Concurrency with thread pools for waiting work
Overlap tasks that mostly wait, such as network or disk requests, and keep results in order.
import time
from concurrent.futures import ThreadPoolExecutor
def fetch(n):
time.sleep(0.2)
return n * n
start = time.perf_counter()
sequential = [fetch(n) for n in range(5)]
sequential_time = time.perf_counter() - start
start = time.perf_counter()
with ThreadPoolExecutor(max_workers=5) as pool:
overlapped = list(pool.map(fetch, range(5)))
overlapped_time = time.perf_counter() - start
print(sequential == overlapped, overlapped)
print(overlapped_time < sequential_time / 2)True [0, 1, 4, 9, 16] True
Explanation
Concurrency means making progress on several tasks during the same period, for example by switching between them while each waits. Parallelism means running tasks at literally the same instant on different processor cores. A program can be concurrent without being parallel. The two are different tools for different problems.
Many programs spend most of their time waiting for the network, a disk, or a database. These are I/O-bound tasks. A thread pool starts a fixed number of worker threads and hands them tasks, so one task's waiting overlaps with another's. Python's concurrent.futures.ThreadPoolExecutor provides this, and its map method returns results in the order of the inputs, not the order in which they finish.
The example uses time.sleep to stand in for waiting on a remote service. Five tasks of 0.2 seconds take about one second one after another, but about 0.2 seconds in a pool of five. This works because a sleeping or I/O-waiting thread releases CPython's global interpreter lock (GIL). Threads do not speed up CPU-bound pure Python code in the default CPython build, because only one thread runs Python bytecode at a time; that case needs processes, as the next lesson shows.
Explore the dry run
import timefrom concurrent.futures import ThreadPoolExecutor def fetch(n): time.sleep(0.2) return n * n start = time.perf_counter()sequential = [fetch(n) for n in range(5)]sequential_time = time.perf_counter() - start start = time.perf_counter()with ThreadPoolExecutor(max_workers=5) as pool: overlapped = list(pool.map(fetch, range(5)))overlapped_time = time.perf_counter() - start print(sequential == overlapped, overlapped)print(overlapped_time < sequential_time / 2)Follow the example
Start to inspect the source, values, and output. These prepared steps explain the published sample; grouped loop steps describe multiple iterations.
View the complete dry-run table
| Step / line | What happens | State after step | Output so far |
|---|---|---|---|
| The task waits 0.2 seconds, standing in for a network or disk request. While it sleeps, the thread does not hold the GIL. | — | — | |
| One after another, the five waits add up to about one second. | ≈ 1.0 s | — | |
| Five worker threads wait at the same time, so the total is close to a single 0.2 s wait. | ≈ 0.2 s | — | |
| map returns results in input order, so both lists are equal. | ≈ 0.2 s | True [0, 1, 4, 9, 16] | |
| The pooled version took less than half the sequential time. | ≈ 0.2 s | True [0, 1, 4, 9, 16] True |
Cost and common mistakes
With w workers and n equal waiting tasks of duration t, total time is about ceil(n / w) × t instead of n × t.
- Threads do not make CPU-bound pure-Python code faster in the default build. Use processes or native extensions for that.
- Exceptions raised in a worker are re-raised when you read its result. Handle them where you collect results.
- Too many threads waste memory and can overload the service you are calling. Size the pool to the work and to the remote limits.
Try it yourself
How long would 10 tasks of 0.2 s take with max_workers=5?
Reveal answer and reasoning
About 0.4 seconds: two rounds of five concurrent tasks, ceil(10 / 5) × 0.2.