A thread pool limits how many functions run at the same time, but it does not automatically limit how much work your producer can queue.

That distinction matters when the input is large or unbounded. A loop can submit millions of tasks to a ThreadPoolExecutor while only a handful of worker threads execute them. The remaining tasks are pending Future objects, along with their arguments and other referenced state. If the producer is much faster than the workers, memory use can grow long before CPU or network capacity is exhausted.

The useful mental model is to separate two limits:

worker limit:    how many tasks may execute concurrently
in-flight limit: how many tasks may be submitted but not yet consumed

This article builds a small bounded-submission pattern with concurrent.futures.wait(). The pattern works with long iterables, provides backpressure to the producer, and makes failure and result-order choices explicit.

A worker limit is not a queue limit

Consider a program that uploads generated records:

from concurrent.futures import ThreadPoolExecutor

with ThreadPoolExecutor(max_workers=8) as executor:
    futures = [
        executor.submit(upload_record, record)
        for record in generate_records()
    ]

    for future in futures:
        future.result()

At most eight worker threads execute upload_record() at once. That is the guarantee provided by ThreadPoolExecutor(max_workers=8).

The list comprehension has a different behavior: it keeps consuming generate_records() and calling submit() without waiting for earlier tasks to finish. If the generator produces 500,000 records quickly, the program can create 500,000 futures even though only eight tasks can run concurrently.

The problem is not that the executor has too many workers. The problem is that production is unbounded relative to consumption.

A bounded design stops pulling new input when enough work is already in flight. When one or more tasks finish, it consumes those results and only then admits more work.

That feedback is backpressure.

Start with a fixed in-flight window

The standard library gives us the pieces we need:

  • Executor.submit() schedules one callable and returns a Future.
  • wait(futures, return_when=FIRST_COMPLETED) waits until at least one future finishes or is cancelled.
  • A set can represent the work currently in flight.

Here is the core pattern:

from concurrent.futures import FIRST_COMPLETED, ThreadPoolExecutor, wait

def process_bounded(items, worker, *, max_workers=8, max_in_flight=32):
    if max_workers <= 0:
        raise ValueError("max_workers must be positive")
    if max_in_flight <= 0:
        raise ValueError("max_in_flight must be positive")

    iterator = iter(items)

    with ThreadPoolExecutor(max_workers=max_workers) as executor:
        in_flight = set()

        for _ in range(max_in_flight):
            try:
                item = next(iterator)
            except StopIteration:
                break
            in_flight.add(executor.submit(worker, item))

        while in_flight:
            done, in_flight = wait(
                in_flight,
                return_when=FIRST_COMPLETED,
            )

            for future in done:
                yield future.result()

                try:
                    item = next(iterator)
                except StopIteration:
                    continue

                in_flight.add(executor.submit(worker, item))

The initial loop fills the window. After that, the function waits for completion before taking more input.

If max_in_flight is 32, the function keeps at most 32 submitted futures in its own in-flight set. More importantly, it does not advance items beyond that window. A lazy producer therefore gets a natural signal to slow down.

The worker limit and the window serve different purposes. Eight workers control execution concurrency. A window of 32 allows some work to remain queued so workers can stay busy even if producing the next item has small timing variations.

Understand the bound precisely

“At most 32 in flight” needs a precise definition.

In this function, an item becomes in flight when executor.submit() returns its future. It leaves the set when wait() reports the future as done. A done future may represent a successful call, an exception, or cancellation.

The function never intentionally stores more than max_in_flight futures in in_flight.

There is one subtle detail in the refill loop. wait(..., FIRST_COMPLETED) can return more than one done future. Several tasks may finish before the waiting thread resumes. The code removes all of them, consumes each result, and submits at most one replacement for each completed future.

That preserves the bound.

This is an application-level admission bound. It does not describe the executor’s private internal data structures, and it should not be presented as a guarantee about their implementation. The guarantee we control is simpler: our code never calls submit() again until space exists in its own window.

Backpressure also limits how far a generator runs ahead

Bounding futures is useful even when each future is small because input generation can have side effects or hold resources.

Consider:

def read_jobs(path):
    with open(path, "rt", encoding="utf-8") as file:
        for line in file:
            yield parse_job(line)

With eager submission, the caller can consume the whole file before workers have processed much of it.

With process_bounded(), iteration pauses whenever the in-flight window is full:

for result in process_bounded(
    read_jobs("jobs.ndjson"),
    process_job,
    max_workers=8,
    max_in_flight=32,
):
    store_result(result)

Only enough jobs are parsed to fill available capacity. When workers fall behind, the file iterator stops advancing until completions create room.

This property is especially useful for generators backed by files, database cursors, paginated APIs, or expensive transformations. It does not make those sources safe automatically; it simply prevents the thread-pool submission loop from draining them eagerly.

Completion order is not input order

The bounded function yields futures from the done set. Results therefore become visible according to completion timing, not according to input position.

Suppose the input is:

A: 800 ms
B:  20 ms
C:  40 ms

A completion-oriented consumer can observe B and C before A.

That is often desirable for independent work because one slow task does not prevent the caller from handling results that are already ready.

It is not appropriate when output order is part of correctness.

If order matters, one option is to attach sequence numbers and reorder completed results before emitting them. That requires buffering results that finish ahead of the next expected sequence number, so the memory trade-off changes. Another option is to use an API whose ordering semantics already match the problem.

Do not accidentally promise input order just because the input itself is sequential.

Exceptions need an explicit policy

Future.result() returns the worker’s value when the call succeeds. If the worker raised an exception, result() raises that exception in the consuming thread.

In the simple implementation:

for future in done:
    yield future.result()

the first failed future encountered stops iteration.

That is a reasonable fail-fast policy for some programs, but understand what happens next. The executor is inside a with statement. Leaving that block calls its shutdown behavior and, by default, waits for already submitted work to finish. Tasks that have already started are not magically interrupted because one sibling failed.

If you need to stop accepting new work after a failure, the current function already does so once future.result() raises. If you also want to request cancellation of work that has not started, handle that policy explicitly:

try:
    for future in done:
        yield future.result()
except BaseException:
    for pending in in_flight:
        pending.cancel()
    raise

Future.cancel() succeeds only when the call has not started running. It cannot forcibly stop a worker function that is already executing.

For tasks with external side effects, cancellation of the future is not a rollback mechanism. Design idempotency, transactions, or compensating actions at the layer that owns those side effects.

A reusable version should validate its limits

There is no correctness requirement that max_in_flight be greater than or equal to max_workers. A smaller window is valid, but then some worker threads can never be occupied because fewer tasks are admitted than workers exist.

For a general helper, it is clearer to validate only positive values:

def validate_limits(max_workers, max_in_flight):
    if max_workers <= 0:
        raise ValueError("max_workers must be positive")
    if max_in_flight <= 0:
        raise ValueError("max_in_flight must be positive")

Then document the performance implication separately:

max_in_flight < max_workers
    valid, but at most max_in_flight tasks can be active

max_in_flight == max_workers
    no extra queued cushion

max_in_flight > max_workers
    some tasks can wait behind running work

A modest cushion can help when task durations vary. A huge cushion may increase memory use and make shutdown or failure handling less responsive without improving throughput.

The right number depends on task cost, input cost, resource budgets, and how much queued work the application is willing to own.

Keep task arguments small when possible

A bound limits the number of submitted tasks, not the size of each task’s retained state.

This still matters:

payload = load_200_megabyte_payload()
executor.submit(process_payload, payload)

If 20 in-flight tasks each retain a distinct 200 MB payload, a bounded window can still require several gigabytes.

Prefer passing compact identifiers when the worker can safely load the data it needs:

executor.submit(process_document, document_id)

That is not universally better. Loading inside workers may move I/O into the concurrency-limited section or cause repeated reads. The point is to account for what each pending future keeps reachable when choosing the window size.

Memory reasoning should use both dimensions:

approximate retained work
    ≈ in-flight task count × retained state per task

This is a planning model, not an exact measurement of Python object memory.

Do not confuse backpressure with rate limiting

A bounded in-flight window limits outstanding work inside this process.

It does not mean “send at most 100 requests per second.”

If eight workers each complete a request in 10 milliseconds, they can collectively start far more than 100 operations per second while still respecting a 32-task in-flight bound.

Rate limiting controls starts or completions over time. Backpressure controls how much unfinished work the producer may get ahead of the consumer.

You may need both:

thread pool       -> limits simultaneous worker execution
in-flight window  -> limits admitted unfinished work
rate limiter      -> limits operation frequency over time

Treating these as separate controls makes production behavior easier to reason about.

Do not hold scarce resources while waiting for admission

Backpressure moves waiting to the producer. That is usually the goal, but it changes where resources can remain held.

Risky structure:

with acquire_database_transaction() as transaction:
    for result in process_bounded(rows, worker):
        transaction.write(result)

If worker calls are slow, the transaction can stay open for the entire processing period.

The same issue applies to locks, leases, sockets, temporary files, and other scoped resources. A bounded pipeline does not shorten those lifetimes for you.

Acquire scarce resources as late as practical and release them as early as correctness permits. When a worker can own a short-lived resource independently, that is often easier to reason about than keeping one outer resource open while waiting for the whole pool.

Executor.map has different trade-offs

Executor.map() is concise:

with ThreadPoolExecutor(max_workers=8) as executor:
    for result in executor.map(worker, items):
        consume(result)

Its result iterator preserves input order. That can be exactly what you want.

Its submission behavior depends on the Python version and arguments. In Python versions before 3.14, Executor.map() collects its input iterables eagerly. Python 3.14 added a buffersize argument that can limit the number of submitted tasks whose results have not yet been yielded.

On Python 3.14 or newer, an ordered bounded pipeline can therefore be as simple as:

with ThreadPoolExecutor(max_workers=8) as executor:
    for result in executor.map(worker, items, buffersize=32):
        consume(result)

Use that when its ordering and version requirements fit your application.

The manual submit() plus wait() pattern remains useful when you need completion-order handling, custom refill behavior, per-future metadata, or compatibility with Python versions that do not provide map(..., buffersize=...).

Do not pass buffersize on an older Python runtime; that parameter was added in Python 3.14.

Avoid waiting on the same pool from inside its workers

Bounded submission does not remove the usual thread-pool deadlock hazards.

A worker that submits more work to the same executor and then waits synchronously for that work can deadlock when all worker threads are occupied by callers doing the same thing.

The smallest example is a one-thread pool:

def outer(executor):
    future = executor.submit(inner)
    return future.result()

If outer() itself occupies the executor’s only worker, inner() cannot start because no worker is free, while outer() waits forever for inner().

The bounded producer pattern avoids this by keeping orchestration in the calling thread. Workers perform their unit of work and return results; they do not synchronously depend on additional tasks in the same saturated pool.

If tasks have dependency relationships, model those dependencies explicitly rather than assuming a larger in-flight window will solve them.

Choose threads for the work they can actually overlap

ThreadPoolExecutor is often useful for work that spends significant time waiting on I/O, such as network calls or file operations.

It does not guarantee speedup for arbitrary CPU-heavy Python code. Whether threads can execute CPU work in parallel depends on the runtime and on whether the code involved releases or otherwise avoids the interpreter’s execution constraints.

Backpressure is still valuable regardless of that distinction: it controls queued work. But do not interpret a bounded thread pool as a general recipe for making CPU-bound algorithms faster.

For CPU-heavy workloads, evaluate process-based or other execution models based on serialization cost, data size, runtime behavior, and deployment constraints.

When to use a bounded submission window

Use this pattern when:

  • the input can be very large or effectively unbounded;
  • producing items is faster than processing them;
  • pending tasks retain enough state that unbounded submission is risky;
  • completion-order processing is acceptable or useful;
  • you want explicit control over admission and failure handling.

A simple executor.map() can be clearer for small finite inputs, especially when ordered results matter and memory is not a concern. On Python 3.14 or newer, map(..., buffersize=...) is often the simplest standard-library choice for ordered bounded mapping.

A queue-based producer/consumer design may be a better fit when workers need to run continuously, multiple producers feed the same pipeline, or task admission has a lifecycle independent of one function call.

Bound the work you own

max_workers answers one question: how many worker threads may execute calls concurrently.

Production systems often need a second answer: how much unfinished work is this process willing to own?

A bounded in-flight window makes that limit explicit. Fill a small window, wait for completions, consume their results, and only then pull more input. The producer naturally slows when workers cannot keep up.

That pattern does not solve rate limiting, rollback, task dependencies, or per-task memory costs. It does give those concerns a stable boundary: instead of an arbitrarily growing backlog, the application has a deliberate amount of admitted work it can reason about.

Python Concurrency Standard Library Performance