Autoregressive generation does not finish every request at the same iteration. One sequence may emit a stop token after a few decode steps while another remains active for hundreds more. A static batch keeps those requests coupled until the batch boundary. Continuous batching breaks that coupling by letting the active set change between model iterations.

The key change is scheduling granularity. A request is no longer the indivisible scheduling unit for the entire generation. The serving system can construct an execution batch for one iteration, update request state after that iteration, remove finished sequences, and admit queued work before the next execution step.

Static batches retain request-level coupling

Consider three requests admitted together:

request A:  18 decode iterations
request B:  73 decode iterations
request C: 141 decode iterations

With request-level batching, the group remains structurally tied to the longest request. Shorter requests may stop contributing useful token work long before the group drains. A newly arrived request can also remain queued even when some capacity inside the original group has become idle.

This behavior is especially mismatched with generation workloads because output length is generally not fixed at admission time. Stop conditions, maximum-token limits, and generated token sequences determine when each request exits. A batch formed from requests with similar prompt sizes can still diverge substantially during decoding.

Continuous batching moves the boundary inward. After an iteration completes, the scheduler can reconsider the active set. The execution shape is therefore allowed to evolve over the lifetime of a request.

Iteration-level scheduling changes batch membership

A simplified state transition looks like this:

queue -> admitted -> active -> finished
                    ^    |
                    |____|
                 next iteration

At each scheduling point, the system has several categories of state: queued requests waiting for admission, active requests with retained generation state, and requests that have just completed. The scheduler selects work for the next model invocation subject to memory, token, and implementation-specific limits.

This mechanism does not imply that every engine uses the same scheduling policy. One system may prioritize decode work, another may admit prefills eagerly, and another may split long prefills into chunks. Continuous batching describes the ability to change membership across execution iterations; priority, fairness, admission control, and preemption remain separate policy decisions.

That distinction matters when comparing serving systems. Two engines can both support continuous batching yet produce different latency distributions because their schedulers make different choices at the same scheduling boundaries.

KV state makes admission a memory decision

A decoding request carries state across iterations. In Transformer serving, that commonly includes a key-value cache representing prior tokens for attention. Removing a finished request can release cache capacity, while admitting a new request creates new state that must fit within the engine’s memory policy.

As a result, an available batch slot is not sufficient evidence that another request can enter. Admission can depend on free KV-cache capacity, prompt length, configured token budgets, cache block allocation, and other engine-specific constraints.

The scheduler therefore operates across at least two resource dimensions: execution work for the next iteration and persistent state retained between iterations. A policy that fills compute capacity aggressively can still run into memory pressure if active sequences accumulate large caches.

Paged or block-based KV-cache managers can make allocation more flexible, but they do not remove this constraint. They change allocation mechanics. The serving layer still needs a policy for deciding which request state remains resident and which new work can be admitted.

Prefill and decode create asymmetric work

Continuous batching is often described through decode iterations, but an incoming request first needs its prompt processed. Prefill can involve many input tokens at once, while a conventional autoregressive decode step advances an active sequence by one generated token.

Mixing those phases creates a scheduling tension. A large prefill can add substantial work to an iteration that also contains latency-sensitive decode requests. Deferring all prefills protects decode cadence but can leave queued requests waiting and can reduce aggregate utilization under some workloads.

Continuous batching alone does not resolve that tension. Systems may use phase prioritization, token budgets, chunked prefill, or other mechanisms. Those are additional designs layered on the same ability to alter batch membership at fine scheduling boundaries.

The architectural boundary is useful: dynamic membership supplies an opportunity to schedule; policy determines how that opportunity is used.

Throughput and latency depend on workload shape

Iteration-level scheduling removes a specific source of idle capacity: completed requests no longer need to keep their original batch intact. It also allows queued requests to enter without waiting for every sequence in an earlier group to finish.

Those properties do not establish a fixed performance multiplier. The effect depends on arrival rate, prompt lengths, output-length distribution, memory capacity, model size, parallelism strategy, scheduler policy, and accelerator behavior. Under low concurrency, there may be little queued work available to fill newly released capacity. Under heavy load, aggressive admission can improve utilization while also changing queueing and per-token latency.

For that reason, continuous batching is better treated as a scheduling mechanism than as a throughput guarantee. Measurements should separate request latency, time to first token, time between output tokens, queueing delay, throughput, and memory occupancy. A single aggregate tokens-per-second number can hide a scheduler that favors one phase or request class at the expense of another.

Dynamic membership is the core boundary

The essential property is narrow: the set of requests participating in execution can change at iteration-level scheduling points. Finished sequences can leave, and eligible queued work can enter, without draining a fixed request group first.

Everything around that property remains a serving-system design choice. Cache allocation determines how persistent state is represented. Admission control decides whether new work fits. Scheduling policy orders prefill and decode work. Fairness rules decide how service is distributed across clients or requests. Preemption determines whether active work can be displaced.

Separating these concerns prevents continuous batching from becoming a catch-all label for inference optimization. Its technical contribution is the removal of static request-group membership across the full autoregressive lifetime, giving the scheduler repeated opportunities to reshape execution as requests arrive, advance, and finish.

Artificial Intelligence LLM Inference Scheduling Serving Systems