A long prompt can occupy an accelerator for a much larger scheduling interval than a single decode iteration. When a serving engine mixes new prefills with requests that are already generating tokens, that difference can show up as irregular time between output tokens. The model has not changed; the interference comes from how two distinct inference phases share execution time.

Prefill processes a prompt and builds the key-value state required by later causal attention. Decode then extends the sequence autoregressively, usually one new token per active request per iteration. Those phases place different pressure on hardware, so treating them as interchangeable scheduling units can produce avoidable stalls.

Chunked prefill changes the unit of scheduling. Instead of admitting an entire long prompt as one indivisible prefill, the scheduler processes a bounded prefix segment, gives active decode work another opportunity to run, then continues with the next segment.

Prefill and decode create different scheduling pressure

For a prompt containing (P) tokens, prefill evaluates those prompt tokens before normal token-by-token generation can continue. Within a transformer layer, causal attention for a prompt position can use earlier prompt positions, and the resulting key-value tensors are retained for subsequent positions.

Decode starts from that retained state. At one decode iteration, each active sequence contributes a small amount of new token work while attention reads the existing cache for its context. The exact compute and memory balance depends on model architecture, batch shape, kernel implementation, parallelism, precision, and accelerator. Still, serving systems commonly observe a useful distinction: prompt processing exposes substantial parallel work, while steady decode repeatedly handles narrow incremental steps.

That distinction matters when both phases occupy the same device. A scheduler that runs a very long prefill in one turn may delay decode requests that were previously emitting tokens at short intervals.

The visible symptom is not necessarily poor total throughput. A system can process many tokens per second and still produce uneven token delivery for an individual request.

A large prefill can become a decode stall

Consider one request, A, already in decode and a new request, B, with a long prompt.

A coarse schedule may resemble:

time -------------------------------------------------------->

A decode | A decode |       B full prefill       | A decode

During B’s prefill, request A has no decode turn. If the prefill interval is large relative to normal decode iteration time, the gap between A’s emitted tokens grows accordingly.

This is a scheduler-level head-of-line effect. It does not require A and B to share text, and it is different from KV-cache capacity pressure. Enough cache memory can exist for both requests while execution time is still monopolized by a large prefill operation.

The problem also differs from ordinary static batching. Continuous serving can admit and retire requests over time, yet a newly admitted prompt can still create a long iteration if prompt work is not bounded.

Chunking bounds the prefill work admitted at once

With chunked prefill, request B is divided into contiguous token ranges:

B prompt:
[ chunk 1 ][ chunk 2 ][ chunk 3 ][ chunk 4 ]

The scheduler can then interleave those chunks with active decode work:

time ---------------------------------------------------------------->

A decode | B chunk 1 | A decode | B chunk 2 | A decode | B chunk 3

The relevant property is that the scheduling interval no longer has to scale with the entire prompt length. It scales with the admitted chunk plus whatever other work the engine places in the same batch.

A production scheduler does not have to alternate exactly as the diagram shows. It may combine a prefill chunk with several decode tokens in one mixed batch, enforce a token budget, prioritize waiting prefills, reserve capacity for decodes, or use separate queues. Chunking supplies a finer scheduling unit; policy still determines which unit runs next.

Chunk boundaries must preserve causal state

Splitting a prefill does not mean evaluating each chunk as an independent sequence. Later chunks require the causal state produced by earlier chunks.

For a prompt token sequence

[ x_1, x_2, \ldots, x_P ]

suppose the first chunk ends at token (c). After processing that chunk, the server retains the key-value tensors corresponding to positions (1) through (c). The next chunk processes positions (c+1) onward with access to that retained prefix state.

A simplified progression is:

chunk 1: tokens 1..c
         -> KV state for 1..c

chunk 2: tokens c+1..2c
         + KV state for 1..c
         -> KV state for 1..2c

chunk 3: tokens 2c+1..
         + earlier KV state
         -> extended KV state

Position handling must also remain consistent with the unchunked sequence. A token in the second chunk is still at its original sequence position; it does not restart at position zero.

With correct cache and position handling, chunking is a serving transformation rather than a change to the intended causal dependency structure. Numerical results can still depend on implementation details such as kernel choice, arithmetic precision, and batching order, so bitwise identity should not be assumed unless the serving stack documents it.

Chunk size sets a latency granularity

Chunk size is not merely a memory knob. It controls how much prefill work the scheduler may place between opportunities for other requests to run.

A larger chunk tends to reduce scheduling overhead and exposes more prompt work per operation. It can also create a longer interval before active decodes are scheduled again.

A smaller chunk creates more interruption points. That can reduce the longest prefill-induced stall, but it can introduce extra scheduler activity, more kernel launches or metadata handling, and less favorable execution shapes in some stacks.

The useful size therefore depends on a concrete target, such as a limit on inter-token latency, time to first token, aggregate throughput, or a combination of service-level constraints. There is no model-independent chunk size that dominates across hardware and workloads.

Token budgets make the policy explicit

One practical formulation is to give each scheduling iteration a token budget (B). Active decode requests consume part of that budget, then prefill work uses the remainder.

For example:

iteration token budget: 2048
active decode tokens:    192
remaining budget:       1856

next prefill chunk <= 1856 tokens

Real serving engines may count work differently, and one token is not a constant-cost unit across all contexts. Attention cost changes with sequence length, and distributed execution adds communication effects. Even so, a budget makes one key constraint explicit: a newly arrived prompt cannot inject unbounded work into a single scheduling turn.

The scheduler can also cap the prefill portion below the total remaining budget when tighter decode latency is required.

Time to first token and inter-token latency can move in opposite directions

Chunking changes more than decode smoothness.

A waiting prompt does not produce its first generated token until its required prefill work has completed. If its chunks repeatedly yield to existing decode traffic, its time to first token can increase relative to a policy that runs the full prompt immediately.

At the same time, existing requests may see smaller gaps between generated tokens because no single prefill occupies the device for as long.

This creates a scheduling tension between:

  • prompt admission latency for newly arriving requests;
  • token delivery regularity for active requests;
  • aggregate accelerator utilization;
  • queue fairness under mixed prompt lengths.

A serving policy should state which latency metric it protects. Optimizing only request throughput can conceal poor inter-token behavior, while optimizing only decode regularity can leave long prompts waiting too long.

Long and short prompts need different scrutiny

The effect is workload-sensitive. If prompts are short, full prefills may already fit comfortably inside the latency budget, and chunking may add complexity without a material benefit.

Long prompts make the scheduling asymmetry more visible. A request carrying a large retrieved context, long conversation state, or sizable code input can create far more prefill work than a short interactive request.

Mixed workloads are especially sensitive because a single queue contains requests with very different prompt costs. Fixed request-count limits do not capture that difference. Two queued requests can represent radically different token volumes.

Prompt-token budgets, estimated execution cost, or phase-aware scheduling provide a more direct control surface than counting requests alone.

Chunking is not prefill-decode separation

Another design is to place prefill and decode on different workers. That removes direct device-time interference between the phases, but introduces a new boundary: the KV state produced during prefill must become available to the decode worker.

Chunked prefill keeps both phases on the same serving resources and reduces interference through scheduling granularity. Phase separation changes resource placement and can allow independent provisioning for prompt processing and generation.

The two approaches solve related pressure at different architectural levels. They can also introduce different bottlenecks: chunking remains subject to local contention, while phase separation must account for state transfer, network topology, queue balance, and capacity on both worker pools.

Measure the gap distribution, not only average throughput

A serving benchmark that reports only tokens per second misses the behavior chunking is intended to control. At minimum, mixed-phase testing should observe time to first token and the distribution of time between generated tokens while long prompts enter the system.

A useful stress case keeps several requests in decode, then injects prompts across a range of lengths. If the scheduler admits full prefills, long prompts can appear as spikes in token gaps. If chunking is effective, those spikes should be bounded by the chosen scheduling policy rather than by the entire prompt size.

The exact bound is implementation-specific because chunk size, batching, kernel duration, queue policy, and parallel execution all contribute to wall-clock latency.

Chunked prefill is therefore best viewed as a control over scheduling granularity. It does not remove prompt computation, reduce the causal state a model needs, or guarantee a universal throughput gain. It gives the serving layer a way to prevent one long prompt from becoming one equally long interruption for every request already decoding.

Artificial Intelligence Language Models Inference LLM Serving Scheduling