An LLM server can receive many requests at the same time, yet processing every request independently is often an inefficient way to use an accelerator. GPUs and similar hardware are designed to perform large amounts of parallel numerical work. A single small request may leave part of that capacity unused.

Batching combines work from multiple requests so the model can process more of it together. This can improve total throughput, but it introduces an important trade-off: waiting to form a batch can delay individual requests, and requests with different sequence lengths do not all consume the same amount of work.

For developers operating LLM applications, the useful question is therefore not simply whether batching is enabled. It is how the serving system should group and schedule tokens while meeting latency and memory requirements.

Start with throughput and latency as separate goals

Two performance measurements are easy to confuse.

Latency describes how long an individual request waits for useful output. For an interactive application, time to first token is especially important because it determines how quickly a response appears to begin.

Throughput describes how much work the service completes over time, such as generated tokens per second across all active requests.

A serving change can improve one while making the other worse. Consider a server that receives four requests a few milliseconds apart:

request A ---->
request B ------>
request C -------->
request D ---------->

Processing each immediately can minimize queueing delay, but may use the accelerator inefficiently. Waiting briefly and processing them together can increase utilization and throughput, but request A now waits for the batch to form.

That is the central batching trade-off.

Why a batch can use hardware more efficiently

Transformer inference is dominated by large numerical operations. Accelerators generally execute these operations more efficiently when enough parallel work is available.

Conceptually, instead of four separate model executions:

[A] -> model
[B] -> model
[C] -> model
[D] -> model

a server may combine compatible work:

[A, B, C, D] -> model

The model still has to perform computation for every sequence. Batching does not make four requests cost the same as one. The benefit comes from executing more useful work together and amortizing some execution overhead.

How much this helps depends on the model, hardware, numerical precision, sequence lengths, serving engine, and current load. Increasing batch size indefinitely does not guarantee proportional throughput gains.

LLM requests are not equal-sized jobs

A request count is a poor approximation of LLM workload size.

Compare these requests:

request A:  120 input tokens,  30 output tokens
request B: 4200 input tokens, 200 output tokens

Both count as one request, but they have very different computational and memory demands. This is why production serving systems often reason about tokens rather than only the number of requests.

The distinction becomes clearer when inference is divided into two phases:

  • prefill processes the input prompt and builds the model state needed for generation;
  • decode generates new tokens autoregressively, usually one token per active sequence at each decoding step.

A long prompt creates substantial prefill work. A long response keeps occupying decode capacity over many steps. A scheduler that ignores these differences can create poor latency even when the nominal request batch looks small.

Static batching is simple but awkward for interactive traffic

The simplest batching strategy waits for a group of requests, processes that group, and finishes the batch before admitting another one.

This works naturally for offline workloads where all inputs are already available:

batch 1: [A, B, C, D] -> finish
batch 2: [E, F, G, H] -> finish

Examples include generating embeddings for a fixed corpus or running model evaluation over a known dataset.

Interactive generation is less convenient. Requests arrive at different times and generate different numbers of tokens. If one sequence finishes quickly while another continues for hundreds of tokens, a rigid batch can leave capacity underused or make later requests wait unnecessarily.

That motivates dynamic scheduling.

Continuous batching changes the batch while generation runs

With continuous batching, also called iteration-level batching in some serving systems, the scheduler can reconsider active work between model iterations. Finished sequences leave, and waiting sequences can enter when capacity becomes available.

A simplified timeline looks like this:

step 1: [A, B, C]
step 2: [A, B, C]
step 3: [A, B]       C finishes
step 4: [A, B, D]    D joins
step 5: [A, D]       B finishes

The exact scheduling algorithm varies by inference engine, but the mental model is useful: the active batch is not necessarily fixed for the lifetime of every request.

This can improve utilization for online workloads because short requests do not reserve a permanent empty slot after they finish. It can also reduce the time newly arrived requests spend waiting for an entire older batch to complete.

Continuous batching is a serving strategy, not a change to the model’s learned behavior. It changes how inference work is scheduled around the model.

Sequence length creates padding and scheduling costs

When sequences of different lengths are represented in a conventional rectangular batch, shorter sequences may need padding so tensor dimensions align.

For example:

A: [token token token token token]
B: [token token PAD   PAD   PAD  ]
C: [token token token PAD   PAD  ]

Naively computing all padded positions wastes work. Modern inference engines can use specialized attention and memory-management techniques to reduce this waste, so the exact cost depends on the serving implementation.

The broader lesson is still important: a batch of similarly sized sequences can behave differently from a batch containing a mixture of very short and very long sequences.

When benchmarking, use the length distribution of real traffic rather than a single convenient prompt length.

Memory puts a practical ceiling on concurrency

More active requests require more runtime state. During autoregressive generation, each active sequence commonly maintains a KV cache containing attention keys and values for tokens already processed.

As active token counts grow, KV-cache memory grows as well. That means the server cannot increase concurrency without considering memory capacity.

A useful operational relationship is:

more active requests
+ longer contexts
+ longer generated outputs
= more active token state
= more memory pressure

This is why a configuration that handles many short conversations may fail to sustain the same concurrency when users submit long documents.

Batch size, context limits, and memory management must be tuned together rather than independently.

Large prefills can interfere with interactive decoding

Imagine several users are already receiving streamed output when another request arrives with a very long prompt. Processing that prompt’s prefill can require a large block of computation.

If the scheduler gives the long prefill too much uninterrupted accelerator time, existing users may observe slower token delivery. If the scheduler always prioritizes decode work, however, large new prompts may wait too long before producing their first token.

This is a scheduling problem rather than a single correct setting. Different products may prefer different policies:

  • a chat interface may prioritize smooth interactive token delivery;
  • an offline generation service may prioritize aggregate throughput;
  • a mixed service may need limits or separate queues for unusually large requests.

Some inference systems can split or chunk large prefills so they coexist more smoothly with decode work. Whether that feature exists and how it behaves is implementation-specific, so it should be evaluated in the actual serving stack rather than assumed.

A batching delay should earn its latency cost

A server under light traffic may not have enough simultaneous requests to fill a large batch. Waiting longer can produce a fuller batch, but every waiting request pays additional queueing latency.

Suppose a service uses a short batching window:

request arrives
      |
      v
wait up to a small window
      |
      +--> compatible work arrives -> batch together
      |
      +--> window expires -> run available work

The useful window depends on traffic and the latency objective. A batch delay that is negligible in a background job may be unacceptable for an interactive autocomplete feature.

Do not choose it from intuition alone. Measure the latency distribution, especially high percentiles, while changing the policy.

Benchmark with a realistic workload shape

A benchmark that sends identical prompts at maximum concurrency can reveal hardware limits, but it does not necessarily predict production behavior.

A more representative test describes at least:

arrival rate
input-token distribution
output-token distribution
concurrent request count
streaming or non-streaming behavior
latency objective

Then record measurements such as:

  • time to first token;
  • inter-token latency or generation rate;
  • end-to-end request latency;
  • total input and output token throughput;
  • queueing time;
  • accelerator utilization;
  • peak memory use;
  • rejected or delayed requests under overload.

Look at percentiles as well as averages. A scheduler can produce an attractive average while a minority of long or unlucky requests experience severe delays.

Backpressure is part of batching design

No batching strategy creates unlimited capacity. Once incoming work exceeds sustainable throughput, the queue grows.

Without an overload policy, a service can enter a bad state where requests remain accepted but wait so long that the application is effectively unavailable. Larger queues may also consume host memory and make recovery slower.

A production design should define what happens when capacity is exhausted. Depending on the product, options can include limiting concurrency, bounding queue size, rejecting excess work, applying per-user quotas, or routing workloads to separate capacity pools.

The correct policy is product-specific, but the principle is general: batching improves how available capacity is used; it does not replace admission control.

Do not optimize throughput in isolation

A higher tokens-per-second number is not automatically a better user experience.

For an interactive assistant, a configuration that increases aggregate throughput by delaying the first token substantially may be a poor trade. For a nightly batch job, the same configuration may be ideal because individual request latency is irrelevant.

Similarly, maximizing the number of concurrent sequences can increase memory pressure and make latency less predictable. The best operating point is usually below a theoretical maximum and depends on the service-level objective.

Tune against the metric the product actually needs.

A practical tuning process

Start with a representative workload and a conservative serving configuration. Measure a baseline before changing batch limits or scheduler behavior.

Then change one dimension at a time:

  1. Measure input and output token-length distributions from realistic requests.
  2. Establish latency and throughput targets.
  3. Increase concurrency or token-batch limits gradually.
  4. Observe throughput, queueing, latency percentiles, and memory together.
  5. Test mixtures of short and long requests rather than only uniform inputs.
  6. Run an overload test to confirm that backpressure behaves intentionally.
  7. Re-run the benchmark when the model, hardware, precision, context limits, or serving engine changes.

This process avoids treating a batch-size value from another system as a universal recommendation.

Conclusion

Batching lets an LLM serving system expose more parallel work to accelerator hardware, which can improve total throughput. The gain is not free: waiting to form batches adds queueing latency, active sequences consume memory, and unequal prompt and output lengths complicate scheduling.

For offline workloads, simple static batches may be sufficient. For interactive generation, continuous batching and token-aware scheduling can use capacity more flexibly as requests arrive and finish.

The practical goal is not the largest possible batch. It is a scheduling policy that keeps the hardware usefully occupied while meeting the application’s latency, memory, and overload requirements. Measure those goals together with realistic traffic, because the best batching strategy is determined by the workload rather than by one universal batch-size setting.

Artificial Intelligence LLM Inference Performance