google/pegasus-xsum is a summarization checkpoint, not a compact text utility. Its latency follows directly from the work performed by a large Transformer encoder-decoder: first encode the source document, then run the decoder repeatedly until the summary is complete. On a CPU, the second phase is usually the part that makes a short output feel disproportionately expensive.

The original PEGASUS work describes a Transformer encoder-decoder pretrained with Gap Sentences Generation and reports a 568M-parameter best model. The XSum checkpoint is fine-tuned for highly abstractive single-document summarization. That combination is useful when summary quality matters, but it also means inference has substantially more machinery than extracting a few source sentences or running a small classifier.

The useful performance question is therefore not simply whether PEGASUS is “large.” It is where time is spent, which settings multiply that work, and which optimizations preserve the behavior that matters.

Summarization has an encode phase and a generate phase

For an input sequence with (N) source tokens, the encoder builds contextual representations for the source. This work happens once per request. A simplified view is:

source text
    |
tokenizer
    |
N source tokens
    |
encoder
    |
source representations
    |
decoder step 1 -> token 1
decoder step 2 -> token 2
decoder step 3 -> token 3
...
decoder step T -> token T

The decoder is autoregressive. If the summary contains (T) generated tokens, generation requires a sequence of decoder steps rather than one pass that emits the entire summary at once.

Caching prevents each decoder step from recomputing all earlier decoder key-value states, but it does not eliminate the repeated forward work. Each new token still requires the model to evaluate the next-token distribution and attend to the encoded source.

This is the first reason latency can look surprising: a 60-token summary is not one output operation. It is roughly 60 sequential generation decisions after the source has already been encoded.

Source length affects the encoder and every cross-attention step

Longer source text increases the initial encoder workload. Self-attention in a conventional Transformer also becomes more expensive as sequence length grows because positions interact across the sequence.

The effect does not end when encoding finishes. Decoder layers use cross-attention over encoder outputs. A longer source therefore leaves a larger encoded memory for the decoder to attend to while generating each output token.

This makes aggressive source-length limits one of the most direct latency controls. The trade-off is semantic rather than merely computational: truncating a document can remove information needed for the summary.

For long documents, blindly taking the first fixed number of tokens is often a poor production policy. A better architecture may segment the document, select relevant passages, summarize chunks, or use a model designed for longer contexts. Those approaches change the quality and latency profile, so they should be evaluated against the actual document distribution rather than treated as interchangeable optimizations.

Beam search can multiply decoder work

Generation policy matters almost as much as model size.

Greedy decoding keeps one candidate sequence at each step:

num_beams = 1

step 1 -> one candidate
step 2 -> one candidate
step 3 -> one candidate

Beam search keeps several candidate sequences alive:

num_beams = 4

step 1 -> four candidates
step 2 -> four candidates
step 3 -> four candidates
...

The exact runtime increase is not a clean four-times multiplier because batching, memory bandwidth, kernels, cache handling, and hardware utilization all matter. Still, a larger beam width increases decoder work and memory traffic. On CPU, the difference can be substantial.

For an application where latency is more important than the quality gain from beam search, num_beams=1 is the first configuration worth benchmarking. If greedy decoding degrades summaries too much, num_beams=2 provides an intermediate point. The decision should come from measurements on representative text, not from a default inherited from an example.

Output length is a latency budget

Because generation is autoregressive, max_new_tokens is also a compute limit.

A request allowed to generate 160 tokens has a much larger worst-case decoder budget than one capped at 60. Setting a large maximum “just in case” makes tail latency harder to control, especially when the stopping behavior varies across documents.

For concise XSum-style output, a tight output budget is usually more coherent with the checkpoint’s purpose than permitting long multi-paragraph generation.

A latency-oriented generation call can start with:

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

model_id = "google/pegasus-xsum"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSeq2SeqLM.from_pretrained(model_id)
model.eval()

inputs = tokenizer(
    text,
    return_tensors="pt",
    truncation=True,
    max_length=512,
)

summary_ids = model.generate(
    **inputs,
    num_beams=1,
    max_new_tokens=64,
)

summary = tokenizer.decode(
    summary_ids[0],
    skip_special_tokens=True,
)

The values are operating choices, not universal optimums. A 512-token source cap may be unacceptable for documents whose important facts occur late, and a 64-token output cap may be too short for another product. Their value is that they make the latency budget explicit.

CPU execution exposes the model’s full cost

A GPU can execute the dense matrix operations behind Transformer inference with much higher parallel throughput than a typical general-purpose CPU. CPU inference remains useful for low request rates, environments without accelerators, offline jobs, and deployments where operational simplicity matters more than latency.

But moving PEGASUS-XSum to CPU does not make its architecture smaller. The encoder, decoder, attention layers, and model weights are still there.

This distinction matters when comparing it with a much smaller general-purpose model. A small decoder-only model may have fewer parameters and cheaper individual steps, even if it was not trained specifically for summarization. PEGASUS-XSum may have the more appropriate task specialization while still losing decisively on latency.

Model selection should therefore separate two questions:

Does the model produce the summary quality and style we need?
Can the deployment hardware deliver it within the latency budget?

A model can pass the first test and fail the second.

Measure tokenization, encoding, and generation separately

A single end-to-end timer tells you that inference is slow but not what to change.

For performance work, split the request into at least tokenization and generation, then inspect input and output token counts. On GPU, synchronize the device around timing boundaries; asynchronous execution otherwise makes measurements misleading.

A minimal CPU benchmark can record:

from time import perf_counter

t0 = perf_counter()
inputs = tokenizer(
    text,
    return_tensors="pt",
    truncation=True,
    max_length=512,
)
t1 = perf_counter()

with torch.inference_mode():
    summary_ids = model.generate(
        **inputs,
        num_beams=1,
        max_new_tokens=64,
    )
t2 = perf_counter()

print({
    "input_tokens": inputs["input_ids"].shape[-1],
    "output_tokens": summary_ids.shape[-1],
    "tokenization_s": t1 - t0,
    "generation_s": t2 - t1,
})

Run the same corpus with beam widths 1, 2, and 4. Then repeat across short, medium, and long inputs. This reveals whether the bottleneck follows source length, generated length, decoding policy, or a combination of them.

Warm-up also matters. The first inference can include one-time costs that do not represent steady-state requests. Report warm and cold latency separately when startup behavior matters.

Batch throughput and single-request latency are different targets

Batching several documents can improve hardware utilization, particularly on an accelerator. It does not automatically make an individual document finish sooner.

A service optimized for offline throughput may intentionally accumulate requests into batches. An interactive API usually cares more about per-request latency and tail latency. The same PEGASUS-XSum deployment can require different batch sizes depending on which metric is binding.

Variable-length documents add padding overhead. Grouping inputs with similar lengths can reduce wasted work in batch-oriented pipelines.

Precision and optimized runtimes help after decoding is bounded

Reduced precision, quantization, graph compilation, ONNX-based runtimes, and hardware-specific kernels can improve inference efficiency. Their gains depend on hardware and operator support, and they can change numerical behavior.

These optimizations are most useful after obvious generation costs are controlled. Quantizing a model while leaving an unnecessarily large beam width and oversized generation limit may optimize the wrong layer of the problem.

A practical order is:

1. measure representative requests
2. bound source and output lengths deliberately
3. benchmark greedy versus beam decoding
4. choose CPU or accelerator from the latency target
5. then evaluate precision, quantization, compilation, and batching

This order makes each change measurable and keeps quality regressions attributable to a specific decision.

The model may simply be too large for the latency target

There is a point where tuning generate() stops being the right answer. If a CPU service needs consistently low latency under concurrency, a 568M-parameter encoder-decoder may be a poor fit regardless of careful beam settings.

At that boundary, replacing the model is an architectural decision. A smaller summarization model, a multilingual sequence-to-sequence model, a compact instruction model, or an extractive pipeline may fit the workload better. Each changes the failure modes: multilingual coverage, factuality, output style, controllability, and document-length behavior all need fresh evaluation.

PEGASUS-XSum is slow for understandable reasons. It encodes the source with a substantial Transformer and then performs sequential decoder work for every generated token; beam search, long sources, and loose output limits add more work on top. Once those costs are measured separately, the choice becomes concrete: reduce decoding work, move it to hardware that can sustain it, or choose a model whose architecture better matches the latency budget.

References

  • Zhang, J., Zhao, Y., Saleh, M., and Liu, P. J. “PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization.” ICML 2020.
  • Hugging Face model card: google/pegasus-xsum.
Artificial Intelligence Summarization PEGASUS Inference Performance