Two prompts can communicate roughly the same amount of information and still consume very different numbers of model tokens. The difference can appear between languages, writing systems, domains, or even formatting styles.

That matters because language-model systems usually operate on tokens rather than characters or words. A context window is measured in tokens. Many hosted APIs account for usage in tokens. Longer token sequences can also increase inference work, although the exact latency and compute effect depends on the model, serving stack, batching, caching, and whether the tokens belong to the input or generated output.

For a multilingual application, a token budget that works comfortably for English may therefore behave differently for Indonesian, Japanese, Arabic, or another language. The right response is not to assume that one language is inherently “expensive.” It is to measure how the tokenizer used by the actual model represents the text your application receives.

This article develops a practical way to measure tokenization efficiency, explains which comparisons are meaningful, and shows how to turn the results into safer context and capacity decisions.

Start with the tokenizer, not the visible text

A tokenizer converts text into the discrete token IDs consumed by a language model. Depending on the tokenizer, a token may correspond to a whole word, part of a word, punctuation, whitespace together with nearby text, a byte-derived unit, or another learned text fragment.

Consider these two fictional tokenizer outputs:

Text:   "reset password"
Tokens: ["reset", " password"]
Count:  2

and:

Text:   "unresettable"
Tokens: ["un", "reset", "table"]
Count:  3

The examples are deliberately simplified; real token boundaries depend on the specific tokenizer. The important point is that token count is not reliably determined by word count or character count.

A tokenizer’s vocabulary was built according to a particular algorithm and training corpus. Text patterns that are represented efficiently by that vocabulary can map to fewer tokens, while unfamiliar character sequences or less well-represented patterns may split into more pieces.

So the useful question is not:

How many words does this language use?

It is:

For this model's tokenizer and this workload,
how many tokens are needed to represent the text we care about?

That framing keeps the measurement tied to the deployed system.

Why token count changes system behavior

Suppose an application reserves 8,000 tokens for the complete model input. A request contains instructions, conversation history, retrieved documents, and the current user message.

If one workload sample becomes 6,000 tokens after tokenization, 2,000 tokens remain in that input budget. If a comparable sample becomes 7,500 tokens, only 500 remain.

Nothing about the visible character count tells you that directly.

Token count can affect several practical constraints.

Context capacity

Model context limits are defined in tokens. More tokens used by the prompt leave fewer tokens available for other input or, depending on the API and model interface, for generation within the applicable context limit.

This is especially important in retrieval-augmented generation. A retrieval system may select five passages because they fit an English test set, then overflow or truncate when the same product is used with text that tokenizes more densely.

Usage-based cost

When a provider prices input or output by token count, tokenization directly affects the billed quantity. The exact price is provider- and model-specific, so token count should be measured separately from currency cost. Convert tokens to money only with the pricing rules that apply to the deployed model.

Inference work

Sequence length influences Transformer computation and memory use, but translating token count into latency is not as simple as multiplying by a constant. Prompt processing and autoregressive generation have different execution patterns. Attention implementations, KV caching, hardware utilization, batching, and serving policies also matter.

Token count is therefore a useful capacity signal, not a universal latency formula.

Choose a denominator that matches the question

A common mistake is to report “tokens per word” as if it were a universal measure of tokenizer quality.

It can be useful inside a language with clear and consistently defined word segmentation. Across languages, however, the definition of a word may not be comparable. Some writing systems do not mark word boundaries with spaces in the same way English does, and different segmentation tools can produce different word counts.

For cross-language engineering comparisons, use metrics whose denominator you can define consistently.

Tokens per character

For a text containing T tokens and C Unicode characters:

tokens_per_character = T / C

This is easy to compute, but Unicode characters do not represent equal amounts of information or equal storage. Combining marks and normalization can also complicate apparently simple character counts.

Tokens per byte

Using the UTF-8 encoded size B:

tokens_per_byte = T / B

This has an unambiguous denominator for a given byte encoding and can be useful for reproducible engineering measurements. It still does not mean that equal byte counts carry equal semantic content.

Tokens per example

For an application dataset, often the most useful metric is simply the distribution of token counts per real request, document, message, or conversation.

If your system processes support tickets, measure tokens per support ticket. If it summarizes articles, measure tokens per article. This metric directly answers capacity questions even when the texts are not exact translations of one another.

No single denominator captures every notion of efficiency. Pick the metric according to the decision you need to make.

Compare like with like

Imagine that you want to compare tokenization for English and Indonesian customer-support messages.

A weak experiment takes 1,000 short English password-reset messages and 1,000 long Indonesian billing complaints. If the Indonesian set has more tokens, you cannot tell whether language, topic, or message length caused the difference.

A stronger design controls the content being compared.

For a translation-based benchmark, start with aligned examples that express the same intended content in each language:

example_id | language | text
-----------|----------|--------------------------------------
42         | en       | I cannot sign in to my account.
42         | id       | Saya tidak dapat masuk ke akun saya.

Then tokenize every version with the same model tokenizer and record the resulting count.

This does not make translations perfectly equivalent. Natural translations can differ in wording and length. But paired examples reduce a major source of noise because each row is trying to communicate the same underlying message.

For production capacity planning, a second benchmark should use naturally occurring traffic. Translation pairs answer, “How does this tokenizer represent comparable content?” Real traffic answers, “What token lengths will this application actually receive?” Both questions are useful, but they are not interchangeable.

Measure distributions instead of one average

Suppose a multilingual support benchmark produces these fictional token counts:

Language A: 18, 20, 21, 22, 24
Language B: 16, 18, 19, 20, 40

Both sets are small, but they demonstrate why a mean alone can hide operational risk. Language B has one much longer example. A system that truncates requests at a fixed threshold may care more about the upper tail than about the average.

For each workload slice, record at least:

  • the number of examples;
  • median token count;
  • a high percentile such as p90 or p95 when the sample is large enough to make that estimate useful;
  • maximum token count for diagnostic purposes;
  • the same statistics for a stable denominator such as characters or bytes when cross-language comparison is needed.

Percentiles should be interpreted with the dataset size in mind. A p99 from a few dozen examples is not a stable description of rare production behavior.

Also keep the raw per-example counts during analysis. Aggregate statistics can tell you that a tail exists; individual examples help explain why.

Use paired ratios carefully

With aligned translations, a paired ratio can make differences easier to inspect.

For example i, let T_A(i) and T_B(i) be token counts for two language versions. Define:

ratio_i = T_B(i) / T_A(i)

If one pair uses 30 tokens in A and 45 in B:

ratio = 45 / 30 = 1.5

For that example, version B requires 50% more tokens under this tokenizer.

Do not turn one ratio into a claim about an entire language. Calculate ratios over a representative paired dataset and inspect their distribution. A tokenizer may behave differently on conversational text, source code, product identifiers, addresses, or specialized terminology.

It is also useful to inspect outliers manually. A large ratio can come from a language-wide vocabulary pattern, but it can also come from a URL, unusual Unicode sequence, transliterated name, or formatting artifact.

Normalize text only when the application does

Unicode allows some visually similar text to have different underlying code-point sequences. Normalization can sometimes make those representations consistent before tokenization.

But benchmark preprocessing should mirror production preprocessing. If the application sends raw user text to the model, measuring only aggressively normalized benchmark text can hide the behavior you will actually see.

The same rule applies to whitespace, markup, case conversion, and repeated separators. Do not clean a benchmark merely to make token counts look better.

Instead, define the pipeline explicitly:

raw application text
        |
        v
production preprocessing
        |
        v
model tokenizer
        |
        v
token count

Then measure at the output of that exact pipeline.

If you are considering a new normalization step, evaluate it as a product change. Check not only token counts but also whether the transformation preserves the text needed for the task.

Turn measurements into a context budget

Tokenization analysis becomes useful when it changes system design.

Suppose your RAG application has a fixed input allowance and the production measurements show that one language slice has a substantially longer upper-tail token count than the others.

Several responses are possible.

You can reduce the number or size of retrieved passages for requests that need more space. You can summarize or compress conversation history. You can reserve budgets dynamically after tokenizing the fixed parts of the request. You can also choose a model with a larger context window if the quality and cost trade-offs justify it.

A simple budgeting sequence is:

1. tokenize fixed instructions
2. tokenize current user input
3. reserve required output capacity if the interface needs it
4. compute the remaining input allowance
5. add optional context only while it fits

The exact accounting rules depend on the model API. Some interfaces add special tokens or message-formatting overhead internally. Use the provider’s documented counting method when available, or the exact supported tokenizer and chat template when token counts must match the service.

Do not assume that encoding only the visible message strings reproduces a hosted API’s final token accounting.

Token efficiency is not model quality

A tokenizer that represents a text in fewer tokens is not automatically paired with a better model.

Model quality depends on training data, architecture, optimization, model capacity, alignment, and many other factors. Tokenization is one part of the system. A model can use more tokens for a language and still produce better answers in that language than another model with a more compact tokenizer.

Likewise, comparing vocabulary sizes alone is insufficient. A larger vocabulary can represent more strings directly, but it also changes the embedding and output layers and does not guarantee that the vocabulary is well allocated for your workload.

Evaluate tokenization efficiency alongside task quality. For a multilingual support assistant, a useful comparison might include:

per language:
- task success or answer-quality metric
- input token distribution
- output token distribution
- end-to-end latency
- actual request cost

This prevents an optimization for compact tokenization from silently degrading the thing users care about.

Common measurement mistakes

Comparing different tokenizers and blaming the language

If model A and model B use different tokenizers, a token-count difference reflects both the text and tokenizer design. State which tokenizer produced every result. Never treat tokens as a model-independent unit.

Using English words as the universal denominator

Word-based metrics can be convenient for one corpus but misleading across writing systems. Prefer application examples, bytes, characters with a documented counting rule, or aligned pairs depending on the question.

Testing only translated benchmark text

Translations control semantic content, but they may not resemble natural user traffic. Production text can contain slang, mixed languages, names, code, URLs, emoji, and domain-specific vocabulary. Validate on real workload samples as well.

Reporting only the mean

Context failures occur on long examples, so inspect the upper tail. A good average does not protect a system from truncation at p95 or from a small set of extreme inputs.

Assuming fewer tokens means proportionally lower latency

Token count affects workload, but serving behavior is more complicated. Measure end-to-end latency on the deployed stack if latency is the decision variable.

Changing text to optimize token count without checking meaning

Removing whitespace, transliterating text, or applying normalization can alter readability or semantics. A preprocessing change is safe only when it preserves what the model and user need.

When tokenization efficiency should influence model choice

Tokenization deserves explicit attention when the application is multilingual, frequently approaches context limits, processes long documents, or has material token-based serving costs.

It matters less when prompts are tiny relative to the context window and token charges are negligible compared with other system costs. In that case, model quality, reliability, or operational simplicity may dominate the decision.

If two candidate models perform similarly on your task, token distributions can become a useful secondary criterion. Compare them using each model’s own tokenizer, then evaluate the complete system rather than comparing raw token counts in isolation.

A practical model-selection table can therefore look like:

candidate | task quality | p95 input tokens | latency | request cost
----------|--------------|------------------|---------|-------------
model A   | measured     | measured         | measured| measured
model B   | measured     | measured         | measured| measured

The values should come from your workload. There is no universal tokenizer ranking that replaces that measurement.

Conclusion

Tokenization efficiency is a property of a tokenizer applied to a particular body of text, not a fixed property of a language. That mental model avoids many misleading comparisons.

For cross-language analysis, use aligned content when you want a controlled comparison and real production samples when you want capacity estimates. Choose a denominator you can defend, inspect distributions rather than only averages, and investigate outliers. Most importantly, use the exact tokenizer and preprocessing pipeline associated with the model you intend to deploy.

The goal is not to minimize token count at any cost. It is to understand how tokenization changes context usage, serving economics, and operational headroom while preserving the model quality your application requires.

Artificial Intelligence LLM Tokenization Model Evaluation