A retrieval-augmented generation (RAG) system can have a strong embedding model and still retrieve poor evidence. One common reason is document chunking: the text was divided into units that are awkward to search or incomplete when read on their own.

If chunks are too large, one embedding must represent several unrelated ideas and retrieval becomes less precise. If chunks are too small, the retrieved text may omit the definitions, qualifiers, or surrounding steps needed to answer correctly. The problem is therefore not to find one universally correct chunk size. It is to create retrieval units that are focused enough to match a query and complete enough to be useful after retrieval.

This article develops that mental model from a small example. You will learn how boundaries, chunk size, overlap, metadata, and evaluation interact, and how to choose a practical chunking strategy without treating a token count as a magic constant.

Think in retrieval units, not arbitrary slices

Suppose a product manual contains this text:

Rotating an API key

Create a replacement key before revoking the old key. Update the
service secret with the replacement, restart workers that cache the
secret, and verify successful requests. Revoke the old key only after
verification.

Troubleshooting

A 401 response after rotation usually means a worker still has the old
key or the replacement key lacks the required scope.

A fixed-width splitter might cut it like this:

chunk 1: Create a replacement key before revoking the old key. Update...
chunk 2: ...restart workers that cache the secret, and verify successful...
chunk 3: ...requests. Revoke the old key only after verification. Troubleshooting...

These pieces may fit a desired token limit, but the boundaries do not reflect the document’s meaning. A query such as When can I revoke the old key? could retrieve a fragment that contains the instruction but lacks the earlier explanation of the safe sequence.

A more useful split preserves the procedure as one retrieval unit and keeps troubleshooting as another:

chunk A: Rotating an API key + the complete rotation procedure
chunk B: Troubleshooting + the 401 explanation

The key question is not merely How many tokens fit? It is:

If this chunk is retrieved by itself, does it represent one useful idea
with enough local context to interpret it?

That question is a better starting point for chunk design.

Chunk size creates a precision-context trade-off

Embedding-based retrieval usually maps each chunk to one vector. That vector must summarize the chunk well enough for similarity search to connect it to relevant queries.

Consider a long section containing three topics:

API key creation
API key rotation
API key audit logs

If all three remain in one chunk, a single vector represents the combined text. A query specifically about audit logs competes with the other content for representation. The chunk can still be retrieved, but its embedding is asked to describe a broader mixture of ideas.

Splitting the section into coherent subsections gives retrieval a more focused unit for each topic. However, making chunks progressively smaller eventually creates the opposite problem. A chunk containing only

Revoke it after verification.

is highly focused but nearly useless without knowing what it and verification refer to.

This gives a practical rule:

  • larger chunks preserve more surrounding context but may mix several retrieval intents;
  • smaller chunks can improve topical focus but may remove information needed to interpret the passage.

The useful point lies between those failure modes and depends on the documents and queries in the application.

Prefer semantic boundaries when the source provides them

Many documents already contain structure that approximates meaningful retrieval units. Headings, paragraphs, list items, table rows, function definitions, and section boundaries carry information about which text belongs together.

A structure-aware splitter can use a hierarchy such as:

split by section
    if section is too large:
        split by subsection
            if subsection is too large:
                split by paragraph
                    if paragraph is still too large:
                        apply a smaller fallback split

This approach does not guarantee perfect chunks, but it avoids breaking coherent sections merely because a character counter reached a threshold.

The fallback limit still matters. Embedding models and generation models accept finite input lengths, and the application may impose tighter budgets for latency or cost. Structure-aware chunking therefore complements size limits rather than replacing them.

Keep headings with the text they describe

A heading often carries essential meaning that does not appear in the paragraph itself. Compare these two chunks:

Wait five minutes before retrying the operation.

and

After changing DNS records
Wait five minutes before retrying the operation.

The second chunk is more self-describing. Including a compact heading path can make both retrieval and downstream interpretation easier:

Networking > DNS > After changing records

For deeply nested documents, avoid copying a large amount of boilerplate into every chunk. Keep only metadata or heading context that helps identify the passage.

Use overlap to protect boundary information, not as a default cure

Chunk overlap repeats some text from the end of one chunk at the beginning of the next. It is useful when an important statement can cross a boundary.

Imagine two consecutive chunks without overlap:

chunk 1: The refresh token expires after 30 days of inactivity. A token...
chunk 2: ...used at least once during that period remains active.

The second chunk is difficult to interpret alone. A small overlap can preserve the relationship:

chunk 2: A token used at least once during that period remains active.

Overlap reduces the chance that a boundary separates tightly connected text, but it has costs. Repeated text creates more indexed tokens, more embeddings, and more near-duplicate candidates. A retriever may return several overlapping chunks that contain essentially the same evidence, consuming context without adding information.

For that reason, overlap should solve an observed boundary problem. If chunks already align with paragraphs or sections, little or no overlap may be necessary. If the source is unstructured prose and boundaries are approximate, modest overlap may be more valuable.

Do not interpret an overlap percentage as a quality guarantee. The useful amount depends on how often important dependencies cross the chosen boundaries.

Separate retrieval chunks from generation context

A useful design distinction is that the unit used for retrieval does not have to be the exact unit sent to the language model.

Suppose a manual is organized as:

section
  paragraph 1
  paragraph 2
  paragraph 3

You might embed individual paragraphs because they produce focused retrieval matches. After paragraph 2 is retrieved, the system can fetch its parent section or neighboring paragraphs before constructing the final prompt.

This pattern can provide:

small retrieval unit -> precise match
larger parent context -> enough information to answer

It is sometimes called parent-child or small-to-large retrieval, but the implementation does not require a particular framework. The essential requirement is that each indexed child retains an identifier linking it to the source material that can be expanded later.

Expansion has a trade-off. Adding neighbors or parents increases prompt tokens and can introduce irrelevant text. Expand only when the extra context is likely to resolve references, preserve a procedure, or supply necessary qualifiers.

Preserve metadata outside the embedding text when appropriate

Every chunk should retain enough metadata to trace it back to its source. Useful fields often include:

document_id
section_id
heading_path
source location or URL
version or updated_at
chunk position

Not every metadata field belongs in the text sent to the embedding model. A database identifier such as doc_8472 usually adds no semantic value. A heading such as Authentication > Token rotation, however, may help describe the chunk and can be useful in the embedded representation.

Metadata also enables operations that similarity search alone cannot express. For example, an application can filter retrieval to the current product version, reconstruct neighboring chunks, or show a source location with the generated answer.

Keep a distinction between semantic text, which helps represent meaning, and control metadata, which helps the retrieval system manage provenance and filtering.

Choose chunking with queries, not documents alone

A chunk can look perfectly readable during ingestion and still be poor for the queries users actually ask. Evaluate chunking from the retrieval direction.

Build a small set of representative queries with known relevant source passages. Include different query shapes:

specific fact:      How long does a refresh token remain valid?
procedure:          How do I rotate a key without downtime?
troubleshooting:    Why do workers return 401 after rotation?
comparison:         How do user keys differ from service keys?

For each chunking configuration, inspect whether the relevant evidence appears in the top retrieved candidates. Metrics such as recall at k can summarize whether at least one relevant chunk is being surfaced, but manual inspection remains valuable because a retrieved chunk can be technically relevant yet incomplete or bloated.

Evaluate the final answer separately from retrieval. If retrieval finds the right evidence but generation fails to use it, changing chunk size may not solve the real problem.

Run a controlled chunking experiment

Avoid changing the embedding model, retriever, prompt, and chunking strategy at the same time. A simple experiment is easier to interpret.

Start with two or three plausible configurations, for example:

A: structure-aware sections, no overlap
B: structure-aware sections with a small boundary overlap
C: smaller paragraph-level retrieval with parent expansion

Use the same document snapshot, embedding model, similarity method, retrieval depth, and evaluation queries for all configurations. Then compare:

  • retrieval recall for known relevant passages;
  • redundancy among the retrieved chunks;
  • average and tail chunk sizes;
  • prompt tokens after any context expansion;
  • end-to-end answer quality on the same questions;
  • indexing and retrieval cost if those differences matter operationally.

The purpose is not to crown a universal winner. It is to find which representation of your documents best matches the information needs of your users.

Watch for common chunking failures

Splitting only by a fixed token count

Token limits are useful constraints, but blind fixed-width splitting can cut through headings, lists, sentences, and procedures. Prefer meaningful boundaries first and use a size limit as a fallback when the document structure allows it.

Adding large overlap everywhere

Large overlap can hide poor boundaries during a small demo while creating many near-duplicate chunks at scale. This increases storage and can reduce the diversity of retrieved context. Measure whether overlap improves retrieval before paying that cost.

Creating chunks that depend on missing context

Pronouns, table cells, code fragments, and short list items may depend heavily on surrounding material. A chunk should either include the necessary context or carry enough structure for the system to reconstruct it after retrieval.

Embedding entire documents by default

Whole-document embeddings can be appropriate when documents are short and each document represents one coherent retrieval target. They are a poor default for long documents containing many independent topics because one vector must represent all of them.

Optimizing only for the embedding model’s maximum input

A model accepting a long input does not imply that every chunk should approach that limit. Maximum accepted length is a capacity constraint, not evidence that a long mixed-topic chunk will retrieve well.

Account for cost and latency

Chunking changes more than retrieval quality. Smaller chunks generally create more records and therefore more embedding work during indexing, more vector entries to store, and potentially more candidates to search or rerank. Large overlap increases those effects because repeated text is indexed multiple times.

Larger chunks create fewer index entries but can increase the amount of text passed downstream when a match is retrieved. If several large chunks are inserted into a prompt, generation input cost and latency can rise even when much of the text is irrelevant.

The exact performance impact depends on the vector index, embedding service, reranking stage, model, hardware, and caching strategy. Measure the deployed pipeline rather than assuming that a particular chunk size is inherently faster or cheaper.

When simpler chunking is enough

Not every RAG system needs a sophisticated parser. Simple paragraph or section splitting is often sufficient when documents are short, consistently structured, and queries target clearly separated topics.

More elaborate strategies are worth considering when documents contain long procedures, nested sections, tables, cross-references, or mixed topics; when retrieval frequently returns incomplete fragments; or when large chunks repeatedly consume context with irrelevant material.

Start with the simplest boundary-aware method that preserves meaning. Add overlap, parent expansion, or specialized parsing only when evaluation shows a concrete failure they can address.

Conclusion

Chunking determines what a RAG retriever is allowed to find. A good chunk is not merely a piece of text under a token limit; it is a retrieval unit that represents a focused idea while carrying enough context to remain useful.

Use document structure to choose boundaries, treat size as a trade-off between retrieval focus and local context, and add overlap only when boundary loss justifies the duplication. Preserve metadata so retrieved pieces can be traced and expanded, and evaluate chunking against representative queries rather than judging it only during ingestion.

The practical goal is not a universal chunk size. It is a document representation in which the evidence users need can be retrieved cleanly and supplied to the model with as little missing or irrelevant context as possible.

Artificial Intelligence RAG Retrieval Embeddings