Chunk size is a retrieval decision, not a formatting one

When building retrieval-augmented generation (RAG) systems, a common stumbling block appears early in the pipeline: how to segment source documents. Engineers often approach this as a text formatting task, slicing documents into arbitrary lengths, perhaps based on a character count or a simple paragraph break. The consequence of this oversight is frequently evident in the system's output: answers that are either incomplete, missing crucial context, or diluted with irrelevant information, forcing the large language model (LLM) to either speculate or produce vague responses.

The problem stems from a fundamental misunderstanding. Chunk size is not about making text aesthetically pleasing or fitting it into a generic window. It is a critical retrieval decision, directly affecting what information the embedding model indexes and what the retrieval system ultimately presents to the LLM. An ill-considered chunking strategy can cripple an otherwise well-designed RAG system, regardless of the sophistication of the embedding model or the LLM itself.

Truncated facts break answers

Consider the approach of using very small chunks, say 200 tokens. The failure mode here is direct truncation. A fact, an argument, or a complete idea often spans more than a few sentences. When a document is chopped into small, rigid segments, there is a high probability that critical information will be split across multiple chunks. For example, a detailed explanation of a technical concept might begin in one 200-token chunk and conclude in the next, or even several chunks later.

What breaks is the LLM's ability to form a coherent, complete answer. If the retrieved chunk contains only "The primary function of a transformer model is to process sequential data, typically text, by relying solely on self-attention mechanisms to draw global dependencies between input and output.", and the crucial follow-up about its encoder-decoder architecture or its advantages over RNNs is in a subsequent, unretrieved chunk, the LLM is left with an incomplete picture. It cannot infer the missing pieces accurately and will either provide a fragmented response, attempt to fill in the gaps with general knowledge (potentially leading to hallucination), or simply state that it cannot find the complete answer. The embeddings generated for such truncated chunks are also less representative of the full concept, making accurate retrieval less likely in the first place.

Bloated chunks dilute relevance

Conversely, some engineers opt for very large chunks, perhaps 1500 tokens or more, with the intention of providing maximum context. While this avoids truncation, it introduces a different failure mode: dilution. When a chunk is excessively large, the specific piece of information relevant to a user's query becomes a small island in a sea of unrelated text.

What breaks here is the precision of retrieval and the efficiency of the LLM's processing. An embedding model's ability to capture the semantic essence of a chunk diminishes as the chunk size increases and the content becomes more diverse. A query for "transformer model architecture" might match a single sentence within a 1500-token chunk that also discusses the history of neural networks, various NLP applications, and future research directions. The embedding for this large chunk will be a generalised representation of all its content, making it less specific to the "transformer model architecture" query. Consequently, the retrieval system might rank more relevant, but smaller, chunks lower, or fail to retrieve the most precise information.

Furthermore, even if the correct large chunk is retrieved, the LLM then has to process a significant amount of irrelevant data. This increases inference time, consumes more of the LLM's context window (potentially leading to a hit on token limits for complex queries or multiple retrieved chunks), and can introduce noise that distracts the LLM from the core answer, potentially leading to less concise or accurate responses.

800 tokens with 80 overlap carries a complete answer

Through empirical observation across various knowledge bases, a chunk size of approximately 800 tokens has proven to be a robust starting point. This size strikes a balance. A passage of 800 tokens is typically long enough to encapsulate a complete thought, a detailed explanation, or a multi-sentence argument, ensuring that the retrieved material is semantically self-contained. It allows the embedding model to generate a rich, focused representation of the chunk's core topic, facilitating more accurate retrieval.

When a user's query matches a specific sentence or paragraph within an 800-token chunk, that chunk is highly likely to contain the surrounding context necessary for the LLM to fully understand and elaborate on the matched information. It avoids the fragmentation of small chunks and the dilution of large ones, ensuring that the LLM receives a complete and relevant information packet.

Overlap prevents boundary issues

Complementing the 800-token chunk size, an overlap of 80 tokens between consecutive chunks addresses the problem of information falling precisely on chunk boundaries. Without overlap, a critical sentence at the very end of chunk A might be semantically linked to the beginning of chunk B. If only chunk A is retrieved, the LLM misses the continuation. If only chunk B is retrieved, it lacks the preceding context.

An 80-token overlap (roughly 10% of the chunk size) acts as a crucial buffer. It ensures that a sentence or a short paragraph that spans a boundary is present in its entirety within both adjacent chunks. This has two key benefits:

1. Robust Embeddings: The embedding model processes the sentence with its immediate preceding and succeeding context, regardless of where it falls. This leads to more stable and accurate embeddings for information near boundaries.
2. Contextual Integrity: If a query matches a sentence that is within the overlapping region, the LLM receives a chunk that contains that sentence plus sufficient context from both sides, ensuring the information is presented coherently and completely. This prevents the LLM from receiving a fragment that starts or ends abruptly, which would necessitate speculative completion.

The 800-token chunk with 80-token overlap strategy is not an arbitrary number but a mechanically derived approach to optimise the retrieval process. It ensures that the retrieved passage is about the thing that matched and still carries a complete answer, directly addressing the failure modes of both under- and over-chunking.

Still a starting point, not a universal constant

While 800 tokens with 80 overlap provides a robust default for many RAG applications, it is not a universally optimal constant. Different types of documents, such as highly structured tables, code snippets, or very short FAQ entries, might benefit from alternative chunking strategies. For instance, code might be best chunked by function or class definitions, and tables might require specific parsing to maintain row/column relationships, rather than a simple token-based split. Similarly, very short, atomic facts might be best left as smaller, distinct chunks to avoid unnecessary contextual overhead.

Furthermore, the optimal chunk size can also be influenced by the specific embedding model used. Some models might be more robust to larger contexts, while others perform better with more concise inputs. The 800/80 rule is a strong empirically-backed starting point that addresses common failure modes, but thorough validation against specific datasets and use cases remains essential. Future work could explore dynamic chunking algorithms that adapt based on content type or query complexity, moving beyond fixed-size approaches altogether.