Person: “What does the inexperienced highlighting imply on this doc?”
RAG system: “Inexperienced highlighted textual content is interpreted as configuration settings.”
the sorts of solutions we anticipate at the moment from Retrieval-Augmented Technology (RAG) techniques.
Over the previous few years, RAG has change into one of many central architectural constructing blocks for knowledge-based language fashions: As a substitute of relying completely on the information saved within the mannequin, RAG techniques mix language fashions with exterior doc sources.
The time period was launched by Lewis et al. and describes an method that’s extensively used to scale back hallucinations, enhance the traceability of solutions, and allow language fashions to work with proprietary knowledge.
I needed to know why a system selects one particular reply as an alternative of a really related different. This resolution is usually made already on the retrieval stage, lengthy earlier than an LLM comes into play.
For that reason, I performed three experiments on this article to research how totally different chunk sizes (80, 220, 500) affect retrieval conduct.
Desk of Contents
1 – Why Chunk Measurement Is Extra Than Only a Parameter
2 – How Does Chunk Measurement Affect the Stability of Retrieval Ends in Small RAG Techniques?
3 – Minimal RAG System With out Output Technology
4 – Three Experiments: Chunk Measurement as a Variable
5 – Ultimate Ideas
1 – Why Chunk Measurement Is Extra Than Only a Parameter
In a typical RAG pipeline, paperwork are first cut up into smaller textual content segments, embedded into vectors, and saved in an index. When a question is issued, semantically related textual content segments are retrieved after which processed into a solution. This closing step is usually carried out together with a language mannequin.
Typical parts of a RAG system embrace:
- Doc preprocessing
- Chunking
- Embedding
- Vector index
- Retrieval logic
- Elective: Technology of the output
On this article, I deal with the retrieval step. This step will depend on a number of parameters:
- Alternative of the embedding mannequin:
The embedding mannequin determines how textual content is transformed into numerical vectors. Totally different fashions seize that means at totally different ranges of granularity and are educated on totally different targets. For instance, light-weight sentence-transformer fashions are sometimes enough for semantic search, whereas bigger fashions could seize extra nuance however include greater computational price. - Distance or similarity metric:
The gap or similarity metric defines how the closeness between two vectors is measured. Widespread selections embrace cosine similarity, dot product or Euclidean distance. For normalized embeddings, cosine similarity is usually used - Variety of retrieved outcomes (Prime-k):
The variety of retrieved outcomes specifies what number of textual content segments are returned by the retrieval step. A small Prime-k can miss related context, whereas a big Prime-k will increase recall however could introduce noise. - Overlap between textual content segments:
Overlap defines how a lot textual content is shared between consecutive chunks. It’s sometimes used to keep away from dropping necessary info at chunk boundaries. A small overlap reduces redundancy however dangers slicing explanations in half, whereas a bigger overlap will increase robustness at the price of storing and processing extra related chunks. - Chunk dimension:
Describes the scale of the textual content models which can be extracted from a doc and saved as particular person vectors. Relying on the implementation, chunk dimension might be outlined primarily based on characters, phrases, or tokens. The dimensions determines how a lot context a single vector represents.
Small chunks comprise little or no context and are extremely particular. Giant chunks embrace extra surrounding info, however at a a lot coarser stage. In consequence, chunk dimension determines which elements of the that means are literally in contrast when a question is matched in opposition to a bit.
Chunk dimension implicitly displays assumptions about how a lot context is required to seize that means, how strongly info could also be fragmented, and the way clearly semantic similarity might be measured.
With this text, I needed to discover precisely this via a small RAG system experiment and requested myself:
How do totally different chunk sizes have an effect on retrieval conduct?
The main target is just not on a system supposed for manufacturing use. As a substitute, I needed to learn the way totally different chunk sizes have an effect on the retrieval outcomes.
2 – How Does Chunk Measurement Affect the Retrieval Ends in Small RAG Techniques?
I due to this fact requested myself the next questions:
- How does chunk dimension change retrieval leads to a small, managed RAG system?
- Which textual content segments make it to the highest of the rating when the queries are equivalent however the chunk sizes differ?
To analyze this, I intentionally outlined a easy setup by which all situations (besides chunk dimension) stay the identical:
- Three Markdown paperwork because the information base
- Three equivalent, fastened questions
- The identical embedding mannequin for vectorizing the texts
The textual content used within the three Markdown recordsdata relies on a documentation from an actual software known as OneLatex. To maintain the experiment targeted on retrieval conduct, the content material was barely simplified and diminished to the core explanations related for the questions.
The three questions I used the place:
"Q1: What's the important benefit of separating content material creation from formatting in OneLatex?"
"Q2: How does OneLatex interpret textual content highlighted in inexperienced in OneNote?"
"Q3: How does OneLatex interpret textual content highlighted in yellow in OneNote?"As well as, I intentionally omitted an LLM for output technology.
The rationale for that is easy: I didn’t need an LLM to show incomplete or poorly-matched textual content segments right into a coherent reply. This makes it a lot clearer what really occurs within the retrieval step, how the parameters of the retrieval work together, and what function the sentence transformer performs.
3 – Minimal RAG System With out Output Technology
For the experiments, I due to this fact used a small RAG system with the next parts: Markdown paperwork because the information base, a easy chunking logic with overlap, a sentence transformer mannequin to generate embeddings, and a rating of textual content segments utilizing cosine similarity.
Because the embedding mannequin, I used all-MiniLM-L6-v2 from the Sentence-Transformers library. This mannequin is light-weight and due to this fact well-suited for operating regionally on a private laptop computer (I ran it regionally on my Lenovo laptop computer with 64 GB of RAM). The similarity between a question and a textual content section is calculated utilizing cosine similarity. As a result of the vectors are normalized, the dot product might be in contrast instantly.
I intentionally saved the system small and due to this fact didn’t embrace any chat historical past, reminiscence or agent logic, or LLM-based reply technology.
As an “reply,” the system merely returns the highest-ranked textual content section. This makes it a lot clearer which content material is definitely recognized as related by the retrieval step.
The total code for the mini RAG system might be present in my GitHub repository:
→ 🤓 Discover the total code within the GitHub Repo 🤓 ←
4 – Three Experiments: Chunk Measurement as a Variable
For the analysis, I ran the three instructions beneath by way of the command line:
#Experiment 1 - Baseline
python important.py --chunk-size 220 --overlap 40 --top-k 3
#Experiment 2 - Small Chunk-Measurement
python important.py --chunk-size 80 --overlap 10 --top-k 3
#Experiment 3 - Huge Chunk-Measurement
python important.py --chunk-size 500 --overlap 50 --top-k 3The setup from Part 3 stays precisely the identical: The identical three paperwork, the identical three questions, and the identical embedding mannequin.
Chunk dimension defines the variety of characters per textual content section. As well as, I used an overlap in every experiment to scale back info loss at chunk boundaries. For every experiment, I computed the semantic similarity scores between the question and all chunks and ranked the highest-scoring segments.
Small Chunks (80 Characters) – Lack of Context
With very small chunks (chunk-size 80), a robust fragmentation of the content material turns into obvious: Particular person textual content segments typically comprise solely sentence fragments or remoted statements with out enough context. Explanations are cut up throughout a number of chunks, in order that particular person segments comprise solely elements of the unique content material.
Formally, the retrieval nonetheless works appropriately: Semantically related fragments are discovered and ranked extremely.
Nevertheless, after we have a look at the precise content material, we see that the outcomes are hardly usable:
The returned chunks are thematically associated, however they don’t present a self-contained reply. The system roughly acknowledges what the subject is about, however it breaks the content material down so strongly that the person outcomes don’t say a lot on their very own.
Medium Chunks (220 characters) – Obvious Stability
With the medium chunks (chunk-size 220), the outcomes already improved clearly. Many of the returned textual content segments contained full explanations and had been content-wise believable. At first look, the retrieval appeared steady and dependable: It normally returned precisely the data one would anticipate.
Nevertheless, a concrete downside grew to become obvious when distinguishing between inexperienced and yellow highlighted textual content. No matter whether or not I requested concerning the that means of the inexperienced or the yellow highlighting, the system returned the chunk concerning the yellow highlighting as the highest end in each instances. The proper chunk was current, however it was not chosen as Prime-1.
The rationale lies within the very related similarity scores of the 2 high outcomes:
- Rating for Prime-1: 0.873
- Rating for Prime-2: 0.774
The system can hardly distinguish between the 2 candidates semantically and in the end selects the chunk with the marginally greater rating.
The issue? It doesn’t match the query content-wise and is just unsuitable.
For us as people, that is very simple to acknowledge. For a sentence transformer like all-MiniLM-L6-v2, it appears to be a problem.
What issues right here is that this: If we solely have a look at the Prime-1 end result, this error stays invisible. Solely by evaluating the scores can we see that the system is unsure on this scenario. Since it’s pressured to make a transparent resolution in our setup, it returns the Prime-1 chunk as the reply.
Giant Chunks (500 characters) – Sturdy Contexts
With the bigger chunks (chunk-size 500), the textual content segments comprise rather more coherent context. There may be additionally hardly any fragmentation anymore: Explanations are now not cut up throughout a number of chunks.
And certainly, the error in distinguishing between inexperienced and yellow now not happens. The questions on inexperienced and yellow highlighting at the moment are appropriately distinguished, and the respective matching chunk is clearly ranked as the highest end result. We will additionally see that the similarity scores of the related chunks at the moment are extra clearly separated.
This makes the rating extra steady and simpler to know. The draw back of this setting, nevertheless, is the coarser granularity: Particular person chunks comprise extra info and are much less finely tailor-made to particular features.
In our setup with three Markdown recordsdata, the place the content material is already thematically nicely separated, this draw back hardly performs a task. With otherwise structured documentation, akin to lengthy steady texts with a number of matters per part, an excessively giant chunk dimension may result in irrelevant info being retrieved along with related content material.
On my Substack Information Science Espresso, I share sensible guides and bite-sized updates from the world of Information Science, Python, AI, Machine Studying, and Tech — made for curious minds like yours.
Take a look and subscribe on Medium or on Substack if you wish to keep within the loop.
5 – Ultimate Ideas
The outcomes of the three quite simple experiments might be traced again to how retrieval works. Every chunk is represented as a vector, and its proximity to the question is calculated utilizing cosine similarity. The ensuing rating signifies how related the query and the textual content section are within the semantic house.
What’s necessary right here is that the rating is just not a measure of correctness. It’s a measure of relative comparability throughout the obtainable chunks for a given query in a single run.
When a number of segments are semantically very related, even minimal variations within the scores can decide which chunk is returned as Prime-1. One instance of this was the wrong distinction between inexperienced and yellow within the medium chunk dimension.
One potential extension could be to permit the system to explicitly sign uncertainty. If the scores of the Prime-1 and Prime-2 chunks are very shut, the system may return an “I don’t know” or “I’m unsure” response as an alternative of forcing a call.
Based mostly on this small RAG system experiment, it isn’t actually potential to derive a “finest chunk dimension” conclusion.
However what we will observe as an alternative is the next:
- Small chunks result in excessive variance: Retrieval reacts very exactly to particular person phrases however rapidly loses the general context.
- Medium-sized chunks: Seem steady at first look, however can create harmful ambiguities when a number of candidates are scored nearly equally.
- Giant chunks: Present extra sturdy context and clearer rankings, however they’re coarser and fewer exactly tailor-made.
Chunk dimension due to this fact, determines how sharply retrieval can distinguish between related items of content material.
On this small setup, this didn’t play a serious function. Nevertheless, after we take into consideration bigger RAG techniques in manufacturing environments, this type of retrieval instability may change into an actual downside: Because the variety of paperwork grows, the variety of semantically related chunks will increase as nicely. Which means many conditions with very small rating variations are more likely to happen. I may think about that such results are sometimes masked by downstream language fashions, when an LLM turns incomplete or solely partially matching textual content segments into believable solutions.
