HNSW at Scale: Why Your RAG System Will get Worse because the Vector Database Grows

a contemporary vector database—Neo4j, Milvus, Weaviate, Qdrant, Pinecone—there’s a very excessive likelihood that Hierarchical Navigable Small World (HNSW) is already powering your retrieval layer. It’s fairly seemingly you didn’t select it whereas constructing the database, nor did you tune it and even know it’s there. And but, HNSW is quietly deciding what your LLM sees as reality. It determines which doc chunks are fed into your RAG pipeline, which recollections your agent remembers, and in the end, whether or not the mannequin solutions accurately—or hallucinates confidently.

As your vector database grows, retrieval high quality degrades progressively:

• No exceptions are raised
• No errors are logged
• Latency usually seems completely wonderful

However the context high quality deteriorates, and your RAG system turns into much less dependable over time—though the embedding mannequin and distance metric stay unchanged.

On this article, I reveal—utilizing managed experiments and actual knowledge—how HNSW impacts retrieval high quality as database measurement grows, why this degradation is worse than flat search, and what you’ll be able to realistically do about it in manufacturing RAG programs.

Particularly, I’ll:

• Construct a sensible, reproducible use case to measure the impact of HNSW on RAG retrieval high quality utilizing Recall@ok.
• Present that, for fastened HNSW settings, recall degrades quicker than flat search because the corpus grows.
• Focus on sensible tuning methods for balancing recall and latency past merely rising ef_search of HNSW.

What’s HNSW?

HNSW is a graph-based algorithm for Approximate Nearest Neighbor (ANN) search. It organizes knowledge into a number of layers of linked neighbors and makes use of this graph construction to hurry up search.

Every vector is linked to a restricted variety of neighbors in every layer. Throughout a search, it performs a grasping search by these layers, and the variety of neighbors checked at every layer is fixed (managed by M and ef_search), which makes the search course of logarithmic with respect to the variety of vectors. In comparison with flat search, the place time complexity is O(N), HNSW search has a time complexity of O(log N), which implies the time required for a search grows very slowly (logarithmically) as in comparison with linear search. We are going to see this in the results of our use case.

Parameters of HNSW index

1. Construct time parameters: M and ef_construction. Might be set earlier than constructing the database solely.

M defines the utmost variety of connections (neighbors) that every vector (node) can have in every layer of the graph. A better M means extra connections, making the graph denser and doubtlessly rising recall however at the price of extra reminiscence and slower indexing.

Ef_construction controls the measurement of the candidate set used in the course of the development of the graph. Primarily, it governs how completely the graph is constructed throughout indexing. A better worth for ef_construction means the graph is constructed extra completely, with extra candidates being thought of earlier than making every connection, which ends up in a larger high quality graph and higher recall at the price of elevated reminiscence and slower indexing.

For a common objective RAG utility, typical values of M are inside a spread of 12 and 48 and ef_construction between 64 and 200.

2. Question time parameter: ef_search

This defines the variety of candidate nodes (or vectors) to discover in the course of the question course of (i.e., in the course of the seek for nearest neighbors). It controls how thorough the search course of is by figuring out what number of candidates are evaluated earlier than the search result’s returned. A better worth for ef_search means the search will discover extra candidates, main to higher recall however doubtlessly slower queries.

What’s Recall@ok?

Recall@ok is a key metric for measuring the accuracy of vector search and RAG programs. It measures the flexibility of the retriever to seek out the related chunks for a person question inside the prime ok outcomes. It’s vital as a result of If the retriever misses the chunks containing the knowledge required to reply the query (low recall), the LLM can not probably generate an correct reply within the response synthesis step, no matter how highly effective it’s.

[ text{Recall}@k = frac{text{relevant items retrieved in top } k}{text{total number of relevant items in the corpus}} ]

In follow, it is a tough metric to measure as a result of the denominator (floor reality paperwork) will not be simply recognized for a real-life manufacturing system. What we are going to do as a substitute, is design a use case the place the bottom reality (eg; vector index) is exclusive and recognized, and Recall@ok will measure the typical variety of occasions it’s retrieved in top-k outcomes, over numerous pattern queries.

As an example, Recall@5 will measure the typical variety of occasions the bottom reality index appeared in top-5 retrievals over 500 queries.

For a RAG, the appropriate vary of Recall@5 is 70-90% and Recall@10 is 80-95%, and we are going to see that our use case adheres to those ranges for the Flat index.

Use Case

To check HNSW, we’d like a vector database with sufficiently giant variety of vectors (> 100,000). There doesn’t appear to be such a big public dataset out there consisting of doc chunks and related question(ies) for which the actual chunk could be thought of as floor reality. And even when it had been there, pure language will be ambiguous, so it’s tough to confidently say which all chunks within the corpus may very well be thought of as related for a question (the denominator in Recall@ok method). Creating such a curated dataset would require discovering numerous paperwork, chunking and embedding them, then growing queries for the chunks. That may be a useful resource intensive course of.

As a substitute, lets re-imagine our RAG drawback as “given a brief caption (question), we want to retrieve probably the most related pictures from the dataset”.

For this method, I utilized the publicly out there LAION-Aesthetics dataset. To entry, you will want to be logged in to Hugging Face, and conform to the phrases talked about. Particulars concerning the dataset is out there on the LAOIN web site right here. It incorporates an enormous variety of rows containing URLs to pictures together with a textual content caption. They appear like the next:

I downloaded a subset of rows and generated 200,000 CLIP embeddings of the pictures to construct the vector database. The textual content captions of the pictures will be conveniently used as queries for RAG. And every caption has just one picture vector as the bottom reality so the denominator of Recall@ok is precisely recognized for all queries. Additionally, the CLIP embeddings of the picture and its caption are by no means a precise match, so there may be sufficient “fuzziness” in retrievals much like a purely doc RAG, the place a textual content question is used to retrieve related doc chunks utilizing a distance metric. This will likely be evident after we see the chart of Recall@ok within the subsequent sections.

Measuring Recall@ok for Flat vs HNSW

We undertake the next method:

1. Embeddings of 200k pictures are saved as .npy file.
2. From the laion dataset, 500 captions(queries) are randomly chosen and embedded utilizing CLIP. The chosen question indices additionally type the bottom reality as they correspond to the distinctive picture for the question.
3. The database is in-built increments of fifty,000 vectors, so 4 iterations of measurement 50k, 100k, 150k and 200k vectors. Each flat and HNSW indexes are constructed. HNSW is constructed utilizing M=16 and ef_construction=100.
4. Recall@ok is calculated for ok = 1, 5, 10, 15 and 20 based mostly upon if the bottom reality indices are included in top-k outcomes.
5. First, the Recall@ok values are calculated for every of the question vectors and averaged over the variety of samples (500).
6. Then, common Recall@ok values are calculated for HNSW ef_search values of 10, 20, 40, 80 and 160.
7. Lastly, 5 charts are drawn, one for every of the Recall@ok values. Every chart depicts the evolution of Recall@ok as database measurement grows for Flat index and completely different ef_search values of HNSW.

import pandas as pd
import numpy as np
import faiss
import torch
import open_clip
import os
import random
import matplotlib.pyplot as plt

def evaluate_subset(measurement, embeddings_all, df_all, query_vectors_all, eval_indices_all, ef_search_values):
    # Subset embeddings
    embeddings = embeddings_all[:size]
    dimension = embeddings.form[1]
    
    # Construct Indices in-memory for this subset measurement
    index_flat = faiss.IndexFlatL2(dimension)
    index_flat.add(embeddings)
    
    index_hnsw = faiss.IndexHNSWFlat(dimension, 16)
    index_hnsw.hnsw.efConstruction = 100
    index_hnsw.add(embeddings)

    num_samples = len(eval_indices_all)
    outcomes = []

    ks = [1, 5, 10, 15, 20]

    # Consider Flat
    flat_recalls = {ok: 0 for ok in ks}
    for i, qv in enumerate(query_vectors_all):
        _, I = index_flat.search(qv, max(ks))
        goal = eval_indices_all[i]
        for ok in ks:
            if goal in I[0][:k]:
                flat_recalls[k] += 1
    
    flat_res = {"Setting": "Flat"}
    for ok in ks:
        flat_res[f"R@{k}"] = flat_recalls[k]/num_samples
    outcomes.append(flat_res)

    # Consider HNSW with completely different efSearch
    for ef in ef_search_values:
        index_hnsw.hnsw.efSearch = ef
        hnsw_recalls = {ok: 0 for ok in ks}
        for i, qv in enumerate(query_vectors_all):
            _, I = index_hnsw.search(qv, max(ks))
            goal = eval_indices_all[i]
            for ok in ks:
                if goal in I[0][:k]:
                    hnsw_recalls[k] += 1
        
        hnsw_res = {"Setting": f"HNSW (ef={ef})", "ef": ef}
        for ok in ks:
            hnsw_res[f"R@{k}"] = hnsw_recalls[k]/num_samples
        outcomes.append(hnsw_res)
    
    return outcomes

def format_table(measurement, outcomes):
    ks = [1, 5, 10, 15, 20]
    traces = []
    traces.append(f"nDatabase Measurement: {measurement}")
    traces.append("="*80)
    header = f"{'Index/efSearch':<20}"
    for ok in ks:
        header += f" | {'R@'+str(ok):<8}"
    traces.append(header)
    traces.append("-" * 80)
    for row in outcomes:
        line = f"{row['Setting']:<20}"
        for ok in ks:
            line += f" | {row[f'R@{k}']:<8.2f}"
        traces.append(line)
    traces.append("="*80)
    return "n".be a part of(traces)

def essential(n):
    dataset_path = r"C:databaselaion_final.parquet"
    embeddings_path = r"C:databaseembeddings.npy"
    results_dir = r"C:outcomes"
    
    db_sizes = [50000, 100000, 150000, 200000]
    ef_search_values = [10, 20, 40, 80, 160]
    num_samples = n
    output_txt = os.path.be a part of(results_dir, f"eval_results_{num_samples}.txt")
    output_png = os.path.be a part of(results_dir, f"recall_vs_dbsize_{num_samples}.png")

    if not os.path.exists(dataset_path) or not os.path.exists(embeddings_path):
        print("Error: Dataset or embeddings not discovered.")
        return
    
    os.makedirs(results_dir, exist_ok=True)

    # Load All Information As soon as
    print("Loading base knowledge...")
    df_all = pd.read_parquet(dataset_path)
    embeddings_all = np.load(embeddings_path).astype('float32')

    # Load CLIP mannequin as soon as
    print("Loading CLIP mannequin (ViT-B-32)...")
    mannequin, _, preprocess = open_clip.create_model_and_transforms('ViT-B-32', pretrained='laion2b_s34b_b79k')
    tokenizer = open_clip.get_tokenizer('ViT-B-32')
    system = "cuda" if torch.cuda.is_available() else "cpu"
    mannequin.to(system)
    mannequin.eval()

    # Use samples legitimate for all subsets
    eval_indices = random.pattern(vary(min(db_sizes)), num_samples)
    print(f"Sampling {num_samples} queries for constant analysis...")

    # Generate question vectors
    query_vectors = []
    for idx in eval_indices:
        textual content = df_all.iloc[idx]['TEXT']
        text_tokens = tokenizer([text]).to(system)
        with torch.no_grad():
            text_features = mannequin.encode_text(text_tokens)
            text_features /= text_features.norm(dim=-1, keepdim=True)
            query_vectors.append(text_features.cpu().numpy().astype('float32'))

    all_output_text = []
    # Gather all outcomes for plotting
    # construction: { 'R@1': { 'Flat': [val1, val2...], 'ef=10': [val1, val2...] }, ... }
    ks = [1, 5, 10, 15, 20]
    plot_data = {f"R@{ok}": { "Flat": [] } for ok in ks}
    for ef in ef_search_values:
        for ok in ks:
            plot_data[f"R@{k}"][f"HNSW ef={ef}"] = []

    for measurement in db_sizes:
        print(f"Evaluating with database measurement: {measurement}...")
        outcomes = evaluate_subset(measurement, embeddings_all, df_all, query_vectors, eval_indices, ef_search_values)
        table_str = format_table(measurement, outcomes)
        
        # Print to display
        print(table_str)
        all_output_text.append(table_str)

        # Gather for plot
        for row in outcomes:
            label = row["Setting"]
            if label == "Flat":
                for ok in ks:
                    plot_data[f"R@{k}"]["Flat"].append(row[f"R@{k}"])
            else:
                ef = row["ef"]
                for ok in ks:
                    plot_data[f"R@{k}"][f"HNSW ef={ef}"].append(row[f"R@{k}"])

    # Save textual content outcomes
    with open(output_txt, "w", encoding="utf-8") as f:
        f.write("n".be a part of(all_output_text))
    print(f"nFinal outcomes saved to {output_txt}")

    # Create Particular person Plots for every Okay
    for ok in ks:
        plt.determine(figsize=(10, 6))
        k_key = f"R@{ok}"
        
        for label, values in plot_data[k_key].objects():
            linestyle = '--' if label == "Flat" else '-'
            marker = 'o' if label == "Flat" else 's'
            plt.plot(db_sizes, values, label=label, linestyle=linestyle, marker=marker)
        
        plt.title(f"Recall@{ok} vs Database Measurement")
        plt.xlabel("Database Measurement")
        plt.ylabel("Recall")
        plt.grid(True)
        plt.legend()
        
        output_png = os.path.be a part of(results_dir, f"recall_vs_dbsize_{ok}.png")
        plt.tight_layout()
        plt.savefig(output_png)
        plt.shut()
        print(f"Plot saved to {output_png}")

if __name__ == "__main__":
    essential(500)

And the outcomes are the next:

Observations

1. For the Flat index (dotted line), Recall@5 and Recall@10 are within the vary of 0.70 – 0.85, as will be anticipated of actual life RAG purposes.
2. Flat index offers the most effective Recall@ok throughout all database sizes and types a benchmark higher sure for HNSW.
3. At any given database measurement, Recall@ok will increase for a better ok. So for database measurement of 100k vectors, Recall@20 > Recall@15 > Recall@10 > Recall@5 > Recall@1. That is comprehensible as with a better ok, there may be extra likelihood that the bottom reality index is current within the retrieved set.
4. Each Flat and HNSW deteriorate constantly because the database measurement grows. It is because high-dimensional vector areas develop into more and more crowded because the variety of vectors grows.
5. Efficiency improves for HNSW for larger ef_search values.
6. Because the database measurement approaches 200k, HNSW seems to degrade quicker than Flat search.

Does HNSW degrade quicker than Flat Search?

To view the relative efficiency of Flat vs HNSW indexes as database measurement grows, a barely completely different method is adopted:

1. The database indexes development and question choice course of stays similar as earlier than.
2. As a substitute of contemplating the bottom reality, we calculate the overlap between the Flat index and every of the HNSW ef_search outcomes for a given retrieval rely(ok).
3. 5 charts are drawn for every of the ok values, denoting the evolution of overlap as database measurement grows. For an ideal match with Flat index, the HNSW line will present a rating of 1. Extra importantly, if the degradation of HNSW outcomes is greater than Flat index, the line may have a detrimental slope, else may have a horizontal or constructive slope.

import pandas as pd
import numpy as np
import faiss
import torch
import open_clip
import os
import random
import matplotlib.pyplot as plt
import time

def evaluate_subset_compare(measurement, embeddings_all, df_all, query_vectors_all, ef_search_values):
    # Subset embeddings
    embeddings = embeddings_all[:size]
    dimension = embeddings.form[1]
    
    # Construct Indices in-memory for this subset measurement
    index_flat = faiss.IndexFlatL2(dimension)
    index_flat.add(embeddings)
    
    index_hnsw = faiss.IndexHNSWFlat(dimension, 16)
    index_hnsw.hnsw.efConstruction = 100
    index_hnsw.add(embeddings)

    num_samples = len(query_vectors_all)
    outcomes = []

    ks = [1, 5, 10, 15, 20]
    max_k = max(ks)

    # 1. Consider Flat as soon as for this subset
    flat_times = []
    flat_results_all = []
    for qv in query_vectors_all:
        start_t = time.perf_counter()
        _, I_flat_all = index_flat.search(qv, max_k)
        flat_times.append(time.perf_counter() - start_t)
        flat_results_all.append(I_flat_all[0])
    
    avg_flat_time_ms = (sum(flat_times) / num_samples) * 1000

    # 2. Consider HNSW relative to Flat
    for ef in ef_search_values:
        index_hnsw.hnsw.efSearch = ef
        
        hnsw_times = []
        # Observe intersection counts for every ok
        overlap_counts = {ok: 0 for ok in ks}
        for i, qv in enumerate(query_vectors_all):
            # HNSW top-max_k
            start_t = time.perf_counter()
            _, I_hnsw_all = index_hnsw.search(qv, max_k)
            hnsw_times.append(time.perf_counter() - start_t)
            
            # Flat outcome was already pre-calculated
            I_flat_all = flat_results_all[i]
            
            for ok in ks:
                set_flat = set(I_flat_all[:k])
                set_hnsw = set(I_hnsw_all[0][:k])
                intersection = set_flat.intersection(set_hnsw)
                overlap_counts[k] += len(intersection) / ok
        
        avg_hnsw_time_ms = (sum(hnsw_times) / num_samples) * 1000
        
        hnsw_res = {
            "Setting": f"HNSW (ef={ef})", 
            "ef": ef,
            "FlatTime_ms": avg_flat_time_ms,
            "HNSWTime_ms": avg_hnsw_time_ms
        }
        for ok in ks:
            # Common over all queries
            hnsw_res[f"R@{k}"] = overlap_counts[k] / num_samples
        outcomes.append(hnsw_res)
    
    return outcomes

def format_all_tables(db_sizes, ef_search_values, all_results):
    ks = [1, 5, 10, 15, 20]
    traces = []
    
    # 1. Create one desk for every Recall@ok
    for ok in ks:
        k_label = f"R@{ok}"
        traces.append(f"nTable: {k_label} (HNSW Overlap with Flat)")
        traces.append("=" * (20 + 12 * len(db_sizes)))
        
        # Header
        header = f"{'ef_search':<18}"
        for measurement in db_sizes:
            header += f" | {measurement:<9}"
        traces.append(header)
        traces.append("-" * (20 + 12 * len(db_sizes)))
        
        # Rows (ef values)
        for ef in ef_search_values:
            row_str = f"{ef:<18}"
            for measurement in db_sizes:
                # Discover the outcome for this measurement and ef
                val = 0
                for r in all_results[size]:
                    if r.get('ef') == ef:
                        val = r.get(k_label, 0)
                        break
                row_str += f" | {val:<9.2f}"
            traces.append(row_str)
        traces.append("=" * (20 + 12 * len(db_sizes)))

    # 2. Create Search Time Desk
    traces.append("nTable: Common Search Time (ms)")
    traces.append("=" * (20 + 12 * len(db_sizes)))
    header = f"{'Index Setting':<18}"
    for measurement in db_sizes:
        header += f" | {measurement:<9}"
    traces.append(header)
    traces.append("-" * (20 + 12 * len(db_sizes)))
    
    # Flat Row
    row_flat = f"{'Flat Index':<18}"
    for measurement in db_sizes:
        # Flat time is similar for all ef in a measurement, so simply take any
        t = all_results[size][0]['FlatTime_ms']
        row_flat += f" | {t:<9.4f}"
    traces.append(row_flat)
    
    # HNSW Rows
    for ef in ef_search_values:
        row_str = f"HNSW (ef={ef:<3})"
        for measurement in db_sizes:
            t = 0
            for r in all_results[size]:
                if r.get('ef') == ef:
                    t = r.get('HNSWTime_ms', 0)
                    break
            row_str += f" | {t:<9.4f}"
        traces.append(row_str)
    traces.append("=" * (20 + 12 * len(db_sizes)))

    return "n".be a part of(traces)

def essential(n):
    dataset_path = r"C:databaselaion_final.parquet"
    embeddings_path = r"C:databaseembeddings.npy"
    results_dir = r"C:outcomes"
    
    db_sizes = [50000, 100000, 150000, 200000]
    ef_search_values = [10, 20, 40, 80, 160]
    num_samples = n
    output_txt = os.path.be a part of(results_dir, f"compare_results_{num_samples}.txt")
    output_png_prefix = "compare_vs_dbsize"

    if not os.path.exists(dataset_path) or not os.path.exists(embeddings_path):
        print("Error: Dataset or embeddings not discovered.")
        return
    
    os.makedirs(results_dir, exist_ok=True)

    # Load All Information As soon as
    print("Loading base knowledge...")
    df_all = pd.read_parquet(dataset_path)
    embeddings_all = np.load(embeddings_path).astype('float32')

    # Load CLIP mannequin as soon as
    print("Loading CLIP mannequin (ViT-B-32)...")
    mannequin, _, preprocess = open_clip.create_model_and_transforms('ViT-B-32', pretrained='laion2b_s34b_b79k')
    tokenizer = open_clip.get_tokenizer('ViT-B-32')
    system = "cuda" if torch.cuda.is_available() else "cpu"
    mannequin.to(system)
    mannequin.eval()

    # Use queries from the primary 50k rows
    eval_indices = random.pattern(vary(min(db_sizes)), num_samples)
    print(f"Sampling {num_samples} queries...")

    # Generate question vectors
    query_vectors = []
    for idx in eval_indices:
        textual content = df_all.iloc[idx]['TEXT']
        text_tokens = tokenizer([text]).to(system)
        with torch.no_grad():
            text_features = mannequin.encode_text(text_tokens)
            text_features /= text_features.norm(dim=-1, keepdim=True)
            query_vectors.append(text_features.cpu().numpy().astype('float32'))

    all_results_data = {}
    ks = [1, 5, 10, 15, 20]
    plot_data = {f"R@{ok}": {} for ok in ks}
    for ef in ef_search_values:
        for ok in ks:
            plot_data[f"R@{k}"][f"ef={ef}"] = []

    for measurement in db_sizes:
        print(f"Evaluating with database measurement: {measurement}...")
        outcomes = evaluate_subset_compare(measurement, embeddings_all, df_all, query_vectors, ef_search_values)
        all_results_data[size] = outcomes

        # Gather for plot
        for row in outcomes:
            ef = row["ef"]
            for ok in ks:
                plot_data[f"R@{k}"][f"ef={ef}"].append(row[f"R@{k}"])

    # Format pivoted tables
    final_output_text = format_all_tables(db_sizes, ef_search_values, all_results_data)
    print(final_output_text)

    # Save textual content outcomes
    with open(output_txt, "w", encoding="utf-8") as f:
        f.write(final_output_text)
    print(f"nFinal outcomes saved to {output_txt}")

    # Create Particular person Plots for every Okay
    for ok in ks:
        plt.determine(figsize=(10, 6))
        k_key = f"R@{ok}"
        
        for label, values in plot_data[k_key].objects():
            plt.plot(db_sizes, values, label=label, marker='s')
        
        plt.title(f"HNSW vs Flat Overlap Recall@{ok} vs Database Measurement")
        plt.xlabel("Database Measurement")
        plt.ylabel("Overlap Ratio")
        plt.grid(True)
        plt.legend()
        
        output_png = os.path.be a part of(results_dir, f"{output_png_prefix}_{ok}.png")
        plt.tight_layout()
        plt.savefig(output_png)
        plt.shut()
        print(f"Plot saved to {output_png}")

if __name__ == "__main__":
    essential(500)

And the outcomes are the next:

Observations

1. In all instances, the traces have a detrimental slope, indicating that HNSW degrades quicker than the Flat index as database grows.
2. Increased ef_search values degrade slower than decrease values, which fall fairly sharply.
3. Increased ef_search values have vital overlap (>90%) with the benchmark flat search as in comparison with the decrease values.

Recall-latency trade-off

We all know that HNSW is quicker than Flat search. To see it in motion, I’ve additionally measured the typical latency within the code of the earlier part. Listed here are the typical search occasions (in ms):

Observations

1. HNSW is orders of magnitude quicker than flat search, which is the first motive for it to be the search algorithm of alternative for nearly all vector databases.
2. Time taken by Flat search will increase nearly linearly with database measurement (O(N) complexity)
3. For a given ef_search worth (a row), HNSW time is sort of fixed. At this scale (200k vectors), HNSW latency stays practically fixed.
4. As ef_search will increase in a column, the HNSW time will increase very considerably. As an example, time taken for ef=160 is 3X that of ef=40

Tuning the RAG pipeline

The above evaluation reveals that whereas HNSW is unquestionably the choice to undertake in a manufacturing situation for latency causes, there’s a must periodically tune the ef_search to take care of the latency-recall stability because the database grows. Some greatest practices that must be adopted are as follows:

1. Given the problem of measuring Recall@ok in a manufacturing database, hold a take a look at case repository of floor reality doc chunks and queries, which will be run at common intervals to examine retrieval high quality. We may begin with probably the most frequent queries requested by the person, and chunks which are wanted for an excellent recall.
2. One other oblique option to verify recall high quality could be to make use of a strong LLM to evaluate the standard of the retrieved context. As a substitute of asking “Did we get the perfect paperwork for the person question?”, which is tough to say exactly for a big database, we are able to ask a barely weaker query “Does the retrieved context truly comprise the reply to the person’s query?” and let the choose LLM reply to that.
3. Gather person suggestions in manufacturing. Consumer ranking of a response together with any handbook correction can be utilized as a set off for efficiency tuning.
4. Whereas tuning ef_search, begin with a conservatively excessive worth, measure Recall@ok, then cut back till latency is appropriate.
5. Measure Recall on the top_k that the RAG makes use of, often between 3 and 10. Take into account stress-free top_k to fifteen or 20 and let the LLM resolve which chunks within the given context to make use of for the response throughout synthesis step. Assuming the context doesn’t develop into too giant to slot in the LLM’s context window, such an method would allow a excessive recall whereas having a reasonable ef_search worth, thereby maintaining latency low.

Hybrid RAG pipeline

HNSW tuning utilizing ef_search can not repair the difficulty of falling recall with rising database measurement past a degree. That’s as a result of vector search even utilizing a flat index, turns into noisy when too many vectors are packed shut collectively within the N dimensional house (N being the variety of dimensions output by the embedding mannequin). Because the charts within the above part present, recall drops by 10%+ as database grows from 50k to 200k. The dependable option to keep recall is to make use of metadata filtering (eg; utilizing a data graph), to establish potential doc ids and run retrieval just for these. I talk about this intimately in my article GraphRAG in Apply: Tips on how to Construct Value-Environment friendly, Excessive-Recall Retrieval Methods

Key Takeaways

• HNSW is the default retrieval algorithm in most vector databases, however it’s hardly ever tuned or monitored in manufacturing RAG programs.
• Retrieval high quality degrades silently because the vector database grows, even when latency stays steady.
• For a similar corpus measurement, Flat search constantly achieves larger Recall@ok than HNSW, serving as a helpful higher sure for analysis.
• HNSW recall degrades quicker than Flat seek for fastened ef_search values as database measurement will increase.
• Growing ef_search improves recall, however latency grows quickly, creating a pointy recall–latency trade-off.
• Merely tuning HNSW parameters is inadequate at scale—vector search itself turns into noisy in dense embedding areas.
• Hybrid RAG pipelines utilizing metadata filters (SQL, graphs, inverted indexes) are probably the most dependable option to keep recall at scale.

Conclusion

HNSW has earned its place because the spine of contemporary vector databases—not as a result of it’s completely correct, however as a result of it’s quick sufficient to make large-scale semantic search sensible.

Nevertheless, in RAG programs, velocity with out recall is a false optimization.

This text reveals that as vector databases develop, retrieval high quality deteriorates quietly—particularly underneath approximate search—whereas latency metrics stay deceptively steady. The result’s a system that seems wholesome from an infrastructure perspective, however progressively feeds weaker context to the LLM, rising hallucinations and lowering reply high quality.

The answer is to not abandon HNSW, nor to arbitrarily improve ef_search.

As a substitute, production-grade RAG programs should:

• Measure retrieval high quality explicitly and recurrently.
• Deal with Flat search as a recall baseline.
• Repeatedly rebalance recall and latency.
• And in the end, transfer towards hybrid retrieval architectures that slender the search house earlier than vector similarity is utilized.

In case your RAG system’s solutions are getting worse as your knowledge grows, the issue might not be your LLM, your prompts, or your embeddings—however the retrieval algorithm you by no means realized you had been counting on.

Join with me and share your feedback at www.linkedin.com/in/partha-sarkar-lets-talk-AI

_{Pictures used on this article are synthetically generated. LAOIN-Aesthetics dataset used underneath CC-BY 4.0 license. Figures and code created by me}