How I cut a RAG system's response time from 4.2s to 1.2s

Quick answers

What chunk size should I use for RAG?

Start at 512 tokens with a 50-token overlap. In my testing, 1024-token chunks dropped context accuracy from 89% to 71% because more irrelevant text entered the context window.

Is vector search enough?

Not for technical content. Pure vector search missed exact terms like "PID controller". Hybrid search (vector + keyword) fixed it.

Which vector database should I use?

If you already run Postgres, start with pgvector. Choose based on your existing infrastructure, not on benchmarks you'll never hit.

The results

Where the time was actually going

The system was a Robotics AI Tutor: students ask questions, and the tutor answers from technical manuals. The stack was React on the front, FastAPI in the middle and a vector database behind it. When answers started taking more than four seconds, everyone blamed the LLM.

So I timed every stage instead of guessing. Embedding the query, searching, re-ranking, building the prompt and generating: each one got a timer. Generation was not the biggest cost. Retrieval was: large chunks meant large prompts, and weak matches meant the model had to read through a lot of noise before it could answer.

import time
from contextlib import contextmanager

timings: dict[str, float] = {}

@contextmanager
def stage(name: str):
    start = time.perf_counter()
    yield
    timings[name] = round((time.perf_counter() - start) * 1000)

with stage("embed"):
    q_vec = embed(query)
with stage("search"):
    hits = search(q_vec, query, k=8)
with stage("generate"):
    answer = generate(query, hits)

print(timings)  # {'embed': 180, 'search': 1400, 'generate': 2300} ← before

Change 1: smaller chunks, with overlap

1024-token chunks felt safe ("more context is better") but they weren't. A single chunk often covered two unrelated sections of a manual, so the model blended them into one confident, wrong answer. Moving to 512 tokens with a 50-token overlap kept each chunk about one idea, and the overlap stopped sentences from being cut in half at the boundary.

def chunk(tokens: list[int], size: int = 512, overlap: int = 50):
    step = size - overlap
    for i in range(0, max(len(tokens) - overlap, 1), step):
        yield tokens[i : i + size]

Change 2: hybrid search

Embeddings are great at meaning and bad at exact strings. A student typing "PID controller" or a part number needs the exact match, not something semantically nearby. I added a keyword score (BM25) next to the vector score and merged them with reciprocal rank fusion. It's simple, it has no extra model, and it fixed most of the misses.

def rrf(vector_hits, keyword_hits, k: int = 60):
    scores: dict[str, float] = {}
    for hits in (vector_hits, keyword_hits):
        for rank, doc_id in enumerate(hits):
            scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank + 1)
    return sorted(scores, key=scores.get, reverse=True)

Change 3: cache what repeats

Students ask the same twenty questions in slightly different words. Caching query embeddings (keyed on normalised text) and hot retrieval results meant repeat questions skipped the most expensive part of the pipeline entirely.

What I'd do first on any slow RAG system

1. Time every stage before changing anything.
2. Check chunk size, then look at what actually lands in the prompt.
3. Add keyword search if your domain has exact terms, codes or names.
4. Build a small evaluation set (30–50 real questions) so you know whether a change helped.
5. Cache repeat work last, once the pipeline is correct.

Most slow RAG systems don't have a model problem. They have a retrieval problem.

Read nextWhat a real citation is in a RAG system (and why most chatbots fake it)