Every founder who asks me to "add AI search" to their product already has a RAG pipeline in their head: embed the docs, stuff them in a vector DB, retrieve top-k, hand it to the LLM. Ship it. That mental model is why so many RAG features get built in a weekend and then quietly stop working three weeks later — not with an error, but with wrong answers that look confident enough that nobody questions them until a customer does.
I've built retrieval into a research-and-writing tool (Researcher), where the entire product is only as good as what the pipeline actually surfaces to the model. When retrieval is bad, the LLM doesn't fail loudly — it hallucinates fluently on top of the wrong context, and everyone in the room comes away thinking it's a "prompting problem." It almost never is. Here's what actually breaks, in the order I check it.
1. Chunking is where you lose the answer before retrieval even runs
Teams spend their engineering time on the embedding model and the vector DB, then chunk documents with whatever the framework's default splitter does — usually a fixed 512 tokens with some overlap, applied uniformly to everything from a legal contract to an API reference.
That's backwards. Chunk size should match how the source document actually carries meaning:
- Dense reference material (legal, compliance, structured docs) wants tighter paragraph-level chunks, around 300-500 tokens, so a single retrieved chunk maps to a single self-contained clause or fact.
- Narrative or technical documentation holds meaning across a whole section, so cramming it into 500-token pieces cuts sentences off mid-argument. 1,000-2,000 token, section-aware chunks retrieve better.
The fix costs almost nothing — it's a splitting strategy, not new infrastructure — and it's the single highest-leverage change I make on a struggling pipeline before touching anything else.
2. No reranker means you're trusting cosine similarity to do too much
Vector similarity search is a coarse filter, not a judge of relevance. It's very good at "these are topically related" and much worse at "this is the specific passage that answers this specific question." Most teams stop at top-k vector retrieval and feed that straight to the LLM.
Adding a cross-encoder reranker as a second pass over your top 20-50 candidates — scoring each chunk against the actual query instead of a cached embedding — routinely improves answer accuracy by 15-25% in my experience, for a component that's a single extra network call. It's cheap insurance that most production RAG systems still skip, mostly because it wasn't in the tutorial they copied the architecture from.
3. Partial index updates are the failure mode nobody plans for
This is the one that actually causes outages, and it's almost invisible until it happens. You kick off a reindex of 10,000 documents. The job dies at document 6,000 — a timeout, a rate limit, a bad document that throws. Now your index has some documents at version N and some at version N+1, with no marker distinguishing them. Retrieval doesn't error. It just silently serves stale or duplicate context depending on which version of a document it happens to pull, and the bug report that comes in three days later looks like a random hallucination.
The fix is architectural, not clever: treat reindexing as an event-driven pipeline with idempotent, resumable steps rather than a single long-running batch job. I wrote about the pattern that solves this generally — decoupled producers and consumers with durable delivery — in my TypeScript pub/sub tutorial. Apply the same event-bus thinking to document ingestion: each document's embedding-and-index step becomes its own message, retries are per-document instead of per-batch, and a crash at document 6,000 doesn't leave the other 4,000 in limbo. You also get a clean way to version documents atomically — swap the pointer to the new version only once every chunk for that document is confirmed indexed.
4. You don't actually know which layer is failing
When a RAG answer is wrong, the instinct is to blame the LLM. Sometimes that's right. More often it's a retrieval precision problem (the right chunk exists but didn't get retrieved), a faithfulness problem (the LLM ignored the retrieved context and used its own knowledge instead), or a chunking problem (the right information existed but got split across two chunks and neither one had enough to answer the question).
You cannot tell these apart by reading transcripts. You need retrieval-stage metrics — precision/recall on a held-out set of query-answer pairs — separate from generation-stage faithfulness metrics. Without that separation, every fix is a guess, and teams end up swapping embedding models or upgrading the LLM when the actual bug was a chunking boundary.
What I'd actually tell you to do first
If you're a founder about to bolt RAG onto your product: don't start with the vector database decision, start with 20 real questions your users will ask and what document, exactly, contains the answer. Chunk around that answer, not around a token count. Add the reranker before you need it, not after a customer complains. And build your ingestion as an event pipeline from day one — retrofitting idempotency into a batch job after it's already dropped documents in production is a much worse conversation to have.
If you're mid-build and something's already misbehaving, I can usually tell you which of these four layers is broken from a transcript and a description of your ingestion pipeline. That's a conversation worth having before you spend more budget on a bigger model. Get in touch if you want a second pair of eyes on it.