Nine Decisions That Make or Break Your RAG Pipeline
A working template for the architecture decisions most teams make by accident in week one — chunking, embedding, retrieval strategy — and never revisit. Each decision includes real tradeoffs, a heuristic, and a place to record your call.
How you split source documents determines what's even possible to retrieve later. Get this wrong and no amount of embedding-model quality fixes it downstream.
- Fixed-size (e.g. 512 tokens, with overlap): simple and predictable, but cuts across semantic boundaries — a sentence can end right before the section explaining it.
- Recursive / structure-aware (split on headers, then size limit): preserves logical units; more implementation work, meaningfully better retrieval precision.
- Semantic (group by embedding-similarity shifts): highest fidelity to meaning, highest compute cost to build and maintain the index.
- Document-native (one chunk per clause/section/row): best for already-structured documents — contracts, tickets, tabular data; doesn't generalize to prose.
The embedding model defines the geometry your entire retrieval system operates in. Swapping it later means re-indexing everything — it isn't a config change.
- General-purpose commercial (OpenAI, Voyage): strong baseline, zero lock-in to run, ongoing per-query cost at scale.
- Open-weight, self-hosted (BGE, E5, GTE): no per-query cost at scale, but you own serving infrastructure and latency.
- Domain-tuned / fine-tuned: best retrieval for specialized vocabulary (legal, medical, internal jargon) — but needs labeled data and retraining discipline.
- Multilingual: required for multi-language corpora — an easy detail to miss until it's a launch blocker.
Built for the team about to write retrieval code — or the one that already did.
You're about to build a RAG pipeline and don't want the chunking strategy decided by whatever the first tutorial you read happened to use. These decisions compound — get them wrong early and no model swap fixes it later.
You already shipped RAG and answers are occasionally wrong in a way nobody can quite explain. Most of the time the cause is one of these nine decisions, made by default, never revisited.
You're inheriting someone else's retrieval pipeline and need to understand what was actually decided — not guess from the code, but see the tradeoffs and reasoning laid out in one place.
Most RAG systems don't fail because of the model. They fail because of nine decisions made in the first week — and never written down, so nobody can tell you why they were made that way.
Nine decisions. Real tradeoffs. A place to record your call.
Plus decision nine — evaluation — and a closing note on how the nine decisions interact.
Michael Legemah is a Principal AI Engineer who has spent over a decade building production systems for AWS, the U.S. Army, and U.S. Space Force — the last several years focused specifically on agentic AI, RAG pipelines, and the evaluation infrastructure that keeps them honest. This template comes from the same retrieval-architecture decisions he's made and revisited directly inside real production systems, not from a summary of someone else's blog posts.
Nine decisions. One sitting. A documented reason for every one of them.
Get the Template — $59