This blueprint is the pattern we ship when a regulated team needs a RAG system that survives audit. It assumes you have a working PoC. It assumes you have an auditor who reads PDFs. It is opinionated where opinion is cheap and explicit where commitment is expensive.
Architecture summary
Four layers, each with a defensible owner and a measurable contract:
- Ingestion + chunking - owned by data engineering. Permissions, lineage, freshness, retention all enforced at this layer. No downstream layer can read a document that did not pass through here.
- Hybrid retrieval - BM25 + dense + reranker. Pure semantic search is a regression you do not need. Hybrid wins on tail queries and explainability.
- Generation with refusal - the model can say "I do not know," "I should not answer," or "I need confirmation." Refusal is not failure; it is the product feature that makes the system safe.
- Evaluation harness - runs in CI, gates merges, runs against a golden set per tenant or per domain. Drift triggers an alert before a customer notices.
The checklist
Ingestion + chunking
- [ ] Source permissions captured at ingestion and kept current; user access enforced again at retrieval.
- [ ] Source lineage stored with every chunk.
- [ ] Retention policy applied per source.
- [ ] Schema drift triggers a freshness alert.
- [ ] Chunk size justified against the use case, not the default tutorial.
Hybrid retrieval
- [ ] BM25 and dense both present, both indexed, both monitored.
- [ ] Reranker chosen against your golden set, not a benchmark.
- [ ] Top-k value justified against precision/recall tradeoffs measured locally.
- [ ] Permission filter applied at retrieval, not deferred to the prompt.
Generation with refusal
- [ ] System prompt explicitly lists refusal conditions.
- [ ] Refusal triggers logged with reason code.
- [ ] No "best effort" mode in production. The model commits or refuses.
- [ ] Citations are required and verified - not just emitted.
Evaluation harness
- [ ] Golden set per use case, 20-200 examples.
- [ ] Rubric written down, versioned, reviewed quarterly.
- [ ] CI runs the harness on every PR that touches retrieval or generation.
- [ ] Cost-per-eval-run is bounded; a regression in cost is treated as a regression in quality.
The non-obvious choices
- Hybrid retrieval over pure RAG. Pure semantic search loses on tail queries and is harder to debug. The cost of BM25 is rounding error; the explainability win is substantial.
- Refusal is a feature. Most teams build generation that always answers. That is a regression. Build a system that can decline; route the decline cases to a human review path.
- Eval harness owns CI. If your eval harness does not run on every PR, it does not exist. You will skip it on the day it matters.
What to do next
Run this checklist against your current RAG system. Anywhere you cannot put a check, that is the work. If five or more boxes are unchecked, do not ship to production - ship to a narrow internal pilot until the gaps close.