Building RAG That Actually Works in Production

Retrieval quality, permissions and evaluation — the parts of RAG that decide whether it survives contact with real users.

Retrieval-augmented generation demos beautifully and breaks quietly. The demo works because the questions are friendly and the documents are clean. Production is neither. What decides whether a RAG system survives real users is rarely the model — it is everything around it.

Retrieval is the whole game

If the right passage never makes it into the prompt, no model can answer correctly — it will just make something up, confidently. So retrieval quality matters far more than model choice. That means paying attention to the unglamorous parts: how you split documents into chunks, whether you use hybrid search (keyword and semantic together) rather than embeddings alone, and whether a reranking step trims the results down to what is actually relevant before it reaches the model.

Permissions are not optional

The fastest way to turn a helpful assistant into a data breach is to let it retrieve documents the user is not allowed to see. RAG has to respect the same access controls as the rest of your system, and it has to enforce them at query time — filtering the search by the user’s permissions before the model reads anything, not after. If your retrieval layer cannot enforce who-can-see-what, it is not ready for anything beyond a demo.

If you cannot measure it, you cannot improve it

Most failed RAG projects never had a way to tell whether a change actually helped. Build an evaluation set early: a few dozen real questions, the answers they should produce, and the documents those answers should come from. Then you can measure whether the retrieved passages were relevant, whether the answer was grounded in them, and whether it addressed the question. Without this you are tuning blind and shipping on vibes.

The boring parts that decide success

• Freshness — a plan for keeping the index in sync as documents change, not a one-time load.
• Citations — every answer should link back to its sources, so users can verify and you can debug.
• Graceful failure — when retrieval finds nothing relevant, the system should say so rather than invent an answer.
• Latency — hybrid search plus reranking plus generation adds up; budget for it from the start.

None of this is exotic. It is simply the difference between a demo and something you would let your team rely on every day.