A technique that combines information retrieval with text generation, allowing LLMs to use external knowledge without retraining the model.
Detailed explanation
RAG (Retrieval-Augmented Generation) is an architecture where, before generating a response, the system retrieves relevant documents from a knowledge base and includes them in the prompt context. This allows the model to use up-to-date and domain-specific information without retraining. The challenge of RAG in production is cost: each retrieved document adds tokens to the context. Kapa.ai demonstrated a technique to cut 68% of RAG context while maintaining 96% recall, showing that RAG optimization is as much a quality issue as a FinOps one. For product managers, RAG is often the architecture of choice for AI products that need to answer based on proprietary or up-to-date data.
How to use it in product decisions
Before choosing RAG, confirm that the answer depends on private, current, or citable knowledge. Measure retrieval before generation: if the right document does not reach the context, a better model only produces a more convincing answer from the wrong evidence. Version sources, record the passages used, and define behavior when evidence is insufficient. Track recall, precision, context cost, latency, and unsupported-answer rate. RAG should improve traceability, not hide uncertainty behind fluent text.