RAG Explained: How Retrieval-Augmented Generation Works
Retrieval-augmented generation grounds an LLM in external information before it answers. Here is how RAG works, why teams use it and where it can fail.
By Maya Chen, Women in AI Editorial Fellow ยท 29 August 2026
Retrieval-augmented generation, usually shortened to RAG, gives a language model relevant external information before asking it to produce an answer.
Instead of relying only on what the model learned during training, the application searches a knowledge source at query time and places the most relevant material into the model's context.
That makes RAG particularly useful for company documents, product information, research libraries and other knowledge that changes frequently or is not part of the model's training data.
The basic RAG flow
A classic RAG pipeline has two broad phases: preparing information and retrieving it when somebody asks a question.
Ingest the documents
The application collects content from sources such as files, websites, databases or knowledge systems.
Split content into chunks
Long documents are usually broken into smaller pieces so the retrieval system can return the section most relevant to a question rather than an entire file.
Microsoft's guidance highlights chunking as an important part of building searchable indexes for generative AI.
Create searchable representations
Many RAG systems create vector embeddings that represent the meaning of each chunk numerically. These can be stored in a vector-capable search index.
Retrieve relevant information
When a user asks a question, the system searches for relevant chunks. Search may use vectors, keywords or a hybrid of both.
Microsoft recommends hybrid approaches in many RAG scenarios because keyword and semantic similarity can retrieve different useful results.
Rerank or filter results
The first search results are not always the best context for the model. Systems may rerank results, apply metadata filters or remove low-confidence matches.
Give the context to the model
The retrieved text is included with the user's question and instructions. The model then generates an answer grounded in that material.
Why use RAG instead of relying on the model?
Foundation models have broad knowledge but do not automatically know an organisation's private documents or the latest version of a policy.
RAG provides that information without changing the model itself.
AWS describes this as one of the practical advantages of RAG: organisations can extend a model into a specific domain or internal knowledge base without retraining it.
RAG can improve freshness and traceability
If the underlying documents are updated and the index is refreshed, the system can retrieve the new information.
RAG can also make citations possible. The application knows which chunks were retrieved and can expose their source documents to users.
That does not guarantee the generated answer is correct, but it gives the system evidence to work from and gives users a way to inspect that evidence.
What can go wrong with RAG?
The right document is never retrieved
If chunking, indexing or search is weak, the model never sees the information it needs.
The system retrieves too much
Large volumes of loosely relevant text can make generation worse rather than better and increase latency and cost.
The documents are wrong
RAG grounds an answer in retrieved content. It does not make the source material accurate. Outdated policies can produce confidently outdated answers.
Permissions are ignored
Enterprise RAG needs access controls. Search results should respect the same permissions that apply to the original information.
The model still misinterprets the evidence
Retrieval improves grounding but does not eliminate model errors. Evaluation remains necessary.
Classic RAG and agentic retrieval
Traditional RAG commonly turns one user query into one retrieval request.
Newer agentic retrieval systems can break a complex question into multiple searches. Microsoft describes its agentic retrieval approach as using an LLM to create focused subqueries, execute them in parallel and combine the strongest grounding information.
This can improve coverage for complex questions at the cost of additional orchestration and latency.
When should you use RAG?
RAG is a strong option when an AI application needs information that is private, changing, domain-specific or too large to include permanently in a prompt.
It is less useful when the task does not require external knowledge or when deterministic retrieval alone can answer the question without generation.
The core idea is simple: retrieve the evidence first, then generate from it. The difficult part is building a retrieval system good enough that the model receives the right evidence consistently.
See our AI Engineering hub for more on retrieval, model evaluation and production AI architecture.