What is retrieval-augmented generation (RAG)?

RAG was introduced in May 2020 by Patrick Lewis and ten co-authors from Facebook AI Research, University College London, and New York University, in a paper presented at NeurIPS 2020 (arXiv:2005.11401). The architecture pairs a parametric memory — a sequence-to-sequence transformer such as BART — with a non-parametric memory — a dense vector index of Wikipedia accessed via Maximum Inner Product Search (MIPS). At query time, a Dense Passage Retrieval (DPR) encoder (arXiv:2004.04906, Karpukhin et al., 2020) projects the user's question into a vector, fetches the top-k passages, and the generator conditions its output on both the question and the retrieved passages.

The motivation is two failure modes of pure parametric models. First, knowledge cutoff: a model trained in January cannot know what happened in February. Second, hallucination: when a model lacks grounding for a claim, it confabulates plausible-sounding falsehoods. The 2020 RAG paper showed state-of-the-art performance on open-domain QA benchmarks Natural Questions, TriviaQA, and WebQuestions, beating both extractive and closed-book baselines.

Modern production RAG diverges from the original paper in three ways. First, the generator is typically a frontier instruction-tuned LLM (GPT-4, Claude, Gemini, Llama 3) rather than BART. Second, the retriever is often a hybrid of dense vectors (BGE, E5, OpenAI text-embedding-3) and sparse lexical matching (BM25, as defined in Robertson & Zaragoza 2009). Third, the index is rarely Wikipedia — it is the customer's own corpus, indexed in a vector database such as Pinecone, Weaviate, Qdrant, Chroma, or pgvector (a PostgreSQL extension first released June 2021).

The canonical pipeline has six stages: chunking (splitting documents into 200–1000 token passages), embedding (encoding each chunk as a vector), indexing (storing vectors in an ANN structure such as HNSW, Malkov & Yashunin 2018, arXiv:1603.09320), retrieval (top-k nearest neighbors at query time), reranking (often using a cross-encoder like Cohere Rerank or BGE-Reranker for higher precision on the top results), and generation (the LLM conditions on the question plus retrieved context).

RAG has known failure modes documented in the literature. Stanford's "Lost in the Middle" paper (Liu et al., 2023, arXiv:2307.03172) showed that LLMs attend more strongly to context at the start and end of the prompt than in the middle — a U-shaped accuracy curve. The "RAGAS" framework (Es et al., 2023, arXiv:2309.15217) and Anthropic's "Contextual Retrieval" research (September 2024) propose metrics and improvements such as contextual chunk headers, which reduce retrieval failure by up to 49% on Anthropic's published benchmarks.

Frameworks have proliferated. LangChain (first commit October 2022) and LlamaIndex (initially "GPT Index," November 2022) dominate the application layer. NVIDIA released the NeMo Retriever blueprint in 2024. Microsoft's GraphRAG (arXiv:2404.16130, April 2024) extends RAG with knowledge-graph construction over the corpus, and Anthropic's Model Context Protocol (MCP), open-sourced November 2024, standardizes how LLM clients connect to retrieval servers.

RAG is not a panacea. Long-context models (Gemini 1.5 Pro at 2M tokens, Claude 3.5 Sonnet at 200K) reduce some use cases for retrieval, and fine-tuning remains better when you need the model to internalize a style or a behavior rather than recall a fact. The practical rule is: use RAG for facts that change, and fine-tune for behavior that doesn't.