How Does RAG Work?

Direct answer

RAG works in two stages: retrieve relevant source material, then generate an answer using that material as context.

Question
  -> retrieve evidence
  -> filter and rank sources
  -> provide context to the model
  -> generate answer
  -> show and evaluate citations

The retrieval stage and generation stage should remain separate. Retrieval finds candidates. Generation synthesizes language. A citation can point to a retrieved passage, but the system still needs to check whether the passage supports the claim.

Prepare the source corpus

Collect documentation, PDFs, API references, tickets, or other approved sources. Validate files and record source ID, owner, version, audience, visibility, canonical URL, update time, and content fingerprint.

For PDF-heavy sources, preserve headings, tables, page numbers, and warnings before indexing. DocuShell’s PDF to JSON workflow is an example of the structured extraction boundary.

Parse and chunk the content

Turn each source into structured records. Keep a section’s parent heading and document title with its text. Chunk by meaning when possible: one procedure, API endpoint, policy clause, troubleshooting path, or table with its header and notes.

Bad chunking can make a good corpus look unreliable. If a chunk begins in the middle of an instruction or separates a table from its label, the model has less context to work with.

Index the chunks

A RAG index may use keyword search, vector search, or both. Keyword retrieval helps exact terms. Semantic retrieval helps paraphrased questions. Metadata filters narrow by version, product, access, and audience.

Read what semantic search is and what vector search is before choosing a retrieval strategy.

Retrieve and rerank

At query time:

1. interpret the question;
2. identify required filters;
3. retrieve keyword and semantic candidates;
4. remove duplicates;
5. rerank by relevance, source quality, version, and scope;
6. return a small evidence set.

Do not pass every matching document to the model. More context can introduce contradictions, stale instructions, and irrelevant passages.

Generate with bounded context

The model receives the question and retrieved sources. The answer contract should require it to answer only from the supplied corpus, preserve version and scope qualifiers, cite the passages used, say when evidence is missing or conflicting, and avoid inventing parameters, steps, or policy exceptions.

Prompting helps, but it cannot make an incomplete corpus complete.

Return useful citations

A citation should help a reader inspect the source. Include a title, section, version, stable URL or PDF page, and the relevant passage when appropriate.

Keep retrieval references separate from verified claims. “This page was retrieved” is not identical to “this sentence is supported by the page.” How to prevent AI hallucinations covers that distinction in more detail.

Evaluate the complete pipeline

Use real questions and include cases that should fail. Measure retrieval recall, answer correctness, citation support, source-version accuracy, permission safety, refusal quality, latency, and cost.

A high-quality answer with a wrong citation is still a failure. A correct refusal when the corpus lacks evidence can be a success.

Common RAG mistakes

• indexing stale and current versions together;
• flattening tables and losing column context;
• skipping permission filters;
• treating vector similarity as proof;
• showing citations only at the bottom of a long answer;
• evaluating only fluent, answerable questions;
• hiding extraction warnings from reviewers.

Conclusion

RAG is retrieval plus generation, not generation with a larger prompt. Keep source preparation, search, filtering, answer synthesis, citations, and evaluation as separate responsibilities. That separation is what makes a RAG system explainable and maintainable.

Practical next steps

For PDF sources, begin with source-aware parsing, then create chunks that preserve page citations. Add claim-level page citations to the answer interface.

Evaluate the system with RAG faithfulness and citation quality. Plan operating spend using How Much Does a RAG Chatbot Cost Per Month? and monthly LLM token-cost estimation.