How Do I Measure RAG Faithfulness and Citation Quality?

Measure claims, not fluent paragraphs

Measure RAG faithfulness by breaking an answer into factual claims and checking whether each claim is supported by the retrieved context. Measure citation quality by checking whether the cited source directly supports the nearby claim and opens to the correct page or region.

Do not combine every failure into one score. Retrieval, generation, and citation linking can fail independently.

Build a useful evaluation set

Include routine questions, questions with no answer, conflicting versions, tables, poor OCR, and adversarial document text. Use the kinds of documents and decisions the product will actually handle.

Create reference evidence, not only reference answers. A model can phrase a correct answer differently while relying on the same source.

Score claims, not paragraphs

Split each answer into checkable factual claims. Mark a claim supported, contradicted, or not established by the retrieved context.

Faithfulness is about support from supplied evidence. Factual correctness against the wider world is a separate check.

Inspect every citation

A citation is not good because the page exists. It should support the exact claim, point to the right document version, and be easy to open.

Use source-aware parsing through DocuShell Parse PDF so chunks keep page and coordinate metadata. The Hallucination Index offers workflow-risk context and links to faithfulness methodology.

Track retrieval and answer failures separately

If the right evidence was never retrieved, changing the generation prompt may not help. If the evidence was present but the answer ignored it, the generation or release policy needs work.

Record retrieval recall, claim support, citation precision, answer completeness, and abstention quality.

Add human review where stakes justify it

Automated evaluators can scale checks, but they can also disagree or reward shallow overlap. Review a sample manually and require human approval for consequential legal, financial, medical, or policy use.

Good evaluation makes failure visible. It does not compress every judgment into a number that nobody can explain.

Define the metrics clearly

Claim support asks whether retrieved evidence entails each factual claim. Citation precision asks how many citations genuinely support the attached claim. Citation recall asks whether claims that needed evidence received a citation. Completeness asks whether the response covered the requested information.

Keep these separate. A response can cite every claim but still omit a critical exception.

Example scoring workflow

Take an answer, split it into atomic claims, and list the citation IDs attached to each. A reviewer or evaluated model marks each claim supported, contradicted, or not established. Then open each citation to verify the page link and source region.

Store the judgment and reason, not only a score. Disagreements reveal unclear rubrics or ambiguous evidence.

Claim: "The policy allows 30 days for refunds."
Citation: Policy v4, section 6.1, page 12
Judgment: Supported
Reason: The cited sentence states the same 30-day window.
Link check: Passed
Version check: Passed

Evaluation rule: a relevant citation is not automatically a supporting citation. Entailment and link correctness are separate checks.

Test abstention and conflict

Include questions whose answer is absent, documents that disagree, and outdated versions. A good system should say when evidence is missing or conflicting rather than combining incompatible sources.

For policy and legal workflows, measure whether the answer identifies the applicable version and jurisdiction.

Use automated judges carefully

Automated evaluation scales, but judge prompts and models have their own failure modes. Calibrate them against human-labeled examples and review a sample of passes and failures.

Start with citation-preserving PDF chunks, implement page-level citations, and use the Hallucination Index for risk context. Compare candidate models using the document Q&A model guide.