How Do You Benchmark Citation Verification in Document RAG?

Direct answer

Benchmark citation verification in document RAG by evaluating claim-citation pairs against frozen, fingerprinted source artifacts. Build a gold dataset containing valid evidence and controlled failures, run the verifier under pinned conditions, and score source identity, locator correctness, literal evidence match, semantic support, citation completeness, failure classification, and reproducibility as separate dimensions.

The most important benchmark design choice is separation. Deterministic checks—whether a document fingerprint, page, quote, table cell, or region exists—must not be blended with probabilistic judgments about whether the evidence semantically supports a paraphrased claim. A single “groundedness score” hides which layer failed.

A credible benchmark publishes its corpus manifest, label policy, fixture hashes, verifier configuration, run command, raw reports, and limitations. Without those artifacts, a percentage is difficult to reproduce and easy to overinterpret.

Define exactly what the benchmark measures

“Citation quality” can refer to at least six different properties:

Specify the target before selecting metrics. A deterministic citation verifier can be excellent at the first three while intentionally declining to judge semantic support. A model-based evaluator may assess support while failing to detect that the cited source fingerprint is stale.

The ALCE work on generating text with citations distinguishes citation quality concerns such as correctness and completeness. RAGAS provides reference-free evaluation components for RAG pipelines. These frameworks are useful context, but a document-verification benchmark still needs exact source artifacts and locator-level labels.

Freeze the benchmark inputs

Reproducibility begins with immutable inputs. A benchmark manifest should record:

• corpus and fixture version;
• SHA-256 digest for every source artifact;
• parser and adapter names and versions;
• embedding, generator, and judge versions if those stages are included;
• chunking and normalization configuration;
• verifier version and verification-config digest;
• operating system, architecture, and relevant runtime versions;
• random seeds and decoding parameters for probabilistic components;
• benchmark schema and label-policy version.

Use license-clean documents that the benchmark can redistribute, or publish scripts that reconstruct them from permitted sources. Synthetic fixtures are valuable because they let maintainers control exact pages, tables, duplicate phrases, and adversarial mutations. Real-world documents add layout and language diversity but require careful licensing and privacy review.

Reproducibility rule: Never silently replace a source PDF while keeping the same fixture ID. A changed byte stream is a new benchmark input, even when the visible text appears unchanged.

Create a gold claim-citation dataset

The atomic benchmark unit should be a claim-citation case, not an entire chatbot conversation. Each case needs enough information to determine the expected result without reconstructing hidden application state.

{
  "case_id": "quote-wrong-page-001",
  "source": {
    "artifact": "fixtures/policy-v3.pdf",
    "sha256": "5f2c8e2b8c1c1d64f37f1f01a5f74cb98a42a8c9b53b51df02de4d96fe11b99c"
  },
  "claim": "Customer backups are retained for 90 days.",
  "citation": {
    "page": 4,
    "expected_text": "Customer backups are retained for 90 days."
  },
  "gold": {
    "source_identity": "match",
    "locator": "not_found",
    "evidence_match": "not_evaluated",
    "semantic_support": "supported_elsewhere",
    "expected_reason": "quote_not_found_on_page"
  }
}

This example deliberately places correct text on the wrong page. A whole-document search may call it correct, while a locator-aware verifier should reject the submitted citation. The gold label records both the failed binding and the fact that supporting text exists elsewhere.

Every case should have a stable ID, source digest, submitted claim, submitted citation, expected outcomes by layer, and evidence for the label. Human-labeled semantic cases should also record annotator count, adjudication procedure, and disagreement.

Use controlled adversarial mutations

A benchmark made only of valid citations measures the happy path and rewards systems that always say “verified.” Pair grounded fixtures with mutations that isolate one failure at a time.

Source mutations

• stale or substituted fingerprint;
• correct document title but different bytes;
• draft versus approved version;
• right text from the wrong tenant or jurisdiction;
• missing fingerprint capability.

Locator mutations

• page incremented or decremented;
• valid element ID from another page;
• row and column swapped;
• table ID changed while cell text remains duplicated;
• bounding box moved outside page bounds;
• coordinate origin declared incorrectly.

Content mutations

• one changed digit, sign, date, unit, or currency;
• inserted or removed negation;
• normalized whitespace versus material punctuation;
• quote truncated before an exception;
• Unicode confusable character substitution;
• value present elsewhere but absent at the submitted locator.

Semantic and coverage mutations

• evidence is relevant but does not entail the claim;
• evidence directly contradicts the claim;
• answer contains an uncited material claim;
• synthesis combines two facts into unsupported causation;
• citation supports only part of a compound claim.

Keep mutations minimal. When a case changes page, text, fingerprint, and authority simultaneously, the benchmark cannot identify which check detected the failure.

Include capability-limited cases

Document evidence is heterogeneous. A plain-text adapter cannot prove the same claims as a source with pages, spans, tables, and coordinates.

Include cases where the correct outcome is neither pass nor ordinary mismatch, but an explicit limitation:

• table-cell verification requested from flattened text;
• region crop requested with unknown coordinate origin;
• character offset requested without stable spans;
• freshness check requested without a source fingerprint;
• scanned page submitted to a no-OCR base path;
• unsupported claim kind submitted to the verifier.

A system should receive credit for refusing to overclaim. Scoring every capability-limited outcome as a generic failure encourages unsafe fallback behavior such as replacing a table-cell check with whole-page text presence.

Split deterministic and semantic tracks

Run at least two benchmark tracks.

Track A: deterministic citation grounding measures source fingerprints, locators, literal or bounded normalized matches, table cells, regions, reason codes, and repeatability. Exact expected outputs can be stored as gold records.

Track B: semantic claim support measures whether evidence entails, contradicts, or fails to address the claim. Use expert adjudication as the reference. If an LLM judge is included, pin the model snapshot and rubric, repeat nondeterministic runs, and measure agreement by class.

RAGTruth provides a corpus for analyzing hallucinations in RAG outputs, while RAGChecker proposes fine-grained diagnosis of retrieval and generation. Those resources reinforce the value of decomposed evaluation. They do not remove the need for locator-specific gold evidence when testing document citations.

Do not let a semantic judge override deterministic facts. If the judge says a claim is supported but the citation points to a nonexistent page, report semantic support and citation failure separately.

Report failure-aware metrics

Overall accuracy can hide dangerous behavior when valid cases outnumber adversarial cases. Publish a confusion matrix and per-class metrics.

For each verification class, report:

• precision, recall, and F1;
• false-accept rate for invalid citations;
• false-reject rate for valid citations;
• reason-code accuracy;
• capability-limit recognition rate;
• claim-level citation coverage;
• macro average across failure classes;
• results by document type, claim kind, and locator type.

False acceptance deserves special attention because it converts unsupported evidence into product trust. Report it separately for stale source, wrong locator, altered quote, wrong table cell, and unsupported claim cases.

For semantic judges, also report repeated-run stability and agreement with experts. If the judge returns confidence, assess calibration rather than assuming the number is a probability.

from dataclasses import dataclass

@dataclass(frozen=True)
class ClassMetrics:
    precision: float
    recall: float
    f1: float

def class_metrics(tp: int, fp: int, fn: int) -> ClassMetrics:
    precision = tp / (tp + fp) if tp + fp else 0.0
    recall = tp / (tp + fn) if tp + fn else 0.0
    f1 = (
        2 * precision * recall / (precision + recall)
        if precision + recall
        else 0.0
    )
    return ClassMetrics(precision=precision, recall=recall, f1=f1)

Use this calculation per class rather than only on pooled outcomes. A verifier that handles quotes well but accepts wrong table cells needs a visible table-cell result.

Measure citation completeness at claim level

Citation correctness asks whether a supplied citation supports its target. Completeness asks whether every material claim has sufficient citation coverage. They require different denominators.

Annotate material claims, then compute:

[ \text{verified coverage}=\frac{\text{material claims with required verified evidence}}{\text{all material claims}} ]

Do not count citation markers. One marker may support several claims, or several markers may collectively support one synthesis. Map claims to evidence checks explicitly.

Use claim-level citation verification to prevent a supported sentence from lending credibility to an adjacent unsupported sentence.

Test deterministic reproducibility

For deterministic verification, repeated runs over identical inputs and pinned configuration should produce the same canonical payload. Exclude explicitly volatile envelope fields such as wall-clock timestamps before comparison, or avoid them in the canonical result.

Run the same fixture multiple times on each supported platform and compare:

• ordered check IDs and statuses;
• source and configuration digests;
• observed evidence projections;
• warnings and capability limits;
• canonical JSON hashes;
• exit codes for pass, verification failure, and invalid request.

Cross-platform differences should be documented rather than hidden. Rendering artifacts may require a separate tolerance policy, while source-bound crop descriptors can remain the canonical evidence object.

Prevent benchmark contamination and leakage

Separate development and final test sets. Keep some adversarial fixtures private when the benchmark is used for release gating. If generators or judges may have seen public questions during training, disclose that limitation and use newly constructed variants.

Do not tune thresholds on the final set. Version every label correction. Preserve prior results so readers can distinguish model improvement from benchmark drift.

For CI, run a small deterministic regression suite on every change and the full corpus on scheduled or release workflows. The guide to testing citation verification in CI shows how to turn these fixtures into release gates.

Publish an auditable benchmark packet

A result should ship with:

• benchmark README and scope statement;
• corpus manifest and source hashes;
• license and privacy notes;
• schema and label guidelines;
• pinned dependency and model versions;
• exact run commands;
• hardware and runtime metadata where performance is reported;
• raw machine-readable reports;
• aggregation code;
• confusion matrices and slice results;
• known limitations and excluded claims.

The benchmark should make weak claims precisely. “98% citation accuracy” is incomplete without class balance, definitions, and raw evidence. “Quote-locator F1 on version 1.2 of this frozen born-digital PDF corpus under configuration X” is narrower but reproducible.

Use the structure in what a RAG verification report should contain for per-case evidence, then aggregate without discarding reasons and limitations.

Where Ethos fits

Ethos can participate in the deterministic grounding track by checking caller-provided citation claims against native or supported foreign-parser GroundingSource evidence and emitting structured verification reports. Its useful benchmark cases include grounded quotes, stale fingerprints, wrong pages, altered values, invalid table cells, unsupported claim kinds, and missing capabilities.

Ethos should not be described as semantically validating an entire answer, and its current repository explicitly does not authorize public benchmark-performance or leaderboard claims. Any future public result should come from a separately governed, reproducible benchmark process with approved evidence and claim language.

In a combined evaluation stack, the harness owns fixtures and orchestration, Ethos supplies deterministic source-bound checks, and a separately calibrated evaluator handles semantic support. This preserves the boundary between evidence facts and probabilistic judgment.

The definitive verdict

The strongest document RAG citation benchmark is not a single score. It is a frozen evidence corpus, explicit claim-citation labels, minimal adversarial mutations, separate deterministic and semantic tracks, failure-aware metrics, and a reproducible result packet.

Start with source identity, locator correctness, evidence match, and capability limits. Add semantic support and claim coverage without overwriting those facts. Treat false acceptance as a first-class risk, publish slice results, and require exact rerun instructions. That benchmark can reveal whether a system merely produces plausible citations or actually verifies document evidence.

Primary keyword: benchmark citation verification in document RAG Optimized meta title: Benchmark Citation Verification in Document RAG Optimized meta description: Build a reproducible document RAG citation benchmark with source-bound fixtures, adversarial cases, explicit labels, and failure-aware metrics. Proposed URL slug: how-do-you-benchmark-citation-verification-in-document-rag