How Do You Test RAG Citations in CI/CD?

Direct answer

Test RAG citations in CI/CD with a layered suite. Run fast deterministic fixtures on every pull request, assert exact outcomes for grounded and adversarial citations, validate schemas and source fingerprints, and reserve live retrieval or LLM evaluation for controlled integration jobs. Block release on deterministic evidence regressions; calibrate semantic thresholds separately.

The test suite must verify failure behavior, not only successful citations. A verifier that accepts correct quotes but mishandles stale documents, wrong table columns, or fabricated evidence IDs is unsafe.

The deterministic RAG gate guide defines the runtime policy. CI proves that the policy continues to work as parsers, schemas, and applications change.

Separate the test pyramid

Do not make every pull request depend on a variable external model response. Equally, do not assume fixture tests prove live retrieval and generation health.

Build a versioned citation fixture

Each fixture should include:

• stable fixture ID;
• license-safe or private controlled source artifact;
• source fingerprint;
• parser and normalization profile;
• citation request;
• expected verification status and reason;
• expected exit behavior;
• optional release-policy decision.

{
  "fixture_id": "wrong-page-quote-001",
  "source": "fixtures/policy.ethos.json",
  "source_fingerprint": "sha256:pinned-fixture-digest",
  "citation": {
    "claim_id": "travel-window",
    "page_index": 4,
    "expected_quote": "Claims must be submitted within 30 days."
  },
  "expected": {
    "status": "not_found",
    "release_action": "block"
  }
}

Store source artifacts and expected reports together when privacy and size permit. Otherwise use controlled test storage with immutable digests.

The minimum adversarial suite

1. Correct quote, source, page, and fingerprint.
2. Correct quote on the wrong page.
3. One changed negation.
4. Missing document or evidence ID.
5. Stale source fingerprint.
6. Correct value with wrong unit, sign, or scale.
7. Correct table value in the wrong row or column.
8. Out-of-range page or region coordinates.
9. Evidence request requiring an unsupported parser capability.
10. Malformed citation schema.
11. Duplicate claim IDs.
12. Grounded evidence attached to an irrelevant claim.
13. Multi-claim answer with incomplete citation coverage.
14. Calculation with grounded inputs and wrong output.
15. Verifier timeout or unavailable dependency.

The first eleven are primarily deterministic. The final four exercise application policy and operational resilience.

Assert exact statuses, not snapshots alone

Full-report golden files are useful for contract drift, but brittle snapshots can obscure intent. Pair them with focused assertions:

import assert from "node:assert/strict";

type VerificationCheck = {
  id: string;
  status: "grounded" | "mismatch" | "not_found" | "capability_limited";
  fingerprint_stale: boolean;
};

export function assertStaleCitation(check: VerificationCheck): void {
  assert.equal(check.status, "mismatch");
  assert.equal(check.fingerprint_stale, true);
}

export function assertReusableGrounding(check: VerificationCheck): void {
  assert.equal(check.status, "grounded");
  assert.equal(check.fingerprint_stale, false);
}

Use the exact status shape from your verifier contract. The example demonstrates test intent rather than defining Ethos schema fields.

Test normalization independently

Quote comparison often normalizes Unicode, whitespace, and line-break hyphenation. Build table-driven unit tests for:

• composed and decomposed Unicode;
• non-breaking spaces;
• line endings;
• repeated whitespace;
• discretionary hyphenation;
• smart punctuation;
• meaningful minus signs and dashes;
• case sensitivity where material;
• decimal and thousands separators;
• currency and percentage symbols.

Normalization must not erase negation, units, signs, or punctuation that changes meaning. Pin a profile version and re-run all citation fixtures when rules change.

Test parser-to-citation stability

Citation verification depends on parsed evidence. Parser changes can alter element IDs, reading order, table structure, or coordinates without changing the source PDF.

For representative documents, assert:

• page count and page identity;
• stable source fingerprint;
• required element text and page assignment;
• table row and column associations;
• bounding-box validity;
• declared parser capabilities;
• expected warnings for unsupported content.

The source-aware parsing guide explains why these invariants matter.

Use Ethos in CI

Ethos verification reports separate negative evidence results from invalid input. The CLI’s fail-on-ungrounded behavior can turn completed but ungrounded verification into a non-zero gate while preserving a structured report for diagnostics.

A representative shell job can be explicit about expected outcomes:

set -euo pipefail

ethos verify fixtures/document.ethos.json \
  --citations fixtures/citations.grounded.json \
  --fail-on-ungrounded \
  --out artifacts/grounded-report.json

if ethos verify fixtures/document.ethos.json \
  --citations fixtures/citations.stale.json \
  --fail-on-ungrounded \
  --out artifacts/stale-report.json; then
  echo "Expected stale citation verification to fail the gate" >&2
  exit 1
fi

The negative test should also parse stale-report.json and assert the expected reason rather than accepting every non-zero exit. An invalid request and an expected stale result are not equivalent.

CI workflow design

name: rag-citation-contract

on:
  pull_request:
  push:
    branches: [main]

jobs:
  citation-fixtures:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Install pinned verifier
        run: npm install --global @docushell/[email protected]
      - name: Run citation contract tests
        run: npm run test:citations
      - name: Upload verification reports
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: citation-verification-reports
          path: artifacts/*.json

Pin action and dependency versions according to your supply-chain policy. PDFium-backed commands require caller-provided PDFium; JSON verification does not need PDFium unless the selected path invokes parsing or rendering.

Keep fixture and live modes distinct

Fixture mode proves the evaluator and integration contract. It does not prove that production retrieval, document parsing, authentication, or model generation currently works.

Live mode should test:

• source ingestion and indexing;
• authorization filters;
• retrieval of required evidence;
• structured claim and citation generation;
• deterministic verification;
• semantic support and relevance;
• final release decision.

Run live tests with dedicated credentials, pinned model snapshots where possible, telemetry controls, redacted artifacts, and cost limits.

Semantic evaluation in CI

LLM judges are useful for support and relevance, but they introduce variability. To use them responsibly:

• maintain expert-labeled calibration cases;
• pin model and rubric versions;
• require structured labels;
• measure false-pass rate by failure class;
• use repeated or adjudicated runs for ambiguous cases;
• separate hard deterministic blockers from statistical thresholds;
• do not expose confidential source text to an unapproved provider.

The LLM-as-a-judge reliability guide provides the calibration workflow.

Release thresholds and baseline changes

Never update golden files automatically because a test failed. Require review that explains:

• why the source or expected result changed;
• whether the new behavior is safer;
• which parser, schema, or policy version caused the change;
• whether historical receipts require revalidation;
• whether documentation and migration notes need updates.

For statistical metrics, store the distribution and sample size, not only an average. Define a minimum acceptable result and a maximum regression relative to baseline.

Protect test evidence

Use public, synthetic, or license-safe fixtures in open repositories. For private enterprise suites:

• restrict artifact access;
• redact model and verifier outputs;
• avoid uploading raw source documents to third-party CI artifacts;
• set retention limits;
• prevent forked pull requests from receiving secrets;
• use representative synthetic fixtures for public contributions.

Diagnosing failures

Route failures by contract rather than assigning every regression to prompt engineering.

Definitive verdict

Reliable RAG citation CI combines fast deterministic contracts with controlled live evaluation. Pin source artifacts and fingerprints, assert specific negative outcomes, test parser invariants, validate complete claim coverage, and preserve reports for diagnosis. Use semantic judges only where interpretation is required and calibrate them separately.

Use DocuShell Parse PDF to build source-aware fixtures, Ethos as the deterministic verification gate, and the DocuShell hallucination index for wider evaluation context. A release should fail when evidence contracts regress—not when a test merely produces a different-looking explanation.

Primary keyword: test RAG citations in CI/CD
Optimized meta title: How to Test RAG Citations in CI/CD
Optimized meta description: Test RAG citations with pinned fixtures, source fingerprints, exact failure assertions, semantic evals, and reliable CI release gates.
Proposed URL slug: how-do-you-test-rag-citations-in-ci-cd