How Do You Build a Fail-Closed Citation Verification Pipeline?

Direct answer

Build a fail-closed citation verification pipeline by placing a mandatory release gate between answer generation and user delivery. The gate must release citation-dependent claims only when required source, fingerprint, locator, evidence, schema, and policy checks produce an understood passing result. Missing reports, unknown statuses, stale evidence, mismatches, timeouts, malformed output, and verifier failures must never default to approval.

Fail-closed does not require showing a generic error for every defect. Safe outcomes include a verified partial answer, an explicit abstention, a review task, or a bounded retry. What it forbids is bypassing the gate because verification was inconvenient or unavailable.

The NIST glossary defines fail secure as terminating functions in a way that prevents loss of secure state when failure occurs or is detected. In RAG, the protected state is “unverified citation-dependent claims are not released.”

Define the protected invariant

Write the invariant before code:

No material citation-dependent claim may reach an unauthorized user-visible
or machine-consumable output unless every mandatory release check for that
claim completed under an approved contract and passed.

This applies to:

• chat responses;
• exported reports;
• emails and notifications;
• API responses;
• agent tool outputs;
• downstream database writes;
• summaries cached for later reuse.

If one channel bypasses the gate, the system is not fail-closed.

The pipeline architecture

authorized request
   -> governed source selection
   -> retrieval with provenance
   -> structured answer + atomic claims + citations
   -> citation request schema validation
   -> deterministic Ethos verification
   -> calculations and semantic policy
   -> claim-level release decision
   -> final / partial / review / abstain / block

The raw model answer should not be streamed directly to the end user when mandatory verification happens afterward. Buffer it as an untrusted candidate, or stream only content classes that policy allows before verification.

Step 1: classify outputs by verification requirement

Define content classes:

Never let the model classify its own unsupported claim as exempt without validation.

Step 2: validate the generated answer contract

Require:

• unique answer and claim IDs;
• claim text and type;
• evidence references for source-dependent claims;
• calculation dependencies;
• no unknown fields when the schema is strict;
• bounded lengths and valid encodings;
• source IDs resolved through a trusted map.

Malformed output is a candidate-generation failure. Retry generation under a bounded policy or abstain; do not pass the raw prose around the verifier.

Step 3: select and authorize the source before verification

The application must:

• select the authoritative source revision;
• enforce tenant and resource authorization;
• resolve model-returned IDs through controlled mappings;
• verify that source artifacts and citations belong to the same request scope;
• prevent arbitrary file paths or URLs;
• capture source-set and policy identity.

Ethos verifies evidence against the GroundingSource it receives. It does not decide whether the application selected the correct corporate, legal, clinical, or policy authority.

Step 4: separate evidence verdicts from infrastructure failures

These categories must not share one false value.

An infrastructure failure says “no trustworthy verdict was produced.” It does not say “citation failed,” and it certainly does not say “citation passed.”

Step 5: call Ethos with explicit error handling

The Python wrapper returns JSON reports for completed verification and raises structured exceptions for command or timeout failures. When fail_on_ungrounded=True, CLI exit 1 with JSON is a completed negative verification verdict, not a process crash.

from dataclasses import dataclass
from pathlib import Path
from typing import Literal, Optional

from ethos_pdf import (
    EthosCli,
    EthosCommandError,
    EthosNotFoundError,
    EthosOutputError,
    EthosTimeoutError,
    proof_summary,
)


@dataclass(frozen=True)
class GateResult:
    state: Literal["pass", "reject", "infrastructure_failure"]
    reason: str
    report: Optional[dict]


def run_grounding_gate(
    *,
    binary: str,
    source: Path,
    citations: Path,
) -> GateResult:
    verifier = EthosCli(binary=binary)
    try:
        report = verifier.verify(
            source=source,
            citations=citations,
            fail_on_ungrounded=True,
            output_format="json",
            timeout=30,
        )
    except (
        EthosNotFoundError,
        EthosTimeoutError,
        EthosOutputError,
        EthosCommandError,
    ) as exc:
        return GateResult(
            state="infrastructure_failure",
            reason=type(exc).__name__,
            report=None,
        )

    summary = proof_summary(report)
    if summary["request_certified"]:
        return GateResult(state="pass", reason="request_certified", report=report)
    return GateResult(
        state="reject",
        reason=summary["proof_status"],
        report=report,
    )

The release caller must handle every state explicitly. Store the completed canonical report even when the outcome is negative. For infrastructure failures, retain privacy-safe diagnostics and request identity without leaking source content.

Step 6: make the release switch exhaustive

def final_action(result: GateResult) -> str:
    if result.state == "pass":
        return "continue_to_semantic_and_policy_checks"
    if result.state == "reject":
        return "block_or_build_verified_partial"
    if result.state == "infrastructure_failure":
        return "abstain_or_route_to_review"
    raise RuntimeError(f"Unhandled gate state: {result.state!r}")

Unknown states raise. They do not fall through to release.

In TypeScript, use a discriminated union and an exhaustive never check. In any language, avoid defaults such as allow = report?.success ?? true.

Step 7: add bounded retries without bypass

Retry only failures likely to be transient:

• worker unavailable;
• temporary filesystem or service error;
• bounded timeout under known load;
• idempotent artifact read.

Do not retry semantic mismatches, stale fingerprints, unsupported claims, or malformed citation requests as though randomness might produce a pass.

Retry rules should specify:

• maximum attempts;
• total request deadline;
• idempotency key;
• backoff and jitter;
• artifact immutability across attempts;
• final abstention or review action;
• metrics and alerting.

A circuit breaker may protect infrastructure, but its open state must not bypass verification. It should move requests to abstention, queue, or review.

Step 8: support safe partial answers

A partial answer is allowed only when:

• reusable grounded checks exist;
• source fingerprint is not stale;
• released claims are relevant source facts;
• dependencies and calculations pass;
• removed claims do not make the remaining answer misleading;
• policy allows partial release;
• the user sees a clear partial-status disclosure.

The partially verified answer guide provides the claim-selection policy. Build final text from approved claim IDs rather than deleting sentences from raw prose.

Step 9: define safe fallback copy

Use deterministic templates:

• No evidence: “The available sources do not support an answer.”
• Verifier unavailable: “I could not verify the cited evidence, so I am not returning a source-dependent answer.”
• Partial: “The following facts were verified; the remaining part could not be confirmed.”
• Review required: “The evidence requires authorized review before this conclusion can be shown.”
• Stale source: “The citation belongs to another source version. The answer must be regenerated.”

Do not ask the model to improvise the fallback after verification fails. It may restate the blocked claim.

Step 10: secure every output path

Audit:

• streaming tokens;
• websocket events;
• server logs;
• traces and metrics;
• error pages;
• cached responses;
• exports and webhooks;
• agent memory;
• notification systems;
• reviewer queues.

The unverified draft may contain sensitive or unsupported content. Restrict it like any other untrusted artifact.

Step 11: set timeout and availability budgets

Verification belongs in the latency budget. Measure:

• p50, p95, and p99 verification latency;
• timeout rate;
• queue time;
• report size;
• failure rate by source and claim kind;
• partial-answer and abstention rate;
• reviewer backlog;
• retry effectiveness.

Do not increase the timeout indefinitely to hide capacity problems. Size workers, isolate expensive crop rendering, cap document and citation complexity, and alert before availability pressure becomes a broad abstention incident.

Step 12: preserve audit evidence

Retain:

• request and answer candidate IDs;
• source-set and fingerprint identity;
• citation request;
• canonical verification report or infrastructure-error envelope;
• proof summary;
• semantic and calculation results;
• release-policy version;
• final action and user-facing copy;
• retries, overrides, and reviewer actions.

The RAG verification report guide defines the canonical evidence fields.

Adversarial test matrix

Test recovery too. After the verifier returns, queued requests must still use current source fingerprints and authorization.

Operational anti-patterns

Fail open after a timeout

This releases the riskiest answers when the control is least available.

Treat missing report as zero findings

No report means no completed verification, not no defects.

Catch every exception and return the model draft

This turns error handling into a verification bypass.

Trust a cached positive report after source change

Cache keys must include source and verification contract identity.

Retry until a probabilistic judge passes

Repeated sampling is not deterministic verification and can select a lucky verdict.

The definitive verdict

A fail-closed citation pipeline protects one invariant: unverified source-dependent claims do not escape. Enforce that invariant across generation, streaming, APIs, caches, logs, and downstream automation. Distinguish completed evidence rejection from capability limitation and infrastructure failure, and make every state lead to an explicit safe action.

Use Ethos for structured deterministic grounding reports, application code for exhaustive release switching, bounded retries for transient infrastructure, and transparent partial or abstention responses for users. The pipeline is fail-closed only when the unavailable verifier is treated as a reason to withhold claims—not a reason to trust them.

Primary keyword: fail-closed citation verification Optimized meta title: How to Build Fail-Closed Citation Verification Optimized meta description: Build a fail-closed RAG citation pipeline that blocks stale, mismatched, unsupported, timed-out, and unverifiable claims before release. Proposed URL slug: how-do-you-build-a-fail-closed-citation-verification-pipeline