What Should a RAG Verification Report Contain?

Direct answer

A RAG verification report should contain enough structured evidence for an authorized reviewer or application to reconstruct what was checked, against which source version, with which verifier configuration, what evidence was observed, what failed, what the source could not prove, and whether the submitted request was certified.

At minimum, record schema version, verification-configuration identity, source fingerprint, parser and adapter identity, declared source capabilities, fingerprint freshness, ordered claim-level checks, stable statuses and reasons, evidence locators, warnings, unsupported claim kinds, capability limitations, and overall grounding state.

Do not replace this report with a confidence score or a green badge. Scores are useful for trends; canonical check records are necessary for release policy, incident response, and audit.

The report has three distinct consumers

Design the canonical report once, then derive views for different consumers.

The user interface should not parse free-form logs. The audit system should not depend on a dashboard that may change. Store the structured report as the durable contract.

Canonical report, proof summary, and release decision

These are related but different artifacts:

canonical verification report
        |
        +--> derived proof summary
        |       verified / partially_verified / unverified
        |       reusable and review check IDs
        |
        +--> application release decision
                relevance + claim type + policy
                show / review / block

The canonical report records evidence-grounding results. A proof summary makes those results easier to consume. An application release decision combines grounding with question relevance, synthesis, calculation, authority, and domain rules.

Do not overwrite the canonical report when the application decides to release a partial answer.

Field group 1: schema and configuration identity

Record:

• report schema version;
• verifier software and contract version;
• verification configuration hash;
• normalization profile;
• run or request ID;
• generation timestamp if policy permits;
• environment or deployment identity where needed.

Why it matters: a result can change when normalization, supported claim kinds, status precedence, or capability policy changes. The configuration hash lets operators determine whether two reports used equivalent verification rules.

Ethos’s canonical schema includes schema_version and verification_config_sha256. The latter is a 64-character lowercase hexadecimal digest without the sha256: prefix under the current schema.

Field group 2: source and grounding identity

Include:

• canonical source ID in the surrounding application record;
• source or document fingerprint when available;
• parser name and version;
• adapter name and version for foreign-parser evidence;
• grounding artifact reference;
• authority, effective-date, and access metadata in the application’s source registry.

The Ethos verification report includes document_fingerprint when the grounding source declares one, plus a grounding.parser object. Business authority does not belong to the deterministic verifier alone; retain it in the linked governed source record.

A matching fingerprint proves source identity under the fingerprint contract. It does not prove that the source is approved, current for the user’s date, or factually correct.

Field group 3: declared source capabilities

The report should say what the source can prove. The current Ethos schema records:

• spans;
• character offsets;
• tables;
• fingerprints;
• coordinate origin: top-left, bottom-left, or unknown;
• crop support.

This prevents a plain-text source from silently satisfying a table-cell or region claim.

Also record structured capability limits such as missing spans, missing character offsets, missing tables, missing fingerprint, unknown coordinate origin, or missing crop support.

Capability limitations are evidence. They explain why a check could not be completed at the requested fidelity.

Field group 4: source freshness

Record:

• expected citation fingerprint;
• observed grounding-source fingerprint where policy allows;
• fingerprint_stale or equivalent status;
• stable reason for missing or mismatched fingerprints;
• source-set or policy-registry reference used to choose the source.

Staleness should be report-level when a citation set was prepared against another source identity. Textually matching checks should not become reusable proof while the report is stale.

Field group 5: ordered claim-level checks

Every check should have:

• stable unique check ID;
• claim kind;
• expected claim text or value when applicable;
• citation locator;
• status;
• stable reason for non-grounded outcomes;
• match method;
• semantic-unverified flag;
• observed evidence when found;
• check-level warnings.

Preserve request order. Stable ordering simplifies reproducible reports, UI mapping, golden tests, and claim-to-check references.

Claim kinds

Examples include:

• quote;
• value;
• presence;
• table cell;
• region where supported;
• other or unsupported kinds that must remain explicit.

Never coerce an unsupported claim kind into a weaker check without recording the downgrade.

Field group 6: citation locators

Depending on the claim, preserve:

• page;
• element ID;
• span ID;
• table ID;
• row and column cell coordinates;
• bounding box;
• region or crop reference where supported.

A report should record the locator submitted, not only the corrected locator an operator might discover later. Correction is a new event and verification run.

Field group 7: statuses and stable reasons

Categorical outcomes are more actionable than one score.

Stable reason codes should distinguish missing locator, missing text, stale fingerprint, missing capabilities, page or element not found, table or cell not found, unknown coordinates, and text mismatch.

Applications should switch on structured codes, never human-readable reason strings.

Field group 8: match method

Record how evidence matched:

• exact equality;
• normalized equality;
• exact containment;
• normalized containment;
• table-cell lookup;
• bounding-box containment;
• presence only;
• none.

The normalization profile must be bounded and versioned. “Normalized match” is meaningless without knowing whether the system changed whitespace only or also removed punctuation, case, signs, or other material content.

Ethos’s current schema explicitly identifies match methods and avoids fuzzy semantic matching in the literal evidence contract.

Field group 9: observed evidence

When permitted, a check can record:

• observed text;
• page;
• bounding box;
• opaque crop or evidence reference;
• typed value context in the application layer.

Minimize sensitive duplication. A report can reference protected evidence rather than embed a large passage. Ensure the reviewer who opens the reference is reauthorized for the source.

Field group 10: warnings and limitations

Warnings may describe:

• low-confidence reading order;
• low-confidence table structure;
• hidden or off-page text;
• low-contrast text;
• annotations or external links;
• image-only pages;
• unsupported annotations;
• partial parsing;
• capability limitations.

Warnings should not silently change a failure into success. Application policy decides whether a grounded check with a warning is releasable, review-only, or blocked.

Field group 11: overall certification

The current Ethos invariant sets all_evidence_grounded to true only when:

1. at least one supported check exists;
2. every supported check is grounded;
3. no check is semantically unverified;
4. unsupported claim kinds are absent;
5. the fingerprint is not stale.

This strict Boolean is suitable as a grounding gate. It is not a claim that the full answer is semantically correct or authoritative.

An illustrative verification report

{
  "schema_version": "1.0.0",
  "document_fingerprint": "sha256:b5d30710d0c25cc38d8dec924ecaf57ae4f81276dd5dc14d75cb3b5b6bde62d3",
  "verification_config_sha256": "4bb224166a04a25fed2dd3ecdb9638ddcc5b398658532b73f1c0547e4983d0b0",
  "grounding": {
    "parser": {
      "name": "approved-parser",
      "version": "pinned-version",
      "adapter": "approved-adapter",
      "adapter_version": "pinned-adapter-version"
    },
    "capabilities": {
      "spans": true,
      "char_offsets": true,
      "tables": true,
      "fingerprint": true,
      "coordinate_origin": "top-left",
      "crop_support": true
    }
  },
  "capability_limits": [],
  "fingerprint_stale": false,
  "all_evidence_grounded": false,
  "checks": [
    {
      "id": "v0001",
      "claim": {
        "kind": "quote",
        "text": "Requests must be submitted within 30 days.",
        "citation": {
          "page": "18",
          "element_id": "p17-e42"
        }
      },
      "status": "grounded",
      "match_method": "normalized_text_contains",
      "semantic_unverified": false,
      "evidence": {
        "text": "Requests must be submitted within 30 days.",
        "page": "18",
        "bbox": [72, 188, 524, 232]
      },
      "warnings": []
    },
    {
      "id": "v0002",
      "claim": {
        "kind": "value",
        "text": "$1,500",
        "citation": {
          "page": "19",
          "element_id": "p18-e08"
        }
      },
      "status": "mismatch",
      "reason": "text_mismatch",
      "match_method": "normalized_text",
      "semantic_unverified": false,
      "evidence": {
        "text": "$1,200",
        "page": "19",
        "bbox": [310, 244, 372, 267]
      },
      "warnings": []
    }
  ],
  "unsupported_claim_kinds": [],
  "warnings": []
}

Privacy, retention, and integrity

Define:

• report access policy;
• encryption and key ownership;
• retention by risk and domain;
• whether source text or crops may be embedded;
• append-only or immutable storage requirements;
• deletion propagation;
• report signing or integrity checks;
• tenant isolation;
• redacted analytics view.

An audit report can contain confidential source text even when the final answer is public. Classify it accordingly.

Testing the report contract

Test:

• schema validation;
• deterministic field ordering where byte stability is required;
• duplicate check IDs;
• unsupported claim kinds;
• stale fingerprints;
• every stable status and reason;
• missing capabilities;
• invalid locators;
• warnings and partial parses;
• report-to-claim mapping;
• privacy redaction;
• old schema migration;
• malformed verifier output;
• release service behavior for unknown future fields or versions.

Fail closed when the release service cannot understand the report version. Silently ignoring new fields can erase a safety condition.

The definitive verdict

A RAG verification report should be a canonical, schema-versioned evidence artifact—not a scorecard. It must identify the source and verification contract, declare what the grounding source can prove, preserve every claim-level check and reason, expose staleness and limitations, and compute overall certification under a documented invariant.

Use Ethos’s report as the citation-grounding record, derive proof summaries for product wording, and create a separate application release decision for relevance, synthesis, calculation, and policy. The audit-ready answer guide shows how those artifacts fit together.

Primary keyword: RAG verification report Optimized meta title: What a RAG Verification Report Should Contain Optimized meta description: Design RAG verification reports with source identity, capabilities, claim checks, evidence, stable reasons, limitations, and release references. Proposed URL slug: what-should-a-rag-verification-report-contain