How Do You Handle Partially Verified AI Answers?

Direct answer

Handle partially verified AI answers at the claim level, not by averaging the answer into a confidence score. Release only claims whose evidence checks are reusable, relevant, and permitted by policy; route synthesis and capability-limited claims to review; block unsupported or stale claims; and disclose when the result is incomplete.

Before showing the verified subset, confirm that removing failed claims does not make the remaining text misleading. An answer can contain individually true sentences whose meaning depends on an omitted exception, denominator, condition, or conclusion.

Ethos’s derived proof summary uses three grounding states: verified, partially_verified, and unverified. That summary is not the canonical evidence artifact. Preserve the full verification report and apply relevance, claim type, calculation, and release rules above it.

What “partially verified” actually means

In the Ethos contract, partially_verified means the request as submitted is not certified, but at least one check remains reusable. A reusable check must be:

• grounded;
• not marked semantic_unverified;
• not invalidated by report-level fingerprint_stale.

This is narrower than “some checks passed.” A stale report can contain textually grounded checks, but none should be reused as proof because the citations were prepared against another source identity.

The application must still ask whether each reusable grounded claim answers the question and whether it is a direct source fact or synthesis.

Why an average score is unsafe

Suppose an answer contains ten evidence checks and nine pass. A score of 90% looks strong, but the failed check may be:

• the source fingerprint for the entire request;
• a contraindication;
• an exception to a policy;
• the denominator in a percentage;
• the limit that determines coverage;
• the only evidence for the final recommendation.

Claim materiality and dependency matter more than the count. Keep statuses categorical and apply explicit release rules.

The partial-answer decision matrix

This matrix separates citation grounding, question relevance, and synthesis level—the three axes in Ethos’s app answer release contract.

Step 1: decompose the answer before filtering

Start with atomic claims:

{
  "claims": [
    {
      "id": "claim-limit",
      "text": "The reimbursement limit is $1,200.",
      "claim_type": "source_fact",
      "check_ids": ["v0001"]
    },
    {
      "id": "claim-deadline",
      "text": "Claims must be submitted within 30 days.",
      "claim_type": "source_fact",
      "check_ids": ["v0002"]
    },
    {
      "id": "claim-eligibility",
      "text": "Your request is eligible for reimbursement.",
      "claim_type": "synthesis",
      "check_ids": ["v0001", "v0002", "v0003"]
    }
  ]
}

The first two claims may be independently grounded. The eligibility conclusion depends on all three checks plus the user’s facts and application policy.

Do not delete unsupported sentences from a free-form paragraph without reconstructing the grammar and logic. Generate final answer text from approved claim IDs or use a deterministic template where possible.

Step 2: derive the Ethos proof summary

from ethos_pdf import proof_summary


summary = proof_summary(verification_report)

print(summary["proof_status"])
print(summary["request_certified"])
print(summary["reusable_grounded_check_ids"])
print(summary["needs_review_check_ids"])
print(summary["proof_limitations"])

The summary helps select candidate evidence. It does not replace the canonical report, which contains check-level statuses, reasons, source identity, warnings, and capability information.

Step 3: label relevance and claim type

For each claim, assign:

Question relevance

• direct_answer: directly answers the user;
• supports_answer: necessary evidence but insufficient alone;
• background_only: true but not responsive;
• unrelated: does not support the requested answer.

Claim type

• source_fact: directly stated by source evidence;
• synthesis: combines facts or adds inference;
• unsupported: cannot be traced to grounded evidence.

These labels come from application policy, a reviewed output schema, a calibrated evaluator, or human review. Ethos does not infer them.

Step 4: apply the app release decision

The Python surface can build a non-canonical application decision envelope after callers supply relevance and claim-type labels.

from ethos_pdf import app_answer_release_decision, proof_summary


summary = proof_summary(verification_report)
decision = app_answer_release_decision(
    "Is this expense eligible for reimbursement?",
    summary,
    [
        {
            "id": "claim-limit",
            "text": "The reimbursement limit is $1,200.",
            "check_ids": ["v0001"],
            "question_relevance": "supports_answer",
            "claim_type": "source_fact",
        },
        {
            "id": "claim-eligibility",
            "text": "Your request is eligible for reimbursement.",
            "check_ids": ["v0001", "v0002", "v0003"],
            "question_relevance": "direct_answer",
            "claim_type": "synthesis",
        },
    ],
    verification_report_ref="audit://verification/vr-0194",
)

print(decision["app_status"])
print(decision["final_answer_claim_ids"])
print(decision["review_claim_ids"])
print(decision["blocked_claim_ids"])

The helper applies the documented conservative release rule; it does not judge relevance or synthesis. Preserve the canonical report alongside this wrapper decision.

Step 5: check claim dependencies

A claim is not independently releasable merely because its own citation passed. Record dependencies:

claim-limit ----> v0001
claim-deadline --> v0002
claim-exception -> v0003
claim-eligibility -> claim-limit + claim-deadline + claim-exception + user facts

Block or review a dependent claim when any mandatory input is unavailable. For synthesis, all referenced Ethos check IDs must be reusable before the synthesis is even reviewable under the documented contract.

Common dependency traps include:

• conclusion remains after its qualifying premise is removed;
• amount remains without unit or time period;
• prohibition remains without its exception;
• recommendation remains after contraindication fails;
• percentage remains after one input becomes stale;
• pronoun or connective refers to a removed sentence.

Step 6: decide whether the partial subset is useful

A partial answer should be released only when:

• at least one relevant source fact is reusable;
• released claims stand independently;
• omitted claims do not reverse or materially qualify them;
• no stale fingerprint invalidates the evidence;
• the user can understand what remains unanswered;
• the domain and risk tier allow partial disclosure;
• review is not mandatory for the use case.

If these conditions fail, abstain: “The available verified sources do not support a complete answer.”

Step 7: write transparent user-facing copy

Good partial-answer copy contains:

1. the verified subset;
2. a clear partial-status disclosure;
3. the unresolved issue;
4. the next safe action.

Example:

The policy states that the reimbursement limit is $1,200. I could not verify the submission deadline or the exception that may apply to your request, so I cannot determine eligibility from the available sources. Please review the cited policy section or route the request to Finance.

Avoid:

Your request is probably eligible. Some details could not be verified.

The second version preserves the unsupported conclusion and hides material uncertainty.

Step 8: handle capability-limited evidence

Capability limitation means the source cannot prove the requested evidence at the required fidelity. Examples:

• flattened text cannot verify a table cell;
• no fingerprint prevents exact version binding;
• unknown coordinates prevent source-region proof;
• no stable spans prevent precise offsets.

Do not convert capability-limited to false or true. Show the limitation, use stronger parsing, inspect the original source, or route to review.

Step 9: handle calculations and synthesis

For calculations:

• require every input check to be reusable;
• verify units, periods, signs, and scope;
• recompute in versioned code;
• record rounding and tolerance;
• invalidate the result if any source fingerprint changes.

For synthesis:

• require all cited facts;
• evaluate semantic support and completeness;
• check conflicts and exceptions;
• review according to domain risk;
• avoid presenting inference as a direct quotation.

Partial grounding is especially dangerous when the omitted claim is the bridge between facts and conclusion.

Step 10: retain the complete audit trail

Store or reference:

• original question and generated draft;
• atomic claims and evidence mappings;
• canonical Ethos verification report;
• derived proof summary;
• relevance and claim-type labels;
• calculation artifacts;
• app release decision;
• final partial answer;
• reviewer actions and overrides;
• correction or regeneration events.

The audit-ready AI answer guide provides a complete envelope design.

Test partial-answer behavior

Include order and phrasing tests. A technically correct disclosure at the bottom may not offset a misleading unsupported headline.

Metrics for partial verification

Track:

• partially verified request rate;
• percentage of claims released, reviewed, and blocked;
• stale-report invalidation rate;
• capability-limited rate;
• user escalation rate;
• reviewer override rate;
• false release and false abstention rates;
• corrections after partial release;
• partial answers that users misinterpreted as complete.

Segment by claim type, source, parser, and domain. A high partial rate may indicate retrieval omissions, poor claim decomposition, weak adapters, or overly broad answers.

The definitive verdict

Partially verified AI answers should be handled as structured claim sets, not watered-down confidence scores. Reuse only checks that are grounded, semantically eligible under the contract, and not invalidated by source staleness. Then apply relevance, claim type, dependency, calculation, and domain-risk policy before generating user-facing text.

Show the verified subset only when it stands independently and cannot mislead. Disclose what remains unresolved, route synthesis and capability limits to review, abstain when essential context is missing, and preserve the full canonical report. Partial verification is useful when it narrows truthfully; it is dangerous when it launders an incomplete conclusion.

Primary keyword: partially verified AI answers Optimized meta title: How to Handle Partially Verified AI Answers Optimized meta description: Release only reusable grounded claims, disclose limits, review synthesis, block stale evidence, and preserve reports for partially verified AI answers. Proposed URL slug: how-do-you-handle-partially-verified-ai-answers