Where Does Ethos Fit in a Modern RAG Evaluation Stack?

Direct answer

Ethos fits in a modern RAG evaluation stack as the deterministic document-evidence verification layer. It runs after an application has citations or evidence references and before those claims are released as trusted answer text. It verifies source identity, freshness, locators, and evidence matching; other tools evaluate retrieval, semantic support, relevance, reasoning, and production behavior.

Ethos does not replace RAGAS, DeepEval, Promptfoo, an observability platform, or human review. It gives those systems a source-bound result they generally do not produce by themselves: whether the submitted citation actually binds to evidence in a trusted document representation.

Read what Ethos is for the product boundary and the RAG metric comparison for terminology.

The modern RAG evaluation stack

Evaluation should mirror the actual pipeline:

source governance
  -> parsing and provenance
  -> indexing and retrieval
  -> answer and citation generation
  -> Ethos citation grounding
  -> semantic support and relevance
  -> calculation and policy checks
  -> release, review, partial answer, or abstention
  -> monitoring and audit

Each stage asks a different question. Combining every stage into one score makes failures harder to diagnose and unsafe to automate.

Layer-by-layer ownership

What Ethos contributes that semantic metrics do not

A faithfulness metric may decide that a claim is supported by retrieved context. It does not necessarily confirm that:

• the displayed page is correct;
• the evidence ID exists in the canonical source;
• the source fingerprint matches the indexed version;
• the quoted text is present at the submitted locator;
• a table value came from the named row and column;
• the parser exposed enough structure to make the claim;
• a source crop can be inspected.

Ethos makes those exact evidence properties explicit.

How Ethos complements RAGAS

RAGAS-style faithfulness evaluates whether answer claims are supported by context. That is a semantic comparison and can be useful even when answers paraphrase evidence.

Ethos asks whether citation claims bind to source evidence. A complete stack can use both:

1. Ethos confirms that the cited source evidence exists and is fresh.
2. RAGAS or another semantic evaluator tests whether the claim follows from the evidence.
3. The application decides whether direct facts, synthesis, or partial answers can be released.

If RAGAS passes while Ethos finds a wrong page, the semantic relationship may be sound but the displayed citation is defective. If Ethos passes while faithfulness fails, the passage exists but does not justify the claim.

How Ethos complements DeepEval

DeepEval supports broader test cases and model-based metrics. Ethos can be wrapped as a deterministic custom assertion:

• invoke the CLI or Python wrapper;
• parse verification_report.json;
• assert full grounding or expected negative status;
• retain the report as test evidence;
• run G-Eval or hallucination metrics separately for semantic quality.

This preserves a clean distinction: model-based metrics evaluate judgment-heavy criteria; Ethos evaluates evidence binding.

How Ethos fits with Promptfoo

Promptfoo and similar harnesses orchestrate providers, cases, assertions, and reports. They can call a provider or script that runs the product workflow and Ethos verification.

A safe assertion envelope can expose:

{
  "fixture_id": "policy-answer-001",
  "source_fingerprint": "sha256:pinned-source",
  "proof_status": "verified",
  "verified_claim_count": 3,
  "unverified_claim_count": 0,
  "release_status": "show_final",
  "audit_ref": "artifact://verification/report-001"
}

Avoid exporting raw confidential answers or source content when compact result metadata is enough. Fixture mode proves the evaluator contract; live mode is needed for end-to-end product health.

How Ethos fits with observability platforms

Tracing platforms can store:

• source-set and index versions;
• retrieved evidence IDs;
• model and prompt versions;
• Ethos report references;
• proof summary;
• semantic evaluation results;
• release actions and reason codes.

Ethos is not an observability product. Its report becomes high-value evidence inside a trace or audit record.

Online versus offline evaluation

The same verifier can support all modes, but latency, privacy, and retention policies differ.

Recommended evaluation sequence

1. Validate source eligibility

Choose approved, authorized, time-valid source versions. Ethos does not decide organizational authority.

2. Validate extraction

Test parser output, coordinates, tables, and capabilities. A verifier cannot recover evidence a parser lost.

3. Measure retrieval

Check required evidence recall and irrelevant-context precision before judging answers.

4. Generate structured claims and citations

Use stable claim IDs and evidence references. Treat model output as untrusted requests.

5. Run Ethos

Verify source fingerprints, locators, evidence, and capability boundaries. Preserve the canonical report.

6. Evaluate semantics

Use a calibrated judge, NLI model, rules, or reviewers for claim support and relevance.

7. Recompute structured conclusions

Use code for arithmetic, dates, units, and thresholds.

8. Apply release policy

Show grounded relevant facts, review synthesis, disclose partial answers, and block unsupported claims.

9. Record and monitor

Attach versioned results to traces and audit records. Monitor failure classes separately.

A decision matrix for tool selection

Tool overlap is normal. The important design choice is which result controls which decision.

Failure combinations and remediation

This matrix demonstrates why one aggregate hallucination score is not enough.

Production boundaries

Ethos is currently a public-beta evaluation surface. The repository documents native JSON verification, an OpenDataLoader-style adapter, stale-fingerprint handling, capability reporting, and narrow born-digital PDF support.

Do not position it as:

• a hosted evaluation service;
• a universal OCR platform;
• a general semantic correctness judge;
• a production-readiness guarantee;
• a public benchmark winner.

Use exact, source-bound product language. The strongest differentiation is deterministic evidence verification, not exaggerated breadth.

Definitive verdict

Ethos belongs between citations and release policy. It converts model-supplied evidence references into structured, source-bound verification results. Retrieval metrics, RAGAS, DeepEval, Promptfoo, observability platforms, application rules, and human review remain complementary parts of the stack.

Use DocuShell Parse PDF for source-aware document preparation, Ethos for citation grounding, the LLM-as-a-judge reliability guide for semantic evaluation design, and the DocuShell hallucination index for broader workflow-risk context.

Primary keyword: Ethos RAG evaluation stack
Optimized meta title: Where Ethos Fits in the RAG Evaluation Stack
Optimized meta description: Place Ethos alongside RAGAS, DeepEval, Promptfoo, LLM judges, observability, source governance, and release policy.
Proposed URL slug: where-does-ethos-fit-in-modern-rag-evaluation-stack