Where Does Ethos Fit in a Modern RAG Evaluation Stack?

Direct answer

Ethos fits in a modern RAG evaluation stack as the deterministic document-evidence verification layer. It runs after an application has citations or evidence references and before those claims are released as trusted answer text. It verifies source identity, freshness, locators, and evidence matching; other tools evaluate retrieval, semantic support, relevance, reasoning, and production behavior.

Ethos does not replace RAGAS, DeepEval, Promptfoo, an observability platform, or human review. It gives those systems a source-bound result they generally do not produce by themselves: whether the submitted citation actually binds to evidence in a trusted document representation.

Read what Ethos is for the product boundary and the RAG metric comparison for terminology.

The modern RAG evaluation stack

Evaluation should mirror the actual pipeline:

source governance
  -> parsing and provenance
  -> indexing and retrieval
  -> answer and citation generation
  -> Ethos citation grounding
  -> semantic support and relevance
  -> calculation and policy checks
  -> release, review, partial answer, or abstention
  -> monitoring and audit

Each stage asks a different question. Combining every stage into one score makes failures harder to diagnose and unsafe to automate.

Layer-by-layer ownership

Layer	Primary question	Typical methods	Ethos role
Source governance	Is this source approved, current, and authorized?	Catalogs, metadata, policy engines	Consumes the trusted source selected upstream
Parser quality	Was document structure extracted correctly?	Fixtures, visual QA, parser benchmarks	Uses exposed evidence and capabilities
Retrieval quality	Was necessary context found?	Recall, precision, ranking tests	Not the retrieval evaluator
Generation quality	Is output relevant, complete, and useful?	LLM judges, human labels, task metrics	Not the general answer judge
Citation grounding	Does each reference bind to source evidence?	Deterministic evidence checks	Core Ethos responsibility
Reasoning and calculation	Do conclusions follow and computations work?	NLI, judges, code, experts	Upstream evidence input, not final decision
Release policy	What can the user see?	Application rules and review workflows	Supplies grounding statuses and limitations
Observability	What happened operationally?	Traces, metrics, logs	Reports can be attached to traces

What Ethos contributes that semantic metrics do not

A faithfulness metric may decide that a claim is supported by retrieved context. It does not necessarily confirm that:

• the displayed page is correct;
• the evidence ID exists in the canonical source;
• the source fingerprint matches the indexed version;
• the quoted text is present at the submitted locator;
• a table value came from the named row and column;
• the parser exposed enough structure to make the claim;
• a source crop can be inspected.

Ethos makes those exact evidence properties explicit.

How Ethos complements RAGAS

RAGAS-style faithfulness evaluates whether answer claims are supported by context. That is a semantic comparison and can be useful even when answers paraphrase evidence.

Ethos asks whether citation claims bind to source evidence. A complete stack can use both:

1. Ethos confirms that the cited source evidence exists and is fresh.
2. RAGAS or another semantic evaluator tests whether the claim follows from the evidence.
3. The application decides whether direct facts, synthesis, or partial answers can be released.

If RAGAS passes while Ethos finds a wrong page, the semantic relationship may be sound but the displayed citation is defective. If Ethos passes while faithfulness fails, the passage exists but does not justify the claim.

How Ethos complements DeepEval

DeepEval supports broader test cases and model-based metrics. Ethos can be wrapped as a deterministic custom assertion:

• invoke the CLI or Python wrapper;
• parse verification_report.json;
• assert full grounding or expected negative status;
• retain the report as test evidence;
• run G-Eval or hallucination metrics separately for semantic quality.

This preserves a clean distinction: model-based metrics evaluate judgment-heavy criteria; Ethos evaluates evidence binding.

How Ethos fits with Promptfoo

Promptfoo and similar harnesses orchestrate providers, cases, assertions, and reports. They can call a provider or script that runs the product workflow and Ethos verification.

A safe assertion envelope can expose:

{
  "fixture_id": "policy-answer-001",
  "source_fingerprint": "sha256:pinned-source",
  "proof_status": "verified",
  "verified_claim_count": 3,
  "unverified_claim_count": 0,
  "release_status": "show_final",
  "audit_ref": "artifact://verification/report-001"
}

Avoid exporting raw confidential answers or source content when compact result metadata is enough. Fixture mode proves the evaluator contract; live mode is needed for end-to-end product health.

How Ethos fits with observability platforms

Tracing platforms can store:

• source-set and index versions;
• retrieved evidence IDs;
• model and prompt versions;
• Ethos report references;
• proof summary;
• semantic evaluation results;
• release actions and reason codes.

Ethos is not an observability product. Its report becomes high-value evidence inside a trace or audit record.

Online versus offline evaluation

Evaluation mode	Goal	Ethos usage
Pull-request fixtures	Prevent deterministic contract regressions	Assert exact grounded and negative statuses
Nightly live tests	Detect retrieval or model drift	Verify generated citations from controlled queries
Pre-release acceptance	Validate full product workflow	Require approved release decisions and reports
Online request gate	Prevent unsafe answer release	Verify citations before user-visible output
Incident revalidation	Reconstruct disputed answers	Re-run pinned claims against recorded source versions
Source-change audit	Detect stale citations	Reverify receipts after fingerprint changes

The same verifier can support all modes, but latency, privacy, and retention policies differ.

Recommended evaluation sequence

1. Validate source eligibility

Choose approved, authorized, time-valid source versions. Ethos does not decide organizational authority.

2. Validate extraction

Test parser output, coordinates, tables, and capabilities. A verifier cannot recover evidence a parser lost.

3. Measure retrieval

Check required evidence recall and irrelevant-context precision before judging answers.

4. Generate structured claims and citations

Use stable claim IDs and evidence references. Treat model output as untrusted requests.

5. Run Ethos

Verify source fingerprints, locators, evidence, and capability boundaries. Preserve the canonical report.

6. Evaluate semantics

Use a calibrated judge, NLI model, rules, or reviewers for claim support and relevance.

7. Recompute structured conclusions

Use code for arithmetic, dates, units, and thresholds.

8. Apply release policy

Show grounded relevant facts, review synthesis, disclose partial answers, and block unsupported claims.

9. Record and monitor

Attach versioned results to traces and audit records. Monitor failure classes separately.

A decision matrix for tool selection

Need	Best first tool category
Find missing retrieval evidence	Retrieval evaluation
Judge paraphrased support	Faithfulness or NLI evaluator
Verify exact page, quote, value, or fingerprint	Ethos
Compare answer relevance and style	LLM judge
Recompute financial or date result	Deterministic application code
Inspect latency and failures	Observability platform
Decide approved source revision	Governance catalog and policy
Resolve ambiguous high-stakes synthesis	Domain reviewer

Tool overlap is normal. The important design choice is which result controls which decision.

Failure combinations and remediation

Retrieval	Ethos	Semantic support	Diagnosis
Pass	Pass	Pass	Candidate for release policy
Pass	Fail	Pass	Citation or source-binding defect
Pass	Pass	Fail	Real evidence does not support claim
Fail	Pass	Pass	Test setup or evidence leakage requires review
Fail	Fail	Fail	Retrieval or source pipeline failure
Pass	Capability limited	Uncertain	Obtain stronger source or review

This matrix demonstrates why one aggregate hallucination score is not enough.

Production boundaries

Ethos is currently a public-beta evaluation surface. The repository documents native JSON verification, an OpenDataLoader-style adapter, stale-fingerprint handling, capability reporting, and narrow born-digital PDF support.

Do not position it as:

• a hosted evaluation service;
• a universal OCR platform;
• a general semantic correctness judge;
• a production-readiness guarantee;
• a public benchmark winner.

Use exact, source-bound product language. The strongest differentiation is deterministic evidence verification, not exaggerated breadth.

Definitive verdict

Ethos belongs between citations and release policy. It converts model-supplied evidence references into structured, source-bound verification results. Retrieval metrics, RAGAS, DeepEval, Promptfoo, observability platforms, application rules, and human review remain complementary parts of the stack.

Use DocuShell Parse PDF for source-aware document preparation, Ethos for citation grounding, the LLM-as-a-judge reliability guide for semantic evaluation design, and the DocuShell hallucination index for broader workflow-risk context.

Primary keyword: Ethos RAG evaluation stack
Optimized meta title: Where Ethos Fits in the RAG Evaluation Stack
Optimized meta description: Place Ethos alongside RAGAS, DeepEval, Promptfoo, LLM judges, observability, source governance, and release policy.
Proposed URL slug: where-does-ethos-fit-in-modern-rag-evaluation-stack