Ethos vs DeepEval: Deterministic Evidence Checks vs LLM Evaluation

Direct answer

Ethos and DeepEval operate at different layers. Ethos deterministically verifies whether AI citations bind to trusted document evidence. DeepEval is a broad LLM evaluation framework for RAG, agents, conversations, safety, custom criteria, and CI testing.

Use Ethos for source fingerprints, page and element locators, quotations, values, table cells, regions, and capability limitations. Use DeepEval for faithfulness, answer relevance, contextual retrieval metrics, G-Eval criteria, multi-turn evaluation, datasets, and test orchestration. Integrate Ethos as a deterministic metric or assertion inside a DeepEval workflow rather than replacing one with the other.

DeepEval’s official RAG evaluation guide separates generator and retriever evaluation. Its metrics documentation explains that most predefined metrics use LLM-as-a-judge techniques.

Comparison at a glance

What DeepEval evaluates

DeepEval’s RAG guidance distinguishes generator and retriever components.

Generator-focused metrics include:

• Faithfulness: whether generated claims are supported by retrieved context;
• Answer Relevancy: whether the response addresses the input.

Retriever-focused metrics include:

• Contextual Relevancy: whether retrieved context is relevant;
• Contextual Precision: whether relevant context ranks appropriately;
• Contextual Recall: whether required information was retrieved.

DeepEval also supports G-Eval for custom natural-language criteria, agent metrics, safety metrics, conversational cases, tracing, and broader test orchestration.

What Ethos evaluates

Ethos checks caller-provided citations against a trusted GroundingSource. Its questions are exact:

• Does the source fingerprint match?
• Does the page, element, span, region, table, or cell exist?
• Does the quote or structured value match?
• Does the source expose the required evidence capability?
• Is the citation stale, mismatched, missing, unsupported, or grounded?

Ethos does not assign general answer-quality scores or judge conversational usefulness.

DeepEval Faithfulness versus Ethos grounding

The tools provide orthogonal evidence.

HallucinationMetric versus FaithfulnessMetric

Current DeepEval documentation advises using its Faithfulness metric for live RAG retrieval_context. The Hallucination metric compares output against curated context treated as reference material and calculates contradiction-oriented results.

This distinction does not make either metric a document-locator verifier. Both operate over context supplied to the test case rather than independently proving the displayed source page or fingerprint.

Where G-Eval fits

G-Eval lets developers define custom semantic criteria in natural language. It can evaluate:

• domain-specific faithfulness;
• completeness;
• tone or professionalism;
• safety criteria;
• relevance;
• synthesis quality.

Do not replace exact Ethos checks with G-Eval. Asking a judge whether a fingerprint “looks fresh” or whether an element ID “probably exists” adds cost and uncertainty to a deterministic proposition.

Use G-Eval after Ethos when a grounded claim needs semantic judgment.

Integrating Ethos as a deterministic metric

A custom DeepEval metric can call Ethos and convert its structured report into a score and reason.

from deepeval.metrics import BaseMetric
from ethos_pdf import EthosCli, proof_summary

class EthosGroundingMetric(BaseMetric):
    threshold = 1.0

    def __init__(self, source: str, citations: str, binary: str = "ethos"):
        self.source = source
        self.citations = citations
        self.ethos = EthosCli(binary=binary)
        self.score = 0.0
        self.reason = "Not measured"
        self.success = False

    def measure(self, test_case):
        report = self.ethos.verify(
            source=self.source,
            citations=self.citations,
            fail_on_ungrounded=False,
            output_format="json",
            timeout=30,
        )
        summary = proof_summary(report)
        self.score = 1.0 if summary["request_certified"] else 0.0
        self.success = self.score >= self.threshold
        self.reason = f"Ethos proof status: {summary['proof_status']}"
        return self.score

    async def a_measure(self, test_case):
        return self.measure(test_case)

    def is_successful(self):
        return self.success

    @property
    def __name__(self):
        return "Ethos Grounding"

Pin this adapter to the installed DeepEval and Ethos API versions, validate actual abstract-method requirements, and preserve the canonical Ethos report separately. A score of 1.0 is only citation grounding, not complete answer correctness.

Combined test case design

from deepeval import evaluate
from deepeval.metrics import AnswerRelevancyMetric, FaithfulnessMetric
from deepeval.test_case import LLMTestCase

test_case = LLMTestCase(
    input="What was Q3 2025 revenue?",
    actual_output="Q3 2025 revenue was $12.4 million.",
    retrieval_context=["Revenue grew to $12.4M in Q3 2025."],
)

evaluate(
    test_cases=[test_case],
    metrics=[
        EthosGroundingMetric(
            source="annual-report.ethos.json",
            citations="citations.json",
        ),
        FaithfulnessMetric(),
        AnswerRelevancyMetric(),
    ],
)

The Ethos metric evaluates source binding. Faithfulness evaluates semantic support against retrieved context. Answer relevance evaluates responsiveness.

CI/CD architecture

Fast deterministic lane

Run Ethos fixtures on every relevant pull request:

• grounded quote;
• wrong page;
• altered quote;
• missing target;
• stale fingerprint;
• capability-blocked table request.

Semantic regression lane

Run DeepEval over representative datasets for faithfulness, relevance, and retrieval metrics. Pin judge configuration and validate thresholds against human labels.

End-to-end release lane

Run the application, capture generated citations, invoke Ethos, compute DeepEval metrics, and apply release policy. Store privacy-safe reports and test-run references.

The RAG citation CI/CD guide provides a full test pyramid.

DeepEval strengths

• Broad metric ecosystem across RAG, agents, conversations, safety, and multimodal applications.
• Custom evaluation through G-Eval, DAG, and self-coded metrics.
• Test-case, dataset, and pytest-oriented workflows.
• Component-level retriever and generator evaluation.
• Multi-turn RAG metrics and tracing integration.
• Score reasoning and debugging for semantic metrics.

Ethos strengths

• Exact source and citation evidence checks.
• Document fingerprint freshness.
• Page, element, span, table-cell, and region locators.
• Explicit capability limitations.
• Local verification without a judge-model call.
• Canonical reports and inspectable evidence artifacts.
• Parser-neutral GroundingSource boundary.

Limitations and mitigation

Privacy, cost, and reproducibility

Ethos exact checks can run locally, avoiding judge tokens for source identity and evidence matching. DeepEval metrics may call configured judge models, creating cost, latency, privacy, and model-version considerations.

Run deterministic checks first. Send bounded verified evidence to semantic evaluation, use approved model providers or local judges, and avoid sending entire confidential documents when claim-level evidence is sufficient.

What neither replaces

Neither tool replaces:

• source authority and access governance;
• parser security isolation;
• complete claim extraction;
• deterministic arithmetic;
• domain-expert review for high-stakes synthesis;
• production monitoring and incident response.

Definitive verdict

Ethos vs DeepEval is a layer comparison, not a universal product contest. Ethos proves source-bound citation properties. DeepEval evaluates semantic, retrieval, conversational, agent, and custom quality criteria and orchestrates tests.

Use DocuShell Parse PDF for source-aware evidence, Ethos as a deterministic custom metric or assertion, DeepEval for broader evaluation, the Ethos release-gate guide for final policy, and the DocuShell hallucination index for wider model-risk context.

Primary keyword: Ethos vs DeepEval
Optimized meta title: Ethos vs DeepEval for RAG Testing
Optimized meta description: Compare Ethos citation grounding with DeepEval RAG metrics, G-Eval, pytest workflows, and LLM-as-a-judge evaluation.
Proposed URL slug: ethos-vs-deepeval-deterministic-evidence-vs-llm-evaluation