Can You Use Ethos with DeepEval for RAG Evaluation?

Direct answer

Yes. Ethos can run with DeepEval as a deterministic custom metric that checks document citations, while DeepEval evaluates semantic qualities such as faithfulness, answer relevance, and retrieval performance. The combination is stronger than treating either system as a complete RAG quality oracle.

The boundary matters. Ethos answers exact evidence questions: does the source fingerprint match, does the cited locator exist, and does the quoted or structured evidence match the source? DeepEval answers broader evaluation questions: is the answer relevant, is it faithful to the supplied retrieval context, and did the retriever return useful evidence? The official DeepEval custom-metric interface explicitly supports metrics that do not call an LLM, which makes it the appropriate extension point for Ethos.

This guide shows the architecture, metric adapter, test data, release policy, and failure analysis needed to make the integration useful in a real RAG pipeline.

Why combine Ethos and DeepEval?

A RAG answer has several independently breakable contracts. Retrieval can miss the governing paragraph. Generation can distort retrieved text. A citation renderer can attach the wrong page. The underlying document can change after indexing. A judge model can approve a fluent answer even though the displayed source link is stale.

No single score identifies all of those failures.

DeepEval’s RAG evaluation guidance separates generator evaluation from retriever evaluation. Ethos adds a third layer: evidence delivery. It verifies that the citation presented to a user actually resolves against the trusted document representation.

A faithful answer can still have a broken citation. A grounded citation can still accompany an overbroad claim. Treat semantic support and evidence binding as separate gates.

The recommended integration architecture

The clean design has four stages:

1. Your RAG application returns an answer, retrieval context, and structured citation references.
2. Ethos verifies those references against native Ethos JSON or a supported foreign-parser source through GroundingSource.
3. A DeepEval custom metric converts the verification outcome into a test score without discarding the canonical Ethos report.
4. DeepEval runs semantic and retrieval metrics beside the deterministic metric, after which CI applies an explicit release policy.

question
   |
   v
RAG application ----> answer + retrieval_context + citations
                           |                    |
                           |                    v
                           |              Ethos verify
                           |                    |
                           v                    v
                 DeepEval semantic       deterministic
                      metrics             proof report
                           \                    /
                            \                  /
                             v                v
                         release policy + retained artifacts

Ethos’s Python package is intentionally thin: EthosCli invokes a caller-provided local ethos CLI and returns structured JSON. The wheel does not bundle the CLI or PDFium. JSON verification does not require parsing, but any PDFium-backed parsing or crop path needs caller-provided PDFium configuration.

Define the test-case contract first

Do not make a metric guess where citations live. Define a case manifest that binds one DeepEval case to its exact verification inputs.

from dataclasses import dataclass
from pathlib import Path
from typing import Optional


@dataclass(frozen=True)
class GroundingCase:
    case_id: str
    source: Path
    citations: Path
    report_output: Path
    grounding_adapter: Optional[str] = None


CASES = {
    "q3-revenue": GroundingCase(
        case_id="q3-revenue",
        source=Path("fixtures/q3-report.ethos.json"),
        citations=Path("runs/q3-revenue/citations.json"),
        report_output=Path("artifacts/q3-revenue/ethos-report.json"),
    ),
}

The manifest prevents a subtle evaluation error: verifying a generated answer against the wrong fixture while the semantic evaluator receives the expected retrieval context. Store source and citation files immutably for each run, and include a stable case ID in application logs.

Build an Ethos custom metric for DeepEval

DeepEval custom single-turn metrics inherit from BaseMetric and implement measurement and success behavior. The adapter below receives a resolver so each LLMTestCase can select its own source and citation files. It writes the complete report before returning a binary grounding score.

import json
from pathlib import Path
from typing import Callable

from deepeval.metrics import BaseMetric
from deepeval.test_case import LLMTestCase
from ethos_pdf import EthosCli, proof_summary


class EthosGroundingMetric(BaseMetric):
    """Fail-closed DeepEval metric backed by an Ethos verification report."""

    def __init__(
        self,
        resolve_case: Callable[[LLMTestCase], GroundingCase],
        *,
        ethos_binary: str = "ethos",
        timeout_seconds: float = 30.0,
    ) -> None:
        self.threshold = 1.0
        self.resolve_case = resolve_case
        self.ethos = EthosCli(binary=ethos_binary)
        self.timeout_seconds = timeout_seconds
        self.score = 0.0
        self.reason = "Ethos has not run"
        self.success = False
        self.error = None

    def measure(self, test_case: LLMTestCase) -> float:
        case = self.resolve_case(test_case)
        report = self.ethos.verify(
            source=case.source,
            citations=case.citations,
            grounding=case.grounding_adapter,
            fail_on_ungrounded=False,
            output_format="json",
            timeout=self.timeout_seconds,
        )
        summary = proof_summary(report)

        case.report_output.parent.mkdir(parents=True, exist_ok=True)
        case.report_output.write_text(
            json.dumps(report, indent=2, sort_keys=True) + "\n",
            encoding="utf-8",
        )

        self.score = 1.0 if summary["request_certified"] else 0.0
        self.success = self.score >= self.threshold
        self.reason = (
            f"proof_status={summary['proof_status']}; "
            f"needs_review={summary['needs_review_check_ids']}; "
            f"limitations={summary['proof_limitations']}"
        )
        return self.score

    async def a_measure(self, test_case: LLMTestCase) -> float:
        return self.measure(test_case)

    def is_successful(self) -> bool:
        return self.success

    @property
    def __name__(self) -> str:
        return "Ethos Citation Grounding"

Pin your DeepEval and ethos-pdf versions, then run this adapter against the installed APIs before copying it into CI. Custom metric interfaces can evolve. More importantly, keep the report file: a 1.0 score is convenient for DeepEval, but the canonical Ethos report contains the check-level evidence needed to diagnose a failure.

Why use a binary score?

A release gate should not average away a required citation failure. Suppose nine quote checks pass but one source fingerprint is stale. A ratio of 0.9 may look healthy even though every citation could refer to an obsolete document version. Binary scoring preserves fail-closed semantics: the request is certified or it is not.

For dashboards, you may separately calculate grounded-check coverage, limitation counts, or partially verified rates. Label those as operational metrics, not proof that an answer is correct.

Run Ethos beside DeepEval RAG metrics

Use an explicit case ID in the test input or metadata so the resolver cannot select the wrong evidence bundle.

from deepeval import evaluate
from deepeval.metrics import AnswerRelevancyMetric, FaithfulnessMetric
from deepeval.test_case import LLMTestCase


def resolve_grounding_case(test_case: LLMTestCase) -> GroundingCase:
    case_id, separator, _ = test_case.input.partition("::")
    if separator != "::" or case_id not in CASES:
        raise ValueError(f"Unknown grounding case in input: {test_case.input!r}")
    return CASES[case_id]


test_case = LLMTestCase(
    input="q3-revenue::What was revenue in Q3 2025?",
    actual_output="Revenue was $12.4 million in Q3 2025 [p. 18].",
    retrieval_context=[
        "For the quarter ended September 30, 2025, revenue was $12.4 million."
    ],
)

evaluate(
    test_cases=[test_case],
    metrics=[
        EthosGroundingMetric(resolve_grounding_case),
        FaithfulnessMetric(threshold=0.8),
        AnswerRelevancyMetric(threshold=0.8),
    ],
)

The exact thresholds are not universal. Calibrate semantic thresholds against representative, expert-labeled examples from your domain. Judge models, prompts, and model versions can change results, so record that configuration with each run.

Interpret combined results correctly

The last row is important. Ethos can verify that cited input values exist; it does not certify a derived calculation merely because the operands are grounded. DeepEval may or may not catch the error depending on context and metric behavior. Recompute financial, dosage, date, and policy-rule outputs in deterministic code.

Design the CI pipeline in three lanes

1. Deterministic pull-request lane

Run small Ethos fixtures for grounded, wrong-page, altered-quote, missing-target, stale-fingerprint, and unsupported-capability cases. These tests are fast, reproducible, and should fail on report-shape drift.

2. Semantic regression lane

Run DeepEval on a curated dataset that covers typical questions, adversarial prompts, missing context, conflicting documents, and abstention. Keep judge configuration pinned and compare changes by metric and segment, not only global average.

3. End-to-end release lane

Execute the actual RAG application, capture its citations, run both evaluators, and retain:

• source and citation fingerprints;
• the canonical Ethos verification report;
• DeepEval metric scores and reasons;
• application, prompt, retriever, and model versions;
• the release decision and policy version.

The RAG citation CI/CD guide explains the broader test pyramid, while the Ethos release-gate guide covers fail-closed policy design.

Production pitfalls to avoid

Treating retrieval context as proof of the displayed citation

DeepEval receives the context you provide. If your application accidentally labels a chunk as page 18 when it came from page 19, a semantic metric can still score the text highly. Verify the user-facing reference against the original source representation.

Discarding partially verified evidence

proof_summary() can distinguish verified, partially_verified, and unverified, but it is derived from the canonical report. Preserve check-level failures and capability limits. A partially verified answer may support a review workflow; it should not silently become fully certified.

Conflating infrastructure errors with negative evidence

A missing Ethos binary, timeout, malformed source, or unreadable citations file is not an ordinary ungrounded result. Fail the evaluation as an infrastructure error, retain stderr safely, and do not convert the exception into a reassuring zero-only metric with no explanation.

Running a judge on sensitive documents unnecessarily

Ethos verification is local and does not require another LLM call. DeepEval semantic metrics may invoke a configured evaluator model. Apply your data-handling policy to retrieval context before sending it to any external service, or use an approved local evaluator path.

Claiming that all green metrics prove truth

Green results mean that specified checks passed against specified inputs. They do not prove that the source document itself is correct, that omitted evidence does not exist, or that the answer is safe for every downstream decision.

The definitive verdict

Use Ethos with DeepEval when you need both source-bound citation proof and broad RAG quality evaluation. Implement Ethos through DeepEval’s custom metric boundary, return a fail-closed score for release policy, and retain the full verification report for audit and debugging. Then run DeepEval faithfulness, answer relevance, and contextual metrics for the semantic and retrieval dimensions Ethos intentionally does not judge.

Start with one representative document and six adversarial citation cases. Once the adapter reliably separates wrong locators, stale fingerprints, unsupported capabilities, semantic overreach, and irrelevant answers, expand the dataset by document type and risk tier. That creates a test system with interpretable failures instead of a single attractive but ambiguous score.

Primary keyword: Ethos DeepEval integration Optimized meta title: Use Ethos with DeepEval for RAG Evaluation Optimized meta description: Integrate Ethos with DeepEval to test deterministic citations, RAG faithfulness, relevance, and retrieval quality with audit-ready CI gates. Proposed URL slug: can-you-use-ethos-with-deepeval-for-rag-evaluation