Do Ethos and RAGAS Compete, or Should You Use Them Together?

Direct answer

Ethos and RAGAS compete for some evaluation attention and budget, but they are not direct substitutes. Ethos verifies whether structured citations bind to trusted document evidence. RAGAS measures semantic, retrieval, and application-quality dimensions such as faithfulness, context precision, and context recall.

Use them together when your RAG product must answer two independent questions:

1. Is the generated claim supported by the retrieved context?
2. Does the citation shown to the user actually resolve to the correct, current source evidence?

RAGAS is designed around systematic LLM-application evaluation and experiments. Its official Faithfulness metric scores whether response claims are supported by retrieved context. Ethos operates lower in the evidence chain: it checks source identity, locators, quotes, values, and declared source capabilities.

That distinction is the basis for a sound architecture and an honest buying decision.

Where Ethos and RAGAS genuinely overlap

Both products can appear in the same conversations:

• reducing unsupported RAG answers;
• testing document-question-answering systems;
• adding quality gates to CI/CD;
• diagnosing retrieval and generation failures;
• producing structured evaluation outputs;
• increasing confidence before release.

This creates market overlap. A team with a limited evaluation budget may ask which project to implement first. A platform owner may want one dashboard rather than two. Search pages may place both under “RAG evaluation tools.”

But category overlap does not mean mechanism equivalence.

Ethos therefore competes with RAGAS only if the requirement is described too broadly as “detect hallucinations.” Once the requirement is decomposed into exact evidence binding, semantic support, retrieval quality, and application policy, the tools become complementary.

What RAGAS proves—and what it does not

RAGAS Faithfulness breaks a response into claims and evaluates whether those claims can be inferred from retrieved context. Its conceptual score is:

faithfulness = supported response claims / total response claims

That is valuable because support is often semantic rather than lexical. A response can paraphrase a passage accurately without repeating the same words.

RAGAS also provides retrieval-focused measurements. Context Recall evaluates whether needed reference information appears in retrieved contexts. Context precision variants evaluate whether relevant chunks are retrieved and ranked effectively. ID-based and non-LLM variants can be used for some retrieval contracts.

However, a high faithfulness score is conditional on the context supplied to the evaluator. It does not independently prove that:

• the chunk came from the PDF named in the user interface;
• the displayed page number is correct;
• the source has not changed since indexing;
• the quotation matches the identified element;
• a table value came from the claimed row and column;
• the parser preserved layout accurately;
• the source document itself is true or authoritative.

Those are different contracts.

What Ethos proves—and what it does not

Ethos verifies caller-provided citation claims against a trusted GroundingSource. Native Ethos JSON is one source type, and supported foreign-parser outputs can enter through adapters. The verifier can check evidence such as:

• source fingerprints;
• pages and document elements;
• quotations and values;
• spans, regions, and coordinates;
• table cells when the source exposes that capability;
• stale, missing, mismatched, unsupported, or capability-limited evidence.

Ethos returns structured check results and a canonical report. Its proof_summary() helper can derive verified, partially_verified, or unverified product-facing status while preserving the canonical report as the source of detail.

Ethos does not claim that a matching quotation semantically entails an entire answer. It does not decide whether retrieved evidence is complete, whether the response addresses the question, or whether a conclusion is legally or clinically sound.

Ethos proves the evidence binding it actually checked. RAGAS estimates the semantic or retrieval quality defined by the selected metric. Neither should be relabeled as universal truth verification.

Why the tools produce different results

Consider a policy answer: “Contractors may export customer records with manager approval,” citing page 14.

The disagreement is useful. It localizes the layer that failed instead of collapsing every problem into “hallucination.”

When Ethos and RAGAS are direct alternatives

There are narrow cases where a team may reasonably choose one.

Choose RAGAS first for broad experimentation

RAGAS is the better first tool when the immediate objective is to compare:

• chunking strategies;
• retriever configurations;
• prompts or model versions;
• answer relevance;
• context precision and recall;
• semantic faithfulness across a labeled dataset.

The official RAGAS documentation positions the library around experiments, metrics, dataset management, and iterative application improvement. This fits a team still discovering which RAG configuration works best.

Choose Ethos first for citation integrity

Ethos is the better first tool when the product already generates source citations but incidents involve:

• wrong PDF pages;
• stale policy versions;
• quotations that cannot be found;
• invalid element or table-cell references;
• missing source capabilities;
• audit teams unable to reproduce evidence.

These are exact document evidence failures. Adding another semantic judge does not fix the broken provenance chain.

Choose neither as the only safety control

For high-impact decisions, add domain rules, deterministic calculations, access controls, source governance, human review, and incident monitoring. Evaluation libraries do not replace application risk management.

Recommended combined architecture

The strongest design keeps one canonical case identifier across ingestion, retrieval, generation, Ethos, and RAGAS.

governed document + expected fingerprint
        |
        v
source-aware parsing and indexing
        |
        v
retrieval ----> retrieved contexts + stable source references
        |
        v
generation ---> answer + claim-level citations
        |                         |
        |                         v
        |                   Ethos verification
        |                         |
        v                         v
RAGAS semantic/retrieval metrics + canonical evidence report
        \                         /
         \                       /
          v                     v
             application release policy

Run Ethos before or alongside RAGAS. If the source fingerprint is stale or citations are malformed, preserve that failure even if a semantic score is high. RAGAS can still be useful diagnostically, but it should not override a mandatory evidence contract.

A combined Python evaluation workflow

The example below runs Ethos locally, retains its canonical JSON report, and scores semantic faithfulness with the current RAGAS collections API. Pin installed versions because evaluator APIs and model behavior evolve.

import asyncio
import json
import os
from pathlib import Path
from typing import Any, Dict, List

from openai import AsyncOpenAI
from ragas.llms import llm_factory
from ragas.metrics.collections import Faithfulness

from ethos_pdf import EthosCli, proof_summary


async def evaluate_document_rag_case(
    *,
    question: str,
    answer: str,
    retrieved_contexts: List[str],
    source_path: Path,
    citations_path: Path,
    artifact_dir: Path,
) -> Dict[str, Any]:
    artifact_dir.mkdir(parents=True, exist_ok=True)

    ethos = EthosCli(binary=os.environ.get("ETHOS_CLI_PATH", "ethos"))
    report = ethos.verify(
        source=source_path,
        citations=citations_path,
        fail_on_ungrounded=False,
        output_format="json",
        timeout=30,
    )
    summary = proof_summary(report)
    (artifact_dir / "ethos-report.json").write_text(
        json.dumps(report, indent=2, sort_keys=True) + "\n",
        encoding="utf-8",
    )

    evaluator_model = os.environ.get("RAGAS_EVALUATOR_MODEL", "gpt-4o-mini")
    evaluator_llm = llm_factory(evaluator_model, client=AsyncOpenAI())
    faithfulness = Faithfulness(llm=evaluator_llm)
    semantic_result = await faithfulness.ascore(
        user_input=question,
        response=answer,
        retrieved_contexts=retrieved_contexts,
    )

    result = {
        "ethos": summary,
        "ragas": {
            "faithfulness": semantic_result.value,
            "reason": semantic_result.reason,
            "evaluator_model": evaluator_model,
        },
    }
    (artifact_dir / "combined-result.json").write_text(
        json.dumps(result, indent=2, sort_keys=True) + "\n",
        encoding="utf-8",
    )
    return result


if __name__ == "__main__":
    combined = asyncio.run(
        evaluate_document_rag_case(
            question="What was Q3 2025 revenue?",
            answer="Q3 2025 revenue was $12.4 million [p. 18].",
            retrieved_contexts=[
                "For the quarter ended September 30, 2025, revenue was $12.4 million."
            ],
            source_path=Path("fixtures/q3-report.ethos.json"),
            citations_path=Path("runs/q3-revenue/citations.json"),
            artifact_dir=Path("artifacts/q3-revenue"),
        )
    )
    print(json.dumps(combined, indent=2, sort_keys=True))

Do not expose unredacted evaluator reasons or source excerpts in public CI logs. The semantic evaluator may send retrieved context to the configured model provider; apply your organization’s data-handling rules.

Release policy: combine gates without averaging them away

A safe policy treats some dimensions as mandatory and others as calibrated thresholds.

def release_allowed(result: dict, minimum_faithfulness: float) -> bool:
    citation_gate = result["ethos"]["request_certified"] is True
    semantic_gate = result["ragas"]["faithfulness"] >= minimum_faithfulness
    return citation_gate and semantic_gate

Do not compute (citation_grounding + faithfulness) / 2 and accept a high average. A score cannot compensate for a stale source fingerprint or a missing required citation. Likewise, a valid citation cannot compensate for a claim that the evidence does not support.

Calibrate the RAGAS threshold using expert-labeled cases from the target domain. Measure false accepts and false rejects by document type, question type, answer length, and risk tier. Record the evaluator model and metric configuration with every run.

Procurement and marketplace positioning

From a market perspective, RAGAS occupies the broad LLM-application evaluation and experimentation category. Ethos occupies the narrower document evidence, citation verification, and auditability layer. Ethos can therefore integrate into the ecosystems around RAGAS, DeepEval, Promptfoo, observability platforms, document parsers, and RAG frameworks rather than trying to displace all of them.

The practical “marketplace” for Ethos is not limited to a single app store. It includes:

• Python, npm, and Rust package ecosystems;
• evaluator integrations and example repositories;
• RAG framework adapters;
• document parser compatibility layers;
• CI actions and release-gate templates;
• observability and experiment-tracking integrations;
• enterprise document QA and audit workflows.

Ethos is strongest when listed as a deterministic evidence integration inside existing evaluation workflows. That positioning is clearer than claiming a broader, undifferentiated hallucination score.

Adoption checklist

Before combining the tools, confirm that your application can preserve:

• a stable case and claim identifier;
• the governed source or grounding artifact;
• the expected source fingerprint;
• retrieved contexts in rank order;
• structured citation references;
• model, prompt, retriever, parser, and evaluator versions;
• canonical Ethos reports and RAGAS metric results;
• a policy decision that distinguishes release, review, and block.

If these data are missing, instrument the pipeline before tuning thresholds. Evaluation performed after provenance has been discarded cannot reconstruct a trustworthy evidence chain.

The definitive verdict

Ethos and RAGAS compete at the category edge, but most serious document RAG systems should treat them as complementary. RAGAS is well suited to semantic and retrieval experiments. Ethos is suited to exact citation identity, freshness, locator, and evidence checks. Together they cover more of the failure surface while keeping each result interpretable.

Adopt RAGAS first if your main question is which retriever, prompt, or model performs better. Adopt Ethos first if your main question is whether user-visible citations are real, current, and reproducible. Use both once semantic quality and evidence integrity become independent release requirements.

Primary keyword: Ethos and RAGAS Optimized meta title: Do Ethos and RAGAS Compete or Work Together? Optimized meta description: See where Ethos and RAGAS overlap, how they differ, and how to combine deterministic citation evidence with semantic RAG evaluation. Proposed URL slug: do-ethos-and-ragas-compete-or-should-you-use-them-together