Can You Use Ethos with Promptfoo to Test Document RAG?

Direct answer

Yes. You can integrate Ethos with Promptfoo by having a custom RAG provider preserve structured citation artifacts and a custom Promptfoo assertion invoke Ethos against the trusted document source. Promptfoo then records deterministic citation grounding beside semantic, retrieval, schema, latency, and cost checks.

Promptfoo is the evaluation harness: it expands test variables, calls providers, and evaluates outputs with assertions. Ethos is the document evidence verifier: it checks source fingerprints, locators, quotations, values, regions, and supported evidence capabilities. Promptfoo’s official assertion documentation supports external JavaScript functions returning a structured grading result, which is the clean integration point.

The result is not one universal “hallucination score.” It is a test matrix that tells you whether the answer was relevant, whether its citations resolved, whether the source was current, and whether operational limits were met.

What each system owns

Promptfoo already includes deterministic assertions such as exact matching, regular expressions, JSON schema checks, and custom JavaScript or Python functions. It also includes model-assisted assertions such as answer relevance and context faithfulness. Ethos adds deterministic source-document evidence checks that are specific to citation-bearing document RAG.

Keep orchestration, generation, evidence verification, and release policy separate. That separation makes a failure diagnosable and prevents a model-graded average from hiding a broken source reference.

Recommended workflow

Use this sequence for every test case:

1. Promptfoo passes a question and document case ID to a custom provider.
2. The provider calls your RAG application and receives an answer, retrieval context, and structured citations.
3. The provider writes citations to a run-unique artifact path and returns the answer plus artifact references in response metadata.
4. A custom JavaScript assertion calls ethos verify with the trusted source and generated citations.
5. The assertion returns a named CitationGrounding result and writes the canonical Ethos report to an artifact directory.
6. Promptfoo runs other assertions for response schema, relevance, context faithfulness, latency, and policy requirements.

Promptfoo test vars
       |
       v
custom RAG provider --> answer + context + citation artifact metadata
       |                                  |
       |                                  v
       |                         custom Ethos assertion
       |                                  |
       v                                  v
semantic and operational assertions + citation grounding result
                         |
                         v
                CI decision and artifacts

The provider/assertion split is preferable to hiding verification inside generation. Promptfoo can show the citation check as its own metric, and a failed verifier does not alter the model’s original output.

Define the RAG response contract

Your application should return machine-readable citations instead of forcing the evaluator to scrape [1] markers from prose.

{
  "answer": "Q3 2025 revenue was $12.4 million [1].",
  "retrieval_context": [
    "For the quarter ended September 30, 2025, revenue was $12.4 million."
  ],
  "citations": [
    {
      "id": "claim-revenue",
      "page": 18,
      "quote": "revenue was $12.4 million",
      "document_fingerprint": "sha256:expected-source-fingerprint"
    }
  ]
}

The exact citation schema must match your Ethos verification contract. Do not invent missing page numbers after generation or let the evaluator search for the “closest” quote. If the application did not produce a valid citation, the test should expose that defect.

Build a custom Promptfoo provider

Promptfoo custom JavaScript providers implement an id() method and a callApi() method. This provider calls a local test endpoint, stores citations under a unique run directory, and returns paths through metadata. It avoids shell interpolation and checks every external response.

// rag-provider.mjs
import { mkdir, writeFile } from 'node:fs/promises';
import { randomUUID } from 'node:crypto';
import path from 'node:path';

export default class DocumentRagProvider {
  id() {
    return 'document-rag-under-test';
  }

  async callApi(prompt, context) {
    const baseUrl = process.env.RAG_TEST_BASE_URL;
    if (!baseUrl) {
      throw new Error('RAG_TEST_BASE_URL is required');
    }

    const response = await fetch(`${baseUrl}/api/test/answer`, {
      method: 'POST',
      headers: { 'content-type': 'application/json' },
      body: JSON.stringify({
        question: prompt,
        document_id: context.vars.document_id,
      }),
      signal: AbortSignal.timeout(30_000),
    });

    if (!response.ok) {
      throw new Error(`RAG endpoint failed with HTTP ${response.status}`);
    }

    const payload = await response.json();
    if (typeof payload.answer !== 'string' || !Array.isArray(payload.citations)) {
      throw new Error('RAG response must include answer and citations');
    }

    const runId = randomUUID();
    const runDir = path.resolve('.promptfoo-artifacts', runId);
    await mkdir(runDir, { recursive: true });
    const citationsPath = path.join(runDir, 'citations.json');
    await writeFile(
      citationsPath,
      `${JSON.stringify(payload.citations, null, 2)}\n`,
      'utf8',
    );

    return {
      output: payload.answer,
      metadata: {
        runId,
        citationsPath,
        sourcePath: path.resolve(String(context.vars.source_path)),
        retrievalContext: payload.retrieval_context ?? [],
      },
    };
  }
}

Run-specific directories matter because Promptfoo may evaluate multiple cases or providers concurrently. A shared citations.json creates a race in which one answer can be verified against another answer’s references.

The source path comes from the controlled test dataset, not the model response. Resolve it under an allowlisted fixture root in high-assurance environments so an untrusted test value cannot make the verifier read an arbitrary file.

Add the Ethos JavaScript assertion

The external assertion below uses execFile, not a shell command, invokes a caller-provided Ethos binary, parses JSON, retains the report, and returns Promptfoo’s structured grading shape.

// assertions/ethos-grounding.cjs
const { execFile } = require('node:child_process');
const { promisify } = require('node:util');
const { mkdir, writeFile } = require('node:fs/promises');
const path = require('node:path');

const execFileAsync = promisify(execFile);

module.exports = async function ethosGroundingAssertion(output, context) {
  const metadata = context.metadata || context.providerResponse?.metadata;
  if (!metadata?.sourcePath || !metadata?.citationsPath || !metadata?.runId) {
    return {
      pass: false,
      score: 0,
      reason: 'Provider did not return Ethos source and citation artifacts',
    };
  }

  const ethosBinary = process.env.ETHOS_CLI_PATH || 'ethos';
  try {
    const { stdout } = await execFileAsync(
      ethosBinary,
      [
        'verify',
        metadata.sourcePath,
        '--citations',
        metadata.citationsPath,
        '--format',
        'json',
      ],
      {
        encoding: 'utf8',
        timeout: 30_000,
        maxBuffer: 10 * 1024 * 1024,
      },
    );

    const report = JSON.parse(stdout);
    const reportDir = path.resolve('.promptfoo-artifacts', metadata.runId);
    await mkdir(reportDir, { recursive: true });
    await writeFile(
      path.join(reportDir, 'ethos-verification-report.json'),
      `${JSON.stringify(report, null, 2)}\n`,
      'utf8',
    );

    const passed = report.all_evidence_grounded === true;
    const failedIds = Array.isArray(report.checks)
      ? report.checks
          .filter((check) => check.status !== 'grounded')
          .map((check) => check.id)
      : [];

    return {
      pass: passed,
      score: passed ? 1 : 0,
      reason: passed
        ? 'All requested citation evidence is grounded'
        : `Ethos did not ground required checks: ${failedIds.join(', ') || 'unknown'}`,
    };
  } catch (error) {
    return {
      pass: false,
      score: 0,
      reason: `Ethos verification infrastructure error: ${error.message}`,
    };
  }
};

Confirm CLI flags against the exact Ethos version installed in your build image. The Python ethos-pdf wrapper is another valid boundary if your evaluation harness is Python-first; it exposes EthosCli.verify() and proof_summary() while still requiring a caller-provided CLI binary.

Do not treat a timeout, missing binary, malformed report, or unreadable file as an ordinary negative citation. It is an evaluation infrastructure failure and should block the lane with a distinct reason.

Configure the Promptfoo test matrix

Promptfoo supports file-based providers and JavaScript assertions. Give each assertion a named metric so reports separate grounding from semantic and operational quality.

# promptfooconfig.yaml
description: Document RAG evidence and quality evaluation

prompts:
  - "{{question}}"

providers:
  - id: file://rag-provider.mjs

defaultTest:
  assert:
    - type: javascript
      value: file://assertions/ethos-grounding.cjs
      metric: CitationGrounding
    - type: answer-relevance
      threshold: 0.8
      metric: AnswerRelevance
    - type: latency
      threshold: 5000
      metric: Latency

tests:
  - description: Q3 revenue with exact page citation
    vars:
      question: What was revenue in Q3 2025?
      document_id: q3-2025-report
      source_path: fixtures/q3-report.ethos.json

  - description: Policy exception must not be omitted
    vars:
      question: Can contractors export customer data?
      document_id: data-export-policy-v7
      source_path: fixtures/data-export-policy-v7.ethos.json

Model-assisted assertions need an evaluator provider and may transmit content to that provider. Configure an approved model path, redact where appropriate, and record the evaluator version. The deterministic Ethos assertion does not itself require a judge-model call.

Run the evaluation with telemetry, cache, and update behavior set according to your organization’s reproducibility policy, and export a machine-readable result in addition to any local HTML view. Promptfoo’s results-output documentation describes supported export formats.

Build adversarial document RAG cases

A useful suite deliberately breaks one layer at a time.

This matrix demonstrates why “faithful” and “verified” are not synonyms. An answer can be faithful to a retrieved chunk while citing the wrong page. It can cite a real paragraph while drawing a conclusion the paragraph does not support.

Apply a fail-closed release policy

Do not make the Promptfoo aggregate score the only release criterion. Define mandatory metrics:

• CitationGrounding must equal 1 for every citation-required case;
• schema and infrastructure assertions must pass;
• critical domain cases must meet their individual semantic thresholds;
• stale fingerprints and required unsupported capabilities must block release;
• partially verified cases must enter review unless an explicit policy permits a bounded response.

Weighted averages are useful for comparing configurations, but they are unsafe for hard evidence obligations. A model should not compensate for a stale policy citation by being faster or more stylistically relevant.

The deterministic RAG release-gate guide explains this policy boundary in more detail.

Operational and security controls

Pin every meaningful input

Record the Promptfoo version, Ethos version, test dataset commit, application revision, model snapshot, prompt revision, retrieval configuration, and source fingerprint. Without those identifiers, a later rerun cannot explain why the result changed.

Keep artifacts privacy-safe

Canonical reports may contain quotes, locators, or document metadata. Store them in access-controlled CI artifacts, redact logs, set retention rules, and avoid embedding sensitive source text in public build summaries.

Avoid arbitrary command and file access

Never construct a shell string from test variables. Use execFile argument arrays, allowlist source roots, validate resolved paths, cap output size, and apply timeouts. Treat provider-returned paths as untrusted unless your test service controls them.

Separate cache policy from evidence freshness

An evaluation cache can conceal a changed source, parser, or retrieval result. Disable or tightly key caching for release runs. Include document and configuration fingerprints in cache keys when caching is necessary.

Preserve the canonical report

The Promptfoo grading result is a concise projection. It is not a replacement for the Ethos verification report, which contains check-level statuses and limitations. Retain both and link them with the same run ID.

The definitive verdict

Use Ethos with Promptfoo when you want a reproducible document RAG test matrix that combines deterministic citation verification with flexible LLM evaluation orchestration. Capture citations as structured artifacts, run Ethos through a dedicated custom assertion, expose the result as a named mandatory metric, and keep semantic or model-graded assertions separate.

Start with eight cases: grounded baseline, wrong page, altered quote, stale fingerprint, missing citation, omitted exception, unsupported capability, and wrong calculation. Once each defect fails in the intended lane, expand across document versions, parsers, retrievers, prompts, and model providers. The goal is not merely more scores; it is a release decision whose evidence can be inspected and reproduced.

Primary keyword: Ethos Promptfoo integration Optimized meta title: Use Ethos with Promptfoo for Document RAG Tests Optimized meta description: Integrate Ethos with Promptfoo to test document RAG citations, relevance, latency, and semantic quality with deterministic CI evidence gates. Proposed URL slug: can-you-use-ethos-with-promptfoo-to-test-document-rag