SimpleQA vs TruthfulQA vs FreshQA: Which Benchmark Should You Use?

Choose the benchmark by the failure you need to study

Use SimpleQA for short factual questions, TruthfulQA for resistance to common misconceptions, and FreshQA for changing facts and false premises. Use none of them as a substitute for testing your own RAG or document workflow.

Benchmark scope warning: a public score measures a model under a particular dataset and evaluation setup. It is not a universal hallucination rate and does not include your parser, retriever, permissions, or citation interface.

SimpleQA

SimpleQA focuses on short questions with concise factual answers. It is useful for comparing factual answer performance where public results are available.

Its narrow format does not reproduce long documents, citations, retrieval failures, or production tool use.

TruthfulQA

TruthfulQA asks questions where people may hold false beliefs. It tests whether a model imitates those misconceptions instead of giving a truthful answer.

This reveals a different risk from simple fact recall, but the fixed set is not a live measure of current-world knowledge.

FreshQA

FreshQA includes questions whose answers can change and questions built on false premises. It is useful for studying freshness and whether a model challenges a bad assumption.

Results depend on model access to current information and the evaluation setup.

How DocuShell uses benchmark context

The DocuShell Hallucination Index links to public sources and keeps unavailable values marked as unavailable. It separates benchmark evidence from its inspectable workflow-risk estimate.

That boundary matters. A benchmark score does not tell you how a model will handle your scanned PDFs, chunking, retrieval, or citation UI.

Build a workflow-specific suite

Use public benchmarks to understand broad behavior and shortlist models. Then test answerable and unanswerable questions from your own documents. Measure retrieval, faithfulness, citation support, and refusal separately.

Benchmarks are useful when their scope is clear. They become misleading when one narrow result is marketed as a universal hallucination score.

Comparison at a glance

Read each benchmark's dataset, scoring method, and model setup before comparing reported numbers from different sources.

Why results are easy to misuse

Scores may come from different model snapshots, prompts, tools, or grading rules. Some models may use retrieval while others answer from internal knowledge. A table that removes those details can imply a fair race where none occurred.

Do not convert benchmark accuracy directly into a universal “hallucination rate.”

Add document-specific evaluation

For PDF Q&A, test clean text, OCR, tables, multiple document versions, unanswerable questions, and citation requirements. Measure whether claims are supported by the retrieved evidence.

Public benchmarks provide background. Your own suite determines release readiness.

Keep benchmark context and product evaluation separate in configuration. For example:

evaluation_plan:
  public_context:
    - benchmark: SimpleQA
      purpose: short_form_factuality
    - benchmark: TruthfulQA
      purpose: misconception_resistance
    - benchmark: FreshQA
      purpose: freshness_and_false_premises
  product_suite:
    corpus: private_policy_pdfs
    required_metrics:
      - retrieval_recall
      - claim_support
      - citation_precision
      - abstention_accuracy
    release_gate:
      unsupported_high_risk_claims: 0

The public benchmark section informs model selection. The product suite decides whether the document workflow is ready to ship.

Keep source dates visible

Benchmark repositories and model results can stop updating. Show the verification date and mark missing results as unavailable rather than estimating them.

Explore the source-linked DocuShell Hallucination Index, then read which LLM hallucinates least for document Q&A and how to measure RAG faithfulness. For practical risk reduction, use the AI hallucination prevention guide.

Primary benchmark sources

• OpenAI simple-evals and SimpleQA reference implementation
• SimpleQA paper
• TruthfulQA paper
• FreshQA paper

Check the source repository and paper before repeating a score. Benchmark coverage, model snapshots, and published result tables change independently.