Why Are Vector Similarity Scores Not Proof That an AI Answer Is Correct?

Direct answer

Vector similarity scores are retrieval ranking signals, not proof that an AI answer is correct. A high score says that a query and a candidate chunk occupy nearby positions under a particular embedding model and similarity function. It does not establish that the chunk entails the generated claim, contains the cited number, represents the correct document version, or is an authoritative source.

This distinction matters because a retrieval system and a verification system answer different questions. Retrieval asks, “Which chunks should the model inspect?” Verification asks, “Does this exact evidence support this exact claim under the release policy?” Treating the first answer as the second produces confident-looking citations without an evidence contract.

The safe sequence is: govern eligible sources, retrieve and rerank candidates, generate an answer, bind claims to evidence, verify those bindings, and then decide whether to release. Vector search is valuable inside that sequence, but its score is not a truth probability.

What a vector similarity score actually measures

An embedding model maps text into a numerical vector. A vector index compares the query vector with stored chunk vectors using a function such as cosine similarity, dot product, or distance. The result orders candidates by closeness according to the representation learned by that model.

For cosine similarity, the common calculation is:

[ \operatorname{cosine}(q,d)=\frac{q \cdot d}{\lVert q\rVert\lVert d\rVert} ]

Here, q is the query vector and d is a document-chunk vector. The equation measures the angle between two vectors. It contains no variable for factual accuracy, source authority, publication date, document version, page identity, or logical entailment.

The Dense Passage Retrieval paper describes learned dense representations for selecting relevant passages. The original Retrieval-Augmented Generation paper uses a dense vector index to retrieve documents that condition generation. Neither turns nearest-neighbor distance into proof of the final answer.

Production rule: Name the field retrieval_score, not confidence, truth_score, or verification_score. A precise name prevents downstream code and dashboards from assigning the number a meaning it does not have.

Why the score is not a universal probability

A similarity value only has meaning within the configuration that produced it. Change the embedding model, normalize vectors differently, replace cosine similarity with dot product, alter chunk length, add metadata filters, or approximate the nearest-neighbor search, and the distribution can change.

Even indexes built from the same documents can produce different score ranges because of:

• embedding model and model version;
• vector normalization;
• distance metric and whether the database reports similarity or distance;
• chunk size, overlap, and boundary placement;
• title, header, and metadata concatenation;
• query rewriting or hypothetical-document expansion;
• index quantization and approximate-search parameters;
• corpus composition and duplicated passages.

A score of 0.82 is therefore not “82% likely to be correct.” It may be a strong retrieval signal in one evaluated index and an ordinary score in another. Thresholds should be calibrated against a retrieval objective—such as recall of an answer-bearing passage—not borrowed from a different model or database.

Eight ways high-similarity context can still produce a wrong answer

The easiest way to understand the boundary is to examine failures where semantic proximity remains high.

Consider the question, “Can contractors export customer records?” A chunk stating “Contractors may access customer records but may not export them” is likely to be highly similar. A generator that compresses it to “Contractors may export customer records” has inverted the governing fact while preserving nearly all of the vocabulary.

Numbers are another common trap. “The retention period is 30 days” and “The retention period changed from 30 to 90 days” may both rank highly. Similarity does not tell the application which number is current, which clause governs, or whether the question concerns backups rather than active storage.

Research datasets such as RAGTruth document hallucinations in retrieval-augmented generation, including unsupported and contradictory content. Retrieval reduces some failures by supplying context, but context availability and answer support remain separate properties.

Retrieval relevance, grounding, and correctness are different tests

Teams often collapse several layers into a single “quality” score. That makes failures hard to diagnose and policies hard to enforce. Keep the layers explicit.

This is why RAG faithfulness, groundedness, and citation verification need separate names. A response may faithfully paraphrase an obsolete policy, cite a real but irrelevant paragraph, or state a true fact with a broken citation. Each condition requires a different remediation.

A safer production architecture

The retrieval score belongs near the beginning of the pipeline, while the release decision belongs at the end.

1. Filter eligible sources. Apply tenant, access-control, document-status, jurisdiction, language, and effective-date rules before semantic ranking where possible.
2. Retrieve broadly enough for recall. Use vector, lexical, or hybrid search to produce candidates. This stage is allowed to be probabilistic.
3. Rerank for the query. A cross-encoder or task-specific scorer may improve candidate order, but its output is still relevance evidence.
4. Generate with locators. Require the model or orchestration layer to retain stable source IDs, page or element locators, and evidence spans.
5. Decompose the answer into material claims. Verification at whole-answer granularity hides which sentence failed.
6. Verify citation integrity. Check that each declared source exists and that the page, quote, table cell, or region resolves against the fingerprinted artifact. See how to verify that an AI citation exists in its source document.
7. Evaluate semantic support. Determine whether the resolved evidence supports, contradicts, or does not address each claim.
8. Apply a release policy. Fail closed when required evidence is missing, malformed, unauthorized, contradictory, or below the application’s coverage requirement.

This separation makes observability useful. A retrieval miss should lead to corpus, chunking, query, or ranking work. A citation-integrity failure should lead to locator or source-binding work. An unsupported claim should lead to generation, claim decomposition, or evidence-coverage work.

Keep ranking code honest about its contract

Production types can make misuse harder. The following TypeScript example filters ineligible candidates before ranking and returns a deliberately narrow result type. It does not expose a field named confidence and it does not decide whether an answer is correct.

type Candidate = {
  sourceId: string;
  sourceFingerprint: string;
  tenantId: string;
  effectiveFrom: string;
  retiredAt?: string;
  embedding: readonly number[];
};

type RetrievedCandidate = {
  sourceId: string;
  sourceFingerprint: string;
  retrievalScore: number;
};

function cosineSimilarity(a: readonly number[], b: readonly number[]): number {
  if (a.length === 0 || a.length !== b.length) {
    throw new Error("Embedding dimensions must match and be non-empty");
  }

  let dot = 0;
  let normA = 0;
  let normB = 0;
  for (let i = 0; i < a.length; i += 1) {
    dot += a[i] * b[i];
    normA += a[i] * a[i];
    normB += b[i] * b[i];
  }

  if (normA === 0 || normB === 0) {
    throw new Error("Zero vectors cannot be ranked with cosine similarity");
  }
  return dot / (Math.sqrt(normA) * Math.sqrt(normB));
}

export function retrieveEligibleCandidates(
  queryEmbedding: readonly number[],
  candidates: readonly Candidate[],
  tenantId: string,
  asOf: Date,
  limit: number,
): RetrievedCandidate[] {
  const asOfIso = asOf.toISOString();

  return candidates
    .filter((candidate) => candidate.tenantId === tenantId)
    .filter((candidate) => candidate.effectiveFrom <= asOfIso)
    .filter((candidate) => !candidate.retiredAt || candidate.retiredAt > asOfIso)
    .map((candidate) => ({
      sourceId: candidate.sourceId,
      sourceFingerprint: candidate.sourceFingerprint,
      retrievalScore: cosineSimilarity(queryEmbedding, candidate.embedding),
    }))
    .sort((left, right) => right.retrievalScore - left.retrievalScore)
    .slice(0, limit);
}

The function answers one question: which eligible candidates are closest to the query vector? A later verifier must receive the generated claims, fingerprinted sources, and precise evidence locators. Mixing those responsibilities would make a retrieval tuning change silently alter the release policy.

Security warning: Post-filtering retrieved results can leak cross-tenant information through logs, timing, caches, or generated text. Apply enforceable authorization before content reaches generation, and reauthorize the source when a user opens a citation.

How to calibrate similarity thresholds correctly

A threshold is useful only when its objective and dataset are named. Build a representative evaluation set of real query types, including short questions, terminology mismatches, table lookups, date-sensitive requests, and questions with no valid answer.

For retrieval, measure:

• Recall@k: how often at least one answer-bearing passage appears in the top k;
• Precision@k: how many retrieved passages are actually useful;
• mean reciprocal rank: how early the first relevant passage appears;
• no-answer behavior: whether unrelated passages still receive deceptively high scores;
• slice performance: results by document type, language, table density, and query class.

Choose a retrieval threshold to balance candidate recall, latency, and context budget. Do not describe it as an answer-correctness threshold. Then evaluate the downstream answer separately with claim support, citation integrity, source authority, coverage, contradiction, and human review metrics.

Recalibrate after changing the embedding model, corpus, chunking strategy, normalization, metadata composition, or index settings. The old score distribution is evidence about the old system, not a permanent constant.

Adversarial tests every RAG system should run

Normal queries rarely expose the semantic shortcuts that cause expensive failures. Add paired tests where relevance stays high but the governing meaning changes.

• Positive and negated versions of the same policy clause.
• Current and superseded documents with nearly identical wording.
• Two jurisdictions with different limits.
• Table cells sharing the same value under different headers.
• A correct paragraph beside an exception paragraph.
• A cited quote that exists on the page but does not support the claim.
• A multi-sentence answer where only one claim lacks evidence.
• A question whose correct outcome is “the supplied sources do not establish this.”

Record retrieval and verification outcomes separately. If a top-five candidate contradicts the answer, retrieval succeeded while generation or support evaluation failed. If no answer-bearing passage appears, do not blame the citation checker.

The guide to why RAG answers can be wrong even with citations provides additional failure patterns. For finer-grained policies, use claim-level citation verification in RAG so one unsupported sentence cannot borrow credibility from a nearby supported sentence.

Where Ethos fits

Ethos is designed for the deterministic evidence boundary after retrieval and generation. It can verify submitted citation evidence against a declared GroundingSource, produce machine-readable checks and receipts, and help a release policy distinguish verified evidence from missing or malformed bindings.

Ethos does not make a vector score into proof, choose the best embedding model, or replace semantic evaluation of whether evidence entails a claim. Its role is narrower and more auditable: establish what source and locator were submitted, whether supported deterministic checks passed, and what evidence artifact can be inspected later.

That makes Ethos complementary to vector databases, hybrid retrievers, rerankers, and model-based evaluators. Retrieval proposes evidence. Semantic evaluation interprets support. Ethos strengthens the source-bound proof layer and the fail-closed contract connecting evidence to release decisions.

The definitive verdict

Vector similarity scores efficiently find candidate context, but they are not proof that an AI answer is correct. They measure closeness within a configured representation space—not entailment, source authority, freshness, citation integrity, or real-world truth.

Use similarity to rank eligible candidates and evaluate it with retrieval metrics. Then verify each material claim against fingerprinted, precisely located evidence; evaluate semantic support; and apply an explicit release policy. A trustworthy RAG system does not ask one number to perform every job.

Primary keyword: vector similarity scores RAG Optimized meta title: Why Vector Similarity Does Not Prove AI Answers Optimized meta description: Learn why vector similarity scores rank RAG context but cannot prove AI answers are supported, authoritative, current, or correct. Proposed URL slug: why-are-vector-similarity-scores-not-proof-ai-answer-is-correct