A RAG Quality Evaluation Workflow

Diagnose the stage that failed

An end-to-end answer score cannot explain whether the corpus lacked the fact, parsing removed it, retrieval missed it, ranking buried it, the model ignored it, or the user question required information outside the authorized collection. Measure stages separately and connect them under one case and run identifier.

Layer	Question	Evidence
Corpus	Does the authorized, current source exist and contain the answer?	coverage, freshness, ownership, access
Ingestion	Was content parsed, chunked, labeled, and versioned correctly?	loss, duplication, metadata, ACL propagation
Retrieval	Were useful chunks ranked and irrelevant chunks controlled?	Recall@k, precision, MRR or NDCG with labels
Generation	Is each claim supported by supplied context?	faithfulness, citation alignment, abstention
Outcome	Does the answer resolve the user task completely and safely?	correctness, relevance, completeness, actionability
Operations	Does quality survive updates, load, failures, and cost limits?	latency, spend, drift, rollback, trace coverage

Freeze the corpus and ingestion version

Record source identifiers, versions, access rules, ingestion time, parser, chunking, normalization, embedding model, metadata schema, and index settings. Evaluation results are not comparable when the source collection changes invisibly. Keep a manifest or hash set for the evaluation snapshot.

Inspect a stratified sample after ingestion: tables, lists, headings, code, PDFs, scanned pages, repeated headers, footnotes, and access metadata. A clean retrieval score on ordinary paragraphs can coexist with systematic loss of the documents users care about.

Build cases from real information needs

Collect questions from support, internal search, analysts, documentation gaps, and subject experts. Preserve language and ambiguity. For each case, record intent, expected answer elements, relevant document and passage identifiers, acceptable alternatives, freshness cutoff, authorization context, and whether abstention is correct.

Case type	What it tests
Answer present once	Basic retrieval and grounding
Answer spread across documents	Multi-hop retrieval and synthesis
Conflicting current and obsolete sources	Freshness, metadata, and uncertainty
No answer in authorized corpus	Abstention and escalation
Near-duplicate documents	Ranking and evidence diversity
Permission-restricted answer	Access control before and after retrieval
Adversarial instructions in a document	Data-versus-instruction separation
Ambiguous or underspecified query	Clarification and intent resolution

Use a development set for tuning and a protected release set. Add production failures only after removing unnecessary personal or confidential data and recording permission. Keep synthetic cases for rare attacks and policy boundaries, but label them so they do not represent estimated traffic frequency.

Label retrieval independently

Ask qualified reviewers which documents and passages are relevant before viewing the generated answer. Use graded relevance when one chunk directly answers the question while another provides context. For small high-value sets, adjudicate disagreements. Track missing judgments; an unlabeled relevant passage can make retrieval appear worse than it is.

Measure Recall@k when missing any supporting passage is costly, precision when irrelevant context distracts generation or raises cost, and ranking metrics such as reciprocal rank or NDCG when position matters. Report k, filters, query rewriting, reranker, and index version. No metric is meaningful without that configuration.

Evaluate the answer against supplied evidence

Break the response into verifiable claims. Mark each as supported, partially supported, contradicted, or not present in retrieved context. Check citations for passage alignment, not only URL validity. Measure whether the answer includes required elements, resolves the question, states uncertainty, and avoids content the user is not permitted to see.

Groundedness and correctness differ. A response can faithfully repeat an obsolete document, or reach a true answer from unsupported model memory. Both should fail a current, evidence-bound task. Completeness also differs from faithfulness: an answer may say nothing false while omitting the decisive exception.

Calibrate automated judges

LLM evaluators can label relevance, groundedness, and completeness at useful scale, but their scores depend on instructions, model, context, and threshold. Create a human-labeled calibration set, measure agreement and important false passes, and version the judge. Use deterministic checks for source IDs, required fields, citations, dates, and access rules whenever possible.

Review all low scores and a random sample of passes. For high-impact cases, require human adjudication. Do not use the same unverified RAG answer as both candidate and reference.

Run controlled experiments

Freeze corpus, cases, model, prompt, filters, and budget.
Change one retrieval or generation factor at a time.
Capture retrieved IDs, scores, final context, answer, citations, latency, usage, and errors.
Compare overall and segment metrics, then inspect changed cases.
Classify failures before choosing the next experiment.

A higher average can conceal regressions on permission filters, current policies, or no-answer cases. Establish protected segments with hard thresholds. Retain the old system until the new one passes and rollback is tested.

Use a failure taxonomy

Corpus gap: create or update an authoritative source. Ingestion loss: repair parsing, chunking, or metadata. Query mismatch: clarify, rewrite, or add synonyms. Ranking failure: change retrieval or reranking. Context overload: remove irrelevant or duplicated chunks. Generation failure: strengthen output contract, grounding checks, or model choice. Policy failure: fix access enforcement outside the model.

Do not answer every failure with more context. Longer context can raise cost, latency, exposure, and distraction. The correct repair targets the failing layer.

Define release and monitoring thresholds

Set minimum retrieval recall for decisive documents, maximum critical grounding failures, required abstention on missing evidence, zero unauthorized retrieval, answer-completeness targets, tail latency, and cost per accepted answer. Define these by risk tier before testing.

In production, sample traces and corrections, monitor source freshness, no-answer rate, retrieval drift, judge disagreement, cost, and permission denials. Re-run the release set after corpus, parser, embedding, reranker, prompt, model, or policy changes. A content update can be as material as a code deployment.

Evaluation record template

For every run, retain case and corpus version, authorization context, query after any rewrite, filters, retrieved document and passage IDs with scores, reranker output, final context, model and prompt version, answer, citations, deterministic checks, judge version, human labels, latency, usage, and error class. Store sensitive text only when necessary; identifiers and hashes can often support reproduction with less exposure.

For every release, publish an internal summary containing case counts, segment results, hard-gate outcomes, changed configuration, known limitations, rollback version, and approving owners. Include examples of important failures, not only aggregate scores. A small improvement that creates one unauthorized retrieval is a failed release.

Schedule freshness cases around the source lifecycle. Test newly effective policy, expired offer, superseded document, and conflicting versions. Measure how long ingestion takes to make an approved source searchable and how quickly a withdrawn source disappears from results and caches.

Review multilingual and format-specific segments independently when they matter to users. Retrieval, parsing, chunking, and automated judging can behave differently across languages, tables, code, scans, and long documents. A strong English paragraph average does not establish quality for a multilingual PDF collection.

Keep an error-budget view for production. Count critical unsupported claims, missing-answer failures, unauthorized retrieval, stale answers, and unresolved cases over a defined window. Pause risky changes or reduce scope when the budget is exhausted, then require evidence that the controlling layer has been repaired.

Measure citation usability for readers as well as machine alignment. A source identifier should open the correct authorized document and, where practical, help locate the supporting passage. Broken deep links, inaccessible sources, and citations to an entire large document increase verification cost even when retrieval was technically correct.

Limitations

Evaluation labels can be incomplete or subjective, especially for open-ended questions. Benchmark results may not transfer to another domain, language, corpus, or access model. Automated groundedness does not prove a source itself is accurate or current.

This workflow does not replace privacy, security, legal, or domain review. Systems handling health, legal, financial, employment, or safety information need qualified experts and stronger governance. RAG reduces some unsupported generation risk but does not eliminate it.

Related guides

Choose between retrieval and action using RAG vs agents, instrument the pipeline with the observability guide, and protect sources through the security checklist. Use the content refresh workflow for source lifecycle.