How to Evaluate AI Agents for Web Research

Define the research job before choosing an agent

“Research” can mean locating one obscure fact, comparing products, mapping a market, tracing a regulatory requirement, reviewing scientific evidence, or monitoring a changing topic. These jobs do not share one correct metric. BrowseComp deliberately uses hard-to-find questions with short, verifiable answers; GAIA combines browsing with reasoning, files, and tools. Both are useful capability probes, but neither automatically represents the long, ambiguous reports most teams request.

Write a task contract for the intended workload. Name the audience, decision, time horizon, allowed and prohibited sources, required sections, citation granularity, freshness rule, budget, and conditions that require abstention. If a purchasing report must compare regional availability and contract terms, a fluent global summary that omits those constraints is a failed result.

Build a representative test set

Begin with 30 to 50 cases sampled from actual work. Include routine lookups, multi-source synthesis, conflicting evidence, missing evidence, current facts, archived facts, paywalled or inaccessible sources, and adversarial pages that contain instructions aimed at the agent. Keep a smaller release set hidden from prompt and workflow authors so repeated tuning does not optimize only the visible examples.

For each case, store the question, why it matters, mandatory sub-questions, acceptable source types, reference facts, evidence links, known conflicts, volatility, prohibited acquisition methods, and maximum acceptable cost. A reference answer should be an evidence map, not merely polished prose. Several valid narratives may satisfy the task while sharing the same supported facts.

Split cases by difficulty and risk. A low-risk background summary can tolerate qualified uncertainty; a claim used for compliance, security, or a major purchase needs stronger sources and human review. Report results by segment so high performance on simple lookups cannot hide weak performance on consequential synthesis.

Use a rubric that exposes tradeoffs

The weights above are a starting point. Increase source quality and uncertainty handling for regulated or high-impact research. Increase efficiency for high-volume monitoring. Do not allow a strong writing score to compensate for fabricated evidence. Set hard rejection rules for nonexistent citations, unsupported decisive claims, privacy violations, or failure to disclose a material conflict.

Check citations at claim level

URL validity is the weakest citation test. Open every sampled source and locate the passage, table, or record that supports the nearby claim. Check entity, date, geography, product version, plan, and whether the report converted correlation into causation. A source may be authoritative yet irrelevant to the sentence attached to it.

Sample all decisive claims and a random portion of ordinary claims. Mark each citation as direct support, partial support, contextual only, contradictory, or inaccessible. Track citation precision—the share of cited claims genuinely supported—and evidence coverage—the share of consequential claims that have adequate support. These measures reveal reports that look well sourced because they contain many links while leaving their strongest conclusions ungrounded.

Require source diversity only when independence matters. Five sites repeating the same press release are one evidence chain. Prefer original documentation for product behavior, official records for rules, original papers for research results, and clearly identified datasets for quantitative claims. Secondary analysis can add interpretation or expose disagreement, but should not obscure the origin of a volatile fact.

Run the comparison reproducibly

1. Freeze the agent version, model, instructions, tool configuration, source permissions, locale, date, and budget.
2. Give every candidate the same task inputs and equivalent access. Record unavailable capabilities rather than compensating silently.
3. Capture queries, URLs, tool errors, elapsed time, token or product charges, report output, and human interventions.
4. Use deterministic checks for URLs, required sections, dates, and reference facts before human scoring.
5. Blind reviewers to the candidate where practical and provide a written rubric with examples.
6. Repeat a subset to measure variance. One lucky run is not reliable production behavior.

Do not secretly repair prompts, add missing sources, or continue a stopped run for one candidate. If operator help is part of the intended workflow, define it as a measured stage and count its time. The correct unit is cost per accepted research outcome, including failed runs and review—not the advertised cost of one model call.

Classify failures before changing prompts

A failure taxonomy prevents random prompt expansion. Retrieval misses may require query planning or source access; evidence mismatch needs citation validation; synthesis errors need decomposition and review; freshness failures need dated sources. Adding a larger model or longer prompt to every problem can raise cost without fixing the controlling failure.

Set a release decision

Define minimum overall performance and non-negotiable segment thresholds before viewing results. A useful gate might require zero fabricated citations in the release set, complete coverage of mandatory questions, a high evidence-support rate for decisive claims, bounded cost and latency, and a documented human escalation path for unresolved conflict. Preserve the baseline so later model, search, or prompt changes are regression-tested.

Choose a bounded deployment first. Limit domains or task categories, require review before a report drives an external decision, and monitor corrections, citation failures, abandoned runs, and cost drift. Expand only when evidence shows the agent is reliable on the new boundary.

Reviewer worksheet

For each report, record task ID, candidate version, reviewer, decisive claims, supported claims, partial or conflicting citations, missed requirements, inaccessible sources, interventions, elapsed time, total charge, and final acceptance. Add one sentence describing the highest-impact failure and the component that should change. This creates evidence for improvement without turning every weak result into a larger prompt.

Review a consistent sample after deployment and compare it with the frozen release set. New domains, languages, report formats, connected data sources, or action capabilities expand the evaluation boundary and require new cases.

Limitations

This framework does not produce a universal ranking. Results depend on the task distribution, source access, date, model, product plan, region, reviewer knowledge, and budget. Public benchmark scores can be affected by task leakage or specialization and may not predict performance on open-ended reports.

LLM-based judges can accelerate review but can share biases with the system being evaluated. Calibrate them against human labels and retain manual review for decisive claims. Research involving legal, medical, financial, personal, or confidential information needs qualified review and additional privacy controls.

Related guides

Use the agent pre-deployment evaluation guide for action-oriented systems, the RAG quality workflow for controlled corpora, and the observability guide to preserve run evidence. Compare runtime choices through the Agents SDK vs LangGraph guide.