How to Evaluate AI Agents Before Deployment

Evaluate the system that can act

A chatbot evaluation that scores the final text is insufficient for an agent. Two runs can return the same sentence while one used an unauthorized source, sent a duplicate message, ignored an approval, or left external state inconsistent. The evaluation unit must include the request, environment, state, evidence, policy, approvals, tool trajectory, side effects, validation, and total resources.

Write an agent contract first: intended users, accepted outcomes, allowed data and tools, prohibited actions, approval boundaries, budgets, stop rules, and named owner. If the team cannot state these, it cannot label a surprising trajectory as correct or wrong.

Construct a representative dataset

Sample real work across ordinary, edge, and high-consequence cases. Preserve the distribution, but deliberately add rare failures that production traffic may not reveal before damage: expired credentials, unavailable dependencies, malformed tool output, delayed approval, conflicting instructions, duplicate events, poisoned retrieved text, ambiguous destinations, and partial writes.

Each case needs initial state, input, policy context, permitted and prohibited actions, reference outcome, validation method, maximum budget, and expected escalation. Prefer executable fixtures or controlled test accounts over prose-only examples. A ticket is not resolved because the agent says it is; query the ticket system. A refund is not correct until amount, account, authorization, and ledger state match.

Keep a development set for iteration, a release set protected from routine tuning, and a small canary set for continuous checks. Version cases and environments. When policy or integrations change, preserve old cases that still represent possible pending work.

Score six layers separately

Do not collapse these into one average. Authority violations and duplicated irreversible actions are hard failures even if outcome quality is high. Trajectory quality is useful for diagnosing waste or fragility, but avoid requiring one exact path when several safe paths can succeed.

Grade outcomes with external evidence

Use deterministic checks wherever possible: schema validity, exact fields, database state, file hashes, message recipients, policy results, and expected records. Human rubrics are appropriate for usefulness, tone, reasoning, and cases with several valid outputs. LLM judges can scale review, but calibrate them against blinded human labels and monitor disagreement by case type.

Decompose complex work into named requirements. A research task may require factual correctness, evidence support, coverage, uncertainty, and safe source access. A browser action may require correct page, account, fields, approval, submission state, and no unrelated navigation. Requirement-level results reveal the component causing failure.

Test authority as a first-class outcome

Give the agent requests that exceed role, scope, destination, amount, data, or time boundaries. Place conflicting instructions in retrieved documents and tool output. Verify that application policy—not a prompt alone—blocks the action. Record both correct denial and overblocking, because a system that escalates every ordinary case may be safe but unusable.

Approval tests should verify the exact proposed action shown to the reviewer, the identity and authority of the reviewer, expiry, edits, rejection, and what happens after resume. A prior approval must not silently authorize changed arguments. Pending approvals need version markers so an old run cannot execute against incompatible policy or code.

Inject recoverable failures

1. Interrupt before a tool call, during a long wait, and immediately after a successful side effect.
2. Return timeouts, rate limits, invalid data, partial results, and conflicting external state.
3. Restart the worker and deploy a compatible code update while work is pending.
4. Deliver the same event twice and resume the same checkpoint more than once.
5. Verify idempotency, compensation, bounded retries, state integrity, and clear escalation.

A retry is not recovery when it repeats a charge or message. Capture failure class, attempts, checkpoint, idempotency key, external transaction identifier, compensation, and final validation. Include the engineering and operator time in cost.

Measure reliability, not a demo

Repeat nondeterministic cases and report pass probability, not only pass/fail from one run. Segment by task, risk, tool, input source, and failure type. Track tail latency and cost; long agent loops can produce acceptable average results while a small group consumes extreme resources.

Compare candidates under the same model, tools, prompts, environment, and budget when evaluating runtimes. When comparing full products, document every capability difference. Never add manual fixes to one candidate without counting them as interventions.

Declare release thresholds in advance

Set thresholds by risk tier. A read-only internal summary and a financial action should not share an acceptance bar. Require explicit sign-off from the business owner, security or privacy owner where relevant, and operator who will respond to failures. A failing gate creates a bounded remediation task; it does not invite changing the score after the fact.

Move from offline tests to a bounded rollout

Start in shadow mode or with reversible, low-impact actions. Compare agent proposals with actual outcomes and human decisions. Then limit users, tools, data, destinations, volume, and spend. Maintain a kill switch and a manual path. Monitor acceptance, intervention, policy denials, duplicate effects, unresolved runs, correction, cost, and trace completeness.

Promote one boundary at a time. A new model, prompt, tool, credential scope, data class, destination, or user population is a material change and triggers relevant regression cases. Retain evaluation evidence with the deployed version.

Maintain an evaluation evidence packet

Store the agent contract, dataset manifest, environment version, graders, human-review instructions, raw traces, aggregate and segment results, known gaps, approval record, and deployment identifier. Protect sensitive test data and traces with access and retention rules. The packet should let a new reviewer reproduce the release decision and identify exactly which cases must run after a proposed change.

Close the loop from production incidents and corrections. Add a minimized regression case, classify the controlling failure, repair the relevant component, and demonstrate that protected existing cases did not regress. Do not publish only the improved average; preserve critical-case results and sample counts.

Assign a review expiry even when no incident occurs. Workload mix, external services, model behavior, and adversarial techniques can change while aggregate dashboards remain stable. Periodically resample real cases and compare them with the original dataset assumptions.

Limitations

No finite test set proves an agent safe or correct in every environment. Rare interactions, changing external systems, novel attacks, and model updates create residual risk. Benchmarks measure selected capabilities and can be contaminated or unlike the organization’s workload.

Evaluation infrastructure can itself be wrong. Validate fixtures, graders, telemetry, and reference states; investigate suspicious improvements. High-impact legal, health, financial, employment, or safety decisions require qualified domain and governance review beyond this framework.

Related guides

Design the telemetry with the agent observability guide, place control points using human approval patterns, and assess authority through the security checklist. Use the readiness checklist before funding a pilot.