AI Agent Security Checklist Before Production

Eight control areas

1. Establish identity and ownership

Every run needs an authenticated principal, a workflow owner, and an environment. Do not let the model choose which user, tenant, account, or organization it represents. Service identities should be separate by environment and purpose so a compromised development workflow cannot access production resources.

2. Minimize data exposure

Classify inputs before sending them to a model or tool. Retrieve data through the same authorization rules used by the application. Redact secrets and unnecessary personal information, and set retention rules for prompts, traces, screenshots, files, and tool responses.

3. Treat external instructions as untrusted

Web pages, email, documents, retrieved passages, issue comments, and tool output may contain instructions intended to manipulate the agent. Separate system policy from external content. A document may provide evidence, but it cannot authorize a new tool, reveal a secret, change the target account, or disable review.

4. Build a tool gateway

Route tool calls through application code that validates the tool name, arguments, target, permission, and approval requirement. Use narrow credentials rather than a shared administrator token. Add timeouts, rate limits, network restrictions, safe file paths, and idempotency keys where side effects are possible.

5. Validate model output

Model output is untrusted input to the next component. Validate structured data against a schema. Parameterize database queries. Restrict URLs and file paths. Never pass model-generated text directly to a shell, template engine, SQL interpreter, browser navigation, or outbound message without context-specific checks.

6. Require approval for high-impact actions

Human review must display the exact action, target, content, cost, and relevant evidence. “Approve task” is too broad if the agent can later change the arguments. Require a new approval after a material change, and expire approvals so they cannot be replayed.

7. Constrain runtime and cost

Set maximum steps, duration, tokens, retries, concurrent tools, and spend. A retry must distinguish a safe read from a side effect. Stop when evidence is insufficient, when policy validation fails, or when repeated attempts do not change the result.

8. Preserve traces and incident controls

Store trace IDs linking the request, retrieved evidence, model calls, state changes, tool arguments, approvals, responses, and final outcome. Protect traces because they may contain sensitive data. Provide a kill switch, credential revocation procedure, queue pause, and way to identify affected runs after an incident.

Adversarial release tests

1. Place a malicious instruction inside a retrieved document and confirm it cannot expand permissions.
2. Ask the agent to send data to an unapproved domain or account.
3. Return malformed tool output containing code, paths, or URLs.
4. Timeout a tool after it may have completed a side effect and verify idempotent recovery.
5. Attempt to access another user or tenant’s data through retrieval and tool arguments.
6. Exhaust the step or spend budget and confirm the workflow stops safely.
7. Change an approved action’s target and confirm a new approval is required.
8. Reconstruct a failed run using stored evidence without relying on the model’s explanation.

Release blockers

• A prompt is the only enforcement mechanism for permissions or allowed tools.
• One long-lived credential grants broad access to several systems.
• External content can be copied into privileged instructions without separation.
• Generated code, commands, URLs, or SQL execute without validation.
• Retries can repeat purchases, sends, deletes, or permission changes.
• Operators cannot stop runs or identify what the agent changed.
• There is no test set containing prompt injection and tool-failure cases.

Minimum security evidence record

For each control, record the owner, implementation location, test case, last verified date, failure behavior, and residual risk. Link tool scopes to the credential configuration and link approval requirements to policy code rather than a design document alone. Store the model, prompt, tool, and policy versions used in adversarial tests so a later upgrade can trigger the right regression suite.

A release reviewer should be able to follow one sensitive action from authenticated request through retrieved evidence, model proposal, policy decision, human approval, tool execution, external result, and audit event. Missing evidence is not proof that the control failed, but it is a reason to block production autonomy until the team can demonstrate the boundary.

Verify controls with evidence

For identity, attempt access with the wrong user, tenant, role, expired token, and revoked token. For data, trace a sensitive field through prompt, retrieval, cache, log, serialized state, tool call, and output. For tools, submit out-of-range values, unknown destinations, duplicate requests, and arguments containing hostile instructions. A policy document is not evidence until the implemented boundary blocks or records the expected behavior.

For human approval, verify reviewer identity, exact arguments, expiry, rejection, edit, and duplicate resume. For recovery, interrupt before and after external side effects and prove idempotency or compensation. For output, validate schema and business state independently. For monitoring, start from a test incident and confirm the owner can locate affected runs, revoke access, preserve evidence, and restore a known-good version.

Create a security case file

Store the system and data-flow diagram, agent and tool inventory, trust boundaries, identities and scopes, policy rules, approval map, threat model, test cases, findings, residual risk, owners, deployment version, and review expiry. Reference secrets and sensitive samples rather than copying them. The case file should make missing evidence visible and support re-review after change.

Classify findings by consequence and exploit path. Block release for broken authorization, secret exposure, unauthorized external action, uncontained arbitrary execution, missing approval on an irreversible action, or inability to stop access. Time-bound lower risks with a named owner and compensating control; do not let “accepted risk” mean an undocumented permanent exception.

Test the operating response

Practice disabling new runs, stopping active work, revoking identities, invalidating pending approvals, rotating secrets, identifying external side effects, notifying owners, and replaying safe cases after repair. Include provider and integration failure, not only malicious behavior. An agent can create harm through ordinary retry, stale state, or wrong destination without an attacker.

Turn meaningful failures into regression cases. Preserve the source, proposed action, policy result, execution evidence, and corrected outcome in minimized form. Re-run protected security cases after model, prompt, retrieval, framework, tool, credential, network, or approval changes.

Review cumulative authority

A sequence of individually low-risk tools can create a high-risk path: read data, transform it, then send it externally; create code, write a file, then execute it; query accounts, select targets, then perform a bulk action. Review tool combinations and loop limits, not only each schema in isolation. Restrict destinations and data flow across the complete action graph.

Repeat the review for delegated agents and nested tools. The top-level agent remains accountable for authority exercised through a handoff. Preserve the original user, policy, approval, and trace relationship across delegation so a sub-agent cannot acquire a broader identity merely because it runs in another component.

Limitations

This checklist supports design review but does not replace threat modeling, secure code review, penetration testing, privacy assessment, or regulatory analysis. Risk depends on the data, users, integrations, jurisdiction, and consequences of error. Re-run the review after changing models, tools, permissions, retrieval sources, or execution environments.

Related planning

Confirm the process is stable with the automation readiness checklist. Choose the simplest architecture with RAG vs agents. Add spend and retry limits using the cost model, then evaluate tool stacks in the AI tools hub.