AI Agent Framework Selection Checklist

First gate: do you need an agent framework?

Use a direct model call when the input and output fit one bounded transformation. Use retrieval when the main problem is finding grounded evidence. Use a deterministic workflow when the sequence is known. Use an agent when the system must inspect intermediate results, choose among tools, adapt the next step, and stop under explicit limits.

A framework is justified when it removes operating work you would otherwise build and maintain: state persistence, tool dispatch, human interruptions, tracing, evaluation hooks, session handling, or multi-agent messaging. It is not justified merely because the documentation produces an attractive demo.

Weighted selection matrix

Criterion	Question	Weight
Determinism and control	Can the team define and test state transitions?	20%
Durability	Must runs pause, resume, retry, or survive deployment?	15%
Tool and identity boundaries	Can each action use a narrow permission and approval policy?	15%
Evaluation	Can behavior be regression-tested against accepted outcomes?	15%
Observability	Can operators reconstruct prompts, state, tool calls, errors, and cost?	15%
Provider and language fit	Does the stack match required models, cloud, SDKs, and team skills?	10%
Operational complexity	Can the on-call team debug and upgrade it?	10%

Score each option from 1 to 5. Multiply the score by the percentage weight, then compare the total and the lowest individual scores. A high total does not excuse a score of 1 for permissions or durability when the workflow handles money, private data, code execution, or external communication.

Choose the architecture family before the vendor

Direct API or small custom loop

Best when the workflow is narrow, the team wants to own state and tool dispatch, and a framework would add more abstraction than capability. The OpenAI Agents SDK documentation itself distinguishes between using the underlying Responses API directly and using an agent runtime for managed turns, tools, guardrails, handoffs, or sessions.

Deterministic workflow engine

Best when transitions and approvals are known in advance. Individual steps can still use models, but business state remains explicit. This is often safer for onboarding, document processing, support triage, and other processes where exceptions can be enumerated.

Graph runtime

Best for long-running, stateful processes that need checkpoints, interruptions, branching, replay, or human review. LangGraph’s documentation emphasizes persistence, durable execution, streaming, and human-in-the-loop control rather than hiding the orchestration layer.

Agent SDK

Best when a team wants a managed agent loop, tools, sessions, guardrails, handoffs, and tracing without adopting a larger workflow platform. Check how tightly the SDK couples the application to a provider and which production surfaces still remain your responsibility.

Multi-agent framework

Best only when independently scoped agents, delegation, or event-driven collaboration solve a real decomposition problem. AutoGen documents both conversational AgentChat and an event-driven Core; CrewAI centers crews, roles, tasks, and flows. Multiple agents add messages, state, failure paths, and cost, so compare against one agent with well-defined tools.

Prototype the failure path

Select one task containing an unreliable tool, ambiguous result, approval point, and recoverable failure.
Use the same model, inputs, tools, and acceptance test across the top two framework options.
Interrupt the run after a side-effect-free step and test whether it can resume safely.
Inject a tool timeout, malformed output, and an instruction embedded in retrieved content.
Measure accepted completion rate, operator debugging time, duplicate side effects, latency, and total cost.
Ask the on-call engineer to reconstruct one failed run using only stored evidence.

Red flags during selection

The framework is chosen before the workflow has acceptance criteria.
The prototype uses a happy path but the production process is dominated by exceptions.
Tracing is treated as console logging rather than a link between state, model calls, tools, and outcomes.
Human review exists in a slide but not as an enforceable interruption.
Retries repeat non-idempotent actions such as sending, purchasing, or deleting.
Model or framework popularity is used as a substitute for testing the team’s actual task.

Limitations

This rubric compares architecture fit, not benchmark performance. Framework capabilities and product packaging change. The final choice should use current official documentation and a representative prototype. Regulated, safety-critical, or high-impact systems require domain-specific legal, security, and governance review beyond this checklist.

Continue the evaluation

Compare current options in the LangChain alternatives guide and AutoGen alternatives guide. Before granting tools, apply the security checklist. Estimate operating burden with the automation cost model.