AI Agent Framework Selection Checklist

First gate: do you need an agent framework?

Use a direct model call when the input and output fit one bounded transformation. Use retrieval when the main problem is finding grounded evidence. Use a deterministic workflow when the sequence is known. Use an agent when the system must inspect intermediate results, choose among tools, adapt the next step, and stop under explicit limits.

A framework is justified when it removes operating work you would otherwise build and maintain: state persistence, tool dispatch, human interruptions, tracing, evaluation hooks, session handling, or multi-agent messaging. It is not justified merely because the documentation produces an attractive demo.

Weighted selection matrix

Score each option from 1 to 5. Multiply the score by the percentage weight, then compare the total and the lowest individual scores. A high total does not excuse a score of 1 for permissions or durability when the workflow handles money, private data, code execution, or external communication.

Choose the architecture family before the vendor

Direct API or small custom loop

Best when the workflow is narrow, the team wants to own state and tool dispatch, and a framework would add more abstraction than capability. The OpenAI Agents SDK documentation itself distinguishes between using the underlying Responses API directly and using an agent runtime for managed turns, tools, guardrails, handoffs, or sessions.

Deterministic workflow engine

Best when transitions and approvals are known in advance. Individual steps can still use models, but business state remains explicit. This is often safer for onboarding, document processing, support triage, and other processes where exceptions can be enumerated.

Graph runtime

Best for long-running, stateful processes that need checkpoints, interruptions, branching, replay, or human review. LangGraph’s documentation emphasizes persistence, durable execution, streaming, and human-in-the-loop control rather than hiding the orchestration layer.

Agent SDK

Best when a team wants a managed agent loop, tools, sessions, guardrails, handoffs, and tracing without adopting a larger workflow platform. Check how tightly the SDK couples the application to a provider and which production surfaces still remain your responsibility.

Multi-agent framework

Best only when independently scoped agents, delegation, or event-driven collaboration solve a real decomposition problem. AutoGen documents both conversational AgentChat and an event-driven Core; CrewAI centers crews, roles, tasks, and flows. Multiple agents add messages, state, failure paths, and cost, so compare against one agent with well-defined tools.

Prototype the failure path

1. Select one task containing an unreliable tool, ambiguous result, approval point, and recoverable failure.
2. Use the same model, inputs, tools, and acceptance test across the top two framework options.
3. Interrupt the run after a side-effect-free step and test whether it can resume safely.
4. Inject a tool timeout, malformed output, and an instruction embedded in retrieved content.
5. Measure accepted completion rate, operator debugging time, duplicate side effects, latency, and total cost.
6. Ask the on-call engineer to reconstruct one failed run using only stored evidence.

Red flags during selection

• The framework is chosen before the workflow has acceptance criteria.
• The prototype uses a happy path but the production process is dominated by exceptions.
• Tracing is treated as console logging rather than a link between state, model calls, tools, and outcomes.
• Human review exists in a slide but not as an enforceable interruption.
• Retries repeat non-idempotent actions such as sending, purchasing, or deleting.
• Model or framework popularity is used as a substitute for testing the team’s actual task.

Compare two candidates with the same recovery drill

Implement one narrow vertical slice in the top two candidates. Use the same fixtures, tool contracts, persistence service, approval rule, and acceptance tests. Stop each run after a side effect but before completion is recorded, resume an approval after a deployment, and force a malformed tool response. Compare code that is unique to the framework, state that must be migrated, trace clarity, and the time an operator needs to explain the result.

Do not add optional features to make one prototype look complete. The exercise is a controlled comparison of the hardest operating requirements. A candidate that produces a shorter happy path but obscures duplicate-action recovery or policy enforcement should not win on developer convenience alone.

Build a decision packet, not a feature spreadsheet

For the two strongest candidates, document the same representative workflow as an executable sequence: accepted input, state transitions, tool contracts, permission checks, approval, interruption, resume, result validation, and operator evidence. Attach the code revision, runtime and model versions, deployment topology, persistence layer, trace sample, failure results, and measured cost. The packet should let someone outside the prototype team understand why one architecture fits.

Test at least four runs beyond the happy path. First, stop the worker immediately after a successful external write and confirm resume does not repeat it. Second, let an approval remain pending across a deployment and verify version compatibility or safe expiration. Third, return malformed or conflicting tool output and confirm bounded recovery. Fourth, place hostile instructions in retrieved content and verify that application policy still controls tools and destinations.

Estimate migration before selection. Inventory prompts, tools, state schemas, checkpoints, traces, evaluations, provider-specific objects, deployment code, operator dashboards, and pending work. Create one small export-and-replay exercise. A framework can be appropriate despite switching cost, but the cost should be visible rather than discovered during an incident or contract change.

Use vetoes before weighted scoring

A high total score must not compensate for a missing hard requirement. Define vetoes for data location, supported language, authorization model, durable recovery, required deployment environment, or an unacceptable license or service dependency. Apply vetoes from current evidence before calculating preferences. Record “unknown” rather than assigning optimistic points where documentation or a prototype cannot demonstrate behavior.

Then run sensitivity analysis. Change the largest weights and uncertain scores within reasonable ranges. If the winner changes easily, the decision is not robust and the team should run a focused test on the uncertain criterion. If one candidate wins across plausible ranges, avoid extending the comparison merely to create a more complicated report.

Plan the first production boundary

The selected framework does not authorize broad autonomy. Begin with one task, a narrow tool set, test identities, explicit destinations, step and spend ceilings, a manual fallback, and a named operator. Define the metrics that permit expansion: accepted outcomes, intervention, duplicate side effects, recovery success, critical policy failures, tail latency, cost, and trace completeness. Expand one boundary at a time.

Schedule a decision review after the pilot and after any material model, tool, state, deployment, or policy change. Revisit the original rejected alternatives only when a requirement or constraint changes; do not restart a market-wide comparison for every feature release. Keeping the record current is cheaper and more reliable than choosing again from memory.

Include the option to use no agent framework. A typed service, existing workflow engine, queue, database, and ordinary model API may satisfy a short bounded process with less state translation and fewer abstractions. Score that baseline with the same criteria so framework adoption must demonstrate value rather than win by default.

Limitations

This rubric compares architecture fit, not benchmark performance. Framework capabilities and product packaging change. The final choice should use current official documentation and a representative prototype. Regulated, safety-critical, or high-impact systems require domain-specific legal, security, and governance review beyond this checklist.

Continue the evaluation

Compare current options in the LangChain alternatives guide and AutoGen alternatives guide. Before granting tools, apply the security checklist. Estimate operating burden with the automation cost model.