Agent frameworks: what to compare

The questions that separate agent frameworks

Compare agent frameworks on five things: how explicitly you control the flow of steps, how state is kept across steps and sessions, how tools are defined and whether MCP servers plug in, how tied the framework is to one model provider, and what you can see when a run fails. Language support and licence decide the shortlist; those five decide the choice.

The agents layer holds the wider set (live products on record: 252) and the agent framework category narrows it to the frameworks. None is ranked here. A framework that suits a research prototype can be the wrong one for a support agent that must behave the same way every Tuesday.

Before comparing any of them, write down the task in plain steps. A task whose steps can be listed, with only a few needing judgement, calls for a framework that lets you write that list and call the model where judgement is needed. A task whose steps cannot be known in advance calls for more autonomy and tighter limits. Many business tasks are the first kind.

Control flow: graph, loop or plain code

Frameworks sit on a line from let the model decide everything to you write every step. At one end the model picks the next tool in a loop until it says it is done. At the other you define a graph of steps with the model filling in decisions at chosen points. Between them sit frameworks that treat agents as ordinary functions you compose.

The loop is quick to demo and hard to bound. A typical failure is not a wrong answer but a run that keeps calling the same search tool with slight variations until it hits the token limit. An explicit graph makes that run harder to write by accident.

Ask whether you can set a limit on steps, tokens and time per run, and what happens when one is hit. Ask whether a person can be put in the flow to approve a step. Those two features matter more in production than the number of agent patterns in the documentation.

State, memory and resuming a run

Ask where state lives between steps and whether a run can be paused and resumed. A long task that fails at step nine should restart at step nine, not step one, and a task waiting for human approval may wait for hours. A framework that writes state to a store you choose makes that possible; one that keeps it only in process memory loses it when the process stops.

Memory across sessions is a separate question. Some frameworks offer it built in, some leave it to you. Either way, it is data about users stored somewhere, with the same questions about retention and deletion as any other store.

Ask how state is versioned when you change the agent. A run started under last week’s graph and resumed under this week’s can fail in ways that are hard to reproduce. Frameworks that record which version a run started on, and can finish it on that version, save a class of bug you will otherwise meet in production first.

Tools, MCP and the model underneath

An agent is only as useful as the tools it can call. Ask how a tool is defined, whether the schema is generated from your code or written by hand, and how errors from a tool are returned to the model. A tool that fails silently teaches the agent nothing.

Ask whether the framework can use MCP servers, the protocol many tools are now published with. The directory reads each MCP server from its own repository, with what it reaches and which credential it wants (servers on record: 8,106), and lists agent skills separately (skills on record: 3,022). A framework that speaks MCP can use them without a wrapper per tool; check the MCP server directory before writing an integration yourself.

Then ask how tied it is to one model provider. A framework that works with any model through a common interface lets you swap to a cheaper or better model later. One built around a single provider’s features may be simpler today and harder to leave.

Check how it handles the model refusing, returning malformed tool arguments, or running past its context window mid-task. These are ordinary events in a long run, and a framework that treats each as a crash will page someone at night for problems a retry would have solved. Ask to see the error handling in the source, not only in the documentation, because the two drift apart.

Seeing inside a run

When an agent does something odd, you need the full tree of what happened: each model call with its prompt, each tool call with arguments and result, and the order they ran in. Ask whether the framework emits traces in a standard format your observability tool can read, or only into its own dashboard.

Then run a small, ugly test before committing. Give each candidate the same task with a tool that sometimes errors and an input that tries to change its instructions, and read the traces. How a framework behaves on the bad path tells you more than its tutorial does on the good one.

Count what you had to write outside the framework to make the test pass: retries, timeouts, logging, a wrapper around a tool. That code is the real cost of the choice, and it is invisible in a feature table. The framework that needed the least of it for your task is a better sign than the one with the longest list of integrations.