Benchmark2026

Agent frameworks, killed mid-run

The same four-step pipeline, killed with SIGKILL in step three under three frameworks. We counted how many steps each one repeats on resume.

Author
View on GitHub

Summary result

LangGraph 1 · LlamaIndex 2 · Agno 3

LangGraph

1 step repeated

Resumed with one repeated step after SIGKILL during step three.

LlamaIndex

2 steps repeated

Re-executed two steps on resume after the mid-run kill.

Agno

3 steps repeated

Re-executed three steps on resume after the mid-run kill.

Why we ran this test

Multi-step LLM agents rarely fail politely in production. Containers get evicted, workers run out of memory, and processes receive hard kills mid-execution.

When an agent pipeline calls external APIs, runs database queries, or spends tokens across several stages, how a framework checkpoints and resumes after an abrupt crash directly affects reliability and cost.

Benchmark design

We implemented the same four-step pipeline across three agent frameworks: LangGraph, LlamaIndex, and Agno.

During execution of step three, each process was terminated abruptly with SIGKILL. On restart, we counted how many steps each framework repeated before finishing the pipeline.

Observed results

Under the exact same SIGKILL interruption in step three, the three frameworks showed distinct recovery behavior:

  • LangGraph repeated 1 step on resume.
  • LlamaIndex repeated 2 steps on resume.
  • Agno repeated 3 steps on resume.

How this shapes our production agent stack

Measuring crash recovery directly informs how we design production agent architectures—pairing LangGraph planner graphs with typed MCP tools and per-call Langfuse tracing so retries, failure boundaries, and per-run costs stay explicit.

The full benchmark code and reproduction setup are published openly on GitHub.

Open repository

Inspect the full benchmark code and harness on GitHub.

Open GitHub repo

Have a problem AI should solve?

Tell us what the data looks like and what “working” would mean for you. One of the founders will reply.