Agent frameworks, killed mid-run
The same four-step pipeline, killed with SIGKILL in step three under three frameworks. We counted how many steps each one repeats on resume.
- DLDuy Le(Co-founder · CEO)
Summary result
LangGraph 1 · LlamaIndex 2 · Agno 3
LangGraph
1 step repeated
Resumed with one repeated step after SIGKILL during step three.
LlamaIndex
2 steps repeated
Re-executed two steps on resume after the mid-run kill.
Agno
3 steps repeated
Re-executed three steps on resume after the mid-run kill.
Why we ran this test
Multi-step LLM agents rarely fail politely in production. Containers get evicted, workers run out of memory, and processes receive hard kills mid-execution.
When an agent pipeline calls external APIs, runs database queries, or spends tokens across several stages, how a framework checkpoints and resumes after an abrupt crash directly affects reliability and cost.
Benchmark design
We implemented the same four-step pipeline across three agent frameworks: LangGraph, LlamaIndex, and Agno.
During execution of step three, each process was terminated abruptly with SIGKILL. On restart, we counted how many steps each framework repeated before finishing the pipeline.
Observed results
Under the exact same SIGKILL interruption in step three, the three frameworks showed distinct recovery behavior:
- LangGraph repeated 1 step on resume.
- LlamaIndex repeated 2 steps on resume.
- Agno repeated 3 steps on resume.
How this shapes our production agent stack
Measuring crash recovery directly informs how we design production agent architectures—pairing LangGraph planner graphs with typed MCP tools and per-call Langfuse tracing so retries, failure boundaries, and per-run costs stay explicit.
The full benchmark code and reproduction setup are published openly on GitHub.
Open repository
Inspect the full benchmark code and harness on GitHub.