Benchmarks for agentic infrastructure

Agents are starting to run real infrastructure. Not just write it, but answer questions about it, find what is broken, and change it. So the tool an agent holds matters as much as the model.

This site collects benchmarks that measure that, and publishes every run.

Benchmarks

• aws-bench

  An open benchmark for AI agents working on AWS. Defines estates, the questions to ask about them, reference answers, and a judge.

  Scenario: ec2-multiregion. Six EC2 instances across three regions, eight questions about reachability, placement, and what is unused.

  Results

That is the only one so far. Others get added as they are published, and the shape of the results does not change, which is what lets a new one slot in.

Whose work this is

Each benchmark defines its own estates, questions, reference answers and judge. That work belongs to whoever published it. What gets added here is a fork that runs it on an emulator, one deployment per toolchain, and the gates that decide whether a run counts.

See Method for how the arms are kept comparable, or Run it yourself to reproduce any of it.