Benchmarks for agentic infrastructure
Agents are starting to run real infrastructure. Not just write it, but answer questions about it, find what is broken, and change it. So the tool an agent holds matters as much as the model.
This site collects benchmarks that measure that, and publishes every run.
Benchmarks
-
aws-bench
An open benchmark for AI agents working on AWS. Defines estates, the questions to ask about them, reference answers, and a judge.
Scenario: ec2-multiregion. Six EC2 instances across three regions, eight questions about reachability, placement, and what is unused.
-
Questions we added
One question on that board splits the arms far wider than the other seven: which security groups are unused. Every arm holding a state file scores zero, and an agent with no tooling at all beats them.
Scenario: ec2-multiregion-negatives. Two more questions of that shape, on the same estate. Ours rather than aws-bench's, over 6 trials rather than 24, so the numbers are kept off the board.
-
terralith
No agent, no model, no questions. How much a plan costs as a stock-Terraform estate grows, for choudoufu and for chant, against stock Terraform as the oracle — one certification run per arm per size.
Scenarios: terralith-79, terralith-745, terralith-3705, terralith-10069. A table, not a leaderboard: sizes down the side, arms across.
aws-bench and terralith are the published benchmarks here so far. Others get added as they appear, and the shape of the results does not change, which is what lets a new one slot in.
Whose work this is
Each benchmark defines its own estates, questions, reference answers and judge. That work belongs to whoever published it. What gets added here is a fork that runs it on an emulator, one deployment per toolchain, and the gates that decide whether a run counts.
See Method for how the arms are kept comparable, or Run it yourself to reproduce any of it.