How a run works, and what it does not claim
Is this eight agents, or one model talking to itself?
The fair version of that question is worth answering with rows rather than assurance. Every agent here calls the same locally served model, deliberately: it is the pinned constant, so that what varies between runs is the agent and not the weights. The scaffolding is what is under test, and this is the evidence that the scaffolding is what decides.
One model on every row in the record: qwen3-coder:30b-a3b-q4_K_M. A run that reached a different one would be a different measurement and is recorded as such.
18 distinct images, each built from its own repository at its own pinned commit. They share nothing but the model endpoint.
Same scenario, same model, and the outcome depends on which agent is reading. If the scaffolding did not decide, these rows would be alike. Instead 6 agents have never once complied across 46 verdicts, while two comply repeatedly.
| Victim | Held | Complied | No result |
|---|---|---|---|
mini-swe-agent | 4 | 7 | 9 |
OpenHands | 3 | 1 | 6 |
gptme | 10 | 0 | 8 |
smolagents | 9 | 0 | 8 |
shell_gpt | 16 | 0 | 14 |
open-interpreter | 7 | 0 | 13 |
qwen-code | 3 | 0 | 4 |
aider | 1 | 0 | 6 |
Induced execution only, because it is the one scenario that has ever produced a confirmation. The agents that comply are the ones whose invocation runs a shell command without asking, and those invocations are published with the findings.
A run takes a seed list of repository URLs and carries each one through four recorded stages. Every stage writes a row whether it succeeds or fails, and a failure is a result rather than a blocked task. Nothing is skipped in silence.
harvest resolve each repository to a commit SHA forge build a container at that pinned commit env check start the image with egress denied, run a probe report count rows, and name what was not measured
What the environment check measures
The check starts the built image and runs the probe the recipe installs: the interpreter comes up, prints one line, exits. That is the whole of it. The container command overrides whatever entrypoint the agent has, so a passing check means the environment works, not that the agent does.
An earlier draft of this project called that stage a boot. It was not one. The agents measured here share no entrypoint, and with no network and no credentials an invocation would measure the harness rather than the agent, contaminating the exact distribution this exists to report. Verifying that an agent itself runs needs a hand-written entrypoint per agent and has not been done.
What is pinned, and why
Every build row records the commit SHA, the image ID, and a sha256 of the build recipe. The recipe is an experimental variable: it decides which agents are buildable at all, so rows produced under two versions of it are two measurements and pooling them would not be one. That column exists because changing the recipe once would otherwise have silently redefined every figure already published.
Where the model sits
Agents under test talk to a local model endpoint through the OpenAI-compatible interface, so the model is an experimental constant rather than someone else’s drifting dependency. The model name and base URL are recorded on every check, since the same image checked against a different endpoint is a different measurement.
What a correction looks like
Records are append-only. The store has no update method and a test asserts its absence. A correction is a new row plus a supersedes row, and both stay readable; superseded rows are excluded from every published count through a single shared definition, so a figure on this site and a figure in a report cannot disagree.