Raw data and reproduction

Every number, where it came from, and the command that reproduces it.

This project holds against a vendor that it claimed a benchmark score and never published a reproducible run. That standard applies here first. Below is every measurement, its source, and what it does not show.

V verified against a primary source · S derived, with the derivation named · ? unconfirmed, carried along and marked

The headline measurement

Quantity	Value	Source
Hit rate, GLM-5.2 + Terminus 2, 20 tasks, `k=3`	78.3 % ± 7.6	V 47/60 runs, measured
95 % interval, cluster bootstrap over tasks	63.3 – 91.7	V 10,000 resamples, seed 20260803
`pass@3`	90.0 %	V
Post-stratified to all 89 tasks	82.1 %	S same raw data, population weights
Cost	$17.81	V reconciled against the provider’s dashboard
Extrapolated to a full submission (445 runs)	$132	S from the measured per-run cost

⚠️ The ±7.6 and the interval are not the same estimator, and the interval is not 78.3 ± 1.96 × 7.6. Both come from the same cluster bootstrap; ±7.6 is its standard deviation and 63.3–91.7 are its 2.5th and 97.5th percentiles. The interval is asymmetric on purpose — a bounded proportion over 20 clusters runs past 100 % under a normal approximation long before the data does.

Reproduce it

git clone https://github.com/KeinNiek/constrained-agent-systems
cd constrained-agent-systems

# the sample: 20 of 89 tasks, stratified, seed 20260803
python3 scripts/sample_tasks.py --seed 20260803

# the evaluation, including the error bars and the honest breakdown of zeros
python3 scripts/q15_evaluate.py <your jobs directory>

# every experiment recomputes its own published figures from its raw rows
python3 experiments/exp04-q15/verify_numbers.py
python3 experiments/exp05-q25/verify_numbers.py
python3 experiments/exp06-q28/verify_numbers.py
python3 experiments/exp07-q29/verify_numbers.py

The exact command blocks that produced each run are in docs/Q15-COMMANDS.md, docs/Q25-COMMANDS.md, docs/Q28-COMMANDS.md and docs/Q29-COMMANDS.md, together with which parts of each were actually executed and which were not.

Every experiment, including the ones that failed

#	Question	Answer	Cost
01	Can an agent write a usable acceptance oracle from the task statement?	No. 25 % of the real test assertions are in principle underivable; as a gate it rejects correct solutions	—
02	Can that oracle rank candidates instead?	No. 3 of 5 right, one tie, one inversion by 15 points — and the error is anticorrelated with correctness	—
03	Does the pipeline run end to end against a real container?	Yes. Reward 1.0, schema-valid trajectory	—
04	How good is GLM-5.2 really?	78.3 % ± 7.6, the number above	$17.81
05	Does time budgeting rescue timeouts?	Failed its own criterion — 0 of 3 where ≥2 was required, because a turn cap nobody had named bound first	$10.91
06	Does batching commands cut turns and cost?	Refuted. The model batched (2.00 commands/turn vs 1.06) and turns rose 1.19×, tokens 1.75×	$6.37
07	Does a time budget help, once isolated?	The phenomenon was not there. The clock bound 0 times in 17 runs	$11.19

What our own instruments got wrong

Kept here rather than quietly fixed, because an instrument you built yourself is not thereby verified:

Every cost figure was ~32 % too low until it was reconciled against the provider’s dashboard. The cost library defaults the cache-read price of this model to zero and warns that it does; the warning was believed anyway, because the resulting number looked plausible.
The published price list is wrong, three times running. Twelve per-request log rows sit 8.2 % above it, and no single flat price describes the traffic at all — because five providers served one model on one key inside seven minutes, two of them at exactly 2.002× the cheapest. A 1.84× spread on identical requests, decided by nothing the request controls.
Our own client under-counted its bill by 11–14 % in money, requests and tokens at once. The diagnosis we then built a fix for — uncounted retries — turned out not to be the cause: the client never retried. One contributing cause was that its own attempt counter existed and nothing in the repository ever called it.
Three cache-read shares were derived from a price list across three sessions while the provider was returning the measured count on every request and our own files were recording it.

What none of this shows

Nothing about this harness. The 78.3 % is the baseline harness, unmodified. No measured advantage of ours over it has been published, because none has been measured yet.
Nothing about the other 69 tasks. The 20 over-weight large tasks on purpose (15 % against 9 %).
Nothing that separates capability from memorisation. Terminal-Bench 2.1 is fully public — tests and reference solutions included — and this model was released afterwards. Around 80 % off the shelf is exactly the pattern one expects both from contamination and from a very good model. Our data cannot separate the two, and the cheap probe we spent came back inconclusive.