Medium Budget-Probe Measurement
The one idea you need
Three different teaching methods have now failed at the same benchmark family — the debugging-style episodes where the model must use feedback across turns. Every remaining training idea therefore tops out at nine of ten families. But all previous measurements gave the model a tight thinking allowance, and the two stuck families are exactly the ones needing long multi-step reasoning. This experiment re-runs the same four already-built models on the medium benchmark with eight times the thinking room: if the stuck families move at all, the goal is still fully reachable and the venue just changes; if nothing moves, the wall is real and the next dose proceeds with honest expectations.
The question
With eight times the thinking allowance, do the two zero-score families finally move for any existing model — is the ten-family goal still reachable?
What we found
The probe never got to ask its question. The benchmark's own referee enforces a wall-clock budget per model, and the untouched base — deliberately sent first because this risk was written into the plan — blew past it with eight times the thinking allowance and was refused without a score. Zero extra compute was wasted, the sealed seed is recorded as spent, and the rule book says no retry at a lower budget in this experiment. What survives: the question of whether more thinking time moves the stuck families is still open at in-between budgets, and one more probe there is the last believable test of this lever.
Why it matters
This is the cheapest possible test standing between the program and a decision about its goal: fund the next training dose knowing its ceiling is nine, or discover the ceiling is still ten and redirect the venue first.
On this page
Results at a glance 1
How to read
Each bar is a preregistered stage; one means complete.
Takeaway → Design, review, and freeze completed; the event stopped at the first arm exactly as the preregistered stop outcome specified.
Data table
| preregistered experiment stage | completed checkpoint |
|---|---|
| intake | 1 |
| design receipt frozen | 1 |
| adversarial review + fixes | 1 |
| 4-model tb8192 event | 0 |
Technical framing
Preregistered stage completion — BUDGET_GATE_STOP: the trusted gateway's hard wall-budget gate refused base at medium/tb8192 before any treated arm ran (safe diagnostic budget_gate_failed; no score emitted; seed 78152 spent by the write-ahead ledger record). The 8x thinking-budget lever is infeasible for paired events at medium; the budget-movement question survives only at intermediate budgets (tb2048-4096), where one further preregistered probe is the lever's last believable test.
In the author’s words from the Overview · “Results”
The preregistered stop outcome fired on the first arm: the gateway refused base at medium/tb8192 with safe diagnostic budget_gate_failed (exit 2, no score emitted, no raw output exposed). Per the frozen order (base first, precisely to minimize spend on this known risk) zero treated arms ran; the write-ahead ledger's opened record marks seed 78,152 spent; the failure receipt is preserved at runs/benchmark/medium_tb8192_seed78152_measurement/base.failure.json. Per the frozen contract there is no retry and no lower-budget re-run inside this directory.
Overview
The last cheap lever on the binding constraint: the same four published composites as the seed-78,150 medium event, re-measured once at think budget 8,192 — does thinking room alone move menders (zero for every arm at tb1024) or rites (zero for all but designed_fresh) off their floors, i.e. is the goal-gate ceiling 9 or 10?
Research Program
- Program:
agentic_breadth_installation. - Program question: can synthetic curricula install general capability that lifts the held-out aggregate without a negative family?
- Prior anchors: the two-tie install (repair kill rule extended to all tested SFT pedagogies — training paths cap at 9/10); the medium measurement (hygiene_explore at 8/10, ties at menders/rites); C44 (serial-compute limit; always give the 4B chain-of-thought); the budgets-maxed directive (caps throttle measured capability); all arms within wall budget at tb1024 (136–230 s).
Question
At medium/tb8192 on a fresh sealed seed, does menders (universally zero at tb1024) or rites (zero for three of four arms) move off its floor — and does the goal gate's reachable ceiling change?
Setup
- Arms (identical pins to the 78,150 event):
base,designed_fresh,replay_repeat,hygiene_explore. - Event: medium, think budget 8,192, sealed seed 78,152, hardened seed-boundary runner, one-seed write-ahead ledger.
- Readings (no promotion): budget movement (fires only for arm/family pairs at 0 in the pinned tb1024 event that turn positive; designed_fresh's already-nonzero rites reported descriptively); budget contrast vs the pinned 78,150 summary (implementation-signature verified equal fail-closed; seed+budget confounds labeled); recorded goal gate; within_budget integrity per arm.
Run
PYTHONDONTWRITEBYTECODE=1 .venv/bin/python -B experiments/qwen35_4b_medium_budget_probe_measurement/scripts/run.py --smoke
.venv/bin/python -B experiments/qwen35_4b_medium_budget_probe_measurement/scripts/run.py --stage benchmarkResults
The preregistered stop outcome fired on the first arm: the gateway refused base at medium/tb8192 with safe diagnostic budget_gate_failed (exit 2, no score emitted, no raw output exposed). Per the frozen order (base first, precisely to minimize spend on this known risk) zero treated arms ran; the write-ahead ledger's opened record marks seed 78,152 spent; the failure receipt is preserved at runs/benchmark/medium_tb8192_seed78152_measurement/base.failure.json. Per the frozen contract there is no retry and no lower-budget re-run inside this directory.
Interpretation
- The full 8× thinking-budget lever is closed at medium: the gateway's per-arm wall budget binds well below tb8192 for base (which passed comfortably at tb1024, 157 s). The budget-movement question (does serial-compute room move menders) remains open only at intermediate budgets.
- Pricing an intermediate successor honestly: base's tb1024 wall was 157 s and hygiene_explore's 230 s — the slowest arm, not base, may bind at higher budgets, so any successor must expect either to trip. One preregistered intermediate probe (tb4096 or tb2048, fresh seed, same stop contract) is the last believable shot at the lever; a second stop would close the budget lever entirely and fix the statechain successor's 9/10 ceiling as the program's honest position.
Knowledgebase Update
- Program evidence updated: the stop, its minimal cost, and the lever's remaining scope recorded.
- Program backlog updated: one intermediate-budget probe queued as the lever's last believable test; the statechain-only dose remains the funded training branch.
- Claim ledger updated: no.
Artifacts
data/design_receipt.json: seed/tier/budget/model/gateway/contrast-source pins.reports/preregistration.md,reports/benchmark_design_review.md: contract and authorization.
Report
Rendered from reports/report.md
Summary
The budget probe closed on its preregistered stop: the gateway's hard wall-budget gate refused base at medium/tb8192 (safe diagnostic budget_gate_failed) before any treated arm ran — the frozen base-first order spent the minimum possible compute on the known risk. The 8× thinking-budget lever is infeasible for paired events at medium; seed 78,152 is spent per the opened ledger record; the budget-movement question survives only at intermediate budgets, where one further preregistered probe is the lever's last believable test.
Research Program Fit
The two-tie install extended the repair kill rule to every tested SFT pedagogy; this probe tests the last cheap non-training lever on the binding constraint and prices the statechain successor honestly.
Method
See the preregistration.
Results
runs/benchmark/medium_tb8192_seed78152_measurement/base.failure.json: gateway exit 2, budget_gate_failed, score_emitted: false, raw_streams_exposed: false; ledger holds the single opened record; no treated arm consumed compute; no readout exists by construction.
Controls
Identical model pins to the reference event; within-event pairing on one seed; the cross-budget contrast explicitly labeled seed-confounded; budget-integrity flags recorded per arm.
Oracle Versus Deployable Evidence
Gateway aggregates and public family scores only; benchmarks/ never read.
Next Stage
Closed. Successor options priced in the README: one intermediate-budget probe (tb4096/tb2048, fresh seed, same stop contract) or accept the 9/10 ceiling and fund the statechain-only dose.
Artifact Manifest
Four composite pins external with committed receipts; everything else in-repo.
Experiment log 3
Show the running log (3 entries, 2026-07-15)
2026-07-15 — Model-free design freeze
- Opened after the two-tie install extended the repair kill rule: every believable training path caps the goal gate at 9/10, and thinking budget at the event is the last cheap untested lever on the binding menders constraint (C44 serial-compute; truncation cascade; the budgets-maxed directive; all arms within wall budget at tb1024).
- Frozen: the same four published composites as seed 78,150, medium tier, think budget 8,192, sealed fresh seed 78,152, hardened seed-boundary runner, four promotion-free readings (budget movement on menders/rites, seed-confounded budget contrast vs 78,150, recorded goal gate, budget integrity flags).
- No model event has run; nothing trains in this cell.
2026-07-15 — Delta review: two reading amendments pre-freeze
- The two-lens review confirmed two MAJORs, both fixed before any commit or event: (1) the movement premise was wrong — designed_fresh already scored rites 0.1 at tb1024, so the absolute >0 boolean could fire on a status-quo repeat; movement is now scoped to (arm, family) pairs at 0 in the pinned tb1024 event, with designed_fresh's rites reported descriptively; (2) the cross-budget contrast never verified benchmark- implementation equality across the two events — the readings now fail closed unless the new receipts' shared implementation signature matches the reference summary's block, with both signatures surfaced.
2026-07-15 — The event: preregistered stop at the first arm
- CI green on the freeze;
--stage benchmarkopened the ledger and ran base first per the frozen order; the gateway refused it withbudget_gate_failed(exit 2, nothing exposed). Zero treated arms ran. - Verdict BUDGET_GATE_STOP exactly as preregistered: failure receipt preserved, seed 78,152 spent, no retry, no lower-budget re-run in this directory. The 8× lever is closed; an intermediate-budget probe under a fresh seed is the lever's last believable test (noting hygiene_explore, not base, was the slowest arm at tb1024 — either could bind).
Reproduce
Smoke test
PYTHONDONTWRITEBYTECODE=1 .venv/bin/python -B experiments/qwen35_4b_medium_budget_probe_measurement/scripts/run.py --smokeFull run
checkpointed scripts/run.py --stage benchmark onlyRun steps are documented inside the experiment folder (README and scripts).