Enumerative Repair Protocol
The one idea you need
One benchmark family — fix-the-broken-procedure episodes — has resisted every teaching attempt. All of those attempts taught the same thing: look at the evidence and INFER which step is wrong. That skill never installed at any dose. But two things the project has proven do work: this model is good at brute-force-style stepping, and it reliably learns turn-by-turn protocols. The family's episodes also give free feedback after every proposed fix. So this experiment teaches the opposite of cleverness: work through every legal one-step change in a fixed, documented order, skip the ones already tried, and propose the next one — letting the episode's own feedback find the winner.
The question
Can 160 lessons teach the model to propose candidate fixes one at a time in a fixed order — and does that discipline finally score on the fix-it benchmark family?
What we found
A split with the sharpest teaching contrast yet. On the fresh exam, the trained model produced the next-in-order untried candidate on 9 of 40 puzzles while BOTH untrained comparison models scored exactly zero — nobody does systematic enumeration without being taught it. But 9 of 40 (22.5%) fell short of the 50% bar the plan froze in advance for claiming the skill was solidly installed, and at the sealed benchmark the debugging family stayed at zero for the trained model — while, in a twist, the untrained control lucked into one item. By the pre-written rule the verdict is failed-on-its-own-terms: this teaching route closes at this dose. The diagnosis is precise, though: half the exam answers never even parsed (the long prompts break the answer format), and where the model got a legal untried candidate out, its ordering was right 56% of the time — the discipline half-took; the plumbing failed first.
Why it matters
This is the last family where the fully documented model scores nothing. If a taught enumeration routine converts, the walled inference skill was never needed — search discipline plus environment feedback substitutes for insight, which would be a new installable mechanism class. If it installs on its own exam but still scores zero on the family, the wall moves to episode integration, and the per-proposal quality readout (legal? untried? in-order?) says exactly which piece failed.
On this page
Results at a glance 1
How to read
Four bars per model: answers that parsed, were legal, were untried, and were exactly next-in-order.
Takeaway → The untrained models are at or near zero on every tier; the trained model leaks half its attempts at the parsing step, then orders correctly 56% of the time.
Data table
| fidelity tier (of 40 rows) | zero_root_parent | replay_ctl6 | enum_repair (trained) |
|---|---|---|---|
| parseable | 4 | 9 | 19 |
| legal | 4 | 5 | 18 |
| legal+untried | 2 | 3 | 16 |
| canonical-next | 0 | 0 | 9 |
Numbers from experiments/qwen35_4b_enumerative_repair_protocol/runs/local/seed88052_promotion.json
Technical framing
Enumeration fidelity cascade on the 40-row holdout (seed 88052) — PILOT_NOT_PROMOTED + FAILED_ON_ITS_OWN_TERMS: the discipline installed with the program's starkest contrast (canonical-next 9/40 vs 0/40 for both controls) but at 22.5% fidelity — below the frozen 0.50 precondition — and did not convert at sealed 78162 (candidate menders 0.0; the untrained replay control drew 0.1; pilot 0.3252 lost to both controls at 0.3502). The bottleneck is long-prompt answer formatting (parseable 19/40); ordering discipline 56% once legal-untried. The pure-enumeration SFT route closes at this dose per the preregistered ordered rule; protocols remain the installable class (5-for-5) with conversion the separate wall.
In the author’s words from the Overview · “Results”
Local gate: PROMOTED — the starkest mechanism contrast recorded: enum_repair 9/40 canonical-next on fresh instances versus BOTH controls at exactly 0/40 (fidelity cascade: parseable 19/40, legal 18, untried 16, canonical-next 9 — the bottleneck is answer formatting on long prompts; ordering discipline 9/16 = 56% once legal-untried); retention 57.67 pooled vs 59.33/59.67, deep in-band. Sealed event at 78,162 (all arms authenticated; the normalized pin held through the fill) (table on the experiment page). Pilot: candidate > base only — NOT promoted. … Read the full result →
Overview
Research Program
- Program:
agentic_breadth_installation - Program question: can DESIGNED synthetic curricula install universal, transferable agentic skills into the one 4B — provably, on fully documented lineage?
- Prior anchors: lifecycle 22 (
qwen35_4b_zero_root_lineage_rebuild— the zero-root composite, tree414f5829…), lifecycle 24 (qwen35_4b_menders_dose_scale— the 10× dose closed the dose-scale mechanism class for the eliminative-inference lesson), and lifecycle 25 (qwen35_4b_clean_gym_mix_dose— mixture dilution hardened the design rule: ONE KIND PER DOSE AT FULL CONCENTRATION).
Question
Lifecycle 26 — the new-mechanism attack on the last goal-gating family (menders). Every failed menders dose taught the model to INFER the right fix (eliminative inference — closed at every dose 80–800; even 2AFC verification sat at chance). Can a dose that teaches SYSTEMATIC ENUMERATION instead — given failure evidence, propose the legal single-step candidates one per turn in a frozen canonical order, let trial feedback decide, stop at first success — install on the axis holdout and convert to the menders family, which is a bounded multi-turn episode WITH rerun feedback?
Hypothesis
The repair kill rules bind the INFERENCE mechanism, not this one. C34 says brute-force search dominates the model's reasoning (a model-level law), and protocols are the line's only reliably installable class (hygiene/explore/termination/statechain). An enumerator converts turn budget into coverage without needing the walled inference skill; a taught frozen-order enumeration protocol should therefore install where taught inference did not, and any candidate menders > 0 where the controls sit at 0 is the mechanism answer.
Setup
- Model: Qwen/Qwen3.5-4B (revision
851bf6e8…), always. - Parent and adapter base: the zero-root composite (
large_artifacts/qwen35_4b_zero_root_lineage_rebuild/merged/zero_root_hygiene_explore, tree414f5829…, weights6e9aad25…), authenticated against lifecycle 22's committed merge receipt (e906caea…; byte-identical provenance copy indata/lineage/provenance/merge.json). - Treatment:
data/sft_enum_repair.jsonl— 160 rows, ONE KINDu_enum_repair, at construction seed 77,190 (shac9b539bf…), 20 rows per formalism across all eight machine formalisms REUSED from the menders dose-scale cell via a byte-identical machinery copy (scripts/gen_feedloop_curriculum.py— imported, never forked). Each row is one PARTIAL enumeration episode in ONE user message: the machine spec with legality clauses PLUS a numbered action list rendering the full bounded grammar in its frozen order; the broken written sequence; both trials' wanted+observed failure evidence; a frozen canonical-order statement (byte-identical in every row: step number ascending, then action-list position); the firstkcanonical candidates already tried, each with its observed two-trial outcome (all failures by construction;kcycles over 0/1/3/6/10); and the ask: name the NEXT untried legal candidate. The think target narrates enumerate → cross off → emit next. The answer is exact-matchSTEP <k>: <corrected step>. The generator verifies per row, by exhaustive re-derivation over the full candidate space, that the target IS the canonical-next untried legal candidate, that exactly ONE candidate repairs both trials, and that every tried entry is legal, canonically ordered, and genuinely failing (each re-simulated against both trials). - Surfaces: ALL inherited from the menders cell by design (no fresh-surface claim); freshness is at the ROW level — zero canonical-user-message overlap against 76 pinned predecessor corpora, streams, and gates (including the formalism-sharing menders corpus/holdout), plus fresh instances by construction.
- Arms:
replay_ctl6(control, trains FIRST) thenenum_repair(candidate); fresh rank-32/alpha-64 adapters, NO warm start, training seed 83, standard recipe (1 epoch over 1,520 rows, 190 optimizer updates, LR 1e-5, batch 1×8, max length 4,096, w_think/w_close 0.2). - Exposure: exact zero-delta three-axis MILP (forward / nonzero-target / absolute loss mass ×5) at the frozen geometry — 1,280-row shared stratified core + 240-row variable block (control: 240 replay; candidate: 160 treatment + 80 fillers), namespace seed 55,170. Matched exactly: 1,436,178 forward / 572,724 nonzero / 629,552 mass×5 per arm; zero skips; 1,280 aligned core rows.
- Local gate (three arms: parent + both trained): 40-row axis holdout at seed 88,052 (all
u_enum_repair, 5 per formalism, one per k-value; same invariants as the treatment) + three 104-row retention screens at 88,053/88,054/88,055 under pooled_k3. All seeds verified grep-fresh in seed contexts; no substitution was needed (88,043/88,047/88,049 remain taken by the counterfactual cell and are avoided). Promotion: axis total strictly > parent AND > replay_ctl6 (ties fail) — single kind, so NO per-kind split exists; pooled retention bands on screen sums (correct −15, caps +9, parsed −9) vs both controls. - Preregistered NON-GATING mechanism readings: (a)
episode_success_simulation— the analytic perfect-enumerator turn count per holdout episode (recorded in the local design receipt, model-free). THE NUMBERS, quoted up front because they scope the transfer bet: holdout from-scratch MEAN 30.4 turns (median 18.5, max 122; 87.5% of episodes need MORE than 10 turns); treatment corpus mean 31.4 (median 23, max 125; 88.1% > 10) — against a family episode budget publicly known only as "bounded". Stated plainly: if the family's budget is materially shorter than these needs, a perfectly-installed enumerator converts few or no episodes. (b)enumeration_fidelity— per axis row at eval time, three booleans about the model's proposal: LEGAL, UNTRIED, CANONICAL-NEXT — a mechanism decomposition beyond raw correctness, summarized per arm; it feeds the frozen zero-draw scoping below. - Conditional benchmark (only on promotion): ONE sealed medium tb1024 event at fresh seed 78,162, four arms in frozen order — base (
26d8ee48…/b654e033…), zero_root_parent (414f5829…), replay_ctl6, enum_repair. Trained-arm pins are six fail-closed TODO-PIN slots inscripts/run_benchmark.py, frozen by check_design's NORMALIZED-HASH pin. Pilot gate: candidate aggregate strictly > base AND > replay_ctl6 AND > zero_root_parent. Recorded either way: the goal gate, the per-family table, and THE MENDERS READING — candidate vs base and vs parent on menders specifically (frozen question: does taught enumeration convert to the family with live rerun feedback?). FROZEN ORDERED CONSEQUENCES, positive first, no third state for the zero draw: (1) ANY candidate menders > 0 where the controls sit at 0 is the mechanism answer; (2) a menders 0 WITH the fidelity precondition met (promoted locally AND holdout canonical-next rate F >= 0.50 AND F strictly above both controls' rates) is TURN_BUDGET_SCOPED — enumeration installed with high fidelity but did not convert within the family's episode budget; the protocol-install mechanism is NOT refuted; what closes is the pure-enumeration route at the family's actual budget; (3) a menders 0 WITHOUT that precondition reads as the install/conversion failing on its own terms. A 10/10 feeds a fresh confirmation cell. (The scoping and quoted simulation numbers were added pre-freeze by review amendment; no seed had been consumed.) - Standalone:
data/lineage/carries the complete clean-chain package — the six zero-root stage datasets, lifecycle 22's stage + merge receipts as provenance documents, the trainer/merger copies, and a clean-chain manifest recording this cell's dose as STAGE 7. NO blend root exists anywhere in this cell (fail-closed). - Hidden-label boundary: gate answers and per-row audits live only in
data/local_tasks_seed*.jsonl; the model-facinglocal_input_seed*.jsonlfiles carry id/messages/meta only. The benchmark suite directory is never read; only the trusted aggregate gateway runs.
Run
Smoke (no GPU, no writes):
PYTHONDONTWRITEBYTECODE=1 .venv/bin/python -B experiments/qwen35_4b_enumerative_repair_protocol/scripts/run.py --smokeFull (one stage per pushed checkpoint, each behind its review verdict):
.venv/bin/python -B experiments/qwen35_4b_enumerative_repair_protocol/scripts/run.py --stage train-control
# then: train-candidate, merge-arms, local, benchmarkStandalone lineage verification (no GPU) / full clean-chain rebuild (GPU):
.venv/bin/python -B experiments/qwen35_4b_enumerative_repair_protocol/scripts/rebuild_clean_chain.py --verify-inputsResults
Local gate: PROMOTED — the starkest mechanism contrast recorded: enum_repair 9/40 canonical-next on fresh instances versus BOTH controls at exactly 0/40 (fidelity cascade: parseable 19/40, legal 18, untried 16, canonical-next 9 — the bottleneck is answer formatting on long prompts; ordering discipline 9/16 = 56% once legal-untried); retention 57.67 pooled vs 59.33/59.67, deep in-band.
Sealed event at 78,162 (all arms authenticated; the normalized pin held through the fill):
| arm | aggregate | menders | goal gate |
|---|---|---|---|
| base | 0.0882 | 0.000 | — |
| zero_root_parent | 0.3502 | 0.000 | 8/10 (ties menders, rites) |
| replay_ctl6 | 0.3502 | 0.100 | 8/10 (ties rites, sirens) |
| enum_repair | 0.3252 | 0.000 | 7/10 (ties menders, sirens; loses rites) |
Pilot: candidate > base only — NOT promoted. The frozen menders rule: candidate_nonzero false, controls_all_zero false (the replay control drew an item), fidelity 22.5% < the 0.50 precondition → FAILED_ON_ITS_OWN_TERMS: the pure-enumeration SFT route closes at this dose, on its own preregistered terms.
Interpretation
Three readings. (1) The INSTALL is genuine and unprecedented in contrast: untrained models score literal zero at canonical-next enumeration and the dose lifted it to 9/40 — protocols remain the installable class (now 5-for-5 on installs). (2) The CONVERSION failed on its own terms: 22.5% local fidelity was too low to earn the budget-scoped reading, the family drew 0 for the candidate while the replay control (which trains nothing) drew an item — re-confirming that menders movement at this granularity remains draw-dominated for everything except a genuinely reliable installed skill, which this dose did not reach. (3) The identified bottleneck is upstream of the discipline: half the holdout rows never parsed into the answer format (long-prompt formatting), and where a legal untried candidate emerged the ordering was right 56% of the time. A formatting-targeted variant is a marginal iteration on a mechanism that just failed its preregistered terms — per calibrate-and-diverge it needs new evidence before funding, and the frozen consequence stands.
Knowledgebase Update
- Program evidence updated: pending results.
- Program backlog updated: pending results.
- Claim ledger updated: pending results.
Artifacts
src/— frozen vLLM runner (byte-identical to the lifecycle 25 cell's).scripts/— staged harness, the new enum-repair generator + the byte-copied menders machinery + canonical retention generator, corpus builder with audits, exposure pipeline, gate, benchmark runner with the menders reading, clean-chain rebuild script, vendored trainer/merger copies.configs/— frozen identity.data/— treatment corpus + manifest, replay copy, exposure streams + receipts, gate files, design receipts, clean-chain lineage package (data/lineage/).runs/— stage receipts (written by the staged GPU runs).reports/artifact_manifest.yaml
Report
Rendered from reports/report.md
Summary
The anti-cleverness bet closed on its own preregistered terms. The enumeration discipline INSTALLED with the program's starkest mechanism contrast — 9/40 canonical-next on fresh instances versus both controls at exactly 0/40, retention deep in-band — but at 22.5% fidelity (below the frozen 0.50 precondition), and at the sealed event it did not convert: candidate menders 0.0 while the untrained replay control drew an item (0.1), pilot lost to both controls (0.3252 vs 0.3502). The frozen ordered rule reads FAILED_ON_ITS_OWN_TERMS: the pure-enumeration SFT route closes at this dose. The mechanism ledger remains sharp — the bottleneck is long-prompt answer formatting (parseable 19/40), not the ordering logic (56% once legal-untried) — and protocols stay the installable class (5-for-5 on installs; conversion remains the separate, harder step).
Research Program Fit
Method
Results
Controls
Oracle Versus Deployable Evidence
Interpretation
Next Experiments
Artifact Manifest
The frozen corpus, streams, receipts, gate inputs, and the clean-chain lineage package are in-repo; trained adapters and merges will live in this cell's own artifact storage with hashes pinned in receipts and reports/artifact_manifest.yaml.
Experiment log 6
Show the running log (6 entries, 2026-07-16)
Scaffold
Created as a new experiment scaffold.
2026-07-16 — Model-free design freeze (lifecycle 26)
- The new-mechanism attack on the last goal-gating family: every failed menders dose taught eliminative INFERENCE (closed 80–800 rows, 2AFC at chance); this cell teaches SYSTEMATIC ENUMERATION — name the next untried legal single-step candidate in a frozen canonical order and let trial feedback decide (C34 + the protocol-install law).
- Vehicle per the freshly hardened dilution rule: ONE KIND (
u_enum_repair) at full 160-row concentration, 20 rows per formalism across the eight machine formalisms reused from the menders dose-scale cell via a byte-identical machinery copy (imported, never forked); the lesson changes completely. Every prompt renders the full numbered action list and a byte-identical canonical-order statement, so the enumeration is entirely in-prompt; tried prefixes cycle k over 0/1/3/6/10; per-row exhaustive re-derivation verifies the target is the canonical-next untried candidate, the both-trials fix is unique, and every tried entry genuinely fails. - Frozen corpus sha
c9b539bf…(manifest975b982e…); zero row overlap against 76 pinned predecessor sources (the menders cell shares all eight formalisms — row-level freshness is the bar); no fresh-surface claim (all vocabulary inherited by design). - Exposure: exact zero-delta three-axis MILP at namespace 55,170 — 1,436,178 forward / 572,724 nonzero / 629,552 mass×5 per arm, 1,280 aligned core rows, zero skips. Streams
replay_ctl6(3e1f79a4…) andenum_repair(a1d4a8f5…); token receipt954386b2…. - Gate frozen at seed 88,052 (40 rows, 5 per formalism, one per k-value) + pooled_k3 screens 88,053/88,054/88,055; single-kind promotion (axis total strict over both controls, ties fail; standard −15/+9/−9 pooled bands). Two preregistered NON-GATING mechanism readings: the analytic episode_success simulation (perfect-enumerator turns per holdout row, recorded in the design receipt) and the eval-time enumeration-fidelity booleans (legal/untried/canonical-next per axis row). All seeds (77,190/55,170/83/88,052–88,055/78,162) verified grep-fresh in seed contexts; no substitution needed (88,043/88,047/88,049 remain taken by the counterfactual cell and are avoided); training seed 83 continues the paired-trial sequence 67/71/73/79/83 with only pre-program, non-lineage hits recorded.
- Conditional benchmark: medium tb1024 at sealed fresh seed 78,162, four arms, six-slot normalized-pin runner (normalized sha
4f29bd26…), THE MENDERS READING recorded either way with the frozen power statement (any candidate menders > 0 where controls sit at 0 is the mechanism answer). - Clean-chain package copied byte-identically (six zero-root datasets + lifecycle 22 provenance receipts + trainer/merger copies), this cell's stage 7 recorded in the lineage manifest (
3f5ab017…); blend root fail-closed absent. 147 tests green;run.py --smokegreen end to end; every write-mode re-run and guarded stage refuses (boundary drills recorded).
2026-07-16 — Review amendment (pre-freeze): the zero draw gets scoped
- Honest ordering note: added PRE-FREEZE after the adversarial review flagged the one MAJOR — the preregistration froze only the POSITIVE consequence (menders > 0 = mechanism answer) while the cell's own analytic simulation shows the modal menders = 0 draw is ambiguous. No model event had run and no seed had been consumed at amendment time.
- The simulation is now QUOTED prominently (preregistration + README + the frozen power statement): a PERFECT canonical enumerator needs mean 30.4 turns from scratch on the holdout (median 18.5, max 122; 87.5% of episodes need > 10 turns); treatment corpus mean 31.4 (median 23, max 125; 88.1% > 10) — against a family budget publicly known only as "bounded". If that budget is materially shorter, a perfectly-installed enumerator converts few or no episodes.
- NEGATIVE SCOPING frozen as an ordered rule beside the positive one, keyed to the preregistered fidelity readout with an a-priori numeric precondition (F = candidate canonical-next rate on the 40-row holdout; HOLDS iff promoted AND F >= 0.50 AND F strictly above both controls): (1) any candidate menders > 0 with controls at 0 = MECHANISM_ANSWER (precedence); (2) menders 0 WITH the precondition = TURN_BUDGET_SCOPED (enumeration installed with high fidelity but did not convert within the family's episode budget; the protocol-install mechanism is NOT refuted; what closes is the pure-enumeration route at the family's actual budget); (3) menders 0 WITHOUT it = failed on its own terms. No third state for the zero draw.
- Implemented in
run_benchmark.py(fidelity_precondition,menders_reading.frozen_interpretation, integer-exact comparisons, the two-direction power statement) — inside the normalized-hash pin, re-frozen atb8c4b5d9…; design receipt regenerated (edba3763…,--checktwice byte-identical; the local gate receipt is untouched — no pinned file changed). Tests extended for the scoping rule and the 0.50 boundary; suite and smoke re-run green.
2026-07-16 — Local gate: PROMOTED with the program's starkest contrast
- The 12-run gate promoted enum_repair on all eight checks: axis 9/40 versus BOTH controls at 0/40 — untrained models cannot produce the canonical-next candidate at all; retention 57.67 pooled vs 59.33/59.67 (deep in-band). Fidelity cascade: parseable 19/40 (the bottleneck is answer formatting on long prompts), legal 18, untried 16, canonical-next 9 (56% ordering discipline once legal-untried).
- Honest pre-event note: 22.5% fidelity is below the frozen 0.50 precondition, so a zero menders draw will read FAILED_ON_ITS_OWN_TERMS per the preregistered rule; MECHANISM_ANSWER (menders > 0, controls 0) takes precedence if it fires. Pins filled; the normalized hash held; PASS_BENCHMARK_EVENT granted; sealed 78,162 opens at the next green checkpoint.
2026-07-16 — The sealed event and closure
- All four arms ran clean at 78,162. Aggregates 0.0882 / 0.3502 / 0.3502 / 0.3252 (base / parent / replay / candidate): pilot NOT promoted (beat base only). The frozen menders rule fired FAILED_ON_ITS_OWN_TERMS: candidate menders 0.0 with fidelity 22.5% below the 0.50 precondition — and the replay control drew a menders item (0.1), denying even the clean zero-contrast. The pure-enumeration SFT route closes at this dose on its own preregistered terms.
- The mechanism ledger stays valuable: the install contrast (9/40 vs 0/40) is the program's starkest; the bottleneck is long-prompt answer formatting (parseable 19/40), not the ordering discipline (56% once legal-untried). Any successor needs new evidence per calibrate-and-diverge; seed 78,162 spent.
2026-07-16 — Post-closure truncation forensics (analysis only)
- The committed gate generations show 20 of the 21 unparseable axis rows are 1,024-token cap truncations caught mid-CORRECT canonical walk — the 22.5% fidelity undercounts the installed discipline; the failure is expression cost (a verbose linear walk), not the ordering logic. Recorded in analysis/truncation_forensics.md as the new-evidence basis for a count-don't-walk successor (index arithmetic over the numbered list, constant-cost in k). The frozen verdict stands unchanged.
Data files 1
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
Reproduce
Smoke test
PYTHONDONTWRITEBYTECODE=1 .venv/bin/python -B experiments/qwen35_4b_enumerative_repair_protocol/scripts/run.py --smokeFull run
.venv/bin/python -B experiments/qwen35_4b_enumerative_repair_protocol/scripts/run.py --stage train-control # then train-candidate, merge-arms, local, benchmarkRun steps are documented inside the experiment folder (README and scripts).