Qwen3.5-4B Neurosymbolic REPL Substrate + Failure Profile
The one idea you need
Each puzzle hides a rule that reshapes one list of numbers into another. Shown a few examples, the model writes Python to reproduce it, then runs the code — an honest calculator, not a bigger brain — to see its real output beside the right one and revise.
The question
If a small model can run its own code and see exactly where it went wrong, can fixing mistakes that way beat just trying again?
What we found
No. Seeing its real error barely helped: the fix-it loop solved 29% of puzzles versus 34% for simply drawing five independent attempts and keeping the best, at equal compute; the error message itself added just two tasks over a control told only to try again. The model's test-time ceiling is whatever its own sampling can reach. Retraining on its verified solutions, though, lifted single-shot accuracy from 22% to 32%.
Why it matters
For a small model, don't spend test-time compute on run-and-fix correction loops; they rarely beat simply sampling more against a checker. Instead fine-tune it on answers it can already verify — that permanently banks the gain into single-shot deployment.
On this page
Results at a glance 4
How to read
Four bars show the share of hidden puzzles solved: one attempt, the run-and-fix loop that reads its real error, a control told only to retry, and drawing five independent attempts then keeping the best. Taller is better.
Takeaway → Sampling five and keeping the best (34%) is tallest; the fix-it loop (29%) barely tops one attempt and the retry control, which both sit at 26%, so reading the real error buys almost nothing.
Data table
| condition | hidden accuracy |
|---|---|
| greedy@1 | 26.3% |
| REPL + real feedback | 28.7% |
| REPL, no feedback (control) | 26.3% |
| sample more + select | 33.8% |
Numbers from runs/repl_summary.json (matches report table)
Technical framing
Execution-feedback REPL vs matched-compute sampling (hidden-test accuracy) — Execution feedback does not beat independent sampling at matched compute; the feedback content adds only +0.024 over the no-feedback control.
How to read
The rising line plots hidden-puzzle accuracy as independent attempts per task grow from one to five with the best kept; two dots mark the fix-it loop and the retry control at about four attempts each. Higher is better.
Takeaway → Both loop dots sit on or below the sampling line at the same attempt budget, so those generations are better spent on fresh independent attempts.
Data table
| generations per task | sample more + visible-select | REPL + real feedback | REPL, no feedback (control) |
|---|---|---|---|
| 1 | 20% | — | — |
| 2 | 23.7% | — | — |
| 3 | 26.3% | — | — |
| 3.9 | — | 28.7% | — |
| 4 | 32.5% | — | 26.3% |
| 5 | 33.8% | — | — |
Numbers from runs/repl_summary.json (sample_more_selected_curve, arm acc/mean_gens)
Technical framing
Accuracy vs generation budget: sampling curve vs REPL arms — The REPL loop at ~4 generations sits on or below the independent-sampling curve — sampling is the stronger use of the same compute.
How to read
Two measures on fresh unseen tasks — single best attempt and best-of-five coverage — each pairing the frozen model against the self-trained one. Higher is better; rising coverage means answer variety was kept.
Takeaway → Self-training lifts single-shot from 22% to 32% and coverage from 31% to 37% — both rise together, so accuracy is banked without collapsing onto one repeated answer.
Data table
| condition | frozen | self-trained |
|---|---|---|
| think greedy@1 | 22.4% | 31.9% |
| think pass@5 | 31% | 37.1% |
Numbers from report table (M3, reports/report.md)
Technical framing
Banking self-solutions: frozen vs self-trained on held-out fresh tasks (N=210) — QLoRA-SFT on the model's own 189 verified solutions lifts deployable single-shot +0.095 (~2.2 SE, +42%) with pass@5 rising — no diversity collapse.
How to read
Three lines run across training rounds, round zero being the untrained model: single best attempt, best-of-five coverage, and accuracy on the hardest deepest puzzles. Higher is better; distance above round zero is accumulated gain.
Takeaway → The top two lines climb but flatten by round three, while the hardest puzzles stay near 9% — the loop widens reach but never cracks the frontier.
Data table
| expert-iteration round (0 = frozen) | greedy@1 | pass@5 | depth-3 greedy@1 |
|---|---|---|---|
| 0 | 26.7% | 37.8% | 6.7% |
| 1 | 35.6% | 40.7% | 11.1% |
| 2 | 38.5% | 45.9% | 6.7% |
| 3 | 39.3% | 46.7% | 8.9% |
Numbers from runs/ei_summary.json (matches report M4 table)
Technical framing
Expert-iteration flywheel: held-out accuracy by training round — Self-training compounds over rounds (+47% greedy@1) with no collapse, but plateaus and never cracks the depth-3 frontier — coverage-bounded.
In the author’s words from the Overview · “Results”
Arc: cleverer test-time readout (execution feedback) does not unearth capability; self-training on verified self-solutions banks it into the weights (and needs clean data to show); iterating that is a real but bounded flywheel — it widens the deployable footprint of the model's own distribution without extending its frontier.
Overview
Research Program
- Program:
structured_execution_and_compilers(extends claim C1: executable intermediates help). - Mission: unearth latent capability in the fixed Qwen3.5-4B weights — no larger models, no distillation, no scaling. The lever is non-stochastic execution feedback: an interpreter is a calculator, not a bigger model, so letting the 4B draft a program, run it, and refine on the real result offloads its weakness (mentally simulating execution) while its strength (writing/debugging code) does the work.
Why a new substrate
MBPP/HumanEval are saturated for this 4B (coverage ~0.89) and contamination-suspect — on a memorized benchmark you cannot tell whether feedback unearthed reasoning or jogged a memory. So this is built on a procedurally-generated, contamination-free substrate: each task is a random depth-D composition of ~23 total list-of-int primitives, presented as input/output examples; the model synthesizes a Python transform(xs) matching held-out examples (graded functionally — any correct program counts). Novel by construction; difficulty = composition depth. A reference oracle solves 100% of generated tasks, so every model failure is real.
Milestones
- Substrate + failure profile (
scripts/run_baseline.py): frozen 4B, thinking on — greedy@1 (deployable) vs pass@k (coverage) vs oracle, by depth. Establishes a hard-but-fair substrate with a coverage→deployment gap to target. - Neurosymbolic REPL loop (
scripts/run_repl.py): draft → execute on visible → real feedback (actual vs expected) → refine, ≤T turns; graded on hidden. Controls: repl_nofb (multi-turn without execution content) isolates whether feedback content matters; sample_more + visible-select at matched compute is the bar (independent sampling beat every trained arm in the corpus). Central question: does execution-grounded self-correction beat independent sampling at equal generation budget? - Bank it (if M2 positive): QLoRA-SFT the 4B on its own successful correction trajectories (no teacher); test durable single-shot improvement on held-out fresh tasks + check for diversity collapse.
Reuse
- Sandbox
src/code_env.py(fromqwen35_4b_retrieval_adapt_verify_scale: AST safety +-Isubprocess isolation + rlimits; visible/hidden protocol; captures actual outputs for feedback). Runtimesrc/gen_lib.py(thinking-budget generation). New:src/gen_tasks.py(procedural generator + oracle).
Run
../../.venv/bin/python scripts/run_baseline.py --per-depth 15 --depths 1 2 3 4 5 6 --k 6 # M1
../../.venv/bin/python scripts/run_repl.py --per-depth 30 --depths <headroom> --turns 5 # M2Results
Full write-up in reports/report.md.
- M1 (failure profile): substrate oracle-solvable 100% but hard — frozen 4B (thinking) greedy@1 0.156, pass@6 0.244; headroom at depths 1–4 (5–6 dead).
- M2 (REPL loop): execution feedback does NOT beat matched-compute sampling. repl_real 0.287 @ 3.9 gens vs sample_more 0.338 @ 5; the feedback content adds only +0.024 over a paired no-feedback control. On a contamination-free substrate, the 4B's ceiling is its own sampling distribution — a clean replication of "sample-more is hard to beat" with no memorization confound.
M3 (banking): self-training on the 4B's OWN 189 verified solutions (no teacher) improves held-out fresh single-shot — think-greedy@1 0.224 → 0.319 (+0.095, ~2.2 SE over N=210, +42% relative), pass@5 up (no diversity collapse), confirmed on two fresh seeds. Works on this contamination-free substrate where the corpus's prior MBPP self-improvement regressed.
- M4 (expert-iteration flywheel, 3 rounds): the banking gain compounds monotonically but with diminishing returns — held-out greedy@1 0.267 → 0.356 → 0.385 → 0.393 (+0.126, +47%), pass@5 rising (no collapse), each round's better model harvesting more data (107→144→162 solved). Lifts depths 1–2 but does not crack the depth-3 frontier the model can't sample — coverage-bounded, as M2 predicts.
Arc: cleverer test-time readout (execution feedback) does not unearth capability; self-training on verified self-solutions banks it into the weights (and needs clean data to show); iterating that is a real but bounded flywheel — it widens the deployable footprint of the model's own distribution without extending its frontier.
Report
Rendered from reports/report.md
Summary
Mission test: can we unearth latent capability in the fixed Qwen3.5-4B weights with non-stochastic execution feedback — no larger models, no distillation, no scaling? On a fresh, procedurally-generated, contamination-free program-synthesis substrate, three findings: (M1) the substrate is hard-but-fair — oracle-solvable 100%, but the frozen 4B (thinking on) gets greedy@1 0.156 / pass@6 0.244, with headroom at depths 1–4. (M2) A neurosymbolic multi-turn REPL loop (draft → execute → real feedback → refine) does NOT beat matched-compute independent sampling (repl_real 0.287 @ ~3.9 gens vs sample_more 0.338 @ 5), and the execution-feedback content adds only +0.024 over a paired no-feedback control (within noise). (M3) QLoRA-SFT on the 4B's OWN 189 verified solutions (no teacher) DOES bank capability into deployable single-shot — held-out fresh think-greedy@1 0.224 → 0.319 (+0.095, ~2.2 SE over N=210, +42% relative), pass@5 up (no diversity collapse), confirmed on two fresh seeds. Net: for this small model, the lever that unearths latent capability is self-training on verified self-solutions, not test-time self-correction — and this self-training works on a contamination-free substrate where the corpus's prior MBPP STaR run regressed, implicating substrate/contamination in that earlier failure. (M4) Scaled into an expert-iteration flywheel (3 rounds), the banking gain COMPOUNDS but with diminishing returns — held-out greedy@1 0.267 → 0.356 → 0.385 → 0.393 (+0.126, +47%), pass@5 rising throughout (no collapse), and each round's improved model harvests more training data (107 → 144 → 162 / 360 tasks solved). It lifts depths 1–2 where coverage exists but does not crack the depth-3 frontier the model can't sample — coverage- bounded, exactly as M2 predicts.
Research Program Fit
structured_execution_and_compilers (extends C1). Directly serves the mission memory [[unearth-latent-capability-mission]]: elicit the fixed 4B, never import capability. The interpreter is a calculator (a tool), not a model, so the loop is on-mission. Also the corpus's first contamination-free program-synthesis substrate — a reusable asset for any future elicitation claim.
Method
- Substrate (
src/gen_tasks.py): each task is a random depth-D composition of ~23 total list-of-int primitives; presented as input/output examples; the model synthesizestransform(xs)graded on held-out examples (functional — any correct program counts). Contamination-free by construction; a reference oracle solves 100%. Sandboxsrc/code_env.py(AST safety +-Isubprocess + rlimits; captures actual outputs). - M1 (
run_baseline.py): frozen 4B, thinking on — greedy@1 / pass@k / oracle by depth. - M2 (
run_repl.py): ≤5 sampled turns;repl_realrefines on real feedback (actual vs expected),repl_nofbis a paired control (reuses real's turn-0, told only "try again"),sample_moredraws 5 independent samples and selects by visible pass-count. Matched-compute accuracy-vs-generations analysis. - M3 (
collect_solutions.py→train_lora.py→eval_lora.py): QLoRA-SFT the 4B on its OWN verified (prompt→code) solutions (no teacher); eval frozen vs trained single-shot on held-out fresh tasks.
Results
M1 — failure profile (thinking on)
| depth | 1 | 2 | 3 | 4 | 5 | 6 |
|---|---|---|---|---|---|---|
| greedy@1 | 0.667 | 0.133 | 0.133 | 0.000 | 0.000 | 0.000 |
| pass@6 | 0.933 | 0.333 | 0.133 | 0.067 | 0.000 | 0.000 |
| oracle | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 |
Overall greedy@1 0.156, pass@6 0.244. Hard-but-fair; headroom at depths 1–4 (5–6 are dead — pass@6 = 0).
M2 — REPL loop vs matched-compute sampling (hidden accuracy)
| arm | acc | mean gens | d1 | d2 | d3 | d4 |
|---|---|---|---|---|---|---|
| greedy@1 | 0.263 | 1.0 | 0.65 | 0.35 | 0.00 | 0.05 |
| repl_real (feedback) | 0.287 | 3.9 | 0.60 | 0.45 | 0.00 | 0.10 |
| repl_nofb (paired control) | 0.263 | 4.0 | 0.60 | 0.40 | 0.00 | 0.05 |
| sample_more (bar) | 0.338 | 5.0 | 0.80 | 0.50 | 0.00 | 0.05 |
sample_more selected curve by #samples: 1→0.20, 2→0.237, 3→0.263, 4→0.325, 5→0.338 (== oracle curve). Figure: analysis/repl_vs_samplemore.png.
- Feedback does not beat sampling. repl_real 0.287 @ 3.9 gens lies on/below the sample_more curve; sample_more reaches 0.338. Independent sampling is the stronger use of the same compute.
- The feedback content barely matters. repl_real 0.287 vs paired repl_nofb 0.263 = +0.024 (2/80 tasks).
- Depth 3 is 0.0 for every arm — genuinely beyond this 4B; feedback cannot manufacture a solution the model can't sample. Gains are confined to the easy end (d1–d2), where sampling already wins.
M3 — banking self-solutions into single-shot
Collected 189 execution-verified (prompt→code) pairs from 146/450 fresh tasks (depths 1–3, seed 202; solve rate d1 104/150, d2 29/150, d3 13/150). QLoRA-SFT (r32/α64, 2 epochs, no teacher). Evaluated frozen vs trained on held-out fresh tasks (seeds 303 & 404, unseen compositions):
| metric (held-out, pooled N=210) | frozen | trained | Δ |
|---|---|---|---|
| think greedy@1 (deployable single-shot) | 0.224 | 0.319 | +0.095 (~2.2 SE, +42%) |
| think pass@5 (coverage) | 0.310 | 0.371 | +0.061 (no collapse) |
| no-think greedy@1 | 0.000 | 0.007 | ~0 |
By depth (confirmation set, greedy@1): d1 0.60→0.69, d2 0.13→0.33, d3 0.07→0.11. The gain is broad across the in-distribution training depths and holds on both fresh seeds.
- Self-training banks capability into deployable single-shot (+0.095, ~2.2 SE) on held-out fresh tasks — a real, generalizing improvement from the model's OWN verified solutions, no teacher.
- No diversity collapse: pass@5 rises (0.310→0.371), the opposite of the STaR failure mode that sank the corpus's contaminated-MBPP self-improvement run.
- No-think stays ~0 for the depths-1–3 adapter; the model still needs thinking (the adapter transferred its task knowledge to the thinking path). Notably a replication adapter trained on depths-1–2 data did bank a no-think one-shot path (0.0 → 0.274 at n=135), suggesting the banked path depends on the data mix.
- Replicated: a second, independently-trained adapter (fresh training data, seed 505) reproduced the held-out think-greedy@1 gain — +0.103 vs the original +0.111 (frozen 0.267) on the same held-out seed-404 set. The effect is robust to the training data, not a lucky adapter.
M4 — expert-iteration flywheel (3 rounds)
Each round: solve a fixed 360-task pool (depths 1–3, seed 202) with the CURRENT model, accumulate verified (prompt→code) pairs, retrain a fresh LoRA from base, eval on the fixed held-out (seed 404, n=135).
| round | 0 (frozen) | 1 | 2 | 3 |
|---|---|---|---|---|
| held-out greedy@1 | 0.267 | 0.356 | 0.385 | 0.393 |
| held-out pass@5 | 0.378 | 0.407 | 0.459 | 0.467 |
| train-pool solved (/360) | 107 | 144 | 162 | — |
| accumulated pairs | 147 | 219 | 287 | — |
| depth-1 / depth-2 / depth-3 greedy | .60/.13/.07 | .69/.27/.11 | .78/.31/.07 | .80/.29/.09 |
Figure: analysis/ei_trajectory.png.
- The flywheel compounds monotonically — 0.267 → 0.356 → 0.385 → 0.393 (+0.126, +47% over frozen) — but with clearly diminishing returns (+0.089, +0.029, +0.008), plateauing by round 3.
- The engine works: each round's improved model harvests MORE verified data (107→144→162 solved; 147→287 pairs), and pass@5 keeps rising (no diversity collapse) — genuine self-improvement, not resampling.
- But it is coverage-bounded: depth-1 climbs steadily (.60→.80), depth-2 plateaus (~.30), and the depth-3 frontier never cracks (~.09 throughout). The loop converts more of the model's reachable distribution into deployable single-shot; it cannot manufacture solutions the model fundamentally can't sample — exactly what M2 predicts.
Controls
Reference oracle (100% solvable) proves every failure is the model's. repl_nofb is paired to real's turn-0, isolating the feedback content from multi-turn re-drafting. sample_more selected == oracle curve shows visible-test selection is loss-free here (no C2 false-passes), so coverage is deployable accuracy — the bottleneck the loop would have to move is pure coverage.
Oracle Versus Deployable Evidence
All arms deployable (only visible info used); pass@k oracle is the coverage ceiling. Crucially the substrate is contamination-free, so — unlike MBPP — a memorization confound cannot inflate any arm.
Interpretation
Two contrasting levers, cleanly separated on the same contamination-free substrate:
- Test-time execution feedback does NOT unearth capability (M2). The model cannot reliably convert "you returned X, expected Y" into a corrected program better than re-sampling; where it can't sample a solution at all (depth ≥3), feedback adds nothing. The frozen model's test-time ceiling is its sampling distribution, and self-correction is not the lever.
- Self-training on verified self-solutions DOES (M3). QLoRA-SFT on the model's own execution-verified solutions moves the weights, banking sampling-accessible capability into deployable single-shot with a significant, generalizing +0.095 on held-out fresh tasks and no diversity collapse.
- Iterating it is a real but bounded flywheel (M4). Expert iteration compounds the gain (+0.126 over 3 rounds, +47%) — each round's better model harvests more verified data — but with diminishing returns that plateau, and it cannot crack the depth-3 frontier the model can't sample. Self-training widens the deployable footprint of the model's own distribution; it does not extend the distribution's frontier.
The synthesis: for a small model, you don't unearth latent capability by reading the frozen weights more cleverly at test time — you bank it into the weights by training on what the model can already verify. And the fact that this self-training works here but regressed on MBPP (qwen35_4b_verifier_guided_self_improvement) is itself a finding: contamination/substrate, not the method, likely explains the earlier failure — a contamination-free, structured substrate is what lets honest self-improvement show up. This is on-mission (no teacher, no scaling) and points the whole capability question toward clean-data self-training.
Limitations
- One model, one task family; budget 512; n=20/depth in M2. Depth ≥3 near-zero limits headroom to d1–d2.
- The loop uses only per-example pass/fail + outputs; richer feedback (traces, counterexamples) is untested.
Next Experiments
- (M3, running) banking; if it also fails, the combined statement is strong: neither test-time feedback nor self-training unearths capability here.
- Richer feedback signals; a substrate whose difficulty sits in the d1–d2 "coverage-exists" band for more mass.
Artifact Manifest
See artifact_manifest.yaml. Small tasks/records/summaries + figures in-repo; the LoRA adapter (~170MB) is a regenerable training artifact.
Experiment log 6
Show the running log (6 entries)
Design
Mission: unearth latent capability in the FIXED Qwen3.5-4B weights via non-stochastic execution feedback (an interpreter is a calculator, not a bigger model). Fresh procedurally-generated, contamination-free substrate (compositional list transforms, PBE, graded by held-out execution; difficulty = depth).
- M1 (
run_baseline.py): failure profile. Substrate solvable (oracle 100%) but hard; coverage->deployment headroom at depths 1-4 (depths 5-6 dead: pass@6=0). - M2 (
run_repl.py): REPL loop (draft -> execute visible -> real feedback -> refine, sampled turns) vs matched-compute sample-more + visible-select, with a paired no-feedback control. Matched-compute accuracy-vs-generations analysis. Central question: does execution feedback beat independent sampling. - M3 (
collect_solutions.py+train_lora.py+eval_lora.py): bank the 4B's OWN verified solutions via QLoRA-SFT (no teacher), test single-shot generalization on held-out FRESH tasks + depth extrapolation.
Env / decisions
- Installed peft 0.19.1 into the .venv (was missing; needed for M3 LoRA). bitsandbytes 0.49.2 already present.
- Trainer uses
AutoModelForCausalLM(NOT the corpus'sAutoModelForImageTextToText) to MATCH the eval runtime (gen_libloads CausalLM), so the LoRA adapter's module paths are compatible frozen<->trained. - M2 uses budget 512 (short synthesis tasks; ~halves thinking cost vs 1024) and SAMPLED REPL turns (greedy per-turn got stuck in the smoke -- 0/5 fixed; sampling gives exploration and a fair match to sample-more).
- REPL no-feedback control is PAIRED (reuses repl_real's turn-0 draft), isolating the feedback content.
M1 results (see runs/baseline.json)
Depths 1-6, 15/depth, thinking on. greedy@1 / pass@6 / oracle by depth: d1 .667/.933/1.0, d2 .133/.333/1.0, d3 .133/.133/1.0, d4 .0/.067/1.0, d5 .0/.0/1.0, d6 .0/.0/1.0. Overall greedy .156, pass@6 .244 (coverage->deployment gap +.089). Substrate is hard-but-fair with real headroom at depths 1-4.
M2 results (see runs/repl_summary.json, analysis/repl_vs_samplemore.png)
80 tasks (depths 1-4, 20/depth), turns 5, budget 512, seed 101. Hidden accuracy: greedy@1 0.263 (1 gen), repl_real 0.287 (3.9 gens), repl_nofb 0.263 (4.0 gens), sample_more 0.338 (5 gens). sample_more selected curve: n1 .20, n2 .237, n3 .263, n4 .325, n5 .338 (== oracle curve: visible-select is perfect here -- 6 visible I/O examples is a near-perfect correctness signal, no C2 false-passes).
NEGATIVE for the loop, cleanly controlled: (1) execution-grounded self-correction does NOT beat matched-compute independent sampling -- repl_real 0.287 @ 3.9 gens sits on/below the sample_more curve; sample_more reaches 0.338. (2) The execution-feedback CONTENT adds only +0.024 over the paired no-feedback control (repl_real 0.287 vs repl_nofb 0.263), within noise. (3) By depth: gains are at d1-d2 (d2: greedy .35 -> repl_real .45 -> sample_more .50); d3 is 0.0 for ALL arms (unsolvable by this 4B, feedback or not); d4 ~.05-.10. So for this 4B on a clean substrate, the capability ceiling is its own sampling distribution and execution feedback does not push past it -- replicating the corpus's "sample-more is hard to beat" theme on contamination-free data.
M3 results (see runs/eval_frozen*.json vs runs/eval_trained*.json)
Collected 189 verified (prompt->code) pairs from 146/450 fresh tasks (depths 1-3, seed 202; d1 104/150, d2 29/150, d3 13/150). QLoRA-SFT r32/a64, 2 epochs, no teacher (train_loss 0.113). Held-out fresh eval (seed 303 n=75 depths 1-5, seed 404 n=135 depths 1-3):
POOLED N=210 -- think_greedy@1: frozen 0.224 -> trained 0.319 (+0.095, ~2.2 SE, +42% rel). think_pass@5: 0.310 -> 0.371 (+0.061, no collapse). no-think greedy ~0 both (model still needs thinking; adapter trained on no-think targets transferred to the thinking path). By depth (confirm greedy): d1 .60->.69, d2 .13->.33, d3 .07->.11 -- broad, both seeds.
POSITIVE, confirmed + REPLICATED: self-training on own verified solutions banks capability into deployable single-shot on held-out fresh tasks, no diversity collapse. A second adapter trained on fresh data (seed 505, depths 1-2) reproduced the gain on the held-out seed-404 set: +0.103 vs the original +0.111 (frozen 0.267) -- robust to training data. (The seed-505 adapter also banked a no-think path: 0.0->0.274.) Contrast with M2 (test-time feedback negative) and with the corpus's MBPP verifier_guided_self_improvement (regressed) -- works on contamination-free data. Adapters (~170MB) gitignored + removed before commit (regenerable via scripts/m3_chain.sh; replication via scripts/replicate.sh).
M4 results (see runs/ei_summary.json, analysis/ei_trajectory.png)
Expert-iteration flywheel (scripts/ei_loop.sh): 3 rounds, each solving a fixed 360-task pool (depths 1-3, seed 202) with the CURRENT model, accumulating verified pairs, retraining a fresh LoRA from base, evaluating on held-out (seed 404, n=135). Held-out greedy@1: frozen 0.267 -> R1 0.356 -> R2 0.385 -> R3 0.393 (+0.126, +47%); pass@5 0.378->0.407->0.459->0.467 (rising, no collapse). Train-pool solved 107->144->162/360; pairs 147->219->287. By depth: d1 .60->.80 (climbs), d2 ~.30 (plateaus), d3 ~.09 (never cracks).
COMPOUNDS but BOUNDED: monotonic gain with clear diminishing returns (+.089,+.029,+.008), plateauing by R3. The engine works (better model -> more verified data each round, coverage grows, no collapse) but it is coverage-bounded -- widens the deployable footprint of the model's OWN distribution without extending its frontier (depth-3 unreachable throughout), exactly as M2 predicts. Added --adapter/--append to collect_solutions.py for the flywheel; adapters removed before commit.
Figures 4
Data files 11
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
runs/baseline.json884 Bruns/ei_eval_1.json439 Bruns/ei_eval_2.json439 Bruns/ei_eval_3.json443 Bruns/ei_summary.json503 Bruns/eval_frozen.json506 Bruns/eval_frozen_big.json417 Bruns/eval_trained.json534 Bruns/eval_trained2_big.json445 Bruns/eval_trained_big.json439 Bruns/repl_summary.json2.3 kB
Reproduce
Run steps are documented inside the experiment folder (README and scripts).