Qwen35 4B Self-Repair Install
The one idea you need
The team is trying to turn a small model that can write one function into one that can carry a task across many steps. A cheap proxy benchmark did not predict real coding, and the first direct attempt (teaching the model to trace code execution) came back flat: it changed WHICH coding tasks got solved, not HOW MANY, and left the multi-step agentic score exactly where it started. Watching those agentic failures, the pattern is clear: the model takes one shot at a multi-step task, hits a failed check, and STOPS instead of figuring out what went wrong and fixing it. This experiment trains that missing loop directly. It generates hundreds of tiny Python functions, breaks each one with a single deliberate mistake (an off-by-one, a flipped comparison, a wrong operator), runs the broken version to capture the exact failing test, and trains the model to read the failure, say what line is wrong and why, and hand back the fixed function. Because the team INJECTED each bug, they already know the correct fix, so every example is self-checked by actually running the code — no bigger 'teacher' model is used. And the training material looks nothing like the exams (the exams say 'write code from a description'; this says 'here is broken code and its failing test, fix it'), so any improvement is real transfer, not memorization.
The question
Does teaching the 4B to debug — read a small function's failing test, diagnose the injected bug, and return the corrected code — transfer to real coding (HumanEval + MBPP) and, the real target, get it to persist-and-repair on multi-step agentic tasks, without breaking its ability to write functions?
What we found
The first bet that moved the needle at all — gently. We taught the 4B to debug by training on 504 examples of [buggy code + the real test-failure message] -> [diagnosis + fix], all self-generated by injecting bugs into correct code and running the tests. Result: HumanEval +3 problems (76.2->78.0%), MBPP -2 (a wash), and on the real multi-step agentic harness 8/35 -> 10/35 — and crucially the wins are asymmetric: it solved 3 tasks the base couldn't while losing only 1, so it is ADDING capability, not just trading it around like the previous (execution-tracing) bet did. Honest caveat: the numbers are small and none is statistically significant on its own, so this is a promising direction, not a proven win. The real lesson: teaching the LOOP behavior (check your work and fix it) does something teaching a passive skill (trace the code) did not. That points the way for the next bets — teach the model WHY, and stack this repair ingredient with it.
Why it matters
This is the program's second direct attempt to INSTALL coding cognition after a proxy and a first curriculum both failed to raise it. The specific thing separating the 4B's 76% function-writing from its 23% agent score is persistence: recovering from a failed check instead of quitting. If self-repair transfers even a little without breaking function-writing, it validates that the missing skill is trainable by a designed curriculum; if it is flat, that is the third independent signal that hand-built supervised curricula reshuffle coding ability without raising it — which is exactly the evidence that funds the pre-committed pivot to reinforcement learning, instead of leaving the team guessing what to try next.
On this page
Results at a glance 1
pass rate / count · coding fitness signal (agentic in raw count of 35) →
Data table
| coding fitness signal (agentic in raw count of 35) | base | self_repair |
|---|---|---|
| HumanEval | 0.762 | 0.78 |
| MBPP | 0.565 | 0.555 |
| Agentic (of 35) | 8 | 10 |
Numbers from experiments/qwen35_4b_self_repair_install/reports/measure_review.md
In the author’s words from the Overview · “Results”
When run, runs/measure/transfer_summary.json records all four pass@1 numbers (base/treatment x HE/MBPP, counts + fractions), the per-problem paired deltas, and the frozen, tightened verdict. Separate deployable evidence (transfer) from the retention guard.
Overview
Research Program
- Program:
agentic_breadth_installation(cognitive-core coding sub-program) - Program question: can real, transferable coding capability be INSTALLED into base Qwen/Qwen3.5-4B by designed, contamination-free curricula, proven by transfer to held-out coding — after the menagerie proxy (McNemar p=1.00) AND bet #1 (execution-tracing, NULL) both reshuffled coding tasks without raising the count?
- Prior anchors: base coding baselines from
experiments/qwen35_4b_coding_fitness_harness— HumanEval 76.2% (strong function coder), MBPP 56.5%, agentic duet-eval 23% (weak agent). Bet #1 (experiments/qwen35_4b_exec_trace_install) was NULL: HumanEval +1, MBPP -3, agentic 8/35 flat. The observed agentic failure mode is a LOOP failure — one shot, a check fails, and the model STOPS instead of verifying and repairing.
Question
Does installing the CHECK-AND-REPAIR loop via a SELF-REPAIR debugging curriculum — buggy functions (bug injected by AST mutation, so the fix is known) + their concrete failing tests -> a localized diagnosis + the corrected code — TRANSFER to real coding (HumanEval + MBPP pass@1), and (the primary target) move the agentic duet-eval, without regressing function-writing?
Hypothesis
Bet #1 showed that installing a PASSIVE cognitive primitive (execution modeling) reshuffles but does not raise coding capability. Self-repair targets the ACTIVE failure mode directly — persist through a failed check, diagnose, and fix. The task ENDS by emitting code (the correction) under a distinct instruction, so it does not compete with code generation (lower forgetting risk than trace-only). Because the training data is buggy-function + failing-test -> corrected-function (nothing like spec -> code), any HumanEval/MBPP movement is genuine TRANSFER. Honest prior on a MEANINGFUL install (>= 3-problem gain with retention): ~25-30%; NULL is the single likeliest verdict and would fund the pre-committed RL pivot.
Setup
- Model: only
Qwen/Qwen3.5-4B(rev851bf6e8…); ONE fresh r32/a64 QLoRA adapter trained from thebase_reserializedcomposite in a single stage. - Dataset/task source:
data/sft_self_repair.jsonl— 504 debugging episodes built byscripts/gen_self_repair_curriculum.py(construction seed 91330). Every row is triple-verified by real execution (correct passes all tests, buggy fails >=1 with a wrong value and crashes on none, correction differs); the bug is injected by AST mutation so the fix is known and provenance-clean. - Train/eval split: training is
buggy-function + failing-test -> corrected code; evaluation is the held-outspec -> codeHumanEval (164) + MBPP (200) — deliberately disjoint surfaces. The agentic duet-eval is the primary real target, run manually as a follow-on. - Baseline: base Qwen/Qwen3.5-4B on the same shared harness.
- Controls: contamination firewall (whole-word banned-benchmark-name audit, zero hits; distinctive code n-gram overlap, zero); the base composite is authenticated fail-closed (tree + weights) before training.
- Primary metric: greedy pass@1 on HumanEval + MBPP (shared fitness harness,
experiments/qwen35_4b_coding_fitness_harness/scripts/eval_pass1.py, referenced not copied). - Oracle-only metrics: none gate here; the agentic duet-eval is a follow-on confirm, not part of this cell's frozen consequence.
- Hidden-label boundary: the frozen, TIGHTENED two-directional consequence (INSTALLED_CODING requires a >= 3-problem gain / RETENTION_FAIL / NULL) is read once from the four pass@1 numbers; benchmarks are executed, never read as data.
Run
Smoke (no GPU, no writes):
python scripts/run.py --smokeGPU stages (each gated behind a staged adversarial review; see reports/preregistration.md for the exact commands and checkpoint order):
python scripts/run.py --stage train # r32/a64, 1 epoch, seed 91331
python scripts/run.py --stage merge # vendored composite merger
python scripts/run.py --stage measure # shared HumanEval+MBPP harness, both armsAgentic confirm (manual follow-on on the merged composite — the PRIMARY real target, base 8/35): run the duet-eval gen4 harness with --model-override set to large_artifacts/qwen35_4b_self_repair_install/merged/self_repair, exactly as bet #1's measure review documented for the exec_trace composite.
Results
Not yet run. The install/merge/measure stages are gated behind staged reviews. When run, runs/measure/transfer_summary.json records all four pass@1 numbers (base/treatment x HE/MBPP, counts + fractions), the per-problem paired deltas, and the frozen, tightened verdict. Separate deployable evidence (transfer) from the retention guard.
Interpretation
Pending the sealed measurement. INSTALLED_CODING makes self_repair the program's reference composite and funds the agentic confirm; RETENTION_FAIL realizes the forgetting risk and funds the --mix-retention re-run; NULL is the third static-SFT curriculum family to reshuffle-without-raising and funds the pre-committed PIVOT to reinforcement learning on the agentic loop.
Knowledgebase Update
- Program evidence updated: pending measurement.
- Program backlog updated: pending measurement.
- Claim ledger updated: pending measurement.
Artifacts
scripts/— self-repair curriculum generator (AST mutation + safe execution), contamination module, vendored trainer/merger, fail-closedtrain_trial.py,measure_transfer.py,run.py.data/sft_self_repair.jsonl— the 504-row curriculum +curriculum_receipt.json;data/provenance/base_reserialized.json;data/contamination/.configs/,reports/,tests/reports/artifact_manifest.yaml
Report
Rendered from reports/report.md
Design-frozen report. The model-free construction is complete and verified; the train/merge/measure GPU stages are gated behind staged adversarial reviews and have not run. Results will be appended to the Results section once the sealed measurement is read.
Summary
Lifecycle 33 — the SECOND curriculum bet of the cognitive-core coding program (cognitive-core coding bet #2). Bet #1 (execution-tracing) came back NULL: it reshuffled which coding tasks the 4B solves without raising the count on HumanEval, MBPP, or the agentic duet-eval (8/35 -> 8/35). The observed agentic failure mode is a LOOP failure — the model one-shots a multi-step task, a check fails, and it STOPS instead of verifying and repairing. This cell installs the CHECK-AND-REPAIR loop directly: a fresh r32/a64 LoRA trains (1 epoch, seed 91331) on 504 debugging episodes, each a synthetic function with a bug INJECTED by AST mutation plus its concrete failing test, targeting a localized diagnosis and the corrected code; it merges onto base and is measured for TRANSFER + RETENTION on the shared HumanEval + MBPP fitness harness under a frozen, TIGHTENED two-directional consequence (INSTALLED_CODING requires a >= 3-problem gain). The training signal is SELF-GENERATED and EXECUTION-VERIFIED (we injected the bug, so we know the fix; every pair is confirmed by real execution) and looks like nothing in the benchmarks (buggy-function + failing-test -> corrected-function, not spec -> code), so any movement is genuine transfer.
Research Program Fit
The program's target is the base 4B's agentic/multi-step cognition gap: HumanEval 76.2% (strong function coder) vs duet-eval 23% (weak agent). Bet #1 showed that installing a PASSIVE cognitive primitive (execution modeling) reshuffles but does not raise coding capability. Bet #2 targets the ACTIVE failure mode — persistence and self-correction after a failed check — which is the loop the agentic eval actually exercises. HumanEval/MBPP serve as the fast transfer + retention signal; the agentic duet-eval (base 8/35) is the PRIMARY real target, run manually as a follow-on (not gated by this cell).
Method
- Curriculum (
scripts/gen_self_repair_curriculum.py, seed 91330). 13 parameterized synthetic function families over integer/list inputs (thresholded counts, scaled sums, factorial/product, largest, spread, even-index sum, position-weighted sum, nested-triangle sum, above-average count, running-cap, countdown, clamp, absolute-gap). A FROZEN mixed schedule biased to medium/long: short 120, medium 192, long 192 (504 total). For each row a correct function + concreteasserttests are generated and executed; a bug is injected by a single AST mutation from a diverse, seeded set (flipped comparison, wrong arithmetic operator, swapped operands, off-by-one loop bound, extra+ 1on the return, off-by-one constant, shifted index — all seven kinds present). Each row: prompt = the buggy function (with docstring) + the tests + the concrete first failure (returned <got>, expected <expected>) + "Diagnose the bug and give the corrected code."; think = a short localized diagnosis (the failing case, the buggy line quoted verbatim, the mechanism, the corrected line); answer = the corrected function. - Triple truth audit (never ship an unverified pair), by REAL CPython execution. (1) the correct function passes ALL its tests; (2) the buggy function fails AT LEAST ONE test with a WRONG VALUE and RAISES on NONE (a crashing or behavior-preserving mutation is rejected); (3) the correction differs from the buggy code (exactly one changed line) and the shipped failure matches the actual first failing test. Safety/termination: restricted builtins (no imports/I/O), only bounded for-loops (never
while), a per-call step cap that aborts runaway code. The committed corpus additionally re-executes end-to-end via--verify-corpus(used in smoke), and the unit tests independently re-grade every row with a separate assert-based grader. - Contamination firewall (
scripts/contamination.py). A committed banned set of all 668 HumanEval + MBPP function names (Python keywords/emitted builtins whitelisted) — zero whole-word hits. A present-only code n-gram aid — zero shared 7-grams carrying a distinctive (non-idiom) token between the corpus's executable code (docstring spec prose excluded — governed by the banned-name gate) and benchmark solution code. Row-level uniqueness (504 unique prompts/task-ids/code-pairs). - Install (
scripts/train_trial.py-> vendoredscripts/train_think.py). One fresh r32/a64 adapter, epochs 1, lr 1e-5, batch 1, grad-accum 8, max-length 4096, w_think 0.2, w_close 0.2, seed 91331 (63 optimizer steps), from thebase_reserializedcomposite. The base is authenticated FAIL-CLOSED (in-cell provenance copy + full tree manifest + full 9 GB weights hash) before training. - Merge (vendored
scripts/merge_adapter.py) with--base-model= the base composite ->merged/self_repair. - Measure (
scripts/measure_transfer.py-> SHARED harness, referenced not copied). Base and self_repair, HumanEval 164 + MBPP 200, greedy pass@1, identical vLLM path; all four numbers (counts + fractions) + per-problem paired deltas + the frozen, tightened verdict recorded.
Results
Pending the sealed measurement. runs/measure/transfer_summary.json will carry pass_at_1{base,self_repair}{humaneval,mbpp}, the pass counts, the McNemar b/c paired deltas per dataset, and the frozen consequence. Deployable transfer evidence is a >= 3-problem pass@1 gain; the retention guard is the paired dataset staying within 0.02.
Construction facts already established (model-free):
- Corpus sha
920cb228172677f005bdbc4501f593ce60dc7a9c4f22cbf177f05660ffc392cb, 504 rows, allself_repair; tiers short 120 / medium 192 / long 192; 3-4 tests per row; 504 unique buggy/corrected code pairs. - Mutation-kind spread (all seven present): arith_op 123, const_offset 107, return_offset 107, compare_op 83, range_bound 65, index_shift 10, operand_swap 9.
- Contamination: 668 banned benchmark names, 0 whole-word hits; 0 distinctive shared 7-grams (80 shared spans, all pure control-flow idioms).
- 55 unit tests green (present-only cache aids RUN with the HF cache; every row independently re-executed: buggy fails >=1 with a wrong value and crashes on none, corrected passes all, they differ).
Controls
- Contamination firewall (banned-name audit + distinctive code n-gram overlap), both zero, so a benchmark movement cannot be memorization.
- Base composite authenticated fail-closed (tree + weights) before training and merge; a swapped composite aborts.
- Identical measurement path for both arms (the shared harness), so base and treatment pass@1 are directly comparable.
- Tightened consequence rule: a noise-level (<3-problem) bump reads NULL, fixing bet #1's letter-of-the-law false positive.
Oracle Versus Deployable Evidence
Deployable evidence = a >= 3-problem HumanEval/MBPP pass@1 gain (real, held-out spec -> code generation). The retention guard (the other dataset within 0.02) is a control on the forgetting risk, not a capability claim. The agentic duet-eval is the eventual deployable target but is a manual follow-on confirm, not gated here. No metric here uses hidden labels beyond the one-shot frozen verdict read.
Interpretation
Pending measurement. INSTALLED_CODING: the check-and-repair curriculum transfers; self_repair becomes the program reference and funds the agentic confirm. RETENTION_FAIL: the forgetting risk is realized; re-run with --mix-retention. NULL: the third static-SFT curriculum family to reshuffle-without-raising — a preserved boundary finding that funds the pre-committed RL pivot to the agentic loop policy, not a fourth static-SFT re-roll.
Next Experiments
- If INSTALLED_CODING: run the agentic duet-eval confirm on the self_repair composite; consider a larger/deeper repair dose.
- If RETENTION_FAIL: re-run the
--mix-retentionvariant and re-measure. - If NULL: execute the pre-committed PIVOT to reinforcement learning on the agentic plan-act-verify-repair loop (self-generated rollouts graded by real test execution, provenance-clean), rather than another static-SFT curriculum.
Artifact Manifest
See artifact_manifest.yaml — the trained adapter and merged composite live under large_artifacts/ (omitted from git); the curriculum, contamination fixture, base provenance copy, and receipts are in-repo and reproducibility- critical.
Experiment log 2
Show the running log (2 entries, 2026-07-18)
2026-07-18 — model-free construction frozen
Lifecycle 33, the SECOND curriculum bet of the cognitive-core coding program (cognitive-core coding bet #2). Mission: install real coding capability into base Qwen/Qwen3.5-4B via a designed, contamination-free curriculum, proven by transfer. Meta-context: the menagerie proxy did NOT transfer (McNemar p=1.00), and bet #1 (execution-tracing) was NULL — it reshuffled which coding tasks the 4B solves without raising the count (HumanEval +1, MBPP -3, agentic 8/35 flat). Two curriculum families now reshuffle-without-raising.
The bet: install the CHECK-AND-REPAIR loop directly, targeting the observed agentic failure mode (one shot, a check fails, the model STOPS instead of verifying and repairing). Training signal: buggy function (bug injected by AST mutation, so the fix is KNOWN) + its concrete failing test -> a localized diagnosis + the corrected code. Self-generated and execution-verified (no larger teacher), disjoint from HumanEval/MBPP.
Built and verified (no GPU, no commit):
scripts/gen_self_repair_curriculum.py(seed 91330) — 504 debugging episodes across 13 parameterized synthetic function families, TRIPLE-verified by real CPython execution (correct passes all tests; buggy fails >=1 with a WRONG VALUE and crashes on none; correction differs, exactly one changed line; the shown failure matches the actual first failing test). Corpus sha920cb228172677f005bdbc4501f593ce60dc7a9c4f22cbf177f05660ffc392cb; tiers short 120 / medium 192 / long 192; all seven mutation kinds present (arith_op 123, const_offset 107, return_offset 107, compare_op 83, range_bound 65, index_shift 10, operand_swap 9); 504 unique code pairs. A--mix-retention Rswitch (default OFF) is prepared as the forgetting guard (blends self-generatedspec -> functionrows).scripts/contamination.py+data/contamination/banned_function_names.json(668 benchmark function names) — 0 whole-word hits; 0 distinctive shared 7-grams between the corpus's executable CODE (docstring spec prose excluded — governed by the banned-name gate; excluded WITHOUT creating false cross-line grams) and benchmark solution code (80 shared spans, all pure control-flow idioms).- Vendored
scripts/train_think.py(sha e0eca2a2…) andscripts/merge_adapter.py(sha cb9af8b4…), byte-identical to the chain trainer/merger. scripts/train_trial.py— fail-closed base authentication (in-cell provenance copy + tree manifest + full weights hash); recipe r32/a64, 1 epoch, lr 1e-5, seed 91331 (63 optimizer steps).scripts/measure_transfer.py— invokes the shared fitness harness for both arms x both datasets; frozen, TIGHTENED INSTALLED_CODING (>= 3-problem gain) / RETENTION_FAIL / NULL consequence (fixes bet #1's letter-of-the-law +1-problem false positive).scripts/run.py— checkpointed--smoke | --stage train | --stage merge | --stage measure; each GPU stage gated behind a staged adversarial review.- 55 unit tests green (present-only HF-cache aids RUN with the cache; every row independently re-executed by a separate assert-based grader);
run.py --smokegreen; boundary drills refuse.
Grep-fresh note: construction seed 91330 and training seed 91331 are fully fresh repo-wide as SEEDS (the only textual matches are incidental substrings inside unrelated hashes/floats). No training-seed collision.
Pre-committed pivot: if this bet is NULL, the next move is NOT a fourth static-SFT curriculum — it is a PIVOT to reinforcement learning on the agentic plan-act-verify-repair loop (self-generated rollouts graded by real test execution, provenance-clean).
GPU stages (train/merge/measure) are pending their staged reviews.
2026-07-18 — Transfer measurement: WEAK POSITIVE (first non-null)
self_repair vs base: HumanEval 0.7622->0.7805 (+3), MBPP 0.5650-> 0.5550 (-2), agentic 8/35->10/35 (discordant 1 base-only / 3 self_repair-only). Frozen tightened rule fires INSTALLED_CODING on the +3 HumanEval. HONEST: weak, underpowered, not individually significant
- but directionally positive on HE + agentic with an asymmetric
agentic discordant (adding, not the flat 5v5 reshuffle of bet #1).
- FINDING: loop-behavior curricula (self-repair) outperform passive- skill curricula (execution-tracing) for coding. First evidence that targeting the LOOP (the observed one-shot-and-stop failure mode) does something the passive-component install did not. self_repair kept as a candidate ingredient to stack + confirm with larger agentic N.
- NEXT: the owner-specified WHY-not-WHAT family (bet #4 inline #WHY: comments first - cleanest test since comments are inert to grading; then bet #3 think-block; anneal on whichever earns it).
Reproduce
Smoke test
python scripts/run.py --smokeFull run
python scripts/run.py --stage train && python scripts/run.py --stage merge && python scripts/run.py --stage measureRun steps are documented inside the experiment folder (README and scripts).