Repository search-compress-bank coding curriculum
The one idea you need
Imagine training a mechanic only on repairs that went perfectly the first time: inspect, fix, test, done. They nail that tidy four-step routine but never watch a stuck bolt get worked loose — so when a fix is rejected, they just resubmit the same broken move.
The question
If you train a small coding agent only on its clean, fully successful bug-fix runs, does it get broadly better at fixing bugs across new codebases?
What we found
No. It became flawless on the six codebase types it trained on, solving every task in exactly four clean steps. But on four never-seen codebase types its success collapsed from 68% to 35% — worse than just running the untrained model twice, which scored 53%. Stripped of every failed attempt, it never learned what to do when a fix gets rejected.
Why it matters
When distilling agent trajectories, keep the failures and recoveries, not just the shortest winning path. Balance error-conditioned moves — failed-test-then-revise — not just how often each action appears, and test recovery on unseen tasks before trusting it.
On this page
Results at a glance 2
How to read
Grouped bars show the share of coding tasks solved. The left pair is the six practiced codebase types, the right pair four unseen types. The lighter bar is before training, the darker bar after clean-wins training. Taller is better.
Takeaway → The practiced pair climbs to a perfect 100%, but the unseen pair drops by a third — the gain was memorized cloning, not transferable coding skill.
Data table
| task family | before training (frozen) | after compact-bank training |
|---|---|---|
| trained family (48 tasks) | 83.3% | 100% |
| 4 unseen families (72 tasks) | 68.1% | 34.7% |
Numbers from reports/result_receipt.json
Technical framing
Compact-bank training helped the trained family but broke transfer — Trained-family success rose to perfect, but on four wholly unseen repository families it fell by a third — family-specific policy cloning, not transferable skill.
How to read
Bars count target training rows per step — inspect, patch, verify, commit — in a 20-repository setup check. All four sit at exactly 20 rows, showing compression kept each step in equal amounts. This is a data check, not a model result.
Takeaway → Perfectly level bars confirm the compression balanced how often each step appears — yet equal counts still failed, because they never say when to recover after a failure.
Data table
| operator | canonical target rows |
|---|---|
| inspect | 20 |
| patch | 20 |
| verify | 20 |
| commit | 20 |
Numbers from experiments/qwen35_4b_repo_search_compress_bank/reports/smoke_receipt.json
Technical framing
Replay compression preserves all four coding operators in smoke — This is a deterministic substrate smoke, not a model result. All 20 test repositories replayed one target for each operator with equal loss mass and no hidden-test leakage.
In the author’s words from the Overview · “Result”
PRIMARY GATE FAILED. The compact arm improved trained-family success from 40/48 to 48/48 (+16.7 points) and reduced the learned path to exactly four valid calls on every task, but this was family-specific policy cloning rather than transferable coding competence. On four wholly held-out repository families it fell from 49/72 to 25/72 (−33.3 points; paired 95% CI [−44.4, −22.2]) and also lost to matched-compute sampling by 18.1 points. Median unrelated-context centered-logit drift was 0.386 against the frozen 0.15 ceiling. The transition audit localizes the failure. After a failed visible test, the apex control patched again on 24/26 transitions; compact did so on 0/48, instead re-testing, re-inspecting, or emitting an invalid action. … Read the full result →
Overview
This experiment asks whether Qwen/Qwen3.5-4B can acquire broader coding-agent competence by searching real procedural repositories, verifying candidate repairs with executable tests, compressing successful tool trajectories to their replay-necessary core, and banking compact plans and actions with explicit operator balance.
Research program
Primary program: agentic_breadth_installation. The experiment follows C53's queued scaffold-distillation direction, composes C12/C22 verified banking with C54 compression advantage, and directly guards the semantic-operator collapse found by the interactive-policy curriculum.
The closest near-duplicate is qwen35_4b_think_ftpo_round2: it introduced a six-family procedural repository agent only as a fresh evaluation gate. It did not search repositories for training data, replay-minimize successful edits, bank tool states, balance operators, or train on coding-agent actions.
Hypothesis
Long successful coding trajectories contain a small causal core: issue-directed inspection, necessary edits, post-edit verification, and commit. Executable minimization should discard failed branches and redundant patches; compact state-specific planning should install the useful transition structure; equal INSPECT/PATCH/VERIFY/COMMIT loss mass should preserve the rare terminal operators that broad DAgger erased.
The hypothesis fails if the compact arm cannot beat (a) an identical action-only bank, (b) the regenerated C54 apex policy on family-disjoint repositories, and (c) two shorter apex samples under the same eight-call/token reservation.
Result
PRIMARY GATE FAILED. The compact arm improved trained-family success from 40/48 to 48/48 (+16.7 points) and reduced the learned path to exactly four valid calls on every task, but this was family-specific policy cloning rather than transferable coding competence. On four wholly held-out repository families it fell from 49/72 to 25/72 (−33.3 points; paired 95% CI [−44.4, −22.2]) and also lost to matched-compute sampling by 18.1 points. Median unrelated-context centered-logit drift was 0.386 against the frozen 0.15 ceiling.
The transition audit localizes the failure. After a failed visible test, the apex control patched again on 24/26 transitions; compact did so on 0/48, instead re-testing, re-inspecting, or emitting an invalid action. All 18 recursive-overlay tasks repeated the same exact patch after the patch tool had already rejected it. Operator-marginal balance preserved the happy-path vocabulary, but success-only compression deleted failure-conditioned recovery behavior.
The necessary gate stopped action-only training, confirmation, and Menagerie. Therefore the result rejects the registered compact-bank recipe as a whole; it does not identify compact plan gradients rather than action gradients as the cause. See the final report, primary gate, and result receipt.
Firewall-clean substrate
Ten fresh Python repository generators materialize source files and a visible test. Six families are eligible for search/training; four algorithmically distinct families are never harvested or trained and are used only for transfer. Hidden test programs and oracle edits live in host memory, are never written into the repository, never enter a model message, and are reduced to pass/fail booleans in receipts. The benchmark directory is never read or imported.
The agent has real bounded tools: tree, read, literal search, exact single-replacement patch, subprocess test, and submit. Source/test paths are constrained to the temporary repository; visible tests are readable but immutable.
Arms and controls
apex_replay: regenerate C54 apex from its committed training data with this experiment's fixed optimizer budget.compact: the same C54 data plus replay-minimized repository rows with compact state-specific plans.action_only: identical repository contexts, actions, row weights, and optimizer budget, but zero loss on the compact plan span.- C53 blend: fixed search teacher and contextual deployment comparator, never a source of private labels.
- Matched sampling: two independent four-turn apex rollouts versus one eight-turn candidate rollout; both reserve eight model calls and 6,144 sampled tokens per task.
Entropy and varentropy may be recorded later to route which live states deserve more search, but verifier outcomes supply correctness and operator class supplies balancing. They never scale token loss or choose a push-up/push-down target.
Frozen stages
- Self-test every generator, path boundary, and hidden-label firewall.
- Search 24 tasks from each of six training families with four eight-turn C53 trajectories.
- Keep private-test successes, greedily delete replay-unnecessary patches, then reconstruct and replay inspect→patch→test→submit traces.
- Train the three registered arms from the pinned base at the same 584 optimizer steps.
- Gate trained-family retention, family-disjoint transfer, compact-vs-action-only advantage, matched-call sampling, verification/commit retention, invalid actions, and unrelated-context logit locality.
- Confirm on a second transfer block. Only then assign fresh union-checked Menagerie seeds and compare aggregate quick/medium scores through the benchmark CLI.
The run stopped during stage 5. Stages 6 and Menagerie were not authorized.
Exact thresholds, seeds, and interpretation rules are frozen in the preregistration, its pre-harvest token-mass implementation amendment, the memory-feasibility audit, the final exact chunked-loss amendment, and configuration. The adversarial review is in reports/design_review.md.
Run
CPU smoke (no model or benchmark):
.venv/bin/python experiments/qwen35_4b_repo_search_compress_bank/scripts/run.py --smokeGPU smoke after the preregistration commit:
.venv-vllm/bin/python experiments/qwen35_4b_repo_search_compress_bank/scripts/run.py --gpu-smokeThe staged full command is intentionally gate-stopping:
.venv/bin/python experiments/qwen35_4b_repo_search_compress_bank/scripts/run.py --fullResult-bearing model outputs and weights live under large_artifacts/qwen35_4b_repo_search_compress_bank; small summaries and receipts are committed. No PR is created: accepted work is rebased and pushed directly to main.
Status
PRIMARY GATE FAILED; EXPERIMENT STOPPED. Both licensed 584-step arms trained and merged successfully. The compact arm overfit the trained procedural families, regressed family-disjoint transfer, lost to matched sampling, failed verification/invalid-action retention, and exceeded the locality ceiling. Action-only, confirmation, and Menagerie were correctly cancelled; zero benchmark seeds were consumed.
Report
Rendered from reports/report.md
Status: negative; primary gate failed and all downstream work stopped. Menagerie remained sealed.
Outcome
The registered recipe installed a fast, exact policy for the six training families but damaged the broader repository agent. Compact improved the trained-family block from 40/48 to 48/48 (+16.7 points, paired 95% CI [+6.3, +27.1]), then fell on four wholly held-out algorithm families from 49/72 to 25/72 (−33.3 points, CI [−44.4, −22.2]). Matched-compute sampling reached 38/72, so compact also lost that comparator by 18.1 points (CI [−27.8, −8.3]).
| Frozen block | Apex replay | Compact | Compact delta |
|---|---|---|---|
| Six trained families, deep | 40/48 (0.833) | 48/48 (1.000) | +0.167 |
| Four held-out families, deep | 49/72 (0.681) | 25/72 (0.347) | −0.333 |
| Four held-out families, matched sampling | 38/72 (0.528) | — | compact −0.181 |
Held-out family success exposes the narrow transfer:
| Family | Apex replay | Compact |
|---|---|---|
dependency_order | 13/18 | 0/18 |
recursive_overlay | 0/18 | 0/18 |
retry_schedule | 18/18 | 18/18 |
ttl_cache | 18/18 | 7/18 |
Compact verification given success was 0.88 versus 1.00 for apex, invalid actions rose from 0.093 to 0.260 per turn, and submit rate fell from 0.889 to 0.306. Commit given verification was preserved (1.00 versus 0.939), so this is not loss of the terminal submit operator after a genuinely successful verification.
The unrelated-context locality guard also failed: median centered non-target logit drift was 0.386 against the frozen 0.15 ceiling. Mean entropy changed only −0.0094, so entropy did not reveal the collateral that the centered-logit audit caught.
Training and artifact validity
Both licensed arms completed the exact registered 584 optimizer steps from pinned Qwen/Qwen3.5-4B revision 851bf6e8, at effective batch 16. Apex ran for 10,007 seconds and compact for 9,633 seconds. The exact checkpointed full-vocabulary loss peaked at 48.58 GB in both arms and crossed the former dense-loss OOM path without approximation. Every one of 128 LoRA modules was nonzero and explicitly merged into each composite checkpoint before vLLM evaluation.
The frozen C53 search policy had covered 129/144 fresh training-family repositories and supplied 376 conjunctive visible+private successes. Replay minimization admitted all 129 covered tasks, collapsed every per-file edit to one patch, and produced 516 rows—exactly 129 each for INSPECT, PATCH, VERIFY, and COMMIT. Weighted action-token mass was exactly 36,110 per operator and compact-plan mass 3,125.8 per operator. All 129 canonical task traces replayed before training. The negative therefore survives the intended data, weighting, padding, merge, and inference-backend checks.
Mechanism diagnosis
The bank compressed away the policy states that matter after something goes wrong.
- On trained tasks, compact produced exactly
INSPECT→PATCH→VERIFY→COMMITon all 48/48 examples, with zero invalid calls, four turns, and only 402 sampled tokens on average. This is a strong family-specific transducer. - Among failed-test observations with another turn available, the apex control chose another
PATCHon 24/26 next transitions (92.3%). Compact chose anotherPATCHon 0/48; it choseVERIFY20 times,INSPECT17 times, and an invalid output 11 times. - After a passing test, both policies retained commit: apex committed on 64/65 next transitions and compact on 22/22.
- Compact encountered rejected exact patches on all 18
recursive_overlaytasks, re-read the file, then repeated the byte-identical rejected patch multiple times on all 18. It had learned the nominal edit shape without learning how to repair an exact-match failure. - On
ttl_cache, compact made a plausible partial fix, observed a visible-test failure, then mostly re-tested or re-inspected instead of revising.retry_schedule, whose repair matched the single local happy-path pattern, stayed perfect.
This sharpens the prior semantic-operator lesson. The interactive-policy curriculum failed because rare verify/commit operators disappeared. This experiment balanced those operator marginals exactly and preserved commit after pass, yet still failed because marginal counts do not specify conditional transitions. A looping agent needs a contingency policy: failed patch → re-inspect and change the edit; failed test → diagnose and revise; passed test → submit. Success-only minimization retained only the last of those.
The locality failure is consistent with additional shared-weight collateral, but it is not a complete causal attribution. Action-only training was intentionally conditional on the necessary gate and did not run. The candidate also displaced some C54 replay examples under the fixed 584-step budget (1.801 mixed-data epochs versus 2.0 apex epochs). The supported conclusion is that the registered compact plan-plus-action recipe fails, not that compact plans alone are harmful.
Decision and learned lessons
The primary gate failed 7 of 11 checks: held-out transfer, both paired-bootstrap lower bounds, matched sampling, invalid actions, verification retention, and locality. Trained-family retention, absolute verification, and both commit checks passed. The orchestrator recorded stop_before_action_only_confirmation_and_menagerie; action-only, confirmation, and Menagerie did not run, and zero benchmark seeds were consumed.
Do not repeat this success-only one-patch bank at another dose. A defensible successor requires a new experiment and fresh procedural splits, with one of two strategies:
- Bank a verifier-conditioned state machine rather than a minimal success trace. Preserve rejected patches, failed tests, changed second edits, and recovery transitions; balance conditional transitions such as
failed_test→PATCH, not only operator totals. Require a frozen perturbation/recovery gate before training. - Keep tool-found scaffolds external and retrieve/execute them conditionally, avoiding a broad shared-weight policy update. Compare this against matched compute and the same held-out algorithm-family transfer.
Either strategy must pass locality and family-disjoint recovery before action-only attribution, confirmation, or Menagerie. The consumed trained_dev and transfer_dev seeds are now analysis data and cannot become future training material or be reused as fresh capability evidence.
The compact machine-readable record is result_receipt.json; paired decisions are in repo_primary_gate.json.
Experiment log 4
Show the running log (4 entries)
Preregistration and implementation
- Re-audited the program state, with C12/C22, C52, C53, C54, the evaluation-only FTPO round-2 repository harness, and the failed interactive-policy curriculum as closest anchors.
- Selected executable replay compression plus operator-balanced compact banking; entropy/varentropy are routing diagnostics only.
- Created ten procedural repository families: six train/search families and four family-disjoint transfer families.
- Verified on CPU that every family starts visible/hidden broken and becomes visible/hidden correct under its host-only oracle.
- Implemented constrained real filesystem tools, answer-region JSON parsing, terminal hidden grading, replay patch deletion, canonical trace reconstruction, operator balancing, and firewall checks.
- Froze seeds, arms, doses, gates, and the conditional benchmark license before result-bearing generation.
- After preregistration commit
462f6274, the first GPU smoke stopped before model load because the current runner template lacked its older local-checkpoint field. Added exact Qwen3.5-4B local-checkpoint support plus an architecture fingerprint test; no scientific seed or output was consumed by the failed attempt. - The corrected GPU smoke loaded the merged C53 checkpoint under vLLM 0.24 and repaired 5/6 one-task-per-family repositories within four turns (implementation evidence only). Its full trajectories remain external and firewall-clean.
- Corrected smoke scoring to require both visible and private tests: the valid count is 4/6; one apparent fifth success passed private edge cases while regressing the visible suite. Added a regression test for this exact failure.
- Replaced equal row-count weighting with exact tokenizer-level action-token mass equality and separately equal compact-plan mass. The action-only control now retains the identical teacher-forced compact text and removes only its plan-span gradient.
- Froze 48 unrelated non-coding contexts for apex→compact centered-logit locality and implemented the full matched-step trainer, explicit composite merge, paired repository analysis, and gate-stopping continuation path while the registered harvest ran.
Registered harvest and bank
- Search teacher covered 129/144 tasks (89.6%; per-family range 58.3–100%) from 576 trajectories and 1,225,314 sampled tokens. There were 376 conjunctive visible+private successes, 370 explicit post-patch verifications, and 351 commits after a pass.
- Replay minimization admitted all 129 covered tasks. Seven source trajectories needed two patch calls, but per-file initial→final collapse reduced every canonical trace to one patch; all canonical visible/private/submission replays passed.
- Built 516 rows: exactly 129 each for INSPECT/PATCH/VERIFY/COMMIT. Exact weighted action-token mass is 36,110 per operator and compact-plan mass 3,125.8 per operator. Longest target sequence is 879 tokens.
- Tokenizer preflight encoded all 4,669 C54 rows and all 516 repository rows with zero skips. Apex replay is exactly 2.0 dataset epochs at 584 steps; both candidates are exactly 1.8011 union epochs at the same 584 steps.
- Every registered pre-training gate passed; training authorized. Menagerie remains sealed.
Training feasibility recovery
- Apex replay with batch 4 × accumulation 4 stopped at optimizer step 52 on a 3,193-token batch: the 9.54 GiB logits allocation exceeded 9.09 GiB free. Loss/gradients were finite, but no adapter or checkpoint was saved.
- Froze the compute-equivalent recovery before rerun: batch 2 × accumulation 8, effective batch 16, 584 steps, 9,344 examples, three apex padding duplicates, and exactly two control epochs. Enabled expandable CUDA segments. All arms share the corrected geometry.
- The 2 × 8 fallback completed three unsaved steps but measured 19–23 s/step. Root cause was dense cross-entropy's full 248k-vocabulary FP32 temporary, not forward activations. Restored the original 4 × 4 geometry with exact gradient-checkpointed 128-position loss chunks; CPU loss/gradient equivalence passes at 1e-6.
- A two-step canary on the eight longest 3,193-token rows passed at batch four, peaking at 48,375,846,912 CUDA bytes. This directly covers the formerly failing allocation path; the smoke adapter is not eligible for evaluation.
Result-bearing training and necessary gate
- Apex replay completed 584/584 optimizer steps in 10,007 seconds; compact completed 584/584 in 9,633 seconds. Both used exact checkpointed full-vocabulary cross-entropy, effective batch 16, and approximately 48.58 GB peak CUDA memory. All 128 LoRA modules were nonzero and explicitly merged for both arms.
- Trained-family deep evaluation: apex 40/48, compact 48/48 (+16.7pp; paired CI [+6.3,+27.1]). Compact used the exact four-operator canonical path on all 48 tasks, with zero invalid calls and 402 versus 1,643 mean sampled tokens.
- Family-disjoint transfer: apex 49/72, compact 25/72 (−33.3pp; CI [−44.4,−22.2]). Matched two-by-four-turn apex sampling scored 38/72, still +18.1pp over compact. Compact invalid calls rose 9.3%→26.0%, verification retention fell 1.00→0.88, and median unrelated-context centered-logit drift was 0.386 versus the 0.15 ceiling.
- Transition audit: after a failed visible test, apex patched again on 24/26 transitions; compact did so on 0/48. After a pass, commit remained intact (64/65 apex, 22/22 compact). All 18 recursive-overlay compact trajectories repeated the identical already-rejected patch. Exact operator marginals preserved the happy path but not verifier-conditioned recovery.
- The primary gate stopped action-only training, confirmation, and Menagerie. Zero benchmark seeds were assigned or consumed. The action-only cancellation leaves plan-gradient attribution unresolved by design.
- Fixed a staged-run footgun found on the negative path: locality correctly wrote a failing receipt and returned status 4, but the orchestrator previously raised before consolidated analysis. Expected gate status is now explicitly accepted, summarized, and returned cleanly; unexpected nonzero statuses still raise, with a regression test.
Data files 1
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
Reproduce
Smoke test
.venv/bin/python experiments/qwen35_4b_repo_search_compress_bank/scripts/run.py --smokeFull run
.venv/bin/python experiments/qwen35_4b_repo_search_compress_bank/scripts/run.py --fullRun steps are documented inside the experiment folder (README and scripts).