Qwen Search-Augmented Rollout Distillation
The one idea you need
Picture a repair apprentice fixing a program one instruction at a time: it edits, runs the code, reads the result, edits again. Handed the correct single fix at every dead-end, it learns only local corrections, never a whole plan that reaches working code.
The question
If you show a code-editing model the verified correct fix for every mistake it makes, will it learn to reliably solve whole tasks?
What we found
No. An automatic search found a verified correct fix for 98% of the dead-ends the model wandered into, yet retraining on those single fixes matched or trailed the simpler training on four of five test sets. The model could pick the right next edit 60% of the time but solved at most 37.5% of whole tasks. Local fixes never became a winning plan.
Why it matters
If you build a step-by-step editing agent, don't expect verified next-step fixes to teach it. The hard part is planning a full sequence that ends in a passing result: reward whole runs, not isolated single edits.
On this page
Results at a glance 3
How to read
Each cluster is one test set. Bars compare the plain model answering directly, the editor trained the normal way, the editor retrained on found fixes, and a perfect edit-sequence teacher. Taller means more tasks solved, so higher is better.
Takeaway → The fix-retrained bars sit at or below the normally-trained bars on most test sets, while the perfect-teacher bars tower far above every trained version.
Data table
| Evaluation split | native Qwen | behavior-cloned (active) | search-distilled (active) | oracle teacher |
|---|---|---|---|---|
| Val mixed | 43.8% | 6.2% | 12.5% | 100% |
| Fresh standard | 50% | 12.5% | 12.5% | 87.5% |
| Fresh paraphrase | 25% | 37.5% | 31.2% | 93.8% |
| Fresh paired | 25% | 18.8% | 18.8% | 100% |
| Hard composition | 50% | 37.5% | 18.8% | 68.8% |
Numbers from experiments/qwen_search_augmented_rollout_distillation/analysis/main_summary.csv
Technical framing
Active VM-controller accuracy by split: search distillation did not beat behavior cloning — Search distillation helped only val mixed (6.2% to 12.5%); the oracle gap (69-100%) shows large headroom the recipe failed to capture.
How to read
The horizontal axis is how many edits the model is allowed; the vertical axis is the share of tasks solved. Each line is one test set, and a higher line means more solved.
Takeaway → Only one test set climbs as more edits are allowed; the rest stay flat or drop, so extra editing does not reliably help.
Data table
| Edit budget K | val mixed | fresh standard | fresh paraphrase | fresh paired | hard composition |
|---|---|---|---|---|---|
| 0 | 12.5% | 0% | 0% | 37.5% | 12.5% |
| 4 | 12.5% | 6.2% | 12.5% | 18.8% | 0% |
| 8 | 12.5% | 0% | 12.5% | 18.8% | 6.2% |
| 12 | 12.5% | 6.2% | 25% | 18.8% | 6.2% |
Technical framing
Accuracy vs edit budget K for the search-distilled policy (greedy learned mode) — No clean monotonic K-scaling: more VM edits help only fresh paraphrase, while other splits stay flat or degrade from the K=0 blank baseline.
How to read
Bars are grouped by training stage, from basic training epochs to fix-retraining epochs. Colors show three kinds of per-step correctness: picking the edit type, its details, and when to stop. Taller means more steps right.
Takeaway → Picking the right edit peaked during basic training and fell after fix-retraining, and stronger per-step scores never produced better whole solutions.
Data table
| Training phase / epoch | action accuracy | argument accuracy | STOP accuracy |
|---|---|---|---|
| BC epoch 1 | 37.3% | 21.5% | 87.8% |
| BC epoch 2 | 60% | 44.1% | 80.5% |
| Search epoch 1 | 38.2% | 42.6% | 73.4% |
| Search epoch 2 | 50.4% | 59.5% | 79.7% |
Numbers from report table (reports/search_augmented_rollout_distillation_report.md, Training Diagnostics)
Technical framing
Local training accuracy by phase and epoch: search labels lowered action accuracy — Retraining on search-repaired labels dropped action accuracy from 60.0% to 50.4%, and local gains never translated into better active rollouts.
In the author’s words from the Report · “Interpretation”
The experiment gives a clear negative result for this specific recipe. The bottleneck is not simply obtaining verified repair labels: the main run found verified completions for 98.1% of policy-visited states. The problem is turning those labels into a policy that chooses useful global trajectories at inference time. The strongest evidence is the gap between local and global metrics. Behavior cloning reached 60.0% action accuracy, and the oracle teacher reached 68.8% to 100.0% by split, but the best active deployable VM score was only 37.5%. Search-distillation increased neither the best active score nor the K-scaling pattern.
Overview
This standalone experiment trains Qwen/Qwen3-4B as a recurrent controller for a typed bytecode VM.
The model receives a task prompt plus dense projected VM-state tokens. It predicts one VM edit action or STOP, the VM executes the edited program, and the same model is called again for the next recurrent step.
The central intervention is search-augmented rollout distillation: policy-visited states are labeled by bounded answer-verified repair search, not only by a gold-program edit trace. The model also receives pairwise action-ranking supervision from repair-positive and repair-negative edits.
Large checkpoints are stored outside this directory:
large_artifacts/qwen_search_augmented_rollout_distillation/checkpoints/Read experiment_log.md for the iteration record. Final reports are written under reports/.
Main outputs:
reports/search_augmented_rollout_distillation_report.mdreports/search_augmented_rollout_distillation_report.htmlanalysis/main_active_accuracy.pnganalysis/main_k_curves_learned.pngruns/main_search_r1_rank00_e2_20260624/
Report
Rendered from reports/search_augmented_rollout_distillation_report.md
Verdict
Search-augmented rollout distillation did not improve the main active VM controller. The behavior-cloned VM controller already produced the best active score on three of five splits, tied search on one, and search improved only val mixed from 6.2% to 12.5%, still far below native Qwen. The oracle teacher remained much higher at 68.8% to 100.0%, so the task and VM action space still have large headroom.
The main positive result is narrow: the behavior-cloned VM controller beat native Qwen on fresh paraphrase tasks when using active K>0 edits (37.5% vs 25.0%). The search-distilled controller did not preserve that gain and did not close the oracle gap.
What Was Tested
This experiment trains Qwen/Qwen3-4B as a recurrent controller for a typed bytecode VM. Each recurrent step is one Qwen forward pass over the task prompt plus dense VM-state tokens. The model predicts one structured edit action or STOP; the VM executes the current program; then the updated VM state is fed back into the model for the next step.
The intervention is search-augmented rollout distillation. After behavior cloning on gold traces, the learned policy is rolled out on training tasks. For each policy-visited state, bounded repair search finds a verified completion when possible. The model is then trained on the first action of that repaired trajectory.
Main Accuracy
These are active K>0 VM-editing scores except for the explicit blank column. This avoids counting passive K=0 blank-program hits as real recurrent computation.
| Split | Native Qwen | Blank K=0 | BC VM active | Search VM active | Oracle teacher |
|---|---|---|---|---|---|
| Val mixed | 43.8% | 12.5% | 6.2% | 12.5% | 100.0% |
| Fresh standard | 50.0% | 0.0% | 12.5% | 12.5% | 87.5% |
| Fresh paraphrase | 25.0% | 0.0% | 37.5% | 31.2% | 93.8% |
| Fresh paired | 25.0% | 37.5% | 18.8% | 18.8% | 100.0% |
| Hard composition | 50.0% | 12.5% | 37.5% | 18.8% | 68.8% |
K Curves
The greedy learned policy shows no clean monotonic K-scaling. Some splits improve with more edits, but others stay flat or degrade. The oracle curve confirms that high accuracy is reachable in the same VM environment when the trajectory is chosen correctly.
Training Diagnostics
| Phase | Epoch | Action acc | Arg acc | STOP acc | Pair-rank acc | States |
|---|---|---|---|---|---|---|
| bc_policy | 1 | 37.3% | 21.5% | 87.8% | 0.0% | 901 |
| bc_policy | 2 | 60.0% | 44.1% | 80.5% | 0.0% | 901 |
| search_r1_policy | 1 | 38.2% | 42.6% | 73.4% | 55.2% | 1703 |
| search_r1_policy | 2 | 50.4% | 59.5% | 79.7% | 62.1% | 1703 |
Behavior cloning reached 60.0% local action accuracy, but active solve accuracy remained much lower. Search-distillation collected many verified repair labels, but training on those labels reduced local action accuracy to 50.4% and did not improve the active controller.
Repair Diagnostics
Search-state collection found verified completions for 787 of 802 policy-visited states (98.1%). It also saw 41 false-stop states and 43.8% rollout success before retraining.
Pilot Result
The first pilot used pairwise positive-vs-negative ranking with weight 0.5. It did not improve rollout accuracy and made the second on-policy collection worse. The selected main recipe disabled the ranking loss and trained two epochs on search-labeled states.
Interpretation
The experiment gives a clear negative result for this specific recipe. The bottleneck is not simply obtaining verified repair labels: the main run found verified completions for 98.1% of policy-visited states. The problem is turning those labels into a policy that chooses useful global trajectories at inference time.
The strongest evidence is the gap between local and global metrics. Behavior cloning reached 60.0% action accuracy, and the oracle teacher reached 68.8% to 100.0% by split, but the best active deployable VM score was only 37.5%. Search-distillation increased neither the best active score nor the K-scaling pattern.
Next Experiment
The next high-impact experiment should stop treating repaired trajectories as ordinary one-step imitation labels. The more direct attack is rollout-level optimization: sample complete VM rollouts, score them with exact VM reward and false-stop penalties, then train preference or policy-gradient updates over full trajectories. The assets from this experiment are enough to do that: a recurrent dense-state controller, a value/distance head, exact executable rewards, and a repair procedure that can produce successful contrastive rollouts.
Experiment log 1
Show the running log (1 entry, 2026-06-24)
2026-06-24
- Created a fresh standalone experiment directory.
- Selected intervention: search-augmented rollout distillation for a dense-state recurrent VM agent.
- Core idea: after behavior cloning on oracle traces, roll out the learned policy, run bounded answer-verified repair search from policy-visited states, and train on the first repair edit plus pairwise positive-vs-negative action ranking.
- Main failure targeted: local edit competence without reliable global trajectory selection.
- Large artifacts will be stored under
large_artifacts/qwen_search_augmented_rollout_distillation/checkpoints/. Implemented the standalone harness:
- behavior cloning from gold VM traces,
- on-policy state collection,
- bounded answer-verified repair search from each visited state,
- first-repair-action supervision,
- same-state positive-vs-negative action ranking,
- greedy, value-gated, forced-step, and value-beam evaluation modes.
- Static syntax check passed for the training script and shared VM core.
- Smoke run
smoke_search_rollout_distill_20260624completed end to end. It exposed a weak search-label source: shallow two-edit repair found verified completions for only 1 of 12 policy-visited states. - Added bounded target-guided completion: candidate first actions are evaluated by whether they can still reach a verified answer within the shortest remaining edit budget.
- Smoke run
smoke_search_rollout_distill_guided_20260624completed end to end. Search found verified completions for 11 of 11 policy-visited states, so the pilot will use guided repair withrepair_max_edits=12. Pilot
pilot_search_r2_rank05_20260624was stopped after the second-round state collection because the first search-distilled policy did not improve rollout accuracy and the second-round collection regressed.- BC K=8 best deployable accuracy: val mixed 25.0%, fresh standard 8.3%,
fresh paraphrase 8.3%, paired 0.0%, hard 8.3%.
- Search-r1 K=8 best deployable accuracy: val mixed 16.7%, fresh standard
8.3%, fresh paraphrase 8.3%, paired 0.0%, hard 8.3%.
- Search-r1 state collection found verified completions for 492/502 states,
but one epoch of rank-weight 0.5 training only reached 53.6% pairwise rank accuracy and reduced action accuracy.
- Search-r2 collection had 46 false-stop states and lower rollout success,
so the run was intentionally interrupted.
- Added
--eval_modesso subsequent pilots can skip expensive value-beam sweeps during iteration. Pilot
pilot_search_r1_rank00_e2_20260624completed with pairwise ranking disabled and two search-distillation epochs.- Search-r1 K=8 improved best deployable accuracy over BC on fresh paired
(0.0% to 16.7%), fresh paraphrase (8.3% to 16.7%), and fresh standard (8.3% to 16.7%).
- Hard composition stayed flat at 8.3% and val mixed stayed flat at 16.7%.
- Search-r1 action accuracy recovered to 39.6%, better than the rank-0.5
pilot but still below BC action accuracy.
- Main recipe selected: rank loss disabled, one search-augmented on-policy round, two search epochs, no value-beam in the broad evaluation sweep.
Main run
main_search_r1_rank00_e2_20260624completed.- Native Qwen accuracy: val mixed 43.8%, fresh standard 50.0%, fresh
paraphrase 25.0%, fresh paired 25.0%, hard composition 50.0%.
- BC VM active K>0 best: val mixed 6.2%, fresh standard 12.5%, fresh
paraphrase 37.5%, fresh paired 18.8%, hard composition 37.5%.
- Search VM active K>0 best: val mixed 12.5%, fresh standard 12.5%, fresh
paraphrase 31.2%, fresh paired 18.8%, hard composition 18.8%.
- Oracle teacher best: val mixed 100.0%, fresh standard 87.5%, fresh
paraphrase 93.8%, fresh paired 100.0%, hard composition 68.8%.
- Search-state repair found verified completions for 787/802 states, but
search retraining reduced local action accuracy from 60.0% to 50.4% and did not improve the active VM controller.
- Generated charts under
analysis/and standalone reports underreports/.
Figures 5
Data files 12
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
analysis/main_summary.csv292 Bruns/main_search_r1_rank00_e2_20260624/metrics.csv21 kBruns/main_search_r1_rank00_e2_20260624/native_qwen_metrics.csv329 Bruns/main_search_r1_rank00_e2_20260624/results.json76 kBruns/pilot_search_r1_rank00_e2_20260624/metrics.csv22 kBruns/pilot_search_r1_rank00_e2_20260624/results.json63 kBruns/pilot_search_r2_rank05_20260624/metrics.csv27 kBruns/pilot_search_r2_rank05_20260624/native_qwen_metrics.csv323 Bruns/smoke_search_rollout_distill_20260624/metrics.csv13 kBruns/smoke_search_rollout_distill_20260624/results.json48 kBruns/smoke_search_rollout_distill_guided_20260624/metrics.csv14 kBruns/smoke_search_rollout_distill_guided_20260624/results.json49 kB
Reproduce
This entry is source code or analysis only — there is no separate run to reproduce.