Qwen Recurrent VM Repair Policy

Finished2026-06-24imported · line YOperator and Skill Inventories Posttraining and Adaptation Process Control and Tool Use Structured Execution and Compilers GitHub ↗

The one idea you need

Instead of a coder forced to write a flawless program in one shot, picture one who runs the code, reads the wrong output, changes a single line, runs it again, and repeats until satisfied. This trains a small model to debug its own answers that way.

The question

When a small model writes a tiny program to solve a task, does letting it run, inspect the result, and fix one line at a time beat writing it in one go?

What we found

Yes, but only partway. Letting the model run its program, read the output, and fix one line at a time roughly tripled accuracy, from about 11% to 34% on the main test and 16% to 40% on reworded prompts. But a perfect editor reaches near 100% and a fully trained program writer hits 68 to 84%, so most of the possible gain stays unclaimed.

Why it matters

Guided self-repair genuinely lifts a small model's accuracy without retraining the base model, but budget the compute: the payoff is capped by how well the fixing skill is trained, not by the task being unreachable.

One-shot vs step-by-step repair11% → 34%share of answers correct on the main test set

On reworded prompts16% → 40%accuracy when the question is paraphrased

Learning from its own mistakes32% → 59%success rate across three rounds of self-correction practice

Room still leftup to 100%accuracy a perfect edit-picker reaches on the same tasks

On this page

Results at a glance
Overview
Report
Experiment log
Figures
Data files
Reproduce
Related

Results at a glance 2

Step-by-step repair versus one-shot and the best-possible ceilings

How to read

Grouped bars across five test sets; bar height is the share of answers correct, so taller is better. Within each set: one-shot answers, the learned step-by-step repair loop, a perfect fixer, and a fully trained program writer.

Takeaway → The repair bar sits two to three times above the one-shot bar on every set, yet stays well below the perfect-fixer and fully-trained bars, so most possible gain is still unused.

Data table

Evaluation split	base compiler	learned repair (best K)	oracle repair	full supervised direct
Validation	10.9%	34.4%	100%	79.7%
Fresh standard	10.9%	35.9%	100%	75%
Fresh paraphrase	15.6%	39.8%	100%	84.4%
Fresh paired	9.4%	32%	100%	84.4%
Hard composition	14.8%	31.2%	95.3%	68%

Numbers from experiments/qwen_recurrent_vm_repair_policy/analysis/main_summary.csv

Technical framing

Learned recurrent repair vs its ceilings, per evaluation split — The learned repair loop lifts base accuracy 2-3x on every split, but oracle and supervised ceilings show most headroom remains.

Practice rounds sharpen the model's own fixing

How to read

Three lines rising left to right across successive practice rounds; height is a success rate, so up is better. Lines track how often its own fixes worked, how often it got the numbers right, and overall answer accuracy.

Takeaway → All three lines climb steadily, with getting-the-numbers-right rising most, showing that repeatedly correcting the model on the mistakes it actually makes is what drives the gains.

Data table

DAgger round (0 = teacher-only policy)	rollout success	argument-edit accuracy	best validation accuracy
0	—	27%	15.6%
1	32.4%	37.4%	25%
2	42.2%	61.6%	28.9%
3	59.3%	82.3%	34.4%

Numbers from experiments/qwen_recurrent_vm_repair_policy/analysis/main_dagger_progress.csv

Technical framing

DAgger rounds fix the policy's own mistakes — On-policy correction raises rollout success 32% to 59% and argument-edit accuracy 27% to 82% over three DAgger rounds.

In the author’s words from the Report · “Abstract”

The final learned recurrent policy improves over the seed compiler on every split, reaching 34.4% validation accuracy and 39.8% fresh-paraphrase accuracy from 10.9% and 15.6% base accuracy. Three rounds of DAgger raise training-rollout success from 32.4% to 59.3%. The oracle repair policy still reaches 95.3%-100.0%, and the full supervised compiler reaches 68.0%-84.4% direct accuracy. The recurrent loop is useful, but the remaining gap is large.

Overview

Standalone experiment for testing whether a Qwen-attached typed-bytecode compiler improves when the model is trained as a repeated repair transition operator instead of a one-shot program generator.

Small artifacts live here:

experiments/qwen_recurrent_vm_repair_policy/

Large checkpoints live separately here:

large_artifacts/qwen_recurrent_vm_repair_policy/checkpoints/

Layout

src/: experiment and analysis scripts.
runs/: per-run metrics, logs, and dataset manifests.
analysis/: aggregate CSVs and generated figures.
reports/: Markdown and HTML reports.
experiment_log.md: running journal.
checkpoint_manifest.csv: run-to-checkpoint map.

Question

Can one Qwen-conditioned forward pass serve as one loop iteration in an execution-feedback repair process?

The recurrent policy receives:

prompt features + current bytecode + VM final value + VM trace

and predicts either one edit action or STOP. The edited program is executed, then fed back to the same policy for another step.

Report

Rendered from reports/qwen_recurrent_vm_repair_policy_report.md

Abstract

This experiment tests whether a frozen 4B language model can be given a learned recurrent program-repair loop at posttraining time. A compiler reads frozen Qwen/Qwen3-4B token hidden states and emits a typed stack-machine program. A repair policy then runs as an MDP: observe the prompt features, current program, and VM execution trace; choose one edit or STOP; execute the edited program; repeat for up to K private steps.

The result is a partial but real positive. The final learned recurrent policy improves over the seed compiler on every split, reaching 34.4% validation accuracy and 39.8% fresh-paraphrase accuracy from 10.9% and 15.6% base accuracy. Three rounds of DAgger raise training-rollout success from 32.4% to 59.3%. The oracle repair policy still reaches 95.3%-100.0%, and the full supervised compiler reaches 68.0%-84.4% direct accuracy. The recurrent loop is useful, but the remaining gap is large.

Setup

Base model: Qwen/Qwen3-4B, frozen and used as a hidden-state feature extractor.
Trainable modules: typed bytecode compiler and recurrent edit/STOP policy.
Seed compiler examples: 192.
Recurrent-policy training examples: 1024.
Full-supervised ceiling examples: 1024, combined with the seed set.
Evaluation size: 128 per split.
VM: stack bytecode with PUSH, arithmetic, comparisons, table lookup, END, and PAD.
Recurrent budgets: K = 0, 1, 2, 4, 8, 16.
Main run: main_recurrent_vm_repair_crossattn_dagger3_s192_c1024.
Large checkpoints: large_artifacts/qwen_recurrent_vm_repair_policy/checkpoints/main_recurrent_vm_repair_crossattn_dagger3_s192_c1024/.

Method

The seed compiler emits an initial program from frozen Qwen hidden states. The repair policy receives:

token-level frozen Qwen hidden states through cross-attention;
the current bytecode program;
VM validity, final value, stack-top trace, and stack-depth trace;
the current recurrent step index.

It predicts one of three action kinds: STOP, edit an opcode slot, or edit an argument slot. Oracle trajectories are generated by comparing the current program to the gold program and taking one edit at a time. DAgger then rolls out the learned policy, labels the states it actually visits with the oracle next action, and retrains on the accumulated state set.

Main Results

Split	Base	Learned STOP	Forced edits	Oracle repair	Full sup. direct	Full sup. search
Validation	10.9%	34.4% (K=2)	32.0% (K=4)	100.0%	79.7%	93.0%
Fresh standard	10.9%	35.9% (K=2)	32.0% (K=4)	100.0%	75.0%	89.8%
Fresh paraphrase	15.6%	39.8% (K=4)	37.5% (K=4)	100.0%	84.4%	96.9%
Fresh paired	9.4%	32.0% (K=16)	29.7% (K=4)	100.0%	84.4%	93.0%
Hard composition	14.8%	31.2% (K=2)	33.6% (K=8)	95.3%	68.0%	89.8%

Final recurrent repair K-curves

Learned recurrent policy versus ceilings

DAgger Dynamics

Phase	Source success	Action acc.	Arg acc.	Val best	Hard best
Teacher policy	100.0%	56.4%	27.0%	15.6% (K=2)	21.1% (K=2)
DAgger round 1	32.4%	56.9%	37.4%	25.0% (K=16)	29.7% (K=4)
DAgger round 2	42.2%	69.2%	61.6%	28.9% (K=16)	28.1% (K=4)
DAgger round 3	59.3%	80.6%	82.3%	34.4% (K=2)	31.2% (K=2)

DAgger progress

The important signal is not just lower training loss. Repeated on-policy correction changes rollout success: the policy succeeds on 32.4% of first-round visited programs, 42.2% in round 2, and 59.3% in round 3. Argument accuracy also rises from 37.4% after round 1 to 82.3% after round 3, which matters because argument edits carry the numeric content of the prompt.

STOP Behavior

STOP behavior

Learned STOP is helpful at moderate K, but it remains imperfect. The final policy often peaks at K=2 or K=4; larger K can introduce false STOPs or unnecessary edits, especially on paired prompts. This is a learning problem, not an environment reachability problem, because the oracle repair policy remains near perfect.

Compiler Ceiling

Compiler ceiling

The full-supervised compiler ceiling is high: direct accuracy reaches 68.0-84.4%, and answer-verified search reaches 89.8-96.9%. That confirms the frozen Qwen features contain enough information for the bytecode task. The recurrent policy has learned a meaningful part of the repair process, but not enough to match dense supervised program learning.

Interpretation

This experiment supports a narrow claim: a Qwen-attached recurrent VM repair loop can turn extra private compute steps into better answers under learned policy control. The strongest evidence is the DAgger progression plus the final learned K-curves. The policy is not merely a static reranker; it executes a sequence of edits, observes the VM after each edit, and improves as the state distribution is corrected.

The result does not support a broad claim of universal intelligence gain. The oracle and full-supervised ceilings show much more is reachable, but the learned policy still leaves most of the oracle gap open. The bottleneck is now policy learning: STOP calibration, choosing the right number of edits, and robust argument edits under prompt variation.

Next Experiments

Add a learned value function over VM states and train the policy with advantage-weighted imitation so STOP is tied to expected answer improvement.
Distill oracle trajectories with dense intermediate value targets, not only next-action labels.
Use beam-style recurrent repair with a learned verifier over resulting VM states, then distill the best repair path back into the one-action policy.
Replace gold-program edit distance with answer-equivalent repair targets so the policy is not punished for alternate correct programs.
Test a token-output bridge where the same recurrent VM state is fed back into Qwen over multiple turns until a learned halt decision fires.

Artifacts

Script: experiments/qwen_recurrent_vm_repair_policy/src/qwen_recurrent_vm_repair_policy_experiment.py
Analysis: experiments/qwen_recurrent_vm_repair_policy/src/analyze_qwen_recurrent_vm_repair_policy.py
Main run: experiments/qwen_recurrent_vm_repair_policy/runs/main_recurrent_vm_repair_crossattn_dagger3_s192_c1024/
Aggregate metrics: experiments/qwen_recurrent_vm_repair_policy/analysis/
Markdown report: experiments/qwen_recurrent_vm_repair_policy/reports/qwen_recurrent_vm_repair_policy_report.md
HTML report: experiments/qwen_recurrent_vm_repair_policy/reports/qwen_recurrent_vm_repair_policy_report.html

Experiment log 2

Show the running log (2 entries, 2026-06-24)

2026-06-24

Created a fresh standalone experiment directory.
Selected intervention: recurrent VM repair policy with learned STOP.
Core idea: train a Qwen-attached policy so that one forward pass proposes one edit to the current VM program, then the VM executes the edited program and feeds the new execution state back into the same policy.
Training plan:
- train a seed compiler from frozen Qwen hidden states;
- collect oracle edit trajectories from seed programs to correct programs;
- train a teacher-forced edit/STOP policy;
- run the learned policy to collect off-trajectory states;
- retrain with DAgger-style mixed teacher/off-policy states;
- evaluate K-step curves and learned STOP behavior.
Large artifacts will be stored in large_artifacts/qwen_recurrent_vm_repair_policy/checkpoints/.

Iteration Notes

Implemented the standalone VM core and recurrent repair experiment script.
Smoke run smoke_recurrent_vm_repair completed end to end.
Smoke diagnostics exposed two implementation issues before scaling: STOP labels were too sparse, and quick validation was reading the base row instead of the learned K-step row. Patched both.
Smoke run smoke_recurrent_vm_repair_v2 verified STOP-state augmentation, but the flat action head was too sparse.
Replaced the flat edit-action head with factorized kind, slot, opcode, and argument heads.
Smoke run smoke_recurrent_vm_repair_factorized verified the factorized policy. The oracle edit curve improved strongly with K, confirming that the recurrent setup has headroom, but the tiny learned policy was not yet useful.
Pilot run pilot_recurrent_vm_repair_s96_c256 completed. The oracle edit loop reached 95-100% accuracy by K=8-16 on validation and paired/hard generalization splits, but the learned policy mainly failed through false STOP decisions on incorrect programs.
Patched the policy rollout path with optional conservative STOP margining, typed VM action masking, and component-level policy diagnostics. This keeps the recurrent VM hypothesis unchanged while targeting the observed learned policy failure mode.
Pilot run pilot_recurrent_vm_repair_masked_s96_c256 showed that typed masking and STOP margining increased imitation diagnostics but harmed final accuracy. Component diagnostics identified argument prediction as the weak point: kind, slot, and opcode accuracy were high, while argument accuracy remained low.
Patched the repair policy to cross-attend to frozen Qwen token hidden states, rather than relying only on a pooled prompt vector. This directly targets numeric/detail recall during edit prediction.
Pilot run pilot_recurrent_vm_repair_crossattn_s96_c256 improved learned rollout behavior relative to the pooled-policy pilot: DAgger rollout success rose from roughly 25% to 30%, and the best validation learned K-curve rose to about 20%. The policy remains far below the oracle, so the next patch adds multiple DAgger rounds to attack off-policy distribution shift.
Pilot run pilot_recurrent_vm_repair_crossattn_dagger3_s96_c256 made the strongest learned-policy improvement so far. The third DAgger trajectory reached roughly 49% rollout success, component argument accuracy rose to roughly 70%, and learned validation accuracy reached roughly 27% at K=8 while the oracle remained near 100%. Main run will use cross-attention and three DAgger rounds.
Main run main_recurrent_vm_repair_crossattn_dagger3_s192_c1024 completed. The final learned recurrent policy improved all splits over the seed compiler: validation 10.9% -> 34.4%, fresh standard 10.9% -> 35.9%, fresh paraphrase 15.6% -> 39.8%, fresh paired 9.4% -> 32.0%, and hard composition 14.8% -> 31.2% with learned STOP. DAgger rollout success rose from 32.4% to 59.3% across three rounds, and argument action accuracy rose to 82.3%. The oracle repair ceiling remained 95.3-100.0%, so the learned policy is useful but still far from the reachable repair ceiling.
Generated aggregate CSVs, five figures, a standalone Markdown report, and a standalone HTML report in analysis/ and reports/.

Figures 5

main oracle gap by split · analysis/figures/

Data files 24

Result tables and metrics copied from the experiment folder — preview inline or open the raw file.

4 more result files not shown here — browse the full folder on GitHub.

Reproduce

This entry is source code or analysis only — there is no separate run to reproduce.

Browse the experiment folder on GitHub ↗