Qwen Action-Conditioned VM-ECHO Policy Iteration
The one idea you need
The model writes a tiny program to solve each puzzle, then runs every draft in a sandbox to watch what it computes. A second model grades those drafts by their consequences, trying to pick the one whose output actually answers the puzzle.
The question
If a model can run each of its draft programs and watch what they compute, can it learn to pick the correct one and get better at writing them?
What we found
Barely. Learning to grade drafts by their run results nudged picking accuracy only from about 10% to 11%, far short of the 37% reachable by always choosing the best available draft. It reliably picked programs that ran, just not ones that were correct. Yet feeding even those imperfect picks back still lifted directly-written program accuracy from 18% to 23%.
Why it matters
When a small model self-improves by running its own candidate answers, the bottleneck isn't generating candidates or reusing good ones — it's the judge. Invest in teaching it to compare rival candidates, not to polish its own writing.
On this page
Results at a glance 5
How to read
Each cluster is a family of puzzles; taller bars mean more solved by writing one program with no retries. The four bars are the frozen starting model, two ways of reusing its own drafts, and a model simply shown the correct answers.
Takeaway → The reuse bars hover right around the starting model — sometimes a touch above, often a touch below — while the shown-answers bar towers far above, so self-teaching helps only slightly.
Data table
| Held-out split | seed (frozen compiler) | learned-selector distill | answer-verified distill | full supervision (ceiling) |
|---|---|---|---|---|
| fresh paired | 18% | 22.7% | 16.4% | 82% |
| fresh paraphrase | 15.6% | 14.8% | 14.8% | 72.7% |
| fresh standard | 14.8% | 19.5% | 18% | 69.5% |
| hard composition | 14.8% | 15.6% | 12.5% | 51.6% |
Numbers from analysis/main_metrics.csv
Technical framing
Direct bytecode-emission accuracy by training phase — Distilling on learned-selected candidates gives modest direct gains (18.0->22.7% on fresh paired), far below the full-supervision ceiling.
How to read
Each cluster is a family of puzzles; taller bars mean more solved when the model may generate and test several drafts and keep a working one. Bars compare the starting model against two ways of reusing its own drafts.
Takeaway → Reusing drafts helps most here — the hardest puzzle family rises from 35% to 52% solved — even though the drafts were picked imperfectly.
Data table
| Held-out split | seed (frozen compiler) | learned-selector distill | answer-verified distill |
|---|---|---|---|
| fresh paired | 51.6% | 64.1% | 57% |
| fresh paraphrase | 35.2% | 43% | 48.4% |
| fresh standard | 44.5% | 52.3% | 57% |
| hard composition | 35.2% | 52.3% | 48.4% |
Numbers from analysis/main_metrics.csv
Technical framing
Answer-search accuracy: distillation gains are largest under search — Even imperfect learned targets help most under candidate search: hard-composition search rises from 35.2% to 52.3%.
How to read
Bars group practice puzzles and fresh ones; taller means the chosen draft is correct more often. Three bars per group: the model's raw hunch, its learned grader, and the score if it always grabbed its best available draft.
Takeaway → The learned grader barely differs from the raw hunch while the best-possible bar sits far higher, showing weak judgment is the real bottleneck.
Data table
| Candidate set | base (compiler logprob) | learned selector | oracle (best candidate) |
|---|---|---|---|
| training candidates | 15.2% | 14.6% | 40.7% |
| validation candidates | 10.2% | 10.9% | 36.7% |
Numbers from analysis/main_consequence_selection_eval.csv
Technical framing
Learned consequence selector roughly matches compiler logprob, far from oracle — The selector recovers almost none of the oracle gap (10.2% -> 10.9% vs 36.7% oracle), so selector precision is the bottleneck.
How to read
Two puzzle families; taller bars mean more solved by direct writing afterward. Bars compare reusing every draft against keeping only the higher-scoring drafts.
Takeaway → Keeping only higher-scoring drafts raises fresh-puzzle accuracy, showing the quality of the reused drafts matters more than their quantity.
Data table
| Pilot split | threshold 0.0 (all targets, 12.5% precision) | threshold 0.7 (filtered, 24.6% precision) |
|---|---|---|
| fresh paired | 6.2% | 9.4% |
| hard composition | 12.5% | 12.5% |
Numbers from analysis/pilot_threshold_comparison.csv
Technical framing
Pilot: higher target threshold trades volume for precision — Filtering learned targets at score 0.7 halves target count but doubles precision and improves pilot fresh-paired direct accuracy.
How to read
Each cluster is a puzzle family; taller bars mean more solved by writing directly. Four bars: the starting model, two self-teaching methods, and a model shown correct answers.
Takeaway → Self-teaching bars edge only a few points above the start, dwarfed by the shown-answers bars reaching 52 to 82%.
Data table
| Evaluation split | Seed supervised | Learned-policy distill | Answer-verified distill | Full supervised (ceiling) |
|---|---|---|---|---|
| fresh_paired | 18% | 22.7% | 16.4% | 82% |
| fresh_standard | 14.8% | 19.5% | 18% | 69.5% |
| fresh_paraphrase | 15.6% | 14.8% | 14.8% | 72.7% |
| hard_composition | 14.8% | 15.6% | 12.5% | 51.6% |
| val_mixed | 10.2% | 11.7% | 12.5% | 74.2% |
Numbers from experiments/qwen_action_conditioned_vm_echo_policy_iteration/analysis/main_metrics.csv
Technical framing
Per-split direct emission: self-distillation gains are tiny next to the full-supervision ceiling — Distilling self-generated candidates lifts direct emission by a few points at most; full supervision reaches 52-82% on the same splits.
In the author’s words from the Report · “Abstract”
The learned consequence selector did not close the oracle gap in the main run: on validation candidates it moved from 10.2% base top-1 accuracy to 10.9%, with an oracle of 36.7%. However, filtered learned-policy distillation still improved the compiler on several generalization splits. Fresh-paired direct accuracy increased from 18.0% to 22.7%, and hard-composition answer-search accuracy increased from 35.2% to 52.3%. The fully supervised ceiling remained much higher: fresh-paired direct accuracy reached 82.0%.
Overview
This standalone experiment tests whether a frozen-Qwen bytecode compiler can improve by learning the consequences of its own generated candidate programs.
Hypothesis
If the model proposes a candidate bytecode program, executing that candidate in the VM gives dense action-conditioned feedback: validity, final value, stack trace, and whether the candidate solves the prompt. A consequence model trained on those observations should rank candidates better than raw compiler logprob, and distilling the best learned candidates back into the compiler should improve direct bytecode emission.
Layout
src/typed_bytecode_core.py: standalone task generator, bytecode VM, typed decoding, and candidate search.src/qwen_action_conditioned_vm_echo_policy_iteration_experiment.py: Qwen feature extraction, compiler training, candidate generation, consequence model training, learned reranking, and policy distillation.src/analyze_qwen_action_conditioned_vm_echo_policy_iteration.py: aggregate metrics, charts, Markdown report, and HTML report.runs/: per-run manifests, logs, and metrics.analysis/: aggregate CSVs and figures.reports/: final writeups.large_artifacts/qwen_action_conditioned_vm_echo_policy_iteration/checkpoints/: checkpoints kept outside the experiment directory.
Primary Metrics
- Greedy direct executable accuracy.
- Learned candidate-selection accuracy.
- Answer-verified oracle/search accuracy.
- Oracle gap recovered by learned consequence ranking.
- Distilled compiler direct accuracy after training on learned-selected candidates.
Report
Rendered from reports/qwen_action_conditioned_vm_echo_policy_iteration_report.md
Abstract
This experiment tests a candidate-conditioned route from program search to better direct program emission. A frozen-Qwen compiler first proposes typed bytecode candidates. A consequence model then receives the prompt representation and a candidate program, and learns from VM execution labels: validity, final value, stack trace, and whether that candidate solves the prompt. The learned selector is then used to choose policy-distillation targets for the compiler.
The main result is mixed. The learned consequence selector did not close the oracle gap in the main run: on validation candidates it moved from 10.2% base top-1 accuracy to 10.9%, with an oracle of 36.7%. However, filtered learned-policy distillation still improved the compiler on several generalization splits. Fresh-paired direct accuracy increased from 18.0% to 22.7%, and hard-composition answer-search accuracy increased from 35.2% to 52.3%. The fully supervised ceiling remained much higher: fresh-paired direct accuracy reached 82.0%.
Setup
- Base model:
Qwen/Qwen3-4B, used only as a frozen hidden-state feature extractor. - Seed examples:
192. - Candidate-training prompts:
1024. - Full-supervised examples:
1024. - Fresh split size:
128. - Candidate search: top-k
3, second-order argument pairs8, max candidates256. - Learned-target threshold:
0.7. - Checkpoints:
large_artifacts/qwen_action_conditioned_vm_echo_policy_iteration/checkpoints/main_action_vm_echo_s192_thr070/.
Main Results
| Phase | Split | Direct | Answer search | Oracle | Learned selector | Program exact |
|---|---|---|---|---|---|---|
| Seed | fresh_paired | 18.0% | 51.6% | 51.6% | 0.8% | |
| Seed | fresh_paraphrase | 15.6% | 35.2% | 35.2% | 3.1% | |
| Seed | fresh_standard | 14.8% | 44.5% | 44.5% | 2.3% | |
| Seed | hard_composition | 14.8% | 35.2% | 35.2% | 1.6% | |
| Learned distill | fresh_paired | 22.7% | 64.1% | 64.1% | 17.2% | 3.1% |
| Learned distill | fresh_paraphrase | 14.8% | 43.0% | 43.0% | 7.0% | 0.8% |
| Learned distill | fresh_standard | 19.5% | 52.3% | 52.3% | 17.2% | 3.1% |
| Learned distill | hard_composition | 15.6% | 52.3% | 52.3% | 13.3% | 1.6% |
| Answer distill | fresh_paired | 16.4% | 57.0% | 57.0% | 18.8% | 0.8% |
| Answer distill | fresh_paraphrase | 14.8% | 48.4% | 48.4% | 13.3% | 2.3% |
| Answer distill | fresh_standard | 18.0% | 57.0% | 57.0% | 18.8% | 4.7% |
| Answer distill | hard_composition | 12.5% | 48.4% | 48.4% | 14.1% | 0.0% |
| Full sup. | fresh_paired | 82.0% | 94.5% | 94.5% | 32.0% | 70.3% |
| Full sup. | fresh_paraphrase | 72.7% | 90.6% | 90.6% | 30.5% | 57.0% |
| Full sup. | fresh_standard | 69.5% | 85.9% | 85.9% | 28.1% | 46.9% |
| Full sup. | hard_composition | 51.6% | 79.7% | 79.7% | 28.9% | 28.9% |
Learned Candidate Selection
The action-conditioned selector learned a real but weak signal. It selected mostly valid programs, but it did not reliably select answer-correct programs at main scale.
Training candidate set:
- Groups:
1024prompts. - Candidates:
246910programs. - Positive candidate rate: 9.3%.
- Prompts with at least one positive candidate: 40.7%.
Policy-Distillation Targets
| phase | targets | oracle_found_rate | selected_correct_rate | selected_valid_rate | changed_rate | mean_selected_score |
|---|---|---|---|---|---|---|
| learned_policy | 470 | 40.7% | 28.9% | 99.6% | 100.0% | 0.884 |
| answer_verified | 417 | 40.7% | 100.0% | 100.0% | 62.6% |
The learned selector chose more targets than answer verification, but with much lower known precision. The useful signal is that even imperfect learned targets improved some direct and search metrics, suggesting that consequence-conditioned filtering is not useless. The limiting factor is selector precision.
Training Dynamics
Threshold Pilot
A small threshold sweep was used to avoid using every learned-selected target. The higher threshold traded volume for precision and produced better pilot direct accuracy.
Interpretation
The candidate-conditioned objective is closer to the desired mechanism than target-trace-only supervision: it asks what a proposed program will do, not only what the correct program should look like. But this implementation still underfits the hardest part: comparing candidate consequences to the prompt-implied answer. The next improvement should strengthen the selector, not the compiler head. Good next changes are pairwise preference training over candidates from the same prompt, harder negative mining, and using the compiler's answer representation directly inside the consequence model.
Artifacts
experiments/qwen_action_conditioned_vm_echo_policy_iteration/runs/main_action_vm_echo_s192_thr070/metrics.csvexperiments/qwen_action_conditioned_vm_echo_policy_iteration/runs/main_action_vm_echo_s192_thr070/target_selection.csvexperiments/qwen_action_conditioned_vm_echo_policy_iteration/runs/main_action_vm_echo_s192_thr070/consequence_train_log.csvexperiments/qwen_action_conditioned_vm_echo_policy_iteration/analysis/main_metrics.csvexperiments/qwen_action_conditioned_vm_echo_policy_iteration/reports/qwen_action_conditioned_vm_echo_policy_iteration_report.mdexperiments/qwen_action_conditioned_vm_echo_policy_iteration/reports/qwen_action_conditioned_vm_echo_policy_iteration_report.html
Experiment log 5
Show the running log (5 entries, 2026-06-24)
2026-06-24
- Created a fresh standalone experiment directory.
- Selected intervention: action-conditioned VM-ECHO policy iteration.
- Core idea: generate candidate bytecode programs from a frozen-Qwen compiler, execute those candidates in the VM, train a consequence model on candidate observations, then distill consequence-selected candidates back into the compiler.
- Large artifacts will be stored in
large_artifacts/qwen_action_conditioned_vm_echo_policy_iteration/checkpoints/.
Implementation Notes
- Built
src/qwen_action_conditioned_vm_echo_policy_iteration_experiment.py. - Built
src/analyze_qwen_action_conditioned_vm_echo_policy_iteration.py. - Reused the typed bytecode VM core as local experiment source in
src/typed_bytecode_core.py, with checkpoint paths redirected to the experiment-specific large-artifact directory. - Added checkpoint tracking in
checkpoint_manifest.csv. - Generated both Markdown and HTML reports under
reports/.
Iteration Record
smoke_action_vm_echo: verified that Qwen feature extraction, candidate generation, consequence training, target selection, distillation, metrics, and checkpoint writing all run end to end.pilot_action_vm_echo_s96: exposed two bookkeeping issues: inconsistent target-selection CSV schemas and quick validation using the wrong prompt feature set.- Patched the target-selection schema, fixed quick-validation prompt features, and restored the best quick-validation consequence checkpoint before target selection.
pilot_action_vm_echo_s96_v2: unfiltered learned target selection chose all available candidates and had low target precision.pilot_action_vm_echo_s96_thr070: tested a 0.7 learned-score threshold. It selected fewer targets with higher known correctness and improved pilot direct accuracy, so the threshold was used for the main run.main_action_vm_echo_s192_thr070: completed the scaled run with 192 seed examples, 1024 candidate-training prompts, 1024 full-supervised examples, and 128-example validation/fresh/hard splits.
Main Result
- Validation selector gap: base top-1
10.2%, learned selector10.9%, candidate oracle36.7%. - Learned-policy distillation improved several downstream splits despite weak learned selection: fresh-paired direct
18.0% -> 22.7%, fresh-paired answer search51.6% -> 64.1%, hard-composition answer search35.2% -> 52.3%. - Fully supervised training remained the practical ceiling: fresh-paired direct
82.0%, fresh-paired answer search94.5%, hard-composition direct51.6%. - Main conclusion: the candidate-conditioned learning signal is real but still too weak. The next experiment should target selector precision with pairwise preference training, harder negatives, or direct answer-representation access.
Final Artifacts
- Markdown report:
reports/qwen_action_conditioned_vm_echo_policy_iteration_report.md. - HTML report:
reports/qwen_action_conditioned_vm_echo_policy_iteration_report.html. - Analysis summary:
analysis/summary.md. - Figures:
analysis/figures/. - Checkpoints:
large_artifacts/qwen_action_conditioned_vm_echo_policy_iteration/checkpoints/.
Figures 5
Data files 19
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
analysis/all_consequence_selection_eval.csv1.2 kBanalysis/all_metrics.csv19 kBanalysis/main_consequence_selection_eval.csv371 Banalysis/main_metrics.csv4.8 kBruns/main_action_vm_echo_s192_thr070/consequence_selection_eval.csv309 Bruns/main_action_vm_echo_s192_thr070/metrics.csv4.8 kBruns/main_action_vm_echo_s192_thr070/results.json16 kBruns/pilot_action_vm_echo_s96/consequence_selection_eval.csv272 Bruns/pilot_action_vm_echo_s96/metrics.csv4.1 kBruns/pilot_action_vm_echo_s96/results.json15 kBruns/pilot_action_vm_echo_s96_thr070/consequence_selection_eval.csv253 Bruns/pilot_action_vm_echo_s96_thr070/metrics.csv4.2 kBruns/pilot_action_vm_echo_s96_thr070/results.json15 kBruns/pilot_action_vm_echo_s96_v2/consequence_selection_eval.csv253 Bruns/pilot_action_vm_echo_s96_v2/metrics.csv4.1 kBruns/pilot_action_vm_echo_s96_v2/results.json15 kBruns/smoke_action_vm_echo/consequence_selection_eval.csv269 Bruns/smoke_action_vm_echo/metrics.csv3.1 kBruns/smoke_action_vm_echo/results.json14 kB
Reproduce
This entry is source code or analysis only — there is no separate run to reproduce.