Qwen In-Policy VM-ECHO Distillation
The one idea you need
A helper reads a frozen model's internal state, writes tiny programs, and runs each in a sandbox. It was also trained to foresee each program's fate: will it crash, how deep it runs, is its answer right. The hope was that this foresight would reveal which program to trust.
The question
If a system answers questions by writing little programs and running them, does teaching it to predict what happens when each program runs help it choose better answers?
What we found
Only partly. It became excellent at predicting mechanical outcomes, like how deep a program runs (about 92% right), but stayed no better than a coin flip at judging which program is actually correct. So it could not rank its own guesses, and ranking did worse than plain answering on every test. First-try accuracy on paired problems stayed stuck near 9%, far below the 77% a fully-supervised version reached.
Why it matters
Teaching a model to simulate its own execution is not the same as teaching it correctness. To improve which answer you pick, train on near-identical attempts that differ only in whether they are right, emphasizing plausible-but-wrong ones.
On this page
Results at a glance 6
How to read
Five training styles run left to right along the bottom. One color is best-of-many success (keep the best attempt), the other is first-try success. Taller is better; the rightmost style trained on complete answer keys.
Takeaway → Watching outcomes nudges best-of-many up a little, yet both bars stay far below the towering full-answer-key style at the right.
Data table
| training phase | answer search | direct decode |
|---|---|---|
| seed | 44.5% | 11.7% |
| answer distill | 47.7% | 14.1% |
| VM-ECHO distill | 51.6% | 14.1% |
| gold trace | 56.2% | 16.4% |
| full supervised | 85.9% | 58.6% |
Numbers from report table (reports/qwen_inpolicy_vm_echo_distillation_report.md)
Technical framing
Hard-composition accuracy across training phases — VM-ECHO lifts hard-composition answer search from 47.7% to 51.6%, but direct accuracy stays far below the 58.6% full-supervised ceiling.
How to read
One bar per training style, left to right; bar height is first-try success on paired puzzles. Taller is better. The rightmost style trained on complete answer keys.
Takeaway → The first three bars sit flat at the same low height; the gold-trace bar rises modestly and only the full-answer-key bar leaps far up, showing outcome-watching moved nothing.
Data table
| training phase | direct decode |
|---|---|
| seed | 8.6% |
| answer distill | 8.6% |
| VM-ECHO distill | 8.6% |
| gold trace | 16.4% |
| full supervised | 76.6% |
Numbers from report table (reports/qwen_inpolicy_vm_echo_distillation_report.md)
Technical framing
Fresh-paired direct accuracy: VM-ECHO does not move the needle — Fresh-paired direct accuracy is stuck at 8.6% through VM-ECHO training while the full-supervised ceiling is 76.6%.
How to read
One bar per kind of prediction about a program's behavior; height is how often the guess was right. Taller is better. Bars span structural traits down to whether the final answer is correct.
Takeaway → Structural bars, like how deep a program runs, tower over the is-the-answer-right bar, which sits near a coin flip.
Data table
| VM observation channel | observation prediction accuracy |
|---|---|
| trace depth | 91.5% |
| validity | 72.9% |
| trace top | 53.9% |
| answer correct | 49.9% |
| final value | 26.2% |
Numbers from analysis/main_metrics.csv (inpolicy_vm_echo_distill rows)
Technical framing
What the VM-ECHO head learned to predict (fresh standard) — ECHO learned structural consequences (trace depth 91.5%, validity 72.9%) far better than which candidate answers correctly (49.9%).
How to read
Four puzzle types run along the bottom. One color is plain first-try answering; the other picks among attempts using the model's own outcome predictions. Taller is better.
Takeaway → The prediction-picking bar is shorter in every group, so using those predictions to choose answers hurts rather than helps.
Data table
| evaluation split | direct decode | ECHO rerank |
|---|---|---|
| fresh standard | 14.8% | 10.9% |
| fresh paraphrase | 14.1% | 10.2% |
| fresh paired | 8.6% | 6.2% |
| hard composition | 14.1% | 8.6% |
Numbers from report table (reports/qwen_inpolicy_vm_echo_distillation_report.md)
Technical framing
ECHO reranking underperforms direct decoding on every split — The learned consequence model is not a useful reranker: selecting candidates by ECHO score loses to plain direct decoding everywhere.
How to read
Three training configurations run along the bottom; colors compare answer-key training against outcome-watching on two puzzle types. Taller is better. The other configs raise the weighting or add a second pass.
Takeaway → Outcome-watching's clear edge over answer-key training shows only in the leftmost setting; stronger weighting or a second pass shrinks it to a near tie.
Data table
| Pilot configuration | Answer distill, fresh paired | VM-ECHO distill, fresh paired | Answer distill, hard composition | VM-ECHO distill, hard composition |
|---|---|---|---|---|
| ECHO weight 0.10 | 12.5% | 21.9% | 7.8% | 17.2% |
| ECHO weight 0.35 | 12.5% | 12.5% | 7.8% | 10.9% |
| Weight 0.10, 2 rounds | 10.9% | 12.5% | 7.8% | 7.8% |
Numbers from experiments/qwen_inpolicy_vm_echo_distillation/analysis/pilot_comparison.csv
Technical framing
Pilot ablations: the VM-ECHO gain vanishes at higher weight or a second round — The pilot gain that motivated the main run appears only at weight 0.10 with one round; 0.35 and a second round erase it.
How to read
Five training styles run along the bottom; four colors mark four puzzle types. Height is best-of-many success. Taller is better. The rightmost style trained on complete answer keys.
Takeaway → Outcome-watching lifts most colored bars slightly above answer-key training, yet the full-answer-key style at the right stands far taller than all.
Data table
| Training phase (main run) | Fresh standard | Fresh paraphrase | Fresh paired | Hard composition |
|---|---|---|---|---|
| Seed | 44.5% | 40.6% | 44.5% | 44.5% |
| Answer distill | 44.5% | 46.9% | 45.3% | 47.7% |
| VM-ECHO distill | 48.4% | 50% | 44.5% | 51.6% |
| Gold trace | 49.2% | 55.5% | 53.1% | 56.2% |
| Full supervised | 90.6% | 88.3% | 95.3% | 85.9% |
Numbers from experiments/qwen_inpolicy_vm_echo_distillation/analysis/all_metrics.csv
Technical framing
Answer-search accuracy by phase: VM-ECHO nudges the search surface, full supervision transforms it — VM-ECHO lifts search on 3 of 4 splits (hard comp 47.7%→51.6%) but full supervision reaches 86-95%, far above any distill phase.
In the author’s words from the Report · “Abstract”
VM-ECHO learned the observation channels: on the main run, trace-depth prediction reached about 91.5% on fresh-standard candidates and trace-top prediction reached 53.9%. It also improved some answer-search metrics over answer-verified distillation, including hard-composition search from 47.7% to 51.6%. But it did not produce a broad direct-accuracy jump: fresh-paired direct accuracy stayed at 8.6%, while the full-supervised ceiling reached 76.6%.
Overview
Standalone experiment for testing whether a frozen-Qwen typed-bytecode compiler improves when it learns the VM consequences of its own proposed programs during repair distillation.
Small artifacts live in this directory:
experiments/qwen_inpolicy_vm_echo_distillation/Large checkpoints live separately here:
large_artifacts/qwen_inpolicy_vm_echo_distillation/checkpoints/Layout
src/: experiment and analysis scripts.runs/: per-run metrics, logs, and dataset manifests.analysis/: aggregate CSVs and generated figures.reports/: Markdown and HTML reports.experiment_log.md: running journal.checkpoint_manifest.csv: run-to-checkpoint map.
Question
Can in-policy VM-observation learning turn sampled program failures into useful posttraining signal, beyond answer-verified repair targets alone?
Report
Rendered from reports/qwen_inpolicy_vm_echo_distillation_report.md
Abstract
This experiment tests whether a Qwen-attached typed-bytecode compiler improves when it learns the VM consequences of its own proposed programs during repair distillation. The compiler first emits executable candidates from frozen Qwen/Qwen3-4B hidden states. Those candidates are executed in a typed VM. The training objective combines answer-verified repair distillation with an integrated observation loss over all sampled candidates: validity, final value, trace top, trace depth, and answer-correctness.
The result is mixed. VM-ECHO learned the observation channels: on the main run, trace-depth prediction reached about 91.5% on fresh-standard candidates and trace-top prediction reached 53.9%. It also improved some answer-search metrics over answer-verified distillation, including hard-composition search from 47.7% to 51.6%. But it did not produce a broad direct-accuracy jump: fresh-paired direct accuracy stayed at 8.6%, while the full-supervised ceiling reached 76.6%.
Setup
- Base model:
Qwen/Qwen3-4B, used as a frozen hidden-state feature extractor. - Compiler: transformer slot decoder over Qwen hidden states.
- Integrated VM-ECHO head: candidate-conditioned transformer sharing the compiler prompt projection.
- Seed examples:
192. - Candidate prompts:
1024. - Candidate programs:
246784. - Candidate positive rate:
8.3%. - Prompt-level oracle found rate:
44.8%. - Repair targets selected:
459. - ECHO loss weight selected by pilot:
0.1. - Checkpoints:
large_artifacts/qwen_inpolicy_vm_echo_distillation/checkpoints/main_inpolicy_vm_echo_s192_w010/.
Main Results
| Phase | Split | Direct | Answer search | Oracle | ECHO rerank | Program exact |
|---|---|---|---|---|---|---|
| Seed | fresh standard | 12.5% | 44.5% | 44.5% | 0.8% | |
| Seed | fresh paraphrase | 14.8% | 40.6% | 40.6% | 0.8% | |
| Seed | fresh paired | 8.6% | 44.5% | 44.5% | 0.0% | |
| Seed | hard composition | 11.7% | 44.5% | 44.5% | 0.0% | |
| Answer distill | fresh standard | 14.1% | 44.5% | 44.5% | 0.0% | |
| Answer distill | fresh paraphrase | 10.9% | 46.9% | 46.9% | 0.0% | |
| Answer distill | fresh paired | 8.6% | 45.3% | 45.3% | 0.0% | |
| Answer distill | hard composition | 14.1% | 47.7% | 47.7% | 0.8% | |
| VM-ECHO distill | fresh standard | 14.8% | 48.4% | 48.4% | 10.9% | 0.8% |
| VM-ECHO distill | fresh paraphrase | 14.1% | 50.0% | 50.0% | 10.2% | 1.6% |
| VM-ECHO distill | fresh paired | 8.6% | 44.5% | 44.5% | 6.2% | 0.0% |
| VM-ECHO distill | hard composition | 14.1% | 51.6% | 51.6% | 8.6% | 0.8% |
| Gold trace | fresh standard | 16.4% | 49.2% | 49.2% | 3.9% | |
| Gold trace | fresh paraphrase | 18.0% | 55.5% | 55.5% | 3.1% | |
| Gold trace | fresh paired | 16.4% | 53.1% | 53.1% | 5.5% | |
| Gold trace | hard composition | 16.4% | 56.2% | 56.2% | 3.9% | |
| Full sup. | fresh standard | 71.9% | 90.6% | 90.6% | 55.5% | |
| Full sup. | fresh paraphrase | 78.9% | 88.3% | 88.3% | 63.3% | |
| Full sup. | fresh paired | 76.6% | 95.3% | 95.3% | 63.3% | |
| Full sup. | hard composition | 58.6% | 85.9% | 85.9% | 39.1% |
Candidate Surface
| Phase | Round | Targets | Oracle found | Changed | Candidate valid |
|---|---|---|---|---|---|
| answer_verified_targets | 1 | 459 | 44.8% | 70.6% | 64.3% |
| echo_repair_targets | 1 | 459 | 44.8% | 70.6% | 64.3% |
The candidate surface is large enough to matter but not large enough to solve the task by itself. The main run generated 246784 candidates from 1024 prompts, and 44.8% of prompts had at least one answer-correct candidate. This puts a ceiling on what one round of repair distillation can learn.
VM Observation Learning
VM-ECHO learned validity and trace structure much better than final answer correctness. This is important: the auxiliary loss did train a consequence model, but the learned consequence model was not a useful no-answer reranker. ECHO reranking was below direct decoding on every main split. The useful effect, where present, came from joint training changing the compiler, not from selecting candidates at inference time.
Training Dynamics
The main ECHO auxiliary loss decreased from 5.35 to 2.67, while repair loss also improved. A higher pilot weight damaged deployable accuracy, so the auxiliary needs to remain secondary to the repair target.
Pilot Results
The pilot sweep selected echo_loss_weight=0.1. A stronger 0.35 weight over-regularized the compiler. A second in-policy round improved some standard/paraphrase cells, but it removed the fresh-paired and hard direct signal that made the one-round pilot attractive.
Interpretation
This experiment supports a narrow claim: candidate-consequence prediction can be trained in the same compiler loop without breaking executable decoding, and it can modestly reshape the answer-search surface. It does not support the stronger claim that this objective currently unlocks the full repair oracle or produces a large direct compiler improvement.
The decisive remaining problem is credit assignment among executable candidates. VM-ECHO learned broad consequences such as validity and stack traces, but it did not learn to identify which candidate solves the prompt. The next version should make the preference signal sharper: train on counterfactual candidate pairs from the same prompt, emphasize hard negatives that share validity or final-value plausibility, and expose a more direct representation of the prompt-implied answer to the candidate scorer.
Artifacts
experiments/qwen_inpolicy_vm_echo_distillation/runs/main_inpolicy_vm_echo_s192_w010/metrics.csvexperiments/qwen_inpolicy_vm_echo_distillation/runs/main_inpolicy_vm_echo_s192_w010/train_log.csvexperiments/qwen_inpolicy_vm_echo_distillation/analysis/main_metrics.csvexperiments/qwen_inpolicy_vm_echo_distillation/reports/qwen_inpolicy_vm_echo_distillation_report.mdexperiments/qwen_inpolicy_vm_echo_distillation/reports/qwen_inpolicy_vm_echo_distillation_report.html
Experiment log 3
Show the running log (3 entries, 2026-06-24)
2026-06-24
- Created a fresh standalone experiment directory.
- Selected intervention: in-policy VM-ECHO distillation.
- Core idea: a frozen-Qwen compiler proposes typed bytecode programs; the VM executes those proposals; integrated observation heads learn validity, final value, trace top, trace depth, and answer-correctness for the compiler's own candidates while answer-verified repairs are distilled into the compiler.
- Controls: answer-verified repair distillation, matched gold-trace distillation, and full-supervised training.
- Large artifacts will be stored in
large_artifacts/qwen_inpolicy_vm_echo_distillation/checkpoints/.
Iteration Notes
- Implemented the standalone VM core, integrated compiler/ECHO experiment script, README, checkpoint manifest, and run log.
- Smoke run
smoke_inpolicy_vm_echopassed end to end, including Qwen feature extraction, seed training, answer-verified control, in-policy VM-ECHO training, gold-trace control, full-supervised control, metrics, and checkpoint writing. - The first smoke exposed a train-log CSV schema issue because ECHO phases write additional loss columns. Patched the CSV appender to rewrite with a union schema when later rows add columns.
- Smoke run
smoke_inpolicy_vm_echo_v2verified the fixed schema. - Pilot
pilot_inpolicy_vm_echo_s96_w010used 96 seed examples and 256 candidate prompts. It found answer-verified repair targets for 33.2% of candidate prompts. In-policy VM-ECHO improved fresh-paired direct accuracy over answer-verified distillation (21.9%vs12.5%) and hard-composition direct accuracy (17.2%vs7.8%). The ECHO reranker itself was weak, so the useful effect appeared through training, not inference-time selection. - Pilot
pilot_inpolicy_vm_echo_s96_w035tested a stronger ECHO loss. It was worse on fresh-paired direct accuracy (12.5%), so the main run will useecho_loss_weight=0.1. - Main run
main_inpolicy_vm_echo_s192_w010completed with 192 seed examples, 1024 candidate prompts, 1024 full-supervised examples, and 128 examples per evaluation split. - Main candidate surface: 246,784 sampled candidates, 8.3% answer-correct candidate rate, 64.3% valid candidate rate, and 44.8% prompt-level oracle found rate. Both answer-only and VM-ECHO branches selected 459 verified repair targets.
- Main result: VM-ECHO learned VM observations and improved some answer-search metrics over answer-verified distillation, including hard-composition search
47.7% -> 51.6%, but it did not produce a broad direct-accuracy gain. Fresh-paired direct accuracy stayed at8.6%, while the full-supervised ceiling reached76.6%direct and95.3%search. - Pilot
pilot_inpolicy_vm_echo_s96_w010_r2tested two in-policy rounds. It did not justify a second main run because gains were inconsistent and the fresh-paired/hard direct signal weakened.
Final Artifacts
- Markdown report:
reports/qwen_inpolicy_vm_echo_distillation_report.md. - HTML report:
reports/qwen_inpolicy_vm_echo_distillation_report.html. - Analysis summary:
analysis/summary.md. - Figures:
analysis/figures/. - Main run files:
runs/main_inpolicy_vm_echo_s192_w010/. - Checkpoints:
large_artifacts/qwen_inpolicy_vm_echo_distillation/checkpoints/.
Figures 5
Data files 14
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
analysis/all_metrics.csv23 kBanalysis/main_metrics.csv4.8 kBruns/main_inpolicy_vm_echo_s192_w010/metrics.csv4.9 kBruns/main_inpolicy_vm_echo_s192_w010/results.json21 kBruns/pilot_inpolicy_vm_echo_s96_w010/metrics.csv4.7 kBruns/pilot_inpolicy_vm_echo_s96_w010/results.json21 kBruns/pilot_inpolicy_vm_echo_s96_w010_r2/metrics.csv4.7 kBruns/pilot_inpolicy_vm_echo_s96_w010_r2/results.json21 kBruns/pilot_inpolicy_vm_echo_s96_w035/metrics.csv4.7 kBruns/pilot_inpolicy_vm_echo_s96_w035/results.json21 kBruns/smoke_inpolicy_vm_echo/metrics.csv3.1 kBruns/smoke_inpolicy_vm_echo/results.json19 kBruns/smoke_inpolicy_vm_echo_v2/metrics.csv3.2 kBruns/smoke_inpolicy_vm_echo_v2/results.json19 kB
Reproduce
The run commands are documented inside the experiment folder (see the README).