Qwen VM-ECHO Trace Distillation
The one idea you need
This system rewrites each word problem as a tiny stack-machine program. The experiment added a side lesson: while the correct program runs, predict what the machine shows at every step. It is like memorizing how a finished dish looks mid-cook without learning to cook your own.
The question
If you train a program-writing model to also predict exactly what its target programs do when they run, does it start writing better programs itself?
What we found
Mostly no. The model got far better at predicting execution, with reading a running program's top value climbing from under 1 percent correct to 43 percent, but that rarely improved the programs it wrote. First-try accuracy on standard problems even slipped from 85 to 81 percent. A few test-and-fix settings gained slightly, 93 to 96 percent, so it is a real limit, not a collapse.
Why it matters
If you bolt a predict-execution side objective onto a code generator, feed it programs the model itself proposed, not the ideal answer. Predicting a correct program's behavior barely transfers to writing or choosing better ones.
On this page
Results at a glance 3
How to read
Three paired bars compare the original helper against the outcome-predicting helper on first-try success, success after test-and-fix retries, and guessing a running program's top value; taller is better.
Takeaway → The outcome-guessing bar leaps from near zero to about 43 percent and retries edge up, but first-try success is identical, so the new skill barely spills over.
Data table
| condition | baseline | VM-ECHO |
|---|---|---|
| Direct accuracy | 84.4% | 84.4% |
| Search accuracy | 93% | 96.1% |
| Trace-top prediction | 0.7% | 43.1% |
Numbers from experiments/qwen_vm_echo_trace_distillation/analysis/main_metrics.csv
Technical framing
Does predicting VM traces help the compiler? (full-supervised, fresh paired split) — VM-ECHO learns to predict execution traces (0.7% to 43.1%) and lifts search 93% to 96%, but direct accuracy is unchanged.
How to read
Four paired bars, one per problem style, compare the original helper with the outcome-predicting one on first-try success; taller is better.
Takeaway → The two sit nearly level everywhere, and on the standard style the outcome-predicting bar is actually shorter, confirming no broad first-try gain.
Data table
| condition | baseline | VM-ECHO |
|---|---|---|
| fresh paired | 84.4% | 84.4% |
| fresh paraphrase | 72.7% | 73.4% |
| fresh standard | 85.2% | 80.5% |
| hard composition | 64.8% | 63.3% |
Numbers from report table (reports/qwen_vm_echo_trace_distillation_report.md)
Technical framing
Direct accuracy by evaluation split (full-supervised phase) — The trace loss does not broadly improve greedy compilation; fresh-standard direct accuracy actually drops from 85.2% to 80.5%.
How to read
Paired bars for two self-practice rounds show the share of problems where each helper found a working program to learn from; taller is better.
Takeaway → The bars stay within a couple of points across both rounds, showing the extra training did not shrink the pool of examples to learn from.
Data table
| Expert-iteration round | Baseline | VM-ECHO |
|---|---|---|
| Round 1 | 37.2% | 38.9% |
| Round 2 | 44.5% | 42.6% |
Numbers from report table
Technical framing
Expert-iteration target collection is unharmed by ECHO loss — At weight 0.03 the ECHO objective does not collapse the search-based target harvest: found rates stay within 2pp of baseline.
In the author’s words from the Report · “Abstract”
VM-ECHO clearly learns the VM observation channels: in the main full-supervised arm, fresh-paired trace-top prediction rises from 0.7% to 43.1%. That extra semantic signal does not translate into a broad direct-accuracy jump. It gives modest local gains in some search/oracle settings, for example hard-composition expert-round-2 search rises from 46.9% to 51.6%, and full-supervised fresh-paired search rises from 93.0% to 96.1%. But full-supervised fresh-standard direct accuracy falls from 85.2% to 80.5%. This is not a universal improvement.
Overview
This experiment tests whether a frozen-Qwen bytecode compiler improves when it is trained not only to emit a target program, but also to predict the VM observations produced by that program.
Hypothesis
A compiler head attached to Qwen hidden states may learn more reusable program semantics if the training signal includes the consequences of execution: validity, final value, stack top after each slot, and stack depth after each slot. The key comparison is a matched baseline versus a VM-ECHO arm with the same Qwen features, same typed decoder, same answer-verified local search, and same initialization.
Layout
src/typed_bytecode_core.py: standalone task generator, typed VM, decoder, candidate search, and utility functions.src/qwen_vm_echo_trace_distillation_experiment.py: frozen-Qwen feature extraction, compiler head, VM-ECHO losses, training, evaluation, and checkpoint manifest updates.src/analyze_qwen_vm_echo_trace_distillation.py: aggregation, charts, and Markdown/HTML report generation.runs/: per-run metrics, logs, and dataset manifests.analysis/: aggregated CSVs and generated figures.reports/: final Markdown and HTML writeups.large_artifacts/qwen_vm_echo_trace_distillation/checkpoints/: checkpoint files kept outside the experiment directory.
Primary Metrics
- Direct executable accuracy from greedy decoded bytecode.
- Search/oracle accuracy from answer-verified candidate repair.
- Exact program match and validity.
- VM observation prediction accuracy: final value, validity, trace top, and trace depth.
Report
Rendered from reports/qwen_vm_echo_trace_distillation_report.md
Abstract
This standalone experiment tests whether a frozen-Qwen typed-bytecode compiler benefits from an auxiliary VM-observation objective. The baseline learns to emit bytecode and a final answer. The VM-ECHO arm gets the same program loss plus a low-weight loss for predicting execution observations: VM validity, final value, stack top after each active bytecode slot, and stack depth after each active bytecode slot.
The result is mixed. VM-ECHO clearly learns the VM observation channels: in the main full-supervised arm, fresh-paired trace-top prediction rises from 0.7% to 43.1%. That extra semantic signal does not translate into a broad direct-accuracy jump. It gives modest local gains in some search/oracle settings, for example hard-composition expert-round-2 search rises from 46.9% to 51.6%, and full-supervised fresh-paired search rises from 93.0% to 96.1%. But full-supervised fresh-standard direct accuracy falls from 85.2% to 80.5%. This is not a universal improvement.
Setup
- Base model:
Qwen/Qwen3-4B, used only as a frozen hidden-state feature extractor. - Compiler: transformer-decoder slot head over Qwen hidden states.
- VM: typed stack bytecode with
192seed examples,1024unlabeled expert-iteration prompts,1024full-supervised examples, and128examples per fresh split. - Main ECHO weight:
0.03. Pilot weights0.35,0.10, and0.03were used only to choose a non-destructive auxiliary-loss scale. - Checkpoints:
large_artifacts/qwen_vm_echo_trace_distillation/checkpoints/main_vm_echo_s192_w003/.
Main Results
| Arm | Phase | Split | Direct | Search | Program exact | Trace-top acc. |
|---|---|---|---|---|---|---|
| baseline | Expert R2 | fresh_paired | 10.9% | 50.0% | 0.8% | 1.1% |
| baseline | Expert R2 | fresh_paraphrase | 14.1% | 43.0% | 0.8% | 0.8% |
| baseline | Expert R2 | fresh_standard | 17.2% | 50.0% | 4.7% | 0.9% |
| baseline | Expert R2 | hard_composition | 8.6% | 46.9% | 3.1% | 0.8% |
| baseline | Full sup. | fresh_paired | 84.4% | 93.0% | 69.5% | 0.7% |
| baseline | Full sup. | fresh_paraphrase | 72.7% | 88.3% | 52.3% | 0.8% |
| baseline | Full sup. | fresh_standard | 85.2% | 93.8% | 63.3% | 1.1% |
| baseline | Full sup. | hard_composition | 64.8% | 85.2% | 47.7% | 1.5% |
| vm_echo | Expert R2 | fresh_paired | 13.3% | 52.3% | 3.1% | 12.0% |
| vm_echo | Expert R2 | fresh_paraphrase | 14.1% | 39.1% | 1.6% | 9.2% |
| vm_echo | Expert R2 | fresh_standard | 21.1% | 50.8% | 4.7% | 12.1% |
| vm_echo | Expert R2 | hard_composition | 8.6% | 51.6% | 3.1% | 7.7% |
| vm_echo | Full sup. | fresh_paired | 84.4% | 96.1% | 65.6% | 43.1% |
| vm_echo | Full sup. | fresh_paraphrase | 73.4% | 87.5% | 53.1% | 38.2% |
| vm_echo | Full sup. | fresh_standard | 80.5% | 91.4% | 58.6% | 41.9% |
| vm_echo | Full sup. | hard_composition | 63.3% | 85.9% | 46.1% | 35.0% |
VM Observation Learning
The auxiliary heads learned the execution-observation task, especially stack depth. Trace-top accuracy also rose substantially in the full-supervised VM-ECHO arm, but final-value prediction stayed modest because it is a 97-way target and the main answer head already carries a separate final-answer signal.
Expert Target Collection
VM-ECHO at weight 0.03 did not collapse the candidate set. It collected slightly more round-1 expert targets than the baseline and slightly fewer round-2 targets.
| arm | round | targets | found_rate | candidate_valid_rate |
|---|---|---|---|---|
| baseline | 1 | 381 | 37.2% | 63.1% |
| baseline | 2 | 456 | 44.5% | 63.4% |
| vm_echo | 1 | 398 | 38.9% | 63.7% |
| vm_echo | 2 | 436 | 42.6% | 64.3% |
Weight Sweep
The pilot sweep showed why the main run used a low weight. At 0.35, VM-ECHO learned observations but damaged candidate search. At 0.03, it preserved the search surface better.
Interpretation
The useful finding is not that VM-ECHO is a breakthrough by itself. The useful finding is sharper: a consequence-prediction loss can be attached to a frozen-Qwen bytecode compiler without breaking typed decoding, and it can make the model learn nontrivial VM-state predictions. However, teacher-forced trace prediction is only weakly coupled to choosing better programs. The next version should condition the observation predictor on candidate programs sampled from the compiler, so the model learns consequences of its own actions rather than consequences of the gold target alone.
Artifacts
experiments/qwen_vm_echo_trace_distillation/runs/main_vm_echo_s192_w003/metrics.csvexperiments/qwen_vm_echo_trace_distillation/runs/main_vm_echo_s192_w003/train_log.csvexperiments/qwen_vm_echo_trace_distillation/analysis/main_metrics.csvexperiments/qwen_vm_echo_trace_distillation/reports/qwen_vm_echo_trace_distillation_report.mdexperiments/qwen_vm_echo_trace_distillation/reports/qwen_vm_echo_trace_distillation_report.html
Experiment log 3
Show the running log (3 entries, 2026-06-24)
2026-06-24
- Created a fresh standalone experiment directory for VM-ECHO trace distillation.
- Chosen intervention: add VM observation prediction losses to the frozen-Qwen typed-bytecode compiler head.
- Initial design: compare
baselineandvm_echoarms from the same randomly initialized compiler head, with identical frozen Qwen feature caches, decoder architecture, candidate search, and training split sizes. - Large artifacts will be stored in
large_artifacts/qwen_vm_echo_trace_distillation/checkpoints/.
Smoke and Pilot Iteration
- Smoke run
smoke_vm_echoverified end-to-end execution, but showed the first trace mask over-weighted padded post-ENDslots. - Updated VM observation labels to supervise only active slots through
END. - Smoke run
smoke_vm_echo_maskedverified the corrected active-slot mask. - Pilot run
pilot_vm_echo_s96withecho_weight=0.35learned observation signals but damaged candidate search. - Pilot runs
pilot_vm_echo_s96_w010andpilot_vm_echo_s96_w003swept lower weights.0.03preserved the candidate search surface best and was selected for the main comparison.
Main Run
- Main run:
main_vm_echo_s192_w003. - Main setting:
echo_weight=0.03, two expert rounds, matched baseline and VM-ECHO arms from the same compiler initialization. - Key result: VM-ECHO learned trace observations but did not produce a broad direct-accuracy improvement.
- Full-supervised fresh paired: direct accuracy tied at 84.4%; search accuracy improved from 93.0% to 96.1%; trace-top observation accuracy improved from 0.7% to 43.1%.
- Expert-round-2 hard composition: search accuracy improved from 46.9% to 51.6%; direct accuracy tied at 8.6%.
Figures 5
Data files 14
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
analysis/all_metrics.csv27 kBanalysis/main_metrics.csv8.5 kBruns/main_vm_echo_s192_w003/metrics.csv8.5 kBruns/main_vm_echo_s192_w003/results.json27 kBruns/pilot_vm_echo_s96/metrics.csv5.9 kBruns/pilot_vm_echo_s96/results.json20 kBruns/pilot_vm_echo_s96_w003/metrics.csv3.1 kBruns/pilot_vm_echo_s96_w003/results.json10 kBruns/pilot_vm_echo_s96_w010/metrics.csv3.1 kBruns/pilot_vm_echo_s96_w010/results.json10 kBruns/smoke_vm_echo/metrics.csv4.1 kBruns/smoke_vm_echo/results.json18 kBruns/smoke_vm_echo_masked/metrics.csv4.3 kBruns/smoke_vm_echo_masked/results.json18 kB
Reproduce
The run commands are documented inside the experiment folder (see the README).