Qwen Hidden VM On-Policy Canonical Repair
The one idea you need
Instead of blurting an answer, the model secretly writes a step-by-step recipe that a fixed calculator runs to reach the result. A checker can spot a corrected recipe sitting right next to the model's. The test: does feeding those corrections back as training make the model write better recipes on its own?
The question
When a checker can automatically repair a model's hidden step-by-step reasoning, does training the model on those repairs make it produce correct steps by itself?
What we found
No. Training the model on automatically corrected recipes reached 61% on new tasks, versus 59% for plain training — a 2-point gap that is basically noise, and it left longer tasks no better. The corrections are genuinely valuable live: a checker searching nearby recipes hits 89%. But that help evaporates the moment you try to bake it into the model itself.
Why it matters
If a checker can fix your model's outputs, use it live at answer time to pick or correct them. Feeding those fixes back into training recovered nothing here and slightly hurt accuracy on longer inputs.
On this page
Results at a glance 8
How to read
Accuracy on a fresh mixed-task set, taller is better. Four bars: answering directly, the hidden-recipe model trained plainly, the same model after training on the checker's fixes, and the best case if a checker patches each recipe.
Takeaway → Both hidden-recipe bars tower over direct answering yet sit nearly level with each other, so training on fixes barely moved anything; only live checker patching reaches much higher.
Data table
| condition | accuracy |
|---|---|
| direct logits | 8.6% |
| hidden VM (trace-only control) | 59% |
| hidden VM (on-policy repair) | 60.9% |
| state-verified repair (headroom) | 89.1% |
Numbers from analysis/all_final_metrics.csv (final, fresh_paired_mixed)
Technical framing
Fresh paired mixed accuracy: direct answer vs hidden VM vs repair headroom — Compiling to a hidden VM lifts accuracy from 8.6% to ~61%, but on-policy repair training adds only +2.0pp over the trace-only control.
How to read
Accuracy (higher is better) moving left to right toward longer tasks; training covered only the shorter lengths. Three lines: the model trained on fixes, the plainly-trained model, and the best case if a checker patches each recipe.
Takeaway → Past the trained length the fix-trained line sits at or below the plainly-trained line, so training on fixes buys no extra robustness on longer tasks.
Data table
| program length (trained up to 6) | on-policy repair (hidden VM) | trace-only control (hidden VM) | repair headroom (on-policy run) |
|---|---|---|---|
| 6 | 72.9% | 68.2% | 96.4% |
| 8 | 53.1% | 54.2% | 85.4% |
| 10 | 31.8% | 35.9% | 65.6% |
Numbers from analysis/all_final_metrics.csv (final; fresh/hard/harder standard_mixed splits)
Technical framing
Length generalization: on-policy repair does not improve hard-length robustness — Beyond the trained length 6, the on-policy repair arm falls at or below the trace-only control, while verified local search keeps large headroom.
How to read
Four training approaches, taller is better. Each has two bars: the model's own recipe-writing accuracy on fresh tasks, and the best case if a checker patches the recipe. Compare heights across approaches.
Takeaway → Every approach produces essentially identical heights, so training on the checker's fixes buys nothing over plain training.
Data table
| training arm | hidden VM | repair headroom |
|---|---|---|
| gold-only control | 60.9% | 89.1% |
| repair-only | 60.9% | 89.1% |
| repair-or-gold | 60.9% | 89.1% |
| trace-only control | 59% | 89.5% |
Numbers from analysis/all_final_metrics.csv (final, fresh_paired_mixed)
Technical framing
Main training arms are indistinguishable on fresh paired accuracy — All repair-target arms land on the same result as gold-only training, so distilling verified repairs adds nothing over the controls.
How to read
Six task types along the bottom, taller is better. Three bars each: answering directly, running the hidden recipe, and the best case if a checker patches the recipe.
Takeaway → The hidden recipe beats direct answering in every task type, and the checker's best case recovers most of what the recipe still missed.
Data table
| domain | direct logits | hidden VM | repair headroom |
|---|---|---|---|
| arithmetic | 0% | 40.9% | 79.5% |
| calendar | 4.5% | 45.5% | 86.4% |
| unit | 0% | 61.9% | 85.7% |
| list | 2.4% | 81% | 100% |
| boolean | 45.2% | 85.7% | 95.2% |
| lookup | 0% | 52.4% | 88.1% |
Numbers from analysis/final_metrics.csv (main_repair_or_gold_s512, fresh_paired_mixed row)
Technical framing
Per-domain accuracy on fresh paired tasks — The hidden VM beats direct answering in every domain, and verified local repair finds a correct program for most remaining failures.
How to read
Three length tiers left to right, taller is better. Bars pair plain wording against reworded wording, shown both for the raw hidden recipe and after a checker patches it.
Takeaway → The gap between plain and reworded bars widens sharply at longer lengths, and patching cannot close it, so rewording bites hardest on the hard tasks.
Data table
| Task length tier | Hidden VM, standard wording | Hidden VM, paraphrased | After repair, standard wording | After repair, paraphrased |
|---|---|---|---|---|
| Fresh length | 72.9% | 62% | 96.4% | 91.7% |
| Hard length | 53.1% | 27.6% | 85.4% | 64.1% |
| Harder length | 31.8% | 16.7% | 65.6% | 33.9% |
Numbers from report table (reports/qwen_hidden_vm_onpolicy_canonical_repair_paper.md, final splits)
Technical framing
Paraphrase wording hurts more as tasks get longer — Paraphrasing costs ~11 points at fresh lengths but ~26-32 points at hard/harder lengths, and repair cannot close the gap.
How to read
Four small-scale training recipes along the bottom, taller is better. Three bars each: the model's own recipe accuracy, the best case after a checker patches it, and how often the recipe was written exactly right.
Takeaway → Recipes leaning hardest on fixes raise the patched best case but drop the model's own accuracy from about 64% to 50%, trading first-try quality for salvage.
Data table
| Pilot training recipe (192 tasks) | Hidden VM | After repair | Program exact match |
|---|---|---|---|
| Head-only | 63.5% | 91.7% | 46.9% |
| Head-only, cap 24 | 60.4% | 86.5% | 42.7% |
| Repair-target | 50% | 96.9% | 37.5% |
| Stable mix | 47.9% | 91.7% | 34.4% |
Numbers from report table (reports/qwen_hidden_vm_onpolicy_canonical_repair_paper.md, run summary)
Technical framing
Pilot ablations: repair-heavy training trades compiler quality for repair rate — Training on repair targets lifts repaired accuracy to 96.9% but drops one-shot compiler accuracy from 63.5% to 50.0%.
How to read
Seven task groups left to right, getting harder and more reworded, taller is better. One bar is how often the checker found any fix; the other is how often that fix was exactly right.
Takeaway → Both bars fall off a cliff on the hardest reworded group — fixes found drop from 95% to 1 in 4 — so the checker's help vanishes exactly where the model needs it.
Data table
| Evaluation split | Repair candidate found | Repaired program exact |
|---|---|---|
| Fresh standard | 94.8% | 87.5% |
| Fresh paraphrase | 90.1% | 80.7% |
| Fresh paired | 88.3% | 73.4% |
| Hard standard | 83.9% | 64.1% |
| Hard paraphrase | 61.5% | 39.6% |
| Harder standard | 60.4% | 39.6% |
| Harder paraphrase | 26% | 10.9% |
Numbers from report table (reports/qwen_hidden_vm_onpolicy_canonical_repair_paper.md, final splits)
Technical framing
The repair search itself collapses on harder paraphrased tasks — Verified repair finds a fix for 94.8% of fresh tasks but only 26.0% of harder paraphrased ones, so headroom vanishes exactly where needed.
How to read
Two bars, before and after training, taller is better. Three measures each: running the hidden recipe, the best case after a checker patches it, and how often the recipe was written exactly right.
Takeaway → The untrained model never writes a perfect recipe; training lifts the checker-patched best case from 28% to 95% on new validation tasks.
Data table
| Checkpoint | Hidden VM execution | After verified repair | Program exact match |
|---|---|---|---|
| Base model (step 0) | 20.8% | 27.8% | 0% |
| After training (step 701) | 58.3% | 95.1% | 41.7% |
Numbers from runs/main_repair_or_gold_s512/train_log.csv
Technical framing
What training buys over the base model on held-out validation — The base model never emits an exactly-correct program; training lifts repaired accuracy from 28% to 95% on validation.
In the author’s words from the Report · “Abstract”
This experiment tests whether a Qwen 4B model can improve a hidden virtual-machine compiler by training on canonical on-policy repair targets. The model emits invisible typed VM slots, a deterministic runtime executes those slots, and local candidate repairs are accepted only when their full intermediate state trajectory matches the canonical trajectory.
Overview
Standalone experiment for training Qwen/Qwen3-4B to compile mixed natural-language tasks into a hidden typed VM, then improve the compiler with canonical on-policy repair targets.
Small files live here:
experiments/qwen_hidden_vm_onpolicy_canonical_repair/Large checkpoints live separately here:
large_artifacts/qwen_hidden_vm_onpolicy_canonical_repair/checkpoints/The main question is whether verified state-equivalent local repairs can be folded back into a Qwen-attached hidden VM compiler without damaging the trace-trained policy.
Report
Rendered from reports/qwen_hidden_vm_onpolicy_canonical_repair_paper.md
Abstract
This experiment tests whether a Qwen 4B model can improve a hidden virtual-machine compiler by training on canonical on-policy repair targets. The model emits invisible typed VM slots, a deterministic runtime executes those slots, and local candidate repairs are accepted only when their full intermediate state trajectory matches the canonical trajectory.
Setup
- Primary run:
main_repair_or_gold_s512 - Model:
Qwen/Qwen3-4B - Variant:
trace - Train examples:
512 - Train steps:
700 - On-policy rounds:
1 - Epochs per round:
1 - Target mode:
repair_or_gold - Repair verifier mode:
state - VM max steps:
10 - Curriculum schedule:
4:240,6:700 - Train length range:
1to6 - Eval length:
6; hard length:8; harder length:10
The hidden VM uses typed operation slots and copied numeric arguments. Direct logits are the model's next-token numeric answer distribution at the answer marker. Hidden VM accuracy is execution of the compiled invisible program. Repair accuracy is target-aware state-verified local search around the compiled program and is reported as a headroom measurement, not as a deployable inference path.
Results
Final Splits
| Split | Direct | Hidden VM | Repair | Program exact | Repair exact | State prefix | Repair found |
|---|---|---|---|---|---|---|---|
| val_mixed | 13.9% | 58.3% | 95.1% | 41.7% | 83.3% | 70.8% | 93.1% |
| fresh_standard_mixed | 7.3% | 72.9% | 96.4% | 56.8% | 87.5% | 76.3% | 94.8% |
| fresh_paraphrase_mixed | 12.5% | 62.0% | 91.7% | 46.9% | 80.7% | 74.5% | 90.1% |
| fresh_paired_mixed | 8.6% | 60.9% | 89.1% | 44.5% | 73.4% | 72.8% | 88.3% |
| hard_standard_mixed | 9.9% | 53.1% | 85.4% | 32.3% | 64.1% | 67.5% | 83.9% |
| hard_paraphrase_mixed | 12.0% | 27.6% | 64.1% | 8.3% | 39.6% | 60.5% | 61.5% |
| harder_standard_mixed | 9.4% | 31.8% | 65.6% | 11.5% | 39.6% | 60.7% | 60.4% |
| harder_paraphrase_mixed | 10.4% | 16.7% | 33.9% | 2.1% | 10.9% | 49.9% | 26.0% |
| domain_arithmetic | 3.1% | 34.4% | 90.6% | 34.4% | 90.6% | 62.5% | 90.6% |
| domain_calendar | 12.5% | 46.9% | 100.0% | 37.5% | 75.0% | 72.4% | 100.0% |
| domain_unit | 0.0% | 37.5% | 84.4% | 37.5% | 84.4% | 66.7% | 84.4% |
| domain_list | 0.0% | 81.2% | 93.8% | 46.9% | 53.1% | 82.8% | 90.6% |
| domain_boolean | 46.9% | 93.8% | 100.0% | 53.1% | 93.8% | 72.9% | 96.9% |
| domain_lookup | 0.0% | 71.9% | 93.8% | 43.8% | 62.5% | 71.4% | 90.6% |
Domain Breakdown
| Domain | n | Direct | Hidden VM | Repair |
|---|---|---|---|---|
| arithmetic | 44.00 | 0.0% | 40.9% | 79.5% |
| calendar | 44.00 | 4.5% | 45.5% | 86.4% |
| unit | 42.00 | 0.0% | 61.9% | 85.7% |
| list | 42.00 | 2.4% | 81.0% | 100.0% |
| boolean | 42.00 | 45.2% | 85.7% | 95.2% |
| lookup | 42.00 | 0.0% | 52.4% | 88.1% |
Training Dynamics
Fresh paired hidden VM accuracy moved from 23.4% at initialization to 60.9% after the full treatment. State-verified local repair on the same split reaches 89.1%. The trace-only control scores 59.0% on fresh paired, 54.2% on hard length 8, and 35.9% on harder length 10.
On-Policy Target Quality
The final target pass used 512.00 source examples. Active rows were 100.0%; canonical repairs were found for 99.6%; changed-program repairs were 17.4%; program-exact repaired targets were 96.5%; average local candidates were 152.59.
Run Summary
| Run | Variant | Direct | Hidden VM | Repair | Program exact | State prefix |
|---|---|---|---|---|---|---|
| main_gold_control_s512 | trace | 8.6% | 60.9% | 89.1% | 44.5% | 72.8% |
| main_repair_only_s512 | trace | 8.6% | 60.9% | 89.1% | 44.5% | 72.8% |
| main_repair_or_gold_s512 | trace | 8.6% | 60.9% | 89.1% | 44.5% | 72.8% |
| main_trace_control_s512 | trace | 8.6% | 59.0% | 89.5% | 44.1% | 71.0% |
| pilot_onpolicy_canonical_headonly_cap24_s192 | trace | 8.3% | 60.4% | 86.5% | 42.7% | 71.0% |
| pilot_onpolicy_canonical_headonly_s192 | trace | 8.3% | 63.5% | 91.7% | 46.9% | 72.0% |
| pilot_onpolicy_canonical_repair_s192 | trace | 8.3% | 50.0% | 96.9% | 37.5% | 69.3% |
| pilot_onpolicy_canonical_stable_s192 | trace | 8.3% | 47.9% | 91.7% | 34.4% | 68.8% |
Interpretation
The primary measurement is fresh paired mixed-domain accuracy. Direct logits score 8.6%, while the on-policy canonical-repair hidden VM scores 60.9% (+52.3 pp). The trace-only control scores 59.0%, so the on-policy treatment changes fresh paired accuracy by +2.0 pp. State-verified local repair scores 89.1%, measuring how often the current top-k neighborhood contains a canonical executable program.
Program-exact accuracy is 44.5% and state-prefix accuracy is 72.8%. The gold-only control scores 60.9% on fresh paired and the repair-only control scores 60.9%; the repair arms therefore do not separate from an extra stabilized gold-trace pass.
The hard-length splits are the decisive stress test. The treatment trains up to length 6 and reaches 53.1% at length 8 and 31.8% at length 10. The trace-only control reaches 54.2% and 35.9% on those same standard hard splits, so the on-policy epoch does not improve length robustness.
Decision
Canonical on-policy repair should not be scaled in this form. The state verifier produces high-quality targets, but distilling those targets into the compiler gives the same fresh paired result as gold-only training and weakens hard-length robustness relative to the trace-only control. The useful outcome is the headroom measurement: state-verified local search still reaches 89.1% on fresh paired and 85.4% on hard length 8. The next step should keep repair as selection or reranking, or train only on uncertainty-targeted repairs with a much stricter preservation objective.
Limitations
- The domains are synthetic and deterministic.
- Answers are integers in a bounded value vocabulary.
- Trace supervision supplies exact hidden programs during the curriculum phase.
- Repair accuracy is target-aware and should be read as verifier-assisted headroom.
- Canonical state verification uses synthetic trajectories available in this harness.
- The runtime is fixed and hand-designed.
- Each main arm is one run unless additional seeds are added.
Artifacts
Small experiment files live in:
experiments/qwen_hidden_vm_onpolicy_canonical_repair/Large artifacts live in:
large_artifacts/qwen_hidden_vm_onpolicy_canonical_repair/checkpoints/Primary files:
analysis/summary.mdanalysis/final_metrics.csvanalysis/all_final_metrics.csvanalysis/figures/split_accuracy.pnganalysis/figures/domain_accuracy.pnganalysis/figures/training_curve.pnganalysis/figures/target_quality.pnganalysis/figures/run_summary.pngruns/main_repair_or_gold_s512/metrics.csvruns/main_repair_or_gold_s512/train_log.csvreports/qwen_hidden_vm_onpolicy_canonical_repair_paper.mdreports/qwen_hidden_vm_onpolicy_canonical_repair_paper.htmlcheckpoint_manifest.csv
Experiment log 1
Show the running log (1 entry, 2026-06-23)
2026-06-23
Created a standalone hidden VM on-policy canonical repair experiment.
Design:
- Use
Qwen/Qwen3-4Bwith QLoRA and a hidden typed VM compiler. - Train on mixed task families: arithmetic chains, calendar shifts, unit transforms, list aggregation, boolean thresholding, and lookup/adjust rules.
- Use a length curriculum before repair training.
- Evaluate standard, paraphrase, paired, hard length, harder length, and per-domain splits.
- Generate local repair candidates from the current compiler policy.
- Accept a repair target only when the candidate's full intermediate state trajectory matches the canonical trajectory, not merely when the final answer matches.
- Train directly on target slot values while retaining gold trace, gold selection, executor, and state losses as stabilizers.
- Keep large model artifacts under
large_artifacts/.
Initial planned runs:
smoke_onpolicy_canonical_repair: tiny end-to-end check.pilot_onpolicy_canonical_repair_s192: small run to verify target quality and stability.main_trace_control_s512: curriculum-only control.main_gold_control_s512: curriculum plus one extra gold-only on-policy-format epoch.main_repair_only_s512: curriculum plus canonical repair-only targets.main_repair_or_gold_s512: curriculum plus canonical repair targets with gold fallback.
Smoke run:
python experiments/qwen_hidden_vm_onpolicy_canonical_repair/src/qwen_hidden_vm_onpolicy_canonical_repair_experiment.py \
--run_name smoke_onpolicy_canonical_repair --variant trace \
--train_examples 12 --val_examples 6 --eval_examples 6 --eval_pairs 4 --domain_eval_examples 2 \
--train_steps 2 --onpolicy_rounds 1 --epochs_per_round 1 --train_batch_size 1 --eval_batch_size 2 \
--max_steps 4 --train_max_len 3 --eval_length 3 --hard_length 4 --harder_length 0 --max_length 384 \
--curriculum_schedule 2:1,3:2 --onpolicy_source_examples 12 --repair_train_topk 2 --repair_eval_topk 2 --repair_max_edits 1 \
--target_mode repair_or_gold --repair_verifier_mode state \
--log_every 1 --eval_every 2 --seed 121 --eval_seed 121001Smoke result:
- Completed successfully in 29.2 seconds.
- The on-policy target pass produced active rows for 100.0% of the 12 source examples.
- Canonical state-verified repairs were found for 41.7% of source examples.
- Changed-program repairs were 33.3%; program-exact repaired targets were 25.0%.
- Final smoke fresh paired hidden-VM accuracy was 25.0%; this is only a wiring check.
- Analyzer generated markdown, HTML, CSV summaries, and charts.
Pilot 1:
python experiments/qwen_hidden_vm_onpolicy_canonical_repair/src/qwen_hidden_vm_onpolicy_canonical_repair_experiment.py \
--run_name pilot_onpolicy_canonical_repair_s192 --variant trace \
--train_examples 192 --val_examples 72 --eval_examples 72 --eval_pairs 48 --domain_eval_examples 16 \
--train_steps 260 --onpolicy_rounds 1 --epochs_per_round 1 --train_batch_size 2 --eval_batch_size 4 \
--max_steps 10 --train_max_len 6 --eval_length 6 --hard_length 8 --harder_length 10 --max_length 768 \
--curriculum_schedule 4:120,6:260 --onpolicy_source_examples 192 --repair_train_topk 3 --repair_eval_topk 3 --repair_max_edits 2 \
--target_mode repair_or_gold --repair_verifier_mode state \
--lr 5e-5 --executor_loss_weight 0.2 --state_loss_weight 0.05 \
--gold_trace_loss_weight 0.25 --gold_selection_loss_weight 0.25 \
--log_every 40 --eval_every 260 --seed 122 --eval_seed 122001Pilot 1 result:
- Completed in 367.6 seconds.
- Curriculum transition before on-policy training was strong: fresh paired 74.0%, hard length-8 standard 63.9%, harder length-10 standard 43.1%.
- State-verified local repair at transition was high: fresh paired 99.0%, hard length-8 standard 95.8%, harder length-10 standard 72.2%.
- Canonical on-policy targets were dense: active rows 100.0%, found 100.0%, changed 17.7%, program-exact 98.4%.
- After one full on-policy epoch, fresh paired fell to 50.0%, hard length-8 standard to 29.2%, and harder length-10 standard to 23.6%.
- Interpretation: target availability is not the problem; the repair phase is too destabilizing at the initial learning rate and regularization strength.
Pilot 2 change:
- Add
--onpolicy_lr_multiplier. - Keep the same curriculum prefix seed, but run the on-policy epoch at a lower effective LR.
- Increase gold trace and gold selection regularization during on-policy training.
Pilot 2:
python experiments/qwen_hidden_vm_onpolicy_canonical_repair/src/qwen_hidden_vm_onpolicy_canonical_repair_experiment.py \
--run_name pilot_onpolicy_canonical_stable_s192 --variant trace \
--train_examples 192 --val_examples 72 --eval_examples 72 --eval_pairs 48 --domain_eval_examples 16 \
--train_steps 260 --onpolicy_rounds 1 --epochs_per_round 1 --onpolicy_lr_multiplier 0.2 \
--train_batch_size 2 --eval_batch_size 4 \
--max_steps 10 --train_max_len 6 --eval_length 6 --hard_length 8 --harder_length 10 --max_length 768 \
--curriculum_schedule 4:120,6:260 --onpolicy_source_examples 192 --repair_train_topk 3 --repair_eval_topk 3 --repair_max_edits 2 \
--target_mode repair_or_gold --repair_verifier_mode state \
--lr 5e-5 --executor_loss_weight 0.2 --state_loss_weight 0.05 \
--repair_trace_loss_weight 0.25 --gold_trace_loss_weight 1.0 --gold_selection_loss_weight 1.0 \
--log_every 40 --eval_every 260 --seed 122 --eval_seed 122001Pilot 2 result:
- Completed in 361.3 seconds.
- The curriculum prefix matched Pilot 1 on the important splits: fresh paired 74.0%, hard length-8 standard 63.9%, harder length-10 standard 43.1%.
- Target quality also matched: active 100.0%, found 100.0%, changed 17.7%, program-exact 98.4%.
- After the gentler on-policy epoch, fresh paired still fell to 47.9%, hard length-8 standard to 34.7%, and harder length-10 standard to 16.7%.
- Interpretation: lower LR and stronger gold regularization are not enough. The likely failure is representation drift or over-updating during the repair phase.
Pilot 3 change:
- Freeze Qwen/LoRA during on-policy training and update only the compiler head.
- Reset the optimizer before on-policy training.
- Keep the same curriculum prefix seed for direct comparison.
Pilot 3:
python experiments/qwen_hidden_vm_onpolicy_canonical_repair/src/qwen_hidden_vm_onpolicy_canonical_repair_experiment.py \
--run_name pilot_onpolicy_canonical_headonly_s192 --variant trace \
--train_examples 192 --val_examples 72 --eval_examples 72 --eval_pairs 48 --domain_eval_examples 16 \
--train_steps 260 --onpolicy_rounds 1 --epochs_per_round 1 --onpolicy_lr_multiplier 0.2 --no-onpolicy_train_lora \
--train_batch_size 2 --eval_batch_size 4 \
--max_steps 10 --train_max_len 6 --eval_length 6 --hard_length 8 --harder_length 10 --max_length 768 \
--curriculum_schedule 4:120,6:260 --onpolicy_source_examples 192 --repair_train_topk 3 --repair_eval_topk 3 --repair_max_edits 2 \
--target_mode repair_or_gold --repair_verifier_mode state \
--lr 5e-5 --executor_loss_weight 0.2 --state_loss_weight 0.05 \
--repair_trace_loss_weight 0.25 --gold_trace_loss_weight 1.0 --gold_selection_loss_weight 1.0 \
--log_every 40 --eval_every 260 --seed 122 --eval_seed 122001Pilot 3 result:
- Completed in 362.9 seconds.
- Same curriculum transition as Pilot 1 and 2: fresh paired 74.0%, hard length-8 standard 63.9%, harder length-10 standard 43.1%.
- Same target quality: active 100.0%, found 100.0%, changed 17.7%, program-exact 98.4%.
- After head-only on-policy training, fresh paired fell to 63.5%, hard length-8 standard to 48.6%, and harder length-10 standard to 36.1%.
- Interpretation: freezing Qwen/LoRA helps, but a full on-policy pass is still too large. The repair phase should be a limited nudge, not a full epoch.
Pilot 4 change:
- Add
--onpolicy_max_batches. - Keep head-only on-policy training, but cap the repair phase to 24 batches.
Pilot 4:
python experiments/qwen_hidden_vm_onpolicy_canonical_repair/src/qwen_hidden_vm_onpolicy_canonical_repair_experiment.py \
--run_name pilot_onpolicy_canonical_headonly_cap24_s192 --variant trace \
--train_examples 192 --val_examples 72 --eval_examples 72 --eval_pairs 48 --domain_eval_examples 16 \
--train_steps 260 --onpolicy_rounds 1 --epochs_per_round 1 --onpolicy_lr_multiplier 0.2 --no-onpolicy_train_lora --onpolicy_max_batches 24 \
--train_batch_size 2 --eval_batch_size 4 \
--max_steps 10 --train_max_len 6 --eval_length 6 --hard_length 8 --harder_length 10 --max_length 768 \
--curriculum_schedule 4:120,6:260 --onpolicy_source_examples 192 --repair_train_topk 3 --repair_eval_topk 3 --repair_max_edits 2 \
--target_mode repair_or_gold --repair_verifier_mode state \
--lr 5e-5 --executor_loss_weight 0.2 --state_loss_weight 0.05 \
--repair_trace_loss_weight 0.25 --gold_trace_loss_weight 1.0 --gold_selection_loss_weight 1.0 \
--log_every 40 --eval_every 260 --seed 122 --eval_seed 122001Pilot 4 result:
- Completed in 325.7 seconds.
- Same curriculum transition: fresh paired 74.0%, hard length-8 standard 63.9%, harder length-10 standard 43.1%.
- Same target quality: active 100.0%, found 100.0%, changed 17.7%, program-exact 98.4%.
- After 24 head-only on-policy batches, fresh paired fell to 60.4%, hard length-8 standard to 47.2%, and harder length-10 standard to 31.9%.
- Interpretation: limiting repair batches did not solve the problem. The main runs should treat canonical on-policy repair as a hypothesis under test, not as an assumed improvement.
Main run plan:
- Use matched seed
124and eval seed124001. - Run a trace-only control.
- Run a gold-only head-only on-policy control.
- Run a repair-only head-only on-policy arm.
- Run a repair-or-gold head-only on-policy arm.
- Select the deployable result by fresh paired and hard-length accuracy, not by repair headroom alone.
Main trace control:
python experiments/qwen_hidden_vm_onpolicy_canonical_repair/src/qwen_hidden_vm_onpolicy_canonical_repair_experiment.py \
--run_name main_trace_control_s512 --variant trace \
--train_examples 512 --val_examples 144 --eval_examples 192 --eval_pairs 128 --domain_eval_examples 32 \
--train_steps 700 --onpolicy_rounds 0 --train_batch_size 2 --eval_batch_size 8 \
--max_steps 10 --train_max_len 6 --eval_length 6 --hard_length 8 --harder_length 10 --max_length 768 \
--curriculum_schedule 4:240,6:700 --onpolicy_source_examples 512 --repair_train_topk 3 --repair_eval_topk 3 --repair_max_edits 2 \
--target_mode repair_or_gold --repair_verifier_mode state \
--lr 5e-5 --executor_loss_weight 0.2 --state_loss_weight 0.05 \
--log_every 70 --eval_every 700 --seed 124 --eval_seed 124001Main trace result:
- Completed in 619.6 seconds.
- Fresh paired hidden-VM accuracy: 59.0%; state-verified repair headroom: 89.5%.
- Hard length-8 standard: 54.2%; hard length-8 paraphrase: 36.5%.
- Harder length-10 standard: 35.9%; harder length-10 paraphrase: 20.3%.
Main gold-only control:
python experiments/qwen_hidden_vm_onpolicy_canonical_repair/src/qwen_hidden_vm_onpolicy_canonical_repair_experiment.py \
--run_name main_gold_control_s512 --variant trace \
--train_examples 512 --val_examples 144 --eval_examples 192 --eval_pairs 128 --domain_eval_examples 32 \
--train_steps 700 --onpolicy_rounds 1 --epochs_per_round 1 --onpolicy_lr_multiplier 0.2 --no-onpolicy_train_lora \
--train_batch_size 2 --eval_batch_size 8 \
--max_steps 10 --train_max_len 6 --eval_length 6 --hard_length 8 --harder_length 10 --max_length 768 \
--curriculum_schedule 4:240,6:700 --onpolicy_source_examples 512 --repair_train_topk 3 --repair_eval_topk 3 --repair_max_edits 2 \
--target_mode gold_only --repair_verifier_mode state \
--lr 5e-5 --executor_loss_weight 0.2 --state_loss_weight 0.05 \
--repair_trace_loss_weight 0.25 --gold_trace_loss_weight 1.0 --gold_selection_loss_weight 1.0 \
--log_every 70 --eval_every 700 --seed 124 --eval_seed 124001Main gold-only result:
- Completed in 821.4 seconds.
- Target pass sanity check: active 100.0%, changed 0.0%, program-exact 100.0%.
- Fresh paired hidden-VM accuracy: 60.9%.
- Hard length-8 standard: 53.6%; hard length-8 paraphrase: 27.6%.
- Harder length-10 standard: 31.8%; harder length-10 paraphrase: 16.7%.
- Interpretation: a head-only extra gold pass gives a small fresh paired gain but weakens hard-length paraphrase and length-10 robustness.
Main repair-only:
python experiments/qwen_hidden_vm_onpolicy_canonical_repair/src/qwen_hidden_vm_onpolicy_canonical_repair_experiment.py \
--run_name main_repair_only_s512 --variant trace \
--train_examples 512 --val_examples 144 --eval_examples 192 --eval_pairs 128 --domain_eval_examples 32 \
--train_steps 700 --onpolicy_rounds 1 --epochs_per_round 1 --onpolicy_lr_multiplier 0.2 --no-onpolicy_train_lora \
--train_batch_size 2 --eval_batch_size 8 \
--max_steps 10 --train_max_len 6 --eval_length 6 --hard_length 8 --harder_length 10 --max_length 768 \
--curriculum_schedule 4:240,6:700 --onpolicy_source_examples 512 --repair_train_topk 3 --repair_eval_topk 3 --repair_max_edits 2 \
--target_mode repair_only --repair_verifier_mode state \
--lr 5e-5 --executor_loss_weight 0.2 --state_loss_weight 0.05 \
--repair_trace_loss_weight 0.25 --gold_trace_loss_weight 1.0 --gold_selection_loss_weight 1.0 \
--log_every 70 --eval_every 700 --seed 124 --eval_seed 124001Main repair-only result:
- Completed in 834.1 seconds.
- Target pass: active 99.6%, found 99.6%, changed 17.4%, program-exact 96.5%.
- Fresh paired hidden-VM accuracy: 60.9%.
- Hard length-8 standard: 53.1%; hard length-8 paraphrase: 27.6%.
- Harder length-10 standard: 31.8%; harder length-10 paraphrase: 16.7%.
- Interpretation: repair-only does not separate from the gold-only control.
Main repair-or-gold:
python experiments/qwen_hidden_vm_onpolicy_canonical_repair/src/qwen_hidden_vm_onpolicy_canonical_repair_experiment.py \
--run_name main_repair_or_gold_s512 --variant trace \
--train_examples 512 --val_examples 144 --eval_examples 192 --eval_pairs 128 --domain_eval_examples 32 \
--train_steps 700 --onpolicy_rounds 1 --epochs_per_round 1 --onpolicy_lr_multiplier 0.2 --no-onpolicy_train_lora \
--train_batch_size 2 --eval_batch_size 8 \
--max_steps 10 --train_max_len 6 --eval_length 6 --hard_length 8 --harder_length 10 --max_length 768 \
--curriculum_schedule 4:240,6:700 --onpolicy_source_examples 512 --repair_train_topk 3 --repair_eval_topk 3 --repair_max_edits 2 \
--target_mode repair_or_gold --repair_verifier_mode state \
--lr 5e-5 --executor_loss_weight 0.2 --state_loss_weight 0.05 \
--repair_trace_loss_weight 0.25 --gold_trace_loss_weight 1.0 --gold_selection_loss_weight 1.0 \
--log_every 70 --eval_every 700 --seed 124 --eval_seed 124001Main repair-or-gold result:
- Completed in 836.3 seconds.
- Target pass: active 100.0%, found 99.6%, changed 17.4%, program-exact 96.5%.
- Fresh paired hidden-VM accuracy: 60.9%; state-verified repair headroom: 89.1%.
- Hard length-8 standard: 53.1%; hard length-8 paraphrase: 27.6%.
- Harder length-10 standard: 31.8%; harder length-10 paraphrase: 16.7%.
- Interpretation: canonical repair targets are high quality, but folding them back into the compiler does not outperform gold-only training and reduces hard-length robustness relative to trace-only.
Main conclusion:
Canonical on-policy repair should not be scaled in this form. The state verifier is valuable as a headroom or candidate-selection mechanism, but direct distillation into the compiler behaves like an extra gold pass and weakens the length-generalization stress tests.
Figures 5
Data files 20
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
analysis/all_final_metrics.csv51 kBanalysis/final_metrics.csv6.8 kBruns/main_gold_control_s512/metrics.csv6.8 kBruns/main_gold_control_s512/results.json116 kBruns/main_repair_only_s512/metrics.csv6.7 kBruns/main_repair_only_s512/results.json116 kBruns/main_repair_or_gold_s512/metrics.csv6.8 kBruns/main_repair_or_gold_s512/results.json116 kBruns/main_trace_control_s512/metrics.csv6.8 kBruns/main_trace_control_s512/results.json85 kBruns/pilot_onpolicy_canonical_headonly_cap24_s192/metrics.csv7.4 kBruns/pilot_onpolicy_canonical_headonly_cap24_s192/results.json115 kBruns/pilot_onpolicy_canonical_headonly_s192/metrics.csv7.3 kBruns/pilot_onpolicy_canonical_headonly_s192/results.json115 kBruns/pilot_onpolicy_canonical_repair_s192/metrics.csv7.3 kBruns/pilot_onpolicy_canonical_repair_s192/results.json115 kBruns/pilot_onpolicy_canonical_stable_s192/metrics.csv7.4 kBruns/pilot_onpolicy_canonical_stable_s192/results.json115 kBruns/smoke_onpolicy_canonical_repair/metrics.csv4.8 kBruns/smoke_onpolicy_canonical_repair/results.json85 kB
Reproduce
This entry is source code or analysis only — there is no separate run to reproduce.