Qwen Counterfactual Trace Preference Distillation
The one idea you need
The system turns each question into a small runnable program and generates dozens of candidate fixes. Choosing a winner with no answer key is like judging a room of working calculators by watching them run: you can tell which ones compute, not which one lands on the right number.
The question
Without an answer key, can a model learn to pick the best of its own attempts just by running each one and watching how it behaves?
What we found
Barely. The self-grader learned to favor programs that run without crashing, but it identified the truly correct one only about 15% of the time, against a 41% best-possible ceiling. On fresh questions it picked worse than just trusting the model's first attempt. Given a real answer key, the same machinery hit 91%. The grader, not the compiler underneath, is the wall.
Why it matters
When you rank a model's own code fixes without ground truth, execution signals flag crashes but not wrong answers, so self-picked labels are mostly wrong. Filter picks by confidence to raise precision before training on them.
On this page
Results at a glance 7
How to read
Bars grouped by the two hardest question sets. Within each: the starting system, targets picked with an answer key, targets the model picked itself, and the answer-key ceiling. Height is single-try accuracy; taller is better.
Takeaway → The self-picked bars nudge just above the starting system, but all three huddle near the floor while the answer-key ceiling towers far above them.
Data table
| held-out split | seed (before distillation) | answer-verified targets | preference-selected targets | full supervised (ceiling) |
|---|---|---|---|---|
| fresh paired | 13.3% | 14.8% | 15.6% | 91.4% |
| hard composition | 13.3% | 10.2% | 14.8% | 68% |
Numbers from report table (reports/qwen_counterfactual_trace_preference_distillation_report.md, Main Results)
Technical framing
Direct accuracy after distillation: preference-selected vs answer-verified targets — Preference-selected distillation gives small direct gains on the hardest splits, but every repair branch sits far below the supervised ceiling.
How to read
For four question sets, three bars each: the model's first attempt, the pick its self-grader chose, and the best candidate available in the batch. Height is accuracy; taller is better.
Takeaway → The self-graded pick sits at or below the first attempt everywhere, while the best-available bar stands far higher, so the headroom goes untouched.
Data table
| held-out split | base direct | preference rerank | oracle over candidates |
|---|---|---|---|
| fresh standard | 14.1% | 9.4% | 38.3% |
| fresh paraphrase | 16.4% | 11.7% | 53.9% |
| fresh paired | 13.3% | 11.7% | 47.7% |
| hard composition | 13.3% | 13.3% | 46.1% |
Numbers from report table (reports/qwen_counterfactual_trace_preference_distillation_report.md, Main Results)
Technical framing
Preference selector picks worse than the base decode on held-out splits — The learned selector stays valid but misses the correct answer: reranking never beats the base decode, while the candidate oracle shows large headroom.
How to read
One bar per source of training examples: answer-key verified, best-quality by answer key, and self-picked. Height is the share that are actually correct; taller means cleaner training material.
Takeaway → The self-picked bar reaches only about a tenth as high as the two answer-key bars, which are fully correct by construction.
Data table
| target source | targets that are answer-correct |
|---|---|
| answer-verified | 100% |
| best-quality | 100% |
| preference-selected | 12.2% |
Numbers from report table (reports/qwen_counterfactual_trace_preference_distillation_report.md, Target Quality)
Technical framing
Why preference distillation is noisy: answer-correct precision of training targets — Preference-selected targets cover all 1024 prompts but only 12.2% are answer-correct, versus 100% for the 466 verified targets.
How to read
For the paired question set, one bar per training stage: starting system, answer-key targets, self-picked targets, and full answer-key training. Height is accuracy when allowed several tries; taller is better.
Takeaway → Self-picked training dips below even the untrained start, while full training nearly maxes out, so self-grading actively hurts multi-try search.
Data table
| distillation phase | answer-search accuracy (fresh paired) |
|---|---|
| seed | 47.7% |
| answer-verified | 53.9% |
| preference-selected | 46.1% |
| full supervised | 97.7% |
Numbers from report table (reports/qwen_counterfactual_trace_preference_distillation_report.md, Main Results)
Technical framing
The trade-off: preference-selected targets degrade search on fresh paired — Preference-selected distillation buys small direct gains at the cost of search accuracy, which drops below even the undistilled seed model.
How to read
For four question sets, four bars: full answer-key training with one try and with several tries, and the starting system with one try and several. Taller is better.
Takeaway → Full-training bars cluster near the top across all sets while the starting bars sit near the floor, showing the task itself is very learnable.
Data table
| Held-out split | Full supervised, direct | Full supervised, search | Seed, direct | Seed, search |
|---|---|---|---|---|
| Fresh standard | 85.9% | 98.4% | 14.1% | 38.3% |
| Fresh paraphrase | 83.6% | 93% | 16.4% | 53.9% |
| Fresh paired | 91.4% | 97.7% | 13.3% | 47.7% |
| Hard composition | 68% | 87.5% | 13.3% | 46.1% |
Numbers from report table
Technical framing
Full-supervised ceiling vs the seed compiler across all held-out splits — With answer labels the compiler reaches 91% direct on fresh paired but drops to 68% on hard composition; the seed sits near 14%.
How to read
For four question sets, four bars: starting system, answer-key repair, self-picked repair, and full answer-key training. Height is how often the exact intended program is recovered; taller is better.
Takeaway → Every repair bar hugs zero while full training reaches roughly half to three-quarters, so repair barely changes the program the system writes.
Data table
| Held-out split | Seed | Answer-verified distill | Preference-selected distill | Full supervised |
|---|---|---|---|---|
| Fresh standard | 0.8% | 1.6% | 0% | 76.6% |
| Fresh paraphrase | 3.1% | 3.9% | 0.8% | 73.4% |
| Fresh paired | 0% | 1.6% | 1.6% | 78.1% |
| Hard composition | 1.6% | 2.3% | 0.8% | 51.6% |
Numbers from report table
Technical framing
Program exact match: repair distillation barely changes program identity — All repair-distilled branches stay under 4% exact-program recovery while full supervision reaches 52-78%.
How to read
For a validation set and the two hardest question sets, three bars: a bare pilot, a pilot given extra execution clues, and the full-size run. Height is how often the grader picked correctly; taller is better.
Takeaway → The extra-clue pilot roughly doubles the bare pilot, but the full-size run slips back toward the bare level on the hardest sets.
Data table
| Evaluation split | Pilot, bytecode-only (m192) | Pilot, +candidate features | Scaled main run |
|---|---|---|---|
| Validation | 7.8% | 15.6% | 14.8% |
| Fresh paired | 9.4% | 20.3% | 11.7% |
| Hard composition | 12.5% | 21.9% | 13.3% |
Numbers from experiments/qwen_counterfactual_trace_preference_distillation/analysis/pilot_selector_comparison.csv
Technical framing
Selector iterations: candidate features helped the pilot, scaling lost the held-out gain — Feature-bridged pilot selection roughly doubled the bytecode-only pilot, but the scaled main run kept only the validation signal.
In the author’s words from the Report · “Abstract”
The candidate surface was real: 45.5% of prompts had an answer-correct candidate, while only 13.8% were already correct at the base decode. That left 31.7% true counterfactual repair groups. The result is mixed. The feature-bridged preference selector learned a weak but real signal on validation, reaching 14.8% selection accuracy against a 41.4% oracle. It did not generalize robustly across all held-out splits: fresh-paired selection was 11.7% against a 47.7% oracle. Distilling the learned selector produced small direct gains on the hardest held-out cells but degraded search relative to answer-verified targets. Fresh-paired direct accuracy was 15.6% for preference-selected distillation versus 14.8% for answer-verified distillation. … Read the full result →
Overview
Standalone experiment for testing whether a Qwen-attached typed-bytecode compiler improves when repair candidates are trained with hard counterfactual trace preferences instead of final-answer labels alone.
Small artifacts live here:
experiments/qwen_counterfactual_trace_preference_distillation/Large checkpoints live separately here:
large_artifacts/qwen_counterfactual_trace_preference_distillation/checkpoints/Layout
src/: experiment and analysis scripts.runs/: per-run metrics, logs, and dataset manifests.analysis/: aggregate CSVs and generated figures.reports/: Markdown and HTML reports.experiment_log.md: running journal.checkpoint_manifest.csv: run-to-checkpoint map.
Question
Can a candidate preference objective over executable VM traces select useful repair programs and distill those choices back into a deployable compiler?
The candidate quality order is:
invalid < valid_wrong < answer_correct < trace_consistent < canonicalThe preference model trains on counterfactual groups where the candidate set contains a better executable program than the base decode.
Report
Rendered from reports/qwen_counterfactual_trace_preference_distillation_report.md
Abstract
This standalone experiment tests whether a Qwen-attached typed-bytecode compiler can learn a no-answer repair selector from hard counterfactual execution traces. The compiler emits executable VM programs from frozen Qwen/Qwen3-4B hidden states. Each prompt gets a local candidate set; every candidate is executed; and candidates are ranked by this quality order:
invalid < valid_wrong < answer_correct < trace_consistent < canonical
The main run generated 246784 candidates from 1024 training prompts. The candidate surface was real: 45.5% of prompts had an answer-correct candidate, while only 13.8% were already correct at the base decode. That left 31.7% true counterfactual repair groups.
The result is mixed. The feature-bridged preference selector learned a weak but real signal on validation, reaching 14.8% selection accuracy against a 41.4% oracle. It did not generalize robustly across all held-out splits: fresh-paired selection was 11.7% against a 47.7% oracle.
Distilling the learned selector produced small direct gains on the hardest held-out cells but degraded search relative to answer-verified targets. Fresh-paired direct accuracy was 15.6% for preference-selected distillation versus 14.8% for answer-verified distillation. Hard-composition direct accuracy was 14.8% versus 10.2%. But fresh-paired search was lower: 46.1% versus 53.9%.
The full-supervised ceiling stayed high: 91.4% direct and 97.7% search on fresh paired, with 68.0% direct on hard composition. The substrate can learn the executable compiler; the preference selector is still the bottleneck.
Setup
- Base model: frozen
Qwen/Qwen3-4Bhidden-state extractor. - Compiler: transformer slot decoder that emits typed stack-machine bytecode.
- Candidate generation: local edit/search around the base decode, capped at
256candidates per prompt. - Preference training set: only counterfactual groups where a better candidate exists than the base decode.
- Preference inputs: prompt feature, candidate bytecode, normalized program prior, prompt answer-head logprob of the candidate's VM final value, VM validity, and VM final value.
- Main run:
192seed examples,1024candidate prompts,128examples per eval split. - Large checkpoints:
large_artifacts/qwen_counterfactual_trace_preference_distillation/checkpoints/main_counterfactual_trace_preference_s192_c1024/.
Main Results
| Phase | Split | Direct | Answer search | Oracle | Preference rerank | Program exact |
|---|---|---|---|---|---|---|
| Seed | fresh standard | 14.1% | 38.3% | 38.3% | 0.8% | |
| Seed | fresh paraphrase | 16.4% | 53.9% | 53.9% | 3.1% | |
| Seed | fresh paired | 13.3% | 47.7% | 47.7% | 0.0% | |
| Seed | hard composition | 13.3% | 46.1% | 46.1% | 1.6% | |
| Preference selector | fresh standard | 14.1% | 38.3% | 38.3% | 9.4% | 0.8% |
| Preference selector | fresh paraphrase | 16.4% | 53.9% | 53.9% | 11.7% | 3.1% |
| Preference selector | fresh paired | 13.3% | 47.7% | 47.7% | 11.7% | 0.0% |
| Preference selector | hard composition | 13.3% | 46.1% | 46.1% | 13.3% | 1.6% |
| Answer-verified | fresh standard | 14.8% | 47.7% | 47.7% | 1.6% | |
| Answer-verified | fresh paraphrase | 18.0% | 57.8% | 57.8% | 3.9% | |
| Answer-verified | fresh paired | 14.8% | 53.9% | 53.9% | 1.6% | |
| Answer-verified | hard composition | 10.2% | 53.1% | 53.1% | 2.3% | |
| Preference-selected | fresh standard | 10.9% | 38.3% | 38.3% | 0.0% | |
| Preference-selected | fresh paraphrase | 12.5% | 54.7% | 54.7% | 0.8% | |
| Preference-selected | fresh paired | 15.6% | 46.1% | 46.1% | 1.6% | |
| Preference-selected | hard composition | 14.8% | 48.4% | 48.4% | 0.8% | |
| Best-quality | fresh standard | 13.3% | 41.4% | 41.4% | 1.6% | |
| Best-quality | fresh paraphrase | 18.0% | 57.0% | 57.0% | 3.1% | |
| Best-quality | fresh paired | 14.8% | 47.7% | 47.7% | 0.0% | |
| Best-quality | hard composition | 12.5% | 50.8% | 50.8% | 1.6% | |
| Full supervised | fresh standard | 85.9% | 98.4% | 98.4% | 76.6% | |
| Full supervised | fresh paraphrase | 83.6% | 93.0% | 93.0% | 73.4% | |
| Full supervised | fresh paired | 91.4% | 97.7% | 97.7% | 78.1% | |
| Full supervised | hard composition | 68.0% | 87.5% | 87.5% | 51.6% |
Candidate Surface
The surface had enough headroom to test selection. Candidate-level answer correctness was only 7.6%, but prompt-level oracle accuracy was 45.5%. Trace-consistent and canonical candidates were much rarer: 0.9% and 0.0% at candidate level. That means the intended quality order was active, but most supervision was still effectively answer-correct versus valid-wrong.
Target Quality
| Target source | Targets | Correct | Canonical | Trace-consistent | Mean quality | Changed |
|---|---|---|---|---|---|---|
| answer_verified_targets | 466 | 100.0% | 10.5% | 10.5% | 2.21 | 69.7% |
| best_quality_targets | 466 | 100.0% | 10.9% | 10.9% | 2.22 | 69.7% |
| preference_selected_targets | 1024 | 12.2% | 1.4% | 1.4% | 1.15 | 100.0% |
Preference-selected targets were broad but noisy: 1024 targets with only 12.2% answer-correct precision. Answer-verified and best-quality targets were perfectly answer-correct by construction, but they covered only 466 prompts. Best-quality targets slightly increased canonical/trace-consistent selection from 10.5% to 10.9%, but this was too small to change deployable accuracy.
Preference Selection
The preference selector did not solve no-answer credit assignment. Its best validation checkpoint improved over the base decode on validation, but on fresh standard, fresh paraphrase, and fresh paired it selected worse than the base direct program. The model learned to stay valid, but not reliably to identify the prompt-correct final value.
Iterations
The first pilot had true counterfactual groups but a weak selector. Adding candidate features, especially the prompt answer-head logprob of the candidate's final value, improved the pilot selector substantially. The scaled main run retained a validation signal but lost much of the held-out gain, which points to calibration/generalization rather than candidate availability as the remaining issue.
Training
The full-supervised branch separated sharply after about eight epochs and reached the high ceiling. The repair-distillation branches trained smoothly but plateaued far below that ceiling, consistent with target quality and selector precision being the limiting factors.
Interpretation
The experiment answers the core question narrowly. Counterfactual trace preference learning can find some no-answer repair signal, and it can produce small direct gains on hard held-out cells. It does not yet provide a reliable path to folding the repair oracle into the compiler. The main failure is that selected preference targets are too noisy: the selector chooses valid programs almost always, but answer-correct programs only 12.2% of the time.
The most useful next step is not simply scaling this objective. The next design should either filter preference-selected targets by confidence to raise precision, or move candidate comparison into a richer Qwen-readable representation where the model can compare prompt semantics, VM final values, and execution traces more directly.
Artifacts
experiments/qwen_counterfactual_trace_preference_distillation/runs/main_counterfactual_trace_preference_s192_c1024/metrics.csvexperiments/qwen_counterfactual_trace_preference_distillation/runs/main_counterfactual_trace_preference_s192_c1024/train_log.csvexperiments/qwen_counterfactual_trace_preference_distillation/runs/main_counterfactual_trace_preference_s192_c1024/target_selection.csvexperiments/qwen_counterfactual_trace_preference_distillation/analysis/main_metrics.csvexperiments/qwen_counterfactual_trace_preference_distillation/reports/qwen_counterfactual_trace_preference_distillation_report.mdexperiments/qwen_counterfactual_trace_preference_distillation/reports/qwen_counterfactual_trace_preference_distillation_report.html
Experiment log 3
Show the running log (3 entries, 2026-06-24)
2026-06-24
- Created a fresh standalone experiment directory.
- Selected intervention: counterfactual trace preference distillation.
- Core idea: build candidate groups from the frozen-Qwen compiler, execute all candidate programs, label each candidate by a strict quality order, train a preference selector on groups where a better candidate exists than the base decode, then use the learned selector as a no-answer repair teacher.
Main comparison plan:
- seed supervised compiler;
- answer-verified repair distillation;
- learned preference-selected distillation;
- best-quality distillation control;
- full-supervised ceiling.
- Large artifacts will be stored in
large_artifacts/qwen_counterfactual_trace_preference_distillation/checkpoints/.
Iteration Notes
- Implemented the standalone VM core and experiment script.
- Smoke run
smoke_counterfactual_prefcompleted end to end, including frozen Qwen feature extraction, seed training, candidate group construction, preference training, answer-verified distillation, preference-selected distillation, best-quality control, full-supervised control, metrics, and checkpoint writing. - Smoke diagnostic: the tiny candidate surface had no true counterfactual repair groups because answer-correct candidates only appeared when the base decode was already answer-correct. The next step is a larger pilot with a stronger seed/search surface so the preference objective has real repairable failures.
- Pilot
pilot_pref_s96_c256_m192produced real counterfactual repair groups (17.2%of unlabeled prompts had a better candidate than the base), but the learned selector was weak (7.8%best validation selection vs31.2%oracle). This exposed a missing channel: the selector could execute a candidate but had only an indirect representation of the prompt-implied answer. - Patched the preference model with no-answer candidate features: normalized program prior, prompt answer-head logprob of the candidate's VM final value, VM validity, and normalized VM final value.
- Smoke run
smoke_counterfactual_pref_v2verified the patched feature path. - Pilot
pilot_pref_s96_c256_featuresvalidated the feature bridge. The selector reached15.6%validation selection against a31.2%oracle, and on held-out splits selected correct candidates at20.3%fresh-paired and21.9%hard-composition from a seed compiler with0%direct accuracy on those two splits. Preference-selected distillation improved deployable direct accuracy to9.4%fresh-paired and17.2%hard-composition. This is still far from the oracle, but it is the first useful no-answer selector signal in this experiment, so the main run will scale this version. - Main run
main_counterfactual_trace_preference_s192_c1024completed with 192 seed examples, 1024 candidate prompts, 1024 full-supervised examples, and 128 examples per evaluation split. - Main candidate surface: 246,784 sampled candidates, 64.2% valid candidate rate, 7.6% answer-correct candidate rate, 45.5% prompt-level oracle found rate, and 31.7% true counterfactual repair groups.
- Main selector result: the learned no-answer preference selector reached 14.8% validation selection accuracy against a 41.4% oracle, but did not generalize robustly across held-out splits. Fresh-paired selector accuracy was 11.7% against a 47.7% oracle.
- Main distillation result: preference-selected distillation slightly improved direct accuracy over answer-verified distillation on fresh-paired (15.6% vs 14.8%) and hard-composition (14.8% vs 10.2%), but its selected targets were noisy (12.2% answer-correct), and its search accuracy was worse than answer-verified distillation on the key fresh-paired split (46.1% vs 53.9%).
- Full-supervised ceiling remained high: 91.4% fresh-paired direct, 97.7% fresh-paired search, and 68.0% hard-composition direct.
Final Artifacts
- Markdown report:
reports/qwen_counterfactual_trace_preference_distillation_report.md. - HTML report:
reports/qwen_counterfactual_trace_preference_distillation_report.html. - Analysis summary:
analysis/summary.md. - Figures:
analysis/figures/. - Main run files:
runs/main_counterfactual_trace_preference_s192_c1024/. - Checkpoints:
large_artifacts/qwen_counterfactual_trace_preference_distillation/checkpoints/.
Figures 6
Data files 12
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
analysis/all_metrics.csv22 kBanalysis/main_metrics.csv5.6 kBruns/main_counterfactual_trace_preference_s192_c1024/metrics.csv5.6 kBruns/main_counterfactual_trace_preference_s192_c1024/results.json21 kBruns/pilot_pref_s96_c256_features/metrics.csv4.7 kBruns/pilot_pref_s96_c256_features/results.json20 kBruns/pilot_pref_s96_c256_m192/metrics.csv4.6 kBruns/pilot_pref_s96_c256_m192/results.json20 kBruns/smoke_counterfactual_pref/metrics.csv4.2 kBruns/smoke_counterfactual_pref/results.json20 kBruns/smoke_counterfactual_pref_v2/metrics.csv4.4 kBruns/smoke_counterfactual_pref_v2/results.json20 kB
Reproduce
This entry is source code or analysis only — there is no separate run to reproduce.