Qwen Typed Bytecode Expert Iteration
The one idea you need
The model turns a word problem into a short program a calculator runs exactly. It generates many programs and keeps any that reach the correct final answer to learn from — like a student trusting any scratch work that matches the answer key, even lucky shortcuts.
The question
If a small model can check whether the programs it writes reach the correct final answer, can learning from those wins match learning from full worked-out solutions?
What we found
It depends. Training the model only on its own attempts that landed on the correct final answer lifted unaided first-try accuracy from 62% to 73% on fresh problems — a real gain that sticks when it writes programs alone. But training on complete worked-out programs reached nearly 100%. Checking final answers helps for free, yet stalls far short of showing full solutions.
Why it matters
When you can score final answers but lack worked solutions, self-training on answer-verified attempts buys a real lift for free. But collect or synthesize complete worked solutions when you need near-perfect reliability, especially on longer multi-step problems.
On this page
Results at a glance 7
How to read
Three groups left to right on fresh problems: starting point, after self-training, full worked examples. Each has two bars — the lighter is the model answering unaided, the darker is it allowed to guess-and-check toward the right answer. Taller is better.
Takeaway → The self-trained unaided bar climbs from 62% to 73%, but the full worked-example bars tower near 100%, dwarfing the self-training gain.
Data table
| training regime | direct decoding | answer-verified search |
|---|---|---|
| seed (384 traces) | 61.5% | 81.6% |
| expert iteration R4 | 73% | 87.3% |
| full supervised (4096 traces) | 99.6% | 100% |
Numbers from experiments/qwen_typed_bytecode_expert_iteration/analysis/final_metrics.csv
Technical framing
Expert iteration helps the deployable compiler, but dense traces nearly saturate — Answer-verified expert iteration lifts direct accuracy +11.5pp over the seed, but dense bytecode supervision reaches 99.6%.
How to read
Two rising lines track unaided accuracy across rounds of self-training, starting at round zero; higher is better. One line is ordinary fresh problems, the other is harder multi-step problems.
Takeaway → Both lines jump at round one then flatten, and the harder-problems line stays well below, showing self-training stalls early and struggles with longer programs.
Data table
| expert-iteration round (0 = seed) | fresh paired | hard composition |
|---|---|---|
| 0 | 61.5% | 45.9% |
| 1 | 70.5% | 51.6% |
| 2 | 70.5% | 52% |
| 3 | 71.1% | 52.5% |
| 4 | 73% | 53.9% |
Numbers from experiments/qwen_typed_bytecode_expert_iteration/analysis/final_metrics.csv
Technical framing
Direct accuracy across expert-iteration rounds (same compiler, no search at test time) — Most of the gain arrives in round 1 and then plateaus; hard compositional prompts improve less than fresh paired ones.
How to read
Three groups on fresh problems: starting point, self-trained, full worked examples. Each has two bars — the light bar is answering unaided, the dark bar is with guess-and-check. Taller is better.
Takeaway → Self-training nearly triples unaided accuracy from 18% to 50%, but the full worked-example bar near 95% again far outreaches it.
Data table
| training regime (bytecode head on frozen Qwen3-4B states) | direct decoding | answer-verified search |
|---|---|---|
| Qwen seed | 17.6% | 53.5% |
| Qwen expert iteration R3 | 50.4% | 74.2% |
| Qwen full supervised | 94.5% | 99.2% |
Numbers from experiments/qwen_typed_bytecode_expert_iteration/analysis/all_final_metrics.csv
Technical framing
Frozen-Qwen pilot: same pattern with a real 4B front end — With frozen Qwen hidden states, expert iteration nearly triples direct accuracy (+32.8pp) but dense traces still dominate.
How to read
Two lines across rounds; higher means more. One is the share of training prompts where guess-and-check found a correct program, the other is the share of those that were genuinely new fixes rather than already-known answers.
Takeaway → The found line climbs to 93% while the new-fix line collapses toward 5%, so later rounds mostly re-confirm what the model already knew.
Data table
| expert-iteration round | answer-verified target found (of 4096 prompts) | target changed by search (of found targets) |
|---|---|---|
| 1 | 83.1% | 26.3% |
| 2 | 89% | 8.7% |
| 3 | 91.3% | 6.1% |
| 4 | 93.4% | 5.2% |
Numbers from experiments/qwen_typed_bytecode_expert_iteration/analysis/expert_target_quality.csv
Technical framing
Expert-target quality over rounds (main run) — Search finds verified targets for 93% of prompts by round 4, but only 5% of found targets are new corrections - the loop runs out of fresh signal.
How to read
Two lines across self-training rounds on fresh problems; higher is better. The lower line is the model answering unaided, the upper line is it allowed to guess-and-check toward the right answer.
Takeaway → The guess-and-check line starts about 20 points above the unaided line at the seed round, then settles into a 14-16 point lead through the later rounds, meaning fixable mistakes remain in the model's first answers.
Data table
| Expert-iteration round (0 = seed) | Direct decoding | Answer-verified search |
|---|---|---|
| 0 | 61.5% | 81.6% |
| 1 | 70.5% | 86.7% |
| 2 | 70.5% | 85.5% |
| 3 | 71.1% | 84.8% |
| 4 | 73% | 87.3% |
Numbers from experiments/qwen_typed_bytecode_expert_iteration/analysis/all_final_metrics.csv
Technical framing
Search keeps beating direct decoding through every round — A 14-16 point search-over-direct gap persists after four rounds, so the compiler still leaves repairable mistakes on the table.
How to read
Four problem-type groups, each with two bars for the two model front ends after training on full worked examples; taller is better. The first three groups are ordinary problems, the last is harder multi-step ones.
Takeaway → Both front ends near-max the first three groups but drop 15-20 points on the harder multi-step group, marking longer programs as the stubborn weak spot.
Data table
| condition | Compact compiler (4,096 gold traces) | Frozen-Qwen head (2,048 gold traces) |
|---|---|---|
| Fresh standard | 99.8% | 94.1% |
| Fresh paraphrase | 99.6% | 96.1% |
| Fresh paired | 99.6% | 94.5% |
| Hard composition | 80.7% | 75.8% |
Numbers from experiments/qwen_typed_bytecode_expert_iteration/analysis/all_final_metrics.csv
Technical framing
Even dense supervision does not close the hard-composition split — Both front ends near-saturate fresh splits with dense traces, but longer compositional programs stay 15-20 points behind.
How to read
Lines across rounds; higher means more. Two lines are the 4B version's share of prompts solved and share needing heavy rewriting; one line is the simpler main version's solved share for comparison.
Takeaway → The 4B version starts far lower and rewrites more heavily, but its solved line climbs toward the main version's, showing a slower version of the same trend.
Data table
| Expert-iteration round | Qwen head: target found | Qwen head: target changed by search | Main compact run: target found |
|---|---|---|---|
| 1 | 53.9% | 66.4% | 83.1% |
| 2 | 66.7% | 50.5% | 89% |
| 3 | 79.2% | 34.6% | 91.3% |
| 4 | — | — | 93.4% |
Numbers from experiments/qwen_typed_bytecode_expert_iteration/analysis/expert_target_quality.csv
Technical framing
Frozen-Qwen pilot finds fewer, noisier expert targets than the main run — The Qwen-attached head starts with far fewer verified targets and heavier search rewrites, but converges toward the main run's pattern.
In the author’s words from the Report · “Abstract”
On fresh paired prompts, the seed compiler reached 61.5%; expert iteration reached 73.0%; dense full supervision reached 99.6%.
Overview
Standalone experiment for a Qwen-oriented typed-bytecode training loop. The experiment tests whether exact execution and answer-verified program search can turn a weak text-to-bytecode compiler into a stronger deployable compiler.
Small artifacts live in this directory. Large checkpoints are stored under:
large_artifacts/qwen_typed_bytecode_expert_iteration/checkpoints/Report
Rendered from reports/qwen_typed_bytecode_expert_iteration_paper.md
Abstract
This experiment tests a typed-bytecode posttraining recipe in a controlled text-to-program compiler. A compact transformer reads natural-language prompts and emits a fixed-length typed stack-machine program. The bytecode is validated and executed by an exact interpreter. The main question is whether answer-verified local search can create useful expert-iteration targets, and how that compares with dense supervised bytecode traces.
The primary run trained a seed compiler on 384 gold bytecode traces, then ran four rounds of answer-verified expert iteration over 4,096 generated training prompts. A separate full-supervised ceiling trained on 4,096 gold traces. On fresh paired prompts, the seed compiler reached 61.5%; expert iteration reached 73.0%; dense full supervision reached 99.6%.
Setup
- Runtime: exact typed stack-machine bytecode over bounded i32 values modulo 97.
- Opcodes:
PUSH, arithmetic, comparisons, min/max, two lookup host calls,END, andPAD. - Domains: modular arithmetic, calendar offsets, unit scaling, list aggregation, boolean thresholds, and table lookup.
- Compiler: compact transformer encoder/decoder over tokenized prompts and fixed program slots.
- Qwen-attached pilot: frozen
Qwen/Qwen3-4Bhidden states with a trainable bytecode compiler head. - Expert iteration: local candidates are generated from compiler logits, executed, and accepted as training targets when their final answer matches the task answer.
- Primary run:
main_typed_bytecode_ei_s384_u4096.
Main Results
| Training regime | Split | Direct | Search | Program exact | Target found |
|---|---|---|---|---|---|
| Seed supervised | Fresh standard | 63.7% | 83.0% | 49.8% | 83.0% |
| Seed supervised | Fresh paraphrase | 63.9% | 83.4% | 50.4% | 83.4% |
| Seed supervised | Fresh paired | 61.5% | 81.6% | 43.4% | 81.6% |
| Seed supervised | Hard composition | 45.9% | 72.9% | 35.5% | 72.9% |
| Expert iteration R4 | Fresh standard | 74.4% | 88.7% | 56.2% | 88.7% |
| Expert iteration R4 | Fresh paraphrase | 71.9% | 87.1% | 56.1% | 87.1% |
| Expert iteration R4 | Fresh paired | 73.0% | 87.3% | 51.0% | 87.3% |
| Expert iteration R4 | Hard composition | 53.9% | 76.2% | 39.5% | 76.2% |
| Full supervised | Fresh standard | 99.8% | 100.0% | 99.6% | 100.0% |
| Full supervised | Fresh paraphrase | 99.6% | 100.0% | 99.6% | 100.0% |
| Full supervised | Fresh paired | 99.6% | 100.0% | 99.6% | 100.0% |
| Full supervised | Hard composition | 80.7% | 91.2% | 72.3% | 91.2% |
Expert-Iteration Curve
Answer-verified expert iteration produced a real deployable improvement, not just a search-time improvement. The same trained compiler is evaluated directly after each round.
Search Headroom
Local answer-verified search remained substantially stronger than direct decoding through the expert-iteration rounds, which means the compiler still leaves repairable mistakes on the table.
Target Quality
| Round | Targets | Found | Changed | Candidates | Valid candidates |
|---|---|---|---|---|---|
| 1 | 3404 | 83.1% | 26.3% | 241.0 | 63.4% |
| 2 | 3647 | 89.0% | 8.7% | 241.0 | 63.8% |
| 3 | 3741 | 91.3% | 6.1% | 240.9 | 63.5% |
| 4 | 3824 | 93.4% | 5.2% | 240.9 | 63.8% |
Frozen-Qwen Attached Pilot
A companion pilot attached the same bytecode head to frozen Qwen/Qwen3-4B token hidden states. Only the bytecode head was trained; Qwen itself was not LoRA-tuned in this run. This checks whether the method still has signal when the text front end is a real 4B model representation.
| Training regime | Split | Direct | Search | Program exact | Target found |
|---|---|---|---|---|---|
| Qwen seed | Fresh standard | 21.9% | 56.2% | 6.6% | 56.2% |
| Qwen seed | Fresh paraphrase | 16.0% | 53.1% | 3.5% | 53.1% |
| Qwen seed | Fresh paired | 17.6% | 53.5% | 4.3% | 53.5% |
| Qwen seed | Hard composition | 16.4% | 57.4% | 5.5% | 57.4% |
| Qwen expert iteration R3 | Fresh standard | 48.8% | 77.7% | 30.9% | 77.7% |
| Qwen expert iteration R3 | Fresh paraphrase | 43.8% | 76.2% | 27.7% | 76.2% |
| Qwen expert iteration R3 | Fresh paired | 50.4% | 74.2% | 34.4% | 74.2% |
| Qwen expert iteration R3 | Hard composition | 40.6% | 75.4% | 26.6% | 75.4% |
| Qwen full supervised | Fresh standard | 94.1% | 98.0% | 92.2% | 98.0% |
| Qwen full supervised | Fresh paraphrase | 96.1% | 99.6% | 94.5% | 99.6% |
| Qwen full supervised | Fresh paired | 94.5% | 99.2% | 92.6% | 99.2% |
| Qwen full supervised | Hard composition | 75.8% | 91.4% | 63.7% | 91.4% |
Qwen-head target quality:
| Round | Targets | Found | Changed | Candidates | Valid candidates |
|---|---|---|---|---|---|
| 1 | 1103 | 53.9% | 66.4% | 241.0 | 64.0% |
| 2 | 1365 | 66.7% | 50.5% | 241.0 | 63.9% |
| 3 | 1621 | 79.2% | 34.6% | 241.0 | 64.4% |
Interpretation
The result is positive for the typed-bytecode substrate and mixed for answer-only expert iteration. Dense bytecode traces are extremely effective: full supervision nearly saturates fresh standard, paraphrase, and paired splits. Expert iteration also helps, moving the seed compiler upward on every fresh split, but it does not approach the dense-trace ceiling. The frozen-Qwen pilot shows the same qualitative pattern: expert iteration improves the trainable Qwen-attached head, while dense bytecode supervision remains much stronger. This suggests that the next method improvement should focus on stronger process verification, multi-input consistency, or prefix-level search targets rather than merely increasing the number of final-answer-verified candidates.
The hard-composition split is the useful warning. Full supervision reached high but not saturated hard accuracy, while expert iteration improved less. This means the bytecode ABI is learnable, but longer or more compositional programs still need either more trace coverage or a better search/value loop.
Limitations
- The primary controlled run uses a compact transformer compiler; the separate Qwen-attached pilot trains only a head on frozen Qwen hidden states, not Qwen LoRA weights.
- The tasks are generated and bounded; they are not open-ended natural language reasoning tasks.
- Answer verification uses known task answers during training-target construction.
- Local search is slot-neighborhood search, not full program synthesis.
- Final-answer verification can accept accidental programs that compute the right scalar answer without matching the intended program.
Artifacts
Small files:
experiments/qwen_typed_bytecode_expert_iteration/runs/main_typed_bytecode_ei_s384_u4096/metrics.csvexperiments/qwen_typed_bytecode_expert_iteration/runs/main_typed_bytecode_ei_s384_u4096/train_log.csvexperiments/qwen_typed_bytecode_expert_iteration/runs/main_typed_bytecode_ei_s384_u4096/expert_targets.csvexperiments/qwen_typed_bytecode_expert_iteration/runs/qwen_head_pilot_s384_u2048/metrics.csvexperiments/qwen_typed_bytecode_expert_iteration/runs/qwen_head_pilot_s384_u2048/expert_targets.csvexperiments/qwen_typed_bytecode_expert_iteration/analysis/final_metrics.csvexperiments/qwen_typed_bytecode_expert_iteration/analysis/summary.mdexperiments/qwen_typed_bytecode_expert_iteration/reports/qwen_typed_bytecode_expert_iteration_paper.mdexperiments/qwen_typed_bytecode_expert_iteration/reports/qwen_typed_bytecode_expert_iteration_paper.html
Large files:
large_artifacts/qwen_typed_bytecode_expert_iteration/checkpoints/main_typed_bytecode_ei_s384_u4096/large_artifacts/qwen_typed_bytecode_expert_iteration/checkpoints/qwen_head_pilot_s384_u2048/
Experiment log 9
Show the running log (9 entries)
Setup
- Created a standalone typed-bytecode expert-iteration experiment directory.
- Required artifact layout:
src/,runs/,analysis/,analysis/figures/,reports/,checkpoint_manifest.csv, and large checkpoints underlarge_artifacts/qwen_typed_bytecode_expert_iteration/checkpoints/.
Iterations
Smoke: smoke_typed_bytecode_ei
- Purpose: validate the end-to-end script, logging, checkpoint layout, and typed VM execution.
- Result: unconstrained slot decoding emitted invalid bytecode, so candidate search had no useful foothold.
- Decision: add stack-depth-constrained bytecode decoding instead of training longer.
Smoke: smoke_typed_bytecode_ei_v2
- Purpose: retest after typed/stack-constrained decoding.
- Result: direct programs became valid often enough to measure, but the model was too small and undertrained for answer-verified search to collect useful targets.
- Decision: move to a real pilot with more warm-start supervision.
Pilot: pilot_typed_bytecode_ei_s128
- Purpose: test seed supervision, answer-verified expert iteration, and dense full supervision on a modest run.
- Result: full supervision reached about 60% fresh paired direct accuracy and about 80% search accuracy; expert iteration was positive but weak.
- Decision: isolate the dense supervised ceiling with larger trace coverage.
Pilot: pilot_supervised_ceiling_s2048
- Purpose: determine whether the typed-bytecode target is learnable with enough dense traces.
- Result: full supervision reached about 93-95% direct accuracy on fresh splits and about 98% with local search.
- Decision: the bytecode ABI is learnable; focus the final run on whether answer-verified expert iteration compounds.
Pilot: pilot_expert_iteration_r3_s256
- Purpose: run a larger answer-verified self-training loop with three rounds.
- Result: fresh paired direct accuracy improved from 32.8% to 56.2%, while search improved from 66.8% to 79.7%.
- Decision: run a larger main configuration combining four expert rounds and a dense supervised ceiling under the same evaluation splits.
Main: main_typed_bytecode_ei_s384_u4096
- Purpose: final standalone measurement.
- Result: fresh paired direct accuracy moved from 61.5% under the seed compiler to 73.0% after four expert-iteration rounds; dense full supervision reached 99.6%. Hard-composition direct accuracy moved from 45.9% to 53.9% under expert iteration and reached 80.7% under full supervision.
- Decision: write up as a positive result for typed-bytecode supervision and a partial result for answer-only expert iteration. The next bottleneck is process verification or prefix-level search, not the bytecode ABI itself.
Frozen-Qwen Pilot: qwen_head_pilot_s384_u2048
- Purpose: attach the typed-bytecode compiler head to frozen
Qwen/Qwen3-4Bhidden states rather than only using the compact controlled compiler. - Result: fresh paired direct accuracy improved from 17.6% under the seed Qwen-head compiler to 50.4% after three expert-iteration rounds. Dense full-supervised Qwen-head training reached 94.5% fresh paired direct accuracy.
- Decision: include this as a Qwen-attached pilot in the standalone paper. The next step should train Qwen adapters or improve process verification; the frozen-head result shows signal but still leaves a large gap to dense traces.
Figures 6
Data files 18
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
analysis/all_final_metrics.csv22 kBanalysis/final_metrics.csv5.0 kBruns/main_typed_bytecode_ei_s384_u4096/metrics.csv5.0 kBruns/main_typed_bytecode_ei_s384_u4096/results.json16 kBruns/pilot_expert_iteration_r3_s256/metrics.csv3.2 kBruns/pilot_expert_iteration_r3_s256/results.json11 kBruns/pilot_supervised_ceiling_s2048/metrics.csv2.5 kBruns/pilot_supervised_ceiling_s2048/results.json8.0 kBruns/pilot_typed_bytecode_ei_s128/metrics.csv3.1 kBruns/pilot_typed_bytecode_ei_s128/results.json10 kBruns/qwen_head_pilot_s384_u2048/metrics.csv3.8 kBruns/qwen_head_pilot_s384_u2048/results.json12 kBruns/smoke_frozen_qwen_head/metrics.csv1.9 kBruns/smoke_frozen_qwen_head/results.json6.9 kBruns/smoke_typed_bytecode_ei/metrics.csv1.6 kBruns/smoke_typed_bytecode_ei/results.json7.0 kBruns/smoke_typed_bytecode_ei_v2/metrics.csv2.6 kBruns/smoke_typed_bytecode_ei_v2/results.json8.1 kB
Reproduce
This entry is source code or analysis only — there is no separate run to reproduce.