Qwen Teacher-Distilled Slot Compiler
The one idea you need
The model acts like a clerk working a 24-line arithmetic worksheet: each line, it lifts one number and one operation out of written wording and runs them in order. The extra training highlighted exactly where each number sat on the page. But the clerk already found them.
The question
If a model already knows which numbers to pull out of a written multi-step math problem, does training that points to exactly where each one sits make it more accurate?
What we found
No. A model trained to copy numbers and operations out of text and run a 24-step calculation got 27% of final answers exactly right; adding the pointing signal landed at 28%, a tie. Worse, agreement between two rewordings of the same problem fell from 73% to 56%. A lighter version of the signal was clearly worse, at 18%.
Why it matters
When a system copies structured values out of text and chains them, do not add supervision on where values sit once the model already locates them over 90% of the time. Spend effort catching and repairing the small copy slips that compound across steps.
On this page
Results at a glance 3
How to read
Three clusters, one per training setup: plain training and two hint-added versions. In each cluster, one bar is how often the final answer is exactly right, the other how often two rewordings of the same problem agree. Taller is better.
Takeaway → The exact-answer bars for plain and full-hint training tie while the light-hint bar sits clearly lower, and plain training's agreement bar towers over both hint versions, so the hints added nothing and cost consistency.
Data table
| training arm | exact execution | paired state consistency |
|---|---|---|
| no-teacher control | 27.1% | 72.7% |
| teacher slot distill | 27.9% | 55.5% |
| soft-position-only teacher | 18.4% | 41% |
Numbers from analysis/selected_retest_metrics.csv (fresh_paired_len24 rows)
Technical framing
Teacher slot distillation does not beat the no-teacher control (fresh length-24 retest) — The teacher arms match or lose to the control on exact execution and are clearly worse on paraphrase-pair consistency.
How to read
Horizontal axis is how many steps the problem has; vertical axis is how often the final answer comes out exactly right. Each line is one training setup; higher and flatter is better.
Takeaway → All three lines drop steeply from short to long problems and sit almost on top of each other, so length, not missing hints, drives the failures.
Data table
| program length (operations) | no-teacher control | teacher slot distill | soft-position-only teacher |
|---|---|---|---|
| 4 | 86.7% | 85.9% | 86.7% |
| 8 | 61.7% | 62.5% | 62.5% |
| 12 | 59.4% | 60.2% | 60.2% |
| 24 | 23.4% | 22.7% | 18.8% |
Numbers from analysis/summary.md final-metrics table (paired_len4/8/12/24 rows, executor_accuracy)
Technical framing
Exact execution degrades with program length in much the same way across arms — All three arms decay similarly as programs get longer (the soft-position arm dips most at length 24), so local slot imitation is not the bottleneck.
How to read
Two groups compare original wording against reworded wording. Inside each group, one bar per training setup shows how often the final answer is exactly right; a big drop from left to right means that setup is fragile to rewording.
Takeaway → The lighter-hint bar collapses from about a quarter correct to roughly one in nine when reworded, while the other two setups barely move.
Data table
| prompt rendering | no-teacher control | teacher slot distill | soft-position-only teacher |
|---|---|---|---|
| standard wording | 30.5% | 30.5% | 27% |
| paraphrased wording | 27.7% | 28.1% | 11.7% |
Numbers from analysis/selected_retest_metrics.csv (fresh_standard_len24 and fresh_paraphrase_len24 rows)
Technical framing
The soft-position-only teacher is fragile under paraphrase (fresh length-24 retest) — Control and full-teacher arms lose only ~2-3 points to paraphrase; the soft-position-only arm collapses from 27% to 12%.
In the author’s words from the Report · “Abstract”
This experiment tests whether oracle slot-teacher losses improve a QLoRA-adapted Qwen/Qwen3-4B compiler on synthetic modular-arithmetic programs. The compiler reads the full prompt hidden sequence, predicts executable slots, copies numbers and operations, and executes the copied program modulo 97. The teacher variants add auxiliary supervision for slot localization, with one arm also matching oracle token representations. The result is negative. On fresh length-24 paired standard/paraphrase programs, the matched no-teacher control scored 27.1% exact execution with 72.7% paired compiler-state consistency. The heavier teacher arm scored 27.9% exact execution but only 55.5% consistency. … Read the full result →
Overview
Standalone experiment testing whether an oracle slot teacher improves a QLoRA-adapted Qwen/Qwen3-4B numeric-copy compiler on long modular-arithmetic programs.
The student compiler reads the full prompt hidden sequence, predicts ordered executable slots, copies numeric and operation symbols from token-level maps, and executes the copied program modulo 97. The teacher condition adds two auxiliary signals during training:
- soft local position targets around the oracle slot span;
- slot-representation matching between the student-attended slot vector and the oracle token vector.
Main Result
Teacher slot distillation did not beat the matched no-teacher control on fresh length-24 paired programs. The control scored 27.1% exact execution and 72.7% paired compiler-state consistency. The heavier teacher arm scored 27.9% exact execution but only 55.5% consistency. The low-weight soft-position-only teacher arm scored 18.4% exact execution and 41.0% consistency.
The practical takeaway is that local oracle slot imitation is not the current bottleneck; the next experiment should target execution-level repair or verifier-guided search.
Layout
src/qwen_teacher_distilled_slot_compiler_experiment.py training and evaluation harness
src/analyze_qwen_teacher_distilled_slot_compiler.py aggregation, selected checkpoint table, and plots
src/evaluate_selected_qwen_teacher_distilled_slot_compiler.py
fresh retest for selected checkpoints
runs/ lightweight JSON and CSV outputs
analysis/ aggregate CSVs and generated figures
reports/ experiment log and standalone write-up
checkpoint_manifest.csv generated list of large checkpoint filesLarge checkpoints are stored outside the experiment directory:
large_artifacts/qwen_teacher_distilled_slot_compiler/checkpoints/Main Metrics
executor_accuracy: exact final answer after copying and executing the compiled program.program_exact: exact full compiled-program correctness.arg_accuracy: per-argument copied value accuracy.arg_pos_accuracy: per-argument slot localization accuracy.compiler_pair_state_consistency: whether standard and paraphrased renderings compile to the same complete state trajectory.selected_checkpoint: saved checkpoint chosen bypaired_len24_executor_accuracy.
Reading Order
reports/qwen_teacher_distilled_slot_compiler_experiment_log.mdanalysis/summary.mdanalysis/selected_checkpoints.csvanalysis/selected_retest_metrics.csvanalysis/final_metrics.csvreports/qwen_teacher_distilled_slot_compiler_paper.md
Report
Rendered from reports/qwen_teacher_distilled_slot_compiler_paper.md
Abstract
This experiment tests whether oracle slot-teacher losses improve a QLoRA-adapted Qwen/Qwen3-4B compiler on synthetic modular-arithmetic programs. The compiler reads the full prompt hidden sequence, predicts executable slots, copies numbers and operations, and executes the copied program modulo 97. The teacher variants add auxiliary supervision for slot localization, with one arm also matching oracle token representations.
The result is negative. On fresh length-24 paired standard/paraphrase programs, the matched no-teacher control scored 27.1% exact execution with 72.7% paired compiler-state consistency. The heavier teacher arm scored 27.9% exact execution but only 55.5% consistency. The lower-weight soft-position-only arm scored 18.4% exact execution and 41.0% consistency. Oracle slot imitation did not improve the selected checkpoint and often made paraphrase-invariant compilation worse.
Question
Can a Qwen-attached numeric compiler bind executable slots more reliably when ordinary trace and state supervision are augmented with oracle slot-teacher losses?
The hypothesis was that a substantial part of the remaining error came from misbinding surface tokens to executable slots. If true, teacher losses should improve long-chain execution and standard/paraphrase consistency, especially on length-24 programs where one copied-step error usually corrupts the final answer.
Method
All main runs used Qwen/Qwen3-4B with 4-bit QLoRA adapters, a copied-symbol program compiler head, a deterministic modular executor, and the same synthetic program curriculum:
| Stage | Length range | Steps |
|---|---|---|
| short | 1-4 | 200 |
| medium | 1-8 | 200 |
| train | 1-12 | 200 |
| long | 8-24 | 300 |
The model was trained with paired standard/paraphrase examples and selected by highest validation paired_len24_executor_accuracy. Large adapter and head checkpoints are stored outside the experiment directory under large_artifacts/qwen_teacher_distilled_slot_compiler/checkpoints/.
Three main arms were run:
| Run | Teacher position weight | Teacher representation weight | Description |
|---|---|---|---|
main_control_light_state_s900 | 0.00 | 0.00 | Matched light-state compiler control |
main_teacher_slot_distill_s900 | 0.10 | 0.05 | Soft local position targets plus oracle slot-representation matching |
main_teacher_softpos_low_s900 | 0.03 | 0.00 | Lower-weight soft position targets only |
The primary held-out retest used 256 fresh length-24 programs per single-rendering split and 256 paired programs rendered in both standard and paraphrased forms, giving 512 examples in the paired row.
Results
Selected Validation Checkpoints
| Run | Selected step | Paired L24 exact | Paired state consistency | Standard L24 exact | Paraphrase L24 exact |
|---|---|---|---|---|---|
| Control | 800 | 30.5% | 67.2% | 37.5% | 25.0% |
| Teacher position + representation | 800 | 28.1% | 59.4% | 37.5% | 21.9% |
| Low soft-position only | 800 | 23.4% | 32.8% | 29.7% | 15.6% |
The matched control was the best validation-selected arm. Both teacher variants selected the same training step but had lower paired exact execution and lower paired compiler-state consistency.
Fresh Selected-Checkpoint Retest
| Run | Fresh standard L24 exact | Fresh paraphrase L24 exact | Fresh paired L24 exact | Fresh paired state consistency |
|---|---|---|---|---|
| Control | 30.5% | 27.7% | 27.1% | 72.7% |
| Teacher position + representation | 30.5% | 28.1% | 27.9% | 55.5% |
| Low soft-position only | 27.0% | 11.7% | 18.4% | 41.0% |
The heavier teacher arm had a tiny fresh paired exact-execution edge over the control, 27.9% versus 27.1%, but this is not a useful win: it came with a large consistency loss, and its selected validation metric was lower. The low-weight position-only arm was clearly worse on the fresh retest, especially on paraphrased prompts.
Interpretation
The teacher losses did not address the dominant bottleneck. The compiler already achieved near-perfect initialization and operation localization, and argument localization stayed around 93-95% at length 24 in the main arms. The remaining failure mode is therefore not simply "the model cannot find the right token." Small residual copy errors compound across 24 steps, and the paired consistency metrics show that the learned compiler is still sensitive to surface rendering.
The representation-matching teacher likely over-constrained the compiler to prompt-surface-specific hidden vectors. That explains why the heavier teacher arm tied answer accuracy but lost paired compiler-state consistency. The low-weight soft-position-only arm briefly improved mid-training transfer at step 600, but the effect did not survive checkpoint selection or fresh retesting.
Conclusion
Oracle slot imitation is not the highest-leverage next direction for this Qwen-attached compiler. It can provide a transient curriculum signal, but it does not improve the best selected checkpoint and can damage paraphrase-invariant compilation.
The most impactful next experiment should move from slot imitation to execution-level pressure:
- Train the compiler with verifier-guided self-correction or search over copied slots, so the model learns from execution failures rather than only from local token labels.
- Add a differentiable or sampled repair loop that proposes alternate arguments for low-confidence slots and backpropagates or reinforces the execution result.
- Use a short policy-gradient stage over compiled programs with a strong supervised anchor, rewarding exact final execution and paired standard/paraphrase program agreement.
The practical recommendation is to start with verifier-guided slot repair, because it directly targets the observed failure mode: high per-slot accuracy that still collapses under long-chain composition.
Experiment log 9
Show the running log (9 entries, 2026-06-22)
Objective
Test whether oracle slot-teacher losses improve a Qwen numeric-copy compiler's long-chain exact execution and paired paraphrase consistency.
Experiment Question
Can a text compiler bind executable slots more reliably when hard trace supervision is augmented with soft local oracle slot targets and slot-representation matching?
Planned Conditions
- Tiny-model smoke test for teacher losses and checkpoint snapshots.
- Qwen smoke test for 4-bit QLoRA teacher training.
- Matched light-state control with no teacher losses.
- Teacher-distilled light-state compiler with soft position and representation losses.
- Low-weight soft-position-only teacher arm.
- Fresh selected-checkpoint retest on length-24 programs.
Primary Selection Rule
Select the saved checkpoint with the highest paired_len24_executor_accuracy.
Primary Metrics
paired_len24_executor_accuracyfresh_paired_len24_executor_accuracyfresh_paired_len24_compiler_pair_state_consistencyarg_accuracyarg_pos_accuracyprogram_exact
Artifact Policy
Lightweight outputs stay in:
experiments/qwen_teacher_distilled_slot_compiler/runs/
experiments/qwen_teacher_distilled_slot_compiler/analysis/
experiments/qwen_teacher_distilled_slot_compiler/reports/Large adapters and head checkpoints stay in:
large_artifacts/qwen_teacher_distilled_slot_compiler/checkpoints/Log
2026-06-22
- Created standalone experiment directory.
- Forked the checkpoint-selected Qwen numeric-copy compiler harness into a teacher-distilled slot harness.
- Added soft oracle position teacher losses.
- Added oracle slot-representation matching losses.
- Added standalone README and experiment log.
- Ran
smoke_tiny_teacherwith a tiny frozen causal LM. The run completed, saved validation checkpoints, generated analysis outputs, and confirmed that teacher position and representation losses are present in the train log. - Ran
smoke_qwen3_4b_teacherwithQwen/Qwen3-4B, 4-bit QLoRA, state loss weight 0.25, teacher position weight 0.1, and teacher representation weight 0.05. The run completed, saved real adapter checkpoints, and showed teacher losses in a reasonable scale relative to the total loss. - Ran
main_control_light_state_s900, the matched light-state control with no teacher losses. The selected checkpoint is step 800 with 30.5% paired length-24 exact execution, 67.2% paired state consistency, 37.5% standard length-24 exact execution, and 25.0% paraphrase length-24 exact execution. The final checkpoint fell to 23.4% paired length-24. - Ran
main_teacher_slot_distill_s900with teacher position weight 0.1 and teacher representation weight 0.05. The selected checkpoint is step 800 with 28.1% paired length-24 exact execution, 59.4% paired state consistency, 37.5% standard length-24 exact execution, and 21.9% paraphrase length-24 exact execution. The final checkpoint fell to 22.7% paired length-24. - Ran a fresh selected-checkpoint retest for the matched control and heavier teacher arms at length 24 with
eval_size=256andeval_seed=92001. The control scored 27.1% paired exact execution and 72.7% paired state consistency. The heavier teacher arm scored 27.9% paired exact execution and 55.5% paired state consistency. The answer accuracy difference was tiny, but the teacher arm lost substantial paired compiler consistency. - Added and ran
main_teacher_softpos_low_s900after inspecting the heavier teacher result. This arm removes representation matching and uses only a low soft-position teacher weight of 0.03. It briefly improved mid-training length-24 transfer at step 600: 13.3% paired length-24 exact execution versus 1.6% for the matched control at the same step. The selected checkpoint was still step 800 and underperformed the control: 23.4% paired length-24 exact execution, 32.8% paired state consistency, 29.7% standard length-24 exact execution, and 15.6% paraphrase length-24 exact execution. - Ran the same fresh selected-checkpoint retest for
main_teacher_softpos_low_s900. It scored 18.4% paired length-24 exact execution and 41.0% paired state consistency. This confirms that low-weight soft slot supervision did not improve the selected checkpoint and damaged paraphrase robustness.
Result
Oracle slot-teacher losses did not beat the matched no-teacher light-state compiler. The best fresh paired length-24 exact execution was effectively tied between the control and the heavier teacher arm, 27.1% versus 27.9%, while paired state consistency dropped from 72.7% to 55.5%. The low-weight soft-position-only variant was worse at 18.4% paired exact execution and 41.0% paired state consistency.
The most useful observation is negative: the compiler already localizes symbols well enough that extra oracle slot imitation is not the current bottleneck. The next experiment should move the training signal closer to execution correctness or search, rather than adding more slot-position supervision.
Figures 5
Data files 9
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
analysis/all_final_metrics.csv11 kBanalysis/final_metrics.csv11 kBanalysis/selected_retest_metrics.csv3.9 kBanalysis/selected_retest_results.json12 kBruns/main_control_light_state_s900/results.json138 kBruns/main_teacher_slot_distill_s900/results.json141 kBruns/main_teacher_softpos_low_s900/results.json140 kBruns/smoke_qwen3_4b_teacher/results.json40 kBruns/smoke_tiny_teacher/results.json40 kB
Reproduce
This entry is source code or analysis only — there is no separate run to reproduce.