Qwen Checkpoint-Selected Scheduled-State Compiler
The one idea you need
The model reads a recipe of add, subtract, and multiply steps, each result wrapping around at 97, then runs the whole 24-step chain. Coaching it during practice to keep every running total correct is a gentle reminder that helps it not lose its place.
The question
When you train a small model to run long chains of arithmetic, does making it keep each running total right help it get the final answer right?
What we found
It depends. A light, steady dose of show-your-work coaching, kept on through the hardest problems, lifted correct answers on 24-step chains from 25% to 33% and nearly doubled agreement between two wordings of the same problem, from 41% to 71%. But heavy coaching switched off for the long chains backfired at 19%, worse than no coaching at all.
Why it matters
If you supervise a model's intermediate steps, keep the pressure light and constant rather than strong-then-removed. And always save versions during training and pick the best one: here the final training step was reliably worse.
On this page
Results at a glance 3
How to read
Each cluster is one coaching recipe. One bar shows how often it solves long chains correctly; the other shows how often plainly worded and reworded versions produce the same running totals. Taller is better.
Takeaway → The two light-coaching-kept-on recipes stand tallest on both measures, while the switched-off-late recipe sinks below giving no coaching at all.
Data table
| state-loss schedule | paired execution accuracy | paired state consistency |
|---|---|---|
| no state loss | 25% | 40.6% |
| constant 0.25 | 32.8% | 55.9% |
| full early, off in long | 19.1% | 29.3% |
| full early, 0.25 in long | 32.2% | 70.7% |
Technical framing
Fresh length-24 retest: state-loss schedules vs no state loss — Keeping light state supervision in the long stage beats no state loss; turning it fully off in the long stage is worst.
How to read
Each recipe gets a pair of bars: its best-scoring saved version versus its final end-of-training version, both measured on solving long chains correctly. Taller is better.
Takeaway → Every final-version bar falls well below its best-saved counterpart, so simply ending training hands you a noticeably worse model.
Data table
| state-loss schedule | selected checkpoint (step 800) | final checkpoint (step 900) |
|---|---|---|
| no state loss | 27.3% | 15.6% |
| constant 0.25 | 30.5% | 12.5% |
| full early, off in long | 22.7% | 18% |
| full early, 0.25 in long | 30.5% | 15.6% |
Technical framing
Checkpoint selection is necessary: selected step 800 vs final step 900 — Every arm collapses by the final training step; reporting the best validation checkpoint roughly doubles length-24 execution.
How to read
Bars are grouped by wording — plainly stated, reworded, and both-must-match — with one colored bar per coaching recipe. Taller is better.
Takeaway → Recipes barely differ on plainly worded problems but split widely on reworded ones, where light coaching stays high and switched-off-late collapses.
Data table
| program rendering | no state loss | constant 0.25 | full early, off in long | full early, 0.25 in long |
|---|---|---|---|---|
| standard | 32% | 34% | 31.2% | 34% |
| paraphrase | 19.5% | 31.2% | 7.4% | 28.1% |
| paired | 25% | 32.8% | 19.1% | 32.2% |
Technical framing
Fresh length-24 retest by rendering: paraphrase robustness — Standard-rendering accuracy is similar across arms; state supervision in the long stage mainly protects paraphrased programs.
In the author’s words from the Report · “Abstract”
This experiment tests whether intermediate-state supervision improves a QLoRA-adapted Qwen/Qwen3-4B numeric-copy compiler when the training process saves real validation checkpoints and reports the best saved checkpoint rather than the final optimizer step. Each prompt describes an initial value and a sequence of add, subtract, and multiply updates modulo 97. The model reads the prompt, copies the initial value, operation sequence, and operation arguments from token-level maps, then executes the copied program with an invisible modular-arithmetic runtime. The primary target is exact execution on length-24 programs, including paired standard/paraphrased renderings of the same latent program. The main result is positive but specific. … Read the full result →
Overview
Standalone experiment testing whether stage-specific intermediate-state supervision and validation checkpoint selection improve a Qwen numeric-copy compiler on long modular-arithmetic programs.
The compiler reads hidden states from Qwen/Qwen3-4B, copies the initial value, operations, and arguments from token-level maps, executes the copied program modulo 97 with an invisible runtime, and reports exact execution metrics. The intervention is not a new parser or task; it is the training policy around state supervision:
- save a real checkpoint at every validation point;
- select the checkpoint by paired length-24 exact execution;
- compare no state loss, constant light state loss, and scheduled state loss that is active before the long-chain stage and reduced or disabled during it.
Layout
src/qwen_checkpoint_scheduled_state_compiler_experiment.py training and evaluation harness
src/analyze_qwen_checkpoint_scheduled_state_compiler.py run aggregation, selected-checkpoint table, and plots
src/evaluate_selected_qwen_checkpoint_scheduled_state_compiler.py
fresh retest for selected checkpoints
runs/ lightweight JSON and CSV outputs
analysis/ aggregate CSVs and generated figures
reports/ experiment log and standalone write-up
checkpoint_manifest.csv generated list of large checkpoint filesLarge checkpoints are stored outside the experiment directory:
large_artifacts/qwen_checkpoint_scheduled_state_compiler/checkpoints/Main Metrics
executor_accuracy: exact final answer after copying and executing the compiled program.program_exact: exact full compiled-program correctness.state_all_exact: every intermediate modular state is correct.state_prefix_fraction: fraction of the program executed before the first state error.compiler_pair_state_consistency: whether standard and paraphrased renderings compile to the same complete state trajectory.selected_checkpoint: saved checkpoint chosen bypaired_len24_executor_accuracy.
Reading Order
reports/qwen_checkpoint_scheduled_state_compiler_experiment_log.mdanalysis/summary.mdanalysis/selected_checkpoints.csvanalysis/selected_retest_metrics.csvanalysis/final_metrics.csvreports/qwen_checkpoint_scheduled_state_compiler_paper.md
Report
Rendered from reports/qwen_checkpoint_scheduled_state_compiler_paper.md
Abstract
This experiment tests whether intermediate-state supervision improves a QLoRA-adapted Qwen/Qwen3-4B numeric-copy compiler when the training process saves real validation checkpoints and reports the best saved checkpoint rather than the final optimizer step.
Each prompt describes an initial value and a sequence of add, subtract, and multiply updates modulo 97. The model reads the prompt, copies the initial value, operation sequence, and operation arguments from token-level maps, then executes the copied program with an invisible modular-arithmetic runtime. The primary target is exact execution on length-24 programs, including paired standard/paraphrased renderings of the same latent program.
The main result is positive but specific. Light intermediate-state supervision improves selected-checkpoint length-24 robustness. On a fresh held-out retest of selected checkpoints, the no-state compiler reaches 25.0% paired length-24 exact execution. Constant state loss weight 0.25 reaches 32.8%, and a full-early/light-long schedule reaches 32.2%. The full-early/light-long schedule also gives the best paired state consistency, 70.7%, versus 40.6% for the no-state compiler. In contrast, full early state loss followed by zero long-stage state loss underperforms at 19.1% paired length-24 exact execution.
Checkpoint selection is essential. Every main arm's best selected checkpoint is at step 800, while the final step 900 checkpoint is worse on paired length-24 exact execution.
Setup
- Base model:
Qwen/Qwen3-4B - Loader:
AutoModelForCausalLM - Quantization: 4-bit NF4
- Trainable update: LoRA rank 8, alpha 16, dropout 0.05, target
all-linear - Trainable LoRA parameters: 16,515,072
- Compiler head width: 768
- Task: modular arithmetic programs modulo 97
- Curriculum:
short:1:4:200,medium:1:8:200,train:1:12:200,long:8:24:300 - Training examples: 512 paired programs per curriculum stage, rendered as standard/paraphrase pairs
- Training batch size: 4
- Validation size during training: 64 examples per unpaired split and 64 paired programs per paired split
- Fresh retest size: 256 standard length-24 programs, 256 paraphrase length-24 programs, and 256 paired length-24 programs per selected checkpoint
- Selection metric:
paired_len24_executor_accuracy - Large checkpoint root:
large_artifacts/qwen_checkpoint_scheduled_state_compiler/checkpoints/
The compiler predicts token positions for the initial value, ordered operation slots, and ordered argument slots. Values and operations are copied from deterministic token maps. The copied program is executed exactly modulo 97. For state-supervised arms, the differentiable executor also produces a distribution over the modular state after every active operation, and training includes NLL against the true state trajectory according to the schedule being tested.
Conditions
| Run | State-loss schedule | Purpose |
|---|---|---|
main_no_state_selected_s900 | all stages 0.0 | Matched compiler with no intermediate-state loss. |
main_state_w025_selected_s900 | all stages 0.25 | Constant light state supervision. |
main_state_l12_off_long_selected_s900 | 1.0 through length-12 training, 0.0 in long stage | Strong early state scaffold, removed for long programs. |
main_state_l12_w025_long_selected_s900 | 1.0 through length-12 training, 0.25 in long stage | Strong early state scaffold, light state objective for long programs. |
All main arms use the same base model, LoRA configuration, curriculum, paired training distribution, explicit training seed, and validation splits. Each arm is launched separately so it starts from the same base model rather than inheriting weights from another condition.
Selected Checkpoints
Each validation point saves an adapter and compiler-head checkpoint. The selected checkpoint is the saved checkpoint with the highest paired length-24 exact execution on the in-training validation split.
| Run | Selected step | Paired L24 exact | Paired state consistency | Standard L24 exact | Paraphrase L24 exact | Final paired L24 exact |
|---|---|---|---|---|---|---|
| No state | 800 | 27.3% | 43.8% | 35.9% | 12.5% | 15.6% |
| State 0.25 constant | 800 | 30.5% | 60.9% | 34.4% | 26.6% | 12.5% |
| State 1.0 then 0.0 | 800 | 22.7% | 32.8% | 34.4% | 4.7% | 18.0% |
| State 1.0 then 0.25 | 800 | 30.5% | 75.0% | 39.1% | 21.9% | 15.6% |
The selected-checkpoint table shows two things. First, light state loss improves the primary validation metric. Second, the final checkpoint is not a reliable estimator of best model quality: all four arms peak before step 900 on paired length-24 exact execution.
Fresh Retest
The selected checkpoints were retested on fresh length-24 programs generated with a different seed and larger sample size.
| Run | Standard L24 exact | Paraphrase L24 exact | Paired L24 exact | Paired state consistency | Paired both-correct |
|---|---|---|---|---|---|
| No state | 32.0% | 19.5% | 25.0% | 40.6% | 17.2% |
| State 0.25 constant | 34.0% | 31.2% | 32.8% | 55.9% | 29.7% |
| State 1.0 then 0.0 | 31.2% | 7.4% | 19.1% | 29.3% | 6.2% |
| State 1.0 then 0.25 | 34.0% | 28.1% | 32.2% | 70.7% | 30.1% |
The fresh retest confirms that the selected-checkpoint improvement is not just a validation-set artifact. Constant light state loss gives the highest fresh paired exact execution, 32.8%. The full-early/light-long schedule is nearly tied on exact execution at 32.2% and gives the strongest representation-level result, with 70.7% paired state consistency.
The full-early/off-long schedule is worse than the no-state control. That result argues against treating strong state loss as a pure early scaffold that should simply be removed for long programs. Some light long-stage state pressure appears useful.
Interpretation
The experiment supports three claims.
First, checkpoint selection is a real experimental requirement for this training setup. The best saved checkpoints occur at step 800, not the final step. Reporting only final checkpoints would reverse or obscure the main result.
Second, intermediate-state supervision helps when it is light enough. Constant weight 0.25 is the best exact-execution arm on the fresh paired retest. The full-early/light-long schedule is the best state-consistency arm and is nearly tied on exact execution. Both outperform the no-state compiler on fresh paired length-24 exact execution.
Third, strong state supervision is not automatically better. Full early state loss followed by zero long-stage state loss underperforms on paired exact execution, paraphrase exact execution, and paired state consistency. The state target is useful as a bias, but too much or poorly scheduled pressure can steer the compiler toward brittle slot/state behavior.
The absolute accuracy remains modest. The best fresh paired length-24 exact execution is 32.8%, so the compiler is still limited by compounding slot and argument errors. The improvement is nevertheless meaningful because it appears on the hardest paired/paraphrased target and survives a fresh retest.
Limitations
- Single main seed.
- Synthetic arithmetic task.
- The selected checkpoint is chosen on a validation split that is evaluated repeatedly during training; the fresh retest mitigates but does not replace multi-seed confirmation.
- The method improves a structured compiler interface, not the base model's unconstrained generation.
- The fresh retest evaluates selected checkpoints only, not every saved checkpoint on fresh data.
Next Experiment
The next experiment should replicate the two best arms with multiple seeds and one training-policy change: reduce or anneal the long-stage learning rate after step 800, or stop long-stage training at the selected checkpoint. The current result shows that useful models are being produced and then degraded. The most direct next question is whether the degradation is an optimizer schedule problem rather than an architecture or objective problem.
Artifacts
Small files:
- Source:
experiments/qwen_checkpoint_scheduled_state_compiler/src/ - Runs:
experiments/qwen_checkpoint_scheduled_state_compiler/runs/ - Analysis:
experiments/qwen_checkpoint_scheduled_state_compiler/analysis/ - Report:
experiments/qwen_checkpoint_scheduled_state_compiler/reports/qwen_checkpoint_scheduled_state_compiler_paper.md - Experiment log:
experiments/qwen_checkpoint_scheduled_state_compiler/reports/qwen_checkpoint_scheduled_state_compiler_experiment_log.md - Manifest:
experiments/qwen_checkpoint_scheduled_state_compiler/checkpoint_manifest.csv
Large files:
- Checkpoints:
large_artifacts/qwen_checkpoint_scheduled_state_compiler/checkpoints/
Experiment log 8
Show the running log (8 entries, 2026-06-22)
Objective
Test whether checkpoint selection and stage-specific intermediate-state supervision improve a Qwen numeric-copy compiler's long-chain exact execution.
Experiment Question
Can a QLoRA-adapted compiler reach better length-24 execution when state supervision is treated as a curriculum tool rather than a fixed objective, and when the reported model is the best saved validation checkpoint rather than the final optimizer step?
Planned Conditions
- Smoke tests for scheduled state weights and checkpoint snapshots.
- Matched no-state curriculum compiler with validation checkpoint selection.
- Constant light scheduled-state compiler with state loss weight 0.25 in every stage.
- State-through-medium compiler with state loss active before the long stage and disabled in the long stage.
- Optional state-through-medium-light-long compiler with full state loss before the long stage and weight 0.25 in the long stage.
Primary Selection Rule
Select the saved checkpoint with the highest paired_len24_executor_accuracy.
Primary Evaluation Metrics
paired_len24_executor_accuracypaired_len24_compiler_pair_state_consistencystandard_len24_executor_accuracyparaphrase_len24_executor_accuracystate_prefix_fractionprogram_exact
Artifact Policy
Lightweight outputs stay in:
experiments/qwen_checkpoint_scheduled_state_compiler/runs/
experiments/qwen_checkpoint_scheduled_state_compiler/analysis/
experiments/qwen_checkpoint_scheduled_state_compiler/reports/Large adapters and head checkpoints stay in:
large_artifacts/qwen_checkpoint_scheduled_state_compiler/checkpoints/Log
2026-06-22
- Created standalone experiment directory.
- Forked the Qwen numeric-copy compiler harness into a checkpoint-selected scheduled-state harness.
- Added stage-specific state-loss schedules.
- Added validation checkpoint snapshots and selected-checkpoint metadata.
- Added standalone README and experiment log.
- Ran
smoke_tiny_scheduled_statewith a tiny frozen causal LM. The run completed, saved validation checkpoints, selected a checkpoint bypaired_len3_executor_accuracy, generated analysis CSVs, and verified that state loss was active in the short stage and inactive in the long stage. - Ran
smoke_qwen3_4b_scheduled_statewithQwen/Qwen3-4B, 4-bit QLoRA, and scheduled state loss. The run completed, saved real adapter checkpoints, generated analysis CSVs, and verified that the large artifacts are stored outside the experiment directory. - Ran
main_no_state_selected_s900. The selected checkpoint is step 800 with 27.3% paired length-24 exact execution, 43.8% paired state consistency, 35.9% standard length-24 exact execution, and 12.5% paraphrase length-24 exact execution. The final step 900 checkpoint fell to 15.6% paired length-24, confirming that checkpoint selection is necessary for this setup. - Ran
main_state_w025_selected_s900. The selected checkpoint is step 800 with 30.5% paired length-24 exact execution, 60.9% paired state consistency, 34.4% standard length-24 exact execution, and 26.6% paraphrase length-24 exact execution. The final step 900 checkpoint fell to 12.5% paired length-24 and 0.0% paraphrase length-24. - Ran
main_state_l12_off_long_selected_s900. The schedule used state loss weight 1.0 through the train stage and 0.0 in the long stage. The selected checkpoint is step 800 with 22.7% paired length-24 exact execution, 32.8% paired state consistency, 34.4% standard length-24 exact execution, and 4.7% paraphrase length-24 exact execution. The final checkpoint reached 37.5% standard length-24 but only 18.0% paired length-24. - Ran
main_state_l12_w025_long_selected_s900. The schedule used state loss weight 1.0 through the train stage and 0.25 in the long stage. The selected checkpoint is step 800 with 30.5% paired length-24 exact execution, 75.0% paired state consistency, 39.1% standard length-24 exact execution, and 21.9% paraphrase length-24 exact execution. The final checkpoint fell to 15.6% paired length-24. - Added an independent selected-checkpoint retest on fresh length-24 programs with 256 standard examples, 256 paraphrase examples, and 256 paired programs per arm. Fresh paired length-24 exact execution: no-state 25.0%, constant 0.25 state 32.8%, full-early/off-long 19.1%, full-early/light-long 32.2%. Fresh paired state consistency: no-state 40.6%, constant 0.25 state 55.9%, full-early/off-long 29.3%, full-early/light-long 70.7%.
Figures 5
Data files 10
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
analysis/all_final_metrics.csv14 kBanalysis/final_metrics.csv14 kBanalysis/selected_retest_metrics.csv5.0 kBanalysis/selected_retest_results.json13 kBruns/main_no_state_selected_s900/results.json135 kBruns/main_state_l12_off_long_selected_s900/results.json138 kBruns/main_state_l12_w025_long_selected_s900/results.json138 kBruns/main_state_w025_selected_s900/results.json137 kBruns/smoke_qwen3_4b_scheduled_state/results.json38 kBruns/smoke_tiny_scheduled_state/results.json38 kB
Reproduce
This entry is source code or analysis only — there is no separate run to reproduce.