Qwen Register-Token Structured Runtime
The one idea you need
The model never says the answer. It writes a step-by-step arithmetic program into a fixed row of hidden memory slots — a starting number, then one operation each — and a fixed calculator runs the slots in order, blindly, unable to notice or fix a wrong one.
The question
Can a small model secretly write a step-by-step math program into hidden memory slots and have a fixed calculator run it correctly, even for long chains?
What we found
Only up to a point. For chains of four to twelve steps the hidden program runs flawlessly, at 100 percent. But at 24 steps exact execution collapses to 25 percent — versus about 1 percent from pure guessing. The catch: each individual step is 80 to 94 percent right, yet a single wrong step anywhere breaks the whole answer, so mostly-right steps still add up to a mostly-wrong long program.
Why it matters
If you route reasoning through hidden slots run by a fixed engine, expect a hard length cliff: 90-percent-per-step still fails most long chains. Measure whole-chain exactness, not per-step accuracy, and add a repair pass over long programs before trusting them.
On this page
Results at a glance 3
How to read
Grouped bars for three 24-step test sets — standard wording, reworded, and paired-wording prompts. Bar height is the share of chains executed exactly right; taller is better. Dark bars are trained for wording consistency, light bars a matched version without it.
Takeaway → Every trained bar beats its control — 25 versus 4 percent on standard wording — yet all stay low, so long-chain execution is helped but far from solved.
Data table
| evaluation split | main (paired consistency) | no-pair control |
|---|---|---|
| standard L24 | 25% | 3.9% |
| paraphrase L24 | 5.5% | 1.6% |
| paired L24 | 11.7% | 1.6% |
Numbers from analysis/all_final_metrics.csv
Technical framing
Length-24 execution: paired-consistency training vs matched control — Paired-consistency training lifts length-24 execution over a matched control, but long prompt-invariant execution stays unsolved.
How to read
Each line follows one prompt type — standard, reworded, paired — as program length grows from 4 to 24 steps along the horizontal axis. Vertical axis is the share of chains executed exactly right; higher is better.
Takeaway → All three lines hug 100 percent through 12 steps, then plunge to between 5 and 25 percent at 24 — a sharp cliff, not a gentle slope.
Data table
| program length (ops) | standard | paraphrase | paired |
|---|---|---|---|
| 4 | 100% | 100% | 100% |
| 8 | 100% | 100% | 100% |
| 12 | 100% | 99.2% | 100% |
| 24 | 25% | 5.5% | 11.7% |
Numbers from analysis/final_metrics.csv
Technical framing
Executor accuracy vs program length (main run) — The trained length range (4-12) is solved essentially perfectly; accuracy collapses at length 24 on every split.
How to read
Bars for each piece of a 24-step standard program: starting number, operation choices, operation inputs, running-state prefix, full program, and final answer. Height is accuracy; higher is better; the left pieces are easiest.
Takeaway → The starting number is perfect and each piece scores 80 to 94 percent, but full-program and final accuracy crash to 25 percent as small errors compound.
Data table
| program component | main run, standard L24 |
|---|---|
| init residue | 100% |
| op routes | 93.8% |
| op args | 89.7% |
| state prefix | 80.5% |
| full program | 25% |
| executor exact | 25% |
Numbers from analysis/final_metrics.csv
Technical framing
Where length-24 fails: per-slot accuracy on standard L24 (main run) — Individual slots are mostly right (80-94%), but small per-op errors compound over 24 steps, so exact execution drops to 25%.
In the author’s words from the Overview · “Result”
The main run solves the trained length range and partially lifts length-24 standard execution, but it does not solve prompt-invariant long-chain execution.
Overview
This standalone experiment tests whether a Qwen-attached model can write a program into fixed register tokens and have that program executed by a fixed cyclic runtime. The bridge reads only register-token hidden states. It predicts an initial residue, primitive operation routes, and operation arguments. A deterministic modulo-97 runtime executes the predicted program.
The main intervention is training pressure on the runtime trajectory:
- supervised executable slots;
- supervised intermediate cyclic states;
- paired consistency between two prompt renderings of the same latent program.
Result
The main run solves the trained length range and partially lifts length-24 standard execution, but it does not solve prompt-invariant long-chain execution.
| Split | Executor exact | Program exact | Init | Op | Arg | Prefix | Pair both | Pair state consistency |
|---|---|---|---|---|---|---|---|---|
| Standard L4 | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | n/a | n/a |
| Standard L8 | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | n/a | n/a |
| Standard L12 | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | n/a | n/a |
| Standard L24 | 25.0% | 25.0% | 100.0% | 93.8% | 89.7% | 80.5% | n/a | n/a |
| Paraphrase L24 | 5.5% | 4.7% | 100.0% | 88.9% | 83.8% | 81.0% | n/a | n/a |
| Paired L24 | 11.7% | 10.2% | 100.0% | 89.7% | 85.7% | 79.1% | 1.6% | 1.6% |
Matched state-loss control without paired consistency:
| Split | Main | No-pair control |
|---|---|---|
| Standard L24 | 25.0% | 3.9% |
| Paraphrase L24 | 5.5% | 1.6% |
| Paired L24 | 11.7% | 1.6% |
Layout
experiments/qwen_register_structured_runtime/
src/ experiment and analysis scripts
reports/ experiment log and final writeup
analysis/ regenerated CSVs, summary, and figures
runs/ per-run metrics and training logs, without large checkpoints
checkpoint_manifest.csv
large_artifacts/qwen_register_structured_runtime/checkpoints/
saved adapters and bridge headsReproduction
Main run:
python experiments/qwen_register_structured_runtime/src/qwen_register_structured_runtime_experiment.py \
--output_dir experiments/qwen_register_structured_runtime/runs/main_structured_trace_state_consistency_s600 \
--checkpoint_dir large_artifacts/qwen_register_structured_runtime/checkpoints/main_structured_trace_state_consistency_s600 \
--variants structured_trace_state_consistency \
--register_style bare \
--curriculum_stages short:1:4:150,medium:1:8:150,train:1:12:150,long:8:24:150 \
--train_size 512 \
--answer_train_size 512 \
--eval_size 128 \
--eval_lengths 4,8,12,24 \
--train_batch_size 4 \
--eval_batch_size 8 \
--register_width 512 \
--register_layers 1 \
--register_heads 4 \
--head_width 512 \
--state_loss_weight 1.0 \
--pair_consistency_loss_weight 0.1 \
--pair_state_consistency_loss_weight 0.5 \
--lr 0.0002 \
--lora_r 8 \
--lora_alpha 16Regenerate analysis after runs:
python experiments/qwen_register_structured_runtime/src/analyze_qwen_register_structured_runtime.pyKey Files
reports/qwen_register_structured_runtime_experiment_log.mdreports/qwen_register_structured_runtime_paper.mdreports/qwen_register_structured_runtime_paper.htmlanalysis/summary.mdanalysis/all_final_metrics.csvcheckpoint_manifest.csv
Report
Rendered from reports/qwen_register_structured_runtime_paper.md
Abstract
This experiment tests whether a Qwen-attached model can write an executable program into fixed register tokens, then execute that program inside a fixed cyclic modulo runtime. Each prompt describes a modular-arithmetic chain. The prompt is followed by an appended register bank: one initial-value marker and one operation and argument marker per possible step. A trainable compiler reads only the hidden states at those marker positions. It predicts an initial residue, operation routes, and arguments. A deterministic runtime executes the predicted program modulo 97.
The main result is mixed. With QLoRA, full executable-slot supervision, full intermediate-state supervision, and paired consistency between prompt renderings, the model reaches 100.0% exact execution through length 12 on standard, paired, and nearly all paraphrase splits. At length 24, it reaches 25.0% standard exact execution, 5.5% paraphrase exact execution, and 11.7% paired exact execution. A matched state-supervised control without paired consistency also solves length 12, but remains near chance at length 24: 3.9% standard, 1.6% paraphrase, and 1.6% paired.
The conclusion is narrow: paired trajectory consistency helps the fixed register runtime start to generalize beyond the trained easy range, but it does not solve long-chain prompt-invariant execution.
Question
Can a Qwen-attached model configure a fixed structured runtime through invisible register tokens, rather than relying on answer-token prediction?
The interface is intentionally strict. The compiler sees only:
- hidden state at
<REG_INIT>; - hidden state at each
<REG_OP_XX>; - hidden state at each
<REG_ARG_XX>.
It does not receive hand-selected prompt spans. If the method works, the model has learned to route source-program information into fixed latent registers.
Task
Each example samples an initial value x modulo 97 and a chain of updates:
x by a
-x by a
*x by aThe target answer is the final value after all active updates. Evaluation uses lengths 4, 8, 12, and 24.
| Split | Meaning |
|---|---|
| Standard | Canonical prompt wording |
| Paraphrase | Alternative wording for the same operation semantics |
| Paired | Two renderings of each sampled program |
Model
The base model is Qwen/Qwen3-4B, loaded in 4-bit NF4 with LoRA adapters. The register compiler is a one-layer transformer over the fixed register bank:
- register width: 512 in the main run;
- attention heads: 4;
- max active steps: 24;
- outputs: init logits over 97 residues, operation logits over three primitives, and argument logits over 97 residues.
The runtime is fixed and cyclic. It applies predicted add, subtract, and multiply primitives modulo 97. The runtime is differentiable during training, so losses can be applied to the final answer distribution and every intermediate state distribution.
Training
The main run uses four losses:
| Loss | Purpose |
|---|---|
| Slot trace loss | Supervise init, operation, and argument registers |
| Final executor loss | Supervise the final runtime answer |
| State trajectory loss | Supervise every intermediate modulo state |
| Paired consistency loss | Align register and state distributions across two prompt renderings |
The main curriculum is:
| Stage | Lengths | Steps |
|---|---|---|
| Short | 1-4 | 150 |
| Medium | 1-8 | 150 |
| Train | 1-12 | 150 |
| Long | 8-24 | 150 |
The matched control uses the same setup but removes the paired consistency losses.
Results
Main Run
| Split | Executor exact | Program exact | Init | Op | Arg | Prefix | Pair both | Pair state consistency |
|---|---|---|---|---|---|---|---|---|
| Standard L4 | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | n/a | n/a |
| Standard L8 | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | n/a | n/a |
| Standard L12 | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | n/a | n/a |
| Standard L24 | 25.0% | 25.0% | 100.0% | 93.8% | 89.7% | 80.5% | n/a | n/a |
| Paraphrase L4 | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | n/a | n/a |
| Paraphrase L8 | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | n/a | n/a |
| Paraphrase L12 | 99.2% | 99.2% | 100.0% | 99.9% | 99.9% | 99.7% | n/a | n/a |
| Paraphrase L24 | 5.5% | 4.7% | 100.0% | 88.9% | 83.8% | 81.0% | n/a | n/a |
| Paired L4 | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% |
| Paired L8 | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% |
| Paired L12 | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% |
| Paired L24 | 11.7% | 10.2% | 100.0% | 89.7% | 85.7% | 79.1% | 1.6% | 1.6% |
The main run completely solves the trained length range. At length 24, the init register is exact and operation/argument accuracy remain high, but exact program execution is much lower. The model makes too many small slot errors for a 24-step chain.
Matched Control
| Split | Main | No-pair control |
|---|---|---|
| Standard L24 executor exact | 25.0% | 3.9% |
| Standard L24 program exact | 25.0% | 2.3% |
| Paraphrase L24 executor exact | 5.5% | 1.6% |
| Paired L24 executor exact | 11.7% | 1.6% |
| Paired L24 pair both-correct | 1.6% | 0.0% |
| Paired L24 state consistency | 1.6% | 0.0% |
Both runs solve length 12. The difference appears at length 24: paired consistency is load-bearing for the observed long-chain lift. It is not enough to make the long-chain latent program stable.
Training Dynamics
The main run developed in stages:
| Step | Stage | Standard L12 | Standard L24 | Paraphrase L24 | Paired L24 |
|---|---|---|---|---|---|
| 150 | Short | 0.8% | 0.0% | 1.6% | 0.8% |
| 300 | Medium | 71.1% | 1.6% | 0.0% | 2.0% |
| 450 | Train | 63.3% | 0.0% | 0.8% | 0.0% |
| 451 | Long start | 93.0% | 13.3% | 4.7% | 6.6% |
| 600 | Long end | 100.0% | 25.0% | 5.5% | 11.7% |
The long stage is essential. The model does not extrapolate to length 24 from short and medium training alone, even with state supervision.
Interpretation
The experiment supports three claims.
First, the fixed register-token interface is trainable. The compiler reads only the appended marker states and reaches perfect exact execution through length 12.
Second, state trajectory supervision alone is not enough for long-chain generalization. The no-pair control has full state loss and still stays near chance at length 24.
Third, paired consistency is useful but incomplete. It raises standard L24 from 3.9% to 25.0% and paired L24 from 1.6% to 11.7%, but paired both-correct and state-consistency remain at 1.6%. The model can become more accurate without becoming prompt-invariant.
The next technical target should be a repair or refinement mechanism over the compiled register program. The current compiler often gets most of the length-24 slots right, but one or two wrong slots are enough to break exact execution.
Limitations
- The task is synthetic modular arithmetic.
- The runtime is fixed and specialized.
- Training uses privileged intermediate state labels.
- The result is one main seed and one matched control seed.
- Length-24 paired consistency remains unsolved.
Artifacts
Small files:
experiments/qwen_register_structured_runtime/Large checkpoints:
large_artifacts/qwen_register_structured_runtime/checkpoints/Primary outputs:
analysis/summary.mdanalysis/final_metrics.csvanalysis/all_final_metrics.csvreports/qwen_register_structured_runtime_experiment_log.mdreports/qwen_register_structured_runtime_paper.mdcheckpoint_manifest.csv
Experiment log 8
Show the running log (8 entries)
Objective
Train and evaluate a Qwen-attached register compiler whose predicted slots are executed by a fixed cyclic modulo runtime. The compiler may read only appended register-token hidden states. The key test is whether full state-trajectory supervision plus paired consistency improves long-chain exact execution and prompt-invariant latent trajectories.
Success Criteria
- Keep this experiment standalone with its own source, reports, analysis, run metadata, and checkpoint manifest.
- Store large checkpoints under
large_artifacts/. - Run smoke, pilot, and main configurations instead of relying on a single run.
- Evaluate standard, paraphrase, and paired length generalization.
- Compare the structured state-consistency condition against at least one control that removes a load-bearing training signal.
Runs
Smoke: Frozen Structured State Consistency
smoke_frozen_structured_state_consistency
- Frozen
Qwen/Qwen3-4Bbackbone. - Four-step register bank.
- Two optimizer steps with tiny train/eval sets.
- Purpose: validate appended register construction, the structured cyclic runtime, full state loss, paired consistency loss, metric writing, and checkpoint writing.
- Result: completed end to end. Accuracy was not expected to move under this tiny setup.
Pilot: QLoRA Structured State Consistency
pilot_structured_state_consistency_s240
- QLoRA-adapted
Qwen/Qwen3-4B. - Full 24-step bare appended register bank.
- Trace loss, full state trajectory loss, final executor loss, and paired program/state consistency loss.
- Curriculum: 80 steps at lengths 1-4, 80 at lengths 1-8, and 80 at lengths 1-12.
- Result: strong through length 12 before any long-chain training. Final exact execution was 85.9% on standard L12, 78.1% on paraphrase L12, and 82.0% on paired L12. Paired state consistency was 95.3% at L12. Length 24 moved above chance but remained weak: 7.8% standard, 9.4% paraphrase, and 9.4% paired exact execution.
Interpretation: the trajectory and consistency losses do not prevent register learning. They produce prompt-stable trajectories through the trained length range. Long-chain exactness still needs an explicit long-stage curriculum.
Main: Structured State Consistency
main_structured_trace_state_consistency_s600
- QLoRA-adapted
Qwen/Qwen3-4B. - Full 24-step bare appended register bank.
- Register transformer width 512.
- Trace loss, final executor loss, full state trajectory loss, paired program-consistency loss, and paired state-consistency loss.
- Curriculum: 150 steps at lengths 1-4, 150 at 1-8, 150 at 1-12, and 150 at 8-24.
Final metrics:
| Split | Executor exact | Program exact | Init | Op | Arg | Prefix | Pair both | Pair state consistency |
|---|---|---|---|---|---|---|---|---|
| Standard L4 | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | n/a | n/a |
| Standard L8 | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | n/a | n/a |
| Standard L12 | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | n/a | n/a |
| Standard L24 | 25.0% | 25.0% | 100.0% | 93.8% | 89.7% | 80.5% | n/a | n/a |
| Paraphrase L24 | 5.5% | 4.7% | 100.0% | 88.9% | 83.8% | 81.0% | n/a | n/a |
| Paired L24 | 11.7% | 10.2% | 100.0% | 89.7% | 85.7% | 79.1% | 1.6% | 1.6% |
Interpretation: the main run completely solves the trained length range and partially improves length-24 standard execution. Long-chain exactness remains fragile because high individual slot accuracy still compounds across 24 steps. The paired L24 state-consistency metric remains low.
Control: State Trace Without Paired Consistency
control_trace_state_no_pair_s600
- Same backbone, register bank, model width, curriculum, trace loss, executor loss, and full state trajectory loss.
- Removed paired program/state consistency losses.
Final length-24 metrics:
| Split | Executor exact | Program exact | Pair both | Pair state consistency |
|---|---|---|---|---|
| Standard L24 | 3.9% | 2.3% | n/a | n/a |
| Paraphrase L24 | 1.6% | 0.0% | n/a | n/a |
| Paired L24 | 1.6% | 0.8% | 0.0% | 0.0% |
Interpretation: paired consistency was not needed for perfect L12 performance, but it was load-bearing for length-24 lift in this run. Without it, the model also solved L4/L8/L12 but stayed near chance at length 24.
Overall Interpretation
The experiment gives a narrow positive result. A fixed register-token interface plus structured cyclic state supervision can make Qwen write executable programs perfectly through length 12. Paired consistency materially improves length-24 standard execution under the tested curriculum.
The negative result is equally important. The method still does not produce a prompt-invariant length-24 latent program. L24 paired both-correct accuracy and paired state consistency remain at 1.6% in the main run. The next change should target long-chain repair or recurrent refinement of the register program, not only stronger per-slot supervision.
Figures 4
Data files 10
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
analysis/all_final_metrics.csv8.2 kBanalysis/final_metrics.csv2.7 kBruns/control_trace_state_no_pair_s600/metrics.csv2.3 kBruns/control_trace_state_no_pair_s600/results.json84 kBruns/main_structured_trace_state_consistency_s600/metrics.csv2.7 kBruns/main_structured_trace_state_consistency_s600/results.json85 kBruns/pilot_structured_state_consistency_s240/metrics.csv2.9 kBruns/pilot_structured_state_consistency_s240/results.json67 kBruns/smoke_frozen_structured_state_consistency/metrics.csv1.4 kBruns/smoke_frozen_structured_state_consistency/results.json17 kB
Reproduce
The run commands are documented inside the experiment folder (see the README).