Qwen Structured Bridge Experiment
The one idea you need
Picture a frozen reader that understands the words but cannot do the math. A small trainable translator turns the reader's internal impressions into calculator instructions, and a separate pocket calculator actually crunches them. The reader stays frozen; only the translator learns.
The question
Can a frozen small model's read of a word problem be turned into a runnable step-by-step program that beats simply asking it for the final answer?
What we found
Yes, but only when you show it the individual steps during training. A tiny translator turning the frozen model's read into calculator instructions solved chains far longer than it trained on: 96% correct at twelve steps and about 88% at twenty-four, versus roughly 1 in 100 when the same model just names the final answer. Rewarding correct answers alone never found the trick.
Why it matters
For multi-step symbolic tasks, don't make a frozen small model blurt out the final answer. Train a light add-on to emit structured operations for a separate calculator. You gain big length generalization cheaply, but you must supply step-level supervision.
On this page
Results at a glance 3
How to read
Grouped bars compare three approaches at chains of 4, 8, and 12 operations (left to right); bar height is the share of answers fully correct, taller is better. Only the step-compiling bars rise near the top; the two answer-guessing versions sit on the floor.
Takeaway → The step-trained translator towers near 100% across every chain length while both the direct-answer and reward-only versions flatline at roughly one-in-a-hundred guessing level.
Data table
| held-out program length | direct answer head | compiler (trace-supervised) | compiler (answer-only loss) |
|---|---|---|---|
| 4 steps | 0% | 100% | 1.6% |
| 8 steps | 1.2% | 99.2% | 0% |
| 12 steps | 0.4% | 95.7% | 0.8% |
Numbers from experiments/qwen_structured_bridge/analysis/final_metrics.csv (main_qwen35_numeric_spans)
Technical framing
Frozen Qwen features drive a latent program executor only with trace supervision — Trace-supervised compilation hits 96-100% while a direct classifier and an answer-only compiler on the same frozen features stay at 97-way chance.
How to read
Lines track accuracy as chains grow from 4 to 24 operations, well past the up-to-four steps seen in training; higher and flatter is better. Two lines for share fully correct sit on top of each other; a third tracks the model's confidence in the right answer. All drift down only slightly toward the right.
Takeaway → Accuracy slopes down gradually to about 88% at 24 operations, six times the training depth, showing graceful decline rather than a collapse.
Data table
| program length (steps) | strict executor accuracy | program exact match | soft target mass |
|---|---|---|---|
| 4 | 100% | 100% | 99.9% |
| 12 | 93.8% | 93.8% | 93.6% |
| 16 | 92.2% | 92.2% | 91.5% |
| 24 | 87.5% | 87.5% | 82.7% |
Numbers from experiments/qwen_structured_bridge/analysis/final_metrics.csv (scale_qwen35_length24_trace)
Technical framing
Length generalization: trained on 1-4 steps, evaluated out to 24 — The compiled executor degrades gracefully far beyond training length, keeping 87.5% strict accuracy at 24 steps (6x the training depth).
How to read
Bars show how often each piece of the compiled program is right at 24 operations: the starting number, the operation, its number, and the whole chain end to end; taller is better. The first three are near-perfect, the last is lower.
Takeaway → Numbers are read essentially perfectly, so the roughly one-in-eight failed chains trace to the occasional wrong operation, whose errors pile up over many steps.
Data table
| compiled program component | per-component accuracy at length 24 |
|---|---|
| initial value | 100% |
| operation | 99.5% |
| argument | 100% |
| whole program exact | 87.5% |
Numbers from experiments/qwen_structured_bridge/analysis/final_metrics.csv (scale_qwen35_length24_trace, len24)
Technical framing
Where 24-step errors come from: rare operation flips, not numeric parsing — Initial values and arguments are parsed perfectly at length 24; the 12.5% program failures trace back to occasional operation misreads.
In the author’s words from the Report · “Abstract”
This experiment tests a frozen Qwen3.5-4B encoder attached to a trainable structured latent executor. The prompt describes a modular arithmetic program. The bridge reads selected Qwen hidden states, compiles them into an initial value and per-step program symbols, and an invisible executor runs the program to produce the answer. With trace supervision, the bridge learns a reliable latent program interface. Training on 1-4 step programs and evaluating on held-out 97-way answers, the compiled executor reaches 100.0% accuracy at length 4, 99.2% at length 8, and 95.7% at length 12. A length-24 scale check reaches 87.5%. … Read the full result →
Overview
This experiment tests a frozen Qwen encoder attached to a structured latent executor. The trainable bridge compiles text into modular program symbols, then an invisible executor runs the compiled program to produce the answer.
Contents
src/qwen_structured_bridge_experiment.py: task generator, Qwen feature extraction, bridge training, executor evaluation, and checkpointing.src/analyze_qwen_structured_bridge.py: analysis tables and figures.reports/qwen_structured_bridge_experiment_log.md: chronological experiment log.reports/qwen_structured_bridge_paper.md: standalone written report.reports/qwen_structured_bridge_paper.html: standalone HTML report.runs/: lightweight JSON and CSV run outputs.analysis/: generated summaries and figures.checkpoint_manifest.csv: saved checkpoint paths and sizes.
Large Files
Trainable bridge checkpoints are stored outside this directory under:
../../large_artifacts/qwen_structured_bridge/checkpoints/Download this experiment directory for the research bundle. Download the large artifact directory only when saved model weights are needed.
Report
Rendered from reports/qwen_structured_bridge_paper.md
Abstract
This experiment tests a frozen Qwen3.5-4B encoder attached to a trainable structured latent executor. The prompt describes a modular arithmetic program. The bridge reads selected Qwen hidden states, compiles them into an initial value and per-step program symbols, and an invisible executor runs the program to produce the answer.
With trace supervision, the bridge learns a reliable latent program interface. Training on 1-4 step programs and evaluating on held-out 97-way answers, the compiled executor reaches 100.0% accuracy at length 4, 99.2% at length 8, and 95.7% at length 12. A length-24 scale check reaches 87.5%. A direct answer classifier trained on the same frozen Qwen features stays at chance, and an answer-only latent compiler also stays at chance.
Task
Each prompt specifies a value x modulo 97 and a sequence of operations:
Initial x = 42.
Step: add 17.
Step: multiply by 3.
Step: subtract 5.The target is the final value of x. The executor supports three operations:
ADD a: setx = x + a (mod 97).SUB a: setx = x - a (mod 97).MUL a: setx = x * a (mod 97).
Training examples contain 1-4 steps. Evaluation uses longer held-out step counts.
Model
Qwen3.5-4B is loaded in 4-bit and kept frozen. The bridge reads hidden states at standardized prompt spans:
- the numeric token span for the initial value,
- the operation line prefix for each step,
- the numeric token span for each operation argument,
- the answer line for the direct-answer control.
The structured compiler predicts:
- initial value logits over 97 residues,
- operation logits over
ADD,SUB, andMUL, - argument logits over 97 residues.
A differentiable executor composes the predicted distributions during training. For strict evaluation, the compiled symbols are argmaxed and executed exactly.
Controls
The main controls are:
direct: an answer classifier trained on the same frozen Qwen answer-line feature.compiler_trace: the structured compiler trained with symbol trace supervision plus executor answer loss.compiler_answer_only: the structured compiler trained only from final answer loss through the soft executor.
These controls separate three questions: whether the frozen features support direct answer prediction, whether they support executable compilation when the latent interface is supervised, and whether final-answer reward alone discovers that interface.
Main Results
The main run uses 1024 training examples with lengths 1-4 and evaluates on lengths 4, 8, and 12.
| Variant | L=4 accuracy | L=8 accuracy | L=12 accuracy | L=12 target mass | L=12 init | L=12 op | L=12 arg | L=12 program exact |
|---|---|---|---|---|---|---|---|---|
direct | 0.0% | 1.2% | 0.4% | n/a | n/a | n/a | n/a | n/a |
compiler_trace | 100.0% | 99.2% | 95.7% | 94.1% | 100.0% | 99.6% | 100.0% | 95.7% |
compiler_answer_only | 1.6% | 0.0% | 0.8% | 1.0% | 1.2% | 33.5% | 3.1% | 0.0% |
Length Scale Check
The scale check trains the trace-supervised compiler on lengths 1-4 and evaluates lengths 4, 12, 16, and 24.
| Length | Executor accuracy | Target mass | Init acc | Op acc | Arg acc | Program exact |
|---|---|---|---|---|---|---|
| 4 | 100.0% | 99.9% | 100.0% | 100.0% | 100.0% | 100.0% |
| 12 | 93.8% | 93.6% | 100.0% | 99.5% | 100.0% | 93.8% |
| 16 | 92.2% | 91.5% | 100.0% | 99.6% | 100.0% | 92.2% |
| 24 | 87.5% | 82.7% | 100.0% | 99.5% | 100.0% | 87.5% |
Interpretation
The result supports a specific claim: frozen Qwen hidden states can drive a small structured bridge that configures and runs an invisible latent executor. The executor is not merely an answer head. It compiles text into program symbols, and strict accuracy tracks whether the whole compiled program is correct.
The direct control is important. It uses the same frozen Qwen backbone and the same task distribution, but it stays at 97-way chance. The structured bridge wins because the output space is decomposed into reusable program symbols that the executor can compose.
The answer-only control is also important. It does not learn the latent program interface under this budget. The trace-supervised condition shows that the interface is usable; the answer-only condition shows that discovering it from sparse final reward remains a separate training problem.
At long lengths, the remaining error is not numeric parsing. Initial value and argument accuracy are 100.0% at length 24. The error comes from rare operation misclassification, which compounds over more steps.
Limitations
The task is synthetic modular arithmetic with standardized text templates. The executor operation set is fixed in advance. The bridge receives direct symbol trace supervision in the successful condition. Qwen is frozen, so this is an attachment experiment rather than full model posttraining.
The result therefore does not show broad intelligence improvement. It shows a concrete mechanism by which a 4B-class model can configure a latent structured runtime and gain large length-generalization benefits on a serial symbolic task.
Artifact Layout
Lightweight code, metrics, figures, and reports live in:
experiments/qwen_structured_bridge/Saved bridge checkpoints live separately in:
large_artifacts/qwen_structured_bridge/checkpoints/The checkpoint manifest is:
experiments/qwen_structured_bridge/checkpoint_manifest.csvExperiment log 10
Show the running log (10 entries, 2026-06-21)
Objective
Test whether a frozen Qwen encoder can be attached to a trainable structured latent executor. The bridge reads hidden states from the prompt, predicts a modular initial value and per-step program symbols, and the executor computes the answer without emitting intermediate text.
Primary Questions
- Can a small bridge compile Qwen hidden states into executable modular program symbols?
- Does structured execution generalize to longer operation chains better than a direct answer classifier trained on the same frozen Qwen features?
- Is trace supervision necessary, or can answer-only supervision discover the latent program interface?
- Which failure mode dominates: initial-value parsing, operation parsing, argument parsing, or accumulated execution error?
Metrics
direct_accuracy: direct answer classifier accuracy from Qwen features.executor_accuracy: accuracy after argmax compilation and exact latent execution.executor_target_mass: soft executor probability assigned to the target answer.init_accuracy: compiled initial value accuracy.op_accuracy: per-step operation accuracy.arg_accuracy: per-step argument accuracy.program_exact: fraction of examples with all compiled symbols correct.
Artifact Layout
- Code and lightweight outputs:
experiments/qwen_structured_bridge/ - Checkpoints:
large_artifacts/qwen_structured_bridge/checkpoints/ - Run outputs:
experiments/qwen_structured_bridge/runs/<variant>/ - Analysis outputs:
experiments/qwen_structured_bridge/analysis/
2026-06-21 Setup
Created the standalone experiment directory:
experiments/qwen_structured_bridge/src/experiments/qwen_structured_bridge/reports/experiments/qwen_structured_bridge/runs/experiments/qwen_structured_bridge/analysis/figures/large_artifacts/qwen_structured_bridge/checkpoints/
Installed peft so LoRA can be added as a later condition if frozen-feature compilation bottlenecks on parsing. The first implementation keeps Qwen frozen to isolate whether the structured bridge works before training Qwen weights.
Next action: implement the Qwen-to-executor bridge harness and run a tiny smoke test.
2026-06-21 Harness Smoke
Implemented the bridge harness:
- Text modular-program generator with line-boundary token positions.
- Frozen-model hidden-state extraction for init, step, and answer lines.
- Direct answer classifier control.
- Program compiler heads for initial value, operation, and argument symbols.
- Differentiable soft modular executor and argmax exact executor.
- Variants:
direct,compiler_trace, andcompiler_answer_only. - External checkpoint writing.
- Analysis summary and figures.
Verification:
- Source compilation passed.
peftimport passed, version0.19.1.- Tiny random Llama smoke completed with all three variants.
- Analysis generation completed.
- Checkpoints were written under
large_artifacts/qwen_structured_bridge/checkpoints/smoke_tiny/.
Smoke interpretation:
- The smoke validates data flow, hidden-state extraction, training, checkpointing, and analysis.
- The tiny random model has no useful parsing signal, so its low accuracy is not an experimental result.
Next action: run a small Qwen pilot to test whether frozen Qwen hidden states support executable program compilation.
2026-06-21 Qwen Pilot
Ran pilot_qwen35_frozen_bridge with frozen Qwen3.5-4B features, training lengths 1-3, and evaluation lengths 3, 6, and 8.
Initial line-boundary feature result:
| Variant | L=3 direct | L=3 executor | L=6 executor | L=8 executor | Init acc | Op acc | Arg acc |
|---|---|---|---|---|---|---|---|
direct | 0.0% | n/a | n/a | n/a | n/a | n/a | n/a |
compiler_trace | n/a | 2.1% | 0.0% | 0.0% | 0.0-4.2% | 95.6-99.7% | 8.7-14.6% |
compiler_answer_only | n/a | 2.1% | 2.1% | 0.0% | 0.0-4.2% | 32.3-33.3% | 0.0-0.5% |
Interpretation:
- Qwen line-end features made operation words easy to classify.
- Numeric initial values and numeric arguments were not recoverable from that feature choice under the small pilot budget.
- The executor path itself was not the failure; it was being configured with wrong numeric symbols.
Patched the harness to read hidden states at numeric token spans for the initial value and step arguments, while keeping operation prediction on the operation-line prefix.
Ran pilot_qwen35_numeric_spans with the patched feature extractor.
Numeric-span result:
| Variant | L=3 direct | L=3 executor | L=6 executor | L=8 executor | Init acc | Op acc | Arg acc | Program exact |
|---|---|---|---|---|---|---|---|---|
direct | 0.0% | n/a | n/a | n/a | n/a | n/a | n/a | n/a |
compiler_trace | n/a | 71.9% | 76.6% | 71.9% | 71.9-85.9% | 98.8-100.0% | 99.5-100.0% | 71.9-76.6% |
compiler_answer_only | n/a | 0.0% | 4.7% | 0.0% | 0.0-3.1% | 33.4-40.1% | 0.0-0.2% | 0.0% |
Pilot interpretation:
- Frozen Qwen hidden states can support executable program compilation when the bridge reads the numeric-token features directly.
- Trace supervision is doing the important work. Answer-only supervision did not discover the latent program interface.
- The main remaining bottleneck is initial-value classification over 97 residues. Operation and argument parsing are already near exact.
Main decision:
- Run a larger Qwen numeric-span bridge with more examples per residue.
- Train on lengths 1-4 and evaluate lengths 4, 8, and 12.
- Keep
direct,compiler_trace, andcompiler_answer_onlyto preserve the direct-answer and answer-only controls.
2026-06-21 Main Qwen Run
Ran main_qwen35_numeric_spans with frozen Qwen3.5-4B features, 1024 training examples, training lengths 1-4, and evaluation lengths 4, 8, and 12.
| Variant | L=4 accuracy | L=8 accuracy | L=12 accuracy | L=12 target mass | L=12 init | L=12 op | L=12 arg | L=12 program exact |
|---|---|---|---|---|---|---|---|---|
direct | 0.0% | 1.2% | 0.4% | n/a | n/a | n/a | n/a | n/a |
compiler_trace | 100.0% | 99.2% | 95.7% | 94.1% | 100.0% | 99.6% | 100.0% | 95.7% |
compiler_answer_only | 1.6% | 0.0% | 0.8% | 1.0% | 1.2% | 33.5% | 3.1% | 0.0% |
Main interpretation:
- The trace-supervised compiler/executor is the first strong Qwen-attached result in this line: the frozen model supplies hidden features that a small bridge can compile into an executable latent program.
- The direct answer head remains at 97-way chance from the same frozen Qwen features.
- Answer-only latent compilation also remains at chance. Final-answer reward alone did not discover the discrete program interface under this setup.
- At length 12, the residual error is almost entirely rare operation misclassification; initial value and numeric argument parsing are exact.
2026-06-21 Length Scale Check
Ran scale_qwen35_length24_trace with the trace-supervised compiler only, training lengths 1-4, and evaluation lengths 4, 12, 16, and 24.
| L | Executor accuracy | Target mass | Init acc | Op acc | Arg acc | Program exact |
|---|---|---|---|---|---|---|
| 4 | 100.0% | 99.9% | 100.0% | 100.0% | 100.0% | 100.0% |
| 12 | 93.8% | 93.6% | 100.0% | 99.5% | 100.0% | 93.8% |
| 16 | 92.2% | 91.5% | 100.0% | 99.6% | 100.0% | 92.2% |
| 24 | 87.5% | 82.7% | 100.0% | 99.5% | 100.0% | 87.5% |
Scale interpretation:
- The bridge generalizes well beyond the training length range, reaching 87.5% exact execution at 24 steps after training only on 1-4 step programs.
- The remaining error scales like accumulated rare operation mistakes.
- Numeric parsing is not the bottleneck in the scaled run: initial value and argument accuracy are 100.0% at length 24.
Next action: write the standalone report, checkpoint manifest, and final audit.
2026-06-21 Final Audit
Final artifacts created:
reports/qwen_structured_bridge_paper.mdreports/qwen_structured_bridge_paper.htmlcheckpoint_manifest.csv
Verification:
- Source compilation passed:
python -m py_compile src/qwen_structured_bridge_experiment.py src/analyze_qwen_structured_bridge.py - Checkpoint manifest validation passed for 16 saved checkpoints.
- Markdown and HTML report image references resolve.
- No
.pt,.pth, or.ckptfiles are stored inside the lightweight experiment directory. - Standalone wording scan passed on the report files.
- Removed the compile cache after verification.
Artifact sizes:
experiments/qwen_structured_bridge/: 452Klarge_artifacts/qwen_structured_bridge/: 98M
Conclusion:
Frozen Qwen3.5-4B hidden states can drive a small trace-supervised structured bridge that compiles text into an executable latent modular program. The main run reaches 95.7% exact execution at length 12 after training on lengths 1-4. The length-24 scale check reaches 87.5%. Direct answer classification and answer-only latent compilation remain at chance.
Figures 3
Data files 7
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
analysis/final_metrics.csv5.1 kBruns/main_qwen35_numeric_spans/results.json24 kBruns/pilot_qwen35_frozen_bridge/results.json21 kBruns/pilot_qwen35_numeric_spans/results.json20 kBruns/scale_qwen35_length24_trace/results.json12 kBruns/smoke_tiny/results.json12 kBruns/smoke_tiny_numeric_spans/results.json12 kB
Reproduce
This entry is source code or analysis only — there is no separate run to reproduce.