Qwen Structural Latent Compiler Expansion

Finished2026-06-24imported · line YBenchmark Generalization Structured Execution and Compilers GitHub ↗

The one idea you need

Rather than writing steps out as text and picking the best-looking guess, the model fills a fixed form — a starting number plus a list of operations — that a calculator then runs. For longer problems, that filled form is copied into a bigger one with more blank rows.

The question

Can a model directly fill in a runnable calculation program instead of writing it out as text, and can that program be stretched to handle much longer calculations?

What we found

Yes, mostly. The model fills fixed slots with operations a calculator runs — no text, no trying many guesses and picking one. After the learned short form was copied into bigger ones, it stayed perfectly correct on 8- and 16-step chains and hit about 83% on the hardest 24-step ones, where reworded versions of a problem still computed identically.

Why it matters

For structured or runnable outputs, let a small model fill fixed operation slots directly rather than generate text and rerank guesses — it is cheaper, repeatable, and you add capacity by copying weights into more slots. Expect some slippage on the longest chains.

Shortest chains, 8 and 16 steps100% correctfinal answer right after the grown solver runs the fixed operation lists

Longest chains, 24 stepsabout 83% correctaccuracy on the hardest, longest problems after growing the slots

Reworded 24-step problems100% matcha differently phrased problem produces the same computation

Both of a reworded pair correct, 24 stepsabout 88%matched problem pairs where both answers came out right

On this page

Results at a glance
Overview
Report
Experiment log
Figures
Data files
Reproduce
Related

Results at a glance 3

How often the grown solver got the right answer, by problem length

How to read

Grouped bars show how often the final grown solver reached the right answer for 8-, 16-, and 24-step problems, with three bars each for the plain, reworded, and paired phrasings. Taller bars are better.

Takeaway → Bars stay at full height for 8- and 16-step problems and dip only for the longest, so stretching the solver cost almost nothing on what it already handled.

Data table

program length	standard	paraphrase	paired
length 8	100%	100%	100%
length 16	100%	100%	100%
length 24	82.8%	100%	93.8%

Numbers from experiments/qwen_structural_latent_compiler_expansion/reports/aggregate_metrics.csv (main run, stage3_max24 rows)

Technical framing

Executable accuracy after expansion to 24 slots, by program length — The final 24-slot compiler keeps perfect accuracy on length-8 and length-16 programs and reaches 82.8-100% on length-24 splits.

Accuracy as longer problems were added in stages

How to read

Lines track answer accuracy over training for the plain phrasing, one line per problem length; each line begins when that length is first introduced. Higher lines are better.

Takeaway → Each length's line holds near the top as longer ones are added — the 16-step line even climbs to perfect — so adding longer problems never dragged shorter ones down.

Data table

training step (stage ends: 300, 450, 750)	length 8	length 16	length 24
300	100%	—	—
450	100%	98.4%	—
750	100%	100%	82.8%

Numbers from experiments/qwen_structural_latent_compiler_expansion/reports/aggregate_metrics.csv (main run, all stages, standard splits)

Technical framing

Accuracy across expansion stages (standard split) — Expanding from 8 to 16 to 24 slots never degrades shorter lengths; length 16 starts at 98.4% and reaches 100%, length 24 ends at 82.8%.

Do reworded versions of a problem solve the same way?

How to read

Grouped bars show three checks — both rewordings correct, the same internal steps, and matching partial steps — for 8-, 16-, and 24-step problems. Taller is better.

Takeaway → Consistency holds at full height for 8- and 16-step problems and eases only to about seven-eighths for the longest, so rewordings mostly solve the same way.

Data table

program length	both paraphrases correct	latent state-trace consistency	state prefix recovered
length 8	100%	100%	100%
length 16	100%	100%	100%
length 24	87.5%	87.5%	99.2%

Numbers from experiments/qwen_structural_latent_compiler_expansion/reports/aggregate_metrics.csv (main run, stage3_max24 paired splits)

Technical framing

Paraphrase-pair consistency of the final compiler — Paraphrased prompts compile to the same latent execution trace 87.5% of the time even at length 24, with near-perfect state prefixes.

In the author’s words from the Report · “Interpretation”

This report is intentionally standalone. The key readout is whether expansion improves or preserves executable accuracy at longer lengths, and whether paraphrase-paired programs compile to the same latent execution trace.

Overview

This experiment tests whether an executable latent program compiler can be expanded structurally from short chains to longer chains without using beam search, candidate reranking, or text program generation.

The model receives a modular-arithmetic prompt plus fixed latent register positions. A Qwen backbone reads the prompt, a direct compiler head predicts an initial value and a sequence of typed operations/arguments, and a differentiable executor supervises the resulting latent program.

Large checkpoints are stored outside this directory under:

/workspace/large_artifacts/qwen_structural_latent_compiler_expansion/checkpoints

Report

Rendered from reports/structural_latent_compiler_expansion_report.md

Question

Can a Qwen-attached executable latent compiler be expanded from short modular programs to longer modular programs while preserving direct executable accuracy, without beam search, candidate reranking, or tokenized program output?

Method

A Qwen causal LM reads the arithmetic prompt and fixed latent register markers.
A structural compiler head predicts one initial value plus typed operation and argument slots.
A differentiable modular executor supervises final answer probability and intermediate state traces.
The compiler is expanded by copying learned short-slot parameters into longer slot structures, then continuing training.
The run reports argmax executable accuracy, exact program recovery, state prefix recovery, and paraphrase-pair consistency.

Runs

run	elapsed_sec	model	stage_max_steps	stage_steps	train_examples	gpu
main_qwen_structural_expansion_s750	1833	Qwen/Qwen3-4B	8,16,24	300,150,300	512	NVIDIA RTX 6000 Ada Generation
pilot_qwen_structural_expansion_s90	84.14	Qwen/Qwen3-4B	8,16,24	30,30,30	64	NVIDIA RTX 6000 Ada Generation
smoke_qwen_structural_expansion	6.944	Qwen/Qwen3-4B	8,16,24	1,1,1	4	NVIDIA RTX 6000 Ada Generation

Results

Final expanded 24-slot compiler, length-24 splits:

split	executor_accuracy	program_exact	state_prefix_fraction	executor_pair_both_correct	compiler_pair_state_consistency
standard_L24	82.8%	82.8%	97.9%
paraphrase_L24	100.0%	100.0%	100.0%
paired_L24	93.8%	93.8%	99.2%	87.5%	87.5%

Best single split executable accuracy was 100.0%; the length-24 table above is the main result.

run	stage	split	global_step	max_steps	executor_accuracy	program_exact	state_prefix_fraction	state_all_exact	executor_pair_both_correct	compiler_pair_state_consistency	length
main_qwen_structural_expansion_s750	stage1_max8	paired_L8	300	8	100.0%	100.0%	100.0%	1	100.0%	100.0%	8
main_qwen_structural_expansion_s750	stage1_max8	paraphrase_L8	300	8	98.4%	98.4%	98.4%	0.9844			8
main_qwen_structural_expansion_s750	stage1_max8	standard_L8	300	8	100.0%	100.0%	100.0%	1			8
main_qwen_structural_expansion_s750	stage2_max16	paired_L8	450	16	100.0%	100.0%	100.0%	1	100.0%	100.0%	8
main_qwen_structural_expansion_s750	stage2_max16	paraphrase_L8	450	16	100.0%	100.0%	100.0%	1			8
main_qwen_structural_expansion_s750	stage2_max16	standard_L8	450	16	100.0%	100.0%	100.0%	1			8
main_qwen_structural_expansion_s750	stage2_max16	paired_L16	450	16	100.0%	100.0%	100.0%	1	100.0%	100.0%	16
main_qwen_structural_expansion_s750	stage2_max16	paraphrase_L16	450	16	98.4%	98.4%	99.7%	0.9844			16
main_qwen_structural_expansion_s750	stage2_max16	standard_L16	450	16	98.4%	98.4%	99.7%	0.9844			16
main_qwen_structural_expansion_s750	stage3_max24	paired_L8	750	24	100.0%	100.0%	100.0%	1	100.0%	100.0%	8
main_qwen_structural_expansion_s750	stage3_max24	paraphrase_L8	750	24	100.0%	100.0%	100.0%	1			8
main_qwen_structural_expansion_s750	stage3_max24	standard_L8	750	24	100.0%	100.0%	100.0%	1			8
main_qwen_structural_expansion_s750	stage3_max24	paired_L16	750	24	100.0%	100.0%	100.0%	1	100.0%	100.0%	16
main_qwen_structural_expansion_s750	stage3_max24	paraphrase_L16	750	24	100.0%	100.0%	100.0%	1			16
main_qwen_structural_expansion_s750	stage3_max24	standard_L16	750	24	100.0%	100.0%	100.0%	1			16
main_qwen_structural_expansion_s750	stage3_max24	paired_L24	750	24	93.8%	93.8%	99.2%	0.9375	87.5%	87.5%	24
main_qwen_structural_expansion_s750	stage3_max24	paraphrase_L24	750	24	100.0%	100.0%	100.0%	1			24
main_qwen_structural_expansion_s750	stage3_max24	standard_L24	750	24	82.8%	82.8%	97.9%	0.8281			24
pilot_qwen_structural_expansion_s90	stage1_max8	paired_L8	30	8	0.0%	0.0%	0.0%	0	0.0%	37.5%	8
pilot_qwen_structural_expansion_s90	stage1_max8	paraphrase_L8	30	8	0.0%	0.0%	0.0%	0			8
pilot_qwen_structural_expansion_s90	stage1_max8	standard_L8	30	8	0.0%	0.0%	0.0%	0			8
pilot_qwen_structural_expansion_s90	stage2_max16	paired_L8	60	16	0.0%	0.0%	0.8%	0	0.0%	75.0%	8
pilot_qwen_structural_expansion_s90	stage2_max16	paraphrase_L8	60	16	6.2%	0.0%	0.8%	0			8
pilot_qwen_structural_expansion_s90	stage2_max16	standard_L8	60	16	0.0%	0.0%	0.0%	0			8
pilot_qwen_structural_expansion_s90	stage2_max16	paired_L16	60	16	12.5%	0.0%	0.0%	0	12.5%	62.5%	16
pilot_qwen_structural_expansion_s90	stage2_max16	paraphrase_L16	60	16	6.2%	0.0%	0.4%	0			16
pilot_qwen_structural_expansion_s90	stage2_max16	standard_L16	60	16	0.0%	0.0%	0.4%	0			16
pilot_qwen_structural_expansion_s90	stage3_max24	paired_L8	90	24	0.0%	0.0%	1.6%	0	0.0%	0.0%	8
pilot_qwen_structural_expansion_s90	stage3_max24	paraphrase_L8	90	24	0.0%	0.0%	0.0%	0			8
pilot_qwen_structural_expansion_s90	stage3_max24	standard_L8	90	24	0.0%	0.0%	0.0%	0			8
pilot_qwen_structural_expansion_s90	stage3_max24	paired_L16	90	24	6.2%	0.0%	0.0%	0	0.0%	0.0%	16
pilot_qwen_structural_expansion_s90	stage3_max24	paraphrase_L16	90	24	0.0%	0.0%	0.0%	0			16
pilot_qwen_structural_expansion_s90	stage3_max24	standard_L16	90	24	0.0%	0.0%	0.0%	0			16
pilot_qwen_structural_expansion_s90	stage3_max24	paired_L24	90	24	0.0%	0.0%	0.0%	0	0.0%	0.0%	24
pilot_qwen_structural_expansion_s90	stage3_max24	paraphrase_L24	90	24	0.0%	0.0%	0.3%	0			24
pilot_qwen_structural_expansion_s90	stage3_max24	standard_L24	90	24	0.0%	0.0%	0.0%	0			24
smoke_qwen_structural_expansion	stage1_max8	paired_L8	1	8	0.0%	0.0%	0.0%	0	0.0%	0.0%	8
smoke_qwen_structural_expansion	stage1_max8	paraphrase_L8	1	8	0.0%	0.0%	0.0%	0			8
smoke_qwen_structural_expansion	stage1_max8	standard_L8	1	8	0.0%	0.0%	0.0%	0			8
smoke_qwen_structural_expansion	stage2_max16	paired_L8	2	16	0.0%	0.0%	0.0%	0	0.0%	0.0%	8
smoke_qwen_structural_expansion	stage2_max16	paraphrase_L8	2	16	0.0%	0.0%	0.0%	0			8
smoke_qwen_structural_expansion	stage2_max16	standard_L8	2	16	0.0%	0.0%	0.0%	0			8
smoke_qwen_structural_expansion	stage2_max16	paired_L16	2	16	0.0%	0.0%	0.0%	0	0.0%	0.0%	16
smoke_qwen_structural_expansion	stage2_max16	paraphrase_L16	2	16	0.0%	0.0%	0.0%	0			16
smoke_qwen_structural_expansion	stage2_max16	standard_L16	2	16	0.0%	0.0%	0.0%	0			16
smoke_qwen_structural_expansion	stage3_max24	paired_L8	3	24	50.0%	0.0%	0.0%	0	0.0%	0.0%	8
smoke_qwen_structural_expansion	stage3_max24	paraphrase_L8	3	24	0.0%	0.0%	0.0%	0			8
smoke_qwen_structural_expansion	stage3_max24	standard_L8	3	24	0.0%	0.0%	0.0%	0			8
smoke_qwen_structural_expansion	stage3_max24	paired_L16	3	24	0.0%	0.0%	0.0%	0	0.0%	0.0%	16
smoke_qwen_structural_expansion	stage3_max24	paraphrase_L16	3	24	0.0%	0.0%	0.0%	0			16
smoke_qwen_structural_expansion	stage3_max24	standard_L16	3	24	0.0%	0.0%	0.0%	0			16
smoke_qwen_structural_expansion	stage3_max24	paired_L24	3	24	0.0%	0.0%	0.0%	0	0.0%	0.0%	24
smoke_qwen_structural_expansion	stage3_max24	paraphrase_L24	3	24	0.0%	0.0%	0.0%	0			24
smoke_qwen_structural_expansion	stage3_max24	standard_L24	3	24	0.0%	0.0%	0.0%	0			24

Figures

Interpretation

Artifacts

Run outputs: experiments/qwen_structural_latent_compiler_expansion/runs/
Reports and figures: experiments/qwen_structural_latent_compiler_expansion/reports/
Large checkpoints: large_artifacts/qwen_structural_latent_compiler_expansion/checkpoints/

Experiment log 5

Show the running log (5 entries, 2026-06-24)

2026-06-24

Created a fresh standalone experiment directory for structural latent compiler expansion.
Motivation: prior beam/search runs were demoted from candidate method to diagnostic because the deployable learned selector repeatedly failed to close the oracle gap. This run removes beam search from the method.
Planned run sequence:
- Smoke: verify the real Qwen QLoRA path, expandable compiler, metrics, checkpoint routing, and report generation.
- Pilot: short staged expansion run to check whether the objective moves held-out executable accuracy.
- Main: longer staged expansion run with selected metrics and standalone Markdown/HTML report.

Smoke Run

Run: smoke_qwen_structural_expansion
Configuration: Qwen/Qwen3-4B QLoRA, stages 8 -> 16 -> 24, one update per stage, tiny eval sets.
Outcome: completed successfully. This verified the real model load, latent register collation, compiler expansion, differentiable executor, CSV/JSON logging, and Markdown/HTML report generation.
Interpretation: accuracy is not meaningful at this size; the smoke was only an integration gate.

Pilot Run

Run: pilot_qwen_structural_expansion_s90
Configuration: Qwen/Qwen3-4B QLoRA, stages 8 -> 16 -> 24, 30 updates per stage, weak default trace/state weights.
Outcome: completed successfully but did not learn executable programs. Final exact program recovery was 0% on all splits and final length-24 executable accuracy remained at 0% on standard/paraphrase/paired splits.
Interpretation: the structural path itself was not falsified; the objective was underweighted and undertrained compared with prior single-compiler recipes. The main run was adjusted to use heavier init/argument trace weights, full state loss, larger compiler width, batch 8, and a longer short-stage bootstrap.

Main Run

Run: main_qwen_structural_expansion_s750
Configuration: Qwen/Qwen3-4B QLoRA, stage expansion 8 -> 16 -> 24, steps 300/150/300, train lengths 1..8 then 1..16 then 8..24.
Outcome:
- After stage 1: length-8 executable/program exact accuracy was 100.0% standard, 98.4% paraphrase, 100.0% paired.
- After stage 2: length-16 executable/program exact accuracy was 98.4% standard, 98.4% paraphrase, 100.0% paired, while length-8 retention was 100.0%.
- After stage 3: length-8 and length-16 retention were 100.0%; length-24 executable/program exact accuracy was 82.8% standard, 100.0% paraphrase, 93.8% paired.
- Final length-24 state-prefix recovery was 97.9% standard, 100.0% paraphrase, 99.2% paired.
Interpretation: this is a clean positive for direct structural expansion of an executable latent compiler without beam search, candidate reranking, or tokenized program output. The remaining weakness is the standard length-24 split, where exact recovery is materially below the paraphrase and paired splits despite very high prefix recovery.