End-to-End Structured Slot Executor Experiment

Finished2026-06-21imported · line YStructured Execution and Compilers GitHub ↗

The one idea you need

Picture a detective's suspect board. You know each suspect's partner sits a fixed number of steps ahead, but not who the first suspect is, so every candidate stays pinned up. As clues arrive, you shift the whole board at once.

The question

Can one small model both build the full set of possible answers from a single clue, then correctly update that whole set as instructions run step by step?

What we found

Yes, but only when both halves carry built-in structure. On the numbers 0 to 30, the full model puts 98% of its confidence on the exactly correct final set of possibilities, nearly matching a version handed the answer, and still 95% on the wider range 0 to 96. Give it a generic starting set and even a flawless updater drops to 29%; a generic updater collapses it to 5%.

Why it matters

For small models that track a set of possible states across many steps, do not lean on a generic memory plus a clever update rule. Bake the domain's structure into both the initial setup and each transition.

Confidence on the exact right answer set98%full structured model, numbers 0-30, longest test programs

Holds up on a bigger number space95%same model on numbers 0-96

Smart updater but generic setup98% → 29%collapses when the starting set lacks structure

Structured setup but generic updater98% → 5%collapses when the step updates lack structure

On this page

Results at a glance
Overview
Report
Experiment log
Figures
Data files
Reproduce
Related

Results at a glance 7

Which parts must be purpose-built to solve the longest puzzles

How to read

Each bar is one build of the model, pairing a purpose-built or generic starting-set part with a purpose-built or generic update part; bar height is confidence on the exactly right final answer set on the longest puzzles. Taller is better.

Takeaway → The four builds using purpose-built parts cluster near the top, around 97 to 100%, while the two using any generic part fall to a quarter or near zero.

Data table

variant (initializer + transition)	strict belief mass at L=24
oracle init + exact transition	100%
oracle init + learned router	99.9%
sinkhorn init + exact transition	96.8%
sinkhorn init + learned router	98.1%
generic init + learned router	28.7%
sinkhorn init + MLP transition	5%

Numbers from report table (reports/end_to_end_structured_slot_paper.md); matches analysis/summary.md

Technical framing

Which components must be structured? Strict belief mass at held-out length 24 (mod 31) — The fully learned structured executor (98.1%) nearly matches the oracle; swapping either part for an unstructured one collapses performance.

Staying accurate as puzzles grow far longer than in training

How to read

The horizontal axis is puzzle length, stretching to three times the longest training puzzle; the vertical axis is confidence on the right answer set. Each line is one build; higher and flatter is better.

Takeaway → The purpose-built lines stay near the top across every length, while the two builds with a generic part slope steeply downward as puzzles get longer.

Data table

held-out program length	oracle init + exact transition	sinkhorn init + learned router	sinkhorn init + exact transition	generic init + learned router	sinkhorn init + MLP transition
4	100%	99.6%	99.4%	52.8%	19%
8	100%	99.5%	99.1%	47.5%	13.9%
12	100%	99.3%	98.9%	42.8%	10%
16	100%	99.1%	98.5%	38.5%	7.8%
24	100%	98.1%	96.8%	28.7%	5%

Numbers from analysis/summary.md table (decoder_belief_target_mass, main modulus 31)

Technical framing

Length generalization: strict belief mass vs held-out program length (mod 31, trained on lengths 1-8) — The learned structured executor stays near oracle out to 3x training length; unstructured init or transition degrades steeply with length.

Holding up on a harder version with far more possibilities

How to read

Same axes on a harder version with roughly three times as many possible answers; two lines compare a version handed the answer against the fully learned build. Higher and flatter is better.

Takeaway → The learned build tracks just below the perfect reference, still keeping 95% of its confidence on the right answer set at the longest puzzle length.

Data table

held-out program length	oracle init + exact transition	sinkhorn init + learned router
4	100%	99%
8	100%	98.7%
12	100%	97.6%
16	100%	96.6%
24	100%	94.9%

Numbers from analysis/summary.md table (decoder_belief_target_mass, scale modulus 97)

Technical framing

Scale check at modulus 97: learned executor vs oracle over held-out length — At modulus 97 the learned executor still reaches 94.9% at length 24; the small gap comes from slow belief-mass diffusion, not routing errors.

How fast each build learns during training

How to read

The horizontal axis is training progress; the vertical axis is the share of answers correct so far. Each line is one build; rising faster and higher is better.

Takeaway → The purpose-built builds climb past 97% early in training, while the two builds with a generic part flatten out around 60 to 80%.

Data table

training step	Sinkhorn init + primitive router	Sinkhorn init + exact transition	Oracle init + primitive router	Generic MLP init + primitive router	Sinkhorn init + MLP transition
1	34.3%	48.5%	36.1%	34.4%	34%
200	—	—	100%	—	—
280	—	—	—	47.7%	—
300	72.5%	73.7%	—	—	—
320	—	—	—	—	47.7%
400	—	—	100%	—	—
560	—	—	—	50.9%	—
600	97.5%	97.7%	100%	—	—
640	—	—	—	—	43.6%
800	—	—	100%	—	—
840	—	—	—	71.4%	—
900	98.8%	98.8%	—	—	—
960	—	—	—	—	47.6%
1120	—	—	—	74.8%	—
1200	99.4%	99.4%	—	—	—
1280	—	—	—	—	56.4%
1400	—	—	—	79.8%	—
1500	99.5%	99.5%	—	—	—
1600	—	—	—	—	60.8%
1800	99.7%	99.7%	—	—	—
2100	99.8%	—	—	—	—
2400	99.8%	—	—	—	—

Numbers from analysis/train_log.csv

Technical framing

Training curves at modulus 31: answer mass over optimizer steps — Sinkhorn-init routers climb past 97% by step ~600, while generic-MLP init and MLP transitions plateau near 60-80%.

Where the failing builds lose the answer

How to read

For each build, two bars: how correct the starting set is before any instructions, and how correct the answer is after the whole puzzle runs. Higher is better.

Takeaway → The generic-setup build loses about 40% before it even starts, while the generic-update build begins decent then erodes from 59% down to 20%.

Data table

condition	Initial belief mass (step 0)	Final answer mass (after 24 steps)
Oracle + exact	100%	100%
Oracle + router	100%	99.9%
Sinkhorn + exact	99.6%	97.7%
Sinkhorn + router	99.7%	98.7%
Sinkhorn + MLP transition	58.9%	19.9%
Generic MLP init + router	58.6%	43%

Numbers from analysis/summary.md table (First K >= L Summary, main modulus 31, length 24)

Technical framing

Where learned variants fail at modulus 31: initial belief vs final answer at length 24 — Generic-MLP init already loses ~41% mass at step 0, while the MLP transition erodes a good Sinkhorn init from 58.9% to 19.9%.

The same ranking repeats on a smaller version

How to read

Horizontal axis is puzzle length; vertical axis is confidence on the right answer, on a smaller version of the task with fewer possibilities. Each line is one build; higher and flatter is better.

Takeaway → The same ordering holds: purpose-built builds stay near or above 99% while the generic-part lines fall off as puzzles get longer.

Data table

program length	Oracle + exact	Sinkhorn + router	Sinkhorn + exact	Sinkhorn + MLP transition	Generic MLP init + router
3	100%	99.7%	99.4%	93.4%	82.4%
6	100%	99.6%	99%	88.6%	76.1%
9	100%	99.4%	98.6%	83.5%	72.9%
12	100%	99.2%	98.2%	78.4%	69.5%

Numbers from analysis/summary.md table (First K >= L Summary, pilot modulus 11)

Technical framing

Replication at modulus 11: answer mass vs held-out program length — The mod-31 ordering replicates at modulus 11: structured init plus router stays above 99% while MLP components decay with length.

A faster training speed rescues one build

How to read

Two builds, each shown with two bars for a slower versus faster training speed; bar height is the share of the answer correct on a small warm-up version. Higher is better.

Takeaway → Raising the training speed lifts the update-step build from 87% to 99%, a far bigger jump than the alternative mixer build gets.

Data table

condition	Base learning rate	Learning rate 0.1
Sinkhorn + primitive router	87.1%	99.4%
Sinkhorn + cyclic mixer	86.5%	91.5%

Numbers from analysis/summary.md table (First K >= L Summary, smoke modulus 7, length 3)

Technical framing

Learning-rate ablation at modulus 7: answer mass at length 3 — Raising the learning rate to 0.1 lifts the smoke-phase router from 87.1% to 99.4%; the cyclic mixer improves less.

In the author’s words from the Report · “Abstract”

On modulus 31, it reaches 98.1% strict belief mass at held-out length 24. On a modulus-97 scale check, the same design reaches 94.9% strict belief mass at held-out length 24. The ablations fail in different ways: a generic initializer with a perfect learned router reaches 28.7% at modulus 31, while a structured initializer with an unstructured MLP transition reaches 5.0%.

Overview

This experiment tests whether a structured slot executor can learn both initial support formation and recurrent modular belief-state transitions in one model.

src/end_to_end_structured_slot_experiment.py: task generator, structured initializers, learned transition routers, checkpointing, and evaluation harness.
src/analyze_end_to_end_structured_slot.py: analysis and figure generation.
reports/end_to_end_structured_slot_experiment_log.md: chronological experiment log.
reports/end_to_end_structured_slot_paper.md: standalone written report.
reports/end_to_end_structured_slot_paper.html: standalone HTML report.
runs/: JSON and CSV run outputs.
analysis/: generated summaries and figures.
checkpoint_manifest.csv: checkpoint paths and sizes.

Checkpoints are written outside the experiment directory under:

../../large_artifacts/end_to_end_structured_slot_executor/checkpoints/

Download this experiment directory for the normal research bundle. Download ../../large_artifacts/end_to_end_structured_slot_executor/ only when saved model weights are needed.

Report

Rendered from reports/end_to_end_structured_slot_paper.md

Abstract

This experiment tests whether a single learned model can form a modular belief state and execute recurrent symbolic updates without being given either the initial state or the transition rule. The task family uses hidden residues A and B with B = A + d (mod p). A program applies arithmetic updates and observations, and the model must recover the final support over (A, B).

The best learned executor combines a Sinkhorn-normalized cyclic initializer with a learned primitive router over modular update candidates. On modulus 31, it reaches 98.1% strict belief mass at held-out length 24. On a modulus-97 scale check, the same design reaches 94.9% strict belief mass at held-out length 24. The ablations fail in different ways: a generic initializer with a perfect learned router reaches 28.7% at modulus 31, while a structured initializer with an unstructured MLP transition reaches 5.0%.

Problem

Each example begins with a relation:

B = A + d (mod p)

The value of A is hidden, so the initial support contains one valid (A, B) pair for each residue. The program then applies operations such as modular increments, decrements, swaps, and observations. The target is not one scalar answer; it is the final belief support induced by the whole program.

The strict metric is the probability mass assigned to the exact final support after evaluating at the first recurrent step budget K that is at least the program length L.

Model

The executor represents belief states with weighted slots. Each slot carries a distribution over A, a distribution over B, and a slot weight. Decoding forms a dense belief over (A, B) from the slot mixture.

The primary learned executor has two structured components:

A Sinkhorn-normalized cyclic initializer that maps the relation offset d into one slot per residue.
A primitive router that selects among equivariant modular update candidates for each program operation.

The key controls are:

oracle_exact: exact initializer and exact transition.
oracle_primitive_router: exact initializer and learned primitive router.
sinkhorn_exact: learned initializer and exact transition.
sinkhorn_primitive_router: learned initializer and learned primitive router.
generic_primitive_router: generic initializer and learned primitive router.
sinkhorn_mlp: learned initializer and unstructured MLP transition.

Results

Main Modulus 31

Training used program lengths 1-8. Evaluation used held-out lengths 4, 8, 12, 16, and 24.

Variant	Init	Transition	L=24 query	L=24 belief	Initial belief	Route acc
`oracle_exact`	oracle	exact	100.0%	100.0%	100.0%	n/a
`oracle_primitive_router`	oracle	primitive router	99.9%	99.9%	100.0%	100.0%
`sinkhorn_exact`	sinkhorn cyclic	exact	97.7%	96.8%	99.6%	n/a
`sinkhorn_primitive_router`	sinkhorn cyclic	primitive router	98.7%	98.1%	99.7%	100.0%
`generic_primitive_router`	generic MLP	primitive router	43.0%	28.7%	58.6%	100.0%
`sinkhorn_mlp`	sinkhorn cyclic	MLP	19.9%	5.0%	58.9%	n/a

Modulus 31 strict belief mass

Modulus 31 initial belief mass

Scale Modulus 97

The modulus-97 scale check used the same held-out maximum length of 24 with a smaller evaluation batch.

Variant	Init	Transition	L=24 query	L=24 belief	Initial belief	Route acc
`oracle_exact`	oracle	exact	100.0%	100.0%	100.0%	n/a
`sinkhorn_primitive_router`	sinkhorn cyclic	primitive router	96.2%	94.9%	99.4%	100.0%

Modulus 97 strict belief mass

Interpretation

The full learned structured executor solves the task end-to-end at modulus 31 and remains strong at modulus 97. The control rows isolate why: the learned router is exact when the state is supplied, and the Sinkhorn initializer forms nearly exact support when the transition is supplied. When both are trained together, the model retains both properties.

The failures are also informative. The generic initializer does not cover the initial modular support, so even a perfect router cannot recover the full belief state. The unstructured MLP transition fails despite a structured initializer, showing that transition equivariance is doing real work.

At modulus 97, the residual error is not caused by wrong route selection or missing residue coverage. It appears as slow diffusion of belief mass through long recurrent execution. That points to sharper recurrent state maintenance as the next technical bottleneck.

Limitations

The model uses strong structure: the slot capacity equals the modulus, the initializer is cyclic, and the transition router chooses among modular primitive candidates. The task is synthetic and arithmetic. The result shows that a learned structured executor can compose these mechanisms reliably, not that an unconstrained neural model would discover the same representation.

Artifact Layout

Lightweight code, metrics, figures, and reports live in:

experiments/end_to_end_structured_slot_executor/

Saved model weights live separately in:

large_artifacts/end_to_end_structured_slot_executor/checkpoints/

The checkpoint manifest is:

experiments/end_to_end_structured_slot_executor/checkpoint_manifest.csv

Experiment log 12

Show the running log (12 entries, 2026-06-21)

Objective

Test whether a single learned structured slot executor can solve modular belief-state programs without oracle initialization or exact transition.

Each example starts from:

B = A + d (mod p), with A unknown

The model must place that support into weighted slots, then recurrently execute arithmetic updates and observation filters. The main learned executor combines a Sinkhorn-normalized cyclic initializer with a learned transition router over equivariant modular update primitives.

Primary Questions

Can the full structured executor train end-to-end from scratch?
Does the learned transition remain reliable when the initializer is learned rather than oracle supplied?
Does the Sinkhorn initializer remain reliable when transition gradients are trained jointly?
Which component is the remaining bottleneck: initialization, transition routing, or their interaction?
How far does the best learned executor get on held-out program lengths?

Metrics

decoder_belief_target_mass: probability assigned to the exact final (A,B) support.
decoder_query_target_mass: probability assigned to the exact final query support after projecting the decoded belief.
init_belief_target_mass: probability assigned to the exact initial (A,B) support before recurrent steps.
init_slot_unique_a_frac: fraction of residues covered by distinct slot-level A argmaxes.
mean_route_accuracy: whether the transition router's highest-probability route matches the true operation.
mean_route_entropy: entropy of transition routing.

The strict headline metric is decoder_belief_target_mass at the first evaluated K >= L.

Artifact Layout

Code and lightweight outputs: experiments/end_to_end_structured_slot_executor/
Checkpoints: large_artifacts/end_to_end_structured_slot_executor/checkpoints/
Run outputs: experiments/end_to_end_structured_slot_executor/runs/<variant>/
Analysis outputs: experiments/end_to_end_structured_slot_executor/analysis/

Planned Sequence

Smoke tests on modulus 7 to verify the combined learned executor and component controls.
Pilot runs on modulus 11 with held-out lengths up to 12.
Main runs on modulus 31 with held-out lengths up to 24.
Optional scale check if the main end-to-end row is strong.
Generate analysis tables, figures, checkpoint manifest, standalone report, HTML report, and final audit.

Variant Plan

oracle_exact: exact initializer and exact transition ceiling.
sinkhorn_exact: learned Sinkhorn initializer with exact transition.
oracle_primitive_router: oracle initializer with learned primitive router.
sinkhorn_primitive_router: full structured end-to-end executor.
sinkhorn_cyclic_mixer: softer learned transition control.
generic_primitive_router: generic initializer ablation with structured transition.
sinkhorn_mlp: structured initializer with unstructured transition ablation.

2026-06-21 Setup

Created the standalone experiment directory:

experiments/end_to_end_structured_slot_executor/src/
experiments/end_to_end_structured_slot_executor/reports/
experiments/end_to_end_structured_slot_executor/runs/
experiments/end_to_end_structured_slot_executor/analysis/figures/
large_artifacts/end_to_end_structured_slot_executor/checkpoints/

Implemented the combined harness:

Structured Sinkhorn cyclic initializer.
Generic MLP and oracle initializer controls.
Exact, MLP, cyclic-mixer, and primitive-router transitions.
Full-prefix belief supervision.
Initializer metrics and route diagnostics.
External checkpoint writing.

Next action: compile the source and run modulus-7 smoke tests.

2026-06-21 Smoke Tests

Source compilation passed for both experiment scripts.

Ran modulus-7 smoke variants with evaluation lengths 2 and 3:

Variant	Init	Transition	Steps	L=3 query	L=3 belief	Initial belief	Route acc
`smoke_oracle_exact`	oracle	exact	0	100.0%	100.0%	100.0%	n/a
`smoke_oracle_primitive_router`	oracle	primitive router	500	100.0%	100.0%	100.0%	100.0%
`smoke_sinkhorn_exact`	sinkhorn cyclic	exact	500	98.4%	96.0%	96.9%	n/a
`smoke_sinkhorn_primitive_router`	sinkhorn cyclic	primitive router	700	87.1%	69.1%	73.9%	100.0%
`smoke_sinkhorn_primitive_router_lr01`	sinkhorn cyclic	primitive router	900	99.4%	98.6%	98.9%	100.0%
`smoke_sinkhorn_cyclic_mixer_lr01`	sinkhorn cyclic	cyclic mixer	900	91.5%	82.5%	89.0%	76.2%
`smoke_generic_primitive_router`	generic MLP	primitive router	700	87.9%	72.8%	75.5%	100.0%
`smoke_sinkhorn_mlp`	sinkhorn cyclic	MLP	700	90.6%	77.0%	61.0%	n/a

Smoke interpretation:

The ceiling rows validate generation, decoding, K-indexed evaluation, and route diagnostics.
Primitive routing is easy under oracle initialization and remains easy under learned Sinkhorn initialization once the initializer trains fast enough.
Joint Sinkhorn+router training is sensitive to learning rate. At lr=0.003 the router learns but the initializer lags; at lr=0.01 the full executor reaches 98.6% strict belief at held-out length 3.
The softer cyclic mixer underperforms because route selection remains diffuse.
The generic initializer and unstructured transition ablations are both clearly below the tuned structured executor.

Pilot decision:

Promote sinkhorn_primitive_router_lr01 as the primary full executor.
Keep oracle_exact, sinkhorn_exact, and oracle_primitive_router as component controls.
Keep generic_primitive_router and sinkhorn_mlp as ablations.
Do not promote the cyclic mixer unless the primitive-router row fails at larger modulus.

2026-06-21 Pilot Sweep

Ran modulus-11 pilot variants with training lengths 1-6 and evaluation lengths 3, 6, 9, and 12:

Variant	Init	Transition	Steps	L=12 query	L=12 belief	Initial belief	Route acc
`pilot_oracle_exact`	oracle	exact	0	100.0%	100.0%	100.0%	n/a
`pilot_oracle_primitive_router`	oracle	primitive router	600	100.0%	100.0%	100.0%	100.0%
`pilot_sinkhorn_exact`	sinkhorn cyclic	exact	900	98.2%	97.3%	98.8%	n/a
`pilot_sinkhorn_primitive_router`	sinkhorn cyclic	primitive router	1400	99.2%	98.9%	99.5%	100.0%
`pilot_generic_primitive_router`	generic MLP	primitive router	1000	69.5%	59.5%	60.4%	100.0%
`pilot_sinkhorn_mlp`	sinkhorn cyclic	MLP	1000	78.4%	70.2%	49.2%	n/a

Pilot interpretation:

The learned primitive router remains exact under both oracle and learned Sinkhorn initialization.
The full structured executor reaches near-ceiling held-out performance at modulus 11.
The generic initializer ablation fails because it does not cover the initial support.
The MLP transition ablation fails despite a structured initializer, showing that transition structure is still required.

Main decision:

Run modulus-31 main rows for oracle_exact, oracle_primitive_router, sinkhorn_exact, sinkhorn_primitive_router, generic_primitive_router, and sinkhorn_mlp.
Keep evaluation lengths 4, 8, 12, 16, and 24.
Use the tuned high learning rate for Sinkhorn structured rows.

2026-06-21 Main Sweep

Ran modulus-31 main variants with training lengths 1-8 and evaluation lengths 4, 8, 12, 16, and 24:

Variant	Init	Transition	Steps	L=24 query	L=24 belief	Initial belief	Route acc
`main_oracle_exact`	oracle	exact	0	100.0%	100.0%	100.0%	n/a
`main_oracle_primitive_router`	oracle	primitive router	800	99.9%	99.9%	100.0%	100.0%
`main_sinkhorn_exact`	sinkhorn cyclic	exact	1800	97.7%	96.8%	99.6%	n/a
`main_sinkhorn_primitive_router`	sinkhorn cyclic	primitive router	2400	98.7%	98.1%	99.7%	100.0%
`main_generic_primitive_router`	generic MLP	primitive router	1400	43.0%	28.7%	58.6%	100.0%
`main_sinkhorn_mlp`	sinkhorn cyclic	MLP	1600	19.9%	5.0%	58.9%	n/a

Main interpretation:

The full learned structured executor succeeds end-to-end at modulus 31, reaching 98.1% strict belief at held-out length 24.
The primitive-router transition remains exact when trained jointly with the learned Sinkhorn initializer.
Both learned pieces are necessary. A generic initializer with a perfect learned router fails, and a structured initializer with an unstructured MLP transition fails.
The full learned row slightly exceeds the Sinkhorn+exact component control at length 24 in this run, likely because the learned primitive router preserves sharper slot-local distributions.

Scale decision:

Run a modulus-97 full-program scale check for sinkhorn_primitive_router.
Include oracle_exact as the standalone ceiling.
Use a smaller batch/evaluation budget to keep the p97 dense pair target tractable.

2026-06-21 Scale Check

Ran modulus-97 scale variants with training lengths 1-8 and evaluation lengths 4, 8, 12, 16, and 24:

Variant	Init	Transition	Steps	L=24 query	L=24 belief	Initial belief	Route acc
`scale_oracle_exact`	oracle	exact	0	100.0%	100.0%	100.0%	n/a
`scale_sinkhorn_primitive_router`	sinkhorn cyclic	primitive router	1800	96.2%	94.9%	99.4%	100.0%

Scale interpretation:

The full learned structured executor remains strong at modulus 97, reaching 94.9% strict belief at held-out length 24.
Initial support formation and route selection are not the limiting factors: initial belief is 99.4%, slot coverage is 100.0%, and route accuracy is 100.0%.
The remaining loss appears as gradual belief diffusion over long recurrent programs rather than a discrete routing failure.
The p97 row is strong enough to include as a scale result in the standalone report, but it is not exact; the limitation should be stated directly.

2026-06-21 Final Audit

Final artifacts created:

reports/end_to_end_structured_slot_paper.md
reports/end_to_end_structured_slot_paper.html
checkpoint_manifest.csv

Verification:

Source compilation passed: python -m py_compile src/end_to_end_structured_slot_experiment.py src/analyze_end_to_end_structured_slot.py
Checkpoint manifest validation passed for 19 saved checkpoints.
Markdown and HTML report image references resolve.
No .pt, .pth, or .ckpt files are stored inside the lightweight experiment directory.
The standalone wording scan passed for the listed external-lineage phrases.
Removed the compile cache after verification.

Artifact sizes:

experiments/end_to_end_structured_slot_executor/: 8.8M
large_artifacts/end_to_end_structured_slot_executor/: 3.8M

Conclusion:

The strongest learned row is sinkhorn_primitive_router. It reaches 98.1% strict belief at modulus 31 and held-out length 24, then reaches 94.9% strict belief at modulus 97 and held-out length 24. The component controls show that both structured support formation and structured transition routing are needed.