Structured Slot Initializer Ladder Experiment

Finished2026-06-21imported · line YBenchmark Generalization Structured Execution and Compilers GitHub ↗

The one idea you need

Before the model computes, it must drop each possible hidden value into its own memory slot, like seating guests in numbered chairs. Left free, a plain network crowds several onto one chair and leaves others empty; a matching rule seats every value exactly once.

The question

Before a small model runs a step-by-step calculation, it loads its starting guesses into fixed memory slots. Does getting this right need a bigger network, or something else?

What we found

Structure, not scale. A plain general-purpose network placed only 55.5% of its belief on the correct starting setup and gave barely half the possible values their own slot, doubling several onto the same one. A rule that forces each value into a distinct slot lifted that to 98.1%, essentially matching a flawless setup. Learning the underlying math alone was not the missing piece.

Why it matters

When a small model must hold a set of distinct possibilities in fixed memory before a later step can use them reliably, don't just enlarge the network. Add an explicit step that gives each possibility its own slot and leaves none uncovered.

Belief on the correct starting setup55.5% → 98.1%plain network vs a one-slot-per-value matching rule, on longer unseen programs

Values given their own slot50% → 100%plain network vs matching rule; the plain one doubled slots onto the same value

Knew the math but botched the filing84% vs 50%the plain network got the value relationship right this often, yet covered only this many values

Gap to a flawless setupabout 2 pointsthe matching-rule setup trails a perfect reference (98.1% vs 100%)

On this page

Results at a glance
Overview
Report
Experiment log
Figures
Data files
Reproduce
Related

Results at a glance 4

How much of the answer lands on the exact target, by setup style

How to read

Each group along the bottom is a setup style, from a flawless reference on the left to a plain general-purpose network on the right. Two bars per group show how much belief landed exactly right; taller is better.

Takeaway → Bars climb steadily as more structure is added; the matching-rule setup nearly touches the flawless reference while the plain network sits far below it.

Data table

initializer	strict belief mass	query mass
oracle	100%	100%
sinkhorn cyclic	98.1%	98.6%
factorized cyclic	92.2%	93.6%
factorized free-B	73.3%	78.9%
generic MLP (full belief)	55.5%	62.8%
generic MLP (final query)	16.7%	30.5%

Numbers from report table (reports/structured_slot_initializer_ladder_paper.md, Main Result)

Technical framing

Initializer ladder at modulus 31: held-out execution at length 24 — Adding structure step by step closes the gap to the oracle; Sinkhorn-constrained slot assignment reaches 98.1% belief mass vs 55.5% for a generic MLP.

Answer quality stays flat as the task runs more steps

How to read

The bottom axis is how many steps the exact calculator runs, up to 24. Height is how much belief stays on the exact right answer, higher being better. Each line is one setup style, flawless reference on top.

Takeaway → Every line runs nearly flat across all lengths, so the calculator neither repairs nor spoils the setup; final quality is locked in at the very start.

Data table

program length	oracle	sinkhorn cyclic	factorized cyclic	factorized free-B	generic MLP (full belief)
4	100%	99.6%	90.1%	72.1%	47.3%
8	100%	99.4%	91.5%	73.5%	52.2%
12	100%	99.3%	92.7%	74.7%	55.8%
16	100%	99.1%	92.5%	75.7%	58.2%
24	100%	98.1%	92.2%	73.3%	55.5%

Numbers from analysis/summary.md (First K >= L Summary, main modulus 31)

Technical framing

Belief mass vs held-out program length (modulus 31) — The exact transition preserves initial support quality: each initializer holds a nearly flat level across lengths 4-24, set by its initialization.

Learning the rule is not enough without covering every value

How to read

Each group is a setup style. One bar is how often it got the value relationship right; the other is whether every value got its own slot. Taller is better on both.

Takeaway → Several setups ace the relationship yet leave many values sharing slots; only the matching-rule setup scores full marks on giving each value its own slot.

Data table

initializer	relation accuracy (B=A+d)	unique-A slot coverage
sinkhorn cyclic	100%	100%
factorized cyclic	100%	90.3%
factorized free-B	89%	83.9%
generic MLP (full belief)	84.5%	50.4%
generic MLP (final query)	52.6%	48.2%

Numbers from report table (reports/structured_slot_initializer_ladder_paper.md, Main Result)

Technical framing

Why initializers fail: relation learning vs slot coverage (modulus 31) — A model can learn the modular relation yet still duplicate slots; only Sinkhorn assignment guarantees every residue gets its own slot.

Matching-rule setup tracks the flawless reference as values multiply

How to read

The bottom axis is the size of the pool of possible values, from small to largest. Height is how much belief lands on target, higher being better. One bar is the learned matching-rule setup, the other the flawless reference.

Takeaway → The learned setup stays within about two points of the flawless reference at every size, holding near-perfect even at the largest pool of values.

Data table

Modulus (hardest evaluation per phase)	Sinkhorn cyclic	Oracle ceiling
Mod 7 (len 3)	98.4%	100%
Mod 11 (len 12)	97.8%	100%
Mod 31 (len 24)	98.6%	100%
Mod 97 (init only)	99.7%	100%

Numbers from experiments/structured_slot_initializer_ladder/analysis/first_k_ge_l_summary.csv

Technical framing

Sinkhorn cyclic tracks the oracle from modulus 7 to 97 — The learned Sinkhorn cyclic initializer stays within about 2 points of the oracle at every modulus tested, including 97 states.

In the author’s words from the Report · “Abstract”

At modulus 31 with held-out program length 24, a generic MLP initializer reaches 55.5% strict final belief mass under full-belief supervision. A free-B factorized initializer reaches 73.3%. A cyclic initializer with soft overlap regularization reaches 92.2% but leaves some residues duplicated. Replacing the soft overlap penalty with Sinkhorn-normalized slot assignment reaches 98.1% strict final belief mass and 98.6% query mass, with 100% measured slot coverage. An initializer-only modulus-97 scale check reaches 98.7% strict initial belief mass.

Overview

This experiment tests which initializer structure is sufficient to populate a sparse modular belief support before an exact recurrent transition executes the program.

src/structured_slot_initializer_ladder_experiment.py: task generator, initializer ladder models, exact transition, checkpointing, and evaluation harness.
src/analyze_structured_slot_initializer_ladder.py: analysis and figure generation.
reports/structured_slot_initializer_ladder_experiment_log.md: chronological experiment log.
reports/structured_slot_initializer_ladder_paper.md: standalone written report.
reports/structured_slot_initializer_ladder_paper.html: standalone HTML report.
runs/: JSON and CSV run outputs.
analysis/: generated summaries and figures.
checkpoint_manifest.csv: checkpoint paths and sizes.

Checkpoints are written outside the experiment directory under:

../../large_artifacts/structured_slot_initializer_ladder/checkpoints/

Download this experiment directory for the normal research bundle. Download ../../large_artifacts/structured_slot_initializer_ladder/ only when saved model weights are needed.

Report

Rendered from reports/structured_slot_initializer_ladder_paper.md

Abstract

This experiment studies a narrow failure mode in recurrent slot execution: forming the initial sparse belief support. Each example begins with an unknown register A and a known modular relation B=A+d (mod p). A slot model must place that support into weighted slots, then an exact recurrent transition executes arithmetic and observation programs. Because the transition is exact, final errors isolate initializer quality.

Task

Each example starts from the support

{(A, B): B = A + d (mod p)}

where d is observed and A is unknown. Programs then apply modular register updates and observation filters:

A=A+c, A=A-c, B=B+c, B=B-c
A=A+B, B=B+A, A=A-B, B=B-A
observations on A mod m or B mod m

The decoded belief is a weighted mixture of slot-local A and B distributions. Evaluation reports strict probability mass on the exact final (A,B) support at the first evaluated K >= L, where L is program length.

Initializers

oracle: exact slot support, used as a ceiling.
generic_mlp: delta and slot-id embeddings passed through an MLP that emits A, B, and weight logits.
factorized_free_b: learned slot-to-A logits, with a free delta-conditioned MLP for B.
factorized_cyclic: learned slot-to-A logits, with B produced by the exact cyclic shift A+d.
sinkhorn_cyclic: learned slot-to-A logits normalized by Sinkhorn iterations, then shifted by d to produce B.

The decisive diagnostic metrics are initial belief mass, relation accuracy B=A+d, unique A slot coverage, and slot overlap.

Main Result

Modulus 31, training lengths 1-8, evaluation lengths 4, 8, 12, 16, and 24:

Initializer	L=24 query mass	L=24 belief mass	Initial belief	Relation acc	Unique A slots
oracle	100.0%	100.0%	100.0%	100.0%	100.0%
sinkhorn_cyclic	98.6%	98.1%	99.7%	100.0%	100.0%
factorized_cyclic	93.6%	92.2%	90.0%	100.0%	90.3%
factorized_free_b	78.9%	73.3%	74.2%	89.0%	83.9%
generic_mlp, full belief	62.8%	55.5%	46.4%	84.5%	50.4%
generic_mlp, final query	30.5%	16.7%	29.4%	52.6%	48.2%

The generic MLP does not reliably form the sparse support at modulus 31. It can learn partial relation structure, but it duplicates many slots and leaves about half of the residue set uncovered. The free-B factorized model improves coverage but still must learn the relation. The cyclic factorization fixes the relation exactly, but a soft overlap penalty is not enough to guarantee full assignment coverage. Sinkhorn normalization supplies that missing assignment constraint.

Main strict belief mass

Main unique slot coverage

Scale Check

The modulus-97 check isolates initialization at K=0; it does not execute full p=97 programs. It tests whether the slot assignment mechanism itself scales to a larger residue set.

Initializer	Initial query mass	Initial belief mass	Relation acc	Unique A slots
oracle	100.0%	100.0%	100.0%	100.0%
sinkhorn_cyclic	99.7%	98.7%	100.0%	100.0%

Scale initial belief mass

Interpretation

The initializer problem is not just learning the modular relation. A model can learn B=A+d and still fail by assigning multiple slots to the same residue. The successful ingredient is a coverage-aware assignment mechanism: the Sinkhorn initializer makes each slot choose a residue while also making each residue receive a slot.

With exact transition fixed, final execution quality closely tracks initial support quality. This makes strict initial belief mass a useful mechanistic predictor: when initial mass and unique coverage are high, held-out program execution remains high; when either is low, the exact transition preserves that initial error rather than repairing it.

Limitations

The recurrent transition is exact by construction, so this experiment does not claim that a neural transition and a learned initializer jointly solve the full problem. The modulus-97 row is initializer-only, not full program execution. The Sinkhorn initializer also uses strong structure: it assumes one slot per residue and hard-wires the cyclic relation.

Conclusion

For modular sparse belief execution, the most important initializer structure is not a larger MLP. It is a constrained assignment from slots to residues plus the exact cyclic relation. Under that structure, a learned initializer reaches near-oracle final execution at modulus 31 and forms near-exact initial support at modulus 97.

Experiment log 13

Show the running log (13 entries, 2026-06-21)

Objective

Test whether a structured initializer can place the sparse modular support

B = A + d (mod p), with A unknown

into weighted slots before an exact recurrent transition executes the program. Each slot stores one distribution over A, one distribution over B, and one mixture weight. The experiment varies only the initializer while keeping the transition update exact by default.

Primary Questions

Can a generic prompt-conditioned initializer learn the support reliably?
Does hard-wiring the cyclic relation B=A+d leave only slot coverage to learn?
Does a free B initializer fail relative to the cyclic-relation initializer?
Do coverage, relation, and slot-purity regularizers improve strict initial belief mass?
Does any learned initializer remain reliable at larger modulus and held-out program length?

Metrics

init_belief_target_mass: probability assigned to the exact initial (A,B) support before any recurrent transition steps.
init_slot_relation_accuracy: fraction of slots whose strongest A and B values satisfy B=A+d.
init_slot_unique_a_frac: fraction of the target residue set represented by distinct slot-level A argmaxes.
decoder_belief_target_mass: probability assigned to the exact final (A,B) support after executing K recurrent steps.
decoder_query_target_mass: probability assigned to the exact final query support after projecting the decoded belief.
mean_slot_purity: mean product of the strongest A and B probabilities per slot.

The strict headline metric is decoder_belief_target_mass at the first evaluated K >= L, with init_belief_target_mass as the mechanistic diagnostic.

Artifact Layout

Code and lightweight outputs: experiments/structured_slot_initializer_ladder/
Checkpoints: large_artifacts/structured_slot_initializer_ladder/checkpoints/
Run outputs: experiments/structured_slot_initializer_ladder/runs/<variant>/
Analysis outputs: experiments/structured_slot_initializer_ladder/analysis/

Planned Sequence

Smoke tests on modulus 7 to validate all initializer modes and diagnostics.
Pilot runs on modulus 11 to compare generic, factorized, and structured regularized initializers.
Main runs on modulus 31 using the most informative pilot variants.
Scale check on modulus 97 if a learned structured initializer is promising.
Generate tables, figures, checkpoint manifest, standalone report, HTML report, and final audit.

Variant Plan

oracle: exact initializer ceiling.
generic_mlp: prompt-conditioned MLP over delta and slot id.
factorized_cyclic: learned slot-to-residue logits for A, with B produced by the exact cyclic shift A+d.
factorized_free_b: learned slot-to-residue logits for A, with a free delta-conditioned MLP for B.
sinkhorn_cyclic: learned slot-to-residue logits normalized with Sinkhorn iterations before applying the exact cyclic shift A+d.
indexed_cyclic: deterministic indexed cyclic initializer control.

2026-06-21 Setup

Created the standalone experiment directory:

experiments/structured_slot_initializer_ladder/src/
experiments/structured_slot_initializer_ladder/reports/
experiments/structured_slot_initializer_ladder/runs/
experiments/structured_slot_initializer_ladder/analysis/figures/
large_artifacts/structured_slot_initializer_ladder/checkpoints/

Implemented the initializer ladder harness:

Exact modular task generator and exact recurrent transition.
Oracle, generic MLP, factorized cyclic, factorized free-B, and indexed cyclic initializers.
Full-prefix belief and final-query supervision.
Optional initializer coverage, relation, weight-uniformity, and slot-entropy regularizers.
Initializer-specific diagnostics for initial support mass, relation accuracy, and unique slot coverage.
External checkpoint writing and analysis script.

Next action: run source compilation and modulus-7 smoke tests.

2026-06-21 Smoke Tests

Source compilation passed for both experiment scripts.

Ran modulus-7 smoke variants with evaluation lengths 2 and 3:

Variant	Init mode	Supervision	Steps	L=3 query	L=3 belief	Initial belief	Relation acc	Unique A slots
`smoke_oracle_ceiling`	oracle	full belief	0	100.0%	100.0%	100.0%	100.0%	100.0%
`smoke_indexed_cyclic`	indexed cyclic	full belief	0	100.0%	100.0%	100.0%	100.0%	100.0%
`smoke_generic_mlp`	generic MLP	full belief	500	93.3%	84.7%	85.4%	90.8%	88.0%
`smoke_generic_mlp_final_query`	generic MLP	final query	500	89.9%	76.1%	77.9%	89.8%	79.6%
`smoke_factorized_cyclic_plain`	factorized cyclic	full belief	500	85.4%	64.4%	69.3%	100.0%	71.4%
`smoke_factorized_cyclic_reg`	factorized cyclic	full belief	500	85.7%	65.1%	70.3%	100.0%	71.4%
`smoke_factorized_cyclic_overlap`	factorized cyclic	full belief	900	92.7%	83.0%	85.3%	100.0%	85.7%
`smoke_factorized_free_b_reg`	factorized free-B	full belief	500	83.8%	62.5%	67.0%	91.1%	85.7%

Smoke interpretation:

The oracle and indexed cyclic controls are exact, validating generation, exact transition, K-indexing, and decoding.
Generic MLP initialization is a strong small-modulus baseline when trained with full prefix-belief supervision.
Final-query supervision learns less initial support than full belief.
Factorized cyclic initialization learns the relation B=A+d immediately by construction, but without an explicit overlap penalty it duplicates slots and leaves part of the residue set uncovered.
Adding slot-overlap regularization fixes most of that failure mode and brings the factorized cyclic initializer close to the generic MLP baseline.
The free-B factorized ablation is weaker, suggesting that the cyclic relation constraint is useful but not sufficient by itself.

Pilot decision:

Run p=11 pilots for oracle, generic full-belief, generic final-query, factorized cyclic with overlap, and factorized free-B with relation/coverage regularization.
Use training lengths 1-6 and held-out evaluation lengths 3, 6, 9, and 12.

2026-06-21 Pilot Sweep

Ran five modulus-11 pilot variants with training lengths 1-6 and evaluation lengths 3, 6, 9, and 12:

Variant	Init mode	Supervision	Steps	L=12 query	L=12 belief	Initial belief	Relation acc	Unique A slots	A overlap
`pilot_oracle_ceiling`	oracle	full belief	0	100.0%	100.0%	100.0%	100.0%	100.0%	0.000
`pilot_factorized_cyclic_overlap`	factorized cyclic	full belief	1400	99.3%	98.9%	99.5%	100.0%	100.0%	0.000
`pilot_factorized_free_b_reg`	factorized free-B	full belief	1100	77.9%	69.0%	76.3%	92.5%	81.8%	0.039
`pilot_generic_mlp_final_query`	generic MLP	final query	1100	71.5%	60.8%	68.3%	89.2%	69.6%	0.085
`pilot_generic_mlp_full_belief`	generic MLP	full belief	1100	66.9%	55.0%	59.2%	79.9%	64.2%	0.142

Pilot interpretation:

The exact ceiling is again perfect, so the task is not lossy under exact transition and sufficient initialization.
The factorized cyclic initializer with overlap regularization is nearly exact through held-out length 12.
The decisive diagnostic is slot coverage: the best structured initializer reaches 100% unique A slots and zero measured A overlap, while every learned baseline leaves duplicated or diffuse slot support.
Freeing B hurts despite relation regularization, so the cyclic relation is not merely a weak auxiliary target; it is the critical structural constraint.
Generic final-query supervision beats generic full-belief supervision at this scale, but both are far below the structured cyclic initializer.

Main decision:

Run modulus-31 main variants for oracle, generic full-belief, generic final-query, factorized free-B, and factorized cyclic with overlap.
Use training lengths 1-8 and held-out evaluation lengths 4, 8, 12, 16, and 24.

2026-06-21 Main Sweep

Ran five modulus-31 main variants with training lengths 1-8 and evaluation lengths 4, 8, 12, 16, and 24:

Variant	Init mode	Supervision	Steps	L=24 query	L=24 belief	Initial belief	Relation acc	Unique A slots	A overlap
`main_oracle_ceiling`	oracle	full belief	0	100.0%	100.0%	100.0%	100.0%	100.0%	0.000
`main_factorized_cyclic_overlap`	factorized cyclic	full belief	2400	93.6%	92.2%	90.0%	100.0%	90.3%	0.007
`main_factorized_free_b_reg`	factorized free-B	full belief	1600	78.9%	73.3%	74.2%	89.0%	83.9%	0.015
`main_generic_mlp_full_belief`	generic MLP	full belief	1400	62.8%	55.5%	46.4%	84.5%	50.4%	0.119
`main_generic_mlp_final_query`	generic MLP	final query	1400	30.5%	16.7%	29.4%	52.6%	48.2%	0.047

Main interpretation:

Structured cyclic initialization scales far better than the generic initializers at modulus 31.
The free-B ablation is substantially better than generic initialization but still well below the exact cyclic relation initializer.
The remaining gap for main_factorized_cyclic_overlap is visibly an assignment gap, not a relation gap: relation accuracy is 100%, but unique A slot coverage is only 90.3%.
The next targeted iteration is a Sinkhorn-normalized cyclic initializer, which turns slot coverage from a soft overlap penalty into a structural constraint.

2026-06-21 Sinkhorn Iteration

Added a sinkhorn_cyclic initializer:

It learns slot-to-residue logits.
Sinkhorn normalization makes the slot/residue assignment approximately doubly stochastic.
The exact cyclic shift then sets B=A+d.

Ran validation rows:

Variant	Modulus	Steps	Eval length	Query	Belief	Initial belief	Relation acc	Unique A slots	A overlap
`smoke_sinkhorn_cyclic`	7	500	3	98.4%	96.0%	96.9%	100.0%	100.0%	0.005
`pilot_sinkhorn_cyclic`	11	800	12	97.8%	96.7%	98.5%	100.0%	100.0%	0.001
`main_sinkhorn_cyclic`	31	2400	24	98.6%	98.1%	99.7%	100.0%	100.0%	0.000

Sinkhorn interpretation:

The Sinkhorn initializer fixes the p=31 assignment gap observed in the overlap-regularized factorized initializer.
At p=31 and held-out length 24, strict belief improves from 92.2% to 98.1%.
The diagnostic metrics show complete slot coverage: 100% relation accuracy, 100% unique A slots, and zero measured A overlap.

2026-06-21 Modulus-97 Initializer Scale Check

Ran an initializer-only scale check at modulus 97. This check evaluates K=0 support formation only; it does not run full p=97 program execution.

Variant	Modulus	Steps	Initial query	Initial belief	Relation acc	Unique A slots	A overlap
`scale_oracle_init_only`	97	0	100.0%	100.0%	100.0%	100.0%	0.000
`scale_sinkhorn_cyclic_init_only`	97	1200	99.7%	98.7%	100.0%	100.0%	0.000

Scale interpretation:

The Sinkhorn assignment mechanism scales to a 97-residue support in the isolated initializer setting.
Because this row is initializer-only, the p=31 full-program result remains the main behavioral result.

Next action: generate the standalone report, HTML artifact, checkpoint manifest, and final audit.

2026-06-21 Final Audit

Report artifacts generated:

reports/structured_slot_initializer_ladder_paper.md
reports/structured_slot_initializer_ladder_paper.html

Consistency checks:

Source compilation passed for src/structured_slot_initializer_ladder_experiment.py and src/analyze_structured_slot_initializer_ladder.py.
The paper and HTML report are self-contained and contain no references to external experiment artifacts or source scripts.
The experiment directory contains no .pt, .pth, or .ckpt files.
Checkpoints are stored under large_artifacts/structured_slot_initializer_ladder/checkpoints/.
checkpoint_manifest.csv contains 18 rows; every listed checkpoint exists and matches the recorded byte count.
The HTML report references 3 local figure files, and all referenced figures exist.
The analysis directory contains 24 generated figure files.
The experiment directory is 7.4M; the external artifact directory is 2.0M.