Research log Small Model Experimentation
GitHub

Joint Register Executor Experiment

Finishedimported 2026-07-12imported · line YStructured Execution and CompilersGitHub ↗
Extra thinking pays off only with linked memory

The one idea you need

Picture two dials showing unknown numbers, locked a fixed distance apart. Track each dial on its own and you lose the lock. This system instead remembers every allowed pair at once and rewrites that whole set, one instruction per internal thinking step.

The question

Can a model run a hidden step-by-step program in its head and keep two linked unknown numbers correct, even for programs longer than anything it trained on?

What we found

Yes, but only with the right memory. When the model held every allowed number-pair together and spent one thinking step per instruction, its confidence in the exactly correct answer set jumped from near-random (about 3%) to between 87 and 98% the moment its steps reached program length, even on programs three times longer than anything in training. A twin that tracked each number separately never beat chance.

Why it matters

If you want a small model to reason harder by spending more internal steps, first make sure its memory can actually represent the linked quantities the task needs. Extra steps cannot rescue a memory that drops the correlations.

Confidence in the exact answer, right memoryabout 3% → 87 to 98%random chance versus the linked-memory model once its steps match the program
Trained short, ran longup to 8 → 24longest program seen in training versus longest solved correctly at test
Separate-memory twinstuck near 1 in 31the version that tracked each number on its own never beat about 3% chance
Enough steps versus too fewabout 4% → 87%the jump on the longest programs once thinking steps reach the instruction count
On this page
  1. Results at a glance
  2. Overview
  3. Report
    1. Abstract
    2. Lay Summary
    3. 1. Question
    4. 2. Task
    5. 3. Models
    6. 4. Metrics
    7. 5. Main Result
    8. 6. Controls
    9. 7. Interpretation
    10. 8. Limits
    11. 9. Next Iterations
    12. 10. Reproducibility
    13. 11. Bottom Line
  4. Experiment log
  5. Figures
  6. Data files
  7. Reproduce
  8. Related

Results at a glance 6

Only the linked-memory model solves the puzzle

How to read

Bars grouped by program length show how much confidence each of three models places on the exactly correct set of number pairs; taller is better. The linked-memory model towers over two simpler models stuck at random-guess level.

0%25%50%75%100%97.8%3.2%7.9%L=495.7%3.2%3.5%L=893.6%3.2%3.2%L=1291.6%3.2%3.2%L=1687.4%3.2%3.2%L=24

Takeaway → Linked-memory bars stay high at every length while both simpler models flatline at random-guess, so the link between the two numbers must be tracked jointly.

Data table
Program length (trained on lengths 1-8)Joint recurrentMarginal recurrentStatic compiler
L=497.8%3.2%7.9%
L=895.7%3.2%3.5%
L=1293.6%3.2%3.2%
L=1691.6%3.2%3.2%
L=2487.4%3.2%3.2%

Numbers from experiments/joint_register_executor/analysis/mod31_control_summary.csv

Technical framing

Belief-state execution (mod 31): joint recurrent executor vs controls — Only joint latent state executes correlated beliefs: marginal and static controls sit at the 3.2% (1/31) baseline, even on held-out lengths.

The answer snaps into place once steps match the instructions

How to read

The horizontal axis is the number of internal thinking steps and the vertical axis is confidence in the correct set of pairs; higher is better. Each line is a different program length, and longer programs need more steps.

0%50%100%150%01020

Takeaway → Every line sits flat near random-guess, then leaps up exactly when the thinking steps reach the number of instructions, the signature of a program being run.

Data table
Internal recurrent steps KL=4L=8L=12 (held-out)L=16 (held-out)L=24 (held-out)
04.4%3.1%3.2%3.1%3.1%
17.9%3.8%3%3.4%3.5%
214.5%4.1%3.1%3.3%3.4%
497.8%5.7%4%3.3%3.3%
897.8%95.7%5.6%3.8%3.2%
1297.8%95.7%93.6%5.2%3.4%
1697.8%95.7%93.6%91.6%3.8%
2497.8%95.7%93.6%91.6%87.4%

Numbers from experiments/joint_register_executor/analysis/final_metrics_long.csv (run main_belief_joint_mod31)

Technical framing

Target-support mass vs internal steps K: sharp threshold at K = program length — Mass stays near baseline until K reaches program length, then jumps to 87-98% — the executor signature, generalizing past training lengths.

Extra thinking helps only once it finishes the program

How to read

For each program length, two bars compare the best result with too few thinking steps against the result once steps match the instruction count; higher is better.

0%25%50%75%100%14.5%97.8%L=45.7%95.7%L=85.6%93.6%L=125.2%91.6%L=163.8%87.4%L=24

Takeaway → The too-few-steps bars sit near random-guess while the matched bars jump to between 87 and 98%, so steps help only when they run the whole program.

Data table
Program lengthBest mass before K>=LMass at first K>=L
L=414.5%97.8%
L=85.7%95.7%
L=125.6%93.6%
L=165.2%91.6%
L=243.8%87.4%

Numbers from experiments/joint_register_executor/analysis/mod31_threshold_summary.csv

Technical framing

Before vs at the K=L threshold (joint executor, mod 31) — Extra internal steps only help when they correspond to executing the program: sub-threshold budgets stay near the 3.2% baseline at every length.

Accuracy holds up as programs get longer

How to read

The horizontal axis is program length and the vertical axis is confidence in the correct set once steps match the instruction count; higher is better. The two lines are a smaller and a larger range of possible numbers.

85%90%95%100%1020Joint recurrent, mod 11 (pilot)Joint recurrent, …Joint recurrent, mod 31 (main)Joint recurrent, …

Takeaway → Both lines slope down only gently as programs lengthen, showing accuracy stays high well past the lengths seen in training.

Data table
Program length LJoint recurrent, mod 11 (pilot)Joint recurrent, mod 31 (main)
397.5%—
4—97.8%
695.2%—
8—95.7%
992.9%—
1290.6%93.6%
16—91.6%
24—87.4%

Numbers from experiments/joint_register_executor/analysis/final_metrics_long.csv (also analysis/summary.md table)

Technical framing

Threshold quality scales across moduli: support mass at K = L — At both moduli, support mass at the K=L threshold decays only gently with program length while top-1 stays at 100%.

During training, only the linked-memory model actually learns

How to read

The horizontal axis is training progress and the vertical axis is the share of pairs the model gets right while learning; higher is better, with one line per model.

0%50%100%150%200400Joint recurrentStatic compiler controlStatic compiler c…Marginal recurrent controlMarginal recurren…

Takeaway → The linked-memory line climbs to a perfect score early while both simpler models crawl along the bottom, never really learning the task.

Data table
Training stepJoint recurrentMarginal recurrent controlStatic compiler control
13.9%2.3%3.3%
2521.1%——
5043.4%3.1%5.5%
7580.5%——
100100%3.9%10%
150100%3.1%12.1%
200100%2.7%13.3%
250100%—16.6%
300100%—16.2%
350100%——
400100%——
450100%——
500100%——

Numbers from experiments/joint_register_executor/runs/main_belief_joint_mod31/results.json, runs/control_belief_marginal_mod31/results.json, runs/control_belief_static_mod31/results.json (train logs)

Technical framing

Training curves (mod 31 belief task): only the joint executor learns — The joint executor hits 100% train pair accuracy by step 100; the marginal control never leaves baseline and the static control plateaus low.

An easier version of the task cannot tell the models apart

How to read

On a simpler task with a single known starting pair, the horizontal axis is thinking steps and the vertical axis is the fraction of pairs correct; higher is better. The two model lines lie on top of each other.

0%50%100%150%0510Joint recurrent (exact task)Joint recurrent (…Marginal recurrent (exact task)Marginal recurren…

Takeaway → Both models trace the same curve and both reach a perfect score, so only the harder task with linked unknowns exposes the difference between them.

Data table
Internal recurrent steps KJoint recurrent (exact task)Marginal recurrent (exact task)
01.1%1.1%
11.1%1.1%
20.9%0.9%
30.6%0.6%
62.2%2.2%
92.5%2.5%
12100%100%

Numbers from experiments/joint_register_executor/runs/pilot_joint_mod11/metrics_step00300.csv, runs/control_marginal_mod11/metrics_step00300.csv (length=12 rows)

Technical framing

Why the belief task is needed: exact-state pilot does not separate models (mod 11, L=12) — On single-state programs the marginal control also reaches 100% at K=L, so only the correlated belief task forces joint state.

In the author’s words from the Report · “Abstract”

On modulo-31 programs trained only on lengths 1-8, it generalizes to held-out lengths 12, 16, and 24. Top-1-on-support accuracy is near baseline when K<L, then reaches 100% exactly when K>=L. Target-support mass at K=L is 97.8%, 95.7%, 93.6%, 91.6%, and 87.4% for lengths 4, 8, 12, 16, and 24. A marginal recurrent control stays at the 3.2% factorized baseline, and a static compiler control stays near baseline on longer lengths. The result supports the claim that joint latent state plus recurrent execution can solve correlated belief-state programs with clean test-time-compute scaling.

Overview

This experiment tests latent recurrent execution for two-register modular programs with cross-register operations such as A=A+B and B=B-A.

Contents

  • src/joint_register_executor_experiment.py: training and evaluation harness.
  • src/analyze_joint_register_executor.py: aggregates run metrics and regenerates figures.
  • reports/joint_register_executor_paper.md: standalone paper-style report.
  • reports/joint_register_executor_paper.html: HTML version of the report.
  • reports/joint_register_executor_experiment_log.md: chronological experiment log.
  • runs/: JSON and CSV run outputs.
  • analysis/: generated summaries and figures.
  • checkpoint_manifest.csv: saved checkpoint list and sizes.

Checkpoints are written outside the experiment directory under:

../../large_artifacts/joint_register_executor/checkpoints/

Intended Download Split

Download this experiment directory for code, reports, figures, and metrics. Download the matching large_artifacts checkpoint directory only if saved weights are needed.

Report

Rendered from reports/joint_register_executor_paper.md

A controlled experiment on recurrent hidden-state computation over uncertain two-register programs

Abstract

This experiment tests whether a latent recurrent runtime can execute programs over a correlated belief state. Each example starts with an unknown register pair constrained by B=A+d (mod p), so the correct hidden state is not one pair but a line-shaped support over (A,B). A program containing constant and cross-register operations transforms that support. The model must assign probability mass to the exact final support after K internal recurrent steps.

The primary joint-state executor stores a categorical distribution over all (A,B) pairs and applies one learned transition per hidden step. On modulo-31 programs trained only on lengths 1-8, it generalizes to held-out lengths 12, 16, and 24. Top-1-on-support accuracy is near baseline when K<L, then reaches 100% exactly when K>=L. Target-support mass at K=L is 97.8%, 95.7%, 93.6%, 91.6%, and 87.4% for lengths 4, 8, 12, 16, and 24. A marginal recurrent control stays at the 3.2% factorized baseline, and a static compiler control stays near baseline on longer lengths. The result supports the claim that joint latent state plus recurrent execution can solve correlated belief-state programs with clean test-time-compute scaling.

Lay Summary

The model does not know the exact starting values of A and B. It only knows a relation:

B = A + d (mod p)

That relation describes many possible starting pairs. The model then receives a hidden program such as:

A = A + 7
B = B - A
A = A + B
...

The correct internal state is the whole set of possible (A,B) pairs after each step. The joint recurrent model learns to update that set one instruction at a time. If the program has 16 instructions, the model needs 16 internal steps. With fewer steps it stays near baseline; with enough steps it puts most probability on the exact final set.

1. Question

The experiment asks whether recurrent latent computation can maintain and update a correlated hidden belief state. The desired evidence has three parts:

  1. Accuracy should depend on the internal step budget K.
  2. The threshold should align with program length L: low when K<L, high when K>=L.
  3. A model without joint state should fail because separate marginals cannot represent the correlation.

The task is deliberately controlled. The goal is not broad language reasoning; it is to test a specific mechanism under conditions where the required computation is exactly known.

2. Task

Programs operate over two registers modulo p.

Initial belief:

{(A, B): B = A + d mod p}

For modulus 31, this support contains 31 possible pairs out of 961 total pairs.

Operations:

OperationMeaning
A=A+cadd a constant to A
A=A-csubtract a constant from A
B=B+cadd a constant to B
B=B-csubtract a constant from B
A=A+Badd B into A
B=B+Aadd A into B
A=A-Bsubtract B from A
B=B-Asubtract A from B

Each example has:

  • a random relation parameter d
  • a random program of length L
  • exact support targets after every prefix step
  • exact final support target after the whole program

Training lengths were 1-8. Evaluation lengths were 4, 8, 12, 16, and 24. Lengths 12, 16, and 24 test recurrent length generalization.

3. Models

Joint Recurrent Executor

The primary model stores a categorical distribution over all (A,B) pairs. Each recurrent step applies a learned transition selected by the current instruction:

  • constant operations use the instruction constant as the selector
  • A cross-register operations use the current B value as the selector
  • B cross-register operations use the current A value as the selector

The transition table has 8 * p * p * p learned logits. For p=31, this is 238,328 transition logits. The model is not handed the modular arithmetic table; it learns the transitions from dense support supervision.

Marginal Recurrent Control

The marginal control uses the same recurrent schedule and transition parameterization, but it stores separate distributions over A and B. On the belief task, both marginals are uniform at initialization, so the relation B=A+d is lost.

Static Compiler Control

The static model receives the relation parameter and program, then predicts the final support in one shot. It has no recurrent execution axis and cannot trade more internal steps for better answers.

4. Metrics

The target is a support set, not a single pair.

  • top1_on_support: whether the highest-probability pair lies inside the exact final support.
  • target_mass: total probability assigned to the exact final support.
  • target_nll: mean negative log probability across support elements.

For modulus 31, a uniform factorized prediction assigns 1/31 = 3.2% mass to the correct support. A perfect prediction assigns 100% mass and has target NLL log(31) = 3.434.

5. Main Result

The modulo-31 joint recurrent executor shows a clean K threshold. Top-1-on-support is 100% exactly once K reaches program length.

Top-1 support heatmap

Target-support mass shows the same structure with a graded distribution-quality signal.

Target mass heatmap

Line curves make the threshold visible by length.

Target mass K curves

Numerically:

Program lengthBest target mass before K>=LFirst K>=LTarget mass at first K>=LTop-1 at first K>=L
414.5%497.8%100.0%
85.7%895.7%100.0%
125.6%1293.6%100.0%
165.2%1691.6%100.0%
243.8%2487.4%100.0%

The held-out lengths 12, 16, and 24 are the key result. The model was trained only on lengths up to 8, but because it learned reusable transitions, it executes longer programs when given enough recurrent steps.

6. Controls

The scaled controls show that the result is not explained by marginal belief tracking or one-shot compilation.

Control comparison

At modulus 31:

ModelL=4L=8L=12L=16L=24
Joint recurrent97.8%95.7%93.6%91.6%87.4%
Marginal recurrent3.2%3.2%3.2%3.2%3.2%
Static compiler7.9%3.5%3.2%3.2%3.2%

The marginal control stays at the 1/31 baseline because it cannot represent the relation between A and B. The static control learns a small short-length signal but does not solve longer programs.

7. Interpretation

This experiment supports a narrow mechanistic claim:

A latent recurrent runtime with joint state can execute correlated belief-state programs, and additional internal steps causally improve performance when those steps correspond to program execution.

The result is stronger than a smooth improvement curve. It has the expected executor signature:

K < L: the runtime has not consumed the whole program -> near-baseline support mass
K >= L: the program has been executed -> high support mass

The marginal control is important because exact single-state programs do not force joint representation. Correlated belief states do: the separate marginals are uniform, while the joint support contains the useful information.

8. Limits

This is still a structured setting.

  • The state is an explicit categorical distribution over (A,B).
  • The model receives dense support supervision.
  • The runtime uses a direct program counter.
  • The operations are modular affine updates.
  • The largest completed run here uses modulus 31, not a much larger state space.

These are not flaws in the result; they define its scope. The experiment shows that the mechanism works when the representation is aligned with the task. It does not show that an unstructured hidden vector will discover the same representation unaided.

9. Next Iterations

The most useful next steps are:

  1. Replace the explicit program counter with attention over instruction tokens and a learned halt/no-op.
  2. Distill the joint categorical belief state into a dense hidden state.
  3. Add observations or queries that require selecting one property of the belief state rather than supervising the full support.
  4. Increase modulus and state size while tracking memory and runtime costs.
  5. Test robustness under noisier supervision where the exact support is not given at every step.

10. Reproducibility

Primary files:

  • Experiment script: ../src/joint_register_executor_experiment.py
  • Analysis script: ../src/analyze_joint_register_executor.py
  • Experiment log: joint_register_executor_experiment_log.md
  • Results directory: ../runs/
  • Analysis directory: ../analysis/

Key run directories:

  • ../runs/main_belief_joint_mod31
  • ../runs/control_belief_marginal_mod31
  • ../runs/control_belief_static_mod31
  • ../runs/pilot_belief_joint_mod11
  • ../runs/control_belief_marginal_mod11
  • ../runs/control_belief_static_mod11

Large checkpoint files are stored outside the experiment bundle under:

Environment:

  • Python 3.12.3
  • PyTorch 2.8.0+cu128
  • GPU: NVIDIA RTX 6000 Ada Generation

11. Bottom Line

The joint-state executor learned to execute correlated belief-state programs. It generalized from training lengths 1-8 to evaluation lengths 12, 16, and 24, with a sharp threshold when internal recurrent steps reached program length. The marginal and static controls stayed near baseline on the scaled task. The core lesson is that recurrent latent execution can scale cleanly with test-time compute, but only when the hidden state can represent the variables and correlations the task actually requires.

Experiment log 18

Show the running log (18 entries)

Objective

Test whether a latent recurrent runtime can execute two-register modular programs that include cross-register operations. The key change is that the hidden state is a categorical distribution over joint register pairs (A, B), rather than separate marginal distributions for A and B.

The experiment is standalone. It asks whether exact algorithmic accuracy rises when the recurrent step budget K reaches the program length L, including on lengths longer than the training horizon.

Hypothesis

If a joint latent state is necessary for cross-register programs, then:

  1. A joint-state recurrent executor should learn exact execution on programs containing both constant and cross-register operations.
  2. Accuracy should be poor when K < L and high when K >= L.
  3. The threshold pattern should hold for held-out lengths beyond training.
  4. A marginal-state recurrent control should struggle because it loses correlations between A and B.
  5. A static one-shot compiler should not show the same K-dependent length generalization.

Task

Programs operate over two registers modulo p.

Operations:

  • A=A+c
  • A=A-c
  • B=B+c
  • B=B-c
  • A=A+B
  • B=B+A
  • A=A-B
  • B=B-A

Each generated example contains random initial registers, a random program, exact trace targets after every prefix, and an exact final pair target.

Models

Joint Recurrent Executor

The hidden state is a categorical distribution over all (A, B) pairs. Each recurrent step applies a learned op-conditioned transition matrix:

  • constant operations use the instruction argument as the transition selector
  • A cross-register operations use current B as the selector
  • B cross-register operations use current A as the selector

This keeps the transition table compact at 8 * p * p * p parameters while retaining joint correlations.

Marginal Recurrent Control

The control uses the same transition parameterization and recurrent schedule, but it stores only separate distributions over A and B. It can model marginal effects but not correlations induced by cross-register operations.

Static Control

The static control embeds the whole program and initial state, then predicts the final pair in one shot.

Planned Sequence

  1. Smoke test at small modulus and a few steps to catch shape/runtime errors.
  2. Pilot joint-state run at small modulus to see whether the mechanism learns.
  3. Matched marginal and static controls.
  4. Main joint-state run at larger modulus with train lengths 1-8 and eval lengths 4/8/12/16/24.
  5. Analyze threshold curves and write a standalone report.

Smoke Test

Run: ../runs/smoke_joint_mod7

Configuration:

  • mode: joint
  • modulus: 7
  • operation family: full
  • train length max: 3
  • train steps: 2
  • eval lengths: 2, 3

Result: the script compiled, trained, evaluated, wrote metrics, and saved checkpoints under ../../../large_artifacts/joint_register_executor/checkpoints/smoke_joint_mod7.

Interpretation: shape logic, trace generation, recurrent updates, metrics, and separated checkpoint writing are functional. The run is intentionally too short and small to test learning.

Pilot 1: Joint Executor, Full Operations, Modulus 11

Run: ../runs/pilot_joint_mod11

Configuration:

  • mode: joint
  • modulus: 11
  • operation family: full
  • train lengths: 1-6
  • eval lengths: 3, 6, 9, 12
  • recurrent budgets: K=0,1,2,3,6,9,12
  • eval examples: 1,024 per length
  • training steps: 300

Result: the joint executor learned rapidly. By step 100 it already showed the intended threshold curve:

  • L=3: 100% exact pair accuracy at K>=3
  • L=6: 100% exact pair accuracy at K>=6
  • L=9: 100% exact pair accuracy at K>=9
  • L=12: 100% exact pair accuracy at K=12

For K < L, pair accuracy stayed near chance, with partial-register accuracy rising as more prefixes were executed.

Step 300 reproduced the same threshold pattern exactly. This is the first positive evidence that the joint latent state can learn reusable transitions for cross-register programs and length-generalize beyond the training horizon.

Control 1: Marginal Executor, Full Operations, Modulus 11

Run: ../runs/control_marginal_mod11

Configuration matched Pilot 1 except mode=marginal.

Result: the marginal executor also reached 100% exact pair accuracy at K>=L, including held-out lengths 9 and 12.

Interpretation: this control changed the experimental diagnosis. With an exact initial pair, the latent state is always a point mass. A marginal state can carry a point-mass A and point-mass B separately, so cross-register operations do not actually require a joint distribution. The current task proves recurrent transition execution, but it does not isolate the need for joint latent state.

Next step: revise the task to use a correlated belief state. The model will receive an initial relation such as B=A+d with A unknown, so the hidden state must track a line of possible (A,B) pairs. The target becomes the final belief distribution after executing the program. A joint executor can represent this distribution; a marginal executor loses the correlation because both marginals are uniform.

Belief-State Task Revision

The revised task initializes a correlated belief state:

B = A + d (mod p), with A unknown

The support contains p possible (A,B) pairs. Programs then transform this support through the same full operation set. The target at each recurrent step is the exact final support distribution, not one sampled pair.

Metrics:

  • top1_on_support: whether the model's highest-probability pair lies inside the correct final support.
  • target_mass: total probability assigned to the correct final support.
  • target_nll: mean negative log probability across support elements.

For modulus 11, a factorized uniform joint gives about 1/11 = 9.1% target mass.

Pilot 2: Joint Belief Executor, Modulus 11

Run: ../runs/pilot_belief_joint_mod11

Configuration:

  • mode: joint
  • task: belief_line
  • modulus: 11
  • operation family: full
  • train lengths: 1-6
  • eval lengths: 3, 6, 9, 12
  • recurrent budgets: K=0,1,2,3,6,9,12
  • eval examples: 1,024 per length
  • training steps: 300

Result at step 300:

  • L=3: K=3 reached 100% top-1-on-support and 97.5% target mass.
  • L=6: K=6 reached 100% top-1-on-support and 95.2% target mass.
  • L=9: K=9 reached 100% top-1-on-support and 92.9% target mass.
  • L=12: K=12 reached 100% top-1-on-support and 90.6% target mass.

For K<L, target mass stayed near the 9.1% factorized/uniform baseline except for partial-prefix cases.

Interpretation: this is the first clean positive result for correlated latent belief execution. The joint state tracks the support and the K threshold generalizes beyond the training horizon.

Control 2: Marginal Belief Executor, Modulus 11

Run: ../runs/control_belief_marginal_mod11

Configuration matched Pilot 2 except mode=marginal.

Result: the marginal control stayed flat throughout training. At step 300, target mass was about 9.1% for every length and every K, with target NLL around 4.796.

Interpretation: this is the expected representational failure. The marginal state sees uniform A and uniform B but cannot store the relation between them.

Control 3: Static Belief Compiler, Modulus 11

Run: ../runs/control_belief_static_mod11

Configuration matched Pilot 2 except mode=static; the static model receives the relation parameter and program, then predicts the final support in one shot.

Result at step 300:

  • L=3: 25.3% top-1-on-support and 18.4% target mass.
  • L=6: 11.0% top-1-on-support and 10.1% target mass.
  • L=9: 9.9% top-1-on-support and 9.1% target mass.
  • L=12: 10.1% top-1-on-support and 9.1% target mass.

Interpretation: the static model learns some short-length signal but does not length-generalize. It does not reproduce the recurrent threshold behavior.

Main Run: Joint Belief Executor, Modulus 31

Run: ../runs/main_belief_joint_mod31

Configuration:

  • mode: joint
  • task: belief_line
  • modulus: 31
  • operation family: full
  • train lengths: 1-8
  • eval lengths: 4, 8, 12, 16, 24
  • recurrent budgets: K=0,1,2,4,8,12,16,24
  • eval examples: 1,024 per length
  • training steps: 500

Result at step 500:

  • L=4: K=4 reached 100% top-1-on-support and 97.8% target mass.
  • L=8: K=8 reached 100% top-1-on-support and 95.7% target mass.
  • L=12: K=12 reached 100% top-1-on-support and 93.6% target mass.
  • L=16: K=16 reached 100% top-1-on-support and 91.6% target mass.
  • L=24: K=24 reached 100% top-1-on-support and 87.4% target mass.

For K<L, target mass stayed near the 1/31 = 3.2% baseline except when a partial prefix overlapped the final support. This is the main positive result.

Scaled Control: Marginal Belief Executor, Modulus 31

Run: ../runs/control_belief_marginal_mod31

Configuration matched the main run except mode=marginal and 200 training steps.

Result: target mass stayed at 3.2% for every length and every K. Target NLL stayed around 6.87, matching the uniform joint baseline.

Interpretation: the marginal recurrent state cannot represent the correlated line support.

Scaled Control: Static Belief Compiler, Modulus 31

Run: ../runs/control_belief_static_mod31

Configuration matched the main run except mode=static and 300 training steps.

Result at step 300:

  • L=4: 10.8% top-1-on-support and 7.9% target mass.
  • L=8: 3.8% top-1-on-support and 3.5% target mass.
  • L=12: 3.0% top-1-on-support and 3.2% target mass.
  • L=16: 2.5% top-1-on-support and 3.2% target mass.
  • L=24: 3.3% top-1-on-support and 3.2% target mass.

Interpretation: the static compiler did not solve the scaled belief task and did not length-generalize beyond the training horizon.

Figures 4

mod31 controls target mass
mod31 controls target mass · analysis/figures/
mod31 target mass heatmap
mod31 target mass heatmap · analysis/figures/
mod31 target mass k curves
mod31 target mass k curves · analysis/figures/
mod31 top1 support heatmap
mod31 top1 support heatmap · analysis/figures/

Data files 24

Result tables and metrics copied from the experiment folder — preview inline or open the raw file.

13 more result files not shown here — browse the full folder on GitHub.

Reproduce

The run commands are documented inside the experiment folder (see the README).

Browse the experiment folder on GitHub ↗