Research log Small Model Experimentation
GitHub

Sampled Query Filter Executor Experiment

Finishedimported 2026-07-12imported · line YActive Evidence AcquisitionStructured Execution and CompilersGitHub ↗
One sampled answer teaches the entire hidden world

The one idea you need

Two unknown numbers sit locked a fixed distance apart. A short program nudges them and drops clues that rule out combinations. To answer any question the model must track every still-possible pair of numbers, though it is graded on just one guessed final answer per problem.

The question

If you only ever grade a model on one possible answer per practice problem, can it still learn the full hidden set of possibilities behind every answer?

What we found

Yes. Graded on just one sampled final answer per problem, the model rebuilt the entire set of still-possible number pairs it was never shown, capturing 94 to 98 percent of it, because holding that full set is the cheapest way to answer varied questions. Simpler models that tracked each number separately, or skipped step-by-step execution, stayed under 6 percent.

Why it matters

When every answer in a task shares one hidden structure, you can supervise only final answers and let the model build that structure itself, but only if its design can hold the correlated whole rather than separate pieces, and you allow enough internal steps.

Hidden set of pairs recovered, never taught94-98%share of the never-shown set of possible number pairs the internal state captured
Same score for the best simpler modelunder 6%controls that track numbers separately or skip step-by-step execution
Stretch beyond training length8 → 24 stepstrained on programs up to 8 steps, still about 95% correct at 24 steps
Longest programs, before vs after enough steps7% → 95%answer accuracy once internal steps reach the program length
On this page
  1. Results at a glance
  2. Overview
  3. Report
    1. Abstract
    2. Lay Summary
    3. 1. Question
    4. 2. Task
    5. 3. Models
    6. 4. Metrics
    7. 5. Main Result
    8. 6. Query Types
    9. 7. Controls
    10. 8. Modulus-11 Diagnostic
    11. 9. Interpretation
    12. 10. Limits
    13. 11. Next Tests
    14. 12. Reproducibility
    15. 13. Bottom Line
  4. Experiment log
  5. Figures
  6. Data files
  7. Reproduce
  8. Related

Results at a glance 6

Answer accuracy on longer programs: full tracker versus two simpler models

How to read

Bars group by program length (4 to 24; models trained only up to 8). Height is probability placed on the correct answer, higher is better. Three bars per length: the full joint tracker, a separate-numbers tracker, and a one-shot no-steps model.

0%25%50%75%100%99.1%47%45.4%length 497.8%30.8%25.7%length 896.6%22.5%13.3%length 1295.8%16.5%8.2%length 1695.1%11.8%5.2%length 24

Takeaway → The full tracker holds above 95 percent even at three times the training length, while both simpler models collapse toward 5 to 12 percent.

Data table
program length (trained on lengths 1-8)joint recurrentmarginal recurrentstatic one-shot
length 499.1%47%45.4%
length 897.8%30.8%25.7%
length 1296.6%22.5%13.3%
length 1695.8%16.5%8.2%
length 2495.1%11.8%5.2%

Numbers from analysis/mod31_control_summary.csv (also analysis/summary.md control table)

Technical framing

Answer accuracy on held-out program lengths: joint recurrent vs controls (mod 31) — With enough internal steps, the joint recurrent model stays above 95% even at 3x training length; both controls collapse.

Accuracy jumps exactly when internal steps reach program length

How to read

The horizontal axis is the number of internal thinking steps allowed; the vertical axis is probability on the correct answer, higher is better. Each line is one program length (4, 8, 12, 16, 24).

0%50%100%150%01020

Takeaway → Every line stays flat and low until the steps reach its own program length, then snaps up to 95 to 99 percent, the signature of genuine step-by-step execution.

Data table
internal recurrent steps Klength 4length 8length 12length 16length 24
035.8%19.8%11.7%8%5.4%
139.1%20.7%12.3%7.7%5.8%
245.9%21.5%12.3%7.9%5%
499.1%25.5%12.6%8.3%5.8%
899.1%97.8%18.5%9.4%5.2%
1299.1%97.8%96.6%16.6%5.3%
1699.1%97.8%96.6%95.8%7.3%
2499.1%97.8%96.6%95.8%95.1%

Numbers from analysis/final_metrics_query_mean.csv (main_joint_mod31, step 1500, mean over query types)

Technical framing

Sharp compute threshold: accuracy jumps exactly when internal steps reach program length — Accuracy stays low until K reaches the program length L, then snaps to ~95-99% -- evidence of step-by-step latent execution.

The never-shown hidden set of pairs is recovered anyway

How to read

Bars group by program length; height is probability placed on the true set of still-possible number pairs, which was withheld from training entirely. Higher is better. Three bars each: full joint tracker, separate-numbers tracker, one-shot model.

0%25%50%75%100%97.8%4.1%5.3%length 496.2%3.4%2.2%length 895.1%2.9%0.7%length 1294.4%2.3%0.3%length 1693.7%1.6%0.2%length 24

Takeaway → The joint tracker recovers 94 to 98 percent of a hidden set it was never taught, while both simpler models stay under 6 percent.

Data table
program lengthjoint recurrentmarginal recurrentstatic one-shot
length 497.8%4.1%5.3%
length 896.2%3.4%2.2%
length 1295.1%2.9%0.7%
length 1694.4%2.3%0.3%
length 2493.7%1.6%0.2%

Numbers from analysis/mod31_control_summary.csv (belief_target_mass column)

Technical framing

Hidden belief state is learned without ever being supervised (mod 31, K >= L) — The full (A,B) belief distribution was withheld from the loss, yet the joint model recovers 94-98% of its mass; controls stay under 6%.

All four question types clear the bar, including the combined ones

How to read

Bars group by unseen program length; height is answer accuracy once internal steps reach program length, higher is better. Four bars per length, one per question type: first number, second number, their sum, and their difference.

0%25%50%75%100%Length 4Length 8Length 12Length 16Length 24

Takeaway → No question type lags: even the harder sum and difference questions stay above 94 percent, so the model holds a genuinely joint state, not two separate ones.

Data table
Held-out program lengthQuery AQuery BQuery A+BQuery A-B
Length 499.3%99.2%98.9%98.8%
Length 898.2%98.1%97.3%97.4%
Length 1297.1%97%96.2%96.2%
Length 1696.4%96.4%95.4%95.2%
Length 2495.6%95.6%94.8%94.7%

Numbers from experiments/sampled_query_filter_executor/analysis/mod31_per_query_threshold_summary.csv

Technical framing

Accuracy at the compute threshold holds across all four query types (mod 31) — No query family lags: single-variable and compound (A+B, A-B) queries all exceed 94% mass once internal steps reach program length.

Hidden-state accuracy climbs during training while chasing the graded answer

How to read

The horizontal axis is training step; the vertical axis is probability on the correct target, higher is better. One line is the graded answer accuracy, the other the never-graded hidden set-of-pairs accuracy.

0%50%100%150%50010001500Query answer mass (supervised)Query answer mass…Hidden belief mass (never supervised)Hidden belief mas…

Takeaway → The hidden line starts near 3 percent while the graded answer sits at 39 percent, then both climb and converge near 98 percent, hidden structure trailing the reward by roughly 200 steps.

Data table
Training stepQuery answer mass (supervised)Hidden belief mass (never supervised)
139.3%2.5%
5040.5%6.7%
10048.8%11.2%
15051.7%12.7%
20051.4%15%
25059%18.9%
30061.3%28.4%
35069.4%40.9%
40076.2%54.9%
45084.1%66.8%
50088%74.6%
55089.3%80.2%
60092.7%85.1%
65094.2%88%
70094.8%89.6%
75095.2%90.6%
80096.1%92.6%
85096.7%93.1%
90096.8%93.6%
95097.1%94.6%
100097.6%95%
105097.9%95.4%
110098%95.7%
115098.2%96.3%
120098.4%96.7%
125098.2%96.7%
130098.5%96.9%
135098.7%97.3%
140098.8%97.5%
145098.8%97.6%
150098.9%97.7%

Numbers from experiments/sampled_query_filter_executor/runs/main_joint_mod31/results.json

Technical framing

Belief-state accuracy rises during training despite never being supervised (mod 31) — Belief mass lags query mass early (2.5% vs 39% at step 1) then converges to 98%, tracking the supervised signal with a delay of ~200 steps.

A separate smaller setup reproduces the same win

How to read

Bars group by program length in an independent smaller-number version of the task; height is answer accuracy at a fixed internal-step budget, higher is better. Three bars each: full joint tracker and two simpler controls.

0%25%50%75%100%99.7%56.3%58.6%Length 399.3%44.4%39.3%Length 699%35.9%24.3%Length 998.8%30.4%17.5%Length 12

Takeaway → The joint tracker holds near 99 percent at every length while both controls decay below 31 percent, so the result is not a fluke of one problem size.

Data table
Held-out program lengthJoint recurrent p=11Marginal recurrent p=11Static p=11
Length 399.7%56.3%58.6%
Length 699.3%44.4%39.3%
Length 999%35.9%24.3%
Length 1298.8%30.4%17.5%

Numbers from experiments/sampled_query_filter_executor/analysis/final_metrics_query_mean.csv

Technical framing

Modulus-11 pilot replicates the mod-31 result: joint recurrent beats both controls — In the independent mod-11 pilot the joint model stays near 99% at every length while marginal and static controls decay below 31%.

In the author’s words from the Report · “Abstract”

Averaged across query types, exact query target mass at the first K>=L step is 99.1%, 97.8%, 96.6%, 95.8%, and 95.1% for lengths 4, 8, 12, 16, and 24. Hidden belief target mass is 97.8%, 96.2%, 95.1%, 94.4%, and 93.7%, despite never being directly supervised. Marginal recurrent and static one-shot controls remain far lower.

Overview

This experiment tests latent recurrent execution over correlated belief states when training supervision is limited to one sampled final query value per example.

Contents

  • src/sampled_query_filter_executor_experiment.py: training and evaluation harness.
  • src/analyze_sampled_query_filter_executor.py: analysis and figure generation.
  • reports/sampled_query_filter_executor_paper.md: standalone writeup.
  • reports/sampled_query_filter_executor_paper.html: standalone HTML report.
  • reports/sampled_query_filter_executor_experiment_log.md: chronological experiment log.
  • runs/: JSON and CSV run outputs.
  • analysis/: generated summaries and figures.
  • checkpoint_manifest.csv: paths and sizes for saved checkpoints.

Checkpoints are written outside the experiment directory under:

../../large_artifacts/sampled_query_filter_executor/checkpoints/

Download this experiment directory for the normal research bundle. Download ../../large_artifacts/sampled_query_filter_executor/ only when saved model weights are needed.

Report

Rendered from reports/sampled_query_filter_executor_paper.md

A controlled experiment on whether one sampled final query value can train hidden-state execution

Abstract

This experiment tests whether a latent recurrent runtime can learn an internal belief-state executor when each training example provides only one sampled final query value. Each example starts with an unknown pair of modular registers constrained by B=A+d (mod p). A program applies arithmetic updates and observation filters. The model receives a query type such as A, B, A+B, or A-B, and the loss is ordinary cross-entropy on one value sampled from the exact final query distribution.

The full final query distribution and the full final belief distribution over (A,B) pairs are withheld from the loss. They are used only for evaluation. On the scaled modulus-31 task, trained on program lengths 1-8 and evaluated on lengths 4, 8, 12, 16, and 24, the joint recurrent model shows a sharp compute threshold. Averaged across query types, exact query target mass at the first K>=L step is 99.1%, 97.8%, 96.6%, 95.8%, and 95.1% for lengths 4, 8, 12, 16, and 24. Hidden belief target mass is 97.8%, 96.2%, 95.1%, 94.4%, and 93.7%, despite never being directly supervised. Marginal recurrent and static one-shot controls remain far lower.

Lay Summary

The model starts from partial knowledge:

B = A + d (mod p)

This relation describes many possible worlds. A program then changes the registers and sometimes filters the possible worlds:

A = A + 7
observe B % 5 = 3
B = B - A
query A + B

Training gives the model only one sampled answer value for the final query. It is not told the full answer distribution, and it is not told the full set of possible (A,B) pairs. The question is whether the model learns the hidden set anyway because that hidden set is the reusable structure needed to answer many sampled queries. In the joint recurrent model, it does.

1. Question

The experiment asks whether sampled-answer supervision can induce a latent recurrent belief-state executor.

The target evidence has five parts:

  1. Query quality should depend on internal step budget K.
  2. The threshold should align with program length L: weak when K<L, strong when K>=L.
  3. The threshold should hold on lengths longer than training.
  4. Hidden belief mass should rise even though the full belief state is not directly supervised.
  5. Controls without joint state or without recurrent execution should fail on held-out lengths.

The setting is deliberately controlled. It is a mechanistic test of whether low-bandwidth final-answer supervision can train reusable latent execution.

2. Task

Programs operate over two registers modulo p.

Initial belief:

{(A, B): B = A + d mod p}

For p=31, the full state space has 961 register pairs. The initial support contains 31 pairs.

Operations:

OperationMeaning
A=A+cadd a constant to A
A=A-csubtract a constant from A
B=B+cadd a constant to B
B=B-csubtract a constant from B
A=A+Badd B into A
B=B+Aadd A into B
A=A-Bsubtract B from A
B=B-Asubtract A from B
OBS_A_BUCKETfilter to states where A % m = r
OBS_B_BUCKETfilter to states where B % m = r

Observation residues are sampled from the live support, so every target support is non-empty. For the scaled run, p=31, observation modulus is 5, and each instruction is an observation with probability 0.3.

Each example samples one final query type:

QueryDistribution being sampled
Afinal distribution of A
Bfinal distribution of B
A_PLUS_Bfinal distribution of A+B mod p
A_MINUS_Bfinal distribution of A-B mod p

The training label is one value sampled from that exact query distribution. The model is trained with one-label cross-entropy. Evaluation still computes the exact final query distribution and exact final belief distribution.

Training used lengths 1-8. Evaluation used lengths 4, 8, 12, 16, and 24. Lengths 12, 16, and 24 test length generalization.

3. Models

Joint Sampled-Query Filter

The primary model stores a categorical distribution over all (A,B) pairs. Each recurrent step reads the next instruction and applies a learned arithmetic transition or learned observation likelihood. The final pair distribution is projected into the selected query distribution, and the loss scores one sampled query value.

The full belief distribution is never used as a training target. It is measured afterward to test whether the model learned a coherent hidden state.

Marginal Sampled-Query Control

The marginal control follows the same recurrent schedule but stores separate distributions over A and B. It can learn some sampled-query signal, but it cannot exactly represent pairwise correlations.

Static Sampled-Query Compiler Control

The static control receives the initial relation and whole program, then predicts a final pair distribution in one pass with a small Transformer encoder. It has no recurrent execution axis.

4. Metrics

The primary metrics evaluate the exact final query distribution:

  • query_target_mass: total probability assigned to the exact query support.
  • query_top1_on_support: whether the most likely queried value is inside the exact query support.
  • query_target_nll: cross-entropy against the exact query distribution.

The audit metrics evaluate the full hidden belief state:

  • belief_target_mass: total pair probability assigned to the exact final (A,B) support.
  • belief_top1_on_support: whether the most likely pair is inside the exact support.
  • belief_target_nll: cross-entropy against the exact final pair distribution.

The hidden-belief metrics are not training objectives.

5. Main Result

The scaled modulus-31 joint recurrent sampled-query filter shows a clean execution threshold. Query mass is low when K<L, then jumps when K reaches program length.

Query top-1 heatmap

Query mass heatmap

The hidden belief audit shows the same threshold, even though the full belief state was not directly supervised.

Hidden belief mass heatmap

The K curves show the threshold by length.

Query mass K curves

Hidden belief mass K curves

Numerically, averaged across query types:

Program lengthBest query mass when K<LBest hidden belief mass when K<LFirst K>=LQuery mass at first K>=LHidden belief mass at first K>=LQuery top-1
445.9%13.8%499.1%97.8%100.0%
825.5%4.0%897.8%96.2%100.0%
1218.5%3.2%1296.6%95.1%100.0%
1616.6%4.1%1695.8%94.4%100.0%
247.3%0.8%2495.1%93.7%100.0%

The held-out lengths are the critical evidence. The model was trained only on lengths up to 8, but it executes lengths 12, 16, and 24 when given enough recurrent steps.

6. Query Types

All four query types work at the execution threshold. At length 24:

QueryQuery mass at K=24Hidden belief mass at K=24
A95.6%93.7%
A_MINUS_B94.7%93.7%
A_PLUS_B94.8%93.8%
B95.6%93.6%

Relational queries matter because they are harder to answer from independent marginals. The joint model solves them along with the direct A and B queries.

7. Controls

The scaled controls show that the result is not explained by shallow sampled-label fitting or one-shot compilation.

Control query mass

Control hidden belief mass

At modulus 31, averaged across query types:

ModelL=4 queryL=8 queryL=12 queryL=16 queryL=24 query
Joint recurrent99.1%97.8%96.6%95.8%95.1%
Marginal recurrent47.0%30.8%22.5%16.5%11.8%
Static compiler45.4%25.7%13.3%8.2%5.2%

Hidden belief target mass separates the models even more sharply:

ModelL=4 beliefL=8 beliefL=12 beliefL=16 beliefL=24 belief
Joint recurrent97.8%96.2%95.1%94.4%93.7%
Marginal recurrent4.1%3.4%2.9%2.3%1.6%
Static compiler5.3%2.2%0.7%0.3%0.2%

The marginal control has recurrence but lacks joint state. The static compiler sees the whole program but lacks recurrent execution. Neither recovers the executor signature.

8. Modulus-11 Diagnostic

A smaller modulus-11 diagnostic used training lengths 1-6 and evaluation lengths 3, 6, 9, and 12. The joint recurrent model reached 98.6-99.7% query mass and 98.2-99.3% hidden belief mass at K=L. The marginal and static controls stayed far lower, especially on held-out lengths. This diagnostic checks that the sampled-answer objective works before scaling the state space to modulus 31.

9. Interpretation

The result supports a specific mechanism claim:

One sampled final query value per example can train a latent recurrent runtime to form and execute a coherent joint belief state when that state is the reusable structure needed to answer varied queries.

The strongest evidence is the combination of:

  1. a sharp K=L threshold,
  2. length generalization beyond the training range,
  3. high hidden-belief mass without direct belief supervision,
  4. failure of marginal and static controls.

The experiment is stronger than training directly on full belief states. The model is rewarded only for one sampled answer value, yet the audited hidden state becomes a high-quality approximation to the full final belief.

10. Limits

This is still a structured setting.

  • The joint model stores an explicit categorical state over (A,B) pairs.
  • The runtime uses a direct program counter.
  • The operation family is modular arithmetic plus bucket observations.
  • The largest completed state space has 961 pairs.
  • Supervision is an exact sample from an exact query distribution, not noisy natural-language feedback.

The experiment shows that sampled-answer supervision can induce the intended latent state when the architecture can represent it. It does not show that an unstructured hidden vector would discover the same state without architectural support.

11. Next Tests

Useful next tests:

  1. Replace the explicit categorical state with a dense latent state and probe how much belief structure remains.
  2. Replace the direct program counter with attention over instruction tokens.
  3. Add a learned halt/no-op policy so the model chooses its compute budget.
  4. Train with noisy sampled labels and measure robustness.
  5. Increase modulus and state size while tracking memory and runtime scaling.

12. Reproducibility

Primary files:

  • Experiment script: ../src/sampled_query_filter_executor_experiment.py
  • Analysis script: ../src/analyze_sampled_query_filter_executor.py
  • Experiment log: sampled_query_filter_executor_experiment_log.md
  • Results directory: ../runs/
  • Analysis directory: ../analysis/
  • Checkpoint manifest: ../checkpoint_manifest.csv

Key run directories:

  • ../runs/main_joint_mod31
  • ../runs/control_marginal_mod31
  • ../runs/control_static_mod31
  • ../runs/pilot_joint_mod11
  • ../runs/control_marginal_mod11
  • ../runs/control_static_mod11

Large checkpoint files are stored outside the experiment bundle under:

../../../large_artifacts/sampled_query_filter_executor/checkpoints/

Environment:

  • Python 3.12.3
  • PyTorch 2.8.0+cu128
  • GPU: NVIDIA RTX 6000 Ada Generation

13. Bottom Line

The joint recurrent sampled-query filter learned to execute arithmetic and observation-filter programs from one sampled final answer value per example. It generalized from training lengths 1-8 to lengths 12, 16, and 24, with a sharp improvement when K reached L. The hidden belief state became accurate even though it was not directly supervised. The marginal and static controls failed on the scaled task.

Experiment log 15

Show the running log (15 entries)

Objective

Test whether a latent recurrent runtime can learn arithmetic and observation-filter execution when each training example provides only one sampled final query value, not the full final query distribution and not the full belief state.

The hidden state is evaluated against the exact final belief distribution over (A, B) pairs, but the training loss only observes a single sampled value for one query:

  • A
  • B
  • A+B mod p
  • A-B mod p

Hypothesis

If sampled-value supervision is sufficient to induce a reusable latent executor, then:

  1. Query performance should improve sharply when K >= L.
  2. The threshold should generalize to held-out lengths longer than training.
  3. Hidden belief mass should rise even though the model never sees full belief targets.
  4. A marginal recurrent control should be weaker because it cannot represent pairwise correlations.
  5. A static one-shot compiler should not reproduce the same length-generalizing K threshold.

Task

Initial belief:

B = A + d (mod p), with A unknown

Programs contain:

  • arithmetic updates over A and B
  • observation filters of the form A % m = r
  • observation filters of the form B % m = r

Observation residues are sampled from the current support, so the target support is never empty.

Each example samples one final query type and one answer value from the exact final query distribution. The model is trained with ordinary one-label cross-entropy on that sampled value. Evaluation still uses exact query distributions and exact hidden belief states.

Models

Joint Sampled-Query Filter

The primary model stores a categorical distribution over all (A,B) pairs. At each recurrent step, it applies the next learned arithmetic transition or learned observation likelihood, then normalizes the belief state. Training loss is computed only after projecting the final pair distribution into the sampled query distribution and scoring the sampled answer value.

Marginal Sampled-Query Control

The marginal control stores separate distributions over A and B. It can learn some marginal answer signal but cannot exactly preserve pairwise correlations.

Static Sampled-Query Compiler Control

The static control receives the initial relation and whole program, then predicts a final pair distribution in one pass. It is trained through the same sampled-value query loss.

Planned Sequence

  1. Smoke test at tiny modulus.
  2. Pilot joint sampled-query filter at small modulus.
  3. Matched marginal and static controls at small modulus.
  4. Main scaled run at modulus 31.
  5. Matched scaled controls.
  6. Aggregate metrics, generate figures, and write a standalone report.

Smoke Test

Run: ../runs/smoke_joint_mod7

Command:

python experiments/sampled_query_filter_executor/src/sampled_query_filter_executor_experiment.py --mode joint --modulus 7 --observe_mod 3 --observe_prob 0.4 --train_max_len 3 --eval_lengths 2,3 --eval_k 0,1,2,3 --eval_query_types all --train_steps 2 --batch_size 16 --eval_batch_size 16 --eval_examples 32 --log_every 1 --eval_every 1 --lr 0.01 --output_dir experiments/sampled_query_filter_executor/runs/smoke_joint_mod7 --checkpoint_dir large_artifacts/sampled_query_filter_executor/checkpoints/smoke_joint_mod7

Status: complete.

Result: the script compiled, trained with sampled query labels, evaluated exact query and hidden-belief metrics, wrote metrics, and saved checkpoints under ../../../large_artifacts/sampled_query_filter_executor/checkpoints/smoke_joint_mod7.

Interpretation: sampled-value targets, exact query evaluation, hidden-belief audit metrics, evaluation aggregation, and separated checkpoint writing are functional. The run is intentionally too short to test learning.

Pilot 1: Joint Sampled-Query Filter, Modulus 11

Run: ../runs/pilot_joint_mod11

Command:

python experiments/sampled_query_filter_executor/src/sampled_query_filter_executor_experiment.py --mode joint --modulus 11 --observe_mod 4 --observe_prob 0.3 --train_max_len 6 --eval_lengths 3,6,9,12 --eval_k 0,1,2,3,6,9,12 --eval_query_types all --train_steps 1200 --batch_size 512 --eval_batch_size 512 --eval_examples 512 --log_every 50 --eval_every 600 --lr 0.03 --output_dir experiments/sampled_query_filter_executor/runs/pilot_joint_mod11 --checkpoint_dir large_artifacts/sampled_query_filter_executor/checkpoints/pilot_joint_mod11

Status: complete.

Interim result at step 600:

  • L=3: at K=3, all query types reached 100% top-1-on-support, 98.8-99.2% query target mass, and 97.6-97.8% hidden belief target mass.
  • L=6: at K=6, all query types reached 100% top-1-on-support, 97.4-98.1% query target mass, and 96.0-96.2% hidden belief target mass.
  • L=9: at K=9, all query types reached 100% top-1-on-support, 96.1-97.2% query target mass, and 94.9-95.0% hidden belief target mass.
  • L=12: at K=12, all query types reached 100% top-1-on-support, 95.4-96.5% query target mass, and 94.3-94.4% hidden belief target mass.

For K<L, query target mass and hidden belief mass were much lower. This interim result shows that one sampled answer value per example is enough to induce a coherent latent belief state at small modulus.

Final result at step 1200:

  • L=3: at K=3, all query types reached 100% top-1-on-support, 99.6-99.7% query target mass, and 99.3% hidden belief target mass.
  • L=6: at K=6, all query types reached 100% top-1-on-support, 99.2-99.4% query target mass, and 98.7-98.9% hidden belief target mass.
  • L=9: at K=9, all query types reached 100% top-1-on-support, 98.8-99.1% query target mass, and 98.4-98.5% hidden belief target mass.
  • L=12: at K=12, all query types reached 100% top-1-on-support, 98.6-99.0% query target mass, and 98.2-98.3% hidden belief target mass.

Interpretation: sampled-value supervision is sufficient to train the joint recurrent executor at small modulus, including held-out lengths.

Control 1: Marginal Sampled-Query Filter, Modulus 11

Run: ../runs/control_marginal_mod11

Command:

python experiments/sampled_query_filter_executor/src/sampled_query_filter_executor_experiment.py --mode marginal --modulus 11 --observe_mod 4 --observe_prob 0.3 --train_max_len 6 --eval_lengths 3,6,9,12 --eval_k 0,1,2,3,6,9,12 --eval_query_types all --train_steps 1200 --batch_size 512 --eval_batch_size 512 --eval_examples 512 --log_every 100 --eval_every 1200 --lr 0.03 --output_dir experiments/sampled_query_filter_executor/runs/control_marginal_mod11 --checkpoint_dir large_artifacts/sampled_query_filter_executor/checkpoints/control_marginal_mod11

Status: complete.

Result at step 1200:

  • L=3: at sufficient K, query target mass reached 43.6-66.2%; hidden belief mass was 10.3-11.5%.
  • L=6: at sufficient K, query target mass reached 35.3-52.0%; hidden belief mass was 11.2-11.9%.
  • L=9: at sufficient K, query target mass reached 28.1-44.2%; hidden belief mass was 10.2-11.1%.
  • L=12: at sufficient K, query target mass reached 23.0-37.6%; hidden belief mass was 9.3-10.5%.

Interpretation: the marginal model learns some sampled-answer signal but does not recover a coherent joint belief state.

Control 2: Static Sampled-Query Compiler, Modulus 11

Run: ../runs/control_static_mod11

Command:

python experiments/sampled_query_filter_executor/src/sampled_query_filter_executor_experiment.py --mode static --modulus 11 --observe_mod 4 --observe_prob 0.3 --train_max_len 6 --eval_lengths 3,6,9,12 --eval_k 0,1,2,3,6,9,12 --eval_query_types all --train_steps 1200 --batch_size 512 --eval_batch_size 512 --eval_examples 512 --log_every 100 --eval_every 1200 --lr 0.001 --dim 128 --heads 4 --compiler_layers 2 --output_dir experiments/sampled_query_filter_executor/runs/control_static_mod11 --checkpoint_dir large_artifacts/sampled_query_filter_executor/checkpoints/control_static_mod11

Status: complete.

Result at step 1200:

  • L=3: query target mass reached 48.0-66.6%; hidden belief mass was 15.1-17.2%.
  • L=6: query target mass reached 32.7-45.3%; hidden belief mass was 8.2-9.4%.
  • L=9: query target mass reached 21.2-28.1%; hidden belief mass was 3.1-3.8%.
  • L=12: query target mass reached 16.4-18.1%; hidden belief mass was 2.0-2.4%.

Interpretation: the static compiler learns short-length signal but does not length-generalize and does not recover a coherent hidden belief state.

Main Run: Joint Sampled-Query Filter, Modulus 31

Run: ../runs/main_joint_mod31

Command:

python experiments/sampled_query_filter_executor/src/sampled_query_filter_executor_experiment.py --mode joint --modulus 31 --observe_mod 5 --observe_prob 0.3 --train_max_len 8 --eval_lengths 4,8,12,16,24 --eval_k 0,1,2,4,8,12,16,24 --eval_query_types all --train_steps 1500 --batch_size 512 --eval_batch_size 512 --eval_examples 512 --log_every 50 --eval_every 750 --lr 0.03 --output_dir experiments/sampled_query_filter_executor/runs/main_joint_mod31 --checkpoint_dir large_artifacts/sampled_query_filter_executor/checkpoints/main_joint_mod31

Status: complete.

Interim result at step 750:

  • L=4: at K=4, all query types reached 100% top-1-on-support, 95.0-97.2% query target mass, and 91.2-91.3% hidden belief target mass.
  • L=8: at K=8, all query types reached 100% top-1-on-support, 89.7-92.6% query target mass, and 84.6-85.4% hidden belief target mass.
  • L=12: at K=12, all query types reached 100% top-1-on-support, 85.1-89.2% query target mass, and 80.9-81.4% hidden belief target mass.
  • L=16: at K=16, all query types reached 100% top-1-on-support, 82.1-85.8% query target mass, and 78.3-79.0% hidden belief target mass.
  • L=24: at K=24, all query types reached 100% top-1-on-support, 78.9-83.8% query target mass, and 75.5-76.9% hidden belief target mass.

Final result at step 1500:

  • L=4: at K=4, all query types reached 100% top-1-on-support, 98.8-99.3% query target mass, and 97.8-97.9% hidden belief target mass.
  • L=8: at K=8, all query types reached 100% top-1-on-support, 97.3-98.2% query target mass, and 96.1-96.3% hidden belief target mass.
  • L=12: at K=12, all query types reached 100% top-1-on-support, 96.2-97.1% query target mass, and 95.0-95.1% hidden belief target mass.
  • L=16: at K=16, all query types reached 100% top-1-on-support, 95.2-96.4% query target mass, and 94.2-94.5% hidden belief target mass.
  • L=24: at K=24, all query types reached 100% top-1-on-support, 94.7-95.6% query target mass, and 93.6-93.8% hidden belief target mass.

For K<L, the same examples remained close to baseline. At L=24, the best pre-threshold query target mass stayed below 10% for all four query types, while K=24 recovered the exact sampled-query answer support and most of the full hidden belief mass.

Scaled Control: Marginal Sampled-Query Filter, Modulus 31

Run: ../runs/control_marginal_mod31

Command:

python experiments/sampled_query_filter_executor/src/sampled_query_filter_executor_experiment.py --mode marginal --modulus 31 --observe_mod 5 --observe_prob 0.3 --train_max_len 8 --eval_lengths 4,8,12,16,24 --eval_k 0,1,2,4,8,12,16,24 --eval_query_types all --train_steps 1500 --batch_size 512 --eval_batch_size 512 --eval_examples 512 --log_every 100 --eval_every 1500 --lr 0.03 --output_dir experiments/sampled_query_filter_executor/runs/control_marginal_mod31 --checkpoint_dir large_artifacts/sampled_query_filter_executor/checkpoints/control_marginal_mod31

Status: complete.

Result at step 1500:

  • L=4: at sufficient K, query target mass reached 36.2-56.2%; hidden belief mass was 3.8-4.3%.
  • L=8: at sufficient K, query target mass reached 23.9-36.8%; hidden belief mass was 3.3-3.5%.
  • L=12: at sufficient K, query target mass reached 16.4-29.0%; hidden belief mass was 2.8-3.0%.
  • L=16: at sufficient K, query target mass reached 11.7-21.5%; hidden belief mass was 2.2-2.6%.
  • L=24: at sufficient K, query target mass reached 8.5-15.0%; hidden belief mass was 1.6%.

Interpretation: the marginal recurrent control learns some sampled-query signal but does not recover the joint hidden state and does not approach the joint recurrent model on held-out lengths.

Scaled Control: Static Sampled-Query Compiler, Modulus 31

Run: ../runs/control_static_mod31

Command:

python experiments/sampled_query_filter_executor/src/sampled_query_filter_executor_experiment.py --mode static --modulus 31 --observe_mod 5 --observe_prob 0.3 --train_max_len 8 --eval_lengths 4,8,12,16,24 --eval_k 0,1,2,4,8,12,16,24 --eval_query_types all --train_steps 1500 --batch_size 512 --eval_batch_size 512 --eval_examples 512 --log_every 100 --eval_every 1500 --lr 0.001 --dim 128 --heads 4 --compiler_layers 2 --output_dir experiments/sampled_query_filter_executor/runs/control_static_mod31 --checkpoint_dir large_artifacts/sampled_query_filter_executor/checkpoints/control_static_mod31

Status: complete.

Result at step 1500:

  • L=4: query target mass reached 36.7-54.0%; hidden belief mass was 4.8-5.9%.
  • L=8: query target mass reached 20.8-29.7%; hidden belief mass was 2.1-2.3%.
  • L=12: query target mass reached 11.5-15.0%; hidden belief mass was 0.6-0.7%.
  • L=16: query target mass reached 7.6-9.2%; hidden belief mass was 0.3-0.4%.
  • L=24: query target mass reached 5.0-5.4%; hidden belief mass was 0.2%.

Interpretation: the static compiler learns some short-length sampled-query signal but does not length-generalize and does not recover the hidden joint belief state.

Figures 7

mod31 belief mass heatmap
mod31 belief mass heatmap · analysis/figures/
mod31 belief mass k curves
mod31 belief mass k curves · analysis/figures/
mod31 controls belief mass
mod31 controls belief mass · analysis/figures/
mod31 controls query mass
mod31 controls query mass · analysis/figures/
mod31 query mass heatmap
mod31 query mass heatmap · analysis/figures/
mod31 query mass k curves
mod31 query mass k curves · analysis/figures/
mod31 query top1 heatmap
mod31 query top1 heatmap · analysis/figures/

Data files 23

Result tables and metrics copied from the experiment folder — preview inline or open the raw file.

Reproduce

The run commands are documented inside the experiment folder (see the README).

Browse the experiment folder on GitHub ↗