Query Filter Executor Experiment
The one idea you need
Two hidden numbers start tied by a fixed relationship, so many value-pairs stay possible. Like a detective narrowing a suspect board as each clue lands, the model crosses off pairs one instruction at a time, though it is graded only on the final answer.
The question
If you only ever grade a model on its final answer, will it still bother to build the full step-by-step reasoning the problem actually needs?
What we found
Yes, and that is the surprise. Trained only to name one final answer, the model taught itself to run each instruction in order and track every still-possible pair of the two hidden numbers. Given enough internal steps, it stayed about 96% accurate on programs three times longer than any it trained on, while simpler-memory and one-pass rivals collapsed to roughly 5 to 12%.
Why it matters
If a task needs an internal working state, cheap answer-only supervision can grow one, and you need not label every step. But only when the architecture can hold that state and you give it enough internal steps to match the task's depth.
On this page
Results at a glance 7
How to read
Bars group by program length, 4 up to 24, with training only up to 8. Height is the share of confidence on the correct answer; taller is better. Three models: joint step-by-step, separate-memory, and one-pass.
Takeaway → The joint bars hold near 96% at every length while both others sink toward 5 to 12%, so only step-by-step execution survives past the training range.
Data table
| program length (trained on lengths 1-8) | joint recurrent | marginal recurrent | static compiler |
|---|---|---|---|
| length 4 | 99.3% | 48% | 46.9% |
| length 8 | 98.2% | 32.3% | 27.2% |
| length 12 | 97.3% | 22.4% | 13.1% |
| length 16 | 96.7% | 16.8% | 8.4% |
| length 24 | 96.2% | 12.1% | 5.1% |
Numbers from analysis/summary.md (Modulus-31 Control Summary table; runs/*_mod31/results.json)
Technical framing
Query accuracy on held-out program lengths: joint belief state vs controls — Only the joint recurrent model keeps ~96%+ query accuracy out to length 24, three times its training length; both controls collapse.
How to read
The horizontal axis is the number of internal steps allowed; the vertical axis is confidence on the correct answer, higher better. Each line is one program length, from 4 to 24.
Takeaway → Every line stays near the floor, then leaps above 96% exactly when the step count reaches its own program length, the fingerprint of true step-by-step execution.
Data table
| internal recurrent steps K | L=4 | L=8 | L=12 | L=16 | L=24 |
|---|---|---|---|---|---|
| 0 | 34.4% | 18.9% | 11.3% | 7.6% | 5.5% |
| 1 | 42.1% | 20.2% | 11.2% | 7% | 5.3% |
| 2 | 51.4% | 22.9% | 11.7% | 7.6% | 4.7% |
| 4 | 99.4% | 28.6% | 13.9% | 7.1% | 6.5% |
| 8 | 99.4% | 98.5% | 25.4% | 11.1% | 5.8% |
| 12 | 99.4% | 98.5% | 97.8% | 21.6% | 6.3% |
| 16 | 99.4% | 98.5% | 97.8% | 97.1% | 10% |
| 24 | 99.4% | 98.5% | 97.8% | 97.1% | 96.5% |
Technical framing
Execution threshold: query accuracy jumps exactly when step budget K reaches program length L — Accuracy stays near chance until K reaches program length L, then jumps above 96% — the signature of step-by-step latent execution.
How to read
Bars group by program length, 4 to 24. Height is the confidence placed on the exact correct set of hidden number-pairs, an audit never used in training; higher is better. Three models compared.
Takeaway → The joint model holds around 95% on this untrained internal state while both others sit under 7%, proving it discovered the hidden pairs unprompted.
Data table
| program length | joint recurrent | marginal recurrent | static compiler |
|---|---|---|---|
| length 4 | 98.3% | 4.2% | 6.3% |
| length 8 | 97% | 3.9% | 2.6% |
| length 12 | 96.1% | 3.1% | 0.7% |
| length 16 | 95.5% | 2.4% | 0.3% |
| length 24 | 95% | 1.8% | 0.2% |
Numbers from analysis/summary.md (Modulus-31 Control Summary table; runs/*_mod31/results.json)
Technical framing
Hidden belief state becomes accurate without ever being supervised — Trained only on final query answers, the joint model still builds a near-exact (A,B) belief state; controls put under 7% mass on it.
How to read
Bars for four question types: each hidden number alone, their sum, and their difference, at the longest program with a full step budget. Two measures each, answer accuracy and hidden-state accuracy; higher is better.
Takeaway → The sum and difference questions, which separate-memory models cannot answer, match the single-number questions near 96%, confirming genuine paired reasoning.
Data table
| query type | query mass | hidden belief mass |
|---|---|---|
| A | 96.5% | 94.9% |
| B | 96.6% | 94.9% |
| A+B | 95.7% | 94.9% |
| A-B | 95.8% | 95.1% |
Numbers from analysis/summary.md (Main Modulus-31 Per-Query Threshold Summary, length 24)
Technical framing
All four query types work at the execution threshold (length 24, K=24) — Relational queries A+B and A-B, which marginals cannot answer, match the direct register queries at the hardest held-out length.
How to read
The horizontal axis is internal step count; the vertical axis is confidence on the correct answer, higher is better. One line per program length, 3 to 12, using a smaller number system.
Takeaway → Each line jumps above 98% right when steps reach its own length, confirming the threshold is not a quirk of one setup.
Data table
| Step budget K | Length 3 | Length 6 | Length 9 | Length 12 |
|---|---|---|---|---|
| 0 | 46.7% | 29.2% | 21.8% | 17.4% |
| 1 | 55.7% | 31.8% | 22.9% | 17.8% |
| 2 | 70.1% | 36% | 24.2% | 18.2% |
| 3 | 99.7% | 42.6% | 25.1% | 18.7% |
| 6 | 99.7% | 99.4% | 36.7% | 22.3% |
| 9 | 99.7% | 99.4% | 99.1% | 34.8% |
| 12 | 99.7% | 99.4% | 99.1% | 98.8% |
Technical framing
Threshold effect replicates at modulus 11 (pilot run) — The same K-equals-L jump appears in an independent pilot with a smaller modulus (11), so the threshold is not specific to modulus 31.
How to read
Bars group by program length, 4 to 24. Height is the best answer accuracy reachable while internal steps stay below the program length. Four question types: each number, their sum, their difference; taller means easier to fake.
Takeaway → Sum and difference bars sit below the single-number bars, and all collapse as programs lengthen, so compound questions cannot be faked without real execution.
Data table
| Program length | A | B | A plus B | A minus B |
|---|---|---|---|---|
| 4 | 51.4% | 51.9% | 43.9% | 42.3% |
| 8 | 28.6% | 29.9% | 22.1% | 23.4% |
| 12 | 25.4% | 23.9% | 13.3% | 14.3% |
| 16 | 21.6% | 20.5% | 11.1% | 10% |
| 24 | 10% | 9.8% | 6.2% | 5.7% |
Numbers from experiments/query_filter_executor/analysis/summary.md (Main Modulus-31 Per-Query Threshold Summary table)
Technical framing
Below the threshold, compound queries are hardest to guess — With too few steps, single-variable queries stay easier to guess than compound ones, and all guessing collapses as programs get longer.
How to read
Bars group by program length, 4 to 24. Height is answer accuracy when internal steps equal the program length. Two snapshots: halfway through training and at the end; higher is better.
Takeaway → By the halfway mark short programs already peak; the second half almost entirely lifts the longest programs, from about 87% to 96%.
Data table
| Program length | Mid-training (step 500) | Final (step 1000) |
|---|---|---|
| 4 | 97.5% | 99.4% |
| 8 | 94.3% | 98.5% |
| 12 | 91.4% | 97.8% |
| 16 | 89.2% | 97.1% |
| 24 | 86.9% | 96.5% |
Technical framing
The second half of training mainly sharpens long-program answers — By step 500 the model already answers at the execution threshold; steps 500 to 1000 mostly close the remaining gap on the longest programs.
In the author’s words from the Report · “Abstract”
Averaged across query types, query target mass at the first K>=L step is 99.3%, 98.2%, 97.3%, 96.7%, and 96.2% for lengths 4, 8, 12, 16, and 24. Hidden belief target mass, despite not being directly supervised, is 98.3%, 97.0%, 96.1%, 95.5%, and 95.0%. Marginal recurrent and static one-shot controls remain far lower on held-out lengths.
Overview
This experiment tests latent recurrent execution over correlated belief states when training supervision is limited to final query answers.
Contents
src/query_filter_executor_experiment.py: training and evaluation harness.src/analyze_query_filter_executor.py: analysis and figure generation.reports/query_filter_executor_paper.md: standalone writeup.reports/query_filter_executor_paper.html: standalone HTML report.reports/query_filter_executor_experiment_log.md: chronological experiment log.runs/: JSON and CSV run outputs.analysis/: generated summaries and figures.checkpoint_manifest.csv: paths and sizes for saved checkpoints.
Checkpoints are written outside the experiment directory under:
../../large_artifacts/query_filter_executor/checkpoints/Download this experiment directory for the normal research bundle. Download ../../large_artifacts/query_filter_executor/ only when saved model weights are needed.
Report
Rendered from reports/query_filter_executor_paper.md
A controlled experiment on recurrent hidden-state execution from final-answer supervision
Abstract
This experiment tests whether a latent recurrent runtime can learn an internal belief-state executor when training supervision is limited to final query answers. Each example starts with an unknown pair of modular registers constrained by B=A+d (mod p). A program applies arithmetic updates and observation filters. The model is trained only to answer a sampled final query: A, B, A+B, or A-B. The exact final belief distribution over (A,B) pairs is held out from the loss and used only as an audit.
The primary joint recurrent model stores a categorical distribution over register pairs and applies one learned update per internal step. On the scaled modulus-31 task, trained on program lengths 1-8 and evaluated on lengths 4, 8, 12, 16, and 24, it shows a sharp execution threshold. Averaged across query types, query target mass at the first K>=L step is 99.3%, 98.2%, 97.3%, 96.7%, and 96.2% for lengths 4, 8, 12, 16, and 24. Hidden belief target mass, despite not being directly supervised, is 98.3%, 97.0%, 96.1%, 95.5%, and 95.0%. Marginal recurrent and static one-shot controls remain far lower on held-out lengths.
Lay Summary
The model starts with a relation, not exact values:
B = A + d (mod p)That relation describes many possible worlds. The program changes the registers and sometimes adds observations:
A = A + 7
observe B % 5 = 3
B = B - A
query A + BTraining only tells the model the final answer distribution for the query. It does not tell the model the full set of possible (A,B) pairs. The question is whether the model learns the hidden set anyway because doing so is the reusable way to answer many queries. In the joint recurrent model, it does.
1. Question
The experiment asks whether final-answer supervision can induce a latent recurrent belief-state executor.
The desired evidence has four parts:
- Query accuracy should depend on internal step budget
K. - The threshold should align with program length
L: weak whenK<L, strong whenK>=L. - Hidden belief mass should also rise, even though the full belief state is not directly supervised.
- Controls without joint state or without recurrent execution should fail on held-out lengths.
The setting is deliberately controlled. The target is not broad natural-language reasoning; it is a mechanistic test of whether query supervision can train reusable latent execution.
2. Task
Programs operate over two registers modulo p.
Initial belief:
{(A, B): B = A + d mod p}For p=31, the full state space has 961 register pairs. The initial support contains 31 pairs.
Operations:
| Operation | Meaning |
|---|---|
A=A+c | add a constant to A |
A=A-c | subtract a constant from A |
B=B+c | add a constant to B |
B=B-c | subtract a constant from B |
A=A+B | add B into A |
B=B+A | add A into B |
A=A-B | subtract B from A |
B=B-A | subtract A from B |
OBS_A_BUCKET | filter to states where A % m = r |
OBS_B_BUCKET | filter to states where B % m = r |
Observation residues are sampled from the live support, so every target support is non-empty. For the scaled run, p=31, observation modulus is 5, and each instruction is an observation with probability 0.3.
Each example samples one final query:
| Query | Target |
|---|---|
A | final distribution of A |
B | final distribution of B |
A_PLUS_B | final distribution of A+B mod p |
A_MINUS_B | final distribution of A-B mod p |
Training used lengths 1-8. Evaluation used lengths 4, 8, 12, 16, and 24. Lengths 12, 16, and 24 test length generalization.
3. Models
Joint Recurrent Query Filter
The primary model stores a categorical distribution over all (A,B) pairs. Each recurrent step reads the next instruction and applies the corresponding learned update. The final pair distribution is projected into the sampled query distribution, and the loss is cross-entropy on that query distribution only.
The full belief distribution is never used as a training target. It is measured afterward to see whether the model learned a coherent hidden state.
Marginal Recurrent Control
The marginal control follows the same recurrent schedule but stores separate distributions over A and B. It can learn some marginal query signal, but it cannot exactly represent pairwise correlations.
Static Compiler Control
The static control receives the initial relation and whole program, then predicts a final pair distribution in one pass with a small Transformer encoder. It has no recurrent execution axis.
4. Metrics
The primary metrics evaluate the sampled final query:
query_target_mass: total probability assigned to the exact query support.query_top1_on_support: whether the most likely queried value is inside the exact query support.query_target_nll: cross-entropy against the exact query distribution.
The audit metrics evaluate the full hidden belief state:
belief_target_mass: total pair probability assigned to the exact final(A,B)support.belief_top1_on_support: whether the most likely pair is inside the exact support.belief_target_nll: cross-entropy against the exact final pair distribution.
The hidden-belief metrics are not training objectives.
5. Main Result
The scaled modulus-31 joint recurrent query filter shows a clean execution threshold. Query mass is low when K<L, then jumps when K reaches program length.
The hidden belief audit shows the same threshold, even though the full belief state was not directly supervised.
The K curves show the threshold by length.
Numerically, averaged across query types:
| Program length | Best query mass when K<L | Best hidden belief mass when K<L | First K>=L | Query mass at first K>=L | Hidden belief mass at first K>=L | Query top-1 |
|---|---|---|---|---|---|---|
| 4 | 47.4% | 14.1% | 4 | 99.3% | 98.3% | 100.0% |
| 8 | 26.0% | 4.1% | 8 | 98.2% | 97.0% | 100.0% |
| 12 | 19.2% | 3.5% | 12 | 97.3% | 96.1% | 100.0% |
| 16 | 15.8% | 3.1% | 16 | 96.7% | 95.5% | 100.0% |
| 24 | 7.6% | 0.4% | 24 | 96.2% | 95.0% | 100.0% |
The held-out lengths are the critical evidence. The model was trained only on lengths up to 8, but it executes lengths 12, 16, and 24 when given enough recurrent steps.
6. Query Types
All four query types work at the execution threshold. At length 24:
| Query | Query mass at K=24 | Hidden belief mass at K=24 |
|---|---|---|
A | 96.5% | 94.9% |
A_MINUS_B | 95.8% | 95.1% |
A_PLUS_B | 95.7% | 94.9% |
B | 96.6% | 94.9% |
Relational queries are important because they are harder to answer from independent marginals. The joint model solves them along with the direct A and B queries.
7. Controls
The scaled controls show that the result is not explained by shallow query fitting or one-shot compilation.
At modulus 31, averaged across query types:
| Model | L=4 query | L=8 query | L=12 query | L=16 query | L=24 query |
|---|---|---|---|---|---|
| Joint recurrent | 99.3% | 98.2% | 97.3% | 96.7% | 96.2% |
| Marginal recurrent | 48.0% | 32.3% | 22.4% | 16.8% | 12.1% |
| Static compiler | 46.9% | 27.2% | 13.1% | 8.4% | 5.1% |
Hidden belief target mass separates the models even more sharply:
| Model | L=4 belief | L=8 belief | L=12 belief | L=16 belief | L=24 belief |
|---|---|---|---|---|---|
| Joint recurrent | 98.3% | 97.0% | 96.1% | 95.5% | 95.0% |
| Marginal recurrent | 4.2% | 3.9% | 3.1% | 2.4% | 1.8% |
| Static compiler | 6.3% | 2.6% | 0.7% | 0.3% | 0.2% |
The marginal control has recurrence but lacks joint state. The static compiler sees the whole program but lacks recurrent execution. Neither recovers the executor signature.
8. Modulus-11 Diagnostic
A smaller modulus-11 diagnostic used the same query-supervised task with training lengths 1-6 and evaluation lengths 3, 6, 9, and 12. The joint recurrent model reached 98.5-99.7% query mass and 98.0-99.2% hidden belief mass at K=L. The marginal and static controls stayed far lower, especially on held-out lengths. This diagnostic confirmed the training objective before the scaled run.
9. Interpretation
The result supports a specific mechanism claim:
Final query supervision can train a latent recurrent runtime to form and execute a coherent joint belief state when that state is the reusable structure needed to answer varied queries.
The strongest evidence is not just high final accuracy. It is the combination of:
- a sharp
K=Lthreshold, - length generalization beyond the training range,
- high hidden-belief mass without direct belief supervision,
- failure of marginal and static controls.
This is a stronger test than training directly on full belief states. The hidden state is useful enough that the model discovers it under a lower-bandwidth objective.
10. Limits
This is still a structured setting.
- The joint model stores an explicit categorical distribution over
(A,B). - The runtime uses a direct program counter.
- The operation family is modular arithmetic plus bucket observations.
- The largest completed state space has 961 pairs.
- Query supervision is exact; it is not noisy natural-language feedback.
The experiment shows that query-only supervision can induce the intended latent state when the architecture can represent it. It does not show that an unstructured hidden vector would discover the same state without architectural support.
11. Next Iterations
Useful next tests:
- Replace the explicit categorical state with a dense latent state and probe how much belief structure remains.
- Replace the direct program counter with attention over instruction tokens.
- Add a learned halt/no-op policy so the model chooses its compute budget.
- Train on sparse scalar rewards or sampled query values instead of exact query distributions.
- Increase modulus and state size while tracking memory and runtime scaling.
12. Reproducibility
Primary files:
- Experiment script:
../src/query_filter_executor_experiment.py - Analysis script:
../src/analyze_query_filter_executor.py - Experiment log:
query_filter_executor_experiment_log.md - Results directory:
../runs/ - Analysis directory:
../analysis/ - Checkpoint manifest:
../checkpoint_manifest.csv
Key run directories:
../runs/main_joint_mod31../runs/control_marginal_mod31../runs/control_static_mod31../runs/pilot_joint_mod11../runs/control_marginal_mod11../runs/control_static_mod11
Large checkpoint files are stored outside the experiment bundle under:
../../../large_artifacts/query_filter_executor/checkpoints/Environment:
- Python 3.12.3
- PyTorch 2.8.0+cu128
- GPU: NVIDIA RTX 6000 Ada Generation
13. Bottom Line
The joint recurrent query filter learned to execute arithmetic and observation-filter programs from final-query supervision alone. It generalized from training lengths 1-8 to lengths 12, 16, and 24, with a sharp improvement when K reached L. The hidden belief state became accurate even though it was not directly supervised. The marginal and static controls failed on the scaled task.
Experiment log 15
Show the running log (15 entries)
Objective
Test whether a latent recurrent runtime can learn to execute arithmetic and observation-filter programs when it is trained only from final query distributions, not from dense belief-state supervision.
The hidden state is still evaluated as a full belief distribution over (A, B) pairs, but the training loss only asks the model to answer one query about the final state:
ABA+B mod pA-B mod p
Hypothesis
If final-query supervision is sufficient to induce a reusable latent executor, then:
- Query accuracy and query target mass should be low when
K < Land high whenK >= L. - The threshold should generalize to held-out lengths longer than training.
- The latent belief state should become coherent even though it is not directly supervised.
- A marginal recurrent control should be weaker, especially on relational queries.
- A static one-shot compiler should not reproduce the same length-generalizing K threshold.
Task
Initial belief:
B = A + d (mod p), with A unknownPrograms contain:
- arithmetic updates over
AandB - observation filters of the form
A % m = r - observation filters of the form
B % m = r
Observation residues are sampled from the current support, so the target support is never empty.
Each example samples one final query type. The query target is the exact distribution of the queried value under the final belief state.
Models
Joint Query Filter
The primary model stores a categorical distribution over all (A,B) pairs. At each recurrent step, it applies the next learned arithmetic transition or learned observation likelihood, then normalizes the belief state. Training loss is computed only after projecting the final pair distribution into the sampled query distribution.
Marginal Query Filter Control
The marginal control stores separate distributions over A and B. It can answer some marginal queries but cannot exactly preserve pairwise correlations.
Static Query Compiler Control
The static control receives the initial relation and whole program, then predicts a final pair distribution in one pass. It is trained through the same final query loss.
Planned Sequence
- Smoke test at tiny modulus.
- Pilot joint query filter at small modulus.
- Matched marginal and static controls at small modulus.
- Main scaled run at modulus 31.
- Matched scaled controls.
- Aggregate metrics, generate figures, and write a standalone report.
Smoke Test
Run: ../runs/smoke_joint_mod7
Command:
python experiments/query_filter_executor/src/query_filter_executor_experiment.py --mode joint --modulus 7 --observe_mod 3 --observe_prob 0.4 --train_max_len 3 --eval_lengths 2,3 --eval_k 0,1,2,3 --eval_query_types all --train_steps 2 --batch_size 16 --eval_batch_size 16 --eval_examples 32 --log_every 1 --eval_every 1 --lr 0.01 --output_dir experiments/query_filter_executor/runs/smoke_joint_mod7 --checkpoint_dir large_artifacts/query_filter_executor/checkpoints/smoke_joint_mod7Result: the script compiled, trained, evaluated all query types, wrote metrics, and saved checkpoints under ../../../large_artifacts/query_filter_executor/checkpoints/smoke_joint_mod7.
Interpretation: query targets, query projection, hidden-belief audit metrics, evaluation aggregation, and separated checkpoint writing are functional. The run is intentionally too short to test learning.
Pilot 1: Joint Query Filter, Modulus 11
Run: ../runs/pilot_joint_mod11
Command:
python experiments/query_filter_executor/src/query_filter_executor_experiment.py --mode joint --modulus 11 --observe_mod 4 --observe_prob 0.3 --train_max_len 6 --eval_lengths 3,6,9,12 --eval_k 0,1,2,3,6,9,12 --eval_query_types all --train_steps 800 --batch_size 512 --eval_batch_size 512 --eval_examples 512 --log_every 50 --eval_every 400 --lr 0.03 --output_dir experiments/query_filter_executor/runs/pilot_joint_mod11 --checkpoint_dir large_artifacts/query_filter_executor/checkpoints/pilot_joint_mod11Result at step 800:
L=3: atK=3, all query types reached 100% top-1-on-support, 99.6-99.7% query target mass, and 99.2% hidden belief target mass.L=6: atK=6, all query types reached 100% top-1-on-support, 99.1-99.4% query target mass, and 98.6-98.7% hidden belief target mass.L=9: atK=9, all query types reached 100% top-1-on-support, 98.7-99.1% query target mass, and 98.3% hidden belief target mass.L=12: atK=12, all query types reached 100% top-1-on-support, 98.5-98.9% query target mass, and 98.0-98.1% hidden belief target mass.
For K<L, query target mass stayed far lower and hidden belief mass stayed low. This is a positive result: final-query supervision induced a coherent latent belief state.
Control 1: Marginal Query Filter, Modulus 11
Run: ../runs/control_marginal_mod11
Command:
python experiments/query_filter_executor/src/query_filter_executor_experiment.py --mode marginal --modulus 11 --observe_mod 4 --observe_prob 0.3 --train_max_len 6 --eval_lengths 3,6,9,12 --eval_k 0,1,2,3,6,9,12 --eval_query_types all --train_steps 800 --batch_size 512 --eval_batch_size 512 --eval_examples 512 --log_every 100 --eval_every 800 --lr 0.03 --output_dir experiments/query_filter_executor/runs/control_marginal_mod11 --checkpoint_dir large_artifacts/query_filter_executor/checkpoints/control_marginal_mod11Result at step 800:
L=3: atK=3, query target mass reached 47.3-66.0%; hidden belief mass was 10.5-11.3%.L=6: atK=6, query target mass reached 35.6-52.0%; hidden belief mass was 11.2-11.5%.L=9: atK=9, query target mass reached 28.2-43.3%; hidden belief mass was 10.1-11.3%.L=12: atK=12, query target mass reached 23.6-38.3%; hidden belief mass was 9.3-10.7%.
Interpretation: the marginal model learns some query signal but does not recover a coherent joint belief state.
Control 2: Static Query Compiler, Modulus 11
Run: ../runs/control_static_mod11
Command:
python experiments/query_filter_executor/src/query_filter_executor_experiment.py --mode static --modulus 11 --observe_mod 4 --observe_prob 0.3 --train_max_len 6 --eval_lengths 3,6,9,12 --eval_k 0,1,2,3,6,9,12 --eval_query_types all --train_steps 800 --batch_size 512 --eval_batch_size 512 --eval_examples 512 --log_every 100 --eval_every 800 --lr 0.001 --dim 128 --heads 4 --compiler_layers 2 --output_dir experiments/query_filter_executor/runs/control_static_mod11 --checkpoint_dir large_artifacts/query_filter_executor/checkpoints/control_static_mod11Result at step 800:
L=3: query target mass reached 52.6-65.9%; hidden belief mass was 16.0-18.5%.L=6: query target mass reached 33.3-45.5%; hidden belief mass was 8.5-9.3%.L=9: query target mass reached 22.0-28.6%; hidden belief mass was 3.5-3.7%.L=12: query target mass reached 17.2-19.1%; hidden belief mass was 2.1-2.5%.
Interpretation: the static compiler learns short-length signal but does not length-generalize and does not recover a coherent final belief state.
Main Run: Joint Query Filter, Modulus 31
Run: ../runs/main_joint_mod31
Command:
python experiments/query_filter_executor/src/query_filter_executor_experiment.py --mode joint --modulus 31 --observe_mod 5 --observe_prob 0.3 --train_max_len 8 --eval_lengths 4,8,12,16,24 --eval_k 0,1,2,4,8,12,16,24 --eval_query_types all --train_steps 1000 --batch_size 512 --eval_batch_size 512 --eval_examples 512 --log_every 50 --eval_every 500 --lr 0.03 --output_dir experiments/query_filter_executor/runs/main_joint_mod31 --checkpoint_dir large_artifacts/query_filter_executor/checkpoints/main_joint_mod31Status: complete.
Interim result at step 500:
L=4: atK=4, all query types reached 100% top-1-on-support, 96.4-97.9% query target mass, and 93.5-93.6% hidden belief target mass.L=8: atK=8, all query types reached 100% top-1-on-support, 92.3-94.6% query target mass, and 88.8-89.0% hidden belief target mass.L=12: atK=12, all query types reached 100% top-1-on-support, 88.7-91.4% query target mass, and 85.1-85.9% hidden belief target mass.L=16: atK=16, all query types reached 100% top-1-on-support, 86.5-89.2% query target mass, and 83.2-83.9% hidden belief target mass.L=24: atK=24, all query types reached 100% top-1-on-support, 83.8-86.9% query target mass, and 81.1-81.8% hidden belief target mass.
For K<L, query and hidden-belief mass stayed much lower. This interim result already shows length-generalizing recurrent execution under query-only supervision.
Final result at step 1000:
L=4: atK=4, all query types reached 100% top-1-on-support, 99.1-99.4% query target mass, and 98.3% hidden belief target mass.L=8: atK=8, all query types reached 100% top-1-on-support, 97.9-98.5% query target mass, and 97.0% hidden belief target mass.L=12: atK=12, all query types reached 100% top-1-on-support, 96.8-97.8% query target mass, and 96.0-96.2% hidden belief target mass.L=16: atK=16, all query types reached 100% top-1-on-support, 96.4-97.1% query target mass, and 95.4-95.6% hidden belief target mass.L=24: atK=24, all query types reached 100% top-1-on-support, 95.7-96.6% query target mass, and 94.9-95.1% hidden belief target mass.
For K<L, the final run preserved the sharp threshold: long programs remained near baseline until the recurrent budget reached the program length.
Scaled Control: Marginal Query Filter, Modulus 31
Run: ../runs/control_marginal_mod31
Command:
python experiments/query_filter_executor/src/query_filter_executor_experiment.py --mode marginal --modulus 31 --observe_mod 5 --observe_prob 0.3 --train_max_len 8 --eval_lengths 4,8,12,16,24 --eval_k 0,1,2,4,8,12,16,24 --eval_query_types all --train_steps 1000 --batch_size 512 --eval_batch_size 512 --eval_examples 512 --log_every 100 --eval_every 1000 --lr 0.03 --output_dir experiments/query_filter_executor/runs/control_marginal_mod31 --checkpoint_dir large_artifacts/query_filter_executor/checkpoints/control_marginal_mod31Status: complete.
Result at step 1000:
L=4: atK=4, query target mass reached 37.0-57.7%; hidden belief mass was 4.0-4.5%.L=8: atK=8, query target mass reached 26.2-39.4%; hidden belief mass was 3.7-4.1%.L=12: atK=12, query target mass reached 17.0-27.9%; hidden belief mass was 2.9-3.2%.L=16: atK=16, query target mass reached 12.5-21.1%; hidden belief mass was 2.3-2.5%.L=24: atK=24, query target mass reached 8.5-15.7%; hidden belief mass was 1.6-2.1%.
Interpretation: the marginal control answers some easy marginal signal but does not recover the joint state and does not approach the joint recurrent model on held-out lengths.
Scaled Control: Static Query Compiler, Modulus 31
Run: ../runs/control_static_mod31
Command:
python experiments/query_filter_executor/src/query_filter_executor_experiment.py --mode static --modulus 31 --observe_mod 5 --observe_prob 0.3 --train_max_len 8 --eval_lengths 4,8,12,16,24 --eval_k 0,1,2,4,8,12,16,24 --eval_query_types all --train_steps 1000 --batch_size 512 --eval_batch_size 512 --eval_examples 512 --log_every 100 --eval_every 1000 --lr 0.001 --dim 128 --heads 4 --compiler_layers 2 --output_dir experiments/query_filter_executor/runs/control_static_mod31 --checkpoint_dir large_artifacts/query_filter_executor/checkpoints/control_static_mod31Status: complete.
Result at step 1000:
L=4: query target mass reached 38.1-53.9%; hidden belief mass was 6.0-6.7%.L=8: query target mass reached 23.2-32.9%; hidden belief mass was 2.4-2.9%.L=12: query target mass reached 11.7-14.7%; hidden belief mass was 0.7-0.8%.L=16: query target mass reached 8.1-9.1%; hidden belief mass was 0.3-0.4%.L=24: query target mass reached 4.9-5.4%; hidden belief mass was 0.2%.
Interpretation: the static compiler learns some short-length signal but collapses on longer programs and does not recover the latent belief state.
Figures 7
Data files 23
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
analysis/all_metrics_long.csv165 kBanalysis/final_metrics_long.csv112 kBanalysis/final_metrics_query_mean.csv27 kBanalysis/mod31_control_summary.csv2.3 kBanalysis/mod31_per_query_threshold_summary.csv1.6 kBanalysis/mod31_threshold_summary.csv829 Bruns/control_marginal_mod11/metrics_step00800.csv16 kBruns/control_marginal_mod11/results.json67 kBruns/control_marginal_mod31/metrics_step01000.csv23 kBruns/control_marginal_mod31/results.json94 kBruns/control_static_mod11/metrics_step00800.csv2.5 kBruns/control_static_mod11/results.json14 kBruns/control_static_mod31/metrics_step01000.csv3.1 kBruns/control_static_mod31/results.json17 kBruns/main_joint_mod31/metrics_step00500.csv22 kBruns/main_joint_mod31/metrics_step01000.csv22 kBruns/main_joint_mod31/results.json184 kBruns/pilot_joint_mod11/metrics_step00400.csv15 kBruns/pilot_joint_mod11/metrics_step00800.csv15 kBruns/pilot_joint_mod11/results.json130 kBruns/smoke_joint_mod7/metrics_step00001.csv4.1 kBruns/smoke_joint_mod7/metrics_step00002.csv4.1 kBruns/smoke_joint_mod7/results.json36 kB
Reproduce
The run commands are documented inside the experiment folder (see the README).