Research log Small Model Experimentation
GitHub

Qwen Python-Shaped Silent Executor

Finishedimported 2026-07-12imported · line YStructured Execution and CompilersGitHub ↗
Private thinking slots did no real thinking

The one idea you need

The model runs a short arithmetic program using a fixed set of blank, silent hidden steps, then reports only the final answer with no work shown. The check: shuffle those steps. If scrambling their order changes nothing, no real step-by-step computing ever happened inside them.

The question

Can a small model learn to run little multi-step programs entirely in its head, using private silent thinking steps, instead of writing out its work as text?

What we found

No. The silent hidden steps never cleared single digits: about 8% at best on the simplest programs, 4.5% on unseen longer ones, barely above a zero untrained model and near the 3% you would get by guessing. The tell: scrambling the steps' order worked just as well, so no real step-by-step computing happened. Even visible written reasoning, which hit 62.5% on the shortest programs, crashed to 0% as they grew longer.

Why it matters

Do not expect extra hidden thinking tokens to buy a small model real multi-step execution. Always run a scrambled-order control: if shuffling the hidden steps does not hurt accuracy, you added raw capacity, not reasoning. Test on unseen program lengths, not trained ones.

Written reasoning, short then long programs62.5% → 0%accuracy writing steps out: shortest programs versus longer ones
Silent thinking, simplest versus unseen-longer programs8% → 4.5%best case on trained-simple programs versus unseen longer ones
Scrambled versus ordered steps, longest programs4.9% vs 4.5%shuffling the hidden steps beat keeping them ordered
Tokens spent per answer~750 vs 1written-out reasoning versus silent hidden steps, per problem
On this page
  1. Results at a glance
  2. Overview
  3. Report
    1. Summary
    2. Setup
    3. Frozen Probe And CoT Baseline
    4. K Scaling
    5. Held-Out Composition
    6. Token Cost
    7. Training Dynamics
    8. Interpretation Guide
    9. Artifacts
  4. Experiment log
  5. Figures
  6. Data files
  7. Reproduce
  8. Related

Results at a glance 7

Accuracy by program length: written steps versus silent hidden steps

How to read

Bars group by program length (4, 8, 16, 24 steps). Height is final-answer accuracy; taller is better. Three bars per group: explicit written-out reasoning, silent ordered hidden steps, and silent shuffled-order hidden steps.

0%20%40%60%80%62.5%8%5.9%length 425%6.6%5.9%length 80%3.8%4.5%length 160%3.5%3.5%length 24

Takeaway → Written reasoning towers at 62.5% on the shortest programs then falls to zero by length 16, while both silent arms stay stuck in single digits everywhere, nearly identical.

Data table
program lengththinking CoT (frozen)silent ordered (K=8)silent shuffled (K=8)
length 462.5%8%5.9%
length 825%6.6%5.9%
length 160%3.8%4.5%
length 240%3.5%3.5%

Numbers from report table (Frozen Probe And CoT Baseline + K Scaling), reports/qwen_python_shaped_silent_executor_report.md

Technical framing

Final accuracy by program length: explicit CoT vs silent latent compute (K=8) — Silent latent compute never leaves the single-digit floor at any length, while explicit CoT works at length 4 then collapses by length 16.

More hidden steps on unseen long programs: ordered versus scrambled

How to read

On unseen 32-step programs. The horizontal axis is the number of private hidden compute steps granted (4 to 32); the vertical axis is final-answer accuracy, higher is better. One line ordered, one scrambled-order.

2%3%4%5%6%102030silent orderedsilent shuffled

Takeaway → Both lines wander between about 2% and 5% with no upward trend as steps increase; the scrambled control even peaks highest at 4.9%, so ordered steps buy nothing.

Data table
private latent compute positions (K)silent orderedsilent shuffled
43.1%3.5%
84.5%2.4%
164.5%4.9%
244.5%4.9%
324.5%4.5%

Numbers from report table (K Scaling, eval_len32), reports/qwen_python_shaped_silent_executor_report.md

Technical framing

K-scaling on held-out length-32 programs: ordered vs shuffled latent positions — The shuffled-order control tracks ordered compute within noise (peaking higher at 4.9%), so extra latent positions add capacity, not sequential execution.

More hidden steps on unfamiliar operation mixes does not help

How to read

On programs combining operations never mixed during training. The horizontal axis is the number of private hidden compute steps (4 to 32); the vertical axis is final-answer accuracy, higher is better. Ordered versus scrambled-order steps.

0%1%2%3%4%102030silent orderedsilent shuffled

Takeaway → Both lines hover near 3% at every step count, dipping as low as 1%; piling on hidden compute never recovers execution when the operation mix is new.

Data table
private latent compute positions (K)silent orderedsilent shuffled
41%2.8%
83.1%2.8%
163.1%3.1%
243.1%3.5%
323.1%3.1%

Numbers from report table (K Scaling, eval_comp24), reports/qwen_python_shaped_silent_executor_report.md

Technical framing

Held-out operation composition (length 24): more latent compute does not help — On unseen operation compositions both arms flatline near 3% at every K, so added latent compute does not recover execution.

How much accuracy swings from the training random seed alone

How to read

Bars group by unseen split (length 32, composition 16, composition 24). Height is final-answer accuracy, higher is better. Three bars per group, one for each of three training seeds.

0%2.5%5%7.5%10%Length 32Length 326.2%4.2%3.1%Composition 16Composition 163.1%8.3%2.1%Composition 24Composition 244.2%4.2%1%

Takeaway → Just by changing the random seed, accuracy swings from about 1% to over 8% across these splits, and more than quadruples on a single split, a lottery as large as any effect the method produces.

Data table
Held-out evaluation splitSeed 101Seed 202Seed 303
Length 326.2%4.2%3.1%
Composition 163.1%8.3%2.1%
Composition 244.2%4.2%1%

Numbers from experiments/qwen_python_shaped_silent_executor/reports/metrics.csv

Technical framing

Seed-to-seed spread on held-out splits (silent ordered, K=8) — Training-seed variance (1-8%) is as large as any latent-compute effect, so no single seed's gain is meaningful.

Trained silent models barely clear a zero untrained baseline

How to read

Bars group by five unseen splits. Height is final-answer accuracy, higher is better. Three bars per group: the untrained frozen model, the trained ordered arm, and the trained scrambled arm.

0%2%4%6%Length 16Length 160%3.8%3.8%Length 24Length 240%3.5%3.5%Length 32Length 320%4.5%4.5%Composition 16Composition 160%4.5%4.9%Composition 24Composition 240%3.1%3.1%

Takeaway → The untrained baseline sits at exactly zero; both trained arms add only 3 to 5 points and match each other everywhere, so training bought almost nothing.

Data table
Held-out evaluation splitFrozen direct likelihoodSilent ordered (trained)Silent shuffled (trained)
Length 160%3.8%3.8%
Length 240%3.5%3.5%
Length 320%4.5%4.5%
Composition 160%4.5%4.9%
Composition 240%3.1%3.1%

Numbers from experiments/qwen_python_shaped_silent_executor/reports/summary_by_arm.csv

Technical framing

Trained silent arms barely clear a zero frozen baseline on held-out splits — Silent training adds only 3-5 points over a 0% frozen baseline, and shuffled latent positions match ordered ones everywhere.

Tokens spent per answer: written steps versus silent hidden steps

How to read

Bars group by program length (4, 8, 16, 24). Height is mean output tokens per problem; fewer is cheaper. Two bars: explicit written-out reasoning versus silent hidden-step execution.

02004006008007461Length 47681Length 87681Length 167681Length 24

Takeaway → Written reasoning emits roughly 750 tokens per answer against 1 for silent, yet still hits 0% on the longer programs, so the cost bought no durable accuracy.

Data table
Program lengthExplicit CoT (frozen)Silent latent (trained)
Length 47461
Length 87681
Length 167681
Length 247681

Numbers from experiments/qwen_python_shaped_silent_executor/runs/main_python_shaped_silent_executor_v1/frozen_probe_metrics.csv

Technical framing

Token cost per answer: explicit CoT vs silent latent compute — CoT emits ~750x more tokens per answer yet still drops to 0% accuracy on lengths 16 and 24.

Training loss never settles over 500 steps

How to read

The horizontal axis is training step (1 to 500); the vertical axis is total training loss, lower is better. Three lines, one per training seed. A learning model would trend steadily downward.

56789200400Seed 303Seed 101Seed 202

Takeaway → All three curves stay high and jagged between 6 and 8 with no downward trend, showing the silent-execution objective was never actually learned.

Data table
Training stepSeed 101Seed 202Seed 303
16.887.077.22
1006.087.756.27
2006.946.225.84
3006.666.978.11
4006.596.676.89
5006.496.297.27

Numbers from experiments/qwen_python_shaped_silent_executor/reports/training_log.csv

Technical framing

Training loss never converges over 500 steps (all three seeds) — Loss stays high and noisy through 500 steps for every seed, so the silent-executor objective was never actually learned.

In the author’s words from the Report · “Summary”

This standalone experiment tests whether a Qwen 4B model can execute Python-shaped mini-programs with private latent compute positions instead of emitting an explicit execution trace. Best silent ordered length-32 accuracy was 6.2% at K=8. The result is a controlled negative for the tested silent-execution recipe: ordered latent compute reaches 8.0% on trained-length-4 programs and 4.5% on held-out length-32 programs, with held-out-composition length-24 at 3.1%. The shuffled-compute control matches or exceeds the ordered arm, so the K curve does not support private sequential execution. Thinking CoT reached 62.5% on length 4 with 746.5 emitted tokens, but fell to 0.0% on length 24 with 768.0 emitted tokens. … Read the full result →

Overview

Standalone experiment for testing whether a Qwen 4B model can learn to execute Python-shaped mini-programs using private latent compute positions instead of emitting an explicit chain-of-thought trace.

Small artifacts live here:

experiments/qwen_python_shaped_silent_executor/
  src/        experiment and reporting code
  runs/       per-run configs, metrics, logs
  reports/    standalone Markdown/HTML report and figures

Large artifacts are stored separately:

large_artifacts/qwen_python_shaped_silent_executor/
  checkpoints/  QLoRA adapters and heads
  caches/       optional cached tensors

Primary report:

reports/qwen_python_shaped_silent_executor_report.md
reports/qwen_python_shaped_silent_executor_report.html

Report

Rendered from reports/qwen_python_shaped_silent_executor_report.md

Summary

This standalone experiment tests whether a Qwen 4B model can execute Python-shaped mini-programs with private latent compute positions instead of emitting an explicit execution trace.

Best silent ordered length-32 accuracy was 6.2% at K=8.

The result is a controlled negative for the tested silent-execution recipe: ordered latent compute reaches 8.0% on trained-length-4 programs and 4.5% on held-out length-32 programs, with held-out-composition length-24 at 3.1%. The shuffled-compute control matches or exceeds the ordered arm, so the K curve does not support private sequential execution.

Thinking CoT reached 62.5% on length 4 with 746.5 emitted tokens, but fell to 0.0% on length 24 with 768.0 emitted tokens.

On length 32, ordered silent compute peaked at 4.5% (K=8, state accuracy 5.5%), while shuffled compute peaked at 4.9% (K=16).

The primary decision criterion is held-out length and held-out operation composition. Accuracy at trained lengths is not sufficient evidence of execution; the silent arm must generalize beyond the lengths and operation compositions used during posttraining. The shuffled-compute arm uses the same number of private latent positions but randomizes their order, separating ordered latent computation from extra capacity.

Setup

  • Base model: Qwen/Qwen3-4B.
  • Train lengths: 4,8,16.
  • Held-out lengths: 24,32.
  • Value range: integers 0..31.
  • Private compute positions: 4,8,16,24,32.
  • Training seeds: 101,202,303.
  • Large artifacts: /workspace/large_artifacts/qwen_python_shaped_silent_executor.

Frozen Probe And CoT Baseline

The frozen probe scores final answers directly with numeric continuation likelihood. The CoT baseline generates explicit intermediate states and is scored on both final answer and state accuracy.

armsplitknfinal_accuracystate_accuracymean_output_tokens
frozen_direct_likelihoodeval_len40160.0%n/a1
frozen_direct_likelihoodeval_len80160.0%n/a1
frozen_direct_likelihoodeval_len160160.0%n/a1
frozen_direct_likelihoodeval_len240160.0%n/a1
frozen_direct_likelihoodeval_len320160.0%n/a1
frozen_direct_likelihoodeval_comp160160.0%n/a1
frozen_direct_likelihoodeval_comp240160.0%n/a1
frozen_cot_generateeval_len4-1862.5%50.0%746.5
frozen_cot_generateeval_len8-1825.0%0.0%768
frozen_cot_generateeval_len16-180.0%2.3%768
frozen_cot_generateeval_len24-180.0%0.0%768

K Scaling

armsplitkfinal_accuracystate_accuracyn
silent_orderedeval_comp1642.1%6.2%96
silent_orderedeval_comp1684.5%5.8%96
silent_orderedeval_comp16164.5%4.8%96
silent_orderedeval_comp16244.5%4.8%96
silent_orderedeval_comp16324.5%4.8%96
silent_orderedeval_comp2441.0%5.3%96
silent_orderedeval_comp2483.1%5.4%96
silent_orderedeval_comp24163.1%5.5%96
silent_orderedeval_comp24243.1%5.1%96
silent_orderedeval_comp24323.1%5.1%96
silent_orderedeval_len1644.2%6.5%96
silent_orderedeval_len1683.8%6.0%96
silent_orderedeval_len16163.8%5.7%96
silent_orderedeval_len16243.8%5.6%96
silent_orderedeval_len16323.8%5.6%96
silent_orderedeval_len2444.2%8.0%96
silent_orderedeval_len2483.5%6.5%96
silent_orderedeval_len24163.5%5.6%96
silent_orderedeval_len24243.5%5.4%96
silent_orderedeval_len24323.5%5.4%96
silent_orderedeval_len3243.1%6.4%96
silent_orderedeval_len3284.5%5.5%96
silent_orderedeval_len32164.5%5.3%96
silent_orderedeval_len32244.5%4.8%96
silent_orderedeval_len32324.5%5.0%96
silent_orderedeval_len446.2%8.2%96
silent_orderedeval_len488.0%8.2%96
silent_orderedeval_len4168.0%8.2%96
silent_orderedeval_len4248.0%8.2%96
silent_orderedeval_len4328.0%8.2%96
silent_orderedeval_len847.3%6.4%96
silent_orderedeval_len886.6%6.2%96
silent_orderedeval_len8166.6%6.2%96
silent_orderedeval_len8246.6%6.3%96
silent_orderedeval_len8326.6%6.3%96
silent_shuffledeval_comp1642.8%5.8%96
silent_shuffledeval_comp1683.1%4.7%96
silent_shuffledeval_comp16164.5%4.9%96
silent_shuffledeval_comp16244.2%4.7%96
silent_shuffledeval_comp16324.9%5.0%96
silent_shuffledeval_comp2442.8%5.5%96
silent_shuffledeval_comp2482.8%5.0%96
silent_shuffledeval_comp24163.1%5.0%96
silent_shuffledeval_comp24243.5%4.9%96
silent_shuffledeval_comp24323.1%4.7%96
silent_shuffledeval_len1644.5%5.5%96
silent_shuffledeval_len1684.5%4.6%96
silent_shuffledeval_len16163.8%4.9%96
silent_shuffledeval_len16243.8%4.8%96
silent_shuffledeval_len16323.8%5.3%96
silent_shuffledeval_len2444.5%4.9%96
silent_shuffledeval_len2483.5%5.3%96
silent_shuffledeval_len24163.1%5.0%96
silent_shuffledeval_len24242.8%4.9%96
silent_shuffledeval_len24323.5%4.9%96
silent_shuffledeval_len3243.5%5.2%96
silent_shuffledeval_len3282.4%4.5%96
silent_shuffledeval_len32164.9%4.7%96
silent_shuffledeval_len32244.9%4.6%96
silent_shuffledeval_len32324.5%4.7%96
silent_shuffledeval_len446.6%6.2%96
silent_shuffledeval_len485.9%6.5%96
silent_shuffledeval_len4167.3%5.6%96
silent_shuffledeval_len4247.6%6.6%96
silent_shuffledeval_len4328.0%5.1%96
silent_shuffledeval_len845.6%4.9%96
silent_shuffledeval_len885.9%4.9%96
silent_shuffledeval_len8166.9%6.1%96
silent_shuffledeval_len8246.2%5.9%96
silent_shuffledeval_len8326.9%5.9%96

Accuracy by length

K curve on length 32

Held-Out Composition

armseedsplitkfinal_accuracystate_accuracymean_output_tokens
silent_ordered101eval_comp1642.1%6.2%1
silent_ordered101eval_comp1683.1%5.5%1
silent_ordered101eval_comp16163.1%4.3%1
silent_ordered101eval_comp16243.1%4.3%1
silent_ordered101eval_comp16323.1%4.3%1
silent_shuffled101eval_comp1642.1%4.4%1
silent_shuffled101eval_comp1683.1%4.4%1
silent_shuffled101eval_comp16163.1%4.2%1
silent_shuffled101eval_comp16243.1%4.4%1
silent_shuffled101eval_comp16323.1%4.4%1
silent_ordered101eval_comp2441.0%4.4%1
silent_ordered101eval_comp2484.2%5.7%1
silent_ordered101eval_comp24164.2%5.5%1
silent_ordered101eval_comp24244.2%5.1%1
silent_ordered101eval_comp24324.2%5.1%1
silent_shuffled101eval_comp2442.1%4.2%1
silent_shuffled101eval_comp2484.2%5.9%1
silent_shuffled101eval_comp24164.2%5.5%1
silent_shuffled101eval_comp24244.2%5.0%1
silent_shuffled101eval_comp24324.2%4.9%1
silent_ordered202eval_comp1642.1%5.7%1
silent_ordered202eval_comp1688.3%5.6%1
silent_ordered202eval_comp16168.3%4.3%1
silent_ordered202eval_comp16248.3%4.2%1
silent_ordered202eval_comp16328.3%4.3%1
silent_shuffled202eval_comp1644.2%5.7%1
silent_shuffled202eval_comp1684.2%4.0%1
silent_shuffled202eval_comp16168.3%4.6%1
silent_shuffled202eval_comp16248.3%4.0%1
silent_shuffled202eval_comp16328.3%4.2%1
silent_ordered202eval_comp2441.0%5.7%1
silent_ordered202eval_comp2484.2%4.8%1
silent_ordered202eval_comp24164.2%5.1%1
silent_ordered202eval_comp24244.2%4.8%1
silent_ordered202eval_comp24324.2%4.8%1
silent_shuffled202eval_comp2444.2%5.7%1
silent_shuffled202eval_comp2483.1%3.5%1
silent_shuffled202eval_comp24164.2%4.4%1
silent_shuffled202eval_comp24244.2%5.2%1
silent_shuffled202eval_comp24324.2%4.8%1
silent_ordered303eval_comp1642.1%6.5%1
silent_ordered303eval_comp1682.1%6.4%1
silent_ordered303eval_comp16162.1%5.8%1
silent_ordered303eval_comp16242.1%5.7%1
silent_ordered303eval_comp16322.1%5.7%1
silent_shuffled303eval_comp1642.1%7.3%1
silent_shuffled303eval_comp1682.1%5.7%1
silent_shuffled303eval_comp16162.1%5.9%1
silent_shuffled303eval_comp16241.0%5.9%1
silent_shuffled303eval_comp16323.1%6.5%1
silent_ordered303eval_comp2441.0%5.7%1
silent_ordered303eval_comp2481.0%5.7%1
silent_ordered303eval_comp24161.0%5.9%1
silent_ordered303eval_comp24241.0%5.3%1
silent_ordered303eval_comp24321.0%5.3%1
silent_shuffled303eval_comp2442.1%6.5%1
silent_shuffled303eval_comp2481.0%5.7%1
silent_shuffled303eval_comp24161.0%5.1%1
silent_shuffled303eval_comp24242.1%4.5%1
silent_shuffled303eval_comp24321.0%4.5%1
frozen_direct_likelihood-1eval_comp1600.0%n/a1
frozen_direct_likelihood-1eval_comp2400.0%n/a1

Held-out composition accuracy

Token Cost

Accuracy versus token cost

Training Dynamics

seedstepkshuffle_trainlossfinal_lossstate_lossquick_best_final_accuracy
10111606.87753.45483.422711.6%
1011001606.084862.882493.202374.7%
1012003206.944513.63563.308914.7%
1013003206.661693.310263.351434.7%
101400806.592443.114153.478299.4%
101500806.485723.261023.22474.7%
20213207.074353.604263.470097.8%
2021001607.747873.97143.776474.7%
202200806.224883.201133.023756.2%
2023001606.973213.546423.42683.1%
2024003206.67443.369983.304424.7%
2025001606.287513.088853.198664.7%
3031807.217153.617733.599424.7%
303100406.271213.3132.9582110.9%
303200805.838253.01052.827756.2%
303300808.107234.235743.871494.7%
303400806.89293.369543.523363.1%
303500807.266733.65193.614826.2%

Training dynamics

Interpretation Guide

A positive result requires the ordered silent arm to improve with K on longer programs while the shuffled-compute control does not show the same curve. It also requires held-out length and held-out composition performance to remain strong; otherwise the model has learned a distribution-specific transducer rather than an executable procedure.

Artifacts

Experiment log 2

Show the running log (2 entries)

Design

This standalone experiment tests silent/internal execution on a Python-shaped mini-language. Programs are executable by a deterministic harness, but prompts present them as readable Python-like code. The model is evaluated in three roles:

  • frozen direct final-answer scoring;
  • explicit chain-of-thought generation with intermediate state reporting;
  • trained private latent compute with direct final/state heads.

The decisive comparisons are held-out program length, held-out operation composition, K-private-compute scaling, and a shuffled-compute control.

Large trainable artifacts are stored under /workspace/large_artifacts/qwen_python_shaped_silent_executor.

Iteration Notes

  1. smoke_python_shaped_silent_executor_v1 tested the first 128-value version. It completed end to end, but raw prompting made the CoT baseline ramble and truncate.

  2. smoke_python_shaped_silent_executor_v2 switched model-facing prompts to Qwen chat formatting. CoT outputs became concise but remained weak.

  3. smoke_python_shaped_silent_executor_v3 enabled Qwen thinking mode for the CoT baseline. This made CoT a stronger and more expensive baseline.

  4. pilot_python_shaped_silent_executor_v1 and v2 showed that the 128-value target space was too diffuse: QLoRA learned frequent-answer priors, not execution, and ordered latent positions did not beat shuffled positions.

  5. The substrate was revised to small integers 0..31 with smaller operation constants. smoke_python_shaped_silent_executor_v4 validated the corrected generator and faster frozen scoring.

  6. pilot_python_shaped_silent_executor_v3 tested the corrected substrate with the default latent recipe. It learned a weak signal but still showed no ordered-vs-shuffled separation.

  7. pilot_python_shaped_silent_executor_v4 removed K=0 from training and increased state supervision. This was more aligned with the execution hypothesis but did not improve the gate.

  8. main_python_shaped_silent_executor_v1 ran the final multi-seed experiment: three QLoRA seeds, train lengths 4,8,16, held-out lengths 24,32, held-out operation compositions, thinking-CoT baseline, ordered latent K-sweep, and shuffled-compute control.

Figures 5

accuracy by length
accuracy by length · reports/figures/
accuracy vs token cost
accuracy vs token cost · reports/figures/
heldout composition accuracy
heldout composition accuracy · reports/figures/
k curve len32
k curve len32 · reports/figures/
training dynamics
training dynamics · reports/figures/

Data files 24

Result tables and metrics copied from the experiment folder — preview inline or open the raw file.

16 more result files not shown here — browse the full folder on GitHub.

Reproduce

This entry is source code or analysis only — there is no separate run to reproduce.

Browse the experiment folder on GitHub ↗