Qwen Fast-Weight Hook Experiment
The one idea you need
Instead of making the model reason out loud in words, we bolted a small loop inside its frozen network — a private scratchpad it can spin extra times before answering. More spins should mean more thinking. We tested whether extra silent spins actually buy accuracy.
The question
If a small model quietly loops through extra thinking steps inside its network before answering, without writing anything down, does it get better at multi-step arithmetic?
What we found
No. Adding more silent internal loops gave no reliable accuracy gain. On small 100-problem tests a few settings looked promising, but rechecking those best moments on 250 problems erased the edge — zero loops did as well or better. A control loop that never actually trained fluked similar bumps, proving the gains were noise, not real internal reasoning.
Why it matters
Do not trust small accuracy bumps from a lightweight add-on measured on a few hundred examples. Always run an untrained control and a larger recheck — noise alone routinely fabricates convincing few-point gains that vanish under scrutiny.
Results at a glance 3
How to read
Bars show accuracy on two problem sets — everyday problems and harder longer-chain problems — as the model runs zero up to eight extra invisible thinking steps before answering. Taller is better; the leftmost zero-step bar is the reference.
Takeaway → The zero-step bar is tallest on both sets, so adding invisible thinking steps left accuracy flat or slightly lower rather than climbing.
Data table
| invisible recurrent steps K at evaluation | validation | hard (longer chains) |
|---|---|---|
| K=0 | 20.8% | 26% |
| K=1 | 19.2% | 24% |
| K=2 | 18.4% | 23.6% |
| K=4 | 19.2% | 23.6% |
| K=8 | 20% | 24.4% |
Numbers from experiments/qwen_fastweight_hook/analysis/large_retests_long.csv
Technical framing
More latent recurrent steps do not help: 250-example retest of the best checkpoint — On the strongest checkpoint (step 200), K=0 beats every K>0 budget on both splits at n=250 — the hoped-for K-scaling is absent.
How to read
Bars compare no extra thinking steps against the best extra-step setting on two problem sets, each scored first on 100 questions then rechecked on 250. Taller is better; the pairs show whether the early lead survives.
Takeaway → Extra-step bars lead on the 100-question checks but fall level or behind on the 250-question rechecks — the early edge was luck of a small sample.
Data table
| split and evaluation size | K=0 (no recurrence) | best K>0 |
|---|---|---|
| val, n=100 | 23% | 27% |
| val, n=250 | 20.8% | 20% |
| hard, n=100 | 22% | 27% |
| hard, n=250 | 26% | 24.4% |
Technical framing
Promising gains at 100 examples evaporate at 250 examples (step-200 checkpoint) — The +4-5pp recurrence advantage seen in small checkpoint sweeps reverses on larger retests — the apparent gains were sampling noise.
How to read
Lines track how many points the best extra-step setting beats zero steps by across training, for the real system and for a control that never trained the thinking loop. The flat zero line means no advantage; above it is better.
Takeaway → Both lines wobble above and below zero, and the untrained control bumps up just as much — the occasional gains reflect noise, not learned computation.
Data table
| training step | full recurrent, val | full recurrent, hard | K=0-only control, val | K=0-only control, hard |
|---|---|---|---|---|
| 0 | 0 | 1 | — | — |
| 50 | 3 | 2 | — | — |
| 100 | 2 | -1 | 0 | 3 |
| 150 | 6 | 3 | — | — |
| 200 | 4 | 5 | 2 | 4 |
| 250 | 6 | 0 | — | — |
| 300 | -2 | 0 | 0 | 1 |
Numbers from experiments/qwen_fastweight_hook/analysis/k_gain_summary.csv
Technical framing
Best recurrence gain over training is unstable, and an untrained-recurrence control bumps too — K gains flicker across checkpoints and a control that never trained the recurrent core shows similar bumps, so isolated gains are noise.
In the author’s words from the Report · “Abstract”
We tested whether a frozen Qwen3.5-4B model augmented with a small internal recurrent fast-weight runtime gains accuracy when given more invisible latent compute steps K. The experiment used generated, exactly-verifiable modular arithmetic multiple-choice tasks, with a harder held-out split containing longer operation chains. The core result is negative: the current implementation does not demonstrate robust K-scaling. Several 100-example checkpoints showed small apparent gains from recurrence, but larger 250-example retests erased or reversed the most favorable effects. A K=0-only training control also produced occasional small K bumps despite never training the recurrent core, showing that isolated few-point gains are not reliable evidence. … Read the full result →
Overview
This experiment tests whether a frozen Qwen3.5-4B model gains accuracy from an inserted invisible recurrent fast-weight runtime.
Contents
src/latent_qwen_fastweight_experiment.py: training and evaluation script.src/analyze_latent_results.py: regenerates analysis CSVs and figures from run metadata.reports/latent_fastweight_qwen_paper.md: standalone paper-style report.reports/latent_fastweight_qwen_paper.html: HTML version of the report.reports/experiment_log.md: chronological run log.analysis/: generated figures, summary Markdown, and analysis CSVs.runs/: small JSON run outputs. Checkpoint.ptfiles are not stored here.checkpoint_manifest.csv: list of saved checkpoints stored outside this directory.
Large Files
Adapter checkpoints are stored at:
../../large_artifacts/qwen_fastweight_hook/checkpoints/Download that directory only if you need to load saved adapter weights. The experiment reports and analysis figures do not require it.
Useful Commands
Regenerate analysis outputs from the stored run metadata:
python experiments/qwen_fastweight_hook/src/analyze_latent_results.pyRun a new experiment from this experiment directory or from the workspace root, passing an explicit --output_dir if you want a named run.
Report
Rendered from reports/latent_fastweight_qwen_paper.md
A local single-GPU experiment on invisible recurrent computation inside Qwen3.5-4B
Abstract
We tested whether a frozen Qwen3.5-4B model augmented with a small internal recurrent fast-weight runtime gains accuracy when given more invisible latent compute steps K. The experiment used generated, exactly-verifiable modular arithmetic multiple-choice tasks, with a harder held-out split containing longer operation chains. The core result is negative: the current implementation does not demonstrate robust K-scaling. Several 100-example checkpoints showed small apparent gains from recurrence, but larger 250-example retests erased or reversed the most favorable effects. A K=0-only training control also produced occasional small K bumps despite never training the recurrent core, showing that isolated few-point gains are not reliable evidence. The best-supported conclusion is that this small bolt-on recurrent adapter sometimes perturbs predictions beneficially, but did not learn stable serial latent computation under this training setup.
Lay Summary
The idea was to give a language model a private scratchpad loop inside its neural network, instead of forcing it to think by writing words. If that worked, giving the model more private steps should make it more accurate on multi-step arithmetic.
We built and ran that test. The model sometimes looked better with a few private steps, but when we checked those promising moments on more examples, the improvement went away. The experiment was still useful: it showed how to run this kind of test, what controls are needed, and why small apparent improvements can be misleading.
1. Motivation
A normal language model has two recurrence channels:
- It can attend over the input sequence inside one forward pass.
- It can generate more output tokens and feed them back into itself.
The second channel is powerful but visible and expensive. The motivating hypothesis was that a model might benefit from an internal recurrent machine:
prompt -> compiler layers -> latent workspace/runtime -> decoder layers -> answerIn the target design, the model does not emit a DSL or call a tool. It updates temporary internal state: workspace tokens, gates over low-rank operators, and differentiable fast-weight memory. The key empirical signature should be:
accuracy(K=0) < accuracy(K=1) < accuracy(K=2/4/8), especially on longer held-out chains2. Important Architecture Caveat
One design assumption was that the backbone would behave like a plain transformer. The actual default model, Qwen/Qwen3.5-4B, is more interesting: its Hugging Face model card describes it as a causal language model with a vision encoder, 4B language-model parameters, hidden size 2560, 32 layers, and a repeated hidden layout of three Gated DeltaNet layers followed by one gated attention layer. NVIDIA's Qwen3.5 documentation likewise describes Qwen3.5 as a hybrid Gated DeltaNet plus standard-attention architecture, with dense and MoE variants in the broader family.
That matters. Gated DeltaNet is already a kind of learned fast-weight/linear-attention substrate. Therefore this experiment should not be read as adding fast weights to a model that lacks them entirely. The sharper question is narrower:
Does adding a small recurrent-depth axis over invisible workspace tokens improve this already hybrid Qwen3.5 model?
Sources: Qwen/Qwen3.5-4B model card, NVIDIA Qwen3.5 documentation.
3. Experimental Setup
Backbone
- Model:
Qwen/Qwen3.5-4B - Loader:
AutoModelForMultimodalLM - Quantization: 4-bit bitsandbytes
- Frozen base parameters: 2,590,093,824 counted in the loaded text/vision model object
- Detected language layers:
model.language_model.layers - Hook insertion point: layer index 28 of 32 (
--hook_layer -4) - GPU: NVIDIA RTX 6000 Ada, about 48 GB VRAM
Runtime Adapter
The adapter reads prompt hidden states only, initializes eight latent workspace tokens, runs K recurrent steps, and writes a residual delta back into the hidden stream before later Qwen layers decode the answer.
Core components:
- workspace tokens: 8
- runtime width: 256
- dynamic low-rank transform bank: 12 bases, rank 16
- temporary fast-weight memory: 128 x 128
- recurrent training budgets:
K in {1,2,4} - evaluation budgets:
K in {0,1,2,4,8}
K=0 is not the frozen model. It is the trained prompt-conditioned injection with zero recurrent loop iterations. The true frozen-model baseline was measured separately with the hook disabled.
Task
The task generator creates exact modular arithmetic problems modulo 97 with five answer choices. Families include single-register chains, reverse-worded chains, and two-register updates. The hard split uses longer operation chains than training by adding three extra steps.
The model is scored by multiple-choice answer-letter log-likelihood. For each candidate letter, we compute the average negative log-likelihood of that answer token and choose the lowest-NLL candidate.
Runs
Three main runs were completed:
| Run | Training K | Extra objective | Purpose |
|---|---|---|---|
| Full recurrent | 1,2,4 | none | Primary test of recurrent latent compute |
K=0-only control | 0 | none | Tests whether untrained recurrent steps create similar bumps |
| Auxiliary value loss | 1,2,4 | 0.2 * CE(answer mod 97) | Tests whether denser numeric supervision helps |
Training used 300 optimizer steps, batch size 4, and 100-example validation/hard evaluations at checkpoints. The most favorable checkpoints were retested on 250 examples per split.
4. Results
Frozen Baseline
The first concern was whether the task was too easy. It was not.
Frozen Qwen3.5-4B, hook disabled:
| Split | Accuracy |
|---|---|
| Validation | 18.0% |
| Hard | 21.0% |
Chance is 20% because there are five choices. This gave the recurrent runtime real headroom.
Training Checkpoints
At 100-example checkpoint evaluations, the full recurrent run sometimes showed positive K>0 gains. For example, at step 150 validation improved from 22% at K=0 to 28% at K=2, and hard improved from 29% at K=0 to 32% at K=1.
But the effect was unstable:
- It was not monotonic in
K. - It appeared and disappeared across checkpoints.
- The final full recurrent checkpoint was negative on validation:
K=0was 26%, whileK=1/2/4/8were 22%, 21%, 22%, and 24%. - The hard split did not show reliable length-generalization gains.
The heatmap below shows the full recurrent run across all checkpoints.
Control Run
The K=0-only control trained the same prompt-conditioned injection path but never trained the recurrent core. If untrained recurrence still creates small K bumps, then isolated gains in the full run are not enough.
That is what happened. At step 200, the control hard split improved from 26% at K=0 to 30% at K=4, even though recurrent dynamics were untrained. This does not mean the control learned recurrence. It means 100-example K sweeps are noisy enough to produce misleading few-point bumps.
Larger Retests
The most important verification was retesting promising checkpoints on 250 examples per split with Wilson 95% intervals.
The strongest apparent full-run checkpoint was step 200. On larger retest:
| Split | K=0 | Best K>0 | Result |
|---|---|---|---|
| Validation | 20.8% | 20.0% (K=8) | no gain |
| Hard | 26.0% | 24.4% (K=8) | no gain |
The auxiliary final checkpoint also failed to show a robust effect:
| Split | K=0 | Best K>0 | Result |
|---|---|---|---|
| Validation | 17.6% | 16.8% (K=4) | no gain |
| Hard | 21.2% | 22.8% (K=2/8) | +1.6 points, not compelling |
The auxiliary value loss itself remained close to random prediction of one of 97 residues, so it did not provide evidence that the workspace learned exact arithmetic.
5. Interpretation
The experiment did not validate the strong hypothesis:
More latent recurrent compute reliably improves accuracy on serial-depth-bound arithmetic.
Instead, it supports a weaker and more cautionary conclusion:
This small 256-dimensional recurrent adapter can perturb Qwen's answer distribution, sometimes beneficially, but under this training budget it does not produce stable, generalizing, monotonic internal computation.
Several observations point in the same direction:
K=0was often as good as or better thanK>0under larger retesting.- Apparent K gains were not monotonic.
- The hard length-generalization split did not improve reliably.
- A
K=0-only control produced occasional K bumps despite untrained recurrence. - The auxiliary value-prediction head did not learn the numeric target.
6. What This Falsifies and What It Does Not
This does not falsify latent recurrent reasoning as a general idea. It falsifies a narrower implementation claim:
A small, hook-inserted, 256-dimensional recurrent fast-weight adapter trained for 300 steps with answer-letter NLL is sufficient to produce robust K-scaling on these modular arithmetic tasks.
That claim is not supported.
The broader idea remains plausible because this implementation is thin relative to the proposal:
- The recurrent loop operates in a small adapter, not through repeated full-width Qwen layers.
- The final objective is mostly one answer-letter token, a low-bandwidth signal.
- The fast-weight memory is a simple leaky outer-product accumulator.
- There is no monotonic-refinement loss, halting objective, teacher trace distillation, or paired per-example recurrence analysis.
- Qwen3.5 already contains Gated DeltaNet layers, so the added fast-weight memory may be redundant unless the recurrent axis is made much stronger.
7. Recommended Next Experiment
The next version should target the failure modes directly:
- Use paired per-example evaluation. Record whether the same example flips correct/incorrect as K changes, then use paired tests rather than independent binomial intervals.
- Train with a real workspace target. Predict the numeric answer from
S_K, but make that objective strong enough to learn; the current auxiliary head did not. - Iterate a wider computation. Re-run a small slice of upper Qwen layers with workspace conditioning, not only a 256-dimensional adapter.
- Preserve all checkpoints automatically. This is now patched in the script.
- Reduce intermediate eval cost. Use small checkpoint sweeps, then large retests only for selected checkpoints.
- Add a refinement objective. Train
V(S_{t+1})to be more predictive thanV(S_t)so recurrence is pressured to improve state rather than merely move it. - Use cleaner gates if interpretability matters. Softmax mixtures are poor evidence for discrete latent opcodes; sigmoid/top-k or vector-quantized program codes would make clustering tests more meaningful.
8. Reproducibility
Primary files:
- Experiment script:
../src/latent_qwen_fastweight_experiment.py - Analysis script:
../src/analyze_latent_results.py - Experiment log:
experiment_log.md - Long-form checkpoint data:
../analysis/training_accuracy_long.csv - Larger retests:
../analysis/large_retests_long.csv - Figures:
../analysis/figures/
Primary run directories:
../runs/main_qwen35_hook_full_seed7../runs/control_qwen35_hook_traink0_seed7../runs/main_qwen35_hook_aux02_seed7../runs/eval_main_step200_n250../runs/eval_aux_final_n250
Large adapter checkpoints are stored outside the experiment bundle under:
../../../large_artifacts/qwen_fastweight_hook/checkpoints/
Environment:
- Python 3.12.3
- PyTorch 2.8.0+cu128
- Transformers 5.12.1
- bitsandbytes 0.49.2
- GPU: NVIDIA RTX 6000 Ada Generation
9. Bottom Line
This was a useful negative result. The experiment found real headroom and successfully inserted an invisible recurrent runtime into Qwen3.5-4B, but it did not find robust evidence that more latent recurrent steps improve exact modular arithmetic. The strongest early positive signs did not survive larger retesting. The next experiment should move from a small residual adapter toward a stronger recurrent-depth mechanism with denser supervision and paired statistical evaluation.
Experiment log 10
Show the running log (10 entries)
Objective
Test whether a frozen Qwen 4B-class model augmented with an invisible recurrent fast-weight hyperadapter learns useful internal computation, operationalized as higher multiple-choice accuracy when given more latent recurrent steps K.
Primary Hypotheses
- If the adapter learns real internal computation, validation accuracy should improve with recurrent budget:
K=0 < K=1 < K=2 < K=4, with possible plateau byK=8. - If fast-weight memory contributes to computation rather than acting as noise, disabling memory should reduce hard-set accuracy or weaken K-scaling.
- If activation-programmed low-rank transforms contribute distinct programmable operators, disabling the dynamic low-rank bank should reduce accuracy or weaken K-scaling.
- If the result is a narrow training artifact, improvements will disappear on harder held-out examples with longer operation chains.
Initial Environment Findings
- Machine: NVIDIA RTX 6000 Ada Generation, about 48 GB VRAM.
- Python: 3.12.3.
- PyTorch: 2.8.0+cu128, CUDA available, bf16 supported.
- Installed missing dependencies:
transformers,accelerate,bitsandbytes,sentencepiece,safetensors,einops,matplotlib,pandas,seaborn,scipy,scikit-learn,markdown,reportlab. - Hugging Face API check showed
Qwen/Qwen3.5-4Band fallbackQwen/Qwen3-4Bare public and ungated. Qwen/Qwen3.5-4Bloaded successfully withAutoModelForMultimodalLMin 4-bit; detectedmodel.language_model.layers, 32 layers, hidden size 2560, hook index 28 for--hook_layer -4.
Script Changes Made
- Added unhooked frozen-model multiple-choice baseline evaluation.
- Added step-0 untrained-adapter evaluation.
- Added run metadata with software versions, GPU details, and script SHA-256.
- Added ablations:
--disable_fast_memoryand--disable_dynamic_lowrank. - Added optional hook-mode workspace auxiliary value loss:
--aux_value_loss. - Verified the modified hook path with a tiny random Llama smoke test.
Planned Experimental Sequence
- Short Qwen pilot: estimate step/eval wall-clock cost and catch runtime failures.
- Main recurrent adapter run with full fast-weight memory and dynamic low-rank bank.
- Static/no-recurrence control using
train_k=0. - Fast-memory ablation.
- Dynamic-low-rank ablation.
- Analyze K-scaling on normal and harder held-out tasks.
- Generate figures and write a paper-style report.
Control Runs
K=0-Only Training Control
Run: ../runs/control_qwen35_hook_traink0_seed7
Purpose: train the same prompt-conditioned hook/readout path with no recurrent steps. At evaluation, K>0 invokes an untrained recurrent core. Any K gains here would weaken the interpretation that recurrent training caused the gains in the full run.
Step 100 eval:
- Validation:
K=0 25.0%,K=1 25.0%,K=2 24.0%,K=4 22.0%,K=8 22.0% - Hard:
K=0 20.0%,K=1 20.0%,K=2 18.0%,K=4 20.0%,K=8 23.0%
Interpretation: unlike the full run's step-100 validation result, untrained recurrent steps do not create a small-K validation improvement. This mildly supports the recurrent-training interpretation.
Step 200 eval:
- Validation:
K=0 19.0%,K=1 19.0%,K=2 17.0%,K=4 16.0%,K=8 21.0% - Hard:
K=0 26.0%,K=1 25.0%,K=2 28.0%,K=4 30.0%,K=8 28.0%
Interpretation: the untrained recurrent core can still create small apparent K bumps on 100-example evals, especially on hard. This weakens any interpretation based only on isolated checkpoint/K improvements.
Step 300 final eval:
- Validation:
K=0 22.0%,K=1 22.0%,K=2 20.0%,K=4 17.0%,K=8 20.0% - Hard:
K=0 19.0%,K=1 17.0%,K=2 19.0%,K=4 16.0%,K=8 20.0%
Interpretation: final K=0-only control is mostly flat or negative with K. The control does not show the same repeated small-K validation gains as the full run, but it did show enough noise at step 200 that small 100-example K bumps cannot be treated as conclusive.
Auxiliary Value-Loss Run
Run: ../runs/main_qwen35_hook_aux02_seed7
Purpose: strengthen supervision by adding 0.2 * CE(value mod 97) from the final workspace, while keeping final evaluation as answer-letter log-likelihood. This tests Claude's critique that letter-only supervision is too low-bandwidth for learning arithmetic-like latent computation.
Step 100 eval:
- Validation:
K=0 18.0%,K=1 20.0%,K=2 23.0%,K=4 19.0%,K=8 20.0% - Hard:
K=0 22.0%,K=1 22.0%,K=2 21.0%,K=4 18.0%,K=8 22.0%
Auxiliary value loss remains close to random (log(97) ~= 4.57) through this checkpoint. Interpretation: the auxiliary head has not yet learned arithmetic, and the eval pattern remains validation-only and unstable.
Step 200 eval:
- Validation:
K=0 20.0%,K=1 20.0%,K=2 18.0%,K=4 17.0%,K=8 19.0% - Hard:
K=0 28.0%,K=1 26.0%,K=2 25.0%,K=4 26.0%,K=8 26.0%
Auxiliary value loss remains near random. Interpretation: the auxiliary objective is not producing useful recurrent computation under this budget; recurrence is now negative.
Step 300 final eval:
- Validation:
K=0 17.0%,K=1 18.0%,K=2 18.0%,K=4 20.0%,K=8 19.0% - Hard:
K=0 15.0%,K=1 18.0%,K=2 20.0%,K=4 18.0%,K=8 17.0%
Auxiliary value loss still did not move convincingly below random. Interpretation: the final checkpoint has a small positive K effect, but the auxiliary mechanism failed at its intended numeric-prediction target, so this is not strong evidence for learned arithmetic computation.
Large retest of final auxiliary checkpoint:
Run: ../runs/eval_aux_final_n250
- 250 validation and 250 hard examples.
- Validation:
K=0 17.6%,K=2 15.2%,K=4 16.8%,K=8 15.6% - Hard:
K=0 21.2%,K=2 22.8%,K=4 20.4%,K=8 22.8%
Interpretation: the validation K effect reversed under larger-sample retesting. The hard-set bump is only +1.6 percentage points and not compelling at this sample size. This reinforces the conclusion that the current implementation does not demonstrate robust latent recurrent K-scaling.
Pilot Results
- Tiny random Llama hook smoke passed after setting
--hook_layer -1for the 2-layer toy model. - Qwen3.5 load-only check passed with 4-bit quantization and hook index 28.
- Qwen3.5 batch-1 pilot passed.
- Qwen3.5 batch-4 pilot passed with
candidate_batch_size=20; no OOM. Three optimizer steps plus two small eval passes took about 32 seconds including reload overhead. This supports running the main sweep withbatch_size=4,grad_accum=1,eval_batch_size=4, andcandidate_batch_size=20.
Main Full-Adapter Run: Early Evidence
Run: ../runs/main_qwen35_hook_full_seed7
Initial frozen-model baseline over 100 validation and 100 hard examples:
- Validation: 18.0%
- Hard length-generalization set: 21.0%
Initial untrained-adapter eval:
- Validation:
K=0 19.0%,K=1 19.0%,K=2 18.0%,K=4 18.0%,K=8 19.0% - Hard:
K=0 28.0%,K=1 26.0%,K=2 26.0%,K=4 29.0%,K=8 29.0%
- Validation:
Interpretation: the default task is already in the right headroom regime; frozen Qwen3.5 is near 5-way chance. Difficulty tuning remains useful as a robustness check, but not as a prerequisite for observing a signal.
Step 50 eval:
- Validation:
K=0 24.0%,K=1 26.0%,K=2 27.0%,K=4 27.0%,K=8 25.0% - Hard:
K=0 26.0%,K=1 28.0%,K=2 28.0%,K=4 27.0%,K=8 27.0%
Interpretation: early result shows a small positive K effect around K=2..4, but the effect is weak enough that it requires later checkpoints, controls, and binomial uncertainty intervals before making any claim.
Step 100 eval:
- Validation:
K=0 21.0%,K=1 23.0%,K=2 23.0%,K=4 22.0%,K=8 22.0% - Hard:
K=0 25.0%,K=1 24.0%,K=2 23.0%,K=4 24.0%,K=8 23.0%
Interpretation: the validation K bump persists weakly, but the hard-set length-generalization result is negative. This argues against a strong current-form recurrent-computation claim.
Step 150 eval:
- Validation:
K=0 22.0%,K=1 25.0%,K=2 28.0%,K=4 25.0%,K=8 24.0% - Hard:
K=0 29.0%,K=1 32.0%,K=2 31.0%,K=4 27.0%,K=8 28.0%
Interpretation: the strongest positive effect so far appears at small recurrent budgets, especially K=2 on validation and K=1..2 on hard. This is evidence for a compute-budget interaction, but not for monotonic K-scaling.
Step 200 eval:
- Validation:
K=0 23.0%,K=1 27.0%,K=2 26.0%,K=4 25.0%,K=8 27.0% - Hard:
K=0 22.0%,K=1 27.0%,K=2 24.0%,K=4 26.0%,K=8 26.0%
Interpretation: another positive K>0 result, strongest at K=1. The effect is inconsistent across K and checkpoints, but recurrent steps have not been purely decorative.
Step 250 eval:
- Validation:
K=0 18.0%,K=1 24.0%,K=2 21.0%,K=4 21.0%,K=8 24.0% - Hard:
K=0 24.0%,K=1 23.0%,K=2 23.0%,K=4 21.0%,K=8 24.0%
Interpretation: validation retains a positive recurrent-step effect, but hard-set length generalization does not. This weakens the serial-computation interpretation.
Step 300 final eval:
- Validation:
K=0 26.0%,K=1 22.0%,K=2 21.0%,K=4 22.0%,K=8 24.0% - Hard:
K=0 15.0%,K=1 15.0%,K=2 15.0%,K=4 14.0%,K=8 15.0%
Interpretation: the final checkpoint does not support K-scaling. Since intermediate checkpoints did show small positive effects, the current evidence points to unstable or undertrained recurrent refinement, not a conclusive latent-computation win.
Preserved checkpoints:
../../../large_artifacts/qwen_fastweight_hook/checkpoints/main_qwen35_hook_full_seed7/latent_adapter_step200.pt../../../large_artifacts/qwen_fastweight_hook/checkpoints/main_qwen35_hook_full_seed7/latent_adapter_step250.pt../../../large_artifacts/qwen_fastweight_hook/checkpoints/main_qwen35_hook_full_seed7/latent_adapter.pt(step 300)
Large retest of step 200 checkpoint:
Run: ../runs/eval_main_step200_n250
- 250 validation and 250 hard examples.
- Validation:
K=0 20.8%,K=1 19.2%,K=2 18.4%,K=4 19.2%,K=8 20.0% - Hard:
K=0 26.0%,K=1 24.0%,K=2 23.6%,K=4 23.6%,K=8 24.4%
Interpretation: the most favorable recurrent checkpoint did not survive larger-sample retesting. K=0 is best on both splits. This is strong evidence against a robust K-scaling effect in the current implementation.
Figures 3
Data files 15
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
analysis/k_gain_summary.csv1.4 kBruns/control_qwen35_hook_traink0_seed7/results.json9.1 kBruns/eval_aux_final_n250/eval_only_results.json3.8 kBruns/eval_main_step200_n250/eval_only_results.json3.8 kBruns/loadcheck_qwen35/results.json2.4 kBruns/main_qwen35_hook_aux02_seed7/results.json11 kBruns/main_qwen35_hook_full_seed7/results.json11 kBruns/main_qwen35_hook_full_seed7/results_through_step200.json8.4 kBruns/main_qwen35_hook_full_seed7/results_through_step250.json9.8 kBruns/pilot_qwen35_hook_batch4/results.json3.7 kBruns/pilot_qwen35_hook_full/results.json4.1 kBruns/smoke_tiny_aux/results.json3.3 kBruns/smoke_tiny_eval_only/eval_only_results.json3.7 kBruns/smoke_tiny_hook/results.json969 Bruns/smoke_tiny_hook_v2/results.json3.7 kB
Reproduce
The run commands are documented inside the experiment folder (see the README).