Research log Small Model Experimentation
GitHub

What has this corpus learned?

One fixed Qwen3.5-4B — no scaling, no bigger teacher — pushed to see how much capability you can draw out of it and install back in. This is the reading surface: 297 finished experiments and 3 in progress, every claim tied to its evidence, every result rendered from its own data — newest first.

Experiments3002026-06-19 → 07-28Programs13lines of inquiryClaims6410 confirmedFigures848rendered on-siteQueued38proposals waiting
64 durable claims, each tied to its evidence — 10 confirmed · 49 promising · 2 open · 3 ruled out Open the ledger →
Glossary — the corpus vocabulary, one line each
deployable
a setting you could actually ship: greedy decoding, one sample, no oracle reranking
greedy@1
single deterministic answer (temperature 0) — the strictest deployable metric
pass@k
chance that at least one of k samples passes the hidden tests
coverage
fraction of tasks where at least one sample in the pool is correct — the ceiling that self-training can bank
oracle
the score if you could always pick the pool's best sample — an upper bound, not deployable
visible / hidden tests
the checks the model may see and run, vs the held-out checks that decide correctness
false-pass
a candidate that passes the visible tests but fails the hidden ones
thinking / no-think
generation with the model's native reasoning channel enabled vs disabled
banking
fine-tuning the fixed model on its own verified successes, so sampled wins become its greedy default
import (lines Y / Z)
the 2026-06-28 bulk import of the predecessor repo's two parallel working tracks

Latest findings

All experiments →
2026-07-19

Qwen35 4B Agentic RLVR Feasibility

GATE PASSED: single-GPU agentic GRPO (execution-reward RLVR) physically works for Qwen3.5-4B on one 24GB card; but the raw base produces zero reward variance (explores then quits without writing) -> a narrow SFT warm-start is the prerequisite before RLVR has signal.

Yes to the machinery: with three fixes (force eager mode so the hybrid architecture doesn't hang, load the model in bf16 not fp32, and split GPU memory 55% to the inference engine with sleep-mode) the loop CLOSES - a movable test reward climbed 0.03->0.38 in 3 steps, proving the generating model really updates (C49 does not bite here). The full agentic loop (model drives tools, runs pytest, gets a reward) runs end-to-end. But NO to a raw-base signal: the base explores 1-2 file reads (~120 tokens) then stops without writing a fix, so every attempt scores the same -> zero variance -> zero gradient, and the GPU can't fit a big enough attempt-group (num_generations>4 OOMs) to catch the base's rare successes. So a narrow SFT warm-start to teach the explore->edit->test->iterate loop is required before RLVR can learn.

Evidence-Conditioned SelectionInterpretability and DiagnosticsPosttraining and Adaptation
2026-07-18

Qwen35 4B WHY-Comment Install

Biggest single-function gain (+5 HumanEval) but flat on the agentic loop — and complementary to the repair bet

The WHY idea paid off where it should — on writing correct functions — and did nothing where it shouldn't. We trained the 4B to write code with the causal reason for each line attached as an inline #WHY: comment (generated by construction, no teacher). Because test-graders ignore comments, this is a perfectly clean test: if the CODE improves, teaching WHY worked. It did — HumanEval +5 problems (76.2->79.3%), the biggest single gain of the whole program, and MBPP about even. But on the multi-step agentic harness it was flat (8/35, same as base). The mirror image of the repair bet, which helped the agentic loop but less on functions. Neither result is statistically airtight on its own, but the pattern is clean and mechanistic: teaching WHY improves per-function correctness; teaching the check-and-fix loop improves multi-step behavior. They fix different things. The obvious next move is to combine them — stack the WHY model and the repair model — and see if we get both gains at once, which would also be strong enough to trust. One operational note: the standard training recipe badly underfit the WHY comments (they are high-entropy text); it took 4x the training to actually install the behavior.

Agentic Breadth InstallationStructured Execution and Compilers
2026-07-18

Qwen35 4B Self-Repair Install

Weak positive (first non-null): teaching the repair loop nudged real coding where teaching a passive skill did not

The first bet that moved the needle at all — gently. We taught the 4B to debug by training on 504 examples of [buggy code + the real test-failure message] -> [diagnosis + fix], all self-generated by injecting bugs into correct code and running the tests. Result: HumanEval +3 problems (76.2->78.0%), MBPP -2 (a wash), and on the real multi-step agentic harness 8/35 -> 10/35 — and crucially the wins are asymmetric: it solved 3 tasks the base couldn't while losing only 1, so it is ADDING capability, not just trading it around like the previous (execution-tracing) bet did. Honest caveat: the numbers are small and none is statistically significant on its own, so this is a promising direction, not a proven win. The real lesson: teaching the LOOP behavior (check your work and fix it) does something teaching a passive skill (trace the code) did not. That points the way for the next bets — teach the model WHY, and stack this repair ingredient with it.

Agentic Breadth InstallationEvidence-Conditioned SelectionInterpretability and Diagnostics
2026-07-18

Qwen35 4B Repair + Why Stack

Mixture dilution: combining the two winning curricula in one corpus washed out both gains

The obvious way to combine our two promising curricula - just train one model on both - backfired. We put the 504 self-repair rows and 504 WHY-comment rows into one 1008-row training set and trained a single adapter. Instead of getting both gains, we got neither: the WHY model's +5 HumanEval collapsed to +1, and the repair model's agentic 10/35 collapsed to 7/35 (basically base). This is a known trap (we hit it in the earlier menagerie work too): when you train one small model on a mixture, each ingredient gets half the concentration and washes out. So the complementary effects are real enough to conflict for capacity, but they don't ADD this way. The right way to combine two separately-trained specialists is in weight space - keep each at full strength and add their learned changes together (task vectors) - which is the cheap next test, no retraining needed.

Agentic Breadth InstallationStructured Execution and Compilers
2026-07-18 →

Qwen35 4B WHY-Think Scale

Design frozen: the CORRECTED dual-channel WHY curriculum - a genuine step-by-step <think> derivation AND strippable #WHY: code comments - built to scale WHY without destroying the 4B's native thinking (GPU sweep not yet run)

This phase built and proved the machinery; the GPU training sweep has not run yet. The generator now emits, for every example, a real hidden reasoning trace generated mechanically from the program's shape - parse the spec, pick an approach (a running total? a builder? a search?), build the code step by step, then trace one of the example's own tests by hand, running the program to get every intermediate value so the trace is true - plus the clean code with strippable WHY comments. Every example is checked by actually running it: strip the comments and the code passes all its tests; the commented version runs identically; and the hidden trace's worked example is recomputed byte-for-byte and must match real execution (a tamper of any traced number is rejected). Measured on a 5000-example sample it hits all 59 program families across 13 task kinds, keeps 100% distinct programs, carries about 1,196 distinct comment-reasoning patterns AND about 4,997 distinct think-derivation skeletons (out of 5000 - the reasoning genuinely varies, it is not one template); at 10,000 examples the think skeletons stay ~100% distinct. Nothing looks like the benchmarks (zero banned benchmark names anywhere - prompt, hidden think, or answer - and zero distinctive shared code fragments through 10,000 examples). Crucially the added thinking fits the training window: the real tokenizer render tops out at 739 tokens (median 467) against a 4096 cap, with zero examples truncated. On top sits a five-rung ladder (2000/5000/10000/20000/40000), each corpus fingerprinted so it rebuilds identically, a fail-closed trainer that trains a fresh small adapter per rung at ONE epoch (unlimited unique data, never re-shown), with a POSITIVE weight on the think channel so the native thinking is preserved and shaped rather than emptied, and a per-rung exam sweep on HumanEval + MBPP run with thinking ON and the base co-measured each rung (no stale thinking-off number carried). Tests pass.

Agentic Breadth InstallationBenchmark GeneralizationPosttraining and Adaptation
2026-07-18 →

Qwen35 4B WHY Scale Ladder

Design frozen: a scale-capable, genuinely-diverse WHY curriculum + a four-rung ladder to find where the +5 fast gain peaks (GPU sweep not yet run)

This phase built and proved the machinery; the GPU training sweep has not run yet. The core blocker was that the original WHY generator saturated fast (about 75 distinct reasons, 438 distinct programs at 504 examples), so scaling it naively would just replay data and fake a flat 'it doesn't scale' result. The rebuilt generator fixes that: it has 59 program families across 13 kinds of task (arithmetic, list work by hand, digit and remainder math, nested loops, adjacent-pair checks, small searches, simple state machines, string work, dictionary counting), and a large pool of true, line-specific reason phrasings. Measured on a 5000-example sample it hits all 59 families, about 1,196 distinct reasoning patterns (numbers and variable names stripped out), and 100% distinct programs; at 20,000 examples the programs are still 100% distinct. Every single example is checked by actually running it: strip the comments and the code still passes all its tests, the commented version runs identically, every comment names something real on its own line, and nothing looks like the benchmarks (zero banned benchmark names, zero distinctive shared code fragments). Each example also fits comfortably in the training window (about 337 tokens on average, 499 at most, versus a 4096 cap). On top of this sits a four-rung ladder (2000/5000/10000/20000), each corpus fingerprinted so it rebuilds identically, a fail-closed trainer that trains a fresh small adapter per rung with an epoch schedule that gives smaller sets more passes, and a per-rung scoring sweep against HumanEval and MBPP that the orchestrator runs one rung at a time to plot the score-vs-size curve and find the peak. 52 tests pass. The honest prior: about a 45% chance some rung clearly beats the base model, with a flat curve still a real and useful possibility that would tell us the WHY trick does not scale on this surface.

Agentic Breadth InstallationBenchmark GeneralizationPosttraining and Adaptation
2026-07-17

Qwen35 4b State Track Confirmation

Confirmed but soft: the state-tracking lift replicates directionally across six seeds, modest and noisy

The lift held up directionally, without becoming a slam dunk. Across six fresh sealed exams, running the same seed through both models so the noise cancels, the state-tracking model beat its parent on 4 of 6 with an average edge of +0.021 aggregate — right in line with the +0.026 seen originally. By the pre-registered rule that is CONFIRMED. But being honest about the statistics: the seed-to-seed spread (SD 0.045) is wide enough that a strict paired t-test (t=1.12, 5 df) would call this not-significant on its own. That is not a surprise or a moved goalpost — the rule was written up front as a LIBERAL directional check, precisely because the decisive value here is the OTHER outcome: a clearly negative mean would have killed the result as noise, and it did not. Across all seven exam seeds now, five are positive and the mean is +0.021. So the read is: state-tracking gives a real but small and noisy general uplift — enough to make it the current best model to carry forward, not enough to call a decisive win. The install-a-universal-skill idea is directionally supported. Next the model goes in front of a real coding agent to see whether any of this proxy improvement shows up where it actually matters.

Agentic Breadth InstallationStructured Execution and Compilers
2026-07-17

Qwen35 4B Exec-Trace Install

Null: teaching the model to trace code did not make it code better (but did not hurt it either)

The first bet at installing coding cognition came back flat. We trained the 4B on 400 self-generated, execution-verified program traces to install an accurate 'mental interpreter,' the idea being that a model that can simulate code should write and debug it better. It did not pan out: HumanEval nudged +1 problem (noise, it is near ceiling), MBPP dropped 3, and the real target — a multi-step agentic coding harness — landed EXACTLY where the base did, 8 of 35, just solving a different 5 tasks. The good news is retention held: teaching the model to trace did not wreck its ability to write code, which was the main risk. But the capability was reshuffled, not raised — the same pattern the menagerie work showed. The lesson: installing a passive skill (understanding execution) does not convert to active coding ability. The next bet targets the actual failure mode we see — the model takes one shot at a multi-step task and stops instead of planning, verifying, and repairing — by training the agentic loop directly.

Agentic Breadth InstallationStructured Execution and Compilers
2026-07-17

Count-Walk Replay Compound (Stage 8)

Bounded: replay compounding stops adding at stage 8 — the first time this reliable move failed

The believed-likelier outcome, delivered cleanly. 'Replay compounding' — retraining on the accumulated replay mixture — had lifted the aggregate score at every previous link in this model's build chain, so it was the safest broad move available. On the eighth link it stopped working: the new model scored 0.342 versus the parent's 0.363, a genuine drop, and one task family (warren) fell far enough to trip the guardrail on its own. The pre-registered rule reads that as BOUNDED: the move has hit diminishing returns on this particular parent, which is already saturated with replay data, so more of the same just reshuffles strength between families instead of adding it (two families up, three down, net negative). The parent model stays the reference. The useful part is the knowledge: we now know the cheap reliable lever has a ceiling here, so the next push has to be a different KIND of move, not another dose of the same one. The new model still crushes the raw base model by 0.238 and wins 8 of 10 families against it.

Agentic Breadth InstallationPosttraining and Adaptation
2026-07-17

Count-Walk Menders Confirmation

Ambiguous: the menders hit did not replicate — a control drew the same score, so no claim is made

The answer the rule was built to force out, delivered without wiggle room. Across the four fresh exams the trained model solved a fix-the-procedure episode exactly once (plus one partial credit that the rules pre-declared doesn't count) — and on that same exam, the comparison model trained WITHOUT the special lessons solved one too. One hit when two were required, and a dead tie against a control, is the pre-written middle verdict: no claim. The clean interpretation is that occasional single-episode solves are background luck this family hands out to roughly one run in ten — the pre-registered noise rate — and the earlier headline result (the trained model scoring while all three controls sat at zero) was most likely that luck landing photogenically. The rule also pre-committed the consequence: no more exam seeds for this comparison; any future attempt at this family must be a genuinely different design, not a re-roll. One quietly encouraging descriptive note: the trained model posted the best overall score on two of the four exams (0.398 and 0.392, its two best readings ever), though those readings carry no claim.

Agentic Breadth InstallationStructured Execution and Compilers
2026-07-17

Coding Fitness Harness (cognitive-core program)

Foundation of the coding-install program: base is a strong function coder (76%) but weak agent (23%)

Surprisingly good at writing single functions (HumanEval 76.2%, MBPP 56.5%) but weak at driving a multi-step coding task in a real agent loop (23%). The harness is validated: it agrees with an independent run to within 1 problem in 164, is deterministic, and correctly passes canonical solutions while failing wrong ones. The 76-vs-23 gap tells us exactly where to aim: not function-writing, which is nearly maxed, but the agentic cognition — planning, tracking state across edits, debugging — that turns a function writer into a coding agent.

Benchmark GeneralizationPosttraining and AdaptationStructured Execution and Compilers
2026-07-17 →

State-Track Installation (Stage 9)

Installed transfer: a new, unrelated skill lifted the aggregate where replay had stalled (single seed)

The believed-unlikelier but hoped-for outcome landed. After the reliable 'just replay again' lever hit its ceiling, this tried a genuinely different lever: teach the model one new, universal skill — keeping a running tally of named counters through a list of updates — using training data that looks nothing like any benchmark. On the sealed exam the new model scored 0.326 aggregate versus the parent's 0.300 and the raw base's 0.168, and crucially no task family dropped by more than a rounding step. The gains showed up on exactly the kinds of tasks where tracking state should help (siftstack +0.2, lockpick +0.1, mirage +0.1). So the pre-registered rule reads INSTALLED_TRANSFER: a fresh transferable skill can still add where more-of-the-same replay cannot. Two honest asterisks keep this from being a victory lap: it is ONE exam seed, and the parent's own score bounces between 0.30 and 0.36 seed-to-seed, so the 0.026 gain has to survive a multi-seed confirmation (the same check that recently exposed an earlier headline as noise) before we trust it. And the ultimate goal — beat the raw base model on EVERY family — is still not met, because one family (warren) sits below base, a weakness inherited from the parent. Real signal, properly hedged; the next step is to confirm it across seeds.

Agentic Breadth InstallationBenchmark GeneralizationPosttraining and Adaptation

What we’ve learned

Full technical synthesis →

This corpus asks one disciplined question: how much more capability can you pull out of — and permanently install into — a single fixed small model, using no bigger model and no smarter teacher, when the bar to beat is simply sampling the model more times? The honest finding is that you cannot outwit the frozen weights at test time — cleverer answer-picking, run-and-revise feedback, and internal steering all fail to beat plain sampling — but you can bank the model's own verified successes back into its weights, and use external tools to reach the few solutions it cannot find alone. The one wall that held everywhere is that the model executes any plan it is handed almost perfectly yet cannot find the plan itself. Just as important is the ledger of honest dead-ends, and the repeated discovery that careful controls, not hopeful demos, decide whether a result is real at all.

  1. Sample a few times, then filter — that's the bar

    A single answer undersells a small model several times over. Draw a handful of samples and keep one that passes a visible test — that alone recovers most of the hidden ability, cheaply. Every fancier method has to beat this baseline first, and most never do.

  2. No test-time trick beats sampling more

    The right answer is often already among the model's samples, so smarter answer-picking, running the code and revising against real errors, and steering its internal activations all seemed promising. Each merely matched plain sampling or did nothing at all. Proposing the right answer — not decoding frozen weights more cleverly — is the real bottleneck.

  3. Bank the model's own verified wins into its weights

    The one lever that genuinely installs new capability: fine-tune the model on answers it produced and verified itself, with no stronger teacher anywhere. This turns occasional lucky samples into dependable one-shot skill, raising single-shot success by about 40%, a gain that compounds to roughly 50% over several repeated rounds.

  4. Self-training saturates at the model's sampling edge

    Fine-tuning on its own verified work reliably solves what the model used to get only sometimes — but never what it can't already sample, and the gains plateau. An external search tool plus a code interpreter can harvest a few solutions beyond that edge to train on, roughly doubling the hardest cases: real, but modest.

  5. Turn on the reasoning mode it left off

    The corpus ran the model with its built-in step-by-step reasoning switched off everywhere — its biggest self-inflicted wound. Turning it on lifted coding accuracy from about 76% to 91%, and controls prove the gain is genuine reasoning: feed it another problem's reasoning and accuracy collapses to about 4%. Too much thinking can still hurt.

  6. It knows it's wrong in numbers, not words

    Ask the model out loud whether it is sure and the answer is worthless — it claims confidence even when wrong. But the probability it quietly assigns its own answer closely tracks whether that answer is right. Read that internal number to pick among samples or abstain — it beats a majority vote, for free.

  7. The model follows a recipe but can't write one

    Hand the model a correct plan and it carries it out almost perfectly, even across long multi-step chains; ask it to find the plan itself and it fails almost at once. The scarce skill is deciding what to do, not doing it — so let cheap tools search and let the model execute.

  8. Controls decide what's real; the failures are the asset

    The corpus's real worth is what it rules out. Careful controls quietly killed flattering wins: 40% of the 'hard' tasks were secretly easy, a serving bug silently trained and measured the untouched original model, and earlier 'self-improvement' failures were just contaminated test data. Audit your tools and data before believing any result.

  9. Extending a fixed model's frontier without a teacher stays open

    Self-training and tools widen what a fixed model reliably does, and tools nudge its frontier slightly past its own sampling reach — but only modestly. Recent attempts to push further, by training tool-use agents on correct actions, backfired and erased working behavior: a narrow-looking signal still shifts the whole model. Cleanly extending a small model's frontier without a bigger teacher remains the open problem.

Load-bearing results

Claims →

The experiments most often cited as evidence by the claim ledger.

  • Qwen3.5-4B Partial-Structure Recognition-Guided Search · cited by 4 claims

    No. Shown a half-finished program skeleton, the four-billion-parameter model's guess at whether it could still be completed was barely above a coin flip — about 51% correct, where 50% is pure chance —

  • Qwen3.5-4B: Can SFT Install the Skill of Induction? · cited by 4 claims

    Yes, but only when it thinks out loud. The trained model cracks brand-new hidden rules perfectly when it writes out its reasoning, yet collapses to worse-than-guessing — about 1 in 100 — when forced t

  • Gauntlet frontier: difficulty escalation past the breadth-install plateau · cited by 4 claims

    No, not both at once. Piling on more, harder, or more varied self-generated practice all stalled at the same ceiling, and even hand-written expert solutions the model could not discover on its own fai

  • Qwen35 4B Agentic RLVR Feasibility · cited by 4 claims

    Yes to the machinery: with three fixes (force eager mode so the hybrid architecture doesn't hang, load the model in bf16 not fp32, and split GPU memory 55% to the inference engine with sleep-mode) the

  • Qwen3.5-4B Foofah Selective Program Fallback · cited by 2 claims

    Trust the program the moment it reproduces the visible worked examples. Doing that lifted exact-match accuracy from 55% to 62% across 250 table tasks, rescuing 18 answers the direct route got wrong wh

  • Qwen3.5-4B: Does Banking Install STRUCTURE? · cited by 2 claims

    Yes, then no. Training lifted a 4-billion-parameter model from never proposing the right step-sequence (0%) to getting it right about half the time (51%) on brand-new tasks—a real new skill, not memor

  • Qwen3.5-4B: Can the Model Localize Its Own Errors in Multi-Step Reasoning? · cited by 2 claims

    Yes, and the dip lands on the exact step, not just late in the chain. Confidence naturally climbs the deeper the model goes, so "least sure" could just mean "last step." Correcting for that, the least

  • Qwen3.5-4B: Does the Confidence Toolkit Survive on Real Code? · cited by 2 claims

    Yes, but not the obvious way. Averaging the model's certainty across every token of a program barely beats a plain majority vote among the tries. The real winner: make the model write the code, then a