Experiments

1. 2026-07-27 → 28In progress

   Self-Written Verifier Fidelity

   Fill this after the run. Separate deployable evidence from oracle/hidden evaluation.

   Benchmark GeneralizationEvidence-Conditioned SelectionInterpretability and Diagnosticsexecutionsmall-modeltable-transform
2. 2026-07-25In progress

   Real-Repo Agentic Instrument

   INSTRUMENT BUILT: 200 execution-verified stub-a-function tasks across 15 real OSS libraries (138 train / 62 held-out, firewalled by repo, scored per-test), replacing an 11-task holdout that sat at 0.818 with every task already solvable. Also established the cause of eight WSL VM deaths that had been blocking the program.

   Yes: 200 tasks over 15 libraries, each one verified two ways - the library's suite passes untouched, and deleting the target function actually breaks specific named tests. Three design traps had to be avoided, and each would have produced confident nonsense. Demanding a perfectly green test suite threw away 11 of 24 libraries over one or two unrelated environment failures, so scoring moved to named per-test sets, which ignores tests that were already failing and makes 'edit the test instead of the code' worthless. A standard editable install made the tests import the ORIGINAL source instead of the agent's copy, which would have handed every episode a free pass. And one common test runner mode hides per-test results, which silently recorded a 1,317-test library as having zero tests.

   Agentic Breadth InstallationStructured Execution and Compilersexecutionsmall-model
3. 2026-07-25In progress

   Depth Recurrence Probe

   DEPTH RECURRENCE LIFTS A FORCED SINGLE-PASS READ 2.8x (0.105 -> 0.245, frozen weights, zero tokens) AND SURVIVES FOUR ADVERSARIAL CONTROLS -- but the original 'matches chain-of-thought' framing is WITHDRAWN: the C59 anchor (0.235) was truncation-bound, and base CoT actually reaches 0.685 (97.2% among episodes that commit). Looping gets ~a third of what tokens buy.

   Partly, and the honest version is smaller than the first claim. Re-running layers 12-15 once more takes a single-pass answer from 0.105 (essentially guessing) to 0.245, on frozen weights with no tokens written, and that effect survives every check we threw at it: it is not an artifact of the answer distribution, only the one four-layer group ending at layer 15 helps, inserting a DIFFERENT group of the same size destroys the model instead (0.000), and both halves of the data replicate. What we got WRONG at first was the comparison: we said this matched what the model achieves by writing its reasoning out. It does not. That older reference number (0.235) had been measured with a generation limit far too small -- the model needs about 1781 tokens of reasoning here, more than twice that old limit -- and with room to finish it actually reaches 0.685, and is right 97% of the time whenever it finishes at all. So writing reasoning out remains roughly three times better than this architectural trick.

   Test-Time Reasoning Budgetsmall-model
4. 2026-07-19

   Qwen35 4B Agentic RLVR Feasibility

   GATE PASSED: single-GPU agentic GRPO (execution-reward RLVR) physically works for Qwen3.5-4B on one 24GB card; but the raw base produces zero reward variance (explores then quits without writing) -> a narrow SFT warm-start is the prerequisite before RLVR has signal.

   Yes to the machinery: with three fixes (force eager mode so the hybrid architecture doesn't hang, load the model in bf16 not fp32, and split GPU memory 55% to the inference engine with sleep-mode) the loop CLOSES - a movable test reward climbed 0.03->0.38 in 3 steps, proving the generating model really updates (C49 does not bite here). The full agentic loop (model drives tools, runs pytest, gets a reward) runs end-to-end. But NO to a raw-base signal: the base explores 1-2 file reads (~120 tokens) then stops without writing a fix, so every attempt scores the same -> zero variance -> zero gradient, and the GPU can't fit a big enough attempt-group (num_generations>4 OOMs) to catch the base's rare successes. So a narrow SFT warm-start to teach the explore->edit->test->iterate loop is required before RLVR can learn.

   Evidence-Conditioned SelectionInterpretability and DiagnosticsPosttraining and Adaptationexecutionlorapolicy
5. 2026-07-18

   Qwen35 4B WHY-Comment Install

   Biggest single-function gain (+5 HumanEval) but flat on the agentic loop — and complementary to the repair bet

   The WHY idea paid off where it should — on writing correct functions — and did nothing where it shouldn't. We trained the 4B to write code with the causal reason for each line attached as an inline #WHY: comment (generated by construction, no teacher). Because test-graders ignore comments, this is a perfectly clean test: if the CODE improves, teaching WHY worked. It did — HumanEval +5 problems (76.2->79.3%), the biggest single gain of the whole program, and MBPP about even. But on the multi-step agentic harness it was flat (8/35, same as base). The mirror image of the repair bet, which helped the agentic loop but less on functions. Neither result is statistically airtight on its own, but the pattern is clean and mechanistic: teaching WHY improves per-function correctness; teaching the check-and-fix loop improves multi-step behavior. They fix different things. The obvious next move is to combine them — stack the WHY model and the repair model — and see if we get both gains at once, which would also be strong enough to trust. One operational note: the standard training recipe badly underfit the WHY comments (they are high-entropy text); it took 4x the training to actually install the behavior.

   Agentic Breadth InstallationStructured Execution and Compilersexecutionrepairsmall-model
6. 2026-07-18

   Qwen35 4B Self-Repair Install

   Weak positive (first non-null): teaching the repair loop nudged real coding where teaching a passive skill did not

   The first bet that moved the needle at all — gently. We taught the 4B to debug by training on 504 examples of [buggy code + the real test-failure message] -> [diagnosis + fix], all self-generated by injecting bugs into correct code and running the tests. Result: HumanEval +3 problems (76.2->78.0%), MBPP -2 (a wash), and on the real multi-step agentic harness 8/35 -> 10/35 — and crucially the wins are asymmetric: it solved 3 tasks the base couldn't while losing only 1, so it is ADDING capability, not just trading it around like the previous (execution-tracing) bet did. Honest caveat: the numbers are small and none is statistically significant on its own, so this is a promising direction, not a proven win. The real lesson: teaching the LOOP behavior (check your work and fix it) does something teaching a passive skill (trace the code) did not. That points the way for the next bets — teach the model WHY, and stack this repair ingredient with it.

   Agentic Breadth InstallationEvidence-Conditioned SelectionInterpretability and Diagnosticsexecutionrepairsmall-model
7. 2026-07-18

   Qwen35 4B Repair + Why Stack

   Mixture dilution: combining the two winning curricula in one corpus washed out both gains

   The obvious way to combine our two promising curricula - just train one model on both - backfired. We put the 504 self-repair rows and 504 WHY-comment rows into one 1008-row training set and trained a single adapter. Instead of getting both gains, we got neither: the WHY model's +5 HumanEval collapsed to +1, and the repair model's agentic 10/35 collapsed to 7/35 (basically base). This is a known trap (we hit it in the earlier menagerie work too): when you train one small model on a mixture, each ingredient gets half the concentration and washes out. So the complementary effects are real enough to conflict for capacity, but they don't ADD this way. The right way to combine two separately-trained specialists is in weight space - keep each at full strength and add their learned changes together (task vectors) - which is the cheap next test, no retraining needed.

   Agentic Breadth InstallationStructured Execution and Compilersexecutionrepairsmall-model
8. 2026-07-18 →

   Qwen35 4B WHY-Think Scale

   Design frozen: the CORRECTED dual-channel WHY curriculum - a genuine step-by-step <think> derivation AND strippable #WHY: code comments - built to scale WHY without destroying the 4B's native thinking (GPU sweep not yet run)

   This phase built and proved the machinery; the GPU training sweep has not run yet. The generator now emits, for every example, a real hidden reasoning trace generated mechanically from the program's shape - parse the spec, pick an approach (a running total? a builder? a search?), build the code step by step, then trace one of the example's own tests by hand, running the program to get every intermediate value so the trace is true - plus the clean code with strippable WHY comments. Every example is checked by actually running it: strip the comments and the code passes all its tests; the commented version runs identically; and the hidden trace's worked example is recomputed byte-for-byte and must match real execution (a tamper of any traced number is rejected). Measured on a 5000-example sample it hits all 59 program families across 13 task kinds, keeps 100% distinct programs, carries about 1,196 distinct comment-reasoning patterns AND about 4,997 distinct think-derivation skeletons (out of 5000 - the reasoning genuinely varies, it is not one template); at 10,000 examples the think skeletons stay ~100% distinct. Nothing looks like the benchmarks (zero banned benchmark names anywhere - prompt, hidden think, or answer - and zero distinctive shared code fragments through 10,000 examples). Crucially the added thinking fits the training window: the real tokenizer render tops out at 739 tokens (median 467) against a 4096 cap, with zero examples truncated. On top sits a five-rung ladder (2000/5000/10000/20000/40000), each corpus fingerprinted so it rebuilds identically, a fail-closed trainer that trains a fresh small adapter per rung at ONE epoch (unlimited unique data, never re-shown), with a POSITIVE weight on the think channel so the native thinking is preserved and shaped rather than emptied, and a per-rung exam sweep on HumanEval + MBPP run with thinking ON and the base co-measured each rung (no stale thinking-off number carried). Tests pass.

   Agentic Breadth InstallationBenchmark GeneralizationPosttraining and Adaptationcurriculumexecutionpolicy
9. 2026-07-18 →

   Qwen35 4B WHY Scale Ladder

   Design frozen: a scale-capable, genuinely-diverse WHY curriculum + a four-rung ladder to find where the +5 fast gain peaks (GPU sweep not yet run)

   This phase built and proved the machinery; the GPU training sweep has not run yet. The core blocker was that the original WHY generator saturated fast (about 75 distinct reasons, 438 distinct programs at 504 examples), so scaling it naively would just replay data and fake a flat 'it doesn't scale' result. The rebuilt generator fixes that: it has 59 program families across 13 kinds of task (arithmetic, list work by hand, digit and remainder math, nested loops, adjacent-pair checks, small searches, simple state machines, string work, dictionary counting), and a large pool of true, line-specific reason phrasings. Measured on a 5000-example sample it hits all 59 families, about 1,196 distinct reasoning patterns (numbers and variable names stripped out), and 100% distinct programs; at 20,000 examples the programs are still 100% distinct. Every single example is checked by actually running it: strip the comments and the code still passes all its tests, the commented version runs identically, every comment names something real on its own line, and nothing looks like the benchmarks (zero banned benchmark names, zero distinctive shared code fragments). Each example also fits comfortably in the training window (about 337 tokens on average, 499 at most, versus a 4096 cap). On top of this sits a four-rung ladder (2000/5000/10000/20000), each corpus fingerprinted so it rebuilds identically, a fail-closed trainer that trains a fresh small adapter per rung with an epoch schedule that gives smaller sets more passes, and a per-rung scoring sweep against HumanEval and MBPP that the orchestrator runs one rung at a time to plot the score-vs-size curve and find the peak. 52 tests pass. The honest prior: about a 45% chance some rung clearly beats the base model, with a flat curve still a real and useful possibility that would tell us the WHY trick does not scale on this surface.

   Agentic Breadth InstallationBenchmark GeneralizationPosttraining and Adaptationcurriculumsmall-model
10. 2026-07-17

    Qwen35 4b State Track Confirmation

    Confirmed but soft: the state-tracking lift replicates directionally across six seeds, modest and noisy

    The lift held up directionally, without becoming a slam dunk. Across six fresh sealed exams, running the same seed through both models so the noise cancels, the state-tracking model beat its parent on 4 of 6 with an average edge of +0.021 aggregate — right in line with the +0.026 seen originally. By the pre-registered rule that is CONFIRMED. But being honest about the statistics: the seed-to-seed spread (SD 0.045) is wide enough that a strict paired t-test (t=1.12, 5 df) would call this not-significant on its own. That is not a surprise or a moved goalpost — the rule was written up front as a LIBERAL directional check, precisely because the decisive value here is the OTHER outcome: a clearly negative mean would have killed the result as noise, and it did not. Across all seven exam seeds now, five are positive and the mean is +0.021. So the read is: state-tracking gives a real but small and noisy general uplift — enough to make it the current best model to carry forward, not enough to call a decisive win. The install-a-universal-skill idea is directionally supported. Next the model goes in front of a real coding agent to see whether any of this proxy improvement shows up where it actually matters.

    Agentic Breadth InstallationStructured Execution and Compilersexecutionlatent-statesmall-model
11. 2026-07-17

    Qwen35 4B Exec-Trace Install

    Null: teaching the model to trace code did not make it code better (but did not hurt it either)

    The first bet at installing coding cognition came back flat. We trained the 4B on 400 self-generated, execution-verified program traces to install an accurate 'mental interpreter,' the idea being that a model that can simulate code should write and debug it better. It did not pan out: HumanEval nudged +1 problem (noise, it is near ceiling), MBPP dropped 3, and the real target — a multi-step agentic coding harness — landed EXACTLY where the base did, 8 of 35, just solving a different 5 tasks. The good news is retention held: teaching the model to trace did not wreck its ability to write code, which was the main risk. But the capability was reshuffled, not raised — the same pattern the menagerie work showed. The lesson: installing a passive skill (understanding execution) does not convert to active coding ability. The next bet targets the actual failure mode we see — the model takes one shot at a multi-step task and stops instead of planning, verifying, and repairing — by training the agentic loop directly.

    Agentic Breadth InstallationStructured Execution and Compilerslatent-statesmall-model
12. 2026-07-17

    Count-Walk Replay Compound (Stage 8)

    Bounded: replay compounding stops adding at stage 8 — the first time this reliable move failed

    The believed-likelier outcome, delivered cleanly. 'Replay compounding' — retraining on the accumulated replay mixture — had lifted the aggregate score at every previous link in this model's build chain, so it was the safest broad move available. On the eighth link it stopped working: the new model scored 0.342 versus the parent's 0.363, a genuine drop, and one task family (warren) fell far enough to trip the guardrail on its own. The pre-registered rule reads that as BOUNDED: the move has hit diminishing returns on this particular parent, which is already saturated with replay data, so more of the same just reshuffles strength between families instead of adding it (two families up, three down, net negative). The parent model stays the reference. The useful part is the knowledge: we now know the cheap reliable lever has a ceiling here, so the next push has to be a different KIND of move, not another dose of the same one. The new model still crushes the raw base model by 0.238 and wins 8 of 10 families against it.

    Agentic Breadth InstallationPosttraining and Adaptationlorasmall-model
13. 2026-07-17

    Count-Walk Menders Confirmation

    Ambiguous: the menders hit did not replicate — a control drew the same score, so no claim is made

    The answer the rule was built to force out, delivered without wiggle room. Across the four fresh exams the trained model solved a fix-the-procedure episode exactly once (plus one partial credit that the rules pre-declared doesn't count) — and on that same exam, the comparison model trained WITHOUT the special lessons solved one too. One hit when two were required, and a dead tie against a control, is the pre-written middle verdict: no claim. The clean interpretation is that occasional single-episode solves are background luck this family hands out to roughly one run in ten — the pre-registered noise rate — and the earlier headline result (the trained model scoring while all three controls sat at zero) was most likely that luck landing photogenically. The rule also pre-committed the consequence: no more exam seeds for this comparison; any future attempt at this family must be a genuinely different design, not a re-roll. One quietly encouraging descriptive note: the trained model posted the best overall score on two of the four exams (0.398 and 0.392, its two best readings ever), though those readings carry no claim.

    Agentic Breadth InstallationStructured Execution and Compilersexecutionlatent-statesmall-model
14. 2026-07-17

    Coding Fitness Harness (cognitive-core program)

    Foundation of the coding-install program: base is a strong function coder (76%) but weak agent (23%)

    Surprisingly good at writing single functions (HumanEval 76.2%, MBPP 56.5%) but weak at driving a multi-step coding task in a real agent loop (23%). The harness is validated: it agrees with an independent run to within 1 problem in 164, is deterministic, and correctly passes canonical solutions while failing wrong ones. The 76-vs-23 gap tells us exactly where to aim: not function-writing, which is nearly maxed, but the agentic cognition — planning, tracking state across edits, debugging — that turns a function writer into a coding agent.

    Benchmark GeneralizationPosttraining and AdaptationStructured Execution and Compilersabicurriculumexecution
15. 2026-07-17 →

    State-Track Installation (Stage 9)

    Installed transfer: a new, unrelated skill lifted the aggregate where replay had stalled (single seed)

    The believed-unlikelier but hoped-for outcome landed. After the reliable 'just replay again' lever hit its ceiling, this tried a genuinely different lever: teach the model one new, universal skill — keeping a running tally of named counters through a list of updates — using training data that looks nothing like any benchmark. On the sealed exam the new model scored 0.326 aggregate versus the parent's 0.300 and the raw base's 0.168, and crucially no task family dropped by more than a rounding step. The gains showed up on exactly the kinds of tasks where tracking state should help (siftstack +0.2, lockpick +0.1, mirage +0.1). So the pre-registered rule reads INSTALLED_TRANSFER: a fresh transferable skill can still add where more-of-the-same replay cannot. Two honest asterisks keep this from being a victory lap: it is ONE exam seed, and the parent's own score bounces between 0.30 and 0.36 seed-to-seed, so the 0.026 gain has to survive a multi-seed confirmation (the same check that recently exposed an earlier headline as noise) before we trust it. And the ultimate goal — beat the raw base model on EVERY family — is still not met, because one family (warren) sits below base, a weakness inherited from the parent. Real signal, properly hedged; the next step is to confirm it across seeds.

    Agentic Breadth InstallationBenchmark GeneralizationPosttraining and Adaptationcurriculumlatent-statelora
16. 2026-07-16

    Qwen35 4b Zero Root Lineage Rebuild

    The mystery ancestor priced: ninety percent of the win was the documented recipe

    ["The rebuild answered the provenance question with numbers. Retracing the six documented training steps from a truly blank starting adapter — same datasets, same seeds, same settings — produced a model with about ninety percent of the original's benchmark gain: 4.9 times the base score, seven of ten families strictly won, nothing lost. The undocumented ancestor adapter turns out to be worth about 0.04 aggregate, concentrated in exactly the three families that match its old training era — and it actually held BACK three other families, where the clean rebuild scores higher. So the formal verdict is 'degraded': the original's perfect sweeps lean on that ancestor's margin, and the clean-provenance upgrade is not available. But the honest headline is the reverse: the program's documented, contamination-free recipe does almost all of the work on its own, and a fully clean lineage extended with the proven skill-converter dose is now a mapped path.", ' ', '[', 'E', 'r', 'r', 'a', 't', 'u', 'm', ' ', '2', '0', '2', '6', '-', '0', '7', '-', '1', '6', ':', ' ', 's', 'w', 'e', 'e', 'p', '-', 'r', 'a', 't', 'e', ' ', 'f', 'r', 'a', 'm', 'i', 'n', 'g', 's', ' ', 'c', 'i', 't', 'i', 'n', 'g', ' ', '~', '5', '0', '%', ' ', 'r', 'e', 'f', 'l', 'e', 'c', 't', 'e', 'd', ' ', 'a', ' ', 'f', 'a', 'v', 'o', 'r', 'a', 'b', 'l', 'e', ' ', 'w', 'i', 'n', 'd', 'o', 'w', ';', ' ', 't', 'h', 'e', ' ', 'a', 'l', 'l', '-', 'e', 'v', 'e', 'n', 't', 's', ' ', 'r', 'a', 't', 'e', ' ', 'i', 's', ' ', '2', '/', '6', ' ', '—', ' ', 's', 'e', 'e', ' ', 't', 'h', 'e', ' ', 's', 'w', 'e', 'e', 'p', '-', 'r', 'a', 't', 'e', ' ', 'c', 'o', 'n', 's', 'o', 'l', 'i', 'd', 'a', 't', 'i', 'o', 'n', ' ', 'c', 'e', 'l', 'l', '.', ']']

    Agentic Breadth InstallationPosttraining and AdaptationStructured Execution and Compilersexecutionlorasmall-model
17. 2026-07-16

    Sweep-Rate Consolidation (Erratum: 2/6, not ~50%)

    Bookkeeping with teeth: the honest sweep rate is one-in-three, not one-in-two

    Two sweeps in six readings — one in three, not one in two — with a wide honest confidence band (roughly 4 to 78 percent at 95%). The texture matters more than the point estimate: the model never lost a single family to the base in sixty comparisons and won the overall score all six times; every failed sweep was a zero-zero tie at the same one family. The corrected figure now stands in every document that carried the old one, as visible errata; no underlying fact changed.

    Agentic Breadth InstallationStructured Execution and Compilersexecutionsmall-model
18. 2026-07-16

    Repair-Verifier Signal Probe

    Coin-flip: the model cannot even recognize the repair it cannot produce

    The gate said no, cleanly. Handed two candidate repairs and the full failure evidence — a task solvable by mentally running each candidate through both trials — the model picked the working one 51.5 percent of the time with its reasoning enabled and 49 percent without: coin-flip, below even the best word-matching shortcut the design team pre-computed and pinned at 53 percent. It was not running out of thinking room (only 7.5 percent of items hit the cap); it finishes its reasoning and still cannot tell. With recognition absent, a learn-from-your-own-attempts loop would have no signal to climb, and the last training class for the stubborn family closes by the pre-written rule. The program's map is complete: the ten-family goal stands demonstrated on two of four sealed seeds, every route to making it reliable is closed by rule rather than fatigue, and the one blocking skill is now precisely characterized — the model can neither produce, nor buy with thinking time, nor even recognize multi-constraint eliminative repairs.

    Agentic Breadth InstallationEvidence-Conditioned SelectionInterpretability and Diagnosticspolicyrepairsmall-model
19. 2026-07-16

    Qwen35 4b Menders Dose Scale

    Ten times the lessons, zero learning: the last teaching route closes

    ["The scale bet returned the cleanest possible no. After ten times the training data — eight hundred feedback-repair lessons across eight invented machines — the trained model solved exactly one of forty fresh test episodes, the same single lucky guess as both untrained comparison models. More lessons did not overcome the zero; the skill simply does not install this way. Forgetting also crept past the allowed margin against the parent. With this, every supervised-teaching route to the one family blocking the program's ten-family goal is closed by pre-written rules: three teaching styles, doses from eighty to eight hundred lessons, and the thinking-time levers. What remains: a fundamentally different training class (learning from the model's own attempts with live feedback), or standing on the program's honest position — the full ten-family sweep demonstrated on two of four sealed seeds."]

    Agentic Breadth Installationsmall-model
20. 2026-07-16

    Enumerative Repair Protocol

    The discipline installed — untrained models score zero at it — but it didn't reach the benchmark

    A split with the sharpest teaching contrast yet. On the fresh exam, the trained model produced the next-in-order untried candidate on 9 of 40 puzzles while BOTH untrained comparison models scored exactly zero — nobody does systematic enumeration without being taught it. But 9 of 40 (22.5%) fell short of the 50% bar the plan froze in advance for claiming the skill was solidly installed, and at the sealed benchmark the debugging family stayed at zero for the trained model — while, in a twist, the untrained control lucked into one item. By the pre-written rule the verdict is failed-on-its-own-terms: this teaching route closes at this dose. The diagnosis is precise, though: half the exam answers never even parsed (the long prompts break the answer format), and where the model got a legal untried candidate out, its ordering was right 56% of the time — the discipline half-took; the plumbing failed first.

    Agentic Breadth Installationrepairsmall-model
21. 2026-07-16

    Count-Don't-Walk Enumeration

    Mechanism answer: menders moves for the first time (0.1 vs all controls 0.0) — but not by the taught mechanism

    Two results that point in opposite directions, both real. The taught compact phrasing did NOT take: at the local exam the trained model still thought all the way to its 1,024-token ceiling on most puzzles (25 of 40 answers cut off, barely fewer than its untreated siblings), and it produced the correct next candidate only 7 times in 40 — nowhere near the pre-set 50% bar. The model keeps its own long-winded style no matter how short the taught examples are. AND YET: at the sealed benchmark, the trained model solved a fix-the-procedure episode that its three comparison models — the raw base, its parent, and a sibling trained on the same replay data without the new lessons — all scored exactly zero on. Under the rules written down before the run, a candidate scoring above zero with every control at zero is the pre-declared positive outcome, and this is the first time it has happened for this family in the whole program. The honest caveats: it is one episode on one exam seed, a control model once hit the same score on a different seed in the previous experiment, and whatever the lessons installed that did the converting, it visibly is not the five-line arithmetic they taught.

    Agentic Breadth Installationsmall-model
22. 2026-07-16

    Clean-Path Statechain Extension

    The skill installs anywhere; the family conversion needed the old ancestor's soil

    ["Split verdict with a clean lesson. The state-tracking dose installed for the THIRD time on its third different parent — the program's most reliable trained effect — and the clean-lineage model beat the untouched base by 2.7 times while keeping its memory inside the calibrated margins. But the headline hope failed: on the original lineage this dose had TRIPLED the protocol-compliance benchmark family, and on the fully clean lineage that conversion vanished (zero, versus the original's 0.30). The pattern reads clearly: that family was one of exactly three the undocumented ancestor adapter was good at, so the taught skill seems to convert into benchmark scores only where the ancestor's training already tilled the soil. One consolation footnote: the clean model scored a strict win on the otherwise-impossible debugging family through a lucky draw. The fully-documented model — every training step receipted from the official base, zero contamination anywhere — stands as the mission's reference artifact."]

    Agentic Breadth InstallationPosttraining and AdaptationStructured Execution and Compilerslatent-statelorasmall-model
23. 2026-07-16

    Clean Gym-Mix Dose

    Three lessons in one dose taught none of them: dilution, re-confirmed

    The mix failed cleanly and instructively. Splitting the standard 160-lesson budget across three skills — sixty trick-instruction episodes, fifty procedure chains, fifty answer-or-abstain puzzles — taught none of them: on its own fresh exam the trained model scored below BOTH untrained comparison models, while forgetting stayed safely inside the calibrated margins (the dose was inert, not harmful). This re-confirms an earlier program law on clean ground: thin slices of many skills dilute below the threshold where anything installs — the one skill that reliably installs (procedure-tracking) always got the full 160 rows in its winning runs. Two bonus findings about the measuring sticks themselves: the answer-or-abstain exam was too easy for untrained models (they nearly aced it, so it can't detect teaching), and the trick-episode exam was too hard for everyone. The three target families now need three separate full-strength experiments; the sealed benchmark seed was never spent.

    Agentic Breadth InstallationAlgorithmic Memory and RetrievalStructured Execution and Compilerslatent-statememorysmall-model
24. 2026-07-15

    Universal-Line Medium-Tier Measurement

    Two tie-flips from the goal: eight family wins, zero losses, menders and rites at zero

    The clean models' first medium-tier outing put all three at eight-of-ten family wins over the base — matching the best historical arms — and the top two lost NOTHING: they only tied on two families where both they and the base scored zero. The install-carrier model leads the aggregate and the quick-tier ranking inverted (the old quick champion came last of the trained three). The all-families goal now needs exactly two zeros flipped: menders, a genuine capability gap, and rites, which this very lineage has already scored on elsewhere.

    Agentic Breadth Installationsmall-model
25. 2026-07-15

    Statechain-Only Dose

    The lesson transferred at last — and the parent quietly swept all ten families

    Three results in one event. First, the state-tracking dose passed its local gate cleanly — the skill installed again and this time forgetting stayed inside the calibrated margin. Second, on the real benchmark the trained model TRIPLED the protocol-compliance family against both matched controls: the first time in this program a taught skill moved its benchmark family. Third, the surprise: the parent model it trained from beat the untouched base on ALL TEN families at once — the program's stated goal, recorded for the first time ever — on razor-thin margins at the two hardest families. One seed is not a claim: a confirmation run on fresh seeds with a sample-more baseline is the immediate next step.

    Agentic Breadth InstallationAlgorithmic Memory and RetrievalStructured Execution and Compilerslatent-statememorysmall-model
26. 2026-07-15

    Retention-Screen Calibration Study

    The measuring stick wobbles twice as much as the pass/fail rule assumed

    The gap wobbles with a standard deviation of 4.3 tasks — the five-task pass/fail margin was barely one wobble wide, so single-quiz forgetting verdicts were close to coin flips. Every historical 'this model forgot 5-10 tasks' reading sits inside the measured noise; averaged over four fresh quizzes the real forgetting is only 1-4 tasks. The frozen fix: future trials must average three fresh quizzes, where the same five-task margin becomes correctly sized.

    Agentic Breadth InstallationInterpretability and DiagnosticsReliability and Safetysmall-model
27. 2026-07-15

    Rank-Capacity Vehicle Cell

    Screen too noisy to judge; guard fired

    The trial could not answer the capacity question — by design. Its built-in guard required the known nine-point forgetting case to reproduce before trusting any comparison, and on this fresh screen that case measured only five. Across the last four screens the same models' forgetting scores wobble by three to four points, which rivals the five-point rule itself. The double-size module measured seven points of forgetting and a weaker skill result, but none of that is judgeable until the measuring stick is calibrated.

    Agentic Breadth InstallationEvidence-Conditioned SelectionInterpretability and Diagnosticslorasmall-modelverification
28. 2026-07-15

    Menders/Sirens + Tier Forensics

    The two-family wall was the measuring tier, not the model

    An artifact. The constants already have committed counterexamples on the line's own instrument, and the all-ten-families-at-once win happened 9 times in 94 historical medium-tier comparisons versus once in 84 quick-tier ones. On medium, the base model never maxes out any family (so strict wins are always available) and the always-one-half family spreads out to a normal range. The catch: those nine winners trained on benchmark-family data, which this program forbids — so the next step is measuring this line's own clean models on medium for the first time.

    Agentic Breadth Installationsmall-model
29. 2026-07-15

    Medium Intermediate Budget Probe

    Refused at the door again: the thinking-time lever is closed for good

    Same outcome as the first probe, one setting lower: the benchmark's wall-clock referee refused the base model at four-times thinking allowance before any trained model ran. Per the plan written before the event, a second refusal closes the thinking-time lever entirely — no more budget probes at any setting. The complete answer cost two sealed seeds and zero exposed scores. The program's honest ceiling with every currently-believable training path is nine of ten families; the stuck debugging-style family now needs a different class of idea, not another variation.

    Agentic Breadth Installationsmall-model
30. 2026-07-15

    Medium Budget-Probe Measurement

    The referee stopped it at the door: eight-fold thinking time exceeds the wall-clock budget

    The probe never got to ask its question. The benchmark's own referee enforces a wall-clock budget per model, and the untouched base — deliberately sent first because this risk was written into the plan — blew past it with eight times the thinking allowance and was refused without a score. Zero extra compute was wasted, the sealed seed is recorded as spent, and the rule book says no retry at a lower budget in this experiment. What survives: the question of whether more thinking time moves the stuck families is still open at in-between budgets, and one more probe there is the last believable test of this lever.

    Agentic Breadth InstallationEvidence-Conditioned Selectioncoveragesmall-model
31. 2026-07-15

    Interleaved-Replay Dose with Medium Pilot

    Review-round theory refuted; search closed

    The review round did not prevent forgetting: the dosed model lost nine to ten retained answers against both comparisons — almost exactly the cost of dosing directly — so the theory drawn from comparing old receipts is refuted by this direct test. The hygiene lesson won its category for the sixth straight measurement; the route-finding lesson lost this time. Per the preregistered escalation rule, the recipe search is closed: the next step is a mechanics study of WHY doses cost retention, not another schedule shuffle.

    Agentic Breadth InstallationEvidence-Conditioned SelectionInterpretability and Diagnosticspolicysmall-modelverification
32. 2026-07-15

    Hygiene-Explore De-stacked Dose with Medium Pilot

    Skills came back; forgetting blocked promotion

    On the fresh start, both reliable lessons came back decisively — the best skill-test result of the whole session (15 vs 11 and 8 of 20) — proving the earlier stall was about over-stacking practice on one model, not about the lessons. But this direct dose made the model forget ten retained answers, and the screen correctly refused it. Comparing receipts across trials isolates the cause: the one dose that never forgot had a full review round between practice doses; this one skipped it.

    Agentic Breadth Installationsmall-model
33. 2026-07-15

    Goal-Gate Confirmation

    Not confirmed at the bar — but the sweep repeated once and the gap narrowed to one family

    The replication returned a split answer. The overall improvement replicated without drama: the trained model beat the base decisively on all three fresh seeds, making four for four all-time. The perfect ten-family sweep repeated on one of the three — two full sweeps across four independent seeds — but the pre-written rule demanded two of three, so the formal verdict is aggregate-only. The near-misses are the striking part: nine-of-ten and eight-of-ten with ZERO losses, blocked purely by zero-zero ties at the debugging-style family that has resisted every teaching method. The goal now hangs on exactly one family, and the queued larger-dose experiment knows precisely what it must produce: any reliable nonzero score there. [Erratum 2026-07-16: the 'two of four seeds' framing omitted the earlier 78,150 reading; over all six recorded readings the sweep rate is 2/6 — see the sweep-rate consolidation cell.]

    Agentic Breadth InstallationStructured Execution and Compilersexecutionsmall-model
34. 2026-07-15

    Feedback-Loop + State-Chain Install

    Half the lesson took: state-tracking installed, feedback-repair failed a third time

    The dose split down the middle. The hidden-state-tracking half installed cleanly — on brand-new test instances the trained model tracked procedures better than both its parent and a matched control. The feedback-repair half failed completely: after eighty lessons on those exact toy machines, the model got zero of twenty fresh ones right, worse than models that never saw a lesson. That is the third teaching method to fail at the repair-style skill. Forgetting was judged for the first time under the new three-quiz averaging rule, which worked exactly as calibrated — and caught the trained model falling just outside the allowed margin against the control. No benchmark shot was taken; the sealed seed stays sealed.

    Agentic Breadth InstallationStructured Execution and Compilerslatent-statesmall-model
35. 2026-07-15

    Dose-Diversity Mechanism Cell

    Forgetting is intrinsic; the trade is priced

    Variety does not protect memory: the larger varied practice set also cost nine retained answers, the known ten-point case reproduced exactly, and even pure review cost five on this fresh screen — so the one past case with zero forgetting was measurement luck. Meanwhile the skills themselves keep landing: the hygiene lesson went ten-for-ten, its seventh straight category win, and the varied-dose model posted the best overall skill score with the cleanest finishing behavior.

    Agentic Breadth InstallationEvidence-Conditioned SelectionInterpretability and Diagnosticslatent-statesmall-modelverification
36. 2026-07-15

    Axis Stack Re-adjudication with Medium Pilot

    Fair screen; half the practice set is dead weight

    Judged fairly on brand-new tasks — with the ceiling quirk handled — the stacked model won the overall skill test for the third straight time (22 vs 15 and 18 of 40) with the cleanest finishing behavior again. But it won only two of the four skill categories: document-hygiene and route-finding practice stick every time, while program-repair practice never took (down to 1 of 10) and procedure-following practice teaches nothing the starting model didn't already know. Two wins fell short of the required three, so the bigger benchmark stayed sealed.

    Agentic Breadth InstallationBenchmark GeneralizationEvidence-Conditioned Selectioncoveragesmall-modeltable-transform
37. 2026-07-15

    Axis Corpus V2 with Staged Repair

    Kill rule fired; repair axis closed

    Even with lessons rebuilt from the autopsy — walking through the search step by step instead of asserting the answer — the repair skills did not stick (a tie and a loss against the comparisons), so the preregistered kill rule closed this approach for good. The screen also revealed something bigger: after three consecutive practice doses on the same model lineage, new practice stopped helping (the practiced model tied its parent and forgot a little), while a third round of plain review won the whole skill test.

    Agentic Breadth InstallationBenchmark GeneralizationEvidence-Conditioned Selectionabicurriculumrepair
38. 2026-07-14 → 15

    Fresh-Surface Budget-Commit Universal Curriculum

    New words worked; rule-guessing wall stayed

    The rewritten practice set beat both comparison models on the big screen — more right answers (69 vs 63 and 62 of 104), many fewer run-on answers (7 vs 18 and 13), and 31 percent shorter output — on vocabulary it had never trained on. But the screen also required four of eight rule-guessing tasks right, and every model scored zero of eight there, so nothing advanced to the benchmark. The budget lesson made things worse, not better.

    Agentic Breadth InstallationBenchmark Generalizationcurriculumsmall-model
39. 2026-07-14 → 15

    Goal-Gap Axis Curriculum

    Skills installed; benchmark chose review

    The targeted practice worked on its own terms — the first screen pass in this program's history (28 vs 22 and 18 of 40 on unseen tasks, with zero forgetting). On the held-out benchmark it beat the untrained model by a wide margin (0.42 vs 0.11) with seven families up, none down, and one stuck family flipped. But plain review of old material scored even higher (0.51), so the frozen rule closed the experiment. Two families never moved for anyone: program repair stayed at zero and injection resistance at exactly one-half, for every model.

    Agentic Breadth InstallationBenchmark GeneralizationStructured Execution and Compilerscurriculumexecutionrepair
40. 2026-07-14 → 15

    Qwen3.5-4B Counterfactual Plan Reflection Transfer

    Ready for review; no model result

    Not known yet. The current checkpoint builds and checks fresh list, text, and register puzzles, plus a control where every plan is deliberately assigned to the wrong puzzle. No model has been loaded or trained.

    Interpretability and DiagnosticsPosttraining and AdaptationStructured Execution and Compilersabismall-model
41. 2026-07-14 → 15

    Axis-on-Replay Stack with Medium Pilot

    Skills survived the stack; one noisy bar blocked

    The stacked model kept the installed skills (24 vs 18 and 15 of 40 on unseen tasks, with the cleanest finishing behavior of any model) and lost nothing on the retained skills. But the screen required beating both comparisons on three of four skill types, one type tied at the ceiling for the second straight trial, and another was lost to a one-off spike in a comparison model — so nothing advanced and the bigger-benchmark question was never asked. A second round of plain review made the model noticeably sloppier, undercutting the idea that review keeps compounding.

    Agentic Breadth Installationsmall-model
42. 2026-07-14

    Policy-Supported Successful-Sibling Universal Curriculum

    Balanced sibling mining cannot start

    This version cannot run its second step. Across 624 first attempts it found 227 failures, but counting and routing had none and selection had only two; the plan required four failures in every skill. It therefore stopped before sampled retries or training.

    Agentic Breadth InstallationBenchmark GeneralizationEvidence-Conditioned Selectioncurriculumpolicysmall-model
43. 2026-07-14

    Natural-Language State-Table Universal Curriculum

    Design frozen; no model result

    Not known yet. CPU construction produced 80 truth-checked lessons and two 320-row streams with exactly 286,814 tokens each. All 48 smoke tests pass, but no model has been trained or evaluated.

    Agentic Breadth InstallationBenchmark GeneralizationEvidence-Conditioned Selectionabicounterexamplescurriculum
44. 2026-07-14

    Search-Scaffold Universal Curriculum

    Staged search fails its local mechanism gate

    No. On 26 fresh paired cases, the staged-search model tied replay at 16 correct and trailed its parent at 18. All three parsed 23 answers and hit the limit three times; the candidate solved none of four execute/induct cases and neither probe-selection case.

    Agentic Breadth InstallationBenchmark GeneralizationEvidence-Conditioned Selectioncurriculumexecutionlatent-state
45. 2026-07-14

    Residual-Skill Successful-Sibling Universal Curriculum

    Stopped: too few correct retries

    Grading the 3,600 retries found 855 short fully-correct ones, and nine of ten weak skills had plenty. But rule-guessing (induction) yielded a usable correct retry on only 2 of its 46 failed tasks, below the required four, so the frozen rule stopped the experiment before any training. No model was trained and no benchmark was touched.

    Agentic Breadth InstallationBenchmark GeneralizationPosttraining and Adaptationcurriculumdistillationsmall-model
46. 2026-07-14

    On-Policy Failure-Prefix Universal Curriculum

    Actual failure-prefix lessons lose to replay

    No. The unmodified parent solved 16 of 26 fresh tasks and equal-compute replay solved 18, while training corrections after the model's own real failures solved only 15. The repair model got none of the six targeted execution, induction, or probing cases right, and it hit the answer limit three times. Broad evaluation stayed sealed.

    Agentic Breadth InstallationBenchmark GeneralizationPosttraining and Adaptationcurriculumpolicyrepair
47. 2026-07-14

    Failure-Selected Counterfactual Restart Curriculum

    Clean restarts are frozen; model test pending

    The data-selection stage passed: from 624 fresh attempts, the frozen rule selected 52 clean restarts, exactly four for each of 13 skills. Forty correct accuracy or cap failures and twelve correct-but-too-long cases are now frozen; training still waits for an exact exposure match to replay.

    Agentic Breadth InstallationBenchmark GeneralizationEvidence-Conditioned Selectioncounterexamplescurriculumsmall-model
48. 2026-07-14

    Qwen3.5-4B Tokenizer-EOS Residual Mechanics Fresh Replay

    Fresh replay is designed, not yet run

    No result exists yet. The scaffold declares seven freshness controls and eight lifecycle controls, reads none of the parent's sampled bundles, and authorizes no model call until independent review and release gates pass.

    Structured Execution and Compilersabismall-model
49. 2026-07-14

    Qwen3.5-4B Tokenizer-EOS Answer Commit Factorial

    Boundary works; capability test failed instrumentally

    The boundary worked cleanly: both no-thinking chat-end conditions produced 48 correct strict answers out of 48, while every matched later-end condition produced zero. The harder transport check also scored 24 of 24. But the final comparison never completed because the analysis code incorrectly demanded that later completed stages be absent.

    Structured Execution and Compilersabismall-model
50. 2026-07-14

    State-Formation Branch Authorization Recovery

    Retry path passed its safety checks

    Yes for the no-model safety check: all six controls passed, the failed first attempt has a third preserved copy, and the two retry-blocking source paths were retired only after that archive commit passed both checks. The actual full-size model retry still waits for the retirement commit to pass.

    Benchmark GeneralizationPosttraining and AdaptationStructured Execution and Compilerslatent-statelorasmall-model
51. 2026-07-14

    State-Formation Branch Handoff Recovery

    Safe handoff is working

    Yes, the narrow handoff passed every safety check, then all three full-size setup controls reached 48 of 48 with their learned path and zero of 48 with that path disabled.

    Benchmark GeneralizationStructured Execution and Compilerslatent-statesmall-modeltable-transform
52. 2026-07-13 → 14

    Close-Weighted Universal Commit Seam

    Close weighting missed the local gate

    No. On 26 fresh procedural cases, ordinary and close-weighted target training both produced 23 well-formed answers and three response-limit contacts. Close weighting scored 16 correct versus 15 for ordinary training, but only by gaining one unrelated abstention case; both remained zero for four targeted execution and rule-finding cases. The best arm missed each frozen numeric gate by one case or contact, so the benchmark stayed sealed.

    Agentic Breadth InstallationBenchmark GeneralizationPosttraining and Adaptationcurriculumlorasmall-model
53. 2026-07-13 → 14

    State-Formation Analysis Recovery

    Recovery path is ready

    Yes, the narrow recovery check passed. It accepts only the one registered file location, still rejects unsafe shortcuts, and has not yet examined any model result.

    Benchmark GeneralizationPosttraining and AdaptationStructured Execution and Compilerslatent-statelorasmall-model
54. 2026-07-13

    Validation-policy counterexample curriculum

    The lesson was already mastered

    The test could not answer the question. Before the newly trained model was allowed to run, the starting model and the equal-size old-lesson control each fixed all 48 practice cases. The required gains were therefore impossible, so the run stopped before transfer or the external benchmark.

    Agentic Breadth InstallationBenchmark GeneralizationEvidence-Conditioned Selectioncounterexamplescurriculumpolicy
55. 2026-07-13

    Replay-Anchored Universal Curriculum Continuation

    Designed mix loses; replay-only sets a stronger anchor

    No at this dose. The designed mix passed its fresh synthetic screen but scored about 42% overall, below both the 44% mature policy and the 49% replay-only comparison. It also fell below base on one family. Replay-only was the useful surprise: it improved eight of ten families, tied the other two, and regressed none.

    Agentic Breadth InstallationBenchmark GeneralizationPosttraining and Adaptationabicurriculumsmall-model
56. 2026-07-13

    Mid-Density Token-Matched Universal Curriculum

    160 lessons nearly pass; benchmark stays sealed

    The 160-lesson stream improved fresh local accuracy from 17 to 19 of 26 cases, raised readable answers from 18 to 23, and cut answer-limit contacts from nine to three. It still missed the readability and length gates by one case each. Adding 80 more lessons erased the accuracy gain, so no model advanced to broad evaluation.

    Agentic Breadth InstallationBenchmark GeneralizationPosttraining and Adaptationcurriculumpolicysmall-model
57. 2026-07-13

    Low-Density Token-Matched Universal Curriculum

    Low-density doses failed the local gate

    No. Neither 40 nor 80 abstract lessons passed the fresh local screen. The stronger 80-lesson stream matched the inherited anchor at 54% accuracy and parsed 62% of answers, but still hit the answer cap 10 times; broad evaluation stayed sealed.

    Agentic Breadth InstallationBenchmark GeneralizationEvidence-Conditioned Selectioncounterexamplescurriculumsmall-model
58. 2026-07-13

    Qwen3.5-4B: Installing Universal Features via Designed Synthetic Curricula

    Synthetic skills installed locally but displaced broad behavior

    Not with either tested training geometry. Continuing from the strong policy raised fresh synthetic accuracy from 50% to 69% and beat the untrained base overall, but it scored 31% aggregate versus 45% for the strong starting policy and fell below base on three of ten families. A second mixture trained from base with broad replay also reached 69% local accuracy, yet only 85% of its answers were well formed and it hit the response limit four times, so its broad test stayed sealed.

    Benchmark GeneralizationEvidence-Conditioned SelectionInterpretability and Diagnosticscurriculumsmall-model
59. 2026-07-13

    State-Formation Capacity Adjudication

    Capacity test is ready, not yet run

    Not run yet. The reviewed test starts with three compact-update training runs. If any required state check misses, six matched controls become mandatory; the final three runs open only if the full-size version also misses.

    Evidence-Conditioned SelectionInterpretability and DiagnosticsOperator and Skill Inventoriescounterexampleslatent-statelora
60. 2026-07-13

    Full-Rank Extra-R Delta: State-Carry Versus State-Bag

    Full-size pilot was blocked, not decisive

    The run worked mechanically, but it did not settle the question. All 892 million full-size update weights trained and fit comfortably, yet macro task-mean joint-state accuracy was only 0.28% against a 40% requirement. At the same time, Carry scored 1.6 points below equal-compute pooling and failed both required query-kind checks. The frozen rules classify that combination as promotion blocked, not as an isolated capacity failure. The full-size construction also shifted the random initialization of the shared state modules relative to the LoRA run.

    Posttraining and AdaptationStructured Execution and Compilerslatent-statelorasmall-model
61. 2026-07-13

    Semantic-policy headroom tournament

    The gap is before the failed test

    No rule produced a stable training target after a failed test: every one of 72 cases reached a correct patch. But before seeing test output, the model got zero of 54 ambiguous first patches fully right and never opened the visible tests first. The run formally stopped because answers also exceeded their length allowance too often.

    Active Evidence AcquisitionAgentic Breadth InstallationBenchmark Generalizationabiactive-learningpolicy
62. 2026-07-13

    Qwen3.5-4B Semantic-Anchor Coordinate Branching

    The late anchor writes names, but the consequence test is invalid

    This run cannot establish that. The internal edit strongly changed the model's choice among candidate names, but none of 440 consequence outputs began with a valid answer token. Even in a restricted twelve-choice readout, the edited state chose the intended consequence only 5 of 44 times, nearly the 4-of-44 control rate. A post-run audit also found that two rotating maps canceled, leaving the final name-to-label relation fixed across tasks.

    Benchmark GeneralizationInterpretability and DiagnosticsStructured Execution and Compilersbridgelatent-statesmall-model
63. 2026-07-13

    Qwen3.5-4B Materialized Residual Sibling Search Fresh Replication

    Cheap materialized ranking fails; free generation has no valid answer interface

    The generation test cannot answer the question because every reasoning trace reached its length cap and the materialized arm parsed only 12 of 52 outputs, with zero successes. The separate one-token ranking test is a clean negative: materialized recall among viable first moves was 0.257, below names alone, shuffled states, listwise choice, and a surface-only rule.

    Evidence-Conditioned SelectionInterpretability and DiagnosticsStructured Execution and Compilersabilatent-statesmall-model
64. 2026-07-13

    Qwen3.5-4B Materialized Residual Sibling Search

    Model-free design is ready for mechanics

    Not yet. The scientific design and every model-free construction check pass, but the model has not run. The frozen test contains 264 fresh functions, and all 38,596 planned prompt renderings fit their assigned context limits.

    Evidence-Conditioned SelectionInterpretability and DiagnosticsStructured Execution and Compilersabismall-model
65. 2026-07-13

    Qwen3.5-4B Materialized Residual Answer-Seam Factorial

    Strict answer seam failed; native commit token isolates one fresh repair

    No registered style qualified: all four scored zero strict parses out of 48, so mechanics stayed sealed. Removing only the final chat-end marker and newline made both no-think styles exact on all 48 rows; thinking still had extra close markers. That post-run clue motivates a fresh answer-stage commit-boundary test, not a retroactive pass.

    Benchmark GeneralizationStructured Execution and Compilerssmall-model
66. 2026-07-13

    Qwen3.5-4B Jacobian Counterfactual Branching

    Balanced internal nudges fail to steer real reasoning

    No. Across all three allowed strengths, the meaningful nudge made its assigned answer win only 4 of 48 times—exactly the one-in-twelve chance rate and identical to a generic nudge. The probabilities barely moved even at the strongest setting. Because the write itself failed, no expensive continuation or correctness run was opened.

    Interpretability and DiagnosticsPosttraining and AdaptationProcess Control and Tool Usepolicysmall-model
67. 2026-07-13

    Qwen3.5-4B Early Text Hypothesis Forking

    A 24-way early-hint test has cleared its model-free checks

    The experiment design and model-free checks now pass, but the model has not run. Review expanded the first draft from twelve operation names to twenty-four fully specified operations, added two fair late-hint comparisons, and proved that all 576 possible two-step rules are covered by the public-data control. The next gate tests whether the model can actually use these hints rather than merely repeat them.

    Evidence-Conditioned SelectionInterpretability and DiagnosticsPosttraining and Adaptationexecutionpolicysmall-model
68. 2026-07-13

    Qwen3.5-4B Counterfactual Order-Support Selector

    Meaningful order helps, but not enough to trust this selector

    Not reliably. The meaningful-order rule got 43 of 113 puzzles right, clearly better than first attempt or majority vote. But it was only two puzzles ahead of simply choosing the most decisive attempt, and a deliberately mismatched scrambled scratchpad did one puzzle better. Those small differences could be noise, so the separate confirmation set stayed closed.

    Evidence-Conditioned SelectionInterpretability and DiagnosticsStructured Execution and Compilersabismall-model
69. 2026-07-13

    Counterfactual evidence-acquisition curriculum

    The chosen start and anchor were too far apart to test the curriculum

    The run stopped at its first checkpoint-compatibility test. Across 48 frozen contexts, the starting checkpoint differed from the comparison anchor by 0.110735 centered logits, above the fixed 0.100000 limit. Entropy stayed healthy, but the failed locality gate sealed behavior sampling and training. This therefore says nothing about whether the evidence-acquisition curriculum would work.

    Active Evidence AcquisitionAgentic Breadth InstallationBenchmark Generalizationactive-learningcurriculumpolicy
70. 2026-07-13

    Qwen3.5-4B: The Compute-Optimal Confidence Policy (select + abstain + escalate)

    Verifier-free SELECT + ABSTAIN work; selectively escalating hard tasks to more think budget is a null result at n=400

    Partly. Picking the answer with the highest single-token P(True) confidence is the best grader-free selector at every compute level, and abstaining below a confidence threshold trades coverage for accuracy cleanly — both robust. But the appealing third move — spending compute on more THINKING for the flagged-hard tasks rather than more samples — showed no benefit once powered up to 400 tasks: an early small-sample win did not replicate (every 95% interval now includes zero). Thinking longer helps a little across the board, just not selectively on the hard tail — because these coding tasks nearly saturate the model's thinking budget already.

    Evidence-Conditioned SelectionInterpretability and DiagnosticsPosttraining and Adaptationabipolicysmall-model
71. 2026-07-12

    Transaction-invariant recovery curriculum

    The structure transferred but one exception did not

    Partly. Focused training lifted practice repairs from 52% to 82% and preserved the test-and-revise loop. But on different transaction APIs it reached 72% versus 70% before training, far below the required ten-point gain. Every target first patch copied state, checked the whole request, and updated atomically, yet all missed the separate negative-amount exception.

    Agentic Breadth InstallationBenchmark GeneralizationEvidence-Conditioned Selectioncurriculumlatent-statepolicy
72. 2026-07-12

    Verifier-conditioned recovery banking curriculum

    A tiny slice of text broke unrelated behavior

    Yes — replaying real failures taught recovery, lifting success from about half of cases to more than nine in ten. But the best version also added a little "explain your fix" text, and that supposedly tiny slice actually carried most of the training force, disturbing unrelated knowledge past a safety limit. The run stopped before it could even test whether recovery transfers to new code.

    Agentic Breadth InstallationBenchmark GeneralizationEvidence-Conditioned Selectioncurriculumlatent-statepolicy
73. 2026-07-12

    State-Carry Versus State-Bag Counterfactual

    LoRA did not form the running state this test needed

    The first matched test did not answer that architecture question because the low-rank update failed to learn the required running state. Carrying memory improved overall accuracy by only 4.3 points, with uncertainty spanning a slight loss to a 9.4-point gain. More decisively, the model decoded the complete joint hidden state on just 0.46% of registered steps, far below the preregistered 40% floor, and swapping memory between examples did not make answers follow the donor. The three-run confirmation therefore stopped exactly as planned.

    Posttraining and AdaptationStructured Execution and Compilerslatent-statelorasmall-model
74. 2026-07-12

    Qwen3.5-4B Same-Prefix Advantage Routing

    Being best overall isn't being best right here

    Only when it replicates, and here one helper didn't. The test actually ran: the deep specialist's local edge held up on a fresh, separate batch, but the quick specialist looked strong on one batch of stuck points and then went slightly negative on a second, so the check refused to trust it and the whole plan was called off. The insight is the reframe: a leaderboard tells you which teacher wins on average, not which one helps at a specific stuck moment, and here the stronger teacher's overall edge barely clears the bar. So each teacher must prove itself where the student fails, on fresh samples, or it sits out.

    Agentic Breadth InstallationBenchmark GeneralizationEvidence-Conditioned Selectioncounterexampleslatent-statesmall-model
75. 2026-07-12

    Repository search-compress-bank coding curriculum

    Training on clean wins erases recovery skill

    No. It became flawless on the six codebase types it trained on, solving every task in exactly four clean steps. But on four never-seen codebase types its success collapsed from 68% to 35% — worse than just running the untrained model twice, which scored 53%. Stripped of every failed attempt, it never learned what to do when a fix gets rejected.

    Agentic Breadth InstallationBenchmark GeneralizationEvidence-Conditioned Selectioncurriculumexecutionoperator-search
76. 2026-07-12

    Public-verifier recovery branch tournament

    Complementary repairs did not transfer

    No. On four new problem types, both policies repaired 74%, and their combined best reached only 75%—exactly tied with two action-only attempts. The selector was correctly stopped before scoring.

    Agentic Breadth InstallationEvidence-Conditioned SelectionInterpretability and Diagnosticsabismall-modelverification
77. 2026-07-12

    Locality-first recovery-reason interpolation

    Safe blend beat both parents, gates stopped it

    Yes on skill, no on shipping. One dial setting recovered from broken code 97% of the time, about 12 points above the act-only version and 15 above a matched-training baseline, while barely moving unrelated behavior. But it tripped two automatic release checks, so the test halted before the transfer stage. On inspection, both checks were mis-specified, not the model.

    Agentic Breadth InstallationPosttraining and AdaptationProcess Control and Tool Usesmall-model
78. 2026-07-12

    Payload-capable recovery agent harness

    Recovery gain did not beat its incumbent

    No. It led on the first unseen task block at 71%, but confirmed at 69%, exactly tied with the action-only model and short of the required lead. The external benchmark stayed sealed.

    Agentic Breadth InstallationProcess Control and Tool UseStructured Execution and Compilersabismall-model
79. 2026-07-12

    Qwen3.5-4B Pareto Policy Integration

    The specialist built for speed lost on speed

    No. Retested on fresh, uncontaminated tasks, the version built for quick work actually lost at quick work by about two points, while the version built for long work won at both quick AND long tasks. One version quietly dominated everywhere, so there was no genuine trade to fuse — the whole merge was cancelled before any training ran.

    Agentic Breadth InstallationEvidence-Conditioned SelectionPosttraining and Adaptationabicounterexamplesdistillation
80. 2026-07-12

    Qwen3.5-4B Native-Thought Seam Budget Ladder

    It never stops thinking to give an answer

    No. Across 48 tries on simple list-transformation puzzles, and at every thinking budget up to 1,024 tokens, the model closed its reasoning and produced an answer exactly zero times. It always burned the whole budget still analyzing. To be usable it needed to stop on its own at least 80 percent of the time, and it never once did. This was not a stuck repeating loop; it kept genuinely reworking the problem, just never committing.

    Benchmark GeneralizationInterpretability and DiagnosticsTest-Time Reasoning Budgetcurriculumsmall-model
81. 2026-07-12

    Qwen3.5-4B Native-Thought Jacobian Value Transport

    The model never stopped thinking to answer

    We could not even reach the test. The whole plan needs the model to finish reasoning and write an answer worth grading. But on all 48 attempts at simple two-step list puzzles, it rambled straight into its 160-token thinking limit and never concluded. Zero natural finishes. That is a setup-and-budget wall, not proof the hidden gauge is missing.

    Interpretability and Diagnosticssmall-model
82. 2026-07-12

    Qwen3.5-4B Jacobian Value Transport

    You can edit the word, not the meaning

    No. Editing one internal direction at a late layer flipped the concept the model said aloud on 18 of 24 tries, up from never, and far beating an ordinary readout-style edit that worked only 1 in 5 times. But when that concept had to feed a follow-up calculation, the edit changed nothing on all 24 items, at every layer. It moved the model's mouth, not its mind.

    Interpretability and DiagnosticsStructured Execution and CompilersTest-Time Reasoning Budgetsmall-model
83. 2026-07-12

    Qwen3.5-4B Jacobian Transport Control Replication

    Rewrite a model's early concept, later steps obey

    Yes. Editing a handful of internal numbers at the early moment the model names its concept made it answer as a completely different concept on all 48 fresh test items, and a separately computed digit that depends on that concept flipped to match too. Same-size random edits changed nothing, and knowing only the output direction changed nothing: the effect needs the real concept numbers.

    Interpretability and Diagnosticsrepairsmall-model
84. 2026-07-12

    Qwen3.5-4B Forced-Commit Jacobian Value Transport

    A stop signal is not an answer slot

    No. Inserting the model's own "done thinking" marker cut the reasoning off but almost never flipped it into answer mode. Across three thinking budgets, only 13% to 19% of forced stops produced anything readable, and just 1 of 48 was correct, versus the 90% readability the test demanded. Usually the model simply restarted its analysis instead of committing.

    Interpretability and DiagnosticsTest-Time Reasoning Budgetsmall-model
85. 2026-07-12

    Qwen3.5-4B Deep-Advantage MOPD

    Right teacher signal, wrong update recipe

    No. The deep specialist really was the better teacher on selected states, and copying it worked slightly better than copying the wrong teacher or training on matched non-winning states. But after four rounds the resulting model still scored below the deep specialist, below a simple weight blend, and far below giving the original blend eight tries. Broad retention and untouched transfer survived, so this is a failure to add capability rather than a general collapse.

    Agentic Breadth InstallationBenchmark GeneralizationPosttraining and Adaptationabidistillationsmall-model
86. 2026-07-12

    Qwen3.5-4B Context-Local Jacobian Clamp

    Perfect concept edit, undone by a rounding error

    Yes, but only when the edit lands on the earlier token that first stores the word. There the model looked up the swapped word's digit on all 48 fresh puzzles, up from zero without the edit, and a wrong-word swap produced that wrong word's own digit, not noise. Yet one of 96 control rows slipped past a precision limit, voiding the formal claim.

    Interpretability and DiagnosticsPosttraining and AdaptationProcess Control and Tool Usepolicysmall-model
87. 2026-07-12

    Qwen3.5-4B Commit-Slot Semantic Power Replication

    Reasoning order matters; one J-value map does not work

    The reasoning result is real, but the proposed internal value meter failed. Ordered scratch work beat the same words shuffled in two independent stages (about 29% versus 14% on confirmation). Yet a task-held-out model of all 120 J-space coordinates ranked better and worse final paths no better than chance overall (50.2%). It looked useful halfway through the thought (60.8%) but reversed near the answer (39.6%), while simpler margin and generic hidden-state readouts were stronger.

    Interpretability and DiagnosticsStructured Execution and CompilersTest-Time Reasoning Budgetlatent-statesmall-model
88. 2026-07-12

    Qwen3.5-4B Commit-Slot Jacobian Value Transport

    Forcing the format made it answer, not reason

    Barely. Forcing the format fixed one problem outright: an allowed word was the model's top choice 85% of the time, versus 4% when it answered freely. But real step-by-step thinking beat the very same thought words scrambled into random order by only about eight points, small enough to be noise, and the edge piled onto just one or two easy words.

    Interpretability and DiagnosticsStructured Execution and CompilersTest-Time Reasoning Budgetlatent-statesmall-model
89. 2026-07-12

    Qwen3.5-4B Balanced-Core Answer-Potential SFT

    Does answer likelihood pick the best reasoning to

    Not run yet — the reasoning bank is fully built (360 tasks, six competing selection rules staged) but no model has been trained or scored. The built test will fine-tune six copies, each fed reasoning chosen a different way, then measure which answers fresh sealed problems most accurately. A key rival is simply keeping the shortest complete reasoning, which may quietly win.

    Evidence-Conditioned SelectionPosttraining and AdaptationStructured Execution and Compilersabilorarepair
90. 2026-07-11

    Entropy-routed think-pivot optimization round 2

    A one-word nudge shifts the whole model

    No. Gently pulling the better word up at 155 hand-picked wrong-turn moments was cleaner than shoving the bad word down, and it carried real signal, beating a scrambled-label control by about 14 points on fresh coding repairs. But it still lost to the untouched model, 54% versus 60% of repositories fixed, and the nudge leaked into unrelated words well past the safety limit.

    Agentic Breadth InstallationPosttraining and AdaptationStructured Execution and Compilersabilatent-statesmall-model
91. 2026-07-11

    Qwen3.5-4B Specialist Policy Integration

    One nearly-perfect skill made the plan impossible

    No — we never found out. Each of four required skills had to gain ten points before merging, but scores top out at 100. One tool-use skill already sat at 99.4%, so its target of 109.4% was impossible. The experiment stopped before a single training step. The real lesson: broad room on the overall task hid one corner already maxed out.

    Agentic Breadth InstallationBenchmark GeneralizationEvidence-Conditioned Selectioncounterexamplesdistillationpolicy
92. 2026-07-11

    Interactive policy curriculum: oracle DAgger to execution-reward RL

    Correcting its live mistakes made it stop verifying

    No — it backfired badly. Success on the practice tasks fell from about 60% to 35%, and on tasks never touched in training from about 69% to 35%. It kept editing fluently but stopped running its checks: the verify-and-commit step was only about 1 in 40 correction examples, so it washed out entirely. On one repair family it never verified once, scoring zero.

    Active Evidence AcquisitionAgentic Breadth InstallationBenchmark Generalizationactive-learningcurriculumdistillation
93. 2026-07-10 → 11

    Think-block FTPO round 1: outcome-conditioned pivot steering as an agentic install recipe

    Nudges fix confident mistakes, not coin-flip forks

    No. Nudging at those forks made the model worse, not better — success on fresh tasks fell about 4 to 8 percent instead of clearing the 5-point gain hoped for. The tell: feeding it deliberately scrambled labels did nearly the same damage, so the harm came from the nudging procedure itself, not the success signal. The nudge only helps when the branch it discourages was already a runaway favorite.

    Agentic Breadth InstallationPosttraining and AdaptationStructured Execution and Compilersabilorasmall-model
94. 2026-07-10 → 11

    Gauntlet frontier: difficulty escalation past the breadth-install plateau

    Fast or slow tasks

    No, not both at once. Piling on more, harder, or more varied self-generated practice all stalled at the same ceiling, and even hand-written expert solutions the model could not discover on its own failed to move it. A new trick, self-training the model's own shortest correct reasoning into its weights, finally cracked the slow-reasoning tier, but only by ceding ground on the fast one.

    Agentic Breadth Installationsmall-model
95. 2026-07-10

    Qwen3.5-4B Verified Macro Invention Long-Context Rerun

    More thinking room only bought longer loops

    No. The model was not short on thinking space, it was stuck repeating itself, and more space only bought longer loops. Handed the blueprint, all 16 tasks finished cleanly. Asked to invent the program from examples alone, every attempt ran off the end of its allowance at every budget, and pure repetition climbed from 9 percent of traces to 83 percent as the room grew.

    Active Evidence AcquisitionBenchmark GeneralizationEvidence-Conditioned Selectionactive-learninglatent-statesmall-model
96. 2026-07-10

    Qwen3.5-4B verified-macro exact CUDA-graph vLLM rerun

    More thinking room only deepened the loops

    No, it just loops harder. At both budgets, all 48 reasoning attempts ran out of thinking room and had to be force-stopped, none finishing on its own. Widening the budget from 49,152 to 61,440 tokens pushed the share stuck endlessly repeating the same lines from 79% up to 83%. More space feeds the loop rather than breaking it.

    Evidence-Conditioned SelectionInterpretability and DiagnosticsOperator and Skill Inventoriescounterexamplessmall-modelverification
97. 2026-07-10

    Qwen3.5-4B verified-macro capacity-fit vLLM rerun

    Fixing the memory crunch just revealed endless looping

    No. Two hidden limits bit first. Configuring the server for 64 simultaneous long prompts demanded about 2.4 million tokens of fast memory from a pool holding roughly 1 million, forcing constant re-reading. Cutting to 19 prompts fixed that overflow — but the model then looped instead of finishing: 37 of 48 attempts fell into exact repeating cycles, and none stopped on their own.

    Evidence-Conditioned SelectionInterpretability and DiagnosticsOperator and Skill Inventoriessmall-modelverification
98. 2026-07-10

    Qwen3.5-4B Partial-Structure Recognition-Guided Search

    Extra reasoning made half-finished-program judging worse, not better

    No. Shown a half-finished program skeleton, the four-billion-parameter model's guess at whether it could still be completed was barely above a coin flip — about 51% correct, where 50% is pure chance — and letting it reason step-by-step scored lower than a snap one-word yes/no. Yet an exact search reading the same skeletons pruned the work 262,144-fold, so the signal was there; the model just could not extract it.

    Benchmark GeneralizationEvidence-Conditioned SelectionStructured Execution and Compilersexecutionsmall-modeltable-transform
99. 2026-07-10

    Qwen3.5-4B Long-Horizon Answer-Potential SFT

    Shortest reasoning outranks the clever quality score

    Barely. That confidence signal does sort stronger reasoning above weaker about 60% of the time — better than a coin flip's 50%, and enough to lift a picked chain's success from 16% to 22%. But two dumb rules beat it: just keeping the shortest chain, or the most typical one, each rank correctly near 70%. The real training test is still pending.

    Evidence-Conditioned SelectionPosttraining and AdaptationTest-Time Reasoning Budgetsmall-model
100. 2026-07-10

     Qwen3.5-4B Answer-Potential Trace SFT

     The signal was real but graded an unreachable

     No. The score carried genuine signal — it tracked what the reasoning actually said, not its length or wording — but its top pick raised fresh-answer success only from about 13% to 20%, short of the useful margin needed to bank it. The deeper flaw: the model almost never stopped thinking on its own, so the score rated an answering moment it could rarely reach.

     Evidence-Conditioned SelectionPosttraining and AdaptationReliability and Safetycounterexampleslatent-statelora
101. 2026-07-09 → 10

     Gauntlet round 1: breadth-first agentic expert iteration

     It knew the answers but never wrote them

     Mostly the second. On a blind benchmark the model scored about 14 percent, with six task types near zero, but it had usually reasoned correctly. It simply hit its thinking limit, restarted explaining instead of writing the answer line, and scored blank. Teaching it to commit an answer from a cut-off chain lifted scores to 36 and 45 percent, and the fix carried to tasks it never practiced.

     Agentic Breadth InstallationEvidence-Conditioned SelectionInterpretability and Diagnosticsabilorasmall-model
102. 2026-07-09

     Qwen3.5-4B Verified Macro Invention

     Perfect shortcut syntax masked wrong rewrites

     No. Even handed the finished plan and asked only to re-express it — no problem to actually solve — the model got it exactly right just one time in four, short of the three-in-four bar. Every output looked flawless: valid syntax, a shortcut always called, nothing cut off. It named the right first shortcut, then invented extra ones that quietly expanded into different steps.

     Benchmark GeneralizationEvidence-Conditioned SelectionInterpretability and Diagnosticssmall-modelverification
103. 2026-07-08 → 092 figures

     Does the installable hypothesize-and-verify skill move the structure wall?

     A taught routine sticks to exactly its practiced

     No. Fine-tuning the model on 1,476 worked guess-and-check traces doubled its success at the two-step depth it practiced on (lists jumped from 37% to 70%), yet did nothing one step deeper: three-step success stayed near 5%, statistically flat and no better than an untaught model. The skill installed cleanly, then refused to stretch a single operation further.

     Evidence-Conditioned SelectionInterpretability and DiagnosticsPosttraining and Adaptationabiexecutionlora
104. 2026-07-082 figures

     Qwen3.5-4B: Does Code Confidence Replicate on HumanEval?

     One yes-or-no check beats averaging the whole answer

     Ask it directly. Averaging the model's confidence across every code token barely helps, nudging correct picks from 77% (blind guessing) to just 79%. But handing the model its own code and reading its confidence in one yes-or-no verdict jumps that to 84% — near the 87% ceiling of a flawless picker. How you ask matters more than whether you ask.

     Benchmark GeneralizationEvidence-Conditioned SelectionInterpretability and Diagnosticssmall-modelverification
105. 2026-07-07 → 081 figure

     Qwen3.5-4B: Can Confidence Replace the Verifier in the Banking Flywheel?

     Confidence can rank answers but can't clean training

     No. When training on answers checked by actually running the code lifted single-shot accuracy from 8% to 24%, confidence-filtered data — fifteen times purer than a random grab of the model's own outputs — landed right on the no-filter floor at 12%, adding nothing. Self-training tolerates almost no wrong examples, and the confident mistakes a filter lets through are plausible near-misses that teach the worst habits.

     Evidence-Conditioned SelectionInterpretability and DiagnosticsPosttraining and Adaptationabiexecutionsmall-model
106. 2026-07-071 figure

     Qwen3.5-4B: Does the Model Know When It Will Fail?

     Knows when it's guessing, but can't say so

     Yes, but only in its numbers, never its words. The probability it quietly places on the digit it writes sorts right answers from wrong ones almost perfectly — 95 out of 100, versus 50 for a coin flip — and beats guessing from surface prompt features alone (61). Yet ask it outright "is your answer correct?" and it scores 46, no better than chance.

     Benchmark Generalizationsmall-model
107. 2026-07-073 figures

     Qwen3.5-4B: Can SFT Install the Skill of Induction?

     Small models crack new rules only by reasoning

     Yes, but only when it thinks out loud. The trained model cracks brand-new hidden rules perfectly when it writes out its reasoning, yet collapses to worse-than-guessing — about 1 in 100 — when forced to answer in a single step. The skill lives in the written steps, not the weights, and taught as a general try-and-check method it even transfers to rule types it never trained on.

     Posttraining and Adaptationlorasmall-model
108. 2026-07-071 figure

     Qwen3.5-4B: Does the Compositional Wall Exist in Language?

     A reasoning wall for code, not language

     No. Written as ordinary sentences, the model follows a chain of invented names near-perfectly through four hops (94 to 100 percent correct), far above the roughly 4 percent a blind guess earns. The three-step ceiling is really about code-style formatting: the identical chain shown as a code lookup table drops it to 3 percent at a single hop, because the model copies the table instead of tracing it.

     Benchmark Generalizationsmall-model
109. 2026-07-071 figure

     Qwen3.5-4B: Does the Structure-Proposal Wall Exist in Language?

     Great at following rules, blind at finding them

     No. Handed the rule outright, the model applies it correctly 86% of the time. Asked to infer that same one-step rule from worked examples, it scores 0% — below even the roughly 6% that pure guessing would earn. Letting it reason step by step lifts inference only to a coin-flip 50%, still short of the 75% it reaches simply executing. It runs rules; it does not discover them.

     Benchmark Generalizationsmall-model
110. 2026-07-071 figure

     Qwen3.5-4B: Is In-Context Learning Retrieval or Induction?

     Follows a new rule but cannot discover one

     Only patterns it already has. Told a scrambled counting order outright, the model applied it almost perfectly — 97 percent right. But shown examples of that same order and asked to work it out, it scored 12 percent, no better than the 10 percent you would get by guessing. Examples surface familiar structure; they do not discover genuinely new structure.

     Algorithmic Memory and RetrievalBenchmark Generalizationmemorysmall-model
111. 2026-07-071 figure

     Qwen3.5-4B: Can the Model Localize Its Own Errors in Multi-Step Reasoning?

     Its confidence dips exactly where it first slipped

     Yes, and the dip lands on the exact step, not just late in the chain. Confidence naturally climbs the deeper the model goes, so "least sure" could just mean "last step." Correcting for that, the least-confident step is the true first slip 56% of the time, versus 36% for guessing the usual trouble spot and 19% for blind chance. When it slips several times, though, only 27%.

     Benchmark Generalizationsmall-model
112. 2026-07-071 figure

     Qwen3.5-4B: Beating Sample-More with the Model's Own Uncertainty

     Trust the surest answer, not the most common

     No. Picking the most common answer wastes the extra tries: accuracy stays flat near 48% however many you draw, because the model keeps confidently repeating the same wrong rule. Picking the answer it was surest of instead climbs from 47% to 62%, beating the popular vote at every budget and approaching the 83% a perfect picker would reach.

     Benchmark Generalizationsmall-model
113. 2026-07-071 figure

     Qwen3.5-4B: Does the Confidence Toolkit Survive on Real Code?

     A single self-check beats averaging whole-program confidence

     Yes, but not the obvious way. Averaging the model's certainty across every token of a program barely beats a plain majority vote among the tries. The real winner: make the model write the code, then answer one yes/no question about it and read that single verdict token. It picks the correct program 76% of the time versus 70% by chance, no tests run — though one real test still beats it, at 82%.

     Benchmark GeneralizationEvidence-Conditioned SelectionInterpretability and Diagnosticsexecutionsmall-modeltools
114. 2026-07-061 figure

     Qwen3.5-4B: When Does the Model's Structure Beat Brute Search? (depth-4)

     Brute-force search wins wider as tasks deepen

     No. Adding one extra dial — a sixteen-times-larger space of combinations — did not flip things. Exhaustive search stayed near-perfect at about 97 percent, while the model's knack for guessing the right combination from memory collapsed from 51 percent to 10 percent. The gap did not shrink toward a crossover; it widened, from 46 to 87 points. Structure fades faster than search slows.

     Posttraining and AdaptationStructured Execution and Compilerslorasmall-model
115. 2026-07-061 figure

     Qwen3.5-4B: Is the Wall Structure or Values? (skeleton-then-fill)

     It picks the wrong steps, not wrong numbers

     The steps. Handed the correct sequence of operations, a cheap number-search finished every single task, so the numbers were never the bottleneck. Left alone, the model almost never even lands the right sequence, and crediting any-numbers-right adds nothing: its right-sequence rate exactly equals its solve rate. There is no hidden pool of right-recipe, wrong-number near-misses waiting to be unlocked.

     Structured Execution and Compilerssmall-model
116. 2026-07-061 figure

     Qwen3.5-4B: Externalize the Latent Readout (probe-to-prompt)

     Hint the exact step, not just its type

     Partly. Writing the concrete first step — exact value and all — into the prompt lifted the single-best-guess solve rate on two-step tasks sixfold, from 3% to 19%, where editing the model's internal state did nothing. But its internals reliably reveal only the step's category, and category alone merely helps it stumble onto answers across several tries — not solve first-shot. The exact value is the real unlock.

     Interpretability and DiagnosticsStructured Execution and Compilerslatent-statesmall-model
117. 2026-07-061 figure

     Qwen3.5-4B: Is the Parameter Latent? (probe the full first op)

     The model computes the operation, not the number

     It splits. The model genuinely computes the KIND of operation inside itself — reading its internal activity names the kind far better than the examples alone do (41% versus 27%, against 6% for blind guessing). But the specific number is not stored anywhere special: a trivial calculator run over the examples recovers it just as well (49% versus 53%). The model reads it off, same as anyone could.

     Interpretability and DiagnosticsStructured Execution and Compilersabilatent-statesmall-model
118. 2026-07-061 figure

     Qwen3.5-4B: Learn from Your Own Failures (DPO)

     Studying its own mistakes backfires

     No. The model already ranked its own correct answer above its wrong one 81% of the time — a sharp internal judge. But training it to favor the correct ones destroyed its writing: single-best-guess solving peaked near 5% then cratered toward zero. The dull alternative won — simply training twice as long on the correct examples alone tripled first-try solving to about 11%.

     Benchmark GeneralizationPosttraining and Adaptationcurriculumlorasmall-model
119. 2026-07-061 figure

     Qwen3.5-4B: Do the Structure Findings Generalize? (string/register)

     Small models get the plan wrong

     The wrong sequence. Across three completely different kinds of programming tasks, a 4-billion-parameter model almost never solved one on its own — 0 to 2 percent — and its misses were wrong-order, not right-order-wrong-number. Hand it the correct sequence of actions and it fills the numbers perfectly every time (100 percent). A blind tool that tries candidate sequences and checks each solved nearly everything, 98 to 100 percent — beating the model outright.

     Structured Execution and Compilerslatent-statesmall-model
120. 2026-07-062 figures

     Qwen3.5-4B: Does Banking Install STRUCTURE?

     Training teaches the plan; brute-force wins anyway

     Yes, then no. Training lifted a 4-billion-parameter model from never proposing the right step-sequence (0%) to getting it right about half the time (51%) on brand-new tasks—a real new skill, not memorized answers. But if you can run code, blindly trying all 4,096 possible step-sequences and keeping what works solves 97.5%, crushing the trained model's 46%. The model becomes unnecessary.

     Posttraining and AdaptationStructured Execution and Compilerslorasmall-model
121. 2026-07-051 figure

     Qwen3.5-4B: Thinking vs the Lookahead Wall

     Thinking sharpens recognition but cannot plan ahead

     No. Asked to name the first of three moves toward a goal, the model stayed stuck at pure guessing — right about 1 time in 32 — no matter how long it thought, even with 2,048 tokens to think first. Yet recognizing a goal just one move away leapt from roughly 28% to 60% as thinking grew. So extra thinking amplifies spotting a near answer, not planning a distant one.

     Structured Execution and Compilerssmall-model
122. 2026-07-051 figure

     Qwen3.5-4B Latent Decomposition: be your own tool-search

     Self-training installed the planning it couldn't do alone

     No, not out of the box. One step from the goal it picks the right operation about eight times better than chance, but on the opening move three steps out it ranks correctly only at chance — recognition, not planning. As its own guide it solved 1 of 80 puzzles, no better than random's 2. Self-training on its verified solutions installed real planning: 18 of 80, nearly matching trying every operation.

     Process Control and Tool UseStructured Execution and Compilerslatent-statesmall-modeltools
123. 2026-07-051 figure

     Qwen3.5-4B Depth Scaling & Controls: saturation, data-vs-compute, depth-4

     Variety keeps teaching, repetition barely helps

     Variety, decisively, and the gains never plateaued. Rereading the same 40 solved examples sixteen times over barely moved the solve rate (9% to 16%, within noise), but replacing them with more distinct examples at the exact same training cost jumped it (16% to 38%). And coverage kept climbing to 54% as distinct examples grew past a thousand, with no ceiling in sight.

     Structured Execution and Compilerssmall-model
124. 2026-07-051 figure

     Qwen3.5-4B: Do Banking and Thinking Stack?

     Extra thinking sharpens easy moves, not long-range planning

     It depends on how far the goal is. One move away, the two boosts stack almost perfectly: a plain model picks the right move 27.5% of the time, extra training lifts that to 52.5%, and adding thinking reaches 85% — the exact sum, no wasted overlap. But three moves out, extra thinking adds nothing (about 17% right, with or without it); only training moved planning. Blind guessing scores about 3%.

     Posttraining and AdaptationStructured Execution and Compilerslorasmall-model
125. 2026-07-052 figures

     Qwen3.5-4B: Bank the Thoughts

     Its own reasoning teaches nothing; real plans do

     No. Training the model on its own successful reasoning taught it nothing beyond showing it the bare answers — both solved about 9 in 100 fresh three-step problems. But a short hand-written plan that builds the solution forward, one step at a time, lifted that to about 22 in 100, roughly 2.6 times better. The model's own "reasoning" was after-the-fact excuse-making, not a real method.

     Benchmark GeneralizationPosttraining and Adaptationcurriculumlorasmall-model
126. 2026-07-041 figure

     Qwen3.5-4B Tool-Seeded Banking: does tool-search + banking cross the depth-3 wall?

     Search finds the harder skill

     Partly. A brute-force search found working three-step programs the model never produces itself, and retraining on them lifted three-step success from a hard zero to 5 of 40 fresh, never-seen tasks when it can reason across sixteen tries — a real, significant crossing. But in one shot, the production setting, it still solved none, versus 15% on two-step tasks. The knowledge crossed the wall, yet barely stuck in the weights.

     Algorithmic Memory and RetrievalEvidence-Conditioned SelectionProcess Control and Tool Usecoveragememorysmall-model
127. 2026-07-041 figure

     Qwen3.5-4B Depth-3 Dose-Response: data-limited or representational cap?

     The reasoning wall was data, not a ceiling

     Just starved for examples. Feeding a fixed 4-billion-parameter model more search-found solutions lifted its solve rate on fresh three-layer puzzles from 0% to 38% when given sixteen tries — climbing steadily at every dose, with no sign of leveling off. It even began solving 1 in 10 fresh puzzles on a single blind attempt with no reasoning shown, versus essentially zero before.

     Algorithmic Memory and RetrievalEvidence-Conditioned SelectionStructured Execution and Compilerscoveragelatent-statememory
128. 2026-07-031 figure

     Qwen3.5-4B Wall Climbing: does banking shallow composition unlock deeper coverage?

     Drilling two-step problems never unlocks three-step ones

     No. Fine-tuning the model on the two-step solutions it could already produce tripled its two-step success on fresh tasks, from 12% to 36%. But its three-step success stayed at exactly zero, unchanged from before: both models solved none of the three-step tasks within sixteen tries. Becoming fluent at one difficulty tier taught it nothing about the tier just above.

     Algorithmic Memory and RetrievalEvidence-Conditioned SelectionStructured Execution and Compilerscoveragememorysmall-model
129. 2026-07-031 figure

     Qwen3.5-4B Latent Composition Probe: is the wall representational or expressive?

     Shallow answers hide inside; deep answers never form

     It depends on the number of steps. For a one-step recipe the first step is almost perfectly written on the internal scratch paper (99% readable), yet the model voices it only 44% of the time: it knows but stays silent. For the hardest three-step recipe the scratch paper is nearly blank: the readable signal (about 13%) barely clears the 14% noise floor, meaning the answer was never worked out.

     Structured Execution and Compilerslatent-statesmall-model
130. 2026-07-031 figure

     Qwen3.5-4B Coverage vs Selection: anatomy of the generation wall

     Small models fail by never guessing, not misjudging

     It never generates it. Whenever a correct program shows up among 32 tries, running each candidate against eight known examples finds it every single time — the model judging its own work, and even a random pick among survivors, do exactly as well. Nine in ten example-passers are truly correct. Smarter selection buys nothing; only whether the right program ever appears matters.

     Algorithmic Memory and RetrievalEvidence-Conditioned SelectionStructured Execution and Compilerscoveragememorysmall-model
131. 2026-07-031 figure

     Qwen3.5-4B Coverage Banking: does banking shift the proposal distribution?

     A model self-taught past its own ceiling

     Yes, but it depends on difficulty. On easy one-step problems it just pulls answers it already knew into its top guess (60% to 80%), gaining no new ground. On harder two-step problems something new happens: on brand-new tasks it proposes correct programs it had never found, tripling its solve rate from 15% to 45%. On the hardest problems, with no wins to learn from, nothing moves.

     Algorithmic Memory and RetrievalEvidence-Conditioned SelectionInterpretability and Diagnosticscoveragememorysmall-model
132. 2026-07-031 figure

     Qwen3.5-4B Activation Steering: is the latent first-op causally usable?

     Reading the model's mind can't steer it

     No. A simple reader picks the model's planned first operation out of its internal state almost perfectly — 99% of the time — yet pushing that exact signal back in during generation barely moves what the model does: at best 3 to 5 points of improvement, no better than pushing a meaningless random direction, and far under the 10-point bar set in advance. Even handing it the known-correct answer failed.

     Structured Execution and Compilerslatent-statesmall-model
133. 2026-07-021 figure

     Qwen3.5-4B Simulation Keystone Repair

     A perfectly repaired skill reached no dependent task

     No. Training made the model trace a multi-step process almost flawlessly — even on longer chains and steps it never studied, so it learned a genuine skill, not memorized answers. Yet every task that supposedly needs tracing stayed frozen: naming the hidden steps from examples crept from 8% to 9%. Oddly, unrelated training that merely matched a task's answer format helped three times more.

     Structured Execution and Compilersabilatent-staterepair
134. 2026-07-022 figures

     Qwen3.5-4B Depth-Wall Anatomy

     It runs any recipe but can't reverse-engineer one

     It's figuring out the steps. Handed the exact sequence of operations, this 4-billion-parameter model writes correct code almost every time, even four steps deep, with zero execution deficit. Left to infer that sequence from before-and-after examples, it solves almost nothing past two steps, spotting each added operation only about twice as well as guessing among 63 candidates.

     Structured Execution and Compilerslatent-statesmall-model
135. 2026-07-021 figure

     Qwen3.5-4B Cross-Family Laws

     It runs any plan but cannot invent one

     No. Handed the exact steps, this fixed 4-billion-parameter model wrote correct code almost every time across three unrelated task types. But asked to infer the same procedure from example inputs and outputs alone, it fell from about 80% at one step toward zero by three stacked steps. Running a plan and discovering one are separate abilities.

     Benchmark GeneralizationStructured Execution and Compilerscurriculumsmall-model
136. 2026-07-021 figure

     Qwen3.5-4B Context Composition

     Fine-tuning installed the skill but hijacked its answers

     Only when its answer survives. The fine-tuned skill is genuinely the sharpest — 95% correct when the model replies in the required form, beating the untrained model's 83% under the same step-by-step procedure. But the training hijacks the answer format half the time, so real usable accuracy collapses to 51% — below the plain untrained model's 74%.

     Process Control and Tool UseStructured Execution and Compilersabismall-modeltools
137. 2026-07-011 figure

     Qwen3.5-4B Decompose-and-Compose Frontier

     Steps plus a tool crack the wall

     Yes — but not because the model got smarter. Taking three steps one at a time, with a tool that runs each and shows the result, solves about 2 in 5 versus 1 in 8 in one shot. The catch: blindly trying all 23 operations does just as well. The step-by-step structure and the tool crack the wall — not the model's judgment about which move is best.

     Posttraining and AdaptationStructured Execution and Compilersabidistillationlatent-state
138. 2026-06-30 → 07-014 figures

     Qwen3.5-4B Neurosymbolic REPL Substrate + Failure Profile

     Fixing its answers failed; retraining on them worked

     No. Seeing its real error barely helped: the fix-it loop solved 29% of puzzles versus 34% for simply drawing five independent attempts and keeping the best, at equal compute; the error message itself added just two tasks over a control told only to try again. The model's test-time ceiling is whatever its own sampling can reach. Retraining on its verified solutions, though, lifted single-shot accuracy from 22% to 32%.

     Posttraining and AdaptationStructured Execution and Compilersdistillationexecutionsmall-model
139. 2026-06-301 figure

     Qwen3.5-4B Verifier vs Visible Selector Showdown

     A cheap test makes slow self-checking pointless

     No. When you can run even a single example test on each candidate, that filter alone lifts the share of shipped programs that fully work from 77% to 85%. Adding a free, instant self-confidence rating reaches 87% — matching a careful step-by-step self-check that costs five times the compute. Slow deliberation only earns its keep when no test can be run.

     Evidence-Conditioned SelectionInterpretability and DiagnosticsReliability and Safetysmall-modelverification
140. 2026-06-301 figure

     Qwen3.5-4B Thinking Content vs Compute

     Only real reasoning helps, wrong reasoning wrecks answers

     It genuinely reasons; the boost is content, not compute. Blank filler of the same length, and the real thinking scrambled into nonsense, both scored like skipping thinking entirely, around 74 to 75 percent. Only coherent thinking lifted accuracy to 86 percent. And splicing in another task's reasoning crashed it to 4 percent, because the model solves whatever problem its notes describe.

     Structured Execution and CompilersTest-Time Reasoning Budgetabismall-model
141. 2026-06-306 figures

     Qwen3.5-4B Thinking-Budget Scaling

     Capped thinking beats both silence and endless rambling

     Yes, but with a sweet spot. Turning on the model's built-in reasoning lifted the accuracy of the answer you would actually ship from 76% to 91% on basic Python tasks — a genuine capability gain, not just a wider pool of guesses to pick from. But more is not better: accuracy peaks around a few hundred to a thousand reasoning tokens, then falls. Unlimited thinking scored only 84%, worse than a firm cap.

     Algorithmic Memory and RetrievalEvidence-Conditioned SelectionInterpretability and Diagnosticscoveragememorypolicy
142. ~2026-06-301 figure

     Qwen3.5-4B Thinking-Budget Controller

     A cheap recheck cuts thinking cost, not accuracy

     For cost, yes; for accuracy, no. Drafting cheaply and thinking harder only when the draft fails the one visible example matches a generous fixed budget's accuracy (88%) using about a fifth of the thinking (113 versus 507 tokens per task). But it never beats the best fixed budget (91%), because that quick check passes a wrong answer about 1 in 10 times.

     Posttraining and AdaptationProcess Control and Tool UseTest-Time Reasoning Budgetpolicysmall-model
143. 2026-06-302 figures

     Qwen3.5-4B Overthinking Content Ladder

     Longer thinking makes reasoning order matter more

     No, the opposite. Giving this small coding model more room to reason made genuine, ordered reasoning matter more, not less. Scrambling its own reasoning steps into random order cost about 10 points of solved tasks at the short budget but 15 points at the longest. Blank filler of the same length bought essentially nothing over not thinking at all, so raw extra compute was not the driver.

     Benchmark GeneralizationTest-Time Reasoning Budgetcurriculumsmall-model
144. 2026-06-29 → 301 figure

     Qwen3.5-4B Thinking Separability Probe

     Scrambled thinking reads correctness as well as real

     Yes, but not for the reason you would expect. From one snapshot of internal activity, whether the model's own code is correct is readable well above a coin-flip, about 64 to 76 percent of the time. Thinking first sharpens that signal. But scrambling the thinking words into gibberish sharpens it just as much, so the boost comes from the act of thinking, not its logic.

     Evidence-Conditioned SelectionInterpretability and DiagnosticsReliability and Safetyabilatent-statesmall-model
145. 2026-06-29 → 301 figure

     Qwen3.5-4B Generator-Verifier Gap

     Small models grade their own code only after

     Only after it thinks. Judging on sight, the model rubber-stamps 91% of its tries as correct while just 77% truly pass — barely a check, mostly agreeing with itself. Given room to reason first, it becomes a real critic: letting it pick its best of eight lifts success from 77% to 86%, closing three-quarters of the gap to a flawless picker's 89% ceiling.

     Evidence-Conditioned SelectionInterpretability and DiagnosticsReliability and Safetyexecutionsmall-modelverification
146. 2026-06-28imported · line Y6 figures

     Qwen Support-Contrastive Meta-ICL

     Models can ace tasks while ignoring your examples

     Not by default. One tuned model scored 49% whether its worked examples were intact or scrambled, meaning it had memorized the task and ignored the examples entirely. A contrastive objective forced genuine reading: 50% with intact examples, collapsing to 7% when scrambled. The catch is this honesty cost a few points versus plain tuning's peak of 56%.

     Benchmark GeneralizationPosttraining and Adaptationlorasmall-modeltable-transform
147. 2026-06-28imported · line Y5 figures

     Qwen Oracle-Distilled Acquisition Policy

     A trained example-picker loses to grabbing variety

     No. The trained picker does read real signal: it beats revealing nothing (50 to 57 percent of tasks fully solved) and crushes a version fed scrambled answers (27 percent). But it lost to simply grabbing a varied spread of examples (67 percent), and even to picking at random (63 percent). Clever ranking underperformed cheap diversity at the same budget.

     Active Evidence AcquisitionBenchmark GeneralizationEvidence-Conditioned Selectionactive-learningdistillationpolicy
148. 2026-06-28imported · line Y6 figures

     Qwen Learned Active Interrogation

     Handpicked examples tie just grabbing them in order

     No. Unlocking four extra answer keys did raise fully-solved tasks from 63% to 70%, and scrambling those answers sank it to 47%, so real labels genuinely matter. But the model's deliberate picks scored exactly the same 70% as blindly taking the first four in order. A hidden best-possible choice reached 80%, so better picks existed; the model simply never found them.

     Active Evidence AcquisitionBenchmark Generalizationactive-learningsmall-modeltable-transform
149. 2026-06-28imported · line Y9 figures

     Counterfactual ICL Public Multiseed Gate

     Cheap example drills triple accuracy; twist adds nothing

     Yes, but not for the reason expected. Tuning tripled whole-task accuracy on real text tasks, from 20% to about 61%, stable across three training runs — and the model genuinely reads its examples: remove them and it collapses to under 2%. But plain tuning on ordinary made-up puzzles reached 64%, so the clever counterintuitive twist added nothing. The lift is generic synthetic tuning.

     Benchmark GeneralizationPosttraining and Adaptationlorasmall-modeltable-transform
150. 2026-06-28imported · line Y4 figures

     Counterfactual Episodic ICL Posttraining

     Contradictory examples teach a model to read its

     Yes. Untrained, a 4-billion-parameter model solved 23% of real text-transformation tasks perfectly; after this training, 57% — but only when it could see the prompt's examples. Scramble those examples and it fell to 17%, below where it started; remove them and it collapsed to about 3%. The entire gain lives in reading the examples, not memorizing a familiar transformation.

     Benchmark GeneralizationPosttraining and AdaptationStructured Execution and Compilersabilorasmall-model
151. 2026-06-28imported · line Y5 figures

     Counterexample-Guided Ephemeral Program

     Clever test-picking tied random guessing; direct answers won

     No. Answering each row directly solved three-quarters of tasks completely, while the best rule-program the method could pick solved only about four in ten — and a perfect picker that peeks at the answers did no better. The clever edge-case grading tied random cases and even meaningless shuffled labels. The real problem: usually no correct program existed to pick.

     Benchmark GeneralizationEvidence-Conditioned SelectionReliability and Safetycounterexamplesexecutionsmall-model
152. 2026-06-28imported · line Z3 figures

     Qwen3.5-4B Tool State Policy LoRA

     Two cheap checks beat a fine-tuned model

     Surprisingly, yes, and with almost no learning. Trusting the program only when it passes a worked example and disagrees with the quick answer lifted accuracy from 56% to 66%, exactly matching the best any picker could reach, and never overwriting a correct answer. Fine-tuning learned most of the signal but landed lower, at 64%. Letting the model commit its own program every time collapsed to 24%, because programs that pass the example are often still secretly wrong.

     Posttraining and AdaptationProcess Control and Tool UseStructured Execution and Compilerslatent-statelorapolicy
153. 2026-06-28imported · line Z3 figures

     Qwen3.5-4B Live Tool DAgger

     Choosing when to run code cuts cost

     Only its cost, not its accuracy. Answering directly solved none of twelve unseen tasks; letting the model write and run code recovered two — about one in six — and spoiled nothing it already had right. But even a flawless, all-knowing chooser topped out at those same two: the ceiling is the model's raw skill, not the decision. The trained controller's real win was reaching that ceiling with half the code runs and zero wasted repairs.

     Posttraining and AdaptationProcess Control and Tool UseStructured Execution and Compilersdistillationlatent-statesmall-model
154. 2026-06-28imported · line Z3 figures

     Qwen3.5-4B Foofah Strategy Discovery Live

     Self-invented recipes beat prompting but unlock nothing new

     Barely, and nothing genuinely new. The self-written recipes lift exactly-solved tables from 42% to 46%, edging out plain prompting. But a ready-made library of solved programs already handled 56 to 58%, and every table the recipes cracked, that library already covered. Net new tables unlocked: exactly zero. Discovery just re-derived a slice of what the model could already reach.

     Benchmark Generalizationfoofahsmall-modeltable-transform
155. 2026-06-28imported · line Z4 figures

     Qwen3.5-4B Foofah Program Strategy Portfolio

     Trust the first passing program, not a vote

     Yes, but modestly, and which passing program you trust matters more than the programs. Asking directly for the finished table got 42% right. The rule the team locked in, commit only when two programs agree, reached just 48%. A simpler rule they also wrote down beforehand, trust the first program that passes the check, hit 56%, nearly the 58% best-possible ceiling. Consensus quietly discarded correct lone hits.

     Benchmark GeneralizationStructured Execution and Compilersexecutionfoofahsmall-model
156. 2026-06-28imported · line Z4 figures

     Qwen3.5-4B Foofah Adaptive Program Budget Router

     A cheap shape check beats running every program

     Yes, and here is the twist: running all five programs on every task scored lower (56%) than the cheap rule (58%), because blanket spending overwrote one answer the quick pass already had right. The rule fires only when the quick answer has fewer columns than the input, flagging exactly the hard cases, beating the quick answer's 42% and tying a perfect-hindsight ceiling at half the compute.

     Benchmark GeneralizationStructured Execution and Compilersexecutionfoofahsmall-model
157. 2026-06-28imported · line Z4 figures

     Qwen3.5-4B Adaptive Tool Controller

     Knowing when tools help beats running them all

     Partly. One structural cue — does the direct answer have fewer columns than the raw data implies? — safely flags the reshaping tasks where a program helps, lifting accuracy from 42% to 50% with zero broken tasks. But it captures only half the gain; the full 58% ceiling needs a slightly broader set of tool calls, not one cheap program.

     Benchmark GeneralizationProcess Control and Tool UseStructured Execution and Compilersexecutionsmall-modeltable-transform
158. imported 2026-07-12imported · line Y7 figures

     Sampled Query Filter Executor Experiment

     One sampled answer teaches the entire hidden world

     Yes. Graded on just one sampled final answer per problem, the model rebuilt the entire set of still-possible number pairs it was never shown, capturing 94 to 98 percent of it, because holding that full set is the cheapest way to answer varied questions. Simpler models that tracked each number separately, or skipped step-by-step execution, stayed under 6 percent.

     Active Evidence AcquisitionStructured Execution and Compilersactive-learningexecutionlatent-state
159. imported 2026-07-12imported · line Y4 figures

     Qwen Slot Repair Distillation

     Fixable programs exist; the editor can't find them

     No. A correct program almost always sits one or two edits away — a search that peeks at the answer lifts solve rates from about a quarter to roughly 86%. But the blind helper couldn't pick which edits to make: on reworded problems it dropped accuracy from 25% to 19%, and on fresh problems it barely moved. It only helped on the exact problems it was tuned on.

     Benchmark GeneralizationPosttraining and AdaptationStructured Execution and Compilersabidistillationexecution
160. imported 2026-07-12imported · line Y3 figures

     Qwen Register Trace Refiner

     Even a perfect picker can't rescue a too

     Rarely. Even a flawless picker that always grabbed the correct edit reached only 37% on plainly worded problems and 7% on reworded ones, because the correct program usually isn't among the roughly 1,300 nearby edits at all. The trained picker did worse, nudging 23% to 27% and leaving reworded problems stuck near 5%.

     Evidence-Conditioned SelectionInterpretability and DiagnosticsReliability and Safetyabiexecutionlatent-state
161. imported 2026-07-12imported · line Y4 figures

     Qwen Register-Token Latent Compiler

     Almost every step right, whole program still wrong

     Only for short chains. Up to twelve steps it builds the correct hidden program about nine times in ten, while stripped-down versions trained on the final answer alone never find the interface and stay at chance. But at twenty-four steps whole-program accuracy crashes to 22 percent — each slot is still about 90 to 97 percent right, yet many near-misses multiply into failure.

     Benchmark GeneralizationStructured Execution and Compilersabibridgeexecution
162. imported 2026-07-12imported · line Y4 figures

     Qwen Register-Token Structured Runtime

     Nearly-right steps compound into wrong long answers

     Only up to a point. For chains of four to twelve steps the hidden program runs flawlessly, at 100 percent. But at 24 steps exact execution collapses to 25 percent — versus about 1 percent from pure guessing. The catch: each individual step is 80 to 94 percent right, yet a single wrong step anywhere breaks the whole answer, so mostly-right steps still add up to a mostly-wrong long program.

     Benchmark GeneralizationStructured Execution and Compilersbridgeexecutionlatent-state
163. imported 2026-07-12imported · line Y5 figures

     Qwen Python-Shaped Silent Executor

     Private thinking slots did no real thinking

     No. The silent hidden steps never cleared single digits: about 8% at best on the simplest programs, 4.5% on unseen longer ones, barely above a zero untrained model and near the 3% you would get by guessing. The tell: scrambling the steps' order worked just as well, so no real step-by-step computing happened. Even visible written reasoning, which hit 62.5% on the shortest programs, crashed to 0% as they grew longer.

     Structured Execution and Compilersexecutionlatent-statesmall-model
164. imported 2026-07-12imported · line Y3 figures

     Qwen Progressive Repair Compiler

     The right repair exists but goes unpicked

     Partly. The correct fix sits in the candidate list almost nine times in ten, yet the judge finds it only about half the time, lifting exactly-correct programs from 30% to 49% against the 88% a flawless chooser would reach. Generating candidate fixes is easy; reliably recognizing the right one is the real wall.

     Benchmark GeneralizationEvidence-Conditioned SelectionInterpretability and Diagnosticsabiexecutionlatent-state
165. imported 2026-07-12imported · line Y1 figure

     Qwen Learned Repair Verifier

     The fix exists; picking it blind is hard

     Partly. Without ever seeing the true answer, the judge lifted correct execution from 30% to 47%. But a checker allowed to peek at the answer key found a correct fix already sitting in the candidate pile 88% of the time, so blind picking claimed only about a third of the fixable errors. Comparing two wordings of the same task and rewarding agreement did better, reaching 51%.

     Evidence-Conditioned SelectionInterpretability and DiagnosticsReliability and Safetyabiexecutionrepair
166. imported 2026-07-12imported · line Y5 figures

     Qwen Latent Beam Program Compiler

     Small model writes exact hidden programs

     Yes, up to a point. It wrote exact programs that a fixed calculator ran perfectly at eight and twelve steps, and its answers matched those programs — so it truly computed rather than guessing, where chance is about one in a hundred. But at twenty-four steps it ran roughly 85% of the chain, then erred late. Producing four candidate programs and keeping the best stayed at chance.

     Structured Execution and Compilersabiexecutionlatent-state
167. imported 2026-07-12imported · line Y1 figure

     Qwen Candidate-Trace Verifier

     Finding the fix is easy; picking it isn't

     Yes. A small checker that reads each candidate's worked-out steps — never the true answer — lifted correctly-running programs from 30% to 54%, and to 56% when it cross-checks two wordings of one task. The twist: a picker handed the answer key hits 88%, so the right fix is usually in the pile; the checker just fails to spot it. Selection, not search, is the wall.

     Benchmark GeneralizationEvidence-Conditioned SelectionInterpretability and Diagnosticsexecutionrepairsmall-model
168. imported 2026-07-12imported · line Z3 figures

     Qwen 3.5 4B Verified Edit Closure

     Trying tiny fixes beats retraining the model

     Yes, on the hardest unseen tasks. Testing small edits of the model's own near-miss program and keeping whichever passes the example cases raised fully-correct answers from 39% to 52%, while re-sampling the model and re-ranking its guesses gave nothing over its first try. But it stalled near 58%: some wrong programs need a rewrite, not a tweak.

     Benchmark GeneralizationEvidence-Conditioned SelectionInterpretability and Diagnosticsexecutionrepairsmall-model
169. imported 2026-07-12imported · line Z3 figures

     Qwen 3.5 4B Typed Sketch Synthesis

     Let the model sketch, let the solver fill

     It depends on difficulty. On the hardest problems, sketching the shape and letting a verified search fill the blanks lifted correct fixes from 33% to 78%, and a safe blend of both methods reached 88%. But on easy problems, forcing sketches backfired — 100% dropped to 75% — because too many blank-fillings look equally right. So trust the model's direct fix whenever it already passes every visible test.

     Benchmark GeneralizationStructured Execution and Compilersexecutionrepairsmall-model
170. imported 2026-07-12imported · line Z3 figures

     Qwen 3.5 4B Static Bridge Ceiling Breaker

     A few harder examples double transfer, then stall

     Partly. Folding in just 60 slightly-harder "bridge" examples, a quarter of the training budget, more than doubled success on deeper, never-seen programs, from 20% to 44% fully repaired, with no loss on familiar skills. But it stalled fast: 80 examples did no better. And the skill is genuine reading, not memorization: feed a scrambled test printout and success sinks below giving no printout at all.

     Benchmark GeneralizationEvidence-Conditioned SelectionStructured Execution and Compilersbridgecoverageexecution
171. imported 2026-07-12imported · line Z2 figures

     Qwen 3.5 4B GraphIR Self Repair

     A fix-it pass catches noise, not real mistakes

     No. The plain one-line formula fully solved 29% of brand-new tasks; the step-by-step diagram managed only 22%, and the fix-it pass recovered part of that gap to 24% — still behind. That pass genuinely works: on randomly scrambled diagrams it lifted success from 3% to 27%, an eightfold jump. But the model's own errors are confident, consistent wrong logic, not obvious damage, so repair rarely helps.

     Benchmark GeneralizationEvidence-Conditioned SelectionInterpretability and Diagnosticsexecutionlatent-staterepair
172. imported 2026-07-12imported · line Y7 figures

     Query Filter Executor Experiment

     Graded only on answers

     Yes, and that is the surprise. Trained only to name one final answer, the model taught itself to run each instruction in order and track every still-possible pair of the two hidden numbers. Given enough internal steps, it stayed about 96% accurate on programs three times longer than any it trained on, while simpler-memory and one-pass rivals collapsed to roughly 5 to 12%.

     Active Evidence AcquisitionStructured Execution and Compilersactive-learningexecutionlatent-state
173. imported 2026-07-12imported · line Y3 figures

     Latent Recurrent Executor Experiment

     Enough hidden steps turn guesswork into perfect answers

     Yes, but only under strict conditions. Built to hold each running total and trained to hit every intermediate value, the network's exact-answer rate stayed under one percent until its private-step count reached the number of instructions, then snapped to a perfect hundred percent, even on programs three times longer than any it trained on. A plain one-shot version and a generic looping version never beat near-random.

     Structured Execution and Compilersexecutionlatent-state
174. imported 2026-07-12imported · line Y4 figures

     Joint Register Executor Experiment

     Extra thinking pays off only with linked memory

     Yes, but only with the right memory. When the model held every allowed number-pair together and spent one thinking step per instruction, its confidence in the exactly correct answer set jumped from near-random (about 3%) to between 87 and 98% the moment its steps reached program length, even on programs three times longer than anything in training. A twin that tracked each number separately never beat chance.

     Structured Execution and Compilersexecutionlatent-state
175. imported 2026-07-12imported · line Y11 figures

     Dense Supervision Ladder Experiment

     Weak feedback, not the model, is the ceiling

     It's the feedback. With the model held fixed, training it on only one sampled final answer left it weak; showing it the full odds of every possible answer at every step roughly doubled how often it solved the hardest 24-step programs, from 9% to 19%, and left its internal tracking more readable than any other recipe. Even the richest feedback, though, left it far from accurate.

     Benchmark GeneralizationStructured Execution and Compilerscurriculumexecutionlatent-state
176. imported 2026-07-12imported · line Y7 figures

     Dense Latent Query Executor Experiment

     Stepping through beats one glance

     Partly. Giving the model enough internal thinking steps to walk through the whole program lifts accuracy sharply, and it clearly beats a model that reads everything in one glance. But the memory stays approximate: at its longest training length it puts only about half its confidence on correct answers, and on the longest programs that slips to about a fifth, with the underlying hidden numbers barely recoverable afterward.

     Active Evidence AcquisitionStructured Execution and Compilersactive-learningexecutionlatent-state
177. imported 2026-07-12imported · line Y4 figures

     Belief Filter Executor Experiment

     One internal step per instruction unlocks exact answers

     Yes. When the model runs one internal update per instruction, it lands over 91% of its confidence on the exact set of still-possible answers, versus under 14% when it stops before finishing. It even runs programs three times longer than any it trained on, but only when it tracks the two hidden numbers as a linked pair, not separately, which scored under 4%.

     Structured Execution and Compilersexecutionlatent-state
178. imported 2026-07-12imported · line Y4 figures

     Adaptive Cognitive Kernel

     Self-rewiring does real work but wins nothing extra

     No advantage. The self-rewiring is genuinely doing ordered work: scrambling the operation order collapses its step-by-step accuracy from about 12% to 2%, and switching the rewiring off cripples it. But it never beats a plain fixed network. On the longest test programs the ordinary one matched or slightly edged it, and both crashed far below their training-length scores.

     Posttraining and AdaptationStructured Execution and Compilersexecutionlora
179. 2026-06-27imported · line Z4 figures

     Real Transform ABI Gate with Counterexamples

     Ready-made toolbox nails clean data, fumbles messy edges

     It depends on how messy the data is. For clean, spreadsheet-style pipeline jobs the toolbox covered every one (100%) and held firm even against deliberately tricky examples. For irregular date, ID, and text cleanup, coverage slipped from 95% to 85% once trick examples exposed functions that only looked right. And a wrong function fit the few shown examples about a third of the time.

     Algorithmic Memory and RetrievalBenchmark GeneralizationEvidence-Conditioned Selectionabicoveragecounterexamples
180. 2026-06-27imported · line Y6 figures

     Qwen Verified Skill Memory RAG

     Handing it a solved example did not help

     No. Handing the model a matched, verified example made it slightly worse, not better: every value came out correct on 47.5% of jobs, versus 50% when it worked through each value alone with no example. The lookup itself worked — the fetched example was the same kind of task 77.5% of the time — yet across 40 jobs it improved none and hurt one. Relevant context was not the missing ingredient.

     Algorithmic Memory and RetrievalBenchmark GeneralizationEvidence-Conditioned Selectionmemorysmall-modeltable-transform
181. 2026-06-27imported · line Y5 figures

     Qwen Recursive Task Decomposition

     One reusable recipe loses to solving cases fresh

     No. Forcing a 4-billion-parameter model to commit to one written recipe lowered accuracy: it got 59% of individual cases right versus 69% when answering each fresh, and 43% of whole tasks perfectly versus 47%. The recipes weren't nonsense — one borrowed from an unrelated task scored even lower — but a single fixed recipe still can't beat the model's case-by-case judgment.

     Benchmark Generalizationsmall-modeltable-transform
182. 2026-06-27imported · line Y5 figures

     Qwen Recursive Ephemeral Program Induction

     Trust the code only when it earns it

     Only when you check the rule first. On its own, a model writing and applying a reusable rule solved 40% of tasks perfectly versus 56% for plain row-by-row answering, and it broke six tasks direct answering had solved. Adding a gate that uses the rule only when it passes the shown examples and isn't a memorized lookup lifted success to 64%, fixing two tasks and breaking none.

     Benchmark GeneralizationStructured Execution and Compilersexecutionsmall-modeltable-transform
183. 2026-06-27imported · line Y5 figures

     Qwen Real Task ABI Coverage Gate

     A fixed toolkit solves the routine

     It depends, and the split is sharp. A frozen kit of reusable office operations, with no training at all, assembled 84% of realistic tasks from stored parts alone, far above the 21% a bare kit managed, and it fully solved even brand-new task types it had never seen built. But every task deliberately designed to need a missing part failed: zero solved.

     Algorithmic Memory and RetrievalEvidence-Conditioned SelectionStructured Execution and Compilersabicoveragememory
184. 2026-06-27imported · line Y6 figures

     Qwen Public PROSE ABI Gate

     Frozen text-cleanup recipes miss most real tasks

     No. The frozen toolkit fully solved only 19% of the 309 outside tasks. In 77% of them no recipe fit even the worked examples, so the toolkit lacked that operation entirely; under 4% overfit. Yet a small four-billion-parameter model, given the same examples, answered a fresh case correctly 73% of the time, and two-thirds right even on tasks the toolkit missed.

     Benchmark GeneralizationStructured Execution and Compilersabismall-modeltable-transform
185. 2026-06-27imported · line Y6 figures

     Pairwise Table Judge

     Table judge picks whichever comes first

     No. Letting the model run a knockout comparison among its own candidate tables did not help: fully-correct tables slipped from 50% (just trusting its first attempt) to 47.5%. On the hard cases where a correct table existed, it picked the right one 100% of the time when that table was shown first and 0% when shown second. It judged position, not content.

     Active Evidence AcquisitionBenchmark GeneralizationEvidence-Conditioned Selectionactive-learningsmall-modeltable-transform
186. 2026-06-27imported · line Y6 figures

     Noisy Row Program Crystallizer

     Distilling noisy guesses into one rule backfires

     No. Keeping the model's direct per-row answers fully solved half of the 40 tasks, while distilling those noisy answers into one fixed rule solved just 22.5% — worse even than a scrambled comparison at 25%. And a flawless rule-picker tops out at 27.5%, because for most tasks no short rule both fits the visible samples and holds on the rest.

     Structured Execution and Compilersexecutionsmall-model
187. 2026-06-27imported · line Y8 figures

     Full-Table Consistency Reranker

     A trained judge never beat the first answer

     No. The trained scorer got every row right on half the tasks, exactly what you get by just keeping the model's first answer, and barely better than scoring tables at random. A flawless table was reachable on five more tasks, but the scorer picked none: it ranked good tables above bad ones well, yet that never became a single extra solve.

     Benchmark GeneralizationInterpretability and Diagnosticssmall-modeltable-transform
188. 2026-06-27imported · line Y5 figures

     Qwen Episodic Soft-Prompt Task Vectors

     Tuned hidden hints do not beat plain prompting

     No. Tuning the primer tokens on each task's own examples solved exactly the same share of tasks perfectly as plain instructions: just over half either way. The tuning genuinely worked, beating a version trained on deliberately scrambled examples by more than 7 points, yet that edge was too small to finish even one extra task, and per-item accuracy inched up by only one point.

     Benchmark Generalizationsmall-modeltable-transform
189. 2026-06-27imported · line Y6 figures

     Qwen Disagreement-Probe Program Induction

     Caution helped; the clever tie-breaker quiz did not

     No. The disagreement quiz picked the same programs whether its judge answers were real, randomly assigned, or skipped entirely — all three landed at 64% of tasks fully solved. The only genuine gain came from a plain caution rule: run a generated program only when it clearly fits, otherwise answer each item by hand. That rule lifted fully-solved tasks from 56% to 64%.

     Benchmark GeneralizationStructured Execution and Compilersexecutionsmall-modeltable-transform
190. 2026-06-27imported · line Y6 figures

     Counterexample-Guided Consistency Projection

     Asking row by row beats forcing one rule

     No. Distilling the model's messy guesses into one reliable formula solved only 20% of tasks completely, versus 50% when it simply transformed each row directly, a 30-point drop. For 29 of the 40 tasks no formula even fit the four examples, and the model's own guesses picked winners no better than randomly shuffled labels.

     Benchmark GeneralizationEvidence-Conditioned SelectionReliability and Safetycounterexamplessmall-modeltable-transform
191. 2026-06-27imported · line Y5 figures

     Qwen Batched Transduction Consistency

     Answering rows together never helped, sometimes hurt

     No. Answering every entry in one combined pass got the whole task right 45 percent of the time, versus 50 percent when each entry was handled alone, a five-point loss. Bundling never rescued a task the solo approach had failed, and it spoiled two tasks that had been fully solved. Asking the model to reason first or format strictly did not close the gap.

     Benchmark Generalizationsmall-modeltable-transform
192. 2026-06-27imported · line Y6 figures

     Qwen Active Example Acquisition

     One well-chosen example helps; random extras backfire

     Barely, and the real lesson is the downside. Letting the model request the single example it was most unsure about lifted fully-solved tasks from 20 to 21 out of 30. But three random extras dropped it to 19, and feeding it examples with scrambled answers sank it to 18. Choosing well matters far more than adding more.

     Active Evidence AcquisitionBenchmark Generalizationactive-learningsmall-modeltable-transform
193. 2026-06-27imported · line Y5 figures

     Qwen Active Crystallizer Public Gate

     Right on each row, not one reliable rule

     No. Using the model's votes to choose a rule worked on 25% of tasks — barely above the 22.5% you get from scrambled, meaningless votes, and it never beat the best rule the candidate pool could offer. The model answered individual unseen rows correctly about 80% of the time, yet almost never turned that skill into one dependable rule.

     Active Evidence AcquisitionBenchmark GeneralizationStructured Execution and Compilersactive-learningexecutionsmall-model
194. 2026-06-27imported · line Z4 figures

     Qwen3.5-4B Transform ABI Compiler Pilot

     Let the model pick recipes, not write code

     Yes. After a light round of tuning, the model chose a recipe that produced the correct output on all 48 test tasks, matching a perfect answer key and beating the untuned model's 92%. It recovered the harder multi-step chains the untuned model flubbed, 19 of 21 up to 21 of 21. Honest caveat: simply matching recipes against the shown examples already scored about 90%, so the menu was fairly easy.

     Benchmark GeneralizationStructured Execution and Compilersabismall-modeltable-transform
195. 2026-06-27imported · line Z5 figures

     Qwen3.5-4B Independent Code ABI Coverage Gate

     Frozen toolbox barely reaches brand-new tasks

     No, hardly any. The locked toolbox solved only about 14% of brand-new tasks, roughly 1 in 7, versus 37% on the familiar tasks it was shaped around. Reshuffling which tasks are unseen barely moves it, around 18%. And nearly every win was a single tool matching by luck, not a real composed solution, far too thin to build a trainable system on top of.

     Algorithmic Memory and RetrievalEvidence-Conditioned SelectionStructured Execution and Compilersabicoveragememory
196. 2026-06-27imported · line Z4 figures

     Qwen3.5-4B Foofah Selective Program Fallback

     Matching the shown examples wins; stress-tests backfire

     Trust the program the moment it reproduces the visible worked examples. Doing that lifted exact-match accuracy from 55% to 62% across 250 table tasks, rescuing 18 answers the direct route got wrong while losing none it got right. The fancier idea of stress-testing the program against fresh made-up inputs did not help and slightly lowered accuracy.

     Benchmark GeneralizationEvidence-Conditioned SelectionStructured Execution and Compilersexecutionfoofahsmall-model
197. 2026-06-27imported · line Z6 figures

     Qwen3.5-4B Foofah Program Repair Agent

     Writing code helps only as a backup plan

     No. Guessing the answer directly won outright, solving 55% of unseen tables versus only 25% for the debugged program. But the program is a useful complement, not a replacement: it rescued 18 tables the direct guess botched, so running both and keeping whichever works lifts success to 62%. And the repair rounds earned their keep, raising program accuracy from 16% to 25% while breaking nothing.

     Benchmark GeneralizationStructured Execution and Compilersexecutionfoofahrepair
198. 2026-06-27imported · line Z5 figures

     Qwen3.5-4B Foofah Program Ensemble Consensus

     Trust the first working program, not a vote

     No. The simplest rule won: run the first program that passes a single worked example. It solved 52% of tables versus 44% when the model just answered directly, rescuing 23 tables it had otherwise botched while breaking only 4. Requiring several programs to agree was more careful but far too timid, recovering fewer mistakes and topping out at 49%.

     Benchmark GeneralizationEvidence-Conditioned SelectionInterpretability and Diagnosticsexecutionfoofahsmall-model
199. 2026-06-27imported · line Z3 figures

     Qwen3.5-4B Foofah External Transform Gate

     A fixed toolbox cannot build most answers

     No. Across 250 real spreadsheet-cleanup tasks, the frozen set of moves could rebuild the correct table for only 18% of them. And the weakness is not picking the right sequence: simply grabbing the first move-sequence that fits already solves 17%, almost touching that 18% ceiling. The moves themselves are missing, and showing more worked examples does not budge it.

     Benchmark Generalizationfoofahsmall-modeltable-transform
200. 2026-06-27imported · line Z5 figures

     Qwen3.5-4B Foofah Ephemeral Program Induction

     Writing self-checked code loses to just asking

     No. Asking directly reshaped 55% of tables correctly; the write-and-test-a-program route managed just 15%. Even a magic chooser that always picked the right route each time would reach only 59% — four points above asking directly. And the self-check leaks: about 1 in 5 programs that passed the one visible example were still wrong on the hidden data.

     Benchmark GeneralizationStructured Execution and Compilersexecutionfoofahsmall-model
201. 2026-06-27imported · line Z3 figures

     Qwen3.5-4B Foofah Direct vs ABI

     Skip the converter, just ask the model

     Just ask the model. Directly generating the reshaped table got 55% of 250 table tasks exactly right, versus only 18% for the fixed-operation converter. The model even nailed 103 reshapes the converter could not even express, while the converter uniquely solved just 10. Direct generation is far from perfect, with 45% still failing, but the converter is a narrow backup, not the main route.

     Benchmark GeneralizationStructured Execution and Compilersabifoofahsmall-model
202. 2026-06-27imported · line Z5 figures

     Qwen3.5-4B Code ABI Oracle Coverage Ladder

     Toolbox builds most tasks but can't pick winners

     Yes, mostly. Snapping together at most three verified functions produced a passing solution for 84% of 160 small programming tasks, up from just 13% with a bare-bones library. But the library is a foundation, not a solver: when you accept the first program that passes one public test, only 59% are actually correct, because most passing candidates still fail hidden tests.

     Algorithmic Memory and RetrievalBenchmark GeneralizationEvidence-Conditioned Selectionabicoveragecurriculum
203. 2026-06-27imported · line Z4 figures

     Qwen3.5-4B Code ABI Compiler Heldout Primitive Pilot

     Reusable code toolbox collapsed on unseen problems

     No. The block kit could build a working solution for 84% of the problems it was tuned against, but only 14% of brand-new ones — a 70-point collapse. Three fresh random batches of new problems all landed near 18%, so it wasn't bad luck. The kit was overfit to its own examples, so model training was skipped entirely.

     Structured Execution and Compilersabismall-model
204. ~2026-06-27imported · line Z

     Qwen 3.5 4B Balanced Discriminative Bridge

     Even practice beats cramming only the hard cases

     An even mix, clearly. Sixty evenly-spread ordinary examples across ten new program types lifted the model's success on unseen hard problems from 60% (with no examples at all) to 99% fully solved. Hand-picking only the trickiest look-alike cases dropped it to 90%, and targeting the model's own past mistakes dropped it further to 83%. Cleverer selection consistently hurt.

     Algorithmic Memory and RetrievalBenchmark GeneralizationEvidence-Conditioned Selectionbridgecoverageexecution
205. 2026-06-26imported · line Y5 figures

     Qwen Trace Procedure Depth Stress

     Writes recipes it cannot cook itself

     Yes. Trained only on single-step tasks, the 4-billion-parameter model wrote six-step procedures that ran correctly 63% of the time once a plain step-follower executed them, versus 0% when the model tried to state the final answer itself. Accuracy slid down gently as tasks lengthened, never collapsing. So it truly composes steps into working procedures, but cannot reliably run them in its own head. Treat it as a recipe-writer, not a calculator.

     Benchmark GeneralizationStructured Execution and Compilersabiexecutionsmall-model
206. 2026-06-26imported · line Y5 figures

     Qwen Program-Only Executable ABI

     Run the code it writes, not its answer

     Yes. Teaching a small model to write a short runnable program lifted brand-new multi-step accuracy from about 44% (just stating an answer) to 73%. And when it wrote out its steps plus an answer, the steps ran correctly 94% of the time while its stated answer was right only 40%. But the gain swung wildly between training runs, so this is a promising lever, not a finished recipe.

     Benchmark GeneralizationStructured Execution and Compilersabiexecutionsmall-model
207. 2026-06-26imported · line Y7 figures

     Qwen Large ABI Nested Compiler

     Bigger toolbox is free; branching must be taught

     Two things. Making the library four times larger, from 32 to 128 operations, did not hurt straight-line programs at all: both stayed perfect on 16-step chains. But branching is a separate skill. Models shown only straight-line programs got the hardest eight-branch tasks almost entirely wrong, about 13% correct, while adding just a handful of small two-to-three-branch examples lifted that to roughly 87%.

     Structured Execution and Compilersabismall-model
208. 2026-06-26imported · line Y6 figures

     Qwen Extrapolation-Bound ABI

     Teach short chains, get long ones free

     Yes. A model trained only on procedures up to three steps long reliably writes correct sixteen-step procedures, with accuracy climbing from 61% under single-step training to a perfect 100%. Surprisingly, adding longer training examples out to eight steps added nothing and slightly hurt, slipping to 89%. Short practice generalizes several times deeper than you would expect.

     Structured Execution and Compilersabismall-model
209. 2026-06-26imported · line Y4 figures

     Qwen Crystallized Trace ABI Tournament

     Showing the work barely helps, costs far more

     Barely. On familiar inputs, writing out each step scored 94% versus 92% for answer-only, a two-point edge that cost three to six times more generated text. On genuinely new combinations of steps the model had never seen, every format collapsed to roughly 30 to 40%, so the extra detail bought almost nothing where it mattered most. Answer-only was actually best on reworded prompts.

     Benchmark GeneralizationStructured Execution and Compilersabiexecutionsmall-model
210. 2026-06-26imported · line Y7 figures

     Qwen Constrained ABI Parser

     Blocking bad steps beats retrying until valid

     Yes, mostly. On the hardest six-step requests, blocking any invalid step as the model writes lifted correctly-running recipes from 60% to 75%, and it won on all five training runs. It also beat merely re-rolling until valid, which reached only about 63%, and writing no steps at all, which got 0%. The catch: the guardrail also makes some wrong recipes look well-formed.

     Benchmark GeneralizationStructured Execution and Compilersabiexecutionlatent-state
211. 2026-06-26imported · line Y6 figures

     Qwen Compositional Curriculum ABI

     A few multi-step examples fix ordering mistakes

     Yes. Adding a few two- and three-step examples lifted correct answers on unseen six-step problems from 72% to 83%, and on eight-step problems from 78% to 89%. But adding only two-step examples did nothing (72% stayed 72%) — you must reach at least one step past your shortest hard case. Every program already followed the format's rules, so this fixed step ordering, not formatting.

     Benchmark GeneralizationStructured Execution and Compilersabicurriculumsmall-model
212. 2026-06-26imported · line Z2 figures

     Qwen3.5-4B Verified Algorithm Retrieval Adaptation

     Reusing solved code rescues a few stuck problems

     Yes, but modestly. Fetching the most similar already-solved problem and adapting its code cracked 3 of the 8 problems that repeated tries kept missing, lifting overall success from 67% to 79%. Matching by meaning clearly beat fetching random code (rescued none) and mislabeled code (rescued one). But only 2 wins truly needed smart matching, and most adaptations that passed the shown tests still failed the hidden ones.

     Algorithmic Memory and RetrievalEvidence-Conditioned SelectionInterpretability and Diagnosticsmemorysmall-modelverification
213. 2026-06-26imported · line Z4 figures

     Qwen3.5-4B Substrate Coverage Ladder

     Hand-written parts solve them; recycled ones solve none

     Only when a building-block was hand-carved for each specific problem. A shared library that reshapes old solutions to fit new ones solved zero of the nine stuck tasks, no better than nothing. Custom-written parts solved all nine, but that only proves the tasks are solvable, not that reuse works. Worse, over half the answers that passed the visible example tests were secretly wrong.

     Algorithmic Memory and RetrievalBenchmark GeneralizationEvidence-Conditioned Selectioncoveragecurriculumexecution
214. 2026-06-26imported · line Z4 figures

     Qwen3.5-4B Retrieval Adapt Verify Scale

     Retrieval finds the fix but can't pick it

     It splits. Pulling a similar solved problem from a 364-entry library and rewriting it recovered 8 of 24 stuck tasks — a third, double the 4 that a random-library control found. But the only safe check, the visible sample tests, passed roughly two-thirds of wrong answers, so the best deployable picker committed 7 right fixes and 7 wrong ones — a coin flip on trust.

     Algorithmic Memory and RetrievalEvidence-Conditioned SelectionInterpretability and Diagnosticscoveragememorysmall-model
215. 2026-06-26imported · line Z4 figures

     Qwen3.5-4B Reliability Exec OPSD Audit

     Leans toward correct code yet still ships bugs

     No. All three cheap fixes failed. Ranking programs by the model's own confidence picked working code less often than just taking the first one that passed the visible sample tests (6 of 24 versus 8), and let more bugs slip through. Showing it one real failing test helped no more than scrambled nonsense — yet leaking the full answer clearly helped, so the test itself could see real signal.

     Interpretability and DiagnosticsReliability and SafetyStructured Execution and Compilersabiexecutionsmall-model
216. 2026-06-26imported · line Z3 figures

     Qwen3.5-4B Prefix Value Guided Search

     A graded head start solves no extra problems

     No. Even when the half-finished drafts were graded with perfect knowledge of the hidden answer tests, finishing only the best-graded one solved the same share of problems as plain full-solution writing — 75% either way. Ungraded random drafts did worse, at 58%. The lone upside: the graded winner used about a quarter of the compute, but only because a flawless grader hand-picked it.

     Structured Execution and Compilerslatent-statesmall-model
217. 2026-06-26imported · line Z3 figures

     Qwen3.5-4B Pass@k Coverage RL

     Cranking up randomness beat retraining for coverage

     No. Retraining the model to spread its tries across different problems actually lowered coverage: it solved about 44% of unseen coding problems within four tries, versus about 69% from simply raising the randomness dial at the same number of tries. It even trailed the plain, untouched model at 50%, and its first-try success sank too. The training collected its reward but never converted it into wider coverage.

     Algorithmic Memory and RetrievalEvidence-Conditioned SelectionPosttraining and Adaptationcoverageloramemory
218. 2026-06-26imported · line Z4 figures

     Qwen3.5-4B OPSD Pressure Locality Audit

     Helpful hints teach copying, not correctness

     No. At the exact spots where correct code diverges from code that passes surface tests but is secretly wrong, the reference hint adds essentially nothing — scoring no better than a scrambled, meaningless hint. The hint only boosts the grader where the code echoes its own wording, or where it leaks the full answer outright. Neither teaches real correctness, so the planned training run was cancelled before it burned any compute.

     Interpretability and DiagnosticsPosttraining and AdaptationProcess Control and Tool Usedistillationpolicysmall-model
219. 2026-06-26imported · line Z5 figures

     Qwen3.5-4B Offline Hard Negative Coverage DPO

     Gentle nudge saves compute, hard shove breaks it

     It depends — and even the gentle version failed its own bar. A light touch matched the task-solving reach of generating twice as many candidate answers at roughly half the compute, and beat a shuffled-pair placebo. But it cut first-try correctness from 50% to 37.5%. Pushed harder, the model collapsed and stopped producing runnable code at all.

     Algorithmic Memory and RetrievalEvidence-Conditioned SelectionPosttraining and Adaptationcoveragecounterexamplesmemory
220. 2026-06-26imported · line Z4 figures

     Qwen3.5-4B Independent Retrieval Consensus

     Independently derived solutions never agreed, so nothing shipped

     No. Requiring independently reshaped solutions to agree on invented tricky inputs, the system committed to zero of 24 unsolved tasks — the separate versions never converged. Simply taking the first solution that passes the public tests got 6 right but 8 wrong, a coin-flip error rate. Worse, forcing source variety cut how many tasks were solvable at all, from 9 down to 7, so it beat neither the simple picker nor a same-topic control.

     Algorithmic Memory and RetrievalEvidence-Conditioned Selectionmemorysmall-model
221. 2026-06-26imported · line Z5 figures

     Qwen3.5-4B Constrained Coverage DPO

     A cheap nudge helps, but more tries win

     Not quite. The nudged model solved 62% of problems within four tries versus 58% for the untrained model, and kept its first-try accuracy — a real, cheap gain. But simply doubling to eight tries solved 67%, still more. The consolation: the nudge reached its score on about half the compute, and cracked one problem the brute-force run missed.

     Algorithmic Memory and RetrievalEvidence-Conditioned SelectionPosttraining and Adaptationcoveragememorysmall-model
222. 2026-06-26imported · line Y4 figures

     Episodic ECHO-TTT

     Scrambled clues help as much as real ones

     No. Letting the model spend a few seconds studying an episode's real observations left its four-choice decisions stuck near blind-guess level, about 25%, no better than doing nothing. The tell: feeding it scrambled or nonsense text helped just as much, and the single best score across the whole experiment, 31%, came from studying nonsense. The study genuinely shifted the model, just not toward understanding the episode.

     Collective Experimentation Infrastructureexperiment
223. 2026-06-25 → 26imported · line Z5 figures

     Qwen3.5-4B Sampler Portfolio Scheduler

     Perfect style-picking wins; no simple rule finds it

     In theory yes, in practice no. A flawless chooser that knew which style would crack each problem solved 75% of coding tasks at the cheapest run's cost, while just doubling tries from one style reached 67% at twice the cost. But every chooser we could actually build tied or lost to simply sampling more. The prize is real; the picker isn't.

     Collective Experimentation InfrastructurePosttraining and AdaptationProcess Control and Tool Usepolicysmall-model
224. 2026-06-24 → 26imported · line Z5 figures

     Qwen3.5-4B Oracle Probe Synthesis MDP

     The menu of questions beat more training

     It's the menu. Just curating which eight test inputs the model chose from raised success from about 43% to 49% — a bigger jump than any training gave. Supervised coaching added a bit more (48% to 51%); preference- and reward-based training added nothing. A flawless picker allowed all 96 candidate tests hit 87%, so the bottleneck is generating good questions, not picking among them.

     Evidence-Conditioned SelectionInterpretability and DiagnosticsOperator and Skill Inventoriesoperator-searchpolicysmall-model
225. 2026-06-25imported · line Y8 figures

     Qwen Tail Repair Stability Critic

     The right fix existed; the picker missed it

     No. A correct fix sat among the candidate rewrites for about nine in ten programs, but the trained judge could not tell which rewrite was right from summary statistics alone, so it played safe and edited nothing, staying at 45% correct, exactly the no-repair mark. A picker allowed to peek at the answer hit 91% from the very same fixes.

     Benchmark GeneralizationStructured Execution and Compilersabiexecutionrepair
226. 2026-06-25imported · line Y5 figures

     Qwen Readable Candidate Verifier

     Giving the model more detail made it worse

     Barely. Training a picker on the model's reading of each fix's pseudocode plus its claimed output raised the share of tasks fixed correctly from 44% to 51%, closing only about 15% of the distance to a perfect picker's 90%. And piling on detail backfired: handing the model a step-by-step execution trace crashed accuracy from 50% down to 8%, far worse than doing nothing.

     Benchmark GeneralizationEvidence-Conditioned SelectionInterpretability and Diagnosticsexecutionrepairsmall-model
227. 2026-06-25imported · line Y4 figures

     Qwen Candidate-Conditioned Trace Verifier

     It couldn't tell working fixes from broken ones

     No. Doing nothing already solved about 44% of tasks, and always choosing a correct candidate could reach 90%. Yet no trained picker captured any of that headroom; the best merely tied doing nothing. Left unchecked, the model rewrote almost every program, including ones already correct, breaking up to 97% and crashing accuracy to about 1%.

     Benchmark GeneralizationEvidence-Conditioned SelectionInterpretability and Diagnosticsexecutionrepairsmall-model
228. 2026-06-25imported · line Z4 figures

     Qwen3.5-4B Verifier-Guided Self-Improvement Report

     Learning from your own wins narrows your range

     No. After a four-billion-parameter model retrained on its own test-passing code, its success on unseen problems barely moved, going from about 65% to about 65% (a hair lower). Retraining made its several attempts look more alike, shrinking variety. The simplest lever won instead: just generating more attempts at solving time reached about 69%, beating every retraining approach.

     Algorithmic Memory and RetrievalEvidence-Conditioned SelectionInterpretability and Diagnosticscoveragecounterexampleslora
229. 2026-06-25imported · line Z6 figures

     Qwen3.5-4B Trained vs Frozen Repair MDP Report

     More fresh tries beat a trained code fixer

     Retry from scratch. Given the same compute budget, a few more fresh attempts rescued 5 of 57 stuck problems, while a fixer trained to patch broken code rescued only 2 — and its repairs secretly failed the hidden tests nearly a third of the time. An untrained fixer helped slightly (3 of 57), but neither repair approach beat plain retries.

     Algorithmic Memory and RetrievalEvidence-Conditioned SelectionPosttraining and Adaptationcoveragememorypolicy
230. 2026-06-25imported · line Z2 figures

     Qwen3.5-4B Strategy Token Diversity LoRA

     Naming coding tactics did not beat plain retries

     No. Feeding the model labeled coding tactics recovered 37.5% of the problems it first failed, slightly below plain extra sampling at the same cost, which recovered 41.7%. Tellingly, scrambling the labels so they meant nothing recovered just as many, proving the model was not truly switching tactics — it only gained from taking more shots. Only far pricier pooling of roughly triple the tries reached 62.5%.

     Posttraining and AdaptationProcess Control and Tool Uselorapolicysmall-model
231. 2026-06-25imported · line Z6 figures

     Qwen3.5-4B Real Sample Verify Commit

     The weak link is writing code

     It's the writing. When a 4-billion-parameter model writes several code attempts per problem, choosing is not the bottleneck. On easier problems nearly every attempt is already correct, so the best result sits near 97%. On harder problems at least one attempt is correct only 60% of the time, and the trained checker captures that entire 60% — grabbing the first attempt that passes the visible examples reaches only 55%. No checker can invent programs that were never written.

     Algorithmic Memory and RetrievalEvidence-Conditioned SelectionInterpretability and Diagnosticscoveragememorysmall-model
232. ~2026-06-25imported · line Z4 figures

     Qwen3.5-4B Oracle-Distilled Semantic Verifier

     Training a code judge helps at home

     Yes, but only on home turf. On the problem set it trained on, the trained judge picked a genuinely-correct program 81% of the time, up from 72% for the same model untrained and just 44% for grabbing the first candidate that passes the visible test. On a fresh, unfamiliar problem set, though, the untrained model judged better, 94% versus 90% — training sharpened its home instinct while dulling its general one.

     Evidence-Conditioned SelectionInterpretability and DiagnosticsPosttraining and Adaptationabidistillationexecution
233. ~2026-06-25imported · line Z4 figures

     Qwen3.5-4B HumanEval Adaptive Evidence Budget

     Deciding when to stop cannot rescue wrong picks

     No. Every strategy landed at the same 16.7% correct-pick rate — running no tests, running all eight, and even a strategy allowed to peek at the right answer to time its stop. The trained model did learn to quit sooner, using under four tests instead of eight, but its choice stayed wrong: grouping programs by matching outputs never reveals which group is actually correct, and a correct program was available for every task.

     Benchmark GeneralizationEvidence-Conditioned SelectionInterpretability and Diagnosticsexecutionsmall-modeltable-transform
234. 2026-06-25imported · line Z5 figures

     Qwen3.5-4B Diversity-Keyed Coverage Gate

     Most missed coding problems needed more varied tries

     Mostly the second. Of 24 Python problems a 4-billion-parameter model missed on four tries, spending more and more varied sampling recovered 15, lifting the share solved from 70% to nearly 89%. Mixing three creativity settings beat any single one, 13 recovered versus 10, because each cracked different problems. Nine stayed unsolved: genuine capability gaps.

     Algorithmic Memory and RetrievalEvidence-Conditioned SelectionStructured Execution and Compilersabicoveragememory
235. 2026-06-24 → 25imported · line Y6 figures

     Qwen Compiler Multi-Seed Reattribution

     Random seed, not recipe, decides whether it works

     No. The best schedule averaged 43% correct on plainly worded problems, yet the identical training swung from total failure to 81% just by changing the random starting number — so no schedule earns credit for the wins. Both growing programs from short to long and jumping straight to full length collapsed unpredictably. The seed, not the recipe, decided the outcome.

     Benchmark GeneralizationCollective Experimentation InfrastructureStructured Execution and Compilersabiexecutionlatent-state
236. 2026-06-24imported · line Y5 figures

     Qwen VM-ECHO Trace Distillation

     Predicting a program's output barely improves writing one

     Mostly no. The model got far better at predicting execution, with reading a running program's top value climbing from under 1 percent correct to 43 percent, but that rarely improved the programs it wrote. First-try accuracy on standard problems even slipped from 85 to 81 percent. A few test-and-fix settings gained slightly, 93 to 96 percent, so it is a real limit, not a collapse.

     Posttraining and AdaptationStructured Execution and Compilersabidistillationexecution
237. 2026-06-24imported · line Y4 figures

     Qwen VM-Agent ECHO QLoRA

     Acting in a loop helped

     Only the acting helped. Editing-and-running in a loop lifted the share of tasks solved from 10% at a blank start to 43%, beating a single one-shot guess near 37%. But adding a second job, predicting the program's output after each edit, dropped success to 36%. The model grew fluent at foreseeing results yet quit before finishing even more often.

     Posttraining and AdaptationStructured Execution and Compilersabiexecutionlora
238. 2026-06-24imported · line Y7 figures

     Qwen Structural Latent Compiler Expansion

     Grown to longer chains

     Yes, mostly. The model fills fixed slots with operations a calculator runs — no text, no trying many guesses and picking one. After the learned short form was copied into bigger ones, it stayed perfectly correct on 8- and 16-step chains and hit about 83% on the hardest 24-step ones, where reworded versions of a problem still computed identically.

     Benchmark GeneralizationStructured Execution and Compilersabiexecutionlatent-state
239. 2026-06-24imported · line Y7 figures

     Qwen Structural Compiler Attribution Ablation

     Graded practice, not clever expansion, builds long programs

     It's the practice schedule. Give the model its full size from the start, then feed examples easy-to-hard — 8 steps, then 16, then 24 — and it solves nearly every standard 24-step program (about 97%). The popular guess, growing short programs by copying earlier steps into the new ones, managed under 2%. And training only up to 16 steps, then testing at 24, got exactly zero.

     Benchmark GeneralizationInterpretability and DiagnosticsStructured Execution and Compilersabiexecutionlatent-state
240. 2026-06-24imported · line Y5 figures

     Qwen Search-Augmented Rollout Distillation

     Perfect one-step fixes still won't teach planning

     No. An automatic search found a verified correct fix for 98% of the dead-ends the model wandered into, yet retraining on those single fixes matched or trailed the simpler training on four of five test sets. The model could pick the right next edit 60% of the time but solved at most 37.5% of whole tasks. Local fixes never became a winning plan.

     Posttraining and AdaptationStructured Execution and Compilersabidistillationsmall-model
241. 2026-06-24imported · line Y5 figures

     Qwen Recurrent VM Repair Policy

     Step-by-step self-repair helps but most gains stay locked

     Yes, but only partway. Letting the model run its program, read the output, and fix one line at a time roughly tripled accuracy, from about 11% to 34% on the main test and 16% to 40% on reworded prompts. But a perfect editor reaches near 100% and a fully trained program writer hits 68 to 84%, so most of the possible gain stays unclaimed.

     Operator and Skill InventoriesPosttraining and AdaptationProcess Control and Tool Useabiexecutionoperator-search
242. 2026-06-24imported · line Y5 figures

     Qwen In-Policy VM-ECHO Distillation

     Learned how programs run, not which ones work

     Only partly. It became excellent at predicting mechanical outcomes, like how deep a program runs (about 92% right), but stayed no better than a coin flip at judging which program is actually correct. So it could not rank its own guesses, and ranking did worse than plain answering on every test. First-try accuracy on paired problems stayed stuck near 9%, far below the 77% a fully-supervised version reached.

     Posttraining and AdaptationProcess Control and Tool UseStructured Execution and Compilersabidistillationexecution
243. 2026-06-24imported · line Y4 figures

     Qwen Fuyu VM GRPO-ECHO

     Self-coaching made the program builder worse

     No. Copying worked solutions alone solved about 10% of tasks; one round of reward-based self-coaching dropped that to 7-8%. The coaching produced encouraging signals — it could rank good fixes over bad ones 80% of the time — yet it scrambled the model's judgment of which edit to make and when to stop. Exhaustive search on the same tasks reaches 83%.

     Posttraining and AdaptationProcess Control and Tool UseStructured Execution and Compilersabipolicysmall-model
244. 2026-06-24imported · line Y4 figures

     Qwen Dense-State DAgger VM Agent

     Step-by-step edits beat prompting but trail the expert

     Mostly yes. Teaching the model to make one edit at a time, then correcting it on the messes it made, roughly doubled accuracy on mixed tasks (22% to 41%) and won on four of five task types. But it lost on ordinary tasks (41% versus 56% for plain prompting) and stayed far below an all-seeing expert solver reaching 81 to 97%. Useful, not a replacement.

     Posttraining and AdaptationStructured Execution and Compilersabidistillationlatent-state
245. 2026-06-24imported · line Y6 figures

     Qwen Counterfactual Trace Preference Distillation

     Self-grading yields running code, not correct code

     Barely. The self-grader learned to favor programs that run without crashing, but it identified the truly correct one only about 15% of the time, against a 41% best-possible ceiling. On fresh questions it picked worse than just trusting the model's first attempt. Given a real answer key, the same machinery hit 91%. The grader, not the compiler underneath, is the wall.

     Posttraining and AdaptationStructured Execution and Compilersabidistillationrepair
246. 2026-06-24imported · line Y5 figures

     Qwen Action-Conditioned VM-ECHO Policy Iteration

     Learning to judge its own code barely helped

     Barely. Learning to grade drafts by their run results nudged picking accuracy only from about 10% to 11%, far short of the 37% reachable by always choosing the best available draft. It reliably picked programs that ran, just not ones that were correct. Yet feeding even those imperfect picks back still lifted directly-written program accuracy from 18% to 23%.

     Posttraining and AdaptationProcess Control and Tool UseStructured Execution and Compilersabiexecutionpolicy
247. 2026-06-24imported · line Z6 figures

     Qwen3.5-4B Sketch Coverage Shift Probe

     You cannot select a program never built

     Only when the template explicitly names the new operation. On tasks needing a brand-new operation, hand-written templates that named and typed it kept the correct program every single time; auto-generated templates kept it just 17% of the time, and stripped-down templates never. And no amount of extra candidates per blank or follow-up test questions could recover a program the builder never proposed.

     Algorithmic Memory and RetrievalBenchmark GeneralizationEvidence-Conditioned Selectioncoveragememorysmall-model
248. 2026-06-24imported · line Z5 figures

     Qwen3.5-4B Oracle Process GRPO

     Simple copying nearly matched every fancier method

     Yes, but with sharp limits. Teaching the model to copy a flawless test-picker lifted its success from about a third of puzzles to 43%, versus roughly 31% for picking blindly. Layering on preference-based and reward-based training gained almost nothing more. And where the available tests simply can't tell two rules apart, even perfect choices topped out near 9%.

     Benchmark GeneralizationEvidence-Conditioned SelectionInterpretability and Diagnosticsexecutionoperator-searchpolicy
249. 2026-06-24imported · line Z5 figures

     Qwen3.5-4B Operator Inventory Search Pilot

     Search finds any operation already in its toolbox

     Yes, but only if that operation sits on the program's list of candidates. Given the full set of seven operations, the program recovered the correct hidden one every time — matching against examples alone already solved about nine in ten cases, and a couple of extra probing inputs closed the rest to perfect. A shorter list that left the operation out recovered zero. Search was never the bottleneck; stocking the candidate was.

     Operator and Skill Inventoriesoperator-searchsmall-model
250. 2026-06-24imported · line Z5 figures

     Qwen3.5-4B Operator Inventory Scaling Stress

     Right answer always present, still hard to pick

     It depends on what you mean by break. The correct program always stays inside the pile you search, even with two blanks and 512 tools. But picking it out from a few examples alone drops to under half, versus always solved with a single blank, and the search balloons to over a quarter-million combinations per problem. The wall is ambiguity and compute cost, not reachability.

     Operator and Skill Inventoriesoperator-searchsmall-model
251. 2026-06-24imported · line Z10 figures

     Qwen3.5-4B Learned Active Trace Policy

     Wins the side tests, loses the main one

     It depends. On the main test set a simple even-splitting rule beat the trained picker after one extra input — 91 percent of programs fully correct versus 87 — and the best-possible choice reached 97 percent. The picker did edge ahead on two other test sets. But its inputs ruled out only about 8 wrong candidates on average, versus 20 for the ideal pick.

     Active Evidence AcquisitionPosttraining and AdaptationProcess Control and Tool Useactive-learningexecutionpolicy
252. 2026-06-24imported · line Z4 figures

     Qwen3.5-4B Joint Shortlister Ladder

     Emitting two looked-up codes at once fails

     No. Across every version — untrained, trained, and with the glossary's descriptions scrambled — the model got both codes exactly right zero percent of the time, even when allowed sixteen guesses. Training pushed single-code accuracy from near zero up to roughly one in eight, but scrambling the glossary barely changed that, proving it was pattern-matching familiar-looking codes, not actually reading the lookup.

     Benchmark GeneralizationOperator and Skill InventoriesStructured Execution and Compilerscurriculumexecutionoperator-search
253. 2026-06-24imported · line Z4 figures

     Qwen3.5-4B Inventory Shortlister Training

     Training failed to teach picking the right tools

     No. Even after extra training, and even allowed to submit its 1,024 strongest paired guesses, it got exactly zero right — no better than the untrained model, and no better than a control with the tool labels scrambled. Training error dropped then leveled off, so the model learned the answer format fine; it simply never learned to choose the correct tools from examples.

     Operator and Skill InventoriesPosttraining and Adaptationloraoperator-searchsmall-model
254. 2026-06-24imported · line Z4 figures

     Qwen3.5-4B Deployable Information Ceiling Sweep

     Missing clues, not a bad strategy

     It's the clues. On the hard hidden rule, the plain strategy got just 4% right when starved of tests and clues, far below the 74% of a rival who secretly knew the answer. But allowing more tests lifted it to 86%, and more starting clues to 91% - nearly matching the cheater. The strategy was already near-best; information was the wall.

     Evidence-Conditioned SelectionPosttraining and AdaptationProcess Control and Tool Usecoveragepolicysmall-model
255. 2026-06-24imported · line Z4 figures

     Qwen3.5-4B Bucket-Belief Probe Ranker

     Training barely beat the simple test-picking rule

     Barely. After training, the 4-billion-parameter model solved 50% of puzzles within three tests, versus 49% for a simple no-learning rule that just picks the cleanest-splitting test, essentially a tie. Even limited to those same candidate tests, a strategy that peeks at the answer reaches 61%; across every available test it hits 87%. The model captured only about 1 of every 10 points on the table.

     Posttraining and AdaptationProcess Control and Tool Usepolicysmall-model
256. 2026-06-24imported · line Z4 figures

     Qwen3.5-4B Adaptive Evidence Budget Policy

     The model learns when it has seen enough

     Yes. Trained just for this single decision, the model matched the accuracy of running all ten checks — solving about 92 of every 100 tasks — while using only about five checks on average, near a perfect-hindsight stopper's roughly four. Untrained, the same model quit almost immediately and solved just 5%. Crude fixed numeric-cutoff rules also collapsed, to between 5 and 7%.

     Active Evidence AcquisitionBenchmark GeneralizationEvidence-Conditioned Selectionexecutionpolicysmall-model
257. 2026-06-24imported · line Z7 figures

     Qwen3.5-4B Active Counterexample Trace Selection

     A few pointed test cases catch hidden bugs

     Yes, but choose them well. Committing on the visible examples alone left about a quarter of picks secretly wrong, even though every one passed all the examples shown. Requesting six new test cases where the surviving programs disagreed most lifted correctness from 78% to 97%, and beat the same number of random tests at every budget.

     Active Evidence AcquisitionEvidence-Conditioned SelectionReliability and Safetyactive-learningcounterexamplesexecution
258. 2026-06-23imported · line Y6 figures

     Qwen Typed Bytecode Expert Iteration

     Self-checking helps, but worked examples win big

     It depends. Training the model only on its own attempts that landed on the correct final answer lifted unaided first-try accuracy from 62% to 73% on fresh problems — a real gain that sticks when it writes programs alone. But training on complete worked-out programs reached nearly 100%. Checking final answers helps for free, yet stalls far short of showing full solutions.

     Evidence-Conditioned SelectionInterpretability and DiagnosticsReliability and Safetyabiexecutionsmall-model
259. 2026-06-23imported · line Y5 figures

     Qwen Semantic Prefix Value Model

     Knowing the answer is reachable doesn't find it

     No. Scoring each step by whether a correct answer is still reachable pushed the top pick to about 68 percent, level with plain confidence search and short of the 71 percent from scoring steps against the known correct program. The "still reachable" signal was genuinely learnable but far too permissive: about one step in three qualifies, so it cannot single out the real winner.

     Benchmark GeneralizationStructured Execution and Compilersabiexecutionsmall-model
260. 2026-06-23imported · line Y4 figures

     Qwen Prefix-State Process Verifier

     Finds the right program, can't pick it out

     Barely. The judge got genuinely good at telling promising partial programs from dead ends, yet it lifted first-try accuracy only a few points, with hard problems going from 41% to 44%. The real gap: on hard problems a correct program sat among the candidates 70% of the time but got chosen only 41%. Knowing good from bad is not the same as surfacing the winner.

     Benchmark GeneralizationEvidence-Conditioned SelectionInterpretability and Diagnosticsabiexecutionlatent-state
261. 2026-06-23imported · line Y5 figures

     Qwen On-Policy Repair-to-Compiler

     Self-correction works, but plain examples work equally

     Yes, but the self-correction isn't the secret ingredient. On brand-new requests the model climbed from about 29% correct programs to 99% after training on its own corrected drafts. The catch: training on plain correct programs alone also reached 99%, and training on corrections alone hit 91%. So the corrections give a usable signal without any answer key, but they aren't uniquely responsible for the jump.

     Evidence-Conditioned SelectionInterpretability and DiagnosticsPosttraining and Adaptationabidistillationexecution
262. 2026-06-23imported · line Y5 figures

     Qwen Mixed-Domain Trace Verifier

     Blind proofreader fixes some mistakes, not most

     Yes, partly. On fresh tasks the frozen model alone got 46% right; the proofreader lifted that to 57%, and a cross-check that compares reworded versions of the same task reached 61%. But a correct recipe was already among the candidates 92% of the time, so the proofreader captured only about a quarter of the reachable gain, leaving most of it untouched.

     Benchmark GeneralizationEvidence-Conditioned SelectionInterpretability and Diagnosticsabiexecutionsmall-model
263. 2026-06-23imported · line Y5 figures

     Qwen LoRA Typed-Bytecode Trace Compiler

     Worked recipes teach the skill, answer keys don't

     Yes — but the win came from the teaching material, not from adapting the model. Fed fully worked recipes, it wrote a runnable recipe that reached the right answer about 68% of the time, versus only 15% when taught with final answers alone. Surprisingly, a completely un-adapted base model already hit 66%, so the extra tuning barely mattered.

     Benchmark GeneralizationPosttraining and AdaptationStructured Execution and Compilersabiexecutionlora
264. 2026-06-23imported · line Y5 figures

     Qwen Iterative Repair Policy

     One-step edits nearly double correct answers

     Yes. The frozen model alone got about 30% of programs exactly right; editing one step at a time lifted that to 53% on fresh problems, closing roughly 38% of the distance to the best a perfect fixer could reach (89%). The first edit did most of the work, and it slightly beat simply picking the single best candidate outright — with no written reasoning generated.

     Posttraining and AdaptationProcess Control and Tool UseStructured Execution and Compilersabiexecutionlatent-state
265. 2026-06-23imported · line Y5 figures

     Qwen Hidden VM On-Policy Canonical Repair

     Folding verified fixes into training changes nothing

     No. Training the model on automatically corrected recipes reached 61% on new tasks, versus 59% for plain training — a 2-point gap that is basically noise, and it left longer tasks no better. The corrections are genuinely valuable live: a checker searching nearby recipes hits 89%. But that help evaporates the moment you try to bake it into the model itself.

     Posttraining and AdaptationProcess Control and Tool UseStructured Execution and Compilersabipolicyrepair
266. 2026-06-23imported · line Y4 figures

     Qwen Hidden VM Mixed Domains

     Writing hidden programs beats guessing the answer

     Yes. Guessing the answer directly worked only about 15% of the time across six kinds of problems — arithmetic, dates, unit conversions, list totals, yes/no thresholds, and lookups. Having the model instead write a hidden program for a fixed calculator to run hit 78%. Showing it the exact worked steps during training, not just final answers, also produced far cleaner programs — 64% exactly right versus 34%.

     Structured Execution and Compilersabiexecutionlatent-state
267. 2026-06-23imported · line Y4 figures

     Qwen Hidden VM Curriculum Repair

     Chasing right answers broke the model's reasoning

     No. Feeding it nearby worksheets that merely land on the correct answer wrecked it. Plain step-by-step training scored 72% on the main mixed test; the same model after answer-chasing repair fell to 35%, and collapsed on longer problems, dropping to 9% at the hardest length versus 30% for plain training. The gradual length curriculum helped; the repair step actively hurt.

     Benchmark GeneralizationEvidence-Conditioned SelectionInterpretability and Diagnosticscurriculumexecutionrepair
268. 2026-06-23imported · line Y6 figures

     Qwen Context-Conditioned Trace Verifier

     The right answer was there but unpickable

     Mostly no. A correct program was almost always sitting in the candidate pile, for 91 to 100 percent of questions, yet the judge barely helped: it nudged easy questions from about 69 to 70 percent and actually made the hardest ones worse, dropping them from 41 to 35 percent. The wall is choosing the right program, not finding one.

     Evidence-Conditioned SelectionInterpretability and DiagnosticsReliability and Safetyexecutionlatent-statesmall-model
269. 2026-06-23imported · line Y6 figures

     Qwen Complete-Program Trace Reranker

     Right answer's always there, picker can't find it

     Barely, and it backfires on hard cases. A correct program sat in the candidate pile 91 to 100 percent of the time, yet the trained picker nudged easy prompts only from 69 to 70 percent and actively hurt longer ones, dropping them from 41 to 35 percent. The ceiling was right there; the picker just could not reach it. Selection, not availability, is the bottleneck.

     Evidence-Conditioned SelectionInterpretability and DiagnosticsReliability and Safetyexecutionlatent-statesmall-model
270. 2026-06-23imported · line Y8 figures

     Qwen Budgeted Action-Value Compiler

     Right answers get written, but rarely chosen

     Barely. On fresh problems the model drafts a correct program among its candidates 81% of the time but ranks it first only 67% of the time. A learned scorer that never sees the answer nudged that to just 70% — while a control that peeks at the answer reached 82%. The wall is choosing, not writing.

     Benchmark GeneralizationStructured Execution and Compilersabiexecutionsmall-model
271. 2026-06-22imported · line Z2 figures

     Targeted Bridge Allocation

     Worked-out reasoning beats clever example targeting

     Barely. Piling examples on the hardest never-seen combinations repaired 33% of test cases versus 28% for spreading them evenly — a three-case edge on sixty tests, and no better than the same budget spent on easy combinations. The real lever was not allocation: strip or scramble the step-by-step solution attached to each example and success crashed from 33% to under 10%.

     Benchmark Generalizationbridgerepair
272. 2026-06-22imported · line Y8 figures

     Qwen Verifier-Guided Slot Repair

     Most failed runs are one or two fixes

     Yes, mostly, but with a catch. A small model copying 24-step calculations got only about 27% exactly right on its own. A checker that knows the correct running number after every step, allowed to swap one or two bad steps, lifted that to 91%; a single swap already reached about 70%. The catch: that step-by-step answer key isn't available in real use.

     Evidence-Conditioned SelectionInterpretability and DiagnosticsPosttraining and Adaptationabiexecutionlatent-state
273. 2026-06-22imported · line Y5 figures

     Qwen Teacher-Distilled Slot Compiler

     Pointing to where the numbers sit didn't help

     No. A model trained to copy numbers and operations out of text and run a 24-step calculation got 27% of final answers exactly right; adding the pointing signal landed at 28%, a tie. Worse, agreement between two rewordings of the same problem fell from 73% to 56%. A lighter version of the signal was clearly worse, at 18%.

     Evidence-Conditioned SelectionInterpretability and DiagnosticsPosttraining and Adaptationabidistillationexecution
274. 2026-06-22imported · line Y5 figures

     Qwen Checkpoint-Selected Scheduled-State Compiler

     Light step-by-step coaching helps; a heavy dose backfires

     It depends. A light, steady dose of show-your-work coaching, kept on through the hardest problems, lifted correct answers on 24-step chains from 25% to 33% and nearly doubled agreement between two wordings of the same problem, from 41% to 71%. But heavy coaching switched off for the long chains backfired at 19%, worse than no coaching at all.

     Benchmark GeneralizationStructured Execution and Compilersabicurriculumexecution
275. 2026-06-22imported · line Z

     Qwen 3.5 4B Unsaturated Frontier Active Bridge

     Spreading practice evenly beats chasing mistakes

     Spread evenly. Giving each of ten problem types the same six extra correction examples let the model fully fix 98% of hard cases. Piling those same examples onto whichever types it failed most reached only 85%, and starved one type all the way to zero. Chasing the loudest failures robbed quieter ones that needed just a little help.

     Active Evidence AcquisitionBenchmark GeneralizationStructured Execution and Compilersactive-learningbridgeexecution
276. 2026-06-22imported · line Z

     Qwen 3.5 4B Model-In-Loop Counterexamples

     Hunting the model's own errors added nothing extra

     No. Building practice cases from the model's actual wrong answers matched, but never beat, simply hand-picking the tricky categories in advance. Both lifted the hardest problems from 64% to a perfect 100% passing every hidden test, and the hand-picked set was even a touch cleaner on a single guess (100% versus 99%). The extra error-hunting loop earned nothing here.

     Benchmark GeneralizationEvidence-Conditioned SelectionReliability and Safetycounterexamplesexecutionsmall-model
277. 2026-06-22imported · line Z

     Qwen 3.5 4B Executable Program Posttraining

     Show the worked steps and runnable fixes transfer

     Yes, but with a catch. Shown worked-through reasoning in the prompt, the model fixed unseen problem types about three-quarters of the time, versus one-in-three when the prompt showed no steps. Strip out or scramble those steps and most of the gain vanishes. One two-condition problem type stayed stubborn, rising from never solved to roughly one in three only after adding extra practice examples.

     Benchmark GeneralizationStructured Execution and Compilersexecutionrepairsmall-model
278. 2026-06-22imported · line Z

     Qwen 3.5 4B Counterexample-Directed DSL

     Curated examples sharpen known skills, not missing ones

     It depends. Hand-picked examples lifted the model's single-best-guess repair rate from 51% to 58% over random examples. But when it generated several candidates and kept the best, the edge vanished (64% slipped to 61%). One task family reached a perfect score and another was rescued, yet a third collapsed to zero, giving the same valid-but-wrong answer on all 24 hidden tests.

     Benchmark GeneralizationEvidence-Conditioned SelectionReliability and Safetycounterexamplesexecutionsmall-model
279. 2026-06-21 → 22imported · line Z2 figures

     Bridge-Dose Recombination Curriculum

     A few exact worked examples unlock skill-combining

     Yes, but only with the right examples. With none, the model solved just 7% of never-seen skill pairings; adding as few as two to four exact worked examples of each pairing lifted that to about 30%, a nearly five-fold jump. Lookalike examples stalled near 8%, and piling on too many quietly dragged familiar-task accuracy down from 86% to 58%.

     Benchmark Generalizationbridgecurriculum
280. 2026-06-21imported · line Y24 figures

     Structured Slot Initializer Ladder Experiment

     One slot per value beats a bigger network

     Structure, not scale. A plain general-purpose network placed only 55.5% of its belief on the correct starting setup and gave barely half the possible values their own slot, doubling several onto the same one. A rule that forces each value into a distinct slot lifted that to 98.1%, essentially matching a flawless setup. Learning the underlying math alone was not the missing piece.

     Benchmark GeneralizationStructured Execution and Compilerscurriculumexecutionlatent-state
281. 2026-06-21imported · line Y16 figures

     Sparse Support Memory Executor Experiment

     One memory slot per possibility recovers exact reasoning

     Only when its scratch memory held one slot for every possible starting value. With that, it answered every question correctly through the longest 24-step programs. Cut the memory roughly in half and accuracy fell to about 69%, because whole batches of candidates got erased mid-program, and about 1 in 3 examples lost every candidate and fell back to a guess. A wider memory vector is not the fix; enough addressable slots is.

     Algorithmic Memory and RetrievalStructured Execution and Compilersexecutionlatent-statememory
282. 2026-06-21imported · line Z3 figures

     Rule-Family Diversity Scaling Experiment

     Variety beats volume for fixing unseen code

     Yes. Holding training at 240 examples but spreading them across 12 kinds of problems instead of 3 lifted fixes on completely unseen problem types from 0% to 29%, while fixes on familiar problems stayed high. But it only works because the model reads the failing-run evidence: strip that evidence and transfer drops to 0%. And the gain is lopsided, with one unseen type hitting 92% while two others never worked at all.

     Collective Experimentation Infrastructureexperiment
283. 2026-06-21imported · line Y3 figures

     Qwen Trace Bootstrap Retention Experiment

     Final answers keep a skill they cannot teach

     Yes. Once step-by-step labels install the skill, training on final answers alone preserves and even sharpens it: 97% of the longest 24-step problems solved exactly, versus about 1 in 100 — no better than guessing — when the same setup learns from final answers only from the start. Final-answer feedback can refine an installed skill but never discovers it.

     Structured Execution and Compilersexecutionlatent-statesmall-model
284. 2026-06-21imported · line Y3 figures

     Qwen Structured Bridge Experiment

     Compile the steps, don't guess the answer

     Yes, but only when you show it the individual steps during training. A tiny translator turning the frozen model's read into calculator instructions solved chains far longer than it trained on: 96% correct at twelve steps and about 88% at twenty-four, versus roughly 1 in 100 when the same model just names the final answer. Rewarding correct answers alone never found the trick.

     Benchmark GeneralizationStructured Execution and Compilersbridgeexecutionlatent-state
285. 2026-06-21imported · line Y5 figures

     Qwen State-Ladder Compiler

     Easy-to-hard practice won; grading every step didn't

     No. Grading the running total after every step never beat an identical model graded only on its final answer, and at full strength it collapsed on the hardest long programs. The real winner was the training schedule: starting on short chains and working up to 24-step ones lifted the hardest cases from near-random to about 1 in 5, and standard cases to about 2 in 5.

     Benchmark GeneralizationStructured Execution and Compilersabicurriculumlatent-state
286. 2026-06-21imported · line Y3 figures

     Qwen Span-Free Compiler

     Works only when you show it where to

     Only when it is first taught where to look. Fed just the frozen model's raw internal notes, a plain reader stayed near random guessing (about 1 in 97). Adding training that also highlighted which spots held the numbers and operations reached 99.6% fully-correct programs at the practiced length. But reword the prompt, or add more steps, and accuracy collapses back to chance.

     Benchmark GeneralizationStructured Execution and Compilersabiexecutionlatent-state
287. 2026-06-21imported · line Y3 figures

     Qwen Slot-Stability Compiler

     A bolted-on calculator beats teaching the model arithmetic

     Yes, mostly. The point-and-compute helper solved about 91% of short problems and around half of medium ones, while training the same model to just emit the final answer never beat random guessing, about 1 in 60, at any length. But an extra rule forcing two rewordings of a problem to agree was a wash: it helped the longest reworded chains yet hurt agreement on medium ones.

     Structured Execution and Compilersabilatent-statesmall-model
288. 2026-06-21imported · line Y3 figures

     Qwen Shared Parser Compiler

     Its hidden math is readable, but fragile

     Only when every step is taught directly. Given step-by-step labels, the add-on rebuilds short programs well — nearly 4 in 5 four-step problems run exactly right — but accuracy fades to 39% at twelve steps and under 1% at twenty-four. Reword the same problems and it collapses toward guessing. Training on final answers alone never finds this ability at all.

     Benchmark GeneralizationStructured Execution and Compilersabiexecutionlatent-state
289. 2026-06-21imported · line Y3 figures

     Qwen Numeric-Copy Compiler

     Point to the numbers, don't recompute them

     Yes. When the model just points to where each number and operation sits and copies the exact symbols for a hidden calculator to run, it solves four-step problems about 90% of the time. A version trained to write the answer itself never left the floor, at most 3% at any length. A shortcut trained only on final answers aced four steps but crashed to zero at eight, never learning the ordered pointing that stretches to longer chains.

     Structured Execution and Compilersabismall-model
290. 2026-06-21imported · line Y3 figures

     Qwen LoRA Parser Compiler

     Small model shows its work but smudges the

     Partly. With step-by-step coaching, a small four-billion-parameter model's hidden states became a readable program: it named the starting number every time and picked the right operation about 98 percent of the time, while ordinary answer-only training stayed near zero. But the actual numbers came through only about 80 percent of the time per step, so exact recipes fell to zero by twelve steps.

     Posttraining and AdaptationStructured Execution and Compilersabilorasmall-model
291. 2026-06-21imported · line Y12 figures

     Learned Sparse Slot Executor Experiment

     Learned state tracking works small, breaks at scale

     Yes, but only small. With eleven possible values and a ready-made scratchpad, the network kept the fully correct answer in view 95.5% of the time, even on longer chains than it trained on. Widen to thirty-one values and the same design collapsed to 12.3%, while a hand-coded updater stayed perfect throughout.

     Algorithmic Memory and RetrievalStructured Execution and Compilersexecutionlatent-statememory
292. 2026-06-21imported · line Z2 figures

     Feature-Factorized Rule Diversity

     Worked steps carry transfer, not the practice mix

     No. All three practice diets fixed only about 1 in 5 brand-new bug combinations, so mixing bought nothing over drilling either kind alone. What actually mattered was showing worked, step-by-step repair reasoning during training: strip it out and success collapsed to about 1 in 30. And most of that thin transfer came from a single lucky bug family, not broad skill.

     Algorithmic Memory and RetrievalBenchmark GeneralizationEvidence-Conditioned Selectioncoveragememoryrepair
293. 2026-06-21imported · line Z2 figures

     Factor Recombination Ladder

     Models master each fix but cannot blend skills

     No. Trained on worked solutions, the model fixed about 81% of bugs when two skills were paired the way it saw in training, but only 8 to 10% when the same familiar skills were paired in a new way. Adding skill labels lifted some familiar cases yet moved that wall not at all. It learns each pairing, not how to recombine skills.

     Benchmark Generalizationcurriculumrepair
294. 2026-06-21imported · line Y28 figures

     End-to-End Structured Slot Executor Experiment

     Tracking every possibility needs structure at both ends

     Yes, but only when both halves carry built-in structure. On the numbers 0 to 30, the full model puts 98% of its confidence on the exactly correct final set of possibilities, nearly matching a version handed the answer, and still 95% on the wider range 0 to 96. Give it a generic starting set and even a flawless updater drops to 29%; a generic updater collapses it to 5%.

     Structured Execution and Compilersexecutionlatent-state
295. 2026-06-21imported · line Y16 figures

     Dense Teacher Distillation Experiment

     Perfect teaching still can't compress exact reasoning

     No. Even with a flawless teacher revealing the exact set of still-possible answers at every step, the fixed-size memory learned only a rough approximation. The best version placed 52% of its confidence on the correct final answer but just 22% on the exact full set of possibilities, and both collapse as programs get longer. More memory helps but never reaches exact.

     Posttraining and AdaptationStructured Execution and Compilersdistillationexecutionlatent-state
296. 2026-06-21imported · line Y20 figures

     Cyclic Transition Ladder Experiment

     Clock-arithmetic parts beat a bigger generic network

     It needs the matching wrap-around parts. A network built from clock-arithmetic moves stayed perfectly exact on programs three times longer than it practiced on. A plain generic network of the same size drifted down to just 6% exact answers, and feeding it clock-style coordinates made it slightly worse, not better. Structure wins here, not raw size.

     Benchmark GeneralizationStructured Execution and Compilerscurriculumexecutionlatent-state
297. 2026-06-20imported · line Z3 figures

     Trace-Keyed Symbol Repair Experiment

     The fix was hiding in the error message

     Yes. After light training, a small coding model fixed all 60 test cases — including value styles it never saw during training — by copying the correct value straight out of the failure message. Take that message away or scramble it and repair drops to none. An untrained model pulls the right value out only about 18 percent of the time, so this skill genuinely had to be trained in.

     program review needed · proposedrepair
298. 2026-06-20imported · line Z3 figures

     Counterexample Rule Repair Experiment

     Failing examples reveal the fix; new rules don't

     Yes, but only for rule shapes it has practiced. Given the failing examples, a small fine-tuned model fixed 91% of bugs on familiar tasks and got unseen inputs right too, versus 0% untrained and near-zero for every version fed no examples or scrambled ones. Strip the real examples out and the same model drops to 0%, proving it truly reads them. Hand it a rule shape never trained on, and nothing works.

     Evidence-Conditioned SelectionReliability and Safetycounterexamplesrepair
299. 2026-06-19 → 20imported · line Z8 figures

     Execution-Conditioned Repair LoRA Experiment

     Trained code fixer memorizes bugs, never generalizes

     No. On bugs built from the same templates it practiced on, the fixer repaired all 60 of 60 cases, versus 11 of 60 with ordinary patch training and 6 of 60 with no training at all. But on bug types it never saw, every method fixed zero of 27, and scrambling or deleting the error message changed nothing. It memorized templates, not repair.

     Posttraining and AdaptationStructured Execution and Compilersexecutionlorarepair
300. 2026-06-19imported · line Y3 figures

     Qwen Fast-Weight Hook Experiment

     Extra silent thinking loops bought no accuracy

     No. Adding more silent internal loops gave no reliable accuracy gain. On small 100-problem tests a few settings looked promising, but rechecking those best moments on 250 problems erased the edge — zero loops did as well or better. A control loop that never actually trained fluked similar bumps, proving the gains were noise, not real internal reasoning.

     program review needed · proposedsmall-model

No experiments match these filters . Clear filters