Confirmed
✓ ConfirmedC1
Make small models show their work, not just answers
Across many experiments, small models got more reliable when they produced a checkable step-by-step output — like a short program a calculator can run, or filled-in operation slots — instead of writing a final answer directly. On 250 table tasks this lifted exact-match accuracy from 55% to 62%, fixing 18 previously wrong answers.
Implication. For any task where you ask a small model for a direct answer, first try a version that emits a checkable, runnable intermediate step and measure whether accuracy improves.
Evidence (3)
- 2026-06-24 Qwen Structural Latent Compiler Expansion
This report is intentionally standalone. The key readout is whether expansion improves or preserves executable accuracy at longer lengths, and whether paraphrase-paired programs co
- 2026-06-27 Qwen3.5-4B Foofah Selective Program Fallback
This standalone experiment tests whether a generated Python table-transform program should be used as a fallback to direct JSON generation on Foofah table-transformation tasks. The
- 2026-06-24 Qwen3.5-4B Operator Inventory Search Pilot
This standalone no-training pilot tests the search-side ceiling for type-colliding operator identification. Every aggregate candidate has signature list[int] -> int; the task is to
Next tests (2)
- On one shared test set, compare plain text answers, calculator-runnable programs, filled operation slots, and step-by-step executors head to head.
- Separately test what changes the output format versus what changes the training signal, so you know which one actually helped.
Avoid
- Reporting another win for step-by-step output without also testing plain direct answers and deliberately broken intermediate steps as controls.
✓ ConfirmedC2
A right answer in the pile is useless if you can't pick it
Across these experiments, a set of guesses often contained a correct answer, yet the model — judging only from what it could actually see — still picked or committed to a wrong one. Pulling a similar past solution surfaced fixes but couldn't reliably choose them; trusting a program because it matched the shown examples still shipped errors.
Implication. Never claim progress from "a correct answer exists in the list." Measure how often the model, using only visible evidence, actually picks the right one and how often it commits to a wrong one.
Evidence (3)
- 2026-06-26 Qwen3.5-4B Retrieval Adapt Verify Scale
This is a positive coverage result and a negative selector result. The positive part is that semantic retrieval plus Qwen adaptation works on this 24-task residual scale: it recove
- 2026-06-27 Qwen3.5-4B Foofah Selective Program Fallback
This standalone experiment tests whether a generated Python table-transform program should be used as a fallback to direct JSON generation on Foofah table-transformation tasks. The
- 2026-06-26 Qwen3.5-4B Independent Retrieval Consensus
The hypothesis was plausible: independent derivations agreeing should provide evidence unavailable to any single-candidate judge. The implementation successfully increased retrieva
Next tests (2)
- Score selection systems only on what they can actually see at decision time, using guesses from problem types they were never trained on.
- Rewrite 'a correct answer exists somewhere' reports as scorecards showing the gap to what actually gets picked and shipped.
Avoid
- Treating 'the right answer is somewhere in the list' as evidence that the system can pick it.
✓ ConfirmedC6
A finding you can trust needs a control that could break it
Across this work, the results that held up were the ones tested against a deliberately broken version of themselves — inputs scrambled, labels shuffled, a part frozen, or the model faced with genuinely new combinations instead of familiar ones. Without such a check, an apparent effect is often just the setup rewarding memorized patterns.
Implication. Before running costly experiments, decide which control would make the effect vanish if it were fake, and run that control alongside the real test.
Evidence (3)
- 2026-06-21 Factor Recombination Ladder
The experiment found a sharp split between in-distribution repair learning and held-out factor recombination. Correct trace training worked well on seen factor combinations: 80.6%
- 2026-06-21 Feature-Factorized Rule Diversity
The best recombination score was the composite-trace adapter at 14/60; mixed trace reached 13/60 and mostly transferred only the sorted_join_holdout family.
- 2026-06-24 Qwen Structural Compiler Attribution Ablation
The strongest arm is the max-24 compiler trained from the start with the staged length curriculum. Copied structural expansion is not the winning explanation in this run.
Next tests (1)
- Build one shared test suite of these break-it controls and reuse it across every line of work.
Avoid
- Do not count gains on the same kind of examples the model trained on as proof it will generalize to new ones.
✓ ConfirmedC7
Never let answer-key scores pose as real-world scores
Across experiments, trustworthy conclusions depended on strict record-keeping. Keep the small files needed to rerun a result, but keep bulky trained model weights out of version control. Above all, clearly flag any score that was only reachable because the true answers were known during scoring, kept separate from scores earned without that peek.
Implication. Tag every result by whether it used a hidden answer key; report the answer-key number and the honest no-peek number separately, and keep trained model weights out of git.
Evidence (4)
Next tests (1)
- Add automatic checks that flag missing rerun files and catch results that quietly relied on knowing the true answers.
Avoid
- Committing trained model weights to git, or letting a score that only works when the true answers are known become your headline real-world claim.
✓ ConfirmedC8
Treat your research repo as an instrument, not just storage
Across this work, the shared repository — its plans, auto-built indexes, running list of tested claims, scorecards, intake notes, templates, and automated checks — turned out to decide whether later experiments build on earlier ones or repeat them. Well-kept records are what let findings accumulate instead of getting lost or rediscovered from scratch.
Implication. When an experiment changes what future work should believe, update the shared claims and evidence records right away — not just your own notes — so the next person inherits the conclusion.
Evidence (4)
Next tests (1)
- Check whether keeping scorecards and intake notes actually cuts duplicate proposals and makes new work cite prior evidence more often.
Avoid
- Calling the process 'done' once it stops actually improving how well past findings are remembered and reused.
✓ ConfirmedC36
Small models pick the wrong plan, not wrong numbers
Across three unrelated programming task types, a 4-billion-parameter model almost never solved a problem alone (0-2 percent), and its failures were the wrong sequence of steps, not right steps with bad values. Given the correct skeleton, filling in values is trivial. So the model computes values; it cannot propose deep structure.
Implication. Stop trying to sample more from the model to find the plan. Wrap it in a tool that brute-force searches over step-sequences, fills in values, and runs the code to pick the answer that works.
Evidence (1)
Next tests (3)
- Check whether training the model to reuse learned skeletons, and its breakdown on deeper problems, also holds across these three task types (so far only tested on one).
- Test whether the split — the model knows which operation to use but must read parameters off the surface — generalizes across task types.
- Investigate why one task type (small-number register machines) lets random guessed values succeed more often; likely the smaller value space causes accidental matches, not a real difference.
Avoid
- Do not assume this is a quirk of one hand-built task language; it repeats on text-character edits and integer register machines too — the wall is always planning, and brute-force search always beats the raw model.
- Do not over-read that random values succeed more often on register machines (about a third of the time); that is just its smaller value space causing accidental matches, not a break in the pattern — the plan is still what fails.
- Remember the scope: this confirms the raw-model finding across task types, but the training-and-depth results and the internal-representation probes were tested on only one task type and still need cross-type checks.
✓ ConfirmedC49
Your fine-tuned adapter may silently never load
One serving tool silently ignored trained fine-tuning adapters for this model: it looked for weights under names the adapter didn't use, matched nothing, and raised no error. Result: two different trained adapters produced byte-for-byte identical outputs because every run was secretly testing the untouched base model. A quick "does anything change" check exposed it.
Implication. Before trusting any fine-tuning evaluation, run the exact same prompt with the adapter on versus off. If outputs are identical, the adapter never loaded and the results are meaningless.
Evidence (1)
Next tests (2)
- Fix the tool's weight-name matching so it finds this model's adapter layers, then re-confirm with the on-versus-off check
- Bake the adapter on-versus-off check into pre-run validation so no future experiment can silently measure the base model
Avoid
- Don't test whether adapters load using an empty adapter — an empty one matches the base model whether or not it loaded
- Don't treat a difference in run time as proof the adapter loaded
- Don't compare scores from two different serving tools against each other — only compare runs from the same tool
✓ ConfirmedC62
NARROW POLICY EDITS PERTURB THE 0.606 pi-DEPLOYMENT WARM-START DOWNWARD; THE REMAINING HEADROOM IS A TERMINATION/FINISHING PROBLEM, NOT ABSENT CAPABILITY
Experiment qwen35_4b_agentic_rlvr_feasibility (owner /goal 2026-07-19). Three narrow policy edits on top of the merged SFT warm-start (the pi-deployment baseline, 0.606 mean pass over an 11-task pi_split holdout, measured through pi-coding-agent itself, k=3) all FAIL to beat it: pi-RFT v1 (execution-filtered SFT on 46 passing pi trajectories, lr 1e-4) CRASHED it to 0.121 by teaching loop-forever; v2 (short-success <=8 turns + lr 2e-5) recovered to 0.576 = NULL; and DPO on synthetic decision-point pass-vs-continue pairs (chosen=terminal stop after ALL PASS, rejected=one more redundant tool call; 45 pairs/24 train tasks; learned emphatically to rewards/accuracies 1.0, margins 0.565) REGRESSED it to 0.515 (-0.091). DPO rescued 2 tasks but damaged 4 that were already solid (min_heap 1.00->0.33). The pi_rft loop-forever timeout is a SINGLE non-terminating generation (92 stream deltas, 1 message_end, <think> to the wall), not a harvestable multi-turn loop, so real timeout rejected could not be captured by the completion-logging proxy; the synthetic negative that replaced it mis-targets (it teaches termination AFTER solving, but the holdout failure is looping BEFORE solving) and, learned too strongly on few pairs, generalizes into premature stopping. FOURTH replication (elicitation/STaR): harvested 15 execution-verified best-of-n successes on the two latent TRAIN tasks (bellman_ford, topo_lex), MIXED with the 46 diverse passes to preserve breadth, SFT at a gentle 2e-5/1epoch -- on the TRAINED task distillation worked (topo_lex single-shot 0.25->0.75, k=8) but the holdout REGRESSED to 0.485 (-0.121, worse than DPO), same damage signature (min_heap/pretty_bytes 1.00->0.33). Best-of-8 also mapped the capability frontier: bellman_ford 0.88 (the earlier 0.00 was unlucky k=2), topo_lex 0.25, allocate 0.12 solvable; but seven tasks (case_convert/deep_merge/schema_lite/semver/glob_match/patch_apply/json_pointer) reach only PARTIAL credit at best-of-8 (mean partial reward 0.13-0.41, up to 0.55 on deep_merge): the model edits solution.py and passes SOME tests every time but never ALL, and every episode TIMES OUT -- 'close-but-can't-finish + termination failure', NOT absent capability. (An earlier version of this claim called them meanR-0.00 ABSENT; that was a bug -- a missing meanR field read as 0.00 via dict.get default. Corrected against episode-level rewards.) Best-of-12 ceiling probe (2026-07-21) settled it: the capability is NOT absent -- schema_lite fully closed 1/12 (maxR 1.00), deep_merge reaches 0.65 partial, and the model edits + passes some tests on 9-11 of 12 tries for most -- but ~100% of all 84 episodes TIME OUT (loop to the wall). The universal failure is TERMINATION/finishing, not missing skill.
Implication. The deployment ceiling is a TERMINATION/finishing problem, not a capability wall: the model engages and reaches partial (occasionally full) solutions but loops instead of closing/stopping. This retroactively explains DPO's failure (its 'stop after solving' signal never fires because these tasks are almost never solved). The believed-in non-training lift is EXECUTION-SELECTED best-of-n at deploy (run N pi rollouts, keep the highest-scoring by the tests) -- it captures the rare full solve + best partial and sidesteps the warm-start's edit-fragility. Training-side, the intervention worth testing targets 'recognize you are stuck -> finish or stop cleanly', not skill installation.
Evidence (1)
Next tests (3)
- Execution-selected best-of-n at deploy on the full holdout: does keeping the highest test-scoring of N=8-12 pi rollouts beat the 0.606 single-shot deployment? (non-training; the ceiling probe implies yes for schema_lite-like tasks).
- A termination/finishing intervention: cap tool iterations with a forced 'submit best attempt' at the budget, or train a stop-when-stuck signal -- test whether reducing the ~100% timeout rate lifts pass rate.
- Train ONE warm-start from base on a larger diverse harvest incl. topo_lex/bellman_ford; compare pi-holdout to 0.606.
Avoid
- Editing the working 0.606 warm-start with narrow SFT/preference signals learned to high accuracy on few examples: RFT-v1 crashed it, DPO regressed it -- the policy is fragile to overshoot.
- Framing the holdout gap as termination discipline: the loop-forever lens addressed the wrong bottleneck; the zeros are a capability gap.
- Trusting a logging reverse-proxy to capture pi_rft timeouts: its loop-forever is a single stream that never completes, so nothing is logged (0-turn rejected).
- Serving the policy to pi as --served-model-name Qwen3.5-4B (the stale pi_episode docstring): pi requests qwen35-4b-pi8k and a name mismatch 404s silently into empty trajectories.
- Editing the 0.606 warm-start with ANY narrow LoRA training (SFT, DPO, or STaR distillation): four methods have now regressed it; it is a robust local optimum and further adapters perturb deployment downward even when they help the trained task.
✓ ConfirmedC63
SINCE EDITING THE 4B's POLICY REGRESSES IT, THE DEPLOYABLE AGENTIC-CODING LIFT COMES FROM INFERENCE-TIME EXECUTION SELECTION: best-of-3 beats single-shot +0.121 on the pi holdout with ZERO training
Experiment qwen35_4b_agentic_rlvr_feasibility (owner /goal). Capstone resolving C62: four LoRA edits of the 0.606 warm-start each REGRESSED pi-holdout deployment (-0.09 to -0.12), and the best-of-12 ceiling probe showed the hard tasks fail by TERMINATION (edit + partial pass, ~100% timeout) not absent capability. Both point to inference-time selection as the lift. Measured from the baseline's own 3 rollouts/task (execution-selected = solved if ANY of 3 fully passes): single-shot 0.606 -> best-of-3 0.727 (+0.121), zero training -- the exact mirror of what every policy edit lost. best-of-3 is the floor; best-of-8/12 would capture the rare closes (schema_lite 1/12). CONFIRMED 2026-07-22 by the dedicated best-of-8 run (T=300, instrumented, checkpoint-resumed through two system crashes): execution-selected best-of-8 = 0.818 (9/11), headline criterion >=0.78 passed, case_convert's first-ever pass captured. Same-day best-of-3@600 also 0.818 at 1.8x less compute (3.1 vs 5.5 GPU-h) -- the selection-efficiency frontier is FLAT in wall-clock T (the T=300 single-shot drop, 0.443 vs 0.621 = 71% retention, matches the disk-scoring prediction of 72% almost exactly; solves land on disk before the wall and are scored from disk). Selection saturates ~0.82 at practical k because the last three tasks are rare-solve (case_convert 1/33, json_pointer 1/33, schema_lite 1/34 pooled). POOLED ACROSS ALL RUNS: 11/11 holdout tasks have >=1 warm-start solve -- the policy's solvable ceiling is 100%; per-task solve RATE is the entire remaining gap.
Implication. Deploy recommendation: N~3 pi rollouts at the full 600s wall, execution-select on the tests -- 0.818 vs 0.606 single-shot on the holdout, and the frontier is flat in T so short walls buy nothing for selection. The remaining gap is rare-solve RATE (three tasks at ~1/33), not solvability: every holdout task is solvable by the warm-start. Raising a rare-solve rate is a different problem from installing a missing capability, and both training (4x regressions, C62) and mechanical termination fixes (cap ladder, killed by its own gate) have failed to do it -- sampling past it is the only proven lever.
Evidence (1)
Next tests (3)
- Adaptive selection at deploy: spend samples only on tasks not yet solved (sequential best-of-N with early stop on first pass) -- pooled data implies ~11/11 reachable at k<=33 worst-case, far cheaper than uniform k.
- Does a partial-credit selector (keep the highest test-score rollout when none fully passes) provide a useful warm continuation for a second round?
- Port the selection protocol to real-repo tasks (the OpenEnv/OSS mining line) where the verifier is the repo's own test suite.
Avoid
- Concluding the hard tasks are unsolvable: schema_lite closed 1/12 and most reach high partial -- they are rare+termination-limited, not absent.
- Spending more on LoRA edits of the warm-start to chase these tasks: four methods regressed it; inference-time selection is the believed-in lift.
✓ ConfirmedC64
THE 4B DOES REAL-CODEBASE AGENTIC CODING: 0.70 single-shot / 0.91 execution-selected on real toolz functions via pi -- the '~0.00 on real repos' result was a HARNESS ARTIFACT
Experiment qwen35_4b_agentic_rlvr_feasibility. Stub-a-function tasks in a real toolz checkout, driven by pi-coding-agent, scored by the repository's OWN pytest suite. 11 tasks at k=3: single-shot 0.697, execution-selected best-of-3 0.909 (9/33 timeouts, mean 311s). Per-task: 4x 3/3, 5x 2/3, 1x 1/3, and only has_keywords never passed (maxR 0.59) -- 10 of 11 real repo functions implemented correctly and verified by the repo's tests. The SAME tasks measured ~0.00 in our own harness, which is why the experiment abandoned real repos for synthetic scenarios; that abandonment rested on a broken measurement, the same class of error as the 0.486-vs-0.810 synthetic harness gap (C62/C63) and the meanR-0.00 'absent tasks' bug.
Implication. Qwen3.5-4B can already do real-codebase agentic coding through a real agent scaffold; the capability was never missing, our measurement was. C63's execution-selected best-of-N replicates on this independent real-code surface (+0.21 with zero training, verifier = the repo's own suite), so the deployable recipe is: drive the 4B with pi, sample N rollouts, keep the one whose tests pass. Any future 'the model cannot do X' claim in this program must be measured in the deployment scaffold before it is believed.
Evidence (1)
Next tests (3)
- Finish the remaining 5 toolz tasks (k=3) and extend to the full 67-task set to tighten 0.70/0.91.
- Harder real repos / larger functions: does the single-shot rate fall with body_lines, and does selection still recover it?
- Re-examine every other 'the model cannot do X' conclusion in this experiment that was measured only in our harness.
Avoid
- Trusting any capability claim measured in a bespoke harness: three separate harness artifacts (real-repo ~0.00, synthetic 0.486 vs 0.810, meanR-0.00 absent tasks) all reversed when measured through pi.
- Concluding a task is unsolvable from single-shot rates: 10/11 real functions passed at least once in 3 tries.
Promising
◑ PromisingC4
Gather extra evidence to improve the decision, not curiosity
Letting a model request extra examples or test cases can help, but only when the choice is aimed at the actual decision it must make: commit, fix, or pick the best answer. One well-targeted example nudged results up; extra inputs chosen for general informativeness added noise and sometimes made things worse.
Implication. Judge any evidence-gathering method by how much it improves the final commit, repair, or pick per extra query gathered, not by how informative the extra checks look on their own.
Evidence (3)
- 2026-06-27 Qwen Active Example Acquisition
The baseline with four examples solves 66.7% of tasks. Active acquisition solves 70.0% with one extra example and 70.0% with three extra examples. Input-diversity acquisition solve
- 2026-06-24 Qwen3.5-4B Active Counterexample Trace Selection
This standalone experiment tests whether a Qwen3.5-4B typed-sketch generator benefits from actively requested execution traces after candidate synthesis. The verifier first complet
- 2026-06-24 Qwen3.5-4B Learned Active Trace Policy
This standalone experiment tests whether a Qwen3.5-4B LoRA can learn a low-budget active trace policy for selecting query inputs after typed-sketch candidate synthesis. The learned
Next tests (2)
- Compare evidence-picking strategies by how much each improves the final choice, given the same number of extra queries.
- Separate extra checks the model can run on its own from ones that need the hidden correct answer to be useful.
Avoid
- Tuning which examples or test cases to gather without showing it improves the final commit, repair, or pick.
◑ PromisingC9
Real reasoning content drives the gain, not extra compute
Turning on the model's built-in reasoning (off by default) lifted its pass rate on basic Python tasks from 76% to 91%. Controls prove the gain is genuine reasoning: same-length blank filler or scrambled reasoning scored like no reasoning, and feeding it another task's reasoning collapsed accuracy to near zero. The benefit grew with longer reasoning.
Implication. Before crediting any reasoning boost as real, compare it against blank filler and scrambled reasoning of the same length — not one best-guess score, which misled here. Re-baseline reasoning-substitute tricks against fair built-in reasoning.
Evidence (6)
- 2026-06-30 Qwen3.5-4B Thinking-Budget Scaling
Headline (n=100 MBPP, k=8), deployable greedy pass@1 by thinking budget (table on the experiment page). Native thinking is a deployable win the corpus disabled: greedy +15pp (0.76→
- ~2026-06-30 Qwen3.5-4B Thinking-Budget Controller
Headline Efficiency win: the visible-test escalation controller Pareto-dominates every fixed budget except the peak — it matches think_256/512 accuracy (~0.88) at ¼–½ the thinking
- 2026-06-29 → 30 Qwen3.5-4B Thinking Separability Probe
Best-layer probe AUC (predict full-test pass from the answer-token activation; n=100, 800/cond) (table on the experiment page). Correctness is moderately decodable from one answer-
- 2026-06-30 Qwen3.5-4B Thinking Content vs Compute
Behavioral ladder (full-pass, n=100) (table on the experiment page). Complete attribution (additive ladder no_think → filler → shuffle → real): pure compute + scaffold (filler − no
- 2026-06-30 Qwen3.5-4B Overthinking Content Ladder
Coherence advantage (real − shuffle) vs budget (table on the experiment page). The coherence advantage grows with budget (+0.105 → +0.150), refuting the "overthinking washes out co
- imported 2026-07-12 Qwen Python-Shaped Silent Executor
This standalone experiment tests whether a Qwen 4B model can execute Python-shaped mini-programs with private latent compute positions instead of emitting an explicit execution tra
Next tests (3)
- Test very long reasoning budgets to check whether the coherent-reasoning advantage shrinks once the model starts overthinking.
- Retry on harder, uncontaminated tasks where the no-reasoning baseline is weaker, leaving more room to improve.
- Trigger deeper reasoning from the model's own uncertainty signals instead of a single visible test case.
Avoid
- Calling a reasoning boost 'real reasoning' without checking that same-length blank filler doesn't produce the same gain.
- Pinning an exact best reasoning length from a single small run of about 100 tasks.
◑ PromisingC10
A small model picks its best answer only after reasoning
A frozen 4B coding model, asked to judge its own guesses as right or wrong, rubber-stamps almost everything (calls 91% correct when only 77% pass) unless it reasons first. Given room to think, its self-judgment sharpens enough to pick the best of 8 tries and close roughly three-quarters of the gap to a perfect picker.
Implication. To pick the winning guess among several, prefer a cheap actual test plus a free instant self-rating; save slow reasoning-based self-checking for when no test can run. Always compare picking methods at equal token cost.
Evidence (2)
- 2026-06-29 → 30 Qwen3.5-4B Generator-Verifier Gap
Checking is easier than doing — but only with thinking. No-think self-verification is weak/yes-biased (AUROC 0.77, says "correct" 91%); thinking makes it a real critic (AUROC 0.93)
- 2026-06-30 Qwen3.5-4B Verifier vs Visible Selector Showdown
The thinking verifier is Pareto-dominated: standalone barely beats visible (0.860 vs 0.850) at ~5x cost, and in combination the no-think verifier ties it (both 0.870). Best deploya
Next tests (3)
- Use the reasoning-based self-check to steer answer selection, with and without a real test, and chart accuracy against token cost.
- Retest self-checking against harder wrong guesses the model reasoned its way into, on data known to be uncontaminated.
- Build a loop where the model generates, checks itself by reasoning, then revises, and measure whether it keeps improving.
Avoid
- Do not call snap self-judgment reliable: without reasoning the model approves 91% of guesses and barely distinguishes right from wrong.
- Do not treat the self-checker's chosen answer as a ceiling: it ships real accuracy but stays below what a perfect picker could reach.
◑ PromisingC11
Bank the model's own verified wins into its weights
On a fresh coding task built with no training-data leakage, two approaches split apart. Letting the model run its code, read the real error, and retry did not beat simply drawing more independent attempts at equal compute. But retraining the model on its own answers that passed the tests reliably raised single-shot success, and kept compounding over repeated rounds without hurting answer variety.
Implication. Skip test-time self-correction loops; instead collect the answers your model already gets right on your checks, retrain on them, and repeat. Expect it to sharpen problems it can already sometimes solve, not unlock ones it never can.
Evidence (1)
Next tests (3)
- The gains plateau and never crack the hardest problems the model can't solve even occasionally — find what pushes that frontier (richer building blocks, a curriculum, or a genuinely new training signal) without leaning on a stronger teacher model.
- Test whether capability banked this way carries over to a different clean task, or only helps within the same family of problems.
- Re-run the earlier failed self-improvement on a standard coding benchmark, this time with leakage controls, to confirm that contaminated data — not the method — caused that failure.
Avoid
- Don't read this as 'execution feedback is useless' — it specifically failed to beat equal-compute sampling for this one model on this one task.
- This still rests on a single task and a single model; whether it holds across other tasks and models is untested.
◑ PromisingC12
A step-checking tool, not smarter planning, extends the frontier
A fixed small model stalls on 3-step synthesis problems it can't reach by training on its own guesses. Solving one step at a time with a tool that runs each step, then fine-tuning on the solutions found, extends its reach — from 1 in 8 solved to about 2 in 5. But blind trial-and-error solves them too, so the tool, not the model's planning, does the work, and true gains stay small.
Implication. To push a small model past its own ceiling without a bigger teacher, harvest verified solutions via step-by-step search with a code-runner, then fine-tune on them — credit the tool, not the model's planning.
Evidence (1)
Next tests (3)
- Repeat the loop — harvest solutions, fine-tune, then harvest again with the improved model — and see whether the gains compound or stall.
- Use a much larger set of building-block operations, where blind enumeration becomes infeasible and the model's guidance must actually carry the search — does it hold up?
- Test deeper problems (4-5 steps) to see whether the fine-tuned model generalizes beyond any depth it was trained on.
Avoid
- Don't claim the model out-searches blind trial-and-error — it solves no more problems overall; its planning only saves steps.
- Don't overstate the fine-tuning gain — absolute solve rates stay low, and the strongest single-attempt improvement is within noise on one small run.
- Don't treat a problem's nominal step-count as its difficulty without checking behavior — 40% of random 3-step problems are actually solvable in 2 or fewer steps.
◑ PromisingC13
Small models run plans but can't reverse-engineer them
This 4-billion-parameter model reliably turns a given step-by-step plan into correct code, even four operations deep. What it can't do is work backwards from observed behavior to the plan that produced it: each added step drops its odds of naming the right sequence roughly 30-fold. The deficit is multi-step mental simulation, not execution.
Implication. Hand the model the plan and let cheap tools (enumeration plus an interpreter) do the step-finding; don't train on execution, which is already near-perfect. Verify a task's true step-depth behaviorally before trusting its label.
Evidence (1)
Next tests (6)
- Test the simulator directly: give a fixed recipe and input, ask only for the output (no code), and watch accuracy fall as the recipe lengthens.
- See whether training on worked traces (input to each intermediate state to final output) can teach the model to simulate multi-step behavior.
- Probe whether showing each step as its own mini-example fixes things, to learn if the deficit lives in splitting a chain into steps.
- Ask the model to pick which of two candidate recipes produced a given behavior, with no code writing involved.
- Sweep how much extra thinking time helps at a fixed step-depth, to see if a longer scratchpad unlocks simulation.
- Check whether the roughly 30-fold-per-step drop in step-identification holds on a different task domain.
Avoid
- Don't read near-perfect plan-following as 'composition solved' — figuring out the steps is still the binding constraint end to end.
- Don't assume the exact 30-fold-per-step drop carries over to other task types until you've tested one.
◑ PromisingC14
Fixing one skill in a small model doesn't spread
Training a 4B model to trace a multi-step process fixed that skill almost perfectly — even on longer chains it never studied, so it truly learned, not memorized. Yet no related task needing the same skill improved. Its abilities are keyed to specific input-to-output formats, not shared inner components; the repaired skill still works when explicitly invoked but won't spread on its own.
Implication. Don't assume a mechanistic explanation predicts what training transfers. Fine-tuning buys only the trained format and close neighbors, and narrow training can damage unrelated instruction-following — so add missing skills via external tools and mix formats.
Evidence (1)
- 2026-07-02 Qwen3.5-4B Simulation Keystone Repair
The simulator was fully repaired: 0.80–0.84 through depth 5 (base 0.30–0.36), +54pp length- generalization beyond trained depths, held-out-primitive transfer 0.42→0.85 (a skill, no
Next tests (3)
- Teach the skill in several formats at once — does that stop the model locking to one format and let it reuse the skill elsewhere?
- Can prompting alone get the model to apply its repaired skill inside a different task that fine-tuning failed to reach?
- Full fine-tuning versus the lightweight adapter method on the same setup — is the failure to spread just an artifact of adapters?
Avoid
- Don't read the model's collapse on the two-choice question as lost reasoning — for these fine-tuned models it reflects forcing answers into their trained output format.
- Don't generalize past this one lightweight-tuning method and model until full fine-tuning and other model families are tested.
◑ PromisingC15
Prompts compose a model's skills but can't create idea-generation
Three ways to add capability to a frozen small model each do a different job. A step-by-step prompt composes skills the model already has (a hard two-choice task rises from 74% to 83%). Training installs the sharpest skill, but it often answers in its trained format instead — cutting usable accuracy roughly in half. Only external tools let the model invent a genuinely new idea; no prompt does.
Implication. Let tools generate candidate ideas, let the prompt supply the step-by-step procedure, and let the model do the comparing. If you train a skill in, also train mixed answer formats, or the skill stays trapped behind its own output style.
Evidence (1)
Next tests (3)
- Train in about 10% mixed answer formats and check whether the skill's 95%-when-well-formatted accuracy becomes usable in practice.
- Test whether offering a fixed menu of idea choices unlocks the model where free-form idea generation fails.
- Re-measure the earlier 'thinking is at chance' result while varying token budget, answer-parser strictness, and required answer format together.
Avoid
- Don't treat the 95% figure as real accuracy — it counts only the half of answers that came out in the required form, so it's an upper bound.
- Don't read the idea-generation score nudging from 9% to 13% as real progress — it's within measurement noise.
◑ PromisingC16
Execution is free; mental simulation needs compact state
Hand a fixed 4-billion-parameter model the exact step-by-step procedure and it runs it correctly nearly every time across three unrelated task types — that reliability is a stable trait. But ask it to infer the procedure from examples and it collapses toward guessing as steps deepen. Whether it can predict a procedure's result in its head, though, depends entirely on how compact the working state is.
Implication. Spend external tools on figuring out the procedure, and on running steps only when the state is bulky — text edits always, growing lists past a few steps. For a handful of small numbers the model tracks it reliably, so a tool call there wastes effort.
Evidence (1)
Next tests (3)
- Try a task whose procedure cannot be written as code (walking a named graph described in words) to see whether reliable execution still holds.
- Keep a task equally easy to track in the head but vary how many possible operations exist, and measure whether a bigger menu makes inferring the procedure harder.
- Re-encode text-edit tasks as small tuples of numbers and check whether in-head prediction jumps to the reliable level, proving the state representation, not the task, drives it.
Avoid
- Do not treat one task type's simulation-decay curve as a fixed model trait — it depends on the task's state; compact-number tasks barely decay while text tasks fail immediately.
- Do not conclude the model cannot simulate from a single task type; the ability to track outcomes in its head is a property of how the state is represented.
- Do not read a small nonzero inference score as 'no wall' — inferring the procedure still collapses, just to a raised floor when the operation set is small and easy to track.
◑ PromisingC17
The wall is never guessing the answer, not picking it
When the correct program shows up anywhere in a batch of tries, picking it out is trivial: running each guess against a handful of known examples finds it every time, and even a random pick among the ones that pass works just as well. So the real limit is that hard problems never get guessed at all. Sampling many times and filtering recovers 2-5x the accuracy of a single try at easier difficulty.
Implication. Don't build smarter answer-pickers — with a few test examples, picking is already solved. To beat plain sample-more, change what the model proposes: give it tools, or bake verified solutions into its weights so the right answer comes up first.
Evidence (1)
Next tests (3)
- Shrink the known examples from eight down to three, then one, and find the point where picking actually gets hard and a real judge beats a random pick.
- Test whether giving the model tools or baking in verified solutions raises first-try accuracy to where sample-many-and-filter already sits today.
- See if the model's confidence or agreement across guesses can flag tasks where a program passes the known examples but is secretly wrong, so it declines instead of shipping it.
Avoid
- Don't read the self-check's perfect score as proof it's a strong judge — with eight known examples any pick works, even a random one.
- Don't build fancier answer-pickers to beat sample-more here; picking is free, so spend effort on changing what gets proposed.
- Don't assume picking stays free with fewer examples — with only a couple, wrong-but-passing programs multiply and picking gets genuinely hard.
- Don't explain the deep-difficulty floor as the model understanding some tasks better — simpler tasks just have fewer possible answers, so the right one gets guessed more often.
◑ PromisingC18
Retraining on its own verified solutions expands what a model reaches
Retraining a small model only on answers it generated and machine-checked as correct does two different things by difficulty. On easy one-step problems it just moves answers it already knew into its first guess (60% to 80%), no new skill. On harder two-step problems it genuinely learns to solve some it previously couldn't — by sharpening its guesses toward correct answers, not guessing more widely. Deep problems, lacking any training examples, don't budge, and the exact size of the two-step gain is still unsettled.
Implication. Bank your model's own verified-correct solutions, retrain on them, then let it sample many times: the ceiling rises wherever you have training examples. For problems too deep to ever solve by luck, seed examples another way first.
Evidence (1)
Next tests (3)
- Seed the training set with three-step solutions found by a tool-assisted search (plain sampling finds almost none), retrain, and test whether the model can newly reach those deeper problems.
- Run a second round of self-training to see whether the newly reachable two-step problems become solvable on the first guess, not just after many tries.
- Vary how many two-step training examples you use (a handful up to a few dozen) to map how the gain grows with example count.
Avoid
- Don't claim self-training beats plain repeated sampling on the first guess — it doesn't; it widens what repeated sampling can reach, so bank first, then sample.
- Don't expect gains at a difficulty level with too few verified examples; plain sampling can't harvest deep solutions, so you must seed them another way.
- Don't read the two-step gain as more varied guesses — the count of distinct programs actually dropped; the model concentrated its guesses onto correct answers.
- Don't cite the tripled two-step result as the effect size — it shrank to nothing under a stricter leak-proof re-run at matched training amount; larger-dose replications are the reliable evidence.
◑ PromisingC19
The reasoning wall is two different failures by depth
On one- and two-step tasks the model's internal state clearly holds the answer's first step even when its written answer leaves it out: it knows but stays quiet. Push to three steps and that internal signal nearly vanishes — it barely computes the answer at all. So a shallow wall is silence; a deep wall is genuine absence.
Implication. Nudging the model to voice what it already computes can only help where that information exists — shallow tasks. At the deep wall there is nothing to surface; install the missing step via training or external tools.
Evidence (1)
Next tests (3)
- Push the internal 'first-step' pattern into the model while it writes a two-step answer — does accuracy rise? That is the real test of whether hidden information is actually usable.
- Take a model trained to overcome the wall and check whether that training makes the first step more readable in its internal state on two- and three-step tasks.
- Read out the second and third steps internally too, not just the first, to map how much of the whole multi-step pipeline is hidden versus only its opening move.
Avoid
- Do not call the three-step wall mere silence — internally the answer barely forms there, so it is a real knowledge gap, not just an unspoken one.
- Do not assume 'readable inside the model' means 'usable for the answer' — presence is not use; the nudging test decides that.
- Do not measure this with a simple yes/no 'is the operation present' check — base rates inflate it; decoding which operation came first is the honest target.
- Do not over-read the one-step gap — the model does use its first step to answer about two-thirds of the time, so it is partly a quirk of the naming task; two-step tasks give the cleanest example of known-but-unspoken.
◑ PromisingC20
Reading a model's mind doesn't let you steer it
Before answering a two-step problem, the model internally settles on which operation to do first, and a simple reader can pick that out of its internal state about 99% of the time. But pushing that exact signal back into the model during generation barely changed what it actually said — the largest effect stayed within measurement noise. A signal you can read is not one you can control.
Implication. Don't assume a skill you can detect inside a model is one you can inject to change its output. When simple activation nudging fails, add the skill by editing weights or giving the model a tool.
Evidence (1)
Next tests (2)
- Try a stronger intervention — replace the internal signal outright, or optimize the nudge to change the output rather than just match the reader — and see if behavior finally moves.
- Check whether directly editing the model's weights to install the skill also strengthens this internal signal, testing whether weight edits add what nudging cannot.
Avoid
- Don't claim a model's readable internal skill can be steered out for free — simple activation nudging didn't move it, even for the signal a reader could pick out almost perfectly.
- Don't read the tiny two-step effect as a win — it sat within noise of a random-nudge control and below the bar set in advance.
- Don't generalize this to all steering — it's a clean failure for one simple method at one layer; stronger, targeted interventions weren't tested.
◑ PromisingC21
Self-training installs skills the model already has, never new depth
Feeding a model its own verified two-step solutions and retraining tripled its two-step success on fresh problems (from 12% to 36%), but three-step success stayed at exactly zero. A strong two-step skill does not stretch into three steps. Self-training spreads what a model can already do; it cannot invent the next rung.
Implication. To reach a harder composition depth, generate correct solutions at that depth externally (tool-assisted search), verify them, and only then retrain. Do not expect plain self-training to climb.
Evidence (1)
Next tests (2)
- Seed retraining with three-step solutions found by tool-assisted search rather than sampling, then check whether three-step success on fresh problems finally rises.
- Re-probe the retrained model to confirm it now internally represents two-step structure while three-step structure remains absent.
Avoid
- Do not expect retraining a model on its own outputs to reach a harder problem depth it cannot already solve.
- Do not misread the zero three-step result as a weak install: the two-step install was strong (tripled on fresh tasks); only cross-depth transfer failed.
- To extend one depth further, you must generate correct solutions at that depth externally, since plain sampling produces almost none.
- The same wall holds for teaching the process, not just answers: training on two-step reasoning traces improved two-step planning but still moved three-step success not at all.
◑ PromisingC22
Outside search finds hard skills; retraining installs them, weakly
A small model almost never solves three-step problems, so it has nothing of its own to learn from. But an outside brute-force search over the allowed operations does find working three-step solutions; retraining the model on those lifts three-step success from a hard zero to 5 of 40 fresh, never-seen tasks. The gain is real but weak, and shows mainly when the model reasons step-by-step, not in one shot.
Implication. To push a model past a difficulty wall, pair an outside search that reaches each harder rung with retraining that installs it — but expect weaker, mostly reasoning-time gains the deeper you go, and seed every rung.
Evidence (1)
Next tests (2)
- Feed more three-step examples and more training passes, and check whether one-shot three-step success actually climbs or just plateaus.
- After installing the three-step skill, test whether searching on the improved model finds four-step solutions more cheaply than before.
Avoid
- Don't claim retraining cleanly installs the three-step skill — the gain is weak and appears only when the model reasons step-by-step, not in a single shot.
- Don't expect installing one difficulty rung to unlock the next — four-step stayed at zero; each rung needs its own found examples.
- Don't confuse the outside search with the model — the search finds the hard solutions; the model and retraining only install them.
◑ PromisingC23
The hard reasoning wall was missing data, not a model limit
A fixed 4-billion-parameter model kept failing three-layer composition puzzles. The block turned out to be too few training examples, not the model's size. Feeding it more solutions found by an external search tool lifted its solve rate on fresh puzzles from 0% to 38% (in sixteen tries), rising steadily with no sign of leveling off.
Implication. When a small model stalls on hard multi-step tasks, try harvesting more verified solutions with an external search tool and training on them before assuming you need a bigger model.
Evidence (1)
Next tests (3)
- Keep adding solutions past the current amount to find where, if ever, the gains stop.
- Separate having more distinct examples from simply training longer, since the current test bundles the two together.
- Test whether the same search-and-train recipe now unlocks even deeper four-layer puzzles more cheaply.
Avoid
- Do not treat the model's earlier weak results as proof it hit a hard capability limit; it was just short on training examples.
- Do not claim a pure data limit, since more examples also meant more training passes in this test; the safe claim is that more distinct found solutions install more skill.
- Do not over-trust the numbers: they come from a single run with no seed-to-seed error bars, and harder puzzles past the tested range are still untested.
◑ PromisingC24
Variety of solved examples keeps teaching, with no plateau
An automated searcher finds verified solutions to small coding puzzles, and training a compact model on them installs the skill. Improvement is driven by the number of DISTINCT solved problems, not repeated passes: rereading 40 examples barely helped (9% to 16%), while adding new ones kept lifting the solve rate with no plateau past 1,150 distinct problems.
Implication. To grow a small model's skill, feed it more distinct verified solutions rather than more training passes over the same set, and expect gains to keep coming well past a thousand examples.
Evidence (1)
Next tests (3)
- Run a data ladder (80, 320, 1,280 examples) on harder four-step tool chains to see if they become reliable with enough distinct examples.
- Repeat the training with at least three random seeds at the key points to get real error bars, since current ranges reflect only test-set noise.
- Push the three-step case well past 1,280 examples with faster training to find where improvement finally stops.
Avoid
- Do not claim a statistically clean gap between adjacent data amounts or for the four-step result; only the broad upward trend and the more-variety comparison are solid.
- Do not credit the gains to extra training passes: rereading the same examples stayed within noise, while adding distinct new problems is what actually helped.
- Do not measure the four-step gain against a from-scratch baseline near zero; the three-step training already carries over to about 7%, so compare against that.
- Do not treat the four-step skill as reliably learned: its single-best-guess solve rate stayed flat near 3% and the result is only marginal.
◑ PromisingC25
Self-training sharpens step-by-step guidance, not real planning
Given a fixed list of 32 operations to build a program, a fresh 4-billion-parameter model ranks its opening move — three steps from the goal — no better than random guessing, though it does well one step out. Retraining it on problems it already solved sharpens these step-by-step rankings, making its search competitive with brute force at low effort — better guessing, not real planning.
Implication. Don't trust a fresh model's early planning moves. Train it on its own solved problems, then measure whether the sharper next-step guidance actually beats brute-force search at your compute budget.
Evidence (2)
Next tests (3)
- Map the full trade-off between search effort and problems solved for the trained guide — does it ever beat brute force, or just tie it?
- Test whether the improvement survives when the model freely proposes operations instead of picking from the fixed 32-item menu.
- Train the model one level deeper and check whether it installs useful guidance one step further from the goal.
Avoid
- Don't call this planning: it's better move-ranking near the goal, not a demonstrated internal search mechanism.
- Don't claim the fresh model is reliably worse than random — it solved one task versus random's two, a gap within noise; say only 'no better than random'.
- Don't credit the built-in distance shortcut for the solves — shortcut-plus-random barely helps; you still need to try every operation.
- Don't oversell training: it makes the guide competitive with brute force at low effort, not better, and only one search width was tested.
- Don't claim improvement on the opening move — training helped only on moves closer to the goal, not the first step.
◑ PromisingC26
Thinking longer sharpens recognition but can't teach planning
Letting an untrained model think longer before answering helps it recognize which single step fits when the situation is already spelled out. But it cannot plan the first of several moves toward a distant goal — that stayed at pure guessing, about 1 in 32, even with 2,048 thinking tokens. Extra thinking amplifies pattern-matching, not forward planning.
Implication. Spend thinking budget on recognition-heavy subproblems where the situation is already laid out. Don't expect longer thinking to close a multi-step planning gap — that needs training on verified solutions, not more compute at answer time.
Evidence (1)
Next tests (3)
- Push thinking budgets much higher (8,000-16,000 tokens) to confirm planning stays flat rather than just being starved of compute.
- Take a model already trained to plan, give it thinking time, and see whether thinking amplifies the planning it now has.
- Re-run with the model freely writing out its answer instead of scoring ranked guesses, to confirm the result holds outside the scoring shortcut.
Avoid
- Don't conclude thinking can never help planning — this model was never trained to reason about this task, so training the thinking process itself remains untested.
- Don't claim longer thinking unlocks planning: naming the first of three moves stayed at chance up to 2,048 thinking tokens.
- Don't count gains on the later steps as planning — the setup hands the model the true intermediate result, so those only measure recognition.
- Don't treat the two levers as interchangeable: training on verified solutions lifts planning; thinking time lifts only recognition.
- Don't compare a freely written answer against ranked-guess scores — that's an unfair channel mismatch; compare think-then-rank against plain rank.
◑ PromisingC28
Train on the plan, not the answer or the after-the-fact story
On fresh three-step problems, a small model trained on short correct plans (break the task into steps, then solve) beat one trained on bare answers — roughly 33% versus 20% solved when allowed multiple tries. But feeding it a wrong plan dropped it below the answer-only model, and training on the model's own rambling successful thoughts helped nothing over answers.
Implication. Train on prompt to explicit step-by-step plan to solution, and let the model think at answer time. Skip its own after-the-fact reasoning — that just narrates an answer it already had.
Evidence (1)
Next tests (2)
- Check whether these plans actually teach the model to pick the right next step in one shot, or only help it cover more solutions by thinking through them.
- Retrain the answer-only model with a matched amount of text and more than one random seed, to rule out that extra training text or luck explains the plan model's edge.
Avoid
- Don't train on the model's own successful reasoning and expect gains — it tied plain answers and lost badly to real plans; those thoughts are after-the-fact justifications, not plans.
- Don't claim the plan-trained model learned to pick the right first step in one shot — it solves more only by thinking things through, which is a different and untested skill.
- Don't credit the plan model's win to it simply thinking longer — a matched model given a wrong plan thought just as long and did worse, so it's the correct plan content that mattered.
- Don't deploy the plan-trained model with thinking turned off — it nearly breaks (about 1% solved); it needs to reason at answer time to work.
◑ PromisingC29
Teaching a model to prefer its right answers wrecks its writing
A small model already rates its own correct answers above its wrong ones about 80% of the time, so it seems like a good judge of itself. But training it to actively favor its correct answers destroyed its ability to generate them: single-best-guess success crashed to near zero. Simply training longer on the correct answers alone worked far better.
Implication. To turn a model's occasional right answers into reliable first-try answers, keep training on those correct answers longer; do not train it to rank its right answers over its wrong ones.
Evidence (1)
Next tests (3)
- Try a gentler version of the prefer-correct training (far fewer steps, a strong pull back toward the original model, stop early on a held-aside score) to see if any careful setting gives a gain instead of a collapse.
- Keep doubling the amount of plain training on correct answers (3, 6, 12, 24 passes) and track whether first-try success keeps climbing toward the model's best-of-many-tries rate.
- Test whether the model's 80% self-judging ability can be used at answer time to pick the correct guess out of several, with no extra training and no collapse.
Avoid
- Don't assume anchoring the prefer-correct training to the original model fixes the collapse: it softens the damage but still ends up worse than plain longer training.
- Don't claim the prefer-correct training helps first-try reliability: both first-try success and best-of-many-tries success crashed, and it never beat simply training longer.
- Don't read the model's 80% self-rating as proof it can pick its own correct answer at answer time: rating two answers side by side is not the same as choosing one while writing, and that remains untested.
- Don't credit the tiny early bump before collapse: it was inside the noise for a single task and vanished immediately.
- Don't compare against an undertrained baseline: doubling the training passes tripled first-try success, so always match training length before judging any method.
◑ PromisingC30
Feed the model its own next step as a hint
A small model often "knows" the first step of a two-step problem in its internal activity even when it can't act on it. Reading that step out and writing it into the prompt as a hint lifted the single-best-guess solve rate on two-step tasks sixfold (3% to 19%) — the first test-time trick to actually raise capability. But only the exact step value helps; naming the step's type alone just widens the guesses.
Implication. Extract the model's own implied next step and inject it as a text hint rather than editing internal activations. Give the concrete value, not just the operation type — the type alone won't fix the top guess.
Evidence (1)
Next tests (3)
- Check whether the exact step value (not just its type) can be read out of the model's internal activity — if it can, the hint could be generated automatically with no extra training; if it can't, that pinpoints what the model never actually computes.
- Test the hint on problems where the exact step is guessable from the input numbers versus where it is hidden, to see when automatic hinting can work.
- Instrument whether the model actually uses the injected step, and repeat across at least two random seeds.
Avoid
- Don't assume reading out just the step TYPE deploys capability — a type-only hint performed no better than no hint at all, because the readout is only right about a third of the time.
- Don't confuse 'more of the guesses are now correct somewhere' with 'the top single guess is correct' — the step type widens the spread of guesses but doesn't fix the best one; you need the exact value for reliable one-shot answers.
- Don't read the hint's benefit as the model just copying the answer from the input: a check on the raw input embedding scored at chance, so the read-out step is genuinely computed internally, and the gains concentrate on the cases where the readout was correct.
- Don't expect this to rescue three-step problems: the internal signal there is too faint, and even a perfect exact-step hint barely moved the needle (about 1%).
◑ PromisingC31
The model computes the operation but just reads off the number
On multi-step tasks, we split what the model knows into two parts: which kind of operation to do, and what number it uses. Reading the model's internal activity names the operation kind far better than the input-output examples alone (41% vs 27%). But the number is fully readable from the examples themselves — a trivial classifier gets it as well as the model does. The operation is genuinely internal; the number is not.
Implication. To supply the missing number, skip the model and use a cheap classifier over the raw inputs and outputs — lengths, sums, min/max, differences. The model holds no special representation of it to extract.
Evidence (1)
Next tests (2)
- Combine the model's read-out of the operation kind with a cheap classifier's read-out of the number into one hint pipeline, and check whether it lifts accuracy as much as being handed both answers directly.
- Recheck other quantities we thought were 'internal to the model' against a plain classifier over inputs and outputs, to see which are truly model-computed versus just readable from the data.
Avoid
- Don't treat the number as hidden model knowledge to extract: a plain classifier over the inputs and outputs reads it as well as the model's internals and actually helps more when used as a hint.
- Don't judge whether something is 'internal to the model' by probing the very first layer — a quirk of the position encoding makes that layer look blank for every task, so it wrongly implies nothing is there. Compare against a real classifier over raw inputs and outputs instead.
- Don't over-read the tiny gain from reading the model's internals: it faithfully reflects what the model decoded, but it's capped because the model only gets the concrete step right about a quarter of the time.
- Don't claim the operation kind is just readable from the examples: the model's internals beat the plain input-output classifier on naming it (41% vs 27%), so that part really is computed inside the model.
◑ PromisingC32
Deep tasks fail on the plan, not the numbers
On three-step chained problems the model almost never proposes the right sequence of operations — under 2% of the time. But hand it the correct sequence and a cheap search over numbers finished every task (100%), while random sequences mostly failed (about 11%). So failures are wrong-plan, not wrong-value; picking which operations in what order is the real wall.
Implication. To clear the wall, search over operation sequences with a tool or enumerator, then cheaply fill in numbers. Don't expect number-side hints or fine-tuning to help — the bottleneck is proposing the plan.
Evidence (1)
Next tests (3)
- Check whether wrong-plan-not-wrong-number holds across other task types, or is specific to this list-manipulation task.
- Sweep the number of chained steps to find exactly where the model's ability to propose the right plan collapses.
- Test whether the 'banking' training method specifically teaches plans: does a banked model propose correct operation sequences more often than the base model?
Avoid
- Don't call this a wrong-number problem: given the right plan the model gets the numbers essentially for free, so its actual failures are wrong-plan.
- Don't over-read the 100% number-fill success as deep — the search is handed the exact set of allowed values, so it only shows numbers are easy once the plan is known.
- Don't measure structural skill by asking the model to write out the operation sequence — formatting alone makes it fail even on one-step tasks; judge by whether its own code behaves like the correct plan.
- Don't treat plan-then-fill as a single forward-pass gain: it's tool-augmented search (generate many, filter), and the win comes from supplying or searching the plan, not from the model.
◑ PromisingC33
Training installs the plan; the base model can't invent it
On a hard task needing a 3-step chained procedure, the untrained 4-billion-parameter model NEVER proposed the right step-sequence (0%). Training it on worked examples lifted that to about 51% on brand-new tasks it had never seen — a real, generalizable skill. What remains is a smaller gap: it picks the right steps but sometimes plugs in wrong values (right plan, about 36% fully correct).
Implication. Spend your training budget teaching the step-by-step plan, not tuning values — the base model's failure is structural. Once the plan is installed, add a cheap pass to fill in correct values, lifting results from ~36% to ~51%.
Evidence (1)
Next tests (3)
- Actually run the two-stage pipeline end-to-end (train the plan, then fill in values) to confirm the ~51% result, which is so far only inferred.
- Check whether a lighter training dose installs proportionally less of the plan.
- Test whether 'training installs the plan' holds across different model families and task types.
Avoid
- Don't credit the gain to memorization: the plan-proposal skill rose on brand-new tasks the model never trained on, so it genuinely generalized.
- Don't expect value-tuning tricks to help the untrained base — it proposes no correct plans at all, so there's nothing to fix on the value side.
- Don't over-trust the size of the leftover value gap or the ~51% end-to-end figure: the sample was small (80 tasks) and the pipeline wasn't run in full.
◑ PromisingC34
With a code interpreter, searching structure beats a trained model
On depth-3 program tasks, letting a machine enumerate every possible step-skeleton, fill in values by running each against known outputs, then pick the answer most guesses agree on solved about 97% of tasks. Using the trained model's own proposed skeleton solved only about 46%, because the model proposes the right shape only half the time.
Implication. When you have an interpreter and the space of possible program shapes is small enough to list exhaustively, enumerate-and-run instead of trusting the model's structure; the model's guess only wins in a single forward pass with no tool.
Evidence (1)
Next tests (3)
- Grow the space of possible program shapes past what you can list exhaustively (more operations, deeper programs) and check whether the model's shape-guessing then beats blind enumeration at equal compute.
- Measure the gap with no interpreter at all, one forward pass only: the trained model solves about 48% here where the untrained base solves none.
- Stress the agree-on-the-answer picker with more probe inputs and adversarial parameters to see if it still avoids answers that only look right on the visible examples.
Avoid
- Do not call the trained-model-structure recipe the best real-world approach: it solves only about 46%, capped by how often the model proposes the right shape, and blind enumeration beats it.
- Do not report how many tasks were merely covered by some candidate: enumeration covers nearly all of them by construction. Only the single final pick that must be correct is a meaningful score.
- Do not assume enumeration always wins: it wins here because roughly four thousand possible shapes is small enough to list; for much larger spaces the model's shape-pruning may become necessary and is untested.
- Do not conclude training the model is useless: it installs the right program structure into a single forward pass, which is the asset when no interpreter is available.
◑ PromisingC35
Brute-force search still beats trained models at program synthesis
For a small list-manipulation language, exhaustively trying every program shape and then filling in its values solved about 97% of tasks even as target programs grew larger. Trained models proposed the correct shape far less often — roughly 51% at one size, dropping to 10% at the next. Even one step deeper, brute force stayed cheap: under two minutes on eight CPUs. Whether a trained guide could ever win is still untested.
Implication. Make exhaustive search your default solver for these small-program tasks. Before betting effort on a trained model to guide it, measure where brute force actually becomes too slow or memory-hungry at greater program sizes.
Evidence (2)
Next tests (3)
- Push to the next program size up and find where exhaustive search finally becomes too slow or memory-hungry in real wall time and memory.
- Test whether training can fix the model's collapsing shape-proposal rate on bigger programs — via curriculum or training on deeper examples — or whether it is a genuine ceiling; note the two rates came from different models, so this is not yet a clean comparison.
- Without an interpreter, using only a single model forward pass, measure how often the trained model outputs a usable program shape by program size, versus the base model's zero.
Avoid
- Do not declare a universal law that trained guidance can never win — no trained guide was ever accurate enough to even run the search here.
- Do not dismiss the model's 10% shape rate on bigger programs as a measurement glitch: exhaustive search solves 97% of those same tasks, so they are solvable — the model simply fails to propose the right shape.
- Do not assume the model can cheaply narrow the search: reading out a program's behavior costs a full enumeration anyway, and the model's guess at whether a half-finished program was completable was barely better than a coin flip.
- Do not read the 51%-to-10% drop as caused by program size — the two numbers come from different, unmatched trained models.
◑ PromisingC37
The reasoning wall is about code, not multi-step thinking
This small model hits a wall linking three or more formal, procedural steps. But give it the same multi-step chain as plain English sentences over made-up names, and it follows the trail near-perfectly through four hops (94-100% correct, versus about 4% for a blind guess). So the limit is specific to code-like formats, not a general inability to reason in steps. Plain-English chains only start slipping at five to six hops.
Implication. Present multi-step tasks as ordinary sentences, not as code or dictionary-style structures, to unlock the model's intact step-by-step reasoning. Watch for it degrading past four or five linked hops.
Evidence (1)
Next tests (3)
- Test whether the plain-English advantage also holds when the model must INFER a hidden multi-hop rule from examples, not just follow a chain it was given.
- Rerun the compare-thinking-vs-not test with a bigger thinking budget and cleaner answer extraction, since the current budget cut answers off.
- Pin down what causes the five-to-six-hop slip: is it chain depth itself, or the number of distracting facts the model must hold in mind?
Avoid
- Do not claim this fixes the harder task of inferring an unknown structure from examples; this only tests following a chain the model is handed.
- Do not read the code-format failure as proof the model can't reason about formal steps: the dictionary format makes it echo the input as a code block instead of reasoning, so that result is muddied by presentation.
- Do not read anything into the thinking-mode results: the thinking budget was too small and cut answers off before completion.
- Do not overclaim there is no wall in plain English at any depth: it does degrade at five to six hops; the finding is only that the three-hop wall seen with code formats is absent.
◑ PromisingC38
The model can follow a rule but not discover one
Given a made-up rule stated outright, the model applied it correctly about 86% of the time. Asked to infer that same simple rule from worked examples, it scored 0% in a single pass — below random guessing. Step-by-step thinking helped partway (about 50%) but stayed error-prone. Discovering rules is hard even in plain language, where executing them is easy.
Implication. Hand the model the rule or structure to apply — its reliable strength — rather than asking it to figure out the underlying rule from examples, which it does poorly even with room to reason.
Evidence (1)
Next tests (3)
- Test rule-discovery with step-by-step thinking on harder cases where the model can still reliably apply a known rule, so the discovery limit is measured on its own.
- Check whether many practice examples, or few-shot prompting, can teach the model to discover rules the way they can install a rule it merely executes.
- Find where the model shifts from applying a rule to failing to discover one — e.g. a vaguely stated rule, or one example versus many.
Avoid
- Do not call rule-discovery impossible in plain language: step-by-step thinking lifted it from 0% to about 50%, so it persists as a difficulty, not a hard wall.
- Do not compare discovery against application on harder multi-step versions of this task — applying the rule itself breaks down there, so the two can only be cleanly compared on the simplest one-step case.
- Do not read this as contradicting the finding that the model executes rules well in language; the two are complementary — executing is easy, discovering is hard.
- Only the simplest depth and a single run were tested; the deeper step-by-step discovery curve is still owed.
◑ PromisingC39
Models follow a new rule but can't discover one
Learning from examples surfaces patterns the model already knows rather than inventing new ones. Told an unusual counting order outright, the model applied it almost perfectly (97%). Shown examples of that same order and asked to figure it out, it scored at chance (12%) — and more examples made it worse, not better.
Implication. When you need the model to follow an unusual rule, state the rule explicitly instead of hoping it infers one from examples — and don't add more examples expecting it to crack a genuinely novel pattern.
Evidence (1)
Next tests (3)
- Gradually scramble the counting order in small steps and check whether the model's ability to work it out from examples fades smoothly as the order drifts further from the normal one.
- Train the model on many unusual orders, then test a brand-new one, to see whether discovering novel patterns can be taught or is a fixed limit.
- Repeat with a different familiar structure, like sorting by a familiar key versus an arbitrary given one, to confirm the follow-versus-discover split isn't specific to counting orders.
Avoid
- Don't claim the model learns nothing from examples — it does work out familiar patterns (45%, well above the 10% chance level). The limit is that this only works for patterns it already knows.
- Don't blame the failure on the rule being hard to apply — when simply told the rule, the model used it 97% correctly. The bottleneck is discovering the rule, not applying it.
- Don't rely on the model's step-by-step thinking mode here — it makes the model write code, which muddies the results; plain prose reasoning is cleaner.
- Don't test this with character-by-character string puzzles — the small model fails at assembling characters regardless, which hides the real signal.
- This is a single run, and even familiar-pattern discovery was imperfect (45%); the real result is the contrast between telling versus showing and familiar versus novel, not the exact scores.
◑ PromisingC40
The model shows its doubt in numbers, not words
When this small model answers, the probability it silently assigns to the digit it writes reveals whether it's right. Within one hard question type, that inner probability sorted right from wrong answers about 95% of the time, far beating surface cues. But when asked out loud to rate itself, it says "100" every time and its yes/no self-checks are no better than a coin flip.
Implication. Use the model's answer-token probability (or its sharpness) as a confidence signal to skip or reroute shaky answers; ignore anything the model says about its own confidence.
Evidence (1)
Next tests (3)
- Check whether answer-token probability works as a skip signal on other task types (like coding problems), or only on this puzzle format.
- On the same task, compare the model judging its own answer versus judging someone else's, to see why its self-check fails while it can verify code well elsewhere.
- Build a deployment rule that skips or reroutes low-confidence answers and attempts high-confidence ones, then measure how much accuracy you gain per answer you're willing to skip.
Avoid
- Don't say the model 'knows itself' outright: the self-knowledge lives only in its output probabilities, never in what it can state.
- Don't treat the fact that average confidence tracks difficulty across question types as real self-knowledge; that's confounded by surface features. The clean result is sorting right from wrong WITHIN one question type.
- Don't ask a small model for a 0-to-100 confidence number: it just answers 100 every time. Read the answer-token probability instead.
- Results are from a single run and the broken self-rating may partly depend on how it was prompted.
◑ PromisingC41
Pick the model's most confident answer, not the most common
Generate several answers, then keep the one the model itself scored as most probable — no code-running or checker needed. That beats majority voting, which stays flat near 48% no matter how many answers you draw, because the model keeps confidently repeating the same wrong rule. Confidence-picking instead climbs from about 47% to 62% as you sample more.
Implication. When you spend compute on multiple samples, choose the answer by the model's own confidence rather than by vote, and skip or escalate problems where even its most confident answer scores low.
Evidence (1)
Next tests (3)
- Test whether reading one yes/no confidence token beats averaging word-by-word probabilities as the confidence signal on real code tasks, then pit the full pick-plus-skip policy against simply sampling more at the same compute.
- Find why steering extra samples toward uncertain problems only tied with spreading samples evenly — look for a cheap early signal, like how varied the first two answers are, that flags which problems more sampling would actually solve.
- Combine the tactics: pick by confidence, skip low-confidence problems, route the unsolvable ones to a tool or bigger model, and chart accuracy against compute cost.
Avoid
- Don't claim that steering more samples toward hard problems helps — here it merely tied with spreading samples evenly; the gain came from picking by confidence and skipping, not from where the budget went.
- Don't select answers by majority vote for this model — accuracy stays flat however many you sample, because the model often confidently repeats the same wrong answer.
- Remember this was shown on one toy task; on real code the signal that transfers is a focused yes/no confidence judgment, not an average of word probabilities.
- Treat this as a single-run, self-reviewed result — the independent design check didn't happen — so confirm it before relying on it.
◑ PromisingC42
A model's confidence dips at its first mistake
When a small model works through a multi-step arithmetic chain, its per-step confidence drops right at the step where it first slipped, not just vaguely late in the chain. Confidence naturally rises deeper in, so you must subtract that trend first; once you do, the dip reliably marks the mistake, letting you redo only from there.
Implication. Read the model's confidence at each step, find the lowest (after removing the natural rising trend), and re-do the chain from that step instead of restarting. This works best when the model slipped only once.
Evidence (1)
Next tests (3)
- Test whether re-doing the located step with some randomness (not a plain retry, which reproduces the same wrong answer) actually fixes chains without peeking at the true answer.
- Reduce messy multi-mistake chains by using shorter/easier chains, or find-and-fix the first error one at a time and re-run until clean.
- Check whether the confidence-dip trick also finds mistakes in non-math step-by-step reasoning, or if it only works for arithmetic.
Avoid
- Do not headline the raw confidence numbers: confidence climbs the deeper the model goes, so you must subtract that per-position trend before the dip actually points at the mistake.
- Do not run this on unfamiliar or scrambled step orders under forced step-by-step output: the model applies one wrong rule everywhere, so every step looks wrong and there is no single mistake to locate.
- Do not claim clean first-mistake location on chains with several mistakes: the dip finds a mistake most of the time but the FIRST one only about a quarter of the time. The clean story holds only for single-mistake chains, which are under half of the error cases.
- A plain retry of the bad step just reproduces the same wrong digit; the reported fixes assumed the right step was found. A real fix needs randomized re-sampling. Results are from one run on familiar-order arithmetic only.
◑ PromisingC43
Fine-tuning nudges rule-discovery but can't fully install it
Show a small model six examples of a hidden rule and ask it to infer the rule and apply it. Untrained, it scores at chance (~9%). Answer-only fine-tuning lifts it to ~40% and is still climbing with data, but stalls well below the ~72% it hits when simply handed the rule, and it learns one specific rule type, not a general knack.
Implication. Don't expect answer-only fine-tuning to install general rule-discovery: it half-works and keeps improving with data, but plateaus below full skill, transfers poorly to new rule types, and erases the model's ability to apply rules it's given.
Evidence (1)
Next tests (4)
- Train on several rule families while holding one type out entirely, so you test whether the model learns general rule-discovery instead of memorizing one rule type.
- Teach the step-by-step reasoning for working out the rule rather than just the final answer: does explaining the procedure install it better and avoid the forgetting? If answers-only fails but reasoning works, the limit is single-pass depth, not missing knowledge.
- Mix in examples of applying already-given rules during training, to check whether learning to infer rules and retaining the ability to apply them can coexist.
- Keep increasing training data (16k, 32k) to see whether rule-inference eventually reaches the ~72% ceiling for applying a known rule or permanently plateaus below it.
Avoid
- Don't claim fine-tuning cleanly installs rule-discovery: it plateaus around 40%, far below the ~72% ceiling for applying a known rule, and only learns one specific rule type (~30% on a new type).
- Don't claim rule-discovery is impossible to install: fine-tuning lifts it from ~9% to ~40% and keeps rising with more data.
- Don't judge poor transfer to a new rule type without first checking the model can even apply that rule when told it (~46% there); part of the shortfall is application difficulty, not failure to transfer the inference skill.
- Don't read the rule-inference gain without noting the cost: answer-only training crashed the ability to apply given rules from ~72% to ~9%. Single seed, and testing one rule type then another is not a real general-inference test.
◑ PromisingC44
Reasoning limits are missing compute, not missing knowledge
Asked to apply a brand-new hidden rule in a single answer, a small model scores at chance — roughly 1 in 100 — even after being trained on that exact rule. Let it write out the steps first and it gets every case right. The knowledge is there; what it lacks is room to compute across tokens.
Implication. Always give the model a scratchpad to work through multi-step rules step by step. Never train it to blurt a one-shot answer, which forces work it cannot do and erases skills it already had.
Evidence (1)
Next tests (3)
- Train on several different hidden rules while holding one out, to see whether writing out reasoning teaches general rule-cracking or just memorizes each taught procedure.
- Give an untrained model a strategy it can actually follow (counting steps, not hard arithmetic) to test whether thinking out loud helps even without being taught the specific rule.
- Chart accuracy against the number of reasoning steps written to find the minimum thinking length below which the model fails.
Avoid
- Do not claim writing out reasoning installs general rule-cracking: it only nails the one specific rule it was taught to execute, and a different rule from the same family transfers at only about 13%.
- Do not read the untrained model's 0% with a strategy hint as 'thinking out loud does not help' — the hint's arithmetic was itself too hard for it; you need an easy-to-follow strategy to test this cleanly.
- Findings are from a single run, and the general-rule test (train on several rules, hold one out) is still owed.
◑ PromisingC45
Teach guess-and-check reasoning and it transfers to unseen rules
A small model was taught a general guess-and-check habit: for a hidden rule, try each candidate, work out the rest from one example, verify on another, keep what fits, and apply it. Trained on several rules and tested on one never shown as the answer, it solved that unseen rule about as well (~91%) as the trained ones — learning the procedure, not just the rules. But only while reasoning out loud.
Implication. To install a missing skill like inferring a rule from examples, teach the general step-by-step strategy across many varied cases and always run it with its reasoning shown — never as a direct one-shot answer.
Evidence (1)
Next tests (3)
- Test a structurally different rule — a two-step or reordering rule — to see if the guess-and-check habit generalizes beyond the single rule shape it was trained on.
- Use a never-trained-on rule whose arithmetic never appeared even as a rejected guess, to prove it learned general inference rather than reusing familiar arithmetic.
- Test whether the same guess-and-check habit carries over to entirely different material, like text strings or lists, instead of just numbers.
Avoid
- Don't claim unlimited general inference: every rule tested has the same shape (multiply then add), and the unseen rule is only a new multiplier within that shape, not a genuinely new kind of rule.
- What generalized is accepting the right answer through verification (the inference logic), not new arithmetic — the never-trained-on rule's math already showed up in training as a rejected guess.
- This works only while the model generates its reasoning out loud; forced to answer in one shot it drops to about 1 in 100. Always run it with reasoning shown and enough output room to finish, since a too-short limit truncates it to zero.
- Results come from a single training run using a small-batch workaround, and the reasoning is long, so allow ample output length for it to complete.
- This habit generalizes only within the same kind of material it was trained on; applied to different material like lists or strings it transfers at essentially zero and even interferes, so don't reuse the trained add-on outside its original domain.
◑ PromisingC46
On real code, asking the model to grade itself beats averaging its confidence
The confidence tricks that worked on toy math carry over to real programming (tested on two standard coding benchmarks), but the winner flips. Averaging the model's word-by-word certainty barely beats random picking. Better: have the model write the code, then answer a strict yes/no "is this correct?" prompt in one shot, and trust that single judgment.
Implication. When no test is available, sample several answers, ask the model to self-grade each with a yes/no prompt, keep the highest-rated, and flag or reroute low-confidence cases. If any runnable test exists, run it first — it beats every confidence signal.
Evidence (2)
Next tests (4)
- Let the model think before it self-grades and see if the better judgment is worth the extra work.
- Compare the full grade-and-reroute policy against simply generating more samples for the same compute cost.
- Grade code step by step to pinpoint the shaky part and rewrite only that section.
- Try confidence-based pruning of guesses in the grammar-learning line once each guess has its own check step.
Avoid
- Do not pick code by averaging word-by-word confidence — it barely beats a plain majority vote among tries. Use the model's yes/no self-grade instead.
- Do not claim confidence beats running a test: one visible test outperforms every confidence signal. Confidence is only for when no test can be run.
- Do not measure self-consistency when there are no visible tests to cluster on — it is undefined there; use the smaller test-backed slice only as a rough ceiling check.
- Score confidence signals problem-by-problem against a code-length baseline, not pooled across all problems, since verbose code and problem difficulty otherwise inflate the numbers.
- Results are still one random seed, eight samples, no deliberation; the coding gains are real but small in absolute terms.
◑ PromisingC47
Confidence ranks answers but can't clean training data
When choosing which of a model's own code answers to train on, keeping only the high-confidence ones did no better than a random grab — even though the confident set was far purer. Only answers checked by actually running the code reliably helped (single-shot accuracy rose from 8% to 24%). Confidence still ranks finished answers well, just can't replace execution.
Implication. Keep running the code to verify answers you train on. Use confidence only to rank finished candidates within a single task, and re-rank every round rather than trusting a fixed cutoff.
Evidence (2)
Next tests (4)
- Collect far more attempts and train on the top-ranked slice — does a larger, purer batch finally beat random selection without running the code?
- Run a real second training round comparing rank-based filtering against a fixed confidence cutoff to see which one quietly decays.
- Try tasks written in plain language instead of code — does cheap no-reasoning confidence become a useful filter there?
- Give the judge more thinking room — does that improve its ranking of the hardest answers, which currently get cut off before it finishes?
Avoid
- Don't judge a filter by its accuracy averaged across all tasks — that hid that it was no better than a coin flip within any single task; measure within-task and control for difficulty.
- Don't set training quotas by counting candidates — wrong answers explode at harder levels; weight by how confident the judge is instead.
- Don't reuse a fixed confidence cutoff across training rounds — scores drift upward on the model's own output while the ranking stays honest; re-rank every round.
- Don't read a drop in no-reasoning accuracy as lost ability — the model just shifted its default outputs onto the trained topic; check its reasoning-mode performance before concluding damage.
- Don't assume that ranking finished programs works on half-finished ones — that was tested separately and came out at chance (about 51% correct).
- Don't trust the default thinking budget for a new judging job — here almost all judgments were forced to stop before finishing.
◑ PromisingC48
Teaching a search routine helps only at the depth you drilled
Fine-tuning a 4-billion-parameter model on ~1,500 worked guess-and-check traces made it far better at the two-step puzzles it practiced (lists rose from 37% to 70% success), but gave no measurable help one step deeper. The skill also failed to carry across problem types, and a routine trained elsewhere actively hurt. Deeper problems may still yield to more thinking budget, which stays untested.
Implication. Train the exact difficulty and problem type you need, and put the reasoning in the model's scratch-thinking channel, not its answer, to avoid erasing skills it already had.
Evidence (2)
Next tests (4)
- Give the model far more room to think (double and quadruple its current thinking budget) on the three-step puzzles and see whether success keeps climbing or flattens out — base success already doubled from 5% to 10% when budget doubled, but answers were still cut off most of the time.
- Train the guess-and-check traces on one problem type only, then test on the others at the same difficulty, to check whether a taught routine stays stuck to its practice material the way a borrowed routine did.
- Rerun the plain instruction-prompt version with cleaner output formatting, since long prompts wrecked the answer format before they could change the actual search behavior.
- Retry judging half-finished programs while showing the model the allowed value ranges and per-example constraints, since judging from the program shape alone currently does no better than a coin flip.
Avoid
- Do not conclude that instruction-prompting fails until you check whether the long prompt simply broke the answer format — formatting collapsed from 89% clean to as low as 44% before search behavior changed at all.
- Do not judge whether a model can propose the right solution shape using only its visible practice cases — require the same proposed shape to also work on fresh unseen inputs, or memorized lookup-table answers will pass by construction.
- Do not reuse a search routine trained on one problem type on a different type — it transferred nothing and actively made performance worse (one two-step task dropped from 37% to 0%).
- Do not set a reproduction check's passing bar from a neighboring result's headline number — use the original experiment's own recorded figure.
- Do not fine-tune the reasoning traces into the answer itself — keeping them in the scratch-thinking channel is what preserved the model's one-step skill (85% held steady) where answer-channel training destroyed it.
- Do not treat the three-step failure at the current thinking budget as a permanent ceiling — success doubled with more budget and answers were still cut off most of the time.
- Do not treat a single fine-tuning run as proof of a general cross-difficulty rule.
◑ PromisingC50
Broad self-training installs general agent skill that transfers
On a blind benchmark a 4B model scored about 14 percent — usually reasoning correctly but hitting its thinking-length limit, restarting its explanation, and never writing a scoreable answer. Training it on its own verified wins across many varied task types, to finish within budget and commit an answer, roughly tripled scores and lifted even task families it never trained on.
Implication. Collect the model's own verified successes across many varied tasks, rewrite each target to a short final answer, and concentrate training on the answer-writing step; where the signal goes beats how much data you add.
Evidence (1)
Next tests (4)
- Confirm the gains hold on the slower, deeper, multi-turn task tiers, not just the quick single-answer ones.
- Push toward harder, longer, multi-step tasks as the next source of gain, since simply repeating the same recipe stops helping.
- Run controlled tests: does fixing the answer-writing step alone drive the gain, and is training across many task types (versus one) what makes unseen tasks improve?
- Recheck earlier findings that gains stay confined to what you trained — were those really about the old recipe rather than the model itself?
Avoid
- Do not just fine-tune the model on copies of its own reasoning and expect behavior to change — that barely differs from doing nothing.
- Do not keep only the runs that finished on their own; that discards the cut-off cases that matter most in real use.
- Do not claim benchmark movement from one serving setup's runs alone; measure with a paired, matched setup on both sides.
- Always check how many training examples were silently dropped for being too long — that quietly rewrote this experiment's own explanation.
◑ PromisingC53
Self-practice installs good habits once, then hits a wall
Teaching a small model a disciplined working style — finish within its time budget, commit answers tersely, act one step at a time — gave a big one-time jump, mostly from a single rich set of worked examples. But more, harder, or more varied practice on the model's own verified answers never pushed past that ceiling; even hand-written expert solutions it couldn't discover itself stayed stuck. Only carefully blending different training data broke the easy-task ceiling past 50 percent, while hard-task scores stayed capped.
Implication. Spend one cheap round installing the working style, then stop scaling that recipe — it won't compound. The next real gain needs a different mechanism: reward-based practice on the exact failures, or distilling solutions the model can't produce on its own.
Evidence (2)
Next tests (4)
- Test a genuinely different mechanism — combining several specialist models — but first confirm each one has real room left to improve so the merge target is actually reachable.
- On the task types that still fail after training, check whether the model can reach the right answer at all given many tries — is the skill even within reach, or truly absent?
- Confirm the hardest-task score ceiling with several random seeds to pin down the true range rather than one noisy number.
- Try teaching multi-step procedures with full worked-out traces, not just single-step examples.
Avoid
- Don't run more rounds of the same self-practice recipe expecting gains to stack — three separate escalations all landed in the same band.
- Don't blame the plateau on the practice tasks being too easy: the model masters even the hardest practice tasks, yet the real-world score still won't move.
- Before merging several specialist models, verify every one has room to hit its improvement target — a skill already sitting near a perfect score cannot gain the required amount, which quietly kills the plan.
◑ PromisingC54
One small model can't master both easy and hard tasks
A 4-billion-parameter model can be pushed well past the target score on quick tasks OR on hard multi-step tasks, but no single trained model clears both at once. Every method tried — more capacity, blending data, averaging model weights, self-generated practice — hit the same wall. The two skill types compete for the model's fixed capacity.
Implication. Don't chase one small model that wins everywhere. Deploy separate specialists (one tuned for quick tasks, one for hard ones) and route each request to the right one, or move to a bigger model.
Evidence (1)
Next tests (4)
- Re-measure any 'we beat the bar on hard tasks' result across at least eight runs — single small samples swing by enough to fake a pass
- Test whether picking the right specialist per task at inference time is the only way one small model can cover both skill types
- Harvest a much larger pool of the model's own long solutions, then compress them, and check if hard-task scores jump further
- Try the 'compress the model's own long solutions into short ones' trick on a genuinely larger model to see if the trade-off is just a small-model limit
Avoid
- Don't blend a quick-task model with a hard-task model hoping the midpoint wins both — the midpoint loses to both ends
- Don't graft the hard-task training onto an already-broad model; it crowds out the breadth. Train the combined skill from the base model instead
- Don't assume hand-written expert solutions unlock hard tasks — only the model's OWN compressed successful attempts moved the needle
- Don't expect self-generated practice to teach a skill the model can't already do; it can only shorten what the model already solves, never add the missing capability
◑ PromisingC55
BUDGET-COMPRESSION LAW: maxing the menagerie think budget (all tiers → 8192, uncapped `huge` tier, max_model_len 65536) reveals the gym-installed advantage was PARTLY compensation for a budget-starved base. A deployment-time compute-response study first confirms the medium wall is SERIAL-COMPUTE (merged absolute medium score rises monotonically 0.337→0.436→0.518 at think budget 1024/2048/4096); then at the new canonical 8192 budget BASE leaps (quick 0.11→0.46, medium 0.13→0.36) and the merged-vs-base DELTA compresses from +0.33/+0.31 to +0.21/+0.15. The install still yields the best ABSOLUTE capability yet (merged 0.666 quick / 0.506 medium) but its MARGINAL value over a fairly-resourced base is ~+0.15–+0.21, not +0.32.
Experiment qwen35_4b_gauntlet_frontier, budget-response phase. Two moves. (1) COMPUTE-RESPONSE STUDY (bench.py --think-budget, paired base-vs-merged on medium items at escalating budgets, fresh seeds): the merged medium ABSOLUTE score rises monotonically with the deployed think budget — 0.337 (n=2) @1024, 0.436 (n=6) @2048, 0.518 (n=2) @4096 — directly confirming AT DEPLOYMENT (not merely inferred from training, as in C54) that the medium wall is a C44 serial-compute limit: the procedure is in the weights and more tokens execute more of it. The merged-minus-base DELTA, however, only rises +0.269→+0.309→+0.301 and PLATEAUS ~+0.31 because base also converts budget into gains. (2) BENCHMARK REDEFINITION (owner-directed, one-off; applied via a context-shielded subagent to keep menagerie internals firewalled): all named tiers → think_budget 8192, a fully-uncapped `huge` tier (65536 == max_model_len), and max_model_len 16384→65536 (verified to init on the RTX 4090 at ~22.5 GB, no OOM); tiers stay ordered by coverage/wall-clock (quick<medium<slow<deep<huge). At the NEW canonical 8192 budget (paired base-vs-merged, n=2/tier, tight): quick base 0.455 / merged 0.666 / delta +0.211; medium base 0.360 / merged 0.506 / delta +0.146. Base leapt from the old budget-starved ~0.11/~0.13, so the delta compressed to ~10× its own n=2 spread below the old +0.31–+0.33. The gym install was thus partly compensating for base not being allowed to think. EPISTEMIC CORRECTION (2026-07-13, per repo owner): 'proven capacity boundary' OVERCLAIMS. The evidence is that ~a dozen recipes (breadth, compression, oracle-injection, episode-mastery, exploration) did not clear +0.32 -- NOT that no training-data sequence can. The true limiter was ITERATION SPEED (~1-2h/recipe), so the data-design space was barely sampled. Reframe: +0.32-on-both is beyond the frontier of the recipes TRIED, and the open problem is to search the data-design space fast enough to find a curriculum that installs the representations + access path. See [[fast-data-design-iteration]].
Implication. The +0.32 quick-AND-medium conjunction was defined against a budget-STARVED base and is the wrong target: give base fair serial compute and it recovers most of the gap. Report ABSOLUTE capability (merged 0.666 quick / 0.506 medium at 8192 — the best measured), not delta-over-a-crippled-base. To beat a fairly-resourced base by a LARGE margin, install what base CANNOT do even with 8192 tokens — the induction / hypothesize-verify walls (C43/C44/C48) — not efficiency or procedure knowledge the base rediscovers once it can think. Old baselines (0.112/0.146/0.138) are superseded; re-baseline at 8192.
Evidence (1)
Next tests (4)
- Tighten quick@8192 and medium@8192 to n>=6 (currently n=2; sd tight ~0.01-0.02 but thin).
- Regenerate full baselines at 8192 (seed 31337) for slow/deep; produce a first `huge`-tier baseline (uncapped, ~18 h wall).
- Does an install TARGETING the induction/hypothesize-verify walls (what base cannot do even at 8192) retain a large delta at maxed budget, unlike the efficiency/procedure install that compresses?
- Sweep merged-vs-base delta at budget 8192->16384->uncapped(`huge`): does it compress to ~0 (base fully catches up) or stabilize at a floor?
Avoid
- Do not compare any post-2026-07-12 menagerie run against the OLD baselines (0.112/0.146/0.138) — they were measured at budget-starved settings and are superseded; re-baseline at 8192.
- Do not read the gym install's old-budget +0.32 as its true value — only ~+0.15–+0.21 survives a fairly-resourced base; the rest was budget-starvation compensation.
- Do not chase quick-AND-medium >+0.32 at maxed budget: base now scores ~0.36–0.46, so a +0.32 medium delta needs merged ~0.68–0.78 — beyond the 4B execution frontier (C44).
- Do not read a shrinking delta as a failed install: the merged ABSOLUTE score is the best measured (0.666/0.506); the delta shrinks because BASE improved, not because merged regressed.
◑ PromisingC56
AXIS-STRUCTURED INSTALL COMPRESSION: at the maxed 8192 menagerie budget the two weakest axes DISSOCIATE — EXPLORATION is installable and transfers (gym burrowmaze mean +0.167 at 8192, L6 0.33->0.67; menagerie medium retain-delta +0.190 > the efficiency install's +0.146) while composed-rule INDUCTION is NOT (gym glyphgate L4-L6 stay ~0.0 before and after; trace-SFT even DEGRADES the easy induction the base could already do, L2 0.93->0.53). No single-4B install flavor clears the +0.32 conjunction at fair budget; decomposed by axis, the residual IS the executor-vs-inducer wall (C39/C44/C48), a serial-compute property of the fixed model, not a data or method gap. Answers C55's open next-test.
Experiment qwen35_4b_gauntlet_frontier, induction/exploration phase (the goal's untried weak-axis prescription, greenlit after C55). MAXED-BUDGET DIAGNOSTIC (gym glyphgate, active induction, greedy@1, tb=8192): base does single-rule induction (L1-L3 1.00/0.93/0.80) but is at a hard 0.0 floor on composed-rule induction (L4-L6), and the broad apex install HURTS induction (L2 0.93->0.47). FOCUSED INSTALL (data/sft_induction.jsonl: 860 glyphgate+burrowmaze oracle hypothesize-verify traces weighted to L4-L6 + 900 broad replay; co-trained from base, emission-seam recipe, adapter induction1). GYM GATE at 8192: glyphgate MEAN -0.056 (L2 0.93->0.53, L4-L6 ~0.0->~0.0) — composed induction NOT installable, trace-SFT trains at 1.0 but does not deploy and degrades easy induction; burrowmaze MEAN +0.167 (L3 0.87->1.00, L4 0.73->1.00, L5 0.67->0.93, L6 0.33->0.67) — exploration IS installable with durable lifts at every hard level, base unsaturated. MENAGERIE TRANSFER (paired base-vs-induction1, n=2/tier, tb=8192): quick +0.183, medium +0.190 — the exploration gain transfers to the held-out benchmark and beats the efficiency apex install on medium (+0.190 vs +0.146; medium carries the multi-turn episodes), despite the combined install also carrying the net-negative glyphgate traces. CLEAN EXPLORATION-ONLY (burrowmaze + replay, no glyphgate; the combined install's +0.190 was a lower bound dragged by the net-negative induction traces): gym burrowmaze MEAN +0.200 at 8192 (L5 0.67->1.00, L6 0.33->0.80), and the MENAGERIE medium retain-delta rises to +0.261 +- 0.048 (n=6; merged mean 0.600 -- tight at 0.575-0.616 -- vs base mean 0.339), quick +0.199 (n=2, sd ~0.001). The merged medium ABSOLUTE 0.60 is the best measured (base ~0.34). This is the definitive exploration ceiling: the strongest single-4B install, yet still below +0.32 on medium and far below on quick, so the conjunction stays unreachable -- exploration lifts medium (episodes) but cannot lift atoms-only quick, and no single install does both (tier-Pareto C54). EPISTEMIC CORRECTION (2026-07-13, per repo owner): 'proven capacity boundary' OVERCLAIMS. The evidence is that ~a dozen recipes (breadth, compression, oracle-injection, episode-mastery, exploration) did not clear +0.32 -- NOT that no training-data sequence can. The true limiter was ITERATION SPEED (~1-2h/recipe), so the data-design space was barely sampled. Reframe: +0.32-on-both is beyond the frontier of the recipes TRIED, and the open problem is to search the data-design space fast enough to find a curriculum that installs the representations + access path. See [[fast-data-design-iteration]].
Implication. Install-value compression at fair budget (C55) is AXIS-STRUCTURED: executable procedures (exploration) install durably and transfer -- the clean exploration-only install is the strongest single-4B install measured (menagerie medium +0.261 +- 0.048 n=6, merged medium absolute 0.60 vs base 0.34) -- while the non-serial inductive leap (composed induction) is walled and un-installable by trace-SFT (and even hurts). No install flavor clears the +0.32 conjunction at fair budget: medium tops out ~+0.26 (exploration) and quick ~+0.20 (efficiency; exploration cannot lift atoms-only quick), and the two tiers need different levers with no single model doing both (tier-Pareto C54). The gauntlet's positive core result is that EXPLORATION is a genuinely installable, budget-robust capability; its negative core result is that the +0.32-on-both target was a budget-starvation artifact (C55) whose residual is the serial-compute induction wall (C39/C44/C48).
Evidence (1)
Next tests (4)
- Do the OTHER executable-procedure weak axes install like exploration? Repeat the isolate-and-measure recipe for program repair (loomfix/patchwheel) and constrained optimization (packhouse/stallwright) at 8192.
- Skin-transfer probe: burrowmaze is SKINNABLE -- does the exploration lift survive fresh pseudo-vocab (procedure, not surface)?
- Tier-router deployment: exploration-merged for medium/episode workloads + efficiency-merged for quick/atom workloads -- the only remaining path to strong deltas on BOTH tiers, since no single model does both.
- Push the exploration install harder (more burrowmaze hard-level traces, expert-iteration on burrowmaze successes) to test whether medium can be pushed decisively past +0.32 while quick is served by a different model.
Avoid
- Do not try to install composed-rule induction via oracle-trace SFT: it trains at 1.0 but deploys ~0.0 (C44 serial-compute) and DEGRADES the single-rule induction the base already does.
- Do not include glyphgate (induction) traces in an exploration install — they are net-negative on their own axis and drag the combined install.
- Do not expect ANY single-4B install flavor to clear +0.32 at the fair 8192 budget: base is ~0.36-0.46, so it would need merged ~0.68-0.78 (beyond the C44 frontier).
- Do not read the exploration lift as budget-compensation: base was NOT saturated at 8192 on burrowmaze (L6 0.33), so the install adds capability rather than compensating starvation.
◑ PromisingC57
COMPUTE-OPTIMAL CONFIDENCE POLICY = confidence-gated adaptive ALLOCATION (not escalation): on the fixed Qwen3.5-4B, committing the greedy answer when its single-token P(True) is high and sampling+conf-selecting only when it is low reaches full-pool MBPP accuracy (0.762) at ~4.25 avg samples vs 9 for uniform sampling — a ~2x compute saving, strictly beating uniform at 7/9 operating points. COMPUTE-OPTIMAL CONFIDENCE POLICY (corrected, powered-up): on the fixed Qwen3.5-4B for MBPP, single-token P(True) confidence-SELECT is the best verifier-free selector on MODERATE-difficulty MBPP (k=9: 0.762 > majority 0.742 > mean-logprob 0.725; per-cand AUROC 0.77) but only DIFFICULTY-DEPENDENTLY so — it ties majority-vote on easy HumanEval (base pass 0.91, both 0.941), max-P(True) ABSTENTION gives a clean risk-coverage curve (solvability AUROC 0.72), and DEPTH (a higher think budget) modestly beats BREADTH on the overall accuracy-vs-tokens frontier (pure-2048 0.593 > pure-256 0.581). BUT selectively ESCALATING the abstained tail to depth does NOT beat matched-compute breadth: at the powered-up n=400 all four abstain-fraction deltas are +0.004..+0.022 with 95% bootstrap CIs spanning 0 — the initial n=24-60 escalation win (+0.15) was a small-sample artifact, caught by the claim's own pre-registered power-up.
Experiment qwen35_4b_confidence_policy. PART 1 (post-hoc, cached 244-task MBPP pool, HF-judge p_true): confidence-select (argmax p_true) is the best verifier-free selector at every k (k=9 conf 0.762 > majority 0.742 > logprob 0.725 > greedy 0.701; exec-line 0.840, oracle 0.848); conf+abstain risk-coverage clean and monotone (cov 1.00->0.757, 0.68->0.866, 0.43->0.902). PART 2 (the escalation arm; regenerated on vLLM ~10x faster than the first HF pass, n=120 -> powered up to n=400 x k=6 at budgets 256 vs 2048; a vLLM judge readout bug was fixed — the model emits the SPACE-PREFIXED ' A'/' B' after 'Answer: ', ids 357/417, not the bare 32/33). The vLLM p_true is a strong signal (per-cand AUROC 0.756-0.769, solvability AUROC 0.715-0.727, matching the HF pool). At MATCHED token-compute with conf-select: pure-2048 modestly dominates pure-256 on the whole frontier (high-k 0.593 vs 0.581), REPLICATING. But selectively escalating the abstained (bottom-by-max-p_true) tail to budget 2048 does NOT beat matched-compute extra breadth at 256: esc-minus-breadth = +0.022 (20%, CI [-0.044,+0.085]), +0.004 (30%, [-0.045,+0.057]), +0.017 (40%, [-0.024,+0.057]), +0.006 (50%, [-0.028,+0.039]) — every 95% CI includes 0. The earlier n=24-60 (HF) escalation win (hardest-20% 0.458 vs 0.304) did NOT replicate; it was a small-sample artifact. Likely mechanism for the null: MBPP nearly SATURATES the budget (the 4B self-limits to ~108-172 think tokens even at budget 2048), so the serial-compute lever is too weak to differentiate here. GENERALIZATION (HumanEval, cached 68-task pool, same schema): the confidence-SELECT advantage is DIFFICULTY-DEPENDENT and does NOT transfer to easy HumanEval — base pass is 0.91 (67/68 solvable), and there confidence-select TIES majority-vote (k=9 both 0.941; majority is slightly ahead at several k). On MBPP (base pass ~0.53) conf-select clearly beat majority (0.762 vs 0.742). So single-token P(True) selection helps only when the task is hard enough that selection matters; when the model is already ~0.9 accurate, self-consistency catches up. Abstention still works on both (HumanEval risk-coverage clean, though only 1 unsolvable task). P(True) > mean-logprob holds on both. DIFFICULTY CURVE (pooled MBPP+HumanEval, 312 tasks binned by per-task pass rate, conf-select vs majority at k=6): the P(True)-select advantage concentrates on the HARD end and vanishes on the easy end — hard (pass 0.08, n=68) conf 0.156 vs majority 0.111 (+0.045); medium (pass 0.54, n=27) 0.762 vs 0.724 (+0.038); easy (pass 0.97, n=217) 0.992 vs 0.999 (-0.007). So conf-select adds value exactly where abstention already flags the task as hard — a self-consistent policy — and majority-vote is fine when the model is already ~0.97 accurate. ADAPTIVE ALLOCATION (the compute-optimal capstone; post-hoc, MBPP 244-pool): a confidence-GATED policy — commit the greedy answer if its P(True) >= threshold, else sample K=8 and conf-select — dominates uniform-k conf-select on the accuracy-vs-average-compute frontier. It reaches the full-pool ceiling 0.762 at ~4.25 avg samples (uniform needs 9), and strictly beats uniform at 7/9 operating points (e.g. avg 1.79: 0.742 vs 0.720; avg 4.25: 0.762 vs 0.747). Mechanism: spend samples ONLY on low-confidence (hard) tasks, exactly where the difficulty curve shows conf-select helps; high-confidence tasks commit for free. So the deployable 'compute-optimal' answer is confidence-gated BREADTH allocation, not the (null) depth escalation. The adaptive-allocation win GENERALIZES to HumanEval too (7/9 operating points; reaches the 0.941 ceiling at ~5 avg samples vs 9 uniform), so confidence-gated allocation saves compute on both an easy and a moderate benchmark. CROSS-DOMAIN: the adaptive-allocation win also generalizes to toy REASONING (the original C41 pool, 240 records; confidence = P(answer)/C40 rather than the P(True) judge): adaptive beats uniform at 6/8 operating points (mid-range +0.02..0.03). So confidence-gated allocation is domain-general across code (MBPP,HumanEval) and reasoning, with EITHER confidence readout.
Implication. The deployable compute-optimal policy for the fixed 4B is a verifier-free, confidence-GATED ADAPTIVE ALLOCATOR: sample greedily once, read the single-token P(True); if high, commit (1 sample); if low, sample K more and pick argmax P(True); abstain below a floor. This reaches full-pool accuracy at ~half the average compute by spending samples only on the hard tasks the confidence signal flags — the difficulty curve proves that is where selection pays. Escalating those hard tasks to more DEPTH (think budget) instead of more BREADTH is a null (n=400) on budget-saturated MBPP. Read one concentrated logit for selection, abstention, AND allocation.
Evidence (2)
Next tests (3)
- Test the escalation lever on a genuinely budget-BOUND task family (harder multi-step reasoning where the 4B does NOT self-limit at ~100-170 think tokens) — MBPP saturates the budget, which likely explains the null here.
- Menagerie arbitration of the SELECT+ABSTAIN policy: does verifier-free P(True) selection + abstention beat majority-vote at matched compute on the held-out benchmark?
- Depth ladder 256->1024->4096->8192 on the abstained tail with n>=400 + CIs: confirm whether depth ever beats breadth selectively at a budget where the model actually keeps thinking.
Avoid
- Do not claim the select-abstain-ESCALATE policy: the escalation (selective depth on the abstained tail) is a NULL result at n=400 (all 95% CIs span 0) — only SELECT + ABSTAIN and a modest OVERALL depth>breadth survive.
- Do not read the original n=24-60 escalation win (+0.15) as real — it did not replicate at n=400; it was a small-sample artifact.
- Do not use the bare 'A'/'B' token ids (32/33) for a vLLM P(True) judge — the model emits the space-prefixed ' A'/' B' (357/417) after 'Answer: '; reading 32/33 yields a degenerate 0.5.
- Do not generalize the escalation null beyond budget-saturated tasks: MBPP self-limits at ~100-170 think tokens, so it under-stresses the serial-compute lever.
- Do not claim conf-select beats majority-vote in general: it is difficulty-dependent — an MBPP-scale win (base ~0.53) that VANISHES to a tie on easy HumanEval (base 0.91), where self-consistency catches up.
◑ PromisingC58
C42 TARGETED REPAIR resolved: a located single-step slip is a NEAR-MISS (the correct digit is the model's RANK-2 choice 85% of the time, median rank 2) and temperature-resampling makes the fix REACHABLE (surfaces d* 80% at n=8), but NO verifier-free CONFIDENCE selector recovers it (greedy re-try and confidence-select both 0.000 -- the erroneous digit IS the argmax). ISOLATED re-computation of just the flagged step, out of the flawed chain context, partially repairs (0.318). So localization enables repair only WITH a verifier or by CONTEXT-REMOVAL, not by confidence selection. END-TO-END: the deployable verifier-free loop (localize min-confidence step -> isolated-recompute -> rerun) lifts whole-chain accuracy +0.048 (oracle-localization ceiling +0.102); repairing a RANDOM step is catastrophic (-0.271), so localization is essential.
Experiment qwen35_4b_error_localization (repair.py), closing C42's owed next-test 'does deployable targeted repair actually fix chains?'. Familiar +k successor chains, scaffolded per-step digit decode; 129 single-slip chains (one located first_local_error) of 560. At the slip step L (d* = the correct successor of the model's own previous digit): (1) RANK of d* in the model's step-L distribution is median 2, rank<=2 0.85, rank<=3 0.90 -- the slip is a near-miss, the model 'almost' knew it; (2) greedy re-try recovers 0.000 (no-op control, confirming C42's warning) and confidence-select recovers 0.000 (the argmax IS the slip); (3) temperature-resample availability is 0.800 (P>=1 of 8 samples == d*) -- the fix is REACHABLE; (4) ISOLATED re-computation -- re-running just that step as a fresh depth-1 chain with the SAME scaffold -- recovers 0.318 (an abstract re-phrasing gave a spurious 0.000 because the model mis-parses 'k forward' as backward; the scaffold-matched prompt is the valid test). Oracle ceiling 1.000. END-TO-END (repair_e2e.py, 560 chains, baseline final-correct 0.450): the deployable verifier-free loop -- localize the min-confidence step, isolated-recompute it, re-run the chain -- LIFTS whole-chain accuracy to 0.498 (+0.048); repairing a RANDOM step is catastrophic (0.179, -0.271, it breaks correct steps); the oracle (repair the true first error) ceiling is 0.552 (+0.102), so the confidence localizer captures ~half the oracle gain (consistent with C42 localization ~0.56). Localization is ESSENTIAL to the repair. SINGLE-SHOT ONLY (repair_iter.py): iterating the loop HURTS -- baseline 0.450 -> 1 repair 0.498 -> 2 repairs 0.463 -> 3 repairs 0.420 (below baseline). After the first repair the localizer targets correct-but-lower-confidence steps and the imperfect isolated-recompute (0.318) breaks them. Deployable rule: repair ONLY the single lowest-confidence step, once.
Implication. Verifier-free IN-CONTEXT confidence repair of a located error is impossible: the erroneous token is exactly the argmax, so any confidence-max selector reproduces the error (greedy/conf-select 0.000). But the fix is a near-miss (rank-2) and reachable by resampling (0.800), so ANY external verifier recovers it cheaply. The one verifier-free move that partially works is CONTEXT-REMOVAL: re-doing the flagged step in isolation repairs ~1/3 (0.318), decomposing slips into ~1/3 context-induced (flawed-chain momentum) vs ~2/3 systematic per-item confusion. Deployable pairing with C57: abstain/localize the low-confidence step, then isolated-recompute it (verifier-free) or resample+verify it (with any checker). DEPLOYABLE: the confidence signal that LOCATES the error also drives a real verifier-free self-repair (localize->isolated-recompute->rerun, +0.048 end-to-end, ~half the oracle-localization ceiling) -- but only because it repairs the RIGHT step; mislocalized repair is catastrophic (-0.271). This is the end-to-end payoff of the C40/C42/C57 confidence arc on multi-step reasoning.
Evidence (1)
Next tests (3)
- Does isolated re-computation + self-consistency (n>1 isolated samples, majority) push past 0.318, and does combining it with resample-availability (0.80) close more of the gap to the oracle?
- Does the near-miss + context-removal repair hold on non-toy multi-step reasoning (GSM8K-style), or is it specific to successor chains?
- C57 x C58: gate isolated re-computation on the per-step confidence dip (only repair the flagged step) and measure end-to-end chain accuracy vs compute.
Avoid
- Do not use confidence to SELECT the repair at a located slip -- the argmax is the error, so greedy/confidence-select are exact no-ops (0.000).
- Do not phrase the isolated re-computation abstractly ('what is k forward from prev'): the model mis-parses it as backward and scores a spurious 0.000; use the EXACT chain scaffold (a depth-1 chain) so the readout is valid.
- Do not claim in-context targeted repair works verifier-free: the fix is reachable but not selectable without an external check or context-removal.
- Do not ITERATE the repair loop: only the FIRST (single lowest-confidence step) repair helps (+0.048); a 2nd/3rd repair damages correct low-confidence steps via the 32%-accurate recompute and drops accuracy BELOW baseline (iter 3: 0.420 vs 0.450). Single-shot only.
◑ PromisingC59
SERIAL COMPUTE CROSSES THE INDUCTION WALL ONLY VIA REASONING CONTENT, not compute-depth or token-count: on held-out shift induction, base single-pass 0.070, LATENT recurrence (N=8 hidden-state feedback, no tokens) 0.090, content-free FILLER tokens (N=32) 0.060 -- all flat at the wall -- while real chain-of-thought GENERATION is 0.220 (3x). Neither continuous latent looping NOR content-free filler tokens help; only the CONTENT of generated reasoning tokens lifts induction. (Naive input-embedding latent recurrence is in fact mildly DEGRADING on affine, 0.193->0.107.)
Experiment qwen35_4b_meta_induction (latent_recurrence.py), probing the corpus's central TOKEN-vs-COMPUTE question: C44 showed the 4B induces held-out rules at ~chance in ONE forward pass but ~1.0 via chain-of-thought GENERATION, and every serial-compute claim (C44/C45/C54/C55/C56) is argued in TOKEN space. This tests compute WITHOUT tokens: a custom HF forward loop takes the last-layer hidden state at the final ('Answer: ') position, appends it as the next input embedding, and re-runs -- N latent 'thought' passes with no token emitted -- then reads the forced-Answer digit (argmax over the 10 digit tokens). On 150 held-out affine (out-of-family) induction episodes: N=0 (base single pass) 0.193; N=1 0.133/0.087; N=2 0.113/0.133; N=4 0.107/0.140; N=8 0.107/0.107 (raw / input-norm-matched). Adding latent passes does NOT help and mildly degrades. Likely mechanism: the model was never trained to interpret its own hidden states as INPUT embeddings, so the appended 'latent token' is out-of-distribution. DECOMPOSITION (compute_decomp.py): four ways to spend test-time compute before the forced-Answer read, on ONE substrate. HELD-OUT SHIFT (in-family, where base CoT works): forced single-pass 0.070, latent-recurrence(N=8) 0.090, filler(N=8/32) 0.030/0.060, real-CoT 0.220 -- only real reasoning CONTENT crosses the wall (3x), latent and filler are flat. OUT-OF-FAMILY AFFINE (the C39 novel-structure wall, where base cannot induce even via CoT): forced 0.193, latent 0.107, filler(N=32) 0.240, real-CoT 0.020 (768-token reasoning often fails to finish/commit) -- everything is low, filler ~ forced, latent degrades. Unifying read: test-time compute helps induction ONLY through the CONTENT of generated reasoning tokens; adding forward passes (latent) or content-free token positions (filler) does not substitute for it. REPLICATED at n=200 (held-out shift): forced 0.090, latent-recurrence(N=8) 0.060, filler(N=8/32) 0.070/0.095, real-CoT 0.235 -- the CoT-vs-compute-only contrast (~2.6x, +0.14 = ~5 SEM) is solid; latent and filler stay flat at the single-pass baseline. Content-only law confirmed. TWO WALL REGIMES (honest bound, NOT a clean replication): the content-law decomposition is clean only where base CoT WORKS. On the harder AFFINE rules (in-family a=3: forced 0.173, latent 0.040, filler(N32) 0.193, real-CoT 0.000; and out-of-family affine real-CoT 0.020) even CoT fails -- the base cannot induce affine novel structure at all (the deeper C39/C43 wall), so nothing crosses and the decomposition is uninformative. So content (CoT) crosses the SHIFT wall but not the AFFINE wall; giving the 4B CoT helps only up to the C39 induction boundary.
Implication. The serial compute that crosses the C44 induction wall is the CONTENT of generated reasoning tokens -- the intermediate values written out -- NOT compute-depth (latent hidden-state recurrence, flat-to-degrading) and NOT token-COUNT (content-free filler, flat). This sharpens every serial-compute claim (C44/C45/C54/C55/C56) and the 'always give it chain-of-thought' prescription: it is CoT's CONTENT, not merely its length or the extra forward passes, that matters. Free test-time compute from re-feeding hidden states or padding tokens will not cross reasoning walls; you must let the model WRITE the intermediate reasoning.
Evidence (1)
Next tests (4)
- Intermediate-LAYER injection / layer-looping (re-run the last hidden state through the decoder layers again) rather than re-embedding at the input -- a less out-of-distribution compute-depth probe.
- Replicate on the CLEAN C44 single-pass 0.01 substrate (this probe's N=0 was 0.193, an easier forced-read prompt) to confirm the flat-with-N result at the true wall.
- Trained latent recurrence / intermediate-layer looping: can SUPERVISION make continuous compute usable, or is token-mediated content fundamentally required?
- Does the content-only law hold on the deployable substrates (MBPP/gym): is CoT content the lever there too, vs filler/latent?
Avoid
- Do not feed a raw last-layer hidden state back as an INPUT embedding expecting compute-depth gains: it is out-of-distribution (the model never learned to read its own hidden states as input) and accuracy is flat-to-declining with N.
- Do not expect free test-time compute on reasoning walls from latent hidden-state recurrence OR content-free filler tokens: both are flat at the wall; only real generated reasoning content (CoT) crosses it (3x on held-out shift).
◑ PromisingC60
CODING THINK BLOCKS MUST BE HARVESTED, NOT AUTHORED: SFT on synthetic AST-templated <think> traces monotonically CRATERS a near-ceiling coder (HumanEval 147/164->129 at 2k rows, ->121 at 5k rows -- WORSE as train loss drops 5.85->2.70), because the base's 89.6% IS its native reasoning and templated traces are strictly worse. At matched recipe+weight, swapping synthetic->NATIVE (self-sampled, execution-verified) think is +35 HumanEval (113->148) and RETAINS coding (+1); native think survives FULL w=1.0 supervision (148). A 2x2 ablation isolates synthetic-think SUPERVISION as the dominant damage (+17 to +32 HE when removed), #WHY code-comments secondary (+7 HE / +13 MBPP). The -9 MBPP residual is training on easy short-trace problems globally SHORTENING thinking (base hits the 8192 budget on 106/200 MBPP, RFT on 21), hurting long-budget-dependent hard problems.
Experiment qwen35_4b_why_think_scale (cognitive-core coding sub-program). Tests whether teaching the 4B WHY via a dual-channel curriculum -- a synthetic step-by-step <think> derivation (AST-derived, genuine, not comment-concat) PLUS strippable inline #WHY: code comments -- can be scaled to an SFT peak worth an RLVR foundation WITHOUT destroying native thinking. Measured thinking-ON (8192 budget) on the shared fitness harness; base HumanEval 147/164 (89.6%), MBPP 151/200 (75.5%). The scale ladder COLLAPSES: rung 2k (loss 5.85) HE -18/MBPP -15, rung 5k (loss 2.70) HE -26/MBPP -32 -- lower loss = worse coding, no pre-collapse peak. Diagnostic: on 29 regressions vs 3 gains the native think SHORTENS (median 608->503 tok) and answers BALLOON (140->319 tok). A 2x2 ablation at 5k (synthetic-think supervised y/n x #WHY-comments y/n): full 121/119, nowhy 113/124, thinkfree(w_think=0) 138/129, cleanfree(clean code + unsupervised think) 145/142 -- synthetic-think supervision is the dominant damage; clean-code SFT with native think is nearly retention-neutral, so the synthetic PROBLEM distribution is fine, the ANNOTATIONS were toxic. Rejection-sampling confirmation: 3000 disjoint synthetic problems sampled from base (K=2, temp 0.8, thinking-budget 8192), execution-filtered vs asserts (100% solved), trained native think + native clean code -- rft w=0.2 HE 148/MBPP 142, rft w=1.0 HE 148/MBPP 139. Same recipe+weight, synthetic-think (nowhy 113) vs native-think (rft 148) = +35 HumanEval.
Implication. Good think blocks for a near-ceiling reasoner must be HARVESTED from the model itself (execution-verified rejection sampling / STaR), never hand-authored -- authored traces regress native reasoning. Native-trace RFT is a retention-SAFE SFT substrate. Separately, the corrected thinking-on baseline reveals HumanEval at 89.6% (near ceiling): the 'push function-writing up with SFT' goal is largely closed (the 76% was a thinking-off measurement artifact); the real prize is the agentic gap (duet-eval 8/35 = 23%), which needs multi-step BEHAVIOR curricula + RLVR, not more function completion. Points the harvest method at a synthetic execution-verified AGENTIC environment feeding SFT warm-start then RLVR.
Evidence (1)
- 2026-07-18 → Qwen35 4B WHY-Think Scale
When run, runs/measure/rung_<rows>.json records each rung's four pass@1 numbers (base co- measured thinking-on), the paired McNemar deltas, and the rung-vs-base problem deltas; the
Next tests (3)
- Build a synthetic execution-verified multi-step agentic env (mirror duet raw 4-tool schema: read_file/write_file/list_dir/run), harvest base multi-turn traces + hint-rationalization for the failing tail, SFT warm-start, measure transfer to duet-eval gen4.
- Fix the MBPP thinking-shortening residual by harvesting on HARDER problems (longer native traces) and confirm it removes the -9 without re-introducing collapse.
- RLVR with execution reward on the same agentic env from the SFT warm-start.
Avoid
- Hand-authoring <think> traces (AST templates, teacher-free synthetic reasoning) for any near-ceiling capability -- it strictly regresses native reasoning; the more faithfully the model fits it (lower loss), the worse it codes.
- Putting rich WHY annotations as inline code comments -- balloons answers 140->319 tok and dents coding.
- Treating HumanEval/MBPP as SFT headroom -- both are near ceiling thinking-on; measure the agentic target instead.
◑ PromisingC61
SINGLE-GPU AGENTIC RLVR IS FEASIBLE FOR Qwen3.5-4B, BUT NEEDS AN SFT WARM-START FOR SIGNAL: TRL 1.8 GRPO colocate closes the loop on one 24GB card (movable reward 0.028->0.38 in 3 steps -> the vLLM-served policy updates, so C49's runtime-LoRA no-op does NOT bite TRL colocate), with a required recipe (enforce_eager monkeypatch for the hybrid arch's CUDAGraph hang; bf16 model_init load not fp32; vllm_gpu_memory_utilization 0.55 + sleep_mode -> peak ~22.6GB). The full agentic loop (environment_factory CodingEnv, thinking-on, pi-mirroring read/write/list/bash tools, pytest reward) runs end-to-end. But the RAW base yields reward_std=0 (explores 1-2 reads ~120 tok then quits WITHOUT writing) and the 24GB card caps num_generations at ~4 (8 OOMs), so a narrow loop-discipline SFT warm-start is the prerequisite before RLVR learns.
Experiment qwen35_4b_agentic_rlvr_feasibility. Owner /goal (2026-07-19): install real-codebase agentic coding into Qwen3.5-4B via pi-coding-agent + OpenEnv, harvest->SFT->RLVR, proven by transfer. Stack: trl==1.8.0 + openenv==0.4.1 into .venv-vllm (no break to vllm 0.24). M0 gate (C49 test): a push-shorter length reward on 20-step colocate GRPO reached 0.38 within 3 steps vs base 0.028 -> served policy updates -> loop closes. Recipe to fit two 4B copies on 24GB: monkeypatch vllm.LLM enforce_eager=True (TRL has no field; hybrid mamba/attn hangs on torch.compile), model_init_kwargs dtype bfloat16 (bf16=True alone loads fp32 16GB), util 0.55 + sleep_mode (<0.55 -> negative KV cache). Integration: GRPOTrainer(environment_factory=CodingEnv) on toolz stub-a-function tasks, thinking-on, ran 3 steps ~25s/step, tools/failure 0. Blocker: across scaffold/system-prompt/num_gen variants, reward_std=0 (frac_reward_zero_std=1) -> zero gradient; logged completions show the base thinks briefly, does 1-2 read/list calls, stops without writing; num_gen 8 OOMs. pi-coding-agent drives the base well headlessly but records NO logprobs (harvest/eval scaffold, not a GRPO rollout source).
Implication. Single-GPU agentic RLVR is a viable install mechanism for this 4B with the pinned recipe, and C49 does not obstruct it (deploy via colocate, not runtime-LoRA-through-a-separate-server). The next stage is a NARROW SFT warm-start (teach explore->edit->test->iterate loop discipline + commit-from-partial, NOT success-only minimization which deletes recovery per the program graveyard) harvested from pi-coding-agent's own execution-verified completed trajectories, then RLVR from the merged warm-start, proven by transfer to a held-out real-coding split (duet-eval final external read only).
Evidence (1)
Next tests (3)
- Harvest pi-coding-agent completed+passing trajectories on solvable real-repo tasks -> multi-turn tool-calling SFT rows -> warm-start; confirm the warm-started model then produces reward_std>0 in the agentic env.
- RLVR (GRPO) from the merged warm-start; reward = FAIL_TO_PASS AND PASS_TO_PASS, no-network sandbox; kill rule: beat matched-compute sample-more on held-out pass-rate.
- Tune GRPO stability (lr, beta/KL, loss_type grpo vs dr_grpo) - the M0 reward trend oscillated.
Avoid
- Expecting RLVR to learn from the raw base on one 24GB GPU: reward variance is zero (base quits without writing) and num_generations can't be grown past ~4 (OOM) to catch rare successes.
- Running colocate vLLM WITHOUT enforce_eager (hangs on Qwen3.5 hybrid arch) or WITHOUT bf16 model_init (loads fp32 -> OOM).
- Using pi-coding-agent rollouts as GRPO samples: pi records no per-token logprobs; use it for harvest/eval, and TRL-driven CodingEnv (pi-mirroring tools) for RLVR generation.