Qwen3.5-4B: Can SFT Install the Skill of Induction?
The one idea you need
You see a few coded messages with their plain meanings, then must decode a new one — but the cipher is hidden. You can't guess it outright; you try each candidate rule against the examples until one fits. Writing the steps out is what makes it work.
The question
If a small model can't spot the hidden rule behind a few examples, can training teach it that skill — and where does the learned skill actually live?
What we found
Yes, but only when it thinks out loud. The trained model cracks brand-new hidden rules perfectly when it writes out its reasoning, yet collapses to worse-than-guessing — about 1 in 100 — when forced to answer in a single step. The skill lives in the written steps, not the weights, and taught as a general try-and-check method it even transfers to rule types it never trained on.
Why it matters
If your small model must infer structure from examples, let it reason in tokens — don't force one-shot answers or cram the skill into the weights. Teach a general try-and-check procedure, and mix in plain rule-following examples so training doesn't wreck existing skills.
On this page
Results at a glance 4
How to read
Each bar is one family of hidden rules; height is how often the model, reasoning step by step, correctly infers and applies it. The three left bars are families seen in training; the highlighted right bar was unseen entirely. Taller is better; guessing is 1 in 10.
Takeaway → The unseen bar (91%) stands as tall as the trained ones (88-96%) — the model learned a general infer-and-check method, not memorized rules.
Data table
| condition | induction accuracy via reasoning (chance 0.1) |
|---|---|
| a=1 (trained) | 95.5% |
| a=3 (trained) | 93% |
| a=9 (trained) | 87.5% |
| a=7 (HELD-OUT, never trained) | 90.5% |
Numbers from
Technical framing
Is GENERAL induction-via-reasoning installable? YES: a general hypothesize-and-verify procedure transfers to a HELD-OUT rule family (a=7: 0.91, as high as in-family) — The most constructive result of the arc. C44 showed induction is a SERIAL-COMPUTE limit but the shift-CoT was shift-SPECIFIC (out-of-family only 0.13). Here: rule families = affine over positions keyed by multiplier a in {1,3,7,9}; a UNIFORM enumerate-and-verify chain-of-thought (try each candidate a, derive b from one example, verify on another, keep the one that fits, apply). Train on {a=1,3,9}, HOLD OUT a=7. RESULT: held-out a=7 induction = 0.905 -- AS HIGH as the trained families (0.88-0.96). The model GENERALIZES to a rule family it never saw as the answer: it learned the general hypothesize-verify-apply PROCEDURE, not just the specific trained rules. So GENERAL induction-via-reasoning IS installable via SFT. Combined with C44 (reasoning 1.00 vs single-forward-pass 0.01), the picture is: the fixed 4B CAN be taught general induction -- infer a novel rule and apply it -- but ONLY as a serial reasoning procedure living in the chain-of-thought tokens, never compressed into the weights. The induction wall (the arc's central limitation) is a serial-compute limit that a general reasoning procedure overcomes generally. Caveat: all families are affine and a=7's arithmetic was seen as a rejected candidate -- what generalizes is the induction LOGIC (accepting a=7 via verify), within the affine hypothesis class.
How to read
Each bar is one setup's accuracy at inferring a unseen hidden rule. Bars span the untrained model and trained models, forced to answer in one step versus allowed to write out reasoning. The two rightmost share one trained model. Higher is better; guessing is 1 in 10.
Takeaway → That trained model scores near zero (1%) when forced to answer instantly but a full 100% when it reasons — the written steps do all the work, not the weights.
Data table
| condition | held-out shift induction accuracy (execute ceiling 0.72, chance 0.1) |
|---|---|
| base (1 forward pass) | 8.7% |
| base (own reasoning) | 0% |
| base+strategy hint | 0% |
| answer-only SFT (1 pass) | 40% |
| reasoning-SFT (1 pass) | 1% |
| reasoning-SFT (reasoning) | 100% |
Numbers from
Technical framing
Is the induction wall a knowledge or serial-compute limit? SERIAL-COMPUTE: the model induces via reasoning (1.00) but at chance in one forward pass (0.01) — The deepest question of the arc, resolved. C43 showed answer-only SFT only PARTIALLY installs scrambled-order shift induction (0.40) and catastrophically forgets execution. Here the SAME base is trained on plain-words chain-of-thought demonstrating the induction procedure (find positions -> derive shift -> apply). THE DISSOCIATION: the reasoning-SFT model induces held-out shifts PERFECTLY when it can reason step-by-step (generation 1.00) but is at CHANCE when forced to answer in a single forward pass (0.01). The chain-of-thought is ~100% LOAD-BEARING -- induction lives entirely in the serial tokens, not the weights. The model literally cannot do the induction computation in one forward pass, even after training; it can only unroll it. So the forward-pass induction wall is a SERIAL-COMPUTE limit, not a knowledge-storage limit. And unlike answer-only SFT (which crashed execution to 0.09), reasoning-SFT largely preserved it (0.57). Caveat: the CoT hand-codes the shift algorithm, so this is executing a taught serial procedure perfectly -- out-of-family affine stays near chance (0.13), so it did not learn GENERAL induction. Connects C38 (thinking rescues induction to 0.50), C13 (broken mental simulation): the model's core limit is running multi-step computation in a forward pass -- give it serial tokens and it works.
How to read
Horizontal axis is how much answer-only training, from none to four thousand to eight thousand practice rounds. One line infers unseen rules of the trained type, another an untrained type; dashed lines mark the apply-a-stated-rule ceiling and pure guessing. Higher is better.
Takeaway → The trained-type line rises to 40% but flattens far below the 72% ceiling, and the untrained-type line barely lifts — a narrow trick, not a general skill.
Data table
| 0=base, 1=SFT 4k episodes, 2=SFT 8k episodes | IN-FAMILY: shift induction (held-out rules) | OUT-OF-FAMILY: affine induction | shift EXECUTE ceiling (base can apply a stated rule) | chance |
|---|---|---|---|---|
| 0 | 8.7% | 21.3% | 72% | 10% |
| 1 | 35% | 26.7% | — | — |
| 2 | 40% | 29.7% | 72% | 10% |
Numbers from
Technical framing
Can SFT install the skill of induction? Partially: shift induction climbs (data-limited) but plateaus below the execute ceiling; OOF affine barely moves — The mission-core 'lift the wall' test. Each episode = a random SCRAMBLED digit order (stated) + a hidden rule + examples + a query; the model must INFER the rule and apply. Base fails at chance (shift-induce 0.087) though it CAN EXECUTE a stated rule (0.72 -- so the wall is induction, not execution, the review's mandatory gate). Answer-only QLoRA on random shift episodes, tested on HELD-OUT shifts (in-family) and AFFINE rules (out-of-family). SFT PARTIALLY lifts the wall and it is DATA-LIMITED: shift induction climbs 0.087 -> 0.35 (4k) -> 0.40 (8k), ~4.6x chance and still rising -- but PLATEAUS well below the execute ceiling (0.72), so only partly installed. Out-of-family affine barely moves (0.21 -> 0.30) -- the model learned a SHIFT-SPECIFIC procedure, not general induction. So the induction wall is neither a hard architectural bound nor cleanly liftable: trained to induce, the fixed 4B learns a specific procedure, not the general skill -- it remains an executor at heart.
How to read
Four bars in two before-and-after pairs. The left pair: how well the model applies an already-stated rule, before then after answer-only training. The right pair: how well it infers a hidden rule, before then after. Higher is better throughout.
Takeaway → Applying a stated rule collapses from 72% to 9% while rule-inference only climbs from 9% to 40% — the training buys a little induction by destroying execution.
Data table
| condition | accuracy |
|---|---|
| shift EXECUTE (base) | 72% |
| shift EXECUTE (after answer-only SFT) | 9.3% |
| shift INDUCE (base) | 8.7% |
| shift INDUCE (SFT 8k) | 40% |
Numbers from
Technical framing
The cost: answer-only SFT installs partial induction (+0.31) but CATASTROPHICALLY FORGETS execution (0.72 -> 0.09) — Answer-only SFT (training the model to output the answer directly) lifts shift induction from chance (0.087) to 0.40, but at a steep cost: it CATASTROPHICALLY FORGETS execution -- the model's ability to APPLY a stated rule crashes from 0.72 to 0.093. So answer-only SFT trades the model's executor competence for a partial, biased induction heuristic. This is consistent with the whole arc's law (the fixed 4B is an executor, not an inducer): even trained to induce, it does not cleanly acquire the general skill, and the training damages what it was good at. Owed: a reasoning-SFT arm (teach the procedure as plain-words chain-of-thought), mixing execute examples to prevent forgetting, and a multi-family leave-one-out design to test GENERAL (not shift-specific) induction.
In the author’s words from the Overview · “Results”
Shift induce: base 0.087 -> SFT-4k 0.35 -> SFT-8k 0.40 (data-limited) but plateaus below execute ceiling 0.72. Affine (OOF) 0.21 -> 0.30 (shift-specific). Catastrophic forgetting: shift execute 0.72 -> 0.09. See reports/report.md, analysis/meta_induction.png.
Overview
Research Program
- Program:
posttraining_and_adaptation/benchmark_generalization - Question (mission-core, lift the wall): can QLoRA SFT install the general skill of inducing a hidden rule from examples (C38/C39: base can't)?
Setup
- Each episode = a random SCRAMBLED digit order (stated) + hidden rule + 6 examples + query -> infer + apply. Families: shift (train), affine a in {3,7,9} (out-of-family). Answer-only QLoRA r32/a64. GATE: base EXECUTE ceiling per family (induction failure meaningful only if base can execute). Eval = forced
Answer:argmax over the 10 digit tokens.
Run
python scripts/gen_data.py --n-train 8000; python scripts/train_lora.py --train data/train_shift.jsonl --out runs/lora_shift8k --epochs 2; python scripts/eval_induction.py --data data/heldout_shift.jsonl --mode induce --adapter runs/lora_shift8k; python scripts/analyze.py.
Results
Shift induce: base 0.087 -> SFT-4k 0.35 -> SFT-8k 0.40 (data-limited) but plateaus below execute ceiling 0.72. Affine (OOF) 0.21 -> 0.30 (shift-specific). Catastrophic forgetting: shift execute 0.72 -> 0.09. See reports/report.md, analysis/meta_induction.png.
Interpretation
The induction wall is neither a hard architectural bound nor cleanly liftable: partial, procedure-specific install with catastrophic forgetting. Trained to induce, the fixed 4B learns a specific procedure, not the general skill -- an executor at heart.
Reasoning arm (C44): serial-compute vs knowledge limit
scripts/gen_cot.py(CoT traces demonstrating the induction procedure),scripts/train_lora.py(reasoning-SFT),scripts/eval_induction.py(--mode induce_gen/strategy),scripts/analyze_reasoning.py.- Result: reasoning-SFT induces held-out shifts PERFECTLY via generation (1.00) but at CHANCE forced-digit (0.01) -> the CoT is 100% load-bearing -> the forward-pass induction wall is a SERIAL-COMPUTE limit, not a knowledge limit. Execution largely preserved (0.57 vs answer-only 0.09). OOF affine 0.13 (shift-specific). See
reports/report_reasoning.md,analysis/induction_serial_compute.png.
Knowledgebase Update
Artifacts
scripts/episode_gen.py,scripts/gen_data.py,scripts/train_lora.py(answer-only QLoRA),scripts/eval_induction.py(forced-digit induce + execute ceiling),scripts/analyze.pyruns/lora_shift*,runs/eval_*.json,runs/verdict.json,analysis/meta_induction.png,reports/{report,design_review}.md
Report
Rendered from reports/report.md
Motivation
The arc's central law: the fixed 4B is an executor/retriever, not an inducer of novel structure (C38/C39). The mission-core question: can QLoRA SFT install the induction skill so it generalizes to held-out rules — lifting the wall — or is induction fundamentally un-installable?
Method (review-hardened)
Each episode = a random scrambled digit order (stated) + a hidden rule + 6 examples + a query; the model must infer the rule and apply. Base fails this at chance (per C39). Rule families over positions in the order: shift f(order[i])=order[(i+k)%10] (train) and affine order[(a·i+b)%10], a∈{3,7,9} (out-of-family). Random orders/params → held-out episodes are genuinely novel. Answer-only QLoRA (r32/α64). The review's mandatory gate: measure the base execute ceiling (rule stated) per family — an induction failure is only meaningful if the base can execute the rule. Eval = forced Answer: argmax over the 10 digit tokens (fair for base + SFT).
Results
| base | SFT-4k | SFT-8k | execute ceiling | |
|---|---|---|---|---|
| shift induce (in-family) | 0.087 | 0.35 | 0.40 | 0.72 |
| affine induce (out-of-family) | 0.213 | 0.267 | 0.297 | 0.457 |
| shift execute | 0.72 | 0.093 | — | — |
- SFT partially lifts the wall, data-limited: shift induction 0.087 (chance) → 0.35 → 0.40, ~4.6× chance and still rising with data — but it plateaus well below the execute ceiling (0.72), so the skill is only partially installed.
- Weak out-of-family transfer: affine induction barely moves (0.21 → 0.30, far below in-family and its 0.46 ceiling) — the model learned a shift-specific procedure, not general induction.
- Two costs: (1) catastrophic forgetting — answer-only SFT crashed execute from 0.72 → 0.09; (2) a default-fallback digit bias that shrinks with data (37% → 20%).
Conclusion
The induction wall is neither a hard architectural bound (SFT lifts it several-fold, scaling with data) nor cleanly liftable (partial, procedure-specific, forgets execution). This is exactly what the arc predicts: trained to induce, the fixed 4B learns a specific procedure, not the general skill — it remains an executor at heart.
Honest scope
- Single seed; answer-only SFT (a reasoning-SFT arm and a multi-family leave-one-out design for general induction are the owed next steps, per the review — this run tests shift→affine procedure-specificity, a weaker probe).
- Affine OOF is partly execution-limited (0.46 base ceiling), so its ceiling is lower than shift's.
- The '8'-bias caveat: SFT-8k accuracy (0.40) far exceeds the frequency baseline (~0.10); most is real induction.
Artifact Manifest
See reports/artifact_manifest.yaml.
Experiment log 1
Show the running log (1 entry)
Scaffold
Created as a new experiment scaffold.
Figures 3
Data files 23
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
runs/eval_base_execute_heldout_shift.json95 Bruns/eval_base_execute_test_affine.json94 Bruns/eval_base_induce_gen_heldout_shift.json97 Bruns/eval_base_induce_heldout_shift.json218 Bruns/eval_base_induce_test_affine.json217 Bruns/eval_base_strategy_heldout_shift.json95 Bruns/eval_gen_induce_gen_gen_heldfam_a7.json97 Bruns/eval_gen_induce_gen_gen_infam_a1.json95 Bruns/eval_gen_induce_gen_gen_infam_a3.json95 Bruns/eval_gen_induce_gen_gen_infam_a9.json95 Bruns/eval_genfix_induce_gen_gen_heldfam_a7.json102 Bruns/eval_genfix_induce_gen_gen_infam_a1.json100 Bruns/eval_genfix_induce_gen_gen_infam_a3.json99 Bruns/eval_genfix_induce_gen_gen_infam_a9.json100 Bruns/eval_rsft_execute_heldout_shift.json95 Bruns/eval_rsft_induce_gen_heldout_shift.json97 Bruns/eval_rsft_induce_gen_test_affine.json96 Bruns/eval_rsft_induce_heldout_shift.json193 Bruns/eval_sft8k_induce_heldout_shift.json218 Bruns/eval_sft8k_induce_test_affine.json218 Bruns/eval_sft_execute_heldout_shift.json95 Bruns/eval_sft_induce_heldout_shift.json214 Bruns/eval_sft_induce_test_affine.json212 B
Reproduce
Smoke test
python scripts/train_lora.py --train data/train_shift.jsonl --smokeFull run
gen_data.py --n-train 8000; train_lora.py (2 epochs); eval_induction.py (induce+execute, base+adapter); analyze.pyRun steps are documented inside the experiment folder (README and scripts).