Qwen3.5-4B: Is In-Context Learning Retrieval or Induction?
The one idea you need
A cook follows any written recipe flawlessly, but can only reverse-engineer a dish back into a recipe when it is one they already know. Hand them a strange new dish to taste and rebuild, and they fail — the very recipe they would have cooked perfectly.
The question
Can a small model work out a brand-new rule just from examples, or can it only reuse patterns it already picked up during training?
What we found
Only patterns it already has. Told a scrambled counting order outright, the model applied it almost perfectly — 97 percent right. But shown examples of that same order and asked to work it out, it scored 12 percent, no better than the 10 percent you would get by guessing. Examples surface familiar structure; they do not discover genuinely new structure.
Why it matters
Do not expect examples to teach a small model a genuinely new rule — adding more only hurt here. Examples re-surface structure it already carries. When a task needs a novel mapping, state the rule outright instead of demonstrating it.
On this page
Results at a glance 2
How to read
Two bar groups: left, the rule is stated outright; right, it must be figured out from examples. In each, one color is the familiar 0-to-9 order, the other a scrambled new order. Taller is more accurate; 10 percent marks pure guessing.
Takeaway → When the rule is stated both bars stand near the top; when it must be inferred, the new-order bar collapses to guessing level — obeying is easy, discovering is not.
Data table
| condition | FAMILIAR order (0-9) | NOVEL order (scrambled, stated) |
|---|---|---|
| EXECUTE (rule stated) | 100% | 97% |
| INDUCE (few-shot) | 45% | 12% |
Numbers from
Technical framing
Is in-context learning retrieval or induction? The model EXECUTES a novel rule (0.97) but cannot INDUCE it (0.12 = chance) — Capstone for the arc: if the model can't induce a novel rule (C38), what is in-context learning doing? Execution-safe single-value task ('advance k steps in a cyclic order', output one digit). Crux (adversarial review): FAMILIAR structure (natural order 0-9, retrievable) vs NOVEL structure (a STATED random cyclic order) at matched 1-parameter complexity, crossed with EXECUTE (rule given) vs INDUCE (rule inferred from few-shot examples). RESULT: EXECUTION is near-perfect and familiarity-INDEPENDENT (familiar 1.00, novel 0.97 -- the model applies the novel scrambled-order rule almost perfectly WHEN TOLD it). INDUCTION is familiarity-BOUND and collapses (familiar 0.45 vs novel 0.12 = chance 0.10): the model CANNOT induce the novel rule from examples, even though it EXECUTES that exact rule at 0.97. So in-context 'learning' surfaces/RETRIEVES familiar structure; it does not create/INDUCE novel structure. Unifies the arc: the model is an EXECUTOR/RETRIEVER of pretrained structure (C37 execution intact), not an INDUCER of novel structure (C38, C32/C36). ICL is the retrieval half of reasoning, not the induction half. Chance = 1/10.
How to read
Horizontal axis is examples shown (five then eight); vertical axis is how often the model works out the rule, higher is better. One line is the familiar order, the other the scrambled new order; both would climb if examples helped.
Takeaway → Both lines slope downward as examples grow — familiar from 45 to 25 percent, new from 15 to 5 — so the limit is familiarity, not too little data.
Data table
| # few-shot examples | FAMILIAR induction | NOVEL induction |
|---|---|---|
| 5 | 45% | 15% |
| 8 | 25% | 5% |
Numbers from
Technical framing
Novel induction is FAMILIARITY-bound, not DATA-bound: more examples do not rescue it (it gets worse) — Does more data rescue novel-structure induction? No -- for both familiar and novel structure, MORE few-shot examples make induction WORSE (familiar 0.45->0.25, novel 0.15->0.05), sinking toward chance (0.10). So the model's failure to induce a novel rule is bounded by structural FAMILIARITY, not by how many examples it sees -- it is not integrating more examples into a better rule estimate (consistent with retrieval/matching rather than genuine induction). Longer prompts also tax this small model's working memory.
In the author’s words from the Overview · “Results”
EXECUTE: familiar 1.00, novel 0.97 (familiarity-independent). INDUCE: familiar 0.45, novel 0.12 (= chance). More examples do not rescue novel induction (0.15->0.05). The model EXECUTES a novel rule perfectly but cannot INDUCE it -> ICL = retrieval of familiar structure, not induction of novel structure. See reports/report.md, analysis/icl_retrieval_vs_induction.png.
Overview
Research Program
- Program:
benchmark_generalization - Capstone question: if the model can't induce a novel rule (C38), what is in-context learning doing? Retrieval of familiar structure, or induction of novel structure?
Setup
- Execution-safe single-value task: "advance k steps in a cyclic order" (output one digit). Crux (adversarial review): FAMILIAR order (natural 0-9, retrievable) vs NOVEL order (a STATED random cyclic order) at matched 1-param complexity, x EXECUTE (rule stated) vs INDUCE (few-shot). No-think (code-mode-free), chance 1/10.
- The first vehicle (letter ciphers) floored (4B can't apply even a given cipher, 0.20 -- char-assembly limit); pivoted to single-value.
Run
python scripts/eval_succ.py --n 60 (2x2), then python scripts/analyze.py.
Results
EXECUTE: familiar 1.00, novel 0.97 (familiarity-independent). INDUCE: familiar 0.45, novel 0.12 (= chance). More examples do not rescue novel induction (0.15->0.05). The model EXECUTES a novel rule perfectly but cannot INDUCE it -> ICL = retrieval of familiar structure, not induction of novel structure. See reports/report.md, analysis/icl_retrieval_vs_induction.png.
Interpretation
Unifies the arc: executor/retriever of pretrained structure (C37), not inducer of novel structure (C38, C32/C36). ICL is the retrieval half of reasoning, not the induction half.
Knowledgebase Update
- Claim ledger: C39
Artifacts
scripts/succ_family.py(single-value successor substrate),scripts/eval_succ.py,scripts/analyze.pyscripts/ic_family.py(original cipher substrate -- floored, kept for the methodological lesson),scripts/eval_icl.pyruns/succ_crux_nothink.json,runs/moreex.json,runs/verdict.json,analysis/icl_retrieval_vs_induction.png,reports/{report,design_review}.md
Report
Rendered from reports/report.md
Motivation
The arc unified into "executor, not inducer" (C37 execution intact in language; C38 induction hard everywhere). But in-context learning is what LLMs are famous for. If the model genuinely can't induce a novel rule from examples (C38), what is ICL doing? Hypothesis: ICL retrieves a familiar pretrained structure the examples point to, rather than inducing a novel rule.
Method
Execution-safe single-value task: "advance k steps in a cyclic order", output one digit. The review's crux: FAMILIAR structure (natural order 0–9, retrievable) vs NOVEL structure (a stated random cyclic order) at matched 1-parameter complexity ("advance k") — crossed with EXECUTE (rule stated) vs INDUCE (rule must be inferred from few-shot examples). Query is an unseen digit (generalization). No-think (code-mode-free); chance = 1/10.
Substrate note (methodological): the first vehicle (letter-substitution ciphers) FLOORED — the 4B cannot apply even a given cipher (application-only 0.20), a character-assembly limit, not induction. Confirmed the floor was harness/char-manipulation and pivoted to the single-value substrate.
Result (no-think, n=60)
| EXECUTE (rule stated) | INDUCE (few-shot) | |
|---|---|---|
| Familiar order (0–9) | 1.00 | 0.45 |
| Novel order (scrambled) | 0.97 | 0.12 (= chance 0.10) |
- Execution is near-perfect and familiarity-independent (1.00 / 0.97): the model applies the novel scrambled-order rule almost perfectly when told it.
- Induction is familiarity-bound and collapses for the novel order (0.45 → 0.12 = chance): the model cannot induce the novel rule from examples — even though it executes that exact rule at 0.97.
- Not data-limited: more examples make novel induction worse (0.15 → 0.05 with 8 examples), so it is bounded by familiarity, not data.
Conclusion
In-context "learning" surfaces/retrieves familiar structure; it does not create/induce novel structure. This unifies the arc: the model is an executor/retriever of pretrained structure (C37; here 0.97–1.00), not an inducer of novel structure (C38, C32/C36). ICL is the retrieval half of "reasoning," not the induction half. For the mission: "unearthing latent capability" means surfacing structure the model already has — the fixed 4B cannot acquire genuinely novel structure in-context, no matter how many examples.
Honest caveats
- Familiar induction is itself only 0.45 (imperfect retrieval).
- The novel arm's induction requires reasoning through a scrambled order (mechanically harder); but the 0.97 execution control shows that mechanism is not the bottleneck — inducing the rule is.
- Single seed; no-think primary (think-mode triggers code-mode on these tasks — a confound documented in runs).
- The char-cipher floor (0.20 application-only) is recorded as a methodological lesson: char-level string tasks are a poor vehicle for this 4B.
Artifact Manifest
See reports/artifact_manifest.yaml.
Experiment log 1
Show the running log (1 entry)
Scaffold
Created as a new experiment scaffold.
Figures 1
Data files 1
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
Reproduce
Smoke test
python scripts/eval_succ.py --n 20Full run
eval_succ.py --n 60 (no-think 2x2); moreex robustness; analyze.pyRun steps are documented inside the experiment folder (README and scripts).