Qwen3.5-4B: Does the Compositional Wall Exist in Language?
The one idea you need
The task is a relay: each made-up name points to the next, and you follow the baton several hops along. Written as plain sentences, this small model runs the relay flawlessly; written as a code-style lookup table, it fumbles the very first handoff.
The question
A small model seems to stall after chaining about three reasoning steps. When the very same follow-the-chain puzzle is written in plain English, does it still stall?
What we found
No. Written as ordinary sentences, the model follows a chain of invented names near-perfectly through four hops (94 to 100 percent correct), far above the roughly 4 percent a blind guess earns. The three-step ceiling is really about code-style formatting: the identical chain shown as a code lookup table drops it to 3 percent at a single hop, because the model copies the table instead of tracing it.
Why it matters
If a small-model task needs multi-step reasoning, phrase it as plain-language sentences, not symbol tables or code. Identical logic in code-style formatting can flip the model into echoing code instead of reasoning, collapsing accuracy to near chance.
On this page
Results at a glance 1
How to read
Horizontal axis is chain length, one to six hops; vertical axis is percent correct out of 80 tries, higher is better. Three lines: plain-English sentences, made-up relation words, and the same chain as a code-style lookup table; about 4 percent is chance.
Takeaway → Both language lines sit near the top through three hops while the code-table line starts near zero, showing format, not the number of steps, sets the wall.
Data table
| reasoning depth (number of hops to chain) | linguistic-semantic (no-think) | linguistic-symbolic 'gorps' (no-think, made-up relation control) | formal-dict (no-think) -- triggers code-mode (confounded) |
|---|---|---|---|
| 1 | 99% | 95% | 3% |
| 2 | 100% | 99% | 79% |
| 3 | 99% | 100% | 75% |
| 4 | 94% | 55% | 29% |
| 5 | 76% | 1% | 6% |
| 6 | 78% | 0% | 42% |
Numbers from
Technical framing
Does the compositional wall exist in LANGUAGE? NO -- the model chains depth-3+ reasoning steps in natural language near-perfectly — All 36 prior claims were formal/procedural. This tests C13's mental-SIMULATION wall (NOT the C32/C36 proposal wall) in the model's NATIVE domain. Contamination-free successor-chains over made-up entities, same chain rendered linguistically vs as a Python dict; shortcut-hardened (answer interior/never sink, random start, recency baseline 0.04). No-think is the PRIMARY (forces mental simulation). RESULT: the model does depth-3 multi-step reasoning in natural language NEAR-PERFECTLY (linguistic-semantic d1-d4 = 0.99/1.00/0.99/0.94), and the made-up-relation control ('gorps') is ALSO perfect through depth-3 (1.00) -> genuine MODALITY effect, not a semantic prior. Both degrade only at depth 5-6 (semantic to 0.76; made-up relation collapses to 0.00 -- semantics aids DEEP chaining). STARK contrast to the formal-composition wall (depth-3, C13-C36): the model chains 3-4 reasoning steps in language near-perfectly, so the 'compositional wall' is NOT a general multi-step limit -- it is specific to FORMAL/procedural composition. Relocates C13's 'broken mental simulation' to the formal modality. SECONDARY: the formal-DICT surface form triggers CODE-MODE (the model echoes the dict as a code block instead of simulating, d1=0.03) -- the presentation determines whether the model REASONS or CODES. Scope: tests SIMULATION (C13), not the structure-PROPOSAL wall (C32/C36); think conditions truncation-confounded (budget 1024).
In the author’s words from the Overview · “Results”
NO depth-3 wall in language: linguistic-semantic d1-d4 = 0.99/1.00/0.99/0.94, made-up-relation control perfect through depth-3 (1.00). Stark contrast to the formal-composition wall (depth-3). The wall is formal-modality-specific. Formal-dict triggers code-mode (confounded). See reports/report.md, analysis/language_reasoning_wall.png.
Overview
Research Program
- Program:
benchmark_generalization/structured_execution_and_compilers - Question: all 36 prior claims are formal/procedural. The model is a LANGUAGE model -- does the compositional (mental-SIMULATION, C13) wall exist in its native linguistic domain?
Setup
- Contamination-free successor-chain traversal over made-up pronounceable entities + distractor chains, shuffled. Same chain rendered linguistic-semantic / linguistic-symbolic('gorps', made-up relation control) / formal-dict. no-think PRIMARY (mental simulation), depths 1-6, n=80. Shortcut-hardened (answer interior/never sink, random start, recency baseline 0.04).
Run
python scripts/eval_reasoning.py --render {ling_sem,ling_sym,formal} --n-per-depth 80 [--think] then python scripts/analyze.py.
Results
NO depth-3 wall in language: linguistic-semantic d1-d4 = 0.99/1.00/0.99/0.94, made-up-relation control perfect through depth-3 (1.00). Stark contrast to the formal-composition wall (depth-3). The wall is formal-modality-specific. Formal-dict triggers code-mode (confounded). See reports/report.md, analysis/language_reasoning_wall.png.
Interpretation
The compositional wall is a property of FORMAL composition, not the model's ability to reason multi-step. In language, mental simulation is intact (depth 4-5). Tests SIMULATION (C13), not the C32/C36 proposal wall.
Knowledgebase Update
- Claim ledger: C37
Artifacts
scripts/reasoning_family.py(contamination-free substrate, 3 renderings),scripts/eval_reasoning.py,scripts/analyze.pyruns/reason_*.json,runs/verdict.json,analysis/language_reasoning_wall.png,reports/{report,design_review}.md
Report
Rendered from reports/report.md
Motivation
All 36 prior claims are formal/procedural: the fixed 4B walls at depth-3 formal composition and is a value-computer not a structure-proposer (model-level law, C36). But it is a LANGUAGE model, and its native domain — multi-step linguistic reasoning — was untouched. This tests C13's mental-SIMULATION wall in language (it does not touch the C32/C36 structure-proposal wall).
Method
Contamination-free successor-chain traversal: random chains over made-up pronounceable entities (Kel → Vor → …)
- confusable distractor chains, shuffled. The same chain rendered three ways — linguistic-semantic
("Kel is directly followed by Vor."), linguistic-symbolic ("Kel gorps Vor." — a made-up relation, the contamination-clean control), and formal-dict (a Python nxt map). Query: "moving forward D steps from the start, which name?" Shortcut-hardened (review): the answer is interior (chain longer than D — never the sink), the start is a random interior node (depth ≠ line number), recency baseline ≈ 0.04. No-think is the primary (it forces mental simulation, where the wall should live). Depths 1–6, n=80.
Result (no-think — mental simulation)
| render | d1 | d2 | d3 | d4 | d5 | d6 |
|---|---|---|---|---|---|---|
| linguistic-semantic | 0.99 | 1.00 | 0.99 | 0.94 | 0.76 | 0.78 |
| linguistic-symbolic ("gorps") | 0.95 | 0.99 | 1.00 | 0.55 | 0.01 | 0.00 |
| formal-dict (confounded — code-mode) | 0.03 | 0.79 | 0.75 | 0.29 | 0.06 | 0.42 |
- No depth-3 wall in language. Linguistic-semantic is near-perfect through depth-4 (0.94–1.00); the made-up relation control is also perfect through depth-3 (1.00) — so it is a genuine modality effect, not a semantic pretraining prior. Both degrade only at depth 5–6 (semantic gracefully to 0.76; the made-up relation collapses to 0.00 — semantics aids deep chaining, not shallow).
- Stark contrast to the formal-composition wall (depth-3, C13–C36). The model chains 3–4 reasoning steps in its native linguistic domain, so the "compositional wall" is NOT a general multi-step limit — it is specific to formal/procedural composition. This relocates C13's "broken mental simulation": mental simulation is intact for multi-step linguistic reasoning (depth 4–5), broken only for formal ops at depth-3.
- Secondary (surface-form effect): the formal-DICT rendering triggers CODE-MODE. The model echoes the dict as a
`python block instead of simulating the lookup (d1 = 0.03, the comprehension gate-failure the review predicted). So the surface presentation determines whether the model reasons or codes — the formal-dict arm is confounded and is not a clean "formal simulation capacity" measurement.
Implication
The wall we spent 36 claims mapping is a property of formal composition, not of the model's ability to reason multi-step. In its native language, there is no depth-3 wall — the model's mental simulation is intact for multi-step linguistic reasoning. This is the first result to locate the compositional wall in the modality (formal/procedural), not the capacity (chaining reasoning steps).
Honest scope
- Tests simulation (chain given → traverse), which is C13; it does not test the C32/C36 structure-proposal wall (nothing here asks the model to propose a hidden structure) — the "value-computer not structure-proposer" headline is untouched.
- Think conditions are truncation-confounded (budget 1024 — the model over-reasons trivial tasks and exhausts the budget before answering); no-think is the clean primary.
- The formal-dict arm is confounded by code-mode; the clean finding rests on the two linguistic arms.
Next
- Linguistic PROPOSAL task (infer a hidden multi-hop rule from I/O examples, mirroring C32): does the "value- computer not structure-proposer" law extend to language, or is proposal also easier linguistically?
- Higher think budget / better answer extraction to get a clean think-vs-no-think (transcription vs mental sim) gap.
Artifact Manifest
See reports/artifact_manifest.yaml.
Experiment log 1
Show the running log (1 entry)
Scaffold
Created as a new experiment scaffold.
Figures 1
Reproduce
Smoke test
python scripts/eval_reasoning.py --render ling_sem --n-per-depth 10 --depths 1 2 3 4Full run
for r in ling_sem formal ling_sym: eval_reasoning.py --render $r --n-per-depth 80 [--think for ling_sem/formal]; analyze.pyRun steps are documented inside the experiment folder (README and scripts).