Qwen3.5-4B: Does the Structure-Proposal Wall Exist in Language?
The one idea you need
Picture a stack of secret decoder rings, each swapping made-up words a fixed way. Handed the right ring, the model decodes flawlessly. But shown only a few finished translations and asked which rings produced them, it must play detective and infer the hidden rule. That detective step is the wall.
The question
Can a small model figure out a hidden rule just by watching a few worked examples, the way it can follow a rule it is handed directly?
What we found
No. Handed the rule outright, the model applies it correctly 86% of the time. Asked to infer that same one-step rule from worked examples, it scores 0% — below even the roughly 6% that pure guessing would earn. Letting it reason step by step lifts inference only to a coin-flip 50%, still short of the 75% it reaches simply executing. It runs rules; it does not discover them.
Why it matters
If your system needs a small model to find the rule behind examples, expect failure — even where it flawlessly applies that rule once stated. State rules explicitly; if it must infer, budget thinking and verify, since inferred rules come out wrong about half the time.
On this page
Results at a glance 2
How to read
Four bars show accuracy on the simplest one-step rule; taller is better. The left two apply a rule the model was handed (first without thinking, then with); the right two infer the rule from examples (without, then with thinking).
Takeaway → The apply bars tower over the infer bars; inference sits flat at zero without thinking and reaches only a coin flip with it — running rules is easy, discovering them is not.
Data table
| condition | accuracy at depth-1 (guess baseline 0.06; C37 simulation = 0.99) |
|---|---|
| application (execute given rule) no-think | 86% |
| application think | 75% |
| INDUCTION (infer rule) no-think | 0% |
| INDUCTION think | 50% |
Numbers from
Technical framing
Does the structure-PROPOSAL wall exist in language? YES -- the model executes a given rule (0.86) but cannot induce one (0.00) — Complement to C37 (the model SIMULATES multi-step reasoning in language perfectly). This tests whether the C32/C36 structure-PROPOSAL wall persists in language: relational-composition INDUCTION (R=4 made-up relations = random bijections over made-up entities, hidden depth-D rule; infer which relations compose from examples + apply to a new query). Critical control (review): an application-only arm (rule GIVEN) -- this multi-relation substrate is harder to EXECUTE than C37's chains, so induction failure is meaningful only where application is easy. RESULT: clean forward-pass dissociation at depth-1 (where application IS easy, 0.86): the model EXECUTES a given rule (0.86) but CANNOT INFER one from examples (0.00 = chance). Induction is at chance no-think at all depths. Thinking only PARTIALLY rescues induction (0.50 at depth-1, budget 4096, no truncation -- reasoning correct but error-prone), still far below application (0.75) and C37's simulation (0.99). So the model is an EXECUTOR, not an INDUCER, in language as in formal domains -- corroborating C32/C36 as a cross-modality law. C37+C38: the compositional wall has TWO components that dissociate by modality -- SIMULATION is modality-dependent (formal walls, language does not), PROPOSAL/INDUCTION is modality-general (hard in both). Caveat: this substrate's application degrades at depth 2+, so induction is cleanly isolable only at depth-1.
How to read
The horizontal axis is rule depth — how many secret rings must be chained; the vertical axis is single-pass accuracy, higher is better. The upper line applies a rule the model was handed; the lower line infers the rule from examples.
Takeaway → The inference line hugs the floor at every depth; applying a given rule starts high then sags as chaining more rings strains the model — inference never leaves the ground.
Data table
| rule depth (relations to compose) | application (execute given rule), no-think | INDUCTION (infer rule), no-think |
|---|---|---|
| 1 | 86% | 0% |
| 2 | 28% | 4% |
| 3 | 46% | 8% |
| 4 | 12% | 2% |
Numbers from
Technical framing
Induction is at CHANCE at all depths in a forward pass; application is easy at depth-1 (degrades deeper on this multi-relation substrate) — No-think forward-pass accuracy by rule depth. INDUCTION (infer the hidden rule from examples) sits at chance (~0.06) at every depth -- the model cannot infer a rule in a single forward pass, even a depth-1 rule (identify which of 4 relations fits the examples). APPLICATION (execute a GIVEN rule) is easy at depth-1 (0.86) but degrades at depth 2+ (0.28-0.46) because chaining 2-3 full-16-entry bijection lookups is hard for the small model without thinking. The clean induction-vs-application dissociation is at depth-1 (0.00 vs 0.86); deeper, the multi-relation application confound prevents clean isolation, but induction is already floored at chance.
In the author’s words from the Overview · “Results”
Clean forward-pass dissociation at depth-1: application (execute given rule) 0.86 vs INDUCTION (infer rule) 0.00. Induction at chance no-think all depths; think only half-rescues (0.50 at d1). The model is an EXECUTOR, not an INDUCER, in language as in formal domains. See reports/report.md, analysis/language_proposal_wall.png.
Overview
Research Program
- Program:
benchmark_generalization/structured_execution_and_compilers - Question (complement to C37): C37 showed the model SIMULATES multi-step reasoning in language. Does the C32/C36 structure-PROPOSAL (rule-INDUCTION) wall also persist in language?
Setup
- Relational-composition INDUCTION: R=4 made-up relations (random bijections over ~16 made-up entities), hidden depth-D rule; give KB + examples + a query start NOT in examples -> infer which relations compose + apply. Min-depth-verified, uniqueness-pinned. Application-only control (rule GIVEN) = the ceiling. no-think + think (budget 4096, truncation-checked).
Run
python scripts/eval_proposal.py --render {ling,app,formal} --depths 1 2 3 4 [--think] then python scripts/analyze.py.
Results
Clean forward-pass dissociation at depth-1: application (execute given rule) 0.86 vs INDUCTION (infer rule) 0.00. Induction at chance no-think all depths; think only half-rescues (0.50 at d1). The model is an EXECUTOR, not an INDUCER, in language as in formal domains. See reports/report.md, analysis/language_proposal_wall.png.
Interpretation
C37+C38: the compositional wall's two components dissociate by modality -- SIMULATION is modality-dependent (formal walls, language does not), PROPOSAL/INDUCTION is modality-general (hard in both).
Knowledgebase Update
- Claim ledger: C38
Artifacts
scripts/reasoning_proposal.py(induction substrate, application-only control),scripts/eval_proposal.py,scripts/analyze.py,scripts/reasoning_family.pyruns/prop_*.json,runs/verdict.json,analysis/language_proposal_wall.png,reports/{report,design_review}.md
Report
Rendered from reports/report.md
Motivation
C37 showed the model SIMULATES multi-step reasoning in language near-perfectly (no depth-3 wall). But that tests C13-style simulation, not the C32/C36 structure-proposal wall. This asks the complementary question: can the model induce a hidden compositional rule from examples in language, or does the proposal wall persist even in its native domain?
Method
Relational-composition induction: R=4 made-up relations (each a random bijection over ~16 made-up entities); a hidden depth-D rule = a fixed sequence of D relations. Give the full relation KB + k examples (start → answer applying the hidden rule) + a query start not among the example starts → the model must infer which relations compose and apply. Min-depth-verified (reject shorter-equivalent rules), uniqueness-pinned (examples uniquely determine the rule), contamination-free. Critical control (review-mandated): an application-only arm (rule GIVEN explicitly) — this multi-relation substrate is harder to execute than C37's chains, so induction failure is meaningful only where application is easy. Renderings linguistic-symbolic (primary); no-think + think (budget 4096, truncation-checked).
Result
| condition | d1 | d2 | d3 | d4 |
|---|---|---|---|---|
| application (execute given rule), no-think | 0.86 | 0.28 | 0.46 | 0.12 |
| application, think | 0.75 | — | — | — |
| INDUCTION (infer rule), no-think | 0.00 | 0.04 | 0.08 | 0.02 |
| INDUCTION, think (budget 4096, no truncation) | 0.50 | — | — | — |
(guess baseline ≈ 0.06; C37 linguistic simulation = 0.99 no-think at depth-3.)
- Clean forward-pass dissociation at depth-1 (where application is easy, so induction is isolable): the model executes a given relational rule (0.86) but cannot infer one from examples (0.00 = chance) in a single forward pass. Induction is at chance no-think at all depths.
- Thinking only partially rescues induction (0.00 → 0.50 at depth-1, budget 4096, verified no truncation — the reasoning is correct but error-prone), still far below application (0.75 with think) and far below C37's linguistic simulation (0.99).
- So the model is an executor, not an inducer, in language as in formal domains — corroborating C32/C36 (value-computer, not structure-proposer) as a cross-modality property.
Implication (C37 + C38 together)
The compositional wall has two components that dissociate by modality:
- Simulation / execution is modality-dependent — the formal wall (depth-3, C13–C36) vanishes in language (C37).
- Proposal / induction is modality-general — hard in both formal (C32/C36) and language (this) — the deeper, more fundamental limit.
The model reasons multi-step in language, but it does not induce rules. The structure-proposal wall is the one part of the whole arc that holds even in the model's native domain.
Honest scope & caveats
- This multi-relation substrate's application itself degrades at depth 2+ (0.28/0.46/0.12 — the small model struggles to chain 2–3 full-bijection lookups no-think), so induction is cleanly isolable only at depth-1; but induction is already at chance there. Deeper induction was not cleanly measured (application confound + very slow think runs).
- Think depths 2–4 not completed (budget-4096 think is ~35 min/condition); d1 think (0.50) is the clean think point. Single seed; n=24–50.
- Formal-dict rendering (code-mode confound, C37) not used as primary; linguistic-symbolic is primary.
Artifact Manifest
See reports/artifact_manifest.yaml.
Experiment log 1
Show the running log (1 entry)
Scaffold
Created as a new experiment scaffold.
Figures 1
Reproduce
Smoke test
python scripts/eval_proposal.py --render app --n-per-depth 12 --depths 1 2 3Full run
eval_proposal.py --render {app,ling} --depths 1 2 3 4 [--think, budget 4096, batch 8]; analyze.pyRun steps are documented inside the experiment folder (README and scripts).