Qwen3.5-4B: Is the Wall Structure or Values? (skeleton-then-fill)
The one idea you need
Picture a three-step recipe: the ordered list of operations, chop then boil then season, is the structure; the exact amount inside each step is the value. This tested a small model on multi-step list-transforming programs, asking whether failure comes from the order of steps or the numbers inside them.
The question
When a small model fails a three-step programming task, is it choosing the wrong sequence of operations, or the right sequence with the wrong numbers plugged in?
What we found
The steps. Handed the correct sequence of operations, a cheap number-search finished every single task, so the numbers were never the bottleneck. Left alone, the model almost never even lands the right sequence, and crediting any-numbers-right adds nothing: its right-sequence rate exactly equals its solve rate. There is no hidden pool of right-recipe, wrong-number near-misses waiting to be unlocked.
Why it matters
Stop fixing deep multi-step failures with better number hints or value tuning; they will not move the wall. Instead supply the operation sequence from outside, via tools, enumeration, or a library of known structures, then let cheap number-fill finish.
On this page
Results at a glance 2
How to read
Four setups, each shown for two-step and three-step tasks; height is solve rate, taller is better. Left to right: single best attempt, best of eight attempts, best of eight credited if any numbers give the right sequence, and tasks solved when the true sequence is supplied.
Takeaway → The first three bars sit near the floor and barely differ, so any-numbers credit adds nothing, then jump to full height once the correct steps are supplied: steps are the bottleneck.
Data table
| condition | depth 2 | depth 3 |
|---|---|---|
| mono greedy@1 | 3.3% | 0.8% |
| mono cov@8 | 9.2% | 1.7% |
| model STRUCTURE-cov@8 (any param) | 10.8% | 1.7% |
| oracle-skeletonfill (true structure) | 100% | 100% |
Numbers from
Technical framing
Is the compositional wall STRUCTURE or VALUES? It is STRUCTURE (value tax ~0; oracle=1.0) — C31 showed the model computes the op-TYPE (structure) but reads the PARAMETER off surface I/O (values). Sharp untested question about the WALL: is depth-3 FAILURE a structure error (wrong op-sequence) or a value error (right skeleton, wrong params)? Measured format-immune (the model's native Python BEHAVIOR matched against the true op-type skeleton with any params). RESULT: the wall is STRUCTURE, decisively. NO VALUE TAX -- the model's STRUCTURE-coverage (right op-type sequence, any param) 0.017 EQUALS its concrete coverage 0.017 at depth-3 -- failures are wrong-skeleton, not right-skeleton-wrong-param. oracle-skeletonfill=1.0 -- if you KNOW the op-type sequence, cheap value-search always finishes (values trivial given structure). So the model cannot propose which operations in which order; once structure is known, values are free. Unifies C19 (depth-3 first-op is a representational thread), C25 (no step-1 lookahead), C31 (param surface-readable). REFUTED the initial 'wall is values' hypothesis. Op-seq GENERATION is a separate 0.00 format failure (not used for the structure signal).
How to read
Horizontal axis: how many random operation sequences were tried (8, 50, 200), each filled by number-search. Vertical axis: solve rate, higher is better. Upper line is two-step tasks, lower line three-step.
Takeaway → Two-step tasks climb to 60% with enough random tries, but three-step tasks reach only about 11%, versus 100% with the true sequence, so the correct structure is genuinely required.
Data table
| R = number of RANDOM op-type skeletons tried (each param-filled + execute-filtered) | depth 2 random-skeletonfill | depth 3 random-skeletonfill |
|---|---|---|
| 8 | 3.3% | 0% |
| 50 | 22.5% | 4.2% |
| 200 | 60% | 10.8% |
Numbers from
Technical framing
Random structure barely cracks depth-3 (R200=0.11): the DSL is NOT value-fungible -- structure genuinely matters — Value-fungibility control: do RANDOM op-type skeletons + value-search crack the tasks (which would make skeletonfill uninformative about structure)? At depth-3, even 200 random skeletons -> only 0.108 (vs oracle=1.0 with the TRUE skeleton). So it is NOT that any skeleton fills -- the correct STRUCTURE is genuinely required, and finding it is the hard part. Depth-2 is more fungible (R200=0.60) because there are fewer/shorter op-sequences. This confirms the wall is a structure-search problem: the model can't propose the structure, and random structure is far from enough -- which is why tool-enumerated structure seeds (C22) and banking (installs structure) were necessary.
In the author’s words from the Overview · “Results”
The wall is STRUCTURE. Value tax (structure-cov - concrete-cov) = +0.000 at depth-3; oracle-skeletonfill = 1.000 (values trivial given structure); random-skeletonfill low (R200=0.108 at depth-3, not value-fungible). See reports/report.md, analysis/structure_or_values.png.
Overview
Research Program
- Program:
structured_execution_and_compilers/interpretability_and_diagnostics - Question (C31 follow-up): when the model fails depth-3, is it a STRUCTURE error (wrong op-type sequence) or a VALUE error (right skeleton, wrong params)?
Setup (pivoted after review + smoke)
- op-seq GENERATION fails (0.00 even depth-1, format handicap), so: model NATIVE Python coverage as baseline; a format-immune STRUCTURE signal (model program BEHAVIOR matched to the true op-type skeleton with any params); oracle-skeletonfill (true structure + value-search); random-skeletonfill (value-fungibility control). Min-depth-verified true-depth tasks, n=120/depth.
Run
python scripts/skeleton_fill.py --n-per-depth 120 --k 8 --randR 8 50 200 --depths 2 3 then python scripts/analyze.py. (scripts/gen_skeletons.py documents the op-seq generation format failure.)
Results
The wall is STRUCTURE. Value tax (structure-cov - concrete-cov) = +0.000 at depth-3; oracle-skeletonfill = 1.000 (values trivial given structure); random-skeletonfill low (R200=0.108 at depth-3, not value-fungible). See reports/report.md, analysis/structure_or_values.png.
Interpretation
The compositional wall is a STRUCTURE-PROPOSAL problem: the model can't propose which ops in which order; once structure is known, values are free. Unifies C19/C25/C31; explains why tool-structure-seeds (C22) and banking were necessary.
Knowledgebase Update
- Claim ledger: C32
Artifacts
scripts/skeleton_fill.py(monolithic Python + format-immune structure signal + oracle/random skeletonfill),scripts/gen_skeletons.py(op-seq gen format probe),scripts/analyze.pyruns/{skelfill_results,verdict}.json,analysis/structure_or_values.png,reports/{report,design_review}.md
Report
Rendered from reports/report.md
Motivation
C31 showed the model computes the op-TYPE (structure) but reads the PARAMETER off surface I/O (values). That raised a sharp question about the WALL itself, which the entire arc (C13–C31) never tested: when the model fails depth-3, does it fail on the STRUCTURE (wrong op-type sequence) or the VALUES (right skeleton, wrong constants)? If it's a value-binding failure, eliciting the skeleton + cheap value-search would crack depth-3.
Design pivot (after adversarial review + smoke)
The natural design — ask the model to output the op-sequence — failed: op-seq generation solves 0.00 even at depth-1 (the model cannot emit DSL op-sequences; a format handicap the repo had hit before). The review also flagged the headline "skeletonfill ≫ direct ⇒ values" as a false dichotomy (a strong value-search fills many wrong skeletons). So the design pivoted to answer the question cleanly on min-depth-verified true-depth tasks:
- model native Python greedy@1 + cov@k (the baseline; op-seq format not required).
- model STRUCTURE-coverage (format-immune): run each model program, check if its behavior matches the true op-type skeleton with any params (right structure, maybe wrong values).
- oracle-skeletonfill: true op-type skeleton + enumerate params + execute-filter-on-visible + check hidden (ceiling: if you know the structure, does value-search finish?).
- random-skeletonfill @ R: R random op-type skeletons, each param-filled (value-fungibility control).
Result (min-depth-verified, n=120/depth)
| depth | mono greedy@1 | mono cov@8 | model STRUCTURE-cov@8 | value tax | oracle-skelfill | random R8/R50/R200 |
|---|---|---|---|---|---|---|
| 2 | 0.033 | 0.092 | 0.108 | +0.017 | 1.000 | 0.033 / 0.225 / 0.600 |
| 3 | 0.008 | 0.017 | 0.017 | +0.000 | 1.000 | 0.000 / 0.042 / 0.108 |
The depth-3 wall is STRUCTURE, decisively:
- No value tax. The model's STRUCTURE-coverage (right op-type sequence, any param) equals its concrete coverage (depth-3: 0.017 = 0.017; depth-2: +0.017). Its failures are wrong-skeleton, not right-skeleton-wrong-param. There is no hidden pool of "right structure, wrong values" solutions to unlock.
- Values are trivial given structure. oracle-skeletonfill = 1.000 — knowing the op-type sequence, cheap value-search always finishes (consistent with C31: the param is surface-readable).
- The DSL is not value-fungible. Random structure barely works (R200 = 0.108 at depth-3): it's not that "any skeleton fills," so structure genuinely matters.
Implication
The compositional wall is a STRUCTURE-PROPOSAL problem: the model can't propose which operations in which order at depth-3; once the structure is known, values are free (oracle 1.0; C31 surface-readable). This unifies the arc — C19 (depth-3 first-op is a representational "thread"), C25 (no step-1 lookahead), C31 (param surface-readable) all point to the same thing: the model reads/computes VALUES easily but cannot propose deep STRUCTURE. It is exactly why tool-enumerated structure seeds (C22) and banking (which installs structure) were necessary, and why value-side interventions (C31 param-hint, DPO on values) don't move the wall. The deployable recipe is structure-search (tool/enumeration) + cheap value-fill — a tool-augmented search (C12/C17 family), not a forward-pass gain.
Honest scope
- "Structure" = op-type sequence on the list DSL; "values" = the arity-1 parameters. Generalization to other substrates/parameter-types is untested.
- op-seq generation = 0.00 is a separate FORMAT failure (documented), not evidence about structural knowledge — the structure signal uses the model's native Python behavior.
- Depth-2 has a tiny value tax (+0.017, within noise at n=120) — even at depth-2 the wall is mostly structure.
Artifact Manifest
See reports/artifact_manifest.yaml.
Experiment log 1
Show the running log (1 entry)
Scaffold
Created as a new experiment scaffold.
Figures 1
Data files 1
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
Reproduce
Smoke test
python scripts/skeleton_fill.py --n-per-depth 15 --k 4 --randR 4 30 --depths 2 3Full run
python scripts/skeleton_fill.py --n-per-depth 120 --k 8 --randR 8 50 200 --depths 2 3; python scripts/analyze.pyRun steps are documented inside the experiment folder (README and scripts).