Qwen3.5-4B: Does Banking Install STRUCTURE?
The one idea you need
Each task needs a tiny three-step program: a sequence of list operations (the plan) plus exact settings (the amounts). Like a recipe, picking the right order of actions—chop, then boil, then season—is the hard part; once the plan is set, filling amounts is easy.
The question
When a small model can't write a correct three-step program, can training on similar tasks teach it the right sequence of steps—and finally make it useful?
What we found
Yes, then no. Training lifted a 4-billion-parameter model from never proposing the right step-sequence (0%) to getting it right about half the time (51%) on brand-new tasks—a real new skill, not memorized answers. But if you can run code, blindly trying all 4,096 possible step-sequences and keeping what works solves 97.5%, crushing the trained model's 46%. The model becomes unnecessary.
Why it matters
Before paying to train program structure into a small model, count your possible program shapes. If a code interpreter can execute candidates and you can list every shape, brute-force search plus output-voting beats the model outright.
On this page
Results at a glance 2
How to read
Grouped bars compare the untrained base model against the trained model on three success measures: single best guess, correct within eight tries, and right step-sequence with any settings within eight tries. Bar height is solve rate on new tasks; taller is better.
Takeaway → The base is flat zero on all three—it never even orders the steps right. Training rises to 51% on step-sequence but only 36% fully correct, exposing a leftover settings gap.
Data table
| condition | base | banked (banked_1280) |
|---|---|---|
| greedy@1 | 0% | 20% |
| cov@8 | 0% | 36.2% |
| STRUCTURE-cov@8 (right op-sequence, any param) | 0% | 51.2% |
Numbers from
Technical framing
Does banking install STRUCTURE? YES: base structure-cov 0.00 -> banked 0.51 (held-out depth-3) — C32 showed the compositional wall is structure-proposal (base can't propose the depth-3 op-sequence; failures are wrong-skeleton) while values are trivially searchable given structure (oracle-skeletonfill=1.0). C22-24 showed banking crosses depth-3. So banking must install STRUCTURE. Ran C32's format-immune structure-coverage (does the model program's BEHAVIOR match the true op-type skeleton with ANY params?) on base vs banked_1280, held-out depth-3 (disjoint from banked's training). RESULT: base has NO structure (structure-cov 0.000 = concrete-cov 0.000). Banking lifts structure-cov to 0.512 on HELD-OUT tasks -> banking installs GENERALIZABLE op-sequence structure (not memorized). And banking CONVERTS the wall from structure-bound to value-bound: the banked model proposes the right skeleton 51% of the time but nails the full concrete program only 36% -- a VALUE TAX of +0.15 (right-skeleton-wrong-param failures the base never had, because it had no skeletons). Since oracle-skeletonfill=1.0 (C32), value-filling the banked model's skeletons would deploy at ~0.51 vs 0.36 alone. Mechanistic closure of C32: banking = STRUCTURE-installation. Explains why value-side interventions (C31 param-hint, C29 DPO) never moved the BASE wall (the base's problem is structure, not values).
How to read
Three bars show deploy success on new three-step tasks: the trained model's single forward-pass guess, its proposed plan plus a settings-search, and brute-forcing every possible plan plus settings-search then voting on outputs. Taller is better.
Takeaway → The brute-force bar towers at 97.5% versus 46% for the model's own plan and 20% single-pass—searching every plan and checking outputs near-solves the task without the model.
Data table
| condition | deploy rate (held-out depth-3, execution-consensus select) |
|---|---|
| banked greedy@1 (forward pass) | 20% |
| bank-fill (model structure + fill + select) | 46.3% |
| brute-fill (SEARCH structure + fill + select) | 97.5% |
Numbers from
Technical framing
Bank+value-fill DEPLOY: the model's structure (0.46) is DOMINATED by brute-force structure-search (0.975) — End-to-end confirmation of the bank+value-fill recipe, with the decisive brute-force control. The banked model emits Python, so recover its proposed STRUCTURE from behavior (infer which op-type skeleton each of k=8 samples implements, no oracle), value-fill against the true outputs, deploy via execution-consensus (plurality output-vector on 16 fresh probe inputs, leakage-free per C17). RESULT: (1) bank-fill deploys at 0.463 = the banked model's structure-coverage 0.475 -> CONFIRMS C33's inferred ~0.51. (2) BUT brute-force structure enumeration (all 4096 depth-3 skeletons, structure NOT from the model) + value-fill + consensus deploys at 0.975 -- near-solving depth-3 without the model. After the 8-visible filter only ~2 skeletons survive (the DSL isn't value-fungible, C32) and consensus picks the right one 97.5% (visible-overfit 2.7%). (3) Using the model's structure is WORSE than ignoring it: bank-fill is CAPPED at struct-cov 0.475 because the banked model proposes the right skeleton only ~48%, while brute enumeration always contains it. NET: with the interpreter available (free selection per C17), free structure-SEARCH dominates the model at deploy; banking's structure (0.20->0.475, C33) is a FORWARD-PASS asset only. Scope: brute-force wins because the depth-3 structure space (4096) is enumerable; the model's structure-pruning would only matter when the space is too large to brute-force.
In the author’s words from the Overview · “Results”
Banking installs STRUCTURE: base structure-cov 0.000 -> banked 0.512 (held-out, generalizable). Banking converts the wall from structure-bound (base struct=concrete=0) to value-bound (banked struct 0.512 > concrete 0.362, value tax +0.15, fillable). See reports/report.md, analysis/banking_installs_structure.png.
Overview
Research Program
- Program:
structured_execution_and_compilers/posttraining_and_adaptation - Question (C32 follow-up): C32 showed the wall is structure-proposal. C22-24 showed banking crosses depth-3. Does banking install the STRUCTURE the base can't propose?
Setup
- Run C32's format-immune structure-coverage (model program BEHAVIOR matches the true op-type skeleton with any params) on BASE vs BANKED_1280 (C24 adapter), on held-out depth-3 (banked's frozen eval, disjoint from training), min-depth-verified, no-think, n=80.
Run
python scripts/bank_structure.py --tag base and python scripts/bank_structure.py --tag banked --adapter <banked_1280>, then python scripts/analyze.py.
Results
Banking installs STRUCTURE: base structure-cov 0.000 -> banked 0.512 (held-out, generalizable). Banking converts the wall from structure-bound (base struct=concrete=0) to value-bound (banked struct 0.512 > concrete 0.362, value tax +0.15, fillable). See reports/report.md, analysis/banking_installs_structure.png.
Interpretation
Mechanistic closure of C32: banking's entire lever is installing op-sequence structure; the residual value tax is small and fillable (oracle-skelfill=1.0). Unifies C22-24/C31/C32: banking = structure-installation.
Knowledgebase Update
- Claim ledger: C33
Artifacts
scripts/bank_structure.py,scripts/skeleton_fill.py+scripts/gen_skeletons.py(C32 structure signal),scripts/analyze.pydata/eval_frozen_d3.jsonl(banked_1280 held-out eval),runs/bank_{base,banked}.json,runs/verdict.json,analysis/banking_installs_structure.png,reports/report.md- Uses external banked_1280 adapter (C24, scratchpad).
Report
Rendered from reports/report.md
Motivation
C32 established that the compositional wall is a STRUCTURE-proposal problem: the base can't propose the depth-3 op-sequence (failures are wrong-skeleton), while values are trivially searchable once structure is known (oracle-skeletonfill = 1.0). C22–24 showed banking crosses depth-3. So banking must install the STRUCTURE the base lacks. This tests it directly.
Method
Run C32's format-immune structure-coverage (does the model program's behavior match the true op-type skeleton with any params?) on the base vs the banked_1280 model, on held-out depth-3 tasks (banked_1280's frozen eval, disjoint from its training), min-depth-verified. no-think, greedy@1 + cov@8.
Result (held-out depth-3, n=80)
| model | greedy@1 | cov@8 | STRUCTURE-cov@8 | value tax (struct−concrete) |
|---|---|---|---|---|
| base | 0.000 | 0.000 | 0.000 | +0.000 |
| banked_1280 | 0.200 | 0.362 | 0.512 (±0.105) | +0.150 |
- Banking installs STRUCTURE. Base structure-coverage 0.000 → banked 0.512 on held-out depth-3. Banking installs generalizable op-sequence structure (held-out tasks ⇒ not memorized skeletons) — the exact capability C32 showed the base lacks.
- Banking converts the wall from structure-bound to value-bound. The base has no skeletons at all (struct = concrete = 0). The banked model proposes the right skeleton 51% of the time but nails the full concrete program only 36% — a value tax of +0.15 (right-skeleton-wrong-param failures the base never had, because it had no skeletons).
- The value tax is fillable. Since oracle-skeletonfill = 1.0 (C32: values trivially searchable given structure), value-filling the banked model's proposed skeletons would deploy at ~0.512 vs 0.362 for the banked model alone — a concrete recipe: bank installs structure; cheap value-search recovers the value tax.
Implication
Mechanistic confirmation and closure of the C32 loop: the compositional wall is structure-proposal (base can't), and banking's entire lever is installing that structure (0 → 0.51 held-out). Once structure is present, the residual is a small, fillable value gap. This unifies the arc — C22–24 (banking crosses depth-3), C32 (the wall is structure), C31 (values surface-readable/searchable): banking = structure-installation. It also explains why value-side interventions (C31 param-hint, C29 DPO) never moved the base wall — the base's problem is structure, not values; only after banking installs structure does a (fillable) value gap even exist.
Honest scope
- "Structure" = op-type sequence on the list DSL; single banked model (banked_1280, C24) and one held-out set (n=80). The +0.15 value tax has a wide CI at n=80. The bank+fill deploy (~0.512) is inferred from structure-coverage + C32's oracle-skeletonfill = 1.0, not run end-to-end here.
Artifact Manifest
See reports/artifact_manifest.yaml. Uses the external banked_1280 adapter (C24, in scratchpad).
Addendum (phase 2): end-to-end bank + value-fill DEPLOY — and the brute-force control that flips the conclusion
C33 inferred bank+value-fill "would deploy at ~0.51". Phase 2 runs it end-to-end (no oracle) and — following the review — adds the decisive brute-force-all-4096 deploy control. Recipe: the banked model emits Python, so recover its proposed STRUCTURE from behavior (run each of k=8 samples, infer which op-type skeleton it implements), value-fill against the true outputs, and deploy via execution-consensus (plurality output-vector on 16 fresh probe inputs — leakage-free, C17-legal). Coverage is tautological (≥ struct-cov), so DEPLOY is the headline.
Result (held-out depth-3, n=80)
| recipe | deploy | coverage |
|---|---|---|
| banked model alone (greedy@1, forward pass) | 0.200 | (cov@8 0.325) |
| bank-fill (model's structure + value-fill + select) | 0.463 ±0.10 | 0.475 |
| brute-fill (all-4096 structure + value-fill + select) | 0.975 ±0.03 | 1.000 |
- C33 confirmed: bank-fill deploys at 0.463 ≈ the banked model's structure-coverage (0.475) — the value-fill recovers the value tax exactly on the tasks where the model proposes the structure.
But the model's structure is DOMINATED by brute-force search. Brute-force structure enumeration + value-fill
- execution-consensus deploys at 0.975 — near-solving depth-3 without the model. After the 8-visible filter
only ~2 skeletons survive per task (the DSL isn't value-fungible, C32), and consensus picks the right one 97.5% of the time (visible-overfit 2.7%).
- Using the model's structure is WORSE than ignoring it: bank-fill (0.463) is capped at the model's structure-coverage because the banked model proposes the right skeleton only ~48% of the time, while brute enumeration always contains it.
Implication
With the interpreter available (mission-legal, free selection per C17), free structure-SEARCH dominates the model at deploy (0.975 vs 0.46 vs 0.20). Banking's installed structure (C33: 0.20 → 0.475) is a forward-pass asset — it matters only if you must deploy in a single pass with no interpreter; the moment you can search+execute, brute-force structure enumeration + value-fill + consensus near-solves the wall and the model is unnecessary. Scope: brute-force wins because the depth-3 structure space (4096) is enumerable; the model's structure-pruning would only become a deployable lever when the space is too large to brute-force (larger op-inventory or deeper compositions).
Experiment log 1
Show the running log (1 entry)
Scaffold
Created as a new experiment scaffold.
Figures 2
Data files 1
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
Reproduce
Smoke test
python scripts/bank_structure.py --tag base --k 4Full run
bank_structure --tag base; bank_structure --tag banked --adapter <banked_1280>; analyzeRun steps are documented inside the experiment folder (README and scripts).