Qwen3.5-4B Context Composition
The one idea you need
Picture an interpreter who studies a new dialect so intensely they start replying in it — even when you asked for plain English. Their understanding genuinely sharpened, but now half their answers come back in a form you cannot read or use.
The question
If you fine-tune a small model to add a skill and then walk it through the exact steps in the prompt, can it reliably use that skill?
What we found
Only when its answer survives. The fine-tuned skill is genuinely the sharpest — 95% correct when the model replies in the required form, beating the untrained model's 83% under the same step-by-step procedure. But the training hijacks the answer format half the time, so real usable accuracy collapses to 51% — below the plain untrained model's 74%.
Why it matters
When you fine-tune a small model to add a skill, measure how often it still answers in your required format, separately from accuracy. A captured output channel can bury a genuinely better skill and make it undeployable — so mix your target format into training.
On this page
Results at a glance 2
How to read
Five setups along the bottom: the untrained model with a plain prompt, a step-by-step procedure, or worked examples, then the fine-tuned model plain and with the procedure. Three bars each — usable accuracy, accuracy among readable answers, and share of replies in the required format. Higher is better everywhere.
Takeaway → The fine-tuned model with the procedure has the tallest readable-answer bar (95%) but the shortest format-survival bar (about half), so its usable-accuracy bar sinks below the untrained model.
Data table
| condition | 2AFC raw (deployable) | 2AFC parse-conditional | parse rate |
|---|---|---|---|
| base plain | 74% | 79% | 94% |
| base orch | 83% | 83% | 100% |
| base ICL | 78% | 81% | 97% |
| SIM plain | 46% | 87% | 53% |
| SIM orch | 51% | 95% | 53% |
Numbers from
Technical framing
Context composes the module (0.95) but format capture crushes deployable 2AFC (0.51) — An explicit simulate-both-compare procedure lifts base 2AFC to 0.83. The weight-installed SIM module reaches 0.95 parse-conditional (it genuinely composes in-context) but format capture (parse 0.53) hijacks the output channel half the time, crushing deployable accuracy to 0.51 — capability = module x interface x procedure.
How to read
Two bars — the untrained model and the fine-tuned model — showing how often each writes the correct program from behavior alone, using a propose-then-test procedure. Higher is better; both sit near the floor of the chart.
Takeaway → Both bars barely clear zero (8% and 13%). No context trick — procedure, worked demos, or a working simulator — lifts writing-the-program-from-scratch; proposing a program from its behavior is the wall nothing crosses.
Data table
| condition | identification (generate-and-test) |
|---|---|
| base | 8% |
| SIM adapter | 13% |
Numbers from
Technical framing
No context strategy moves hypothesis generation (the un-composable wall) — Neither a procedure, worked demos, nor a working simulator moves generation — proposal cannot be assembled in-context from forward primitives.
In the author’s words from the Overview · “Results (see reports/report.md; figure analysis/context_composition.png)”
Identification (all context strategies): base 0.08, SIM 0.13 — unmoved. Context composes discrimination: the explicit procedure lifts base to 0.83, flat through depth 4. The weight-installed module IS accessible in-context and adds capability — SIM+orchestrated hits 0.95 parse-conditional (vs base 0.83, same procedure) — but format capture gates the interface (parse 0.53), crushing deployable accuracy to 0.51. The module composes; the interface is captured. Hypothesis GENERATION is the un-composable wall: no context strategy (procedure, demos, working simulator) moves bare identification (0.08–0.13). Inverse inference cannot be assembled in-context from forward primitives. … Read the full result →
Overview
Research Program
- Program:
structured_execution_and_compilers(+posttraining_and_adaptation). Pre-registered (reports/prereg.md). The third capability-installation mechanism: C12/C13 measured TOOLS (externalization works), C14 measured WEIGHTS (SFT is format-local, doesn't compose). This measures CONTEXT: can explicit orchestration or few-shot demonstration compose what weight-training cannot?
Design
Same 120 verified ladder tasks + same 2AFC decoys/seed as the keystone experiment (exact comparability). Conditions: {base, SIM adapter (regenerated; simulator verified 0.8+ to depth 5)} × {plain-think@1024 (budget control), ORCHESTRATED (explicit simulate-both-compare procedure), ICL (2 worked examples)} for 2AFC; orchestrated generate-and-test for bare identification.
Results (see reports/report.md; figure analysis/context_composition.png)
| 2AFC | raw | parse-rate | parse-conditional |
|---|---|---|---|
| base plain@1024 | 0.74 | 0.94 | 0.79 |
| base ORCHESTRATED | 0.83 | 1.00 | 0.83 |
| base ICL | 0.78 | 0.97 | 0.81 |
| SIM plain@1024 | 0.46 | 0.53 | 0.87 |
| SIM ORCHESTRATED | 0.51 | 0.53 | 0.95 |
Identification (all context strategies): base 0.08, SIM 0.13 — unmoved.
- Context composes discrimination: the explicit procedure lifts base to 0.83, flat through depth 4.
- The weight-installed module IS accessible in-context and adds capability — SIM+orchestrated hits 0.95 parse-conditional (vs base 0.83, same procedure) — but format capture gates the interface (parse 0.53), crushing deployable accuracy to 0.51. The module composes; the interface is captured.
- Hypothesis GENERATION is the un-composable wall: no context strategy (procedure, demos, working simulator) moves bare identification (0.08–0.13). Inverse inference cannot be assembled in-context from forward primitives.
- Retro-correction of P12 (keystone/C13): "thinking-2AFC at chance" was inflated by budget-512 + a weak first-char parser; at budget 1024 with strict answer format, base thinking-2AFC ≈ the no-think logit read (0.74–0.79 vs 0.73–0.78). Thinking doesn't hurt discrimination; it just doesn't beat the surface heuristic without an explicit procedure.
Insight (claim C15): the three installation mechanisms have distinct failure modes — weights install capability but capture the interface; context composes procedures but cannot create generators; tools alone cross the generation wall. Deployable capability = module × interface × procedure.
Report
Rendered from reports/report.md
Summary
The third capability-installation mechanism, pre-registered (reports/prereg.md). C14 showed WEIGHTS install capability format-locally (a fully repaired simulator moved nothing downstream). Here we test CONTEXT: explicit orchestration and few-shot demonstration on the same verified tasks, same decoys, base vs the SIM adapter. Four findings. (1) Context composes discrimination: an explicit simulate-both-compare procedure lifts base 2AFC to 0.83 (parse 1.00), flat through depth 4 — where the plain condition sits at 0.74. (2) The weight-installed module IS accessible in-context: under the identical procedure the SIM adapter reaches 0.95 parse-conditional (+12pp over base) — the trained simulator genuinely composes — but format capture gates the interface (parse rate 0.53: half its generations answer in trained format instead of Answer: X), crushing deployable accuracy to 0.51. (3) Hypothesis generation is the un-composable wall: no context strategy moves bare identification (base 0.08, SIM 0.13) — procedure, demonstrations, and a working simulator all fail to help the model propose hypotheses. (4) Retro-correction: the keystone's "thinking-2AFC at chance" (P12) was inflated by budget-512 + a weak first-char parser; with budget 1024 and a strict answer format, base thinking-2AFC ≈ the no-think logit read. Net insight (C15): deployable capability = module × interface × procedure — weights install modules but capture interfaces; context supplies procedures but cannot create generators; only tools cross the generation wall.
Research Program Fit
structured_execution_and_compilers + posttraining_and_adaptation. Completes the installation-mechanism triptych (tools C12/C13, weights C14, context C15) and executes C14's next_tests #1–2 (prompt-bridging; format-capture characterization).
Method
Same 120 verified ladder tasks (d {2,3,4} × k {0,2}) and identical 2AFC items/decoys (fixed seed 4242) as qwen35_4b_simulation_keystone_repair. SIM adapter regenerated from the committed recipe. Conditions — 2AFC (greedy, thinking budget 1024, strict Answer: A/B format, last-match parsing): plain (budget control), ORCHESTRATED (stepwise simulate-both-compare procedure, Step i: [...] lines), ICL (two programmatically-constructed worked examples, disjoint tasks); identification: orchestrated generate-and-test (propose → simulate stepwise → check → revise → emit code), pass@2.
Results
| 2AFC | raw | parse | parse-conditional |
|---|---|---|---|
| base plain@1024 | 0.74 | 0.94 | 0.79 |
| base orchestrated | 0.83 | 1.00 | 0.83 |
| base ICL | 0.78 | 0.97 | 0.81 |
| SIM plain@1024 | 0.46 | 0.53 | 0.87 |
| SIM orchestrated | 0.51 | 0.53 | 0.95 |
Identification: base gen-and-test 0.08 (= bare 0.08); SIM gen-and-test 0.13 (bare 0.09).
- P-C1 (SIM+orch ≥ 0.70): REFUTED on raw (0.51) — but 0.95 parse-conditional. The star cell splits along the module/interface distinction the prereg didn't anticipate.
- P-C2 (orchestration helps base, gain shrinking with depth): direction confirmed (0.74→0.83) but the gain does NOT shrink — 0.85 at d4. Discrimination needs only partial simulation (the pipelines differ in one op), so the procedure stays viable at depth.
- P-C3 (ICL < +0.10): CONFIRMED (+0.04).
- P-C4 (gen-and-test lifts < 2×): CONFIRMED — no material lift for either model.
- P-C5 (interaction): raw A5−A2 = −0.32 (interface capture dominates); parse-conditional +0.12 (the module adds real capability when invocable). Both facts are the finding.
Controls
Same tasks/decoys/seed as the keystone (paired); budget control (plain@1024) separates budget from procedure; parse rates reported everywhere; ICL demos constructed programmatically on disjoint tasks; identification graded hidden, code-extracted as before.
Oracle Versus Deployable Evidence
Raw accuracy is deployable; parse-conditional is diagnostic (module capability given a working interface). The gap between them (0.51 vs 0.95) is itself the finding: format capture is an interface failure, not a capability failure.
Interpretation
The installation-mechanism triptych, completed:
| mechanism | installs | fails at |
|---|---|---|
| tools (C12/C13) | search + simulation externally | nothing measured — but costs interpreter calls |
| weights/SFT (C14) | modules, format-locally | interface capture; no implicit propagation |
| context (C15) | procedures (composition recipes) | cannot create generators (hypothesis proposal) |
Deployable capability = module × interface × procedure. The keystone's "sealed modules" softens: the module was never sealed — its output channel was hijacked. And the deepest wall of the arc sharpens: everything except hypothesis generation can now be installed or composed by some mechanism; proposing candidate programs from behavior remains untouched by weights, context, and procedure alike — only external enumeration (tools) crosses it. C13's deployment rule gets its final form: let tools generate, let context orchestrate, let the model simulate-and-transcribe.
Limitations
Parse-conditional accuracy conditions on a non-random half of SIM generations (tasks where the format survives may be easier — treat 0.95 as an upper bound); one substrate; QLoRA adapter; P12 correction rests on a prompt+budget+parser change measured jointly, not factorially.
Next Experiments
- Interface repair: brief mixed-format SFT (chains +
Answer: X+ code) — does 10% format diversity restore parse rates and make the 0.95 deployable? - Generation-wall autopsy: why can't proposal be composed? (Constrained candidate menus vs free proposal.)
- Factorial P12 re-measurement (budget × parser × format) to close the correction cleanly.
Artifact Manifest
See artifact_manifest.yaml. Adapter regenerable; per-task records + prereg + figure in-repo.
Experiment log 2
Show the running log (2 entries)
Design
Third installation mechanism (context), pre-registered. Same 120 verified tasks + identical 2AFC decoys (seed 4242) as the keystone. Conditions: {base, SIM adapter} x {plain@1024, orchestrated, ICL} for 2AFC; orchestrated generate-and-test for identification. SIM adapter regenerated (train_loss 0.021-equivalent).
Results
2AFC raw/parse/parse-conditional: base plain .74/.94/.79; base orch .83/1.00/.83; base ICL .78/.97/.81; SIM plain .46/.53/.87; SIM orch .51/.53/.95. Identification: base gen&test 0.08 (=bare), SIM 0.13 (bare .09).
- Context COMPOSES discrimination (procedure lifts base to 0.83, flat to d4 -- discrimination only needs partial simulation of the differing op).
- The trained module IS accessible in-context (+12pp parse-conditional over base under the same procedure) but FORMAT CAPTURE gates the interface (parse 0.53) -> raw 0.51. Module composes; interface captured.
- Hypothesis GENERATION un-composable: no context strategy moves identification (0.08-0.13).
- RETRO-CORRECTION: keystone P12 "thinking-2AFC at chance" was budget-512 + weak first-char parser; at 1024 + strict format base = 0.74-0.79 ~ no-think logit. "Thinking hurts" retracted -> "doesn't help without a procedure".
- P-C1 refuted on raw / split by module-vs-interface; P-C2 direction confirmed (gain does NOT shrink with depth); P-C3 confirmed (+0.04); P-C4 confirmed; P-C5 raw negative / conditional positive.
Ops: smoke revealed answer-truncation (answer_max 700 -> 1100). SIM ident smoke 0.38 was a cell artifact (first 8 tasks = d2k0). Claim C15; C13 P12 clause + C14 sealed-modules clause corrected.
Figures 1
Reproduce
Full run
bash scripts/full_chain.shRun steps are documented inside the experiment folder (README and scripts).