Does the installable hypothesize-and-verify skill move the structure wall?

Results at a glance 3

Overview

Research Program

• Program: structured_execution_and_compilers (× posttraining_and_adaptation)
• Program question: the structure-wall arc terminates in C36 ("the fixed 4B cannot PROPOSE deep op-structure — a model-level law across substrates"); the induction-install arc terminates in C45 ("a GENERAL hypothesize-and-verify serial strategy is installable and transfers to a held-out family"). These two terminal laws make OPPOSITE predictions about one untested cell: aim the strategy at the wall.
• Prior anchors: C32/C36 (qwen35_4b_structure_or_values, qwen35_4b_crosssubstrate_structure — the wall + the behavioral skeleton metric), C45 (qwen35_4b_meta_induction — the installable strategy, committed train_general.jsonl), C44 (serial-compute law; CoT mandatory), C39 (execute-given-rule gate discipline), C25/C34 (guided search never beat brute — but guidance was logit-level, never a TAUGHT strategy).

Question

Can the model's own serial hypothesize-and-verify loop — elicited by prompt scaffold, transferred zero-shot from the C45 adapter, or installed by a small DSL-native reasoning-SFT dose — lift op-STRUCTURE proposal on the list/string DSL substrates where the wall was measured? C36 predicts no arm moves (the wall is a capability bound); C45's logic predicts the serial strategy converts proposal into enumerate-and-check, which the model CAN execute (C32: execution-given-structure ≈ 1.0).

Hypothesis

The wall arc never tested a TAUGHT serial strategy — its negatives are direct sampling (C17), logit-guided search (C25/C34), and steering (C20). C44 says the missing ingredient for induction is serial compute in the CoT channel; C45 says the loop template generalizes across (affine) families. If the wall is a proposal-PROCEDURE deficit, arms 1–3 climb (scaffold ≥ some lift; DSL-native SFT most); if it is a hypothesis-SPACE deficit (the model cannot even enumerate candidate op-types against I/O evidence), all arms stay at base and C36 hardens. Falsifiers are asymmetric and both informative: any significant lift breaks C36's "un-installable" reading; a flat result with a passing trap-gate scopes C45's strategy to retrieval-adjacent families.

Setup

• Model: Qwen3.5-4B; QLoRA r32/α64 for SFT arms (C45 recipe: bs 2 + grad-accum 8, 2 epochs).
• Dataset/task source: contamination-free procedural DSL identification tasks from the C36 families.py (byte-identical copy): list (16 prims) and string (13 prims) families; behavioral min-depth verification rejects shallower-equivalent compositions (C13 discipline).
• Train/eval split: frozen eval set, committed (data/eval_tasks.jsonl): n=30 tasks per family × depth {2,3} (120 total), generated once (seed 71), each with 8 visible + 6 hidden examples. SFT-arm training tasks are depth-1/2 compositions with op-composition dedup vs ALL eval tasks (0 leakage, verified at build time). The C45-adapter arm trains only on the committed digit-affine train_general.jsonl (different substrate entirely — zero leakage by construction).

• Arms (shared eval; matched K=12 think samples at budget 1024, identical decode params; the ONLY variable is strategy provenance):

  • base: C36's ident_prompt unchanged — the sample-more baseline (known: skeleton-coverage

  falls off a cliff by depth; replicates the wall).

  • scaffold (training-free, the pure-elicitation arm): same task rendering + an explicit

  enumerate-and-verify procedure in the prompt ("shortlist candidate op types consistent with the I/O evidence; compose a candidate pipeline; mentally execute it on example 1; check against example 2; revise; only then write the function").

  • c45_zero: C45 adapter (regenerated from committed train_general.jsonl), zero-shot on the

  DSLs with the base prompt — does the installed meta-skill transfer across substrates?

  • dsl_sft: reasoning-SFT on ~1,500 PROGRAMMATIC hypothesize-and-verify CoT traces (C45's

  trace-template method — interpreter-generated, no teacher model) on depth-1/2 list+string tasks disjoint from eval; deploy think-mode. The install ceiling.

• Baseline to beat: base arm at matched K (sample-more); secondary anchor: rand-skelfill@R (random skeletons + value-fill at matched interpreter budget — C32's value-fungibility control, CPU-only).

• Controls / gates (pre-registered, run.py hard-stops):

  1. Trap gate (C43-mandatory): oracle-skelfill (true skeleton + value search) must solve

  ≥ 0.85 of eval tasks per family×depth — otherwise a proposal null is uninterpretable. 2. Skill-installed gate for dsl_sft: the trained model must reproduce the taught trace format on ≥ 0.8 of held-out depth-1 tasks AND its depth-1 correctness must not collapse vs base in the deploy channel (C29/C43 forgetting check) — else the arm tests a failed install. 3. c45_zero regen-sanity gate: the regenerated C45 adapter must reproduce C45's committed headline (held-out a7 induction-via-generation 0.905; gate = historical − 2×SE at n=200 → 0.87; initially miscalibrated to 0.95 from C44's shift number, corrected — see log) BEFORE its DSL eval is spent — else a flat arm is a failed rebuild, not evidence. Measured: 0.920. 4. Leakage: dedup at op-TYPE-SEQUENCE (skeleton) level between all SFT training tasks and all eval tasks (0 exact-skeleton overlap at any depth, verified at build); depth-3 window overlap is stratified, not excluded (see decision rule). 5. Smoke path with _smoke artifact suffixes throughout (smoke can never poison a full run); run.py sets PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True itself; launch with pipefail per docs/compute_environment.md. 6. Base-arm anchors: a cheap NO-think base pass ties this frozen eval to C36's historical numbers, with a pre-registered branch: if base-THINK@1024 itself clears the historical wall (skeleton-coverage@K ≥ 0.25 at depth 3), the experiment's frame shifts from "can the strategy move the wall" to "serial-compute dose already moves it" (a C44 result) and arm contrasts are interpreted against that elevated base. Decode pinned in configs/default.yaml (temperature 0.8/top_p 0.95 sampling, greedy pass separate; answer_max 512).

• Primary metric: probe-robust skeleton-coverage@K per cell — ≥1 of K samples structure-correct under a MIMICRY-ROBUST extension of the C36 metric (review must-fix): some fill of the TRUE skeleton must reproduce the model program's behavior on the 8 visible inputs AND on 6 fresh probe INPUTS (labels never shown to the model) — lookup-table/hardcoded programs match visible but diverge on probes and are counted as a per-arm mimicry rate, reported alongside the legacy C36 metric for continuity. Secondary (characterization only, non-verdictive): greedy skeleton@1, full-solve coverage@K, per-family cells, depth-2 cells, code-parse rate per arm (a null with parse-rate far below base is pre-registered as "format-confounded", not "no transfer").
• Decision rule (pre-registered; power-corrected per review): PRIMARY contrast = each arm vs base on probe-robust skeleton-coverage@K, pooled list+string at depth 3 (n=60 tasks/arm; min detectable ~0.067, power 0.90 at true +0.15), one-sided paired bootstrap, Holm-corrected across the 3 arm-vs-base contrasts. C36's "un-installable/un-elicitable" reading falsified iff any corrected contrast has CI lower bound > 0 AND point diff ≥ +0.10 — with depth-3 results STRATIFIED by trained-window overlap (both/one/zero depth-2 sub-skeletons of the eval task seen in dsl_sft training): the falsification claim requires lift in the zero-or-one-window stratum (both-windows lift = composition of banked shallow structure, C24/C28 territory, reported as such). C45 scoped iff no corrected contrast moves (honestly: "no effect ≥ ~0.15 detectable") with all gates passing. Scaffold lift without SFT lift = elicitation-sufficient; SFT-only lift = installable-only (noting the pre-registered asymmetry: dsl_sft carries vocabulary+strategy, the scaffold strategy only). Prior-art conditioning: C28/C33 already installed depth-3 structure FROM oracle depth-3 content; the novel cell here is PROCEDURE-only transfer across depth (traces are depth-1/2 only; C21 is the negative prior for answer-banking across depth).
• Compute-parity (review must-fix): report per-arm mean prompt tokens, generated tokens/sample, total tokens/task, forced-close rate; base is additionally re-presented at K′ matched to the scaffold arm's total generated tokens. Pre-committed contingency: if a NULL arm shows forced-close > 50% at depth 3, run a budget-2048 probe on the depth-3 subset before concluding.
• Treatment freezing (review must-fix): the scaffold prompt text is checked in VERBATIM at configs/scaffold_prompt.txt (zero op names / substrate hints — C30/C31 hint-leak discipline; includes an explicit do-not-hardcode instruction) and the dsl_sft trace generator implements the SAME factorized procedure: evidence extraction (length/sign/order/duplicate deltas) → per-stage candidate SHORTLIST from a fixed truth-independent feature→candidate rulebook → compose from shortlists in fixed order → mentally execute on example 1 with intermediate states → on failure localize the divergent stage and revise → verify on examples 2–3 → only then emit code. Traces are truth-BLIND by construction (candidate order and shortlists are pure functions of visible I/O; verified at build time by regenerating with the oracle blinded and byte-comparing); tasks the blind procedure cannot solve within the trace budget are dropped (C45 cot→None pattern, drop rate reported); kept traces' final candidate must also solve the hidden set (train-data purity filter, C45-standard). Traces train into the THINK channel with the final code block as the answer (bank-the-thoughts train_lora_think pattern — a stated recipe change vs C45, which aligns train and deploy channels).
• Oracle-only metrics: oracle-skelfill gate, purity/leakage checks, rand-skelfill anchor. No oracle signal reaches any arm's prompt, training data selection, or decoding.
• Hidden-label boundary: hidden examples grade full-solve only; skeleton metric uses visible behavior + the true skeleton (evaluation-side oracle, standard for the wall arc).
• Known limits (pre-registered): single LoRA seed per SFT arm; K=12/n=30 resolves ~0.25 effects, not 0.10 subtleties; budget 1024 may truncate deep enumerate loops (report forced-close rates — C45 needed ≥400 tokens, C47 saw 99% truncation still work); register family deferred.

Run

Smoke:

python scripts/run.py --smoke

Full (measured ~6 h main pipeline + ~3 h budget-2048 contingency probes on the RTX 4090; pre-run estimate was 9–11 h):

python scripts/run.py            # idempotent; safe to re-run after interruption
python scripts/analyze.py

Results

Full narrative in reports/report.md; stats in runs/verdict.json; saga in experiment_log.md.

1. The wall holds (C36 hardens). No arm clears the pre-registered pooled depth-3 Holm contrast: dsl_sft +0.033 (p_holm 0.55), scaffold +0.033 (0.41), c45_zero −0.017 (0.82); base depth-3 probe-robust cov@12 = 0.05 (wall replicates at think@1024; no C44 frame-shift — base_think_clears_wall=false). Zero-or-one-trained-window stratum: +0.019, n.s.
2. The procedure installs, dramatically, WITHIN taught depths. dsl_sft (1,476 truth-blind d1/d2 traces, think-channel) doubles depth-2 structure proposal: list 0.37→0.70, string 0.33→0.57; deployable greedy@1 list 0.10→0.37; parse 1.00; zero d1 forgetting (0.85→0.85). Flat at depth-3 (0.10/0.07 vs 0.03/0.07): serial-strategy installs are depth-local — C21's cross-depth negative extends from banked answers to banked procedures.
3. C45's skill is substrate-local. The regenerated adapter beats its own in-family headline (0.920 vs 0.905) yet transfers at ~zero to the DSLs, with active interference (list d2 0.37→0.00) and degraded parsing (0.70–0.79).
4. Scaffold null is format-confounded (pre-registered reading): parse collapses to 0.44–0.67 vs base ~0.89 while coverage matches/edges base — the treatment text disrupts formatting more than it elicits search.
5. Guards: mimicry ≤0.014 everywhere (probe-robust metric armed, barely needed); oracle-skelfill 1.00 all cells; rand-skelfill ≤0.07; traces byte-verified truth-blind; 0 task-level skeleton leakage (51/1,476 trace found-pipelines coincide with eval d2 skeletons, 0 with d3).
6. Budget-2048 contingency (pre-committed): the d3 wall is partially BUDGET-limited for everyone. c45_zero 0.017 (null doubly confirmed); scaffold 0.150 — but the post-hoc base@2048 control doubles too (0.05→0.10, forced-close 0.84→0.75), and the paired matched-budget scaffold edge is +0.050 (CI-lo +0.000, p 0.043–0.051 across seeds — borderline, post-hoc, not verdict-grade). Serial-compute dose is the operative depth-3 margin; a small procedure margin beyond dose is unresolved at this n; even 2048 truncates 75–100% of thinking (dose-curve unfinished).

Interpretation

The wall is not a missing procedure. The model demonstrably learns and executes the full hypothesize-shortlist-check-revise loop (that is what 0.37→0.70 at parse 1.00 with zero forgetting means) and gains nothing one composition step past where the loop was practiced. With C21 (answers don't climb), C24 (diversity is in-depth dose), and C44 (induction is serial-compute limited): depth itself is the resource, and neither answers, diversity, nor procedure banks it across. C36 survives its strongest challenger; C45 gains a regime clause (family-general within its substrate, zero across substrates). Deployable takeaway: DSL-native procedure-SFT is the strongest in-depth structure-proposal lever measured on this substrate; the depth frontier still belongs to external search (C34/C35).

Knowledgebase Update

• Program evidence updated: structured_execution_and_compilers × posttraining_and_adaptation
• Claim ledger updated: C48 (this experiment); C21 annotated (cross-depth negative extends to procedures); C45 annotated (substrate-local scope clause)

Artifacts

• src/ — families.py (byte-identical C36 copy), gen_lib.py (C36 copy + the torch-2.12 OOM_ERRORS AcceleratorError patch from qwen35_4b_verifier_free_banking)
• scripts/ — make_tasks.py (frozen eval + SFT-train tasks, leakage check), gen_traces.py (programmatic DSL hypothesize-and-verify CoT), train_lora.py (C45 recipe), eval_arms.py (shared K-sample eval + skeleton metric), run.py (orchestrator, gates), analyze.py
• data/ — frozen eval sets, SFT training pairs, trace corpus
• runs/ — adapters (moved external before commit), eval JSONs, verdict
• reports/ — design_review.md, report.md, artifact_manifest.yaml

Report

Rendered from reports/report.md

Summary

The wall holds — and the way it holds is the finding. Aimed the C45 hypothesize-and-verify serial strategy at the C32/C36 structure-proposal wall via three routes (training-free prompt scaffold, zero-shot C45-adapter transfer, DSL-native reasoning-SFT). No arm clears the pre-registered pooled depth-3 contrast (Holm-corrected; dsl_sft +0.033 p=0.55, scaffold +0.033 p=0.41, c45_zero −0.017 p=0.82; base depth-3 probe-robust coverage 0.05 — the wall replicates in-house at think@1024). But the install itself is dramatic within taught depths: DSL-native SFT on 1,476 truth-blind procedure traces (depths 1–2 only) lifts depth-2 structure proposal from 0.37→0.70 (list) and 0.33→0.57 (string) at parse-rate 1.00, deployable greedy 0.10→0.37 (list) — with zero depth-1 forgetting (0.85→0.85) and full trace-format adoption (1.00). The strategy is installable but depth-local: it does not extend one composition step beyond its training depth. C21's cross-depth negative (banked answers don't climb) now extends to banked procedures. And C45's skill is substrate-local: the regenerated adapter (0.920 on its own held-out family, above its 0.905 headline) transfers at ~zero to the DSLs and actively interferes (list d2 0.37→0.00). Both terminal laws survive with new scope clauses: the wall is not a missing procedure (C36 hardens); the installable-induction result is a within-regime skill, not a transferable faculty (C45 scoped).

Research Program Fit

structured_execution_and_compilers × posttraining_and_adaptation: the single cell where the program's two most mature terminal laws — C36 ("the fixed 4B cannot propose deep op-structure, a model-level law") and C45 ("a general hypothesize-and-verify strategy is installable serial compute") — made opposite predictions. Either outcome had to rewrite a law's scope; both got rewritten narrower.

Method

Frozen eval (committed, seed 71): list + string DSL identification tasks (byte-identical C36 families.py), 30 tasks per family × depth {2,3} = 120, each 8 visible + 6 hidden examples + 6 fresh probe INPUTS (labels never in any prompt), behavioral min-depth verified. Four arms at matched K=12 think@1024 samples (temp 0.8/top_p 0.95, answer_max 512) + 1 greedy pass; a no-think K=8 base anchor ties to C36's historical decode. Arms: base (C36 ident_prompt), scaffold (the same + a verbatim-frozen evidence→shortlist→compose-and-check procedure text, configs/scaffold_prompt.txt, zero op vocabulary), c45_zero (C45 adapter regenerated from its committed digit-affine train_general.jsonl, seed-pinned; gated at 0.920 ≥ historical 0.905 on 200 held-out-a7 episodes before its DSL eval was spent), dsl_sft (QLoRA r32/α64 on 1,476 programmatic hypothesize-and-verify traces — evidence extraction → truth-independent feature→candidate shortlists → fixed-order compose → mentally-executed checks with intermediate states → revise-at-first-divergence — generated on depth-1/2 tasks only, op-type-sequence-deduped vs ALL eval tasks (0 overlap at the task level; 51/1,476 traces' behaviorally-equivalent FOUND pipelines coincide with eval depth-2 skeletons, 0 with depth-3 — the primary d3 contrast is clean), truth-BLIND by construction (byte-verified by regenerating with the oracle fields deleted), trained into the THINK channel). Primary metric: probe-robust skeleton-coverage@K (the C36 behavioral metric hardened against lookup-table mimicry: the same true-skeleton fill must reproduce the model program's behavior on visible AND probe inputs). Decision rule: pooled list+string depth-3 (n=60), one-sided paired bootstrap, Holm across the 3 arm-vs-base contrasts; falsification requires CI>0 AND diff ≥ +0.10 in the zero-or-one trained-window stratum. Gates: trap (oracle-skelfill), c45 regen-sanity, install (format + no d1 collapse). Pre-committed contingency: budget-2048 depth-3 re-probe for any null arm with forced-close > 50%.

Results

1. The wall replicates and holds (C36 hardens). Base depth-3 probe-robust coverage@12 = 0.033 (list) / 0.067 (string); pooled 0.05. No arm's Holm-corrected pooled contrast is significant and none reaches the +0.10 falsification bar (dsl_sft +0.033, CI-lo −0.033; scaffold +0.033, CI-lo 0.000, p_holm 0.41; c45_zero −0.017). The pre-registered zero-or-one-window stratum for dsl_sft: +0.019 (p=0.42). c36_falsified_for_content_free_installs = false.

2. The procedure INSTALLS — massively — within its taught depths. dsl_sft depth-2 probe-robust coverage: list 0.367→0.700, string 0.333→0.567; full-solve (hidden set) list 0.333→0.700; deployable greedy@1 list 0.100→0.367, string 0.067→0.200; parse-rate 1.00; install gate: trace-format 1.00, depth-1 correctness 0.85 vs base 0.85 (zero forgetting — the think-channel recipe avoids C43's answer-only collapse entirely). The same model that runs the taught loop to a doubled depth-2 proposal rate gains nothing at depth-3 (0.10/0.07 vs base 0.03/0.07). The procedure does not compose one step past its training depth.

3. C45's skill is substrate-local and negatively transferring. c45_zero, whose regenerated adapter beats its own source headline in-family (0.920 vs 0.905), scores 0.00–0.07 across all DSL cells — below base at list depth-2 (0.37→0.00) — with parse-rate degraded to 0.70–0.79. The "general" in C45 was family-general (affine a∈{1,3,9}→7), not substrate-general.

4. The scaffold null is format-confounded (pre-registered reading). Scaffold parse-rate collapses to 0.44–0.67 vs base 0.86–0.91; its coverage still matches or edges base (list d3 0.10 vs 0.03, greedy d2 up to 0.20 vs 0.07 on string). Per pre-registration this arm reads "the scaffold disrupts output formatting", not "elicitation fails"; per-parsed-sample rates and the 2048-budget re-probe (below) qualify it further.

5. Pre-committed budget-2048 contingency: the depth-3 wall is partially BUDGET-limited — for everyone. The forced-close trigger (>50% at d3) fired for two of the three null arms (scaffold and c45_zero, both 1.00; dsl_sft 0.32 did not trigger; base was 0.84). At budget 2048 on the depth-3 subset (K=12): c45_zero 0.017 (was 0.033) — null doubly confirmed. Scaffold 0.150 (was 0.083) — which prompted a post-hoc base@2048 control: base itself doubles, 0.050 → 0.100 (forced-close 0.84 → 0.75, still truncation-bound). Paired at matched 2048 budget the scaffold edge is +0.050 (one-sided CI-lo +0.000, p 0.043–0.051 across bootstrap seeds — runs/verdict.json contingency_b2048) — borderline and post-hoc, not verdict-grade in either direction. Reading: serial-compute DOSE moves depth-3 coverage (a C44-consistent softening: 2× budget ≈ 2× coverage, from a very low floor); whether the taught procedure adds a small margin beyond dose is unresolved at this n. Even 2048 truncates 75–100% of thinking — the depth-3 budget dose-curve (4096+) is unfinished business, not settled wall.

6. Mimicry guard: armed, unneeded. Mimicry rates ≤ 0.014 in every cell — the lookup-table leak the design review predicted did not materialize at these budgets, but legacy-vs-probe-robust gaps (base list d2 0.43 vs 0.37) show the guard did catch real instances.

7. Compute parity. Scaffold costs more per sample (prompt 523 vs 286 tokens; gen 1459 vs 1195 at d2) and still doesn't move depth-3; K′-matched base (K′=14, capped at stored K=12) leaves base d3 at 0.05 — no accounting trick closes the dsl_sft d2 gap (dsl_sft traces are also shorter at deploy, parse 1.00, making its d2 win compute-favorable).

Controls

Trap gate: oracle-skelfill 1.00 in all four cells (a proposal null is interpretable; C39/C43 discipline), rand-skelfill@12 ≤ 0.07 (structure is not value-searchable by luck at this budget). Blindness: traces byte-identical when regenerated with oracle fields deleted. Leakage: 0 op-type-sequence overlaps between SFT tasks and eval; depth-3 window-overlap stratified instead of excluded. Regen-sanity: the c45 gate initially FAILED at a miscalibrated 0.95 threshold (C44's shift number, not C45's 0.905 headline) — corrected to historical−2SE (0.87) with the measurement (0.920) unchanged; lesson codified. No-think anchor: base no-think d2 0.07–0.10, d3 0.00–0.03 — consistent with C36's historical floor; think@1024 alone does NOT clear the wall (no C44 frame-shift; base_think_clears_wall=false).

Oracle Versus Deployable Evidence

Oracle appears only in: eval grading, the trap gate, trace purity filtering (C45-standard programmatic generation), and probe labels computed at eval time. Deployable evidence: the dsl_sft depth-2 gains are fully deployable (greedy@1 0.10→0.37 list at parse 1.00 with no oracle at inference); the depth-3 null is measured under the same deployable protocol.

Interpretation

The wall is not a missing procedure — it is not fixable by teaching the model how to search. The dsl_sft arm proves the model can learn, format, and execute the full hypothesize-shortlist-check-revise loop (doubling proposal exactly where the loop was practiced) while gaining nothing one composition step deeper. Combined with C21 (banked answers don't climb), C24 (gains scale with in-depth diversity), and C44 (induction is serial-compute-limited), the sharpest available statement is: serial-strategy training buys depth-local competence; depth itself is the resource that neither answers, nor diversity, nor procedure can bank across. C45's "general induction is installable" survives only with a regime clause — general across families within a substrate and depth regime, zero across substrates (and negative-transfer prone). For deployment: DSL-native procedure-SFT is the best in-depth structure-proposal lever measured to date on this substrate (d2 coverage 0.70 vs banking-era numbers), but the depth frontier still belongs to external search (C34/C35 brute dominance) — among model-side levers only raw serial- compute BUDGET moved depth-3 at all (0.05→0.10 at 2× budget, Result 5), and that dose-curve, not any taught procedure, is the remaining open edge.

Next Experiments

1. Depth-3 budget dose-curve: base coverage doubled 1024→2048 and thinking is still 75–100% truncated — run 4096/8192 on the d3 subset (all arms cheap at n=60) to learn whether the wall asymptotes or keeps yielding to serial compute (this would sharpen both C36 and C44).
2. Depth-curriculum probe: traces at depths 1–3 (oracle content at d3, C28-style) vs depths 1–2 — is the depth-locality about practice at depth or about d3 procedure traces being ungeneratable blind? (The blind rulebook keeps 45–46% of d2 tasks after the purity filter — blind solve ≈51%; measure its d3 yield first, CPU-only.)
3. Scaffold format repair: the scaffold arm's parse collapse (0.44–0.67) is mechanical — a format-hardened scaffold (explicit "end with the code block" + one worked example) isolates elicitation from formatting; cheap single-arm rerun.
4. The C46 next-test still standing: skeleton-level P(True) ranking + judge-pruned beam at depth 4–5 (proposal is walled; RECOGNITION of partial structure is untested — forest-review option B).
5. Cross-substrate install transfer: train the trace procedure on list, eval on string (and register) — is procedure-SFT substrate-local like C45's adapter, or does within-DSL transfer hold at matched depth?

Artifact Manifest

runs/verdict.json (primary contrasts, cells, strata, parity, gates), runs/eval_*.json (5 arms

• 3 budget-2048 probes: 2 pre-committed contingency + 1 post-hoc base control), runs/gate_*.json, data/ (frozen eval with probe inputs, 1,476 traces,

SFT tasks, gate tasks — all committed), configs/scaffold_prompt.txt (verbatim frozen treatment), analysis/hypothesize_verify_wall.png, reports/design_review.md (3-lens pre-registration review), experiment_log.md (full saga: gate miscalibration stop, recovery-path empty_cache crash, both codified). Adapters (lora_dsl, lora_c45, ~182 MB each) moved external — see reports/artifact_manifest.yaml.

Experiment log 8

Figures 2

Data files 8

Result tables and metrics copied from the experiment folder — preview inline or open the raw file.

• previewruns/eval_base_b2048.json158 kB
• previewruns/eval_base_nothink_smoke.json11 kB
• previewruns/eval_base_smoke.json11 kB
• previewruns/eval_c45_zero_b2048.json158 kB
• previewruns/eval_c45_zero_smoke.json11 kB
• previewruns/eval_dsl_sft_smoke.json11 kB
• previewruns/eval_scaffold_b2048.json158 kB
• previewruns/eval_scaffold_smoke.json11 kB

Reproduce

Smoke test

python scripts/run.py --smoke

Full run

python scripts/run.py && python scripts/eval_arms.py --arm scaffold --budget-override 2048 --depths-only 3 --out-suffix _b2048 && python scripts/eval_arms.py --arm c45_zero --adapter runs/lora_c45 --budget-override 2048 --depths-only 3 --out-suffix _b2048 && python scripts/eval_arms.py --arm base --budget-override 2048 --depths-only 3 --out-suffix _b2048 && python scripts/analyze.py

Run steps are documented inside the experiment folder (README and scripts).

Browse the experiment folder on GitHub ↗

Related

Cited by claims

• ◑ Promising C48 · Hypothesize-and-verify SFT lifts taught depth 2 but not depth 3 at think@1024; cross-depth and cross-substrate transfer are null while the budget curve remains open

Related experiments

Nearest by shared name, tags, and programs.

• 2026-06-22 Qwen Verifier-Guided Slot Repair
• 2026-06-22 Qwen Teacher-Distilled Slot Compiler
• 2026-07-19 Qwen35 4B Agentic RLVR Feasibility
• 2026-07-09 → 10 Gauntlet round 1: breadth-first agentic expert iteration
• 2026-07-07 → 08 Qwen3.5-4B: Can Confidence Replace the Verifier in the Banking Flywheel?
• ~2026-06-25 Qwen3.5-4B Oracle-Distilled Semantic Verifier