Qwen35 4B WHY Scale Ladder

Results at a glance 1

Overview

construction is complete and frozen (generator + sha-pinned ladder + harness + 52 tests) and all three staged reviews PASSed (PASS_CONTROL_TRAINING / PASS_CONTROL_MERGE / PASS_MEASURE), but the sweep stopped mid-ladder — rungs 2000/5000 trained (1 epoch, loss 5.67 / 2.67; receipts in runs/training/) and rung 5000 merged, rung 10000 was SIGKILLed at 19% (why_scale_10000.failure.json, rc −9), rungs 20000/40000 never started, and MEASURE never ran for any rung: no HumanEval/MBPP pass@1, no scaling curve, no claim in the ledger. Voided by commit 115805e4 (the shared coding harness had been evaluating thinking-OFF; corrected base HumanEval 76.2% → 89.6%), which killed this ladder's minimal-<think> premise; the question moved to experiments/qwen35_4b_why_think_scale.

Research Program

• Program: agentic_breadth_installation (cognitive-core coding sub-program)
• Program question: can real, transferable coding capability be INSTALLED into base Qwen/Qwen3.5-4B by designed, contamination-free curricula, proven by transfer — and specifically, does teaching the 4B WHY a correct answer is correct SCALE with more (genuinely diverse) data to a peak worth building an RLVR foundation on?
• Prior anchors: experiments/qwen35_4b_coding_fitness_harness — HumanEval 76.2% (0.7622), MBPP 56.5% (0.565), agentic duet-eval 8/35. Bet #4 (qwen35_4b_why_comment_install) gave the program's biggest fast gain — HumanEval +5 (0.7622 -> 0.7927) on the clean, comment-inert test — but was underpowered (McNemar p=0.33), flat on the agentic loop, and did not survive combination.

Question

Does teaching the 4B WHY each line of correct code is correct — as inline #WHY: comments the grader ignores — SCALE? Trained at genuinely diverse rung sizes (2000, 5000, 10000, 20000), does HumanEval/MBPP pass@1 CLIMB to a peak (WHY is a real but underpowered signal), stay FLAT (the +5 was noise), or COLLAPSE (the narrow synthetic surface overfits)? The peak rung is the SFT foundation for the subsequent RLVR phase.

Hypothesis

The 504-row WHY gain was the low-N corner of a rising curve, not noise. With a generator that produces GENUINELY diverse data at scale (>= 50 families, >= 300 distinct WHY reasoning patterns, ~100% unique programs — removing the saturation that would have faked a null), pass@1 should climb with real scale to a peak before overfit/collapse. Because comments are inert to the execution grader, any gain is an unconfounded CODE improvement. Honest prior: P(a rung meaningfully beats base, >= +3 problems with retention) ~= 0.45; a flat curve remains a likely, informative outcome that reprices the WHY family.

Setup

• Model: only Qwen/Qwen3.5-4B (rev 851bf6e8…); one fresh r32/a64 QLoRA adapter per rung, trained from the base_reserialized composite in a single stage.
• Dataset/task source: scripts/gen_why_scale_curriculum.py (construction seed 94100) — 59 parameterized synthetic families producing spec -> correct solution rows where each meaningful line carries a trailing #WHY: causal comment, emitted BY CONSTRUCTION (NO teacher model), every row verified by real execution. Rungs at 2000/5000/10000/20000 rows are sha-pinned in data/ladder_manifest.json; the corpora are large, deterministically regenerable, and live gitignored under large_artifacts/.
• Train/eval split: training is spec -> #WHY:-commented correct code over synthetic families; evaluation is the held-out spec -> code HumanEval (164) + MBPP (200) — disjoint surfaces, grader ignores comments.
• Baseline: base Qwen/Qwen3.5-4B, co-measured on the same shared harness per rung (must beat it, not just move — "sample more" is the standing bar).
• Controls: contamination firewall (whole-word banned-name audit zero hits; distinctive code 7-gram overlap zero at scale); comments inert to the grader; the base composite authenticated fail-closed before training.
• Primary metric: greedy pass@1 on HumanEval + MBPP (shared fitness harness, experiments/qwen35_4b_coding_fitness_harness/scripts/eval_pass1.py, referenced not copied), swept per rung.
• Oracle-only metrics: none gate here; the agentic duet-eval is a follow-on confirm on the peak composite.
• Hidden-label boundary: benchmarks are executed, never read as data; the ladder curve is read once per rung from the four pass@1 numbers.

Run

Smoke (no GPU, no writes) — compiles, verifies the base provenance + fixture, builds a small verified corpus, checks the ladder-manifest shas, runs the tests:

python scripts/run.py --smoke

Build the ladder corpora locally from the committed manifest (CPU, model-free):

python scripts/run.py --stage gen-ladder

GPU stages (each gated behind a staged adversarial review; per rung; see reports/preregistration.md for the exact commands and checkpoint order):

python scripts/run.py --stage train   --rows 2000    # r32/a64, epochs 4, seed 94101
python scripts/run.py --stage merge   --rows 2000    # vendored composite merger
python scripts/run.py --stage measure --rows 2000    # shared HumanEval+MBPP harness
# ... repeat --rows 5000 / 10000 / 20000 (epochs 2 / 1 / 1) and assemble the curve.

Results

Not yet run. The install/merge/measure stages are gated behind staged reviews and are a SWEEP the orchestrator runs rung-by-rung. When run, runs/measure/rung_<rows>.json records each rung's four pass@1 numbers, the paired McNemar deltas, and the rung-vs-base problem deltas; the assembled curve pass@1(rows) locates the peak. Separate deployable evidence (a pass@1 gain over base, a real code improvement because the grader ignores comments) from the retention guard.

Model-free construction facts already established (see reports/report.md):

• 59 families / 13 categories; 5000-row sample: 59/59 families, 1196 distinct normalized WHY templates, 100% unique programs; 20000-row build 100% unique (raw draw ~82%).
• Contamination: 663 banned benchmark names after whitelist, 0 whole-word hits; 0 distinctive shared 7-grams (78 shared structural idioms) at 5000 rows.
• Token budget: full training render max 499 tokens (median 337) over 5000 rows — well under the 4096 cap; 0 rows truncate.
• 52 unit tests green; run.py --smoke green; boundary drills refuse.

Interpretation

Pending the sweep. A rising-then-peaking curve makes the peak rung the SFT foundation for the RLVR phase (Phase B) and funds the agentic confirm. A flat curve reprices the WHY-comment mechanism (does not scale on this surface) and takes RLVR from base instead. A collapse bounds the usable WHY dose and makes the small rung the foundation.

Knowledgebase Update

• Program evidence updated: pending the ladder sweep.
• Program backlog updated: pending the ladder sweep.
• Claim ledger updated: pending the ladder sweep (design-only work manufactures no claim).

Artifacts

• scripts/gen_why_scale_curriculum.py — the scale-capable, high-diversity WHY curriculum generator (59 families, phrase-pool rationales, per-row truth audit, contamination self-heal, deterministic per seed).
• scripts/build_ladder.py — builds the four rung corpora + the sha-pinned data/ladder_manifest.json; --verify regenerates and checks shas.
• scripts/contamination.py + data/contamination/banned_function_names.json.
• scripts/train_trial.py (fail-closed per-rung trainer), vendored scripts/train_think.py + scripts/merge_adapter.py, scripts/measure_transfer.py (per-rung sweep via the shared harness), scripts/run.py.
• data/ladder_manifest.json, data/provenance/base_reserialized.json.
• configs/, reports/ (preregistration, report, artifact manifest), tests/.

Report

Rendered from reports/report.md

Design-frozen report. The model-free construction (the scale-capable generator, the sha-pinned four-rung ladder, the train/eval harness, the tests) is complete and verified; the per-rung train/merge/measure GPU stages are gated behind staged adversarial reviews and have not run. Rung pass@1 numbers and the assembled scaling curve will be appended to the Results section as the sweep is read.

Summary

Phase A of the owner's scale-then-RLVR plan. Bet #4 (qwen35_4b_why_comment_install) taught the 4B WHY each line of a correct solution is correct via inline #WHY: comments and produced the program's biggest fast gain — HumanEval 0.7622 -> 0.7927 (+5 problems) on the cleanest possible test (comments are inert to the execution grader) — but was underpowered (McNemar p=0.33), flat on the agentic loop, and did not survive combination. The owner's directive is to SCALE the WHY curriculum to find its peak before overfit/collapse, merge the best rung as an SFT foundation, and then RLVR. This cell removes the blocker to that test — the sibling generator SATURATES (~75 WHY templates, 438 unique programs at 504 rows), so naive 20x replay would overfit and read as a false negative — by rebuilding the generator to produce GENUINELY DIVERSE data at scale, and it builds the sha-pinned four-rung ladder (2000/5000/10000/20000) and the per-rung train/eval sweep harness.

Research Program Fit

The program's target is the base 4B's coding-cognition gap: HumanEval 76.2% (strong function coder) alongside a weak agentic loop (8/35). Bet #4 showed teaching the GENERATING REASON of a correct answer is the strongest fast lever found, but at a single underpowered dose. This cell asks the scaling question the sibling could not: with genuinely diverse data, does the WHY signal CLIMB to a peak worth building an RLVR foundation on, stay flat (noise), or COLLAPSE (overfit)? HumanEval/MBPP are the fast transfer + retention signal; the agentic duet-eval is a follow-on confirm on the peak composite.

Method

• Scale-capable generator (scripts/gen_why_scale_curriculum.py, construction seed 94100). --rows N produces exactly N verified rows for any N up to ~30000, deterministically. 59 parameterized synthetic families across 13 categories (arithmetic accumulation, list reduce, list transform-by-hand, conditional chains, parity/modular/digit arithmetic, nested loops, pairwise/adjacent comparisons, bounded search, state machines, string manipulation, dict aggregation). For each row a correct function is generated BY CONSTRUCTION and each meaningful line is annotated with a trailing #WHY: comment stating the true causal reason that line is correct — chosen from a phrase-pool of TRUE, line-specific rationale variants per code construct so the corpus carries many distinct REASONING patterns, not a handful replayed. Prompt = a plain write a function framing (spec + signature + tests, NO instruction to comment — the WHY-writing behavior is the model's DEFAULT); think = a single minimal line; answer = the correct solution with the #WHY: comments.
• Per-row truth audit (never ship an unverified row), by REAL CPython execution. (1) STRIP the #WHY: comments -> the clean code passes ALL its tests; (2) the COMMENTED code runs and passes them IDENTICALLY (comments inert); (3) the marker is distinctive and mechanically strippable (# appears only as the marker); (4) every #WHY: comment is line-specific and the comments VARY within the row. Safety/termination: restricted builtins (no imports/I/O), only bounded for-loops (never while), a per-call step cap that aborts runaway code. A banned-vocabulary self-heal rejects any row that would carry a benchmark name.
• Contamination firewall (scripts/contamination.py, committed fixture of all 668 HumanEval + MBPP function names, 663 after the language whitelist; identical to the sibling cell). Zero whole-word hits over code + spec prose + #WHY: prose; a present-only code-only 7-gram aid finds zero distinctive shared spans vs the benchmark solutions (the accumulator/list-param pools avoid benchmark code idioms total/res/prod/arr/lst/nums).
• Scale ladder (scripts/build_ladder.py). Corpora at 2000/5000/10000/20000 rows (fixed seed 94100, different N), each verified + contamination-audited, sha-pinned in data/ladder_manifest.json (which also pins the generator sha and the fixture sha). The corpora are large and deterministically regenerable, so they live gitignored under large_artifacts/; --verify regenerates each rung and checks its sha.
• Install (scripts/train_trial.py -> vendored scripts/train_think.py). One fresh r32/a64 adapter per rung, lr 1e-5, batch 1, grad-accum 8, max-length 4096, w_think 0.2, w_close 0.2, seed 94101, from the base_reserialized composite (authenticated FAIL-CLOSED: in-cell provenance copy + full tree manifest + full 9 GB weights hash). Epoch schedule epochs = max(1, round(8000/rows)) = 4 / 2 / 1 / 1 (exposures 8k / 10k / 10k / 20k; optimizer steps 1000 / 1250 / 1250 / 2500).
• Merge (vendored scripts/merge_adapter.py) with --base-model = the base composite -> merged/why_scale_<rows>.
• Measure (scripts/measure_transfer.py -> SHARED harness, referenced not copied). Base and each rung composite, HumanEval 164 + MBPP 200, greedy pass@1, identical vLLM path. This is a SWEEP: all four numbers (counts + fractions) + the per-problem paired McNemar deltas + the rung-vs-base problem deltas are recorded per rung; the orchestrator assembles pass@1(rows) to find the peak. No single-shot install/null verdict here.

Results

Pending the sweep. runs/measure/rung_<rows>.json will carry pass_at_1{base,rung}{humaneval,mbpp}, the pass counts, the McNemar b/c paired deltas per dataset, and the rung-vs-base problem deltas; the assembled curve pass@1(rows) locates the peak. Deployable evidence is a pass@1 gain over the co-measured base (a real code improvement because the grader ignores comments); the retention guard is the paired dataset staying within tolerance.

Model-free construction facts already established:

• Diversity (the whole point — vs the sibling's ~75 templates / 438 programs at 504 rows). 5000-row sample: 59/59 families exercised across 13 categories, 1196 distinct normalized WHY reasoning templates (numbers/vars removed), 100% unique clean programs. 10000-row: 1197 templates. 20000-row: 100% unique programs (raw no-dedup draw ~82% unique — the diversity is genuine, not a dedup artifact). Rung templates: 2000 -> 1179, 5000 -> 1196, 10000/20000 -> 1197.
• Contamination: 663 banned benchmark names after whitelist, 0 whole-word hits over every row; 0 distinctive shared 7-grams (78 shared structural control-flow idioms) at 5000 rows, 0 at 10000 rows (HF cache present so the aid RAN).
• Token budget: full training render (chat + think + </think> + answer) max 499 tokenizer tokens (median 337, p95 455) over 5000 rows, measured against the pinned tokenizer — well under the 4096 cap; 0 rows truncate, so the trainer's zero-skip contract holds by construction. Character render max 1567.
• Determinism: the corpus is a pure function of (seed 94100, N); two rebuilds are byte-identical; the ladder --verify regenerates every rung to its pinned sha (2000 608192fa..., 5000 2a0fb91a..., 10000 d038452c..., 20000 e32584d0...).
• 52 unit tests green (diversity at 5000/10000/20000; WHY-truth re-executed by a separate assert-based grader; safety/termination; contamination zero at 10000; determinism; base auth fail-closed; epoch schedule; ladder-manifest sha pinning).

Controls

• Contamination firewall (banned-name audit + distinctive code 7-gram overlap), both zero at scale, so any benchmark movement cannot be memorization.
• Comments are inert to the execution grader, so any pass@1 gain is a CODE gain, not a grader artifact — the design property that makes this the clean test.
• The diversity fix is itself a control: because the data is genuinely diverse at scale (>= 50 families, >= 300 WHY patterns, ~100% unique programs), a flat curve cannot be blamed on replayed data, and a rising curve is real scaling — the saturation that would have faked a null is removed.
• Base composite authenticated fail-closed (tree + weights) before every rung's training and merge; a swapped composite aborts.
• Identical measurement path for base and every rung (the shared harness), with the base co-measured per rung, so deltas are directly comparable.

Oracle Versus Deployable Evidence

Deployable evidence = a pass@1 gain over the co-measured base on HumanEval/MBPP (real, held-out spec -> code generation, comments ignored by the grader), assembled into the scaling curve. The retention guard (the paired dataset within tolerance) is a control on the forgetting risk, not a capability claim. The agentic duet-eval is the eventual deployable target but is a manual follow-on confirm on the peak composite, not gated here. No metric here uses hidden labels beyond the per-rung pass@1 reads.

Interpretation

Pending the sweep. A rising-then-peaking curve makes the peak rung the SFT foundation for the RLVR phase (Phase B) and confirms the 504-row +5 was the low-N corner of a real scaling signal; it funds the agentic confirm. A flat curve reprices the WHY-comment mechanism — it does not scale on this synthetic surface — and takes the RLVR phase from base (or the sibling's 504 composite) instead. A collapse at the large rungs bounds the usable WHY dose and makes the small rung the foundation.

Next Experiments

• Phase B (RLVR) from the peak rung composite as the SFT foundation — the pre-committed successor if a rung clears the bar.
• If flat: reprice the WHY family; take RLVR from base or the 504-row composite.
• If collapse: document the overfit boundary; the small rung is the foundation.
• Follow-on: the agentic duet-eval confirm on the peak composite; consider stacking the peak WHY rung with the self_repair ingredient (the sibling's cross-bet complementarity finding).

Artifact Manifest

See artifact_manifest.yaml — the rung corpora (large, deterministically regenerable, sha-pinned in data/ladder_manifest.json) and the trained adapters / merged composites live under large_artifacts/ (omitted from git); the generator, contamination fixture, ladder manifest, base provenance copy, and receipts are in-repo and reproducibility-critical.

Experiment log 2

Reproduce

Smoke test

python scripts/run.py --smoke

Full run

python scripts/run.py --stage train --rows N && python scripts/run.py --stage merge --rows N && python scripts/run.py --stage measure --rows N  # per rung in {2000,5000,10000,20000}

Run steps are documented inside the experiment folder (README and scripts).

Browse the experiment folder on GitHub ↗

Related

Related experiments

Nearest by shared name, tags, and programs.

• 2026-07-18 → Qwen35 4B WHY-Think Scale
• 2026-07-14 Search-Scaffold Universal Curriculum
• 2026-07-14 Failure-Selected Counterfactual Restart Curriculum
• 2026-07-13 Validation-policy counterexample curriculum
• 2026-07-13 Replay-Anchored Universal Curriculum Continuation
• 2026-07-13 Mid-Density Token-Matched Universal Curriculum