Qwen3.5-4B: Installing Universal Features via Designed Synthetic Curricula
The one idea you need
The curriculum teaches abstract procedures with invented symbols that do not resemble the benchmark. The crucial test is not whether the model learns those exercises, but whether the same update improves every unrelated family without erasing behavior it already had.
The question
Can carefully checked synthetic lessons add general procedures that improve every unseen benchmark family while preserving a strong starting policy?
What we found
Not with either tested training geometry. Continuing from the strong policy raised fresh synthetic accuracy from 50% to 69% and beat the untrained base overall, but it scored 31% aggregate versus 45% for the strong starting policy and fell below base on three of ten families. A second mixture trained from base with broad replay also reached 69% local accuracy, yet only 85% of its answers were well formed and it hit the response limit four times, so its broad test stayed sealed.
Why it matters
The lessons clearly changed the model, but local learning was not a universal gain. The result narrows the search toward gentler joint training that keeps mature broad behavior active during every update, with a replay-only comparison to separate designed-skill transfer from ordinary continued practice.
On this page
Results at a glance 1
How to read
For each family, compare the strong starting policy's change from base with the designed continuation's change from base. Bars below zero mark regressions.
Takeaway → The designed lessons create large wins in two families but regress three below base and lose overall to the strong starting policy.
Data table
| public benchmark family | blend minus base | sequential designed continuation minus base |
|---|---|---|
| chronicle | 0.5 | 0.75 |
| lockpick | 0.5 | 0 |
| menders | 0 | 0.0208 |
| mirage | 0.5 | 0.125 |
| rites | 0.125 | -0.125 |
| siftstack | 0.5 | 0.75 |
| sirens | 0.125 | 0.125 |
| stockade | 0.333 | -0.125 |
| toolsmith | 0.333 | 0.0104 |
| warren | -0.125 | -0.125 |
Numbers from experiments/qwen35_4b_universal_curriculum/runs/benchmark/quick_tb1024_seed78131_pilot1_tb1024/summary.json
Technical framing
Designed-only continuation specializes instead of lifting every family — Aggregate-only quick@1024, seed 78131, same merged qwen_vllm backend. The designed continuation makes two large positive moves but creates three negative families and loses 0.1385 aggregate to blend.
In the author’s words from the Report · “Summary”
Neither preregistered integration geometry installed a universal feature. Continuing the mature C53 blend adapter for one epoch on 800 designed-only rows raised fresh synthetic exact accuracy from 0.500 to 0.692 and reduced cap contacts from 10/26 to 1/26. On held-out quick@1,024, it remained well above base (+0.1406 aggregate) but regressed three families and lost 0.1385 to blend. From-base co-training on designed plus broad replay also reached 0.692 local accuracy, but failed its prospective parse and cap gates, so its benchmark remained sealed. The parent factorial is negative.
Overview
Outcome: sequential arm failed aggregate transfer; from-base replay union failed its frozen local gate before benchmark.
This experiment turns the doctrine in docs/installing_universal_features.md into a contamination-controlled search. It asks whether hand-designed, executable synthetic lessons can add procedures that transfer beyond their surfaces while retaining the strongest existing broad emission-policy install.
Research program
- Primary program:
agentic_breadth_installation. - Supporting programs:
benchmark_generalizationandstructured_execution_and_compilers. - Prior anchors: C11 (self-harvest is coverage-bounded), C49 (runtime LoRA is a silent no-op for this composite), C53 (the broad emission install is a strong but saturating control), C56 (executable exploration transfers while answer narration does not), and C59 (reasoning content—not nominal token count—crosses serial walls).
- Intake and closest duplicates:
idea_intake.md.
Frozen pilot question
Does a truth-audited, surface-varied curriculum of generic search, execution, verification, repair, uncertainty, state, and routing procedures add broad held-out transfer beyond the existing C53 blend install?
The pilot succeeds only if candidate-minus-base aggregate is positive and none of the ten public family deltas is negative on a fresh quick event. That is a screening result, not a universal-feature claim. Confirmation requires independent quick seeds, strictly positive mean deltas for every family, medium-tier transfer, paired uncertainty, and a matched-compute sampling baseline.
Design
- Model: only
Qwen/Qwen3.5-4B, pinned revision851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a. - Generator:
scripts/gen_curriculum.pyemits 13 lesson types—induction, execution, selection, tracing, verification, counting, repair, optimization, abstention, state carry, ordering, probe choice, and routing—over six randomized abstract surfaces. - Truth controls: every row comes from an executable specification; induction rows require query-identifiability, a real dead end, and a deterministic witness that the claimed two-step rule does not collapse to one primitive.
- Frozen doses: the full v2 corpus has 2,300 rows. The fast search tier is exactly 800 rows and one epoch. Exact tokenizer receipts require zero boundary merges, truncations, or skipped targets.
- Arms: pinned base, frozen C53
blend, designed-only, designed-plus-replay, and the first sequential armblend_then_designed_fast. The initial event compares base,blend, and the sequential arm. - Firewall: benchmark access is exclusively through
scripts/run_benchmark_aggregate.py. Each promoted arm is explicitly merged and evaluated with the sameqwen_vllmbackend, tier, and seed. Raw suite output never enters this experiment.
The adversarial review and frozen interpretation rules are in reports/design_review.md and reports/preregistration.md. Hardware caps are frozen in reports/protocol_amendment_001.md; the quantitative from-base replay-union local gate is frozen in reports/protocol_amendment_002.md.
Reproduction
Generate and test the frozen corpus:
.venv/bin/python experiments/qwen35_4b_universal_curriculum/scripts/gen_curriculum.py
.venv/bin/python -m unittest \
experiments.qwen35_4b_universal_curriculum.tests.test_curriculum -v
.venv/bin/python experiments/qwen35_4b_universal_curriculum/scripts/validate_curriculum.py \
--data experiments/qwen35_4b_universal_curriculum/data/sft_universal_fast.jsonl \
--mix induct=80,execute=60,select=50,trace=60,verify=60,count=30,repair=90,optimize=70,abstain=70,state=80,order=50,probe=50,route=50 \
--max-length 2048 \
--receipt experiments/qwen35_4b_universal_curriculum/data/sft_universal_fast.receipt.jsonThe first training command is preserved verbatim in runs/training/blend_then_designed_fast.json. The fail-closed entry point is scripts/train_trial.py; it authenticates corpus hashes, exact token exposure, model lineage, trainer row counts, and the final adapter.
Fresh synthetic screening uses scripts/eval_curriculum.py. Promoted adapters use scripts/merge_trial.py, and paired aggregate events use scripts/run_benchmark.py. Both refuse stale or unauthenticated artifacts.
Results so far
Corpus and training gates
- The original v1 corpus was rejected before training: 16/600 induction traces contradicted their answers and at least 33/600 nominal two-step rules collapsed to a primitive. The advertised historical run had never started.
The deterministic v2 full corpus contains 2,300 valid rows; the fast corpus contains 800. All six generator tests pass across process hash seeds.
blend_then_designed_fastconsumed 800/800 rows with zero skips at max length 2,048. It ran for 346.7 seconds, ending at train loss 0.3495. Adapter weights SHA-256:72af43458777245f7236f0df6edf89013eeade67140a9a16c4aee279f95e6e77.designed_plus_replay_fast_b1consumed all 3,040 frozen designed-plus-replay rows from base with zero skips at max length 4,096. The batch-2 launch hit a first-step WSL/CUDA residency failure; the exact effective-batch-8 recovery used batch 1 / accumulation 8, ran 380 steps in 2,622.6 seconds, and ended at finite train loss 1.366. Adapter weights SHA-256:e551c1d291fca993f94bdd03c1cbaeef43b1b7b7bd4f4f16d277cfa12bc6412b.
Fresh synthetic screen
On 26 unseen generator tasks at seed 88001 and a 1,024-token cap:
| arm | exact accuracy | parse rate | cap contacts | mean generated tokens |
|---|---|---|---|---|
frozen blend | 0.500 | 0.615 | 10/26 | 763.7 |
blend_then_designed_fast | 0.692 | 0.962 | 1/26 | 231.1 |
This passes installability/emission screening but is not transfer evidence. Induction and repair remain imperfect locally, and the sequential arm over-abstained on two route items; those negatives are retained in the local result.
The from-base replay union was screened prospectively on fresh seed 88002:
| arm | exact accuracy | parse rate | cap contacts | mean generated tokens |
|---|---|---|---|---|
designed_plus_replay_fast_b1 | 0.692 | 0.846 | 4/26 | 394.4 |
It met the 0.65 accuracy bar and recovered routing to 2/2, but failed the frozen parse rate (required at least 0.90) and cap-contact (required at most 2/26) gates. Induction was 0/2 and execution 0/2. Benchmark seed 78132 was therefore never consumed.
Aggregate transfer
Native quick@8,192 was rejected before any score: the permitted estimator marks it over the 60-second hardware gate on this RTX 4090. Two raw-suppressed failures and one interrupted diagnostic are retained. Protocol amendment 001 freezes quick@1,024 and medium@2,048, the highest power-of-two caps that pass their tier estimates.
Quick@1,024 seed 78131 completed through the paired qwen_vllm gateway:
| arm | aggregate | delta vs. base | positive / nonnegative families | minimum family delta |
|---|---|---|---|---|
| base | 0.1667 | — | — | — |
frozen blend | 0.4458 | +0.2791 | 8 / 9 | -0.1250 |
blend_then_designed_fast | 0.3073 | +0.1406 | 6 / 7 | -0.1250 |
The sequential candidate fails the pilot gate. It moves real axes—chronicle and siftstack are each +0.75 versus base, and aggregate is +0.1406—but rites, stockade, and warren are negative. Relative to blend, it loses 0.1385 aggregate and sharply regresses lockpick, mirage, rites, stockade, and toolsmith. This is specialization plus catastrophic displacement, not a universal feature. The full authenticated event is runs/benchmark/quick_tb1024_seed78131_pilot1_tb1024/summary.json.
Artifacts
- Checked in: deterministic corpora, tokenizer receipts, source, tests, intake, preregistration, design review, training/local receipts, and aggregate-only events.
- External: adapters and 9 GB composite checkpoints under
large_artifacts/qwen35_4b_universal_curriculum/. - Manifest:
reports/artifact_manifest.yaml.
Negative controls and failed infrastructure attempts are never overwritten. Any post-result adaptive curriculum or confirmation moves to a successor experiment. The parent factorial is complete; the next registered mechanism is qwen35_4b_universal_replay_anchor.
Report
Rendered from reports/pilot_report.md
Summary
Neither preregistered integration geometry installed a universal feature. Continuing the mature C53 blend adapter for one epoch on 800 designed-only rows raised fresh synthetic exact accuracy from 0.500 to 0.692 and reduced cap contacts from 10/26 to 1/26. On held-out quick@1,024, it remained well above base (+0.1406 aggregate) but regressed three families and lost 0.1385 to blend. From-base co-training on designed plus broad replay also reached 0.692 local accuracy, but failed its prospective parse and cap gates, so its benchmark remained sealed. The parent factorial is negative.
Research Program Fit
- Program:
agentic_breadth_installation. - Question: can correct human-designed executable procedures add the capability core that self-harvest/replay curricula leave behind?
- Prior anchors: C14, C49, C53, C54, C56, and C59.
Method
The inherited v1 experiment was audited before GPU work. It contained 16 contradictory induction traces, at least 33 behaviorally collapsed two-step rules, a dead smoke command, a fail-open shell chain, a firewall-invalid direct benchmark path, and no evidence that its advertised training run had started.
The replacement deterministic generator emits 13 executable lesson types over six abstract surfaces. Induction rows must be query-identifiable across every probe-consistent composition, contain a genuine dead end, and differ from every primitive on a deterministic witness bank. Six unit tests cover byte determinism across hash seeds, truth gates, depth, mix size, and smoke breadth.
The first arm warm-started immutable blend and trained on the frozen 800-row fast corpus for one epoch at learning rate 5e-5, rank 32, alpha 64, effective batch 8, max length 2,048, and think loss weight 0.2. All 800 rows encoded; zero were skipped. Training and adapter identity are authenticated in runs/training/blend_then_designed_fast.json.
The second arm trained from base on the union of those 800 rows and the frozen 2,240-row C53 replay corpus at the same learning rate, rank, alpha, effective batch, and think weight, with max length 4,096. A batch-2 first-step CUDA residency failure was preserved; batch 1 / accumulation 8 kept the exact dose and effective batch. It completed 380 steps, 3,040/3,040 rows, zero skips, and finite loss 1.366.
Infrastructure amendment
Native quick@8,192 cannot satisfy this host's public suite budget: the permitted estimator reports 321.9 expected / 440.3 worst seconds against a 60-second gate. Two scoreless gateway failures and one interrupted diagnostic occurred before any benchmark value was exposed. Protocol amendment 001 freezes quick@1,024 and medium@2,048, the highest power-of-two caps whose estimates remain within their tier gates. All arms use explicitly serialized/merged composites and the same qwen_vllm backend.
Results
Fresh synthetic seed 88001
| arm | accuracy | parse rate | cap contacts | mean generated tokens |
|---|---|---|---|---|
blend | 0.500 | 0.615 | 10/26 | 763.7 |
| sequential designed continuation | 0.692 | 0.962 | 1/26 | 231.1 |
The candidate improved execution, optimization, probe choice, tracing, and verification, but remained wrong on both induction and repair examples and incorrectly abstained on both feasible routing examples.
Aggregate-only quick@1,024 seed 78131
| family | base | blend | candidate | candidate − base | candidate − blend |
|---|---|---|---|---|---|
| chronicle | 0.1250 | 0.6250 | 0.8750 | +0.7500 | +0.2500 |
| lockpick | 0.0000 | 0.5000 | 0.0000 | +0.0000 | -0.5000 |
| menders | 0.0000 | 0.0000 | 0.0208 | +0.0208 | +0.0208 |
| mirage | 0.1250 | 0.6250 | 0.2500 | +0.1250 | -0.3750 |
| rites | 0.1250 | 0.2500 | 0.0000 | -0.1250 | -0.2500 |
| siftstack | 0.0000 | 0.5000 | 0.7500 | +0.7500 | +0.2500 |
| sirens | 0.3750 | 0.5000 | 0.5000 | +0.1250 | +0.0000 |
| stockade | 0.1250 | 0.4581 | 0.0000 | -0.1250 | -0.4581 |
| toolsmith | 0.6667 | 1.0000 | 0.6771 | +0.0104 | -0.3229 |
| warren | 0.1250 | 0.0000 | 0.0000 | -0.1250 | +0.0000 |
| aggregate | 0.1667 | 0.4458 | 0.3073 | +0.1406 | -0.1385 |
The candidate has six positive, one zero, and three negative family deltas versus base. It fails both the no-negative-family pilot gate and the strong-control comparison.
From-base union local gate, seed 88002
| arm | accuracy | parse rate | cap contacts | mean generated tokens |
|---|---|---|---|---|
| from-base designed plus replay | 0.692 | 0.846 | 4/26 | 394.4 |
The arm passed accuracy and feasible-route abstention checks but failed the frozen parse-rate and cap-contact checks. Induction was 0/2, execution 0/2, and four answers were unparsable. Per protocol amendment 002, no merge or benchmark event ran and seed 78132 remains unconsumed.
Controls and validity
- No benchmark source, item, transcript, verifier detail, or raw child stream was read or retained. Only the gateway's aggregate/public-family schema crossed the boundary.
- Base/control/candidate used the same seed, quick tier, 1,024 cap, vLLM backend, composite config/tokenizer/template serialization, and benchmark source inventory.
- Every output was within the public suite budget. The event summary SHA-256 is
d01ff14a6ef13a6503dfa586b445531d51b88cabe0cedcfe834cd4c4a13f87ca. - The first result was not used to rewrite the from-base arm. It was evaluated only on its prospectively frozen local gate; any new integration geometry is in
qwen35_4b_universal_replay_anchor.
Interpretation
Designed executable supervision is not inert: it made large, coherent moves on unseen surfaces and two held-out axes. But a fixed trace/template distribution at full continuation rate overwrote broad policy components faster than it installed the hard procedures. Local accuracy gains therefore cannot serve as a proxy for universal transfer. From-base co-training did not reproduce the sequential arm's concise emission behavior despite broad replay. The remaining discriminating test is mature warm-start replay anchoring: if it retains blend while keeping the new axis gains, the bottleneck was integration; if not, the designed content or its dose is too local.
Next experiments
- In the result-separated successor, test low-rate mature warm start with replay in every optimizer window against a matched replay-only refresh.
- Benchmark only locally retained candidates; require strict positive family deltas.
- Confirm a winner on independent quick seeds, medium@2,048, paired uncertainty, and matched-compute sampling before updating a shared claim.
Data files 1
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
Reproduce
Smoke test
.venv/bin/python experiments/qwen35_4b_universal_curriculum/scripts/gen_curriculum.py --smoke --out /tmp/qwen35_universal_smoke.jsonlFull run
Rebuild the checked-in token receipt, run scripts/train_trial.py using the frozen parameters in reports/preregistration.md, pass scripts/eval_curriculum.py, then use scripts/materialize_base.py, scripts/merge_trial.py, and scripts/run_benchmark.py.Run steps are documented inside the experiment folder (README and scripts).