Research log Small Model Experimentation
GitHub

Qwen3.5-4B Specialist Policy Integration

Finished2026-07-11Agentic Breadth InstallationBenchmark GeneralizationEvidence-Conditioned SelectionPosttraining and Adaptation+2 moreGitHub ↗
One nearly-perfect skill made the plan impossible

The one idea you need

Picture four high-jumpers, each ordered to clear a bar ten centimeters above their own best. The gym ceiling caps everyone. Three have room, but the fourth already jumps just under the ceiling, so ten extra centimeters simply have nowhere to go.

The question

Can you train four separate skill specialists, then merge them into one small model that combines all their strengths on brand-new tasks?

What we found

No — we never found out. Each of four required skills had to gain ten points before merging, but scores top out at 100. One tool-use skill already sat at 99.4%, so its target of 109.4% was impossible. The experiment stopped before a single training step. The real lesson: broad room on the overall task hid one corner already maxed out.

Why it matters

Before spending compute on a multi-specialist plan, check improvement room skill by skill, not just on the averaged task. A fixed points-gained target is unreachable once a skill nears its ceiling — size targets to the room actually left.

Starting score on the tool-use skill99.4%already pressed against the perfect ceiling of 100%
Score the rule demanded109.4%impossible, since the maximum is 100%
Room to improve versus room required0.6 vs 10 pointsthe fixed gain bar dwarfed the headroom left
Specialists actually trainedzerothe plan halted before any training ran
On this page
  1. Results at a glance
  2. Overview
  3. Report
    1. Status
    2. Research Program Fit
    3. Reached Evidence
    4. Terminal Feasibility Result
    5. Interpretation
    6. Unreached by Design
    7. Artifact Manifest
  4. Experiment log
  5. Data files
  6. Reproduce
  7. Related

Results at a glance 1

Required scores versus the 100 percent ceiling, per skill

How to read

Four skill domains sit along the bottom: discovery-and-repair, stateful control, tool use, and composition. Each shows three bars — current score, the score its rule required, and the 100% ceiling. A required bar staying under the ceiling means the target is reachable.

00.511.50.5130.6131discover0.5230.6231control0.9941.091tools0.180.281compose

Takeaway → Only the tool-use skill's required bar pokes above the ceiling, so its target can never be met; the other three sit well below and stay feasible.

Data table
specialist domainincumbent scorerequired specialist scorescore ceiling
discover0.5130.6131
control0.5230.6231
tools0.9941.091
compose0.180.281

Numbers from experiments/qwen35_4b_specialist_policy_integration/analysis/specialist_headroom_gate.json

Technical framing

One mandatory specialist threshold exceeded the score ceiling — The tools incumbent was already 0.994, so its frozen +0.10 qualification bar required 1.094 despite a hard ceiling of 1.0. The preregistered gate stopped before best-of-8 or specialist training; this does not test the integration mechanism.

In the author’s words from the Report · “Interpretation”

The most promising follow-up is a new experiment—not a threshold amendment: keep the same regenerated incumbent, discovery/control/compose cores, installation gates, corrected MOPD loss, and matched controls; replace or structurally harden the saturated tools/provenance core using a disjoint calibration pool (an existing provenance candidate such as gatepost has measured room, but requires a new split and held-out family); require every core's 1 - S0_macro to exceed every frozen absolute gain bar before best-of-8 or any training; and freeze fresh paired seeds only after this domain-level feasibility gate. … Read the full result →

Overview

Status: stopped negative on 2026-07-12 before best-of-8 or specialist training. The tools core's incumbent score is 0.994, making its frozen S0 + 0.10 qualification target 1.094 on a score bounded by 1.0. Because all four specialists were mandatory, teacher audit and integration are unlicensed.

This experiment tests whether independently execution-improved, same-origin specialists can be consolidated on the student's own trajectories into one Qwen/Qwen3.5-4B policy that composes their capabilities on held-out tasks.

Research Programs

Question and Hypothesis

Does execution reward first create policies with real headroom, and can same-observation on-policy multi-teacher distillation integrate that headroom without the privileged-context shortcuts, exposure bias, or mixture see-saw seen in adjacent methods?

The mechanism is supported only if qualified specialists improve exact student-prefix continuations, correctly routed MOPD beats wrong routing and matched integration controls, and the final student exceeds every individual teacher on never-trained compound pairings/depths.

Substrate

Primitive specialist domains:

  • discovery/repair: glyphgate, loomfix;
  • stateful control: kilnrite, burrowmaze;
  • tools/provenance: ferrier plus permitted foundry_ledger atom replay.

Composition specialist training:

  • cipherkiln: infer a cyclic code mapping, then execute a legal protocol;
  • mazeferry: explore a partial map, find tools, and carry typed handles through a dependency chain.

No-new-exposure/held-out evaluation:

  • primitive: patchwheel, spindle, gatepost;
  • compound: patchferry, tripleforge, plus order reversals.

Every compound family has an exact oracle and explicit primitive-removal policies. The CPU smoke currently records oracle score 1.0 and ablation full- success 0.0 at every L1-L4 cell.

Stages and Stop Rules

  1. Regenerate and merge the C53 incumbent S0; pass HF/vLLM and nonzero- composite gates.
  2. Verify that every frozen pass-one specialist gain bar has mathematical headroom under the environment score ceiling. This postmortem gate now stops before best-of-8 or training when a target is unreachable.
  3. Produce four DAgger-to-execution-RL specialists. Every specialist must beat DAgger, extra SFT, shuffled reward, and S0 best-of-8.
  4. Audit correct versus KL-matched wrong teachers on exact S0 prefixes and run the five-update exact-logit locality pilot.
  5. Integrate qualified teachers with corrected top-50 MOPD. Compare end-to-end matched joint RL, off-policy SFT, parameter merge, and wrong routing.
  6. Evaluate three seeds on individual domains, primitive transfer, and held-out compounds. Open the benchmark CLI only if every whitebox gate passes.

The exact thresholds, seeds, metrics, and interpretation are frozen in reports/preregistration.md and configs/default.yaml. The adversarial review is reports/design_review.md.

Firewall

  • The only model is Qwen/Qwen3.5-4B at the pinned revision.
  • Nothing under benchmarks/ is imported, read, or used for training.
  • Programmatic state labels DAgger but is never placed in model input.
  • MOPD teacher and student see the identical observable prompt and prefix.
  • All comparable generations use the same pinned vLLM backend.
  • Transfer families are excluded from every new training and replay row.

Run

CPU scientific smoke:

python3 experiments/qwen35_4b_specialist_policy_integration/scripts/run.py --smoke

The smoke writes runs/smoke/summary.json and verifies all compound oracles, necessity ablations, state-aware experts, and split/replay invariants.

Reached model stages are resumable and fail closed on missing upstream receipts:

python3 experiments/qwen35_4b_specialist_policy_integration/scripts/run.py --stage model-smoke
python3 experiments/qwen35_4b_specialist_policy_integration/scripts/run.py --stage incumbent
python3 experiments/qwen35_4b_specialist_policy_integration/scripts/run.py --stage calibration-gate
python3 experiments/qwen35_4b_specialist_policy_integration/scripts/run.py --stage baseline-eval
python3 experiments/qwen35_4b_specialist_policy_integration/scripts/run.py --stage dagger-collect --domain discover
python3 experiments/qwen35_4b_specialist_policy_integration/scripts/run.py --stage dagger-train --domain discover
python3 experiments/qwen35_4b_specialist_policy_integration/scripts/run.py --stage rl-collect --domain discover
python3 experiments/qwen35_4b_specialist_policy_integration/scripts/run.py --stage specialist-train --domain discover
python3 experiments/qwen35_4b_specialist_policy_integration/scripts/run.py --stage controls --domain discover
python3 experiments/qwen35_4b_specialist_policy_integration/scripts/run.py --stage specialist-eval --domain discover
python3 experiments/qwen35_4b_specialist_policy_integration/scripts/run.py --stage specialist-analyze --domain discover

Replace discover with control, tools, or compose. Teacher-audit and integration stages remain deliberately unavailable until all specialist gates pass; a request cannot silently bypass the stop hierarchy. In the reached result, --stage baseline-eval resumes the committed greedy baseline, rewrites the negative headroom receipt deterministically, and stops before best-of-8.

Current Results

Reached evidence:

  • four compound families deterministic and JSON-safe;
  • exact oracle 1.0 at every L1-L4 cell;
  • generic random policy 0.0;
  • every registered primitive-removal policy full-success 0.0; and
  • state-aware live experts solve every cell at 1.0;
  • the pinned model/runtime answered 4/4 generic semantic smoke prompts and honored the explicit CUDA-graph geometry on the live L40.
  • the pinned Transformers path produced finite logits with both Qwen fast-path extensions, a two-step QLoRA made 128/128 nonzero mapped deltas, and vLLM loaded the explicitly merged local composite.
  • a one-step QLoRA smoke exited successfully but yielded an all-zero adapter; the merge gate rejected and preserved it, demonstrating that Trainer exit status alone is not an installation check.
  • the full incumbent ran all 333 optimizer steps over 2,117 encoded rows (123/2,240 rows skipped at the frozen 2,048-token cap), then produced 128/128 nonzero explicitly mapped deltas with summed Frobenius norm 161.39; the CUDA FP32/no-TF32 merged composite is weight-hashed in its receipt.
  • all 7/7 frozen visible-prefix canaries changed versus the pinned base under identical greedy prompts, runner hash, sampling, CUDA graphs, and runtime lock. The aggregate incumbent provenance/install gate passed every check.
  • on a disjoint 288-episode compound calibration (four families, L2-L4, 24/cell), the incumbent scored 0.135 macro: cipherkiln 0.227, mazeferry 0.296, patchferry 0.012, and tripleforge 0.005. The strict <0.60 headroom gate and every scope/decode/seed/atom-firewall check passed.

The full paired greedy baseline then resolved feasibility:

Specialist coreFrozen familiesS0 macroRequired scoreFeasible under cap 1.0
discoverglyphgate, loomfix0.5130.613yes
controlkilnrite, burrowmaze0.5230.623yes
toolsferrier0.9941.094no
composecipherkiln, mazeferry0.1800.280yes

The all-process macro was 0.458 over 864 episodes; atom-retention macro was 0.681 over 1,344 items. Every baseline protocol check passed. The originally scheduled all-family best-of-8 was interrupted during engine warmup before any sampled output, and the new feasibility gate now deterministically refuses it.

This is a design-negative result, not evidence for or against MOPD/OPSD: the experiment failed to provide a falsifiable four-teacher test because one mandatory improvement bar was impossible before training. No DAgger, GRPO, specialist, teacher-audit, integration, confirmatory, or benchmark stage ran. Any follow-up must live in a new experiment, retain the 0.10 bar, and calibrate headroom independently for every specialist domain before GPU production.

Artifacts

  • src/gym/families/compound_core.py: shared exact compound mechanics.
  • src/curriculum.py: state-aware expert interface.
  • runs/smoke/summary.json: committed CPU gate receipt.
  • analysis/specialist_headroom_gate.json: terminal negative stop receipt.
  • runs/proxy_eval/incumbent_calibration/: paired greedy/atom baseline.
  • reports/artifact_manifest.yaml: external checkpoint policy.
  • future large weights: large_artifacts/qwen35_4b_specialist_policy_integration/.

Report

Rendered from reports/report.md

Status

Stopped negative on 2026-07-12. Runtime, incumbent installation, and compound-headroom gates passed, but the paired greedy baseline made the tools specialist's frozen gain bar mathematically unreachable. No specialist or integration capability claim is available.

Research Program Fit

This is the registered beyond-C53 mechanism for agentic_breadth_installation: execution-reward RL produces headroom and on-policy multi-teacher distillation attempts to integrate it. It also directly tests post-training interference and held-out composition.

Reached Evidence

runs/smoke/summary.json records:

  • exact oracle score 1.0 for cipherkiln, mazeferry, patchferry, and tripleforge at every L1-L4 cell;
  • generic random policy score 0.0;
  • all discovery/control/navigation/repair/tool removal policies at 0.0 full success; and
  • state-aware live expert score 1.0 in all 16 family/level cells.

This establishes substrate validity only. It does not show that the fixed model can learn any primitive or composition.

The generic runtime smoke loaded the pinned revision on the live NVIDIA L40, resolved the requested full-decode CUDA-graph sizes exactly, and answered 4/4 format/semantic probes. It validates the inference path only.

The Transformers smoke found finite padded-vocabulary logits and both required Qwen fast paths. A two-step rank-32 QLoRA produced 128 nonzero composite-mapped deltas (summed Frobenius norm 8.742), and the merged checkpoint loaded through the same vLLM path. The first one-step attempt is a preserved negative: Trainer reported success, but every delta was zero and the merge correctly refused it.

The full incumbent then completed 333/333 optimizer steps in 3,054.4 seconds on the live L40. The frozen 2,048-token encoder admitted 2,117/2,240 rows and skipped 123 (5.49%): 116 skipped rows were forced-close atoms and seven were episodes, a disclosed concentration inherited from the exact C53 recipe. The explicit merge applied 128/128 nonzero deltas (summed norm 161.39, maximum 2.90) on CUDA in FP32 with TF32 disabled. All seven frozen visible-prefix canaries changed versus the pinned base while prompt, runner, sampling, graph, and environment-lock metadata matched. analysis/incumbent_gate.json passes all source-data, encoding, hyperparameter, optimizer, merge, and installation checks. This proves the intended checkpoint was installed; it does not yet show compound headroom or improvement.

The disjoint compound-only calibration then evaluated 288 greedy episodes at L2-L4 (24 per family/level) under episode seed base 80500, with atom generation disabled. The family means were 0.2271 cipherkiln, 0.2961 mazeferry, 0.0115 patchferry, and 0.0053 tripleforge, for macro 0.1350 against the pre-registered exclusive ceiling 0.60. Exact success was 5.56%, 1.39%, 0%, and 0%. Every protocol check passed, and the compacted artifact contains no simulator spec, expert label, message transcript, or raw top-20 payload. This establishes endpoint headroom and provisionally passed that gate; the later per-core feasibility result shows that the complete specialist-production plan still could not satisfy its frozen qualification rules. It is not evidence that a training method improved the model.

Terminal Feasibility Result

The full paired greedy baseline evaluated all 12 process families at L2-L4, 24 episodes/cell (864 episodes), plus 1,344 atom-retention items. It used the committed clean runner/environment, 3,893,188 sampled episode tokens, 4,909,001 logical episode-input tokens, and 4,991.3 wall seconds. Process macro was 0.4582 and atom macro 0.6806.

CoreIncumbent macroFrozen +0.10 targetMaximum possible gainFeasible?
discover0.51270.61270.4873yes
control0.52300.62300.4770yes
tools0.99401.09400.0060no
compose0.17970.27970.8203yes

All registered environment scores are bounded in [0, 1]. Therefore no possible ferrier specialist can satisfy the frozen S0 + 0.10 requirement on these paired cells. This is not sampling uncertainty: the decision rule uses the observed paired baseline, and its required score is above the score contract's hard ceiling. Because the preregistration requires all four specialists, the experiment stops before best-of-8, DAgger, execution RL, controls, teacher audit, or integration.

The old all-family best-of-8 subprocess was interrupted during vLLM warmup, before any sampled request or output artifact. A committed fail-closed analyzer now reproduces the negative receipt and blocks that spend automatically.

Interpretation

This experiment did not test whether specialist RL creates headroom or whether MOPD integrates it. It found a prior design error: compound-level headroom does not imply headroom for every mandatory teacher. The adversarial review correctly guarded average-masking after training but missed the simpler upper-bound feasibility check before training.

The most promising follow-up is a new experiment—not a threshold amendment:

  1. keep the same regenerated incumbent, discovery/control/compose cores, installation gates, corrected MOPD loss, and matched controls;
  2. replace or structurally harden the saturated tools/provenance core using a disjoint calibration pool (an existing provenance candidate such as gatepost has measured room, but requires a new split and held-out family);
  3. require every core's 1 - S0_macro to exceed every frozen absolute gain bar before best-of-8 or any training; and
  4. freeze fresh paired seeds only after this domain-level feasibility gate.

Lowering the +0.10 bar, dropping the tools teacher, or reusing the current evaluation cells to tune a replacement would answer a different question.

Unreached by Design

  • execution-filtered best-of-8 (zero sampled outputs);
  • all four specialist/control pipelines;
  • same-prefix teacher/locality audit;
  • MOPD and matched integration controls;
  • confirmatory and benchmark evaluation.

Artifact Manifest

See artifact_manifest.yaml. Large adapters and merged checkpoints remain external and receive checksums and regeneration commands when produced.

Experiment log 6

Show the running log (6 entries, 2026-07-11 → 12)

Scaffold

Created as a new experiment scaffold attached to agentic breadth installation, post-training/adaptation, and benchmark generalization.

2026-07-11 — preregistration and CPU substrate gate

  • Accepted the specialize -> distill -> compose decision record.
  • Copied the prior interactive-policy harness into this standalone experiment.
  • Added four procedural compound families with exact oracles and explicit primitive-removal policies.
  • Locked the preregistration and adversarial design review before model output.
  • CPU scientific smoke passed all L1-L4 oracle, random, necessity, live-expert, split, and replay-exclusion checks. No model baseline or training was run.

2026-07-11 — runtime implementation checkpoint

  • Discovered that the live GPU is an NVIDIA L40 (46,068 MiB), not the RTX 6000 Ada recorded by the previous pod; updated the shared environment docs.
  • Recreated separate pinned vLLM and Transformers environments from committed locks. The first pinned vLLM load resolved the explicit CUDA-graph geometry and answered four generic semantic probes correctly. These probes contain no gym item and license no capability claim.
  • Fixed the merged-composite path so the current runner genuinely loads and fingerprints a local checkpoint instead of accepting an unusable harness argument.
  • Added resumable domain-isolated DAgger, GRPO, extra-SFT, shuffled-reward, paired evaluation, diagnostic, and gate stages.
  • Pre-baseline amendment: increased extra-SFT from 120 to 300 steps because GRPO has multiple forward passes per optimizer step. This preserves the preregistered compute-overmatched control; no task model output existed.
  • The first one-step QLoRA preflight ran for 110.1 seconds and exited normally, but all 128 reconstructed LoRA deltas were zero; explicit merge refused it. The adapter remains under the external smoke artifacts and its compact failure receipt is committed.
  • A two-step, accumulation-one rerun exercised a nonzero optimizer update: all 128 mapped deltas were nonzero (summed Frobenius norm 8.742), explicit FP32/no-TF32 merge succeeded, and vLLM loaded and generated from the local composite. HF/vLLM prompt-token counts matched 4/4.
  • A concurrent main-branch environment update introduced the repository's canonical PEFT 0.19.1, bitsandbytes 0.49.2, accelerate 1.14.0, and xFormers pins. The work was rebased rather than overwritten, a full dependency lock was regenerated, and the entire finite-logit/train/merge/local-vLLM preflight passed again under that exact lock.
  • Primary-paper correction before any specialist output: MOPD equation (5) adds -p_student + p_teacher to each teacher-top-k reverse-KL summand. The earlier “corrected tail mass” wording was inaccurate; no lumped tail bucket will be implemented. The registered top-50 choice and all gates are unchanged.
  • A second pre-output orchestration audit found that the compound headroom call would have inherited seeds.proxy_eval_base instead of the separately frozen split.calibration_seed_base. Before any gym-model generation, the call and its analyzer were made explicit and fail-closed on seed namespace; unrelated atom generation was also disabled for this compound-only gate.
  • The same audit strengthened the frozen shuffled-reward control: advantage vectors are now deranged within each family/level cell with no fixed-point groups. A plain permutation would retain roughly one correctly routed group per cell in expectation and unnecessarily dilute the negative control.
  • Specialist qualification now also fails closed on partial or guard-stopped DAgger/GRPO runs, stale evaluation-to-merge fingerprints, adapter hash mismatches, zero/partial merge mappings, and compute-short controls. Stopped checkpoints remain preserved and evaluable for diagnosis but cannot qualify.
  • The preregistered behavioral installation rule is enforced before downstream use for every DAgger, extra-SFT, shuffled-reward, and real-reward composite. Reusable seven-prefix canaries are hash-bound to the source and candidate merge receipts; a candidate must change at least one greedy token sequence under identical runner, sampling, graph, prompt, and environment metadata.

2026-07-11 — incumbent regeneration and installation gate

  • The frozen C53 recipe completed all 333 optimizer steps over 2,117 encoded rows in 3,054.4 seconds on the L40 (12.82 GB peak allocated CUDA memory).
  • The independent encoder audit found 123/2,240 skips (5.49%): 116 forced-close atom rows and seven episode rows. No row was truncated or silently relabeled.
  • Explicit CUDA FP32/no-TF32 merge applied 128/128 nonzero deltas, with summed Frobenius norm 161.39 and merged-weight SHA-256 56e2bec45199ebcc....
  • All 7/7 frozen visible-prefix greedy canaries changed relative to the pinned base with identical prompt, runner, engine, sampling, graph, and environment metadata. The aggregate incumbent gate passed every registered check.
  • These receipts license the disjoint compound-headroom calibration only; they do not establish capability improvement.

2026-07-11 — disjoint compound-headroom gate

  • Ran 288 greedy episodes over cipherkiln, mazeferry, patchferry, and tripleforge, L2-L4 with 24 episodes/cell, at the separately frozen episode seed base 80500. Atom generation was disabled.
  • Family means were 0.2271, 0.2961, 0.0115, and 0.0053 respectively; macro 0.1350 passed the preregistered exclusive <0.60 ceiling by a wide margin.
  • Every protocol check passed. Generation used 1,942,775 sampled tokens and 2,229,494 logical input tokens across 3,742 turns in 2,225.5 wall seconds.
  • The post-run firewall check confirmed that neither hidden specs/expert labels nor raw top-20 logprob payloads survived in the committed evaluation rows.
  • This is a headroom/measurement result only. It licenses matched baselines and specialist production; it does not show a capability gain.
  • Before matched baseline output, unused atom passes were removed from the best-of-8, DAgger, extra-SFT, and shuffled-reward evaluations. Qualification reads atom retention only from the paired incumbent and real specialist; episode prompts, seeds, decoding, and every registered comparison are unchanged.
  • Before best-of-8 output, its qualification scope was narrowed to the seven specialist-training families and its token ledger made own-domain paired. Primitive/compound transfer best-of-8 is not consumed by specialist gates; the held-out compound comparison remains a separate 128/cell confirmatory arm if integration is reached. This preserves the stop hierarchy and avoids spending eight rollouts on five families before a teacher exists.

2026-07-12 — terminal specialist-headroom stop

  • Completed the paired greedy baseline on all 12 process families: 864 episodes, macro 0.4582, plus 1,344 atom items at macro 0.6806. The clean committed runner generated 3,893,188 episode tokens in 4,991.3 seconds.
  • Core macros were discover 0.5127, control 0.5230, tools 0.9940, and compose 0.1797. With the frozen +0.10 pass-one bar and score ceiling 1.0, the tools target is 1.0940 and its maximum possible gain is only 0.0060.
  • Interrupted the just-started old-scope best-of-8 subprocess during engine warmup. It generated zero sampled requests and wrote no result artifact.
  • Added analyze_specialist_headroom.py; the resulting terminal receipt passes every baseline protocol check, marks only tools impossible, and authorizes stop_before_best8_and_specialist_production. A stage resume reproduces the stop before GPU load.
  • No DAgger, GRPO, specialist, control, teacher-audit, MOPD, confirmatory, or benchmark stage ran. The OPSD/MOPD mechanism remains untested here.
  • Durable design lesson: calibrate theoretical pass-one headroom independently for every mandatory teacher before sampling baselines or producing teachers. Any replacement tools core or split is a new preregistered experiment; the current threshold and outcome are not amended.

Data files 3

Result tables and metrics copied from the experiment folder — preview inline or open the raw file.

Reproduce

Smoke test

python3 experiments/qwen35_4b_specialist_policy_integration/scripts/run.py --smoke

Full run

see the staged Run section in README.md

Run steps are documented inside the experiment folder (README and scripts).

Browse the experiment folder on GitHub ↗