Qwen3.5-4B Verified Macro Invention Long-Context Rerun

Results at a glance 2

Overview

plan-given interface passed 16/16 records at think@16,384, overturning the parent's apparent low-budget interface limit. The fresh base-smoke arm then proved that 16,384 was still binding for induction (131/144 unresolved contacts and 60/144 answer truncations), so those rows were excluded and think@32,768 also rejected from content-blind finish/count metadata plus the preregistered token-ID periodicity test: 81/144 exact loops, 63/144 unresolved contacts, and 37/144 answer-limit contacts. The max-seqs-64 K=4 probe at think@49,152 then finished with 34/48 exact loops, 14/48 unresolved contacts, and 13/48 answer-limit contacts, but amendment 12 had already made it diagnostic-only from pre-result cache-capacity evidence. Fresh long-rung selection moved to the capacity-fit follow-up, whose 61k attempt was stopped before a receipt after an audit found that its implicit CUDA-graph list did not cover the active batch width. A separate exact-capture follow-up then rejected fresh 49k and 61k probes even though both live-KV and exact-graph gates passed. Its terminal selector records pass=false and no selected budget, so no K=12 matrix, semantic analysis, or macro claim is authorized.

Research program

• Primary: operator_and_skill_inventories
• Secondary: structured_execution_and_compilers, benchmark_generalization, and test_time_reasoning_budget
• Parent: qwen35_4b_verified_macro_invention at commit 1c8c5bbb81d2a67618891597205ceb2f40f498d8
• Intake: idea_intake.md
• Preregistration: reports/preregistration.md
• External scientific-smoke durability amendment: reports/preregistration_amendment_10.md
• Fail-early full-run audit amendment: reports/preregistration_amendment_11.md
• Capacity-fit scheduler amendment: reports/preregistration_amendment_12.md
• Adversarial design review: reports/design_review.md
• Frozen-data provenance: data/source_provenance.json

Question

With enough context and a nonbinding, explicitly calibrated reasoning allowance, can a verified abstraction library built only from prior solved programs improve visible-only selected accuracy on fresh behaviorally true-depth-5 programs beyond matched-compute sampling over the original primitives?

Why this rerun exists

The parent did not reach the scientific comparison. Its budgeted plan-given attempt gave every completion only 768 thinking tokens and a 128-token answer: all 16 samples hit the thinking cap and 12/16 hit the answer cap. A later no-thinking transcription probe removed truncation but measured a different low-compute interface. Neither path establishes that verified macros help or fail under a properly provisioned induction protocol.

This follow-up changes the inference envelope, not the hypothesis. It keeps the parent's frozen construction corpus, macro libraries, never-prompted v2 smoke, never-prompted full tasks, scientific arms, K values, analyzer logic, hidden-label boundary, and confirmatory thresholds. It replaces the single 768-token assumption with content-blind termination calibration and whole-matrix escalation.

Setup

• Only model: Qwen/Qwen3.5-4B at repository-pinned revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a.
• Only inference backend: the experiment-local src/vllm_runner.py under .venv-vllm; no Transformers inference and no backend mixing.
• Engine envelope: max_model_len=65536, with exact prompt-plus-generation preflight.
• Scientific reasoning ladder: budgeted thinking at 16,384, 32,768, 49,152, then 61,440 tokens after amendments 3--5 separated exact loops and extended the setup envelope.
• Answer allowance: 512 tokens at every rung. Forced-close stage 2 is directly capped; a naturally closed stage-1 answer reaching 512 also rejects the rung under amendment 7.
• Reasoning-cap accounting: amendment 9 counts a forced intervention or a close in the final stage-1 slot as contact. An earlier natural </think> followed by a stage-1 length finish is an answer-restart diagnostic, not evidence that reasoning exhausted its allowance; the fresh answer still has to clear the unchanged 512-token rule.
• Largest context guard: the frozen train-only proposal prompt is 3,478 tokens. Its largest-rung total is 65,432, and the CPU regression guard preserves the remaining 104-token headroom.
• Substrate: the byte-identical contamination-free procedural corpus copied from the parent: 800 construction programs, 12 unseen v2 smoke tasks, and 120 unseen full tasks (80 motif-reuse, 40 primitive-multiset-matched no-reuse).
• Sampling: macro arms K=12; base K=24 for the matched-token sample-more curve.
• Primary metric: visible-only selected hidden-all accuracy on the reuse split.
• Oracle-only metric: whether any sampled candidate passes every hidden case.

The frozen tasks.json SHA-256 is 82fbbd57e26fd392aa8f30ec6f26d370dc08dd78b3279bed6ee2e2174aea5073; the frozen libraries.json SHA-256 is a2ae3663753a3a0d0c9614a5d7c1d250506c74fd7879e11e99b66f5c1e43f865.

Anti-censoring gates

1. The original train-only calibration ladder used four deterministic plan-given records sampled 16 times each. Amendment 4 selected think@16,384 using finish/count metadata plus the frozen token-ID periodicity test: below-5% unresolved-cap and answer-limit-contact rates, at most 80% p99 productive thinking use, and at most 50% p99 answer use. Generated text and correctness were unavailable to selection.
2. At that rung, a disjoint train-only plan-given gate ran 16 records with four samples each. It passed 16/16 record coverage; 63/64 samples were strict valid macro-using surfaces, all 12 cap contacts were classified by the frozen exact-token periodic-tail detector, none remained unresolved, and no answer truncated.
3. Fresh induction has its own workload-conditioned ladder. A completed inadequate arm can reject a lower rung immediately; every remaining arm is recorded as skipped, and all lower rows remain diagnostic and unscored. A rung is selectable only when the complete base/designed matrix is termination-adequate at one shared budget. If either completed K=12 arm rejects the 32,768 matrix, higher rungs first use all 12 base prompts at non-scored K=4 to locate a viable allowance; the selected rung is then rerun from scratch at the original complete K=12 matrix.
4. Raw stage-1 length, forced intervention, final-slot boundary contact, earlier-close answer restart, and fresh-answer limit contact are reported separately. Runtime selection and offline analysis use the same amendment-9 definition.
5. Exhausting 61,440 is an inconclusive setup failure, never a capability result. Full generation remains blocked until an uncensored smoke matrix passes the original semantic gates.
6. Amendment 12 found before the 49k receipt that max-seqs 64 overcommits the measured 995,328-token KV cache at the long rungs. That completed probe is decision-ineligible regardless of its termination result. Fresh 49k/61k probes and any complete matrix move to the separate capacity-fit follow-up at max-seqs 19/15 respectively.

The full decision remains the parent's conjunction: mined must beat base sampling, the non-callable mined_hint, and matched random composite libraries under the registered effect-size, paired-interval, compute, macro-use, and reuse-specificity thresholds. The qwen_ranked verdict remains separate.

Run

All commands use the uv-managed vLLM environment.

CPU preparation and tests:

.venv-vllm/bin/python -m unittest discover -s experiments/qwen35_4b_verified_macro_long_context_rerun/tests -v
.venv-vllm/bin/python experiments/qwen35_4b_verified_macro_long_context_rerun/scripts/run.py --prepare

Before resuming a legacy repository-local scientific-smoke cache, stop the old runner and perform the model-free staged migration. The first command validates a staging copy, writes receipts and a deterministic catalog, atomically installs the external tree, and deliberately preserves the local source for review. The second command revalidates byte identity before removing only the canonical local tier/probe directories. Cleanup is resumable: any remaining local bundles need only be an exact verified subset of an external inventory that may already have advanced:

.venv-vllm/bin/python experiments/qwen35_4b_verified_macro_long_context_rerun/scripts/run.py --migrate-scientific-artifacts
.venv-vllm/bin/python experiments/qwen35_4b_verified_macro_long_context_rerun/scripts/run.py --migrate-scientific-artifacts --remove-local-scientific-artifacts

Scientific matrix and termination-probe bundles then live under /workspace/large_artifacts/qwen35_4b_verified_macro_long_context_rerun/scientific_smoke_v1/. QWEN35_MACRO_SCIENTIFIC_ARTIFACT_ROOT may select a byte-identical absolute copy on another host. Do not hand-copy individual JSONL files: each bundle is valid only as a preflight alone or as the exact preflight/rows/metadata triplet plus its last-written receipt. Unknown files, symlinks, partials, or catalog drift fail closed before model allocation or row interpretation. A narrow crash window is self-healing under the shared stage lock: when unchanged protocol bytes and exact-valid external bundles prove that a cataloged preflight merely gained its last-written receipt, the catalog is reconciled before model allocation. Source drift, unknown bundles, and rows/metadata partials are never reconciled.

Calibrate the rung, run the heldout interface gate, and run the fresh smoke:

.venv-vllm/bin/python experiments/qwen35_4b_verified_macro_long_context_rerun/scripts/run.py --smoke

Run the full matrix only after all gates pass:

.venv-vllm/bin/python experiments/qwen35_4b_verified_macro_long_context_rerun/scripts/run.py --full

Full generation follows preregistration amendments 8 and 11 and does not create runs/full/*.jsonl. Canonical raw artifacts live at /workspace/large_artifacts/qwen35_4b_verified_macro_long_context_rerun/full/think_<budget>/<arm>/shard_<index>/. Base uses 20 six-task shards at K=24; every macro arm uses 10 twelve-task shards at K=12. Each shard therefore contains exactly 144 completions and preserves the frozen 2:1 reuse mix. A final shard is reusable only when its directory contains exactly preflight.json, rows.jsonl, runner.meta.json, and the last-written receipt.json, and every ordered prompt, identity, byte size, and SHA-256 validates. Re-running --full first scans and exact-validates every existing final in the active rung, including downstream arms, and only then resumes missing work. .tmp-* directories from interruptions are inventoried but never salvaged.

A persistent sibling .full.lock serializes scientific migration, smoke, and full processes. It uses nonblocking flock and is acquired before frozen-data checks, smoke replay, proposal work, model construction, or external-root/catalog mutation. Before vLLM loads, the runner also rejects repository-local full raw rows, symlinks, unknown budgets/arms/shards, malformed finals, partial Qwen ensembles, and stale protocol/data/catalog bindings. A malformed final shard_<index> fails closed and is never overwritten. Inspect it, move the whole directory to an explicitly named quarantine location outside the canonical arm directory, then rerun the entire shard. Do not copy individual rows or samples back. At a successful selected rung, analysis/full_artifact_catalog.json is the tracked logical pointer; no raw promotion copy is made. Its schema-2 inventory is checkpointed from startup onward, including diagnostic receipts, nonreusable temporary directories, rejected rungs, and a selected-null setup-inconclusive finish. It binds the exact config, hidden-label-bearing tasks, post-proposal libraries, smoke gates, selection, source files, plan, receipts, and protocol identities. Offline analysis revalidates that binding and recomputes selected-arm termination adequacy before scoring any completion.

Analyze an existing full run without loading the model:

.venv-vllm/bin/python experiments/qwen35_4b_verified_macro_long_context_rerun/scripts/analyze.py --run full

Results

The inference repair has already changed the diagnosis:

• max-seqs-64 calibration at think@16,384: 3/64 unresolved contacts, 0 answer truncations, productive p99 11,629 tokens;
• independent plan-given interface: 16/16 records covered, 63/64 strict valid macro-using samples, 0 unresolved contacts, and 0 answer truncations;
• fresh base smoke at the same allowance: 144/144 raw cap contacts, 13 exact periodic tails, 131/144 unresolved contacts, and 60/144 answer-limit contacts;
• fresh base smoke at think@32,768: 144/144 forced interventions at the reasoning boundary, 81 exact periodic tails, 63/144 unresolved contacts, 37/144 answer-limit contacts, and no earlier-close answer restarts.

The max-seqs-64 K=4 diagnostic at 49,152 also forced all 48 reasoning traces closed: 34 were exact loops, 14 remained unresolved, and 13 answers hit 512 tokens. It sampled 2,366,620 tokens in 4,035.356 seconds (586.47 tokens/s). Amendment 12 had made it selection-ineligible before its receipt existed because 48 admitted long sequences required 2,433,024 cache tokens against the 995,328-token engine cache with prefix caching off.

These are workload/scheduler-calibration failures, not macro results. The 16k arm consumed 2,391,698 sampled tokens in 2,138.606 seconds (1,118.34 tokens/s); 32k consumed 4,739,527 in 5,971.182 seconds (793.73 tokens/s). All three diagnostics were excluded before parsing or correctness inspection. The capacity-fit follow-up then rejected a fresh 49k probe but exposed a second scheduler defect: its requested capture maximums 19/15 resolved only through widths 16/8. Its 61k attempt was stopped before a receipt, with no reusable rows. The separate exact-capture follow-up covered width 19 exactly at 49k while fitting 963,072 cache tokens into 996,864 live tokens, but still rejected the probe content-blind: 38/48 exact loops, 10/48 unresolved contacts, and 6/48 answer-limit contacts. It sampled 2,363,163 tokens in 4,809.081 seconds (491.396 tokens/s). At 61k, vLLM resolved [1, 2, 4, 8, 15] exactly in FULL mode and fit 950,400 required cache tokens into 997,888 live tokens with 47,488 of headroom. The fresh probe nevertheless failed all three thresholds: 40/48 exact loops, 8/48 unresolved contacts, and 4/48 answer-limit contacts, at 397.688 sampled tokens/s. The terminal selection is pass=false with no selected budget. No decoded or scored content informed either decision, and no K=12 matrix or macro comparison was authorized. The next registered follow-up should stop increasing context and test a symmetric loop-control protocol in a separate experiment while preserving the unresolved-contact and answer-limit gates.

Artifacts

• data/: byte-identical frozen parent inputs plus explicit source provenance.
• configs/: the registered ladder, context envelope, sampling, gates, and scientific constants.
• runs/: train-only calibration and interface vLLM rows with exact runner metadata. Scientific smoke/probe rows use the canonical external root above; full rows use the external shard root.
• analysis/: machine-readable gates/verdicts, budget selections, and deterministic external checksum catalogs. Smoke and full per-task output is compact: hashes, aggregate counts/tokens, and the selected program/grades only; reasoning text and raw token arrays stay external.
• reports/: preregistration, design review, final report, and artifact manifest.
• src/vllm_runner.py: the single-file, pinned vLLM inference wrapper used by every arm.

Report

Rendered from reports/report.md

Summary

This follow-up has repaired the parent's apparent interface failure but has not yet produced an eligible scientific comparison. Under the uv-managed vLLM stack, the disjoint plan-given interface passed 16/16 records at think@16,384. The fresh induction base arm at that same allowance then hit a different termination regime: 131/144 samples remained unresolved after exact-loop exclusions and 60/144 answer stages truncated. Those rows were rejected before parser or correctness inspection. At think@32,768, all 144 samples still required forced closure: 81 exact loops, 63 unresolved contacts, and 37 answer-limit contacts. That rung was also excluded. The max-seqs-64 K=4 probe at think@49,152 finished with 34/48 loops, 14/48 unresolved contacts, and 13/48 answer-limit contacts, but pre-result KV-capacity evidence had already made it diagnostic-only. Fresh selection is running through separate scheduler follow-ups. The capacity-fit 61k attempt stopped before a receipt when its implicit CUDA-graph list proved not to cover the active width. Fresh exact-capture 49k and 61k probes then failed termination despite passing both live-KV and exact-graph gates. The terminal selector is pass=false with no selected budget, so no K=12 matrix or semantic analysis was authorized. No capped row is a negative macro result.

Research Program Fit

Primary program: operator_and_skill_inventories. The experiment also supplies direct evidence to test_time_reasoning_budget: a budget calibrated on supplied-plan transcription did not transfer to fresh program induction in the same procedural substrate. Structured execution and benchmark generalization become relevant only after an uncensored mined/base/hint/random comparison exists.

Method

The only model is Qwen/Qwen3.5-4B at revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a. Every model-facing request uses the experiment-local single-file vLLM runner; Transformers inference and backend mixing are forbidden. Inputs are copied byte-for-byte from parent commit 1c8c5bbb81d2a67618891597205ceb2f40f498d8: 800 train-only construction programs, 12 fresh v2 smoke tasks, and 120 never-prompted full tasks with exact execution and behaviorally verified depth five.

The original calibration and independent interface gate used supplied train-only plans. Scientific smoke uses fresh I/O-only induction tasks and compares base primitives with the generator-known designed macro ceiling at matched K=12. Amendments 1--12 record scheduler diagnostics, the Ada batch-invariance boundary, exact periodic-loop handling, concurrency optimization, the extended scientific ladder [16384, 32768, 49152, 61440], exact cap accounting, external scientific/full durability, and fail-early full-run audit hardening. If either completed K=12 arm rejects the 32,768 matrix, the two higher rungs first use non-scored K=4 base workload probes to locate a viable allowance. Lower rungs and probes can be rejected from content-blind finish/count metadata plus the preregistered token-ID periodicity test, but a rung is scored only when both arms are complete at K=12 and adequate at the same budget. Amendment 12 additionally freezes max-seqs 19 at 49k and 15 at 61k in a separate protocol because the measured cache cannot hold the former max-seqs-64 scheduling wave without recomputation.

Results

Corrected setup and interface

The interface produced 63/64 strict valid macro-using samples; every one of its 16 records had at least one exact optimal rewrite. Its 12 raw cap contacts were all classified by the frozen exact-token periodic-tail detector, so none remained unresolved.

The fresh base arm sampled 2,391,698 tokens in 2,138.606 seconds (1,118.34 tokens/s). All 144 samples reached 16,384; 13 tails passed the periodic-loop detector and 131 did not. Because base alone irreversibly rejected the rung, the automatically started designed arm was interrupted before it returned rows. No decoded output, parser status, visible score, hidden grade, or oracle result informed the amendment or escalation.

At 32,768, the amendment-9 replay found no earlier-close answer restarts: every sample reached the reasoning boundary and required intervention. Eighty-one tails were exact periodic loops, while 63 remained unresolved; 37 fresh answers hit 512 tokens. The arm sampled 4,739,527 tokens in 5,971.182 seconds (793.73 tokens/s). Base alone again rejected the rung without a designed-arm call, so the precommitted branch advanced to the non-scored K=4 base probe at 49,152. This decision also used no decoded output, parser result, visible score, hidden grade, or oracle result.

Before that probe returned, amendment 12 proved from engine capacity and installed scheduler source that max-seqs 64 could not hold its long active contexts without eviction/recomputation. The probe therefore became irrevocably decision-ineligible. It later confirmed 48/48 forced interventions, 34 exact periodic loops, 14 unresolved contacts, and 13 answers at 512 tokens. Sampling 2,366,620 tokens took 4,035.356 seconds (586.47 tokens/s). The receipt-bound diagnostic was classified from finish/count metadata and token IDs only; no decoded or scored content was inspected. Fresh budget selection moved to qwen35_4b_verified_macro_capacity_fit_rerun at max-seqs 19/15.

That capacity-fit follow-up completed its fresh 49k K=4 probe within live KV capacity, but all 48 samples still contacted the reasoning boundary: 37 were exact loops, 11 remained unresolved, and 9 answers reached their limit. It generated 2,364,643 sampled tokens in 5,012.451 seconds (471.754 tokens/s). A source/runtime audit then found that the implicit CUDA-graph configuration resolved only through widths 16 and 8 for requested maxima 19 and 15. The 61k capacity-fit attempt was therefore stopped before a receipt; it produced no reusable rows and no decoded or scored content was inspected.

The separate qwen35_4b_verified_macro_exact_cudagraph_rerun froze explicit graph lists and fresh artifacts. At 49k, vLLM resolved [1, 2, 4, 8, 16, 19] exactly and the live audit fit 963,072 required cache tokens into 996,864, leaving 33,792. The fresh probe nevertheless failed all three termination thresholds: 38/48 exact loops, 10/48 unresolved contacts, and 6/48 answer-limit contacts. It generated 2,363,163 sampled tokens in 4,809.081 seconds (491.396 tokens/s), a descriptive 4.16% improvement over the closest implicit-capture probe. This is not a causal throughput benchmark because scheduling can alter sampled trajectories.

The terminal 61k probe also passed its runtime envelope: vLLM resolved FULL decode graphs at [1, 2, 4, 8, 15], covering max-seqs 15 exactly, and fit 950,400 required cache tokens into 997,888 with 47,488 of headroom. All 48 samples still reached the reasoning boundary: 40 were exact loops, 8 remained unresolved, and 4 answers reached 512 tokens. It generated 2,951,995 sampled tokens in 7,422.886 seconds (397.688 tokens/s). The terminal selection therefore records pass=false and selected_thinking_budget=null; no K=12 arm, semantic analysis, or macro result is authorized. Neither probe decision used decoded or scored content.

Scientific smoke and full comparison

Pending an uncensored complete rung. Full generation remains blocked by the unchanged smoke gate.

Controls

• Byte-level frozen-data verification and zero forbidden train/evaluation overlap.
• Only one model, revision, backend, prompt family, and shared rung per comparison matrix.
• Content-blind finish/count and preregistered token-ID periodicity decisions; no decoded semantics or scores enter budget selection, and lower rungs are never pooled or scored.
• Exact periodic-tail classification reported separately from unresolved cap contact.
• Stage-independent 512-token answer-limit contact; natural answers cannot bypass the stage-2 cap.
• Amendment-9 reasoning-cap accounting separates forced intervention and final-slot close from an earlier natural close whose partial answer was merely restarted after the raw stage-1 length cap.
• A model-free proposal-record/hash guard freezes the 3,478-token proposal bound, 65,432-token largest-rung total, and at least 104 tokens of context headroom; exact runtime preflight remains mandatory.
• Strict one-line parser, visible-only selection, and hidden-only final grading remain unchanged.
• Base/design smoke precedes all mined, hint, random, Qwen-ranked, and full-task generation.

Oracle Versus Deployable Evidence

Not yet eligible. Oracle and selected correctness are deliberately unavailable until the complete smoke matrix passes termination. If it does, full results will report hidden oracle coverage separately from the visible-only selected line and from the matched-token sample-more base prefix.

Interpretation

The parent's 768-token result was not a durable alias-interface limit: adequate reasoning cleared the broader K=4 record-level gate. The new roadblock is workload-conditioned provisioning. Supplied-plan transcription and I/O-only induction share the same syntax and model but have radically different trace-length distributions. The operational lesson is to calibrate on the actual workload class, retain exact termination evidence, and treat a generation ceiling as missing evidence rather than task failure.

This still says nothing about whether mined macros improve induction. That claim requires the unchanged full conjunction against base sampling, non-callable hints, matched random libraries, macro mediation, and the no-reuse control.

Next Experiments

Stop increasing the context allowance: the exact-capture ladder has reached its registered terminal rung without selecting a budget. The next attempt should be a separate, preregistered symmetric loop-control experiment that applies the same intervention to every compared arm while preserving the unresolved-boundary and answer-limit gates. Do not reuse censored rows, decode them to tune the intervention, or claim a macro effect without a fresh termination-adequate K=12 matrix.

Artifact Manifest

Raw superseded scheduler/context diagnostics are preserved under the standard external artifact root with a deterministic tree checksum. Current compact audits, config, preregistration amendments, metadata, and reasonably sized calibration/interface rows are listed in artifact_manifest.yaml. Raw scientific smoke and full rows live only in their canonical external roots and are bound by tracked deterministic catalogs and per-file receipt hashes; repository-local promotion copies are forbidden.

Experiment log 10

Reproduce

Smoke test

.venv-vllm/bin/python experiments/qwen35_4b_verified_macro_long_context_rerun/scripts/run.py --smoke

Full run

.venv-vllm/bin/python experiments/qwen35_4b_verified_macro_long_context_rerun/scripts/run.py --full

Run steps are documented inside the experiment folder (README and scripts).

Browse the experiment folder on GitHub ↗

Related

Related experiments

Nearest by shared name, tags, and programs.

• 2026-07-10 Qwen3.5-4B verified-macro exact CUDA-graph vLLM rerun
• 2026-07-10 Qwen3.5-4B verified-macro capacity-fit vLLM rerun
• 2026-07-09 Qwen3.5-4B Verified Macro Invention
• 2026-07-11 Interactive policy curriculum: oracle DAgger to execution-reward RL
• imported 2026-07-12 Qwen 3.5 4B Verified Edit Closure
• 2026-06-23 Qwen Context-Conditioned Trace Verifier