Qwen3.5-4B verified-macro exact CUDA-graph vLLM rerun

Finished2026-07-10Evidence-Conditioned Selection Interpretability and Diagnostics Operator and Skill Inventories Reliability and Safety+2 moreGitHub ↗

The one idea you need

Picture the model working through a problem on scratch paper before it answers. On these tasks it kept rewriting the same few lines instead of concluding. Hand it a bigger pad and it just fills more pages with the identical loop.

The question

If a small model runs out of room to think before answering, does giving it more room help it finish, or does it just keep looping?

What we found

No, it just loops harder. At both budgets, all 48 reasoning attempts ran out of thinking room and had to be force-stopped, none finishing on its own. Widening the budget from 49,152 to 61,440 tokens pushed the share stuck endlessly repeating the same lines from 79% up to 83%. More space feeds the loop rather than breaking it.

Why it matters

Before enlarging a small reasoning model's thinking budget, measure how often its runs end by repeating themselves. If they loop, more budget just burns compute; the real fix is anti-repetition decoding or training, not more room.

Attempts that finished thinking on their own0 of 48every sample hit the reasoning limit and was force-stopped, at each budget

Stuck endlessly repeating the same lines79% → 83%share of samples in exact loops as the budget rose from 49,152 to 61,440 tokens

Loop rate still allowed to pass25% at mostregistered ceiling; the actual rate ran past three times higher

Speed gain from the engineering fixabout 4% fasterpre-recording the exact GPU batch shapes vs letting the engine guess them

On this page

Results at a glance
Overview
Report
Experiment log
Reproduce
Related

Results at a glance 2

Spelling out batch sizes covered every width the model used

How to read

Four inference setups along the bottom: two thinking budgets, each run once with the engine guessing fast-path batch sizes and once with them spelled out. Per setup, one bar is the width actually used, the other the largest width pre-built; matching them is better.

Takeaway → Guessing pre-built only up to widths 16 and 8, below the 19 and 15 actually used; spelling them out matched exactly, closing the slow uncovered gap.

Data table

condition	active batch width	largest resolved CUDA graph
implicit 49k	19	16
exact 49k	19	19
implicit 61k	15	8
exact 61k	15	15

Numbers from ../qwen35_4b_verified_macro_capacity_fit_rerun/analysis/scientific_smoke_49k_termination_audit.json; analysis/scientific_smoke_49k_termination_audit.json; analysis/scientific_smoke_61k_termination_audit.json

Technical framing

Explicit graph lists cover every registered active width — Pinned vLLM's implicit sparse lists missed active widths 17-19 at 49k and 9-15 at 61k. Explicit lists resolved exactly through widths 19 and 15 in both completed fresh probes. The 61k run also fit 950,400 required KV tokens below 997,888 live tokens.

More thinking room raised the share of looping runs

How to read

Two thinking budgets side by side, each over 48 attempts. Three colored shares per budget: attempts stuck repeating a block, attempts hitting the limit for other reasons, and attempts running out of final-answer space. Lower is better on all three.

Takeaway → Moving to the bigger budget nudged the looping share up from 79% to 83% — extra room deepened the loop instead of resolving it.

Data table

condition	unresolved boundary contacts	exact periodic loops	answer-limit contacts
exact 49k	20.8%	79.2%	12.5%
exact 61k	16.7%	83.3%	8.3%

Numbers from analysis/scientific_smoke_49k_termination_audit.json; analysis/scientific_smoke_61k_termination_audit.json; analysis/smoke_budget_selection.json

Technical framing

More context increased loops without clearing termination — At 49k, the fresh K=4 probe recorded 38 loops, 10 unresolved contacts, and 6 answer-limit contacts; at terminal 61k it recorded 40, 8, and 4. Registered maxima for unresolved, loops, and answer-limit shares are 5%, 25%, and 5% respectively, so both rungs failed every threshold. The terminal selector is pass=false with no selected budget; no decoded or scored content was inspected.

In the author’s words from the Report · “Summary”

Termination result: both fresh exact-capture K=4 probes passed their infrastructure gates but failed all three registered content-blind termination thresholds. The 61k rung is terminal: selection records pass=false with no selected budget. No K=12 arm, decoded or scored inspection, semantic analysis, or macro result is eligible. The experiment isolates a concrete inference defect: the capacity-fit predecessor requested CUDA graph maxima 19 and 15 but relied on vLLM's implicit sparse shape list, which resolved only through 16 and 8. This follow-up supplies explicit lists ending at the active widths and fails unless the constructed engine resolves them exactly under a full-decode CUDA-graph mode.

Overview

Status: terminal setup negative. Both fresh exact-capture K=4 probes are complete and termination-inadequate. No K=12 matrix, semantic analysis, or macro result is eligible, and no decoded or scored content has been inspected.

Research program

Primary: operator_and_skill_inventories.
Secondary: structured_execution_and_compilers and test_time_reasoning_budget.
Closest near-duplicate: qwen35_4b_verified_macro_capacity_fit_rerun.

This is a separate experiment because explicit CUDA-graph capture shapes change the vLLM inference protocol on Ada. No predecessor output may be imported, pooled, promoted, or used to skip a rung.

Question

The capacity-fit rerun correctly limited concurrent near-65k contexts to the live KV cache, but its runner supplied only max_cudagraph_capture_size=max_num_seqs. Pinned vLLM's implicit sparse list resolved a requested maximum of 19 to 16 and 15 to 8. Does explicitly capturing through the exact active widths remove that avoidable eager-dispatch region while preserving the scientific gates?

This experiment does not assume the answer is yes. CUDA-graph warmup, padding, memory, and changed termination trajectories can erase or reverse a throughput benefit.

Frozen protocol

Model: only Qwen/Qwen3.5-4B, revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a.
Backend: only the experiment-local vLLM runner, schema 4, SHA-256 3a98eb8da787054aded56a1ec3fd040ee2edaacc7d0694b4aec5a0309488774a.
49k rung: thinking budget 49,152, max_num_seqs=19, capture sizes [1,2,4,8,16,19].
61k rung: thinking budget 61,440, max_num_seqs=15, capture sizes [1,2,4,8,15].
Engine: max_model_len=65536, bf16, tensor parallel 1, gpu_memory_utilization=0.9, max_num_batched_tokens=32768, prefix caching off, asynchronous scheduling off.
Sampling: temperature 0.6, top-p 0.95, top-k 20, answer allowance 512, seed 2701.
Probe: fresh base K=4 and termination-only. Selectable matrix: fresh base and designed K=12 at the first adequate rung.

Each constructed engine must independently pass two checks before generation:

live block-rounded KV demand fits the exposed cache capacity; and
vLLM's resolved capture-size list and maximum exactly equal the registered values, with decode_mode=FULL, full CUDA graphs enabled, and the full active width covered.

Setup with `uv`

From the repository root, create the pinned vLLM environment only if it is absent:

uv venv --python 3.12 .venv-vllm
uv pip sync --python .venv-vllm/bin/python --torch-backend=cu129 requirements-vllm.lock.txt
uv pip check --python .venv-vllm/bin/python

Model-free gates:

.venv-vllm/bin/python -m unittest discover \
  -s experiments/qwen35_4b_verified_macro_exact_cudagraph_rerun/tests -v
.venv-vllm/bin/python \
  experiments/qwen35_4b_verified_macro_exact_cudagraph_rerun/scripts/run.py --validate

GPU runbook

Do not launch until the current GPU owner and an independent design reviewer give GO. Every command below creates one engine, invokes one experiment phase, commits one bundle, and exits; it never automatically advances.

Start with the fresh 49k probe:

.venv-vllm/bin/python \
  experiments/qwen35_4b_verified_macro_exact_cudagraph_rerun/scripts/run.py \
  --phase probe --budget 49152

If and only if its content-blind termination gate is adequate, run the new K=12 base arm:

.venv-vllm/bin/python experiments/qwen35_4b_verified_macro_exact_cudagraph_rerun/scripts/run.py \
  --phase base --budget 49152

If and only if that K=12 base arm is also termination-adequate, run the designed arm:

.venv-vllm/bin/python experiments/qwen35_4b_verified_macro_exact_cudagraph_rerun/scripts/run.py \
  --phase designed --budget 49152

The completed 49k probe authorized one separately fresh 61k probe, which was run as:

.venv-vllm/bin/python \
  experiments/qwen35_4b_verified_macro_exact_cudagraph_rerun/scripts/run.py \
  --phase probe --budget 61440

The 61k probe also failed, so the ladder is now terminal. The following analysis command was never authorized because no K=12 matrix was selected:

.venv-vllm/bin/python \
  experiments/qwen35_4b_verified_macro_exact_cudagraph_rerun/scripts/analyze.py

Decision rule

Termination selection uses counts and finish metadata plus token IDs only for the frozen periodic loop detector. It never decodes or scores while selecting a rung. Adequacy requires unresolved cap contacts below 5%, answer-limit contacts below 5%, and periodic-loop contacts at most 25%.

Semantic smoke remains an interface gate: parse rate at least 0.5 in each K=12 arm, valid macro candidates on at least two reuse tasks, and designed reuse oracle coverage no lower than base. A positive smoke does not beat matched-compute sampling and cannot support a macro capability claim.

49k probe result

The completed fresh K=4 probe passed both infrastructure gates. Its live engine exposed 996,864 KV tokens in 528-token blocks; 19 block-rounded 50,688-token reservations required 963,072 tokens and left a 33,792-token margin. vLLM resolved the registered full-decode graph list exactly as [1,2,4,8,16,19], including the active width 19.

All 48 samples contacted the frozen reasoning boundary and required force-close. The token-only periodicity audit found 38 exact loops; 10 contacts remained unresolved and six answer stages reached their limit. The corresponding rates---79.17%, 20.83%, and 12.50%---fail all three registered thresholds, so 49k was rejected before decoding or scoring. At that checkpoint the fresh exact-capture 61k probe was authorized; its terminal result is recorded below.

The probe sampled 2,363,163 tokens in 4,809.081 seconds (491.396 sampled tokens/s), including late JIT warnings in elapsed generation time. That is 4.16% faster than the closest implicit-capture capacity-fit probe and 16.21% slower than the invalidly overcommitted max-seqs-64 diagnostic. These cross-protocol timings are descriptive rather than a causal benchmark. The result is termination and inference evidence only, with no macro claim.

61k probe and terminal result

The separately fresh K=4 probe at 61,440 passed both infrastructure gates. Its live engine exposed 997,888 KV tokens in 528-token blocks; 15 block-rounded 63,360-token reservations required 950,400 tokens and left a 47,488-token margin. vLLM resolved full-decode CUDA graphs exactly as [1,2,4,8,15], covering the active width.

All 48 samples again contacted the frozen reasoning boundary and required force-close. The content-blind token-ID audit found 40 exact periodic loops, eight unresolved contacts, and four answer stages at their limit. Their rates---83.33%, 16.67%, and 8.33%---all fail the registered thresholds. The run sampled 2,951,995 tokens in 7,422.886 generation seconds (397.688 sampled tokens/s), after a 100.294-second load.

The terminal selection therefore records pass=false and selected_thinking_budget=null. No K=12 base or designed arm exists; decoding, scoring, and semantic analysis remain prohibited. This is a preserved negative about the setup: exact CUDA-graph coverage and a 61k reasoning allowance did not produce termination-adequate samples. It is not a negative macro-capability result.

Artifacts

idea_intake.md: novelty, closest duplicate, controls, and falsifier.
reports/preregistration.md: frozen geometry, branch logic, and inspection boundary.
reports/design_review.md: adversarial risks and required controls.
data/source_provenance.json: exact copies, derived-runner hash, and noninheritance boundary.
src/vllm_runner.py: single-file vLLM wrapper with explicit/resolved capture validation.
src/scientific_artifacts.py: fail-closed external receipts, catalogs, and selection.
scripts/run.py: one-engine/one-phase orchestrator and live preflight.
scripts/analyze.py: three-pass termination-then-semantics analyzer.
reports/artifact_manifest.yaml: fresh external namespace and regeneration contract.
analysis/scientific_smoke_49k_termination_audit.json: content-blind capacity, graph, termination, and timing audit for the rejected 49k probe.
analysis/scientific_smoke_61k_termination_audit.json: content-blind audit for the rejected terminal 61k probe and fail-closed selection.

Report

Rendered from reports/report.md

Summary

Termination result: both fresh exact-capture K=4 probes passed their infrastructure gates but failed all three registered content-blind termination thresholds. The 61k rung is terminal: selection records pass=false with no selected budget. No K=12 arm, decoded or scored inspection, semantic analysis, or macro result is eligible.

The experiment isolates a concrete inference defect: the capacity-fit predecessor requested CUDA graph maxima 19 and 15 but relied on vLLM's implicit sparse shape list, which resolved only through 16 and 8. This follow-up supplies explicit lists ending at the active widths and fails unless the constructed engine resolves them exactly under a full-decode CUDA-graph mode.

Research program fit

The scientific target remains reusable composite operators, so the primary program is operator_and_skill_inventories. This variant repairs the inference protocol needed to make the verified-macro smoke interpretable; it is not a new macro mechanism.

Method

Only pinned Qwen/Qwen3.5-4B through vLLM.
Frozen 12-task smoke-v2 inputs and identical sampling/interface rules.
Live KV-safe max-seqs 19 at 49k and 15 at 61k.
Explicit CUDA-graph lists [1,2,4,8,16,19] and [1,2,4,8,15].
Post-construction equality check against vLLM's resolved compilation config, including decode_mode=FULL and full CUDA graphs enabled.
Fresh K4 termination probe at each reached rung; no probe is semantic evidence.
Fresh same-rung base/designed K12 matrix only after a passing probe.
New fail-closed external namespace, receipt-last writes, and content-blind selection.

Results

Infrastructure gates

The independent reviewer gave prelaunch GO against frozen binding 9d2692c6acad35d3b7ab56ddf368c9974c1ddaf6e0a06997b01015c0de397158 before engine construction. The exact 49k invocation then passed both live checks:

996,864 live KV tokens with 528-token cache blocks;
963,072 block-rounded tokens required by max-seqs 19, leaving 33,792 tokens;
resolved decode_mode=FULL, full CUDA graphs enabled; and
requested and resolved capture sizes exactly [1,2,4,8,16,19].

The receipt SHA-256 is 61da6f616365bf080e97f341bd0c2305b889998c4d161c61e63d06e5dfb5923c. The complete external four-file tree is 30,653,162 bytes with SHA-256 654d44119fc46fe83428c154680ee502073c00022ccb9cdb0922c1896ca20685.

After the content-blind 49k rejection, the separately fresh 61k invocation passed the same gates:

997,888 live KV tokens with 528-token cache blocks;
950,400 block-rounded tokens required by max-seqs 15, leaving 47,488 tokens; and
requested and resolved full-decode capture sizes exactly [1,2,4,8,15].

Its receipt SHA-256 is 8f00535a773e347ec4f90a48eb6b00960935d7010c3500af7ccdf57fedd6f2e1. The final external eight-file tree is 70,191,578 bytes with SHA-256 4aa311ab579c301f2b2d7383591e3e68ed66035d184bbaaa2659d59bd95542d3.

Termination result

All 48 samples ended at the frozen reasoning boundary and required force-close. The content-blind token-ID audit found 38 exact periodic loops, with periods recorded in analysis/scientific_smoke_49k_termination_audit.json; the remaining 10 contacts were unresolved. Six answer stages reached the 512-token limit. Loop (79.17%), unresolved (20.83%), and answer-limit (12.50%) rates each fail the registered threshold. Therefore 49k was rejected before decoding or scoring.

At 61k, all 48 fresh samples likewise ended at the reasoning boundary and required force-close. The audit found 40 exact periodic loops, eight unresolved cap contacts, and four answer-limit contacts: rates of 83.33%, 16.67%, and 8.33%. All three fail their thresholds. The terminal selection SHA-256 is acbaf7cdb84ee5633e4f86b0716360c382f2262eb50e79ca92ce02b6e157fb07; it records pass=false and selected_thinking_budget=null, so no K=12 matrix or semantic analysis is eligible.

Operational throughput

The run sampled 2,363,163 tokens in 4,809.081014 generation seconds, or 491.395964 sampled tokens/s. Late JIT warnings are included in this elapsed time. Exact active-width capture was 4.1636% faster than the otherwise closest implicit-capture capacity-fit probe at 471.753824 tokens/s, consistent with the operational diagnosis that uncaptured widths 17--19 imposed avoidable overhead. It remained 16.2114% slower than the invalidly overcommitted max-seqs-64 diagnostic at 586.471154 tokens/s.

These are descriptive cross-protocol comparisons, not a clean causal benchmark: scheduler geometry can change trajectories, and sampled-token throughput does not count recomputed prefix work.

The exact-capture 61k probe sampled 2,951,995 tokens in 7,422.885983 generation seconds, or 397.688312 sampled tokens/s, after 100.293580 seconds of model loading. Its lower aggregate rate is also descriptive: the budget, active width, and generated trajectories differ from 49k.

The experiment-local CPU suite had already passed 42/42 tests and scripts/run.py --validate with both frozen record hashes and capture mappings. No decoded, semantic, oracle, or deployable metric is available.

Controls

The runner rejects an explicit list whose maximum differs from max_num_seqs. The live preflight rejects vLLM normalization/truncation, eager or piecewise-only decode, lack of active-width coverage, and insufficient block-rounded KV capacity. Storage validators bind both requested and resolved geometry into preflight, metadata, receipt, and catalog. Predecessor output and external roots are ineligible.

Interpretation

Exact graph coverage improved 49k aggregate sampled-token throughput modestly relative to the closest implicit-capture run, so active-width capture matters, but it did not recover the overcommitted diagnostic's speed. The stronger scientific obstacle is now terminal for this ladder: simply increasing the reasoning allowance from 49k to 61k left every probe sample at the boundary, mostly in exact periodic loops. This is a useful negative about the setup, not a negative macro result, because the protocol never reached a termination-adequate K12 matrix.

No outcome in this experiment by itself establishes verified-macro capability. A complete termination-adequate K12 matrix is required even for semantic smoke, and a separate matched-compute experiment is required for a capability claim.

Next action

Stop this ladder and preserve both rejected probes. Do not decode either probe or launch a K12 arm. Any future attempt to address periodic reasoning loops must be preregistered as a separate design variant rather than extending this result-bearing experiment.

Experiment log 5

Show the running log (5 entries, 2026-07-10)

2026-07-10 — experiment split and protocol freeze

Created a new experiment attached to operator_and_skill_inventories; named qwen35_4b_verified_macro_capacity_fit_rerun as the closest near-duplicate.
Copied only frozen tasks, demonstrations, libraries, prompt identities, macro DSL, and model harness. Exact hashes and the no-output inheritance boundary are in data/source_provenance.json.
Derived one local vLLM runner from predecessor SHA fd9972bdcb3a9e8b9841b45ed8e2849017a6e80b601e924817cdaaa5144b8782. The new runner accepts an explicit capture-size tuple, requires its maximum to equal max_num_seqs, records vLLM's resolved compilation config, and fails unless sizes match under a full-decode graph mode. Frozen derived SHA: 3a98eb8da787054aded56a1ec3fd040ee2edaacc7d0694b4aec5a0309488774a.
Registered 49k shapes [1,2,4,8,16,19] with max-seqs 19 and 61k shapes [1,2,4,8,15] with max-seqs 15. Both remain subject to the independent live-KV fit gate.
Preserved one-engine/one-phase execution, K4 nonpromotion, content-blind first-adequate selection, receipt-last storage, full-history verification, and three-pass analysis.
Assigned a new external artifact namespace and forbade both predecessor roots.
Added model-free tests for exact mapping, truncated resolved-list rejection, engine-config invariants, storage/state-machine behavior, parser/domain behavior, and hidden-content nonaccess.
No GPU process or model call was launched.

2026-07-10 — model-free verification

All 42 experiment-local CPU tests passed under the uv-managed vLLM environment, including explicit rejection of resolved NONE and piecewise-only CUDA-graph modes.
uv pip check --python .venv-vllm/bin/python passed for all 189 installed packages.
scripts/run.py --validate passed with record hashes bd66aa64942f9e57e1fe55ae716c154ea1231480d6163f1811a07828ba364907 (base) and c5a6cd00d9600b7a63c8e2c132e202b25da30f30af299afb3735a8f5525d9e86 (designed ceiling).
The validated exact capture mappings are 49,152 → [1,2,4,8,16,19] and 61,440 → [1,2,4,8,15]; their maxima exactly equal max-seqs 19 and 15.
The frozen protocol binding at this verification point is 9d2692c6acad35d3b7ab56ddf368c9974c1ddaf6e0a06997b01015c0de397158.
No GPU process or model call was launched by either validation command.

2026-07-10 — independent prelaunch GO

Before engine construction, a separate read-only reviewer checked the frozen binding 9d2692c6acad35d3b7ab56ddf368c9974c1ddaf6e0a06997b01015c0de397158, exact-width resolution assertions, live block-rounded KV gate, fresh artifact namespace, receipt-last state machine, content-blind first-adequate rule, K4 nonpromotion, and stop conditions.
The independent verdict was GO for only the registered fresh 49,152-token K=4 probe. It did not authorize automatic rung advancement, K=12 generation, decoded inspection, or scoring.

2026-07-10 — exact-capture 49k K=4 result

The live capacity audit passed: 996,864 exposed KV tokens, 528-token blocks, 963,072 tokens of block-rounded demand at max-seqs 19, and 33,792 tokens of remaining margin.
The constructed engine resolved full-decode CUDA graphs exactly at [1,2,4,8,16,19]; width 19 was covered as preregistered.
Receipt SHA-256 61da6f616365bf080e97f341bd0c2305b889998c4d161c61e63d06e5dfb5923c commits 48/48 completed K4 samples. The external four-file tree is 30,653,162 bytes with SHA-256 654d44119fc46fe83428c154680ee502073c00022ccb9cdb0922c1896ca20685.
All 48 samples contacted the reasoning boundary and were force-closed. The content-blind audit found 38 periodic loops, 10 unresolved cap contacts, and six answer-limit contacts. Rates of 79.17%, 20.83%, and 12.50% fail all three registered thresholds, rejecting the rung.
Generation sampled 2,363,163 tokens in 4,809.081014 seconds, or 491.395964 sampled tokens/s. Late JIT warnings remain included in elapsed time. This is 4.1636% faster than the closest implicit-capture capacity-fit probe and 16.2114% slower than the overcommitted max-seqs-64 diagnostic; the comparisons are descriptive across changed scheduler protocols.
No decoded or scored content was inspected. At this checkpoint the 61k K4 probe was authorized; its later terminal result is recorded below. No K12 matrix, semantic analysis, or macro claim was authorized.

2026-07-10 — exact-capture 61k K=4 terminal result

Ran one separately fresh 61,440-token K=4 probe after the 49k rejection. The live capacity gate passed with 997,888 KV tokens, 528-token blocks, 950,400 block-rounded tokens required at max-seqs 15, and a 47,488-token margin.
The constructed engine resolved full-decode CUDA graphs exactly at [1,2,4,8,15], covering the registered active width 15.
Receipt SHA-256 8f00535a773e347ec4f90a48eb6b00960935d7010c3500af7ccdf57fedd6f2e1 commits 48/48 completed K4 samples. Across both probes the external eight-file tree is 70,191,578 bytes with SHA-256 4aa311ab579c301f2b2d7383591e3e68ed66035d184bbaaa2659d59bd95542d3.
All 48 samples contacted the reasoning boundary and were force-closed. The content-blind audit found 40 periodic loops, eight unresolved cap contacts, and four answer-limit contacts. Rates of 83.33%, 16.67%, and 8.33% fail all three registered thresholds.
Generation sampled 2,951,995 tokens in 7,422.885983 seconds, or 397.688312 sampled tokens/s, after 100.293580 seconds of model loading.
Terminal selection SHA-256 acbaf7cdb84ee5633e4f86b0716360c382f2262eb50e79ca92ce02b6e157fb07 records pass=false and selected_thinking_budget=null. No K12 arm, decoded inspection, semantic analysis, or macro result is authorized. The setup negative is preserved rather than erased or promoted.

Reproduce

Run steps are documented inside the experiment folder (README and scripts).

Browse the experiment folder on GitHub ↗