Qwen3.5-4B Verified Macro Invention
The one idea you need
A "shortcut" here is a named stand-in for a fixed recipe of basic steps — like writing "make a roux" instead of its five exact stirring instructions. It counts as right only if spelling it back out reproduces those exact steps, unchanged.
The question
If you hand a small model ready-made shortcuts and the exact step-by-step plan, can it simply re-write that plan using the right shortcuts?
What we found
No. Even handed the finished plan and asked only to re-express it — no problem to actually solve — the model got it exactly right just one time in four, short of the three-in-four bar. Every output looked flawless: valid syntax, a shortcut always called, nothing cut off. It named the right first shortcut, then invented extra ones that quietly expanded into different steps.
Why it matters
When a small model drives a tool or shortcut interface, never treat clean formatting and high shortcut-usage rates as passing. Verify that every call, expanded literally, reproduces the intended steps — score meaning, not surface.
On this page
Results at a glance 2
How to read
Two bar groups, one per attempt: thinking budgeted to 768 tokens, then thinking turned off. The tall bar counts how many of four test cases were rewritten exactly right; the flat reference marks the three needed to pass. Higher is better.
Takeaway → Both attempts fell short — two, then one, of four — so the run halted at the interface check before the real shortcut question was ever tested.
Data table
| condition | records with any exact optimal output | records required to pass |
|---|---|---|
| thinking at 768 tokens | 2 | 3 |
| thinking off | 1 | 3 |
Numbers from reports/report.md (Results); analysis/interface_v{2,3}_gate_failed.json
Technical framing
Exact plan transcription never cleared the 3-of-4 interface gate — The primitive plan was supplied, so this is an interface check rather than an induction result. Both attempts failed before the fresh scientific smoke or full evaluation was shown to the model.
How to read
Two groups over sixteen samples each: thinking at 768 tokens, then thinking off. One bar counts cleanly parsed samples, the other counts samples cut off mid-answer. Thinking off pushed clean parses to all sixteen and cut-offs to zero. More parsed, fewer cut off is better.
Takeaway → Turning thinking off made every sample parse cleanly and eliminated all cut-offs, yet exact expansion still failed on thirteen of sixteen — clean output was not correct output.
Data table
| condition | strictly parsed samples | answer-truncated samples |
|---|---|---|
| thinking at 768 tokens | 4 | 12 |
| thinking off | 16 | 0 |
Numbers from reports/report.md (Results); analysis/interface_v3_audit.json
Technical framing
No-think fixed formatting and truncation, but not exact shortcut placement — Thinking off made all 16 outputs parse and removed all truncation. Exact expansion still failed on 13 samples because the model added unrelated aliases and exceeded the five-primitive limit.
In the author’s words from the Overview · “Results”
Smoke v1 failed the interface gate. The matched base/designed pool parsed at 0.5972 overall (base 0.6111; designed 0.5694), but answer truncation was 0.40046, the designed arm produced zero valid macro-using candidates, and its oracle coverage was 0 versus base 0.0833. All 1,440 solver samples force-closed; 607 answer stages truncated. The sole base oracle solve was a no-reuse task. The macro-proposal whole-answer parser also accepted 0/16 samples. A post-failure, exploratory line-local audit found 18 behaviorally unique train-supported candidates, showing parser loss but not establishing a usable Qwen library. Neither diagnostic is evidence against or for verified macros. Full generation has not run, so the research hypothesis remains unresolved. … Read the full result →
Overview
smoke or full evaluation was run; the verified-macro hypothesis remains unresolved.
2026-07-10 follow-up: the separate qwen35_4b_verified_macro_long_context_rerun later cleared a broader, disjoint plan-given interface gate on 16/16 records under adequately budgeted vLLM inference. This directory's historical outputs and reported numbers remain unchanged; the additive forward link does not rewrite its registered low-budget stop. That stop should not be read as a durable model-level shortcut-interface failure.
Research program
- Primary:
operator_and_skill_inventories - Secondary:
structured_execution_and_compilers,benchmark_generalization - Intake: idea_intake.md
- Preregistration: reports/preregistration.md
- Fresh-smoke interface amendment: reports/preregistration_amendment_1.md
- No-think transcription amendment: reports/preregistration_amendment_2.md
- Design review: reports/design_review.md
- Preserved smoke-v1 failure: reports/smoke_v1_failure.md
- Final interface failure: reports/interface_v3_failure.md
Question
Can a verified abstraction library built only from prior solved programs turn fresh, behaviorally deep programs into shallower decisions and improve visible-only selected accuracy beyond matched-compute sampling over the original primitives?
Hypothesis
The fixed-vocabulary assumption is part of the composition bottleneck. Recurring depth-2/3 motifs can be packaged as exact callable macros, reducing the surface decision depth of unseen behaviorally true-depth-5 combinations. If that representation is useful, frequent train-only macros should beat both base-primitive sampling and matched random composites; correct treatment-only solutions should actually call macros; and the advantage should concentrate on a preregistered motif-reuse split.
Why this is not another operator-bank experiment
The closest experiments grow or shortlist fixed, human-authored atomic inventories. Here the inventory itself is learned from prior programs. The main scientific comparison separates:
- deterministic tool-mined abstractions;
- Qwen-proposed, locally verified abstractions;
- highlighted-but-not-callable subsequences;
- random composite entries matched on count, length, and train support;
- generator-known motifs as a clearly labeled ceiling.
The experiment therefore asks both whether abstraction helps the system and whether Qwen adds anything to a deterministic miner.
Setup
- Only model:
Qwen/Qwen3.5-4B, repository-pinned revision. - Inference: experiment-local src/vllm_runner.py under
.venv-vllm; no Transformers inference and no mixed backend. - Substrate: contamination-free procedural list transformations with exact execution.
- Construction corpus: primitive-rendered programs from a frozen latent-motif grammar.
- Full evaluation: 80 motif-reuse and 40 primitive-multiset-matched no-reuse tasks at behaviorally verified depth 5, plus a disjoint smoke set.
- Examples: eight visible, eight hidden-grade, and eight unlabeled probe inputs per task.
- Sampling: budgeted thinking, K=12 per macro arm; base K=24 supplies the matched-token sample-more curve.
- Primary metric: visible-only selected hidden-all accuracy, pooled on the reuse split.
- Oracle-only metric: whether any sampled candidate passes all hidden cases.
- Hidden boundary: macro construction sees train programs only; prompts see visible I/O; hidden outputs enter only the committed analyzer.
The latent-motif generator is necessary rather than cosmetic: a uniform primitive generator has no real recurring abstraction to discover. The no-reuse control preserves each paired task's primitive multiset while permuting away the three evaluation-recurrent motifs; train-only decoy motifs may remain, making this a conservative rather than artificially macro-hostile control.
Gates
- CPU split/min-depth/leakage checks and an oracle compression check.
- Adversarial design review saved before model generation.
- Separate vLLM smoke: parse rate at least 0.50 and nonzero macro use in the designed-ceiling arm, with no tuning on full evaluation outputs. Smoke v1 failed the interface gate and is preserved. The task-independent plan-given attempts 2 and 3 then failed before any fresh smoke prompt. Amendment 2's stop rule is now final.
- Full result only after the smoke gate passes. It never did, so full generation is forbidden.
The macro mechanism clears only under the decision rule in the preregistration: at least +0.10 selected-accuracy lift over base with a positive paired interval, survival at matched token cost, macro use carrying treatment-only successes, and a materially smaller effect on no-reuse tasks.
Run
All commands use the uv-managed vLLM environment.
CPU preparation and tests:
.venv-vllm/bin/python -m unittest discover -s experiments/qwen35_4b_verified_macro_invention/tests -v
.venv-vllm/bin/python experiments/qwen35_4b_verified_macro_invention/scripts/run.py --prepareHistorical reproduction of the stopped interface path (expected to terminate at the failed gate; earlier attempts are preserved under versioned archive paths):
.venv-vllm/bin/python experiments/qwen35_4b_verified_macro_invention/scripts/run.py --smokeFull (registered for provenance but never run and now forbidden by the stop rule):
.venv-vllm/bin/python experiments/qwen35_4b_verified_macro_invention/scripts/run.py --fullAnalyze an existing run without loading the model:
.venv-vllm/bin/python experiments/qwen35_4b_verified_macro_invention/scripts/analyze.py --run fullResults
Smoke v1 failed the interface gate. The matched base/designed pool parsed at 0.5972 overall (base 0.6111; designed 0.5694), but answer truncation was 0.40046, the designed arm produced zero valid macro-using candidates, and its oracle coverage was 0 versus base 0.0833. All 1,440 solver samples force-closed; 607 answer stages truncated. The sole base oracle solve was a no-reuse task.
The macro-proposal whole-answer parser also accepted 0/16 samples. A post-failure, exploratory line-local audit found 18 behaviorally unique train-supported candidates, showing parser loss but not establishing a usable Qwen library. Neither diagnostic is evidence against or for verified macros. Full generation has not run, so the research hypothesis remains unresolved.
Amendment 1 freezes a fresh-seed v2 smoke at the already planned full thinking budget, with a shared surface-first procedure, a strict solver parser, a train-only plan-given interface probe, and matched K=12 base/designed arms. Full generation remains blocked until that gate passes.
The first execution of that task-independent probe, interface attempt 2, failed before any fresh induction prompt was shown. Of 4 records and 16 samples, records 00 and 02 succeeded, 4 samples strictly parsed and all 4 used macros, but answer truncation was 12/16 = 0.75. Amendment 2 therefore retries only the identical plan-transcription gate with vLLM thinking: off, n=4, and the unchanged 128-token answer cap and 3/4-plus-truncation gate. If it passes, the still-unseen induction smoke remains think@768 exactly as frozen in amendment 1.
Interface attempt 3 removed the mechanical failures but still failed exact alias fidelity. All 16/16 samples strictly parsed, all 16 used macros, and none truncated, yet only record 00 succeeded; record coverage was 1/4 against the registered 3/4 gate. Qualitatively, most inspected errors called the intended leading alias and then hallucinated extra aliases for primitive suffix operations that those aliases did not match. The committed raw-row audit makes the pattern exact: all 13 failed samples used multiple aliases and expanded past depth five; 10/13 included the correct designated alias and 3/13 omitted it.
The stop rule fired. No fresh induction-smoke or full prompt was ever generated. Consequently this experiment provides a strong interface lesson—syntax and macro invocation can be perfect while literal expansion fidelity fails—but no evidence for or against the macro-invention hypothesis. There is no claim-ledger update. Further work must start as a new material follow-up, not amendment 4 in this directory.
Artifacts
data/: frozen corpus, task splits, libraries, prompts, manifests, and the smoke-v1 snapshot.runs/: raw vLLM generations and exact runtime/token metadata, including versioned failed-v1 proposal/solver outputs and failed interface attempts 2 and 3.analysis/: derived task-level tables and machine-readable verdicts, including smoke v1 and both task-independent interface gates.archive/: exact source snapshots needed to explain the historical v1, interface-v2, and interface-v3 outputs.reports/: preregistration, amendments, design review, failure/final reports, and manifest.
Report
Rendered from reports/report.md
Summary
The first vLLM smoke failed its interface gate. Two task-independent, plan-given repairs then separated answer-boundary failure from semantic alias failure. No-think made all 16 final-gate outputs valid, macro-using, and non-truncated, but only 1/4 records reproduced the supplied plan exactly. The registered stop rule fired before any fresh induction prompt. This experiment is closed with the verified-macro hypothesis unresolved and no claim-ledger update.
Follow-up context (2026-07-10)
The additive long-context follow-up qwen35_4b_verified_macro_long_context_rerun later passed a disjoint 16-record plan-given gate at think@16,384: every record had at least one exact macro-using rewrite, with no unresolved cap contact or answer truncation. The historical numbers below are unchanged. Their proper scope is the two original low-compute interfaces, not a general inability to place verified aliases.
Research Program Fit
The question remains attached to operator_and_skill_inventories, with secondary relevance to structured_execution_and_compilers and benchmark_generalization. Existing work studies fixed operator banks; this experiment still targets the unresolved question of whether train-derived, exact composite entries can change held-out proposal coverage beyond sample-more and matched random entries.
Method
The procedural list DSL, construction corpus, exact interpreter, true-depth-5 full tasks, mined and placebo libraries, visible-only selector, and vLLM-only inference path were frozen before model generation. Smoke v1 used a disjoint 12-task set. The base arm sampled K=24 and each macro arm K=12; the registered interface gate pooled the matched base/design comparison and separately checked each arm.
After v1, amendment 1 inserted a non-scored train-only plan-given gate before any new induction task. Four prompts each supplied the verified primitive plan and asked for an optimal rendering with designed aliases; each received four samples. This isolates whether the model can call the surface when no program induction is required.
Results
| Smoke-v1 gate metric | Result |
|---|---|
| Pooled base/designed parse | 0.5972 |
| Base parse | 0.6111 |
| Designed-ceiling parse | 0.5694 |
| Pooled base/designed answer truncation | 0.40046 |
| Designed valid macro-using candidates | 0 |
| Designed oracle coverage | 0/12 = 0.0000 |
| Base oracle coverage | 1/12 = 0.0833 |
All 1,440 solver samples force-closed and 607 answer stages truncated across all generated arms. The only base oracle solve was in the no-reuse split. The whole-answer proposal parser accepted 0/16 outputs. An exploratory line-local audit after failure found 18 behaviorally unique, train-supported candidates, but that reparse is not part of the v1 result and cannot backfill a Qwen-ranked arm.
Interface attempt 2 then produced:
| Plan-given interface metric | Result |
|---|---|
| Records / samples | 4 / 16 |
| Successful records | 2 (00, 02) |
| Strictly valid samples | 4 |
| Macro-using samples | 4 |
| Answer truncation | 12/16 = 0.75 |
It failed the required at-least-3/4 successful records and below-0.05 truncation conjunction. The pipeline stopped before generating a fresh smoke prompt.
Interface attempt 3 then changed only the plan-given gate's thinking channel:
| No-think interface metric | Result |
|---|---|
| Records / samples | 4 / 16 |
| Successful records | 1 (00) |
| Strictly valid samples | 16 |
| Macro-using samples | 16 |
| Answer truncation | 0/16 = 0.0000 |
Attempt 3 passed the truncation gate but failed the unchanged at-least-3/4 record requirement. The committed raw-row audit found three exact samples, all on record 00. Each of the 13 failed samples used multiple macros and expanded beyond the five-primitive limit (depth 6--10); 10/13 included the correct designated alias but appended unrelated aliases, while 3/13 omitted it. No fresh induction or full prompt was ever generated.
Controls
No full control contrast was run. Construction/full overlap remained zero, full tasks and hidden outputs were not used for repair, and every model call used the same experiment-local vLLM backend. Smoke v1 is preserved rather than overwritten. Amendment 1 uses a new seed and ids, matched K=12 base/designed arms, identical surface-first instructions, and a train-only plan-given mechanical probe.
The failed plan-given attempts are also preserved separately. Amendment 2 changed only the final gate's thinking mode; it did not alter prompts, targets, parser, thresholds, fresh smoke, proposal ranking, or the full protocol.
Attempt 3 confirms why exact verification was non-negotiable: parse and raw macro use were both 16/16, yet exact record success was 1/4. Neither plan-given attempt is pooled with another or with scientific-task evidence.
Oracle Versus Deployable Evidence
Neither deployable nor oracle abstraction evidence exists yet. Designed-ceiling oracle coverage was zero because the interface produced no valid macro-using candidate; this prevents attribution to library quality. The base arm's single no-reuse solve is too small and on the wrong slice to support a scientific comparison.
Interpretation
Smoke v1 diagnosed a budget-and-surface failure. Attempt 2 established that designed aliases were sometimes callable but budgeted thinking spilled into the answer. Attempt 3 showed that removing thinking fixed syntax and termination completely without fixing exact alias substitution. A model can call the intended leading abstraction, then over-compress unrelated suffix operations with plausible but behaviorally wrong aliases.
The durable lesson is methodological: strict syntax and macro-use rates are insufficient. A composite call is valid only when literal expansion preserves the intended plan. The scientific macro-invention question itself remains unresolved because its fresh task was never attempted.
Next Experiments
Do not add amendment 4. A further attempt would need a materially different exact-call interface, such as a constrained representation or separately verified rewrite procedure, and is therefore a new experiment. It should preserve the same one-model/vLLM boundary, establish plan-given expansion fidelity before induction, and retain matched-compute sampling and no-reuse controls.
Artifact Manifest
The adjacent artifact_manifest.yaml records the vLLM-only reproduction commands and the complete versioned smoke-v1, interface-v2, and interface-v3 archives. There are no external model or adapter artifacts.
Experiment log 8
Show the running log (8 entries)
Scaffold
Created as a new experiment scaffold after the user selected verified macro invention from the 2026-07-09 forest review.
Design freeze, before GPU
- Attached to
operator_and_skill_inventories; no new program needed. - Related-work search found fixed human-authored inventory scaling and failed large-bank shortlisting, but no result-bearing experiment that derives executable composite operators from a prior solved-program corpus.
- Chose the experiment-local vLLM runner for every proposal and solver sample. No Transformers inference comparison is permitted.
- Added a frozen latent-motif source grammar because uniform independent primitives contain no genuine abstraction distribution to recover.
- Adversarial review required paired no-reuse tasks, exhaustive true-depth checks, multiple matched random libraries, a highlighted-but-not-callable control, and conjunctive verdicts.
- Scoped the Qwen arm honestly as proposal/ranking within the closed length-2/3 macro language; Qwen-specific invention requires exclusive verified entries that carry unique correct solves.
- No model generation was launched before saving the intake, preregistration, and design review.
CPU preflight
- Full generation produced 932 unique concrete programs and 932 unique frozen-probe behavior signatures: 800 construction, 12 smoke, and 120 scored tasks.
Every scored task is behaviorally verified at true depth 5 by exhaustive search through depth 4. Construction/evaluation concrete and behavioral overlap are both zero.
- The designed library reduces reuse tasks by exactly two surface calls (80/80); paired no-reuse median reduction is zero. These are substrate gates, not evidence.
- Repeated preparation was byte-identical after sorting multiset permutations before seeded shuffling.
- A tokenizer-only preflight found the original 200-program proposal view produced an 18,007-token prompt (18,903 with the registered generation reserve), exceeding the 16,384 vLLM context. Before any model load, reduced the frozen proposal view to 150 programs and regenerated every dependent library/hash. No scored output existed.
- Clarified before GPU that all arms share the same parent run seed and decode configuration, while the generic vLLM runner intentionally derives deterministic effective seeds from arm-qualified record ids; this is not a common-random-numbers design.
Smoke v1: failed interface gate
- Ran the macro-proposal stage and solver smoke entirely through the experiment-local vLLM runner with the pinned
Qwen/Qwen3.5-4Brevision. No full generation was launched. - The registered matched base/designed pool had 0.5972 overall parse rate: base 0.6111 and designed ceiling 0.5694, all above the 0.50 parser threshold.
- The same pool had 0.40046 answer truncation, far above the 0.05 ceiling. Every one of the 1,440 all-arm solver samples force-closed its thinking stage, and 607 answer stages truncated.
- The designed ceiling produced zero valid macro-using candidates and zero oracle solves. Base oracle coverage was 1/12 = 0.0833, with the sole solve on a no-reuse task. The smoke gate failed.
- The strict whole-answer macro-proposal parser accepted 0/16 samples. A post-failure line-local audit found 18 behaviorally unique, train-supported candidate expansions in those same raw outputs. This audit is exploratory only: it does not populate the v1 Qwen arm or alter the failed-v1 verdict.
- Interpretation: v1 did not establish a usable macro surface. It did not test the full macro hypothesis. Saved the complete diagnosis in
reports/smoke_v1_failure.md.
V1 preservation and v2 refreeze, before another GPU call
- Preserved the historical config, smoke data, failed proposal, failed solver outputs, analyses, and exact source under
configs/smoke_v1.yaml,data/smoke_v1_frozen/,runs/proposal_v1_failed/,runs/smoke_v1_failed/,analysis/smoke_v1_failed/, andarchive/smoke_v1_source/. - Left the construction corpus, proposal view, libraries, full tasks and hashes, hidden boundary, analyzer, controls, and full decision rules unchanged.
- Froze amendment 1 before v2 generation. V2 uses fresh seed
20260710and idssmoke-v2-reuse-NNN/smoke-v2-no-reuse-NNN, with explicit disjointness against train, v1 smoke, and full evaluation. - Matched the v2 smoke to the preregistered full think@768 budget while retaining the 128-token answer cap. Added the same surface-first procedure and abstract alias-use example to both arms, while keeping the solver parser strict.
- Restricted the scored v2 gate to base and designed ceiling at matched K=12. The macro-use gate now requires valid alias use on at least two distinct reuse tasks.
- Added a non-scored train-only plan-given interface probe. It may diagnose mechanical formatting and alias calling only; it is not hypothesis evidence.
- Repaired the train-only proposal interface with a compact program-only prompt and a frozen line-local first-eight extraction rule. Full generation remains blocked until v2 passes.
Interface attempt 2: failed before fresh smoke
- Ran only the non-scored, train-only, plan-given designed-alias transcription gate from amendment 1. It contained 4 records and n=4, for 16 total vLLM samples.
- Two records succeeded:
interface-v2-00::designed_ceilingandinterface-v2-02::designed_ceiling. A successful record had at least one strict completion that used a macro, had optimal surface length, and expanded exactly to the supplied plan. - Four samples were strictly valid and all four used a macro. This shows some alias calling, but only 2/4 records cleared the required 3/4 reliability gate.
- Answer truncation was 12/16 = 0.75, failing the below-0.05 gate. All 16 samples exhausted the 768-token thinking allowance and force-closed before the answer stage.
- The runner raised the registered gate failure and stopped. No fresh
smoke-v2-*evaluation prompt was generated or shown to the model; the fresh smoke tasks remain model-unseen. Full generation also remains unrun. - Preserved the exact attempt under
runs/interface_v2_failed/,analysis/interface_v2_gate_failed.json,configs/interface_v2.yaml, andarchive/interface_v2_source/.
Amendment 2: interface attempt 3, frozen before GPU
- Classified the plan-given gate as transcription/formatting rather than induction: the verified primitive plan is already present in the prompt, so extended reasoning is not part of the capability being tested.
- Froze a retry of only that gate using the copied vLLM runner's exact
thinking: offmode, n=4, and answer cap 128. Prompt contents, four targets, designed aliases, parser, executor, parent seed family, and success definition remain unchanged. - Retained the same requirements: exact macro-using optimal transcription on at least 3/4 records and answer truncation below 0.05 across 16 samples. Failure stops before induction smoke.
- If attempt 3 passes, the scientific induction smoke still uses think@768 on the same unseen fresh tasks under amendment 1. No full metric or decision rule changes.
Interface attempt 3: final gate failure and stop
- Retried only the same 4 task-independent, plan-given records through the experiment-local vLLM runner with
thinking: off, n=4, and answer cap 128. - All 16/16 samples passed the strict program parser, all 16/16 used at least one supplied macro, and 0/16 truncated. No-think therefore repaired the formatting, termination, and raw alias-use failures seen under think@768.
- Only
interface-v3-00::designed_ceilingsucceeded under the full exact criterion. Record coverage was 1/4, below the frozen at-least-3/4 gate, so attempt 3 failed. - A committed post-gate audit regenerated the error taxonomy from raw rows. All 13 failed samples used multiple aliases and expanded beyond depth five (depth 6--10); 10/13 included the correct designated alias but appended unrelated aliases, while 3/13 omitted it. This audit describes the already-failed gate and does not change its decision.
- Preserved the exact config, outputs, verdict, and source under
configs/interface_v3.yaml,runs/interface_v3_failed/,analysis/interface_v3_gate_failed.json, andarchive/interface_v3_source/. - Amendment 2 required a stop on failure. No fresh
smoke-v2-*induction prompt and no full prompt was ever generated or shown to the model. The fresh scientific question remains untested. - Closed the experiment as interface gate failed; macro hypothesis unresolved. No claim-ledger update is warranted. Any additional interface design is a material follow-up and must receive a new experiment directory, intake, design review, and preregistration rather than amendment 4.
Data files 1
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
Reproduce
Smoke test
.venv-vllm/bin/python experiments/qwen35_4b_verified_macro_invention/scripts/run.py --smokeFull run
.venv-vllm/bin/python experiments/qwen35_4b_verified_macro_invention/scripts/run.py --fullRun steps are documented inside the experiment folder (README and scripts).