Qwen3.5-4B Early Text Hypothesis Forking
The one idea you need
Imagine giving the same puzzle to twenty-four independent scratchpads. Each starts with a different concrete guess about the first transformation, may reject that guess, and must still write the complete two-step rule. Public examples—not the hidden answer—choose one result.
The question
Does giving each possible first transformation before reasoning produce better complete rules than giving it late or simply buying more ordinary attempts?
What we found
The experiment design and model-free checks now pass, but the model has not run. Review expanded the first draft from twelve operation names to twenty-four fully specified operations, added two fair late-hint comparisons, and proved that all 576 possible two-step rules are covered by the public-data control. The next gate tests whether the model can actually use these hints rather than merely repeat them.
Why it matters
The previous internal-state experiment could change a candidate name without showing that later reasoning used it. This test moves a plain-text candidate to the start of reasoning and demands a full computed rule, while matched ordinary attempts and exhaustive search prevent extra branching from receiving free credit.
On this page
Results at a glance 1
How to read
Each bar is one public diagnostic list. Its height is the number of fully specified operations that produce a distinct expected result; all four reach the required twenty-four.
Takeaway → The upcoming model check cannot pass by exploiting two candidates that happen to mean the same thing on a diagnostic input.
Data table
| public diagnostic input | distinct operation results |
|---|---|
| diagnostic 1 | 24 |
| diagnostic 2 | 24 |
| diagnostic 3 | 24 |
| diagnostic 4 | 24 |
Numbers from experiments/qwen35_4b_early_text_hypothesis_forking/runs/smoke/summary.json
Technical framing
All four model-free diagnostics separate the 24 bound operations — Model-free design check only. The refreshed smoke also exhausts all 576 two-step programs on each of 144 fresh tasks, finds zero readable-ancestor behavior collisions, and makes no model call.
In the author’s words from the Report · “Summary”
The authenticated mechanics run is terminal INVALID_INTERFACE_PARSE. Supplying an exact bound operation before reasoning produced strong direct execution control, but the frozen answer interface and full-program reachability requirements did not pass. Qualification and confirmation were correctly not run. This is a useful mechanism-localized negative, not a capability gain.
Overview
This experiment tests whether supplying each fully bound first operation at the start of Qwen3.5-4B reasoning changes complete two-step program proposals enough to beat late hints and compute-matched sampling under visible-only selection.
The original design is supplemented by a pre-model authenticated-mechanics amendment. It freezes an eight-case parameter-stratified reachability ceiling, contextwise adherence gates, receipt-last generation, and a conservative live KV no-preemption check. No model was constructed before this amendment.
Research program
- Primary:
structured_execution_and_compilers. - Secondary:
evidence_conditioned_selection,test_time_reasoning_budget, andinterpretability_and_diagnostics. - Immediate parent:
qwen35_4b_semantic_anchor_coordinate_branching. - Closest near-duplicate:
qwen35_4b_hypothesize_verify_wall, which used one generic procedure scaffold rather than a trajectory for every bound first operation.
Question
Does placing a concrete first-operation hypothesis at the beginning of native thinking shift Qwen3.5-4B's full-program proposal distribution, and can one frozen visible-only selector turn that shift into more correct programs than late injection and compute-matched ordinary sampling?
Why this follows the Jacobian work
The parent late-anchor experiment did not establish a valid consequence effect: its unrestricted interface failed and its composed mappings accidentally cancelled. Its constrained diagnostic nevertheless supplied one useful timing clue—late state could write an operation name without evidence that subsequent computation consumed it. This successor tests that clue with text, before reasoning, without claiming J-space transport or internal certainty.
Frozen setup
- Model: only
Qwen/Qwen3.5-4B, revision851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a. - Backend: one experiment-local pinned vLLM runner, bf16, for every generation arm. No backend mixing.
- Substrate: fresh exact-depth-two list transformations with 8 visible, 8 hidden, and 16 unlabeled probe inputs; 48 qualification tasks in two frozen blocks and 96 sealed confirmation tasks.
- Bound bank: all 24 legal first operations—8 parameter-free, 6 additions, 3 multiplications, 4 takes, and 3 rotations.
negateremains a distractor but is not generated as the gold first step because of pervasive equivalent reorderings. - Task admission: exhaustive enumeration of all 24² programs must prove no depth-one fit, unique identification of the bound first operation, and hidden/probe equivalence of every visible-consistent program.
- Output: a strict Python helper function with exactly two allowed assignments. An AST parser canonicalizes the calls and an interpreter executes them; generated code is never evaluated.
- Selector: canonical deduplication, exact visible pass, clustering by outputs on unlabeled probes, cluster support, then a frozen task hash. Gold is opened only after prompts, outputs, resources, and selected IDs are written.
Arms
early_concrete_24: one independently sampled trajectory per bound operation; the exact hypothesis tokens are inserted immediately inside open<think>.late_equal_total_24: an independent blind 512-token prefix per branch, the same hypothesis tokens, then 512 more thought tokens.late_equal_post_24: the same 512-token blind construction followed by 1,024 post-hypothesis tokens, deliberately overmatching early's usable suffix.early_duplicate_24: 24 branches with one task-hash-selected operation.early_placebo_24: an exact-scaffold neutral hypothesis bank with token use charged.neutral_sample_more_masterandplain_sample_more_master: independently frozen 48-sample pools. Prefixes selected before grading provide sampled-token and logical-model-token match points plus first-over-budget controls.cpu_exhaustive: all 576 two-step programs tested against visible examples. This is the symbolic scope ceiling, not a model oracle.
Every model arm uses the same decode distribution and final visible-only selector. Every prompt, blind prefix, resumed-prefix prefill, injected token, thought token, forced close, answer token, invalid output, and duplicate is charged in resource receipts.
Stages
- CPU smoke proves inventory, task construction, ancestor disjointness, strict parse/execution, gold-mutation invariance, outcome-blind compute matching, branch permutation independence, and composed slot-to-behavior variation.
- Label-free mechanics crosses four public diagnostic inputs with all 24 bound operations. It requires natural parse, correct computed lists, broad support, low cap contact, and adherence above deranged/duplicate/placebo controls.
- Qualification runs 48 tasks only after mechanics passes. Early must beat both late controls and every token-matched duplicate/placebo/sample-more construction under paired task uncertainty and Holm correction.
- Confirmation repeats the frozen protocol on 96 untouched tasks. Splits are never pooled. Training is not part of this experiment.
Run
Model-free smoke:
.venv/bin/python experiments/qwen35_4b_early_text_hypothesis_forking/scripts/run.py --stage smokeDeterministic tokenizer-only mechanics preparation:
PYTHONDONTWRITEBYTECODE=1 .venv-vllm/bin/python experiments/qwen35_4b_early_text_hypothesis_forking/scripts/run_mechanics.py --stage prepareAfter the implementation-lock file itself is committed and pushed, the exact live command is:
PYTHONDONTWRITEBYTECODE=1 .venv-vllm/bin/python experiments/qwen35_4b_early_text_hypothesis_forking/scripts/run_mechanics.py --stage run --lock experiments/qwen35_4b_early_text_hypothesis_forking/runs/mechanics/implementation_lock.jsonThe runner refuses construction before that lock is tracked on published main. Each completed invocation is immutable and safely skipped on resume; an ambiguous started-only invocation is never resampled.
Current result
The locked mechanics run completed all 392 rows and passed raw authentication. Its terminal decision is INVALID_INTERFACE_PARSE, so qualification and confirmation remain sealed.
The preregistered adherence subgate was strongly positive: the systematic arm executed the injected registered operation on 84/96 rows (0.875), while the length-matched deranged arm executed that same registered operation on 0/96. The deranged arm instead executed its own supplied operation on 84/96. This pattern held above every contextwise threshold, covered all 24 operations and all four contexts, and was not reproduced by duplicate (4/96 registered) or placebo (0/96) hints. It is evidence that an exact early text hypothesis can control this one-operation diagnostic, not that it solves the task.
The experiment stopped for two independent reasons. First, the answer-interface gate failed: systematic and deranged cap-contact rates were 0.0938 and 0.1042 against a 0.05 ceiling, while duplicate/placebo parse rates were only 0.7292 and 0.5312. Second, the noncausal full-program ceiling solved only 3/8 visible cases (0.375 versus 0.50), although the parameterized stratum reached 2/4. All eight programs parsed and none contacted the answer cap. The model therefore consumed bound early text reliably for a direct execution diagnostic but did not turn it into sufficiently reliable complete two-step proposals under this interface.
The model-free smoke remains CPU_SMOKE_PASS. The preparation receipt is 2d6b668a6d43e1bd657124c3645d85ea9996d9aaaea8f81225b97472a2f5b292, the published implementation lock is 12c298aef3ba9cc83bd4d1cdadc304aa0daa7d71bde4be3fce81514fdf4b3148, and the post-run authentication receipt is 921699b45e585e4990defb3c81fd95334c52d71f953446c537d21a3f640bcac7. See the result report and adversarial result review.
Claim boundary
This failed mechanics run supports no deployable capability or proposal-coverage claim. The positive adherence subgate is scoped to four public one-operation diagnostics and cannot be promoted past the failed interface and full-program ceiling. It does not show consciousness, internal certainty, J-space transfer, weight installation, autonomous solving, or superiority to exhaustive symbolic search. Any residualization, interface repair, higher-depth, or installation test must be a new experiment.
Artifacts
idea_intake.md: novelty and nearest-neighbor decision.reports/design_review.md: adversarial attacks and mandatory resolutions.reports/preregistration.md: frozen arms, gates, and stop logic.reports/preregistration_amendment_1.md: pushed pre-model mechanics boundary.reports/implementation_review.md: adversarial code findings and repairs.reports/result_review.md: independent post-run authentication, gate, and interpretation audit.configs/default.yaml: exact model, data, compute, and thresholds.runs/smoke/summary.json: model-free validation receipt.runs/mechanics/prepared/: independently rebuildable exact-token requests and the pre-outcome receipt.runs/mechanics/implementation_lock.json: added only after the implementation commit is published; it is the final prerequisite to model construction.runs/mechanics/raw/: exact prompts/outputs, engine sidecars, immutable transaction receipts, and the authentication receipt.runs/mechanics/scored/andruns/mechanics/summary.json: strict-parser outputs and the terminal preregistered decision.reports/artifact_manifest.yaml: external/omitted artifact policy.
Report
Rendered from reports/report.md
Summary
The authenticated mechanics run is terminal INVALID_INTERFACE_PARSE. Supplying an exact bound operation before reasoning produced strong direct execution control, but the frozen answer interface and full-program reachability requirements did not pass. Qualification and confirmation were correctly not run. This is a useful mechanism-localized negative, not a capability gain.
Research program fit
This is a fresh timing and proposal-shaping test in structured_execution_and_compilers, motivated by—but scientifically separate from—the invalid late semantic-anchor mechanics result. It asks whether explicit bound text hypotheses present before thought can elicit more useful full-program proposals than late text or matched ordinary sampling.
Frozen method
- all 24 bound DSL operations receive one early branch;
- two independent-prefix late arms distinguish total budget from usable post-hypothesis budget;
- duplicate, exact-scaffold placebo, neutral sample-more, and plain sample-more controls use one frozen visible-only selector;
- all 576 public two-operation programs form a deterministic scope ceiling; and
- qualification/confirmation gold remains sealed until pre-grade artifacts and resource matches are written.
The complete arm, gate, compute, and claim contracts are frozen in reports/preregistration.md; the attacks that materially changed the design are recorded in reports/design_review.md.
Model-free evidence
The terminal model-free decision is CPU_SMOKE_PASS:
- 24 bound candidates and 576 exhaustive two-step programs are frozen;
- 48 qualification plus 96 confirmation tasks are deterministic and behavior- disjoint from readable ancestor procedural artifacts;
- every visible-consistent program is hidden- and probe-equivalent;
- four diagnostics each distinguish all 24 bound-operation consequences;
- all 144 taskwise slot-to-operation-to-behavior compositions are distinct and gold positions are balanced; and
- strict parser, visible selector, exact-token runner, gold-mutation firewall, and outcome-free resource matcher tests pass;
- four exact-token diagnostic arms each contain 96 independently rebuilt rows;
- the eight full-program cases cover four parameter-free and four parameterized first-operation families; and
- the deterministic prepare receipt has SHA-256
2d6b668a6d43e1bd657124c3645d85ea9996d9aaaea8f81225b97472a2f5b292.
All 39 experiment tests plus 33 parameterized runner subtests passed before the model call. At that boundary model_loaded=false and outcomes_loaded=false; these checks validated the instrument, not the hypothesis.
The implementation-lock receipt binds the pushed amendment and implementation commits and every critical-file blob. Its SHA-256 is 12c298aef3ba9cc83bd4d1cdadc304aa0daa7d71bde4be3fce81514fdf4b3148, with model_calls_before_lock=0.
Authenticated mechanics result
The live preflight passed with the pinned bf16 Qwen3.5-4B revision, exact vLLM 0.24.0 engine arguments, explicit full CUDA graphs, and 760,366 KV-cache tokens. All five receipt-last invocations completed: 96 rows each for systematic, deranged, duplicate, and placebo, plus eight noncausal program-ceiling rows. The analysis-only authentication decision is MECHANICS_AUTHENTICATION_PASS; all raw, metadata, prepared-request, runner, preflight, model, revision, engine, seed, prompt-token, completion-token, and environment-lock identities match.
The strict frozen scorer produced:
| Arm | Parse | Registered execution | Supplied execution | Answer-cap contact |
|---|---|---|---|---|
| systematic | 87/96 (0.9062) | 84/96 (0.8750) | 84/96 (0.8750) | 9/96 (0.0938) |
| deranged | 87/96 (0.9062) | 0/96 (0.0000) | 84/96 (0.8750) | 10/96 (0.1042) |
| duplicate | 70/96 (0.7292) | 4/96 (0.0417) | 70/96 (0.7292) | 22/96 (0.2292) |
| placebo | 51/96 (0.5312) | 0/96 (0.0000) | 0/96 (0.0000) | 39/96 (0.4062) |
The adherence gate passed globally and separately in all four contexts. The systematic-minus-deranged registered-operation delta was +0.875, with successes covering all 24 bound operations and all four contexts. The symmetric 84/96 supplied-operation rate in systematic and deranged is especially informative: the model followed the actual injected text rather than a task-row shortcut. Duplicate and placebo registered-target rates stayed below their 0.20 caps.
The primary interface gate nevertheless failed. Every diagnostic arm was required to parse at least 0.90 and contact the answer cap at most 0.05. The systematic and deranged parse rates barely passed, but both cap rates failed; duplicate and placebo failed both requirements. All diagnostic parse errors had the frozen result_answer_shape category, so no post-hoc parser repair is allowed.
The separate eight-row full-program ceiling had parse 8/8 and zero cap contact, but visible correctness was only 3/8 against a 4/8 requirement. Its four parameterized cases reached 2/4, exactly meeting their stratum threshold, while only 1/4 parameter-free cases passed. The registered first operation appeared as the generated first operation in 5/8 proposals, but only three of those completed the correct visible mapping. This ceiling is noncausal reachability only; even a pass would not establish that the hint caused correctness.
The frozen decision order therefore emits INVALID_INTERFACE_PARSE before the independent NO_CORRECT_HYPOTHESIS_CEILING failure. Both failures are real, and neither may be repaired by changing this result-bearing experiment. The summary sets qualification_authorized=false; no qualification, confirmation, training, or benchmark call occurred.
Teardown anomaly and recovery
After all five immutable COMPLETE receipts were written, the same process's automatic analysis attempt falsely reported installed packaging 26.0 rather than locked 26.2. vLLM's extension path had imported setuptools, appending setuptools/_vendor—which contains packaging-26.0.dist-info—to sys.path. The analyzer's last-write-wins package scan selected that vendored duplicate. Generation metadata had snapshotted the real 26.2 environment before this mutation. Independent forensics recomputed every transaction hash, and the preregistered fresh-process --stage analyze path then authenticated and scored the completed rows without constructing a model or resampling. This is an infrastructure false abort after generation, not a sample-integrity defect.
Interpretation
The exact early hypothesis is a real, broad behavioral address for direct one-operation execution under the tested prompt: replacing it with a deranged hypothesis redirects the output to the deranged operation. That is stronger than the predecessor's late next-token write. But it is not yet a proposal controller: complete two-step reachability remains below gate, and neutral or duplicated branches destabilize the direct-result ABI. The warranted successor must make the consequence of each bound hypothesis explicit—such as by materializing residualized examples—then establish a fresh interface gate and beat matched-compute sampling. Simply increasing budgets, relaxing the parser, or running the sealed qualification would be post-hoc.
Controls and claim boundary
This experiment supports no deployable gain. Its positive adherence subgate is diagnostic and scoped to direct one-operation outputs. It is not installed capability, J-space transport, autonomous solving, or a win over CPU search. Any follow-up must be a new experiment and retain late timing, duplicate, placebo, and sampled/logical-token-matched ordinary-sampling controls.
Artifact manifest
All raw outputs, scored rows, transaction receipts, and summaries are small and tracked in-tree. reports/artifact_manifest.yaml records no external or omitted artifact.
Experiment log 7
Show the running log (7 entries, 2026-07-13)
2026-07-13 — Scaffold
Created from synchronized origin/main as the deployable successor to the terminal-invalid late semantic-anchor experiment. The initial design freezes early systematic, early duplicate, late systematic, and matched sample-more arms; no model has been loaded.
2026-07-13 — Adversarial redesign before GPU use
Two independent reviews rejected the initial type-only bank. The frozen design now supplies all 24 bound operations, exhaustively audits the 24² grammar, requires visible-equivalence of every public-data fit, uses a strict Python AST answer ABI, and adds independent-prefix equal-total and equal-post late arms. Duplicate, exact-scaffold placebo, neutral/plain matched-sampling, and CPU exhaustive controls are mandatory. Gold-mutation, resource-matching, composed- map, and token-stitching audits were promoted to pre-GPU gates. No model outcome was observed before these changes.
2026-07-13 — Refreshed CPU smoke passes
Regenerated the complete 48/96 split after bound-operation hardening. The smoke exhausted all 576 programs per task, found zero readable-ancestor behavior collisions, verified 24 distinct consequences in each of four diagnostics, serialized 144 unique composed branch maps with balanced gold slots, and froze the pre-grade mutation/resource firewall. The experiment-local test suite passed 31 tests and 33 parameterized subtests. model_loaded=false, outcomes_loaded=false, and all model stages remain fail closed.
2026-07-13 — Pre-model mechanics amendment
Implementation-level adversarial review found that the unspecified four-case program ceiling happened to cover only parameter-free first operations. Before any model construction, generation, or outcome, the design was amended to eight cases: four parameter-free plus add_k(-2), mul_k(3), take_k(3), and rotate_k(2). The ceiling now requires .50 visible pass overall and within the parameterized stratum, strict .90 parse, and at most .05 cap contact. It is explicitly non-causal reachability evidence. The amendment also freezes per-context adherence gates, exact terminal-token matching for padded controls, authenticated receipt-last generation, and a conservative live KV no-preemption gate. No threshold was relaxed.
2026-07-13 — Adversarial implementation audit passes
Three independent code reviews found and closed raw-authentication, lock, terminal-padding, context-aggregation, parameter-coverage, live-capacity, and crash-resume defects before model construction. Prepared prompts are now independently rebuilt; raw text must equal decoded token IDs; natural and forced continuation accounting, seeds, exact engine/sampling settings, and the complete pinned package set are authenticated. Full package parity is checked before tokenizer or engine construction and again against generation metadata. Receipt-last transactions permit verification-only finalization but never resample a started-only invocation. The deterministic preparation froze four 96-row diagnostic arms and eight program cases with receipt SHA-256 2d6b668a6d43e1bd657124c3645d85ea9996d9aaaea8f81225b97472a2f5b292. All 39 tests plus 33 parameterized subtests pass, refreshed smoke remains CPU_SMOKE_PASS, and model_loaded=false, outcomes_loaded=false.
2026-07-13 — Implementation lock frozen
The separately published implementation commit is a7bd9fbe093b3f02b3ebdecd5ab533b816e133b3. The receipt-last lock binds that commit, the original design, amendment af9c8431, and the exact 20-file critical allowlist. Lock SHA-256 is 12c298aef3ba9cc83bd4d1cdadc304aa0daa7d71bde4be3fce81514fdf4b3148 and records model_calls_before_lock=0. Live mechanics remains unrun at this boundary.
2026-07-13 — Authenticated mechanics stops before qualification
The first locked live call passed engine preflight and wrote immutable complete receipts for systematic, deranged, duplicate, and placebo (96 rows each) and the eight-case program ceiling. Independent forensic review recomputed all request/raw/metadata/receipt hashes and found no partial or mutable transaction. The authentication receipt is MECHANICS_AUTHENTICATION_PASS.
Early bound text exerted broad direct control: systematic executed the injected registered operation on 84/96 rows, deranged executed that registered operation on 0/96 but its own supplied operation on 84/96, and all 24 operations and four contexts had support. Duplicate registered execution was 4/96 and placebo 0/96. The preregistered adherence subgate passed.
The formal outcome is nevertheless terminal INVALID_INTERFACE_PARSE. Systematic/deranged answer-cap contacts were 9/96 and 10/96 against a maximum of 0.05; duplicate/placebo parse was 70/96 and 51/96 and cap contact was 22/96 and 39/96. Independently, the noncausal full-program ceiling reached only 3/8 visible passes versus the required 4/8, with its parameterized stratum at 2/4. Qualification and confirmation remain sealed.
The automatic same-process analysis initially false-aborted after engine teardown because setuptools/_vendor introduced a duplicate packaging 26.0 distribution after generation. Every generation sidecar had already recorded the real locked 26.2 package set. The documented fresh-process analysis-only stage authenticated and scored existing receipts without a model call or resample. This operational footgun is now documented in docs/vllm_inference.md.
Data files 2
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
Reproduce
Smoke test
.venv/bin/python experiments/qwen35_4b_early_text_hypothesis_forking/scripts/run.py --stage smokeFull run
PYTHONDONTWRITEBYTECODE=1 .venv-vllm/bin/python experiments/qwen35_4b_early_text_hypothesis_forking/scripts/run_mechanics.py --stage run --lock experiments/qwen35_4b_early_text_hypothesis_forking/runs/mechanics/implementation_lock.jsonRun steps are documented inside the experiment folder (README and scripts).