Qwen3.5-4B Materialized Residual Sibling Search
The one idea you need
Imagine trying every possible first move, writing down the concrete intermediate state each move creates, and asking the same solver to finish from each state. Public examples then choose among the finished candidates.
The question
Can concrete intermediate states help a small model finish multi-step rules better than names alone or the same spending on ordinary attempts?
What we found
Not yet. The scientific design and every model-free construction check pass, but the model has not run. The frozen test contains 264 fresh functions, and all 38,596 planned prompt renderings fit their assigned context limits.
Why it matters
The controls prevent extra attempts, a cheaper exhaustive solver, or a broken answer format from masquerading as an improvement. Treat this as a trustworthy launch pad, not capability evidence.
On this page
Results at a glance 1
How to read
The horizontal groups count tasks with one, two, three, or four publicly valid first moves; bar height is the number of frozen tasks. No height is better—the planned 8:8:4:4 balance repeats in every block.
Takeaway → The first two bars match at 88 and the next two match at 44, confirming the planned ambiguity balance across 264 tasks.
Data table
| publicly enumerable first-operation residuals | fresh exact-depth-three tasks |
|---|---|
| 1 live sibling | 88 |
| 2 live siblings | 88 |
| 3 live siblings | 44 |
| 4 live siblings | 44 |
Numbers from experiments/qwen35_4b_materialized_residual_sibling_search/data/procedural/manifest.json
Technical framing
Frozen construction spans four levels of residual ambiguity — Model-free construction receipt only. Every 24-task block has an exact 8/8/4/4 live-sibling balance; all 264 common-panel function fingerprints are unique, and no model was loaded or called.
In the author’s words from the Overview · “Results”
The model-free smoke passed on the frozen 264-task construction: 24 mechanics, 48 qualification, and 192 confirmation tasks. It found 3,525 eligible exact depth-three function fingerprints, rebuilt the split deterministically, independently re-audited public-live sets on 34 registered tasks, and obtained 0.966 compound pass probability at the registered confirmation alternative. The 38,596 exact rendered prompts across every frozen task, candidate, and condition span 259 to 941 tokens; the 259-token minimum is the short supplied echo ceiling. The receipt records zero model loads, zero model calls, and no benchmark reads. This is a construction and design result only; no Qwen3.5-4B capability result exists. … Read the full result →
Overview
Outcome: sealed without a durable, authenticated capability result. Attempt 1 aborted in live preflight; attempt 2 reached one 52-request invocation but lost all returned rows before durable persistence when termination metadata authentication failed.
Research Program
- Primary:
structured_execution_and_compilers. - Secondary:
evidence_conditioned_selection,interpretability_and_diagnostics, andtest_time_reasoning_budget. - Immediate parent:
qwen35_4b_early_text_hypothesis_forking. - Closest algorithmic near-duplicate:
qwen35_4b_decompose_compose_frontier. - Closest representation-negative:
qwen35_4b_partial_structure_search.
Question
On fresh exact-depth-three tasks, can Qwen3.5-4B complete a useful two-step residual when an external interpreter materializes each candidate first operation's public consequences? Does the resulting all-sibling explorer beat candidate names and candidate-blind sampling at taskwise matched sampled and logical model tokens? Secondarily, can a cheap no-think viability score retain most of the all-sibling coverage with only four completions?
Hypothesis
The parent found broad one-step semantic routing but almost no downstream composition. Here each candidate h produces a concrete relation h(x_i) -> y_i, reducing a depth-three inverse problem to a depth-two suffix. The primary test does not pretend the model can discover h: it treats all 24 siblings symmetrically and measures whether materialized consequences change the suffix proposal distribution. A shuffled state/target alignment preserves the treatment's token multiset while destroying its semantics.
Setup
- Model: only
Qwen/Qwen3.5-4Bat revision851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a, bf16. - Backend: the experiment-local pinned vLLM runner for every model arm; no backend mixing and no training in this experiment.
- Tasks: fresh procedural integer-list functions whose output signature on a frozen common panel is realizable at depth three but not at depth zero, one, or two. Functions, concrete triples, and registered suffixes are disjoint across splits.
- Inputs: list lengths four through eight and IID discrete-uniform integer values from -9 through 9; within-list duplicates are allowed, while every visible, hidden, and probe input in a task is distinct.
- Splits: 24 public mechanics tasks, 48 qualification tasks in two frozen shards, and 192 untouched confirmation tasks in eight frozen shards.
- Public-live sibling: a candidate first operation for which at least one of the 24² legal suffixes fits every visible row. This is a deployable, model-free label, not hidden gold. Every 24-task block has exactly 8/8/4/4 tasks with one/two/three/four public-live siblings.
- Primary treatment: generate one strict two-operation suffix from every one of the 24 materialized sibling prompts, assemble full programs, and select using visible executions and independently generated unlabeled probes.
- Primary controls: all-24 name-only siblings; all-24 task-hash-deranged state/target alignments; candidate-blind full-program sampling matched taskwise at conservative sampled-token and logical-token first-over points; exact public viability; and exhaustive CPU enumeration as an explicit dominance reference.
- Ranking secondary: no-think targeted raw log probabilities for binary materialized viability, candidate-name viability, the deranged relation, and a C25-style listwise original-I/O next-operation scorer. Any top-four policy gets its own real four-request generation run; all-24 outputs are not reused.
- Primary metrics: hidden-correct proposal coverage and visible-only selected hidden accuracy. Ranking uses within-task live recall@4 and live hit@4.
- Hidden boundary: hidden outputs may score only already frozen pools and selections. Prompts, stopping, resource matching, and selected IDs use visible examples, deterministic candidate states, unlabeled probe inputs, and frozen hashes only.
Construction and execution boundaries
The DSL has one shared typed INVALID result for any illegal or safety-bound execution. Empty or invalid target trajectories are rejected. Candidate executions that become empty or invalid are simply ineligible; model-produced code is never executed. The 24 concrete operation aliases and both two-op and three-op parsers are frozen before model use.
Admission never filters on hidden candidate correctness or selector-probe agreement. This keeps held-out selection fallible. A frozen common-input function fingerprint, rather than a hash of each task's own rows, prevents the same function from appearing in multiple splits.
Stages and stop logic
- CPU smoke proves exact minimum depth, common-panel and split disjointness, public-live enumeration, live-count balance, candidate-state distinction, partial-operation semantics, strict assembly/execution, selector blindness, taskwise resource matching, and key discrete threshold geometry.
- Public mechanics runs two independent checks: a live-sibling suffix/direct ABI ceiling and no-think sibling ranking. Interface failure seals all model qualification. Suffix failure seals the all-24 explorer. Ranking failure seals only the top-four efficiency secondary.
- Qualification compares the all-24 materialized explorer against every primary control on 48 new tasks using point and shard-consistency futility gates. It makes no claim-grade significance statement. The optional top-four decision is separate and cannot veto primary confirmation.
- Confirmation repeats the untouched frozen protocol on 192 tasks. The sole claim-grade family is selected accuracy versus name-only, shuffled-state, and the two conservative direct first-over baselines, using exact paired one-sided McNemar tests with Holm familywise correction. Any top-four result is a descriptive operational secondary and cannot alter the primary result.
- A replicated pass supplies positive prior evidence for a separate residual-policy supervision experiment. Failure seals this untrained interface, not the logically independent training question.
Run
Model-free smoke:
.venv/bin/python experiments/qwen35_4b_materialized_residual_sibling_search/scripts/run.py --stage smokeHistorical mechanics command (sealed permanently because attempt 2 has a terminal STARTED transaction; do not rerun):
.venv-vllm/bin/python experiments/qwen35_4b_materialized_residual_sibling_search/scripts/run_mechanics.py --stage runResults
The model-free smoke passed on the frozen 264-task construction: 24 mechanics, 48 qualification, and 192 confirmation tasks. It found 3,525 eligible exact depth-three function fingerprints, rebuilt the split deterministically, independently re-audited public-live sets on 34 registered tasks, and obtained 0.966 compound pass probability at the registered confirmation alternative. The 38,596 exact rendered prompts across every frozen task, candidate, and condition span 259 to 941 tokens; the 259-token minimum is the short supplied echo ceiling. The receipt records zero model loads, zero model calls, and no benchmark reads. This is a construction and design result only; no Qwen3.5-4B capability result exists.
The first live attempt initialized the exact engine but failed its cache receipt before the first experimental generation request. No invocation transaction or sampled output exists. The failed preflight is preserved as incident evidence; it is not a capability result and its embedded PASS label is unauthenticated.
After the append-only repair and lock were independently reviewed, pushed, and green in CI, attempt 2 passed the corrected cache preflight. The first suffix_materialized invocation returned all 52 rows in memory, but its post-generation authenticator falsely required the model EOS ID 248044 in the tokenizer-EOS receipt field. The pinned tokenizer correctly reports <|im_end|> as EOS ID 248046. Authentication therefore failed before raw rows or metadata were written. No output text was printed or inspected, no later invocation began, and no sampled bytes are recoverable. The immutable STARTED receipt forbids replay, so this experiment ends without a durable, authenticated model result.
Interpretation
The scientific design remains untested, but this experiment instance is not eligible for another run. A valid successor must use a new experiment, fresh task/record identities and sampling seeds, and durable write-before- semantic-authentication quarantine. Even a later successor pass would be an external structured-search result. CPU 24³ enumeration remains exact and cheap at this depth.
Knowledgebase Update
- Program evidence updated: incident recorded; scientific belief unchanged.
- Program backlog updated: fresh-identity successor required.
- Claim ledger updated: no; no result exists.
Artifacts
src/scripts/configs/data/runs/analysis/reports/reports/artifact_manifest.yamlidea_intake.mdreports/preregistration.mdreports/design_review.md
Report
Rendered from reports/report.md
Summary
No durable, authenticated model result exists. The first design was rejected and rewritten before implementation. The replacement passed adversarial review and its model-free construction/protocol smoke. Two live mechanics attempts then ended in preserved infrastructure incidents: one before any experimental request and one after 52 returned rows but before durable output persistence. The terminal second transaction permanently seals this experiment.
Research Program Fit
This is a gated successor to qwen35_4b_early_text_hypothesis_forking. It asks whether externally computed candidate consequences improve residual proposals when all siblings are completed, with cheap ranking only a secondary. It is narrower than J-space and does not claim an internal coordinate, certainty signal, or installed capability.
Method
The locked design uses 264 fresh exact-depth-three integer-list functions with one to four publicly enumerable live first-operation siblings. The primary policy completes all 24 materialized residuals and compares visible/probe-only selection against all-24 name-only and deranged controls plus taskwise sampled/logical-first-over direct sampling. Cheap no-think sibling ranking is a separate descriptive efficiency secondary and cannot veto the primary.
Results
The CPU smoke passed. It found 3,525 eligible exact-depth-three function fingerprints, filled 24/48/192 mechanics/qualification/confirmation tasks with global function/triple/suffix disjointness, and independently re-audited 34 public-live sets. The complete tokenizer audit covers every task, all 24 candidates, and every prompt family; all 38,596 rendered prompts span 259 to 941 tokens and fit their condition-specific 4,096-token context reserves. The registered compound confirmation rule passed 483/500 simulations at the design alternative. Receipts record zero model loads, zero model calls, and no benchmark reads.
The first authorized model attempt initialized the engine but failed an incorrect hybrid-cache receipt before any experimental request. Its append-only repair passed review and CI. The second attempt passed live preflight and the first suffix_materialized invocation returned 52 rows in memory. A post-generation authenticator then falsely required tokenizer EOS 248044; the pinned tokenizer correctly declares <|im_end|> EOS 248046, distinct from the model-config EOS 248044. Because raw writes followed semantic authentication, zero sampled-output bytes survived. No output was inspected and no later invocation began. Independent review returned BLOCK on replay.
Controls
The locked design includes all-24 candidate-name and token-preserving state/target-derangement controls, exact public enumeration, CPU exhaustive search, and taskwise sampled/logical-first-over candidate-blind sampling. Ranking secondaries include original-I/O listwise, surface, and random ranks.
Oracle Versus Deployable Evidence
Public-live sibling identity is deployably computable by exact visible enumeration and is not called an oracle. Hidden outputs and hidden proposal coverage remain scoring fields. Ranking, stopping, selection, and resource matching use only visible examples, deterministic candidate states, unlabeled probe inputs, and frozen hashes.
Interpretation
This result validates only construction and protocol. It is not evidence that materialization improves or harms Qwen3.5-4B. The second attempt is an interface-invalid, terminal STARTED transaction, not a negative capability result. Recovery requires a new registered experiment with fresh task/record identities and sampling seeds plus durable quarantine before semantic authentication.
Next Experiments
First run a fresh-identity mechanics successor that changes no scientific arm or outcome gate. A replicated search pass there would license a separate, disjoint residual-policy supervision experiment. A genuine mechanics failure would retire this inference-time interface without ruling out supervised installation; the bookkeeping incident here does not.
Artifact Manifest
artifact_manifest.yaml records the local smoke, both locks and preflights, the terminal attempt-2 STARTED receipt, incident receipt, and adversarial review. No external, omitted, or model result artifact exists.
Experiment log 2
Show the running log (2 entries)
Scaffold
- 2026-07-13: Created as a new experiment scaffold under
structured_execution_and_compilers. - 2026-07-13: Read the immediate parent and closest related lines through the repository discovery workflow. Narrowed novelty to symmetric materialized sibling ranking and residual completion; interpreter materialization itself is prior art.
- 2026-07-13: Drafted the preregistration and launched three independent read-only adversarial reviews covering statistics, task construction, and scientific/resource confounds. No model call was made.
- 2026-07-13: All three reviews rejected the first draft. The required unique-first balance was impossible for
negateandtake_k(1); hidden selection was tautological; a 512-token-per-sibling ranker was dominated by simply completing all siblings; suffix reachability was untested; and the registered qualification inference was underpowered. - 2026-07-13: Rewrote the design around multi-label public-live siblings, common-panel function fingerprints, non-filtered hidden/probe outcomes, all-24 materialized completion as the primary explorer, cheap no-think raw log-probability ranking as a secondary, a public suffix ABI ceiling, taskwise first-over resource matching, qualification-only futility gates, and a 192-task paired confirmation family. Launched a second adversarial read before implementation. No model call was made.
- 2026-07-13: Implemented and ran the model-free construction/protocol smoke. It deterministically filled 264 globally disjoint exact-depth-three tasks, independently re-audited 34 public-live sets, verified strict protocol and taskwise resource machinery, and simulated 0.966 compound confirmation pass probability at the registered design alternative. No model was loaded or called and no benchmark content was read.
- 2026-07-13: The second adversarial read found and closed runner, resource, interface, statistical, and documentation blockers. In particular, raw 24-way log probabilities are authenticated; mechanics covers every live sibling; top-four runs are independent; shuffled alignment is in the four-comparator confirmation family; and optional top-four failures cannot veto the primary all-24 decisions.
- 2026-07-13: Expanded tokenizer smoke from a mechanics subset to every frozen task, all 24 candidates, and every prompt family with condition-specific reserves and prompt/token-ID hashes. Bound the design documents, configuration, source, and tests into the final model-free receipt.
- 2026-07-13: Three independent reviewers accepted the construction layer after the final fixes. Published design/data lock still authorizes no model call; mechanics implementation requires a separate audit and lock.
- 2026-07-13: Implemented the complete mechanics boundary and passed two independent adversarial code audits. The frozen candidate reconstructs 1,984 requests, 24 surface-control folds, and 4,032 targeted raw-logprob values; authenticates all 189 pinned environment distributions plus tokenizer, runtime, transaction, and result-chain receipts; and made zero model calls. Model authorization remains withheld until the prepared artifacts are pushed and a separate implementation lock is then published and pushed.
- 2026-07-13: Ran deterministic mechanics preparation twice under the pinned vLLM environment. Both passes accepted the same 1,984-request inventory and existing bytes; the committed preoutcome receipt records zero model loads and calls and no hidden, qualification, confirmation, or benchmark reads.
- 2026-07-13: Pushed the reviewed implementation at
48ef078f, observed both repository-validation and research-site CI succeed, and generated a separate mechanics-only implementation lock binding that pushed commit and every critical source/prepared hash. No model was loaded or called; execution remains sealed until this lock is independently committed and pushed. - 2026-07-13: Pushed the separate lock at
cd82e649and observed both CI workflows pass. The exact engine then initialized, but the live preflight aborted before the first experimental generation request because the validator tried to invert vLLM's intentionally floored group-aware token capacity. No armSTARTEDreceipt, model output, score, or summary exists; ordinary internal engine profiling/warmup did occur. - 2026-07-13: Preserved the failed preflight byte-for-byte and opened an append-only v2 repair. Independent incident audits required versioned active paths, an incident-bound lock that discloses the prior engine initialization, the exact 11-block Qwen hybrid-cache identity, conservative block-based arm fit, and validation before publishing a PASS receipt. Retry remains sealed pending a reviewed, committed, pushed v2 implementation and separate lock.
- 2026-07-13: Closed every v2 repair blocker. The stable candidate directly binds all attempt-1 evidence, uses authoritative floored capacity and exact 11-block hybrid geometry, rejects the 703/704-block boundary, validates before writing PASS, isolates versioned active state, and normalizes only expected Git metadata while keeping the scientific runtime exact. Two independent final-byte adversaries returned
FREEZE; 45 mechanics and 71 full experiment tests passed, and source-bound preparation reproduced twice with zero new model loads or calls. - 2026-07-13: Pushed the frozen v2 repair at
fa942eefafter rebasing three concurrentmaincommits; both CI workflows passed. Generated the separate v2 lock binding 33 critical files and explicitly recording one prior engine initialization, zero experimental requests, and zero sampled outputs. Retry remains sealed until that lock is itself committed, pushed, and green in CI. - 2026-07-13: Pushed the separate v2 lock at
6629de7f; repository-validation and research-site CI both passed. From a clean synchronized tree and idle GPU, attempt 2 initialized the exact pinned engine and passed the corrected live hybrid-cache preflight. - 2026-07-13: The first
suffix_materializedinvocation returned 52 rows and metadata in memory, then failed termination authentication before durable raw/metadata writes. The runner correctly distinguished model EOS248044from tokenizer<|im_end|>EOS248046; the experiment authenticator and its fake-tokenizer test incorrectly required248044for both. No output was printed or inspected, no later invocation began, and no returned bytes are recoverable. TheSTARTEDtransaction is terminal and will not be replayed. This experiment is sealed without a durable, authenticated model result.
Pending
- Publish the attempt-2 incident and open a separately reviewed successor with fresh task/record identities, fresh sampling seeds, and durable quarantine before semantic authentication.
Data files 1
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
Reproduce
Smoke test
.venv/bin/python experiments/qwen35_4b_materialized_residual_sibling_search/scripts/run.py --stage smokeFull run
.venv-vllm/bin/python experiments/qwen35_4b_materialized_residual_sibling_search/scripts/run_mechanics.py --stage runRun steps are documented inside the experiment folder (README and scripts).