Qwen3.5-4B Materialized Residual Answer-Seam Factorial

Results at a glance 1

Overview

Terminal 2026-07-14: NO_VALID_RESIDUAL_ANSWER_SEAM; mechanics remained sealed.

The registered answer interface failed before residual mechanics opened. All 240 calibration outputs authenticated, but every arm scored 0/48 strict parses and exact echoes. After removing only the terminal <|im_end|> plus newline, the two no-think arms pass the frozen exact parser on 48/48 rows. Thinking arms remain partly invalid because some emit another </think> boundary. This is evidence for a fresh native answer-stage commit-boundary experiment, not permission to repair this result in place.

Research Program

• Program: structured_execution_and_compilers
• Program question: can externalized intermediate state become a deployable compositional interface rather than only local semantic routing?
• Prior anchors: qwen35_4b_materialized_residual_sibling_search_fresh_replication, qwen35_4b_commit_slot_semantic_power_replication, and qwen35_4b_early_text_hypothesis_forking.

Question

Can a separately calibrated, autonomously scored short answer seam make materialized residual generation measurable, and if so does it beat name-only, token-preserving shuffled, and matched-compute direct sampling controls?

Hypothesis

The prior model could copy a structured echo on 20/24 non-cap rows but every thought hit its cap. A complete 2x2 crossing think@512/no-think with freeform/literal-PROGRAM: prefill can isolate answer syntax from reasoning policy without supplying answer identity. A separately calibrated policy may expose residual completion that the invalid free-form interface hid.

Setup

• Model: only Qwen/Qwen3.5-4B revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a, bf16.
• Dataset/task source: fresh procedural exact-depth-three list transforms; never benchmarks/.
• Split: 48 known-answer interface-calibration tasks and 24 disjoint mechanics tasks, with new task/request/seed domains.
• Interface arms: think512_freeform, think512_program_slot, no_think_freeform, and no_think_program_slot; all answer aliases are sampled autonomously with the same 24-token tail cap.
• Baseline: taskwise matched-compute direct full-program sampling on the same backend and selected interface budget.
• Controls: name-only siblings, task-hash shuffled materialized states/targets, exact echo, candidate-blind direct sampling, and exhaustive CPU ceiling.
• Primary metric: hidden exact accuracy of the pre-hidden visible-only selector, gated behind the interface calibration.
• Oracle-only diagnostics: all-sibling hidden-correct proposal coverage, exact candidate viability, and hidden program success; none may affect interface choice, prompts, budgets, or selected IDs.
• Calibration gates: >=44/48 exact echoes, >=44/48 parses, <=2/48 answer-cap contacts, plus >=22/24 exact/parse and <=1/24 cap contacts in each arity.
• Winner: first qualifier in the fixed least-departure priority, never the best observed metric.
• Hidden-label boundary: mechanics remains inaccessible until a committed winner receipt and second lock; hidden scoring remains inaccessible until a committed visible-selection receipt. Qualification/confirmation and all benchmark content remain unread.

Run

Smoke:

python experiments/qwen35_4b_materialized_residual_answer_seam_factorial/scripts/run.py --smoke

Fresh construction and append-only v2 smoke:

python experiments/qwen35_4b_materialized_residual_answer_seam_factorial/scripts/construct.py
python experiments/qwen35_4b_materialized_residual_answer_seam_factorial/scripts/run.py --design-smoke

Full:

.venv-vllm/bin/python experiments/qwen35_4b_materialized_residual_answer_seam_factorial/scripts/run_calibration.py --stage lock
.venv-vllm/bin/python experiments/qwen35_4b_materialized_residual_answer_seam_factorial/scripts/run_calibration.py --stage run

Results

The exact reviewed implementation received PASS_RELEASE_LIVE_CALLS; its calibration lock was committed and passed both workflows before the live run. The five durable transactions then authenticated 240/240 outputs with exact shared-thought and answer-seed pairing. The fixed gate returned NO_VALID_RESIDUAL_ANSWER_SEAM:

• all four arms: 0/48 strict parses and 0/48 exact echoes;
• think512_freeform: 48/48 thought-cap and 18/48 answer-cap contacts;
• think512_program_slot: 48/48 thought-cap and 0/48 answer-cap contacts; and
• both no-think arms: zero thought/answer-cap contacts.

Mechanics, qualification, confirmation, hidden, and benchmark reads stayed empty. A frozen post-decision diagnostic removed only the exact decoded <|im_end|>\n terminal suffix. The frozen parser then accepted 24/48 think/freeform, 38/48 think/PROGRAM:, and 48/48 in each no-think arm. A looser expected-answer-tail diagnostic was 29/48, 48/48, 48/48, and 48/48 respectively, but it hides five/ten additional thinking-close boundaries and is not full- string exactness. Neither diagnostic changes the registered zero-parse result.

Interpretation

The tested strict HF-model-EOS interface is invalid, so residualization remains unadjudicated. The negative is not evidence that the no-think model lacked the requested program: both no-think arms became 48/48 full-string exact after one post hoc terminal-suffix removal. Thinking has an additional close-boundary failure. The only warranted continuation is a new experiment that registers first tokenizer EOS as the answer-stage deployment commit event on fresh tasks, keeps strict parsing, and reruns matched HF-EOS and malformed-terminator controls. No mechanics stage from this experiment may be opened.

Knowledgebase Update

• Program evidence updated: yes; strict interface failure and tokenizer-EOS diagnostic recorded.
• Program backlog updated: yes; fresh commit-boundary successor required.
• Claim ledger updated: no; this is a single terminal interface result, not a capability claim.

Artifacts

• src/
• scripts/
• configs/
• runs/
• runs/calibration/decision.json
• analysis/calibration_terminal_diagnostic.json
• runs/smoke/summary.json
• reports/
• reports/artifact_manifest.yaml

Report

Rendered from reports/report.md

Summary

The registered calibration terminated at NO_VALID_RESIDUAL_ANSWER_SEAM, so mechanics never opened. All 240 outputs in five durable transactions authenticated, but every arm scored 0/48 strict parses and exact echoes.

This is a clean interface negative, not a residual-mechanics result. A frozen post-decision diagnostic found that both no-think arms became 48/48 exact under the frozen parser after removal of only tokenizer EOS <|im_end|> plus newline. Thinking arms retained additional close-boundary failures. The runner intentionally waited for the later HF model EOS, so the registered parser correctly rejected the extra terminal bytes. This licenses a fresh answer-stage tokenizer-EOS experiment; it does not change this experiment's decision.

Research Program Fit

The closest parent cleanly falsified its cheap viability ranker but could not adjudicate materialized residual generation because every thought hit cap and the answer ABI failed. This experiment tested that prerequisite on separate known-answer tasks before reopening composition. It neither revives the failed ranker nor treats echo as a capability result.

Method

Fresh construction uses the same 24-operation list DSL but a new namespace and seeds. Exact common-panel functions exclude depth <=2 behavior. Every accepted task exhaustively enumerates 24 first operations x 576 two-operation suffixes, giving exact public-live labels. The 72 tasks are split 48 calibration / 24 mechanics with unique public functions, target triples, target suffixes, and input rows.

Calibration crosses think@512/no-think with freeform/PROGRAM: prefill. It uses 48 mechanics-length echo rows balanced across A-X in every answer position. All aliases remain autonomously sampled. The fixed winner is the first gate passer in preregistered least-departure order.

Mechanics would have generated all 24 residual siblings under materialized, name-only, and shuffled evidence plus a frozen 96-sample direct pool per task. The calibration gate failed, so none of those prepared mechanics requests or protected labels were opened.

Results

Construction summary SHA-256: b39e0ad1ccf49503eb48353eac118500432953f32ad27ae2acc1448ed99f622d.

• 48 calibration and 24 mechanics tasks;
• 24/24/12/12 single/double/triple/quad public-live strata overall;
• 72 unique public-instance fingerprints;
• zero overlap with 264 authenticated parent public instances;
• 4,104 prepared rows and 2,952 unique canonical request IDs;
• three suffix controls share exact IDs/order;
• zero parent request-ID, seed-key, or user-prompt overlap;
• every A-X alias appears once in each calibration answer position;
• real-tokenizer receipt 61ff7292... authenticates PROGRAM: [78041,25], close [248069,271], all 14,400 canonical answer compositions, 1,396-token worst-case context, and zero overlap with 1,984 parent rendered prompts;
• a reviewed implementation lock with zero prior model calls/outputs and empty protected-read receipts; and
• 82/82 model-free tests before execution, including append-only transaction and crash-recovery mutations.

After five exact-hash adversarial reviews, two independent archive audits returned PASS_RELEASE_LIVE_CALLS. The lock-bearing commit passed both GitHub workflows before calibration. The live run then authenticated:

• five registered invocations and 240/240 sampled outputs;
• exact shared thought-token pairing across the two thinking continuations;
• exact answer-seed pairing and registered prefix assignment; and
• empty benchmark, mechanics, qualification, confirmation, and hidden reads.

The frozen gate metrics were:

No arm approached the registered >=44/48 exact/parse gates. The fixed winner is therefore null and NO_VALID_RESIDUAL_ANSWER_SEAM is terminal.

Post-decision terminal-boundary diagnostic

This diagnostic is report-only and cannot change qualification. It first removes only an exact final decoded <|im_end|>\n, then reruns the frozen full- string parser. A separate expected-tail diagnostic takes the final segment after a thinking close and can therefore hide an additional </think>; it is reported only to locate the requested answer, not as exact output. The sampled terminal sequence is tokenizer EOS 248046, newline 198, and registered HF model EOS 248044.

Thus tokenizer-EOS stopping is a complete post hoc explanation only for the two no-think cells. The thinking cells also have extra close boundaries; freeform thinking adds 18 answer-cap contacts. A fresh successor must treat thinking as a control rather than assume that one stop-token change repairs it.

Controls

The model boundary is exact Qwen/Qwen3.5-4B revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a. Prepared arms freeze backend, requests, ordering, seeds, caps, and direct-pool ceiling. Forbidden benchmark, mechanics, qualification, confirmation, and hidden-read receipts are empty. These are application-level receipts, not OS syscall telemetry. The post-decision diagnostic was computed only from already opened calibration outputs.

Oracle Versus Deployable Evidence

No mechanics outcome exists. The deployable selector, hidden proposal coverage, and exhaustive CPU ceiling all remain unopened because the interface gate failed.

Interpretation

The strict HF-model-EOS answer ABI is invalid for this workload, and residual generation remains unadjudicated. The result should not be summarized as “the model could not copy a program”: both no-think arms became 48/48 exact after removal of one terminal suffix. Nor should the expected-tail diagnostic for thinking be called full-string exactness, or any diagnostic be called a successful interface after the fact: parser, stopping policy, cap accounting, and transaction semantics were registered around the later HF EOS.

The narrow causal hypothesis for the successor is now concrete. Treat tokenizer EOS as an explicit answer-stage deployment commit event, stop at its first sampled occurrence, remove only that registered terminal token, and require every preceding byte to pass the same exact grammar. Fresh tasks, record IDs, and seeds must be used. Controls must include the current HF-EOS boundary, freeform versus literal PROGRAM: prefix, think versus no-think, early/interior/missing terminators, extra pre-commit bytes, exact stop/token/cost authentication, fresh transport, and the same mechanics firewall. The paired no-think cells share rows/seeds and are not independent replications. Only a newly qualified interface may reopen residual mechanics.

Remaining Work

This experiment is complete and terminal. Publish the authenticated negative and create a separate fresh tokenizer-EOS commit-boundary successor. Do not run the existing mechanics lock, transport, generation, visible, or hidden stages.

Artifact Manifest

All construction, calibration transactions, decision, and derived diagnostic are tracked. The manifest records their controlling hashes. There are no external or omitted artifacts.

Experiment log 2

Data files 2

Result tables and metrics copied from the experiment folder — preview inline or open the raw file.

• previewruns/construction/summary.json15 kB
• previewruns/smoke/summary.json696 B

Reproduce

Smoke test

python scripts/run.py --smoke && python scripts/run.py --design-smoke

Full run

.venv-vllm/bin/python experiments/qwen35_4b_materialized_residual_answer_seam_factorial/scripts/run_calibration.py --stage run

Run steps are documented inside the experiment folder (README and scripts).

Browse the experiment folder on GitHub ↗

Related

Related experiments

Nearest by shared name, tags, and programs.

• 2026-07-14 Qwen3.5-4B Tokenizer-EOS Answer Commit Factorial
• 2026-07-13 Qwen3.5-4B Materialized Residual Sibling Search Fresh Replication
• 2026-07-13 Qwen3.5-4B Materialized Residual Sibling Search
• 2026-07-14 Residual-Skill Successful-Sibling Universal Curriculum
• 2026-07-14 Qwen3.5-4B Tokenizer-EOS Residual Mechanics Fresh Replay
• 2026-07-12 Qwen3.5-4B Native-Thought Seam Budget Ladder