Qwen3.5-4B Commit-Slot Semantic Power Replication

Results at a glance 2

Overview

This study tests whether the parent experiment's fixed-1,024 ordered-thought advantage over shuffled thought is a task-general semantic effect rather than a five-task, alias-concentrated near miss.

Research Program

• Primary: interpretability_and_diagnostics.
• Secondary: test_time_reasoning_budget and structured_execution_and_compilers.
• Direct parent: qwen35_4b_commit_slot_jacobian_value_transport, terminal COMMIT_SLOT_SEAM_FAIL.
• J mechanism anchor: qwen35_4b_jacobian_transport_control_replication.

Question

At one fixed 1,024-token thought budget, does ordered native thought reliably improve the next semantic alias choice over both an immediate slot and an exact- length permutation of the same thought tokens, across enough fresh task and alias units to support a later J-space value experiment?

Parent evidence and hypothesis

The parent repaired answer mode: an alias was the unmasked top token on 41/48 long traces. Ordered thought scored 15/48 versus the equivalent 12/48 no-thought and 11/48 shuffled. It passed both pooled gap gates but had five mixed tasks versus six required; task-bootstrap intervals crossed zero and effects were alias concentrated. Post-hoc bias subtraction did not improve the slot.

The narrow hypothesis is that the +8.33pp ordered-over-shuffled task effect is real but underpowered. This replication fixes cap 1,024, expands each seam stage to 113 tasks (339 traces), balances all 11 target operations, and requires both task-level uncertainty and semantic-support gates. It does not change syntax, aliases, decoding, model, task family, or the three-trace policy.

Setup

• Only Qwen/Qwen3.5-4B, revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a.
• Exact replicated 24-coordinate lens, SHA-256 e373b6e93956fdfc5cb446e9bee8249655707c8258a7868f0653d11f1ffd0213, frozen layers 4--8. It remains unused unless the seam replicates.
• Transformers bf16 SDPA, unpadded batch one; cached native generation and exact cache-free full-prefill slot/control logits.
• 322 new exact-depth-two procedural tasks: 113 semantic qualification, 113 untouched semantic confirmation, 48 value fit, and 48 causal confirmation.
• All visible sets have one identifiable first-operation type and no depth-one fit; fingerprints are unique and disjoint from five direct parents.
• Fixed cap 1,024; three traces/task; temperature 0.6, top-p 0.95, top-k 20.
• Policy: append exactly </think>\n\nFirst: and take argmax over the 12 public one-token aliases.
• Controls: immediate no-thought slot, deterministic exact-token-multiset shuffle, unmasked full-vocabulary logits, and same-prefix close-only free-form output.

The slot is a constrained deployment interface. It supplies syntax and a closed vocabulary, never answer identity.

Power and frozen gates

The parent task-level ordered-minus-shuffled mean was 0.08333 with SD 0.35486. A one-sided alpha-0.05 normal planning approximation requires 113 task units for 80% power; both seam stages use exactly 113. The actual decision uses a nonparametric task bootstrap, not the approximation.

Each stage independently requires:

• real slot accuracy in 20%--70%;
• at least 28 tasks with both correct and incorrect real traces;
• at least +3pp over no-thought and +5pp over shuffled thought;
• one-sided 95% task-bootstrap lower bound above zero for real minus shuffled;
• correct successes spanning at least eight target aliases;
• at least eight distinct chosen aliases;
• unmasked top-is-alias rate at least 75% and mean alias mass at least 50%; and
• 100% finite real rows, with every evaluated control finite.

Selection tests only fixed cap 1,024. If it passes, one untouched 113-task confirmation must satisfy the identical gates. Splits may not be pooled to rescue a miss. Only POWERED_COMMIT_SLOT_SEAM_REPLICATED may reopen value-code implementation; all J/value/control/causal commands currently fail closed.

Run

PYTHONDONTWRITEBYTECODE=1 .venv/bin/python -m pytest \
  experiments/qwen35_4b_commit_slot_semantic_power_replication/tests -q
PYTHONDONTWRITEBYTECODE=1 .venv/bin/python \
  experiments/qwen35_4b_commit_slot_semantic_power_replication/scripts/run.py \
  --stage smoke
PYTHONDONTWRITEBYTECODE=1 .venv/bin/python \
  experiments/qwen35_4b_commit_slot_semantic_power_replication/scripts/power_audit.py

After anchoring the design boundary:

.venv/bin/python experiments/qwen35_4b_commit_slot_semantic_power_replication/scripts/run.py --stage model-smoke
.venv/bin/python experiments/qwen35_4b_commit_slot_semantic_power_replication/scripts/run.py --stage seam-selection
.venv/bin/python experiments/qwen35_4b_commit_slot_semantic_power_replication/scripts/run.py --stage seam-confirmation

After the replicated seam and the separately anchored value implementation:

.venv/bin/python experiments/qwen35_4b_commit_slot_semantic_power_replication/scripts/run.py --stage value-model-smoke
.venv/bin/python experiments/qwen35_4b_commit_slot_semantic_power_replication/scripts/run.py --stage prefix-value

The value boundary is anchored to pushed implementation commit ddbc1969. Both value commands have run exactly once. control-calibration and causal-confirmation remain unimplemented and sealed by the negative value decision.

Status

Terminal seam decision: POWERED_COMMIT_SLOT_SEAM_REPLICATED. Qualification and its equally powered untouched confirmation independently passed every frozen gate. All 678 paths remained open to the fixed 1,024 cap and all row/control contracts passed.

The parent hint therefore independently generalizes twice across 226 fresh task units. Ordered thought contributes answer-relevant information beyond an identical shuffled token multiset and the syntax-only no-thought slot. A deterministic post-decision audit gives separate two-sided task-bootstrap intervals of [7.96pp, 19.17pp] and [8.26pp, 21.83pp]; pooling is descriptive only. Correct-answer mention strata do not explain the effect.

The remaining identity nuisance is load-bearing: confirmation had no successful horse target rows, while tiger and river were favored by shuffle. The seam is replicated, but any J/value model must be task-held-out and prove incremental value beyond correct-alias activity, ordinary slot margin, and alias identity. The outcome-blind prefix-value implementation and adversarial audit completed with 16 passing tests and hash-anchored to pushed commit ddbc1969. Neither reserved split was opened before the outcome-blind value-model smoke, which passed with five rank-24 dictionaries, finite 120-wide J/non-J features, and maximum non-J span leakage 2.67e-7; it recorded no outcome or trace text.

The one scientific value run is terminal NO_PREFIX_J_VALUE. All 144 traces and 288 prefix rows were complete and finite, but the shared task-held-out J score was 0.5021 AUC with one-sided task-bootstrap lower 0.4417, missing the 0.65/0.50 gates. It lost to slot margin (0.5448) and equal-width non-J residual features (0.5292). Midpoint prospective AUC was 0.6083, but endpoint AUC reversed to 0.3958, so the frozen shared coordinate cancelled to chance. This phase-specific hint cannot rescue the decision. causal_confirmation remains unopened and all causal stages stay unavailable.

The allowed deterministic post-decision phase audit also rejects the apparent midpoint successor. Refitting the frozen ridge separately by phase reduced the midpoint J AUC to 0.5375 (one-sided task lower 0.4417), below equal-width non-J state at 0.6000 and effectively tied with slot margin at 0.5396. Endpoint J was 0.4292. Centered midpoint/end J coordinates were not stable (mean coordinate correlation -0.0386; mean paired-row cosine -0.0544), and the two fitted J coefficient vectors were nearly orthogonal with a slight negative cosine (-0.0681). This is post hoc and cannot alter NO_PREFIX_J_VALUE; it removes, rather than supports, the rationale for a fresh midpoint-J replication.

Scope

This experiment replicates constrained semantic elicitation but rejects one fixed shared J-value readout. It does not show that no prospective state exists: the registered midpoint slice was above chance while endpoint geometry reversed. It does show that this all-coordinate shared ridge is neither J-specific nor stable enough to license causal patching or capability work. A phase-specific successor must be a new experiment with fresh data and independent replication.

Knowledgebase Update

• Update all three program ledgers and shared synthesis at terminal gates.
• Reserve no claim ID while the claim re-grade remains open.

Artifacts

• assets/context_lens.pt: byte-identical mechanism anchor.
• data/procedural/: four frozen fresh splits and manifest.
• runs/smoke/: CPU, reachability, and power receipts.
• reports/preregistration.md and reports/design_review.md: immutable rules.
• reports/pre_selection_implementation_audit.md: outcome-blind code audit.
• reports/post_confirmation_adversarial_audit.md: post-decision scope and nuisance audit.
• scripts/run.py: fixed-cap seam harness; later stages fail closed.
• runs/seam_selection*.json* and runs/seam_confirmation*.json*: complete, hash-locked passing stages.
• analysis/analyze_replication.py and analysis/replication_audit.json: deterministic stagewise and descriptive cross-stage audit.
• analysis/analyze_prefix_phase.py and analysis/prefix_phase_diagnostics.json: deterministic, post-decision phase audit that cannot alter the registered negative or open causal data.
• configs/prefix_value.yaml, reports/prefix_value_preregistration.md, and reports/pre_value_design_review.md: frozen prospective-value rules.
• reports/pre_value_implementation_audit.md: outcome-blind code/firewall audit covered by the anchored implementation boundary.
• reports/post_value_model_smoke_audit.md: outcome-free context, rank, dimension, non-J geometry, and reserved-data firewall receipt.
• reports/post_prefix_value_adversarial_audit.md: terminal negative scope and the only allowed phase-specific post-decision diagnostics.
• src/coordinates.py and src/value_probe.py: exact coordinate geometry and pure task-held-out analysis; no causal patcher is implemented.
• runs/prefix_value*.json*: complete negative value rows, trace receipt, frozen final fit, and automatic terminal summary. Causal data remain sealed.

Report

Rendered from reports/report.md

Status

Terminal seam result POWERED_COMMIT_SLOT_SEAM_REPLICATED. Qualification and untouched confirmation independently passed every frozen gate. J stages remain sealed pending a new outcome-blind implementation and adversarial audit.

Purpose

The parent fixed slot repaired answer mode and showed a +8.33pp ordered-over- shuffled hint at cap 1,024, but it missed the mixed-task gate and task-level uncertainty crossed zero. This experiment fixes that one cap/interface and tests the hint with 113 fresh task units per seam stage plus bootstrap and diversity gates.

Results

CPU only: 322/322 unique exact-depth tasks, zero overlap with five parents, balanced 10--11 target tasks per seam split, exact lens hash, reachable gates, and seven passing unit tests. Parent-effect planning requires and assigns 113 tasks per stage for approximate power 0.802745; the actual decision uses a 10,000-resample task bootstrap.

No correctness row, scientific trace, or scientific summary exists yet.

After the immutable design boundary, outcome-blind model smoke passed the exact revision, 32-layer/2,560-wide architecture, five rank-24 lens matrices, 12 distinct leading-space aliases, fixed slot tokens [271, 5170, 25], finite logits, and native/free-form cache contracts. Peak allocation was 8,514,319,872 bytes. The receipt stores no correctness, chosen alias, or trace text. A final implementation audit then verified task bootstrap, diversity, exact cardinality, and confirmation hash locks before any scientific run.

Qualification result

The fixed-cap scientific stage completed exactly 339 native traces, 339 real slots, 339 exact-token-multiset shuffled slots, 339 close-only free-form controls, and 113 no-thought slots. Every native path contacted cap 1,024. All cache, finite, exact-shuffle, cardinality, data, design, and power contracts passed.

The no-thought task-bootstrap lower diagnostic was also positive (0.120944), though it was not a powered primary gate. Correct-alias probability averaged 0.266564 under real thought versus 0.164722 shuffled. Alias mentions were not required: only 16.2% of real prefixes contained the correct alias token.

The interface itself remains valid without relying mainly on masking: an alias was already the unmasked top token on 299/339 rows and the 12 aliases held 66.8% mean full-vocabulary probability. Close-only free-form remained much worse: 57/339 parsed, 20/339 were correct, and 310/339 exhausted 16 answer tokens.

Qualification sampled 347,136 native thought tokens in 11,669.621 seconds and processed 492,435 real plus 492,435 shuffled slot-prefill tokens. Peak allocated GPU memory was 8,706,993,152 bytes.

The automatic qualification decision was POWERED_COMMIT_SLOT_SEAM_QUALIFIED at the only registered cap 1,024. It opened exactly one hash-locked confirmation and no J stage by itself.

Independent confirmation result

The untouched confirmation then completed exactly the same 339 native traces, 339 real slots, 339 exact-token-multiset shuffled slots, 339 close-only controls, and 113 no-thought slots. Every path again contacted cap 1,024. No selection row was pooled into a confirmation decision.

The no-thought task lower diagnostic was 0.165192. Correct-alias probability averaged 0.239230 under ordered thought versus 0.153041 shuffled. Only 19.8% of ordered prefixes contained the correct alias token, and the post-decision mention strata were not favorable to copying: success was 26.9% with a mention versus 29.4% without one. The unrestricted next token was already an alias on 297/339 rows, and aliases carried 66.35% mean full-vocabulary mass. Close-only free-form again remained unusable: 55/339 parsed, 20/339 succeeded, and 316/339 exhausted its answer cap.

Confirmation sampled 347,136 native thought tokens in 11,690.539 seconds and processed 491,457 real plus 491,457 shuffled slot-prefill tokens. Peak allocated GPU memory was 8,704,452,608 bytes. The automatic terminal seam decision is POWERED_COMMIT_SLOT_SEAM_REPLICATED.

Post-decision replication audit

The deterministic audit preserves the two stage decisions and adds two-sided task-bootstrap diagnostics. Ordered-minus-shuffled was 0.135693 [0.079646, 0.191740] in qualification and 0.150442 [0.082596, 0.218289] in confirmation. The difference between those two independent effects was 0.014749 with interval [-0.073746, 0.103245], providing no evidence of stage drift. At the paired-path level, ordered-only wins versus shuffled-only wins were 60:14 and 64:13. Task effects were positive/zero/negative on 35/72/6 and 34/71/8 tasks.

The pooled 226-task effect, explicitly diagnostic and unnecessary for either pass, was 0.143068 [0.098820, 0.187316]. It cannot rescue a failed stage.

Alias identity remains an important nuisance. Confirmation successes spanned 10 of 11 target aliases, but horse had 0/30 ordered successes. Shuffle beat ordered thought for tiger (28/30 versus 21/30) and river (13/30 versus 11/30), while most other targets favored ordered thought. This heterogeneity does not defeat the registered breadth/task gates, but it forbids treating raw alias logits or identity as a certainty coordinate. Any value model must use task-held-out evaluation and demonstrate incremental signal over correct-alias activity, slot margin, and alias identity.

Boundary

This is replicated constrained semantic elicitation: ordered native thought changes the fixed semantic commit choice beyond syntax and identical token-bag controls. It is not autonomous termination, free-form capability, J certainty, or installed capability. Gold labels evaluate the seam. J/value/control/causal commands still fail closed until a separately committed outcome-blind audit and implementation boundary.

Prefix-value implementation boundary

After the seam decision, a new preregistration and 30-point adversarial design review froze a prospective rather than merely endpoint readout: midpoint coordinates must rank each path's later full-cap correct-alias probability, and endpoint signal cannot rescue a midpoint miss. The primary 120 J features use all 24 coordinates at layers 4--8 with task-held-out, within-task/fraction centered ridge evaluation. Mandatory matched pipelines use five gold-alias J activities, ordinary slot margin, alias identity, and 120 layer-matched random coordinates orthogonal to the complete J span. Task bootstrap and 32 within- group shuffled refits are load-bearing.

Implementation is complete and 16 outcome-blind tests pass, including exact three-path groups, future-label timing, whole-task folds, train-only scaling, non-J projection <=1e-5, reserved-data loader isolation, and pending-boundary failure before model load. The exact code/audit is anchored to pushed commit ddbc1969; only the outcome-blind value-model smoke is now authorized. No value_fit or causal_confirmation row had been opened at that boundary.

The subsequent one outcome-blind model smoke passed at 8,510,865,408 peak allocated bytes. All five J dictionaries retained rank 24; J and non-J feature widths were each 120 and finite; non-J projection into J-space was at most 2.67e-7. The live feature sequence was 384 tokens with no close/slot, while the separate slot prefill was exactly four tokens longer. The receipt stores no outcome, correctness, chosen alias, probability, or trace text, and both reserved splits remained unopened.

Prefix-value result

The one authorized value run then opened only value_fit and completed all 48 tasks, 144 cap-1,024 traces, and 288 live prefix states. Every trace reached the cap without natural close; all cache, cardinality, lens-rank, context-separation, feature-width, finite, fold, and non-J geometry contracts passed. The terminal slot scored 41/144 paths; the midpoint slot scored 28/144. Mean terminal correct- alias probability was 0.228749.

The automatic decision is NO_PREFIX_J_VALUE:

The one-sided 95% task-bootstrap lower bound for primary AUC was 0.441667. Lower bounds for J minus gold-alias activity, slot margin, and non-J residual features were -0.039583, -0.138542, and -0.104167. Fold AUCs were 0.4917, 0.5750, 0.4667, and 0.4750. Thus neither the headline nor any incremental uncertainty gate passed.

The scientifically interesting pattern is phase reversal: the same shared OOF model ranked terminal outcomes above chance from midpoint states but below chance from endpoint states, cancelling to chance overall. That midpoint point estimate does not rescue the registered decision, and the equal-width non-J baseline also beat shared J. It motivates only outcome-labeled post-decision diagnostics and, if warranted, a new fresh phase-specific replication—not causal access inside this experiment.

The run sampled 147,456 thought tokens and processed 346,146 feature-prefill tokens in 4,835.588 seconds, with 8,670,490,112 peak allocated bytes. causal_confirmation remained unopened. Control calibration and causal confirmation remain fatal-unavailable.

Post-decision phase diagnostic

The preregistered audit permitted separate phase refits only as hypothesis diagnosis. Under the same folds and analysis, midpoint-only J reached 0.5375 (one-sided task lower 0.4417), below its matched non-J readout at 0.6000 and effectively equal to slot margin at 0.5396. Endpoint-only J reached 0.4292. Cross-phase J transfer was 0.5292 from midpoint to endpoint and 0.4458 in the reverse direction; neither lower bound exceeded chance.

This is not evidence of a simple sign flip. Across the 144 paired paths, midpoint/end centered J rows had mean cosine -0.0544, the 120 coordinate correlations averaged -0.0386, and separately fitted coefficient vectors had standardized cosine -0.0681. The stronger explanation is an unstable readout plus sampling/model flexibility. The apparent midpoint lead is retired, the registered negative is unchanged, and no causal data were opened.

Artifact Manifest

See artifact_manifest.yaml.

Experiment log 9

Data files 2

Result tables and metrics copied from the experiment folder — preview inline or open the raw file.

• previewruns/model_smoke/result.json3.3 kB
• previewruns/value_model_smoke/result.json3.3 kB

Reproduce

Smoke test

PYTHONDONTWRITEBYTECODE=1 .venv/bin/python experiments/qwen35_4b_commit_slot_semantic_power_replication/scripts/run.py --stage smoke

Full run

.venv/bin/python experiments/qwen35_4b_commit_slot_semantic_power_replication/scripts/run.py --stage prefix-value

Run steps are documented inside the experiment folder (README and scripts).

Browse the experiment folder on GitHub ↗

Related

Related experiments

Nearest by shared name, tags, and programs.

• 2026-07-12 Qwen3.5-4B Commit-Slot Jacobian Value Transport
• 2026-07-13 Qwen3.5-4B Materialized Residual Sibling Search Fresh Replication
• 2026-07-13 Qwen3.5-4B Semantic-Anchor Coordinate Branching
• 2026-06-22 Qwen Verifier-Guided Slot Repair
• 2026-06-22 Qwen Teacher-Distilled Slot Compiler
• 2026-07-13 Semantic-policy headroom tournament