Full-Rank Extra-R Delta: State-Carry Versus State-Bag
The one idea you need
The first test changed each large weight matrix through a narrow bottleneck, which is memory-efficient but limits the directions training can move. This successor gives every repeated-pass matrix a full-size change of its own while leaving the ordinary one-pass model untouched.
The question
Was the failed running memory caused by the serial architecture, or did the low-rank training update prevent the model from building a sufficiently rich internal state?
What we found
The run worked mechanically, but it did not settle the question. All 892 million full-size update weights trained and fit comfortably, yet macro task-mean joint-state accuracy was only 0.28% against a 40% requirement. At the same time, Carry scored 1.6 points below equal-compute pooling and failed both required query-kind checks. The frozen rules classify that combination as promotion blocked, not as an isolated capacity failure. The full-size construction also shifted the random initialization of the shared state modules relative to the LoRA run.
Why it matters
This valid diagnostic says the direct full-size recipe also failed; it does not prove that LoRA was innocent. The decisive follow-up must give LoRA and full rank exactly the same state-module start and stochastic training stream, prove the state readout can learn, and repeat the fixed final comparison across three seeds. Downstream answer and causal tests should wait until a readable state exists.
On this page
Results at a glance 2
How to read
Each depth has a Carry bar and an equal-compute pooled Bag bar. Compare the paired heights across depths five through twelve; higher means more answers were correct.
Takeaway → Carry is uneven and averages 1.6 points below Bag, while the separate hidden-state score is almost zero. Because several gates failed together, this diagnoses a failed recipe but cannot isolate full-size capacity as the cause.
Data table
| held-out semantic depth | State-Carry | State-Bag |
|---|---|---|
| depth 5 | 37.5% | 40.6% |
| depth 6 | 34.4% | 37.5% |
| depth 7 | 37.5% | 43.8% |
| depth 8 | 21.9% | 12.5% |
| depth 9 | 25% | 37.5% |
| depth 10 | 28.1% | 18.8% |
| depth 11 | 43.8% | 37.5% |
| depth 12 | 18.8% | 31.2% |
Numbers from experiments/qwen35_4b_state_carry_vs_state_bag_fullrank_delta/analysis/k_depth_curves.csv
Technical framing
Blocked full-rank pilot: State-Carry does not beat equal-compute State-Bag — Seed-7401 diagnostic pilot, 32 paired tasks per depth. Carry is positive at only three of eight depths and averages 1.56 points below Bag; paired 95% CI -6.64 to +3.91 points. Macro task-mean joint-step accuracy is 0.277% versus 40% required. Because state sufficiency, Carry positivity, and both query-kind checks failed simultaneously, the frozen ladder gives PILOT_PROMOTION_BLOCKED; this chart is diagnostic, not a clean capacity verdict.
How to read
The three bars split memory among the update weights, their gradients, and Adam's two running statistics. The statistics bar is twice either single-tensor bar because Adam keeps two values per weight.
Takeaway → The real optimizer and worst-depth gate fit comfortably, so the negative state result cannot be dismissed as an unexecuted or memory-infeasible full-rank proposal.
Data table
| full-rank delta training component | FP32 memory |
|---|---|
| delta parameters | 3.32 |
| delta gradients | 3.32 |
| Adam moments | 6.65 |
Numbers from experiments/qwen35_4b_state_carry_vs_state_bag_fullrank_delta/analysis/planned_delta_memory.csv
Technical framing
Full-rank control carried 13.30 GiB of delta training state — Exact setup geometry, not the behavioral result. The 892,272,640 direct full-rank delta parameters require 13.296 GiB for FP32 parameters, gradients, and Adam moments before the frozen model, activations, kernels, allocator slack, or optimizer temporaries. Live G0 passed on the 48GB Ada GPU, establishing feasibility but not a clean capacity conclusion.
In the author’s words from the Overview
Confirmation, edge cut, G3, and G4 remain unlicensed for these checkpoints. The complete seed-7401 Carry/Bag pilot used identical initialization receipts, data order, prompt tokens (2,594,937 each), decoder-layer-token applications (145,316,472 each), fixed 300-step schedule, source/config/data identities, and G0 lineage. Both final checkpoints reloaded with exact K=1 parity (0.0). Full-rank Carry's macro task-mean joint node+phase+checksum step accuracy was 0.0027686, versus the frozen 0.40 sufficiency gate; node accuracy was 0.06167. The corresponding micro count was 7 jointly correct states over 2,176 registered steps (0.0032169). … Read the full result →
Overview
Authoritative terminal disposition: PILOT_PROMOTION_BLOCKED. The preserved analyzer receipt historically emitted PILOT_STATE_FORMATION_MISS, but that classifier gave state failure priority over simultaneous non-capacity failures. The frozen preregistration says those outcomes are mutually distinct. Because Carry-minus-Bag and both registered answer strata also failed, the held-fixed LoRA-rank capacity branch is not closed. Confirmation, edge cut, G3, and G4 remain unlicensed for these checkpoints.
Pilot result and terminal disposition
The complete seed-7401 Carry/Bag pilot used identical initialization receipts, data order, prompt tokens (2,594,937 each), decoder-layer-token applications (145,316,472 each), fixed 300-step schedule, source/config/data identities, and G0 lineage. Both final checkpoints reloaded with exact K=1 parity (0.0).
Full-rank Carry's macro task-mean joint node+phase+checksum step accuracy was 0.0027686, versus the frozen 0.40 sufficiency gate; node accuracy was 0.06167. The corresponding micro count was 7 jointly correct states over 2,176 registered steps (0.0032169). Carry minus Bag on the 256 matched primary tasks was -0.015625 (pilot 95% interval [-0.06640625, 0.0390625]), with only three of eight depths positive. Unseen-K scaling was -0.0078125 ([-0.0625, 0.046875]), and joint-holdout Carry minus Bag was +0.01953125 ([-0.0234375, 0.0625]). Under bidirectional swaps, donor following fell by 0.0078125 ([-0.0390625, 0.015625]) and remained 0.078125 below recipient preservation.
All registered cells were complete, the +0.05 answer gate was reachable, and Carry's answer interface was valid (0.97265625 full-top-is-answer). However, the realized checks simultaneously had joint_state_sufficient=false, positive_carry_minus_bag=false, and query_kinds_positive=false: node was exactly 0.0 and checksum was -0.03125. The latter two are non-capacity promotion failures under the frozen taxonomy, so the raw state miss is a useful descriptive result but cannot license the registered capacity conclusion.
The cross-experiment comparison also did not preserve a bit-identical shared state-module initialization or dropout RNG stream: constructing PEFT LoRA and constructing then zeroing 892M direct parameters consume different random streams before training. Direct deltas also change optimizer geometry and the effect of global gradient clipping, even with the same nominal learning rate and schedule. Thus this experiment establishes that one mechanically valid direct-full-shape recipe also failed to learn the registered state; it does not isolate LoRA rank as the cause or non-cause. A fresh, independently preregistered capacity adjudication is mandatory. The current state was not readable, so these checkpoints do not license the readable-but-unused interface successor either.
Program and question
- Primary research program:
structured_execution_and_compilers. - Parent/closest near-duplicate:
qwen35_4b_state_carry_vs_state_bag. - Trigger: the parent's complete, source-bound
PILOT_MECHANISM_MISS, with a reachable answer-level gate andjoint_state_sufficient=false. - Question: holding the parent task, base path, supervision, recurrence, seeds, Carry/Bag control, and causal tests fixed, does removing LoRA's rank constraint permit the deeper joint representation to form?
The trigger is checked cryptographically and semantically at G0. A missing, incomplete, infeasible, or wrong-parent identity-bound receipt aborts rather than silently turning this into an unregistered architecture search.
Registered intervention
The base is the pinned Qwen/Qwen3.5-4B revision. Layers 12–19 are still the two complete repeated Qwen motifs. The parent LoRA is replaced by 62 direct, zero-initialized, FP32 full-shape weight deltas totaling exactly 892,272,640 parameters. For a targeted linear on an extra R call only:
y = W_base x + 2 * DeltaW * dropout(x, p=0.05)W_base stays frozen. DeltaW is disabled for the prelude, first R call, and coda. Thus K=1 is the exact original base-model path, not a merged or approximately cancelled adaptation. Carry and Bag instantiate the identical wrapper; only the source of state at calls 2..K differs within this successor. That makes the successor's Carry/Bag comparison clean. It does not make the comparison with the parent LoRA run a one-factor randomized contrast, because parameterization-specific initialization, random-stream, and optimizer geometry were not matched across experiments.
G0 is a real feasibility gate
The direct deltas alone require 3.324 GiB for FP32 parameters, 3.324 GiB for gradients, and 6.648 GiB for Adam moments: 13.296 GiB steady state before the frozen model, activations, kernels, allocator slack, or optimizer temporaries. G0 therefore runs an actual scheduled AdamW step, proves that every one of the 62 delta tensors has two finite, shape-matched FP32 Adam moments, records allocated and reserved peaks plus reserved headroom, verifies finite K=12 recurrent logits, and round-trips both delta and loop-state files after deliberately destroying the live tensors. It also independently proves zero delta calls for Carry K=1 and Bag K=1, exact expected calls at K=4/K=12, nonzero gradients in both arms, and no base-model gradient.
If G0 OOMs, stop. Reserved headroom is a diagnostic, not a post-hoc threshold. Do not lower precision, switch models, reduce targets, use a high-rank LoRA surrogate, or reinterpret the result as scientific evidence.
The live gate passed on the 48GB RTX 6000 Ada. It discovered exactly 62 targets and 892,272,640 FP32 delta parameters, produced nonzero delta gradients in both arms, allocated both finite FP32 Adam moments for every delta, preserved exact K=1 parity before and after the optimizer step, and produced finite K=12 logits. Peak allocation was 24.49 GiB and peak reservation was 24.93 GiB, leaving 22.57 GiB of reserved headroom. A 3,571,392,174-byte delta-plus-loop checkpoint was destroyed and reloaded with recurrent-logit error 0.0. This establishes live feasibility only; it is not a behavioral result.
Data and evidence firewall
The generator is self-contained and never imports parent or benchmark code. Nevertheless, every full split is bound to a parent-derived hash of canonical decompressed rows. Regeneration checks row count, order, IDs, and every content field for all 11 splits; when parent artifacts are present it also compares them directly. Every model-bearing stage recomputes those canonical receipts from the current artifacts and checks the full parity metadata rather than trusting a copied pass flag.
The canonical preparation pass produced all 11 splits and 27,744 rows. It matched both the frozen canonical-row contract and the locally available parent artifacts exactly, with zero cross-split structural duplicates and zero benchmark reads.
The seed-7401 pilot was a non-evidentiary promotion gate. Only a complete, reachable pilot whose joint state specifically remains insufficient receives PILOT_STATE_FORMATION_MISS and closes the held-fixed capacity branch. Incomplete diagnostics or failures on another promotion requirement stop the run as PILOT_INCOMPLETE or PILOT_PROMOTION_BLOCKED without licensing that capacity conclusion. Only PILOT_PROMOTION_READY licenses seeds 7411–7413 and the full causal ladder through G3. G4/sample-more is deliberately deferred: this successor was designed to resolve the LoRA-capacity counterfactual but cannot make a deployment claim. The realized cross-experiment controls were not strong enough to identify rank, as detailed below.
Every pilot checkpoint embeds the exact G0 receipt path, file hash, identity hash, status, and phase that licensed training. Full checkpoints additionally embed the corresponding G1 promotion receipt lineage. Checkpoint loading and analysis fail closed on missing or malformed lineage, so later evidence cannot silently outlive the gate chain that authorized it.
The immutable analyzer emitted historical PILOT_STATE_FORMATION_MISS, but the post-result audit applies the frozen taxonomy and authoritatively reclassifies the run as PILOT_PROMOTION_BLOCKED, with capacity_branch_closed=false. Seeds 7411–7413, the same-checkpoint edge cut, G3, and G4 were correctly not run. The pilot swaps are a one-seed diagnostic, not completed G3 causal identification.
A fresh adjudication must use fresh procedural evaluation rows, bit-identical shared loop-state initialization and controlled CPU/CUDA/dropout RNG streams across LoRA and direct-delta arms, an early held-out trained-depth state positive control, and a full fixed-final multi-seed comparison. Representation formation and answer/mechanism promotion must be separate verdict axes so a simultaneous downstream miss cannot silently decide the capacity question.
See reports/preregistration.md, reports/design_review.md, and reports/terminal_science_audit.md for the authoritative disposition. Generated corpora and checkpoints are omitted under reports/artifact_manifest.yaml.
Report
Rendered from reports/report.md
Verdict
PILOT_PROMOTION_BLOCKED — authoritative post-result disposition. The raw, immutable analyzer receipt historically emitted PILOT_STATE_FORMATION_MISS, but its classifier did not implement the frozen mutually exclusive taxonomy for simultaneous failures. Carry failed both non-capacity sign requirements while also failing state sufficiency, so capacity_branch_closed=false. This stops these checkpoints but does not close the LoRA-rank question.
Integrity and feasibility
All 11 splits and 27,744 rows match the parent task exactly under both frozen canonical hashes and direct artifact comparison, with no structural duplicates or benchmark reads.
On the current 48GB RTX 6000 Ada, G0 discovered the preregistered 62 targets and 892,272,640 FP32 delta parameters. Carry and Bag had identical 892,840,988 total trainable-parameter/value receipts; every delta tensor received nonzero gradients in both arms; and Adam allocated 124 finite, shape-matched FP32 moment tensors. Exact base/K=1 and Carry/Bag parity remained 0.0 before and after the real optimizer step. K=12 was finite with 682 delta calls per arm. Peak allocation was 24.49 GiB, peak reservation was 24.93 GiB, and reserved headroom was 22.57 GiB. The 3.571 GB checkpoint round trip restored recurrent logits with error 0.0.
G0 emitted MODEL_SMOKE_PASS, proving that the full-rank capacity control was mechanically executable rather than an arithmetic-only proposal.
Matched pilot
Carry and Bag independently started from identical 892,840,988-parameter/value receipts and consumed the same ordered rows, G0 receipt, 300 optimizer steps, 2,594,937 prompt tokens, and 145,316,472 decoder-layer-token applications. Both checkpoint reloads preserved exact K=1 logits. Peak training allocation was 26.93 GiB in both arms. Final pilot-validation accuracy was 0.28125 for Carry and 0.328125 for Bag; these were diagnostics, not verdict gates.
The complete locked analysis found:
- primary Carry minus Bag
-0.015625, 95% CI[-0.06640625, 0.0390625], with 3/8 positive depths; - unseen-K Carry gain
-0.0078125, CI[-0.0625, 0.046875]; - macro task-mean joint node+phase+checksum step accuracy
0.0027686versus the0.40state-sufficiency gate; micro accuracy 7/2,176 =0.0032169; node step accuracy0.0616713; - joint-holdout Carry minus Bag
+0.01953125, CI[-0.0234375, 0.0625]; - donor-follow gain under 128 bidirectional swap directions
-0.0078125, CI[-0.0390625, 0.015625], with donor follow minus recipient preservation-0.078125; - Carry answer-interface validity
0.97265625; the required +0.05 answer gain was mathematically reachable from the Bag baseline0.32421875.
The check vector was therefore simultaneous: joint_state_sufficient=false, positive_carry_minus_bag=false, and query_kinds_positive=false (node difference 0.0, checksum difference -0.03125). Preregistration assigns a complete pilot with any non-capacity promotion failure to PILOT_PROMOTION_BLOCKED; the state-specific label is reserved for a state miss without those unrelated failures. The historical classifier reversed that precedence. See terminal_science_audit.md.
The state was unreadable under the registered macro metric, and pilot swaps did not support donor following. These are useful descriptive negatives, not G3: seeds 7411–7413, the same-checkpoint edge cut, and G3 were never run. The phrase "causally inert" is therefore not licensed by this one-seed diagnostic.
Capacity interpretation and next experiment
The successor proves that the direct full-shape branch fit, optimized, and also failed under this 300-step seed-7401 recipe. It does not isolate rank. PEFT LoRA and the 892M direct-delta construction consume different random streams before the shared state modules and dropout, so the same integer seed does not give a bit-identical cross-experiment initialization. Full matrices also change Adam and global-clipping geometry even when the nominal optimizer schedule is held.
A fresh capacity adjudication is mandatory rather than another stage from these checkpoints. It must use fresh procedural evaluation rows, exact shared loop- state initialization and controlled CPU/CUDA/dropout RNG receipts across LoRA and direct-delta arms, an early held-out trained-depth state positive control, and fixed-final multi-seed training. It must assign representation formation and downstream answer/mechanism promotion to separate verdict axes. The current unreadable state does not license an interface successor.
Experiment log 4
Show the running log (4 entries, 2026-07-13)
2026-07-13 — successor scaffold and adversarial review
- Created as the parent LoRA pilot's preregistered full-rank capacity successor.
- Copied the self-contained task, recurrent mechanics, evaluation, and analysis harness; no parent source is imported at runtime.
- Replaced PEFT with 62 zero-initialized FP32 full-shape deltas on layers 12–19, active only for extra R calls.
- Added a strict parent-trigger reader, canonical parent-row parity contract, real Adam-state/memory G0, independent Carry/Bag K=1 call checks, and an observable delta-plus-loop checkpoint/logit round trip.
- Removed G4 from the CLI and verdict; it remains explicitly deferred.
- Ran CPU unit/static tests only. No data-preparation or model-bearing stage was run; there are no scientific results.
2026-07-13 — canonical data and live G0
- Canonical CPU smoke passed under config digest
bb0abb85766c0e5eb848492a503b1db0e5c005b5d6521e554a3c30d25d514ccdand source contractc18c44fe8ed6c65fe18be6592ded644a788954a5002256a1dd1730c1fdc8bcba. - Regenerated all 11 parent-matched splits (27,744 rows). Frozen canonical-row hashes and direct comparison with the available parent artifacts both passed; structural duplicates and benchmark reads were zero. Manifest SHA256:
1ad19fd3e74e43c52d7e9dc1fbdfc3d9ea0ac4f2b697f6e7e4f7454a40281da5. - Live G0 loaded only the pinned
Qwen/Qwen3.5-4Brevision on the RTX 6000 Ada and emittedMODEL_SMOKE_PASS(receipt identity0832423e632a5c056e701eacb5b7e70387595956cccbadbe9453cb583c8346fc). - Exact receipts: 62 targets, 892,272,640 delta parameters, zero initial delta, both-arm nonzero gradients, 124 complete FP32 Adam moment tensors, K=1 base and Carry/Bag error
0.0before and after AdamW, and finite K=12 logits with 682 active delta calls per arm. - Peak allocation/reservation was 24.49/24.93 GiB with 22.57 GiB reserved headroom. The 3,571,392,174-byte checkpoint round trip restored recurrent logits exactly and removed its temporary payload.
- These are setup and feasibility receipts only. The seed-7401 Carry/Bag pilot is the next authorized scientific stage.
2026-07-13 — matched full-rank pilot and historical analyzer output
- Trained independent Carry and Bag seed-7401 arms for the fixed 300 steps. Both consumed 2,594,937 prompt tokens and 145,316,472 decoder-layer-token applications in identical order (
97813bf9a2c7b81cf55db1a405e8e999e7e4bf953b2d50434a007140019b0e4f), under the same source/config/data/G0 lineage and identical initialization receipt. Peak allocation was 26.93 GiB in each arm. - Carry final validation was 0.28125 and Bag was 0.328125. Both fixed-final checkpoints passed exact K=1 reload parity and yielded complete 768-row pilot evaluations; Carry also yielded all 128 bidirectional swap directions.
- Deterministic analysis was rerun byte-identically (summary SHA256
2f3508202b08928aa6cd2867656e82b6f54859c3c1b075fdb373daa4a2cffa83) and historically emittedPILOT_STATE_FORMATION_MISSwith receipt identity7697cf03066ff00e41ef02bb0bd3a33b24b42e465106c2db7e474d5f860a0dc0. - Macro task-mean joint-state step accuracy was 0.0027686 versus the 0.40 gate; the micro count was 7/2,176 = 0.0032169. Carry minus Bag was -0.015625 (CI -0.06640625 to +0.0390625), unseen-K gain -0.0078125 (CI -0.0625 to +0.046875), and swaps reduced donor following by 0.0078125.
- All required cells were complete, the answer gate was reachable, and the answer interface was valid. The emitted check vector nevertheless also had
positive_carry_minus_bag=falseandquery_kinds_positive=falsein addition tojoint_state_sufficient=false. - Confirmation seeds, edge cut, G3, and G4 were not run. No interface successor is licensed because the state was not readable.
2026-07-13 — post-result terminal science audit
- An adversarial read-only audit found that the analyzer's precedence did not implement the frozen mutually exclusive pilot taxonomy. The preregistration assigns a complete pilot with any non-capacity promotion failure to
PILOT_PROMOTION_BLOCKED; the implementation instead let the simultaneous state failure dominate. - The authoritative disposition is therefore
PILOT_PROMOTION_BLOCKED, withcapacity_branch_closed=false. The raw summary, receipt identity, and hash above are preserved unchanged as historical flawed-classifier output rather than rewritten after seeing the result. - The audit also found that the parent LoRA and successor direct-delta builds did not share bit-identical state-module initialization or dropout RNG streams. Their parameterizations consume different random streams before training, and the 892M-parameter direct optimizer has different global-clipping geometry. The same integer seed, learning rate, and schedule do not remove those cross-experiment confounds.
- The result remains strong descriptive evidence that this mechanically valid direct-full-shape recipe did not learn the registered state. It does not close LoRA rank as the cause or non-cause, and it did not run edge cuts or G3.
- A fresh experiment is mandatory: use fresh procedural evaluation rows, bit-identical shared loop-state initialization, isolated and reset RNG streams, an early trained-depth state-readability positive control, full fixed-final multi-seed LoRA/direct-delta arms, and separate representation versus answer/mechanism verdict axes.
Data files 3
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
analysis/summary.json7.8 kBruns/pilot_bag_seed7401/summary.json18 kBruns/pilot_carry_seed7401/summary.json18 kB
Reproduce
Smoke test
python3 experiments/qwen35_4b_state_carry_vs_state_bag_fullrank_delta/scripts/run.py --smokeFull run
historical only; preserve receipts and follow reports/terminal_science_audit.md for the corrected dispositionRun steps are documented inside the experiment folder (README and scripts).