Qwen3.5-4B Long-Horizon Answer-Potential SFT
The one idea you need
You hold the answer key. To choose which fully worked-out reasoning chains a small model learns from, score each by how much more confident the model grows in the correct answer after reading it — keeping the notes that make the right answer feel most inevitable.
The question
When a small model reasons for as long as it wants, can its own confidence in the correct answer flag which reasoning chains are worth training on?
What we found
Barely. That confidence signal does sort stronger reasoning above weaker about 60% of the time — better than a coin flip's 50%, and enough to lift a picked chain's success from 16% to 22%. But two dumb rules beat it: just keeping the shortest chain, or the most typical one, each rank correctly near 70%. The real training test is still pending.
Why it matters
Before building a confidence scorer to curate which reasoning a small model learns from, race it against trivial baselines — chain length and how typical the chain is. If those match or beat it, your fancy signal is mostly a proxy for length and habit.
On this page
Results at a glance 1
How to read
Three bars, one per stage, on a vertical log scale of candidate reasoning chains gathered: an earlier length-capped run near 2,000, this study's calibration near 9,000, and its full planned pool near 95,000. Taller means a bigger search, not better quality.
Takeaway → The full pool towers about 46 times over the capped pilot, because uncapped, naturally finished reasoning needs a far larger haul to surface the rare high-quality chains.
Data table
| Experiment stage | candidate thoughts |
|---|---|
| C51 scorer run | 2048 |
| new calibration | 8640 |
| new full pool | 95.04k |
Numbers from experiments/qwen35_4b_long_horizon_answer_potential_sft/configs/default.yaml and README.md
Technical framing
Preregistered candidate scale: censored pilot versus long-horizon follow-up — Pre-run design chart: the new pool uses natural close with a 12,288-token allowance and does not stop before SFT for a modest calibration effect.
In the author’s words from the Report · “Summary”
Pre-run design only. No scientific result has been observed.
Overview
Status
Design frozen before scientific GPU work. This follow-up is intentionally not a larger replay of C51: it removes the operative 512-token cutoff, requires naturally closed thoughts, scales the pool to about 95,000 candidate traces, banks complete traces before any compression, and runs the SFT comparison even if calibration shows only modest predictive signal.
The original run is now paused after 331/1,080 train tasks because its measured wall time exceeded the available budget. Its design and partial artifacts remain intact; no terminal SFT claim is made here. The checksum-preserving, prospectively frozen completion is qwen35_4b_balanced_core_answer_potential_sft.
The immutable protocol is in reports/preregistration.md, and the pre-run adversarial review is in reports/design_review.md.
Research Program
- Programs:
posttraining_and_adaptation,evidence_conditioned_selection, andtest_time_reasoning_budget. - Closest near-duplicate:
qwen35_4b_answer_potential_trace_sft/ C51. It sampled 2,048 thoughts at 512 tokens, force-closed 99.37% of them, found real-but-modest answer-potential signal, and stopped before training. - Other anchors: C28 (own successful thoughts can be inert rationalizations), C50 (the answer-emission seam and weighted loss are load-bearing), C44/C45 (serial reasoning can carry installed skill), and C24 (training diversity matters more than repeated gradient exposure).
Question
If Qwen3.5-4B is allowed to finish long reasoning naturally, does the same model's likelihood of a known correct answer identify complete reasoning strategies that are better SFT targets than random natural thoughts, binary successful-answer rejection sampling, answer-only SFT, and task-shuffled thoughts?
The stronger mechanistic question is whether a deployment-matched joint score for close-token + ANSWER + correct answer improves on answer-only potential. The original answer-only score remains a primary treatment; the joint score is the predeclared seam repair.
Why This Is A New Experiment
C51 did not train anything. Its fresh-continuation labels showed positive top-choice deltas over random and shortest traces, and real thoughts beat shuffled and foreign controls. Its dominant fact was instead interface censorship: nearly every thought was still running at 512 tokens. This experiment changes four load-bearing variables together and therefore gets a new directory and fresh tasks:
- natural termination with a 12,288-token allowance and continuation of rare non-loop contacts;
- about 95,000 candidate traces rather than 2,048;
- complete-trace banking with no brevity objective; and
- an actual controlled SFT/evaluation matrix, with no predictive-effect gate before training.
Model And Firewall
- Only
Qwen/Qwen3.5-4B, revision851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a. - Fresh procedural atom tasks copied locally from the firewall-clean gauntlet gym.
- No content under
benchmarks/is read, imported, scored into training data, or used to tune this experiment. - Reference answers and verifiers are oracle-side curation/evaluation instruments only. They are never included in prompts at deployment.
- Nine finite-answer training families are used. Stallwright's combinatorial answer-rendering family is excluded from potential curation and retained as a transfer-only evaluation family. Brinework and spindle remain fully held-out families.
Frozen Fresh Splits
| split | construction | items | purpose |
|---|---|---|---|
| termination pilot | 9 families x L1-L3 x 1 | 27 | capacity and natural-close mechanics only |
| calibration | 9 families x L1-L3 x 5 | 135 | selector diagnostics and threshold freeze |
| harvest/train | 9 families x L1-L3 x 40 | 1,080 | candidate pool and SFT rows |
| IID evaluation | 9 families x L1-L3 x 20 | 540 | primary capability evaluation |
| harder evaluation | 9 families x L4 x 20 | 180 | difficulty transfer |
| held-family evaluation | 2 families x L1-L3 x 30 | 180 | family transfer |
| rendering-held evaluation | stallwright x L1-L4 x 20 | 80 | transfer to an uncurated answer interface |
IDs, prompts, prompt digests, and family-generator seeds must be disjoint across every split before any model call.
Long-Horizon Sampling
The experiment has no 512-token reasoning cutoff. Thought sampling stops only when the model emits its own </think> token or reaches the 12,288-token context-safety allowance. A non-loop allowance contact is resumed once from its exact prefix for up to 2,048 additional tokens; periodic loops are recorded and excluded rather than laundered into complete reasoning. The finite safety envelope is required by the 16,384-token model context and is not used as a forced-close training intervention.
- independent sampling: temperature 1.0, top-p 0.95, top-k 20;
- calibration:
135 x 64 = 8,640thoughts; - harvest:
1,080 x 64 = 69,120independent thoughts; - pivot branches:
1,080 x 16 = 17,280suffix resamples; - combined planned candidate pool:
95,040thoughts before deterministic exclusions; - sampled-token log-probability is captured for the trace-prior diagnostic;
- every raw shard, seed, context contact, loop diagnostic, and sampled token is counted.
Natural closure is a mechanical validity diagnostic, not an effectiveness veto. The full SFT matrix runs whenever each training task has at least two trainable natural traces after the continuation path. If that minimum is missed, the runner completes additional independent natural samples for only the deficient tasks; it does not lower an AUROC bar or force-close traces.
Scores
For prompt x, complete natural thought z, and canonical answer y*:
answer_gain(z) = log p(y* | x, z, close, ANSWER) - log p(y* | x, empty, close, ANSWER)
joint_gain(z) = log p(close, ANSWER, y* | x, z)
- log p(close, ANSWER, y* | x, empty)answer_gain is the original idea under a deployment-valid natural-close pool. joint_gain makes the close/commit seam part of the event rather than teacher-forcing it away. Both are computed uniformly by a Transformers bf16 SDPA teacher-forced scorer because full-sequence logits are the required internal measurement; 32 held smoke rows must agree with the exact vLLM targeted readout within 0.15 mean nats/token before bulk scoring. All generative comparisons remain on vLLM.
Calibration obtains four fresh answer continuations per trace. Harvest obtains one per trace, which both validates ranking and constructs the binary-success RFT baseline. Calibration reports within-task AUROC, top-k success curves, length/prior baselines, pre-answer-mention checkpoints, family heterogeneity, and answer-versus-joint score disagreement. These results freeze selector tolerances but do not cancel SFT.
Potential-Guided Pivot Branching
For the highest-scoring independent natural trace per training task, score at most eight natural sentence/newline checkpoints. Choose the boundary immediately before the largest positive joint-gain jump; if no positive jump exceeds 0.05 nats per canonical-answer token, use the natural boundary nearest half the trace. Preserve that prefix and sample 16 fresh suffixes to natural close.
This is suffix resampling, never arbitrary token editing. Independent-only N=64 results remain a complete nested baseline, and every preserved-prefix prefill plus sampled suffix token is counted. The branch pool is included in selection regardless of whether it wins; branch source and adoption are reported so a null cannot disappear.
Quality, Diversity, Then Full-Trace Banking
Selection is within task and does not reward brevity:
- retain natural, non-loop, finite-score, trainable-length traces;
- rank separately by answer gain and joint gain;
- retain the top 12 candidates per score;
- choose the top trace, then the most structurally distant trace within 0.25 nats per answer token of the top, using identifier/number-normalized token-trigram Jaccard distance; and
- keep both complete traces exactly. No prefix compression or shortest-near-best rule is applied.
This yields up to 2,160 full-trace rows per potential arm. Compression is explicitly deferred: if full-trace banking works, a separate follow-up may re-harvest from the trained model and test iterative compression without contaminating this first causal comparison.
SFT Arms
All adapters start from the same pinned base, use the same canonical answers, and use seed 42.
| arm | trace target | question |
|---|---|---|
empty | empty thought | does answer learning alone explain gains? |
random_natural | same-task natural trace nearest treatment length | do long thought tokens/style explain gains? |
success_rft | same-task R1-successful natural trace | does ordinary binary rejection sampling suffice? |
answer_potential | diverse top answer-gain full traces | original treatment |
joint_potential | diverse top close-plus-answer-gain full traces | deployment-seam repair |
potential_shuffle | answer-potential traces reassigned within family/level/length | is task-specific reasoning content causal? |
Potential-versus-success comparisons include both the full training sets and the exact common-task intersection. Success-RFT may have fewer unique tasks; it is oversampled only to match optimizer steps, and its unique coverage is reported rather than hidden.
QLoRA: rank 32, alpha 64, dropout 0.05, two epochs, learning rate 2e-4, batch 1 x gradient accumulation 16, maximum sequence length 16,000. Prompt loss is 0, full thought loss is 0.5, and close/answer loss is 1.0. The larger thought weight deliberately tests strategy banking; C50's answer-seam weighting remains. Trace arms are matched on rows, selected-task quotas, optimizer steps, and length as closely as their definitions permit. Actual supervised and forward tokens are reported.
Adapters remain external. Each is merged into a full composite checkpoint, then must produce a real on-versus-off behavioral difference before evaluation; vLLM runtime LoRA is prohibited by C49.
Evaluation
Every base and merged-trained arm generates naturally on vLLM with an allowance of 12,288 tokens; no evaluation arm is force-closed at 512. Primary: fresh IID greedy exact-answer accuracy and paired answer_potential - random_natural, answer_potential - success_rft, and answer_potential - potential_shuffle deltas.
Secondary:
- joint-potential comparisons;
- natural-close, parse, and parse-conditional accuracy;
- mean/median/p95 thinking tokens;
- family macro, L4, held-family, and stallwright transfer;
- k=8 sampled coverage, majority selection, diversity, and oracle pass@8 ceiling;
- base sample-more accuracy versus actual forward tokens; and
- branch adoption and independent-N scaling at N in {8, 16, 32, 64}.
The primary seed-42 matrix always completes. If either potential arm beats the strongest matched trace baseline by at least 0.03 IID with paired 95% lower confidence bound above zero and no >0.02 parse/family regression, that treatment and the strongest baseline are replicated at training seed 43.
Decision Rules
- Scorer informative / banking negative: scores predict outcomes, but potential SFT does not beat random natural and success-RFT controls.
- Full-trace banking positive: a potential arm beats both controls by at least 0.03 IID, paired 95% lower bound above zero, beats shuffled content, and preserves parse/family macro within 0.02.
- Replicated banking positive: the seed-43 contrast has the same sign and the pooled paired interval excludes zero.
- Mission positive: replicated banking positive and the trained greedy point beats base sample-more at some matched actual-forward-token point.
- Compression licensed: full-trace banking positive. No compression claim is made in this experiment.
There is deliberately no pre-SFT AUROC/effect-size stop. A terminal claim about the overarching idea is made only after the complete seed-42 SFT matrix and fresh evaluation.
Run
The restartable staged interface is:
.venv-vllm/bin/python experiments/qwen35_4b_long_horizon_answer_potential_sft/scripts/run.py --stage smoke
.venv-vllm/bin/python experiments/qwen35_4b_long_horizon_answer_potential_sft/scripts/run.py --stage fullGranular stages include pilot, calibration-generate, scorer-parity, calibration-score, calibration-rollouts, harvest-generate, harvest-score, pivot-plan, branch-generate, branch-score, train-rollouts, select, train, merge, deployment-probe, both evaluation modes, analysis, and conditional replication. Sharding, external paths, and exact commands are registered in the artifact manifest and experiment log.
Results
Termination and training-envelope gates
The registered 27-task termination pilot is complete (108 traces). Every trace exceeded the old 512-token cutoff; median thought length was 4,636 tokens and p95/max was 14,336. The model closed naturally on 96/108 traces (88.9%), with zero exact periodic loops. Thirteen traces reached the initial 12,288-token allowance; one then closed during the exact-prefix continuation. The 12 still-open traces were all loomfix and remain ineligible rather than being force-closed. Correctness was not inspected in this operational pilot.
The exact-token QLoRA path was also validated before dataset construction. Ordinary 3--4k rows train in 4.7 seconds per two-example optimizer step. A deliberately worst-case 14,687-token row initially exposed a quadratic SDPA backward workspace, then passed untruncated in 29.1 seconds at 15.0 GiB peak after the training-only full-attention kernel was moved to xFormers and >8k rows received explicit layer/loss checkpointing. The six-arm scientific matrix has not yet reported a result. The complete calibration harvest is now banked: 8,640 independent traces over 135 tasks (N=64), totaling 45,728,102 sampled thought tokens. Of these, 7,814 closed naturally and 27 exact periodic loops were detected; loops and every unresolved allowance contact remain mechanically ineligible. The preregistered 32-row HF/vLLM canonical-answer likelihood parity gate passed with a maximum difference of 0.000448 nats per answer token (threshold 0.15). Calibration answer rollouts and full-prefix scoring are in progress.
Loomfix is the important termination stress case: its 960 calibration traces consumed 12,676,528 sampled tokens, but only 204 (21.3%) closed naturally within 12,288 plus the single exact-prefix 2,048 continuation. This is preserved as evidence that even the enlarged protocol does not cover every search horizon; it does not veto the complete SFT matrix, and unresolved traces are never force-closed or used as training examples.
Calibration scoring and R=4 answer rollouts are also complete for all 7,814 eligible traces (31,256 answer rollouts). Answer gain has task-macro AUROC 0.597 and its top-1 trace averages 22.46% rollout success versus 15.63% for seeded random (+6.84 percentage points). Joint close+answer gain has AUROC 0.678 and top-1 success 21.88% (+6.25 points). Canonical versus one-newline score rankings are stable (task-macro Kendall tau-b 0.841). These are encouraging selector diagnostics, not a banking result: negative length (AUROC 0.690; top-1 26.56%) and the sampled-trace prior (AUROC 0.700; top-1 22.46%) are strong controls. Per the preregistration, none of these diagnostics gates the 1,080-task harvest or the complete six-arm SFT matrix.
No scientific result yet. This file records the complete pre-run plan; later results are added above this boundary without rewriting the frozen preregistration.
Artifacts
idea_intake.md: novelty and near-duplicate decisionreports/preregistration.md: immutable protocol and decision rulesreports/design_review.md: adversarial pre-run reviewconfigs/default.yaml: frozen counts, seeds, model, and recipesreports/artifact_manifest.yaml: sharded pools, adapters, merged checkpoints, and checksumsruns/: compact receipts and scientific summaries retained in git- external root:
/workspace/large_artifacts/qwen35_4b_long_horizon_answer_potential_sft
Report
Rendered from reports/report.md
Summary
Pre-run design only. No scientific result has been observed.
Research Program Fit
Follow-up to C51 across posttraining, evidence-conditioned selection, and test-time reasoning budgets.
Method
See ../README.md and preregistration.md; the full protocol is frozen before GPU work.
Results
Pending. The experiment may not stop for a modest calibration effect; the seed-42 SFT matrix is the claim test.
Controls
Oracle Versus Deployable Evidence
Interpretation
Next Experiments
Artifact Manifest
See artifact_manifest.yaml for the external sharded pools, adapters, and merged checkpoints.
Experiment log 8
Show the running log (8 entries, 2026-07-10 → 12)
Scaffold
Created as a new experiment after C51 showed real signal under a 99.37% force-close regime but stopped before SFT. The follow-up plan removes the 512-token cutoff, scales to about 95k candidates, retains full natural traces, and commits to the complete training matrix without an effectiveness gate.
No scientific GPU work had run at this boundary. No benchmark content was read.
2026-07-10 — Pre-GPU Implementation Boundary
- Design committed at
6f97f0ce; config now verifies the committed README, preregistration, and design review hashes rather than the mutable worktree. - Generated all seven fresh procedural splits and passed ID/prompt/digest/family-seed disjointness.
- Added atomic gzip shard storage, natural-thought continuation, sampled-trace prior capture, a memory-bounded Transformers scorer using
logits_to_keep, and restartable generation/scoring stages. - Twenty-six CPU tests pass. No scientific GPU call has run yet.
2026-07-10 — Integrated Smoke and Pre-Pilot Correction
- The integrated four-trace smoke passed HF/vLLM canonical-answer likelihood parity at
0.000735nats per answer token and preserved finite sampled-trace priors. - Two of four traces closed naturally between 3,171 and 3,665 tokens. The other two reached the smoke-only 4,096-token allowance. This directly confirms useful behavior far beyond 512 tokens.
- The first smoke also exposed an implementation mismatch before the termination pilot: a legacy global trigram-frequency heuristic marked all long coherent prose as loops. The preregistered exclusion is for exact periodic loops, so the detector now requires an exact periodic suffix of at least four repeats and 64 tokens. Global trigram frequency remains descriptive only. This correction changes neither a scientific outcome nor any selection threshold, and is frozen before the registered pilot.
- Transformers' causal-LM auto mapping passed Qwen3.5's multimodal wrapper config to its text-only class and failed on the missing outer
vocab_size. The scorer now uses the checkpoint-native conditional generation class in text-only mode; a real 4,096-token forward and the integrated parity gate passed.
2026-07-11 — Termination Pilot and Training Envelope
- Registered pilot complete: 108 traces, 108/108 longer than 512 tokens, median 4,636, p95/max 14,336, 96/108 natural closes, zero exact periodic loops, and 13 initial allowance contacts. The one non-loomfix contact closed after continuation; all 12 residual open traces were loomfix and remain mechanically ineligible rather than force-closed. Correctness was not inspected.
- Built an isolated pinned training environment. Both Qwen hybrid fast-path checks pass. Exact text-only loading maps all 426 language weights and exposes 42,467,328 rank-32 LoRA parameters.
- The first checkpointed 3--4k loss implementation was unnecessarily slow (96.5 s/two rows). Bounded bf16 full logits preserved the same loss and reduced it to 4.2 s at 17.7 GiB peak.
- The required 14,687-token stress row exposed a quadratic SDPA backward allocation (12.86 GiB) and OOMed. Training now uses xFormers memory-efficient causal attention, explicit per-layer checkpoints plus 256-token recomputed vocabulary chunks only above 8,192 tokens. The exact untruncated 14,687-token row passed in 29.1 s at 15.0 GiB peak; the ordinary path remained 4.7 s/two rows. These are operational kernel/memory repairs made before any SFT dataset exists, not result-conditioned design changes.
2026-07-11 — Calibration Harvest and Scorer Parity
- The preregistered 32-row HF/vLLM canonical-answer likelihood parity gate passed. Maximum absolute difference was 0.000448 nats per answer token against the frozen 0.15 threshold.
- Completed all 135 calibration tasks at N=64: 8,640 traces and 45,728,102 sampled thought tokens. There were 7,814 natural closes, 27 exact periodic loops, and finite sampled-trace priors on all 8,640 rows. Loops and unresolved allowance contacts remain ineligible for scoring and selection.
- Loomfix was the clear outlier: 204/960 natural closes (21.3%) and 12,676,528 sampled tokens. Its harder tiers frequently remained open after the exact 12,288+2,048 protocol. This is recorded as a support boundary, not used to abort the experiment, and no incomplete trace is force-closed into training.
- Generation was restart-safe at per-task atomic/checksummed shard boundaries. Calibration answer rollouts and full-prefix scoring began only after the full harvest completed; no score/outcome was inspected during generation.
2026-07-11 — Calibration Informativeness
- Completed full-prefix canonical and one-newline-variant scoring for all 7,814 eligible traces in 6,293 seconds, plus 31,256 fresh answer rollouts (R=4) in 14,033 seconds.
- Task-macro AUROC was 0.597 for answer gain and 0.678 for joint close+answer gain. Mean top-1 rollout success was 22.46% and 21.88%, respectively, versus 15.63% for seeded random: +6.84 and +6.25 points.
- Canonical/format-variant rankings were stable (task-macro Kendall tau-b 0.841). Answer and joint rankings were only moderately aligned (tau-b 0.282), justifying both preregistered treatment arms.
- Important caution: negative trace length was stronger (AUROC 0.690; top-1 26.56%), and sampled-trace prior was also strong (AUROC 0.700; top-1 22.46%). The calibration therefore supports running SFT but is not itself evidence that answer-potential banking transfers. No effectiveness gate was applied; the frozen 1,080-task N=64 harvest started unchanged.
2026-07-11 — Main-Branch Integration
- Paused the restartable harvest after 55 complete task shards, rebased eight experiment commits over 12 concurrent
maincommits, regenerated the combined knowledge indexes, passedmake check, and pushed the integrated tree. Both GitHub validation and Pages deployment passed ond32f1a2c. - Rebase changed the immutable design commit ID from original
6f97f0ceto rebased261680d3. The README, preregistration, and design-review hashes at both commits are byte-identical. The ancestry pin was re-anchored to261680d3; no design content or threshold changed. A guard run failed closed before model load, exposing the stale pointer, and this operational repair is committed before harvest resume.
2026-07-12 — Parent Paused; Balanced-Core Forked
- Paused after 331/1,080 train tasks at a safe atomic boundary: 21,184 traces, 97,883,041 sampled thought tokens, 20,917 natural closes, and four exact periodic loops. The in-flight 332nd task was discarded; no completed shard was lost.
- Observed throughput made the remaining nine-family plus pivot protocol incompatible with the user's time budget. The original design and partial artifacts remain preserved and this experiment makes no terminal banking claim.
- The user selected a prospectively frozen, resource-constrained follow-up at
qwen35_4b_balanced_core_answer_potential_sft. It imports the 331 shards by checksum, finishes the balanced 360-task three-family core, adds the observed strongest shortest-trace control, skips branches, and gates all expansion on fresh SFT outcomes.
Data files 6
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
runs/calibration_independent_summary.json410 Bruns/calibration_rollouts_r4_summary.json309 Bruns/calibration_scores_summary.json303 Bruns/scorer_parity_32.json22 kBruns/termination_pilot_summary.json396 Bruns/smoke/result.json42 kB
Reproduce
Smoke test
.venv-vllm/bin/python experiments/qwen35_4b_long_horizon_answer_potential_sft/scripts/run.py --stage smokeFull run
.venv-vllm/bin/python experiments/qwen35_4b_long_horizon_answer_potential_sft/scripts/run.py --stage fullRun steps are documented inside the experiment folder (README and scripts).