Qwen3.5-4B Answer-Potential Trace SFT

Results at a glance 2

Overview

Status

Terminal verdict: SCORER_NEGATIVE. The preregistered G0 calibration failed, so the gated orchestrator refused the N=128 harvest and no selector, adapter, or SFT evaluation was run. The answer-potential signal was real but not actionable under this protocol: task-macro within-task AUROC was 0.617 versus the 0.65 gate, and top-one rollout-success uplifts over seeded-random and shortest selection were +0.073 and +0.058 versus the required +0.10.

The full plan is in reports/preregistration.md, and the adversarial review is in reports/design_review.md. Both were frozen in commit 3441dd23 before any GPU-scale work. The terminal report is in reports/report.md, with compact result tables in analysis/.

Result

Calibration sampled 2,048 thoughts for 64 fresh procedural prompts; 58 prompts (1,856 thoughts) admitted a finite confirmatory answer event. Three of eight implementation-level gate booleans passed:

The score therefore detects some task-relevant structure but does not reliably identify a best deployable thought. Its top choice succeeded on 20.3% of fresh continuations versus 12.9% for the seeded-random choice and 14.4% for the shortest choice; both gains had positive paired confidence intervals, but both missed the preregistered effect-size bar.

The most important failure was the deployment seam. Only 13/2,048 thoughts closed naturally; 2,035/2,048 contacted the 512-token cap. After force-closing and asking for a short answer, only 13.2% of 14,848 rollouts parsed, although 86.9% of parsed answers were correct. Teacher-forcing the canonical answer after an injected </think>\n\nANSWER: boundary measured a useful counterfactual answer state, but usually not one the model could enter and express on its own.

One instrumentation limitation is preserved rather than repaired post-result: thought-generation log-probabilities were not requested, so the trace-prior comparison was unavailable and its criterion failed closed. This does not determine the verdict: AUROC, both +0.10 top-one uplift gates, and the pre-answer-mention gate independently failed.

Learned Lessons

1. Canonical-answer likelihood is not mere answer-format leakage here: real thoughts beat length-matched shuffled and foreign controls, and rankings survived an answer-format change.
2. A statistically positive selector lift is not automatically large enough to justify banking. The top-gain trace beat both cheap selectors, but not by the frozen actionable margin.
3. Score the state that deployment can actually reach. A forced-close teacher-forced seam can be informative while remaining a poor proxy for autonomous continuation success.
4. Natural closure and answer parseability belong in the launch gate, not only in downstream diagnostics. Increasing N would multiply mostly cap-bound traces and is not a licensed fix.
5. Any follow-up should be a new experiment that first repairs or models the close/commit seam—for example, compare joint likelihood of </think>\n\nANSWER: y* with answer-only potential, or restrict to genuinely early-closing traces after demonstrating adequate coverage. It must not simply rerun this selector with more samples.

Research Program

• Programs: posttraining_and_adaptation, test_time_reasoning_budget, evidence_conditioned_selection
• Program question: can a dense, answer-conditioned signal identify self-generated thinking that is worth banking, where binary correct-answer rejection sampling selected inert rationalizations?
• Closest duplicate: qwen35_4b_bank_the_thoughts Phase 2 (C28), which selected the model's own thoughts only after a sampled answer happened to be correct and found no coverage gain over answer-only SFT.
• Other anchors: C9 (coherent thinking content is load-bearing), C46/C47 (within-task probability readouts and pooled-score failure), and C50 (breadth plus answer-seam-weighted SFT transfers).

Question

Does teacher-forced likelihood of a known canonical answer, measured after a sampled thought but before sampling an answer, identify concise and diverse thoughts that improve held-out deployable accuracy after SFT more than binary successful-answer rejection sampling?

This is RL-free but not oracle-free. Reference answers are used only during training-data curation; they are unavailable to every deployed selector and appear only in evaluation graders after the split is frozen.

Hypothesis

For prompt x, pre-answer thought z, and canonical answer y*, sample z ~ p(z|x) and score

gain(z) = log p(y* | x, z) - log p(y* | x, empty_thought).

Because thoughts are already sampled from the model prior, reweighting them by p(y*|x,z) is an importance-sampling approximation to the posterior over thoughts conditioned on the correct answer. Unlike a one-rollout binary filter, the score marginalizes answer-emission luck. The treatment should therefore enrich for thoughts that place the model in a genuinely better pre-answer state.

The claim is false if answer gain cannot rank fresh trace-conditioned answer rollouts within a task, if it merely selects short/format-priming/answer-copying traces, or if the selected traces do not beat length-matched random and binary-success trace SFT on fresh tasks.

Novelty Boundary

This is not a claim that future-token likelihood or reasoning-potential search is globally new. It is the repository's first controlled test of the following combination on the fixed model:

1. sample only the visible <think>...</think> region without showing the answer;
2. use the same model's teacher-forced canonical-answer likelihood as a dense oracle-side trace score;
3. validate that cheap score against fresh continuation success before training;
4. select quality first, then structural diversity, then the shortest near-best representative;
5. optionally branch at measured potential drops rather than perturbing arbitrary tokens; and
6. bank the resulting trace through matched, answer-seam-weighted QLoRA SFT.

Substrate And Firewall

• Model: only Qwen/Qwen3.5-4B, pinned to repository revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a.
• Substrate: a self-contained copy of the procedurally generated atom families from qwen35_4b_gauntlet_breadth_round1; no episode tasks in this first test.
• Training families: the ten original trained atom families.
• Family-transfer evaluation: the two original held-out atom families, never used in calibration, harvesting, threshold selection, or training.
• No content under benchmarks/ is imported, inspected, or used for training.
• All generator seeds are split-disjoint. Generated item IDs and canonical prompt digests must be disjoint across calibration, train, IID evaluation, held-family evaluation, and harder-level evaluation before any model call.

The atom-only choice is deliberate: short canonical ANSWER: targets make answer likelihood a clean measurement. Multi-turn action likelihood is a separate follow-up if this mechanism works.

Frozen Splits

Counts may shrink only for deterministic generator rejection or an explicit pre-result amendment. They may not be changed after observing treatment metrics.

Sampling Protocol

Calibration uses N=32 independent thoughts per prompt. The full pool uses N=128 with nested first-16/32/64/128 analyses from the single N=128 artifact; different vLLM batch shapes are never treated as paired continuations.

• temperature 1.0
• top-p 0.95
• top-k 20
• maximum sampled thinking tokens 512
• stop at the first </think> token; do not sample an answer in the treatment harvest
• explicit fixed shards, request seeds, prompt order, vLLM scheduler, capacity-fit receipt, and sampled-token accounting

Natural-close rate is a launch diagnostic. Forced-close thoughts may be scored, but they are not supervised as complete reasoning unless a sentence-boundary compression step explicitly turns them into a registered early-close target.

Answer-Potential Score

Render the exact thinking-channel prefix plus the sampled thought, the exact </think>\n\nANSWER: boundary, and the canonical answer content. Gather only the observed answer-token conditional log-probabilities. The pre-result operational amendment in the preregistration records why the implementation uses vLLM's exact targeted next-token readout instead of its pathological all-prompt-position API. Exclude the fixed boundary and terminal token.

For each trace store:

• summed and mean canonical-answer log-likelihood;
• gain against the same prompt with an empty thought;
• first answer-content-token log-probability;
• correct-versus-procedural-decoy margin when the family admits unambiguous decoys;
• sampled trace log-probability, length, closure, duplicate, and loop diagnostics;
• whether and where the canonical answer first appears verbatim in the thought; and
• scores at natural sentence/newline checkpoints for compression and branching.

Raw scores are never pooled across tasks. Every calibration statistic is task-macro or within-task. Multiple valid answer renderings are scored as an equivalence set or the item is excluded from the confirmatory scorer gate.

Gate G0: Cheap Scorer Validation

On the 64 calibration prompts, generate eight fresh short answer continuations from every thought using seeds not used for thought generation. These continuation outcomes approximate the expensive rollout potential and are not used in the likelihood score.

Proceed to the N=128 harvest only if all conditions hold:

1. task-macro within-task AUROC of answer gain against fresh rollout correctness is at least 0.65;
2. top-one-by-gain rollout success exceeds random-trace and shortest-trace selection by at least 0.10, with paired task-bootstrap lower confidence bound above zero for both comparisons;
3. answer gain beats trace length and trace prior-likelihood as within-task rankers;
4. real thoughts beat length-matched token-shuffled and foreign-task controls;
5. answer-format perturbations retain rank correlation Kendall tau >= 0.80; and
6. at least 75% of selected traces show positive gain before their first verbatim answer mention, or contain no verbatim answer mention.

If G0 fails, stop before SFT. The negative result is that canonical-answer likelihood is not a valid trace-value interface under this protocol.

Trace Selection

Quality, diversity, and brevity are lexicographic rather than collapsed into one tuned scalar:

1. discard malformed loops, exact duplicates, nonfinite scores, and traces below a frozen prior- likelihood floor;
2. retain the high-answer-gain set using the threshold frozen on calibration;
3. normalize identifiers, numbers, and whitespace, then use token-trigram Jaccard distance and deterministic farthest-first selection to retain distinct trace structures;
4. within each selected cluster, choose the shortest trace within the frozen near-best gain tolerance; and
5. cap at two traces per item and apply level/kind round-robin plus per-family and global-template caps, targeting 900-1,200 rows across at least eight families.

For prefix compression, score natural boundaries and choose the earliest boundary within the frozen tolerance of the trace's maximum score. Never cut inside a token or unfinished sentence.

Conditional Pivot/Branch Arm

The perturbation arm resamples suffixes; it never edits arbitrary tokens in-place. For each seed trace, preserve prefixes through positive potential jumps and branch immediately before the first material score drop or long plateau. Compare:

• independent_128; and
• independent_64_plus_branch_64.

Include every repeated-prefix prefill and sampled suffix token in matched-compute accounting. The branch pool is eligible for SFT only if it improves top-selected fresh-rollout success over independent sampling with a paired lower confidence bound above zero and does not reduce task or family coverage. Otherwise it remains a negative pool-level ablation.

SFT Arms

All arms use identical prompts, canonical answers, row counts, family/level quotas, optimizer steps, and base initialization. Only the trace-selection rule changes.

The applied potential arm may cover prompts with no successful sampled continuation. A matched-task intersection analysis separates better trace quality from broader task coverage.

Training format:

prompt -> <think> selected thought </think>\n\nANSWER: canonical answer

QLoRA recipe: rank 32, alpha 64, dropout 0.05, two epochs, learning rate 2e-4, prompt loss 0, thought-token loss 0.2, and close/answer loss 1.0. Truncated recovery contexts receive thought loss 0. The answer is retained in every arm because C50 found the answer-emission seam load-bearing; the experiment tests which thought context should be learned, not whether to suppress answer learning.

Screen every arm at training seed 42. If potential beats the strongest baseline by at least 0.03 on the frozen IID screen without a parse-rate loss, replicate potential and that baseline at seed 43 before making a positive claim. Otherwise the training verdict is negative/inconclusive and no selective replication is launched.

Evaluation

Run all base and trained arms through one inference backend and identical prompt renderer. Because vLLM 0.24 runtime LoRA silently no-ops for this model (C49), merge each adapter into the composite checkpoint and require a real greedy on-versus-off behavioral-difference gate before accepting its evaluation.

Primary:

• fresh IID greedy exact-answer accuracy at think budget 512;
• paired potential - success_rft and potential - random_length deltas;
• actual sampled thinking tokens and total forward-token accounting.

Secondary:

• parse rate and parse-conditional accuracy;
• no-think off-diagonal deployment;
• coverage/pass@8 and unique-answer diversity;
• held-family and harder-level macro accuracy;
• selected-trace length, natural-close rate, answer-copy rate, and generated-trace adoption;
• base sample-more curves using deployable majority/confidence selection; and
• oracle pass@k only as a clearly labeled ceiling.

The repository mission gate is stricter than an adapter delta: the potential-trained curve must beat the base sample-more curve at at least one matched actual-forward-token point without losing greedy accuracy or family-macro coverage. One-time training and curation cost is reported separately as an amortization curve.

Decision Rules

• Scorer negative: G0 fails; do not harvest or train.
• Selector positive / training negative: G0 passes but potential does not beat both matched trace controls after SFT; retain the scorer as a measurement result only.
• Local SFT positive: potential beats success_rft by at least 0.05 IID absolute accuracy, paired 95% confidence lower bound above zero, and beats random_length and potential_shuffle, with no parse or family-macro regression larger than 0.02.
• Efficiency positive: in addition, median thought length is at most 70% of success_rft, or treatment strictly Pareto-dominates it in accuracy versus sampled tokens.
• Mission positive: local SFT positive plus a win over matched-forward-token base sample-more.
• Transfer positive: held-family delta is positive with paired lower confidence bound above zero; otherwise any win is explicitly substrate/family-local.

Run

CPU and tiny GPU smoke:

.venv-vllm/bin/python experiments/qwen35_4b_answer_potential_trace_sft/scripts/run.py --smoke

Calibration gate:

.venv-vllm/bin/python experiments/qwen35_4b_answer_potential_trace_sft/scripts/run.py --stage calibrate

CPU-only reduction from the preserved calibration artifacts:

.venv-vllm/bin/python experiments/qwen35_4b_answer_potential_trace_sft/scripts/run.py --stage analyze-g0
.venv-vllm/bin/python experiments/qwen35_4b_answer_potential_trace_sft/scripts/analyze.py

Full-stage guard (expected to refuse for this terminal result):

.venv-vllm/bin/python experiments/qwen35_4b_answer_potential_trace_sft/scripts/run.py --stage full

The orchestrator refuses --stage full without a committed design-boundary receipt and a passing G0 artifact. The preserved runs/full_refusal.json confirms that it did so.

Results

G0 is terminally negative; see the Result section above and reports/report.md. The frozen pre-run design remains available in reports/preregistration.md, commit 3441dd23, and git history. No downstream training result exists.

Interpretation

The experiment is designed so every terminal outcome compounds knowledge: scorer failure rejects the interface cheaply; scorer success plus SFT failure separates measurement from learnability; and a controlled SFT win would establish answer-potential posterior mining as a better thought-bank builder than binary rejection sampling for the fixed 4B.

Knowledgebase Update

The terminal result is recorded as claim C51 and integrated into all three owning program ledgers, shared synthesis, the model playbook, the practitioner brief, and native chart specifications.

Artifacts

• idea_intake.md: novelty and routing decision
• reports/preregistration.md: immutable design specification
• reports/design_review.md: pre-run adversarial review
• configs/default.yaml: frozen counts, seeds, thresholds, and recipes
• src/vllm_runner.py: pinned common inference backend
• reports/artifact_manifest.yaml: terminal no-external-artifact receipt and regeneration commands
• runs/: compact receipts, gates, summaries, and scored rows
• analysis/: derived tables and plots

Report

Rendered from reports/report.md

Status

Terminal verdict: SCORER_NEGATIVE. The preregistered G0 gate failed. In accordance with the frozen decision rule, the full N=128 harvest, pivot/branch arm, trace selection, QLoRA training, and adapter evaluation were not run. The full-stage command was exercised and wrote ../runs/full_refusal.json before refusing to proceed.

This is a result about the scorer prerequisite, not a direct SFT comparison. It rules out banking traces selected by this exact answer-potential protocol; it does not establish that every possible answer-conditioned trace score or trace-SFT design fails.

Design Boundary

• Model: only Qwen/Qwen3.5-4B, revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a.
• The complete plan, gates, controls, and split policy were frozen in commit 3441dd23 before any GPU-scale call.
• Substrate: fresh, split-disjoint procedural atom tasks copied into the experiment; no benchmark content was read or used.
• Calibration: 64 prompts, 32 thoughts per prompt, temperature 1.0, top-p 0.95, top-k 20, and a 512-token thinking cap. Six prompts whose answer equivalence class was combinatorial were excluded from confirmatory potential scoring under the frozen equivalence-set exclusion rule.
• Outcome label: eight fresh short answer continuations per scored thought, generated with disjoint seeds on the same vLLM backend.
• The dated pre-result scoring amendment in preregistration.md replaced vLLM's pathological full-vocabulary prompt_logprobs path with mathematically identical targeted next-token reads. HF bf16 SDPA parity passed at a 0.060-nat/token maximum discrepancy versus the frozen 0.15 tolerance.

Primary Gate

Calibration produced 2,048 thoughts. The 58 scorable prompts contributed 1,856 scored thoughts and 14,848 fresh rollout outcomes. The six numbered G0 conditions expand to eight stored booleans because the two selector comparisons and two corruption controls are evaluated independently.

Overall: 3/8 stored criteria passed; G0 failed. Full numerical output and artifact checksums are preserved in ../runs/calibration/g0.json, with a compact table in ../analysis/g0_metrics.csv.

What The Signal Did And Did Not Do

The top trace by answer gain had fresh rollout success 0.2026, compared with 0.1293 for the seeded random trace and 0.1444 for the shortest trace. Both paired bootstrap lower bounds were positive. Thus the score has modest within-task ranking value; the failure is not equivalent to chance. The preregistered gate required a practically useful +0.10 uplift, however, and observed uplifts were only +0.0733 and +0.0582.

The mechanism controls also show real trace dependence. A real thought improved canonical-answer gain over a length-matched token shuffle by 0.555 nats task-macro, and over a foreign-task thought by 4.791 nats; both confidence intervals excluded zero. Changing the answer rendering preserved rankings at Kendall tau 0.830. A positive canonical-versus-procedural-decoy margin occurred for 91.3% of scored thoughts. The scorer is therefore sensitive to task-relevant thought content and robust to the tested format perturbation, but that information did not translate into sufficient autonomous-answer lift.

The trace-prior ranker could not be evaluated because the thought-generation request did not capture sample log-probabilities. The implementation writes JSON strictly and represents the missing metric as null; the criterion fails closed. This instrumentation miss is material and recorded, but it cannot rescue the gate: four other independent conditions failed.

The Dominant Failure: An Unreachable Answer State

The scorer teacher-forced y* after an injected </think>\n\nANSWER: seam. Almost every sampled thought was still running when that seam was injected. The continuation test, by contrast, required the model to recover from the forced close and emit a parseable terse answer. The extremely low parse rate and high parse-conditional accuracy locate the main loss at this interface: when the model committed in the expected form it was usually right, but it rarely entered that form after a cap-bound thought.

This explains how answer potential can pass the corruption and format tests while failing selection: it measures whether the reference answer is locally compatible with a counterfactual forced answer state, not whether the model will autonomously terminate and express that answer. It also aligns with C50's independent finding that the answer-emission seam can dominate deployable behavior.

The pre-answer-mention diagnostic adds a second caution. Only 33/58 selected traces showed positive gain before their first verbatim answer mention or never mentioned it; the required count was at least 44/58. Ten selected traces never mentioned the answer, while only 23 of the remaining 48 passed before mention. Some high gain therefore arrives too late to rule out answer-copy or answer-rehearsal effects.

Heterogeneity

The scorer was not uniform across task families. Task-macro AUROC exceeded 0.65 for caravan (0.653), foundry_ledger (0.710), and gatepost (0.764). Glyphgate's nominal 0.955 came from only one mixed task and is not stable evidence. Ferrier, kilnrite, loomfix, and runeward were near chance. Parse rates ranged from 0.5% to 56.2%, so family-level discrimination is entangled with whether the answer interface works for that family. The full table is ../analysis/family_summary.csv.

Compute And Provenance

• Counted logical tokens: 32,805,906.
• Thought sampling: 490,784 prompt plus 1,046,911 sampled tokens.
• Targeted scoring: 19,474,315 repeated-prefix prefill plus 26,702 one-token reads.
• Rollouts: 11,110,544 prompt plus 656,650 sampled tokens.
• Sum of timed GPU operations: 3,128.8 seconds (52.15 minutes); this is an operation-time sum, not a claim about wall-clock exclusivity.

Every GPU artifact was written atomically before reduction. The first reduction then exposed a configuration-key typo (premember versus premention). No inference was regenerated: the added CPU-only --stage analyze-g0 path verified and reduced the saved artifacts. Strict JSON serialization also caught and removed a non-standard NaN, replacing the unavailable trace-prior metric with null. These recovery details and the design receipt remain part of the committed provenance.

Learned Lessons And Decision

1. Dense oracle-side signal is not enough. Answer potential carried real trace-specific information, yet its ranking lift missed the actionable bar. Selector validation must precede expensive curation or SFT.
2. Deployment matching is part of score validity. Scoring a teacher-forced answer after a forced boundary can validate a counterfactual state the model almost never reaches. Termination and parsing need to be part of the measured event.
3. Do not answer a closure failure with larger N. At 99.37% cap contact, N=128 would mostly buy four times as many unfinished traces. The matched-compute sample-more baseline would remain the bar, and no evidence licenses that expense.
4. Capture every declared baseline at generation time. Trace-prior log-probability was a frozen comparator but was not recorded. Future harnesses should assert required fields before the first scientific shard, not merely fail closed during reduction.
5. Preserve effect-size gates. The positive confidence intervals could have invited a post-hoc success story. The preregistered +0.10 threshold correctly separated detectable signal from a method worth training on.
6. The next useful question is the close/commit seam, not a selector retune. A fresh experiment could compare joint likelihood of </think>\n\nANSWER: y* against answer-only potential, or first create adequate naturally early-closing coverage and then repeat the within-task gate. It should retain shuffled/foreign/length/prior controls and treat autonomous parseability as confirmatory.

Decision: stop this experiment at G0. No pivot branching, diversity optimization, SFT, adapter, or held-out capability claim is authorized. The complete terminal summary is ../analysis/summary.md.

Smoke Evidence

• Frozen procedural split construction produced 64 calibration, 600 train, 400 IID, 100 held-family, and 100 hard items with zero ID, prompt, digest, or generator-seed overlap.
• Twenty-five CPU tests pass across the firewall, verifier-equivalent answers, controls, statistics, and vLLM geometry.
• The corrected real-model smoke produced four finite trace scores and eight fresh answer continuations with exact registered CUDA-graph geometry and a passing live-KV capacity receipt.
• HF bf16 SDPA versus vLLM bf16 targeted likelihood differed by at most 0.060 nats per answer token, below the pre-calibration plumbing tolerance of 0.15. HF rows are diagnostic only.
• The 64-token smoke traces all contacted the thought cap; their zero rollout score is intentionally non-scientific and does not enter G0.

These smoke rows were plumbing checks only and were not included in any scientific statistic above.

Experiment log 4

Data files 6

Result tables and metrics copied from the experiment folder — preview inline or open the raw file.

• previewanalysis/compute_summary.json709 B
• previewanalysis/family_summary.csv1.0 kB
• previewanalysis/g0_metrics.csv831 B
• previewanalysis/g0_summary.json786 B
• previewanalysis/selected_trace_summary.csv6.6 kB
• previewruns/smoke/result.json33 kB

Reproduce

Smoke test

.venv-vllm/bin/python experiments/qwen35_4b_answer_potential_trace_sft/scripts/run.py --smoke

Run steps are documented inside the experiment folder (README and scripts).

Browse the experiment folder on GitHub ↗

Related

Cited by claims

• ✗ Negative C51 · Answer-only potential after cap-bound thoughts is a real but NON-ACTIONABLE selector: it detects relevant trace content yet fails G0 because the teacher-forced answer seam is rarely deployable

Related experiments

Nearest by shared name, tags, and programs.

• 2026-07-12 Qwen3.5-4B Balanced-Core Answer-Potential SFT
• 2026-07-10 Qwen3.5-4B Long-Horizon Answer-Potential SFT
• 2026-07-13 State-Formation Capacity Adjudication
• imported 2026-07-12 Qwen Register Trace Refiner
• 2026-06-24 Qwen3.5-4B Active Counterexample Trace Selection
• 2026-06-23 Qwen Context-Conditioned Trace Verifier