Entropy-routed think-pivot optimization round 2
The one idea you need
The goal was to fix the model's word choice at one confident wrong turn in its reasoning. But it's like retuning a single guitar string through a peg geared to all six: you can't shift that one note without detuning the whole chord.
The question
Can you make a small reasoning model smarter by gently editing the exact moments where it confidently takes a wrong turn in its thinking?
What we found
No. Gently pulling the better word up at 155 hand-picked wrong-turn moments was cleaner than shoving the bad word down, and it carried real signal, beating a scrambled-label control by about 14 points on fresh coding repairs. But it still lost to the untouched model, 54% versus 60% of repositories fixed, and the nudge leaked into unrelated words well past the safety limit.
Why it matters
Before trusting a targeted fine-tune to fix specific reasoning mistakes, measure how far it moves unrelated word-scores, not just its target, and always run a scrambled-label control. If drift clears your locality budget, more examples won't rescue it; shrink the dose.
On this page
Results at a glance 3
How to read
Three training styles along the bottom: push the wrong word down, pull the right word up, and pull up with scrambled labels. Bar height is how far unrelated word-scores drifted; lower is better, and a safety line marks the allowed spillover.
Takeaway → Pull-up's bar is about a third shorter than push-down's, yet all three, even scrambled labels, rise above the safety line, so no method stayed local.
Data table
| training arm | median neighboring-logit movement |
|---|---|
| push failed token down | 0.229 |
| pull successful token up | 0.145 |
| pull up, scrambled labels | 0.12 |
Numbers from
Technical framing
Pulling up is less disruptive than pushing down, but still misses locality — The frozen safety ceiling was 0.10 logits. Positive-only pull-up cut collateral by 36.6% versus demotion, but every arm still failed locality; even scrambled labels moved the network broadly.
How to read
Five agent versions along the bottom; bar height is the share of repository repairs passing hidden tests, higher is better: untouched model over eight serial turns, push down, pull up, pull up with scrambled labels, and the untouched model split into two shorter tries.
Takeaway → The untouched eight-turn agent leads near 60%; the best edit reaches 54%, and splitting the calls into two short tries collapses to 31%, so long feedback loops mattered most.
Data table
| agent arm | hidden-test repository repairs |
|---|---|
| base, 8 serial turns | 59.7% |
| push down | 47.2% |
| pull up | 54.2% |
| pull up, scrambled | 40.3% |
| base, two 4-turn tries | 30.6% |
Numbers from
Technical framing
The unchanged deep coding agent remains the capability bar — Pull-up solved 39/72 versus deep base 43/72, so it failed the preregistered capability gate. It beat matched-compute branching, showing that preserving a long feedback loop mattered more than splitting the calls.
How to read
Four evaluations along the bottom; bar height is the real pull-up minus the scrambled pull-up, so above zero means real labels helped: shorter and longer fresh reasoning, the practice gym, and the coding agent.
Takeaway → Real labels beat scrambled by up to 14 points on coding and the practice gym, but dip negative at the longer reasoning budget, evidence of signal rather than a reliable gain.
Data table
| evaluation | real pull-up minus scrambled pull-up |
|---|---|
| fresh reasoning, 1024 | 2.5% |
| fresh reasoning, 2048 | -2% |
| parent-style gym | 6.2% |
| repository agent | 13.9% |
Numbers from
Technical framing
True outcome directions help locally, but not consistently across reasoning budgets — Real labels separated on the gym and coding agent (the coding 95% interval touches zero), but the sign flipped at the longer whitebox budget. This is evidence for label signal, not a general capability gain.
In the author’s words from the Overview · “Result”
P0 and every broad P4 guard passed, but P1 locality, P2 fresh whitebox, and P3 agentic coding all failed. The selector retained 155 confident-wrong-turn rows. Positive-only uplift moved its target on 75.5% of them and reduced median non-target drift from demotion's 0.229 logits to 0.145, but missed the 0.10 locality ceiling. On 72 fresh repository repairs, deep base passed 43, uplift 39, demote 34, and shuffled uplift 29. Uplift's +13.89pp separation from shuffled labels (paired-bootstrap 95% CI [0.00,+27.78]) and +6.25pp gym separation show that the outcome directions were not empty; neither effect overcame the generic shared-weight update. Whitebox uplift was +0.26pp at think@1024 and −3.06pp at think@2048 versus base. … Read the full result →
Overview
This completed follow-up tested whether entropy-routed, outcome-labeled thought pivots make Qwen3.5-4B better at held-out reasoning and multi-turn coding, comparing bounded successful-token uplift against conventional FTPO demotion and a shuffled-outcome control. The preregistered verdict is LOW_DOSE_NULL: true labels retained a local signal, but no trained arm elicited general capability.
Research Program
- Primary:
agentic_breadth_installation; cross-cuttingposttraining_and_adaptationandtest_time_reasoning_budget. - Program question: after breadth-SFT's one-time C50 gain and round 1 FTPO's C52 failure, can a different, localized update mechanism move general agentic capability rather than merely suppress surface pathologies?
- Prior anchors: C52 (near-parity FTPO harms think flow), C50 (signal placement can transfer broadly), C29 (preference training is fragile), C41/C42 (uncertainty localizes useful choices), and the positive-pressure locality audit (incremental signal must exist at the exact fork).
Question
Round 1 pushed successful and failed sibling tokens two logits apart even when the base regarded them as peers; its shuffled labels caused the same delayed closure and degradation. This round asks two separable questions:
- Does conventional FTPO work when restricted to true confident wrong turns?
- At those same turns, is it safer or more useful to pull up the empirically fruitful continuation by a bounded amount while pinning the failed token?
Entropy and varentropy are measured at the actual harvest temperature. They do not supply correctness labels; they distinguish a focused attractor from broad noise and a deterministic groove from a spiky distribution with a plausible alternative tail.
Hypothesis
A failed argmax token with P≥0.5, a ≥0.5-logit lead, low entropy, and nonzero varentropy has the outlier geometry missing from round 1. Pairwise demotion may therefore become safe. The primary hypothesis is stronger: a +0.5-logit, positive-only lift of its successful sibling will preserve think flow better than manufacturing a two-logit chosen-over-rejected margin, and the gain will not appear under shuffled outcomes.
Setup
- Model: only
Qwen/Qwen3.5-4B, pinned revision. - Training source: round 1's exact committed, already-regularized real and shuffled FTPO rows. No new labels, teacher, benchmark content, or gold reasoning enters training.
- Geometry: frozen-base exact logits, one context per forward (the hybrid architecture fails padded equivalence); entropy/varentropy at T=0.6.
- Arms: base,
demote,uplift(primary),uplift_shuffled. - Trainer: LoRA r256/α128, two epochs maximum, final-position logits only, two-tier raw-logit tether.
demoteis published FTPO;uplifttargets a +0.5 chosen-logit gain and treats rejected as non-target. - Fresh evaluation: parent whitebox/gym substrates on new seeds plus a new held-out six-family repository-repair agent.
- Coding agent: iterative tree/read/search/test/exact-patch/submit tools over materialized Python repos; final hidden-test grading after eight turns.
- Matched-compute baseline: two independent four-turn base trajectories, the same maximum eight model calls and 6,144 sampled tokens per task. Explicit submit rate is separate from final-workspace correctness.
- Blackbox: menagerie only through
run.pyCLI and aggregate scores, and only after a preregistered whitebox gate. - Full predictions and outcome labels: preregistration.
Run
CPU smoke:
python3 scripts/run.py --smokeGPU smoke (two exact-logit rows + one task per repository family):
python3 scripts/run.py --gpu-smokeFull staged pipeline through analysis:
python3 scripts/run.py --full \
--artifact-root ../../large_artifacts/qwen35_4b_think_ftpo_round2If a future exact reproduction sets analysis/summary.json to menagerie_eligible=true, run the two frozen paired quick events and regenerate analysis:
python3 scripts/bench.py --tier quick --seed 62011 --arms base uplift \
--merged uplift=../../large_artifacts/qwen35_4b_think_ftpo_round2/merged/uplift
python3 scripts/bench.py --tier quick --seed 62012 --arms base uplift \
--merged uplift=../../large_artifacts/qwen35_4b_think_ftpo_round2/merged/uplift
python3 scripts/analyze.pyResult
P0 and every broad P4 guard passed, but P1 locality, P2 fresh whitebox, and P3 agentic coding all failed. The selector retained 155 confident-wrong-turn rows. Positive-only uplift moved its target on 75.5% of them and reduced median non-target drift from demotion's 0.229 logits to 0.145, but missed the 0.10 locality ceiling.
On 72 fresh repository repairs, deep base passed 43, uplift 39, demote 34, and shuffled uplift 29. Uplift's +13.89pp separation from shuffled labels (paired-bootstrap 95% CI [0.00,+27.78]) and +6.25pp gym separation show that the outcome directions were not empty; neither effect overcame the generic shared-weight update. Whitebox uplift was +0.26pp at think@1024 and −3.06pp at think@2048 versus base. Menagerie was ineligible and zero benchmark seeds were consumed. See the final report.
Entropy/varentropy were diagnostically useful but not monotone steering coordinates: the lowest-varentropy uplift quartile had the cleanest update (0.122 median non-target drift), while the third quartile was worst (0.176). The next lever is parameter locality—a lower-dose or genuinely context-gated intervention that clears P1—not simply more rows or higher-varentropy pivots.
Interpretation boundary
The parent pool can supply only a few hundred qualified rows. A flat outcome is therefore a low-dose null, not proof that thought steering cannot work. Control-equivalent harm, termination damage, or a positive signal with clean guards are informative at this dose and decide whether a larger harvest is worth the GPU time.
Artifacts
data/rows_{pivot,shuffled}.jsonl.gz: exact parent inputs; selected rows are generated after P0.src/repo_{tasks,agent}.py: procedural repositories and iterative harness.scripts/score_rows.py: dominance/entropy/varentropy census and matched set.scripts/train_sparse.py,audit_logits.py: objectives and locality audit.scripts/eval_whitebox.py,eval_gym.py,eval_repo_agent.py: fresh gates.analysis/: regenerated paired-bootstrap verdict after the full run.- External adapters and merged checkpoints are declared in
reports/artifact_manifest.yaml; model weights never enter git.
Report
Rendered from reports/report.md
Verdict
LOW_DOSE_NULL (preregistered label). Entropy-routed, outcome-conditioned single-token updates did not elicit a general capability gain from Qwen3.5-4B. The primary positive-only uplift arm was less destructive than conventional demote and separated from shuffled labels on two local substrates, but it did not beat the frozen model on the held-out repository agent: 39/72 versus 43/72 (−5.56pp, paired-bootstrap 95% CI [−19.44,+8.33]). It also missed both fresh whitebox gates.
The experiment resolves the round-1 follow-up: selecting confident wrong turns is not sufficient to make weight-space FTPO local. Pulling the successful token up changes the damage profile and preserves some outcome-label signal, but the shared LoRA update still moves non-target logits too far. No menagerie event was run; zero blackbox benchmark seeds were consumed.
Here LOW_DOSE_NULL does not mean “nothing moved.” It is the frozen analyzer's label for missing the capability gates when the real-label arm separates from the shuffled arm enough that the stricter GENERIC_TRAINING_HARM rule does not apply. The absolute held-out result remains negative.
Design
The same 155 frozen-base-qualified real pivot rows fed two arms:
demote: the published pairwise FTPO objective with a two-logit margin;uplift: a bounded positive-only objective targeting a +0.5 chosen-token logit gain while treating the failed token as non-target.
An otherwise identical uplift_shuffled arm used outcome-permuted parent labels. Rows qualified at the actual harvest temperature, T=0.6, only when the failed token was base argmax, led the successful sibling by at least 0.5 logits, had P(failed)≥0.5, entropy≤1.5 nats, varentropy≥0.1 nats², and retained a plausible successful alternative. Full thresholds, gates, and outcome labels were frozen in preregistration.md.
The capability north star was a fresh, procedural six-family repository- repair suite. An eight-turn tool loop could inspect files, search, run visible tests, apply exact patches, and submit; final workspaces were scored by hidden tests. The matched-compute sample-more baseline unioned two independent four-turn base trajectories under the same eight-call / 6,144-reserved-token ceiling.
Gate results
| Gate | Result | Evidence |
|---|---|---|
| P0 geometry | PASS | 155 matched rows; minimum 128 |
| P1 targeted mechanism | FAIL, all arms | hit-rate bars passed, but mean per-row median absolute non-target drift was 0.229/0.145/0.120 logits for demote/uplift/shuffled; ceiling 0.10 |
| P2 fresh whitebox | FAIL | uplift +0.26pp at think@1024 and −3.06pp at think@2048 vs base; required ≥+3pp plus control separation and termination guards |
| P3 repository agent | FAIL | uplift 39/72 vs base 43/72; required ≥+8pp and a sample-more win |
| P4 broad guards | PASS | C49, gym floor, collapse, and no-think guards all passed |
| P5 menagerie | NOT ELIGIBLE | P1, P2, and P3 failed |
Geometry and training mechanism
The selector premise was real but sparse. Of 615 real and 661 shuffled parent rows, 155 (25.2%) and 166 (25.1%) qualified; seeded matching retained 155 per arm. Full-pool medians for the real rows were P(failed)=0.397, failed-minus- best-successful gap=0.0 logits, entropy=0.692 nats, and varentropy=0.441 nats².
All arms hit the training safety stop early: demote after 8/20 optimizer steps, uplift and shuffled after 5/20. The exact-logit audit shows why downstream evaluation remained unsafe despite intended-target movement:
| Arm | Objective hit | Chosen gain | Failed-token drift | Pair-gap shift | Median non-target drift | P95 non-target drift |
|---|---|---|---|---|---|---|
| demote | 40.0% | +0.382 | −1.043 | +1.425 | 0.229 | 0.547 |
| uplift | 75.5% | +0.781 | +0.502 | +0.280 | 0.145 | 0.363 |
| uplift_shuffled | 76.1% | +0.762 | +0.550 | +0.212 | 0.120 | 0.304 |
Positive-only uplift cut median collateral by 36.6% relative to demotion and avoided manufacturing a large chosen-over-failed margin. It did not isolate the edit: even the tethered failed token rose +0.50 logits on average. The shuffled arm's similar movement identifies a generic shared-parameter update, not outcome-conditioned steering, as the dominant P1 failure.
Entropy and varentropy routing
Entropy and varentropy were useful instruments for locating and auditing forks, but they were not monotone measures of editability. For the real uplift arm, entropy quartiles Q1→Q4 had objective-hit rates 79.5%, 89.5%, 76.9%, 56.4% and non-target drifts 0.163, 0.134, 0.137, 0.147. Varentropy quartiles Q1→Q4 had hit rates 89.7%, 71.1%, 84.6%, 56.4% and drifts 0.122, 0.148, 0.176, 0.136.
The cleanest stratum was lowest varentropy, not highest. The preregistered minimum-varentropy filter removed deterministic grooves, but increasing varentropy beyond that did not create safer or more fruitful weight edits. This is a routing result, not permission to post-hoc train on Q1: any such filter needs an independent experiment and still must clear P1.
Fresh whitebox outcomes
The nominal N=400 allocation produced 392 paired tasks because integer per-cell allocation was used (a 2% protocol deviation; no task was selected or removed by outcome).
| Budget | Base | Demote Δ | Uplift Δ | Shuffled Δ | Uplift−shuffled |
|---|---|---|---|---|---|
| think@1024 | 53.57% | −2.81pp [-6.63,+0.77] | +0.26pp [-3.57,+4.08] | −2.30pp [-6.38,+1.79] | +2.55pp [-1.53,+6.89] |
| think@2048 | 58.16% | −3.32pp [-7.40,+0.77] | −3.06pp [-7.40,+1.28] | −1.02pp [-5.87,+3.83] | −2.04pp [-6.63,+2.81] |
Uplift did not transfer consistently across budgets. At think@2048 it raised natural closure from 15.05% to 19.13%, but answer-limit contacts also rose from 37.24% to 40.56%, violating the ≤2pp termination guard. Demotion raised answer-limit contacts by 3.32pp/2.81pp at 1024/2048. Loops remained rare in every arm (0–0.77%), reinforcing that repetition was not the deployed-budget bottleneck.
Repository-agent north star
| Arm | Hidden-test pass | Δ vs deep base (95% CI) | Submit | Invalid actions / turn | Mean sampled tokens |
|---|---|---|---|---|---|
| base, eight-turn deep | 43/72 (59.72%) | — | 45.83% | 9.98% | 2,510 |
| demote | 34/72 (47.22%) | −12.50pp [−23.61,−1.39] | 38.89% | 13.84% | 2,597 |
| uplift | 39/72 (54.17%) | −5.56pp [−19.44,+8.33] | 38.89% | 12.27% | 2,643 |
| uplift_shuffled | 29/72 (40.28%) | −19.44pp [−33.33,−5.56] | 26.39% | 14.26% | 2,816 |
| base sample-more, 2×4 turns | 22/72 (30.56%) | — | 1.39% | 4.86% | 2,341 |
Uplift exceeded shuffled by +13.89pp, with its paired-bootstrap interval touching zero [0.00,+27.78]. This is compatible with useful information in the true outcome directions, as is the gym separation below, but it is not a general capability gain: uplift remained 5.56pp below the stronger serial base agent. It beat matched-compute branching by +23.61pp [+11.11,+36.11], but the preregistration required beating both sample-more and deep base. Here, preserving eight serial feedback turns was much stronger than splitting them into two short trajectories.
Broad guards and controls
Gym aggregate success was base 48.55%, demote 49.29%, uplift 53.27%, and shuffled 47.02%. Thus true-label uplift exceeded shuffled by 6.25pp on the parent-style gym, while held-out-family uplift was 50.0% versus base 53.57%. The direction signal is substrate-local, not held-out breadth.
All merged checkpoints passed the mandatory C49 on-vs-off behavioral gate (0/8 outputs identical to base). Collapse guards passed: base greedy/pass@8 were 4.17%/9.17%, versus demote 5.00%/12.50%, uplift 5.83%/10.83%, and shuffled 6.67%/11.67%. No-think success was base 36.67%, demote 39.17%, uplift 37.50%, and shuffled 36.67%. This is not C29-style collapse; the damage is specific enough to evade coarse guards yet broad enough to erase transfer.
Interpretation
Round 2 falsifies the simple rescue proposed after round 1. A confident failed argmax plus low entropy and nonzero varentropy restores the published geometry, but not the required parameter locality. Conventional demotion is decisively harmful on the coding agent. Positive-only pressure is the better direction—it retains a real-label advantage over shuffled training and reduces collateral—but shared LoRA weights still change neighboring logits and agent behavior more than the sparse label signal can repay at 155 rows.
The next experiment should therefore not merely harvest more rows or increase varentropy. First earn locality with a smaller (+0.25) uplift or a genuinely context-gated last-layer/activation intervention, using P1 as a hard preflight. Only a mechanism below 0.10 median non-target drift should receive a fresh, larger outcome harvest and another agentic transfer test. Long-context loop FTPO remains a separate 16k+ question; this experiment provides no reason to apply it at deployed budgets.
Reproducibility and artifacts
The result is regenerated by scripts/run.py --full; machine-readable gates are in analysis/summary.json. Training rows and all small run receipts are committed. Adapters and merged checkpoints are external, checksummed in artifact_manifest.yaml. The full run used only Qwen/Qwen3.5-4B, kept HF exact-logit work separate from vLLM generation, and evaluated every vLLM arm as a merged checkpoint.
Experiment log 5
Show the running log (5 entries, 2026-07-11)
Design and smoke (before scientific run)
- Routed to
agentic_breadth_installation; closest duplicate is round 1/C52. - Exploratory replay of the already-regularized real rows found 290/615 failed tokens were base argmax and 172 led their successful sibling by ≥0.5 logits. Entropy and varentropy expose a spiky-conflicted subset; thresholds were then frozen before scoring the shuffled pool or training.
- Uncapping prefix-tree nodes yielded only 885 raw nodes vs 879 previously, so round 2 is explicitly a low-dose controlled pilot rather than a new harvest.
- CPU invariants pass for all repository families, sandbox paths, parser, and both objectives. Exact-logit and vLLM GPU paths loaded successfully.
- The initial one-operator repair tasks saturated (base patched 6/6) and were rejected during smoke. Replaced with semantic multi-line maintenance faults; tiny calibration: deep base final-workspace 2/6, matched two-by-four-turn baseline 1/6. These adaptive smoke items are not scientific evidence.
Frozen design checkpoint
- Design, code, smoke receipts, and preregistration were committed before the scientific run as
c6480deeand pushed toagent/think-ftpo-round2. Full command:
PYTHONDONTWRITEBYTECODE=1 python3 \ experiments/qwen35_4b_think_ftpo_round2/scripts/run.py --full \ --artifact-root \ /workspace/small-model-experimentation/large_artifacts/qwen35_4b_think_ftpo_round2
Full run (2026-07-11)
- P0 passed: 155/615 real rows and 166/661 shuffled rows met the frozen confident-wrong-turn/entropy/varentropy geometry; seeded matching retained 155 rows per arm (minimum 128).
- Training safety-stopped as registered: demote at 8/20 steps (165 s), uplift at 5/20 (104 s), and shuffled uplift at 5/20 (103 s).
- P1 failed for every arm. Objective hit/non-target drift were 40.0%/0.229 logits (demote), 75.5%/0.145 (uplift), and 76.1%/0.120 (shuffled); the frozen bars were ≥35% and ≤0.10. Uplift was 36.6% less collateral than demotion but did not localize the update.
- Entropy was not monotone with safety. Uplift entropy-quartile drift was 0.163/0.134/0.137/0.147. Varentropy was more diagnostic but ran opposite the naive “more conflict is better” read: Q1 drift 0.122 was cleanest, Q3 drift 0.176 was worst. These strata are explanatory only, not a post-hoc selector.
- P2 failed. Fresh paired whitebox uplift-minus-base was +0.26pp at 1024 (95% CI −3.57,+4.08) and −3.06pp at 2048 (−7.40,+1.28). Uplift-minus- shuffled changed sign (+2.55pp/−2.04pp). At 2048, natural closure improved +4.08pp but answer-limit contacts worsened +3.32pp, failing the termination guard.
- P3 failed. On 72 hidden-tested repository repairs, deep base passed 43, demote 34, uplift 39, and shuffled uplift 29. Paired uplift-minus-base was −5.56pp (95% CI −19.44,+8.33); uplift-minus-shuffled was +13.89pp with the interval touching zero (0.00,+27.78). The matched two-by-four-turn base baseline passed 22; uplift beat it by +23.61pp, but the registered north star also required beating the stronger deep base.
- P4 passed for all trained arms: merged C49 on/off, gym floor, collapse greedy/pass@8, and no-think guards were clean. Gym aggregate was 48.55% base, 49.29% demote, 53.27% uplift, and 47.02% shuffled.
- Final registered label:
LOW_DOSE_NULL. P5 was ineligible; menagerie was not run and no blackbox seeds were consumed.
Protocol deviation
The whitebox plan named N=400, while the frozen per-cell integer allocator materialized 392 paired tasks (98 per cell), a 2% shortfall. No outcomes were inspected or filtered and every materialized task was retained. The effective N and exact paired intervals are reported rather than silently rounding to the nominal target.
Read and next decision
Outcome labels carried a substrate-local direction signal relative to the shuffled control, but shared-weight collateral exceeded it. Confident-outlier geometry is necessary but not sufficient for FTPO. Do not scale this LoRA recipe or select higher-varentropy rows. First test a lower +0.25 uplift or a genuinely context-gated last-layer/activation edit behind the same P1 locality gate; only then fund a larger fresh harvest and agentic transfer run.
Data files 7
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
runs/eval_gym_base.json685 Bruns/eval_gym_demote.json780 Bruns/eval_gym_uplift.json781 Bruns/eval_gym_uplift_shuffled.json738 Bruns/train_summary_demote.json1.9 kBruns/train_summary_uplift.json1.4 kBruns/train_summary_uplift_shuffled.json1.4 kB
Reproduce
Smoke test
python3 experiments/qwen35_4b_think_ftpo_round2/scripts/run.py --gpu-smokeFull run
python3 experiments/qwen35_4b_think_ftpo_round2/scripts/run.py --full --artifact-root large_artifacts/qwen35_4b_think_ftpo_round2Run steps are documented inside the experiment folder (README and scripts).