Qwen3.5-4B Reliability Exec OPSD Audit
The one idea you need
The model writes several candidate programs by adapting retrieved algorithms. Wherever a correct and a buggy version diverge, it already leans the right way almost every time — yet, like a chess player who sees the best move but plays a worse one, it rarely assembles a working program (a workable one exists for only about a third of tasks).
The question
When a small coding model writes several programs that all pass the quick sample checks a user can see, can cheap tricks reliably pick the truly correct one?
What we found
No. All three cheap fixes failed. Ranking programs by the model's own confidence picked working code less often than just taking the first one that passed the visible sample tests (6 of 24 versus 8), and let more bugs slip through. Showing it one real failing test helped no more than scrambled nonsense — yet leaking the full answer clearly helped, so the test itself could see real signal.
Why it matters
Don't trust a small model's confidence, steadier decoding, or a single failing example to screen out buggy code — it already knows the right branch and still fails. Gather independent evidence: run several distinct solutions and commit only when they agree.
On this page
Results at a glance 3
How to read
Each bar is a type of hint fed to the model; height is how much more strongly it then leaned toward the correct program. Bars include a scrambled-nonsense hint and an answer-leaking hint for comparison. Taller means a stronger, more useful nudge.
Takeaway → The real failing-test bar sits just below the scrambled-nonsense bar and far below the answer-leak bar, so a real failing test carries no genuine task-specific information.
Data table
| teacher context at task-specific forks | delta over no-hint student |
|---|---|
| failing input only | 0.026 |
| failing input + correct output | 0.063 |
| shuffled execution observation | 0.079 |
| full-reference leakage ceiling | 0.237 |
Numbers from reports/exec_pressure_summary.json (also final_report.md table)
Technical framing
Does execution evidence point the teacher at the correct branch? — Real execution evidence (+0.063) does not beat shuffled evidence (+0.079); only leaking the answer (+0.237) helps, so the OPSD gate failed.
How to read
Two bars per method over 24 tasks: working programs picked (want it tall) and buggy programs picked (want it short). Methods are taking the first program that passes the visible tests, ranking by the model's confidence, and a perfect picker.
Takeaway → Confidence ranking has a shorter working bar and a taller buggy bar than simply taking the first passing program, so it makes selection worse.
Data table
| selector (semantic temperature-union pool, 24 tasks) | hidden-correct commits | hidden-wrong commits |
|---|---|---|
| first visible | 8 | 6 |
| MAP mean logprob | 6 | 8 |
| oracle (hidden tests) | 8 | 0 |
Numbers from reports/map_selector_semantic_temps_summary.json
Technical framing
MAP likelihood selector picks worse than first-visible — Raw model likelihood selects fewer hidden-correct and more hidden-wrong candidates than simply taking the first visible pass.
How to read
Across three settings from most predictable to most varied: how often at least one workable program appeared and how often the model's writing was correct on the first try, both over 24 tasks. Higher is better for both.
Takeaway → No setting wins: the most predictable one gets slightly more first-try correct answers but fewer workable candidates — a wash with no clear winner.
Data table
| decoding temperature | pool coverage | pass@1 proxy |
|---|---|---|
| T=0.0 (greedy) | 29.2% | 16.7% |
| T=0.1 | 33.3% | 12.5% |
| T=0.2 (baseline) | 33.3% | 12.5% |
Numbers from final_report.md low-temperature probe table
Technical framing
Lower decoding temperature does not improve retrieval adaptation — Greedy decoding slightly raises pass@1 proxy but lowers coverage; no temperature dominates the existing T=0.2 pool.
In the author’s words from the Overview
Low-temperature retrieval adaptation: compare semantic adaptation at T=0.0, T=0.1, and the local T=0.2 baseline. Execution-grounded locality audit: at same-prefix code forks, compare teacher preference for the hidden-correct branch versus the hidden-wrong branch when the teacher is given a failing input, observed wrong output, and correct output. Run OPSD training only if execution-grounded hints add task-specific correct-branch preference beyond the no-hint student and shuffled-observation control. Full-reference hints are included only as a leakage ceiling.
Overview
Standalone experiment testing two reliability probes and one execution-grounded OPSD locality audit.
The experiment asks whether hidden-correct retrieval adaptations are already preferred by Qwen's own likelihood and whether behavioral counterexample evidence gives a hinted teacher task-specific branch signal that weak retrieval hints did not provide.
Parts
- MAP likelihood selector: score visible-pass retrieval-adapt candidates by raw code likelihood under the task prompt and select the highest mean logprob candidate.
- Low-temperature retrieval adaptation: compare semantic adaptation at
T=0.0,T=0.1, and the localT=0.2baseline. - Execution-grounded locality audit: at same-prefix code forks, compare teacher preference for the hidden-correct branch versus the hidden-wrong branch when the teacher is given a failing input, observed wrong output, and correct output.
Gate
Run OPSD training only if execution-grounded hints add task-specific correct-branch preference beyond the no-hint student and shuffled-observation control. Full-reference hints are included only as a leakage ceiling.
Report
Rendered from reports/final_report.md
Date: 2026-06-26
Decision
Do not train the execution-grounded OPSD variant from this audit. The position-matched fork gate failed: execution evidence with the correct output added +0.063 nats over the no-hint student on task-specific forks, while shuffled execution evidence added +0.079 nats. The full-reference leakage ceiling added +0.237 nats, so the audit is capable of seeing signal when the answer is leaked.
The reliability probes also failed to produce a deployable selector: raw MAP likelihood selected fewer hidden-correct candidates than first-visible in both candidate pools, and increased hidden-wrong visible-pass selections.
Question
This no-training experiment tested two cheap hypotheses before any OPSD run:
- If the base model already weakly prefers correct fork tokens, can lower-temperature decoding or raw model likelihood turn that into reliable retrieval-adapt selection?
- If retrieval hints are only surface-level, can an execution-grounded teacher with counterexample input and correct output localize positive pressure onto task-specific discriminating tokens?
Inputs
- Residual retrieval-adapt slice: 24 MBPP held-out tasks.
- Model: Qwen3.5-4B used as generator/scorer.
- Generated new semantic retrieval-adapt pools at T=0.0 and T=0.1, top-3 retrieved algorithms per task.
- Used existing semantic T=0.2, random, shuffled, and copy/rename pools as controls and pair sources.
- No model training was performed.
Low-Temperature Probe
| arm | coverage | pass1 proxy | visible coverage | visible-pass hidden-wrong candidates | functional diversity | forward tokens |
|---|---|---|---|---|---|---|
| T=0.0 | 7/24 (29.2%) | 4/24 (16.7%) | 7/24 (29.2%) | 13/23 | 62.5% | 25,057 |
| T=0.1 | 8/24 (33.3%) | 3/24 (12.5%) | 8/24 (33.3%) | 11/20 | 63.9% | 24,961 |
| T=0.2 | 8/24 (33.3%) | 3/24 (12.5%) | 8/24 (33.3%) | 14/24 | 61.1% | 24,352 |
Lower temperature did not dominate the existing semantic T=0.2 pool. Greedy decoding produced slightly better pass1 proxy but lower pool coverage than T=0.1/T=0.2. T=0.1 matched the best 8/24 coverage but lower pass1 proxy.
MAP Likelihood Selector
MAP scoring used raw average token log-probability of each candidate code under the task prompt, then selected among visible-passing candidates.
Semantic temperature union:
| selector | hidden-correct commits | hidden-wrong commits | commit rate | false-pass rate |
|---|---|---|---|---|
| first_visible | 8 | 6 | 58.3% | 42.9% |
| map_mean | 6 | 8 | 58.3% | 57.1% |
| oracle_hidden | 8 | 0 | 33.3% | 0.0% |
Copy plus semantic T=0.2:
| selector | hidden-correct commits | hidden-wrong commits | commit rate | false-pass rate |
|---|---|---|---|---|
| first_visible | 7 | 7 | 58.3% | 50.0% |
| map_mean | 6 | 8 | 58.3% | 57.1% |
| oracle_hidden | 8 | 0 | 33.3% | 0.0% |
Result: MAP likelihood is not a reliable selector for this near-miss pool. It selected 6/24 hidden-correct candidates in both views, below first-visible's 8/24 and 7/24, and increased visible-pass hidden-wrong selections.
Execution-Grounded OPSD Audit
Matched-pair builder found 59 correct-vs-hidden-wrong adaptation pairs across tasks [35, 44, 87]. It produced 216 position-matched fork rows: 54 task-specific and 162 hint-overlap.
Gate:
- Passed:
False - Reason: execution observation does not add task-specific correct-branch preference beyond student and shuffled control
- Task-specific forks: 54
| context | task-specific preference | delta over student | fraction prefers correct |
|---|---|---|---|
| failing input only | 4.599 | 0.026 | 98.1% |
| failing input + correct output | 4.636 | 0.063 | 98.1% |
| shuffled execution observation | 4.652 | 0.079 | 98.1% |
| full-reference leakage ceiling | 4.811 | 0.237 | 98.1% |
Interpretation: the execution observation moves the model in the right direction a little, but not beyond shuffled execution evidence. The full-reference ceiling moves substantially more, confirming that the audit can detect a teacher that truly contains task-specific information.
Readout
The task-specific fork result repeats the important reliability pattern from the prior audit: the no-hint student already strongly prefers the correct branch at almost every task-specific fork (98.1% under execution-observation rows, same fork set), but the margin is not converted into reliable whole-program assembly. The missing ingredient is not a weak retrieval or execution hint that tells the teacher where the task-specific token is; these hints do not beat shuffled controls at the exact fork.
Next Best Direction
The evidence points away from another token-credit training run and back toward adding independent behavioral evidence before selection. The most promising next experiment is independent-retrieval consensus: retrieve and adapt from several semantically distinct source algorithms, execute them on generated disagreement inputs, and commit only when independently sourced adaptations converge on outputs. That directly attacks the current selection wall with new evidence rather than another learned judge over thin public tests.
Artifacts
- Records:
data/ - Run logs:
run_logs/ - Summaries and figures:
reports/ - Experiment log:
logs/experiment_log.md - Large-artifact policy: large model/checkpoint/cache files are outside this directory; see
large_artifacts_manifest.md.
Experiment log 4
Show the running log (4 entries, 2026-06-26)
2026-06-26
- Created standalone experiment package.
- Copied local residual candidate pools, retrieval plan, verified library, and generic evaluator/model utilities into this package.
- Localized experiment identity to
qwen35_4b_reliability_exec_opsd_audit. - Planned three measurements: MAP likelihood selector, low-temperature semantic adaptation, and execution-grounded locality audit.
2026-06-26T22:13:20Z
- Completed semantic retrieval-adapt top-3 at T=0.0 and T=0.1, using the existing T=0.2 pool as baseline comparison.
- T=0.0: 24 records, coverage 7/24, pass1 proxy 4/24, forward tokens 25,057.
- T=0.1: 24 records, coverage 8/24, pass1 proxy 3/24, forward tokens 24,961.
- Both arms wrote records and manifests under
data/. - Next step: raw likelihood/MAP candidate scoring to test whether the base model's own probability favors hidden-correct visible-pass adaptations.
2026-06-26T22:16:00Z
- Completed raw likelihood/MAP selector scoring.
- Semantic temperature union pool coverage was 8/24. First-visible selected 8/24 hidden-correct with 6 visible-pass hidden-wrong selections. MAP selected 6/24 hidden-correct with 8 hidden-wrong selections.
- Copy+semantic T=0.2 pool coverage was 8/24. First-visible selected 7/24 hidden-correct with 7 hidden-wrong selections. MAP selected 6/24 hidden-correct with 8 hidden-wrong selections.
- Interpretation before final report: raw model likelihood is not a reliable selector for these near-miss retrieval-adapt pools.
2026-06-26T22:30:00Z
- Built 59 matched correct-vs-hidden-wrong adaptation pairs across 3 tasks.
- Pair set contains 216 position-matched fork rows: 54 task-specific forks and 162 hint-overlap forks.
- Scored teacher contexts: no hint/student, execution observation with correct output, failing-input-only observation, shuffled execution observation, and full-reference leakage ceiling.
Stage-1 execution-grounded OPSD gate failed:
- Student task-specific correct-branch preference: 4.573 nats.
- Execution observation: 4.636 nats, +0.063 over student.
- Shuffled execution observation: 4.652 nats, +0.079 over student.
- Full-reference leakage ceiling: 4.811 nats, +0.237 over student.
- Interpretation before final report: execution feedback with expected output is not producing a task-specific teacher signal beyond the shuffled control on this audit set; do not train this OPSD variant.
Figures 4
Data files 4
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
reports/exec_pair_summary.json594 Breports/exec_pressure_summary.json19 kBreports/map_selector_copy_semantic_t0p2_summary.json1.5 kBreports/map_selector_semantic_temps_summary.json1.6 kB
Reproduce
Runnable scripts exist in the experiment folder, but the exact invocation was not written down.