Qwen3.5-4B HumanEval Adaptive Evidence Budget
The one idea you need
Picture choosing an answer by popularity vote: the system writes several candidate programs, feeds them the same sample inputs, groups programs whose outputs match, and commits to the biggest group. But agreement is not correctness, and there is no answer key. The model only decides whether to run one more test.
The question
If a small model could smartly decide when to stop running test inputs on candidate programs, would it pick correct code more often while wasting less effort?
What we found
No. Every strategy landed at the same 16.7% correct-pick rate — running no tests, running all eight, and even a strategy allowed to peek at the right answer to time its stop. The trained model did learn to quit sooner, using under four tests instead of eight, but its choice stayed wrong: grouping programs by matching outputs never reveals which group is actually correct, and a correct program was available for every task.
Why it matters
When candidate code is chosen by which programs agree on unlabeled test inputs, spend effort on grounding the choice — a trustworthy answer key or a stronger selector — not on tuning when to stop. More tests cannot fix a blind chooser.
On this page
Results at a glance 2
How to read
Grouped bars across five stopping strategies. In each pair, the full-height bar shows a correct program existed among the candidates (always 100%); the shorter bar shows how often the chosen program was actually correct. Higher is better; the short bars never rise.
Takeaway → Every strategy's correct-pick bar sits at the same low height while availability stays full — the chooser almost never lands on the correct program that was there all along.
Data table
| stopping policy | hidden-correct selected | candidate-pool coverage |
|---|---|---|
| fixed budget 0 | 16.7% | 100% |
| fixed budget 8 | 16.7% | 100% |
| oracle stop | 16.7% | 100% |
| base stop/more | 16.7% | 100% |
| SFT stop/more | 16.7% | 100% |
Numbers from reports/summary_overall.csv
Technical framing
Every stopping policy selects the hidden-correct candidate at the same 16.7% rate — A correct candidate exists for every task (100% coverage), yet no probe-spending policy lifts selection above 16.7% — a clean negative.
How to read
One bar per strategy showing the average number of test inputs run before committing; shorter means cheaper. The trained model's bar sits well below the run-all-eight bar, but correct-pick rate from the other chart is flat across all of them.
Takeaway → The trained model learned to stop early and spend fewer test runs, yet since accuracy never changes, that saving buys nothing useful.
Data table
| stopping policy | avg probes used |
|---|---|
| fixed budget 0 | 0 |
| fixed budget 3 | 3 |
| fixed budget 8 | 8 |
| oracle stop | 6.67 |
| base stop/more | 0.333 |
| SFT stop/more | 3.83 |
Numbers from reports/summary_overall.csv
Technical framing
Executable probes spent per task by stopping policy — The SFT controller learned to stop cheaper than fixed budget 8 (3.83 vs 8 probes), but since accuracy is flat the saving buys nothing.
In the author’s words from the Overview · “Summary”
This standalone experiment tests whether Qwen3.5-4B can be posttrained as a STOP/MORE controller for an executable Python verifier on HumanEval tasks. The verifier generates candidate implementations, chooses unlabeled probes by target-independent output-agreement split, and commits the first candidate in the largest output-agreement cluster. The model only decides whether to commit or spend one more executable probe.
Overview
This top-level README was generated during repository normalization because the imported experiment did not include one.
- Source track:
track-z - Primary report: reports/qwen35_4b_humaneval_adaptive_budget_report.md
- Metadata: metadata.yaml
How To Read
Start with the primary report, then inspect data/, reports/, analysis/, src/, and scripts/ as available. This folder remains self-contained; do not move its run data into shared directories.
Summary
This standalone experiment tests whether Qwen3.5-4B can be posttrained as a STOP/MORE controller for an executable Python verifier on HumanEval tasks. The verifier generates candidate implementations, chooses unlabeled probes by target-independent output-agreement split, and commits the first candidate in the largest output-agreement cluster. The model only decides whether to commit or spend one more executable probe.
Report
Rendered from reports/qwen35_4b_humaneval_adaptive_budget_report.md
Objective
This standalone experiment tests whether Qwen3.5-4B can be posttrained as a STOP/MORE controller for an executable Python verifier on HumanEval tasks. The verifier generates candidate implementations, chooses unlabeled probes by target-independent output-agreement split, and commits the first candidate in the largest output-agreement cluster. The model only decides whether to commit or spend one more executable probe.
Dataset
- Source:
openai/openai_humaneval. - Train tasks: 24; eval tasks: 12.
- Visible tests per task: 1; probe pool: 8; generated hidden tests: 0.
- Candidate implementations per task: 16 maximum, from canonical-solution mutations and generic fallback bodies.
- Public doctest examples are the only labeled visible tests. Generated probes are unlabeled and are used only to form candidate agreement clusters. Reference outputs for generated probes are stored only as audit metadata.
- STOP/MORE train states: 216; eval states: 108.
Key Result
- SFT STOP/MORE reached 16.7% hidden-correct selection with 3.83 probes on average.
- Fixed budget 8 reached 16.7% with 8.00 probes.
- Oracle stopping reached 16.7% with 6.67 probes.
- Base Qwen STOP/MORE reached 16.7%.
- Candidate-pool coverage was 100.0%: a hidden-correct candidate was present for every eval task, but the leak-free agreement selector usually did not choose it.
Overall
| Policy | Hidden-correct selected | Candidate-pool coverage | Avg probes | Agreement clusters | Selected-cluster share | Hidden-correct survivors |
|---|---|---|---|---|---|---|
| Fixed budget 0 | 16.7% | 100.0% | 0.00 | 1.00 | 100.0% | 1.33 |
| Fixed budget 3 | 16.7% | 100.0% | 3.00 | 2.92 | 56.6% | 1.33 |
| Fixed budget 6 | 16.7% | 100.0% | 6.00 | 2.92 | 56.6% | 1.33 |
| Fixed budget 8 | 16.7% | 100.0% | 8.00 | 2.92 | 56.6% | 1.33 |
| Stop if cluster >=70% | 16.7% | 100.0% | 0.00 | 1.00 | 100.0% | 1.33 |
| Stop if cluster >=90% | 16.7% | 100.0% | 0.00 | 1.00 | 100.0% | 1.33 |
| Oracle stop | 16.7% | 100.0% | 6.67 | 2.92 | 56.6% | 1.33 |
| Base Qwen stop/more | 16.7% | 100.0% | 0.33 | 1.75 | 84.1% | 1.33 |
| SFT Qwen stop/more | 16.7% | 100.0% | 3.83 | 2.08 | 69.8% | 1.33 |
Interpretation
This pilot is a clean negative for STOP/MORE budget control under this leak-free HumanEval evidence model. The candidate pool contains a hidden-correct implementation for every eval task, so the low final accuracy is not a coverage failure. The failure is that unlabeled generated probes split the visible-passing candidates into agreement clusters without grounding which cluster is correct. Spending more probes changes cluster structure but does not move the selected candidate onto the hidden-correct implementation, so even a hidden-aware oracle stopping rule has no accuracy to recover.
The SFT controller learned a cheaper stopping behavior than fixed budget 8, but because the selected candidate remained wrong on 11 of 12 eval tasks, the cost saving is not useful. For this benchmark shape, the next useful lever is not a better STOP/MORE controller over unlabeled tests; it is either a stronger deployable candidate selector, a trustworthy labeled-test generator, or an adaptive generation budget that samples more candidate programs when the visible-passing pool is poorly grounded.
This is a pilot-scale result. HumanEval public examples and randomly generated in-domain probes limited the usable split to 24 train and 12 eval tasks. The conclusion should be read as a substrate diagnostic, not as a benchmark-level pass-rate estimate.
Figures
reports/figures/accuracy_vs_probes.pngreports/figures/accuracy_by_policy.pngreports/figures/probes_by_policy.pngreports/figures/budget_sft_loss.png
Reproduction
python scripts/build_dataset.py --train-tasks 24 --eval-tasks 12 --visible-tests 1 --probe-tests 8 --hidden-tests 0 --candidate-count 16 --max-budget 8
python scripts/eval_budget_policy.py --policy fixed --name fixed_budget0 --fixed-budget 0 --max-budget 8
python scripts/eval_budget_policy.py --policy fixed --name fixed_budget3 --fixed-budget 3 --max-budget 8
python scripts/eval_budget_policy.py --policy fixed --name fixed_budget6 --fixed-budget 6 --max-budget 8
python scripts/eval_budget_policy.py --policy fixed --name fixed_budget8 --fixed-budget 8 --max-budget 8
python scripts/eval_budget_policy.py --policy threshold --name threshold_70 --threshold 70 --max-budget 8
python scripts/eval_budget_policy.py --policy threshold --name threshold_90 --threshold 90 --max-budget 8
python scripts/eval_budget_policy.py --policy oracle_stop --name oracle_stop --max-budget 8
python scripts/eval_budget_policy.py --policy base --name base_budget_policy --max-budget 8
python scripts/train_budget_sft.py --max-steps 160 --batch-size 2 --grad-accum 2
python scripts/eval_budget_policy.py --policy adapter --name sft_budget_policy --adapter-dir /workspace/large_artifacts/qwen35_4b_humaneval_adaptive_budget/models/budget_sft_lora --max-budget 8
python scripts/make_report.pyExperiment log 6
Show the running log (6 entries)
Objective
Run a standalone external-validity pilot for adaptive evidence budgeting on real executable Python tasks. The model's role is limited to deciding STOP versus MORE. The verifier owns candidate execution, probe ranking, and deterministic commit selection.
Design Decisions
- Dataset:
openai/openai_humaneval. - Visible evidence: public doctest examples parsed from the task prompt.
- Probe evidence: generated inputs are unlabeled at inference time. They are used only to cluster candidate implementations by output agreement.
- Hidden evaluation: official HumanEval
check(candidate)is reserved for evaluation and for constructing supervised STOP/MORE labels. - Candidate pool: up to 16 implementations per task from canonical-solution mutations plus generic fallback bodies. The canonical solution is included to measure whether selection can recover a correct candidate when coverage exists.
- Commit rule: choose the first candidate in the largest output-agreement cluster among candidates passing public visible examples.
- Probe rule: greedily choose the unused probe that minimizes expected remaining agreement-cluster size.
- Model action space:
A = STOP,B = MORE. - Large artifact placement: the QLoRA adapter and tokenizer are stored under
/workspace/large_artifacts/qwen35_4b_humaneval_adaptive_budget/models/budget_sft_lora.
Iteration Notes
- Initial smoke build with 3 train and 2 eval tasks succeeded, validating the executor, timeout path, public doctest parser, state builder, and cheap policy evaluators.
- A stricter full build with 2 public examples and 32 generated probes plus 32 generated hidden checks accepted too few tasks. The filter was too selective for HumanEval.
- A second full build with 1 public example and 16 probes plus 16 generated hidden checks still accepted too few tasks.
- The final pilot uses 1 public visible example, 8 generated unlabeled probes, 0 generated hidden checks, and official HumanEval checks as the hidden target. This yielded 24 train tasks and 12 eval tasks from the first 53 raw tasks scanned.
- Audit found a public-doctest parser bug: extracting examples from the full function prompt sometimes included the closing triple quote in the expected output. The parser was corrected to extract the function docstring first and skip malformed doctest blocks.
- The corrected dataset was rebuilt from scratch with the same pilot settings: 24 train tasks, 12 eval tasks, 8 unlabeled probes, and one public visible test.
- Corrected non-model baselines showed 100.0% candidate-pool coverage on eval but only 16.7% hidden-correct selected accuracy across all fixed budgets. This diagnosed a selection/grounding failure before model training.
- Base Qwen STOP/MORE evaluation reached the same 16.7% selected accuracy with 0.33 probes on average.
- QLoRA SFT was retrained for 160 optimizer steps on 216 corrected train states. The loss frequently fell near zero but had intermittent spikes, consistent with the small heterogeneous pilot split.
- SFT evaluation reached the same 16.7% selected accuracy with 3.83 probes on average.
Result Summary
| Policy | Hidden-correct selected | Candidate-pool coverage | Avg probes |
|---|---|---|---|
| Fixed budget 0 | 16.7% | 100.0% | 0.00 |
| Fixed budget 8 | 16.7% | 100.0% | 8.00 |
| Oracle stop | 16.7% | 100.0% | 6.67 |
| Base Qwen stop/more | 16.7% | 100.0% | 0.33 |
| SFT Qwen stop/more | 16.7% | 100.0% | 3.83 |
Read
This pilot is a clean negative for STOP/MORE budget control under the leak-free agreement-only evidence model. The hidden-correct implementation is always in the eval candidate pool, but unlabeled probes do not identify which output-agreement cluster is correct. More probes split candidates but do not move deterministic selection onto the correct candidate, leaving no useful headroom for a stopping controller.
Reproduction
python scripts/build_dataset.py --train-tasks 24 --eval-tasks 12 --visible-tests 1 --probe-tests 8 --hidden-tests 0 --candidate-count 16 --max-budget 8
python scripts/eval_budget_policy.py --policy fixed --name fixed_budget0 --fixed-budget 0 --max-budget 8
python scripts/eval_budget_policy.py --policy fixed --name fixed_budget3 --fixed-budget 3 --max-budget 8
python scripts/eval_budget_policy.py --policy fixed --name fixed_budget6 --fixed-budget 6 --max-budget 8
python scripts/eval_budget_policy.py --policy fixed --name fixed_budget8 --fixed-budget 8 --max-budget 8
python scripts/eval_budget_policy.py --policy threshold --name threshold_70 --threshold 70 --max-budget 8
python scripts/eval_budget_policy.py --policy threshold --name threshold_90 --threshold 90 --max-budget 8
python scripts/eval_budget_policy.py --policy oracle_stop --name oracle_stop --max-budget 8
python scripts/eval_budget_policy.py --policy base --name base_budget_policy --max-budget 8
python scripts/train_budget_sft.py --max-steps 160 --batch-size 2 --grad-accum 2
python scripts/eval_budget_policy.py --policy adapter --name sft_budget_policy --adapter-dir /workspace/large_artifacts/qwen35_4b_humaneval_adaptive_budget/models/budget_sft_lora --max-budget 8
python scripts/make_report.pyFigures 4
Data files 2
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
Reproduce
Run steps are documented inside the experiment folder (README and scripts).