Qwen3.5-4B Real Sample Verify Commit
The one idea you need
Think of an editor choosing the best essay from a stack of drafts a writer produced. On easy prompts a good draft always exists, so any editor succeeds. On hard prompts, if not one draft is actually correct, even a flawless editor cannot pick a winner.
The question
When a small coding model writes several attempts at a problem and a checker picks one to submit, is the weak link the picking or the writing?
What we found
It's the writing. When a 4-billion-parameter model writes several code attempts per problem, choosing is not the bottleneck. On easier problems nearly every attempt is already correct, so the best result sits near 97%. On harder problems at least one attempt is correct only 60% of the time, and the trained checker captures that entire 60% — grabbing the first attempt that passes the visible examples reaches only 55%. No checker can invent programs that were never written.
Why it matters
Building a write-several-then-pick loop for a small model? Spend effort generating more and better attempts, not on a smarter checker. The checker already captures nearly every correct program that exists in the pool.
On this page
Results at a glance 6
How to read
Two groups of bars, one per problem set. Within each: the ceiling of problems where a correct attempt existed, the result of grabbing the first attempt that passed the visible example tests, and the trained checker's choice. Taller means more problems solved.
Takeaway → The trained checker reaches the same height as the ceiling on both sets, losing nothing; the harder set's ceiling itself is simply lower.
Data table
| Dataset | pool coverage ceiling | first visible-passing commit | SFT verifier commit |
|---|---|---|---|
| MBPP eval | 60% | 55% | 60% |
| HumanEval | 96.7% | 96.7% | 96.7% |
Numbers from report table (reports/qwen35_4b_real_sample_verify_commit_report.md, Key Result + Fixed-Budget Summary)
Technical framing
Coverage is the bottleneck, not selection (pass rate at max budget) — The SFT verifier captures the full candidate-pool ceiling on both sets; missing MBPP accuracy comes from no correct sample existing, not bad selection.
How to read
Two lines, the harder set and the easier set. The horizontal axis is how many attempts you write per problem; the vertical axis is the share of problems where at least one attempt is correct. Higher and rising is better.
Takeaway → The easier line starts near the top and barely moves; the harder line climbs from a third of problems to about six in ten, then flattens well below it.
Data table
| Sampling budget (candidates per task; 9 = max) | MBPP eval | HumanEval |
|---|---|---|
| 1 | 35% | 90% |
| 2 | 35% | 90% |
| 4 | 50% | 93.3% |
| 8 | 60% | 96.7% |
| 9 | 60% | 96.7% |
Numbers from report table (Fixed-Budget Summary) / reports/summary_fixed_budget.csv
Technical framing
Candidate-pool coverage vs sampling budget — HumanEval is nearly solved at 1 sample; MBPP coverage climbs from 35% to 60% and plateaus, leaving a 40% generator-side gap.
How to read
Two groups of bars, harder set and easier set. Bars show the average number of attempts written per problem: always writing the maximum, an ideal stop-as-soon-as-solved rule, and a learned stopping rule. Fewer attempts is cheaper and better.
Takeaway → The learned rule writes far fewer attempts than always writing the maximum, especially on the easier set — the same results for a fraction of the work.
Data table
| Dataset | fixed max budget | oracle stop | SFT stop controller |
|---|---|---|---|
| MBPP eval | 7.8 | 5.35 | 4.2 |
| HumanEval | 5.47 | 1.6 | 1.7 |
Numbers from report table (Adaptive-Budget Summary + Fixed-Budget Summary) / reports/summary_adaptive_budget.csv
Technical framing
Adaptive stopping cuts sample cost vs always sampling to max — The learned stop controller matches the 96.7% HumanEval ceiling with 1.7 samples instead of 5.5, trading 5pp on MBPP (55% vs 60%) for fewer samples.
How to read
Each group of bars is one picking rule, from the best-possible pick down to simple habits like taking the first attempt that passed the visible tests or the answer most attempts agree on. Bar height is share of problems solved; taller is better. Shown for both sets.
Takeaway → Nearly every bar sits at the same height within each set — smarter picking rules do not beat simple ones; only a couple dip slightly on the harder set.
Data table
| Selection policy | MBPP eval (20 tasks) | HumanEval (30 tasks) |
|---|---|---|
| oracle_coverage | 60% | 96.7% |
| base_verifier | 60% | 96.7% |
| sft_verifier | 60% | 96.7% |
| shortest_visible | 60% | 96.7% |
| first_visible | 55% | 96.7% |
| public_signature_majority | 55% | 96.7% |
Numbers from experiments/qwen35_4b_real_sample_verify_commit/reports/summary_fixed_budget.csv (budget 9 rows)
Technical framing
Which selection policy commits the right answer (max budget) — Verifier scoring matches the oracle on both sets; only first-visible and signature-majority drop 5pp on MBPP.
How to read
Three stopping rules side by side per set: an ideal stop-as-soon-as-solved rule and two practical learned rules. Bar height is the share of problems still solved after stopping early; taller means less accuracy given up. Shown for both sets.
Takeaway → On the easier set all three match; on the harder set the practical rules sit slightly below the ideal one — a small accuracy cost for stopping early.
Data table
| Stopping policy | MBPP eval | HumanEval |
|---|---|---|
| oracle_stop | 60% | 96.7% |
| sft_stop_controller | 55% | 96.7% |
| threshold_sft_score | 55% | 96.7% |
Numbers from experiments/qwen35_4b_real_sample_verify_commit/reports/summary_adaptive_budget.csv
Technical framing
Adaptive stopping: what accuracy each controller keeps — Learned stopping matches the HumanEval ceiling but gives back 5pp vs oracle stopping on MBPP in exchange for fewer samples.
How to read
Two lines, harder and easier set. The horizontal axis is the number of attempts allowed per problem; the vertical axis is how many were actually written before an attempt passed the visible tests and writing stopped. Lower means it stopped sooner.
Takeaway → Both lines bend below the maximum at higher limits — writing stops early once an attempt passes the visible tests, and the easier set stops sooner than the harder one.
Data table
| Sampling budget (candidates allowed) | MBPP eval | HumanEval |
|---|---|---|
| 1 | 1 | 1 |
| 2 | 2 | 1.97 |
| 4 | 4 | 3.73 |
| 8 | 7.45 | 5.4 |
| 9 | 7.8 | 5.47 |
Numbers from experiments/qwen35_4b_real_sample_verify_commit/reports/summary_fixed_budget.csv (sampled_candidates_mean)
Technical framing
Samples actually drawn vs nominal budget — Even fixed-budget runs stop early on visible-test passes: at budget 8, HumanEval draws only 5.4 samples while harder MBPP draws 7.45.
In the author’s words from the Overview · “Summary”
Run the full sample -> verify -> commit loop using genuine Qwen3.5-4B code samples. The primary measurement decomposes final pass rate into candidate-pool coverage and selector capture.
Overview
This top-level README was generated during repository normalization because the imported experiment did not include one.
- Source track:
track-z - Primary report: reports/qwen35_4b_real_sample_verify_commit_report.md
- Metadata: metadata.yaml
How To Read
Start with the primary report, then inspect data/, reports/, analysis/, src/, and scripts/ as available. This folder remains self-contained; do not move its run data into shared directories.
Summary
Run the full sample -> verify -> commit loop using genuine Qwen3.5-4B code samples. The primary measurement decomposes final pass rate into candidate-pool coverage and selector capture.
Report
Rendered from reports/qwen35_4b_real_sample_verify_commit_report.md
Objective
Run the full sample -> verify -> commit loop using genuine Qwen3.5-4B code samples. The primary measurement decomposes final pass rate into candidate-pool coverage and selector capture.
Candidate Generation
- MBPP train records: 40.
- MBPP eval records: 20.
- HumanEval eval records: 30.
- Direct samples per task: 8.
- Model repair attempts per task: 1.
- Temperatures: 0.2,0.7,1.0.
- Max new tokens: 220.
No mutation-generated candidates are used. Every candidate is either a direct Qwen sample or a Qwen repair sample.
Key Result
- MBPP eval coverage at max budget: 60.0%.
- MBPP eval first-visible commit: 55.0%.
- MBPP eval SFT verifier commit: 60.0%.
- HumanEval coverage at max budget: 96.7%.
- HumanEval first-visible commit: 96.7%.
- HumanEval SFT verifier commit: 96.7%.
- MBPP SFT stop controller: 55.0% using 4.20 samples on average.
- HumanEval SFT stop controller: 96.7% using 1.70 samples on average.
Readout
On these genuine sampled pools, selection is not the main bottleneck. HumanEval is already easy for the generator at this budget: coverage reaches 96.7%, and first-visible, frozen verifier, SFT verifier, and oracle selection all match the ceiling. MBPP is coverage-limited: max-budget coverage is 60.0%, and the verifier can capture that ceiling, but it cannot create missing correct programs. The adaptive controller mostly trades a small amount of MBPP accuracy for fewer samples, while matching the HumanEval ceiling with far fewer samples.
Fixed-Budget Summary
| Dataset | Policy | Budget | Visible coverage | Selected hidden-pass | Coverage captured | Samples seen |
|---|---|---|---|---|---|---|
| humaneval | base_verifier | 1 | 90.0% | 90.0% | 100.0% | 1.00 |
| humaneval | base_verifier | 2 | 90.0% | 90.0% | 100.0% | 1.97 |
| humaneval | base_verifier | 4 | 93.3% | 93.3% | 100.0% | 3.73 |
| humaneval | base_verifier | 8 | 96.7% | 96.7% | 100.0% | 5.40 |
| humaneval | base_verifier | 9 | 96.7% | 96.7% | 100.0% | 5.47 |
| humaneval | first_visible | 1 | 90.0% | 90.0% | 100.0% | 1.00 |
| humaneval | first_visible | 2 | 90.0% | 90.0% | 100.0% | 1.97 |
| humaneval | first_visible | 4 | 93.3% | 93.3% | 100.0% | 3.73 |
| humaneval | first_visible | 8 | 96.7% | 96.7% | 100.0% | 5.40 |
| humaneval | first_visible | 9 | 96.7% | 96.7% | 100.0% | 5.47 |
| humaneval | oracle_coverage | 1 | 90.0% | 90.0% | 100.0% | 1.00 |
| humaneval | oracle_coverage | 2 | 90.0% | 90.0% | 100.0% | 1.97 |
| humaneval | oracle_coverage | 4 | 93.3% | 93.3% | 100.0% | 3.73 |
| humaneval | oracle_coverage | 8 | 96.7% | 96.7% | 100.0% | 5.40 |
| humaneval | oracle_coverage | 9 | 96.7% | 96.7% | 100.0% | 5.47 |
| humaneval | public_signature_majority | 1 | 90.0% | 90.0% | 100.0% | 1.00 |
| humaneval | public_signature_majority | 2 | 90.0% | 90.0% | 100.0% | 1.97 |
| humaneval | public_signature_majority | 4 | 93.3% | 93.3% | 100.0% | 3.73 |
| humaneval | public_signature_majority | 8 | 96.7% | 96.7% | 100.0% | 5.40 |
| humaneval | public_signature_majority | 9 | 96.7% | 96.7% | 100.0% | 5.47 |
| humaneval | sft_verifier | 1 | 90.0% | 90.0% | 100.0% | 1.00 |
| humaneval | sft_verifier | 2 | 90.0% | 90.0% | 100.0% | 1.97 |
| humaneval | sft_verifier | 4 | 93.3% | 93.3% | 100.0% | 3.73 |
| humaneval | sft_verifier | 8 | 96.7% | 96.7% | 100.0% | 5.40 |
| humaneval | sft_verifier | 9 | 96.7% | 96.7% | 100.0% | 5.47 |
| humaneval | shortest_visible | 1 | 90.0% | 90.0% | 100.0% | 1.00 |
| humaneval | shortest_visible | 2 | 90.0% | 90.0% | 100.0% | 1.97 |
| humaneval | shortest_visible | 4 | 93.3% | 93.3% | 100.0% | 3.73 |
| humaneval | shortest_visible | 8 | 96.7% | 96.7% | 100.0% | 5.40 |
| humaneval | shortest_visible | 9 | 96.7% | 96.7% | 100.0% | 5.47 |
| mbpp | base_verifier | 1 | 35.0% | 35.0% | 100.0% | 1.00 |
| mbpp | base_verifier | 2 | 35.0% | 35.0% | 100.0% | 2.00 |
| mbpp | base_verifier | 4 | 50.0% | 50.0% | 100.0% | 4.00 |
| mbpp | base_verifier | 8 | 60.0% | 60.0% | 100.0% | 7.45 |
| mbpp | base_verifier | 9 | 60.0% | 60.0% | 100.0% | 7.80 |
| mbpp | first_visible | 1 | 35.0% | 35.0% | 100.0% | 1.00 |
| mbpp | first_visible | 2 | 35.0% | 35.0% | 100.0% | 2.00 |
| mbpp | first_visible | 4 | 50.0% | 50.0% | 100.0% | 4.00 |
| mbpp | first_visible | 8 | 60.0% | 55.0% | 91.7% | 7.45 |
| mbpp | first_visible | 9 | 60.0% | 55.0% | 91.7% | 7.80 |
| mbpp | oracle_coverage | 1 | 35.0% | 35.0% | 100.0% | 1.00 |
| mbpp | oracle_coverage | 2 | 35.0% | 35.0% | 100.0% | 2.00 |
| mbpp | oracle_coverage | 4 | 50.0% | 50.0% | 100.0% | 4.00 |
| mbpp | oracle_coverage | 8 | 60.0% | 60.0% | 100.0% | 7.45 |
| mbpp | oracle_coverage | 9 | 60.0% | 60.0% | 100.0% | 7.80 |
| mbpp | public_signature_majority | 1 | 35.0% | 35.0% | 100.0% | 1.00 |
| mbpp | public_signature_majority | 2 | 35.0% | 35.0% | 100.0% | 2.00 |
| mbpp | public_signature_majority | 4 | 50.0% | 50.0% | 100.0% | 4.00 |
| mbpp | public_signature_majority | 8 | 60.0% | 55.0% | 91.7% | 7.45 |
| mbpp | public_signature_majority | 9 | 60.0% | 55.0% | 91.7% | 7.80 |
| mbpp | sft_verifier | 1 | 35.0% | 35.0% | 100.0% | 1.00 |
| mbpp | sft_verifier | 2 | 35.0% | 35.0% | 100.0% | 2.00 |
| mbpp | sft_verifier | 4 | 50.0% | 50.0% | 100.0% | 4.00 |
| mbpp | sft_verifier | 8 | 60.0% | 60.0% | 100.0% | 7.45 |
| mbpp | sft_verifier | 9 | 60.0% | 60.0% | 100.0% | 7.80 |
| mbpp | shortest_visible | 1 | 35.0% | 35.0% | 100.0% | 1.00 |
| mbpp | shortest_visible | 2 | 35.0% | 35.0% | 100.0% | 2.00 |
| mbpp | shortest_visible | 4 | 50.0% | 50.0% | 100.0% | 4.00 |
| mbpp | shortest_visible | 8 | 60.0% | 60.0% | 100.0% | 7.45 |
| mbpp | shortest_visible | 9 | 60.0% | 60.0% | 100.0% | 7.80 |
Adaptive-Budget Summary
| Dataset | Policy | Visible coverage | Selected hidden-pass | Coverage captured | Mean samples used |
|---|---|---|---|---|---|
| humaneval | oracle_stop | 96.7% | 96.7% | 100.0% | 1.60 |
| humaneval | sft_stop_controller | 96.7% | 96.7% | 100.0% | 1.70 |
| humaneval | threshold_sft_score | 96.7% | 96.7% | 100.0% | 1.60 |
| mbpp | oracle_stop | 60.0% | 60.0% | 100.0% | 5.35 |
| mbpp | sft_stop_controller | 55.0% | 55.0% | 100.0% | 4.20 |
| mbpp | threshold_sft_score | 55.0% | 55.0% | 100.0% | 4.20 |
Interpretation
This experiment uses genuinely sampled candidate pools, so the central fork is visible directly: under this sampling setup, the verifier is not failing on subtle visible-passing near-misses at this scale. The dominant open problem is generating a correct candidate on harder MBPP tasks. For HumanEval, the first visible-passing sample is usually already correct, so verifier posttraining and adaptive control add little accuracy headroom, though adaptive stopping reduces sample cost.
The main limitation is scale: this run uses 20 MBPP eval tasks and 30 HumanEval tasks because genuine Qwen sampling is the expensive step. The conclusion should be read as a measured pilot of the real candidate distribution, not as a final benchmark score.
The next high-leverage step is generator-side: increase coverage through better sampling, repair, or verifier-guided self-improvement. Verifier work should focus on larger, more adversarial sampled pools, because this run did not surface a meaningful selector wall.
Figures
reports/figures/mbpp_coverage_curve.pngreports/figures/mbpp_fixed_budget_accuracy.pngreports/figures/mbpp_adaptive_budget.pngreports/figures/humaneval_coverage_curve.pngreports/figures/humaneval_fixed_budget_accuracy.pngreports/figures/humaneval_adaptive_budget.png
Reproduction
python scripts/sample_candidates.py --mbpp-train 40 --mbpp-eval 20 --humaneval-eval 30 --samples-per-task 8 --repair-per-task 1 --max-new-tokens 220 --generation-batch-size 4 --temperatures 0.2,0.7,1.0 --top-p 0.95
python scripts/build_verifier_examples.py
python scripts/train_action_sft.py --train data/train_verifier_examples.jsonl --output-dir /workspace/large_artifacts/qwen35_4b_real_sample_verify_commit/models/verifier_sft_lora --loss-out reports/verifier_sft_training_losses.json --method sampled_semantic_verifier_sft --max-steps 160 --batch-size 2 --grad-accum 2
python scripts/eval_commit.py --records data/mbpp_train_records.jsonl --policy sft_verifier --name sft_verifier --out reports/eval/mbpp_train_sft_verifier.json --budgets 1,2,4,8,max --adapter-dir /workspace/large_artifacts/qwen35_4b_real_sample_verify_commit/models/verifier_sft_lora
python scripts/build_stop_examples.py --scores reports/eval/mbpp_train_sft_verifier.json --budgets 1,2,4,8,max
python scripts/train_action_sft.py --train data/train_stop_examples.jsonl --output-dir /workspace/large_artifacts/qwen35_4b_real_sample_verify_commit/models/stop_sft_lora --loss-out reports/stop_sft_training_losses.json --method sampled_generation_budget_stop_sft --max-steps 120 --batch-size 2 --grad-accum 2
bash scripts/run_evaluation_suite.sh
python scripts/make_report.pyExperiment log 12
Show the running log (12 entries)
Date: 2026-06-25
Experiment directory: /workspace/experiments/qwen35_4b_real_sample_verify_commit
Large artifacts directory: /workspace/large_artifacts/qwen35_4b_real_sample_verify_commit
Objective
Run the sample -> verify -> commit loop on genuinely sampled Qwen3.5-4B code completions, not mutation-derived candidate pools. The goal is to measure real candidate-pool coverage and selector capture without assuming that a correct candidate is present or that incorrect candidates are easy to reject.
The primary measurement is:
final pass@1 = candidate coverage ceiling x selector capture
Where:
candidate coverage ceilingmeans at least one sampled candidate passes hidden tests.selector capturemeans the selected candidate is hidden-correct when the pool contains a hidden-correct candidate.- Hidden tests are used only for evaluation labels, verifier training labels, and oracle diagnostic arms. Selection inputs at eval time remain leak-free.
Hypothesis
The candidate distribution itself can dominate the result. If genuine Qwen sampling creates realistic subtle-bug near-misses, selection may collapse even when coverage is high. If coverage is low, the wall is the generator rather than the verifier.
This run is designed to distinguish three outcomes:
- High coverage and high selector capture: sample + verify + commit is deployable.
- High coverage and poor selector capture: verifier/selection is the wall.
- Low coverage: generation is the wall.
Model And Data
Base model: /workspace/.cache/huggingface/models--Qwen--Qwen3.5-4B/snapshots/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a
Datasets:
- MBPP train subset: 40 records.
- MBPP eval subset: 20 records.
- HumanEval eval subset: 30 records.
Candidate generation:
- Direct samples per task: 8.
- Repair attempts per task: 1.
- Temperatures:
0.2,0.7,1.0. - Top-p:
0.95. - Max new tokens:
220. - Seed:
20260625. - HumanEval public tests used during candidate filtering: up to 3.
- No mutation-generated candidates.
Candidate pool manifest:
| Split | Records | Mean candidates | Mean parsed/safe | Mean visible-passers | Coverage | Visible coverage |
|---|---|---|---|---|---|---|
| MBPP train | 40 | 6.925 | 6.675 | 4.075 | 77.5% | 77.5% |
| MBPP eval | 20 | 7.800 | 7.500 | 3.200 | 60.0% | 60.0% |
| HumanEval eval | 30 | 5.467 | 5.333 | 4.433 | 96.7% | 96.7% |
Implementation Notes
The package is intentionally split so the downloadable experiment directory does not contain large checkpoints or adapters:
- Code, reports, CSVs, logs, and JSONL data are under the experiment directory.
- LoRA adapters are under
large_artifacts/qwen35_4b_real_sample_verify_commit/models.
Core scripts:
scripts/sample_candidates.py: genuine Qwen direct sampling plus Qwen repair sampling.scripts/build_verifier_examples.py: verifier SFT data from parsed/safe public-failing negatives plus visible-passing positives/negatives.scripts/train_action_sft.py: shared LoRA trainer for semantic verifier and stop controller.scripts/eval_commit.py: fixed-budget commit policies.scripts/build_stop_examples.py: adaptive generation-budget STOP/MORE training examples.scripts/tune_threshold.py: threshold baseline tuning on MBPP train.scripts/eval_adaptive_budget.py: adaptive generation-budget policies.scripts/make_report.py: standalone markdown report, CSV summaries, and figures.scripts/run_evaluation_suite.sh: reruns fixed-budget, threshold, and adaptive evaluation after adapters exist.
Debugging And Fixes
This run intentionally used the real model and real Python execution, so several harness issues surfaced and were fixed:
- MBPP prompt construction initially used the wrong field name (
textinstead of normalizedtask_text). Fixed prompt builders to use normalized task text. - The Qwen model implementation rejected
generator=ingenerate. Fixed sampling to settorch.manual_seedper completion instead. - Qwen initially emitted thinking text rather than code-only completions. Fixed prompts to use the tokenizer chat template with
enable_thinking=Falsewhere supported. - The static safety checker rejected safe typing imports such as
from typing import List. Fixed import checking to validate the module name, not each imported alias as a module. - HumanEval prompts with only a function signature and docstring were being accepted as valid because the entry function existed. Fixed validation to require a non-docstring function body.
- Verifier training data initially had too few negatives because genuine visible-passers were often correct. Fixed
build_verifier_examples.pyto include parsed/safe public-failing candidates as negative examples by default, while keeping--visible-onlyavailable. - Report generation initially mixed MBPP train scoring files into fixed-budget summaries. Fixed
make_report.pyto exclude_train_evaluation files.
Training
Verifier SFT:
python scripts/train_action_sft.py \
--train data/train_verifier_examples.jsonl \
--output-dir /workspace/large_artifacts/qwen35_4b_real_sample_verify_commit/models/verifier_sft_lora \
--loss-out reports/verifier_sft_training_losses.json \
--method sampled_semantic_verifier_sft \
--max-steps 160 \
--batch-size 2 \
--grad-accum 2Verifier training examples:
- Total: 267.
- Positive: 148.
- Negative: 119.
- Includes parsed/safe public-failing negatives.
Stop controller SFT:
python scripts/train_action_sft.py \
--train data/train_stop_examples.jsonl \
--output-dir /workspace/large_artifacts/qwen35_4b_real_sample_verify_commit/models/stop_sft_lora \
--loss-out reports/stop_sft_training_losses.json \
--method sampled_generation_budget_stop_sft \
--max-steps 120 \
--batch-size 2 \
--grad-accum 2Stop controller training examples:
- Total: 200.
- STOP: 153.
- MORE: 47.
The threshold baseline was tuned on MBPP train verifier scores. Best threshold: -19.875, with MBPP-train selected hidden-pass 77.5% at 2.75 samples average.
Fixed-Budget Results
Budgets are candidate-prefix budgets. max means all available generated candidates for that record.
| Dataset | Policy | Max-budget coverage | Max-budget selected hidden-pass | Coverage captured |
|---|---|---|---|---|
| MBPP eval | oracle coverage | 60.0% | 60.0% | 100.0% |
| MBPP eval | first visible | 60.0% | 55.0% | 91.7% |
| MBPP eval | public-signature majority | 60.0% | 55.0% | 91.7% |
| MBPP eval | shortest visible | 60.0% | 60.0% | 100.0% |
| MBPP eval | base verifier | 60.0% | 60.0% | 100.0% |
| MBPP eval | SFT verifier | 60.0% | 60.0% | 100.0% |
| HumanEval eval | oracle coverage | 96.7% | 96.7% | 100.0% |
| HumanEval eval | first visible | 96.7% | 96.7% | 100.0% |
| HumanEval eval | public-signature majority | 96.7% | 96.7% | 100.0% |
| HumanEval eval | shortest visible | 96.7% | 96.7% | 100.0% |
| HumanEval eval | base verifier | 96.7% | 96.7% | 100.0% |
| HumanEval eval | SFT verifier | 96.7% | 96.7% | 100.0% |
Coverage curves:
| Dataset | Budget 1 | Budget 2 | Budget 4 | Budget 8 | Max |
|---|---|---|---|---|---|
| MBPP eval | 35.0% | 35.0% | 50.0% | 60.0% | 60.0% |
| HumanEval eval | 90.0% | 90.0% | 93.3% | 96.7% | 96.7% |
Adaptive Generation-Budget Results
The adaptive resource here is number of generated candidates inspected before committing, matching the real-code cost structure more closely than probe/test count.
| Dataset | Policy | Visible coverage | Selected hidden-pass | Coverage captured | Mean samples used |
|---|---|---|---|---|---|
| MBPP eval | oracle stop | 60.0% | 60.0% | 100.0% | 5.35 |
| MBPP eval | threshold on SFT verifier score | 55.0% | 55.0% | 100.0% | 4.20 |
| MBPP eval | SFT stop controller | 55.0% | 55.0% | 100.0% | 4.20 |
| HumanEval eval | oracle stop | 96.7% | 96.7% | 100.0% | 1.60 |
| HumanEval eval | threshold on SFT verifier score | 96.7% | 96.7% | 100.0% | 1.60 |
| HumanEval eval | SFT stop controller | 96.7% | 96.7% | 100.0% | 1.70 |
Interpretation
This pilot uses genuine sampled pools and the outcome is clear: on these pools, selection is not the main bottleneck.
For HumanEval, Qwen sampling already creates a correct visible-passing candidate on 96.7% of the sampled tasks, and the first visible-passing candidate is already hidden-correct whenever coverage exists. That leaves almost no selection headroom for either the base verifier or the SFT verifier.
For MBPP, max-budget coverage is only 60.0%. The verifier can capture that ceiling, but it cannot create candidates that sampling missed. This points at generation/repair coverage as the current wall, not verifier ranking.
The adaptive generation-budget result is useful but modest. On HumanEval, the controller matches the 96.7% ceiling after inspecting only about 1.7 samples on average. On MBPP, the SFT stop controller and tuned threshold trade 5 points of accuracy for fewer inspected samples relative to oracle-stop/max-budget selection.
The important fork is therefore:
- Not enough evidence for a selector wall in this sampled-pool pilot.
- Strong evidence that MBPP needs better candidate generation or repair.
- HumanEval at this small scale is too forgiving to stress the verifier.
Limitations
- Scale is small because genuine Qwen3.5-4B sampling is expensive on the available hardware: 20 MBPP eval tasks and 30 HumanEval eval tasks.
- HumanEval appears too easy under this sampling setup because public-visible passers are usually hidden-correct.
- MBPP coverage is the decisive bottleneck, but this experiment does not yet improve the generator.
- Verifier SFT data is small and partly trained with public-failing negatives; this is appropriate for a pilot but not a final verifier benchmark.
- The code execution harness uses subprocess timeouts and static checks, not a full container sandbox.
Decision
The next high-impact experiment should move upstream to coverage: verifier-guided self-improvement or repair generation on genuine sampled failures, still reported as coverage x selector capture. A larger verifier-only run is lower priority unless the candidate generator can produce harder realistic near-miss pools where coverage is high and first-visible selection fails.
The concrete next step I would run:
- Generate larger genuine candidate pools on MBPP train/eval.
- Use the verifier/executor to identify public-failing and hidden-failing candidates.
- Train a repair generator or rejection-sampling distillation loop on verifier-selected candidates.
- Re-run this exact coverage x selection report to see whether MBPP coverage rises above 60% without reducing selector capture.
Reproduction Commands
python scripts/sample_candidates.py --mbpp-train 40 --mbpp-eval 20 --humaneval-eval 30 --samples-per-task 8 --repair-per-task 1 --max-new-tokens 220 --generation-batch-size 4 --temperatures 0.2,0.7,1.0 --top-p 0.95
python scripts/build_verifier_examples.py
python scripts/train_action_sft.py --train data/train_verifier_examples.jsonl --output-dir /workspace/large_artifacts/qwen35_4b_real_sample_verify_commit/models/verifier_sft_lora --loss-out reports/verifier_sft_training_losses.json --method sampled_semantic_verifier_sft --max-steps 160 --batch-size 2 --grad-accum 2
python scripts/eval_commit.py --records data/mbpp_train_records.jsonl --policy sft_verifier --name sft_verifier --out reports/eval/mbpp_train_sft_verifier.json --budgets 1,2,4,8,max --adapter-dir /workspace/large_artifacts/qwen35_4b_real_sample_verify_commit/models/verifier_sft_lora
python scripts/build_stop_examples.py --scores reports/eval/mbpp_train_sft_verifier.json --budgets 1,2,4,8,max
python scripts/train_action_sft.py --train data/train_stop_examples.jsonl --output-dir /workspace/large_artifacts/qwen35_4b_real_sample_verify_commit/models/stop_sft_lora --loss-out reports/stop_sft_training_losses.json --method sampled_generation_budget_stop_sft --max-steps 120 --batch-size 2 --grad-accum 2
bash scripts/run_evaluation_suite.sh
python scripts/make_report.pyFigures 6
Data files 12
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
reports/summary_adaptive_budget.csv633 Breports/summary_fixed_budget.csv4.7 kBreports/eval/humaneval_base_verifier.json97 kBreports/eval/humaneval_first_visible.json65 kBreports/eval/humaneval_oracle_coverage.json66 kBreports/eval/humaneval_oracle_stop.json12 kBreports/eval/humaneval_public_signature_majority.json67 kBreports/eval/humaneval_sft_stop_controller.json13 kBreports/eval/humaneval_sft_verifier.json97 kBreports/eval/humaneval_shortest_visible.json66 kBreports/eval/humaneval_threshold_sft_score.json13 kBreports/eval/mbpp_threshold_sft_score.json7.9 kB
Reproduce
Run steps are documented inside the experiment folder (README and scripts).