Qwen3.5-4B Oracle-Distilled Semantic Verifier
The one idea you need
Several draft programs all pass the one test you can run, but the real grading tests stay hidden. A small model plays judge, betting which draft secretly passes. It learns by studying past programs already labeled by whether they cleared those hidden tests.
The question
When several candidate programs all pass the one test you can see, can a small model learn to pick the one that secretly passes the hidden tests?
What we found
Yes, but only on home turf. On the problem set it trained on, the trained judge picked a genuinely-correct program 81% of the time, up from 72% for the same model untrained and just 44% for grabbing the first candidate that passes the visible test. On a fresh, unfamiliar problem set, though, the untrained model judged better, 94% versus 90% — training sharpened its home instinct while dulling its general one.
Why it matters
If you fine-tune a small model to judge which code is correct, expect gains only on problems resembling your training set. An off-the-shelf model may already be the stronger judge elsewhere, so train on broad, varied code.
On this page
Results at a glance 2
How to read
Bars show how often the chosen program also passes the hidden tests (taller is better) across six picking methods, from naive rules to the two model judges to a perfect picker. Two colors compare the familiar problem set against a fresh one.
Takeaway → The trained judge's bar is tallest on the familiar set but slightly shorter than the untrained judge on the fresh set; every judge towers over naive rules.
Data table
| selection policy | MBPP validation | HumanEval transfer |
|---|---|---|
| shortest visible-pass | 25% | 32.3% |
| random visible-pass | 40.6% | 64.5% |
| first visible-pass | 43.8% | 77.4% |
| frozen base verifier | 71.9% | 93.5% |
| SFT verifier | 81.2% | 90.3% |
| oracle ceiling | 100% | 100% |
Numbers from reports/summary_overall.csv
Technical framing
Which candidate-selection policy picks hidden-test-passing programs? — SFT beats the frozen verifier in-domain on MBPP (+9.4pp) but trails it by 3.2pp on HumanEval transfer; both crush naive selection.
How to read
For each problem set, a tall bar counts programs per task that pass the visible test and a shorter bar counts how many of those also pass the hidden tests. A bigger gap means picking the right one is harder.
Takeaway → On the familiar set barely half the visible-passing programs survive the hidden tests, versus about three-quarters on the fresh set, so choosing well matters more there.
Data table
| Eval set | Visible-test-passing candidates | Also pass hidden tests |
|---|---|---|
| HumanEval (transfer) | 11.1 | 8.16 |
| MBPP validation | 5.66 | 3.12 |
Numbers from experiments/qwen35_4b_oracle_distilled_semantic_verifier/reports/summary_overall.csv
Technical framing
How hard is candidate selection? Pool size vs hidden-pass candidates per task — MBPP pools are harder: barely half of visible-passing candidates also pass hidden tests, vs ~73% on HumanEval, so selection matters more there.
In the author’s words from the Overview · “Summary”
Train Qwen3.5-4B as a deployable verifier for Python candidate programs. The training oracle labels visible-test-passing candidates by hidden-test execution. At inference, the model sees only the task, public tests, public-test status, and candidate code; it ranks candidates by the probability that they pass hidden tests.
Overview
This top-level README was generated during repository normalization because the imported experiment did not include one.
- Source track:
track-z - Primary report: reports/qwen35_4b_oracle_distilled_semantic_verifier_report.md
- Metadata: metadata.yaml
How To Read
Start with the primary report, then inspect data/, reports/, analysis/, src/, and scripts/ as available. This folder remains self-contained; do not move its run data into shared directories.
Summary
Train Qwen3.5-4B as a deployable verifier for Python candidate programs. The training oracle labels visible-test-passing candidates by hidden-test execution. At inference, the model sees only the task, public tests, public-test status, and candidate code; it ranks candidates by the probability that they pass hidden tests.
Report
Rendered from reports/qwen35_4b_oracle_distilled_semantic_verifier_report.md
Objective
Train Qwen3.5-4B as a deployable verifier for Python candidate programs. The training oracle labels visible-test-passing candidates by hidden-test execution. At inference, the model sees only the task, public tests, public-test status, and candidate code; it ranks candidates by the probability that they pass hidden tests.
Data
- Train records: 90 MBPP tasks.
- Validation records: 32 MBPP tasks.
- Transfer eval records: 31 HumanEval tasks.
- Visible tests per task: 1.
- Candidate implementations per task: up to 18.
- Train verifier examples: 489.
Key Result
- HumanEval first-visible baseline: 77.4%.
- HumanEval base Qwen verifier: 93.5%.
- HumanEval candidate-pool coverage: 100.0%.
- HumanEval SFT verifier: 90.3%.
- MBPP validation first-visible baseline: 43.8%.
- MBPP validation base Qwen verifier: 71.9%.
- MBPP validation SFT verifier: 81.2%.
Readout
The SFT verifier improves MBPP validation selection by +9.4 points over the frozen base verifier and +37.5 points over first visible-pass selection. On HumanEval it improves +12.9 points over first visible-pass selection, but trails the frozen base verifier by 3.2 points. This is a useful but not complete transfer result: oracle-distilled posttraining teaches an in-domain candidate verifier, while the frozen model is already a very strong out-of-domain verifier under this candidate-generation setup.
Overall
| Dataset | Policy | Candidate-pool coverage | Selected hidden-pass | Coverage captured | Visible candidates | Hidden-pass candidates |
|---|---|---|---|---|---|---|
| humaneval | First visible-pass | 100.0% | 77.4% | 77.4% | 11.13 | 8.16 |
| humaneval | Shortest visible-pass | 100.0% | 32.3% | 32.3% | 11.13 | 8.16 |
| humaneval | Random visible-pass | 100.0% | 64.5% | 64.5% | 11.13 | 8.16 |
| humaneval | Oracle coverage | 100.0% | 100.0% | 100.0% | 11.13 | 8.16 |
| humaneval | Base Qwen verifier | 100.0% | 93.5% | 93.5% | 11.13 | 8.16 |
| humaneval | SFT Qwen verifier | 100.0% | 90.3% | 90.3% | 11.13 | 8.16 |
| mbpp | First visible-pass | 100.0% | 43.8% | 43.8% | 5.66 | 3.12 |
| mbpp | Shortest visible-pass | 100.0% | 25.0% | 25.0% | 5.66 | 3.12 |
| mbpp | Random visible-pass | 100.0% | 40.6% | 40.6% | 5.66 | 3.12 |
| mbpp | Oracle coverage | 100.0% | 100.0% | 100.0% | 5.66 | 3.12 |
| mbpp | Base Qwen verifier | 100.0% | 71.9% | 71.9% | 5.66 | 3.12 |
| mbpp | SFT Qwen verifier | 100.0% | 81.2% | 81.2% | 5.66 | 3.12 |
Interpretation
The primary question is whether oracle-labeled posttraining teaches a semantic candidate verifier that captures candidate-pool coverage under leak-free public evidence. The answer here is mixed. The adapter clearly improves in-domain MBPP selection, which means the hidden-test labels provide a learnable signal. The HumanEval result is weaker: SFT remains above simple selection baselines, but the frozen Qwen verifier is better on this eval set. The most likely reading is that this small MBPP-only SFT run partly overfits the mutation and task distribution instead of improving the model's general verifier prior. The next iteration should either train on a larger and more varied code corpus, or distill preferences from a stronger oracle while preserving the frozen model's general-code prior.
Figures
reports/figures/selected_hidden_pass.pngreports/figures/coverage_captured.pngreports/figures/selection_vs_coverage.pngreports/figures/verifier_sft_loss.png
Reproduction
python scripts/build_dataset.py --mbpp-train 90 --mbpp-valid 32 --humaneval-eval 31 --visible-tests 1 --candidate-count 18
python scripts/eval_verifier.py --records data/mbpp_valid_records.jsonl --policy first --name first_visible --out reports/eval/mbpp_first_visible.json
python scripts/eval_verifier.py --records data/mbpp_valid_records.jsonl --policy shortest --name shortest_visible --out reports/eval/mbpp_shortest_visible.json
python scripts/eval_verifier.py --records data/mbpp_valid_records.jsonl --policy random --name random_visible --out reports/eval/mbpp_random_visible.json
python scripts/eval_verifier.py --records data/mbpp_valid_records.jsonl --policy oracle --name oracle_coverage --out reports/eval/mbpp_oracle_coverage.json
python scripts/eval_verifier.py --records data/humaneval_eval_records.jsonl --policy first --name first_visible --out reports/eval/humaneval_first_visible.json
python scripts/eval_verifier.py --records data/humaneval_eval_records.jsonl --policy shortest --name shortest_visible --out reports/eval/humaneval_shortest_visible.json
python scripts/eval_verifier.py --records data/humaneval_eval_records.jsonl --policy random --name random_visible --out reports/eval/humaneval_random_visible.json
python scripts/eval_verifier.py --records data/humaneval_eval_records.jsonl --policy oracle --name oracle_coverage --out reports/eval/humaneval_oracle_coverage.json
python scripts/eval_verifier.py --records data/mbpp_valid_records.jsonl --policy base --name base_verifier --out reports/eval/mbpp_base_verifier.json
python scripts/eval_verifier.py --records data/humaneval_eval_records.jsonl --policy base --name base_verifier --out reports/eval/humaneval_base_verifier.json
python scripts/train_verifier_sft.py --max-steps 220 --batch-size 2 --grad-accum 2
python scripts/eval_verifier.py --records data/mbpp_valid_records.jsonl --policy adapter --name sft_verifier --out reports/eval/mbpp_sft_verifier.json --adapter-dir /workspace/large_artifacts/qwen35_4b_oracle_distilled_semantic_verifier/models/verifier_sft_lora
python scripts/eval_verifier.py --records data/humaneval_eval_records.jsonl --policy adapter --name sft_verifier --out reports/eval/humaneval_sft_verifier.json --adapter-dir /workspace/large_artifacts/qwen35_4b_oracle_distilled_semantic_verifier/models/verifier_sft_lora
python scripts/make_report.pyExperiment log 7
Show the running log (7 entries)
Objective
Train a deployable Qwen3.5-4B semantic verifier for Python candidate programs. Training labels come from hidden-test execution. Inference uses only public task information and candidate code.
Initial Design
- Training substrate: MBPP.
- Transfer substrate: HumanEval.
- Action interface:
A = candidate will pass hidden tests,B = candidate will fail hidden tests. - Candidate pool: canonical solution, source mutations, and generic default-return variants.
- Train examples: visible-test-passing candidates only, balanced to keep hard negatives.
- Evaluation metric: selected hidden-pass rate and fraction of candidate-pool coverage captured.
- Large artifacts: LoRA adapter and tokenizer under
/workspace/large_artifacts/qwen35_4b_oracle_distilled_semantic_verifier.
Reproduction
python scripts/build_dataset.py --mbpp-train 90 --mbpp-valid 32 --humaneval-eval 31 --visible-tests 1 --candidate-count 18
python scripts/train_verifier_sft.py --max-steps 220 --batch-size 2 --grad-accum 2
python scripts/make_report.pyRun Notes
- Dataset construction initially targeted 40 HumanEval eval records. With leak-free public tests and a required visible-passing hidden-failing hard negative, only 31 HumanEval tasks were usable. I kept the hard-negative and leak-free constraints and reduced the HumanEval eval target to 31.
- MBPP train and validation targets remained at 90 and 32 records.
Dataset Manifest
- MBPP train: 90 records, 489 balanced verifier examples, 5.86 visible-passing candidates per task on average.
- MBPP validation: 32 records, 181 verifier examples, 5.66 visible-passing candidates per task on average.
- HumanEval eval: 31 records, 345 verifier examples, 11.13 visible-passing candidates per task on average.
- Candidate-pool coverage is 100% for all retained splits.
- HumanEval skips before reaching 31 usable records: 36 tasks had no public doctest-style tests, 33 had no visible-passing hidden-failing hard negative.
Evaluation Results
| Dataset | Policy | Selected hidden-pass |
|---|---|---|
| MBPP validation | First visible-pass | 43.8% |
| MBPP validation | Shortest visible-pass | 25.0% |
| MBPP validation | Random visible-pass | 40.6% |
| MBPP validation | Frozen Qwen verifier | 71.9% |
| MBPP validation | SFT Qwen verifier | 81.2% |
| MBPP validation | Oracle coverage | 100.0% |
| HumanEval | First visible-pass | 77.4% |
| HumanEval | Shortest visible-pass | 32.3% |
| HumanEval | Random visible-pass | 64.5% |
| HumanEval | Frozen Qwen verifier | 93.5% |
| HumanEval | SFT Qwen verifier | 90.3% |
| HumanEval | Oracle coverage | 100.0% |
Interpretation
The SFT verifier learned a real in-domain selection signal: MBPP validation improved from 71.9% for the frozen Qwen verifier to 81.2% for the adapter. The HumanEval result is weaker. SFT stayed above simple baselines, but it underperformed the frozen Qwen verifier by 3.2 points. This suggests the small MBPP-only oracle-distillation run partly overfit the candidate mutation and task distribution instead of cleanly improving the general code-verification prior.
The next iteration should keep the oracle-labeled verifier idea, but train on a broader code-candidate distribution and include an explicit regularizer or distillation term that preserves the frozen model's strong HumanEval behavior.
Figures 4
Data files 8
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
reports/candidate_scores.csv76 kBreports/summary_overall.csv1.1 kBreports/eval/humaneval_base_verifier.json82 kBreports/eval/humaneval_first_visible.json13 kBreports/eval/humaneval_oracle_coverage.json13 kBreports/eval/humaneval_random_visible.json13 kBreports/eval/humaneval_sft_verifier.json81 kBreports/eval/humaneval_shortest_visible.json14 kB
Reproduce
Run steps are documented inside the experiment folder (README and scripts).