Qwen3.5-4B Sampler Portfolio Scheduler
The one idea you need
Writing code with a small model is like fishing with different lures: each sampling style hooks problems the others miss. Casting more with one favorite lure catches more fish, but only knowing which lure to tie on per spot reaches them all.
The question
When a small model writes code, is it better to keep several different sampling styles and pick the right one per problem, or just sample more from one style?
What we found
In theory yes, in practice no. A flawless chooser that knew which style would crack each problem solved 75% of coding tasks at the cheapest run's cost, while just doubling tries from one style reached 67% at twice the cost. But every chooser we could actually build tied or lost to simply sampling more. The prize is real; the picker isn't.
Why it matters
Don't pour effort into brute-force sampling or scaling one style for coverage. The unclaimed win is a cheap predictor of which sampling style will add a new solution for a problem, before you spend the tokens.
On this page
Results at a glance 3
How to read
Each bar is one way of choosing generation styles; bar height is the share of 24 coding problems solved, so taller is better. The far-right bar is a perfect after-the-fact picker; the rest are approaches you could actually ship.
Takeaway → Only the far-right perfect-hindsight bar clearly rises above the pack; every buildable approach sits level with or below simply sampling more from one style.
Data table
| Generation arm / schedule | coverage |
|---|---|
| base hot @4 | 58.3% |
| base hot @8 | 66.7% |
| constrained @4 | 62.5% |
| hot4 + constrained4 | 66.7% |
| visible-test gate | 62.5% |
| oracle arm choice | 75% |
Numbers from report table (reports/final_report.md, Constrained-Arm Subset Results)
Technical framing
Constrained subset: oracle arm choice beats sampling more, deployable schedulers don't — Oracle per-task arm choice reaches 75% coverage; every deployable schedule ties or trails simply sampling 8 from one policy (66.7%).
How to read
Each bar is a way of spending generation effort; height is the share of 80 problems solved, taller is better. The far-right bar keeps every candidate answer per problem, ignoring cost.
Takeaway → The trained picker dips below the plain fixed recipes, and the towering far-right bar reveals many solvable problems no practical approach reached.
Data table
| Portfolio schedule | coverage |
|---|---|
| base prefix @4 | 70% |
| prefix2 + mid4 | 70% |
| learned scheduler | 65% |
| oracle best block | 71.2% |
| full union (all candidates) | 86.2% |
Numbers from report table (reports/final_report.md, Broad Source-Policy Results)
Technical framing
Broad 80-task pool: learned scheduler loses coverage vs static prefix baseline — The learned scheduler (65%) underperforms static schedules (70%); the full union shows 86.2% coverage exists in the pool but costs 8x the tokens.
How to read
Each bar is one approach; height is the total compute it spent across 24 problems, so shorter is better. Read it beside the first chart, which shows how many problems each one solved.
Takeaway → The perfect-picker bar is among the shortest, about half the compute of sampling more, yet it solved the most, so its payoff is cheap not costly.
Data table
| Generation arm / schedule | forward tokens (24 tasks) |
|---|---|
| base hot @4 | 23.43k |
| base hot @8 | 44.22k |
| constrained @4 | 21.79k |
| hot4 + constrained4 | 45.22k |
| visible-test gate | 29.18k |
| oracle arm choice | 22.22k |
Numbers from report table (reports/final_report.md, Constrained-Arm Subset Results)
Technical framing
Token cost per constrained arm: oracle headroom comes at half the cost of sampling more — Oracle arm choice hits its 75% coverage at 22k tokens, roughly half the 44k spent by base hot @8, which only reaches 66.7%.
In the author’s words from the Report · “Interpretation”
The constrained subset is sharper: the oracle arm chooser reaches 75.0% at roughly the same token cost as K4, while ordinary hot K8 reaches 66.7% at about double the tokens. That means the next valuable target is a better policy-value estimator, not more blind sampling and not a one-arm LoRA scale-up. The concrete next direction is to collect a larger multi-policy training set where every task has matched candidates from each policy arm, then train a policy-value model to predict which arm is likely to add new hidden coverage from prompt plus cheap prefix evidence. This pilot says there is arm-choice headroom, but it is sparse and not exposed cleanly enough to the simple visible-feature schedulers used here.
Overview
Standalone experiment package for testing whether multiple Qwen3.5-4B generation policies can be combined more efficiently than simply sampling more from one policy.
The experiment treats each generation policy as an arm that can contribute candidate programs. It evaluates static portfolios, oracle portfolio headroom, and a small learned scheduler that observes only a short base-hot prefix before choosing which policy block receives the remaining budget.
Large artifacts, if any are added later, should be stored outside this directory under:
/workspace/large_artifacts/qwen35_4b_sampler_portfolio_scheduler
Final report: reports/final_report.md
Report
Rendered from reports/final_report.md
Question
Can a portfolio of generation policies, selected by static schedules or a small deployable scheduler, improve the coverage/pass@1/forward-token Pareto frontier over simply sampling more from one policy?
This package evaluates two views:
- a broad 80-task source-policy pool with low/mid/high/diverse candidate blocks;
- a 24-task constrained-arm subset comparing base-hot sampling with a constrained preference policy.
Oracle rows are reported only as headroom. Deployable rows use fixed schedules, visible-test gates, or schedulers trained without hidden eval labels.
Broad Source-Policy Results
| arm | n | coverage | pass@1 | candidates/task | parse/task | functional diversity | forward tokens |
|---|---|---|---|---|---|---|---|
| base_prefix_k4 | 80 | 70.0% | 61.3% | 3.45 | 3.36 | 47.7% | 62270 |
| prefix2_mid4 | 80 | 70.0% | 61.3% | 3.15 | 3.08 | 50.9% | 57013 |
| learned_scheduler_after_prefix2 | 80 | 65.0% | 61.3% | 2.80 | 2.73 | 53.5% | 51517 |
| oracle_best_block_after_prefix2 | 80 | 71.2% | 61.3% | 2.30 | 2.27 | 58.4% | 40154 |
| full_union_all_candidates | 80 | 86.2% | 61.3% | 24.09 | 22.23 | 34.0% | 504344 |
Constrained-Arm Subset Results
| arm | n | coverage | pass@1 | candidates/task | parse/task | functional diversity | forward tokens |
|---|---|---|---|---|---|---|---|
| subset_base_hot_k4 | 24 | 58.3% | 37.5% | 4.00 | 3.54 | 51.0% | 23434 |
| subset_base_hot_k8 | 24 | 66.7% | 41.7% | 7.71 | 7.50 | 32.1% | 44219 |
| subset_constrained_k4 | 24 | 62.5% | 41.7% | 3.83 | 3.58 | 54.9% | 21785 |
| subset_hot4_plus_constrained4 | 24 | 66.7% | 37.5% | 7.83 | 7.12 | 34.1% | 45219 |
| subset_visible_gate_hot4_then_constrained | 24 | 62.5% | 37.5% | 4.83 | 4.29 | 47.4% | 29184 |
| subset_oracle_choose_arm | 24 | 75.0% | 37.5% | 4.17 | 3.96 | 53.8% | 22223 |
Scheduler Diagnostics
Source scheduler train labels: {'high4': 5, 'low4': 12, 'mid4': 11, 'stop': 52}
Source scheduler eval actions: {'high4': 3, 'low4': 11, 'mid4': 11, 'stop': 55}
Constrained leave-one-task-out labels: {'constrained4': 1, 'hot_next4': 7, 'stop': 16}
Constrained leave-one-task-out summary: {'action_counts': {'constrained4': 2, 'hot_next4': 6, 'stop': 16}, 'candidate_count_mean': 5.333333333333333, 'coverage': 0.625, 'covered_tasks': [11, 12, 13, 14, 17, 18, 19, 22, 23, 27, 28, 29, 30, 32, 33], 'distinct_functional_rate_mean': 0.4479166666666667, 'forward_tokens': 30495, 'parse_success_mean': 5.125, 'pass1_proxy': 0.4166666666666667, 'records': 24, 'visible_coverage': 0.8333333333333334}
Gate Readout
Broad source-policy run: base prefix K4 reached 70.0%; learned scheduling reached 65.0%; oracle after the same prefix reached 71.2%. The full source-policy union reached 86.2%, but at 504344 forward tokens. Constrained-arm subset: base hot K8 reached 66.7%; hot4+constrained4 tied it at 66.7%; oracle arm choice reached 75.0% at 22223 forward tokens. Leave-one-task-out constrained scheduler reached 62.5% with action counts {'constrained4': 2, 'hot_next4': 6, 'stop': 16}. Gate readout: no deployable scheduler in this pilot beat the single-policy sample-more reference. The positive signal is oracle headroom for choosing among policy arms, not an already solved scheduler.
Interpretation
The portfolio idea is not dead, but the naive scheduler is not enough. The broad pool shows that large unions contain much more coverage, while small static or learned schedules do not extract it. The constrained subset is sharper: the oracle arm chooser reaches 75.0% at roughly the same token cost as K4, while ordinary hot K8 reaches 66.7% at about double the tokens. That means the next valuable target is a better policy-value estimator, not more blind sampling and not a one-arm LoRA scale-up.
The concrete next direction is to collect a larger multi-policy training set where every task has matched candidates from each policy arm, then train a policy-value model to predict which arm is likely to add new hidden coverage from prompt plus cheap prefix evidence. This pilot says there is arm-choice headroom, but it is sparse and not exposed cleanly enough to the simple visible-feature schedulers used here.
Experiment log 5
Show the running log (5 entries, 2026-06-26)
2026-06-26
- Created standalone sampler-portfolio scheduler experiment package.
- Primary question: can complementary generation policies improve the coverage/pass@1/forward-token Pareto frontier compared with simply sampling more from one policy?
Policies under test:
- base hot sampling;
- mixed-temperature base sampling;
- constrained preference LoRA sampling.
Planned readouts:
- static portfolio coverage and token cost;
- oracle best-block headroom after a short base-hot prefix;
- learned scheduler using only deployable prefix features;
- pass@1 and parseability guardrails.
- Success criterion: a deployable portfolio or learned scheduler must beat the base sample-more reference on the coverage/pass@1/forward-token Pareto frontier.
Iteration Notes
- Started a fresh
train_base_mixed_k8generation run for 36 MBPP train tasks. - Stopped the run after 4/36 tasks because K=8 mixed sampling was taking roughly one minute per task, which would have delayed the scheduler test by hours before producing any new information.
- Pivoted to complete candidate pools copied into this package under
data/. The package is standalone: all analysis scripts read only local files in this experiment directory. Broad source-policy pool:
- train:
data/train_policy_pool_records.jsonl; - eval:
data/eval_policy_union_records.jsonl.
- train:
Constrained-arm subset:
data/subset_base_hot_k4_records.jsonl;data/subset_base_hot_k8_records.jsonl;data/subset_constrained_dpo_k4_records.jsonl.
Evaluations Run
scripts/evaluate_source_portfolios.py- trained a linear scheduler on 80 train tasks using prompt and two-sample prefix features;
- evaluated base/static/learned/oracle source-policy schedules on 80 eval tasks.
scripts/evaluate_constrained_subset.py- evaluated static base-hot/constrained portfolios and oracle arm choice on 24 matched tasks.
scripts/diagnose_constrained_scheduler.py- ran a leave-one-task-out scheduler diagnostic for stop vs hot-next vs constrained choice.
scripts/make_report.py- wrote the final report and figures.
Results
Broad source-policy run:
| arm | coverage | pass@1 | forward tokens |
|---|---|---|---|
| base_prefix_k4 | 70.0% | 61.3% | 62270 |
| prefix2_mid4 | 70.0% | 61.3% | 57013 |
| learned_scheduler_after_prefix2 | 65.0% | 61.3% | 51517 |
| oracle_best_block_after_prefix2 | 71.2% | 61.3% | 40154 |
| full_union_all_candidates | 86.2% | 61.3% | 504344 |
Constrained-arm subset:
| arm | coverage | pass@1 | forward tokens |
|---|---|---|---|
| subset_base_hot_k4 | 58.3% | 37.5% | 23434 |
| subset_base_hot_k8 | 66.7% | 41.7% | 44219 |
| subset_constrained_k4 | 62.5% | 41.7% | 21785 |
| subset_hot4_plus_constrained4 | 66.7% | 37.5% | 45219 |
| subset_visible_gate_hot4_then_constrained | 62.5% | 37.5% | 29184 |
| subset_oracle_choose_arm | 75.0% | 37.5% | 22223 |
Scheduler diagnostic:
- Source scheduler labels: stop 52, low4 12, mid4 11, high4 5.
- Source learned scheduler actions: stop 55, low4 11, mid4 11, high4 3.
- Constrained leave-one-task-out labels: stop 16, hot_next4 7, constrained4 1.
- Constrained leave-one-task-out result: 62.5% coverage at 30495 forward tokens.
Readout
- No deployable scheduler or static portfolio beat the single-policy sample-more reference in this pilot.
- The constrained-arm oracle chooser shows real arm-selection headroom: 75.0% coverage at 22223 forward tokens versus base hot K=8 at 66.7% and 44219 tokens.
- The simple feature schedulers did not recover that headroom.
- Next direction: collect a larger matched multi-policy training set and train a policy-value estimator that predicts per-arm marginal coverage from prompt plus cheap prefix evidence.
Figures 5
Data files 3
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
reports/constrained_subset_results.json9.0 kBreports/report_summary.json54 kBreports/source_portfolio_results.json24 kB
Reproduce
Runnable scripts exist in the experiment folder, but the exact invocation was not written down.