Qwen3.5-4B Bucket-Belief Probe Ranker
The one idea you need
Like Twenty Questions for code: to find one hidden program in a large field, you run tests that sort the survivors into groups by their output. The model was trained to guess which group hides the real answer, then pick the test that narrows the field most.
The question
Can a small model learn to pick the most revealing tests, beating a simple fixed rule, when hunting for one hidden program in a large field of candidates?
What we found
Barely. After training, the 4-billion-parameter model solved 50% of puzzles within three tests, versus 49% for a simple no-learning rule that just picks the cleanest-splitting test, essentially a tie. Even limited to those same candidate tests, a strategy that peeks at the answer reaches 61%; across every available test it hits 87%. The model captured only about 1 of every 10 points on the table.
Why it matters
When candidate tests are already pre-filtered by a simple splitting rule, bolting a learned ranker on top wins almost nothing. The leverage is in generating better tests or a richer view of the puzzle, not re-ranking the handful the rule already surfaced.
On this page
Results at a glance 2
How to read
Each bar is one strategy; height is the share of puzzles fully solved after three narrowing tests, so taller is better. Left bars are the simple rule, the untrained model, and the trained model; the two right bars peek at the hidden answer to set the ceiling.
Takeaway → The trained bar sits almost level with the simple-rule bar and far below the answer-peeking bars, showing training added almost nothing.
Data table
| Policy arm | hidden-all accuracy @ budget 3 |
|---|---|
| split top-1 | 48.8% |
| base bucket ranker | 48.8% |
| SFT bucket ranker | 50% |
| oracle over top-8 | 61.3% |
| full-pool oracle | 86.9% |
Numbers from experiments/qwen35_4b_bucket_belief_probe_ranker/reports/summary_overall.csv
Technical framing
Budget-3 hidden-all accuracy: SFT bucket ranker barely beats split top-1 — The SFT ranker captures 1.25 of the 12.5-point oracle gap over split top-1; most target-aware headroom stays unconverted.
How to read
Bars are grouped into an easy family and a hard family; within each group, colored bars compare the strategies and height is the share solved, so taller is better. The answer-peeking bars show how much room exists.
Takeaway → On the easy family every strategy nearly maxes out, but on the hard family the trained model stays near the floor while the peeking strategy climbs, so the unclaimed gap lives there.
Data table
| Template | split top-1 | base bucket ranker | SFT bucket ranker | oracle over top-8 | full-pool oracle |
|---|---|---|---|---|---|
| affine-mod | 93.8% | 93.8% | 95% | 98.8% | 100% |
| compare-gate | 3.8% | 3.8% | 5% | 23.8% | 73.8% |
Numbers from experiments/qwen35_4b_bucket_belief_probe_ranker/reports/summary_by_template.csv
Technical framing
Budget-3 accuracy by task template — Affine-mod is nearly solved by every arm; compare-gate is where the oracle gap lives, and the SFT ranker recovers almost none of it.
In the author’s words from the Report · “Results”
At budget 3, the SFT bucket ranker reached 50.0% hidden-all accuracy versus 48.8% for target-independent split top-1, 48.8% for the base bucket ranker, 61.3% for an oracle over the same top-8 probe set, and 86.9% for the full-pool oracle. The recoverable top-8 oracle gap is 12.5 points over split top-1; the SFT ranker captured 1.3 points of that gap.
Overview
This standalone experiment tests whether Qwen3.5-4B can convert target-aware probe headroom into a deployable probe-ranking policy.
The model does not emit operators. For each verifier state, the system mines the top 8 candidate probes by target-independent split quality. For each probe, the prompt shows the output buckets that surviving candidate programs would produce. Qwen is trained to predict which bucket contains the hidden target program. At rollout time, the model scores each candidate probe by predicted expected survivors and the verifier executes the probe with the smallest score.
Large model artifacts are intentionally outside this directory:
/workspace/large_artifacts/qwen35_4b_bucket_belief_probe_ranker/models/bucket_sft_lora/workspace/large_artifacts/qwen35_4b_bucket_belief_probe_ranker/cache/bucket_train_examples.jsonl
Main outputs:
reports/qwen35_4b_bucket_belief_probe_ranker_report.mdreports/figures/reports/*.csvreports/eval/*.jsonrun_logs/logs/experiment_log.md
Reproduction order:
python scripts/build_dataset.py --train-per-cell 50 --eval-per-cell 20 --states-per-record 3 --query-pool-cases 96 --action-source mined8python scripts/build_bucket_dataset.pypython scripts/build_bucket_dataset.py --records data/eval_records.jsonl --states data/process_eval_states.jsonl --out data/bucket_eval_examples.jsonlpython scripts/eval_bucket_ranker.py --policy split_top1 --name split_top1 --max-budget 3python scripts/eval_bucket_ranker.py --policy oracle_topk --name oracle_top8 --max-budget 3python scripts/eval_bucket_ranker.py --policy fullpool_oracle --name fullpool_oracle --max-budget 3python scripts/eval_bucket_ranker.py --policy base_bucket --name base_bucket_ranker --max-budget 3python scripts/train_bucket_sft.py --max-steps 220 --batch-size 2 --grad-accum 2python scripts/eval_bucket_ranker.py --policy adapter_bucket --name sft_bucket_ranker --adapter-dir /workspace/large_artifacts/qwen35_4b_bucket_belief_probe_ranker/models/bucket_sft_lora --max-budget 3python scripts/make_report.py
Report
Rendered from reports/qwen35_4b_bucket_belief_probe_ranker_report.md
Objective
This standalone experiment tests whether a Qwen3.5-4B LoRA can turn verifier state into a deployable probe-ranking policy by predicting which candidate-output bucket contains the hidden target program. The model does not name operators and does not see the answer at inference time; it scores candidate probes by the expected survivors implied by its bucket probabilities.
Experimental Design
- Substrate: two-operator typed program search with a 96-case probe pool, four visible observations, sixteen hidden checks, and library sizes 64, 128, 256, and held-out 512 at evaluation.
- Candidate probes for the learned ranker: top 8 remaining probes by target-independent split statistics.
- Training target: for each candidate probe, predict the output bucket that contains the true target program.
- Rollout rule: choose the probe with the smallest model-predicted expected survivor count, observe its true output, update the verifier candidate set, and repeat for three probes.
- Controls: target-independent split top-1, target-aware oracle over the same top-8 candidate probes, target-aware oracle over the full 96-case pool, and the untrained base model under the same bucket-scoring rule.
Data Summary
- Process records: train=300, eval=160.
- Bucket SFT examples: 4952.
- Bucket SFT states: 619.
Results
- At budget 3, the SFT bucket ranker reached 50.0% hidden-all accuracy versus 48.8% for target-independent split top-1, 48.8% for the base bucket ranker, 61.3% for an oracle over the same top-8 probe set, and 86.9% for the full-pool oracle.
- The recoverable top-8 oracle gap is 12.5 points over split top-1; the SFT ranker captured 1.3 points of that gap.
Budget-3 Overall
| Arm | Hidden-all accuracy | Candidates left | Hidden-equivalent left |
|---|---|---|---|
| Split top-1 | 48.8% | 1005.0 | 149.1 |
| Oracle over top-8 | 61.3% | 657.7 | 146.8 |
| Oracle over full pool | 86.9% | 267.1 | 145.5 |
| Base bucket ranker | 48.8% | 795.6 | 156.6 |
| SFT bucket ranker | 50.0% | 1020.1 | 149.2 |
Budget-3 by Template
| Arm | Affine-mod | Compare-gate |
|---|---|---|
| Split top-1 | 93.8% | 3.8% |
| Oracle over top-8 | 98.8% | 23.8% |
| Oracle over full pool | 100.0% | 73.8% |
| Base bucket ranker | 93.8% | 3.8% |
| SFT bucket ranker | 95.0% | 5.0% |
Interpretation
The experiment is intentionally decisive about whether target-aware oracle headroom can be converted into deployable ranking by a learned bucket-belief model. A lift over split top-1 means the model learned a useful non-uniform belief over output buckets. A result near split top-1 means the oracle gap is mostly unavailable without additional state, candidate representation, or truly generative probe construction.
The base-model arm matters because it distinguishes learned bucket inference from prompt priors. The oracle-over-top-8 arm matters because it bounds what any ranker can gain when it is restricted to the same split-mined candidate probes. The full-pool oracle remains a headroom measurement, not a deployable result.
Figures
reports/figures/budget_curve.pngreports/figures/budget3_accuracy.pngreports/figures/budget3_by_template.pngreports/figures/bucket_sft_loss.png
Reproduction
Run from this experiment directory:
python scripts/build_dataset.py --train-per-cell 50 --eval-per-cell 20 --states-per-record 3 --query-pool-cases 96 --action-source mined8
python scripts/build_bucket_dataset.py
python scripts/build_bucket_dataset.py --records data/eval_records.jsonl --states data/process_eval_states.jsonl --out data/bucket_eval_examples.jsonl
python scripts/eval_bucket_ranker.py --policy split_top1 --name split_top1 --max-budget 3
python scripts/eval_bucket_ranker.py --policy oracle_topk --name oracle_top8 --max-budget 3
python scripts/eval_bucket_ranker.py --policy fullpool_oracle --name fullpool_oracle --max-budget 3
python scripts/eval_bucket_ranker.py --policy base_bucket --name base_bucket_ranker --max-budget 3
python scripts/train_bucket_sft.py --max-steps 220 --batch-size 2 --grad-accum 2
python scripts/eval_bucket_ranker.py --policy adapter_bucket --name sft_bucket_ranker --adapter-dir /workspace/large_artifacts/qwen35_4b_bucket_belief_probe_ranker/models/bucket_sft_lora --max-budget 3
python scripts/make_report.pyExperiment log 4
Show the running log (4 entries)
Setup
- Created a standalone experiment directory at
/workspace/experiments/qwen35_4b_bucket_belief_probe_ranker. - Created a separate large-artifact root at
/workspace/large_artifacts/qwen35_4b_bucket_belief_probe_ranker. - Hypothesis: a Qwen3.5-4B LoRA can improve deployable active probing if it learns to infer which output bucket contains the hidden target program, rather than only selecting probes by target-independent split quality.
Planned Arms
split_top1: choose the best remaining probe by target-independent expected survivors.oracle_top8: choose the target-aware best probe from the same top-8 split-ranked candidate probes.fullpool_oracle: choose the target-aware best probe from the full 96-case pool.base_bucket_ranker: untrained Qwen3.5-4B bucket-belief scoring over the top-8 probes.sft_bucket_ranker: QLoRA SFT bucket-belief scoring over the top-8 probes.
Notes
- The full-pool and top-8 oracle arms are headroom measurements, not deployable policies.
- A deployable win requires
sft_bucket_rankerto beatsplit_top1andbase_bucket_ranker. - If the SFT ranker stays near
split_top1, the result means target-aware oracle headroom is not recoverable from this bucket prompt alone.
Run Results
- Built 300 train records, 160 eval records, 619 train process states, and 334 eval process states.
- Built 4,952 train bucket examples and 2,672 eval bucket examples.
- Trained Qwen3.5-4B QLoRA bucket-belief adapter for 220 optimizer steps.
Budget-3 hidden-all accuracy:
split_top1: 48.8%oracle_top8: 61.3%fullpool_oracle: 86.9%base_bucket_ranker: 48.8%sft_bucket_ranker: 50.0%
Budget-3 compare-gate accuracy:
split_top1: 3.8%oracle_top8: 23.8%fullpool_oracle: 73.8%base_bucket_ranker: 3.8%sft_bucket_ranker: 5.0%
- Interpretation: bucket-belief SFT produced a real but tiny deployable lift, capturing about 1.3 points of a 12.5-point top-8 oracle gap. The larger full-pool oracle gap remains mostly unrecovered.
- Moved the generated train prompt JSONL to
/workspace/large_artifacts/qwen35_4b_bucket_belief_probe_ranker/cache/bucket_train_examples.jsonlafter training to keep the experiment directory compact.
Figures 4
Data files 3
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
reports/summary_by_cell.csv13 kBreports/summary_by_template.csv2.4 kBreports/summary_overall.csv1.3 kB
Reproduce
Run steps are documented inside the experiment folder (README and scripts).