Qwen3.5-4B Oracle Process GRPO

Finished2026-06-24imported · line ZBenchmark Generalization Evidence-Conditioned Selection Interpretability and Diagnostics Operator and Skill Inventories+4 moreGitHub ↗

The one idea you need

Picture "Guess Who?", but the hidden faces are secret math rules. The model never guesses the rule itself. It only picks which of eight test-inputs to try; a search engine runs that test and crosses off every rule that disagrees, for up to three rounds.

The question

Can a small model get better at solving puzzles by learning which diagnostic questions to ask, instead of trying to guess the answer outright?

What we found

Yes, but with sharp limits. Teaching the model to copy a flawless test-picker lifted its success from about a third of puzzles to 43%, versus roughly 31% for picking blindly. Layering on preference-based and reward-based training gained almost nothing more. And where the available tests simply can't tell two rules apart, even perfect choices topped out near 9%.

Why it matters

For small-model agents that pick tools or diagnostic steps, plainly imitating good decisions can capture nearly all the gain; heavier reward-based training may add little. But no training beats an information ceiling: fix which tests are available first.

Solved within three tests, untrained then trained33% → 44%share of puzzles pinned down, before training versus the best trained model

Plain copying vs fancier training43% vs 44%imitating good picks nearly matched preference- and reward-based training

Picking tests blindlyabout 31%random-choice baseline the untrained model barely beat

Ceiling on low-information puzzlesabout 9%where tests can't separate rules, even a perfect picker maxes out here

On this page

Results at a glance
Overview
Report
Experiment log
Figures
Data files
Reproduce
Related

Results at a glance 3

Puzzles solved within three tests, by test-picker

How to read

Each bar is one test-picking method; height is the share of puzzles where it pinned down the hidden rule within three tests. Taller is better. The far-right bar is a flawless picker; two bars are controls — one shuffles the training rewards, the other scrambles the puzzle clues.

Takeaway → Trained pickers rise well above blind guessing and the untrained model, nearly reaching the flawless picker, while copying almost ties the fancier methods; shuffling the rewards drops scores partway back, and scrambling the clues collapses all the way to blind guessing.

Data table

policy	hidden-all solved @ budget 3
random	30.5%
base Qwen	32.8%
SFT	43%
process-DPO	43.8%
shuffled-reward DPO	36.7%
scrambled-features DPO	31.2%
GRPO	43.8%
max-split heuristic	44.5%
oracle	46.9%

Numbers from reports/overall_by_policy_budget.csv

Technical framing

Solved at 3 probes: trained controllers vs baselines and oracle — Posttraining lifts base 32.8% to 43.8% (DPO/GRPO), ~78% of oracle headroom; shuffled-reward and scrambled-feature controls collapse.

More tests solve more puzzles; trained pickers pull ahead

How to read

Horizontal axis is how many tests are allowed, from zero to three; vertical is the share of puzzles solved. All lines start together low, then higher is better. The top line is the flawless picker.

Takeaway → Lines start together, then trained pickers climb faster with each added test, tracking the flawless picker's curve far more closely than the untrained model.

Data table

probe budget (verifier queries)	base Qwen	SFT	process-DPO	GRPO	oracle
0	3.9%	3.9%	3.9%	3.9%	3.9%
1	13.3%	18%	20.3%	23.4%	28.9%
2	27.3%	33.6%	32%	34.4%	43%
3	32.8%	43%	43.8%	43.8%	46.9%

Numbers from reports/overall_by_policy_budget.csv

Technical framing

Solve rate grows with probe budget, trained policies pull ahead — All policies start equal at budget 0; trained controllers track the oracle curve more closely at every extra probe.

Gains land on information-rich puzzles, not sparse ones

How to read

Bars group two puzzle types: an information-rich one where each test reveals a lot, and a sparse one where tests barely narrow the options. Height is share solved; taller is better; one bar per method.

Takeaway → Training lifts scores sharply on information-rich puzzles but barely moves sparse ones, where even a flawless picker solves under one in ten, a hard ceiling.

Data table

output regime	base Qwen	SFT	process-DPO	GRPO	oracle
pair_affine_mod	62.5%	79.7%	81.2%	81.2%	84.4%
pair_compare_gate	3.1%	6.2%	6.2%	6.2%	9.4%

Numbers from report table

Technical framing

Gains concentrate in the information-rich regime — Training helps where probes are informative; the compare-gate regime is identifiability-bounded — even the oracle reaches only 9.4%.

In the author’s words from the Report · “Interpretation”

This supports the process-control version of the neurosymbolic hypothesis: let exhaustive search and execution make answers reachable, then train Qwen to orchestrate verifier actions. The cleanest signal is not GRPO alone; it is the full stack: SFT learns the action interface and makes a large jump over base. Process-DPO uses the perfect per-step verifier oracle and gives a smaller additional improvement. GRPO is viable but did not beat DPO in this short run. Shuffled reward and scrambled feature controls collapse, so the improvement is not just formatting or letter bias. The hard ceiling is also clear. The controller cannot solve states where three observations do not identify a hidden-correct candidate. … Read the full result →

Overview

This standalone experiment trains Qwen3.5-4B as a process controller inside an executable verifier MDP. The model does not emit operators directly. It chooses among displayed probe actions, and exhaustive search updates the candidate set deterministically.

Large model artifacts are intentionally outside this directory:

/workspace/large_artifacts/qwen35_4b_oracle_process_grpo/models/sft_process_lora
/workspace/large_artifacts/qwen35_4b_oracle_process_grpo/models/dpo_process_lora
/workspace/large_artifacts/qwen35_4b_oracle_process_grpo/models/dpo_shuffled_lora
/workspace/large_artifacts/qwen35_4b_oracle_process_grpo/models/grpo_process_lora

Main outputs:

reports/qwen35_4b_oracle_process_grpo_report.md
reports/figures/
reports/*.csv
run_logs/
logs/experiment_log.md

Reproduction order:

python scripts/build_dataset.py --train-per-cell 80 --eval-per-cell 16 --states-per-record 3
Run non-model baselines with scripts/eval_policy.py.
python scripts/train_sft_policy.py --max-steps 360 --batch-size 2 --grad-accum 2
python scripts/train_dpo_policy.py --max-steps 160 --batch-size 2 --grad-accum 2
python scripts/train_dpo_policy.py --max-steps 160 --batch-size 2 --grad-accum 2 --shuffle-rewards --output-dir /workspace/large_artifacts/qwen35_4b_oracle_process_grpo/models/dpo_shuffled_lora
python scripts/train_grpo_policy.py --max-steps 120 --batch-size 2 --grad-accum 2 --group-size 8
python scripts/make_report.py

Report

Rendered from reports/qwen35_4b_oracle_process_grpo_report.md

Question

Can Qwen3.5-4B learn to make useful decisions inside a deterministic verifier MDP when the training oracle can score process actions exactly?

The model is not asked to name operators directly. It receives a compact process state containing visible executions, candidate-set size, and eight concrete probe choices with candidate-output bucket summaries. The action is one letter, A-H. The verifier then executes that probe, filters candidates, and repeats for up to three probes. Evaluation is deployable: learned policies do not see hidden labels or the target pair.

Design

Base model: Qwen3.5-4B, 4-bit QLoRA adapters.
Train records: 480; eval records: 128.
Informative train states: 1138; informative eval states: 299.
Eval ladder: library sizes 64, 128, 256, 512; two output regimes, pair_affine_mod and pair_compare_gate.
Probe budget: 0-3 verifier queries.
Optimizers: oracle-action SFT, process-DPO, shuffled-reward DPO control, and GRPO.

Main Result

At budget 3, Qwen posttraining substantially improved the process controller over the base policy:

policy	hidden-all @3	exact pair @3	survivors @3
Random	30.5%	17.2%	1949.6
Base Qwen	32.8%	18.8%	1678
SFT	43.0%	26.6%	1428.4
Process-DPO	43.8%	28.1%	1576.1
Shuffled-DPO	36.7%	22.7%	1694
DPO scrambled features	31.2%	18.0%	1726.9
GRPO	43.8%	28.1%	1536.2
Max-split	44.5%	28.9%	1693.2
Oracle	46.9%	31.2%	1138.7

SFT recovered most of the available same-budget oracle headroom: base 32.8% -> SFT 43.0%. Process-DPO added a small further gain to 43.8%, which is 77.8% of the base-to-oracle headroom. GRPO matched DPO at 43.8% but did not clearly exceed it in the short run.

The shuffled-reward and scrambled-feature controls are important. Shuffled-DPO fell to 36.7%, and feature-scrambled DPO fell to 31.2%. That means the verifier-aligned rewards and displayed candidate-bucket summaries both mattered.

Regime Split

The learned policy helped mainly where the observations carry enough information:

pair_affine_mod: base 62.5%, SFT 79.7%, DPO 81.2%, GRPO 81.2%, oracle 84.4%.
pair_compare_gate: base 3.1%, SFT 6.2%, DPO 6.2%, GRPO 6.2%, oracle 9.4%.

The low-information comparison regime remains bounded by identifiability. Even the same-budget oracle reaches only 9.4% there, because many candidate programs remain hidden-equivalent after three probes.

Interpretation

SFT learns the action interface and makes a large jump over base.
Process-DPO uses the perfect per-step verifier oracle and gives a smaller additional improvement.
GRPO is viable but did not beat DPO in this short run.
Shuffled reward and scrambled feature controls collapse, so the improvement is not just formatting or letter bias.

The hard ceiling is also clear. The controller cannot solve states where three observations do not identify a hidden-correct candidate. That pushes the next step toward joint optimization of policy and observation design, with either larger query budgets or richer probe-generation actions.

Figures

Artifacts

Large LoRA adapters are outside the experiment directory under /workspace/large_artifacts/qwen35_4b_oracle_process_grpo. This directory contains the standalone source, generated datasets, run logs, metrics, figures, and report.

Experiment log 1

Show the running log (1 entry, 2026-06-24)

2026-06-24

Created a standalone verifier-MDP experiment for Qwen3.5-4B process control.
Implemented a typed list[int] -> int operator-pair environment with two output regimes:
- pair_affine_mod: higher-information numeric outputs.
- pair_compare_gate: lower-information symbolic outputs.
Implemented deterministic candidate filtering, same-budget oracle probe selection, max-split heuristic, and deployable model-action evaluation.
Smoke-built a tiny dataset and found zero-signal post-solve states. Updated dataset generation to omit states where all probe actions have identical reward.
Built the full dataset: 480 train records, 128 eval records, 1138 informative train states, 299 informative eval states.
Ran non-model baselines: random, max-split, oracle.
Ran Qwen base action policy.
Trained SFT process policy for 360 optimizer steps.
Evaluated SFT adapter on the full held-out ladder.
Trained process-DPO for 160 optimizer steps from the SFT adapter.
Evaluated process-DPO.
Trained shuffled-reward process-DPO control for 160 optimizer steps from the same SFT adapter.
Evaluated shuffled-reward control.
Trained GRPO for 120 optimizer steps from the SFT adapter.
Evaluated GRPO.
Evaluated feature-scrambled process-DPO to test dependence on candidate-bucket summaries.