Qwen3.5-4B Oracle Probe Synthesis MDP

Finished2026-06-24 → 26imported · line ZEvidence-Conditioned Selection Interpretability and Diagnostics Operator and Skill Inventories Posttraining and Adaptation+2 moreGitHub ↗

The one idea you need

The model plays detective: a rule is hidden, so it runs a few test inputs and watches the outputs, each result ruling out suspects like narrowing options in Guess Who. The catch — researchers curated which test inputs it was even allowed to try.

The question

When a small model uncovers a hidden rule by running test inputs, does it help more to train it harder or to hand it better tests to choose from?

What we found

It's the menu. Just curating which eight test inputs the model chose from raised success from about 43% to 49% — a bigger jump than any training gave. Supervised coaching added a bit more (48% to 51%); preference- and reward-based training added nothing. A flawless picker allowed all 96 candidate tests hit 87%, so the bottleneck is generating good questions, not picking among them.

Why it matters

When a small model gathers information by probing its environment, invest first in the quality of the moves you offer it, not in reward-based training. A better menu of test inputs — and ultimately generating them — beats coaching the picker.

Success from a sharper test menu42.5% → 49.4%hidden rules fully cracked with three tests: random inputs vs curated ones

Success from supervised coaching47.5% → 51.2%base model vs supervised-trained, same curated inputs

Ceiling with a flawless picker over every question86.9%perfect selection across the full 96-question bank

Gain from preference and reward trainingnoneboth at or below supervised: 47.5% and 48.8% vs 51.2%

On this page

Results at a glance
Overview
Report
Experiment log
Figures
Data files
Reproduce
Related

Results at a glance 3

Cracking the hidden rule with three tests: trained models versus every-question search

How to read

Each bar is one strategy; height is the share of hidden rules fully solved after three test inputs, taller is better. Bars run from random questions and a plain model up to trying every one of the 96 possible questions.

Takeaway → Sharper questions lift every bar and supervised training adds a small bump, but the try-everything bar towers far above all the trained models — headroom remains.

Data table

policy	hidden-all success @ budget 3
max-split (random 8)	42.5%
max-split (mined 8)	49.4%
base Qwen	47.5%
SFT	51.2%
SFT (scrambled features)	47.5%
DPO	47.5%
GRPO	48.8%
oracle (mined 8)	61.3%
oracle (full 96-probe pool)	86.9%

Numbers from reports/overall_by_policy_budget.csv (budget=3 rows)

Technical framing

Solving the hidden pair with 3 probes: learned policies vs oracles — Mining better probes lifts every arm; SFT gives a real but modest gain over base, while the full-pool oracle shows a large remaining headroom.

Success climbs as the model is allowed more test inputs

How to read

The horizontal axis is how many test inputs were used, zero to three; the vertical axis is the share of hidden rules fully solved, higher is better. Each line is one strategy, from a plain model up to trying every possible question.

Takeaway → The try-everything line sits above every other from the very first test, so the gap comes from asking better questions, not from later choices.

Data table

probe budget (verifier queries)	max-split (mined 8)	base Qwen (mined 8)	SFT (mined 8)	oracle (mined 8)	oracle (full pool)
0	1.9%	1.9%	1.9%	1.9%	1.9%
1	27.5%	22.5%	25.6%	32.5%	46.2%
2	44.4%	40.6%	43.1%	48.8%	67.5%
3	49.4%	47.5%	51.2%	61.3%	86.9%

Numbers from reports/overall_by_policy_budget.csv

Technical framing

Success vs number of verifier probes used — The full-pool oracle pulls ahead from the very first probe, so the gap is in proposing high-information probes, not in later selection.

Easy versus hard rules: where the room to improve lives

How to read

Bars are grouped by strategy with two colors, one for an easy rule type and one for a hard rule type; height is the share of hidden rules fully solved after three test inputs, higher is better.

Takeaway → On the hard rule type trained models stay near one in twenty, yet trying every question reaches nearly three-quarters — the needed questions exist but current methods rarely find them.

Data table

policy	pair_affine_mod	pair_compare_gate
base Qwen	90%	5%
SFT	96.2%	6.2%
oracle (mined 8)	98.8%	23.8%
oracle (full pool)	100%	73.8%

Numbers from report table (Regime Split section of reports/qwen35_4b_oracle_probe_synthesis_mdp_report.md)

Technical framing

Easy vs hard template: where the headroom lives — pair_compare_gate looks unsolvable to learned policies but reaches 73.8% with full-pool oracle probing — the environment has the needed probes.

In the author’s words from the Report · “Interpretation”

This experiment supports a sharper line-2 thesis: the next leverage is not more operator naming, and not naive RL over the same eight choices. It is probe generation. Exhaustive search can expose high-value observations, and Qwen can learn a modest but real selection improvement over base and max-split once those observations are displayed. However, the full-pool oracle is far ahead of the learned policies, so the main remaining gap is proposing the right high-information probes under deployable constraints. The negative DPO/GRPO result is useful. Preference or on-policy optimization over the same mined-eight actions was not enough; SFT was the robust learned component. … Read the full result →

Overview

This standalone line-2 experiment tests whether Qwen3.5-4B can exploit richer verifier probes. The model does not emit operators. It chooses among eight displayed probe inputs, but those probes are mined from a 96-case bank using target-independent candidate-bucket statistics.

Large model artifacts are intentionally outside this directory:

/workspace/large_artifacts/qwen35_4b_oracle_probe_synthesis_mdp/models/sft_process_lora
/workspace/large_artifacts/qwen35_4b_oracle_probe_synthesis_mdp/models/dpo_process_lora
/workspace/large_artifacts/qwen35_4b_oracle_probe_synthesis_mdp/models/grpo_process_lora

Main outputs:

reports/qwen35_4b_oracle_probe_synthesis_mdp_report.md
reports/figures/
reports/*.csv
reports/eval/*.json
run_logs/
logs/experiment_log.md

Reproduction order:

python scripts/build_dataset.py --train-per-cell 50 --eval-per-cell 20 --states-per-record 3 --query-pool-cases 96 --action-source mined8
Run non-model baselines with scripts/eval_policy.py for random, max_split, oracle, fullpool_max_split, and fullpool_oracle.
python scripts/eval_policy.py --policy base --name base_mined8 --action-source mined8 --max-budget 3
python scripts/train_sft_policy.py --max-steps 260 --batch-size 2 --grad-accum 2
python scripts/train_dpo_policy.py --max-steps 120 --batch-size 2 --grad-accum 2
python scripts/train_grpo_policy.py --max-steps 80 --batch-size 2 --grad-accum 2 --group-size 8
python scripts/eval_policy.py --policy adapter --name sft_scrambled_features --adapter-dir /workspace/large_artifacts/qwen35_4b_oracle_probe_synthesis_mdp/models/sft_process_lora --action-source mined8 --max-budget 3 --scramble-features
python scripts/make_report.py

Report

Rendered from reports/qwen35_4b_oracle_probe_synthesis_mdp_report.md

Question

Can Qwen3.5-4B exploit a richer, deployable probe-generation layer inside a deterministic verifier MDP?

The model still does not name operators. It sees visible executions, the current surviving candidate count, and eight proposed probe inputs. The difference from a fixed small action set is that those eight probes are mined from a 96-case bank by target-independent candidate-bucket statistics. Training labels and rewards then use the verifier oracle to identify which displayed probe actually shrinks the target-retaining candidate set.

Design

Base model: Qwen3.5-4B with 4-bit QLoRA adapters.
Train records: 300; eval records: 160.
Informative train states: 619; informative eval states: 334.
Probe bank: 96 candidate inputs per task; displayed action set: 8 mined probes.
Eval ladder: library sizes 64, 128, 256, 512; templates pair_affine_mod and pair_compare_gate.
Probe budget: 0-3 verifier queries.
Arms: random8 and mined8 controls, full-pool upper bounds, base Qwen, SFT, feature-scrambled SFT, DPO, and GRPO.

Main Result

policy	hidden-all @3	exact pair @3	survivors @3
Max-split random8	42.5%	28.1%	1441.4
Oracle random8	54.4%	40.6%	1199.6
Random mined8	46.9%	32.5%	1033.8
Max-split mined8	49.4%	35.6%	1004.9
Max-split full pool	48.8%	35.0%	1005
Base Qwen mined8	47.5%	33.8%	999.5
SFT mined8	51.2%	37.5%	996.7
SFT scrambled features	47.5%	31.9%	1029.7
DPO mined8	47.5%	35.6%	1060.4
GRPO mined8	48.8%	34.4%	1449.4
Oracle mined8	61.3%	48.1%	657.7
Oracle full pool	86.9%	71.9%	267.1

The action-source result is the largest signal. Moving from random-eight to mined-eight improves max-split from 42.5% to 49.4%, and the same-budget oracle from 54.4% to 61.3%. Scanning the full 96-probe bank with target-aware oracle selection reaches 86.9%, so the earlier low-information ceiling was partly an action-space ceiling, not just an intrinsic task ceiling.

The best learned policy is the SFT warm start: base Qwen 47.5% -> SFT 51.2%. SFT beats target-independent mined max-split and falls when features are scrambled (47.5%), so the learned gain depends on the candidate-bucket summaries. DPO and GRPO did not improve on SFT in this run: DPO reached 47.5% and GRPO reached 48.8%.

Regime Split

pair_affine_mod: base 90.0%, SFT 96.2%, DPO 88.8%, GRPO 91.2%, mined oracle 98.8%, full-pool oracle 100.0%.
pair_compare_gate: base 5.0%, SFT 6.2%, DPO 6.2%, GRPO 6.2%, mined oracle 23.8%, full-pool oracle 73.8%.

The full-pool oracle changes the low-information story: pair_compare_gate rises to 73.8% under target-aware full-pool probing. That means the verifier environment contains useful discriminating probes, but the current deployable mining heuristic and Qwen action policy do not reliably surface or select them.

Interpretation

The negative DPO/GRPO result is useful. Preference or on-policy optimization over the same mined-eight actions was not enough; SFT was the robust learned component. The next step should make the action generator itself trainable or differentiably rankable, rather than only training a selector over the top eight target-independent probes.

Figures

Artifacts

Large LoRA adapters are outside the experiment directory under /workspace/large_artifacts/qwen35_4b_oracle_probe_synthesis_mdp. The experiment directory contains standalone source, generated datasets, run logs, metrics, plots, and this report.

Experiment log 3

Show the running log (3 entries, 2026-06-24)

2026-06-24

Created a standalone verifier-MDP experiment for Qwen3.5-4B probe synthesis and process control.
Implemented the verifier-MDP harness as a standalone experiment package.
Added a 96-case probe bank per task and a deployable mined8 action source that selects eight probes by target-independent candidate-bucket statistics.
Added full-pool non-model baselines to measure the value of scanning the whole probe bank.
Smoke-built a 24-query-pool dataset and confirmed the oracle labels were not position-collapsed.
The first full 96-query build was too slow due repeated operator-output recomputation. Patched the environment to compute operator output tables once per state.
Built the full dataset: 300 train records, 160 eval records, 619 informative train states, 334 informative eval states.
Ran non-model baselines:
- max_split_random8, oracle_random8
- random_mined8, max_split_mined8, oracle_mined8
- fullpool_max_split, fullpool_oracle
Evaluated base Qwen on mined-eight prompts.
Trained SFT for 260 optimizer steps from Qwen3.5-4B.
Evaluated SFT on the full mined-eight eval ladder.
Trained process-DPO for 120 optimizer steps from SFT.
Evaluated process-DPO on the full mined-eight eval ladder.
Trained process-GRPO for 80 optimizer steps from SFT.
Evaluated process-GRPO on the full mined-eight eval ladder.
Evaluated SFT with scrambled displayed features.

Main Readout

Random-eight max-split reached 42.5% hidden-all at budget 3.
Mined-eight max-split reached 49.4%, showing the deployable mined action source improves the baseline action set.
Mined-eight oracle reached 61.3%, while full-pool oracle reached 86.9%, showing major remaining probe-generation headroom.
Base Qwen reached 47.5%; SFT improved to 51.3% and dropped to 47.5% under feature scrambling.
DPO and GRPO did not improve on SFT in this run.

Interpretation

The main line-2 bottleneck is now sharper: the verifier can expose much better observations than the current mined-eight action source reliably surfaces, and Qwen can learn a modest selector improvement once those probes are displayed. The next target should be trainable probe generation or ranking over a larger candidate bank, not more optimization over the same eight actions.