Qwen3.5-4B Deployable Information Ceiling Sweep
The one idea you need
Picture a deduction game where you identify a hidden rule by choosing which inputs to test and reading the outputs. A plain strategy always runs the test that most evenly splits the still-possible rules, racing against a rival who secretly already knows the answer.
The question
When a small system can't pin down a hidden rule, is that because its questioning strategy is too weak, or because it simply hasn't gathered enough clues?
What we found
It's the clues. On the hard hidden rule, the plain strategy got just 4% right when starved of tests and clues, far below the 74% of a rival who secretly knew the answer. But allowing more tests lifted it to 86%, and more starting clues to 91% - nearly matching the cheater. The strategy was already near-best; information was the wall.
Why it matters
When a small model fails a task that needs probing or tool calls, don't just hunt for a cleverer selection policy. First check whether it is gathering enough observations - more query budget or evidence may close most of the gap.
On this page
Results at a glance 3
How to read
Grouped bars set the plain strategy against a rival who already knows the answer, across three setups from hardest to easier. Bar height is how often the hidden rule is correctly identified; taller is better.
Takeaway → In the hardest leftmost setup the plain strategy's bar is tiny beside the cheat's, but in the other two they stand almost level - more clues or tests closes the gap.
Data table
| information regime | greedy (deployable) | target-aware oracle |
|---|---|---|
| 4 visible, budget 3 | 3.8% | 73.8% |
| 4 visible, budget 10 | 86.2% | 98.8% |
| 16 visible, budget 3 | 91.2% | 97.5% |
Numbers from report table (reports/qwen35_4b_deployable_information_ceiling_sweep_report.md)
Technical framing
Compare-gate: deployable greedy probing vs target-aware oracle — More budget or more visible observations closes most of the 70-point oracle gap: the bottleneck is information volume, not policy.
How to read
Two lines track how often the hidden rule is correctly identified as the number of allowed tests rises left to right, with starting clues held fixed. One line is the plain strategy, the other the answer-knowing rival; higher is better.
Takeaway → The plain strategy's line climbs steadily to the top while the rival's flattens early, so extra tests keep helping the plain strategy long after the cheat has maxed out.
Data table
| active probe budget | greedy (deployable) | target-aware oracle |
|---|---|---|
| 0 | 0% | 0% |
| 1 | 0% | 7.5% |
| 2 | 2.5% | 35% |
| 3 | 3.8% | 73.8% |
| 4 | 15% | 92.5% |
| 5 | 33.8% | 97.5% |
| 6 | 48.8% | 98.8% |
| 7 | 58.8% | 98.8% |
| 8 | 72.5% | 98.8% |
| 9 | 78.8% | 98.8% |
| 10 | 86.2% | 98.8% |
Numbers from reports/summary_by_template.csv (pair_compare_gate, visible_total=4, selected_hidden_all)
Technical framing
Compare-gate accuracy vs probe budget (4 visible observations) — In the hardest regime greedy climbs steadily with budget, from 0% to 86%; the oracle saturates by budget 5.
How to read
Two lines show identification accuracy as the number of clues shown up front increases left to right, with the number of tests held low. One line is the plain strategy, the other the answer-knowing rival; higher is better.
Takeaway → The plain strategy's line rises steeply and converges toward the rival's, so each extra batch of clues shrinks the gap between them from wide to nearly nothing.
Data table
| initial visible observations | greedy (deployable) | target-aware oracle |
|---|---|---|
| 4 | 3.8% | 73.8% |
| 8 | 46.2% | 93.8% |
| 12 | 73.8% | 95% |
| 16 | 91.2% | 97.5% |
Numbers from report table (Budget-3 Summary, pair_compare_gate rows)
Technical framing
Compare-gate accuracy vs visible observations (budget 3) — Holding budget at 3, each extra batch of visible observations lifts greedy sharply and shrinks the oracle gap from 70 to 6 points.
In the author’s words from the Report · “Interpretation”
The greedy policy is the deployable one-step Bayesian experiment-design rule for a uniform posterior over all candidates consistent with observed executions. If additional budget or additional visible observations lift greedy performance, the bottleneck is information volume. If the target-aware oracle remains far above greedy, that gap should be read as target-knowledge headroom rather than directly recoverable deployable headroom.
Overview
This standalone diagnostic asks whether the hard low-information regime is limited by deployable information or by a trainable probe-selection policy.
There is no model training in this package. The deployable policy is greedy max expected information gain under a uniform posterior over all verifier-surviving candidates. The comparison policy is a target-aware oracle and is only a headroom measurement.
Main outputs:
reports/qwen35_4b_deployable_information_ceiling_sweep_report.mdreports/figures/reports/*.csvreports/eval/*.jsonlogs/experiment_log.md
Reproduction:
python scripts/eval_information_sweep.py --max-budget 10 --visible-extra 0 4 8 12
python scripts/make_report.pyReport
Rendered from reports/qwen35_4b_deployable_information_ceiling_sweep_report.md
Objective
This standalone diagnostic measures whether the hard low-information regime is limited by deployable information or by a trainable probe-selection policy. It uses no model training. The deployable policy is greedy max expected information gain under a uniform posterior over surviving verifier candidates. The oracle policy is target-aware and is included only as non-deployable headroom.
Key Checks
- With four visible observations and budget 3, compare-gate greedy accuracy is 3.8% versus 73.8% for the target-aware oracle.
- Keeping four visible observations but raising active budget to 10, compare-gate greedy accuracy is 86.2% versus 98.8% for the target-aware oracle.
- Raising initial visible observations to sixteen while keeping budget 3 gives compare-gate greedy accuracy 91.2% versus 97.5% for the target-aware oracle.
Interpretation
The greedy policy is the deployable one-step Bayesian experiment-design rule for a uniform posterior over all candidates consistent with observed executions. If additional budget or additional visible observations lift greedy performance, the bottleneck is information volume. If the target-aware oracle remains far above greedy, that gap should be read as target-knowledge headroom rather than directly recoverable deployable headroom.
Budget-3 Summary
| Template | Visible observations | Greedy | Target-aware oracle | Gap points | Greedy candidates left | Greedy hidden-equivalent left |
|---|---|---|---|---|---|---|
| pair_affine_mod | 4 | 93.8% | 100.0% | 6.2 | 4.6 | 4.4 |
| pair_affine_mod | 8 | 100.0% | 100.0% | 0.0 | 4.3 | 4.3 |
| pair_affine_mod | 12 | 100.0% | 100.0% | 0.0 | 4.3 | 4.3 |
| pair_affine_mod | 16 | 100.0% | 100.0% | 0.0 | 4.3 | 4.3 |
| pair_compare_gate | 4 | 3.8% | 73.8% | 70.0 | 2005.4 | 293.8 |
| pair_compare_gate | 8 | 46.2% | 93.8% | 47.5 | 1162.6 | 263.0 |
| pair_compare_gate | 12 | 73.8% | 95.0% | 21.2 | 1256.9 | 248.7 |
| pair_compare_gate | 16 | 91.2% | 97.5% | 6.2 | 314.1 | 234.2 |
Budget-10 Summary
| Template | Visible observations | Greedy | Target-aware oracle | Gap points | Greedy candidates left | Greedy hidden-equivalent left |
|---|---|---|---|---|---|---|
| pair_affine_mod | 4 | 100.0% | 100.0% | 0.0 | 4.3 | 4.3 |
| pair_affine_mod | 8 | 100.0% | 100.0% | 0.0 | 4.3 | 4.3 |
| pair_affine_mod | 12 | 100.0% | 100.0% | 0.0 | 4.3 | 4.3 |
| pair_affine_mod | 16 | 100.0% | 100.0% | 0.0 | 4.3 | 4.3 |
| pair_compare_gate | 4 | 86.2% | 98.8% | 12.5 | 707.4 | 244.8 |
| pair_compare_gate | 8 | 95.0% | 98.8% | 3.8 | 760.6 | 229.3 |
| pair_compare_gate | 12 | 96.2% | 98.8% | 2.5 | 529.1 | 222.8 |
| pair_compare_gate | 16 | 98.8% | 98.8% | 0.0 | 246.4 | 213.2 |
Figures
reports/figures/budget_curve_pair_affine_mod.pngreports/figures/budget_curve_pair_compare_gate.pngreports/figures/oracle_gap_pair_affine_mod.pngreports/figures/oracle_gap_pair_compare_gate.png
Reproduction
python scripts/eval_information_sweep.py --max-budget 10 --visible-extra 0 4 8 12
python scripts/make_report.pyExperiment log 4
Show the running log (4 entries)
Setup
- Created a standalone diagnostic directory at
/workspace/experiments/qwen35_4b_deployable_information_ceiling_sweep. - Packaged 160 evaluation records covering library sizes 64, 128, 256, and 512 across
pair_affine_modandpair_compare_gate. - No model training is used.
Hypothesis
If the low-information regime is information-limited, then more active probes or more initial observations should raise the deployable greedy uniform-posterior policy. If the target-aware oracle remains far above the deployable policy, that residual gap should be treated as target-knowledge headroom rather than directly recoverable deployable performance.
Planned Sweep
Policies:
greedy_uniform_split: choose the full-pool probe with minimum expected survivors under a uniform posterior over verifier-surviving candidates.target_aware_oracle: choose the full-pool probe that minimizes survivors for the actual hidden target.
- Visible observations: 4, 8, 12, and 16 total.
- Active probe budgets: 0 through 10.
Results
- Completed the full sweep over 160 records.
pair_affine_modis solved by the deployable greedy policy once either budget or visible observations are modestly increased.pair_compare_gateis not intrinsically stuck:- 4 visible, budget 3: greedy 3.8%, oracle 73.8%.
- 4 visible, budget 10: greedy 86.2%, oracle 98.8%.
- 8 visible, budget 3: greedy 46.2%, oracle 93.8%.
- 12 visible, budget 3: greedy 73.8%, oracle 95.0%.
- 16 visible, budget 3: greedy 91.2%, oracle 97.5%.
- 16 visible, budget 10: greedy 98.8%, oracle 98.8%.
- Interpretation: the low-information failure at budget 3 is primarily an observation-budget problem. The target-aware oracle gap at four visible cases is not directly trainable selector headroom, but the deployable policy catches up when given enough observations.
Figures 4
Data files 3
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
reports/summary_by_cell.csv50 kBreports/summary_by_template.csv13 kBreports/summary_overall.csv5.6 kB
Reproduce
Run steps are documented inside the experiment folder (README and scripts).