Qwen3.5-4B Adaptive Tool Controller
The one idea you need
Picture a data clerk reshaping spreadsheets, usually answering by eye. One tell flags the hard cases: when her quick answer has fewer columns than the raw data holds, real restructuring is needed, so she runs a scripted tool instead of guessing.
The question
When a small model can answer a table task directly or run a slower program, can one simple rule decide when the program is worth the extra cost?
What we found
Partly. One structural cue — does the direct answer have fewer columns than the raw data implies? — safely flags the reshaping tasks where a program helps, lifting accuracy from 42% to 50% with zero broken tasks. But it captures only half the gain; the full 58% ceiling needs a slightly broader set of tool calls, not one cheap program.
Why it matters
Before wiring an expensive tool loop into a small-model system, look for one cheap structural signal that predicts when the tool actually helps, and route on it. Firing every tool by default costs many times more and gains nothing extra.
On this page
Results at a glance 3
How to read
Bars give exact-match accuracy over 50 test tasks, one per policy: answer directly, the learned one-cue controller, the best fixed shape rule, and two perfect-hindsight oracles. Taller is better; the oracles mark the 58% ceiling.
Takeaway → The learned controller lifts 42% to 50% but stalls halfway; the fixed shape rule and both oracles reach 58%, so one cue is not enough.
Data table
| Policy | exact-match accuracy |
|---|---|
| direct only | 42% |
| learned controller | 50% |
| best fixed shape rule | 58% |
| oracle: tools only when helpful | 58% |
| oracle: full budget, best pick | 58% |
Numbers from experiments/qwen35_4b_adaptive_tool_controller/reports/final_summary.json
Technical framing
Held-out accuracy: learned controller vs direct answering and diagnostics — A one-feature learned controller lifts accuracy 42% to 50% cheaply, but a fixed shape rule reaches the 58% oracle ceiling.
How to read
Each point plots a policy's accuracy against its average tokens per task, the compute cost. One line links the three deployable policies, another the two hindsight oracles. Up and to the left — more accuracy, less compute — is better.
Takeaway → The deployable policies climb cheaply to 58%; the full-budget a perfect chooser sits far to the right at the same 58%, so heavy extra spending buys no accuracy.
Data table
| Avg forward tokens per task | deployable policies | oracle reference policies |
|---|---|---|
| 1.48e+03 | 42% | — |
| 2.97e+03 | 50% | — |
| 3.96e+03 | — | 58% |
| 4.63e+03 | 58% | — |
| 17.8k | — | 58% |
Numbers from experiments/qwen35_4b_adaptive_tool_controller/reports/final_summary.json
Technical framing
Accuracy vs token cost: the controller is a low-cost Pareto point — Direct-only, learned controller, and the fixed shape rule form the deployable frontier; the full-budget oracle costs ~4x more for the same 58%.
How to read
Paired bars per task family, five tasks each: darker is direct answering, lighter is the learned controller. Higher is better; equal-height pairs mean tools changed nothing. Families run from structural splits to already-easy folds.
Takeaway → Both gains come from two split-and-reshape families rising from 0% to 40%; every other family is unchanged, so all gains sit in a narrow structural niche.
Data table
| Task family | direct only | learned controller |
|---|---|---|
| potters_wheel_merge_split | 0% | 40% |
| synthetic_8 | 0% | 40% |
| crime_data_wrangler | 0% | 0% |
| agriculture | 80% | 80% |
| proactive_wrangling_fold | 100% | 100% |
Numbers from experiments/qwen35_4b_adaptive_tool_controller/reports/final_summary.json
Technical framing
Where the controller helps: per-family accuracy, direct vs controller — All controller gains come from two structural-split families; it never hurts families the direct answer already handles.
In the author’s words from the Report · “Summary”
This experiment trains a small offline controller to choose between direct answering and external executable-program tool actions. The controller is selected on pilot train/dev records and frozen before held-out test scoring. Selected controller: Feature: direct_out_cols_less_new_in If true: single_split_fold_unpivot If false: direct_only Candidates passing train gate: 4468
Overview
This standalone experiment evaluates a small adaptive controller over a fixed table-transformation candidate pool. Each task has a direct JSON attempt and five executable-program tool attempts. The controller decides whether to stop with the direct answer or spend additional forward-token budget on tool actions.
Run:
python scripts/eval_tool_controller.py \
--root /workspace/experiments/qwen35_4b_adaptive_tool_controllerOutputs:
reports/final_summary.jsonreports/report.mdreports/figures/*.pngreports/decisions/*.json
Report
Rendered from reports/report.md
Summary
This experiment trains a small offline controller to choose between direct answering and external executable-program tool actions. The controller is selected on pilot train/dev records and frozen before held-out test scoring.
Selected controller:
- Feature:
direct_out_cols_less_new_in - If true:
single_split_fold_unpivot - If false:
direct_only - Candidates passing train gate:
4468
Held-Out Test Result
| Policy | Exact | Accuracy | Tokens | Avg tokens/task | Program commits | Recoveries | Losses | Commit precision |
|---|---|---|---|---|---|---|---|---|
direct_only | 21/50 | 42.0% | 73,911 | 1478 | 0 | 0 | 0 | n/a |
learned_controller | 25/50 | 50.0% | 148,451 | 2969 | 4 | 4 | 0 | 100.0% |
fixed_visible_out_less_cols_full_else_direct | 29/50 | 58.0% | 231,365 | 4627 | 8 | 8 | 0 | 100.0% |
prefix5_first_visible | 28/50 | 56.0% | 890,030 | 17801 | 23 | 8 | 1 | 78.3% |
oracle_budget_full_only_when_helpful | 29/50 | 58.0% | 197,951 | 3959 | 8 | 8 | 0 | 100.0% |
oracle_best_available_full_budget | 29/50 | 58.0% | 890,030 | 17801 | 8 | 8 | 0 | 100.0% |
The learned controller improves over direct answering and becomes a low-cost Pareto point. It does not reach the strongest fixed shape rule included as a diagnostic; the learned policy buys the first four recoveries cheaply, while the fixed shape rule buys the remaining four recoveries with additional tool budget.
Pilot Selection
| Rank | Feature | True action | False action | Train exact | Dev exact | Dev tokens | Dev losses |
|---|---|---|---|---|---|---|---|
| 1 | direct_out_cols_less_new_in | single_split_fold_unpivot | direct_only | 9/30 | 5/10 | 6,821 | 0 |
| 2 | visible_out_less_cols | single_split_fold_unpivot | direct_only | 9/30 | 5/10 | 9,939 | 0 |
| 3 | visible_rows_change | direct_only | single_row_column_rule | 9/30 | 5/10 | 11,339 | 0 |
| 4 | visible_row_expansion_or_col_contraction | single_split_fold_unpivot | direct_only | 9/30 | 5/10 | 11,588 | 0 |
| 5 | direct_out_cols_less_new_in | canary_split_fold_unpivot_disagree_escalate | direct_only | 9/30 | 5/10 | 13,050 | 0 |
| 6 | visible_rows_change | direct_only | canary_split_fold_unpivot_disagree_escalate | 9/30 | 5/10 | 15,872 | 0 |
| 7 | direct_out_cols_less_new_in | canary_row_column_rule_disagree_escalate | direct_only | 9/30 | 5/10 | 16,016 | 0 |
| 8 | visible_out_less_cols | canary_split_fold_unpivot_disagree_escalate | direct_only | 9/30 | 5/10 | 16,168 | 0 |
| 9 | visible_rows_change | direct_only | canary_row_column_rule_disagree_escalate | 9/30 | 5/10 | 17,162 | 0 |
| 10 | visible_row_expansion_or_col_contraction | canary_split_fold_unpivot_disagree_escalate | direct_only | 9/30 | 5/10 | 17,817 | 0 |
| 11 | direct_out_rows_less_new_in | direct_only | canary_split_fold_unpivot_disagree_escalate | 9/30 | 5/10 | 18,114 | 0 |
| 12 | visible_col_expansion_or_row_contraction | direct_only | canary_split_fold_unpivot_disagree_escalate | 9/30 | 5/10 | 18,114 | 0 |
Family Breakdown
| Family | n | Direct | Controller | Recoveries | Losses | Program commits | Tokens |
|---|---|---|---|---|---|---|---|
agriculture | 5 | 4/5 | 4/5 | 0 | 0 | 0 | 5,453 |
crime_data_wrangler | 5 | 0/5 | 0/5 | 0 | 0 | 5 | 66,089 |
potters_wheel_merge_split | 5 | 0/5 | 2/5 | 2 | 0 | 4 | 5,696 |
potters_wheel_unfold | 5 | 2/5 | 2/5 | 0 | 0 | 1 | 3,987 |
potters_wheel_unfold2 | 5 | 0/5 | 0/5 | 0 | 0 | 0 | 1,409 |
proactive_wrangling_fold | 5 | 5/5 | 5/5 | 0 | 0 | 0 | 1,708 |
synthetic_12 | 5 | 5/5 | 5/5 | 0 | 0 | 0 | 11,102 |
synthetic_25 | 5 | 4/5 | 4/5 | 0 | 0 | 0 | 9,942 |
synthetic_48 | 5 | 1/5 | 1/5 | 0 | 0 | 0 | 2,242 |
synthetic_8 | 5 | 0/5 | 2/5 | 2 | 0 | 5 | 40,823 |
Pareto Frontier
direct_only: 21/50 (42.0%), 73,911 tokenslearned_controller: 25/50 (50.0%), 148,451 tokensfixed_visible_out_less_cols_full_else_direct: 29/50 (58.0%), 231,365 tokens
Figures
Interpretation
The controller confirms that external program tools are valuable only on a narrow structural subset. A one-feature policy selected from pilot data recovers a useful low-cost slice without paying full portfolio cost on every task. The stronger fixed shape diagnostic shows that the remaining recoveries require broader portfolio calls, not just a cheaper single-tool action.
This is an orchestration result, not a generation-capability result. The controller changes how budget is allocated over already-generated tool candidates; it does not create new candidates outside the recorded pool.
Limitations
- Offline evaluation over a fixed candidate pool; no fresh model generations are produced in this package.
- The pilot split is small, so feature selection is unstable. The fixed shape-rule diagnostic should be validated across more held-out family splits.
- The depth-1 controller is intentionally simple. A richer sequential controller should be trained only after this signal replicates under regenerated candidates.
Experiment log 1
Show the running log (1 entry, 2026-06-28)
2026-06-28
- Created standalone adaptive tool-controller package.
- Copied fixed candidate-pool and task records into
data/. - Planned an offline learned-controller evaluation with pilot train/dev policy selection and held-out test scoring.
Figures 4
Data files 1
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
Reproduce
Run steps are documented inside the experiment folder (README and scripts).