Qwen3.5-4B Foofah Adaptive Program Budget Router
The one idea you need
Picture a returns desk that opens a package for full inspection only when the box comes back lighter than it shipped, a weight drop hinting something inside went missing. Here the quick answer having fewer columns than the input table is that lighter box, triggering the slow repair programs.
The question
Can a cheap rule decide when a small model should run its slow, expensive extra program attempts on a table task, instead of always running them?
What we found
Yes, and here is the twist: running all five programs on every task scored lower (56%) than the cheap rule (58%), because blanket spending overwrote one answer the quick pass already had right. The rule fires only when the quick answer has fewer columns than the input, flagging exactly the hard cases, beating the quick answer's 42% and tying a perfect-hindsight ceiling at half the compute.
Why it matters
Before training a learned judge, check whether your expensive fallback's wins cluster behind one cheap, visible signal; a hand-written trigger may match the best achievable accuracy. And beware: running every tool everywhere can clobber correct answers, not just waste compute.
On this page
Results at a glance 7
How to read
Bars show exactly-correct accuracy on 50 tasks for four policies: quick answer only (42%), running all five programs everywhere (56%), the cheap shape rule (58%), and a perfect-hindsight policy that runs programs only when they help (58%). Taller is better.
Takeaway → The cheap rule reaches the perfect-hindsight ceiling and edges past run-everything, while the quick answer trails far below, so selective spending beats blanket spending.
Data table
| policy | test accuracy (50 tasks) |
|---|---|
| direct JSON only | 42% |
| full fixed portfolio (5 programs) | 56% |
| adaptive shape router | 58% |
| oracle (run programs only when helpful) | 58% |
Numbers from reports/report.md table; verified against reports/decisions/*.json
Technical framing
Adaptive shape router matches the oracle ceiling at a fraction of the fixed portfolio's cost — The pilot-selected router (run programs when direct output loses columns) hits the 58% oracle ceiling; always-run-everything gets 56%.
How to read
Bars show average tokens per task for the same four policies: quick answer cheapest at 1,478, run-everything priciest at 17,801, the shape rule in between at 9,017, perfect-hindsight lowest among spenders at 3,959. Shorter is better.
Takeaway → The rule costs about half of run-everything yet scores higher, and the perfect-hindsight bar shows even more compute could still be shaved.
Data table
| policy | avg tokens per task |
|---|---|
| direct JSON only | 1478 |
| full fixed portfolio (5 programs) | 17.8k |
| adaptive shape router | 9017 |
| oracle (run programs only when helpful) | 3959 |
Numbers from reports/report.md table
Technical framing
Token cost per task for the same policies — The adaptive router spends about half the tokens of the fixed portfolio while beating its accuracy; the oracle shows further headroom.
How to read
A rising line tracks accuracy as 0 to 5 programs are forced on every task, climbing from 42% to 56%; a flat reference line marks the shape rule at 58%. Higher is better.
Takeaway → The always-run line climbs then stalls at 56%, always under the rule's 58%, so spending the whole budget blindly cannot match spending it selectively.
Data table
| Number of program strategies always run | fixed budget (first-visible commit) | adaptive shape router |
|---|---|---|
| 0 | 42% | 58% |
| 1 | 50% | — |
| 2 | 52% | — |
| 3 | 54% | — |
| 4 | 54% | — |
| 5 | 56% | 58% |
Technical framing
Fixed program budgets: accuracy saturates below the adaptive router — Spending the whole 5-program budget on every task never reaches the 58% the adaptive router gets by spending it selectively.
How to read
Paired bars per task family (five tasks each) compare quick answer against the shape rule across ten families. Higher is better. Two families jump from 0 to 80%; every other family is unchanged.
Takeaway → All eight rescues cluster in the merge/split and one synthetic family, and the rule never dents a family the quick answer already solved.
Data table
| task family (5 tasks each) | direct JSON only | adaptive shape router |
|---|---|---|
| agriculture | 80% | 80% |
| crime_data_wrangler | 0% | 0% |
| potters_wheel_merge_split | 0% | 80% |
| potters_wheel_unfold | 40% | 40% |
| potters_wheel_unfold2 | 0% | 0% |
| proactive_wrangling_fold | 100% | 100% |
| synthetic_12 | 100% | 100% |
| synthetic_25 | 80% | 80% |
| synthetic_48 | 20% | 20% |
| synthetic_8 | 0% | 80% |
Numbers from reports/report.md family breakdown table
Technical framing
Where the router's gains come from: per-family test accuracy — All eight recoveries are concentrated in two families (merge/split and synthetic_8); the router never hurts a family the direct answer already solves.
How to read
Paired bars per program strategy compare running that strategy alone against a version that escalates only when its answer disagrees with the quick answer. Higher is better; all bars sit between 42% and 50%.
Takeaway → Escalating after one program adds at most two points, and no single-strategy approach reaches the shape rule's 58%.
Data table
| Program strategy | Single strategy only | Canary: escalate on disagreement |
|---|---|---|
| verified_structural | 50% | 50% |
| cell_parser | 42% | 42% |
| row_column_rule | 50% | 50% |
| header_aware | 42% | 44% |
| split_fold_unpivot | 50% | 50% |
Numbers from reports/final_summary.json (test_metrics: single_* and canary_*_disagree_escalate accuracy)
Technical framing
Paying for one program strategy first (canary) barely beats running it outright — Canary escalation adds at most 2pp over the lone strategy and every variant stays below the shape router's 58%.
How to read
Grouped bars at three portfolio sizes compare committing to the first plausible program against requiring two or three programs to agree. Higher is better; the agreement bars are shortest.
Takeaway → Demanding agreement drops accuracy 8 to 12 points versus trusting the first plausible program, so consensus voting throws away correct lone answers.
Data table
| Fixed portfolio budget | First visible-consistent commit | Consensus of 2 | Consensus of 3 |
|---|---|---|---|
| 3-strategy prefix | 54% | 46% | 42% |
| 4-strategy prefix | 54% | 46% | 44% |
| 5-strategy prefix | 56% | 48% | 48% |
Numbers from reports/final_summary.json (test_metrics: prefix*_first_visible, prefix*_consensus2, prefix*_consensus3 accuracy)
Technical framing
Requiring consensus among program candidates hurts accuracy at every budget — Demanding agreement between program outputs drops accuracy by 8-12pp versus committing to the first visible-consistent program.
How to read
Bars show total tokens over 50 tasks for six trigger rules, from the tightest fewer-columns rule (231,365) to always-run (890,030). Shorter is better; the five selective rules all score 29 of 50, or 58%.
Takeaway → The five selective triggers tie at 58% accuracy while the loosest costs about three and a half times the tightest, so trigger precision is what saves compute.
Data table
| Shape trigger rule | Total forward tokens (50 tasks) |
|---|---|
| visible out has fewer cols | 231.4k |
| direct out cols < new input | 450.9k |
| fewer cols or direct mismatch | 516.5k |
| visible cols change | 699.2k |
| any visible shape change | 814.1k |
| always run portfolio | 89k |
Numbers from reports/final_summary.json (test_metrics: shape_* total_forward_tokens)
Technical framing
Looser shape triggers reach the same 29/50 accuracy at up to 3.5x the token bill — The five selective triggers all score 29/50; only 'always' slips to 28/50 while paying the full 890k-token portfolio cost.
In the author’s words from the Report · “Summary”
This experiment evaluates whether a cheap deployable router can decide when to spend the executable-program portfolio budget for Foofah-style table transformations. The router is selected only on pilot data and then frozen for test. Pilot-selected router: direct_out_cols_less_new_in. It runs when the direct JSON output has fewer columns than the new input table; otherwise it returns the direct JSON output.
Overview
This standalone package evaluates adaptive budget policies over a fixed Foofah-style table-transformation candidate pool. Each task has one direct JSON completion and five executable-program strategy completions. The experiment asks whether a cheap deployable router can decide when the expensive program portfolio is worth running.
Primary comparison:
- Direct JSON only.
- Fixed program budgets: prefixes of the five-strategy portfolio.
- Adaptive public-shape router selected on pilot data and frozen on test.
- Canary routers that pay for one program strategy before deciding whether to escalate.
- Nondeployable oracle diagnostics.
Run:
python scripts/eval_adaptive_router.py \
--root /workspace/experiments/qwen35_4b_foofah_adaptive_program_budget_routerOutputs are written under reports/.
Report
Rendered from reports/report.md
Summary
This experiment evaluates whether a cheap deployable router can decide when to spend the executable-program portfolio budget for Foofah-style table transformations. The router is selected only on pilot data and then frozen for test.
Pilot-selected router: direct_out_cols_less_new_in. It runs when the direct JSON output has fewer columns than the new input table; otherwise it returns the direct JSON output.
Headline Test Result
| Policy | Exact | Accuracy | Tokens | Avg tokens/task | Program commits | Recoveries | Losses | Commit precision |
|---|---|---|---|---|---|---|---|---|
direct_only | 21/50 | 42.0% | 73,911 | 1478 | 0 | 0 | 0 | n/a |
shape_direct_out_cols_less_new_in_all_first_visible | 29/50 | 58.0% | 450,853 | 9017 | 8 | 8 | 0 | 100.0% |
shape_visible_out_less_cols_all_first_visible | 29/50 | 58.0% | 231,365 | 4627 | 8 | 8 | 0 | 100.0% |
prefix5_first_visible | 28/50 | 56.0% | 890,030 | 17801 | 23 | 8 | 1 | 78.3% |
oracle_budget_run_all_only_on_helpful_tasks | 29/50 | 58.0% | 197,951 | 3959 | 8 | 8 | 0 | 100.0% |
oracle_best_available_full_budget | 29/50 | 58.0% | 890,030 | 17801 | 8 | 8 | 0 | 100.0% |
The pilot-selected adaptive router matches the hidden oracle union accuracy on this test set while using fewer tokens than the full fixed portfolio. It recovers eight direct misses with no direct-correct losses.
The best observed deployable diagnostic is shape_visible_out_less_cols_all_first_visible at 29/50 (58.0%) with 231,365 tokens. Treat this as a test-set diagnostic, not the preselected primary router.
Pilot Selection
Variant order selected on pilot train: verified_structural, cell_parser, row_column_rule, header_aware, split_fold_unpivot. Shape trigger selected on pilot dev: direct_out_cols_less_new_in.
| Trigger | Pilot train exact | Pilot dev exact | Pilot dev tokens | Pilot dev losses |
|---|---|---|---|---|
direct_out_cols_less_new_in | 8/30 | 5/10 | 21,632 | 0 |
visible_out_less_cols | 9/30 | 5/10 | 34,887 | 0 |
visible_less_cols_or_direct_mismatch | 9/30 | 5/10 | 48,440 | 0 |
visible_cols_change | 11/30 | 5/10 | 58,685 | 0 |
visible_cols_change_or_direct_mismatch | 11/30 | 5/10 | 58,685 | 0 |
never | 9/30 | 4/10 | 5,235 | 0 |
direct_parse_fail | 9/30 | 4/10 | 5,235 | 0 |
visible_out_more_rows | 9/30 | 4/10 | 16,737 | 0 |
Test Family Breakdown
| Family | n | Direct | Adaptive | Recoveries | Losses | Program commits | Tokens |
|---|---|---|---|---|---|---|---|
agriculture | 5 | 4/5 | 4/5 | 0 | 0 | 0 | 5,453 |
crime_data_wrangler | 5 | 0/5 | 0/5 | 0 | 0 | 0 | 233,360 |
potters_wheel_merge_split | 5 | 0/5 | 4/5 | 4 | 0 | 4 | 21,126 |
potters_wheel_unfold | 5 | 2/5 | 2/5 | 0 | 0 | 0 | 12,913 |
potters_wheel_unfold2 | 5 | 0/5 | 0/5 | 0 | 0 | 0 | 1,409 |
proactive_wrangling_fold | 5 | 5/5 | 5/5 | 0 | 0 | 0 | 1,708 |
synthetic_12 | 5 | 5/5 | 5/5 | 0 | 0 | 0 | 11,102 |
synthetic_25 | 5 | 4/5 | 4/5 | 0 | 0 | 0 | 9,942 |
synthetic_48 | 5 | 1/5 | 1/5 | 0 | 0 | 0 | 2,242 |
synthetic_8 | 5 | 0/5 | 4/5 | 4 | 0 | 4 | 151,598 |
Pareto Readout
Deployable policies on the accuracy/token Pareto frontier:
shape_visible_out_less_cols_all_first_visible: 29/50 (58.0%), 231,365 tokensprefix1_first_visible: 25/50 (50.0%), 214,562 tokensdirect_only: 21/50 (42.0%), 73,911 tokens
Figures
Interpretation
The experiment finds a simple, deployable budget rule rather than a learned judge. Public and direct-output table-shape signals identify the cases where the executable-program portfolio is worth its cost. On the test set, the pilot-selected rule triggers on 15 tasks, commits a program on eight of them, and captures every direct-miss recovery available to the fixed portfolio without taking the fixed portfolio's one direct-correct loss.
The result is an efficiency win, not a claim that the candidate pool contains more hidden-correct outputs than the full portfolio. The nondeployable oracle diagnostics show the available ceiling in this recorded pool. The adaptive router reaches that ceiling because the useful program cases are concentrated in a public structural signature.
Limitations
- The policy is selected from a small pilot split and evaluated on 50 test tasks; the trigger should be re-run across additional family splits.
- This is an offline router over a recorded candidate pool. It charges only the candidates each policy would generate, but it does not regenerate model outputs.
- The router relies on visible table-shape structure; it may not transfer to transformations where useful program candidates are not aligned with column contraction.
Experiment log 1
Show the running log (1 entry, 2026-06-28)
2026-06-28
- Created standalone package with copied Foofah candidate-pool records and task cases.
- Planned an offline adaptive-budget router evaluation over direct JSON outputs plus five executable-program strategies.
- Primary policy selection is restricted to pilot data. Test labels are reserved for final evaluation.
Figures 4
Data files 1
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
Reproduce
Run steps are documented inside the experiment folder (README and scripts).