Qwen3.5-4B Live Tool DAgger

Finished2026-06-28imported · line ZPosttraining and Adaptation Process Control and Tool Use Structured Execution and Compilers GitHub ↗

The one idea you need

Picture a clerk reshaping a messy spreadsheet with one example row already solved. The clerk both eyeballs an answer and writes a formula, tests that formula against the visible row, then a supervisor decides which answer to hand in — never seeing the final grade.

The question

If a small model can both guess a table's new layout and write code to compute it, does learning when to trust the code make it more accurate?

What we found

Only its cost, not its accuracy. Answering directly solved none of twelve unseen tasks; letting the model write and run code recovered two — about one in six — and spoiled nothing it already had right. But even a flawless, all-knowing chooser topped out at those same two: the ceiling is the model's raw skill, not the decision. The trained controller's real win was reaching that ceiling with half the code runs and zero wasted repairs.

Why it matters

Before engineering a policy for when your model reaches for tools, measure the ceiling with a perfect-hindsight chooser. If it only ties a cheap rule, your bottleneck is tool capability — spend the policy on cutting wasted tool calls, not accuracy.

Direct answers alone0 of 12unseen tasks solved without touching tools

With a code tool added0 of 12 → 2 of 12tools recovered two cases, spoiled none

Cost to reach that same score6 runs, 0 repairs vs 12 runs, 18 repairstrained controller versus always-run-code

Programs it chose to commit100% correctthe controller never submitted a wrong program

On this page

Results at a glance
Overview
Report
Experiment log
Figures
Data files
Reproduce
Related

Results at a glance 3

Unseen-task accuracy for six ways of choosing tools

How to read

Each bar is one strategy's share of twelve unseen table tasks solved exactly; taller is better. The six: answer directly, always run code, a learned rule, a trained controller, a scrambled-label control, and a perfect-hindsight chooser.

Takeaway → Direct-only and the scrambled control sit at zero; every genuine code-using strategy, even the perfect chooser, ties at one in six, so decision quality never lifts the ceiling.

Data table

Policy	test accuracy
direct only	0%
always tool	16.7%
learned rule	16.7%
LoRA policy	16.7%
shuffled-label LoRA	0%
oracle actions	16.7%

Numbers from reports/final_summary.json (policy_results.*.test.accuracy)

Technical framing

Held-out test accuracy by controller policy — Tool-using controllers recover 2/12 held-out cases that direct answering misses; the shuffled-label control collapses to 0.

Practice-set versus unseen-task accuracy for each strategy

How to read

Paired bars per strategy: the lighter bar is the practice set, the darker the harder unseen families; height is share solved exactly, taller better. Same six strategies, from direct-only to the perfect-hindsight chooser.

Takeaway → Every strategy falls sharply from practice to unseen; even the perfect chooser drops from about 58% to one in six, so the unseen families are simply harder.

Data table

Policy	dev	test
direct only	25%	0%
always tool	58.3%	16.7%
learned rule	58.3%	16.7%
LoRA policy	33.3%	16.7%
shuffled-label LoRA	25%	0%
oracle actions	58.3%	16.7%

Numbers from reports/final_summary.json (policy_results.*.dev/test.accuracy)

Technical framing

Dev vs held-out test accuracy: the generalization gap — Every policy drops sharply from dev to the harder held-out families; even oracle action choice caps at 2/12 on test.

Compute spent to reach the same score, by strategy

How to read

Five strategies that all solve two of twelve unseen tasks. For each, one bar counts programs written and run, the other counts repair attempts across twelve cases; lower is better since scores are equal.

Takeaway → Always-run burns twelve program runs plus eighteen repairs; the trained controller needs six runs and zero repairs, the perfect chooser just two — routing cuts compute, not accuracy.

Data table

Policy	program generations	repair actions
always tool	12	18
visible-disagree rule	12	18
learned rule	12	0
LoRA policy	6	0
oracle actions	2	0

Numbers from reports/final_summary.json (policy_results.*.test program_generations, repair_actions)

Technical framing

Tool budget on held-out test: same accuracy, very different cost — All these policies score 2/12, but the LoRA controller gets there with 6 generations and no repairs vs 12+18 for always-tool.

In the author’s words from the Report · “Summary”

This standalone experiment generates fresh tool-environment traces and trains/evaluates a sequential controller over visible tool state. The controller chooses among DIRECT, WRITE, FIX, and PROGRAM. It is evaluated by simulating those actions on the freshly generated traces.

Overview

Standalone live tool-state DAgger-style pilot.

The experiment generates fresh Qwen3.5-4B traces for a table transformation environment:

produce a direct JSON answer,
write an executable transform(table) program,
run the program on the public example,
repair when the public example fails,
decide whether to commit the direct output or the program output.

It derives oracle action labels from held-out correctness for training/evaluation analysis, but deployed policy inputs contain only visible tool state.

Run smoke:

python scripts/run_live_tool_dagger.py \
  --root /workspace/experiments/qwen35_4b_live_tool_dagger \
  --generate-traces \
  --limit-total 6 \
  --max-repairs 1

Run full pilot:

python scripts/run_live_tool_dagger.py \
  --root /workspace/experiments/qwen35_4b_live_tool_dagger \
  --generate-traces \
  --train-lora \
  --train-shuffled-lora \
  --max-repairs 2 \
  --max-steps 60 \
  --resume

Outputs are written under reports/.

Report

Rendered from reports/report.md

Summary

This standalone experiment generates fresh tool-environment traces and trains/evaluates a sequential controller over visible tool state.

The controller chooses among DIRECT, WRITE, FIX, and PROGRAM. It is evaluated by simulating those actions on the freshly generated traces.

Split And Trace Counts

dev: 12 records, direct 3, program 5, oracle union 7, families: potters_wheel_merge_split, proactive_wrangling_fold, synthetic_22, synthetic_51
test: 12 records, direct 0, program 2, oracle union 2, families: potters_wheel_divide, reshape_table_structure_data_wrangler, synthetic_13, synthetic_45
train: 36 records, direct 6, program 5, oracle union 10, families: agriculture, craigslist_data_wrangler, crime_data_wrangler, potters_wheel_fold, potters_wheel_fold_2, potters_wheel_split_fold, potters_wheel_unfold, potters_wheel_unfold2, proactive_wrangling_complex, synthetic_11, synthetic_5, synthetic_8

Held-Out Test Result

Policy	Exact	Accuracy	Program gens	Repairs	Program commits	Recoveries	Program precision
`direct_only`	0/12	0.0%	0	0	0	0	n/a
`always_tool_visible`	2/12	16.7%	12	18	3	2	66.7%
`visible_disagree_rule`	2/12	16.7%	12	18	2	2	100.0%
`learned_rule`	2/12	16.7%	12	0	2	2	100.0%
`lora_seq_policy`	2/12	16.7%	6	0	2	2	100.0%
`lora_shuffled_seq`	0/12	0.0%	0	0	0	0	n/a
`oracle_seq`	2/12	16.7%	2	0	2	2	100.0%

Gate Verdict

The best deployable rule reached 2/12 with 2 recoveries and 0 losses. The sequential LoRA policy reached 2/12 with 2 recoveries and 0 losses. The shuffled-label control reached 0/12, providing the label-noise control for the LoRA arm.

Learned Rule

{
  "dev": {
    "accuracy": 0.5833333333333334,
    "direct_correct_losses": 0,
    "direct_miss_recoveries": 4,
    "exact": 7,
    "n": 12,
    "policy": "rule",
    "program_commits": 4,
    "program_correct": 4,
    "program_generations": 12,
    "program_precision": 1.0,
    "repair_actions": 0,
    "split": "dev"
  },
  "false": "WRITE",
  "feature": "program_disagrees_direct",
  "true": "PROGRAM"
}

Figures

Accuracy

Recoveries and losses

Tool budget

Limitations

This is a balanced pilot split, not a full benchmark run.
The policy is evaluated on fresh precomputed traces; WRITE and FIX reveal the corresponding fresh generated tool outputs from those traces.
Hidden labels are used only for oracle labels and evaluation, not in policy state.

Experiment log 1

Show the running log (1 entry, 2026-06-28)

2026-06-28

Created standalone live tool-state controller package.
Copied source cases and sandbox utilities into the package.
Configured a family-disjoint balanced pilot split with train/dev/test families and fresh trace generation.
Ran a 9-case smoke pass with one repair round to exercise trace generation, safe execution, report writing, and chart writing.
Ran an initial full pass, but its held-out split had no oracle headroom on test, so it was retained only as a diagnostic artifact and not used for the final readout.
Tightened split construction to seed recovery-positive families across train/dev/test before filling neutral families.
Ran the corrected split2 pilot with 60 fresh traces: 36 train, 12 dev, 12 test.
Final held-out result on split2: direct-only 0/12, learned rule 2/12, sequential LoRA 2/12, shuffled-label LoRA 0/12, oracle 2/12.
The sequential LoRA matched the oracle on the held-out split while using 6 program generations, versus 12 for the best fixed/rule policies and 2 for the oracle.

Figures 3

Data files 1

Result tables and metrics copied from the experiment folder — preview inline or open the raw file.

reports/final_summary.json26 kB

Reproduce

Run steps are documented inside the experiment folder (README and scripts).

Browse the experiment folder on GitHub ↗

Qwen3.5-4B Live Tool DAgger

Results at a glance 3

Overview

Report

Summary

Split And Trace Counts

Held-Out Test Result

Gate Verdict

Learned Rule

Figures

Limitations

Experiment log 1

2026-06-28

Figures 3

Data files 1

Reproduce

Related

Cited by claims

Related experiments