Qwen3.5-4B Tool State Policy LoRA

Finished2026-06-28imported · line ZPosttraining and Adaptation Process Control and Tool Use Structured Execution and Compilers GitHub ↗

The one idea you need

For each table puzzle the model gives two answers: a quick guess, and a small program it writes and test-runs on one example whose answer is known. A referee commits just one, trusting the program only when it passes that test and disagrees with the guess.

The question

When a small model can either answer a table task directly or run a program it wrote, what is the best way to decide which answer to submit?

What we found

Surprisingly, yes, and with almost no learning. Trusting the program only when it passes a worked example and disagrees with the quick answer lifted accuracy from 56% to 66%, exactly matching the best any picker could reach, and never overwriting a correct answer. Fine-tuning learned most of the signal but landed lower, at 64%. Letting the model commit its own program every time collapsed to 24%, because programs that pass the example are often still secretly wrong.

Why it matters

Before fine-tuning a selector, try cheap tool signals: does the generated program pass its worked example, and does it disagree with the direct answer? Those two checks captured every recoverable win here. Never let a small model commit its own code unchecked.

Two-check rule vs answering directly56% → 66%correct on 50 unseen table tasks: 28 of 50 up to 33 of 50

Best any picker could reach66%perfect-hindsight ceiling, which the simple rule tied

Fine-tuned model64%one task below the free rule and the ceiling

Letting the model commit its own program24%worse than never using it; overwrote 21 of 50 correct answers

On this page

Results at a glance
Overview
Report
Experiment log
Figures
Data files
Reproduce
Related

Results at a glance 3

Answer accuracy for six ways of choosing which answer to submit

How to read

Each bar is test accuracy on unseen tasks for one commit policy, left to right: answer directly, commit the program blindly, fine-tune on scrambled labels, the fine-tuned policy, the two-check rule, and a perfect-hindsight ceiling. Taller is better.

Takeaway → The two-check rule reaches the perfect-hindsight ceiling, while committing the program blindly sits lowest, below even always answering directly. Knowing when to override beats raw effort.

Data table

commit policy	test accuracy
always direct	56%
base zero-shot	24%
shuffled-label LoRA	46%
LoRA action policy	64%
learned rule	66%
oracle ceiling	66%

Numbers from reports/final_summary.json (policy_results.*.test.accuracy)

Technical framing

Held-out accuracy: can a policy pick DIRECT vs PROGRAM from tool state? — A simple tool-state rule hits the oracle ceiling (66%); the LoRA policy learns most of the signal (64%) but not all.

Wrong answers rescued versus right answers ruined, per policy

How to read

For five policies, two bars each out of 50 tasks: wrong direct answers the program rescued, and correct direct answers a bad program overwrote. More rescues with zero overwrites is best.

Takeaway → The two-check rule and fine-tuned policy rescue about 5 tasks with zero overwrites, while committing the program blindly destroys 10 to 21 correct answers.

Data table

commit policy	direct-miss recoveries	direct-correct losses
base zero-shot	5	21
shuffled-label LoRA	5	10
LoRA action policy	4	0
learned rule	5	0
oracle ceiling	5	0

Numbers from reports/final_summary.json (policy_results.*.test.direct_miss_recoveries / direct_correct_losses)

Technical framing

Recoveries vs losses on held-out test (out of 50 tasks) — The learned rule and LoRA policy recover direct misses without ever overwriting a correct direct answer; controls lose 10-21 tasks.

Does the tool state hold a real, learnable choose signal?

How to read

Two training-loss curves over 80 steps: fitting the real pick-the-answer labels versus randomly shuffled labels. Lower loss means the model found a pattern; the shuffled line is the cannot-learn-noise baseline.

Takeaway → Real-label loss plunges toward zero while shuffled-label loss stays stuck high, confirming the tool state carries a genuine choose signal, not memorized noise.

Data table

training step	real labels	shuffled labels
2	3.24	2.69
4	0.377	0.497
6	0.274	0.378
8	0.41	0.344
10	0.023	0.12
12	0.0646	0.21
14	0.0284	0.394
16	7.8e-04	0.293
18	0.134	0.701
20	0.00352	0.36
22	2.2e-04	0.308
24	5.7e-04	0.358
26	3.7e-04	0.342
28	0.0014	0.278
30	0.0133	0.177
32	0.0147	0.33
34	0.00703	0.188
36	0.0178	0.38
38	0.0426	0.277
40	0.0502	0.354
42	0.00151	0.276
44	0.00113	0.558
46	4.6e-04	0.311
48	4.8e-04	0.38
50	3.9e-04	0.2
52	3.1e-04	0.333
54	2.0e-04	0.382
56	2.2e-04	0.351
58	1.7e-04	0.325
60	1.3e-04	0.398
62	1.2e-04	0.437
64	1.1e-04	0.144
66	1.0e-04	0.496
68	6.0e-05	0.359
70	6.0e-05	0.26
72	7.0e-05	0.289
74	6.0e-05	0.236
76	6.0e-05	0.745
78	7.0e-05	0.371
80	6.0e-05	0.129

Numbers from reports/final_summary.json (lora.losses, lora_shuffled.losses; every 2nd step)

Technical framing

LoRA training loss: real labels vs shuffled-label control — Real-label loss collapses to near zero while shuffled labels stay unlearnable, confirming the tool-state signal is genuine, not memorization.

In the author’s words from the Report · “Summary”

This standalone experiment tests whether a small action policy can use executable-tool observations to choose between a direct table answer and a repaired program output. The policy is not asked to produce a table or a program. It sees a compact tool state and chooses DIRECT or PROGRAM.

Overview

Standalone experiment for tool-state action-policy learning.

The package uses precomputed Qwen3.5-4B table-transformation traces. Each trace contains:

a direct JSON answer,
an executable transform(table) program attempt,
visible-example execution observations,
repair-loop observations,
hidden held-out exactness labels used only for training labels and evaluation.

The policy sees only deployable tool-state observations and chooses:

DIRECT: commit the direct JSON table,
PROGRAM: commit the final visible-verified executable program output.

Run the fast non-neural baselines:

python scripts/run_tool_state_policy.py \
  --root /workspace/experiments/qwen35_4b_tool_state_policy_lora \
  --skip-lora

Run the LoRA action-policy arm:

python scripts/run_tool_state_policy.py \
  --root /workspace/experiments/qwen35_4b_tool_state_policy_lora \
  --train-lora \
  --max-steps 80

Outputs are written under reports/.

Report

Rendered from reports/report.md

Summary

This standalone experiment tests whether a small action policy can use executable-tool observations to choose between a direct table answer and a repaired program output.

The policy is not asked to produce a table or a program. It sees a compact tool state and chooses DIRECT or PROGRAM.

Split

Train: 150 records across 30 families
Dev: 50 records across 10 families
Test: 50 records across 10 families

Held-Out Test Result

Policy	Exact	Accuracy	Program commits	Recoveries	Losses	Program precision
`direct_only`	28/50	56.0%	0	0	0	n/a
`program_if_visible_else_direct`	33/50	66.0%	14	5	0	85.7%
`program_if_visible_and_disagrees_else_direct`	33/50	66.0%	5	5	0	100.0%
`parse_fallback_program_else_direct`	32/50	64.0%	4	4	0	100.0%
`learned_rule`	33/50	66.0%	5	5	0	100.0%
`base_zero_shot_action`	12/50	24.0%	50	5	21	24.0%
`lora_action_policy`	32/50	64.0%	4	4	0	100.0%
`lora_shuffled_labels`	23/50	46.0%	27	5	10	44.4%
`oracle_action`	33/50	66.0%	5	5	0	100.0%

Gate Verdict

The environment state contains a deployable selection signal. The selected rule reaches the oracle action ceiling on held-out test: 33/50, with 5 direct-miss recoveries and 0 losses. The rule is simple: choose PROGRAM only when the program passes the public example and disagrees with the direct output on the new input.

The LoRA action-policy arm is a partial positive. It improves over direct-only (32/50 vs 28/50), beats the shuffled-label control (32/50 vs 23/50), and has 0 losses, but it does not match the simple rule/oracle ceiling (33/50). The posttraining signal is real but not the best controller in this small split.

Learned Rule

The non-neural rule search selected:

{
  "dev": {
    "accuracy": 0.66,
    "action_counts": {
      "DIRECT": 43,
      "PROGRAM": 7
    },
    "direct_correct_losses": 0,
    "direct_miss_recoveries": 5,
    "exact": 33,
    "n": 50,
    "policy": "rule",
    "program_commits": 7,
    "program_correct": 5,
    "program_precision": 0.7142857142857143,
    "split": "dev"
  },
  "false_action": "DIRECT",
  "feature": "direct_program_disagree_visible_pass",
  "train": {
    "accuracy": 0.6,
    "action_counts": {
      "DIRECT": 136,
      "PROGRAM": 14
    },
    "direct_correct_losses": 0,
    "direct_miss_recoveries": 8,
    "exact": 90,
    "n": 150,
    "policy": "rule",
    "program_commits": 14,
    "program_correct": 8,
    "program_precision": 0.5714285714285714,
    "split": "train"
  },
  "true_action": "PROGRAM"
}

LoRA Training

Real-label LoRA train examples: 122
Real-label LoRA tokenized examples: 122
Real-label final loss: 6.317748193396255e-05
Shuffled-label LoRA train examples: 122
Shuffled-label final loss: 0.12911884486675262

Figures

Held-out accuracy

Recoveries and losses

Family headroom

Interpretation

The decisive comparison is whether the learned action policy converts program-only headroom into held-out recoveries without committing hidden-wrong visible-pass programs on direct-correct tasks.

The oracle row is an upper bound that uses hidden labels to pick PROGRAM exactly when the program is correct and the direct answer is not. Deployable policies do not see that label.

The main positive result is not that LoRA is necessary; it is that a small deployable tool-state observation collapses the selection problem on this held-out split. Posttraining learned most of that signal, but the rule exposes the cleaner mechanism.

Limitations

The environment traces are precomputed; this package trains and evaluates the commit policy over those observed states.
The policy observes the final repair-loop state, so policies that use tool observations pay the full repair-loop generation cost even when they choose DIRECT.
The family-disjoint split is deterministic but still small. Repeat across multiple split seeds before treating the learned policy as stable.

Experiment log 1

Show the running log (1 entry, 2026-06-28)

2026-06-28

Created a standalone tool-state action-policy package.
Copied table-transformation cases and precomputed Qwen3.5-4B tool-environment traces into data/.
Chose a family-disjoint split seed (6137) that keeps direct-miss program-recovery labels present in train, dev, and test.
Planned a two-stage run: first validate non-neural baselines and oracle ceilings, then run the LoRA action-policy arm if the package compiles and the split is sane.
Ran the fast baseline/rule stage. The selected rule was PROGRAM only when the final program passed the visible example and disagreed with the direct output.
The selected rule reached 33/50 on held-out test, exactly matching the oracle action ceiling and recovering 5 direct misses with 0 losses.
Ran a one-step LoRA smoke test. Model loading, adapter attachment, training, scoring, and artifact writing all worked.
Started an 80-step LoRA run with autoregressive action generation, then interrupted it during post-train evaluation because generation was too slow for a two-action classifier.
Patched evaluation to score DIRECT vs PROGRAM by next-token logits. Re-ran the full 80-step real-label LoRA plus 80-step shuffled-label control.
Final LoRA result: real-label LoRA reached 32/50, recovering 4 direct misses with 0 losses. Shuffled-label LoRA reached 23/50, with 10 direct-correct losses. The adapter learned useful state signal but did not match the simpler rule/oracle ceiling.

Figures 3

Data files 1

Result tables and metrics copied from the experiment folder — preview inline or open the raw file.

reports/final_summary.json33 kB

Reproduce

Run steps are documented inside the experiment folder (README and scripts).

Browse the experiment folder on GitHub ↗

Qwen3.5-4B Tool State Policy LoRA

Results at a glance 3

Overview

Report

Summary

Split

Held-Out Test Result

Gate Verdict

Learned Rule

LoRA Training

Figures

Interpretation

Limitations

Experiment log 1

2026-06-28

Figures 3

Data files 1

Reproduce

Related

Related experiments