Qwen3.5-4B Learned Active Trace Policy

Finished2026-06-24imported · line ZActive Evidence Acquisition Posttraining and Adaptation Process Control and Tool Use Structured Execution and Compilers GitHub ↗

The one idea you need

Picture Guess Who, but with thousands of candidate programs that all match the examples you have already seen. Each move you run one new input; the best input is the question that splits the survivors evenly, knocking out the most wrong programs at once.

The question

When a small model must choose which extra input to run to narrow thousands of candidate programs, can it learn to pick better than a simple rule?

What we found

It depends. On the main test set a simple even-splitting rule beat the trained picker after one extra input — 91 percent of programs fully correct versus 87 — and the best-possible choice reached 97 percent. The picker did edge ahead on two other test sets. But its inputs ruled out only about 8 wrong candidates on average, versus 20 for the ideal pick.

Why it matters

Choosing an informative test is harder than generating candidates. Before you train a picker model, benchmark a cheap hand-coded splitting rule first — it may already beat your model, and its edge can widen as more tests compound.

Right program is always among the candidates100%so every failure is choosing which test to run, not building the program

Main test, one extra input: simple rule vs learned picker91% vs 87%share of programs fully correct; the simple rule won

Another test set, one extra input85% → 88%here the learned picker pulled ahead of the simple rule

Wrong candidates each chosen input rules out8 vs 20learned picker versus the ideal pick, main test average

On this page

Results at a glance
Overview
Report
Experiment log
Figures
Data files
Reproduce
Related

Results at a glance 4

With one extra input, share of programs fully solved by each method

How to read

Three groups of bars, one per test set (the main set plus two others). Within each group, four bars compare picking a random input, a simple even-splitting rule, the learned picker, and the best-possible pick. Height is the share of programs fully solved; taller is better.

Takeaway → The learned picker tops the simple rule on the two secondary sets but falls short on the main set; the best-possible pick leads everywhere, showing real headroom.

Data table

eval split	random extra query	hand-coded max-split	learned Qwen policy	oracle elimination
ceiling	86.2%	90.8%	86.7%	96.7%
support	65%	69.2%	70.8%	85.8%
IID	81.7%	85%	88.3%	91.7%

Numbers from reports/policy_summary.csv (also report table)

Technical framing

One extra query: learned policy vs baselines, by eval split — With one query the learned policy beats the hand-coded heuristic on support and IID but loses on the primary ceiling split.

Main test: share solved as more extra inputs are allowed

How to read

On the main test set, each line tracks the share of programs fully solved as the picker is allowed more extra inputs, from none to three. Four lines compare a random pick, a simple rule, the learned picker, and the best-possible pick. Higher is better.

Takeaway → The learned line starts below the simple rule at one input, then climbs to match it by three; the best-possible line stays clearly above both.

Data table

extra query budget	random extra query	hand-coded max-split	learned Qwen policy	oracle elimination
0	84.2%	84.2%	84.2%	84.2%
1	86.2%	90.8%	86.7%	96.7%
2	87.9%	94.2%	93.3%	100%
3	90.8%	95.8%	95.8%	100%

Numbers from reports/policy_summary.csv (also report table)

Technical framing

Ceiling split: success vs query budget — On the primary ceiling split the learned policy trails max-split at budget 1, then catches up to match it at budget 3.

A harder test set: share solved as more extra inputs are allowed

How to read

On the hardest test set, each line shows the share of programs fully solved as extra inputs increase from none to three, comparing a random pick, a simple rule, the learned picker, and the best-possible pick. Higher is better.

Takeaway → The learned picker leads the simple rule at one and two inputs, then slips behind it at three; the best-possible pick stays far ahead throughout.

Data table

extra query budget	random extra query	hand-coded max-split	learned Qwen policy	oracle elimination
0	60%	60%	60%	60%
1	65%	69.2%	70.8%	85.8%
2	69.6%	77.5%	82.5%	95.8%
3	72.1%	89.2%	86.7%	96.7%

Numbers from reports/policy_summary.csv (also report table)

Technical framing

Support split: success vs query budget — On the hard support split the learned policy leads the heuristic at budgets 1-2 but falls behind at budget 3.

How many wrong candidates each chosen input rules out

How to read

Three pairs of bars, one per test set. Each pair compares the wrong candidates ruled out by a single input: the learned picker's choice versus the best-possible choice. Taller means the chosen input carried more information.

Takeaway → In every test set the learned picker's bar sits under half the best-possible height — a real but only mild signal in its choices.

Data table

eval split	learned policy's chosen action	oracle best action
ceiling	8.37	19.9
support	11.4	25.6
IID	5.71	12

Numbers from report table (Learned Policy Diagnostics), backed by reports/eval/learned_*.json

Technical framing

Learned choices eliminate far fewer candidates than the oracle action — The learned first-query signal is real but weak: its chosen queries eliminate under half as many wrong candidates as the oracle's.

In the author’s words from the Report · “Summary”

This standalone experiment tests whether a Qwen3.5-4B LoRA can learn a low-budget active trace policy for selecting query inputs after typed-sketch candidate synthesis. The learned controller is trained by oracle-action distillation: each training state displays visible examples and candidate-output buckets for possible query inputs, while the target answer is the displayed query option that eliminates the most wrong candidates. The query prompt does not reveal the held-out expected output for candidate query inputs. The policy must infer which output bucket is likely correct from the visible examples and candidate-output structure.

Overview

Standalone experiment package for training and evaluating a learned active execution-query policy after typed sketch synthesis.

The experiment trains two local Qwen3.5-4B LoRA adapters:

a sketch adapter that emits typed DSL sketches from visible traces;
a policy adapter that chooses the next query input from candidate-output bucket summaries.

Large artifacts such as adapters and checkpoints are stored outside this directory under:

/workspace/large_artifacts/qwen35_4b_learned_active_trace_policy

Directory Layout

configs/: experiment configuration.
data/: generated JSONL datasets and manifests.
logs/: human-readable experiment log.
reports/: result JSON, CSV summaries, figures, and final report.
run_logs/: command stdout/stderr captures.
scripts/: runnable experiment scripts.
src/: local DSL, data-generation, sketch, prompt, modeling, and active-policy code.

Main Commands

python scripts/build_dataset.py
python scripts/build_policy_dataset.py --train-data data/static_bridge_80/dsl_train.jsonl --eval-data data/seed/dsl_train.jsonl
python scripts/train_adapter.py --task sketch --target-field target_sketch ...
python scripts/train_policy_adapter.py --train data/policy/policy_train.jsonl --eval data/policy/policy_eval.jsonl ...
python scripts/eval_learned_policy.py --data data/eval/dsl_eval_ceiling.jsonl ...
python scripts/make_report.py

Report

Rendered from reports/qwen35_4b_learned_active_trace_policy_report.md

Summary

The query prompt does not reveal the held-out expected output for candidate query inputs. The policy must infer which output bucket is likely correct from the visible examples and candidate-output structure.

Key Findings

Candidate coverage was perfect on all splits: candidate oracle and exact target synthesis both reached 100%. Remaining failures are policy/control failures, not synthesis failures.
The learned Qwen policy was mechanically reliable: parse rate was 1.0 on ceiling, support, and IID, with no fallback actions.
The learned policy improved the single-query regime on support and IID:
- support +1: learned 85/120 vs active max-split 83/120;
- IID +1: learned 53/60 vs active max-split 51/60.
The learned policy did not improve the primary ceiling +1 regime: learned 104/120 vs active max-split 109/120, with oracle elimination at 116/120.
At larger low budgets, the learned policy was mixed:
- ceiling +3 matched active max-split at 115/120;
- support +2 beat active max-split, 99/120 vs 93/120;
- support +3 and IID +2/+3 fell behind active max-split.
The learned policy chose nontrivial option ranks, but its chosen actions eliminated far fewer candidates than the oracle action on average. This points to a real learned first-query signal, but weak multi-step state-update robustness.

Bottom line: this run validates the learned-controller formulation and finds a real but incomplete gain. The current supervised policy is not yet the best controller for the primary split; the next iteration should focus on compounding, on-policy state distribution, and value-aware training rather than more sketch synthesis.

Training Notes

Sketch LoRA: trained for 2 epochs on static bridge sketch targets; final eval loss 0.000829.
Policy LoRA: trained on 481 oracle-distilled query states; checkpoint 40 eval loss was 0.399913, checkpoint 80 worsened to 0.405868, so checkpoint 40 was selected for evaluation.
Policy training examples only covered query steps 1-3, so the full evaluation uses budgets 0,1,2,3.

Policy Distillation Data

Train examples: 481. Eval examples: 95.

Policy dataset examples

Candidate Coverage

split	records	visible_selected_hidden_all_pct	candidate_oracle_hidden_all_pct	target_program_synthesized_pct	avg_synthesized_programs	avg_visible_consistent_candidates
ceiling	120	84.167	100	100	3114.73	388.342
iid	60	78.333	100	100	1104.25	58.783
support	120	60	100	100	3311.1	353.492

Candidate coverage

Policy Results

split	policy	budget	rows	hidden_all_pct	observed_all_pct	avg_hidden_passes	avg_queries_used
ceiling	active_max_split	0	120	84.167	100	17.583	0
ceiling	active_max_split	1	120	90.833	100	17.8	1
ceiling	active_max_split	2	120	94.167	100	17.875	2
ceiling	active_max_split	3	120	95.833	100	17.892	3
ceiling	learned_qwen_policy	0	120	84.167	100	17.583	0
ceiling	learned_qwen_policy	1	120	86.667	100	17.683	1
ceiling	learned_qwen_policy	2	120	93.333	100	17.875	2
ceiling	learned_qwen_policy	3	120	95.833	100	17.933	3
ceiling	oracle_elimination	0	120	84.167	100	17.583	0
ceiling	oracle_elimination	1	120	96.667	100	17.942	1
ceiling	oracle_elimination	2	120	100	100	18	2
ceiling	oracle_elimination	3	120	100	100	18	3
ceiling	random_extra	0	240	84.167	100	17.583	0
ceiling	random_extra	1	240	86.25	100	17.683	1
ceiling	random_extra	2	240	87.917	100	17.746	2
ceiling	random_extra	3	240	90.833	100	17.837	3
ceiling	visible_prior	0	120	84.167	100	17.583	0
iid	active_max_split	0	60	78.333	100	17	0
iid	active_max_split	1	60	85	100	17.533	1
iid	active_max_split	2	60	95	100	17.917	2
iid	active_max_split	3	60	96.667	100	17.967	3
iid	learned_qwen_policy	0	60	78.333	100	17	0
iid	learned_qwen_policy	1	60	88.333	100	17.7	1
iid	learned_qwen_policy	2	60	91.667	100	17.817	2
iid	learned_qwen_policy	3	60	91.667	100	17.817	3
iid	oracle_elimination	0	60	78.333	100	17	0
iid	oracle_elimination	1	60	91.667	100	17.85	1
iid	oracle_elimination	2	60	93.333	100	17.883	2
iid	oracle_elimination	3	60	96.667	100	17.967	3
iid	random_extra	0	120	78.333	100	17	0
iid	random_extra	1	120	81.667	100	17.342	1
iid	random_extra	2	120	85	100	17.525	2
iid	random_extra	3	120	87.5	100	17.6	3
iid	visible_prior	0	60	78.333	100	17	0
support	active_max_split	0	120	60	100	16.458	0
support	active_max_split	1	120	69.167	100	16.95	1
support	active_max_split	2	120	77.5	100	17.358	2
support	active_max_split	3	120	89.167	100	17.725	3
support	learned_qwen_policy	0	120	60	100	16.458	0
support	learned_qwen_policy	1	120	70.833	100	16.942	1
support	learned_qwen_policy	2	120	82.5	100	17.442	2
support	learned_qwen_policy	3	120	86.667	100	17.633	3
support	oracle_elimination	0	120	60	100	16.458	0
support	oracle_elimination	1	120	85.833	100	17.633	1
support	oracle_elimination	2	120	95.833	100	17.95	2
support	oracle_elimination	3	120	96.667	100	17.967	3
support	random_extra	0	240	60	100	16.458	0
support	random_extra	1	240	65	100	16.8	1
support	random_extra	2	240	69.583	100	17.008	2
support	random_extra	3	240	72.083	100	17.125	3
support	visible_prior	0	120	60	100	16.458	0

Ceiling success by budget

Support success by budget

IID success by budget

Budget 1 comparison

Budget 3 comparison

Oracle gap budget 1

Oracle gap budget 3

Learned Policy Diagnostics

split	learned_queries	parse_ok_rate	avg_chosen_rank	avg_chosen_actual_eliminated	avg_oracle_actual_eliminated
ceiling	720	1	4.233	8.372	19.915
iid	360	1	3.792	5.708	12.033
support	720	1	5.343	11.429	25.608

Learned choice rank

Interpretation

The central test is whether learned query selection closes the low-budget gap between the hand-coded max-split heuristic and oracle elimination. A useful learned controller should improve especially at budgets +1 to +3, where one or two high-value traces matter more than broad random coverage.

Reproducibility

Config: configs/experiment.json
Dataset manifest: data/dataset_manifest.json
Policy dataset manifest: data/policy/policy_dataset_manifest.json
Eval JSON files: reports/eval/learned_iid.json, reports/eval/learned_support.json, reports/eval/learned_ceiling.json
Policy summary CSV: reports/policy_summary.csv
Candidate summary CSV: reports/candidate_summary.csv
Large artifacts: /workspace/large_artifacts/qwen35_4b_learned_active_trace_policy

Experiment log 14

Show the running log (14 entries, 2026-06-24)

2026-06-24 05:00 UTC

Initialized a standalone learned active trace policy experiment package.

Design commitments:

Use Qwen/Qwen3.5-4B.
Train a fresh sketch LoRA and a fresh policy LoRA for this experiment.
Keep model adapters and checkpoints under /workspace/large_artifacts/qwen35_4b_learned_active_trace_policy.
Keep downloadable experiment files under /workspace/experiments/qwen35_4b_learned_active_trace_policy.
Train the policy by oracle-action distillation from candidate-program ambiguity states.
Do not reveal query-option expected outputs to the policy prompt; only visible examples and candidate-output buckets are shown.
Evaluate visible-only, random extra trace, active max-split, learned Qwen policy, and oracle elimination on IID, support, and ceiling splits.

Initial implementation:

Added src/active_core.py for shared candidate-bank, query-option, prompt, policy, and summary logic.
Added scripts/build_policy_dataset.py for oracle-distilled policy SFT examples.
Added scripts/train_policy_adapter.py for the Qwen policy LoRA.
Added scripts/eval_learned_policy.py for model-in-loop learned-policy evaluation.
Replaced scripts/make_report.py with a learned-policy report generator.

Validation:

python -m py_compile scripts/build_dataset.py scripts/build_policy_dataset.py scripts/train_adapter.py scripts/train_policy_adapter.py scripts/eval_learned_policy.py scripts/make_report.py src/*.py
Result: passed.
Removed generated __pycache__ directories from the experiment tree.

2026-06-24 05:05 UTC

Built the standalone DSL dataset.

Command:

python scripts/build_dataset.py > run_logs/dataset_build_console.log 2>&1

Observed:

Static80 sketch-training records: 240.
Seed/policy-validation records: 240.
IID eval records: 60.
Support eval records: 120.
Ceiling eval records: 120.
Active query pool before policy cap: 384 cases per eval record.

Next check: build a small policy-distillation smoke dataset, inspect labels and prompt shape, then build the full policy SFT dataset.

2026-06-24 05:11 UTC

Policy-distillation smoke iteration 1 produced zero examples.

Command:

python scripts/build_policy_dataset.py --train-data data/static_bridge_80/dsl_train.jsonl --eval-data data/seed/dsl_train.jsonl --out-dir data/policy_smoke --max-train-records 12 --max-eval-records 8 --max-steps 3 --max-options 24 --max-policy-candidates 128 --max-query-pool-cases 48 --max-total-programs-per-record 4000 > run_logs/policy_dataset_smoke_console.log 2>&1

Cause:

The first dataset build attached case_pool only to eval records.
Policy SFT needs query pools on training records too.

Fix:

Patched src/data_gen.py so static_bridge_records accepts case_pool_count.
Patched scripts/build_dataset.py to attach active case pools to seed, static-base, and bridge training records.
Rebuilt the dataset with python scripts/build_dataset.py > run_logs/dataset_build_v2_console.log 2>&1.

Verification:

data/static_bridge_80/dsl_train.jsonl first record has 384 query-pool cases.
data/seed/dsl_train.jsonl first record has 384 query-pool cases.
data/dataset_manifest.json records train_records_include_active_case_pools: true.

2026-06-24 05:23 UTC

Policy-distillation smoke iteration 2 succeeded.

Command:

python scripts/build_policy_dataset.py --train-data data/static_bridge_80/dsl_train.jsonl --eval-data data/seed/dsl_train.jsonl --out-dir data/policy_smoke --max-train-records 12 --max-eval-records 8 --max-steps 3 --max-options 24 --max-policy-candidates 128 --max-query-pool-cases 48 --max-total-programs-per-record 4000 > run_logs/policy_dataset_smoke_v2_console.log 2>&1

Observed:

Train examples: 8.
Eval examples: 2.
First prompt displayed 24 options and hid the query-option expected output.
First label: Q00, with 2 actual candidate eliminations.

Decision:

Use both static bridge training files for the full policy source to increase supervised states.
Keep seed records as policy-validation source.

2026-06-24 05:34 UTC

Built the full oracle-distilled policy dataset.

Command:

python scripts/build_policy_dataset.py --train-data data/static_bridge_80/dsl_train.jsonl data/static_bridge_60/dsl_train.jsonl --eval-data data/seed/dsl_train.jsonl --out-dir data/policy --max-eval-records 120 --max-steps 3 --max-options 24 --max-policy-candidates 128 --max-query-pool-cases 48 --max-total-programs-per-record 4000 > run_logs/policy_dataset_full_console.log 2>&1

Observed:

Train source records: 480.
Policy train examples: 481.
Policy eval examples: 95.
Train examples by step: 241 at step 1, 165 at step 2, 75 at step 3.
Average target actual eliminations: 27.744.
Target action distribution is nontrivial: Q00 occurs 204/481, with many labels deeper in the displayed list.
Token length check with the Qwen tokenizer: max 2882, p95 2673, 0 examples above 4096.

Next step: train a fresh sketch LoRA, then train the policy LoRA.

2026-06-24 05:54 UTC

Sketch LoRA training completed successfully.

Command:

python scripts/train_adapter.py --train data/static_bridge_80/dsl_train.jsonl --eval data/eval/dsl_eval_iid.jsonl --task sketch --target-field target_sketch --prompt-mode trace --output-dir /workspace/large_artifacts/qwen35_4b_learned_active_trace_policy/models/sketch_lora --epochs 2.0 --lr 1.5e-4 --rank 32 --alpha 64 --dropout 0.05 --grad-accum 8 --save-steps 30 --eval-steps 30 > run_logs/training_sketch_lora_console.log 2>&1

Observed:

Trainable parameters: 42,467,328.
Step 30 eval loss: 0.0001971.
Final eval loss at epoch 2: 0.000829.
Adapter/checkpoint tree size: 709M.
Large artifact location: /workspace/large_artifacts/qwen35_4b_learned_active_trace_policy/models/sketch_lora.

Next step: train the Qwen active-query policy LoRA on data/policy/policy_train.jsonl.

2026-06-24 06:37 UTC

Policy LoRA training was run with checkpointed validation and stopped early after the validation loss stopped improving.

Command:

python scripts/train_policy_adapter.py --train data/policy/policy_train.jsonl --eval data/policy/policy_eval.jsonl --output-dir /workspace/large_artifacts/qwen35_4b_learned_active_trace_policy/models/policy_lora --epochs 3.0 --lr 1.2e-4 --rank 32 --alpha 64 --dropout 0.05 --grad-accum 8 --save-steps 40 --eval-steps 40 --max-eval-records 95 > run_logs/training_policy_lora_console.log 2>&1

Observed:

Trainable parameters: 42,467,328.
Checkpoint 40 eval loss: 0.399913.
Checkpoint 80 eval loss: 0.405868.
Checkpoint 80 was worse than checkpoint 40, so training was intentionally interrupted at step 81 rather than spending another long epoch on likely overfit/noise.
Selected policy checkpoint for evaluation: /workspace/large_artifacts/qwen35_4b_learned_active_trace_policy/models/policy_lora/checkpoint-40.
Policy artifact tree size after two checkpoints: 528M.

Next step: run target-sketch smoke evaluation with the selected policy checkpoint to verify parsing/action behavior, then run full model-sketch evaluations.

2026-06-24 06:40 UTC

Target-sketch learned-policy smoke evaluation completed.

Command:

python scripts/eval_learned_policy.py --data data/eval/dsl_eval_ceiling.jsonl --sketch-source target --policy-adapter /workspace/large_artifacts/qwen35_4b_learned_active_trace_policy/models/policy_lora/checkpoint-40 --output reports/eval/_smoke_target_policy_ceiling10.json --max-records 10 --budgets 0,1,2,3 --random-repeats 1 --max-total-programs-per-record 4000 --max-policy-candidates 128 --max-query-pool-cases 48 --policy-max-options 24 > run_logs/eval_smoke_target_policy_ceiling10_console.log 2>&1

Observed:

Candidate oracle: 10/10.
Exact target synthesized: 10/10.
Visible-only: 10/10.
Learned policy parse rate: 1.0.
Learned traces: 60 parseable actions, 0 fallbacks.
The smoke slice was already solved, so it validates mechanics and parsing but not policy advantage.

Next step: run a model-sketch smoke where visible selection is expected to be less saturated.

2026-06-24 06:45 UTC

Full-path model-sketch learned-policy smoke evaluation completed.

Command:

python scripts/eval_learned_policy.py --data data/eval/dsl_eval_ceiling.jsonl --sketch-source model --sketch-adapter /workspace/large_artifacts/qwen35_4b_learned_active_trace_policy/models/sketch_lora --policy-adapter /workspace/large_artifacts/qwen35_4b_learned_active_trace_policy/models/policy_lora/checkpoint-40 --output reports/eval/_smoke_model_policy_ceiling10.json --max-records 10 --budgets 0,1,2,3 --random-repeats 1 --num-samples 3 --max-total-programs-per-record 4000 --max-policy-candidates 128 --max-query-pool-cases 48 --policy-max-options 24 > run_logs/eval_smoke_model_policy_ceiling10_console.log 2>&1

Observed:

Candidate oracle: 10/10.
Exact target synthesized: 10/10.
Visible-only: 10/10.
Learned policy parse rate: 1.0.
Learned traces: 60 parseable actions, 0 fallbacks.
Both Qwen adapters loaded and ran together successfully.

Decision:

Full eval will use budgets 0,1,2,3.
Reason: the policy SFT data contains oracle trajectories through step 3, and the main experimental question is low-budget learned control.

2026-06-24 07:18 UTC

Full ceiling evaluation completed.

Command:

python scripts/eval_learned_policy.py --data data/eval/dsl_eval_ceiling.jsonl --sketch-source model --sketch-adapter /workspace/large_artifacts/qwen35_4b_learned_active_trace_policy/models/sketch_lora --policy-adapter /workspace/large_artifacts/qwen35_4b_learned_active_trace_policy/models/policy_lora/checkpoint-40 --output reports/eval/learned_ceiling.json --budgets 0,1,2,3 --random-repeats 2 --num-samples 3 --max-total-programs-per-record 4000 --max-policy-candidates 128 --max-query-pool-cases 48 --policy-max-options 24 > run_logs/eval_learned_ceiling_console.log 2>&1

Observed:

Candidate oracle: 120/120.
Exact target synthesized: 120/120.
Visible-only: 101/120.
Active max-split: 109/120 at +1, 113/120 at +2, 115/120 at +3.
Learned Qwen policy: 104/120 at +1, 112/120 at +2, 115/120 at +3.
Oracle elimination: 116/120 at +1, 120/120 at +2 and +3.
Learned parse rate: 1.0.

Interpretation before retention checks:

The learned policy is mechanically reliable and catches up to active max-split by +3.
It does not beat active max-split in the highest-leverage +1 regime on the primary split.
The oracle gap remains large at +1, so the policy-learning formulation is still relevant, but the current SFT controller has not solved it.

Next step: run support and IID retention evaluations with the same settings.

2026-06-24 07:50 UTC

Full support evaluation completed.

Command:

python scripts/eval_learned_policy.py --data data/eval/dsl_eval_support.jsonl --sketch-source model --sketch-adapter /workspace/large_artifacts/qwen35_4b_learned_active_trace_policy/models/sketch_lora --policy-adapter /workspace/large_artifacts/qwen35_4b_learned_active_trace_policy/models/policy_lora/checkpoint-40 --output reports/eval/learned_support.json --budgets 0,1,2,3 --random-repeats 2 --num-samples 3 --max-total-programs-per-record 4000 --max-policy-candidates 128 --max-query-pool-cases 48 --policy-max-options 24 > run_logs/eval_learned_support_console.log 2>&1

Observed:

Candidate oracle: 120/120.
Exact target synthesized: 120/120.
Visible-only: 72/120.
Active max-split: 83/120 at +1, 93/120 at +2, 107/120 at +3.
Learned Qwen policy: 85/120 at +1, 99/120 at +2, 104/120 at +3.
Oracle elimination: 103/120 at +1, 115/120 at +2, 116/120 at +3.
Learned parse rate: 1.0.

Interpretation:

Learned policy beats active max-split at +1 and +2 on support retention.
Learned policy falls behind active max-split by +3, suggesting the learned policy can choose useful early discriminators but compounds less reliably across multiple queries.

Next step: run IID retention evaluation.

2026-06-24 08:02 UTC

Full IID evaluation completed.

Command:

python scripts/eval_learned_policy.py --data data/eval/dsl_eval_iid.jsonl --sketch-source model --sketch-adapter /workspace/large_artifacts/qwen35_4b_learned_active_trace_policy/models/sketch_lora --policy-adapter /workspace/large_artifacts/qwen35_4b_learned_active_trace_policy/models/policy_lora/checkpoint-40 --output reports/eval/learned_iid.json --budgets 0,1,2,3 --random-repeats 2 --num-samples 3 --max-total-programs-per-record 4000 --max-policy-candidates 128 --max-query-pool-cases 48 --policy-max-options 24 > run_logs/eval_learned_iid_console.log 2>&1

Observed:

Candidate oracle: 60/60.
Exact target synthesized: 60/60.
Visible-only: 47/60.
Active max-split: 51/60 at +1, 57/60 at +2, 58/60 at +3.
Learned Qwen policy: 53/60 at +1, 55/60 at +2, 55/60 at +3.
Oracle elimination: 55/60 at +1, 56/60 at +2, 58/60 at +3.
Learned parse rate: 1.0.

Interpretation:

Learned policy improves the single-query regime on IID.
It compounds worse than active max-split after additional queries.
Across splits, this points to a real learned first-query signal but insufficient recurrent/state-update robustness.

Next step: generate report, plots, and CSV summaries.

2026-06-24 08:04 UTC

Generated report, figures, and CSV summaries.

Command:

python scripts/make_report.py > run_logs/report_generation_console.log 2>&1

Artifacts:

Final report: reports/qwen35_4b_learned_active_trace_policy_report.md.
Summary CSVs: reports/policy_summary.csv, reports/candidate_summary.csv, reports/learned_parse_summary.csv, reports/learned_query_trace_summary.csv.
Figures:

Manual report edit after generation:

Added key findings and training notes to make the writeup reflect the actual result: reliable parsing, real +1 gains on support/IID, no +1 gain on ceiling, and weak multi-step compounding.

2026-06-24 08:10 UTC

Final packaging and audit completed.

Checks:

Removed generated __pycache__ directories.
Removed an unused copied evaluator script that belonged to a different experiment package.
Searched the final experiment tree for stale old-experiment references; no matches remained.
Verified that no file larger than 50M remains under /workspace/experiments/qwen35_4b_learned_active_trace_policy.
Verified experiment package size: 85M.
Verified large artifact package size: 1.3G.
Verified all 10 generated PNG figures are readable and non-empty.

Final locations:

Downloadable experiment package: /workspace/experiments/qwen35_4b_learned_active_trace_policy.
Large adapters/checkpoints: /workspace/large_artifacts/qwen35_4b_learned_active_trace_policy.
Final report: reports/qwen35_4b_learned_active_trace_policy_report.md.