Qwen Recursive Ephemeral Program Induction

Finished2026-06-27imported · line YBenchmark Generalization Structured Execution and Compilers GitHub ↗

The one idea you need

Give a model four before-and-after examples of a text cleanup and ask it to write one small reusable rule for the pattern, like a spreadsheet formula. Some rules truly generalize; others just hard-code the four examples. The trick is knowing which to trust.

The question

If a model turns a handful of examples into a small reusable rule, does running that rule beat just answering each new row directly?

What we found

Only when you check the rule first. On its own, a model writing and applying a reusable rule solved 40% of tasks perfectly versus 56% for plain row-by-row answering, and it broke six tasks direct answering had solved. Adding a gate that uses the rule only when it passes the shown examples and isn't a memorized lookup lifted success to 64%, fixing two tasks and breaking none.

Why it matters

Don't deploy a self-written transformation rule blind. Run it against your known examples, reject rules that merely memorize those examples, and fall back to answering row-by-row otherwise. That gate turns a net loss into a clean gain.

Plain row-by-row answering56%share of tasks solved perfectly, every row right

Self-written rule, used blindly40%worse than answering directly; broke 6 tasks it had solved

Self-written rule with a trust gate64%fixed 2 tasks, broke none versus direct answering

Scrambled-example sanity check4%rules built from mismatched examples almost never solve a task

On this page

Results at a glance
Overview
Report
Experiment log
Figures
Data files
Reproduce
Related

Results at a glance 6

Task success by method: the trust-gated rule wins

How to read

Each group is one way of solving the tasks; taller bars are better. One bar is the share of tasks solved perfectly with every row right, the other the share of individual rows correct. The gated rule stands tallest; the scrambled-example control sits near the floor.

Takeaway → The trust-gated rule is the tallest whole-task bar, edging out plain answering, while blind rules and the scrambled control fall far below both.

Data table

method	full-task exact	row exact
gated program + fallback	64%	82.7%
direct batch	60%	76%
direct row-by-row	56%	80.7%
recursive program	40%	44.7%
monolithic program	36%	40.7%
shuffled-label control	4%	6%

Numbers from experiments/qwen_recursive_ephemeral_program_induction/analysis/summary.csv

Technical framing

Held-out task success by method: gated program induction beats direct answering — Raw induced programs underperform direct answering (40% vs 56%), but gating on non-memorizing train-passing programs lifts it to 64%.

Tasks each method rescued versus tasks it broke

How to read

For each method, one bar counts tasks it newly solved and another counts tasks it broke that plain answering had solved, out of 25. More rescued and fewer broken is better.

Takeaway → Blind rules break six or seven tasks to win two, but the gated version keeps both wins with a zero-height broken bar.

Data table

method	tasks helped	tasks hurt
gated program + fallback	2	0
direct batch	2	1
recursive program	2	6
monolithic program	2	7
shuffled-label control	1	14

Numbers from experiments/qwen_recursive_ephemeral_program_induction/analysis/summary.csv

Technical framing

Tasks helped vs hurt relative to the direct row-by-row baseline — Ungated programs win 2 tasks but break 6-7 that direct answering solved; the gate keeps both wins while introducing zero losses.

Whole-task success by task type: where rules help or break

How to read

Each group is a task type such as numbers, dates, names, file paths, billing codes, and ratings; bars compare plain answering, blind rules, and the gated rule. Taller is better.

Takeaway → Blind rules genuinely lift number tasks yet flatten to zero on dates, file paths, and billing codes, while the gated bar matches whichever approach is best in every type.

Data table

task family	direct row-by-row	recursive program	gated program + fallback
Number	28.6%	42.9%	57.1%
DateTime	28.6%	0%	28.6%
Name	100%	66.7%	100%
FilePath	100%	0%	100%
BillingCode	100%	0%	100%
Rating	100%	100%	100%

Numbers from experiments/qwen_recursive_ephemeral_program_induction/analysis/family_summary.csv

Technical framing

Full-task exact by task family: where programs help and where they break — Program induction genuinely helps Number tasks but wrecks DateTime, FilePath and BillingCode; the gate matches the best arm per family.

Getting rows right versus getting whole tasks perfect

How to read

Each method has two bars: the share of individual rows correct and the share of whole tasks correct with no mistakes anywhere. Taller is better.

Takeaway → Plain answering loses a big chunk moving from rows to perfect whole tasks, while rule methods stay low on both because they tend to miss an entire task at once.

Data table

Method	Row exact	Full-task exact
Gated program+direct	82.7%	64%
Direct batch	76%	60%
Direct row-by-row	80.7%	56%
Recursive program	44.7%	40%
Program oracle	44.7%	40%
Monolithic program	40.7%	36%
Shuffled-label control	6%	4%

Numbers from report table (reports/qwen_recursive_ephemeral_program_induction_report.md, Main Result)

Technical framing

Row-level accuracy vs strict full-task exactness by method — Direct answering loses 25pts moving from row to full-task exactness; program arms lose little because they fail whole tasks at once.

Fitting the shown examples does not mean solving new rows

How to read

For each rule method, one bar shows how often a rule matched the few shown examples and the other how often it solved the whole task on new rows. A truly working rule would have both bars high.

Takeaway → Blind rules match the shown examples about half the time but solve fewer new tasks, and the scrambled control fits examples while solving almost nothing, exposing false matches.

Data table

Program arm	Train-pass rate (visible examples)	Full-task exact (held-out)
Gated program+direct	44%	64%
Recursive program	52%	40%
Monolithic program	48%	36%
Shuffled-label control	20%	4%

Numbers from report table (reports/qwen_recursive_ephemeral_program_induction_report.md, Main Result)

Technical framing

Passing visible examples does not mean generalizing to held-out rows — Raw programs pass visible examples on 52% of tasks but solve only 40% held-out; the shuffled control even train-passes 20% yet solves 4%.

Row accuracy by task type: where rules collapse

How to read

Each task type has two bars comparing plain answering with blind rules on the share of individual rows correct on new data. Taller is better.

Takeaway → Rules track plain answering on currency, email, phone, and rating tasks but collapse toward zero on dates, billing codes, and file paths even row by row.

Data table

Task family	Direct row-by-row	Recursive program
BillingCode	100%	0%
City	66.7%	33.3%
Currency	100%	100%
DateTime	59.5%	9.5%
Email	100%	100%
FilePath	100%	0%
Name	100%	66.7%
Number	76.2%	45.2%
Phone	100%	100%
Rating	100%	100%

Numbers from analysis/family_summary.csv

Technical framing

Row-level accuracy by task family: where induced programs collapse — Programs match direct answering on Currency/Email/Phone/Rating but collapse on DateTime, BillingCode, and FilePath even at the row level.

In the author’s words from the Report · “Interpretation”

Direct row-by-row answering solves 56.0% of tasks under strict full-task exactness. The selected recursive executable program solves 40.0%. The gated recursive method, which falls back to direct answering when no non-memorizing train-passing program is available, solves 64.0%. The hidden diagnostic oracle over train-passing generated programs solves 40.0%, and at least one recursive candidate passes visible examples on 52.0% of tasks. The gated recursive method helps 2 tasks and hurts 0 tasks relative to direct row-by-row answering. The shuffled-label control is included to check whether executable programs can be induced from mismatched examples. A useful executable-program result should beat both direct answering and this shuffled control.

Overview

Standalone experiment for inducing task-local executable transformation programs from sparse examples.

Large reusable artifacts are stored under:

/workspace/large_artifacts/qwen_recursive_ephemeral_program_induction

Primary outputs:

src/qwen_recursive_ephemeral_program_induction.py
experiment_log.md
runs/
analysis/
reports/

Report

Rendered from reports/qwen_recursive_ephemeral_program_induction_report.md

Question

Can a frozen language model convert sparse input-output examples into a task-local executable program that is more consistent than direct row-by-row answering?

The method asks the model to write a Python transform(row) function. Candidate programs are executed on visible examples. Only visible examples are used for selection and repair. Held-out rows are used only for final scoring.

Setup

Run: main_v1
Dataset: public text-transformation tasks.
Tasks: 25
Visible examples per task: 4
Held-out cap per task: 6
Program variants: monolithic,robust
Repair rounds: 1
Elapsed seconds: 1444.2

Main Result

method	tasks	row_exact	full_task_exact	train_pass_rate	tasks_helped_vs_direct	tasks_hurt_vs_direct
recursive_gated_direct	25	82.7%	64.0%	44.0%	2	0
direct_batch	25	76.0%	60.0%		2	1
direct_row	25	80.7%	56.0%		0	0
recursive_program	25	44.7%	40.0%	52.0%	2	6
recursive_oracle	25	44.7%	40.0%	52.0%	2	6
monolithic_program	25	40.7%	36.0%	48.0%	2	7
recursive_shuffled	25	6.0%	4.0%	20.0%	1	14

Interpretation

Direct row-by-row answering solves 56.0% of tasks under strict full-task exactness. The selected recursive executable program solves 40.0%. The gated recursive method, which falls back to direct answering when no non-memorizing train-passing program is available, solves 64.0%. The hidden diagnostic oracle over train-passing generated programs solves 40.0%, and at least one recursive candidate passes visible examples on 52.0% of tasks.

The gated recursive method helps 2 tasks and hurts 0 tasks relative to direct row-by-row answering.

The shuffled-label control is included to check whether executable programs can be induced from mismatched examples. A useful executable-program result should beat both direct answering and this shuffled control.

Program Diagnostics

The raw executable-program arm is not deployable by itself: it passes visible examples on 52.0% of tasks but drops to 40.0% full-task exact and hurts 6 tasks that direct row-by-row answering solved. Most damage comes from train-fitting programs that are too narrow, often literal branches or partial parsers.

The gated arm uses an executable program only when a train-passing candidate does not look like a literal example table; otherwise it falls back to direct row-by-row answering. It uses generated programs on 11/25 tasks, solves 16/25 tasks overall, and captures both program-only wins without introducing any direct-baseline losses.

The shuffled-label control solves only 1/25 tasks and hurts 14 direct-baseline successes, so the generated programs are not succeeding from formatting alone. The useful signal is real, but sparse.

Charts

Method full-task exact

Row scatter

Wins and losses

Family breakdown

Train-passing candidates

Task Details

task_id	family	heldout_rows	direct_full_exact	recursive_full_exact	recursive_gated_full_exact	recursive_oracle_full_exact	recursive_train_pass_count	recursive_gated_used_program	recursive_gated_helped_vs_direct	recursive_gated_hurt_vs_direct
Number.000044	Number	6	False	True	True	True	1	True	True	False
Number.000093	Number	3	False	True	True	True	1	True	True	False
BillingCode.000002	BillingCode	6	True	False	True	False	0	False	False	False
City.000011	City	3	False	False	False	False	1	False	False	False
Currency.000004	Currency	6	True	True	True	True	1	True	False	False
DateTime.000012	DateTime	6	False	False	False	False	1	True	False	False
DateTime.000035	DateTime	6	True	False	True	False	0	False	False	False
DateTime.000077	DateTime	6	False	False	False	False	0	False	False	False
DateTime.000083	DateTime	6	False	False	False	False	0	False	False	False
DateTime.000088	DateTime	6	False	False	False	False	0	False	False	False
DateTime.000094	DateTime	4	True	False	True	False	0	False	False	False
DateTime.000098	DateTime	6	False	False	False	False	0	False	False	False
Email.000013	Email	6	True	True	True	True	2	True	False	False
FilePath.000001	FilePath	6	True	False	True	False	0	False	False	False
Name.000013	Name	6	True	True	True	True	2	True	False	False
Name.000026	Name	6	True	False	True	False	0	False	False	False
Name.000028	Name	6	True	True	True	True	2	True	False	False
Number.000048	Number	4	True	False	True	False	0	False	False	False
Number.000074	Number	6	False	False	False	False	0	False	False	False
Number.000075	Number	6	False	False	False	False	0	False	False	False
Number.000081	Number	6	False	False	False	False	1	False	False	False
Number.000088	Number	6	True	True	True	True	1	True	False	False
Phone.000005	Phone	6	True	True	True	True	1	True	False	False
Phone.000017	Phone	6	True	True	True	True	1	True	False	False
Rating.000001	Rating	6	True	True	True	True	1	True	False	False

Family Summary

family	direct_row	monolithic_program	recursive_gated_direct	recursive_oracle	recursive_program
BillingCode	1	0	1	0	0
City	0	0	0	0	0
Currency	1	1	1	1	1
DateTime	0.286	0	0.286	0	0
Email	1	1	1	1	1
FilePath	1	0	1	0	0
Name	1	0.667	1	0.667	0.667
Number	0.286	0.429	0.571	0.429	0.429
Phone	1	1	1	1	1
Rating	1	0	1	1	1

Limitations

Generated code is sandboxed by a conservative AST pass, so some potentially valid programs may be rejected. The benchmark tasks are public text transformations and do not cover arbitrary software engineering problems. Full-task exact is intentionally strict and can be much lower than row accuracy.

Artifacts

Run directory: /workspace/experiments/qwen_recursive_ephemeral_program_induction/runs/main_v1
Summary: analysis/summary.csv
Task details: analysis/task_summary.csv
Candidate programs: analysis/candidates.csv
Figures: analysis/figures/

Experiment log 6

Show the running log (6 entries)

Setup

Created a fresh standalone experiment directory.
Large artifacts are kept under /workspace/large_artifacts/qwen_recursive_ephemeral_program_induction.
Primary metric: strict full-task exact on held-out rows.
Secondary metrics: row exact, train-pass rate, generated-program validity, and oracle headroom among train-passing generated programs.

Run `smoke_no_qwen`

Time UTC: 2026-06-27T10:43:22.094485+00:00
Elapsed seconds: 0.6
Config: {"answer_max_new_tokens": 64, "batch_max_new_tokens": 320, "code_max_new_tokens": 520, "heldout_cap": 3, "min_heldout": 3, "no_qwen": true, "repair_max_new_tokens": 560, "repair_rounds": 1, "repair_variants": "minimal,broaden,conditional", "run_name": "smoke_no_qwen", "run_shuffled": true, "sample_seed": 20260627, "task_limit": 3, "train_n": 4, "variants": "monolithic,helpers,robust"}
Tasks: 3
Direct row-by-row full-task exact: 0.0%
Recursive selected-program full-task exact: 0.0%
Recursive train-pass rate: 0.0%
Recursive oracle among train-passing candidates: 0.0%

Run `smoke_qwen_3`

Time UTC: 2026-06-27T10:47:05.311065+00:00
Elapsed seconds: 198.9
Config: {"answer_max_new_tokens": 48, "batch_max_new_tokens": 220, "code_max_new_tokens": 360, "heldout_cap": 3, "min_heldout": 3, "no_qwen": false, "repair_max_new_tokens": 380, "repair_rounds": 1, "repair_variants": "minimal,broaden,conditional", "run_name": "smoke_qwen_3", "run_shuffled": true, "sample_seed": 20260627, "task_limit": 3, "train_n": 4, "variants": "monolithic,helpers,robust"}
Tasks: 3
Direct row-by-row full-task exact: 66.7%
Recursive selected-program full-task exact: 33.3%
Recursive train-pass rate: 66.7%
Recursive oracle among train-passing candidates: 33.3%

Run `smoke_qwen_5_v2`

Time UTC: 2026-06-27T10:54:41.415308+00:00
Elapsed seconds: 402.9
Config: {"answer_max_new_tokens": 48, "batch_max_new_tokens": 220, "code_max_new_tokens": 360, "heldout_cap": 3, "min_heldout": 3, "no_qwen": false, "repair_max_new_tokens": 380, "repair_rounds": 1, "repair_variants": "minimal,broaden,conditional", "run_name": "smoke_qwen_5_v2", "run_shuffled": true, "sample_seed": 20260627, "task_limit": 5, "train_n": 4, "variants": "monolithic,helpers,robust"}
Tasks: 5
Direct row-by-row full-task exact: 60.0%
Recursive selected-program full-task exact: 20.0%
Recursive train-pass rate: 40.0%
Recursive oracle among train-passing candidates: 20.0%

Run `smoke_qwen_5_helpers`

Time UTC: 2026-06-27T11:05:15.279262+00:00
Elapsed seconds: 530.7
Config: {"answer_max_new_tokens": 48, "batch_max_new_tokens": 220, "code_max_new_tokens": 420, "heldout_cap": 3, "min_heldout": 3, "no_qwen": false, "repair_max_new_tokens": 420, "repair_rounds": 1, "repair_variants": "minimal,broaden,conditional", "run_name": "smoke_qwen_5_helpers", "run_shuffled": true, "sample_seed": 20260627, "task_limit": 5, "train_n": 4, "variants": "monolithic,helpers,robust"}
Tasks: 5
Direct row-by-row full-task exact: 60.0%
Recursive selected-program full-task exact: 20.0%
Recursive train-pass rate: 80.0%
Recursive oracle among train-passing candidates: 20.0%

Run `main_v1`

Time UTC: 2026-06-27T11:32:03.844775+00:00
Elapsed seconds: 1444.3
Config: {"answer_max_new_tokens": 56, "batch_max_new_tokens": 300, "code_max_new_tokens": 420, "heldout_cap": 6, "min_heldout": 3, "no_qwen": false, "repair_max_new_tokens": 420, "repair_rounds": 1, "repair_variants": "minimal,broaden,conditional", "run_name": "main_v1", "run_shuffled": true, "sample_seed": 20260627, "task_limit": 25, "train_n": 4, "variants": "monolithic,robust"}
Tasks: 25
Direct row-by-row full-task exact: 56.0%
Recursive selected-program full-task exact: 40.0%
Recursive gated-direct full-task exact: 64.0%
Recursive train-pass rate: 52.0%
Recursive oracle among train-passing candidates: 40.0%

Figures 5

method full task exact · analysis/figures/

row scatter direct vs recursive · analysis/figures/

train passing candidates · analysis/figures/

wins losses vs direct · analysis/figures/

Data files 18

Result tables and metrics copied from the experiment folder — preview inline or open the raw file.

analysis/family_summary.csv2.2 kB
analysis/summary.csv434 B
analysis/task_summary.csv6.0 kB
runs/main_v1/family_summary.csv2.2 kB
runs/main_v1/summary.csv434 B
runs/main_v1/task_summary.csv6.0 kB
runs/smoke_no_qwen/family_summary.csv429 B
runs/smoke_no_qwen/summary.csv298 B
runs/smoke_no_qwen/task_summary.csv1.1 kB
runs/smoke_qwen_3/family_summary.csv489 B
runs/smoke_qwen_3/summary.csv523 B
runs/smoke_qwen_3/task_summary.csv1.1 kB
runs/smoke_qwen_5_helpers/family_summary.csv690 B
runs/smoke_qwen_5_helpers/summary.csv344 B
runs/smoke_qwen_5_helpers/task_summary.csv1.6 kB
runs/smoke_qwen_5_v2/family_summary.csv645 B
runs/smoke_qwen_5_v2/summary.csv374 B
runs/smoke_qwen_5_v2/task_summary.csv1.5 kB

Reproduce

The run commands are documented inside the experiment folder (see the README).

Browse the experiment folder on GitHub ↗

Qwen Recursive Ephemeral Program Induction

Results at a glance 6

Overview

Report

Question

Setup

Main Result

Interpretation

Program Diagnostics

Charts

Task Details

Family Summary

Limitations

Artifacts

Experiment log 6

Setup

Run smoke_no_qwen

Run smoke_qwen_3

Run smoke_qwen_5_v2

Run smoke_qwen_5_helpers

Run main_v1

Figures 5

Data files 18

Reproduce

Related

Related experiments

Run `smoke_no_qwen`

Run `smoke_qwen_3`

Run `smoke_qwen_5_v2`

Run `smoke_qwen_5_helpers`

Run `main_v1`