Qwen Recursive Task Decomposition

Finished2026-06-27imported · line YBenchmark Generalization GitHub ↗

The one idea you need

Shown four worked examples of a text reformat — say turning "12 Jan 2002" into "2002-01-12" — a small model can tackle each new case fresh, or first write one fixed recipe and follow it blindly on every case. This tests whether committing to that recipe helps.

The question

If a small model writes one reusable rule from a few examples and applies it to every new case, does it stay more accurate than solving each case fresh?

What we found

No. Forcing a 4-billion-parameter model to commit to one written recipe lowered accuracy: it got 59% of individual cases right versus 69% when answering each fresh, and 43% of whole tasks perfectly versus 47%. The recipes weren't nonsense — one borrowed from an unrelated task scored even lower — but a single fixed recipe still can't beat the model's case-by-case judgment.

Why it matters

Don't make a small model freeze one rule and apply it mechanically; its per-case reasoning beats any single rule it can articulate. And checking the rule against the practice examples cannot catch this — every failing rule fit the examples perfectly.

Individual cases right: fresh vs one fixed recipe69% → 59%new cases exactly correct when answering each directly vs committing to one written recipe

Whole tasks fully correct: fresh vs one fixed recipe47% → 43%share of tasks where every new case is exact

Recipe borrowed from an unrelated task58%individual cases right — below both real approaches, so the recipes do carry genuine task signal, just not enough

Best-possible step breakdown vs flat formula34% vs 29%top achievable whole-task accuracy from an automatic formula library, but the version actually picked from examples ties the flat one at 23%

On this page

Results at a glance
Overview
Report
Experiment log
Figures
Data files
Reproduce
Related

Results at a glance 7

Committing to one recipe does not beat solving cases one at a time

How to read

Three approaches along the bottom: solve each case fresh, commit to one recipe first, or borrow a recipe from an unrelated task. Two bars each — one counts individual cases right, the other counts tasks where every case is right. Taller is better.

Takeaway → Solving fresh is tallest on both counts; committing to one recipe drops individual-case accuracy from 69% to 59%, and a borrowed recipe falls slightly lower still.

Data table

Qwen arm (30 tasks)	full-task exact	row exact
direct	46.7%	69.4%
locked rule	43.3%	59.4%
shuffled rule	33.3%	58.3%

Numbers from experiments/qwen_recursive_task_decomposition/analysis/qwen_summary.csv

Technical framing

Frozen Qwen: locking in a rule does not beat answering rows directly — Committing to one reusable rule loses 10pp of row accuracy vs direct answering; shuffled rules drop further, so rules carry signal but hurt.

Breaking a formula into steps raises the ceiling, not what you can pick

How to read

Share of tasks a fixed formula library solves perfectly, taller better: a one-shot formula and a step-by-step formula, each shown as its best-possible pick and the pick chosen from the few examples, plus a scrambled-labels control.

Takeaway → Step-by-step lifts the best-possible score from 29% to 34%, but the version actually chosen from examples ties the one-shot at 23%, and the scrambled control collapses near zero.

Data table

static method (309 tasks)	full-task exact
monolithic (examples)	22.7%
monolithic (oracle)	29.4%
recursive (examples)	22.7%
recursive (oracle)	34.3%
recursive (shuffled labels)	0.3%

Numbers from experiments/qwen_recursive_task_decomposition/analysis/static_summary.csv

Technical framing

Static executable decomposition: recursion lifts the oracle, not example selection — Recursive trees raise the held-out oracle from 29.4% to 34.3%, but example-selected recursion ties monolithic at 22.7%; the shuffled control collapses to 0.3%.

Writing three recipe styles and keeping the best still did not recover accuracy

How to read

Individual-case accuracy for the same three approaches, taller better. Two bars each compare a first attempt using one recipe against a second attempt that writes three recipe styles and keeps the one best matching the examples.

Takeaway → Even keeping the best-matching recipe, the recipe approach stays below solving cases one at a time, so more recipe attempts did not help.

Data table

Qwen arm (same 30 tasks)	v1 single rule (row exact)	v2 train-verified (row exact)
direct	69.4%	69.4%
locked rule	61.7%	59.4%
shuffled rule	53.6%	58.3%

Numbers from experiments/qwen_recursive_task_decomposition/analysis/iteration_comparison.csv

Technical framing

Train-verifying rules did not recover the lost row accuracy — Generating three rule styles and picking the one that best fits the training examples still leaves locked-rule row accuracy below direct answering.

Step-by-step formulas only help cleanly separated task types

How to read

For each type of task, two bars show the best-possible perfect-task rate, taller better: a one-shot formula versus a step-by-step formula.

Takeaway → Step-by-step jumps to 100% on neatly delimited types like logs and web addresses, while dates and numbers barely move.

Data table

Task family	Monolithic oracle	Recursive oracle
Log	0%	100%
Url	0%	100%
Language	50%	100%
Address	17%	50%
Phone	50%	56%
Name	61%	64%
DateTime	11%	14%
Number	19%	21%

Numbers from report table (Static Coverage By Family), reports/qwen_recursive_task_decomposition_report.md

Technical framing

Where recursive decomposition lifts the static oracle, by task family — Recursion's oracle gain is concentrated in delimiter-structured families (Log, Url, Language, Address); DateTime and Number barely move.

Picking a recipe from a few examples loses most of the step-by-step gain

How to read

For the ten largest task types, two bars: the best step-by-step recipe available versus the one actually chosen from the few examples. Taller is better.

Takeaway → Example-picking loses the entire gain on addresses and cities and a large share on names and phones, keeping full coverage only on emails.

Data table

Task family (10 largest)	Recursive oracle	Recursive, selected on train examples
DateTime	14%	9%
Number	21%	13%
Name	64%	43%
Phone	56%	38%
ShippingCode	80%	70%
City	11%	0%
UserAgent	57%	43%
Email	67%	67%
Address	50%	0%
BillingCode	100%	67%

Numbers from report table (Static Coverage By Family), reports/qwen_recursive_task_decomposition_report.md

Technical framing

Static selection gap: oracle vs train-example-selected recursive programs, by family — Train-example selection loses the whole recursive gain in Address and City and a large share in Name, Phone, and BillingCode; only Email keeps oracle-level coverage.

Locking a recipe rescues two tasks but wrecks several already solved

How to read

For each task where locking a recipe changed the outcome, two bars compare individual-case accuracy: solving fresh versus committing to one recipe. Taller is better.

Takeaway → Locking rescues two date tasks but collapses number tasks the model had solved outright, dropping some from 100% to 17% or even 0%.

Data table

Task (only tasks where locking changed row accuracy)	Direct answering	Locked rule
DateTime.000076	66.7%	100%
DateTime.000114	0%	16.7%
DateTime.000116	50%	33.3%
Number.000075	66.7%	50%
DateTime.000017	100%	66.7%
DateTime.000051	33.3%	0%
Number.000077	33.3%	0%
City.000011	75%	25%
Number.000016	83.3%	0%
Number.000022	100%	16.7%

Numbers from report table (Qwen Task-Level Details), reports/qwen_recursive_task_decomposition_report.md

Technical framing

Per-task effect of locking a rule: helps 2 tasks, damages 8 — Rule locking rescues two DateTime tasks but collapses Number tasks Qwen solved directly (100% to 16.7%, 83.3% to 0%).

Matching the practice examples cannot tell which recipes will fail

How to read

For the ten worst tasks, two bars: how perfectly the recipe matched the practice examples versus how well it then worked on new cases. Taller is better.

Takeaway → Every recipe matched the practice examples perfectly yet many scored zero on new cases, so fitting the examples cannot flag a recipe that will fail.

Data table

Task (10 lowest locked-rule scores)	Rule accuracy on training examples	Locked-rule held-out row exact
DateTime.000051	100%	0%
DateTime.000115	100%	0%
Number.000016	100%	0%
Number.000077	100%	0%
DateTime.000114	100%	16.7%
Number.000022	100%	16.7%
City.000011	100%	25%
Number.000049	100%	25%
Address.000002	100%	33.3%
BillingCode.000007	100%	33.3%

Numbers from runs/main_v2_train_verified_rules/qwen_rules.csv (train_verify_exact) and analysis/qwen_task_details.csv (locked_row_exact)

Technical framing

Train verification is a blind filter: every rule fits training yet fails held-out rows — All 30 selected rules scored 1.0 on training rows, so train verification cannot distinguish rules that will fail on held-out rows.

In the author’s words from the Report · “Abstract”

This standalone experiment tests whether recursive task decomposition improves task-level consistency on public text-transformation tasks. It compares executable recursive decomposition trees with a frozen language-model rule-locking procedure: first infer one reusable rule, then apply that same rule to every held-out row.

Overview

Standalone experiment testing whether recursive task decomposition improves task-level consistency on public text-transformation tasks.

Question

Given a few input-output examples, can a solver decompose the transformation into reusable substeps and then apply the same decomposition consistently across all held-out rows?

Main Arms

direct_qwen: frozen Qwen answers each held-out row directly from the examples.
locked_rule_qwen: frozen Qwen first writes a reusable rule/decomposition, then answers every held-out row while conditioned on that same rule.
shuffled_rule_qwen: same as locked_rule_qwen, but uses a rule from another task.
static_mono_examples: shortest monolithic train-fitting expression from a deterministic expression library.
static_mono_oracle: held-out oracle over monolithic train-fitting expressions.
static_recursive_examples: shortest recursive output-template decomposition that fits train examples.
static_recursive_oracle: held-out oracle over recursive train-fitting decompositions.
static_recursive_shuffled: recursive decomposition fit on rotated train labels, then evaluated on the real held-out labels.

Layout

src/qwen_recursive_task_decomposition.py: experiment runner, analysis, and report generator.
runs/: raw per-run outputs.
analysis/: consolidated CSVs and figures.
reports/: Markdown and HTML reports.

Report

Rendered from reports/qwen_recursive_task_decomposition_report.md

Abstract

Method

Dataset: public Microsoft PROSE Transformation.Text tasks.
Split: first 4 examples are training examples; up to 50 held-out examples are scored for static methods.
Primary metric: full-task exact. A task is correct only if every held-out row is exact.
Static recursive decomposition: derive output templates from delimiters or token/literal structure, synthesize child transformations for each slot, and compose the children into a tree.
Qwen locked-rule decomposition: frozen Qwen writes one reusable rule/decomposition from the examples, then answers each held-out row while conditioned on that same rule.
Shuffled controls: static recursive synthesis on rotated labels, and Qwen application with a rule from another task.

Run Configuration

Suite: main_v2_train_verified_rules.
Static tasks: 309.
Qwen tasks: 30.
Qwen model: Qwen/Qwen3-4B.
Static max candidates: 12000.
Recursive depth: 2; child limit: 5.
Qwen held-out cap: 6 rows per task.

Primary Results

Static Executable Decomposition

method	tasks	full_exact	row_exact
static_mono_examples	309	22.7%	27.2%
static_mono_oracle	309	29.4%	n/a
static_recursive_examples	309	22.7%	27.2%
static_recursive_oracle	309	34.3%	n/a
static_recursive_shuffled	309	0.3%	0.4%

Frozen Qwen Decomposition

method	tasks	full_exact	row_exact
direct_qwen	30	46.7%	69.4%
locked_rule_qwen	30	43.3%	59.4%
shuffled_rule_qwen	30	33.3%	58.3%

Qwen Iteration Comparison

The first Qwen iteration used one generated decomposition rule per task. The second iteration generated three rule styles and selected the rule that best reproduced the training examples before held-out application. Both iterations used the same 30-task sample.

iteration	method	tasks	full_exact	row_exact
v1_single_rule	direct_qwen	30	46.7%	69.4%
v1_single_rule	locked_rule_qwen	30	46.7%	61.7%
v1_single_rule	shuffled_rule_qwen	30	36.7%	53.6%
v2_train_verified	direct_qwen	30	46.7%	69.4%
v2_train_verified	locked_rule_qwen	30	43.3%	59.4%
v2_train_verified	shuffled_rule_qwen	30	33.3%	58.3%

Static Coverage By Family

family	tasks	mono_oracle	recursive_oracle	mono_examples	recursive_examples
Column	2	0.00	0.00	0.00	0.00
Product	2	0.00	0.00	0.00	0.00
Author	1	0.00	0.00	0.00	0.00
FilePath	1	0.00	0.00	0.00	0.00
Meteorite	1	0.00	0.00	0.00	0.00
City	9	0.11	0.11	0.00	0.00
DateTime	106	0.11	0.14	0.09	0.09
Number	84	0.19	0.21	0.13	0.13
Gender	3	0.33	0.33	0.33	0.33
Address	6	0.17	0.50	0.00	0.00
EmergencyCall	2	0.50	0.50	0.00	0.00
Rating	2	0.50	0.50	0.50	0.50
Phone	16	0.50	0.56	0.38	0.38
UserAgent	7	0.57	0.57	0.43	0.43
Name	28	0.61	0.64	0.43	0.43
Email	6	0.67	0.67	0.67	0.67
ShippingCode	10	0.80	0.80	0.70	0.70
BillingCode	6	1.00	1.00	0.67	0.67
Log	4	0.00	1.00	0.00	0.00
Currency	3	1.00	1.00	1.00	1.00
Language	2	0.50	1.00	0.50	0.50
Abbreviation	1	1.00	1.00	1.00	1.00
Airline	1	1.00	1.00	1.00	1.00
Noise	1	1.00	1.00	1.00	1.00
Song	1	1.00	1.00	1.00	1.00
State	1	1.00	1.00	1.00	1.00
Team	1	1.00	1.00	1.00	1.00
Url	1	0.00	1.00	0.00	0.00
ZipCode	1	1.00	1.00	1.00	1.00

Qwen Task-Level Details

task_id	family	heldout_rows	direct_row_exact	locked_row_exact	shuffled_row_exact	direct_full_exact	locked_full_exact	shuffled_full_exact	locked_minus_direct_rows
DateTime.000076	DateTime	6	66.7%	100.0%	66.7%	False	True	False	0.33
City.000010	City	3	100.0%	100.0%	100.0%	True	True	True	0.00
Column.000001	Column	6	100.0%	100.0%	100.0%	True	True	True	0.00
DateTime.000004	DateTime	6	100.0%	100.0%	100.0%	True	True	True	0.00
DateTime.000007	DateTime	6	100.0%	100.0%	100.0%	True	True	True	0.00
DateTime.000025	DateTime	6	100.0%	100.0%	66.7%	True	True	False	0.00
DateTime.000094	DateTime	4	100.0%	100.0%	50.0%	True	True	False	0.00
DateTime.000104	DateTime	6	100.0%	100.0%	100.0%	True	True	True	0.00
DateTime.000108	DateTime	6	100.0%	100.0%	83.3%	True	True	False	0.00
FilePath.000001	FilePath	6	100.0%	100.0%	100.0%	True	True	True	0.00
Language.000002	Language	6	100.0%	100.0%	100.0%	True	True	True	0.00
Phone.000011	Phone	3	100.0%	100.0%	100.0%	True	True	True	0.00
UserAgent.000003	UserAgent	6	100.0%	100.0%	100.0%	True	True	True	0.00
DateTime.000114	DateTime	6	0.0%	16.7%	0.0%	False	False	False	0.17
Address.000002	Address	3	33.3%	33.3%	33.3%	False	False	False	0.00
BillingCode.000007	BillingCode	3	33.3%	33.3%	33.3%	False	False	False	0.00
DateTime.000081	DateTime	6	50.0%	50.0%	33.3%	False	False	False	0.00
DateTime.000115	DateTime	6	0.0%	0.0%	0.0%	False	False	False	0.00
Gender.000001	Gender	3	66.7%	66.7%	66.7%	False	False	False	0.00
Number.000008	Number	6	33.3%	33.3%	33.3%	False	False	False	0.00
Number.000049	Number	4	25.0%	25.0%	25.0%	False	False	False	0.00
ShippingCode.000008	ShippingCode	3	33.3%	33.3%	0.0%	False	False	False	0.00
DateTime.000116	DateTime	6	50.0%	33.3%	33.3%	False	False	False	-0.17
Number.000075	Number	6	66.7%	50.0%	50.0%	False	False	False	-0.17
DateTime.000017	DateTime	6	100.0%	66.7%	50.0%	True	False	False	-0.33
DateTime.000051	DateTime	3	33.3%	0.0%	33.3%	False	False	False	-0.33
Number.000077	Number	3	33.3%	0.0%	100.0%	False	False	True	-0.33
City.000011	City	4	75.0%	25.0%	25.0%	False	False	False	-0.50
Number.000016	Number	6	83.3%	0.0%	33.3%	False	False	False	-0.83
Number.000022	Number	6	100.0%	16.7%	33.3%	True	False	False	-0.83

Example Locked Rules

task_id	family	rule_chars	rule
Address.000002	Address	245	Take the input and split it into parts using commas as delimiters. The first part contains the name and a number followed by a place and direction. Extract the number and place portion, then take the first part of the split result as the output.
BillingCode.000007	BillingCode	116	Keep the input as is.

Split the input into components: Separate the date into month, day, and year (or equivalent parts).
Map the month abbreviation to its full name: Convert abbreviations like "Mar" to "March" (though examples use abbreviations, this step is for consistency).
Reformat the day: Convert the day into a two-digit format if necessary (e.g., "1" → "01", "30" → "30").
Reformat the year: Extract the last two digits of the year (e.g., "2002" → "02", "1953" → "53|

|DateTime.000051|DateTime|365|If the input is a time in HH:MM:SS format: If the hour is 11, then format as "11:00AM-11:30AM"; If the hour is 8, then format as "8:00AM-8:30AM"; If the hour is 12, then format as "12:00PM-12:30PM"; If the hour is 0, then format as "12:00AM-12:30AM"; If the input is empty or null, then return "N/A"; If the input is an abbreviation like "11:00AM", then| |DateTime.000076|DateTime|235|Take the input time in the format HH:MM. Extract the hour part (HH). If the hour is between 00 and 19, output "0:00-19:00". If the hour is between 20 and 39, output "20:00-39:00". If the hour is between 40 and 59, output "40:00-59:00".| |DateTime.000081|DateTime|434|Rule: Given a time in HH:MMAM/PM format, decompose into:

Extract hour and minute.
If minute < 15, subtract (minute - 0) from hour, add 15 minutes, and set the new start time.
If minute >= 15 and < 45, set start time to hour:15 and end time to hour:45.
If minute >= 45, add (60 - minute) to hour, set start time to (hour + 1):15 and end time to (hour + 1):45.
Combine start and end times into "start-time|

|DateTime.000094|DateTime|281|Parse the input date into components (year, month, day). If the date is in the format "DD.MM.YYYY" or "DD. MMM YYYY", extract the day, month, and year. If the date is in the format "MMM DD YYYY" or "DD MMM YYYY", extract the day, month, and year. Format the output as "YYYY-MM-DD".| |DateTime.000104|DateTime|241|Extract the third character from the date portion of the input string. The date portion is in the format "DD-Mon-YYYY", so the third character is the first letter of the month (e.g., "Jan", "Feb", "Mar"). Return that character as the output.| |DateTime.000108|DateTime|243|Parse the input date and time string. Extract the day and month from the date part. Format the output as MM/DD/YYYY, where MM is the month abbreviation (Jan=1, Feb=2, ..., Dec=12), DD is the day, and YYYY is the year. The time part is ignored.| |DateTime.000114|DateTime|149|Take the day of the month from the input date. If the day is less than or equal to 15, output "15-30". If the day is greater than 15, output "30-45".|

Figures

Interpretation

Executable recursion changes the static oracle from 29.4% to 34.3%. The examples-selected recursive tree reaches 22.7%, compared with 22.7% for monolithic expressions and 0.3% for the shuffled-label control. For frozen Qwen, direct answering reaches 69.4% row exact and 46.7% full-task exact. The locked-rule decomposition reaches 59.4% row exact and 43.3% full-task exact. The shuffled-rule control reaches 58.3% row exact and 33.3% full-task exact. The locked-rule full-task delta over direct answering is -3.3%. Positive values mean decomposition improved consistency; negative values mean rule commitment damaged useful row-level inference. The iteration comparison is negative for this form of recursive decomposition. A single generated rule tied direct Qwen on full-task exact but reduced row exact. Train-verifying three rule styles did not recover the lost row competence and lowered full-task exact by one task. The shuffled-rule controls are lower than the real-rule arms, so the generated rules carry task signal; the problem is that the signal is not reliably better than Qwen's direct row-level inference.

Limitations

The Qwen arm is capped for runtime and uses deterministic decoding. The static recursive tree only explores delimiter and token-template decompositions, not arbitrary semantic subgoals. Full-task exact is intentionally strict and can be much lower than row exact when a method is inconsistent across rows.

Artifacts

Static details: analysis/static_details.csv
Qwen rules: analysis/qwen_rules.csv
Qwen task summary: analysis/qwen_task_details.csv
Qwen row details: analysis/qwen_row_details.csv
Iteration comparison: analysis/iteration_comparison.csv
Figures: analysis/figures/
Benchmark mirror: /workspace/large_artifacts/qwen_recursive_task_decomposition/prose-benchmarks

Experiment log 6

Show the running log (6 entries)

Setup

Created a fresh standalone experiment directory.
Target benchmark: public Microsoft PROSE Transformation.Text tasks mirrored under /workspace/large_artifacts/qwen_recursive_task_decomposition/prose-benchmarks.
Primary metric: full-task exact on held-out rows. A task counts only if every held-out row is exact.
Secondary metric: row exact, used only to diagnose task-level consistency.

Run `smoke_static`

Time UTC: 2026-06-27T05:12:36.848797+00:00
Static tasks: 12; Qwen tasks: 0
Config: {"answer_max_new_tokens": 64, "child_limit": 4, "heldout_cap": 8, "max_candidates": 8000, "qwen_heldout_cap": 6, "qwen_min_heldout": 3, "qwen_task_limit": 12, "recursive_depth": 2, "rule_max_new_tokens": 180, "run_qwen": false, "sample_seed": 20260627, "suite": "smoke_static", "task_limit": 12, "train_n": 4}
Static summary:
- static_mono_examples full-task exact: 33.3%
- static_mono_oracle full-task exact: 50.0%
- static_recursive_examples full-task exact: 33.3%
- static_recursive_oracle full-task exact: 50.0%
- static_recursive_shuffled full-task exact: 0.0%

Run `smoke_qwen`

Time UTC: 2026-06-27T05:13:59.780118+00:00
Static tasks: 20; Qwen tasks: 4
Config: {"answer_max_new_tokens": 48, "child_limit": 3, "heldout_cap": 6, "max_candidates": 5000, "qwen_heldout_cap": 3, "qwen_min_heldout": 3, "qwen_task_limit": 4, "recursive_depth": 1, "rule_max_new_tokens": 120, "run_qwen": true, "sample_seed": 20260627, "suite": "smoke_qwen", "task_limit": 20, "train_n": 4}
Static summary:
- static_mono_examples full-task exact: 30.0%
- static_mono_oracle full-task exact: 50.0%
- static_recursive_examples full-task exact: 30.0%
- static_recursive_oracle full-task exact: 50.0%
- static_recursive_shuffled full-task exact: 0.0%
Qwen summary:
- direct_qwen row exact: 75.0%; full-task exact: 50.0%
- locked_rule_qwen row exact: 58.3%; full-task exact: 50.0%
- shuffled_rule_qwen row exact: 58.3%; full-task exact: 25.0%

Run `main_v1`

Time UTC: 2026-06-27T05:35:27.915748+00:00
Static tasks: 309; Qwen tasks: 30
Config: {"answer_max_new_tokens": 56, "child_limit": 5, "heldout_cap": 50, "max_candidates": 12000, "qwen_heldout_cap": 6, "qwen_min_heldout": 3, "qwen_task_limit": 30, "recursive_depth": 2, "rule_max_new_tokens": 160, "run_qwen": true, "sample_seed": 20260627, "suite": "main_v1", "task_limit": 0, "train_n": 4}
Static summary:
- static_mono_examples full-task exact: 22.7%
- static_mono_oracle full-task exact: 29.4%
- static_recursive_examples full-task exact: 22.7%
- static_recursive_oracle full-task exact: 34.3%
- static_recursive_shuffled full-task exact: 0.3%
Qwen summary:
- direct_qwen row exact: 69.4%; full-task exact: 46.7%
- locked_rule_qwen row exact: 61.7%; full-task exact: 46.7%
- shuffled_rule_qwen row exact: 53.6%; full-task exact: 36.7%

Run `syntax_check`

Time UTC: 2026-06-27T05:36:58.054670+00:00
Static tasks: 3; Qwen tasks: 0
Config: {"answer_max_new_tokens": 64, "child_limit": 2, "heldout_cap": 3, "max_candidates": 1000, "qwen_heldout_cap": 6, "qwen_min_heldout": 3, "qwen_task_limit": 12, "recursive_depth": 1, "rule_max_new_tokens": 180, "rule_styles": "decompose", "run_qwen": false, "sample_seed": 20260627, "suite": "syntax_check", "task_limit": 3, "train_n": 4, "train_verify_rules": false}
Static summary:
- static_mono_examples full-task exact: 33.3%
- static_mono_oracle full-task exact: 33.3%
- static_recursive_examples full-task exact: 33.3%
- static_recursive_oracle full-task exact: 33.3%
- static_recursive_shuffled full-task exact: 0.0%

Run `main_v2_train_verified_rules`

Time UTC: 2026-06-27T05:49:25.174356+00:00
Static tasks: 309; Qwen tasks: 30
Config: {"answer_max_new_tokens": 56, "child_limit": 5, "heldout_cap": 50, "max_candidates": 12000, "qwen_heldout_cap": 6, "qwen_min_heldout": 3, "qwen_task_limit": 30, "recursive_depth": 2, "reuse_static_from": "main_v1", "rule_max_new_tokens": 150, "rule_styles": "decompose,terse,conditional", "run_qwen": true, "sample_seed": 20260627, "suite": "main_v2_train_verified_rules", "task_limit": 0, "train_n": 4, "train_verify_rules": true}
Static summary:
- static_mono_examples full-task exact: 22.7%
- static_mono_oracle full-task exact: 29.4%
- static_recursive_examples full-task exact: 22.7%
- static_recursive_oracle full-task exact: 34.3%
- static_recursive_shuffled full-task exact: 0.3%
Qwen summary:
- direct_qwen row exact: 69.4%; full-task exact: 46.7%
- locked_rule_qwen row exact: 59.4%; full-task exact: 43.3%
- shuffled_rule_qwen row exact: 58.3%; full-task exact: 33.3%

Figures 5

iteration comparison · analysis/figures/

method full task scores · analysis/figures/

qwen family locked effect · analysis/figures/

static recursive gain · analysis/figures/

Data files 12

Result tables and metrics copied from the experiment folder — preview inline or open the raw file.

Reproduce

This entry is source code or analysis only — there is no separate run to reproduce.

Browse the experiment folder on GitHub ↗

Qwen Recursive Task Decomposition

Results at a glance 7

Overview

Question

Main Arms

Layout

Report

Abstract

Method

Run Configuration

Primary Results

Static Executable Decomposition

Frozen Qwen Decomposition

Qwen Iteration Comparison

Static Coverage By Family

Qwen Task-Level Details

Example Locked Rules

Figures

Interpretation

Limitations

Artifacts

Experiment log 6

Setup

Run smoke_static

Run smoke_qwen

Run main_v1

Run syntax_check

Run main_v2_train_verified_rules

Figures 5

Data files 12

Reproduce

Related

Related experiments

Run `smoke_static`

Run `smoke_qwen`

Run `main_v1`

Run `syntax_check`

Run `main_v2_train_verified_rules`