Qwen Active Crystallizer Public Gate

Finished2026-06-27imported · line YActive Evidence Acquisition Benchmark Generalization Structured Execution and Compilers GitHub ↗

The one idea you need

Spreadsheets can watch you fill a few cells and infer one rule for the whole column. Here, thousands of candidate rules fit the shown examples, and a small model's yes/no votes on invented rows had to pick the single rule that holds on every future row.

The question

A model can usually reformat one text example correctly. Can its yes/no votes be distilled into a single fixed rule that reliably handles every future row?

What we found

No. Using the model's votes to choose a rule worked on 25% of tasks — barely above the 22.5% you get from scrambled, meaningless votes, and it never beat the best rule the candidate pool could offer. The model answered individual unseen rows correctly about 80% of the time, yet almost never turned that skill into one dependable rule.

Why it matters

A model being right row by row does not mean you can compile it into one reusable rule. If you need a rule that holds on every input, test whole-task consistency, not sampled rows — the per-row skill collapses.

Model votes picked a fully general rule25% vs 22.5%real model votes versus scrambled votes choosing a winning rule

Correct on a single unseen rowabout 80%the model answering one unseen row at a time

Correct on every row of a task37.5% vs 12.5%model answering directly versus the model-picked rule, all rows must match

Best any candidate rule could reach29.4%ceiling if you always picked the perfect available rule

On this page

Results at a glance
Overview
Report
Experiment log
Figures
Data files
Reproduce
Related

Results at a glance 6

Picking a rule with model votes versus answering row by row

How to read

Each bar is one method; height is the share of tasks solved on unseen rows, taller is better. Bars cover model votes, scrambled votes, plain shortest-rule picking, the best rule available, and the model answering one row directly.

Takeaway → The model-vote bar sits level with the scrambled-vote bar and the best-available ceiling, all far below the tall direct-answering bar.

Data table

method	held-out success rate
direct Qwen, one held-out row	70%
candidate oracle ceiling	25%
Qwen probe-label selection	25%
shuffled-label control	22.5%
examples-only shortest	20.8%

Numbers from analysis/qwen_summary.csv

Technical framing

Program selection vs direct answering on the Qwen-probe subset (120 tasks) — Qwen probe labels reach the 25% oracle ceiling but add only 2.5pp over shuffled labels, far below Qwen's 70% direct row answering.

Getting each row right versus getting a whole task right

How to read

Three bars on the same forty tasks; height is exact-match rate, taller is better. Left: model answering single rows; middle: model getting every row of a task right; right: one picked rule getting every row right.

Takeaway → The tall single-row bar collapses when every row must match, and the picked-rule bar collapses furthest, showing consistency is the weak point.

Data table

metric	exact-match rate
direct Qwen, per-row exact	79.9%
direct Qwen, all rows exact	37.5%
selected program, all rows exact	12.5%

Numbers from analysis/qwen_direct_full_summary.csv; 12.5% (5/40) from report table

Technical framing

Strict diagnostic: row accuracy vs full-task consistency (same 40 tasks) — Direct Qwen gets 79.9% of rows right but only 37.5% of whole tasks; crystallized programs manage just 12.5% on the same tasks.

Fitting the shown examples is easy; generalizing is not

How to read

Three bars, height is share of tasks, taller is better. Left: tasks where some rule fits the shown examples; middle: tasks where a fitting rule also handles unseen rows; right: picking the shortest fitting rule.

Takeaway → Every task has a rule fitting the examples, yet only about a third of those rules generalize, so fitting examples is easy but generalizing is not.

Data table

condition	task rate
some program fits train examples	100%
oracle: a train-fitting program is held-out valid	29.4%
examples-only shortest selection	22.7%

Numbers from analysis/static_summary.csv

Technical framing

Candidate DSL coverage across all static tasks (309 tasks) — Every task has train-fitting programs, but only 29.4% have one that generalizes: the bottleneck is held-out validity, not train fit.

Best-possible rule coverage by kind of task

How to read

Task kinds along the bottom; paired bars show the best rule available versus the plain shortest-rule pick; taller is better. The two largest kinds, date-and-time and numbers, appear first.

Takeaway → The date-and-time and number bars are lowest despite being the biggest groups, dragging overall rule coverage down below simpler kinds.

Data table

Task family (10 largest, by task count)	Candidate oracle coverage	Examples-only shortest program
DateTime (106)	11%	9%
Number (84)	19%	13%
Name (28)	61%	43%
Phone (16)	50%	38%
ShippingCode (10)	80%	70%
City (9)	11%	0%
Email (6)	67%	67%
BillingCode (6)	100%	67%
Address (6)	17%	0%
UserAgent (7)	57%	43%

Numbers from report table (Family Breakdown, reports/qwen_active_crystallizer_public_gate_report.md)

Technical framing

Oracle DSL ceiling vs examples-only selection by task family — The DSL ceiling collapses on the two biggest families, DateTime and Number, so overall coverage is capped well below simpler families.

Results hold steady across three run sizes

How to read

Three run sizes along the bottom; grouped bars show each method's share of tasks solved on unseen rows, taller is better. Bars include model votes, scrambled votes, the best rule available, shortest-rule picking, and direct answering.

Takeaway → Across all three sizes the model-vote bars stay pinned near the scrambled-vote and best-available bars, while direct answering stays far higher.

Data table

Run suite (Qwen-probe tasks)	Candidate oracle	Qwen probe select	Shuffled probe control	Examples shortest	Qwen direct (first row)
Smoke (12 tasks)	50%	33.3%	25%	33.3%	75%
Pilot (50 tasks)	20%	18%	18%	16%	70%
Main (120 tasks)	25%	25%	22.5%	20.8%	70%

Numbers from runs/smoke_v1/qwen_summary.csv, runs/pilot_v1/qwen_summary.csv, runs/main_v1/qwen_summary.csv

Technical framing

Probe-selection methods across smoke, pilot, and main suites — The pattern is stable across suite sizes: probe selection hugs the oracle ceiling with only a thin margin over shuffled labels.

Best-possible rule coverage depends on the task sample

How to read

Three sample sizes along the bottom; paired bars show the best rule available versus the plain shortest-rule pick; taller is better. Only the largest sample gives a stable reading.

Takeaway → The best-available bar swings widely between small samples, so only the full-size run gives a trustworthy estimate of how far rules can reach.

Data table

Static task sample	Candidate oracle coverage	Examples-only shortest program
Smoke (35 tasks)	45.7%	31.4%
Pilot (100 tasks)	18%	13%
Main (309 tasks)	29.4%	22.7%

Numbers from runs/smoke_v1/static_summary.csv, runs/pilot_v1/static_summary.csv, runs/main_v1/static_summary.csv

Technical framing

DSL oracle ceiling estimate depends on the task sample — Oracle coverage swings from 46% to 18% across samples; only the full 309-task run gives a trustworthy 29% ceiling.

In the author’s words from the Report · “Abstract”

This standalone experiment tests whether frozen Qwen probe labels can select a deterministic transformation program from sparse examples. The model labels synthetic train-like probes; held-out benchmark rows are used only for evaluation.

Overview

Standalone experiment testing whether frozen Qwen probe labels can turn sparse example-level transformation behavior into a deterministic program selected from a broad candidate DSL.

Question

Can a model's direct few-shot transformation behavior be crystallized into a single executable program that generalizes across held-out rows?

Method

Use public Microsoft PROSE Transformation.Text tasks.
Split each task into train and held-out rows.
Generate a broad deterministic candidate DSL from train inputs only.
Generate synthetic probe inputs from train inputs only.
Ask frozen Qwen/Qwen3-4B to label selected probes.
Select among train-fitting candidate programs using probe-label agreement.
Compare against examples-only selection, shuffled probe labels, and oracle candidate coverage.

Artifacts

Source: src/qwen_active_crystallizer_public_gate.py
Reports: reports/
Metrics and figures: analysis/
Public benchmark checkout: /workspace/large_artifacts/qwen_active_crystallizer_public_gate/prose-benchmarks

Report

Rendered from reports/qwen_active_crystallizer_public_gate_report.md

Abstract

Method

Dataset: public Microsoft PROSE Transformation.Text tasks.
Split: first 4 examples are train examples; up to 50 following examples are held out.
Candidate DSL: extraction, casing, regex groups, date/time/number utilities, affixes, two-part concatenation, and small finite maps.
Candidate oracle: whether any train-fitting candidate also matches all held-out rows.
Examples-only selector: shortest train-fitting candidate.
Active crystallizer: Qwen labels synthetic train-like probes chosen to maximize candidate disagreement; the selected program is then evaluated on all held-out rows.
Shuffled-label control: same probes, but labels are rotated before program selection.
Direct Qwen baseline: one held-out query per sampled task, not a full-task program-consistency metric.

Run Configuration

Suite: main.
Static candidate tasks: 309.
Qwen-probe tasks: 120.
Qwen model: Qwen/Qwen3-4B.
Max candidates per task: 40000.
Max Qwen probes per task: 4.

Primary Results

Static Candidate Coverage

method	tasks	score
candidate_oracle	309	29.4%
examples_shortest	309	22.7%
has_train_match	309	100.0%

Qwen-Probe Subset

method	tasks	score
candidate_oracle	120	25.0%
examples_shortest	120	20.8%
qwen_probe_select	120	25.0%
shuffled_probe_select	120	22.5%
qwen_direct_first_row	120	70.0%

Strict Direct-Qwen Full-Heldout Diagnostic

This diagnostic uses the same train examples and asks frozen Qwen to answer every held-out row for a capped task subset. A task counts only if every held-out row is exact.

metric	tasks	rows	score
row_exact	40	458	79.9%
full_task_exact	40	458	37.5%

On the same 40 tasks, active selected-program full-task exact is 12.5% (5/40).
Direct Qwen full-task exact is 37.5% (15/40).

Family Breakdown

family	tasks	oracle_coverage	examples_score	train_match_rate
Author	1	0.00	0.00	100.0%
Column	2	0.00	0.00	100.0%
Meteorite	1	0.00	0.00	100.0%
FilePath	1	0.00	0.00	100.0%
Log	4	0.00	0.00	100.0%
Product	2	0.00	0.00	100.0%
Url	1	0.00	0.00	100.0%
City	9	0.11	0.00	100.0%
DateTime	106	0.11	0.09	100.0%
Address	6	0.17	0.00	100.0%
Number	84	0.19	0.13	100.0%
Gender	3	0.33	0.33	100.0%
Rating	2	0.50	0.50	100.0%
EmergencyCall	2	0.50	0.00	100.0%
Language	2	0.50	0.50	100.0%
Phone	16	0.50	0.38	100.0%
UserAgent	7	0.57	0.43	100.0%
Name	28	0.61	0.43	100.0%
Email	6	0.67	0.67	100.0%
ShippingCode	10	0.80	0.70	100.0%
Airline	1	1.00	1.00	100.0%
Abbreviation	1	1.00	1.00	100.0%
BillingCode	6	1.00	0.67	100.0%
Currency	3	1.00	1.00	100.0%
Song	1	1.00	1.00	100.0%
Noise	1	1.00	1.00	100.0%
Team	1	1.00	1.00	100.0%
State	1	1.00	1.00	100.0%
ZipCode	1	1.00	1.00	100.0%

Qwen-Probe Task Examples

task_id	family	features	oracle_covered	examples_full_exact	qwen_probe_full_exact	shuffled_probe_full_exact	qwen_direct_first_exact	probe_count	qwen_program
Abbreviation.000001	Abbreviation	Concatenation,Conditional,Substring	True	True	True	True	True	4	initials(COL0)
BillingCode.000007	BillingCode	Concatenation	True	True	True	True	False	4	affix['',']'](COL0)
DateTime.000003	DateTime	DateTime	True	True	True	True	True	4	number_int(COL0)
DateTime.000004	DateTime	Concatenation,DateTime,Multicolumn	True	True	True	True	True	4	concat' '
DateTime.000013	DateTime	Conditional,DateTime	True	True	True	True	True	4	COL0
DateTime.000103	DateTime	DateTime	True	True	True	True	True	0	word4
DateTime.000104	DateTime	DateTime	True	True	True	True	True	4	title(alpha(COL0))
DateTime.000107	DateTime	DateTime	True	True	True	True	True	4	word2
EmergencyCall.000003	EmergencyCall	Casing,Substring	True	False	True	True	True	4	title(field;,1)
Gender.000003	Gender	Conditional	True	True	True	True	True	1	mapl->0,m->1,k->2)
Language.000002	Language	Multicolumn,Substring	True	True	True	True	True	4	word3
Name.000017	Name	Substring	True	True	True	True	False	4	title(slice0:4)
Name.000027	Name	Substring	True	True	True	True	True	4	alpha(COL0)
Name.000028	Name	Substring	True	True	True	True	True	4	word0
Name.000029	Name	Concatenation,Substring	True	False	True	True	True	4	concat', ',word0)
Name.000038	Name	Casing,Concatenation,Substring	True	False	True	True	True	4	affix'','@'))
Number.000007	Number	Numeric,NumericRounding	True	False	True	True	True	4	affix'','0')
Number.000029	Number	Numeric,NumericRounding	True	True	True	True	True	4	number_round10(COL0)
Number.000044	Number	Numeric	True	True	True	True	True	4	number_int(COL0)
Number.000049	Number	Numeric,NumericRounding	True	True	True	False	False	4	number_round100(COL0)
Number.000051	Number	Numeric,NumericRounding	True	True	True	True	True	4	number_round100(COL0)
Number.000077	Number	Numeric,NumericRounding	True	False	True	False	False	4	affix'','0')
Number.000078	Number	Numeric,NumericRounding	True	True	True	True	True	4	number_round10(COL0)
Phone.000011	Phone	Concatenation,Substring	True	True	True	True	True	4	title(COL0)

Qwen-Probe Misses

task_id	family	features	oracle_covered	qwen_probe_full_exact	qwen_direct_first_exact	direct_target	direct_prediction
Address.000002	Address	Substring	False	False	False	880 81th Place	880 81th Place SE
Address.000003	Address	Substring	False	False	True	319 09th Lane	319 09th Lane
Address.000013	Address	Conditional,Substring	False	False	True	89	89
City.000004	City	Conditional	False	False	True	San Francisco	San Francisco
City.000008	City	Conditional	False	False	False
City.000010	City	Conditional,Numeric	False	False	True	9	9
City.000011	City	Conditional	False	False	True	New York City	New York City
Column.000001	Column	Concatenation,Conditional,Substring	False	False	True	Col1	Col1
DateTime.000005	DateTime	Conditional,DateTime	False	False	True	Jun 2027	Jun 2027
DateTime.000007	DateTime	DateTime	False	False	True	Sep 2007	Sep 2007
DateTime.000012	DateTime	Conditional,DateTime	False	False	True	2033	2033
DateTime.000014	DateTime	DateTime	False	False	False	Friday #1 February 2013	Saturday #1 February 2013
DateTime.000015	DateTime	DateTime	False	False	False	Friday, 2013W05	Thursday, 2013W01
DateTime.000017	DateTime	DateTime	False	False	True	30/3/2241	30/3/2241
DateTime.000018	DateTime	DateTime	False	False	True	30 Mar 2241	30 Mar 2241
DateTime.000023	DateTime	DateTime	False	False	True	03302241	03302241
DateTime.000025	DateTime	DateTime	False	False	True	Mar 41	Mar 41
DateTime.000027	DateTime	DateTime	False	False	False	Q1 '2241	Q1 '241
DateTime.000029	DateTime	DateTime	False	False	False	Tuesday	Sunday
DateTime.000032	DateTime	DateTime	False	False	True	10:02 PM	10:02 PM
DateTime.000034	DateTime	DateTime,Substring	False	False	True	2002-09-12 16:15:08	2002-09-12 16:15:08
DateTime.000035	DateTime	DateTime,Multicolumn	False	False	True	2002-09-12 16:15:08	2002-09-12 16:15:08
DateTime.000040	DateTime	DateTime,Substring	False	False	True	January 31, 1846	January 31, 1846
DateTime.000044	DateTime	DateTime	False	False	False	January the 31th 1846	January the 31st 1846

Interpretation

On the Qwen-probe subset, active crystallization selects a full held-out-valid program for 25.0% of tasks, compared with 22.5% for the shuffled-label control and a candidate-oracle ceiling of 25.0%. Frozen Qwen direct answering reaches 70.0% on one held-out row per sampled task, which is a different metric: it measures row-level inference, not whether a single executable program generalizes over all held-out rows. The strict direct-Qwen diagnostic narrows that comparison: direct row accuracy is 79.9%, but full-task consistency drops to 37.5%. Direct Qwen is still ahead of active selected programs on the matched subset, but it is not a solved consistency baseline. Across all static tasks, the candidate DSL has a full-heldout oracle ceiling of 29.4%, while examples-only shortest selection reaches 22.7%. The main failure is not a lack of train-fitting programs: finite maps can fit train examples for every task. The failure is that most train-fitting programs are not held-out-valid, and Qwen probe labels only add a small margin over the shuffled-label control. Under this setup, fuzzy model labels did not crystallize Qwen's row-level competence into broadly reliable executable programs.

Limitations

This run uses synthetic probes generated from train inputs, not human-authored counterexamples. The Qwen-probe subset is capped for runtime. Direct Qwen is scored on one held-out row per task, while program methods require one executable program to match all held-out rows.

Artifacts

Static details: analysis/static_details.csv
Qwen-probe details: analysis/qwen_probe_details.csv
Static summary: analysis/static_summary.csv
Qwen summary: analysis/qwen_summary.csv
Strict direct-Qwen full-heldout diagnostic: analysis/qwen_direct_full_summary.csv
Public benchmark checkout: /workspace/large_artifacts/qwen_active_crystallizer_public_gate/prose-benchmarks

Experiment log 9

Show the running log (9 entries)

Setup

Created fresh experiment directory: /workspace/experiments/qwen_active_crystallizer_public_gate
Large artifacts directory: /workspace/large_artifacts/qwen_active_crystallizer_public_gate
Public benchmark checkout is available under the large artifacts directory.
Core question: whether Qwen probe labels can select an executable transformation program from a candidate DSL under held-out validation.
Report format: standalone Markdown and HTML with plots.

Smoke Debug Note

First smoke attempt exposed a duplicate-probe loop in synthetic probe generation before producing metrics. Added an attempt cap and deterministic task seed before rerunning.

Smoke Debug Note 2

Second smoke attempt showed finite-map enumeration was running over concat-expanded candidates. Restricted maps to base/wrapped expressions and lowered concat breadth before rerunning.

Run `smoke_v1`

Started: 2026-06-27 04:36:01 UTC
Static tasks: 35; Qwen-probe tasks: 12
Completed in 23.8s.
Candidate-oracle full-heldout coverage: 31.4%.
Examples-only full-heldout score: 31.4%.
Qwen-probe selected-program score: 33.3%.
Shuffled-label selected-program score: 33.3%.
Direct Qwen first-heldout-row score: 66.7%.

Smoke Debug Note 3

Third inspection showed train-signature deduplication erased candidate ambiguity. Removed that deduplication so probe labels can distinguish train-equivalent programs, and made Qwen tasks sampled by seed.

Run `smoke_v1`

Started: 2026-06-27 04:37:26 UTC
Static tasks: 35; Qwen-probe tasks: 12
Completed in 46.8s.
Candidate-oracle full-heldout coverage: 45.7%.
Examples-only full-heldout score: 31.4%.
Qwen-probe selected-program score: 33.3%.
Shuffled-label selected-program score: 25.0%.
Direct Qwen first-heldout-row score: 75.0%.

Run `pilot_v1`

Started: 2026-06-27 04:38:41 UTC
Static tasks: 100; Qwen-probe tasks: 50
Completed in 170.9s.
Candidate-oracle full-heldout coverage: 18.0%.
Examples-only full-heldout score: 13.0%.
Qwen-probe selected-program score: 18.0%.
Shuffled-label selected-program score: 18.0%.
Direct Qwen first-heldout-row score: 70.0%.

Run `main_v1`

Started: 2026-06-27 04:42:01 UTC
Static tasks: 309; Qwen-probe tasks: 120
Completed in 384.9s.
Candidate-oracle full-heldout coverage: 29.4%.
Examples-only full-heldout score: 22.7%.
Qwen-probe selected-program score: 25.0%.
Shuffled-label selected-program score: 22.5%.
Direct Qwen first-heldout-row score: 70.0%.

Strict Direct-Qwen Full-Heldout Diagnostic

Model: Qwen/Qwen3-4B
Tasks: 40; held-out rows: 458
Row exact: 79.9% (366/458).
Full-task exact: 37.5% (15/40).
Full-task exact requires every held-out row for the task to be answered exactly.

Figures 5

direct full consistency · analysis/figures/

failure decomposition · analysis/figures/

family oracle coverage · analysis/figures/

train match ambiguity · analysis/figures/

Data files 9

Result tables and metrics copied from the experiment folder — preview inline or open the raw file.

Reproduce

This entry is source code or analysis only — there is no separate run to reproduce.

Browse the experiment folder on GitHub ↗

Qwen Active Crystallizer Public Gate

Results at a glance 6

Overview

Question

Method

Artifacts

Report

Abstract

Method

Run Configuration

Primary Results

Static Candidate Coverage

Qwen-Probe Subset

Strict Direct-Qwen Full-Heldout Diagnostic

Family Breakdown

Qwen-Probe Task Examples

Qwen-Probe Misses

Interpretation

Limitations

Artifacts

Experiment log 9

Setup

Smoke Debug Note

Smoke Debug Note 2

Run smoke_v1

Smoke Debug Note 3

Run smoke_v1

Run pilot_v1

Run main_v1

Strict Direct-Qwen Full-Heldout Diagnostic

Figures 5

Data files 9

Reproduce

Related

Related experiments

Run `smoke_v1`

Run `smoke_v1`

Run `pilot_v1`

Run `main_v1`