Research log Small Model Experimentation
GitHub

Noisy Row Program Crystallizer

Finished2026-06-27imported · line YStructured Execution and CompilersGitHub ↗
Distilling noisy guesses into one rule backfires

The one idea you need

Picture a hasty clerk reformatting every entry in a list, typing each one a little differently each pass. This hunts for a single fixed recipe that fits the four entries you can check and best matches the clerk's noisy retypes on all the rest.

The question

If a model gives messy, inconsistent answers for each row of a text-cleanup task, can you boil those guesses into one clean rule that solves the whole task?

What we found

No. Keeping the model's direct per-row answers fully solved half of the 40 tasks, while distilling those noisy answers into one fixed rule solved just 22.5% — worse even than a scrambled comparison at 25%. And a flawless rule-picker tops out at 27.5%, because for most tasks no short rule both fits the visible samples and holds on the rest.

Why it matters

For per-row text cleanup, trust and vote among the model's direct answers. A single rigid rule is all-or-nothing, and the narrow library of rules caps you below direct answering — reach for a rule only when a simple one truly exists.

Whole tasks fully solved: direct answers vs one rule50% → 22.5%out of 40 text-transformation tasks
Best a flawless rule-picker could reach27.5%hard ceiling of the single-rule approach, still below direct answering
The rule versus a scrambled comparison22.5% vs 25%the rule lost to scrambling, so its signal is not real
A correct answer usually sat in the model's samplesmore than 6 in 10 tasksa fully correct answer existed among the samples but the rule left it unpicked
On this page
  1. Results at a glance
  2. Overview
  3. Report
    1. Question
    2. Setup
    3. Main Result
    4. Interpretation
    5. Diagnostics
    6. Charts
    7. Family Breakdown
    8. Reachable Headroom Tasks
    9. Candidate-Support Changes
    10. Task Details
    11. Files
  4. Experiment log
  5. Figures
  6. Data files
  7. Reproduce
  8. Related

Results at a glance 8

Boiling rows into one rule solves fewer whole tasks than using the rows directly

How to read

Each bar is one method; height is the share of 40 tasks fully solved, taller is better. Left bars keep the model's direct per-row answers; right bars force those answers into a single fixed rule.

0%20%40%60%80%row-candidate oraclerow-candidate oracle62.5%direct greedy rowsdirect greedy rows50%row majority voterow majority vote50%pseudo-program + direct fallbackpseudo-program + direct fallback50%program oracleprogram oracle27.5%examples-only shortest programexamples-only shortest program25%shuffled-support controlshuffled-support control25%candidate-support programcandidate-support program22.5%pseudo-program tablepseudo-program table17.5%shuffled-pseudo controlshuffled-pseudo control0%

Takeaway → Every direct-answer bar towers over every single-rule bar, and the rule bars even dip below a scrambled comparison, so distilling into a rule hurts.

Data table
methodfull-task exact
row-candidate oracle62.5%
direct greedy rows50%
row majority vote50%
pseudo-program + direct fallback50%
program oracle27.5%
examples-only shortest program25%
shuffled-support control25%
candidate-support program22.5%
pseudo-program table17.5%
shuffled-pseudo control0%

Numbers from experiments/qwen_noisy_row_program_crystallizer/analysis/summary.csv

Technical framing

Crystallizing rows into one program loses to just using the rows (full-task exact, 40 tasks) — Selecting a deterministic program via candidate support (22.5%) is far below direct greedy row inference (50%) and even below its shuffled control (25%).

Individual rows right versus whole tasks right, across methods

How to read

Two bars per method: share of individual rows correct and share of whole tasks correct, taller is better. Direct-answer methods sit left, fixed-rule methods and the best-possible pick to the right.

0%20%40%60%80%direct greedy rowsdirect greedy rows72.3%50%row majority voterow majority vote72.7%50%candidate-support programcandidate-support program27.1%22.5%program oracleprogram oracle27.5%27.5%row-candidate oraclerow-candidate oracle62.5%62.5%

Takeaway → Direct answers get most rows right but fewer whole tasks; a rule is all-or-nothing, flattening both to the same low level.

Data table
methodrow exactfull-task exact
direct greedy rows72.3%50%
row majority vote72.7%50%
candidate-support program27.1%22.5%
program oracle27.5%27.5%
row-candidate oracle62.5%62.5%

Numbers from experiments/qwen_noisy_row_program_crystallizer/analysis/summary.csv

Technical framing

Row-level vs strict full-task accuracy: programs are exact or nothing, direct rows are partially right — Direct row inference gets 72% of individual rows right but only 50% of whole tasks; program methods collapse both to the same low rate.

How many of 40 tasks each step can possibly solve

How to read

Each bar is a step; height is tasks solved out of 40, taller is better. Top bars are best-possible ceilings; lower bars are what building a rule actually reaches.

0102030some row candidate is correct (oracle)some row candidate is correct (or…25direct greedy rowsdirect greedy rows20pseudo-program + direct fallbackpseudo-program + direct fallback20train-fitting program exists (oracle)train-fitting program exists (ora…11candidate-support selectioncandidate-support selection9pseudo-program selector alonepseudo-program selector alone7

Takeaway → A fixed rule even exists for only 11 of 40 tasks, below the 20 the direct answers already solve, so the rule library is the bottleneck.

Data table
stagetasks solved
some row candidate is correct (oracle)25
direct greedy rows20
pseudo-program + direct fallback20
train-fitting program exists (oracle)11
candidate-support selection9
pseudo-program selector alone7

Numbers from report table (Diagnostics section, reports/qwen_noisy_row_program_crystallizer_report.md); oracle 25 = 62.5% of 40 from analysis/summary.csv

Technical framing

Program headroom: how many of 40 tasks each stage can solve — The DSL itself is the bottleneck: only 11 of 40 tasks even have a train-fitting deterministic program, below the 20 direct inference already solves.

Whole-task accuracy by task type: direct answers versus one fixed rule

How to read

Task types with their counts run along the axis; paired bars compare using the model's answers directly against forcing them into one rule, taller is better.

0%25%50%75%100%DateTime (16)DateTime (16)56.2%12.5%Number (10)Number (10)30%20%Address (2)Address (2)0%0%City (2)City (2)50%0%Phone (2)Phone (2)100%100%

Takeaway → On the two biggest types the rule collapses far below direct — date tasks fall from 56% to 13% — tying only on tiny, trivially rule-like types.

Data table
task family (count)direct greedy rowscandidate-support program
DateTime (16)56.2%12.5%
Number (10)30%20%
Address (2)0%0%
City (2)50%0%
Phone (2)100%100%

Numbers from experiments/qwen_noisy_row_program_crystallizer/analysis/family_summary.csv

Technical framing

Per-family full-task exact: crystallization hurts most on DateTime and Number, the biggest families — Crystallization only matches direct inference on trivially programmatic families like Phone; it collapses on DateTime (56% to 12.5%).

Do the rule-picking methods beat their own scrambled version?

How to read

Two rule-picking methods on the axis; for each, one bar for the real signal and one for a scrambled control, taller is better. A genuine method should beat its scramble.

0%10%20%30%Candidate-support programCandidate-support program22.5%25%Pseudo-label program tablePseudo-label program table17.5%0%

Takeaway → One method beats its scramble (17.5% vs 0%), but the candidate-vote method ties or trails its scramble (22.5% vs 25%), so its signal is not real.

Data table
Crystallization methodReal support signalShuffled-support control
Candidate-support program22.5%25%
Pseudo-label program table17.5%0%

Numbers from experiments/qwen_noisy_row_program_crystallizer/analysis/summary.csv

Technical framing

Shuffle controls: pseudo-label selection is a real signal, candidate-support voting is not — Shuffling destroys pseudo-label selection (17.5% to 0%) but candidate-support does no better than its shuffled control.

Best answer available versus the model's direct pick, by task type

How to read

Task types on the axis; three bars each — best answer anywhere in the sample pool, a table-wide best, and the model's direct pick — taller is better.

0%20%40%60%80%DateTime (16 tasks)DateTime (16 tasks)62.5%62.5%56.2%Number (10)Number (10)70%60%30%City (2)City (2)50%50%50%Address (2)Address (2)0%0%0%

Takeaway → On number tasks a correct answer sits in the pool 70% of the time but direct picking lands only 30% — real recoverable room the rule fails to capture.

Data table
Task familyRow-candidate oracle (best answer in pool)Table-candidate oracleDirect greedy rows
DateTime (16 tasks)62.5%62.5%56.2%
Number (10)70%60%30%
City (2)50%50%50%
Address (2)0%0%0%

Numbers from experiments/qwen_noisy_row_program_crystallizer/analysis/family_summary.csv

Technical framing

Oracle ceiling vs greedy decoding by family: the recoverable headroom is in Number tasks — On Number tasks the sampled pool contains a fully correct answer 70% of the time but greedy only solves 30%; Address is unreachable either way.

Agreement among answers does not mean the chosen rule is right

How to read

Each bar is one task where candidate voting swapped in a different rule; height is how much of the answer pool backed the winner. All ten shown were wrong on the hidden rows.

0%25%50%75%100%Address.000002Address.0000020%City.000011City.00001164.3%DateTime.000051DateTime.0000510%DateTime.000081DateTime.00008116.7%DateTime.000115DateTime.000115100%Gender.000001Gender.00000190.5%Number.000043Number.0000430%Number.000077Number.00007757.1%ShippingCode.000008ShippingCode.0000080%BillingCode.000007BillingCode.00000771.4%

Takeaway → Vote share ranged from none to unanimous yet every pick was wrong; on one task a 71% majority even overrode a correct rule.

Data table
Task where support changed the selected programVote share of support-selected program (strictly wrong on hidden rows)
Address.0000020%
City.00001164.3%
DateTime.0000510%
DateTime.00008116.7%
DateTime.000115100%
Gender.00000190.5%
Number.0000430%
Number.00007757.1%
ShippingCode.0000080%
BillingCode.00000771.4%

Numbers from report table (Candidate-Support Changes, reports/qwen_noisy_row_program_crystallizer_report.md)

Technical framing

Candidate agreement does not certify correctness: vote share of the 10 support-changed picks, all wrong — Every program that candidate support swapped in was wrong, yet vote share spanned 0-100%; on BillingCode.000007 it even overrode a correct program at 71.4%.

The same loss shows up at both 6 tasks and 40 tasks

How to read

Two run sizes on the axis; three bars each for direct answers, the best-possible rule ceiling, and the built single rule, taller is better.

0%20%40%60%80%Smoke run (6 tasks)Smoke run (6 tasks)66.7%33.3%16.7%Main run (40 tasks)Main run (40 tasks)50%27.5%22.5%

Takeaway → At both sizes the rule ceiling trails direct answers, confirming the rule space itself caps performance rather than being a fluke of a small run.

Data table
Run scaleDirect greedy rowsProgram oracleCandidate-support program
Smoke run (6 tasks)66.7%33.3%16.7%
Main run (40 tasks)50%27.5%22.5%

Numbers from experiments/qwen_noisy_row_program_crystallizer/experiment_log.md

Technical framing

The crystallization loss replicates across task-set sizes (6-task smoke vs 40-task main) — At both 6 and 40 tasks, even the best train-fitting program trails direct rows, so the program space itself is the ceiling.

In the author’s words from the Report · “Interpretation”

Candidate support changes strict full-task exactness by -27.5 points relative to direct greedy row inference and by -2.5 points relative to the shortest train-fitting deterministic program. The shuffled-support control is separated by -2.5 points. The pseudo-label crystallizer reaches 17.5% as a pure table selector and 50.0% with conservative direct fallback. Its shuffled-pseudo control reaches 0.0%. The deterministic train-fitting program oracle solves 27.5% of tasks and the row-candidate oracle solves 62.5%. Relative to the direct-to-program oracle gap, candidate-support gap capture is not defined because the deterministic-program oracle is below the direct row baseline.

Overview

Standalone experiment directory for candidate-supported deterministic program selection.

Report

Rendered from reports/qwen_noisy_row_program_crystallizer_report.md

Question

Can noisy row-level candidate outputs be crystallized into one deterministic program for an entire text-transformation task?

The experiment uses the language model only to propose row outputs. It then enumerates deterministic programs that exactly match the visible examples and selects the program whose held-out predictions receive the strongest support from the row-candidate pool. Hidden outputs are used only for evaluation and oracle diagnostics.

Setup

  • Run: main_final
  • Dataset: public text-transformation tasks.
  • Tasks: 40
  • Visible examples per task: 4
  • Held-out cap per task: 6
  • Max deterministic programs enumerated per task: 40000
  • Max candidate tables per task: 32
  • Pseudo-program support threshold for fallback: 0.75
  • Row-candidate rows used: 1428

Main Result

methodtasksrow_exactfull_task_exact
row_candidate_oracle4062.5%62.5%
table_candidate_oracle4060.0%60.0%
direct_row_greedy4072.3%50.0%
row_majority4072.7%50.0%
pseudo_program_or_direct4072.3%50.0%
program_oracle4027.5%27.5%
examples_shortest_program4025.0%25.0%
shuffled_support_program4028.7%25.0%
candidate_support_program4027.1%22.5%
pseudo_program_table4019.2%17.5%
shuffled_pseudo_program_table400.0%0.0%

Interpretation

Candidate support changes strict full-task exactness by -27.5 points relative to direct greedy row inference and by -2.5 points relative to the shortest train-fitting deterministic program. The shuffled-support control is separated by -2.5 points.

The pseudo-label crystallizer reaches 17.5% as a pure table selector and 50.0% with conservative direct fallback. Its shuffled-pseudo control reaches 0.0%.

The deterministic train-fitting program oracle solves 27.5% of tasks and the row-candidate oracle solves 62.5%. Relative to the direct-to-program oracle gap, candidate-support gap capture is not defined because the deterministic-program oracle is below the direct row baseline.

Diagnostics

  • Direct row inference solves 20 of 40 tasks.
  • A deterministic train-fitting program can solve 11 of 40 tasks.
  • Candidate-support selection solves 9 of 40 tasks.
  • Pseudo-label crystallization solves 7 tasks as a pure selector and 20 with direct fallback.
  • The direct fallback uses the pseudo-program table on 8 tasks.
  • There are 4 direct-missed tasks with a hidden-valid deterministic program; candidate support captures 2 of them.
  • Candidate support changes the selected program on 10 tasks: 0 helped and 1 hurt on strict full-task exactness.

Charts

Full-task exact by method

Row versus full-task exact

Program headroom

Support selection changes

Support vote-share distribution

Family heatmap

Family Breakdown

methodfamilytasksrow_exactfull_task_exact
candidate_support_programAddress20.0%0.0%
candidate_support_programBillingCode133.3%0.0%
candidate_support_programCity225.0%0.0%
candidate_support_programColumn10.0%0.0%
candidate_support_programDateTime1612.5%12.5%
candidate_support_programFilePath1100.0%100.0%
candidate_support_programGender166.7%0.0%
candidate_support_programLanguage1100.0%100.0%
candidate_support_programName1100.0%100.0%
candidate_support_programNumber1023.3%20.0%
candidate_support_programPhone2100.0%100.0%
candidate_support_programShippingCode10.0%0.0%
candidate_support_programUserAgent10.0%0.0%
direct_row_greedyAddress250.0%0.0%
direct_row_greedyBillingCode133.3%0.0%
direct_row_greedyCity287.5%50.0%
direct_row_greedyColumn1100.0%100.0%
direct_row_greedyDateTime1672.9%56.2%
direct_row_greedyFilePath1100.0%100.0%
direct_row_greedyGender166.7%0.0%
direct_row_greedyLanguage1100.0%100.0%
direct_row_greedyName1100.0%100.0%
direct_row_greedyNumber1061.7%30.0%
direct_row_greedyPhone2100.0%100.0%
direct_row_greedyShippingCode133.3%0.0%
direct_row_greedyUserAgent1100.0%100.0%
examples_shortest_programAddress20.0%0.0%
examples_shortest_programBillingCode1100.0%100.0%
examples_shortest_programCity20.0%0.0%
examples_shortest_programColumn10.0%0.0%
examples_shortest_programDateTime1612.5%12.5%
examples_shortest_programFilePath1100.0%100.0%
examples_shortest_programGender10.0%0.0%
examples_shortest_programLanguage1100.0%100.0%
examples_shortest_programName1100.0%100.0%
examples_shortest_programNumber1020.0%20.0%
examples_shortest_programPhone2100.0%100.0%
examples_shortest_programShippingCode10.0%0.0%
examples_shortest_programUserAgent10.0%0.0%
program_oracleAddress20.0%0.0%
program_oracleBillingCode1100.0%100.0%
program_oracleCity20.0%0.0%
program_oracleColumn10.0%0.0%
program_oracleDateTime1612.5%12.5%
program_oracleFilePath1100.0%100.0%
program_oracleGender10.0%0.0%
program_oracleLanguage1100.0%100.0%
program_oracleName1100.0%100.0%
program_oracleNumber1030.0%30.0%
program_oraclePhone2100.0%100.0%
program_oracleShippingCode10.0%0.0%
program_oracleUserAgent10.0%0.0%
pseudo_program_or_directAddress250.0%0.0%
pseudo_program_or_directBillingCode133.3%0.0%
pseudo_program_or_directCity287.5%50.0%
pseudo_program_or_directColumn1100.0%100.0%
pseudo_program_or_directDateTime1672.9%56.2%
pseudo_program_or_directFilePath1100.0%100.0%
pseudo_program_or_directGender166.7%0.0%
pseudo_program_or_directLanguage1100.0%100.0%
pseudo_program_or_directName1100.0%100.0%
pseudo_program_or_directNumber1061.7%30.0%
pseudo_program_or_directPhone2100.0%100.0%
pseudo_program_or_directShippingCode133.3%0.0%
pseudo_program_or_directUserAgent1100.0%100.0%
pseudo_program_tableAddress20.0%0.0%
pseudo_program_tableBillingCode133.3%0.0%
pseudo_program_tableCity20.0%0.0%
pseudo_program_tableColumn10.0%0.0%
pseudo_program_tableDateTime1612.5%12.5%
pseudo_program_tableFilePath1100.0%100.0%
pseudo_program_tableGender10.0%0.0%
pseudo_program_tableLanguage1100.0%100.0%
pseudo_program_tableName1100.0%100.0%
pseudo_program_tableNumber103.3%0.0%
pseudo_program_tablePhone2100.0%100.0%
pseudo_program_tableShippingCode10.0%0.0%
pseudo_program_tableUserAgent10.0%0.0%
row_candidate_oracleAddress20.0%0.0%
row_candidate_oracleBillingCode10.0%0.0%
row_candidate_oracleCity250.0%50.0%
row_candidate_oracleColumn1100.0%100.0%
row_candidate_oracleDateTime1662.5%62.5%
row_candidate_oracleFilePath1100.0%100.0%
row_candidate_oracleGender10.0%0.0%
row_candidate_oracleLanguage1100.0%100.0%
row_candidate_oracleName1100.0%100.0%
row_candidate_oracleNumber1070.0%70.0%
row_candidate_oraclePhone2100.0%100.0%
row_candidate_oracleShippingCode10.0%0.0%
row_candidate_oracleUserAgent1100.0%100.0%
row_majorityAddress250.0%0.0%
row_majorityBillingCode133.3%0.0%
row_majorityCity287.5%50.0%
row_majorityColumn1100.0%100.0%
row_majorityDateTime1674.0%56.2%
row_majorityFilePath1100.0%100.0%
row_majorityGender166.7%0.0%
row_majorityLanguage1100.0%100.0%
row_majorityName1100.0%100.0%
row_majorityNumber1061.7%30.0%
row_majorityPhone2100.0%100.0%
row_majorityShippingCode133.3%0.0%
row_majorityUserAgent1100.0%100.0%
shuffled_pseudo_program_tableAddress20.0%0.0%
shuffled_pseudo_program_tableBillingCode10.0%0.0%
shuffled_pseudo_program_tableCity20.0%0.0%
shuffled_pseudo_program_tableColumn10.0%0.0%
shuffled_pseudo_program_tableDateTime160.0%0.0%
shuffled_pseudo_program_tableFilePath10.0%0.0%
shuffled_pseudo_program_tableGender10.0%0.0%
shuffled_pseudo_program_tableLanguage10.0%0.0%
shuffled_pseudo_program_tableName10.0%0.0%
shuffled_pseudo_program_tableNumber100.0%0.0%
shuffled_pseudo_program_tablePhone20.0%0.0%
shuffled_pseudo_program_tableShippingCode10.0%0.0%
shuffled_pseudo_program_tableUserAgent10.0%0.0%
shuffled_support_programAddress20.0%0.0%
shuffled_support_programBillingCode1100.0%100.0%
shuffled_support_programCity225.0%0.0%
shuffled_support_programColumn10.0%0.0%
shuffled_support_programDateTime1612.5%12.5%
shuffled_support_programFilePath1100.0%100.0%
shuffled_support_programGender166.7%0.0%
shuffled_support_programLanguage1100.0%100.0%
shuffled_support_programName1100.0%100.0%
shuffled_support_programNumber1023.3%20.0%
shuffled_support_programPhone2100.0%100.0%
shuffled_support_programShippingCode10.0%0.0%
shuffled_support_programUserAgent10.0%0.0%
table_candidate_oracleAddress20.0%0.0%
table_candidate_oracleBillingCode10.0%0.0%
table_candidate_oracleCity250.0%50.0%
table_candidate_oracleColumn1100.0%100.0%
table_candidate_oracleDateTime1662.5%62.5%
table_candidate_oracleFilePath1100.0%100.0%
table_candidate_oracleGender10.0%0.0%
table_candidate_oracleLanguage1100.0%100.0%
table_candidate_oracleName1100.0%100.0%
table_candidate_oracleNumber1060.0%60.0%
table_candidate_oraclePhone2100.0%100.0%
table_candidate_oracleShippingCode10.0%0.0%
table_candidate_oracleUserAgent1100.0%100.0%

Reachable Headroom Tasks

task_idfamilyfeaturesdirect_full_exactexamples_full_exactsupport_full_exactshuffled_full_exactsupport_avg_vote_sharetrain_match_countsupport_programoracle_program
BillingCode.000007BillingCodeConcatenationFalseTrueFalseTrue71.4%1239affix['[',']'](field[,-1)affix['',']'](COL0)
Number.000077NumberNumeric,NumericRoundingFalseFalseFalseFalse57.1%687affix'','0')affix'','0')
Number.000029NumberNumeric,NumericRoundingFalseTrueTrueTrue57.1%617number_round10(COL0)number_round10(COL0)
Number.000049NumberNumeric,NumericRoundingFalseTrueTrueTrue14.3%624number_round100(COL0)number_round100(COL0)

Candidate-Support Changes

task_idfamilyfeaturesexamples_full_exactsupport_full_exactsupport_avg_vote_sharesupport_supported_rowsexamples_programsupport_program
Address.000002AddressSubstringFalseFalse0.0%0mapAysu Fatma Ahmed 492 24th Place NW,Edison,AK,(896) 388-9065,000-93-6876,38891->492 24th Place,Fiamma Greco 967 03th Place SE,Long Beach,OK,(129) 734-1247,000-61-4879,03719->967 03th Place SEmapPlace->492 24th Place,SE,Long->967 03th Place SE)
City.000011CityConditionalFalseFalse64.3%3mapNew York City->New York City,n.y.c.->New York City,New York City ->New York Cityaffix'','ew York City'))
DateTime.000051DateTimeDateTimeRange,DateTimeRounding,DateTimeFalseFalse0.0%0map11:12:29->11:00AM-11:30AM,08:29:52->8:00AM-8:30AMmap1->11:00AM-11:30AM,0->8:00AM-8:30AM)
DateTime.000081DateTimeDateTimeRange,DateTimeRounding,DateTimeFalseFalse16.7%1map6:25PM->6:15PM-6:45PM,9:44PM->9:15PM-9:45PM,7:00AM->6:45AM-7:15AM,11:34PM->11:15PM-11:45PMmap2->6:15PM-6:45PM,4->9:15PM-9:45PM,0->6:45AM-7:15AM,1->11:15PM-11:45PM))
DateTime.000115DateTimeDateTimeRange,DateTimeRounding,DateTimeFalseFalse100.0%6map31-Jan-2031 05:54:18->0-20,17-Jan-1990 13:32:01->0-20,14-Feb-2034 05:36:07->0-20,14-Mar-2002 13:16:16->0-20affix'0','20')
Gender.000001GenderConditionalFalseFalse90.5%3mapM->0,F->1mapM->0,F->1)
Number.000043NumberDateTimeFalseFalse0.0%0map26/4->04-26,5/11->11-05,23/9->09-23,8/12->12-08map30->04-26,0->11-05,20->09-23,10->12-08)
Number.000077NumberNumeric,NumericRoundingFalseFalse57.1%3map112->110,117->110affix'','0')
ShippingCode.000008ShippingCodeConcatenation,SubstringFalseFalse0.0%0map1Z 39V 80D 24 0712 870 8->39V 870,1Z AI7 S7L 39 2136 908 9->AI7 2136map9->39V 870,->AI7 2136))
BillingCode.000007BillingCodeConcatenationTrueFalse71.4%3affix['',']'](COL0)affix['[',']'](field[,-1)

Task Details

task_idfamilyheldout_rowstrain_match_countcandidate_tablescoherent_pseudo_tablesprogram_oracletable_candidate_oracledirect_full_exactexamples_full_exactsupport_full_exactpseudo_full_exactpseudo_or_direct_full_exactshuffled_pseudo_full_exactpseudo_avg_vote_share
BillingCode.000007BillingCode3123981TrueFalseFalseTrueFalseFalseFalseFalse71.4%
Number.000077Number3687182TrueTrueFalseFalseFalseFalseFalseFalse57.1%
Address.000002Address3140410FalseFalseFalseFalseFalseFalseFalseFalse0.0%
Address.000013Address6116890FalseFalseFalseFalseFalseFalseFalseFalse0.0%
City.000011City486440FalseFalseFalseFalseFalseFalseFalseFalse0.0%
DateTime.000027DateTime61026320FalseFalseFalseFalseFalseFalseFalseFalse0.0%
DateTime.000051DateTime3105140FalseFalseFalseFalseFalseFalseFalseFalse0.0%
DateTime.000076DateTime656860FalseTrueFalseFalseFalseFalseFalseFalse0.0%
DateTime.000081DateTime675060FalseFalseFalseFalseFalseFalseFalseFalse0.0%
DateTime.000114DateTime6862320FalseFalseFalseFalseFalseFalseFalseFalse0.0%
DateTime.000115DateTime6132711FalseFalseFalseFalseFalseFalseFalseFalse100.0%
DateTime.000116DateTime690710FalseFalseFalseFalseFalseFalseFalseFalse0.0%
Gender.000001Gender354030FalseFalseFalseFalseFalseFalseFalseFalse0.0%
Number.000008Number676880FalseFalseFalseFalseFalseFalseFalseFalse0.0%
Number.000015Number6691320FalseFalseFalseFalseFalseFalseFalseFalse0.0%
Number.000016Number6687320FalseTrueFalseFalseFalseFalseFalseFalse0.0%
Number.000075Number664580FalseTrueFalseFalseFalseFalseFalseFalse0.0%
ShippingCode.000008ShippingCode3118410FalseFalseFalseFalseFalseFalseFalseFalse0.0%
Number.000029Number361760TrueFalseFalseTrueTrueFalseFalseFalse0.0%
Number.000049Number4624320TrueFalseFalseTrueTrueFalseFalseFalse0.0%
City.000010City357410FalseTrueTrueFalseFalseFalseTrueFalse0.0%
Column.000001Column649110FalseTrueTrueFalseFalseFalseTrueFalse0.0%
DateTime.000007DateTime697410FalseTrueTrueFalseFalseFalseTrueFalse0.0%
DateTime.000017DateTime61027320FalseTrueTrueFalseFalseFalseTrueFalse0.0%
DateTime.000025DateTime6102780FalseTrueTrueFalseFalseFalseTrueFalse0.0%
DateTime.000034DateTime648410FalseTrueTrueFalseFalseFalseTrueFalse0.0%
DateTime.000094DateTime494810FalseTrueTrueFalseFalseFalseTrueFalse0.0%
DateTime.000108DateTime686210FalseTrueTrueFalseFalseFalseTrueFalse0.0%
DateTime.000111DateTime686220FalseTrueTrueFalseFalseFalseTrueFalse0.0%
Number.000022Number6701240FalseTrueTrueFalseFalseFalseTrueFalse0.0%
Number.000028Number376810FalseTrueTrueFalseFalseFalseTrueFalse0.0%
Number.000043Number673610FalseTrueTrueFalseFalseFalseTrueFalse0.0%
UserAgent.000003UserAgent680510FalseTrueTrueFalseFalseFalseTrueFalse0.0%
DateTime.000004DateTime6189111TrueTrueTrueTrueTrueTrueTrueFalse100.0%
DateTime.000104DateTime6119911TrueTrueTrueTrueTrueTrueTrueFalse100.0%
FilePath.000001FilePath658211TrueTrueTrueTrueTrueTrueTrueFalse100.0%
Language.000002Language6129211TrueTrueTrueTrueTrueTrueTrueFalse100.0%
Name.000028Name6105411TrueTrueTrueTrueTrueTrueTrueFalse100.0%
Phone.000008Phone6122811TrueTrueTrueTrueTrueTrueTrueFalse100.0%
Phone.000011Phone3109311TrueTrueTrueTrueTrueTrueTrueFalse100.0%

Files

  • runs/main_final/row_candidates.csv
  • runs/main_final/task_details.csv
  • runs/main_final/method_details.csv
  • runs/main_final/summary.csv
  • analysis/summary.csv
  • analysis/task_details.csv
  • analysis/method_details.csv
  • analysis/family_summary.csv

Experiment log 8

Show the running log (8 entries, 2026-06-27)

2026-06-27

  • Created standalone experiment directory.
  • Copied deterministic expression DSL into local src/dsl_core.py for standalone execution.

Run smoke_v1

  • Tasks: 6
  • Candidate rows: 210
  • Candidate-support full-task exact: 16.7%
  • Direct greedy full-task exact: 66.7%
  • Program oracle full-task exact: 33.3%

Run main_v1

  • Tasks: 40
  • Candidate rows: 1428
  • Candidate-support full-task exact: 22.5%
  • Direct greedy full-task exact: 50.0%
  • Program oracle full-task exact: 27.5%

Run smoke_v2

  • Tasks: 6
  • Candidate rows: 210
  • Candidate-support full-task exact: 16.7%
  • Direct greedy full-task exact: 66.7%
  • Program oracle full-task exact: 33.3%

Iteration note

  • main_v2 with 128 real pseudo tables and 128 shuffled pseudo tables was interrupted because the shuffled control dominated wall-clock before the 10-task progress marker.
  • Adjusted the implementation so the shuffled-pseudo control has its own smaller table/program budget while the real pseudo-label crystallizer remains bounded separately.

Run main_v2_bounded

  • Tasks: 40
  • Candidate rows: 1428
  • Candidate-support full-task exact: 22.5%
  • Direct greedy full-task exact: 50.0%
  • Program oracle full-task exact: 27.5%

Run main_final

  • Tasks: 40
  • Candidate rows: 1428
  • Candidate-support full-task exact: 22.5%
  • Pseudo-program table full-task exact: 17.5%
  • Pseudo-program with direct fallback full-task exact: 50.0%
  • Direct greedy full-task exact: 50.0%
  • Program oracle full-task exact: 27.5%

Run main_final

  • Tasks: 40
  • Candidate rows: 1428
  • Candidate-support full-task exact: 22.5%
  • Pseudo-program table full-task exact: 17.5%
  • Pseudo-program with direct fallback full-task exact: 50.0%
  • Direct greedy full-task exact: 50.0%
  • Program oracle full-task exact: 27.5%

Figures 6

family heatmap
family heatmap · analysis/figures/
method full task exact
method full task exact · analysis/figures/
program headroom
program headroom · analysis/figures/
row vs full task
row vs full task · analysis/figures/
support selection changes
support selection changes · analysis/figures/
support vote share distribution
support vote share distribution · analysis/figures/

Data files 12

Result tables and metrics copied from the experiment folder — preview inline or open the raw file.

Reproduce

This entry is source code or analysis only — there is no separate run to reproduce.

Browse the experiment folder on GitHub ↗