Counterexample-Guided Consistency Projection
The one idea you need
You have four examples of how a column should be reformatted, say dates or phone numbers. Instead of transforming each new row directly, you hunt for one exact formula, invent extra test inputs, and let the model's own answers vote for which formula to trust.
The question
When a small model reformats data messily, can you distill its noisy guesses into one clean, reliable rule that works correctly on every new row?
What we found
No. Distilling the model's messy guesses into one reliable formula solved only 20% of tasks completely, versus 50% when it simply transformed each row directly, a 30-point drop. For 29 of the 40 tasks no formula even fit the four examples, and the model's own guesses picked winners no better than randomly shuffled labels.
Why it matters
When you need a small model to reformat data reliably, don't reverse-engineer one formula from a few examples: its self-labeled guesses are too noisy to steer the search. Transform each row directly, and reserve a fixed rule only where a valid one provably exists.
On this page
Results at a glance 7
How to read
Each bar is one method; height is the share of tasks solved perfectly, every row correct, so taller is better. The left three answer each row directly; the right three boil the guesses into one fixed rule.
Takeaway → The three direct bars sit near 50% while the three rule bars flatten to 20%, and real guesses tie shuffled ones, so the rule approach threw away 30 points.
Data table
| method | full-task exact |
|---|---|
| direct row-by-row | 50% |
| gated projection | 50% |
| output majority | 47.5% |
| probe-guided projection | 20% |
| shuffled-probe projection | 20% |
| train-only selection | 20% |
Numbers from experiments/qwen_counterexample_guided_projection/analysis/summary.csv
Technical framing
Full-task exactness: direct row inference beats probe-guided projection — Probe-guided projection loses 30 points to direct inference, and real probe labels do no better than shuffled labels or train-only selection.
How to read
For each of five methods, one bar shows the share of individual rows answered correctly and a second shows the share of tasks solved completely; taller is better for both.
Takeaway → Direct methods get about 72% of rows right but only half of tasks fully right; the rule method collapses to roughly 21% and 20% on both.
Data table
| method | row exact | full-task exact |
|---|---|---|
| direct row-by-row | 72.1% | 50% |
| gated projection | 71.5% | 50% |
| output majority | 71.5% | 47.5% |
| probe-guided projection | 21.2% | 20% |
| train-only selection | 21.2% | 20% |
Numbers from experiments/qwen_counterexample_guided_projection/analysis/summary.csv
Technical framing
Row-level vs strict full-task accuracy by method — Direct methods get ~72% of rows right but only 50% of tasks fully right; projection collapses both because it rarely has a usable candidate.
How to read
Bars split the 40 tasks by how many valid fixed rules were found: none, at least one, or more than one; a tall left bar means the rule method had little to work with.
Takeaway → 29 of 40 tasks had zero valid rules, and every one of the 11 that had any had several, so on most tasks there was nothing to pick among.
Data table
| train-consistent deterministic candidates per task | tasks |
|---|---|
| zero candidates | 29 |
| at least one | 11 |
| more than one | 11 |
Technical framing
Candidate availability was the bottleneck (40 tasks) — 29 of 40 tasks had no train-consistent deterministic candidate at all, so probe labels had nothing to select among.
How to read
Grouped bars for five task types (addresses, cities, dates, numbers, phones) show three methods side by side; height is the share of tasks solved completely, so taller is better.
Takeaway → On the two biggest groups, dates and numbers, the rule bars fall to near zero while direct answers hold 56% and 30%.
Data table
| Task family (multi-task families) | Direct row-by-row | Output majority | Probe-guided projection |
|---|---|---|---|
| Address | 0% | 0% | 0% |
| City | 50% | 50% | 0% |
| DateTime | 56.2% | 56.2% | 12.5% |
| Number | 30% | 20% | 0% |
| Phone | 100% | 100% | 100% |
Numbers from experiments/qwen_counterexample_guided_projection/analysis/family_summary.csv
Technical framing
Strict full-task exactness by task family — Projection collapses on the two largest families (DateTime, Number) while direct inference keeps 56% and 30% full-task exactness.
How to read
The same five task types, three methods side by side; height is the share of individual rows answered correctly, so taller is better.
Takeaway → The rule method drops to zero on several groups while direct answers stay high; voting among guesses nudges dates up but drags numbers down.
Data table
| Task family (multi-task families) | Direct row-by-row | Output majority | Probe-guided projection |
|---|---|---|---|
| Address | 50% | 50% | 0% |
| City | 87.5% | 87.5% | 25% |
| DateTime | 70.8% | 76% | 12.5% |
| Number | 64.2% | 56.7% | 0% |
| Phone | 100% | 100% | 100% |
Numbers from experiments/qwen_counterexample_guided_projection/analysis/family_summary.csv
Technical framing
Row-level accuracy by task family — Majority voting nudges DateTime rows up but hurts Number rows, so it never converts into more fully-solved tasks than direct inference.
How to read
Each bar is one task; height is how many valid fixed rules were found; of the 11 tasks that had any rule at all, 10 are shown.
Takeaway → Counts ranged from about two dozen up to 400, one hitting the cap, but this covered just 11 of 40 tasks while the rest had none.
Data table
| Task (10 of 11 tasks with candidates shown) | Train-consistent candidates |
|---|---|
| DateTime.000004 | 400 |
| BillingCode.000007 | 104 |
| DateTime.000104 | 90 |
| FilePath.000001 | 88 |
| Number.000077 | 77 |
| City.000011 | 59 |
| Language.000002 | 53 |
| Phone.000011 | 44 |
| DateTime.000115 | 31 |
| Name.000028 | 24 |
Technical framing
Train-consistent candidate counts on tasks that had any — Where the grammar covered a task at all, candidates ranged 24-400 (one hit the 400 cap); the other 29 tasks had zero.
How to read
For the only two tasks where high guess-agreement triggered the safeguard, paired bars compare direct answers against the gated rule; height is the share of rows correct, taller is better.
Takeaway → On both tasks the gate locked in a single constant answer; one task got worse and the other stayed at zero, so the safeguard never helped.
Data table
| Task where high probe agreement triggered the gate | Direct row-by-row | Gated projection |
|---|---|---|
| City.000011 | 75% | 50% |
| DateTime.000115 | 0% | 0% |
Technical framing
The only two gate-triggered tasks: gating never helped — High probe agreement (87.5% and 100%) selected constant programs; one task got worse at the row level and the other stayed at zero.
In the author’s words from the Overview · “Summary”
Can noisy row-level model guesses be converted into a stable task-level deterministic transformation?
Overview
This top-level README was generated during repository normalization because the imported experiment did not include one.
- Source track:
track-y - Primary report: reports/qwen_counterexample_guided_projection_report.md
- Metadata: metadata.yaml
How To Read
Start with the primary report, then inspect data/, reports/, analysis/, src/, and scripts/ as available. This folder remains self-contained; do not move its run data into shared directories.
Summary
Can noisy row-level model guesses be converted into a stable task-level deterministic transformation?
Report
Rendered from reports/qwen_counterexample_guided_projection_report.md
Question
Can noisy row-level model guesses be converted into a stable task-level deterministic transformation?
The experiment generates counterexample-style probe inputs from each task's training examples, labels those probes with a row-level model, and selects among train-consistent deterministic expressions using probe-label agreement plus a small complexity penalty. The selected expression is then evaluated on held-out rows from the same public transformation tasks.
Setup
- Benchmark root:
/workspace/large_artifacts/qwen_counterexample_guided_projection/prose-benchmarks - Run:
main_qwen_probe_40 - Tasks: 40
- Train rows per task: 4
- Held-out cap per task: 6
- Generated probes per task: 8
- Candidate cap per task: 400
- Probe labels used: 320
- New probe-label calls during this invocation: no
Main Result
| method | tasks | row_exact | full_task_exact | median_candidates | median_probe_agreement |
|---|---|---|---|---|---|
| direct_row_by_row | 40 | 72.1% | 50.0% | 0.00 | 0.0% |
| qwen_projection_gated_direct | 40 | 71.5% | 50.0% | 0.00 | 0.0% |
| output_majority | 40 | 71.5% | 47.5% | 0.00 | 0.0% |
| qwen_projection | 40 | 21.2% | 20.0% | 0.00 | 0.0% |
| qwen_projection_random | 40 | 21.2% | 20.0% | 0.00 | 0.0% |
| qwen_projection_shuffled | 40 | 21.2% | 20.0% | 0.00 | 0.0% |
| train_only | 40 | 21.2% | 20.0% | 0.00 | 0.0% |
Interpretation
The probe-guided projection underperformed direct row inference by 30.0 points. Real and shuffled probe labels were separated by only 0.0 points. Probe-guided and train-only deterministic selection were separated by only 0.0 points.
The primary success condition is not row-level accuracy alone. The goal is to improve strict full-task exactness: every held-out row for a task must be correct under one stable transformation.
Diagnostic Findings
- Candidate availability was the main bottleneck: 29 of 40 tasks had zero train-consistent deterministic candidates, 11 had at least one, and 11 had more than one.
- The ungated projection did not improve over train-only selection. This means the probe labels did not rescue the candidate-selection objective on this benchmark slice.
- The gated projection changed 2 tasks: 0 helped, 2 tied, and 0 hurt relative to direct row inference on strict full-task exactness.
- The output-majority baseline did not improve over direct row inference, so simple agreement among prompt variants was not enough to stabilize full-task behavior.
Charts
Family Breakdown
| method | family | tasks | row_exact | full_task_exact |
|---|---|---|---|---|
| direct_row_by_row | Address | 2 | 50.0% | 0.0% |
| direct_row_by_row | BillingCode | 1 | 33.3% | 0.0% |
| direct_row_by_row | City | 2 | 87.5% | 50.0% |
| direct_row_by_row | Column | 1 | 100.0% | 100.0% |
| direct_row_by_row | DateTime | 16 | 70.8% | 56.2% |
| direct_row_by_row | FilePath | 1 | 100.0% | 100.0% |
| direct_row_by_row | Gender | 1 | 66.7% | 0.0% |
| direct_row_by_row | Language | 1 | 100.0% | 100.0% |
| direct_row_by_row | Name | 1 | 100.0% | 100.0% |
| direct_row_by_row | Number | 10 | 64.2% | 30.0% |
| direct_row_by_row | Phone | 2 | 100.0% | 100.0% |
| direct_row_by_row | ShippingCode | 1 | 33.3% | 0.0% |
| direct_row_by_row | UserAgent | 1 | 100.0% | 100.0% |
| output_majority | Address | 2 | 50.0% | 0.0% |
| output_majority | BillingCode | 1 | 0.0% | 0.0% |
| output_majority | City | 2 | 87.5% | 50.0% |
| output_majority | Column | 1 | 100.0% | 100.0% |
| output_majority | DateTime | 16 | 76.0% | 56.2% |
| output_majority | FilePath | 1 | 100.0% | 100.0% |
| output_majority | Gender | 1 | 66.7% | 0.0% |
| output_majority | Language | 1 | 100.0% | 100.0% |
| output_majority | Name | 1 | 100.0% | 100.0% |
| output_majority | Number | 10 | 56.7% | 20.0% |
| output_majority | Phone | 2 | 100.0% | 100.0% |
| output_majority | ShippingCode | 1 | 33.3% | 0.0% |
| output_majority | UserAgent | 1 | 100.0% | 100.0% |
| qwen_projection | Address | 2 | 0.0% | 0.0% |
| qwen_projection | BillingCode | 1 | 100.0% | 100.0% |
| qwen_projection | City | 2 | 25.0% | 0.0% |
| qwen_projection | Column | 1 | 0.0% | 0.0% |
| qwen_projection | DateTime | 16 | 12.5% | 12.5% |
| qwen_projection | FilePath | 1 | 100.0% | 100.0% |
| qwen_projection | Gender | 1 | 0.0% | 0.0% |
| qwen_projection | Language | 1 | 100.0% | 100.0% |
| qwen_projection | Name | 1 | 100.0% | 100.0% |
| qwen_projection | Number | 10 | 0.0% | 0.0% |
| qwen_projection | Phone | 2 | 100.0% | 100.0% |
| qwen_projection | ShippingCode | 1 | 0.0% | 0.0% |
| qwen_projection | UserAgent | 1 | 0.0% | 0.0% |
| qwen_projection_gated_direct | Address | 2 | 50.0% | 0.0% |
| qwen_projection_gated_direct | BillingCode | 1 | 33.3% | 0.0% |
| qwen_projection_gated_direct | City | 2 | 75.0% | 50.0% |
| qwen_projection_gated_direct | Column | 1 | 100.0% | 100.0% |
| qwen_projection_gated_direct | DateTime | 16 | 70.8% | 56.2% |
| qwen_projection_gated_direct | FilePath | 1 | 100.0% | 100.0% |
| qwen_projection_gated_direct | Gender | 1 | 66.7% | 0.0% |
| qwen_projection_gated_direct | Language | 1 | 100.0% | 100.0% |
| qwen_projection_gated_direct | Name | 1 | 100.0% | 100.0% |
| qwen_projection_gated_direct | Number | 10 | 64.2% | 30.0% |
| qwen_projection_gated_direct | Phone | 2 | 100.0% | 100.0% |
| qwen_projection_gated_direct | ShippingCode | 1 | 33.3% | 0.0% |
| qwen_projection_gated_direct | UserAgent | 1 | 100.0% | 100.0% |
| qwen_projection_random | Address | 2 | 0.0% | 0.0% |
| qwen_projection_random | BillingCode | 1 | 100.0% | 100.0% |
| qwen_projection_random | City | 2 | 25.0% | 0.0% |
| qwen_projection_random | Column | 1 | 0.0% | 0.0% |
| qwen_projection_random | DateTime | 16 | 12.5% | 12.5% |
| qwen_projection_random | FilePath | 1 | 100.0% | 100.0% |
| qwen_projection_random | Gender | 1 | 0.0% | 0.0% |
| qwen_projection_random | Language | 1 | 100.0% | 100.0% |
| qwen_projection_random | Name | 1 | 100.0% | 100.0% |
| qwen_projection_random | Number | 10 | 0.0% | 0.0% |
| qwen_projection_random | Phone | 2 | 100.0% | 100.0% |
| qwen_projection_random | ShippingCode | 1 | 0.0% | 0.0% |
| qwen_projection_random | UserAgent | 1 | 0.0% | 0.0% |
| qwen_projection_shuffled | Address | 2 | 0.0% | 0.0% |
| qwen_projection_shuffled | BillingCode | 1 | 100.0% | 100.0% |
| qwen_projection_shuffled | City | 2 | 25.0% | 0.0% |
| qwen_projection_shuffled | Column | 1 | 0.0% | 0.0% |
| qwen_projection_shuffled | DateTime | 16 | 12.5% | 12.5% |
| qwen_projection_shuffled | FilePath | 1 | 100.0% | 100.0% |
| qwen_projection_shuffled | Gender | 1 | 0.0% | 0.0% |
| qwen_projection_shuffled | Language | 1 | 100.0% | 100.0% |
| qwen_projection_shuffled | Name | 1 | 100.0% | 100.0% |
| qwen_projection_shuffled | Number | 10 | 0.0% | 0.0% |
| qwen_projection_shuffled | Phone | 2 | 100.0% | 100.0% |
| qwen_projection_shuffled | ShippingCode | 1 | 0.0% | 0.0% |
| qwen_projection_shuffled | UserAgent | 1 | 0.0% | 0.0% |
| train_only | Address | 2 | 0.0% | 0.0% |
| train_only | BillingCode | 1 | 100.0% | 100.0% |
| train_only | City | 2 | 25.0% | 0.0% |
| train_only | Column | 1 | 0.0% | 0.0% |
| train_only | DateTime | 16 | 12.5% | 12.5% |
| train_only | FilePath | 1 | 100.0% | 100.0% |
| train_only | Gender | 1 | 0.0% | 0.0% |
| train_only | Language | 1 | 100.0% | 100.0% |
| train_only | Name | 1 | 100.0% | 100.0% |
| train_only | Number | 10 | 0.0% | 0.0% |
| train_only | Phone | 2 | 100.0% | 100.0% |
| train_only | ShippingCode | 1 | 0.0% | 0.0% |
| train_only | UserAgent | 1 | 0.0% | 0.0% |
Selected Program Examples
| task_id | family | method | candidate_count | probe_agreement_rate | program | row_exact | full_task_exact |
|---|---|---|---|---|---|---|---|
| Address.000002 | Address | direct_row_by_row | 0 | 0.0% | 33.3% | False | |
| Address.000013 | Address | direct_row_by_row | 0 | 0.0% | 66.7% | False | |
| BillingCode.000007 | BillingCode | direct_row_by_row | 104 | 0.0% | 33.3% | False | |
| City.000010 | City | direct_row_by_row | 0 | 0.0% | 100.0% | True | |
| City.000011 | City | direct_row_by_row | 59 | 0.0% | 75.0% | False | |
| Column.000001 | Column | direct_row_by_row | 0 | 0.0% | 100.0% | True | |
| DateTime.000004 | DateTime | direct_row_by_row | 400 | 0.0% | 100.0% | True | |
| DateTime.000007 | DateTime | direct_row_by_row | 0 | 0.0% | 100.0% | True | |
| DateTime.000017 | DateTime | direct_row_by_row | 0 | 0.0% | 100.0% | True | |
| DateTime.000025 | DateTime | direct_row_by_row | 0 | 0.0% | 100.0% | True | |
| DateTime.000027 | DateTime | direct_row_by_row | 0 | 0.0% | 33.3% | False | |
| DateTime.000034 | DateTime | direct_row_by_row | 0 | 0.0% | 100.0% | True | |
| DateTime.000051 | DateTime | direct_row_by_row | 0 | 0.0% | 33.3% | False | |
| DateTime.000076 | DateTime | direct_row_by_row | 0 | 0.0% | 66.7% | False | |
| DateTime.000081 | DateTime | direct_row_by_row | 0 | 0.0% | 50.0% | False | |
| DateTime.000094 | DateTime | direct_row_by_row | 0 | 0.0% | 100.0% | True | |
| DateTime.000104 | DateTime | direct_row_by_row | 90 | 0.0% | 100.0% | True | |
| DateTime.000108 | DateTime | direct_row_by_row | 0 | 0.0% | 100.0% | True | |
| DateTime.000111 | DateTime | direct_row_by_row | 0 | 0.0% | 100.0% | True | |
| DateTime.000114 | DateTime | direct_row_by_row | 0 | 0.0% | 0.0% | False | |
| DateTime.000115 | DateTime | direct_row_by_row | 31 | 0.0% | 0.0% | False | |
| DateTime.000116 | DateTime | direct_row_by_row | 0 | 0.0% | 50.0% | False | |
| FilePath.000001 | FilePath | direct_row_by_row | 88 | 0.0% | 100.0% | True | |
| Gender.000001 | Gender | direct_row_by_row | 0 | 0.0% | 66.7% | False | |
| Language.000002 | Language | direct_row_by_row | 53 | 0.0% | 100.0% | True | |
| Name.000028 | Name | direct_row_by_row | 24 | 0.0% | 100.0% | True | |
| Number.000008 | Number | direct_row_by_row | 0 | 0.0% | 33.3% | False | |
| Number.000015 | Number | direct_row_by_row | 0 | 0.0% | 33.3% | False | |
| Number.000016 | Number | direct_row_by_row | 0 | 0.0% | 83.3% | False | |
| Number.000022 | Number | direct_row_by_row | 0 | 0.0% | 100.0% | True | |
| Number.000028 | Number | direct_row_by_row | 0 | 0.0% | 100.0% | True | |
| Number.000029 | Number | direct_row_by_row | 0 | 0.0% | 66.7% | False | |
| Number.000043 | Number | direct_row_by_row | 0 | 0.0% | 100.0% | True | |
| Number.000049 | Number | direct_row_by_row | 0 | 0.0% | 25.0% | False | |
| Number.000075 | Number | direct_row_by_row | 0 | 0.0% | 66.7% | False | |
| Number.000077 | Number | direct_row_by_row | 77 | 0.0% | 33.3% | False | |
| Phone.000008 | Phone | direct_row_by_row | 8 | 0.0% | 100.0% | True | |
| Phone.000011 | Phone | direct_row_by_row | 44 | 0.0% | 100.0% | True | |
| ShippingCode.000008 | ShippingCode | direct_row_by_row | 0 | 0.0% | 33.3% | False | |
| UserAgent.000003 | UserAgent | direct_row_by_row | 0 | 0.0% | 100.0% | True | |
| Address.000002 | Address | output_majority | 0 | 0.0% | 33.3% | False | |
| Address.000013 | Address | output_majority | 0 | 0.0% | 66.7% | False | |
| BillingCode.000007 | BillingCode | output_majority | 104 | 0.0% | 0.0% | False | |
| City.000010 | City | output_majority | 0 | 0.0% | 100.0% | True | |
| City.000011 | City | output_majority | 59 | 0.0% | 75.0% | False | |
| Column.000001 | Column | output_majority | 0 | 0.0% | 100.0% | True | |
| DateTime.000004 | DateTime | output_majority | 400 | 0.0% | 100.0% | True | |
| DateTime.000007 | DateTime | output_majority | 0 | 0.0% | 100.0% | True | |
| DateTime.000017 | DateTime | output_majority | 0 | 0.0% | 100.0% | True | |
| DateTime.000025 | DateTime | output_majority | 0 | 0.0% | 100.0% | True | |
| DateTime.000027 | DateTime | output_majority | 0 | 0.0% | 66.7% | False | |
| DateTime.000034 | DateTime | output_majority | 0 | 0.0% | 100.0% | True | |
| DateTime.000051 | DateTime | output_majority | 0 | 0.0% | 33.3% | False | |
| DateTime.000076 | DateTime | output_majority | 0 | 0.0% | 66.7% | False | |
| DateTime.000081 | DateTime | output_majority | 0 | 0.0% | 50.0% | False | |
| DateTime.000094 | DateTime | output_majority | 0 | 0.0% | 100.0% | True | |
| DateTime.000104 | DateTime | output_majority | 90 | 0.0% | 100.0% | True | |
| DateTime.000108 | DateTime | output_majority | 0 | 0.0% | 100.0% | True | |
| DateTime.000111 | DateTime | output_majority | 0 | 0.0% | 100.0% | True | |
| DateTime.000114 | DateTime | output_majority | 0 | 0.0% | 50.0% | False | |
| DateTime.000115 | DateTime | output_majority | 31 | 0.0% | 0.0% | False | |
| DateTime.000116 | DateTime | output_majority | 0 | 0.0% | 50.0% | False | |
| FilePath.000001 | FilePath | output_majority | 88 | 0.0% | 100.0% | True | |
| Gender.000001 | Gender | output_majority | 0 | 0.0% | 66.7% | False | |
| Language.000002 | Language | output_majority | 53 | 0.0% | 100.0% | True | |
| Name.000028 | Name | output_majority | 24 | 0.0% | 100.0% | True | |
| Number.000008 | Number | output_majority | 0 | 0.0% | 16.7% | False | |
| Number.000015 | Number | output_majority | 0 | 0.0% | 50.0% | False | |
| Number.000016 | Number | output_majority | 0 | 0.0% | 50.0% | False | |
| Number.000022 | Number | output_majority | 0 | 0.0% | 33.3% | False | |
| Number.000028 | Number | output_majority | 0 | 0.0% | 100.0% | True | |
| Number.000029 | Number | output_majority | 0 | 0.0% | 66.7% | False | |
| Number.000043 | Number | output_majority | 0 | 0.0% | 100.0% | True | |
| Number.000049 | Number | output_majority | 0 | 0.0% | 0.0% | False | |
| Number.000075 | Number | output_majority | 0 | 0.0% | 83.3% | False | |
| Number.000077 | Number | output_majority | 77 | 0.0% | 66.7% | False | |
| Phone.000008 | Phone | output_majority | 8 | 0.0% | 100.0% | True | |
| Phone.000011 | Phone | output_majority | 44 | 0.0% | 100.0% | True | |
| ShippingCode.000008 | ShippingCode | output_majority | 0 | 0.0% | 33.3% | False | |
| UserAgent.000003 | UserAgent | output_majority | 0 | 0.0% | 100.0% | True |
Projection-Induced Program Changes
| task_id | probe_agreement_rate | train_only_program | projection_program | train_only_full_task_exact | projection_full_task_exact | train_only_row_exact | projection_row_exact |
|---|---|---|---|---|---|---|---|
| BillingCode.000007 | 25.0% | affix('',COL0,']') | affix('',upper(COL0),']') | True | True | 100.0% | 100.0% |
| Language.000002 | 37.5% | file_stem(COL1) | lower(file_stem(COL1)) | True | True | 100.0% | 100.0% |
| Name.000028 | 12.5% | first_word(COL0) | title(first_word(COL0)) | True | True | 100.0% | 100.0% |
Gated Projection Changes
| task_id | probe_agreement_rate | program | direct_row | gated_row | direct_full | gated_full |
|---|---|---|---|---|---|---|
| City.000011 | 87.5% | const('New York City') | 0.75 | 0.50 | False | False |
| DateTime.000115 | 100.0% | const('0-20') | 0.00 | 0.00 | False | False |
Files
runs/main_qwen_probe_40/task_details.csvruns/main_qwen_probe_40/row_details.csvruns/main_qwen_probe_40/probe_details.csvruns/main_qwen_probe_40/qwen_probe_labels.csvanalysis/summary.csvanalysis/family_summary.csvanalysis/task_details.csvanalysis/row_details.csvanalysis/probe_details.csv
Experiment log 9
Show the running log (9 entries, 2026-06-27)
Objective
Test whether a model's row-level transformation guesses can be crystallized into a stable task-level deterministic transducer by generating counterexample-style probe inputs, labeling those probes with the model, and selecting a train-consistent program using a noise-aware objective.
The experiment is standalone: it uses public text-transformation tasks, writes all local outputs under this directory, and stores any large reusable artifacts under /workspace/large_artifacts/qwen_counterexample_guided_projection.
Initial Plan
- Build a standalone runner for public text-transformation tasks.
- Enumerate deterministic candidate expressions that exactly match training examples.
- Generate task-local probe inputs by mutating and recombining training inputs.
- Label probes with Qwen row inference.
- Select among train-consistent expressions using probe-label agreement plus a complexity penalty.
- Compare against train-only deterministic selection, shuffled probe labels, direct Qwen held-out outputs, and simple output-level ensemble/majority baselines when available.
- Generate CSVs, charts, a Markdown report, and an HTML report.
Run Notes
2026-06-27 06:48 UTC - Smoke: smoke_no_qwen
- Created a new standalone experiment directory.
- Ran a 6-task no-Qwen smoke test to verify benchmark loading, deterministic candidate enumeration, CSV writing, chart generation, Markdown report, and HTML report.
- Finding: the scaffold worked, but the initial candidate deduplication would have collapsed all train-consistent candidates into one because train-consistent expressions share the same train signature. Fixed that before real probe labeling.
2026-06-27 06:51 UTC - Candidate-class iteration
- Added token-span extraction and simple affix expressions.
- Reran a no-Qwen smoke test.
- Finding: the broader grammar helped some substring-style tasks but many public tasks still had zero train-consistent candidates.
2026-06-27 06:53 UTC - Matched-split diagnostic: diagnostic_no_qwen_matched
- Aligned the task split with the cached direct row-by-row baseline: seed
20260627, 40 tasks, 4 train rows, up to 6 held-out rows. - Train-only deterministic selection reached 20.0% full-task exact and 21.2% row exact.
- Direct row-by-row baseline reached 50.0% full-task exact and 72.1% row exact.
- Candidate availability was the main bottleneck: 29/40 tasks had zero train-consistent candidates; 11/40 had at least one candidate.
2026-06-27 06:56 UTC - Qwen probe pilot: pilot_qwen_6
- Queried Qwen on 24 generated probe rows across 6 tasks.
- Probe labels were often sensible, especially for date/month and numeric-format tasks.
- The projection still could not act on most tasks because there were no train-consistent deterministic candidates.
- Added a high-agreement gated projection arm that falls back to direct row inference unless the selected program strongly agrees with probe labels.
- Added a concrete-output majority baseline over existing row/batch prompt variants.
2026-06-27 07:01 UTC - Main run: main_qwen_probe_40
- Queried Qwen for 320 generated probe labels: 40 tasks x 8 probes.
Main results:
direct_row_by_row: 72.1% row exact, 50.0% full-task exact.qwen_projection: 21.2% row exact, 20.0% full-task exact.train_only: 21.2% row exact, 20.0% full-task exact.qwen_projection_shuffled: 21.2% row exact, 20.0% full-task exact.qwen_projection_gated_direct: 71.5% row exact, 50.0% full-task exact.output_majority: 71.5% row exact, 47.5% full-task exact.
Interpretation:
- Real Qwen probe labels did not separate from shuffled/random probe controls on full-task accuracy.
- The ungated projection was limited by deterministic hypothesis coverage.
- The gated projection safely tied direct full-task exact but did not improve it, and it slightly reduced row exact.
- Simple output-majority across prompt variants also did not improve over direct row-by-row.
2026-06-27 07:05 UTC - Report hardening
- Added candidate availability diagnostics.
- Added gated-decision diagnostics.
- Regenerated the standalone Markdown and HTML reports with six charts.
Figures 6
Data files 2
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
Reproduce
This entry is source code or analysis only — there is no separate run to reproduce.