Counterfactual ICL Public Multiseed Gate
The one idea you need
The training puzzles follow invented rules that contradict anything memorized — like a cipher whose key changes every round. The only way to solve one is to decode the worked examples shown in the prompt, forcing the model to read them instead of recalling a familiar format.
The question
Can cheap made-up puzzles that force a small model to copy its examples teach it to follow examples on real tasks — and is the special recipe why it works?
What we found
Yes, but not for the reason expected. Tuning tripled whole-task accuracy on real text tasks, from 20% to about 61%, stable across three training runs — and the model genuinely reads its examples: remove them and it collapses to under 2%. But plain tuning on ordinary made-up puzzles reached 64%, so the clever counterintuitive twist added nothing. The lift is generic synthetic tuning.
Why it matters
Before crediting a clever data recipe for a small model's gains, run two cheap controls: tune on plain synthetic puzzles, and strip the examples at test time. Here both showed the win was generic synthetic tuning, not the advertised mechanism.
On this page
Results at a glance 7
How to read
Bars give whole-task exact-match accuracy on public text tasks with examples intact; left to right: untuned base, the counterfactual adapter (mean of three seeds), the plain-synthetic adapter, and the shuffled-label adapter. Taller is better.
Takeaway → All three tuned bars cluster near 60 to 64%, roughly tripling the untuned 20% and indistinguishable from each other, so the clever counterfactual ingredient adds nothing visible.
Data table
| training arm | full-task exact (public PROSE, intact support) |
|---|---|
| base (no tuning) | 20% |
| counterfactual adapter (3 seeds) | 60.7% |
| ordinary adapter | 64.4% |
| shuffled-label adapter | 62.2% |
Numbers from reports/qwen_counterfactual_icl_public_multiseed_report.md table (analysis/aggregate_summary.csv)
Technical framing
Public text-transformation accuracy: base model vs LoRA-tuned adapters — Synthetic ICL-style LoRA tuning triples public accuracy, but the ordinary and shuffled-label controls match it — the counterfactual ingredient isn't isolated.
How to read
For each adapter, bars show public whole-task accuracy under intact, shuffled, and removed support examples; colors mark the counterfactual, ordinary, and shuffled-label-trained adapters; higher is better.
Takeaway → Every adapter collapses to near zero once examples are removed, proving they read them, but the shuffled-label adapter still scores 51% on shuffled examples, hinting at learned formatting.
Data table
| support condition at eval | counterfactual adapter | ordinary adapter | shuffled-label-trained adapter |
|---|---|---|---|
| intact support | 60.7% | 64.4% | 62.2% |
| shuffled support | 28.1% | 26.7% | 51.1% |
| no support | 1.5% | 0% | 2.2% |
Numbers from reports/qwen_counterfactual_icl_public_multiseed_report.md table (analysis/aggregate_summary.csv)
Technical framing
Adapter accuracy depends on the in-context support examples — Corrupting or removing support examples collapses performance, so adapters do read the examples — but shuffled-label training still yields high shuffled-support accuracy.
How to read
Two evaluation splits — the synthetic training-style tasks and the unseen public tasks; two bars each, untuned base versus the counterfactual adapter, both with intact examples; higher is better.
Takeaway → The adapter lifts both splits, from 38% to 93% on synthetic and 20% to 61% on public, so the skill transfers to genuinely unseen tasks, not just the generator.
Data table
| evaluation split | base | counterfactual adapter (mean of 3 seeds) |
|---|---|---|
| synthetic counterfactual split | 38.3% | 92.8% |
| public PROSE split | 20% | 60.7% |
Numbers from reports/qwen_counterfactual_icl_public_multiseed_report.md table (analysis/aggregate_summary.csv)
Technical framing
Synthetic training gains transfer to the public benchmark — Training only on synthetic episodes with answer-only supervision lifts unseen public tasks by 40.7 points, not just the synthetic generator.
How to read
For each of three training seeds, one bar counts public tasks that improved and one counts those that worsened versus the untuned base, out of 45; more helped and fewer hurt is better.
Takeaway → Every seed improves about 20 of 45 tasks and worsens only one, so the gain is a stable effect rather than one lucky run.
Data table
| training seed | tasks helped | tasks hurt |
|---|---|---|
| seed 1 | 18 | 1 |
| seed 2 | 20 | 1 |
| seed 3 | 20 | 1 |
Numbers from reports/qwen_counterfactual_icl_public_multiseed_report.md 'Public Task Flips Versus Base' table
Technical framing
Per-seed public task flips versus base (45 tasks) — The gain is consistent across seeds: about 20 of 45 public tasks improve while only 1 regresses in every seed.
How to read
Ten public task families across the axis; paired bars give untuned base versus counterfactual-adapter whole-task accuracy; higher is better.
Takeaway → Structured formats like codes, file paths and currency jump to 100%, dates stay hardest, and the single address task actually regresses to zero.
Data table
| Public task family | Base model | Counterfactual adapter |
|---|---|---|
| Column | 0% | 100% |
| Currency | 0% | 100% |
| EmergencyCall | 0% | 100% |
| FilePath | 0% | 100% |
| ShippingCode | 0% | 88.9% |
| Number | 10% | 63.3% |
| Name | 0% | 53.3% |
| Phone | 66.7% | 100% |
| DateTime | 20% | 44.4% |
| Address | 100% | 0% |
Technical framing
Which public task families the adapter fixes — Gains span structured families (codes, paths, currency); DateTime stays hardest and the single Address task regresses to 0%.
How to read
Four arms on the synthetic split — base and the counterfactual, ordinary, and shuffled-label adapters; two bars each for normal versus shuffled support examples; higher is better.
Takeaway → Most arms roughly halve under shuffled examples, but the shuffled-label adapter holds 92%, evidence it learned output formatting rather than truly reading the examples.
Data table
| Model arm (synthetic counterfactual split) | Normal support examples | Shuffled support examples |
|---|---|---|
| Base | 38.3% | 16.7% |
| Counterfactual adapter | 92.8% | 51.1% |
| Ordinary adapter | 95% | 46.7% |
| Shuffled-label adapter | 95% | 91.7% |
Technical framing
Shuffled-label training control still tolerates shuffled support — The adapter trained on shuffled labels keeps 91.7% under shuffled support, so a formatting explanation for the gains remains plausible.
How to read
Four arms on the public split — base and the three adapters; two bars each, lenient per-row scoring versus strict whole-task scoring where every row must match; higher is better.
Takeaway → Requiring every row to match costs the adapters about 10 points but nearly halves the base model, so strict scoring widens the tuned-versus-untuned gap.
Data table
| Model arm (public split, normal support) | Per-row exact | All rows in task exact |
|---|---|---|
| Base | 37% | 20% |
| Counterfactual adapter | 70.6% | 60.7% |
| Ordinary adapter | 74.1% | 64.4% |
| Shuffled-label adapter | 74.1% | 62.2% |
Technical framing
Strict whole-task scoring costs every arm on the public benchmark — Requiring every row of a task to match costs adapters about 10 points but nearly halves the base model's score.
In the author’s words from the Report · “Interpretation”
The synthetic counterfactual split shows the intended training effect: the adapter improves strict task consistency, and that improvement depends on intact support examples. The public split shows positive transfer from synthetic counterfactual episodes to unseen public text-transformation tasks. The public gain is support-sensitive: corrupting the support examples removes a material fraction of the effect. The ordinary synthetic control is close enough that the counterfactual ingredient is not isolated. The shuffled-label training control is close enough that a formatting explanation remains plausible.
Overview
Standalone experiment testing whether counterfactual episodic LoRA posttraining improves example-conditioned public text transformation accuracy across seeds.
Large artifacts, checkpoints, and the benchmark mirror live under /workspace/large_artifacts/qwen_counterfactual_icl_public_multiseed.
Planned Main Arms
cf_s1,cf_s2,cf_s3: counterfactual synthetic few-shot curriculum, three seeds.ordinary_s1: ordinary synthetic few-shot curriculum control.shuffled_train_s1: counterfactual input distribution with shuffled support labels during training.
All arms use the same synthetic/public evaluation seed. The final report is generated by src/aggregate_multiseed.py and includes both Markdown and HTML.
Final Read
The counterfactual adapter result is stable over three seeds: public full-task exact is 60.7% ± 2.6% versus 20.0% for base on the fixed public sample. It is also support-sensitive: shuffled support drops to 28.1% and no support drops to 1.5%.
The attribution is not clean. The ordinary synthetic-training control reaches 64.4%, and the shuffled-label training control reaches 62.2% with high shuffled-support performance. This supports a real public improvement from synthetic ICL-style LoRA tuning, but not a uniquely counterfactual support-induction mechanism.
Reports:
- Markdown:
reports/qwen_counterfactual_icl_public_multiseed_report.md - HTML:
reports/qwen_counterfactual_icl_public_multiseed_report.html
Report
Rendered from reports/qwen_counterfactual_icl_public_multiseed_report.md
Question
Can LoRA posttraining on counterfactual few-shot episodes make Qwen3-4B rely more on the support examples of a text-transformation task, and does that transfer to a public benchmark rather than only to the synthetic generator?
The training signal is answer-only. No public benchmark labels are used for training. The controls test whether the effect survives support shuffling, no-support prompts, ordinary synthetic training, and deliberately shuffled training support labels.
Headline
- Public PROSE full-task exact: base
20.0%; counterfactual adapter mean60.7%with seed spread2.6%; delta40.7%. - Synthetic counterfactual full-task exact: base
38.3%; counterfactual adapter mean92.8%. - Public support controls for the counterfactual adapter: normal
60.7%, shuffled28.1%, no support1.5%. - Ordinary synthetic-training control on public PROSE:
64.4%. - Shuffled-label training control on public PROSE:
62.2%.
Aggregate Metrics
| arm | split | support_mode | runs | tasks | rows | row_exact_mean | row_exact_std | full_task_exact_mean | full_task_exact_std |
|---|---|---|---|---|---|---|---|---|---|
| base | public_prose | normal | 1 | 45 | 135 | 37.0% | 0.0% | 20.0% | 0.0% |
| base | public_prose | shuffled | 1 | 45 | 135 | 11.1% | 0.0% | 2.2% | 0.0% |
| counterfactual_adapter | public_prose | none | 3 | 45 | 135 | 3.7% | 0.7% | 1.5% | 1.3% |
| counterfactual_adapter | public_prose | normal | 3 | 45 | 135 | 70.6% | 1.5% | 60.7% | 2.6% |
| counterfactual_adapter | public_prose | shuffled | 3 | 45 | 135 | 45.4% | 3.7% | 28.1% | 2.6% |
| ordinary_adapter | public_prose | none | 1 | 45 | 135 | 5.2% | 0.0% | 0.0% | 0.0% |
| ordinary_adapter | public_prose | normal | 1 | 45 | 135 | 74.1% | 0.0% | 64.4% | 0.0% |
| ordinary_adapter | public_prose | shuffled | 1 | 45 | 135 | 48.9% | 0.0% | 26.7% | 0.0% |
| shuffled_label_adapter | public_prose | none | 1 | 45 | 135 | 4.4% | 0.0% | 2.2% | 0.0% |
| shuffled_label_adapter | public_prose | normal | 1 | 45 | 135 | 74.1% | 0.0% | 62.2% | 0.0% |
| shuffled_label_adapter | public_prose | shuffled | 1 | 45 | 135 | 64.4% | 0.0% | 51.1% | 0.0% |
| base | synthetic_counterfactual | normal | 1 | 60 | 120 | 53.3% | 0.0% | 38.3% | 0.0% |
| base | synthetic_counterfactual | shuffled | 1 | 60 | 120 | 20.8% | 0.0% | 16.7% | 0.0% |
| counterfactual_adapter | synthetic_counterfactual | none | 3 | 60 | 120 | 5.3% | 3.4% | 2.8% | 1.9% |
| counterfactual_adapter | synthetic_counterfactual | normal | 3 | 60 | 120 | 96.1% | 0.5% | 92.8% | 1.9% |
| counterfactual_adapter | synthetic_counterfactual | shuffled | 3 | 60 | 120 | 60.3% | 2.9% | 51.1% | 2.5% |
| ordinary_adapter | synthetic_counterfactual | none | 1 | 60 | 120 | 2.5% | 0.0% | 0.0% | 0.0% |
| ordinary_adapter | synthetic_counterfactual | normal | 1 | 60 | 120 | 96.7% | 0.0% | 95.0% | 0.0% |
| ordinary_adapter | synthetic_counterfactual | shuffled | 1 | 60 | 120 | 58.3% | 0.0% | 46.7% | 0.0% |
| shuffled_label_adapter | synthetic_counterfactual | none | 1 | 60 | 120 | 0.8% | 0.0% | 0.0% | 0.0% |
| shuffled_label_adapter | synthetic_counterfactual | normal | 1 | 60 | 120 | 97.5% | 0.0% | 95.0% | 0.0% |
| shuffled_label_adapter | synthetic_counterfactual | shuffled | 1 | 60 | 120 | 95.8% | 0.0% | 91.7% | 0.0% |
Seed-Level Normal-Support Metrics
| run_name | seed | arm | split | tasks | rows | row_exact | full_task_exact |
|---|---|---|---|---|---|---|---|
| cf_s1 | 20260628 | base | public_prose | 45 | 135 | 37.0% | 20.0% |
| cf_s1 | 20260628 | base | synthetic_counterfactual | 60 | 120 | 53.3% | 38.3% |
| cf_s1 | 20260628 | counterfactual_adapter | public_prose | 45 | 135 | 68.9% | 57.8% |
| cf_s1 | 20260628 | counterfactual_adapter | synthetic_counterfactual | 60 | 120 | 95.8% | 91.7% |
| cf_s2 | 20260629 | counterfactual_adapter | public_prose | 45 | 135 | 71.1% | 62.2% |
| cf_s2 | 20260629 | counterfactual_adapter | synthetic_counterfactual | 60 | 120 | 96.7% | 95.0% |
| cf_s3 | 20260630 | counterfactual_adapter | public_prose | 45 | 135 | 71.9% | 62.2% |
| cf_s3 | 20260630 | counterfactual_adapter | synthetic_counterfactual | 60 | 120 | 95.8% | 91.7% |
| ordinary_s1 | 20260628 | ordinary_adapter | public_prose | 45 | 135 | 74.1% | 64.4% |
| ordinary_s1 | 20260628 | ordinary_adapter | synthetic_counterfactual | 60 | 120 | 96.7% | 95.0% |
| shuffled_train_s1 | 20260628 | shuffled_label_adapter | public_prose | 45 | 135 | 74.1% | 62.2% |
| shuffled_train_s1 | 20260628 | shuffled_label_adapter | synthetic_counterfactual | 60 | 120 | 97.5% | 95.0% |
Interpretation
The synthetic counterfactual split shows the intended training effect: the adapter improves strict task consistency, and that improvement depends on intact support examples. The public split shows positive transfer from synthetic counterfactual episodes to unseen public text-transformation tasks. The public gain is support-sensitive: corrupting the support examples removes a material fraction of the effect. The ordinary synthetic control is close enough that the counterfactual ingredient is not isolated. The shuffled-label training control is close enough that a formatting explanation remains plausible.
Charts
Public Task Flips Versus Base
| run_name | seed | helped | hurt | tied | tasks |
|---|---|---|---|---|---|
| cf_s1 | 20260628 | 18 | 1 | 26 | 45 |
| cf_s2 | 20260629 | 20 | 1 | 24 | 45 |
| cf_s3 | 20260630 | 20 | 1 | 24 | 45 |
Public Family Breakdown
| family | tasks_base | tasks_counterfactual_adapter | full_task_exact_base | full_task_exact_counterfactual_adapter | row_exact_base | row_exact_counterfactual_adapter | delta |
|---|---|---|---|---|---|---|---|
| Column | 1 | 1 | 0.0% | 100.0% | 0.0% | 100.0% | 100.0% |
| Currency | 1 | 1 | 0.0% | 100.0% | 66.7% | 100.0% | 100.0% |
| EmergencyCall | 1 | 1 | 0.0% | 100.0% | 0.0% | 100.0% | 100.0% |
| FilePath | 1 | 1 | 0.0% | 100.0% | 0.0% | 100.0% | 100.0% |
| ShippingCode | 3 | 3 | 0.0% | 88.9% | 33.3% | 92.6% | 88.9% |
| Number | 10 | 10 | 10.0% | 63.3% | 30.0% | 72.2% | 53.3% |
| Name | 5 | 5 | 0.0% | 53.3% | 20.0% | 68.9% | 53.3% |
| Phone | 3 | 3 | 66.7% | 100.0% | 66.7% | 100.0% | 33.3% |
| DateTime | 15 | 15 | 20.0% | 44.4% | 35.6% | 53.3% | 24.4% |
| Meteorite | 1 | 1 | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% |
| 2 | 2 | 50.0% | 50.0% | 83.3% | 83.3% | 0.0% | |
| Song | 1 | 1 | 100.0% | 100.0% | 100.0% | 100.0% | 0.0% |
| Address | 1 | 1 | 100.0% | 0.0% | 100.0% | 66.7% | -100.0% |
Public Error Sample
| run_name | task_id | family | input | target | prediction |
|---|---|---|---|---|---|
| cf_s1 | Address.000013 | Address | One Main Parkway, Allentown, ND 41230 | nan | One |
| cf_s1 | DateTime.000016 | DateTime | Jan 10 1975 | Fri W2 | Wed W1 |
| cf_s1 | DateTime.000016 | DateTime | 1 Feb 2013 | Fri W5 | Wed W5 |
| cf_s1 | DateTime.000016 | DateTime | 12 Dec 2002 | Thu W50 | Wed W46 |
| cf_s1 | DateTime.000017 | DateTime | 03302241 | 30/3/2241 | 3/3/2241 |
| cf_s1 | DateTime.000029 | DateTime | 03302241 | Tuesday | Monday |
| cf_s1 | DateTime.000029 | DateTime | 02-Aug-2160 | Saturday | Monday |
| cf_s1 | DateTime.000029 | DateTime | 23 May 1984 | Wednesday | Thursday |
| cf_s1 | DateTime.000032 | DateTime | 09:53 | 9:53 AM | 09:53 AM |
| cf_s1 | DateTime.000032 | DateTime | 1956-12-16 20:18 | 8:18 PM | December 16, 1956 at 8:18 PM |
| cf_s1 | DateTime.000041 | DateTime | 4.2.1743 | 2/4/1743 | 4.2.1743 |
| cf_s1 | DateTime.000081 | DateTime | 11:48PM | 11:45PM-12:15AM | 11:15PM-11:45PM |
| cf_s1 | DateTime.000084 | DateTime | 3/30/2241 16:15 | Tue 4:30 PM | Fri 16:15 |
| cf_s1 | DateTime.000084 | DateTime | 8/2/2160 23:48 | Sat 12:00 AM | Wed 12:00 AM |
| cf_s1 | DateTime.000084 | DateTime | 5/23/1984 22:24 | Wed 10:30 PM | Thu 10:00 PM |
| cf_s1 | DateTime.000086 | DateTime | 16:15:08 | 4:00PM | 4:15PM |
| cf_s1 | DateTime.000086 | DateTime | 23:48:20 | 11:30PM | 11:48PM |
| cf_s1 | DateTime.000086 | DateTime | 22:24:59 | 10:00PM | 10:24PM |
| cf_s1 | DateTime.000109 | DateTime | 21-Jan-1985 05:44:43 | Monday, January 21, 1985 | Wednesday, January 21, 1985 |
| cf_s1 | DateTime.000109 | DateTime | 16-Aug-1985 01:11:56 | Friday, August 16, 1985 | Wednesday, August 16, 1985 |
| cf_s1 | DateTime.000109 | DateTime | 20-Dec-2033 18:36:29 | Tuesday, December 20, 2033 | Friday, December 20, 2033 |
| cf_s1 | DateTime.000115 | DateTime | 21-Jan-1985 05:44:43 | 40-60 | 0-20 |
| cf_s1 | DateTime.000115 | DateTime | 16-Aug-1985 01:11:56 | 40-60 | 0-20 |
| cf_s1 | DateTime.000115 | DateTime | 20-Dec-2033 18:36:29 | 20-40 | 0-20 |
| cf_s1 | Email.000006 | iñaki | iñaki@proseware.com | iñaki@contoso.com | |
| cf_s1 | Meteorite.000001 | Meteorite | Zunyi | 682 | 2243-10-11 11:33:09 |
| cf_s1 | Meteorite.000001 | Meteorite | Elgin | 53648 | 2139-04-25 19:02:49 |
| cf_s1 | Meteorite.000001 | Meteorite | New Haven | 066 | 2041-05-26 22:15:47 |
| cf_s1 | Name.000023 | Name | Cecep Sutresna | Sutr | Sutre |
| cf_s1 | Name.000023 | Name | Milica Zujovic | Zujo | Zujovic |
| cf_s1 | Name.000037 | Name | Marcela Kubatova | nan | ova |
| cf_s1 | Name.000037 | Name | Mr. Hadar Caspit | nan | Mr. |
| cf_s1 | Number.000008 | Number | -13578 | -135.78 | -13.578 |
| cf_s1 | Number.000008 | Number | -1961.1180 | -19.611180 | -19.6118 |
| cf_s1 | Number.000047 | Number | 1284.42 | 1285.00 | 1280.00 |
| cf_s1 | Number.000047 | Number | 23224.98 | 23225.00 | 23220.00 |
| cf_s1 | Number.000047 | Number | 1024.21 | 1025.00 | 1020.00 |
| cf_s1 | Number.000069 | Number | 1202.3433 | 1200 | 1202 |
| cf_s1 | Number.000069 | Number | 23224.1 | 23225 | 23224 |
| cf_s1 | Number.000069 | Number | -23224.1 | -23225 | -23224 |
| cf_s1 | ShippingCode.000002 | ShippingCode | 1Z I81 6QF 90 9601 169 4 | 6QF | I81 |
| cf_s1 | ShippingCode.000002 | ShippingCode | 1Z O63 B7Z 35 2550 248 6 | B7Z | O63 |
| cf_s2 | Address.000013 | Address | One Main Parkway, Allentown, ND 41230 | nan | One |
| cf_s2 | DateTime.000016 | DateTime | Jan 10 1975 | Fri W2 | Wed W52 |
| cf_s2 | DateTime.000016 | DateTime | 12 Dec 2002 | Thu W50 | Thu W49 |
| cf_s2 | DateTime.000029 | DateTime | 03302241 | Tuesday | Monday |
| cf_s2 | DateTime.000029 | DateTime | 02-Aug-2160 | Saturday | Monday |
| cf_s2 | DateTime.000029 | DateTime | 23 May 1984 | Wednesday | Monday |
| cf_s2 | DateTime.000081 | DateTime | 4:15PM | 4:15PM-4:45PM | 4:00PM-4:30PM |
| cf_s2 | DateTime.000081 | DateTime | 11:48PM | 11:45PM-12:15AM | 11:15PM-11:45PM |
| cf_s2 | DateTime.000084 | DateTime | 3/30/2241 16:15 | Tue 4:30 PM | Fri 4:00 PM |
| cf_s2 | DateTime.000084 | DateTime | 8/2/2160 23:48 | Sat 12:00 AM | Wed 12:00 AM |
| cf_s2 | DateTime.000084 | DateTime | 5/23/1984 22:24 | Wed 10:30 PM | Tue 10:00 PM |
| cf_s2 | DateTime.000086 | DateTime | 16:15:08 | 4:00PM | 4:15PM |
| cf_s2 | DateTime.000086 | DateTime | 23:48:20 | 11:30PM | 11:40PM |
| cf_s2 | DateTime.000086 | DateTime | 22:24:59 | 10:00PM | 10:15PM |
| cf_s2 | DateTime.000109 | DateTime | 21-Jan-1985 05:44:43 | Monday, January 21, 1985 | Wednesday, January 21, 1985 |
| cf_s2 | DateTime.000109 | DateTime | 16-Aug-1985 01:11:56 | Friday, August 16, 1985 | Monday, August 16, 1985 |
| cf_s2 | DateTime.000109 | DateTime | 20-Dec-2033 18:36:29 | Tuesday, December 20, 2033 | Monday, December 20, 2033 |
| cf_s2 | DateTime.000115 | DateTime | 21-Jan-1985 05:44:43 | 40-60 | 0-20 |
Artifacts
- Experiment root:
/workspace/experiments/qwen_counterfactual_icl_public_multiseed - Large artifacts root:
/workspace/large_artifacts/qwen_counterfactual_icl_public_multiseed - Per-run artifacts:
/workspace/experiments/qwen_counterfactual_icl_public_multiseed/runs - Adapter checkpoints:
/workspace/large_artifacts/qwen_counterfactual_icl_public_multiseed/checkpoints - Aggregate CSV:
/workspace/experiments/qwen_counterfactual_icl_public_multiseed/analysis/aggregate_summary.csv - Combined row predictions:
/workspace/experiments/qwen_counterfactual_icl_public_multiseed/analysis/aggregate_row_predictions.csv - Combined task metrics:
/workspace/experiments/qwen_counterfactual_icl_public_multiseed/analysis/aggregate_task_metrics.csv
Limitations
The public evaluation is capped for runtime, and exact-match scoring is strict. The main counterfactual arm is multiseed; ordinary and shuffled-label controls are single-seed controls in this run. The experiment tests transfer from a synthetic counterfactual training distribution, not broad open-ended transformation ability.
Experiment log 9
Show the running log (9 entries)
Setup
- Created fresh standalone experiment directory:
/workspace/experiments/qwen_counterfactual_icl_public_multiseed. - Large artifacts directory:
/workspace/large_artifacts/qwen_counterfactual_icl_public_multiseed. - Public benchmark mirror copied under the large-artifact directory.
- Primary question: does counterfactual episodic LoRA posttraining improve support-conditioned public text transformation accuracy across seeds, beyond output-format or generic synthetic-training effects?
Run smoke_cf
- Time UTC:
2026-06-28T05:50:51.863838+00:00 - Config:
{"batch_size": 1, "eval_seed": 20260701, "grad_accum": 2, "log_every": 1, "lora_alpha": 32, "lora_dropout": 0.05, "lora_r": 16, "lr": 0.0002, "max_grad_norm": 1.0, "max_length": 768, "max_new_tokens": 24, "model": "Qwen/Qwen3-4B", "no_train": false, "public_heldout_n": 3, "public_task_limit": 2, "query_n": 2, "run_name": "smoke_cf", "seed": 20260628, "skip_base_eval": false, "smoke": true, "support_n": 4, "synthetic_eval_pairs": 2, "train_episodes": 16, "train_mode": "counterfactual", "train_rows_per_episode": 2, "train_steps": 2, "warmup_steps": 20, "weight_decay": 0.0} - Elapsed seconds:
80.8 - Synthetic adapter full-task exact:
25.0% - Public adapter full-task exact:
0.0% - Report:
/workspace/experiments/qwen_counterfactual_icl_public_multiseed/reports/qwen_counterfactual_icl_public_multiseed_report.md
Aggregation Harness
- Added
src/aggregate_multiseed.py. - Compile check passed for the training and aggregate scripts.
- Smoke aggregation passed against
smoke_cf. - Main gate plan: three counterfactual seeds plus ordinary-training and shuffled-label controls, all with fixed public/synthetic eval seed.
Main Gate Results
- Completed
cf_s1,cf_s2,cf_s3,ordinary_s1, andshuffled_train_s1. - Fixed eval set: 60 synthetic counterfactual tasks and 45 public PROSE tasks.
- Base public full-task exact:
20.0%. - Counterfactual adapter public full-task exact: mean
60.7%, std2.6%over 3 seeds. - Counterfactual adapter public controls: shuffled support
28.1%, no support1.5%. - Ordinary synthetic-training control public full-task exact:
64.4%. - Shuffled-label training control public full-task exact:
62.2%; shuffled support remains high at51.1%. - Main read: the public improvement is stable, but it is not uniquely explained by counterfactual support-induction training. Generic synthetic ICL tuning and even corrupted-support training reproduce much of the public gain, so the effect is likely a broader prompt-format/task-family adaptation plus some support sensitivity.
- Final Markdown report:
/workspace/experiments/qwen_counterfactual_icl_public_multiseed/reports/qwen_counterfactual_icl_public_multiseed_report.md - Final HTML report:
/workspace/experiments/qwen_counterfactual_icl_public_multiseed/reports/qwen_counterfactual_icl_public_multiseed_report.html
Run cf_s1
- Time UTC:
2026-06-28T05:56:28.260234+00:00 - Config:
{"batch_size": 1, "eval_seed": 20260701, "grad_accum": 4, "log_every": 10, "lora_alpha": 32, "lora_dropout": 0.05, "lora_r": 16, "lr": 0.0002, "max_grad_norm": 1.0, "max_length": 768, "max_new_tokens": 24, "model": "Qwen/Qwen3-4B", "no_train": false, "public_heldout_n": 3, "public_task_limit": 45, "query_n": 2, "run_name": "cf_s1", "seed": 20260628, "skip_base_eval": false, "smoke": false, "support_n": 4, "synthetic_eval_pairs": 30, "train_episodes": 800, "train_mode": "counterfactual", "train_rows_per_episode": 2, "train_steps": 100, "warmup_steps": 20, "weight_decay": 0.0} - Elapsed seconds:
729.1 - Synthetic adapter full-task exact:
91.7% - Public adapter full-task exact:
57.8% - Report:
/workspace/experiments/qwen_counterfactual_icl_public_multiseed/reports/qwen_counterfactual_icl_public_multiseed_report.md
Run cf_s2
- Time UTC:
2026-06-28T06:08:54.410400+00:00 - Config:
{"batch_size": 1, "eval_seed": 20260701, "grad_accum": 4, "log_every": 10, "lora_alpha": 32, "lora_dropout": 0.05, "lora_r": 16, "lr": 0.0002, "max_grad_norm": 1.0, "max_length": 768, "max_new_tokens": 24, "model": "Qwen/Qwen3-4B", "no_train": false, "public_heldout_n": 3, "public_task_limit": 45, "query_n": 2, "run_name": "cf_s2", "seed": 20260629, "skip_base_eval": true, "smoke": false, "support_n": 4, "synthetic_eval_pairs": 30, "train_episodes": 800, "train_mode": "counterfactual", "train_rows_per_episode": 2, "train_steps": 100, "warmup_steps": 20, "weight_decay": 0.0} - Elapsed seconds:
409.3 - Synthetic adapter full-task exact:
95.0% - Public adapter full-task exact:
62.2% - Report:
/workspace/experiments/qwen_counterfactual_icl_public_multiseed/reports/qwen_counterfactual_icl_public_multiseed_report.md
Run cf_s3
- Time UTC:
2026-06-28T06:16:00.820755+00:00 - Config:
{"batch_size": 1, "eval_seed": 20260701, "grad_accum": 4, "log_every": 10, "lora_alpha": 32, "lora_dropout": 0.05, "lora_r": 16, "lr": 0.0002, "max_grad_norm": 1.0, "max_length": 768, "max_new_tokens": 24, "model": "Qwen/Qwen3-4B", "no_train": false, "public_heldout_n": 3, "public_task_limit": 45, "query_n": 2, "run_name": "cf_s3", "seed": 20260630, "skip_base_eval": true, "smoke": false, "support_n": 4, "synthetic_eval_pairs": 30, "train_episodes": 800, "train_mode": "counterfactual", "train_rows_per_episode": 2, "train_steps": 100, "warmup_steps": 20, "weight_decay": 0.0} - Elapsed seconds:
404.6 - Synthetic adapter full-task exact:
91.7% - Public adapter full-task exact:
62.2% - Report:
/workspace/experiments/qwen_counterfactual_icl_public_multiseed/reports/qwen_counterfactual_icl_public_multiseed_report.md
Run ordinary_s1
- Time UTC:
2026-06-28T06:22:59.909062+00:00 - Config:
{"batch_size": 1, "eval_seed": 20260701, "grad_accum": 4, "log_every": 10, "lora_alpha": 32, "lora_dropout": 0.05, "lora_r": 16, "lr": 0.0002, "max_grad_norm": 1.0, "max_length": 768, "max_new_tokens": 24, "model": "Qwen/Qwen3-4B", "no_train": false, "public_heldout_n": 3, "public_task_limit": 45, "query_n": 2, "run_name": "ordinary_s1", "seed": 20260628, "skip_base_eval": true, "smoke": false, "support_n": 4, "synthetic_eval_pairs": 30, "train_episodes": 800, "train_mode": "ordinary", "train_rows_per_episode": 2, "train_steps": 100, "warmup_steps": 20, "weight_decay": 0.0} - Elapsed seconds:
408.6 - Synthetic adapter full-task exact:
95.0% - Public adapter full-task exact:
64.4% - Report:
/workspace/experiments/qwen_counterfactual_icl_public_multiseed/reports/qwen_counterfactual_icl_public_multiseed_report.md
Run shuffled_train_s1
- Time UTC:
2026-06-28T06:30:03.483561+00:00 - Config:
{"batch_size": 1, "eval_seed": 20260701, "grad_accum": 4, "log_every": 10, "lora_alpha": 32, "lora_dropout": 0.05, "lora_r": 16, "lr": 0.0002, "max_grad_norm": 1.0, "max_length": 768, "max_new_tokens": 24, "model": "Qwen/Qwen3-4B", "no_train": false, "public_heldout_n": 3, "public_task_limit": 45, "query_n": 2, "run_name": "shuffled_train_s1", "seed": 20260628, "skip_base_eval": true, "smoke": false, "support_n": 4, "synthetic_eval_pairs": 30, "train_episodes": 800, "train_mode": "shuffled_labels", "train_rows_per_episode": 2, "train_steps": 100, "warmup_steps": 20, "weight_decay": 0.0} - Elapsed seconds:
371.2 - Synthetic adapter full-task exact:
95.0% - Public adapter full-task exact:
62.2% - Report:
/workspace/experiments/qwen_counterfactual_icl_public_multiseed/reports/qwen_counterfactual_icl_public_multiseed_report.md
Figures 9
Data files 20
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
analysis/aggregate_summary.csv2.5 kBanalysis/summary.csv501 Bruns/cf_s1/eval_episodes.csv44 kBruns/cf_s1/summary.csv829 Bruns/cf_s1/task_metrics.csv37 kBruns/cf_s2/eval_episodes.csv44 kBruns/cf_s2/summary.csv502 Bruns/cf_s2/task_metrics.csv23 kBruns/cf_s3/eval_episodes.csv44 kBruns/cf_s3/summary.csv529 Bruns/cf_s3/task_metrics.csv23 kBruns/ordinary_s1/eval_episodes.csv44 kBruns/ordinary_s1/summary.csv482 Bruns/ordinary_s1/task_metrics.csv23 kBruns/shuffled_train_s1/eval_episodes.csv44 kBruns/shuffled_train_s1/summary.csv501 Bruns/shuffled_train_s1/task_metrics.csv22 kBruns/smoke_cf/eval_episodes.csv2.3 kBruns/smoke_cf/summary.csv551 Bruns/smoke_cf/task_metrics.csv2.3 kB
Reproduce
The run commands are documented inside the experiment folder (see the README).