Counterfactual Episodic ICL Posttraining
The one idea you need
Picture flashcards where the very same word demands a different answer in each deck, so memorizing never pays off — the only way to score is to first read that deck's example pairs. Training on such deliberately contradictory decks forces the model to read the prompt's examples.
The question
If you train a small model on made-up tasks where the same input can demand opposite answers, does it learn to actually read the examples instead of guessing?
What we found
Yes. Untrained, a 4-billion-parameter model solved 23% of real text-transformation tasks perfectly; after this training, 57% — but only when it could see the prompt's examples. Scramble those examples and it fell to 17%, below where it started; remove them and it collapsed to about 3%. The entire gain lives in reading the examples, not memorizing a familiar transformation.
Why it matters
To make a small model obey in-context examples instead of falling back on a familiar guess, train it on example sets whose answers can only come from the examples — and always retest with the examples shuffled or removed; real conditioning makes accuracy crater.
On this page
Results at a glance 4
How to read
Bars show the share of tasks solved completely correctly; taller is better. Four conditions along the bottom: untrained with examples, trained with examples, trained with scrambled examples, trained with no examples. One color is made-up unseen tasks, the other real public tasks.
Takeaway → Trained-with-examples towers over every other bar; scrambling or removing the examples crashes accuracy to near or below the untrained level — winning requires reading the examples.
Data table
| condition | synthetic held-out counterfactual | public text-transformation tasks |
|---|---|---|
| base + support | 25% | 23.3% |
| adapter + support | 91.7% | 56.7% |
| adapter + shuffled support | 31.7% | 16.7% |
| adapter, no support | 1.7% | 3.3% |
Numbers from runs/main_v2/summary.csv
Technical framing
Full-task exact match: counterfactual-episode LoRA vs base — The adapter learns to read the support examples: 25%→92% held-out and 23%→57% public, and it collapses when support is shuffled or removed.
How to read
Bars show the share of individual output lines matched exactly, not whole tasks; taller is better. Same four conditions and same two task groups — made-up unseen versus real public — as the whole-task chart.
Takeaway → The looser per-line scoring shows the identical shape: big gains only when correct examples are present, confirming genuine example-reading rather than an artifact of strict all-or-nothing scoring.
Data table
| condition | synthetic held-out counterfactual | public text-transformation tasks |
|---|---|---|
| base + support | 41.7% | 44.4% |
| adapter + support | 95% | 73.3% |
| adapter + shuffled support | 43.3% | 33.3% |
| adapter, no support | 3.3% | 10% |
Numbers from runs/main_v2/summary.csv
Technical framing
Row-level exact match under the same conditions — Per-row accuracy shows the same pattern as strict full-task scoring: gains come from support conditioning, not a memorized task prior.
How to read
A single line traces the model's training error (vertical) against training step from 1 to 120 (horizontal); lower is better. Only the model's answer tokens are scored during training, not the prompt examples.
Takeaway → Error plunges from about 7 to near zero within roughly 40 steps, then stays flat — the model masters the contradictory tasks quickly and cheaply.
Data table
| training step | training loss |
|---|---|
| 1 | 7.09 |
| 10 | 5.97 |
| 20 | 0.947 |
| 30 | 0.159 |
| 40 | 0.327 |
| 50 | 0.124 |
| 60 | 0.169 |
| 70 | 0.158 |
| 80 | 0.032 |
| 90 | 0.0104 |
| 100 | 0.122 |
| 110 | 0.0609 |
| 120 | 0.0496 |
Numbers from runs/main_v2/training_log.csv
Technical framing
LoRA training loss (main_v2, 120 steps) — Answer-only loss on counterfactual episodes converges within ~40 steps of the 120-step run.
How to read
Whole-task accuracy bars; taller is better. Groups pair each task set — made-up unseen and real public — across a smaller 240-task run and the larger 800-task run. Three colors: untrained, trained with examples, trained with scrambled examples.
Takeaway → Going from 240 to 800 training tasks raises real-task accuracy from 38% to 57% and deepens the drop under scrambled examples — stronger, more genuine example-reading.
Data table
| Eval split and run | Base, normal support | Adapter, normal support | Adapter, shuffled support |
|---|---|---|---|
| Synthetic held-out — pilot_v1 | 18.8% | 93.8% | 43.8% |
| Synthetic held-out — main_v2 | 25% | 91.7% | 31.7% |
| Public PROSE — pilot_v1 | 12.5% | 37.5% | 37.5% |
| Public PROSE — main_v2 | 23.3% | 56.7% | 16.7% |
Numbers from experiments/qwen_counterfactual_episodic_icl/runs/pilot_v1/summary.csv; runs/main_v2/summary.csv
Technical framing
Training scale: full-task exact match, pilot (240 episodes) vs main (800 episodes) — Scaling 240 to 800 train episodes lifts public transfer and makes shuffled support hurt more, evidence of real support conditioning.
In the author’s words from the Report · “Interpretation”
On held-out synthetic counterfactual episodes, the adapter changes full-task exactness from 25.0% to 91.7%. With shuffled support examples, the adapter scores 31.7%. On the public text-transformation sample, the adapter changes full-task exactness from 23.3% to 56.7%. With shuffled support examples, the adapter scores 16.7%. The synthetic split shows a real support-conditioning effect: performance improves and shuffled support degrades it. The public split determines whether that learned behavior transfers outside the synthetic generator. The public benchmark transfer is positive at the tested scale.
Overview
This standalone experiment tests whether answer-only LoRA posttraining on counterfactual few-shot episodes improves Qwen's ability to infer a text transformation from support examples.
The training data is synthetic and deliberately counterfactual: the same input style can require incompatible outputs depending on the examples in the prompt. Public benchmark examples are used only for evaluation.
Main Run
- Run:
main_v2 - Model:
Qwen/Qwen3-4B - Adapter:
/workspace/large_artifacts/qwen_counterfactual_episodic_icl/checkpoints/main_v2/adapter - Synthetic held-out counterfactual full-task exact: base
25.0%, adapter91.7%, adapter with shuffled support31.7%. - Public text-transformation full-task exact: base
23.3%, adapter56.7%, adapter with shuffled support16.7%.
Artifacts
- Report:
reports/qwen_counterfactual_episodic_icl_report.md - HTML report:
reports/qwen_counterfactual_episodic_icl_report.html - Source:
src/qwen_counterfactual_episodic_icl.py - Run CSVs:
runs/main_v2/ - Figures:
analysis/figures/ - Large artifacts:
/workspace/large_artifacts/qwen_counterfactual_episodic_icl
Report
Rendered from reports/qwen_counterfactual_episodic_icl_report.md
Question
Can answer-only LoRA posttraining on counterfactual few-shot episodes improve a 4B model's ability to infer a task from support examples, rather than relying on a task-family prior?
The training episodes are synthetic and deliberately counterfactual: the same kind of query input can require incompatible outputs depending on the support examples. Public benchmark outputs are used only for evaluation.
Setup
- Run:
main_v2 - Model:
Qwen/Qwen3-4B - Train episodes:
800 - Train steps:
120 - LoRA rank:
16 - Synthetic eval counterfactual pairs:
30 - Public PROSE tasks:
30 - Elapsed seconds:
588.9
Main Result
| method | split | support_mode | tasks | rows | row_exact | full_task_exact |
|---|---|---|---|---|---|---|
| adapter | public_prose | none | 30 | 90 | 10.0% | 3.3% |
| adapter | public_prose | normal | 30 | 90 | 73.3% | 56.7% |
| adapter | public_prose | shuffled | 30 | 90 | 33.3% | 16.7% |
| base | public_prose | normal | 30 | 90 | 44.4% | 23.3% |
| base | public_prose | shuffled | 30 | 90 | 12.2% | 6.7% |
| adapter | synthetic_counterfactual | none | 60 | 120 | 3.3% | 1.7% |
| adapter | synthetic_counterfactual | normal | 60 | 120 | 95.0% | 91.7% |
| adapter | synthetic_counterfactual | shuffled | 60 | 120 | 43.3% | 31.7% |
| base | synthetic_counterfactual | normal | 60 | 120 | 41.7% | 25.0% |
| base | synthetic_counterfactual | shuffled | 60 | 120 | 13.3% | 8.3% |
Interpretation
On held-out synthetic counterfactual episodes, the adapter changes full-task exactness from 25.0% to 91.7%. With shuffled support examples, the adapter scores 31.7%. On the public text-transformation sample, the adapter changes full-task exactness from 23.3% to 56.7%. With shuffled support examples, the adapter scores 16.7%.
The synthetic split shows a real support-conditioning effect: performance improves and shuffled support degrades it. The public split determines whether that learned behavior transfers outside the synthetic generator. The public benchmark transfer is positive at the tested scale.
Charts
Task-Level Details
| method | split | support_mode | task_id | family | row_exact | full_task_exact | rows |
|---|---|---|---|---|---|---|---|
| adapter | public_prose | none | Address.000014 | Address | 0.0% | 0.0% | 3 |
| adapter | public_prose | normal | Address.000014 | Address | 100.0% | 100.0% | 3 |
| adapter | public_prose | shuffled | Address.000014 | Address | 100.0% | 100.0% | 3 |
| base | public_prose | normal | Address.000014 | Address | 100.0% | 100.0% | 3 |
| base | public_prose | shuffled | Address.000014 | Address | 0.0% | 0.0% | 3 |
| adapter | public_prose | none | Currency.000003 | Currency | 66.7% | 0.0% | 3 |
| adapter | public_prose | normal | Currency.000003 | Currency | 100.0% | 100.0% | 3 |
| adapter | public_prose | shuffled | Currency.000003 | Currency | 0.0% | 0.0% | 3 |
| base | public_prose | normal | Currency.000003 | Currency | 66.7% | 0.0% | 3 |
| base | public_prose | shuffled | Currency.000003 | Currency | 0.0% | 0.0% | 3 |
| adapter | public_prose | none | DateTime.000003 | DateTime | 0.0% | 0.0% | 3 |
| adapter | public_prose | normal | DateTime.000003 | DateTime | 100.0% | 100.0% | 3 |
| adapter | public_prose | shuffled | DateTime.000003 | DateTime | 0.0% | 0.0% | 3 |
| base | public_prose | normal | DateTime.000003 | DateTime | 33.3% | 0.0% | 3 |
| base | public_prose | shuffled | DateTime.000003 | DateTime | 0.0% | 0.0% | 3 |
| adapter | public_prose | none | DateTime.000009 | DateTime | 0.0% | 0.0% | 3 |
| adapter | public_prose | normal | DateTime.000009 | DateTime | 100.0% | 100.0% | 3 |
| adapter | public_prose | shuffled | DateTime.000009 | DateTime | 100.0% | 100.0% | 3 |
| base | public_prose | normal | DateTime.000009 | DateTime | 0.0% | 0.0% | 3 |
| base | public_prose | shuffled | DateTime.000009 | DateTime | 0.0% | 0.0% | 3 |
| adapter | public_prose | none | DateTime.000012 | DateTime | 66.7% | 0.0% | 3 |
| adapter | public_prose | normal | DateTime.000012 | DateTime | 66.7% | 0.0% | 3 |
| adapter | public_prose | shuffled | DateTime.000012 | DateTime | 0.0% | 0.0% | 3 |
| base | public_prose | normal | DateTime.000012 | DateTime | 66.7% | 0.0% | 3 |
| base | public_prose | shuffled | DateTime.000012 | DateTime | 0.0% | 0.0% | 3 |
| adapter | public_prose | none | DateTime.000018 | DateTime | 0.0% | 0.0% | 3 |
| adapter | public_prose | normal | DateTime.000018 | DateTime | 100.0% | 100.0% | 3 |
| adapter | public_prose | shuffled | DateTime.000018 | DateTime | 66.7% | 0.0% | 3 |
| base | public_prose | normal | DateTime.000018 | DateTime | 0.0% | 0.0% | 3 |
| base | public_prose | shuffled | DateTime.000018 | DateTime | 0.0% | 0.0% | 3 |
| adapter | public_prose | none | DateTime.000030 | DateTime | 0.0% | 0.0% | 3 |
| adapter | public_prose | normal | DateTime.000030 | DateTime | 0.0% | 0.0% | 3 |
| adapter | public_prose | shuffled | DateTime.000030 | DateTime | 0.0% | 0.0% | 3 |
| base | public_prose | normal | DateTime.000030 | DateTime | 33.3% | 0.0% | 3 |
| base | public_prose | shuffled | DateTime.000030 | DateTime | 0.0% | 0.0% | 3 |
| adapter | public_prose | none | DateTime.000032 | DateTime | 0.0% | 0.0% | 3 |
| adapter | public_prose | normal | DateTime.000032 | DateTime | 66.7% | 0.0% | 3 |
| adapter | public_prose | shuffled | DateTime.000032 | DateTime | 33.3% | 0.0% | 3 |
| base | public_prose | normal | DateTime.000032 | DateTime | 33.3% | 0.0% | 3 |
| base | public_prose | shuffled | DateTime.000032 | DateTime | 33.3% | 0.0% | 3 |
| adapter | public_prose | none | DateTime.000033 | DateTime | 0.0% | 0.0% | 3 |
| adapter | public_prose | normal | DateTime.000033 | DateTime | 100.0% | 100.0% | 3 |
| adapter | public_prose | shuffled | DateTime.000033 | DateTime | 0.0% | 0.0% | 3 |
| base | public_prose | normal | DateTime.000033 | DateTime | 33.3% | 0.0% | 3 |
| base | public_prose | shuffled | DateTime.000033 | DateTime | 0.0% | 0.0% | 3 |
| adapter | public_prose | none | DateTime.000075 | DateTime | 0.0% | 0.0% | 3 |
| adapter | public_prose | normal | DateTime.000075 | DateTime | 66.7% | 0.0% | 3 |
| adapter | public_prose | shuffled | DateTime.000075 | DateTime | 0.0% | 0.0% | 3 |
| base | public_prose | normal | DateTime.000075 | DateTime | 33.3% | 0.0% | 3 |
| base | public_prose | shuffled | DateTime.000075 | DateTime | 0.0% | 0.0% | 3 |
| adapter | public_prose | none | DateTime.000091 | DateTime | 0.0% | 0.0% | 3 |
| adapter | public_prose | normal | DateTime.000091 | DateTime | 100.0% | 100.0% | 3 |
| adapter | public_prose | shuffled | DateTime.000091 | DateTime | 66.7% | 0.0% | 3 |
| base | public_prose | normal | DateTime.000091 | DateTime | 100.0% | 100.0% | 3 |
| base | public_prose | shuffled | DateTime.000091 | DateTime | 0.0% | 0.0% | 3 |
| adapter | public_prose | none | DateTime.000097 | DateTime | 0.0% | 0.0% | 3 |
| adapter | public_prose | normal | DateTime.000097 | DateTime | 66.7% | 0.0% | 3 |
| adapter | public_prose | shuffled | DateTime.000097 | DateTime | 33.3% | 0.0% | 3 |
| base | public_prose | normal | DateTime.000097 | DateTime | 66.7% | 0.0% | 3 |
| base | public_prose | shuffled | DateTime.000097 | DateTime | 0.0% | 0.0% | 3 |
| adapter | public_prose | none | DateTime.000116 | DateTime | 0.0% | 0.0% | 3 |
| adapter | public_prose | normal | DateTime.000116 | DateTime | 66.7% | 0.0% | 3 |
| adapter | public_prose | shuffled | DateTime.000116 | DateTime | 66.7% | 0.0% | 3 |
| base | public_prose | normal | DateTime.000116 | DateTime | 66.7% | 0.0% | 3 |
| base | public_prose | shuffled | DateTime.000116 | DateTime | 66.7% | 0.0% | 3 |
| adapter | public_prose | none | Email.000013 | 0.0% | 0.0% | 3 | |
| adapter | public_prose | normal | Email.000013 | 100.0% | 100.0% | 3 | |
| adapter | public_prose | shuffled | Email.000013 | 100.0% | 100.0% | 3 | |
| base | public_prose | normal | Email.000013 | 100.0% | 100.0% | 3 | |
| base | public_prose | shuffled | Email.000013 | 100.0% | 100.0% | 3 | |
| adapter | public_prose | none | Language.000002 | Language | 0.0% | 0.0% | 3 |
| adapter | public_prose | normal | Language.000002 | Language | 100.0% | 100.0% | 3 |
| adapter | public_prose | shuffled | Language.000002 | Language | 0.0% | 0.0% | 3 |
| base | public_prose | normal | Language.000002 | Language | 66.7% | 0.0% | 3 |
| base | public_prose | shuffled | Language.000002 | Language | 0.0% | 0.0% | 3 |
| adapter | public_prose | none | Name.000008 | Name | 0.0% | 0.0% | 3 |
| adapter | public_prose | normal | Name.000008 | Name | 0.0% | 0.0% | 3 |
| adapter | public_prose | shuffled | Name.000008 | Name | 0.0% | 0.0% | 3 |
| base | public_prose | normal | Name.000008 | Name | 0.0% | 0.0% | 3 |
| base | public_prose | shuffled | Name.000008 | Name | 0.0% | 0.0% | 3 |
| adapter | public_prose | none | Name.000025 | Name | 0.0% | 0.0% | 3 |
| adapter | public_prose | normal | Name.000025 | Name | 100.0% | 100.0% | 3 |
| adapter | public_prose | shuffled | Name.000025 | Name | 0.0% | 0.0% | 3 |
| base | public_prose | normal | Name.000025 | Name | 0.0% | 0.0% | 3 |
| base | public_prose | shuffled | Name.000025 | Name | 0.0% | 0.0% | 3 |
| adapter | public_prose | none | Name.000037 | Name | 0.0% | 0.0% | 3 |
| adapter | public_prose | normal | Name.000037 | Name | 0.0% | 0.0% | 3 |
| adapter | public_prose | shuffled | Name.000037 | Name | 33.3% | 0.0% | 3 |
| base | public_prose | normal | Name.000037 | Name | 33.3% | 0.0% | 3 |
| base | public_prose | shuffled | Name.000037 | Name | 33.3% | 0.0% | 3 |
| adapter | public_prose | none | Number.000047 | Number | 0.0% | 0.0% | 3 |
| adapter | public_prose | normal | Number.000047 | Number | 0.0% | 0.0% | 3 |
| adapter | public_prose | shuffled | Number.000047 | Number | 33.3% | 0.0% | 3 |
| base | public_prose | normal | Number.000047 | Number | 0.0% | 0.0% | 3 |
| base | public_prose | shuffled | Number.000047 | Number | 0.0% | 0.0% | 3 |
| adapter | public_prose | none | Number.000054 | Number | 33.3% | 0.0% | 3 |
| adapter | public_prose | normal | Number.000054 | Number | 66.7% | 0.0% | 3 |
| adapter | public_prose | shuffled | Number.000054 | Number | 66.7% | 0.0% | 3 |
| base | public_prose | normal | Number.000054 | Number | 33.3% | 0.0% | 3 |
| base | public_prose | shuffled | Number.000054 | Number | 0.0% | 0.0% | 3 |
| adapter | public_prose | none | Number.000062 | Number | 0.0% | 0.0% | 3 |
| adapter | public_prose | normal | Number.000062 | Number | 100.0% | 100.0% | 3 |
| adapter | public_prose | shuffled | Number.000062 | Number | 0.0% | 0.0% | 3 |
| base | public_prose | normal | Number.000062 | Number | 0.0% | 0.0% | 3 |
| base | public_prose | shuffled | Number.000062 | Number | 0.0% | 0.0% | 3 |
| adapter | public_prose | none | Number.000069 | Number | 0.0% | 0.0% | 3 |
| adapter | public_prose | normal | Number.000069 | Number | 0.0% | 0.0% | 3 |
| adapter | public_prose | shuffled | Number.000069 | Number | 0.0% | 0.0% | 3 |
| base | public_prose | normal | Number.000069 | Number | 0.0% | 0.0% | 3 |
| base | public_prose | shuffled | Number.000069 | Number | 0.0% | 0.0% | 3 |
| adapter | public_prose | none | Number.000082 | Number | 100.0% | 100.0% | 3 |
| adapter | public_prose | normal | Number.000082 | Number | 100.0% | 100.0% | 3 |
| adapter | public_prose | shuffled | Number.000082 | Number | 0.0% | 0.0% | 3 |
| base | public_prose | normal | Number.000082 | Number | 100.0% | 100.0% | 3 |
| base | public_prose | shuffled | Number.000082 | Number | 33.3% | 0.0% | 3 |
| adapter | public_prose | none | Number.000084 | Number | 0.0% | 0.0% | 3 |
| adapter | public_prose | normal | Number.000084 | Number | 66.7% | 0.0% | 3 |
| adapter | public_prose | shuffled | Number.000084 | Number | 33.3% | 0.0% | 3 |
| base | public_prose | normal | Number.000084 | Number | 33.3% | 0.0% | 3 |
| base | public_prose | shuffled | Number.000084 | Number | 0.0% | 0.0% | 3 |
Error Examples
| split | task_id | family | rule_name | input | target | prediction |
|---|---|---|---|---|---|---|
| synthetic_counterfactual | cf_0002_number_last2 | number | number_last2 | COBALT 321 | 21 | 13 |
| synthetic_counterfactual | cf_0005_last_three | span | last_three | harbordelta | lta | delta |
| synthetic_counterfactual | cf_0013_initials | words | initials | Sonia Rees Cobalt | SRC | SRE |
| synthetic_counterfactual | cf_0017_last_three | span | last_three | fablelambda | bda | amb |
| synthetic_counterfactual | cf_0017_last_three | span | last_three | fablenectar | tar | ter |
| synthetic_counterfactual | cf_0022_last_three | span | last_three | cobaltlambda | bda | lambda |
| public_prose | Name.000008 | Name | public | Antonia A. Guachiac | A G | A A |
| public_prose | Name.000008 | Name | public | Nikolajs Serdar Pirc | N P | N S |
| public_prose | Name.000008 | Name | public | Jaap H. Kleefstra | J K | J H |
| public_prose | Name.000037 | Name | public | Dr. Christophe Beaulieu, Sr. | Sr. | Sr |
| public_prose | Name.000037 | Name | public | Marcela Kubatova | NULL | null |
| public_prose | Name.000037 | Name | public | Mr. Hadar Caspit | NULL | |
| public_prose | Number.000084 | Number | public | 199 | 190-209 | 190-199 |
| public_prose | DateTime.000030 | DateTime | public | 03302241 | Tuesday, March 30, 2241 | Saturday, March 30, 2241 |
| public_prose | DateTime.000030 | DateTime | public | 02-Aug-2160 | Saturday, August 2, 2160 | Monday, August 2, 2160 |
| public_prose | DateTime.000030 | DateTime | public | 23 May 1984 | Wednesday, May 23, 1984 | Monday, May 23, 1984 |
| public_prose | DateTime.000075 | DateTime | public | 10:24PM | 10:00PM-10:29PM | 10:30PM-10:59PM |
| public_prose | DateTime.000012 | DateTime | public | Unknown | None | Unknown |
| public_prose | Number.000047 | Number | public | 1284.42 | 1285.00 | 1280.00 |
| public_prose | Number.000047 | Number | public | 23224.98 | 23225.00 | 23200.00 |
| public_prose | Number.000047 | Number | public | 1024.21 | 1025.00 | 1020.00 |
| public_prose | DateTime.000097 | DateTime | public | 6.30.2220 | 6 | 2220 |
| public_prose | Number.000054 | Number | public | 12541253 | 12541253 | 000012541253 |
| public_prose | DateTime.000032 | DateTime | public | 1956-12-16 20:18 | 8:18 PM | December 16, 2056 at 8:18 PM |
| public_prose | Number.000093 | Number | public | 4759 | 04759 | 4759 |
| public_prose | Number.000093 | Number | public | 7204 | 07204 | 7204 |
| public_prose | DateTime.000116 | DateTime | public | 20-Dec-2033 18:36:29 | 5PM-7PM | 6PM-8PM |
| public_prose | Number.000069 | Number | public | 1202.3433 | 1200 | 1202 |
| public_prose | Number.000069 | Number | public | 23224.1 | 23225 | 23224 |
| public_prose | Number.000069 | Number | public | -23224.1 | -23225 | -23224 |
Artifacts
- Run directory:
/workspace/experiments/qwen_counterfactual_episodic_icl/runs/main_v2 - Adapter checkpoint:
/workspace/large_artifacts/qwen_counterfactual_episodic_icl/checkpoints/main_v2/adapter - Row predictions:
/workspace/experiments/qwen_counterfactual_episodic_icl/runs/main_v2/row_predictions.csv - Task metrics:
/workspace/experiments/qwen_counterfactual_episodic_icl/runs/main_v2/task_metrics.csv - Summary:
/workspace/experiments/qwen_counterfactual_episodic_icl/runs/main_v2/summary.csv - Training log:
/workspace/experiments/qwen_counterfactual_episodic_icl/runs/main_v2/training_log.csv - Large artifacts directory:
/workspace/large_artifacts/qwen_counterfactual_episodic_icl
Limitations
This run trains on synthetic counterfactual transformations, so public benchmark transfer is the decisive external signal. The public evaluation is capped for runtime. Exact-match scoring is intentionally strict and does not award partial credit for near-correct formats.
Experiment log 6
Show the running log (6 entries)
Setup
- Created fresh standalone experiment directory:
/workspace/experiments/qwen_counterfactual_episodic_icl. - Large artifacts directory:
/workspace/large_artifacts/qwen_counterfactual_episodic_icl. - Public benchmark mirror is stored under the large-artifact directory.
- Primary question: can counterfactual episodic posttraining improve sparse-example task induction rather than merely memorizing a family prior?
Run smoke_no_train
- Time UTC:
2026-06-28T04:16:45.068880+00:00 - Config:
{"batch_size": 1, "grad_accum": 8, "log_every": 10, "lora_alpha": 32, "lora_dropout": 0.05, "lora_r": 16, "lr": 0.0002, "max_grad_norm": 1.0, "max_length": 768, "max_new_tokens": 32, "model": "Qwen/Qwen3-4B", "no_train": true, "public_heldout_n": 3, "public_task_limit": 1, "query_n": 2, "run_name": "smoke_no_train", "seed": 20260628, "smoke": true, "support_n": 4, "synthetic_eval_pairs": 1, "train_episodes": 16, "train_rows_per_episode": 2, "train_steps": 1, "warmup_steps": 20, "weight_decay": 0.0} - Elapsed seconds:
36.3 - Synthetic adapter full-task exact:
0.0% - Public adapter full-task exact:
0.0% - Report:
/workspace/experiments/qwen_counterfactual_episodic_icl/reports/qwen_counterfactual_episodic_icl_report.md
Run smoke_train
- Time UTC:
2026-06-28T04:18:04.969847+00:00 - Config:
{"batch_size": 1, "grad_accum": 2, "log_every": 1, "lora_alpha": 32, "lora_dropout": 0.05, "lora_r": 16, "lr": 0.0002, "max_grad_norm": 1.0, "max_length": 768, "max_new_tokens": 32, "model": "Qwen/Qwen3-4B", "no_train": false, "public_heldout_n": 3, "public_task_limit": 1, "query_n": 2, "run_name": "smoke_train", "seed": 20260628, "smoke": true, "support_n": 4, "synthetic_eval_pairs": 1, "train_episodes": 16, "train_rows_per_episode": 2, "train_steps": 2, "warmup_steps": 20, "weight_decay": 0.0} - Elapsed seconds:
60.1 - Synthetic adapter full-task exact:
0.0% - Public adapter full-task exact:
0.0% - Report:
/workspace/experiments/qwen_counterfactual_episodic_icl/reports/qwen_counterfactual_episodic_icl_report.md
Run pilot_v1
- Time UTC:
2026-06-28T04:20:11.971541+00:00 - Config:
{"batch_size": 1, "grad_accum": 4, "log_every": 5, "lora_alpha": 32, "lora_dropout": 0.05, "lora_r": 16, "lr": 0.0002, "max_grad_norm": 1.0, "max_length": 768, "max_new_tokens": 24, "model": "Qwen/Qwen3-4B", "no_train": false, "public_heldout_n": 3, "public_task_limit": 8, "query_n": 2, "run_name": "pilot_v1", "seed": 20260628, "smoke": false, "support_n": 4, "synthetic_eval_pairs": 8, "train_episodes": 240, "train_rows_per_episode": 2, "train_steps": 50, "warmup_steps": 20, "weight_decay": 0.0} - Elapsed seconds:
174.7 - Synthetic adapter full-task exact:
93.8% - Public adapter full-task exact:
37.5% - Report:
/workspace/experiments/qwen_counterfactual_episodic_icl/reports/qwen_counterfactual_episodic_icl_report.md
Run main_v1
- Time UTC:
2026-06-28T04:23:25.047618+00:00 - Config:
{"batch_size": 1, "grad_accum": 4, "log_every": 10, "lora_alpha": 32, "lora_dropout": 0.05, "lora_r": 16, "lr": 0.0002, "max_grad_norm": 1.0, "max_length": 768, "max_new_tokens": 24, "model": "Qwen/Qwen3-4B", "no_train": false, "public_heldout_n": 3, "public_task_limit": 30, "query_n": 2, "run_name": "main_v1", "seed": 20260628, "smoke": false, "support_n": 4, "synthetic_eval_pairs": 30, "train_episodes": 800, "train_rows_per_episode": 2, "train_steps": 120, "warmup_steps": 20, "weight_decay": 0.0} main_v1was interrupted before model load after a boundedness bug was found in counterfactual rule-pair construction. No result artifacts were produced for that run. The generator was patched to discard infeasible rule pairs after bounded search.
Run main_v2
- Time UTC:
2026-06-28T04:27:24.417084+00:00 - Config:
{"batch_size": 1, "grad_accum": 4, "log_every": 10, "lora_alpha": 32, "lora_dropout": 0.05, "lora_r": 16, "lr": 0.0002, "max_grad_norm": 1.0, "max_length": 768, "max_new_tokens": 24, "model": "Qwen/Qwen3-4B", "no_train": false, "public_heldout_n": 3, "public_task_limit": 30, "query_n": 2, "run_name": "main_v2", "seed": 20260628, "smoke": false, "support_n": 4, "synthetic_eval_pairs": 30, "train_episodes": 800, "train_rows_per_episode": 2, "train_steps": 120, "warmup_steps": 20, "weight_decay": 0.0} - Elapsed seconds:
588.9 - Synthetic adapter full-task exact:
91.7% - Public adapter full-task exact:
56.7% - Report:
/workspace/experiments/qwen_counterfactual_episodic_icl/reports/qwen_counterfactual_episodic_icl_report.md
Figures 4
Data files 13
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
analysis/summary.csv809 Bruns/main_v2/eval_episodes.csv36 kBruns/main_v2/summary.csv809 Bruns/main_v2/task_metrics.csv33 kBruns/pilot_v1/eval_episodes.csv9.4 kBruns/pilot_v1/summary.csv610 Bruns/pilot_v1/task_metrics.csv8.8 kBruns/smoke_no_train/eval_episodes.csv1.2 kBruns/smoke_no_train/summary.csv515 Bruns/smoke_no_train/task_metrics.csv1.2 kBruns/smoke_train/eval_episodes.csv1.2 kBruns/smoke_train/summary.csv515 Bruns/smoke_train/task_metrics.csv1.2 kB
Reproduce
This entry is source code or analysis only — there is no separate run to reproduce.