Qwen Episodic Soft-Prompt Task Vectors
The one idea you need
A soft prompt is a handful of invisible primer tokens slipped in front of every request to a frozen model. Here each reformatting task gets its own tiny set, shaped by that task's four worked examples to steer output, while the model itself never changes.
The question
If you tune a small set of hidden primer tokens for each reformatting task, does a frozen small model nail more of those tasks perfectly than plain instructions?
What we found
No. Tuning the primer tokens on each task's own examples solved exactly the same share of tasks perfectly as plain instructions: just over half either way. The tuning genuinely worked, beating a version trained on deliberately scrambled examples by more than 7 points, yet that edge was too small to finish even one extra task, and per-item accuracy inched up by only one point.
Why it matters
When adapting a frozen small model from just a few examples per task, a tuned hidden prefix is not worth the training cost here: plain instructions already capture what those few examples can teach. Test that gap before investing.
On this page
Results at a glance 6
How to read
Bars group six methods, from plain instructions to the tuned hidden prefix. Height is accuracy on examples never trained on; taller is better. Each method shows two bars: the whole task solved perfectly, and individual answers correct.
Takeaway → The plain-instructions bar and the tuned-prefix bar reach identical height for whole tasks, so the tuning added nothing over asking the model directly.
Data table
| method | full-task exact | row exact |
|---|---|---|
| direct (row-by-row) | 52.5% | 70.4% |
| learned soft prompt (row) | 52.5% | 71.5% |
| untrained soft prompt (row) | 50% | 71.7% |
| shuffled-label soft prompt (row) | 45% | 63.5% |
| learned soft prompt (batch) | 42.5% | 63.7% |
| direct (batch) | 40% | 65% |
Numbers from analysis/summary.csv
Technical framing
Held-out accuracy by method: learned soft prompt vs direct inference and controls — The learned per-task soft prompt ties direct row-by-row inference on full-task exact (52.5%) and beats the shuffled-label control, but never beats direct.
How to read
Bars show each method's change in how many tasks it solved perfectly, measured against plain instructions at the zero line. Above zero means improvement; every bar here rests on or dips below the line.
Takeaway → The tuned prefix's bar sits exactly on zero and every other method falls below it, so nothing beat plain instructions on fully solved tasks.
Data table
| method | full-task exact delta |
|---|---|
| learned soft prompt (row) | 0% |
| untrained soft prompt (row) | -2.5% |
| shuffled-label soft prompt (row) | -7.5% |
| learned soft prompt (batch) | -10% |
| direct (batch) | -12.5% |
Numbers from analysis/method_deltas.csv
Technical framing
Change vs direct row-by-row baseline (full-task exact) — No condition beats the direct baseline on fully solved tasks; the learned prefix is neutral (1 task helped, 1 hurt, 38 tied).
How to read
Two lines track how far off the prefix's fit is as tuning proceeds step by step; lower means a better fit. One line trains on the task's real examples, the other on deliberately scrambled ones.
Takeaway → The real-example line drops steadily and far lower, showing the prefix genuinely learned, so the flat end result is not a training failure.
Data table
| optimization step | learned (real labels) | shuffled labels |
|---|---|---|
| 0 | 1.06 | 3.92 |
| 1 | 0.747 | 3.43 |
| 2 | 0.588 | 3.04 |
| 3 | 0.487 | 2.63 |
| 4 | 0.372 | 2.21 |
| 5 | 0.289 | 1.87 |
| 6 | 0.248 | 1.58 |
Numbers from report table 'Train Loss Log' (per-step means of analysis/train_log.csv)
Technical framing
Soft-prompt training loss: real labels vs shuffled labels — The prefix optimizes cleanly on real labels (mean loss 1.06 to 0.25 in 6 steps), so the null result is not a failure to train.
How to read
For each of five methods, three bars count how many of the forty tasks it improved, worsened, or left unchanged compared with plain instructions. More improved and fewer worsened is better.
Takeaway → Nearly every task lands in the unchanged pile; the tuned prefix helped one task and hurt one, a wash overall.
Data table
| Method | Tasks helped | Tasks hurt | Tasks tied |
|---|---|---|---|
| Learned soft prompt (row) | 1 | 1 | 38 |
| Untrained init prefix (row) | 0 | 1 | 39 |
| Shuffled-label prefix (row) | 1 | 4 | 35 |
| Learned soft prompt (batch) | 0 | 4 | 36 |
| Direct (batch) | 0 | 5 | 35 |
Numbers from experiments/qwen_episodic_soft_prompt_task_vectors/analysis/method_deltas.csv
Technical framing
Tasks helped vs hurt relative to direct row-by-row (40-task run) — Almost every task ties the direct baseline; the learned prefix helps 1 task and hurts 1, while controls only hurt.
How to read
Bars show each method's change in how often individual answers matched exactly, against plain instructions at the zero line; above zero is better. Both the tuned prefix and an untuned prefix rise slightly.
Takeaway → An untuned prefix rises just as much as the tuned one, so the tiny gain comes from having any prefix, not from tuning it.
Data table
| Method | Change in row exact vs direct row |
|---|---|
| Learned soft prompt (row) | 1% |
| Untrained init prefix (row) | 1.2% |
| Shuffled-label prefix (row) | -6.9% |
| Learned soft prompt (batch) | -6.7% |
| Direct (batch) | -5.4% |
Numbers from experiments/qwen_episodic_soft_prompt_task_vectors/analysis/method_deltas.csv
Technical framing
Row-level accuracy change vs direct row-by-row baseline (40-task run) — At row level the untrained init prefix gains as much as the learned one, so training adds nothing beyond having any prefix.
How to read
Bars show how often single answers matched exactly for each of six methods, measured on a smaller and a larger set of test tasks; taller is better. The two sizes check whether the ranking holds.
Takeaway → Method ordering barely shifts between the two sizes, and the tuned prefix's edge over plain instructions stays tiny either way.
Data table
| Method | 20 tasks | 40 tasks |
|---|---|---|
| Direct (row) | 67.5% | 70.4% |
| Learned soft prompt (row) | 70.4% | 71.5% |
| Untrained init prefix (row) | 68.8% | 71.7% |
| Shuffled-label prefix (row) | 62.5% | 63.5% |
| Learned soft prompt (batch) | 67.1% | 63.7% |
| Direct (batch) | 61.3% | 65% |
Technical framing
Held-out row accuracy at 20-task vs 40-task evaluation budgets — Method ordering is roughly stable across budgets: row-by-row decoding beats batch, and the learned prefix edge over direct stays tiny.
In the author’s words from the Report · “Interpretation”
The learned soft-prompt row method changes strict full-task exact by 0.0 points relative to direct row-by-row inference. It changes full-task exact by 2.5 points relative to the untrained initialized prefix and by 7.5 points relative to a prefix optimized on shuffled training outputs. The learned prefix changes row exact by 1.0 points relative to direct row-by-row inference. It helps 1 tasks and hurts 1 tasks on strict full-task exact. A positive result requires learned_soft_row to beat direct inference and both soft-prefix controls. This run is neutral for strict full-task exact and weakly positive only at row level. … Read the full result →
Overview
Standalone experiment testing whether a short continuous prefix optimized per task can improve Qwen's held-out text-transformation consistency.
Main Run
- Run directory:
runs/main_qwen_soft_prompt_40_s6_lr001 - Markdown report:
reports/qwen_episodic_soft_prompt_task_vectors_report.md - HTML report:
reports/qwen_episodic_soft_prompt_task_vectors_report.html - Persistent log:
experiment_log.md
Key CSVs
analysis/summary.csv: method-level metrics.analysis/method_deltas.csv: wins/losses versus direct row-by-row inference.analysis/task_details.csv: task-level outputs and exactness.analysis/row_details.csv: row-level outputs and exactness.analysis/train_log.csv: leave-one-out soft-prompt training loss.
Key Charts
analysis/figures/method_full_task_exact.pnganalysis/figures/row_vs_full_task.pnganalysis/figures/wins_losses_vs_direct.pnganalysis/figures/family_heatmap.pnganalysis/figures/train_loss_curves.png
Data Placement
Large benchmark data is kept outside the experiment directory under:
/workspace/large_artifacts/qwen_episodic_soft_prompt_task_vectors
Report
Rendered from reports/qwen_episodic_soft_prompt_task_vectors_report.md
Question
Can a frozen language model become more consistent on text-transformation tasks by learning a small continuous task vector from that task's examples?
Each task receives its own optimized soft prefix. The model weights are frozen. The prefix is trained on leave-one-out versions of the task's training rows, then evaluated on held-out rows.
Setup
- Dataset root:
/workspace/large_artifacts/qwen_episodic_soft_prompt_task_vectors/prose-benchmarks - Run:
main_qwen_soft_prompt_40_s6_lr001 - Model:
Qwen/Qwen3-4B - Tasks: 40
- Soft tokens: 8
- Optimization steps: 6
- Learning rate: 0.01
- Train examples per task: 4
- Held-out cap per task: 4
Main Result
| method | tasks | row_exact | full_task_exact | parse_ok | avg_outputs |
|---|---|---|---|---|---|
| direct_row | 40 | 70.4% | 52.5% | 100.0% | 3.750 |
| learned_soft_row | 40 | 71.5% | 52.5% | 100.0% | 3.750 |
| init_soft_row | 40 | 71.7% | 50.0% | 100.0% | 3.750 |
| shuffled_soft_row | 40 | 63.5% | 45.0% | 100.0% | 3.750 |
| learned_soft_batch | 40 | 63.7% | 42.5% | 95.0% | 3.750 |
| direct_batch | 40 | 65.0% | 40.0% | 95.0% | 3.750 |
Interpretation
The learned soft-prompt row method changes strict full-task exact by 0.0 points relative to direct row-by-row inference. It changes full-task exact by 2.5 points relative to the untrained initialized prefix and by 7.5 points relative to a prefix optimized on shuffled training outputs.
The learned prefix changes row exact by 1.0 points relative to direct row-by-row inference. It helps 1 tasks and hurts 1 tasks on strict full-task exact. A positive result requires learned_soft_row to beat direct inference and both soft-prefix controls.
This run is neutral for strict full-task exact and weakly positive only at row level. The learned prefix beats the shuffled-label control, showing that the optimization target matters, but it does not improve the number of fully solved tasks over direct row-by-row inference.
Charts
Deltas Versus Direct Row Baseline
| method | tasks | full_task_delta | row_exact_delta | tasks_helped | tasks_hurt | tasks_tied |
|---|---|---|---|---|---|---|
| learned_soft_row | 40 | 0.0% | 1.0% | 1 | 1 | 38 |
| init_soft_row | 40 | -2.5% | 1.2% | 0 | 1 | 39 |
| shuffled_soft_row | 40 | -7.5% | -6.9% | 1 | 4 | 35 |
| learned_soft_batch | 40 | -10.0% | -6.7% | 0 | 4 | 36 |
| direct_batch | 40 | -12.5% | -5.4% | 0 | 5 | 35 |
Task Details
| task_id | family | method | row_exact | full_task_exact | parse_ok | parse_status | output_count |
|---|---|---|---|---|---|---|---|
| Address.000002 | Address | direct_batch | 33.3% | False | True | ok | 3 |
| Address.000013 | Address | direct_batch | 50.0% | False | True | ok | 4 |
| BillingCode.000007 | BillingCode | direct_batch | 0.0% | False | True | ok | 3 |
| City.000010 | City | direct_batch | 100.0% | True | True | ok | 3 |
| City.000011 | City | direct_batch | 75.0% | False | True | ok | 4 |
| Column.000001 | Column | direct_batch | 100.0% | True | True | ok | 4 |
| DateTime.000004 | DateTime | direct_batch | 100.0% | True | True | ok | 4 |
| DateTime.000007 | DateTime | direct_batch | 100.0% | True | True | ok | 4 |
| DateTime.000017 | DateTime | direct_batch | 75.0% | False | True | ok | 4 |
| DateTime.000025 | DateTime | direct_batch | 75.0% | False | True | ok | 4 |
| DateTime.000027 | DateTime | direct_batch | 50.0% | False | True | ok | 4 |
| DateTime.000034 | DateTime | direct_batch | 100.0% | True | True | ok | 4 |
| DateTime.000051 | DateTime | direct_batch | 0.0% | False | True | ok | 3 |
| DateTime.000076 | DateTime | direct_batch | 75.0% | False | True | ok | 4 |
| DateTime.000081 | DateTime | direct_batch | 0.0% | False | True | ok | 4 |
| DateTime.000094 | DateTime | direct_batch | 100.0% | True | True | ok | 4 |
| DateTime.000104 | DateTime | direct_batch | 100.0% | True | True | ok | 4 |
| DateTime.000108 | DateTime | direct_batch | 100.0% | True | True | ok | 4 |
| DateTime.000111 | DateTime | direct_batch | 75.0% | False | True | ok | 4 |
| DateTime.000114 | DateTime | direct_batch | 75.0% | False | True | ok | 4 |
| DateTime.000115 | DateTime | direct_batch | 0.0% | False | True | ok | 4 |
| DateTime.000116 | DateTime | direct_batch | 50.0% | False | True | ok | 4 |
| FilePath.000001 | FilePath | direct_batch | 100.0% | True | True | ok | 4 |
| Gender.000001 | Gender | direct_batch | 66.7% | False | True | ok | 3 |
| Language.000002 | Language | direct_batch | 100.0% | True | True | ok | 4 |
| Name.000028 | Name | direct_batch | 100.0% | True | True | ok | 4 |
| Number.000008 | Number | direct_batch | 25.0% | False | True | ok | 4 |
| Number.000015 | Number | direct_batch | 75.0% | False | True | ok | 4 |
| Number.000016 | Number | direct_batch | 0.0% | False | False | parse_fail | 4 |
| Number.000022 | Number | direct_batch | 50.0% | False | True | ok | 4 |
| Number.000028 | Number | direct_batch | 100.0% | True | True | ok | 3 |
| Number.000029 | Number | direct_batch | 0.0% | False | True | ok | 3 |
| Number.000043 | Number | direct_batch | 100.0% | True | True | ok | 4 |
| Number.000049 | Number | direct_batch | 0.0% | False | False | parse_fail | 4 |
| Number.000075 | Number | direct_batch | 50.0% | False | True | ok | 4 |
| Number.000077 | Number | direct_batch | 66.7% | False | True | ok | 3 |
| Phone.000008 | Phone | direct_batch | 100.0% | True | True | ok | 4 |
| Phone.000011 | Phone | direct_batch | 100.0% | True | True | ok | 3 |
| ShippingCode.000008 | ShippingCode | direct_batch | 33.3% | False | True | ok | 3 |
| UserAgent.000003 | UserAgent | direct_batch | 100.0% | True | True | ok | 4 |
| Address.000002 | Address | direct_row | 33.3% | False | True | row_clean | 3 |
| Address.000013 | Address | direct_row | 50.0% | False | True | row_clean | 4 |
| BillingCode.000007 | BillingCode | direct_row | 33.3% | False | True | row_clean | 3 |
| City.000010 | City | direct_row | 100.0% | True | True | row_clean | 3 |
| City.000011 | City | direct_row | 100.0% | True | True | row_clean | 4 |
| Column.000001 | Column | direct_row | 100.0% | True | True | row_clean | 4 |
| DateTime.000004 | DateTime | direct_row | 100.0% | True | True | row_clean | 4 |
| DateTime.000007 | DateTime | direct_row | 100.0% | True | True | row_clean | 4 |
| DateTime.000017 | DateTime | direct_row | 100.0% | True | True | row_clean | 4 |
| DateTime.000025 | DateTime | direct_row | 100.0% | True | True | row_clean | 4 |
| DateTime.000027 | DateTime | direct_row | 25.0% | False | True | row_clean | 4 |
| DateTime.000034 | DateTime | direct_row | 100.0% | True | True | row_clean | 4 |
| DateTime.000051 | DateTime | direct_row | 33.3% | False | True | row_clean | 3 |
| DateTime.000076 | DateTime | direct_row | 50.0% | False | True | row_clean | 4 |
| DateTime.000081 | DateTime | direct_row | 50.0% | False | True | row_clean | 4 |
| DateTime.000094 | DateTime | direct_row | 100.0% | True | True | row_clean | 4 |
| DateTime.000104 | DateTime | direct_row | 100.0% | True | True | row_clean | 4 |
| DateTime.000108 | DateTime | direct_row | 100.0% | True | True | row_clean | 4 |
| DateTime.000111 | DateTime | direct_row | 100.0% | True | True | row_clean | 4 |
| DateTime.000114 | DateTime | direct_row | 0.0% | False | True | row_clean | 4 |
| DateTime.000115 | DateTime | direct_row | 0.0% | False | True | row_clean | 4 |
| DateTime.000116 | DateTime | direct_row | 50.0% | False | True | row_clean | 4 |
| FilePath.000001 | FilePath | direct_row | 100.0% | True | True | row_clean | 4 |
| Gender.000001 | Gender | direct_row | 66.7% | False | True | row_clean | 3 |
| Language.000002 | Language | direct_row | 100.0% | True | True | row_clean | 4 |
| Name.000028 | Name | direct_row | 100.0% | True | True | row_clean | 4 |
| Number.000008 | Number | direct_row | 50.0% | False | True | row_clean | 4 |
| Number.000015 | Number | direct_row | 25.0% | False | True | row_clean | 4 |
| Number.000016 | Number | direct_row | 50.0% | False | True | row_clean | 4 |
| Number.000022 | Number | direct_row | 100.0% | True | True | row_clean | 4 |
| Number.000028 | Number | direct_row | 100.0% | True | True | row_clean | 3 |
| Number.000029 | Number | direct_row | 33.3% | False | True | row_clean | 3 |
| Number.000043 | Number | direct_row | 100.0% | True | True | row_clean | 4 |
| Number.000049 | Number | direct_row | 50.0% | False | True | row_clean | 4 |
| Number.000075 | Number | direct_row | 50.0% | False | True | row_clean | 4 |
| Number.000077 | Number | direct_row | 33.3% | False | True | row_clean | 3 |
| Phone.000008 | Phone | direct_row | 100.0% | True | True | row_clean | 4 |
| Phone.000011 | Phone | direct_row | 100.0% | True | True | row_clean | 3 |
| ShippingCode.000008 | ShippingCode | direct_row | 33.3% | False | True | row_clean | 3 |
| UserAgent.000003 | UserAgent | direct_row | 100.0% | True | True | row_clean | 4 |
| Address.000002 | Address | init_soft_row | 33.3% | False | True | row_clean | 3 |
| Address.000013 | Address | init_soft_row | 50.0% | False | True | row_clean | 4 |
| BillingCode.000007 | BillingCode | init_soft_row | 33.3% | False | True | row_clean | 3 |
| City.000010 | City | init_soft_row | 100.0% | True | True | row_clean | 3 |
| City.000011 | City | init_soft_row | 75.0% | False | True | row_clean | 4 |
| Column.000001 | Column | init_soft_row | 100.0% | True | True | row_clean | 4 |
| DateTime.000004 | DateTime | init_soft_row | 100.0% | True | True | row_clean | 4 |
| DateTime.000007 | DateTime | init_soft_row | 100.0% | True | True | row_clean | 4 |
| DateTime.000017 | DateTime | init_soft_row | 100.0% | True | True | row_clean | 4 |
| DateTime.000025 | DateTime | init_soft_row | 100.0% | True | True | row_clean | 4 |
| DateTime.000027 | DateTime | init_soft_row | 50.0% | False | True | row_clean | 4 |
| DateTime.000034 | DateTime | init_soft_row | 100.0% | True | True | row_clean | 4 |
| DateTime.000051 | DateTime | init_soft_row | 33.3% | False | True | row_clean | 3 |
| DateTime.000076 | DateTime | init_soft_row | 75.0% | False | True | row_clean | 4 |
| DateTime.000081 | DateTime | init_soft_row | 50.0% | False | True | row_clean | 4 |
| DateTime.000094 | DateTime | init_soft_row | 100.0% | True | True | row_clean | 4 |
| DateTime.000104 | DateTime | init_soft_row | 100.0% | True | True | row_clean | 4 |
| DateTime.000108 | DateTime | init_soft_row | 100.0% | True | True | row_clean | 4 |
| DateTime.000111 | DateTime | init_soft_row | 100.0% | True | True | row_clean | 4 |
| DateTime.000114 | DateTime | init_soft_row | 25.0% | False | True | row_clean | 4 |
| DateTime.000115 | DateTime | init_soft_row | 0.0% | False | True | row_clean | 4 |
| DateTime.000116 | DateTime | init_soft_row | 50.0% | False | True | row_clean | 4 |
| FilePath.000001 | FilePath | init_soft_row | 100.0% | True | True | row_clean | 4 |
| Gender.000001 | Gender | init_soft_row | 66.7% | False | True | row_clean | 3 |
| Language.000002 | Language | init_soft_row | 100.0% | True | True | row_clean | 4 |
| Name.000028 | Name | init_soft_row | 100.0% | True | True | row_clean | 4 |
| Number.000008 | Number | init_soft_row | 50.0% | False | True | row_clean | 4 |
| Number.000015 | Number | init_soft_row | 25.0% | False | True | row_clean | 4 |
| Number.000016 | Number | init_soft_row | 50.0% | False | True | row_clean | 4 |
| Number.000022 | Number | init_soft_row | 100.0% | True | True | row_clean | 4 |
| Number.000028 | Number | init_soft_row | 100.0% | True | True | row_clean | 3 |
| Number.000029 | Number | init_soft_row | 33.3% | False | True | row_clean | 3 |
| Number.000043 | Number | init_soft_row | 100.0% | True | True | row_clean | 4 |
| Number.000049 | Number | init_soft_row | 50.0% | False | True | row_clean | 4 |
| Number.000075 | Number | init_soft_row | 50.0% | False | True | row_clean | 4 |
| Number.000077 | Number | init_soft_row | 33.3% | False | True | row_clean | 3 |
| Phone.000008 | Phone | init_soft_row | 100.0% | True | True | row_clean | 4 |
| Phone.000011 | Phone | init_soft_row | 100.0% | True | True | row_clean | 3 |
| ShippingCode.000008 | ShippingCode | init_soft_row | 33.3% | False | True | row_clean | 3 |
| UserAgent.000003 | UserAgent | init_soft_row | 100.0% | True | True | row_clean | 4 |
| Address.000002 | Address | learned_soft_batch | 33.3% | False | True | ok | 3 |
| Address.000013 | Address | learned_soft_batch | 50.0% | False | True | ok | 4 |
| BillingCode.000007 | BillingCode | learned_soft_batch | 0.0% | False | True | ok | 3 |
| City.000010 | City | learned_soft_batch | 100.0% | True | True | ok | 3 |
| City.000011 | City | learned_soft_batch | 50.0% | False | True | ok | 4 |
| Column.000001 | Column | learned_soft_batch | 100.0% | True | True | ok | 4 |
| DateTime.000004 | DateTime | learned_soft_batch | 100.0% | True | True | ok | 4 |
| DateTime.000007 | DateTime | learned_soft_batch | 100.0% | True | True | ok | 4 |
| DateTime.000017 | DateTime | learned_soft_batch | 100.0% | True | True | ok | 4 |
| DateTime.000025 | DateTime | learned_soft_batch | 100.0% | True | True | ok | 4 |
| DateTime.000027 | DateTime | learned_soft_batch | 25.0% | False | True | ok | 4 |
| DateTime.000034 | DateTime | learned_soft_batch | 100.0% | True | True | ok | 4 |
| DateTime.000051 | DateTime | learned_soft_batch | 0.0% | False | True | ok | 3 |
| DateTime.000076 | DateTime | learned_soft_batch | 75.0% | False | True | ok | 4 |
| DateTime.000081 | DateTime | learned_soft_batch | 0.0% | False | True | ok | 4 |
| DateTime.000094 | DateTime | learned_soft_batch | 100.0% | True | True | ok | 4 |
| DateTime.000104 | DateTime | learned_soft_batch | 100.0% | True | True | ok | 4 |
| DateTime.000108 | DateTime | learned_soft_batch | 100.0% | True | True | ok | 4 |
| DateTime.000111 | DateTime | learned_soft_batch | 75.0% | False | True | ok | 4 |
| DateTime.000114 | DateTime | learned_soft_batch | 0.0% | False | False | parse_fail | 4 |
| DateTime.000115 | DateTime | learned_soft_batch | 0.0% | False | True | ok | 4 |
| DateTime.000116 | DateTime | learned_soft_batch | 50.0% | False | True | ok | 4 |
| FilePath.000001 | FilePath | learned_soft_batch | 100.0% | True | True | ok | 4 |
| Gender.000001 | Gender | learned_soft_batch | 66.7% | False | True | ok | 3 |
| Language.000002 | Language | learned_soft_batch | 100.0% | True | True | ok | 4 |
| Name.000028 | Name | learned_soft_batch | 100.0% | True | True | ok | 4 |
| Number.000008 | Number | learned_soft_batch | 50.0% | False | True | ok | 4 |
| Number.000015 | Number | learned_soft_batch | 75.0% | False | True | ok | 4 |
| Number.000016 | Number | learned_soft_batch | 50.0% | False | True | ok | 4 |
| Number.000022 | Number | learned_soft_batch | 50.0% | False | True | ok | 4 |
| Number.000028 | Number | learned_soft_batch | 0.0% | False | False | parse_fail | 3 |
| Number.000029 | Number | learned_soft_batch | 66.7% | False | True | ok | 3 |
| Number.000043 | Number | learned_soft_batch | 100.0% | True | True | ok | 4 |
| Number.000049 | Number | learned_soft_batch | 50.0% | False | True | ok | 4 |
| Number.000075 | Number | learned_soft_batch | 50.0% | False | True | ok | 4 |
| Number.000077 | Number | learned_soft_batch | 33.3% | False | True | ok | 3 |
| Phone.000008 | Phone | learned_soft_batch | 100.0% | True | True | ok | 4 |
| Phone.000011 | Phone | learned_soft_batch | 100.0% | True | True | ok | 3 |
| ShippingCode.000008 | ShippingCode | learned_soft_batch | 0.0% | False | True | ok | 3 |
| UserAgent.000003 | UserAgent | learned_soft_batch | 100.0% | True | True | ok | 4 |
| Address.000002 | Address | learned_soft_row | 33.3% | False | True | row_clean | 3 |
| Address.000013 | Address | learned_soft_row | 50.0% | False | True | row_clean | 4 |
| BillingCode.000007 | BillingCode | learned_soft_row | 33.3% | False | True | row_clean | 3 |
| City.000010 | City | learned_soft_row | 100.0% | True | True | row_clean | 3 |
| City.000011 | City | learned_soft_row | 50.0% | False | True | row_clean | 4 |
| Column.000001 | Column | learned_soft_row | 100.0% | True | True | row_clean | 4 |
| DateTime.000004 | DateTime | learned_soft_row | 100.0% | True | True | row_clean | 4 |
| DateTime.000007 | DateTime | learned_soft_row | 100.0% | True | True | row_clean | 4 |
| DateTime.000017 | DateTime | learned_soft_row | 100.0% | True | True | row_clean | 4 |
| DateTime.000025 | DateTime | learned_soft_row | 100.0% | True | True | row_clean | 4 |
| DateTime.000027 | DateTime | learned_soft_row | 75.0% | False | True | row_clean | 4 |
| DateTime.000034 | DateTime | learned_soft_row | 100.0% | True | True | row_clean | 4 |
| DateTime.000051 | DateTime | learned_soft_row | 33.3% | False | True | row_clean | 3 |
| DateTime.000076 | DateTime | learned_soft_row | 100.0% | True | True | row_clean | 4 |
| DateTime.000081 | DateTime | learned_soft_row | 50.0% | False | True | row_clean | 4 |
| DateTime.000094 | DateTime | learned_soft_row | 100.0% | True | True | row_clean | 4 |
| DateTime.000104 | DateTime | learned_soft_row | 100.0% | True | True | row_clean | 4 |
| DateTime.000108 | DateTime | learned_soft_row | 100.0% | True | True | row_clean | 4 |
| DateTime.000111 | DateTime | learned_soft_row | 100.0% | True | True | row_clean | 4 |
| DateTime.000114 | DateTime | learned_soft_row | 25.0% | False | True | row_clean | 4 |
| DateTime.000115 | DateTime | learned_soft_row | 0.0% | False | True | row_clean | 4 |
| DateTime.000116 | DateTime | learned_soft_row | 50.0% | False | True | row_clean | 4 |
| FilePath.000001 | FilePath | learned_soft_row | 100.0% | True | True | row_clean | 4 |
| Gender.000001 | Gender | learned_soft_row | 66.7% | False | True | row_clean | 3 |
| Language.000002 | Language | learned_soft_row | 100.0% | True | True | row_clean | 4 |
| Name.000028 | Name | learned_soft_row | 100.0% | True | True | row_clean | 4 |
| Number.000008 | Number | learned_soft_row | 50.0% | False | True | row_clean | 4 |
| Number.000015 | Number | learned_soft_row | 25.0% | False | True | row_clean | 4 |
| Number.000016 | Number | learned_soft_row | 75.0% | False | True | row_clean | 4 |
| Number.000022 | Number | learned_soft_row | 100.0% | True | True | row_clean | 4 |
| Number.000028 | Number | learned_soft_row | 100.0% | True | True | row_clean | 3 |
| Number.000029 | Number | learned_soft_row | 66.7% | False | True | row_clean | 3 |
| Number.000043 | Number | learned_soft_row | 100.0% | True | True | row_clean | 4 |
| Number.000049 | Number | learned_soft_row | 25.0% | False | True | row_clean | 4 |
| Number.000075 | Number | learned_soft_row | 50.0% | False | True | row_clean | 4 |
| Number.000077 | Number | learned_soft_row | 0.0% | False | True | row_clean | 3 |
| Phone.000008 | Phone | learned_soft_row | 100.0% | True | True | row_clean | 4 |
| Phone.000011 | Phone | learned_soft_row | 100.0% | True | True | row_clean | 3 |
| ShippingCode.000008 | ShippingCode | learned_soft_row | 0.0% | False | True | row_clean | 3 |
| UserAgent.000003 | UserAgent | learned_soft_row | 100.0% | True | True | row_clean | 4 |
| Address.000002 | Address | shuffled_soft_row | 33.3% | False | True | row_clean | 3 |
| Address.000013 | Address | shuffled_soft_row | 100.0% | True | True | row_clean | 4 |
| BillingCode.000007 | BillingCode | shuffled_soft_row | 33.3% | False | True | row_clean | 3 |
| City.000010 | City | shuffled_soft_row | 100.0% | True | True | row_clean | 3 |
| City.000011 | City | shuffled_soft_row | 50.0% | False | True | row_clean | 4 |
| Column.000001 | Column | shuffled_soft_row | 100.0% | True | True | row_clean | 4 |
| DateTime.000004 | DateTime | shuffled_soft_row | 100.0% | True | True | row_clean | 4 |
| DateTime.000007 | DateTime | shuffled_soft_row | 100.0% | True | True | row_clean | 4 |
| DateTime.000017 | DateTime | shuffled_soft_row | 75.0% | False | True | row_clean | 4 |
| DateTime.000025 | DateTime | shuffled_soft_row | 100.0% | True | True | row_clean | 4 |
| DateTime.000027 | DateTime | shuffled_soft_row | 25.0% | False | True | row_clean | 4 |
| DateTime.000034 | DateTime | shuffled_soft_row | 100.0% | True | True | row_clean | 4 |
| DateTime.000051 | DateTime | shuffled_soft_row | 33.3% | False | True | row_clean | 3 |
| DateTime.000076 | DateTime | shuffled_soft_row | 75.0% | False | True | row_clean | 4 |
| DateTime.000081 | DateTime | shuffled_soft_row | 25.0% | False | True | row_clean | 4 |
| DateTime.000094 | DateTime | shuffled_soft_row | 100.0% | True | True | row_clean | 4 |
| DateTime.000104 | DateTime | shuffled_soft_row | 100.0% | True | True | row_clean | 4 |
| DateTime.000108 | DateTime | shuffled_soft_row | 100.0% | True | True | row_clean | 4 |
| DateTime.000111 | DateTime | shuffled_soft_row | 100.0% | True | True | row_clean | 4 |
| DateTime.000114 | DateTime | shuffled_soft_row | 25.0% | False | True | row_clean | 4 |
Train Loss Log
| arm | step | loss |
|---|---|---|
| learned | 0 | 1.059 |
| learned | 1 | 0.747 |
| learned | 2 | 0.588 |
| learned | 3 | 0.487 |
| learned | 4 | 0.372 |
| learned | 5 | 0.289 |
| learned | 6 | 0.248 |
| shuffled | 0 | 3.920 |
| shuffled | 1 | 3.429 |
| shuffled | 2 | 3.038 |
| shuffled | 3 | 2.631 |
| shuffled | 4 | 2.213 |
| shuffled | 5 | 1.868 |
| shuffled | 6 | 1.580 |
Files
runs/main_qwen_soft_prompt_40_s6_lr001/task_details.csvruns/main_qwen_soft_prompt_40_s6_lr001/row_details.csvruns/main_qwen_soft_prompt_40_s6_lr001/train_log.csvruns/main_qwen_soft_prompt_40_s6_lr001/summary.csvruns/main_qwen_soft_prompt_40_s6_lr001/method_deltas.csvanalysis/summary.csvanalysis/method_deltas.csvanalysis/task_details.csvanalysis/row_details.csvanalysis/train_log.csv
Experiment log 11
Show the running log (11 entries, 2026-06-27)
Objective
Test whether a small continuous task vector learned from a task's training examples can improve a frozen model's held-out text-transformation consistency.
The core intervention is episodic soft-prompt optimization: for each task, freeze the model, optimize a short sequence of continuous prefix embeddings on the task's training examples, and then prepend that learned task vector during held-out inference.
The primary metric is strict full-task exact on held-out rows: a task counts as correct only if every held-out output is exactly correct.
The experiment is standalone. It stores run-local caches, learned-prefix diagnostics, analyses, charts, Markdown report, and HTML report under this directory. Large benchmark data is referenced through /workspace/large_artifacts/qwen_episodic_soft_prompt_task_vectors.
Initial Plan
- Create a fresh experiment directory and separate large-artifact root.
- Load public text-transformation tasks.
- Compare direct row-by-row and batched baselines against learned soft prompts.
Include controls:
- frozen random soft prompt
- shuffled-label learned soft prompt
- zero/initialized soft prompt baseline
- Track train loss, train exactness, held-out row exactness, held-out full-task exactness, parse success, and overfit signatures.
- Run no-Qwen smoke, small real-Qwen pilot, then a main run.
- Generate CSVs, charts, Markdown report, and HTML report.
Run Notes
2026-06-27 09:08 UTC - Scaffold
- Created fresh experiment directory:
/workspace/experiments/qwen_episodic_soft_prompt_task_vectors. - Created separate large-artifact root:
/workspace/large_artifacts/qwen_episodic_soft_prompt_task_vectors. - Added standalone runner:
src/qwen_episodic_soft_prompt_task_vectors.py. - Added persistent log and later added
README.md.
2026-06-27 09:09 UTC - No-Qwen Smoke
Command:
python -m py_compile /workspace/experiments/qwen_episodic_soft_prompt_task_vectors/src/qwen_episodic_soft_prompt_task_vectors.py
python /workspace/experiments/qwen_episodic_soft_prompt_task_vectors/src/qwen_episodic_soft_prompt_task_vectors.py \
--run_name smoke_no_qwen \
--task_limit 3 \
--heldout_cap 3 \
--steps 1 \
--no_qwenResult:
- Smoke completed on 3 tasks.
- CSVs, charts, Markdown report, and HTML report were generated.
- Metrics are not interpreted because
--no_qwenleaves generations blank.
2026-06-27 09:10 UTC - Tiny Real-Qwen Gradient/Generation Pilot
Command:
python /workspace/experiments/qwen_episodic_soft_prompt_task_vectors/src/qwen_episodic_soft_prompt_task_vectors.py \
--run_name pilot_qwen_2_step1 \
--task_limit 2 \
--heldout_cap 3 \
--soft_tokens 4 \
--steps 1 \
--lr 0.05Result:
- Real-model training and soft-prefix generation completed successfully.
- This validated that gradients through the frozen 4-bit model to input soft prompts work in this environment.
- One-step optimization was too aggressive on a numeric task: learned and shuffled soft prompts both degraded that row output.
2026-06-27 09:12 UTC - Lower-LR Pilot
Command:
python /workspace/experiments/qwen_episodic_soft_prompt_task_vectors/src/qwen_episodic_soft_prompt_task_vectors.py \
--run_name pilot_qwen_4_lr001_s6 \
--task_limit 4 \
--heldout_cap 3 \
--soft_tokens 8 \
--steps 6 \
--lr 0.01Result:
direct_row: 83.3% row exact, 75.0% full-task exact.learned_soft_row: 75.0% row exact, 75.0% full-task exact.shuffled_soft_row: 75.0% row exact, 75.0% full-task exact.- Training loss sometimes overshot by the final step, so final-step prefix selection was not safe.
2026-06-27 09:13 UTC - Best-Loss Prefix Selection
Changed the optimizer to return the best-loss prefix checkpoint instead of the final optimization step.
Reran the four-task pilot:
python /workspace/experiments/qwen_episodic_soft_prompt_task_vectors/src/qwen_episodic_soft_prompt_task_vectors.py \
--run_name pilot_qwen_4_best_lr001_s6 \
--task_limit 4 \
--heldout_cap 3 \
--soft_tokens 8 \
--steps 6 \
--lr 0.01Result:
- All row methods tied on full-task exact at 75.0%.
learned_soft_rowno longer damaged the numeric task relative to direct row-by-row.- Chose
soft_tokens=8,steps=6,lr=0.01, best-loss checkpoint selection for the main runs.
2026-06-27 09:15 UTC - Main Run, 20 Tasks
Command:
python /workspace/experiments/qwen_episodic_soft_prompt_task_vectors/src/qwen_episodic_soft_prompt_task_vectors.py \
--run_name main_qwen_soft_prompt_20_s6_lr001 \
--task_limit 20 \
--heldout_cap 4 \
--soft_tokens 8 \
--steps 6 \
--lr 0.01Result:
direct_row: 67.5% row exact, 50.0% full-task exact.learned_soft_row: 70.4% row exact, 50.0% full-task exact.init_soft_row: 68.8% row exact, 50.0% full-task exact.shuffled_soft_row: 62.5% row exact, 45.0% full-task exact.
Read:
- Learned soft prompts improved row exact slightly but did not solve more full tasks.
- The result was close enough to warrant a 40-task run.
2026-06-27 09:16 UTC - Main Run, 40 Tasks
Command:
python /workspace/experiments/qwen_episodic_soft_prompt_task_vectors/src/qwen_episodic_soft_prompt_task_vectors.py \
--run_name main_qwen_soft_prompt_40_s6_lr001 \
--task_limit 40 \
--heldout_cap 4 \
--soft_tokens 8 \
--steps 6 \
--lr 0.01Result:
- 40 tasks.
- Runtime: 481.45 seconds.
direct_row: 70.4% row exact, 52.5% full-task exact.learned_soft_row: 71.5% row exact, 52.5% full-task exact.init_soft_row: 71.7% row exact, 50.0% full-task exact.shuffled_soft_row: 63.5% row exact, 45.0% full-task exact.learned_soft_batch: 63.8% row exact, 42.5% full-task exact.direct_batch: 65.0% row exact, 40.0% full-task exact.
Deltas versus direct_row:
learned_soft_row: +0.0 full-task points, +1.0 row-exact points; helped 1 task, hurt 1 task, tied 38 tasks.init_soft_row: -2.5 full-task points, +1.2 row-exact points.shuffled_soft_row: -7.5 full-task points, -6.9 row-exact points.learned_soft_batch: -10.0 full-task points.direct_batch: -12.5 full-task points.
Task-level movement:
learned_soft_rowfixedDateTime.000076.learned_soft_rowbrokeCity.000011.- Net strict full-task exact was unchanged.
Final diagnosis:
- Neutral for strict full-task exact.
- Weakly positive at row level.
- The learned target matters: shuffled-label prefixes are worse.
- The untrained initialized prefix also changes row behavior, so not all row-level movement should be credited to task learning.
- Episodic soft prompts do not produce a deployable improvement under this configuration, but they are not inert: they can move individual rows and one full task in each direction.
2026-06-27 09:25 UTC - Report Hardening
- Added report-only regeneration path to avoid rerunning Qwen when editing the report.
- Regenerated the Markdown and HTML reports from the 40-task CSVs.
- Updated report interpretation to state the neutral full-task result and weak row-level gain explicitly.
Final artifacts:
README.mdexperiment_log.mdsrc/qwen_episodic_soft_prompt_task_vectors.pyruns/main_qwen_soft_prompt_40_s6_lr001/task_details.csvruns/main_qwen_soft_prompt_40_s6_lr001/row_details.csvruns/main_qwen_soft_prompt_40_s6_lr001/train_log.csvruns/main_qwen_soft_prompt_40_s6_lr001/summary.csvruns/main_qwen_soft_prompt_40_s6_lr001/method_deltas.csvanalysis/*.csvanalysis/figures/*.pngreports/qwen_episodic_soft_prompt_task_vectors_report.mdreports/qwen_episodic_soft_prompt_task_vectors_report.html
Figures 5
Data files 7
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
analysis/summary.csv345 Bruns/main_qwen_soft_prompt_20_s6_lr001/summary.csv307 Bruns/main_qwen_soft_prompt_40_s6_lr001/summary.csv345 Bruns/pilot_qwen_2_step1/summary.csv320 Bruns/pilot_qwen_4_best_lr001_s6/summary.csv356 Bruns/pilot_qwen_4_lr001_s6/summary.csv314 Bruns/smoke_no_qwen/summary.csv260 B
Reproduce
This entry is source code or analysis only — there is no separate run to reproduce.