Qwen Verified Skill Memory RAG

Finished2026-06-27imported · line YAlgorithmic Memory and Retrieval Benchmark Generalization Evidence-Conditioned Selection Interpretability and Diagnostics+1 moreGitHub ↗

The one idea you need

Before a small model reformats messy data — dates, phone numbers, addresses — into a target style, it is handed a verified, already-solved example of a similar reformatting job, like glancing at the answer key for a related puzzle before tackling your own.

The question

When a small model reformats messy data like dates and phone numbers, does handing it a verified, already-solved example of a similar job make more batches come out perfectly right?

What we found

No. Handing the model a matched, verified example made it slightly worse, not better: every value came out correct on 47.5% of jobs, versus 50% when it worked through each value alone with no example. The lookup itself worked — the fetched example was the same kind of task 77.5% of the time — yet across 40 jobs it improved none and hurt one. Relevant context was not the missing ingredient.

Why it matters

If you run a small model through repetitive data-cleanup, do not count on a library of similar solved examples to lift reliability. Here the gains came from clean one-value-at-a-time prompting, not from looking up past work — even accurate lookups added nothing.

Working alone vs given a matched example50% → 47.5%share of jobs where every value was exactly right

Fetched example was the right kind of task77.5%versus 20% for a random pick — the lookup worked

Jobs the matched example improved0 of 40and it made one job worse

Given a random, unrelated example instead42.5%jobs fully correct — worse still

On this page

Results at a glance
Overview
Report
Experiment log
Figures
Data files
Reproduce
Related

Results at a glance 8

How often each method got a whole batch perfectly right

How to read

Each bar is one working style; taller means more jobs where every value was exactly right. The leftmost bar is the model working through values one at a time alone; the others add a looked-up example or process the whole batch at once. Taller is better.

Takeaway → The plain one-at-a-time bar is tallest; every version that adds a looked-up example or batches the job sits lower, so the extra example never paid off.

Data table

method	full-task exact
direct row-by-row	50%
skill RAG	47.5%
corrupted skill RAG	45%
direct batch	42.5%
random skill RAG	42.5%

Numbers from analysis/summary.csv

Technical framing

Strict full-task exact accuracy by method — Retrieving a verified analogous skill card did not help: skill RAG lands below plain row-by-row inference and near its controls.

How often each method got a single value right

How to read

Each bar is one working style; taller means more individual values exactly correct — a looser test than needing the whole batch perfect. The leftmost bar is the model working one value at a time alone. Taller is better.

Takeaway → Working alone stays tallest even under this gentler scoring; every added-example or batched version lands lower, confirming the pattern is not an artifact of strict grading.

Data table

method	row exact
direct row-by-row	70.8%
skill RAG	65.2%
corrupted skill RAG	63.1%
direct batch	65.2%
random skill RAG	67.1%

Numbers from analysis/summary.csv

Technical framing

Row-level exact accuracy by method — The same ordering holds at the row level: every skill-card and batching variant loses rows versus direct row-by-row inference.

How many jobs each shortcut helped versus hurt

How to read

For each working style, two bars count the jobs it improved versus the jobs it worsened, against the model working one value at a time alone. Across 40 jobs, a taller help bar and a shorter hurt bar is better.

Takeaway → Help bars sit at or near zero everywhere while hurt bars climb; the matched example improved none of the 40 jobs and worsened one.

Data table

method	tasks helped	tasks hurt
skill RAG	0	1
corrupted skill RAG	0	2
direct batch	0	3
random skill RAG	1	4

Numbers from analysis/method_deltas.csv

Technical framing

Tasks helped vs hurt relative to direct row-by-row (40 tasks) — Skill RAG helped 0 of 40 tasks and hurt 1 versus the direct baseline; no variant produced net wins.

How often the lookup found a genuinely similar example

How to read

Two bars: the real lookup versus a random pick. Each shows how often the fetched example came from the same kind of task. Taller means the fetched example was more relevant to the job at hand.

Takeaway → The real lookup fetched a genuinely similar example most of the time, far above the random pick, so the flat result is not a broken lookup.

Data table

retrieval condition	same-family at rank 1
skill RAG retriever	77.5%
random skill control	20%

Numbers from analysis/retrieval_summary.csv

Technical framing

Retrieval quality: top-1 same-family hit rate — The retriever worked (77.5% same-family vs 20% random), so the null result is not a retrieval failure — the skill cards just didn't help.

Tiny six-job trials swung wildly from run to run

How to read

Each group is one working style; the three bars within it are three separate tiny trials of only six jobs each. Height is the share of values correct. Watch how much the matched-example bars jump between trials.

Takeaway → The matched-example bars swing from worst to best across the three tiny trials, showing six jobs are too few to trust and motivating the full 40-job test.

Data table

Method	Pilot top-3 skills	Pilot top-3 repeat	Pilot top-1 skill
direct_row	77.8%	77.8%	77.8%
direct_batch	83.3%	83.3%	83.3%
skill_rag	61.1%	83.3%	88.9%
random_skill_rag	77.8%	83.3%	83.3%
corrupt_skill_rag	55.6%	77.8%	83.3%

Numbers from runs/pilot_qwen_6/summary.csv, runs/pilot_qwen_6_v2/summary.csv, runs/pilot_qwen_6_top1/summary.csv

Technical framing

Pilot runs were unstable: row accuracy across three 6-task pilots — skill_rag swung from worst (61%) to best (89%) across 6-task pilots, motivating the 40-task main run where its edge vanished.

On the hardest task type, examples did not rescue accuracy

How to read

Ten of the hardest number-formatting jobs run along the bottom. For each, one bar is the model working alone and the other is the model given a matched example. Height is the share of values correct; taller is better.

Takeaway → On these number jobs the matched-example bars are mostly equal or shorter — shorter on five of the ten and taller on just two — so the failures are not about finding a relevant example.

Data table

Number-family task ID	Direct row-by-row	Retrieved skill card
000008	33.3%	16.7%
000015	33.3%	0%
000016	66.7%	33.3%
000022	100%	33.3%
000028	100%	100%
000029	33.3%	66.7%
000043	100%	100%
000049	50%	25%
000075	66.7%	66.7%
000077	33.3%	66.7%

Numbers from analysis/task_details.csv

Technical framing

Hardest family (Number): skill cards do not rescue per-task accuracy — On the weakest family, skill retrieval loses on 4 of 10 tasks and wins on 2; the numeric-rounding failures are not a retrieval problem.

Only processing the whole batch at once broke the format

How to read

Each bar is one working style; height is the share of jobs whose output could be read back cleanly. Taller is better. The bar worth watching is the one that processes the whole batch in a single pass.

Takeaway → Every style produced clean, readable output except processing the whole batch at once, which garbled the format on a small slice of jobs.

Data table

Method	Parse success rate
direct_row	100%
skill_rag	100%
corrupt_skill_rag	100%
random_skill_rag	100%
direct_batch	95%

Numbers from analysis/summary.csv

Technical framing

Output-format reliability: only batched decoding drops parses — All prompt-augmented methods parse cleanly; direct batched JSON output fails to parse on 5% of tasks, part of its full-task gap.

The single best-matched example is the most relevant

How to read

Two lines run across the first, second, and third best-matched examples. One tracks how similar the match is; the other tracks how often it is the same kind of task. Higher means more relevant.

Takeaway → Both lines slope downward from the top match, so lower-ranked examples are less relevant — the reason only the single best example was ever handed to the model.

Data table

Retrieval rank	Mean similarity score	Same-family hit rate
1	0.783	1
2	0.702	0.833
3	0.631	0.833

Numbers from runs/pilot_qwen_6/retrieval_summary.csv

Technical framing

Retrieval quality decays with rank in the top-3 pilot — Ranks 2-3 retrieve weaker, more off-family skills, which is why the main run kept only the top-1 skill card.

In the author’s words from the Report · “Interpretation”

The retrieved-skill method changes strict full-task exact by -2.5 points relative to row-by-row direct inference and by 5.0 points relative to direct batched inference. The random-skill control scores 42.5% full-task exact and the corrupted-skill control scores 45.0%. A retrieval-memory gain is only meaningful if skill_rag beats both controls and the direct baselines. This run is negative for the tested retrieval-memory mechanism. Top-1 retrieval found a same-family verified skill 77.5% of the time, so the retriever was not random, but skill_rag helped 0 tasks and hurt 1 task relative to row-by-row direct inference. … Read the full result →

Overview

Standalone experiment testing whether train-only verified transformation examples help Qwen solve public text-transformation tasks more consistently.

Main Run

Run directory: runs/main_qwen_skill_memory_40_top1
Markdown report: reports/qwen_verified_skill_memory_rag_report.md
HTML report: reports/qwen_verified_skill_memory_rag_report.html
Persistent log: experiment_log.md

Key CSVs

analysis/summary.csv: method-level metrics.
analysis/method_deltas.csv: wins/losses versus direct row-by-row inference.
analysis/task_details.csv: task-level outputs and exactness.
analysis/row_details.csv: row-level outputs and exactness.
analysis/retrieval_details.csv: retrieved skill cards per task.
analysis/retrieval_summary.csv: retrieval quality summary.

Key Charts

Data Placement

Large benchmark data is kept outside the experiment directory under:

/workspace/large_artifacts/qwen_verified_skill_memory_rag

Report

Rendered from reports/qwen_verified_skill_memory_rag_report.md

Question

Can a frozen language model solve text-transformation tasks more consistently when it retrieves analogous verified transformation skills from a train-only memory?

The experiment evaluates strict full-task exact: a task is correct only if every held-out row is exactly correct.

Setup

Dataset root: /workspace/large_artifacts/qwen_verified_skill_memory_rag/prose-benchmarks
Run: main_qwen_skill_memory_40_top1
Model: Qwen/Qwen3-4B
Evaluation tasks: 40
Memory tasks: 269
Retrieved skills per task: 1
Skill-card examples: 4
Train examples per target task: 4
Held-out cap per task: 6
Generation rows: 364

Main Result

method	tasks	row_exact	full_task_exact	parse_ok	avg_outputs
direct_row	40	70.8%	50.0%	100.0%	5.100
skill_rag	40	65.2%	47.5%	100.0%	5.100
corrupt_skill_rag	40	63.1%	45.0%	100.0%	5.100
direct_batch	40	65.2%	42.5%	95.0%	5.100
random_skill_rag	40	67.1%	42.5%	100.0%	5.100

Interpretation

The retrieved-skill method changes strict full-task exact by -2.5 points relative to row-by-row direct inference and by 5.0 points relative to direct batched inference.

The random-skill control scores 42.5% full-task exact and the corrupted-skill control scores 45.0%. A retrieval-memory gain is only meaningful if skill_rag beats both controls and the direct baselines.

This run is negative for the tested retrieval-memory mechanism. Top-1 retrieval found a same-family verified skill 77.5% of the time, so the retriever was not random, but skill_rag helped 0 tasks and hurt 1 task relative to row-by-row direct inference. The failure is therefore not retrieval failure alone; adding a verified analogous skill card did not reliably improve the model's target transformation.

Charts

Full-task exact by method

Row versus full-task accuracy

Family heatmap

Retrieval family agreement

Top retrieval scores

Wins and losses versus direct

Deltas Versus Direct Row Baseline

method	tasks	full_task_delta	row_exact_delta	tasks_helped	tasks_hurt	tasks_tied
skill_rag	40	-2.5%	-5.6%	0	1	39
corrupt_skill_rag	40	-5.0%	-7.7%	0	2	38
direct_batch	40	-7.5%	-5.6%	0	3	37
random_skill_rag	40	-7.5%	-3.7%	1	4	35

Retrieval Diagnostics

method	rank	mean_score	same_family
corrupt_skill_rag	1	0.714	77.5%
random_skill_rag	1	0.000	20.0%
skill_rag	1	0.714	77.5%

Task Details

task_id	family	method	row_exact	full_task_exact	parse_ok	parse_status	output_count
Address.000002	Address	corrupt_skill_rag	33.3%	False	True	ok	3
Address.000013	Address	corrupt_skill_rag	66.7%	False	True	ok	6
BillingCode.000007	BillingCode	corrupt_skill_rag	0.0%	False	True	ok	3
City.000010	City	corrupt_skill_rag	100.0%	True	True	ok	3
City.000011	City	corrupt_skill_rag	100.0%	True	True	ok	4
Column.000001	Column	corrupt_skill_rag	100.0%	True	True	ok	6
DateTime.000004	DateTime	corrupt_skill_rag	100.0%	True	True	ok	6
DateTime.000007	DateTime	corrupt_skill_rag	100.0%	True	True	ok	6
DateTime.000017	DateTime	corrupt_skill_rag	66.7%	False	True	ok	6
DateTime.000025	DateTime	corrupt_skill_rag	100.0%	True	True	ok	6
DateTime.000027	DateTime	corrupt_skill_rag	66.7%	False	True	ok	6
DateTime.000034	DateTime	corrupt_skill_rag	100.0%	True	True	ok	6
DateTime.000051	DateTime	corrupt_skill_rag	0.0%	False	True	ok	3
DateTime.000076	DateTime	corrupt_skill_rag	83.3%	False	True	ok	6
DateTime.000081	DateTime	corrupt_skill_rag	66.7%	False	True	ok	6
DateTime.000094	DateTime	corrupt_skill_rag	100.0%	True	True	ok	4
DateTime.000104	DateTime	corrupt_skill_rag	100.0%	True	True	ok	6
DateTime.000108	DateTime	corrupt_skill_rag	100.0%	True	True	ok	6
DateTime.000111	DateTime	corrupt_skill_rag	100.0%	True	True	ok	6
DateTime.000114	DateTime	corrupt_skill_rag	0.0%	False	True	ok	6
DateTime.000115	DateTime	corrupt_skill_rag	0.0%	False	True	ok	6
DateTime.000116	DateTime	corrupt_skill_rag	50.0%	False	True	ok	6
FilePath.000001	FilePath	corrupt_skill_rag	100.0%	True	True	ok	6
Gender.000001	Gender	corrupt_skill_rag	0.0%	False	True	ok	3
Language.000002	Language	corrupt_skill_rag	100.0%	True	True	ok	6
Name.000028	Name	corrupt_skill_rag	100.0%	True	True	ok	6
Number.000008	Number	corrupt_skill_rag	16.7%	False	True	ok	6
Number.000015	Number	corrupt_skill_rag	0.0%	False	True	ok	6
Number.000016	Number	corrupt_skill_rag	33.3%	False	True	ok	6
Number.000022	Number	corrupt_skill_rag	33.3%	False	True	ok	6
Number.000028	Number	corrupt_skill_rag	100.0%	True	True	ok	3
Number.000029	Number	corrupt_skill_rag	66.7%	False	True	ok	3
Number.000043	Number	corrupt_skill_rag	100.0%	True	True	ok	6
Number.000049	Number	corrupt_skill_rag	25.0%	False	True	ok	4
Number.000075	Number	corrupt_skill_rag	50.0%	False	True	ok	6
Number.000077	Number	corrupt_skill_rag	0.0%	False	True	ok	3
Phone.000008	Phone	corrupt_skill_rag	100.0%	True	True	ok	6
Phone.000011	Phone	corrupt_skill_rag	66.7%	False	True	ok	3
ShippingCode.000008	ShippingCode	corrupt_skill_rag	0.0%	False	True	ok	3
UserAgent.000003	UserAgent	corrupt_skill_rag	100.0%	True	True	ok	6
Address.000002	Address	direct_batch	33.3%	False	True	ok	3
Address.000013	Address	direct_batch	66.7%	False	True	ok	6
BillingCode.000007	BillingCode	direct_batch	0.0%	False	True	ok	3
City.000010	City	direct_batch	100.0%	True	True	ok	3
City.000011	City	direct_batch	75.0%	False	True	ok	4
Column.000001	Column	direct_batch	100.0%	True	True	ok	6
DateTime.000004	DateTime	direct_batch	100.0%	True	True	ok	6
DateTime.000007	DateTime	direct_batch	100.0%	True	True	ok	6
DateTime.000017	DateTime	direct_batch	83.3%	False	True	ok	6
DateTime.000025	DateTime	direct_batch	100.0%	True	True	ok	6
DateTime.000027	DateTime	direct_batch	66.7%	False	True	ok	6
DateTime.000034	DateTime	direct_batch	100.0%	True	True	ok	6
DateTime.000051	DateTime	direct_batch	0.0%	False	True	ok	3
DateTime.000076	DateTime	direct_batch	66.7%	False	True	ok	6
DateTime.000081	DateTime	direct_batch	0.0%	False	True	ok	6
DateTime.000094	DateTime	direct_batch	100.0%	True	True	ok	4
DateTime.000104	DateTime	direct_batch	100.0%	True	True	ok	6
DateTime.000108	DateTime	direct_batch	100.0%	True	True	ok	6
DateTime.000111	DateTime	direct_batch	83.3%	False	True	ok	6
DateTime.000114	DateTime	direct_batch	33.3%	False	True	ok	6
DateTime.000115	DateTime	direct_batch	0.0%	False	True	ok	6
DateTime.000116	DateTime	direct_batch	50.0%	False	True	ok	6
FilePath.000001	FilePath	direct_batch	100.0%	True	True	ok	6
Gender.000001	Gender	direct_batch	66.7%	False	True	ok	3
Language.000002	Language	direct_batch	100.0%	True	True	ok	6
Name.000028	Name	direct_batch	100.0%	True	True	ok	6
Number.000008	Number	direct_batch	16.7%	False	True	ok	6
Number.000015	Number	direct_batch	66.7%	False	True	ok	6
Number.000016	Number	direct_batch	0.0%	False	False	parse_fail	6
Number.000022	Number	direct_batch	33.3%	False	True	ok	6
Number.000028	Number	direct_batch	100.0%	True	True	ok	3
Number.000029	Number	direct_batch	0.0%	False	True	ok	3
Number.000043	Number	direct_batch	100.0%	True	True	ok	6
Number.000049	Number	direct_batch	0.0%	False	False	parse_fail	4
Number.000075	Number	direct_batch	66.7%	False	True	ok	6
Number.000077	Number	direct_batch	66.7%	False	True	ok	3
Phone.000008	Phone	direct_batch	100.0%	True	True	ok	6
Phone.000011	Phone	direct_batch	100.0%	True	True	ok	3
ShippingCode.000008	ShippingCode	direct_batch	33.3%	False	True	ok	3
UserAgent.000003	UserAgent	direct_batch	100.0%	True	True	ok	6
Address.000002	Address	direct_row	33.3%	False	True	row_clean	3
Address.000013	Address	direct_row	66.7%	False	True	row_clean	6
BillingCode.000007	BillingCode	direct_row	33.3%	False	True	row_clean	3
City.000010	City	direct_row	100.0%	True	True	row_clean	3
City.000011	City	direct_row	100.0%	True	True	row_clean	4
Column.000001	Column	direct_row	100.0%	True	True	row_clean	6
DateTime.000004	DateTime	direct_row	100.0%	True	True	row_clean	6
DateTime.000007	DateTime	direct_row	100.0%	True	True	row_clean	6
DateTime.000017	DateTime	direct_row	66.7%	False	True	row_clean	6
DateTime.000025	DateTime	direct_row	100.0%	True	True	row_clean	6
DateTime.000027	DateTime	direct_row	33.3%	False	True	row_clean	6
DateTime.000034	DateTime	direct_row	100.0%	True	True	row_clean	6
DateTime.000051	DateTime	direct_row	33.3%	False	True	row_clean	3
DateTime.000076	DateTime	direct_row	50.0%	False	True	row_clean	6
DateTime.000081	DateTime	direct_row	50.0%	False	True	row_clean	6
DateTime.000094	DateTime	direct_row	100.0%	True	True	row_clean	4
DateTime.000104	DateTime	direct_row	100.0%	True	True	row_clean	6
DateTime.000108	DateTime	direct_row	100.0%	True	True	row_clean	6
DateTime.000111	DateTime	direct_row	100.0%	True	True	row_clean	6
DateTime.000114	DateTime	direct_row	0.0%	False	True	row_clean	6
DateTime.000115	DateTime	direct_row	0.0%	False	True	row_clean	6
DateTime.000116	DateTime	direct_row	50.0%	False	True	row_clean	6
FilePath.000001	FilePath	direct_row	100.0%	True	True	row_clean	6
Gender.000001	Gender	direct_row	66.7%	False	True	row_clean	3
Language.000002	Language	direct_row	100.0%	True	True	row_clean	6
Name.000028	Name	direct_row	100.0%	True	True	row_clean	6
Number.000008	Number	direct_row	33.3%	False	True	row_clean	6
Number.000015	Number	direct_row	33.3%	False	True	row_clean	6
Number.000016	Number	direct_row	66.7%	False	True	row_clean	6
Number.000022	Number	direct_row	100.0%	True	True	row_clean	6
Number.000028	Number	direct_row	100.0%	True	True	row_clean	3
Number.000029	Number	direct_row	33.3%	False	True	row_clean	3
Number.000043	Number	direct_row	100.0%	True	True	row_clean	6
Number.000049	Number	direct_row	50.0%	False	True	row_clean	4
Number.000075	Number	direct_row	66.7%	False	True	row_clean	6
Number.000077	Number	direct_row	33.3%	False	True	row_clean	3
Phone.000008	Phone	direct_row	100.0%	True	True	row_clean	6
Phone.000011	Phone	direct_row	100.0%	True	True	row_clean	3
ShippingCode.000008	ShippingCode	direct_row	33.3%	False	True	row_clean	3
UserAgent.000003	UserAgent	direct_row	100.0%	True	True	row_clean	6
Address.000002	Address	random_skill_rag	33.3%	False	True	ok	3
Address.000013	Address	random_skill_rag	66.7%	False	True	ok	6
BillingCode.000007	BillingCode	random_skill_rag	33.3%	False	True	ok	3
City.000010	City	random_skill_rag	100.0%	True	True	ok	3
City.000011	City	random_skill_rag	50.0%	False	True	ok	4
Column.000001	Column	random_skill_rag	100.0%	True	True	ok	6
DateTime.000004	DateTime	random_skill_rag	100.0%	True	True	ok	6
DateTime.000007	DateTime	random_skill_rag	100.0%	True	True	ok	6
DateTime.000017	DateTime	random_skill_rag	83.3%	False	True	ok	6
DateTime.000025	DateTime	random_skill_rag	83.3%	False	True	ok	6
DateTime.000027	DateTime	random_skill_rag	66.7%	False	True	ok	6
DateTime.000034	DateTime	random_skill_rag	100.0%	True	True	ok	6
DateTime.000051	DateTime	random_skill_rag	33.3%	False	True	ok	3
DateTime.000076	DateTime	random_skill_rag	66.7%	False	True	ok	6
DateTime.000081	DateTime	random_skill_rag	66.7%	False	True	ok	6
DateTime.000094	DateTime	random_skill_rag	100.0%	True	True	ok	4
DateTime.000104	DateTime	random_skill_rag	100.0%	True	True	ok	6
DateTime.000108	DateTime	random_skill_rag	100.0%	True	True	ok	6
DateTime.000111	DateTime	random_skill_rag	83.3%	False	True	ok	6
DateTime.000114	DateTime	random_skill_rag	50.0%	False	True	ok	6
DateTime.000115	DateTime	random_skill_rag	0.0%	False	True	ok	6
DateTime.000116	DateTime	random_skill_rag	50.0%	False	True	ok	6
FilePath.000001	FilePath	random_skill_rag	100.0%	True	True	ok	6
Gender.000001	Gender	random_skill_rag	66.7%	False	True	ok	3
Language.000002	Language	random_skill_rag	100.0%	True	True	ok	6
Name.000028	Name	random_skill_rag	100.0%	True	True	ok	6
Number.000008	Number	random_skill_rag	16.7%	False	True	ok	6
Number.000015	Number	random_skill_rag	0.0%	False	True	ok	6
Number.000016	Number	random_skill_rag	16.7%	False	True	ok	6
Number.000022	Number	random_skill_rag	33.3%	False	True	ok	6
Number.000028	Number	random_skill_rag	100.0%	True	True	ok	3
Number.000029	Number	random_skill_rag	100.0%	True	True	ok	3
Number.000043	Number	random_skill_rag	100.0%	True	True	ok	6
Number.000049	Number	random_skill_rag	0.0%	False	True	ok	4
Number.000075	Number	random_skill_rag	50.0%	False	True	ok	6
Number.000077	Number	random_skill_rag	0.0%	False	True	ok	3
Phone.000008	Phone	random_skill_rag	100.0%	True	True	ok	6
Phone.000011	Phone	random_skill_rag	100.0%	True	True	ok	3
ShippingCode.000008	ShippingCode	random_skill_rag	33.3%	False	True	ok	3
UserAgent.000003	UserAgent	random_skill_rag	100.0%	True	True	ok	6
Address.000002	Address	skill_rag	33.3%	False	True	ok	3
Address.000013	Address	skill_rag	66.7%	False	True	ok	6
BillingCode.000007	BillingCode	skill_rag	0.0%	False	True	ok	3
City.000010	City	skill_rag	100.0%	True	True	ok	3
City.000011	City	skill_rag	100.0%	True	True	ok	4
Column.000001	Column	skill_rag	100.0%	True	True	ok	6
DateTime.000004	DateTime	skill_rag	100.0%	True	True	ok	6
DateTime.000007	DateTime	skill_rag	100.0%	True	True	ok	6
DateTime.000017	DateTime	skill_rag	66.7%	False	True	ok	6
DateTime.000025	DateTime	skill_rag	100.0%	True	True	ok	6
DateTime.000027	DateTime	skill_rag	50.0%	False	True	ok	6
DateTime.000034	DateTime	skill_rag	100.0%	True	True	ok	6
DateTime.000051	DateTime	skill_rag	0.0%	False	True	ok	3
DateTime.000076	DateTime	skill_rag	66.7%	False	True	ok	6
DateTime.000081	DateTime	skill_rag	66.7%	False	True	ok	6
DateTime.000094	DateTime	skill_rag	100.0%	True	True	ok	4
DateTime.000104	DateTime	skill_rag	100.0%	True	True	ok	6
DateTime.000108	DateTime	skill_rag	100.0%	True	True	ok	6
DateTime.000111	DateTime	skill_rag	100.0%	True	True	ok	6
DateTime.000114	DateTime	skill_rag	0.0%	False	True	ok	6
DateTime.000115	DateTime	skill_rag	0.0%	False	True	ok	6
DateTime.000116	DateTime	skill_rag	50.0%	False	True	ok	6
FilePath.000001	FilePath	skill_rag	100.0%	True	True	ok	6
Gender.000001	Gender	skill_rag	0.0%	False	True	ok	3
Language.000002	Language	skill_rag	100.0%	True	True	ok	6
Name.000028	Name	skill_rag	100.0%	True	True	ok	6
Number.000008	Number	skill_rag	16.7%	False	True	ok	6
Number.000015	Number	skill_rag	0.0%	False	True	ok	6
Number.000016	Number	skill_rag	33.3%	False	True	ok	6
Number.000022	Number	skill_rag	33.3%	False	True	ok	6
Number.000028	Number	skill_rag	100.0%	True	True	ok	3
Number.000029	Number	skill_rag	66.7%	False	True	ok	3
Number.000043	Number	skill_rag	100.0%	True	True	ok	6
Number.000049	Number	skill_rag	25.0%	False	True	ok	4
Number.000075	Number	skill_rag	66.7%	False	True	ok	6
Number.000077	Number	skill_rag	66.7%	False	True	ok	3
Phone.000008	Phone	skill_rag	100.0%	True	True	ok	6
Phone.000011	Phone	skill_rag	100.0%	True	True	ok	3
ShippingCode.000008	ShippingCode	skill_rag	0.0%	False	True	ok	3
UserAgent.000003	UserAgent	skill_rag	100.0%	True	True	ok	6

Files

runs/main_qwen_skill_memory_40_top1/generations.csv
runs/main_qwen_skill_memory_40_top1/task_details.csv
runs/main_qwen_skill_memory_40_top1/row_details.csv
runs/main_qwen_skill_memory_40_top1/retrieval_details.csv
analysis/summary.csv
analysis/task_details.csv
analysis/row_details.csv
analysis/retrieval_details.csv
analysis/retrieval_summary.csv
analysis/method_deltas.csv

Experiment log 11

Show the running log (11 entries, 2026-06-27)

Objective

Test whether a frozen model becomes more task-consistent when it can retrieve analogous verified transformation skills from a train-only memory.

The primary metric is strict full-task exact on held-out rows: a task is counted correct only when every held-out output is exact.

The experiment is standalone. It stores run-local caches, retrieval records, summaries, charts, Markdown report, and HTML report under this directory. Large benchmark data is referenced through /workspace/large_artifacts/qwen_verified_skill_memory_rag.

Initial Plan

Create a fresh experiment directory and separate large-artifact root.
Load public text-transformation tasks.
Split tasks into an evaluation set and a train-only skill-memory set.
Build verified skill cards from memory tasks.
Retrieve analogous skill cards for each evaluation task using only the task's training examples.
Compare:
- direct row-by-row Qwen
- direct batched Qwen
- retrieved-skill batched Qwen
- random-skill control
- corrupted-skill control
Generate CSVs, charts, Markdown report, and HTML report.

Run Notes

2026-06-27 08:45 UTC - Scaffold

Created fresh experiment directory: /workspace/experiments/qwen_verified_skill_memory_rag.
Created separate large-artifact root: /workspace/large_artifacts/qwen_verified_skill_memory_rag.
Added standalone runner: src/qwen_verified_skill_memory_rag.py.
Added persistent log and later added README.md.

2026-06-27 08:46 UTC - No-Qwen Smoke

Command:

python -m py_compile /workspace/experiments/qwen_verified_skill_memory_rag/src/qwen_verified_skill_memory_rag.py
python /workspace/experiments/qwen_verified_skill_memory_rag/src/qwen_verified_skill_memory_rag.py \
  --run_name smoke_no_qwen \
  --task_limit 4 \
  --heldout_cap 3 \
  --top_k 2 \
  --no_qwen

Result:

Smoke completed on 4 tasks.
CSVs, charts, Markdown report, and HTML report were generated.
Retrieval diagnostics were structurally valid; top retrievals were same-family on the tiny smoke sample.
Metrics are not interpreted because --no_qwen leaves uncached generations blank.

2026-06-27 08:47 UTC - Real-Qwen Pilot, Top-3 Retrieval

Command:

python /workspace/experiments/qwen_verified_skill_memory_rag/src/qwen_verified_skill_memory_rag.py \
  --run_name pilot_qwen_6 \
  --task_limit 6 \
  --heldout_cap 3 \
  --top_k 3

Result:

6 evaluation tasks.
direct_batch: 83.3% row exact, 66.7% full-task exact.
direct_row: 77.8% row exact, 66.7% full-task exact.
skill_rag: 61.1% row exact, 50.0% full-task exact.
random_skill_rag: 77.8% row exact, 66.7% full-task exact.
corrupt_skill_rag: 55.6% row exact, 33.3% full-task exact.

Diagnosis:

Top-3 retrieval over-steered the model on a date-format task: retrieved date examples pulled the model into a different format even though target examples were clear.
Prompt needed to make target examples more authoritative.

2026-06-27 08:50 UTC - Prompt Iteration

Changed the memory prompt so the target examples appear first and are explicitly described as authoritative. Reference transformations are labeled optional and ignorable when they conflict with the target.

2026-06-27 08:51 UTC - Real-Qwen Pilot, Top-3 Retrieval With Stronger Target Authority

Command:

python /workspace/experiments/qwen_verified_skill_memory_rag/src/qwen_verified_skill_memory_rag.py \
  --run_name pilot_qwen_6_v2 \
  --task_limit 6 \
  --heldout_cap 3 \
  --top_k 3

Result:

direct_batch: 83.3% row exact, 66.7% full-task exact.
direct_row: 77.8% row exact, 66.7% full-task exact.
skill_rag: 83.3% row exact, 50.0% full-task exact.
random_skill_rag: 83.3% row exact, 66.7% full-task exact.
corrupt_skill_rag: 77.8% row exact, 50.0% full-task exact.

Diagnosis:

The stronger prompt reduced row-level damage but still failed to beat direct baselines.
Top-3 retrieval still looked too intrusive.

2026-06-27 08:52 UTC - Real-Qwen Pilot, Top-1 Retrieval

Command:

python /workspace/experiments/qwen_verified_skill_memory_rag/src/qwen_verified_skill_memory_rag.py \
  --run_name pilot_qwen_6_top1 \
  --task_limit 6 \
  --heldout_cap 3 \
  --top_k 1

Result:

direct_batch: 83.3% row exact, 66.7% full-task exact.
direct_row: 77.8% row exact, 66.7% full-task exact.
skill_rag: 88.9% row exact, 66.7% full-task exact.
random_skill_rag: 83.3% row exact, 66.7% full-task exact.
corrupt_skill_rag: 83.3% row exact, 66.7% full-task exact.

Diagnosis:

Top-1 retrieval was cleaner than top-3.
It improved row exact on the pilot but did not improve strict full-task exact, and controls still tied it on full-task exact.
Chose top-1 for the main run because it was least damaging and still tested the intended mechanism.

2026-06-27 08:53 UTC - Main Run

Command:

python /workspace/experiments/qwen_verified_skill_memory_rag/src/qwen_verified_skill_memory_rag.py \
  --run_name main_qwen_skill_memory_40_top1 \
  --task_limit 40 \
  --heldout_cap 6 \
  --top_k 1

Result:

40 evaluation tasks.
269 train-only memory tasks.
364 unique cached generation records.
No blank generations in the main cache.
Top-1 retrieved skill was same-family 77.5% of the time, versus 20.0% for random retrieval.

Main metrics:

direct_row: 70.8% row exact, 50.0% full-task exact.
skill_rag: 65.2% row exact, 47.5% full-task exact.
corrupt_skill_rag: 63.1% row exact, 45.0% full-task exact.
direct_batch: 65.2% row exact, 42.5% full-task exact.
random_skill_rag: 67.1% row exact, 42.5% full-task exact.

Deltas versus direct_row:

skill_rag: -2.5 full-task points, -5.6 row-exact points; helped 0 tasks, hurt 1 task, tied 39 tasks.
corrupt_skill_rag: -5.0 full-task points.
direct_batch: -7.5 full-task points.
random_skill_rag: -7.5 full-task points.

Final diagnosis:

Negative for the tested verified skill-memory mechanism.
The retriever itself was not random: it found same-family memory tasks at a high rate.
The failure is in converting a retrieved analogous skill card into better target outputs. The card changes the prompt distribution but does not reliably improve task consistency.
Direct row-by-row inference remains the best method in this experiment.

2026-06-27 08:59 UTC - Report Hardening

Added analysis/method_deltas.csv.
Added analysis/figures/wins_losses_vs_direct.png.
Updated Markdown and HTML reports to state the negative conclusion explicitly.
Regenerated reports from the cached main run with --no_qwen; no additional model calls were required.

Final artifacts:

README.md
experiment_log.md
src/qwen_verified_skill_memory_rag.py
runs/main_qwen_skill_memory_40_top1/generations.csv
runs/main_qwen_skill_memory_40_top1/task_details.csv
runs/main_qwen_skill_memory_40_top1/row_details.csv
runs/main_qwen_skill_memory_40_top1/retrieval_details.csv
runs/main_qwen_skill_memory_40_top1/summary.csv
runs/main_qwen_skill_memory_40_top1/retrieval_summary.csv
runs/main_qwen_skill_memory_40_top1/method_deltas.csv
analysis/*.csv
analysis/figures/*.png
reports/qwen_verified_skill_memory_rag_report.md
reports/qwen_verified_skill_memory_rag_report.html

Figures 6

method full task exact · analysis/figures/

retrieval family agreement · analysis/figures/

top retrieval scores · analysis/figures/

wins losses vs direct · analysis/figures/

Data files 18

Result tables and metrics copied from the experiment folder — preview inline or open the raw file.

Reproduce

This entry is source code or analysis only — there is no separate run to reproduce.

Browse the experiment folder on GitHub ↗

Qwen Verified Skill Memory RAG

Results at a glance 8

Overview

Main Run

Key CSVs

Key Charts

Data Placement

Report

Question

Setup

Main Result

Interpretation

Charts

Deltas Versus Direct Row Baseline

Retrieval Diagnostics

Task Details

Files

Experiment log 11

Objective

Initial Plan

Run Notes

2026-06-27 08:45 UTC - Scaffold

2026-06-27 08:46 UTC - No-Qwen Smoke

2026-06-27 08:47 UTC - Real-Qwen Pilot, Top-3 Retrieval

2026-06-27 08:50 UTC - Prompt Iteration

2026-06-27 08:51 UTC - Real-Qwen Pilot, Top-3 Retrieval With Stronger Target Authority

2026-06-27 08:52 UTC - Real-Qwen Pilot, Top-1 Retrieval

2026-06-27 08:53 UTC - Main Run

2026-06-27 08:59 UTC - Report Hardening

Figures 6

Data files 18

Reproduce

Related

Cited by claims

Related experiments