Qwen3.5-4B Verified Algorithm Retrieval Adaptation
The one idea you need
A programmer keeps a cookbook of solutions that provably worked on past problems. Stuck on a new one, they pull the closest-matching entry and tweak it to fit. Here a small coding model does exactly that with a library of its own verified solutions.
The question
When a small coding model keeps failing certain problems, does handing it a similar problem it already solved and letting it adapt that code help it finally pass?
What we found
Yes, but modestly. Fetching the most similar already-solved problem and adapting its code cracked 3 of the 8 problems that repeated tries kept missing, lifting overall success from 67% to 79%. Matching by meaning clearly beat fetching random code (rescued none) and mislabeled code (rescued one). But only 2 wins truly needed smart matching, and most adaptations that passed the shown tests still failed the hidden ones.
Why it matters
When a small model plateaus, a searchable library of its own verified solutions plus adaptation is a cheap way to claw back a few failures. But pair it with a stronger checker: thin public tests wave wrong answers through.
On this page
Results at a glance 3
How to read
Five bars, each a different pool of candidate solutions; bar height is the share of 24 new problems solved, so taller is better. The leftmost is trying many times alone; the rest add retrieved solutions of differing relevance.
Takeaway → The meaning-matched bar rises highest, 67% to 79%, while the random-retrieval bar stays flat at 67%, so the content of the retrieved solution, not just having one, does the work.
Data table
| candidate pool | coverage (24 held-out tasks) |
|---|---|
| direct sampling only | 66.7% |
| + copy/rename retrieval | 70.8% |
| + semantic retrieval | 79.2% |
| + random retrieval | 66.7% |
| + shuffled retrieval | 70.8% |
Numbers from reports/report_summary.json (combined)
Technical framing
Held-out coverage: direct sampling alone vs plus retrieval adaptation — Adapting semantically retrieved verified algorithms lifts coverage from 66.7% to 79.2%; random retrieval adds nothing.
How to read
Four bars, one per type of retrieved solution; height is how many of the 8 stubborn problems that repeated tries kept missing were newly solved, so taller is better.
Takeaway → Meaning-matched code cracks 3 problems versus 0 for random and 1 each for mislabeled or blind-copied code, showing the match earns most of the gain.
Data table
| retrieval arm | misses recovered |
|---|---|
| semantic | 3 |
| shuffled | 1 |
| copy/rename | 1 |
| random | 0 |
Numbers from reports/report_summary.json (retrieval_rows zero_to_one)
Technical framing
Direct-sampling misses recovered by each retrieval arm (of 8 misses) — Semantic matching recovers 3 misses vs 0 for random and 1 for shuffled, so the retrieved algorithm content matters.
How to read
Four bars, one per type of retrieved solution; height is the share of answers that passed the shown example tests yet still failed the hidden checks, so lower is better.
Takeaway → Every bar is tall, from about 4 in 7 up to all of them, meaning passing the shown examples alone is not safe enough to trust an answer.
Data table
| retrieval arm | visible-pass hidden-wrong rate |
|---|---|
| semantic | 57.1% |
| copy/rename | 66.7% |
| shuffled | 85.7% |
| random | 100% |
Numbers from reports/report_summary.json (retrieval_rows visible_hidden_wrong_rate)
Technical framing
Failure mode: visible-pass candidates that fail hidden tests — Public tests are too thin: 57-100% of adaptations that pass visible tests still fail hidden tests, motivating a verifier before commit.
In the author’s words from the Report · “Interpretation”
Semantic retrieval adaptation passes the primary pilot gate: it recovers three direct-sampling misses, compared with zero for random retrieval and one for shuffled retrieval. Combined with the direct sample-more pool, coverage rises from 66.7% to 79.2% on this 24-task slice at an additional 7,699 forward tokens. The control read is important. Copy/rename and shuffled retrieval both recover task 20, so that task is not strong evidence for semantic matching. The stronger semantic-specific lift is tasks 15 and 25, where matched retrieved algorithms map cleanly onto the target operation family. The main failure mode is also clear: visible-pass hidden-wrong rates are high for all retrieval arms. … Read the full result →
Overview
Standalone experiment package for testing verified algorithm retrieval plus Qwen3.5-4B adaptation on MBPP-style code tasks.
The experiment builds a verified algorithm library from training tasks, retrieves nearest algorithms for held-out tasks, asks Qwen to adapt retrieved code to the target function and public tests, and evaluates whether this recovers tasks missed by direct sampling.
Large artifacts, if any are added later, should be stored outside this directory under:
/workspace/large_artifacts/qwen35_4b_verified_algorithm_retrieval_adaptation
Final report: reports/final_report.md
Report
Rendered from reports/final_report.md
Question
Can verified algorithm retrieval plus Qwen adaptation recover held-out tasks that direct sampling missed?
The experiment builds a verified algorithm library from training tasks, retrieves top-k candidate algorithms for each held-out miss, adapts them to the target task with Qwen, and evaluates hidden tests only after candidate generation.
Setup
- Verified library entries: 364
- Eval baseline tasks: 24
- Direct sample-more coverage: 66.7%
- Direct sample-more misses: [15, 16, 20, 21, 24, 25, 26, 31]
- Retrieval top-k: 3
Retrieval-Only Results
| arm | retrieval coverage | zero-to-one | visible-pass hidden-wrong | parse/task | tokens |
|---|---|---|---|---|---|
| retrieval_copy_rename_top3 | 12.5% | 1 (12.5%) | 2/3 (66.7%) | 3.00 | 0 |
| retrieval_adapt_semantic_top3 | 37.5% | 3 (37.5%) | 4/7 (57.1%) | 2.62 | 7699 |
| retrieval_adapt_random_top3 | 0.0% | 0 (0.0%) | 5/5 (100.0%) | 2.75 | 7982 |
| retrieval_adapt_shuffled_top3 | 12.5% | 1 (12.5%) | 6/7 (85.7%) | 2.88 | 7881 |
Combined With Direct Sampling
| combined arm | coverage | zero-to-one tasks | forward tokens |
|---|---|---|---|
| base_sample_more | 66.7% | [] | 45406 |
| base_plus_retrieval_copy_rename_top3 | 70.8% | [20] | 45406 |
| base_plus_retrieval_adapt_semantic_top3 | 79.2% | [15, 20, 25] | 53105 |
| base_plus_retrieval_adapt_random_top3 | 66.7% | [] | 53388 |
| base_plus_retrieval_adapt_shuffled_top3 | 70.8% | [20] | 53287 |
Recovered Tasks
| arm | recovered task | task | winner sources |
|---|---|---|---|
| retrieval_copy_rename_top3 | 20 | Write a function to check if the given number is woodball or not. | ['retrieval_copy_rename_top3_semantic_r0'] |
| retrieval_adapt_semantic_top3 | 15 | Write a function to split a string at lowercase letters. | ['retrieval_adapt_semantic_top3_semantic_r0_s0'] |
| retrieval_adapt_semantic_top3 | 20 | Write a function to check if the given number is woodball or not. | ['retrieval_adapt_semantic_top3_semantic_r0_s0'] |
| retrieval_adapt_semantic_top3 | 25 | Write a python function to find the product of non-repeated elements in a given array. | ['retrieval_adapt_semantic_top3_semantic_r0_s0'] |
| retrieval_adapt_shuffled_top3 | 20 | Write a function to check if the given number is woodball or not. | ['retrieval_adapt_shuffled_top3_shuffled_r0_s0'] |
Gate Readout
Semantic retrieval adaptation recovered 3 direct-sampling misses. Random retrieval adaptation recovered 0. Shuffled retrieval adaptation recovered 1.
Interpretation
Semantic retrieval adaptation passes the primary pilot gate: it recovers three direct-sampling misses, compared with zero for random retrieval and one for shuffled retrieval. Combined with the direct sample-more pool, coverage rises from 66.7% to 79.2% on this 24-task slice at an additional 7,699 forward tokens.
The control read is important. Copy/rename and shuffled retrieval both recover task 20, so that task is not strong evidence for semantic matching. The stronger semantic-specific lift is tasks 15 and 25, where matched retrieved algorithms map cleanly onto the target operation family.
The main failure mode is also clear: visible-pass hidden-wrong rates are high for all retrieval arms. Retrieval gives Qwen useful external algorithmic memory, but public tests are too thin to safely commit every visible-passing adaptation. The next iteration should scale this to a larger held-out slice and add a retrieval-candidate verifier/reranker or generated counterexample tests before commit.
Experiment log 5
Show the running log (5 entries, 2026-06-26)
2026-06-26
- Created standalone verified algorithm retrieval + adaptation experiment package.
- Primary question: can retrieval of verified training algorithms plus Qwen adaptation recover held-out tasks missed by direct sampling?
- Primary metric: zero-to-one lift on tasks where direct K=8 sampling has no hidden-correct candidate.
Planned arms:
- direct base sampling baseline;
- literal retrieved-code copy/rename baseline;
- semantic retrieval adaptation;
- random retrieval adaptation control;
- shuffled retrieval adaptation control;
- oracle over generated retrieval candidates as headroom.
Guardrails:
- hidden tests are evaluation only;
- public tests are the only deployable evidence given to adaptation prompts;
- random and shuffled retrieval controls must underperform semantic retrieval for a positive read.
Setup
- Added local direct-sampling baseline pool:
data/base_sample_more_k8_records.jsonl. Built verified algorithm library from MBPP train references:
- loaded 374 train records;
- kept 364 verified algorithms;
- dropped 10 unverified references.
Built retrieval plan for the eight direct-sampling misses:
- tasks 15, 16, 20, 21, 24, 25, 26, 31;
- top-k semantic retrieval = 3;
- random and shuffled controls matched at top-k = 3.
Iteration Notes
- First ran
copy_semantic, a no-model literal copy/rename baseline. - The first copy/rename attempt exposed a regex escaping bug in
copy_rename_code; patched and reran. Then ran three Qwen adaptation arms in one model-load session:
- semantic retrieval adaptation;
- random retrieval adaptation;
- shuffled retrieval adaptation.
Results
| arm | retrieval coverage | zero-to-one | visible-pass hidden-wrong | parse/task | tokens |
|---|---|---|---|---|---|
| retrieval_copy_rename_top3 | 12.5% | 1/8 | 2/3 | 3.00 | 0 |
| retrieval_adapt_semantic_top3 | 37.5% | 3/8 | 4/7 | 2.62 | 7699 |
| retrieval_adapt_random_top3 | 0.0% | 0/8 | 5/5 | 2.75 | 7982 |
| retrieval_adapt_shuffled_top3 | 12.5% | 1/8 | 6/7 | 2.88 | 7881 |
Combined with direct sample-more:
| arm | combined coverage | recovered tasks | forward tokens |
|---|---|---|---|
| base_sample_more | 66.7% | [] | 45406 |
| base_plus_retrieval_copy_rename_top3 | 70.8% | [20] | 45406 |
| base_plus_retrieval_adapt_semantic_top3 | 79.2% | [15, 20, 25] | 53105 |
| base_plus_retrieval_adapt_random_top3 | 66.7% | [] | 53388 |
| base_plus_retrieval_adapt_shuffled_top3 | 70.8% | [20] | 53287 |
Readout
- Semantic retrieval adaptation passes the pilot gate: 3/8 direct misses recovered versus 0/8 random and 1/8 shuffled.
- Task 20 is not strong semantic evidence because copy/rename and shuffled retrieval also recovered it.
- Tasks 15 and 25 are the semantic-specific lift.
- Main failure mode: high visible-pass hidden-wrong rate, so retrieval adaptation needs a verifier/reranker or counterexample generation before commit.
Figures 2
Data files 3
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
reports/library_summary.json267 Breports/report_summary.json12 kBreports/retrieval_plan_summary.json290 B
Reproduce
Run steps are documented inside the experiment folder (README and scripts).