Qwen3.5-4B Retrieval Adapt Verify Scale
The one idea you need
A small coding model that just failed a problem flips through a cookbook of 364 already-solved recipes, grabs the closest match, and rewrites it to fit the new problem. Some failures aren't beyond it — they were just missing the right recipe to start from.
The question
When a coding model gets stuck, can pulling a similar solved problem from a library and adapting it fix the miss — and can safe checks trust the result?
What we found
It splits. Pulling a similar solved problem from a 364-entry library and rewriting it recovered 8 of 24 stuck tasks — a third, double the 4 that a random-library control found. But the only safe check, the visible sample tests, passed roughly two-thirds of wrong answers, so the best deployable picker committed 7 right fixes and 7 wrong ones — a coin flip on trust.
Why it matters
For small-model coding, the hard part isn't generating a fix once you retrieve a good hint — it's proof. Thin sample tests and code-only re-rankers can't separate right from wrong; invest in counterexample tests and cross-candidate agreement before committing.
On this page
Results at a glance 2
How to read
Each bar is one lookup strategy; bar height is the share of the 24 stuck tasks it rescued into the candidate pool, so taller is better. Meaning-matched lookup is tallest; plain copying is shortest.
Takeaway → The meaning-matched bar reaches a third of stuck tasks, roughly double the random-lookup bar — matching by meaning, not surface words, is what recovers misses.
Data table
| retrieval arm | residual pool coverage (of 24 misses) |
|---|---|
| copy/rename top-3 | 4.2% |
| semantic adapt top-3 | 33.3% |
| random adapt top-3 | 16.7% |
| shuffled-query adapt top-3 | 12.5% |
Technical framing
Residual tasks recovered into the candidate pool by retrieval arm — Semantic retrieval plus Qwen adaptation recovers a third of direct-sampling misses, doubling the random-retrieval control.
How to read
For each picking method, paired bars show correct fixes committed versus wrong ones, out of 24 stuck tasks; fewer wrong is better. Only the last method, a cheat that peeks at the hidden answers, has no wrong bar.
Takeaway → For every real method the wrong bar is at least as tall as the correct bar — first-visible ties 7-to-7 while the other three commit even more wrong than right; the safe checks can't separate right from wrong, and only the peeking baseline avoids mistakes.
Data table
| selector | correct commits | hidden-wrong commits |
|---|---|---|
| first visible | 7 | 7 |
| shortest visible | 6 | 8 |
| consensus visible | 6 | 8 |
| frozen-Qwen rerank | 5 | 8 |
| hidden oracle | 8 | 0 |
Numbers from experiments/qwen35_4b_retrieval_adapt_verify_scale/reports/report_summary.json (selector_counts)
Technical framing
Selector commits on residual tasks: correct vs hidden-wrong — Every deployable selector commits about as many hidden-wrong candidates as correct ones; only the non-deployable oracle avoids false commits.
In the author’s words from the Report · “Interpretation”
This is a positive coverage result and a negative selector result. The positive part is that semantic retrieval plus Qwen adaptation works on this 24-task residual scale: it recovers a third of the direct-sampling residual, doubles random-retrieval coverage, beats shuffled-query retrieval, and adds four control-clean residual tasks. This supports the external algorithmic-memory direction: some misses are not beyond adaptation; they are missing the right algorithmic hint. The negative part is deployable selection. Public tests are too thin: most visible-pass candidates in the main pool are hidden-wrong, and neither agreement probes nor a frozen-Qwen reranker reduce that risk. … Read the full result →
Overview
Standalone experiment: scale retrieval-and-adaptation on direct-sampling misses, then test whether deployable verification can turn new candidate-pool coverage into selected pass@1.
The experiment uses Qwen3.5-4B as the adapter/reranker and a verified MBPP train library as external algorithmic memory. Hidden tests are used only for evaluation summaries and oracle ceilings.
Primary Question
Can semantic retrieval plus adaptation recover tasks missed by direct Qwen sampling, and can public-test-safe selectors distinguish the hidden-correct adaptations from visible-pass hidden-wrong ones?
Arms
base_direct_k4: 80 held-out MBPP tasks, four direct samples per task.retrieval_copy_rename_top3: copy top-3 retrieved train-library algorithms after renaming the function.retrieval_adapt_semantic_top3: Qwen adapts the top-3 semantic retrievals.retrieval_adapt_random_top3: random-library control.retrieval_adapt_shuffled_top3: shuffled-query retrieval control.Selector arms over residual tasks:
first_visibleshortest_visibleconsensus_visibleusing target-independent agreement probes.frozen_qwen_visible_rerankoracle_hiddenas non-deployable headroom.
Main Metrics
- Direct baseline coverage on all 80 tasks.
- Residual zero-to-one pool coverage on baseline misses.
- Visible-pass hidden-wrong rate.
- Selected recovery rate for deployable selectors.
- Combined all-task coverage/pass@1 implied by direct baseline plus retrieval recovery.
- Forward-token cost.
Artifact Layout
configs/experiment.json: complete run configuration.data/: JSONL candidate pools, retrieval plan, selector records, summaries.reports/: final report, figures, machine-readable summary.run_logs/: stdout/stderr logs for each script.logs/experiment_log.md: chronological experiment log./workspace/large_artifacts/qwen35_4b_retrieval_adapt_verify_scale/: reserved for large files such as checkpoints. This run should not put small downloadable artifacts there.
Report
Rendered from reports/final_report.md
Motivation
Direct Qwen3.5-4B sampling covers many MBPP held-out tasks, but leaves a residual set with no hidden-correct candidate in the sample pool. This experiment tests an external-memory route around that gap: retrieve verified train-library algorithms, ask Qwen to adapt them to each residual task, then measure both hidden-test pool coverage and deployable selection.
Hidden tests are used only for evaluation and oracle ceilings. Public tests, candidate code, and target-independent agreement probes are the only deployable evidence used by selectors.
Setup
- Base pool: 80 MBPP held-out tasks, K=4 direct samples per task.
- Base coverage: 56/80 (70.0%).
- Residual tasks: 24 direct-sampling misses.
- Verified algorithm library: 364 MBPP train references.
- Retrieval: TF-IDF top-3 semantic retrieval, plus random and shuffled-query controls.
- Adaptation: one Qwen3.5-4B completion per retrieved algorithm.
- Selector pool: copy/rename top-3 plus semantic adaptations.
Candidate-Pool Coverage
| arm | residual recovered | rate | recovered tasks | forward tokens |
|---|---|---|---|---|
| copy/rename top-3 | 1/24 | 4.2% | [67] | 0 |
| semantic adapt top-3 | 8/24 | 33.3% | [15, 35, 36, 42, 44, 67, 73, 87] | 24352 |
| random adapt top-3 | 4/24 | 16.7% | [36, 42, 67, 84] | 25603 |
| shuffled-query adapt top-3 | 3/24 | 12.5% | [36, 42, 73] | 26127 |
Semantic retrieval is the strongest pool-coverage arm: 8/24 residual recoveries versus 4/24 random and 3/24 shuffled. The semantic-only recoveries beyond copy/random/shuffled are [15, 35, 44, 87]. If those hidden-correct candidates were selectable perfectly, all-task coverage would rise from 56/80 to 64/80 (80.0%).
Selection and False Passes
The caveat is still visible-pass hidden-wrong noise. In the main copy+semantic pool, 20/31 visible-pass candidates fail hidden tests (64.5%).
| selector | correct residual commits | wrong visible-pass commits | no commit | selected recovery |
|---|---|---|---|---|
| first visible | 7 | 7 | 10 | 29.2% |
| consensus visible | 6 | 8 | 10 | 25.0% |
| shortest visible | 6 | 8 | 10 | 25.0% |
| frozen-Qwen rerank | 5 | 8 | 11 | 20.8% |
| hidden oracle | 8 | 0 | 16 | 33.3% |
The simplest deployable selector, first-visible, captures 7/8 oracle recoveries but also commits 7 hidden-wrong visible passers. Target-independent agreement probes do not help here, and the frozen-Qwen reranker is worse than first-visible.
Interpretation
This is a positive coverage result and a negative selector result.
The positive part is that semantic retrieval plus Qwen adaptation works on this 24-task residual scale: it recovers a third of the direct-sampling residual, doubles random-retrieval coverage, beats shuffled-query retrieval, and adds four control-clean residual tasks. This supports the external algorithmic-memory direction: some misses are not beyond adaptation; they are missing the right algorithmic hint.
The negative part is deployable selection. Public tests are too thin: most visible-pass candidates in the main pool are hidden-wrong, and neither agreement probes nor a frozen-Qwen reranker reduce that risk. The main bottleneck after retrieval is not generating a candidate; it is obtaining enough trustworthy evidence to commit it.
Next Direction
The next high-value run should keep semantic retrieval+adaptation, but replace weak public-test selection with stronger deployable evidence:
- generate or mine counterexample tests with output agreement, not expected answers;
- require candidates to survive multiple independently retrieved/adapted implementations by consensus;
- use a verifier only after the evidence set is enlarged, because code-only reranking did not separate correct from hidden-wrong candidates here.
Artifacts
data/base_direct_k4_records.jsonldata/retrieval_plan.jsonldata/retrieval_adapt_semantic_top3_records.jsonldata/retrieval_adapt_random_top3_records.jsonldata/retrieval_adapt_shuffled_top3_records.jsonldata/selector_copy_semantic_records.jsonldata/qwen_rerank_copy_semantic_records.jsonlreports/report_summary.json
Experiment log 1
Show the running log (1 entry, 2026-06-26)
2026-06-26
- Created standalone experiment package.
- Copied only generic evaluator/model utilities and retrieval scripts into the standalone package.
- Localized the experiment identity to
qwen35_4b_retrieval_adapt_verify_scale. - Copied the 80-task MBPP heldout direct K=4 baseline into this package as the scale substrate.
- Added selector scripts for target-independent agreement probes and frozen-Qwen visible-candidate reranking.
- Rebuilt base manifest: 80 records, 56/80 direct coverage, 24 residual misses.
- Verified 364 MBPP train reference algorithms for the retrieval library.
- Planned top-3 semantic, random, and shuffled retrieval for all 24 residual tasks.
- Ran copy/rename retrieval: 1/24 residual pool recovery.
Ran generated retrieval adaptation arms:
- semantic top-3: 8/24 residual pool recovery, 24,352 forward tokens;
- random top-3: 4/24 residual pool recovery, 25,603 forward tokens;
- shuffled-query top-3: 3/24 residual pool recovery, 26,127 forward tokens.
Evaluated copy+semantic selectors:
- residual oracle pool recovery: 8/24;
- first-visible selected recovery: 7/24, with 7 visible-pass hidden-wrong commits;
- agreement-consensus selected recovery: 6/24, with 8 visible-pass hidden-wrong commits;
- frozen-Qwen visible rerank selected recovery: 5/24, with 8 visible-pass hidden-wrong commits.
- Generated final report, machine-readable summary, and four figures under
reports/.
Figures 4
Data files 8
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
reports/library_summary.json256 Breports/qwen_rerank_copy_semantic_summary.json493 Breports/report_summary.json2.7 kBreports/retrieval_plan_summary.json408 Breports/selector_copy_semantic_summary.json3.8 kBreports/selector_random_summary.json3.0 kBreports/selector_semantic_summary.json3.0 kBreports/selector_shuffled_summary.json2.9 kB
Reproduce
Runnable scripts exist in the experiment folder, but the exact invocation was not written down.