Qwen3.5-4B Independent Retrieval Consensus
The one idea you need
Imagine pulling several proven algorithms from a small library, reshaping each independently into a solution for a new coding task, then trusting only answers where those separate versions agree on a tricky test input you invented yourself, with no answer key. Here they almost never agreed.
The question
If you fetch several unrelated proven algorithms, reshape each to a new coding task, and trust wherever their independent versions agree, can that agreement pick correct answers without an answer key?
What we found
No. Requiring independently reshaped solutions to agree on invented tricky inputs, the system committed to zero of 24 unsolved tasks — the separate versions never converged. Simply taking the first solution that passes the public tests got 6 right but 8 wrong, a coin-flip error rate. Worse, forcing source variety cut how many tasks were solvable at all, from 9 down to 7, so it beat neither the simple picker nor a same-topic control.
Why it matters
Diversifying retrieval sources does not manufacture trustworthy agreement. When your library lacks several genuinely relevant algorithms per task, forcing variety just cuts relevance. Invest in a larger verified library or stronger counterexample generation, not diversity for its own sake.
On this page
Results at a glance 2
How to read
Each pair of bars is one picking rule; height counts how many of 24 problems it committed an answer to. Green is commits that were actually correct, red actually wrong. More green and less red is better; the rightmost pair is the best any picker could do.
Takeaway → Both independent-agreement rules sit flat at zero — requiring separate solutions to agree never committed to anything, while the loose first-pass rule commits often but is wrong more than right.
Data table
| selector | hidden-correct commits | hidden-wrong commits |
|---|---|---|
| first-visible (independent) | 6 | 8 |
| independent consensus min-2 | 0 | 0 |
| independent consensus min-3 | 0 | 0 |
| same-neighborhood consensus min-2 | 2 | 3 |
| same-neighborhood consensus min-3 | 0 | 2 |
| oracle (independent pool) | 7 | 0 |
Numbers from experiments/qwen35_4b_independent_retrieval_consensus/reports/consensus_summary.json
Technical framing
Consensus selection outcomes on 24 residual MBPP tasks — Strict independent consensus never committed (0/24); same-neighborhood consensus committed rarely and mostly on wrong answers.
How to read
For two ways of pulling reference solutions, one bar shows the share of problems the pool could actually solve (higher is better), the other the share of passing-looking answers that were secretly wrong (lower is better). Compare the diversified independent pool against the same-topic pool.
Takeaway → The diversified pool has the lower solve share and the taller wrong-answer bar — forcing variety made the source material less relevant, not more trustworthy.
Data table
| candidate pool | hidden-test coverage | visible-pass hidden-wrong rate |
|---|---|---|
| independent retrieval-adapt top-6 | 29.2% | 76.2% |
| same-neighborhood retrieval-adapt top-6 | 37.5% | 64.1% |
Numbers from experiments/qwen35_4b_independent_retrieval_consensus/reports/consensus_summary.json
Technical framing
Pool quality: coverage vs false-pass risk by retrieval pool — Diversified independent retrieval lowered coverage below the same-neighborhood control and raised the visible-pass false-positive rate.
In the author’s words from the Report · “Interpretation”
The hypothesis was plausible: independent derivations agreeing should provide evidence unavailable to any single-candidate judge. The implementation successfully increased retrieval independence, but that independence did not translate into useful agreement. Instead, it mostly reduced relevance and coverage. The non-independent control remained more coverage-effective, and consensus over it did not suppress false-pass enough to beat first-visible. This is a negative for independent-retrieval consensus as implemented here. … Read the full result →
Overview
Standalone experiment for independent-retrieval consensus selection on MBPP residual tasks.
The experiment tests whether independently retrieved and adapted verified algorithms can provide deployable selection evidence by agreeing on generated disagreement inputs. The package contains all scripts, local data copies, run logs, summaries, figures, and a final report. Large checkpoints and model caches are kept outside this directory.
Report
Rendered from reports/final_report.md
Date: 2026-06-26
Decision
Independent-retrieval consensus does not pass the deployable gate in this run. The retrieval planner did create more independent sources, but the generated independent adaptations did not converge on disagreement probes. With the mechanism-faithful rule that requires at least one generated disagreement probe, independent consensus committed on 0/24 tasks. First-visible on the same independent pool selected 6/24 hidden-correct tasks, with 8 hidden-wrong visible-pass selections.
The stronger pool was the non-independent same-neighborhood control: it had 9/24 oracle coverage versus 7/24 for independent retrieval and 7/24 for direct K12 sampling. But same-neighborhood consensus still failed selection: min-2 consensus selected 2/24 hidden-correct and 3 hidden-wrong tasks.
Question
Can independently retrieved verified algorithms supply the missing behavioral evidence for selecting correct retrieval-adapt candidates without hidden-test labels?
Setup
- Residual tasks: 24 MBPP heldout tasks missed by the base direct K=4 pool.
- Library: 364 verified algorithms.
- Retrieval arms: top-6 same-neighborhood semantic retrieval and top-6 MMR-diversified independent retrieval.
- Generation: one Qwen3.5-4B adaptation per retrieved algorithm, T=0.2, top-p 0.95.
- Consensus: generate up to 64 input probes from public test perturbations, choose up to 8 calls that split visible-passing candidates, and commit only when at least 2 or 3 distinct retrieved sources agree on the same output signature.
- Baseline: direct K12 sample-more on the same 24 residual tasks.
Independence Gate
| retrieval set | code distance | task-token distance | mean retrieval score |
|---|---|---|---|
| same-neighborhood | 0.701 | 0.635 | 0.237 |
| independent MMR | 0.803 | 0.700 | 0.226 |
The build gate passed: independent retrieval increased source-code and task-token distance while retaining similar semantic score.
Pool Coverage
| pool | coverage | visible-pass hidden-wrong rate | candidates/task | forward tokens |
|---|---|---|---|---|
| independent retrieval-adapt top-6 | 29.2% | 76.2% | 5.62 | 51,282 |
| same-neighborhood retrieval-adapt top-6 | 37.5% | 64.1% | 5.38 | 52,001 |
| direct sample-more K12 | 29.2% | not selector pool | 11.50 | 74,780 |
Independence did not improve coverage. It lowered coverage relative to the same-neighborhood control and matched direct K12.
Consensus Selection
| selector | commits | hidden-correct commits | hidden-wrong commits | false-pass rate |
|---|---|---|---|---|
| first visible independent | 14 | 6 | 8 | 57.1% |
| independent consensus min-2 | 0 | 0 | 0 | 0.0% |
| independent consensus min-3 | 0 | 0 | 0 | 0.0% |
| first visible same-neighborhood | 15 | 7 | 8 | 53.3% |
| same-neighborhood consensus min-2 | 5 | 2 | 3 | 60.0% |
| same-neighborhood consensus min-3 | 2 | 0 | 2 | 100.0% |
| oracle independent | 7 | 7 | 0 | 0.0% |
| oracle same-neighborhood | 9 | 9 | 0 | 0.0% |
| oracle union | 9 | 9 | 0 | 0.0% |
Strict independent consensus had no deployable commits because no covered task had cross-source agreement on actual disagreement probes. Same-neighborhood consensus committed sometimes, but its false-pass rate remained high.
Probe Diagnostics
| pool | tasks with generated disagreement probes | mean selected probes/task |
|---|---|---|
| independent | 7/24 | 1.25 |
| same-neighborhood | 8/24 | 1.54 |
Covered independent tasks were [15, 35, 36, 42, 44, 67, 87]. Covered same-neighborhood tasks were [15, 35, 36, 42, 44, 67, 73, 77, 87].
The key failure mode is not just a conservative threshold. Independent adaptations often produced no source-agreement cluster after target-independent disagreement probes. When agreement existed in same-neighborhood candidates, it was often agreement on the wrong behavior.
Cost
- Independent retrieval-adapt: 51,282 forward tokens.
- Same-neighborhood retrieval-adapt: 52,001 forward tokens.
- Direct K12 sample-more: 74,780 forward tokens.
The strongest coverage/cost point here is same-neighborhood retrieval-adapt, not independent consensus. It still does not solve deployable selection.
Interpretation
The hypothesis was plausible: independent derivations agreeing should provide evidence unavailable to any single-candidate judge. The implementation successfully increased retrieval independence, but that independence did not translate into useful agreement. Instead, it mostly reduced relevance and coverage. The non-independent control remained more coverage-effective, and consensus over it did not suppress false-pass enough to beat first-visible.
This is a negative for independent-retrieval consensus as implemented here. The result narrows the next direction: the missing evidence probably needs either stronger, task-targeted counterexample generation with a real oracle, or a larger/higher-quality verified library where multiple genuinely relevant independent algorithms exist for the same residual task. Merely diversifying among the current 364 MBPP-derived algorithms did not produce independent correct convergence.
Artifacts
- Data:
data/ - Run logs:
run_logs/ - Summaries and figures:
reports/ - Experiment log:
logs/experiment_log.md - Large artifact manifest:
large_artifacts_manifest.md
Experiment log 4
Show the running log (4 entries, 2026-06-26)
2026-06-26
- Created standalone experiment package.
- Copied base direct K=4 records, verified algorithm library, and generic evaluator/model utilities into this package.
- Localized experiment identity to
qwen35_4b_independent_retrieval_consensus. - Planned measurements: independent retrieval gate, independent vs same-neighborhood adaptation pools, disagreement-input consensus selector, and direct sample-more baseline on the same residual tasks.
2026-06-26 Retrieval Independence Gate
- Built retrieval plan over 24 base-miss residual tasks and 364 verified library entries.
- Same-neighborhood top-6 mean pairwise code distance: 0.701.
- MMR independent top-6 mean pairwise code distance: 0.803.
- Same-neighborhood top-6 mean pairwise task-token distance: 0.635.
- MMR independent top-6 mean pairwise task-token distance: 0.700.
- Gate passed; proceed to generation.
2026-06-26 Adaptation Pools
- Generated independent top-6 adaptation pool: 24 records, 144 model calls, coverage 7/24, pass1 proxy 3/24, forward tokens 51,282.
- Generated same-neighborhood top-6 adaptation pool: 24 records, 144 model calls, coverage 9/24, pass1 proxy 3/24, forward tokens 52,001.
- Early read: independence increased source diversity but did not increase pool coverage; same-neighborhood control is stronger at the coverage level.
2026-06-26 Direct Sample-More Baseline
- Generated direct K12 baseline on the same 24 residual tasks.
- Direct K12: 288 model calls, coverage 7/24, pass1 proxy 1/24, forward tokens 74,780.
- Same-neighborhood retrieval-adapt remains the strongest pool coverage before consensus selection.
Figures 4
Data files 2
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
Reproduce
Runnable scripts exist in the experiment folder, but the exact invocation was not written down.