Qwen3.5-4B Substrate Coverage Ladder

Finished2026-06-26imported · line ZAlgorithmic Memory and Retrieval Benchmark Generalization Evidence-Conditioned Selection Interpretability and Diagnostics+2 moreGitHub ↗

The one idea you need

Imagine a locksmith with nine stubborn locks. Cutting a fresh key shaped for each one opens all nine, but that only proves the locks are pickable. Reaching for keys already in the drawer, reshaped to fit, opened none, the real test of reuse.

The question

When a coding model gets stuck on hard problems, can a library of reusable code building-blocks, adapted from past solutions, express answers it could not find on its own?

What we found

Only when a building-block was hand-carved for each specific problem. A shared library that reshapes old solutions to fit new ones solved zero of the nine stuck tasks, no better than nothing. Custom-written parts solved all nine, but that only proves the tasks are solvable, not that reuse works. Worse, over half the answers that passed the visible example tests were secretly wrong.

Why it matters

Do not expect a shared library of recycled solutions to expand what a small model can solve on genuinely new problems. And never mine training targets from visible example tests alone: here most answers that passed them were actually wrong.

Stuck problems solved by hand-written parts9 of 9a custom building-block authored for each task

Stuck problems solved by recycling past solutions0 of 9reshape-an-existing-solution approach, beat nothing

Visible-passing answers that were secretly wrong57%17 of 30 failed the hidden checks

Recycled answers that looked right but were wrongall 8every reused answer that passed sample tests failed hidden ones

On this page

Results at a glance
Overview
Report
Experiment log
Figures
Data files
Reproduce
Related

Results at a glance 4

Share of stuck tasks solved, by how the answer was built

How to read

Four bars, one per building method; height is the share of the nine stuck tasks fully solved, taller is better. Expanded custom-built parts and the combined pool reach the top; a smaller custom set reaches under half; recycling past solutions sits flat at the bottom.

Takeaway → The recycle bar is flat at zero while custom-built parts reach full height, so reusing past solutions solved nothing and hand-written parts solved everything.

Data table

substrate rung	hidden coverage
hand-authored core	44.4%
hand-authored expanded	100%
retrieved transplant	0%
combined	100%

Numbers from reports/main_summary.json (rungs.*.coverage); matches README/report table

Technical framing

Hidden-test oracle coverage of 9 residual MBPP tasks by substrate rung — Task-specific hand-authored templates cover all 9 residual tasks; reusing retrieved train-split solutions covers none.

How often answers that looked right on sample tests were secretly wrong

How to read

Four bars, one per building method; height is the share of answers that passed the visible sample tests but failed the full hidden checks, so lower is better.

Takeaway → Every bar is high, from about 41% up to 100%, meaning passing the visible sample tests routinely hides a wrong answer.

Data table

substrate rung	false-pass rate among public-pass candidates
hand-authored core	63.6%
hand-authored expanded	40.9%
retrieved transplant	100%
combined	56.7%

Numbers from reports/main_summary.json (rungs.*.visible_pass_hidden_fail_rate)

Technical framing

Public-test trap: share of public-pass candidates that fail hidden tests — 40-100% of candidates that pass public tests are secretly wrong, so public tests alone cannot supply trustworthy training targets.

Answers that passed sample tests versus those actually correct

How to read

For each building method, the taller bar counts answers passing the visible sample tests and the shorter bar counts those that also passed the full hidden checks; a big gap means many false wins.

Takeaway → For recycled past solutions, 8 answers passed the sample tests but zero survived the full hidden checks.

Data table

substrate rung	passed public tests	also passed hidden tests
hand-authored core	11	4
hand-authored expanded	22	13
retrieved transplant	8	0
combined	30	13

Numbers from reports/main_summary.json (rungs.*.visible_pass_candidates, hidden_pass_candidates)

Technical framing

Public-pass vs hidden-verified candidates per rung — The retrieved-transplant rung produced 8 public-pass candidates and every one failed hidden tests.

How many tries each method needed per task

How to read

Four bars, one per building method; height is the average number of candidate answers generated per task, so lower is cheaper.

Takeaway → Expanded custom-built parts reached full coverage at about 15 tries per task, far cheaper than the pooled roughly 57.

Data table

substrate rung	mean candidates per task
hand-authored core	9.67
hand-authored expanded	14.7
retrieved transplant	42.1
combined	56.8

Numbers from reports/main_summary.json (rungs.*.mean_candidates_per_task)

Technical framing

Candidate cost per residual task by rung — The expanded hand-authored rung reaches full coverage at about 15 candidates per task, far cheaper than the combined pool.

In the author’s words from the Overview · “Summary”

Can an executable kernel/template substrate express MBPP held-out tasks that remained unsolved by a large direct-sampling candidate pool? This is an oracle-ceiling experiment: no model is trained, and hidden tests are used only to measure whether the substrate contains a correct graph/program and how often public-test filtering would produce spurious targets.

Overview

This top-level README was generated during repository normalization because the imported experiment did not include one.

Source track: track-z
Primary report: reports/final_report.md
Metadata: metadata.yaml

How To Read

Start with the primary report, then inspect data/, reports/, analysis/, src/, and scripts/ as available. This folder remains self-contained; do not move its run data into shared directories.

Summary

Report

Rendered from reports/final_report.md

Question

Dataset

Residual tasks: 9 MBPP held-out tasks.
Residual task ids: [16, 26, 31, 39, 43, 44, 48, 60, 77].
Train reference library used for retrieval/transplant controls: 374 train-split entries.
Primary metric: hidden-test oracle coverage on the residual set.

Results

rung	hidden coverage	solved	candidates/task	visible-pass	hidden-pass	visible-pass hidden-fail	false-pass rate
combined	100.0%	9	56.8	30	13	17	56.7%
manual_core	44.4%	4	9.7	11	4	7	63.6%
manual_expanded	100.0%	9	14.7	22	13	9	40.9%
retrieved_transplant	0.0%	0	42.1	8	0	8	100.0%

coverage by rung

visible false passes

candidate cost

task solve heatmap

Per-Task Solve Matrix

task_id	combined	manual_core	manual_expanded	retrieved_transplant	winning templates
16	yes	yes	yes	no	lowercase_underscore_full, lowercase_underscore_one_or_more_groups
26	yes	yes	yes	no	tuple_list_all_k, tuple_list_all_k_set
31	yes	no	yes	no	topk_frequency_minheap_pop_order
39	yes	no	yes	no	rearrange_no_adjacent_heap
43	yes	yes	yes	no	lowercase_underscore_full, lowercase_underscore_one_or_more_groups
44	yes	yes	yes	no	word_at_start, word_at_start_alpha
48	yes	no	yes	no	odd_bit_mask_positions_0_2_4
60	yes	no	yes	no	longest_subseq_adjacent_diff_le_1
77	yes	no	yes	no	alternating_digit_sums_divisible_by_11

Interpretation

The combined substrate solved 9/9 residual tasks (100.0% hidden-test oracle coverage), but this is only a weak expressivity ceiling. The coverage came from the manual_expanded rung (9/9), whose winning templates are task-specific kernels such as top-k frequency, odd-bit masking, no-adjacent string rearrangement, and longest subsequence with adjacent difference. This should not be read as evidence that a reusable substrate generalizes to new residual tasks.

The stricter reusable-substrate readout failed. The retrieved_transplant rung, which retrieves train-split reference solutions and adapts their function signature, solved 0/9 residual tasks. It produced 8 public-test-passing candidates and all 8 were hidden-wrong. That is evidence against the specific reuse hypothesis tested here.

The public-test trap is also visible. The combined arm produced 30 public-test-passing candidates, but 17 of those failed hidden tests, for a false-pass rate of 56.7% among public-pass candidates. This is the load-bearing warning for any follow-up configurator: visible tests alone are not a trustworthy source of perfect latent programs.

Gate Readout

Weak Gate 1, any readable substrate expression: cleared.
Strict Gate 1, reusable substrate expression: failed.
Gate 2, target trust: requires filtering beyond public tests.

Next Step Implied By This Run

Do not train a graph configurator from this result. The meaningful reusable-substrate precondition did not clear, and training on these residual tasks would risk learning a lookup over task-specific hand-authored templates. A follow-up should first test transfer: author substrate templates on one residual slice, then measure hidden-test coverage on a disjoint slice for which no task-specific templates were authored. If that transfer coverage looks like the retrieved-transplant rung, the substrate is not a reusable frontier-expansion mechanism.

If any configurator is later tested, it should not train on arbitrary public-pass substrate candidates. It should either train only from hidden-verified train tasks or include a verifier strong enough to reject the public-pass hidden-fail cases.

Experiment log 3

Show the running log (3 entries)

Purpose

This standalone experiment tests the oracle substrate ceiling before any model training. The question is whether a fixed or mined executable substrate can express held-out MBPP tasks that a large direct sampling pool did not solve. Hidden tests are used only for measurement of oracle coverage and false-pass rate.

Protocol

Build the residual task set from local baseline coverage artifacts.
Prepare a train-split reference library for retrieval/transplant controls.
Run smoke search on two residual tasks.
Run pilot search with the manual substrate.
Run the full ladder: manual core, manual expanded, retrieved transplant, and combined.
Generate figures and a final report.

Running Notes

2026-06-26: Created the experiment package and copied local baseline coverage inputs into data/.
2026-06-26: Prepared the dataset. The K128-residual set contains 9 MBPP held-out tasks: 16, 26, 31, 39, 43, 44, 48, 60, and 77. The retrieval library contains 374 MBPP train-split reference entries.
2026-06-26: Smoke run on two residual tasks passed. Both tasks had hidden-test-correct substrate candidates, and both also had public-test-passing hidden-wrong candidates.
2026-06-26: Manual-only pilot v1 solved 8/9 residual tasks. The miss was the top-k-frequency task, where the substrate had a heap ranking candidate but returned heap-array order instead of heap-pop order.
2026-06-26: Added the generic heap-pop-order top-k-frequency kernel and reran the manual pilot. Manual expanded coverage reached 9/9 on hidden tests.
2026-06-26: Main ladder run completed. Manual core solved 4/9; manual expanded solved 9/9; retrieved train-reference transplants solved 0/9 and added 8 public-test-passing hidden-fail candidates. Combined coverage is 9/9, with 17 visible-pass hidden-fail candidates out of 30 visible-pass candidates.
2026-06-26: Gate interpretation corrected after review. The weak expressivity gate cleared, but the meaningful reusable-substrate gate did not: 9/9 coverage came from task-specific manual templates, while the genuine reuse arm solved 0/9.