Research log Small Model Experimentation
GitHub

Qwen3.5-4B Sketch Coverage Shift Probe

Finished2026-06-24imported · line ZAlgorithmic Memory and RetrievalBenchmark GeneralizationEvidence-Conditioned SelectionInterpretability and Diagnostics+1 moreGitHub ↗
You cannot select a program never built

The one idea you need

Picture a fill-in-the-blank code template, its gaps filled with pieces from a fixed toolbox; every combination runs and anything that passes the tests is kept. If a task needs a tool the toolbox lacks and the template never asks for it, the right program is never assembled.

The question

When a coding task suddenly needs an operation the template-based program builder has never seen, does the correct program still turn up among the candidates it generates?

What we found

Only when the template explicitly names the new operation. On tasks needing a brand-new operation, hand-written templates that named and typed it kept the correct program every single time; auto-generated templates kept it just 17% of the time, and stripped-down templates never. And no amount of extra candidates per blank or follow-up test questions could recover a program the builder never proposed.

Why it matters

Before investing in a smarter candidate-picker or more test queries, make sure your template actually proposes the operations new tasks demand. No selector can rescue a correct program the generation step never produced.

Correct program kept when the template names the new operation100%hand-written template, tasks needing a brand-new operation
Correct program kept when the template is auto-generated17%same brand-new-operation tasks
Correct program kept when the template is stripped to output format only0%same brand-new-operation tasks
Final pick correct after two targeted test questions50% → 100%but only when the correct program was already among the candidates
On this page
  1. Results at a glance
  2. Overview
  3. Report
    1. Summary
    2. Key Findings
    3. Coverage Tables
    4. Active Query Diagnostic
    5. Artifacts
    6. Conclusion
  4. Experiment log
  5. Figures
  6. Reproduce
  7. Related

Results at a glance 4

Does the correct program survive when the task shifts?

How to read

Bars grouped by task type: a familiar task, the same task with a renamed operation, and a task needing a brand-new operation. Within each group, three bars for auto-generated, hand-written, and stripped templates. Bar height is the share of tasks where the correct program stayed a candidate; taller is better.

0%25%50%75%100%control (in bank)control (in bank)100%100%33.3%name shiftname shift100%100%0%primitive shiftprimitive shift16.7%100%0%

Takeaway → Hand-written bars stay at full height across all three task types, while the auto-generated bar collapses to a sliver and the stripped bar to zero once the task needs a brand-new operation.

Data table
task shift typeauto sketchmanual sketcherased sketch
control (in bank)100%100%33.3%
name shift100%100%0%
primitive shift16.7%100%0%

Numbers from experiments/qwen35_4b_sketch_coverage_shift_probe/reports/coverage_by_shift.csv

Technical framing

Does the correct program stay in the candidate set when the task shifts? — Coverage survives shift only when the sketch names and types the shifted operator; auto sketches collapse to 17% on primitive shift.

Trying more options per blank does not rescue auto templates

How to read

On brand-new-operation tasks, the horizontal axis is how many fill-in options each blank may try (8, 16, then 28); the vertical axis is the share of tasks keeping the correct program. Three lines: hand-written, auto-generated, stripped templates. Higher is better.

0%50%100%150%10152025manual sketchauto sketcherased sketch

Takeaway → Only the hand-written line rises to full coverage; the auto line stays flat near one in six and the stripped line at zero, because more options cannot propose an operation the template never names.

Data table
hole options per typed hole (cap)manual sketchauto sketcherased sketch
883.3%16.7%0%
16100%16.7%0%
28100%16.7%0%

Numbers from experiments/qwen35_4b_sketch_coverage_shift_probe/reports/coverage_by_shift.csv

Technical framing

Raising the hole-option cap does not rescue auto sketches on primitive shift — More candidates per hole only helps manual sketches; auto and erased sketches stay flat because the shifted operator is never proposed.

Follow-up test questions help only when the right program exists

How to read

The horizontal axis is how many follow-up test questions are allowed (zero to three); the vertical axis is the share where the finally chosen program passes hidden tests. Two lines had the correct program available, one did not. Higher is better.

0%25%50%75%100%125%0123name shift, auto sketchname shift, auto …primitive shift, manual sketchprimitive shift, …primitive shift, auto sketchprimitive shift, …

Takeaway → The two lines with the correct program present climb to full correctness within two or three questions, while the line missing it stays flat no matter how many questions are asked.

Data table
oracle-elimination query budgetname shift, auto sketchprimitive shift, manual sketchprimitive shift, auto sketch
050%91.7%16.7%
183.3%97.9%16.7%
2100%100%16.7%
3100%100%16.7%

Numbers from experiments/qwen35_4b_sketch_coverage_shift_probe/reports/active_by_shift.csv

Technical framing

Active queries disambiguate candidates but cannot create missing coverage — With coverage, 2-3 oracle queries reach 100%; without coverage (primitive shift, auto sketch) extra queries change nothing.

Which task families keep the correct program, by template style

How to read

A grid: each row is one of twelve task families (familiar, renamed, and brand-new-operation groups); each column is a template style (auto-generated, hand-written, stripped). Cell shade is the share of that family's tasks keeping the correct program; brighter is higher.

control_length_containscontrol_length_contains100%100%0%control_sum_modcontrol_sum_mod100%100%0%control_tuple_mod_gatecontrol_tuple_mod_gate100%100%100%alias_length_containsalias_length_contains100%100%0%alias_sum_modalias_sum_mod100%100%0%alias_tuple_mod_gatealias_tuple_mod_gate100%100%0%primitive_abs_min_deltaprimitive_abs_min_delta0%100%0%primitive_max_gateprimitive_max_gate0%100%0%primitive_max_modprimitive_max_mod0%100%0%primitive_mul_sumprimitive_mul_sum100%100%0%primitive_prefix_gateprimitive_prefix_gate0%100%0%primitive_prod_modprimitive_prod_mod0%100%0%automanualerased0%100%

Takeaway → The hand-written column is fully lit down every row, the auto column goes dark across most brand-new-operation families (one still keeps the program), and the stripped column is dark almost everywhere — the template, not the family, decides survival.

Data table
automanualerased
control_length_contains100%100%0%
control_sum_mod100%100%0%
control_tuple_mod_gate100%100%100%
alias_length_contains100%100%0%
alias_sum_mod100%100%0%
alias_tuple_mod_gate100%100%0%
primitive_abs_min_delta0%100%0%
primitive_max_gate0%100%0%
primitive_max_mod0%100%0%
primitive_mul_sum100%100%0%
primitive_prefix_gate0%100%0%
primitive_prod_mod0%100%0%

Numbers from reports/coverage_by_family.csv

Technical framing

Does the correct program survive each task family and sketch mode? (cap 28) — Share of tasks where the correct program stayed in the candidate set. Manual sketches keep it everywhere; auto sketches lose it on the primitive families; erasing the sketch loses it almost everywhere.

In the author’s words from the Report · “Summary”

This standalone experiment tests whether typed-sketch verified completion keeps the correct executable program in its bounded candidate set when the task substrate shifts. No new adapter is trained. The executor can score shifted primitives, while the completion bank is held fixed for the falsification pass. Each record is evaluated under three sketch conditions: auto: generated by the typed target-sketch function. manual: hand-typed sketch with the intended operator shape and typed holes. erased: low-information sketch that preserves only output format or branch labels.

Overview

Standalone falsification experiment for typed-sketch verified completion under task shift.

The experiment does not train a new adapter. It tests the load-bearing coverage assumption behind typed-sketch synthesis: when the task substrate changes, does the correct executable completion still appear in the bounded candidate set?

Large artifacts, if any are produced later, belong outside this directory:

/workspace/large_artifacts/qwen35_4b_sketch_coverage_shift_probe

Layout

  • configs/: experiment configuration.
  • data/: generated shifted task records and dataset manifest.
  • logs/: chronological experiment log.
  • reports/: metrics, CSVs, figures, and final writeup.
  • run_logs/: captured command output.
  • scripts/: dataset, evaluation, and reporting entry points.
  • src/: standalone executor, typed-sketch completer, and shifted task definitions.

Main Commands

python scripts/build_dataset.py
python scripts/run_coverage_probe.py --data data/shifted_coverage_eval.jsonl --output reports/coverage_probe.json
python scripts/make_report.py

Report

Rendered from reports/qwen35_4b_sketch_coverage_shift_probe_report.md

Summary

This standalone experiment tests whether typed-sketch verified completion keeps the correct executable program in its bounded candidate set when the task substrate shifts. No new adapter is trained. The executor can score shifted primitives, while the completion bank is held fixed for the falsification pass.

Each record is evaluated under three sketch conditions:

  • auto: generated by the typed target-sketch function.
  • manual: hand-typed sketch with the intended operator shape and typed holes.
  • erased: low-information sketch that preserves only output format or branch labels.

Key Findings

  • Control target coverage at cap 28 with auto sketches was 100.0%.
  • Primitive-shift target coverage at cap 28 was 16.7% for auto, 100.0% for manual, and 0.0% for erased.
  • Name-shift target coverage at cap 28 was 100.0% for manual, but only 0.0% for erased.
  • The decisive failure mode is operator support in the sketch: if the sketch explicitly names and types the shifted operator, completion often works; if the operator must be discovered by the bank, coverage collapses.
  • Active querying helps only after coverage exists. It cannot recover a target program that was never synthesized.

Coverage Tables

Primary cap-28 shift summary:

shift_typesketch_moderecordstarget_coverage_pctoracle_hidden_all_pctselected_hidden_all_pctavg_program_count
control_in_bankauto24100.0100.066.7493.2
control_in_bankerased2433.333.333.328.0
control_in_bankmanual24100.0100.066.7493.2
name_shiftauto24100.0100.050.0501.3
name_shifterased240.04.24.228.0
name_shiftmanual24100.0100.050.0501.3
primitive_shiftauto4816.716.716.7698.7
primitive_shifterased480.00.00.028.0
primitive_shiftmanual48100.0100.091.7117.2

Target coverage by shift

Oracle hidden coverage by shift

Selected hidden coverage by shift

Cap sensitivity

Family heatmap

Active Query Diagnostic

Active selection is reported at hole option cap 28. These rows test disambiguation among synthesized candidates; they are not a substitute for coverage.

shift_typesketch_modepolicybudgetrecordsselected_hidden_all_pct
control_in_bankautoactive_max_split02466.7
control_in_bankautoactive_max_split12470.8
control_in_bankautoactive_max_split22475.0
control_in_bankautoactive_max_split32487.5
control_in_bankautooracle_elimination02466.7
control_in_bankautooracle_elimination12491.7
control_in_bankautooracle_elimination22495.8
control_in_bankautooracle_elimination324100.0
control_in_bankmanualactive_max_split02466.7
control_in_bankmanualactive_max_split12470.8
control_in_bankmanualactive_max_split22475.0
control_in_bankmanualactive_max_split32487.5
control_in_bankmanualoracle_elimination02466.7
control_in_bankmanualoracle_elimination12491.7
control_in_bankmanualoracle_elimination22495.8
control_in_bankmanualoracle_elimination324100.0
name_shiftautoactive_max_split02450.0
name_shiftautoactive_max_split12462.5
name_shiftautoactive_max_split22475.0
name_shiftautoactive_max_split32491.7
name_shiftautooracle_elimination02450.0
name_shiftautooracle_elimination12483.3
name_shiftautooracle_elimination224100.0
name_shiftautooracle_elimination324100.0
name_shiftmanualactive_max_split02450.0
name_shiftmanualactive_max_split12462.5
name_shiftmanualactive_max_split22475.0
name_shiftmanualactive_max_split32491.7
name_shiftmanualoracle_elimination02450.0
name_shiftmanualoracle_elimination12483.3
name_shiftmanualoracle_elimination224100.0
name_shiftmanualoracle_elimination324100.0
primitive_shiftautoactive_max_split04816.7
primitive_shiftautoactive_max_split14816.7
primitive_shiftautoactive_max_split24816.7
primitive_shiftautoactive_max_split34816.7
primitive_shiftautooracle_elimination04816.7
primitive_shiftautooracle_elimination14816.7
primitive_shiftautooracle_elimination24816.7
primitive_shiftautooracle_elimination34816.7
primitive_shiftmanualactive_max_split04891.7
primitive_shiftmanualactive_max_split14893.8
primitive_shiftmanualactive_max_split24895.8
primitive_shiftmanualactive_max_split34895.8
primitive_shiftmanualoracle_elimination04891.7
primitive_shiftmanualoracle_elimination14897.9
primitive_shiftmanualoracle_elimination248100.0
primitive_shiftmanualoracle_elimination348100.0

Active query diagnostic

Artifacts

  • Dataset: data/shifted_coverage_eval.jsonl
  • Dataset manifest: data/dataset_manifest.json
  • Full result JSON: reports/coverage_probe.json
  • Coverage CSVs: reports/coverage_by_shift.csv, reports/coverage_by_family.csv
  • Active CSV: reports/active_by_shift.csv
  • Large artifacts: /workspace/large_artifacts/qwen35_4b_sketch_coverage_shift_probe

Conclusion

The coverage assumption does not survive all task shifts. The strongest result is conditional: typed completion can still work on shifted primitives when the sketch names the shifted operator and uses correctly typed holes. The weak result is equally important: erased sketches and automatically mistyped sketches often lose the target completely. The next scaling step should therefore widen and train sketch/operator coverage before investing in a stronger selector.

Experiment log 7

Show the running log (7 entries, 2026-06-24)

2026-06-24 08:20 UTC

Initialized a standalone coverage-shift probe.

Design commitments:

  • Use a local executable substrate with typed sketches and verified completion.
  • Do not train a new Qwen adapter for the first pass.
  • Keep the completion bank intentionally unchanged while extending only the executor to score shifted primitives.

  • Compare three sketch conditions:

    • auto: the existing target-sketch generator.
    • manual: a hand-typed sketch with the intended operator shape and typed holes.
    • erased: a low-information sketch that preserves only output format or branch labels.

  • Compare three task regimes:

    • control_in_bank: operations and names the completion bank is tuned for.
    • name_shift: same operations but renamed variables and constants.
    • primitive_shift: executor-visible primitives that the completion bank was not tuned to propose.
  • Evaluate candidate coverage first; only interpret active-query selection where coverage exists.

Initial implementation:

  • Created standalone experiment directory.
  • Copied local executor and sketch completion mechanics into src/.
  • Extended the executor with mul, abs, max, min, prod, and startswith.
  • Added shifted task definitions with visible, hidden, and query-pool cases.
  • Added dataset builder, README, config, and large-artifact manifest.

Validation:

  • python -m py_compile scripts/build_dataset.py src/*.py
  • Result: passed.

2026-06-24 08:31 UTC

Built the first shifted dataset with 12 records per family.

Command:

python scripts/build_dataset.py > run_logs/dataset_build_console.log 2>&1

Observed:

  • Total records: 144.
  • control_in_bank: 36.
  • name_shift: 36.
  • primitive_shift: 72.
  • Visible cases per record: 6.
  • Hidden cases per record: 18.
  • Query-pool cases per record: 48.

Sketch inspection showed the intended contrast:

  • Control and name-shift auto sketches were typed correctly.
  • Shifted primitives such as max, min, prod, and startswith were mistyped by auto sketches in most families.
  • manual sketches preserved the shifted operator shape and typed holes.
  • erased sketches preserved only output format or branch labels.

2026-06-24 08:38 UTC

Ran evaluator smoke test.

Command:

python scripts/run_coverage_probe.py --data data/shifted_coverage_eval.jsonl --output reports/_smoke_coverage_probe.json --max-records 3 --hole-options 8,16 --active-hole-options 16 --max-programs-per-sketch 500 > run_logs/coverage_smoke_console.log 2>&1

Observed:

  • Smoke completed successfully.
  • Control target coverage was present.
  • Primitive auto/erased failures appeared in the smoke slice.
  • Primitive manual sketches recovered targets in the smoke slice.
  • Active-query rows were emitted for both max-split and oracle-elimination policies.

Decision:

  • Proceed to the full grid, but watch runtime because high-arity sketches can enumerate many completions.

2026-06-24 08:44 UTC

Started a full grid with a 4000-program completion cap, then stopped it after the first minute because high-arity rows were too slow for the information gain.

Stopped command:

python scripts/run_coverage_probe.py --data data/shifted_coverage_eval.jsonl --output reports/coverage_probe.json --hole-options 8,16,28 --active-hole-options 28 --max-programs-per-sketch 4000 > run_logs/coverage_full_console.log 2>&1

Reason:

  • Early tuple-style rows were taking tens of seconds each.
  • The smoke run had already recovered the relevant high-arity targets within 500 completions.
  • The falsification question is coverage under shift, not exhaustive enumeration at a very large cap.

Adjustment:

  • Rebuilt the dataset at 8 records per family to preserve family balance while reducing runtime.
  • Used 1000 completions per sketch for the full run.

Command:

python scripts/build_dataset.py --records-per-family 8 > run_logs/dataset_build_records8_console.log 2>&1

Observed:

  • Total records: 96.
  • control_in_bank: 24.
  • name_shift: 24.
  • primitive_shift: 48.

2026-06-24 08:55 UTC

Completed the full coverage and active-query grid.

Command:

python scripts/run_coverage_probe.py --data data/shifted_coverage_eval.jsonl --output reports/coverage_probe.json --hole-options 8,16,28 --active-hole-options 28 --max-programs-per-sketch 1000 > run_logs/coverage_full_cap1000_console.log 2>&1

Runtime:

  • 864 coverage jobs completed in 9:35.

Primary cap-28 coverage results:

  • control_in_bank, auto: target coverage 24/24, candidate oracle 24/24, visible-selected hidden all-pass 16/24.
  • control_in_bank, manual: target coverage 24/24, candidate oracle 24/24, visible-selected hidden all-pass 16/24.
  • control_in_bank, erased: target coverage 8/24, candidate oracle 8/24, visible-selected hidden all-pass 8/24.
  • name_shift, auto: target coverage 24/24, candidate oracle 24/24, visible-selected hidden all-pass 12/24.
  • name_shift, manual: target coverage 24/24, candidate oracle 24/24, visible-selected hidden all-pass 12/24.
  • name_shift, erased: target coverage 0/24, candidate oracle 1/24, visible-selected hidden all-pass 1/24.
  • primitive_shift, auto: target coverage 8/48, candidate oracle 8/48, visible-selected hidden all-pass 8/48.
  • primitive_shift, manual: target coverage 48/48, candidate oracle 48/48, visible-selected hidden all-pass 44/48.
  • primitive_shift, erased: target coverage 0/48, candidate oracle 0/48, visible-selected hidden all-pass 0/48.

Family-level primitive-shift cap-28 target coverage:

  • primitive_mul_sum: auto 8/8, manual 8/8, erased 0/8.
  • primitive_abs_min_delta: auto 0/8, manual 8/8, erased 0/8.
  • primitive_max_gate: auto 0/8, manual 8/8, erased 0/8.
  • primitive_max_mod: auto 0/8, manual 8/8, erased 0/8.
  • primitive_prefix_gate: auto 0/8, manual 8/8, erased 0/8.
  • primitive_prod_mod: auto 0/8, manual 8/8, erased 0/8.

Active-query cap-28 diagnostic:

  • Control auto/manual: visible-selected 16/24; active max-split reached 21/24 at budget 3; oracle elimination reached 24/24.
  • Name-shift auto/manual: visible-selected 12/24; active max-split reached 22/24 at budget 3; oracle elimination reached 24/24 by budget 2.
  • Primitive-shift manual: visible-selected 44/48; active max-split reached 46/48 at budget 3; oracle elimination reached 48/48 by budget 2.
  • Primitive-shift auto: stayed at 8/48 for every active budget because coverage was absent for five of six families.
  • Primitive-shift erased: stayed at 0/48.

Interpretation:

  • Completion coverage survives name shift when the sketch carries the correct structure.
  • Completion coverage does not survive primitive shift when the sketcher mistypes or omits the shifted operator.
  • If the shifted operator is explicitly present in a correctly typed sketch, the completion bank can usually fill the arguments and recover target coverage.
  • Active querying is useful for disambiguating survivors after coverage exists, but it cannot recover absent programs.

2026-06-24 09:08 UTC

Generated report, CSVs, and figures.

Command:

python scripts/make_report.py > run_logs/report_generation_v2_console.log 2>&1

Artifacts:

2026-06-24 09:14 UTC

Final packaging and audit completed.

Checks:

  • python -m py_compile scripts/*.py src/*.py passed before cache cleanup.
  • Removed generated __pycache__ directories.
  • Verified no stale package-name references in the standalone experiment tree.
  • Verified no file larger than 50M under /workspace/experiments/qwen35_4b_sketch_coverage_shift_probe.
  • Verified experiment package size: 4.1M.
  • Verified large artifact directory size: 0.
  • Verified all 6 generated PNG figures are readable and non-empty.

Final locations:

Figures 6

active query diagnostic
active query diagnostic · reports/figures/
cap sensitivity
cap sensitivity · reports/figures/
family target coverage heatmap
family target coverage heatmap · reports/figures/
oracle hidden by shift
oracle hidden by shift · reports/figures/
selected hidden by shift
selected hidden by shift · reports/figures/
target coverage by shift
target coverage by shift · reports/figures/

Reproduce

Run steps are documented inside the experiment folder (README and scripts).

Browse the experiment folder on GitHub ↗