Qwen 3.5 4B Static Bridge Ceiling Breaker

Finishedimported 2026-07-12imported · line ZBenchmark Generalization Evidence-Conditioned Selection Structured Execution and Compilers GitHub ↗

The one idea you need

Picture the model as a code mechanic handed a broken tiny program plus a printout that, for each failing test, shows the answer it should have produced beside the wrong one it actually did. Its job: rewrite the program so every hidden test passes.

The question

If you mix a small batch of slightly-harder practice problems into a small model's training, does it learn a repair skill that carries to entirely new, deeper problems?

What we found

Partly. Folding in just 60 slightly-harder "bridge" examples, a quarter of the training budget, more than doubled success on deeper, never-seen programs, from 20% to 44% fully repaired, with no loss on familiar skills. But it stalled fast: 80 examples did no better. And the skill is genuine reading, not memorization: feed a scrambled test printout and success sinks below giving no printout at all.

Why it matters

When teaching a small model to repair code, add a thin slice of slightly-harder examples and always show real execution results in the prompt. Expect a hard ceiling, and confirm it truly reads that feedback by scrambling the results and watching accuracy collapse.

Fixing deeper, never-seen programs20% → 44%share fully repaired after adding 60 bridge examples, up from none

Bridge examples added60 of 240a quarter of a fixed training budget

More examples, no payoff44% vs 41%60 versus 80 bridge examples on deeper programs

Correct vs scrambled printout38% → 7%a shuffled test printout hurts more than none at all

On this page

Results at a glance
Overview
Report
Experiment log
Figures
Reproduce
Related

Results at a glance 3

A few harder examples lift both trained and never-seen program types

How to read

Bars grouped by training mix: no bridge examples, 60 added, 80 added. One color is the familiar trained program families, the other the deeper unseen families never seen in training. Height is the share of programs whose fix passes every hidden test; taller is better.

Takeaway → Trained families jump to a perfect 100% and hold, while unseen families roughly double from 20% to 44% then flatten, so transfer is real but capped.

Data table

training condition	support families (trained)	ceiling families (held out)
seed (no bridge data)	53.3%	20%
static bridge 60	100%	44.2%
static bridge 80	100%	40.8%

Numbers from README table; reports/qwen35_4b_static_bridge_ceiling_breaker_report.md tables

Technical framing

Bridge training transfers to held-out ceiling families, but only partially — Adding 60 static bridge records saturates support families (120/120) and roughly doubles held-out ceiling transfer (24 to 53 of 120).

Fixing new programs collapses when the test printout is wrong or missing

How to read

Three bars, the same 60-bridge model on the same unseen programs, differing only in the test printout shown: correct, removed, or scrambled. Height is the share of programs whose fix passes every hidden test; taller means it leaned on that printout more.

Takeaway → The correct-printout bar towers over the rest, and the scrambled bar sinks below the removed one, proving the model genuinely reads the feedback rather than guessing.

Data table

prompt condition	static 60, ceiling greedy
aligned trace	38.3%
no trace	15%
shuffled trace	6.7%

Numbers from reports/qwen35_4b_static_bridge_ceiling_breaker_report.md, Ceiling Trace Controls table

Technical framing

Ceiling gains depend on a correct execution trace (static 60 controls) — Removing the trace drops accuracy to 15% and a shuffled trace is worse than none, so the model is reading the trace, not pattern-matching.

Bridge training helps some new families fully, others not at all

How to read

Six unseen program families along the bottom. Paired bars compare the no-bridge model against the 60-bridge model. Height is the share of programs whose fix passes every hidden test; taller is better.

Takeaway → Several families leap from zero to near-perfect, one stays frozen at zero, and one regresses, so the benefit is family-specific, never a uniform lift.

Data table

held-out ceiling family	seed (no bridge data)	static bridge 60
text_absent_mod	0%	100%
token_absent_length	0%	91.7%
token_count_mod_length	41.7%	83.3%
text_value_gate	83.3%	100%
tuple_value_mod	50%	8.3%
sorted_join_contains	0%	0%

Numbers from reports/qwen35_4b_static_bridge_ceiling_breaker_report.md, Ceiling By Family table

Technical framing

Ceiling transfer is bimodal across held-out families — Some families jump from 0% to near-perfect, others stay at zero or regress, so bridge transfer is family-specific rather than uniform.

In the author’s words from the Overview

This standalone experiment tests whether static bridge posttraining learns a trace-conditioned executable repair interface that transfers from support bridge families to deeper held-out composition families. The model receives: an input schema, a current wrong DSL program, visible execution cases with expected and got values, and must output one corrected executable DSL expression. The trained conditions are: seed_lora: 240 base-family random-trace records. static60_lora: 180 base-family records plus 60 equal support bridge records. static80_lora: 160 base-family records plus 80 equal support bridge records. The main evaluation split is dsl_eval_ceiling.jsonl, whose families are absent from bridge training. … Read the full result →

Overview

The model receives:

an input schema,
a current wrong DSL program,
visible execution cases with expected and got values,
and must output one corrected executable DSL expression.

The trained conditions are:

seed_lora: 240 base-family random-trace records.
static60_lora: 180 base-family records plus 60 equal support bridge records.
static80_lora: 160 base-family records plus 80 equal support bridge records.

The main evaluation split is dsl_eval_ceiling.jsonl, whose families are absent from bridge training.

Layout

configs/experiment.json: fixed design and hyperparameters.
src/: standalone DSL, data, prompt, and model utilities.
scripts/: dataset generation, training, evaluation, and report entry points.
data/: generated JSONL datasets and manifest.
reports/: evaluation JSON files and final report.
figures/: generated charts.
logs/ and run_logs/: experiment notebook and command output.
large_artifacts_manifest.md: pointers to adapter directories stored outside this compact directory.

Report

Final report:

reports/qwen35_4b_static_bridge_ceiling_breaker_report.md

Key results:

Support reranked hidden all-pass: seed 64/120, Static60 120/120, Static80 120/120.
Ceiling reranked hidden all-pass: seed 24/120, Static60 53/120, Static80 49/120.
IID retention: all three adapters 60/60.
Static60 trace controls on ceiling, greedy hidden: aligned 46/120, no trace 18/120, shuffled trace 8/120.

Large adapters and checkpoints are intentionally outside this directory:

/workspace/large_artifacts/qwen35_4b_static_bridge_ceiling_breaker/

Report

Rendered from reports/qwen35_4b_static_bridge_ceiling_breaker_report.md

Question

Can fixed-budget static bridge posttraining learn a trace-conditioned repair interface that transfers from support bridge families to deeper held-out composition families?

Design

Base model: Qwen/Qwen3.5-4B.
Model output: one executable DSL expression.
Training: 4-bit NF4 QLoRA adapters.
Candidate selection: choose the valid candidate with the most visible-case passes.
Large adapter/checkpoint files are stored outside the compact experiment directory.
Seed adapter: 240 base-family random-trace records.
Static 60 adapter: 180 base-family records plus 60 equal support bridge records.
Static 80 adapter: 160 base-family records plus 80 equal support bridge records.
Main test: held-out ceiling families absent from bridge training.

Dataset

Seed train records: 240.
Static 60 train records: 240 (180 base + 60 bridge).
Static 80 train records: 240 (160 base + 80 bridge).
IID eval records: 60.
Support eval records: 120.
Ceiling eval records: 120.
Visible cases per record: 6.
Hidden cases per record: 18.
Support bridge families: 10.
Ceiling families: 10.
Static 60 selector summary: {'avg_eliminated_wrong_programs': 4.1, 'avg_remaining_wrong_programs': 0.0, 'records': 60}.
Static 80 selector summary: {'avg_eliminated_wrong_programs': 4.1, 'avg_remaining_wrong_programs': 0.0, 'records': 80}.

Support Split Results

Condition	Data	Prompt	Samples	Greedy Hidden	Rerank Hidden
Seed, support	`dsl_eval_support.jsonl`	`trace`	3	50.8% (61/120)	53.3% (64/120)
Static 60, support	`dsl_eval_support.jsonl`	`trace`	3	99.2% (119/120)	100.0% (120/120)
Static 80, support	`dsl_eval_support.jsonl`	`trace`	3	100.0% (120/120)	100.0% (120/120)

Ceiling Split Results

Condition	Data	Prompt	Samples	Greedy Hidden	Rerank Hidden
Seed, ceiling	`dsl_eval_ceiling.jsonl`	`trace`	3	12.5% (15/120)	20.0% (24/120)
Static 60, ceiling	`dsl_eval_ceiling.jsonl`	`trace`	3	38.3% (46/120)	44.2% (53/120)
Static 80, ceiling	`dsl_eval_ceiling.jsonl`	`trace`	3	40.0% (48/120)	40.8% (49/120)

Ceiling Trace Controls

Condition	Data	Prompt	Samples	Greedy Hidden	Rerank Hidden
Static 60, no trace ceiling	`dsl_eval_ceiling.jsonl`	`no_trace`	0	15.0% (18/120)	15.0% (18/120)
Static 60, shuffled trace ceiling	`dsl_eval_ceiling.jsonl`	`shuffled_trace`	0	6.7% (8/120)	6.7% (8/120)

IID Retention Results

Condition	Data	Prompt	Greedy Hidden	Rerank Hidden
Seed, IID	`dsl_eval_iid.jsonl`	`trace`	100.0% (60/60)	100.0% (60/60)
Static 60, IID	`dsl_eval_iid.jsonl`	`trace`	100.0% (60/60)	100.0% (60/60)
Static 80, IID	`dsl_eval_iid.jsonl`	`trace`	100.0% (60/60)	100.0% (60/60)

Ceiling By Family

Family	Seed, ceiling	Static 60, ceiling	Static 80, ceiling
`sum_length_mod_gate_label`	0.0% (0/12)	8.3% (1/12)	0.0% (0/12)
`sorted_index_sum_branch_label`	0.0% (0/12)	0.0% (0/12)	0.0% (0/12)
`token_absent_length_code`	0.0% (0/12)	91.7% (11/12)	58.3% (7/12)
`token_count_mod_length_code`	41.7% (5/12)	83.3% (10/12)	58.3% (7/12)
`text_value_gate_label`	83.3% (10/12)	100.0% (12/12)	100.0% (12/12)
`tuple_value_mod_label`	50.0% (6/12)	8.3% (1/12)	50.0% (6/12)
`sorted_join_contains_code`	0.0% (0/12)	0.0% (0/12)	0.0% (0/12)
`text_absent_mod_code`	0.0% (0/12)	100.0% (12/12)	100.0% (12/12)
`sum_len_mod_label`	0.0% (0/12)	8.3% (1/12)	0.0% (0/12)
`tuple_sum_mod_gate_label`	25.0% (3/12)	41.7% (5/12)	41.7% (5/12)

Support By Family

Family	Seed, support	Static 60, support	Static 80, support
`modulo_sum_label`	100.0% (12/12)	100.0% (12/12)	100.0% (12/12)
`length_contains_code`	66.7% (8/12)	100.0% (12/12)	100.0% (12/12)
`tuple_branch_label`	66.7% (8/12)	100.0% (12/12)	100.0% (12/12)
`sum_offset_mod_label`	0.0% (0/12)	100.0% (12/12)	100.0% (12/12)
`length_mod_contains_code`	66.7% (8/12)	100.0% (12/12)	100.0% (12/12)
`sum_length_branch_label`	100.0% (12/12)	100.0% (12/12)	100.0% (12/12)
`sorted_index_offset_label`	0.0% (0/12)	100.0% (12/12)	100.0% (12/12)
`contains_count_length_code`	58.3% (7/12)	100.0% (12/12)	100.0% (12/12)
`tuple_sum_gate_label`	75.0% (9/12)	100.0% (12/12)	100.0% (12/12)
`not_contains_length_code`	0.0% (0/12)	100.0% (12/12)	100.0% (12/12)

Readout

Ceiling reranked hidden all-pass: seed 20.0% (24/120), static 60 44.2% (53/120), static 80 40.8% (49/120).
Ceiling greedy hidden all-pass: seed 12.5% (15/120), static 60 38.3% (46/120), static 80 40.0% (48/120).
Static 60, ceiling trace controls: aligned 38.3% (46/120), no trace 15.0% (18/120), shuffled trace 6.7% (8/120).

Figures

Failure Signatures

sum_length_mod_gate_label seed greedy top: 6x (if (and (gt (sum values) 0) (eq (mod (len text) modulus) target)) high_label low_label); 6x (if (and (gt (sum values) 0) (eq (mod (sum values) modulus) target)) high_label low_label); static 60 greedy top: 6x (if (and (contains text "a") (eq (mod (sum values) modulus) target)) high_label low_label); 3x (if (and (contains text "e") (eq (mod (sum values) modulus) target)) high_label low_label); static 80 greedy top: 5x (if (and (gt (sum values) 0) (eq (mod (len text) modulus) target)) high_label low_label); 2x (if (and (contains text "a") (gt (sum values) 0)) high_label low_label).
sorted_index_sum_branch_label seed greedy top: 3x (if (and (gt (tuple_get values index) threshold) (gt (sum values) 0)) high_label low_label); 2x (if (and (gt (sum values) threshold) (eq index 0)) high_label low_label); static 60 greedy top: 7x (if (gt (tuple_get (sort values) index) threshold) high_label low_label); 5x (if (gt (tuple_get values index) threshold) high_label low_label); static 80 greedy top: 7x (if (gt (tuple_get (sort values) index) threshold) high_label low_label); 4x (if (gt (tuple_get values index) threshold) high_label low_label).
token_absent_length_code seed greedy top: 3x (if (and (contains tokens needle) (gt (count_eq tokens needle) min_len)) "FOUND" "OTHER"); 3x (if (and (contains tokens needle) (gt (len tokens) min_len)) "ABSENT_LONG" "OTHER"); static 60 greedy top: 9x (if (and (not (contains tokens needle)) (gt (len tokens) min_len)) "ABSENT_LONG" "OTHER"); 2x (if (and (contains tokens needle) (gt (len tokens) min_len)) "ABSENT_LONG" "OTHER"); static 80 greedy top: 6x (if (and (not (contains tokens needle)) (gt (len tokens) min_len)) "ABSENT_LONG" "OTHER"); 5x (if (and (not (contains tokens needle)) (gt (count_eq tokens needle) min_len)) "ABSENT_LONG" "OTHER").
token_count_mod_length_code seed greedy top: 5x (if (and (contains tokens needle) (gt (len tokens) min_len) (and (eq (mod (count_eq tokens needle) modulus) target))) "COUNT_MOD_LONG" "MISS"); 2x (if (and (contains tokens needle) (gt (len tokens) min_len) (and (gt (mod (len tokens) modulus) 0) (gt target 0))) "COUNT_MOD_LONG" "MISS"); static 60 greedy top: 5x (if (and (contains tokens needle) (eq (mod (count_eq tokens needle) modulus) target) (gt (len tokens) min_len)) "COUNT_MOD_LONG" "MISS"); 4x (if (and (contains tokens needle) (gt (len tokens) min_len) (eq (mod (count_eq tokens needle) modulus) target)) "COUNT_MOD_LONG" "MISS"); static 80 greedy top: 6x (if (and (contains tokens needle) (gt (len tokens) min_len) (eq (mod (count_eq tokens needle) modulus) target)) "COUNT_MOD_LONG" "MISS"); 6x (if (and (contains tokens needle) (gt (len tokens) min_len) (eq (mod (len tokens) modulus) target)) "COUNT_MOD_LONG" "MISS").
text_value_gate_label seed greedy top: 10x (if (and (contains text needle) (gt (sum values) threshold) (gt (len text) min_len)) high_label low_label); 1x (if (and (gt (len text) min_len) (contains text needle)) (gt (sum values) threshold) high_label low_label); static 60 greedy top: 8x (if (and (contains text needle) (gt (sum values) threshold) (gt (len text) min_len)) high_label low_label); 4x (if (and (contains text needle) (gt (len text) min_len) (gt (sum values) threshold)) high_label low_label); static 80 greedy top: 12x (if (and (contains text needle) (gt (sum values) threshold) (gt (len text) min_len)) high_label low_label).
tuple_value_mod_label seed greedy top: 4x (format "TV{}" (add (tuple_get item index) (mod (sum values) modulus))); 2x (format "TV{}" (tuple_get item index)); static 60 greedy top: 9x (format "TV{}" (mod (add (tuple_get item index) (sum values)) modulus)); 2x (format "TV{}" (mod (add (tuple_get item index) (first values)) modulus)); static 80 greedy top: 6x (format "TV{}" (add (tuple_get item index) (mod (sum values) modulus))); 4x (format "TV{}" (mod (add (tuple_get item index) (sum values)) modulus)).
sorted_join_contains_code seed greedy top: 4x (if (and tokens (contains tokens needle)) "JOIN_HAS" "JOIN_MISS"); 2x (if (contains (join "" tokens) needle) "JOIN_HAS" "JOIN_MISS"); static 60 greedy top: 6x (if (contains (join "" tokens) needle) "JOIN_HAS" "JOIN_MISS"); 5x (if (contains (sort tokens) needle) "JOIN_HAS" "JOIN_MISS"); static 80 greedy top: 5x (if (and (contains tokens needle) (count_eq tokens needle)) "JOIN_HAS" "JOIN_MISS"); 3x (if (and (contains (join "" tokens) needle) (eq (count_eq tokens needle) 0)) "JOIN_HAS" "JOIN_MISS").
text_absent_mod_code seed greedy top: 5x (if (and (contains text needle) (eq (mod (len text) modulus) target)) "FOUND" "OTHER"); 3x (if (and (contains text needle) (eq (mod (len text) modulus) target)) "FOUND" "ABSENT_MOD"); static 60 greedy top: 12x (if (and (not (contains text needle)) (eq (mod (len text) modulus) target)) "ABSENT_MOD" "OTHER"); static 80 greedy top: 12x (if (and (not (contains text needle)) (eq (mod (len text) modulus) target)) "ABSENT_MOD" "OTHER").
sum_len_mod_label seed greedy top: 6x (format "SL{}" (mod (sum values) modulus)); 5x (format "SL{}" (add (sum values) (mod (len text) modulus))); static 60 greedy top: 7x (format "SL{}" (mod (sum values) modulus)); 4x (format "SL{}" (mod (len text) (mod (sum values) modulus))); static 80 greedy top: 12x (format "SL{}" (mod (sum values) modulus)).
tuple_sum_mod_gate_label seed greedy top: 5x (if (and (gt (get item index) threshold) (eq (mod (sum item) modulus) target)) high_label low_label); 5x (if (and (gt (tuple_get item index) threshold) (eq (mod (tuple_get item index) modulus) target)) high_label low_label); static 60 greedy top: 8x (if (and (gt (tuple_get item index) threshold) (eq (mod (tuple_get item index) modulus) target)) high_label low_label); 4x (if (and (gt (tuple_get item index) threshold) (eq (mod (sum item) modulus) target)) high_label low_label); static 80 greedy top: 7x (if (and (gt (tuple_get item index) threshold) (eq (mod (tuple_get item index) modulus) target)) high_label low_label); 5x (if (and (gt (tuple_get item index) threshold) (eq (mod (sum item) modulus) target)) high_label low_label).

Per-Condition Details

seed_lora_ceiling

Adapter: /workspace/large_artifacts/qwen35_4b_static_bridge_ceiling_breaker/models/seed_lora.
Data: data/eval/dsl_eval_ceiling.jsonl.
Prompt mode: trace.
Samples: 3.
Greedy hidden all-pass: 12.5% (15/120).
Rerank hidden all-pass: 20.0% (24/120).

Family	Greedy Hidden	Rerank Hidden	Greedy Visible	Rerank Visible
sorted_index_sum_branch_label	0.0% (0/12)	0.0% (0/12)	0.0% (0/12)	0.0% (0/12)
sorted_join_contains_code	0.0% (0/12)	0.0% (0/12)	16.7% (2/12)	16.7% (2/12)
sum_len_mod_label	0.0% (0/12)	0.0% (0/12)	0.0% (0/12)	0.0% (0/12)
sum_length_mod_gate_label	0.0% (0/12)	0.0% (0/12)	16.7% (2/12)	16.7% (2/12)
text_absent_mod_code	0.0% (0/12)	0.0% (0/12)	0.0% (0/12)	0.0% (0/12)
text_value_gate_label	83.3% (10/12)	83.3% (10/12)	91.7% (11/12)	100.0% (12/12)
token_absent_length_code	0.0% (0/12)	0.0% (0/12)	16.7% (2/12)	16.7% (2/12)
token_count_mod_length_code	8.3% (1/12)	41.7% (5/12)	0.0% (0/12)	41.7% (5/12)
tuple_sum_mod_gate_label	0.0% (0/12)	25.0% (3/12)	8.3% (1/12)	33.3% (4/12)
tuple_value_mod_label	33.3% (4/12)	50.0% (6/12)	33.3% (4/12)	50.0% (6/12)

seed_lora_iid

Adapter: /workspace/large_artifacts/qwen35_4b_static_bridge_ceiling_breaker/models/seed_lora.
Data: data/eval/dsl_eval_iid.jsonl.
Prompt mode: trace.
Samples: 0.
Greedy hidden all-pass: 100.0% (60/60).
Rerank hidden all-pass: 100.0% (60/60).

Family	Greedy Hidden	Rerank Hidden	Greedy Visible	Rerank Visible
contains_and_count_code	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)
contains_code	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)
contains_count_label	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)
length_and_mod_code	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)
length_label	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)
length_mod_label	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)
mod_scalar_label	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)
scalar_branch_label	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)
sorted_first_label	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)
sum_add_label	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)
sum_and_scalar_code	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)
sum_label	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)
sum_threshold_label	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)
tuple_get_label	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)
tuple_sum_label	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)

seed_lora_support

Adapter: /workspace/large_artifacts/qwen35_4b_static_bridge_ceiling_breaker/models/seed_lora.
Data: data/eval/dsl_eval_support.jsonl.
Prompt mode: trace.
Samples: 3.
Greedy hidden all-pass: 50.8% (61/120).
Rerank hidden all-pass: 53.3% (64/120).

Family	Greedy Hidden	Rerank Hidden	Greedy Visible	Rerank Visible
contains_count_length_code	41.7% (5/12)	58.3% (7/12)	58.3% (7/12)	75.0% (9/12)
length_contains_code	66.7% (8/12)	66.7% (8/12)	75.0% (9/12)	75.0% (9/12)
length_mod_contains_code	66.7% (8/12)	66.7% (8/12)	75.0% (9/12)	75.0% (9/12)
modulo_sum_label	100.0% (12/12)	100.0% (12/12)	100.0% (12/12)	100.0% (12/12)
not_contains_length_code	0.0% (0/12)	0.0% (0/12)	16.7% (2/12)	16.7% (2/12)
sorted_index_offset_label	0.0% (0/12)	0.0% (0/12)	0.0% (0/12)	0.0% (0/12)
sum_length_branch_label	100.0% (12/12)	100.0% (12/12)	100.0% (12/12)	100.0% (12/12)
sum_offset_mod_label	0.0% (0/12)	0.0% (0/12)	0.0% (0/12)	0.0% (0/12)
tuple_branch_label	66.7% (8/12)	66.7% (8/12)	66.7% (8/12)	66.7% (8/12)
tuple_sum_gate_label	66.7% (8/12)	75.0% (9/12)	66.7% (8/12)	75.0% (9/12)

static60_lora_ceiling

Adapter: /workspace/large_artifacts/qwen35_4b_static_bridge_ceiling_breaker/models/static60_lora.
Data: data/eval/dsl_eval_ceiling.jsonl.
Prompt mode: trace.
Samples: 3.
Greedy hidden all-pass: 38.3% (46/120).
Rerank hidden all-pass: 44.2% (53/120).

Family	Greedy Hidden	Rerank Hidden	Greedy Visible	Rerank Visible
sorted_index_sum_branch_label	0.0% (0/12)	0.0% (0/12)	8.3% (1/12)	8.3% (1/12)
sorted_join_contains_code	0.0% (0/12)	0.0% (0/12)	0.0% (0/12)	0.0% (0/12)
sum_len_mod_label	0.0% (0/12)	8.3% (1/12)	0.0% (0/12)	8.3% (1/12)
sum_length_mod_gate_label	0.0% (0/12)	8.3% (1/12)	16.7% (2/12)	25.0% (3/12)
text_absent_mod_code	100.0% (12/12)	100.0% (12/12)	100.0% (12/12)	100.0% (12/12)
text_value_gate_label	100.0% (12/12)	100.0% (12/12)	100.0% (12/12)	100.0% (12/12)
token_absent_length_code	75.0% (9/12)	91.7% (11/12)	83.3% (10/12)	100.0% (12/12)
token_count_mod_length_code	75.0% (9/12)	83.3% (10/12)	75.0% (9/12)	91.7% (11/12)
tuple_sum_mod_gate_label	33.3% (4/12)	41.7% (5/12)	41.7% (5/12)	50.0% (6/12)
tuple_value_mod_label	0.0% (0/12)	8.3% (1/12)	0.0% (0/12)	8.3% (1/12)

static60_lora_iid

Adapter: /workspace/large_artifacts/qwen35_4b_static_bridge_ceiling_breaker/models/static60_lora.
Data: data/eval/dsl_eval_iid.jsonl.
Prompt mode: trace.
Samples: 0.
Greedy hidden all-pass: 100.0% (60/60).
Rerank hidden all-pass: 100.0% (60/60).

Family	Greedy Hidden	Rerank Hidden	Greedy Visible	Rerank Visible
contains_and_count_code	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)
contains_code	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)
contains_count_label	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)
length_and_mod_code	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)
length_label	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)
length_mod_label	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)
mod_scalar_label	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)
scalar_branch_label	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)
sorted_first_label	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)
sum_add_label	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)
sum_and_scalar_code	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)
sum_label	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)
sum_threshold_label	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)
tuple_get_label	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)
tuple_sum_label	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)

static60_lora_no_trace_ceiling

Adapter: /workspace/large_artifacts/qwen35_4b_static_bridge_ceiling_breaker/models/static60_lora.
Data: data/eval/dsl_eval_ceiling.jsonl.
Prompt mode: no_trace.
Samples: 0.
Greedy hidden all-pass: 15.0% (18/120).
Rerank hidden all-pass: 15.0% (18/120).

Family	Greedy Hidden	Rerank Hidden	Greedy Visible	Rerank Visible
sorted_index_sum_branch_label	0.0% (0/12)	0.0% (0/12)	8.3% (1/12)	8.3% (1/12)
sorted_join_contains_code	0.0% (0/12)	0.0% (0/12)	0.0% (0/12)	0.0% (0/12)
sum_len_mod_label	0.0% (0/12)	0.0% (0/12)	0.0% (0/12)	0.0% (0/12)
sum_length_mod_gate_label	0.0% (0/12)	0.0% (0/12)	16.7% (2/12)	16.7% (2/12)
text_absent_mod_code	0.0% (0/12)	0.0% (0/12)	0.0% (0/12)	0.0% (0/12)
text_value_gate_label	41.7% (5/12)	41.7% (5/12)	66.7% (8/12)	66.7% (8/12)
token_absent_length_code	0.0% (0/12)	0.0% (0/12)	8.3% (1/12)	8.3% (1/12)
token_count_mod_length_code	58.3% (7/12)	58.3% (7/12)	58.3% (7/12)	58.3% (7/12)
tuple_sum_mod_gate_label	0.0% (0/12)	0.0% (0/12)	8.3% (1/12)	8.3% (1/12)
tuple_value_mod_label	50.0% (6/12)	50.0% (6/12)	50.0% (6/12)	50.0% (6/12)

static60_lora_shuffled_trace_ceiling

Adapter: /workspace/large_artifacts/qwen35_4b_static_bridge_ceiling_breaker/models/static60_lora.
Data: data/eval/dsl_eval_ceiling.jsonl.
Prompt mode: shuffled_trace.
Samples: 0.
Greedy hidden all-pass: 6.7% (8/120).
Rerank hidden all-pass: 6.7% (8/120).

Family	Greedy Hidden	Rerank Hidden	Greedy Visible	Rerank Visible
sorted_index_sum_branch_label	0.0% (0/12)	0.0% (0/12)	8.3% (1/12)	8.3% (1/12)
sorted_join_contains_code	0.0% (0/12)	0.0% (0/12)	0.0% (0/12)	0.0% (0/12)
sum_len_mod_label	0.0% (0/12)	0.0% (0/12)	0.0% (0/12)	0.0% (0/12)
sum_length_mod_gate_label	0.0% (0/12)	0.0% (0/12)	0.0% (0/12)	0.0% (0/12)
text_absent_mod_code	16.7% (2/12)	16.7% (2/12)	16.7% (2/12)	16.7% (2/12)
text_value_gate_label	25.0% (3/12)	25.0% (3/12)	25.0% (3/12)	25.0% (3/12)
token_absent_length_code	8.3% (1/12)	8.3% (1/12)	8.3% (1/12)	8.3% (1/12)
token_count_mod_length_code	16.7% (2/12)	16.7% (2/12)	16.7% (2/12)	16.7% (2/12)
tuple_sum_mod_gate_label	0.0% (0/12)	0.0% (0/12)	8.3% (1/12)	8.3% (1/12)
tuple_value_mod_label	0.0% (0/12)	0.0% (0/12)	0.0% (0/12)	0.0% (0/12)

static60_lora_support

Adapter: /workspace/large_artifacts/qwen35_4b_static_bridge_ceiling_breaker/models/static60_lora.
Data: data/eval/dsl_eval_support.jsonl.
Prompt mode: trace.
Samples: 3.
Greedy hidden all-pass: 99.2% (119/120).
Rerank hidden all-pass: 100.0% (120/120).

Family	Greedy Hidden	Rerank Hidden	Greedy Visible	Rerank Visible
contains_count_length_code	100.0% (12/12)	100.0% (12/12)	100.0% (12/12)	100.0% (12/12)
length_contains_code	100.0% (12/12)	100.0% (12/12)	100.0% (12/12)	100.0% (12/12)
length_mod_contains_code	100.0% (12/12)	100.0% (12/12)	100.0% (12/12)	100.0% (12/12)
modulo_sum_label	100.0% (12/12)	100.0% (12/12)	100.0% (12/12)	100.0% (12/12)
not_contains_length_code	91.7% (11/12)	100.0% (12/12)	91.7% (11/12)	100.0% (12/12)
sorted_index_offset_label	100.0% (12/12)	100.0% (12/12)	100.0% (12/12)	100.0% (12/12)
sum_length_branch_label	100.0% (12/12)	100.0% (12/12)	100.0% (12/12)	100.0% (12/12)
sum_offset_mod_label	100.0% (12/12)	100.0% (12/12)	100.0% (12/12)	100.0% (12/12)
tuple_branch_label	100.0% (12/12)	100.0% (12/12)	100.0% (12/12)	100.0% (12/12)
tuple_sum_gate_label	100.0% (12/12)	100.0% (12/12)	100.0% (12/12)	100.0% (12/12)

static80_lora_ceiling

Adapter: /workspace/large_artifacts/qwen35_4b_static_bridge_ceiling_breaker/models/static80_lora.
Data: data/eval/dsl_eval_ceiling.jsonl.
Prompt mode: trace.
Samples: 3.
Greedy hidden all-pass: 40.0% (48/120).
Rerank hidden all-pass: 40.8% (49/120).

Family	Greedy Hidden	Rerank Hidden	Greedy Visible	Rerank Visible
sorted_index_sum_branch_label	0.0% (0/12)	0.0% (0/12)	8.3% (1/12)	8.3% (1/12)
sorted_join_contains_code	0.0% (0/12)	0.0% (0/12)	0.0% (0/12)	8.3% (1/12)
sum_len_mod_label	0.0% (0/12)	0.0% (0/12)	0.0% (0/12)	0.0% (0/12)
sum_length_mod_gate_label	0.0% (0/12)	0.0% (0/12)	8.3% (1/12)	25.0% (3/12)
text_absent_mod_code	100.0% (12/12)	100.0% (12/12)	100.0% (12/12)	100.0% (12/12)
text_value_gate_label	100.0% (12/12)	100.0% (12/12)	100.0% (12/12)	100.0% (12/12)
token_absent_length_code	50.0% (6/12)	58.3% (7/12)	66.7% (8/12)	75.0% (9/12)
token_count_mod_length_code	58.3% (7/12)	58.3% (7/12)	58.3% (7/12)	58.3% (7/12)
tuple_sum_mod_gate_label	41.7% (5/12)	41.7% (5/12)	50.0% (6/12)	50.0% (6/12)
tuple_value_mod_label	50.0% (6/12)	50.0% (6/12)	50.0% (6/12)	50.0% (6/12)

static80_lora_iid

Adapter: /workspace/large_artifacts/qwen35_4b_static_bridge_ceiling_breaker/models/static80_lora.
Data: data/eval/dsl_eval_iid.jsonl.
Prompt mode: trace.
Samples: 0.
Greedy hidden all-pass: 100.0% (60/60).
Rerank hidden all-pass: 100.0% (60/60).

Family	Greedy Hidden	Rerank Hidden	Greedy Visible	Rerank Visible
contains_and_count_code	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)
contains_code	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)
contains_count_label	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)
length_and_mod_code	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)
length_label	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)
length_mod_label	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)
mod_scalar_label	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)
scalar_branch_label	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)
sorted_first_label	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)
sum_add_label	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)
sum_and_scalar_code	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)
sum_label	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)
sum_threshold_label	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)
tuple_get_label	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)
tuple_sum_label	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)	100.0% (4/4)

static80_lora_support

Adapter: /workspace/large_artifacts/qwen35_4b_static_bridge_ceiling_breaker/models/static80_lora.
Data: data/eval/dsl_eval_support.jsonl.
Prompt mode: trace.
Samples: 3.
Greedy hidden all-pass: 100.0% (120/120).
Rerank hidden all-pass: 100.0% (120/120).

Family	Greedy Hidden	Rerank Hidden	Greedy Visible	Rerank Visible
contains_count_length_code	100.0% (12/12)	100.0% (12/12)	100.0% (12/12)	100.0% (12/12)
length_contains_code	100.0% (12/12)	100.0% (12/12)	100.0% (12/12)	100.0% (12/12)
length_mod_contains_code	100.0% (12/12)	100.0% (12/12)	100.0% (12/12)	100.0% (12/12)
modulo_sum_label	100.0% (12/12)	100.0% (12/12)	100.0% (12/12)	100.0% (12/12)
not_contains_length_code	100.0% (12/12)	100.0% (12/12)	100.0% (12/12)	100.0% (12/12)
sorted_index_offset_label	100.0% (12/12)	100.0% (12/12)	100.0% (12/12)	100.0% (12/12)
sum_length_branch_label	100.0% (12/12)	100.0% (12/12)	100.0% (12/12)	100.0% (12/12)
sum_offset_mod_label	100.0% (12/12)	100.0% (12/12)	100.0% (12/12)	100.0% (12/12)
tuple_branch_label	100.0% (12/12)	100.0% (12/12)	100.0% (12/12)	100.0% (12/12)
tuple_sum_gate_label	100.0% (12/12)	100.0% (12/12)	100.0% (12/12)	100.0% (12/12)

Experiment log 7

Show the running log (7 entries)

Objective

Test whether fixed-budget static bridge posttraining transfers from support bridge families to deeper held-out composition families that are absent from bridge training.

Design Commitments

Use only Qwen/Qwen3.5-4B.
Keep each trained adapter at 240 records.
Keep the compact experiment directory downloadable by storing trained adapters and checkpoints outside it.
Train a seed baseline, a 60-record static bridge adapter, and an 80-record static bridge adapter.
Evaluate support-family generalization, held-out ceiling-family generalization, IID retention, and trace controls.
Generate a final markdown report and charts.

Starting Hypotheses

Static bridge examples should strongly improve support-family repair relative to the seed adapter.
Held-out ceiling families should be harder than support families and may expose whether bridge training learns a reusable trace interface or mostly family templates.
Increasing bridge dose from 60 to 80 may help if the ceiling result is limited by bridge-token exposure.
Shuffled traces should be harmful if the adapter is using visible execution semantics rather than ignoring the trace.

Planned Runs

Build deterministic datasets from seed 20260630.
Train seed_lora.
Train static60_lora.
Train static80_lora.
Evaluate all adapters on support, ceiling, and IID splits.
Run ceiling trace controls for the stronger static bridge adapter, and for both static adapters if time permits.
Generate charts and a final report.
Audit compact artifact size and large artifact separation.

Step Log

Initialized standalone experiment directory and large artifact directory.
Added held-out ceiling composition families to the data generator.
Replaced dataset builder with the seed/static60/static80 ceiling-breaker design.
Replaced report builder with markdown plus PNG chart generation.
python -m compileall src scripts passed before dataset generation.
Built deterministic datasets with seed 20260630.
Dataset counts: seed train 240, static60 train 240, static80 train 240, support eval 120, ceiling eval 120, IID eval 60.
Static 60 bridge allocation: 6 records per support bridge family, 60 total.
Static 80 bridge allocation: 8 records per support bridge family, 80 total.
Static 60 selector summary: average 4.10 eliminated wrong programs per bridge record, zero remaining static distractors on average.
Static 80 selector summary: average 4.10 eliminated wrong programs per bridge record, zero remaining static distractors on average.
Confirmed that no held-out ceiling family appears in either static bridge training set.
Trained seed_lora for 2 epochs / 60 optimizer steps on 240 seed records.
seed_lora training summary: runtime 835.7s, train loss 0.1080, eval loss 0.0001593 on the 24-record training-time eval subset.
Saved seed_lora under /workspace/large_artifacts/qwen35_4b_static_bridge_ceiling_breaker/models/seed_lora.
Trained static60_lora for 2 epochs / 60 optimizer steps on 240 records.
static60_lora training summary: runtime 873.7s, train loss 0.1047, eval loss 0.00004824 on the 24-record training-time eval subset.
Saved static60_lora under /workspace/large_artifacts/qwen35_4b_static_bridge_ceiling_breaker/models/static60_lora.
Trained static80_lora for 2 epochs / 60 optimizer steps on 240 records.
static80_lora training summary: runtime 868.4s, train loss 0.1058, eval loss 0.0001918 on the 24-record training-time eval subset.
Saved static80_lora under /workspace/large_artifacts/qwen35_4b_static_bridge_ceiling_breaker/models/static80_lora.
Support eval, seed_lora, trace prompt, 3 samples: greedy hidden 61/120, reranked hidden 64/120.
Support eval, static60_lora, trace prompt, 3 samples: greedy hidden 119/120, reranked hidden 120/120.
Support eval, static80_lora, trace prompt, 3 samples: greedy hidden 120/120, reranked hidden 120/120.
Ceiling eval, seed_lora, trace prompt, 3 samples: greedy hidden 15/120, reranked hidden 24/120.
Ceiling eval, static60_lora, trace prompt, 3 samples: greedy hidden 46/120, reranked hidden 53/120.
Ceiling eval, static80_lora, trace prompt, 3 samples: greedy hidden 48/120, reranked hidden 49/120.
IID eval, seed_lora, trace prompt, greedy only: hidden 60/60.
IID eval, static60_lora, trace prompt, greedy only: hidden 60/60.
IID eval, static80_lora, trace prompt, greedy only: hidden 60/60.
Trace control, static60_lora, no-trace prompt, greedy only on ceiling: hidden 18/120.
Trace control, static60_lora, shuffled-trace prompt, greedy only on ceiling: hidden 8/120.
Generated final report at reports/qwen35_4b_static_bridge_ceiling_breaker_report.md.
Generated figures: figures/support_ceiling_rerank_hidden.png, figures/ceiling_trace_controls.png, and figures/ceiling_by_family.png.

Main Readout

Static bridge posttraining saturated the support split: 64/120 reranked hidden for the seed adapter versus 120/120 for both static bridge adapters.
The same training did transfer to held-out ceiling families, but only partially: 24/120 reranked hidden for the seed adapter, 53/120 for static60_lora, and 49/120 for static80_lora.
The 80-record bridge dose did not improve the ceiling result over the 60-record bridge dose, despite equal support saturation and perfect IID retention.
Trace alignment mattered. On static60_lora, aligned ceiling greedy hidden was 46/120, no-trace was 18/120, and shuffled-trace was 8/120.
Strong transfer families were trace-aligned absent/contains/count compositions: text_absent_mod_code 12/12, text_value_gate_label 12/12, token_absent_length_code 11/12, and token_count_mod_length_code 10/12 under static60_lora.
Persistent failures concentrated in sort/join and deeper numeric composition: sorted_index_sum_branch_label 0/12, sorted_join_contains_code 0/12, sum_length_mod_gate_label 1/12, and sum_len_mod_label 1/12 under static60_lora.

Final Interpretation

The experiment supports a narrower version of the trace-conditioned bridge hypothesis. A small amount of static bridge posttraining can teach an executable repair interface that generalizes beyond the support families, and the trace controls show the gain is not just prompt formatting. However, support-family saturation was not enough to break the deeper ceiling: the adapter learned some reusable symbolic moves but did not robustly compose sort, join, sum, length, modulo, and branch operations in the hardest held-out families. The next experiment should therefore target breadth and compositional coverage of bridge families rather than increasing the dose of the same support families.

Figures 3

support ceiling rerank hidden · figures/

Reproduce

Runnable scripts exist in the experiment folder, but the exact invocation was not written down.

Browse the experiment folder on GitHub ↗

Qwen 3.5 4B Static Bridge Ceiling Breaker

Results at a glance 3

Overview

Layout

Report

Report

Question

Design

Dataset

Support Split Results

Ceiling Split Results

Ceiling Trace Controls

IID Retention Results

Ceiling By Family

Support By Family

Readout

Figures

Failure Signatures

Per-Condition Details

seed_lora_ceiling

seed_lora_iid

seed_lora_support

static60_lora_ceiling

static60_lora_iid

static60_lora_no_trace_ceiling

static60_lora_shuffled_trace_ceiling

static60_lora_support

static80_lora_ceiling

static80_lora_iid

static80_lora_support

Experiment log 7

Objective

Design Commitments

Starting Hypotheses

Planned Runs

Step Log

Main Readout

Final Interpretation

Figures 3

Reproduce

Related

Related experiments