Qwen 3.5 4B Verified Edit Closure
The one idea you need
A small model's wrong program is usually one slip from correct: a swapped operation, a missing "not." Picture a spell-checker that only knows single-edit fixes. It generates every nearby variant, runs each against the example cases, and keeps whichever one now passes.
The question
When a small model writes a program that is almost right but fails, can automatically trying tiny edited versions and testing them fix more than just asking again?
What we found
Yes, on the hardest unseen tasks. Testing small edits of the model's own near-miss program and keeping whichever passes the example cases raised fully-correct answers from 39% to 52%, while re-sampling the model and re-ranking its guesses gave nothing over its first try. But it stalled near 58%: some wrong programs need a rewrite, not a tweak.
Why it matters
For small-model code repair, wrap generation in a cheap edit-and-run loop before spending on more training. Accept a fix only when it passes every example the model missed, which halves the edits that break working code.
On this page
Results at a glance 3
How to read
Five bars, one per repair method; height is the share of 120 hard unseen tasks fully solved on the hidden tests. Left to right: single guess, re-ranking guesses, edit testing, strict edit testing, best-possible edit. Taller is better.
Takeaway → Both edit-testing bars jump from about 39% to roughly 52%, nearly reaching the 58% best-possible bar: verified edits, not more sampling, carry the gain.
Data table
| policy | hidden all-pass rate |
|---|---|
| greedy | 40% |
| visible rerank | 39.2% |
| edit closure | 51.7% |
| strict closure | 50.8% |
| hidden oracle | 57.5% |
Technical framing
Held-out ceiling split: hidden all-pass rate by repair policy — Verified symbolic edit closure lifts hard held-out repair from ~39% to ~52%, close to the 57.5% oracle ceiling of the same edit neighborhood.
How to read
Six task families across the bottom, three bars each, counting fully-solved tasks out of 12: re-ranking guesses, edit testing, and best-possible edit. Higher is better; families are ordered by how much editing helped.
Takeaway → Editing lifts one family from zero to nine solved, but two families stay flat at zero even for the best-possible edit: no such fix exists in the menu.
Data table
| ceiling task family | baseline rerank | edit closure | hidden oracle |
|---|---|---|---|
| tuple_value_mod_label | 0 | 9 | 9 |
| sorted_index_sum_branch_label | 0 | 2 | 2 |
| token_count_mod_length_code | 8 | 10 | 12 |
| tuple_sum_mod_gate_label | 5 | 5 | 7 |
| sorted_join_contains_code | 0 | 0 | 0 |
| sum_len_mod_label | 0 | 0 | 0 |
Numbers from report table (reports/qwen35_4b_verified_edit_closure_report.md, Ceiling Families)
Technical framing
Where edit closure helps: hidden all-pass records by ceiling family (out of 12) — Closure recovers whole families the baseline misses (tuple_value 0 to 9), but two families stay at zero even for the oracle: the edit operators, not the selector, are the limit there.
How to read
Three acceptance rules along the bottom; paired bars count tasks the edits improved versus damaged, out of 120. More improved is good, fewer damaged is good. The strictest rule, accepting only fixes that pass every example, sits on the right.
Takeaway → Moving to strict acceptance drops improved tasks from 49 to 33 but halves damaged tasks from 8 to 4: the safer setting to deploy.
Data table
| acceptance policy | records improved | records damaged |
|---|---|---|
| closure visible select | 49 | 8 |
| conservative accept | 48 | 8 |
| strict accept visible-all | 33 | 4 |
Numbers from report table (reports/qwen35_4b_verified_edit_closure_report.md, Ceiling Policies)
Technical framing
Acceptance policy trade-off: hidden pass-count improved vs damaged records — Strict visible-all acceptance halves damage (8 to 4 records) while keeping nearly the same hidden all-pass count (61 vs 62 of 120).
In the author’s words from the Overview · “Final Results”
IID baseline rerank hidden all-pass: 60/60. Support baseline rerank hidden all-pass: 120/120. Ceiling baseline rerank hidden all-pass: 47/120. Ceiling visible-selected edit closure hidden all-pass: 62/120. Ceiling strict visible-all closure hidden all-pass: 61/120. Ceiling hidden-oracle closure all-pass: 69/120. Strict closure accepted 39/120 edits and had 4 hidden pass-count damage cases.
Overview
This standalone experiment tests whether Qwen 3.5 4B executable-DSL repair improves when model candidates are expanded by a bounded symbolic edit neighborhood and selected by visible execution.
The model receives:
- an input schema,
- a current wrong DSL program,
- visible execution cases with expected and got values,
- and must output one corrected executable DSL expression.
The experiment trains one fixed-budget adapter:
static60_lora: 180 base-family records plus 60 support bridge records.
The inference-time edit closure starts from the adapter's generated candidates and enumerates local DSL edits such as primitive substitutions, missing negation, raw/sorted tuple access, modulo scope changes, and missing conjunction terms. Variants are executed on visible cases, and the best visible-passing variant is scored on hidden cases.
Large adapters and checkpoints are intentionally outside this compact directory:
/workspace/large_artifacts/qwen35_4b_verified_edit_closure/
Layout
configs/experiment.json: fixed design and hyperparameters.src/: standalone DSL, data, prompt, model, and edit-closure utilities.scripts/: dataset generation, training, baseline evaluation, edit-closure evaluation, and report entry points.data/: generated JSONL datasets and manifest.reports/: evaluation JSON files, final report, and generated charts underreports/figures/.logs/andrun_logs/: experiment notebook and command output.large_artifacts_manifest.md: pointers to adapter directories stored outside this compact directory.
Report
Final report:
reports/qwen35_4b_verified_edit_closure_report.md
Final Results
- IID baseline rerank hidden all-pass: 60/60.
- Support baseline rerank hidden all-pass: 120/120.
- Ceiling baseline rerank hidden all-pass: 47/120.
- Ceiling visible-selected edit closure hidden all-pass: 62/120.
- Ceiling strict visible-all closure hidden all-pass: 61/120.
- Ceiling hidden-oracle closure all-pass: 69/120.
- Strict closure accepted 39/120 edits and had 4 hidden pass-count damage cases.
Report
Rendered from reports/qwen35_4b_verified_edit_closure_report.md
Executive Summary
A fresh Qwen/Qwen3.5-4B LoRA adapter solved IID and support DSL evals perfectly. On the held-out ceiling split, normal visible reranking reached 47/120 hidden all-pass records. Bounded symbolic edit closure raised visible-selected hidden all-pass to 62/120, while the hidden oracle inside the same closure neighborhoods reached 69/120.
The main positive result is that local verified edits added real recoverable capability on hard held-out compositions. The main negative result is that six visible cases are not enough to choose safely in every family: the strict visible-all acceptance policy kept 61/120 hidden all-pass and reduced pass-count damage to 4 records, but it did not eliminate it.
Setup
- Base model:
Qwen/Qwen3.5-4B. - Adapter: LoRA, trained for 2 epochs on 240 static DSL records.
- Data: 60 IID eval records, 120 support eval records, 120 held-out ceiling records.
- Each eval record has 6 visible cases and 18 hidden cases.
- Normal baseline: greedy plus three sampled candidates for support and ceiling, selected by visible execution.
- Closure: bounded two-round local DSL edits from up to four model candidates, selected by visible execution.
- Strict policy: accept a closure program only when it reaches all visible cases and the baseline did not.
- Hidden oracle: best hidden-case result inside the closure candidate set, used only as a diagnostic.
Split Results
| Split | Records | Greedy Hidden | Rerank Hidden | Closure Hidden | Strict Hidden | Oracle Hidden |
|---|---|---|---|---|---|---|
| IID | 60 | 60/60 (100.0%) | 60/60 (100.0%) | 60/60 (100.0%) | 60/60 (100.0%) | 60/60 (100.0%) |
| Support | 120 | 120/120 (100.0%) | 120/120 (100.0%) | 120/120 (100.0%) | 120/120 (100.0%) | 120/120 (100.0%) |
| Ceiling | 120 | 48/120 (40.0%) | 47/120 (39.2%) | 62/120 (51.7%) | 61/120 (50.8%) | 69/120 (57.5%) |
Ceiling Policies
| Policy | Hidden All-Pass | Visible All-Pass | Hidden Pass-Count Improved | Hidden Pass-Count Damaged | Accepted |
|---|---|---|---|---|---|
| Baseline visible rerank | 47/120 (39.2%) | 53/120 (44.2%) | 0 | 0 | 0 |
| Closure visible select | 62/120 (51.7%) | 92/120 (76.7%) | 49 | 8 | 120 |
| Conservative accept visible gain | 61/120 (50.8%) | 92/120 (76.7%) | 48 | 8 | 59 |
| Strict accept visible all | 61/120 (50.8%) | 92/120 (76.7%) | 33 | 4 | 39 |
| Hidden oracle diagnostic | 69/120 (57.5%) | hidden-only | diagnostic | diagnostic | 120 |
Ceiling Families
| Family | Base | Closure | Strict | Oracle | Strict Accepted | Strict Damaged |
|---|---|---|---|---|---|---|
| sorted_index_sum_branch_label | 0 | 2 | 2 | 2 | 9 | 0 |
| sorted_join_contains_code | 0 | 0 | 0 | 0 | 4 | 0 |
| sum_len_mod_label | 0 | 0 | 0 | 0 | 0 | 0 |
| sum_length_mod_gate_label | 0 | 1 | 1 | 3 | 6 | 0 |
| text_absent_mod_code | 12 | 12 | 12 | 12 | 0 | 0 |
| text_value_gate_label | 12 | 12 | 12 | 12 | 0 | 0 |
| token_absent_length_code | 10 | 11 | 11 | 12 | 1 | 0 |
| token_count_mod_length_code | 8 | 10 | 9 | 12 | 4 | 0 |
| tuple_sum_mod_gate_label | 5 | 5 | 5 | 7 | 6 | 4 |
| tuple_value_mod_label | 0 | 9 | 9 | 9 | 9 | 0 |
Search Budget
| Split | Generated Median | Generated P90 | Valid Median | Valid P90 |
|---|---|---|---|---|
| IID | 23 | 971 | 23 | 971 |
| Support | 970 | 989 | 970 | 989 |
| Ceiling | 973 | 1358 | 973 | 1358 |
Interpretation
The edit closure helped most when the model produced a structurally nearby but semantically wrong program. The clearest gains were in tuple_value_mod_label, sorted_index_sum_branch_label, and token_count_mod_length_code. These are cases where the symbolic neighborhood contained useful repairs and visible execution usually moved selection in the right direction.
The ceiling oracle result, 69/120, is only seven records above pure visible closure at 62/120. That means the local edit space is a real constraint, not just the selector. Families such as sorted_join_contains_code and sum_len_mod_label had zero hidden-oracle all-pass records, so the current edit operators do not generate the needed programs for those records.
The strict policy is the better deployable readout than pure closure: it gives nearly the same hidden all-pass count as conservative closure, accepts fewer edits, and halves pass-count damage. The remaining damage is concentrated in tuple_sum_mod_gate_label, where visible cases are ambiguous among several plausible tuple and sum predicates.
Iteration Notes
The first closure selector used shortest-program tie-breaking among visible-equivalent programs. That caused IID hidden regressions by selecting degenerate but visible-perfect simplifications. The selector was changed to stable first-seen tie-breaking, which preserves the nearest seed candidate under ties. After that change, IID and support closure both stayed perfect.
A second policy layer was then added: conservative acceptance requires visible pass-count gain, and strict acceptance additionally requires all visible cases. Strict acceptance is the cleanest summary of what a visible-only repair policy can safely claim here.
Conclusion
This experiment supports the hypothesis that a symbolic program-edit region can amplify a small LLM's held-out executable-task performance, but not enough by itself for a step-change result. The next most direct improvement is not more LoRA training. It is stronger visible discrimination: adaptive counterexample generation or active visible-case expansion targeted at closure ties.
Experiment log 5
Show the running log (5 entries)
Objective
Test whether a bounded verifier-guided symbolic edit closure can improve held-out executable DSL repair over normal candidate sampling for Qwen 3.5 4B.
Design Commitments
- Use only
Qwen/Qwen3.5-4B. - Train a fresh DSL baseline adapter inside this standalone experiment.
- Keep the training budget fixed at 240 records.
- Keep adapter/checkpoint files outside the compact experiment directory.
- Evaluate normal model candidates and edit-closure-expanded candidates on IID, support, and held-out ceiling splits.
- Report both verifier-selected hidden success and hidden-oracle closure coverage.
- Generate a final markdown report and charts.
Hypotheses
- Many held-out failures are valid DSL programs that are one or two local semantic edits away from the target.
- Visible execution can select the corrected local edit without hidden-case access.
- If hidden-oracle closure coverage is much higher than verifier-selected closure success, the bottleneck is visible-case discrimination rather than candidate support.
- If hidden-oracle closure coverage is low, this edit family is not enough and a larger program search or different representation is needed.
Planned Runs
- Build deterministic datasets from seed
20260630. - Train
static60_loraon 180 base records plus 60 support bridge records. - Evaluate
static60_loraon IID, support, and ceiling splits with normal visible reranking. - Run edit closure on the normal baseline results for IID, support, and ceiling.
- If edit closure materially improves ceiling hidden success, inspect whether the improvement is selected by visible cases or only present under hidden oracle.
- Generate charts and final report.
- Audit compact artifact size and large artifact separation.
Step Log
- Initialized standalone experiment directory and large artifact directory.
- Copied stable DSL, data generation, prompt, training, and baseline evaluator utilities.
- Added bounded symbolic DSL edit closure and closure evaluation script.
python -m compileall src scriptspassed before dataset generation.- Local closure smoke test passed: the edit closure generated the intended
len textrepair from acount_eq text needlefailure within two edit rounds. - Built deterministic datasets with seed
20260630. - Dataset counts used for the main experiment: static60 train 240, IID eval 60, support eval 120, ceiling eval 120.
- Training mix for
static60_lora: 180 base-family records plus 60 support bridge records. - Confirmed that no held-out ceiling family appears in the static60 training set.
- Confirmed that target DSL programs execute correctly on all visible and hidden cases for static60 train, IID eval, support eval, and ceiling eval.
- Trained
static60_lorafromQwen/Qwen3.5-4Bfor 2 epochs on the 240-record static60 DSL set. - Training completed in 909.3 seconds with final
train_loss=0.1042. - Final trainer IID eval on the 60-record IID eval file produced
eval_loss=8.394e-05. - Large adapter/checkpoint artifacts were written outside the compact experiment directory at
/workspace/large_artifacts/qwen35_4b_verified_edit_closure/models/static60_lora(445Mafter training). - Baseline eval completed for all splits.
- IID baseline greedy and reranked hidden all-pass: 60/60.
- Support baseline greedy and reranked hidden all-pass: 120/120.
- Ceiling baseline greedy hidden all-pass: 48/120; ceiling baseline visible-reranked hidden all-pass: 47/120.
- Ran closure smoke check on 5 ceiling records; schema and runtime were valid.
- First closure selector iteration exposed visible-tie regressions on IID because shortest-program tie-breaking chose degenerate visible-perfect programs.
- Replaced closure visible tie-breaking with stable first-seen selection so nearer seed candidates are preserved under equal visible pass counts.
Added conservative and strict closure acceptance policies:
- conservative accepts closure only when visible pass count improves over baseline;
- strict accepts closure only when closure reaches all visible cases and baseline does not.
- Reran closure evals with the revised selector and acceptance metrics.
- IID closure hidden all-pass after selector revision: 60/60.
- Support closure hidden all-pass after selector revision: 120/120.
- Ceiling visible-selected closure hidden all-pass: 62/120.
- Ceiling conservative closure hidden all-pass: 61/120.
- Ceiling strict visible-all closure hidden all-pass: 61/120.
- Ceiling hidden-oracle closure all-pass: 69/120.
- Strict closure accepted 39/120 ceiling edits and had 4 hidden pass-count damage cases.
- Generated final report at
reports/qwen35_4b_verified_edit_closure_report.md. - Generated charts under
reports/figures/:ceiling_hidden_success.png,closure_by_family.png, andclosure_candidate_counts.png. Final audit:
python -m compileall src scriptspassed.- Removed generated
__pycache__directories after compile audit. - No files larger than 50M are present in the compact experiment directory.
- Compact experiment directory size:
6.8M. - Large adapter/checkpoint directory size:
445M. - No train or eval processes remained running.
- Stale-reference scan returned no hits for old experiment names or non-Qwen3.5 model names.
- Final dataset line counts: train 240, IID 60, support 120, ceiling 120.
Figures 3
Reproduce
Runnable scripts exist in the experiment folder, but the exact invocation was not written down.