Qwen Span-Free Compiler
The one idea you need
A frozen language model silently reads a math word-problem and scribbles private notes across a long strip. A small add-on translator turns those notes into exact calculator steps, pointing out the starting number, each operation, and each amount, though nobody ever tells it where in the notes they sit.
The question
Can a small add-on read a frozen model's internal notes and reconstruct the exact math steps of a problem, without being told where the numbers and operations sit?
What we found
Only when it is first taught where to look. Fed just the frozen model's raw internal notes, a plain reader stayed near random guessing (about 1 in 97). Adding training that also highlighted which spots held the numbers and operations reached 99.6% fully-correct programs at the practiced length. But reword the prompt, or add more steps, and accuracy collapses back to chance.
Why it matters
Frozen small models really do carry recoverable program structure in their internals, but only a narrow, brittle slice. Do not expect an extracted interface to survive reworded prompts or longer inputs unless training explicitly covers those cases.
On this page
Results at a glance 8
How to read
Each bar is one way of building the system; bar height is how often the whole step-sequence came out exactly right on the practiced problem style. Taller is better. The two guided-attention translators stand near the top; the direct answer reader and answer-only translator sit at the floor.
Takeaway → Only the two translators taught where to focus reach the top, near 100%, while the plain methods stay at random-guess level.
Data table
| Training variant | exact execution |
|---|---|
| direct answer head | 1.6% |
| answer-only compiler | 0% |
| trace-bootstrap compiler | 99.6% |
| trace then answer-only | 99.6% |
Numbers from analysis/final_metrics.csv (run main_qwen35_attention_len4_retention, standard_len4)
Technical framing
Exact execution on trained length-4 programs: trace bootstrap vs controls — Attention-aligned trace bootstrap reaches 99.6% exact execution; direct and answer-only controls stay at chance (~1%).
How to read
The horizontal axis is how many math steps the problem has; the vertical axis is how often the whole sequence comes out exactly right. Higher is better. Each line is one build of the system. The two trace-trained lines are high only at the practiced four-step length and crash toward zero beyond it, while the two control builds sit at random-guess level at every length.
Takeaway → The two trace-trained lines plunge from near-perfect at four steps to near-zero at eight, twelve, and twenty-four steps, while the control lines never leave random-guess level, showing the skill is locked to one length.
Data table
| Program length (steps) | trace-bootstrap compiler | trace then answer-only | answer-only compiler | direct answer head |
|---|---|---|---|---|
| 4 | 99.6% | 99.6% | 0% | 1.6% |
| 8 | 0.4% | 1.6% | 0.8% | 1.2% |
| 12 | 1.2% | 1.6% | 0.8% | 0% |
| 24 | 0.8% | 0.4% | 0% | 1.2% |
Numbers from analysis/final_metrics.csv (run main_qwen35_attention_len4_retention)
Technical framing
Execution accuracy vs program length: no transfer beyond trained slots — The installed interface is length-specific: near-perfect at the trained length 4, chance-level at untrained lengths 8-24.
How to read
Bars are grouped by what the translator had to read out: the starting number, each operation, each amount, and the whole sequence. Height is accuracy; taller is better. Practiced-wording bars sit near the top; reworded-prompt bars are much shorter except for the starting number.
Takeaway → Reworded bars stay tall only for the starting number and roughly halve for operations and amounts, dragging the whole-sequence bar down to almost nothing.
Data table
| Predicted program symbol | standard template | paraphrased prompt |
|---|---|---|
| initial value | 100% | 94.9% |
| operation | 100% | 58% |
| argument | 99.9% | 54% |
| exact execution | 99.6% | 1.6% |
Numbers from analysis/final_metrics.csv (run main_qwen35_attention_len4_retention, compiler_trace)
Technical framing
Trained wording vs paraphrase at length 4: symbol extraction breaks down — Paraphrasing mostly preserves initial-value extraction but halves op/arg accuracy, collapsing exact execution to 1.6%.
How to read
Horizontal axis is chain length; vertical axis is accuracy; higher is better. Two lines track how often each single operation and each single amount is right; the third tracks solving the whole sequence. The per-step lines stay fairly high while the whole-sequence line sinks fast.
Takeaway → Even with each step around 90% right at twelve steps, the whole-sequence line drops under 6%, because small per-step errors pile up.
Data table
| Program length (steps) | per-step operation | per-step argument | exact execution |
|---|---|---|---|
| 4 | 100% | 99.8% | 99.2% |
| 8 | 93.3% | 94.4% | 37.9% |
| 12 | 86.3% | 88.9% | 5.9% |
| 24 | 56.5% | 41.5% | 2% |
Numbers from analysis/final_metrics.csv (run pilot_qwen35_independent_attention_len12_standard_strong, compiler_trace)
Technical framing
Stronger length-12 training: high per-step accuracy still compounds to failure — Per-step symbol accuracy near 90% at L=12 still yields under 6% exact execution: a compounding-error regime.
How to read
Each bar is a different way of building the translator; height is how often it solved the whole practiced sequence exactly. Taller is better. The three simpler designs that just pool everything sit at the floor; the two that learn where to focus stand near the top.
Takeaway → Accuracy jumps from random-guess level to near-perfect only for the two designs that learn where to point, marking focus as the key ingredient.
Data table
| Compiler readout variant | Trace-trained pilot, shortest trained length |
|---|---|
| Span-free pooled | 1.6% |
| Split-init pooled | 0.8% |
| Token pool | 1% |
| Attention | 99.2% |
| Generated-query attention | 99.2% |
Technical framing
Readout architecture ablation: only attention readouts bootstrap execution — Pooled readouts stay at chance (~1%); adding attention-aligned queries jumps trace-trained execution to 99% at the trained length.
How to read
Horizontal axis is chain length; vertical axis is how often the whole sequence is solved; higher is better. Each line is a different training schedule. All lines collapse toward the floor by twelve steps, and some start low even at short chains.
Takeaway → Every schedule falls to near-zero by twelve steps, and the fixed and mixed schedules give up much of their short-chain accuracy too.
Data table
| Evaluation program length (steps) | Fixed-length-12 curriculum | Mixed-length curriculum | Standard curriculum | Standard strong (longer training) |
|---|---|---|---|---|
| 4 | 50% | 44.5% | 96.1% | 99.2% |
| 8 | 1.6% | 1.6% | 13.3% | 37.9% |
| 12 | 0% | 1.6% | 3.1% | 5.9% |
| 24 | 0% | 1.6% | 0.8% | 2% |
Technical framing
Length-12 curriculum grid: no schedule escapes the length cliff — Every length-12 curriculum still collapses by 12 steps; fixed and mixed schedules also give up most length-4 accuracy.
How to read
Bars are grouped by training schedule, with one bar for practiced wording and one for reworded wording; height is whole-sequence accuracy, taller is better. High-accuracy schedules show a tall practiced bar beside a floor-level reworded bar; the milder schedules show two similar mid-height bars.
Takeaway → The most accurate schedules drop from near-perfect to random-guess level under rewording, while milder schedules hold similar accuracy both ways but never climb as high.
Data table
| Length-12 training curriculum | Trained wording | Paraphrased wording |
|---|---|---|
| Fixed-length-12 | 50% | 46.1% |
| Mixed-length | 44.5% | 38.3% |
| Standard | 96.1% | 1.6% |
| Standard strong | 99.2% | 3.1% |
Technical framing
Curriculum trade-off: mixed-length training buys paraphrase robustness, costs accuracy — Mixed and fixed curricula keep most accuracy under paraphrase, while high-accuracy standard curricula drop from 96-99% to near chance.
How to read
Horizontal axis is training progress; vertical axis is how often the whole sequence is solved; higher is better. One line is practiced wording, the other reworded wording. The practiced line shoots up early and flattens near the top; the reworded line stays pinned at the bottom throughout.
Takeaway → The practiced-wording line reaches near-perfect within the first stretch and holds, while the reworded line never lifts off random-guess level.
Data table
| Training step | Trained wording | Paraphrased wording |
|---|---|---|
| 1 | 1.2% | 1.2% |
| 500 | 99.2% | 2.7% |
| 1000 | 99.2% | 3.1% |
| 1500 | 99.2% | 3.9% |
| 2000 | 99.6% | 1.6% |
Numbers from experiments/qwen_span_free_compiler/runs/main_qwen35_attention_len4_retention/train_log.csv (compiler_trace variant)
Technical framing
Training curve: trace bootstrap saturates by step 500, paraphrase never moves — The compiler locks onto trained wording within 500 steps and stays at 99%, while paraphrase accuracy stays flat at chance throughout.
In the author’s words from the Report · “Abstract”
This experiment tests whether a small trainable compiler can read the full hidden sequence of a frozen Qwen3.5-4B model and configure an executable latent program without receiving token-span features at inference time. A plain query-context reader fails to extract numeric program symbols. Token-local classification improves argument extraction but still fails to locate the initial value. Adding trace-time attention alignment changes the result: the compiler reaches 99.6% exact execution on the trained length-4 standard template, while a direct answer head and an answer-only compiler remain at chance. Answer-only continuation preserves the installed interface. The result does not transfer to untrained later step slots or paraphrased prompt wording.
Overview
This experiment tests whether a small trainable latent compiler can read a frozen Qwen hidden sequence and configure an executable modular program without being given token-span features for the numeric values or operation words.
Lightweight files live in this directory. Saved checkpoints live under:
large_artifacts/qwen_span_free_compiler/checkpoints/Run outputs are written under runs/, analysis outputs under analysis/, and the chronological experiment log plus final report under reports/.
Report
Rendered from reports/qwen_span_free_compiler_paper.md
Abstract
This experiment tests whether a small trainable compiler can read the full hidden sequence of a frozen Qwen3.5-4B model and configure an executable latent program without receiving token-span features at inference time. A plain query-context reader fails to extract numeric program symbols. Token-local classification improves argument extraction but still fails to locate the initial value. Adding trace-time attention alignment changes the result: the compiler reaches 99.6% exact execution on the trained length-4 standard template, while a direct answer head and an answer-only compiler remain at chance. Answer-only continuation preserves the installed interface. The result does not transfer to untrained later step slots or paraphrased prompt wording.
Setup
The task is modular program execution. Each prompt gives an initial value x, a sequence of add, subtract, and multiply updates modulo 97, and an answer marker. The model must recover the final value.
The frozen model is used only as a hidden-state source. The trainable compiler receives:
- the padded full hidden sequence,
- a sequence mask,
- no numeric token span features at inference time.
The compiler predicts:
- initial value,
- per-step operation,
- per-step argument.
Those symbols are executed by a differentiable modular executor during training and by argmax symbolic execution for exact accuracy.
Variants
| Variant | Training signal |
|---|---|
direct | answer classification from the answer-marker hidden state |
compiler_answer_only | final-answer loss through the executor |
compiler_trace | symbol trace loss, executor loss, and attention alignment |
compiler_trace_then_answer | trace bootstrap followed by final-answer-only continuation |
Attention alignment is used only while trace loss is active. At evaluation time, the compiler still reads the full hidden sequence and must attend for itself.
Main Result
Run: main_qwen35_attention_len4_retention
Training: standard-template length-4 programs, 2048 bootstrap examples, 2048 answer-continuation examples, frozen Qwen3.5-4B features, independent step queries, attention-aligned trace bootstrap.
| Variant | L=4 exec | L=8 exec | L=12 exec | L=24 exec | L=4 init | L=4 op | L=4 arg | L=4 exact |
|---|---|---|---|---|---|---|---|---|
direct | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a |
compiler_trace | 99.6% | 0.4% | 1.2% | 0.8% | 100.0% | 100.0% | 99.9% | 99.6% |
compiler_answer_only | 0.0% | 0.8% | 0.8% | 0.0% | 0.4% | 35.5% | 1.7% | 0.0% |
compiler_trace_then_answer | 99.6% | 1.6% | 1.6% | 0.4% | 100.0% | 100.0% | 99.9% | 99.6% |
The direct answer control stays at chance:
| Split | Direct accuracy |
|---|---|
| standard L=4 | 1.6% |
| standard L=8 | 1.2% |
| standard L=12 | 0.0% |
| standard L=24 | 1.2% |
Stress Tests
The trained interface is narrow. When evaluated on paraphrased length-4 prompts, the compiler does not transfer:
| Variant | paraphrase L=4 exec | paraphrase L=4 init | paraphrase L=4 op | paraphrase L=4 arg |
|---|---|---|---|---|
compiler_trace | 1.6% | 94.9% | 58.0% | 54.0% |
compiler_trace_then_answer | 2.0% | 94.5% | 62.9% | 55.7% |
Longer standard chains also fail when later step slots are not trained:
| Variant | standard L=8 exec | standard L=12 exec | standard L=24 exec |
|---|---|---|---|
compiler_trace | 0.4% | 1.2% | 0.8% |
compiler_trace_then_answer | 1.6% | 1.6% | 0.4% |
A stronger length-12 trace condition improves per-step symbol accuracy but not long-chain exactness:
| Split | Exec | Init | Op | Arg | Program exact |
|---|---|---|---|---|---|
| standard L=4 | 99.2% | 100.0% | 100.0% | 99.8% | 99.2% |
| standard L=8 | 37.9% | 100.0% | 93.3% | 94.4% | 37.9% |
| standard L=12 | 5.9% | 100.0% | 86.3% | 88.9% | 4.3% |
This is a compounding-error regime: per-step symbol accuracy that looks high is still not high enough for exact long-chain execution.
Interpretation
The positive result is real but bounded. A small trainable module can compile frozen Qwen hidden states into an executable latent program without span features at inference time. Trace-time attention alignment is the critical ingredient; without it, numeric extraction does not form reliably. Once the interface is installed, final-answer-only continuation preserves it on the trained distribution.
The negative result is equally important. The method does not yet produce a general latent program interface. It does not discover the interface from answer-only supervision, does not extrapolate to untrained step slots, and does not handle paraphrased wording under the tested training budget.
Conclusion
The experiment supports a narrow claim: frozen Qwen hidden states can feed a span-free executable latent compiler when bootstrap supervision teaches both symbols and attention. It does not support a broad claim of universal posttraining improvement. The next technically meaningful step is to replace independent step queries with a parser-like sequence tagger or to train a small QLoRA adapter so Qwen exposes stable program-token features across wording and length.
Artifacts
- Source:
experiments/qwen_span_free_compiler/src/ - Runs:
experiments/qwen_span_free_compiler/runs/ - Analysis:
experiments/qwen_span_free_compiler/analysis/ - Checkpoints:
large_artifacts/qwen_span_free_compiler/checkpoints/ - Manifest:
experiments/qwen_span_free_compiler/checkpoint_manifest.csv
Experiment log 11
Show the running log (11 entries, 2026-06-21)
Objective
Test whether a frozen Qwen hidden sequence contains enough information for a small trainable compiler to locate program-relevant tokens, emit executable program symbols, and preserve that interface when trace loss is removed.
Primary Questions
- Can a learned query-attention compiler recover initial values, operations, and arguments from the full prompt hidden sequence without token-span input?
- Does trace bootstrap install an executable interface that generalizes to longer chains?
- Does answer-only continuation preserve or improve the installed interface?
- Does answer-only training from scratch discover the interface?
- How sensitive is the learned compiler to prompt wording and line-format changes?
Metrics
executor_accuracy: accuracy after argmax compilation and exact execution.executor_target_mass: differentiable executor probability assigned to the target answer.init_accuracy: compiled initial value accuracy.op_accuracy: per-step operation accuracy.arg_accuracy: per-step argument accuracy.program_exact: fraction of examples with all compiled symbols correct.direct_accuracy: direct answer classifier accuracy from the frozen Qwen answer-position feature.
Artifact Layout
- Code and lightweight outputs:
experiments/qwen_span_free_compiler/ - Checkpoints:
large_artifacts/qwen_span_free_compiler/checkpoints/ - Run outputs:
experiments/qwen_span_free_compiler/runs/<run>/ - Analysis outputs:
experiments/qwen_span_free_compiler/analysis/
2026-06-21 Setup
Created the standalone experiment directory:
experiments/qwen_span_free_compiler/src/experiments/qwen_span_free_compiler/reports/experiments/qwen_span_free_compiler/runs/experiments/qwen_span_free_compiler/analysis/figures/large_artifacts/qwen_span_free_compiler/checkpoints/
Implementation plan:
- Cache padded full-sequence hidden states and attention masks.
- Replace span-selected compiler inputs with learned query cross-attention over the whole sequence.
- Keep the direct answer head as a control on the answer-position feature.
- Add standard, mixed, and paraphrase prompt templates for wording-shift evaluation.
- Preserve staged variants: direct, trace, answer-only, trace-then-answer, and low-learning-rate trace-then-answer.
2026-06-21 Harness Smoke
Implemented the span-free harness:
- Full hidden-sequence feature caching with sequence masks.
- Learned query cross-attention compiler over the whole prompt.
- Direct answer classifier control from the answer-marker hidden state.
- Standard and paraphrase template evaluation splits.
- Staged variants for trace bootstrap and answer-only retention.
- External checkpoint writing.
- Analysis summary and figures.
Verification:
- Source compilation passed.
- Stale inherited-path scan passed.
- Tiny random Llama smoke completed.
- Analysis generation completed.
- Checkpoints were written under
large_artifacts/qwen_span_free_compiler/checkpoints/smoke_tiny/.
Smoke interpretation:
- The smoke validates data flow, padded hidden-sequence caching, query-attention compiler calls, staged training, checkpointing, and analysis.
- The tiny random model has no meaningful parsing signal, so its low accuracy is only a plumbing check.
Next action: run a small Qwen pilot with frozen Qwen features to estimate whether the span-free compiler can learn symbol extraction from the full prompt.
2026-06-21 Qwen Pilot: Query-Context Reader
Ran pilot_qwen35_span_free with frozen Qwen3.5-4B features, full-sequence hidden-state caching, learned query-context attention, bootstrap training lengths 1-3, answer-continuation training lengths 1-6, and evaluation lengths 3, 6, and 12 under standard and paraphrase templates.
Final standard-template metrics:
| Variant | L=3 exec | L=6 exec | L=12 exec | L=3 init | L=3 op | L=3 arg |
|---|---|---|---|---|---|---|
direct | n/a | n/a | n/a | n/a | n/a | n/a |
compiler_trace | 1.6% | 3.1% | 4.7% | 0.0% | 75.5% | 7.3% |
compiler_answer_only | 3.1% | 0.0% | 0.0% | 0.0% | 29.2% | 0.0% |
compiler_trace_then_answer | 0.0% | 0.0% | 1.6% | 1.6% | 81.8% | 0.5% |
Pilot interpretation:
- The direct answer control remains at chance.
- The trace-supervised compiler partially learns operation words, but numeric initial values and numeric arguments remain near chance on held-out examples.
- Training loss for initial values falls while held-out initial accuracy stays near chance, indicating memorization rather than a reusable numeric reader.
- The simplest query-context pooling is too weak for the span-free condition.
Next action: replace context-only pooling with token-logit pooling. Each learned query still attends over the full prompt, but the class evidence is computed at each token before pooling, which should preserve local numeric features.
2026-06-21 Qwen Pilot: Token-Logit Pooling
Patched the compiler so learned queries pool token-local class logits instead of only pooling hidden-state contexts.
Ran smoke_tiny_token_pool; source compilation and tiny-model smoke passed.
Ran pilot_qwen35_token_pool_trace with the trace-supervised compiler only, 512 bootstrap examples, 512 continuation examples, standard-template training, and standard/paraphrase evaluation.
Final metrics:
| Split | Exec | Target mass | Init | Op | Arg | Program exact |
|---|---|---|---|---|---|---|
| standard L=3 | 1.0% | 1.0% | 1.0% | 55.9% | 41.7% | 0.0% |
| standard L=6 | 0.0% | 1.0% | 3.1% | 56.4% | 35.4% | 0.0% |
| standard L=12 | 3.1% | 1.0% | 2.1% | 41.1% | 15.0% | 0.0% |
| paraphrase L=3 | 3.1% | 1.0% | 1.0% | 40.3% | 4.2% | 0.0% |
| paraphrase L=6 | 1.0% | 1.0% | 2.1% | 38.7% | 9.4% | 0.0% |
| paraphrase L=12 | 3.1% | 1.0% | 1.0% | 36.1% | 6.2% | 0.0% |
Pilot interpretation:
- Token-logit pooling helps argument extraction substantially compared with context-only pooling, but it does not solve initial-value extraction.
- The shared numeric token classifier is a likely bottleneck because initial values are uniform over the full modulus while arguments occupy a restricted range.
Next action: split initial-value and argument token classifiers, and expose trace-loss weights so the hard initial-value channel can be emphasized.
2026-06-21 Alignment Bootstrap and Query Tests
Added optional attention-alignment supervision for trace-active stages. The compiler still receives only the full hidden sequence at inference time; the alignment labels are used only during bootstrap.
Key diagnostics:
| Run | Trained range | Main split | Exec | Init | Op | Arg | Program exact |
|---|---|---|---|---|---|---|---|
pilot_qwen35_attention_trace | L=3 | standard L=3 | 99.2% | 100.0% | 100.0% | 99.7% | 99.2% |
pilot_qwen35_attention_trace | L=3 | standard L=6 | 0.0% | 100.0% | 58.5% | 39.6% | 0.0% |
pilot_qwen35_generated_query_attention | L=3 | standard L=3 | 99.2% | 100.0% | 100.0% | 99.7% | 99.2% |
pilot_qwen35_generated_query_attention | L=3 | standard L=6 | 0.0% | 100.0% | 58.6% | 35.3% | 0.0% |
Interpretation:
- Attention-aligned bootstrap solves trained step slots.
- Generated step queries did not extrapolate to untrained later slots.
- Independent step queries are more appropriate for the capacity test.
Ran length-12 stress conditions:
| Run | Training template | Training range | Split | Exec | Init | Op | Arg | Program exact |
|---|---|---|---|---|---|---|---|---|
pilot_qwen35_independent_attention_len12_mixed | mixed | L=1-12 | standard L=12 | 1.6% | 100.0% | 57.7% | 47.5% | 0.0% |
pilot_qwen35_independent_attention_len12_fixed | mixed | L=12 | standard L=12 | 0.0% | 100.0% | 60.1% | 51.2% | 0.0% |
pilot_qwen35_independent_attention_len12_standard | standard | L=12 | standard L=12 | 3.1% | 100.0% | 80.5% | 82.5% | 0.8% |
pilot_qwen35_independent_attention_len12_standard_strong | standard | L=12 | standard L=12 | 5.9% | 100.0% | 86.3% | 88.9% | 4.3% |
Interpretation:
- Initial-value extraction is solved by attention-aligned bootstrap.
- Longer chains fail mainly through compounding operation and argument errors.
- Mixed-template localization is materially harder than standard-template localization under the current budget.
Decision:
- Use standard-template, independent-query, attention-aligned bootstrap for the main retention run.
- Treat longer lengths and paraphrase wording as stress tests, not as solved conditions.
2026-06-21 Main Qwen Run
Ran main_qwen35_attention_len4_retention with frozen Qwen3.5-4B features, full-sequence hidden-state caching, independent step queries, attention-aligned trace bootstrap, and standard-template length-4 training.
Final standard-template metrics:
| Variant | L=4 exec | L=8 exec | L=12 exec | L=24 exec | L=4 init | L=4 op | L=4 arg | L=4 exact |
|---|---|---|---|---|---|---|---|---|
direct | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a |
compiler_trace | 99.6% | 0.4% | 1.2% | 0.8% | 100.0% | 100.0% | 99.9% | 99.6% |
compiler_answer_only | 0.0% | 0.8% | 0.8% | 0.0% | 0.4% | 35.5% | 1.7% | 0.0% |
compiler_trace_then_answer | 99.6% | 1.6% | 1.6% | 0.4% | 100.0% | 100.0% | 99.9% | 99.6% |
Direct answer control:
| Split | Direct accuracy |
|---|---|
| standard L=4 | 1.6% |
| standard L=8 | 1.2% |
| standard L=12 | 0.0% |
| standard L=24 | 1.2% |
Paraphrase stress result:
| Variant | paraphrase L=4 exec | paraphrase L=4 init | paraphrase L=4 op | paraphrase L=4 arg |
|---|---|---|---|---|
compiler_trace | 1.6% | 94.9% | 58.0% | 54.0% |
compiler_trace_then_answer | 2.0% | 94.5% | 62.9% | 55.7% |
Main interpretation:
- A span-free inference-time compiler can be installed when trace bootstrap includes attention alignment.
- Answer-only continuation preserves the installed length-4 standard-template interface.
- Answer-only training from scratch remains at chance.
- The learned interface is narrow: later step slots and paraphrased wording do not transfer under this configuration.
Final artifacts created:
analysis/final_metrics.csvanalysis/summary.mdanalysis/figures/direct_accuracy.pnganalysis/figures/executor_accuracy.pnganalysis/figures/program_exact.pngcheckpoint_manifest.csv
Next action: write the standalone report and run the final artifact audit.
2026-06-21 Final Audit
Final artifacts created:
reports/qwen_span_free_compiler_paper.mdreports/qwen_span_free_compiler_paper.htmlcheckpoint_manifest.csv
Verification:
- Source compilation passed:
python -m py_compile src/qwen_span_free_compiler_experiment.py src/analyze_qwen_span_free_compiler.py - Checkpoint manifest validation passed for 23 saved checkpoints.
- No
.pt,.pth, or.ckptfiles are stored inside the lightweight experiment directory. - Standalone wording scan passed on the report files, README, and source.
- Compile caches were removed after verification.
Artifact sizes:
experiments/qwen_span_free_compiler/: 900Klarge_artifacts/qwen_span_free_compiler/: 274M
Conclusion:
The span-free inference-time compiler succeeds on the trained length-4 standard template when trace bootstrap includes attention alignment. The main retention row reaches 99.6% exact execution at length 4 after trace loss is removed, matching the trace-supervised compiler. The direct answer head and answer-only compiler remain at chance. The interface does not transfer to later step slots or paraphrased prompts under the tested configuration.
Figures 3
Data files 14
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
analysis/final_metrics.csv20 kBruns/main_qwen35_attention_len4_retention/results.json102 kBruns/pilot_qwen35_attention_len12_mixed/results.json18 kBruns/pilot_qwen35_attention_trace/results.json9.2 kBruns/pilot_qwen35_generated_query_attention/results.json9.2 kBruns/pilot_qwen35_independent_attention_len12_fixed/results.json18 kBruns/pilot_qwen35_independent_attention_len12_mixed/results.json18 kBruns/pilot_qwen35_independent_attention_len12_standard/results.json18 kBruns/pilot_qwen35_independent_attention_len12_standard_strong/results.json22 kBruns/pilot_qwen35_span_free/results.json77 kBruns/pilot_qwen35_split_init_trace/results.json8.7 kBruns/pilot_qwen35_token_pool_trace/results.json21 kBruns/smoke_tiny/results.json55 kBruns/smoke_tiny_token_pool/results.json32 kB
Reproduce
This entry is source code or analysis only — there is no separate run to reproduce.