Qwen3.5-4B Operator Inventory Scaling Stress
The one idea you need
Picture a recipe with two blanks, each filled from a spice rack of 512 interchangeable spices. That is a quarter-million possible pairings, and many taste identical on the handful of dishes you sampled, so the samples alone cannot reveal which pairing is truly the right one.
The question
When you build little programs by filling blanks from a growing toolbox of number-crunching functions, does scaling to hundreds of tools break your ability to find the right program?
What we found
It depends on what you mean by break. The correct program always stays inside the pile you search, even with two blanks and 512 tools. But picking it out from a few examples alone drops to under half, versus always solved with a single blank, and the search balloons to over a quarter-million combinations per problem. The wall is ambiguity and compute cost, not reachability.
Why it matters
Do not brute-force two-blank program search over a large toolbox: cost grows with the square of the tool count. Train a small model to shortlist a few likely tools, then ask a couple of targeted examples to break the remaining ties.
On this page
Results at a glance 4
How to read
The horizontal axis is toolbox size, from a handful of tools up to 512. The vertical axis is how often a few examples alone pick the exact right program; higher is better. One line is single-blank programs, the other is two-blank.
Takeaway → Single-blank programs stay near always-solved throughout, while two-blank programs sag toward half as the toolbox grows and identical-looking combinations pile up.
Data table
| Operator library size | one-hole templates | two-hole templates |
|---|---|---|
| 8 | 100% | 75% |
| 16 | 100% | 62.5% |
| 32 | 91.7% | 50% |
| 64 | 87.5% | 45.8% |
| 128 | 83.3% | 37.5% |
| 256 | 91.7% | 54.2% |
| 512 | 100% | 45.8% |
Numbers from reports/library_depth_summary.csv
Technical framing
Zero-query program selection as the operator library grows — The target always stays reachable, but zero-query selection for two-hole programs falls to ~46% at 512 operators as ambiguity grows.
How to read
The horizontal axis is toolbox size. The vertical axis is how often the correct two-blank program still falls inside a capped shortlist of combinations you agree to check; higher is better. Each line is a different shortlist size.
Takeaway → Every capped shortlist catches the answer for small toolboxes, then collapses toward zero once the toolbox reaches the dozens, so a fixed shortlist stops working.
Data table
| Operator library size | budget 1024 | budget 4096 | budget 16384 |
|---|---|---|---|
| 8 | 100% | 100% | 100% |
| 16 | 100% | 100% | 100% |
| 32 | 100% | 100% | 100% |
| 64 | 37.5% | 100% | 100% |
| 128 | 0% | 29.2% | 100% |
| 256 | 4.2% | 8.3% | 37.5% |
| 512 | 0% | 4.2% | 12.5% |
Numbers from reports/prefix_summary.csv
Technical framing
Would a small fixed candidate budget still contain the target? (two-hole) — Naive fixed budgets collapse for two-hole search past 64 operators (0-12.5% at 512), quantifying what a learned top-k shortlister must beat.
How to read
The horizontal axis is how many extra targeted examples you request. The vertical axis is how often you then pick the exact right two-blank program at the largest toolbox; higher is better. One line is a practical question-picking rule, one is the best-possible questions.
Takeaway → Both lines climb sharply from under half with no extra examples to the seventies or eighties after three, so a few well-chosen clues recover most of the lost accuracy.
Data table
| Query budget | max-split heuristic | oracle elimination |
|---|---|---|
| 0 | 45.8% | 45.8% |
| 1 | 62.5% | 66.7% |
| 2 | 66.7% | 83.3% |
| 3 | 70.8% | 83.3% |
Numbers from reports/active_summary.csv
Technical framing
Active queries recover selection at 512 operators (two-hole) — A few active queries lift selection from 46% to 71-83%, but only after the full N^2 candidate set has already been enumerated.
How to read
Bars show how often examples pick the exact right two-blank program, at two toolbox sizes, for two program shapes: a scale-and-remainder arithmetic shape versus a compare-two-things-and-choose shape. Taller bars are better.
Takeaway → The compare-and-choose shape sits near zero to a quarter while the arithmetic shape stays two-thirds or better, because comparing reveals too little to separate the look-alikes.
Data table
| Operator library size | pair_affine_mod | pair_compare_gate |
|---|---|---|
| 128 operators | 75% | 0% |
| 512 operators | 66.7% | 25% |
Numbers from reports/library_template_summary.csv
Technical framing
Which two-hole template is hard: low-information comparison gates — The comparison-gate template leaves thousands of visible-consistent candidates, so zero-query selection collapses relative to affine-mod.
In the author’s words from the Report · “Summary”
Target coverage remains 100.0% for two-hole programs because the target is still in the library and visible cases retain it, but zero-query selection drops to 45.8% as visible-consistent ambiguity grows. The practical bottleneck is now compute budget and residual ambiguity, not target reachability. For two-hole programs at 512 operators, fixed prefix coverage is 1024: 0.0%, 4096: 4.2%, 16384: 12.5%; a deployable top-k shortlister must preserve target coverage while avoiding full N^2 enumeration.
Overview
Standalone scaling stress test for typed operator inventory search.
The experiment expands a same-signature operator library from single digits to hundreds of list[int] -> int operators, then compares exhaustive operator-hole search across one-hole and two-hole program templates. The goal is to find where full inventory enumeration remains cheap and identifiable, and where a Qwen3.5-4B inventory-conditioned top-k shortlister would become necessary.
Large artifacts, if any are produced later, belong outside this directory:
/workspace/large_artifacts/qwen35_4b_operator_inventory_scaling_stress
Layout
configs/: experiment configuration.data/: generated scaling benchmark records.logs/: chronological experiment log.reports/: result JSON, CSVs, figures, and final report.run_logs/: captured command output.scripts/: dataset, evaluation, and reporting entry points.src/: standalone operator library, task generator, and vectorized search.
Main Commands
python scripts/build_dataset.py
python scripts/eval_scaling.py --data data/operator_scaling_eval.jsonl --output reports/operator_scaling_results.json
python scripts/make_report.pyReport
Rendered from reports/qwen35_4b_operator_inventory_scaling_stress_report.md
Summary
This standalone no-training experiment scales a same-signature list[int] -> int operator inventory from 8 to 512 operators. It measures one-hole templates, where exhaustive search scales as N, and two-hole templates, where exhaustive search scales as N^2.
At 512 operators, one-hole exhaustive search enumerates 512 candidates per record, while two-hole exhaustive search enumerates 262144 candidates per record. Target coverage remains 100.0% for two-hole programs because the target is still in the library and visible cases retain it, but zero-query selection drops to 45.8% as visible-consistent ambiguity grows.
The practical bottleneck is now compute budget and residual ambiguity, not target reachability. For two-hole programs at 512 operators, fixed prefix coverage is 1024: 0.0%, 4096: 4.2%, 16384: 12.5%; a deployable top-k shortlister must preserve target coverage while avoiding full N^2 enumeration.
Key Scaling Rows
| library | holes | records | raw candidates | target visible % | oracle % | selected % | visible candidates | target rank p90 |
|---|---|---|---|---|---|---|---|---|
| 8 | 1 | 24 | 8 | 100.00 | 100.00 | 100.00 | 1.00 | 7.00 |
| 8 | 2 | 24 | 64 | 100.00 | 100.00 | 75.00 | 2.42 | 43.10 |
| 64 | 1 | 24 | 64 | 100.00 | 100.00 | 87.50 | 1.96 | 59.10 |
| 64 | 2 | 24 | 4096 | 100.00 | 100.00 | 45.80 | 300.79 | 3027.10 |
| 512 | 1 | 24 | 512 | 100.00 | 100.00 | 100.00 | 26.54 | 459.80 |
| 512 | 2 | 24 | 262144 | 100.00 | 100.00 | 45.80 | 8695.79 | 226195.10 |
Template Breakdown
The hard case is the low-information comparison template. At 512 operators, pair_compare_gate leaves far more visible-consistent candidates than pair_affine_mod, and zero-query selection falls accordingly.
| library | template | selected % | visible candidates | target rank p90 |
|---|---|---|---|---|
| 128 | pair_affine_mod | 75.00 | 2.83 | 13415.40 |
| 128 | pair_compare_gate | 0.00 | 3729.92 | 10655.40 |
| 512 | pair_affine_mod | 66.70 | 357.50 | 229748.60 |
| 512 | pair_compare_gate | 25.00 | 17034.08 | 220053.90 |
Active Query Diagnostic
For two-hole programs, active querying reduces ambiguity but does not remove the search-cost issue. It helps after the full candidate set has already been generated and filtered. The oracle-elimination curve is a ceiling on query choice quality; max-split is the deployable heuristic.
| library | policy | budget | selected % | candidate count |
|---|---|---|---|---|
| 64 | active_max_split | 0 | 45.80 | 300.79 |
| 64 | active_max_split | 1 | 50.00 | 138.29 |
| 64 | active_max_split | 2 | 70.80 | 71.71 |
| 64 | active_max_split | 3 | 83.30 | 37.54 |
| 64 | oracle_elimination | 0 | 45.80 | 300.79 |
| 64 | oracle_elimination | 1 | 70.80 | 20.83 |
| 64 | oracle_elimination | 2 | 87.50 | 9.42 |
| 64 | oracle_elimination | 3 | 95.80 | 6.33 |
| 512 | active_max_split | 0 | 45.80 | 8695.79 |
| 512 | active_max_split | 1 | 62.50 | 4199.46 |
| 512 | active_max_split | 2 | 66.70 | 1990.25 |
| 512 | active_max_split | 3 | 70.80 | 1110.21 |
| 512 | oracle_elimination | 0 | 45.80 | 8695.79 |
| 512 | oracle_elimination | 1 | 66.70 | 1451.33 |
| 512 | oracle_elimination | 2 | 83.30 | 573.33 |
| 512 | oracle_elimination | 3 | 83.30 | 403.04 |
Fixed Candidate-Budget Diagnostic
The fixed-prefix diagnostic is deliberately simple: it asks whether a small canonical candidate budget would contain the target before semantic shortlisting. The answer becomes no as two-hole search grows, which quantifies the budget a learned shortlister has to beat.
Decision
The target remains reachable under exhaustive inventory search through 512 operators, including two-operator compositions. The reason to train an inventory-conditioned Qwen3.5-4B sketcher is now sharply defined: top-k shortlisting for large two-hole libraries. The training target should be coverage at fixed candidate budgets, especially 1024, 4096, and 16384, with active querying retained as a post-shortlist disambiguator.
Artifacts
- Dataset:
data/operator_scaling_eval.jsonl - Dataset manifest:
data/dataset_manifest.json - Full result JSON:
reports/operator_scaling_results.json - CSVs:
reports/library_depth_summary.csv,reports/library_template_summary.csv,reports/target_bucket_summary.csv,reports/prefix_summary.csv,reports/active_summary.csv - Large artifacts:
/workspace/large_artifacts/qwen35_4b_operator_inventory_scaling_stress
Experiment log 4
Show the running log (4 entries, 2026-06-24)
2026-06-24
Initialized a standalone operator inventory scaling stress test.
Design commitments:
- Generate a large library of same-signature
list[int] -> intoperators. - Sweep inventory sizes from 8 through 512 operators.
- Include both one-hole and two-hole templates so candidate count scales as
NandN^2. - Keep the first run no-training: measure exhaustive search cost, coverage, visible ambiguity, active-query lift, and fixed-budget prefix coverage.
- Store any future model artifacts outside the experiment directory.
Initial implementation:
- Added a standalone library generator with 512 same-signature operators across core, order-statistic, count, modular, and bounded aggregate families.
Added four templates:
single_modsingle_offsetpair_affine_modpair_compare_gate
- Added vectorized exhaustive search over one-hole and two-hole candidate ids.
- Added active query evaluation for max-split and oracle-elimination policies.
- Added reporting with CSV summaries and PNG figures.
Smoke Validation
Commands:
python -m py_compile scripts/*.py src/*.py
python scripts/build_dataset.py --library-sizes 8,16 --records-per-template 2 > run_logs/dataset_smoke_console.log 2>&1
python scripts/eval_scaling.py \
--data data/operator_scaling_eval.jsonl \
--output reports/_smoke_operator_scaling_results.json \
> run_logs/eval_smoke_console.log 2>&1Result:
- Syntax check passed.
- Smoke dataset contained 16 records.
- Candidate rows: 16.
- Active rows: 128.
- Target-visible coverage was 100% for both one-hole and two-hole records.
- Two-hole ambiguity appeared immediately: average visible-consistent candidates rose from 1.5 at 8 operators to 11.75 at 16 operators.
Full Scaling Sweep
Commands:
python scripts/build_dataset.py > run_logs/dataset_build_console.log 2>&1
python scripts/eval_scaling.py \
--data data/operator_scaling_eval.jsonl \
--output reports/operator_scaling_results.json \
> run_logs/eval_scaling_console.log 2>&1
python scripts/make_report.py > run_logs/report_generation_console.log 2>&1Dataset:
- 336 total records.
- Library sizes: 8, 16, 32, 64, 128, 256, 512.
- 48 records per library size.
- 24 one-hole records and 24 two-hole records per library size.
- 6 visible cases, 18 hidden cases, and 48 query-pool cases per record.
- 512 generated same-signature operators.
Primary result by library size and hole count:
| library | holes | records | raw candidates | target visible | oracle hidden-all | selected hidden-all | visible candidates |
|---|---|---|---|---|---|---|---|
| 8 | 1 | 24 | 8 | 100.0% | 100.0% | 100.0% | 1.00 |
| 8 | 2 | 24 | 64 | 100.0% | 100.0% | 75.0% | 2.42 |
| 64 | 1 | 24 | 64 | 100.0% | 100.0% | 87.5% | 1.96 |
| 64 | 2 | 24 | 4096 | 100.0% | 100.0% | 45.8% | 300.79 |
| 512 | 1 | 24 | 512 | 100.0% | 100.0% | 100.0% | 26.54 |
| 512 | 2 | 24 | 262144 | 100.0% | 100.0% | 45.8% | 8695.79 |
Two-hole template breakdown:
| library | template | selected hidden-all | visible candidates |
|---|---|---|---|
| 128 | pair_affine_mod | 75.0% | 2.83 |
| 128 | pair_compare_gate | 0.0% | 3729.92 |
| 512 | pair_affine_mod | 66.7% | 357.50 |
| 512 | pair_compare_gate | 25.0% | 17034.08 |
Active-query result on two-hole records:
| library | policy | budget 0 | budget 1 | budget 2 | budget 3 |
|---|---|---|---|---|---|
| 64 | max-split | 45.8% | 50.0% | 70.8% | 83.3% |
| 64 | oracle-elimination | 45.8% | 70.8% | 87.5% | 95.8% |
| 512 | max-split | 45.8% | 62.5% | 66.7% | 70.8% |
| 512 | oracle-elimination | 45.8% | 66.7% | 83.3% | 83.3% |
Fixed-prefix coverage on two-hole records:
| library | 1024 candidates | 4096 candidates | 16384 candidates |
|---|---|---|---|
| 64 | 37.5% | 100.0% | 100.0% |
| 128 | 0.0% | 29.2% | 100.0% |
| 256 | 4.2% | 8.3% | 37.5% |
| 512 | 0.0% | 4.2% | 12.5% |
Interpretation:
- Exhaustive target reachability remains 100% through 512 operators because the target is in the inventory and full search enumerates all candidates.
- The real failure mode is the combination of quadratic candidate growth and residual visible ambiguity.
pair_compare_gateis the stress case: binary outputs leave thousands of visible-consistent two-operator candidates at large library sizes.- Active querying helps, but it operates after full enumeration and does not solve the candidate-budget problem.
- The next trained experiment should target Qwen3.5-4B inventory-conditioned top-k shortlisting for two-hole programs, measured by coverage at fixed budgets of 1024, 4096, and 16384 candidates.
Generated artifacts:
- Report:
reports/qwen35_4b_operator_inventory_scaling_stress_report.md - Full results:
reports/operator_scaling_results.json - CSV summaries:
reports/library_depth_summary.csv,reports/library_template_summary.csv,reports/target_bucket_summary.csv,reports/prefix_summary.csv,reports/active_summary.csv - Figures:
reports/figures/*.png
Final Audit
Commands/checks:
python -m py_compile scripts/*.py src/*.py
find . -type d -name __pycache__ -prune -exec rm -rf {} +
find . -type f -size +50M -print
du -sh . /workspace/large_artifacts/qwen35_4b_operator_inventory_scaling_stressAudit result:
- Final syntax check passed.
- No Python cache directories remain.
- No file larger than 50 MB is present in the experiment directory.
- Experiment directory size: 4.0 MB.
- External large-artifact directory size: 0.
- A dynamic text scan against sibling experiment directory names found no references.
- No standalone-forbidden temporal references were found.
- PNG figures were opened and verified with PIL.
Figures 5
Data files 6
Result tables and metrics copied from the experiment folder — preview inline or open the raw file.
reports/_smoke_operator_scaling_results.json101 kBreports/active_summary.csv6.1 kBreports/library_depth_summary.csv1.8 kBreports/library_template_summary.csv3.4 kBreports/prefix_summary.csv1.2 kBreports/target_bucket_summary.csv4.0 kB
Reproduce
Run steps are documented inside the experiment folder (README and scripts).