CI and local qualification policy

Kiln separates inexpensive repository checks from accelerator evidence. GitHub-hosted CI can prove that selected code builds, tests pass, contracts validate, and repository policy holds. It cannot, by itself, prove that a real CUDA, ROCm, Vulkan, or Metal device is correct, stable, or fast.

Start here

A skipped job usually means its path filter or manual-dispatch condition did not match. It is not a pass. Open the workflow file and confirm the trigger before drawing a conclusion from the check list.

What a green check proves

Every check has a bounded claim:

• A CPU test proves only the behavior exercised by that test.
• A feature build proves that the selected source and toolchain compiled and linked. It does not prove on-device execution.
• A software Vulkan run can catch API and shader regressions, but it does not qualify a physical Vulkan implementation.
• A portable receipt check proves that checked-in evidence matches its schema and retained-artifact policy. It does not rerun the workload.
• A local receipt proves only the declared source, model, workload, backend, device, and pass criteria. One machine never becomes a device allowlist or a universal performance claim.

Mocks, source-text assertions, and tests that return success after skipping required hardware do not count as accelerator evidence.

Automatic checks

The repository currently uses these automatic workflows:

Path filters intentionally avoid unrelated work. When a change crosses a contract boundary that the filters do not express, run the affected checker locally and update the workflow filter in the same change.

The ordinary ci.yml path launches one automatic Linux job. It does not run a redundant cargo build before cargo test, retest unchanged leaf crates, or start a second cold Rust runner for OpenEnv. Missing diff history safely falls back to the full workspace, and a weekly scheduled run exercises the complete workspace as a safety net. Accelerator jobs require manual dispatch.

Manual backend checks

Dispatch a backend check against the exact ref under review:

gh workflow run ci.yml --ref <commit-or-branch> -f backend_build=metal
gh workflow run ci.yml --ref <commit-or-branch> -f backend_build=vulkan
gh workflow run ci.yml --ref <commit-or-branch> -f backend_build=cuda
gh workflow run ci.yml --ref <commit-or-branch> -f backend_build=rocm
gh workflow run ci.yml --ref <commit-or-branch> -f backend_build=all

The lanes do not all provide the same evidence:

The CUDA and ROCm architecture values in those workflow jobs are compile-time targets chosen to bound hosted cost. They are not runtime support policy. Backend dispatch and user-facing defaults must remain capability-driven.

Local qualification

Use Local hardware qualification for the complete new-machine, workload, validation, comparison, and publication procedure. The short form is:

1. Start from the exact source revision you intend to claim.
2. Select an existing backend-generic workload, or add one with explicit pass criteria.
3. Build before starting the device workload; do not overlap another Cargo or accelerator job.
4. Capture the environment and run the workload on the requested real device.
5. Validate the receipt, required cases, source identity, and local artifact hashes.
6. Compare only receipts with compatible workload and comparison policy.
7. Commit compact receipts and manifests; keep raw logs, traces, and profiles in ignored .qualification/ storage.

Capture an environment receipt with an operator-chosen stable host label:

python3 scripts/qualification/environment.py \
  --backend vulkan \
  --host-id <stable-host-id>

Replace the backend with cuda, rocm, or metal as appropriate. The captured device identity belongs in the receipt; do not copy it into runtime dispatch, workload admission, or product defaults.

Validate every portable qualification and serving-benchmark receipt:

mapfile -d '' receipts < <(
  find qualification/receipts -type f -name '*.json' -print0 | sort -z
)
if ((${#receipts[@]})); then
  python3 scripts/qualification/receipt.py "${receipts[@]}"
fi

mapfile -d '' benchmark_receipts < <(
  find benchmarks/receipts -type f -name '*.json' -print0 | sort -z
)
if ((${#benchmark_receipts[@]})); then
  python3 scripts/bench-concurrent-batch.py \
    --validate-receipt "${benchmark_receipts[@]}"
fi

On the qualification machine, add the strict local checks to the relevant receipt paths:

python3 scripts/qualification/receipt.py \
  --require-current-source \
  --require-local-artifacts \
  qualification/receipts/<backend>/<host-id>/*.json

A passing hardware claim requires a clean source tree, the requested real device, no skipped required case, exact source/model/workload identity, and a validated receipt.

Bounded local Cargo work

On Linux qualification hosts, run ad hoc Cargo work through scripts/cargo-bounded.sh. It serializes builds, checks available memory, reserves host memory, and contains the compiler/linker process tree within a memory-capped, no-swap cgroup.

Source-bound qualification cases run inside a PID and network boundary. Declared builds use the wrapper’s transient-service mode so the build retains an aggregate memory limit, private network, runtime limit, and control-group cleanup. Its versioned environment policy admits only the paths, locale, home, and user-service variables required for the build; ambient compiler flags, target directories, credentials, and API tokens are excluded.

Examples:

scripts/cargo-bounded.sh check --locked -p kiln-server --lib
scripts/cargo-bounded.sh test --locked \
  -p kiln-server --lib config::tests -- --test-threads=1

Always name --lib, --bin, or --test for a filtered package test. A test name filters execution, but Cargo otherwise still builds every integration test target in that package.

Manual packaging

Packaging is deliberate and is not triggered by tag creation alone. Inspect the intended tag and relevant local receipts before dispatch:

gh workflow run server-release.yml --ref kiln-vX.Y.Z
gh workflow run docker-server-release.yml --ref kiln-vX.Y.Z
gh workflow run desktop-build.yml --ref desktop-vX.Y.Z
gh workflow run runpod-image.yml --ref <commit-or-branch>

A run from the wrong ref is not release evidence. Confirm the checked-out ref, artifact version, digests or attestations, and publication result in the workflow summary.

Failure triage

Historical cost record

The automatic/manual split was introduced after measurements from completed GitHub runs on 2026-07-09. The previous Rust workflow started nine jobs for an ordinary matching change. In one representative run, accelerator compile jobs used 40.5 of 46.75 aggregate runner-minutes; CUDA and ROCm used 31.3 minutes. The first successful run after the split used 3m52s wall time and about 4m36s of aggregate hosted time.

Those figures explain the policy; they are not a current performance promise. Re-measure representative runs before using them for capacity or cost planning.

On 2026-07-31, a representative OpenEnv push fanned out to seven workflows; the Rust workflow alone used about 7.2 aggregate runner-minutes and the standalone OpenEnv lane compiled an overlapping workspace again. The current router consolidates the three static-policy workflows into one runner and the three Rust jobs into one affected-package job, with the pinned live matrix in that same warm job. Weekly upstream-edge coverage remains independent.

The legacy performance-fixture workflow remains manual while structured local qualification replaces it. Its machine-bound thresholds are compatibility fixtures, not backend support policy or portable performance baselines.

Ownership

• GitHub Actions owns inexpensive syntax, CPU behavior, dependency and repository policy, and portable evidence validation.
• The operator of each declared local machine owns backend correctness, latency, memory, soak, and performance evidence from that machine.
• Release operators own checking the relevant receipt set and dispatching packaging from the intended tag.
• Contributors own updating path filters when a new source begins to affect an existing contract.