Complete configuration reference

This document is the canonical operator reference for Kiln's typed server configuration. It describes the configuration accepted by the current KilnConfig and RequestLogConfig implementations, including current defaults, validation, mechanically derived environment overrides, and runtime provenance.

The machine-readable contract is contracts/kiln-config-v1.schema.json. The documentation website renders its complete field reference directly from that schema. python3 scripts/check_config_schema.py --self-test verifies that the schema, these tables, and kiln.example.toml agree.

Use kiln config --file kiln.toml --json for the complete hardware-free effective startup object. A running server publishes its immutable startup snapshot in GET /v1/config under effective_configuration, beside the backend-derived and live capacity diagnostics that can only be resolved after hardware selection.

Kiln's intended public environment-variable naming rule is mechanical:

KILN_<SECTION>_<FIELD>

Both the section and field are converted to uppercase snake case. For example, server.host maps to the canonical target KILN_SERVER_HOST, and request_log.max_file_bytes maps to KILN_REQUEST_LOG_MAX_FILE_BYTES.

Every fixed field with an environment override implements its canonical name. The "Alternate env spelling(s)" column is deliberately none for every field:

• implemented means the mechanically derived canonical name works;
• none means there is no supported environment override for that field;
• target only; not implemented means setting the mechanically derived name currently has no effect because the field is intentionally config-file-only.

There are no environment aliases. Retired non-canonical names are ignored, so they cannot silently override the typed configuration. The complete migration index maps every retired public spelling to its canonical replacement.

Do not assume that an arbitrary KILN_* name is public configuration. The repository contains internal diagnostics, qualification switches, kernel experiments, and build/toolchain inputs. Only the canonical names listed here are supported public field overrides.

The generated Runtime Environment Inventory catalogs every direct environment read and mutation in crate-owned source, classifies its owner boundary, and exposes the remaining typed-config migration queue. It is an engineering inventory, not a second list of supported settings.

Resolution and startup

Config file selection

Kiln selects at most one TOML file, in this order:

1. The explicit path supplied by the command, such as kiln --config /path/kiln.toml serve.
2. The path in KILN_CONFIG.
3. ./kiln.toml, if it exists in the process working directory.
4. No file; built-in defaults are used.

An explicitly selected file that cannot be read or parsed is a startup error. KILN_CONFIG selects the file; it is not an override for a TOML field and is not included in the field counts below.

Value precedence

Effective startup values resolve in this order, highest priority first:

1. Typed CLI overrides applied after environment resolution (serve --served-model-id and serve --eval-mode).
2. The mechanically derived canonical environment name listed here.
3. The selected TOML file.
4. The built-in default.

The -v/-vv and --quiet logging flags override logging.level for the running command. RUST_LOG, when present, takes precedence over that resolved level when the tracing filter is built.

Environment resolution happens once at startup. Restart the server after changing a file or environment value. Model, training, and request execution receive resolved policy rather than re-reading public configuration from the process environment.

Strict failure behavior

The top-level document and every typed section reject unknown fields. This includes [request_log], [eval], [agent], [teachers], and each dynamic teacher credential. A misspelled key does not fall back to a default. The one intentional open subtree is agent.self_improve, whose structured request body is validated later by the self-improvement subsystem.

TOML type errors, invalid enum values, malformed URLs, out-of-range values, and semantically invalid combinations stop startup. Present centralized environment overrides must be valid UTF-8 and must parse successfully; malformed values stop startup and the diagnostic names the variable and raw value. The centralized loader's boolean overrides accept, case-insensitively and with surrounding whitespace ignored:

true:  true, 1, yes, on
false: false, 0, no, off

Optional TOML values are represented by omitting the field; TOML has no null literal. Environment-specific clearing syntax is documented per field.

Validate without starting the server:

kiln config --file kiln.toml
kiln config --file kiln.toml --backend rocm

This command uses the same typed loader and environment precedence as serving, but its pretty output is a summary rather than a complete effective-config dump. The optional --backend cpu|cuda|rocm|metal|vulkan selector resolves that target's static scheduling capabilities and applies the same actor-prefill startup validator as kiln serve. It does not enumerate devices, probe memory, open an accelerator, or load model weights.

Coverage summary

The accepted TOML surface contains 16 top-level sections and 112 fixed leaf fields. Dynamic teachers.credentials.<id> entries add two leaf fields per credential. Of the 112 fixed fields:

• 107 implement the canonical mechanical environment name;
• 5 are config-file-only and have no environment override;
• no field has an alternate environment spelling.

The tables below cover all 112 fixed fields and both dynamic credential fields. The migration index separately tracks all 82 former public spellings; entries marked removed are inert and intentionally have no replacement field. Unknown TOML fields are rejected, including the removed speculative.enabled and streaming_prefill.enabled toggles. Their canonical replacement fields are documented below.

[server]

Serving profiles

The profile owns GPU mutation and admission policy; individual requests cannot override it.

The default stable profile is the normal full product: inference, training, adapter transitions, memory management, and guarded graph capture do not need a special opt-in. experimental is reserved for backend routes under an explicit correctness quarantine; it is not a faster user mode.

The only profile-gated configuration is an offline forced KV resize:

server.debug_model_state is not an execution experiment: it only exposes the trusted diagnostics route and does not alter inference, cache, training, or evaluation semantics. It is therefore source-tracked and restart-bound but not profile-gated. Logging level debug is likewise observability, not an execution selector.

[accelerator]

This section owns process-lifetime accelerator execution behavior that must be fixed before the primary device context or model runner is created. The resolved object uses schema kiln.accelerator-runtime-policy.v16. Startup, kiln config, GET /v1/config, /health, trusted debug state, and the dashboard all report the same configured/effective/source values, plus the compiled Vulkan kernel-policy schema ID; lower model, tensor, and kernel paths do not re-read these public environment names.

accelerator.kt_api_mode replaces the former KILN_USE_KT_API_*, KILN_DISABLE_KT_API_*, and KILN_USE_KT_PAGED_KV_* debugging switches. Those per-operation names are not compatibility aliases: they created order-dependent route combinations that could not be represented, validated, or reported as one startup policy. Use the typed field or its mechanically derived canonical environment name instead.

accelerator.cuda_kernel_profile owns these twenty-five CUDA model/backend decisions:

The CUDA-specific variables KILN_DISABLE_CUDA_FULL_ATTN_QKV_IN_PROJ, KILN_DISABLE_CUDA_GDN_AB_IN_PROJ, KILN_DISABLE_CUDA_GDN_PREFILL_AB_IN_PROJ, KILN_DISABLE_CUDA_GDN_PREFILL_GATES, KILN_DISABLE_FUSED_GDN_DECODE, KILN_DISABLE_GDN_DECODE_UNEXPANDED_QK, KILN_DISABLE_CUDA_GDN_DECODE_QK_NORM_RECURRENT, KILN_DISABLE_CUDA_GDN_DECODE_QK_NORM_RECURRENT_RMSNORM, KILN_DISABLE_CUDA_LORA_DECODE_ADD, and KILN_DISABLE_GDN_FULL_CHUNK_FORWARD_MULTIBLOCK no longer control CUDA and are not aliases. The same is true of the former CUDA/generic switches for direct paged decode, rotary QK, decode QKV preparation, MLP fusion, attention sigmoid fusion, RMSNorm forward/backward, L2 QK normalization, dynamic-length paged decode, GDN pre-permute, training MLP chunking, and split Q/gate training. Use the complete profile; individual route combinations are not a supported product configuration.

accelerator.cuda_marlin_profile owns the complete CUDA Marlin weight-layout choice:

The former KILN_W4A16 and KILN_W4A16_GDN_OUT_PROJ switches are removed, not aliases. KILN_DISABLE_PARALLEL_PACK is also removed: production packing always uses the established parallel path, while serial comparisons belong in an explicit benchmark harness. Marlin remains CUDA-only; selecting a profile does not itself claim model-quality, correctness, memory, or performance qualification for a particular NVIDIA system.

accelerator.cuda_flash_backward_mode is the sole CUDA FlashAttention backward accumulation authority. fast is the historical default; deterministic selects split accumulation. The former KILN_FLASH_ATTN_BWD_DETERMINISTIC spelling is removed rather than retained as an alias. This setting concerns CUDA training backward only and does not imply that server.deterministic makes every accelerator kernel bitwise deterministic.

Two unused or unsafe-to-retune training-kernel controls are deleted without replacement. KILN_DISABLE_OPD_LOSS_KERNEL had no live caller. KILN_FLCE_ACTIVE_ROW_TILE is fixed at the established bounded 4,096 active rows; unsupported shapes continue through the existing backend fallback.

accelerator.metal_kernel_profile owns these forty-six Metal backend decisions:

The former Metal switches are removed, not accepted as aliases. They were: KILN_DISABLE_METAL_SDPA, KILN_DISABLE_METAL_SDPA_FULL, KILN_DISABLE_METAL_CONV1D_PREFILL, KILN_DISABLE_FUSED_CONV1D, KILN_DISABLE_METAL_FUSED_CONV1D, KILN_DISABLE_GDN_KERNEL, KILN_DISABLE_FUSED_GDN_GATES, KILN_DISABLE_METAL_GDN_GATES, KILN_DISABLE_METAL_GDN_FORWARD_SUBSTITUTION, KILN_DISABLE_METAL_GDN_RECURRENT, KILN_DISABLE_METAL_GDN_DECODE_GATES_RECURRENT, KILN_DISABLE_METAL_GDN_DECODE_GATES_RECURRENT_RMSNORM, KILN_DISABLE_METAL_GATED_RMSNORM, KILN_DISABLE_METAL_GDN_QK_NORM, KILN_DISABLE_METAL_GDN_QKV_CONV_NORM, KILN_DISABLE_METAL_GDN_PREFILL_QKV_CONV_SPLIT, KILN_DISABLE_METAL_GDN_PREFILL_DECAY_RECURRENT, KILN_DISABLE_METAL_GDN_PREFILL_AB_IN_PROJ, KILN_DISABLE_RMSNORM_KERNEL, KILN_DISABLE_METAL_RMSNORM, KILN_DISABLE_METAL_MLP_GATE_UP_FUSION, KILN_DISABLE_METAL_MLP_GATE_UP_ROW_PAIR, KILN_DISABLE_METAL_MLP_GATE_UP_ROW_QUAD, KILN_DISABLE_METAL_MLP_GATE_UP_ROW_TRIPLE, KILN_DISABLE_METAL_MLP_GATE_UP_ROW_QUAD_VECTOR_LOAD, KILN_DISABLE_METAL_MLP_GATE_UP_SERIAL_VECTOR_LOAD, KILN_DISABLE_METAL_MLP_GATE_UP_SERIAL_DEDICATED, KILN_DISABLE_METAL_MLP_SILU_MUL, KILN_DISABLE_METAL_ATTN_GATE_FUSION, KILN_DISABLE_METAL_FUSED_QKV_PROJ, KILN_DISABLE_METAL_LORA_DELTA_DECODE, KILN_DISABLE_METAL_GDN_IN_PROJ_FUSION, KILN_DISABLE_METAL_GDN_IN_PROJ_ROW_PAIR, KILN_DISABLE_METAL_GDN_IN_PROJ_ROW_QUAD, KILN_DISABLE_METAL_GDN_IN_PROJ_ROW_TRIPLE, KILN_DISABLE_METAL_GDN_IN_PROJ_SERIAL_VECTOR_LOAD, KILN_DISABLE_METAL_GDN_IN_PROJ_SERIAL_X2_LOAD, KILN_ENABLE_METAL_LM_HEAD_ARGMAX, KILN_DISABLE_METAL_LM_HEAD_ARGMAX, KILN_DISABLE_METAL_LM_HEAD_ARGMAX_ROWS, KILN_DISABLE_METAL_LM_HEAD_ARGMAX_GPU_REDUCE, KILN_DISABLE_METAL_LM_HEAD_SAMPLE, KILN_DISABLE_METAL_PAGED_ATTN_DECODE_CONTIGUOUS, KILN_DISABLE_METAL_PAGED_KV_WRITE_TOKEN_MAJOR, KILN_DISABLE_METAL_TRANSPOSED_COOP_GEMV, KILN_DISABLE_METAL_TRANSPOSED_COOP_GEMV_TILE8, KILN_DISABLE_METAL_TRANSPOSED_COOP_GEMV_TILE16, KILN_DISABLE_METAL_TRANSPOSED_COOP_GEMV_ROW_PAIR, KILN_DISABLE_METAL_TRANSPOSED_COOP_GEMV_ROW_QUAD, KILN_DISABLE_METAL_TRANSPOSED_COOP_GEMV_ROW_QUAD_TILE8, and KILN_DISABLE_METAL_TRANSPOSED_COOP_GEMV_ROW_TRIPLE_TILE8. The profile is intentionally closed: use a whole-profile comparison plus route telemetry and a focused reproduction rather than reconstructing arbitrary per-kernel combinations.

Fixed model, loader, and fallback policy

The following former unsectioned switches are removed without aliases. They were debug or rollback controls, not stable product configuration, and allowed behavior to change after typed startup validation. Their replacement behavior is fixed or owned by the typed backend profile shown above.

server.deterministic remains the public deterministic control, with KILN_SERVER_DETERMINISTIC as its mechanically derived environment target and no alternate spelling. Standalone kiln-tensor consumers do not read process environment for this policy; without an owning application calling the typed configuration hook, deterministic execution defaults to false.

accelerator.rocm_kernel_profile replaces the model-layer ROCm controls KILN_DISABLE_ROCM_ATTN_DECODE_QKV_PREP, KILN_ENABLE_ROCM_GDN_STREAMING_FASTPATHS, KILN_DISABLE_ROCM_LONG_FLASH_ATTN, KILN_ROCM_FLASH_NATIVE_RECTANGULAR_CAUSAL, KILN_DISABLE_ROCM_TRAINING_MLP_CHUNKING, KILN_ROCM_TRAINING_MLP_CHUNK_TOKENS, KILN_DISABLE_ROCM_W8_SAMPLED_LM_HEAD, KILN_DISABLE_ROCM_SPLIT_Q_GATE_TRAINING, KILN_ROCM_SPLIT_Q_GATE_ROW_TILE_TOKENS, KILN_DISABLE_ROCM_SPLIT_Q_GATE_F32_OUTPUT, KILN_DISABLE_ROCM_GQA_SDPA_F32, KILN_ROCM_TAPE_FLASH_MATERIALIZED, KILN_ROCM_W8A16, KILN_ROCM_W8A8, KILN_ROCM_W8A8_SAMPLED_LM_HEAD, and KILN_DISABLE_ROCM_W8_SWIGLU, plus the lower-level tensor spellings KILN_DISABLE_ROCM_SPLIT_PAGED_ATTN, KILN_ROCM_PAGED_ATTN_SPLIT_TOKENS, KILN_ROCM_PAGED_ATTN_MAX_SPLITS, KILN_DISABLE_ROCM_GQA_PAGED_ATTN, KILN_DISABLE_ROCM_GQA_D128_PARALLEL, KILN_DISABLE_ROCM_GQA_D256_PARALLEL, KILN_DISABLE_ROCM_CONCAT_SAFE_ROW_ASSEMBLY, KILN_ROCM_CONCAT_AXIS0_ROW_COPY, KILN_ROCM_IS_FINITE_LARGE_HOST_SCAN, KILN_ROCM_IS_FINITE_HOST_SCAN_ELEMENTS, and KILN_ROCM_RMSNORM_ROW_TILE_ROWS. None is an alias. The two default-off investigation switches for streaming GDN and materialized tape flash have no profile leaf because no qualified product uses them. The default-off axis-zero concat experiment and its dead vectorized kernel were also deleted rather than promoted.

The context-owned flash-attention subsection replaces or deletes all 50 former flash build/runtime spellings: KILN_ROCM_ATTENTION_COOPERATIVE_YIELD, KILN_ROCM_ATTENTION_YIELD_MS, KILN_ROCM_DISABLE_CK_FMHA, KILN_ROCM_ENABLE_CK_FMHA, KILN_ROCM_ENABLE_CK_FMHA_FWD, KILN_ROCM_F32_MATMUL_INNER_TILE, KILN_ROCM_FLASH_BWD_PRECOMPUTE_DELTA, KILN_ROCM_FLASH_BWD_PRECOMPUTE_DELTA_MAX_SEQ, KILN_ROCM_FLASH_CK, KILN_ROCM_FLASH_COLLAPSED_GQA_BWD, KILN_ROCM_FLASH_DISABLE_CK_BWD_COLLAPSED_GQA, KILN_ROCM_FLASH_DISABLE_WMMA_GQA_QBLOCK, KILN_ROCM_FLASH_DISABLE_WMMA_GQA_R64K16, KILN_ROCM_FLASH_DISABLE_WMMA_GQA_R64K32, KILN_ROCM_FLASH_DISABLE_WMMA_GQA_R64K32_LOG2, KILN_ROCM_FLASH_MATMUL_BWD, KILN_ROCM_FLASH_NATIVE_BWD, KILN_ROCM_FLASH_NATIVE_BWD_FORCE, KILN_ROCM_FLASH_NATIVE_BWD_LONG_MIN_SEQ, KILN_ROCM_FLASH_NATIVE_BWD_MAX_SEQ, KILN_ROCM_FLASH_NATIVE_DIRECT_COLLAPSED_GQA_BWD, KILN_ROCM_FLASH_NATIVE_DIRECT_COLLAPSED_GQA_QPAR, KILN_ROCM_FLASH_NATIVE_FFI_SYNC, KILN_ROCM_FLASH_NATIVE_GQA_QBLOCK, KILN_ROCM_FLASH_NATIVE_GQA_QBLOCK_MIN_SEQ, KILN_ROCM_FLASH_NATIVE_KEYSPLIT, KILN_ROCM_FLASH_NATIVE_KEYSPLIT_MIN_SEQ, KILN_ROCM_FLASH_NATIVE_KEY_TILE, KILN_ROCM_FLASH_NATIVE_MAX_SEQ, KILN_ROCM_FLASH_NATIVE_QUERY_TILE, KILN_ROCM_FLASH_NATIVE_RECTANGULAR_CAUSAL, KILN_ROCM_FLASH_NATIVE_SCALAR, KILN_ROCM_FLASH_NATIVE_SCALAR_FORCE, KILN_ROCM_FLASH_NATIVE_SINGLE_MAX_SEQ, KILN_ROCM_FLASH_NATIVE_STREAMING, KILN_ROCM_FLASH_NATIVE_STREAMING_FORCE, KILN_ROCM_FLASH_NATIVE_STREAMING_MIN_SEQ, KILN_ROCM_FLASH_NATIVE_TILED, KILN_ROCM_FLASH_NATIVE_TILED_FORCE, KILN_ROCM_FLASH_NATIVE_WMMA_QBLOCK, KILN_ROCM_FLASH_NATIVE_WMMA_QBLOCK_MIN_SEQ, KILN_ROCM_FLASH_ONLINE, KILN_ROCM_FLASH_ONLINE_BWD, KILN_ROCM_FLASH_ONLINE_MATMUL_BATCH_GROUP, KILN_ROCM_FLASH_USE_WMMA_GQA_R64K16, KILN_ROCM_FLASH_USE_WMMA_GQA_R64K32, KILN_ROCM_FLASH_USE_WMMA_GQA_R64K32_LOG2, KILN_ROCM_FLASH_WMMA_GQA_R64K16_MIN_SEQ, KILN_ROCM_FLASH_WMMA_GQA_R64K32_LOG2_MIN_SEQ, and KILN_ROCM_FLASH_WMMA_GQA_R64K32_MIN_SEQ. None is accepted as an alias. Positive/negative/force combinations collapse into the closed profile; cooperative sleeps and forced FFI synchronization are deleted; the unused key-split/WMMA branches and numerically incorrect CK backward route are removed rather than exposed as policy.

The generic GPU training and fused-route comparison switches in the fixed policy table above are no longer migration inputs for any backend. Their ROCm decisions remain leaves of the complete typed policy.

accelerator.full_attention_score_budget_mib replaces the undocumented KILN_FULL_ATTN_SCORE_BUDGET_MB, KILN_ROCM_FLASH_SCORE_BUDGET_MB, KILN_FULL_ATTN_SCORE_TILE_MAX_ELEMENTS, KILN_ROCM_FULL_ATTN_SCORE_ELEMENT_CAP, KILN_ROCM_FLASH_SCORE_TILE_MAX_ELEMENTS, KILN_ROCM_FLASH_ONLINE_SCORE_BUDGET_MB, KILN_ROCM_FLASH_ONLINE_QUERY_TILE, and KILN_ROCM_FLASH_ONLINE_KEY_TILE controls. Those names are intentionally not aliases. Previously, model and ROCm flash paths parsed different permissive values during execution and recomputed score budgets from changing free-memory snapshots, permitting route geometry to change between layers or requests. Policy v16 fixes one bounded ceiling before execution; the ROCm allocator governor may still reject a planned operation when its exact working set is no longer admissible, but it cannot silently shrink or expand that plan.

Vulkan device policy

kiln.vulkan-device-policy.v1 is the model/kernel boundary installed from the two typed fields above before the first logical Vulkan device is created. A same-value reinstall is idempotent; a conflicting late owner fails closed. The selected VulkanDevice retains its physical enumeration index for diagnostics. The removed GGML_VK_VISIBLE_DEVICES compatibility behavior is not a Vulkan loader contract and is not accepted as a Kiln alias. Use the typed field or its mechanically derived canonical environment name. External loader/driver selectors such as MESA_VK_DEVICE_SELECT, DRI_PRIME, and Vulkan ICD selectors remain identity remaps and cause device-scoped memory-probe validation to fail closed rather than claim the wrong physical device.

Vulkan kernel policy

Vulkan kernel selection is an immutable implementation contract, not a public configuration surface. Product execution uses kiln.vulkan-kernel-policy.v6, derived once from the selected physical device's reported Vulkan limits and memory types before model routes are projected. There is no TOML, CLI, request, environment, device-name, vendor-ID, device-ID, driver-name, or PCI-ID override for its leaves. Device identity and machine qualification data do not alter product dispatch.

Kiln negotiates the instance API exposed by the Vulkan loader, capped at 1.2. The common shader set targets Vulkan 1.0. Three tiled attention shaders use subgroup basic/arithmetic operations and target Vulkan 1.1; they are selected only when the physical device reports compute-stage support for both operations, otherwise the Vulkan 1.0 untiled shader remains active.

Policy derivation checks each shader family's local workgroup geometry, invocation count, shared-memory requirement, storage-buffer binding count, and push-constant size against VkPhysicalDeviceLimits. Workload-dependent dispatch-grid and buffer-range checks remain at dispatch. A coherent host-visible recurrent-state route additionally requires a host-visible, device-local Vulkan memory type; cached staging memory remains a separate choice. A device that cannot create the complete route set skips full prewarm and lazily creates only compatible selected pipelines.

Changing a requirement or workload crossover requires a reviewed source change, a new policy version, backend/oracle parity, performance evidence, and cross-device qualification.

The capability policy owns these decisions:

Receipts for earlier policy versions describe those historical binaries, not the current product policy, and do not by themselves qualify v6.

The former variables represented by this table, including the applicable KILN_DISABLE_VULKAN_*, KILN_ENABLE_VULKAN_*, kernel-threshold, split-K, kernel-stage profiling, model-route, packed-weight, recurrence-residency, and resident-decode controls, were deleted without aliases or replacement fields. They permitted route mixtures and permissive fallback below the typed startup boundary, and the model duplicated a different copy of several thresholds. Historical optimization documents may name those variables as experiment evidence; they do not describe current runtime controls. The research microbenchmark remains a separate executable surface and must not be treated as product configuration.

The serving library and product kernel paths no longer read residual KILN_VK_* tiling or CPU-fallback switches. The separate vulkan_decode_microbench research executable also has no KILN_* input: its complete experiment policy is a typed, fail-fast command line documented below. It is not server configuration, and invoking it does not create a product qualification receipt. Non-Vulkan internal runtime controls are tracked separately in the environment inventory.

Standalone benchmark configuration

kiln-bench loads the same typed KilnConfig as the server. --config selects the TOML file; in its absence normal config discovery applies. In particular, the [speculative] section supplies the method, draft proposal count, and draft depth. The three --spec-* arguments are explicit per-invocation overrides for controlled A/B work. The executable has no direct KILN_BENCH_*, KILN_SPEC_*, or qualification environment reads.

The speculative benchmark opt-in is deliberately not named qualification. Only a declared workload executed by scripts/qualification/run.py and a strictly checked retained receipt constitute qualification evidence. A direct benchmark command, including one using the experimental opt-in, does not.

vulkan_decode_microbench is an offline kernel research tool. It parses all arguments before opening a Vulkan device. Unknown arguments/cases, empty list items, malformed integers, and zero numeric values are fatal. Its complete base and geometry surface is:

The microbenchmark kernel-policy arguments are explicit research A/B controls; they never modify product policy:

ROCm synchronization semantics

legacy_host_barriers is the portable default. It preserves the conservative device-wide barriers around large hipBLASLt results and BF16/F32 cast boundaries, the historical active-stream waits after eager tensor operations, and a device-wide external-yield drain before generated progress is published. Choosing it retains ordering safety without relying on device identity.

stream_ordered is an explicit local-qualification mode. It omits only barriers whose call site asserts that producer and consumer are FIFO-ordered on the same HIP stream. It does not omit host readback, host-buffer lifetime, in-place mutation, allocator reclaim, graph transition, error recovery, or global-state drains. External yield waits for the primary producer stream rather than the whole device. A replayed graph records completion on its capture stream and makes the primary stream wait on that event before the external-yield stream wait, so replay work is included.

ROCm matmul route semantics

The portable defaults are disabled strided batching and f32_then_cast output. They select per-row GEMMs and F32 output followed by an on-device BF16 cast without inspecting the device model or ROCm version. enabled and native_bf16 are explicit diagnostic comparison routes.

The retired machine-shaped auto values fail parsing. The effective values are reported with source attribution by kiln config, /v1/config, health, and trusted debug state. Former force/disable and trace variables are ignored; no matmul path reads process environment during execution.

ROCm token-only LM-head qualification fixture

The default native_default profile disables the W8 token-only LM-head routes, as does portable_fallback; both retain the ordinary BF16 LM head. Historical W8/W8A8 route attribution is available only in the explicit hardware-qualification example build; it is not a server profile or environment-selectable product path. See Local Hardware Qualification for fixture behavior, telemetry, and receipt interpretation.

ROCm synchronization telemetry and quarantine

The primary ROCm context stores the immutable policy. A second caller asking for a different policy on the same device receives a startup error naming the already-installed and requested policies. This prevents two model/runtime owners from silently mixing synchronization disciplines.

Synchronization telemetry has 23 fixed reason values. /health reports device wait count, stream wait count, waited nanoseconds, and skipped count for every reason under decode_runtime.rocm_synchronization. The same object exposes cleanup_quarantined: when true, a fatal execution or synchronization failure made device state unsafe. The flag is shared by every context for that device ordinal and is process-lifetime sticky: a later recovery drain may collect diagnostics, but cannot re-enable execution or cleanup. New ROCm execution is rejected, possibly in-flight HIP resources are retained instead of being destroyed unsafely, backend admission fails closed, and the process must be restarted. Active ROCm telemetry becoming unavailable is treated as the same fail-closed health condition rather than as a false quarantine value. The central server health gate observes and latches this state before adapter, training, import, or other backend mutation admission, so those paths cannot slip through before the model runner notices the fault. This is live state, so it is reported by health and trusted debug rather than /v1/config; the latter remains the immutable policy authority. /metrics exports the same fixed dimensions as kiln_rocm_synchronizations_total, kiln_rocm_synchronization_wait_seconds_total, and kiln_rocm_synchronization_skipped_total, plus policy, availability, and kiln_rocm_cleanup_quarantined gauges. Treat the quarantine gauge as unknown unless kiln_rocm_synchronization_telemetry_available is 1. Reading these surfaces loads atomics only; it never synchronizes the device or probes the driver.

capture_rollback and error_recovery are deliberately different reasons. After a native capture error, capture_rollback may settle the device while execution admission remains open; eager continuation is allowed only after the runner also proves that its logical recurrent state was restored. A settlement failure, a logical-rollback failure, or an armed/unclassified capture guard leaving scope publishes the process-lifetime STOP state first, then uses error_recovery only as a diagnostic drain. That fatal path remains cleanup-quarantined even if the later drain succeeds and always requires a process restart.

Stream-ordered mode is not promoted by configuration alone. Qualify legacy and stream-ordered runs on the same source tree using token/logit parity, throughput at concurrency 1/8/16/32/64, p50/p95/p99/max inter-token latency, reasoned synchronization time, and stable memory plateaus before changing the default.

ROCm graph lifecycle

• disabled bypasses graph-shaped warmup, capture, and replay.
• warmup_then_eager runs one graph-shaped eager warmup, never captures, and remains eager. It exists for controlled graph-state-machine comparisons.
• lazy_capture_replay warms eagerly and may capture/replay eligible decode shapes while serving. Capture/fallback/replay state remains visible in /health.
• profile is the recommended default: stable and development serving select guarded lazy capture/replay, while drained maintenance stays eager.

Entry count and retained bytes are independent hard limits. Byte accounting deduplicates physical ROCm allocations by device pointer, counts graph-stable I/O, freeze-pointer capture arenas, each private stream's actual hipBLASLt workspace, and persistent recurrent/conv slot state once per owner. Health, trusted debug, and Prometheus split those categories and report peak bytes, evictions, admission rejections, and the number of opaque HIP graph, exec, stream, and event objects. The byte budget deliberately does not pretend that HIP exposes the allocator overhead of those opaque objects; live driver memory pressure remains a separate complementary signal.

Before native capture, the runner reclaims eligible idle owners at entry/byte saturation. If those cannot make room, it computes a deterministic fair-LRU order across active owners. The incoming candidate counts toward its owner's projected share; each step selects the most represented owner, then its oldest exact geometry with stable owner/geometry tie-breakers. Admission retires only the minimum entries needed and preserves at least one graph per active owner after the candidate, plus every live recurrent-state slot, row assignment, and decode-continuity timeline. No active graph is retired while unused global entry and byte headroom remains. The runner then performs one settled warm Record pass, measures the candidate's exact queryable allocation identities, rechecks matching-device pressure, and atomically reserves global governor headroom. Native capture begins only after that sequence admits it. A governor device-selector mismatch fails closed rather than consuming another accelerator's headroom. The Record pass necessarily allocates the candidate before its exact size is knowable. Phase telemetry independent of the model and graph-runner locks plus last/peak transient-candidate bytes make that residual exposure observable. Before any capture-parity snapshot allocation, a separate matching-device reservation covers two GDN state snapshots, the eager hidden copy, all current K/V rows, one current-row gather, and the largest exact U8 comparison mask. The later exact candidate reservation overlaps it through native capture, so the transient high-water mark includes both working sets. A candidate that alone exceeds the byte budget or cannot be accounted exactly makes its geometry non-capture-safe for this runner. Aggregate byte-budget suppression clears after ownership or budget relief; global-reservation denial retries after the matching device reports enough cached headroom. A selector mismatch remains fail-closed rather than consuming another accelerator's budget. None repeats a full warm pass on every token.

Ordinary budget eviction is deterministic least-recently-used eviction of idle owners as a unit, including every graph for the owner and its retained slot state. Those owners are always considered before fair active relief. At actual saturation, boundary pre_capture_active_fair_share settles and retires the selected active graph entries in one narrow transaction. A cache hit is never an admission candidate, and the algorithm never removes an active owner's last projected graph merely to grow another owner's share. Every completed transaction emits rocm_graph_cache_eviction with its boundary, typed reason, removed graph/slot counts, released requested bytes, and active_graph_only_owner_count. A nonzero active count identifies the settled prior-geometry retirement path rather than an idle-owner eviction. Moderate pressure permits retained replay but blocks cache growth. Tight pressure additionally reclaims eligible idle owners while preserving an active cache hit. Critical or unavailable pressure disables replay, reclaims all graphs only after device settlement, and uses eager execution only if both pre-drop and post-drop settlement prove cleanup safe. A destructor, async-free, or settlement failure activates the process-lifetime device quarantine; no reclaimed-byte success is reported and no eager retry follows.

ROCm graph observability contract

GET /v1/config and trusted GET /v1/debug/model-state expose two independent live values and a reason beside each one:

• rocm_graphs is the nonblocking full cache/counter snapshot, with rocm_graphs_unavailable_reason.
• rocm_graph_telemetry is the current/completed phase and transient-candidate snapshot, independent of the model and graph-runner locks, with rocm_graph_telemetry_unavailable_reason.

Each value is either an object with a null reason or null with exactly one of backend_without_graph_runner, model_runner_busy, model_runner_lock_poisoned, graph_runner_busy, or graph_runner_lock_poisoned. The full snapshot can use any of those five. The phase handle is stored outside both runner locks, so a real backend retains live phase telemetry through model-runner and graph-runner contention or poison; currently only backend_without_graph_runner makes that value null. The phase reason family nevertheless uses the same closed schema. Do not interpret any null value as an empty cache or zero activity.

Health flattens both authorities into decode_runtime.rocm_graphs for an operational view. Its state is enabled or disabled when full statistics are present, busy only for model_runner_busy or graph_runner_busy, and unavailable for the no-runner and poisoned-lock reasons. unavailable_reason belongs to the full statistics. phase_telemetry_available and phase_telemetry_unavailable_reason independently govern current_phase, current_phase_elapsed_micros, six phase summary objects, and last/peak transient candidate bytes. Every unavailable field is serialized as null; the server never fabricates a zero snapshot.

For a coherent point-in-time full snapshot, rocm_graphs also serializes the six completed phase summaries and last/peak transient bytes. The separate rocm_graph_telemetry object is the authoritative current-phase source and continues advancing when full statistics cannot acquire either runner lock. Health sources its phase and transient fields from that separate channel.

The phase names are exactly pre_candidate_headroom, candidate_warm, pre_native_reservation, native_capture, native_replay, and rejected_candidate_cleanup. Each completed phase object contains calls, slow, total_duration_micros, and max_duration_micros; slow means at least 100 ms. current_phase_elapsed_micros uses a monotonic clock while a phase is active. native_capture remains active through capture, the settled first launch, defensive cache admission/publication, and the blocking committed governor debit. rejected_candidate_cleanup begins only when an unretained candidate enters destruction and settlement. last_transient_candidate_bytes and peak_transient_candidate_bytes measure the peak matching-device reservation for the exact deduplicated pre-admission candidate plus the capture-parity snapshot/gather/mask working set. Already-owned recurrent slot state and opaque native objects remain excluded. These values are not retained-cache bytes; only the candidate subset is committed when admission retains the graph.

native_replay covers per-replay device input updates, the default-stream to capture-stream dependency, and native launch submission. It does not include the external-yield settlement that makes the result safe to publish; that wait is reported separately as request synchronization_ms. Direct and batched ROCm serving snapshot phase totals while holding the graph-runner lock for exactly one invocation, so their request graph_capture_ms and graph_replay_ms fields cannot absorb another concurrent request's work. A zero value means the phase ran below microsecond resolution; null means that route did not run the phase.

For batched decode, the same invocation envelope is attached only to the ready requests whose rows participated. It is intentionally a per-request overlap candidate, so summing requests can duplicate shared work by the decode width; process-work totals remain authoritative in the six lifetime phase summaries. ROCm mixed-load qualification rejects request totals above lifetime phase time times maximum observed decode width, which also catches contamination from prefilling or backpressured active slots.

Native ROCm HIP-graph capture supports both single-row decode and the contiguous BF16 multi-row route. Batched graphs are keyed by row count and bucketed attention geometry. They retain graph-stable token IDs, positions, RoPE tables, block tables, sequence lengths, KV slots, per-layer attention scratch, output hidden state, and any GDN recurrent state. Final normalization, LM head, and sampling remain eager after the graph. A width-four graph therefore reports one active graph slot across changing request cohorts; it is not an idle single-row owner. multi_row_batch_unsupported remains in the closed schema so historical receipts stay valid, but a supported current batched route must leave it zero and show real capture/replay activity instead.

Before a new batched graph becomes replayable, its first launch is compared exactly with the already-required eager warm pass. The comparison covers hidden output, every recurrent and convolution state tensor, and the current K/V rows for every full-attention layer. Native same-dtype equality produces one bounded U8 mask at a time and reads back only the reduced scalar. Structured event rocm_graph_capture_parity_check reports passed, failed, or error, whether comparison completed, compared bytes, duration, hidden equality and the first recurrent, convolution, K, and V layer mismatch, or the comparison error. Comparison failure follows the capture rollback/settlement contract, counts as capture_failure, disables further graph execution for the runner, and reaches eager only after state restoration and containment succeed.

Health makes the admission proof durable with process-lifetime batched_capture_attempts, batched_capture_successes, batched_capture_deferrals, batched_capture_failures, capture_parity_checks, capture_parity_passes, capture_parity_failures, capture_parity_errors, capture_parity_compared_bytes for completed equality operations, and capture_parity_duration_micros. Batched attempts and parity checks each reconcile to their closed outcome sets. Every successful batched capture has a prior passed check, so cumulative successes cannot exceed cumulative passes. Equality is required for a clean qualification window; a passed check followed by a later cache-admission error is retained as a pass plus capture failure rather than falsified into a successful admission.

The eager-fallback reasons are exactly multi_row_batch_unsupported, cold_cache_host_round_trip, persistent_host_round_trip, shape_dependent_attention, graph_cache_capacity, graph_cache_byte_budget, graph_accounting_incomplete, moderate_memory_pressure, tight_memory_pressure, critical_memory_pressure, memory_reservation_denied, memory_governor_selector_mismatch, capture_failure, and replay_failure. The full snapshot reports one counter per reason plus total, slow, total-duration, and maximum-duration values.

Prometheus always emits two availability gauges and two closed one-hot reason families. A zero kiln_rocm_graph_telemetry_available is explained by kiln_rocm_graph_snapshot_unavailable{reason} and omits only the full cache/counter families. A zero kiln_rocm_graph_phase_telemetry_available is explained by kiln_rocm_graph_phase_telemetry_unavailable{reason} and omits only current phase, phase latency, and transient-candidate families. Both reason labels use the same five-value set above. Phase availability remains one for a real backend while full snapshot availability is zero during either runner-lock contention or poison. Full families cover state; cache, slot, owner, retained-byte and opaque-object gauges; admissions, evictions and their four causes; three post-capture rejection reasons; five pre-capture skip reasons; capture/replay outcomes; batched capture attempts/outcomes; first-launch parity checks/outcomes, logical bytes, and duration; and all 14 fallback reasons and latency. The fixed-cardinality families are kiln_rocm_graph_batched_capture_attempts_total, kiln_rocm_graph_batched_capture_outcomes_total{outcome="success|deferred|failure"}, kiln_rocm_graph_capture_parity_checks_total, kiln_rocm_graph_capture_parity_outcomes_total{outcome="passed|failed|error"}, kiln_rocm_graph_capture_parity_compared_bytes_total, and kiln_rocm_graph_capture_parity_duration_seconds_total. Live families cover the one-hot current phase, active elapsed seconds, calls/slow/total/max duration for all six phases, and last/peak transient bytes. Every label set is closed; request, shape, allocation, and configured-byte values never become labels.

Graph-stable paged metadata is a correctness invariant rather than a knob. The runtime no longer reads KILN_ROCM_GRAPH_STABLE_PAGED_METADATA, graph cache/capture flags, matmul synchronization thresholds, or full-attention handoff flags inside decode. The three historical graph variables listed in the table are parsed once by the typed startup loader. Setting both historical graph-mode aliases is rejected even when their booleans happen to agree; use the canonical enum to state one unambiguous lifecycle.

[batching]

These values are resolved once, after backend selection and after server.deterministic, server.max_decode_batch, and server.max_batch_tokens determine the effective decode width. Every change requires a process restart; none is a live tuning control.

Every real backend uses the batching actor. There is no activation switch and no second direct-stream decode scheduler. This invariant gives cancellation, backpressure, admission, adapter barriers, KV ownership, and shutdown one state machine. Mock mode keeps the deterministic scheduler-backed test engine and does not construct a production actor.

Backend-owned auto policy

auto preserves backend qualification rather than imposing one cross-device default. The matrix is evaluated against the final effective decode width, not the unconstrained backend maximum.

burst_prefill_admission is backend-owned and intentionally has no TOML or public environment field. On CUDA it lets admission refill the decode width in one actor turn. It is reported with the four primary-actor settings so an observed CUDA/Vulkan/ROCm difference is not mistaken for a hidden runtime override. actor_prefill_tile_alignment_required is also backend-owned. ROCm sets it because local deterministic parity proved that changing a prompt partition can change greedy output. Startup therefore requires actor prompt chunks and the qualified streaming-prefill tile policy to preserve one numerical partition.

Effective policy and provenance

GET /v1/config reports the exact startup result at batching.configuration and whether an actor was actually constructed at batching.actor_active. A typical ROCm default is:

{
  "batching": {
    "configuration": {
      "rowwise_decode": { "enabled": false, "source": "default" },
      "prefix_aware_admission": { "enabled": true, "source": "default" },
      "prefill_admission_quantum": {
        "configured": null,
        "configured_source": "default",
        "backend_policy": 4,
        "effective": 4,
        "effective_source": "backend_policy"
      },
      "actor_cycle_idle": {
        "milliseconds": 0,
        "source": "default",
        "enabled": false,
        "command_poll_milliseconds": 5
      },
      "burst_prefill_admission": false,
      "actor_prefill_tile_alignment_required": true
    },
    "actor_active": true
  }
}

For explicit values, the configured source is config_file or environment. The complete effective-source enum is default, backend_policy, config_file, environment, or effective_decode_width. The admission quantum uses effective_decode_width when the final decode width lowers the selected value. A nonzero actor-cycle idle has no backend-derived effective value: its exact configured milliseconds and source are immutable for the process. The actor applies it only after a cycle performed model work, not while idle, queued, or waiting for a request. Because the delay begins only after synchronous accelerator return, it does not suspend outstanding device work. It is reported so qualification can measure its throughput and ITL cost. A configured quantum is retained even when clamped, so intent and execution cannot be confused.

GET /health repeats the same immutable object at decode_runtime.batching_configuration beside the optional live decode_runtime.batching_engine snapshot. The trusted debug response keeps the batching_engine.enabled actor-activity flag and the immutable object at batching_engine.configuration. A real backend always reports the actor live; mock mode reports no production actor snapshot.

When the actor exists, the live health/debug snapshot reports actor_cycle_idle_ms, actor_cycle_idle_source, actor_cycle_idle_active, actor_cycle_idle_count, total_actor_cycle_idle_ms, and max_actor_cycle_idle_ms. The same snapshot reports actor_barrier_adapter_active and actor_barrier_resize_active while an ordered mutation is waiting for the active batch to drain or executing at the between-requests boundary. Prometheus exports the same process-lifetime state as kiln_batching_engine_actor_cycle_idle_configured_seconds, kiln_batching_engine_actor_cycle_idle_active, kiln_batching_engine_actor_cycle_idles_total, kiln_batching_engine_actor_cycle_idle_seconds_total, and kiln_batching_engine_actor_cycle_idle_max_seconds, plus the live gauges kiln_batching_engine_actor_barrier_adapter_active and kiln_batching_engine_actor_barrier_resize_active. These counters make an intentional duty-cycle delay distinguishable from an unexplained inference stall, while the barrier gauges distinguish an intentional model/KV mutation from ordinary actor work without adding a control-plane wait. They do not replace request-local adapter_ms or resize_ms. Serving benchmark driver v19 records the actor as the only request route, snapshots an idle boundary outside each measured request window and fails its actor_cycle_idle_accounted gate when the source, count, elapsed time, maximum, or final active state contradicts the configured policy.

kiln config --file <path> validates and prints all four [batching] startup values: rowwise decode, prefix-aware admission, prefill quantum, and actor-cycle idle. It also prints the combined actor-cycle budget, prompt-token and layer ceilings, configured decode-width ceiling, streaming mode, threshold, base, tape, and detached full-attention tiles with source provenance. It does not construct an accelerator. Add --backend rocm (or another target) to resolve the alignment requirement, decode width, streaming dispatch, and tiles from the same backend policies used at serve startup. The command applies the real actor-prefill contract and exits nonzero before any hardware access when the combination is invalid. Without --backend, it names those values as a serve-startup contract instead of guessing a target. Inspect /v1/config after restart for actual selected-device and actor state.

[model]

[paths]

This section owns process-lifetime filesystem locations shared across backend crates. Resolution happens before model loading or accelerator construction, so inference never changes cache location in response to process environment.

When omitted, paths.cache_root resolves from the operating-system account database to ~/.cache/kiln on Linux and other Unix hosts or ~/Library/Caches/kiln on macOS. Kiln does not consult HOME or XDG_CACHE_HOME; use the typed field or its mechanical environment spelling instead. If the account database has no home, Kiln uses the system temporary directory. The root owns BLAS/rocBLAS autotune data, Vulkan pipeline caches, and transposed model weights. kiln config prints the same resolved path.

[memory]

Capacity detection is device-scoped. Discrete accelerators use the selected device's driver-reported VRAM and never count GTT as device-local capacity. Linux DRM topology keeps the primary VRAM heap and a separately admissible host-backed GTT tier distinct. Any nonzero VRAM heap remains the primary pool, even when GTT is larger, because that is also a common discrete-AMD shape.

The host-backed tier is independently bounded by GTT free bytes, MemAvailable, the most restrictive finite memory.max and memory.high headroom across the visible cgroup hierarchy, and the unified-memory reserve. Kiln's current Vulkan paged KV cache and recurrent prefix state are host-resident, so startup requires them to fit both the primary accelerator budget and this host-backed budget. Prefix state is reserved first; automatic KV sizing uses the configured inference fraction of the remaining host-backed headroom. Missing or exhausted host-tier evidence yields a startup error before the zero-filled Rust allocation is attempted. CUDA and ROCm KV pools remain device-resident and are not capped by this separate tier.

Only a driver reporting no local VRAM and a nonzero GTT heap is automatically treated as fully host-shared. A host-shared primary pool retains max(6 GiB, 25% of backing capacity) for the OS and CPU workloads. Apple Silicon applies the same reserve to its unified pool and samples live host pressure. The optional memory.gpu_memory_gb cap is applied only after safe physical capacity is established.

The Vulkan scratch recycler is a cache, not an allocation entitlement. Its memory.vulkan_buffer_pool_gb cap is applied to Vulkan's exact bound memory requirements, including alignment, across device-local and host-visible routes. Borrowed buffers count against the cap and are never evicted while a caller owns them. When there is insufficient idle capacity, Kiln either evicts the least-recently-used idle entries or returns an uncached buffer that is destroyed normally. The memory governor can trim only idle entries and only when the serving profile permits allocator reclaim; changing the cap requires a restart. Recycler misses are counted separately for device-local and host-visible allocation routes. GET /health retains only the most recent miss, including its process-lifetime sequence, requested size, rounded bucket, route, and Rust caller file and line; it reports null before any miss. This diagnostic record has constant memory cost and is not a per-allocation log.

Device-local scratch lookup requires the exact rounded bucket. Host-visible staging lookup checks its exact bucket first, then selects the smallest sufficient idle larger bucket for the same Vulkan device and memory type, with oldest-use order breaking ties. This is sound because every host staging operation retains its logical copy/read extent; it prevents arbitrary prompt-tail shapes from allocating a new staging buffer while a larger compatible one is idle. Undersized and borrowed buffers are never candidates, and cross-device or cross-memory-type reuse is prohibited.

The paged KV block manager also favors residency locality: a completed request's block run returns to the front of the free list in its original logical order. The next compatible request reuses those already-touched pages before walking untouched sparse-zero capacity. On host-resident Vulkan KV pools, this bounds delayed anonymous/transparent-huge-page first touch without prefaulting the full configured cache; logical block ownership, prefix-cache leases, shrink retirement, and physical-resize safety are unchanged.

Physical-device identity remains fail-closed where backend enumeration and OS memory probes expose unrelated logical ordinals. Typed accelerator.vulkan_device_index now selects and reports the same Vulkan index through kernel, model, and server code, but the subsequent memory-probe gate still accepts it only when ordinal zero is the sole relevant DRM candidate. This means an available nonzero Vulkan index is selected faithfully and then rejected before model upload rather than being budgeted against another GPU. CUDA and ROCm have the same single-candidate/ordinal-zero restriction. Multi-device hosts, failed candidate enumeration, and driver-level visibility or remapping controls such as CUDA_VISIBLE_DEVICES, ROCR_VISIBLE_DEVICES, HIP_VISIBLE_DEVICES, MESA_VK_DEVICE_SELECT, or DRI_PRIME are rejected. The canonical KILN_ACCELERATOR_VULKAN_DEVICE_INDEX override is typed application configuration, not a driver remap; the retired KILN_VULKAN_DEVICE and GGML_VK_VISIBLE_DEVICES names are ignored. Automatic memory probing remains diagnostic-only; CPU performs no accelerator probe, and Apple Silicon uses its single unified physical memory pool. Multi-device startup remains unavailable until backend selection and probing share a typed PCI address or UUID.

External accelerator environment safety

Driver visibility and ordinal-remapping variables are not Kiln configuration, do not have TOML or KILN_* aliases, and cannot be changed through the API. Kiln snapshots the presence of this closed set at the first accelerator memory- probe validation, before model weights are loaded, and reuses that immutable snapshot for the lifetime of the process:

If any variable relevant to the selected family is present, device/probe identity fails closed rather than interpreting its value or potentially budgeting one physical device from another device's memory counters. Unset the driver remap. Vulkan exposes the typed accelerator.vulkan_device_index selector, subject to the single-device identity restriction above; CUDA and ROCm do not yet expose a product multi-device selector. Multi-device support requires a shared PCI-address or UUID identity and cannot be enabled by adding another environment spelling.

[training]

Training GPU work is also governed by server.serving_profile; the default stable profile grants coordinated training GPU ownership through the same writer-priority protocol used for adapter transitions and physical memory changes. maintenance is only required when inference must be fully drained.

Optimizer support is a resident capability, not configuration

There is intentionally no startup or request switch for optimizer fallback or rounding. Product SFT, GRPO, OPD, the distinct DistillRefresh workload, and OPD-backed distillation use immutable round-to-nearest. Stochastic rounding remains available only to explicit Rust optimizer-library callers; it is not a server product mode. Consequently, the mechanical KILN_<SECTION>_<FIELD> rule produces no optimizer-policy environment variable.

The following old process-global inputs were removed without aliases:

• KILN_BF16_STOCHASTIC_ROUND
• KILN_TRAINING_HOT_PATH_DEBUG_FALLBACK
• KILN_CUDA_TRAINING_OPTIMIZER_FALLBACK
• KILN_ROCM_TRAINING_OPTIMIZER_FALLBACK
• KILN_METAL_TRAINING_OPTIMIZER_FALLBACK
• KILN_VULKAN_TRAINING_OPTIMIZER_FALLBACK

Delete them from service definitions. There is no replacement field: silent host fallback during product training would change performance, precision, residency, and checkpoint identity. Unknown environment variables are not a supported way to select behavior.

GET /v1/config exposes the resident contract at training.optimizer_support. A real runner returns schema ID kiln.training-optimizer-support and version 1; a mock runner or a runner whose snapshot cannot be read returns null. immutable_after_startup=true means this is a process-lifetime capability snapshot, not mutable configuration. The object has this complete shape (the arrays always enumerate all four workloads and all three optimizer kinds):

{
  "schema": {"id": "kiln.training-optimizer-support", "version": 1},
  "backend": "rocm",
  "device": "rocm:0",
  "base_weight_dtype": "bf16",
  "resolved_lora_parameter_dtype": "bf16",
  "immutable_after_startup": true,
  "rounding_modes": ["round_to_nearest"],
  "backend_implementation_rounding_modes": ["round_to_nearest"],
  "optimizer_tuple_kinds": ["muon", "adam_w", "sgd"],
  "workloads": [
    {
      "workload": "sft",
      "supported": true,
      "unavailable_reason": null,
      "allowed_optimizer_kinds": ["muon", "adam_w", "sgd"]
    },
    {
      "workload": "grpo",
      "supported": true,
      "unavailable_reason": null,
      "allowed_optimizer_kinds": ["muon", "adam_w", "sgd"]
    },
    {
      "workload": "opd",
      "supported": true,
      "unavailable_reason": null,
      "allowed_optimizer_kinds": ["muon", "adam_w", "sgd"]
    },
    {
      "workload": "distill_refresh",
      "supported": false,
      "unavailable_reason": "distill_refresh is unavailable until admission pins separate exact SFT and OPD phase plans, prepares the exact SFT rows, and reserves the maximum sequential working set",
      "allowed_optimizer_kinds": []
    }
  ],
  "optimizers": [
    {
      "kind": "muon",
      "backend_implementation": {
        "supported": true,
        "route": "native_device_hook",
        "native_device_hook": true,
        "parameter_dtypes": ["f32", "bf16"]
      },
      "optimizer_tuple": {
        "supported": true,
        "unavailable_reason": null,
        "lora_rank": {
          "minimum": 2,
          "maximum": 48,
          "backend_maximum": 48,
          "model_maximum": 1024,
          "live_memory_admission_required": true
        }
      }
    },
    {
      "kind": "adam_w",
      "backend_implementation": {
        "supported": true,
        "route": "native_device_hook",
        "native_device_hook": true,
        "parameter_dtypes": ["f32", "bf16"]
      },
      "optimizer_tuple": {
        "supported": true,
        "unavailable_reason": null,
        "lora_rank": {
          "minimum": 1,
          "maximum": 1024,
          "backend_maximum": null,
          "model_maximum": 1024,
          "live_memory_admission_required": true
        }
      }
    },
    {
      "kind": "sgd",
      "backend_implementation": {
        "supported": true,
        "route": "native_device_hook",
        "native_device_hook": true,
        "parameter_dtypes": ["f32", "bf16"]
      },
      "optimizer_tuple": {
        "supported": true,
        "unavailable_reason": null,
        "lora_rank": {
          "minimum": 1,
          "maximum": 1024,
          "backend_maximum": null,
          "model_maximum": 1024,
          "live_memory_admission_required": true
        }
      }
    }
  ]
}

Read the response as four distinct gates:

1. backend_implementation reports whether the optimizer library contains a raw executable update implementation, its portable_reference or native_device_hook route, whether that route is an accelerator-native hook, and its parameter dtypes. It does not claim that resident server weights can execute a training workload. CPU therefore reports portable_reference with native_device_hook=false. An unavailable raw implementation reports supported=false, route unavailable, and native_device_hook=false; parameter_dtypes remains the backend's advertised kind-specific dtype set.
2. optimizer_tuple combines that implementation with the exact resident backend/device identity, base-weight dtype, resolved LoRA dtype, immutable product rounding rule, optimizer kind, and static LoRA-rank range. Within lora_rank, backend_maximum is the optimizer backend's optional ceiling, model_maximum is derived from the smallest dimension among every trained projection, and maximum is their effective minimum. A null backend_maximum means the backend adds no ceiling; it never makes the effective maximum null because the resident model still supplies one. optimizer_tuple_kinds is the summary of kinds whose resident tuple is supported. A supported tuple has unavailable_reason: null; an unsupported tuple retains its concrete reason next to the kind.
3. Each member of workloads independently reports whether the complete server substrate for sft, grpo, opd, or distill_refresh is executable. Its allowed_optimizer_kinds is the intersection of that workload gate and optimizer_tuple_kinds; it is empty whenever the workload is unavailable, and that entry carries a concrete unavailable_reason. This per-workload array, not a raw hook or tuple, is the static HTTP and dashboard admission authority.
4. live_memory_admission_required=true is deliberately outside the static promise. A supported tuple and workload can still be rejected when the exact request shape does not fit the current memory budget. Admission rejects the request; it never lowers rank, changes optimizer, or switches execution route.

rounding_modes is the server-product policy and currently contains only round_to_nearest. backend_implementation_rounding_modes describes what the resident backend optimizer implementation reports. Those fields are separate so a future implementation capability cannot silently become a product mode. resolved_lora_parameter_dtype: null means the resident base dtype cannot be resolved under the backend precision policy; the per-kind tuple reasons carry the failure rather than treating the dtype as unknown.

The sibling training.native_training_supported and native_training_unavailable_reason fields are a coarse compatibility summary. They do not replace workloads: a true summary does not promise that all four workloads are available, and clients selecting SFT, GRPO, OPD, or DistillRefresh must inspect the matching workload member.

For canonical Qwen3.5-4B, whose trained-projection model ceiling is 1024, the resident optimizer-tuple matrix before workload and live-memory admission is:

This table is not a server-execution matrix. Current CPU processes expose the portable F32 optimizer tuples for diagnostics and direct library testing, while their sft, grpo, opd, and distill_refresh workload entries remain unsupported. A Vulkan process can likewise expose native hooks and resident optimizer tuples while a workload fails closed. In the usual hybrid Vulkan server, model weights are CPU-hosted, so exact native backend/device identity and resident-weight checks reject every workload even though raw Vulkan hooks exist.

Static workload admission requires all of the following: a real readable runner; a serving profile that grants training GPU ownership; agreement between configured and resident weight devices; a training runtime that resolves those weights; exact native backend and device identity; no Marlin-packed model projection; and the authoritative kt_tape_authoritative forward/backward route. SFT additionally rejects effective multi-segment checkpointing with a full_logits loss route. OPD additionally requires both its loss route and phase-B backward route. GRPO uses the common authoritative-tape gate. The specific failed condition appears in workloads[].unavailable_reason.

DistillRefresh is not an OPD alias for admission. It is a sequential composite with an SFT knowledge phase and an OPD behavior-recovery phase. The distill_refresh row is unconditionally fail-closed with this stable reason: distill_refresh is unavailable until admission pins separate exact SFT and OPD phase plans, prepares the exact SFT rows, and reserves the maximum sequential working set. Enabling it requires one admission artifact that binds both exact phase plans, the precise SFT rows that phase one will consume, and a reservation for the larger of the two sequential working sets. Adding the phases' estimates or admitting only the OPD-shaped request is not equivalent.

F16 is inference-only on CUDA/ROCm. maximum is the effective static ceiling, computed as min(backend_maximum, model_maximum) while treating a null backend maximum as unbounded. Live memory may impose a lower request-time ceiling but does not rewrite these static fields. The server validates the cheap workload gate, optimizer kind, hyperparameters, base/LoRA dtype, rounding, and rank before checkpoint loading or corpus scanning across dedicated training endpoints, the intent-tagged front door, recipes, judge/self-improve, DistillRefresh, and OPD-backed distillation. Checkpoint loading and corpus preparation happen only after those checks. The worker revalidates the checkpoint and the workload/tuple before memory reservation and residency. Invalid kind, rank, or hyperparameters use structured training_invalid_request; an unsupported base dtype, backend/device identity, or workload substrate uses training_backend_unsupported.

Cheap teacher-alias validation and metadata pinning retain their established request-error ordering and may precede the workload guard. Remote/local teacher materialization, checkpoint loading, corpus scanning, memory preflight, and GPU reservation do not.

For a queued resume, admission fully validates the manifest and every declared artifact hash, then retains a compact checkpoint-ID/manifest-digest identity plus the effective seed. The manifest digest covers its artifact hash entries. Before memory reservation at dequeue, the worker fully reloads the checkpoint and requires both the recomputed identity and effective seed to match. This is not an external-file snapshot: Kiln does not copy or pin checkpoint files in the queue, and a mutation after the reload remains a filesystem race. Keep the checkpoint directory immutable for its entire use.

GET /v1/recipes applies the same static checks to every built-in recipe step and returns admission: {supported, unavailable_reason} on each descriptor. A false descriptor identifies the first unsupported workload or optimizer/rank tuple. A true descriptor is not a memory reservation: recipe submission repeats the full preflight before any step is prepared, and live-memory admission still occurs for each queued job.

Tape scope is internal execution authority, not configuration

The kt autograd tape is a required workload substrate. Kiln opens its thread-local scope internally around the forward/backward region that must record gradients and closes it when that region completes. SFT, GRPO, and OPD admission first require an authoritative tape route; a request cannot open a scope, suppress one recorder, or choose a partially tape-backed execution. There is intentionally no TOML field, API member, CLI flag, or mechanically derived environment name for tape activation.

Scope presence is also the complete inference boundary. Ordinary inference does not open a training tape scope and therefore retains forward-only fast paths. In particular, GDN chunkwise recurrence and CUDA's weight-aware embedding lookup defer to their tape-aware alternatives only while a scope is actually active. They do not consult a cached process-global tape setting. This prevents a default-on global flag from changing inference routing when no gradient graph exists.

The following former correctness switches were removed without compatibility aliases or replacement fields:

• KILN_USE_TAPE_FORWARD
• KILN_USE_TAPE_FLASH_ATTN
• KILN_USE_TAPE_SDPA
• KILN_USE_TAPE_LORA_ADD
• KILN_USE_TAPE_GDN
• KILN_USE_TAPE_GDN_CONV
• KILN_USE_TAPE_GDN_QK_NORM
• KILN_USE_TAPE_GDN_GATED_NORM

Delete these names from service definitions instead of translating them. A debug or performance control may select an implementation only when the active scope still records the required analytic backward and preserves the graph. It must not disable the tape, bypass an individual recorder, or fall through to a forward-only kernel. If no graph-preserving route exists, the workload fails closed through training.optimizer_support.workloads[] rather than accepting a correctness opt-out.

This contract proves static ownership and route isolation; it does not by itself qualify numerical correctness, latency, memory use, or stability on a particular accelerator. Those claims require the platform-specific qualification workload and committed evidence described in NATIVE_SFT_PROFILE.md.

Frozen parameters are constants, not optimizer leaves

Native SFT, GRPO, and OPD train only LoRA A/B tensors. Embedding tables, base projection matrices, RMSNorm and GDN gated-RMSNorm weights, GDN gate parameters, MTP projection weights, and loss-head transposes are saved constants. Their tape nodes record only differentiable activations and LoRA leaves. Backward therefore returns only required activation and LoRA gradients; it does not allocate, retain, or deposit frozen-weight gradients, and frozen projection paths do not execute a dWeight matrix multiplication.

The split Q/gate projection keeps the original full LoRA A/B tensor IDs on both recorded chunks. Each chunk pads its B contribution to the full shape before the tape combines them, and tape-aware reshapes preserve the activation chain. Sliced base matrices and temporary B slices are never trainable leaves. This is an execution invariant with no TOML, request, CLI, debug, or environment override. It reduces frozen-gradient memory and compute by construction, but does not by itself qualify backend throughput.

The optimizer boundary independently requires an exact LoRA gradient set. The observed tensor IDs must equal the configured trainable leaves; shape, backward-compute dtype, master device, and finite values must match before any parameter, optimizer state, or step counter is changed. Missing or unknown entries reject the step, while a valid all-zero gradient is accepted. Checkpointed execution applies the same identity and metadata contract to the exact leaves in each layer range. The tape bridge reduces distinct recorded input contributions to one accumulated entry per leaf, but rejects the result if any registered differentiable input is absent. A connected clone therefore cannot hide a disconnected use of the same leaf. Split output-projection chunks assemble zero-padded contributions under the original full leaf ID rather than temporary slice IDs. A checkpoint range with no configured LoRA leaves accepts only an empty gradient set, and the merge rejects duplicate leaf IDs across disjoint segments. Token-level GRPO accumulation validates identity and metadata for each source but defers its finite-value reduction to the single final optimizer boundary, avoiding a per-completion device synchronization multiplier. None of these checks has a configuration or environment bypass. CUDA, ROCm, and Vulkan use backend finite reducers. Metal performs a synchronized full-gradient host scan at the same boundary until a native Metal reducer is qualified; this is a correctness fallback, not an advertised throughput path.

Backend-owned SFT loss route is not configuration

There is intentionally no [training] field for the native SFT loss route. The resident backend reports one typed capability: CUDA and ROCm report kt_tape_flce, Vulkan reports vulkan_active_rows, and Metal reports full_logits. These values are capability and artifact names, not accepted TOML enums, request fields, CLI flags, or /v1/config controls. Because no configuration field exists, the mechanical naming rule does not produce an environment variable for this choice.

The old KILN_USE_FLCE input has been removed. It is not a deprecated alias, the typed loader does not consume it, and setting it has no effect on current SFT loss selection. Operators should remove it from service definitions rather than translating it to a new name. Backend loss routing is not an A/B switch.

SFT admission resolves the route from the resident model runner and applies a route-specific, saturating memory upper bound. CUDA/ROCm account for kt-tape FLCE buffers and any F32 head promotion; Vulkan accounts for its maximum legal active-row chunk and F32 workspace; full_logits accounts for dense [T, V] logits and cross-entropy forward/backward residency. Overflow saturates toward rejection. An over-budget request returns HTTP 413 with estimated and available capacity and a breakdown containing loss workspace ... (route=<route>).

The admitted enum is stored with the queued SFT job. The worker revalidates it against the resident runner before memory reservation or reclamation, and the trainer revalidates it against its execution backend before resident or trainable allocations. The pinned value drives each forward/backward step, appears in SFT receipts as runtime.sft_loss_route, and participates in the SFT exact-resume planning identity. full_logits is not compatible with a multi-segment checkpoint plan; that combination returns training_invalid_request before queue publication and is checked again by the trainer. These are immutable execution contracts, not omitted configuration knobs.

SFT checkpoint-boundary policy

These four fields control activation boundaries inside gradient-checkpointed native SFT. They do not enable or disable gradient checkpointing; that remains the job of no_grad_checkpoint and the resolved segment plan. When replay is inactive, SFT retains every segment input. When replay is active, it retains a sparse set of anchors and recomputes intervening segment inputs during the backward pass.

For checkpoint_boundary_anchor_stride = "auto", Kiln computes the bytes in one boundary as sequence length * hidden size * bytes per element, determines how many anchors fit in checkpoint_boundary_cache_gb, reserves one slot for replay, and spreads the remaining anchors across the planned segments. The automatically selected result is always at least one, and auto mode selects stride one for a one-segment plan. An explicit stride wins even for a one-segment plan and bypasses only this shape calculation. It does not override the recompute mode.

Startup resolves all four values into one immutable CheckpointBoundaryPolicy. The typed loader retains default, config_file, or environment source attribution for each configured value. Admission and execution receive the same copyable policy and call the same pure threshold and shape functions, so a request cannot be admitted under one boundary-memory estimate and execute another. Trainer and preflight code do not re-read any of the four environment names. Changing any value requires restarting the server; queued and running jobs retain the policy installed at startup.

Sparse checkpoint-boundary replay is currently SFT-only. Native GRPO and OPD retain all num_segments + 1 boundaries and do not use these fields to choose their live boundary layout. The common training runtime still records the resolved policy in exact-resume planning identity. GRPO and OPD use kiln.training-checkpoint-planning.v3. SFT extends the same object to v4 with its pinned backend loss route. A changed boundary value, including one that is execution-inert for GRPO or OPD today, is therefore exact-resume drift. An SFT v3 identity also cannot authorize continuation under v4 because it cannot prove the admitted loss implementation. See Native Training Checkpoints.

The resolved object has this stable JSON shape:

{
  "recompute_mode": "auto",
  "recompute_threshold_tokens": 8192,
  "anchor_stride": null,
  "cache_target_bytes": 6442450944
}

GET /v1/config exposes it at training.checkpoint_boundary_policy alongside the gradient-checkpoint plan. GET /health and its /v1/health alias expose it at training.checkpoint_boundary_policy; trusted GET /v1/debug/model-state exposes the same object at training.checkpoint_boundary_policy. The dashboard's Runtime Config training group renders mode, threshold, explicit-or-auto stride, cache target, and the restart requirement. These are observations of immutable application state, not live controls or fresh environment reads.

[openenv]

This section governs server-owned OpenEnv runs launched from the dashboard or /v1/openenv/*. The kiln openenv CLI uses the same protocol client and artifacts but is not disabled or origin-restricted by this server control-plane policy.

Protected environments use server-owned, exact-origin credentials:

[openenv]
allow_remote_environments = true

[openenv.credentials.production-arcade]
origin = "https://arcade.example.com"
bearer_token_env = "ARCADE_OPENENV_TOKEN"

POST /v1/openenv/inspect, POST /v1/openenv/tasks, and POST /v1/openenv/runs accept "credential_ids":["production-arcade"], aligned positionally with environment_urls. Use null for an unauthenticated slot in a mixed list. Kiln resolves the handle only after URL and origin validation, applies the bearer header to HTTP discovery, Task API requests, and the WebSocket upgrade, and never places the token in typed config, run status, logs, metrics, summaries, replay manifests, or receipts.

Each active run uses the selected resident policy through Kiln's authoritative in-process chat handler, so normal inference admission, adapter identity, metrics, and limits apply. A train run then enters the same bounded native GRPO queue as POST /v1/train/grpo; it does not create a second trainer. The OpenEnv run remains the workflow-level owner, projects the trainer and linked post-evaluation state, forwards cancellation to those authoritative executors, and becomes terminal only when the complete requested workflow does. New failed runs retain kiln.openenv-run-failure.v1 with a closed stage and code, retryability, a bounded detail, an actionable hint, and exact protocol/HTTP evidence when available. Alert on the fixed-cardinality kiln_openenv_run_failures_total{stage,retryable} metric; its labels never contain endpoints, run IDs, or peer messages.

[logging]

Logging is bootstrapped before the full file is validated so parse failures can be reported through the requested renderer. The authoritative loader then validates the complete file and stops startup if it differs or is invalid. Malformed or non-UTF-8 RUST_LOG is fatal.

[prefix_cache]

The TOML object records operator intent. GET /v1/config reports that object at prefix_cache.configuration and reports the backend-qualified result beside it: effective_enabled, effective_reason, effective_max_blocks, effective_max_entries, and effective_max_state_bytes. Reasons are active, configuration, vulkan_correctness_quarantine, or backend_unavailable. GET /health and GET /v1/health expose the live capability both as prefix_cache.enabled and, when the batching actor exists, decode_runtime.batching_engine.prefix_cache_enabled. Trusted debug state publishes the same two views. Prometheus publishes kiln_batching_engine_prefix_cache_enabled for the actor capability plus the existing kiln_prefix_cache_* activity and residency series.

For batching-actor requests, enabled changes storage and reuse only. Cache admission does not change the canonical block-aligned final prefill boundary: an enabled cache miss and a disabled-cache fresh prefill execute the same token and layer quanta. With the cache disabled, Kiln skips lookup, recurrent-state snapshotting, prompt registration, rolling snapshots, retained blocks, and leases; it does not select a numerically different prompt partition. This invariant matters for hybrid GDN models because each resumable chunk boundary materializes recurrent state at the configured precision. Qualification must compare tokens, recurrent state, and the following decode token before treating the cache-off arm as a storage-only diagnostic.

While Vulkan is quarantined, all prefix-cache lookup, hit, miss, retained-block, retained-entry, recurrent-state, lease, and pending-release values must remain zero. Requests use fresh generic prefill, including exact repeats. There is no request override or alternate environment flag that bypasses the quarantine. Re-enablement requires a source change and production-model semantic parity across first use, exact repeats, strict descendants, changing concurrency, cancellation, and repeated process history; cache mechanics tests alone are not sufficient.

[speculative]

The loaded SpeculativeDecodingConfig is the sole serving authority. TOML and the canonical environment overrides are resolved and validated once during typed startup, then the immutable result is passed to serving state. Request dispatch does not re-read speculative environment variables. Restart the server to apply a change.

Current fail-closed status: off is the only serving method available. Every enabled method, including skip_layer and mtp, stops startup until its cancellation, owner-settlement, EOS, context-capacity, and burst-admission contracts pass local accelerator qualification. The fields remain part of the typed configuration so their intended contract is explicit and can be qualified without introducing a second configuration path. kiln config --file ... applies the same availability gate as serve, native MTP weights remain deferred (load_mtp=false), request dispatch contains no speculative branch, and Desktop always launches with speculative serving off. Benchmark-only speculative paths require the explicit --allow-experimental-speculative acknowledgment. That flag permits research benchmarks only; it does not qualify a serving route.

The former KILN_MTP_DEBUG, KILN_MTP_DUMP_*, KILN_MTP_SWAP_*, and KILN_MTP_*FP32* experiment controls are retired and are not configuration inputs. They have no typed replacements because they selected completed diagnostic branches rather than supported runtime policy. Current source retains one canonical MTP execution path; historical commands and conclusions remain labeled as such in PROFILING.md.

[streaming_prefill]

streaming_prefill.mode is the single mode authority. The removed TOML enabled toggle and non-canonical environment spellings are not alternate paths into this policy: the TOML field is rejected and the environment names are ignored.

The selected backend supplies the following auto policy. Dispatch thresholds are inclusive. Tile counts are tokens:

mode = "enabled" can deliberately force non-empty CPU or Vulkan inputs onto the streaming path; this is an operator override, not a claim that the route is qualified or faster on that backend. Conversely, mode = "disabled" is the clean isolation control for a monolithic-prefill A/B on backends that admit it; ROCm rejects this setting. A threshold override only adjusts an existing CUDA, ROCm, or Metal automatic crossover. Dispatch eligibility and physical splitting are distinct: an eligible route creates multiple tiles only when its sequence is longer than the effective tile for that route.

Startup resolves this section once after backend selection and injects the same immutable execution policy into ordinary generation, prompt-logprob scoring, native SFT/GRPO/OPD admission and training forwards, local OPD teachers, MTP alignment, checkpoint planning, and the benchmark harness. Server-owned training and teacher construction fail closed if that startup policy is missing. No production model or trainer path re-reads these public environment names. Every change requires a restart; an existing job or request cannot observe a mid-process change.

Vulkan resumable-prompt GDN state residency is intentionally absent from this configuration section because the production route is correctness-quarantined. There is no TOML field, canonical environment override, alternate spelling, or internal enable variable that can activate it. The typed Vulkan runtime policy sets prompt residency to false. Ordinary actor chunking therefore materializes each layer's recurrent state at the completed token-chunk boundary, including the BF16 dtype boundary, before a request can yield and resume. Layer-group yields inside one token chunk retain their existing in-progress state and do not add an extra numerical boundary.

The lower-level registry, stable (serving row ID, linear-attention layer index) ownership, TensorId alias transfer, device-side BF16 boundary, exact owner eviction, and observability remain compiled and test-covered so the fault can be investigated without reconstructing the experiment. Production cannot enter those prompt scopes. Re-enabling them requires a source change, the focused real-device suite, a clean full-model semantic pass with prompt residency enabled and decode residency disabled, the cancellation/drain probe, and a checked soak. A micro-kernel or small-state parity result alone is not a release gate. The source-bound failure and rollback receipts belong in Local Hardware Qualification; they do not define a machine-specific product default.

Prompt recurrent-state residency remains disabled by kiln.vulkan-kernel-policy.v6; resident decode is enabled only when the selected device satisfies the complete resident shader limit set. Their former enable and disable variables were deleted without aliases, so process environment cannot compose an unreported route mixture. Changing either decision requires a reviewed source policy version and the full evidence sequence above.

Trusted GET /v1/debug/model-state exposes current ownership as caches.resident_recurrent_state.entry_count, buffer_bytes, and allocation_bytes. /metrics publishes the same state as kiln_gdn_recurrent_state_resident_entries and kiln_gdn_recurrent_state_resident_bytes{kind="buffer|allocation"}. A drained server must report zero for all three values. While prompt residency is quarantined, production also expects zero throughout prompt execution; a nonzero value is an invalid route activation or ownership defect. Their wire types are generated from contracts/kiln-observability-v1.schema.json; no copied environment variable or undocumented toggle controls them.

Kiln prompt-logprob teacher identity uses kiln.prompt-logprobs.inference-config.v2 and hashes the complete resolved execution policy: mode, threshold, every base/tape/detached tile variant, and last-token LM-head behavior. Changing any of these fields therefore changes inference_config_sha256, the canonical teacher content revision, and every identity-bound logit-cache key. A deployment must re-register the teacher and must not resume an OPD checkpoint pinned to the previous revision.

Prompt-logprob selection has no environment variable or runtime tuning field. The runtime chooses either compact_device or bounded_host_fallback from the backend implementation, and fixed-cardinality Prometheus counters report the resolved route. The HTTP API Contract defines scoring behavior and the Latency Observability guide defines its operational evidence.

GET /v1/config exposes the complete resolved object at streaming_prefill. Health repeats it at prefill_runtime.streaming_prefill, and trusted debug exposes it at top-level streaming_prefill. The object separates:

• configured mode/source, backend dispatch rule, effective dispatch rule, and effective authority;
• configured/backend/effective threshold, including whether an override actually changed a threshold-based backend policy;
• configured/backend/effective values and sources for base, tape, detached, detached-boundary, and detached-replay tiles;
• last-token LM-head configured/effective state and source; and
• immutable_after_startup plus restart_required_to_change.

An inherited specialized tile reports a closed source such as inherited_from_tile_tokens_config_file; an untouched automatic value reports backend_policy. There is intentionally no single effective_enabled boolean: in automatic mode dispatch depends on the current prompt length.

Examples:

# Keep backend dispatch and tile policy (recommended default).
[streaming_prefill]
mode = "auto"
threshold_tokens = "auto"
tile_tokens = "auto"
tape_tile_tokens = "auto"
detached_full_attn_tile_tokens = "auto"
last_token_lm_head = true

# On non-ROCm backends, isolate a suspected tiled-prefill pause while keeping
# actor ownership.
[streaming_prefill]
mode = "disabled"

# ROCm diagnostic: lower the actor and streaming tile boundaries together.
[server]
max_batch_tokens = 256
max_prefill_tokens_per_cycle = 128

[streaming_prefill]
mode = "enabled"
threshold_tokens = 128
tile_tokens = 128
tape_tile_tokens = "auto"
detached_full_attn_tile_tokens = "auto"

[adapters]

Because all three typed defaults are Some(...) and TOML has no null value, the supported way to opt out is the canonical environment override with 0 or an empty value.

[teachers]

teachers.credentials is a dynamic map of server-owned credential handles. There is no field-level environment override for the map. Each credential points to a separately named secret environment variable; that secret is not stored in TOML, AppState, API responses, logs, receipts, or caches. One narrow credential-provider adapter performs every process-environment secret lookup; model, training, request-handler, and configuration code cannot read it directly.

[teachers.credentials.primary-vllm]
origin = "https://vllm.example.com:8443"
api_key_env = "VLLM_TEACHER_API_KEY"

Credential ids must be 1..=64 ASCII letters, digits, _, or -. A handle is accepted only for its exact configured origin. Non-loopback teachers require a configured credential handle; credential-free teachers are restricted to loopback. Startup validates every configured secret before device or model initialization. Availability is checked again when a handle is resolved, and the provider reads the bearer value immediately before each outbound request; missing, non-Unicode, or whitespace-only values fail closed without exposing the configured variable name or remote response body. Operators should still treat secret changes as restart-scoped configuration because configuration validation and registered-teacher identity are startup-owned.

[eval]

The section is optional. When absent, the runtime uses the same queue defaults shown below and roots eval data at <adapter_dir>/.eval. It creates suites, datasets, and judgments below that root.

Names such as eval.root, eval.max_tracked_eval_jobs, and an [eval.generation] table are not accepted by the current typed schema.

[request_log]

Request logging is enabled by default. It records inference request and response JSON on a dedicated bounded writer thread; overload drops log rows instead of blocking inference. Rotation and retention bound disk use.

The default is intentionally data-bearing. Review log-directory permissions, retention, and the captured request content before exposing the service to untrusted traffic.

[agent]

The section is optional. Its absence disables the scheduled self-improvement loop, while the embedded run engine still receives the default concurrency and timeout.

Terminal and embedded-run access default to loopback_only. The host decision, the pi executable, session root, adapter-library URL, and teacher-logit cache root are resolved once during startup and published under operational in GET /v1/config. Request handlers use that immutable snapshot; changing TOML, PATH or a canonical configuration input requires a restart.

Effective values and provenance

Library runtime boundaries

Applications embedding kiln-model directly must initialize accelerator memory governance as an explicit startup step. Call InferenceMemoryRuntime::initialize(device, governor_config) before building the runner, then pass the returned binding to ModelRunner::new_with_initialized_runtime. Initialization validates the backend/probe physical-device identity, detects safe capacity, treats GovernorConfig::capacity_limit_bytes as a cap, publishes a live sample, and starts the background sampler. The fallible runner constructor verifies the binding against the model-weight device and installed process policy without probing again. ModelRunner::new, new_with_options, and new_with_runtime_options remain non-probing compatibility constructors for owners such as kiln-server that install the same process-wide policy before model construction.

Accelerator training has the same explicit identity rule. Construct TrainingRuntimeContext::new_for_device(device, effective_vram, policy) and use the *_with_runtime SFT, GRPO, streamed-GRPO, or OPD entry point. A context without a device binding cannot authorize accelerator-backed weights. Weight storage and native training execution must currently name the same device. Vulkan serving uses CPU-host weight handles, but its full-model resident upload path is not production-qualified and is therefore rejected before queue or dataset admission with training_backend_unsupported; there is no environment escape hatch. Vulkan-device training fixtures remain valid when their weights are genuinely Vulkan-resident. Use the HF/TRL export/import workflow for production training while serving through the hybrid Vulkan backend.

Kiln exposes complementary startup-input and runtime-derived views:

• kiln config --file <path> performs authoritative parsing and validation and prints a concise human-oriented summary. kiln config --file <path> --json emits kiln.config-check.v1: the complete source-aware startup object, resolved accelerator policy, application paths, checkpoint-boundary policy, and optional backend preview. It does not probe hardware or load weights. Adding --backend <target> resolves hardware-independent scheduling capabilities and rejects an invalid actor-prefill contract, but deliberately does not claim device availability or live route state.
• GET /v1/config reports runtime diagnostics for serving profile, the complete immutable effective_configuration startup snapshot, the versioned accelerator runtime policy, effective decode width, speculative configuration and availability, VRAM/KV state, native-training runtime/weight devices and support reason, the versioned implementation/resident-tuple/per-workload optimizer contract, gradient-checkpoint segmentation, the immutable SFT checkpoint-boundary policy, the absolute shared cache root and source, memory budgets, and generation defaults. Its training.checkpoint_boundary_policy object reports resolved recompute mode, inclusive automatic threshold, optional explicit stride, and integral cache target bytes. Its speculative object distinguishes configured_method and configured_effective_method from the actual serving_effective_method (off), reports serving_routable=false, the stable serving_unavailable_reason, the K ceiling, and immutable backend MTP support facts. It does not expose live routing, admission, or weight-readiness fields because those are not serving authorities. A running process cannot have a non-off configured method through the supported startup path. training.native_training_supported=false is accompanied by training.native_training_unavailable_reason and matches the admission error without scanning a corpus. training.optimizer_support separately exposes backend implementations, exact resident optimizer tuples, and per-workload executable kinds. Consumers must use workloads[].supported and workloads[].allowed_optimizer_kinds, not infer product support from backend_implementation or optimizer_tuple_kinds. When checkpoint execution is disabled, training.checkpoint_segments is 0; an optional segment count retained in training.checkpoint_policy is provenance, not an active execution plan. The effective_configuration is the complete input object; sibling fields are the authoritative backend-derived, hardware-derived, and live results.
• GET /health and /v1/health repeat the accelerator policy and reasoned ROCm synchronization counters under decode_runtime, and repeat the resolved boundary policy under training.checkpoint_boundary_policy; trusted GET /v1/debug/model-state reports the identical object at the same path. The dashboard reads /v1/config and renders the same process-lifetime values. None of these surfaces parses configuration or rereads process environment.
• GET /v1/config also reports cached-sample age, maximum age, staleness, sampler requirement/liveness/health, cgroup memory.high, and the separately bounded vram.live.raw_observations.host_backed tier. /health exposes the same liveness decision, while /metrics exports fixed-cardinality gauges for alerting. A stale or failed sample remains visible diagnostically but contributes zero allocation headroom.
• /health and /v1/debug/model-state expose config hashes and other runtime identity data. Trusted debug additionally exposes the current and process-lifetime batched recurrent-state cache lifecycle at caches.batched_recurrent_state, plus the separate direct prompt/scoped-decode GDN registry at caches.resident_recurrent_state; /metrics publishes the same fixed-cardinality ownership, reuse, rejection, concurrency, and eviction counters. The complete field and interpretation contracts are in resumable prefill residency telemetry and batched recurrent-state cache telemetry.
• Startup logs record serving-profile provenance and configured/backend/final decode-width sources.

The versioned kiln.effective-configuration.v1 object is a deterministic flat map keyed by canonical TOML paths. It always contains all 112 fixed typed leaves, including materialized defaults for optional [eval] and [agent] sections, plus two dynamic leaves for every configured teacher credential: teachers.credentials.<id>.origin and teachers.credentials.<id>.api_key_env. Every entry contains:

• effective_value: the post-precedence typed JSON value, or null when redacted;
• source: exactly default, config_file, environment, or command_line;
• canonical_environment: the mechanically derived supported environment name, or no name for config-file-only/dynamic fields;
• compatibility_environment: retained in schema v1 and always empty, making the absence of alternate spellings machine-readable;
• redacted: whether the value is deliberately withheld; and
• restart_required_to_change: always true for this startup snapshot.

The object also reports its schema/version, fixed and dynamic field counts, the typed effective-config hash, and the invariant that every represented value requires restart. Field keys are lexicographically ordered. The loader parses the selected TOML document once, validates the typed object, records which leaves were explicit, applies centralized environment overrides, then applies the two supported serve CLI overrides. Unknown TOML fields and invalid values still fail before the dump can be produced.

Secrets and request-bearing values are never serialized into this diagnostic. agent.self_improve, eval.webhook_url, training.webhook_url, and every teachers.credentials.<id>.api_key_env entry are present with their real source but redacted=true and effective_value=null. The named bearer secret itself is never part of KilnConfig, the effective dump, application state, or the config hash.

The effective_config_hash binds only the serialized effective typed configuration and is repeated in the human-readable dump. Execution provenance separately hashes the redacted set of present KILN_* environment inputs, so ambient process state cannot silently change the meaning of the configuration digest. Backend-derived adjustments are intentionally not rewritten into the startup map: use --backend for a hardware-free scheduling preview and the sibling /v1/config objects for the actual selected backend, capacity, availability, and live state.

Configuration migration

The canonical-only environment contract is a breaking cleanup. The former public spellings below are ignored rather than accepted as aliases. Replace them before upgrading; kiln config --json exposes the final source of every effective field so deployment checks can prove the replacement took effect.

For ordinary aliases, preserve the old value under the replacement name. The former boolean-to-mode and force/disable controls require an explicit value:

• KILN_DEFAULT_NO_THINK becomes KILN_SERVER_DEFAULT_THINKING_ENABLED=false.
• KILN_TERMINAL and KILN_AGENT_RUNS map false to disabled and true to enabled on their replacement access fields.
• KILN_BATCHING_ENGINE, KILN_DECODE_BATCHER, KILN_STREAMING_PREFILL, KILN_STREAMING_PREFILL_ENABLED, and KILN_RECOMPUTE_CHECKPOINT_BOUNDARIES map false to disabled and true to enabled on their replacement mode fields.
• KILN_ROCM_GRAPHS maps false to disabled and true to profile. KILN_ROCM_GRAPH_CAPTURE maps false to warmup_then_eager and true to profile.
• For the two retired ROCm matmul force/disable pairs, true force-strided maps to enabled, true disable-strided maps to disabled, true force-F32-output maps to f32_then_cast, and true disable-F32-output maps to native_bf16. A former false value delegated to the removed machine-shaped auto policy and therefore has no exact replacement. Choose the portable defaults (disabled and f32_then_cast) or explicitly qualify the corresponding experimental route.
• KILN_SPECULATIVE_ENABLED and KILN_SPEC_ENABLED map false to off and true to skip_layer on KILN_SPECULATIVE_METHOD. Non-off serving remains fail-closed. The method override now accepts only off, skip_layer, and mtp; historical synonyms must be replaced with those exact enum values.

TOML also has two removed fields: replace speculative.enabled with speculative.method, using the boolean mapping above, and replace streaming_prefill.enabled with streaming_prefill.mode, mapping false to disabled and true to enabled. Unknown fields are fatal, so stale files fail at startup instead of becoming inert.

Every public runtime field now resolves exactly once and is passed as immutable typed configuration to its owner; the environment contract is generated mechanically, and repository policy rejects direct production environment reads outside the closed driver-remap and credential-provider adapters. Internal experiment and qualification flags remain separate from this public API rather than being promoted by documentation.

Repository policy also scans executable and operational source under .github/, scripts/, capabilities/, and desktop/ against the complete retirement index. A retired public name in those surfaces fails scripts/check_config_schema.py; current launchers, training recipes, generated corpus examples, and workflow code must use the canonical name and value semantics above. Exact-revision material under an archive/ directory is historical evidence rather than a current command. The only non-archive source exceptions are two smoke checks that assert retirement behavior and one historical benchmark receipt generator that records the command originally measured. Those exceptions may mention a retired spelling, but may not use it as current runtime configuration.