Kiln is one Rust process that serves, trains, and hot-swaps adapters.

Single-process server

Kiln runs serving, training, adapter management, and evaluation in one deployable Rust binary. The process owns the Axum HTTP API, scheduler, model engine, adapter registry, and background workers. The normal SFT and GRPO paths need neither a Python sidecar nor a second model load.

inference
  -> HTTP/API
  -> scheduler + block manager
  -> Qwen/Qwen3.5-4B engine
  -> sampler or streaming response

training
  -> HTTP/API
  -> LoRA training queue
  -> saved or hot-swapped adapter

OpenEnv RL
  -> discovery + stateful WebSocket episodes
  -> canonical scored GRPO groups + exact replay
  -> native training + held-out environment returns

“One process” does not mean that every task runs at the same time. Inference takes shared accelerator ownership; training and weight-changing adapter operations require exclusive ownership and are admitted only by a serving profile that permits them.

OpenEnv episodes are a native training input

kiln-openenv owns implementation-neutral HTTP discovery, typed schemas, optional Task API catalogs, authentication, and the lock-step stateful WS /ws protocol. kiln-server composes those sessions with the ordinary in-process chat handler, persists run lifecycle, and submits canonical scored groups through the same native GRPO admission path as every other job. kiln openenv start enters that persisted state machine from a bounded request file. A strict FIFO admits at most the configured number of complete workflows; queued runs hold no environment or model capacity and remain position-visible and immediately cancellable. Never-admitted v5 entries resume after restart; pristine v4 entries are first sealed once into v5; admitted stateful work fails explicitly rather than guessing at recovery. A bounded persisted idempotency key atomically maps identical retries to the original retained run and rejects drift.

persisted run request + environment URLs + aligned credentials/reset plan
  -> idempotent persisted creation
  -> bounded FIFO workflow admission
  -> canonicalize complete discovery identity and compile self-contained action schema
  -> round-robin seed-matched candidate episodes
  -> validate every model action before its environment step
  -> environment-owned returns
  -> revalidate complete discovery identity
  -> immutable admitted training contract
  -> dataset + exact replay + summary v5 receipt
  -> native GRPO
  -> paired held-out environment evaluation
  -> promote, retain, or reject adapter
  -> completed evidence or typed terminal failure

Current inspection binds a canonical kiln.openenv-discovery.v1 SHA-256 over the complete raw metadata, schema, environment inventory, and OpenAPI values before typed projection. Whitespace and object-key order are irrelevant; unknown extension fields remain identity-bearing. Every session also repeats status-only health immediately before its WebSocket upgrade. Rollout and corpus provenance retain the digest, while legacy v1 evidence without it remains readable.

The advertised action schema is enforced, not merely shown to the policy. Inspection auto-detects its JSON Schema draft, supports internal references, and rejects external HTTP/filesystem references without resolving them. A schema-invalid action never reaches the environment; it is retained as invalid_model_action with bounded keyword/JSON-Pointer evidence and ACTION_SCHEMA_VALIDATION_FAILED.

Thinking-on is the native OpenEnv default. A non-empty observation input_text is pasted verbatim as the user turn. Kiln retains separated model reasoning plus the final answer as the trainable GRPO Action trajectory, while only the independently parsed, schema-valid final answer reaches the environment. The template-owned thinking opener remains prompt context; the action mask begins on the first generated reasoning token.

Train admission atomically persists kiln.openenv-training-contract.v1 with the effective GRPO config and optional post-eval suite. Restart, trainer submission, CLI, and dashboard consume that sealed value instead of recomputing defaults; summary v5 embeds it before first artifact publication so a later handoff failure remains auditable.

After the last episode and before artifact derivation, the run enters revalidating and re-reads every endpoint’s complete stable discovery identity. Metadata, advertised names, OpenAPI version, authentication mode, URLs, schema digest, and all action/observation/state schemas must exactly match the initial inspection. A redeploy becomes environment_identity_changed at identity_verification; no mixed-identity corpus is published or trained. Live replay uses the same complete comparison.

Failure is part of that state machine, not an unstructured side channel. New terminal failures retain kiln.openenv-run-failure.v1: a closed lifecycle stage and reason code, retryability, bounded detail, operator hint, timestamp, and exact protocol or HTTP evidence when supplied. CLI, API, dashboard, and fixed-cardinality metrics project the same diagnosis; the historical error string remains only for compatibility.

One reset object may be shared, or a heterogeneous portfolio may align one object with each endpoint. Kiln removes caller-supplied seeds, inserts deterministic group seeds, and requires every endpoint to run. The receipt hashes the ordered seed-free plan; offline verification reconstructs it from replay and rejects missing endpoints or drift.

Policy generation may be slower than an environment server’s WebSocket idle timeout. Kiln pumps Ping/Pong control frames while the model thinks and also sends periodic read-only state exchanges. A timeout, malformed or unsolicited frame, transport failure, or wrong response type poisons the session permanently—lock-step has no request ID with which to resynchronize.

Every retained reset, action, observation, error, and final state is charged on arrival against one 512 MiB aggregate collection budget. Candidate data moves into group, replay, and receipt ownership without full-object clones; reset files are bounded before read, hashes stream, and replay encoding stops at its cap. Exhaustion publishes no partial bundle.

Only manifest-declared artifacts are downloadable. Publication hashes bounded regular files, and every request rechecks exact byte count and SHA-256 on the same descriptor before streaming. kiln openenv artifact repeats the manifest, header, length, and digest checks while staging locally, then publishes atomically. Replacement, symlinks, growth, truncation, or mutation fail closed instead of serving drifted bytes.

Exact action/observation transcripts, reward arithmetic, recovery, capacity backoff, cancellation, origin-scoped credentials, metrics, and paired promotion evidence remain server-owned. The production path depends only on OpenEnv compatibility. Reproducible CI derives the complete published environment matrix from a pinned oracle checkout, while a scheduled edge lane repeats that byte-real gate against upstream main. The oracle inventory is never production policy. See the OpenEnv training guide and replay contract.

Request path and batching

Startup resolves one immutable BatchingRuntimeConfig after selecting a backend and decode width. Streaming and non-streaming requests, sampling, cancellation, response backpressure, adapter barriers, KV ownership, and shutdown enter the same actor state machine. Live snapshots distinguish queue time, prefill, decode, response delivery, and cleanup.

Prefix-cache admission must change reuse, not numerical prompt partitioning. A cache miss follows the same block-aligned final prefill boundary as a fresh request. A request is not reported complete until its scheduler row and backend-owned resources have drained.

One streaming-prefill authority for inference and training

Typed configuration and canonical startup overrides resolve into an immutable StreamingPrefillRuntimeConfig. Inference, prompt-logprob scoring, native training, local teachers, checkpoint planning, and benchmarks consume that same value. They do not reread environment variables while work is running.

Backend policy supplies automatic dispatch thresholds and route-specific tile defaults. Explicit settings may override those defaults, but startup validates their geometry before accepting traffic. The resolved values and their sources appear in config, health, and trusted-debug output.

Changing prefill policy changes execution identity. Identity-bound teacher caches and resumable training state therefore fail closed instead of silently reusing evidence produced by another route. See the configuration reference for exact defaults and validation.

Accelerator policy is resolved once before execution

Kiln combines typed [accelerator] settings with the serving profile before it creates the device or loads the model. The result is immutable kiln.accelerator-runtime-policy.v16, shared by the tensor context, model runner, training runtime, and diagnostics. Requests cannot change it.

Vulkan policy is capability-driven

Vulkan adds a device-capability policy derived from standardized limits and features—not a device name, marketing name, vendor ID, device ID, driver string, PCI ID, host name, or qualification receipt. After physical-device selection, Kiln reads the API version, workgroup and storage limits, shared-memory and push-constant limits, subgroup support, and memory properties. It installs kiln.vulkan-kernel-policy.v6 once.

Each optional Vulkan route is admitted only when its own shader, buffer, dispatch-grid, tensor-shape, and dtype requirements fit. If one route is unavailable, only that route takes its explicit fallback; unrelated fast paths remain eligible. This lets any sufficiently capable Vulkan device use accelerated routes without appearing in an allowlist. A policy change requires a version change and cross-device correctness evidence.

The research-only vulkan_decode_microbench accepts typed command arguments. Its route experiments and machine receipts are not server settings and cannot qualify product behavior. Exact device-selection, validation-layer, kernel-profile, and fallback contracts belong in the configuration reference and architecture deep dive.

Other backends follow the same boundary

CUDA and Metal use typed kernel profiles; ROCm product execution defaults to its correctness-qualified native paged-decode routes and falls back portably when a request is ineligible, alongside typed synchronization, matmul, and graph policy. Backend-specific optimized routes may have narrower eligibility than baseline inference. Their presence in source does not mean that stable serving activates them, and no hot path is allowed to reinterpret a changed shell variable between requests.

GET /v1/config reports resolved startup intent and source attribution. GET /health adds live runtime state. Trusted debug output and Prometheus expose backend-specific activity, fallback, synchronization, memory, and quarantine evidence. Missing telemetry is never presented as zero activity.

Why the Gated DeltaNet hybrid matters

Qwen3.5-4B is not an all-attention transformer. Its 32 layers comprise 24 Gated DeltaNet linear-attention layers and 8 full GQA layers. Kiln targets that exact architecture.

The architecture deep dive explains recurrence, cache ownership, paging, and kernel boundaries.

LoRA hot-swap and training queue

Accepted SFT, GRPO, and OPD jobs enter a FIFO background queue. The worker takes exclusive accelerator ownership, trains a LoRA adapter against the loaded base model, checkpoints progress, and saves the completed adapter. The base model remains fixed.

Stable serving runs the complete product loop: inference, training, evaluation, adapter transitions, guarded memory management, and correctness-qualified acceleration. Maintenance drains inference for exclusive operational work; experimental is reserved for qualifying quarantined backend routes. See the serving-profile matrix.

Backend-owned SFT loss routing

The selected training backend owns one typed SftFlceLossRoute. Requests cannot select it, and there is no TOML field or mechanically derived environment name for it. Admission estimates the route-specific loss workspace and rejects unsupported checkpoint geometry before queue publication.

This matrix identifies the loss implementation after native training is admitted. It does not claim that the complete training path is available on every serving configuration or device. Read the generated Native SFT Profile for the normative route, memory, checkpoint, and receipt contract.

GPU backend crates

Kiln keeps the Qwen3.5-4B execution path in focused Rust crates, with native kernels where the model needs them. Backend boundaries are explicit so policy, correctness fallbacks, and telemetry stay attributable.

• kiln-server owns configuration, HTTP routes, serving-profile admission, metrics, and workers.
• kiln-scheduler and kiln-core coordinate requests, batching, and KV blocks.
• kiln-model loads Qwen3.5-4B, applies adapters, and drives forward execution.
• kiln-flash-attn, CUDA support crates, kiln-vulkan-kernel, and Metal shaders provide backend-specific kernels.
• kiln-train owns SFT, GRPO, OPD, checkpoints, receipts, and training-job state.
• kiln-openenv owns protocol discovery, typed wire messages, stateful sessions, and bounded interoperability semantics.
• kiln-eval owns suites, scorers, outcomes, and dataset-to-suite synthesis.

Evals as a peer of training

A separate FIFO worker resolves a registered or inline suite, generates each completion in process, scores it, and records per-example outcomes plus aggregate accuracy. Eval generation can share the inference side of accelerator ownership and waits while training holds exclusive GPU ownership.

See the Evals guide for suites, scorers, post-training gates, judgments, comparisons, and strict replay.