Architecture deep dive

Kiln is one Rust process that serves Qwen3.5-4B, trains LoRA adapters, evaluates them, and manages the resulting artifacts. This page is for contributors and operators who need to understand the boundaries between the HTTP service, scheduler, model runtime, accelerator backends, training system, and durable state.

If you want to run Kiln rather than study its internals, begin with the Quickstart. For exact request and response shapes, use the HTTP API reference.

System at a glance

clients and dashboard
        │
        ▼
HTTP validation, policy, and identity
        │
        ├──────────────► OpenEnv HTTP discovery + stateful WS episodes
        │                         │
        │                         └──► canonical grouped trajectories
        │
        ├──────────────► durable datasets, suites, jobs, and receipts
        │
        ▼
scheduler and accelerator ownership
        │
        ▼
model runner and tensor/autograd substrate
        │
        ▼
CUDA · ROCm · Metal · Vulkan · CPU

The same process owns four related workflows:

These workflows share model identity, memory accounting, adapter state, and accelerator ownership. They do not communicate through Python workers or a second model server.

Package boundaries

Kiln is a Cargo workspace. The exact crate list changes as backend work moves, so the useful boundary is responsibility rather than a fixed crate count:

Feature flags compile only the backend stack requested by a build. Product code talks to focused backend traits for attention, linear algebra, sampling, residency, training loss, optimizer steps, and replay. Backend implementations may select different kernels, but they must preserve the same public request contract.

The generated backend capability report is the source-tree inventory for current trait coverage and known gaps. It is an engineering report, not a promise that every route is equally fast or qualified on every device.

Startup establishes the runtime contract

Startup is the boundary where mutable inputs become immutable runtime policy:

1. Parse command-line options and merge defaults, kiln.toml, and canonical KILN_<SECTION>_<FIELD> overrides.
2. Validate the complete configuration, including cross-field constraints.
3. Resolve the serving profile and backend-specific execution policies.
4. Select a physical device and verify that its memory probe refers to the same accelerator.
5. Inspect device capabilities and install immutable kernel-selection policy.
6. Read and verify the model checkpoint, then upload or bind its weights.
7. Size the memory governor, KV cache, decode buffers, and other resident resources.
8. Precompile or prewarm routes required before readiness.
9. Start the batching, training, and evaluation workers.
10. Bind the HTTP server and publish readiness.

Runtime code consumes the resolved objects. It does not repeatedly reinterpret environment variables in individual layers or kernels. A policy change that affects memory ownership, kernel selection, graph capture, or serving behavior requires a restart.

The effective values and their provenance are available from GET /v1/config; readiness and live state are available from GET /health.

Serving path

An OpenAI-compatible chat request passes through these ownership boundaries:

POST /v1/chat/completions
        │
        ▼
schema validation and serving-profile admission
        │
        ▼
chat-template rendering and tokenization
        │
        ▼
model, adapter, sampling, and cache identity
        │
        ▼
batching actor
        │
        ├── prefill prompt chunks
        ├── decode active rows
        └── settle cancellation or completion
        │
        ▼
JSON response or server-sent events

The HTTP layer owns wire compatibility, validation, and error responses. The batching actor owns admission, request progress, cancellation, and terminal cleanup. ModelRunner owns model execution, adapter application, sampling, backend health, and accelerator-resident state.

Streaming changes response delivery, not model ownership. Generated events travel through a bounded channel to the async response task. Cancellation, timeout, disconnect, and panic paths must still establish whether accelerator work completed before buffers or cache blocks can be reused.

Continuous batching

The real-model batching actor uses a token budget for each scheduling cycle. The default combined budget is 512 tokens, but the resolved configuration is authoritative. Within a cycle, the actor prioritizes:

1. one decode step for each ready row;
2. resumable prompt prefills already in progress;
3. newly admitted prefills that fit the remaining budget.

Long prompts can therefore be split across cycles without preventing ready decode rows from progressing. Partial prefills rotate rather than allowing one long prompt to monopolize every cycle.

The actor assembles compatible rows into backend-supported batches. Effective decode width, prefill quantum, shape constraints, and route support come from the resolved backend policy. A narrow direct-stream worker remains as a compatibility path when the actor is unavailable; it is not a second production scheduler.

Terminal cleanup is part of request lifetime

A request is not complete merely because its final token was sampled. Before the actor removes the row, it must release or settle:

• graph or command-replay ownership;
• recurrent Gated DeltaNet state;
• prefix-cache leases;
• private KV blocks;
• response delivery and cancellation state.

Health counters keep a row active through this cleanup. A reported zero-active boundary therefore means those resources have settled, not only that the row left a scheduling list.

Batch completions and thinking budgets

POST /v1/completions/batch admits a bounded set of prompts and returns one aggregated response. Its rows still use the same scheduler, adapter identity, memory accounting, and cancellation rules as ordinary generation.

Thinking budgets are request-local decode state. Omitted limits inherit server defaults, JSON null means unlimited, and a nonnegative value is an explicit limit. When a budget closes an open reasoning span, the controller emits a tokenizer-validated closing sequence through the normal decode path before answer generation continues. The inference schema defines the exact wire fields.

Model execution

Kiln's current model profile targets Qwen3.5-4B, a 32-layer hybrid model:

• 24 Gated DeltaNet (GDN) linear-attention layers maintain recurrent state;
• 8 grouped-query full-attention layers use a paged KV cache;
• LoRA deltas may be applied to supported projections;
• final normalization, the language-model head, penalties, and sampling produce the next token.

This hybrid shape matters operationally. Only the full-attention layers need the ordinary K/V history, while GDN layers carry fixed-shape recurrent and convolution state. Prefill chunking must preserve both kinds of state across cycle boundaries.

Prefill

Prefill renders the prompt into model state. Depending on the selected backend, prompt length, and immutable streaming-prefill policy, the model runner may use one materialized forward or a tiled forward. Tiling is an execution choice, not a semantic choice: enabled and disabled cache paths must preserve the same numerical prompt partition where a backend requires that invariant.

The full-attention layers write K/V entries. GDN layers update recurrent and convolution state. The final prompt position produces the first decode candidate.

Decode

Decode advances each active row by one sampled token. The backend may use eager execution or an eligible replay mechanism:

Replay is never assumed from the backend name alone. The serving profile, request shape, resource identity, capture status, and backend support decide whether a request is eligible. Failure to prove safe replay falls back only where the route defines a contained fallback; an unknown completion state quarantines the backend.

Speculative decoding implementations remain research and qualification substrate. The serving policy accepts only effective off, and startup rejects an effective non-off configuration before model weights are loaded.

Memory and resident state

Kiln treats accelerator capacity as a shared budget, not as a collection of unrelated allocations.

Physical-device identity

Device selection, the tensor backend, and the operating-system memory probe must resolve to the same physical accelerator. Startup fails if that identity cannot be established. A configured memory value is a cap on detected capacity, not permission to invent capacity that the device does not have.

The memory governor samples live pressure and accounts for model weights, KV cache, training reservations, replay buffers, allocator pools, and other tracked categories. Admission uses the same planning inputs that execution uses, reducing estimate/runtime drift.

Paged KV cache

The production block manager divides K/V storage into 64-token blocks. Each request holds a logical block table that maps token positions to physical slots. Blocks are allocated incrementally and returned only after request cleanup proves they are no longer in use.

Paging avoids a separate maximum-context allocation for every request and allows concurrent rows to share one bounded pool. The exact number of blocks is resolved from the effective memory budget unless the operator supplies a valid override.

Optional FP8 K/V storage uses half the bytes per element of BF16. That is a capacity tradeoff, not a universal quality or speed guarantee; it remains opt-in and must be evaluated for the deployment workload.

Prefix cache

The production prefix cache reuses complete, block-aligned prompt state. Cache identity includes the information needed to keep reused state consistent with the active model, adapter, and generation contract. Retained blocks are reference-counted and evicted within configured block, entry, and state-byte budgets.

Requested and effective cache state are separate. In particular, Vulkan prefix reuse is currently disabled by a source-level correctness quarantine. GET /v1/config reports vulkan_correctness_quarantine when configuration requests the cache but the selected execution route cannot safely use it. Starting successfully does not override that restriction.

Recurrent and replay state

GDN state, graph buffers, command batches, and resident activations have explicit owners and generations. Reusing a resident object requires a matching request identity, shape, device, allocation generation, and active adapter where applicable. Adapter transitions invalidate state whose weight pointers or numerical identity changed.

Accelerator backends

Every backend has two separate obligations:

1. implement a correct route or decline it explicitly;
2. choose compatible, efficient kernels from the active device's capabilities.

Support predicates are request-aware. A backend can support an operation for one dtype, layout, or shape and decline another. A declined optimized route may use a defined reference implementation only when that fallback preserves the operation's residency and performance contract. Missing hot-path support must not silently become an unbounded host round trip.

The backend capability report distinguishes native support, constrained support, portable fallback, and unavailable routes. Hardware qualification adds evidence for a recorded build, driver, device, and workload; it does not rewrite product dispatch.

Vulkan policy is capability-driven

Vulkan runtime selection does not branch on a marketing device name, qualification-machine identity, driver string, vendor ID, or device ID. Those values may appear in diagnostic or benchmark receipts, but they are not execution capabilities.

After selecting the physical device, Kiln reads limits and memory properties such as:

• API version;
• workgroup counts, sizes, and invocation limits;
• shared-memory and push-constant limits;
• descriptor and storage-buffer limits;
• compute subgroup support;
• coherent device-local host-visible memory;
• cached host-visible staging memory.

VulkanKernelPolicy::from_capabilities derives compatible routes from those facts. Individual dispatch sites add workload-dependent checks such as tensor shape, storage range, and dispatch-grid bounds. The resulting policy is installed once for the process; conflicting reinstallation fails.

This design has two consequences:

• a capable Vulkan device can select fast routes without appearing in a model or vendor allowlist;
• a device that lacks a shader requirement retains only compatible routes and explicit fallbacks.

Vulkan's current serving path may execute Vulkan kernels while retaining the frozen model representation on CPU. The runtime therefore tracks execution device and model-weight device separately. This is an explicit hybrid design, not evidence that work silently left Vulkan.

The stable serving profile keeps Vulkan resident prefill disabled until its release qualification closes. The experimental profile may admit that route, subject to capability and request checks. This profile gate is about evidence for a route; it is not a special case for one machine.

CUDA, ROCm, and Metal

CUDA and ROCm expose backend-specific kernel and graph policies; Metal uses MPS, Metal shaders, and indirect command buffers where supported. These backends follow the same boundary as Vulkan: startup resolves policy, runtime queries support for the actual request, and observability reports intended versus effective behavior.

Backend-specific optimizations can have narrower eligibility than baseline inference. Marlin W4A16, FP8 K/V storage, live graph capture, and specialized GDN or attention routes are examples. Their presence in source does not imply that stable serving activates them.

Accelerator ownership and serving profiles

Inference, training, evaluation, adapter transitions, and physical memory maintenance can touch the same accelerator state. A shared coordination lock makes that ownership explicit:

• inference takes shared ownership;
• training and physical mutations take exclusive ownership;
• adapter swaps also cross the batching actor's quiescence barrier;
• GPU-backed eval work obeys the same lock rather than running through an independent device path.

The immutable serving profile decides which owners are admitted:

One process therefore does not mean inference and training execute on the GPU at the same time. A queued training or eval job may exist while inference is active, but accelerator work waits for the ownership policy. Stable serving rejects real-backend training instead of quietly pausing traffic.

If an exclusive owner waits behind accelerator work whose completion becomes unknown, health checks interrupt the wait. Kiln does not free or reuse potentially live device resources merely to make the queue move.

Adapter lifecycle

Adapters have both durable and live identities:

• the adapter directory and manifest identify the saved artifact;
• content hashes identify the exact files;
• the runtime records which revision is physically loaded;
• the server default identifies what adapter-free requests should select.

The live model has one physical adapter state at a time. Loading, unloading, or switching an adapter pauses new actor admission, lets active rows settle, takes exclusive ownership, changes the weights, invalidates dependent caches and replay state, and then resumes admission. Queued requests validate that the revision they expected is still the revision being served.

Per-request adapter selection uses this same transition machinery; it is not an invisible second copy of the model. Adapter composition first produces a content-addressed composed artifact, then loads that artifact through the same validated path.

Adapter publication is transactional. Temporary output is validated before it is renamed into the durable adapter directory. Manifests and receipts bind the artifact to its base model, training inputs, effective configuration, and provenance. Canary failures can quarantine an adapter independently of backend health.

See the adapter manifest and execution provenance references for the exact identity contracts.

OpenEnv rollout path

OpenEnv is the environment-facing half of native reinforcement learning:

CLI train/start/status/artifact · dashboard · /v1/openenv/*
        │
        ├── train preflight → effective GRPO config · exact policy · suite · backend/optimizer
        │                     (before persistence or direct discovery/episodes/GPU work)
        │
        ├── bounded GET health · metadata · schema · environments · OpenAPI
        ├── canonical complete-discovery digest (including extension fields)
        ├── compile self-contained action schema (no HTTP/file resolution)
        │
        ▼
content-addressed environment identity
        │
        ▼
one WS /ws connection per candidate episode
        │
        ├── reset(same seed for every candidate in the group)
        ├── Kiln chat generation → one JSON object
        ├── local advertised-schema validation → reject before step on mismatch
        ├── environment step → observation + tagged reward + done
        └── repeat under explicit step/session/data bounds
        │
        ▼
re-read complete discovery identity for every endpoint
        ├── exact match → continue to artifact derivation
        └── drift/unavailable → typed identity-verification failure, publish nothing
        │
        ▼
AgenticGroup JSONL
        ├── Action segments → GRPO policy loss
        ├── Observation segments → ECHO environment loss
        ├── environment + behavior-policy identity → scored-payload hash and receipt
        └── exact typed protocol transcript → offline verification and live replay
        │
        ▼
ordinary /v1/train/grpo admission and training queue
        │
        ▼
run-owned corpus lineage + train receipt + adapter manifest

kiln-openenv owns the protocol boundary and has no model or optimizer responsibility. kiln-server::openenv_cli composes sessions with the existing chat and training APIs. The server control plane invokes the authoritative chat handler in process, so OpenEnv policy actions retain the ordinary serving profile, adapter, scheduler, metrics, and request-log semantics without a loopback HTTP dependency. It submits completed groups through the same typed GRPO admission function used by POST /v1/train/grpo. kiln-train owns the optional OpenEnv provenance inside canonical ScoredRollout; this lets the same JSONL travel through native GRPO without a parallel training representation.

Both OpenEnv training entrances first materialize the exact config that will later reach native GRPO. Persisted kind=train admission does so before it creates a run; direct kiln openenv train calls POST /v1/openenv/training/preflight before environment discovery and must submit the receipt's exact effective config after collection. Both fix rollout-owned fields and validate the environment-token loss and policy contract, LoRA scale, checkpoint interval, behavior-adapter layout, installed post_eval suite, serving profile, backend workload, optimizer tuple, and model rank ceiling. Failure is synchronous and persists no run or direct artifact, so a typo cannot spend episodes before failing at trainer handoff. Direct preflight also returns a point-in-time queue and tracked-job capacity snapshot with capacity_reserved=false; the final native queue admission still rechecks these invariants plus transient queue and live-memory capacity after collection. Rollout-only requests reject every training-only field. Both the dashboard's static Prove it after training suite and paired environment_eval remain ordinary parts of the same owned lifecycle.

The admitted values become one typed kiln.openenv-training-contract.v1. Persisted v5 status writes it to run.json atomically with queue admission and the executor submits from that field, not from request defaults. Direct train builds the identical type from its preflight receipt. Every sampled action returns the exact base-model, inference-config/runtime, and optional adapter content revision. Collection rejects identity drift within an episode or corpus; native admission compares that identity with the selected policy. After the final episode, the collector enters revalidating, re-reads every endpoint's complete stable discovery identity and schemas, and requires exact equality with the initial inspection before it derives artifacts. A redeploy is retained as environment_identity_changed at identity_verification; no mixed-identity corpus is published or submitted. Summary v5 embeds both contract and identity before the first artifact publication, so failed handoff is still auditable. Under the adapter-mutation barrier, execution revalidates the identity and privately snapshots a distinct behavior adapter before GPU work.

Server-owned runs are bounded by [openenv] policy and persisted below <adapter_dir>/.openenv/runs/<run-id>/. run.json records request, immutable training contract, discovery identity, progress, artifact-bound rollout quality/recovery statistics, lifecycle state, artifacts, error, and optional training job. max_active_runs admits complete workflows in strict FIFO order; excess valid submissions remain bounded by max_tracked_runs, expose one-based queue position, stable admission sequence, and admission wait, and can be cancelled without touching an environment or model. The execution permit spans collection, training, static evaluation, and paired environment evaluation. A restart resumes v5 FIFO entries that provably never acquired a slot; pristine v4 entries are sealed once into v5 first. It converts interrupted admitted work into explicit terminal failure and never guesses whether a stateful external episode can resume. An optional bounded idempotency_key is part of the persisted normalized request. While its run is retained, atomic concurrent retries return the original status without adding queue work; reuse with any changed request field returns a conflict. Dataset, replay, and summary downloads stream from those owned paths. Collection is cooperatively cancellable through capacity backoff and episode boundaries; after handoff, the workflow forwards cancellation to the authoritative native trainer and linked evaluators. kiln openenv start submits the same bounded run request and follows that state machine. kiln openenv artifact consumes only an exact manifest entry, disables redirects, verifies response headers, length, and SHA-256 while staging beside the destination, and publishes atomically only after the independent check. When its trainer completes, the run projects the admitted TrainingDataProvenance and atomically retains the validated native train_receipt.json and adapter_manifest.json. Publication proves successful status, output-adapter identity, exact corpus digest, semantic OpenEnv plan, and the manifest-to-receipt hash link. These small records are capped at 4 MiB each and become ordinary run-manifest artifacts, so the OpenEnv lifecycle remains self-contained after adapter retention or installation changes.

The WebSocket path is load-bearing. Every connection repeats the status-only GET /health readiness precondition immediately before upgrade. OpenEnv's HTTP /reset and /step routes construct a fresh environment for each request and cannot carry episode state. The client is strictly lock-step because OpenEnv has no correlation IDs and permits no server-initiated application messages. While the selected policy is generating, the client keeps polling the socket solely to answer Ping control frames; application data in that idle window is a protocol violation. Once an exchange times out, returns an unreadable frame, or answers with the wrong message type, the session is permanently poisoned because no safe resynchronization exists.

The action schema is an executable protocol boundary, not prompt decoration. kiln-openenv compiles it during every inspection with draft auto-detection and without HTTP or filesystem resolvers; internal references remain available, while malformed or external-reference schemas fail discovery as a non-retryable environment protocol error. The collector reuses the compiled validator for every generated action. A mismatch produces bounded keyword and JSON Pointer evidence, receives the configured protocol-error reward, terminates as invalid_model_action, and never writes the action to the episode socket.

Discovery identity is computed before typed projection. Kiln canonicalizes and hashes the complete /metadata, /schema, /list_environments, and /openapi.json JSON values under kiln.openenv-discovery.v1; object-key order and whitespace are irrelevant, while unknown extension fields remain bound. Current rollout and corpus provenance carry that digest. Legacy v1 artifacts without it remain readable, but every current collection and revalidation requires it.

Every group is assigned exactly one environment and reset seed. Candidates may run concurrently, but their initial messages must be identical or collection fails. Step rewards—not reset rewards—sum into episode return. Environment done, Kiln's max-step cutoff, invalid policy JSON/schema, and protocol errors remain distinct termination states. Recoverable OpenEnv errors remain on the same lock-step socket and become policy feedback under a bounded budget. A CAPACITY_REACHED socket is terminal, but episode acquisition is not: Kiln backs off and opens a fresh socket under an explicit wait deadline.

Reset configuration is either one shared object or an ordered object per environment. The collector removes any caller-supplied seed, inserts its deterministic group seed, and assigns whole groups round-robin. It requires at least one group per configured endpoint. Summary v5 hashes the ordered, seed-free plan; offline verification reconstructs that plan from replay groups and rejects missing environments, per-environment drift, or a digest mismatch.

The canonical JSONL stays the only training representation. A separate content-addressed replay manifest retains exact reset payloads, reset observations, ordered action/result exchanges, and final states. The summary links both artifacts by digest. Offline verification cross-checks hashes, counts, provenance, and reward arithmetic; live replay also pins the discovered discovery identity and all schemas before comparing every wire result exactly.

The server's artifact manifest is the publication boundary. A known filename inside a run directory is not downloadable until its digest and byte count appear in persisted run status. Publication hashes bounded regular, non-symlink files and cross-checks the dataset and replay against the summary; download repeats both checks on one file descriptor and streams that same descriptor with exact length and digest headers. Disk drift therefore becomes a structured integrity failure, never an unverified training-data response. The CLI repeats the manifest, header, length, and digest checks rather than trusting either a guessed pathname or transport success.

The exact summary statistics are projected into persisted run status rather than hidden behind an artifact download. CLI follow and dashboard cards expose return range, terminal outcomes, recoverable errors, capacity retries, environment steps, policy tokens, and latency for training and both paired-eval halves. /metrics accumulates the same work with only closed termination labels; environment URLs, adapter names, and run IDs never become metric dimensions.

The boundary is intentionally bounded: redirects are disabled, discovery bodies and WebSocket frames have independent byte limits, client messages, environment count, sessions, groups, candidates, steps, recoveries, capacity wait, action tokens, replay/data artifact bytes, and the inline corpus are capped, and terminal OpenEnv errors latch the session closed. The collector also charges every retained reset, action, observation, error, and final state against one 512 MiB aggregate budget before appending it. Candidate structures are compacted into group/replay/receipt ownership by move, reset files are bounded before read, group hashing streams, and replay encoding refuses the next write at its 256 MiB cap. A hostile but protocol-legal group therefore fails without publishing partial artifacts instead of accumulating the full candidate-count × step-count × frame-size product in memory. Reproducible CI uses a pinned miniopenenv counter and derives every published arcade, synthesis, and math server from that checkout's build graph; a scheduled edge lane repeats the same matrix against upstream main. This oracle inventory remains CI-only. No production setting, type, or runtime branch identifies that implementation; every server follows the same OpenEnv contract.

See the OpenEnv training guide for the operator workflow and artifact contract.

Training path

Training submissions follow this path:

HTTP request or dashboard form
        │
        ▼
schema and dataset validation
        │
        ▼
backend-route and memory admission
        │
        ▼
FIFO training queue
        │
        ▼
exclusive accelerator ownership
        │
        ▼
forward → loss → backward → optimizer
        │
        ▼
checkpoint and adapter publication
        │
        ├── optional post-training eval gate
        └── optional explicit or configured activation

The training worker executes jobs sequentially. Supported workloads are:

• SFT: supervised next-token loss over prepared examples;
• GRPO: grouped rollouts, reward scoring, normalized advantages, and policy updates;
• OPD: on-policy sampling followed by teacher-guided distillation loss.

Admission asks the selected backend for its training tape, loss, precision, and optimizer routes. A compiled backend is not sufficient: if the exact workload requires an unavailable route, the request fails before taking accelerator ownership.

SFT may use gradient checkpointing and sparse boundary replay to trade additional computation for lower activation memory. Planning identity records the effective checkpoint and loss route. Exact resume rejects drift in those inputs instead of continuing with a different memory or numerical contract.

Checkpoints include the state required by their declared resume mode, while final adapters use the project artifact format. Cancellation is checked at defined work boundaries and must settle backend work before a job becomes terminal.

Evaluation pipeline

The evaluation subsystem stores named datasets and suites under the configured eval.eval_dir. A separate FIFO worker executes registered suites, inline suites, comparisons, and replay jobs. CPU-only scoring can proceed without accelerator ownership; generation and model-based judging use the shared model runtime and its ownership rules.

Each suite combines examples with one or more scorers. Scorers include exact or structured comparisons, regular-expression and numeric checks, code execution where explicitly configured, and local model judging. A local judge adapter is loaded through the same adapter transition boundary as any other adapter.

Strict replay binds the suite, generation settings, seeds, model and adapter identities, scorer configuration, execution provenance, and raw completion references. Admission verifies that the recorded environment still matches; execution rechecks the loaded candidate and judge before generation. The terminal verdict is matched, mismatch, or error.

A training request can attach a post-eval gate. On successful training, Kiln queues the adapter and optional baseline evaluations, records their IDs on the training job, and applies the configured promotion rule to the resulting evidence. Auto-load on pass is explicit policy, not an implication of job completion.

The judgment workflow turns locally collected A/B preferences into training examples for a judge adapter. It does not require a hosted frontier model unless the operator separately configures a remote teacher.

Failure and health boundaries

Kiln distinguishes failures by the state they can invalidate:

The backend health latch preserves the first reason an asynchronous completion can no longer be proven. Once latched, new accelerator work is rejected and potentially in-flight resources are retained rather than freed unsafely. Readiness reports the quarantine and restart requirement.

Fallback is therefore route-specific. Kiln may use a declared eager or reference route after a contained, fully settled failure. It may not treat an unknown accelerator state as a recoverable performance miss.

Observability

The main operator surfaces answer different questions:

The debug model-state endpoint is disabled unless server.debug_model_state = true or eval mode enables it. It is intended for a trusted diagnostic boundary.

Kiln's HTTP API has no built-in authentication. The default loopback bind is the safe local boundary. If you expose the server beyond a trusted host, put it behind an authenticated reverse proxy and appropriate network controls. Request observability can include sensitive prompt or completion text, especially the recent-request diagnostics and optional durable request log.

Latency reporting separates queue time, accelerator-lock wait, prefill, decode, graph capture or replay, response backpressure, training, resizing, trimming, and adapter transitions where those phases are observable. That separation is necessary: a slow request is not automatically a slow kernel.

Configuration and source of truth

Do not copy configuration fields from this architecture page. Use:

• Configuration guide for precedence, validation, and examples;
• complete configuration reference for every supported field;
• configuration schema for machine-readable constraints;
• HTTP API reference for endpoints and wire shapes;
• observability schema for health, config, metrics, and diagnostic payloads.

For a claim about current backend coverage, regenerate or inspect docs/backend-capability-report.md. For a performance claim, use Benchmarks and retain its exact revision, build, device, driver, workload, and metric provenance. A benchmark result is evidence about that recorded run; it is never a device-name dispatch rule.

Where to continue

• Architecture overview for the shorter system map;
• GRPO guide for the generate–score–train workflow;
• Evals guide for suites, scorers, comparisons, and replay;
• Latency observability for phase definitions;
• Troubleshooting for operator-facing diagnosis;
• Security policy for deployment and vulnerability-reporting boundaries.