CLI guide · serve · openenv · pi-setup · train · eval · adapters

Run Kiln from the terminal

Use the CLI to start a headless server, automate a workflow, or diagnose a running Kiln instance. The examples use Qwen/Qwen3.5-4B; see Quickstart for installation and model download. Release archives contain the kiln binary only. Eval, benchmark, replay, and Vulkan research clients require a source build, documented below.

Start the server Check health Find each binary Configure pi Train with OpenEnv Submit training Run evals Run offline tools Manage adapters

Command chooser

Choose a task

Start the server

Point Kiln at your model directory, then serve the OpenAI-compatible API.

export KILN_MODEL_PATH=./Qwen3.5-4B
kiln serve

Verify readiness

Use the readable tree for humans or JSON for scripts and CI probes.

kiln health
kiln health --json

Train in an OpenEnv

Discover a stateful RL environment, collect seed-matched episodes, and train from its rewards.

kiln openenv inspect --environment http://127.0.0.1:8990
kiln openenv tasks --environment http://127.0.0.1:8990 --split train
kiln openenv train --environment http://127.0.0.1:8990 --output-adapter counter-agent
kiln openenv verify --summary openenv.rollout-summary.json

Submit training

Send native training data, export or import a verified HF/TRL handoff, or check progress.

kiln train sft --file corrections.jsonl --adapter support-bot
kiln train grpo --file scored-groups.jsonl --adapter support-bot
kiln train hf export-sft --file /data/corrections.jsonl --name support-hf-01
kiln train hf export-grpo --file /data/recorded.jsonl --name support-grpo-01
kiln train hf import-peft --bundle ./support-hf-01.kiln-hf --name support-v2
kiln train status

Point pi at Kiln

Back up and merge pi's models/settings files so pi uses the running Kiln server.

kiln pi-setup
kiln pi-setup --kiln-url http://office-kiln:8420

Manage adapters

List adapters, load a saved LoRA, or unload the active adapter.

kiln adapters list
kiln adapters load support-bot
kiln adapters unload

Run and replay evals

Grade an adapter, compare candidates, or verify a completed run's exact decoder bytes.

./target/release/kiln-eval run --suite smoke --adapter support-bot --watch
./target/release/kiln-eval compare --suite smoke --adapter "" --adapter support-bot --watch
./target/release/kiln-eval replay --job eval_123 --json

Validate config

Check a TOML config and optionally resolve one target backend’s static scheduling contract before using it with kiln serve --config.

kiln config --file kiln.toml
kiln config --file kiln.toml --backend rocm
kiln config --file kiln.toml --backend rocm --json

Binary availability

Know which commands the release archive includes

Current prebuilt server archives contain only kiln. Build the auxiliary clients from a source checkout when you need them. The commands below place each binary in ./target/release/; use that path explicitly or add the directory to your PATH.

cargo build --release --locked -p kiln-server \
  --bin kiln-eval --bin kiln-bench

cargo build --release --locked -p kiln-train \
  --bin kiln-replay

cargo build --release --locked -p kiln-vulkan-kernel \
  --bin vulkan_decode_microbench

Binary	Purpose	Distribution
`kiln`	Serve, health, configuration, training, adapters, pi setup, and rollout generation.	Prebuilt server archive.
`kiln-eval`	Register, run, compare, probe, and strict-replay evals.	Source build.
`kiln-bench`	Offline model-path benchmark driver.	Source build.
`kiln-replay`	Historical request-lineage hash inspection; not model replay.	Source build.
`vulkan_decode_microbench`	Vulkan kernel research, not serving qualification.	Source build.

Server

Start serving Qwen3.5-4B

Point KILN_MODEL_PATH at the local model directory, then start the OpenAI-compatible server. Running kiln with no subcommand starts the server just like kiln serve.

export KILN_MODEL_PATH=./Qwen3.5-4B
kiln serve

The default server listens on 127.0.0.1:8420; open /ui there for the dashboard.

Configuration

Use config files and model IDs

--config (or -c) loads a TOML config file. --served-model-id changes the model name returned by /v1/models and accepted by OpenAI-compatible clients.

kiln config
kiln config --file kiln.toml
kiln config --file kiln.toml --backend rocm
kiln config --file kiln.toml --backend rocm --json
kiln serve -c kiln.toml
kiln serve --config kiln.toml
kiln serve --served-model-id Qwen3.5-4B-local

Use kiln config to validate built-in defaults plus KILN_* environment overrides, or kiln config --file / kiln config -f to validate a TOML file before starting the server. --backend cpu|cuda|rocm|metal|vulkan resolves hardware-independent scheduling policy and applies the serve-startup actor-prefill validator without probing hardware or loading model weights. --json emits the complete kiln.config-check.v1 machine document.

Advanced startup policy Effective values, CUDA graphs, batching, prefix caching, and streaming prefill

Effective configuration

Inspect resolved startup values and their sources

The JSON check contains kiln.effective-configuration.v1. Its flat, sorted field map records every typed value after precedence resolution, the default/config_file/environment/command_line source, canonical environment spelling, compatibility names, redaction state, and restart requirement.

kiln config --file kiln.toml --json | jq '.effective_configuration'
kiln config --file kiln.toml --backend vulkan --json | jq '.backend_preview'
curl -fsS http://localhost:8420/v1/config | jq '.effective_configuration.fields["server.max_decode_batch"]'

The CLI path parses and validates configuration without probing hardware or loading weights. A running server publishes its immutable startup snapshot in GET /v1/config under effective_configuration; sibling accelerator, batching, memory, and availability objects are authoritative for backend-derived and live results. Self-improvement requests, webhook URLs, and teacher credential-provider names remain represented with their source but serialize as redacted null values.

CUDA graph policy

Bound retained decode graphs at startup

Single-row CUDA graph capture and its retained geometry limit are typed memory policy. The cache limit is 1..=64, defaults to 8, and every change requires restart. Stable paged metadata is mandatory; batched CUDA graph capture is unavailable and has no hidden opt-in.

[server]
serving_profile = "stable"

[memory]
cuda_graphs = true
cuda_graph_cache_entries = 8

kiln config --file kiln.toml
kiln serve --config kiln.toml
curl -fsS http://localhost:8420/v1/config | jq '.cuda_graphs'
curl -fsS http://localhost:8420/health | jq '.decode_runtime.cuda_graphs'

The canonical cache override is KILN_MEMORY_CUDA_GRAPH_CACHE_ENTRIES. The default stable profile permits guarded single-row capture; drained maintenance resolves to eager execution. Decode never re-reads process environment.

Batching policy

Resolve scheduler controls at startup

Put production scheduling controls in typed TOML. The canonical environment rule is mechanically derived as KILN_BATCHING_<FIELD>, but files are easier to review and bind into benchmark evidence. Every change requires a server restart.

[batching]
rowwise_decode = false
prefix_aware_admission = true
prefill_admission_quantum = "auto"
actor_cycle_idle_ms = 0

kiln config --file kiln.toml --backend rocm
kiln serve --config kiln.toml
curl -fsS http://localhost:8420/v1/config | jq '.batching'

kiln config validates and prints all four batching fields plus the combined actor-cycle budget, prompt-token and layer ceilings, decode-width ceiling, streaming mode, threshold, and three configured tiles, each with source provenance. Every real backend uses one batching actor; activation and the former direct worker are not configurable. actor_cycle_idle_ms is an explicit 0–60,000 ms safe-boundary duty-cycle delay: zero is unpaced; nonzero trades throughput and ITL for lower sustained accelerator duty cycle and polls control commands within 5 ms. Its canonical environment override is KILN_BATCHING_ACTOR_CYCLE_IDLE_MS. Add --backend to resolve the alignment requirement, decode width, streaming dispatch, and tiles and reject an invalid contract without probing hardware or loading model weights. Without it, the command leaves backend-effective values unresolved. After restart, /v1/config reports backend policy, effective decode-width clamping, and actor_active.

Prefix-cache policy

Separate requested policy from effective admission

The three typed fields express startup intent. Omitted capacities are derived only when the active backend admits cross-request KV and recurrent-state reuse.

[prefix_cache]
enabled = true
# max_blocks = 2048
# max_entries = 16

kiln config --file kiln.toml
kiln serve --config kiln.toml
curl -fsS http://localhost:8420/v1/config | jq '.prefix_cache'
curl -fsS http://localhost:8420/health | jq '.prefix_cache, .decode_runtime.batching_engine.prefix_cache_enabled'

enabled defaults to true; max_blocks and max_entries are optional positive integers. CPU, CUDA, ROCm, and Metal currently honor the request. Vulkan reports effective_enabled=false and effective_reason="vulkan_correctness_quarantine", allocates no prefix-cache capacity, and fresh-prefills exact repeats. No request or environment override can bypass that source-level quarantine.

Streaming-prefill policy

Resolve long-prefill and training tiles once

Use typed TOML for the six process-lifetime controls. auto preserves backend dispatch and tile policy; every concrete tile must be a positive multiple of 64. A concrete base tile is inherited by auto tape and detached routes, including detached boundary and replay work.

[streaming_prefill]
mode = "auto"
threshold_tokens = "auto"
tile_tokens = "auto"
tape_tile_tokens = "auto"
detached_full_attn_tile_tokens = "auto"
last_token_lm_head = true

Disable for an A/B

[streaming_prefill]
mode = "disabled"

Force non-empty prompts

[streaming_prefill]
mode = "enabled"

Tune ROCm crossover

[streaming_prefill]
mode = "auto"
threshold_tokens = 4096
tile_tokens = 2048

kiln config --file kiln.toml
kiln serve --config kiln.toml
curl -fsS http://localhost:8420/v1/config | jq '.streaming_prefill'
curl -fsS http://localhost:8420/health | jq '.prefill_runtime.streaming_prefill'

Canonical overrides are mechanically derived: KILN_STREAMING_PREFILL_MODE, KILN_STREAMING_PREFILL_THRESHOLD_TOKENS, KILN_STREAMING_PREFILL_TILE_TOKENS, KILN_STREAMING_PREFILL_TAPE_TILE_TOKENS, KILN_STREAMING_PREFILL_DETACHED_FULL_ATTN_TILE_TOKENS, and KILN_STREAMING_PREFILL_LAST_TOKEN_LM_HEAD. Six old shorter spellings warn and conflict-check; legacy TOML enabled is compatibility-only. Startup injects one policy into inference and native training, so restart after every change. The two diagnostic objects must match.

Logging

Tune startup output

Kiln uses the configured log level by default. Add global -v / --verbose for debug startup detail, repeat it as -vv for trace-level kernel and scheduler detail, or use -q / --quiet when you only want warnings and errors. Add --help when you want the exact flags from the installed binary; the examples below stay focused on copy-paste startup and health commands.

kiln -v serve
kiln -vv serve
kiln -q health

Put verbosity flags before or after the subcommand; they are global CLI options and are mutually exclusive between verbose and quiet modes.

Health

Check server readiness

kiln health prints a readable tree with model, adapter, scheduler, and training status.

kiln health
kiln health --url http://localhost:8420

Point --url at a remote, Tailscale, or reverse-proxied server when Kiln runs on another machine.

Use --json in scripts or CI probes when you want the raw health payload.

kiln health --json \
  | python3 -m json.tool

pi integration

Point pi or OpenCode at Kiln

kiln pi-setup merges a kiln-local provider into pi’s existing configuration and sets Kiln as the default provider and model. The local examples target http://localhost:8420/v1; Kiln does not require an API key on its loopback API. OpenCode’s OpenAI-compatible provider accepts a local endpoint without an apiKey.

pi · local setup

kiln pi-setup
pi -p "Use the bash tool to run: pwd"

Existing models.json and settings.json are copied to timestamped .bak-... files before the merge.

pi · remote server

kiln pi-setup --kiln-url http://office-kiln:8420

--kiln-url is an alias for --url. If the URL omits /v1, Kiln appends it for pi.

OpenCode · provider config

Add a Kiln provider to ~/.config/opencode/opencode.json, then run opencode and pick Kiln · Qwen3.5-4B with /models. OpenCode documents @ai-sdk/openai-compatible, options.baseURL, and an optional options.apiKey for custom providers; see its custom-provider guide.

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "kiln": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "Kiln (local)",
      "options": { "baseURL": "http://localhost:8420/v1" },
      "models": { "Qwen3.5-4B": { "name": "Kiln · Qwen3.5-4B" } }
    }
  }
}

Tool-calling clients receive OpenAI tool_calls even when Qwen3.5 internally emits its native XML tool-call form, so pi and opencode execute the tool instead of showing raw XML.

Kiln does not add API authentication. Use a remote URL only on a trusted network or behind an authenticated reverse proxy, especially when training and adapter-management routes are enabled.

OpenEnv reinforcement learning

Act, observe, reward, and train in one command

kiln openenv is the implementation-neutral, native environment-facing GRPO workflow. It content-addresses every standard OpenEnv discovery surface, health-checks each stateful WebSocket connection, gives every candidate in a comparison group the same reset seed, asks the running Kiln model for action-schema JSON, preserves every action and observation for ECHO, and sums environment step rewards into episode return. Recoverable protocol errors become corrective turns on the same socket; capacity saturation uses bounded fresh-session acquisition.

Inspect, catalog, and collect

kiln openenv inspect \
  --environment http://127.0.0.1:8990

kiln openenv tasks \
  --environment http://127.0.0.1:8990 \
  --split train --start 0 --limit 25

kiln openenv rollout \
  --environment http://127.0.0.1:8990 \
  --groups 16 \
  --group-size 4 \
  --output counter.rollouts.jsonl \
  --replay-output counter.replay.json

Inspection reports metadata, action/observation/state schemas, the derived WS /ws URL, protocol profile, schema SHA-256, and a canonical complete-discovery SHA-256 over raw metadata, schema, inventory, and OpenAPI JSON. Unknown extension fields are bound; whitespace and object-key order are not. tasks safely pages an optional provider catalog; conforming 501 means unsupported, and catalog rows are never implicitly treated as reset data. Rollout collection writes canonical GRPO JSONL, an exact environment transcript, and a content-addressed summary receipt without starting training.

Collect and train

kiln openenv train \
  --environment http://127.0.0.1:8990 \
  --environment-reset-options arcade.json \
  --environment http://127.0.0.1:8991 \
  --environment-reset-options math.json \
  --adapter base \
  --groups 32 \
  --group-size 8 \
  --max-steps 8 \
  --output-adapter counter-agent

Before environment discovery, direct training calls POST /v1/openenv/training/preflight, validates the server-owned effective GRPO config, and later submits those exact values. Summary v5 embeds the resulting kiln.openenv-training-contract.v1 before first artifact publication. Rejection contacts no environment and writes no artifacts. The returned capacity snapshot is explicitly non-reserving, so final native queue and memory admission still apply after collection. Discovery compiles each self-contained action JSON Schema without external HTTP/filesystem resolution, every session repeats status-only health immediately before upgrade, and every model action is validated before step; a mismatch never reaches the environment and is counted as invalid_model_action. After the last episode Kiln revalidates every endpoint’s canonical complete-discovery identity; a mid-run redeploy publishes no artifacts and reports environment_identity_changed. Successful artifacts are published before native on-policy GRPO submission. Use one shared --reset-options FILE, or repeat --environment-reset-options FILE_OR_DASH exactly once per URL for a heterogeneous portfolio; - means an empty object. Kiln owns seed, requires at least one group per endpoint, and binds the ordered plan in the verifiable receipt. Environment done, max-step cutoffs, invalid model JSON/schema, terminal errors, and exhausted recovery budgets remain distinct.

Authenticate a protected environment

ARCADE_OPENENV_TOKEN=... kiln openenv train \
  --environment https://arcade.example.com/openenv \
  --credential-env ARCADE_OPENENV_TOKEN \
  --output-adapter arcade-agent

Direct CLI commands name a secret environment variable. Dashboard/API workflows instead submit an opaque credential_ids entry backed by exact-origin [openenv.credentials.<id>] server configuration. Kiln sends the bearer header on discovery and the WebSocket upgrade, records only authentication: bearer in environment identity, and excludes handles, variable names, and tokens from artifacts.

Gate promotion on held-out environment returns

"environment_eval": {
  "groups": 20,
  "group_size": 1,
  "gate": {
    "min_mean_return": 0.5,
    "min_mean_improvement": 0.05
  }
}

kiln openenv start --request openenv-run.json \
  --idempotency-key experiment:arcade:17 --follow

Add this field to openenv-run.json and submit the complete persisted workflow with start, send the same object to POST /v1/openenv/runs, or choose the paired gate in Training → OpenEnv. Kiln evaluates behavior and candidate policies on identical, disjoint seeds and defers auto-load until at least 20 seed groups show a significant exact sign-test win over per-seed mean returns plus configured thresholds. Same-seed replications do not inflate significance. Status and kiln.openenv-environment-evaluation.v1 preserve the evidence and outcome.

Verify offline, replay live

kiln openenv verify --summary counter.rollout-summary.json
kiln openenv replay --summary counter.rollout-summary.json

verify rehashes and cross-checks the dataset, exact transcript, ordered per-environment reset plan, reward arithmetic, provenance, admitted training contract, and receipt without network access; legacy v2/v3 receipts remain readable. replay verifies first, pins each live target’s complete captured OpenEnv discovery identity and schemas, and compares every reset, observation, tagged reward, error, done flag, and final state exactly.

Follow a server-owned workflow

kiln openenv start --request openenv-run.json \
  --idempotency-key experiment:arcade:17 --follow
kiln openenv runs
kiln openenv status 80a26e21-8451-4a64-8666-890c06fd80bd --follow
kiln openenv artifact 80a26e21-8451-4a64-8666-890c06fd80bd \
  environment_eval_receipt --output receipt.json
kiln openenv cancel 80a26e21-8451-4a64-8666-890c06fd80bd

start submits the same full run contract used by the dashboard and API. For train runs, Kiln preflights the exact effective GRPO config, behavior adapter, installed post_eval suite, backend/workload, optimizer, and rank before it persists a run or opens an environment session. V5 status and run.json seal those values as kiln.openenv-training-contract.v1; queue resume and submission use that field rather than current defaults. Add the static suite in the JSON request or use Prove it after training in the dashboard. Bind one non-secret --idempotency-key to an experiment attempt: an exact retry recovers the retained run, while changed semantics fail closed. When execution slots are occupied, status follows a bounded FIFO with live position and admission wait; cancel removes queued work immediately or cooperatively stops the active phase. artifact accepts only the exact kind and same-server URL returned by status, requires the declared length and ETag, rehashes the bounded stream, and atomically publishes no partial or accidental replacement.

OpenEnv servers remain untrusted inputs even when authenticated. Prefer loopback/private networks or HTTPS/WSS with exact-origin credentials. The production runtime depends only on OpenEnv-compatible discovery and WebSocket behavior; miniopenenv is used solely as a pinned byte-real CI oracle. See the complete OpenEnv Training Guide for wire semantics, limits, reward typing, identity, interoperability, and troubleshooting.

Training

Run native jobs or move a verified HF/TRL handoff

Training commands talk to an already-running server. SFT reads JSONL with one chat correction example per line, each with a messages array. GRPO accepts either one JSON request/batch with groups or streamed JSONL with one group per line; each group has prompt messages plus candidate completions containing text and reward scores. OPD reads one /v1/train/opd request object, or a JSON prompt array paired with --teacher, and binds the registered teacher's exact content revision. The nested train hf commands create and manage immutable bundles for broader Transformers/TRL/PEFT training.

The default server admits training

The default stable profile supports training and coordinated live adapter transitions. Use a drained maintenance process only for artifact production without inference or eval. These CLI commands do not expose auto_load or post_eval; they inherit the server’s auto_load: true training default. Use the HTTP training API with auto_load: false or a held-out post_eval gate when activation must depend on evidence. Training takes exclusive accelerator ownership while the queued job runs.

SFT corrections

kiln train sft \
  --file corrections.jsonl \
  --adapter support-bot \
  --checkpoint-interval 25
kiln train sft --file corrections.jsonl --adapter support-bot --url http://gpu-box:8420

SFT flags and resume contract

Each JSONL line is one chat correction with a messages array. Native SFT uses the fixed native_online_lora_v1 profile: one conversation per update, constant learning rate, and no accumulation, warmup, decay, or clipping. Add --epochs, --lr, or --lora-rank to override its bounded settings. Continue an interrupted run with the same file and options plus --resume-checkpoint BASENAME. --adapter-smoke-test runs post-training canaries; --adapter-smoke-prompts-file loads a JSON string array or one text prompt into request-local config. Add --detect-anomaly only when localizing a NaN or Inf; it scans every backward operation and can substantially slow the job.

GRPO rewards

kiln train grpo \
  --file scored-groups.jsonl \
  --adapter support-bot \
  --checkpoint-interval 25

kiln train grpo \
  --file scored-groups.jsonl \
  --adapter support-bot \
  --checkpoint-interval 25 \
  --resume-checkpoint support-bot-checkpoint-step-00000025.kiln-checkpoint

GRPO input, resume, and diagnostic flags

Use /v1/completions/batch or another generator to create prompts and candidate completions first, score them, then submit the scored groups. A .jsonl file selects the memory-bounded streamed route; JSON containing groups selects inline training. Exact resume requires the identical file, route, adapter, and options. Adapter canaries use the same --adapter-smoke-test and --adapter-smoke-prompts-file controls as SFT. --no-shared-prefix-reference selects the exact per-completion KL-reference fallback for a deliberate comparison. --detect-anomaly enables the same request-local per-operation NaN/Inf diagnostic as SFT. See the GRPO Guide.

Verified HF/TRL SFT and GRPO handoff

kiln train hf export-sft \
  --file /data/corrections.jsonl \
  --name support-hf-01

mkdir -p ./handoffs
kiln train hf export-sft \
  --file /data/corrections.jsonl \
  --name support-hf-01 \
  --output ./handoffs/support-hf-01.tar.gz

# Or snapshot a named server dataset
kiln train hf export-sft \
  --dataset corrections:active \
  --name corrections-hf-01

# Recorded GRPO: canonical final-LF JSONL with exact rollout provenance
kiln train hf export-grpo \
  --file /data/recorded-rollouts.jsonl \
  --name support-grpo-01 \
  --output ./handoffs/support-grpo-01.tar.gz

# After extracting and running the embedded trainer
kiln train hf import-peft \
  --bundle ./support-hf-01.kiln-hf \
  --name support-v2

kiln train hf list
kiln train hf list --json
kiln train hf delete \
  --name retained-export \
  --export-sha256 sha256:<64-hex-from-list>

Export and import integrity contract Input location, archive validation, conditional cleanup, and no-replace publication

--file is read by the server process; it is not uploaded from the CLI host. SFT requires exactly one --file or --dataset; GRPO requires canonical compact, final-LF --file input whose uniform-width groups carry exact recorded provenance. Add --input-adapter NAME, --split-manifest FILE, or --keep-server-copy to either route; --invalid-row-policy skip is SFT-only. Both exports refuse redirects and existing outputs, require response ETag identity agreement, bound idle reads and compressed/expanded bytes, stream into a sibling temporary file, reject links, special entries, duplicate paths, trailing gzip data, and any root other than {name}.kiln-hf, verify every manifest-bound byte, then publish atomically. Automatic cleanup and safe manual retry use identity-conditional If-Match through --export-sha256, so a concurrently reused server name is never deleted; omit that flag only for deliberate operator cleanup of the current name or a damaged bundle. Import takes the completed extracted .kiln-hf directory, fails local verification before connecting, streams a deterministic ten-file envelope through bounded memory, and requires HTTP 201 plus exact locally predicted receipt, ETag, PEFT revision, installed size, and file count. It never changes or removes the source bundle; an existing target is rejected without replacement.

OPD teacher distillation

kiln train opd \
  --file opd-request.json \
  --adapter distilled-bot \
  --teacher qwen35@vllm \
  --checkpoint-interval 25

kiln train opd \
  --file opd-request.json \
  --adapter distilled-bot \
  --teacher qwen35@vllm \
  --checkpoint-interval 25 \
  --resume-checkpoint distilled-bot-checkpoint-step-00000025.kiln-checkpoint

OPD resume and sampler flags

OPD defaults to an exact checkpoint every 25 committed optimizer steps. Exact resume requires the identical request data, effective configuration, output adapter, and currently registered teacher content revision; a same-name replacement is rejected. Job detail and the dashboard expose the candidate cursor and immutable basename. --sampler-segments configures the memory-bounded student sampler; --rollout-prompt-rendering selects the qualified admitted-token boundary or the experimental chat-template algorithm. --detect-anomaly opts this job into per-operation NaN/Inf scans and is retained in exact resume state.

Queue status

kiln train status
kiln train status --job-id train_123
kiln train status --url http://gpu-box:8420

Use the dashboard when you want visual progress, recent job history, or a fail-closed resume handoff. The same --url flag works for SFT, GRPO, OPD, status, and HF/TRL export-management commands.

Request-lineage integrity

./target/release/kiln-replay verify ./adapters/support-bot
./target/release/kiln-replay show ./adapters/support-bot

kiln-replay is a historical command name. It recomputes request-lineage hashes only; it does not load a model, execute training or inference, compare outputs, or prove reproducibility. Use a validated .kiln-checkpoint for exact continuation. See the request-lineage integrity contract.

Evaluation

Run, compare, and strictly replay evals

kiln-eval is a source-built client, not part of the prebuilt server archive. It talks to the running server selected by KILN_SERVER_URL, and can register suites, run registered or local suite files, compare adapters on shared sampling inputs, create one-example probes, and strictly replay a completed run.

Suites and runs

./target/release/kiln-eval list
./target/release/kiln-eval register --file smoke.json --force
./target/release/kiln-eval run --suite smoke --adapter support-bot --seed 42 --watch
./target/release/kiln-eval run --file smoke.json --adapter support-bot --watch --json
./target/release/kiln-eval compare --suite smoke --adapter "" --adapter support-bot --seed 42 --watch
./target/release/kiln-eval probe --prompt "1+1?" --target 2 --scorer numeric --adapter support-bot --seed 42

run accepts exactly one registered --suite or local --file. --watch polls to terminal state; --json preserves full result, execution, base-weight, and replay evidence. Compare uses one effective seed across every adapter.

Strict byte replay

./target/release/kiln-eval replay --job eval_123
./target/release/kiln-eval replay --job eval_compare_123 --run-index 1 --json

--run-index defaults to zero and selects one source arm. The command always waits for a terminal matched, mismatch, or error verdict and exits nonzero unless the complete replay identity and every raw decoder continuation match byte-for-byte. Admission refuses legacy records and any execution, base-weight, candidate, or judge identity drift. See Strict replay for the exact proof boundary.

Offline tools

Configure benchmark runs explicitly

kiln-bench loads the same typed KilnConfig as the server. The Vulkan microbenchmark is a separate kernel-research executable. Both are source-built developer tools rather than release-archive binaries. Neither binary reads private KILN_BENCH_* or KILN_VK_MICROBENCH_* controls; every experiment-specific choice is visible in the command. Invalid enums, lists, numeric values, and unknown flags fail before accelerator work.

./target/release/kiln-bench --model-path ./Qwen3.5-4B --config kiln.toml \
  --paged --latency-only --prompt-tokens 512 --max-output-tokens 128

./target/release/vulkan_decode_microbench --only full_step_resident,full_token_resident_paged \
  --batches 1,4,8 --warmup-iters 10 --timed-iters 30 --repeats 5

Complete kiln-bench argument table Defaults, validation, research flags, and qualification boundary

Complete `kiln-bench` arguments

Argument	Default	Contract
`--config <path>`	normal discovery	Typed TOML loaded before device selection.
`--model-path <path>`	required	Local model/tokenizer directory.
`--max-output-tokens <n>`, `--prompt-tokens <n>`	128, 512	Both must be positive.
`--training-steps <n>`, `--skip-training`	10, false	Controls the SFT arm.
`--paged`, `--latency-only`, `--latency-warmup-runs <n>`	false, false, 0	Production paged path, phase selection, and whole-pass warmup.
`--seed <u64>`, `--temperature <f32>`	42, 0.0	Temperature must be finite and non-negative.
`--chat-template`, `--prompt-subset <name>`	false, all	Subset is `all\|gsm8k\|humaneval\|c4`.
`--spec-method <method>`	`[speculative]`	`off\|skip_layer\|mtp`; explicit per-run override.
`--spec-num-tokens <n>`, `--spec-draft-layers <n>`	`[speculative]`	Validated against the model; draft window is currently 1–4.
`--force-mtp`, `--log-tokens`, `--log-itl`	false	Explicit research routing and paged-baseline diagnostics.
`--allow-experimental-speculative`	false	Required for non-off speculation. Acknowledges unsupported research behavior; it never makes a direct run qualification evidence.
`-v`, `--verbose`, `-vv`, `-q`, `--quiet`	warn	Quiet wins over verbosity.

Only a declared workload executed by scripts/qualification/run.py and a strictly checked retained receipt constitute qualification evidence. The experimental benchmark opt-in is only an accident-prevention acknowledgment.

Complete Vulkan microbenchmark argument table Kernel cases, batch sizes, thresholds, and route toggles

Complete Vulkan microbenchmark arguments

Argument	Exact default/effect
`--only <case,...>`	All cases. Names: `full_attn_qkv`, `mlp_bf16_gu_f32_d`, `mlp_bf16w`, `linear_decode`, `causal_conv1d_update`, `gdn_gated_norm`, `qwen_rmsnorm`, `gdn_gates`, `gdn_in_proj`, `gdn_block_resident_batched`, `full_step_resident`, `full_step_resident_batched`, `full_token_resident_batched`, `full_token_resident_mixed_batched`, `full_token_resident_mixed_paged`, `full_token_resident_paged`.
`--batches <n,...>`	`1,4,8,16,32,64`; positive integers only.
`--warmup-iters`, `--timed-iters`, `--repeats`	10, 30, 5; each positive.
`--attention-history`, `--paged-history`, `--paged-block-size`	256, 256, 16; each positive.
`--mlp-rows8-min-batch`, `--mlp-gate-up-rows4-min-batch`, `--mlp-down-rows4-min-batch`	256, 8, 16.
`--linear-rows8-min-batch`, `--linear-rows4-min-batch`	64, 16.
`--gdn-in-proj-rows8-min-batch`, `--gdn-in-proj-rows4-min-batch`	64, 16.
`--full-attn-qkv-rows8-min-batch`, `--full-attn-qkv-rows4-min-batch`	64, 2.
`--disable-linear-rows4`, `--disable-linear-rows8`	Both routes default enabled.
`--disable-mlp-rows8`, `--disable-mlp-gate-up-rows4`, `--disable-mlp-down-rows4`	All three routes default enabled.
`--disable-full-attn-qkv-rows4`, `--disable-full-attn-qkv-rows8`	Both routes default enabled.
`--disable-gdn-in-proj-pair-qkv-z`, `--disable-gdn-in-proj-row-pair`, `--disable-gdn-in-proj-row-quad`	All three routes default enabled.
`--enable-gdn-in-proj-row-octet`, `--enable-gdn-in-proj-conv-split-fusion`	Both routes default disabled.
`--disable-gdn-qk-norm-recurrent-fusion`	Fusion defaults enabled.

These switches affect only the offline Vulkan research binary. Product serving chooses dispatch through the backend’s versioned runtime policy, not these microbenchmark flags or a machine-specific override. See Architecture. A microbenchmark result is not a serving qualification receipt.

Adapters

Manage LoRA adapters

Adapter commands call the running server’s adapter API. Listing is read-only; loading and unloading weights are coordinated under the default stable profile, while maintenance drains inference first. Use these commands in scripts; use /ui for upload, download, merge, or visual confirmation before deletion.

List, load, unload

kiln adapters list
kiln adapters list --url http://gpu-box:8420
kiln adapters load support-bot
kiln adapters unload
kiln adapters unload support-bot

The named unload form is accepted for backwards compatibility; the server unloads the active adapter. Add --url to target a remote server.

Delete

kiln adapters delete support-bot

Delete permanently removes an idle adapter from the server registry; an active or physically loaded adapter is rejected. There is no trash or automatic undo. Prefer the UI for one-off cleanup.