Integrity and artifacts

Qualification workload contract

Define a deterministic, device-neutral local workload with bounded inputs, owned evidence, and explicit comparison rules.

Closed, source-bound contract for one deterministic local qualification workload. JSON Schema validates the wire shape; scripts/qualification/workload.py additionally enforces placeholder delivery, bounded execution, canonical ordering, result ownership, and comparison semantics.

Schema identity

Property	Value
Title	Kiln local qualification workload
`$id`	`https://kiln.local/schemas/qualification-workload-v1.json`
Dialect	`https://json-schema.org/draft/2020-12/schema`
Root type	`object`
Root object	closed

Root fields

Field	Required	Type	Constraints and default	Description
`schema_version`	yes	`any`	const 1	Workload contract version. The validator accepts only version 1.
`workload_id`	yes	`id`	-	Stable identifier recorded in each case result and qualification receipt.
`kind`	yes	`any`	enum "environment", "correctness", "serving", "performance", "training", "eval", "soak"	Purpose of the workload. The kind constrains seed delivery and which comparison modes are valid.
`description`	yes	`string`	minimum length 1; maximum length 512	Human explanation of the behavior, boundary, or performance claim being qualified.
`determinism`	yes	`determinism`	-	Seed, repetition, ordering, concurrency, and network rules owned by the runner.
`variables`	yes	`array<variable>`	-	Sorted declarations for operator-supplied scalar placeholders. Every declaration must be consumed, and every required variable must be consumed by every variant.
`variants`	yes	`array<variant>`	minimum items 1	Sorted backend/configuration variants. Each variant runs its cases serially in declared order.
`comparison_policy`	yes	`comparisonPolicy \| null`	-	Explicit metric comparison contract, or null only for environment and soak workloads.

Definitions

idLowercase stable identifier, 3–128 characters, using letters, digits, dots, underscores, or hyphens.0 fields

Type: string. Constraints: pattern ^[a-z0-9][a-z0-9._-]{2,127}$.

nullableScalarFinite JSON scalar or null. The semantic validator rejects non-finite numeric values.0 fields

Type: string | number | boolean | null.

configValueOne finite scalar, null, or recursively nested configuration object.0 fields

Type: nullableScalar | configObject.

Composition and conditional rules

The following JSON is copied exactly from this schema node.

{
  "oneOf": [
    {
      "$ref": "#/$defs/nullableScalar"
    },
    {
      "$ref": "#/$defs/configObject"
    }
  ]
}

configObjectConfiguration tree whose lowercase keys can be addressed as dot-separated comparison paths.0 fields

Type: object.

Composition and conditional rules

The following JSON is copied exactly from this schema node.

{
  "propertyNames": {
    "pattern": "^[a-z][a-z0-9_-]*$"
  },
  "additionalProperties": {
    "$ref": "#/$defs/configValue"
  }
}

determinismRunner-enforced execution envelope. Cases are serial, ordered, offline, and bounded.6 fields

Type: object. Constraints: closed object.

Field	Required	Type	Constraints and default	Description
`seed`	yes	`integer \| null`	-	Non-negative workload seed. It is null only for environment workloads.
`seed_delivery`	yes	`any`	enum "not_applicable", "fixed_fixture", "argv", "environment"	How every variant receives the seed. argv and environment require the reserved ${seed} placeholder in the corresponding surface.
`repetitions`	yes	`integer`	minimum 1; maximum 1000	Number of times the runner executes the declared case sequence.
`case_order`	yes	`any`	const "declared"	Cases run in the order stored in each variant.
`max_parallel_cases`	yes	`any`	const 1	The runner executes at most one case at a time to avoid undeclared contention.
`network_access`	yes	`any`	const "forbidden"	Cases must use committed or local inputs and cannot depend on network state.

variableOne operator-resolved scalar placeholder declaration. Variable names are sorted and unique; runner-owned seed and model_path are reserved.6 fields

Type: object. Constraints: closed object.

Field	Required	Type	Constraints and default	Description
`name`	yes	`string`	pattern ^[a-z][a-z0-9_]{1,63}$	Placeholder name without ${...}. It must not collide with a runner-owned placeholder.
`description`	yes	`string`	minimum length 1; maximum length 256	Operator-facing meaning and unit of the value.
`type`	yes	`any`	enum "string", "integer", "number", "boolean"	Exact JSON scalar type accepted by parameter resolution.
`required`	yes	`boolean`	-	Whether the operator must supply the value. Required variables must use a null default and be consumed by every variant.
`default`	yes	`nullableScalar`	-	Optional resolved value when required is false. It must satisfy the declared type and constraints.
`constraints`	yes	`object`	closed object	Complete type-aware bounds. Unused bound kinds remain explicitly null.

outputAssertionRegular-expression assertion applied to a case's captured output. Duplicate assertions are rejected.3 fields

Type: object. Constraints: closed object.

Field	Required	Type	Constraints and default	Description
`stream`	yes	`any`	enum "stdout", "stderr", "combined"	Captured stream against which the regular expression is evaluated.
`match`	yes	`any`	enum "required", "forbidden"	Whether the pattern must occur or must not occur.
`pattern`	yes	`string`	minimum length 1; maximum length 512	Valid regular expression evaluated by the qualification runner.

caseOne bounded argv-based command and its exact exit, output, and structured-result contract.10 fields

Type: object. Constraints: closed object.

Field	Required	Type	Constraints and default	Description
`id`	yes	`id`	-	Stable result identifier. Compared variants must declare identical case/result contracts.
`description`	yes	`string`	minimum length 1; maximum length 512	Human explanation of the operation and its expected evidence.
`required`	yes	`boolean`	-	Whether a skip or failure makes qualification fail. At least one case per variant must be required.
`command`	yes	`array<string>`	minimum items 1	Argument vector executed without shell-string evaluation. A placeholder must occupy a complete argument; the executable cannot be a placeholder.
`working_directory`	yes	`string`	minimum length 1	Repository-relative POSIX path. Absolute paths, backslashes, and parent traversal are rejected.
`environment`	yes	`object`	-	Case-owned environment additions. Runner-owned variables cannot be overridden, and placeholders must occupy complete values.
`timeout_seconds`	yes	`integer`	minimum 1; maximum 172800	Per-execution deadline in seconds. The semantic validator also caps aggregate declared wall time.
`expected_exit_codes`	yes	`array<integer>`	minimum items 1; unique items	Unique accepted process exit codes in ascending order.
`output_assertions`	yes	`array<outputAssertion>`	-	Required and forbidden patterns checked after command completion.
`result_protocol`	yes	`object`	closed object	Ownership and declared metric names for the strict case-result file.

variantOne backend and effective-configuration selection with a complete serial case list. Variant IDs are sorted and unique.7 fields

Type: object. Constraints: closed object.

Field	Required	Type	Constraints and default	Description
`id`	yes	`id`	-	Stable variant identifier used by parameter resolution and comparison endpoints.
`description`	yes	`string`	minimum length 1; maximum length 512	Human explanation of the backend or configuration selection.
`backend`	yes	`any`	enum "cpu", "cuda", "rocm", "vulkan", "metal"	Portable backend family. The contract does not select a machine model.
`device_requirement`	yes	`any`	enum "none", "required"	Whether the selected backend device must exist. Accelerator variants must use required.
`skip_policy`	yes	`any`	enum "allow", "fail"	How an unavailable case is classified. A required device forces fail.
`effective_config`	yes	`configObject`	-	Resolved, comparison-visible configuration tree for this variant. Non-empty trees require command-produced results.
`cases`	yes	`array<case>`	minimum items 1	Non-empty serial case list. Compared variants must expose identical required flags, producers, and declared metrics by case ID.

variantPairSame-backend A/B endpoint pair. Both variants must consume the same variables and declare identical result contracts.3 fields

Type: object. Constraints: closed object.

Field	Required	Type	Constraints and default	Description
`baseline_variant_id`	yes	`id`	-	Declared same-backend baseline variant.
`candidate_variant_id`	yes	`id`	-	Declared same-backend candidate variant.
`allowed_effective_config_differences`	yes	`array<string>`	minimum items 1; unique items	Sorted dot paths that must exactly equal the scalar leaf differences between baseline and candidate effective_config.

backendPairCanonical cross-backend correctness pair. Endpoints use different backend families but identical variables, effective configuration, and result contracts.5 fields

Type: object. Constraints: closed object.

Field	Required	Type	Constraints and default	Description
`backend_a`	yes	`any`	enum "cpu", "cuda", "rocm", "vulkan", "metal"	Backend family for canonical endpoint A.
`variant_a_id`	yes	`id`	-	Variant whose declared backend must equal backend_a.
`backend_b`	yes	`any`	enum "cpu", "cuda", "rocm", "vulkan", "metal"	Different backend family for canonical endpoint B.
`variant_b_id`	yes	`id`	-	Variant whose declared backend must equal backend_b.
`allowed_environment_differences`	yes	`array<string>`	minimum items 1; unique items	Sorted environment-identity paths acknowledged as expected backend differences. This list does not relax metric equality.

metricRuleOne required or advisory comparison over a metric declared by a required case in every selected endpoint.11 fields

Type: object. Constraints: closed object.

Field	Required	Type	Constraints and default	Description
`scope`	yes	`any`	const "result"	Metrics are owned by one case result.
`result_id`	yes	`id`	-	Required case ID that declares the named metric in every compared variant.
`metric`	yes	`string`	pattern ^[a-z][a-z0-9_.-]{0,127}$	Exact metric key emitted by the declared result producer.
`metric_class`	yes	`any`	enum "correctness", "performance"	Evidence class. Cross-backend rules allow correctness only; performance modes require performance.
`unit`	yes	`string`	minimum length 1	Unit attached to every metric observation. Runner-owned metrics have fixed canonical units.
`aggregation`	yes	`string`	minimum length 1	Declared reduction across repetitions. Runner-owned metrics have fixed canonical aggregations.
`lower_is_better`	yes	`boolean`	-	Metric direction used to validate one-sided operators.
`operator`	yes	`any`	enum "equal", "not_greater", "not_less"	Candidate comparison: equality within tolerance, no greater than baseline, or no less than baseline.
`absolute_tolerance`	yes	`number`	minimum 0	Non-negative tolerance in the declared unit.
`relative_tolerance`	yes	`number`	minimum 0	Non-negative fractional tolerance relative to the baseline magnitude.
`required`	yes	`boolean`	-	Whether a failed or missing comparison makes the verdict fail. A non-empty rule set needs at least one required rule.

comparisonPolicyClosed declaration of which receipts may be compared and how each result metric decides pass or fail.4 fields

Type: object. Constraints: closed object.

Field	Required	Type	Constraints and default	Description
`mode`	yes	`any`	enum "same_environment_performance", "declared_ab_variants", "cross_backend_correctness", "self_contained_correctness"	Comparison topology. Environment and soak workloads omit the policy; self-contained correctness uses command assertions without metric pairs.
`variant_pairs`	yes	`array<variantPair>`	-	Same-backend A/B pairs, used only by declared_ab_variants.
`backend_pairs`	yes	`array<backendPair>`	-	Cross-backend endpoint pairs, used only by cross_backend_correctness.
`metric_rules`	yes	`array<metricRule>`	-	Metric verdict rules. Performance comparisons require command-produced performance evidence; cross-backend correctness requires equal, required correctness rules.