Kiln Documentation

Operations and development

Source-parsing test debt

Keep source-text assertions out of correctness claims, understand the scanner’s exact scope, and replace any future debt with behavioral evidence.

Generated by python3 scripts/check_source_parsing_tests.py --write; do not edit by hand.

This gate finds tests that read implementation or qualification-driver source as text and then treat spelling, substring, or split assertions as behavioral proof. Kiln does not accept that pattern as correctness evidence: it can detect a source edit while never exercising the behavior it claims to protect.

This is the exact zero-debt gate and migration queue for that anti-pattern. It is not an inventory of every test, guarantee, platform, or coverage gap in Kiln. Use the Verification policy for evidence classes and claim ownership.

Current baseline

  • Tests: 0
  • Direct or module-bound source reads: 0
  • Text search/split assertions: 0
  • Rust tests: 0
  • Python qualification tests: 0

The three limits are monotonic: --write lowers them after a migration but refuses to bless an increase. The contract is contracts/source-parsing-test-inventory-v1.json.

What zero proves

A zero baseline proves that the scanner found no in-scope Rust test or Python qualification test that both reads implementation source as text and uses the ratcheted text-assertion patterns. It does not prove that:

  • runtime behavior is correct;
  • every requirement has a test;
  • every platform executed;
  • tests assert the right outcome;
  • source files contain no string operations; or
  • documentation-rendering tests are forbidden from checking rendered text.

The scanner recognizes Rust #[test] and #[...::test] functions plus Python qualification test methods. Canonical contracts, schemas, fixtures, and workload documents are structured test inputs; production and qualification-driver source files are not.

Replacement policy

Classification Required replacement
generated_artifact_text Parse the canonical JSON/schema and assert typed fields; reserve rendered-text checks for the documentation renderer.
implementation_source_text Replace with compile-time trait/type constraints, injected runtime behavior, property tests, or structured ownership metadata consumed by production.
qualification_driver_source_text Exercise a public helper or CLI preflight and assert its structured output instead of matching the driver's Python source.

A compile-only type constraint proves interface shape. An injected runtime test proves routing and failure behavior. A property/state-machine test proves transition invariants. Canonical JSON or schema validation proves generated metadata. Substring presence in a .rs, .cu, or driver file proves none of those things.

Ownership concentration

None. No test currently enters the source-parsing migration queue.

Migration queue

Empty. Any future row is technical debt, not correctness evidence.

Gate

Run:

python3 scripts/check_source_parsing_tests.py

If a change introduces an entry, replace the source-text assertion with the appropriate compile-time, runtime, property/state-machine, or structured-contract test. After removing existing debt, run the same command with --write; it regenerates the exact contract and report and lowers each applicable ceiling.

Regeneration refuses an increase. Raising a ceiling requires an explicit reviewed contract edit; there is no hidden command-line bypass.