tests/ — read them; they are the specification
V=1 make test
That is the cheapest orientation available in this repository, and it is the
first thing ../docs/TOUR.md tells you to do. Every case
name here is a sentence, and read end to end they are the specification of the
wire layer:
a_control_frame_may_sit_between_two_fragments
nesting_is_capped_before_the_stack_is
no_cut_changes_the_answer
a_request_becomes_slices_of_the_caller_s_own_bytes
the_two_transports_must_disagree
one_frame_two_caps_and_the_reader_decides_which
With V unset the suite prints one line per binary; with V=1 it prints every
claim it is about to check.
There is no framework
t.h is the entire harness and it is sixty-odd lines. No registration macro, no
runner binary, no discovery: a test file is a program, main calls its cases
in order, and the exit status is the result. make test links each t_*.c
against the library and runs it.
The one thing worth more than sixty lines is the failure message, because a
byte-exact project fails by one byte. T_EQ_MEM prints both buffers escaped,
with a caret under the first byte that differs and its hex value — the
difference between "assertion failed" and knowing what to fix.
What runs, and what it costs
25 binaries, 11,200 checks, run twice: as built,
then under ASan + UBSan with OE_ARENA_DEBUG_MALLOC=1. That switch turns every
arena allocation into a separately-poisoned malloc, so a retained pointer is a
diagnosed use-after-free rather than a value that happens to still be there.
Almost none of it opens a socket. That is not thrift; it is the layering
paying out. src/proto/ is a pure function from one frame to one frame, so the
protocol can be tested by replaying recorded bytes, and src/server/'s seam
makes no system call either.
| Binary | The claim it decides | Socket? |
|---|---|---|
t_replay | 103 of the 118 reference-corpus exchanges, byte for byte; the other 15 named, with reasons | no |
t_conform | the six official criteria, then six the official suite has no equivalent of | forks a server |
t_hollow | a server that scores 6/6 and runs no episode — the same six criteria, shown necessary but not sufficient | forks a server |
t_conn | five connection states, the capacity refusal, the handshake deadline | no |
t_encoders | that the two transports disagree, on purpose | no |
t_pyrepr | Python's repr(), the two arms no codec reached: the \xNN escape and the buffer-edge cut | no |
t_decode, t_validate, t_errors, t_schema, t_routes, t_jsonrpc, t_tool | the protocol layer | no |
t_json_parse, t_json_write, t_ws, t_http, t_utf8, t_str, t_buf, t_arena, t_bytes, t_hash | the wire layer | no |
t_client | parts 1–5 the codec; part 6 forks our own server and drives a real episode. make demo-two-readers runs it for the two-readers demonstration in part 4 | part 6 |
t_net | the one file with #ifdefs in it | yes |
t_replay — the corpus, the fifteen it diverges on, and the two normalisations
One hundred and three cases — a hundred and eighteen exchanges — recorded byte for byte from
six real OpenEnv servers, fed back into the four functions of
<openenv/embed.h> and diffed against what came out. No port, no clock, no
Python, single-digit milliseconds.
One hundred and three of the one hundred and eighteen replay. The binary says so itself, in the lines it always prints:
REPLAYED 103 exchanges, byte for byte
EXPECTED-DIVERGENT 15 exchanges
REFUSED 14 bytes we will not emit, on evidence
UNREPRODUCIBLE 1 a Python library's artefact, not ours to forge
NOT-YET 0 implementable, and not yet built
OUT-OF-SCOPE 0 exchanges
103 cases -> 118 exchanges: 88 replay whole, 14 do not, 1 split
The verdict is per EXCHANGE and not per case, and that is the whole difference.
A WebSocket case is a connection of up to eight frames, and while a case stood
or fell whole, one bad frame took three good ones out of the corpus with it.
The fifteen are named individually with their reasons, below the counts, on
every run, each naming the exact field. They are: two OpenAPI documents this
project deliberately writes shorter (one of them the production-mode document);
six MCP rows where the reference's answer is a Python library's serialisation
rather than OpenEnv's; one tool-door row — the undeclared keyword argument
CPython refuses for a tool, which our agent door now refuses too but cannot
spell in the interpreter's own words; and six task-API rows where the reference
env declares a task provider and we answer 501, a refusal on evidence
(GAP-24). The two other tool-door rows and the three
production-mode 404s became REPLAYs as items 7, 8 and 9 landed. GET /schema
now publishes the framework's action model; the step door enforces it; and
production is a parameter that removes /reset, /step and /state. Each
divergence also carries one of three verdicts — refused on evidence,
unreproducible without being a Python library, or not yet built — and the three
subtotals shown above are asserted. The not-yet count is now zero: the arc
that closed it drove the bucket to empty. It stays asserted, so a future
capture we could match but have not costs somebody a line rather than
fossilising out of sight. The divergences are enumerated here for the same
reason the two normalisations below are: a subtraction nobody states is a
subtraction nobody can check.
Two normalisations, stated here because a silent one would be a lie:
date:andserver: uvicornare deleted from every recorded response before the comparison. The first needs a wall clock, which the pure layer does not have and must not have; the second is a claim about a program this is not. Every other byte — the status line, the header names, their order, their spelling, and the body — is compared exactly.- A WebSocket close is compared as silence. The recorded response is the RFC
6455 close frame's payload, which belongs to
src/wire/ws.cand is tested int_ws.c. What this file asserts is the protocol fact: that the session appends no text frame at all.
And a third thing, which is not a normalisation: fifteen of the
one hundred and eighteen exchanges do not replay. The phrasing matches t_replay's
own banner above, so the prose and the program output can be diffed by eye. Every
one is printed with its reason whether it passed or not, and the test asserts
all four counts. So a case the classifier ever quietly dropped is a failure,
not a shrinking number nobody watches.
../docs/PROVENANCE.md is where they are accounted for.
t_conform — and the line it prints in the middle
$ V=1 ./build/rel/tests/t_conform
- conf_01_openapi_version_available
...
validate --url: 6/6 required criteria passed, profile openenv-http/1.x
--- beyond the official suite: the WebSocket no criterion
opens, and the episode that exists only there ---
- the_handshake_the_official_suite_never_performs
...
The six criteria above the separator are openenv validate --url reimplemented
in C, so the official conformance suite runs on every commit without a Python
interpreter. The separator is the teaching device: the official suite never
opens a WebSocket, so the thing OpenEnv exists for is unverified by it, and the
six checks below the line are what that costs.
The subdirectories
fixtures/ | reference-corpus.json — the 103 captured cases. Produced by interop/recapture.py, which needs six reference servers running and ships launchers for five; deliberately checked in |
fuzz/ | six targets with invariant oracles, standalone_main.c so they run without libFuzzer, and a checked-in seed corpus including the regression inputs for two real bugs. make fuzz-run |
autobahn/ | echo.c — src/wire/ws.c with a socket and nothing in between — plus expected.txt, one line per case with its reason. make autobahn |
embed/ | three third-party consumers, each really built and really run: pkg-config after a staged install, the sources vendored under a hostile flag set, and <openenv/embed.h> alone. make check-embed |
counter_env.h and reward_env.h are the two shipped environments without a
main, because the replay must link no socket and a program must not include
from tests/. Each is paired with its envs/ twin: change one, change the
other.
What is deliberately not here
Anything that needs a second implementation. make test is hermetic on
purpose — no Python, no network, no clock — so it runs anywhere and never
flakes. The live oracles are make interop, and they skip cleanly on a clone
with no reference tree. ../docs/PROVENANCE.md says why
a corpus alone is not enough: a fossil cannot be asked a new question.
Mechanical rules, too. Those are tools/check.sh, which decides the layer
DAG, the banned functions, src/proto/ purity and the staleness of the
generated documents. make check runs it and then this suite.
See also
../docs/FUZZING.md— the six targets and their oracles, the Autobahn results case by case, and the two library bugs they found../docs/PROVENANCE.md— the corpus, and the fifteen live oracles beyond it../docs/PACKAGING.md— whattests/embed/proves../CONVENTIONS.md↗ — §12, every command that gates a change