miniopenenv

tests/README.md → html

tests/ — read them; they are the specification

V=1 make test

That is the cheapest orientation available in this repository, and it is the first thing ../docs/TOUR.md tells you to do. Every case name here is a sentence, and read end to end they are the specification of the wire layer:

a_control_frame_may_sit_between_two_fragments
nesting_is_capped_before_the_stack_is
no_cut_changes_the_answer
a_request_becomes_slices_of_the_caller_s_own_bytes
the_two_transports_must_disagree
one_frame_two_caps_and_the_reader_decides_which

With V unset the suite prints one line per binary; with V=1 it prints every claim it is about to check.

§ 1 · There is no framework

There is no framework

t.h is the entire harness and it is sixty-odd lines. No registration macro, no runner binary, no discovery: a test file is a program, main calls its cases in order, and the exit status is the result. make test links each t_*.c against the library and runs it.

The one thing worth more than sixty lines is the failure message, because a byte-exact project fails by one byte. T_EQ_MEM prints both buffers escaped, with a caret under the first byte that differs and its hex value — the difference between "assertion failed" and knowing what to fix.

§ 2 · What runs, and what it costs

What runs, and what it costs

25 binaries, 11,200 checks, run twice: as built, then under ASan + UBSan with OE_ARENA_DEBUG_MALLOC=1. That switch turns every arena allocation into a separately-poisoned malloc, so a retained pointer is a diagnosed use-after-free rather than a value that happens to still be there.

Almost none of it opens a socket. That is not thrift; it is the layering paying out. src/proto/ is a pure function from one frame to one frame, so the protocol can be tested by replaying recorded bytes, and src/server/'s seam makes no system call either.

BinaryThe claim it decidesSocket?
t_replay103 of the 118 reference-corpus exchanges, byte for byte; the other 15 named, with reasonsno
t_conformthe six official criteria, then six the official suite has no equivalent offorks a server
t_hollowa server that scores 6/6 and runs no episode — the same six criteria, shown necessary but not sufficientforks a server
t_connfive connection states, the capacity refusal, the handshake deadlineno
t_encodersthat the two transports disagree, on purposeno
t_pyreprPython's repr(), the two arms no codec reached: the \xNN escape and the buffer-edge cutno
t_decode, t_validate, t_errors, t_schema, t_routes, t_jsonrpc, t_toolthe protocol layerno
t_json_parse, t_json_write, t_ws, t_http, t_utf8, t_str, t_buf, t_arena, t_bytes, t_hashthe wire layerno
t_clientparts 1–5 the codec; part 6 forks our own server and drives a real episode. make demo-two-readers runs it for the two-readers demonstration in part 4part 6
t_netthe one file with #ifdefs in ityes
§ 3 · t_replay — the corpus, the fifteen it diverges on, and the two normalisations

t_replay — the corpus, the fifteen it diverges on, and the two normalisations

One hundred and three cases — a hundred and eighteen exchanges — recorded byte for byte from six real OpenEnv servers, fed back into the four functions of <openenv/embed.h> and diffed against what came out. No port, no clock, no Python, single-digit milliseconds.

One hundred and three of the one hundred and eighteen replay. The binary says so itself, in the lines it always prints:

REPLAYED             103 exchanges, byte for byte
EXPECTED-DIVERGENT   15 exchanges
  REFUSED           14  bytes we will not emit, on evidence
  UNREPRODUCIBLE      1  a Python library's artefact, not ours to forge
  NOT-YET             0  implementable, and not yet built
OUT-OF-SCOPE          0 exchanges
103 cases -> 118 exchanges: 88 replay whole, 14 do not, 1 split

The verdict is per EXCHANGE and not per case, and that is the whole difference. A WebSocket case is a connection of up to eight frames, and while a case stood or fell whole, one bad frame took three good ones out of the corpus with it. The fifteen are named individually with their reasons, below the counts, on every run, each naming the exact field. They are: two OpenAPI documents this project deliberately writes shorter (one of them the production-mode document); six MCP rows where the reference's answer is a Python library's serialisation rather than OpenEnv's; one tool-door row — the undeclared keyword argument CPython refuses for a tool, which our agent door now refuses too but cannot spell in the interpreter's own words; and six task-API rows where the reference env declares a task provider and we answer 501, a refusal on evidence (GAP-24). The two other tool-door rows and the three production-mode 404s became REPLAYs as items 7, 8 and 9 landed. GET /schema now publishes the framework's action model; the step door enforces it; and production is a parameter that removes /reset, /step and /state. Each divergence also carries one of three verdicts — refused on evidence, unreproducible without being a Python library, or not yet built — and the three subtotals shown above are asserted. The not-yet count is now zero: the arc that closed it drove the bucket to empty. It stays asserted, so a future capture we could match but have not costs somebody a line rather than fossilising out of sight. The divergences are enumerated here for the same reason the two normalisations below are: a subtraction nobody states is a subtraction nobody can check.

Two normalisations, stated here because a silent one would be a lie:

  1. date: and server: uvicorn are deleted from every recorded response before the comparison. The first needs a wall clock, which the pure layer does not have and must not have; the second is a claim about a program this is not. Every other byte — the status line, the header names, their order, their spelling, and the body — is compared exactly.
  2. A WebSocket close is compared as silence. The recorded response is the RFC 6455 close frame's payload, which belongs to src/wire/ws.c and is tested in t_ws.c. What this file asserts is the protocol fact: that the session appends no text frame at all.

And a third thing, which is not a normalisation: fifteen of the one hundred and eighteen exchanges do not replay. The phrasing matches t_replay's own banner above, so the prose and the program output can be diffed by eye. Every one is printed with its reason whether it passed or not, and the test asserts all four counts. So a case the classifier ever quietly dropped is a failure, not a shrinking number nobody watches. ../docs/PROVENANCE.md is where they are accounted for.

§ 4 · t_conform — and the line it prints in the middle

t_conform — and the line it prints in the middle

$ V=1 ./build/rel/tests/t_conform
  - conf_01_openapi_version_available
  ...
     validate --url: 6/6 required criteria passed, profile openenv-http/1.x
     --- beyond the official suite: the WebSocket no criterion
         opens, and the episode that exists only there ---
  - the_handshake_the_official_suite_never_performs
  ...

The six criteria above the separator are openenv validate --url reimplemented in C, so the official conformance suite runs on every commit without a Python interpreter. The separator is the teaching device: the official suite never opens a WebSocket, so the thing OpenEnv exists for is unverified by it, and the six checks below the line are what that costs.

§ 5 · The subdirectories

The subdirectories

fixtures/reference-corpus.json — the 103 captured cases. Produced by interop/recapture.py, which needs six reference servers running and ships launchers for five; deliberately checked in
fuzz/six targets with invariant oracles, standalone_main.c so they run without libFuzzer, and a checked-in seed corpus including the regression inputs for two real bugs. make fuzz-run
autobahn/echo.c — src/wire/ws.c with a socket and nothing in between — plus expected.txt, one line per case with its reason. make autobahn
embed/three third-party consumers, each really built and really run: pkg-config after a staged install, the sources vendored under a hostile flag set, and <openenv/embed.h> alone. make check-embed

counter_env.h and reward_env.h are the two shipped environments without a main, because the replay must link no socket and a program must not include from tests/. Each is paired with its envs/ twin: change one, change the other.

§ 6 · What is deliberately not here

What is deliberately not here

Anything that needs a second implementation. make test is hermetic on purpose — no Python, no network, no clock — so it runs anywhere and never flakes. The live oracles are make interop, and they skip cleanly on a clone with no reference tree. ../docs/PROVENANCE.md says why a corpus alone is not enough: a fossil cannot be asked a new question.

Mechanical rules, too. Those are tools/check.sh, which decides the layer DAG, the banned functions, src/proto/ purity and the staleness of the generated documents. make check runs it and then this suite.

§ 7 · See also

See also