The tour
This page is the guided reading order for the repository: which files to read, in what order, and what question each one answers. The dependency graph is the compiler's order. This document is yours.
Sixteen numbered steps — twenty-two stops, counting 6b, 7b, 7c, 8b, 8c and 13b — about an hour. Steps 1–7 are the complete answer to "what is OpenEnv"; everything after them is how a server that has to survive strangers is built underneath it.
Every step below exists in the tree today.
The path is also IN the tree, not only in this file. Every source file below carries its step in its opening line, so an editor is enough:
grep -rn 'READ:' src include envs tour examples | sort -t: -k3
tools/check.sh requires the set of READ:n markers in the source and the set
of numbered rows in the table below to be equal. So neither can be edited
without the other. Step 1 is this repository's README and is the one row with
no file to mark.
| # | Read | What it answers |
|---|---|---|
| 1 | README.md | What this is, and the two-column line count |
| 2 | tour/01_minimal.c (and its cited Python twin) | The protocol, entire. 64 statements of C beside 78 of Python. make lines prints the split |
| 3 | include/openenv/openenv.h | The public API, three tags, no ceremony |
| 4 | envs/echo.c | What an environment author actually writes — 45 statements, no allocator, no error branch |
| 5 | envs/counter.c | ... with state, and a done condition. A literal port of interop/probe_env.py, which is the environment the corpus was captured against. tests/counter_env.h is the same table with no main, because the replay links no socket |
| 6 | src/proto/types.h | The message vocabulary, named once — five message types on one socket |
| 6b | src/proto/routes.h | ... and the other half of it: sixteen routes, as a table |
| 7 | src/proto/encode.c | The two encoders, and the diff between them. envs/rewards.c is that diff as a program: one step, 1.0 on HTTP and true on the WebSocket, and its banner is the shell recipe |
| 7b | src/proto/errors.c | What a refusal looks like, and the url rule that is not what it looks like |
| 7c | src/proto/validate.c | Lax coercion, and why a strict validator fails the replay |
| 8 | tour/02_conformant.c | The same environment, written with the library. 77 statements against 01's 254, no socket half at all, sixteen routes instead of two — and the same five server frames byte for byte. Its banner carries the diff that proves it |
| 8b | src/proto/client_codec.c | The other end of the same five messages. What a client may send, what it must read from the top level of data rather than from inside the observation, and the URL derivation — the code behind the twenty MUST 8.B.n requirements in PROTOCOL.md. It is the client half in a pure function with no socket in it, which is why tests/t_client.c drives the whole client protocol without opening one. An RL trainer is a client, not a server, so this is the half that matters most if you are writing one |
| 8c | tour/03_client.c | The client protocol, hand-written and counted — the other chair. 8b is the client codec with the library; this is the same nine messages written with src/wire/ and nothing from src/proto/, the way tour/01_minimal.c (step 2) is the server. Fenced into a protocol half and a socket half so make lines-tour counts them: 73 statements of client protocol stand behind the twenty 8.B MUSTs, and the socket half is the cost of no websockets library. It drives the genuine Python reference to done (interop.sh oracle 3c) and sends byte-for-byte the same client frames as examples/oe_agent (oracle 3d) |
| 9 | include/openenv/embed.h | Using the protocol without our main loop — and the seam the corpus replays through |
| 10 | src/server/session.c | One episode, one arena, five message types |
| 11 | src/server/http_stateless.c | The HTTP half, and why it validates differently |
| 12 | src/wire/limits.h | Every hard cap, with its reason. Read this one twice |
| 13 | src/wire/json_parse.c | RFC 8259 by recursive descent, and CPython's diagnostics |
| 13b | examples/oe_probe.c | What a WebSocket frame actually is — run it. Every bit of every header is decoded beside the bytes it came from: FIN, RSV, opcode, which of the three length forms, the mask key and the XOR rule, and the accept token verified against its own SHA-1. The gentler on-ramp to step 14, and the only place the transport is shown rather than described |
| 14 | src/wire/ws.c | RFC 6455. Last, or never — but after 13b it is survivable |
| 15 | src/server/conn.h | Five states, one fixed-capacity buffer, and why a parser never sees a partial message |
| 16 | src/server/serve.c | One poll() loop, and the wire-level reason it is not four processes |
examples/oe_agent.c is the other end of the same thing and needs no step. It
is what a consumer writes after make install: it includes nothing but
<openenv/openenv.h> and <openenv/embed.h>, and it drives an environment it
has never heard of using GET /schema alone. Run it against all three of
envs/ and watch it not care. examples/README.md is
the pair of them: oe_probe shows the wire and uses no library, oe_agent uses
the library and never sees the wire.
Two files answer "what do I write?", and they are not the same answer.
Step 4 (envs/echo.c) is what an environment author writes; step 13b
(examples/oe_probe.c) and oe_agent.c are what a consumer writes. The
per-directory READMEs — envs/,
tour/, examples/ — each open
with the one command that decides their directory's central claim.
Before you read anything
Run the suite in verbose mode:
V=1 make test
Every case name in tests/ is a sentence, and read end to end they are the
specification of the wire layer:
a_control_frame_may_sit_between_two_fragments,
nesting_is_capped_before_the_stack_is,
no_cut_changes_the_answer,
a_request_becomes_slices_of_the_caller_s_own_bytes. Four hundred and fifty-two
case names per pass, and it is the cheapest orientation available.
Then:
make lines
which prints the claim step 2 exists to make, as a measurement rather than a paragraph. And:
./build/rel/tests/t_replay
which pushes all one hundred and three captured reference cases — the corpus,
tests/fixtures/reference-corpus.json — through the four pure functions in
include/openenv/embed.h. No socket, no port, no Python, under a millisecond.
It prints, by name and with its reason, every case it did not replay.
Eighty-eight replay whole and one is split; the
other fourteen are deliberate divergences, each named on stdout with the field
that diverges and the Python type that produced it. Nothing is out of scope
any more — that bucket prints zero, which is a claim rather than an absence.
Two more instruments, when you want them
make fuzz-run # replay the fuzz corpus under ASan + UBSan, ~1 s
make autobahn # 517 RFC 6455 conformance cases, in docker, ~12 s
make check-embed # three third-party consumers, each really built and run
make docker # FROM scratch, then prove the container contract
The first pushes 631 seed inputs — 523 derived from the captured corpus plus
108 committed regression inputs — through six targets. The six are the entry
points of include/openenv/embed.h and the two layers below them, plus the
client codec and the tool vocabulary. The client codec is the only target that
reads bytes a remote SERVER chose; the tool vocabulary is the only one whose
input reaches an environment author's own code. And the targets check not for
crashes but for invariants: that no cut of a byte stream changes what the
WebSocket reader answers, that writing a parsed document twice produces the
same bytes, that an oe_json_dup copy survives its source arena being
released. The second is the specification's own test suite, run against
tests/autobahn/echo.c, which is src/wire/ws.c with a socket and nothing in
between.
docs/FUZZING.md is what both of them found. Two library bugs, both fixed and
both now carrying a unit test: an (int64_t) conversion of an out-of-range
double, which is undefined behaviour one POST /step away; and oe_json_dup
refusing, by one level, a document oe_json_parse accepts.
The other two are about leaving the tree. make check-embed builds three
strangers against this library: pkg-config after a staged make install, the
sources vendored and compiled with a flag set chosen to be hostile to us, and
<openenv/embed.h> alone. The third one's nm -u output is the whole
operating-system dependency of the OpenEnv protocol — fifteen libc functions
and not one system call. make docker builds the FROM scratch image and then
asserts the container contract on a running container. docs/PACKAGING.md is
both, with the transcripts.
The directory READMEs
cat src/README.md src/*/README.md envs/README.md tour/README.md \
examples/README.md tests/README.md
A sixty-second architectural tour. Each one opens with a claim you can check with one command, and each says what is deliberately not in its directory.
../arcade/README.md is the other side of the boundary:
game servers built on this protocol, cited by no test and by no corpus case,
there to be trained against. One oracle names one of them and skips when it is
absent, which is why no verdict here depends on the arcade being built.
The rest of the documentation
Three of these are generated from the source by make docs, and make check fails the build when they go stale — so the index cannot lie about the
code, because it is derived from it.
Start here if you are not reading code yet
PROTOCOL.md | our statement of the protocol. The sixteen routes, the five and four message types, the envelope, the seven error strings, the twenty-three numbered server requirements CONF-nn and MUST 8.A.n cite, and the twenty client requirements MUST 8.B.n cites |
NOT-THE-PROTOCOL.md | every refusal, with the reference's own observed bytes. Absences, out-of-scope surfaces, and eight deliberate divergences with their costs |
PROVENANCE.md | how the protocol was re-derived, the 103-case corpus, the fifteen oracles, and the statement that no upstream code is vendored |
The three generated documents
tools/gendocs.sh writes three files, not the five an earlier design named. The
CONF- index folded into WARTS.md and the SAFETY: index into
BOUNDARY.md. A fourth marker taxonomy did not need a fourth file, and an
index nobody opens is worse than a section somebody scrolls past. Both merges
are recorded in the generator's own banner.
WARTS.md | every WART-nn with its four lines verbatim, plus the CONF-nn citations — twenty-five of them, covering the twenty-three numbered server requirements. Generated |
GAPS.md | every GAP-nn with its three lines verbatim — the places where the protocol turns out not to exist. Generated, and separate from WARTS.md on purpose |
BOUNDARY.md | every [WIRE]/[SDK]/[POLICY] tag with its symbol and site, then every SAFETY: line. Two boundaries, one index. Generated |
The models, once each
MEMORY.md | the arena model, the one rule an author carries, and the three antidotes to the hazard it creates |
ERRORS.md | one flat enum, why none of its names ever reaches a peer, and the collapse onto what one does |
SECURITY.md | the threat model, every cap and why that number, and two limits that were wrong |
PORTING.md | the one file with platform conditionals in it, and what a port has to keep true |
What the instruments found
FUZZING.md | the six fuzz targets, their invariant oracles, and Autobahn case by case |
PACKAGING.md | install, pkg-config, vendoring, embedding, and the container |
evidence/OBSERVED-WIRE.md | captured traffic, byte for byte |
evidence/REFERENCE-NOTES.md | where each measured constant came from |
And ../CONVENTIONS.md ↗ is the contract the code is written
under — roughly half of it enforced by tools/check.sh rather than by review,
and the enforced half is marked.