miniopenenv

docs/TOUR.md → html

The tour

This page is the guided reading order for the repository: which files to read, in what order, and what question each one answers. The dependency graph is the compiler's order. This document is yours.

Sixteen numbered steps — twenty-two stops, counting 6b, 7b, 7c, 8b, 8c and 13b — about an hour. Steps 1–7 are the complete answer to "what is OpenEnv"; everything after them is how a server that has to survive strangers is built underneath it.

Every step below exists in the tree today.

The path is also IN the tree, not only in this file. Every source file below carries its step in its opening line, so an editor is enough:

grep -rn 'READ:' src include envs tour examples | sort -t: -k3

tools/check.sh requires the set of READ:n markers in the source and the set of numbered rows in the table below to be equal. So neither can be edited without the other. Step 1 is this repository's README and is the one row with no file to mark.

#ReadWhat it answers
1README.mdWhat this is, and the two-column line count
2tour/01_minimal.c (and its cited Python twin)The protocol, entire. 64 statements of C beside 78 of Python. make lines prints the split
3include/openenv/openenv.hThe public API, three tags, no ceremony
4envs/echo.cWhat an environment author actually writes — 45 statements, no allocator, no error branch
5envs/counter.c... with state, and a done condition. A literal port of interop/probe_env.py, which is the environment the corpus was captured against. tests/counter_env.h is the same table with no main, because the replay links no socket
6src/proto/types.hThe message vocabulary, named once — five message types on one socket
6bsrc/proto/routes.h... and the other half of it: sixteen routes, as a table
7src/proto/encode.cThe two encoders, and the diff between them. envs/rewards.c is that diff as a program: one step, 1.0 on HTTP and true on the WebSocket, and its banner is the shell recipe
7bsrc/proto/errors.cWhat a refusal looks like, and the url rule that is not what it looks like
7csrc/proto/validate.cLax coercion, and why a strict validator fails the replay
8tour/02_conformant.cThe same environment, written with the library. 77 statements against 01's 254, no socket half at all, sixteen routes instead of two — and the same five server frames byte for byte. Its banner carries the diff that proves it
8bsrc/proto/client_codec.cThe other end of the same five messages. What a client may send, what it must read from the top level of data rather than from inside the observation, and the URL derivation — the code behind the twenty MUST 8.B.n requirements in PROTOCOL.md. It is the client half in a pure function with no socket in it, which is why tests/t_client.c drives the whole client protocol without opening one. An RL trainer is a client, not a server, so this is the half that matters most if you are writing one
8ctour/03_client.cThe client protocol, hand-written and counted — the other chair. 8b is the client codec with the library; this is the same nine messages written with src/wire/ and nothing from src/proto/, the way tour/01_minimal.c (step 2) is the server. Fenced into a protocol half and a socket half so make lines-tour counts them: 73 statements of client protocol stand behind the twenty 8.B MUSTs, and the socket half is the cost of no websockets library. It drives the genuine Python reference to done (interop.sh oracle 3c) and sends byte-for-byte the same client frames as examples/oe_agent (oracle 3d)
9include/openenv/embed.hUsing the protocol without our main loop — and the seam the corpus replays through
10src/server/session.cOne episode, one arena, five message types
11src/server/http_stateless.cThe HTTP half, and why it validates differently
12src/wire/limits.hEvery hard cap, with its reason. Read this one twice
13src/wire/json_parse.cRFC 8259 by recursive descent, and CPython's diagnostics
13bexamples/oe_probe.cWhat a WebSocket frame actually is — run it. Every bit of every header is decoded beside the bytes it came from: FIN, RSV, opcode, which of the three length forms, the mask key and the XOR rule, and the accept token verified against its own SHA-1. The gentler on-ramp to step 14, and the only place the transport is shown rather than described
14src/wire/ws.cRFC 6455. Last, or never — but after 13b it is survivable
15src/server/conn.hFive states, one fixed-capacity buffer, and why a parser never sees a partial message
16src/server/serve.cOne poll() loop, and the wire-level reason it is not four processes

examples/oe_agent.c is the other end of the same thing and needs no step. It is what a consumer writes after make install: it includes nothing but <openenv/openenv.h> and <openenv/embed.h>, and it drives an environment it has never heard of using GET /schema alone. Run it against all three of envs/ and watch it not care. examples/README.md is the pair of them: oe_probe shows the wire and uses no library, oe_agent uses the library and never sees the wire.

Two files answer "what do I write?", and they are not the same answer. Step 4 (envs/echo.c) is what an environment author writes; step 13b (examples/oe_probe.c) and oe_agent.c are what a consumer writes. The per-directory READMEs — envs/, tour/, examples/ — each open with the one command that decides their directory's central claim.

§ 1 · Before you read anything

Before you read anything

Run the suite in verbose mode:

V=1 make test

Every case name in tests/ is a sentence, and read end to end they are the specification of the wire layer: a_control_frame_may_sit_between_two_fragments, nesting_is_capped_before_the_stack_is, no_cut_changes_the_answer, a_request_becomes_slices_of_the_caller_s_own_bytes. Four hundred and fifty-two case names per pass, and it is the cheapest orientation available.

Then:

make lines

which prints the claim step 2 exists to make, as a measurement rather than a paragraph. And:

./build/rel/tests/t_replay

which pushes all one hundred and three captured reference cases — the corpus, tests/fixtures/reference-corpus.json — through the four pure functions in include/openenv/embed.h. No socket, no port, no Python, under a millisecond. It prints, by name and with its reason, every case it did not replay. Eighty-eight replay whole and one is split; the other fourteen are deliberate divergences, each named on stdout with the field that diverges and the Python type that produced it. Nothing is out of scope any more — that bucket prints zero, which is a claim rather than an absence.

§ 2 · Two more instruments, when you want them

Two more instruments, when you want them

make fuzz-run              # replay the fuzz corpus under ASan + UBSan, ~1 s
make autobahn              # 517 RFC 6455 conformance cases, in docker, ~12 s
make check-embed           # three third-party consumers, each really built and run
make docker                # FROM scratch, then prove the container contract

The first pushes 631 seed inputs — 523 derived from the captured corpus plus 108 committed regression inputs — through six targets. The six are the entry points of include/openenv/embed.h and the two layers below them, plus the client codec and the tool vocabulary. The client codec is the only target that reads bytes a remote SERVER chose; the tool vocabulary is the only one whose input reaches an environment author's own code. And the targets check not for crashes but for invariants: that no cut of a byte stream changes what the WebSocket reader answers, that writing a parsed document twice produces the same bytes, that an oe_json_dup copy survives its source arena being released. The second is the specification's own test suite, run against tests/autobahn/echo.c, which is src/wire/ws.c with a socket and nothing in between.

docs/FUZZING.md is what both of them found. Two library bugs, both fixed and both now carrying a unit test: an (int64_t) conversion of an out-of-range double, which is undefined behaviour one POST /step away; and oe_json_dup refusing, by one level, a document oe_json_parse accepts.

The other two are about leaving the tree. make check-embed builds three strangers against this library: pkg-config after a staged make install, the sources vendored and compiled with a flag set chosen to be hostile to us, and <openenv/embed.h> alone. The third one's nm -u output is the whole operating-system dependency of the OpenEnv protocol — fifteen libc functions and not one system call. make docker builds the FROM scratch image and then asserts the container contract on a running container. docs/PACKAGING.md is both, with the transcripts.

§ 3 · The directory READMEs

The directory READMEs

cat src/README.md src/*/README.md envs/README.md tour/README.md \
    examples/README.md tests/README.md

A sixty-second architectural tour. Each one opens with a claim you can check with one command, and each says what is deliberately not in its directory.

../arcade/README.md is the other side of the boundary: game servers built on this protocol, cited by no test and by no corpus case, there to be trained against. One oracle names one of them and skips when it is absent, which is why no verdict here depends on the arcade being built.

§ 4 · The rest of the documentation

The rest of the documentation

Three of these are generated from the source by make docs, and make check fails the build when they go stale — so the index cannot lie about the code, because it is derived from it.

Start here if you are not reading code yet

PROTOCOL.mdour statement of the protocol. The sixteen routes, the five and four message types, the envelope, the seven error strings, the twenty-three numbered server requirements CONF-nn and MUST 8.A.n cite, and the twenty client requirements MUST 8.B.n cites
NOT-THE-PROTOCOL.mdevery refusal, with the reference's own observed bytes. Absences, out-of-scope surfaces, and eight deliberate divergences with their costs
PROVENANCE.mdhow the protocol was re-derived, the 103-case corpus, the fifteen oracles, and the statement that no upstream code is vendored

The three generated documents

tools/gendocs.sh writes three files, not the five an earlier design named. The CONF- index folded into WARTS.md and the SAFETY: index into BOUNDARY.md. A fourth marker taxonomy did not need a fourth file, and an index nobody opens is worse than a section somebody scrolls past. Both merges are recorded in the generator's own banner.

WARTS.mdevery WART-nn with its four lines verbatim, plus the CONF-nn citations — twenty-five of them, covering the twenty-three numbered server requirements. Generated
GAPS.mdevery GAP-nn with its three lines verbatim — the places where the protocol turns out not to exist. Generated, and separate from WARTS.md on purpose
BOUNDARY.mdevery [WIRE]/[SDK]/[POLICY] tag with its symbol and site, then every SAFETY: line. Two boundaries, one index. Generated

The models, once each

MEMORY.mdthe arena model, the one rule an author carries, and the three antidotes to the hazard it creates
ERRORS.mdone flat enum, why none of its names ever reaches a peer, and the collapse onto what one does
SECURITY.mdthe threat model, every cap and why that number, and two limits that were wrong
PORTING.mdthe one file with platform conditionals in it, and what a port has to keep true

What the instruments found

FUZZING.mdthe six fuzz targets, their invariant oracles, and Autobahn case by case
PACKAGING.mdinstall, pkg-config, vendoring, embedding, and the container
evidence/OBSERVED-WIRE.mdcaptured traffic, byte for byte
evidence/REFERENCE-NOTES.mdwhere each measured constant came from

And ../CONVENTIONS.md ↗ is the contract the code is written under — roughly half of it enforced by tools/check.sh rather than by review, and the enforced half is marked.