ADR-0010 Architecture decision record

BDD acceptance tests via Cucumber.js (black-box, scenario-isolated)

jr2's per-package tests are node --test suites that run in-process and socket-free (e.g. the CLI's cli.test.ts calls main(argv, io) with fetch bound to app.request). Those are fast and cover unit and integration behavior, but nothing exercises the system the way a user does — the real jr2 binary, a real HTTP socket, exit codes, signals. This ADR adds a second, complementary tier: black-box end-to-end acceptance tests, written in Gherkin and run by Cucumber.js, living in a top-level ./features/ workspace package (@jr2/e2e).

What we decided

• Two tiers, not one. In-process node --test stays the unit/integration tier (fast, no sockets, per package). Cucumber.js is the e2e tier: each step spawns the real jr2 binary as a child process against a real orchestrator server on an ephemeral port, and asserts only on what a shell sees — captured stdout (the one machine-readable result), stderr (human activity), and the exit code. The e2e tier does not re-test what the in-process tier already covers.
• The per-scenario orchestrator is the image entrypoint, not a CLI verb. There is no host dev mode (ADR-0019), but the Orchestrator's server entrypoint is an ordinary Node process — the test fixture boots exactly that process on the host, with --url/token env pointing the spawned jr2 binary at it. Same real server the cluster runs, no cluster, no user-facing verb. The wire-compatible stub Harness (ADR-0011) is a fixture owned by this tier and served the same way.
• Feature → Rule → Scenario maps to criteria. A Rule is one acceptance criterion; a Scenario is an example of it. This is organizational only — Rules carry no shared state.
• Scenario is the isolation unit — and the environment is part of it. Each scenario gets its own scaffolded temp instance (via the real jr2 init), its own server process + sqlite store, torn down in an After hook. It also gets its own environment: the World spawns nothing from the ambient one, because a tier that inherits the developer's shell has made that shell a silent participant in every scenario. Two classes are sealed. Every JR2_* variable is scrubbed, since they say where to talk and as whom — an exported JR2_URL would point a whole suite at someone's deployed instance, and JR2_NAMESPACE reaches the entrypoint's data-plane switch (composesSandbox && namespace !== undefined), which would turn workspace.feature's "no cluster, so it faults pointedly" into a real Sandbox provisioned on whatever cluster was configured. Each tier adds back exactly what it means to set, and the variables a scenario's own config reads arrive through the instance's .env, a file the step wrote. KUBECONFIG is pinned to /dev/null wherever there is no cluster in the story, so a run-verb falling through to the kube path finds nothing rather than the last cluster a human used; @kind and @dist own a namespace on a real cluster and keep the real one. (The leak this closes was not theoretical: cli-contract.feature's exit-code scenario hung for its full 120s timeout against an unreachable home cluster, where it should fail in milliseconds — and it hung only on the machine that had one.) Nothing is shared across scenarios, so the suite is safe under cucumber-js --parallel.
• An opt-in @kind tier for the data plane (added once the workspace() slice landed — ADR-0012). Workspaces are always real Sandboxes, so the only way to test them is against a real cluster: the @kind scenarios drive the operator's Sandbox CR, a pod, the node's read-only Repo cache, an in-pod git worktree, and the Harness endpoint — faking only the LLM. (Originally the Sandbox ran a dev Harness image, a wire-compatible stub in a container with git; ADR-0038 retired it. The pod runs the STOCK Harness now — real pi, the real Menu through the Custodian, real Working tools — and the substitution moved to the provider: a scripted OpenAI-compatible endpoint the World serves from the host. Which is what makes this tier a second pi canary.) They are tagged @kind and excluded from the default profile, so the everyday suite needs no docker. Bring-up is the product's own path (ADR-0019): one shared VANILLA kind cluster and nothing else, then jr2 up per scenario into a fresh namespace — which since ADR-0038 builds and loads every image it deploys, so no image is pre-loaded by hand. Namespace-as-identity makes the scenario the isolation unit here too, so nothing is instance-bound to the cluster. The host address the pods dial back on is discovered, not assumed. The scripted provider runs host-side, so the tier needs the one address that routes from a pod back to this process — and it is not the same on every machine. On Linux the daemon IS the host, so the kind bridge's gateway is the answer. Under a VM-backed daemon (colima, Docker Desktop) that gateway is the VM: measured on colima, 172.18.0.1 reaches nothing while host.docker.internal, host.lima.internal and 192.168.5.2 all answer — and on Linux those names mean nothing. So the candidates are ordered by platform and the winner is VERIFIED, by curling the provider that is already listening from inside a kind node (a pod's egress routes through it), for the price of one docker exec memoized per worker process. Taking the bridge network's first IPAM entry is what broke: docker lists the IPv6 subnet first on some hosts and it carries no gateway at all, so the tier now reads every entry that HAS one, IPv4 first. Failing to find any address is one error naming every candidate tried, not an assertion in each scenario's Before hook. Measured: 22/22 on macOS/colima, ~3m20 warm at degree 4 against the ~1m35 Linux figure below — the VM is slower, not different. The tier runs parallel, and three properties are what make that safe. Scenario isolation (own namespace, own scripted model port); no shared mutable image name between concurrent converges of one checkout — a Sandbox Image builds straight to its content tag, with no intermediate, because the Harness arrives by volume at pod time (ADR-0037) — while ADR-0041 makes a warm scenario's converge build nothing (~13s fresh-namespace converge, every build disk-skipped); and connection retries at the two seats that dial a Sandbox before anything else ever has — ADR-0042 has the diagnosis (a Sandbox CR at phase: Ready is not yet ROUTABLE, so an unretried first dial is a lost turn; the Menu read was a second instance of the same defect). Parallelism does not create those windows, it only samples them more often — which is why any tier flake that vanishes at --parallel 1 should be read as a routability-class defect first, not as an isolation bug. Measured: degree 4 → 8 consecutive runs, 88/88 scenarios, 1m30–1m43s each on a sealed tree (serial: ~5m). The wired default is now --parallel 4 (in the kind profile, so a direct npx cucumber-js --profile kind gets it too); pass --parallel 1 to bisect a suspected isolation bug. Three tier changes came out of the hunt and stay. A FAILED @kind scenario dumps its evidence (the provider's request count, jr2 status, the Harness's ?view=history, every pod log with timestamps, events, EndpointSlices) to features/.tmp/kind-failures/ before its namespace is deleted. The tier's own workflow ROUTES agent.fault instead of ignoring it — an ignored terminal fault is an invisible one, and a tier that hangs where it could name the reason is a tier that costs a session per defect. And every scenario, passing or not, is now scraped for what ADR-0042's retries COST (jr2.routability seat=… attempts=… ms=… last=…), against a budget of 30s or 8 attempts judged per worker in AfterAll — two meters because a REJECT spends attempts and a dropped SYN spends time, and the tier measured one of each in its first two runs (2 attempts / 10667ms, then 3 attempts / 502ms / ECONNREFUSED). That last one is the tier watching the fix rather than merely enjoying it: the retry is absorbed by design, so without it a cluster drifting toward the 90s window would keep passing — slightly slower, silently — until the day it did not.
• An opt-in @console tier for the Console's UX (added with ADR-0032). The Console's risky behavior is interaction — token-mode switching, the frame-triggered gate inbox, selection vs. folding — which no reducer test sees and no CLI step drives. These scenarios hold a Playwright page inside ordinary Cucumber steps (Playwright the library, not the @playwright/test runner): the browser is a second driver of the same black box, reusing the per-scenario orchestrator fixture unchanged, and a scenario may drive both (the CLI starts a run; the browser sees it park). Tagged @console and excluded from the default profile exactly as @kind is — the everyday suite must not require a Chromium install — but they need no docker: a host-booted orchestrator serves the Console fine. Scope discipline: they assert what a user does and sees, never pixels or layout; what the page believes stays in console-store.test.ts.
• Harness conformance, in @jr2/harness, in the default gate (ADR-0027). The three tiers above all reach the Harness through the stub Harness, which is a hand-written model of the wire — it holds no conversation, so it can report a turn settled ended or aborted without ever having had a turn to settle. That is the right fixture for testing jr2, and it is blind by construction to what the real runtime keeps, drops, or sends onward. Those claims are covered by the conformance suite in packages/harness/test/, driven through the real turn loop — pi at the exact pin, the Menu over a real socket to the Custodian's address, a scripted provider — so a turn's shape (where it stalls, what it half-emits) is chosen by the test rather than by a model. The witness is the message array the provider receives on the next turn, which is the only place these claims are visible. Not a separate tier: it runs as part of pnpm -r test (no docker, no cluster), and it is the canary for pi bumps — run it before bumping the pin. (A predecessor tier that ran the retired foreign harness runtime at its pin dissolved into this suite — ADR-0027.)
• The Custodian suite, opt-in, against the real engine (ADR-0059). The Custodian's claims — the swap on the wire, the strip, the refusals, the dial guard, TLS verification upstream, SSE timing, abort, and a real SDK through HTTPS_PROXY — are Envoy's behaviour under jr2's configuration, so only the pinned Envoy image can prove them. just custodian-test runs node --test (packages/cli/test/custodian.docker.ts) against it under docker, on the host network with free ports, with local HTTPS upstreams under their own CA; about 23 seconds. Opt-in for the @kind reason — it needs docker — and NOT part of pnpm -r test. What the default gate keeps is the rendering: custodian.test.ts pins the bootstrap and script a pod's maker writes, socket-free. The @kind tier then proves the Custodian in a real pod, in both placements, against a model provider that refuses any request without the key.
• An opt-in @model tier, against a real model (ADR-0066). Every tier above fakes the model, so none can see a change in what a model DOES with what jr2 shows it — and ADR-0029/0062 measured that such changes are large. This tier is @kind with the last fake removed: the same cluster and bring-up, jr2 up per scenario into a fresh namespace, and behind the Custodian a real endpoint named by a profile (features/model-endpoints/<name>.json, selected by JR2_MODEL_ENDPOINT). Its scenarios are CLAIMS — an ADR's measured claim as N trials with a floor, scored into features/.tmp/model-scorecard.json against a committed baseline — and THE RUN, one workflow from a failing test to a commit verified in the pod. It runs the endpoint profile's workers scenarios at once, and workers × the profile's parallel runs must stay within what the endpoint serves and what the host holds. Not in CI and not in pnpm -r test: just e2e-model probes the endpoint and says skipped: when nothing answers, and whoever changes what the model reads runs it before calling the change done. A third pi canary.
• Shared cluster state is the one thing a scenario cannot own, and the pool is PROCESSES. --parallel runs cucumber workers as separate processes, so a fixture memoized "once per process" runs once per WORKER — four times, concurrently, against one cluster. The tier has exactly one such fixture: the git seed (steps/seed.ts) it serves in-cluster, which is cluster-scoped where everything else is namespaced per scenario. kubectl apply creates with GET-then-POST, so the workers that lose that race are told AlreadyExists — which an ensure treats as success, because the object being there is the whole ask. The rollout wait that follows is the real rendezvous, and it is idempotent for every worker. (The same property is why the @dist profile is serial: its publish would otherwise stand up one verdaccio per worker on a single port.)
• A workspace package, not a bare folder. ./features/ is @jr2/e2e so it owns its own xstate (+ cucumber) dependency: a scaffolded instance's workflows/*.ts import "xstate", resolved by walking up from the temp dir, and the repo root has no xstate. The package's node_modules satisfies it. It is top-level (not under packages/cli/) because an e2e test exercises most packages at once — the CLI is the common entry point, not the only conceivable one.
• Zero-build, like the rest of the repo. Cucumber.js v13 runs .ts ESM step definitions directly under Node 24's native type-stripping — no ts-node, no transpile step. (Verified before adopting it.)

Considered options

• Express BDD as a convention in node --test (a Given/When/Then helper, no .feature files): keeps zero deps, but loses the plain-English acceptance specs that are the point of writing them at the user boundary. Rejected — we want the .feature files as readable, reviewable criteria.
• In-process e2e (drive main(argv, io) instead of a subprocess): faster, but it is integration, not e2e — it can't catch bin/shebang, exit-code, signal, or attach bugs. That's already the job of the in-process tier; the e2e tier exists precisely to cover the subprocess boundary.
• Requiring a cluster for all e2e (every scenario through jr2 up): honest but slow, and it gates the everyday suite on docker. Rejected — the control-plane surface is fully exercisable against the bare server process; only the data plane needs kind, and it has its own tier.

Consequences

• @cucumber/cucumber is the first test-framework dependency in a repo that otherwise uses only node --test — a deliberate deviation, scoped to @jr2/e2e (it does not leak into the kit packages). playwright joins it there, same scoping, browser binaries installed only where @console runs (CI, or npx playwright install chromium locally).
• The suite is run with pnpm --filter @jr2/e2e test:e2e; @jr2/e2e exposes no test script, so pnpm -r test stays fast and unit-only.
• Human-in-the-loop delivery is covered at the gates API (features/agent-and-gates.feature — the agent played over /agents/<iid>/*, the human over /runs/:id/gates/:gate/events); the jr2 send --gate CLI verb is covered at the in-process tier. A CLI-driven e2e gate scenario remains a candidate follow-up Rule.