# BDD acceptance tests via Cucumber.js (black-box, scenario-isolated) jr2's per-package tests are `node --test` suites that run in-process and socket-free (e.g. the CLI's `cli.test.ts` calls `main(argv, io)` with `fetch` bound to `app.request`). Those are fast and cover unit and integration behavior, but nothing exercises the _system_ the way a user does — the real `jr2` binary, a real HTTP socket, exit codes, signals. This ADR adds a second, complementary tier: **black-box end-to-end acceptance tests, written in Gherkin and run by Cucumber.js**, living in a top-level `./features/` workspace package (`@jr2/e2e`). ## What we decided - **Two tiers, not one.** In-process `node --test` stays the unit/integration tier (fast, no sockets, per package). Cucumber.js is the **e2e** tier: each step spawns the real `jr2` binary as a child process against a real orchestrator server on an ephemeral port, and asserts only on what a shell sees — captured stdout (the one machine-readable result), stderr (human activity), and the exit code. The e2e tier does not re-test what the in-process tier already covers. - **The per-scenario orchestrator is the image entrypoint, not a CLI verb.** There is no host dev mode (ADR-0019), but the Orchestrator's server entrypoint is an ordinary Node process — the test fixture boots exactly that process on the host, with `--url`/token env pointing the spawned `jr2` binary at it. Same real server the cluster runs, no cluster, no user-facing verb. The wire-compatible **stub Harness** (ADR-0011) is a fixture owned by this tier and served the same way. - **`Feature` → `Rule` → `Scenario` maps to criteria.** A `Rule` is one acceptance criterion; a `Scenario` is an example of it. This is organizational only — `Rule`s carry no shared state. - **`Scenario` is the isolation unit — and the environment is part of it.** Each scenario gets its own scaffolded temp instance (via the real `jr2 init`), its own server process + sqlite store, torn down in an `After` hook. It also gets its own **environment**: the World spawns nothing from the ambient one, because a tier that inherits the developer's shell has made that shell a silent participant in every scenario. Two classes are sealed. Every `JR2_*` variable is scrubbed, since they say where to talk and as whom — an exported `JR2_URL` would point a whole suite at someone's deployed instance, and `JR2_NAMESPACE` reaches the entrypoint's data-plane switch (`composesSandbox && namespace !== undefined`), which would turn `workspace.feature`'s "no cluster, so it faults pointedly" into a real Sandbox provisioned on whatever cluster was configured. Each tier adds back exactly what it means to set, and the variables a scenario's own config reads arrive through the instance's `.env`, a file the step wrote. `KUBECONFIG` is pinned to `/dev/null` wherever there is no cluster in the story, so a run-verb falling through to the kube path finds nothing rather than the last cluster a human used; `@kind` and `@dist` own a namespace on a real cluster and keep the real one. (The leak this closes was not theoretical: `cli-contract.feature`'s exit-code scenario hung for its full 120s timeout against an unreachable home cluster, where it should fail in milliseconds — and it hung only on the machine that had one.) Nothing is shared across scenarios, so the suite is safe under `cucumber-js --parallel`. - **An opt-in `@kind` tier for the data plane** (added once the `workspace()` slice landed — ADR-0012). Workspaces are always real Sandboxes, so the only way to test them is against a real cluster: the `@kind` scenarios drive the operator's Sandbox CR, a pod, the node's read-only Repo cache, an in-pod git worktree, and the Harness endpoint — faking only the LLM. (Originally the Sandbox ran a **dev Harness image**, a wire-compatible stub in a container with `git`; [ADR-0038](0038-jr2-up-builds-every-image-it-deploys.md) retired it. The pod runs the STOCK Harness now — real pi, the real Menu through the Custodian, real Working tools — and the substitution moved to the provider: a scripted OpenAI-compatible endpoint the World serves from the host. Which is what makes this tier a second pi canary.) They are tagged `@kind` and **excluded from the default profile**, so the everyday suite needs no docker. Bring-up is the product's own path (ADR-0019): one shared VANILLA kind cluster and nothing else, then **`jr2 up` per scenario into a fresh namespace** — which since ADR-0038 builds and loads every image it deploys, so no image is pre-loaded by hand. Namespace-as-identity makes the scenario the isolation unit here too, so nothing is instance-bound to the cluster. **The host address the pods dial back on is discovered, not assumed.** The scripted provider runs host-side, so the tier needs the one address that routes from a pod back to this process — and it is not the same on every machine. On Linux the daemon IS the host, so the kind bridge's gateway is the answer. Under a VM-backed daemon (colima, Docker Desktop) that gateway is the VM: measured on colima, `172.18.0.1` reaches nothing while `host.docker.internal`, `host.lima.internal` and `192.168.5.2` all answer — and on Linux those names mean nothing. So the candidates are ordered by platform and the winner is VERIFIED, by `curl`ing the provider that is already listening from inside a kind node (a pod's egress routes through it), for the price of one `docker exec` memoized per worker process. Taking the bridge network's first IPAM entry is what broke: docker lists the IPv6 subnet first on some hosts and it carries no gateway at all, so the tier now reads every entry that HAS one, IPv4 first. Failing to find any address is one error naming every candidate tried, not an assertion in each scenario's `Before` hook. Measured: **22/22 on macOS/colima, ~3m20 warm at degree 4** against the ~1m35 Linux figure below — the VM is slower, not different. **The tier runs parallel, and three properties are what make that safe.** Scenario isolation (own namespace, own scripted model port); no shared mutable image name between concurrent converges of one checkout — a Sandbox Image builds straight to its content tag, with no intermediate, because the Harness arrives by volume at pod time ([ADR-0037](0037-an-instance-builds-its-sandbox-images-jr2-injects-the-harness.md)) — while [ADR-0041](0041-a-build-the-host-already-holds-is-not-spent-again.md) makes a warm scenario's converge build nothing (~13s fresh-namespace converge, every build disk-skipped); and connection retries at the two seats that dial a Sandbox before anything else ever has — [ADR-0042](0042-ready-is-not-routable-an-admission-retries-its-connection.md) has the diagnosis (a Sandbox CR at `phase: Ready` is not yet ROUTABLE, so an unretried first dial is a lost turn; the Menu read was a second instance of the same defect). Parallelism does not create those windows, it only samples them more often — which is why any tier flake that vanishes at `--parallel 1` should be read as a routability-class defect first, not as an isolation bug. Measured: **degree 4 → 8 consecutive runs, 88/88 scenarios, 1m30–1m43s each** on a sealed tree (serial: ~5m). **The wired default is now `--parallel 4`** (in the `kind` profile, so a direct `npx cucumber-js --profile kind` gets it too); pass `--parallel 1` to bisect a suspected isolation bug. Three tier changes came out of the hunt and stay. A FAILED `@kind` scenario dumps its evidence (the provider's request count, `jr2 status`, the Harness's `?view=history`, every pod log with timestamps, events, EndpointSlices) to `features/.tmp/kind-failures/` before its namespace is deleted. The tier's own workflow ROUTES `agent.fault` instead of ignoring it — an ignored terminal fault is an invisible one, and a tier that hangs where it could name the reason is a tier that costs a session per defect. And **every** scenario, passing or not, is now scraped for what ADR-0042's retries COST (`jr2.routability seat=… attempts=… ms=… last=…`), against a budget of 30s or 8 attempts judged per worker in `AfterAll` — two meters because a REJECT spends attempts and a dropped SYN spends time, and the tier measured one of each in its first two runs (2 attempts / 10667ms, then 3 attempts / 502ms / `ECONNREFUSED`). That last one is the tier watching the fix rather than merely enjoying it: the retry is absorbed by design, so without it a cluster drifting toward the 90s window would keep passing — slightly slower, silently — until the day it did not. - **An opt-in `@console` tier for the Console's UX** (added with [ADR-0032](0032-the-console-unlocks-with-the-instance-token.md)). The Console's risky behavior is interaction — token-mode switching, the frame-triggered gate inbox, selection vs. folding — which no reducer test sees and no CLI step drives. These scenarios hold a **Playwright page inside ordinary Cucumber steps** (Playwright the _library_, not the `@playwright/test` runner): the browser is a second driver of the same black box, reusing the per-scenario orchestrator fixture unchanged, and a scenario may drive both (the CLI starts a run; the browser sees it park). Tagged `@console` and **excluded from the default profile** exactly as `@kind` is — the everyday suite must not require a Chromium install — but they need no docker: a host-booted orchestrator serves the Console fine. Scope discipline: they assert what a user does and sees, never pixels or layout; what the page _believes_ stays in `console-store.test.ts`. - **Harness conformance, in `@jr2/harness`, in the default gate** ([ADR-0027](0027-the-harness-is-jr2s-own-server-flue-retires-the-wire-stays.md)). The three tiers above all reach the Harness through the **stub Harness**, which is a hand-written model of the _wire_ — it holds no conversation, so it can report a turn settled `ended` or `aborted` without ever having had a turn to settle. That is the right fixture for testing jr2, and it is blind by construction to what the real runtime keeps, drops, or sends onward. Those claims are covered by the conformance suite in `packages/harness/test/`, driven through the real turn loop — pi at the exact pin, the Menu over a real socket to the Custodian's address, a **scripted provider** — so a turn's shape (where it stalls, what it half-emits) is chosen by the test rather than by a model. The witness is the message array the provider receives on the _next_ turn, which is the only place these claims are visible. Not a separate tier: it runs as part of `pnpm -r test` (no docker, no cluster), and it is the canary for pi bumps — **run it before bumping the pin**. (A predecessor tier that ran the retired foreign harness runtime at its pin dissolved into this suite — ADR-0027.) - **The Custodian suite, opt-in, against the real engine** ([ADR-0059](0059-a-harness-holds-stand-ins-and-the-custodian-holds-the-keys.md)). The Custodian's claims — the swap on the wire, the strip, the refusals, the dial guard, TLS verification upstream, SSE timing, abort, and a real SDK through `HTTPS_PROXY` — are Envoy's behaviour under jr2's configuration, so only the pinned Envoy image can prove them. `just custodian-test` runs `node --test` (`packages/cli/test/custodian.docker.ts`) against it under docker, on the host network with free ports, with local HTTPS upstreams under their own CA; about 23 seconds. Opt-in for the `@kind` reason — it needs docker — and NOT part of `pnpm -r test`. What the default gate keeps is the rendering: `custodian.test.ts` pins the bootstrap and script a pod's maker writes, socket-free. The `@kind` tier then proves the Custodian in a real pod, in both placements, against a model provider that refuses any request without the key. - **An opt-in `@model` tier, against a real model** ([ADR-0066](0066-a-model-tier-runs-a-real-model-and-a-change-to-what-it-reads-runs-the-tier.md)). Every tier above fakes the model, so none can see a change in what a model DOES with what jr2 shows it — and ADR-0029/0062 measured that such changes are large. This tier is `@kind` with the last fake removed: the same cluster and bring-up, `jr2 up` per scenario into a fresh namespace, and behind the Custodian a real endpoint named by a profile (`features/model-endpoints/.json`, selected by `JR2_MODEL_ENDPOINT`). Its scenarios are CLAIMS — an ADR's measured claim as N trials with a floor, scored into `features/.tmp/model-scorecard.json` against a committed baseline — and THE RUN, one workflow from a failing test to a commit verified in the pod. It runs the endpoint profile's `workers` scenarios at once, and `workers` × the profile's `parallel` runs must stay within what the endpoint serves and what the host holds. Not in CI and not in `pnpm -r test`: `just e2e-model` probes the endpoint and says `skipped:` when nothing answers, and whoever changes what the model reads runs it before calling the change done. A third pi canary. - **Shared cluster state is the one thing a scenario cannot own, and the pool is PROCESSES.** `--parallel` runs cucumber workers as separate processes, so a fixture memoized "once per process" runs once per WORKER — four times, concurrently, against one cluster. The tier has exactly one such fixture: the git seed (`steps/seed.ts`) it serves in-cluster, which is cluster-scoped where everything else is namespaced per scenario. `kubectl apply` creates with GET-then-POST, so the workers that lose that race are told `AlreadyExists` — which an `ensure` treats as success, because the object being there is the whole ask. The rollout wait that follows is the real rendezvous, and it is idempotent for every worker. (The same property is why the `@dist` profile is serial: its publish would otherwise stand up one verdaccio per worker on a single port.) - **A workspace package, not a bare folder.** `./features/` is `@jr2/e2e` so it owns its own `xstate` (+ cucumber) dependency: a scaffolded instance's `workflows/*.ts` `import "xstate"`, resolved by walking up from the temp dir, and the repo root has no `xstate`. The package's `node_modules` satisfies it. It is top-level (not under `packages/cli/`) because an e2e test exercises most packages at once — the CLI is the common entry point, not the only conceivable one. - **Zero-build, like the rest of the repo.** Cucumber.js v13 runs `.ts` ESM step definitions directly under Node 24's native type-stripping — no `ts-node`, no transpile step. (Verified before adopting it.) ## Considered options - **Express BDD as a convention in `node --test`** (a Given/When/Then helper, no `.feature` files): keeps zero deps, but loses the plain-English acceptance specs that are the point of writing them at the user boundary. Rejected — we want the `.feature` files as readable, reviewable criteria. - **In-process e2e** (drive `main(argv, io)` instead of a subprocess): faster, but it is integration, not e2e — it can't catch bin/shebang, exit-code, signal, or attach bugs. That's already the job of the in-process tier; the e2e tier exists precisely to cover the subprocess boundary. - **Requiring a cluster for all e2e** (every scenario through `jr2 up`): honest but slow, and it gates the everyday suite on docker. Rejected — the control-plane surface is fully exercisable against the bare server process; only the data plane needs kind, and it has its own tier. ## Consequences - `@cucumber/cucumber` is the **first test-framework dependency** in a repo that otherwise uses only `node --test` — a deliberate deviation, scoped to `@jr2/e2e` (it does not leak into the kit packages). `playwright` joins it there, same scoping, browser binaries installed only where `@console` runs (CI, or `npx playwright install chromium` locally). - The suite is run with `pnpm --filter @jr2/e2e test:e2e`; `@jr2/e2e` exposes **no** `test` script, so `pnpm -r test` stays fast and unit-only. - Human-in-the-loop delivery is covered at the gates API (`features/agent-and-gates.feature` — the agent played over `/agents//*`, the human over `/runs/:id/gates/:gate/events`); the `jr2 send --gate` CLI verb is covered at the in-process tier. A CLI-driven e2e gate scenario remains a candidate follow-up `Rule`.