ADR-0022 Architecture decision record

Observation is a level-triggered feed, and an API before it is a page

The visualizer never updated (2026-07-19). Not slowly — structurally, in three independent ways, and each one looked healthy while it failed.

New runs could not appear. The page's run list came from a one-shot GET /workflows/:name/runs at boot, with a ↻ button beside it. There was no push channel to subscribe to: RunHost held per-run listener sets, and start() notified nobody, because nobody existed to notify. A run started after page load was unreachable by any means short of a reload. The ↻ button was not a convenience; it was the workaround.

A dropped stream was permanently and invisibly fatal. The page set feed.onerror = () => feed.close(), defeating EventSource's own reconnect. onerror cannot distinguish a clean terminal close from a blip, so the handler had to assume the harmless case and got the other one — after which the diagram sat there, wearing its last highlight, saying nothing.

Nothing wrote when nothing happened. Every feed wrote only on transitions. A run parked on a gate for hours wrote zero bytes, so any idle intermediary dropped the connection. This is the same gap that surfaced as error: terminated on an attached jr2 run — undici's message for a fetch() body that died mid-read, which reads like a run failure and was not one. The run was fine the whole time.

The common cause is that observation had been built as a feature of a page rather than as a surface. A page feature can get away with fetching once, because a person can press a button; an API cannot. So the fix is not a page patch.

Decision

• The observation API is real-time, and the page is its first consumer — not its purpose. An external dashboard, a CI annotator, or a chat bot should be able to track a workflow without running our HTML. Every choice below is weighed against that reader rather than against ours.
• GET /workflows/:name/events (open band) — one connection carrying a whole workflow: every run of it appearing, moving, emitting and leaving, for as long as the caller stays attached. It outlives every run on it.
• Level-triggered, not event-sourced. status always carries a whole RunObservation, never a patch, and the opening runs frame carries the entire live set. A reconnecting client therefore converges with no replay buffer, no Last-Event-ID, and no per-client state on the server; re-delivery of any frame is idempotent by construction, so there is no gap to detect and no cursor to resume from. Same reconciliation idiom as the Lease (ADR-0021) and jr2 up (ADR-0019): say the whole truth, repeatedly, and let the reader converge.
• Two granularities, one vocabulary. The per-run feed stays — tracking exactly one run is a legitimate thing to want, and it is what jr2 run and jr2 logs -f already do. Every run-scoped frame carries runId even where the route makes it redundant, so one parser serves both.
• gone is a fact, not an inference. A run can leave the live set without a terminal status: stop() deliberately leaves the stored status "live" so a later restore() picks the run back up. A client watching for status === "done" would show a stopped run as live forever, so departure is stated outright.
• The workflow's listener set belongs to the host, keyed by name — not to any LiveRun. persist() clears a settled run's own listener set, and a workflow's watcher must survive every run it watches. Placing the set on the host makes that structurally true instead of a comment asking the next reader not to clear the wrong one.
• A quiet feed pings: an SSE comment frame every 15s, on all three feeds. It is deliberately not Lease vocabulary — GLOSSARY.md puts heartbeat and keepalive on that entry's Avoid list, and this asserts nothing and expects no answer. The write doubles as the liveness probe: a failed write is how a silently-dead peer is discovered, which is why it is wired to the handler's exit.
• A settled run is the client's own memory. A run appears in the page's settled list iff this page watched it leave. After a hard reload they are gone, and that is correct rather than a defect: serving a finished run's state to an unauthenticated reader is a read-through, which stays behind the Instance token on /runs/:id (ADR-0014).
• jr2 visualize is deleted. Its job was to find an orchestrator, forward a port, and open a browser. With the page now useful on its own, that is one kubectl port-forward the reader already knows how to run — and the command's ephemeral-fallback path was quietly forwarding to a different instance than the deployed one. Its one unique piece of value, the opaque-states warning, moves onto the page as MachineDoc.opaqueStates, where the person looking at the diagram can read it.
• A frame is rendered once, and a slow reader is closed, not queued. Every subscriber of a feed receives the same wire string, rendered once per event, never once per subscriber. A subscriber has a bounded backlog of unread writes, judged at each ping: one that still leaves the bound's worth of writes unread a whole ping interval after they were sent has fallen behind, and is closed. Lag, not burst size — one transition's Emits reach every subscriber in the same tick, and a reader that keeps up drains them before any ping judges it. Closing is safe because the feed is level-triggered: the client reconnects and receives the whole current truth, having missed no state — only activity (Emits, retries) that fell in the gap. A page reconnects with EventSource's own retry; the CLI (jr2 run, jr2 logs -f) re-attaches any feed that ends before a terminal status, and never reads such an end as the run's — except gone: a run stopped before it settled (parked, ADR-0025) reads through as active, so its one-shot feed says gone after the status, and the CLI ends there instead of re-attaching in a loop. The server keeps no per-client queue and no slow-client accounting beyond counting the subscriber's writes (scaling review R10, 2026-09-30).
• Shutdown ends feeds explicitly. RunHost.close() delivers closed to every listener. server.close() waits for in-flight requests and an observation feed has no end of its own, so without this a single attached watcher wedges jr2 dev shutdown indefinitely.

The new route stays inside the band ADR-0014 drew: no context, no instanceId and no fault at any depth, emit type only, scoped to one workflow the caller can already name, and no store read-through.

Considered options

• Poll faster. The ↻ button on an interval. Rejected because it is wrong at both ends: too slow to watch a run by, and pure waste against an instance where nothing is happening — which is most of the time. It also leaves the onerror and idle-timeout failures untouched, since those are not staleness.
• A patch/delta feed with a replay buffer. More efficient on the wire, and genuinely wrong here: it makes every client stateful, forces a Last-Event-ID cursor and server-side retention, and turns a dropped frame into permanent divergence with no way to notice. A whole observation is a few hundred bytes on transitions a human is watching. We are nowhere near the volume that would justify buying complexity with correctness.
• Fold the per-run feed into the workflow feed. One route is simpler. Rejected: a consumer tracking one run would have to receive and discard every sibling's traffic, and jr2 run — the most common consumer — is exactly that case.
• Give the page a WebSocket. Bidirectional, and nothing here needs a back-channel; control is on the guarded surface, deliberately. SSE reconnects on its own, survives proxies, and needs no client library.
• Keep jr2 visualize, deprecated. Rejected as the worse of the two: a command whose whole remaining value is opening a browser, still carrying the port-forward path that could attach to the wrong instance.

Consequences

• Kill-detection latency is untouched and remains the ~5-minute Lease poll (ADR-0021). The feed reports what the Orchestrator believes; a workspace lost 4 minutes ago is still believed live, and now says so in real time. Lowering that interval, or watching Sandbox CRs, is a separate decision.
• A body machine cannot ignore workspace.lost: workspace() refuses a body with no route for it, and an event that no transition takes faults the run (ADR-0074). The feed reports the machine; it does not second-guess it.
• The page has no reload button and no reload path for settled runs — by design, per the memory rule above.
• viz/store.js exists so the client's logic is a pure reducer testable under node:test. The repo still has no jsdom and no browser driver, and this ADR is not an argument for adding one: the renderer is a pure function of (doc, status), and everything else the page believes now lives in a function that takes a store and a frame.