ADR-0007 Architecture decision record

Durable Machine state: serializable context, live things constructed from input

The Orchestrator is a forever-running daemon that must survive a kill -9 and resume in-flight work, so every Machine has to be reconstructable from a persisted xstate snapshot. That forces one invariant across everything that runs in a Machine:

Machine context and actor input hold plain serializable data only — ids, counters, status, data-shaped handles (e.g. a Sandbox's endpoint string). Live infrastructure — HTTP servers, clients, live actor references — is constructed per-invocation from serializable input (ADR-0011's static-import doctrine) and never persisted. Parent↔child coordination uses sendParent events plus serializable ids, never a held child-actor reference.

PoC #7 discovered this the hard way (a live http.Server in context crashed JSON.stringify); PoC #8 was designed to it from the start (the top Machine keeps only counts + ids, children report completion UP via sendParent('FEATURE_DONE')). They are the same invariant — #8's discipline is precisely what makes #7's snapshot/restore possible.

The store's own durability — what a persist survives, and how long a finished run's row lives — is ADR-0065.

Durable agent handles do not ride Machine context at all: the host persists an iid → admission ledger beside the snapshot in the same save (ADR-0016). Iids are globally unique, so restore needs no context-tree walking to find them. An admission names the Turn ID of the Turn that ledgered it, because the continued Turns of one conversation share an iid. A restore re-attaches only that Turn (ADR-0016).

Effects keep a ledger beside the snapshot in the same save (ADR-0067): one row per Effect, by its id. An intent row is written before the call — saved at the write, so the snapshot that holds the Effect and the row land together, before the request leaves; a settled row is written with the output, saved at the write before the transition the output drives and again in the save that carries that transition; an aborted row when the Effect's state exited while it ran, kept in memory and saved by the save that carries that exit (a cancel's final save included), never by a save inside the transition. A restore reads the row and never guesses: settled → the output is delivered (the transition's save was lost); intent → per kind: a command that names no secret re-attaches to its exec by id, or settles unknown when the Harness no longer knows it; an http call (a Worktree file body included) asks every replica of its Effector for its id and follows the one that holds it, and when none does it is sent again only when the author marked it idempotent, else it settles unknown; a push and a credentialed command ask their Sandbox Effector for their id, and when it no longer holds them are sent again with only the commit the intent row pinned — never the Agent's tip now — or settle unknown when no commit was pinned; no row → nothing was sent, so it starts. jr2 runs an Effect it started a second time only where its author marked it idempotent, or where its run is a function of a pinned commit (a push, a credentialed command); a command that names no secret never. A row replaces the one before it and is never dropped: a restored child that is done or stopped never starts, so no restore reads a finished Effect's row, and the settled and aborted rows keep the Effect's id, kind and outcome for the run's life: the record of what the run's Effects did. A settled row's output TAIL is dropped once the save that carries its transition is on disk — that save's snapshot no longer holds the Effect running, so no restore reads it, and the full stream is on the run's feed. A push row keeps its Repo Slot, its published commit, its source, its squashed and its sync's answer whole: every later push of its Workspace and Repo Slot sends them as one of its pairs, and the newest push's sync, when it did not finish, runs again (ADR-0067); git's words are its tail, and so is its bundle, except on the newest push of its Repo Slot while that push's sync is busy or not-synced: that row keeps the bundle, which the sync that runs again moves the branch from. Only the newest push's sync runs again, so once a newer push of the slot published, the older row's bundle goes at the next save — at most one bundle per Repo Slot. The ledger grows by one short row per Effect a run starts, never by its tails, and lives as long as the run's row (ADR-0065). An outstanding pick's hold is saved in the same blob and keeps its chain whole until the pick is answered (ADR-0067); that is the only place an Effect's tails accumulate.

Pending timers do not ride the snapshot either. The host persists a timer table beside it in the same save: one row per pending after, named delay, delayed raise or delayed sendTo in the run's tree, with its absolute deadline. The author writes plain xstate after; there is no kit timer actor, and after is refused nowhere.

Mechanism (the non-obvious parts)

• Re-attach is driven by the child's persisted input, not the parent invoke. On restore, xstate v5 re-spawns an invoked actor from its persisted input — it does not re-evaluate the parent's invoke: { input: … }. So the re-attach lever is rewriting the child's persisted input on hydrate (drop prompt, mark it attaching) so the run resumes its stream instead of re-POSTing. Keep the parent-side input expression as defence-in-depth, but know it is not what fires on restore.
• A timer is re-armed by the host, because xstate 5 drops it. xstate 5 keeps a delayed event on the actor SYSTEM's scheduler, not in the snapshot, and a restore runs no entry action again, so a pending after never fires after a restore (Pool's pollEvery included, ADR-0017). At each save the host reads the scheduler's table and writes each timer as { source, target, id, event, delay, deadline }, actors named by their child-id path below the root. The deadline is a time on the host's wall clock (RunHostOptions.now, Date.now by default; tests pass one that does not step). On restore the order is: start(), then re-arm each row with deadline − now clamped to [0, delay], in deadline order. Re-armed before start(), a timer is armed twice, because the system's own start() schedules its table. The id is xstate's own, so a state's exit after the re-arm cancels a re-armed timer. start() drains the mailbox, so a child that sends on start can exit a state before any row is re-armed; the host records each cancel during start() and does not re-arm a row it names. A child stopped during start() is stopped with xstate's cancelAll, which sees an empty table, and its parent state entered again puts a new child at the same path; so the host resolves each row's actors before start() and re-arms a row only on the same actors after it. So a timer fires once, after the time that remained; at once if its deadline passed while the Orchestrator was down, and timers that are all overdue fire in the order they fell due; never if an exit cancelled it, before the save, during start() or after the re-arm; and never later than its own delay when the wall clock went backwards. A wall clock that jumped forwards fires it early: only the wall clock survives a process. "Once" is with respect to the saved state: a timer that fired after the last save fires again on restore, as any event after the last save replays. A delayed send to an actor outside the run's tree is not recorded; jr2 has no such idiom.
• The re-arm reads an xstate internal (system.getSnapshot()._scheduledEvents, system.scheduler.schedule, and system.scheduler.cancel, wrapped for the length of start(); packages/orchestrator/src/timers.ts). The same two scheduler functions are also wrapped for the life of the run's system, to tell a timer that fires from an event an actor sends at once: both reach the inspection stream alike, and the scheduler forgets a timer as it fires it. An outstanding pick's stopped answer needs that difference (ADR-0024). Raw-xstate tests that show xstate itself dropping the timer are the canary for an xstate bump. xstate 6 persists timers in the snapshot and re-arms them with the same deadline and clamp; on xstate 6 the timer table is deleted.
• Reconcile against the live world before re-attaching. Restore is not blind resume: a present Sandbox CR → re-attach; an absent one → the defined failure path (workspace.lost into the body — ADR-0012), never a silent re-provision.
• Reconcile against the MACHINE too (added 2026-08-02, ADR-0030). "Reconstructable from a persisted snapshot" was only ever true of the Machine that WROTE it, and a run resolves its workflow by filename while the state volume outlives the image — so a run parked across a deploy meets whatever was baked in next. The snapshot now carries a digest of its Machine's shape, and a mismatch is refused rather than interpreted. This is not belt-and-braces: xstate accepts a value naming a state the chart no longer has and starts with value: undefined, so nothing downstream would have caught it.

Consequences

• Actors may not stash live connections in context for convenience; anything live is rebuilt per-invocation from serializable input (the Harness client from an endpoint, the kube client from ambient config) — which is also why a mock↔real swap happens at the wire (ADR-0011's stub Harness), never inside actor code.
• An Effect has no at-most-once sliver: its intent row is on disk before the call leaves, so a restart either finds no row (nothing was sent) or finds the intent and settles unknown when it cannot re-attach — or, for an http call the author marked idempotent that no replica holds, sends it again under the same id.
• The known durability edge stands for Agents: a crash between "run admitted" and "admission persisted" degrades to an at-most-once re-POST (a live-run probe replays from -1; a genuinely-absent run re-POSTs), and an in-flight POST at the instant of the kill is the irreducible at-most-once sliver.