ADR-0031 Architecture decision record

Menu-only Agents run on the Instance Harness

A workflow should be able to run an Agent whose whole job is a decision — read some inputs, pick the next event — with no Workspace, no worktree, and no per-run pod: the statelyai/agent shape, at conversational latency, on any configured model. jr2's Menu already is that decision surface (the invoking state derives what the Agent may say — ADR-0015/ADR-0029), and the mechanism was already Sandbox-agnostic: agentRun needs only an endpoint URL, a registration may carry no Sandbox (tokens.ts), and the stub Harness proved a Harness is "only a different URL". What was missing was the hosting: the only real Harness anywhere was the one inside a Sandbox pod, so a Turn without a workspace() had nowhere to run. Adopting statelyai/agent instead was rejected outright — it would be a second, weaker agent mechanism (no Submission/Admission ledger, no re-attach, no Settlement, a menu that is not guard-narrowed) beside the one jr2 already built.

Decision

• One Instance Harness per instance, deployed by convention. jr2 up converges a Harness StatefulSet + Service — stock jr2-harness:<kitversion> image, same wire — whenever any Agent slot carried by a registered Machine (ADR-0049's walk) declares workspace: "none" (ADR-0028). The scan is static and definition-level, deliberately not workflow-level (workflow internals are not statically recoverable — the same line ADR-0018 drew for model preflight); a declared-but-never-invoked "none" Agent over-deploys, erring toward "the convention works when you need it". No config key names, sizes, addresses, or enables it.
• Placement is definition-wins, not nearest-wins. A workspace: "none" Agent's Turn runs on the Instance Harness always — even invoked from inside a workspace(). It is ADR-0028's thesis extended one step: an Agent that touches no Workspace has no reason to run in one, and a Sandbox — torn down at its Workspace's final state, replaced after an eviction — is the wrong home for a conversation whose definition says it needs no Workspace. A conversation is an Instance ID on one Harness (the server holds the history), so placement must be a function of the definition, not of where an invocation happens to sit; nearest-wins would make the Harness that holds a "none" conversation an accident of authoring. Definition-wins also makes "which pod's logs" a function of the Agent alone.
• The cohesive per-run log is solved by projection, not placement. The cost of definition-wins — decisions vanishing from the Workspace pod's log — is repaid by ADR-0023's echo: the enclosing Workspace's Harness prints the run's narrative (admission + pick markers for remotely-hosted Turns, Emits, the backfilled preamble). Instance Harness log = every "none" conversation, instance-wide, interleaved (the less useful top-to-bottom read, accepted); Workspace log = everything that ran there plus the narrative of the run that owns it.
• The Instance Harness pod is Harness + Custodian — the one Harness shape, minus the Workspace. No user container, no /work volume, no attach step. The Custodian stays although "none" Agents cannot execute code (the ADR-0013 rationale technically lapses): the Harness has exactly one route to its Menu — the Custodian on localhost — and forking that path for one pod buys a divergence in the component this ADR keeps deliberately uniform. The pod is composed by the same function a Sandbox's is (ADR-0059), so its Custodian carries the Instance Harness's model calls the same way. It holds only the model keys of the "none" Agents it hosts — the keys their providers read — and none of the held secrets a workspace() lists: a "none" Agent cannot execute code, so it had no way to use the rest. Its held objects are its own (held-instance.json, jr2-held-secrets-instance). Defense-in-depth is the bonus, not the reason.
• The Instance Effector is the second Instance-level pod, beside it (ADR-0068), on the same convention: jr2 up converges it when a registered Machine places an Effect there — an http Effect — and deletes it otherwise. It runs the same jr2-harness image with /opt/jr2/bin/effector as its command, holds no conversation and runs no Agent, and is a Deployment, not a StatefulSet: it keeps nothing, and a rollout must run the new pod beside the old one. No config key names, sizes, addresses, or enables it.
• No image config at all. The Harness image was once configured as sandbox.image because the Sandbox pod was the only place a Harness ran — a misnomer once the Instance Harness exists, and config is the wrong seat regardless: jr2 up builds and resolves every image it deploys (ADR-0038), and the per-Workspace images (Sandbox Image, User Container) are static workspace() options carried by the Machine itself (ADR-0049, ADR-0037, ADR-0005). sandbox.image, the per-component image keys, and operator.image dissolve; the sandbox config section disappears until something genuinely pod-shaped and user-tunable exists.

The Instance Harness holds a conversation as long as its run, and frees the rest

The scaling review (R11, 2026-09-30) found the conversations map with no delete path: every Menu-only Agent instance of every run stayed until the pod restarted, and the restart lost every conversation at once. The research that preceded it checked Agent Substrate's own model against the primary sources: an actor is stateful, the platform snapshots its memory and its DurableDir volumes and resumes it on any worker, and its control plane holds metadata and snapshot pointers only; memory snapshots are tied to one template version, so the state with a documented path across a code change is a volume the actor reloads on a cold boot. These decisions take that shape.

• A conversation is persisted to a directory, and rebuilt from it on boot. The Harness writes each conversation as it goes: jr2's own record — the stream log, the Settlements, the history messages, in the wire's shapes — plus the engine part, in a folder the engine names (engine/<name>), which the record does not read. Today it is jr2's transcript (ADR-0027): engine/pi/transcript.jsonl, the pi-ai messages the model read and each Compaction. Rebuilding reads jr2's record first; an engine that cannot read the engine part (another engine, another version) continues from the history messages. The record is the contract; the engine part is not. A Submission in flight at the restart settles failed (the typed error harness_restarted, which the Agent's Actor reads as "Turn lost") on the rebuilt stream, so the Orchestrator reads a Settlement, not a 404, and the conversation's next Turn continues it.
• A restart while the Agent's state is busy cuts the Turn and is no fault. When the Agent's state is not at Rest — its pick is outstanding, or work it started in runs below it — or its pick's answer was read and never reached the model, the Agent's Actor raises no agent.fault for a restart: the Machine would route it and stop the work the Turn did not want stopped (ADR-0024). It waits for Rest, then re-admits the Agent on the same conversation, which the rebuilt Harness holds: a jr2 prompt ("Your previous turn was cut: the process that ran you restarted while the workflow was busy. The workflow did not stop. Continue your task from here.") with the answer as a pick-answer notice (cut: true) in the Briefing's Turn part, and the Allowed picks after it (ADR-0062). The re-admission asks the rebuilt Harness for the conversation first, as a continued Turn does: one it cannot read gets a conversation-new notice ahead of the answer, and the notices pending for the Turn ride along. In the rebuilt engine part the cut Menu call has no result; the Harness answers it in the history with jr2's text — the pick reached the workflow, which does not run it twice, and its result comes in the next message — before pi-ai would answer "No result provided", which reads as a failed pick. That text prints once in the rebuilt Harness's log, before its first prompt (ADR-0023). The engine runs a model answer's calls in order (ADR-0027). So only the first call with no result can have been out. Every call after it never started, a Working-tool call too. Each one gets jr2's text "This call did not run: your turn ended before it started.", as an error result. A Menu call that an abort cut while the model was still writing it (the abort lands mid-stream) did not reach the workflow: the engine runs no call of a message the model did not finish. The Harness tells the two cuts apart by whether the call was delivered, read before it re-tags an aborted message so that pi-ai replays it: a call in a finished message keeps the text above, and a call in an aborted message gets jr2's text in the same form — the Turn was cut while the model was still writing this call, it did not reach the workflow, nothing ran, and the model may call it again if the newest "Allowed now" still lists it. That text prints once too, before the conversation's next prompt, in the same process or the next. The Harness reads whether a call was delivered off its message, not off a record of the POST, and that is the limit of the test: a SIGKILL of the Harness after the model finished the message and before the pick's POST reached the Orchestrator leaves a call that never reached the workflow, and the rebuilt Harness answers it with the restart's text. Closing that window needs a delivery record written before the POST, which is not built. A Menu call in a message that a provider error ended mid-stream (stopReason error) is neither cut: pi runs no call of such a message (pi-agent-core 1.4.0 agent-loop.js line 148 returns after a message that ended error or aborted) and drops it from every later request (pi-ai 1.4.0 transform-messages.js lines 172 to 180), so the Harness does not re-tag it and adds no answer for its call: an answer would be a tool result with no call before it, which the providers refuse. The pick did not reach the workflow, the Submission settles failed as any provider error does, and on the next request the model reads no trace of the call and is free to make it again. A poll takes the answer only while its connection lives: the killed Harness's last poll, still waiting at the Orchestrator, ends unanswered when its connection closes, so an answer that lands after the kill, and before the rebuilt stream reports the restart, is still unread. The same holds for a Sandbox's Harness, whose restart the watch sees (ADR-0021). A restart at Rest with nothing unread is the fault it always was. Only the Turn's OWN pick counts as unread: a pick of an earlier Turn on the same continued address, whose route left the Agent's state, was answered as a notice to this Turn, and a restart at Rest in this Turn is still the fault. A route that leaves the Agent's state ends the Turn while it waits, as any route does.
• Both placements persist, to different volumes. The Instance Harness to a PersistentVolumeClaim of its own; a Sandbox's Harness to an emptyDir mounted into the Harness container alone (the User Container must not read it), so a Harness container that restarts inside a living pod keeps its conversations. A pod that is gone takes the emptyDir with it — that Workspace is Lost (ADR-0021), and no conversation persistence should say otherwise.
• The Instance Harness is a StatefulSet of one, with a per-ordinal volume. The value is the stable volume; the stable name is what N > 1 would pin a conversation to (the admission ledger already records its endpoint). N is a kit value, 1 until a measured need; never an HPA, which would kill conversations on a scale-down and land a continuation on a replica that never saw it. A rollout at N = 1 has no surge, so admissions are refused while the new pod starts; the Orchestrator already absorbs an admission refused at the connection (ADR-0042), so a rollout delays a Turn and faults nothing.
• A rollout drains Turns. On SIGTERM the Harness stops admitting, finishes the Turns it holds (the queued ones too), then exits; the termination grace is a Turn's worst case. A Turn still generating or running a tool keeps that graceful wait. A Turn whose only pending work is an outstanding pick's poll — whatever the pick waits on: a Gate, an Effect, a child Machine — is cut at once, or as soon as it comes to that: its answer can wait on a human (a Gate), often the one who is stopping the Harness, so no grace period would help. The cut drops the poll unanswered (the answer stays unread at the Orchestrator), leaves the Menu call without a result, asks the model nothing more, and settles the Submission failed, harness_restarted, with everything queued behind it. To the Orchestrator it is a restart while the Agent's state is busy (the bullet above): the pick stays outstanding, and once the workflow is back at Rest jr2 re-admits the Agent on the same conversation with the answer, and the replacement Harness answers the cut call with jr2's text. It refuses an admission with a 503 before it reads or queues anything, and closes that connection, so the Orchestrator re-sends it on ADR-0042's ladder and the re-send dials the Service anew. Readiness stays up (the probe is the Harness's own /healthz, which it keeps answering), and the dialed Service is a ClusterIP one, which kube-proxy still routes to a terminating pod that serves while no other is ready — so reads keep answering and a wait on the draining pod still reads its Settlement. A headless name would not: DNS stops naming a terminating pod, so the StatefulSet's headless governing Service is a second one, dialed by no one. The Custodian is a native sidecar, stopped only after the Harness exits, so the drain keeps its Menu and its model to the end. A deploy loses no Turn and, with the volume, no conversation — as long as the drain and the replacement's start fit ADR-0042's window; a longer one faults the waiting admission with the Harness's own 503 words.
• The Orchestrator states the live set; the Harness frees the rest. After every restore and every 5 minutes (the Lease's cadence) the Orchestrator tells the Instance Harness which conversations its live runs hold — from the admission ledgers in memory, never from the store — and the Harness frees memory and files of every other one. Level-triggered, idempotent, a missed statement costs nothing, and a crashed Orchestrator's leftovers go at its next boot. No idle TTL: a run parked on a Gate for a day keeps its conversation, and nothing else can decide that. A Sandbox's Harness is not told; its Workspace's teardown frees everything.
• The Orchestrator keeps holding handles only (ADR-0007): no conversation state crosses to it. Storing sessions centrally and re-sending them to a fresh pod was considered and rejected — it is the design Substrate does not have, and it puts every Menu-only Turn's history on the single writer's path.

Considered options

• statelyai/agent (or any in-process decision actor). Rejected above; the long form: every ADR from 0011 to 0029 would stop being the single story of how models drive Machines, and the copy would be the lesser one — its picks bypass the Menu's guard-narrowing, its runs bypass the admission ledger, its failures bypass Settlement.
• Unconditional ambient fallback (no enclosing workspace() → Instance Harness, for every Agent). Rejected: a coder invoked outside its workspace() by mistake would run bash in the shared pod instead of failing loudly — precisely the mistake today's error catches. The fallback is gated on workspace: "none"; everything else keeps the loud error, now sharper ("agent x has workspace: "write" — invoke it inside a workspace()").
• Nearest-wins placement (a "none" Agent invoked inside a workspace() runs on that Workspace's Harness). Genuinely attractive — the Workspace pod log stays the one coherent story with zero new mechanism — and rejected on the continuation amnesia above plus three lesser costs: decisioner turns die with workspace churn, share limits with builds, and scatter one Agent's history across pods. The log story it bought is recovered by ADR-0023's echo.
• On-demand pod per workspace-less run. Rejected: pays pod-start latency on the path whose whole point is a fast pick.
• Hosting the Turn in the Orchestrator process. Rejected: no pod logs, and it crosses the line the architecture draws hardest — the Orchestrator holds handles; models act remotely.
• A host-local dev Harness. Considered as a dev story and dropped when the premise died: ADR-0019 removed jr2 dev — there is no host dev mode to serve. Local development of a decisioning workflow is jr2 up against kind, where the Instance Harness converges like everything else; an instructions tweak rides the next admission, no image build. The stub Harness keeps its one job: a test fixture reached by explicit endpoint.

Consequences

• The Agent's resolution grows one arm: explicit endpoint (tests) → workspace: "none" → Instance Harness (deterministic Service DNS) → enclosing workspace()'s ambient handles → loud error naming the definition's workspace value.
• A "none" conversation survives the Instance Harness pod's restart: the StatefulSet restores the endpoint and the volume restores the history. A Turn in flight at the restart settles failed; nothing is continued silently — a Turn whose state is busy is re-admitted with a prompt that says it was cut.
• An instance with no "none" definitions deploys nothing new. The feature is invisible until the first agent({ ..., workspace: "none" }) a Machine carries, which is its entire user-facing surface.
• The harness conformance suite (ADR-0027) gains the Menu-only shape: a Turn whose definition withholds every Working tool, settled by pick alone — no infra, same scripted provider.
• GLOSSARY.md: Harness's "inside a Sandbox" loosens to name the second placement; Instance Harness becomes a glossary term.