ADR-0064 Architecture decision record

A Workspace waits for capacity

Since ADR-0060 a Sandbox reserves its whole Size, so a full cluster shows as a Pending pod, not as eviction. But the provision's 120-second budget started at the CR's write, so time spent Unschedulable used it up: the run errored, and the CR, pod and Secret stayed until the operator's idle GC. A burst of 10 Workspaces at 500m each on the home cluster (2026-09-29) placed 2; the other 8 errored at 120s and left Pending pods behind, one of which could take a node later for nobody. Worse, the Lease was renewed only in running, so the operator counted creation as the Lease and would reap a Sandbox that waited past its 30-minute idle timeout. The scaling review asked two more questions: should jr2 cap live Workspaces for the Instance (R4), and concurrent Turns per model provider (R5)?

Decision

• A Sandbox with no node waits, and jr2 sets no deadline on the wait. jr2 cannot tell "full now" from "never fits": the cluster autoscaler, a finished neighbour or a drain can each make room, and the node set moves (ADR-0052). jr2 up already warns when no node could hold a Size (ADR-0060). A Machine that wants a bound puts after on the state that holds the Workspace; stopping it is the body's policy, as for workspace.lost (ADR-0021).
• A quota refusal is the same wait. A ResourceQuota (or Kueue) is the cluster owner's tool; jr2 neither writes one nor replaces it. The operator publishes the refusal on the Sandbox's status and keeps retrying the pod create.
• The Workspace Machine has a state for the wait: placing, before provisioning. placing writes the CR and waits for the pod to be scheduled, with no deadline. provisioning waits for Ready under the 120-second budget, which now measures only what jr2 controls: the image pull, the Harness start and the Repos. A restart re-enters placing: the apply is idempotent and the watch's first list answers at once (ADR-0063).
• The wait is visible; its reason is behind the token.
  • The state is structure, so the Console lights placing and observe() carries it with no new field.
  • The authenticated RunStatus gains waiting: [{ child, on: "node" | "quota", message, since }]. message is the scheduler's or the quota's own words, which can name nodes and taints. jr2 status <run> prints it.
  • The feed has two lines per wait: when it starts, with the reason, and when it ends ("placed after 3m12s"). The duration of a wait this Orchestrator process opened is read on the monotonic clock, so it is never negative. A wait opened before an Orchestrator restart has only its stored wall-clock start, so its duration is two wall-clock reads, clamped at 0. A changed reason updates waiting; it adds no feed line.
  • jr2 runs marks a run that has a Workspace in placing.
  • The Machine receives no event: scheduling detail is not the body's concern, and after covers a bound.
• The Lease covers the Sandbox from its write to its teardown, in every state, so the idle GC reaps only what the Orchestrator has abandoned (ADR-0001).
• Delete what never ran; keep what ran. A Sandbox whose wait ends without a node — a parent after, a stopped run, a faulted run — is deleted at once: nothing on it can be inspected, and left alone it could take a node later. The delete does not touch a name that a live Workspace of the run holds then: a parent that re-enters the state, or invokes the same Machine again, gives its new Workspace the same invoke id and so the same name, and that Workspace has placed it again (ADR-0073). A Sandbox that was placed and then faulted keeps ADR-0021's rule: it stays inspectable (logs, jr2 ssh) and ages out at the idle timeout.
• jr2 caps nothing of its own. Not live Workspaces (R4), not concurrent Turns per provider (R5). See the options.
• A provider's refusal is named. A Turn that ends because the Harness's provider retries ran out has a fault reason with the fixed prefix provider limit, as a memory kill has memory limit (ADR-0061), so jr2 status and the Machine can tell provider pressure from a bug.

Considered options

• Keep a deadline on the wait, or make it a config value. Rejected: any number either faults a Workspace the autoscaler was about to place or waits for one that never fits, and the Machine's own after already states the body's bound where the body can act on it.
• An Instance-wide cap on live Workspaces (sandbox.max), first come first served, persisted: Agent Substrate's request parking in jr2's terms. Rejected: a count of Workspaces of mixed Sizes is not a measure of resources. A spend ceiling is a ResourceQuota on requests.cpu and requests.memory, exact in resource units and refused before any pod exists, so the autoscaler never scales for it. Placement order among Pending pods of one priority is not promised, and nothing in jr2 needs strict order. Substrate's parking is driven by capacity, not a count.
• An Instance-wide cap on concurrent Turns per provider (R5). Rejected:
  • A Turn is the wrong unit: it is many model calls with tool runs between them, and a slot held while tsc runs blocks the provider while it has nothing to do.
  • A fixed number is the wrong measure: hosted providers limit tokens and requests per minute, vLLM by batch and KV cache.
  • The wait already happens in the right place: vLLM queues in arrival order, and the Harness retries twice, through pi-ai, with backoff, and honors retry-after up to 60 s (pi-ai 1.4.0). pi-ai retries what the SDKs retry: a 408, a 409, a 429, each 5xx, and a connection error (an error with no status). So a quota 429 is asked again too; its settlement keeps the quota's own words, never provider limit. The scaling review measured slow Turns at 40 concurrent (about 90s against 14s alone), which is throughput a cap cannot add, and no failed ones.
  • A slow provider cannot make a Runaway: ADR-0035 counts steps and repeats, not time.
  • If a hosted provider's 429 storm is ever measured, the answer is shared backoff at the Custodian (Envoy's global rate limit), not a Turn count.
• Watch for a Pending pod from the Orchestrator. Rejected by ADR-0063: the operator reads the pod and publishes the scheduling condition.

Consequences

• ADR-0012's lifecycle gains placing. ADR-0021: the Lease is no longer only in running, so "attach must stay well inside the idle timeout" goes away. ADR-0001: the operator publishes a quota refusal like an Unschedulable pod.
• A Workspace parked in placing holds no node, only a CR and a Secret. A Workspace that waits a long time still holds its run's place in any pool(): the Pool's cap is the Machine's own concurrency policy.
• GLOSSARY.md gains Placing, and Lease covers the Sandbox's whole life.