ADR-0042 Architecture decision record

Ready is not routable: the two calls a turn starts with re-ask a Service that has not answered

ADR-0016 sorted an Agent run's failures into classes and said which ones jr2 absorbs. Its infra class is stated as already-covered: "provider-stream retry lives inside the turn in the Harness, and wait reconnects indefinitely from the offset ledger; a dead Harness surfaces as a fault". That sentence walks the turn from its stream backwards and never reaches the two steps in front of it. Both legs that run BEFORE a model is ever asked were single, unretried fetches — the Orchestrator's admission POST to the Workspace Harness, and the Harness's GET /agents/:iid/surface that builds the turn's Menu — and either one rejecting settles the turn, which actor.ts reports as the terminal agent.fault. One dropped packet, one lost turn.

The gap is not theoretical and not rare, because both of those calls dial a Service that has only just started mattering. The admission is the FIRST HTTP request jr2 ever sends over a Workspace's Service: the wrapper waits on the Sandbox CR's phase: Ready, which the operator computes from the POD's readiness, and the attach then reaches into the pod through the API server (kubectl exec) — neither proves the Service dialable. The surface read dials the Orchestrator's Service, which is between EndpointSlices every time the Orchestrator restarts, and restarting is ordinary (ADR-0007's restore is built on it). Kubernetes programs the EndpointSlice behind a ClusterIP after the pod passes its probe, and until it does, kube-proxy REJECTs. Ready is a statement about a pod; routable is a statement about a Service, and jr2 was reading the first as the second — at both ends of the same turn.

Measured: this is what cost the @kind tier its parallel default (ADR-0010). Roughly one parallel run in three lost a scenario whose first turn never reached the scripted model, and the symptom was read for a long time as "a slow first turn". It is not. It is a lost turn, and the tier's failure dump caught both legs losing it:

• The admission, three times. The pod's Harness answered ?view=history with 404 — no conversation (POST is what creates one), the scripted model had received zero streaming requests, and the run settled {"outcome":"faulted","reason":"fetch failed"}. In the same window the run-narrative echo — un-retried, and the only other thing that dials that Service — logged the same fetch failed against the same address. Nothing had reached the pod at all.
• The surface read, once the admission was fixed and the tier ran again: submission settled failed: … {"error":"fetch failed"} — the surface read's failure, in the scenario that RESTARTS the Orchestrator. Which is the same defect one hop over: the Menu leg looks pod-local, and it is not — its hop leaves the pod for the Orchestrator's Service.

Parallelism causes neither; it multiplies the number of Services crossing into service at once, so it samples both windows more often. That is why the flake was degree-independent and serial runs never saw it.

Decision

• A never-delivered admission is re-sent. send re-POSTs while the request demonstrably never left the host, backing off on the same capped ladder wait uses. This is wait's own rule applied one step earlier and for the same reason: a request that could not connect is not a Harness that refused the prompt.
• Only never-delivered. Admission is accept-and-queue (ADR-0027): a POST the Harness received but could not answer has already queued a Submission, so re-sending it would run the turn twice — a worse failure than losing it. The test is therefore the narrow structural question "did any byte reach the wire" (the errno's syscall/code, read down the cause chain), never the broad "does this look transient". A reset connection is not retried.
• One answer is re-sent too: a draining Harness's 503. A Harness that received SIGTERM refuses an admission before it reads or queues anything, and closes the connection (ADR-0031), so that 503 is as undelivered as a refused connection — nothing ran, and the re-send dials the Service anew. It rides the same ladder and window. Every other status is the Harness's word about this admission, and stays a fault.
• The window is bounded, where wait's is not. wait may reconnect forever because its Submission is already admitted, so the lease owns the reporting (ADR-0021's workspace.lost). Nothing is admitted yet at this seat, so there is no turn for a lease to be about: an address that never answers is a fault this call must name itself, and it names the address it kept trying. 90 seconds — a measurement, not a derivation. The mechanism argues for far less: kube-proxy programs the EndpointSlice in well under a second, and CoreDNS's 30s negative TTL is the only other obvious floor. Reasoning from the mechanism alone is how this ADR first arrived at 60s, and 60s clears neither number the field has actually recorded — kubernetes#88986 measures 63s on bare metal (a SYN-retransmit ladder, 1-2-4-8-16-32, so those packets were DROPPED rather than refused), and kind#2280 measures up to 77s with the EndpointSlices already populated. The bound covers both. Its cost is paid only by an address that is genuinely wrong, and that one faults with the address in hand either way.
• The ladder is jittered, because the callers arrive together. Sandboxes that converge together cross the same window together, and every Custodian in the cluster dials the same Orchestrator Service — so an un-jittered ladder does not merely fail to help, it organizes the callers into a herd that retries and misses in lockstep. Each sleep is drawn from the TOP HALF of its rung (equal jitter). The random half decorrelates them; the half that stays a floor is the deliberate part, and it is what full jitter would give away — it is what still holds four clients off a Service that is genuinely down.
• A transport failure says what it was. fetch reports every one of them as the same three words, and reason: "fetch failed" on a run is a diagnosis of nothing — the errno one level down (connect ECONNREFUSED …, getaddrinfo ENOTFOUND …) is the whole answer. Both the admission fault and the echo's log line now carry it.
• The Harness re-asks an unanswered surface read, on ANY transport failure. An unanswered read is the Harness's own transport failure, or the Custodian's word that the Orchestrator never answered it (a 502 it marks as its own, ADR-0059), or no answer within 10 seconds. The Custodian itself retries only a connection reset: any reset under the read, and under a pick or an ask only one before the request was sent. It also drops a pooled connection after 4 seconds idle, before Node's 5-second keep-alive closes it. A GET is idempotent, so the narrow never-delivered test the admission needs buys nothing here — there is no second turn to accidentally start, so the rule is simply "an answered request is an answer". A 503 is the Orchestrator's word that it is stopping, so it is re-asked too (ADR-0026). A 404 stays ADR-0026's turn-is-over and a 403 stays a scope refusal: both are answers, and neither is re-asked.
• deliver does not retry, and that is not an oversight. A failed pick reaches the model as a tool error it can act on — pick again, or pick differently — so the turn survives one; and a POST that may have been delivered must never be re-sent, because a duplicate pick is a duplicate transition. The asymmetry between the Menu's two calls is the same one between send and wait: what may be re-asked is decided by what a second copy would do, never by how transient the failure looks.
• The fault class is unchanged. Exhausting either window is still ADR-0016's single terminal agent.fault, and where it routes is still workflow policy. What changes is that it now means "this endpoint never answered", not "the EndpointSlice was a few hundred milliseconds behind".

What this does not fix

An unrouted agent.fault is not this ADR's. This ADR does not change what a workflow with no agent.fault route does with a lost Turn. ADR-0074 decides it: the walk refuses a state that invokes an Agent with no route on it or above it, and a fault that no transition takes at runtime ends the run error, with a fault that names the event, the state and the reason.

Consequences

• The @kind tier's wired default becomes --parallel 4 (ADR-0010 carries the measurements).
• A genuinely wrong address at either seat now costs up to its window before it faults, instead of failing at once. That is the intended trade: the address is right and the Service is late far more often than the address is wrong, and the fault message names the address either way.
• The rule generalizes, and the next seat to need it should say so out loud. Two hops needed this and were found one at a time, each by a failure. The claim is not "retry the network"; it is that jr2 dials Services at the exact moment they come into existence, and that a call which never got an answer has learned nothing worth acting on. Any new jr2→Service call made at a lifecycle edge starts with that exposure until someone decides otherwise.
• The retry is invisible to the workflow by construction (ADR-0016's absorption principle): no new event, no new budget on the authoring surface, no bookkeeping in Machine context. But absorbed is not the same as unmeasured, and the difference is load-bearing. Absorption alone would leave the tier benefiting from a window it cannot see: if routability got twice as slow tomorrow, all 88 scenarios would still pass, a little slower, in silence, until the day the window finally closed and the suite went red as a fresh mystery. Being unseeable is precisely how this class survived three sessions, so each seat emits ONE line when a retry actually cost something — jr2.routability seat=… attempts=… ms=… last=… url=… — and the @kind tier holds a budget against it, judged once per worker in AfterAll. A log line, not a run-feed event: the authoring surface is untouched, so nothing above contradicts itself. What the tier now asserts is not that the retry worked — the scenarios already assert that — but how much of the window is being spent, which is what turns 90s from a number read off two GitHub issues into a measurement of the cluster in front of us.
• The first live reading immediately corrected this ADR, which is the argument for having taken it. The budget was going to be attempts alone. The tier's first measurement was 2 attempts costing 10667ms — one dropped SYN sitting on undici's 10s connect timeout, not a ladder being climbed. So the refusal that dominates the cost here is a DROP, not the kube-proxy REJECT this ADR leads with; both are real, and they cost in opposite currencies. A REJECT is instant, so it spends attempts and almost no time. A DROP spends time and almost no attempts. An attempts-only budget of 4 would therefore have waved through ~40s of a 90s window without a word. The very next run measured the other mode — 3 attempts, 502ms, ECONNREFUSED — which settled the shape: neither meter sees the other's failure, so the budget is 30s or 8 attempts, whichever trips first, and the line carries the final errno so nobody has to infer the mode from the arithmetic. The two are calibrated to trip at comparable severity, which is what stops them being one meter written twice: against the jittered 250ms→5s ladder, 8 attempts is ~9–18s of refusal, and 30s is three connect timeouts. The first draft budgeted 4 attempts against an observed 3 — which is not a budget, it is a flake — and only the second reading showed it.
• This is the ecosystem's answer, not a jr2 workaround, which is worth recording because "retry" reads as a patch. kind#2280 is this defect exactly — EndpointSlices populated, connection still refused, up to 77s — and it establishes that waiting for the EndpointSlice is NOT sufficient, which retires gating the CR's phase: Ready on it as a narrowing rather than a fix. Knative's activator arrives at the same place from the other side: it probes the pod itself and treats its own successful probe as authoritative regardless of what Kubernetes says, exactly so traffic can start before the control plane catches up. jr2's version is that with one fewer moving part — the first real request IS the probe — which is only safe because neverDelivered is what makes it re-sendable. A separate probe would add a round trip and put the race back (probe succeeds, the rule changes, the real request lands).