# A Harness holds stand-ins; the Custodian holds the keys [ADR-0058](0058-a-harness-answers-the-orchestrator-alone.md) recorded an open gap: model API keys were in the Harness container. `JR2_PROVIDER_API_KEY` and literal `harness.env` values (for example `ANTHROPIC_API_KEY`) came from the `jr2-harness-env` Secret by `envFrom`. Working tools give the Agent code execution in that container, and every tool child inherits the env. So the Agent could read a key and send it out of the pod. Dropping pi's catalog was not an option: the user's real setup is Bedrock through a LiteLLM gateway, with a fixed `ANTHROPIC_API_KEY` and `anthropic/…` models, and the catalog holds the limits, costs and thinking levels of those models. The same problem had one solution already, for one credential. The Adapter held the Sandbox token in a container the Agent executes nothing in ([ADR-0013](0013-the-agent-reaches-its-machine-through-a-container-it-cannot-read.md)). This decision generalizes that container. It holds every credential the Harness in its pod uses, and it replaces the Adapter. ## Decision - **A held secret is a credential the Harness uses and never holds.** The Harness container's env holds its **Stand-in**, `jr2-held-`. The Stand-in is deterministic and public. It means something only to the Custodian, and only toward a host the secret is bound to. It is unique per name, it is a valid header and env value, and it never contains `sk-ant-oat` (pi reads such a key as an Anthropic OAuth token). The word is not "placeholder", because that is the word of an Open part. - **The Custodian holds them.** It is a container in every Harness pod — every Sandbox and the Instance Harness — beside the Harness, and it holds the held secrets its own pod names (below). The Agent executes no code in it, and `shareProcessNamespace` stays off ([ADR-0005](0005-sandbox-pod-composition.md)). So the Agent cannot read its env, its files or its `/proc`. Its engine is Envoy: the upstream distroless image, pinned by digest (`CUSTODIAN_IMAGE` in `@jr2/orchestrator`). The pod's maker writes Envoy's whole configuration — the Orchestrator a Sandbox's at provision, `jr2 up` the Instance Harness's — from the manifest `jr2 up` declares (see "Objects, per pod"). There is no admin listener and no xDS, so nothing in the pod can change the configuration or read it back. - **The Custodian replaces the Adapter.** The `@jr2/adapter` package and the `jr2-adapter` image are deleted. The Sandbox token is a held secret of the kit's own: the Harness container holds `JR2_SANDBOX_TOKEN=jr2-held-JR2_SANDBOX_TOKEN`, and the Custodian holds the token as a file from the `-token` Secret (the Instance Harness: the `JR2_INSTANCE_HARNESS_TOKEN` key of `jr2-instance`). No other container mounts it. - **The Harness reads its Menu itself.** There is no MCP. For each Submission, `menu.ts` reads `GET /agents//surface` and delivers a pick with `POST /agents//events`, both to the Custodian on `127.0.0.1:8081`, with the Stand-in as the bearer. `menu-tools.ts` gives pi each accepted event as a tool. The tool names stay `mcp__jr2__`, because shipped instructions and the printer depend on them; the name is no longer a transport. `menu-tools.ts` adds the prefix, and `menu.ts` keeps the bare name, because an MCP client adds `mcp____` itself. The Harness keeps what the Adapter did: a turn that is over is never prompted ([ADR-0026](0026-a-turn-that-is-over-is-never-prompted.md)), a receipt is prose ([ADR-0024](0024-an-agents-turn-ends-with-the-state-that-asked-for-it.md), [ADR-0029](0029-a-menu-is-fixed-for-a-conversation-and-a-turn-is-told-its-allowed-picks.md)), the surface read retries on the ladder with the `jr2.routability` line ([ADR-0042](0042-ready-is-not-routable-an-admission-retries-its-connection.md)), and a `poll` event refuses the turn (`deferred` is retired, ADR-0006). The event's own JSON Schema is the tool's parameters, so the model sees the contract the Orchestrator validates against. - **The control listener forwards four routes and no other.** `127.0.0.1:8081`, plain HTTP, to the Orchestrator's Service (`jr2-orchestrator..svc:4000`, written into the pod's bootstrap by whoever makes the pod — see "Objects, per pod"): ``` GET /agents//surface → the same path the Menu POST /agents//events → the same path a pick GET /agents//deliveries/ → the same path an outstanding pick's answer (ADR-0024) POST /fetch → /sandboxes//fetch the ask (ADR-0053) anything else → 404, from the Custodian ``` A read — the surface, a delivery's poll — is retried on any reset; a pick and the ask only on a reset before they were sent, so one the Orchestrator may have read is not sent twice. A poll holds up to about 25 s, far below the listener's 3600 s idle limit, so an outstanding pick never rides one long request (ADR-0002). On each forwarded request, the Custodian swaps `Authorization: Bearer ` for the token and removes every other credential header. A request without the Stand-in gets 403. The ask's path comes from the Custodian (`JR2_SANDBOX` in its env, set per pod); on the Instance Harness there is none, and the ask gets 404. The route allowlist is the second layer. The control stays in the Orchestrator: the token's scope, the live registration, and the event's name and payload are judged there, as before (ADR-0013). - **Model keys go the same way, over HTTPS.** When its pod holds a held secret, the Harness container also gets: ```sh =jr2-held- one per held secret of the pod HTTPS_PROXY / https_proxy=http://127.0.0.1:15001 NO_PROXY / no_proxy=localhost,127.0.0.1,::1 NODE_USE_ENV_PROXY=1 NODE_EXTRA_CA_CERTS=/etc/jr2/ca/extra.crt SSL_CERT_FILE / REQUESTS_CA_BUNDLE / GIT_SSL_CAINFO=/etc/jr2/ca/bundle.crt ``` These ride `env`, after `harness.env` and before the wire's digest. In Kubernetes `env` wins over `envFrom`, so a Stand-in overrides a same-named key of an `envFrom` Secret, even one that changes later. There is no `HTTP_PROXY`: the Custodian carries TLS only, and plain HTTP (the Custodian's control listener, an in-cluster seed repository) goes direct. `NO_PROXY` is loopback only, so an in-cluster gateway that a secret is bound to still goes through the Custodian. - **The egress listener: CONNECT only.** `127.0.0.1:15001`. The CONNECT target decides everything (lowercased, without a trailing dot) — never the SNI, never the `Host` header. A target that a held secret is bound to goes to an internal listener for that target alone. That listener ends TLS with a leaf `jr2 up` issued (ALPN `http/1.1` only) and, per request: - The method must be GET, HEAD, POST, PUT, PATCH, DELETE or OPTIONS, else 405. A server that answers TRACE echoes the request, the value among it, back to the Agent. A header name with `_` gives 400: `x_api_key` is not `x-api-key` to the strip, but some servers read it as one. - `Host` must name the target, else 421. An encoded `/`, `.` or `\` in the path gives 400. A path outside the secret's `paths` gives 403. It is checked after Envoy removes dot segments and reads `\` as `/` and `..;` as `..`, and the upstream gets that path. So IIS and Tomcat cannot read a different path. A credential header sent twice gives 400. - The swap: in each header the secret names, the Stand-in alone, or one auth scheme and the Stand-in (`Bearer `), becomes the value. Nothing else is rewritten. - Every credential header that is not the product of a swap is removed. A request with no swap gets 403 and never leaves the pod: `jr2 custodian: needs the stand-in of in `. - Envoy opens its own TLS to the target, verifies the chain against `upstream.crt` and the name against the target's SAN (DNS or IP), and adds no header either way (`server`, `x-envoy-*`, `x-request-id` and `x-forwarded-*` are off). A failure gives 502 and a body that names the target and the reason. Redirects pass back unchanged. - When the target gives no answer, the Custodian answers 502 with a body that names the target, says what jr2 knows about the request, and ends with Envoy's reason. It replaces Envoy's own answer for ten Envoy response flags, in two mappers. One reply can carry several flags, and Envoy uses the first mapper that matches, in order. The first mapper is for `UF`, `UH` and `DF`, so a reply with one of them gets the first text whatever else it carries: - `UF` (the connection to the target failed), `UH` (no healthy host) and `DF` (the DNS lookup failed) say that Envoy did not send the request. The body is `jr2 custodian: could not be reached:
`. - `UC` (the target closed the connection), `DPE` (a downstream protocol error, which a body framing error can cause after the headers went), `UPE` (an upstream protocol error), `UR` (the target reset the stream) and `LR` (Envoy reset it) can come after the request went to the target. The body is `jr2 custodian: did not answer; the request may have reached it:
`. `UMSDR` (the maximum stream duration) is in this mapper: it ends a stream that was sent. `URX` (the retry limit) is in it too: Envoy sets it with the last attempt's own flag, so `UF,URX` gets the first text and `UC,URX` the second. - Envoy does not set `URX` or `UMSDR` on these routes, because they set no retry policy and no `max_stream_duration`. The two flags are in the mappers only to keep their union equal to the Orchestrator route's list. The second text is there because the first would be false after those flags. The target may have the request, so a `curl -X POST` or a `gh api` call to a bound host can write twice when the Agent sends it again. "Could not be reached" would tell the Agent that the first call went nowhere. The status stays 502 for both, the status of a gateway whose target gave no valid answer. No code reads the words. The Harness's provider adapter reads the status, and no model reads its own provider's 502. The Agent's own tools show the body to the model. Without the mappers, Envoy answers each of the ten flags itself: 503 for most of them, and 502 for `UPE`. That body names no target, so a caller can read it as the target's own answer. With them, every caller sees that the Custodian gave the answer, and for which target. A reset after the target's answer started is not replaced: the caller sees its stream end. The tunnel (below) keeps one mapper for all ten flags, with the first text. Envoy answers a `CONNECT` with 200 only after its TCP connection to the target is up, and no byte of the Agent's request goes before that 200. So a 502 on the tunnel always comes before the request reached the target, and "could not be reached" is true for each flag. The Orchestrator route is different ([ADR-0002](0002-duplex-streaming-actor.md)). Its mapper adds the header `x-jr2-custodian: unreachable` and the flags, and the Harness reads them to decide whether a pick may be sent again (`menu.ts`). It says "never reached" only when each flag of the reply is `UF`, `UH`, `DF` or `URX`, and "not known" when any other flag is there. - Bodies stream both ways. A route has no total timeout; an idle stream has an hour; a client that goes away resets the upstream request. Every other target is tunneled, byte for byte, after the dial guard (below). - **The engine choices.** The values and every credential decision are in a Lua filter. Its program is fixed (`program.lua`); each pod's own `custodian.lua` is a generated `HELD` table — names, headers, paths and targets, never a value — that ends with `dofile` of the program, so the program runs after the table it reads. At start the program reads each value from `/etc/jr2/custodian/values/`, takes off one trailing `\n` or `\r\n`, and stops the start-up with a line that names the secret when a value is missing, empty, or has a byte outside `0x20`–`0x7E`. Envoy's `credential_injector` is not used: it is alpha, it breaks on a secret file that ends in a newline (1.39.1), and it replaces a whole header, where the swap must see the Stand-in. Values never enter a ConfigMap. Envoy runs one worker, with hot restart off, and flushes its access log twice a second. - **The log is a contract.** One line per decision on stdout, with the prefix `jr2.custodian` and `key=value` pairs: `ready secrets=… hosts=N` (at the first probe), `action=intercept target=… secret=… method=… status=…`, `action=tunnel target=…`, `action=refuse target=… reason=…`. Never a header value, a body, a path, a query, a value or a Stand-in. - **The dial guard, in two halves.** A tunnel never reaches a loopback, link-local (the cloud metadata address included, and AWS's `fd00:ec2::254`) or unspecified address, or an IPv4-mapped form of one; it gets 403 and a `reason=guard` line. The script refuses a target that SPELLS one, and `localhost` or a `.localhost` name, before anything is resolved. The dynamic forward proxy filter saves the address it resolved a name to (`save_upstream_address`), and an RBAC filter after it refuses that address when it is in `GUARDED_RANGES` — so a name that resolves to one is refused like the literal. The address judged is the address dialed: both come from the one DNS cache entry. - **Health.** `0.0.0.0:15021`, `GET /healthz`. The kubelet cannot reach a loopback listener, and a distroless image has no shell for an exec probe. The Custodian's readiness probe is on it. Pod Ready needs every container ready, and the operator holds a Sandbox's Ready on pod Ready, so no Turn is admitted before the Custodian serves. The Instance Harness's rollout wait covers it the same way. It is the one listener a peer can reach, where the ADR-0058 NetworkPolicy is not enforced. So it is outside Envoy's global connection limit, holds 16 connections at most, and closes a connection idle for 5 seconds: a peer's idle connections cannot stop the Menu or the egress. - **The config surface** (`jr2.config.ts`, [ADR-0018](0018-instance-agents-are-definitions-jr2-assembles-the-harness.md)): the top-level `secrets` list, whose entry is `{ name; value | valueFrom.secretKeyRef; hosts?; headers?, default `authorization`, `x-api-key`, `x-goog-api-key`, `api-key`; paths? }`, and `harness.catalog` (a catalog provider id → `{ baseUrl }`). The list serves both sides ([ADR-0068](0068-an-effector-holds-an-effects-secrets-and-signs-its-wire.md)). `harness.provider.apiKey` is an entry too, implicitly: `JR2_PROVIDER_API_KEY`, bound to its `baseUrl`'s host. `loadConfig` checks the shape in both worlds. - **Naming is placement: a pod gets the held secrets something in it names.** Placement is read, never declared (`secretPlacementsOf`), and an entry's `hosts`, `headers` and `paths` are its binding — where the Custodian may put the value — never its placement: | Pod | The held secrets its Custodian holds | | --------------------- | ------------------------------------------------------------------------------------------ | | A Workspace's Sandbox | the names its `workspace()` lists under `secrets: [...]`, and the model keys of its Agents | | The Instance Harness | the model keys of the `"none"` Agents it hosts, and nothing else | A model key is claimed by the PROVIDER: an Agent names its model, and jr2 knows which keys that provider reads (`MODEL_KEYS`; the custom provider's id for `JR2_PROVIDER_API_KEY`). An Agent's Turns run in the Workspace below which it is declared, down to the next `workspace()`, unless it is `"none"`, whose Turns run at the Instance Harness ([ADR-0031](0031-menu-only-agents-run-on-the-instance-harness.md)). A `"none"` Agent cannot execute code, so the Instance Harness loses nothing by holding only model keys. Listing a model key under a `workspace()`'s `secrets` is allowed, and `jr2 up` reports it as redundant. A name a `workspace()` lists that the list does not hold is refused, with the line to add. An entry nothing names — no `workspace()`, no provider, no Effect — goes nowhere, and `jr2 up` warns. `jr2 up` reports per pod: `Workspace of workflow "coding": Custodian holds ANTHROPIC_API_KEY (provider), JIRA_TOKEN`. `jr2 up` alone checks the claimed entries (`heldSecretsOf`), host-side, because `.env` values are absent in-cluster. An entry no pod claims is not checked as a held secret: | # | Refused | | --- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | R1 | A known model key (`MODEL_KEYS`) in `harness.env`, or as a key of a Secret or ConfigMap `harness.envFrom` names. | | R2 | A held-secret name that is `JR2_*` or a proxy or trust variable. | | R3 | One name held twice, or held and set by `harness.env`. | | R4 | Not exactly one of `value`/`valueFrom`; an empty value; a byte outside `0x20`–`0x7E`; leading or trailing space. | | R5 | A claimed entry with no `hosts` where no default exists; a wildcard, userinfo, a non-`https` scheme, a bare-host path, a bad port; two secrets on one host in one header. | | R6 | A `headers` entry that is not a lowercase token, or that frames or routes the request; a duplicate. | | R7 | `provider.apiKey` with a non-`https` `baseUrl`; a non-`https` catalog `baseUrl`. | | R8 | A catalog key that is not a known provider, or that is the custom provider's id. | | R9 | Explicit `hosts` on a known model key that leave out where a carried Agent's provider sends its calls. | | R10 | A `secretKeyRef` whose Secret or key does not exist. | | R11 | While a secret is held: `harness.env` sets a proxy or trust variable. | | R12 | A `paths` entry that does not start with `/`, or holds `..`, `;`, `\`, `%2e`, `%2f` or `%5c`. | | R13 | A held value that also reaches the Harness: its Secret loaded by `harness.envFrom`, its key read by a `harness.env` `secretKeyRef`, or its literal set by `harness.env`. A kit Secret (`jr2-instance`, `jr2-held-*`, `jr2-instance-effector`, `jr2-effector-*`, `-effector`, `jr2-git-*`) named by either. | Each message names the config path, never a value, and says what to write instead. A SigV4 or ADC credential in `harness.env` is warned about, not refused. A known model key with no `hosts` is bound to the host its provider's calls go to: the host of `harness.catalog..baseUrl` when set, else pi's catalog host. The key and the calls then read one config and cannot drift. `MODEL_KEYS` is pi's data at the pin, written as plain data by `just model-keys` (a script in `@jr2/harness`) at each pi bump, so the Orchestrator has no pi dependency. A test in `@jr2/harness` checks the table against pi, and checks that the file is the script's exact text. So a pi bump that moves a provider, or a hand edit of the file, fails the default gate until the recipe runs. - **A catalog provider can move to a gateway.** pi has no env var for a catalog provider's base URL. It hands the provider the model it is given. So `resolveModel` returns a copy of the catalog model with `baseUrl` replaced, for every model of that provider. The catalog's other facts ride along. The model id sent upstream is the catalog id, so the gateway must serve that name (a LiteLLM `model_name` alias). - **Certificates.** `jr2 up` keeps a per-Instance CA in the `jr2-held-ca` Secret, which no pod mounts; it reads the CA over the kube API. It issues one leaf per bound `host:port` into `jr2-held-tls` before a pod starts, so the CA's key never enters a Harness pod. Keys are ECDSA P-256, written by a small DER writer in `@jr2/cli` over `node:crypto`: no dependency, and no `openssl` on the host. `jr2 up` never rotates the CA; a human deletes the Secret to rotate it. A leaf is reused while its SAN is the same, it chains to the current CA, and more than 30 days remain. - **Trust.** The `jr2-ca` ConfigMap carries `ca.crt` (the user's `caBundle`), and while a secret is held: `extra.crt` (`caBundle` + the jr2 CA, for `NODE_EXTRA_CA_CERTS`), `bundle.crt` (the roots + both, for the tools whose variable replaces their store), and `upstream.crt` (the roots + `caBundle`, and never the jr2 CA, for the Custodian). "Roots" are the Mozilla roots of the Node that runs `jr2 up`. The ConfigMap is applied server-side, because two copies of the roots are more than client-side apply's annotation can hold. - **Every Secret is applied server-side.** A client-side apply copies the whole object into its `kubectl.kubernetes.io/last-applied-configuration` annotation, so the values of a Secret are then in plaintext for each reader of its metadata. So `jr2 up` applies each Secret that it writes server-side: `jr2-instance`, `jr2-harness-env`, `jr2-held-ca`, `jr2-held-tls`, `jr2-held-secrets-`, the Effector Secrets, the git Secrets, the deploy key and the check pod's Secrets. The field manager is `jr2`, with `--force-conflicts`, because jr2 is the only writer of these Secrets. A `stringData` goes as `data`, because a server-side apply owns the fields that it names, and `stringData` is write-only. On a Secret that an older `jr2 up` applied client-side, the first server-side apply migrates it: kubectl (1.26 and later) moves the fields of the client-side manager to `jr2`, so that apply removes the annotation and each key that it does not name, and `jr2` is then the only manager (measured on kind with kubectl 1.35.9). When that migration fails, kubectl only writes a warning on stderr. So after each apply of Secrets, a fallback removes the annotation (`kubectl annotate … last-applied-configuration-`). The removal of an annotation that is not there changes nothing, so the fallback needs no record of what was removed. When the fallback fails, `jr2 up` writes one warning that names the Secrets, and the converge goes on. The rule is in the apply seam (`kubectlAdmin.apply`), so a Secret that a later change adds is applied in the same way. The values go on stdin, never in argv. No ConfigMap holds a value. - **Objects, per pod.** The held manifest, the Custodian's files and the literal values are each a pod's own, from the one derivation that places Effect secrets. A pod is keyed `instance` for the Instance Harness, and `ws-` for a Workspace, the digest of its registered workflow and the placement route to its `workspace()` (the pod key below) — the route the walk reads statically and the run host reads off the live actor at provision, so both name one pod alike. Nothing rendered per pod lands in a shared object: `jr2 up` declares, and the pod's maker renders. `jr2-held` (always) holds `held-.json` for each pod (what it holds: names, bindings, headers, never a value) and `program.lua` once per Instance, so it grows by one manifest per Workspace. A pod's two rendered files, Envoy's `envoy.json` and its `custodian.lua`, are in a ConfigMap of that pod's own: the Orchestrator renders a Sandbox's at provision, from the manifest it reads, into `-custodian`, applied after the CR and owned by it like the `-token` Secret, so it goes with the pod (the Orchestrator's Role may create and patch a ConfigMap, and no more); `jr2 up` renders the Instance Harness's into `jr2-held-custodian`, applied with its StatefulSet and deleted with it. A pod's `custodian-config` volume projects its own ConfigMap and `program.lua` from `jr2-held`. The reason: every Kubernetes object stops at 1 MiB, and client-side apply's annotation at 256 KiB; one copy of Envoy's files per pod in `jr2-held` passed that limit at nine Workspaces, and a dedupe or a server-side apply only moves the ceiling. `jr2-held-secrets-` holds one pod's literal values, labeled `jr2.dev/held-pod`, and `jr2 up` deletes the one a pod no longer needs. `jr2-held-tls` (every bound target's leaf) and `jr2-held-ca` are the Instance's. The Orchestrator mounts `jr2-held` at `/etc/jr2/held` and reads the manifest of the Workspace it provisions, per provision — the `jr2-images` pattern ([ADR-0038](0038-jr2-up-builds-every-image-it-deploys.md)); a Workspace `jr2 up` never walked has none, and its provision is refused, naming `jr2 up`. A pod key is the workflow and the Workspace's placement route: the `actors` slot keys down to its `workspace()`, a wrapper's body skipped, where a plain Machine invoked inline adds xstate's generated invoke src for that hop (`xstate.invoke..`), held so it can never equal a slot key. `jr2 up` computes it from the state node, and the provision reads it off the live actor tree, so both derive one key. A key reached by slot keys alone is what it was. A wrapper's body adds no hop, so a `workspace()` reached through another `workspace()`'s body (`workspace(workspace(...))`, `workspace(pool(workspace(...)))`) has the outer one's key: `jr2 up` refuses that Machine, with or without a secret, since the two pods' secrets could not be kept apart, and names the fix (the inner `workspace()` under a key of a plain Machine's `actors`, which has a route of its own); every other Workspace has a key of its own. A Workspace reached through a Machine object handed to `spawnChild`, which no walk can see, fails at provision with the slot form. One function, `custodianComposition`, builds what a pod gains from its pod's manifest; `kubeSandbox` and `instanceHarnessObjects` both call it, so the two pod shapes cannot drift. The Instance Harness's pod template carries `jr2.dev/held-digest`, an HMAC over its pod's held-secret inputs keyed with the signing key, so a held-secret edit rolls it at the next `jr2 up`. Its Custodian reads each value and its files once at start, so the inputs are its two rendered files, the program, a literal's value, a `valueFrom`'s reference, and a digest of the value in the user's Secret, keyed with the signing key: `jr2 up` reads that Secret to check the key exists, and keeps the digest in place of the value, so a value edited there rolls the pod too. A live Sandbox keeps what its Custodian read at start. ## Considered options - **jr2's own proxy in Node, in a second container of the Adapter's image** (branch `held-secrets/node`). No new image and no second language; jr2 then owns TLS-parsing code at the trust boundary, and needs its own tests for every edge Envoy already handles. It keeps the Adapter. - **Envoy in a container beside the Adapter** (branch `held-secrets/envoy-sidecar`). The same Envoy configuration, and three containers per pod. Two containers hold two credentials, so a fault in one engine does not reach the other's credential. This branch trades that split for one container fewer and one component fewer: the Custodian holds both. - **The Orchestrator serves MCP, and Envoy forwards `/mcp/`.** Rejected. It reverses ADR-0013's "the Orchestrator hosts no MCP", puts a transport and session handling back in the control plane, and keeps the MCP SDK on both ends — to carry two plain HTTP calls. What ADR-0013 rejected about translation inside the Harness was the credential in that container. The Custodian now holds it, so that reason is gone. - **The Adapter relays model calls** (ADR-0058's first direction). Rejected: every provider needs its `baseUrl` rewritten to the relay, and pi's catalog has no env var for a base URL, so the catalog could not follow. - **Drop pi's catalog and name every model as a custom provider.** Rejected by the user: the catalog holds the model facts the user would have to copy by hand. - **iron-proxy.** It intercepts every host and cannot pass one through, and it makes leaves at runtime, so the CA's key would be in the pod. The Sandbox Image is the user's, and its tools may not trust a jr2 CA. - **mitmproxy, Squid.** mitmproxy streams SSE only if the addon asks, and its swap is code jr2 writes anyway; Squid brings the GPL and a poor CVE record. Both would be a fourth image. - **Envoy's `credential_injector`, or the value in a bootstrap Secret with `request_headers_to_add`.** Rejected: the first is alpha and fails on a trailing newline; both replace a header without seeing the Stand-in, and the second cannot hold a `secretKeyRef` value, which `jr2 up` never copies. - **Intercept every host.** Rejected for the iron-proxy reason: a tool that does not trust the jr2 CA would then fail on every host, not only on bound ones. ## Consequences - **What this closes, and what it does not.** The Agent cannot take a model key or the Sandbox token out of the pod. It can still **spend** a key through the Custodian while its pod lives, on any path `paths` allows. A key with admin rights can mint a new key, and a minted key is one the Agent can take away: use a key with no admin rights, and narrow `paths`. The User Container shares the pod's loopback, so a human in it can spend a key the same way. That human cannot read one either. - **Response echo.** Responses pass unchanged. An upstream that echoes the full credential in a response hands it to the Agent. Providers mask keys in errors; jr2 does not scrub responses. - **Bypass.** A client that ignores `HTTPS_PROXY` goes direct and sends the Stand-in, which is worth nothing. It is not blocked: egress control stays the cluster's decision (ADR-0058's open egress gap). The same holds for the control plane: a caller that skips the Custodian and sends the Stand-in to the Orchestrator gets 401 on every route (`auth.test.ts`). - **Clients that do not trust the jr2 CA** (Java, a pinned client) fail TLS to a bound host. They fail closed and send nothing. Unbound hosts still work for them. - **`SSL_CERT_FILE` replaces an OpenSSL tool's store.** A private CA baked into the Sandbox Image, and not put in `caBundle`, stops being trusted by curl, git and Python in a pod with a held secret. The fix is `caBundle`. The gain: `caBundle` now reaches curl, git and Python, which [ADR-0020](0020-private-ca-trust-is-a-named-concept.md) recorded as missing. - **Signing credentials cannot be held.** A SigV4 or ADC credential signs the request with the secret, and a Stand-in cannot be swapped into a signature. `jr2 up` warns. The fix is the Effector ([ADR-0068](0068-an-effector-holds-an-effects-secrets-and-signs-its-wire.md)): an Effect secret is the real value, in a container the Agent executes nothing in. Its `http` kind sends a bearer, a Basic pair or a header the author writes, and the credentialed `command` ([ADR-0067](0067-an-effect-is-an-actor-slot-and-runs-at-most-once-unless-marked-idempotent.md)) runs the signing tool there (`terraform`, `aws`, `gcloud`), its child confined with Landlock from the Effector's key and every secret it does not name. `ANTHROPIC_OAUTH_TOKEN` is held like a key, but pi sends a Stand-in as `x-api-key`, which an OAuth token cannot ride; hold it as `ANTHROPIC_AUTH_TOKEN` (`Authorization: Bearer`) instead. - **Rotation of a value** reaches a live Sandbox only when its pod is replaced. - **The walk reads definitions, not Turns.** A pod's model keys come from the models its Agents' definitions name. A Turn whose invocation overrides the model with another provider's (`AgentTurnInput.model`) finds no key for that provider in its pod: the Harness container gets a Stand-in only for the names in its own pod's manifest, so that provider's key variable is not set at all. The call then has no key (a catalog provider) or sends the custom provider's placeholder `unused`, and it fails. This holds in a Workspace and in the Instance Harness alike. The fix is a definition that names the provider; in a Workspace, the key under the `workspace()`'s `secrets` also works. The Instance Harness has no such list: its pod holds only the model keys of the `"none"` Agents it hosts. - **TLS-only bindings.** A key for a plain-HTTP endpoint is refused (R7). A LAN `llama-server --api-key` needs TLS and its CA in `caBundle`, or no key. - **No proxy chaining.** The Custodian cannot forward to a corporate proxy. R11 says so rather than fail silently. - **Two Harness env shapes.** With no held secret in it, a pod gets the Custodian's address and the token's Stand-in and no proxy, and its HTTPS goes direct. The Custodian runs in both shapes, because it carries the Menu. - **The dial guard adds no confinement while egress is open.** The Agent can dial those addresses itself. The guard matters when a cluster closes egress to everything but the Custodian: then the Custodian is not a way around it. - **One container holds two kinds of credential.** A fault in Envoy's TLS or HTTP parsing is a fault in the container that holds both the Sandbox token and the model keys. The branch with a separate Adapter keeps them apart. - **The Harness container now speaks the Orchestrator's two agent routes.** The Agent could always make those calls (the Adapter would forward any MCP call it asked for); now the Harness makes them itself. Nothing is judged in the pod. - **`jr2-upload-pack` carries the Stand-in as a constant.** The User Container gets no env (ADR-0005), and asks like any other seat. A test pins the Go literal to `standIn("JR2_SANDBOX_TOKEN")`. - **The Custodian's image is a Pinned image.** It is deployed and never built, which [ADR-0038](0038-jr2-up-builds-every-image-it-deploys.md) now states. `kitRegistry` re-homes it and `jr2 kit push` mirrors it ([ADR-0044](0044-kit-images-live-at-a-canonical-home-a-self-host-mirrors-it.md)). The image sweep never takes it, because jr2 never stamped it ([ADR-0039](0039-image-garbage-collects-by-reachability.md)). A kind node pulls it from its registry, so the `@kind` tier needs network to `docker.io`. `just kit-push` does not seed it. - **Where the Custodian suite runs.** `just custodian-test`: node's test runner against the pinned image under docker, on the host network with free ports, with local HTTPS upstreams under their own CA. It is opt-in, like `@kind`, because it needs docker ([ADR-0010](0010-bdd-acceptance-tests.md)), and the release job runs it on every tag ([ADR-0055](0055-a-release-is-a-pushed-tag-and-the-publish-guard-is-the-credential.md)): the image moves only with its pin, but the configuration jr2 renders moves with any release. About 35 seconds. It checks the swap, the strip, 403 with no Stand-in, 405 on a method, 400 on an underscore header, a header `Connection` names, two `Host` headers, idle connections on the health listener, 421, 400 and 403 on paths, the tunnel, the dial guard by every spelling and by resolved name, 502 on an untrusted upstream, SSE timing, abort, a 302, the start-up refusal, the log, and the Anthropic SDK through `HTTPS_PROXY`. The first SSE event arrives through the Custodian about half a millisecond later than direct. - **The `@kind` tier's model wants a key.** Its fake provider serves HTTPS under the tier's CA and refuses a request without the key, which is `harness.provider.apiKey` — so every turn in the tier crosses a Custodian, in both placements. A Rule proves the key is in no env and no file of either Harness container. - **The provider preflight crosses a Custodian.** A provider's key is held. So `jr2 up` checks the provider from a pod with a Custodian that holds its keys, and the stock Harness's `provider-preflight.ts` ([ADR-0019](0019-one-converging-command-against-the-current-context.md)). The check pod reads its own copies of the held objects. The check runs before `jr2 up` writes an object that the Instance or the operator reads. A literal key is in the check pod's own Secret while the pod lives. A wrong or missing key, no egress to the bound host, and a gateway that does not answer fail the converge, not a Turn. - **The CA has no name constraints.** Constraints would force a new CA whenever the bound hosts change, and that breaks a live pod whose Custodian restarts.