ADR-0004 Architecture decision record

A Sandbox's workspace is a private local clone of a read-only cache, not a shared worktree

Each Sandbox gets its own .git by running a local git clone --shared from a canonical bare checkout that is mounted read-only, rather than a git worktree add against a single shared .git. The clone borrows objects from the checkout (via alternates) for fast, disk-cheap setup but owns its refs, index, HEAD, config, and — critically — its lock files, so a Sandbox contends only with itself. The checkout it borrows from is a node-local cache the operator keeps (ADR-0051), one per repository per node, so a Sandbox can be scheduled on any node.

Why

jr gave every feature a git worktree, but all worktrees resolved through one default/.git. Git serializes access to that shared state with lock files, so concurrent commit/fetch/rebase/ref updates across worktrees blocked each other through the filesystem. That contention is the bottleneck the whole per-pod isolation design exists to kill — so the worktree model had to change, not just move into pods. A private .git removes the shared lock surface by construction; that is the load-bearing claim, and it does not depend on a measurement. A PoC corroborated it directly (same concurrent churn: lock-retries present with a shared .git, zero with private clones) and confirmed setup stays worktree-cheap (--shared .git ~7M vs a worktree's ~7M, vs a full copy ~33M).

The isolation is the point, not the speed: the read-only source plus a private pod-local clone is what confines an Agent's blast radius to its own pod (it physically cannot write — or gc — the canonical source or any sibling's work), which is the Sandbox thesis (GLOSSARY.md) applied to git.

We keep jr's worktree ergonomics — fast spin-up, a working dir that looks like a totally normal checkout, one canonical checkout everything branches from — because borrowing objects sets up about as cheaply as a worktree while giving an independent .git.

The --shared invariant, and how it is enforced

A --shared clone reads the cache's object store via objects/info/alternates; if an object it borrows is ever deleted from the cache, the clone corrupts (a PoC reproduced this by gc-ing a writable checkout under a live clone). So the one invariant: objects in a cache are never deleted while clones borrow them. Deletion only happens via git gc/prune — including the gc --auto that porcelain commands (fetch among them) run past a loose-object threshold, which prunes unreachable objects (e.g. commits a force-push orphaned that a long-lived clone still borrows). Enforcement, by side:

• Sandboxes cannot violate it: the cache is mounted read-only, and gc is a write. Structural.
• The cache agent is the one writer per node, and it is config-pinned: it sets gc.auto=0, gc.pruneExpire=never, and maintenance.auto=false on every checkout it clones or adopts, before any fetch — so neither its own fetches nor a human running git inside the checkout can trigger object deletion. Only a deliberate git gc --prune=now defeats it, which is rm -rf territory.
• Maintenance that deletes no object is allowed, and it is the agent's (scaling review R13, 2026-09-30). A cache that is only ever fetched accumulates a pack per fetch until every object lookup in every borrowing clone opens thousands of packs. So the agent consolidates on its interval, once loose objects or packs pass a count: git repack -a -d -k (one pack holding EVERY object, unreachable ones included — -k is the invariant, spelled as a flag: a force-pushed branch's commits stay for the clone that borrows them), git commit-graph write and git pack-refs --all. Git is built for readers under a repack — an open pack stays readable after its unlink, and a missed object re-scans the pack list — so a borrowing clone never sees an absent object. The rule for anyone who edits that command: no --prune, no gc, and gc.pruneExpire=never stays.
• Eviction is deletion of the whole checkout, never of objects inside one, and only once no pod on the node mounts it (ADR-0051). A live clone never loses what it borrows.

The fallback where borrowing can't be made safe (a source jr2 doesn't control) is a full copy — fully independent, gc-proof, at full disk and the slowest setup. An immutable generation-swap scheme (refresh = a new snapshot, old generations released as their Sandboxes drain) was considered again as OCI image volumes in ADR-0051 and rejected there: it buys nothing gc pinning does not, and a repack costs a full re-upload.

Refresh is fetch-in-place; the Repo resource is the catalog

The catalog is the set of Repo custom resources in the Instance's namespace (ADR-0051): the Orchestrator creates one per repository its Machines bind, at boot from the walk and at first attach for a per-run url, and the operator reconciles each onto the nodes that need it. A cache is cloned when missing and fetched when present — on the resource's interval, on demand before every attach, so a Workspace starts from the remote's now, and on demand for every fetch run inside a pod — the worktrees' origin asks the cache, and the cache asks the remote (ADR-0053). fetch only adds objects — safe under the invariant by nature. A repository that fails to sync degrades that repository, never the Instance: a cold clone that fails fails the one provision that needed it, pointedly; a fetch that fails on a warm cache lets the attach proceed on what the cache holds, announced as stale (ADR-0048). The cache agent runs in-cluster, so every repository must be fetchable from the cluster (an HTTPS token or deploy-key Secret for private ones — ADR-0019, ADR-0047); a working copy that exists only on someone's host is not a valid source. Local-path urls are cloned like any other url, never used in place. A checkout found in a node's cache directory with no Repo resource is adopted and pinned but not refreshed, and is evicted like any other once nothing mounts it.

A repository is identified by its url — host plus path, with scheme, user, a trailing / and .git dropped — so every spelling a Machine writes for one repository lands on ONE cache per node, and the expensive clone happens once per repository per node. The directory name is derived from that identity, printed beside the url by jr2 status, and never chosen by a human; there is no name to derive twice or to orphan.

Storage shape: a cache per node

The contract every Sandbox needs is narrow — a read-only directory per repository it attaches, on the node it runs on — and the writable side is always pod-local (emptyDir), which scales with agent count by construction. The backing is a hostPath directory per node that the Instance's cache agent (a DaemonSet) populates and the operator mounts read-only into Sandboxes, with a soft affinity toward nodes already holding the repositories a Sandbox names. A node that lacks one clones it on first need — the image-pull model, and the reason the cluster's node count is not a limit on Sandbox placement. A single-node cluster (kind) is the one-node case of the same model; there is no separate volume path to keep.

Read-only-ness is load-bearing twice over: nobody can write the shared thing (no write contention), and nobody can gc it (the structural half of the invariant). Shell-in (kubectl exec) needs none of this — the cache exists purely for fast setup.

Where setup runs

Worktree provisioning is a post-Ready step owned by the Orchestrator: the Sandbox CRD names repositories by identity so the operator can place and mount them, but it knows nothing about clones or worktrees (ADR-0001, ADR-0051). workspace()'s attach is one idempotent in-pod script the Harness runs on POST /attach (ADR-0012, ADR-0063), once Ready says every cache is present and fetched.

In-Sandbox layout: worktrees off the per-Sandbox clone (gwtmux convention)

"Not a shared worktree" above means not a worktree off a fleet-shared .git. Inside a Sandbox, worktrees are the intended ergonomic — taken off the Sandbox's own clone. The per-Sandbox clone is the default/, and branch working trees are its siblings, matching the layout gwtmux already expects, under the slot the Machine gave the repository (ADR-0051):

/work/<slot>/default/        the per-Sandbox `git clone --shared` (holds the pod-local .git; the default branch checked out)
/work/<slot>/<branch>/        `git worktree add` siblings, one per branch worked

Because the layout is identical to the one used outside the cluster, existing worktree tooling works unchanged when you exec into a Sandbox. Object resolution falls through: a branch worktree shares the per-Sandbox default/.git, whose alternates point at the read-only cache — existing objects are read from the cache, new commits land in the pod-local .git. default/ is a checkout of the Repo's default branch, as gwtmux's default/ is outside the cluster: work happens in the branch worktrees, and default/ is the reference tree a human or a tool finds where it expects one. Not --no-checkout: an empty working tree reads as every tracked file deleted to git status, which misleads an Agent that looks. The branch name default is refused by the spec guard: the branch Worktree is a sibling of that directory, never it. origin's push url is the binding's own spelling, so a Machine that bound over ssh pushes over ssh even when the cache was cloned over https.

This does not reintroduce jr's contention: the shared .git here is per-Sandbox, and it is accessed serially under the standing invariant that one Agent runs at a time per Workspace — structural since ADR-0012 (one sequential body per workspace). A Machine that runs concurrent Agents in one Sandbox, or an Agent that spawns parallel git-touching subagents, reintroduces contention (pod-scoped, not fleet-wide) and must opt into separate Sandboxes instead.