Appearance

Size a Workspace ​

Give a Workspace the memory and cpu that its Sandbox needs.

The Size is the most memory and cpu that the whole Sandbox can use. jr2 reserves all of it on a node, so the Sandbox always has it. Write it in the resources option of workspace(). In this example, the Workflow workflows/e2e.ts runs Playwright with 4 GiB of memory and 2 cpus:

ts
import { z } from "zod";
import { command, fail, jr2Setup, workspace, type Workspaced } from "@jr2/orchestrator";

const runInput = z.object({ branch: z.string() });
type BodyInput = Workspaced<z.infer<typeof runInput>, "app">;

const body = jr2Setup({
  types: {} as { input: BodyInput },
  actors: {
    // JR2_CPUS holds the cpu limit of the container. Playwright does not read that limit.
    e2e: command(["sh", "-c", "npx playwright test --workers=$JR2_CPUS"], { timeout: "30m" }),
    fail: fail(),
  },
}).createMachine({
  id: "e2e",
  on: { "workspace.lost": ".lost" },
  initial: "testing",
  states: {
    testing: {
      invoke: {
        src: "e2e",
        onDone: [
          {
            guard: ({ event }) => event.output.outcome === "exited" && event.output.code === 0,
            target: "passed",
          },
          { target: "failed" },
        ],
      },
    },
    passed: { type: "final" },
    // Not final states: each one fails the run, and the run ends `failed`.
    failed: { invoke: { src: "fail", input: { reason: "the Playwright tests did not pass" } } },
    lost: { invoke: { src: "fail", input: { reason: "the Sandbox was lost" } } },
  },
});

export const machine = workspace(body, {
  input: runInput,
  // The Size of the whole Sandbox: 4 GiB of memory and 2 cpus. jr2 reserves all of it on a node.
  resources: { limits: { memory: "4Gi", cpu: "2" } },
  repos: { app: "https://github.com/you/app.git" },
  spec: ({ input }) => ({ branch: input.branch }),
});

The Sandbox Image must have Playwright and its browsers. Add tools to the Sandbox tells how.

Choose the numbers ​

A Size is { limits: { memory, cpu } }, and nothing else. Each value is a Kubernetes quantity: "4Gi" or "512Mi" for memory, "2" or "1500m" for cpu.

The kit default is memory 2Gi and cpu 1. It is enough for tsc and vitest, for cargo build, and for Playwright with 2 workers. State a larger Size for Playwright with more workers, or for a large monorepo.

A larger Size is not free. The Sandbox holds all of its Size while the run waits, also at a Gate. When no node has room for the Size, the Workspace waits in the Placing state, with no time limit. Workspaces tells how to see the wait.

jr2 up warns when no node can hold a Size now. It does not stop, because the nodes can change. For example, a kind node with 16 cpus cannot hold a Size of memory 64Gi and cpu 32:

text
…
sandbox nodes: jr2-control-plane
warning: no Sandbox node right now has the allocatable for a Size of memory 64Gi, cpu 32 (workflow "e2e") — its Sandbox stays Pending until one does
…

Where the Size comes from ​

The Size is a fact of the Machine. A run cannot change it. jr2 uses the first of these that states a Size:

  1. customize(machine, { resources }), in the Instance that registers the Machine.
  2. resources on the workspace().
  3. sandbox.resources in jr2.config.ts, such as sandbox: { resources: { limits: { memory: "3Gi" } } }.
  4. The kit default.

These rules apply to the order:

  • customize() writes over the Size of the workspace() one field at a time. Thus customize(machine, { resources: { limits: { memory: "6Gi" } } }) keeps the cpu of the Machine.
  • sandbox.resources applies only to a Workspace that states no Size. It never changes a Size that a Machine states.
  • A field that the chosen Size does not state comes from the kit default. For example, under resources: { limits: { memory: "4Gi" } }, the cpu is 1, also when sandbox.resources states a cpu.

Use customize() to change the Size of a Machine from a package, such as task from @jr2/machines. See Bind a shipped Machine's Open parts.

How jr2 divides the Size in the pod ​

The pod has the Size as one limit. Inside the pod, jr2 divides it among the containers:

ContainerIts part
The CustodianA fixed part: memory 64Mi, cpu 50m.
The User ContainerThe part that user.resources gives. With no part, it shares the rest.
The Sandbox EffectorNo part of its own. It shares the rest.
The harness containerThe rest, as its own limit. The Agent's processes and the commands run here.

jr2 up refuses a Size that leaves the harness container less than 256Mi of memory, or no cpu.

To give the User Container a part, write user as an object: user: { image: import.meta.resolve("./shell"), resources: { limits: { memory: "256Mi", cpu: "250m" } } }. The part comes out of the part of the harness container. It is not added to the Size.

The Size holds only memory and cpu. jr2 also sets these, and you cannot change them:

  • /dev/shm in the harness container is 25% of the container's memory, and not less than 64Mi. It counts in that memory. To get more, state a larger Size.
  • /work has a disk request of 10Gi, with no limit. The disk is not part of the Size.

Use the cpus ​

jr2 enforces the cpu limit. Many tools read the limit, such as Node's availableParallelism(), Go, cargo and the JVM. For tools that do not read it, these environment variables in the harness container hold its cpu limit, rounded up to a whole number: JR2_CPUS, OMP_NUM_THREADS, PYTHON_CPU_COUNT and GOMAXPROCS.

Node's os.cpus() gives the cpus of the node, not the limit. Playwright uses it to choose its number of workers. Thus the example gives --workers=$JR2_CPUS. A command sees these variables.

When the memory is not enough ​

The Harness watches the memory of its container. Near the limit, it stops all of the Agent's processes. Then the Agent's bash call returns this text, and the Agent can try again with less:

text
killed: memory limit (peak <peak> of <limit>); use fewer workers or a larger Size

The Harness also watches when no bash call runs. A process that a Turn left running, such as a dev server, can pass the limit while the workflow waits at a Gate. Then no bash call returns the text. jr2 tells the next Agent that works in the Workspace: the next Turn, or an open Turn that jr2 asks again.

The Harness can miss a fast increase. Then the kernel stops the whole container, and the Turn faults. The reason starts with memory limit:

text
memory limit (OOMKilled, limit <memory>): the Workspace's processes passed the Harness container's memory limit and the kernel killed the container, the conversation with it. Use fewer workers, or give the Workspace a larger Size (ADR-0060, ADR-0061).

The next Agent in the Workspace gets a notice of the kill. When you see either text again and again, state a larger Size.

Rules ​

  • workspace(), customize() and the config refuse a key other than limits, where you write it. <where> is workspace(): resources, customize(): resources, <file> at sandbox.resources, workspace() user.resources or customize() user.resources: <where>.requests is not accepted — jr2 reserves the whole Size, so every request equals its limit — a Size is { limits: { memory?, cpu? } } and nothing else — Agent Substrate's template size (ADR-0060) and <where>.<key> is not accepted — a Size is { limits: { memory?, cpu? } } and nothing else — Agent Substrate's template size (ADR-0060).
  • They refuse a key in limits other than memory and cpu: <where>.limits.<key> is not accepted — a Size is { limits: { memory?, cpu? } } and nothing else — Agent Substrate's template size (ADR-0060).
  • They refuse a value that is not a quantity more than zero: <where>.limits.memory: <value> is not a positive quantity (e.g. "2Gi", "512Mi") and <where>.limits.cpu: <value> is not a positive quantity (e.g. "2", "1500m").
  • They refuse a Size, or a limits, that is not an object: <where>: expected an object (got <value>) — a Size is { limits: { memory?, cpu? } } and nothing else — Agent Substrate's template size (ADR-0060) and the same text for <where>.limits.
  • customize() refuses resources for a Machine that has no workspace(): customize(): `resources` say what a SANDBOX is made of, how big it is and which Repos it attaches, and machine "<id>" composes none — only a workspace() wrapper carries those seats, that Size and those slots (ADR-0049, ADR-0037, ADR-0051, ADR-0060).
  • jr2 up refuses a Size that leaves the harness container less than 256Mi of memory. The text names each number and where it came from. The part − the User Container's <memory> (its user.resources.limits.memory) is there only when user.resources gives a memory: refusing: workflow "<workflow>": the Size leaves the Harness container <memory> memory, below its 256Mi floor: memory <size> (<source>) − the Custodian's 64Mi − the User Container's <memory> (its user.resources.limits.memory) — raise the Size's limits.memory, or shrink the User Container's split (ADR-0060). The text can also name the Machine that states the Size, and it can end with the customize() line that changes the Size. For cpu, the text holds the Size leaves the Harness container no cpu:.
  • jr2 up refuses a cluster that drops pod-level resources: refusing: <context> (Kubernetes <version>) drops pod-level resources — the PodLevelResources feature gate is off, and a Workspace's Size cannot hold (ADR-0060). Enable the gate on the API server and kubelets. The namespace was created; nothing else was applied.

The docs of Kit version 0.5.1. Each page is also at its .md URL.