Fix a failed converge

Find the cause of a failed jr2 up in its output, and fix it.

The examples on this page use the Get started names: the Instance my-instance and the kubectl context kind-jr2.

Read the output

jr2 up prints one line for each step. When it refuses, the line starts with refusing:. When a step fails, the line starts with error:. In the two cases, jr2 up exits with the code 1, and the line tells you what to change. Many lines end with the ADR that gives the rule.

For example, change input.message to input.mesage in the context line of workflows/ping.ts. Then run jr2 up:

jr2 up

…
jr2 up — instance "my-instance" → context kind-jr2 / namespace my-instance
typecheck: tsc --noEmit (the instance's own compiler)
refusing: this instance does not typecheck — nothing was built and nothing was applied
  workflows/ping.ts(45,45): error TS2551: Property 'mesage' does not exist on type 'Input'. Did you mean 'message'?

The first line (…) is the .env line, when .env sets a variable that your shell does not set. The refusal comes before jr2 up reads the cluster, so the cluster did not change. Each line after it is a line from the TypeScript compiler of the Instance. To see the same errors without jr2 up, run jr2 check.

Correct the typo. Then run jr2 up again. You can run jr2 up again after each fix. It does not build an image again when the inputs of the image did not change.

The checks of the config, the compiler and the Machines come before jr2 up reads the cluster. jr2 check runs these checks with the same code and no cluster, and gives the same refusing: and error: lines. Thus you, or an Assistant that cannot run jr2 up, can fix them before a converge. The checks of the context, the API server, the Kubernetes version and the namespace come before jr2 up changes the cluster. Thus these refusals change nothing in the cluster. The pod-level resources refusal comes after jr2 up makes the namespace. Other errors can come after jr2 up applied some parts. When you fix the cause, run jr2 up again. A converge that fails removes no image. Converge gives the order of the steps.

Fix a rollout that does not complete

After jr2 up applies each part, it waits for the rollout of that part. When a rollout does not complete in time, jr2 up reads the pods and prints what they say. In this example, the kitRegistry mirror does not hold the image of the Orchestrator:

…
orchestrator: applying (image <registry>/jr2-orchestrator:<version>)
orchestrator: waiting for rollout
error: jr2-orchestrator: rollout did not complete in namespace my-instance — the pods say:

  pod jr2-orchestrator-5cd48ff77f-m7qhw (Pending) on node jr2-control-plane (amd64)
    container orchestrator: ImagePullBackOff
      image <registry>/jr2-orchestrator:<version>
      Back-off pulling image "<registry>/jr2-orchestrator:<version>": ErrImagePull: rpc error: code = NotFound desc = failed to pull and unpack image "<registry>/jr2-orchestrator:<version>": failed to resolve reference "<registry>/jr2-orchestrator:<version>": <registry>/jr2-orchestrator:<version>: not found
    event Warning Failed: Error: ImagePullBackOff
    event Normal Pulling: Pulling image "<registry>/jr2-orchestrator:<version>"
    event Warning Failed: Failed to pull image "<registry>/jr2-orchestrator:<version>": rpc error: code = NotFound desc = failed to pull and unpack image "<registry>/jr2-orchestrator:<version>": failed to resolve reference "<registry>/jr2-orchestrator:<version>": <registry>/jr2-orchestrator:<version>: not found
    event Warning Failed: Error: ErrImagePull

diagnosis: the kubelet could not pull <registry>/jr2-orchestrator:<version>.
  it resolves to <registry>/jr2-orchestrator:<version>
  check the push landed there and that this cluster may pull from it (credentials, network, mirror).

Command failed: kubectl --namespace my-instance rollout status deployment/jr2-orchestrator --timeout=180s
error: timed out waiting for the condition

<registry> is the value of kitRegistry, and <version> is the Kit version. The pod name of your run is different. For each pod, jr2 up prints its phase, its node, the state of each container and the last events. When a container has a log, the last lines of the log come after the events. The output ends with the error of kubectl. Here, jr2 up waited 180 seconds for the rollout.

The first line names the part: jr2-orchestrator, jr2-controller-manager (the operator, in jr2-system), jr2-repo-cache, jr2-instance-harness or jr2-instance-effector. Read the reason, the events and the logs of the pod. When jr2 up knows the cause, it adds a diagnosis: line. The diagnosis explains only the evidence above it.

The diagnosis starts with	The cause and the fix
diagnosis: <pod> carries an image built for another platform	The image is for a different CPU architecture than the node. Run jr2 up --force to build the image again. Or set platforms.
diagnosis: the kubelet could not pull <image>.	The node cannot get the image. The next line gives the full address. On kind, for an image that jr2 built, run jr2 up --force to build the image and load it again. With a registry, check that the push arrived and that the cluster can pull from the registry.
diagnosis: <pod> starts and exits	The container stops after it starts. The log lines above give the reason. When there are no log lines, run the kubectl -n <namespace> logs <pod> --previous command that the diagnosis gives, after the pod starts again.
diagnosis: the pod's env references <kind> "<name>"	The pod names a Secret or a ConfigMap that is not in the namespace. Make it. Then run jr2 up again.
diagnosis: the container's configuration cannot be built	The pod names an object that the namespace does not have. The message above names the object. Make it. Then run jr2 up again.

A Kit image that the cluster cannot pull is at ghcr.io/snapwich, or at the kitRegistry mirror. See Mirror the Kit images.

Rules

The list gives the refusals and the errors in the order that jr2 up meets them. Each refusing: line and each error: line makes jr2 up exit with the code 1. A warning: line does not stop the converge. Give the cluster git credentials gives the refusals of git.credentials and of the git ssh key. Use a hosted provider gives the other refusals of secrets and harness.env. Run a command or call a URL with an Effect gives the refusals of an Effect secret. Write a Workflow gives the refusals of a Workflow module. Push and sign a branch gives the refusals of a push and of the Effector image. Use a local model gives the refusals of caBundle.

• jr2 up runs only in an Instance folder: error: not inside a jr2 instance — no jr2.config.ts found walking up from cwd

• The Instance and the jr2 CLI must use the same Kit version. If not, jr2 up prints error: Kit version mismatch — an Instance has one (ADR-0056):, then three lines. The third line is one of these fixes:

  • install the Instance's dependencies (<folder>/package.json pins "@jr2/orchestrator" and "@jr2/cli" at one exact version). Run npm install in the Instance folder.
  • pin "@jr2/orchestrator" and "@jr2/cli" at ONE exact version in <folder>/package.json and reinstall. Give the two packages the same version. Then run npm install.
  • add "@jr2/cli": "<version>" to devDependencies in <folder>/package.json and reinstall — the Instance's own jr2 then runs

• jr2.config.ts must have a default export: error: <folder>/jr2.config.ts has no default export (use `export default defineConfig({…})`)

• jr2 up checks the shape of git, secrets, harness, sandbox and priorityClasses when it loads jr2.config.ts. The error names the field, and then the problem: error: <folder>/jr2.config.ts at <field>: <problem>. See jr2.config.ts.

• jr2 up needs a kubectl context: error: no kube context — `kind create cluster` (or point kubectl at one), then re-run `jr2 up`. See Install.

• The Instance must have the tsconfig.json that jr2 init writes: error: this instance has no tsconfig.json — `jr2 init` scaffolds one that extends `@jr2/orchestrator/tsconfig.instance.json`, and `jr2 up` typechecks the folder before it builds anything

• The Instance must install its TypeScript compiler. Run npm install. The refusal starts with: error: this instance's TypeScript compiler is missing — install its dependencies (npm, pnpm, or bun), then re-run `jr2 up`.

• The Instance must typecheck: refusing: this instance does not typecheck — nothing was built and nothing was applied

• A registered Machine with a door must be made by jr2Setup, because the setup() of xstate does not read the input of createMachine. See Write a Workflow: refusing: workflow "<workflow>": machine "<id>" has a zod `input` on createMachine, but jr2Setup did not make this Machine, so no door reads that input: a run starts with any input, and the Console start form is only a field for JSON — make the Machine with jr2Setup in place of setup() or createMachine() (leave out `events` if it takes no workflow event) (ADR-0033)

• A registered Machine must have no Open part. The refusal gives the customize() line that binds the part. Put it in workflows/<workflow>.ts. Replace <import> with the name that you import the Machine as. Give the values in the <>. See Bind a shipped Machine's Open parts.

  • refusing: workflow "<workflow>" leaves Repo Slot "<slot>" open — bind it where the Machine is registered: export const machine = customize(<import>, { repos: { <slot>: "<url>" } }) (ADR-0051)
  • refusing: workflow "<workflow>" leaves its Repo Slots open — bind it where the Machine is registered: export const machine = customize(<import>, { repos: { <slot>: "<url>" } }) (ADR-0051)
  • refusing: workflow "<workflow>" leaves Agent "<agent>" open — bind it where the Machine is registered: export const machine = customize(<import>, { agents: { <agent>: { model: "<provider>/<model>" } } }) (ADR-0054)
  • An Effect. <parts> is url, argv, secret, url and secret or argv and secret. The binding object has one sample for each part, from url: "https://<host>/<path>", argv: ["<program>", "<arg>"] and secret: "<name>". For example: refusing: workflow "<workflow>" leaves the url and secret of Effect "<effect>" open — bind it where the Machine is registered: export const machine = customize(<import>, { effects: { <effect>: { url: "https://<host>/<path>", secret: "<name>" } } }) (ADR-0054)
  • When the part is on a Machine that a different Machine invokes, the refusal adds (on the Machine composed as "<slot>") after open. Its customize() line binds the part through actors, for example customize(<import>, { actors: { <slot>: { repos: { <repo slot>: "<url>" } } } }). At each deeper level, the parenthetical adds → "<slot>", and the customize() line adds one more actors.
  • When the Machine is invoked inline, no customize() line can reach it: refusing: workflow "<workflow>" leaves <part> open on a Machine invoked inline — no customize() reaches an actor without a slot key; declare that Machine under setup({ actors }) and bind it there (ADR-0049, <ADR>)

• The Size of each Workspace must leave the Harness container enough memory and cpu. The refusal starts with refusing: workflow "<workflow>": the Size leaves the Harness container. It gives the numbers that it subtracts and where each one comes from, and the field to make larger. See Size a Workspace. It ends with this text: ; a composer retunes it where the Machine is registered: export const machine = customize(<import>, { resources: { limits: { memory: "<memory>" } } }) When the Machine is invoked inline, the refusal has no customize() line.

  • When the Size is on a Machine that a different Machine invokes, the refusal starts with refusing: workflow "<workflow>" (on the Machine composed as "<slot>"): the Size leaves the Harness container. Its customize() line sets the memory of the Size through actors: customize(<import>, { actors: { <slot>: { resources: { limits: { memory: "<memory>" } } } } }). At each deeper level, the parenthetical adds → "<slot>", and the customize() line adds one more actors.

• jr2 up makes the Assembly before it reads the cluster. The Assembly must not hold a second copy of a Kit package, such as @jr2/orchestrator, xstate or zod: error: cannot stage the Assembly: it would carry a second copy of a Kit package (ADR-0069). Each line after it names one copy: <path> is a second copy of <package>, nested by <package>: the Orchestrator's image ships <package> for @jr2/orchestrator <version>, and an Assembly that carries its own copy loads Machines whose Menus are empty (ADR-0054) — align that dependency with the <package> @jr2/orchestrator uses, or remove it, and reinstall. Give the dependency that nests the copy a version that uses the same <package> as @jr2/orchestrator. Then run npm install.

• A symlink in the Assembly must stay in the Assembly: error: cannot seal the Assembly: the symlink <path> points outside the Assembly ("<target>") — an Assembly carries only what it holds; <fix>. The fix is install that dependency as a copy, not a link for a link in node_modules, and make the link relative, with a target inside the Instance for a different link.

• A file in the Assembly must not hold the path where jr2 up makes the Assembly. The refusal starts with error: cannot seal the Assembly: <file> holds the staging path (<path>). A postinstall step of a dependency is the usual cause.

• The Assembly must hold each Workflow of the folder. The refusal starts with refusing: the staged Assembly does not hold the Workflows this folder holds (ADR-0069), and names each file. It ends with — the Assembly holds what the Instance's install keeps; check the `files:` list in package.json.

• jr2 up must reach the API server of the context: refusing: cannot reach the API server of <context> (<error>). Nothing was applied. For example, with --context nope, the error is error: context "nope" does not exist. Check the context with kubectl config current-context. Check that the cluster runs.

• The cluster must run Kubernetes 1.35 or later: refusing: <context> runs Kubernetes <version>; jr2 needs Kubernetes 1.35 or newer (every Sandbox mounts jr2's runtime as an image volume, ADR-0037). Nothing was applied. For kind, install a newer kind release. Then make the cluster again.

• The namespace must not belong to a different Instance: refusing: namespace "<namespace>" on <context> belongs to another instance ("<owner>"), then pick a different namespace (-n) or context (--context). Set name in jr2.config.ts, or add -n <namespace>.

• Before jr2 up uses a namespace for the first time, it asks: first contact: deploy instance "<instance>" to context <context> (create namespace "<namespace>")? [y/N]. For a namespace that exists but holds no Instance, it asks to adopt existing namespace "<namespace>". When you do not type y, it prints aborted — nothing was changed. When no terminal is attached, it prints <question> — not a tty; pass --yes to proceed non-interactively first. In CI, add --yes.

• The cluster must keep pod-level resources. jr2 up makes the namespace, and then it stops: refusing: <context> (Kubernetes <version>) drops pod-level resources — the PodLevelResources feature gate is off, and a Workspace's Size cannot hold (ADR-0060). Enable the gate on the API server and kubelets. The namespace was created; nothing else was applied.

• When jr2 up cannot confirm pod-level resources, it warns and continues: warning: could not confirm pod-level resources on <context>: <error>

• jr2 builds only for the platforms that the Kit releases images for: error: <what> — the kit releases images for <kit platforms> and nothing else, so an image built for anything else could not be joined by a Kit image in the same pod. Set `platforms` in jr2.config.ts (docker platform strings, e.g. "linux/arm64") to name the set to build for. <what> is `platforms` names <platforms>, or this cluster's schedulable nodes report <architectures>. Set platforms to platforms from the list after the kit releases images for.

• When no node can hold a Sandbox, jr2 up warns and continues: warning: no Sandbox node right now — a Sandbox stays Pending until one appears:. Each line after it gives a node, and why a Sandbox cannot use it. A Sandbox node can appear later. To admit a node with a taint or a label, set sandbox in jr2.config.ts.

• When no Sandbox node has the memory and cpu for a Size, jr2 up warns and continues: warning: no Sandbox node right now has the allocatable for a Size of <size> (workflow "<workflow>") — its Sandbox stays Pending until one does

• Images for two or more platforms need a registry: error: this cluster needs a <platforms> build, which docker buildx delivers by PUSHING a manifest list (kind load cannot carry one) — set `registry` in jr2.config.ts, or name a single platform with `platforms`

• A cluster that is not kind needs a registry: error: context <context> is not a kind cluster and no `registry` is configured — set `registry` in jr2.config.ts (from env) so the image can be pushed (ADR-0019). jr2 knows a kind cluster by its context name, which starts with kind-. See Move from kind to a shared cluster.

• The Instance folder must hold the lockfile of one package manager. <folder> is the path of the Instance folder:

  • Lockfiles of two managers: error: <folder> holds lockfiles from more than one package manager (<manager>: <lockfile>; <manager>: <lockfile>) — delete the stale one, so the bundle installs the dependency graph the instance really uses. Delete the lockfile of the manager that you do not use.
  • A yarn.lock: error: <folder> has a yarn.lock, and yarn is not supported — use npm, pnpm, or bun. Install with npm, pnpm or bun, and delete yarn.lock.
  • No lockfile: error: <folder> has no lockfile — jr2 installs the instance's dependencies from npm (package-lock.json), pnpm (pnpm-lock.yaml), bun (bun.lock or bun.lockb); run your package manager's install and commit the lockfile it writes. Run npm install, and commit package-lock.json.

• To build for a different CPU architecture, docker needs qemu emulation. The refusal gives the command that installs it: error: this host cannot build for <missing platforms>: a foreign architecture's RUN steps need qemu binfmt emulation, and none is registered (this builder targets <builder platforms>). Install it, then re-run:, then docker run --privileged --rm tonistiigi/binfmt --install <arch>

• The folder of each Sandbox Image that a workspace() names must exist: error: the Sandbox Image context <folder> cannot be read (<code>) — a `workspace()` names its image with `import.meta.resolve("./<dir>")`, so that directory must exist beside the Machine's own module (ADR-0037/0049). See Add tools to the Sandbox.

• Each Secret that harness.envFrom names must exist: error: harness.envFrom references Secret "<name>", which does not exist in namespace "<namespace>" — create it first: kubectl -n <namespace> create secret generic <name> --from-literal=KEY=...

• Each secrets entry that a pod or an Effect uses needs one value. A variable that is not set in .env gives no value: error: secrets[<i>] needs exactly one of value or valueFrom — <NAME> has neither, which is what an unset .env variable reads as: set <NAME> in .env, or reference a Secret with valueFrom: { secretKeyRef: { name, key } } (ADR-0059). Add the variable to .env. An entry with the two gives not both in place of the text after the dash.

• A value must not be empty: error: secrets[<i>].value is empty — set <NAME> in .env (ADR-0059)

• A value must hold only printable ASCII, with no space at its start or end: error: secrets[<i>].value holds a character a header cannot carry (outside printable ASCII) or leading or trailing space — a pasted newline, most likely; the value itself is not shown (ADR-0059). Remove the newline or the space from the value in .env.

• The Secret that a valueFrom names must exist, with its key: error: secrets[<i>].valueFrom names Secret "<name>", which does not exist in the instance's namespace — create it first: kubectl create secret generic <name> --from-literal=<key>=... (ADR-0019, ADR-0059), and error: secrets[<i>].valueFrom names key "<key>" of Secret "<name>", which has no such key (ADR-0059)

• When harness.provider is set and an Agent uses its models, a pod in the cluster asks the provider for one tool call for each model. When a call fails: error: provider preflight failed for model "<model>" against <host> (from a pod on <where>, as a Turn calls it): <error>. Four lines follow, with the usual causes. A fifth line follows when apiKey is set: the key may be wrong.

    - localhost never works from a pod; use a LAN address
    - the model id must be exactly what the endpoint serves (vLLM: GET /v1/models)
    - vLLM needs --enable-auto-tool-choice and a matching --tool-call-parser
    - stop reason "length" means the model ran out of tokens before the call (a thinking model reasons first)

  When the endpoint refused the key (a 401 or a 403), the endpoint answered. So three of the four usual lines are left out: the line about localhost, the line about vLLM tool calls, and the line about stop reason "length". Two lines follow: the line about the key, and the line about the model id. When apiKey is set, the line about the key says that the held secret may be wrong. When apiKey is not set, it says that the endpoint wants a key.

    - the held secret <NAME> may be wrong (harness.provider.apiKey)
    - the model id must be exactly what the endpoint serves (vLLM: GET /v1/models)

  See Use a local model.

• When an Agent uses a model of pi's catalog (for example anthropic/claude-haiku-5-5), a pod in the cluster asks the provider for one tool call for each model. The pod has the placement of the Harness pods that run the model, and a Custodian that holds the secrets of those pods. When a call fails: error: provider preflight failed for model "<model>" toward <host> (from a pod on <where>, through a Custodian, as a Turn calls it): <error>. When the pods hold no secret, the error says direct in place of through a Custodian. Each failed model has its own error. Lines with the usual causes follow:

    - the pods may have no egress to <host> (a NetworkPolicy, a firewall, no route out of the cluster)
    - the held secret <NAME> may be empty, wrong, or without access to this model (`secrets` in jr2.config.ts)
    - a private CA on that path must be in harness.caBundle

  When the provider refused the key (a 401 or a 403), the line about the held secret comes first, and the line about egress is left out, because the provider answered. When no Harness pod holds a key of the provider, the line about the key says which key to add to secrets. With a harness.catalog gateway for the provider, a line also says that the gateway may not answer, or may not serve the model. A 503 is an error: a gateway with no backend, or a server that still loads its model, answers 503.

  When the check pod cannot start, the error says why:

  • a node cannot pull an image;
  • a Secret or a key that harness.envFrom or a harness.env valueFrom names is missing;
  • a Secret or a key that a secrets valueFrom names is missing (the volume does not mount);
  • the Custodian refused a held value;
  • a container ran out of memory.

  While the check pod waits to start, jr2 up writes why, for example provider: the check pod is not running yet — Unschedulable: …. A provider that is busy after the retries (a 429, a 529, an overload) is a warning, not an error. A model that answers the request of a Turn in text, not with a tool call, is a warning too. When a model of the custom provider, or behind a harness.catalog gateway, gives no answer in time (no answer in 120 s), the error adds that it can be a model that loads on first use: run jr2 up again once it has loaded. A provider that answers 429 and then never answers a retry is a warning; for Gemini and Vertex it is an error, because their tries are not seen. When the check pod stops before the end, the error says which model the check was on, and not checked for each model that it never reached.

  The check has a limit. The check pod has labels of its own, so a NetworkPolicy that selects the Harness pods by label does not select the check pod. When such a policy closes the egress of the Harness pods, the check can pass while a Turn fails.

  A check can fail when the model works, for example a model that loads for longer than the check waits. When you know that the models work, run jr2 up --skip-provider-preflight. No check pod runs, and the converge goes on. jr2 up writes one line: warning: provider: preflight skipped (--skip-provider-preflight) — these models were not checked: <model>, …. A model that does not work then fails in a run, as agent.fault, and not at the converge. Remove the flag when the check works again.

• Each rollout must complete. The error starts with error: <part>: rollout did not complete in namespace <namespace> — the pods say:. See Fix a rollout that does not complete. Every rollout comes after the provider check, the rollout of the operator (jr2-controller-manager) too.

• When the rollout made no pod, the error says: no pod matches <selector> — the ReplicaSet made none (check quota, node taints, and the selector). For the Repo cache and the Instance Harness, it names the DaemonSet or the StatefulSet.

• After each rollout, the pods must run the image that jr2 up expected. <layer> is operator, orchestrator, repo cache, instance harness or instance effector: error: <layer>: rollout reported success, but the running pod carries <images> — expected <image>. The cluster is serving code this converge did not deploy; `kubectl -n <namespace> rollout restart deploy` and re-run `jr2 up` to resolve it.

• When the rollout completes but no pod matches, jr2 up cannot check the image. It continues: <layer>: WARNING — rollout succeeded but no pod matched <selector> (nothing verified)

• Before the publish, the Orchestrator must read the image map that jr2 up wrote. When it does not read it in time, jr2 up publishes nothing: refusing: after <seconds>s the Orchestrator still reads an older <maps> from its mount — the kubelet has not refreshed it; nothing was published, and the previous Assembly is still current. Run `jr2 up` again

• The Orchestrator checks the Assembly again before it makes it current. It loads each Workflow and jr2.config.ts. It refuses an Open part that nobody bound, a secret that the config does not declare, a Kit version that is not its own, and a zod input that no door reads (a Machine that jr2Setup did not make). The refusal gives the status code and the reason, then one line for each problem: refusing: the Orchestrator refused Assembly <hash> (HTTP <status>): <error> — the previous Assembly is still current. Fix each problem. Then run jr2 up again.

Read next

• Converge
• ADR-0046: why a failed rollout prints the evidence of its pods.
• ADR-0050: why jr2 up typechecks the Instance first.
• ADR-0054: why jr2 up refuses an Open part.
• ADR-0056: why an Instance has one Kit version.
• ADR-0069: why jr2 up makes an Assembly and publishes it.
• ADR-0019: the current context and the namespace.
• ADR-0045: how jr2 up chooses the platforms.
• jr2 up, jr2.config.ts
• Clean up