The Kubernetes API client

Everything chant does against a live cluster — chant lifecycle diff --live, chant lifecycle plan, chant import --from, the kubectlApply and waitForReady Op activities, and behold’s overlay — goes through a typed API client rather than a kubectl subprocess.

What that buys

Coverage. A declared entity type becomes an API address through a table generated by the same codegen pass that emits the declarable classes — 180-odd types, every bundled CRD included. The address is then confirmed against the cluster’s own API discovery, which is the only thing that knows the plural and the scope for the version that cluster serves. Before this, the lexicon carried a hand-written twenty-entry map, and every type outside it — every CRD, a HorizontalPodAutoscaler, a PodDisruptionBudget — was reported as unreadable.

Concurrency. Reads run through a bounded pool (eight in flight by default). A hundred declared entities are a hundred concurrent HTTP GETs sharing one connection and one cached credential, not a hundred serial process spawns.

Typed failures. The API server sends a Status object with a numeric code and a reason enum. Chant reads those fields instead of matching English on stderr, so 403 Forbidden becomes “not observed, no credentials” and 404 NotFound becomes a genuine absence, by construction rather than by regex.

No kubectl in the image. A process running kubectlApply or waitForReady needs no kubectl binary. (The activity is still called kubectlApply: core’s activity registry keys the activity map by export name and an Op step resolves its fn string against it, so renaming the export would break every Op that names the step.)

Installation

The client is @intentius/chant-k8s-client, an optional dependency of @intentius/chant-lexicon-k8s. A normal npm install brings it in; npm install --omit=optional does not.

Two things follow from it being separate:

• chant build never resolves it. No module on the build path imports it, statically or otherwise — the lexicon reaches it through a dynamic import from modules only the observation and Op paths load. This is the first chant code that holds live cluster credentials, so that boundary is structural rather than a lint rule. examples/k8s-client-boundary.test.ts walks the static import graph and builds the whole k8s corpus to keep it that way.
• If it is missing, chant says so. Observation reports every declared entity as not-observed with an install hint, rather than reporting an empty cluster — which would classify as a proposal to create everything.

Which cluster, and with whose credentials

The environment→cluster binding is unchanged from k8s.profiles.<env>.context:

chant.config.ts

import type { ChantConfig } from "@intentius/chant/config";
// Brings the `k8s` key into ChantConfig (chant #1344).
import "@intentius/chant-lexicon-k8s";

export default {
  lexicons: ["k8s"],
  k8s: {
    profiles: {
      prod: { context: "prod-eks" },
      staging: { context: "staging-eks" },
    },
  },
} satisfies ChantConfig;

A declared binding is passed explicitly on every request. If the kubeconfig’s own current-context disagrees with it, chant refuses before touching a single resource and names both contexts — reading the wrong cluster would report every declared resource as missing, which is a confident and wrong list of creates. With no binding, the kubeconfig’s current-context is used and a warning says so.

Exec credential plugins

On EKS, AKS and GKE, kubeconfig authentication is itself a subprocess: aws eks get-token, kubelogin, gke-gcloud-auth-plugin. So on the clusters most people actually run, the client still spawns a process — once for a token instead of once per resource read. That is the win, but it is worth stating rather than implying, because an exec plugin runs an arbitrary binary named in a file chant did not write.

Three things follow:

• The plugin must be allowlisted. Those three plus kubectl are allowed by default. Anything else is refused by name, before it runs.
• Tokens are cached for their stated lifetime. One KubeConfig is built per session and reused, so a 200-entity observation invokes the plugin once, not 200 times.
• The credential path is recorded, so the provenance of an observation is legible: a read authorized by aws eks get-token and one authorized by a static service-account token are not equally trustworthy inputs to a drift report.

To allow another plugin, name it — this replaces the default list:

k8s: {
  profiles: { prod: { context: "prod-eks" } },
  execCredentialPlugins: ["aws", "my-org-oidc-helper"],
}

Caution

The allowlist matches the command name, not its full path: aws, /usr/local/bin/aws and aws.exe are the same entry. It expresses which tool may run, not which file.

Applying

kubectlApply performs a server-side apply, rather than the client-side three-way merge kubectl apply does by default. Server-side apply is where Kubernetes itself has gone, it removes the last-applied-configuration annotation from the story, and it is what makes field ownership something the API server tracks rather than something chant has to infer.

ApplyOp({ target: "kubectl" }) goes through the same path. The kubectl branch of nativeApply used to shell kubectl apply -f; it now dispatches into this lexicon, where the apply is server-side and the prune runs against the typed client. nativeApply, the dispatcher itself, stays in core, because “which mechanism applies this target” is not Kubernetes knowledge; applying to Kubernetes is.

chant’s field manager

The manager chant applies as is derived from the project’s ownership stack:

ownership.stack	field manager
unset (or ownership.enabled: false)	chant
web	chant:web

This is the same identity the chant.intentius.io/stack label already carries. The label answers a binary, whole-object question — is this chant’s? — and server-side apply answers the sub-object version of it, per field, from the API server. One identity, two granularities.

ownership.env is deliberately not part of the manager. Two environments of one stack only ever touch the same object if they share a namespace and a name, and at that point they are fighting over it; an env-qualified manager would let each own half of it without either noticing.

To see what chant owns on a live object:

$ kubectl get deploy/web -n prod -o jsonpath='{.metadata.managedFields}' | jq

The entry whose manager is chant:<stack> and whose operation is Apply lists exactly the fields the manifest declared.

Conflicts

If another manager already owns a field the apply would set, the API server refuses with a 409 and says precisely which fields and who holds them. chant presents that rather than passing the raw status through:

k8s: server-side apply of apps/v1 Deployment prod/web was refused — 2 fields are
owned by another field manager.

  "kubectl-client-side-apply" owns:
    .spec.replicas
    .spec.template.spec.containers[name="web"].image

chant applied as field manager "chant:web". Taking these fields means the managers above
stop owning them, and will contest them again on their next apply.

chant does not force this for you. Either:
  - remove the contested fields from your chant source, leaving them to their current owner; or
  - re-run this apply with force-conflicts on, deliberately (the `force: true` activity argument,
    or `forceConflicts: true` on ApplyOp), which transfers ownership to chant.

The error is a FieldManagerConflictError — still a K8sApiError, so anything catching 409s keeps working — carrying conflicts, byManager, managers, fields and the fieldManager chant applied as.

Caution

chant never forces a conflict on its own. Taking a field from another manager decides who owns production, and a tool that decides it silently is how “it works on my cluster” happens. The opt-in exists and is never a default:

export const { op } = ApplyOp({
  name: "prod-apply",
  env: "prod",
  target: "kubectl",
  forceConflicts: true, // adopt objects a previous `kubectl apply` created
});

Deletes

ApplyOp’s delete: "owned-only" (and "gated") prunes chant-owned objects the manifest no longer declares, which is what kubectl apply --prune --selector app.kubernetes.io/managed-by=chant used to do. It keeps both of that command’s safety properties and narrows one:

• Marker-scoped. The label selector goes to the server, and the marker is re-checked on every object before it is deleted. An object chant never stamped is never a candidate. With ownership.stack set, the selector also pins the stack, so one stack never prunes another’s.
• Namespace-scoped to the apply. Namespaced kinds are swept only in the namespaces the apply set touched. A manifest that mentions prod never reaches into staging.
• Kind-scoped to the union of the kinds in the apply set and chant’s default sweep set. The union is what lets a kind deleted from source entirely still be pruned — it appears nowhere in the apply set.

Generated-once secrets and the retained outcome

The ensure-secret capability and the ensureSecret(...) op step (#1829) have a Kubernetes store: k8sSecretStore(namespace, client?) from @intentius/chant-lexicon-k8s/secret-store, plus k8sEnsureSecretCapability(...) to build the registered capability over it. The contract is read-then-write — if the Secret exists, its key NAMES and metadata are verified and nothing is written; if absent, one value per declared key is minted at the write and never travels back. describe projects key names and labels/annotations only; no code path returns, logs, or compares a data value (#1365 decision 6).

A minted Secret carries the ownership marker plus chant.intentius.io/generated-once: "true", and that label puts it outside every prunable set:

• The owned-only prune above skips it and reports it under retained in the apply result, with a console line saying so.
• chant lifecycle teardown <env> --yes reports it as a retained row — the env sweep deletes everything else and says loudly that the environment is not clean while the Secret exists.

Both are deliberate: the stored bytes are the only copy of material chant never held, so no sweep may destroy them. Deleting a generated-once Secret is an explicit act — kubectl delete secret <name> today, a gated op later — never a side effect of “no longer declared”.

Drift keeps the same line. Deep observation masks every data.*/stringData.* value on a K8s::Core::Secret on both sides of the diff, so chant lifecycle diff --live classifies presence and key-name drift (a key injected out of band, a declared key gone) without ever holding a value — a value-only change is invisible by design.

managedFields as a read surface

@intentius/chant-k8s-client exports the decoding primitives, since fieldsV1 is not a shape a caller should have to parse:

import { chantOwnedFields, fieldOwners, fieldSetsOf } from "@intentius/chant-k8s-client";

chantOwnedFields(live);        // ['.spec', '.spec.replicas', …] — every chant manager
fieldOwners(live).get(".spec.replicas");  // ['chant:web', 'kubectl-client-side-apply']
fieldSetsOf(live);             // one decoded entry per manager, subresources included

Paths render the way sigs.k8s.io/structured-merge-diff does — .spec.template.spec.containers[name="web"].image — which is the same syntax a conflict cause uses, so the two are comparable as strings. Subresource entries (a controller’s status writes) are excluded by default.

What ownership means for drift

chant lifecycle diff --live reads managedFields on every declared object, but ownership decides who gets named on a drift line, not whether the line is reported. The rule:

Live field is	Reported as
Declared in source, whoever owns it live	changed (or nothing, when the values agree). A kubectl scale that took spec.replicas away from chant still shows, with owner: kubectl-client-side-apply.
Not declared, owned by another manager	undeclared, with the owner named — kubectl label deploy web team=platform shows as metadata.labels.team owned by kubectl-edit; an HPA that owns spec.replicas on a Deployment whose source is silent about replicas shows as spec.replicas owned by kube-controller-manager.
Not declared, stamped by Kubernetes itself	Nothing. deployment.kubernetes.io/revision, kubectl.kubernetes.io/last-applied-configuration, pod-template-hash and the other controller-written annotations and labels are on a static allowlist (K8S_SYSTEM_METADATA_PRUNE_PATTERNS in @intentius/chant/managed-fields), alongside status and the server-minted metadata.

The second row is the one that changed in chant #1191. Until then a foreign-owned field nobody declared was pruned before the diff ran, which silenced the console-edit case the deep read exists for. The noise valve is the accepted baseline: accept the HPA’s spec.replicas once with --update-baseline and it stays quiet until the value moves. Acceptance is value-bound, so accepting team=platform does not bless team=security.

What is still shelled out

kubectl exec, attach, port-forward and cp need SPDY or the 1.30+ WebSocket subprotocol, are not on any critical path, and stay as they are. Chant is not reimplementing kubectl.

Testing against it

@intentius/chant-k8s-client/testing ships the fakes chant’s own tests use — a literal kubeconfig and a recording request layer that replaces the HTTP send and nothing above it, so kubeconfig parsing, context selection, credential policy, discovery and URL construction all still run for real:

import { fakeCluster, objectKey } from "@intentius/chant-lexicon-k8s/api/fake-cluster";

const cluster = fakeCluster({
  objects: {
    [objectKey("ray.io/v1", "RayCluster", "ml", "ray")]: {
      apiVersion: "ray.io/v1",
      kind: "RayCluster",
      metadata: { name: "ml", namespace: "ray", uid: "uid-ray" },
      status: { phase: "ready" },
    },
  },
});

await describeResources({ environment: "prod", /* … */ }, cluster.connector);
expect(cluster.layer.paths()).toContain("/apis/ray.io/v1/namespaces/ray/rayclusters/ml");

No test in this repository reads an ambient kubeconfig or opens a socket to a cluster.