Operator

A ConvergeOp tick re-observes and re-derives everything every time, so a crash recovery and a normal tick are the same act and a controller loop is enough. chant operator runs those ticks on a schedule, with gates. Git is its memory, chant lifecycle plan --live is the clock, and git update-ref is the lock. It needs no extra service or state store.

Pipeline ops vs converge ops

The two kinds of Op need different runtimes because durability means something different for each:

	Pipeline Op	ConvergeOp
Shape	A fixed sequence of steps, run start to finish	A loop that observes, classifies, and dispatches from scratch every tick
What a crash loses	Whatever the run had not finished when it died	Nothing; the next tick re-observes and re-derives everything
What “durable” means	Run it again, on a machine that is still there	Re-tick: run the whole loop again, cheap because ticks are idempotent
Waiting (a schedule, a gate)	A fact the run records and exits on	A symptom the next tick reads fresh, or a fact the tick records and moves past
Runtime	chant run <op>, here or --on a steward	chant operator, or the same steward

Both are convergent verbs; what differs is who decides when they run. A pipeline Op runs because somebody or something asked for it, so what it needs from a runtime is a machine that is still standing when the ask arrives. A ConvergeOp runs because the world might have moved, so what it needs is a cadence and a lease. It is level-triggered in the Kubernetes-controller sense, comparing the live level of the world against the declared level, recomputed from scratch every time. Re-running one beats resuming one, so nothing here resumes a run.

chant operator is the smallest thing that supplies the cadence and the lease: your own machine, or a CronJob, ticking on git alone. A fountain steward is the same cadence hosted, with the environment’s thread as the record. See Ops as teammates for how the pieces divide up.

When a step genuinely cannot be re-run

Compensation and mid-sequence resumption are workarounds for non-idempotent steps. The apply contract (tri-state envelope, owned-only prune, re-apply converges) exists so that chant verbs are already convergent, so a step that cannot be re-run safely usually belongs in a pipeline Op somebody invokes deliberately, rather than as a converge rule’s run() target that a tick may fire at any time.

Running it

chant operator                       # tick every discovered ConvergeOp, forever
chant operator --env staging         # scope to one environment
chant operator --once                # one round, then exit — cron/CI/CronJob invokers use this
chant operator status                # last tick, outcomes, pending gates — reads the ledger only
chant operator log                   # the whole tick history, and the gate resolutions against it
chant approve fountain-apply rollout-gate   # resolve a gate a tick recorded

chant operator is a deliberately small daemon: a timer, a lease, and the existing local executor (runOpLocally, the same path chant run <name> takes) in a loop. Every round:

1. Discover every ConvergeOp in the project (any Op whose labels.Converge === "true", which ConvergeOp(...) always sets), optionally filtered to --env.
2. Acquire or renew the lease for each one (see below). If another process holds it, skip and report (skipped=1(lease-held:<holder>)); never queue.
3. Tick the ones this process’s lease covers, through the same local executor chant run <name> uses, so a tick this daemon runs and a tick you run by hand are identical, byte for byte.
4. Sleep --interval (default 60s) and repeat, until Ctrl-C.

The daemon is optional. If it dies, nothing breaks and nothing is lost; the environment stops converging until something ticks it again: this process restarted, cron, a systemd timer, a Kubernetes CronJob, or a bare chant run <name>. Any of them can safely run the same tick, because ticks are idempotent and lease-fenced.

The lease: single-writer coordination over a git ref

Single-writer per environment is a lease, and git update-ref already supports compare-and-swap: pass the ref’s last-observed value as its old value, and the write only lands if nothing moved it in between. chant operator uses exactly that and nothing else:

• Each ConvergeOp gets a dedicated ref, refs/chant/lease/<op-name>, separate from the chant/lifecycle ledger branch’s own history. In a workspace member the ref is refs/chant/lease/_members/<member>/<op-name> (see Ledgers). The ref’s value is the lease record: a bare git blob (holder, a fencing token, acquiredAt, expiresAt). It has no tree and no commit.
• Acquire: read the ref’s current value; if it’s absent, expired, or already held by this same process, write a new value with update-ref <ref> <new> <old>. A concurrent acquirer loses the CAS outright. git update-ref’s own lockfile-then-rename is already an atomic local mutex, so no flock is needed on top of it.
• Renew: the same acquire call, made every round for an op this process still holds. It keeps the fencing token and pushes out the expiry.
• Fencing token: a fresh token is minted only when the lease actually changes hands (first acquire, or a re-acquire after the previous holder’s lease expired). A tick checks, right before trusting its own work, that its token is still the live one, so it detects a lease stolen mid-tick (the process stalled past its TTL).
• Cross-machine: the ref is pushed/fetched through the project’s configured remote, the same way the chant/lifecycle branch itself is. A project with no remote is single-machine by construction — the local CAS above is then the only coordination. If two operators on two different machines share no remote, nothing arbitrates between them; don’t run more than one in that setup.

chant operator --lease-ttl 5m        # default; long enough for a normal tick, short enough to recover promptly

Crash recovery

An operator that is stopped cleanly (Ctrl-C or SIGTERM) releases the Op leases it holds, so a restart ticks on its first round instead of printing skipped=1(lease-held:<old holder>) until they expire (#2879). A killed one cannot, and that is the case the TTL covers.

Restarting is ticking; there is no separate crash-recovery code path. A killed operator stops renewing its lease; the lease expires on its own TTL; the next round (this process restarted, a different machine’s operator, or a bare chant run <name> from cron) re-acquires and re-ticks. A converge tick re-derives everything from live observation every time (ConvergeOp’s own design), so re-ticking after a crash and ticking on a normal schedule run the same code.

All operator state (ticks, gates, leases) lives on git. rm -rf of any local scratch directory changes nothing: the tick history is on the chant/lifecycle orphan branch, and the lease is a ref, both re-fetchable from the remote.

One crash needs a human. If the killed process was interrupted mid-write (between git update-ref creating its lock file and renaming it into place), that lock file can be left behind, and it blocks every future write to that op’s lease ref until it’s removed; TTL expiry alone doesn’t help, since the ref update itself can’t land while the lock file sits there. chant operator never mistakes this for ordinary lease contention (a live holder elsewhere): a round reports it per-op as its own outcome (error=1(lease acquire failed — ...), naming the .lock path and the fix), logs it, and moves on to every other discovered op rather than aborting the whole round. Once the file is removed (rm <path>.lock), the next round acquires normally.

Gate-as-fact

A gate is a fact on the ledger. Nothing is suspended or holding a machine open for a decision: reaching a gate, a run reads the gate ledger, and either walks through on a resolution someone has already written or records that it is waiting and stops.

Run a gated Op directly and you see both halves:

chant run prod-apply

[phase] Plan
  ✓ terraformPlan(root=app)   2.1s
  • gate:rollout-gate   0ms
  • terraformApply(root=app)   skipped
Op "prod-apply" is gated on "rollout-gate" after 2.1s
  release manager signs off
  approve : chant approve prod-apply rollout-gate
  expires : 2026-09-07T12:00:00.000Z

The run exits 3, not 1. A gate is a decision nobody has made yet, so a CI job that reads 3 can report “waiting on approval” instead of paging whoever is on call. No phase after the gate runs, and no onFailure phase runs either: nothing failed, so there is nothing to compensate for. The pending fact lands on _gates/<op>.jsonl beside the resolutions, with the gate’s timeout as its expiry (48h by default). Running again before it expires re-reads the same fact rather than writing another one, so an operator ticking every minute against an unapproved gate does not grow the ledger by a line a minute.

A converge tick reports the same event in its own vocabulary. Its run() dispatch of a gated Op sees exit 3 and records one ConvergeRuleOutcome with action: "gated" and the gate’s name, on the same ledger record every other tick outcome lands on (<env>/converge.jsonl):

converge(staging): drifted=1 remediated=0 reported=0 skipped-budget=0 skipped-flap=0 gated=1 unobserved=0 adopted=0

The tick ends there for that rule; it does not block, retry, or queue. The next tick re-observes independently and may fire the same rule again. Either way it shows in chant operator status’s pending-gates list.

The pending fact also carries the commit the run was at (gitSha) and the run that reached the gate (runId), and chant approve copies both onto the approval (#4020). That is how a gate is joined to the commit it is waiting on, to its run’s record and to the release it admits. Chant 0.124.28 and newer write them.

The pending fact carries an address when the run has one. A run inside the PR (or MR) job that carries the change knows where approval happens, so it records that link, and chant operator status prints it under the pending gate:

  pending gates:
    - fountain-apply gate "rollout-gate" (rule drift-apply) — resolve: chant approve fountain-apply rollout-gate
      expires: 2026-09-07T12:00:00.000Z
      approve at: https://github.com/org/repo/pull/123

A run with no PR behind it records no url; the field is absent, never a synthesized link. A gate an ordinary chant run <op> stopped at appears under a pending gates (no converge tick) heading, since no rule dispatched it.

Resolution is out-of-band, the same two paths Reconciling Lifecycle already uses for a human gate:

chant approve fountain-apply rollout-gate --actor alex --url "https://github.com/org/repo/pull/123"

writes the counterpart fact, a GateResolutionRecord at _gates/<op>.jsonl; a merged PR resolves it just as well. --url is the typed link, so a reader gets “resolved by this PR” without sniffing free text; omit it inside a PR/MR CI job and that job’s own PR is used. --note stays for prose.

Then run the Op again. It reads the resolution, walks through the gate carrying the approver onto the step record, and finishes:

  ✓ gate:rollout-gate   0ms
    [approved] alex at 2026-09-05T13:04:00.000Z (https://github.com/org/repo/pull/123)
  ✓ terraformApply(root=app)   41.2s
Op "prod-apply" completed in 43.4s

Ordering is what makes this safe. A resolution counts only if it is newer than the gate’s newest pending fact, so approving in advance does not clear a gate recorded since, and last month’s approval does not answer the fact this morning’s run wrote. To take back a gate nobody intends to finish, chant approve <op> <gate> --expire clears the standing pending fact without approving anything, and the next run decides the gate from scratch.

chant run --components <name> behaves identically. A gate in a component’s composition stops that component pending chant approve <component> <gate>, exit 3, with no saga unwind and no rollback phase. The steps that ran before the gate are the steps the author meant to run before the approval.

The limits of chant approve

A resolution is a durable, queryable fact rather than an authorization check. Anyone who can run chant approve locally can write one, the same trust boundary a local git commit already has, and issue #1485 scopes signed resolutions out on purpose. Re-running is also what clears a gate; nothing resumes a run in place. That holds on a hosted runtime too. A steward’s turn ends at the gate rather than sitting on the thread waiting, and chant run approve <op> <gate> --on fountain posts chant run <op> back onto the thread once the resolution is written, so the sandbox re-runs and reads the fact.

Reading the ledger as an audit trail

@intentius/chant/lifecycle/gate-audit turns _gates/<op>.jsonl lines into typed audit entries and reports what the ledger is missing. It is a pure function that takes ledger lines and returns entries and gaps. It touches no git, clock or file, and it bundles small enough for a product’s single CI file. The product keeps storage and rendering.

import { deriveGateAudit, gateLedgerLines } from "@intentius/chant/lifecycle/gate-audit";

const { entries, gaps } = deriveGateAudit({
  ledgers: [{ op: "tf-apply", lines: gateLedgerLines(fileText) }],
  applies: [{ ref: "reports/1", at, gate: "wave-1", digest }],
});

Entry kinds are approval-requested, approved (with seal, signer, approver, relayedBy and review), override-requested, override-recorded (reason in note), revoked, and, for the applies a product passes in, applied (with the approval it ran under) and override-spent. A line a commit removed is a revocation: feed gateLedgerLinesFromLog the text of git log -p --unified=0 over the file. An override gate is one of op policy-override or named override-<rule> unless isOverride says otherwise. Gaps are approval-without-request, unsealed (when sealed(op, gate) says the gate needs a seal), approval and override (an apply with no approval or override behind it), and malformed-line. readGateAuditLedgers(port, ops) reads the current ledgers through a GateLedgerPort.

Op-wave decision records carry the outcome of every hook (outcomes, with skipped moments and chained holds). Pass them as decisions: [{ ref, at, record }] (or text, the file as written) and each outcome becomes a hook entry with hook: { moment, name, target, outcome, exit, reason }, where outcome is passed, failed, held or skipped. The entry carries the wave (wave). It links to the wave’s approval through under and to its applied entry through applied, matched on gate and digest. A record holds no time or place, so ref and at come from the caller; readGateAuditDecisions(port, dir, at) reads every .json under a directory of a LedgerReadPort for a product that keeps them on the ledger branch. A record that is not a decision record is a malformed-line gap.

Each ledger entry’s source names its line: path, line (the file’s line number from gateLedgerLines, otherwise the line’s place in lines) and text, exactly as passed in. Lines from gateLedgerLinesFromLog keep the commit that added or removed them, with its date, author and, if the log format ends in %x1f%ae, the author’s email, as source.author. Approvals and overrides read that way also carry committedBy, the author of the commit that added the line. Under approval: ledger anyone who can push may write any resolvedBy, so committedBy is the check on who actually pushed it. The default id digests the source and the record. A product that already holds its own ids passes id: (entry) => string, which receives each entry with its default id and returns the id to use. applied and override-spent entries point at the returned ids through under.

seal records how an approval was authorised. A line that came through a pull request review (via: "pr-review", or a review field) is review even if chant approve also sealed it, because the review is what let it through. A sealed line that did not come through a review is sealed, and a line with neither is unsigned. Any line that carries a seal also gets sealed: true and its signer, so a sealed review keeps both facts.

An approval a run carried from a gate’s alias (#3949) is an approved entry with carriedFrom: { op, gate, at }. Its request is the old name’s pending fact, so pass the old Op’s ledger too. With it, a missing request is an approval-without-request gap, and a cited line that isn’t in the old ledger is a carried-without-source gap. Without the old ledger neither is checked.

chant operator status

Everything below is read from the chant/lifecycle orphan branch alone, so no daemon needs to be running and nothing queries a live cluster:

chant operator status
chant operator status --env staging --json

staging-converge (staging)
  last tick : 2026-08-25T10:04:00.000Z
  converge(staging): drifted=1 remediated=0 reported=0 skipped-budget=0 skipped-flap=0 gated=1 unobserved=0 adopted=0
  lease     : held by mac.local:4821:9f2c1a3b (expires 2026-08-25T10:09:00.000Z)
  pending gates:
    - fountain-apply gate "rollout-gate" (rule drift-apply) — resolve: chant approve fountain-apply rollout-gate

A gate stops appearing as pending the moment chant approve (or a resolution recorded from elsewhere) is dated after the tick that recorded it. chant operator status re-checks the gate ledger on every read, so a stale/superseded approval (dated before the gated tick) doesn’t count.

chant operator log

status answers “where is this environment right now” and shows exactly one tick. log is the history behind it: every tick record for the discovered ConvergeOps and every gate resolution against them, merged into one timestamp-ordered timeline, oldest first, read from the same orphan branch:

chant operator log
chant operator log --env staging --op staging-converge
chant operator log --since 2026-08-25T00:00:00Z --limit 50 --json

2026-08-25T10:00:00.000Z  staging-converge@staging  [7f3c1a9b]  converge(staging): drifted=0 remediated=0 ...
2026-08-25T10:04:00.000Z  staging-converge@staging  [c02de451]  converge(staging): drifted=1 ... gated=1 ...
    gated  drift-apply → fountain-apply gate "rollout-gate"  https://github.com/org/repo/pull/123
2026-08-25T10:41:00.000Z  gate-resolved  fountain-apply/rollout-gate by alex  https://github.com/org/repo/pull/123

The bracketed value is the tick’s own id, so an outcome can be traced back to the tick that produced it. --limit keeps the newest n entries and still prints them oldest-first; --since takes an ISO-8601 instant and is inclusive.

--json emits {entries, malformed} as one document on stdout, with diagnostics on stderr, the same discipline operator status --json follows. malformed carries the per-ledger count of lines that could not be parsed, so a consumer can say “3 lines of this ledger were unreadable” rather than silently rendering a shorter timeline. Without --json the same count is a warning on stderr.

This exists so a reader never has to reach past the CLI. git show chant/lifecycle:<env>/converge.jsonl works, and pins whoever does it to the orphan branch’s name, the <env>/converge.jsonl and _gates/<op>.jsonl path conventions, and the record’s on-disk encoding. None of those are promised contracts.

Change signals: a second wake source, never a second source of truth

The loop above is time-triggered, so detection latency is bounded by --interval. A lexicon can shorten that by telling the operator when something moved, through the optional subscribeChanges seam on its plugin (#1981). Today one lexicon implements it: k8s, over the Kubernetes Watch API.

Three rules govern it.

The signal is a trigger, and never a fact. onChange() takes no arguments. There is no channel through which a watch event could reach a change set, a snapshot row or a diff line, because there is nowhere to put one. The round a signal wakes is the round the timer would have run: it re-observes and re-derives everything from the ordinary read path. Waking early changes when a tick runs, and nothing about what it concludes. A test in packages/core/src/op/operator.test.ts runs the same round on a timer and on a fabricated signal and asserts the two verdicts are identical.

A dropped subscription degrades to the timer and says so. A watch that dies is reported as one line, and so is one that never opened. The loop keeps ticking on --interval either way, and the next round re-subscribes. Losing the stream slows detection back down. It never stops the loop, and it never reads as a clean estate.

The timer stays. A signal shortens the sleep and never replaces it, so a substrate that silently stops sending still converges on --interval.

Signals are coalesced behind a floor of 5 seconds (DEFAULT_SIGNAL_FLOOR_MS). A rollout or a controller resync produces hundreds of truthful “something moved” events in a second, and a tick per event would turn a convergence loop into a load generator pointed at the cluster it is watching. So a pending signal wakes the sleep no earlier than 5 seconds after the last round started: a storm inside one window costs exactly one early round, and a substrate that never stops sending settles into ticking on the floor rather than on the interval. The floor is measured from the round, not from the last signal, so it is a rate limit. A debounce would delay the tick longer the busier the cluster gets.

Subscriptions need an environment, since they resolve the same binding a live read does, so chant operator --env <env> is what turns them on. Every subscription is closed and its abort signal fired when the operator stops.

Which substrates have a change stream worth using

A substrate qualifies when its notification primitive is complete and trustworthy enough to shorten a scheduled watch with nothing deployed into the observed estate. One family qualifies.

Lexicon	Native primitive	Usable as a signal	Cost
k8s	Watch API on any resource, resourceVersion-based, bookmarks	Yes, for every declared kind	One long-lived connection per watched kind and namespace. Re-list on 410 Gone
k3d, k3s	The same API server	Yes, through k8s	Same
helm	None of its own. A release is a Secret in the cluster	Indirectly, through the k8s watch	Same
docker	docker events on the local daemon	Yes, for the local daemon. Not implemented	A child process holding a stream
aws	EventBridge, CloudTrail, Config configuration items	No	Requires a rule, a target and a delivery path deployed into the observed account, plus a receiver. A read-only watch stops being read-only. CloudTrail coverage is per-service and delivery is minutes-delayed
gcp	Cloud Asset Inventory feeds to Pub/Sub	No	The same shape, a feed and a topic in the observed project, plus a subscriber
azure	Event Grid system topics, Activity Log	No	The same shape again, and system-topic coverage varies by resource provider
github, gitlab, forgejo	Webhooks	No	Complete for repo and org events, but needs an ingress endpoint reachable from the forge. chant runs no such receiver and should not
fly	The Machines API exposes no change stream	No
cedar, fountain, render, cpln	No change feed on the surfaces chant reads	No

Every cloud substrate fails on the same point. Subscribing means writing infrastructure into the account being observed, which breaks the property that makes a watch safe to point at production. Leaving them out is deliberate: on aws, gcp and azure the operator’s timer is the detection mechanism, and --interval is the latency.

Local-only durability

chant operator’s daemon lives entirely on your machine and your project’s git remote, with no server or database. Hosting the same cadence on a machine that is not yours is what a fountain steward is for, and OperatorStack (the in-cluster estate runtime, tracked separately) is deliberately out of scope here. If your project has no configured remote, the lease and the ledger are local-only, and a second machine running chant operator against the “same” environment has no way to see the first one’s lease at all.

See also

• Converging Lifecycle — defining a ConvergeOp’s rule table
• Ops as teammates — the general shape this page sharpens for the converge case
• chant operator CLI reference — flags and subcommands
• Lexicon authoring overview, where subscribeChanges sits among the optional plugin members
• Reconciling Lifecycle — the PR-as-gate pattern chant approve mirrors