Release a state lock a killed job left

Result

When a runner kills a job mid-apply (a timeout or a cancelled run), the binary never releases its hold on the state. The root’s next plan then waits until its timeout and fails. terragucci unlock-state runs force-unlock with checks a hand run lacks:

Check	What it guards against
no run that began before it was taken is alive	freeing state that a job still applying holds, so that two applies write one state
an approval of its ID	a release only the person typing agreed to, or one of a newer hold taken since
a record on chant/lifecycle	a release nobody can find later; the audit trail lists it as unlock

It reads the lock where the backend keeps it:

Backend	Lock	ID
s3 with use_lockfile = true	the lock file <key>.tflock	the ID in the file
gcs	the lock object <prefix>/<workspace>.tflock	the object’s generation, as the binary prints it
azurerm	a lease on the state blob, with the lock info in its metadata	the lease’s ID
GitLab-managed state	GitLab’s lock on the state	the ID in the lock

A comment never runs it. With binary: choudoufu and a record store, a killed job holds no lock, so there is none to release (locking with choudoufu).

Free the state

1. Find the root. The failed job’s log names the state and the ID:

   Error: Error acquiring the state lock
   Lock Info:
     ID:        1f0c6a52-...
     Path:      acme-state/envs/prod/app.tfstate
     Operation: OperationTypeApply
     Who:       root@runner-7
     Created:   2026-10-09 18:02:11.420 +0000 UTC

2. Make sure the job that took it has ended, and stop it on the forge if not.

3. In a checkout of the default branch, with the backend’s credentials and the forge token in the variable token_env names (FORGEJO_TOKEN, GITHUB_TOKEN or GITLAB_TOKEN by default), run:

   npx terragucci unlock-state envs/prod/app

   A run that began before the hold and is still running may be the holder; then nothing is freed, and the command exits 1 naming the run. With no such run, it records the release as waiting and prints the command that approves it:

   terragucci unlock-state: envs/prod/app: releasing lock 1f0c6a52-..., OperationTypeApply by root@runner-7 at 2026-10-09T18:02:11.420Z waits for an approval of digest jcs1-sha256:6d2b.... Approve it with:
   terragucci unlock-state:   terragucci approve unlock envs/prod/app --plan jcs1-sha256:6d2b...

4. Whoever approves your waves runs that command, with --sign under approval: sealed. The approval names this ID and no other.

5. Run npx terragucci unlock-state envs/prod/app again. It asks the forge about the runs once more and frees the state with the binary’s force-unlock of that ID. The release goes in _gates/tf-unlock/done.jsonl with the approver and the person who ran it.

6. Rerun the wave or push the next change; its plan takes the state as usual.

GitLab-managed state

GitLab has no call that reads a lock, so unlock-state asks for it. A held lock answers with its ID, the GitLab user who took it and when. A free lock is taken and given back at once. The release is a DELETE of that ID on the lock endpoint, which GitLab refuses when another lock holds the state.

You need	Why
lock_address and unlock_address in the block, or TF_HTTP_LOCK_ADDRESS and TF_HTTP_UNLOCK_ADDRESS	without them the backend takes no lock
TF_HTTP_USERNAME, and a personal access token with the Maintainer role in TF_HTTP_PASSWORD	GitLab lets Maintainers lock and unlock state

The log names the lock endpoint:

terragucci unlock-state: app: https://gitlab.example.com/api/v4/projects/42/terraform/state/app/lock holds lock 1f0c... by deploy-bot at 2026-10-09T18:02:11.420Z

Flags and exit codes are on CLI commands.