Waves and approvals

A shared module change goes out in batches, each after the last; a batch waits only when the gate policy holds it.

Waves

The roots in waves.canary go first, then one dependency layer per wave, so no root in a wave reads another in it. The report’s wave 3 is the gate wave-3.

A root’s dependencies are the roots whose state it reads through terraform_remote_state. For plain roots that read nothing of each other, waves.after gives the order instead:

waves:
  after:
    app: [database]
    database: [network]

wave 1  network
wave 2  database   (after network)
wave 3  app        (after database)

A pull request that changes network plans all three, and its blast radius names database and app.

Terragrunt units split the same way, the canary units’ layers first. Each wave is one terragrunt run --all behind its own gate. Once approved, it applies exactly the plans its job saved.

Where	Terragrunt waves come from
init	terragrunt find; without Terragrunt on the path, the plain paths in each unit’s dependency and dependencies blocks
each apply job	terragrunt find in the job, which splits a wave whose units read each other; the last job also runs any wave past the jobs init wrote

Approval binding

The wave’s set digest is what an approval binds, so it covers exactly those plans. Under approval: ledger, the default, that is all it takes, so anyone who can push to chant/lifecycle can approve in anyone’s name. approval: sealed also requires its ssh seal to verify against the signers file.

The approval key, the signers and the gates in chant.workspace.json come from the applied commit’s first parent (the default branch before an applying pull request), so a change cannot loosen its own rule. Only a multi-commit rebase breaks this; merge or squash such changes.

A gated wave reads the newest approval of its gate:

That approval names	The wave
the digest it plans now	applies, after it records on chant/lifecycle that it used the approval
another digest, which a run applied	waits for an approval of its own digest, with the command to give it
another digest, which no run applied	applies nothing and exits 4: its plans changed after approval

An approval never counts for a digest other than its own. Under on-destructive a re-plan with no destroy applies; under never nothing is refused.

Plans are made late

Each wave plans on real outputs after the last one applied, and an approval binds only that plan’s digest. A Terragrunt unit never plans on mock_outputs (details).

Linked roots

A root that reads another root’s state through terraform_remote_state goes in a later wave. When a pull request’s change reaches both, the reading root plans on the other root’s planned outputs instead of its last applied ones:

An output it reads	In the pull request’s plan
known when the upstream plans	the new value
known only once the upstream applies	(known after apply)
of an upstream the change does not reach	the applied value

The plan note lists each root planned this way and the outputs it read before they were known. A wave that read such an output plans again once the waves it reads have applied, then waits for an approval of that plan. No digest for it appears in the note, and under approval: pr-review the pull request’s review does not approve it. With only known outputs read, the wave plans the same digest again when its upstream applies as planned, so an approval or review of the note’s digest still counts.

For that plan terragucci adds terragucci_linked.tf with a local holding the upstream’s planned outputs and points the root’s references to the block at it. The files are restored after the plan. A root in Terraform’s JSON syntax, such as a CDK Terrain stack’s cdk.tf.json, is linked the same way: its backend and its terraform_remote_state blocks order the waves, and its references are rewritten inside their ${...} templates. These still plan on the applied state, and the report’s roots[].reads says why:

Case	It plans on
an upstream that has never applied	nothing yet: the root waits for it, and the report and note name it as waiting
a block with count or for_each	the applied state
a plan on the planned outputs that fails, such as a for_each over a value known only after apply	the applied state
a block in a child module	the applied state; it does not order the waves either
terragucci plan on your machine	the applied state

A Terragrunt unit’s dependency is previewed through Terragrunt itself, and a unit that reads a value known only once its upstream applies is not planned at all: see Use Terragrunt.

The run view

With reports.bucket set, each apply also writes one page for its commit, <prefix>/<project>/runs/<commit>/run.html, and its data, run.json. It shows the waves left to right with their roots and the roots each reads, and where each wave stands:

State	The wave
not started	no job of it has run on this commit yet
waiting	waits for an approval; the page gives its digest and the approval command
applying	is applying, or its share jobs are
applied	applied, or had nothing to apply
refused	applied nothing: its plans changed after an approval, or the policy denied a root
failed	a root failed to plan or apply

Each wave’s job rewrites its own column when it starts applying and when it ends, and links its report. A wave’s report.json says the same in waves[].state, and a pull request’s report marks the waves that plan again in waves[].replans_after.

Below the waves the page has two more parts:

Part	Shows
Blast radius	the roots whose plan in a wave changes something, every root that reads their state through terraform_remote_state or that waves.after puts after them (in a Terragrunt repo, every unit whose dependency or dependencies block names it), followed through, and the dependency graph of the commit’s roots with both marked
Timeline	each wave’s plan, its wait at the gate and its apply on one time axis, a wait that has not ended drawn open, and the time in each as a table

A wave that waited shows the plan that asked for the approval and the later plan that applies. In run.json each wave carries changed (the roots whose plan changes something) and spans, each with phase (plan, gate or apply), start and, once it ended, end. Each root carries state, the state its backend block names, and external, its reads of a state no root of the repo holds, which the estate page matches to other projects’ roots.

Progress

With choudoufu, a wave’s column also follows each resource its plans change while it applies:

2 of 4 resources done

• random_password.db done
• terraform_data.seed done
• time_sleep.warm_up in flight
• terraform_data.config waiting

The wave reads its estate’s records every 5 seconds (TG_PROGRESS_SECONDS), not the apply’s output. A resource is done once its record is written or removed, which choudoufu does as each resource’s apply returns. It is in flight once the resources it depends on are done, and waiting before that. A resource with no record of its own is done when its root’s apply returns. While a wave applies, the page reloads every 15 seconds, the job writes the same list to terragucci-report/progress.json, run.json carries it in waves[].progress, and the estate page shows the counts.

Blast radius in the plan note

Pull requests get the same radius in their plan note, by wave: the roots whose plan changes something and each downstream root that reads their state. A root the run did not plan, such as one --root left out, is marked “not planned in this run”.

**Blast radius:** 1 root changes (`envs/dev/platform`), and 3 roots downstream read their state:

- `envs/dev/orders` (wave 2) reads `envs/dev/platform`
- `envs/dev/payments` (wave 2) reads `envs/dev/platform`
- `envs/dev/search` (wave 2) reads `envs/dev/platform`

In a Terragrunt repo the radius is the units whose own plan changes something, not a preview, and each unit that depends on them through a dependency or dependencies block, followed through:

**Blast radius:** 1 unit changes (`live/dev/vpc`), and 1 unit downstream depends on them:

- `live/dev/app` (wave 2) depends on `live/dev/vpc`; not planned in this run

The note leaves it out when nothing reads or depends on a changed root. report.json carries it as blast.

Resources in the radius

For plain roots, the note also lists each changed resource and the resources that depend on it, in its own root and in each root that reads an output it reaches:

**By resource:**

- `aws_sqs_queue.jobs` in `queue` (update) reaches:
  - `aws_sqs_queue_policy.jobs`
  - `aws_lambda_event_source_mapping.jobs` in `worker`, through output `jobs_arn` of `queue`
  - `aws_lambda_function.worker` in `worker`, through output `jobs_arn` of `queue`

The edges come from each plan’s references: expressions, count, for_each, depends_on, module arguments and outputs, and locals. Between roots, a resource is listed only when it reads an output the change reaches, through terraform_remote_state or terraform_estate_outputs. A reader of worker’s outputs is followed the same way. A root the run did not plan stays in the root list only. report.json carries the list as blast.resources.

Waiting waves

The gate policy decides which waves wait for a person.

Policy	Waits for an approval when
always	the wave has at least one change; a wave whose plans change nothing never waits
on-destructive	the wave’s plans destroy or replace something
never	never; only pull request review and default branch protection stand in front of it

A waiting wave exits with code 3 and prints the approval command; run it, then rerun the stage. A rerun resumes after the approval or failed root and never applies a root twice.

A wide wave across jobs

parallelism bounds what one job runs at once, and every wave gets a single job, so a wave of a few hundred roots keeps one runner busy while the rest idle. Set waves.jobs to spread it over several:

waves:
  jobs: 3

Job	What it does
apply-wave-<n>	plans every root, runs the policy, decides the gate on the set digest and records the approval it uses, then hands each plan digest to the shares and changes nothing
apply-wave-<n>-share-<s>	plans its part again and applies it if each digest matches what apply-wave-<n> decided; otherwise exits 4 and changes nothing

Shares run side by side as far as runners are free, and the next wave starts after the last of them. Each wave still has one gate with one ledger record and one approve command. Each share copies its report to the bucket under tf-apply-wave-<n>-share-<s>.

In a Terragrunt repo the wave’s job plans every unit with one run --all, and each share plans and applies its own units with one run --all --filter. When the last wave splits, an apply-rest job after its shares runs any wave Terragrunt’s edges cut past the pipeline’s, as the last wave’s job does otherwise.

On GitLab the shares take each other’s place in the apply resource group, which runs one job at a time, so with waves.jobs the apply jobs hold a lock on the remote instead, as on Forgejo; the pipeline’s CI_JOB_TOKEN cannot push, so the project’s token does. init refuses the key under apply.when: pull-request. A comment’s apply and the resume job keep a single job per wave.

Applies side by side

Two pushes that change different roots apply at the same time, and a root’s own applies take turns at its state lock; with choudoufu, waves wait only for a run changing one of their resources. A wave a newer push superseded stands down. With apply.when: pull-request, an applying pull request also locks each root it reaches until it merges or closes. With locks: plan, a pull request holds that lock from its first plan. Per forge.

Next

• Approvals as records in your repo
• Approve a waiting wave
• Fix a refused wave