Setting up CI for many teams

For a platform team that runs the pipelines of many schema projects: one renderer, the same jobs on each forge, and a pinned image.

Works

• yodel ci renders every project's pipelines from the installed package, and yodel ci --check fails a pipeline edited by hand or not rendered again
• ci.forges picks GitHub, GitLab or Forgejo, and ci.jobs keeps a project's own jobs across renders
• ci.image runs every job in the CI image, pinned by its digest
• Waves: one environment after another, each behind its own gate

Differs

By database

ClickHouse

• A project owns ClickHouse databases (yodel create <dir> --clickhouse --database <name>).
• A rebuild keeps the old table until yodel cleanup drops it, and lint flags mutations and rebuilds (ch-mutation, ch-rebuild).

Postgres

• A project owns Postgres schemas (yodel create <dir> --postgres --schema <name>).
• Roles stay the environment's: the schema declares the policies and grants that name them.

By forge

GitHub

• The writer's secrets are secrets of a GitHub environment limited to main.
• The plan comment and the drift issue use the job's own token.

GitLab

• The writer's variables are protected and scoped to the environment; the readers' are not protected.
• The plan comment and the drift issue need GITLAB_TOKEN, and each drift watch runs from a pipeline schedule you create.

Forgejo

• Secrets belong to the repository, with no environments, so limit who can push branches.
• Jobs need a runner with the docker label, and a job gets no OIDC token for cloud roles.

First step

npm run ci

The pipelines: yodel ci

Proof

Each claim runs what this room relies on against a real server, once plain (it passes) and once with the behaviour broken (the claim catches it). Claims status lists every claim.

Claim	What it says	Plain, broken
waves	the apply pipeline runs one wave per environment, in order, each behind its gate policy read from the base commit, applies only what the wave before applied, and applies a tenant set's migrations to every tenant behind one gate; a sealed wave counts only an approval sealed by a signer the base commit lists	ClickHouse: pass, caughtPostgres: pass, caught
template	a project from the starter template, on Forgejo: apply only after approval, lint with replay and the plan comment on a pull request, and the approved change applied on merge; a sealed wave applies only on an approval sealed by a signer listed at the base, and a pr-review wave on the review of a writer other than the author; a pull request job cannot write, a forked migration fails lint and is annotated, a stale or hand-edited pipeline fails yodel ci --check, the CI image pinned by digest runs a pull request's jobs, a command token source mints the reader's password, and the drift watch keeps one tracking issue	ClickHouse: pass, caughtPostgres: pass, caught

Then read

Tasks

Setting up each forgeLint and plan on pull requests only

Background

The pipelines: yodel ciWaves: a gate per environmentLeast privilege on each forge

Reference

yodel.config.tsThe commands