CockroachDB Multi-Region on GKE

One TypeScript project, eight output files, nine CockroachDB nodes across three GCP regions. GCP’s VPC is global and routes between regions natively, so there is no VPN gateway and no two-pass build: every value is known at synthesis. Config Connector on a management cluster turns the GCP half of the output into real infrastructure; the Kubernetes half goes to the three clusters it creates.

What you’ll build

┌──────────────────────────────────────────────────────────────────┐
│                    GCP VPC: crdb-multi-region                    │
│                                                                  │
│  ┌──────────────┐     ┌──────────────┐     ┌──────────────┐    │
│  │  GKE East    │◄───►│  GKE Central │◄───►│  GKE West    │    │
│  │  us-east4    │     │  us-central1 │     │  us-west1    │    │
│  │  3 CRDB nodes│     │  3 CRDB nodes│     │  3 CRDB nodes│    │
│  └──────────────┘     └──────────────┘     └──────────────┘    │
│                    crdb.internal private DNS                     │
│         GCS backups + KMS + Cloud Armor WAF + Secret Manager     │
└──────────────────────────────────────────────────────────────────┘

What you’ll learn

• Why advertiseHostDomain exists — a cluster-local FQDN does not resolve from another cluster, and nodes that advertise one come up healthy, never find each other, and sit there
• How three composites collapse a 3-region estate to two files per region
• How a 13-step deploy script becomes phases, and how the one wait nobody can automate became a converge rule
• Why the operator you install with Helm belongs in the build output, pinned
• How to prove a multi-region database forms, on a laptop, with no cloud account

Try it without a cloud account

cd examples/cockroachdb-multi-region-gke
npm install
npm run smoke        # chant run crdb-k3d-smoke

Three CockroachDB regions in three namespaces of one k3d cluster: a shared CA, three secure nodes, one logical cluster, three regions known to SQL, and a REGIONAL BY ROW table that takes a write. About a minute once the CockroachDB image is cached. Needs k3d, kubectl and docker, and no credentials.

The cluster’s own shape is declared with the k3d lexicon and built to the config k3d cluster create --config consumes, so the declaration and the cluster cannot drift:

k3d/src/k3d-cluster.ts

export const smokeCluster = new Cluster({
  metadata: { name: "crdb-smoke" },
  servers: 1,
  agents: 0,
  options: new Options({
    k3d: new K3dOptions({ wait: true }),
    k3s: new K3sOptions({
      extraArgs: [
        new K3sExtraArg({ arg: "--disable=traefik", nodeFilters: ["server:*"] }),
        new K3sExtraArg({ arg: "--disable=servicelb", nodeFilters: ["server:*"] }),
      ],
    }),
  }),
});

One cluster, not three

Cross-cluster gossip is the part that needs three k3d clusters, plus something standing in for Cloud DNS and triple the memory. One cluster still proves everything short of that: the manifests apply, a secure cluster forms across three localities on a shared CA, exactly one region initialises, and multi-region SQL works. What it cannot cover — ExternalDNS, Workload Identity, External Secrets, GCE Ingress, Cloud Armor, and NetworkPolicy enforcement (k3s runs flannel, which ignores it) — is listed in k3d/src/regions.ts, next to the declarations it applies to.

Key patterns

advertiseHostDomain: the bug that looks like nothing

By default a CockroachDB node advertises its cluster-local FQDN, cockroachdb-0.cockroachdb.crdb-east.svc.cluster.local. That resolves inside east and nowhere else. Central and west receive an address they cannot reach, gossip never converges, and the symptom is nine healthy pods that never become one cluster.

CockroachDbRegionStack takes advertiseHostDomain, and every node advertises a name the private zone resolves from all three clusters:

src/east/k8s.ts

export const east = CockroachDbRegionStack({
  region: "east",
  namespace: "crdb-east",
  internalDomain: "east.crdb.internal",
  cockroachdb: {
    joinAddresses: config.joinAddresses,        // all nine
    advertiseHostDomain: "east.crdb.internal",  // → cockroachdb-0.east.crdb.internal
    extraCertNodeAddresses: NODE_ADDRESSES.east,
  },
  // ...
});

There is a second trap underneath it. Kubernetes exec-form args do not expand shell variables, so command: ["/cockroach/cockroach"] would emit the literal ${HOSTNAME}. The composite uses shell form:

command: ["/bin/sh", "-c"]
args: ["cockroach start ... --advertise-host=${HOSTNAME}.east.crdb.internal"]

Both properties are asserted in examples/examples.test.ts rather than left to documentation.

Three composites, two files per region

src/shared/   config.ts  infra.ts  platform.ts  secrets.ts  iam.ts
src/east/     config.ts  infra.ts  k8s.ts
src/central/  config.ts  infra.ts  k8s.ts
src/west/     config.ts  infra.ts  k8s.ts
platform/     eso.ts

Composite	Lexicon	What it emits
MultiRegionVpc	gcp	VPC, node + pod subnets per region, a router and NAT per region, allow-internal firewall
GkeCrdbRegion	gcp	GKE cluster + node pools, public DNS zone, ExternalDNS identity with WI + dns.admin, CockroachDB identity with WI + backup-bucket access
CockroachDbRegionStack	k8s	namespace with quota, limits and default-deny; pd-ssd StorageClass; the StatefulSet, services, RBAC and PDB; ClusterSecretStore + two ExternalSecrets; managed cert, FrontendConfig and GCE Ingress; Cloud Armor BackendConfig; ExternalDNS; Prometheus

A whole region is two calls:

src/east/infra.ts

export const east = GkeCrdbRegion({
  region: "us-east4",
  clusterName: "gke-crdb-east",
  network: "crdb-multi-region",
  subnetwork: "crdb-multi-region-east-nodes",
  domain: config.domain,
  project: config.projectId,
  crdbNamespace: "crdb-east",
  masterCidr: "172.16.0.0/28",
  backupBucket: BACKUP_BUCKET,
  nodeConfig: { machineType: "n2-standard-2", maxNodeCount: 3, diskSizeGb: 100 },
});

A gap in a composite is a lexicon fix

When one of these does not quite do what an estate needs, the answer is to change the composite, not to work around it in the example. Two of these three were extracted from an earlier version of this very example.

Build parameters, not process.env

Three values vary per deployment. Reading them from process.env at module scope is exactly the ambient read synthesis refuses to fold — and it is not free: it forced nine of each region’s eleven source files onto the run path, and every regional stack had to switch EVL001 off to permit it.

chant.config.ts

buildParams: {
  projectId:     { type: "string", default: "my-project",      env: "GCP_PROJECT_ID" },
  projectNumber: { type: "string", default: "000000000000",    env: "GCP_PROJECT_NUMBER" },
  domain:        { type: "string", default: "crdb.example.com", env: "CRDB_DOMAIN" },
}

src/shared/config.ts

import { params } from "@intentius/chant/params";

export const GCP_PROJECT_ID = params.projectId as string;

The env mapping keeps set -a && source .env && set +a working exactly as before. Every file in all four stacks now folds. See Build-Time Parameters.

projectNumber is there for one reason worth knowing: Google-managed service agents are addressed by project number, not id, and the GCS agent needs permission on the CMEK key before the backup bucket will accept it.

The deploy is an Op

scripts/deploy.sh used to be 205 lines and thirteen numbered steps. Each thing it did badly is something an Op does by construction.

ops/deploy.op.ts

phase("Network", [kubectlApply("dist/shared-infra.yaml", { context: MGMT, profile: "longInfra" })]),

phase("Clusters",
  REGIONS.map((r) => kubectlApply(`dist/${r}-infra.yaml`, { context: MGMT, profile: "longInfra" })),
  { parallel: true }),

phase("Clusters ready", [
  ...REGIONS.map((r) => waitForReady(CC_CLUSTER,   `gke-crdb-${r}`,       { context: MGMT, profile: "longInfra" })),
  ...REGIONS.map((r) => waitForReady(CC_NODE_POOL, `gke-crdb-${r}-nodes`, { context: MGMT, profile: "longInfra" })),
], { parallel: true }),

The readiness phase is the clearest trade. The script waited on kubectl wait --for=condition=Ready containercluster/... and then polled gcloud container node-pools describe sixty times in a bash loop. Both are Config Connector resources, so waitForReady reads their own Ready condition — and fails fast on a terminal one instead of polling out the timeout.

Where the wait goes

Between two steps the old script printed a DNS-delegation reminder in an ASCII box and carried straight on. Delegating three subdomains at a registrar is the one action nobody can automate from inside GCP, and Google will not issue the managed certificates until the names resolve — so the deploy “succeeded” while three certificates sat in PROVISIONING and the UIs served 502 until somebody remembered.

Publishing the UIs is therefore its own Op, and the database does not wait on any of it:

ops/publish-ui.op.ts

export default Op({
  name: "crdb-publish-ui",
  depends: ["crdb-deploy"],
  phases: [
    phase("Nameservers",  [shell("gcloud dns managed-zones describe ...")]),
    phase("Certificates", [shell("kubectl wait managedcertificate/... --timeout=45m")]),
    phase("Verify",       REGIONS.map((r) => httpCheck(`https://${r}.${DOMAIN}/health`)), { parallel: true }),
  ],
});

That Op held a 72-hour gate once, released by somebody who had to remember to send a signal after they had already done the work at the registrar. It holds nothing now. Run before delegation it fails on the certificate wait, which is the honest answer: the names do not resolve yet. Run after, it succeeds.

Which means nobody has to remember. ops/publish-ui-converge.op.ts is a ConvergeOp that observes prod every quarter hour and dispatches crdb-publish-ui while the UI stack is short of what src/ declares:

export const { op } = ConvergeOp({
  name: "crdb-ui-converge",
  env: "prod",
  dial: "apply",
  schedule: "*/15 * * * *",
  budget: 1,
  rules: [
    when(gt("updateCount", 0), run("crdb-publish-ui"), {
      id: "ui-unpublished",
      why: "The UI ingresses and their certificates are declared but not yet live …",
    }),
  ],
});

The first tick after the NS records propagate is the one that publishes. The delegation is the signal.

chant run crdb-deploy            # the database, which waits on nothing
chant run crdb-publish-ui        # once, by hand: prints the nameservers you delegate
chant run crdb-ui-converge       # one tick, or leave the cadence to notice

ops/fountain.ts declares the steward those cadences live on. Out of it come an Environment and a Vault, an Agent speaking ACP over chant acp with a Teammate seat, and one Schedule per Op that carries a cron. chant run <op> --on fountain posts the command line to that thread instead of running it here, so the conversation is the estate’s operational history. Every Op above still runs locally with no fountain at all.

npm run build:fountain           # dist/fountain.yaml
chant run crdb-ui-converge --on fountain

The operator belongs in the build output

External Secrets Operator was installed by helm repo add + helm upgrade --install in the deploy script. That left kubectl apply -f dist/ short of the actual deployment, and pinned no version at all — the chart has gone from 0.10 to 2.9 since this example was written, across a major version, and every region’s ClusterSecretStore depends on the CRDs it installs.

platform/eso.ts

export const ESO_CHART_VERSION = "2.9.0";

export const externalSecrets = HelmRender({
  name: "external-secrets",
  repo: "https://charts.external-secrets.io",
  chart: "external-secrets",
  version: ESO_CHART_VERSION,
  namespace: "kube-system",
  values: { installCRDs: true, serviceAccount: { name: "external-secrets-sa" }, /* … */ },
});

HelmRender runs helm template at synthesis and turns each rendered manifest into a Declarable, so the operator lands in dist/eso.yaml — 44 resources, 25 of them CRDs — and is applied like anything else. The render caches under ~/.chant/helm-renders, keyed by repo, chart, version and values.

Getting the operator installed is only half of it. The deploy waits for both ExternalSecrets in every region to report SecretSynced before it waits on a single CockroachDB pod:

phase("Secrets synced",
  REGIONS.flatMap((r) => ["cockroachdb-node-certs-eso", "cockroachdb-client-certs-eso"].map((name) =>
    waitForReady(EXTERNAL_SECRET, name, { namespace: `crdb-${r}`, context: r, spec: SECRET_SYNCED }))),
  { parallel: true }),

That gate is there because the chain was decorative until recently. The cert script created those Secrets itself with kubectl create secret, and ESO will not adopt a Secret it does not own — so both ExternalSecrets sat in permanent error while the deploy reported success on the pre-created bytes. Secret Manager, the ESO install and the Workload Identity bindings behind it delivered nothing, and nothing said so.

It lives in platform/, outside src/, because rendering wants the helm binary and, on a cold cache, the network. The four stacks under src/ stay pure synthesis.

Ownership, and the observe position

chant.config.ts

ownership: { stack: "crdb-multi-region" },

Every one of the 193 emitted resources carries chant.intentius.io/stack: crdb-multi-region. That marker lives on the live resource, not in a state file chant hosts, which is what makes a later prune precise — and what makes this work:

chant lifecycle diff prod --live

See Lifecycle Models for why the marker and the snapshot are deliberately different things.

No two-pass build

The GKE and EKS microservice examples build twice: infra first, then extract service-account emails or ARNs, then build the manifests with real values. This estate does not need it. One global VPC, one IAM system, and names that follow from the project and the domain — so everything resolves at synthesis, from three build parameters.

Cost

A dollar or two an hour at steady state — four regional control planes, twelve nodes, and the disks under them. The example README lists what to price; the numbers move, so price it for your own regions. Tear it down after testing.

Deploy it

cd examples/cockroachdb-multi-region-gke
cp .env.example .env && $EDITOR .env
set -a && source .env && set +a

npm install
npm run bootstrap    # once: management cluster + Config Connector
npm run deploy       # chant run crdb-deploy

The example README has the full deploy, verification and troubleshooting walkthrough — including the one that costs the most time to diagnose: cockroach sql against /cockroach/cockroach-certs fails with password authentication failed for user root, because the node certs secret contains no client certificate.

Further reading

• GCP Config Connector lexicon — resource reference and composites
• GKE Composites — GkeExternalDnsAgent, GceIngress, CockroachDbRegionStack, and the rest of the GKE surface
• Ops — phases, gates, profiles, and compensation
• Build-Time Parameters — declaring what varies per deployment
• Multi-Stack Projects — splitting one project across stacks
• The Steward — the machine the cadences above run from, declared