Evidence · How close AWS is
reference-k8s-shared-objects
hand-written reference shape kept in this repository, the kubernetes lane’s two-estate estate (#1882, epic #1885): estate platform owns a Namespace, a ConfigMap, a Secret, a Service, a Deployment on the node image’s pause container and a two-instance count ConfigMap as whole objects, plus one label on the cluster’s default Namespace through kubernetes_labels; estate app owns no object, only fields of platform’s objects and of the kind node, each written under the field manager choudoufu:
Set: growing. Lane: kubernetes.
Clear. Every headline stage passes.
| Stage | Verdict | Duration | Detail |
|---|---|---|---|
| Cold deploy | pass | 1m5s | two roots from plain terraform against kind v1.37.0: platform’s 8 instances (Namespace, 2 ConfigMaps plus a 2-instance count ConfigMap, Secret, Service, Deployment, and a kubernetes_labels on the cluster’s default Namespace) and then app’s 6 field-granular instances writing into them and into the node (kubernetes_labels, kubernetes_annotations, kubernetes_config_map_v1_data, kubernetes_secret_v1_data, kubernetes_env, kubernetes_node_taint), each root with a real terraform.tfstate; zero tofu-estate labels, and every field app writes owned by stock’s default manager Terraform, read from metadata.managedFields with kubectl; the same two roots cold-deployed by stock on a second cluster as every later stage’s oracle |
| Migrate | pass | 3s | platform: 8 of 8 newly stamped, 0 failed, 0 skipped - its 7 objects carry tofu-estate=shared-platform and the default-Namespace label moved from Terraform to choudoufu:shared-platform; app: 6 of 6 newly stamped, 0 failed, 0 skipped - #1869’s hand-over moved exactly the fields each block writes (the Namespace label, the Service annotation, the ConfigMap and Secret keys, the container’s APP_MODE env, the node taint) from Terraform to choudoufu:shared-app with no value changed, and Terraform owns none of them now; all read from metadata.managedFields with kubectl |
| Replan from nothing | pass | 24s | with no state file both plans are empty - platform’s and app’s, each over objects the other writes into, so neither proposes touching the other’s fields; platform’s 7 identities (NAMESPACE/NAME) confirmed with kubectl, and app’s 6 by value as the fields choudoufu:shared-app owns in metadata.managedFields (the patched object is the identity, the manager the marker), platform’s default-Namespace label by choudoufu:shared-platform. BREAK_BOUNDARY=1 drops platform’s ignore_changes entries and its plan correctly proposes taking app’s fields |
| No-op apply | pass | 24s | both no-op applies were 0 added, 0 changed, 0 destroyed; platform’s objects carrying tofu-estate=shared-platform unchanged at 7 over the five kinds the count covers, and the objects choudoufu:shared-app owns fields on unchanged at 6 (the field-granular estate’s count, read from metadata.managedFields since no label names it) |
| Drift and reconverge | pass | 57s | one of platform’s keys (settings.region) tampered with kubectl patch, in the ConfigMap app also writes app_mode into: platform proposed exactly kubernetes_config_map_v1.settings (0 add, 1 change, 0 destroy) and nothing about app_mode, matching stock’s own plan on the oracle cluster; app’s plan stayed empty; the apply changed 1, region reads back eu, and app_mode still reads shared, owned by choudoufu:shared-app alone in metadata.managedFields; both plans empty afterwards. BREAK=1 tampers a second object and the single-object assertion correctly fails |
| Rename | pass | 46s | moved block on a field-granular block: app’s kubernetes_config_map_v1_data.settings -> .app_settings planned No changes. Your infrastructure matches the configuration, no create, no destroy, no replace - the marker is the field manager and carries no address, so nothing on the object is rewritten - and applied without releasing the field: app_mode still owned by choudoufu:shared-app alone in metadata.managedFields; stock’s plan for the same moved block on the oracle cluster is zero churn; both plans empty afterwards. BREAK=1 points the block at another ConfigMap instead, which changes its identity and plans a create or a destroy |
| Remove a block | pass | 47s | deleting app’s kubernetes_config_map_v1_data block proposed exactly one destroy, at the sweep’s synthetic orphan address kubernetes_config_map_v1_data.orphan_shared-objs_settings - found by the fields choudoufu:shared-app owns, with no selector and no label - and the apply released the field and nothing else (#1869): app_mode is gone from platform’s ConfigMap and owned by nobody, the ConfigMap is still there carrying tofu-estate=shared-platform with region and every other platform key intact, its data identical to stock’s end state for the same removal on the oracle cluster (stock: 0 add, 0 change, 1 destroy); both plans empty afterwards. BREAK_REMOVE=1 keeps the block and no destroy is proposed |
| Change count | pass | 1m9s | scaling platform’s kubernetes_config_map_v1.shard from 2 to 1 destroyed exactly shard-1, planned at the sweep’s orphan address kubernetes_config_map_v1.orphan_shared-objs_shard-1 (shard-0 untouched, both read with kubectl); back to 2 created exactly kubernetes_config_map_v1.shard[1]; both estates’ plans empty afterwards; stock’s plans for the same two changes on the oracle cluster have the identical shape. BREAK_COUNT=1 asserts the lower index was destroyed and correctly fails |
| Replace with create_before_destroy | pass | 59s | a create_before_destroy ConfigMap whose content-hashed name changes (cfg-a -> cfg-b) plans as stock’s replace: ‘kubernetes_config_map.hashed must be replaced’, +/- create replacement and then destroy, 1 add and 1 destroy, with no orphan destroy beside it, because cfg-a carries the block’s address annotation and the sweep binds it (#1640). At -parallelism=1 the apply log shows cfg-b’s creation complete (line 64) before cfg-a’s deposed destroy starts (line 65), the same order stock’s apply shows on the oracle cluster; kubectl confirms cfg-b alone remains, carrying the annotation, and the next plan is empty (#1541). The block is removed from both roots afterwards. BREAK_REPLACE=1 recreates cfg-a carrying the block’s annotation and the next plan correctly proposes destroying it |
| Crash mid-apply | pass | 3m54s | app’s apply of two field-granular blocks, each labelling one of platform’s shards, was interrupted by a real SIGTERM (exit 1) the engine delivered itself at -parallelism=1 the instant kubernetes_labels.crash_first’s write committed; crash_second’s label reads crash_first’s name, so the walker cannot have reached it - kubectl confirms shard-0’s label owned by choudoufu:shared-app and shard-1 unlabelled, and the interrupted apply wrote crash_first’s record (records 8 -> 9). The next plan proposed exactly the remainder (Plan: 1 to add, 0 to change, 0 to destroy, kubernetes_labels.crash_second created) and nothing for crash_first, which it bound by the field its manager owns - not a second write, not an orphan release - matching stock’s own plan from the same position on the oracle cluster; the recovery added exactly one, shard-1 reads after-shard-0, and both estates’ plans are empty afterwards. BREAK_CRASH=1 asserts nothing is proposed and correctly fails; BREAK_CRASH_UNBOUND=1 takes the label off shard-0 and the recovery check correctly fails. The create_before_destroy rename window is interrupted too (#1768): a kubernetes_config_map renamed under create_before_destroy was killed by the engine’s own hook the instant the new object’s create committed, at -parallelism=1, leaving both objects carrying the block’s address annotation and the record holding the old one as the address’s deposed object. The verdict is the end state stock’s replace leaves, not the plan’s wording: after one more apply exactly the new object remains, the old one is gone, the record’s deposed entry is cleared and the replan is empty. Both of the rerun’s paths reached it - orphan leg: Plan: 0 to add, 0 to change, 1 to destroy, the destroy at ‘orphan_shared-objs_crash-rename-a’; deposed leg: Plan: 0 to add, 0 to change, 1 to destroy, the destroy at ‘(deposed object’; the same configuration through the orphan destroy, and the name read from another block’s attribute (#1539’s shape) through the deposed record (#1683), where stock’s plan reads the same deposed-object destroy. |
| Teardown | pass | 1m25s | app first: apply -destroy released exactly its 7 field-granular instances in one apply, choudoufu:shared-app owns no label, annotation or data key, env item or taint anywhere afterwards (a released block’s manager entry is left owning only an empty map or env list, and is not counted), and platform’s 7 objects still carry tofu-estate=shared-platform with its plan empty - app’s teardown left platform converged. Then, with app’s fields written back, platform’s apply -destroy removed exactly its 8 instances (its labelled objects and the default-Namespace label) in one apply, the same count stock’s destroy of the same estate removed on the oracle cluster with app’s fields present there too; the namespace is gone, no object carries tofu-estate=shared-platform and platform’s label is released. With platform gone, app’s next plan is refused by name (‘Patched object does not exist’, exit 1) once for each of its six blocks whose object platform’s destroy deleted, while stock’s plan on the oracle cluster reports ‘No changes. Your infrastructure matches the configuration’ from its stale state - the provider’s read keeps a missing object’s state - which choudoufu, with no state, does not reproduce (ruled 2026-10-04 on #1885); app’s final destroy released the node taint |
| Plan, review, apply | pass | 2m53s | plan -out wrote one update (settings gains reviewed=yes, in the ConfigMap app writes app_mode into); the world then moved out of band (a stray label on shard-0, kubectl) and apply of the saved plan refused with “The approved plan no longer matches the live system” at exit 3, nothing applied; with the label removed the identical file applied, 0 added, 1 changed, 0 destroyed, reviewed=yes reads back and app_mode is untouched. Stock’s own planfile applied on the oracle cluster in the unchanged case. The estate boundary’s refusals, each an extra block of app’s on the cluster’s default Namespace, where platform owns one label under choudoufu:shared-platform: without force the plan warned “Field owned by another estate” naming shared-platform and the API server refused the apply with a conflict naming choudoufu:shared-platform; with force = true the plan refused at exit 1, “Force refused over another estate’s field”, naming shared-platform, nothing applied and the label still choudoufu:shared-platform’s; force over a label kubectl wrote planned and applied as stock’s force does, the label then choudoufu:shared-app’s alone, and removing the block released it. Two blocks of app’s on one Deployment (kubernetes_env.web and a kubernetes_annotations) were refused, “Two field-granular blocks patch one object”. Both plans empty afterwards. BREAK_APPROVAL=1 expects success after the move and correctly fails; BREAK_FORCE=1 makes the forced write with the stock binary under choudoufu:shared-app and it takes the label |
| Greenfield apply | pass | 49s | both estates applied fresh with live blocks and no terraform.tfstate: platform 8 added, its 7 objects labelled tofu-estate=shared-platform and its default-Namespace label owned by choudoufu:shared-platform; app 6 added, every field it writes owned by choudoufu:shared-app in metadata.managedFields; both replanned empty. The cluster’s inventory - platform’s objects and every field app wrote into them, the default-Namespace label and the node taint - matches stock’s cold deploy of the same two roots on the same cluster field by field, labels and managers normalised out. BREAK=1 drops settings’ data from the expected inventory and the match correctly fails |
| Strict profile (not a headline stage) | pass | 30s | every strict toggle on (secrets = refuse, no_source_create = refuse, marker_repair = never with a markers “record” selection naming kubernetes_config_map_v1) against a scratch estate carrying random_password.db: exactly one refusal, Logical resource is not admitted under strict { secrets = “refuse” }; the other two toggles are on and silent. BREAK_STRICT=1 turns secrets back to “store” and the refusal disappears |
| Plan with no local state (not a headline stage) | pass | 45s | with both estates’ local record stores and state caches deleted - a fresh clone with only the cluster left - neither plan created, destroyed or replaced anything: platform’s objects found by their tofu-estate label, app’s six field-granular instances by the fields choudoufu:shared-app owns on objects it does not own. plan_calls_no_local_state=21 plan_calls_cache_serving=21 ratio=1.00x (both estates’ plans summed, the provider’s HTTP requests counted from TF_LOG=DEBUG); stock in this position has no plan at all. BREAK_NO_LOCAL_STATE=1 strips settings’ label and the check correctly fails |
Last run at commit 4bfb459f95 on 2026-10-05T02:38:36Z, exit code 0, against substrate image kindest/node:v1.37.0@sha256:a1ed56cfb0e7b93589bdf97c8cd566405a265939e3620fc4f5de89adff580ae5. Total run time 16m52s.
Oracle: stock terraform 1.15.8, stock tofu 1.12.5. Stale: the current pin is terraform 1.16.1, tofu 1.13.0.
Engine: OpenTofu base 1.13.0 (matches the current base).
Reproduce it
go run ./tools/gauntlet run reference-k8s-shared-objects
Needs Docker (the emulator is pulled at the pinned digest), the AWS CLI, and a
stock terraform or tofu binary on PATH for the cold deploy. The script is
live/e2e/reference-k8s-shared-objects/run.sh; BREAK=1 corrupts its assertions to show they are load-bearing.