# Kubernetes change gate: tiered by readiness state **Decided by:** Bernd Worsch (founder), exercising `GOVERN @ estate`. **Date:** 2026-09-21. **Recorded by:** the-custodian. **Vocabulary:** SecurityCanon Mode of Authority v0.2.0 (`security-canon/infospace/vocabulary/mode-of-authority/`), which is still a draft. Where that vocabulary changes, this record follows it. This record answers gate-house's open question: which engine, if any, should own ops-mason's contact with the Kubernetes API. ## The question, stated in the canon's terms Applying manifests to a running cluster is `ADMINISTER @ realm:kubernetes/railiance01`. Changing the manifests is `CONSTRUCT @ `. A GitOps controller separates the two: `CONSTRUCT` produces the definition, and the controller reconciles the cluster to it. The question is what gates `ADMINISTER @ realm:kubernetes`, and whether that gate differs between experimental and production workloads. ## Decision 1. **No engine owns the contact.** The gate is a quality gate on a change, not an authorization decision. railiance-master ADR-0006 already keeps readiness and permission on separate axes: `production-approved` "is not an authorization decision". So no PDP belongs in this path. The contact is with `realm:kubernetes/railiance01`. This record assigns no layer to that realm and no layer to any rail. Mapping rails onto layers is statute §20.3, which ADR-0009 Decision 4 deliberately leaves unset. The first version of this record said the contact "stays a Tooling contact, owned by rail-kubernetes". That was an unintended §20.3 mapping, and it is withdrawn. 2. **The gate is tiered by ADR-0006 `readiness_state`, not by `Environment`.** railiance01 is a single cluster, and every workload on it shares one `Environment`. The canon's `Environment` dimension therefore cannot separate experimental from production here. The readiness state can. | Readiness state of the target | Path for a change | `Activation` | |---|---|---| | `declared`, `installed`, `verified` | Direct `ADMINISTER @ realm:kubernetes` by ops-mason | `APPROVED`: founder approval of the construction plan | | `production-approved` | `CONSTRUCT @ manifest repository`, reconciled by ArgoCD (`railiance-platform`) | `APPROVED` for the change; the merge is the gate | | `production-approved`, emergency | Direct `ADMINISTER` | `BREAK_GLASS`, recorded, reconciled back into the repository afterwards | Three rules settle the edges of the table (founder, 2026-09-21): - **Unmapped targets are production-tier until mapped.** The readiness contract defines states only for rails and bound rapps, and nothing yet maps a Kubernetes object or namespace to its binding. So a target without a mapping is production-tier. The friction this creates is deliberate: it is what motivates building the namespace-to-binding mapping (ops-mason MASON-WP-0006, with railiance-master). - **An evidence lapse does not loosen the path.** When a binding's evidence fails or expires, the contract returns it to its last proven state. The change path does not follow it down. Once a binding has reached `production-approved`, its changes stay on the git/ArgoCD path until the binding is explicitly re-scoped. A lapse weakens the readiness claim, and that is exactly when the gate must not loosen. - **`deprecated` keeps the tier it had before deprecation.** A deprecated production workload may still carry production traffic until it is removed. 3. **Why production goes through git: the `EvidenceBoundary` dimension.** A direct apply is `target-audited`: the only evidence of the change is held by the realm being changed. A change through the manifest repository adds `external-audited` evidence, which survives the actor and the realm. For production that is the property that matters, and it is a statement about evidence, not about trust in ops-mason. ## Explicit tier placements Some targets have no binding readiness state, but the founder has placed them in a tier explicitly. An explicit placement overrides the "unmapped is production-tier" default for that target only. | Target | Tier | Decided | Reason | |---|---|---|---| | namespace `whitehat` (whitehat-security; ops-mason bundle `whitehat-foundational-plane`) | non-production | founder, 2026-09-21 | A security-testing workload that no other layer depends on at runtime. Placing it explicitly keeps the unmapped default strict for real platform objects, and makes whitehat a deliberate exception rather than an accidental block. | A placement is revisited if the target gains dependants, or if it gets a binding with its own readiness state. If it gets a binding, the binding's state governs from then on. ## Transition for rapp-policy-nexus `rapp-policy-nexus` is `production-approved` (reef-railiance `bindings/rapps.yaml`), but ArgoCD does not manage it. Its `railiance-apps-root` applications are external-secrets, issue-core, openbao-secretstore and target-revenue. So the production path does not yet exist for the one workload that needs it. - **Until 2026-12-21:** changes to policy-nexus keep `activation=APPROVED` by founder plan approval. Each change additionally records that its target is production-tier, so the transition is visible and does not look like conformance. - **By 2026-12-21:** policy-nexus is onboarded to ArgoCD. From then on, the production row of the table applies to it. 2026-12-21 is the same review date as the ops-mason plan-approval gate, so both come up together. ## Scope of ops-mason's contact, as found ops-mason's executor is narrower than a generic `kubectl apply`: - it confines each plan to one expected namespace; - it refuses the `Pod` and `Secret` kinds; - it rejects any manifest carrying `data` or `stringData`. This decision relies on those limits. Widening them is a new decision. ## Verified 2026-09-21: ArgoCD is not on railiance01 A read-only check, run by the custodian over SSH with the founder's approval, established: - The node is `92.205.62.239`, the single k3s control-plane node. - The `argocd` namespace does not exist. - The `applications.argoproj.io` CRD does not exist. - The resource type `applications` is unknown to the API server. Only `get` was used; nothing was applied, patched or synced. **So the production row has no working path on railiance01.** Until one exists, the transition rule applies to every production-tier target there. Installing ArgoCD on railiance01 is a cluster-layer change and needs its own founder go-ahead. It is not part of this decision. ### What the records said before the check Whether ArgoCD runs on railiance01 at all is unverified, and railiance-platform's own records point the other way: - Its later records name the controller "coulombcore ArgoCD". - Its only health record, "healthy on railiance01", predates the 2026-07-02 correction, before which "railiance01" often meant coulombcore. - No evidence records `railiance-apps-root` syncing on 92.205.62.239. If ArgoCD is not on railiance01, **the production row has no working path there yet**. Until one exists, production-tier changes on railiance01 fall under the transition rule in the next paragraph, and installing ArgoCD on railiance01 is a cluster-layer change needing its own go-ahead. railiance-platform RPF-WP-0043-T01 holds the read-only check. Until the production row has a working path, the policy-nexus transition rule applies to every production-tier target on railiance01, not only to policy-nexus: `activation=APPROVED`, with each change recorded as production-tier. ## Owners - **gate-house:** the open question is answered. No engine owns the contact, and the gate is quality, not authorization. - **ops-mason:** phase 4 checks the target's readiness state and refuses a direct apply to a production-tier target, except under `BREAK_GLASS` or the policy-nexus transition. - **railiance-platform:** the ArgoCD lane is the production path, and policy-nexus onboarding is due by 2026-12-21. - **railiance-master:** ADR-0006 gains a deploy-path consequence for `production-approved`. - **rapp-policy-nexus:** the transition and its date. ## ArgoCD installed on railiance01 (phase A), 2026-09-21 **Decided by:** Bernd Worsch (founder), exercising `GOVERN @ estate`. This settles railiance-platform ArchitectureBlueprint §5.6 as option 1, **adopt properly**: ArgoCD runs on railiance01, and the live workloads are brought under GitOps one application at a time. **What was installed:** upstream Argo CD Core (headless: no API server, UI or Dex), **v3.5.3**. - Manifest: `core-install.yaml` at tag v3.5.3, sha256 `1a87025d8eb2eae621653fd312fb9ca51df1b4b3b6992a030e3a9ef38e45c448`. - Images: `quay.io/argoproj/argocd:v3.5.3` and `public.ecr.aws/docker/library/redis:8.2.3-alpine`. - Namespace `argocd`, created 2026-09-21T13:55:29Z. - 34 objects server-side applied after a clean server-side dry run. **Verified after the install:** - all four pods are Ready with 0 restarts; - there are 0 Applications and 0 AppProjects; - 0 namespaces carry an ArgoCD instance label, so nothing is adopted; - no Service is of type LoadBalancer or NodePort, so there is no new listener (ADR-0008). **The mode it was done in:** `ADMINISTER @ realm:kubernetes/railiance01`, `activation=APPROVED` by the founder, performed by the custodian session over SSH. By this record's own table the ArgoCD install is a platform object with no readiness state, so it is production-tier. It was done by direct apply under the transition rule, because the production row did not yet exist: this install is what creates it. Its declared state now has to be captured in its owning repository, so the install is not itself a hand-deployed object that nothing declares. **Rollback:** delete namespace `argocd` and the three `argoproj.io` CRDs. With no Applications, nothing else is affected. **Resource hygiene, open:** upstream core sets no resource requests or limits, so the ArgoCD pods are BestEffort. That makes them the first evicted under pressure, and unbounded. Set requests and limits when the install is captured in its owning repository. **Phase B**, adopting each application, and **phase C**, retiring coulombcore's ArgoCD, each need their own founder go-ahead. Phase B starts with the AppProjects and the root app with automated sync off. Then, per application, a diff, then a manual sync with prune off, and prune and self-heal only after that application is proven. ## Found during phase A: the node's CPU requests are exhausted Before the install, railiance01 already had **100% of its allocatable CPU requested** (4 of 4 cores); limits are at 828%. Short scheduled jobs cannot be placed, and have been failing with `Insufficient cpu` for days: | Pod | Pending since | |---|---| | user-engine database backup | 2026-09-16 | | authelia backup | 2026-09-20 | | lldap backup | 2026-09-20 | | privacyidea backup | 2026-09-20 | | audit-core attest-chain | 2026-09-20 | | activity-core cert-manager ACME solver | 2026-09-19 | ArgoCD did not cause this: it requests no CPU. It is an existing capacity defect with a direct consequence: **identity and user-engine backups have not run.** The custodian has not changed anything; it is reported to the founder. ## CPU requests released, 2026-09-21 **Founder go-ahead, 2026-09-21.** The mode was `ADMINISTER @ realm:kubernetes/railiance01` with `activation=APPROVED`, performed by the custodian session over SSH. Both targets are non-production tier: rail-knative's readiness is `verified`, and the stuck drill pod belongs to a drill. **Changed.** Only CPU requests were changed; limits are unchanged. | Deployment | Before | After | |---|---|---| | knative-serving/activator | 300m | 50m | | kourier-system/3scale-kourier-gateway | 200m | 50m | | knative-serving/net-kourier-controller | 200m | 30m | | knative-serving/controller | 100m | 30m | | knative-serving/webhook | 100m | 30m | | knative-serving/autoscaler | 100m | 30m | The HPAs on the activator, gateway and webhook target 100% of the request. At observed use they now read about 2–10%. The stuck pod `platform-pg-drill/platform-pg-drill-1` was force-deleted; its deletion had been requested on 2026-08-13 and it had no finalizers. **What happened.** The rolling updates deadlocked at first. The new pods are created before the old ones stop, and on a node with no requestable CPU none of the new pods could be scheduled. Force-deleting the drill pod freed 50m, which let the first new pod schedule; its old pod then stopped, and the rest followed. **Lesson:** on a node whose requests are exhausted, free capacity *before* changing a Deployment's requests, or a request cut cannot roll out. **Verified afterwards:** - all six rollouts are complete, with no old pods left; - `ksvc/rapp-qonto` is Ready; - 0 Pending pods cluster-wide; - the authelia, lldap, privacyidea and user-engine backups and audit-core's attest-chain have Completed, and fresh backups ran within minutes; - node CPU is at 69% actual use. **Open:** - rail-knative must declare these requests, or the next upstream apply resets them. - Actual CPU use is about 70% on a single node. That is a capacity question for the railiance02 move, not something to settle here. - state-hub uses about 447m against a 100m request. It is *under*-requested. ## State Hub request restored, 2026-09-21 State Hub's CPU request was **100m → 250m**, the chart's own default. The mode was `ADMINISTER @ realm:kubernetes/railiance01`, `activation=APPROVED` by the founder, done through state-hub's own promotion path. The 100m had been a declared override in `deploy/railiance/apps/helm/state-hub-values.yaml`, working around a node with less than 250m unallocated; that constraint is gone. **Steps:** 1. Changed the value in state-hub (commit `6b97d01`). 2. On railiance01, rendered the chart and diffed it against the live objects. The only substantive change was `cpu: 100m → 250m`, and the chart had not drifted from the live revision 63. 3. Ran the chart's headroom preflight: `ok: true`, 830m remaining. 4. Ran `helm upgrade --atomic` with the pinned image `main-0747b4d`, giving **revision 64**, 1/1 ready. Hub health afterwards: `ok`, db connected, schema `b5e6f7a8c9d0` as expected. There was no downtime (maxUnavailable=0). ## Phase B begun, and external-secrets repaired, 2026-09-21 All of the following were done with the founder's go-ahead, in `ADMINISTER @ realm:kubernetes/railiance01` with `activation=APPROVED`, by the custodian session. **1. ArgoCD resources** (railiance-enablement RAIL-EN-WP-0002-T02). Requests and limits were applied from the declared kustomization. All four pods are now Burstable, Ready, with 0 restarts. Free capacity was checked first, with 680m free, so the rollouts did not deadlock. **2. Phase B layout** (RPF-WP-0044-T01). The founder chose option A: a railiance01-only path, `argocd/railiance01/`. Nothing under `argocd/applications/` or `argocd/bootstrap/`, which coulombcore's ArgoCD reads, was changed. **Hazard:** `make argocd-bootstrap-deploy` still renders the old root with automated sync and must not be run on railiance01. **3. T02 and T03.** - **T02:** 3 AppProjects and a root app with automated sync off were applied at `c3ebd6d`. The root is Synced/Healthy with zero children, which shows the repo-server reaches Forgejo. - **T03:** openbao-secretstore was adopted: - the diff was rc=0; - the child was merged in `182e788`; - a manual sync at `d2dbc19`, with the apply strategy and prune off, succeeded; - `ClusterSecretStore/openbao` is Valid and tracked, and issue-core-runtime is SecretSynced. **It is the first workload under GitOps on railiance01.** It is now in its 24-hour proving period. `selfHeal` needs a second go-ahead, and `prune` a third. **4. external-secrets Helm release.** It had been stuck in `pending-upgrade` (rev 7) since 2026-07-07. Rev 6 and rev 7 were identical in values and manifests, and live matched both exactly, so the upgrade had fully applied and only the status was stuck. It was fixed with `helm rollback external-secrets 6`: dry run first, the chart has no hooks, and the result is rev 8, deployed. All three pods kept their names and their 36-day age, so nothing restarted. **Found and routed, not fixed:** two ClusterSecretStores have been failing since 2026-09-13 (`openbao-email-connect`) and 2026-09-19 (`openbao-activity-core`). Their static OpenBao tokens are rejected with 403 on `lookup-self`, so 5 of 37 ExternalSecrets are not syncing, including activity-core's offsite-backup credential. This is credential custody, so it was routed to railiance-platform, activity-core and email-connect. No token was read or minted.