the-custodian/docs/kubernetes-change-gate-decision.md

Kubernetes change gate: tiered by readiness state

Decided by: Bernd Worsch (founder), exercising GOVERN @ estate. Date: 2026-09-21. Recorded by: the-custodian.

Vocabulary: SecurityCanon Mode of Authority v0.2.0 (security-canon/infospace/vocabulary/mode-of-authority/), which is still a draft. Where that vocabulary changes, this record follows it.

This record answers gate-house's open question: which engine, if any, should own ops-mason's contact with the Kubernetes API.

The question, stated in the canon's terms

Applying manifests to a running cluster is ADMINISTER @ realm:kubernetes/railiance01. Changing the manifests is CONSTRUCT @ <manifest repository>. A GitOps controller separates the two: CONSTRUCT produces the definition, and the controller reconciles the cluster to it.

The question is what gates ADMINISTER @ realm:kubernetes, and whether that gate differs between experimental and production workloads.

Decision

1. No engine owns the contact. The gate is a quality gate on a change, not an authorization decision. railiance-master ADR-0006 already keeps readiness and permission on separate axes: production-approved "is not an authorization decision". So no PDP belongs in this path. The contact is with realm:kubernetes/railiance01. This record assigns no layer to that realm and no layer to any rail. Mapping rails onto layers is statute §20.3, which ADR-0009 Decision 4 deliberately leaves unset. The first version of this record said the contact "stays a Tooling contact, owned by rail-kubernetes". That was an unintended §20.3 mapping, and it is withdrawn.

2. The gate is tiered by ADR-0006 readiness_state, not by Environment. railiance01 is a single cluster, and every workload on it shares one Environment. The canon's Environment dimension therefore cannot separate experimental from production here. The readiness state can.

   Readiness state of the target	Path for a change	Activation
   declared, installed, verified	Direct ADMINISTER @ realm:kubernetes by ops-mason	APPROVED: founder approval of the construction plan
   production-approved	CONSTRUCT @ manifest repository, reconciled by ArgoCD (railiance-platform)	APPROVED for the change; the merge is the gate
   production-approved, emergency	Direct ADMINISTER	BREAK_GLASS, recorded, reconciled back into the repository afterwards

   Three rules settle the edges of the table (founder, 2026-09-21):

   • Unmapped targets are production-tier until mapped. The readiness contract defines states only for rails and bound rapps, and nothing yet maps a Kubernetes object or namespace to its binding. So a target without a mapping is production-tier. The friction this creates is deliberate: it is what motivates building the namespace-to-binding mapping (ops-mason MASON-WP-0006, with railiance-master).
   • An evidence lapse does not loosen the path. When a binding's evidence fails or expires, the contract returns it to its last proven state. The change path does not follow it down. Once a binding has reached production-approved, its changes stay on the git/ArgoCD path until the binding is explicitly re-scoped. A lapse weakens the readiness claim, and that is exactly when the gate must not loosen.
   • deprecated keeps the tier it had before deprecation. A deprecated production workload may still carry production traffic until it is removed.

3. Why production goes through git: the EvidenceBoundary dimension. A direct apply is target-audited: the only evidence of the change is held by the realm being changed. A change through the manifest repository adds external-audited evidence, which survives the actor and the realm. For production that is the property that matters, and it is a statement about evidence, not about trust in ops-mason.

Explicit tier placements

Some targets have no binding readiness state, but the founder has placed them in a tier explicitly. An explicit placement overrides the "unmapped is production-tier" default for that target only.

Target	Tier	Decided	Reason
namespace whitehat (whitehat-security; ops-mason bundle whitehat-foundational-plane)	non-production	founder, 2026-09-21	A security-testing workload that no other layer depends on at runtime. Placing it explicitly keeps the unmapped default strict for real platform objects, and makes whitehat a deliberate exception rather than an accidental block.

A placement is revisited if the target gains dependants, or if it gets a binding with its own readiness state. If it gets a binding, the binding's state governs from then on.

Transition for rapp-policy-nexus

rapp-policy-nexus is production-approved (reef-railiance bindings/rapps.yaml), but ArgoCD does not manage it. Its railiance-apps-root applications are external-secrets, issue-core, openbao-secretstore and target-revenue. So the production path does not yet exist for the one workload that needs it.

• Until 2026-12-21: changes to policy-nexus keep activation=APPROVED by founder plan approval. Each change additionally records that its target is production-tier, so the transition is visible and does not look like conformance.
• By 2026-12-21: policy-nexus is onboarded to ArgoCD. From then on, the production row of the table applies to it.

2026-12-21 is the same review date as the ops-mason plan-approval gate, so both come up together.

Scope of ops-mason's contact, as found

ops-mason's executor is narrower than a generic kubectl apply:

• it confines each plan to one expected namespace;
• it refuses the Pod and Secret kinds;
• it rejects any manifest carrying data or stringData.

This decision relies on those limits. Widening them is a new decision.

Verified 2026-09-21: ArgoCD is not on railiance01

A read-only check, run by the custodian over SSH with the founder's approval, established:

• The node is 92.205.62.239, the single k3s control-plane node.
• The argocd namespace does not exist.
• The applications.argoproj.io CRD does not exist.
• The resource type applications is unknown to the API server.

Only get was used; nothing was applied, patched or synced.

So the production row has no working path on railiance01. Until one exists, the transition rule applies to every production-tier target there. Installing ArgoCD on railiance01 is a cluster-layer change and needs its own founder go-ahead. It is not part of this decision.

What the records said before the check

Whether ArgoCD runs on railiance01 at all is unverified, and railiance-platform's own records point the other way:

• Its later records name the controller "coulombcore ArgoCD".
• Its only health record, "healthy on railiance01", predates the 2026-07-02 correction, before which "railiance01" often meant coulombcore.
• No evidence records railiance-apps-root syncing on 92.205.62.239.

If ArgoCD is not on railiance01, the production row has no working path there yet. Until one exists, production-tier changes on railiance01 fall under the transition rule in the next paragraph, and installing ArgoCD on railiance01 is a cluster-layer change needing its own go-ahead. railiance-platform RPF-WP-0043-T01 holds the read-only check.

Until the production row has a working path, the policy-nexus transition rule applies to every production-tier target on railiance01, not only to policy-nexus: activation=APPROVED, with each change recorded as production-tier.

Owners

• gate-house: the open question is answered. No engine owns the contact, and the gate is quality, not authorization.
• ops-mason: phase 4 checks the target's readiness state and refuses a direct apply to a production-tier target, except under BREAK_GLASS or the policy-nexus transition.
• railiance-platform: the ArgoCD lane is the production path, and policy-nexus onboarding is due by 2026-12-21.
• railiance-master: ADR-0006 gains a deploy-path consequence for production-approved.
• rapp-policy-nexus: the transition and its date.

ArgoCD installed on railiance01 (phase A), 2026-09-21

Decided by: Bernd Worsch (founder), exercising GOVERN @ estate. This settles railiance-platform ArchitectureBlueprint §5.6 as option 1, adopt properly: ArgoCD runs on railiance01, and the live workloads are brought under GitOps one application at a time.

What was installed: upstream Argo CD Core (headless: no API server, UI or Dex), v3.5.3.

• Manifest: core-install.yaml at tag v3.5.3, sha256 1a87025d8eb2eae621653fd312fb9ca51df1b4b3b6992a030e3a9ef38e45c448.
• Images: quay.io/argoproj/argocd:v3.5.3 and public.ecr.aws/docker/library/redis:8.2.3-alpine.
• Namespace argocd, created 2026-09-21T13:55:29Z.
• 34 objects server-side applied after a clean server-side dry run.

Verified after the install:

• all four pods are Ready with 0 restarts;
• there are 0 Applications and 0 AppProjects;
• 0 namespaces carry an ArgoCD instance label, so nothing is adopted;
• no Service is of type LoadBalancer or NodePort, so there is no new listener (ADR-0008).

The mode it was done in: ADMINISTER @ realm:kubernetes/railiance01, activation=APPROVED by the founder, performed by the custodian session over SSH. By this record's own table the ArgoCD install is a platform object with no readiness state, so it is production-tier. It was done by direct apply under the transition rule, because the production row did not yet exist: this install is what creates it. Its declared state now has to be captured in its owning repository, so the install is not itself a hand-deployed object that nothing declares.

Rollback: delete namespace argocd and the three argoproj.io CRDs. With no Applications, nothing else is affected.

Resource hygiene, open: upstream core sets no resource requests or limits, so the ArgoCD pods are BestEffort. That makes them the first evicted under pressure, and unbounded. Set requests and limits when the install is captured in its owning repository.

Phase B, adopting each application, and phase C, retiring coulombcore's ArgoCD, each need their own founder go-ahead. Phase B starts with the AppProjects and the root app with automated sync off. Then, per application, a diff, then a manual sync with prune off, and prune and self-heal only after that application is proven.

Found during phase A: the node's CPU requests are exhausted

Before the install, railiance01 already had 100% of its allocatable CPU requested (4 of 4 cores); limits are at 828%. Short scheduled jobs cannot be placed, and have been failing with Insufficient cpu for days:

Pod	Pending since
user-engine database backup	2026-09-16
authelia backup	2026-09-20
lldap backup	2026-09-20
privacyidea backup	2026-09-20
audit-core attest-chain	2026-09-20
activity-core cert-manager ACME solver	2026-09-19

ArgoCD did not cause this: it requests no CPU. It is an existing capacity defect with a direct consequence: identity and user-engine backups have not run. The custodian has not changed anything; it is reported to the founder.

CPU requests released, 2026-09-21

Founder go-ahead, 2026-09-21. The mode was ADMINISTER @ realm:kubernetes/railiance01 with activation=APPROVED, performed by the custodian session over SSH. Both targets are non-production tier: rail-knative's readiness is verified, and the stuck drill pod belongs to a drill.

Changed. Only CPU requests were changed; limits are unchanged.

Deployment	Before	After
knative-serving/activator	300m	50m
kourier-system/3scale-kourier-gateway	200m	50m
knative-serving/net-kourier-controller	200m	30m
knative-serving/controller	100m	30m
knative-serving/webhook	100m	30m
knative-serving/autoscaler	100m	30m

The HPAs on the activator, gateway and webhook target 100% of the request. At observed use they now read about 2–10%. The stuck pod platform-pg-drill/platform-pg-drill-1 was force-deleted; its deletion had been requested on 2026-08-13 and it had no finalizers.

What happened. The rolling updates deadlocked at first. The new pods are created before the old ones stop, and on a node with no requestable CPU none of the new pods could be scheduled. Force-deleting the drill pod freed 50m, which let the first new pod schedule; its old pod then stopped, and the rest followed. Lesson: on a node whose requests are exhausted, free capacity before changing a Deployment's requests, or a request cut cannot roll out.

Verified afterwards:

• all six rollouts are complete, with no old pods left;
• ksvc/rapp-qonto is Ready;
• 0 Pending pods cluster-wide;
• the authelia, lldap, privacyidea and user-engine backups and audit-core's attest-chain have Completed, and fresh backups ran within minutes;
• node CPU is at 69% actual use.

Open:

• rail-knative must declare these requests, or the next upstream apply resets them.
• Actual CPU use is about 70% on a single node. That is a capacity question for the railiance02 move, not something to settle here.
• state-hub uses about 447m against a 100m request. It is under-requested.

State Hub request restored, 2026-09-21

State Hub's CPU request was 100m → 250m, the chart's own default. The mode was ADMINISTER @ realm:kubernetes/railiance01, activation=APPROVED by the founder, done through state-hub's own promotion path. The 100m had been a declared override in deploy/railiance/apps/helm/state-hub-values.yaml, working around a node with less than 250m unallocated; that constraint is gone.

Steps:

1. Changed the value in state-hub (commit 6b97d01).
2. On railiance01, rendered the chart and diffed it against the live objects. The only substantive change was cpu: 100m → 250m, and the chart had not drifted from the live revision 63.
3. Ran the chart's headroom preflight: ok: true, 830m remaining.
4. Ran helm upgrade --atomic with the pinned image main-0747b4d, giving revision 64, 1/1 ready.

Hub health afterwards: ok, db connected, schema b5e6f7a8c9d0 as expected. There was no downtime (maxUnavailable=0).

Phase B begun, and external-secrets repaired, 2026-09-21

All of the following were done with the founder's go-ahead, in ADMINISTER @ realm:kubernetes/railiance01 with activation=APPROVED, by the custodian session.

1. ArgoCD resources (railiance-enablement RAIL-EN-WP-0002-T02). Requests and limits were applied from the declared kustomization. All four pods are now Burstable, Ready, with 0 restarts. Free capacity was checked first, with 680m free, so the rollouts did not deadlock.

2. Phase B layout (RPF-WP-0044-T01). The founder chose option A: a railiance01-only path, argocd/railiance01/. Nothing under argocd/applications/ or argocd/bootstrap/, which coulombcore's ArgoCD reads, was changed. Hazard: make argocd-bootstrap-deploy still renders the old root with automated sync and must not be run on railiance01.

3. T02 and T03.

• T02: 3 AppProjects and a root app with automated sync off were applied at c3ebd6d. The root is Synced/Healthy with zero children, which shows the repo-server reaches Forgejo.

• T03: openbao-secretstore was adopted:

  • the diff was rc=0;
  • the child was merged in 182e788;
  • a manual sync at d2dbc19, with the apply strategy and prune off, succeeded;
  • ClusterSecretStore/openbao is Valid and tracked, and issue-core-runtime is SecretSynced.

  It is the first workload under GitOps on railiance01. It is now in its 24-hour proving period. selfHeal needs a second go-ahead, and prune a third.

4. external-secrets Helm release. It had been stuck in pending-upgrade (rev 7) since 2026-07-07. Rev 6 and rev 7 were identical in values and manifests, and live matched both exactly, so the upgrade had fully applied and only the status was stuck. It was fixed with helm rollback external-secrets 6: dry run first, the chart has no hooks, and the result is rev 8, deployed. All three pods kept their names and their 36-day age, so nothing restarted.

Found and routed, not fixed: two ClusterSecretStores have been failing since 2026-09-13 (openbao-email-connect) and 2026-09-19 (openbao-activity-core). Their static OpenBao tokens are rejected with 403 on lookup-self, so 5 of 37 ExternalSecrets are not syncing, including activity-core's offsite-backup credential. This is credential custody, so it was routed to railiance-platform, activity-core and email-connect. No token was read or minted.

target-revenue adopted, and a credential-exposure finding, 2026-09-21

RPF-WP-0044-T05, target-revenue. Adopted with the founder's go-ahead.

• The server-side diff conflicted on limits.cpu (1 vs 1000m, the same quantity, owned by an old client-side apply). The client-side diff, which is what the apply strategy uses, was rc=0.
• A manual sync with the apply strategy and prune off succeeded, so 0 hook Jobs ran against the production database.
• The pod is unchanged, CNPG is healthy, and /healthz returns 200.
• Automated sync waits on target-revenue confirming its two Sync-hook Jobs are safe to re-run.

RPF-WP-0045. The runbook for moving openbao-activity-core and openbao-email-connect to OpenBao Kubernetes auth, as the three recovered stores already use, is written. Nothing is applied. One step needs the OpenBao admin credential and is the founder's to run.

Credential exposure (routed to railiance-platform). Secrets created with kubectl apply carry their full data in the kubectl.kubernetes.io/last-applied-configuration annotation, so a metadata-only read prints the secret. This was observed on the two dead ESO tokens. It is suspected, not verified, on five live *-eso-token Secrets. The custodian did not inspect them. Remediation — a check that prints no value, stripping the annotation, deciding on rotation, and fixing the creation paths — belongs to custody.

ESO stores recovered, 2026-09-21 (RPF-WP-0045 T01–T04)

openbao-activity-core and openbao-email-connect were moved off their dead static tokens to OpenBao Kubernetes auth. Each role is bound to one ServiceAccount in one namespace, issues 15-minute tokens, and carries a single policy scoped to exactly the paths its ExternalSecrets read. The founder's go-ahead covered T01, T03 and T04.

T01 was run through warden access openbao-platform-admin-login --exec. It took four more attempts to succeed:

1. The child's output made warden fail closed; the lane requires a silent child.
2. BAO_ADDR pointed at an unreachable public name; the operator path is the tunnel at 127.0.0.1:18200.
3. WSL had no browser opener; fixed with a ~/.local/bin/xdg-open shim to explorer.exe.
4. The fifth run succeeded, but warden reported the session revocation "could not be confirmed". It also exits 0 on a failed login and audits every attempt as ok.

These are sent to ops-warden as one improvement suggestion (6a1ce1bb). The undeclared platform-admin role and policy, and the suspected missing revoke-self capability, are sent to railiance-platform (641673a4).

Result: both stores are Ready, the five stalled ExternalSecrets synced at 17:39Z (activity-core's offsite-backup credential among them), and 37/37 ExternalSecrets are ready cluster-wide. The two dead token Secrets are deleted in T06, after 24 hours of clean syncs.