Pin apps-pg targets to railiance01 by cluster identity; seed RPF-WP-0020
All checks were successful
CI Smoke / host-smoke (push) Successful in 0s
CI Smoke / container-smoke (push) Successful in 1s

Two reachable clusters each carry a CNPG Cluster named apps-pg in a
namespace named databases. KUBECONFIG is an environment variable, so the
Makefile ?= default never applied, and RAILIANCE01_KUBECONFIG pointed at
config-hosteurope - a different cluster. Had the environment pointed at the
other reachable cluster instead of an unauthorized one, make apps-pg-deploy
would have applied RPF-WP-0019 connection limits, role timeouts and backup
config to the wrong cluster and reported success. The Unauthorized error was
the only thing that prevented it.

Filename selection cannot protect against this: both kubeconfigs resolve to
a 127.0.0.1 tunnel port and the environment wins either way. railiance01-guard
pins identity instead, comparing the live kube-system namespace UID against
RAILIANCE01_CLUSTER_UID, and fails closed on mismatch or unreachability. It
gates apps-pg deploy, backup-deploy, overflow-dry-run, status and shell.
Verified refusing on the wrong cluster, refusing when unreachable, and
passing on railiance01. Not global: db-status legitimately targets the other
cluster for gitea-db.

RPF-WP-0019 blocker note corrected - the cluster was never unreachable, our
wiring was wrong.

RPF-WP-0020 seeded for the pre-existing CCR test failure, which is two
unrelated problems: CCR-2026-0010 is an active lane missing its whole
openbao.auth block, and CCR-2026-0011 is an honest in-flight draft the suite
has no way to express.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
codex 2026-08-18 15:18:37 +02:00
parent 1cbde550b5
commit b1f973c2d5
4 changed files with 227 additions and 13 deletions

View file

@ -1,9 +1,25 @@
SHELL := /usr/bin/env bash
.DEFAULT_GOAL := help
KUBECONFIG ?= $(firstword $(wildcard $(HOME)/.kube/config) $(HOME)/.kube/config-hosteurope)
KUBECONFIG ?= $(firstword $(wildcard $(HOME)/.kube/config-railiance01) $(wildcard $(HOME)/.kube/config))
KUBECTL_BIN ?= $(firstword $(shell command -v kubectl 2>/dev/null) $(wildcard $(HOME)/.local/bin/kubectl) kubectl)
KUBECTL := $(KUBECTL_BIN) --kubeconfig=$(KUBECONFIG)
# Cluster identity pin (RPF-WP-0019).
#
# Two reachable clusters each carry a CNPG Cluster named `apps-pg` in a
# namespace named `databases`. Selecting by kubeconfig FILENAME does not
# protect against applying to the wrong one: KUBECONFIG is an environment
# variable, `?=` does not override it, and both files resolve to a
# 127.0.0.1 tunnel port. So the guard pins cluster IDENTITY instead —
# the kube-system namespace UID, which is unique per cluster and stable
# for its lifetime.
#
# railiance01 (k3s v1.35.1) carries platform-pg, forgejo-db and both
# apps-pg consumers. The other cluster carries gitea-db and only one
# apps-pg consumer; `db-status` legitimately targets it, which is why the
# guard is applied per-target rather than globally.
RAILIANCE01_CLUSTER_UID ?= a553c742-0115-43d4-99a4-a5ca56fe0786
HELM := helm --kubeconfig=$(KUBECONFIG)
NAMESPACE := platform
@ -82,10 +98,31 @@ consumption-preflight: ## Refuse a new order that exceeds a restricted entity's
##@ Shared apps-pg (S5 application databases)
cluster-id: ## Print the kube-system UID of the cluster KUBECONFIG currently selects
@echo "kubeconfig: $(KUBECONFIG)"
@echo "cluster-uid: $$($(KUBECTL) get ns kube-system -o jsonpath='{.metadata.uid}' 2>/dev/null || echo UNREACHABLE)"
railiance01-guard: ## Fail closed unless KUBECONFIG selects railiance01
@uid=$$($(KUBECTL) get ns kube-system -o jsonpath='{.metadata.uid}' 2>/dev/null); \
if [ -z "$$uid" ]; then \
echo "railiance01-guard: cluster unreachable via $(KUBECONFIG)" >&2; \
echo " hint: unset KUBECONFIG, or set it to ~/.kube/config-railiance01" >&2; \
exit 2; \
fi; \
if [ "$$uid" != "$(RAILIANCE01_CLUSTER_UID)" ]; then \
echo "railiance01-guard: REFUSING - wrong cluster" >&2; \
echo " kubeconfig: $(KUBECONFIG)" >&2; \
echo " expected: $(RAILIANCE01_CLUSTER_UID)" >&2; \
echo " found: $$uid" >&2; \
echo " both clusters carry apps-pg in namespace databases; applying here would hit the wrong one." >&2; \
exit 2; \
fi; \
echo "railiance01-guard: ok ($$uid)"
apps-pg-verify-capacity: ## Verify cell ceiling, role limits and distinct backup prefixes
python3 tools/verify_apps_pg_capacity.py helm/apps-pg-cluster.yaml helm/apps-pg-2-cluster.yaml
apps-pg-deploy: apps-pg-verify-capacity ## Apply shared apps-pg cnpg Cluster + NetworkPolicies
apps-pg-deploy: railiance01-guard apps-pg-verify-capacity ## Apply shared apps-pg cnpg Cluster + NetworkPolicies
@if [ -n "$(CONSUMING_ENTITY)" ]; then \
$(MAKE) consumption-preflight CONSUMING_ENTITY='$(CONSUMING_ENTITY)' \
ESTIMATE_EUR='$(ESTIMATE_EUR)' CONSUMPTION_CLASS=new-order; \
@ -93,11 +130,11 @@ apps-pg-deploy: apps-pg-verify-capacity ## Apply shared apps-pg cnpg Cluster + N
$(KUBECTL) apply -f helm/apps-pg-cluster.yaml
$(KUBECTL) apply -f helm/apps-pg-networkpolicies.yaml
apps-pg-backup-deploy: ## Apply the daily backup after the governed S3 Secret exists
apps-pg-backup-deploy: railiance01-guard ## Apply the daily backup after the governed S3 Secret exists
@$(KUBECTL) get secret platform-pg-backup-s3 -n databases >/dev/null || { echo 'missing governed platform-pg-backup-s3 secret' >&2; exit 2; }
$(KUBECTL) apply -f helm/apps-pg-backup.yaml
apps-pg-overflow-dry-run: apps-pg-verify-capacity ## Server-validate the unapplied overflow cell
apps-pg-overflow-dry-run: railiance01-guard apps-pg-verify-capacity ## Server-validate the unapplied overflow cell
@$(KUBECTL) get secret apps-pg-2-credentials -n databases >/dev/null || { echo 'missing distinct apps-pg-2-credentials secret' >&2; exit 2; }
$(KUBECTL) apply --dry-run=server -f helm/apps-pg-2-cluster.yaml
$(KUBECTL) apply --dry-run=server -f helm/apps-pg-2-networkpolicies.yaml
@ -108,11 +145,11 @@ apps-pg-overflow-deploy: apps-pg-overflow-dry-run ## Provision only for an appro
$(KUBECTL) apply -f helm/apps-pg-2-networkpolicies.yaml
$(KUBECTL) apply -f helm/apps-pg-2-backup.yaml
apps-pg-status: ## Show apps-pg cnpg cluster health
apps-pg-status: railiance01-guard ## Show apps-pg cnpg cluster health
$(KUBECTL) cnpg status apps-pg -n databases 2>/dev/null || \
$(KUBECTL) get cluster apps-pg -n databases -o wide
apps-pg-shell: ## Open psql shell on apps-pg primary as apps_admin / apps_meta
apps-pg-shell: railiance01-guard ## Open psql shell on apps-pg primary as apps_admin / apps_meta
$(KUBECTL) cnpg psql apps-pg -n databases -- -U apps_admin apps_meta 2>/dev/null || \
$(KUBECTL) exec -it -n databases apps-pg-1 -- psql -U apps_admin apps_meta
@ -383,7 +420,7 @@ argocd-status: ## Show Railiance ArgoCD projects, root app, and registered repos
##@ Backup
RAILIANCE01_KUBECONFIG ?= $(HOME)/.kube/config-hosteurope
RAILIANCE01_KUBECONFIG ?= $(HOME)/.kube/config-railiance01
backup: ## Backup platform services (PostgreSQL logical dump) — age-encrypted to Nextcloud
@test -x tools/cmd/railiance-backup || { echo "tools/cmd/railiance-backup not installed; use forgejo-backup for Forgejo." >&2; exit 1; }

View file

@ -18,6 +18,30 @@ databases. It lives in the `databases` namespace and is owned by
into application namespaces, use it in S5 runtime configuration, or treat
it as a consumer credential.
## Which cluster this is
**Two reachable clusters each carry a CNPG `Cluster` named `apps-pg` in a
namespace named `databases`.** The one this document describes is
**railiance01** (k3s v1.35.1) — it also carries `platform-pg` and `forgejo-db`,
and holds both apps-pg consumers (`vergabe_db`, `coulomb_social_db`). The other
cluster carries `gitea-db` and only one apps-pg consumer.
Selecting the right one by kubeconfig filename is not safe: `KUBECONFIG` is an
environment variable that overrides the Makefile default, and both files
resolve to a `127.0.0.1` tunnel port. Every `apps-pg-*` target therefore runs
`railiance01-guard` first, which compares the live `kube-system` namespace UID
against `RAILIANCE01_CLUSTER_UID` and fails closed on mismatch or
unreachability.
```bash
make cluster-id # what KUBECONFIG currently selects
make railiance01-guard # assert it is railiance01, or refuse
```
If a target refuses, the fix is `KUBECONFIG=~/.kube/config-railiance01`, not
`--force` and not editing the pinned UID. `db-status` deliberately targets the
other cluster for `gitea-db`, which is why the guard is per-target.
## Consumer Onboarding
Each S5 application gets its own role, database, and runtime Secret. The

View file

@ -85,12 +85,36 @@ against a live shared rail, in this order:
already.
5. T04's probes, in an announced window, after 13 have settled.
**Blocker as of this session:** the cluster is unreachable from the
workstation — `kubectl` returns `Unauthorized` against
`config-hosteurope`. Credentials or the tunnel need attention before any of
the above runs. This is a session-local access problem, not a defect in the
manifests: `make apps-pg-verify-capacity` passes and the capacity tests are
green.
**Blocker resolved 2026-08-18, and it was ours.** The earlier note recorded
this as `kubectl` returning `Unauthorized` — an access problem outside the
repo. That was wrong. railiance01 is reachable and healthy
(`~/.kube/config-railiance01`, k3s v1.35.1, `apps-pg` 9d, both consumers
present). What failed was our own wiring: `KUBECONFIG` is an environment
variable, so the Makefile's `?=` default never applied, and
`RAILIANCE01_KUBECONFIG` pointed at `config-hosteurope` — a different cluster
that happens to be unauthorized from here.
**The near-miss is the finding.** Two reachable clusters each carry a CNPG
`Cluster` named `apps-pg` in a namespace named `databases`. The other one
(k3s v1.30.3) holds `gitea-db` and only one apps-pg consumer. Had `KUBECONFIG`
pointed there instead of at an unauthorized file, `make apps-pg-deploy` would
have applied this workplan's connection limits, role timeouts and backup
configuration **to the wrong cluster, and reported success.** The
`Unauthorized` error was the only thing that prevented it.
Fixed by pinning cluster *identity* rather than kubeconfig *filename*:
`railiance01-guard` compares the live `kube-system` namespace UID against
`RAILIANCE01_CLUSTER_UID` and fails closed on mismatch or unreachability. It
gates `apps-pg-deploy`, `apps-pg-backup-deploy`, `apps-pg-overflow-dry-run`,
`apps-pg-status` and `apps-pg-shell`. Filename selection could not have
protected against this: both kubeconfigs resolve to a `127.0.0.1` tunnel port,
and the environment overrides the default either way. `make cluster-id` prints
what is currently selected.
The guard is deliberately **not** global. `db-status` legitimately targets the
other cluster for `gitea-db`, so a blanket guard would break a working target
and teach people to bypass it.
**Do not treat the rollout as evidence.** T04's P1 claim and the R-axis both
need artifacts produced *after* application, and `docs/placement-policy.md`

View file

@ -0,0 +1,129 @@
---
id: RPF-WP-0020
type: workplan
title: "Close CCR schema drift: one active lane unmigrated, one draft the suite cannot express"
domain: financials
repo: railiance-platform
status: proposed
owner: codex
topic_slug: railiance
created: "2026-08-18"
updated: "2026-08-18"
related:
- RPF-WP-0014
origin: residual
origin_ref: RPF-WP-0019
---
# RPF-WP-0020 — CCR schema drift
## Goal
Make `tests/test_credential_change.py::test_all_repo_ccrs_validate` pass for
the right reason: because every credential-change request in the repo is
either valid or explicitly declared in-flight — not because the assertion was
loosened until it stopped complaining.
## Why this is its own workplan
The test has been failing on `main`. It was found while closing
`RPF-WP-0019` and confirmed to predate that work (it fails at `HEAD` with the
apps-pg changes stashed), so it is neither a regression from that workplan nor
something to fix inside it.
A single red test is hiding **two unrelated problems with opposite remedies**.
That is the reason to split them out and name them, rather than fix whichever
one makes the suite green first.
## The two problems
**P1 — an active credential lane is unmigrated.**
`credential-change-requests/CCR-2026-0010-email-connect-transactional.yaml`
carries `status: active` and `readiness: ready`, and is missing the entire
`openbao.auth` block: `method`, `mount`, `role`, `policies`, `bound_claims`.
The validator gained those requirements and this CCR was never brought
forward.
This is the one that matters. A live lane whose declaration does not describe
how the workload authenticates is a governance gap, not a lint failure — the
document that is supposed to be the authority on the lane cannot answer the
first question anyone would ask of it. The lane itself is presumably working,
which is exactly what makes it easy to leave.
**P2 — a genuine in-flight draft the suite cannot express.**
`CCR-2026-0011-scaleway-object-storage-bootstrap.yaml` carries
`status: apply_pending` and `readiness: waiting-on-ui-replace-of-xxx-placeholders`.
It is a founder-bootstrap credential still holding placeholder values, and
`ops-warden` already tracks it as a draft lane. Its errors include a
`readiness` value outside the permitted enum, which is the file honestly
reporting that it is not finished.
This one is not a defect. The defect is that a suite asserting *all* CCRs
validate has no way to say "this one is deliberately incomplete", so an honest
draft and a real gap produce identical output.
## Boundaries
- **No secret values are read, written or rotated.** This is declaration
hygiene. Anything requiring a mint, a rotation or an apply belongs to the
credential-change approval flow in `docs/credential-change-approval.md`.
- P2 is not closed by filling in placeholders. The Scaleway bootstrap key is
a live commercial credential with an owner outside this repo.
- The validator's rules are not relaxed to accommodate either file.
## Tasks
```task
id: RPF-WP-0020-T01
status: todo
priority: high
```
**Migrate CCR-2026-0010 to the current schema.** Determine the actual
authentication path for the `email-connect` transactional lane — Kubernetes
auth mount, role, bound claims, and the policy name that must match
`openbao.policy_name` — from the live OpenBao configuration and the existing
policy file, not by inventing plausible values. If the live configuration and
the declaration disagree, the live configuration is the fact and the
disagreement is the finding.
```task
id: RPF-WP-0020-T02
status: todo
priority: medium
```
**Give the suite a way to express a draft.** Add an explicit in-flight state
so `CCR-2026-0011` is skipped *by declaration* rather than by exception list —
a `status` the validator recognises as not-yet-complete, with the test
asserting that such files are still well-formed in every other respect. An
allowlist of filenames would work today and rot on the next draft.
```task
id: RPF-WP-0020-T03
status: todo
priority: medium
```
**Report the drift rather than only fixing it.** If the `openbao.auth`
requirement was added without migrating existing active CCRs, other repos
carrying CCRs may have the same gap and no failing test to reveal it. Confirm
whether the requirement originated here or upstream, and notify accordingly.
```task
id: RPF-WP-0020-T04
status: todo
priority: low
```
**Make the suite green and keep it that way.** With T01 and T02 done, the full
suite passes. Record in `docs/credential-change-approval.md` that a new
required field obliges a migration pass over existing active CCRs — the
omission that produced P1.
## Risks
**T01 invents values to make the test pass.** The likeliest failure and the
worst one: a declaration that is well-formed and wrong is more dangerous than
one that is visibly incomplete, because it stops anyone looking. Mitigation is
that T01 reads live configuration and treats disagreement as a finding.
**T02 becomes a way to silence future failures.** Mitigation is that the
in-flight state still requires the file to be well-formed in every other
respect, so it suppresses the completeness assertion and nothing else.