--- id: RPF-WP-0019 type: workplan title: "apps-pg: backup, per-consumer controls, and the isolation probes they make possible" domain: financials repo: railiance-platform status: active owner: codex topic_slug: railiance created: "2026-08-17" updated: "2026-08-18" related: - RPF-WP-0018 origin: residual origin_ref: RPF-WP-0018 state_hub_workstream_id: "160e226d-27b0-4c94-9e99-331f4354dd12" --- # RPF-WP-0019 — apps-pg recoverability and per-consumer controls ## Goal Close the three defects `RPF-WP-0018` surfaced in `apps-pg` by writing its quota disclosure. Documentation found them; only this workplan fixes them. ## Why this is separate from RPF-WP-0018 That workplan declared posture and policy. This one changes a live cluster. Keeping them apart matters: a declaration workplan that quietly starts editing production is how "we wrote it down" becomes indistinguishable from "we fixed it". The declaration is published as-is, with the defects visible, and this is the record of closing them. ## The three defects **D1 — `apps-pg` has no backup.** No `barmanObjectStore`, no `retentionPolicy`, nothing. This is not a short retention window; it is no recovery path at all, on a cluster holding two S5 application databases. Its R level is `R0` and `R0` here means unrecoverable, not merely un-erasable. **D2 — no per-consumer controls.** The connection pool is unpartitioned, so one consumer can exhaust the cluster while staying politely inside its own expectations. No `statement_timeout`, no `idle_in_transaction_session_timeout`, no CPU or memory limits — the pod is BestEffort QoS and is the first thing evicted under node pressure. **D3 — no isolation probes**, so the `P1` levels recorded for `vergabe` and `coulomb_social` in `docs/placement-policy.md` §3.1 are provisioning declarations without the §13 artifact. ## Sequencing, and why it is not the obvious one **D1 first.** It is the only one whose failure is unrecoverable. A cluster with no backup is one bad afternoon from data loss that no amount of isolation evidence compensates for. **D2 before D3, necessarily.** Tenancy Posture §13.4: an artifact must assert something achievable. The noisy-neighbour artifact requires showing the governance controls *bind*. With no controls there is nothing to bind, so a probe written now could only demonstrate degradation — an artifact that "can only fail, or that passes by being run gently enough". Writing the probe first would produce an overclaim wearing the costume of evidence. D1 also depends on a backup target, which is `resource-control`'s bucket and the `platform-pg-backup-s3` credential lane — the same handoff `make postgres-backup-deploy` waits on. Check whether that is now live before assuming this is blocked. ## Status 2026-08-18 — repository-complete, live-blocked Everything this repo can do without touching the cluster is done and committed. What remains on T01, T02 and T04 is a single operator window against a live shared rail, in this order: 1. `make apps-pg-deploy` — Cluster reconcile: role connection limits, Burstable requests/limits, explicit aggregate parameters. 2. Apply `helm/apps-pg-consumer-controls.sql` — the two 15s role timeouts. Idempotent; CNPG 1.28 has no managed-role settings field, so this is operator SQL by necessity, not by preference. 3. `make apps-pg-backup-deploy` — the ScheduledBackup, once the governed Secret is confirmed live. 4. Capture `LastBackupSucceeded=True` and a scratch restore. **Until both exist, `apps-pg` R stays 0** — declared configuration is not a §13 artifact, and this workplan exists because that distinction was missed once already. 5. T04's probes, in an announced window, after 1–3 have settled. **Blocker resolved 2026-08-18, and it was ours.** The earlier note recorded this as `kubectl` returning `Unauthorized` — an access problem outside the repo. That was wrong. railiance01 is reachable and healthy (`~/.kube/config-railiance01`, k3s v1.35.1, `apps-pg` 9d, both consumers present). What failed was our own wiring: `KUBECONFIG` is an environment variable, so the Makefile's `?=` default never applied, and `RAILIANCE01_KUBECONFIG` pointed at `config-hosteurope` — a different cluster that happens to be unauthorized from here. **The near-miss is the finding.** Two reachable clusters each carry a CNPG `Cluster` named `apps-pg` in a namespace named `databases`. The other one (k3s v1.30.3) holds `gitea-db` and only one apps-pg consumer. Had `KUBECONFIG` pointed there instead of at an unauthorized file, `make apps-pg-deploy` would have applied this workplan's connection limits, role timeouts and backup configuration **to the wrong cluster, and reported success.** The `Unauthorized` error was the only thing that prevented it. Fixed by pinning cluster *identity* rather than kubeconfig *filename*: `railiance01-guard` compares the live `kube-system` namespace UID against `RAILIANCE01_CLUSTER_UID` and fails closed on mismatch or unreachability. It gates `apps-pg-deploy`, `apps-pg-backup-deploy`, `apps-pg-overflow-dry-run`, `apps-pg-status` and `apps-pg-shell`. Filename selection could not have protected against this: both kubeconfigs resolve to a `127.0.0.1` tunnel port, and the environment overrides the default either way. `make cluster-id` prints what is currently selected. The guard is deliberately **not** global. `db-status` legitimately targets the other cluster for `gitea-db`, so a blanket guard would break a working target and teach people to bypass it. **Do not treat the rollout as evidence.** T04's P1 claim and the R-axis both need artifacts produced *after* application, and `docs/placement-policy.md` §3.1 and `tenancy.yaml` should be updated only then. ## Tasks ```task id: RPF-WP-0019-T01 status: progress priority: high state_hub_task_id: "0f5175c7-bde3-46a0-87a8-b50b5eeac32c" ``` **Establish a backup target for `apps-pg`.** Confirm the state of the `resource-control` bucket and the `platform-pg-backup-s3` OpenBao Secret; if live, configure `barmanObjectStore` and a `retentionPolicy` on the cluster. If not live, record the dependency and say so — do not leave the absence undocumented a second time. 2026-08-18 repository readiness: the governed Secret exists live and reviewed desired state now carries a distinct `apps-pg/` prefix, 30-day retention, continuous WAL and a daily 02:15 backup. NetworkPolicy admits the DNS/HTTPS path archival needs. The ScheduledBackup is not applied and no successful backup/restore evidence exists, so T01 remains progress. ```task id: RPF-WP-0019-T02 status: progress priority: high state_hub_task_id: "c5b0b2ac-b9f5-42fb-8e75-2fac2ab1e852" ``` **Declare and enforce per-consumer controls.** Per-consumer connection allowance, `statement_timeout`, `idle_in_transaction_session_timeout`, and pod resource requests/limits to lift `apps-pg` off BestEffort QoS. Publish every value in `docs/s3-consumer-interfaces.md` before it takes effect — §10.2 is a disclosure rule, and applying a timeout consumers learn about by hitting it would breach the rule while implementing it. 2026-08-18 repository readiness: both roles declare a 20-connection limit, the pod has Burstable requests/limits, aggregate/logging parameters are explicit, and controlled operator SQL sets both 15s role timeouts. Every value was published in `docs/s3-consumer-interfaces.md` before application. Live SQL and Cluster reconciliation remain an operator window, so T02 remains progress. ```task id: RPF-WP-0019-T03 status: done priority: medium state_hub_task_id: "736cbc11-1992-4f1b-9ff0-cb4622ff39a5" ``` **Declare the ceiling and overflow target.** Owed under this repo's own Rule P-4.1 before `apps-pg`'s third consumer; it is at two. Name the binding resource per Rule P-4.2 — memory or connections — and a named overflow substrate per P-4.3. Completed 2026-08-18. The declared ceiling is three, memory is the binding constraint, and `apps-pg-2` is a named, source-provisionable overflow cell with a distinct credential and backup prefix. `make apps-pg-verify-capacity` rejects a fourth consumer per cell and unbounded/duplicate roles. The cell intentionally remains absent until a fourth consumer is approved. ```task id: RPF-WP-0019-T04 status: wait priority: medium state_hub_task_id: "2fccdd26-c9c6-43cb-b069-27ed5668afac" ``` **Isolation probes, after T02.** Consumer-boundary probes on the `rapp-postgres` model, then the §13 noisy-neighbour artifact: per-consumer baseline, saturation run, evidence the controls bind, degradation measured and judged against each consumer's declared service class. Update `docs/placement-policy.md` §3.1 and `docs/tenancy-posture.md` when the P1 claims become evidenced. Waiting on T02 live application and an announced probe window. No saturation or destructive recovery experiment is run against the shared production rail as part of repository preparation. ## Boundaries - `apps-pg` only. `platform-pg`'s equivalents are `rapp-postgres`'s. - No consumer is migrated. This changes the cluster, not who is on it. - Values are published before they are enforced, never after. ## Risks **Applying limits to a live cluster breaks a consumer that was relying on their absence.** Most likely with `statement_timeout`. Mitigation is the disclosure-first ordering in T02, which gives consumers a window to object. **T01 stays blocked on a handoff outside this repo and the cluster keeps no backup meanwhile.** Mitigation is that T01 requires the dependency be recorded explicitly rather than left as a silent absence — which is exactly how D1 survived this long.