railiance-platform/workplans/RPF-WP-0019-apps-pg-recoverability-and-controls.md

---
id: RPF-WP-0019
type: workplan
title: "apps-pg: backup, per-consumer controls, and the isolation probes they make possible"
domain: financials
repo: railiance-platform
status: active
owner: codex
topic_slug: railiance
created: "2026-08-17"
updated: "2026-08-18"
related:
  - RPF-WP-0018
origin: residual
origin_ref: RPF-WP-0018
state_hub_workstream_id: "160e226d-27b0-4c94-9e99-331f4354dd12"
---

# RPF-WP-0019 — apps-pg recoverability and per-consumer controls

## Goal

Close the three defects `RPF-WP-0018` surfaced in `apps-pg` by writing its
quota disclosure. Documentation found them; only this workplan fixes them.

## Why this is separate from RPF-WP-0018

That workplan declared posture and policy. This one changes a live cluster.
Keeping them apart matters: a declaration workplan that quietly starts editing
production is how "we wrote it down" becomes indistinguishable from "we fixed
it". The declaration is published as-is, with the defects visible, and this is
the record of closing them.

## The three defects

**D1 — `apps-pg` has no backup.** No `barmanObjectStore`, no `retentionPolicy`,
nothing. This is not a short retention window; it is no recovery path at all,
on a cluster holding two S5 application databases. Its R level is `R0` and
`R0` here means unrecoverable, not merely un-erasable.

**D2 — no per-consumer controls.** The connection pool is unpartitioned, so
one consumer can exhaust the cluster while staying politely inside its own
expectations. No `statement_timeout`, no `idle_in_transaction_session_timeout`,
no CPU or memory limits — the pod is BestEffort QoS and is the first thing
evicted under node pressure.

**D3 — no isolation probes**, so the `P1` levels recorded for `vergabe` and
`coulomb_social` in `docs/placement-policy.md` §3.1 are provisioning
declarations without the §13 artifact.

## Sequencing, and why it is not the obvious one

**D1 first.** It is the only one whose failure is unrecoverable. A cluster
with no backup is one bad afternoon from data loss that no amount of isolation
evidence compensates for.

**D2 before D3, necessarily.** Tenancy Posture §13.4: an artifact must assert
something achievable. The noisy-neighbour artifact requires showing the
governance controls *bind*. With no controls there is nothing to bind, so a
probe written now could only demonstrate degradation — an artifact that "can
only fail, or that passes by being run gently enough". Writing the probe first
would produce an overclaim wearing the costume of evidence.

D1 also depends on a backup target, which is `resource-control`'s bucket and
the `platform-pg-backup-s3` credential lane — the same handoff
`make postgres-backup-deploy` waits on. Check whether that is now live before
assuming this is blocked.

## Status 2026-08-18 — repository-complete, live-blocked

Everything this repo can do without touching the cluster is done and
committed. What remains on T01, T02 and T04 is a single operator window
against a live shared rail, in this order:

1. `make apps-pg-deploy` — Cluster reconcile: role connection limits,
   Burstable requests/limits, explicit aggregate parameters.
2. Apply `helm/apps-pg-consumer-controls.sql` — the two 15s role timeouts.
   Idempotent; CNPG 1.28 has no managed-role settings field, so this is
   operator SQL by necessity, not by preference.
3. `make apps-pg-backup-deploy` — the ScheduledBackup, once the governed
   Secret is confirmed live.
4. Capture `LastBackupSucceeded=True` and a scratch restore. **Until both
   exist, `apps-pg` R stays 0** — declared configuration is not a §13
   artifact, and this workplan exists because that distinction was missed once
   already.
5. T04's probes, in an announced window, after 1–3 have settled.

**Blocker resolved 2026-08-18, and it was ours.** The earlier note recorded
this as `kubectl` returning `Unauthorized` — an access problem outside the
repo. That was wrong. railiance01 is reachable and healthy
(`~/.kube/config-railiance01`, k3s v1.35.1, `apps-pg` 9d, both consumers
present). What failed was our own wiring: `KUBECONFIG` is an environment
variable, so the Makefile's `?=` default never applied, and
`RAILIANCE01_KUBECONFIG` pointed at `config-hosteurope` — a different cluster
that happens to be unauthorized from here.

**The near-miss is the finding.** Two reachable clusters each carry a CNPG
`Cluster` named `apps-pg` in a namespace named `databases`. The other one
(k3s v1.30.3) holds `gitea-db` and only one apps-pg consumer. Had `KUBECONFIG`
pointed there instead of at an unauthorized file, `make apps-pg-deploy` would
have applied this workplan's connection limits, role timeouts and backup
configuration **to the wrong cluster, and reported success.** The
`Unauthorized` error was the only thing that prevented it.

Fixed by pinning cluster *identity* rather than kubeconfig *filename*:
`railiance01-guard` compares the live `kube-system` namespace UID against
`RAILIANCE01_CLUSTER_UID` and fails closed on mismatch or unreachability. It
gates `apps-pg-deploy`, `apps-pg-backup-deploy`, `apps-pg-overflow-dry-run`,
`apps-pg-status` and `apps-pg-shell`. Filename selection could not have
protected against this: both kubeconfigs resolve to a `127.0.0.1` tunnel port,
and the environment overrides the default either way. `make cluster-id` prints
what is currently selected.

The guard is deliberately **not** global. `db-status` legitimately targets the
other cluster for `gitea-db`, so a blanket guard would break a working target
and teach people to bypass it.


**Do not treat the rollout as evidence.** T04's P1 claim and the R-axis both
need artifacts produced *after* application, and `docs/placement-policy.md`
§3.1 and `tenancy.yaml` should be updated only then.

## Tasks

```task
id: RPF-WP-0019-T01
status: progress
priority: high
state_hub_task_id: "0f5175c7-bde3-46a0-87a8-b50b5eeac32c"
```
**Establish a backup target for `apps-pg`.** Confirm the state of the
`resource-control` bucket and the `platform-pg-backup-s3` OpenBao Secret; if
live, configure `barmanObjectStore` and a `retentionPolicy` on the cluster. If
not live, record the dependency and say so — do not leave the absence
undocumented a second time.

2026-08-18 repository readiness: the governed Secret exists live and reviewed
desired state now carries a distinct `apps-pg/` prefix, 30-day retention,
continuous WAL and a daily 02:15 backup. NetworkPolicy admits the DNS/HTTPS
path archival needs. The ScheduledBackup is not applied and no successful
backup/restore evidence exists, so T01 remains progress.

```task
id: RPF-WP-0019-T02
status: progress
priority: high
state_hub_task_id: "c5b0b2ac-b9f5-42fb-8e75-2fac2ab1e852"
```
**Declare and enforce per-consumer controls.** Per-consumer connection
allowance, `statement_timeout`, `idle_in_transaction_session_timeout`, and
pod resource requests/limits to lift `apps-pg` off BestEffort QoS. Publish
every value in `docs/s3-consumer-interfaces.md` before it takes effect —
§10.2 is a disclosure rule, and applying a timeout consumers learn about by
hitting it would breach the rule while implementing it.

2026-08-18 repository readiness: both roles declare a 20-connection limit,
the pod has Burstable requests/limits, aggregate/logging parameters are
explicit, and controlled operator SQL sets both 15s role timeouts. Every value
was published in `docs/s3-consumer-interfaces.md` before application. Live SQL
and Cluster reconciliation remain an operator window, so T02 remains progress.

```task
id: RPF-WP-0019-T03
status: done
priority: medium
state_hub_task_id: "736cbc11-1992-4f1b-9ff0-cb4622ff39a5"
```
**Declare the ceiling and overflow target.** Owed under this repo's own
Rule P-4.1 before `apps-pg`'s third consumer; it is at two. Name the binding
resource per Rule P-4.2 — memory or connections — and a named overflow
substrate per P-4.3.

Completed 2026-08-18. The declared ceiling is three, memory is the binding
constraint, and `apps-pg-2` is a named, source-provisionable overflow cell with
a distinct credential and backup prefix. `make apps-pg-verify-capacity`
rejects a fourth consumer per cell and unbounded/duplicate roles. The cell
intentionally remains absent until a fourth consumer is approved.

```task
id: RPF-WP-0019-T04
status: wait
priority: medium
state_hub_task_id: "2fccdd26-c9c6-43cb-b069-27ed5668afac"
```
**Isolation probes, after T02.** Consumer-boundary probes on the
`rapp-postgres` model, then the §13 noisy-neighbour artifact: per-consumer
baseline, saturation run, evidence the controls bind, degradation measured and
judged against each consumer's declared service class. Update
`docs/placement-policy.md` §3.1 and `docs/tenancy-posture.md` when the P1
claims become evidenced.

Waiting on T02 live application and an announced probe window. No saturation
or destructive recovery experiment is run against the shared production rail
as part of repository preparation.

## Boundaries

- `apps-pg` only. `platform-pg`'s equivalents are `rapp-postgres`'s.
- No consumer is migrated. This changes the cluster, not who is on it.
- Values are published before they are enforced, never after.

## Risks

**Applying limits to a live cluster breaks a consumer that was relying on
their absence.** Most likely with `statement_timeout`. Mitigation is the
disclosure-first ordering in T02, which gives consumers a window to object.

**T01 stays blocked on a handoff outside this repo and the cluster keeps no
backup meanwhile.** Mitigation is that T01 requires the dependency be recorded
explicitly rather than left as a silent absence — which is exactly how D1
survived this long.