Finish RPF-WP-0019 apps-pg controls and recovery

2026-08-20 22:58:45 +02:00 · 2026-08-20 22:58:45 +02:00 · cb321deeab
commit cb321deeab
parent 272c1691c3
16 changed files with 517 additions and 68 deletions
--- a/.gitignore
+++ b/.gitignore
@ -10,6 +10,7 @@ helm/*.yaml
 !helm/*-networkpolicies.yaml
 !helm/*-databases.yaml
 !helm/*-backup.yaml
+!helm/*-probe.yaml

 # ArgoCD repository credentials — encrypt locally, never commit
 argocd/repositories/*.repository.sops.yaml
--- a/4
+++ b/4
@ -134,6 +134,9 @@ apps-pg-backup-deploy: railiance01-guard ## Apply the daily backup after the gov
 	@$(KUBECTL) get secret platform-pg-backup-s3 -n databases >/dev/null || { echo 'missing governed platform-pg-backup-s3 secret' >&2; exit 2; }
 	$(KUBECTL) apply -f helm/apps-pg-backup.yaml

+apps-pg-isolation-probe: railiance01-guard ## Run bounded live consumer-boundary and connection-saturation probes
+	KUBECONFIG='$(KUBECONFIG)' KUBECTL_BIN='$(KUBECTL_BIN)' tools/apps-pg-isolation-probe.sh
+
 apps-pg-overflow-dry-run: railiance01-guard apps-pg-verify-capacity ## Server-validate the unapplied overflow cell
 	@$(KUBECTL) get secret apps-pg-2-credentials -n databases >/dev/null || { echo 'missing distinct apps-pg-2-credentials secret' >&2; exit 2; }
 	$(KUBECTL) apply --dry-run=server -f helm/apps-pg-2-cluster.yaml
@ -460,4 +463,5 @@ help: ## Show this help
 	  /^[a-zA-Z_-]+:.*?##/ { printf "  \033[36m%-22s\033[0m %s\n", $$1, $$2 } \
 	  /^##@/ { printf "\n\033[1m%s\033[0m\n", substr($$0, 5) }' $(MAKEFILE_LIST)

+.PHONY: apps-pg-isolation-probe
 .PHONY: db-deploy db-status db-shell db-logs forgejo-db-deploy forgejo-db-status forgejo-db-shell apps-pg-deploy apps-pg-status apps-pg-shell apps-pg-logs net-kingdom-pg-inter-hub-networkpolicy-deploy pg-deploy pg-status pg-pgpool-check valkey-deploy valkey-status openbao-repo openbao-dry-run openbao-overlay-apply openbao-verify-login-overlay openbao-deploy openbao-status openbao-verify openbao-verify-post-unseal openbao-configure-initial openbao-configure-ssh openbao-verify-ssh openbao-verify-authenticated openbao-configure-external-secrets-issue-core openbao-configure-external-secrets-activity-core openbao-configure-external-secrets-forgejo openbao-validate-restore-evidence openbao-validate-emergency-evidence credential-grants-validate credential-change-applier-dry-run credential-change-applier-apply-plan credential-change-applier-apply credential-change-runbook credential-change-record-evidence credential-change-lifecycle-plan credential-change-lifecycle-event credential-change-import-inventory openbao-credential-change-appliers-dry-run openbao-configure-credential-change-appliers openbao-token-grants-dry-run openbao-configure-token-grants openbao-verify-token-grants-dry-run openbao-verify-token-grants openbao-verify-token-grants-smoke credential-helper-dry-run credential-tests credential-exec-ops-warden-smoke argocd-bootstrap-dry-run argocd-bootstrap-deploy argocd-repo-apply argocd-status backup forgejo-backup forgejo-backup-dry-run forgejo-backup-status forgejo-package-prune forgejo-package-prune-dry-run reef-deployables apps-pg-evidence consumption-preflight help
--- a/WORK-RECORDS.md
+++ b/WORK-RECORDS.md
@ -21,7 +21,7 @@
 | workplan | RAILIANCE-WP-0016 | finished | — | workplans/RAILIANCE-WP-0016-architecture-cleanup-backlog.md |
 | workplan | RAILIANCE-WP-0017 | finished | — | workplans/RAILIANCE-WP-0017-consumption-mode-enforcement.md |
 | workplan | RPF-WP-0018 | finished | — | workplans/RPF-WP-0018-policy-surface-alignment.md |
-| workplan | RPF-WP-0019 | active | — | workplans/RPF-WP-0019-apps-pg-recoverability-and-controls.md |
+| workplan | RPF-WP-0019 | finished | — | workplans/RPF-WP-0019-apps-pg-recoverability-and-controls.md |
 | workplan | RPF-WP-0020 | proposed | — | workplans/RPF-WP-0020-ccr-schema-drift.md |
 | workplan | RPF-WP-0021 | active | — | workplans/RPF-WP-0021-core-hub-platform-onboarding.md |
 | task | RAILIANCE-WP-0005-T01 | done | — | workplans/RAILIANCE-WP-0005-credential-request-and-lease-broker.md |
@ -92,10 +92,10 @@
 | task | RPF-WP-0018-T05 | done | — | workplans/RPF-WP-0018-policy-surface-alignment.md |
 | task | RPF-WP-0018-T06 | done | — | workplans/RPF-WP-0018-policy-surface-alignment.md |
 | task | RPF-WP-0018-T07 | done | — | workplans/RPF-WP-0018-policy-surface-alignment.md |
-| task | RPF-WP-0019-T01 | progress | — | workplans/RPF-WP-0019-apps-pg-recoverability-and-controls.md |
-| task | RPF-WP-0019-T02 | progress | — | workplans/RPF-WP-0019-apps-pg-recoverability-and-controls.md |
+| task | RPF-WP-0019-T01 | done | — | workplans/RPF-WP-0019-apps-pg-recoverability-and-controls.md |
+| task | RPF-WP-0019-T02 | done | — | workplans/RPF-WP-0019-apps-pg-recoverability-and-controls.md |
 | task | RPF-WP-0019-T03 | done | — | workplans/RPF-WP-0019-apps-pg-recoverability-and-controls.md |
-| task | RPF-WP-0019-T04 | wait | — | workplans/RPF-WP-0019-apps-pg-recoverability-and-controls.md |
+| task | RPF-WP-0019-T04 | done | — | workplans/RPF-WP-0019-apps-pg-recoverability-and-controls.md |
 | task | RPF-WP-0020-T01 | todo | — | workplans/RPF-WP-0020-ccr-schema-drift.md |
 | task | RPF-WP-0020-T02 | todo | — | workplans/RPF-WP-0020-ccr-schema-drift.md |
 | task | RPF-WP-0020-T03 | todo | — | workplans/RPF-WP-0020-ccr-schema-drift.md |
--- a/docs/apps-pg.md
+++ b/docs/apps-pg.md
@ -123,8 +123,9 @@ applied; coulomb-social connectivity smoke from labeled consumer ns OK.

 `apps-pg` remains a conservative single-instance, 10Gi cluster. The desired
 state now includes continuous WAL archival, a daily 02:15 UTC base backup and
-30-day retention under the distinct `apps-pg/` object-store prefix. The
-credential lane is shared with `platform-pg`; the backup data path is not.
+30-day retention under the distinct `platform-pg/apps-pg/` object-store
+prefix. The credential lane and its bucket-policy top-level prefix are shared
+with `platform-pg`; the backup data path beneath that prefix is not.

 The declared ceiling is three consumers. Each gets at most 20 connections;
 40 of the explicit 100-connection aggregate remains for CNPG and operator
@ -140,11 +141,14 @@ resource envelopes, or a reused backup path.
 `statement_timeout` and `idle_in_transaction_session_timeout` are 15 seconds
 per consumer role. CNPG 1.28 cannot express those role settings, so
 `helm/apps-pg-consumer-controls.sql` is the idempotent controlled-operator
-step; `connectionLimit` remains declaratively reconciled by CNPG.
+step; `connectionLimit` remains declaratively reconciled by CNPG. The same SQL
+removes PostgreSQL's default `PUBLIC` `CONNECT`/`TEMPORARY` grants and grants
+each consumer role access only to its own database.

 Resource evidence for `resource:railiance:apps-pg` (capacity, recovery,
 labor, allocation drivers) is published under
 `docs/evidence/RAILIANCE-WP-0016-apps-pg-resource-evidence.md`. The
-2026-08-14 observation remains historically correct. The 2026-08-18 desired
-state is not called verified recovery until the first backup succeeds and a
-scratch restore artifact is recorded.
+2026-08-14 observation remains historically correct. Live backup, restore and
+isolation evidence captured on 2026-08-20 is in
+`docs/evidence/RPF-WP-0019-backup-restore-2026-08-20.md` and
+`docs/evidence/RPF-WP-0019-isolation-2026-08-20.md`.
--- a/docs/evidence/RPF-WP-0019-backup-restore-2026-08-20.md
+++ b/docs/evidence/RPF-WP-0019-backup-restore-2026-08-20.md
@ -0,0 +1,88 @@
+# RPF-WP-0019 — apps-pg backup and restore evidence
+
+Date: 2026-08-20
+
+Live target: railiance01 (`kube-system` UID `a553c742-0115-43d4-99a4-a5ca56fe0786`)
+
+PostgreSQL: 16, CloudNativePG 1.25.1
+
+No credential value was read or recorded while producing this evidence.
+
+## Controls rollout
+
+The guarded reconcile changed `apps-pg-1` from BestEffort to Burstable QoS:
+
+| Resource | Request | Limit |
+| --- | ---: | ---: |
+| CPU | 100m | 1 |
+| Memory | 256Mi | 1Gi |
+
+PostgreSQL reported both `vergabe` and `coulomb_social` with connection limit
+20, `statement_timeout=15s`, and
+`idle_in_transaction_session_timeout=15s`. `max_connections=100` and the
+declared connection, lock-wait, slow-statement, and I/O timing logging settings
+were also live. The cluster returned to `Ready=True` after its single-instance
+restart.
+
+## Archive-path correction
+
+The first WAL attempt against the reviewed sibling prefix `apps-pg/` failed
+with Scaleway `AccessDenied` on `PutObject`. This was expected behavior from
+the already-evidenced bucket policy, which permits this identity only beneath
+`platform-pg/*`; the manifest and policy had drifted.
+
+The live and source destination was corrected to the distinct governed
+sub-prefix:
+
+`s3://railiance-platform-pg-backup/platform-pg/apps-pg/`
+
+After reconcile, the Cluster condition changed to
+`ContinuousArchiving=True` / `ContinuousArchivingSuccess` at
+2026-08-20T20:41:40Z.
+
+## Base backup
+
+| Field | Value |
+| --- | --- |
+| ScheduledBackup | `apps-pg-daily` (02:15 UTC, immediate first run) |
+| Backup | `apps-pg-daily-20260820204148` |
+| Backup ID | `20260820T204148` |
+| Started | 2026-08-20T20:41:48Z |
+| Completed | 2026-08-20T20:41:56Z (8 seconds) |
+| WAL | `000000010000000000000032` through the same segment |
+| Result | `phase=completed`; Cluster `LastBackupSucceeded=True` |
+
+## Scratch restore
+
+A separate Cluster named `apps-pg-restore-20260820` was created in namespace
+`apps-pg-restore-20260820`; production was never recovered in place. The
+governed Secret was copied namespace-to-namespace without printing its values.
+
+| Event | UTC |
+| --- | --- |
+| Restore applied | 2026-08-20T20:43:45Z |
+| Scratch `Ready=True` | 2026-08-20T20:44:41Z |
+| Measured restore time | 56 seconds |
+
+Production and scratch matched exactly for the four production databases and
+their sizes: `apps_meta` 7,830,551 bytes, `coulomb_social_db` 8,985,623 bytes,
+`postgres` 7,830,551 bytes, and `vergabe_db` 7,830,551 bytes. `vergabe_db` was
+empty on both sides. All 13 user tables in `coulomb_social_db` matched by name
+and exact row count, including 36 permissions, 20 migrations, 23 sessions, one
+member, one user, and one space.
+
+Both restored consumer roles also retained the 20-connection and two
+15-second settings. The scratch catalog additionally contained an empty
+7,912,471-byte `app` database that was not present in the live catalog at
+verification time. It is recorded as recovered historical state, not silently
+described as a current production database.
+
+Production remained `Ready=True`. The scratch namespace was deleted after
+these comparisons were captured.
+
+## Claim
+
+This proves a completed off-cluster base backup and a full instance restore of
+both `apps-pg` consumer databases. It does not prove point-in-time recovery,
+single-consumer physical restore, HA, node-loss recovery, or isolation; those
+claims need their own artifacts.
--- a/docs/evidence/RPF-WP-0019-isolation-2026-08-20.md
+++ b/docs/evidence/RPF-WP-0019-isolation-2026-08-20.md
@ -0,0 +1,65 @@
+# RPF-WP-0019 — apps-pg isolation evidence
+
+Date: 2026-08-20
+
+Live target: railiance01 / `databases/apps-pg`
+
+Harness: `make apps-pg-isolation-probe`
+
+The repeatable harness creates a labeled temporary namespace, projects the two
+existing consumer Secrets into isolated probe pods without printing their
+values, exercises the real consumer credentials over the service endpoint,
+and deletes the namespace on exit.
+
+## Finding before the passing run
+
+The first privilege preflight found that `apps_meta`, `vergabe_db`, and
+`coulomb_social_db` retained PostgreSQL's default `PUBLIC` `CONNECT` and
+`TEMPORARY` privileges. A database-per-consumer declaration was therefore not
+yet a database connection boundary. The controlled SQL now revokes both
+privileges from `PUBLIC` on all three databases and grants them only to each
+database's owner. The passing run occurred after that correction; the failed
+precondition is not counted as a pass.
+
+## Consumer boundary
+
+All eleven assertions passed:
+
+- each role connected to its own database through the live RW Service;
+- each role was denied connection to the peer database;
+- `coulomb_social` was denied connection to `apps_meta`;
+- a consumer could not alter the peer role;
+- an own-database table remained readable;
+- the real credential observed both 15-second timeouts and connection limit
+  20; and
+- a 20-second statement was terminated by `statement_timeout`.
+
+## Noisy-neighbour run
+
+The harness measured five sequential `vergabe` queries, filled every available
+`coulomb_social` role slot with client-side-idle sessions, verified 20 live
+connections, and attempted one more connection.
+
+| Observation | Result |
+| --- | --- |
+| `vergabe` baseline, five queries | 5,285 ms |
+| `coulomb_social` at ceiling | 20 connections |
+| excess `coulomb_social` connection | denied |
+| `vergabe` during saturation, five queries | 8,146 ms |
+| measured wall-time increase | 54.1% |
+| victim availability | 5/5 queries succeeded |
+
+The timings include five separate API/exec/process startups, so they are an
+operator-path comparison rather than an application latency benchmark. Both
+co-residents declare `interactive`, but neither declares a numeric database
+latency objective. The result therefore supports the P1 consumer-boundary and
+availability claim: one consumer exhausted its own allowance, the control
+bound, and the peer retained service. It does **not** establish a latency SLO,
+CPU fairness, memory fairness, tenant isolation within either consumer, HA, or
+P2 placement.
+
+## Result
+
+`14 passed, 0 failed`. Production `apps-pg` remained Ready with continuous
+archiving and the last backup healthy; `deployment/coulomb-social` remained
+1/1 Ready. The temporary probe namespace was deleted.
--- a/docs/placement-policy.md
+++ b/docs/placement-policy.md
@ -79,43 +79,40 @@ triggers. It is not the same role as the operator of the cluster.
 | `user-engine` | `user-engine-pg` | P2 | `net-kingdom` | this repo | not assessed here |
 | `target-revenue` | `target-revenue-pg` | P2 | `target-revenue` | this repo | not assessed here |
 | `forgejo` | `forgejo-db` | P2 | this repo | — | single consumer |
-| `vergabe` | `apps-pg` | P1 | `vergabe-teilnahme` | this repo | **absent** |
-| `coulomb_social` | `apps-pg` | P1 | `coulomb-social` | this repo | **absent** |
+| `vergabe` | `apps-pg` | P1 | `vergabe-teilnahme` | this repo | 14 live boundary/control probes |
+| `coulomb_social` | `apps-pg` | P1 | `coulomb-social` | this repo | 14 live boundary/control probes |

-### 3.1 Two of these P1 levels are asserted without evidence
+### 3.1 The two apps-pg P1 levels are now evidenced

 §13.1 claims a level only with its artifact present. The P1–P4 artifact is
 *"provisioning declaration plus the platform's isolation probes"*.

 `platform-pg` has them — `rapp-postgres` runs 19 adversarial probes against the
-consumer boundary. **`apps-pg` has none.** `scripts/capture-apps-pg-evidence.py`
-captures *capacity* evidence for `resource-control`; it makes no isolation
-assertion. So the P1 levels recorded above for `vergabe` and `coulomb_social`
-are, as of 2026-08-17, **provisioning declarations without probes** — stated
-here rather than quietly carried, because a table of levels that mixes
-evidenced and unevidenced entries without saying which is which is the failure
-§13.1 exists to prevent.
+consumer boundary. Since 2026-08-20, `apps-pg` has a repeatable 14-assertion
+live harness too. It uses both real consumer credentials, denies cross-database
+and bootstrap connections, verifies the timeouts and role allowance, fills
+one consumer's allowance, proves the next connection is rejected, and keeps
+the peer available. Evidence:
+`docs/evidence/RPF-WP-0019-isolation-2026-08-20.md`.

-Recorded as owed by this repo. Until the probes exist, those two rows should be
-read as *intended* placement, not *verified* placement.
-
-### 3.2 `apps-pg` cannot produce the noisy-neighbour artifact at all
+### 3.2 `apps-pg` noisy-neighbour artifact and its limit

 Worth separating from the gap above, because it is not a matter of writing a
 missing test. §13's P1–P2 noisy-neighbour artifact requires a per-consumer
 usage baseline, a saturation run, and **evidence that the governance controls
 bind — that the greedy consumer is held at its limits.**

-`apps-pg` has no per-consumer governance controls. Its connection pool is not
-partitioned, there is no per-consumer connection allowance, no statement
-timeout and no resource limit on the pod. There is nothing that could be shown
-to bind, so a run against it could only demonstrate that a saturating consumer
-degrades its co-residents — which §13.4 explicitly rejects as an artifact that
-"can only fail, or that passes by being run gently enough".
+RPF-WP-0019 supplied the missing 20-connection role allowances, 15-second
+timeouts, database ACL boundary, and Burstable pod envelope before running the
+artifact. Saturating `coulomb_social` bound at 20 and rejected an excess
+connection while five `vergabe` queries remained available. Five-query
+operator-path wall time rose from 5,285ms to 8,146ms (54.1%).

-The honest sequence is therefore: **controls first, artifact second.** Writing
-the probe before the limits exist would produce an overclaim wearing the
-costume of evidence. Tracked as `RPF-WP-0019`.
+Both consumers are `interactive`, but neither publishes a numeric database
+latency objective, and the measurement includes five separate `kubectl exec`
+and `psql` startups. The artifact therefore evidences P1 boundary and continued
+availability, not a latency SLO or CPU/memory fairness. That limit is part of
+the artifact rather than silently promoted into a stronger claim.

 **The consuming repo is the placement owner; this repo co-signs.** The
 consumer knows its latency requirement, its erasure horizon and its compliance
--- a/docs/s3-consumer-interfaces.md
+++ b/docs/s3-consumer-interfaces.md
@ -65,18 +65,18 @@ before the corresponding live rollout:
 | Per-consumer connections | 20 | CNPG `managed.roles[].connectionLimit`, for both current roles. |
 | `statement_timeout` | 15s | Per role; idempotent operator SQL because CNPG 1.28 has no managed-role settings field. |
 | `idle_in_transaction_session_timeout` | 15s | Per role, same enforcement path. |
+| Database boundary | own database only | `CONNECT` and `TEMPORARY` are revoked from `PUBLIC`; each consumer role receives them only on its own database. |
 | CPU / memory | 100m / 256Mi requests; 1 CPU / 1Gi limits | Burstable QoS; memory is expected to bind before connections if workload size grows. |
 | Instances | 1 | No HA. Restart recovery only, per the single-node rail. |
 | Storage | 10Gi | |
-| Backup | 30d; daily 02:15 UTC + continuous WAL | S3 prefix `s3://railiance-platform-pg-backup/apps-pg/`; rollout is not evidence of recoverability until first backup and restore proof exist. |
+| Backup | 30d; daily 02:15 UTC + continuous WAL | S3 prefix `s3://railiance-platform-pg-backup/platform-pg/apps-pg/`; backup and restore proof captured 2026-08-20. |
 | Effective erasure horizon | 30d, or the max across co-residents | Shared-cluster rule from placement policy §7. |
 | Declared ceiling / overflow | 3 consumers / `apps-pg-2` | Memory is the binding safety constraint; provision the named overflow substrate before admitting a fourth. |

-The manifests close the configuration gap, not the evidence gap. Until the
-cluster change is applied and `LastBackupSucceeded=True` plus a scratch restore
-are captured, recovery remains operationally unproven. Likewise the P1 noisy-
-neighbour claim waits for the controlled probe window; source configuration is
-not substituted for either artifact.
+The configuration and evidence gaps closed on 2026-08-20:
+`LastBackupSucceeded=True`, a 56-second scratch restore, and a 14/14 live
+boundary/noisy-neighbour run. Source configuration is not substituted for
+those artifacts; see `docs/evidence/RPF-WP-0019-*.md`.

 ## `rapp-postgres.v1`

--- a/docs/tenancy-posture.md
+++ b/docs/tenancy-posture.md
@ -180,6 +180,16 @@ Declaring `P: 1` for `apps-pg` would read as though the cluster had a
 placement, and a later reader would use it to argue about isolation. `n/a`
 plus the provider statement is the accurate canonical form.

+For `apps-pg`, the provider statement is now backed by live evidence dated
+2026-08-20. A separately restored cluster recovered both consumer databases in
+56 seconds from the governed 30-day backup path, establishing current R2/V1
+for the provider. The 14-assertion isolation harness then proved own-database
+access, cross-database denial, both role timeouts, the 20-connection boundary,
+and peer availability while one role exhausted its allowance. The measured
+wall-time increase is recorded but is not a latency SLO. Evidence:
+`docs/evidence/RPF-WP-0019-backup-restore-2026-08-20.md` and
+`docs/evidence/RPF-WP-0019-isolation-2026-08-20.md`.
+
 ## Finding adopted by `net-kingdom`

 **The six ladders describe a consumer of storage. They do not describe a
--- a/helm/apps-pg-2-cluster.yaml
+++ b/helm/apps-pg-2-cluster.yaml
@ -36,7 +36,7 @@ spec:
  backup:
    retentionPolicy: "30d"
    barmanObjectStore:
-      destinationPath: s3://railiance-platform-pg-backup/apps-pg-2/
+      destinationPath: s3://railiance-platform-pg-backup/platform-pg/apps-pg-2/
      endpointURL: https://s3.nl-ams.scw.cloud
      s3Credentials:
        accessKeyId: {name: platform-pg-backup-s3, key: ACCESS_KEY_ID}
--- a/helm/apps-pg-cluster.yaml
+++ b/helm/apps-pg-cluster.yaml
@ -70,7 +70,9 @@ spec:
  backup:
    retentionPolicy: "30d"
    barmanObjectStore:
-      destinationPath: s3://railiance-platform-pg-backup/apps-pg/
+      # The bucket policy grants this runtime identity only platform-pg/*.
+      # Keep apps-pg distinct beneath that governed top-level prefix.
+      destinationPath: s3://railiance-platform-pg-backup/platform-pg/apps-pg/
      endpointURL: https://s3.nl-ams.scw.cloud
      s3Credentials:
        accessKeyId:
--- a/helm/apps-pg-consumer-controls.sql
+++ b/helm/apps-pg-consumer-controls.sql
@ -5,3 +5,11 @@ ALTER ROLE vergabe SET statement_timeout = '15s';
 ALTER ROLE vergabe SET idle_in_transaction_session_timeout = '15s';
 ALTER ROLE coulomb_social SET statement_timeout = '15s';
 ALTER ROLE coulomb_social SET idle_in_transaction_session_timeout = '15s';
+
+-- PostgreSQL grants CONNECT and TEMPORARY to PUBLIC by default. A database per
+-- consumer is not an isolation boundary until those defaults are removed.
+REVOKE CONNECT, TEMPORARY ON DATABASE apps_meta FROM PUBLIC;
+REVOKE CONNECT, TEMPORARY ON DATABASE vergabe_db FROM PUBLIC;
+REVOKE CONNECT, TEMPORARY ON DATABASE coulomb_social_db FROM PUBLIC;
+GRANT CONNECT, TEMPORARY ON DATABASE vergabe_db TO vergabe;
+GRANT CONNECT, TEMPORARY ON DATABASE coulomb_social_db TO coulomb_social;
--- a/helm/apps-pg-isolation-probe.yaml
+++ b/helm/apps-pg-isolation-probe.yaml
@ -0,0 +1,67 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: apps-pg-isolation-probe
+  labels:
+    railiance.io/postgres-client: apps-pg
+    railiance.io/purpose: apps-pg-isolation-probe
+---
+apiVersion: v1
+kind: Pod
+metadata:
+  name: coulomb-probe
+  namespace: apps-pg-isolation-probe
+  labels:
+    app.kubernetes.io/name: apps-pg-isolation-probe
+spec:
+  restartPolicy: Never
+  automountServiceAccountToken: false
+  containers:
+    - name: psql
+      image: ghcr.io/cloudnative-pg/postgresql:16
+      command: [sh, -c, "sleep 1800"]
+      env:
+        - name: PGPASSWORD
+          valueFrom:
+            secretKeyRef:
+              name: coulomb-social-app-credentials
+              key: password
+      resources:
+        requests: {cpu: 10m, memory: 32Mi}
+        limits: {cpu: 100m, memory: 128Mi}
+      securityContext:
+        allowPrivilegeEscalation: false
+        capabilities: {drop: [ALL]}
+        runAsNonRoot: true
+        runAsUser: 26
+        seccompProfile: {type: RuntimeDefault}
+---
+apiVersion: v1
+kind: Pod
+metadata:
+  name: vergabe-probe
+  namespace: apps-pg-isolation-probe
+  labels:
+    app.kubernetes.io/name: apps-pg-isolation-probe
+spec:
+  restartPolicy: Never
+  automountServiceAccountToken: false
+  containers:
+    - name: psql
+      image: ghcr.io/cloudnative-pg/postgresql:16
+      command: [sh, -c, "sleep 1800"]
+      env:
+        - name: PGPASSWORD
+          valueFrom:
+            secretKeyRef:
+              name: vergabe-app-credentials
+              key: password
+      resources:
+        requests: {cpu: 10m, memory: 32Mi}
+        limits: {cpu: 100m, memory: 128Mi}
+      securityContext:
+        allowPrivilegeEscalation: false
+        capabilities: {drop: [ALL]}
+        runAsNonRoot: true
+        runAsUser: 26
+        seccompProfile: {type: RuntimeDefault}
--- a/tenancy.yaml
+++ b/tenancy.yaml
@ -6,9 +6,9 @@ services:
  - service: apps-pg
    role: relational-storage-provider
    tenancy:
-      current: { I: 0, A: 0, E: 0, P: "n/a", R: 0, V: 0 }
+      current: { I: 0, A: 0, E: 0, P: "n/a", R: 2, V: 1 }
      target:  { I: 0, A: 0, E: 0, P: "n/a", R: 2, V: 1 }
-      reviewed: "2026-08-17"
+      reviewed: "2026-08-20"
      review_due: "2027-02-17"
      service_class: interactive
      permanent: [I, A, E, P]
@ -17,14 +17,11 @@ services:
        A: "No tenant context is bound at the database provider."
        E: "Tenant enforcement belongs to each consuming application."
        P: "apps-pg provides P1; a substrate is not itself at a P level."
-        R: "A backup target and 30-day retention are declared in source and
-            not yet applied; no successful backup or restore exists. R stays 0
-            until evidenced, per §13.1 — declared configuration is not an
-            artifact."
-        V: "No recovery exercise or failover evidence exists."
-      gap:
-        R: "RPF-WP-0019 establishes a backup target and declared horizon."
-        V: "Exercise recovery after the backup target is established."
+        R: "Continuous WAL and a completed base backup use the governed
+            30-day object-store path; a scratch restore recovered both
+            consumer databases on 2026-08-20."
+        V: "A separate scratch Cluster restored Ready in 56 seconds; this is
+            single-node recovery evidence, not HA or failover."
    provider:
      capability: data.relational
      profile: s5-application-databases
@ -41,16 +38,20 @@ services:
          maximum: 2
          conditions:
            - "P2 requires a new dedicated cluster for the consumer."
+          evidence:
+            - "docs/evidence/RPF-WP-0019-isolation-2026-08-20.md"
        R:
-          available: 0
+          available: 2
          maximum: 2
-          conditions:
-            - "A governed backup target must exist before R2 is available."
+          evidence:
+            - "docs/evidence/RPF-WP-0019-backup-restore-2026-08-20.md"
        V:
-          available: 0
+          available: 1
          maximum: 1
          conditions:
-            - "Single-node recovery must be documented and exercised."
+            - "V1 is single-node recovery only; no HA or failover is claimed."
+          evidence:
+            - "docs/evidence/RPF-WP-0019-backup-restore-2026-08-20.md"

  - service: forgejo-db
    role: relational-storage-provider
--- a/tools/apps-pg-isolation-probe.sh
+++ b/tools/apps-pg-isolation-probe.sh
@ -0,0 +1,155 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)
+KUBECONFIG=${KUBECONFIG:-/home/worsch/.kube/config-railiance01}
+KUBECTL_BIN=${KUBECTL_BIN:-kubectl}
+EXPECTED_CLUSTER_UID=${RAILIANCE01_CLUSTER_UID:-a553c742-0115-43d4-99a4-a5ca56fe0786}
+NAMESPACE=apps-pg-isolation-probe
+DB_HOST=apps-pg-rw.databases.svc.cluster.local
+K=("$KUBECTL_BIN" --kubeconfig="$KUBECONFIG")
+pass=0
+fail=0
+namespace_created=0
+
+cleanup() {
+  if [[ "$namespace_created" == 1 && "${KEEP:-0}" != 1 ]]; then
+    "${K[@]}" delete namespace "$NAMESPACE" --wait=false >/dev/null 2>&1 || true
+  fi
+}
+trap cleanup EXIT
+
+record() {
+  local outcome=$1 name=$2 detail=${3:-}
+  if [[ "$outcome" == pass ]]; then
+    pass=$((pass + 1))
+    printf 'PASS|%s%s\n' "$name" "${detail:+|$detail}"
+  else
+    fail=$((fail + 1))
+    printf 'FAIL|%s%s\n' "$name" "${detail:+|$detail}"
+  fi
+}
+
+psql_as() {
+  local pod=$1 role=$2 database=$3 sql=$4
+  "${K[@]}" exec -n "$NAMESPACE" "$pod" -- \
+    psql -X -v ON_ERROR_STOP=1 -At -h "$DB_HOST" -U "$role" -d "$database" -c "$sql"
+}
+
+psql_super() {
+  local sql=$1
+  "${K[@]}" exec -n databases apps-pg-1 -c postgres -- \
+    psql -X -v ON_ERROR_STOP=1 -U postgres -d postgres -At -c "$sql"
+}
+
+expect_value() {
+  local name=$1 pod=$2 role=$3 database=$4 sql=$5 expected=$6 got
+  if ! got=$(psql_as "$pod" "$role" "$database" "$sql" 2>&1); then
+    record fail "$name" "query failed"
+  elif [[ "$got" == "$expected" ]]; then
+    record pass "$name" "$got"
+  else
+    record fail "$name" "expected=$expected got=$got"
+  fi
+}
+
+expect_denied() {
+  local name=$1 pod=$2 role=$3 database=$4 sql=$5
+  if psql_as "$pod" "$role" "$database" "$sql" >/dev/null 2>&1; then
+    record fail "$name" "attempt succeeded"
+  else
+    record pass "$name" "denied"
+  fi
+}
+
+copy_secret() {
+  local name=$1
+  "${K[@]}" get secret "$name" -n databases -o json \
+    | jq --arg ns "$NAMESPACE" \
+      'del(.metadata.annotations, .metadata.creationTimestamp, .metadata.resourceVersion,
+           .metadata.uid, .metadata.ownerReferences, .metadata.managedFields) |
+       .metadata.namespace=$ns' \
+    | "${K[@]}" apply -f - >/dev/null
+}
+
+measure_vergabe_ms() {
+  local start end
+  start=$(date +%s%3N)
+  for _ in 1 2 3 4 5; do
+    psql_as vergabe-probe vergabe vergabe_db 'SELECT 1' >/dev/null
+  done
+  end=$(date +%s%3N)
+  echo $((end - start))
+}
+
+uid=$("${K[@]}" get namespace kube-system -o jsonpath='{.metadata.uid}')
+if [[ "$uid" != "$EXPECTED_CLUSTER_UID" ]]; then
+  echo "REFUSING: expected railiance01 UID $EXPECTED_CLUSTER_UID, got $uid" >&2
+  exit 2
+fi
+if "${K[@]}" get namespace "$NAMESPACE" >/dev/null 2>&1; then
+  echo "REFUSING: probe namespace already exists: $NAMESPACE" >&2
+  exit 2
+fi
+
+"${K[@]}" apply -f "$ROOT/helm/apps-pg-isolation-probe.yaml" >/dev/null
+namespace_created=1
+copy_secret coulomb-social-app-credentials
+copy_secret vergabe-app-credentials
+"${K[@]}" wait --for=condition=Ready pod/coulomb-probe pod/vergabe-probe \
+  -n "$NAMESPACE" --timeout=60s >/dev/null
+
+echo 'SECTION|consumer-boundary'
+expect_value 'coulomb own database connects' coulomb-probe coulomb_social coulomb_social_db 'SELECT 1' 1
+expect_value 'vergabe own database connects' vergabe-probe vergabe vergabe_db 'SELECT 1' 1
+expect_denied 'coulomb cannot connect to vergabe database' coulomb-probe coulomb_social vergabe_db 'SELECT 1'
+expect_denied 'vergabe cannot connect to coulomb database' vergabe-probe vergabe coulomb_social_db 'SELECT 1'
+expect_denied 'coulomb cannot connect to platform bootstrap database' coulomb-probe coulomb_social apps_meta 'SELECT 1'
+expect_denied 'consumer cannot alter peer role' coulomb-probe coulomb_social coulomb_social_db 'ALTER ROLE vergabe CONNECTION LIMIT 999'
+expect_value 'coulomb sees its own table' coulomb-probe coulomb_social coulomb_social_db 'SELECT count(*) FROM members_user' 1
+expect_value 'coulomb statement timeout binds' coulomb-probe coulomb_social coulomb_social_db 'SHOW statement_timeout' 15s
+expect_value 'coulomb idle transaction timeout binds' coulomb-probe coulomb_social coulomb_social_db 'SHOW idle_in_transaction_session_timeout' 15s
+expect_value 'coulomb connection limit is 20' coulomb-probe coulomb_social coulomb_social_db "SELECT rolconnlimit FROM pg_roles WHERE rolname=current_user" 20
+expect_denied 'overlong statement is terminated' coulomb-probe coulomb_social coulomb_social_db 'SELECT pg_sleep(20)'
+
+echo 'SECTION|noisy-neighbour-connection-saturation'
+baseline_ms=$(measure_vergabe_ms)
+existing=$(psql_super "SELECT count(*) FROM pg_stat_activity WHERE usename='coulomb_social'")
+hold=$((20 - existing))
+if (( hold < 1 )); then
+  record fail 'saturation setup has free role slots' "existing=$existing"
+else
+  "${K[@]}" exec -n "$NAMESPACE" coulomb-probe -- sh -c '
+    count=$1
+    i=1
+    while [ "$i" -le "$count" ]; do
+      psql -X -v ON_ERROR_STOP=1 -At -h "$2" -U coulomb_social -d coulomb_social_db -c "\! sleep 25" >/dev/null 2>&1 &
+      i=$((i + 1))
+    done
+    wait
+  ' sh "$hold" "$DB_HOST" &
+  saturator_pid=$!
+
+  saturated=0
+  for _ in $(seq 1 50); do
+    current=$(psql_super "SELECT count(*) FROM pg_stat_activity WHERE usename='coulomb_social'")
+    if [[ "$current" == 20 ]]; then saturated=1; break; fi
+    sleep 0.2
+  done
+  if [[ "$saturated" == 1 ]]; then
+    record pass 'coulomb reaches its declared connection ceiling' 'connections=20'
+  else
+    record fail 'coulomb reaches its declared connection ceiling' "connections=$current"
+  fi
+  expect_denied 'coulomb excess connection is rejected' coulomb-probe coulomb_social coulomb_social_db 'SELECT 1'
+  if during_ms=$(measure_vergabe_ms); then
+    record pass 'vergabe remains available during peer saturation' "baseline_5q_ms=$baseline_ms saturated_5q_ms=$during_ms"
+  else
+    during_ms=unavailable
+    record fail 'vergabe remains available during peer saturation'
+  fi
+  wait "$saturator_pid" || true
+fi
+
+echo "SUMMARY|passed=$pass|failed=$fail|baseline_5q_ms=$baseline_ms|saturated_5q_ms=${during_ms:-not-run}"
+(( fail == 0 ))
--- a/workplans/RPF-WP-0019-apps-pg-recoverability-and-controls.md
+++ b/workplans/RPF-WP-0019-apps-pg-recoverability-and-controls.md
@ -4,11 +4,11 @@ type: workplan
 title: "apps-pg: backup, per-consumer controls, and the isolation probes they make possible"
 domain: financials
 repo: railiance-platform
-status: active
+status: finished
 owner: codex
 topic_slug: railiance
 created: "2026-08-17"
-updated: "2026-08-18"
+updated: "2026-08-20"
 related:
  - RPF-WP-0018
 origin: residual
@ -120,11 +120,29 @@ and teach people to bypass it.
 need artifacts produced *after* application, and `docs/placement-policy.md`
 §3.1 and `tenancy.yaml` should be updated only then.

+## Status 2026-08-20 — recoverability and controls live
+
+The guarded live rollout completed against railiance01. The pod reconciled
+from BestEffort to Burstable QoS, both consumer roles now have a 20-connection
+limit, the explicit aggregate/logging parameters bind, and controlled SQL
+applied and verified both 15-second role timeouts. T02 is complete.
+
+The first WAL archive attempt exposed a repository defect before any recovery
+claim was made: the runtime identity is intentionally restricted by bucket
+policy to `platform-pg/*`, while the reviewed manifest used sibling prefix
+`apps-pg/`. Scaleway correctly denied `PutObject`. Desired state now keeps the
+cell distinct beneath the governed prefix at `platform-pg/apps-pg/` (and the
+unprovisioned overflow at `platform-pg/apps-pg-2/`). Continuous archiving then
+became healthy, the immediate base backup completed in eight seconds, and a
+separate scratch Cluster restored both consumer databases in 56 seconds.
+Evidence is `docs/evidence/RPF-WP-0019-backup-restore-2026-08-20.md`. T01 and
+T02 are complete. T04 subsequently passed 14/14 and the workplan is finished.
+
 ## Tasks

 ```task
 id: RPF-WP-0019-T01
-status: progress
+status: done
 priority: high
 state_hub_task_id: "0f5175c7-bde3-46a0-87a8-b50b5eeac32c"
 ```
@ -134,15 +152,22 @@ live, configure `barmanObjectStore` and a `retentionPolicy` on the cluster. If
 not live, record the dependency and say so — do not leave the absence
 undocumented a second time.

-2026-08-18 repository readiness: the governed Secret exists live and reviewed
-desired state now carries a distinct `apps-pg/` prefix, 30-day retention,
-continuous WAL and a daily 02:15 backup. NetworkPolicy admits the DNS/HTTPS
-path archival needs. The ScheduledBackup is not applied and no successful
-backup/restore evidence exists, so T01 remains progress.
+2026-08-18 repository readiness: the governed Secret existed live and source
+carried 30-day retention, continuous WAL and a daily 02:15 backup. At that
+point it incorrectly used sibling prefix `apps-pg/`; the 2026-08-20 live run
+proved the bucket policy denied it and corrected the path beneath
+`platform-pg/`. The ScheduledBackup had not yet been applied and no successful
+backup/restore evidence existed, so T01 correctly remained progress then.
+
+Completed live 2026-08-20. The governed path is
+`platform-pg/apps-pg/`, continuous archiving is healthy, Backup
+`apps-pg-daily-20260820204148` completed in eight seconds, and a separately
+named scratch Cluster restored and matched both consumer databases in 56
+seconds. See `docs/evidence/RPF-WP-0019-backup-restore-2026-08-20.md`.

 ```task
 id: RPF-WP-0019-T02
-status: progress
+status: done
 priority: high
 state_hub_task_id: "c5b0b2ac-b9f5-42fb-8e75-2fac2ab1e852"
 ```
@ -156,8 +181,14 @@ hitting it would breach the rule while implementing it.
 2026-08-18 repository readiness: both roles declare a 20-connection limit,
 the pod has Burstable requests/limits, aggregate/logging parameters are
 explicit, and controlled operator SQL sets both 15s role timeouts. Every value
-was published in `docs/s3-consumer-interfaces.md` before application. Live SQL
-and Cluster reconciliation remain an operator window, so T02 remains progress.
+was published in `docs/s3-consumer-interfaces.md` before application. At that
+point, live SQL and Cluster reconciliation still required an operator window.
+
+Completed live 2026-08-20. The railiance01 identity guard passed, the single
+instance reconciled healthy with Burstable QoS, and PostgreSQL reported both
+roles at connection limit 20 with `statement_timeout=15s` and
+`idle_in_transaction_session_timeout=15s`. The declared aggregate and logging
+parameters were also verified live.

 ```task
 id: RPF-WP-0019-T03
@ -178,7 +209,7 @@ intentionally remains absent until a fourth consumer is approved.

 ```task
 id: RPF-WP-0019-T04
-status: wait
+status: done
 priority: medium
 state_hub_task_id: "2fccdd26-c9c6-43cb-b069-27ed5668afac"
 ```
@ -193,6 +224,22 @@ Waiting on T02 live application and an announced probe window. No saturation
 or destructive recovery experiment is run against the shared production rail
 as part of repository preparation.

+Started 2026-08-20 after T02 completed. The first privilege preflight found
+that all three databases still inherited PostgreSQL's default `PUBLIC`
+`CONNECT` and `TEMPORARY` grants, so either consumer could connect to the
+other's database even though relation grants remained separate. The probe did
+not launder that into a P1 pass. The controlled SQL and published interface now
+revoke the defaults and grant each role access only to its own database; live
+boundary and noisy-neighbour evidence follows that enforcement.
+
+Completed 2026-08-20. `make apps-pg-isolation-probe` passed 14/14 using the
+real consumer credentials. The greedy role bound at 20 connections and the
+next connection was denied; five peer queries stayed available, with measured
+five-query wall time increasing from 5,285ms to 8,146ms. Both consumers are
+interactive but declare no numeric database latency objective, so this proves
+P1 boundary and continued service, not a latency SLO or resource fairness.
+Evidence: `docs/evidence/RPF-WP-0019-isolation-2026-08-20.md`.
+
 ## Boundaries

 - `apps-pg` only. `platform-pg`'s equivalents are `rapp-postgres`'s.