Finish RPF-WP-0019 apps-pg controls and recovery

docs/evidence/RPF-WP-0019-backup-restore-2026-08-20.md

@@ -0,0 +1,88 @@
+# RPF-WP-0019 — apps-pg backup and restore evidence
+
+Date: 2026-08-20
+
+Live target: railiance01 (`kube-system` UID `a553c742-0115-43d4-99a4-a5ca56fe0786`)
+
+PostgreSQL: 16, CloudNativePG 1.25.1
+
+No credential value was read or recorded while producing this evidence.
+
+## Controls rollout
+
+The guarded reconcile changed `apps-pg-1` from BestEffort to Burstable QoS:
+
+| Resource | Request | Limit |
+| --- | ---: | ---: |
+| CPU | 100m | 1 |
+| Memory | 256Mi | 1Gi |
+
+PostgreSQL reported both `vergabe` and `coulomb_social` with connection limit
+20, `statement_timeout=15s`, and
+`idle_in_transaction_session_timeout=15s`. `max_connections=100` and the
+declared connection, lock-wait, slow-statement, and I/O timing logging settings
+were also live. The cluster returned to `Ready=True` after its single-instance
+restart.
+
+## Archive-path correction
+
+The first WAL attempt against the reviewed sibling prefix `apps-pg/` failed
+with Scaleway `AccessDenied` on `PutObject`. This was expected behavior from
+the already-evidenced bucket policy, which permits this identity only beneath
+`platform-pg/*`; the manifest and policy had drifted.
+
+The live and source destination was corrected to the distinct governed
+sub-prefix:
+
+`s3://railiance-platform-pg-backup/platform-pg/apps-pg/`
+
+After reconcile, the Cluster condition changed to
+`ContinuousArchiving=True` / `ContinuousArchivingSuccess` at
+2026-08-20T20:41:40Z.
+
+## Base backup
+
+| Field | Value |
+| --- | --- |
+| ScheduledBackup | `apps-pg-daily` (02:15 UTC, immediate first run) |
+| Backup | `apps-pg-daily-20260820204148` |
+| Backup ID | `20260820T204148` |
+| Started | 2026-08-20T20:41:48Z |
+| Completed | 2026-08-20T20:41:56Z (8 seconds) |
+| WAL | `000000010000000000000032` through the same segment |
+| Result | `phase=completed`; Cluster `LastBackupSucceeded=True` |
+
+## Scratch restore
+
+A separate Cluster named `apps-pg-restore-20260820` was created in namespace
+`apps-pg-restore-20260820`; production was never recovered in place. The
+governed Secret was copied namespace-to-namespace without printing its values.
+
+| Event | UTC |
+| --- | --- |
+| Restore applied | 2026-08-20T20:43:45Z |
+| Scratch `Ready=True` | 2026-08-20T20:44:41Z |
+| Measured restore time | 56 seconds |
+
+Production and scratch matched exactly for the four production databases and
+their sizes: `apps_meta` 7,830,551 bytes, `coulomb_social_db` 8,985,623 bytes,
+`postgres` 7,830,551 bytes, and `vergabe_db` 7,830,551 bytes. `vergabe_db` was
+empty on both sides. All 13 user tables in `coulomb_social_db` matched by name
+and exact row count, including 36 permissions, 20 migrations, 23 sessions, one
+member, one user, and one space.
+
+Both restored consumer roles also retained the 20-connection and two
+15-second settings. The scratch catalog additionally contained an empty
+7,912,471-byte `app` database that was not present in the live catalog at
+verification time. It is recorded as recovered historical state, not silently
+described as a current production database.
+
+Production remained `Ready=True`. The scratch namespace was deleted after
+these comparisons were captured.
+
+## Claim
+
+This proves a completed off-cluster base backup and a full instance restore of
+both `apps-pg` consumer databases. It does not prove point-in-time recovery,
+single-consumer physical restore, HA, node-loss recovery, or isolation; those
+claims need their own artifacts.

docs/evidence/RPF-WP-0019-isolation-2026-08-20.md

@@ -0,0 +1,65 @@
+# RPF-WP-0019 — apps-pg isolation evidence
+
+Date: 2026-08-20
+
+Live target: railiance01 / `databases/apps-pg`
+
+Harness: `make apps-pg-isolation-probe`
+
+The repeatable harness creates a labeled temporary namespace, projects the two
+existing consumer Secrets into isolated probe pods without printing their
+values, exercises the real consumer credentials over the service endpoint,
+and deletes the namespace on exit.
+
+## Finding before the passing run
+
+The first privilege preflight found that `apps_meta`, `vergabe_db`, and
+`coulomb_social_db` retained PostgreSQL's default `PUBLIC` `CONNECT` and
+`TEMPORARY` privileges. A database-per-consumer declaration was therefore not
+yet a database connection boundary. The controlled SQL now revokes both
+privileges from `PUBLIC` on all three databases and grants them only to each
+database's owner. The passing run occurred after that correction; the failed
+precondition is not counted as a pass.
+
+## Consumer boundary
+
+All eleven assertions passed:
+
+- each role connected to its own database through the live RW Service;
+- each role was denied connection to the peer database;
+- `coulomb_social` was denied connection to `apps_meta`;
+- a consumer could not alter the peer role;
+- an own-database table remained readable;
+- the real credential observed both 15-second timeouts and connection limit
+  20; and
+- a 20-second statement was terminated by `statement_timeout`.
+
+## Noisy-neighbour run
+
+The harness measured five sequential `vergabe` queries, filled every available
+`coulomb_social` role slot with client-side-idle sessions, verified 20 live
+connections, and attempted one more connection.
+
+| Observation | Result |
+| --- | --- |
+| `vergabe` baseline, five queries | 5,285 ms |
+| `coulomb_social` at ceiling | 20 connections |
+| excess `coulomb_social` connection | denied |
+| `vergabe` during saturation, five queries | 8,146 ms |
+| measured wall-time increase | 54.1% |
+| victim availability | 5/5 queries succeeded |
+
+The timings include five separate API/exec/process startups, so they are an
+operator-path comparison rather than an application latency benchmark. Both
+co-residents declare `interactive`, but neither declares a numeric database
+latency objective. The result therefore supports the P1 consumer-boundary and
+availability claim: one consumer exhausted its own allowance, the control
+bound, and the peer retained service. It does **not** establish a latency SLO,
+CPU fairness, memory fairness, tenant isolation within either consumer, HA, or
+P2 placement.
+
+## Result
+
+`14 passed, 0 failed`. Production `apps-pg` remained Ready with continuous
+archiving and the last backup healthy; `deployment/coulomb-social` remained
+1/1 Ready. The temporary probe namespace was deleted.