Finish RPF-WP-0019 apps-pg controls and recovery

docs/evidence/RPF-WP-0019-backup-restore-2026-08-20.md

@@ -0,0 +1,88 @@
+# RPF-WP-0019 — apps-pg backup and restore evidence
+
+Date: 2026-08-20
+
+Live target: railiance01 (`kube-system` UID `a553c742-0115-43d4-99a4-a5ca56fe0786`)
+
+PostgreSQL: 16, CloudNativePG 1.25.1
+
+No credential value was read or recorded while producing this evidence.
+
+## Controls rollout
+
+The guarded reconcile changed `apps-pg-1` from BestEffort to Burstable QoS:
+
+| Resource | Request | Limit |
+| --- | ---: | ---: |
+| CPU | 100m | 1 |
+| Memory | 256Mi | 1Gi |
+
+PostgreSQL reported both `vergabe` and `coulomb_social` with connection limit
+20, `statement_timeout=15s`, and
+`idle_in_transaction_session_timeout=15s`. `max_connections=100` and the
+declared connection, lock-wait, slow-statement, and I/O timing logging settings
+were also live. The cluster returned to `Ready=True` after its single-instance
+restart.
+
+## Archive-path correction
+
+The first WAL attempt against the reviewed sibling prefix `apps-pg/` failed
+with Scaleway `AccessDenied` on `PutObject`. This was expected behavior from
+the already-evidenced bucket policy, which permits this identity only beneath
+`platform-pg/*`; the manifest and policy had drifted.
+
+The live and source destination was corrected to the distinct governed
+sub-prefix:
+
+`s3://railiance-platform-pg-backup/platform-pg/apps-pg/`
+
+After reconcile, the Cluster condition changed to
+`ContinuousArchiving=True` / `ContinuousArchivingSuccess` at
+2026-08-20T20:41:40Z.
+
+## Base backup
+
+| Field | Value |
+| --- | --- |
+| ScheduledBackup | `apps-pg-daily` (02:15 UTC, immediate first run) |
+| Backup | `apps-pg-daily-20260820204148` |
+| Backup ID | `20260820T204148` |
+| Started | 2026-08-20T20:41:48Z |
+| Completed | 2026-08-20T20:41:56Z (8 seconds) |
+| WAL | `000000010000000000000032` through the same segment |
+| Result | `phase=completed`; Cluster `LastBackupSucceeded=True` |
+
+## Scratch restore
+
+A separate Cluster named `apps-pg-restore-20260820` was created in namespace
+`apps-pg-restore-20260820`; production was never recovered in place. The
+governed Secret was copied namespace-to-namespace without printing its values.
+
+| Event | UTC |
+| --- | --- |
+| Restore applied | 2026-08-20T20:43:45Z |
+| Scratch `Ready=True` | 2026-08-20T20:44:41Z |
+| Measured restore time | 56 seconds |
+
+Production and scratch matched exactly for the four production databases and
+their sizes: `apps_meta` 7,830,551 bytes, `coulomb_social_db` 8,985,623 bytes,
+`postgres` 7,830,551 bytes, and `vergabe_db` 7,830,551 bytes. `vergabe_db` was
+empty on both sides. All 13 user tables in `coulomb_social_db` matched by name
+and exact row count, including 36 permissions, 20 migrations, 23 sessions, one
+member, one user, and one space.
+
+Both restored consumer roles also retained the 20-connection and two
+15-second settings. The scratch catalog additionally contained an empty
+7,912,471-byte `app` database that was not present in the live catalog at
+verification time. It is recorded as recovered historical state, not silently
+described as a current production database.
+
+Production remained `Ready=True`. The scratch namespace was deleted after
+these comparisons were captured.
+
+## Claim
+
+This proves a completed off-cluster base backup and a full instance restore of
+both `apps-pg` consumer databases. It does not prove point-in-time recovery,
+single-consumer physical restore, HA, node-loss recovery, or isolation; those
+claims need their own artifacts.