Finish RPF-WP-0019 apps-pg controls and recovery

workplans/RPF-WP-0019-apps-pg-recoverability-and-controls.md

@@ -4,11 +4,11 @@ type: workplan
 title: "apps-pg: backup, per-consumer controls, and the isolation probes they make possible"
 domain: financials
 repo: railiance-platform
-status: active
+status: finished
 owner: codex
 topic_slug: railiance
 created: "2026-08-17"
-updated: "2026-08-18"
+updated: "2026-08-20"
 related:
   - RPF-WP-0018
 origin: residual
@@ -120,11 +120,29 @@ and teach people to bypass it.
 need artifacts produced *after* application, and `docs/placement-policy.md`
 §3.1 and `tenancy.yaml` should be updated only then.
 
+## Status 2026-08-20 — recoverability and controls live
+
+The guarded live rollout completed against railiance01. The pod reconciled
+from BestEffort to Burstable QoS, both consumer roles now have a 20-connection
+limit, the explicit aggregate/logging parameters bind, and controlled SQL
+applied and verified both 15-second role timeouts. T02 is complete.
+
+The first WAL archive attempt exposed a repository defect before any recovery
+claim was made: the runtime identity is intentionally restricted by bucket
+policy to `platform-pg/*`, while the reviewed manifest used sibling prefix
+`apps-pg/`. Scaleway correctly denied `PutObject`. Desired state now keeps the
+cell distinct beneath the governed prefix at `platform-pg/apps-pg/` (and the
+unprovisioned overflow at `platform-pg/apps-pg-2/`). Continuous archiving then
+became healthy, the immediate base backup completed in eight seconds, and a
+separate scratch Cluster restored both consumer databases in 56 seconds.
+Evidence is `docs/evidence/RPF-WP-0019-backup-restore-2026-08-20.md`. T01 and
+T02 are complete. T04 subsequently passed 14/14 and the workplan is finished.
+
 ## Tasks
 
 ```task
 id: RPF-WP-0019-T01
-status: progress
+status: done
 priority: high
 state_hub_task_id: "0f5175c7-bde3-46a0-87a8-b50b5eeac32c"
 ```
@@ -134,15 +152,22 @@ live, configure `barmanObjectStore` and a `retentionPolicy` on the cluster. If
 not live, record the dependency and say so — do not leave the absence
 undocumented a second time.
 
-2026-08-18 repository readiness: the governed Secret exists live and reviewed
-desired state now carries a distinct `apps-pg/` prefix, 30-day retention,
-continuous WAL and a daily 02:15 backup. NetworkPolicy admits the DNS/HTTPS
-path archival needs. The ScheduledBackup is not applied and no successful
-backup/restore evidence exists, so T01 remains progress.
+2026-08-18 repository readiness: the governed Secret existed live and source
+carried 30-day retention, continuous WAL and a daily 02:15 backup. At that
+point it incorrectly used sibling prefix `apps-pg/`; the 2026-08-20 live run
+proved the bucket policy denied it and corrected the path beneath
+`platform-pg/`. The ScheduledBackup had not yet been applied and no successful
+backup/restore evidence existed, so T01 correctly remained progress then.
+
+Completed live 2026-08-20. The governed path is
+`platform-pg/apps-pg/`, continuous archiving is healthy, Backup
+`apps-pg-daily-20260820204148` completed in eight seconds, and a separately
+named scratch Cluster restored and matched both consumer databases in 56
+seconds. See `docs/evidence/RPF-WP-0019-backup-restore-2026-08-20.md`.
 
 ```task
 id: RPF-WP-0019-T02
-status: progress
+status: done
 priority: high
 state_hub_task_id: "c5b0b2ac-b9f5-42fb-8e75-2fac2ab1e852"
 ```
@@ -156,8 +181,14 @@ hitting it would breach the rule while implementing it.
 2026-08-18 repository readiness: both roles declare a 20-connection limit,
 the pod has Burstable requests/limits, aggregate/logging parameters are
 explicit, and controlled operator SQL sets both 15s role timeouts. Every value
-was published in `docs/s3-consumer-interfaces.md` before application. Live SQL
-and Cluster reconciliation remain an operator window, so T02 remains progress.
+was published in `docs/s3-consumer-interfaces.md` before application. At that
+point, live SQL and Cluster reconciliation still required an operator window.
+
+Completed live 2026-08-20. The railiance01 identity guard passed, the single
+instance reconciled healthy with Burstable QoS, and PostgreSQL reported both
+roles at connection limit 20 with `statement_timeout=15s` and
+`idle_in_transaction_session_timeout=15s`. The declared aggregate and logging
+parameters were also verified live.
 
 ```task
 id: RPF-WP-0019-T03
@@ -178,7 +209,7 @@ intentionally remains absent until a fourth consumer is approved.
 
 ```task
 id: RPF-WP-0019-T04
-status: wait
+status: done
 priority: medium
 state_hub_task_id: "2fccdd26-c9c6-43cb-b069-27ed5668afac"
 ```
@@ -193,6 +224,22 @@ Waiting on T02 live application and an announced probe window. No saturation
 or destructive recovery experiment is run against the shared production rail
 as part of repository preparation.
 
+Started 2026-08-20 after T02 completed. The first privilege preflight found
+that all three databases still inherited PostgreSQL's default `PUBLIC`
+`CONNECT` and `TEMPORARY` grants, so either consumer could connect to the
+other's database even though relation grants remained separate. The probe did
+not launder that into a P1 pass. The controlled SQL and published interface now
+revoke the defaults and grant each role access only to its own database; live
+boundary and noisy-neighbour evidence follows that enforcement.
+
+Completed 2026-08-20. `make apps-pg-isolation-probe` passed 14/14 using the
+real consumer credentials. The greedy role bound at 20 connections and the
+next connection was denied; five peer queries stayed available, with measured
+five-query wall time increasing from 5,285ms to 8,146ms. Both consumers are
+interactive but declare no numeric database latency objective, so this proves
+P1 boundary and continued service, not a latency SLO or resource fairness.
+Evidence: `docs/evidence/RPF-WP-0019-isolation-2026-08-20.md`.
+
 ## Boundaries
 
 - `apps-pg` only. `platform-pg`'s equivalents are `rapp-postgres`'s.