Finish RPF-WP-0019 apps-pg controls and recovery
This commit is contained in:
codex
parent
272c1691c3
commit
cb321deeab
16 changed files with 517 additions and 68 deletions
|
|
@ -123,8 +123,9 @@ applied; coulomb-social connectivity smoke from labeled consumer ns OK.
|
|||
|
||||
`apps-pg` remains a conservative single-instance, 10Gi cluster. The desired
|
||||
state now includes continuous WAL archival, a daily 02:15 UTC base backup and
|
||||
30-day retention under the distinct `apps-pg/` object-store prefix. The
|
||||
credential lane is shared with `platform-pg`; the backup data path is not.
|
||||
30-day retention under the distinct `platform-pg/apps-pg/` object-store
|
||||
prefix. The credential lane and its bucket-policy top-level prefix are shared
|
||||
with `platform-pg`; the backup data path beneath that prefix is not.
|
||||
|
||||
The declared ceiling is three consumers. Each gets at most 20 connections;
|
||||
40 of the explicit 100-connection aggregate remains for CNPG and operator
|
||||
|
|
@ -140,11 +141,14 @@ resource envelopes, or a reused backup path.
|
|||
`statement_timeout` and `idle_in_transaction_session_timeout` are 15 seconds
|
||||
per consumer role. CNPG 1.28 cannot express those role settings, so
|
||||
`helm/apps-pg-consumer-controls.sql` is the idempotent controlled-operator
|
||||
step; `connectionLimit` remains declaratively reconciled by CNPG.
|
||||
step; `connectionLimit` remains declaratively reconciled by CNPG. The same SQL
|
||||
removes PostgreSQL's default `PUBLIC` `CONNECT`/`TEMPORARY` grants and grants
|
||||
each consumer role access only to its own database.
|
||||
|
||||
Resource evidence for `resource:railiance:apps-pg` (capacity, recovery,
|
||||
labor, allocation drivers) is published under
|
||||
`docs/evidence/RAILIANCE-WP-0016-apps-pg-resource-evidence.md`. The
|
||||
2026-08-14 observation remains historically correct. The 2026-08-18 desired
|
||||
state is not called verified recovery until the first backup succeeds and a
|
||||
scratch restore artifact is recorded.
|
||||
2026-08-14 observation remains historically correct. Live backup, restore and
|
||||
isolation evidence captured on 2026-08-20 is in
|
||||
`docs/evidence/RPF-WP-0019-backup-restore-2026-08-20.md` and
|
||||
`docs/evidence/RPF-WP-0019-isolation-2026-08-20.md`.
|
||||
|
|
|
|||
88
docs/evidence/RPF-WP-0019-backup-restore-2026-08-20.md
Normal file
88
docs/evidence/RPF-WP-0019-backup-restore-2026-08-20.md
Normal file
|
|
@ -0,0 +1,88 @@
|
|||
# RPF-WP-0019 — apps-pg backup and restore evidence
|
||||
|
||||
Date: 2026-08-20
|
||||
|
||||
Live target: railiance01 (`kube-system` UID `a553c742-0115-43d4-99a4-a5ca56fe0786`)
|
||||
|
||||
PostgreSQL: 16, CloudNativePG 1.25.1
|
||||
|
||||
No credential value was read or recorded while producing this evidence.
|
||||
|
||||
## Controls rollout
|
||||
|
||||
The guarded reconcile changed `apps-pg-1` from BestEffort to Burstable QoS:
|
||||
|
||||
| Resource | Request | Limit |
|
||||
| --- | ---: | ---: |
|
||||
| CPU | 100m | 1 |
|
||||
| Memory | 256Mi | 1Gi |
|
||||
|
||||
PostgreSQL reported both `vergabe` and `coulomb_social` with connection limit
|
||||
20, `statement_timeout=15s`, and
|
||||
`idle_in_transaction_session_timeout=15s`. `max_connections=100` and the
|
||||
declared connection, lock-wait, slow-statement, and I/O timing logging settings
|
||||
were also live. The cluster returned to `Ready=True` after its single-instance
|
||||
restart.
|
||||
|
||||
## Archive-path correction
|
||||
|
||||
The first WAL attempt against the reviewed sibling prefix `apps-pg/` failed
|
||||
with Scaleway `AccessDenied` on `PutObject`. This was expected behavior from
|
||||
the already-evidenced bucket policy, which permits this identity only beneath
|
||||
`platform-pg/*`; the manifest and policy had drifted.
|
||||
|
||||
The live and source destination was corrected to the distinct governed
|
||||
sub-prefix:
|
||||
|
||||
`s3://railiance-platform-pg-backup/platform-pg/apps-pg/`
|
||||
|
||||
After reconcile, the Cluster condition changed to
|
||||
`ContinuousArchiving=True` / `ContinuousArchivingSuccess` at
|
||||
2026-08-20T20:41:40Z.
|
||||
|
||||
## Base backup
|
||||
|
||||
| Field | Value |
|
||||
| --- | --- |
|
||||
| ScheduledBackup | `apps-pg-daily` (02:15 UTC, immediate first run) |
|
||||
| Backup | `apps-pg-daily-20260820204148` |
|
||||
| Backup ID | `20260820T204148` |
|
||||
| Started | 2026-08-20T20:41:48Z |
|
||||
| Completed | 2026-08-20T20:41:56Z (8 seconds) |
|
||||
| WAL | `000000010000000000000032` through the same segment |
|
||||
| Result | `phase=completed`; Cluster `LastBackupSucceeded=True` |
|
||||
|
||||
## Scratch restore
|
||||
|
||||
A separate Cluster named `apps-pg-restore-20260820` was created in namespace
|
||||
`apps-pg-restore-20260820`; production was never recovered in place. The
|
||||
governed Secret was copied namespace-to-namespace without printing its values.
|
||||
|
||||
| Event | UTC |
|
||||
| --- | --- |
|
||||
| Restore applied | 2026-08-20T20:43:45Z |
|
||||
| Scratch `Ready=True` | 2026-08-20T20:44:41Z |
|
||||
| Measured restore time | 56 seconds |
|
||||
|
||||
Production and scratch matched exactly for the four production databases and
|
||||
their sizes: `apps_meta` 7,830,551 bytes, `coulomb_social_db` 8,985,623 bytes,
|
||||
`postgres` 7,830,551 bytes, and `vergabe_db` 7,830,551 bytes. `vergabe_db` was
|
||||
empty on both sides. All 13 user tables in `coulomb_social_db` matched by name
|
||||
and exact row count, including 36 permissions, 20 migrations, 23 sessions, one
|
||||
member, one user, and one space.
|
||||
|
||||
Both restored consumer roles also retained the 20-connection and two
|
||||
15-second settings. The scratch catalog additionally contained an empty
|
||||
7,912,471-byte `app` database that was not present in the live catalog at
|
||||
verification time. It is recorded as recovered historical state, not silently
|
||||
described as a current production database.
|
||||
|
||||
Production remained `Ready=True`. The scratch namespace was deleted after
|
||||
these comparisons were captured.
|
||||
|
||||
## Claim
|
||||
|
||||
This proves a completed off-cluster base backup and a full instance restore of
|
||||
both `apps-pg` consumer databases. It does not prove point-in-time recovery,
|
||||
single-consumer physical restore, HA, node-loss recovery, or isolation; those
|
||||
claims need their own artifacts.
|
||||
65
docs/evidence/RPF-WP-0019-isolation-2026-08-20.md
Normal file
65
docs/evidence/RPF-WP-0019-isolation-2026-08-20.md
Normal file
|
|
@ -0,0 +1,65 @@
|
|||
# RPF-WP-0019 — apps-pg isolation evidence
|
||||
|
||||
Date: 2026-08-20
|
||||
|
||||
Live target: railiance01 / `databases/apps-pg`
|
||||
|
||||
Harness: `make apps-pg-isolation-probe`
|
||||
|
||||
The repeatable harness creates a labeled temporary namespace, projects the two
|
||||
existing consumer Secrets into isolated probe pods without printing their
|
||||
values, exercises the real consumer credentials over the service endpoint,
|
||||
and deletes the namespace on exit.
|
||||
|
||||
## Finding before the passing run
|
||||
|
||||
The first privilege preflight found that `apps_meta`, `vergabe_db`, and
|
||||
`coulomb_social_db` retained PostgreSQL's default `PUBLIC` `CONNECT` and
|
||||
`TEMPORARY` privileges. A database-per-consumer declaration was therefore not
|
||||
yet a database connection boundary. The controlled SQL now revokes both
|
||||
privileges from `PUBLIC` on all three databases and grants them only to each
|
||||
database's owner. The passing run occurred after that correction; the failed
|
||||
precondition is not counted as a pass.
|
||||
|
||||
## Consumer boundary
|
||||
|
||||
All eleven assertions passed:
|
||||
|
||||
- each role connected to its own database through the live RW Service;
|
||||
- each role was denied connection to the peer database;
|
||||
- `coulomb_social` was denied connection to `apps_meta`;
|
||||
- a consumer could not alter the peer role;
|
||||
- an own-database table remained readable;
|
||||
- the real credential observed both 15-second timeouts and connection limit
|
||||
20; and
|
||||
- a 20-second statement was terminated by `statement_timeout`.
|
||||
|
||||
## Noisy-neighbour run
|
||||
|
||||
The harness measured five sequential `vergabe` queries, filled every available
|
||||
`coulomb_social` role slot with client-side-idle sessions, verified 20 live
|
||||
connections, and attempted one more connection.
|
||||
|
||||
| Observation | Result |
|
||||
| --- | --- |
|
||||
| `vergabe` baseline, five queries | 5,285 ms |
|
||||
| `coulomb_social` at ceiling | 20 connections |
|
||||
| excess `coulomb_social` connection | denied |
|
||||
| `vergabe` during saturation, five queries | 8,146 ms |
|
||||
| measured wall-time increase | 54.1% |
|
||||
| victim availability | 5/5 queries succeeded |
|
||||
|
||||
The timings include five separate API/exec/process startups, so they are an
|
||||
operator-path comparison rather than an application latency benchmark. Both
|
||||
co-residents declare `interactive`, but neither declares a numeric database
|
||||
latency objective. The result therefore supports the P1 consumer-boundary and
|
||||
availability claim: one consumer exhausted its own allowance, the control
|
||||
bound, and the peer retained service. It does **not** establish a latency SLO,
|
||||
CPU fairness, memory fairness, tenant isolation within either consumer, HA, or
|
||||
P2 placement.
|
||||
|
||||
## Result
|
||||
|
||||
`14 passed, 0 failed`. Production `apps-pg` remained Ready with continuous
|
||||
archiving and the last backup healthy; `deployment/coulomb-social` remained
|
||||
1/1 Ready. The temporary probe namespace was deleted.
|
||||
|
|
@ -79,43 +79,40 @@ triggers. It is not the same role as the operator of the cluster.
|
|||
| `user-engine` | `user-engine-pg` | P2 | `net-kingdom` | this repo | not assessed here |
|
||||
| `target-revenue` | `target-revenue-pg` | P2 | `target-revenue` | this repo | not assessed here |
|
||||
| `forgejo` | `forgejo-db` | P2 | this repo | — | single consumer |
|
||||
| `vergabe` | `apps-pg` | P1 | `vergabe-teilnahme` | this repo | **absent** |
|
||||
| `coulomb_social` | `apps-pg` | P1 | `coulomb-social` | this repo | **absent** |
|
||||
| `vergabe` | `apps-pg` | P1 | `vergabe-teilnahme` | this repo | 14 live boundary/control probes |
|
||||
| `coulomb_social` | `apps-pg` | P1 | `coulomb-social` | this repo | 14 live boundary/control probes |
|
||||
|
||||
### 3.1 Two of these P1 levels are asserted without evidence
|
||||
### 3.1 The two apps-pg P1 levels are now evidenced
|
||||
|
||||
§13.1 claims a level only with its artifact present. The P1–P4 artifact is
|
||||
*"provisioning declaration plus the platform's isolation probes"*.
|
||||
|
||||
`platform-pg` has them — `rapp-postgres` runs 19 adversarial probes against the
|
||||
consumer boundary. **`apps-pg` has none.** `scripts/capture-apps-pg-evidence.py`
|
||||
captures *capacity* evidence for `resource-control`; it makes no isolation
|
||||
assertion. So the P1 levels recorded above for `vergabe` and `coulomb_social`
|
||||
are, as of 2026-08-17, **provisioning declarations without probes** — stated
|
||||
here rather than quietly carried, because a table of levels that mixes
|
||||
evidenced and unevidenced entries without saying which is which is the failure
|
||||
§13.1 exists to prevent.
|
||||
consumer boundary. Since 2026-08-20, `apps-pg` has a repeatable 14-assertion
|
||||
live harness too. It uses both real consumer credentials, denies cross-database
|
||||
and bootstrap connections, verifies the timeouts and role allowance, fills
|
||||
one consumer's allowance, proves the next connection is rejected, and keeps
|
||||
the peer available. Evidence:
|
||||
`docs/evidence/RPF-WP-0019-isolation-2026-08-20.md`.
|
||||
|
||||
Recorded as owed by this repo. Until the probes exist, those two rows should be
|
||||
read as *intended* placement, not *verified* placement.
|
||||
|
||||
### 3.2 `apps-pg` cannot produce the noisy-neighbour artifact at all
|
||||
### 3.2 `apps-pg` noisy-neighbour artifact and its limit
|
||||
|
||||
Worth separating from the gap above, because it is not a matter of writing a
|
||||
missing test. §13's P1–P2 noisy-neighbour artifact requires a per-consumer
|
||||
usage baseline, a saturation run, and **evidence that the governance controls
|
||||
bind — that the greedy consumer is held at its limits.**
|
||||
|
||||
`apps-pg` has no per-consumer governance controls. Its connection pool is not
|
||||
partitioned, there is no per-consumer connection allowance, no statement
|
||||
timeout and no resource limit on the pod. There is nothing that could be shown
|
||||
to bind, so a run against it could only demonstrate that a saturating consumer
|
||||
degrades its co-residents — which §13.4 explicitly rejects as an artifact that
|
||||
"can only fail, or that passes by being run gently enough".
|
||||
RPF-WP-0019 supplied the missing 20-connection role allowances, 15-second
|
||||
timeouts, database ACL boundary, and Burstable pod envelope before running the
|
||||
artifact. Saturating `coulomb_social` bound at 20 and rejected an excess
|
||||
connection while five `vergabe` queries remained available. Five-query
|
||||
operator-path wall time rose from 5,285ms to 8,146ms (54.1%).
|
||||
|
||||
The honest sequence is therefore: **controls first, artifact second.** Writing
|
||||
the probe before the limits exist would produce an overclaim wearing the
|
||||
costume of evidence. Tracked as `RPF-WP-0019`.
|
||||
Both consumers are `interactive`, but neither publishes a numeric database
|
||||
latency objective, and the measurement includes five separate `kubectl exec`
|
||||
and `psql` startups. The artifact therefore evidences P1 boundary and continued
|
||||
availability, not a latency SLO or CPU/memory fairness. That limit is part of
|
||||
the artifact rather than silently promoted into a stronger claim.
|
||||
|
||||
**The consuming repo is the placement owner; this repo co-signs.** The
|
||||
consumer knows its latency requirement, its erasure horizon and its compliance
|
||||
|
|
|
|||
|
|
@ -65,18 +65,18 @@ before the corresponding live rollout:
|
|||
| Per-consumer connections | 20 | CNPG `managed.roles[].connectionLimit`, for both current roles. |
|
||||
| `statement_timeout` | 15s | Per role; idempotent operator SQL because CNPG 1.28 has no managed-role settings field. |
|
||||
| `idle_in_transaction_session_timeout` | 15s | Per role, same enforcement path. |
|
||||
| Database boundary | own database only | `CONNECT` and `TEMPORARY` are revoked from `PUBLIC`; each consumer role receives them only on its own database. |
|
||||
| CPU / memory | 100m / 256Mi requests; 1 CPU / 1Gi limits | Burstable QoS; memory is expected to bind before connections if workload size grows. |
|
||||
| Instances | 1 | No HA. Restart recovery only, per the single-node rail. |
|
||||
| Storage | 10Gi | |
|
||||
| Backup | 30d; daily 02:15 UTC + continuous WAL | S3 prefix `s3://railiance-platform-pg-backup/apps-pg/`; rollout is not evidence of recoverability until first backup and restore proof exist. |
|
||||
| Backup | 30d; daily 02:15 UTC + continuous WAL | S3 prefix `s3://railiance-platform-pg-backup/platform-pg/apps-pg/`; backup and restore proof captured 2026-08-20. |
|
||||
| Effective erasure horizon | 30d, or the max across co-residents | Shared-cluster rule from placement policy §7. |
|
||||
| Declared ceiling / overflow | 3 consumers / `apps-pg-2` | Memory is the binding safety constraint; provision the named overflow substrate before admitting a fourth. |
|
||||
|
||||
The manifests close the configuration gap, not the evidence gap. Until the
|
||||
cluster change is applied and `LastBackupSucceeded=True` plus a scratch restore
|
||||
are captured, recovery remains operationally unproven. Likewise the P1 noisy-
|
||||
neighbour claim waits for the controlled probe window; source configuration is
|
||||
not substituted for either artifact.
|
||||
The configuration and evidence gaps closed on 2026-08-20:
|
||||
`LastBackupSucceeded=True`, a 56-second scratch restore, and a 14/14 live
|
||||
boundary/noisy-neighbour run. Source configuration is not substituted for
|
||||
those artifacts; see `docs/evidence/RPF-WP-0019-*.md`.
|
||||
|
||||
## `rapp-postgres.v1`
|
||||
|
||||
|
|
|
|||
|
|
@ -180,6 +180,16 @@ Declaring `P: 1` for `apps-pg` would read as though the cluster had a
|
|||
placement, and a later reader would use it to argue about isolation. `n/a`
|
||||
plus the provider statement is the accurate canonical form.
|
||||
|
||||
For `apps-pg`, the provider statement is now backed by live evidence dated
|
||||
2026-08-20. A separately restored cluster recovered both consumer databases in
|
||||
56 seconds from the governed 30-day backup path, establishing current R2/V1
|
||||
for the provider. The 14-assertion isolation harness then proved own-database
|
||||
access, cross-database denial, both role timeouts, the 20-connection boundary,
|
||||
and peer availability while one role exhausted its allowance. The measured
|
||||
wall-time increase is recorded but is not a latency SLO. Evidence:
|
||||
`docs/evidence/RPF-WP-0019-backup-restore-2026-08-20.md` and
|
||||
`docs/evidence/RPF-WP-0019-isolation-2026-08-20.md`.
|
||||
|
||||
## Finding adopted by `net-kingdom`
|
||||
|
||||
**The six ladders describe a consumer of storage. They do not describe a
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue