resource-control/workplans/RESOURCE-WP-0002-procure-postgres-backup-storage.md

---
id: RESOURCE-WP-0002
type: workplan
title: "Procure and operationalize PostgreSQL backup object storage"
domain: financials
repo: resource-control
status: finished
owner: codex
topic_slug: railiance
created: "2026-08-10"
updated: "2026-08-15"
state_hub_workstream_id: "921496a3-280b-4dc8-a3c0-b4ec314142f5"
---

# RESOURCE-WP-0002 — procure PostgreSQL backup storage

## Goal

Provide an off-host object-storage resource for `rapp-postgres` continuous WAL
archiving and physical base backups, and prove that it supports both full
recovery and point-in-time recovery (PITR). Record the resource, its real cost,
utilization, failure domain, contract, and exit path so fin-hub can incorporate
it into burn rate and resource-control can continuously test whether another
provider or procurement model is more attractive.

The candidate provider baseline is:

- **A — Host Europe**
- **B — Scaleway**
- **C — Hetzner**

This workplan chooses on evidence. It does not presume that keeping compute and
backup at one provider is cheaper or safer.

Inventory output follows `docs/operational-reference-convention.md`
(`RESOURCE-WP-0006`). Operating attributes are `reef:storage/…` references.
Credentials are `secret:railiance-platform/backup`. Do not write a Scaleway
endpoint or key into `data/resources/`. `rapp-postgres` consumes the
reviewed destination; it does not procure it.

## Context and current evidence

`rapp-postgres` has deployed `platform-pg` on reef-railiance at Host Europe.
Its workplan requires continuous WAL archiving, scheduled physical backups, a
30-day recovery window, and timed full/PITR restore drills.

Host Europe's **Backup Storage** is reachable by FTP, SCP, SFTP, and rsync. It
is suitable for an encrypted secondary logical-backup copy but is not directly
usable by CloudNativePG's Barman object-store integration. Host Europe also
publishes an older Cloud Storage specification describing an S3-compatible
service; current orderability, endpoint, features, support, and price must be
confirmed rather than inferred from that document.

Scaleway offers S3-compatible Object Storage in European regions, including a
Multi-AZ tier with usage-based storage pricing. Hetzner offers S3-compatible
Object Storage in Nuremberg, Falkenstein, and Helsinki with a monthly base
price including 1 TB storage and egress. Exact effective prices are recorded at
decision time, not copied permanently from changing marketing pages.

Primary references:

- Host Europe Backup Storage:
  `https://www.hosteurope.de/faq/server/virtual-server/backup-storage`
- Host Europe published Cloud Storage specification:
  `https://www.hosteurope.de/download/PVL/Preis_und_Leistungsverzeichnis_CloudStorage_1-0-1.pdf`
- Scaleway storage pricing: `https://www.scaleway.com/en/pricing/storage/`
- Hetzner Object Storage:
  `https://www.hetzner.com/de/storage/object-storage/`
- CNPG 1.25 object-store requirements:
  `https://cloudnative-pg.io/docs/1.25/appendixes/object_stores/`

## Boundaries

This workplan owns requirements, comparison, procurement evidence, resource
registration, cost/utilization monitoring, and proof that the purchased
resource works.

The purchased object is a five-facet resource (`RESOURCE-WP-0006`):
description, decision, `reef:` attribute refs, `secret:` handles, and
potential vs actual consumers. Non-secret operating attributes (endpoint,
bucket, region) will be owned by `reef-storage` (`RMASTER-WP-0022`), not
inlined here and not placed on `reef-railiance`. See
`docs/operational-reference-convention.md`.

- Human financial authority approves purchase and accepts contractual terms.
- `railiance-platform` owns provider credential custody and the approved
  Kubernetes Secret/credential-broker delivery.
- `rapp-postgres` owns CNPG manifests, schedules, retention configuration, and
  restore procedure.
- `fin-hub` owns budget, burn-rate, runway, and viability signals.

No access key, secret, payment instrument, or credential-bearing endpoint may
be committed to this repository or sent through State Hub.

## Acceptance requirements

The selected resource must provide:

- S3-compatible HTTPS endpoint accepted by the live CNPG/Barman version
- AWS Signature V4 and scoped access keys
- private buckets, multipart upload, list/get/put/delete operations
- versioning and lifecycle/retention controls, or a documented equivalent
- encryption in transit and at rest
- EU data location with provider and region recorded
- failure-domain analysis relative to reef-railiance
- at least 100 GiB initial usable capacity without a long commitment
- daily physical base backups and continuous WAL archive support
- target RPO no worse than five minutes and initially measured RTO
- export/delete path and credential revocation at exit
- monthly price and usage evidence attributable to `platform:audit-storage`

Object lock is desirable, but compatibility with CNPG retention deletion must
be tested before it becomes mandatory. Same-provider placement is allowed only
with an explicit correlated-failure decision and an independent second copy.

## T01 — Establish demand and cost model

```task
id: RESOURCE-WP-0002-T01
status: done
priority: high
state_hub_task_id: "f578a9ec-dbdb-4b26-93bc-e53f7bc87ad1"
```

Measure the current `platform-pg` database size, daily WAL generation, expected
growth, restore-test traffic, and retention requirement. Build low/base/high
12-month forecasts. Normalize each provider into monthly storage, ingress,
egress, operations, minimum charge, tax, support, commitment, and estimated
operator-labor cost.

Include the cost of one monthly restore drill and one complete provider exit.
Do not compare only advertised price per GB.

Done when the demand forecast and cost model can calculate effective monthly
cost for A, B, and C under the same scenarios.

Progress 2026-08-10: added a timestamped live database/WAL observation,
low/base/high 12-month inputs, and a tested fail-closed calculator in
`data/`, `tools/cost_model.py`, and
`docs/evidence/RESOURCE-WP-0002-demand-and-cost-model-2026-08-10.md`.
Scaleway and in-quota Hetzner costs calculate. Host Europe remains deliberately
`null` until a current account quote exists; Hetzner excess pricing is also
required before its high scenario and full exit can calculate.

Expanded 2026-08-10: added 1/2/3-node self-managed Garage estimates for Host
Europe and Hetzner plus managed AWS S3, Azure Blob, Google Cloud Storage, and
STACKIT Object Storage price points. Fixed-capacity VM options fail closed when
forecast demand exceeds usable replicated capacity. Currency conversion,
request charges, setup labor, recurring operations, and topology limitations
are explicit in
`docs/evidence/RESOURCE-WP-0002-expanded-storage-comparison-2026-08-10.md`.

Refined 2026-08-10: every quote now separates monthly infrastructure from
internal operations labor and internal setup from external services. Added a
normalized 320 GB running-cost view so bundled, fixed-capacity, and elastic
products compare on the same stored volume and request/restore pattern.

Done 2026-08-15: the account holder confirmed Host Europe offers no fully
managed S3 and no storage-provisioning API, so there is no managed product to
price. Host Europe's real form in this comparison is self-managed Garage on
their VMs, which was already priced from the public virtual-server catalogue:
EUR 335.77 per month for the 3-node option, roughly five times the chosen
Scaleway option, and it fails closed at month 12 when base demand (458 GB)
exceeds the 400 GB usable capacity. A, B, and C therefore calculate under the
same scenarios. Evidence:
`docs/evidence/RESOURCE-WP-0002-host-europe-resolution-2026-08-15.md`.

Control refinement 2026-08-10: added an immutable 12-month forecast record,
monthly actual-observation schema, and variance calculator covering stored
bytes, database size, WAL, requests, restore egress, invoice cost, and internal
labor. `docs/forecast-actual-control.md` defines error thresholds, evidence,
revision discipline, and the three-month recalibration loop.

## T02 — Complete provider due diligence

```task
id: RESOURCE-WP-0002-T02
status: done
priority: high
state_hub_task_id: "6dedb8db-c08f-4b58-90f2-32b1ea832d98"
```

For Host Europe, obtain current written confirmation of whether its
S3-compatible Cloud Storage can be ordered for this account, plus endpoint,
region, Signature V4, versioning, lifecycle, durability, availability, support,
current price, and cancellation terms. Treat SFTP Backup Storage as a separate
secondary-copy product.

For Scaleway and Hetzner, collect the same facts from current primary sources.
Record data location, durability/availability statements, minimum billing,
traffic rules, API limitations, credential granularity, versioning, lifecycle,
object-lock behavior, and provider status visibility.

Done when every acceptance requirement has evidence, `unknown`, or a blocking
answer for each provider—no blank cells and no marketing inference.

Done 2026-08-15: Host Europe's remaining cells are answered by a blocking
answer, which this task's acceptance criterion permits — the account holder
confirmed the managed S3 product is not offered, so the 2020 published
specification describes nothing currently orderable. `host-europe-cloud-storage`
moves to `not-offered`. Provenance is recorded as account-holder knowledge, not
vendor-written confirmation. No acceptance requirement is now blank for any
provider. Evidence:
`docs/evidence/RESOURCE-WP-0002-host-europe-resolution-2026-08-15.md`.

Progress 2026-08-10: completed the public-primary-source evidence matrix in
`docs/evidence/RESOURCE-WP-0002-provider-due-diligence-2026-08-10.md` and added
a provisional Scaleway inventory record. Host Europe current S3 orderability
and commercial terms require written account-support evidence. Scaleway is the
provisional primary because it documents Multi-AZ service, durability, S3
features, and managed at-rest encryption; selection still requires a live
Barman preflight, contract review, and human approval.

## T03 — Select and procure the primary resource

```task
id: RESOURCE-WP-0002-T03
status: done
priority: high
state_hub_task_id: "e4184350-dab2-4a0b-bee5-1a641e8a2df3"
```

Produce a decision record ranking the three providers on total cost,
compatibility, resilience, sovereignty, operational effort, and exit cost.
Recommend one primary and one independent secondary-copy path.

Human financial authority approves the purchase and contractual terms. Create
the account/project/bucket with public access disabled, the narrowest practical
service identity, versioning/lifecycle settings, cost alerts, and an owner.

Done when the decision is approved and the purchased resource has a non-secret
inventory record with provider resource ID, region, service class, contract,
renewal/cancellation dates, capacity model, owner, and cost-attribution key.

Progress 2026-08-14: decision record written —
`docs/evidence/RESOURCE-WP-0002-primary-selection-2026-08-14.md`.
Primary: Scaleway Multi-AZ `nl-ams`. Secondary copy: Host Europe Backup
Storage or Nextcloud (T06). **Purchase approved 2026-08-14.** After the
Object Storage policy attached, `scw object bucket list/get` succeeded.
Private bucket `railiance-platform-pg-backup` exists in `nl-ams`
(created 2026-08-14T16:21:56Z), versioning on, 30-day current and
noncurrent lifecycle applied, ACL owner-only. Inventory is
`data/resources/platform-audit-storage.json` (`status: ordered`,
`ordered_on: 2026-08-14`, `provider_resource_id` = bucket name).
Operating facts live on
`reef:storage/substrate/object-stores/platform-audit-storage.yaml`.
Founder set a Scaleway billing alert (email). Scaleway has no
euro hard-stop; that gap is accepted. Scoped backup key remains T04.

## T04 — Establish credential custody and hand off to rapp-postgres

```task
id: RESOURCE-WP-0002-T04
status: done
priority: high
state_hub_task_id: "a2dc370a-b5e7-44b1-b46a-f3b84815b14a"
```

Route the scoped **backup** object-store credential to
`railiance-platform` through the approved OpenBao/credential-broker
lane. Principal is IAM application `resource-control`; API key and
policy are `Scoped backup access`. Permit only the project-scoped
object-store operations backup consumers need. Keep credential values
out of Git, State Hub, shell history, logs, and chat.

Hand the non-secret endpoint, bucket/prefix, region, CA/TLS
requirements, and retention parameters to `rapp-postgres` as the first
consumer. Verify the empty-WAL-archive check before enabling continuous
archiving.

Done when credential delivery is renewable/revocable, a negative credential
cannot access the bucket, and `rapp-postgres` can perform the storage preflight
without disclosing a value.

Started 2026-08-14: bootstrap key cannot create IAM applications. Lane
retargeted the same day to a general backup identity — `CCR-2026-0012`
(proposed), mason plan `backup-object-storage` (draft), founder put
instructions, handle `secret:railiance-platform/backup`, first consumer
projection Secret `platform-pg-backup-s3`. Non-secret destination
handed at
`rapp-postgres/docs/handoff/RESOURCE-WP-0002-T04-barman-destination.md`.
2026-08-14: founder approved CCR-2026-0012 and put the backup key in
OpenBao (`ACCESS_KEY`/`SECRET_KEY` + org/project ids, version 1).
Positive S3 list/get and prefix put/get/delete succeeded. Negative:
bogus secret denied; IAM/billing/k8s list denied. Distinct from the
bootstrap key. Done 2026-08-14: OpenBao policy `workload-kv-read-backup-object-storage`
and AppRole `external-secrets-backup-object-storage` applied.
ClusterSecretStore `openbao-backup-object-storage` Valid; ExternalSecret
synced Secret `databases/platform-pg-backup-s3` (keys present, values
not read). Prefix `platform-pg/` is empty (0 objects). Cluster still
has no `barmanObjectStore`. WAL remains T05.

## T05 — Prove backup, full restore, and PITR

```task
id: RESOURCE-WP-0002-T05
status: done
priority: high
state_hub_task_id: "693953bf-f223-484b-b835-41369f5d2a02"
```

Enable continuous WAL archiving and daily base backups. Confirm archive health,
backup age, object growth, retention behavior, and alerts. Then:

1. Restore the latest physical backup to a scratch CNPG cluster and compare
   per-database counts/checksums.
2. Create a restore point, write a marker, record a target time, perform a
   destructive change, and restore to immediately before that change.
3. Record measured RPO, full-restore RTO, PITR RTO, bytes transferred, and
   provider charges.
4. Delete scratch resources only after evidence is durable.

Done when both restores are verified and timed. A successful upload alone is
not backup acceptance.

Done 2026-08-14: first Backup completed in 48s; full restore 65s
(`audit_core.events` 30=30); PITR 65s after retarget (`keep` present,
`destroy` absent). Evidence
`docs/evidence/RESOURCE-WP-0002-T05-restore-2026-08-14.json`. Scratch
namespace deleted. Production stayed Ready.

## T06 — Establish the independent logical-copy lane

```task
id: RESOURCE-WP-0002-T06
status: done
priority: medium
state_hub_task_id: "76745e5b-a0eb-43b5-a6b7-9d19c06f6110"
```

Retain encrypted per-database logical dumps outside the primary object-storage
failure domain, using Host Europe Backup Storage or the existing governed
Nextcloud lane as justified by the final provider choice. Test a single-consumer
logical restore.

Done when one primary-object-store failure or credential loss does not remove
both recovery copies, and audit-core can be restored independently through the
slower logical path.

Done 2026-08-14: reused CCR-2026-0004 Nextcloud + age (not a new Host
Europe product). Dumped `audit_core`, `platform_meta`,
`isolation_probe`; uploaded to prefix `r01-platform-pg/`. Restore-drill
decrypted `platform-pg-audit_core-20260814T184337Z.dump.age` into a
scratch postgres pod: `audit_core.events` 30=30, trigger present.
Evidence `docs/evidence/RESOURCE-WP-0002-T06-logical-offsite-2026-08-14.md`.

## T07 — Feed fin-hub and operate the optimization loop

```task
id: RESOURCE-WP-0002-T07
status: done
priority: medium
state_hub_task_id: "e6c848ce-ea4b-4ac6-99da-5890de242b50"
```

Publish monthly actual cost, forecast, stored bytes, WAL/base-backup growth,
egress, backup success, and restore-test cost under
`platform:audit-storage`. Set thresholds for budget variance, abnormal growth,
stale backups, and unused commitment.

Re-evaluate A/B/C at least quarterly and before renewal. A switch recommendation
must include migration traffic, overlap period, operator labor, restore proof,
contract timing, and credential revocation—not only a cheaper unit price.

Done when fin-hub receives normalized evidence and resource-control can identify
rightsizing or provider-switch opportunities from observed utilization and
total cost.

Done 2026-08-14: the loop runs on real evidence. First observation
`data/actuals/2026-08.json` records database 0.6365 GB, stored 0.0066 GB over
8 objects, backup success 1/1, restore RTO 1.08 min, and five null proxies each
with a named owner. Thresholds are declared in
`data/thresholds/platform-audit-storage.json` and evaluated fail-closed by
`make thresholds PERIOD=2026-08`: two within, one not applicable, six
unmeasured, zero breaches — `unmeasured` never counts as passing.
`make exchange-usage` emits `technical_usage` records to fin-hub with
measurement gaps carried through and no infrastructure amount, since fin-hub
owns the booked fact. The A/B/C case
`opt:platform-audit-storage:2026-08` is now `approved` by the founder with
Hetzner a costed reject and Host Europe excluded for want of written terms.
The resource moved `ordered` → `active`, commissioned 2026-08-14. Evidence:
`docs/evidence/RESOURCE-WP-0002-T07-control-loop-2026-08-14.md`.

August produces no variance and should not: the decision forecast starts at
2026-09, so August is a commissioning baseline and 2026-09 is the first
comparable month. Total cost stays unknown until Scaleway invoices and fin-hub
books it under `FIN-WP-0004`.