railiance-platform/workplans/RPF-WP-0036-platform-service-assurance.md

---
id: RPF-WP-0036
type: workplan
title: "Close S3 service assurance and ownership gaps"
domain: financials
repo: railiance-platform
status: blocked
owner: codex
created: "2026-09-05"
updated: "2026-09-05"
state_hub_workstream_id: "ca639c3d-3a87-5fa4-ad13-6f2e014b0c84"
---

# S3 service assurance and ownership gaps

Source: `history/2026-09-05-platform-intent-workplan-assessment.md`.
Reviewed against current repository evidence. This plan supplies the missing
continuing obligations; it does not reopen completed bootstrap projects or
duplicate incident/lane work. Repository design and read-only implementation
can progress now. Every live drill, scheduler, credential operation or migration
retains its own owner and execution gate.

## Record the portfolio assessment and consolidate source work

```task
id: RPF-WP-0036-T01
status: done
priority: high
state_hub_task_id: "9c6d0439-8a14-5921-9d88-95a10b303057"
```

Completed 2026-09-05. Assessed all 37 existing plans and their 168 task records,
corrected SCOPE, grouped the remaining obligations, consolidated the three
design follow-ups under RPF-WP-0035, archived completed plans with identities
preserved, and recorded owner handoffs and before/after inventory in history.
This certifies the source review, not live service health or external acceptance.

## Publish achievable service guarantees and recovery ownership

```task
id: RPF-WP-0036-T02
status: done
priority: high
state_hub_task_id: "eebcd5c7-4cfd-5084-9a21-ca1e4e748bdd"
```

For apps-pg, platform-pg, OpenBao and each supported backup delivery lane,
publish a versioned service record: accountable S3/package/operator owners,
consumers, failure domain, availability objective, RPO/RTO, retention, recovery
key/quorum availability, maintenance/abort path and evidence freshness budget.
Separate measured results from accepted targets and unknowns. A 56-second
scratch restore is not an RTO commitment; one replica on one host is not HA.
Reuse `docs/s3-consumer-interfaces.md` and existing package declarations.

**Done when:** every supported service has owner-reviewed numeric targets or
an explicit unsupported guarantee and decision owner; consumer requirements
are compared to the current substrate; any HA/node-loss gap has an exact S1/S2
and package dependency rather than a blanket new-cluster project here.

## Make backup freshness and recurring recovery evidence checkable

```task
id: RPF-WP-0036-T03
status: wait
priority: high
state_hub_task_id: "d64446fb-870b-5ae9-9628-f1fb9d06c5a4"
```

Inventory authoritative CNPG backup/PITR, OpenBao snapshot/isolated restore,
encrypted off-host copy and custody-recovery receipts. Reuse existing validators
and package status commands. Define cadence/expiry from T02; return distinct
healthy, stale, missing and unavailable states using metadata only. Schedule
execution only through the accepted execution owner and separately approved
authority. Keep RPF-WP-0015's pending database/reboot experiments as the sole
live tasks for those experiments; RPF-WP-0029 retains provider-key recovery.

**Done when:** a current off-host backup and a current isolated restore receipt
exist for each supported data service, the approved cadence is installed and
its execution is evidenced, and missing/stale/failed evidence reaches a named
operator. A template, dated successful snapshot, or same-PVC reboot does not
pass as restore proof. Record independent recovery-key access without values.

## Produce S3 signals and prove their delivery to the evidence owner

```task
id: RPF-WP-0036-T04
status: wait
priority: high
state_hub_task_id: "5351e0e4-6263-58f0-afb2-7c78c4cd6f68"
```

Define service-owned health semantics for backup/WAL age, restore age, seal
state, ESO freshness, connection/memory headroom and consumer ceiling. Reuse
package emitters and the Q2 owner's standard contract; retain an explicit
unmonitored state and named manual checker until transport is accepted.
Request a concrete receiving contract from railiance-telemetry when routing is
authorized; do not implement a competing monitoring plane in S3.

**Done when:** bounded metadata-only samples pass contract validation, a
controlled stale/failure sample reaches a named recipient through the accepted
Q2 route, and missing emission itself is detectable. Local fixture tests may
finish before the receiver, but end-to-end acceptance cannot.

## Reconcile admission, placement and consumer interface drift

```task
id: RPF-WP-0036-T05
status: done
priority: high
state_hub_task_id: "5cded2e9-7edd-57bd-9f7e-7977b75004dc"
```

Join actual package declarations and authorized metadata to the S3 interface,
tenancy and placement records. Correct stale platform-pg occupancy/co-residency
(Core Hub admission versus older tenant-engine descriptions), distinguish
desired placement from observed placement, and verify the named overflow
targets remain provisionable. Add a bounded check for missing owners,
unsupported retention requests, quota/ceiling drift and stale evidence; consume
package admission checks instead of reimplementing them.

**Done when:** every admitted consumer has one authoritative placement/contract,
capacity and retention disclosures match package source and dated live proof,
and synthetic invalid admissions fail before provisioning. No workload moves
under this task without its own owner-reviewed migration.

## Obtain acceptance for compatibility assets and derived-record cleanup

```task
id: RPF-WP-0036-T06
status: wait
priority: medium
state_hub_task_id: "3b74f79c-7366-5d00-bf0e-31c0464a8f33"
```

Prepare exact source/entry-point inventories and owner-ready handoffs for
Forgejo backup/pruning/image inventory (`railiance-forge`, activity-core
execution), retained OpenBao package wrappers (`rapp-openbao`), and ArgoCD
bootstrap/application manifests (S2/S4/S5 according to artifact). Keep S3
custody contracts and the RPF-WP-0029 exposure obligation here until accepted
closure. No new framework or app-specific helper belongs here by default.

Supply repo-manager/State Hub with the exact legacy alias/source identity map
from the assessment. Their apparent duplicate active records and stale brief
are derived-state defects, not additional workplans. Use scoped reconciliation;
never change managed UUIDs or blanket-acknowledge retirements to clean a view.

**Done when:** each retained compatibility surface has an accepting owner,
canonical replacement and tested callers or a dated retention decision; the
repo-filtered projection and generated brief agree with source identities.
Unaccepted transfer remains explicitly pending. No requests were sent during
the assessment and this task does not assert acceptance for another repo.

## Decide demand and reuse for undeployed stateful capabilities

```task
id: RPF-WP-0036-T07
status: done
priority: medium
state_hub_task_id: "b32709d2-581d-5750-a04c-e3494a42d27e"
```

Review cache, general object storage and messaging separately with potential
consumers. Inventory existing providers/contracts (including artifact-store
and the external backup bucket) before selecting an engine. Record workload,
durability/latency/retention needs, capacity, tenancy, custody, package owner,
recovery cost and operating owner for any accepted demand. Ask railiance-master
to resolve fleet Q3 ownership through its architecture process; do not assign
it to S3 by implication.

**Done when:** each capability has a dated decision to reuse, defer with a
review trigger, or start a bounded consumer-backed delivery plan with explicit
acceptance criteria. “No accepted demand; keep deploy gated” is a valid result.
No Valkey, MinIO, RabbitMQ or new provider purchase is authorized by this plan.

## Implementation and remaining acceptance — 2026-09-05

T02 completed with `assurance/service-records.json` and ADR-0004: all three
CNPG cells, OpenBao and both backup delivery surfaces disclose unsupported
availability/RPO/RTO guarantees, existing evidence, retention and named decision
owners. Numeric commitments have not been invented or approved for other owners.

T05 completed with the owner-native admission checker, hash-bound baseline and
placement registry. It reuses local/package validators and rejects unowned
consumers, fifth consumers, unhonoured retention and source disclosure drift.
Corrected tenant-engine's completed PostgreSQL cutover and the deployed SBOM
overflow cell. Live metadata matches one instance, 1Gi limit, 100 connections
and 30-day retention on all three deployed cells. Dated owner evidence remains
the authority for database/application cutover; this run moved no workloads.

T07 completed with ADR-0005: reuse approved backup storage only for backup;
evaluate artifact-store's existing S3 interface for artifact demand; defer
cache, general object store and broker deployment with concrete demand/review
triggers. The fleet Q3 question is retained in T06's prepared master handoff;
no external architecture assignment or provisioning occurred.

T03/T04 have working read-only `assurance-capture`, `assurance-check` and
`assurance-admission` entry points plus adversarial tests. The collector pins
the cluster UID and reads selected status, native aggregate connection counts,
metrics and unauthenticated seal status. It captures no Secrets, SQL text from
sessions, application rows or logs. Healthy/stale/missing/unavailable/failed
are distinct; unknown fields, future timestamps and wrong clusters fail closed.
The output explicitly says unmonitored and unsupported guarantees.

- **T03 waits:** fresh owner-validated isolated restore/snapshot/offsite receipts,
  independent recovery access, approved cadence and execution evidence. Existing
  live experiments stay in RPF-WP-0015; offsite exposure stays in RPF-WP-0029.
  No restore, snapshot creation, upload, scheduler or seal/unseal was run.
- **T04 waits:** accepted Q2 receiving contract and controlled failure/absence
  delivery to a named recipient. The local producer is not a ratified Q2
  contract. Live metadata also surfaced failed ESO resources: forgejo-mailer,
  reuse-surface-runtime and target-revenue-runtime. Their current/obsolete scope
  and exact repair require consumer/platform acceptance before lane mutation;
  failures remain visible rather than excluded.
- **T06 waits:** ownership acceptance and scoped alias/brief repair. The exact
  path/hash/caller inventory, dated retention decision through 2026-10-05 and
  prepared owner requests are in `assurance/ownership-handoffs.json` and
  `docs/platform-ownership-handoffs.md`. The installed brief generator still
  includes open legacy aliases; rerunning it alone cannot satisfy acceptance.
  No messages or external handoff acceptance were fabricated.

Evidence: `docs/evidence/RPF-WP-0036-assurance-2026-09-05.json`.
The plan is blocked on these explicit live/owner gates, not finished merely
because its repository implementation and tests pass.


2026-09-05 ESO recovery follow-up: RPF-WP-0037 resolved all three delivery
failures using exact Kubernetes authentication. Custody values were unchanged,
negative checks and repeated refresh passed, obsolete invalid delivery tokens
were removed, and all 27 ExternalSecrets now report Ready. The earlier dated
assurance snapshot is retained as historical evidence. T04 still waits on the
accepted Q2 receiver and controlled failure/absence transport proof.