coulomb-social/workplans/CSOC-WP-0012-durable-recovery-service-readiness.md

id	type	title	domain	repo	status	owner	topic_slug	created	updated	origin	origin_ref	state_hub_workstream_id
CSOC-WP-0012	workplan	Durable app recovery and service readiness	communication	coulomb-social	active	codex	coulomb-social	2026-10-08	2026-10-30	residual	CSOC-WP-0008	0cd1f513-f26e-5707-b646-8b4bbe6369fc

Residual acceptance from CSOC-WP-0008, split at the user's request. Existing implementation/release/acceptance evidence remains in the originating workplan; these tasks own the remaining concrete outputs. Source checks never grant runtime or productive acceptance.

T01 — Admit independent offsite custody and retention

id: CSOC-WP-0012-T01
status: progress
priority: high
needs_human: true
intervention_note: "Scoped API custody, protected kit and first application recovery are native-verified; no repeat scope/schedule approval or routine personal login is needed. Historical local production-pair copies still need explicit owner disposition. Retention/capacity and monitored scheduling remain open."
state_hub_task_id: "dc470337-585f-517d-b2a5-92e89a7dbec8"

Use existing RPF-WP-0053-T02/T03/T06 credential-pair/prefix/escrow admission. Own the coulomb-social consumer acceptance, not another credential provisioning lane. Record capacity, retained independent retrieval receipts, RPO/RTO and expiry of temporary production-pair copies; no secrets in records.

2026-10-27: User approved the scoped node executor/delivery (decision 9557e2b8-2b37-5b94-a5e6-e5fe08d0351e). Native own-reader/version/revocation, five reader denials, wrong-account/audience denials, own multipart create/abort, and sibling/native-prefix/delete/bucket-policy/IAM denials pass. The exact hash-pinned helper delivered only this application's version-2 pair to root:root 0600 under a 0700 parent, with equal readback and revoked token. No completed object or timer created. Proofs: docs/release/backup-boundaries-2026-10-27.json and backup-delivery-2026-10-27.json. Independent recovery/receipt custody, retention/capacity and first paired recovery remain open.

2026-10-27 supplement: native DeleteObjectVersion also denies403 on a unique never-published probe key; no retained object touched. Wrong-namespace login with the same ServiceAccount name also denies403, no data or sibling credential read. Receipts: docs/release/backup-version-delete-boundary-2026-10-27.json and backup-wrong-namespace-boundary-2026-10-27.json.

2026-10-28: Prepared docs/release/recovery-custody-review-2026-10-28.md for the additional independent custody boundary: dedicated Coulomb age identity, separate prefix-read-only recovery caller, platform-root recovery/failure owner, protected off-node/workstation kit and last recovered point, and explicit rolling retention/headroom compatibility. Requested that additional admission, without reopening the existing publisher/executor or schedule approval. No recovery key, grant, kit location, capture, timer or lifecycle created while it is pending.

User subsequently approved the packet: decision 27bbc1ac-e275-42be-ba34-6cf16eae4e78. Admission is recorded in RPF-WP-0053; no second approval of that scope is needed. Operator access is a separate execution prerequisite: metadata checks found no SCW operator environment, standard local config or live local browser-debug session. Warden's bootstrap route remains draft/unresolvable, so it is not an executable credential lane. Continue source preparation and use an attended platform-owned operator front door for native provisioning; never substitute a sibling recovery caller/key.

The user identified the existing Scaleway coulomb-social-backup-primary application. It is the already delivered publisher; its tested IAM-administration denial excludes using it to provision a separate recovery reader/escrow. Removed the consumer's old shared public-recipient default. Live execution now requires explicit dedicated public-recipient configuration and validates it before provider auth/capture. Pending sealed ciphertext and final receipts bind that recipient; changed/missing retry binding refuses publication. The paired unit requires an owner-delivered public-only recipient.env. No private identity belongs on the node.

Guard installed on the admitted node after exact prior-source/cluster UID checks, atomic root-owned 0600 replacement and source-hash readback. Native main-path proof with child commands disabled rejects missing recipient before commands or state creation. No pending directories or recipient file exists there; no private identity was installed; no capture/upload/timer activation occurred. Metadata-only proof: docs/release/backup-recipient-guard-2026-10-28.json. Full gate 237 passes, focused backup/recovery 25 passes, unit verification passes. The recovery review packet now contains exact attended Console resource/boundary steps, including isolated project selection to avoid project-wide read leakage. Native operator access and independent custody/recovery/retention proofs remain required.

User clarified that Console login may be used once more for bootstrap but must not become later routine access. Decision 6139e02b-3fb7-4eff-be8a-efe127c45ce5: establish scoped programmatic publisher, archive-reader and custodian escrow access under governed custody, independent of primary node/workstation loss; retire the temporary personal Console session after native verification. No standing organization-wide operator key or publisher IAM administration follows. Stopped the aborted capture (terminal exit1, no state saved), then opened a fresh attended browser and requested login confirmation. Routine operations must not depend on retaining that personal session. No provider resources created yet.

2026-10-29: Scoped API bootstrap is now native-verified. Isolated recovery project, three dedicated applications, four project-only policies, two private/versioned buckets and four protected nonexpiring secret versions are present. Independent custodian API readback validates the dedicated age identity and both routine API pairs without personal login. Thirty native checks prove version reads/listing, publisher create/readback/own abort and denials on deletion, policy/IAM admin, outside-prefix reads, publisher kit access, reader writes and escrow access by publisher/reader (synthetic canary). Publisher pair unchanged; no org-wide grant. The temporary construction key was revoked with GET404 verification, its pair removed and the personal context closed without saved state. Routine front door and public provider coordinates live in railiance-platform's docs/credential-lane-designs/coulomb-recovery-api.{json,md}. Routine pair expiry is 2027-10-28; platform-root owns renewal before the recorded deadlines.

Evidence: docs/release/recovery-api-bootstrap-2026-10-29.json. Synthetic objects are boundary fixtures, not application recovery points. No automatic expiry, capture, restore or timers activated. Full protected app kit/last recovered point, retention/headroom, historical-pair disposition and first app recovery remain open. Native bootstrap no longer requires the personal Console session.

Delivered only the independently verified public recipient to the node's root:root0600 recipient.env after cluster UID and installed guard hash checks; equal readback and real age recipient validation pass. No private identity or recovery API pair was installed there. Receipt: docs/release/backup-public-recipient-delivery-2026-10-29.json.

2026-10-30 retention reconciliation: scoped routine API observations prove the primary bucket's global current/noncurrent 30-day expiry covers Coulomb, while independent archive and protected-kit buckets have no lifecycle rules. Complete prefix inventories and four exact-version HEADs match known primary/mirror/kit/ proof hashes and sizes. No private identity or personal session was used, and no provider mutation occurred. Added an offline reviewer using the hash-pinned owner 14-daily/four-weekly selector, validating complete fresh evidence and independent protected recovery bindings. Actual plan keeps one point, with zero expiration candidates. A real-selector synthetic gap keeps 13 points older than 30 days (oldest47days), so blanket 30-day mirror expiry cannot replace generation selection. Preserve that policy; a separately admitted mirror-only exact-version expiry executor remains needed, with publisher/reader deletion denials intact. Review/proof: docs/release/generation-retention-review-2026-10-30.{md,json}. This is not recurring retention enforcement or additional expiry admission.

T02 — Activate and prove the first encrypted paired recovery point

id: CSOC-WP-0012-T02
status: done
priority: high
state_hub_task_id: "25ae6330-d26f-54ee-a599-2eca3a2e274d"

After admission, run scripts/paired-backup.py consumer, retrieve the exact immutable version independently, decrypt privately and rehearse application recovery with content/hash/membership/denied-read checks. Reconcile post-capture writes before any production restore. Do not call local temporary copies offsite coverage.

2026-10-29: Live preflight verified cluster/image/one Ready replica, all three installed helper hashes and no pending pair. Database physical9.1MB, content2.4MB, node free120GB; conservative capture reserve1GiB. First coherent pair captured and app resumed in54.3s before encryption/upload. Immutable primary version downloaded and hash-checked; ciphertext2,120,392bytes. A separate project mirror was conditionally published and retrieved through the independent reader. Custodian API supplied the dedicated identity; decryption/pair hashes pass without original plaintext.

Isolated PostgreSQL restore matches13tables/112rows/sequences and82files,4.23s for restore only. The exact packaged app renders all39recovered pages through actual views; restored owner access, synthetic viewer read/mutation403 and same-tenant ungranted404 on all39pages pass. Fixture transaction rolled back;82file hashes unchanged. No production restore or external identity-provider call. Portable scripts/verify-restored-application.py checks actual internal Docker network, container/image/port boundaries before executing. Full make check246passes.

Receipts: first-pair-preflight-2026-10-29.json, first-primary-pair-2026-10-29.json, first-mirrored-pair-2026-10-29.json and first-isolated-app-recovery-2026-10-29.json under docs/release/.

The full277,585,920byte kit is now independently stored/retrieved at an immutable version in the protected bucket with live versioning/no-expiry checks. It holds only encrypted application data, exact app/PostgreSQL images, recovery tools, locked SDK and receipts; private identity/API pairs remain in separate escrow. Provider-only recovery verifies all2098members, fetches the age identity through the scoped API, loads the retained images without registry pulls, decrypts/restores and repeats all39page/access checks in30.232s monotonic. Receipt: docs/release/provider-kit-recovery-2026-10-29.json. No original node/plaintext used; operator account recovery time and a fresh Docker daemon are not included. The platform public contract binds the protected kit and exact last recovered point. The successful recovery proof is itself conditionally stored as a separate nonexpiring protected object and independently read back, so the verdict/kit reference also survives node/workstation loss. T02 is done; T01 retention/copy disposition and T03 monitoring/service readiness remain open. No timer or production restore was activated.

T03 — Establish scheduled backup, monitoring and service evidence

id: CSOC-WP-0012-T03
status: progress
priority: high
state_hub_task_id: "ded6fc1d-0819-57c5-9213-e64db76bc6ed"

Enable reviewed backup/freshness units only after the exact first-run return. Verify failed-run/pending retry, stale alert, recovery runbook, observed restore time, cost attribution, SLO and business-app service-contract evidence. Reconcile the existing CSOC-IN-0004 stage-1 live-smoke and deployment receipts with docs/identity/smoke.md before closing that intake. Current revision16 and persistent CONTENT_ROOT are already deployed. Keep apex/Bubble retirement in WP-0014.

2026-10-27: Read-only node check confirms the Coulomb primary credential file is absent; current reader CCR retains false node/provider/boundary admission flags. No backup activation performed. Added requirement-by-requirement service contract §§1–6 assessment in docs/release/service-contract-conformance-2026-10-27.md; conformance remains unproven, including full-client offboarding versus personal export, current finance join and ops registration. The older August recovery assumptions are historical; founder-approved paired schedule already exists in RPF-WP-0053 and does not need repeat schedule approval. Current findings and next returns: docs/release/acceptance-audit-2026-10-27.md.

2026-10-30: Replaced the candidate freshness unit's narrow last-success check with a read-only operating checker covering pending/stalled captures, exact primary and mirror receipt binding, local ciphertext integrity and explicit capacity parameters. A fresh previous point no longer conceals an abandoned newer capture. Native node execution (without installation) found the primary fresh, no pending capture, 2,123,252 state bytes and 119,716,765,696 free bytes, but no matching worker mirror receipt. Both timers remain not-found/inactive. The first independently verified external mirror/kit is intact; recurring mirror/receipt promotion is still missing. No provider contact, alert send, key read, node write or activation. 35 focused tests and source systemd verification pass. Required operating return and proposed capacity limits: docs/release/backup-operating-preflight-2026-10-30.md; native metadata/hash proof is its JSON companion. Received monitoring, retention, pre-capture capacity guard and service-contract returns remain incomplete.

Recurring mirror follow-through: prepared mirror-paired-backup.py using the hash-pinned platform engine and only the own publisher environment. It preserves primary key/version/hash retry binding, conditionally retains a filtered public receipt in the own archive prefix, verifies both exact readbacks and only then promotes last-mirror. Candidate service hooks retry the previous mirror before capture and mirror the new point afterward. The monitor now requires the retained receipt as well as ciphertext evidence. Actual pinned-engine synthetic integration proves normal/lost-reply paths without duplicate retained objects; 53 focused cases pass. Exact units verify on the target with temporary review files only; installed SDK supports the conditional PUT. No provider call, credential read, installed helper change, unit start or timer activation. Review and proof: docs/release/mirror-worker-operating-review-2026-10-30.md and mirror-worker-rehearsal-2026-10-30.json. Native publication/independent retrieval, received alerts, retention and pre-capture capacity remain required.

Capacity follow-through: prepared a shared capacity guard inside each writer's backup lock, before capture/provider work. Candidate profile reserves the complete pipeline's scratch, keeps an 8 GiB free-space floor and 8 GiB local budget, checks inodes/private state and bounds new files to 1 GiB through inherited RLIMIT_FSIZE. Sealed retry reserves less new scratch and preserves all previous pending bytes. Actual subprocess tests prove oversized-write refusal and finally cleanup. Read-only bounded native preflight fits current 2,134,016 state bytes against 118,842,212,352 free bytes; exact candidate units verify. Installed old helpers remain unchanged and storage proposal3389 remains sealed; new operating sources must not replace its expected helper before release execution or explicit rebase. Review/proof: docs/release/backup-capacity-review-2026-10-30.{md,json}. Limits are still proposed; provider retention, native operating publication/retrieval, received alerts and historical-copy disposition remain open. No timer activated.

Completion

Finish when every task above has its stated evidence and acceptance, or hand off any newly discovered residual to a live work record first. Cross-owner dependencies remain their existing canonical records.