coulomb-social/workplans/CSOC-WP-0012-durable-recovery-service-readiness.md

id	type	title	domain	repo	status	owner	topic_slug	created	updated	origin	origin_ref	state_hub_workstream_id
CSOC-WP-0012	workplan	Durable app recovery and service readiness	communication	coulomb-social	active	codex	coulomb-social	2026-10-08	2026-10-31	residual	CSOC-WP-0008	0cd1f513-f26e-5707-b646-8b4bbe6369fc

Residual acceptance from CSOC-WP-0008, split at the user's request. Existing implementation/release/acceptance evidence remains in the originating workplan; these tasks own the remaining concrete outputs. Source checks never grant runtime or productive acceptance.

Current execution map — 2026-10-31

Completed: T01 independent custody, T02 recovery, T05 dual-destination proof and T06 recurring source/rehearsal. T04 owns retention/capacity; T07 the pending attended route; T08 inactive installation; T09 native operational proof; T10 schedule activation/observation. T11/T12 expose the previously buried service-contract work. T03 remains the final service acceptance gate and is not an additional implementation estimate on top of T07–T12.

The four-workplan assessment records the scope split and effort estimate. Task counts are not effort percentages. Historical notes retain the evidence trail; this map and the task blocks govern current status. No original acceptance requirement has been dropped.

T01 — Admit and verify independent offsite custody

id: CSOC-WP-0012-T01
status: done
priority: high
state_hub_task_id: "dc470337-585f-517d-b2a5-92e89a7dbec8"
needs_human: false

Current scope after the October 31 tracking audit: Independent publisher/reader/key custody and protected-kit retrieval are complete. The original retention, capacity and historical-copy disposition scope continues in live task T04. First application recovery remains completed T02; ongoing monitoring belongs to T03 and T07–T10.

Earlier dated notes below are retained as history, not the current task state.

Use existing RPF-WP-0053-T02/T03/T06 credential-pair/prefix/escrow admission. Own the coulomb-social consumer acceptance, not another credential provisioning lane. Record capacity, retained independent retrieval receipts, RPO/RTO and expiry of temporary production-pair copies; no secrets in records.

2026-10-27: User approved the scoped node executor/delivery (decision 9557e2b8-2b37-5b94-a5e6-e5fe08d0351e). Native own-reader/version/revocation, five reader denials, wrong-account/audience denials, own multipart create/abort, and sibling/native-prefix/delete/bucket-policy/IAM denials pass. The exact hash-pinned helper delivered only this application's version-2 pair to root:root 0600 under a 0700 parent, with equal readback and revoked token. No completed object or timer created. Proofs: docs/release/backup-boundaries-2026-10-27.json and backup-delivery-2026-10-27.json. Independent recovery/receipt custody, retention/capacity and first paired recovery remain open.

2026-10-27 supplement: native DeleteObjectVersion also denies403 on a unique never-published probe key; no retained object touched. Wrong-namespace login with the same ServiceAccount name also denies403, no data or sibling credential read. Receipts: docs/release/backup-version-delete-boundary-2026-10-27.json and backup-wrong-namespace-boundary-2026-10-27.json.

2026-10-28: Prepared docs/release/recovery-custody-review-2026-10-28.md for the additional independent custody boundary: dedicated Coulomb age identity, separate prefix-read-only recovery caller, platform-root recovery/failure owner, protected off-node/workstation kit and last recovered point, and explicit rolling retention/headroom compatibility. Requested that additional admission, without reopening the existing publisher/executor or schedule approval. No recovery key, grant, kit location, capture, timer or lifecycle created while it is pending.

User subsequently approved the packet: decision 27bbc1ac-e275-42be-ba34-6cf16eae4e78. Admission is recorded in RPF-WP-0053; no second approval of that scope is needed. Operator access is a separate execution prerequisite: metadata checks found no SCW operator environment, standard local config or live local browser-debug session. Warden's bootstrap route remains draft/unresolvable, so it is not an executable credential lane. Continue source preparation and use an attended platform-owned operator front door for native provisioning; never substitute a sibling recovery caller/key.

The user identified the existing Scaleway coulomb-social-backup-primary application. It is the already delivered publisher; its tested IAM-administration denial excludes using it to provision a separate recovery reader/escrow. Removed the consumer's old shared public-recipient default. Live execution now requires explicit dedicated public-recipient configuration and validates it before provider auth/capture. Pending sealed ciphertext and final receipts bind that recipient; changed/missing retry binding refuses publication. The paired unit requires an owner-delivered public-only recipient.env. No private identity belongs on the node.

Guard installed on the admitted node after exact prior-source/cluster UID checks, atomic root-owned 0600 replacement and source-hash readback. Native main-path proof with child commands disabled rejects missing recipient before commands or state creation. No pending directories or recipient file exists there; no private identity was installed; no capture/upload/timer activation occurred. Metadata-only proof: docs/release/backup-recipient-guard-2026-10-28.json. Full gate 237 passes, focused backup/recovery 25 passes, unit verification passes. The recovery review packet now contains exact attended Console resource/boundary steps, including isolated project selection to avoid project-wide read leakage. Native operator access and independent custody/recovery/retention proofs remain required.

User clarified that Console login may be used once more for bootstrap but must not become later routine access. Decision 6139e02b-3fb7-4eff-be8a-efe127c45ce5: establish scoped programmatic publisher, archive-reader and custodian escrow access under governed custody, independent of primary node/workstation loss; retire the temporary personal Console session after native verification. No standing organization-wide operator key or publisher IAM administration follows. Stopped the aborted capture (terminal exit1, no state saved), then opened a fresh attended browser and requested login confirmation. Routine operations must not depend on retaining that personal session. No provider resources created yet.

2026-10-29: Scoped API bootstrap is now native-verified. Isolated recovery project, three dedicated applications, four project-only policies, two private/versioned buckets and four protected nonexpiring secret versions are present. Independent custodian API readback validates the dedicated age identity and both routine API pairs without personal login. Thirty native checks prove version reads/listing, publisher create/readback/own abort and denials on deletion, policy/IAM admin, outside-prefix reads, publisher kit access, reader writes and escrow access by publisher/reader (synthetic canary). Publisher pair unchanged; no org-wide grant. The temporary construction key was revoked with GET404 verification, its pair removed and the personal context closed without saved state. Routine front door and public provider coordinates live in railiance-platform's docs/credential-lane-designs/coulomb-recovery-api.{json,md}. Routine pair expiry is 2027-10-28; platform-root owns renewal before the recorded deadlines.

Evidence: docs/release/recovery-api-bootstrap-2026-10-29.json. Synthetic objects are boundary fixtures, not application recovery points. No automatic expiry, capture, restore or timers activated. Full protected app kit/last recovered point, retention/headroom, historical-pair disposition and first app recovery remain open. Native bootstrap no longer requires the personal Console session.

Delivered only the independently verified public recipient to the node's root:root0600 recipient.env after cluster UID and installed guard hash checks; equal readback and real age recipient validation pass. No private identity or recovery API pair was installed there. Receipt: docs/release/backup-public-recipient-delivery-2026-10-29.json.

2026-10-30 retention reconciliation: scoped routine API observations prove the primary bucket's global current/noncurrent 30-day expiry covers Coulomb, while independent archive and protected-kit buckets have no lifecycle rules. Complete prefix inventories and four exact-version HEADs match known primary/mirror/kit/ proof hashes and sizes. No private identity or personal session was used, and no provider mutation occurred. Added an offline reviewer using the hash-pinned owner 14-daily/four-weekly selector, validating complete fresh evidence and independent protected recovery bindings. Actual plan keeps one point, with zero expiration candidates. A real-selector synthetic gap keeps 13 points older than 30 days (oldest47days), so blanket 30-day mirror expiry cannot replace generation selection. Preserve that policy; a separately admitted mirror-only exact-version expiry executor remains needed, with publisher/reader deletion denials intact. Review/proof: docs/release/generation-retention-review-2026-10-30.{md,json}. This is not recurring retention enforcement or additional expiry admission.

2026-10-31 current-inventory return: the date-operation 02:07 point is now mirrored with its immutable public receipt and independent reader verification. A repeatable read-only observer refreshes complete prefix listings, lifecycle, 12 exact-version HEADs and two public-receipt GET comparisons. Four logical points are mapped; the pinned selector keeps three and identifies the earlier October 31 00:27 point as one candidate with its exact receipt. Nothing was deleted. Protected kit/proof custody remains unchanged, all unknown versions and fixtures are preserved, and no age-only mirror expiry was added. Source and native proof: docs/release/generation-retention-review-2026-10-31.{md,json}. Separate expiry executor/caller admission, native boundaries and historical-copy disposition remain open; the fresh review is not recurring retention enforcement.

2026-10-31 expiry-executor preparation: exact proposal6af0564e recomputes the pinned generation selector and binds source/runtime, inventory, protected recovery and targets. Prepared executor journals intent durably before each version-specific mirror delete, verifies kept/protected objects before every action, and reconciles lost replies by native listing/HEAD before retrying. Candidate ciphertext precedes its receipt; unknown versions, primary and kit objects remain outside deletion. Twenty-six synthetic cases and full make check441passes validate the source. Native read-only preflight streamed/verified8,669,264mirror bytes and exact protected version metadata with no mutation. Current candidate remains retained. Review: docs/release/mirror-expiry-executor-review-2026-10-31.md; exact preflight/source proof: mirror-expiry-executor-preflight-2026-10-31.{json,sources.txt}. No admitted local expiry pair is present. Separate caller/custody admission, native boundaries, attended execution and recurring integration remain open; no source or fixture test is substituted for those returns. Inventory drift during a partial run requires owner reconciliation with the journal retained.

2026-10-31 post-date protected-point follow-through: provider kit4166eb8b, version1793432303941465, recovers the corrected07:13 point without original node or local plaintext. Separate escrow identity,2098verified kit members,13tables/ 114rows/86files and40owner/40outsider checks pass in46.692s. Matching successful proof is protected at1793432409572654 with independent readback. The platform public pointer is reconciled from staleOctober29 to this actual recovered point; old kits remain retained. The newly created drill identity copy was removed. Fresh read-only retention observation covers5points/15exact HEADs/3public receipts; the pinned selector keeps3points and identifies2candidates (00:27 and02:07), with nothing deleted. Old expiry proposal6af0564e must be refreshed against this inventory and protected binding before admitted execution. Native caller boundaries, capacity/monitoring, recurring activation and historical-copy disposition remain open. Review: docs/release/provider-kit-recovery-post-date-2026-10-31.md.

T02 — Activate and prove the first encrypted paired recovery point

id: CSOC-WP-0012-T02
status: done
priority: high
state_hub_task_id: "25ae6330-d26f-54ee-a599-2eca3a2e274d"

After admission, run scripts/paired-backup.py consumer, retrieve the exact immutable version independently, decrypt privately and rehearse application recovery with content/hash/membership/denied-read checks. Reconcile post-capture writes before any production restore. Do not call local temporary copies offsite coverage.

2026-10-29: Live preflight verified cluster/image/one Ready replica, all three installed helper hashes and no pending pair. Database physical9.1MB, content2.4MB, node free120GB; conservative capture reserve1GiB. First coherent pair captured and app resumed in54.3s before encryption/upload. Immutable primary version downloaded and hash-checked; ciphertext2,120,392bytes. A separate project mirror was conditionally published and retrieved through the independent reader. Custodian API supplied the dedicated identity; decryption/pair hashes pass without original plaintext.

Isolated PostgreSQL restore matches13tables/112rows/sequences and82files,4.23s for restore only. The exact packaged app renders all39recovered pages through actual views; restored owner access, synthetic viewer read/mutation403 and same-tenant ungranted404 on all39pages pass. Fixture transaction rolled back;82file hashes unchanged. No production restore or external identity-provider call. Portable scripts/verify-restored-application.py checks actual internal Docker network, container/image/port boundaries before executing. Full make check246passes.

Receipts: first-pair-preflight-2026-10-29.json, first-primary-pair-2026-10-29.json, first-mirrored-pair-2026-10-29.json and first-isolated-app-recovery-2026-10-29.json under docs/release/.

The full277,585,920byte kit is now independently stored/retrieved at an immutable version in the protected bucket with live versioning/no-expiry checks. It holds only encrypted application data, exact app/PostgreSQL images, recovery tools, locked SDK and receipts; private identity/API pairs remain in separate escrow. Provider-only recovery verifies all2098members, fetches the age identity through the scoped API, loads the retained images without registry pulls, decrypts/restores and repeats all39page/access checks in30.232s monotonic. Receipt: docs/release/provider-kit-recovery-2026-10-29.json. No original node/plaintext used; operator account recovery time and a fresh Docker daemon are not included. The platform public contract binds the protected kit and exact last recovered point. The successful recovery proof is itself conditionally stored as a separate nonexpiring protected object and independently read back, so the verdict/kit reference also survives node/workstation loss. T02 is done; T01 retention/copy disposition and T03 monitoring/service readiness remain open. No timer or production restore was activated.

2026-10-31 founder refresh: primary version1793404296527269 at23:50:40Z was verified from the native node receipt, independently mirrored/retrieved and restored. All13 tables/113 rows/sequences and86 files match. The new complete protected provider kit (version1793405649619843) recovers the actual app and40 pages in42.651s, with a separately protected proof. A native synthetic startup observation exposed early Unix-socket readiness during PostgreSQL initialization; the isolated restore helper now queries the final target DB over TCP and supports an explicit protected work root. The final kit includes those fixes; make check 396passes. The first refreshed kit's failed drill and historical raw/private copies remain preserved for explicit disposition under T01; no deletion/expiry or recurring activation is inferred. Proof: docs/release/provider-kit-recovery-2026-10-31.json; release review: docs/release/storage-release-review-2026-10-31.md.

T03 — Establish scheduled backup, monitoring and service evidence

id: CSOC-WP-0012-T03
status: progress
priority: high
state_hub_task_id: "ded6fc1d-0819-57c5-9213-e64db76bc6ed"

Enable reviewed backup/freshness units only after the exact first-run return. Verify failed-run/pending retry, stale alert, recovery runbook, observed restore time, cost attribution, SLO and business-app service-contract evidence. Reconcile the existing CSOC-IN-0004 stage-1 live-smoke and deployment receipts with docs/identity/smoke.md before closing that intake. Current revision16 and persistent CONTENT_ROOT are already deployed. Keep apex/Bubble retirement in WP-0014.

2026-10-27: Read-only node check confirms the Coulomb primary credential file is absent; current reader CCR retains false node/provider/boundary admission flags. No backup activation performed. Added requirement-by-requirement service contract §§1–6 assessment in docs/release/service-contract-conformance-2026-10-27.md; conformance remains unproven, including full-client offboarding versus personal export, current finance join and ops registration. The older August recovery assumptions are historical; founder-approved paired schedule already exists in RPF-WP-0053 and does not need repeat schedule approval. Current findings and next returns: docs/release/acceptance-audit-2026-10-27.md.

2026-10-30: Replaced the candidate freshness unit's narrow last-success check with a read-only operating checker covering pending/stalled captures, exact primary and mirror receipt binding, local ciphertext integrity and explicit capacity parameters. A fresh previous point no longer conceals an abandoned newer capture. Native node execution (without installation) found the primary fresh, no pending capture, 2,123,252 state bytes and 119,716,765,696 free bytes, but no matching worker mirror receipt. Both timers remain not-found/inactive. The first independently verified external mirror/kit is intact; recurring mirror/receipt promotion is still missing. No provider contact, alert send, key read, node write or activation. 35 focused tests and source systemd verification pass. Required operating return and proposed capacity limits: docs/release/backup-operating-preflight-2026-10-30.md; native metadata/hash proof is its JSON companion. Received monitoring, retention, pre-capture capacity guard and service-contract returns remain incomplete.

Recurring mirror follow-through: prepared mirror-paired-backup.py using the hash-pinned platform engine and only the own publisher environment. It preserves primary key/version/hash retry binding, conditionally retains a filtered public receipt in the own archive prefix, verifies both exact readbacks and only then promotes last-mirror. Candidate service hooks retry the previous mirror before capture and mirror the new point afterward. The monitor now requires the retained receipt as well as ciphertext evidence. Actual pinned-engine synthetic integration proves normal/lost-reply paths without duplicate retained objects; 53 focused cases pass. Exact units verify on the target with temporary review files only; installed SDK supports the conditional PUT. No provider call, credential read, installed helper change, unit start or timer activation. Review and proof: docs/release/mirror-worker-operating-review-2026-10-30.md and mirror-worker-rehearsal-2026-10-30.json. Native publication/independent retrieval, received alerts, retention and pre-capture capacity remain required.

Capacity follow-through: prepared a shared capacity guard inside each writer's backup lock, before capture/provider work. Candidate profile reserves the complete pipeline's scratch, keeps an 8 GiB free-space floor and 8 GiB local budget, checks inodes/private state and bounds new files to 1 GiB through inherited RLIMIT_FSIZE. Sealed retry reserves less new scratch and preserves all previous pending bytes. Actual subprocess tests prove oversized-write refusal and finally cleanup. Read-only bounded native preflight fits current 2,134,016 state bytes against 118,842,212,352 free bytes; exact candidate units verify. Installed old helpers remain unchanged and storage proposal3389 remains sealed; new operating sources must not replace its expected helper before release execution or explicit rebase. Review/proof: docs/release/backup-capacity-review-2026-10-30.{md,json}. Limits are still proposed; provider retention, native operating publication/retrieval, received alerts and historical-copy disposition remain open. No timer activated.

2026-10-30 client-offboarding follow-through: added a private operator client dataset export and schema-derived inventory, separate from personal portability. It selects tenant records/permissions/admin history and local content, assets, aliases and all retained revisions; rejects ambiguous ownership, pending recovery, source drift and unclassified application schema; excludes session credentials and requires unusable local passwords. No source deletion, teardown or backup expiry is performed. Review: docs/release/client-data-export-review-2026-10-30.md. Current-target release/coherent export acceptance and the owner-governed teardown/ session/log/backup-expiry contract remain open under service-contract §6.2. The exact module overlaid on the retained candidate exported the recovered client's four application record sets and all81local content/revision files with matching archive hashes, zero external bindings and all13source tables/82source files unchanged. Temporary private archive removed. Proof: docs/release/client-data-export-rehearsal-2026-10-30.json. Full make check340passes, zero skips. No deployment, production export, teardown or expiry was performed.

2026-10-30 operational inventory: the prepared date supervisor introduces an operator-private root date-operation store containing original client index bytes, plan and backup binding. Client export inventory now names this store explicitly; the app cannot read/export it. Include any future retained journals in operator offboarding handoff and explicit expiry under the open service contract work. No production date journal or expiry was created by the isolated rehearsal.

2026-10-30 service-source/attribution follow-through: read-only native metadata still observes the old digest and one replica. An entrypoint-bypassed, network-none, read-only retained-image comparison proves all69packaged files match dd83c90; the registry's796a1557source differs in eight paths. Corrected only the app release-unit source pin. The actual fin-hub parser rejects the resource inventory ID as a client attribution key. The resource/default tenant (tenant:coulomb), accepted-cohort SQL tenant (tenant:platform) and financial entity (entity:coulomb) must remain distinct until an authoritative client/instance mapping is returned. No identity rekey, cost, finance ingestion or ops bootstrap is inferred. Local service-catalog slug404/repo0rows does not establish owner-surface absence. Proof/next returns: docs/release/service-provenance-reconciliation-2026-10-30.{md,json}.

2026-10-31 owner-surface follow-through: the documented operations public API fails certificate verification, so no service record was verified there. The finance default local ledger was inspected with SQLite mode=ro: seven legacy entries, no client/application/instance attribution columns. It cannot prove current finance ingestion or costs, nor does it establish absence of another owner-held ledger. No TLS bypass, external write, schema migration or fabricated mapping was used. Bounded observations and required owner returns are appended to docs/release/service-provenance-reconciliation-2026-10-30.md; §§4–5 remain open.

2026-10-31 storage-release return: founder proposal5fd7ccd8 is verified at Helm revision18/image51281946/source17d0eb6. All73packaged source files match; tracked deployment-owner values and native receipt are pushed at8434134 and render exactly the sealed candidate. Snapshot helper4ff81b79/server dry-run and unchanged paired worker43e119ac are verified. This closes the storage dependency; recurring delivery, monitoring/retention/capacity, owner operations registration, canonical finance attribution and the other service-contract returns remain open.

2026-10-31 native mirror protocol return: unchanged candidate worker executed twice from a protected local copy of the founder00:27:56Z primary point, with all four platform runtime pins checked. The admitted publisher creates/readbacks one ciphertext and one1664byte public receipt version; retry preserves those exact versions and local primary/ciphertext bytes. Separate reader exact-version fetches verify both hashes/sizes and the receipt's complete explicit public-field filter. Native exact-point inventory changes0to2versions, with one of each after retry. Proof: docs/release/mirror-worker-native-2026-10-31.{json,sources.txt}. No capture, decryption identity, node helper/state change, timer activation or deletion. Node service execution/received alerts and remaining operating/service-contract returns remain open. Defer installed helper replacement while pending date capsule47829c7a still pins the current installation.

2026-10-31 date-dependency return: founder correction47829c7a is independently verified complete/resumed, with all13tables/114rows/86files matching its exact derived result and current helper pins unchanged. Its installation hold is cleared. Operating delivery/admission, received alerts, retention/capacity/disposition and service-contract returns remain separate; no timer activation is inferred. The new root date journal now exists and remains included in the already tracked operator offboarding/expiry obligation. No journal or historical pair was deleted.

2026-10-31 latest-existing-point mirror: unchanged worker executed twice from a private copy of the native 02:07 date-operation archive. Independent reader GETs verify the single ciphertext and public receipt version; retry changes neither. The node's durable receipt and encrypted bytes were checked before copying. Proof: docs/release/mirror-date-point-native-2026-10-31.{json,sources.txt}. This point predates the date correction and is not a new capture or app restore. No node worker/state/unit/timer changed. Fresh generation observation/selection is recorded under T01; node operating execution, received alerts and the full service contract remain required under T03.

2026-10-31 operations-owner boundary: native core-hub metadata confirms no Ingress and three Ready deployments. The deployment-owner private verification path returns candidate health 200/ok, protected hubs 401 and OpenAPI 200. Public exposure is intentionally gated by HUB-WP-0012-T05/T07 and RAPPCOREHUB-WP-0002-T05; do not treat the public TLS failure as authority to restore Ingress. The schema advertises generic registry/projection ports but no explicit service/client/environment routes; no authenticated catalog read occurred, so record absence is not inferred. Client-hub scoping does not supply the missing canonical client/instance IDs or finance facts. Existing T03 retains the complete registration/attribution/report obligation. Proof: docs/release/service-owner-boundary-2026-10-31.{json,sources.txt}; interpretation and concrete owner dependencies in service-provenance-reconciliation-2026-10-30.md. No credential, catalog write, live configuration or task acceptance changed.

2026-10-31 operating-source reconciliation: the repository snapshot helper still pinned the pre-release image. Updated that source constant to the accepted live digest; its complete hash now equals the already installed helper 4ff81b79. Native default server dry-run passes without capture/pause/provider calls. Sealed the 15 operating source/unit/dependency-evidence files plus manifest into a private review bundle; all 16 member bytes verified and current installed hashes/unit states retained. All four node units remain absent/inactive. Full make check: 441 passed, zero skips. Review: docs/release/backup-operating-delivery-2026-10-31.{md,json}. Capacity admission, rolling retention/expiry, historical-copy disposition, named alert receiver and received signals, exact installation/rollback and native service execution remain open. Requested the missing receiver/response-owner information; no notification was sent. Existing schedule/custody approvals remain valid. No installed helper, unit, timer, data or task acceptance changed.

2026-10-31 packaged client-export candidate: built/published exact source 23ffe2c at immutable digest 522bcd7f, with independently read-back registry digest and all 80 packaged files matching Git. Its actual packaged exporter, without an app source overlay, exports all 85 selected content/revision files on the retained October 30 recovery point; all 13 tables/86 source files remain unchanged and the temporary ZIP is removed. Candidate and existing image each render 40 owner pages and deny 40 outsider requests. All 25 packaged identity/export tests pass; source gate 441 passes. See docs/release/identity-export-candidate-review-2026-10-31.md. This is not a current post-date-correction export or live release. A separate revision 18 image-only execution packet/fresh paired rollback return is required; the old migration launcher must not be replayed. §6.2 teardown/session/log/backup expiry and productive offboarding acceptance remain open.

2026-10-31 current-target image packet: candidate522bcd7f now has sealed proposal 9f1daf6c, with native read-only preflight against revision18, 92business rows and 86post-correction content files plus exact migration history. Nineteen owner release safety cases pass. Founder execution is pending under decision75b94bb6 and RAPPS-WP-0015-T05; it creates/readbacks a new paired primary before upgrade. The prior isolated candidate export used the older recovered point. Independent mirror/retrieval/recovery and packaged client export of the new execution point remain required follow-through, alongside the original operating/service contract obligations. Packet: docs/release/identity-export-release-review-2026-10-31.md.

2026-10-31 founder image-release return: proposal9f1daf6c completed at07:15:24Z, Helm revision19/image522bcd7f. Independent complete-spec/Ready/identity checks, all80packaged source hashes, unchanged92business rows/86files/migration history, 40owner renders/40outsider denials, public endpoints and refreshed helperdcc8dd2a all pass. Live owner values/product pins are reconciled. Fresh07:13 primary point is mirrored, independently retrieved and restored (13tables/114rows/86files); packaged client export verifies85files without source mutation. Full make check 441passes, zero skips. Release decision75b94bb6 is resolved only for that exact execution; RAPPS-WP-0015-T05 is complete. Productive flow and service/offboarding acceptance remain open. Return: docs/release/identity-export-release-return-2026-10-31.md.

The operating delivery sources were resealed for the actual revision19 image; old bundle76af8159 is historical and must not be installed. New bundle060620da retains proposed limits and missing alert/retention/installation returns. No timers were installed or activated, and the previous protected kit remains retained.

2026-10-31 backup alert destination acceptance: founder explicitly selected both the platform-root State Hub inbox and bernd.worsch@gmail.com, response owner platform-root. Resolved decision f4df16b5-32f8-43a7-a8ae-d1c44d1d13ef; configuration deploy/backup/alert-routing.json records both as required. Authorized verification message 8d70c5a7-776b-4ad0-8bf3-11716b70a44f was received centrally and exact fields independently read back. No email has been sent. Prepared a bounded founder-attended SMTP proof using the existing telemetry projection, with native namespace/DNS/policy/ESO checks and server dry-run. Review: docs/release/backup-alert-delivery-review-2026-10-31.md. Email receipt, recurring dual-channel integration, failed/pending/stale signal proof and the existing capacity/retention/copy-disposition/installation obligations remain open under T03/T01. No timers or recurring delivery were activated. The sealed operating bundle needs rebasing with the eventual notifier; destination approval is settled and must not be requested again. Backup response ownership does not settle general reply/support/bounce ownership under WP-0015.

2026-10-31 backup email proof return: founder executed exact proposal51f6cf1f. Native read-only verification matched Job UID6ab74cbb-ab48-4586-9e05-52e322e990e1, cluster/namespace, the complete reviewed Job fields and SMTP Message-ID; one successful completion at09:01:46Z. Founder separately confirmed “Received in inbox” at bernd.worsch@gmail.com. Both selected test destinations are verified. Original provider receipt and independent/human confirmation are preserved in docs/release/backup-alert-email-2026-10-31/{founder-result,verification}.json. No resend, timer activation or production incident occurred. T03 remains progress: recurring failed/pending/stale integration, durable per-channel retry/receipt handling and native received-signal evidence are still required, alongside the existing operating/service-contract obligations. Destination and one-off test acceptance are complete and should not be requested again.

2026-10-31 recurring alert implementation: durable per-channel outbox now connects the candidate hourly health observer and systemd failure handlers to State Hub and the existing Alertmanager email owner. Stable event IDs, independent channel retry, restart persistence, route readback and newer-paired-point failure clearing are implemented; no credentials are added to the consumer. Three-hour alert leases require hourly refresh and independent producer monitoring. A dedicated route proposal preserves unrelated routing. Native amtool config/positive and negative route checks pass; native systemd verifies all five candidate units. An isolated real Alertmanager delivered the synthetic event to loopback SMTP; this caught and fixed the API's millisecond timestamp precision requirement. Review: docs/release/backup-alert-recurring-integration-2026-10-31.md. Live Alertmanager still has only the pending receiver. Owner route/mount review, current operating bundle rebuild/install, native dual-channel incident/replay and producer-death proof remain required. Neither old sealed operating bundle is a complete current installer. All runtime/timer/capacity/retention and other service acceptance gates remain unchanged; no new external alert was sent this turn.

2026-10-31 exact owner route and bundle preparation: live telemetry revision5 has only the empty receiver and no SMTP mount. Owner proposal4bcad54f now preserves the complete87-resource release graph except two reviewed Alertmanager resources. Native Helm server dry-run equals the expected graph exactly, including Grafana's existing PVC binding; native merged amtool config/three routes pass. Owner12 unit tests and package check pass; family validation is blocked by unrelated rapp-forgejo's missing declaration. Review is in rapp-telemetry/docs/ coulomb-backup-route-review-2026-10-31.md; existing RAPP-TELEMETRY-WP-0001-T04 owns the route execution boundary. Rebuilt22-member consumer source bundleeb5dd9ba, manifest84f67186, all163840bytes read back, retaining snapshotdcc8dd2a/platform pins and adding alert helper/handler/env/route evidence. Public inventory: docs/release/backup-operating-delivery-alerts-2026-10-31.json. Executing wrapper, durable receipt/rollback checks and the final supervised installation remain to prepare; no route, unit or timer activated. Existing capacity/retention/copy disposition, received recurring signals and service-contract requirements persist.

2026-10-31 owner execution ready: rapp-telemetry exact execution digest sha256:4d8481061dc9ce6835169defc96887945052332a0db8d4234ae84032d63cc328 binds the reviewed route/mount proposal, launcher/runtime and native identities. Final live87-resource preflight passes; all23 owner tests and isolated native runtime-config matching pass. Durable single-attempt receipt and read-only rollback verification are implemented. Founder command: /home/worsch/rapp-telemetry/docs/ coulomb-backup-route-execution-2026-10-31.md. Awaiting this exact supervised action; no release/route/timer change or external alert was made by preparation. After the founder return, independently verify the receipt and actual route, then complete consumer installation and native received-signal/replay/producer-death proof. Existing capacity/retention/copy-disposition and WP13/14/15 obligations persist.

2026-10-31 inactive installation primitive: strict sealed-bundle reader and transactional source installer now validate the22-member/19-target archive, preserve unchanged/shared pinned helpers, journal preimages and staged identities before replacement, verify inactive units and all file readbacks, and support explicit interruption-safe rollback. Scope excludes credentials, capture and timer starts. Native target observation finds only3 existing files,16 missing, all5 units absent/inactive and both locks free at observation time. Route receipt 4d848106 remains absent and native receiver is still pending-only; the founder command is unchanged. Review: docs/release/backup-source-installer-review-2026-10-31.md. The terminal caller/lock/route/profile gate and exact final installation review remain to prepare; no production target was modified. Original native execution, signal/replay/dead-producer and capacity/retention/disposition/service gates remain.

T04 — Admit retention, capacity and historical-copy disposition

id: CSOC-WP-0012-T04
status: progress
priority: high
state_hub_task_id: "a36722a5-5f13-5728-9d58-24f030c2bb00"
needs_human: true
intervention_note: "Eight-entry completed-evidence custody repair is prepared and native preflight passes; founder terminal command is in docs/release/backup-completed-evidence-custody-review-2026-10-31.md, proposal fa2763d1dcbe92fe638a29c9a7a7d51c03d30849c4232f2d43f81ef705bae611. No change executed. Historical pair disposition and retention/capacity admission remain open; existing destinations and schedule are settled."

Remaining scope from T01. Close with owner-admitted capacity/headroom limits, a current generation/expiry plan, and explicit retention or deletion disposition for historical private production-pair copies. Execute only admitted expiry and retain native receipts. Proposed limits and a deletion candidate list are not admission. Evidence: docs/release/backup-capacity-review-2026-10-30.md and generation-retention-post-date-2026-10-31.json. No deletion has occurred. Existing RPF-WP-0053 custody/expiry ownership remains canonical.

Native installer preflight now identifies an additional concrete custody return: four root-owned0755 release/capsule directories and four helper-owned UID10001 date-operation entries are inside the capacity-accounted state tree. The existing strict guard refuses unsafe_state_entry. Metadata-only evidence is retained in docs/release/backup-state-custody-observation-2026-10-31.json. Prepare a reviewed disposition preserving historical evidence and its ownership/access requirements; do not bypass accounting or silently change/delete it. No body was read or changed.

Prepared the exact bounded custody repair under docs/release/backup-completed-evidence-custody-review-2026-10-31.md. Fresh native checks bind three unchanged terminal operation receipts, absent recorded PIDs, absent date helper and no state hostPath mounts; final preflight passes. The founder-only entry point takes all four locks, journals descriptor-based metadata changes, verifies unchanged bytes/paths and runs the unchanged whole-state custody guard afterward. Nineteen dedicated cases cover refusal, real locks and rollback. No historical ownership/mode change has executed. Receipt/body hashes were computed natively; no private journal body was printed or sent to inference.

The future-producer correction is now completed source task T13, with the release owner return RAPPS-WP-0015-T06. It prevents new intermediate-directory and completed helper ownership exceptions. T04 still owns the exact historical eight-entry repair, retention/capacity admission and historical-copy disposition. Source correction does not execute that repair or establish native recurring-service acceptance.

T05 — Verify both approved backup alert destinations

id: CSOC-WP-0012-T05
status: done
priority: high
state_hub_task_id: "380ecf13-cd24-5930-88c3-fb5dabc70dec"

State Hub central readback verifies event816e6775 and inbox message8d70c5a7 to platform-root. The one-off SMTP proof was independently verified and the user explicitly confirmed receipt at bernd.worsch@gmail.com. Destination decisionf4df16b5 is settled. Evidence: docs/release/backup-alert-destinations-2026-10-31.json and backup-alert-email-2026-10-31/verification.json. The original provider receipt remains unchanged; human receipt confirmation is separate. This closes destination/delivery proof only; recurring monitoring is T07–T10. Do not resend the proof.

T06 — Implement and rehearse recurring alert delivery sources

id: CSOC-WP-0012-T06
status: done
priority: high
state_hub_task_id: "5adde49c-e9a3-51bd-bdce-cba60d6f3ec1"

Durable independent channel attempts, retry/replay, failed/pending/stale episodes and unit integration are implemented. Native isolated Alertmanager-to-local-SMTP, route and systemd validation pass. The sealed operating archive has 22 members and 19 targets. Evidence: docs/release/backup-alert-recurring-integration-2026-10-31.md, backup-alert-isolated-smtp-2026-10-31.json and backup-operating-delivery-alerts-2026-10-31.json. This is source/rehearsal completion; installation and production event delivery remain T07–T09.

T07 — Deploy and independently verify the recurring email route

id: CSOC-WP-0012-T07
status: wait
wait_kind: human
blocking_reason: "Exact reviewed route is prepared; founder terminal execution and its durable verified receipt are pending."
priority: high
state_hub_task_id: "8c8e4810-d967-54b2-ae49-25d367eac459"
needs_human: true
intervention_note: "Existing attended route packet4d848106 awaits founder execution and independent verification. Destination/test-send approval is settled; do not resend the one-off proof."

The exact owner execution packet is ready; durable receipt was absent at the 10:04:37Z observation. Founder execution and independent complete-resource/runtime-config postflight remain. Use the existing reviewed owner packet in /home/worsch/rapp-telemetry/docs/coulomb-backup-route-execution-2026-10-31.md, digest4d848106; inspect any existing attempt before further action. No renewed recipient approval is needed. Close only on durable verified native success, preserving the existing PVC and unrelated routes.

T08 — Finish the supervised source installer and install inactive units

id: CSOC-WP-0012-T08
status: wait
priority: high
state_hub_task_id: "ca41b874-5e09-538e-9692-d31d990d679b"
depends_on:
- CSOC-WP-0012-T04
- CSOC-WP-0012-T07

The installation/rollback library and final supervised wrapper are implemented. The wrapper binds sources, archive/manifest, native preimages, both exclusive locks, full owner route verification, capacity/profile, SDK/custody and inactive/ disabled/no-queued-job guards. Explicit rollback retains preimages and refuses interference. Current native preflight reaches and refuses the capacity guard's historical custody exceptions; T04 now records that exact blocker. T07's route receipt is also still absent. No installation attempt or target change occurred. Then obtain the attended return and independently verify installed hashes and disabled units. Review: docs/release/backup-source-installation-execution-review-2026-10-31.md; source/gate evidence: backup-supervised-installer-validation-2026-10-31.json.

Preparation is complete; task status is now wait for the T04 custody/profile return and T07 attended route. The exact eight-entry custody repair has a passing native preflight but has not executed. Installation and independent postflight still prevent this task from being done. T07 likewise waits for its founder run; neither task represents a currently running native job.

T09 — Prove native paired service and recurring failure reception

id: CSOC-WP-0012-T09
status: todo
priority: high
state_hub_task_id: "c30bc200-6e0c-5651-b53a-be7a6f3dd207"

After T07/T08 and applicable T04 operating admission, prove a native paired service capture, exact primary/mirror receipt, independent retrieval, received failed/pending/stale events in both channels, retry and restart continuity, and outside-producer failure detection. Retain correlated receipts without claiming an Alertmanager handoff is recipient delivery. Simulated source tests and the one-off email cannot close this task.

T10 — Activate and observe the approved recurring schedule

id: CSOC-WP-0012-T10
status: todo
priority: high
state_hub_task_id: "009c9b51-32d4-5a35-a771-7cff0059c042"

After T04 and T07–T09, activate the already approved 03:15 UTC schedule with five-minute jitter, RPO24h/stale26h and attended RTO60min target. Verify native timer/service state and subsequent scheduled capture/freshness evidence. Record measured recovery limits; cached-image rehearsal is not cold-start recovery. Do not ask the user to approve the schedule again. Close on observed operation, not merely enabling a timer.

T11 — Close service registration, provenance, cost and smoke evidence

id: CSOC-WP-0012-T11
status: todo
priority: high
state_hub_task_id: "61f5f196-40e4-5901-a10e-318304809a00"

Remaining T03 service-contract §§1–5 evidence: authenticated ops registration, canonical client × app × instance finance mapping and current ingestion/report, Forgejo successor provenance and governing identity-specialization decision, current SLO evidence, and CSOC-IN-0004 stage-1 smoke reconciliation. Consume existing identity/cohort returns from WP-0014/0015 without duplicating owner implementations. Use docs/release/service-contract-conformance-2026-10-27.md and service-provenance-reconciliation-2026-10-30.md. Resource requests and planning labor are not booked costs.

T12 — Agree and prove complete client offboarding and expiry

id: CSOC-WP-0012-T12
status: todo
priority: high
state_hub_task_id: "54af74b0-3212-537e-a02c-d2d8a091caf9"

Remaining T03 service-contract §6: resolve external/unattributed data, obtain private delivery/acceptance of complete client export, and agree instance teardown, sessions, logs and primary/mirror/protected-kit expiry. Current packaged export proof covers 85 selected files and four record sets; it is not offboarding acceptance. Coordinate retention with T04 and personal portability with WP-0013-T03. Record non-destructive review first; do not infer authorization to delete data.

Completion

Finish when every task above has its stated evidence and acceptance, or hand off any newly discovered residual to a live work record first. Cross-owner dependencies remain their existing canonical records.

T13 — Correct backup evidence producers and recovery closeout

id: CSOC-WP-0012-T13
status: done
priority: high
needs_human: false

Bounded source correction split from T04 after native installer preflight found historical custody exceptions. The current railiance-apps image-release controller creates every capsule directory privately and refuses unsafe existing parents. RAPPS-WP-0015-T06 owns that source return; the historical storage capsule is retained. The current date supervisor leaves helper evidence usable until application resume, then records file fingerprints and transfers the closed evidence to root custody. Interrupted closeout resumes against those fingerprints without another capture, apply, helper creation or scale operation. A stale failure-phase bug during a second recovery interruption is also corrected; completed attempts refuse replay.

Validation: docs/release/backup-producer-custody-validation-2026-10-31.json. Focused release and supervisor cases cover nested modes, unsafe parents, active mounts/locks, changed or unexpected files, partial ownership transfer, and productive writes after resume. Root ownership transitions use a synthetic metadata adapter; ordinary descriptor/file preservation checks use real temporary files. Full make check includes protected browser journeys. No production operation, frozen historical capsule or pending repair/installation/route packet was changed. Future reviewed capsules carry these sources; the existing eight-entry repair remains pending T04, and native recurring evidence remains T09. This task closes source prevention only.