From 5567fa979e52ab50fb174811a496335ce6a7e89b Mon Sep 17 00:00:00 2001 From: codex Date: Sun, 6 Sep 2026 01:36:39 +0200 Subject: [PATCH] Record successful full offsite Forgejo recovery and remaining custody gate Assistant: codex Assistant-Model: gpt-6-astra Assistant-Session: 01a06ecb-456a-71c2-b41e-0755d336e883 --- SCOPE.md | 5 +-- docs/backup-provider-coverage.md | 8 +++- ...9-rejected-fixture-cleanup-2026-09-06.json | 17 +++++++ ...-WP-0029-secondary-restore-2026-09-06.json | 24 ++++++++++ ...secondary-restore-attempt1-2026-09-06.json | 13 ++++++ ...secondary-restore-attempt2-2026-09-06.json | 15 +++++++ ...WP-0029-secondary-transfer-2026-09-06.json | 25 +++++++++++ history/2026-09-06-backup-problem-followup.md | 45 ++++++++++++++++++- workplans/README.md | 2 +- ...-0029-backup-credential-default-removal.md | 37 +++++++++++++-- ...-0038-forgejo-scaleway-primary-coverage.md | 6 +++ 11 files changed, 187 insertions(+), 10 deletions(-) create mode 100644 docs/evidence/RPF-WP-0029-rejected-fixture-cleanup-2026-09-06.json create mode 100644 docs/evidence/RPF-WP-0029-secondary-restore-2026-09-06.json create mode 100644 docs/evidence/RPF-WP-0029-secondary-restore-attempt1-2026-09-06.json create mode 100644 docs/evidence/RPF-WP-0029-secondary-restore-attempt2-2026-09-06.json create mode 100644 docs/evidence/RPF-WP-0029-secondary-transfer-2026-09-06.json diff --git a/SCOPE.md b/SCOPE.md index fb0b9ae..5bcafe1 100644 --- a/SCOPE.md +++ b/SCOPE.md @@ -127,9 +127,8 @@ validated restore adapters and Q2 delivery remain open. See [service assurance](docs/service-assurance.md) and RPF-WP-0036. The dedicated Nextcloud Backup account is active with a 10 GiB provider quota -and a create-only workload share. Encrypted fixture recovery and consumer -refresh are verified; full application restore and predecessor invalidation -remain RPF-WP-0029-T02. See the +and a create-only workload share. Full offsite application recovery and consumer +refresh are verified; predecessor invalidation remains RPF-WP-0029-T02. See the [latest blocked-workplan review](history/2026-09-05-blocked-workplan-closure-review.md). diff --git a/docs/backup-provider-coverage.md b/docs/backup-provider-coverage.md index ce49202..e490aca 100644 --- a/docs/backup-provider-coverage.md +++ b/docs/backup-provider-coverage.md @@ -10,7 +10,7 @@ Backup, 10 GiB quota. Provider selection does not establish asset coverage. | platform-pg | Scaleway Barman base backups + WAL, `platform-pg/` | Earlier package restore evidence; independent logical Nextcloud copy | | platform-pg-2 | Scaleway Barman base backups + WAL, `platform-pg/platform-pg-2/` | Earlier package restore evidence; independent logical Nextcloud copy | | forgejo-db | Scaleway Barman base backups + WAL, `platform-pg/forgejo-db/`; daily 02:35 UTC, 30-day retention | Fresh isolated physical restore passed in 45.08 seconds; full archive coverage remains separate | -| Forgejo repositories/packages/blobs | No reviewed Scaleway archive destination found | Corrected full-archive capture; 5.35 GB verified encrypted artifact staged; secondary download/application restore still pending | +| Forgejo repositories/packages/blobs | No reviewed Scaleway archive destination found | Corrected full-archive capture; 5.35 GB encrypted secondary download and isolated Forgejo recovery passed, including all 2,040 package blobs | | net-kingdom-pg / state-hub-db | No native Barman destination observed | Do not infer protection from the shared cells' healthy backup status | | OpenBao / S1 host configuration | Not evaluated by this database restore | Their own encrypted snapshot/host backup and recovery contracts still apply | @@ -49,3 +49,9 @@ and replacement recovery. The full primary coverage gap belongs to S3 assurance RPF-WP-0038-T04 tracks the remaining primary full-archive delivery and application recovery contract. Native database recovery verified 142 repository records, six users and 2,040 package blob records; it did not recover the blob files. + +Secondary full recovery evidence is +`docs/evidence/RPF-WP-0029-secondary-restore-2026-09-06.json`. Immediate quota +after rejected-fixture cleanup still reported 5.23 GB free versus the 5.35 GB +current archive. Recheck quota before another upload and establish bounded +retention/capacity; the 10 GiB account is not an indefinite archive store. diff --git a/docs/evidence/RPF-WP-0029-rejected-fixture-cleanup-2026-09-06.json b/docs/evidence/RPF-WP-0029-rejected-fixture-cleanup-2026-09-06.json new file mode 100644 index 0000000..51c43cf --- /dev/null +++ b/docs/evidence/RPF-WP-0029-rejected-fixture-cleanup-2026-09-06.json @@ -0,0 +1,17 @@ +{ + "schema": "platform.rejected-recovery-fixture-cleanup.v1", + "status": "removed", + "quota_before": { + "quota_bytes": 10737418240, + "used_bytes": 5502701722, + "available_bytes": 5234716518 + }, + "artifact": "wp0029-recovery-forgejo-dump-20260904T001507Z.zip.age", + "delete_http_status": 204, + "quota_after": { + "quota_bytes": 10737418240, + "used_bytes": 5502701722, + "available_bytes": 5234716518 + }, + "warden_exit_code": 0 +} diff --git a/docs/evidence/RPF-WP-0029-secondary-restore-2026-09-06.json b/docs/evidence/RPF-WP-0029-secondary-restore-2026-09-06.json new file mode 100644 index 0000000..5301131 --- /dev/null +++ b/docs/evidence/RPF-WP-0029-secondary-restore-2026-09-06.json @@ -0,0 +1,24 @@ +{ + "schema": "platform.forgejo-isolated-restore.v1", + "status": "restored", + "images": [ + "code.forgejo.org/forgejo/forgejo@sha256:e2684fd8707d486329084a695ed91999a4072a798e5409d45c1eb8a2911ff4b9", + "postgres@sha256:ff23cdce56cac62ada6f66013e1a50864c0abbe79d132d40b6e05bd80f378a70" + ], + "offsite_artifact": "wp0029-recovery-forgejo-dump-20260905-verified.zip.age", + "ciphertext_sha256": "00d2a5b099bc0e65b2109287fca3bd9d6eb1c8a4e43623520a6e9f3379904018", + "stage": "package_blob_recovery", + "database_import": true, + "database_counts": { + "repositories": 142, + "users": 6, + "package_blobs": 2040 + }, + "application_health": true, + "repositories_verified": [ + "coulomb/key-cape", + "coulomb/adaptive-pricing" + ], + "package_blobs_verified": 2040, + "cleanup": true +} diff --git a/docs/evidence/RPF-WP-0029-secondary-restore-attempt1-2026-09-06.json b/docs/evidence/RPF-WP-0029-secondary-restore-attempt1-2026-09-06.json new file mode 100644 index 0000000..633aef0 --- /dev/null +++ b/docs/evidence/RPF-WP-0029-secondary-restore-attempt1-2026-09-06.json @@ -0,0 +1,13 @@ +{ + "schema": "platform.forgejo-isolated-restore.v1", + "status": "failed", + "images": [ + "code.forgejo.org/forgejo/forgejo@sha256:e2684fd8707d486329084a695ed91999a4072a798e5409d45c1eb8a2911ff4b9", + "postgres@sha256:ff23cdce56cac62ada6f66013e1a50864c0abbe79d132d40b6e05bd80f378a70" + ], + "offsite_artifact": "wp0029-recovery-forgejo-dump-20260905-verified.zip.age", + "ciphertext_sha256": "00d2a5b099bc0e65b2109287fca3bd9d6eb1c8a4e43623520a6e9f3379904018", + "stage": "isolated_database", + "cleanup": true, + "error": "isolated_restore_failed" +} diff --git a/docs/evidence/RPF-WP-0029-secondary-restore-attempt2-2026-09-06.json b/docs/evidence/RPF-WP-0029-secondary-restore-attempt2-2026-09-06.json new file mode 100644 index 0000000..dc9fadc --- /dev/null +++ b/docs/evidence/RPF-WP-0029-secondary-restore-attempt2-2026-09-06.json @@ -0,0 +1,15 @@ +{ + "schema": "platform.forgejo-isolated-restore.v1", + "status": "failed", + "images": [ + "code.forgejo.org/forgejo/forgejo@sha256:e2684fd8707d486329084a695ed91999a4072a798e5409d45c1eb8a2911ff4b9", + "postgres@sha256:ff23cdce56cac62ada6f66013e1a50864c0abbe79d132d40b6e05bd80f378a70" + ], + "offsite_artifact": "wp0029-recovery-forgejo-dump-20260905-verified.zip.age", + "ciphertext_sha256": "00d2a5b099bc0e65b2109287fca3bd9d6eb1c8a4e43623520a6e9f3379904018", + "stage": "isolated_database", + "failure_kind": "command_nonzero", + "command_exit_code": 2, + "cleanup": true, + "error": "isolated_restore_failed" +} diff --git a/docs/evidence/RPF-WP-0029-secondary-transfer-2026-09-06.json b/docs/evidence/RPF-WP-0029-secondary-transfer-2026-09-06.json new file mode 100644 index 0000000..3e01c45 --- /dev/null +++ b/docs/evidence/RPF-WP-0029-secondary-transfer-2026-09-06.json @@ -0,0 +1,25 @@ +{ + "schema": "platform.real-offsite-recovery.v1", + "status": "offsite_fetched_pending_isolated_restore", + "quota_before": { + "quota_bytes": 10737418240, + "used_bytes": 149677429, + "available_bytes": 10587740811 + }, + "artifact": "wp0029-recovery-forgejo-dump-20260905-verified.zip.age", + "source_backup": "forgejo-dump-20260905-verified.zip.age", + "kv_version": 3, + "upload_http_status": 201, + "download_http_status": 200, + "ciphertext_bytes": 5353024293, + "ciphertext_sha256": "00d2a5b099bc0e65b2109287fca3bd9d6eb1c8a4e43623520a6e9f3379904018", + "decrypted": true, + "plaintext_sha256": "fe53d156461387ea487cddc5d6a01f0b5e3debda6fa494c8b87afc8bc62b55b2", + "quota_after": { + "quota_bytes": 10737418240, + "used_bytes": 5502701722, + "available_bytes": 5234716518 + }, + "warden_exit_code": 0, + "attended_session_completed": true +} diff --git a/history/2026-09-06-backup-problem-followup.md b/history/2026-09-06-backup-problem-followup.md index 112ae62..0b48454 100644 --- a/history/2026-09-06-backup-problem-followup.md +++ b/history/2026-09-06-backup-problem-followup.md @@ -7,7 +7,11 @@ routing probe returned the expected Keycloak host and localhost callback. Implemented a narrowly validated, repository-scoped Windows browser bridge and retained Warden's contained credential session. The replacement login reached the silent owner command and began the real secondary archive transfer. -Transfer and revocation results are recorded separately when complete. +The 5,353,024,293-byte encrypted archive uploaded with HTTP 201, downloaded +with HTTP 200 and matched the source ciphertext hash. Decryption matched the +validated source archive hash. Warden exited 0 after its contained session, +including its self-revocation/cleanup path. The isolated application recovery +is recorded separately below. Forgejo's native PostgreSQL backup is now active on the primary Scaleway bucket at `platform-pg/forgejo-db/`, using the existing CCR-2026-0012 backup identity. @@ -31,3 +35,42 @@ Evidence: All 204 repository tests passed after the browser fix and generalized primary restore helper. The live primary recovery additionally verified that helper against the new Forgejo Barman destination. + +The first two isolated application attempts stopped in database import; both +removed their temporary resources. The first did not record a precise failure +cause. Increasing the timeout and importing the 97 MB INSERT-based SQL dump +in one transaction did not resolve the second attempt's psql connection exit +(code 2). Inspection of the pinned PostgreSQL image confirmed its initialization +server listens only on a Unix socket and then shuts down. The old socket-based +readiness probe could accept that temporary server. Changed readiness to the +final TCP listener, then retained trusted local-socket import inside the +disposable container. The helper now records fixed failure categories without +printing SQL, application rows or captured diagnostics. + +The corrected full application recovery PASSED from the actual Nextcloud +download. Forgejo started healthy with 142 repository records, six users and +2,040 package blob records. Public repositories coulomb/key-cape and +coulomb/adaptive-pricing both cloned over the isolated HTTP endpoint and passed +`git fsck --full`. Every one of the 2,040 stored package blobs matched its +database SHA-256 digest. Disposable containers, network and extracted staging +were removed. This verifies startup, sampled Git delivery and all blob bytes; +it does not claim every package installation workflow, runner recovery, or +production failover. All 207 repository tests passed. + +Evidence: `docs/evidence/RPF-WP-0029-secondary-transfer-2026-09-06.json` and +`docs/evidence/RPF-WP-0029-secondary-restore-2026-09-06.json`. + +WP-0029-T02 now retains only the old Bernd-owned share invalidation/custody +receipt. Its replacement full-application recovery requirement is satisfied. +The exposed recovery-key history remains; upload-account replacement does not +clear it. Primary full-archive delivery remains WP-0038-T04. + +The exact rejected September 4 drill copy was conditionally deleted by its +size and strong ETag after replacement recovery (HTTP 204). Warden exited 0. +The temporary fetched plaintext ZIP was removed; good encrypted backups were +preserved. Immediate Nextcloud quota metadata still reported 5,234,716,518 +bytes free, less than this 5,353,024,293-byte artifact. Do not infer freed quota +from DELETE or promise another full upload will fit. Recheck provider quota and +agree archive retention/capacity within the existing owner contract. No retained +good backup or personal-account share was deleted. +Evidence: `docs/evidence/RPF-WP-0029-rejected-fixture-cleanup-2026-09-06.json`. diff --git a/workplans/README.md b/workplans/README.md index 98b1603..3199787 100644 --- a/workplans/README.md +++ b/workplans/README.md @@ -8,7 +8,7 @@ plans is not a count of missing implementations or independent incidents. | Workplan | Purpose and next gate | S3 boundary | | --- | --- | --- | | [RPF-WP-0027](RPF-WP-0027-keycape-live-secret-exposure-recovery.md) | Incident custody and final evidence; accept NetKingdom's residual disposition and publish exact custody handoff | The bundle was already rotated. Provider/MFA reconciliation belongs to NetKingdom. | -| [RPF-WP-0029](RPF-WP-0029-backup-credential-default-removal.md) | Backup account cutover complete; old share invalidation and full offsite application restore remain | S3 retains custody acceptance; S1 and forge own their backup execution. | +| [RPF-WP-0029](RPF-WP-0029-backup-credential-default-removal.md) | Backup cutover and full offsite application recovery complete; old share invalidation receipt remains | S3 retains custody acceptance; S1 and forge own their backup execution. | | [RPF-WP-0025](RPF-WP-0025-openbao-operator-only-access.md) | Private OpenBao access; fresh attended callback/login then guarded retraction | Coordinate package, issuer, tunnel and DNS owners; keep the window separate. | | [RPF-WP-0015](RPF-WP-0015-audit-core-custody-and-recovery-coordination.md) | Two prepared recovery exercises; registered load driver exists; fresh sender/window/abort approvals and custody readiness remain | S3 contributes lease/ESO and snapshot/unseal proof; S1/S2 and audit-core execute their parts. | | [RPF-WP-0035](RPF-WP-0035-credential-lane-implementation.md) | Two remaining lanes: secrets-engine JWT and Fluid operator KV | Signing T04 is complete; JWT and Fluid retain separate issuer/consumer gates. | diff --git a/workplans/RPF-WP-0029-backup-credential-default-removal.md b/workplans/RPF-WP-0029-backup-credential-default-removal.md index 822d92c..a124f7b 100644 --- a/workplans/RPF-WP-0029-backup-credential-default-removal.md +++ b/workplans/RPF-WP-0029-backup-credential-default-removal.md @@ -7,7 +7,7 @@ repo: railiance-platform status: blocked owner: codex created: "2026-09-05" -updated: "2026-09-05" +updated: "2026-09-06" state_hub_workstream_id: "bb326ebb-a313-549e-b35f-1bf17e1c58fd" --- @@ -45,11 +45,13 @@ Provider-side invalidation and replacement custody need the attended provider ow Prepared owner execution procedure: `docs/backup-credential-recovery.md`. The dedicated Backup account cutover is complete under T03. Awaiting owner -authority for invalidating the old Bernd-owned share and a real offsite restore. Activity-core is +authority for invalidating the old Bernd-owned share. Full offsite application +recovery passed on September 6 (evidence below). Activity-core is also a consumer of this upload lane. Preserve AGE_PRIVATE_KEY and historical exposure evidence; upload-token rotation cannot clear recovery-key taint. -T03 proves encrypted fixture transport and decryption; full application restore -and historical predecessor invalidation remain open. +T03 proves encrypted fixture transport and decryption; September 6 evidence +also proves full application recovery. Historical predecessor invalidation +remains open. ## Portfolio review — 2026-09-05 @@ -128,3 +130,30 @@ Barman destination and the Forgejo full-archive uploader still targets Nextcloud Do not conflate this coverage gap with the old-share incident or silently move archives into a database-owned prefix. WP-0029's secondary acceptance gates remain explicit. Source/platform assurance records now name the correct primary. + + +## Follow-up — 2026-09-06 + +Repaired the missing WSL browser-launcher path while retaining Warden +containment. Fresh attended login reached the owner command, the full verified +5.35 GB archive uploaded to Backup (201), downloaded (200), matched ciphertext +and decrypted archive hashes, and Warden exited 0 after session cleanup. +Evidence: `docs/evidence/RPF-WP-0029-secondary-transfer-2026-09-06.json`. +Application recovery and predecessor invalidation remain separate gates. + +The earlier primary gap observation is superseded for forgejo-db: native +Scaleway base backup/WAL and isolated database recovery now pass under +RPF-WP-0038-T02/T03. Forgejo's primary full-archive delivery remains T04. +The Backup account's 10 GiB quota holds only about two current full archives; +growth and other consumers require a bounded retention policy, not automatic +deletion of retained backups. + +Replacement recovery PASSED: isolated Forgejo healthy, 142 repositories, six +users, two public Git clones plus fsck, and all 2,040 package blob digests +verified. Disposable resources removed. Evidence: +`docs/evidence/RPF-WP-0029-secondary-restore-2026-09-06.json`. +T02 remains `wait` solely for the Bernd-owned predecessor invalidation/custody +receipt; do not repeat the completed replacement restore as an open gate. + +Rejected drill-copy cleanup completed with conditional DELETE 204; attended +session exited 0. Temporary plaintext removed; good encrypted backups retained. diff --git a/workplans/RPF-WP-0038-forgejo-scaleway-primary-coverage.md b/workplans/RPF-WP-0038-forgejo-scaleway-primary-coverage.md index a96c83f..b72ab81 100644 --- a/workplans/RPF-WP-0038-forgejo-scaleway-primary-coverage.md +++ b/workplans/RPF-WP-0038-forgejo-scaleway-primary-coverage.md @@ -83,3 +83,9 @@ The existing 5.35 GB encrypted verified archive is available in private staging. Keep Nextcloud's 10 GiB secondary budget separate. The current helper still uploads only to Nextcloud until this task's full delivery is implemented and verified; native database backup alone cannot close the workplan. + +Secondary capacity observation (not a native-primary acceptance gate): after +the successful full recovery and rejected-fixture DELETE, immediate Nextcloud +metadata still reported 5,234,716,518 free bytes, below one current archive's +5,353,024,293 bytes. Recheck quota and settle bounded archive retention/capacity +with the owning package; do not automatically delete retained good backups.