Record successful full offsite Forgejo recovery and remaining custody gate
Assistant: codex Assistant-Model: gpt-6-astra Assistant-Session: 01a06ecb-456a-71c2-b41e-0755d336e883
This commit is contained in:
codex
parent
603c2926fa
commit
5567fa979e
11 changed files with 187 additions and 10 deletions
5
SCOPE.md
5
SCOPE.md
|
|
@ -127,9 +127,8 @@ validated restore adapters and Q2 delivery remain open. See
|
|||
[service assurance](docs/service-assurance.md) and RPF-WP-0036.
|
||||
|
||||
The dedicated Nextcloud Backup account is active with a 10 GiB provider quota
|
||||
and a create-only workload share. Encrypted fixture recovery and consumer
|
||||
refresh are verified; full application restore and predecessor invalidation
|
||||
remain RPF-WP-0029-T02. See the
|
||||
and a create-only workload share. Full offsite application recovery and consumer
|
||||
refresh are verified; predecessor invalidation remains RPF-WP-0029-T02. See the
|
||||
[latest blocked-workplan review](history/2026-09-05-blocked-workplan-closure-review.md).
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -10,7 +10,7 @@ Backup, 10 GiB quota. Provider selection does not establish asset coverage.
|
|||
| platform-pg | Scaleway Barman base backups + WAL, `platform-pg/` | Earlier package restore evidence; independent logical Nextcloud copy |
|
||||
| platform-pg-2 | Scaleway Barman base backups + WAL, `platform-pg/platform-pg-2/` | Earlier package restore evidence; independent logical Nextcloud copy |
|
||||
| forgejo-db | Scaleway Barman base backups + WAL, `platform-pg/forgejo-db/`; daily 02:35 UTC, 30-day retention | Fresh isolated physical restore passed in 45.08 seconds; full archive coverage remains separate |
|
||||
| Forgejo repositories/packages/blobs | No reviewed Scaleway archive destination found | Corrected full-archive capture; 5.35 GB verified encrypted artifact staged; secondary download/application restore still pending |
|
||||
| Forgejo repositories/packages/blobs | No reviewed Scaleway archive destination found | Corrected full-archive capture; 5.35 GB encrypted secondary download and isolated Forgejo recovery passed, including all 2,040 package blobs |
|
||||
| net-kingdom-pg / state-hub-db | No native Barman destination observed | Do not infer protection from the shared cells' healthy backup status |
|
||||
| OpenBao / S1 host configuration | Not evaluated by this database restore | Their own encrypted snapshot/host backup and recovery contracts still apply |
|
||||
|
||||
|
|
@ -49,3 +49,9 @@ and replacement recovery. The full primary coverage gap belongs to S3 assurance
|
|||
RPF-WP-0038-T04 tracks the remaining primary full-archive delivery and application
|
||||
recovery contract. Native database recovery verified 142 repository records,
|
||||
six users and 2,040 package blob records; it did not recover the blob files.
|
||||
|
||||
Secondary full recovery evidence is
|
||||
`docs/evidence/RPF-WP-0029-secondary-restore-2026-09-06.json`. Immediate quota
|
||||
after rejected-fixture cleanup still reported 5.23 GB free versus the 5.35 GB
|
||||
current archive. Recheck quota before another upload and establish bounded
|
||||
retention/capacity; the 10 GiB account is not an indefinite archive store.
|
||||
|
|
|
|||
|
|
@ -0,0 +1,17 @@
|
|||
{
|
||||
"schema": "platform.rejected-recovery-fixture-cleanup.v1",
|
||||
"status": "removed",
|
||||
"quota_before": {
|
||||
"quota_bytes": 10737418240,
|
||||
"used_bytes": 5502701722,
|
||||
"available_bytes": 5234716518
|
||||
},
|
||||
"artifact": "wp0029-recovery-forgejo-dump-20260904T001507Z.zip.age",
|
||||
"delete_http_status": 204,
|
||||
"quota_after": {
|
||||
"quota_bytes": 10737418240,
|
||||
"used_bytes": 5502701722,
|
||||
"available_bytes": 5234716518
|
||||
},
|
||||
"warden_exit_code": 0
|
||||
}
|
||||
24
docs/evidence/RPF-WP-0029-secondary-restore-2026-09-06.json
Normal file
24
docs/evidence/RPF-WP-0029-secondary-restore-2026-09-06.json
Normal file
|
|
@ -0,0 +1,24 @@
|
|||
{
|
||||
"schema": "platform.forgejo-isolated-restore.v1",
|
||||
"status": "restored",
|
||||
"images": [
|
||||
"code.forgejo.org/forgejo/forgejo@sha256:e2684fd8707d486329084a695ed91999a4072a798e5409d45c1eb8a2911ff4b9",
|
||||
"postgres@sha256:ff23cdce56cac62ada6f66013e1a50864c0abbe79d132d40b6e05bd80f378a70"
|
||||
],
|
||||
"offsite_artifact": "wp0029-recovery-forgejo-dump-20260905-verified.zip.age",
|
||||
"ciphertext_sha256": "00d2a5b099bc0e65b2109287fca3bd9d6eb1c8a4e43623520a6e9f3379904018",
|
||||
"stage": "package_blob_recovery",
|
||||
"database_import": true,
|
||||
"database_counts": {
|
||||
"repositories": 142,
|
||||
"users": 6,
|
||||
"package_blobs": 2040
|
||||
},
|
||||
"application_health": true,
|
||||
"repositories_verified": [
|
||||
"coulomb/key-cape",
|
||||
"coulomb/adaptive-pricing"
|
||||
],
|
||||
"package_blobs_verified": 2040,
|
||||
"cleanup": true
|
||||
}
|
||||
|
|
@ -0,0 +1,13 @@
|
|||
{
|
||||
"schema": "platform.forgejo-isolated-restore.v1",
|
||||
"status": "failed",
|
||||
"images": [
|
||||
"code.forgejo.org/forgejo/forgejo@sha256:e2684fd8707d486329084a695ed91999a4072a798e5409d45c1eb8a2911ff4b9",
|
||||
"postgres@sha256:ff23cdce56cac62ada6f66013e1a50864c0abbe79d132d40b6e05bd80f378a70"
|
||||
],
|
||||
"offsite_artifact": "wp0029-recovery-forgejo-dump-20260905-verified.zip.age",
|
||||
"ciphertext_sha256": "00d2a5b099bc0e65b2109287fca3bd9d6eb1c8a4e43623520a6e9f3379904018",
|
||||
"stage": "isolated_database",
|
||||
"cleanup": true,
|
||||
"error": "isolated_restore_failed"
|
||||
}
|
||||
|
|
@ -0,0 +1,15 @@
|
|||
{
|
||||
"schema": "platform.forgejo-isolated-restore.v1",
|
||||
"status": "failed",
|
||||
"images": [
|
||||
"code.forgejo.org/forgejo/forgejo@sha256:e2684fd8707d486329084a695ed91999a4072a798e5409d45c1eb8a2911ff4b9",
|
||||
"postgres@sha256:ff23cdce56cac62ada6f66013e1a50864c0abbe79d132d40b6e05bd80f378a70"
|
||||
],
|
||||
"offsite_artifact": "wp0029-recovery-forgejo-dump-20260905-verified.zip.age",
|
||||
"ciphertext_sha256": "00d2a5b099bc0e65b2109287fca3bd9d6eb1c8a4e43623520a6e9f3379904018",
|
||||
"stage": "isolated_database",
|
||||
"failure_kind": "command_nonzero",
|
||||
"command_exit_code": 2,
|
||||
"cleanup": true,
|
||||
"error": "isolated_restore_failed"
|
||||
}
|
||||
25
docs/evidence/RPF-WP-0029-secondary-transfer-2026-09-06.json
Normal file
25
docs/evidence/RPF-WP-0029-secondary-transfer-2026-09-06.json
Normal file
|
|
@ -0,0 +1,25 @@
|
|||
{
|
||||
"schema": "platform.real-offsite-recovery.v1",
|
||||
"status": "offsite_fetched_pending_isolated_restore",
|
||||
"quota_before": {
|
||||
"quota_bytes": 10737418240,
|
||||
"used_bytes": 149677429,
|
||||
"available_bytes": 10587740811
|
||||
},
|
||||
"artifact": "wp0029-recovery-forgejo-dump-20260905-verified.zip.age",
|
||||
"source_backup": "forgejo-dump-20260905-verified.zip.age",
|
||||
"kv_version": 3,
|
||||
"upload_http_status": 201,
|
||||
"download_http_status": 200,
|
||||
"ciphertext_bytes": 5353024293,
|
||||
"ciphertext_sha256": "00d2a5b099bc0e65b2109287fca3bd9d6eb1c8a4e43623520a6e9f3379904018",
|
||||
"decrypted": true,
|
||||
"plaintext_sha256": "fe53d156461387ea487cddc5d6a01f0b5e3debda6fa494c8b87afc8bc62b55b2",
|
||||
"quota_after": {
|
||||
"quota_bytes": 10737418240,
|
||||
"used_bytes": 5502701722,
|
||||
"available_bytes": 5234716518
|
||||
},
|
||||
"warden_exit_code": 0,
|
||||
"attended_session_completed": true
|
||||
}
|
||||
|
|
@ -7,7 +7,11 @@ routing probe returned the expected Keycloak host and localhost callback.
|
|||
Implemented a narrowly validated, repository-scoped Windows browser bridge
|
||||
and retained Warden's contained credential session. The replacement login
|
||||
reached the silent owner command and began the real secondary archive transfer.
|
||||
Transfer and revocation results are recorded separately when complete.
|
||||
The 5,353,024,293-byte encrypted archive uploaded with HTTP 201, downloaded
|
||||
with HTTP 200 and matched the source ciphertext hash. Decryption matched the
|
||||
validated source archive hash. Warden exited 0 after its contained session,
|
||||
including its self-revocation/cleanup path. The isolated application recovery
|
||||
is recorded separately below.
|
||||
|
||||
Forgejo's native PostgreSQL backup is now active on the primary Scaleway bucket
|
||||
at `platform-pg/forgejo-db/`, using the existing CCR-2026-0012 backup identity.
|
||||
|
|
@ -31,3 +35,42 @@ Evidence:
|
|||
All 204 repository tests passed after the browser fix and generalized primary
|
||||
restore helper. The live primary recovery additionally verified that helper
|
||||
against the new Forgejo Barman destination.
|
||||
|
||||
The first two isolated application attempts stopped in database import; both
|
||||
removed their temporary resources. The first did not record a precise failure
|
||||
cause. Increasing the timeout and importing the 97 MB INSERT-based SQL dump
|
||||
in one transaction did not resolve the second attempt's psql connection exit
|
||||
(code 2). Inspection of the pinned PostgreSQL image confirmed its initialization
|
||||
server listens only on a Unix socket and then shuts down. The old socket-based
|
||||
readiness probe could accept that temporary server. Changed readiness to the
|
||||
final TCP listener, then retained trusted local-socket import inside the
|
||||
disposable container. The helper now records fixed failure categories without
|
||||
printing SQL, application rows or captured diagnostics.
|
||||
|
||||
The corrected full application recovery PASSED from the actual Nextcloud
|
||||
download. Forgejo started healthy with 142 repository records, six users and
|
||||
2,040 package blob records. Public repositories coulomb/key-cape and
|
||||
coulomb/adaptive-pricing both cloned over the isolated HTTP endpoint and passed
|
||||
`git fsck --full`. Every one of the 2,040 stored package blobs matched its
|
||||
database SHA-256 digest. Disposable containers, network and extracted staging
|
||||
were removed. This verifies startup, sampled Git delivery and all blob bytes;
|
||||
it does not claim every package installation workflow, runner recovery, or
|
||||
production failover. All 207 repository tests passed.
|
||||
|
||||
Evidence: `docs/evidence/RPF-WP-0029-secondary-transfer-2026-09-06.json` and
|
||||
`docs/evidence/RPF-WP-0029-secondary-restore-2026-09-06.json`.
|
||||
|
||||
WP-0029-T02 now retains only the old Bernd-owned share invalidation/custody
|
||||
receipt. Its replacement full-application recovery requirement is satisfied.
|
||||
The exposed recovery-key history remains; upload-account replacement does not
|
||||
clear it. Primary full-archive delivery remains WP-0038-T04.
|
||||
|
||||
The exact rejected September 4 drill copy was conditionally deleted by its
|
||||
size and strong ETag after replacement recovery (HTTP 204). Warden exited 0.
|
||||
The temporary fetched plaintext ZIP was removed; good encrypted backups were
|
||||
preserved. Immediate Nextcloud quota metadata still reported 5,234,716,518
|
||||
bytes free, less than this 5,353,024,293-byte artifact. Do not infer freed quota
|
||||
from DELETE or promise another full upload will fit. Recheck provider quota and
|
||||
agree archive retention/capacity within the existing owner contract. No retained
|
||||
good backup or personal-account share was deleted.
|
||||
Evidence: `docs/evidence/RPF-WP-0029-rejected-fixture-cleanup-2026-09-06.json`.
|
||||
|
|
|
|||
|
|
@ -8,7 +8,7 @@ plans is not a count of missing implementations or independent incidents.
|
|||
| Workplan | Purpose and next gate | S3 boundary |
|
||||
| --- | --- | --- |
|
||||
| [RPF-WP-0027](RPF-WP-0027-keycape-live-secret-exposure-recovery.md) | Incident custody and final evidence; accept NetKingdom's residual disposition and publish exact custody handoff | The bundle was already rotated. Provider/MFA reconciliation belongs to NetKingdom. |
|
||||
| [RPF-WP-0029](RPF-WP-0029-backup-credential-default-removal.md) | Backup account cutover complete; old share invalidation and full offsite application restore remain | S3 retains custody acceptance; S1 and forge own their backup execution. |
|
||||
| [RPF-WP-0029](RPF-WP-0029-backup-credential-default-removal.md) | Backup cutover and full offsite application recovery complete; old share invalidation receipt remains | S3 retains custody acceptance; S1 and forge own their backup execution. |
|
||||
| [RPF-WP-0025](RPF-WP-0025-openbao-operator-only-access.md) | Private OpenBao access; fresh attended callback/login then guarded retraction | Coordinate package, issuer, tunnel and DNS owners; keep the window separate. |
|
||||
| [RPF-WP-0015](RPF-WP-0015-audit-core-custody-and-recovery-coordination.md) | Two prepared recovery exercises; registered load driver exists; fresh sender/window/abort approvals and custody readiness remain | S3 contributes lease/ESO and snapshot/unseal proof; S1/S2 and audit-core execute their parts. |
|
||||
| [RPF-WP-0035](RPF-WP-0035-credential-lane-implementation.md) | Two remaining lanes: secrets-engine JWT and Fluid operator KV | Signing T04 is complete; JWT and Fluid retain separate issuer/consumer gates. |
|
||||
|
|
|
|||
|
|
@ -7,7 +7,7 @@ repo: railiance-platform
|
|||
status: blocked
|
||||
owner: codex
|
||||
created: "2026-09-05"
|
||||
updated: "2026-09-05"
|
||||
updated: "2026-09-06"
|
||||
state_hub_workstream_id: "bb326ebb-a313-549e-b35f-1bf17e1c58fd"
|
||||
---
|
||||
|
||||
|
|
@ -45,11 +45,13 @@ Provider-side invalidation and replacement custody need the attended provider ow
|
|||
|
||||
Prepared owner execution procedure: `docs/backup-credential-recovery.md`.
|
||||
The dedicated Backup account cutover is complete under T03. Awaiting owner
|
||||
authority for invalidating the old Bernd-owned share and a real offsite restore. Activity-core is
|
||||
authority for invalidating the old Bernd-owned share. Full offsite application
|
||||
recovery passed on September 6 (evidence below). Activity-core is
|
||||
also a consumer of this upload lane. Preserve AGE_PRIVATE_KEY and historical
|
||||
exposure evidence; upload-token rotation cannot clear recovery-key taint.
|
||||
T03 proves encrypted fixture transport and decryption; full application restore
|
||||
and historical predecessor invalidation remain open.
|
||||
T03 proves encrypted fixture transport and decryption; September 6 evidence
|
||||
also proves full application recovery. Historical predecessor invalidation
|
||||
remains open.
|
||||
|
||||
## Portfolio review — 2026-09-05
|
||||
|
||||
|
|
@ -128,3 +130,30 @@ Barman destination and the Forgejo full-archive uploader still targets Nextcloud
|
|||
Do not conflate this coverage gap with the old-share incident or silently move
|
||||
archives into a database-owned prefix. WP-0029's secondary acceptance gates
|
||||
remain explicit. Source/platform assurance records now name the correct primary.
|
||||
|
||||
|
||||
## Follow-up — 2026-09-06
|
||||
|
||||
Repaired the missing WSL browser-launcher path while retaining Warden
|
||||
containment. Fresh attended login reached the owner command, the full verified
|
||||
5.35 GB archive uploaded to Backup (201), downloaded (200), matched ciphertext
|
||||
and decrypted archive hashes, and Warden exited 0 after session cleanup.
|
||||
Evidence: `docs/evidence/RPF-WP-0029-secondary-transfer-2026-09-06.json`.
|
||||
Application recovery and predecessor invalidation remain separate gates.
|
||||
|
||||
The earlier primary gap observation is superseded for forgejo-db: native
|
||||
Scaleway base backup/WAL and isolated database recovery now pass under
|
||||
RPF-WP-0038-T02/T03. Forgejo's primary full-archive delivery remains T04.
|
||||
The Backup account's 10 GiB quota holds only about two current full archives;
|
||||
growth and other consumers require a bounded retention policy, not automatic
|
||||
deletion of retained backups.
|
||||
|
||||
Replacement recovery PASSED: isolated Forgejo healthy, 142 repositories, six
|
||||
users, two public Git clones plus fsck, and all 2,040 package blob digests
|
||||
verified. Disposable resources removed. Evidence:
|
||||
`docs/evidence/RPF-WP-0029-secondary-restore-2026-09-06.json`.
|
||||
T02 remains `wait` solely for the Bernd-owned predecessor invalidation/custody
|
||||
receipt; do not repeat the completed replacement restore as an open gate.
|
||||
|
||||
Rejected drill-copy cleanup completed with conditional DELETE 204; attended
|
||||
session exited 0. Temporary plaintext removed; good encrypted backups retained.
|
||||
|
|
|
|||
|
|
@ -83,3 +83,9 @@ The existing 5.35 GB encrypted verified archive is available in private staging.
|
|||
Keep Nextcloud's 10 GiB secondary budget separate. The current helper still
|
||||
uploads only to Nextcloud until this task's full delivery is implemented and
|
||||
verified; native database backup alone cannot close the workplan.
|
||||
|
||||
Secondary capacity observation (not a native-primary acceptance gate): after
|
||||
the successful full recovery and rejected-fixture DELETE, immediate Nextcloud
|
||||
metadata still reported 5,234,716,518 free bytes, below one current archive's
|
||||
5,353,024,293 bytes. Recheck quota and settle bounded archive retention/capacity
|
||||
with the owning package; do not automatically delete retained good backups.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue