Record verified Forgejo Scaleway backup and recovery
Assistant: codex Assistant-Model: gpt-6-astra Assistant-Session: 01a06ecb-456a-71c2-b41e-0755d336e883
This commit is contained in:
parent
be45c4e3fd
commit
4a9d6e6c7f
8 changed files with 114 additions and 11 deletions
9
SCOPE.md
9
SCOPE.md
|
|
@ -56,7 +56,7 @@ replacement entry points; source files are not deleted merely to tidy scope.
|
||||||
|
|
||||||
## Work that closes the intent gaps
|
## Work that closes the intent gaps
|
||||||
|
|
||||||
The [current work index](workplans/README.md) groups six open workplans.
|
The [current work index](workplans/README.md) groups seven open workplans.
|
||||||
The [full assessment](history/2026-09-05-platform-intent-workplan-assessment.md)
|
The [full assessment](history/2026-09-05-platform-intent-workplan-assessment.md)
|
||||||
records every reviewed plan and the consolidation mapping.
|
records every reviewed plan and the consolidation mapping.
|
||||||
|
|
||||||
|
|
@ -137,12 +137,15 @@ remain RPF-WP-0029-T02. See the
|
||||||
|
|
||||||
Scaleway Standard Multi-AZ (`nl-ams`) is the primary backup destination under
|
Scaleway Standard Multi-AZ (`nl-ams`) is the primary backup destination under
|
||||||
RESOURCE-WP-0002. Nextcloud is the independent secondary-copy lane. The live
|
RESOURCE-WP-0002. Nextcloud is the independent secondary-copy lane. The live
|
||||||
Scaleway paths cover apps-pg, platform-pg and platform-pg-2. An isolated apps-pg
|
Scaleway paths cover apps-pg, platform-pg, platform-pg-2 and forgejo-db. An isolated apps-pg
|
||||||
restore from that primary succeeded in 42.64 seconds; production remained ready
|
restore from that primary succeeded in 42.64 seconds; production remained ready
|
||||||
and scratch resources were removed. See
|
and scratch resources were removed. See
|
||||||
[primary recovery evidence](docs/evidence/scaleway-primary-restore-2026-09-06.json).
|
[primary recovery evidence](docs/evidence/scaleway-primary-restore-2026-09-06.json).
|
||||||
|
|
||||||
Coverage is per asset: live forgejo-db, net-kingdom-pg and state-hub-db have no
|
Forgejo native primary backup and isolated recovery now pass (45.08 seconds;
|
||||||
|
142 repository records, six users, 2,040 package blob records). See
|
||||||
|
[Forgejo primary evidence](docs/evidence/forgejo-scaleway-restore-2026-09-06.json).
|
||||||
|
Coverage is per asset: net-kingdom-pg and state-hub-db still have no observed
|
||||||
native Barman destination. The Forgejo full-archive helper still targets
|
native Barman destination. The Forgejo full-archive helper still targets
|
||||||
Nextcloud; no Forgejo blob/archive destination on Scaleway was evidenced.
|
Nextcloud; no Forgejo blob/archive destination on Scaleway was evidenced.
|
||||||
The account cutover and archive-integrity fix do not establish that coverage.
|
The account cutover and archive-integrity fix do not establish that coverage.
|
||||||
|
|
|
||||||
|
|
@ -127,3 +127,21 @@ prove that every restored package can be installed or every image pulled.
|
||||||
Package-consumer acceptance and broader disaster-recovery guarantees remain
|
Package-consumer acceptance and broader disaster-recovery guarantees remain
|
||||||
with their owning assurance tasks. Full backups share the account's 10 GiB
|
with their owning assurance tasks. Full backups share the account's 10 GiB
|
||||||
quota; a newly measured archive size must inform the separate retention decision.
|
quota; a newly measured archive size must inform the separate retention decision.
|
||||||
|
|
||||||
|
## WSL attended-login browser repair
|
||||||
|
|
||||||
|
If this host has no `xdg-open`, `x-www-browser` or `www-browser`, the installed
|
||||||
|
Bao browser launcher cannot open its login page. Warden contains that output,
|
||||||
|
so this presents as a failed callback before the owner command starts. Use:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 scripts/openbao-attended-exec.py -- /absolute/path/to/reviewed-silent-command
|
||||||
|
```
|
||||||
|
|
||||||
|
The wrapper keeps Warden's `openbao-platform-admin-login --exec` envelope and
|
||||||
|
adds a repository-scoped launcher only when all native providers are absent.
|
||||||
|
The launcher allows only the Keycloak HTTPS host and exact local OIDC callback;
|
||||||
|
it sends the URL to Windows PowerShell through stdin without recording it.
|
||||||
|
No token helper or captured Warden output may be inspected. A receipt written
|
||||||
|
by the owner command proves handoff; session cleanup is assessed separately
|
||||||
|
from the final Warden exit status.
|
||||||
|
|
|
||||||
|
|
@ -9,7 +9,7 @@ Backup, 10 GiB quota. Provider selection does not establish asset coverage.
|
||||||
| apps-pg | Scaleway Barman base backups + WAL, `platform-pg/apps-pg/` | Fresh isolated physical restore passed in 42.64 seconds; Nextcloud logical copy is separate |
|
| apps-pg | Scaleway Barman base backups + WAL, `platform-pg/apps-pg/` | Fresh isolated physical restore passed in 42.64 seconds; Nextcloud logical copy is separate |
|
||||||
| platform-pg | Scaleway Barman base backups + WAL, `platform-pg/` | Earlier package restore evidence; independent logical Nextcloud copy |
|
| platform-pg | Scaleway Barman base backups + WAL, `platform-pg/` | Earlier package restore evidence; independent logical Nextcloud copy |
|
||||||
| platform-pg-2 | Scaleway Barman base backups + WAL, `platform-pg/platform-pg-2/` | Earlier package restore evidence; independent logical Nextcloud copy |
|
| platform-pg-2 | Scaleway Barman base backups + WAL, `platform-pg/platform-pg-2/` | Earlier package restore evidence; independent logical Nextcloud copy |
|
||||||
| forgejo-db | No native Barman destination observed | Logical SQL/full archive helper targets Nextcloud; primary coverage needs implementation |
|
| forgejo-db | Scaleway Barman base backups + WAL, `platform-pg/forgejo-db/`; daily 02:35 UTC, 30-day retention | Fresh isolated physical restore passed in 45.08 seconds; full archive coverage remains separate |
|
||||||
| Forgejo repositories/packages/blobs | No reviewed Scaleway archive destination found | Corrected full-archive capture; 5.35 GB verified encrypted artifact staged; secondary download/application restore still pending |
|
| Forgejo repositories/packages/blobs | No reviewed Scaleway archive destination found | Corrected full-archive capture; 5.35 GB verified encrypted artifact staged; secondary download/application restore still pending |
|
||||||
| net-kingdom-pg / state-hub-db | No native Barman destination observed | Do not infer protection from the shared cells' healthy backup status |
|
| net-kingdom-pg / state-hub-db | No native Barman destination observed | Do not infer protection from the shared cells' healthy backup status |
|
||||||
| OpenBao / S1 host configuration | Not evaluated by this database restore | Their own encrypted snapshot/host backup and recovery contracts still apply |
|
| OpenBao / S1 host configuration | Not evaluated by this database restore | Their own encrypted snapshot/host backup and recovery contracts still apply |
|
||||||
|
|
@ -37,11 +37,15 @@ or assume a top-level `forgejo/` prefix is permitted. Before extending coverage:
|
||||||
3. Use a streaming multipart S3 uploader for growing archives, with abort/cleanup
|
3. Use a streaming multipart S3 uploader for growing archives, with abort/cleanup
|
||||||
and immutable object naming. Verify completion and content, then recover by
|
and immutable object naming. Verify completion and content, then recover by
|
||||||
GET from Scaleway into the isolated Forgejo procedure.
|
GET from Scaleway into the isolated Forgejo procedure.
|
||||||
4. Set native forgejo-db Barman coverage through its owning package/source,
|
4. Native forgejo-db Barman coverage and isolated recovery are complete in
|
||||||
with a separate tested recovery and no production in-place restore.
|
RPF-WP-0038-T02/T03. See `docs/evidence/forgejo-scaleway-restore-2026-09-06.json`.
|
||||||
5. Record provider-native retention and primary failure reporting separately
|
5. Record provider-native retention and primary failure reporting separately
|
||||||
from the 10 GiB secondary budget. No retained backup deletion is implicit.
|
from the 10 GiB secondary budget. No retained backup deletion is implicit.
|
||||||
|
|
||||||
WP-0029 remains the secondary credential incident: old Bernd-share invalidation
|
WP-0029 remains the secondary credential incident: old Bernd-share invalidation
|
||||||
and replacement recovery. The full primary coverage gap belongs to S3 assurance
|
and replacement recovery. The full primary coverage gap belongs to S3 assurance
|
||||||
(RPF-WP-0036-T03), with forge requirements and package/storage-owner inputs.
|
(RPF-WP-0036-T03), with forge requirements and package/storage-owner inputs.
|
||||||
|
|
||||||
|
RPF-WP-0038-T04 tracks the remaining primary full-archive delivery and application
|
||||||
|
recovery contract. Native database recovery verified 142 repository records,
|
||||||
|
six users and 2,040 package blob records; it did not recover the blob files.
|
||||||
|
|
|
||||||
13
docs/evidence/forgejo-scaleway-activation-2026-09-06.json
Normal file
13
docs/evidence/forgejo-scaleway-activation-2026-09-06.json
Normal file
|
|
@ -0,0 +1,13 @@
|
||||||
|
{
|
||||||
|
"schema": "platform.forgejo-primary-backup.v1",
|
||||||
|
"status": "verified",
|
||||||
|
"backup_contract_applied": true,
|
||||||
|
"started_at": "2026-09-05T22:50:09.663787+00:00",
|
||||||
|
"backup_name": "forgejo-db-daily-20260905225013",
|
||||||
|
"backup_id": "20260905T225115",
|
||||||
|
"backup_phase": "completed",
|
||||||
|
"production_ready": true,
|
||||||
|
"continuous_archiving": true,
|
||||||
|
"destination": "s3://railiance-platform-pg-backup/platform-pg/forgejo-db/",
|
||||||
|
"retention": "30d"
|
||||||
|
}
|
||||||
22
docs/evidence/forgejo-scaleway-restore-2026-09-06.json
Normal file
22
docs/evidence/forgejo-scaleway-restore-2026-09-06.json
Normal file
|
|
@ -0,0 +1,22 @@
|
||||||
|
{
|
||||||
|
"schema": "platform.scaleway-primary-restore.v1",
|
||||||
|
"status": "verified",
|
||||||
|
"namespace": "forgejo-db-recovery-ad1aff99",
|
||||||
|
"primary_destination": "s3://railiance-platform-pg-backup/platform-pg/forgejo-db/",
|
||||||
|
"last_successful_backup": "2026-09-05T22:51:32Z",
|
||||||
|
"source": "Scaleway Barman base backup and WAL",
|
||||||
|
"started_at": "2026-09-05T22:54:35.917727+00:00",
|
||||||
|
"stage": "database_acceptance",
|
||||||
|
"restore_seconds": 45.08,
|
||||||
|
"databases": [
|
||||||
|
"app",
|
||||||
|
"forgejo",
|
||||||
|
"postgres"
|
||||||
|
],
|
||||||
|
"repository_count": 142,
|
||||||
|
"user_count": 6,
|
||||||
|
"package_blob_count": 2040,
|
||||||
|
"production_ready": true,
|
||||||
|
"cleanup": true,
|
||||||
|
"finished_at": "2026-09-05T22:55:54.893587+00:00"
|
||||||
|
}
|
||||||
33
history/2026-09-06-backup-problem-followup.md
Normal file
33
history/2026-09-06-backup-problem-followup.md
Normal file
|
|
@ -0,0 +1,33 @@
|
||||||
|
# Backup problem follow-up — 2026-09-06
|
||||||
|
|
||||||
|
The attended-login failures were caused by missing Linux browser launchers on
|
||||||
|
this WSL host. The installed Bao browser dependency tries `xdg-open`,
|
||||||
|
`x-www-browser` and `www-browser`; none was installed. The unauthenticated OIDC
|
||||||
|
routing probe returned the expected Keycloak host and localhost callback.
|
||||||
|
Implemented a narrowly validated, repository-scoped Windows browser bridge
|
||||||
|
and retained Warden's contained credential session. The replacement login
|
||||||
|
reached the silent owner command and began the real secondary archive transfer.
|
||||||
|
Transfer and revocation results are recorded separately when complete.
|
||||||
|
|
||||||
|
Forgejo's native PostgreSQL backup is now active on the primary Scaleway bucket
|
||||||
|
at `platform-pg/forgejo-db/`, using the existing CCR-2026-0012 backup identity.
|
||||||
|
No IAM expansion or new credential delivery was needed. A resource-version
|
||||||
|
guarded patch changed only the live backup field; source manifests include
|
||||||
|
HTTPS/DNS egress, daily 02:35 UTC base backup, WAL archiving and 30-day retention.
|
||||||
|
The fresh backup completed with production Ready and continuous archiving.
|
||||||
|
|
||||||
|
An isolated Scaleway recovery became ready in 45.08 seconds and preserved
|
||||||
|
142 repository records, six users and 2,040 package blob records. Production
|
||||||
|
remained Ready and the scratch namespace was removed. These are database
|
||||||
|
records, not recovered repository/package files. WP-0038-T01/T02/T03 are done.
|
||||||
|
T04 retains the separate full-archive primary delivery and application recovery
|
||||||
|
contract; WP-0029 retains the secondary incident and predecessor invalidation.
|
||||||
|
|
||||||
|
Evidence:
|
||||||
|
|
||||||
|
- `docs/evidence/forgejo-scaleway-activation-2026-09-06.json`
|
||||||
|
- `docs/evidence/forgejo-scaleway-restore-2026-09-06.json`
|
||||||
|
|
||||||
|
All 204 repository tests passed after the browser fix and generalized primary
|
||||||
|
restore helper. The live primary recovery additionally verified that helper
|
||||||
|
against the new Forgejo Barman destination.
|
||||||
|
|
@ -1,6 +1,6 @@
|
||||||
# Current platform work
|
# Current platform work
|
||||||
|
|
||||||
Reviewed 2026-09-05. Six open workplans: six blocked on explicit owner/live
|
Reviewed 2026-09-06. Seven open workplans: seven blocked on explicit owner/live
|
||||||
gates; RPF-WP-0036 now has its repository implementation. Completed designs and implementations are
|
gates; RPF-WP-0036 now has its repository implementation. Completed designs and implementations are
|
||||||
under `archived/`; their IDs and UUIDs are preserved. The number of blocked
|
under `archived/`; their IDs and UUIDs are preserved. The number of blocked
|
||||||
plans is not a count of missing implementations or independent incidents.
|
plans is not a count of missing implementations or independent incidents.
|
||||||
|
|
@ -13,6 +13,7 @@ plans is not a count of missing implementations or independent incidents.
|
||||||
| [RPF-WP-0015](RPF-WP-0015-audit-core-custody-and-recovery-coordination.md) | Two prepared recovery exercises; registered load driver exists; fresh sender/window/abort approvals and custody readiness remain | S3 contributes lease/ESO and snapshot/unseal proof; S1/S2 and audit-core execute their parts. |
|
| [RPF-WP-0015](RPF-WP-0015-audit-core-custody-and-recovery-coordination.md) | Two prepared recovery exercises; registered load driver exists; fresh sender/window/abort approvals and custody readiness remain | S3 contributes lease/ESO and snapshot/unseal proof; S1/S2 and audit-core execute their parts. |
|
||||||
| [RPF-WP-0035](RPF-WP-0035-credential-lane-implementation.md) | Two remaining lanes: secrets-engine JWT and Fluid operator KV | Signing T04 is complete; JWT and Fluid retain separate issuer/consumer gates. |
|
| [RPF-WP-0035](RPF-WP-0035-credential-lane-implementation.md) | Two remaining lanes: secrets-engine JWT and Fluid operator KV | Signing T04 is complete; JWT and Fluid retain separate issuer/consumer gates. |
|
||||||
| [RPF-WP-0036](RPF-WP-0036-platform-service-assurance.md) | Implemented local assurance/admission; waits for recurring restore evidence, Q2 reception and owner handoff | Run the assurance commands; live acceptance and external ownership remain gated. |
|
| [RPF-WP-0036](RPF-WP-0036-platform-service-assurance.md) | Implemented local assurance/admission; waits for recurring restore evidence, Q2 reception and owner handoff | Run the assurance commands; live acceptance and external ownership remain gated. |
|
||||||
|
| [RPF-WP-0038](RPF-WP-0038-forgejo-scaleway-primary-coverage.md) | Forgejo native Scaleway backup and isolated database recovery complete; full archive delivery remains | Separate archive prefix, runtime delivery and application recovery contract required. |
|
||||||
|
|
||||||
RPF-WP-0036-T02/T05/T07 are complete; T03/T04/T06 retain the remaining
|
RPF-WP-0036-T02/T05/T07 are complete; T03/T04/T06 retain the remaining
|
||||||
acceptance gates. Treat credential exposure closure as the highest-priority attended
|
acceptance gates. Treat credential exposure closure as the highest-priority attended
|
||||||
|
|
@ -27,7 +28,8 @@ still shows them active. Use source IDs, and follow AGENTS.md for verified sync.
|
||||||
## Latest closure review
|
## Latest closure review
|
||||||
|
|
||||||
[2026-09-05 blocker review](../history/2026-09-05-blocked-workplan-closure-review.md):
|
[2026-09-05 blocker review](../history/2026-09-05-blocked-workplan-closure-review.md):
|
||||||
12 unfinished tasks across six genuine blocked plans. All terminal plans have
|
At that review: 12 unfinished tasks across six genuine blocked plans.
|
||||||
|
The September 6 follow-up adds WP-0038 with one remaining full-archive task. All terminal plans have
|
||||||
only done/cancel tasks. Completed ESO recovery RPF-WP-0037 is archived.
|
only done/cancel tasks. Completed ESO recovery RPF-WP-0037 is archived.
|
||||||
Three retired Hub aliases still appear open; they are a derived-view defect,
|
Three retired Hub aliases still appear open; they are a derived-view defect,
|
||||||
not three more workplans. Use this file before the dated generated brief.
|
not three more workplans. Use this file before the dated generated brief.
|
||||||
|
|
|
||||||
|
|
@ -4,7 +4,7 @@ type: workplan
|
||||||
title: "Close Forgejo primary backup coverage on Scaleway"
|
title: "Close Forgejo primary backup coverage on Scaleway"
|
||||||
domain: financials
|
domain: financials
|
||||||
repo: railiance-platform
|
repo: railiance-platform
|
||||||
status: active
|
status: blocked
|
||||||
owner: codex
|
owner: codex
|
||||||
created: "2026-09-06"
|
created: "2026-09-06"
|
||||||
updated: "2026-09-06"
|
updated: "2026-09-06"
|
||||||
|
|
@ -35,7 +35,7 @@ server directory. Retention follows the existing 30-day primary contract.
|
||||||
|
|
||||||
```task
|
```task
|
||||||
id: RPF-WP-0038-T02
|
id: RPF-WP-0038-T02
|
||||||
status: progress
|
status: done
|
||||||
priority: high
|
priority: high
|
||||||
state_hub_task_id: "b6960c2e-1ad3-5c23-bfb6-9f1f16d0f0c2"
|
state_hub_task_id: "b6960c2e-1ad3-5c23-bfb6-9f1f16d0f0c2"
|
||||||
```
|
```
|
||||||
|
|
@ -45,11 +45,15 @@ base backup. Apply only the reviewed backup field with a resource-version
|
||||||
guard, preserving unrelated live settings. Require a completed fresh Backup,
|
guard, preserving unrelated live settings. Require a completed fresh Backup,
|
||||||
continuous archiving and source readiness; record metadata only.
|
continuous archiving and source readiness; record metadata only.
|
||||||
|
|
||||||
|
Completed: fresh backup `forgejo-db-daily-20260905225013`, backup ID
|
||||||
|
`20260905T225115`, source Ready and continuous archiving healthy. Evidence:
|
||||||
|
`docs/evidence/forgejo-scaleway-activation-2026-09-06.json`.
|
||||||
|
|
||||||
## Prove isolated primary database recovery
|
## Prove isolated primary database recovery
|
||||||
|
|
||||||
```task
|
```task
|
||||||
id: RPF-WP-0038-T03
|
id: RPF-WP-0038-T03
|
||||||
status: todo
|
status: done
|
||||||
priority: high
|
priority: high
|
||||||
state_hub_task_id: "b47b5905-83c8-5da8-9951-e8cb7b7459f9"
|
state_hub_task_id: "b47b5905-83c8-5da8-9951-e8cb7b7459f9"
|
||||||
```
|
```
|
||||||
|
|
@ -59,6 +63,10 @@ Verify expected Forgejo database and nonempty repository/user/package metadata,
|
||||||
source readiness and scratch cleanup. This is database recovery, not combined
|
source readiness and scratch cleanup. This is database recovery, not combined
|
||||||
Forgejo blob/application recovery.
|
Forgejo blob/application recovery.
|
||||||
|
|
||||||
|
Completed: restored in 45.08 seconds, 142 repository records, six users and
|
||||||
|
2,040 package blob records. Production remained Ready; scratch namespace
|
||||||
|
removed. Evidence: `docs/evidence/forgejo-scaleway-restore-2026-09-06.json`.
|
||||||
|
|
||||||
## Establish primary full-archive delivery and application recovery
|
## Establish primary full-archive delivery and application recovery
|
||||||
|
|
||||||
```task
|
```task
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue