Verify full and essentials recovery and implement bounded retention tooling
Assistant: codex Assistant-Model: gpt-6-astra Assistant-Session: 01a06ecb-456a-71c2-b41e-0755d336e883
This commit is contained in:
codex
parent
11322a5256
commit
a3ca4b708f
16 changed files with 461 additions and 4 deletions
|
|
@ -10,7 +10,7 @@ Backup, 10 GiB quota. Provider selection does not establish asset coverage.
|
|||
| platform-pg | Scaleway Barman base backups + WAL, `platform-pg/` | Earlier package restore evidence; independent logical Nextcloud copy |
|
||||
| platform-pg-2 | Scaleway Barman base backups + WAL, `platform-pg/platform-pg-2/` | Earlier package restore evidence; independent logical Nextcloud copy |
|
||||
| forgejo-db | Scaleway Barman base backups + WAL, `platform-pg/forgejo-db/`; daily 02:35 UTC, 30-day retention | Fresh isolated physical restore passed in 45.08 seconds; full archive coverage remains separate |
|
||||
| Forgejo repositories/packages/blobs | No reviewed Scaleway archive destination found | Corrected full-archive capture; 5.35 GB encrypted secondary download and isolated Forgejo recovery passed, including all 2,040 package blobs |
|
||||
| Forgejo repositories/packages/blobs | Full archive at `platform-pg/application-archives/forgejo/`; retrieved full-application recovery passed | Full secondary copy preserved; new 273 MiB essentials copy also passed isolated reduced recovery. Scheduled tier cutover remains pending. |
|
||||
| net-kingdom-pg / state-hub-db | No native Barman destination observed | Do not infer protection from the shared cells' healthy backup status |
|
||||
| OpenBao / S1 host configuration | Not evaluated by this database restore | Their own encrypted snapshot/host backup and recovery contracts still apply |
|
||||
|
||||
|
|
|
|||
|
|
@ -90,3 +90,57 @@ when overlap/headroom cannot be met. Never delete the last verified recovery poi
|
|||
upload overlap, failed-upload preservation and safe expiration within 10 GiB.
|
||||
6. Switch scheduled secondary delivery only after these gates pass. Preserve the
|
||||
current good full Nextcloud copy until a verified replacement permits retirement.
|
||||
|
||||
## Verified implementation — 2026-09-06
|
||||
|
||||
The actual encrypted essentials artifact is **286,298,520 bytes (273 MiB)**,
|
||||
versus 5,353,024,293 bytes for the tested full archive: about 94.7% smaller.
|
||||
Nine distinct copies at this measured size need about 2.40 GiB, before other
|
||||
services, incoming overlap, growth and provider accounting. Keep the 600 MiB
|
||||
admission ceiling; current size is not a perpetual promise.
|
||||
|
||||
The full archive was multipart-uploaded to the separate existing-policy prefix
|
||||
`platform-pg/application-archives/forgejo/`, retrieved by its returned object
|
||||
version, hash-checked, decrypted and restored successfully. Forgejo startup,
|
||||
two Git clones/fsck and all 2,040 package blob hashes passed. This is a verified
|
||||
primary archive and recovery; recurring full-archive scheduling is still pending.
|
||||
The existing bucket contract is versioned with 30-day current/noncurrent lifecycle;
|
||||
a separate calendar archive-expiration policy has not been activated.
|
||||
|
||||
The smaller candidate was uploaded/downloaded through Backup, decrypted and
|
||||
restored without primary access. Forgejo started with 142 repositories, six users
|
||||
and 2,050 package metadata records; the package service was explicitly disabled.
|
||||
Two Git clones/fsck passed. The newer essentials capture and older full archive
|
||||
are different recovery points; they are not claimed to be an atomic matching pair.
|
||||
Both disposable recovery environments were removed. The public pinned Forgejo
|
||||
and PostgreSQL recovery images were successfully pulled independently of both
|
||||
Scaleway and the production forge. Other five database clusters total about
|
||||
209 MiB of physical data; compressed export budgets remain to be measured.
|
||||
|
||||
Implementation:
|
||||
|
||||
- `scripts/capture_forgejo_archive.py --profile essentials` uses native exclusions,
|
||||
seals the explicit manifest and enforces the size ceiling. Default remains full.
|
||||
- `scripts/forgejo_essentials_profile.py` also removes generated Actions logs while
|
||||
preserving unique attachments/LFS/Actions artifacts and rejects unsafe paths.
|
||||
- `scripts/scaleway_forgejo_archive.py` performs bounded multipart PUT, aborts failed
|
||||
in-flight multipart uploads, then downloads and verifies bytes. Run with
|
||||
`uv run --no-project --with boto3==1.40.18 python ...`; credential values stay in
|
||||
captured memory from the existing governed databases Secret.
|
||||
- `scripts/backup_retention_plan.py` selects seven daily/two weekly points without
|
||||
duplicate uploads; protects the explicitly pinned recovered archive and newest
|
||||
point, ignores unknown/unverified objects and requires 2 GiB upload headroom.
|
||||
- `scripts/execute_nextcloud_retention.py` previews by default. With `--apply`, the
|
||||
separate attended owner executor revalidates protected ciphertext, retained
|
||||
object presence and candidate sizes/ETags before conditional deletion. It uses
|
||||
a per-user local lock; deploy only one authoritative owner executor. It never
|
||||
changes workload credentials or deletes full archives. Unknown drill filenames
|
||||
are intentionally ineligible. No live expiration was executed in this review.
|
||||
|
||||
Before scheduled cutover, bind the durable primary caller and dependencies,
|
||||
produce canonical `forgejo-essentials-YYYYMMDDTHHMMSSZ.zip.age` names and verified
|
||||
inventory, and bind the separate owner retention execution/fresh quota check.
|
||||
The current local `forgejo-backup` cron remains unchanged. The activity-core
|
||||
worker has vendored age/kubectl, which its CNPG resolver adds to PATH; absence
|
||||
from its default PATH did not establish missing tools. Neither that mount nor
|
||||
an attended workstation proof establishes a scheduled primary archive job.
|
||||
|
|
|
|||
|
|
@ -0,0 +1,22 @@
|
|||
{
|
||||
"schema": "platform.essentials-bootstrap-assessment.v1",
|
||||
"pinned_recovery_images_public_pull_verified": true,
|
||||
"image_bytes": [
|
||||
75323786,
|
||||
160190278
|
||||
],
|
||||
"other_database_physical_bytes": {
|
||||
"apps-pg-1": 32477276,
|
||||
"net-kingdom-pg-1": 41300060,
|
||||
"platform-pg-1": 53373778,
|
||||
"platform-pg-2-1": 25556393,
|
||||
"state-hub-db-1": 66697262
|
||||
},
|
||||
"package_type_counts": {
|
||||
"container": 21,
|
||||
"npm": 2,
|
||||
"pypi": 2
|
||||
},
|
||||
"all_packages_assumed_rebuildable": false,
|
||||
"note": "Physical database/image sizes, not compressed offsite exports. Other service backup coverage and independent recovery-key custody retain their own acceptance contracts."
|
||||
}
|
||||
26
docs/evidence/RPF-WP-0038-essentials-capture-2026-09-06.json
Normal file
26
docs/evidence/RPF-WP-0038-essentials-capture-2026-09-06.json
Normal file
|
|
@ -0,0 +1,26 @@
|
|||
{
|
||||
"schema": "platform.forgejo-essentials-capture.v1",
|
||||
"status": "captured_pending_offsite_restore",
|
||||
"archive_bytes": 286228448,
|
||||
"archive_sha256": "940b27f6782ad614fe16ab4f6f31dd431adb60b6dea249662346c573617b3a10",
|
||||
"repository_heads": 142,
|
||||
"zip_crc_passed": true,
|
||||
"budget_bytes": 629145600,
|
||||
"manifest": {
|
||||
"complete_application_backup": false,
|
||||
"created_at": "2026-09-06T06:12:09.581148+00:00",
|
||||
"kept_members": 70433,
|
||||
"omitted_prefixes": [
|
||||
"data/packages/",
|
||||
"data/repo-archive/",
|
||||
"data/indexers/",
|
||||
"data/actions_log/",
|
||||
"log/"
|
||||
],
|
||||
"package_registry_available": false,
|
||||
"profile": "essentials",
|
||||
"recovery_contract": "Git and collaboration recovery; packages require full primary or an independent artifact source.",
|
||||
"schema": "platform.forgejo-backup-profile.v1",
|
||||
"unique_data_policy": "Keep repositories, database, configuration, attachments, LFS and Actions artifacts when present."
|
||||
}
|
||||
}
|
||||
27
docs/evidence/RPF-WP-0038-essentials-restore-2026-09-06.json
Normal file
27
docs/evidence/RPF-WP-0038-essentials-restore-2026-09-06.json
Normal file
|
|
@ -0,0 +1,27 @@
|
|||
{
|
||||
"schema": "platform.forgejo-isolated-restore.v1",
|
||||
"status": "restored_essentials",
|
||||
"images": [
|
||||
"code.forgejo.org/forgejo/forgejo@sha256:e2684fd8707d486329084a695ed91999a4072a798e5409d45c1eb8a2911ff4b9",
|
||||
"postgres@sha256:ff23cdce56cac62ada6f66013e1a50864c0abbe79d132d40b6e05bd80f378a70"
|
||||
],
|
||||
"source_provider": "Nextcloud",
|
||||
"offsite_artifact": "wp0029-recovery-forgejo-essentials-20260906.zip.age",
|
||||
"ciphertext_sha256": "23b5250469a63a3e0097047851cb02036ec4e24928979db7a972a3245f93b60a",
|
||||
"archive_profile": "essentials",
|
||||
"stage": "repository_recovery",
|
||||
"database_import": true,
|
||||
"database_counts": {
|
||||
"repositories": 142,
|
||||
"users": 6,
|
||||
"package_blobs": 2050
|
||||
},
|
||||
"application_health": true,
|
||||
"repositories_verified": [
|
||||
"coulomb/key-cape",
|
||||
"coulomb/adaptive-pricing"
|
||||
],
|
||||
"package_registry_available": false,
|
||||
"primary_storage_accessed": false,
|
||||
"cleanup": true
|
||||
}
|
||||
|
|
@ -0,0 +1,25 @@
|
|||
{
|
||||
"schema": "platform.real-offsite-recovery.v1",
|
||||
"status": "offsite_fetched_pending_isolated_restore",
|
||||
"quota_before": {
|
||||
"quota_bytes": 10737418240,
|
||||
"used_bytes": 5385385643,
|
||||
"available_bytes": 5352032597
|
||||
},
|
||||
"artifact": "wp0029-recovery-forgejo-essentials-20260906.zip.age",
|
||||
"source_backup": "forgejo-essentials-20260906.zip.age",
|
||||
"kv_version": 3,
|
||||
"upload_http_status": 201,
|
||||
"download_http_status": 200,
|
||||
"ciphertext_bytes": 286298520,
|
||||
"ciphertext_sha256": "23b5250469a63a3e0097047851cb02036ec4e24928979db7a972a3245f93b60a",
|
||||
"archive_profile": "essentials",
|
||||
"decrypted": true,
|
||||
"plaintext_sha256": "940b27f6782ad614fe16ab4f6f31dd431adb60b6dea249662346c573617b3a10",
|
||||
"quota_after": {
|
||||
"quota_bytes": 10737418240,
|
||||
"used_bytes": 5385385643,
|
||||
"available_bytes": 5352032597
|
||||
},
|
||||
"warden_exit_code": 0
|
||||
}
|
||||
|
|
@ -0,0 +1,16 @@
|
|||
{
|
||||
"schema": "platform.forgejo-primary-archive.v1",
|
||||
"status": "primary_fetched_pending_application_restore",
|
||||
"stage": "transfer_verified",
|
||||
"destination": "s3://railiance-platform-pg-backup/platform-pg/application-archives/forgejo/2026/09/06/060650-6cf2063187f52d6a2b65f79b.zip.age",
|
||||
"ciphertext_sha256": "00d2a5b099bc0e65b2109287fca3bd9d6eb1c8a4e43623520a6e9f3379904018",
|
||||
"ciphertext_bytes": 5353024293,
|
||||
"uploaded_bytes": 5353024293,
|
||||
"multipart_completed": true,
|
||||
"version_pinned": true,
|
||||
"downloaded_bytes": 5353024293,
|
||||
"download_hash_matches": true,
|
||||
"decrypted": true,
|
||||
"plaintext_sha256": "fe53d156461387ea487cddc5d6a01f0b5e3debda6fa494c8b87afc8bc62b55b2",
|
||||
"warden_exit_code": 0
|
||||
}
|
||||
|
|
@ -0,0 +1,26 @@
|
|||
{
|
||||
"schema": "platform.forgejo-isolated-restore.v1",
|
||||
"status": "restored",
|
||||
"images": [
|
||||
"code.forgejo.org/forgejo/forgejo@sha256:e2684fd8707d486329084a695ed91999a4072a798e5409d45c1eb8a2911ff4b9",
|
||||
"postgres@sha256:ff23cdce56cac62ada6f66013e1a50864c0abbe79d132d40b6e05bd80f378a70"
|
||||
],
|
||||
"source_provider": "Scaleway",
|
||||
"offsite_artifact": "s3://railiance-platform-pg-backup/platform-pg/application-archives/forgejo/2026/09/06/060650-6cf2063187f52d6a2b65f79b.zip.age",
|
||||
"ciphertext_sha256": "00d2a5b099bc0e65b2109287fca3bd9d6eb1c8a4e43623520a6e9f3379904018",
|
||||
"archive_profile": "full",
|
||||
"stage": "package_blob_recovery",
|
||||
"database_import": true,
|
||||
"database_counts": {
|
||||
"repositories": 142,
|
||||
"users": 6,
|
||||
"package_blobs": 2040
|
||||
},
|
||||
"application_health": true,
|
||||
"repositories_verified": [
|
||||
"coulomb/key-cape",
|
||||
"coulomb/adaptive-pricing"
|
||||
],
|
||||
"package_blobs_verified": 2040,
|
||||
"cleanup": true
|
||||
}
|
||||
|
|
@ -0,0 +1,13 @@
|
|||
{
|
||||
"schema": "platform.forgejo-primary-archive.v1",
|
||||
"status": "primary_fetched_pending_application_restore",
|
||||
"stage": "transfer_verified",
|
||||
"destination": "s3://railiance-platform-pg-backup/platform-pg/application-archives/forgejo/2026/09/06/060650-6cf2063187f52d6a2b65f79b.zip.age",
|
||||
"ciphertext_sha256": "00d2a5b099bc0e65b2109287fca3bd9d6eb1c8a4e43623520a6e9f3379904018",
|
||||
"ciphertext_bytes": 5353024293,
|
||||
"uploaded_bytes": 5353024293,
|
||||
"multipart_completed": true,
|
||||
"version_pinned": true,
|
||||
"downloaded_bytes": 5353024293,
|
||||
"download_hash_matches": true
|
||||
}
|
||||
Loading…
Add table
Add a link
Reference in a new issue