Verify full and essentials recovery and implement bounded retention tooling
Assistant: codex Assistant-Model: gpt-6-astra Assistant-Session: 01a06ecb-456a-71c2-b41e-0755d336e883
This commit is contained in:
parent
11322a5256
commit
a3ca4b708f
16 changed files with 461 additions and 4 deletions
6
SCOPE.md
6
SCOPE.md
|
|
@ -146,7 +146,11 @@ Forgejo native primary backup and isolated recovery now pass (45.08 seconds;
|
|||
[Forgejo primary evidence](docs/evidence/forgejo-scaleway-restore-2026-09-06.json).
|
||||
Coverage is per asset: net-kingdom-pg and state-hub-db still have no observed
|
||||
native Barman destination. The Forgejo full-archive helper still targets
|
||||
Nextcloud; no Forgejo blob/archive destination on Scaleway was evidenced.
|
||||
Nextcloud. A full archive at `platform-pg/application-archives/forgejo/` now
|
||||
passed Scaleway retrieval and isolated application recovery. A 273 MiB
|
||||
Nextcloud essentials candidate also passed reduced recovery without primary
|
||||
access. Recurring tiered scheduling, quota and owner retention activation
|
||||
remain RPF-WP-0038-T04.
|
||||
The account cutover and archive-integrity fix do not establish that coverage.
|
||||
WP-0029 retains secondary credential invalidation/recovery; its completion
|
||||
must not be presented as full primary-backup assurance.
|
||||
|
|
|
|||
|
|
@ -10,7 +10,7 @@ Backup, 10 GiB quota. Provider selection does not establish asset coverage.
|
|||
| platform-pg | Scaleway Barman base backups + WAL, `platform-pg/` | Earlier package restore evidence; independent logical Nextcloud copy |
|
||||
| platform-pg-2 | Scaleway Barman base backups + WAL, `platform-pg/platform-pg-2/` | Earlier package restore evidence; independent logical Nextcloud copy |
|
||||
| forgejo-db | Scaleway Barman base backups + WAL, `platform-pg/forgejo-db/`; daily 02:35 UTC, 30-day retention | Fresh isolated physical restore passed in 45.08 seconds; full archive coverage remains separate |
|
||||
| Forgejo repositories/packages/blobs | No reviewed Scaleway archive destination found | Corrected full-archive capture; 5.35 GB encrypted secondary download and isolated Forgejo recovery passed, including all 2,040 package blobs |
|
||||
| Forgejo repositories/packages/blobs | Full archive at `platform-pg/application-archives/forgejo/`; retrieved full-application recovery passed | Full secondary copy preserved; new 273 MiB essentials copy also passed isolated reduced recovery. Scheduled tier cutover remains pending. |
|
||||
| net-kingdom-pg / state-hub-db | No native Barman destination observed | Do not infer protection from the shared cells' healthy backup status |
|
||||
| OpenBao / S1 host configuration | Not evaluated by this database restore | Their own encrypted snapshot/host backup and recovery contracts still apply |
|
||||
|
||||
|
|
|
|||
|
|
@ -90,3 +90,57 @@ when overlap/headroom cannot be met. Never delete the last verified recovery poi
|
|||
upload overlap, failed-upload preservation and safe expiration within 10 GiB.
|
||||
6. Switch scheduled secondary delivery only after these gates pass. Preserve the
|
||||
current good full Nextcloud copy until a verified replacement permits retirement.
|
||||
|
||||
## Verified implementation — 2026-09-06
|
||||
|
||||
The actual encrypted essentials artifact is **286,298,520 bytes (273 MiB)**,
|
||||
versus 5,353,024,293 bytes for the tested full archive: about 94.7% smaller.
|
||||
Nine distinct copies at this measured size need about 2.40 GiB, before other
|
||||
services, incoming overlap, growth and provider accounting. Keep the 600 MiB
|
||||
admission ceiling; current size is not a perpetual promise.
|
||||
|
||||
The full archive was multipart-uploaded to the separate existing-policy prefix
|
||||
`platform-pg/application-archives/forgejo/`, retrieved by its returned object
|
||||
version, hash-checked, decrypted and restored successfully. Forgejo startup,
|
||||
two Git clones/fsck and all 2,040 package blob hashes passed. This is a verified
|
||||
primary archive and recovery; recurring full-archive scheduling is still pending.
|
||||
The existing bucket contract is versioned with 30-day current/noncurrent lifecycle;
|
||||
a separate calendar archive-expiration policy has not been activated.
|
||||
|
||||
The smaller candidate was uploaded/downloaded through Backup, decrypted and
|
||||
restored without primary access. Forgejo started with 142 repositories, six users
|
||||
and 2,050 package metadata records; the package service was explicitly disabled.
|
||||
Two Git clones/fsck passed. The newer essentials capture and older full archive
|
||||
are different recovery points; they are not claimed to be an atomic matching pair.
|
||||
Both disposable recovery environments were removed. The public pinned Forgejo
|
||||
and PostgreSQL recovery images were successfully pulled independently of both
|
||||
Scaleway and the production forge. Other five database clusters total about
|
||||
209 MiB of physical data; compressed export budgets remain to be measured.
|
||||
|
||||
Implementation:
|
||||
|
||||
- `scripts/capture_forgejo_archive.py --profile essentials` uses native exclusions,
|
||||
seals the explicit manifest and enforces the size ceiling. Default remains full.
|
||||
- `scripts/forgejo_essentials_profile.py` also removes generated Actions logs while
|
||||
preserving unique attachments/LFS/Actions artifacts and rejects unsafe paths.
|
||||
- `scripts/scaleway_forgejo_archive.py` performs bounded multipart PUT, aborts failed
|
||||
in-flight multipart uploads, then downloads and verifies bytes. Run with
|
||||
`uv run --no-project --with boto3==1.40.18 python ...`; credential values stay in
|
||||
captured memory from the existing governed databases Secret.
|
||||
- `scripts/backup_retention_plan.py` selects seven daily/two weekly points without
|
||||
duplicate uploads; protects the explicitly pinned recovered archive and newest
|
||||
point, ignores unknown/unverified objects and requires 2 GiB upload headroom.
|
||||
- `scripts/execute_nextcloud_retention.py` previews by default. With `--apply`, the
|
||||
separate attended owner executor revalidates protected ciphertext, retained
|
||||
object presence and candidate sizes/ETags before conditional deletion. It uses
|
||||
a per-user local lock; deploy only one authoritative owner executor. It never
|
||||
changes workload credentials or deletes full archives. Unknown drill filenames
|
||||
are intentionally ineligible. No live expiration was executed in this review.
|
||||
|
||||
Before scheduled cutover, bind the durable primary caller and dependencies,
|
||||
produce canonical `forgejo-essentials-YYYYMMDDTHHMMSSZ.zip.age` names and verified
|
||||
inventory, and bind the separate owner retention execution/fresh quota check.
|
||||
The current local `forgejo-backup` cron remains unchanged. The activity-core
|
||||
worker has vendored age/kubectl, which its CNPG resolver adds to PATH; absence
|
||||
from its default PATH did not establish missing tools. Neither that mount nor
|
||||
an attended workstation proof establishes a scheduled primary archive job.
|
||||
|
|
|
|||
|
|
@ -0,0 +1,22 @@
|
|||
{
|
||||
"schema": "platform.essentials-bootstrap-assessment.v1",
|
||||
"pinned_recovery_images_public_pull_verified": true,
|
||||
"image_bytes": [
|
||||
75323786,
|
||||
160190278
|
||||
],
|
||||
"other_database_physical_bytes": {
|
||||
"apps-pg-1": 32477276,
|
||||
"net-kingdom-pg-1": 41300060,
|
||||
"platform-pg-1": 53373778,
|
||||
"platform-pg-2-1": 25556393,
|
||||
"state-hub-db-1": 66697262
|
||||
},
|
||||
"package_type_counts": {
|
||||
"container": 21,
|
||||
"npm": 2,
|
||||
"pypi": 2
|
||||
},
|
||||
"all_packages_assumed_rebuildable": false,
|
||||
"note": "Physical database/image sizes, not compressed offsite exports. Other service backup coverage and independent recovery-key custody retain their own acceptance contracts."
|
||||
}
|
||||
26
docs/evidence/RPF-WP-0038-essentials-capture-2026-09-06.json
Normal file
26
docs/evidence/RPF-WP-0038-essentials-capture-2026-09-06.json
Normal file
|
|
@ -0,0 +1,26 @@
|
|||
{
|
||||
"schema": "platform.forgejo-essentials-capture.v1",
|
||||
"status": "captured_pending_offsite_restore",
|
||||
"archive_bytes": 286228448,
|
||||
"archive_sha256": "940b27f6782ad614fe16ab4f6f31dd431adb60b6dea249662346c573617b3a10",
|
||||
"repository_heads": 142,
|
||||
"zip_crc_passed": true,
|
||||
"budget_bytes": 629145600,
|
||||
"manifest": {
|
||||
"complete_application_backup": false,
|
||||
"created_at": "2026-09-06T06:12:09.581148+00:00",
|
||||
"kept_members": 70433,
|
||||
"omitted_prefixes": [
|
||||
"data/packages/",
|
||||
"data/repo-archive/",
|
||||
"data/indexers/",
|
||||
"data/actions_log/",
|
||||
"log/"
|
||||
],
|
||||
"package_registry_available": false,
|
||||
"profile": "essentials",
|
||||
"recovery_contract": "Git and collaboration recovery; packages require full primary or an independent artifact source.",
|
||||
"schema": "platform.forgejo-backup-profile.v1",
|
||||
"unique_data_policy": "Keep repositories, database, configuration, attachments, LFS and Actions artifacts when present."
|
||||
}
|
||||
}
|
||||
27
docs/evidence/RPF-WP-0038-essentials-restore-2026-09-06.json
Normal file
27
docs/evidence/RPF-WP-0038-essentials-restore-2026-09-06.json
Normal file
|
|
@ -0,0 +1,27 @@
|
|||
{
|
||||
"schema": "platform.forgejo-isolated-restore.v1",
|
||||
"status": "restored_essentials",
|
||||
"images": [
|
||||
"code.forgejo.org/forgejo/forgejo@sha256:e2684fd8707d486329084a695ed91999a4072a798e5409d45c1eb8a2911ff4b9",
|
||||
"postgres@sha256:ff23cdce56cac62ada6f66013e1a50864c0abbe79d132d40b6e05bd80f378a70"
|
||||
],
|
||||
"source_provider": "Nextcloud",
|
||||
"offsite_artifact": "wp0029-recovery-forgejo-essentials-20260906.zip.age",
|
||||
"ciphertext_sha256": "23b5250469a63a3e0097047851cb02036ec4e24928979db7a972a3245f93b60a",
|
||||
"archive_profile": "essentials",
|
||||
"stage": "repository_recovery",
|
||||
"database_import": true,
|
||||
"database_counts": {
|
||||
"repositories": 142,
|
||||
"users": 6,
|
||||
"package_blobs": 2050
|
||||
},
|
||||
"application_health": true,
|
||||
"repositories_verified": [
|
||||
"coulomb/key-cape",
|
||||
"coulomb/adaptive-pricing"
|
||||
],
|
||||
"package_registry_available": false,
|
||||
"primary_storage_accessed": false,
|
||||
"cleanup": true
|
||||
}
|
||||
|
|
@ -0,0 +1,25 @@
|
|||
{
|
||||
"schema": "platform.real-offsite-recovery.v1",
|
||||
"status": "offsite_fetched_pending_isolated_restore",
|
||||
"quota_before": {
|
||||
"quota_bytes": 10737418240,
|
||||
"used_bytes": 5385385643,
|
||||
"available_bytes": 5352032597
|
||||
},
|
||||
"artifact": "wp0029-recovery-forgejo-essentials-20260906.zip.age",
|
||||
"source_backup": "forgejo-essentials-20260906.zip.age",
|
||||
"kv_version": 3,
|
||||
"upload_http_status": 201,
|
||||
"download_http_status": 200,
|
||||
"ciphertext_bytes": 286298520,
|
||||
"ciphertext_sha256": "23b5250469a63a3e0097047851cb02036ec4e24928979db7a972a3245f93b60a",
|
||||
"archive_profile": "essentials",
|
||||
"decrypted": true,
|
||||
"plaintext_sha256": "940b27f6782ad614fe16ab4f6f31dd431adb60b6dea249662346c573617b3a10",
|
||||
"quota_after": {
|
||||
"quota_bytes": 10737418240,
|
||||
"used_bytes": 5385385643,
|
||||
"available_bytes": 5352032597
|
||||
},
|
||||
"warden_exit_code": 0
|
||||
}
|
||||
|
|
@ -0,0 +1,16 @@
|
|||
{
|
||||
"schema": "platform.forgejo-primary-archive.v1",
|
||||
"status": "primary_fetched_pending_application_restore",
|
||||
"stage": "transfer_verified",
|
||||
"destination": "s3://railiance-platform-pg-backup/platform-pg/application-archives/forgejo/2026/09/06/060650-6cf2063187f52d6a2b65f79b.zip.age",
|
||||
"ciphertext_sha256": "00d2a5b099bc0e65b2109287fca3bd9d6eb1c8a4e43623520a6e9f3379904018",
|
||||
"ciphertext_bytes": 5353024293,
|
||||
"uploaded_bytes": 5353024293,
|
||||
"multipart_completed": true,
|
||||
"version_pinned": true,
|
||||
"downloaded_bytes": 5353024293,
|
||||
"download_hash_matches": true,
|
||||
"decrypted": true,
|
||||
"plaintext_sha256": "fe53d156461387ea487cddc5d6a01f0b5e3debda6fa494c8b87afc8bc62b55b2",
|
||||
"warden_exit_code": 0
|
||||
}
|
||||
|
|
@ -0,0 +1,26 @@
|
|||
{
|
||||
"schema": "platform.forgejo-isolated-restore.v1",
|
||||
"status": "restored",
|
||||
"images": [
|
||||
"code.forgejo.org/forgejo/forgejo@sha256:e2684fd8707d486329084a695ed91999a4072a798e5409d45c1eb8a2911ff4b9",
|
||||
"postgres@sha256:ff23cdce56cac62ada6f66013e1a50864c0abbe79d132d40b6e05bd80f378a70"
|
||||
],
|
||||
"source_provider": "Scaleway",
|
||||
"offsite_artifact": "s3://railiance-platform-pg-backup/platform-pg/application-archives/forgejo/2026/09/06/060650-6cf2063187f52d6a2b65f79b.zip.age",
|
||||
"ciphertext_sha256": "00d2a5b099bc0e65b2109287fca3bd9d6eb1c8a4e43623520a6e9f3379904018",
|
||||
"archive_profile": "full",
|
||||
"stage": "package_blob_recovery",
|
||||
"database_import": true,
|
||||
"database_counts": {
|
||||
"repositories": 142,
|
||||
"users": 6,
|
||||
"package_blobs": 2040
|
||||
},
|
||||
"application_health": true,
|
||||
"repositories_verified": [
|
||||
"coulomb/key-cape",
|
||||
"coulomb/adaptive-pricing"
|
||||
],
|
||||
"package_blobs_verified": 2040,
|
||||
"cleanup": true
|
||||
}
|
||||
|
|
@ -0,0 +1,13 @@
|
|||
{
|
||||
"schema": "platform.forgejo-primary-archive.v1",
|
||||
"status": "primary_fetched_pending_application_restore",
|
||||
"stage": "transfer_verified",
|
||||
"destination": "s3://railiance-platform-pg-backup/platform-pg/application-archives/forgejo/2026/09/06/060650-6cf2063187f52d6a2b65f79b.zip.age",
|
||||
"ciphertext_sha256": "00d2a5b099bc0e65b2109287fca3bd9d6eb1c8a4e43623520a6e9f3379904018",
|
||||
"ciphertext_bytes": 5353024293,
|
||||
"uploaded_bytes": 5353024293,
|
||||
"multipart_completed": true,
|
||||
"version_pinned": true,
|
||||
"downloaded_bytes": 5353024293,
|
||||
"download_hash_matches": true
|
||||
}
|
||||
46
history/2026-09-06-backup-tiers-implementation.md
Normal file
46
history/2026-09-06-backup-tiers-implementation.md
Normal file
|
|
@ -0,0 +1,46 @@
|
|||
# Backup tiers implementation and recovery evidence
|
||||
|
||||
Implemented and live-verified the two data paths under WP-0038-T04. The full
|
||||
5.35 GB encrypted archive now has a separate Scaleway application-archive copy
|
||||
and passed actual retrieval, decryption and isolated Forgejo recovery, including
|
||||
all 2,040 package blob hashes. Native database backup remains independently active.
|
||||
|
||||
The explicit essentials profile produced a 286,298,520-byte encrypted archive
|
||||
(273 MiB), about 94.7% smaller. It preserves all 142 repository roots, database,
|
||||
configuration and unique-data categories. The manifest declares omitted packages,
|
||||
generated archives/indexes/logs and the reduced recovery contract. Its real
|
||||
Nextcloud round trip and isolated recovery passed with package service disabled,
|
||||
142 repository records, six users, and two successful Git clone/fsck checks.
|
||||
The newer essentials database has 2,050 blob metadata records; these two captures
|
||||
are different recovery points, not a synchronized full/essential pair.
|
||||
|
||||
Public pinned recovery images were fetched without the production registry or
|
||||
Scaleway. Inventory found 21 container, two npm and two Python packages; no blanket
|
||||
rebuildability claim is made. Five other database clusters total about 209 MiB
|
||||
physical storage, a sizing input rather than compressed backup coverage proof.
|
||||
|
||||
Implemented tested retention planning and a separate attended owner executor:
|
||||
protected recovered anchor, known verified names only, incoming headroom,
|
||||
retained-object checks, ciphertext anchor revalidation, conditional ETag deletion
|
||||
and a single-writer local lock. Workload delivery remains create-only. No retained
|
||||
full archive or provider retention setting was changed.
|
||||
|
||||
Initial capture/transfer attempts failed before useful diagnostics; safe stage
|
||||
and exception-type diagnostics were added. Retried operations passed. Their
|
||||
precise initial failure cause was not established. No raw captured credential
|
||||
outputs were inspected or recorded.
|
||||
|
||||
Remaining T04 gate is operational activation: bind durable scheduling/caller and
|
||||
its dependencies, canonical generation names and verified inventory, plus the
|
||||
separate owner retention/fresh-quota operation. Current local cron still invokes
|
||||
the old full-to-Nextcloud helper. This turn's manual tests do not close recurring
|
||||
primary coverage or authorize silently dropping a recovery component.
|
||||
|
||||
Evidence is under `docs/evidence/RPF-WP-0038-*.json`; the complete operational
|
||||
contract and limitations are in `docs/backup-storage-tiers.md`.
|
||||
|
||||
All 221 repository tests passed. Removed the three drill plaintext ZIPs after
|
||||
verified recovery; all good encrypted copies and provider backups remain.
|
||||
Current create-only workload credentials cannot perform owner quota/expiration.
|
||||
The new owner executor remains attended; no owner password or new delete grant
|
||||
was delivered to a workload. Operational binding remains explicit in T04.
|
||||
55
scripts/backup_retention_plan.py
Normal file
55
scripts/backup_retention_plan.py
Normal file
|
|
@ -0,0 +1,55 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Plan bounded essentials retention; never delete provider objects."""
|
||||
import argparse
|
||||
from datetime import datetime, timezone
|
||||
import json
|
||||
from pathlib import Path
|
||||
import re
|
||||
|
||||
NAME=re.compile(r'^forgejo-essentials-(\d{8}T\d{6}Z)\.zip\.age$')
|
||||
DIGEST=re.compile(r'^[0-9a-f]{64}$')
|
||||
RESERVE=2*1024**3
|
||||
MAX_INCOMING=600*1024**2
|
||||
|
||||
|
||||
def plan(inventory, protected, incoming_bytes, available_bytes, now=None):
|
||||
now=now or datetime.now(timezone.utc)
|
||||
if not protected or not 0<incoming_bytes<=MAX_INCOMING or available_bytes<0:
|
||||
raise ValueError('invalid_retention_contract')
|
||||
items={};ignored=[]
|
||||
for item in inventory:
|
||||
name=item['name'];match=NAME.fullmatch(name)
|
||||
if not match or item.get('verified') is not True:
|
||||
ignored.append(name);continue
|
||||
stamp=datetime.strptime(match[1],'%Y%m%dT%H%M%SZ').replace(tzinfo=timezone.utc)
|
||||
etag=item.get('etag','')
|
||||
if (name in items or stamp>now or not DIGEST.fullmatch(item.get('sha256',''))
|
||||
or not etag.startswith('"') or not etag.endswith('"')
|
||||
or any(c in etag for c in '\r\n') or not 0<item.get('bytes',0)<=MAX_INCOMING):
|
||||
raise ValueError('unverified_object_identity')
|
||||
items[name]=(stamp,item)
|
||||
if not set(protected)<=items.keys(): raise ValueError('protected_recovery_point_missing')
|
||||
ordered=sorted(items,key=lambda name:items[name][0],reverse=True)
|
||||
daily={};weekly={}
|
||||
for name in ordered:
|
||||
stamp=items[name][0]
|
||||
daily.setdefault(stamp.date(),name)
|
||||
weekly.setdefault(stamp.isocalendar()[:2],name)
|
||||
keep=set(protected)|set(list(daily.values())[:7])|set(list(weekly.values())[:2])
|
||||
keep.add(ordered[0])
|
||||
candidates=[{'name':name,'etag':items[name][1]['etag'],'bytes':items[name][1]['bytes']}
|
||||
for name in ordered if name not in keep]
|
||||
return {'schema':'platform.essentials-retention-plan.v1','status':'ready' if available_bytes>=incoming_bytes+RESERVE else 'insufficient_upload_headroom',
|
||||
'keep':sorted(keep),'protected':sorted(protected),'delete_candidates':candidates,'ignored':sorted(ignored),
|
||||
'incoming_bytes':incoming_bytes,'available_bytes':available_bytes,'reserved_bytes':RESERVE,
|
||||
'upload_before_expiration':True,'provider_mutations':False,
|
||||
'execution_gate':'Separate owner executor must revalidate ETags, protected recovery receipts and completed replacement before expiration.'}
|
||||
|
||||
|
||||
def main():
|
||||
p=argparse.ArgumentParser(description=__doc__);p.add_argument('--inventory',required=True,type=Path)
|
||||
p.add_argument('--protected',required=True,action='append');p.add_argument('--incoming-bytes',required=True,type=int)
|
||||
p.add_argument('--available-bytes',required=True,type=int);a=p.parse_args()
|
||||
result=plan(json.loads(a.inventory.read_text()),a.protected,a.incoming_bytes,a.available_bytes)
|
||||
print(json.dumps(result,indent=2));return int(result['status']!='ready')
|
||||
if __name__=='__main__': raise SystemExit(main())
|
||||
86
scripts/execute_nextcloud_retention.py
Normal file
86
scripts/execute_nextcloud_retention.py
Normal file
|
|
@ -0,0 +1,86 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Attended owner expiration of reviewed essentials candidates; no workload deletion grant."""
|
||||
import argparse
|
||||
import base64
|
||||
import json
|
||||
import hashlib
|
||||
import fcntl
|
||||
import os
|
||||
from pathlib import Path
|
||||
import urllib.request
|
||||
from backup_retention_plan import NAME, RESERVE
|
||||
from migrate_nextcloud_backup_account import HOST, OPERATOR_PATH, NoRedirect, quota, require
|
||||
from state_hub_preflight_lane import bao,data
|
||||
|
||||
|
||||
def validate(plan, restore):
|
||||
require(plan.get('schema')=='platform.essentials-retention-plan.v1' and plan.get('status')=='ready','ready_plan_required')
|
||||
keep=set(plan['keep']); protected=set(plan['protected'])
|
||||
require(bool(protected) and protected<=keep,'protected_anchor_required')
|
||||
require(restore.get('status')=='restored_essentials' and restore.get('cleanup') is True
|
||||
and restore.get('offsite_artifact') in protected,'verified_recovery_anchor_required')
|
||||
candidates=plan['delete_candidates']; names=[item['name'] for item in candidates]
|
||||
require(len(names)==len(set(names)) and not keep.intersection(names),'ambiguous_expiration_plan')
|
||||
for name in list(keep)+names: require(NAME.fullmatch(name) is not None,'managed_essentials_name_required')
|
||||
for item in candidates:
|
||||
etag=item['etag']
|
||||
require(etag.startswith('"') and etag.endswith('"') and not any(c in etag for c in '\r\n'),'strong_etag_required')
|
||||
return candidates
|
||||
|
||||
|
||||
def execute(plan,restore,receipt):
|
||||
candidates=validate(plan,restore)
|
||||
if not candidates:
|
||||
receipt['status']='no_expiration_needed';return
|
||||
entry=data(bao(['read','-format=json',OPERATOR_PATH]))['data']['data']
|
||||
require(entry['BACKUP_USERNAME']=='Backup','backup_owner_required')
|
||||
auth=(entry['BACKUP_USERNAME'],entry['BACKUP_PASSWORD'])
|
||||
headers={'Authorization':'Basic '+base64.b64encode((auth[0]+':'+auth[1]).encode()).decode()}
|
||||
opener=urllib.request.build_opener(NoRedirect())
|
||||
root=HOST+'/remote.php/dav/files/Backup/railiance-backups/forgejo/'
|
||||
receipt['quota_before']=quota(auth)
|
||||
require(receipt['quota_before']['available_bytes']>=RESERVE,'headroom_recheck_failed')
|
||||
# Verify every retained recovery point is present before any expiration.
|
||||
for name in plan['keep']:
|
||||
with opener.open(urllib.request.Request(root+name,method='HEAD',headers=headers),timeout=60) as response:
|
||||
require(response.status==200 and int(response.headers['Content-Length'])>0,'retained_point_missing')
|
||||
# Revalidate the exact recovered ciphertext, not merely a nonempty name.
|
||||
with opener.open(urllib.request.Request(root+restore['offsite_artifact'],headers=headers),timeout=60) as response:
|
||||
require(0<int(response.headers['Content-Length'])<=600*1024**2,'anchor_size_outside_budget')
|
||||
digest=hashlib.sha256()
|
||||
while block:=response.read(1024*1024): digest.update(block)
|
||||
require(digest.hexdigest()==restore['ciphertext_sha256'],'recovery_anchor_changed')
|
||||
receipt['removed']=[]
|
||||
for item in candidates:
|
||||
with opener.open(urllib.request.Request(root+item['name'],method='HEAD',headers=headers),timeout=60) as response:
|
||||
require(response.headers.get('ETag')==item['etag'] and int(response.headers['Content-Length'])==item['bytes'],'candidate_changed')
|
||||
conditional=dict(headers,**{'If-Match':item['etag']})
|
||||
with opener.open(urllib.request.Request(root+item['name'],method='DELETE',headers=conditional),timeout=60) as response:
|
||||
require(response.status==204,'conditional_expiration_failed')
|
||||
receipt['removed'].append(item['name'])
|
||||
receipt.update(status='expired',quota_after=quota(auth))
|
||||
|
||||
|
||||
def main():
|
||||
p=argparse.ArgumentParser(description=__doc__)
|
||||
p.add_argument('--plan',required=True,type=Path);p.add_argument('--restore-receipt',required=True,type=Path)
|
||||
p.add_argument('--receipt',required=True);p.add_argument('--apply',action='store_true')
|
||||
args=p.parse_args();result={'schema':'platform.essentials-retention-execution.v1','status':'failed'}
|
||||
fd=os.open(args.receipt,os.O_WRONLY|os.O_CREAT|os.O_EXCL,0o600)
|
||||
try:
|
||||
plan=json.loads(args.plan.read_text());restore=json.loads(args.restore_receipt.read_text())
|
||||
candidates=validate(plan,restore)
|
||||
if args.apply:
|
||||
lock=os.open('/tmp/railiance-platform-essentials-retention-'+str(os.getuid())+'.lock',
|
||||
os.O_WRONLY|os.O_CREAT|os.O_NOFOLLOW,0o600)
|
||||
try:
|
||||
require(os.fstat(lock).st_uid==os.getuid(),'retention_lock_owner_mismatch')
|
||||
fcntl.flock(lock,fcntl.LOCK_EX|fcntl.LOCK_NB)
|
||||
execute(plan,restore,result)
|
||||
finally: os.close(lock)
|
||||
else: result.update(status='preview',candidate_count=len(candidates),provider_mutations=False)
|
||||
except Exception: result['error']='bounded_essentials_expiration_refused'
|
||||
finally:
|
||||
with os.fdopen(fd,'w') as output: json.dump(result,output,indent=2)
|
||||
return int(result['status']=='failed')
|
||||
if __name__=='__main__': raise SystemExit(main())
|
||||
47
tests/test_backup_retention_plan.py
Normal file
47
tests/test_backup_retention_plan.py
Normal file
|
|
@ -0,0 +1,47 @@
|
|||
from datetime import datetime,timedelta,timezone
|
||||
from pathlib import Path
|
||||
import sys
|
||||
import unittest
|
||||
sys.path.insert(0,str(Path(__file__).resolve().parents[1]/'scripts'))
|
||||
from backup_retention_plan import plan
|
||||
|
||||
class RetentionPlan(unittest.TestCase):
|
||||
def objects(self):
|
||||
now=datetime(2026,9,6,tzinfo=timezone.utc)
|
||||
return now,[{'name':'forgejo-essentials-'+(now-timedelta(days=i)).strftime('%Y%m%dT%H%M%SZ')+'.zip.age',
|
||||
'verified':True,'etag':'"v'+str(i)+'"','sha256':'a'*64,'bytes':100} for i in range(30)]
|
||||
def test_pinned_recovery_survives_daily_weekly_expiration(self):
|
||||
now,items=self.objects();protected=items[-1]['name']
|
||||
result=plan(items,[protected],100,3*1024**3,now)
|
||||
self.assertIn(protected,result['keep']);self.assertIn(items[0]['name'],result['keep'])
|
||||
self.assertGreater(len(result['delete_candidates']),0)
|
||||
self.assertNotIn(protected,[i['name'] for i in result['delete_candidates']])
|
||||
def test_unknown_full_and_unverified_objects_are_untouched(self):
|
||||
now,items=self.objects();unknown={'name':'forgejo-dump-full.zip.age'}
|
||||
items.append(unknown);items[-2]['verified']=False
|
||||
result=plan(items,[items[0]['name']],100,3*1024**3,now)
|
||||
self.assertIn(unknown['name'],result['ignored']);self.assertIn(items[-2]['name'],result['ignored'])
|
||||
def test_missing_recovery_anchor_refuses(self):
|
||||
now,items=self.objects()
|
||||
with self.assertRaises(ValueError): plan(items,['missing'],100,3*1024**3,now)
|
||||
def test_low_space_does_not_assume_deletion_frees_upload_room(self):
|
||||
now,items=self.objects();result=plan(items,[items[0]['name']],100,0,now)
|
||||
self.assertEqual(result['status'],'insufficient_upload_headroom');self.assertFalse(result['provider_mutations'])
|
||||
|
||||
class OwnerExpiration(unittest.TestCase):
|
||||
def fixture(self):
|
||||
now,objects=RetentionPlan().objects()
|
||||
p=plan(objects,[objects[0]['name']],100,3*1024**3,now)
|
||||
return p,{'status':'restored_essentials','cleanup':True,'offsite_artifact':objects[0]['name']}
|
||||
def test_pinned_anchor_cannot_be_expired(self):
|
||||
from execute_nextcloud_retention import validate
|
||||
p,r=self.fixture();p['delete_candidates'].append({'name':r['offsite_artifact'],'etag':'"v"','bytes':100})
|
||||
with self.assertRaises(Exception):validate(p,r)
|
||||
def test_full_archive_is_outside_executor_scope(self):
|
||||
from execute_nextcloud_retention import validate
|
||||
p,r=self.fixture();p['delete_candidates'].append({'name':'forgejo-dump-full.zip.age','etag':'"v"','bytes':100})
|
||||
with self.assertRaises(Exception):validate(p,r)
|
||||
def test_missing_recovery_proof_refuses(self):
|
||||
from execute_nextcloud_retention import validate
|
||||
p,r=self.fixture();r['status']='failed'
|
||||
with self.assertRaises(Exception):validate(p,r)
|
||||
|
|
@ -1,6 +1,6 @@
|
|||
# Current platform work
|
||||
|
||||
Reviewed 2026-09-06. Seven open workplans: seven blocked on explicit owner/live
|
||||
Reviewed 2026-09-06. Seven open workplans: WP-0038 active, six blocked on explicit owner/live
|
||||
gates; RPF-WP-0036 now has its repository implementation. Completed designs and implementations are
|
||||
under `archived/`; their IDs and UUIDs are preserved. The number of blocked
|
||||
plans is not a count of missing implementations or independent incidents.
|
||||
|
|
@ -13,7 +13,7 @@ plans is not a count of missing implementations or independent incidents.
|
|||
| [RPF-WP-0015](RPF-WP-0015-audit-core-custody-and-recovery-coordination.md) | Two prepared recovery exercises; registered load driver exists; fresh sender/window/abort approvals and custody readiness remain | S3 contributes lease/ESO and snapshot/unseal proof; S1/S2 and audit-core execute their parts. |
|
||||
| [RPF-WP-0035](RPF-WP-0035-credential-lane-implementation.md) | Two remaining lanes: secrets-engine JWT and Fluid operator KV | Signing T04 is complete; JWT and Fluid retain separate issuer/consumer gates. |
|
||||
| [RPF-WP-0036](RPF-WP-0036-platform-service-assurance.md) | Implemented local assurance/admission; waits for recurring restore evidence, Q2 reception and owner handoff | Run the assurance commands; live acceptance and external ownership remain gated. |
|
||||
| [RPF-WP-0038](RPF-WP-0038-forgejo-scaleway-primary-coverage.md) | Forgejo native Scaleway backup and isolated database recovery complete; full archive delivery remains | Separate archive prefix, runtime delivery and application recovery contract required. |
|
||||
| [RPF-WP-0038](RPF-WP-0038-forgejo-scaleway-primary-coverage.md) | Native backup, full Scaleway archive recovery and 273 MiB Nextcloud essentials recovery verified; scheduled tier cutover remains | Bind recurring caller/dependencies, verified inventory, quota checks and separate owner retention. |
|
||||
|
||||
RPF-WP-0036-T02/T05/T07 are complete; T03/T04/T06 retain the remaining
|
||||
acceptance gates. Treat credential exposure closure as the highest-priority attended
|
||||
|
|
|
|||
|
|
@ -96,3 +96,13 @@ archives on Scaleway and design a measured essentials-only Nextcloud profile.
|
|||
T04 acceptance includes package/bootstrap audit, primary-independent essentials
|
||||
recovery, quota preflight and separate bounded retention execution before
|
||||
changing scheduled secondary scope. No blanket package exclusion is activated.
|
||||
|
||||
Implementation evidence, September 6: full Scaleway multipart transfer, versioned
|
||||
GET, decryption and isolated application recovery passed (all 2,040 package
|
||||
blobs). Essentials capture is 273 MiB; real Nextcloud round trip and isolated
|
||||
source/collaboration recovery passed with package service disabled. Retention
|
||||
planner and separate attended owner executor are implemented and tested; no
|
||||
live expiration or cron cutover performed. T04 remains in progress for durable
|
||||
scheduled caller/dependency binding, canonical verified inventory, fresh quota
|
||||
checks and owner retention activation. See
|
||||
`history/2026-09-06-backup-tiers-implementation.md`.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue