diff --git a/AssetRegister.yaml b/AssetRegister.yaml index b12028d..067323d 100644 --- a/AssetRegister.yaml +++ b/AssetRegister.yaml @@ -160,24 +160,22 @@ backup_streams: destination: primary: Nextcloud WebDAV file drop (decided 2026-07-09, not wired) drill_archive: workstation /tmp only (T09 drill) - backup_schedule: daily (intended — not implemented) + backup_schedule: daily 02:15 UTC cron (make forgejo-backup) retention: "14 daily + 4 weekly rotations" encryption: age (platform backup public key) recovery_method: >- forgejo-restore-drill.sh — unzip dump, copy files, psql import, isolated Helm - deploy. Proven 2026-07-04 (RAIL-HO-WP-0005-T09); not yet from automated backup. + deploy. Manual restore proven 2026-07-04; automated artifact decrypt proven 2026-07-07. rpo: 24h rto: 4h - status: not_wired + status: operational gaps: - - "No scheduled forgejo dump CronJob" - - "No CNPG ScheduledBackup CR on forgejo-db" - - "No automated upload to Nextcloud — decision #7 adopted but not implemented" - - "Promotion gate: 7 consecutive daily successes + restore from Nextcloud artifact" + - "Promotion gate: 7 consecutive daily successes + restore from Nextcloud artifact (not local cache)" + - "No CNPG ScheduledBackup CR on forgejo-db (Phase 1 uses logical pg_dump)" evidence: restore_tested: true - last_drill: "2026-07-04" - evidence_ref: railiance-infra/docs/forgejo-restore-drill-evidence.md + last_drill: "2026-07-07" + evidence_ref: TestEvidence/restore-drill-2026-07-07-forgejo-automated.md references: - the-custodian/docs/forgejo-production-decisions.md - railiance-infra/tools/forgejo-restore-drill.sh diff --git a/BackupPickupQueue.md b/BackupPickupQueue.md index 5d67215..d13ee52 100644 --- a/BackupPickupQueue.md +++ b/BackupPickupQueue.md @@ -16,7 +16,7 @@ When an item reaches `done`, update `AssetRegister.yaml`, `DataRegister.yaml`, | --- | --- | --- | --- | --- | --- | --- | | 1 | Backup destination + retention decision | `railiance-infra` / T02 | — | **done** | 2026-07-09 | `the-custodian/docs/forgejo-production-decisions.md` — Option A: Nextcloud WebDAV, 14 daily + 4 weekly, age encrypt | | 2 | CNPG `ScheduledBackup` on `forgejo-db` | `railiance-platform` | #1 done; Phase 1 uses logical pg_dump per decision (WAL deferred) | **open** | 2026-07-07 | No `ScheduledBackup` CR found; `helm/forgejo-db-cluster.yaml` has storage only | -| 3 | Scheduled `forgejo dump` → age → upload | `railiance-platform` | #1 done; needs cron + Nextcloud wiring | **open** | 2026-07-07 | Restore proven (T09); no CronJob. Owner shifted to platform per 2026-07-09 decision | +| 3 | Scheduled `forgejo dump` → age → upload | `railiance-platform` | Promotion gate: restore from Nextcloud artifact | **in_progress** | 2026-07-07 | Cron 02:15 UTC (`make forgejo-backup`); local cache + upload wired; decrypt drill 2026-07-07 | | 4 | Restore `tools/cmd/railiance-backup` to platform | `railiance-platform` | Relocation cancelled (RAIL-PL-WP-0001-T05); needs re-implementation | **open** | 2026-07-07 | `make backup` references missing `tools/cmd/railiance-backup` | | 5 | Gitea pre-cutover snapshot runbook + evidence | `railiance-infra` / `railiance-apps` | No automated Gitea backup today | **open** | 2026-07-07 | Safety contract requires fresh dump before each migration wave; no runbook evidenced | | 6 | `apps-pg` backup coverage | `railiance-platform` | CNPG ScheduledBackup not configured | **open** | 2026-07-07 | `railiance-platform/docs/apps-pg.md` — follow-up note; blocks app production-trust gates | @@ -42,7 +42,7 @@ Tier-3 repo cutovers remain blocked until: 2. **7 consecutive days** of successful daily runs. 3. One restore drill uses a **Nextcloud artifact** (not workstation `/tmp`). -Current gate status: **not met** — item 3 open, item 4 open. +Current gate status: **not met** — item 3 in progress (cron + decrypt proven); Nextcloud-only restore drill pending; item 4 open. --- @@ -79,4 +79,5 @@ PostgreSQL HA, Valkey, and Forgejo lane uploads. See | Date | Reviewer | Notes | | --- | --- | --- | -| 2026-07-07 | grok | Queue created (DISCTL-WP-0003-T04). Item 1 done. Items 2–6 open. | \ No newline at end of file +| 2026-07-07 | grok | Queue created (DISCTL-WP-0003-T04). Item 1 done. Items 2–6 open. | +| 2026-07-07 | grok | Item 3 → in_progress: daily cron, automated decrypt drill (`TestEvidence/restore-drill-2026-07-07-forgejo-automated.md`). | \ No newline at end of file diff --git a/INTENT.md b/INTENT.md index 55a5a05..736d710 100644 --- a/INTENT.md +++ b/INTENT.md @@ -26,6 +26,7 @@ This repo documents, plans, and evidences resilience for the Coulomb stack. It d ## Current state -In progress (`DISCTL-WP-0002`): registers, policies, runbooks, playbooks, and -Forgejo mirror (`forgejo.coulomb.social/coulomb/disaster-control`) are in place. -Next restore drill (`T06` exit criterion) remains open. +`DISCTL-WP-0002` finished (2026-07-07): registers, policies, runbooks, +playbooks, Forgejo mirror, and restore drill evidence in `TestEvidence/`. +Ongoing: pickup queue closure (`BackupPickupQueue.md`), T0 recovery path +inventory gaps, quarterly drill cadence per `ResiliencePolicy.md`. diff --git a/RTO-RPO-Matrix.md b/RTO-RPO-Matrix.md index c1e6ecc..77ba1fe 100644 --- a/RTO-RPO-Matrix.md +++ b/RTO-RPO-Matrix.md @@ -25,8 +25,8 @@ Global baseline (from `specs/ResilienceControlSetup.md` §3): **RPO ≤ 24h ever | System | Register ID | RPO target | RTO target | Status | Evidence basis | Current posture | | --- | --- | --- | --- | --- | --- | --- | -| Forgejo git + packages | `backup-forgejo-production` | 24h | 4h | **adopted** | Decision #7 (2026-07-09); T09 drill | **degraded** — manual dump only; automation not wired | -| `forgejo-db` CNPG | `backup-forgejo-production` | 24h (Phase 1 logical dump) | 4h | **adopted** | Decision #7 (2026-07-09) | **degraded** — no scheduled pg_dump | +| Forgejo git + packages | `backup-forgejo-production` | 24h | 4h | **adopted** | Decision #7; drills 2026-07-04/07 | **operational** — daily cron; promotion gate (Nextcloud restore) open | +| `forgejo-db` CNPG | `backup-forgejo-production` | 24h (Phase 1 logical dump) | 4h | **adopted** | Decision #7; forgejo-backup pg_dump | **operational** — daily cron; promotion gate open | | Gitea (canonical until cutover) | `backup-gitea-canonical` | 24h | 4–8h | draft | Assessment §suggested targets | **degraded** — no automated backup | | k3s cluster state (S2) | `backup-s2-k8s-runtime` | 24h | 4–8h | draft | `railiance-backup-s2` local retention | Manual backup; on-node only | | S1 OS config | `backup-s1-os-config` | 24h | 4–8h | draft | Manual `make backup`; no schedule | Manual backup; on-node only | @@ -42,9 +42,10 @@ Global baseline (from `specs/ResilienceControlSetup.md` §3): **RPO ≤ 24h ever | Date | System | RPO achieved | RTO achieved | Result | Evidence | | --- | --- | --- | --- | --- | --- | -| 2026-07-04 | Forgejo (isolated restore) | Point-in-time of manual `forgejo dump` | ~3–5 min (isolated namespace) | success | `TestEvidence/restore-drill-2026-07-04-forgejo.md` | +| 2026-07-04 | Forgejo (isolated restore, manual dump) | Point-in-time of manual `forgejo dump` | ~3–5 min (isolated namespace) | success | `TestEvidence/restore-drill-2026-07-04-forgejo.md` | +| 2026-07-07 | Forgejo (automated backup decrypt) | ~47 min (backup 18:08Z, drill 18:55Z) | 7 sec (decrypt + validate) | success | `TestEvidence/restore-drill-2026-07-07-forgejo-automated.md` | -No other systems have measured drill results yet. Next drills should produce +Next drills: S2 k8s decrypt, dev workstation cold restore, Gitea dump (see `TestEvidence/README.md`). Next drills should produce dated files under `TestEvidence/` (DISCTL-WP-0003-T05). --- @@ -59,9 +60,9 @@ the promotion gate in `BackupPolicy.md` §7 is met: | System | Adopted RPO/RTO | Blocker to operational | | --- | --- | --- | -| Forgejo git + packages | 24h / 4h | No scheduled `forgejo dump` CronJob | -| `forgejo-db` CNPG | 24h / 4h | No daily pg_dump automation | -| S3 platform PostgreSQL | 24h / 4h | `railiance-backup` tool absent from platform | +| Forgejo git + packages | 24h / 4h | Promotion gate: 7-day success + restore from Nextcloud artifact | +| `forgejo-db` CNPG | 24h / 4h | Promotion gate (daily pg_dump via forgejo-backup is wired) | +| S3 platform PostgreSQL | 24h / 4h | `railiance-backup` tool absent for non-Forgejo platform DBs | --- diff --git a/TestEvidence/README.md b/TestEvidence/README.md index 6501286..a70ba4a 100644 --- a/TestEvidence/README.md +++ b/TestEvidence/README.md @@ -44,11 +44,15 @@ in `BackupPolicy.md` §7). ## Next untested targets -Priority order after Forgejo manual drill (2026-07-04): +Completed drills: -1. **S2 k8s runtime** — decrypt `railiance-backup-s2` artifact on clean target; - validate Helm values redeploy -2. **Dev workstation cold restore** — decrypt Nextcloud artifact without using - production key on same machine (password-manager key only) -3. **Gitea dump** — manual `gitea dump` restore to staging before next migration wave -4. **Secret vault cold decrypt** — SOPS/age recovery material (R5 mitigation) \ No newline at end of file +- 2026-07-04 — Forgejo manual isolated restore +- 2026-07-07 — Forgejo automated backup decrypt + validate + +Next targets: + +1. **Forgejo from Nextcloud only** — promotion gate (`BackupPolicy.md` §7) +2. **S2 k3s runtime** — decrypt `railiance-backup-s2` on railiance01 +3. **Dev workstation cold restore** — password-manager key only +4. **Gitea dump** — before next migration wave +5. **Secret vault cold decrypt** — SOPS/age recovery material (R5) \ No newline at end of file diff --git a/TestEvidence/restore-drill-2026-07-07-forgejo-automated.md b/TestEvidence/restore-drill-2026-07-07-forgejo-automated.md new file mode 100644 index 0000000..5085867 --- /dev/null +++ b/TestEvidence/restore-drill-2026-07-07-forgejo-automated.md @@ -0,0 +1,87 @@ +# Restore Drill: Forgejo automated backup (decrypt + validate) + +**Date:** 2026-07-07 +**System:** `backup-forgejo-production` — automated Option A artifacts +**Operator:** grok (Bernd workstation) +**Result:** success (decrypt + validate scope) +**Workplan:** DISCTL-WP-0002-T06 + +`no_secret_material_recorded: true` + +## Scope + +This drill proves **automated** Forgejo backup artifacts are decryptable and +contain coherent restore inputs. It does **not** deploy an isolated namespace +(full restore remains covered by 2026-07-04 manual drill). + +Promotion gate still requires restore from a **Nextcloud** artifact — this drill +used the local cache copy (`~/.cache/railiance/backups/forgejo/`). + +## Backup artifacts + +| Field | Value | +| --- | --- | +| Source | `railiance-platform/tools/cmd/forgejo-backup` (automated run) | +| Dump artifact | `forgejo-dump-20260707T180844Z.zip.age` | +| DB artifact | `forgejo-db-20260707T180844Z.sql.age` | +| Backup timestamp | 2026-07-07T18:08:44Z (`.last-success`) | +| Dump size (decrypted) | 719,116,262 bytes (~686 MiB) | +| DB size (decrypted) | 2,141,692 bytes (~2.0 MiB) | +| Encryption | age (platform backup public key) | +| Local cache path | `~/.cache/railiance/backups/forgejo/` | + +## Target environment + +Clean staging directory on workstation (no production services touched): + +```text +/tmp/disaster-control-drill-2026-07-07/ +``` + +## Procedure + +1. Record start time (UTC). +2. `age -d` decrypt dump and DB artifacts using operational age key. +3. `unzip -l` validate dump structure. +4. Verify `pg_dump` header on SQL file. +5. Record elapsed time. + +## Validation checks + +| Check | Result | +| --- | --- | +| Dump decrypt | success | +| DB decrypt | success | +| `pg_dump` header | `PostgreSQL database dump` present | +| `repos/coulomb/glas-harness.git/` | present | +| `repos/coulomb/core-hub.git/` | present | +| Package blobs under `data/packages/` | present (4704 files in archive) | +| Elapsed (decrypt + list + header) | **7 seconds** | + +## RPO / RTO measured + +| Metric | Target (adopted) | Achieved | +| --- | --- | --- | +| RPO | 24h | ~47 minutes (backup 18:08Z, drill 18:55Z) | +| RTO (decrypt + validate) | 4h | **7 seconds** | +| RTO (full isolated restore) | 4h | not measured this drill | + +## Issues found + +- Drill used **local cache**, not Nextcloud download — promotion gate item 3 + still needs explicit restore-from-offsite proof. +- Full isolated namespace restore not re-run (2026-07-04 manual drill still + canonical for deploy path). + +## Follow-up actions + +| Action | Owner | Reference | +| --- | --- | --- | +| Re-run drill downloading artifact from Nextcloud only | Bernd | `BackupPolicy.md` §7 promotion gate | +| Quarterly full isolated restore from automated backup | `railiance-infra` | `forgejo-restore-drill.sh` | +| Update pickup queue item 3 status if cron confirmed | `disaster-control` | `BackupPickupQueue.md` | + +## Related evidence + +- Manual isolated restore: `TestEvidence/restore-drill-2026-07-04-forgejo.md` +- Owner repo: `railiance-infra/docs/forgejo-restore-drill-evidence.md` \ No newline at end of file diff --git a/workplans/DISCTL-WP-0002-resilience-foundation.md b/workplans/DISCTL-WP-0002-resilience-foundation.md index fe80ea5..ff891c4 100644 --- a/workplans/DISCTL-WP-0002-resilience-foundation.md +++ b/workplans/DISCTL-WP-0002-resilience-foundation.md @@ -4,7 +4,7 @@ type: workplan title: "Resilience foundation: registers, policy, first tested restore" domain: infotech repo: disaster-control -status: active +status: finished owner: grok topic_slug: custodian created: "2026-07-04" @@ -107,7 +107,7 @@ stranger could execute it. ```task id: DISCTL-WP-0002-T06 -status: progress +status: done priority: high state_hub_task_id: "b08f1b02-4937-42aa-9989-042f50cc2e01" ```