railiance-apps/docs/app-data-backup-restore-handoff.md

# App Data Backup And Restore Handoff

This document defines the S5 app release boundary for data durability. It does
not create backup jobs, authorize a live restore drill, or move platform
backup ownership into `railiance-apps`.

## Current App Data

The live invited pilot uses `vergabe_demo_company` on `databases/apps-pg`,
with runtime Secret reference `vergabe-demo-company/vergabe-demo-env`. It has
separate Bound local-path claims for media (5Gi) and issue state (1Gi). See
[the current binding](vergabe-demo-company-binding.md). Historical `vergabe_db`
and July backup receipts below do not establish this new company's recovery.

RAPPS-WP-0014-T03 remains waiting: the September 25 rehearsal proved database
restore and restart, with empty file stores and no off-host upload. A populated,
coherent three-store recovery point and restored workflow are still required.
The CNPG/storage backup mechanisms remain owned by `railiance-platform`.

## Ownership Matrix

| Concern | S5 app repo owns | Upstream owner |
| --- | --- | --- |
| App database request | App name, namespace, database name, role name, intended use, and production-readiness need | `railiance-platform` reviews and provisions the role/database |
| Runtime DB Secret use | Secret name in the app namespace and URL-encoded DSN rebuild helper | `railiance-platform` owns platform credential source and future secret delivery |
| Database backup job | Readiness gate and consumer evidence requirement | `railiance-platform` owns CNPG backup and restore implementation |
| App restore verification | App-specific post-restore checks, migrations, login/smoke path, and rollback note | `railiance-platform` restores the backing database |
| Forge images/packages | Artifact identity and consumer evidence cited by app runbooks | `railiance-forge` owns registry/package restore evidence |
| App media/blob data | PVC declaration and app-level restore checks if enabled | `railiance-platform` owns storage backup mechanism once media is production-critical |

## Production Readiness Gate

Before an app release treats data as production-critical, the app runbook should
record:

- data class: disposable, externally reproducible, or production-critical;
- owning platform workplan or doc for the backup mechanism;
- latest non-secret backup evidence reference;
- latest restore-drill evidence reference, ideally from an isolated
  environment;
- app-specific post-restore checks, such as migrations, health endpoint,
  login/admin path, and representative business workflow;
- rollback or disable path if restore fails;
- assertion that no secret material was copied into Git, logs, screenshots, or
  State Hub notes.

If this gate is missing, the app can still be used for smoke, development, or
migration validation, but promotion beyond that should create or link a
`railiance-platform` workplan.

## Historical production cluster inventory (2026-07-10)

Custodian delivery-lane snapshot
(`the-custodian/docs/evidence/vergabe-teilnahme-delivery-lane-20260710.json`)
confirmed all four production CNPG clusters on railiance01 have
`spec.backup=null` and no `ScheduledBackup` resources:

| Cluster | Consumer impact |
| --- | --- |
| `apps-pg` | `vergabe_db` and future S5 app databases |
| `forgejo-db` | Forgejo metadata and package registry state |
| `net-kingdom-pg` | Net Kingdom relational data |
| `state-hub-db` | State Hub API persistence |

Check current posture:

```bash
make cnpg-backup-status
```

**Option A (decided, RAILIANCE-WP-0015):** age-encrypted logical dumps →
Nextcloud WebDAV via CronJobs in `manifests/cnpg-option-a-backup.yaml`.

```bash
bao login -method=oidc -path=netkingdom role=railiance-backup-workload-kv-read
make cnpg-backup-offsite-secret-apply   # Secret from OpenBao (no private key)
make cnpg-option-a-apply                # schedule CM + RBAC + suspended CronJobs
make cnpg-logical-backup                # immediate / daily workstation run
make cnpg-backup-status
```

**Historical workstation schedule (superseded):** RAPPS-WP-0002 records the
July 22 move to in-cluster CronJobs on CoulombCore and activity-core on
Railiance01. This old workstation example is not a current pilot backup receipt:

```cron
# Daily 02:30 UTC — Option A multi-cluster logical backup (RPO 24h)
30 2 * * * cd $HOME/railiance-apps && make cnpg-logical-backup >>$HOME/.cache/railiance/backups/cnpg/cron.log 2>&1
```

Barman `ObjectStore` / CNPG `ScheduledBackup` remain deferred (Phase 2 stubs in
`manifests/cnpg-backup-readiness.yaml`).

## Historical `vergabe_db` gate

Current posture (2026-07-12):

- database: `vergabe_db` on `databases/apps-pg`;
- app role Secret: `vergabe-app-credentials`;
- env Secret: `vergabe-teilnahme-env`;
- **Phase 1 backup lane:** `make apps-pg-backup` uploads encrypted logical dumps
  to the platform offsite lane (latest:
  `apps-pg-vergabe_db-20260711T231430Z.dump.age`);
- **CNPG Option A posture:** `make cnpg-backup-status` reports Option A
  CronJob + last-success coverage (not barman `ScheduledBackup`); see
  RAILIANCE-WP-0015 for fleet rollout;
- **restore drill:** isolated restore with row-count gate 195/195 pass
  (`docs/evidence/apps-pg-restore-drill-20260711T230725Z.json`);
- **S5 production-trust gate:** **satisfied for Phase 1** — backup + restore
  evidence recorded in `docs/evidence/apps-pg-backup-lane-20260711.json`;
  app migration smoke on drill DB deferred (row gate sufficient for interim gate).

Evidence index (`RAILIANCE-WP-0013`):

| Check | Evidence |
| --- | --- |
| Backup lane auth + upload | `docs/evidence/apps-pg-backup-lane-20260711.json` |
| Isolated restore drill | `docs/evidence/apps-pg-restore-drill-20260711T230725Z.json` |
| Operator commands | `make apps-pg-backup`, `make apps-pg-restore-drill` |

Remaining before full production-critical promotion (post–Phase 1):

- `vergabe-teilnahme` migration smoke on restored database;
- health endpoint and HTTPS smoke checks after restore;
- representative tender-management workflow verification;
- Option A unattended coverage for all four CNPG clusters
  (`manifests/cnpg-option-a-backup.yaml`, RAILIANCE-WP-0015);
- any app media path remains disabled or has its own storage restore evidence.

## Forge Artifact Evidence

S5 runbooks may cite forge-owned package and blob restore evidence, but must not
own Gitea package backup procedures or registry credentials. Use
`/home/worsch/railiance-forge/docs/backup-restore-secret-handoff.md` for the
forge artifact boundary.

For app releases, cite:

- image repository, tag, and digest when available;
- source commit and package version;
- forge publish job or evidence reference;
- package/blob restore drill evidence when the artifact is production-critical;
- namespace-local pull Secret or approved workload secret path, without token
  values.

## Filing Upstream Gaps

When the missing durability item is not local to S5:

1. Keep the S5 task focused on the app release impact.
2. Create or link the platform/forge workplan that owns the missing mechanism.
3. Mark the S5 task `wait` and its workplan `blocked` when the app release cannot safely continue
   without that upstream evidence.
4. Record the State Hub workstream/task id in the app runbook or workplan.
5. Revisit the S5 promotion gate after upstream evidence exists.