railiance-platform/docs/backup-credential-recovery.md

WP-0029 provider recovery procedure

Primary platform backup is Scaleway (RESOURCE-WP-0002). This procedure covers the independent Nextcloud secondary-copy lane.

Scope: invalidate the exposed Nextcloud upload predecessor and prove replacement encrypted upload and offsite recovery under CCR-2026-0004. The route is a Nextcloud file-drop share, not a platform-admin OpenBao credential. OpenBao administration does not confer Nextcloud share-owner authority.

Required owner inputs

• Nextcloud share owner/account and the reviewed share-management session.
• A replacement file-drop share/token delivered through protected custody; never chat, argv, Git or State Hub. Record only custody coordinates.
• Owner-authorized retrieval access for the encrypted offsite artifact. Do not assume an upload-only file-drop token can list or download backups.
• A protected reference to the predecessor for a bounded rejection probe, or accepted provider-side revocation evidence. Do not reconstruct it into logs.

Execution order

1. Confirm current OpenBao KV version and preserve every existing field. The lane contains AGE_PRIVATE_KEY; replacing only upload fields must use CAS and preserve recovery escrow. Never use a partial kv put that drops it.
2. Confirm the exact Nextcloud share and all backup consumers. Besides the workstation helpers, activity-core consumes NC_WEBDAV_TOKEN and NC_WEBDAV_URL through ExternalSecret actcore-backup-offsite. Coordinate its next invocation so process-cached credentials cannot produce a false replacement-success claim.
3. Create the replacement under provider authority. Verify its permissions match the file-drop lane; no production delete/admin authority is needed. Seed both upload fields through contained OpenBao CAS and verify delivery with boolean comparisons. Keep provider secrets out of evidence.
4. Upload a uniquely named age-encrypted artifact with the replacement. Require HTTP 200/201/204; redirects and network errors are not success. Record only artifact identity and safe status. A synthetic fixture can prove transport, but it does not establish Forgejo application recovery.
5. Invalidate the predecessor at the provider. Verify rejection with the old credential in protected memory against the same bounded operation. A 401/403 or accepted provider invalidation receipt is evidence; a timeout is not. Preserve existing retained backup objects and access to their encryption key.
6. Retrieve the newly uploaded encrypted artifact through the separate owner recovery path, decrypt in a protected temporary directory, and verify its expected content. For Forgejo restore acceptance, use a fetched real backup and the owning isolated restore procedure with repository/blob/database checks. Require the local fetched artifact to exist before invoking the historical infra drill: that script otherwise creates a fresh production dump, which cannot prove offsite recovery.
7. Record provider invalidation, replacement KV version, upload/download, decryption, isolated restore and cleanup receipts. Mark T02 done only after the agreed recovery checks pass. No receipt may contain the predecessor's value, fingerprint, length or shape.

Recovery-key exposure remains separate

CCR-2026-0004 records historical exposure of the age recovery key as well as upload credentials. Upload-share rotation alone must not clear that taint. Retain old age recovery material until all retained artifacts are accounted for and any owner-approved re-encryption is complete. Do not silently rotate or discard the age key as part of this upload-token repair.

Prepared transport

lib/railiance-backup-common.sh passes curl credentials and credential-bearing URLs via stdin configuration, disables curlrc, suppresses backend diagnostics, requires HTTPS and rejects non-success responses. The Python backend streams the artifact, refuses redirects and suppresses credential-bearing exception text. These source changes are preparation; no provider rotation, upload or restore is claimed by their tests.

Dedicated Backup account (2026-09-05)

The operator-provisioned account is Backup on nx4069.your-storageshare.de. Its credentials remain in OpenBao KVv2 operators/nextcloud/backup, fields BACKUP_USERNAME and BACKUP_PASSWORD (API operators/data/nextcloud/backup). The UI's secrets/ prefix is not a separate KV mount.

Future uploads use the account's /railiance-backups folder through a dedicated create-only share. Only its derived NC_WEBDAV_TOKEN and NC_WEBDAV_URL enter the existing workload lane and activity-core ExternalSecret; the account password is not a production credential. Recovery download and cleanup use operator account authority. Preserve AGE_PRIVATE_KEY and all unrelated KV fields with CAS.

The server-reported quota is 10 GiB (10,737,418,240 bytes). This is a hard capacity limit shared by the account's files. Retention counts such as 14 daily and four weekly backups do not guarantee they fit. Check quota before large uploads; a quota rejection is a failed backup. Pruning retained backups requires an explicit retention decision and operator authority; upload jobs cannot delete. Existing Bernd-owned backups remain in their original location and require their existing recovery access. This cutover neither migrates them nor revokes Bernd's personal account or old share.

scripts/migrate_nextcloud_backup_account.py performs the attended account, quota, create-only share, encrypted fixture recovery and CAS checks. Run it only through warden access openbao-platform-admin-login --exec -- ..., with a fresh private receipt path and --confirm 'MIGRATE TO Backup'. The helper is silent; receipts contain only status and metadata. Review a failed receipt before retrying: a failure can leave the dedicated folder/share or an identified test fixture, and a cleanup failure after CAS can occur after credentials have already changed. Refresh actcore-backup-offsite, verify delivery by contained comparison, and restart its three environment consumers (actcore-api, actcore-event-router, actcore-worker). Record their readiness and loaded-value comparisons.

Real archive acceptance

capture_forgejo_archive.py requires the exact producer's successful exit marker, checks every transferred chunk's length and the complete SHA-256 against the producer, then validates ZIP structure and member CRCs before encryption. The 2026-09-05 recovery attempt exposed a truncated September 4 encrypted archive; successful age decryption and upload alone cannot certify backup completeness.

verify_nextcloud_offsite_restore.py streams a real encrypted archive through the current upload lane and downloads it separately through owner recovery custody. It emits a hash-bound transfer receipt. Run this silent helper through the attended Warden envelope, with fresh private staging and receipt paths. restore_forgejo_offsite_locally.py requires both that receipt and the fetched archive. Its disposable local Docker network is internal, exposes no host ports, and uses an independent database password/configuration. Production app.ini is never activated. It checks database import, application health, repository API and clone/fsck, and package blob integrity; cleanup removes its named containers, anonymous volumes, network and extracted staging. Remaining fetched plaintext must be removed after the acceptance receipt is preserved.

This bounded application-data recovery does not test replacement runners or prove that every restored package can be installed or every image pulled. Package-consumer acceptance and broader disaster-recovery guarantees remain with their owning assurance tasks. Full backups share the account's 10 GiB quota; a newly measured archive size must inform the separate retention decision.

WSL attended-login browser repair

If this host has no xdg-open, x-www-browser or www-browser, the installed Bao browser launcher cannot open its login page. Warden contains that output, so this presents as a failed callback before the owner command starts. Use:

python3 scripts/openbao-attended-exec.py -- /absolute/path/to/reviewed-silent-command

The wrapper keeps Warden's openbao-platform-admin-login --exec envelope and adds a repository-scoped launcher only when all native providers are absent. The launcher allows only the Keycloak HTTPS host and exact local OIDC callback; it sends the URL to Windows PowerShell through stdin without recording it. No token helper or captured Warden output may be inspected. A receipt written by the owner command proves handoff; session cleanup is assessed separately from the final Warden exit status.