railiance-platform/docs/backup-credential-recovery.md

# WP-0029 provider recovery procedure

Primary platform backup is Scaleway (RESOURCE-WP-0002). This procedure covers
the independent Nextcloud secondary-copy lane.

Scope: invalidate the exposed Nextcloud upload predecessor and prove replacement
encrypted upload and offsite recovery under CCR-2026-0004. The route is a
Nextcloud file-drop share, not a platform-admin OpenBao credential. OpenBao
administration does not confer Nextcloud share-owner authority.

## Required owner inputs

- Nextcloud share owner/account and the reviewed share-management session.
- A replacement file-drop share/token delivered through protected custody;
  never chat, argv, Git or State Hub. Record only custody coordinates.
- Owner-authorized retrieval access for the encrypted offsite artifact. Do not
  assume an upload-only file-drop token can list or download backups.
- A protected reference to the predecessor for a bounded rejection probe, or
  accepted provider-side revocation evidence. Do not reconstruct it into logs.

## Execution order

1. Confirm current OpenBao KV version and preserve every existing field. The
   lane contains `AGE_PRIVATE_KEY`; replacing only upload fields must use CAS
   and preserve recovery escrow. Never use a partial `kv put` that drops it.
2. Confirm the exact Nextcloud share and all backup consumers. Besides the
   workstation helpers, activity-core consumes `NC_WEBDAV_TOKEN` and
   `NC_WEBDAV_URL` through ExternalSecret `actcore-backup-offsite`. Coordinate
   its next invocation so process-cached credentials cannot produce a false
   replacement-success claim.
3. Create the replacement under provider authority. Verify its permissions
   match the file-drop lane; no production delete/admin authority is needed.
   Seed both upload fields through contained OpenBao CAS and verify delivery
   with boolean comparisons. Keep provider secrets out of evidence.
4. Upload a uniquely named age-encrypted artifact with the replacement. Require
   HTTP 200/201/204; redirects and network errors are not success. Record only
   artifact identity and safe status. A synthetic fixture can prove transport,
   but it does not establish Forgejo application recovery.
5. Invalidate the predecessor at the provider. Verify rejection with the old
   credential in protected memory against the same bounded operation. A 401/403
   or accepted provider invalidation receipt is evidence; a timeout is not.
   Preserve existing retained backup objects and access to their encryption key.
6. Retrieve the newly uploaded encrypted artifact through the separate owner
   recovery path, decrypt in a protected temporary directory, and verify its
   expected content. For Forgejo restore acceptance, use a fetched real backup
   and the owning isolated restore procedure with repository/blob/database
   checks. Require the local fetched artifact to exist before invoking the
   historical infra drill: that script otherwise creates a fresh production
   dump, which cannot prove offsite recovery.
7. Record provider invalidation, replacement KV version, upload/download,
   decryption, isolated restore and cleanup receipts. Mark T02 done only after
   the agreed recovery checks pass. No receipt may contain the predecessor's
   value, fingerprint, length or shape.

## Recovery-key exposure remains separate

CCR-2026-0004 records historical exposure of the age recovery key as well as
upload credentials. Upload-share rotation alone must not clear that taint.
Retain old age recovery material until all retained artifacts are accounted for
and any owner-approved re-encryption is complete. Do not silently rotate or
discard the age key as part of this upload-token repair.

## Prepared transport

`lib/railiance-backup-common.sh` passes curl credentials and credential-bearing
URLs via stdin configuration, disables curlrc, suppresses backend diagnostics,
requires HTTPS and rejects non-success responses. The Python backend streams
the artifact, refuses redirects and suppresses credential-bearing exception
text. These source changes are preparation; no provider rotation, upload or
restore is claimed by their tests.

## Dedicated Backup account (2026-09-05)

The operator-provisioned account is `Backup` on `nx4069.your-storageshare.de`.
Its credentials remain in OpenBao KVv2 `operators/nextcloud/backup`, fields
`BACKUP_USERNAME` and `BACKUP_PASSWORD` (API `operators/data/nextcloud/backup`).
The UI's `secrets/` prefix is not a separate KV mount.

Future uploads use the account's `/railiance-backups` folder through a dedicated
create-only share. Only its derived `NC_WEBDAV_TOKEN` and `NC_WEBDAV_URL` enter
the existing workload lane and activity-core ExternalSecret; the account password
is not a production credential. Recovery download and cleanup use operator
account authority. Preserve `AGE_PRIVATE_KEY` and all unrelated KV fields with CAS.

The server-reported quota is **10 GiB (10,737,418,240 bytes)**. This is a hard
capacity limit shared by the account's files. Retention counts such as 14 daily
and four weekly backups do not guarantee they fit. Check quota before large
uploads; a quota rejection is a failed backup. Pruning retained backups requires
an explicit retention decision and operator authority; upload jobs cannot delete.
Existing Bernd-owned backups remain in their original location and require their
existing recovery access. This cutover neither migrates them nor revokes Bernd's
personal account or old share.

`scripts/migrate_nextcloud_backup_account.py` performs the attended account,
quota, create-only share, encrypted fixture recovery and CAS checks. Run it only
through `warden access openbao-platform-admin-login --exec -- ...`, with a fresh
private receipt path and `--confirm 'MIGRATE TO Backup'`. The helper is silent;
receipts contain only status and metadata. Review a failed receipt before retrying:
a failure can leave the dedicated folder/share or an identified test fixture, and
a cleanup failure after CAS can occur after credentials have already changed.
Refresh `actcore-backup-offsite`, verify delivery by contained comparison, and
restart its three environment consumers (`actcore-api`, `actcore-event-router`,
`actcore-worker`). Record their readiness and loaded-value comparisons.

## Real archive acceptance

`capture_forgejo_archive.py` requires the exact producer's successful exit marker,
checks every transferred chunk's length and the complete SHA-256 against the
producer, then validates ZIP structure and member CRCs before encryption. The
2026-09-05 recovery attempt exposed a truncated September 4 encrypted archive;
successful age decryption and upload alone cannot certify backup completeness.

`verify_nextcloud_offsite_restore.py` streams a real encrypted archive through
the current upload lane and downloads it separately through owner recovery
custody. It emits a hash-bound transfer receipt. Run this silent helper through
the attended Warden envelope, with fresh private staging and receipt paths.
`restore_forgejo_offsite_locally.py` requires both that receipt and the fetched
archive. Its disposable local Docker network is internal, exposes no host ports,
and uses an independent database password/configuration. Production app.ini is
never activated. It checks database import, application health, repository API
and clone/fsck, and package blob integrity; cleanup removes its named containers,
anonymous volumes, network and extracted staging. Remaining fetched plaintext
must be removed after the acceptance receipt is preserved.

This bounded application-data recovery does not test replacement runners or
prove that every restored package can be installed or every image pulled.
Package-consumer acceptance and broader disaster-recovery guarantees remain
with their owning assurance tasks. Full backups share the account's 10 GiB
quota; a newly measured archive size must inform the separate retention decision.

## WSL attended-login browser repair

If this host has no `xdg-open`, `x-www-browser` or `www-browser`, the installed
Bao browser launcher cannot open its login page. Warden contains that output,
so this presents as a failed callback before the owner command starts. Use:

```bash
python3 scripts/openbao-attended-exec.py -- /absolute/path/to/reviewed-silent-command
```

The wrapper keeps Warden's `openbao-platform-admin-login --exec` envelope and
adds a repository-scoped launcher only when all native providers are absent.
The launcher allows only the Keycloak HTTPS host and exact local OIDC callback;
it sends the URL to Windows PowerShell through stdin without recording it.
No token helper or captured Warden output may be inspected. A receipt written
by the owner command proves handoff; session cleanup is assessed separately
from the final Warden exit status.