net-kingdom/docs/identity-restore-drill.md

# Quarterly isolated identity restore drill

Work record: `NK-WP-0053` (from `NK-WP-0049-T24` gap G3, founder decision
`e4c288d9`, 2026-10-23). Cadence: quarterly, aligned with disaster-control
`BackupPolicy.md`. First drill: the attended `admin` run of the week of 2026-10-28, well ahead of
**2027-01-31**, the end of the single-provider risk acceptance. Later drills are
scheduled through disaster-control's restore-proof register.

The drill re-proves restores that have each been proven once (NK49-T16 LLDAP,
T31 privacyIDEA token material, T20 audit). It uses the existing reviewed
tooling unchanged and adds no grant, scheduler or production write. One
attended operator runs it in their own terminal, using the independently held
backup age key and the Backup-account credential through Platform custody.

## Scope

| Step | Tool | Proves |
|---|---|---|
| 1. Independent retrieval | `tools/bootstrap-recovery/retrieve-identity.py --source-receipt <node receipt> --receipt <private transfer receipt>` ([ceremony](bootstrap-recovery-ceremony.md) step 5) | The newest successful generation re-downloads and binds by ETag/size/hash |
| 2. LLDAP | `tools/bootstrap-recovery/run-isolated-lldap.sh <user> <transfer receipt>` | Directory restore and authentication in isolation |
| 3. privacyIDEA | `tools/bootstrap-recovery/run-isolated-privacyidea.sh <transfer receipt>` | Database, token-material decryption and audit signatures in isolation |
| 4. Authelia | `tools/bootstrap-recovery/run-isolated-authelia.sh admin <transfer receipt>` | Authelia restored against the restored directory accepts the existing user `admin` in isolation |
| 5. By-user MFA | `tools/bootstrap-recovery/run-isolated-privacyidea-mfa.sh <transfer receipt>`, login `admin` | `admin`'s retained OTP token from the backup validates a fresh code in isolation |

Every stage is run by `admin`, the operator of record and the MFA test subject;
`platform-root` is kept for break-glass. Stage 5 needs a generation captured after
2026-10-25 03:45 UTC, the first daily capture holding `admin`'s OTP token; an older
generation proves nothing for `admin`.

Out of scope: KeyCape consumer continuity (NK49-T18, a separate admitted
attempt), any production restore or cutover, and
`run-offline-recovery.sh` (it publishes kit files, which is a production
effect).

## Before the drill

- Review the [crisis custody list](crisis-custody-list.md): every row must have a holder, a location and a last-proven date, and no row may rely on a login to a service being restored. The operator of record is `admin`.
- Pick the generation from the daily backup (the newest that passed health).
  Record its date, not its content.
- Stage the pinned images locally. Each launcher refuses if they are missing.
- Open an OpenBao session as `admin` in the operator shell:
  `bao login -method=oidc -path=netkingdom role=platform-admin` (MFA). Stage 1 reads
  the Backup-account credential from OpenBao custody through that token and never
  prompts; without a valid `platform-admin` token it writes a failed receipt
  (`contained_owner_retrieval_failed`) and prints nothing. Check `echo $?` after stage 1.
- Confirm no outbound notification or provider write is configured. The
  launchers enforce isolation; confirm their preflight passes.

## Pass and fail

A drill passes only if every launcher exits 0, each safe receipt reports
cleanup verified with no uncertainty, and the ciphertext digest in every receipt
equals the transfer receipt. Any other result is a **failed drill**. Record it
as a finding under NK-WP-0049 or its successor, with the safe receipts. Do not
rerun with modified inputs until the cause is understood, and never repair a
receipt by hand.

## Evidence

Keep the value-safe receipts under
`docs/evidence/<YYYY-MM-DD>-identity-restore-drill/`:
`transfer.json` (public fields only), `lldap-recovery.json`,
`privacyidea-recovery.json`, `authelia-recovery.json`, `privacyidea-mfa-recovery.json`
and a short `drill.json` with the date, generation, verdict, next due date and, per
stage, the identity that ran it and the login it tested. Before committing, run the repository
secret scan (the pre-commit hook does this). No key, password, token or
plaintext may enter Git, State Hub or chat.

After each drill, put the next due date (drill date plus 3 months) on the
NK-WP-0053 record and tell disaster-control, so its restore-proof register
stays current.