net-kingdom/docs/bootstrap-recovery-ceremony.md

# NK-WP-0049 recovery ceremony and closure packet

Status: actual off-host privacyIDEA database drill passed; full identity backup
coverage remains in preparation. This document grants no credential export,
provider access, revocation, rotation, root retrieval or production bootstrap.

## What has been established

On 2026-10-15 the operator identified Scaleway and Storage Share as the off-host
backup providers. This identifies destinations, not the exact recovery artifact,
its freshness, its file coverage or independently held decryption key. Platform's
current backup contracts distinguish Scaleway database/full archives from the
Nextcloud secondary copy. An upload-only share is not a retrieval identity.

Read-only primary-cluster metadata is retained under
`docs/evidence/2026-10-15-bootstrap-recovery/`. LLDAP, Authelia, KeyCape,
privacyIDEA, identity-provisioner and User Engine each have one Ready replica.
The three SQLite/key backup CronJobs report successful October 15 invocations.
Their current commands target local PVCs; neither an external export nor complete
off-host key/database coverage is established by those observations. The
privacyIDEA command mentions its encryption key, but not its audit key pair.
This command inspection is a coverage question, not proof that no other backup
exists. The personal-token and trigger-admin Secrets still exist. Neither their
validity nor provider-side revocation has been tested. A scan of 34 current
Deployment/StatefulSet/DaemonSet/CronJob/Pod resources in sso, mfa and user-engine
found no references to those two old Secrets; it still found the expected audit
delivery reference in User Engine. Other namespaces, historic jobs, external
consumers and issued provider credentials are outside that scan.

## Assemble the exact owner packet before execution

Record only safe references and checksums. Keep credentials in their current
owner custody and pass the decryption key through attended temporary unlock.

| Required owner input | Acceptance |
| --- | --- |
| Off-host artifact | Provider, account/folder or object/version, backup timestamp, ciphertext SHA256 and independent retrieval receipt; do not substitute a new local export for the existing off-host backup. |
| Independent key | Human confirms matching custody outside the servers and OpenBao. A successful decrypt using only the OpenBao escrow key does not prove independent recovery. Never send the key in chat. |
| Required assets | Exact manifest of matching service env/config files, LLDAP and Authelia data, privacyIDEA database plus original encryption/audit keys, and KeyCape registry/signing material. Resolve assets held through other current owner lanes explicitly. |
| Recovery environment | Named disposable target, pinned images available independently of the failed production registry, no production ingress, no production identity/client execution, bounded egress, private storage and exact cleanup scope. |
| Native proof | Owner-specific database integrity, identity/group counts, decryption of retained token material, audit signature continuity, issuer/client/signing configuration preservation and admitted login/MFA checks. Distinguish an isolated restore from current production credential validity. |
| Current consumer source | Reconcile the deployed revisions with the repaired caller/provider source. The October 14 reserved platform-group fix must be deployed and independently checked before claiming live enforcement. |
| Legacy credential disposition | IAM owner identifies the exact historical personal-token/trigger principals, actual consumers and revocation method before any deletion. Deleting a Secret does not revoke an issued provider credential. |

The operator supplied the independently held `railiance-backup` age key only to
the attended local custody helper. The actual off-host PI ciphertext decrypted
and restored successfully; the agent received no private key material. The full
identity asset manifest remains unfulfilled. The backup lane recipient and
bootstrap `keys/age.pub` must not be assumed to be the same.
Do not draft a blanket restore or grant approval while full coverage is unknown.
Once fixed, bind the source revisions, target identities, allowed operations,
recovery/cleanup sequence and maintenance window to the reviewed live packet.
Existing eligibility or another client's approval is not this ceremony's grant.

## Execute in stages

1. Use the existing Platform/provider recovery lane to retrieve the identified
   ciphertext, verify its object/version and digest, and retain a safe transfer
   receipt. Do not request or expose the provider credential to the agent.
2. The human unlocks the independently held age key in the existing custody
   helper. Validate it against the registered public recipient. No permanent
   key is installed in HOME, generated as a replacement, or copied into a bundle.
3. Run the local transport/coverage check below. It contacts no cluster, keeps
   plaintext in a private temporary directory outside Git, refuses unsafe
   archives/private age keys, tests an unrelated disposable wrong key, reads the
   restored files back and removes that staging. It emits a closed non-secret
   receipt. This is one prerequisite, not whole-service recovery acceptance.
4. Under the exact separately reviewed owner ceremony, restore the matching
   service data/key/config sets into the disposable environment and perform the
   native checks above. Never run the full production bootstrap to test source
   repairs. Unknown writes stop for reconciliation rather than automatic retry.
5. Verify cleanup, preserve the original ciphertext and independent custody,
   and reconcile current callers/legacy credentials through their owners.
   Retain exact source/object/image/actor/time and positive/negative receipts.
6. Close T02/T03 only after the recorded remaining requirements are satisfied.
   Update the workplan first, run `statehub fix-consistency --repo net-kingdom`,
   and verify the Hub projection. A transport fixture or generated evidence
   worksheet cannot mark this plan finished.

## Prepared local transport check

The human invokes this through the existing attended custody helper after the
exact artifact and required-file list are agreed. `NETKINGDOM_CUSTODY_AGE_KEY_FILE`
is the temporary private file provided inside that envelope. The tool never
reads an ambient HOME key. The receipt destination must be new.

```bash
python3 tools/bootstrap-recovery/verify-bundle.py \
  --bundle /private/recovery/fetched-ops-bundle.tar.age \
  --bundle-sha256 <independently-recorded-ciphertext-sha256> \
  --required-file lldap/secrets.env \
  --required-file privacyidea/secrets.env \
  --required-file privacyidea/pi.enc \
  --required-file privacyidea/private.pem \
  --required-file privacyidea/public.pem \
  --receipt /private/recovery/new-transport-receipt.json
```

This example file list is not an accepted whole-stack manifest. Add every
agreed service file supported by the closed transport profile; database copies
and other archive formats use their native owner restore tools. A successful
receipt explicitly leaves independent custody, current service validity and
native consumer recovery unproven. Off-host retrieval lineage and human custody
confirmation must accompany it; the tool cannot infer where a local input came
from. A failed check publishes no acceptance receipt.

## Rotation boundaries

CUST-WP-0073-T05 remains founder-deferred under decision
`1591daa1-6e85-4dfd-aadd-5a9fe1ef47fa`. This work does not reopen that rotation,
replace signing keys, change directory passwords or erase old backup keys.
Existing historical exposure, provider revocation and coordinated rotation
requirements stay explicit in their live owner records. If completion needs a
protected native operation, prepare that exact operation and obtain its scoped
disposition; never treat a source repair or successful backup job as permission.

## Knowledge/source scope

Emergent Security 0.2.6, package digest
`8d1d6a9d4d07fec7b619618de7dfe2be2254c1790f85f4198c7fe19db9751c02`:
start/root cards withheld, orientation only. Current SECURITY-COMPANION,
Platform backup/custody procedures, owner runtime contracts and the workplan
were reviewed as sources. No execution authority is derived from the cards.


## Off-host inventory return — 2026-10-15

Both bounded metadata inventories completed. Storage Share's inspected
`Backup/railiance-backups` scope includes the current privacyIDEA database
ciphertext and the October14 OpenBao snapshot. It did not reveal a matching
bootstrap/identity-file bundle in the inspected folders. Scaleway's inspected
`railiance-platform-pg-backup/platform-pg/` tree contains database backup trees
and an application archive subtree, with no identity-file bundle in this bounded
scope. Other accounts/buckets/prefixes are unassessed. See
[the discovery summary](evidence/2026-10-15-bootstrap-recovery/offsite-discovery-summary.json).
No object contents or age keys were read; the operator envelope exited0 and
self-revoked. Existing scoped backup credentials were reused without changes;
the Scaleway bootstrap API key was not used.

The operator subsequently reported the hidden-terminal check passed and the
temporary key file was removed. The later actual database drill also passed;
the private key was not provided to the agent.

## Prepared bounded privacyIDEA database drill — 2026-10-16

The identified existing off-host ciphertext was downloaded through the same
attended Platform Backup-account procedure. The before/GET/after strong-ETag
binding and exact byte count passed. Ciphertext SHA256:
`cc81da442755928171d2c51bce93eda7de4392a0a1aa9d0ffb83a25ba34e5ead`.
See `privacyidea-transfer-r1.json` in the October15 evidence folder. The owner
envelope exited0 and self-revoked; no age key or plaintext was read by the fetch.
Ciphertext is retained in the fresh private `/tmp/nk49-offhost-j9dzl0ge/` directory.

The following exact human command performs only an isolated local database
drill. It checks the transfer receipt/digest before opening the temporary key,
checks the backup recipient, refuses an unrelated wrong key, decrypts to private
temporary storage and restores into disposable PostgreSQL. Image
`postgres@sha256:ff23cdce56cac62ada6f66013e1a50864c0abbe79d132d40b6e05bd80f378a70`
is already cached locally (pg_restore16.14). There is no image pull, network,
published port, host data mount, production connection or IAM/client execution.
Database storage is a bounded temporary memory filesystem. Only fixed aggregate
counts for `token`, `resolver`, and `realm` leave the database. The container,
volumes and temporary plaintext are removed and cleanup is checked. The original
ciphertext remains for reproducibility. This proves database recovery only;
token decryption, audit signature continuity and full identity service recovery
require matching key/config files and further native owner checks.

Executed successfully on 2026-10-16 (local time). The command is retained for
provenance; its r1 receipt now exists and must not be overwritten or rerun merely
to repeat evidence. Run future reviewed drills with a new receipt. Paste the key
only into the hidden prompt, then share the success/error line or safe JSON
receipt. The receipt destination must be new; do not overwrite a failed receipt.

```bash
cd /home/worsch/net-kingdom
bash sso-mfa/bootstrap/sops-custody-unlock.sh \
  --expected-recipient age1zvryunvjhvpkmasskauga2heeg0ztnte9ymgppvjge36ekumk50syr3tsz \
  --prompt -- python3 tools/bootstrap-recovery/check-privacyidea-database.py \
  --transfer-receipt docs/evidence/2026-10-15-bootstrap-recovery/privacyidea-transfer-r1.json \
  --receipt docs/evidence/2026-10-15-bootstrap-recovery/privacyidea-database-drill-r1.json
```

Source verification:23 focused checks passed, including a real local Docker
custom-format dump, age encryption, temporary fixture-key unlock, wrong-key
refusal, native restore/readback and cleanup. Those tests use synthetic data and
fresh fixture keys. They are not evidence of actual backup decryption. If the
real drill fails, review its closed receipt; do not expose PostgreSQL/age stderr
or dump contents in chat. T02/T03 remain open.

## Actual database recovery result — 2026-10-16

The operator reported success and the actual receipt was reviewed against the
exact checker source, pinned image and fetched ciphertext SHA256. The retained
ciphertext digest was independently rechecked. Decryption, unrelated wrong-key
refusal, native PostgreSQL restore, fixed aggregate readback and container/plaintext
cleanup all passed. The restored database has five tokens, two resolvers and
two realms. The local Docker inventory contains no remaining NK49 containers.
Evidence: `privacyidea-database-drill-r1.json` and `database-drill-review.json`
in the October15 evidence directory. No production connection occurred.

This proves recovery of the selected database using the operator's independently
held key. It does not prove the recoverability of token plaintext, old audit
signatures, directory accounts, issuer keys or a full identity environment.

## Missing coverage prepared at the operator's request

The operator confirmed there is no known full identity bundle and requested
preparation of the missing coverage. The machine-readable proposal is
[`identity-backup-profile.json`](../tools/bootstrap-recovery/identity-backup-profile.json).
The read-only preparer checked the primary cluster UID, four current Deployment
UIDs/readiness/observed generations, five Bound PVC UIDs and the complete current
Secret/ConfigMap reference sets. It captured names and identities only; it did
not read Secret values or execute in pods. Its exact result is
`identity-backup-preparation-r1.json`. Fourteen tests cover foreign/stale targets,
new unreviewed dependencies, projected Secrets and private-value suppression.

| Component | Required new coverage | Required recovery proof |
| --- | --- | --- |
| LLDAP | Consistent `users.db` snapshot, remaining durable data and current credentials/configuration | User, group and membership preservation; existing-user authentication |
| Authelia | Consistent storage DB, original encryption/OIDC/session keys and current configuration | Storage decryption, signing identity and isolated authentication |
| privacyIDEA | Original enckey plus matching audit pair, configuration/scripts and a linked native database dump | Existing token decryption/MFA, old audit signature verification and new isolated audit entry |
| KeyCape | Current registry/config and signing key, policy/receipts, every referenced credential's recovery or independent reissue disposition | Issuer/JWKS identity, registry/policy continuity and positive/negative consumer acceptance |

The proposed primary prefix is
`platform-pg/application-archives/net-kingdom/`; the proposed secondary folder is
`Backup/railiance-backups/net-kingdom-identity/`. These are **not admitted**.
Platform's current generic application archive adapter only admits
`coulomb-social` and `email-connect`, and their publisher custody profiles are
application-bound. A new NetKingdom profile and scoped publisher require owner
admission; existing credentials must not be borrowed. The operator's Scaleway
bootstrap API key is unnecessary for preparation.

The initial goal is one verified generation, with a measured encrypted size and
fresh provider quota/upload-overlap checks. A proposed 512 MiB ceiling is a
refusal bound, not evidence that the data fits the secondary budget. Seven daily
and two weekly points remain a later measured proposal. No retention deletion
or recurring schedule was installed. Capture must preserve original keys and
use SQLite online-backup/PostgreSQL logical snapshots, record each capture
interval and refuse changed source identities/configuration. It must not claim
global atomicity or substitute a fresh export for the already recovered old dump.

Implementation remains: owner-admitted capture/sealing and new-profile transfer
adapters, exact credential disposition, immutable create plus retrieved readback,
then the isolated full-service recovery checks above. The current bootstrap
secrets-only tar checker cannot verify this new data-bearing profile. Approval
of preparation does not make these adapters implemented or the proposed
destinations active. The profile lists these gates explicitly; no live grant or
deployment approval is requested from this incomplete execution packet.