net-kingdom/docs/identity-service-restore-packet.md

# Isolated identity service restore packet

NK-WP-0049-T15. Prepared 2026-10-16. **Actual LLDAP recovery accepted at
18:37 UTC; native privacyIDEA token-material/audit recovery accepted at
19:10 UTC; native Authelia recovery accepted 2026-10-24 at 15:24 UTC; existing-token MFA accepted 2026-10-24 at 16:48 UTC. The KeyCape consumer run remains unexecuted.**
Accountable preparer: codex. Execution/acceptance: Railiance Platform and the
LLDAP, Authelia, KeyCape and privacyIDEA service owners.

## Fixed starting point

Use the actual recovered artifact and receipt in
`docs/evidence/2026-10-16-identity-backup-live/actual-offsite.json`, with matching
original keys and manifest. The completed archive recovery verified 36 file
hashes, SQLite integrity, RSA pairs and native PostgreSQL restore. It did not
prove service authentication or audit-signature continuity. A newer generation
needs equivalent provenance and archive checks before replacing this input.

## Targets and image admission still required

The bounded LLDAP stage below selects the local Linux Docker engine
`bnt-lap001`, ID `74f93783-6cef-43a8-a3c4-f62586cd3bc1`; the executor refuses a
different engine or a remote Docker endpoint. All five exact image references
have been saved, loaded and inspected on that workstation; receipt:
`docs/evidence/2026-10-16-identity-outage-recovery/offline-service-images.json`.
This establishes local image availability without production registry access,
not an off-host image escrow or exact historical mutable-tag provenance.

No broader execution target has been admitted. Select a disposable host/context and
record its immutable identity, operator, private endpoints, maximum lifetime,
resource limits and exact owned cleanup inventory. Never select the production
context or reuse live PVCs. Use fresh private storage; no production host mounts,
service-account tokens, ingress controller or external DNS update credentials.

The October 16 read-only pod observation is recorded in
`docs/evidence/2026-10-16-identity-outage-recovery/image-observation.json`.
These are candidate inputs, not confirmation of the images in the selected archive:

| Component | Candidate reference | Admission gap |
| --- | --- | --- |
| PostgreSQL | `postgres@sha256:ff23cdce56cac62ada6f66013e1a50864c0abbe79d132d40b6e05bd80f378a70` | Passed archive DB restore; stage independently of failed registry |
| LLDAP | `docker.io/lldap/lldap@sha256:9e605a66c02514bfcffd1b67cafb1e98d50992216bb2871d7ae44622047dd09d` | Actual running digest resolved; compare selected archive/runtime lineage and stage it |
| Authelia | `docker.io/authelia/authelia@sha256:bd97cff4fcbf715b5ff1f9ae286afbe6033afce385302520b0368122d43a6f54` | Compare selected archive deployment and stage image |
| KeyCape | `forgejo.coulomb.social/coulomb/key-cape@sha256:cdec9dd7c515f7e0a6169fc3fa560ee2cc47bf40d899755dec808c362f5ce010` | Compare selected archive; require independent image availability |
| privacyIDEA | `ghcr.io/gpappsoft/privacyidea-docker@sha256:af7841adad262f129e0c1d4f553af13f21cb2f4dc713533f316cfe43ed0b4473` | Actual main and sidecar digest observed separately; compare archive/runtime lineage and stage it |

Record image archive digests and offline load/inspect results. Resolve mutable
references from trusted runtime/image metadata rather than pulling today's tag.
The encrypted archive deployment records must be compared privately; do not
export literal environment values, Secret payloads or arbitrary configs.

## Restore order and containment

1. Freeze the selected artifact/manifest/image/target references. The owner
   supplies existing-user credentials and an existing retained MFA token through
   private inputs. Agree time synchronization and test identities without
   exposing passwords, seed material, client secrets or user records.
2. Establish an isolated network with default-deny external egress, no published
   ports, no host networking and no production DNS/ingress. Permit only named
   internal service links. The test browser, if needed, must use a separately
   reviewed local route and hosts mapping. Preserve issuer/client identity while
   keeping all test traffic private. Verify denied external connection attempts.
3. Import manifest-checked data and original keys into fresh private storage.
   Do not apply archived Kubernetes objects wholesale: metadata UIDs, mounts,
   service accounts, endpoints and sidecars require explicit translation. Keep
   notification/provider credentials sealed and unused; disable sending paths.
4. Restore the linked privacyIDEA database with native PostgreSQL first; restore
   LLDAP data/config and verify directory readiness. Run directory authentication
   before dependent provider checks. Restore privacyIDEA with its original enckey
   and audit pair against the isolated database/directory. Restore Authelia with
   original storage/OIDC keys and private directory routes. Restore KeyCape with
   original signing/config/policies and private MFA/directory dependencies.
5. Run the individual acceptance checks below; do not start provisioning,
   enrollment, rotation, scheduled jobs or production consumers automatically.
   An unsupported dependency is a named gap, never an implicit production route.
6. Remove only the named disposable containers/network/storage and temporary
   plaintext/key files. Confirm absence and production resources unchanged;
   retain sanitized receipts, original ciphertext and independently held keys.

## Accepted owner step: isolated LLDAP recovery

The actual owner run passed on 2026-10-16. Evidence:
`docs/evidence/2026-10-16-identity-outage-recovery/actual-isolated-lldap-recovery.json`
and `actual-isolated-lldap-review.json`. All 36 archive hashes, the exact
source/ciphertext/target bindings, native existing-user bind and explicit
invalid-credential refusal passed. Seven users, 17 groups, 13 memberships and
available attribute/schema rows were preserved; isolation and container/tmpfs
cleanup passed. T16 is done. The procedure below remains for a future drill;
there is no need to repeat it for this acceptance.

Execute in your own terminal, using an existing account and the password valid
at the backup's capture time. No password belongs in the arguments:

```bash
bash tools/bootstrap-recovery/run-isolated-lldap.sh EXISTING_USER \
  /tmp/nk49-offline-handoff-1X660a5t/transfer.json
```

Replace `EXISTING_USER` with the selected existing username. The command asks
privately for the independently held `railiance-backup` age key and then the
account password. It rechecks ciphertext size/hash, recipient and every archive
file before importing anything. Decrypted staging is in a private Linux tmpfs;
the key helper removes its temporary file at exit.

The executor creates only uniquely named `nk49-lldap-*` containers on the fixed
local engine. LLDAP has `--network none`, no published ports, read-only root,
all capabilities dropped, no added privileges, user1000, 256 MiB memory/one CPU
and private tmpfs for data/config. The client shares only that loopback network
namespace and verifies there is no external interface or route. Container
isolation is inspected before private import. No Kubernetes/OpenBao/SSO API,
production directory endpoint or provider grant is used. Containers have a
30-minute holding lifetime and are removed on exit; plaintext and containers
are checked for cleanup. A forced host/process termination can still require
owner reconciliation of the named resources.

The original server key/seed, JWT secret and directory data are preserved.
Admin-password reset, private-key replacement, SMTP and LDAPS listeners are
disabled. The private account password travels only through memory/stdin; it is
not placed in Docker arguments, environment values, logs or evidence. A native
LDAP bind must pass, and an explicit LDAP invalid-credentials result must reject
the wrong password. After a graceful stop, users/groups/memberships and available
attribute tables are compared with the original snapshot. Only equality and
aggregate counts leave the process.

The actual-image tests use disposable synthetic accounts, a real encrypted
bundle and the same restore executor. They cover positive and wrong-credential
paths, isolation, original-key refusal, directory preservation and cleanup.
The separate actual owner receipt now supplies real-backup proof for T16.
For future runs, return only the outcome and evidence directory.

This step does not restore Authelia, KeyCape or privacyIDEA, and does not prove
current production password validity, MFA or audit continuity.

## Prepared owner step: isolated Authelia recovery (T17)

Prepared and fixture-tested 2026-10-24 under T41. The actual owner run passed
the same day (15:24 UTC); evidence is in
`docs/evidence/2026-10-24-nk49-t17-authelia/`. T17 is done. Execute in your own terminal with a verified transfer receipt from an
independent retrieval (the T23 procedure), an existing account and the password
valid at the backup's capture time:

```bash
bash tools/bootstrap-recovery/run-isolated-authelia.sh EXISTING_USER \
  /path/to/verified/transfer.json
```

The command asks privately for the `railiance-backup` age key and then the
password. It rechecks ciphertext size/hash, recipient and every archive file,
and decrypts into private Linux tmpfs, as the LLDAP step does.

Topology: restored LLDAP runs as the `--network none` anchor. Authelia and the
probe client join its network namespace (`container:<anchor>`) and use loopback
only. All containers are `nk49-authelia-*`, read-only, capabilities dropped, no
added privileges, user 1000, tmpfs-only mounts, no published ports. Isolation
is inspected before private import; the probe verifies there is no external
interface or route.

The archived deployment, ConfigMap and `authelia-secrets` are used as captured.
Secret files are placed at their archived mount paths; the environment carries
only file paths. Four confinement rewrites are made, and only these: the LDAP
address points to the restored directory on loopback; the SMTP notifier becomes
a tmpfs file notifier and `authelia-smtp` is not imported; the NTP startup check
is off (outbound UDP); the container healthcheck file is off (the root stays
read-only). KeyCape's archived `keycape-config` supplies the client id, client
secret and the portal hostname.

Checks, in order: `authelia validate-config` on the original configuration;
`authelia storage encryption check` with the original key; the issuer JWKS
served by Authelia contains the archived issuer public key; existing-user
first factor through `/api/firstfactor` against the restored LLDAP; a wrong
password refused; the archived `keycape` client secret passes client
authentication at `/api/oidc/token` (`invalid_grant` for a deliberately invalid
code, so no grant is issued); a wrong client secret refused (`invalid_client`).
The receipt carries booleans, the RFC 7638 issuer key thumbprint, image IDs and
source hashes only. Containers and tmpfs staging are removed and checked on
every exit.

Native fixture tests (`NK49_RUN_ISOLATED_LLDAP_TEST=1
NK49_RUN_ISOLATED_AUTHELIA_TEST=1 pytest tests/test_isolated_authelia_recovery.py`)
use this repository's own Authelia manifests with synthetic secrets on the
pinned images. They cover the positive path, wrong password, wrong storage key
and an encrypted-bundle run through `main()`.

This step does not prove the full KeyCape→Authelia→LLDAP→privacyIDEA login,
MFA, or current production password validity. It reuses `isolated-lldap.py`
and the bundle modules unedited, so the pinned KeyCape attempt sources stay as
reviewed.

## Acceptance matrix

| Task / owner lane | Positive check | Negative check | Safe evidence |
| --- | --- | --- | --- |
| T16 / LLDAP | Preserved user/group/membership identities; existing-user bind/login | Wrong password rejected | Image/target IDs, aggregate comparisons and booleans |
| T17 / Authelia | Original config validates; storage key opens store; issuer JWKS equals archived key; `keycape` client verifier accepts KeyCape's archived secret; existing-user first factor against restored LLDAP | Wrong password and wrong client secret refused; no grant issued | Public key thumbprint, outcomes, file-notifier only (stage prepared in T41) |
| T19 / privacyIDEA MFA | Existing retained token decrypts internally and actual existing-token MFA succeeds | Wrong enckey and invalid code refused in disposable copies | Token material never exported; bounded outcomes only |
| T20 / privacyIDEA audit | Verify historical signatures; create and verify one isolated entry with original key | Mismatched public key/tampered disposable record rejected | Signature counts, continuity result, cleanup |
| T18 / KeyCape | Same issuer/JWKS, registry/policy/receipt continuity; named current consumer authenticates | Wrong secret and denied scope rejected | Public fingerprints, owner-selected test IDs and outcomes |

Do not silently reset credentials or re-enroll tokens to pass. Historical audit
verification failures must be reported, not hidden with replacement keys.

## Accepted privacyIDEA token-material and stored-audit stage

```bash
bash tools/bootstrap-recovery/run-isolated-privacyidea.sh \
  /tmp/nk49-offline-handoff-1X660a5t/transfer.json
```

This uses the already independently retrieved ciphertext. The owner enters only
the `railiance-backup` age key through the existing hidden helper; no password,
MFA code or provider credential is requested. The exact local Docker target and
cached images are checked first. The main executor rechecks ciphertext/recipient
and all archive hashes, with private plaintext staging on Linux tmpfs.

`isolated-privacyidea.py` restores the linked dump into fresh PostgreSQL on
`--network none`, listening only on its loopback interface. The pinned
privacyIDEA 3.12.2 worker shares that same namespace. Both have read-only roots,
all capabilities dropped, no added privileges, fixed unprivileged UIDs, no host
mounts/ports and only tmpfs writable storage. Isolation is inspected before
private import. The database has a 2 GiB memory limit and one GiB tmpfs data
limit; the worker has a one GiB memory limit. Hold processes expire after 30
minutes (database forced termination after a further 15 seconds); normal cleanup
removes both containers and plaintext staging. An interrupted host/process still
requires reconciliation of the exact `nk49-pi-check-*` resources.

The worker uses original enckey/audit keys and the original pepper/session-secret
fields. It generates a narrowly scoped private config pointing only at the
isolated database, and explicitly selects that config over image defaults.
It does not execute archived Python config. A literal archived
`PI_CHECK_OLD_SIGNATURES` setting is preserved; unknown dynamic expressions are
refused. No production URI or provider-admin bearer is used, no user is enrolled,
and no original key is regenerated or replaced.

Native checks decrypt every nonempty encrypted token-key field internally,
compare against a separate disposable wrong-key result, and emit only counts.
This proves bounded native material readback, **not existing-token MFA**; padding
and readback do not establish successful authentication. That remaining positive
and wrong-code/wrong-key authentication proof stays T19. T31 tracks the separate
actual token-material readback receipt.

For T20, the native audit implementation verifies every stored historical row
(up to two million, otherwise refusal), reports signed/unsigned/invalid/legacy
counts, then creates and verifies one isolated audit entry using the original
signing identity. Tampering and an unrelated public key must fail. No historical
row or signature is rewritten. This checks stored rows, not event completeness.
The worker has a ten-minute execution bound; timeout is not an accepted partial
scan. Counts/signatures/keys are never replaced with a sample to claim success.

Exit0 means these bounded checks and cleanup passed. Exit2 means checks completed
but historical audit exceptions need owner disposition; T20 must stay open.
Exit1 indicates another failure. All outcomes have a safe receipt in the printed
private evidence directory. Return only that directory and the outcome. Actual owner recovery passed at 19:10 UTC on 2026-10-16: five retained token
materials read internally, 279,209 historical signatures verified, no unsigned or
invalid rows, new-entry verification, both negative checks and cleanup passed.
T20/T31 are done. Source/ciphertext/target bindings and the safe receipt are in
`evidence/2026-10-16-identity-outage-recovery/actual-isolated-privacyidea-review.json`.
Existing-token MFA remains T19; event completeness is not claimed.

## Prepared owner step: isolated existing-token MFA (T19)

Prepared and fixture-tested 2026-10-24 under T42. The actual owner run passed the
same day (16:48 UTC); evidence is in `docs/evidence/2026-10-24-nk49-t19-privacyidea-mfa/`.
T19 is done. The procedure below remains for a future drill. Execute in your own terminal with a verified transfer receipt (the T23
procedure) and your authenticator ready:

```bash
bash tools/bootstrap-recovery/run-isolated-privacyidea-mfa.sh /path/to/verified/transfer.json
```

Prompts, in order: the `railiance-backup` age key (hidden); your login name (and
the realm, if more than one exists); then a token PIN plus current code exactly as
typed at login (hidden). Any one of your usable tokens works. A TOTP code is valid
for tens of seconds, so enter it only when asked and keep the authenticator open.

**Why by user.** A first version checked a token by serial. On the real backup every
check stopped with `ResolverError`: real tokens belong to an LDAP user, and the
archived resolver `lldap-coulomb` points at the in-cluster address
`ldap://lldap.sso.svc.cluster.local:3890`, which does not exist in isolation. Fixture
tokens had no owner, so tests had not shown it, and the run correctly claimed no proof.

**Topology.** The restored database container is the `--network none` anchor.
Restored LLDAP and the privacyIDEA worker join its namespace (`container:<anchor>`)
and talk over loopback only. All three are read-only, capabilities dropped, user
non-root, tmpfs-only, with isolation inspected before private import.

**Resolver rewrite, disposable copy only.** After `pg_restore`, every `ldapresolver`
`LDAPURI` must equal the archived in-cluster address exactly, or the run is refused
as an owner decision. They are then repointed to `ldap://127.0.0.1:3890`. The stored
bind password is not touched, so the original encryption key must decrypt it and the
restored LLDAP must accept it before any user can be looked up.

**Checks.** The call is `POST /validate/check` with user, realm and pass, through
privacyIDEA's own application, so stored policies and the resolver apply as for KeyCape:

1. The user is found through the restored resolver and directory, and has a usable token.
2. **Wrong key.** The application is pointed at a random key file (the run asserts the
   configuration really names it). The typed code must be refused; privacyIDEA returns
   its own error 907.
3. **Wrong code.** The last digit of what you typed is changed; refused as a wrong value.
4. **Your code.** Accepted with the original key and the retained token material.

The order is deliberate: only the last step can spend a counter or replay-protected
code. An earlier fixture run in which the wrong-key step silently used the original key
showed why; the configuration assertion now prevents that vacuous result, and the
receipt claims proof only when all three outcomes line up.

The receipt carries booleans, bounded refusal categories (for example `error_907`,
`wrong_value`), token types, counts, a SHA-256 of the login and source hashes. No login,
PIN or code is recorded, and the disposable copy is destroyed. Fixture tests cover HOTP
and TOTP-with-PIN owned by an LDAP user, a wrong code alone, an unknown user, a malformed
original key, the rewrite refusing an unknown address, and an encrypted bundle through
`main()`.

**Limits.** Policies that call outside the namespace (for example event handlers that
send notifications) cannot succeed and are not exercised. Token ownership is checked for
the one user you name, not for every user.

## Unresolved inputs before execution

- Named immutable disposable target, approved private browser route if required,
  resource limits, cleanup inventory and operator window.
- Confirm archive/runtime lineage for the observed image digests and stage all
  images independently of production and the failed registry.
- A retained user and MFA token available in private owner custody; exact native
  verification commands reviewed against the chosen service versions.
- T21 per-caller owner admission, including legacy token, diagnostic, resolver
  and LDAP-group boundaries. The T02 current-caller ledger is delivered
  ([caller-ledger.json](evidence/2026-10-23-nk49-t02-caller-ledger/caller-ledger.json),
  2026-10-23). Submit the acceptance from its `current` rows. Its two historical
  Secrets (`sso/keycape-pi-token`, `mfa/privacyidea-trigger-admin`) are excluded
  from restore scope: they are retired in source, with revocation routed to T22.
- Service-owner acceptance of network/config translations and all required
  grants. New grants or production effects require their own owner decisions.

Until resolved, T15 stays in progress and T18 remains waiting; T16, T17, T19, T20 and T31 are done. Production
cutover, full-host/OpenBao restoration and application database recovery are
separate owner procedures.