net-kingdom/docs/identity-service-restore-packet.md

Isolated identity service restore packet

NK-WP-0049-T15. Prepared 2026-10-16. Actual LLDAP recovery accepted at 18:37 UTC; native privacyIDEA token-material/audit recovery accepted at 19:10 UTC. Existing-token MFA, Authelia and KeyCape remain unexecuted. Accountable preparer: codex. Execution/acceptance: Railiance Platform and the LLDAP, Authelia, KeyCape and privacyIDEA service owners.

Fixed starting point

Use the actual recovered artifact and receipt in docs/evidence/2026-10-16-identity-backup-live/actual-offsite.json, with matching original keys and manifest. The completed archive recovery verified 36 file hashes, SQLite integrity, RSA pairs and native PostgreSQL restore. It did not prove service authentication or audit-signature continuity. A newer generation needs equivalent provenance and archive checks before replacing this input.

Targets and image admission still required

The bounded LLDAP stage below selects the local Linux Docker engine bnt-lap001, ID 74f93783-6cef-43a8-a3c4-f62586cd3bc1; the executor refuses a different engine or a remote Docker endpoint. All five exact image references have been saved, loaded and inspected on that workstation; receipt: docs/evidence/2026-10-16-identity-outage-recovery/offline-service-images.json. This establishes local image availability without production registry access, not an off-host image escrow or exact historical mutable-tag provenance.

No broader execution target has been admitted. Select a disposable host/context and record its immutable identity, operator, private endpoints, maximum lifetime, resource limits and exact owned cleanup inventory. Never select the production context or reuse live PVCs. Use fresh private storage; no production host mounts, service-account tokens, ingress controller or external DNS update credentials.

The October 16 read-only pod observation is recorded in docs/evidence/2026-10-16-identity-outage-recovery/image-observation.json. These are candidate inputs, not confirmation of the images in the selected archive:

Component	Candidate reference	Admission gap
PostgreSQL	postgres@sha256:ff23cdce56cac62ada6f66013e1a50864c0abbe79d132d40b6e05bd80f378a70	Passed archive DB restore; stage independently of failed registry
LLDAP	docker.io/lldap/lldap@sha256:9e605a66c02514bfcffd1b67cafb1e98d50992216bb2871d7ae44622047dd09d	Actual running digest resolved; compare selected archive/runtime lineage and stage it
Authelia	docker.io/authelia/authelia@sha256:bd97cff4fcbf715b5ff1f9ae286afbe6033afce385302520b0368122d43a6f54	Compare selected archive deployment and stage image
KeyCape	forgejo.coulomb.social/coulomb/key-cape@sha256:cdec9dd7c515f7e0a6169fc3fa560ee2cc47bf40d899755dec808c362f5ce010	Compare selected archive; require independent image availability
privacyIDEA	ghcr.io/gpappsoft/privacyidea-docker@sha256:af7841adad262f129e0c1d4f553af13f21cb2f4dc713533f316cfe43ed0b4473	Actual main and sidecar digest observed separately; compare archive/runtime lineage and stage it

Record image archive digests and offline load/inspect results. Resolve mutable references from trusted runtime/image metadata rather than pulling today's tag. The encrypted archive deployment records must be compared privately; do not export literal environment values, Secret payloads or arbitrary configs.

Restore order and containment

1. Freeze the selected artifact/manifest/image/target references. The owner supplies existing-user credentials and an existing retained MFA token through private inputs. Agree time synchronization and test identities without exposing passwords, seed material, client secrets or user records.
2. Establish an isolated network with default-deny external egress, no published ports, no host networking and no production DNS/ingress. Permit only named internal service links. The test browser, if needed, must use a separately reviewed local route and hosts mapping. Preserve issuer/client identity while keeping all test traffic private. Verify denied external connection attempts.
3. Import manifest-checked data and original keys into fresh private storage. Do not apply archived Kubernetes objects wholesale: metadata UIDs, mounts, service accounts, endpoints and sidecars require explicit translation. Keep notification/provider credentials sealed and unused; disable sending paths.
4. Restore the linked privacyIDEA database with native PostgreSQL first; restore LLDAP data/config and verify directory readiness. Run directory authentication before dependent provider checks. Restore privacyIDEA with its original enckey and audit pair against the isolated database/directory. Restore Authelia with original storage/OIDC keys and private directory routes. Restore KeyCape with original signing/config/policies and private MFA/directory dependencies.
5. Run the individual acceptance checks below; do not start provisioning, enrollment, rotation, scheduled jobs or production consumers automatically. An unsupported dependency is a named gap, never an implicit production route.
6. Remove only the named disposable containers/network/storage and temporary plaintext/key files. Confirm absence and production resources unchanged; retain sanitized receipts, original ciphertext and independently held keys.

Accepted owner step: isolated LLDAP recovery

The actual owner run passed on 2026-10-16. Evidence: docs/evidence/2026-10-16-identity-outage-recovery/actual-isolated-lldap-recovery.json and actual-isolated-lldap-review.json. All 36 archive hashes, the exact source/ciphertext/target bindings, native existing-user bind and explicit invalid-credential refusal passed. Seven users, 17 groups, 13 memberships and available attribute/schema rows were preserved; isolation and container/tmpfs cleanup passed. T16 is done. The procedure below remains for a future drill; there is no need to repeat it for this acceptance.

Execute in your own terminal, using an existing account and the password valid at the backup's capture time. No password belongs in the arguments:

bash tools/bootstrap-recovery/run-isolated-lldap.sh EXISTING_USER \
  /tmp/nk49-offline-handoff-1X660a5t/transfer.json

Replace EXISTING_USER with the selected existing username. The command asks privately for the independently held railiance-backup age key and then the account password. It rechecks ciphertext size/hash, recipient and every archive file before importing anything. Decrypted staging is in a private Linux tmpfs; the key helper removes its temporary file at exit.

The executor creates only uniquely named nk49-lldap-* containers on the fixed local engine. LLDAP has --network none, no published ports, read-only root, all capabilities dropped, no added privileges, user1000, 256 MiB memory/one CPU and private tmpfs for data/config. The client shares only that loopback network namespace and verifies there is no external interface or route. Container isolation is inspected before private import. No Kubernetes/OpenBao/SSO API, production directory endpoint or provider grant is used. Containers have a 30-minute holding lifetime and are removed on exit; plaintext and containers are checked for cleanup. A forced host/process termination can still require owner reconciliation of the named resources.

The original server key/seed, JWT secret and directory data are preserved. Admin-password reset, private-key replacement, SMTP and LDAPS listeners are disabled. The private account password travels only through memory/stdin; it is not placed in Docker arguments, environment values, logs or evidence. A native LDAP bind must pass, and an explicit LDAP invalid-credentials result must reject the wrong password. After a graceful stop, users/groups/memberships and available attribute tables are compared with the original snapshot. Only equality and aggregate counts leave the process.

The actual-image tests use disposable synthetic accounts, a real encrypted bundle and the same restore executor. They cover positive and wrong-credential paths, isolation, original-key refusal, directory preservation and cleanup. The separate actual owner receipt now supplies real-backup proof for T16. For future runs, return only the outcome and evidence directory.

This step does not restore Authelia, KeyCape or privacyIDEA, and does not prove current production password validity, MFA or audit continuity.

Acceptance matrix

Task / owner lane	Positive check	Negative check	Safe evidence
T16 / LLDAP	Preserved user/group/membership identities; existing-user bind/login	Wrong password rejected	Image/target IDs, aggregate comparisons and booleans
T17 / Authelia	Native storage decryption; same signing identity/client configs; isolated authentication	Wrong credentials or unregistered client refused	Public key thumbprint, test outcome, notification suppression
T19 / privacyIDEA MFA	Existing retained token decrypts internally and actual existing-token MFA succeeds	Wrong enckey and invalid code refused in disposable copies	Token material never exported; bounded outcomes only
T20 / privacyIDEA audit	Verify historical signatures; create and verify one isolated entry with original key	Mismatched public key/tampered disposable record rejected	Signature counts, continuity result, cleanup
T18 / KeyCape	Same issuer/JWKS, registry/policy/receipt continuity; named current consumer authenticates	Wrong secret and denied scope rejected	Public fingerprints, owner-selected test IDs and outcomes

Do not silently reset credentials or re-enroll tokens to pass. Historical audit verification failures must be reported, not hidden with replacement keys.

Accepted privacyIDEA token-material and stored-audit stage

bash tools/bootstrap-recovery/run-isolated-privacyidea.sh \
  /tmp/nk49-offline-handoff-1X660a5t/transfer.json

This uses the already independently retrieved ciphertext. The owner enters only the railiance-backup age key through the existing hidden helper; no password, MFA code or provider credential is requested. The exact local Docker target and cached images are checked first. The main executor rechecks ciphertext/recipient and all archive hashes, with private plaintext staging on Linux tmpfs.

isolated-privacyidea.py restores the linked dump into fresh PostgreSQL on --network none, listening only on its loopback interface. The pinned privacyIDEA 3.12.2 worker shares that same namespace. Both have read-only roots, all capabilities dropped, no added privileges, fixed unprivileged UIDs, no host mounts/ports and only tmpfs writable storage. Isolation is inspected before private import. The database has a 2 GiB memory limit and one GiB tmpfs data limit; the worker has a one GiB memory limit. Hold processes expire after 30 minutes (database forced termination after a further 15 seconds); normal cleanup removes both containers and plaintext staging. An interrupted host/process still requires reconciliation of the exact nk49-pi-check-* resources.

The worker uses original enckey/audit keys and the original pepper/session-secret fields. It generates a narrowly scoped private config pointing only at the isolated database, and explicitly selects that config over image defaults. It does not execute archived Python config. A literal archived PI_CHECK_OLD_SIGNATURES setting is preserved; unknown dynamic expressions are refused. No production URI or provider-admin bearer is used, no user is enrolled, and no original key is regenerated or replaced.

Native checks decrypt every nonempty encrypted token-key field internally, compare against a separate disposable wrong-key result, and emit only counts. This proves bounded native material readback, not existing-token MFA; padding and readback do not establish successful authentication. That remaining positive and wrong-code/wrong-key authentication proof stays T19. T31 tracks the separate actual token-material readback receipt.

For T20, the native audit implementation verifies every stored historical row (up to two million, otherwise refusal), reports signed/unsigned/invalid/legacy counts, then creates and verifies one isolated audit entry using the original signing identity. Tampering and an unrelated public key must fail. No historical row or signature is rewritten. This checks stored rows, not event completeness. The worker has a ten-minute execution bound; timeout is not an accepted partial scan. Counts/signatures/keys are never replaced with a sample to claim success.

Exit0 means these bounded checks and cleanup passed. Exit2 means checks completed but historical audit exceptions need owner disposition; T20 must stay open. Exit1 indicates another failure. All outcomes have a safe receipt in the printed private evidence directory. Return only that directory and the outcome. Actual owner recovery passed at 19:10 UTC on 2026-10-16: five retained token materials read internally, 279,209 historical signatures verified, no unsigned or invalid rows, new-entry verification, both negative checks and cleanup passed. T20/T31 are done. Source/ciphertext/target bindings and the safe receipt are in evidence/2026-10-16-identity-outage-recovery/actual-isolated-privacyidea-review.json. Existing-token MFA remains T19; event completeness is not claimed.

Unresolved inputs before execution

• Named immutable disposable target, approved private browser route if required, resource limits, cleanup inventory and operator window.
• Confirm archive/runtime lineage for the observed image digests and stage all images independently of production and the failed registry.
• A retained user and MFA token available in private owner custody; exact native verification commands reviewed against the chosen service versions.
• T02 current-caller ledger and T21 per-caller owner admission, including legacy token, diagnostic, resolver and LDAP-group boundaries.
• Service-owner acceptance of network/config translations and all required grants. New grants or production effects require their own owner decisions.

Until resolved, T15 stays in progress and T17–T19 remain waiting; T16, T20 and T31 are done. Production cutover, full-host/OpenBao restoration and application database recovery are separate owner procedures.