net-kingdom/docs/identity-service-restore-packet.md

# Isolated identity service restore packet

NK-WP-0049-T15. Prepared 2026-10-16. **LLDAP stage ready for the private
operator run; remaining service stages are draft and unexecuted.**
Accountable preparer: codex. Execution/acceptance: Railiance Platform and the
LLDAP, Authelia, KeyCape and privacyIDEA service owners.

## Fixed starting point

Use the actual recovered artifact and receipt in
`docs/evidence/2026-10-16-identity-backup-live/actual-offsite.json`, with matching
original keys and manifest. The completed archive recovery verified 36 file
hashes, SQLite integrity, RSA pairs and native PostgreSQL restore. It did not
prove service authentication or audit-signature continuity. A newer generation
needs equivalent provenance and archive checks before replacing this input.

## Targets and image admission still required

The bounded LLDAP stage below selects the local Linux Docker engine
`bnt-lap001`, ID `74f93783-6cef-43a8-a3c4-f62586cd3bc1`; the executor refuses a
different engine or a remote Docker endpoint. All five exact image references
have been saved, loaded and inspected on that workstation; receipt:
`docs/evidence/2026-10-16-identity-outage-recovery/offline-service-images.json`.
This establishes local image availability without production registry access,
not an off-host image escrow or exact historical mutable-tag provenance.

No broader execution target has been admitted. Select a disposable host/context and
record its immutable identity, operator, private endpoints, maximum lifetime,
resource limits and exact owned cleanup inventory. Never select the production
context or reuse live PVCs. Use fresh private storage; no production host mounts,
service-account tokens, ingress controller or external DNS update credentials.

The October 16 read-only pod observation is recorded in
`docs/evidence/2026-10-16-identity-outage-recovery/image-observation.json`.
These are candidate inputs, not confirmation of the images in the selected archive:

| Component | Candidate reference | Admission gap |
| --- | --- | --- |
| PostgreSQL | `postgres@sha256:ff23cdce56cac62ada6f66013e1a50864c0abbe79d132d40b6e05bd80f378a70` | Passed archive DB restore; stage independently of failed registry |
| LLDAP | `docker.io/lldap/lldap@sha256:9e605a66c02514bfcffd1b67cafb1e98d50992216bb2871d7ae44622047dd09d` | Actual running digest resolved; compare selected archive/runtime lineage and stage it |
| Authelia | `docker.io/authelia/authelia@sha256:bd97cff4fcbf715b5ff1f9ae286afbe6033afce385302520b0368122d43a6f54` | Compare selected archive deployment and stage image |
| KeyCape | `forgejo.coulomb.social/coulomb/key-cape@sha256:cdec9dd7c515f7e0a6169fc3fa560ee2cc47bf40d899755dec808c362f5ce010` | Compare selected archive; require independent image availability |
| privacyIDEA | `ghcr.io/gpappsoft/privacyidea-docker@sha256:af7841adad262f129e0c1d4f553af13f21cb2f4dc713533f316cfe43ed0b4473` | Actual main and sidecar digest observed separately; compare archive/runtime lineage and stage it |

Record image archive digests and offline load/inspect results. Resolve mutable
references from trusted runtime/image metadata rather than pulling today's tag.
The encrypted archive deployment records must be compared privately; do not
export literal environment values, Secret payloads or arbitrary configs.

## Restore order and containment

1. Freeze the selected artifact/manifest/image/target references. The owner
   supplies existing-user credentials and an existing retained MFA token through
   private inputs. Agree time synchronization and test identities without
   exposing passwords, seed material, client secrets or user records.
2. Establish an isolated network with default-deny external egress, no published
   ports, no host networking and no production DNS/ingress. Permit only named
   internal service links. The test browser, if needed, must use a separately
   reviewed local route and hosts mapping. Preserve issuer/client identity while
   keeping all test traffic private. Verify denied external connection attempts.
3. Import manifest-checked data and original keys into fresh private storage.
   Do not apply archived Kubernetes objects wholesale: metadata UIDs, mounts,
   service accounts, endpoints and sidecars require explicit translation. Keep
   notification/provider credentials sealed and unused; disable sending paths.
4. Restore the linked privacyIDEA database with native PostgreSQL first; restore
   LLDAP data/config and verify directory readiness. Run directory authentication
   before dependent provider checks. Restore privacyIDEA with its original enckey
   and audit pair against the isolated database/directory. Restore Authelia with
   original storage/OIDC keys and private directory routes. Restore KeyCape with
   original signing/config/policies and private MFA/directory dependencies.
5. Run the individual acceptance checks below; do not start provisioning,
   enrollment, rotation, scheduled jobs or production consumers automatically.
   An unsupported dependency is a named gap, never an implicit production route.
6. Remove only the named disposable containers/network/storage and temporary
   plaintext/key files. Confirm absence and production resources unchanged;
   retain sanitized receipts, original ciphertext and independently held keys.

## Ready owner step: isolated LLDAP recovery

Execute in your own terminal, using an existing account and the password valid
at the backup's capture time. No password belongs in the arguments:

```bash
bash tools/bootstrap-recovery/run-isolated-lldap.sh EXISTING_USER \
  /tmp/nk49-offline-handoff-1X660a5t/transfer.json
```

Replace `EXISTING_USER` with the selected existing username. The command asks
privately for the independently held `railiance-backup` age key and then the
account password. It rechecks ciphertext size/hash, recipient and every archive
file before importing anything. Decrypted staging is in a private Linux tmpfs;
the key helper removes its temporary file at exit.

The executor creates only uniquely named `nk49-lldap-*` containers on the fixed
local engine. LLDAP has `--network none`, no published ports, read-only root,
all capabilities dropped, no added privileges, user1000, 256 MiB memory/one CPU
and private tmpfs for data/config. The client shares only that loopback network
namespace and verifies there is no external interface or route. Container
isolation is inspected before private import. No Kubernetes/OpenBao/SSO API,
production directory endpoint or provider grant is used. Containers have a
30-minute holding lifetime and are removed on exit; plaintext and containers
are checked for cleanup. A forced host/process termination can still require
owner reconciliation of the named resources.

The original server key/seed, JWT secret and directory data are preserved.
Admin-password reset, private-key replacement, SMTP and LDAPS listeners are
disabled. The private account password travels only through memory/stdin; it is
not placed in Docker arguments, environment values, logs or evidence. A native
LDAP bind must pass, and an explicit LDAP invalid-credentials result must reject
the wrong password. After a graceful stop, users/groups/memberships and available
attribute tables are compared with the original snapshot. Only equality and
aggregate counts leave the process.

The actual-image tests use disposable synthetic accounts, a real encrypted
bundle and the same restore executor. They cover positive and wrong-credential
paths, isolation, original-key refusal, directory preservation and cleanup.
They are not the real-backup proof: T16 remains waiting for the owner's private
run and safe receipt. Return only the outcome and evidence directory.

This step does not restore Authelia, KeyCape or privacyIDEA, and does not prove
current production password validity, MFA or audit continuity.

## Acceptance matrix

| Task / owner lane | Positive check | Negative check | Safe evidence |
| --- | --- | --- | --- |
| T16 / LLDAP | Preserved user/group/membership identities; existing-user bind/login | Wrong password rejected | Image/target IDs, aggregate comparisons and booleans |
| T17 / Authelia | Native storage decryption; same signing identity/client configs; isolated authentication | Wrong credentials or unregistered client refused | Public key thumbprint, test outcome, notification suppression |
| T19 / privacyIDEA MFA | Existing retained token decrypts internally and actual existing-token MFA succeeds | Wrong enckey and invalid code refused in disposable copies | Token material never exported; bounded outcomes only |
| T20 / privacyIDEA audit | Verify historical signatures; create and verify one isolated entry with original key | Mismatched public key/tampered disposable record rejected | Signature counts, continuity result, cleanup |
| T18 / KeyCape | Same issuer/JWKS, registry/policy/receipt continuity; named current consumer authenticates | Wrong secret and denied scope rejected | Public fingerprints, owner-selected test IDs and outcomes |

Do not silently reset credentials or re-enroll tokens to pass. Historical audit
verification failures must be reported, not hidden with replacement keys.

## Unresolved inputs before execution

- Named immutable disposable target, approved private browser route if required,
  resource limits, cleanup inventory and operator window.
- Confirm archive/runtime lineage for the observed image digests and stage all
  images independently of production and the failed registry.
- A retained user and MFA token available in private owner custody; exact native
  verification commands reviewed against the chosen service versions.
- T02 current-caller ledger and T21 per-caller owner admission, including legacy
  token, diagnostic, resolver and LDAP-group boundaries.
- Service-owner acceptance of network/config translations and all required
  grants. New grants or production effects require their own owner decisions.

Until resolved, T15 stays in progress and T16–T20 remain waiting. Production
cutover, full-host/OpenBao restoration and application database recovery are
separate owner procedures.