net-kingdom/workplans/NK-WP-0049-bootstrap-secret-free-recovery-handoff.md

id	type	title	domain	repo	status	owner	topic_slug	created	updated	flavor	depends_on	state_hub_workstream_id
NK-WP-0049	workplan	Remove bootstrap secret output and verify separated recovery custody	infotech	net-kingdom	active	codex	netkingdom	2026-10-12	2026-10-16	implementation		17d315f0-829e-57c7-85cc-3a7914152804

Native owner work for CUST-WP-0080 G08. This workplan remains active until the remaining caller, service-recovery and custody acceptance gates are satisfied. NK-WP-0045 covers the non-secret root-grant record. Founder-deferred exposure rotation remains in CUST-WP-0073-T05; this plan does not silently reopen it.

Replanned 2026-10-16 at the operator's request: substantial completed work is now represented by completed tasks, and remaining work has separate deliverables, dependencies and acceptance criteria. T01–T03 retain their historical IDs and Hub UUIDs. Completed parts of the old T02/T03 umbrellas are extracted into T04–T14; T02 now covers the current-caller ledger and T03 retains final owner acceptance. These are retrospective task records of evidenced work, not additional delivery or newly granted authority. The earlier dated narrative remains in Git at cc2dd01; its evidence references are carried into the tasks below.

Current position: 22 tasks done; 2 in progress; 1 ready to start (todo); 6 waiting on explicit prerequisites. These are task counts, not an effort-based completion percentage. Assignees are also written in task descriptions because the current Forge projector does not populate the dedicated Hub assignee field. The outage-retrieval tool and restore-packet draft are now prepared; independent outage retrieval and Storage Share publication are accepted; service admission remains an explicit gate.

Next actions: finish the read-only caller ledger (T02); admit the drafted isolated restore packet (T15) and record residual-scope handoffs (T24). Independent outage retrieval (T23) and guide publication (T26) are done. After owner admission, execute the individual native service and caller checks. A task's owner is accountable for the result; another repository's approval or credential grant still requires its own owner path.

Closure: T03 assembles owner acceptance after its named dependencies pass. Passing archive recovery does not prove existing-user authentication, retained MFA-token decryption, historical audit signatures or current caller admission. The verified backup job stays active while those separate tasks are completed.

Remove secret-bearing output and ambient root discovery

id: NK-WP-0049-T01
status: done
priority: high
assignee: codex
state_hub_task_id: "d7ca8d5f-c5ff-5cba-bd25-038cb53ffe4e"

Accountable worker: codex. Service/credential-owner acceptance remains with the named owner boundary.

2026-10-12: regression reproduced emergency/reprint root export, generation prefix disclosure, permissive file modes, dry-run side effects and unattended entry. Replaced emergency export with a non-secret guide (legacy secret-input arguments refused), suppressed generation prefixes/tracing and set private permissions. Bootstrap requires the registered public recipient and an attended terminal; never generates/discovers a permanent private key. Encrypted resume requires an explicit matching temporary custody key before cluster activity. Dry-run is a non-mutating plan. The live cluster/apply path was not executed.

A fresh uniquely named encrypted ops bundle follows apply; old delivery flags cannot skip current acknowledgment. New acknowledgment state binds its SHA256 and timestamp. Declining stops before cleanup/completion. This is an operator statement, not an independent restore receipt. Historical fields remain unchanged. Make/help/status/console and current credential guidance reflect that distinction. Evidence and exact source manifest: Custodian docs/evidence/2026-10-12-netkingdom-security/bootstrap-handoff.json.

Harden custody, SOPS and rotation source entry points

id: NK-WP-0049-T04
status: done
priority: high
assignee: codex
depends_on: [NK-WP-0049-T01]
state_hub_task_id: "04c4e8be-5bd8-5d5c-99b1-79d8ae758462"

Accountable worker: codex. Service/credential-owner acceptance remains with the named owner boundary.

Completed 2026-10-12–13. Public-recipient-only encryption, explicit temporary custody, private no-overwrite output, sanitized diagnostics, stdin-only secret transport and failed-operation cleanup replace ambient-key and output paths. SOPS read/edit/data-key rotation preserve recipient/MAC/source-version guards; component rotations preserve unrelated fields and retain recovery candidates. This records source and disposable-native proof, not live rotation or revocation.

Evidence: Custodian docs/evidence/2026-10-12-netkingdom-security/ receipts custody-entrypoints.json, sops-make-entrypoints.json, rotation-output.json, and admin-bootstrap.json; October13 sops-write-callers.json.

Repair component, diagnostic, LDAP and resolver callers

id: NK-WP-0049-T05
status: done
priority: high
assignee: codex
depends_on: [NK-WP-0049-T04]
state_hub_task_id: "e8e38f42-609e-5c07-a8e6-e73c8eb4ae95"

Accountable worker: codex. Service/credential-owner acceptance remains with the named owner boundary.

Completed 2026-10-13–14. Repaired KeyCape configuration and component Secret publication, original privacyIDEA key export/restore transport, scoped MFA counts, LDAP group maintenance and configuration-preserving resolver cutover. Retired unsafe personal-token mint/refresh and direct LDAP lifecycle shortcuts. Disposable native PI/LLDAP tests and source regressions passed; estate provider admission, historical credential validity and service authentication remain T21/T22.

Evidence: Custodian October13 keycape-caller-reconciliation.json, service-secret-callers.json, privacyidea-key-custody.json, mfa-state-diagnostic.json, realm-bootstrap-repair.json; October14 legacy-lldap-retirement.json, platform-group-boundary.json, lldap-group-maintenance.json, resolver-cutover.json under the corresponding docs/evidence/YYYY-MM-DD-netkingdom-security/ directories.

Inventory source callers and retire remaining unsafe delivery paths

id: NK-WP-0049-T06
status: done
priority: high
assignee: codex
depends_on: [NK-WP-0049-T04, NK-WP-0049-T05]
state_hub_task_id: "a5f50891-c5b2-513b-8670-e1915ee52015"

Accountable worker: codex. Service/credential-owner acceptance remains with the named owner boundary.

Completed 2026-10-15. Enumerated all 87 current non-test shell/Python callers in the declared bootstrap/k8s scope plus Make/console/delivery adapters. Repaired streamed encrypted bundle publication and fail-closed bootstrap/lab paths; retired unsafe delivery copiers, root prompts, registry patchers and stale rollback recipes. The attended inventory records source dispositions and their native-proof limits. T02 separately maps these to current live users/owners.

Evidence: caller inventory, attended procedures and the October15 recovery evidence directory. No live production bootstrap or lab-init acceptance is claimed.

Discover off-host coverage and recover the existing PI database

id: NK-WP-0049-T07
status: done
priority: high
assignee: codex
state_hub_task_id: "2d7302a2-319a-5633-bf58-d51ccd25ebf6"

Accountable worker: codex. Service/credential-owner acceptance remains with the named owner boundary.

Completed 2026-10-15–16. Bounded Storage Share/Scaleway inventories identified existing database coverage and the missing full identity bundle. The operator independently matched the backup recipient, then recovered the exact existing PI dump: correct-key decryption, wrong-key refusal, isolated native PostgreSQL restore/readback and cleanup passed (5 tokens, 2 resolvers, 2 realms).

Evidence: October15 recovery offsite-discovery-summary.json, privacyidea-transfer-r1.json, privacyidea-database-drill-r1.json, database-drill-review.json. This database-only proof led to T08–T14.

Implement the full identity capture and archive recovery toolchain

id: NK-WP-0049-T08
status: done
priority: high
assignee: codex
depends_on: [NK-WP-0049-T06, NK-WP-0049-T07]
state_hub_task_id: "317ca173-c784-5329-9810-5e3d300604b8"

Accountable worker: codex. Service/credential-owner acceptance remains with the named owner boundary.

Completed 2026-10-16. Defined exact LLDAP/Authelia/KeyCape/privacyIDEA assets, Deployment/PVC/reference bindings, original keys and snapshot consistency limits. Implemented native SQLite/PG capture, streamed age sealing, strict data-bearing manifest checks, provider readback, independent retrieval, bounded retention, root runtime and attended archive verifier. All 56 focused checks passed, including actual synthetic encrypted/native PostgreSQL recovery.

Evidence: profile, implementation review, and the live evidence's tested ESO-default/host-binding corrections. Production archive recovery is separately evidenced in T13.

Admit and verify the two scoped backup credential deliveries

id: NK-WP-0049-T09
status: done
priority: high
assignee: codex
depends_on: [NK-WP-0049-T08]
state_hub_task_id: "4a47eb2a-7e42-5ae7-9df1-f5ede746eca1"

Accountable worker: codex. Service/credential-owner acceptance remains with the named owner boundary.

Completed 2026-10-16. Platform CCR-2026-0048/0049 are verified. The operator approved folder-only provider-normalized publisher21/retention25 rights, including re-sharing, in addition to the original rollout. Coding-agent denials precede KV publication. Exact 900-second single-policy readers, wrong-SA/namespace/audience rejection, metadata/sibling/old-escrow/token-create denials, ESO equality and reader/session revocation passed. Operator passwords and age private keys are not job credentials.

Evidence: live publisher-setup-verified.json, retention-setup-verified.json, publisher-native-delivery-verified.json, retention-native-delivery-verified.json; approval decisions 40bfae40-6108-4e51-8613-073339f5345f and 5edd5203-fae7-461b-8775-5d22b7873971; Platform CCR files.

Apply capture isolation and install the guarded node runtime

id: NK-WP-0049-T10
status: done
priority: high
assignee: codex
depends_on: [NK-WP-0049-T08, NK-WP-0049-T09]
state_hub_task_id: "0519444c-d559-5455-aa27-8c5dc8b832b0"

Accountable worker: codex. Service/credential-owner acceptance remains with the named owner boundary.

Completed 2026-10-16. Applied exact UID/resource-version/spec-guarded DNS selector exclusions, preserving ordinary service matches. Capture refuses any remaining helper allow policy. Installed immutable runtime sources bound to the actual hostname, machine identity and cluster UID. Disabled-unit replacement checked previous contents and retained rollback bytes. Health unit validation passed. Activation remained gated until T13.

Evidence: live isolation-applied.json, capture-ready-plan.json, installed-runtime-packet-r2.json, upgrade-disabled-runtime.py. Actual complete hardened-job execution is recorded in T14.

Connect recovery to Railiance bootstrap and reuse discovery

id: NK-WP-0049-T11
status: done
priority: high
assignee: codex
depends_on: [NK-WP-0049-T08]
state_hub_task_id: "d69af493-4026-56a8-a526-c267d90dce18"

Accountable worker: codex. Service/credential-owner acceptance remains with the named owner boundary.

Completed 2026-10-16. Added the Railiance bootstrap identity-recovery dispatcher and independent-custody → isolated verification → layer/service-owner handoff. Updated capability.security.bootstrap-recovery, linked bootstrap sequencing and composed reuse-surface federation. Keyword discovery for identity backup, restore, privacyIDEA and disaster recovery was verified. The operator actually used this handoff in T13. No unattended production importer is claimed.

Evidence: ceremony, owning capability entry, railiance-bootstrap/docs/identity-recovery.md, bootstrap commit d1dcfde and reuse-surface federation commit 662782e.

Create and verify the first full off-host identity bundle

id: NK-WP-0049-T12
status: done
priority: high
assignee: codex
depends_on: [NK-WP-0049-T09, NK-WP-0049-T10]
state_hub_task_id: "decb415a-8012-521c-9dbe-d73c6ff0bbfb"

Accountable worker: codex. Service/credential-owner acceptance remains with the named owner boundary.

Completed 2026-10-16. Captured original service keys/current data plus the linked native PI database dump, sealed and published a unique Storage Share artifact, then downloaded it and verified size/digest before publishing its index. Helper and plaintext cleanup passed. Artifact: net-kingdom-identity-20261016T063159Z-5882771aa2ce.tar.age (109738344 bytes).

Evidence: live actual-capture.json, actual-offsite.json; ciphertext SHA256 7117b3fe1e4840c86b301c94c85e6b16709e5626bb0884bd0580bde630e30e30. No age private key was needed by the capture job.

Independently retrieve and recover the complete actual archive

id: NK-WP-0049-T13
status: done
priority: high
assignee: codex
depends_on: [NK-WP-0049-T11, NK-WP-0049-T12]
state_hub_task_id: "602d4921-cfac-5e5b-9993-4fbf67195c7d"

Accountable worker: codex. Service/credential-owner acceptance remains with the named owner boundary.

Completed 2026-10-16. Independently downloaded the exact first generation through attended Backup-account custody, rehashed it locally and revoked the session. The operator used the independently held age key through the bootstrap handoff. Correct/wrong-key cases, all 36 manifest files/hashes, original RSA pairs, SQLite integrity and isolated native PG16.14 restore/readback passed; plaintext and containers were removed. No key or restored secret values reached the agent.

Evidence: live independent-owner-transfer.json, actual-full-archive-recovery.json, actual-full-recovery-review.json. Existing-user/MFA authentication and old audit signatures are deliberately separate T16–T20 outcomes.

Activate and prove the complete recurring backup and retention job

id: NK-WP-0049-T14
status: done
priority: high
assignee: codex
depends_on: [NK-WP-0049-T10, NK-WP-0049-T13]
state_hub_task_id: "eaf0708f-b3b7-5531-8172-95edf30c3a19"

Accountable worker: codex. Service/credential-owner acceptance remains with the named owner boundary.

Completed 2026-10-16 after T13. Both daily backup/retention and hourly health timers are enabled/active. The actual hardened systemd service passed retention before/after a second capture/upload/download (Result=success, ExecMainStatus=0). Both verified generations are retained, the independently recovered point is pinned, and nothing was deleted. Helpers/plaintext were cleaned; failure markers and reporting outbox are empty. Health check succeeded.

Evidence: live activation-result.json, activation-review.json, recurring-job-review.json. Cadence: daily 03:45 UTC plus up to 5 minutes jitter, hourly health, 25-hour freshness bound. Second-provider/host-loss gaps go to T24; this completed task is not a full-estate recovery claim.

Map current callers, owners and native acceptance gaps

id: NK-WP-0049-T02
status: progress
priority: high
assignee: codex
depends_on: [NK-WP-0049-T06]
state_hub_task_id: "479d6970-6516-50f5-9436-bb5b56d58aad"

Accountable worker: codex. Service/credential-owner acceptance remains with the named owner boundary.

The source inventory and known legacy objects exist; a complete current-use ledger is not yet accepted. Next action is read-only reconciliation of the 93 entries in the latest October15 source inventory (the earlier snapshot had 87) against active Make/bootstrap/operator routes and owner evidence.

Done when: each entry is identified as current, retired or owner-deferred, with its owner, credential route, current consumer(s), native evidence or exact missing proof, and next responsible task. Explicitly cover the historical sso/keycape-pi-token and mfa/privacyidea-trigger-admin objects, lab init, factor diagnostic delivery, LDAP group/provider boundary, resolver cutover and root-output/isolation boundaries. Secret presence is not credential validity. Send current native checks to T21 and historical revocation/rotation decisions to T22. Do not reopen completed source repairs without a new finding.

Prepare the isolated full-service restore and acceptance packet

id: NK-WP-0049-T15
status: progress
priority: high
assignee: codex
depends_on: [NK-WP-0049-T08, NK-WP-0049-T13, NK-WP-0049-T27, NK-WP-0049-T28, NK-WP-0049-T29, NK-WP-0049-T30]
state_hub_task_id: "1d1eb903-a4b0-5bc4-a421-0b842d7f68ae"

Accountable worker: codex. Service/credential-owner acceptance remains with the named owner boundary.

Prepared draft scope moved to T27. See docs/identity-service-restore-packet.md: restore order, containment, safe evidence and per-service positive/negative checks are drafted. Read-only current runtime image digests were observed for all four services; correspondence to the selected archive and independent image availability are not yet proved.

Next action: admit a named disposable target/private route, verify and stage matching images independently, obtain private existing-user/MFA test inputs and review version-specific native commands with service owners. Incorporate T02's ledger before submitting current-caller acceptance.

2026-10-16 checkpoint: T28 completed the exact local image save/load/inspect and T29 prepared and natively tested the bounded LLDAP stage. Its actual owner run is T16 and can proceed independently of the remaining multi-service packet. The other service adapters, matching archive/runtime lineage and private existing-user/MFA inputs are still required; no blanket T15 completion is claimed.

Done when: service owners accept the exact targets/procedure, existing-user and MFA test identities/input custody are available without posting secrets, outbound notification/provider writes are disabled, and every required new credential or production effect has its own owner decision. The verifier deletes its plaintext; it is not a production importer. This task prepares authority and inputs for T16–T20, not an automatic production restore.

Prove restored LLDAP directory and existing-user authentication

id: NK-WP-0049-T16
status: done
priority: high
assignee: codex
depends_on: [NK-WP-0049-T13, NK-WP-0049-T29]
state_hub_task_id: "b64d4d6a-e851-58d1-9929-f03343147de0"

Accountable worker: codex. Service/credential-owner acceptance remains with the named owner boundary.

Owner boundary: NetKingdom directory/User Engine owners. Restore the captured original directory/configuration in the admitted isolated environment.

Done when: user, group and membership identities are preserved; an independently supplied existing-user credential authenticates; a wrong credential is refused; provider writes/outbound effects stay confined; cleanup is verified. Native fixtures from T05 and SQLite integrity from T13 do not satisfy this current-data acceptance. Record only safe counts, identities and outcomes.

Completed 2026-10-16 at 18:37 UTC. The operator privately supplied the independent age key and an existing backup-era account credential to the exact local executor. All 36 archive hashes and the known ciphertext/source/target bindings matched. Native existing-user LDAP bind passed; explicit invalid credentials were rejected. The original seven users, 17 groups, 13 memberships and the available attribute/schema rows were preserved across service startup, authentication and graceful shutdown. The container had only loopback/no external routes; pre-import isolation and container/tmpfs cleanup passed. No production endpoint was contacted or secret exported to the agent.

Evidence: actual-isolated-lldap-recovery.json and actual-isolated-lldap-review.json in docs/evidence/2026-10-16-identity-outage-recovery/. This closes the actual LLDAP stage independently of the remaining T15 packet. It does not prove Authelia/KeyCape authentication, retained-token MFA or audit continuity. T17, T18 and T19 retain their remaining adapter/input prerequisites.

Prove restored Authelia storage and OIDC continuity

id: NK-WP-0049-T17
status: wait
priority: high
assignee: codex
depends_on: [NK-WP-0049-T15, NK-WP-0049-T16]
blocking_reason: "LLDAP recovery T16 passed; await the exact isolated Authelia adapter/procedure and private authentication-flow inputs from T15."
state_hub_task_id: "b47f9975-d6d7-5bb8-93ce-886deb1dfa70"

Accountable worker: codex. Service/credential-owner acceptance remains with the named owner boundary.

Owner boundary: SSO/Authelia and directory owners. Restore the original storage, encryption/session/signing material and configuration in isolation.

Done when: native storage decryption succeeds, issuer/signing identity and client configuration are preserved, and an isolated authentication flow has positive/negative results using the restored directory. Disable outbound notifications and verify cleanup. Archive integrity/RSA parsing alone are not native Authelia acceptance.

Prove retained privacyIDEA token decryption and existing-token MFA

id: NK-WP-0049-T19
status: wait
priority: high
assignee: codex
depends_on: [NK-WP-0049-T15, NK-WP-0049-T16, NK-WP-0049-T31]
blocking_reason: "LLDAP T16, token-material T31 and audit T20 passed; await the exact isolated MFA adapter and owner-selected retained-token/factor inputs."
state_hub_task_id: "6f8af1a7-2a3f-5fa8-8202-83103ebd55b2"

Accountable worker: codex. Service/credential-owner acceptance remains with the named owner boundary.

Owner boundary: MFA/privacyIDEA and directory owners. Use the exact linked dump and original enckey from the independently recovered generation.

Done when: retained token material decrypts inside the isolated process, a selected existing token completes a native MFA check, a wrong key/code is refused, and cleanup is verified. Obtain any token/user input through attended private custody; never emit token plaintext, seeds or credentials. Native database row counts and disposable synthetic-token fixtures do not prove this outcome.

2026-10-16 refinement: native token-material readback is separately tracked in T31 using the prepared T30 executor. T19 retains actual existing-token MFA and wrong-code/wrong-key authentication refusal. Internal decryption or padding success does not authenticate a user. No MFA code is needed for T20/T31; the selected user's private factor input and exact isolated MFA adapter remain T19.

Prove historical and new privacyIDEA audit-signature continuity

id: NK-WP-0049-T20
status: done
priority: high
assignee: codex
depends_on: [NK-WP-0049-T13, NK-WP-0049-T30]
state_hub_task_id: "375fa8f9-b8a5-5048-97a7-ec141aa7c45d"

Accountable worker: codex. Service/credential-owner acceptance remains with the named owner boundary.

Owner boundary: MFA/privacyIDEA audit owner. Use the original recovered audit pair and linked database in the admitted isolated target.

Done when: existing pre-backup audit signatures are verified natively, a new isolated audit entry is signed and verified with the preserved identity, a wrong key fails, and cleanup is recorded. Report any historical verification failure for explicit owner disposition; do not regenerate keys to hide it. Matching RSA pairs from T13 are a prerequisite, not audit-history verification.

Completed 2026-10-16 at 19:10 UTC using the actual independently retrieved bundle and original audit identity on the fixed isolated local engine. All 279,209 stored rows were signed and verified; zero unsigned, invalid or legacy signatures. A new isolated entry verified, and tampering/unrelated-public-key checks were refused. Container/plaintext cleanup passed. Receipt and reviewed source/ciphertext/target bindings: actual-isolated-privacyidea-recovery.json and actual-isolated-privacyidea-review.json in docs/evidence/2026-10-16-identity-outage-recovery/. Stored-row continuity is accepted; event completeness and existing-token MFA are separate and are not claimed.

Prove restored KeyCape issuer, policy and consumer continuity

id: NK-WP-0049-T18
status: wait
priority: high
assignee: codex
depends_on: [NK-WP-0049-T15, NK-WP-0049-T16, NK-WP-0049-T19]
blocking_reason: "LLDAP recovery T16 passed; await the exact isolated KeyCape adapter/procedure and retained-token MFA proof for the selected flow."
state_hub_task_id: "141dd773-c1fa-56c1-a852-bbe2372e5125"

Accountable worker: codex. Service/credential-owner acceptance remains with the named owner boundary.

Owner boundary: KeyCape and the selected current consumer owners. Restore the original signing/configuration, registry, authentication policies and receipts.

Done when: issuer/JWKS signing identity, registry/client configuration and policy/receipt continuity are preserved; selected current consumers pass native positive/negative authentication using the restored dependencies; and isolation and cleanup hold. Confirm the actual dependency graph in T15. Do not route a production consumer to the isolated issuer or mint new client grants implicitly.

Verify current caller and provider admission at their actual boundaries

id: NK-WP-0049-T21
status: wait
priority: high
assignee: codex
depends_on: [NK-WP-0049-T02]
blocking_reason: "Await the current-caller ledger and exact per-caller owner admission/test inputs; the backup credential grants do not admit unrelated callers."
state_hub_task_id: "77aa42be-7a39-5d93-8042-03d78b4094b4"

Accountable worker: codex. Service/credential-owner acceptance remains with the named owner boundary.

Owner boundary: each current caller's service/provider and credential owner. Use T02's ledger to select the exact current procedures and required owner lanes.

Done when: each current caller has scoped delivery and native positive/negative acceptance or an explicit accepted retirement/disposition. Include dedicated factor-diagnostic admission (do not borrow KeyCape's reader), deployed reserved LDAP group denial, resolver/cutover input provenance and functional proof, and bootstrap/root-output/isolation callers. Fixtures, an existing tty and historical bootstrap receipts are not current authority. Propose exact write/cutover actions before execution; record deferred actions separately rather than faking a pass.

Resolve historical credential and configuration-preserving rotation dispositions

id: NK-WP-0049-T22
status: wait
priority: high
assignee: codex
depends_on: [NK-WP-0049-T02, NK-WP-0049-T21]
blocking_reason: "Await the caller/consumer ledger, current replacement-lane acceptance and explicit credential-owner dispositions; existing exposure-rotation deferral is unchanged."
state_hub_task_id: "3ea6bf00-a87f-58ea-bdce-328c13a03a55"

Accountable worker: codex. Service/credential-owner acceptance remains with the named owner boundary.

Owner boundary: Platform/custody and affected consumer owners. Review historical personal-admin/trigger credential copies and unsafe predecessor paths against known current consumers and rollback custody.

Done when: every affected credential has an owner-accepted disposition: verified revocation/replacement, continued bounded use with acceptance, or an explicit linked deferral. Signing/client-secret changes require preservation and native multi-consumer proof if actually approved. CUST-WP-0073-T05's founder rotation deferral and decision 1591daa1-6e85-4dfd-aadd-5a9fe1ef47fa remain in force; record their exact applicability rather than automatically rotating or treating source retirement as provider revocation.

Provide retrieval that works when OpenBao and SSO are unavailable

id: NK-WP-0049-T23
status: done
priority: high
assignee: codex
depends_on: [NK-WP-0049-T13, NK-WP-0049-T25]
state_hub_task_id: "1ec537ee-dfee-5576-be36-a81bdba417d1"

Accountable worker: codex. Service/credential-owner acceptance remains with the named owner boundary.

Completed 2026-10-16 at 18:02 UTC. The operator used independently held Backup account access in the private prompt. The actual protected 109,738,344-byte ciphertext was downloaded with unchanged ETag/size/SHA256, with no OpenBao/SSO contact or provider writes by the retrieval path. The human-held age key matched the registered recipient; wrong-key refusal, all 36 manifest hashes, SQLite integrity, original key pairs and native isolated PostgreSQL 16.14 restore passed. The database retained five token, two resolver and two realm rows. Disposable container and temporary plaintext cleanup passed. The agent received no secret.

Evidence: actual-offline-transfer.json, actual-offline-recovery.json and actual-offline-review.json under docs/evidence/2026-10-16-identity-outage-recovery/. The review rechecked the local ciphertext digest and verifier source. This proves the independent path without requiring a deliberate production outage. Service authentication, retained-token decryption and historical audit verification remain T16–T20. T26 separately records accepted guide publication.

Record owner disposition for second-provider and external monitoring gaps

id: NK-WP-0049-T24
status: todo
priority: medium
assignee: codex
depends_on: [NK-WP-0049-T14]
state_hub_task_id: "474f5a78-a643-59c6-bcf9-fc4a908dfcb0"

Accountable worker: codex. Service/credential-owner acceptance remains with the named owner boundary.

Next action: record the current single-provider and host-local-health limits, then obtain explicit owner scope decisions for a second encrypted provider copy, external host-loss/freshness detection and recurring recovery-drill cadence.

Done when: each gap has either an accepted bounded residual risk or a linked live intake/workplan with owner and trigger. Existing daily RPO/25-hour health settings remain the approved baseline. Scaleway full-identity publication and an external watchdog are not implemented; do not add their grants/jobs implicitly. This task triages the new insights instead of silently expanding NK49's required implementation to an entire disaster-recovery programme.

Deliver and verify the offline retrieval adapter and recovery tools kit

id: NK-WP-0049-T25
status: done
priority: high
assignee: codex
depends_on: [NK-WP-0049-T13]
state_hub_task_id: "1c5ea7a1-edf7-5072-b36b-8be7f827957a"

Accountable worker: codex.

Completed 2026-10-16. retrieve-identity-offline.py uses privately entered independent Backup-account access or an exact read-capable identity-folder share. It performs only fixed-provider HEAD/GET, binds the reviewed generation by strong ETag/length/SHA256, refuses redirects/proxies/unreviewed scope and removes failed downloads. No OpenBao/SSO, age key or standing agent credential is required. recovery-kit.py builds a reproducible allowlisted ZIP containing the tools, public profile, known-good ciphertext receipt and recovery guides, with file checksums. Its publication path creates only three versioned guide/ZIP/checksum objects and verifies readback without overwriting existing conflicting objects. run-offline-recovery.sh provides the owner's two-hidden-prompt handoff.

Evidence: tests/test_identity_offline_recovery.py and the bounded test/build receipt in docs/evidence/2026-10-16-identity-outage-recovery/. The focused run passed 65 tests including real disposable age/PostgreSQL fixtures. ZIP-contained entry points import without sibling repositories. This is source/fixture proof, not actual independent-provider retrieval (T23) or publication (T26).

2026-10-16 correction: the first owner handoff stopped before its credential prompt because Python text r+ requires a seekable stream. Reproduced against a real controlling PTY; replaced both prompts with the shared hidden-input helper, refusing redirected input and echo fallback. Added real-terminal and both-CLI regressions plus closed stage/HTTP-code diagnostics (no raw responses). The focused rerun passed 36 tests. Evidence: terminal-prompt-fix.json in the outage evidence directory. This source correction does not complete T23/T26.

Publish and verify the disaster recovery guide on Storage Share

id: NK-WP-0049-T26
status: done
priority: high
assignee: codex
depends_on: [NK-WP-0049-T25, NK-WP-0049-T27]
state_hub_task_id: "94d054fe-e5d8-5ddf-8e48-80714123d184"

Accountable worker: codex; operator holds provider access privately.

Requested by the operator on 2026-10-16; decision fc3ef8db-0a33-41d5-a0ba-d13169c5f884 records the bounded scope. Guide source: docs/disaster-recovery-storage-share.md. Document scope, independently held provider/key requirements, exact known-good fallback, copy/download verification, archive-check commands, service restore order, recurring schedule and remaining coverage limits. Include the offline tools so failed Forgejo does not prevent access to the recovery scripts; retain an independent copy and trusted digest.

Completed 2026-10-16 at 18:01 UTC through the private independent-provider owner handoff. All three objects were conditionally created and read back with matching strong ETag, length and content; the recorded digests match a fresh reproducible local build. No existing object or provider permission was changed. The failed first attempt and terminal correction remain documented in T25.

Published location: Storage Share account Backup, railiance-backups/net-kingdom-identity/:

• net-kingdom-recovery-57d1940947aa25ff-START-HERE.md
• net-kingdom-recovery-57d1940947aa25ff.zip
• net-kingdom-recovery-57d1940947aa25ff.zip.sha256

Trusted ZIP SHA256: 57d1940947aa25ff83d0f25ddf30090eb40595cccaab64225de882a2fd59ea2f. Evidence: actual-offline-publication.json and actual-offline-review.json in docs/evidence/2026-10-16-identity-outage-recovery/. The published guide is the exact pre-drill revision; this acceptance record records the later successful drill without implying an unperformed republication. Keep an independent copy and trusted digest outside the failed service/provider dependency.

Draft the isolated service restore packet and observe candidate images

id: NK-WP-0049-T27
status: done
priority: high
assignee: codex
depends_on: [NK-WP-0049-T08, NK-WP-0049-T13]
state_hub_task_id: "e273e803-bee7-5d90-bc51-ecc08c0c8eff"

Accountable worker: codex.

Completed 2026-10-16. Draft in docs/identity-service-restore-packet.md names the known recovered input, restore order, original key/data ownership, default-deny containment, blocked production/provider/notification effects, cleanup and five native acceptance lanes with positive/negative checks. Read-only metadata from the pinned primary cluster resolved all four current service image digests, including the separately observed privacyIDEA main container and LLDAP's mutable source tag. Evidence: docs/evidence/2026-10-16-identity-outage-recovery/image-observation.json.

This task extracts completed preparation from T15. Named target admission, archive/runtime image-lineage matching and independent image availability, private existing-user/MFA inputs and owner-reviewed native commands remain T15; no service restore, production change or owner acceptance is claimed.

Stage and verify the five exact service images for local offline recovery

id: NK-WP-0049-T28
status: done
priority: high
assignee: codex
depends_on: [NK-WP-0049-T27]
state_hub_task_id: "4829b091-76d2-5b10-bdfc-032800190c0a"

Accountable worker: codex.

Completed 2026-10-16. All five pinned references (PostgreSQL, LLDAP, Authelia, KeyCape, privacyIDEA) were inspected on the fixed local Linux Docker engine bnt-lap001, ID 74f93783-6cef-43a8-a3c4-f62586cd3bc1. Missing LLDAP and privacyIDEA images were pulled only by their observed immutable digests. Saved a private 347,521,536-byte image archive, loaded it and re-inspected every exact reference. Evidence and archive SHA256/location: docs/evidence/2026-10-16-identity-outage-recovery/offline-service-images.json.

The local drill no longer requires the production registry. This is not off-host image escrow or proof of historical mutable-tag/backup lineage; those limits remain in T15. No service custody or age key was supplied to image caching.

Prepare and verify the bounded native LLDAP recovery executor

id: NK-WP-0049-T29
status: done
priority: high
assignee: codex
depends_on: [NK-WP-0049-T13, NK-WP-0049-T27, NK-WP-0049-T28]
state_hub_task_id: "2f8c73b0-1cdd-5b22-99b6-017ccde4454b"

Accountable worker: codex; operator retains actual private recovery inputs.

Completed 2026-10-16. isolated-lldap.py and run-isolated-lldap.sh provide the exact owner-run stage in docs/identity-service-restore-packet.md: fixed local engine/images, no remote Docker endpoint, original key/seed/JWT/directory data, Linux-tmpfs decryption, memory/stdin-only private inputs, network-none/read-only containers, isolation inspection before import, actual existing-user bind, explicit invalid-credential denial, stable directory rows and cleanup checks. Missing original server keys are refused; reset/replacement/notification paths are disabled. No live Kubernetes/OpenBao/SSO path or provider grant is used.

Evidence: tests/test_isolated_lldap_recovery.py exercises the actual pinned image, disposable retained-account directory and a full synthetic encrypted bundle through the same native executor. The private real-bundle run and its acceptance remain T16; fixture success is not actual user/MFA/audit recovery. See isolated-lldap-preparation.json in the October16 outage evidence directory for exact test/source/target provenance and limitations.

Prepare the bounded native privacyIDEA token-material and audit executor

id: NK-WP-0049-T30
status: done
priority: high
assignee: codex
depends_on: [NK-WP-0049-T13, NK-WP-0049-T28]
state_hub_task_id: "6e4a43db-9be2-5662-bdbf-1595fce03052"

Accountable worker: codex; operator retains private recovery inputs.

Completed 2026-10-16. isolated-privacyidea.py and its owner launcher use the fixed local Docker target and exact cached PostgreSQL/privacyIDEA3.12.2 images. The linked dump, original enckey/audit pair and required secret fields are restored only into a loopback-only, nonroot, read-only, capability-free, no-host-mount/port, tmpfs-backed environment with inspected pre-import isolation. Decrypted host staging is tmpfs; normal cleanup removes all private data and owned containers, with a bounded holding lifetime if the parent is interrupted.

The native provider decrypts token-key material internally, checks every stored audit row, adds one isolated signed entry and checks wrong-public-key/tampering negatives. Original policy for legacy signature verification is retained only from literal archived configuration; archived Python is never executed. Invalid or unsigned history remains reported, not repaired or sampled away. Native material readback is not existing-token MFA or proof of event completeness.

Evidence: tests/test_isolated_privacyidea_recovery.py and isolated-privacyidea-preparation.json in the October16 outage evidence folder. Synthetic pinned-image tests exercise valid/tampered/unsigned history, malformed keys, encrypted archive to native restore, isolation and cleanup. Real original material/audit acceptance remains T31/T20; actual MFA remains T19.

Verify retained privacyIDEA token material inside the isolated provider

id: NK-WP-0049-T31
status: done
priority: high
assignee: codex
depends_on: [NK-WP-0049-T13, NK-WP-0049-T30]
state_hub_task_id: "26383841-8076-5c53-bbd1-9db64e32544a"

Accountable worker: codex; operator supplies only the independently held age key in the hidden helper. This task extracts actual material-readback scope from T19, preserving T19's existing-token MFA acceptance gate and historical identity.

Completed 2026-10-16 at 19:10 UTC in the same actual run as T20. All five retained token materials were decrypted internally with the original enckey; an unrelated key could not reproduce any of the five. All 36 manifest hashes, exact executor/ciphertext/target bindings, isolation and cleanup passed. Safe receipts are the actual-isolated-privacyidea-{recovery,review}.json files referenced under T20. No retained-factor authentication is claimed. Done when: exact ciphertext/source/target/manifest bindings pass; the native provider reads every nonempty encrypted token-key field using the original enckey without exporting material; an unrelated key cannot reproduce that material; aggregate counts and container/plaintext cleanup are recorded.

Only bounded internal readback is accepted here. Padding/readback success does not prove the recovered secret authenticates a retained factor; actual positive MFA and wrong-key/wrong-code authentication refusal remain T19. No token seed, plaintext, serial or user record belongs in the receipt, Git, State Hub or chat.

Accept remaining recovery and caller evidence and close the G08 handoff

id: NK-WP-0049-T03
status: wait
priority: high
assignee: codex
depends_on: [NK-WP-0049-T02, NK-WP-0049-T16, NK-WP-0049-T17, NK-WP-0049-T18, NK-WP-0049-T19, NK-WP-0049-T20, NK-WP-0049-T21, NK-WP-0049-T22, NK-WP-0049-T23, NK-WP-0049-T24, NK-WP-0049-T26]
blocking_reason: "Await the separately tracked native service proofs, current/legacy caller dispositions and residual-scope decisions; archive recovery, backup activation, independent outage retrieval and guide publication are complete."
state_hub_task_id: "d68cd4d3-1ba3-530e-88f9-26ebb8dd6eac"

Accountable worker: codex. Service/credential-owner acceptance remains with the named owner boundary.

Owner boundary: NetKingdom, Platform and affected service owners, with Custodian G08 receiving the final evidence. The completed archive/job milestones remain accepted independently of this final gate.

Done when: T02 and T15–T31 have their stated completed or explicitly accepted handoff outcomes; service authentication/token/audit proof and current/legacy caller dispositions are accepted; the source workplan and projected tasks agree; and G08 has an exact evidence/handoff record. Record actionable residuals as live work records before workplan closure. Preserve historical IDs and receipts; do not call a deferred rotation or an untested full-estate restore complete.

Evidence locations and refinement history

“Live” above means docs/evidence/2026-10-16-identity-backup-live/ in NetKingdom. Implementation preparation is in the October16 implementation directory; earlier backup discovery/database recovery is in docs/evidence/2026-10-15-bootstrap-recovery/. The source-repair receipts explicitly labelled Custodian belong to that owning repository. Use the recovery ceremony for exact procedure and approval history. No secret values belong in task evidence.

Refinement rule: substantial work and newly understood remaining scope must be visible in task blocks at a checkpoint, not only in progress prose. This replan preserves T01–T03 identity and original evidence while making the completed milestones and each remaining acceptance gate separately manageable.

Managed identifier assignment also consumed a preexisting staged generated brief; the follow-up commit restores its prior published baseline while preserving the operator's working brief and original staged version. This is a projection-tool side effect, not new recovery evidence.