diff --git a/docs/bootstrap-recovery-ceremony.md b/docs/bootstrap-recovery-ceremony.md index c17cf47..0333858 100644 --- a/docs/bootstrap-recovery-ceremony.md +++ b/docs/bootstrap-recovery-ceremony.md @@ -7,6 +7,12 @@ continuity and current/legacy caller acceptance remain open for NK-WP-0049. Approved execution is recorded in decisions40bfae40 and5edd5203; this document records their scope and evidence. +**Outage quick start:** [Storage Share disaster recovery guide](disaster-recovery-storage-share.md). +The independently prompted provider download and offline tools ZIP are prepared; +actual outage-path retrieval and guide publication are separately tracked in +T23/T26. The [isolated service restore packet](identity-service-restore-packet.md) +contains the draft native-service sequence and remaining T15 admission inputs. + ## What has been established On 2026-10-15 the operator identified Scaleway and Storage Share as the off-host diff --git a/docs/disaster-recovery-storage-share.md b/docs/disaster-recovery-storage-share.md new file mode 100644 index 0000000..7926f0d --- /dev/null +++ b/docs/disaster-recovery-storage-share.md @@ -0,0 +1,136 @@ +# NetKingdom disaster recovery — start here + +Updated 2026-10-16. Owner: NetKingdom with Railiance Platform and the individual +identity service owners. This guide covers the identity backup, not whole-host, +OpenBao, Forgejo, User Engine or application recovery. + +## If the identity services are down + +1. Use a trusted recovery workstation. Obtain Storage Share access independently + from your password safe/offline custody, and the independently held + `railiance-backup` age private key. Never paste either into chat or a ticket. + The private key must match this **public** recipient: + `age1zvryunvjhvpkmasskauga2heeg0ztnte9ymgppvjge36ekumk50syr3tsz`. + The provider password and encryption key are separate requirements. +2. Open using the independent provider + account, not NetKingdom SSO. In the Backup account, navigate to + `railiance-backups/net-kingdom-identity/`. Recovery documentation and a small + versioned recovery-tools ZIP are placed beside the encrypted archives. + Keep a copy of this guide and ZIP outside Storage Share as well. +3. Choose an intact recovery generation. The full identity filename starts + `net-kingdom-identity-` and ends `.tar.age`. A `privacyidea_db-*.dump.age` + database-only backup cannot replace the matching service keys and configs. + Each generation has a `.receipt.json` index. Preserve the original objects. +4. Check timestamp, size and SHA256 against a separately trusted receipt. The + index is useful metadata, not a signed independent trust anchor. For the + first actually recovered, protected generation the trusted values are: + + - Artifact: `net-kingdom-identity-20261016T063159Z-5882771aa2ce.tar.age` + - Bytes: `109738344` + - SHA256: `7117b3fe1e4840c86b301c94c85e6b16709e5626bb0884bd0580bde630e30e30` + - Strong ETag at publication: `"28ec1f7014e492de4c40e1ae75502f08"` + + This is a known good fallback, not a claim that it is the latest backup. + For a newer generation, reconcile its index with the independently retained + publisher receipt before constructing a new download input. Never edit a + receipt just to make a failed integrity check pass. ETag changes require + owner reconciliation even if the filename is unchanged. +5. On Linux with Python 3, unpack the recovery-tools ZIP into a private working + directory and enter it. Verify its `SHA256SUMS` against a trusted repository + copy or independently held kit digest before executing code. A checksum + stored only alongside the download detects damage, not a malicious provider. + Alternatively use an independently available trusted net-kingdom checkout. + + ```bash + python3 tools/bootstrap-recovery/retrieve-identity-offline.py \ + --source-receipt docs/evidence/2026-10-16-identity-backup-live/actual-offsite.json \ + --auth account --receipt /tmp/nk49-outage-transfer.json + ``` + + In your own terminal, enter the independently held **Backup** account app + password at the hidden prompt. The fixed account name is `Backup`. + If your independently held credential is a read-capable public share token + for this exact identity folder, use `--auth folder-share` instead. An + upload-only token, another account's password or a share for the backup root + is insufficient for this command. A 401/403/404 stops; do not widen permissions + or substitute another token silently. Download performs only HEAD/GET against + the fixed provider, rejects redirects/proxies, and verifies ETag/size/hash. + Failure removes its incomplete local download. Success leaves ciphertext + in a private `/tmp/nk49-identity-fetched-*` directory named in the receipt. +6. Verify and decrypt through the existing hidden key helper. Requirements: + `age`, `age-keygen`, `openssl`, Python 3 and Docker; the following PostgreSQL + image must already be present on the recovery host: + `postgres@sha256:ff23cdce56cac62ada6f66013e1a50864c0abbe79d132d40b6e05bd80f378a70`. + Stage and verify this image before a drill. The tools ZIP contains no images. + Allow enough private disk and RAM for a roughly 110 MB ciphertext, a bounded + 512 MiB archive and the disposable PostgreSQL container (2 GiB memory limit). + + ```bash + bash sso-mfa/bootstrap/sops-custody-unlock.sh \ + --expected-recipient age1zvryunvjhvpkmasskauga2heeg0ztnte9ymgppvjge36ekumk50syr3tsz \ + --prompt -- \ + python3 tools/bootstrap-recovery/verify-identity.py \ + --transfer-receipt /tmp/nk49-outage-transfer.json \ + --receipt /tmp/nk49-outage-archive-recovery.json + ``` + + Use new receipt names on every run. The helper removes its temporary private + key on exit. The verifier checks the encrypted manifest and all file hashes, + original key pairs, SQLite integrity, wrong-key refusal and an isolated + PostgreSQL restore. It removes decrypted staging and the disposable database. + It deliberately does not install restored secrets or alter production. +7. Preserve the sanitized receipts and original ciphertext. Remove your local + downloaded ciphertext directory when no longer needed. If the verifier fails, + stop and check the bounded receipt with the service owner; do not regenerate + encryption/signing keys, reset user passwords or delete the provider backup. + +## What the backup contains + +| Component | Recovery material | +| --- | --- | +| LLDAP | Native SQLite snapshot, remaining durable data, deployment and referenced secrets | +| Authelia | Native SQLite snapshot, encryption/session/OIDC material, current configuration and referenced secrets | +| KeyCape | Original signing/configuration material, client configuration, authentication policies and receipts | +| privacyIDEA | Fresh PostgreSQL dump, original encryption key, matching original audit RSA key pair, current configuration and required durable files | + +The encrypted manifest records exact files and capture intervals. This is not +a global atomic snapshot of all services. Historical backups and selected +notification/log files are excluded as documented by the capture profile. +The backup contains sensitive service credentials; only ciphertext belongs on +ordinary persistent recovery media. The age private key is never in the bundle. + +## Restoring working services + +**Archive recovery passed on 2026-10-16. Full restored-service acceptance remains +open.** The archive verifier deletes its plaintext and is not a service importer. +Follow `docs/identity-service-restore-packet.md` with service owners before an +isolated service drill. Its pending target and image fields must be resolved. + +Restore the exact matching data/config/original keys into an isolated recovery +environment first. Keep production ingress, public DNS, notifications, external +provider writes and real consumer traffic disconnected. Establish directory and +database readiness; prove LLDAP existing-user authentication; then privacyIDEA +retained-token MFA/audit continuity, Authelia storage/OIDC, and KeyCape/current +consumer continuity according to their actual dependencies. Existing user/MFA +inputs stay in operator custody. Public issuer identity must be preserved while +test traffic is routed privately; do not change production DNS for a drill. + +Only after owner acceptance of native positive/negative tests should the owners +approve a production cutover, including target identity, rollback, DNS/ingress, +current consumers, credential dispositions and monitoring. This guide does not +authorize a production restore or automatic credential rotation. + +## Schedule, protection and limits + +Daily capture is scheduled at 03:45 UTC with up to five minutes jitter. Hourly +host-local checks flag a backup older than 25 hours. Retention protects the +independently recovered pinned generation, newest generation, seven distinct +daily and two weekly points; unknown/unindexed objects are not auto-deleted. + +At this revision the full identity backup has one provider copy on Storage Share. +Existing Scaleway database backups do not establish a second full identity copy. +Host-local health cannot report a dead host. Full service login/MFA/audit proofs, +an independently executed outage-path retrieval drill, a second provider copy +and external host-loss monitoring must not be assumed from an upload or this +guide. Track remaining acceptance in NK-WP-0049; current task state lives in the +repository and may be newer than this offline guide. diff --git a/docs/evidence/2026-10-16-identity-outage-recovery/image-observation.json b/docs/evidence/2026-10-16-identity-outage-recovery/image-observation.json new file mode 100644 index 0000000..14eaa0f --- /dev/null +++ b/docs/evidence/2026-10-16-identity-outage-recovery/image-observation.json @@ -0,0 +1,97 @@ +{ + "schema": "netkingdom.identity-recovery-image-observation.v1", + "observed_at": "2026-10-16T15:56:10.384927+00:00", + "cluster_uid": "a553c742-0115-43d4-99a4-a5ca56fe0786", + "read_only": true, + "secret_values_read": false, + "archive_generation_match_proven": false, + "independent_image_availability_proven": false, + "pods": [ + { + "namespace": "sso", + "pod": "authelia-7f57c4c4fc-mk5hb", + "pod_uid": "7678e756-0af9-4eb9-bc25-c8b09ca490fc", + "owners": [ + { + "kind": "ReplicaSet", + "name": "authelia-7f57c4c4fc", + "uid": "de04027d-ebff-45bc-8aff-73583907b57e" + } + ], + "containers": [ + { + "name": "authelia", + "image_id": "docker.io/authelia/authelia@sha256:bd97cff4fcbf715b5ff1f9ae286afbe6033afce385302520b0368122d43a6f54", + "ready": true + } + ] + }, + { + "namespace": "sso", + "pod": "keycape-5fc5dd9b7f-zv5xm", + "pod_uid": "bf0d4674-cbe4-4e44-b105-b799eb4c124e", + "owners": [ + { + "kind": "ReplicaSet", + "name": "keycape-5fc5dd9b7f", + "uid": "8d862e2d-532c-4f1f-b510-cbe489ec579b" + } + ], + "containers": [ + { + "name": "keycape", + "image_id": "forgejo.coulomb.social/coulomb/key-cape@sha256:cdec9dd7c515f7e0a6169fc3fa560ee2cc47bf40d899755dec808c362f5ce010", + "ready": true + } + ] + }, + { + "namespace": "sso", + "pod": "lldap-f6df76764-jjp97", + "pod_uid": "b05329a6-8b9d-45d6-a573-c03c04e667be", + "owners": [ + { + "kind": "ReplicaSet", + "name": "lldap-f6df76764", + "uid": "306a97b4-a5f7-421f-971d-36a3dd5fbc47" + } + ], + "containers": [ + { + "name": "lldap", + "image_id": "docker.io/lldap/lldap@sha256:9e605a66c02514bfcffd1b67cafb1e98d50992216bb2871d7ae44622047dd09d", + "ready": true + } + ] + }, + { + "namespace": "mfa", + "pod": "privacyidea-86b695c88c-9kh7g", + "pod_uid": "2a44aa01-84e0-4778-a7c3-d0597b192259", + "owners": [ + { + "kind": "ReplicaSet", + "name": "privacyidea-86b695c88c", + "uid": "7db5316a-0b89-431e-a678-a5dad372fc9c" + } + ], + "containers": [ + { + "name": "factor-recovery", + "image_id": "ghcr.io/gpappsoft/privacyidea-docker@sha256:af7841adad262f129e0c1d4f553af13f21cb2f4dc713533f316cfe43ed0b4473", + "ready": true + }, + { + "name": "privacyidea", + "image_id": "ghcr.io/gpappsoft/privacyidea-docker@sha256:af7841adad262f129e0c1d4f553af13f21cb2f4dc713533f316cfe43ed0b4473", + "ready": true + } + ] + } + ], + "knowledge": { + "version": "0.2.6", + "package_digest": "8d1d6a9d4d07fec7b619618de7dfe2be2254c1790f85f4198c7fe19db9751c02", + "source_status": "withheld; current SECURITY-COMPANION and owner route sources reviewed directly; orientation only" + } +} diff --git a/docs/evidence/2026-10-16-identity-outage-recovery/preparation-review.json b/docs/evidence/2026-10-16-identity-outage-recovery/preparation-review.json new file mode 100644 index 0000000..331bc51 --- /dev/null +++ b/docs/evidence/2026-10-16-identity-outage-recovery/preparation-review.json @@ -0,0 +1,66 @@ +{ + "schema": "netkingdom.identity-outage-preparation.v1", + "date": "2026-10-16", + "status": "source_fixture_verified_owner_execution_pending", + "tests": { + "command": "NK49_RUN_DISPOSABLE_POSTGRES_TEST=1 python3 -m pytest -q tests/test_identity_offline_recovery.py tests/test_identity_offsite.py tests/test_identity_bundle.py tests/test_privacyidea_offhost_recovery.py", + "passed": 65, + "failed": 0, + "skipped": 0 + }, + "kit_preview": { + "schema": "netkingdom.recovery-kit.v1", + "status": "built", + "secret_values_exported": false, + "new_grants_created": false, + "files": [ + { + "name": "net-kingdom-recovery-62b9464609b8e0fd-START-HERE.md", + "bytes": 8381, + "sha256": "1531675f2e7d3713a3a7fc28a5143fb24f201b19cd05d14c59ee485a0bc11e61" + }, + { + "name": "net-kingdom-recovery-62b9464609b8e0fd.zip", + "bytes": 34527, + "sha256": "62b9464609b8e0fde2c05527fc79eff742ee3f626262d847df8fb579548a9470" + }, + { + "name": "net-kingdom-recovery-62b9464609b8e0fd.zip.sha256", + "bytes": 108, + "sha256": "55e375291b27c2977049b89a5deddaed5e39c7ee9179013ce73d2ad43f2fb308" + } + ], + "finished_at": "2026-10-16T15:56:50.813341+00:00" + }, + "sources": [ + { + "path": "tools/bootstrap-recovery/retrieve-identity-offline.py", + "sha256": "80e9f755b2304c222822b49ab563600832b9452c0c387a0523f4323e591e6270" + }, + { + "path": "tools/bootstrap-recovery/recovery-kit.py", + "sha256": "55df6283898fadae316b89304bacf485dec88459b35ea2b9599c5d387c44ca8c" + }, + { + "path": "tools/bootstrap-recovery/run-offline-recovery.sh", + "sha256": "0e3cfe8cc66d63b91a5c3f90912ce700c648c55b685ac11151631d22d87b40fd" + }, + { + "path": "docs/disaster-recovery-storage-share.md", + "sha256": "1531675f2e7d3713a3a7fc28a5143fb24f201b19cd05d14c59ee485a0bc11e61" + }, + { + "path": "docs/identity-service-restore-packet.md", + "sha256": "34f90e0ee1425083b26a763777b0150241e1cd5fb7dbdf8917f7a7a99482376d" + }, + { + "path": "tests/test_identity_offline_recovery.py", + "sha256": "d88a752fd862983e3c01ebe6aea4baffa4168802d8ec616db161585a80c65f62" + } + ], + "actual_outage_retrieval_verified": false, + "actual_storage_share_guide_publication_verified": false, + "service_restore_executed": false, + "new_grants_created": false, + "secret_values_exported": false +} diff --git a/docs/identity-service-restore-packet.md b/docs/identity-service-restore-packet.md new file mode 100644 index 0000000..10104b2 --- /dev/null +++ b/docs/identity-service-restore-packet.md @@ -0,0 +1,97 @@ +# Isolated identity service restore packet + +NK-WP-0049-T15. Prepared 2026-10-16; **draft, not admitted or executed**. +Accountable preparer: codex. Execution/acceptance: Railiance Platform and the +LLDAP, Authelia, KeyCape and privacyIDEA service owners. + +## Fixed starting point + +Use the actual recovered artifact and receipt in +`docs/evidence/2026-10-16-identity-backup-live/actual-offsite.json`, with matching +original keys and manifest. The completed archive recovery verified 36 file +hashes, SQLite integrity, RSA pairs and native PostgreSQL restore. It did not +prove service authentication or audit-signature continuity. A newer generation +needs equivalent provenance and archive checks before replacing this input. + +## Targets and image admission still required + +No execution target has been admitted. Select a disposable host/context and +record its immutable identity, operator, private endpoints, maximum lifetime, +resource limits and exact owned cleanup inventory. Never select the production +context or reuse live PVCs. Use fresh private storage; no production host mounts, +service-account tokens, ingress controller or external DNS update credentials. + +The October 16 read-only pod observation is recorded in +`docs/evidence/2026-10-16-identity-outage-recovery/image-observation.json`. +These are candidate inputs, not confirmation of the images in the selected archive: + +| Component | Candidate reference | Admission gap | +| --- | --- | --- | +| PostgreSQL | `postgres@sha256:ff23cdce56cac62ada6f66013e1a50864c0abbe79d132d40b6e05bd80f378a70` | Passed archive DB restore; stage independently of failed registry | +| LLDAP | `docker.io/lldap/lldap@sha256:9e605a66c02514bfcffd1b67cafb1e98d50992216bb2871d7ae44622047dd09d` | Actual running digest resolved; compare selected archive/runtime lineage and stage it | +| Authelia | `docker.io/authelia/authelia@sha256:bd97cff4fcbf715b5ff1f9ae286afbe6033afce385302520b0368122d43a6f54` | Compare selected archive deployment and stage image | +| KeyCape | `forgejo.coulomb.social/coulomb/key-cape@sha256:cdec9dd7c515f7e0a6169fc3fa560ee2cc47bf40d899755dec808c362f5ce010` | Compare selected archive; require independent image availability | +| privacyIDEA | `ghcr.io/gpappsoft/privacyidea-docker@sha256:af7841adad262f129e0c1d4f553af13f21cb2f4dc713533f316cfe43ed0b4473` | Actual main and sidecar digest observed separately; compare archive/runtime lineage and stage it | + +Record image archive digests and offline load/inspect results. Resolve mutable +references from trusted runtime/image metadata rather than pulling today's tag. +The encrypted archive deployment records must be compared privately; do not +export literal environment values, Secret payloads or arbitrary configs. + +## Restore order and containment + +1. Freeze the selected artifact/manifest/image/target references. The owner + supplies existing-user credentials and an existing retained MFA token through + private inputs. Agree time synchronization and test identities without + exposing passwords, seed material, client secrets or user records. +2. Establish an isolated network with default-deny external egress, no published + ports, no host networking and no production DNS/ingress. Permit only named + internal service links. The test browser, if needed, must use a separately + reviewed local route and hosts mapping. Preserve issuer/client identity while + keeping all test traffic private. Verify denied external connection attempts. +3. Import manifest-checked data and original keys into fresh private storage. + Do not apply archived Kubernetes objects wholesale: metadata UIDs, mounts, + service accounts, endpoints and sidecars require explicit translation. Keep + notification/provider credentials sealed and unused; disable sending paths. +4. Restore the linked privacyIDEA database with native PostgreSQL first; restore + LLDAP data/config and verify directory readiness. Run directory authentication + before dependent provider checks. Restore privacyIDEA with its original enckey + and audit pair against the isolated database/directory. Restore Authelia with + original storage/OIDC keys and private directory routes. Restore KeyCape with + original signing/config/policies and private MFA/directory dependencies. +5. Run the individual acceptance checks below; do not start provisioning, + enrollment, rotation, scheduled jobs or production consumers automatically. + An unsupported dependency is a named gap, never an implicit production route. +6. Remove only the named disposable containers/network/storage and temporary + plaintext/key files. Confirm absence and production resources unchanged; + retain sanitized receipts, original ciphertext and independently held keys. + +## Acceptance matrix + +| Task / owner lane | Positive check | Negative check | Safe evidence | +| --- | --- | --- | --- | +| T16 / LLDAP | Preserved user/group/membership identities; existing-user bind/login | Wrong password rejected | Image/target IDs, aggregate comparisons and booleans | +| T17 / Authelia | Native storage decryption; same signing identity/client configs; isolated authentication | Wrong credentials or unregistered client refused | Public key thumbprint, test outcome, notification suppression | +| T19 / privacyIDEA MFA | Existing retained token decrypts internally and actual existing-token MFA succeeds | Wrong enckey and invalid code refused in disposable copies | Token material never exported; bounded outcomes only | +| T20 / privacyIDEA audit | Verify historical signatures; create and verify one isolated entry with original key | Mismatched public key/tampered disposable record rejected | Signature counts, continuity result, cleanup | +| T18 / KeyCape | Same issuer/JWKS, registry/policy/receipt continuity; named current consumer authenticates | Wrong secret and denied scope rejected | Public fingerprints, owner-selected test IDs and outcomes | + +Do not silently reset credentials or re-enroll tokens to pass. Historical audit +verification failures must be reported, not hidden with replacement keys. + +## Unresolved inputs before execution + +- Named immutable disposable target, approved private browser route if required, + resource limits, cleanup inventory and operator window. +- Confirm archive/runtime lineage for the observed image digests and stage all + images independently of production and the failed registry. +- A retained user and MFA token available in private owner custody; exact native + verification commands reviewed against the chosen service versions. +- T02 current-caller ledger and T21 per-caller owner admission, including legacy + token, diagnostic, resolver and LDAP-group boundaries. +- Service-owner acceptance of network/config translations and all required + grants. New grants or production effects require their own owner decisions. + +Until resolved, T15 stays in progress and T16–T20 remain waiting. Production +cutover, full-host/OpenBao restoration and application database recovery are +separate owner procedures. diff --git a/registry/capabilities/capability.security.bootstrap-recovery.md b/registry/capabilities/capability.security.bootstrap-recovery.md index da03ee2..57b156c 100644 --- a/registry/capabilities/capability.security.bootstrap-recovery.md +++ b/registry/capabilities/capability.security.bootstrap-recovery.md @@ -52,7 +52,8 @@ discovery: privacyIDEA keys plus linked PostgreSQL dump - Public-recipient age sealing, Storage Share create/readback, separate retention and attended independent-key archive verification - - Railiance bootstrap handoff without requiring recovered SSO for off-host retrieval + - Railiance bootstrap handoff and a prepared independent-provider download path + requiring neither OpenBao nor SSO; actual outage-path owner drill remains pending excludes: - Source repairs and disposable native fixtures do not prove estate root isolation or recovery @@ -69,6 +70,11 @@ availability: - tools/bootstrap-recovery/identity-backup-profile.json - tools/bootstrap-recovery/capture-identity.py - tools/bootstrap-recovery/retrieve-identity.py + - tools/bootstrap-recovery/retrieve-identity-offline.py + - tools/bootstrap-recovery/recovery-kit.py + - tools/bootstrap-recovery/run-offline-recovery.sh + - docs/disaster-recovery-storage-share.md + - docs/identity-service-restore-packet.md - tools/bootstrap-recovery/verify-identity.py consumption_modes: - CLI @@ -95,6 +101,7 @@ evidence: - tests/test_identity_bundle.py - tests/test_identity_offsite.py - tests/test_identity_backup_preparation.py + - tests/test_identity_offline_recovery.py consumer_guidance: recommended_for: - Operator recovers identity, custody or data with independently held recovery material @@ -140,3 +147,12 @@ Assessment: CUST-WP-0081, 2026-10-14. Relations identify interfaces to compose; ## Identity recovery and bootstrap handoff Start with [the recovery ceremony](../../docs/bootstrap-recovery-ceremony.md). It names the exact capture, retrieval, hidden-key verification, activation gates and remaining full-service proof. Railiance bootstrap sequences the isolated recovery and layer handoff; NetKingdom owns identity assets and native acceptance. Discovery does not authorize grants or live restore. + +For an outage, use [the Storage Share recovery guide](../../docs/disaster-recovery-storage-share.md) +and the self-contained kit built by `tools/bootstrap-recovery/recovery-kit.py`. +The private operator handoff publishes the versioned guide/kit with readback, +retrieves the protected generation using independently held provider access, +and prompts separately for the human-held age key. Source/fixture preparation +passed; consult NK-WP-0049-T23/T26 for actual outage drill/publication acceptance. +The [isolated service packet](../../docs/identity-service-restore-packet.md) +records the remaining service-owner admission and native tests. diff --git a/tests/test_identity_offline_recovery.py b/tests/test_identity_offline_recovery.py new file mode 100644 index 0000000..9262d70 --- /dev/null +++ b/tests/test_identity_offline_recovery.py @@ -0,0 +1,155 @@ +"""Outage download boundaries, failure cleanup and independently usable kit.""" +import hashlib +import importlib.util +import io +import json +from pathlib import Path +import subprocess +import sys +import urllib.error +import zipfile + +import pytest + +ROOT = Path(__file__).resolve().parents[1] + + +def load(name, root=ROOT): + spec = importlib.util.spec_from_file_location(name, root / 'tools/bootstrap-recovery' / (name + '.py')) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +m = load('retrieve-identity-offline') +k = load('recovery-kit') + + +class Response(io.BytesIO): + def __init__(self, url, raw, etag='"original"', status=200): + super().__init__(raw) + self.url = url + self.status = status + self.headers = {'Content-Length': str(len(raw)), 'ETag': etag} + + +class Provider: + def __init__(self, body): + self.body = body + self.calls = [] + self.tamper = False + self.etag = '"original"' + self.objects = {} + + def open(self, request, timeout): + method = request.get_method() + self.calls.append((method, request.full_url, dict(request.headers))) + assert request.full_url.startswith((m.ACCOUNT_BASE, m.offsite.BASE)) + if self.objects or method == 'PUT': + if method == 'PUT': + assert request.get_header('If-none-match') == '*' + assert request.full_url not in self.objects + self.objects[request.full_url] = request.data + return Response(request.full_url, b'', status=201) + if request.full_url not in self.objects: + raise urllib.error.HTTPError(request.full_url, 404, 'missing', {}, None) + return Response(request.full_url, self.objects[request.full_url]) + raw = self.body if not self.tamper or method == 'HEAD' else b'X' * len(self.body) + return Response(request.full_url, raw, self.etag) + + +@pytest.fixture +def record(): + return dict(schema='netkingdom.identity-offsite.v1', status='readback_verified', + provider='Storage Share', folder='Backup/railiance-backups/net-kingdom-identity', + artifact='net-kingdom-identity-20261016T063159Z-5882771aa2ce.tar.age', + ciphertext_bytes=32, ciphertext_sha256=hashlib.sha256(b'A' * 32).hexdigest(), + etag='"original"') + + +@pytest.mark.parametrize('mode', ['account', 'folder-share']) +def test_offline_success_uses_only_fixed_provider_reads(record, mode): + provider = Provider(b'A' * 32) + record['untrusted_extra'] = 'must-not-export' + result = m.retrieve(record, 'fixtureSecret1234', mode, provider) + try: + assert Path(result['fetched_path']).read_bytes() == b'A' * 32 + assert not result['openbao_contacted'] and not result['sso_contacted'] + assert {c[0] for c in provider.calls} == {'HEAD', 'GET'} + assert 'fixtureSecret1234' not in json.dumps(result) + assert 'must-not-export' not in json.dumps(result) + assert provider.calls[1][2]['If-match'] == record['etag'] + finally: + path = Path(result['fetched_path']) + path.unlink() + path.parent.rmdir() + + +@pytest.mark.parametrize('field,value', [ + ('artifact', '../another.tar.age'), ('folder', 'other'), + ('ciphertext_sha256', 'x' * 64), ('ciphertext_bytes', True), + ('provider', 'other'), ('etag', 'W/"weak"'), +]) +def test_unreviewed_scope_refuses_before_contact(record, field, value): + record[field] = value + provider = Provider(b'A' * 32) + with pytest.raises(m.offsite.Refused): + m.retrieve(record, 'fixtureSecret1234', 'account', provider) + assert not provider.calls + + +def test_digest_failure_removes_incomplete_ciphertext(record, monkeypatch, tmp_path): + directory = tmp_path / 'download' + directory.mkdir() + monkeypatch.setattr(m.tempfile, 'mkdtemp', lambda **kwargs: str(directory)) + provider = Provider(b'A' * 32) + provider.tamper = True + with pytest.raises(m.offsite.Refused): + m.retrieve(record, 'fixtureSecret1234', 'account', provider) + assert not directory.exists() + + +def test_changed_etag_refuses_before_download(record): + provider = Provider(b'A' * 32) + provider.etag = '"changed"' + with pytest.raises(m.offsite.Refused): + m.retrieve(record, 'fixtureSecret1234', 'account', provider) + assert len(provider.calls) == 1 + + +def test_kit_contains_all_offline_imports_and_checksums(tmp_path): + files = k.build() + archive = next(value for name, value in files.items() if name.endswith('.zip')) + with zipfile.ZipFile(io.BytesIO(archive)) as kit: + kit.extractall(tmp_path) + for line in kit.read('SHA256SUMS').decode().splitlines(): + digest, name = line.split(' ', 1) + assert hashlib.sha256((tmp_path / name).read_bytes()).hexdigest() == digest + for script in ('verify-identity.py', 'retrieve-identity-offline.py'): + result = subprocess.run([sys.executable, str(tmp_path / 'tools/bootstrap-recovery' / script), '--help'], + capture_output=True) + assert result.returncode == 0, result.stderr + assert k.build() == files + + +def test_kit_publication_conditional_and_exact_readback(): + provider = Provider(b'') + # Ensure initial HEAD returns 404 instead of the download-fixture response. + provider.objects['unrelated'] = b'preserved' + files = k.build() + rows = k.publish(files, {'Authorization': 'fixture'}, provider) + assert len(rows) == 3 and all(row['readback_verified'] for row in rows) + assert len([c for c in provider.calls if c[0] == 'PUT']) == 3 + k.publish(files, {'Authorization': 'fixture'}, provider) + assert len([c for c in provider.calls if c[0] == 'PUT']) == 3 + assert provider.objects['unrelated'] == b'preserved' + + +def test_kit_conflicting_existing_object_never_overwritten(): + provider = Provider(b'') + files = k.build() + first = next(iter(files)) + provider.objects[m.ACCOUNT_BASE + first] = b'X' * len(files[first]) + with pytest.raises(k.offsite.Refused): + k.publish(files, {'Authorization': 'fixture'}, provider) + assert not any(c[0] == 'PUT' for c in provider.calls) diff --git a/tools/bootstrap-recovery/README.md b/tools/bootstrap-recovery/README.md index 6b38587..b7f1ff8 100644 --- a/tools/bootstrap-recovery/README.md +++ b/tools/bootstrap-recovery/README.md @@ -3,6 +3,20 @@ Owner work: NK-WP-0049. Follow [the exact recovery ceremony](../../docs/bootstrap-recovery-ceremony.md). +For an identity/custody outage, start with the +[Storage Share disaster recovery guide](../../docs/disaster-recovery-storage-share.md). +`retrieve-identity-offline.py` performs fixed-provider ciphertext retrieval from +privately entered independent provider access, without OpenBao/SSO. Its source +receipt pins the object, strong ETag, size and digest; failed downloads are removed. +`recovery-kit.py` builds an allowlisted standalone tools ZIP and can publish only +three new versioned guide/ZIP/checksum files beside the backups with exact +readback. `run-offline-recovery.sh` is the owner's private two-prompt handoff for +publication, outage-path retrieval and the existing archive recovery check. +Run secret prompts in the owner's own terminal, never an agent-captured PTY. +Source/fixture checks do not replace the actual owner drill/publication receipts. +Service restore admission remains in the +[isolated restore packet](../../docs/identity-service-restore-packet.md). + `verify-bundle.py` validates a digest-bound age-encrypted `secrets/` tar archive using an explicit private temporary key and registered `keys/age.pub`. It checks an exact required-file list, unrelated wrong-key refusal, safe isolated file diff --git a/tools/bootstrap-recovery/recovery-kit.py b/tools/bootstrap-recovery/recovery-kit.py new file mode 100644 index 0000000..a26b997 --- /dev/null +++ b/tools/bootstrap-recovery/recovery-kit.py @@ -0,0 +1,172 @@ +#!/usr/bin/env python3 +"""Build a secret-free offline recovery kit; optionally publish exact new files. + +Publication creates only three fixed versioned objects beside identity backups, +never changes shares, credentials, existing files, retention or runtime jobs. +""" +import argparse +import base64 +from datetime import datetime, timezone +import getpass +import hashlib +import importlib.util +import io +import json +import os +from pathlib import Path +import sys +import urllib.error +import urllib.request +import zipfile + +ROOT = Path(__file__).resolve().parents[2] +HERE = Path(__file__).resolve().parent +spec = importlib.util.spec_from_file_location('offline', HERE / 'retrieve-identity-offline.py') +offline = importlib.util.module_from_spec(spec) +spec.loader.exec_module(offline) +offsite = offline.offsite +FILES = [ + 'docs/disaster-recovery-storage-share.md', + 'docs/identity-service-restore-packet.md', + 'docs/evidence/2026-10-16-identity-backup-live/actual-offsite.json', + 'docs/evidence/2026-10-16-identity-outage-recovery/image-observation.json', + 'tools/bootstrap-recovery/retrieve-identity-offline.py', + 'tools/bootstrap-recovery/identity-offsite.py', + 'tools/bootstrap-recovery/verify-identity.py', + 'tools/bootstrap-recovery/identity-bundle.py', + 'tools/bootstrap-recovery/check-privacyidea-database.py', + 'tools/bootstrap-recovery/identity-backup-profile.json', + 'sso-mfa/bootstrap/sops-custody-unlock.sh', + 'sso-mfa/bootstrap/sops-private.py', + 'keys/age.pub', +] + + +def build(): + content = {name: (ROOT / name).read_bytes() for name in FILES} + if any(b'AGE-SECRET-KEY-1' in value for name, value in content.items() + if not name.endswith(('.py', '.sh', '.md'))): + raise offsite.Refused() + sums = ''.join(hashlib.sha256(value).hexdigest() + ' ' + name + '\n' + for name, value in content.items()).encode() + output = io.BytesIO() + with zipfile.ZipFile(output, 'w', compression=zipfile.ZIP_DEFLATED) as archive: + for name, value in {**content, 'SHA256SUMS': sums}.items(): + entry = zipfile.ZipInfo(name, date_time=(2026, 10, 16, 0, 0, 0)) + entry.external_attr = 0o100600 << 16 + entry.compress_type = zipfile.ZIP_DEFLATED + archive.writestr(entry, value) + kit = output.getvalue() + version = hashlib.sha256(kit).hexdigest() + stem = 'net-kingdom-recovery-' + version[:16] + return { + stem + '-START-HERE.md': content['docs/disaster-recovery-storage-share.md'], + stem + '.zip': kit, + stem + '.zip.sha256': (version + ' ' + stem + '.zip\n').encode(), + } + + +def publish(files, headers, client=None): + client = client or offsite.opener() + result = [] + for name, raw in files.items(): + url = offline.ACCOUNT_BASE + name + digest = hashlib.sha256(raw).hexdigest() + # A retry may accept an identical object after exact readback, but must + # never overwrite a conflicting object or retry an unknown PUT outcome. + present = False + try: + with client.open(urllib.request.Request(url, method='HEAD', headers=headers), timeout=30) as response: + version = offsite.etag(response, url, len(raw)) + present = True + except urllib.error.HTTPError as error: + if error.code != 404: + raise offsite.Refused() from None + if not present: + h = dict(headers, **{'If-None-Match': '*', 'Content-Length': str(len(raw)), + 'Content-Type': 'application/zip' if name.endswith('.zip') else 'text/plain; charset=utf-8'}) + with client.open(urllib.request.Request(url, data=raw, method='PUT', headers=h), timeout=60) as response: + if response.status != 201 or response.url != url: + raise offsite.Refused() + with client.open(urllib.request.Request(url, method='HEAD', headers=headers), timeout=30) as response: + version = offsite.etag(response, url, len(raw)) + h = dict(headers, **{'If-Match': version}) + with client.open(urllib.request.Request(url, headers=h), timeout=60) as response: + if offsite.etag(response, url, len(raw)) != version: + raise offsite.Refused() + downloaded = response.read(len(raw) + 1) + if downloaded != raw: + raise offsite.Refused() + result.append({'name': name, 'bytes': len(raw), 'sha256': digest, + 'etag': version, 'readback_verified': True}) + return result + + +def main(): + p = argparse.ArgumentParser(description=__doc__) + p.add_argument('--output-dir', required=True) + p.add_argument('--publish', action='store_true') + p.add_argument('--auth-source', choices=('prompt', 'attended-openbao'), default='prompt') + p.add_argument('--receipt', required=True) + p.add_argument('--offline-transfer-receipt', help='After prompted publication, retrieve the known recovered generation using the same independently held access') + a = p.parse_args() + os.umask(0o077) + fd = os.open(a.receipt, os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW, 0o600) + r = {'schema': 'netkingdom.recovery-kit.v1', 'status': 'failed', + 'secret_values_exported': False, 'new_grants_created': False} + try: + if a.offline_transfer_receipt and (not a.publish or a.auth_source != 'prompt'): + raise offsite.Refused() + files = build() + directory = Path(a.output_dir) + directory.mkdir(mode=0o700, exist_ok=False) + for name, raw in files.items(): + (directory / name).write_bytes(raw) + r.update(status='built', files=[{'name': n, 'bytes': len(v), 'sha256': hashlib.sha256(v).hexdigest()} + for n, v in files.items()]) + if a.publish: + if a.auth_source == 'prompt': + with open('/dev/tty', 'r+') as terminal: + if not terminal.isatty(): + raise offsite.Refused() + secret = getpass.getpass('Independently held Storage Share Backup app password: ', stream=terminal) + else: + spec = importlib.util.spec_from_file_location('inventory', HERE / 'inventory-storage-share.py') + owner = importlib.util.module_from_spec(spec) + spec.loader.exec_module(owner) + _, credentials = owner.caller_credentials() + secret = credentials['BACKUP_PASSWORD'] + if not secret or any(c in secret for c in '\r\n\x00'): + raise offsite.Refused() + try: + headers = {'Authorization': 'Basic ' + base64.b64encode(('Backup:' + secret).encode()).decode(), + 'Accept-Encoding': 'identity'} + r.update(status='published_readback_verified', files=publish(files, headers), + folder='Backup/railiance-backups/net-kingdom-identity', + provider='Storage Share', credential_source=a.auth_source) + if a.offline_transfer_receipt: + transfer_fd = os.open(a.offline_transfer_receipt, os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW, 0o600) + transfer = {'status': 'failed', 'error': 'offline_retrieval_failed_private_diagnostics_suppressed'} + try: + record = json.loads((ROOT / FILES[2]).read_text()) + transfer = offline.retrieve(record, secret, 'account') + finally: + with os.fdopen(transfer_fd, 'w') as out: + json.dump(transfer, out, indent=2) + out.write('\n') + r['offline_retrieval_verified'] = transfer['status'] == 'readback_verified' + finally: + secret = None + r['finished_at'] = datetime.now(timezone.utc).isoformat() + except (Exception, KeyboardInterrupt): + r.update(status='failed', error='kit_operation_failed_private_diagnostics_suppressed') + finally: + with os.fdopen(fd, 'w') as output: + json.dump(r, output, indent=2) + output.write('\n') + # Silent: safe to use in the existing captured attended-owner conduit. + return int(r['status'] == 'failed') + + +if __name__ == '__main__': + sys.exit(main()) diff --git a/tools/bootstrap-recovery/retrieve-identity-offline.py b/tools/bootstrap-recovery/retrieve-identity-offline.py new file mode 100644 index 0000000..e546269 --- /dev/null +++ b/tools/bootstrap-recovery/retrieve-identity-offline.py @@ -0,0 +1,138 @@ +#!/usr/bin/env python3 +"""Download one reviewed ciphertext using privately entered provider access. + +No OpenBao, SSO, Kubernetes, credential discovery, provider writes or age key. +Run by the operator in their own terminal, never through a recorded agent PTY. +""" +import argparse +import base64 +from datetime import datetime, timezone +import getpass +import hashlib +import importlib.util +import json +import os +from pathlib import Path +import re +import shutil +import signal +import sys +import tempfile +import urllib.request + +HERE = Path(__file__).resolve().parent +spec = importlib.util.spec_from_file_location('offsite', HERE / 'identity-offsite.py') +offsite = importlib.util.module_from_spec(spec) +spec.loader.exec_module(offsite) +ACCOUNT_BASE = offsite.HOST + '/remote.php/dav/files/Backup/railiance-backups/net-kingdom-identity/' + + +def validate(record): + if (record.get('schema') != 'netkingdom.identity-offsite.v1' + or record.get('status') != 'readback_verified' + or record.get('provider') != 'Storage Share' + or record.get('folder') != 'Backup/railiance-backups/net-kingdom-identity' + or not offsite.NAME.fullmatch(record.get('artifact', '')) + or not re.fullmatch('[a-f0-9]{64}', record.get('ciphertext_sha256', '')) + or type(record.get('ciphertext_bytes')) is not int + or not 0 < record['ciphertext_bytes'] <= offsite.LIMIT + or not re.fullmatch(r'"[^"\x00-\x20]{1,180}"', record.get('etag', ''))): + raise offsite.Refused() + + +def retrieve(record, secret, mode, client=None): + validate(record) + if mode not in {'account', 'folder-share'} or not secret or any(c in secret for c in '\r\n\x00'): + raise offsite.Refused() + auth = offsite.headers(secret) if mode == 'folder-share' else { + 'Authorization': 'Basic ' + base64.b64encode(('Backup:' + secret).encode()).decode(), + 'Accept-Encoding': 'identity'} + url = (offsite.BASE if mode == 'folder-share' else ACCOUNT_BASE) + record['artifact'] + client = client or offsite.opener() + directory = None + complete = False + try: + with client.open(urllib.request.Request(url, method='HEAD', headers=auth), timeout=30) as response: + if offsite.etag(response, url, record['ciphertext_bytes']) != record['etag']: + raise offsite.Refused() + directory = Path(tempfile.mkdtemp(prefix='nk49-identity-fetched-', dir='/tmp')) + target = directory / record['artifact'] + auth['If-Match'] = record['etag'] + count = 0 + digest = hashlib.sha256() + with client.open(urllib.request.Request(url, headers=auth), timeout=180) as response: + if offsite.etag(response, url, record['ciphertext_bytes']) != record['etag']: + raise offsite.Refused() + fd = os.open(target, os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW, 0o600) + with os.fdopen(fd, 'wb') as output: + while block := response.read(1024 * 1024): + count += len(block) + if count > record['ciphertext_bytes']: + raise offsite.Refused() + digest.update(block) + output.write(block) + output.flush() + os.fsync(output.fileno()) + if count != record['ciphertext_bytes'] or digest.hexdigest() != record['ciphertext_sha256']: + raise offsite.Refused() + with client.open(urllib.request.Request(url, method='HEAD', headers=auth), timeout=30) as response: + if offsite.etag(response, url, record['ciphertext_bytes']) != record['etag']: + raise offsite.Refused() + complete = True + # Closed allowlist: never copy arbitrary source-receipt fields to output. + result = {k: record[k] for k in ('schema', 'provider', 'folder', 'artifact', + 'ciphertext_bytes', 'ciphertext_sha256', 'etag')} + result.update(status='readback_verified', fetched_path=str(target), + independent_owner_retrieval_verified=True, + credential_source='independently_held_provider_access', + provider_auth_mode=mode, openbao_contacted=False, sso_contacted=False, + provider_writes_performed=False, age_private_key_used=False, + secret_values_exported=False, object_unchanged_during_transfer=True, + downloaded_at=datetime.now(timezone.utc).isoformat()) + return result + finally: + if directory is not None and not complete: + shutil.rmtree(directory) + + +def interrupted(signum, frame): + raise InterruptedError() + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('--source-receipt', required=True) + parser.add_argument('--receipt', required=True) + parser.add_argument('--auth', choices=('account', 'folder-share'), default='account') + args = parser.parse_args() + os.umask(0o077) + fd = os.open(args.receipt, os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW, 0o600) + result = {'status': 'failed', 'error': 'offline_retrieval_failed_private_diagnostics_suppressed'} + try: + record = json.loads(Path(args.source_receipt).read_text()) + validate(record) + # getpass must never fall back to echoed stdin (e.g. a captured agent tool). + with open('/dev/tty', 'r+') as terminal: + if not terminal.isatty(): + raise offsite.Refused() + secret = getpass.getpass('Storage Share Backup app password or exact folder-share token: ', stream=terminal) + try: + result = retrieve(record, secret, args.auth) + finally: + secret = None + except (Exception, KeyboardInterrupt): + pass + finally: + with os.fdopen(fd, 'w') as output: + json.dump(result, output, indent=2) + output.write('\n') + ok = result['status'] == 'readback_verified' + print('OK: ciphertext retrieved without OpenBao or SSO; ETag, size and SHA256 verified.' if ok + else 'ERROR: retrieval refused; private diagnostics suppressed. Use a new receipt path to retry.') + return 0 if ok else 1 + + +if __name__ == '__main__': + for sig in (signal.SIGINT, signal.SIGTERM, signal.SIGHUP): + signal.signal(sig, interrupted) + sys.exit(main()) diff --git a/tools/bootstrap-recovery/run-offline-recovery.sh b/tools/bootstrap-recovery/run-offline-recovery.sh new file mode 100644 index 0000000..8e4d82a --- /dev/null +++ b/tools/bootstrap-recovery/run-offline-recovery.sh @@ -0,0 +1,33 @@ +#!/usr/bin/env bash +# Run in the owner's own terminal. Provider password and age key are hidden inputs. +set +x +set -euo pipefail +ROOT="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." && pwd)" +cd -- "$ROOT" +umask 077 +[[ -t 0 && -t 1 ]] || { printf '%s\n' 'ERROR: run in your own interactive terminal.'; exit 1; } +for tool in python3 age age-keygen openssl docker; do + command -v "$tool" >/dev/null || { printf 'ERROR: required tool unavailable: %s\n' "$tool"; exit 1; } +done +docker image inspect 'postgres@sha256:ff23cdce56cac62ada6f66013e1a50864c0abbe79d132d40b6e05bd80f378a70' >/dev/null 2>&1 || { + printf '%s\n' 'ERROR: stage the pinned PostgreSQL image before this drill.'; exit 1; +} +NK49_HANDOFF_DIR="$(mktemp -d /tmp/nk49-offline-handoff-XXXXXXXX)" +printf 'Recovery evidence directory: %s\n' "$NK49_HANDOFF_DIR" +printf '%s\n' 'This publishes three new versioned guide/kit/checksum files beside the backups, then downloads the protected recovered generation.' +printf '%s\n' 'Enter the independently held Backup account app password at the next hidden prompt. No OpenBao or SSO login is used.' +python3 tools/bootstrap-recovery/recovery-kit.py \ + --output-dir "$NK49_HANDOFF_DIR/kit" --publish \ + --receipt "$NK49_HANDOFF_DIR/publication.json" \ + --offline-transfer-receipt "$NK49_HANDOFF_DIR/transfer.json" || { + printf '%s\n' 'ERROR: publication/retrieval stopped. Inspect the safe receipts; do not paste credentials into chat.'; exit 1; + } +printf '%s\n' 'Guide publication and outage-path retrieval passed. Next enter the independently held railiance-backup age key.' +bash sso-mfa/bootstrap/sops-custody-unlock.sh \ + --expected-recipient age1zvryunvjhvpkmasskauga2heeg0ztnte9ymgppvjge36ekumk50syr3tsz \ + --prompt -- \ + python3 tools/bootstrap-recovery/verify-identity.py \ + --transfer-receipt "$NK49_HANDOFF_DIR/transfer.json" \ + --receipt "$NK49_HANDOFF_DIR/recovery.json" +printf 'OK: private-provider retrieval and archive recovery passed. Safe receipts: %s\n' "$NK49_HANDOFF_DIR" +printf '%s\n' 'Service authentication remains separate acceptance. Return only the evidence directory path and outcome, never credentials.' diff --git a/workplans/NK-WP-0049-bootstrap-secret-free-recovery-handoff.md b/workplans/NK-WP-0049-bootstrap-secret-free-recovery-handoff.md index 989a408..5185f7f 100644 --- a/workplans/NK-WP-0049-bootstrap-secret-free-recovery-handoff.md +++ b/workplans/NK-WP-0049-bootstrap-secret-free-recovery-handoff.md @@ -28,10 +28,11 @@ These are retrospective task records of evidenced work, not additional delivery or newly granted authority. The earlier dated narrative remains in Git at `cc2dd01`; its evidence references are carried into the tasks below. -**Current position:** 12 tasks done; 1 in progress; 3 ready to start (`todo`); -8 waiting on explicit prerequisites. These are task counts, not an effort-based +**Current position:** 14 tasks done; 2 in progress; 1 ready to start (`todo`); +10 waiting on explicit prerequisites. These are task counts, not an effort-based completion percentage. Assignees are also written in task descriptions because -the current Forge projector does not populate the dedicated Hub assignee field. No remaining task is being executed by this replan. +the current Forge projector does not populate the dedicated Hub assignee field. The outage-retrieval tool and restore-packet draft are now prepared; +owner execution/publication and service admission remain explicit gates. **Next actions:** finish the read-only caller ledger (T02); prepare the isolated restore packet (T15), independent offline retrieval (T23), and residual-scope @@ -369,8 +370,8 @@ state_hub_task_id: "479d6970-6516-50f5-9436-bb5b56d58aad" the named owner boundary. The source inventory and known legacy objects exist; a complete current-use -ledger is not yet accepted. Next action is read-only reconciliation of the 87 -source entries against active Make/bootstrap/operator routes and owner evidence. +ledger is not yet accepted. Next action is read-only reconciliation of the 93 +entries in the latest October15 source inventory (the earlier snapshot had 87) against active Make/bootstrap/operator routes and owner evidence. **Done when:** each entry is identified as current, retired or owner-deferred, with its owner, credential route, current consumer(s), native evidence or exact @@ -385,21 +386,26 @@ T22. Do not reopen completed source repairs without a new finding. ```task id: NK-WP-0049-T15 -status: todo +status: progress priority: high assignee: codex -depends_on: [NK-WP-0049-T08, NK-WP-0049-T13] +depends_on: [NK-WP-0049-T08, NK-WP-0049-T13, NK-WP-0049-T27] state_hub_task_id: "1d1eb903-a4b0-5bc4-a421-0b842d7f68ae" ``` **Accountable worker:** codex. Service/credential-owner acceptance remains with the named owner boundary. -**Next action:** turn the recovered manifest and current caller gaps into a -reviewable procedure with selected immutable images, restore order, isolated -endpoints/network, data/key ownership, cleanup and per-service positive/negative -checks. Preparation can start with the existing archive; incorporate T02's ledger -before submitting current-caller acceptance. +**Prepared draft scope moved to T27.** See +`docs/identity-service-restore-packet.md`: restore order, containment, safe evidence +and per-service positive/negative checks are drafted. Read-only current runtime +image digests were observed for all four services; correspondence to the selected +archive and independent image availability are not yet proved. + +**Next action:** admit a named disposable target/private route, verify and stage +matching images independently, obtain private existing-user/MFA test inputs and +review version-specific native commands with service owners. Incorporate T02's +ledger before submitting current-caller acceptance. **Done when:** service owners accept the exact targets/procedure, existing-user and MFA test identities/input custody are available without posting secrets, @@ -585,22 +591,29 @@ source retirement as provider revocation. ```task id: NK-WP-0049-T23 -status: todo +status: wait priority: high assignee: codex -depends_on: [NK-WP-0049-T13] +depends_on: [NK-WP-0049-T13, NK-WP-0049-T25] +blocking_reason: "Await the operator-run private independent-provider download and hidden-key recovery drill; bounded tooling and self-contained kit are prepared in T25." state_hub_task_id: "1ec537ee-dfee-5576-be36-a81bdba417d1" ``` **Accountable worker:** codex. Service/credential-owner acceptance remains with the named owner boundary. -The existing independent retrieval succeeded through a healthy attended OpenBao -path. A full custody outage still lacks a reviewed offline provider adapter. +The previous independent retrieval succeeded through a healthy attended OpenBao +path. Source/fixture preparation of a separate offline adapter is now complete +in T25. Actual independently held provider access has not yet been exercised +for this outage path; no completed drill is claimed. -**Next action:** prepare bounded read-only download/receipt tooling using -independently held provider access in private operator input, with fixed provider, -artifact/ETag/size/digest guards, sanitized output and no standing agent secret. +**Unblock action:** operator runs the prepared private terminal handoff +`bash tools/bootstrap-recovery/run-offline-recovery.sh` (combined guide +publication and known-generation retrieval) or the standalone read-only +`retrieve-identity-offline.py` command in the guide. Independently held provider +access and age key go only into the respective hidden prompts; inspect safe +receipts and perform the real archive recovery check. Never use a captured agent +PTY for secret entry. T26 separately tracks actual guide publication. **Done when:** the owner can retrieve the exact off-host bundle without relying on the failed OpenBao/SSO instance, run the existing hidden-key checker, and @@ -633,6 +646,91 @@ an external watchdog are not implemented; do not add their grants/jobs implicitl This task triages the new insights instead of silently expanding NK49's required implementation to an entire disaster-recovery programme. +## Deliver and verify the offline retrieval adapter and recovery tools kit + +```task +id: NK-WP-0049-T25 +status: done +priority: high +assignee: codex +depends_on: [NK-WP-0049-T13] +``` + +**Accountable worker:** codex. + +Completed 2026-10-16. `retrieve-identity-offline.py` uses privately entered +independent Backup-account access or an exact read-capable identity-folder share. +It performs only fixed-provider HEAD/GET, binds the reviewed generation by strong +ETag/length/SHA256, refuses redirects/proxies/unreviewed scope and removes failed +downloads. No OpenBao/SSO, age key or standing agent credential is required. +`recovery-kit.py` builds a reproducible allowlisted ZIP containing the tools, +public profile, known-good ciphertext receipt and recovery guides, with file +checksums. Its publication path creates only three versioned guide/ZIP/checksum +objects and verifies readback without overwriting existing conflicting objects. +`run-offline-recovery.sh` provides the owner's two-hidden-prompt handoff. + +Evidence: `tests/test_identity_offline_recovery.py` and the bounded test/build +receipt in `docs/evidence/2026-10-16-identity-outage-recovery/`. The focused run +passed 65 tests including real disposable age/PostgreSQL fixtures. ZIP-contained +entry points import without sibling repositories. This is source/fixture proof, +not actual independent-provider retrieval (T23) or publication (T26). + +## Publish and verify the disaster recovery guide on Storage Share + +```task +id: NK-WP-0049-T26 +status: wait +priority: high +assignee: codex +depends_on: [NK-WP-0049-T25, NK-WP-0049-T27] +blocking_reason: "Await private operator provider-access execution of the prepared publication command and actual three-object readback receipts; guide and kit are built." +``` + +**Accountable worker:** codex; operator holds provider access privately. + +Requested by the operator on 2026-10-16. Guide source: +`docs/disaster-recovery-storage-share.md`. Document scope, independently held +provider/key requirements, exact known-good fallback, copy/download verification, +archive-check commands, service restore order, recurring schedule and remaining +coverage limits. Include the offline tools so failed Forgejo does not prevent +access to the recovery scripts; retain an independent copy and trusted digest. + +**Unblock action:** operator runs `run-offline-recovery.sh`, or the reviewed +`recovery-kit.py --publish` command through the existing attended Backup-account +owner conduit. Existing publisher/retention workload grants are not repurposed. +The user authorized this documentation upload; no new grant is required by the +prepared command. Credentials remain private; no provider permissions are widened. + +**Done when:** the versioned START-HERE guide, self-contained ZIP and ZIP SHA256 +file are present beside the identity backups with exact conditional-create and +ETag/length/content readback proof, and their folder/name/digests are recorded for +future discovery. Local build or attempted upload alone does not close this task. + +## Draft the isolated service restore packet and observe candidate images + +```task +id: NK-WP-0049-T27 +status: done +priority: high +assignee: codex +depends_on: [NK-WP-0049-T08, NK-WP-0049-T13] +``` + +**Accountable worker:** codex. + +Completed 2026-10-16. Draft in `docs/identity-service-restore-packet.md` names the +known recovered input, restore order, original key/data ownership, default-deny +containment, blocked production/provider/notification effects, cleanup and five +native acceptance lanes with positive/negative checks. Read-only metadata from +the pinned primary cluster resolved all four current service image digests, +including the separately observed privacyIDEA main container and LLDAP's mutable +source tag. Evidence: `docs/evidence/2026-10-16-identity-outage-recovery/image-observation.json`. + +This task extracts completed preparation from T15. Named target admission, +archive/runtime image-lineage matching and independent image availability, +private existing-user/MFA inputs and owner-reviewed native commands remain T15; +no service restore, production change or owner acceptance is claimed. + ## Accept remaining recovery and caller evidence and close the G08 handoff ```task @@ -640,7 +738,7 @@ id: NK-WP-0049-T03 status: wait priority: high assignee: codex -depends_on: [NK-WP-0049-T02, NK-WP-0049-T16, NK-WP-0049-T17, NK-WP-0049-T18, NK-WP-0049-T19, NK-WP-0049-T20, NK-WP-0049-T21, NK-WP-0049-T22, NK-WP-0049-T23, NK-WP-0049-T24] +depends_on: [NK-WP-0049-T02, NK-WP-0049-T16, NK-WP-0049-T17, NK-WP-0049-T18, NK-WP-0049-T19, NK-WP-0049-T20, NK-WP-0049-T21, NK-WP-0049-T22, NK-WP-0049-T23, NK-WP-0049-T24, NK-WP-0049-T26] blocking_reason: "Await the separately tracked native service proofs, current/legacy caller acceptance and independent outage retrieval; full archive recovery and backup activation are already complete." state_hub_task_id: "d68cd4d3-1ba3-530e-88f9-26ebb8dd6eac" ``` @@ -652,7 +750,7 @@ Owner boundary: NetKingdom, Platform and affected service owners, with Custodian G08 receiving the final evidence. The completed archive/job milestones remain accepted independently of this final gate. -**Done when:** T02 and T15–T24 have their stated completed or explicitly accepted +**Done when:** T02 and T15–T27 have their stated completed or explicitly accepted handoff outcomes; service authentication/token/audit proof and current/legacy caller dispositions are accepted; the source workplan and projected tasks agree; and G08 has an exact evidence/handoff record. Record actionable residuals as live