coulomb/net-kingdom
5
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Prepare independent outage recovery and Storage Share documentation

Browse source 
Assistant: codex
Assistant-Model: gpt-6-astra
Assistant-Session: 01a13eab-1e6a-7b33-9c37-372e90ada3a2
This commit is contained in:
tegwick 2026-10-16 17:59:18 +02:00
parent 2a3800e7f1
commit 1764f40b43
12 changed files with 1050 additions and 22 deletions
Download patch file Download diff file Expand all files Collapse all files

6
docs/bootstrap-recovery-ceremony.md
View file

@ -7,6 +7,12 @@ continuity and current/legacy caller acceptance remain open for NK-WP-0049.
Approved execution is recorded in decisions40bfae40 and5edd5203; this document
records their scope and evidence.
**Outage quick start:** [Storage Share disaster recovery guide](disaster-recovery-storage-share.md).
The independently prompted provider download and offline tools ZIP are prepared;
actual outage-path retrieval and guide publication are separately tracked in
T23/T26. The [isolated service restore packet](identity-service-restore-packet.md)
contains the draft native-service sequence and remaining T15 admission inputs.
## What has been established
On 2026-10-15 the operator identified Scaleway and Storage Share as the off-host

136
docs/disaster-recovery-storage-share.md Normal file
View file

@ -0,0 +1,136 @@
# NetKingdom disaster recovery — start here
Updated 2026-10-16. Owner: NetKingdom with Railiance Platform and the individual
identity service owners. This guide covers the identity backup, not whole-host,
OpenBao, Forgejo, User Engine or application recovery.
## If the identity services are down
1. Use a trusted recovery workstation. Obtain Storage Share access independently
from your password safe/offline custody, and the independently held
`railiance-backup` age private key. Never paste either into chat or a ticket.
The private key must match this **public** recipient:
`age1zvryunvjhvpkmasskauga2heeg0ztnte9ymgppvjge36ekumk50syr3tsz`.
The provider password and encryption key are separate requirements.
2. Open <https://nx4069.your-storageshare.de> using the independent provider
account, not NetKingdom SSO. In the Backup account, navigate to
`railiance-backups/net-kingdom-identity/`. Recovery documentation and a small
versioned recovery-tools ZIP are placed beside the encrypted archives.
Keep a copy of this guide and ZIP outside Storage Share as well.
3. Choose an intact recovery generation. The full identity filename starts
`net-kingdom-identity-` and ends `.tar.age`. A `privacyidea_db-*.dump.age`
database-only backup cannot replace the matching service keys and configs.
Each generation has a `.receipt.json` index. Preserve the original objects.
4. Check timestamp, size and SHA256 against a separately trusted receipt. The
index is useful metadata, not a signed independent trust anchor. For the
first actually recovered, protected generation the trusted values are:
- Artifact: `net-kingdom-identity-20261016T063159Z-5882771aa2ce.tar.age`
- Bytes: `109738344`
- SHA256: `7117b3fe1e4840c86b301c94c85e6b16709e5626bb0884bd0580bde630e30e30`
- Strong ETag at publication: `"28ec1f7014e492de4c40e1ae75502f08"`
This is a known good fallback, not a claim that it is the latest backup.
For a newer generation, reconcile its index with the independently retained
publisher receipt before constructing a new download input. Never edit a
receipt just to make a failed integrity check pass. ETag changes require
owner reconciliation even if the filename is unchanged.
5. On Linux with Python 3, unpack the recovery-tools ZIP into a private working
directory and enter it. Verify its `SHA256SUMS` against a trusted repository
copy or independently held kit digest before executing code. A checksum
stored only alongside the download detects damage, not a malicious provider.
Alternatively use an independently available trusted net-kingdom checkout.
```bash
python3 tools/bootstrap-recovery/retrieve-identity-offline.py \
--source-receipt docs/evidence/2026-10-16-identity-backup-live/actual-offsite.json \
--auth account --receipt /tmp/nk49-outage-transfer.json
```
In your own terminal, enter the independently held **Backup** account app
password at the hidden prompt. The fixed account name is `Backup`.
If your independently held credential is a read-capable public share token
for this exact identity folder, use `--auth folder-share` instead. An
upload-only token, another account's password or a share for the backup root
is insufficient for this command. A 401/403/404 stops; do not widen permissions
or substitute another token silently. Download performs only HEAD/GET against
the fixed provider, rejects redirects/proxies, and verifies ETag/size/hash.
Failure removes its incomplete local download. Success leaves ciphertext
in a private `/tmp/nk49-identity-fetched-*` directory named in the receipt.
6. Verify and decrypt through the existing hidden key helper. Requirements:
`age`, `age-keygen`, `openssl`, Python 3 and Docker; the following PostgreSQL
image must already be present on the recovery host:
`postgres@sha256:ff23cdce56cac62ada6f66013e1a50864c0abbe79d132d40b6e05bd80f378a70`.
Stage and verify this image before a drill. The tools ZIP contains no images.
Allow enough private disk and RAM for a roughly 110 MB ciphertext, a bounded
512 MiB archive and the disposable PostgreSQL container (2 GiB memory limit).
```bash
bash sso-mfa/bootstrap/sops-custody-unlock.sh \
--expected-recipient age1zvryunvjhvpkmasskauga2heeg0ztnte9ymgppvjge36ekumk50syr3tsz \
--prompt -- \
python3 tools/bootstrap-recovery/verify-identity.py \
--transfer-receipt /tmp/nk49-outage-transfer.json \
--receipt /tmp/nk49-outage-archive-recovery.json
```
Use new receipt names on every run. The helper removes its temporary private
key on exit. The verifier checks the encrypted manifest and all file hashes,
original key pairs, SQLite integrity, wrong-key refusal and an isolated
PostgreSQL restore. It removes decrypted staging and the disposable database.
It deliberately does not install restored secrets or alter production.
7. Preserve the sanitized receipts and original ciphertext. Remove your local
downloaded ciphertext directory when no longer needed. If the verifier fails,
stop and check the bounded receipt with the service owner; do not regenerate
encryption/signing keys, reset user passwords or delete the provider backup.
## What the backup contains
| Component | Recovery material |
| --- | --- |
| LLDAP | Native SQLite snapshot, remaining durable data, deployment and referenced secrets |
| Authelia | Native SQLite snapshot, encryption/session/OIDC material, current configuration and referenced secrets |
| KeyCape | Original signing/configuration material, client configuration, authentication policies and receipts |
| privacyIDEA | Fresh PostgreSQL dump, original encryption key, matching original audit RSA key pair, current configuration and required durable files |
The encrypted manifest records exact files and capture intervals. This is not
a global atomic snapshot of all services. Historical backups and selected
notification/log files are excluded as documented by the capture profile.
The backup contains sensitive service credentials; only ciphertext belongs on
ordinary persistent recovery media. The age private key is never in the bundle.
## Restoring working services
**Archive recovery passed on 2026-10-16. Full restored-service acceptance remains
open.** The archive verifier deletes its plaintext and is not a service importer.
Follow `docs/identity-service-restore-packet.md` with service owners before an
isolated service drill. Its pending target and image fields must be resolved.
Restore the exact matching data/config/original keys into an isolated recovery
environment first. Keep production ingress, public DNS, notifications, external
provider writes and real consumer traffic disconnected. Establish directory and
database readiness; prove LLDAP existing-user authentication; then privacyIDEA
retained-token MFA/audit continuity, Authelia storage/OIDC, and KeyCape/current
consumer continuity according to their actual dependencies. Existing user/MFA
inputs stay in operator custody. Public issuer identity must be preserved while
test traffic is routed privately; do not change production DNS for a drill.
Only after owner acceptance of native positive/negative tests should the owners
approve a production cutover, including target identity, rollback, DNS/ingress,
current consumers, credential dispositions and monitoring. This guide does not
authorize a production restore or automatic credential rotation.
## Schedule, protection and limits
Daily capture is scheduled at 03:45 UTC with up to five minutes jitter. Hourly
host-local checks flag a backup older than 25 hours. Retention protects the
independently recovered pinned generation, newest generation, seven distinct
daily and two weekly points; unknown/unindexed objects are not auto-deleted.
At this revision the full identity backup has one provider copy on Storage Share.
Existing Scaleway database backups do not establish a second full identity copy.
Host-local health cannot report a dead host. Full service login/MFA/audit proofs,
an independently executed outage-path retrieval drill, a second provider copy
and external host-loss monitoring must not be assumed from an upload or this
guide. Track remaining acceptance in NK-WP-0049; current task state lives in the
repository and may be newer than this offline guide.

97
docs/evidence/2026-10-16-identity-outage-recovery/image-observation.json Normal file
View file

@ -0,0 +1,97 @@
{
"schema": "netkingdom.identity-recovery-image-observation.v1",
"observed_at": "2026-10-16T15:56:10.384927+00:00",
"cluster_uid": "a553c742-0115-43d4-99a4-a5ca56fe0786",
"read_only": true,
"secret_values_read": false,
"archive_generation_match_proven": false,
"independent_image_availability_proven": false,
"pods": [
{
"namespace": "sso",
"pod": "authelia-7f57c4c4fc-mk5hb",
"pod_uid": "7678e756-0af9-4eb9-bc25-c8b09ca490fc",
"owners": [
{
"kind": "ReplicaSet",
"name": "authelia-7f57c4c4fc",
"uid": "de04027d-ebff-45bc-8aff-73583907b57e"
}
],
"containers": [
{
"name": "authelia",
"image_id": "docker.io/authelia/authelia@sha256:bd97cff4fcbf715b5ff1f9ae286afbe6033afce385302520b0368122d43a6f54",
"ready": true
}
]
},
{
"namespace": "sso",
"pod": "keycape-5fc5dd9b7f-zv5xm",
"pod_uid": "bf0d4674-cbe4-4e44-b105-b799eb4c124e",
"owners": [
{
"kind": "ReplicaSet",
"name": "keycape-5fc5dd9b7f",
"uid": "8d862e2d-532c-4f1f-b510-cbe489ec579b"
}
],
"containers": [
{
"name": "keycape",
"image_id": "forgejo.coulomb.social/coulomb/key-cape@sha256:cdec9dd7c515f7e0a6169fc3fa560ee2cc47bf40d899755dec808c362f5ce010",
"ready": true
}
]
},
{
"namespace": "sso",
"pod": "lldap-f6df76764-jjp97",
"pod_uid": "b05329a6-8b9d-45d6-a573-c03c04e667be",
"owners": [
{
"kind": "ReplicaSet",
"name": "lldap-f6df76764",
"uid": "306a97b4-a5f7-421f-971d-36a3dd5fbc47"
}
],
"containers": [
{
"name": "lldap",
"image_id": "docker.io/lldap/lldap@sha256:9e605a66c02514bfcffd1b67cafb1e98d50992216bb2871d7ae44622047dd09d",
"ready": true
}
]
},
{
"namespace": "mfa",
"pod": "privacyidea-86b695c88c-9kh7g",
"pod_uid": "2a44aa01-84e0-4778-a7c3-d0597b192259",
"owners": [
{
"kind": "ReplicaSet",
"name": "privacyidea-86b695c88c",
"uid": "7db5316a-0b89-431e-a678-a5dad372fc9c"
}
],
"containers": [
{
"name": "factor-recovery",
"image_id": "ghcr.io/gpappsoft/privacyidea-docker@sha256:af7841adad262f129e0c1d4f553af13f21cb2f4dc713533f316cfe43ed0b4473",
"ready": true
},
{
"name": "privacyidea",
"image_id": "ghcr.io/gpappsoft/privacyidea-docker@sha256:af7841adad262f129e0c1d4f553af13f21cb2f4dc713533f316cfe43ed0b4473",
"ready": true
}
]
}
],
"knowledge": {
"version": "0.2.6",
"package_digest": "8d1d6a9d4d07fec7b619618de7dfe2be2254c1790f85f4198c7fe19db9751c02",
"source_status": "withheld; current SECURITY-COMPANION and owner route sources reviewed directly; orientation only"
}
}

66
docs/evidence/2026-10-16-identity-outage-recovery/preparation-review.json Normal file
View file

@ -0,0 +1,66 @@
{
"schema": "netkingdom.identity-outage-preparation.v1",
"date": "2026-10-16",
"status": "source_fixture_verified_owner_execution_pending",
"tests": {
"command": "NK49_RUN_DISPOSABLE_POSTGRES_TEST=1 python3 -m pytest -q tests/test_identity_offline_recovery.py tests/test_identity_offsite.py tests/test_identity_bundle.py tests/test_privacyidea_offhost_recovery.py",
"passed": 65,
"failed": 0,
"skipped": 0
},
"kit_preview": {
"schema": "netkingdom.recovery-kit.v1",
"status": "built",
"secret_values_exported": false,
"new_grants_created": false,
"files": [
{
"name": "net-kingdom-recovery-62b9464609b8e0fd-START-HERE.md",
"bytes": 8381,
"sha256": "1531675f2e7d3713a3a7fc28a5143fb24f201b19cd05d14c59ee485a0bc11e61"
},
{
"name": "net-kingdom-recovery-62b9464609b8e0fd.zip",
"bytes": 34527,
"sha256": "62b9464609b8e0fde2c05527fc79eff742ee3f626262d847df8fb579548a9470"
},
{
"name": "net-kingdom-recovery-62b9464609b8e0fd.zip.sha256",
"bytes": 108,
"sha256": "55e375291b27c2977049b89a5deddaed5e39c7ee9179013ce73d2ad43f2fb308"
}
],
"finished_at": "2026-10-16T15:56:50.813341+00:00"
},
"sources": [
{
"path": "tools/bootstrap-recovery/retrieve-identity-offline.py",
"sha256": "80e9f755b2304c222822b49ab563600832b9452c0c387a0523f4323e591e6270"
},
{
"path": "tools/bootstrap-recovery/recovery-kit.py",
"sha256": "55df6283898fadae316b89304bacf485dec88459b35ea2b9599c5d387c44ca8c"
},
{
"path": "tools/bootstrap-recovery/run-offline-recovery.sh",
"sha256": "0e3cfe8cc66d63b91a5c3f90912ce700c648c55b685ac11151631d22d87b40fd"
},
{
"path": "docs/disaster-recovery-storage-share.md",
"sha256": "1531675f2e7d3713a3a7fc28a5143fb24f201b19cd05d14c59ee485a0bc11e61"
},
{
"path": "docs/identity-service-restore-packet.md",
"sha256": "34f90e0ee1425083b26a763777b0150241e1cd5fb7dbdf8917f7a7a99482376d"
},
{
"path": "tests/test_identity_offline_recovery.py",
"sha256": "d88a752fd862983e3c01ebe6aea4baffa4168802d8ec616db161585a80c65f62"
}
],
"actual_outage_retrieval_verified": false,
"actual_storage_share_guide_publication_verified": false,
"service_restore_executed": false,
"new_grants_created": false,
"secret_values_exported": false
}

97
docs/identity-service-restore-packet.md Normal file
View file

@ -0,0 +1,97 @@
# Isolated identity service restore packet
NK-WP-0049-T15. Prepared 2026-10-16; **draft, not admitted or executed**.
Accountable preparer: codex. Execution/acceptance: Railiance Platform and the
LLDAP, Authelia, KeyCape and privacyIDEA service owners.
## Fixed starting point
Use the actual recovered artifact and receipt in
`docs/evidence/2026-10-16-identity-backup-live/actual-offsite.json`, with matching
original keys and manifest. The completed archive recovery verified 36 file
hashes, SQLite integrity, RSA pairs and native PostgreSQL restore. It did not
prove service authentication or audit-signature continuity. A newer generation
needs equivalent provenance and archive checks before replacing this input.
## Targets and image admission still required
No execution target has been admitted. Select a disposable host/context and
record its immutable identity, operator, private endpoints, maximum lifetime,
resource limits and exact owned cleanup inventory. Never select the production
context or reuse live PVCs. Use fresh private storage; no production host mounts,
service-account tokens, ingress controller or external DNS update credentials.
The October 16 read-only pod observation is recorded in
`docs/evidence/2026-10-16-identity-outage-recovery/image-observation.json`.
These are candidate inputs, not confirmation of the images in the selected archive:
| Component | Candidate reference | Admission gap |
| --- | --- | --- |
| PostgreSQL | `postgres@sha256:ff23cdce56cac62ada6f66013e1a50864c0abbe79d132d40b6e05bd80f378a70` | Passed archive DB restore; stage independently of failed registry |
| LLDAP | `docker.io/lldap/lldap@sha256:9e605a66c02514bfcffd1b67cafb1e98d50992216bb2871d7ae44622047dd09d` | Actual running digest resolved; compare selected archive/runtime lineage and stage it |
| Authelia | `docker.io/authelia/authelia@sha256:bd97cff4fcbf715b5ff1f9ae286afbe6033afce385302520b0368122d43a6f54` | Compare selected archive deployment and stage image |
| KeyCape | `forgejo.coulomb.social/coulomb/key-cape@sha256:cdec9dd7c515f7e0a6169fc3fa560ee2cc47bf40d899755dec808c362f5ce010` | Compare selected archive; require independent image availability |
| privacyIDEA | `ghcr.io/gpappsoft/privacyidea-docker@sha256:af7841adad262f129e0c1d4f553af13f21cb2f4dc713533f316cfe43ed0b4473` | Actual main and sidecar digest observed separately; compare archive/runtime lineage and stage it |
Record image archive digests and offline load/inspect results. Resolve mutable
references from trusted runtime/image metadata rather than pulling today's tag.
The encrypted archive deployment records must be compared privately; do not
export literal environment values, Secret payloads or arbitrary configs.
## Restore order and containment
1. Freeze the selected artifact/manifest/image/target references. The owner
supplies existing-user credentials and an existing retained MFA token through
private inputs. Agree time synchronization and test identities without
exposing passwords, seed material, client secrets or user records.
2. Establish an isolated network with default-deny external egress, no published
ports, no host networking and no production DNS/ingress. Permit only named
internal service links. The test browser, if needed, must use a separately
reviewed local route and hosts mapping. Preserve issuer/client identity while
keeping all test traffic private. Verify denied external connection attempts.
3. Import manifest-checked data and original keys into fresh private storage.
Do not apply archived Kubernetes objects wholesale: metadata UIDs, mounts,
service accounts, endpoints and sidecars require explicit translation. Keep
notification/provider credentials sealed and unused; disable sending paths.
4. Restore the linked privacyIDEA database with native PostgreSQL first; restore
LLDAP data/config and verify directory readiness. Run directory authentication
before dependent provider checks. Restore privacyIDEA with its original enckey
and audit pair against the isolated database/directory. Restore Authelia with
original storage/OIDC keys and private directory routes. Restore KeyCape with
original signing/config/policies and private MFA/directory dependencies.
5. Run the individual acceptance checks below; do not start provisioning,
enrollment, rotation, scheduled jobs or production consumers automatically.
An unsupported dependency is a named gap, never an implicit production route.
6. Remove only the named disposable containers/network/storage and temporary
plaintext/key files. Confirm absence and production resources unchanged;
retain sanitized receipts, original ciphertext and independently held keys.
## Acceptance matrix
| Task / owner lane | Positive check | Negative check | Safe evidence |
| --- | --- | --- | --- |
| T16 / LLDAP | Preserved user/group/membership identities; existing-user bind/login | Wrong password rejected | Image/target IDs, aggregate comparisons and booleans |
| T17 / Authelia | Native storage decryption; same signing identity/client configs; isolated authentication | Wrong credentials or unregistered client refused | Public key thumbprint, test outcome, notification suppression |
| T19 / privacyIDEA MFA | Existing retained token decrypts internally and actual existing-token MFA succeeds | Wrong enckey and invalid code refused in disposable copies | Token material never exported; bounded outcomes only |
| T20 / privacyIDEA audit | Verify historical signatures; create and verify one isolated entry with original key | Mismatched public key/tampered disposable record rejected | Signature counts, continuity result, cleanup |
| T18 / KeyCape | Same issuer/JWKS, registry/policy/receipt continuity; named current consumer authenticates | Wrong secret and denied scope rejected | Public fingerprints, owner-selected test IDs and outcomes |
Do not silently reset credentials or re-enroll tokens to pass. Historical audit
verification failures must be reported, not hidden with replacement keys.
## Unresolved inputs before execution
- Named immutable disposable target, approved private browser route if required,
resource limits, cleanup inventory and operator window.
- Confirm archive/runtime lineage for the observed image digests and stage all
images independently of production and the failed registry.
- A retained user and MFA token available in private owner custody; exact native
verification commands reviewed against the chosen service versions.
- T02 current-caller ledger and T21 per-caller owner admission, including legacy
token, diagnostic, resolver and LDAP-group boundaries.
- Service-owner acceptance of network/config translations and all required
grants. New grants or production effects require their own owner decisions.
Until resolved, T15 stays in progress and T16–T20 remain waiting. Production
cutover, full-host/OpenBao restoration and application database recovery are
separate owner procedures.

18
registry/capabilities/capability.security.bootstrap-recovery.md
View file

@ -52,7 +52,8 @@ discovery:
privacyIDEA keys plus linked PostgreSQL dump
- Public-recipient age sealing, Storage Share create/readback, separate retention
and attended independent-key archive verification
- Railiance bootstrap handoff without requiring recovered SSO for off-host retrieval
- Railiance bootstrap handoff and a prepared independent-provider download path
requiring neither OpenBao nor SSO; actual outage-path owner drill remains pending
excludes:
- Source repairs and disposable native fixtures do not prove estate root isolation
or recovery
@ -69,6 +70,11 @@ availability:
- tools/bootstrap-recovery/identity-backup-profile.json
- tools/bootstrap-recovery/capture-identity.py
- tools/bootstrap-recovery/retrieve-identity.py
- tools/bootstrap-recovery/retrieve-identity-offline.py
- tools/bootstrap-recovery/recovery-kit.py
- tools/bootstrap-recovery/run-offline-recovery.sh
- docs/disaster-recovery-storage-share.md
- docs/identity-service-restore-packet.md
- tools/bootstrap-recovery/verify-identity.py
consumption_modes:
- CLI
@ -95,6 +101,7 @@ evidence:
- tests/test_identity_bundle.py
- tests/test_identity_offsite.py
- tests/test_identity_backup_preparation.py
- tests/test_identity_offline_recovery.py
consumer_guidance:
recommended_for:
- Operator recovers identity, custody or data with independently held recovery material
@ -140,3 +147,12 @@ Assessment: CUST-WP-0081, 2026-10-14. Relations identify interfaces to compose;
## Identity recovery and bootstrap handoff
Start with [the recovery ceremony](../../docs/bootstrap-recovery-ceremony.md). It names the exact capture, retrieval, hidden-key verification, activation gates and remaining full-service proof. Railiance bootstrap sequences the isolated recovery and layer handoff; NetKingdom owns identity assets and native acceptance. Discovery does not authorize grants or live restore.
For an outage, use [the Storage Share recovery guide](../../docs/disaster-recovery-storage-share.md)
and the self-contained kit built by `tools/bootstrap-recovery/recovery-kit.py`.
The private operator handoff publishes the versioned guide/kit with readback,
retrieves the protected generation using independently held provider access,
and prompts separately for the human-held age key. Source/fixture preparation
passed; consult NK-WP-0049-T23/T26 for actual outage drill/publication acceptance.
The [isolated service packet](../../docs/identity-service-restore-packet.md)
records the remaining service-owner admission and native tests.

155
tests/test_identity_offline_recovery.py Normal file
View file

@ -0,0 +1,155 @@
"""Outage download boundaries, failure cleanup and independently usable kit."""
import hashlib
import importlib.util
import io
import json
from pathlib import Path
import subprocess
import sys
import urllib.error
import zipfile
import pytest
ROOT = Path(__file__).resolve().parents[1]
def load(name, root=ROOT):
spec = importlib.util.spec_from_file_location(name, root / 'tools/bootstrap-recovery' / (name + '.py'))
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return module
m = load('retrieve-identity-offline')
k = load('recovery-kit')
class Response(io.BytesIO):
def __init__(self, url, raw, etag='"original"', status=200):
super().__init__(raw)
self.url = url
self.status = status
self.headers = {'Content-Length': str(len(raw)), 'ETag': etag}
class Provider:
def __init__(self, body):
self.body = body
self.calls = []
self.tamper = False
self.etag = '"original"'
self.objects = {}
def open(self, request, timeout):
method = request.get_method()
self.calls.append((method, request.full_url, dict(request.headers)))
assert request.full_url.startswith((m.ACCOUNT_BASE, m.offsite.BASE))
if self.objects or method == 'PUT':
if method == 'PUT':
assert request.get_header('If-none-match') == '*'
assert request.full_url not in self.objects
self.objects[request.full_url] = request.data
return Response(request.full_url, b'', status=201)
if request.full_url not in self.objects:
raise urllib.error.HTTPError(request.full_url, 404, 'missing', {}, None)
return Response(request.full_url, self.objects[request.full_url])
raw = self.body if not self.tamper or method == 'HEAD' else b'X' * len(self.body)
return Response(request.full_url, raw, self.etag)
@pytest.fixture
def record():
return dict(schema='netkingdom.identity-offsite.v1', status='readback_verified',
provider='Storage Share', folder='Backup/railiance-backups/net-kingdom-identity',
artifact='net-kingdom-identity-20261016T063159Z-5882771aa2ce.tar.age',
ciphertext_bytes=32, ciphertext_sha256=hashlib.sha256(b'A' * 32).hexdigest(),
etag='"original"')
@pytest.mark.parametrize('mode', ['account', 'folder-share'])
def test_offline_success_uses_only_fixed_provider_reads(record, mode):
provider = Provider(b'A' * 32)
record['untrusted_extra'] = 'must-not-export'
result = m.retrieve(record, 'fixtureSecret1234', mode, provider)
try:
assert Path(result['fetched_path']).read_bytes() == b'A' * 32
assert not result['openbao_contacted'] and not result['sso_contacted']
assert {c[0] for c in provider.calls} == {'HEAD', 'GET'}
assert 'fixtureSecret1234' not in json.dumps(result)
assert 'must-not-export' not in json.dumps(result)
assert provider.calls[1][2]['If-match'] == record['etag']
finally:
path = Path(result['fetched_path'])
path.unlink()
path.parent.rmdir()
@pytest.mark.parametrize('field,value', [
('artifact', '../another.tar.age'), ('folder', 'other'),
('ciphertext_sha256', 'x' * 64), ('ciphertext_bytes', True),
('provider', 'other'), ('etag', 'W/"weak"'),
])
def test_unreviewed_scope_refuses_before_contact(record, field, value):
record[field] = value
provider = Provider(b'A' * 32)
with pytest.raises(m.offsite.Refused):
m.retrieve(record, 'fixtureSecret1234', 'account', provider)
assert not provider.calls
def test_digest_failure_removes_incomplete_ciphertext(record, monkeypatch, tmp_path):
directory = tmp_path / 'download'
directory.mkdir()
monkeypatch.setattr(m.tempfile, 'mkdtemp', lambda **kwargs: str(directory))
provider = Provider(b'A' * 32)
provider.tamper = True
with pytest.raises(m.offsite.Refused):
m.retrieve(record, 'fixtureSecret1234', 'account', provider)
assert not directory.exists()
def test_changed_etag_refuses_before_download(record):
provider = Provider(b'A' * 32)
provider.etag = '"changed"'
with pytest.raises(m.offsite.Refused):
m.retrieve(record, 'fixtureSecret1234', 'account', provider)
assert len(provider.calls) == 1
def test_kit_contains_all_offline_imports_and_checksums(tmp_path):
files = k.build()
archive = next(value for name, value in files.items() if name.endswith('.zip'))
with zipfile.ZipFile(io.BytesIO(archive)) as kit:
kit.extractall(tmp_path)
for line in kit.read('SHA256SUMS').decode().splitlines():
digest, name = line.split(' ', 1)
assert hashlib.sha256((tmp_path / name).read_bytes()).hexdigest() == digest
for script in ('verify-identity.py', 'retrieve-identity-offline.py'):
result = subprocess.run([sys.executable, str(tmp_path / 'tools/bootstrap-recovery' / script), '--help'],
capture_output=True)
assert result.returncode == 0, result.stderr
assert k.build() == files
def test_kit_publication_conditional_and_exact_readback():
provider = Provider(b'')
# Ensure initial HEAD returns 404 instead of the download-fixture response.
provider.objects['unrelated'] = b'preserved'
files = k.build()
rows = k.publish(files, {'Authorization': 'fixture'}, provider)
assert len(rows) == 3 and all(row['readback_verified'] for row in rows)
assert len([c for c in provider.calls if c[0] == 'PUT']) == 3
k.publish(files, {'Authorization': 'fixture'}, provider)
assert len([c for c in provider.calls if c[0] == 'PUT']) == 3
assert provider.objects['unrelated'] == b'preserved'
def test_kit_conflicting_existing_object_never_overwritten():
provider = Provider(b'')
files = k.build()
first = next(iter(files))
provider.objects[m.ACCOUNT_BASE + first] = b'X' * len(files[first])
with pytest.raises(k.offsite.Refused):
k.publish(files, {'Authorization': 'fixture'}, provider)
assert not any(c[0] == 'PUT' for c in provider.calls)

14
tools/bootstrap-recovery/README.md
View file

@ -3,6 +3,20 @@
Owner work: NK-WP-0049. Follow
[the exact recovery ceremony](../../docs/bootstrap-recovery-ceremony.md).
For an identity/custody outage, start with the
[Storage Share disaster recovery guide](../../docs/disaster-recovery-storage-share.md).
`retrieve-identity-offline.py` performs fixed-provider ciphertext retrieval from
privately entered independent provider access, without OpenBao/SSO. Its source
receipt pins the object, strong ETag, size and digest; failed downloads are removed.
`recovery-kit.py` builds an allowlisted standalone tools ZIP and can publish only
three new versioned guide/ZIP/checksum files beside the backups with exact
readback. `run-offline-recovery.sh` is the owner's private two-prompt handoff for
publication, outage-path retrieval and the existing archive recovery check.
Run secret prompts in the owner's own terminal, never an agent-captured PTY.
Source/fixture checks do not replace the actual owner drill/publication receipts.
Service restore admission remains in the
[isolated restore packet](../../docs/identity-service-restore-packet.md).
`verify-bundle.py` validates a digest-bound age-encrypted `secrets/` tar archive
using an explicit private temporary key and registered `keys/age.pub`. It checks
an exact required-file list, unrelated wrong-key refusal, safe isolated file

172
tools/bootstrap-recovery/recovery-kit.py Normal file
View file

@ -0,0 +1,172 @@
#!/usr/bin/env python3
"""Build a secret-free offline recovery kit; optionally publish exact new files.
Publication creates only three fixed versioned objects beside identity backups,
never changes shares, credentials, existing files, retention or runtime jobs.
"""
import argparse
import base64
from datetime import datetime, timezone
import getpass
import hashlib
import importlib.util
import io
import json
import os
from pathlib import Path
import sys
import urllib.error
import urllib.request
import zipfile
ROOT = Path(__file__).resolve().parents[2]
HERE = Path(__file__).resolve().parent
spec = importlib.util.spec_from_file_location('offline', HERE / 'retrieve-identity-offline.py')
offline = importlib.util.module_from_spec(spec)
spec.loader.exec_module(offline)
offsite = offline.offsite
FILES = [
'docs/disaster-recovery-storage-share.md',
'docs/identity-service-restore-packet.md',
'docs/evidence/2026-10-16-identity-backup-live/actual-offsite.json',
'docs/evidence/2026-10-16-identity-outage-recovery/image-observation.json',
'tools/bootstrap-recovery/retrieve-identity-offline.py',
'tools/bootstrap-recovery/identity-offsite.py',
'tools/bootstrap-recovery/verify-identity.py',
'tools/bootstrap-recovery/identity-bundle.py',
'tools/bootstrap-recovery/check-privacyidea-database.py',
'tools/bootstrap-recovery/identity-backup-profile.json',
'sso-mfa/bootstrap/sops-custody-unlock.sh',
'sso-mfa/bootstrap/sops-private.py',
'keys/age.pub',
]
def build():
content = {name: (ROOT / name).read_bytes() for name in FILES}
if any(b'AGE-SECRET-KEY-1' in value for name, value in content.items()
if not name.endswith(('.py', '.sh', '.md'))):
raise offsite.Refused()
sums = ''.join(hashlib.sha256(value).hexdigest() + ' ' + name + '\n'
for name, value in content.items()).encode()
output = io.BytesIO()
with zipfile.ZipFile(output, 'w', compression=zipfile.ZIP_DEFLATED) as archive:
for name, value in {**content, 'SHA256SUMS': sums}.items():
entry = zipfile.ZipInfo(name, date_time=(2026, 10, 16, 0, 0, 0))
entry.external_attr = 0o100600 << 16
entry.compress_type = zipfile.ZIP_DEFLATED
archive.writestr(entry, value)
kit = output.getvalue()
version = hashlib.sha256(kit).hexdigest()
stem = 'net-kingdom-recovery-' + version[:16]
return {
stem + '-START-HERE.md': content['docs/disaster-recovery-storage-share.md'],
stem + '.zip': kit,
stem + '.zip.sha256': (version + ' ' + stem + '.zip\n').encode(),
}
def publish(files, headers, client=None):
client = client or offsite.opener()
result = []
for name, raw in files.items():
url = offline.ACCOUNT_BASE + name
digest = hashlib.sha256(raw).hexdigest()
# A retry may accept an identical object after exact readback, but must
# never overwrite a conflicting object or retry an unknown PUT outcome.
present = False
try:
with client.open(urllib.request.Request(url, method='HEAD', headers=headers), timeout=30) as response:
version = offsite.etag(response, url, len(raw))
present = True
except urllib.error.HTTPError as error:
if error.code != 404:
raise offsite.Refused() from None
if not present:
h = dict(headers, **{'If-None-Match': '*', 'Content-Length': str(len(raw)),
'Content-Type': 'application/zip' if name.endswith('.zip') else 'text/plain; charset=utf-8'})
with client.open(urllib.request.Request(url, data=raw, method='PUT', headers=h), timeout=60) as response:
if response.status != 201 or response.url != url:
raise offsite.Refused()
with client.open(urllib.request.Request(url, method='HEAD', headers=headers), timeout=30) as response:
version = offsite.etag(response, url, len(raw))
h = dict(headers, **{'If-Match': version})
with client.open(urllib.request.Request(url, headers=h), timeout=60) as response:
if offsite.etag(response, url, len(raw)) != version:
raise offsite.Refused()
downloaded = response.read(len(raw) + 1)
if downloaded != raw:
raise offsite.Refused()
result.append({'name': name, 'bytes': len(raw), 'sha256': digest,
'etag': version, 'readback_verified': True})
return result
def main():
p = argparse.ArgumentParser(description=__doc__)
p.add_argument('--output-dir', required=True)
p.add_argument('--publish', action='store_true')
p.add_argument('--auth-source', choices=('prompt', 'attended-openbao'), default='prompt')
p.add_argument('--receipt', required=True)
p.add_argument('--offline-transfer-receipt', help='After prompted publication, retrieve the known recovered generation using the same independently held access')
a = p.parse_args()
os.umask(0o077)
fd = os.open(a.receipt, os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW, 0o600)
r = {'schema': 'netkingdom.recovery-kit.v1', 'status': 'failed',
'secret_values_exported': False, 'new_grants_created': False}
try:
if a.offline_transfer_receipt and (not a.publish or a.auth_source != 'prompt'):
raise offsite.Refused()
files = build()
directory = Path(a.output_dir)
directory.mkdir(mode=0o700, exist_ok=False)
for name, raw in files.items():
(directory / name).write_bytes(raw)
r.update(status='built', files=[{'name': n, 'bytes': len(v), 'sha256': hashlib.sha256(v).hexdigest()}
for n, v in files.items()])
if a.publish:
if a.auth_source == 'prompt':
with open('/dev/tty', 'r+') as terminal:
if not terminal.isatty():
raise offsite.Refused()
secret = getpass.getpass('Independently held Storage Share Backup app password: ', stream=terminal)
else:
spec = importlib.util.spec_from_file_location('inventory', HERE / 'inventory-storage-share.py')
owner = importlib.util.module_from_spec(spec)
spec.loader.exec_module(owner)
_, credentials = owner.caller_credentials()
secret = credentials['BACKUP_PASSWORD']
if not secret or any(c in secret for c in '\r\n\x00'):
raise offsite.Refused()
try:
headers = {'Authorization': 'Basic ' + base64.b64encode(('Backup:' + secret).encode()).decode(),
'Accept-Encoding': 'identity'}
r.update(status='published_readback_verified', files=publish(files, headers),
folder='Backup/railiance-backups/net-kingdom-identity',
provider='Storage Share', credential_source=a.auth_source)
if a.offline_transfer_receipt:
transfer_fd = os.open(a.offline_transfer_receipt, os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW, 0o600)
transfer = {'status': 'failed', 'error': 'offline_retrieval_failed_private_diagnostics_suppressed'}
try:
record = json.loads((ROOT / FILES[2]).read_text())
transfer = offline.retrieve(record, secret, 'account')
finally:
with os.fdopen(transfer_fd, 'w') as out:
json.dump(transfer, out, indent=2)
out.write('\n')
r['offline_retrieval_verified'] = transfer['status'] == 'readback_verified'
finally:
secret = None
r['finished_at'] = datetime.now(timezone.utc).isoformat()
except (Exception, KeyboardInterrupt):
r.update(status='failed', error='kit_operation_failed_private_diagnostics_suppressed')
finally:
with os.fdopen(fd, 'w') as output:
json.dump(r, output, indent=2)
output.write('\n')
# Silent: safe to use in the existing captured attended-owner conduit.
return int(r['status'] == 'failed')
if __name__ == '__main__':
sys.exit(main())

138
tools/bootstrap-recovery/retrieve-identity-offline.py Normal file
View file

@ -0,0 +1,138 @@
#!/usr/bin/env python3
"""Download one reviewed ciphertext using privately entered provider access.
No OpenBao, SSO, Kubernetes, credential discovery, provider writes or age key.
Run by the operator in their own terminal, never through a recorded agent PTY.
"""
import argparse
import base64
from datetime import datetime, timezone
import getpass
import hashlib
import importlib.util
import json
import os
from pathlib import Path
import re
import shutil
import signal
import sys
import tempfile
import urllib.request
HERE = Path(__file__).resolve().parent
spec = importlib.util.spec_from_file_location('offsite', HERE / 'identity-offsite.py')
offsite = importlib.util.module_from_spec(spec)
spec.loader.exec_module(offsite)
ACCOUNT_BASE = offsite.HOST + '/remote.php/dav/files/Backup/railiance-backups/net-kingdom-identity/'
def validate(record):
if (record.get('schema') != 'netkingdom.identity-offsite.v1'
or record.get('status') != 'readback_verified'
or record.get('provider') != 'Storage Share'
or record.get('folder') != 'Backup/railiance-backups/net-kingdom-identity'
or not offsite.NAME.fullmatch(record.get('artifact', ''))
or not re.fullmatch('[a-f0-9]{64}', record.get('ciphertext_sha256', ''))
or type(record.get('ciphertext_bytes')) is not int
or not 0 < record['ciphertext_bytes'] <= offsite.LIMIT
or not re.fullmatch(r'"[^"\x00-\x20]{1,180}"', record.get('etag', ''))):
raise offsite.Refused()
def retrieve(record, secret, mode, client=None):
validate(record)
if mode not in {'account', 'folder-share'} or not secret or any(c in secret for c in '\r\n\x00'):
raise offsite.Refused()
auth = offsite.headers(secret) if mode == 'folder-share' else {
'Authorization': 'Basic ' + base64.b64encode(('Backup:' + secret).encode()).decode(),
'Accept-Encoding': 'identity'}
url = (offsite.BASE if mode == 'folder-share' else ACCOUNT_BASE) + record['artifact']
client = client or offsite.opener()
directory = None
complete = False
try:
with client.open(urllib.request.Request(url, method='HEAD', headers=auth), timeout=30) as response:
if offsite.etag(response, url, record['ciphertext_bytes']) != record['etag']:
raise offsite.Refused()
directory = Path(tempfile.mkdtemp(prefix='nk49-identity-fetched-', dir='/tmp'))
target = directory / record['artifact']
auth['If-Match'] = record['etag']
count = 0
digest = hashlib.sha256()
with client.open(urllib.request.Request(url, headers=auth), timeout=180) as response:
if offsite.etag(response, url, record['ciphertext_bytes']) != record['etag']:
raise offsite.Refused()
fd = os.open(target, os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW, 0o600)
with os.fdopen(fd, 'wb') as output:
while block := response.read(1024 * 1024):
count += len(block)
if count > record['ciphertext_bytes']:
raise offsite.Refused()
digest.update(block)
output.write(block)
output.flush()
os.fsync(output.fileno())
if count != record['ciphertext_bytes'] or digest.hexdigest() != record['ciphertext_sha256']:
raise offsite.Refused()
with client.open(urllib.request.Request(url, method='HEAD', headers=auth), timeout=30) as response:
if offsite.etag(response, url, record['ciphertext_bytes']) != record['etag']:
raise offsite.Refused()
complete = True
# Closed allowlist: never copy arbitrary source-receipt fields to output.
result = {k: record[k] for k in ('schema', 'provider', 'folder', 'artifact',
'ciphertext_bytes', 'ciphertext_sha256', 'etag')}
result.update(status='readback_verified', fetched_path=str(target),
independent_owner_retrieval_verified=True,
credential_source='independently_held_provider_access',
provider_auth_mode=mode, openbao_contacted=False, sso_contacted=False,
provider_writes_performed=False, age_private_key_used=False,
secret_values_exported=False, object_unchanged_during_transfer=True,
downloaded_at=datetime.now(timezone.utc).isoformat())
return result
finally:
if directory is not None and not complete:
shutil.rmtree(directory)
def interrupted(signum, frame):
raise InterruptedError()
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('--source-receipt', required=True)
parser.add_argument('--receipt', required=True)
parser.add_argument('--auth', choices=('account', 'folder-share'), default='account')
args = parser.parse_args()
os.umask(0o077)
fd = os.open(args.receipt, os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW, 0o600)
result = {'status': 'failed', 'error': 'offline_retrieval_failed_private_diagnostics_suppressed'}
try:
record = json.loads(Path(args.source_receipt).read_text())
validate(record)
# getpass must never fall back to echoed stdin (e.g. a captured agent tool).
with open('/dev/tty', 'r+') as terminal:
if not terminal.isatty():
raise offsite.Refused()
secret = getpass.getpass('Storage Share Backup app password or exact folder-share token: ', stream=terminal)
try:
result = retrieve(record, secret, args.auth)
finally:
secret = None
except (Exception, KeyboardInterrupt):
pass
finally:
with os.fdopen(fd, 'w') as output:
json.dump(result, output, indent=2)
output.write('\n')
ok = result['status'] == 'readback_verified'
print('OK: ciphertext retrieved without OpenBao or SSO; ETag, size and SHA256 verified.' if ok
else 'ERROR: retrieval refused; private diagnostics suppressed. Use a new receipt path to retry.')
return 0 if ok else 1
if __name__ == '__main__':
for sig in (signal.SIGINT, signal.SIGTERM, signal.SIGHUP):
signal.signal(sig, interrupted)
sys.exit(main())

33
tools/bootstrap-recovery/run-offline-recovery.sh Normal file
View file

@ -0,0 +1,33 @@
#!/usr/bin/env bash
# Run in the owner's own terminal. Provider password and age key are hidden inputs.
set +x
set -euo pipefail
ROOT="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." && pwd)"
cd -- "$ROOT"
umask 077
[[ -t 0 && -t 1 ]] || { printf '%s\n' 'ERROR: run in your own interactive terminal.'; exit 1; }
for tool in python3 age age-keygen openssl docker; do
command -v "$tool" >/dev/null || { printf 'ERROR: required tool unavailable: %s\n' "$tool"; exit 1; }
done
docker image inspect 'postgres@sha256:ff23cdce56cac62ada6f66013e1a50864c0abbe79d132d40b6e05bd80f378a70' >/dev/null 2>&1 || {
printf '%s\n' 'ERROR: stage the pinned PostgreSQL image before this drill.'; exit 1;
}
NK49_HANDOFF_DIR="$(mktemp -d /tmp/nk49-offline-handoff-XXXXXXXX)"
printf 'Recovery evidence directory: %s\n' "$NK49_HANDOFF_DIR"
printf '%s\n' 'This publishes three new versioned guide/kit/checksum files beside the backups, then downloads the protected recovered generation.'
printf '%s\n' 'Enter the independently held Backup account app password at the next hidden prompt. No OpenBao or SSO login is used.'
python3 tools/bootstrap-recovery/recovery-kit.py \
--output-dir "$NK49_HANDOFF_DIR/kit" --publish \
--receipt "$NK49_HANDOFF_DIR/publication.json" \
--offline-transfer-receipt "$NK49_HANDOFF_DIR/transfer.json" || {
printf '%s\n' 'ERROR: publication/retrieval stopped. Inspect the safe receipts; do not paste credentials into chat.'; exit 1;
}
printf '%s\n' 'Guide publication and outage-path retrieval passed. Next enter the independently held railiance-backup age key.'
bash sso-mfa/bootstrap/sops-custody-unlock.sh \
--expected-recipient age1zvryunvjhvpkmasskauga2heeg0ztnte9ymgppvjge36ekumk50syr3tsz \
--prompt -- \
python3 tools/bootstrap-recovery/verify-identity.py \
--transfer-receipt "$NK49_HANDOFF_DIR/transfer.json" \
--receipt "$NK49_HANDOFF_DIR/recovery.json"
printf 'OK: private-provider retrieval and archive recovery passed. Safe receipts: %s\n' "$NK49_HANDOFF_DIR"
printf '%s\n' 'Service authentication remains separate acceptance. Return only the evidence directory path and outcome, never credentials.'

140
workplans/NK-WP-0049-bootstrap-secret-free-recovery-handoff.md
View file

@ -28,10 +28,11 @@ These are retrospective task records of evidenced work, not additional delivery
or newly granted authority. The earlier dated narrative remains in Git at
`cc2dd01`; its evidence references are carried into the tasks below.
**Current position:** 12 tasks done; 1 in progress; 3 ready to start (`todo`);
8 waiting on explicit prerequisites. These are task counts, not an effort-based
**Current position:** 14 tasks done; 2 in progress; 1 ready to start (`todo`);
10 waiting on explicit prerequisites. These are task counts, not an effort-based
completion percentage. Assignees are also written in task descriptions because
the current Forge projector does not populate the dedicated Hub assignee field. No remaining task is being executed by this replan.
the current Forge projector does not populate the dedicated Hub assignee field. The outage-retrieval tool and restore-packet draft are now prepared;
owner execution/publication and service admission remain explicit gates.
**Next actions:** finish the read-only caller ledger (T02); prepare the isolated
restore packet (T15), independent offline retrieval (T23), and residual-scope
@ -369,8 +370,8 @@ state_hub_task_id: "479d6970-6516-50f5-9436-bb5b56d58aad"
the named owner boundary.
The source inventory and known legacy objects exist; a complete current-use
ledger is not yet accepted. Next action is read-only reconciliation of the 87
source entries against active Make/bootstrap/operator routes and owner evidence.
ledger is not yet accepted. Next action is read-only reconciliation of the 93
entries in the latest October15 source inventory (the earlier snapshot had 87) against active Make/bootstrap/operator routes and owner evidence.
**Done when:** each entry is identified as current, retired or owner-deferred,
with its owner, credential route, current consumer(s), native evidence or exact
@ -385,21 +386,26 @@ T22. Do not reopen completed source repairs without a new finding.
```task
id: NK-WP-0049-T15
status: todo
status: progress
priority: high
assignee: codex
depends_on: [NK-WP-0049-T08, NK-WP-0049-T13]
depends_on: [NK-WP-0049-T08, NK-WP-0049-T13, NK-WP-0049-T27]
state_hub_task_id: "1d1eb903-a4b0-5bc4-a421-0b842d7f68ae"
```
**Accountable worker:** codex. Service/credential-owner acceptance remains with
the named owner boundary.
**Next action:** turn the recovered manifest and current caller gaps into a
reviewable procedure with selected immutable images, restore order, isolated
endpoints/network, data/key ownership, cleanup and per-service positive/negative
checks. Preparation can start with the existing archive; incorporate T02's ledger
before submitting current-caller acceptance.
**Prepared draft scope moved to T27.** See
`docs/identity-service-restore-packet.md`: restore order, containment, safe evidence
and per-service positive/negative checks are drafted. Read-only current runtime
image digests were observed for all four services; correspondence to the selected
archive and independent image availability are not yet proved.
**Next action:** admit a named disposable target/private route, verify and stage
matching images independently, obtain private existing-user/MFA test inputs and
review version-specific native commands with service owners. Incorporate T02's
ledger before submitting current-caller acceptance.
**Done when:** service owners accept the exact targets/procedure, existing-user
and MFA test identities/input custody are available without posting secrets,
@ -585,22 +591,29 @@ source retirement as provider revocation.
```task
id: NK-WP-0049-T23
status: todo
status: wait
priority: high
assignee: codex
depends_on: [NK-WP-0049-T13]
depends_on: [NK-WP-0049-T13, NK-WP-0049-T25]
blocking_reason: "Await the operator-run private independent-provider download and hidden-key recovery drill; bounded tooling and self-contained kit are prepared in T25."
state_hub_task_id: "1ec537ee-dfee-5576-be36-a81bdba417d1"
```
**Accountable worker:** codex. Service/credential-owner acceptance remains with
the named owner boundary.
The existing independent retrieval succeeded through a healthy attended OpenBao
path. A full custody outage still lacks a reviewed offline provider adapter.
The previous independent retrieval succeeded through a healthy attended OpenBao
path. Source/fixture preparation of a separate offline adapter is now complete
in T25. Actual independently held provider access has not yet been exercised
for this outage path; no completed drill is claimed.
**Next action:** prepare bounded read-only download/receipt tooling using
independently held provider access in private operator input, with fixed provider,
artifact/ETag/size/digest guards, sanitized output and no standing agent secret.
**Unblock action:** operator runs the prepared private terminal handoff
`bash tools/bootstrap-recovery/run-offline-recovery.sh` (combined guide
publication and known-generation retrieval) or the standalone read-only
`retrieve-identity-offline.py` command in the guide. Independently held provider
access and age key go only into the respective hidden prompts; inspect safe
receipts and perform the real archive recovery check. Never use a captured agent
PTY for secret entry. T26 separately tracks actual guide publication.
**Done when:** the owner can retrieve the exact off-host bundle without relying
on the failed OpenBao/SSO instance, run the existing hidden-key checker, and
@ -633,6 +646,91 @@ an external watchdog are not implemented; do not add their grants/jobs implicitl
This task triages the new insights instead of silently expanding NK49's required
implementation to an entire disaster-recovery programme.
## Deliver and verify the offline retrieval adapter and recovery tools kit
```task
id: NK-WP-0049-T25
status: done
priority: high
assignee: codex
depends_on: [NK-WP-0049-T13]
```
**Accountable worker:** codex.
Completed 2026-10-16. `retrieve-identity-offline.py` uses privately entered
independent Backup-account access or an exact read-capable identity-folder share.
It performs only fixed-provider HEAD/GET, binds the reviewed generation by strong
ETag/length/SHA256, refuses redirects/proxies/unreviewed scope and removes failed
downloads. No OpenBao/SSO, age key or standing agent credential is required.
`recovery-kit.py` builds a reproducible allowlisted ZIP containing the tools,
public profile, known-good ciphertext receipt and recovery guides, with file
checksums. Its publication path creates only three versioned guide/ZIP/checksum
objects and verifies readback without overwriting existing conflicting objects.
`run-offline-recovery.sh` provides the owner's two-hidden-prompt handoff.
Evidence: `tests/test_identity_offline_recovery.py` and the bounded test/build
receipt in `docs/evidence/2026-10-16-identity-outage-recovery/`. The focused run
passed 65 tests including real disposable age/PostgreSQL fixtures. ZIP-contained
entry points import without sibling repositories. This is source/fixture proof,
not actual independent-provider retrieval (T23) or publication (T26).
## Publish and verify the disaster recovery guide on Storage Share
```task
id: NK-WP-0049-T26
status: wait
priority: high
assignee: codex
depends_on: [NK-WP-0049-T25, NK-WP-0049-T27]
blocking_reason: "Await private operator provider-access execution of the prepared publication command and actual three-object readback receipts; guide and kit are built."
```
**Accountable worker:** codex; operator holds provider access privately.
Requested by the operator on 2026-10-16. Guide source:
`docs/disaster-recovery-storage-share.md`. Document scope, independently held
provider/key requirements, exact known-good fallback, copy/download verification,
archive-check commands, service restore order, recurring schedule and remaining
coverage limits. Include the offline tools so failed Forgejo does not prevent
access to the recovery scripts; retain an independent copy and trusted digest.
**Unblock action:** operator runs `run-offline-recovery.sh`, or the reviewed
`recovery-kit.py --publish` command through the existing attended Backup-account
owner conduit. Existing publisher/retention workload grants are not repurposed.
The user authorized this documentation upload; no new grant is required by the
prepared command. Credentials remain private; no provider permissions are widened.
**Done when:** the versioned START-HERE guide, self-contained ZIP and ZIP SHA256
file are present beside the identity backups with exact conditional-create and
ETag/length/content readback proof, and their folder/name/digests are recorded for
future discovery. Local build or attempted upload alone does not close this task.
## Draft the isolated service restore packet and observe candidate images
```task
id: NK-WP-0049-T27
status: done
priority: high
assignee: codex
depends_on: [NK-WP-0049-T08, NK-WP-0049-T13]
```
**Accountable worker:** codex.
Completed 2026-10-16. Draft in `docs/identity-service-restore-packet.md` names the
known recovered input, restore order, original key/data ownership, default-deny
containment, blocked production/provider/notification effects, cleanup and five
native acceptance lanes with positive/negative checks. Read-only metadata from
the pinned primary cluster resolved all four current service image digests,
including the separately observed privacyIDEA main container and LLDAP's mutable
source tag. Evidence: `docs/evidence/2026-10-16-identity-outage-recovery/image-observation.json`.
This task extracts completed preparation from T15. Named target admission,
archive/runtime image-lineage matching and independent image availability,
private existing-user/MFA inputs and owner-reviewed native commands remain T15;
no service restore, production change or owner acceptance is claimed.
## Accept remaining recovery and caller evidence and close the G08 handoff
```task
@ -640,7 +738,7 @@ id: NK-WP-0049-T03
status: wait
priority: high
assignee: codex
depends_on: [NK-WP-0049-T02, NK-WP-0049-T16, NK-WP-0049-T17, NK-WP-0049-T18, NK-WP-0049-T19, NK-WP-0049-T20, NK-WP-0049-T21, NK-WP-0049-T22, NK-WP-0049-T23, NK-WP-0049-T24]
depends_on: [NK-WP-0049-T02, NK-WP-0049-T16, NK-WP-0049-T17, NK-WP-0049-T18, NK-WP-0049-T19, NK-WP-0049-T20, NK-WP-0049-T21, NK-WP-0049-T22, NK-WP-0049-T23, NK-WP-0049-T24, NK-WP-0049-T26]
blocking_reason: "Await the separately tracked native service proofs, current/legacy caller acceptance and independent outage retrieval; full archive recovery and backup activation are already complete."
state_hub_task_id: "d68cd4d3-1ba3-530e-88f9-26ebb8dd6eac"
```
@ -652,7 +750,7 @@ Owner boundary: NetKingdom, Platform and affected service owners, with Custodian
G08 receiving the final evidence. The completed archive/job milestones remain
accepted independently of this final gate.
**Done when:** T02 and T15–T24 have their stated completed or explicitly accepted
**Done when:** T02 and T15–T27 have their stated completed or explicitly accepted
handoff outcomes; service authentication/token/audit proof and current/legacy
caller dispositions are accepted; the source workplan and projected tasks agree;
and G08 has an exact evidence/handoff record. Record actionable residuals as live