net-kingdom/docs/bootstrap-recovery-ceremony.md

# NK-WP-0049 recovery ceremony and closure packet

Status: the actual full off-host identity bundle was independently retrieved and
recovered with the human-held age key. Daily backup/retention and hourly health
timers are active. Full native service authentication, token decryption, audit
continuity and current/legacy caller acceptance remain open for NK-WP-0049.
Approved execution is recorded in decisions40bfae40 and5edd5203; this document
records their scope and evidence.

**Outage quick start:** [Storage Share disaster recovery guide](disaster-recovery-storage-share.md).
The guide and standalone tools ZIP were published and read back on 2026-10-16.
The operator independently retrieved the protected bundle without OpenBao/SSO
and passed the full archive/native PostgreSQL check. T23/T26 are done; see
[the accepted publication and drill record](storage-share-recovery-publication.md).
The [isolated service restore packet](identity-service-restore-packet.md)
contains the draft native-service sequence and remaining T15 admission inputs.
Its actual LLDAP stage passed on 2026-10-16: native existing-user bind, wrong
credential refusal, stable directory/attribute rows, isolation and cleanup.
T16 is done; Authelia/KeyCape/MFA/audit continuity remains separate acceptance.

## What has been established

On 2026-10-15 the operator identified Scaleway and Storage Share as the off-host
backup providers. This identifies destinations, not the exact recovery artifact,
its freshness, its file coverage or independently held decryption key. Platform's
current backup contracts distinguish Scaleway database/full archives from the
Nextcloud secondary copy. An upload-only share is not a retrieval identity.

Read-only primary-cluster metadata is retained under
`docs/evidence/2026-10-15-bootstrap-recovery/`. LLDAP, Authelia, KeyCape,
privacyIDEA, identity-provisioner and User Engine each have one Ready replica.
The three SQLite/key backup CronJobs report successful October 15 invocations.
Their current commands target local PVCs; neither an external export nor complete
off-host key/database coverage is established by those observations. The
privacyIDEA command mentions its encryption key, but not its audit key pair.
This command inspection is a coverage question, not proof that no other backup
exists. The personal-token and trigger-admin Secrets still exist. Neither their
validity nor provider-side revocation has been tested. A scan of 34 current
Deployment/StatefulSet/DaemonSet/CronJob/Pod resources in sso, mfa and user-engine
found no references to those two old Secrets; it still found the expected audit
delivery reference in User Engine. Other namespaces, historic jobs, external
consumers and issued provider credentials are outside that scan.

## Assemble the exact owner packet before execution

Record only safe references and checksums. Keep credentials in their current
owner custody and pass the decryption key through attended temporary unlock.

| Required owner input | Acceptance |
| --- | --- |
| Off-host artifact | Provider, account/folder or object/version, backup timestamp, ciphertext SHA256 and independent retrieval receipt; do not substitute a new local export for the existing off-host backup. |
| Independent key | Human confirms matching custody outside the servers and OpenBao. A successful decrypt using only the OpenBao escrow key does not prove independent recovery. Never send the key in chat. |
| Required assets | Exact manifest of matching service env/config files, LLDAP and Authelia data, privacyIDEA database plus original encryption/audit keys, and KeyCape registry/signing material. Resolve assets held through other current owner lanes explicitly. |
| Recovery environment | Named disposable target, pinned images available independently of the failed production registry, no production ingress, no production identity/client execution, bounded egress, private storage and exact cleanup scope. |
| Native proof | Owner-specific database integrity, identity/group counts, decryption of retained token material, audit signature continuity, issuer/client/signing configuration preservation and admitted login/MFA checks. Distinguish an isolated restore from current production credential validity. |
| Current consumer source | Reconcile the deployed revisions with the repaired caller/provider source. The October 14 reserved platform-group fix must be deployed and independently checked before claiming live enforcement. |
| Legacy credential disposition | IAM owner identifies the exact historical personal-token/trigger principals, actual consumers and revocation method before any deletion. Deleting a Secret does not revoke an issued provider credential. |

The operator supplied the independently held `railiance-backup` age key only to
the attended local custody helper. The actual off-host PI ciphertext decrypted
and restored successfully; the agent received no private key material. The full
identity asset manifest remains unfulfilled. The backup lane recipient and
bootstrap `keys/age.pub` must not be assumed to be the same.
Do not draft a blanket restore or grant approval while full coverage is unknown.
Once fixed, bind the source revisions, target identities, allowed operations,
recovery/cleanup sequence and maintenance window to the reviewed live packet.
Existing eligibility or another client's approval is not this ceremony's grant.

## Execute in stages

1. Use the existing Platform/provider recovery lane to retrieve the identified
   ciphertext, verify its object/version and digest, and retain a safe transfer
   receipt. Do not request or expose the provider credential to the agent.
2. The human unlocks the independently held age key in the existing custody
   helper. Validate it against the registered public recipient. No permanent
   key is installed in HOME, generated as a replacement, or copied into a bundle.
3. Run the local transport/coverage check below. It contacts no cluster, keeps
   plaintext in a private temporary directory outside Git, refuses unsafe
   archives/private age keys, tests an unrelated disposable wrong key, reads the
   restored files back and removes that staging. It emits a closed non-secret
   receipt. This is one prerequisite, not whole-service recovery acceptance.
4. Under the exact separately reviewed owner ceremony, restore the matching
   service data/key/config sets into the disposable environment and perform the
   native checks above. Never run the full production bootstrap to test source
   repairs. Unknown writes stop for reconciliation rather than automatic retry.
5. Verify cleanup, preserve the original ciphertext and independent custody,
   and reconcile current callers/legacy credentials through their owners.
   Retain exact source/object/image/actor/time and positive/negative receipts.
6. Close T02/T03 only after the recorded remaining requirements are satisfied.
   Update the workplan first, run `statehub fix-consistency --repo net-kingdom`,
   and verify the Hub projection. A transport fixture or generated evidence
   worksheet cannot mark this plan finished.

## Prepared local transport check

The human invokes this through the existing attended custody helper after the
exact artifact and required-file list are agreed. `NETKINGDOM_CUSTODY_AGE_KEY_FILE`
is the temporary private file provided inside that envelope. The tool never
reads an ambient HOME key. The receipt destination must be new.

```bash
python3 tools/bootstrap-recovery/verify-bundle.py \
  --bundle /private/recovery/fetched-ops-bundle.tar.age \
  --bundle-sha256 <independently-recorded-ciphertext-sha256> \
  --required-file lldap/secrets.env \
  --required-file privacyidea/secrets.env \
  --required-file privacyidea/pi.enc \
  --required-file privacyidea/private.pem \
  --required-file privacyidea/public.pem \
  --receipt /private/recovery/new-transport-receipt.json
```

This example file list is not an accepted whole-stack manifest. Add every
agreed service file supported by the closed transport profile; database copies
and other archive formats use their native owner restore tools. A successful
receipt explicitly leaves independent custody, current service validity and
native consumer recovery unproven. Off-host retrieval lineage and human custody
confirmation must accompany it; the tool cannot infer where a local input came
from. A failed check publishes no acceptance receipt.

## Rotation boundaries

CUST-WP-0073-T05 remains founder-deferred under decision
`1591daa1-6e85-4dfd-aadd-5a9fe1ef47fa`. This work does not reopen that rotation,
replace signing keys, change directory passwords or erase old backup keys.
Existing historical exposure, provider revocation and coordinated rotation
requirements stay explicit in their live owner records. If completion needs a
protected native operation, prepare that exact operation and obtain its scoped
disposition; never treat a source repair or successful backup job as permission.

## Knowledge/source scope

Emergent Security 0.2.6, package digest
`8d1d6a9d4d07fec7b619618de7dfe2be2254c1790f85f4198c7fe19db9751c02`:
start/root cards withheld, orientation only. Current SECURITY-COMPANION,
Platform backup/custody procedures, owner runtime contracts and the workplan
were reviewed as sources. No execution authority is derived from the cards.


## Off-host inventory return — 2026-10-15

Both bounded metadata inventories completed. Storage Share's inspected
`Backup/railiance-backups` scope includes the current privacyIDEA database
ciphertext and the October14 OpenBao snapshot. It did not reveal a matching
bootstrap/identity-file bundle in the inspected folders. Scaleway's inspected
`railiance-platform-pg-backup/platform-pg/` tree contains database backup trees
and an application archive subtree, with no identity-file bundle in this bounded
scope. Other accounts/buckets/prefixes are unassessed. See
[the discovery summary](evidence/2026-10-15-bootstrap-recovery/offsite-discovery-summary.json).
No object contents or age keys were read; the operator envelope exited0 and
self-revoked. Existing scoped backup credentials were reused without changes;
the Scaleway bootstrap API key was not used.

The operator subsequently reported the hidden-terminal check passed and the
temporary key file was removed. The later actual database drill also passed;
the private key was not provided to the agent.

## Prepared bounded privacyIDEA database drill — 2026-10-16

The identified existing off-host ciphertext was downloaded through the same
attended Platform Backup-account procedure. The before/GET/after strong-ETag
binding and exact byte count passed. Ciphertext SHA256:
`cc81da442755928171d2c51bce93eda7de4392a0a1aa9d0ffb83a25ba34e5ead`.
See `privacyidea-transfer-r1.json` in the October15 evidence folder. The owner
envelope exited0 and self-revoked; no age key or plaintext was read by the fetch.
Ciphertext is retained in the fresh private `/tmp/nk49-offhost-j9dzl0ge/` directory.

The following exact human command performs only an isolated local database
drill. It checks the transfer receipt/digest before opening the temporary key,
checks the backup recipient, refuses an unrelated wrong key, decrypts to private
temporary storage and restores into disposable PostgreSQL. Image
`postgres@sha256:ff23cdce56cac62ada6f66013e1a50864c0abbe79d132d40b6e05bd80f378a70`
is already cached locally (pg_restore16.14). There is no image pull, network,
published port, host data mount, production connection or IAM/client execution.
Database storage is a bounded temporary memory filesystem. Only fixed aggregate
counts for `token`, `resolver`, and `realm` leave the database. The container,
volumes and temporary plaintext are removed and cleanup is checked. The original
ciphertext remains for reproducibility. This proves database recovery only;
token decryption, audit signature continuity and full identity service recovery
require matching key/config files and further native owner checks.

Executed successfully on 2026-10-16 (local time). The command is retained for
provenance; its r1 receipt now exists and must not be overwritten or rerun merely
to repeat evidence. Run future reviewed drills with a new receipt. Paste the key
only into the hidden prompt, then share the success/error line or safe JSON
receipt. The receipt destination must be new; do not overwrite a failed receipt.

```bash
cd /home/worsch/net-kingdom
bash sso-mfa/bootstrap/sops-custody-unlock.sh \
  --expected-recipient age1zvryunvjhvpkmasskauga2heeg0ztnte9ymgppvjge36ekumk50syr3tsz \
  --prompt -- python3 tools/bootstrap-recovery/check-privacyidea-database.py \
  --transfer-receipt docs/evidence/2026-10-15-bootstrap-recovery/privacyidea-transfer-r1.json \
  --receipt docs/evidence/2026-10-15-bootstrap-recovery/privacyidea-database-drill-r1.json
```

Source verification:23 focused checks passed, including a real local Docker
custom-format dump, age encryption, temporary fixture-key unlock, wrong-key
refusal, native restore/readback and cleanup. Those tests use synthetic data and
fresh fixture keys. They are not evidence of actual backup decryption. If the
real drill fails, review its closed receipt; do not expose PostgreSQL/age stderr
or dump contents in chat. T02/T03 remain open.

## Actual database recovery result — 2026-10-16

The operator reported success and the actual receipt was reviewed against the
exact checker source, pinned image and fetched ciphertext SHA256. The retained
ciphertext digest was independently rechecked. Decryption, unrelated wrong-key
refusal, native PostgreSQL restore, fixed aggregate readback and container/plaintext
cleanup all passed. The restored database has five tokens, two resolvers and
two realms. The local Docker inventory contains no remaining NK49 containers.
Evidence: `privacyidea-database-drill-r1.json` and `database-drill-review.json`
in the October15 evidence directory. No production connection occurred.

This proves recovery of the selected database using the operator's independently
held key. It does not prove the recoverability of token plaintext, old audit
signatures, directory accounts, issuer keys or a full identity environment.

## Missing coverage prepared at the operator's request

The operator confirmed there is no known full identity bundle and requested
preparation of the missing coverage. The machine-readable proposal is
[`identity-backup-profile.json`](../tools/bootstrap-recovery/identity-backup-profile.json).
The read-only preparer checked the primary cluster UID, four current Deployment
UIDs/readiness/observed generations, five Bound PVC UIDs and the complete current
Secret/ConfigMap reference sets. It captured names and identities only; it did
not read Secret values or execute in pods. Its exact result is
`identity-backup-preparation-r1.json`. Fourteen tests cover foreign/stale targets,
new unreviewed dependencies, projected Secrets and private-value suppression.

| Component | Required new coverage | Required recovery proof |
| --- | --- | --- |
| LLDAP | Consistent `users.db` snapshot, remaining durable data and current credentials/configuration | User, group and membership preservation; existing-user authentication |
| Authelia | Consistent storage DB, original encryption/OIDC/session keys and current configuration | Storage decryption, signing identity and isolated authentication |
| privacyIDEA | Original enckey plus matching audit pair, configuration/scripts and a linked native database dump | Existing token decryption/MFA, old audit signature verification and new isolated audit entry |
| KeyCape | Current registry/config and signing key, policy/receipts, every referenced credential's recovery or independent reissue disposition | Issuer/JWKS identity, registry/policy continuity and positive/negative consumer acceptance |

The proposed primary prefix is
`platform-pg/application-archives/net-kingdom/`; the proposed secondary folder is
`Backup/railiance-backups/net-kingdom-identity/`. These are **not admitted**.
Platform's current generic application archive adapter only admits
`coulomb-social` and `email-connect`, and their publisher custody profiles are
application-bound. A new NetKingdom profile and scoped publisher require owner
admission; existing credentials must not be borrowed. The operator's Scaleway
bootstrap API key is unnecessary for preparation.

The initial goal is one verified generation, with a measured encrypted size and
fresh provider quota/upload-overlap checks. A proposed 512 MiB ceiling is a
refusal bound, not evidence that the data fits the secondary budget. Seven daily
and two weekly points remain a later measured proposal. No retention deletion
or recurring schedule was installed. Capture must preserve original keys and
use SQLite online-backup/PostgreSQL logical snapshots, record each capture
interval and refuse changed source identities/configuration. It must not claim
global atomicity or substitute a fresh export for the already recovered old dump.

The source implementation below supersedes the earlier adapter-preparation gap.
The Scaleway primary/Storage Share secondary proposal above is deferred: the
initial rollout implements Storage Share only. Neither destination has yet
received a full identity bundle. Owner admission and actual recovery remain open.


## Implemented identity backup packet — 2026-10-16

The new capture, sealing, upload/readback, independent retrieval, archive/native
PostgreSQL verification, separate retention and node installer are implemented
under `tools/bootstrap-recovery/identity-*.py`, `capture-identity.py`,
`retrieve-identity.py`, `verify-identity*.py` and `install-identity-backup.py`.
Source profile: `identity-backup-profile.json`. No production keys were collected,
no new credential grants were made and no timers were installed in this source
checkpoint. Decision `40bfae40-6108-4e51-8613-073339f5345f` remains pending.

The exact new owner requests are:

- [CCR-2026-0048 publisher](../../railiance-platform/credential-change-requests/CCR-2026-0048-net-kingdom-backup-publisher.yaml): a read/create share for only `Backup/railiance-backups/net-kingdom-identity/`, delivered through a dedicated 15-minute, audience/namespace/SA-bound ESO reader.
- [CCR-2026-0049 retention](../../railiance-platform/credential-change-requests/CCR-2026-0049-net-kingdom-backup-retention.yaml): a separate read/delete share for that same folder and separate process. Compromise of this credential can delete points in the folder; it must not be given to the publisher process.

Both require Platform, NetKingdom and Backup-share owner review. Only the scoped
share tokens enter exact new workload KV paths and ESO-managed Secrets. Neither
the operator account password nor the age private key is delivered to the job.
The setup helper refuses proposed/unapproved or hash-mismatched CCRs. Native
positive and wrong-SA/namespace/audience probes must prove the admitted reader
boundary, followed by equality to the exact ESO delivery; temporary readers are
revoked. These are new grants, so prior approval to prepare the missing coverage
does not count as completed credential-owner admission.

The initial provider is Storage Share only. Ciphertext and safe receipt are
created once under a unique generation name, read back using strong ETags and
checked by size/SHA256. A 512 MiB refusal ceiling and two GiB fresh provider
headroom apply; unsupported public-share quota fails closed. Public DAV behavior
and permission denials are checked with disposable public-recipient fixture
ciphertext before accepting the grants. No Scaleway credential or another
workload publisher is reused. Nextcloud documents the public-share DAV interface
in its [official WebDAV guide](https://docs.nextcloud.com/server/25/user_manual/en/files/access_webdav.html).

Two temporary helper Pods mount only reviewed PVCs read-only, have no service
account token, use pinned images, private tmpfs scratch and deny-all network
policies, then are deleted with exact UID preconditions. Kubernetes network
allows combine, so `identity-isolation.py` first prepares two guarded patches to
exclude the capture label from the existing namespace-wide DNS allows. Every
peer/port and ordinary service selector match is preserved. Plans bind the
current cluster/policy UIDs, resource versions and full original specs, and
refuse an existing pod using the capture label. Capture refuses any remaining
matching allow policy, source drift or unknown helper cleanup outcome.

Capture uses native SQLite online backups and a fresh native PostgreSQL dump;
all required current Secrets/configuration and original key pairs are sealed
with the public recipient. The manifest and plaintext hashes are inside the
ciphertext. privacyIDEA service logs, historical LLDAP/Authelia/PI `backups/`
subtrees and Authelia notification.txt are omitted; the audit database is in the
linked PostgreSQL dump, and the full KeyCape policy/receipt tree is preserved.
There is no claim of a globally atomic multi-service snapshot. Drift/partial
cleanup stops future runs and requires exact owner reconciliation.

### Ordered live rollout after exact packet approval

1. Record all required owner reviews in CCR-2026-0048/0049 and bind their exact file hashes. Run `identity-credential-setup.py publisher|retention --ccr-sha256 HASH --receipt PRIVATE_NEW_PATH` through the attended Platform-admin owner conduit. Never inspect private helper output or copy operator credentials into the runtime. Verify actual provider permissions, initial CAS-only KV publication, exact policy/role, agent high-risk denials and revoked fixture readers.
2. Apply only `identity-custody.json`; wait for its two SecretStores/ExternalSecrets to be Ready. Run `verify-identity-delivery.py publisher|retention --ccr-sha256 HASH --kubeconfig REVIEWED_PATH --receipt PRIVATE_NEW_PATH` through that same owner conduit. Resolve any native denial before accepting delivery; do not set confirmed-claim flags from fixtures.
3. Generate a fresh `identity-isolation.py plan --kubeconfig REVIEWED_PATH --receipt PRIVATE_PLAN`; review it, server-dry-run it, then `apply --plan PRIVATE_PLAN --plan-sha256 HASH` using the same explicit kubeconfig and a fresh receipt. Each patch tests UID/resourceVersion/spec. Regenerate stale plans rather than weakening the guards. Confirm the cluster's network policy enforcement through its owning layer.
4. Render `install-identity-backup.py render --decision-id 40bfae40-6108-4e51-8613-073339f5345f --receipt PRIVATE_PACKET`. Transfer only the reviewed runtime files in that packet plus the packet to the admitted node. As root on railiance01, run installer `install --packet PRIVATE_PACKET --packet-sha256 HASH --receipt PRIVATE_NEW_PATH`. Installation verifies exact cluster/source hashes and unit syntax; timers remain disabled. Execute the packet's exact `first_capture_command`. All output receipts/state and encrypted copies stay under root-private `/var/lib/net-kingdom-backup`.
5. Retrieve the successful generation's safe off-site receipt from that private state, then run attended `retrieve-identity.py --source-receipt PRIVATE_NODE_RECEIPT --receipt PRIVATE_INDEPENDENT_TRANSFER` with the independently authorized Backup account via Platform custody. It re-downloads and binds the ciphertext by ETag/size/hash. The receipt names the new private local ciphertext path. A published index alone is not independent retrieval proof.
6. The human independently verifies the downloaded bundle using the hidden-key command below. Review the actual receipt and exact source/artifact/hash. Copy only this safe receipt to private node state; activate with `install-identity-backup.py activate --accepted-receipt PRIVATE_ACTUAL_RECOVERY --receipt PRIVATE_NEW_PATH`. Activation binds the actual artifact to its local successful off-site receipt, pins that generation, checks health and enables the timers.
7. Confirm both active timers, last success, retained protection and safe Hub progress. Full NK49 native identity/caller and legacy-credential acceptance remains separately open.

All actions accept source/target guards; flags do not create authority. The
installer runs as the already admitted node owner. If installation or a grant
partly fails, stop and reconcile its safe receipt/exact resources before retry.
Do not rotate keys, change existing service credentials or remove older backups
as an implicit repair. For rollback, disable the two exact new timers first,
retain the verified ciphertext and restore receipt, then have the owners revoke
only these newly issued shares/roles and remove their delivery resources after
confirming no consumer remains. Never delete existing unrelated backup jobs.

### Independent full archive verification and bootstrap connection

The key is entered only in the operator's hidden terminal, using the independently
held backup key already checked against this public recipient:

```bash
sso-mfa/bootstrap/sops-custody-unlock.sh \
  --expected-recipient age1zvryunvjhvpkmasskauga2heeg0ztnte9ymgppvjge36ekumk50syr3tsz \
  --prompt -- python3 tools/bootstrap-recovery/verify-identity.py \
  --transfer-receipt /PRIVATE/independent-transfer.json \
  --receipt /PRIVATE/new-full-archive-recovery.json
```

With the reviewed sibling `railiance-bootstrap` checkout, the equivalent handoff
is `bash ../railiance-bootstrap/bin/railiance identity-recovery` with the same
transfer/receipt arguments, run inside that hidden-key wrapper. Its
[bootstrap sequence](../../railiance-bootstrap/docs/identity-recovery.md) links
independent custody and isolated verification to replacement-host/layer handoff.
The wrapper delegates to this verifier and never imports into production.

The complete manifest, SQLite integrity, original RSA pairs, wrong-key refusal,
native isolated PostgreSQL restore and verified cleanup are the archive gate for
recurring backups. A successful archive gate does not prove existing-token
plaintext/MFA, historical/new audit signatures, existing-user login or issuer
consumer flows. Those require separately reviewed isolated service acceptance
before T03 and NK49 can close. The verifier deletes its plaintext; it is not a
production importer.

Today's attended retrieval helper requires healthy OpenBao. For total custody
loss, independently held provider access and a separately reviewed offline
retrieval receipt procedure are still required; that offline adapter is not
implemented. No bare-VM unattended restore is claimed. The backup public key is
not interchangeable with the separate bootstrap encryption recipient.

### Recurrence, retention and health

After the first actual archive gate, the prepared root unit runs daily at
03:45 UTC with up to five minutes jitter. Retention runs in separate processes
before/after capture, protecting newest, seven distinct daily points, two
distinct weekly points and the independently recovered pinned generation.
Unknown/unindexed objects are never candidates. Both pinned and newest points
are re-downloaded and hashed before any expiry; each candidate is checked and
deleted conditionally by ETag. Partial/unknown expiry stops subsequent expiry
until reconciled. A matching guarded local-cache prune preserves protected
points and leaves unknown/failed runs for owner inspection. There is no claim
that 10 GiB is sufficient indefinitely: measured quota/headroom is checked each
run and an excess refuses new capture rather than deleting a protected point.

An hourly node health check reports missing/older-than-25-hour backups or partial
failure to State Hub, with durable local nonsecret outbox/replay. This host-local
check cannot detect or report a dead host; external outage monitoring remains a
separate gap. The daily backup RPO is explicit and distinct from the one-hour
ecosystem coordination frame. Timers are currently disabled/uninstalled.

Find this functionality through the owning registry entry
`capability.security.bootstrap-recovery` (keywords backup, restore,
disaster-recovery, age) federated by reuse-surface, related to
`capability.railiance.bootstrap-sequencing`. Maturity remains D3/A2/C1/R0;
source fixtures and discoverability do not assert live production acceptance.


## Live admission update — 2026-10-16

The operator approved the exact packet in decision
`40bfae40-6108-4e51-8613-073339f5345f`. The first owner login succeeded but
Nextcloud returned publisher permission21 for requested5. Setup refused before
publishing any new KV value. Official Nextcloud link-share normalization adds
permission16 (re-sharing) to read-enabled public links when outgoing federation
is enabled. The operator then explicitly approved folder-only publisher21
(read/create/re-share) and retention25 (read/delete/re-share), recorded in
`5edd5203-fae7-461b-8775-5d22b7873971`. The two CCRs and setup guards now require
that additional explicit approval. Neither credential gains update rights,
other-folder or account administration. New-share metadata and failed-attempt
receipts are retained in the October16 live evidence folder. This supersedes
the initial no-re-sharing provider assumption; it does not infer a native test
pass from approval.

Node preflight also verified that `railiance01` is an SSH alias whose actual
hostname is `239.62.205.92.host.secureserver.net`. Runtime/installer now require
that exact hostname, the reviewed machine-ID hash and the original cluster UID,
plus root. No server hostname or machine identity was changed.52 focused
checks passed including the real synthetic encrypted/native PG fixture after
this guard repair. Unapproved-CCR negative tests use proposed fixtures even
when the actual owner requests have been approved.

Both exact guarded DNS-selector patches passed server-dry-run and were applied.
The node has active k3s with network-policy rules present. The source-bound
runtime release was installed and unit validation completed; both timers remain
disabled. Provider admission/readback and actual full capture remain pending;
no periodic capture/expiry is enabled by installation alone. One subsequent
OIDC attempt timed out before owner-command handoff; it did not run mutations,
and its possible session revocation was unconfirmed. Only the contained owner
login lane is retried, without switching to root or inspecting captured bytes.


The native publisher fixture subsequently proved read/create, denied overwrite
and delete, and exact fixture cleanup. Its malformed parent path was rejected
withHTTP400. The checker now records that separately and also requires a valid
outside-account-folder request to fail authentication; accepting a malformed
path rejection alone would not establish authorization confinement.16 provider
checks pass after this correction. A fresh metadata capture plan confirms zero
allow policies matching helper Pods. The next owner login timed out before
handoff; no corrected provider probe or KV/ESO publication ran in that attempt.
The installed runtime's timers remain disabled while attended admission resumes.


## First full bundle created — 2026-10-16

Both exact roles passed native login, effective single-policy/900-second lifetime,
metadata/sibling/old-escrow/token-create denials, wrong-SA/namespace/audience
rejection, ESO value equality and temporary-reader revocation. Both ExternalSecrets
are Ready and KV versions are1. The operator session self-revoked. CCR-0048/0049
are applied with actual bound-claim evidence; the provider-only request5/9
assumption has been superseded by separately approved21/25.

ESO adds known no-transformation defaults (`conversionStrategy: Default`,
`decodingStrategy: None`, `metadataPolicy: None`) to its spec. The delivery guard
now accepts only those defaults while rejecting extra sources or altered
conversion/property/version/reader bindings. The disabled node units were
replaced with exact previous-packet/content guards, timers verified disabled and
inactive, and preserved rollback bytes; source release
`758ba4d170ba5ed4b6c563d3a379e496c5f81e8bff28c45d142a6bacbfaa21bf`
is installed. 56 focused checks passed, including native synthetic full-archive
PostgreSQL recovery. The task-only disabled-upgrade source and packet are in the
live evidence folder; this is not permission for an unchecked active-job upgrade.

Actual first capture completed under the approved node owner, with all required
source identity guards and native data snapshots. Both temporary helpers and
private plaintext were cleaned. The exact new encrypted artifact is:

- Name: `net-kingdom-identity-20261016T063159Z-5882771aa2ce.tar.age`
- Folder: Storage Share `Backup/railiance-backups/net-kingdom-identity/`
- Bytes:109738344
- Ciphertext SHA256:`7117b3fe1e4840c86b301c94c85e6b16709e5626bb0884bd0580bde630e30e30`
- Status: unique publication and fresh downloaded readback verified; safe index published last.

Private node receipts live under `/var/lib/net-kingdom-backup/runs/` for that
artifact. No age private key was used in capture/upload. The first independent
owner retrieval and human hidden-key archive proof follow; both timers remain
disabled until the actual proof is reviewed. Full directory/issuer/MFA/token and
audit-continuity acceptance remains distinct from archive recovery.


## Actual independent full recovery and activation — 2026-10-16

The attended Backup-account download succeeded and its owner session self-revoked.
The independently fetched 109738344-byte ciphertext was rehashed locally against
the exact published digest. The operator then ran the hidden-key ceremony through
`railiance-bootstrap/bin/railiance identity-recovery`. The original independent
backup key matched the declared recipient; an unrelated key was refused; all 36
manifest files/hashes, original key pairs and SQLite integrity checked; the native
networkless disposable PostgreSQL 16.14 restore read back 5 tokens/2 resolvers/2 realms.
The checker source SHA256 and exact artifact/digest were reviewed, plaintext and
container cleanup passed, and no NK49 Docker containers remain. The agent
received no age private key or restored secret values.

Safe evidence: `independent-owner-transfer.json`, `actual-full-archive-recovery.json`
and `actual-full-recovery-review.json` in the October16 live evidence directory.
This proves actual archive/file/native-database recovery, with full-service
authentication/token decryption/audit-continuity fields explicitly false.

The accepted receipt was copied as root-private state and bound to the exact
successful node off-site receipt. The approved installer activation succeeded;
both timers are enabled and active. Next backup:2026-10-17T03:47:39Z (05:47CEST,
including the configured jitter). Next hourly health run:2026-10-16T08:00:00Z
(10:00CEST). The actual health unit completed with Result=success/ExecMainStatus=0.
A live retention preview listed the recovered artifact as protected, selected no
expiry candidates and deleted nothing; unknown objects remain untouched. No
capture/retention failure marker or pending event outbox remains. Runtime cadence
is daily 03:45 UTC plus up to 5 minutes jitter, hourly health and 25-hour freshness;
the recorded next times are observations, not permanent calendar values.

`activation-result.json` and `activation-review.json` record the actual state.
The verified first generation stays pinned until a separately reviewed owner
change. Both archive creation/retrieval/recovery and the backup job setup are now
complete. NK49 remains active because broader native service/caller and legacy
custody acceptance are separate open closure criteria.


The installed hardened systemd backup service was then run once end to end.
`Result=success` and `ExecMainStatus=0` cover pre-retention, scheduled capture,
fresh upload/download verification and post-retention. It produced a second
verified generation, `net-kingdom-identity-20261016T073927Z-1224cb9362f8.tar.age`,
SHA256 `2b88375c378aac52fbd82f2bfa21fbbd9af61d8f88abdad2ed3f997dc9993b9c`.
Actual retention kept both the independently recovered first point and the newer
point, selected/deleted nothing and removed no local generations. Helpers and
plaintext were cleaned; failure markers and event outbox remain empty. This
validates the actual hardened recurring-job path, beyond a direct CLI capture.
Evidence: `recurring-job-review.json`. No further validation captures are needed
before the next scheduled run absent a new failure or source change.


## Native token-material and audit acceptance — 2026-10-16

The independently retrieved identity generation passed native privacyIDEA checks
on the fixed isolated local engine: all five retained token materials were read
internally using the original enckey; unrelated-key results differed for all five.
All 279,209 stored historical signatures verified, with zero unsigned/invalid
rows. A new isolated entry verified with the preserved identity; tampering and
an unrelated public key were refused. Container/plaintext cleanup passed.
T20/T31 are done. Safe evidence and source/ciphertext/target bindings are in
`evidence/2026-10-16-identity-outage-recovery/actual-isolated-privacyidea-review.json`.
This accepts internal readback and stored-row signature continuity. Existing-token
MFA, Authelia and KeyCape authentication remain separate; event completeness is
not claimed. The published Storage Share kit remains its recorded version.

## KeyCape original attempt assembly (NK-WP-0049-T33) — 2026-10-21

This is the attended order for the original KeyCape attempt. It is preparation and
authorizes nothing. T18 owns the actual run. Abort at any non-zero step, before the
hidden key prompt.

1. **Founder attempt admission (surviving offline authorization).** The founder
   signs a `secrets-engine.recovery-attempt-admission.v1` document with namespace
   `secrets-engine-recovery-attempt`, case `secrets15-nk49-readiness`, a window of
   at most 24 h, issuer `https://kc.coulomb.social`, and the exact hashes below
   (secrets-engine `docs/founder-admission.md`; approved in principle by hub decision
   `d1ab097d-cc79-4612-9c17-bb9b364081ac`, not yet signed). The verifier anchor
   `/etc/secrets-engine/admission/allowed_signers` (root-owned, runtime-unwritable)
   must exist on the attempt host. It is absent on the workstation today.
2. **Selector preflight with admission. This step is required.** From a secrets-engine checkout at
   `0f44fbeea18147e65f379632621f2f29ded5f310` (or a replacement the owner names):
   ```sh
   python3 -I tools/check-keycape-recovery-selector.py \
     --selector docs/keycape-recovery-selector.json \
     --source-sha256 sha256:c0ade429edf452ad17f05d690b7ebef091db7f1a42b5ac350f1e40955c8586c0 \
     --admission <attempt.json>
   ```
   Proceed only with exit 0 and status `founder_admitted_attempt_not_native_acceptance`.
   The selector bytes must hash to
   `sha256:02077af3c18b24f5e4a85992a3d4d0df51a8e458de1e48b48f7eb5f482257012`. The
   NK49 admission's consumer block must equal the selector's eight fields
   (`secrets-engine-openbao`, `KEYCAPE_SECRETS_ENGINE_OPENBAO_CLIENT_SECRET`,
   audience `secrets-engine-openbao`, scope `openbao:login`, subject
   `service:secrets-engine`, tenant `tenant:platform`, roles `secrets-engine`,
   deny `openbao:admin`). A pass shows founder authority for this one attempt. It
   is not native acceptance and not single use. The NK49 execution keeps single-use
   consumption and cleanup.
3. **TLS/trust input: ACME reissue** (founder decision
   `9f9d0d88-837a-433c-8a11-d30b633b0073`, resolved 2026-10-21 = A). The
   `kc.coulomb.social` TLS identity is reissued through `letsencrypt-prod` ACME
   HTTP-01 and chains to ISRG Root X1 (trust sha256 `22b557a2…43d1`). The original
   key is not required, and no self-signed substitute is allowed.
   **Before the real attempt**, the owner (railiance-cluster) rehearses the same
   issuance against Let's Encrypt staging. This protects the duplicate-certificate
   limit of 5 per week. At attempt time, record the public chain with
   railiance-cluster `tools/public-tls-metadata.py` and check SAN
   `kc.coulomb.social` plus the ISRG Root X1 chain. Abort if HTTP-01 cannot
   complete or the chain differs. The TLS key stays in the owner's own terminal.
4. **Existing six-argument launcher** `tools/bootstrap-recovery/run-isolated-keycape.sh`
   with the reviewed admission digest. Its T34 source closure and abort/cleanup
   behaviour are unchanged. The private manifest is derived and the archive
   registration is compared to the selector inside custody only.

5. **Safe receipt and independent verification.** The launcher prints
   `Safe receipt: <tmpdir>/recovery.json`. Before anyone copies it, verify it
   from a separate shell against a clean checkout at the reviewed commit:
   ```sh
   python3 -I tools/bootstrap-recovery/verify-keycape-attempt-receipt.py \
     --receipt <tmpdir>/recovery.json \
     --transfer-receipt <verified transfer receipt> \
     --source-root <clean checkout>/tools/bootstrap-recovery \
     --not-after <admission window end, ISO 8601 with offset>
   ```
   The admission window, at most 24 h and signed by the founder, is the execution
   deadline. Retain the receipt only on exit 0 (`verified`), and keep it at
   `docs/evidence/<date>-nk49-t18-keycape-attempt/recovery.json` together with the
   verifier output. On exit 1, keep it out of Git and report the findings. Do not
   repair it by hand. The verifier refuses secret-shaped content, open cleanup
   obligations, daemon uncertainty, any claim of consumer reconnect or human MFA,
   a ciphertext that differs from the transfer receipt, and source that differs
   from the checkout. A pass is not consumer continuity (T18), retained-factor MFA
   (T19) or owner acceptance.

Still open before T33 can close (2026-10-23): the anchor installed on the attempt
host (SECRETS-WP-0015-T04, founder- and root-owned) and the owner's Let's Encrypt
staging rehearsal from step 3. T35, T36 and T39 are done. The deadline, receipt
location and verifier are fixed above. Public observations are in
`docs/evidence/2026-10-21-nk49-t33-readiness/readiness-observation.json`.