diff --git a/Makefile b/Makefile index 7f04965..0dab731 100644 --- a/Makefile +++ b/Makefile @@ -139,7 +139,7 @@ creds-bundle: ## Age-encrypt the ops bundle for offsite storage creds-apply: ## Apply all create-secrets.sh scripts to the cluster in dependency order bash sso-mfa/bootstrap/creds-apply.sh -creds-verify: ## Check all expected K8s secrets exist and update creds-state.yaml +creds-verify: ## Observe current Secret presence without rewriting native acceptance bash sso-mfa/bootstrap/creds-verify.sh creds-status: ## Print human-readable credential state from creds-state.yaml diff --git a/docs/attended-procedure-inventory.md b/docs/attended-procedure-inventory.md index 14f9735..8a3ab68 100644 --- a/docs/attended-procedure-inventory.md +++ b/docs/attended-procedure-inventory.md @@ -1,4 +1,4 @@ -# Attended procedure inventory — 2026-09-05 +# Attended procedure inventory — 2026-09-05 baseline, reviewed 2026-10-15 Reviewed all 32 shell scripts under `sso-mfa/k8s/` for NK-WP-0034-T03. Exercise status: unknown unless the row states otherwise. Existing deployment @@ -11,12 +11,12 @@ runbook or a dedicated procedure. Missing runbooks are listed separately. | Script (under `sso-mfa/k8s/`) | Runbook | Exercise status | | --- | --- | --- | | `authelia/create-secrets.sh` | [README.md](../sso-mfa/k8s/authelia/README.md), [smooth-bootstrap-guide.md](../docs/smooth-bootstrap-guide.md) | unknown | -| `keycape/configure-openbao-oidc.sh` | [README.md](../sso-mfa/k8s/keycape/README.md) | unknown | -| `keycape/create-pi-token.sh` | [README.md](../sso-mfa/k8s/keycape/README.md) | unknown | +| `keycape/configure-openbao-oidc.sh` | [README.md](../sso-mfa/k8s/keycape/README.md) | retired 2026-10-15 before token/cluster access; Platform governed owner operation replaces the root-token pod prompt | +| `keycape/create-pi-token.sh` | [README.md](../sso-mfa/k8s/keycape/README.md) | retired; current factor reader is Platform-owned RPF-WP-0040; historical Secret/provider revocation remains open | | `keycape/create-secrets.sh` | [README.md](../sso-mfa/k8s/keycape/README.md), [smooth-bootstrap-guide.md](../docs/smooth-bootstrap-guide.md) | unknown | -| `keycape/patch-openbao-client.sh` | [README.md](../sso-mfa/k8s/keycape/README.md) | unknown | -| `keycape/refresh-pi-token-live.sh` | [README.md](../sso-mfa/k8s/keycape/README.md) | unknown | -| `keycape/register-coulomb-social.sh` | None found | unknown | +| `keycape/patch-openbao-client.sh` | [README.md](../sso-mfa/k8s/keycape/README.md) | retired 2026-10-15; use reviewed guarded registration/rollout adapters | +| `keycape/refresh-pi-token-live.sh` | [README.md](../sso-mfa/k8s/keycape/README.md) | retired; no personal-admin token refresh or live acceptance claimed | +| `keycape/register-coulomb-social.sh` | None found | retired 2026-10-15; raw registry replacement/output removed | | `keycape/verify-openbao-client.sh` | [README.md](../sso-mfa/k8s/keycape/README.md), [smooth-bootstrap-guide.md](../docs/smooth-bootstrap-guide.md) | unknown | | `lldap/bootstrap-users.sh` | [README.md](../sso-mfa/k8s/lldap/README.md), [OPERATOR-GROUPS.md](../sso-mfa/k8s/lldap/OPERATOR-GROUPS.md) | source + isolated native fixture verified 2026-10-14; estate exercise still unknown; [procedure](../sso-mfa/k8s/lldap/GROUP-MAINTENANCE.md) | | `lldap/break-glass.sh` | [platform-root-custody.md](../docs/platform-root-custody.md) | retired 2026-10-14; [owner routes](legacy-lldap-lifecycle-retirement.md) | @@ -29,12 +29,12 @@ runbook or a dedicated procedure. Missing runbooks are listed separately. | `privacyidea/bootstrap-realm.sh` | [privacyIDEA repair contract](../sso-mfa/k8s/privacyidea/README.md#configuration-preserving-realm-repair) | source repaired: literal private files, preview/explicit apply, configuration preservation; live acceptance pending | | `privacyidea/check-user-mfa-state.sh` | [smooth-bootstrap-guide.md](../docs/smooth-bootstrap-guide.md) | source repaired: explicit admitted private JWT, scoped read-only counts; live diagnostic delivery not admitted by this repair | | `privacyidea/create-secrets.sh` | [README.md](../sso-mfa/k8s/privacyidea/README.md), [smooth-bootstrap-guide.md](../docs/smooth-bootstrap-guide.md) | unknown | -| `privacyidea/enckey-bootstrap.sh` | [README.md](../sso-mfa/k8s/privacyidea/README.md) | unknown | +| `privacyidea/enckey-bootstrap.sh` | [README.md](../sso-mfa/k8s/privacyidea/README.md) | source/isolated native recovery adapter verified 2026-10-13; current estate recovery remains NK-WP-0049-T03 | | `privacyidea/reconcile-lldap-resolver-live.sh` | [keycape-exposure-resolver-reconciliation.md](../docs/keycape-exposure-resolver-reconciliation.md) | historical check 2026-09-23 with predecessor unproven; current core exercised 2026-10-14 on disposable PI/LLDAP, full production procedure unexercised | | `privacyidea/repair-realm-live.sh` | [privacyIDEA repair contract](../sso-mfa/k8s/privacyidea/README.md#configuration-preserving-realm-repair), [verify-t06.md](../docs/verify-t06.md) | source repaired: attended in-memory input and propagated functional proof; live acceptance pending | | `privacyidea/update-lldap-resolver-live.sh` | [keycape-exposure-resolver-reconciliation.md](keycape-exposure-resolver-reconciliation.md) | core exercised 2026-10-14 on disposable PI/LLDAP; production unexercised | -| `user-engine/verify-operability.sh` | None found | unknown | -| `verify-identity-cutover-dependencies.sh` | None found | unknown | +| `user-engine/verify-operability.sh` | None found | retired 2026-10-15; use rapp-user-engine verification; no stale manifest rollback | +| `verify-identity-cutover-dependencies.sh` | None found | read-only source review; tracing disabled and private staging 2026-10-15; no new native exercise claim | | `verify-t02.sh` | [README.md](../sso-mfa/k8s/README.md) | unknown | | `verify-t03.sh` | [DR-RUNBOOK.md](../sso-mfa/k8s/backup/DR-RUNBOOK.md) | unknown | | `verify-t04.sh` | [DR-RUNBOOK.md](../sso-mfa/k8s/backup/DR-RUNBOOK.md) | unknown | @@ -57,3 +57,31 @@ header now says unexercised to successful completion. Existing incident authority/window requirements still apply; this documentation review does not authorize that attended cutover. The changed helper must be included in the revision recorded by the operator. + +## Bootstrap/root-output caller review — 2026-10-15 + +The original 32-shell-script snapshot above is historical and does not cover the +subsequently added Python adapters or bootstrap/Make callers. The current exact +scope, source digests and per-entrypoint dispositions are recorded in +[the caller source inventory](evidence/2026-10-15-bootstrap-recovery/caller-source-inventory.json). +It enumerates all current non-test shell/Python files under bootstrap and k8s, +plus Make, the console, legacy delivery copier and isolated transport checker. +Enumeration is complete for that declared scope; it is not a claim that every +helper has current production acceptance or that no future defect can exist. + +| Caller group | Current source boundary | Native acceptance | +| --- | --- | --- | +| Generate/encrypt/decrypt/bundle | Public recipient for encryption; explicit private temporary key for decryption; private files; fresh no-overwrite bundle output; no age private key in bundle; no plaintext tar intermediate | Disposable actual age checks; exact off-host artifact/current coverage pending | +| SOPS setup/read/edit/encrypt/rotate/unlock | Existing private-input and recipient checks; no automatic permanent HOME key installation | Existing source/disposable receipts; independent recovery pending | +| Component create/apply/rotation | Literal private input, create-or-match or explicit named field update; no credential argv; contained errors; unknown writes require reconciliation | Current multi-consumer recovery/revocation remains owner work | +| Bootstrap orchestration | Attended entry; no automatic erase/regeneration of retained service custody; failed OpenBao verification stops before final cleanup; encrypted handoff is not proof of recovery | No production bootstrap executed in this review | +| Lab OpenBao init/unseal | Explicit lab model, admitted kubeconfig and exact cluster UID; private no-overwrite init receipt; shares stdin only; no automatic post-init root configuration | Captured fake-cluster transport tests only; historical lab receipt does not cover new revision | +| Legacy User Engine delivery copier | Retired; scoped email ESO delivery remains authoritative; audit delivery replacement needs owner custody path; existing Secrets untouched | No provider token copied, rotated or revoked | +| Legacy registry patchers/root prompt | Retired; raw Python registry patch output disabled; guarded owner adapters remain available | Existing registry/clients unchanged | +| Secret presence diagnostic | Current scoped factor Secret replaces old personal-token expectation; historical applied/native state never rewritten from presence | Presence only | +| Dependency/client verification | No tracing of decoded config; private staging; suppressed provider response/port-forward diagnostics | No new native acceptance claimed | +| Legacy managed workload rollback | Retired; current rapp-user-engine release/verification contract is authoritative | No workload rollback or reapply performed | + +The [recovery ceremony](bootstrap-recovery-ceremony.md) records exact outstanding +owner evidence. Retired callers are not evidence of provider revocation. A +successful local backup job is not evidence of an off-host matching key/data set. diff --git a/docs/bootstrap-recovery-ceremony.md b/docs/bootstrap-recovery-ceremony.md new file mode 100644 index 0000000..d571c8c --- /dev/null +++ b/docs/bootstrap-recovery-ceremony.md @@ -0,0 +1,118 @@ +# NK-WP-0049 recovery ceremony and closure packet + +Status: preparation; no live restore, credential read/grant, revocation, rotation, +root retrieval or production bootstrap is authorized by this document. + +## What has been established + +On 2026-10-15 the operator identified Scaleway and Storage Share as the off-host +backup providers. This identifies destinations, not the exact recovery artifact, +its freshness, its file coverage or independently held decryption key. Platform's +current backup contracts distinguish Scaleway database/full archives from the +Nextcloud secondary copy. An upload-only share is not a retrieval identity. + +Read-only primary-cluster metadata is retained under +`docs/evidence/2026-10-15-bootstrap-recovery/`. LLDAP, Authelia, KeyCape, +privacyIDEA, identity-provisioner and User Engine each have one Ready replica. +The three SQLite/key backup CronJobs report successful October 15 invocations. +Their current commands target local PVCs; neither an external export nor complete +off-host key/database coverage is established by those observations. The +privacyIDEA command mentions its encryption key, but not its audit key pair. +This command inspection is a coverage question, not proof that no other backup +exists. The personal-token and trigger-admin Secrets still exist. Neither their +validity nor provider-side revocation has been tested. A scan of 34 current +Deployment/StatefulSet/DaemonSet/CronJob/Pod resources in sso, mfa and user-engine +found no references to those two old Secrets; it still found the expected audit +delivery reference in User Engine. Other namespaces, historic jobs, external +consumers and issued provider credentials are outside that scan. + +## Assemble the exact owner packet before execution + +Record only safe references and checksums. Keep credentials in their current +owner custody and pass the decryption key through attended temporary unlock. + +| Required owner input | Acceptance | +| --- | --- | +| Off-host artifact | Provider, account/folder or object/version, backup timestamp, ciphertext SHA256 and independent retrieval receipt; do not substitute a new local export for the existing off-host backup. | +| Independent key | Human confirms matching custody outside the servers and OpenBao. A successful decrypt using only the OpenBao escrow key does not prove independent recovery. Never send the key in chat. | +| Required assets | Exact manifest of matching service env/config files, LLDAP and Authelia data, privacyIDEA database plus original encryption/audit keys, and KeyCape registry/signing material. Resolve assets held through other current owner lanes explicitly. | +| Recovery environment | Named disposable target, pinned images available independently of the failed production registry, no production ingress, no production identity/client execution, bounded egress, private storage and exact cleanup scope. | +| Native proof | Owner-specific database integrity, identity/group counts, decryption of retained token material, audit signature continuity, issuer/client/signing configuration preservation and admitted login/MFA checks. Distinguish an isolated restore from current production credential validity. | +| Current consumer source | Reconcile the deployed revisions with the repaired caller/provider source. The October 14 reserved platform-group fix must be deployed and independently checked before claiming live enforcement. | +| Legacy credential disposition | IAM owner identifies the exact historical personal-token/trigger principals, actual consumers and revocation method before any deletion. Deleting a Secret does not revoke an issued provider credential. | + +The current independent key availability and exact artifact/manifest are pending. +Do not draft a blanket restore or grant approval while these remain unknown. +Once fixed, bind the source revisions, target identities, allowed operations, +recovery/cleanup sequence and maintenance window to the reviewed live packet. +Existing eligibility or another client's approval is not this ceremony's grant. + +## Execute in stages + +1. Use the existing Platform/provider recovery lane to retrieve the identified + ciphertext, verify its object/version and digest, and retain a safe transfer + receipt. Do not request or expose the provider credential to the agent. +2. The human unlocks the independently held age key in the existing custody + helper. Validate it against the registered public recipient. No permanent + key is installed in HOME, generated as a replacement, or copied into a bundle. +3. Run the local transport/coverage check below. It contacts no cluster, keeps + plaintext in a private temporary directory outside Git, refuses unsafe + archives/private age keys, tests an unrelated disposable wrong key, reads the + restored files back and removes that staging. It emits a closed non-secret + receipt. This is one prerequisite, not whole-service recovery acceptance. +4. Under the exact separately reviewed owner ceremony, restore the matching + service data/key/config sets into the disposable environment and perform the + native checks above. Never run the full production bootstrap to test source + repairs. Unknown writes stop for reconciliation rather than automatic retry. +5. Verify cleanup, preserve the original ciphertext and independent custody, + and reconcile current callers/legacy credentials through their owners. + Retain exact source/object/image/actor/time and positive/negative receipts. +6. Close T02/T03 only after the recorded remaining requirements are satisfied. + Update the workplan first, run `statehub fix-consistency --repo net-kingdom`, + and verify the Hub projection. A transport fixture or generated evidence + worksheet cannot mark this plan finished. + +## Prepared local transport check + +The human invokes this through the existing attended custody helper after the +exact artifact and required-file list are agreed. `NETKINGDOM_CUSTODY_AGE_KEY_FILE` +is the temporary private file provided inside that envelope. The tool never +reads an ambient HOME key. The receipt destination must be new. + +```bash +python3 tools/bootstrap-recovery/verify-bundle.py \ + --bundle /private/recovery/fetched-ops-bundle.tar.age \ + --bundle-sha256 \ + --required-file lldap/secrets.env \ + --required-file privacyidea/secrets.env \ + --required-file privacyidea/pi.enc \ + --required-file privacyidea/private.pem \ + --required-file privacyidea/public.pem \ + --receipt /private/recovery/new-transport-receipt.json +``` + +This example file list is not an accepted whole-stack manifest. Add every +agreed service file supported by the closed transport profile; database copies +and other archive formats use their native owner restore tools. A successful +receipt explicitly leaves independent custody, current service validity and +native consumer recovery unproven. Off-host retrieval lineage and human custody +confirmation must accompany it; the tool cannot infer where a local input came +from. A failed check publishes no acceptance receipt. + +## Rotation boundaries + +CUST-WP-0073-T05 remains founder-deferred under decision +`1591daa1-6e85-4dfd-aadd-5a9fe1ef47fa`. This work does not reopen that rotation, +replace signing keys, change directory passwords or erase old backup keys. +Existing historical exposure, provider revocation and coordinated rotation +requirements stay explicit in their live owner records. If completion needs a +protected native operation, prepare that exact operation and obtain its scoped +disposition; never treat a source repair or successful backup job as permission. + +## Knowledge/source scope + +Emergent Security 0.2.6, package digest +`8d1d6a9d4d07fec7b619618de7dfe2be2254c1790f85f4198c7fe19db9751c02`: +start/root cards withheld, orientation only. Current SECURITY-COMPANION, +Platform backup/custody procedures, owner runtime contracts and the workplan +were reviewed as sources. No execution authority is derived from the cards. diff --git a/docs/evidence/2026-10-15-bootstrap-recovery/backup-command-metadata.json b/docs/evidence/2026-10-15-bootstrap-recovery/backup-command-metadata.json new file mode 100644 index 0000000..44d4484 --- /dev/null +++ b/docs/evidence/2026-10-15-bootstrap-recovery/backup-command-metadata.json @@ -0,0 +1,58 @@ +{ + "observed_at": "2026-10-15T20:51:39.050158+00:00", + "read_only": true, + "jobs": [ + { + "namespace": "sso", + "name": "lldap-backup", + "uid": "6f150dff-3ef9-4a35-8c20-cfedf3fc3d0c", + "images": [ + "nouchka/sqlite3:latest" + ], + "command_sha256": "7aad6525ff542febe20aa23a4a9d57754014b291b56cff8c1d35a9c85bc7d7de", + "pvc_refs": [ + "lldap-data" + ], + "command_mentions_enckey": false, + "command_mentions_private_audit_key": false, + "command_mentions_public_audit_key": false, + "command_mentions_s3_or_webdav": false, + "coverage_semantics": "Command-string observations only; neither success nor complete off-host recovery is inferred." + }, + { + "namespace": "sso", + "name": "authelia-backup", + "uid": "b00880db-acbb-413d-ad0c-bce2354449b6", + "images": [ + "nouchka/sqlite3:latest" + ], + "command_sha256": "2f2cd0b65e840edbfa5a618d52286035bc769167a29540646fae11c89f6e69fc", + "pvc_refs": [ + "authelia-data" + ], + "command_mentions_enckey": false, + "command_mentions_private_audit_key": false, + "command_mentions_public_audit_key": false, + "command_mentions_s3_or_webdav": false, + "coverage_semantics": "Command-string observations only; neither success nor complete off-host recovery is inferred." + }, + { + "namespace": "mfa", + "name": "privacyidea-backup", + "uid": "8bce84b6-a10a-445d-a3c2-bfd2e3ab8d8c", + "images": [ + "busybox:stable" + ], + "command_sha256": "b4146f4b110ae3448f86552da2087b6c612b920ce33041d2cc9c2287c5287eaf", + "pvc_refs": [ + "privacyidea-data" + ], + "command_mentions_enckey": true, + "command_mentions_private_audit_key": false, + "command_mentions_public_audit_key": false, + "command_mentions_s3_or_webdav": false, + "coverage_semantics": "Command-string observations only; neither success nor complete off-host recovery is inferred." + } + ], + "offhost_asset_coverage_confirmed": false +} diff --git a/docs/evidence/2026-10-15-bootstrap-recovery/backup-schedule-metadata.json b/docs/evidence/2026-10-15-bootstrap-recovery/backup-schedule-metadata.json new file mode 100644 index 0000000..214fb90 --- /dev/null +++ b/docs/evidence/2026-10-15-bootstrap-recovery/backup-schedule-metadata.json @@ -0,0 +1,32 @@ +{ + "observed_at": "2026-10-15T20:51:17.348168+00:00", + "read_only": true, + "rows": [ + { + "namespace": "mfa", + "name": "privacyidea-backup", + "uid": "8bce84b6-a10a-445d-a3c2-bfd2e3ab8d8c", + "schedule": "30 3 * * *", + "suspend": false, + "last_successful_time": "2026-10-15T03:30:05Z" + }, + { + "namespace": "sso", + "name": "authelia-backup", + "uid": "b00880db-acbb-413d-ad0c-bce2354449b6", + "schedule": "15 3 * * *", + "suspend": false, + "last_successful_time": "2026-10-15T03:15:04Z" + }, + { + "namespace": "sso", + "name": "lldap-backup", + "uid": "6f150dff-3ef9-4a35-8c20-cfedf3fc3d0c", + "schedule": "0 3 * * *", + "suspend": false, + "last_successful_time": "2026-10-15T03:00:06Z" + } + ], + "offsite_retrieval_verified": false, + "current_credential_validity_verified": false +} diff --git a/docs/evidence/2026-10-15-bootstrap-recovery/caller-source-inventory.json b/docs/evidence/2026-10-15-bootstrap-recovery/caller-source-inventory.json new file mode 100644 index 0000000..636ce45 --- /dev/null +++ b/docs/evidence/2026-10-15-bootstrap-recovery/caller-source-inventory.json @@ -0,0 +1,446 @@ +{ + "schema": "netkingdom.bootstrap-caller-inventory.v1", + "observed_at": "2026-10-15T20:54:09.856047+00:00", + "scope": "all current .sh/.py non-test files under sso-mfa/bootstrap and sso-mfa/k8s, Makefile, console, delivery copier and recovery transport tool", + "entry_count": 87, + "entries": [ + { + "path": "Makefile", + "sha256": "5d2114f6141bb8504c5c780044302126c07958a5732afdb0d0c0218a4f84210e", + "classification": "owner-scoped adapter/caller; source presence supplies no live approval or acceptance" + }, + { + "path": "sso-mfa/bootstrap/apply-service-secrets.py", + "sha256": "ea9c3418ed00527b9019ab9893f1f3c9fe29d0697f1f0c879e28fdbae373889b", + "classification": "owner-scoped adapter/caller; source presence supplies no live approval or acceptance" + }, + { + "path": "sso-mfa/bootstrap/creds-apply.sh", + "sha256": "a70dba396484b2a353a39349c3d3719f7b6695b9f0e35025ef79e3db0ec2ff53", + "classification": "owner-scoped adapter/caller; source presence supplies no live approval or acceptance" + }, + { + "path": "sso-mfa/bootstrap/creds-bootstrap-agent.sh", + "sha256": "cc27c4f3969c292a913a0ae325cd0d4ec2665f9fffbbd1eb1c5f73b08e398a32", + "classification": "owner-scoped adapter/caller; source presence supplies no live approval or acceptance" + }, + { + "path": "sso-mfa/bootstrap/creds-rotate.sh", + "sha256": "09b387d3555ce2beaeb8d7f000c2d52a1ffea274c793f30bce80a8f3ef5dc2f8", + "classification": "owner-scoped adapter/caller; source presence supplies no live approval or acceptance" + }, + { + "path": "sso-mfa/bootstrap/creds-status.sh", + "sha256": "f0fe0b885fd22f4789baa86cc2438272700aff8634cfa6b27d55c2c860c9f36d", + "classification": "diagnostic; structural observations do not prove recovery or revoke credentials" + }, + { + "path": "sso-mfa/bootstrap/creds-verify.sh", + "sha256": "101b12948c862f21a668b49b3a82efae691ba885d5c8fd7a94a10e095c16a24a", + "classification": "diagnostic; structural observations do not prove recovery or revoke credentials" + }, + { + "path": "sso-mfa/bootstrap/decrypt-secrets.sh", + "sha256": "1572c170ceb9eff9ff408409acb82650ac911edb02961cde48806c68fa80ce76", + "classification": "private custody/local operation; explicit input; owner procedure required" + }, + { + "path": "sso-mfa/bootstrap/emergency-bundle.sh", + "sha256": "c1a0e9853dae6b71537ecb708b05ce967eacf797334192156485ca943dc8c71e", + "classification": "owner-scoped adapter/caller; source presence supplies no live approval or acceptance" + }, + { + "path": "sso-mfa/bootstrap/encrypt-secrets.sh", + "sha256": "5b4774488e09a22b409079021e7cd10632c7cf6cf9db6f98cd85e3878d34d435", + "classification": "private custody/local operation; explicit input; owner procedure required" + }, + { + "path": "sso-mfa/bootstrap/gen-secrets.sh", + "sha256": "b4a910cca93427c0e70e1f4c7646fc94d9b885aeae65a10a01cf8e11548ce03d", + "classification": "private custody/local operation; explicit input; owner procedure required" + }, + { + "path": "sso-mfa/bootstrap/openbao-init-unseal.sh", + "sha256": "49485948a8d44576e293b6f709f1bb444e36bed3310c843d47393c3c8d4c1d0b", + "classification": "lab-only custody adapter; explicit target; production forbidden" + }, + { + "path": "sso-mfa/bootstrap/openbao-lab.py", + "sha256": "0c14e38f86e189f6123b99e036e4423ebb31266d90f4c4423d3619c2060c3bc7", + "classification": "lab-only custody adapter; explicit target; production forbidden" + }, + { + "path": "sso-mfa/bootstrap/pack-bundle.sh", + "sha256": "53abc7af3dcda6cfa9f52e9c10d352b66defa2cbfa7cd71b07c483c6f4a60f66", + "classification": "private custody/local operation; explicit input; owner procedure required" + }, + { + "path": "sso-mfa/bootstrap/rotation-env.py", + "sha256": "bab079a51b8e7ad0199676e76e473a13e70f46e0495d8c942502c952236ec893", + "classification": "private custody/local operation; explicit input; owner procedure required" + }, + { + "path": "sso-mfa/bootstrap/sops-custody-unlock.sh", + "sha256": "8a6c02ace974b489d23869a9e99d3305896f4bdc4c430164df09bb4ce1710959", + "classification": "private custody/local operation; explicit input; owner procedure required" + }, + { + "path": "sso-mfa/bootstrap/sops-private.py", + "sha256": "113d17ee4cd7ad405e73cc51e1823dc02bd2e618e7fa67c1b624f0226e334ec6", + "classification": "private custody/local operation; explicit input; owner procedure required" + }, + { + "path": "sso-mfa/bootstrap/sops-write.py", + "sha256": "5a785e9c8e6a40ca9120a668d93992ea5c3115566ada21933536cc2d931f1ddf", + "classification": "private custody/local operation; explicit input; owner procedure required" + }, + { + "path": "sso-mfa/k8s/authelia/create-secrets.sh", + "sha256": "137e137d973993e3938dbfe656b036e8d4d9643f84b5e8ca7ab30fbc5810ea1c", + "classification": "owner-scoped adapter/caller; source presence supplies no live approval or acceptance" + }, + { + "path": "sso-mfa/k8s/keycape/approval-clients-rollout.py", + "sha256": "488d5cc59e081993dd2b5fc6d9e46bbddcaec4f5170b8f7a5ad0aae3a211a790", + "classification": "owner-scoped adapter/caller; source presence supplies no live approval or acceptance" + }, + { + "path": "sso-mfa/k8s/keycape/authentication-policy-rollout.py", + "sha256": "2c1bd23fded4855bae1eb3a08dbc4510d7f60f2bc4d625d60e471ab78f6f0251", + "classification": "owner-scoped adapter/caller; source presence supplies no live approval or acceptance" + }, + { + "path": "sso-mfa/k8s/keycape/configure-openbao-oidc.sh", + "sha256": "bf214ac511fbd2269935f3270c6d4491c729d834e8138b9d7404aecf23c0825f", + "classification": "retired entry point; refuses before credential/cluster access" + }, + { + "path": "sso-mfa/k8s/keycape/create-pi-token.sh", + "sha256": "2f750aea638c06eb11dfe7afcf7b3d6117ba9e964cbf2a48bb12590054178152", + "classification": "retired entry point; refuses before credential/cluster access" + }, + { + "path": "sso-mfa/k8s/keycape/create-secrets.py", + "sha256": "5da672fff36eb94101741ad99e2a1a96ae81fea15ed1871304c40e63ce27c787", + "classification": "owner-scoped adapter/caller; source presence supplies no live approval or acceptance" + }, + { + "path": "sso-mfa/k8s/keycape/create-secrets.sh", + "sha256": "f5131160d91641a59c3163a17a8dd4f9b97d228eb8a5e0d8ad04cc2f7ff51644", + "classification": "owner-scoped adapter/caller; source presence supplies no live approval or acceptance" + }, + { + "path": "sso-mfa/k8s/keycape/exercise-approval-clients.py", + "sha256": "e2d3dd741ac52af28dce752344227a4443b78fedd4cfcc06ba902e679dfc9351", + "classification": "owner-scoped adapter/caller; source presence supplies no live approval or acceptance" + }, + { + "path": "sso-mfa/k8s/keycape/keycape-client.py", + "sha256": "e31ee2883fad4f93635700608ad98e05984c9e0453515dd7710c897ca856b8c8", + "classification": "owner-scoped adapter/caller; source presence supplies no live approval or acceptance" + }, + { + "path": "sso-mfa/k8s/keycape/openbao-client-config.py", + "sha256": "1b908b4c36134212d78f9fbe7a2a201eb3ced06d31bb1dcea4a59d80689d84b1", + "classification": "owner-scoped adapter/caller; source presence supplies no live approval or acceptance" + }, + { + "path": "sso-mfa/k8s/keycape/openbao-negative-node-wrapper.py", + "sha256": "76ac013d1a676a0dfa58aea1ce87b0c3e72483ccd21f586a22c5475e9132f6c1", + "classification": "owner-scoped adapter/caller; source presence supplies no live approval or acceptance" + }, + { + "path": "sso-mfa/k8s/keycape/openbao-negative-session.py", + "sha256": "24b7b803e4eb7e6e7c0700a069dad0ec315ee7eb5538a88b9e56147f8fab5cdd", + "classification": "owner-scoped adapter/caller; source presence supplies no live approval or acceptance" + }, + { + "path": "sso-mfa/k8s/keycape/openbao-signin-diagnosis-rollout.py", + "sha256": "a57b957d6eed9e8fb09d2a7aa7405a39deeafd9aaf57a9e6dc907afabd380371", + "classification": "owner-scoped adapter/caller; source presence supplies no live approval or acceptance" + }, + { + "path": "sso-mfa/k8s/keycape/patch-openbao-client.sh", + "sha256": "cfd0dbd0528e799ee59d8aa4cd1cefc2189227cbc92d251f2a67bb25212aecad", + "classification": "retired entry point; refuses before credential/cluster access" + }, + { + "path": "sso-mfa/k8s/keycape/playwright-activation.py", + "sha256": "5ac683b3324b00e6253f6040f4c73057027b1f98f7071160a6153794ba522757", + "classification": "owner-scoped adapter/caller; source presence supplies no live approval or acceptance" + }, + { + "path": "sso-mfa/k8s/keycape/playwright-registration.py", + "sha256": "bf2d6b83d93629df8ddaf6c31d6c7408ae3866969d63895714df0e08a570da99", + "classification": "owner-scoped adapter/caller; source presence supplies no live approval or acceptance" + }, + { + "path": "sso-mfa/k8s/keycape/portal-client-rollout.py", + "sha256": "158e1d36eb881cdd0a2eda91b472adea5e8fde5061e00d26dda368580aeb871f", + "classification": "owner-scoped adapter/caller; source presence supplies no live approval or acceptance" + }, + { + "path": "sso-mfa/k8s/keycape/refresh-pi-token-live.sh", + "sha256": "2f750aea638c06eb11dfe7afcf7b3d6117ba9e964cbf2a48bb12590054178152", + "classification": "retired entry point; refuses before credential/cluster access" + }, + { + "path": "sso-mfa/k8s/keycape/register-coulomb-social.py", + "sha256": "b40ff4bdba71b797f8f393e79cb3c9d73b790045a0f7df6ecea35803369793c7", + "classification": "retired entry point; refuses before credential/cluster access" + }, + { + "path": "sso-mfa/k8s/keycape/register-coulomb-social.sh", + "sha256": "cfd0dbd0528e799ee59d8aa4cd1cefc2189227cbc92d251f2a67bb25212aecad", + "classification": "retired entry point; refuses before credential/cluster access" + }, + { + "path": "sso-mfa/k8s/keycape/register-user-engine-portal.py", + "sha256": "67173edc44f5457e7b6a3e0fb2321b398159cfce5dea8ead01eabf835fb38d76", + "classification": "owner-scoped adapter/caller; source presence supplies no live approval or acceptance" + }, + { + "path": "sso-mfa/k8s/keycape/secrets-engine-openbao-client-rollout.py", + "sha256": "8b834e5f0353a5e11797637420609e06e7bf7de47777c7e78b774f74d4a6afbd", + "classification": "owner-scoped adapter/caller; source presence supplies no live approval or acceptance" + }, + { + "path": "sso-mfa/k8s/keycape/vergabe-client-rollout.py", + "sha256": "8cee6cc5ff3e3e8617304055263d04b393d0e005c34d9af12cb7e8bc685bac1a", + "classification": "owner-scoped adapter/caller; source presence supplies no live approval or acceptance" + }, + { + "path": "sso-mfa/k8s/keycape/verify-openbao-client.sh", + "sha256": "6f8e75ad16f0d35f776d1a8da85842dc1174085a730f0189a7f5a36a0d7db83f", + "classification": "diagnostic; structural observations do not prove recovery or revoke credentials" + }, + { + "path": "sso-mfa/k8s/keycape/verify-user-engine-portal.py", + "sha256": "ccd5106395bd5d7b4f3dbbcbad449e3793aff217d4429b5d5af9e52ab9a58103", + "classification": "diagnostic; structural observations do not prove recovery or revoke credentials" + }, + { + "path": "sso-mfa/k8s/lldap/bootstrap-users.sh", + "sha256": "309da1de430638e4ed0b9bfe11b7bbb472a6bfe364cdfb6c69499c6fb78b66d6", + "classification": "owner-scoped adapter/caller; source presence supplies no live approval or acceptance" + }, + { + "path": "sso-mfa/k8s/lldap/break-glass.sh", + "sha256": "a30cc22a990d368b9066e2e8b7aaef1c993c1c8a86e191ad2afb0e5c4d4f3294", + "classification": "retired entry point; refuses before credential/cluster access" + }, + { + "path": "sso-mfa/k8s/lldap/create-secrets.sh", + "sha256": "fb2ee3b5af858297d20f8bc8821ee13a86d6ebc9235756b9ed9a8e85a54d84a1", + "classification": "owner-scoped adapter/caller; source presence supplies no live approval or acceptance" + }, + { + "path": "sso-mfa/k8s/lldap/create-user.sh", + "sha256": "a30cc22a990d368b9066e2e8b7aaef1c993c1c8a86e191ad2afb0e5c4d4f3294", + "classification": "retired entry point; refuses before credential/cluster access" + }, + { + "path": "sso-mfa/k8s/lldap/dry-run-nonroot-user.sh", + "sha256": "a30cc22a990d368b9066e2e8b7aaef1c993c1c8a86e191ad2afb0e5c4d4f3294", + "classification": "retired entry point; refuses before credential/cluster access" + }, + { + "path": "sso-mfa/k8s/lldap/identity-provisioner-reconcile.py", + "sha256": "e124b94c3a5749d4e3fdfc5fd054acfb295594beac24cd399a7b24a8617683d6", + "classification": "owner-scoped adapter/caller; source presence supplies no live approval or acceptance" + }, + { + "path": "sso-mfa/k8s/lldap/lldap_admin.py", + "sha256": "5e39e2eac53846a97cd25151c0a4af1ffc0230db881a6325efb66de6a51bfc36", + "classification": "owner-scoped adapter/caller; source presence supplies no live approval or acceptance" + }, + { + "path": "sso-mfa/k8s/lldap/manage-group-members.sh", + "sha256": "3c85e4e882ad003a4dc513d9085d136531511273db8adf27e1fcf5e749ae04d1", + "classification": "owner-scoped adapter/caller; source presence supplies no live approval or acceptance" + }, + { + "path": "sso-mfa/k8s/postgresql/create-secrets.sh", + "sha256": "801ae8a404f0a8fd2bb349003091ab6a509df2533bc1a0f456dd7d5940d917e9", + "classification": "owner-scoped adapter/caller; source presence supplies no live approval or acceptance" + }, + { + "path": "sso-mfa/k8s/privacyidea/activate-onboarding-policy.py", + "sha256": "edd6d3d4507e11e1632002e56f948bb2e81465132500627b5ddc7fa0f06203d1", + "classification": "owner-scoped adapter/caller; source presence supplies no live approval or acceptance" + }, + { + "path": "sso-mfa/k8s/privacyidea/bootstrap-admin-remote.py", + "sha256": "c83c3a5c9a753a8afec90c35ce5ddbfc15f7b8aff437065f428109d9c5774b89", + "classification": "owner-scoped adapter/caller; source presence supplies no live approval or acceptance" + }, + { + "path": "sso-mfa/k8s/privacyidea/bootstrap-admin.py", + "sha256": "b81b5c3806dbdf03ea7cac37884cc368de0f0ef7880ac326b67af3a291c35890", + "classification": "owner-scoped adapter/caller; source presence supplies no live approval or acceptance" + }, + { + "path": "sso-mfa/k8s/privacyidea/bootstrap-admin.sh", + "sha256": "1a11738c1790cc6b5f23ed40f02971591eb76afa486be89d5535b0027aca6989", + "classification": "owner-scoped adapter/caller; source presence supplies no live approval or acceptance" + }, + { + "path": "sso-mfa/k8s/privacyidea/bootstrap-realm.sh", + "sha256": "d3dc530d378ce826f4875c4cf066a6f7f916c3abd5492eb5ee237a8c98b7957b", + "classification": "owner-scoped adapter/caller; source presence supplies no live approval or acceptance" + }, + { + "path": "sso-mfa/k8s/privacyidea/check-user-mfa-state.sh", + "sha256": "18901decfdc974ba5a6e871b7a4fe8022be59c73123774a31c57303fd5f657be", + "classification": "owner-scoped adapter/caller; source presence supplies no live approval or acceptance" + }, + { + "path": "sso-mfa/k8s/privacyidea/create-secrets.sh", + "sha256": "6644b37f2a0ba09b67e135516204950474bd8fa28a243f62e29b898d00c08d91", + "classification": "owner-scoped adapter/caller; source presence supplies no live approval or acceptance" + }, + { + "path": "sso-mfa/k8s/privacyidea/deploy-factor-recovery.py", + "sha256": "9a46aa7a1fdf7c202981f2529bc138dcdd952fbefab1d18a88ac65176936a389", + "classification": "owner-scoped adapter/caller; source presence supplies no live approval or acceptance" + }, + { + "path": "sso-mfa/k8s/privacyidea/deploy-onboarding-guard.py", + "sha256": "95a3195d01775ea1d800aaa37131d440acc92f2a50215a108c0b63a3441cce94", + "classification": "owner-scoped adapter/caller; source presence supplies no live approval or acceptance" + }, + { + "path": "sso-mfa/k8s/privacyidea/enable-factor-confirmation.py", + "sha256": "f71e22d6a7a99f4293fb688567ec11c6826a543223b27168e0577e8a6070581e", + "classification": "owner-scoped adapter/caller; source presence supplies no live approval or acceptance" + }, + { + "path": "sso-mfa/k8s/privacyidea/enckey-bootstrap.sh", + "sha256": "e47b86ae422f043145525b3deaa4c57ce6ee92974db090695be7fafbe7b61a2e", + "classification": "owner-scoped adapter/caller; source presence supplies no live approval or acceptance" + }, + { + "path": "sso-mfa/k8s/privacyidea/enckey-restore.sh", + "sha256": "fb31831c1492006ad5f32e16f2ce47baf6e20d0cb56a592432da0698052bf362", + "classification": "owner-scoped adapter/caller; source presence supplies no live approval or acceptance" + }, + { + "path": "sso-mfa/k8s/privacyidea/key-custody-remote.py", + "sha256": "27b2b885c81923d85054e3999c5613e104fe627779b2b50398ed9bed5088f47e", + "classification": "owner-scoped adapter/caller; source presence supplies no live approval or acceptance" + }, + { + "path": "sso-mfa/k8s/privacyidea/key-custody.py", + "sha256": "e43d21694cb9c99bd02257a4f225973e992c9af4b0e71788929f949b9752f08a", + "classification": "owner-scoped adapter/caller; source presence supplies no live approval or acceptance" + }, + { + "path": "sso-mfa/k8s/privacyidea/keycape-scope-probe.py", + "sha256": "847b8d7c72c11deab15b3d16964e2619011e21d219b1d82c25f0346493abbbef", + "classification": "owner-scoped adapter/caller; source presence supplies no live approval or acceptance" + }, + { + "path": "sso-mfa/k8s/privacyidea/mfa_state.py", + "sha256": "3f0c2e404d7df41af7f8e1fe66a4a6d392338391456ef298c7dd0219ad3d58dc", + "classification": "diagnostic; structural observations do not prove recovery or revoke credentials" + }, + { + "path": "sso-mfa/k8s/privacyidea/pi_api.py", + "sha256": "01bbf3a2c1f8c423f864dc4bd2145c5c89f938fa9e8b0ffbbecf6d674484896b", + "classification": "owner-scoped adapter/caller; source presence supplies no live approval or acceptance" + }, + { + "path": "sso-mfa/k8s/privacyidea/realm_bootstrap.py", + "sha256": "2885086fdc4fd64b58496f90e82c75eefc7171017004e0804defaaabf791b3a9", + "classification": "owner-scoped adapter/caller; source presence supplies no live approval or acceptance" + }, + { + "path": "sso-mfa/k8s/privacyidea/reconcile-lldap-resolver-live.sh", + "sha256": "89f5b279cde441ef8a7c63436e63c6242527d36949aba4bad2134c890a9e0b04", + "classification": "owner-scoped adapter/caller; source presence supplies no live approval or acceptance" + }, + { + "path": "sso-mfa/k8s/privacyidea/repair-realm-live.sh", + "sha256": "2684a6caca848f5d9e74191af4ca2864adba92fe3ee410a4236045d1743b99e6", + "classification": "owner-scoped adapter/caller; source presence supplies no live approval or acceptance" + }, + { + "path": "sso-mfa/k8s/privacyidea/resolver_cutover.py", + "sha256": "6ed2df95671897f073a3dc8fee62b780cdff87b08d617d5a337e1fbba206e10a", + "classification": "owner-scoped adapter/caller; source presence supplies no live approval or acceptance" + }, + { + "path": "sso-mfa/k8s/privacyidea/update-lldap-resolver-live.sh", + "sha256": "ffa7c551dbcc6ab74d2cfc0853fd0647dd026259b1541f99e4a84dcbac16588c", + "classification": "owner-scoped adapter/caller; source presence supplies no live approval or acceptance" + }, + { + "path": "sso-mfa/k8s/privacyidea/verify_mfa.py", + "sha256": "118d0052433edb85dd53e3c988943392f59d0aa00641d45f8ff3676a15f8af19", + "classification": "diagnostic; structural observations do not prove recovery or revoke credentials" + }, + { + "path": "sso-mfa/k8s/user-engine/verify-operability.sh", + "sha256": "2e1cbde1d2aa2ce9f27685baf93137d5f13d33a77421a2fba1649f874f2b7f48", + "classification": "retired entry point; refuses before credential/cluster access" + }, + { + "path": "sso-mfa/k8s/verify-identity-cutover-dependencies.sh", + "sha256": "96da30c4eca28615685ff384bbc7d00d81c7562e8e572a3ad54c3cc6fdd297ca", + "classification": "diagnostic; structural observations do not prove recovery or revoke credentials" + }, + { + "path": "sso-mfa/k8s/verify-t02.sh", + "sha256": "7a9fe613fc83841397f2e3b0b4c26c5464d7b903d5e56a83ec786e2289719b24", + "classification": "diagnostic; structural observations do not prove recovery or revoke credentials" + }, + { + "path": "sso-mfa/k8s/verify-t03.sh", + "sha256": "7b3a888ae20394b761d7cd9cf1bc81f273813794a3002202951b061bd6c27963", + "classification": "diagnostic; structural observations do not prove recovery or revoke credentials" + }, + { + "path": "sso-mfa/k8s/verify-t04.sh", + "sha256": "74128536919f79f17bffc107678718848c4ca095de06140cdc453e68953aabd1", + "classification": "diagnostic; structural observations do not prove recovery or revoke credentials" + }, + { + "path": "sso-mfa/k8s/verify-t05.sh", + "sha256": "bb226bad1e5fb60a5d6f87fa67ecee2d6af84d6d6a84fa02d9839d8bca58a197", + "classification": "diagnostic; structural observations do not prove recovery or revoke credentials" + }, + { + "path": "sso-mfa/k8s/verify-t06.sh", + "sha256": "717317c1d51f7c3361f3b711d79cae52019c36cd6c397e5fa61f07eaad3632e2", + "classification": "diagnostic; structural observations do not prove recovery or revoke credentials" + }, + { + "path": "sso-mfa/k8s/verify-t07.sh", + "sha256": "026406ff1fec9e4bf542741c08002c45ea8a94a4ce5a50980665b601485882df", + "classification": "diagnostic; structural observations do not prove recovery or revoke credentials" + }, + { + "path": "sso-mfa/k8s/verify-t08.sh", + "sha256": "7efe14c0d71f2c846434d19f8d05702d69d3c40a617a700b08fe13b3764d0650", + "classification": "diagnostic; structural observations do not prove recovery or revoke credentials" + }, + { + "path": "tools/bootstrap-recovery/verify-bundle.py", + "sha256": "885b05a2011e711228e42e316bda70bbe0d97d481383d61aa8469d0f10452370", + "classification": "diagnostic; structural observations do not prove recovery or revoke credentials" + }, + { + "path": "tools/security-bootstrap-console/security_bootstrap_console.py", + "sha256": "e9419ab1de0725b12b06fc29098c49584cb8acf47ec0ca09d552915f964a2c48", + "classification": "owner-scoped adapter/caller; source presence supplies no live approval or acceptance" + }, + { + "path": "tools/sync-user-engine-delivery-secret.sh", + "sha256": "7c0fb6b066e4ad8b103e75551beecd5de65a4bbb22570d95b42e5da6395e2dfc", + "classification": "retired entry point; refuses before credential/cluster access" + } + ], + "source_inventory_complete_for_declared_scope": true, + "native_exercise_or_full_security_acceptance_claimed": false, + "remaining": "T02 legacy identity/credential disposition, provider enforcement and owner acceptance; T03 exact off-host asset/key lineage and native recovery. Historical exposure rotation stays founder-deferred." +} diff --git a/docs/evidence/2026-10-15-bootstrap-recovery/legacy-runtime-reference-metadata.json b/docs/evidence/2026-10-15-bootstrap-recovery/legacy-runtime-reference-metadata.json new file mode 100644 index 0000000..3a37f6c --- /dev/null +++ b/docs/evidence/2026-10-15-bootstrap-recovery/legacy-runtime-reference-metadata.json @@ -0,0 +1,28 @@ +{ + "observed_at": "2026-10-15T20:55:09.988184+00:00", + "scope": "deployment,statefulset,daemonset,cronjob,pod in sso,mfa,user-engine; not all estate clients or issued provider credentials", + "resources_scanned": 34, + "references": [ + { + "namespace": "user-engine", + "kind": "Deployment", + "name": "user-engine", + "uid": "e1c4274a-4b6d-4ea2-b619-e09630b745b0", + "secret_names": [ + "user-engine-delivery" + ] + }, + { + "namespace": "user-engine", + "kind": "Pod", + "name": "user-engine-6c894f6b76-bcvpf", + "uid": "ebe2bab8-67b2-4f8d-afea-74b0cdc9d829", + "secret_names": [ + "user-engine-delivery" + ] + } + ], + "secret_values_exported": false, + "provider_revocation_proven": false, + "legacy_tokens_may_have_other_consumers": true +} diff --git a/docs/evidence/2026-10-15-bootstrap-recovery/legacy-secret-metadata.json b/docs/evidence/2026-10-15-bootstrap-recovery/legacy-secret-metadata.json new file mode 100644 index 0000000..0bf4971 --- /dev/null +++ b/docs/evidence/2026-10-15-bootstrap-recovery/legacy-secret-metadata.json @@ -0,0 +1,149 @@ +{ + "observed_at": "2026-10-15T20:50:28.605750+00:00", + "operation": "read-only Secret metadata output", + "secret_values_exported": false, + "rows": [ + { + "namespace": "sso", + "name": "authelia-secrets", + "uid": "2e6c32d4-4178-4c1e-9813-14362d330eaf", + "created_at": "2026-03-25T00:29:22Z", + "type": "Opaque" + }, + { + "namespace": "sso", + "name": "authelia-smtp", + "uid": "5d34559e-8378-4065-b10b-f1ea046b73d5", + "created_at": "2026-10-06T18:30:54Z", + "type": "Opaque" + }, + { + "namespace": "sso", + "name": "identity-provisioner-token", + "uid": "f8449bb9-0c87-46a9-a012-2c2fce9d9b6a", + "created_at": "2026-07-27T23:02:12Z", + "type": "Opaque" + }, + { + "namespace": "sso", + "name": "keycape-approval-engine-operator-client", + "uid": "599a61a7-8244-4618-8c44-6473195bbe4e", + "created_at": "2026-09-09T00:09:47Z", + "type": "Opaque" + }, + { + "namespace": "sso", + "name": "keycape-config", + "uid": "2e94519d-1550-41c7-9701-2efe47fe1fd3", + "created_at": "2026-03-25T00:36:59Z", + "type": "Opaque" + }, + { + "namespace": "sso", + "name": "keycape-factor-read", + "uid": "ee07b0d9-bccf-464a-9d1b-e38d8d577985", + "created_at": "2026-09-13T14:15:49Z", + "type": "Opaque" + }, + { + "namespace": "sso", + "name": "keycape-informed-decision-sitting-requester-client", + "uid": "361d439b-3e11-43b4-842f-cf2028acd4a4", + "created_at": "2026-09-15T18:28:30Z", + "type": "Opaque" + }, + { + "namespace": "sso", + "name": "keycape-pi-token", + "uid": "1d459ab6-8b42-454d-9220-0fa2ffe4e819", + "created_at": "2026-03-25T00:51:32Z", + "type": "Opaque" + }, + { + "namespace": "sso", + "name": "keycape-rapp-qonto-client", + "uid": "d2a06c1b-8836-4ce4-9e3a-feb3604fadae", + "created_at": "2026-07-27T18:08:12Z", + "type": "Opaque" + }, + { + "namespace": "sso", + "name": "keycape-secrets-engine-approval-client", + "uid": "eaa28bdf-04af-4e4e-a1fc-fe395c7689a7", + "created_at": "2026-09-09T00:09:47Z", + "type": "Opaque" + }, + { + "namespace": "sso", + "name": "keycape-secrets-engine-openbao-client", + "uid": "c5986225-bbcc-40c8-b55d-79c62648a46f", + "created_at": "2026-10-07T00:10:55Z", + "type": "Opaque" + }, + { + "namespace": "sso", + "name": "keycape-secrets-engine-requester-client", + "uid": "8a279143-5553-439a-a529-1fc928010062", + "created_at": "2026-09-14T00:23:36Z", + "type": "Opaque" + }, + { + "namespace": "sso", + "name": "lldap-secrets", + "uid": "c6a9e6be-5bb5-47e6-9faa-06b8d72afec3", + "created_at": "2026-03-25T00:29:16Z", + "type": "Opaque" + }, + { + "namespace": "sso", + "name": "lldap-tls", + "uid": "1ab2442f-4c57-450c-aadd-dc4b4e3eb938", + "created_at": "2026-03-25T00:44:05Z", + "type": "kubernetes.io/tls" + }, + { + "namespace": "mfa", + "name": "privacyidea-auditkeys", + "uid": "6f0b191f-4e52-4122-9df7-31f791d124f8", + "created_at": "2026-03-25T00:49:48Z", + "type": "Opaque" + }, + { + "namespace": "mfa", + "name": "privacyidea-config", + "uid": "3cd352c9-4f8e-4954-8181-d80fc9f62e0a", + "created_at": "2026-03-25T00:29:26Z", + "type": "Opaque" + }, + { + "namespace": "mfa", + "name": "privacyidea-enckey", + "uid": "b3ee83fe-cacc-4ab3-a110-baac56328e94", + "created_at": "2026-03-25T00:49:46Z", + "type": "Opaque" + }, + { + "namespace": "mfa", + "name": "privacyidea-trigger-admin", + "uid": "d1c643be-672d-4d13-8e46-3a58cb37ae39", + "created_at": "2026-03-25T00:50:53Z", + "type": "Opaque" + }, + { + "namespace": "user-engine", + "name": "identity-provisioner-client", + "uid": "3ac5147a-df73-42b5-b596-c467c0571d23", + "created_at": "2026-07-27T23:02:12Z", + "type": "Opaque" + }, + { + "namespace": "user-engine", + "name": "user-engine-delivery", + "uid": "2560f99d-621c-4f35-9472-c64ec3f6627e", + "created_at": "2026-08-13T13:02:50Z", + "type": "Opaque" + } + ], + "revocation_proven": false, + "deletion_performed": false +} diff --git a/docs/evidence/2026-10-15-bootstrap-recovery/live-workload-metadata.json b/docs/evidence/2026-10-15-bootstrap-recovery/live-workload-metadata.json new file mode 100644 index 0000000..9bdf786 --- /dev/null +++ b/docs/evidence/2026-10-15-bootstrap-recovery/live-workload-metadata.json @@ -0,0 +1,234 @@ +{ + "observed_at": "2026-10-15T20:45:27.835371+00:00", + "cluster_uid": "a553c742-0115-43d4-99a4-a5ca56fe0786", + "read_only": true, + "deployments": [ + { + "namespace": "sso", + "name": "lldap", + "uid": "743a86bd-5cb0-4ca0-9c56-a3438f4746a8", + "generation": 7, + "observed_generation": 7, + "ready_replicas": 1, + "containers": [ + { + "name": "lldap", + "image": "lldap/lldap:stable", + "secret_key_refs": [ + { + "env": "LLDAP_JWT_SECRET", + "secret": "lldap-secrets", + "key": "LLDAP_JWT_SECRET" + }, + { + "env": "LLDAP_LDAP_USER_PASS", + "secret": "lldap-secrets", + "key": "LLDAP_LDAP_USER_PASS" + } + ] + } + ], + "volume_refs": [ + { + "volume": "data", + "pvc": "lldap-data" + } + ] + }, + { + "namespace": "sso", + "name": "authelia", + "uid": "dc40a925-2306-4a3b-89c0-e714187503c3", + "generation": 28, + "observed_generation": 28, + "ready_replicas": 1, + "containers": [ + { + "name": "authelia", + "image": "docker.io/authelia/authelia@sha256:bd97cff4fcbf715b5ff1f9ae286afbe6033afce385302520b0368122d43a6f54", + "secret_key_refs": [] + } + ], + "volume_refs": [ + { + "volume": "smtp", + "secret": "authelia-smtp" + }, + { + "volume": "secrets", + "secret": "authelia-secrets" + }, + { + "volume": "data", + "pvc": "authelia-data" + } + ] + }, + { + "namespace": "sso", + "name": "keycape", + "uid": "99ddd83c-cb3f-4847-bcf8-35f1aa87627f", + "generation": 65, + "observed_generation": 65, + "ready_replicas": 1, + "containers": [ + { + "name": "keycape", + "image": "forgejo.coulomb.social/coulomb/key-cape@sha256:cdec9dd7c515f7e0a6169fc3fa560ee2cc47bf40d899755dec808c362f5ce010", + "secret_key_refs": [ + { + "env": "KEYCAPE_SECRETS_ENGINE_OPENBAO_CLIENT_SECRET", + "secret": "keycape-secrets-engine-openbao-client", + "key": "client-secret" + }, + { + "env": "KEYCAPE_RAPP_QONTO_CLIENT_SECRET", + "secret": "keycape-rapp-qonto-client", + "key": "client-secret" + }, + { + "env": "KEYCAPE_SECRETS_ENGINE_APPROVAL_CLIENT_SECRET", + "secret": "keycape-secrets-engine-approval-client", + "key": "client-secret" + }, + { + "env": "KEYCAPE_APPROVAL_ENGINE_OPERATOR_CLIENT_SECRET", + "secret": "keycape-approval-engine-operator-client", + "key": "client-secret" + }, + { + "env": "KEYCAPE_SECRETS_ENGINE_REQUESTER_CLIENT_SECRET", + "secret": "keycape-secrets-engine-requester-client", + "key": "client-secret" + }, + { + "env": "KEYCAPE_INFORMED_DECISION_SITTING_REQUESTER_CLIENT_SECRET", + "secret": "keycape-informed-decision-sitting-requester-client", + "key": "client-secret" + } + ] + } + ], + "volume_refs": [ + { + "volume": "authentication-policy", + "pvc": "keycape-authentication-policy" + }, + { + "volume": "factor-token", + "secret": "keycape-factor-read" + }, + { + "volume": "config-secret", + "secret": "keycape-config" + } + ] + }, + { + "namespace": "mfa", + "name": "privacyidea", + "uid": "58c7f96d-61cb-4dd4-bca2-54661c0ac375", + "generation": 11, + "observed_generation": 11, + "ready_replicas": 1, + "containers": [ + { + "name": "privacyidea", + "image": "ghcr.io/gpappsoft/privacyidea-docker:3.12.2", + "secret_key_refs": [] + }, + { + "name": "factor-recovery", + "image": "ghcr.io/gpappsoft/privacyidea-docker@sha256:af7841adad262f129e0c1d4f553af13f21cb2f4dc713533f316cfe43ed0b4473", + "secret_key_refs": [] + } + ], + "volume_refs": [ + { + "volume": "data", + "pvc": "privacyidea-data" + }, + { + "volume": "logs", + "pvc": "privacyidea-logs" + } + ] + }, + { + "namespace": "sso", + "name": "identity-provisioner", + "uid": "13d2fadb-6929-4fa9-b2ba-635ab8572f54", + "generation": 18, + "observed_generation": 18, + "ready_replicas": 1, + "containers": [ + { + "name": "provisioner", + "image": "forgejo.coulomb.social/coulomb/identity-provisioner@sha256:bafb48604d60f8bb98f58d27019df21d17966efcb742cfc11a6a5ab87f3f9d1f", + "secret_key_refs": [ + { + "env": "LLDAP_ADMIN_PASSWORD", + "secret": "lldap-secrets", + "key": "LLDAP_LDAP_USER_PASS" + }, + { + "env": "PROVISIONER_SERVICE_TOKEN", + "secret": "identity-provisioner-token", + "key": "token" + } + ] + } + ], + "volume_refs": [] + }, + { + "namespace": "user-engine", + "name": "user-engine", + "uid": "e1c4274a-4b6d-4ea2-b619-e09630b745b0", + "generation": 76, + "observed_generation": 76, + "ready_replicas": 1, + "containers": [ + { + "name": "portal", + "image": "forgejo.coulomb.social/coulomb/user-engine@sha256:1e6547c40c93c49f01438edcf76c20f561ccebc110220bae020723505aa56c70", + "secret_key_refs": [ + { + "env": "USER_ENGINE_DATABASE_URL", + "secret": "user-engine-pg-app", + "key": "uri" + }, + { + "env": "USER_ENGINE_PROXY_SECRET", + "secret": "user-engine-runtime", + "key": "proxy-secret" + }, + { + "env": "USER_ENGINE_EVENT_TOKEN", + "secret": "user-engine-delivery", + "key": "event-token" + }, + { + "env": "USER_ENGINE_MAIL_TOKEN", + "secret": "user-engine-scoped-mail", + "key": "token" + }, + { + "env": "USER_ENGINE_REGISTRATION_VERIFICATION_TOKEN", + "secret": "user-engine-scoped-mail", + "key": "token" + }, + { + "env": "USER_ENGINE_PROVISIONING_TOKEN", + "secret": "identity-provisioner-client", + "key": "token" + } + ] + } + ], + "volume_refs": [] + } + ], + "secret_resources_read": false, + "deployment_literal_env_values_exported": false +} diff --git a/docs/evidence/2026-10-15-bootstrap-recovery/readiness.json b/docs/evidence/2026-10-15-bootstrap-recovery/readiness.json new file mode 100644 index 0000000..fc5495b --- /dev/null +++ b/docs/evidence/2026-10-15-bootstrap-recovery/readiness.json @@ -0,0 +1,26 @@ +{ + "workplan": "NK-WP-0049", + "observed_at": "2026-10-15T20:56:58.471680+00:00", + "status": "source_prepared_pending_owner_inputs_and_native_acceptance", + "backup_destinations_operator_reported": [ + "Scaleway", + "Storage Share" + ], + "exact_offhost_artifact_confirmed": false, + "independently_held_matching_key_confirmed": false, + "recovery_key_or_backup_contents_retrieved": false, + "legacy_token_secret_present": true, + "legacy_trigger_secret_present": true, + "legacy_provider_revocation_proven": false, + "native_recovery_proven": false, + "deferred_exposure_rotation_reopened": false, + "caller_inventory": "caller-source-inventory.json", + "verification": "source-verification.json", + "ceremony": "../../bootstrap-recovery-ceremony.md", + "outstanding": [ + "Exact off-host artifact/version/digest and complete matching data/key/config coverage", + "Independent age key availability/custody confirmation", + "Owner admitted isolated recovery target/procedure and native service continuity/cleanup evidence", + "T02 current caller/provider source admission and exact legacy provider credential disposition; retain existing deferred-rotation boundary" + ] +} diff --git a/docs/evidence/2026-10-15-bootstrap-recovery/source-verification.json b/docs/evidence/2026-10-15-bootstrap-recovery/source-verification.json new file mode 100644 index 0000000..51d8fdb --- /dev/null +++ b/docs/evidence/2026-10-15-bootstrap-recovery/source-verification.json @@ -0,0 +1,23 @@ +{ + "broad": { + "command": "python3 -m pytest -q tests tools/security-bootstrap-console/tests sso-mfa/k8s/keycape sso-mfa/k8s/lldap/test_identity_provisioner_reconcile.py", + "passed": 723, + "skipped": 65, + "warnings": 2, + "duration_seconds": 171.05, + "scope_note": "Skipped checks are not acceptance evidence; native/estate recovery is not established." + }, + "final_focused": { + "command": "python3 -m pytest -q tests/test_bootstrap_secret_output.py tests/test_custody_entrypoints.py tests/test_retirement_and_bundle_boundaries.py tests/test_openbao_lab_boundary.py tests/test_bootstrap_recovery_transport.py", + "passed": 57, + "duration_seconds": 7.45 + }, + "reproduced_baseline_failures": 6, + "syntax_checks": { + "modified_shell_files": 13, + "modified_python_files": 4, + "passed": true + }, + "production_mutations_performed": false, + "production_recovery_claimed": false +} diff --git a/docs/openbao-unseal-custody-models.md b/docs/openbao-unseal-custody-models.md index 771cf6d..e829926 100644 --- a/docs/openbao-unseal-custody-models.md +++ b/docs/openbao-unseal-custody-models.md @@ -32,8 +32,13 @@ during bootstrap and rebuild. uninitialized (init JSON written straight into the age-custody secrets dir), replays unseal shares stdin-to-stdin, verifies post-unseal state, and emits non-secret `openbao_initialized` / `openbao_post_unseal_verified` evidence. - Set `OPENBAO_RUN_CONFIGURE_INITIAL=1` to chain `railiance-platform: make - openbao-configure-initial`. + The current adapter additionally requires an explicit `lab` profile, an + owner-admitted `KUBECONFIG`, `OPENBAO_EXPECTED_CLUSTER_UID`, and owned private + custody paths. Existing init files are never replaced; diagnostics and shares + remain contained. `--dry-run` is a non-mutating plan without cluster reads. + Platform configuration is a separate owner operation; + `OPENBAO_RUN_CONFIGURE_INITIAL=1` is refused. The July greenfield receipt is + historical, not execution proof for this October revision. - Enables **unattended rebuild test cycles** on a 3-node slate. - **Not** production trust posture — use to prove S1→S3→SSH engine automation, then graduate to stronger models. diff --git a/sso-mfa/bootstrap/README.md b/sso-mfa/bootstrap/README.md index c7d3ccb..1e19e0a 100644 --- a/sso-mfa/bootstrap/README.md +++ b/sso-mfa/bootstrap/README.md @@ -11,10 +11,12 @@ KeePassXC is the pre-cluster source of truth. After the K3s cluster is running, secrets migrate into HashiCorp Vault (T01 Phase 0b) and KeePassXC becomes the break-glass / dev-local backup. -## KeePassXC database structure +## Historical KeePassXC layout -Create a new `.kdbx` database. Use a strong master password stored in your -personal password manager (Bitwarden, 1Password, etc.). +This early layout is historical; Keycloak is backlog, not a current runtime +requirement. Use existing independently held custody and the current asset +manifest. Do not generate replacement credentials or enroll a new recovery key +merely to make this layout match a failed recovery. Recommended group structure: @@ -49,17 +51,18 @@ chmod +x gen-secrets.sh ./gen-secrets.sh ./secrets ``` -This writes `.env` files to `./secrets/` (gitignored). Inspect each file, -then copy each value into the appropriate KeePassXC entry. +This writes `.env` files to `./secrets/` (gitignored). Handle them only through +the private custody procedure; never print them or copy values into chat. -### Step 2 — Create the age encryption key (one-time) +### Step 2 — Confirm independent custody and the registered recipient -```bash -age-keygen -o ~/net-kingdom-ops-bundle.key -``` - -The public key prints to stdout. The private key is in the `.key` file. -Store the `.key` file somewhere safe (NOT in this repo; NOT in the secrets/ dir). +Use the existing independently registered public recipient in `keys/age.pub`. +Encryption needs only that public file. Do not generate, discover or install a +permanent private key as part of bootstrap. The matching private key stays in +its independently held password-safe/offline custody; temporary attended unlock +uses the [custody procedure](../../docs/security-bootstrap-age-custody.md). +If it is unavailable, stop and resolve custody through the owner rather than +create a new key that cannot decrypt existing backups. ### Step 3 — Create the encrypted ops bundle @@ -69,6 +72,10 @@ chmod +x pack-bundle.sh ``` Store `ops-bundle.tar.age` offsite (cloud storage, external drive, separate location). +The destination must be new. Inputs must be owned private regular files, without +links, special files or age private keys. There is no plaintext tar staging file. +Independently verify recovery and service-file coverage before source cleanup; +bundle creation alone proves neither current credential validity nor recovery. ### Step 4 — Shred the generated files @@ -77,25 +84,15 @@ find ./secrets -type f -exec shred -u {} \; rm -rf ./secrets ``` -### Step 5 — PI_ENCFILE (after privacyIDEA container is running — T04) +### Step 5 — privacyIDEA encryption and audit keys -```bash -# Generate the encryption key inside the running container: -kubectl exec -n mfa -- pi-manage create_enckey - -# Extract it: -kubectl cp -n mfa :/etc/privacyidea/enckey ./pi.enc - -# Store as a binary attachment in KeePassXC → net-kingdom/privacyIDEA/PI_ENCFILE - -# Create the K8s Secret: -kubectl create secret generic privacyidea-enckey \ - --from-file=PI_ENCFILE=./pi.enc \ - --namespace mfa - -# Shred the local copy: -shred -u ./pi.enc -``` +Follow the current [key recovery procedure](../k8s/privacyidea/KEY-RECOVERY.md). +Export the verified existing PVC encryption key and matching audit pair through +`enckey-bootstrap.sh` into owned private custody. Missing recovery keys are not +regenerated. Creating an audit pair is an explicit first-bootstrap-only action, +not recovery. The old unconditional `pi-manage create_enckey`, raw `kubectl cp` +and `PI_ENCFILE` Secret recipes are withdrawn; they could destroy continuity or +misrepresent the application's actual PVC key path. ## Notes on secret reuse @@ -110,11 +107,10 @@ Use KeePassXC references (`{REF:P@T:UUID}`) to avoid maintaining two copies. ## Phase 0b — HashiCorp Vault (after T02, once K3s is running) -See `../vault/` (created in T01 Phase 0b) for: -- Vault Helm chart values -- ESO (External Secrets Operator) configuration -- Vault secret path layout -- Migration procedure: KeePassXC → Vault +Current OpenBao configuration, ESO delivery and custody belong to +`railiance-platform`; use its exact reviewed procedure. Legacy bootstrap +references do not authorize live initialization, a new root, or replaying old +service values into existing workloads. ## Current bootstrap and recovery handoff diff --git a/sso-mfa/bootstrap/creds-bootstrap-agent.sh b/sso-mfa/bootstrap/creds-bootstrap-agent.sh index ba52d90..a5086f1 100755 --- a/sso-mfa/bootstrap/creds-bootstrap-agent.sh +++ b/sso-mfa/bootstrap/creds-bootstrap-agent.sh @@ -129,12 +129,8 @@ step "1 — Generate secrets" if [[ "$(state_get secrets_generated)" == "true" ]]; then ok "secrets already generated — skipping" else - # Clean up any partial generation from a failed prior run - if [[ -d "$SECRETS_DIR" ]]; then - warn "leftover secrets/ found from previous run — removing" - find "$SECRETS_DIR" -type f -exec shred -u {} \; 2>/dev/null || true - rm -rf "$SECRETS_DIR" - fi + # Existing custody may be the only rollback copy after a partial run. + [[ ! -e "$SECRETS_DIR" && ! -L "$SECRETS_DIR" ]] || die "existing service custody requires owner reconciliation; never remove it to regenerate" log "running gen-secrets.sh..." if [[ "$DRY_RUN" == false ]]; then @@ -310,7 +306,7 @@ step "7 — Final verification (all components)" if [[ "$DRY_RUN" == false ]]; then (cd "$SCRIPT_DIR" && bash creds-verify.sh) - ok "all secrets verified" + ok "expected Secret names present; native service and recovery acceptance remain separate" else echo " [dry-run] would run: bash creds-verify.sh" fi @@ -344,8 +340,7 @@ if kubectl get namespace openbao &>/dev/null; then "chore(creds): encrypted OpenBao init material [agent]" fi else - warn "OpenBao init/unseal did not complete — see output" - warn "(gate unselected or pod not ready; bootstrap continues)" + die "OpenBao init/unseal not verified; retain all private custody and reconcile through the owner before cleanup or retry" fi else echo " [dry-run] would run: openbao-init-unseal.sh $SECRETS_DIR" @@ -358,7 +353,7 @@ fi step "8 — Create ops bundle (age-encrypted snapshot)" -BUNDLE_PATH=$(mktemp "$REPO_ROOT/ops-bundle-$(date +%Y%m%dT%H%M%S)-XXXXXXXX.tar.age") +BUNDLE_PATH="$REPO_ROOT/ops-bundle-$(date +%Y%m%dT%H%M%S)-$(python3 -c 'import uuid; print(uuid.uuid4().hex)').tar.age" # Always snapshot the current service files after apply. An old phase boolean # must not cause a newly generated/rotated file to be omitted from recovery. diff --git a/sso-mfa/bootstrap/creds-verify.sh b/sso-mfa/bootstrap/creds-verify.sh index c061a5c..dfa8618 100755 --- a/sso-mfa/bootstrap/creds-verify.sh +++ b/sso-mfa/bootstrap/creds-verify.sh @@ -1,5 +1,5 @@ #!/usr/bin/env bash -# creds-verify.sh — check all expected K8s secrets exist and update creds-state.yaml. +# creds-verify.sh — observe Secret presence only; no credential/native acceptance. # # Usage: # bash sso-mfa/bootstrap/creds-verify.sh @@ -12,8 +12,9 @@ # mfa/privacyidea-config → secrets_applied.privacyidea # sso/keycape-config → secrets_applied.keycape # mfa/privacyidea-enckey → enckey_bootstrapped -# sso/keycape-pi-token → pi_admin_created +# sso/keycape-factor-read → current owner-managed factor delivery +set +x set -euo pipefail SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" @@ -30,19 +31,10 @@ secret_exists() { kubectl get secret "$name" --namespace="$ns" --ignore-not-found -o name 2>/dev/null | grep -q . } -update_state_top() { - local key="$1" value="$2" - if [[ -f "$STATE_FILE" ]]; then - sed -i "s|^$key: .*|$key: $value|" "$STATE_FILE" - fi -} - -update_state_nested() { - local key="$1" value="$2" - if [[ -f "$STATE_FILE" ]]; then - sed -i "s|^ $key: .*| $key: $value|" "$STATE_FILE" - fi -} +# Presence cannot establish application, issuance, effective MFA or recovery. +# Historical applied/verification state is not rewritten by this diagnostic. +update_state_top() { :; } +update_state_nested() { :; } # ── Results table ───────────────────────────────────────────────────────────── pass=0 @@ -90,15 +82,15 @@ check "enckey (privacyidea-enckey)" \ mfa privacyidea-enckey \ update_state_top enckey_bootstrapped -check "pi-admin token (keycape-pi-token)" \ - sso keycape-pi-token \ - update_state_nested keycape +check "scoped factor reader (keycape-factor-read)" \ + sso keycape-factor-read \ + update_state_top factor_delivery_present echo "" echo "Results: $pass present, $fail missing" if [[ -f "$STATE_FILE" ]]; then - echo "State file updated: $STATE_FILE" + echo "Historical state preserved; presence is not native consumer or recovery acceptance." fi [[ "$fail" -eq 0 ]] diff --git a/sso-mfa/bootstrap/decrypt-secrets.sh b/sso-mfa/bootstrap/decrypt-secrets.sh index a5b8ced..8e0cb74 100755 --- a/sso-mfa/bootstrap/decrypt-secrets.sh +++ b/sso-mfa/bootstrap/decrypt-secrets.sh @@ -10,6 +10,23 @@ OUTPUT_DIR="${1:-./secrets}" AGE_KEY="${2:-${NETKINGDOM_CUSTODY_AGE_KEY_FILE:-}}" [[ $# -le 2 ]] || fail "unsupported decrypt argument" [[ -n "$AGE_KEY" && -f "$AGE_KEY" ]] || fail "explicit temporary custody key required" +# Validate an owned, private, non-linked age identity before passing it to age. +# No HOME/plugin/SSH identity discovery is admitted by this recovery path. +python3 - "$AGE_KEY" 2>/dev/null <<'PYKEY' || fail "custody key must be an owned private regular age identity" +import os, pathlib, stat, sys +p = pathlib.Path(os.path.abspath(sys.argv[1])) +if any(x.is_symlink() for x in (p, *p.parents)): raise ValueError() +fd = os.open(p, os.O_RDONLY | os.O_NOFOLLOW | os.O_NONBLOCK) +try: + st = os.fstat(fd) + if not stat.S_ISREG(st.st_mode) or st.st_uid != os.getuid() or st.st_mode & 0o077 or st.st_nlink != 1 or st.st_size > 16384: + raise ValueError() + data = os.read(fd, 16385).decode('ascii') + values = [x for x in data.splitlines() if x and not x.startswith('#')] + if len(values) != 1 or not values[0].startswith('AGE-SECRET-KEY-1'): raise ValueError() +finally: os.close(fd) +PYKEY + ENC_DIR="$(dirname "$OUTPUT_DIR")/secrets.enc" [[ -d "$ENC_DIR" && ! -L "$ENC_DIR" ]] || fail "encrypted source directory unavailable" [[ ! -e "$OUTPUT_DIR" && ! -L "$OUTPUT_DIR" ]] || fail "output already exists" diff --git a/sso-mfa/bootstrap/gen-secrets.sh b/sso-mfa/bootstrap/gen-secrets.sh index 8a6191e..c9e45a0 100755 --- a/sso-mfa/bootstrap/gen-secrets.sh +++ b/sso-mfa/bootstrap/gen-secrets.sh @@ -23,8 +23,8 @@ umask 077 OUT_DIR="${1:-./secrets}" -if [[ -e "$OUT_DIR" ]]; then - echo "ERROR: $OUT_DIR already exists. Delete it first or choose a different path." >&2 +if [[ -e "$OUT_DIR" || -L "$OUT_DIR" ]]; then + echo "ERROR: $OUT_DIR already exists. Preserve it for reconciliation or choose a new path." >&2 exit 1 fi diff --git a/sso-mfa/bootstrap/openbao-init-unseal.sh b/sso-mfa/bootstrap/openbao-init-unseal.sh index c245e44..3380966 100755 --- a/sso-mfa/bootstrap/openbao-init-unseal.sh +++ b/sso-mfa/bootstrap/openbao-init-unseal.sh @@ -1,204 +1,7 @@ #!/usr/bin/env bash -# openbao-init-unseal.sh — SOPS-held OpenBao init/unseal automation (NET-WP-0020 T2) -# -# Usage: -# bash sso-mfa/bootstrap/openbao-init-unseal.sh [--dry-run] [SECRETS_DIR] -# -# Implements the `sops-held-automation` unseal custody model from -# docs/openbao-unseal-custody-models.md: -# - refuses to run unless the security bootstrap console has selected -# `sops-held-automation` (attended-ceremony/auto-unseal-transit are never -# automated here); -# - if OpenBao is uninitialized: runs `bao operator init` inside the pod and -# writes the init material to SECRETS_DIR/openbao/ (age/SOPS custody via -# encrypt-secrets.sh — never Git plaintext, never stdout); -# - if OpenBao is sealed: replays the SOPS-held unseal shares via stdin until -# the threshold is met; -# - verifies post-unseal state and emits a NON-SECRET JSON evidence line -# (flags only: openbao_initialized, openbao_post_unseal_verified). -# -# Post-unseal platform configuration stays owned by railiance-platform: -# make -C ../railiance-platform openbao-configure-initial -# It is invoked automatically only when OPENBAO_RUN_CONFIGURE_INITIAL=1. -# -# Environment: -# OPENBAO_NAMESPACE k8s namespace (default: openbao) -# OPENBAO_RELEASE helm release / pod prefix (default: openbao) -# OPENBAO_KEY_SHARES init key shares (default: 3) -# OPENBAO_KEY_THRESHOLD init key threshold (default: 2) -# OPENBAO_CONSOLE_METADATA console metadata file (default: /.local/security-bootstrap.json) -# OPENBAO_RUN_CONFIGURE_INITIAL 1 = run platform configure-initial after unseal -# RAILIANCE_PLATFORM_DIR path to railiance-platform checkout -# (default: sibling of this repo) - +# Lab-only adapter. Explicit lab profile, custody model, kubeconfig and +# OPENBAO_EXPECTED_CLUSTER_UID are required; never use this on production. +set +x set -euo pipefail - -SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" -REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" - -DRY_RUN=false -SECRETS_DIR="$SCRIPT_DIR/secrets" -for arg in "$@"; do - case "$arg" in - --dry-run) DRY_RUN=true ;; - *) SECRETS_DIR="$arg" ;; - esac -done - -NAMESPACE="${OPENBAO_NAMESPACE:-openbao}" -RELEASE="${OPENBAO_RELEASE:-openbao}" -POD="${RELEASE}-0" -KEY_SHARES="${OPENBAO_KEY_SHARES:-3}" -KEY_THRESHOLD="${OPENBAO_KEY_THRESHOLD:-2}" -CONSOLE_METADATA="${OPENBAO_CONSOLE_METADATA:-$REPO_ROOT/.local/security-bootstrap.json}" -RAILIANCE_PLATFORM_DIR="${RAILIANCE_PLATFORM_DIR:-$(dirname "$REPO_ROOT")/railiance-platform}" -OPENBAO_SECRETS="$SECRETS_DIR/openbao" -INIT_FILE="$OPENBAO_SECRETS/init.json" - -log() { echo " [openbao] $*"; } -die() { echo " [openbao] ERROR: $*" >&2; exit 1; } - -evidence() { - # NON-SECRET evidence only: booleans, counts, model, namespace. - python3 - "$@" <<'PY' -import json, sys -keys = [a.split("=", 1) for a in sys.argv[1:]] -def coerce(v): - if v in ("true", "false"): - return v == "true" - try: - return int(v) - except ValueError: - return v -print("EVIDENCE " + json.dumps({k: coerce(v) for k, v in keys}, sort_keys=True)) -PY -} - -# ── Custody model gate ──────────────────────────────────────────────────────── -# Only sops-held-automation may be automated. attended-ceremony and -# auto-unseal-transit must never reach `bao operator init/unseal` from here. -MODEL="" -PROFILE="" -if [[ -f "$CONSOLE_METADATA" ]]; then - MODEL=$(python3 -c 'import json,sys; print(json.load(open(sys.argv[1])).get("openbao_unseal_custody_model") or "")' "$CONSOLE_METADATA") - PROFILE=$(python3 -c 'import json,sys; print(json.load(open(sys.argv[1])).get("deployment_profile") or "")' "$CONSOLE_METADATA") -fi -if [[ "$PROFILE" == "production" ]]; then - die "deployment profile is 'production' — sops-held automation is blocked. - Use the attended ceremony (docs/openbao-attended-ceremony-runbook.md) or - auto-unseal-transit, or reselect: select-deployment-profile --profile lab" -fi -if [[ "$MODEL" != "sops-held-automation" ]]; then - die "unseal custody model is '${MODEL:-unselected}' — automation requires 'sops-held-automation'. - Select it first (see docs/openbao-unseal-custody-models.md): - python3 tools/security-bootstrap-console/security_bootstrap_console.py \\ - select-openbao-unseal-custody-model --model sops-held-automation \\ - --metadata $CONSOLE_METADATA" -fi -log "custody model gate passed: $MODEL" - -command -v kubectl >/dev/null 2>&1 || die "kubectl not found" -command -v python3 >/dev/null 2>&1 || die "python3 not found" - -kubectl get namespace "$NAMESPACE" >/dev/null 2>&1 \ - || die "namespace '$NAMESPACE' not found — deploy OpenBao first (railiance-platform: make openbao-deploy)" -kubectl -n "$NAMESPACE" get pod "$POD" >/dev/null 2>&1 \ - || die "pod '$POD' not found in namespace '$NAMESPACE'" - -# ── Status probe ────────────────────────────────────────────────────────────── -# `bao status` exits 2 when sealed — capture output regardless of exit code. -bao_status() { - kubectl -n "$NAMESPACE" exec "$POD" -- bao status -format=json 2>/dev/null || true -} - -STATUS_JSON="$(bao_status)" -[[ -n "$STATUS_JSON" ]] || die "could not read bao status from $NAMESPACE/$POD" -INITIALIZED=$(python3 -c 'import json,sys; print(str(json.loads(sys.stdin.read()).get("initialized", False)).lower())' <<<"$STATUS_JSON") -SEALED=$(python3 -c 'import json,sys; print(str(json.loads(sys.stdin.read()).get("sealed", True)).lower())' <<<"$STATUS_JSON") -log "status: initialized=$INITIALIZED sealed=$SEALED" - -# ── Init (only when uninitialized) ──────────────────────────────────────────── -DID_INIT=false -if [[ "$INITIALIZED" != "true" ]]; then - if [[ "$DRY_RUN" == true ]]; then - log "[dry-run] would run: bao operator init -key-shares=$KEY_SHARES -key-threshold=$KEY_THRESHOLD" - else - log "initializing OpenBao ($KEY_SHARES shares, threshold $KEY_THRESHOLD)..." - mkdir -p "$OPENBAO_SECRETS" - chmod 700 "$OPENBAO_SECRETS" - umask 077 - kubectl -n "$NAMESPACE" exec "$POD" -- \ - bao operator init -key-shares="$KEY_SHARES" -key-threshold="$KEY_THRESHOLD" -format=json \ - > "$INIT_FILE" - chmod 600 "$INIT_FILE" - DID_INIT=true - log "init material written to $INIT_FILE (plaintext — encrypt + shred via encrypt-secrets.sh)" - fi - STATUS_JSON="$(bao_status)" - SEALED=$(python3 -c 'import json,sys; print(str(json.loads(sys.stdin.read()).get("sealed", True)).lower())' <<<"$STATUS_JSON") -fi - -# ── Unseal (when sealed) ────────────────────────────────────────────────────── -if [[ "$SEALED" == "true" ]]; then - if [[ ! -f "$INIT_FILE" ]]; then - die "OpenBao is sealed but $INIT_FILE is missing. - Decrypt SOPS-held custody first: bash decrypt-secrets.sh $SECRETS_DIR" - fi - if [[ "$DRY_RUN" == true ]]; then - log "[dry-run] would replay $KEY_THRESHOLD unseal share(s) from $INIT_FILE via stdin" - else - SHARE_COUNT=$(python3 -c 'import json,sys; print(len(json.load(open(sys.argv[1])).get("unseal_keys_b64") or []))' "$INIT_FILE") - [[ "$SHARE_COUNT" -gt 0 ]] || die "no unseal shares found in $INIT_FILE" - log "replaying unseal shares (have $SHARE_COUNT)..." - for idx in $(seq 0 $((SHARE_COUNT - 1))); do - # Share travels stdin→stdin; never argv, never logs. - # `bao operator unseal -` does not read stdin (needs a TTY or the - # share in argv), so use the sys/unseal API with key=- instead. - python3 -c 'import json,sys; print(json.load(open(sys.argv[1]))["unseal_keys_b64"][int(sys.argv[2])])' "$INIT_FILE" "$idx" \ - | kubectl -n "$NAMESPACE" exec -i "$POD" -- bao write sys/unseal key=- >/dev/null - STATUS_JSON="$(bao_status)" - SEALED=$(python3 -c 'import json,sys; print(str(json.loads(sys.stdin.read()).get("sealed", True)).lower())' <<<"$STATUS_JSON") - log "unseal share $((idx + 1)) applied (sealed=$SEALED)" - [[ "$SEALED" == "false" ]] && break - done - fi -fi - -# ── Post-unseal verification ────────────────────────────────────────────────── -VERIFIED=false -if [[ "$DRY_RUN" == true ]]; then - log "[dry-run] would verify post-unseal status" -else - STATUS_JSON="$(bao_status)" - INITIALIZED=$(python3 -c 'import json,sys; print(str(json.loads(sys.stdin.read()).get("initialized", False)).lower())' <<<"$STATUS_JSON") - SEALED=$(python3 -c 'import json,sys; print(str(json.loads(sys.stdin.read()).get("sealed", True)).lower())' <<<"$STATUS_JSON") - if [[ "$INITIALIZED" == "true" && "$SEALED" == "false" ]]; then - VERIFIED=true - log "post-unseal verified: initialized, unsealed" - else - die "post-unseal verification failed: initialized=$INITIALIZED sealed=$SEALED" - fi -fi - -evidence \ - "custody_model=$MODEL" \ - "namespace=$NAMESPACE" \ - "pod=$POD" \ - "openbao_initialized=$([[ "$INITIALIZED" == "true" || "$DID_INIT" == "true" ]] && echo true || echo false)" \ - "openbao_did_init_this_run=$DID_INIT" \ - "openbao_post_unseal_verified=$VERIFIED" \ - "dry_run=$DRY_RUN" - -# ── Platform handoff (railiance-platform owns post-unseal configuration) ───── -if [[ "${OPENBAO_RUN_CONFIGURE_INITIAL:-0}" == "1" && "$DRY_RUN" == false ]]; then - [[ -d "$RAILIANCE_PLATFORM_DIR" ]] || die "railiance-platform not found at $RAILIANCE_PLATFORM_DIR" - log "running railiance-platform post-unseal configuration..." - make -C "$RAILIANCE_PLATFORM_DIR" openbao-configure-initial -else - log "next (railiance-platform): make -C $RAILIANCE_PLATFORM_DIR openbao-configure-initial" -fi - -if [[ "$DID_INIT" == true ]]; then - log "REMINDER: encrypt + shred the new init material now:" - log " bash $SCRIPT_DIR/encrypt-secrets.sh $SECRETS_DIR && commit secrets.enc/" -fi +umask 077 +exec python3 "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/openbao-lab.py" "$@" diff --git a/sso-mfa/bootstrap/openbao-lab.py b/sso-mfa/bootstrap/openbao-lab.py new file mode 100644 index 0000000..32d4183 --- /dev/null +++ b/sso-mfa/bootstrap/openbao-lab.py @@ -0,0 +1,148 @@ +#!/usr/bin/env python3 +"""Contained lab-only init/unseal. Production uses the Platform custody ceremony. + +The local model selection is a scope check, not an authorization grant. Caller +must supply an owner-admitted lab kubeconfig and independently observed cluster +UID. Private init output is retained for reconciliation after any unknown write. +""" +import importlib.util +import json +import os +from pathlib import Path +import re +import subprocess +import sys + +spec = importlib.util.spec_from_file_location('private_service_files', Path(__file__).with_name('apply-service-secrets.py')) +private = importlib.util.module_from_spec(spec) +spec.loader.exec_module(private) + + +class Refused(Exception): + pass + + +def run(arguments, payload=None, allowed=(0,)): + # Inherit only routing/executable inputs, never ambient bearer credentials or + # shell hooks. Diagnostics and command output remain inside this process. + env = {k: os.environ[k] for k in ('PATH', 'HOME', 'KUBECONFIG') if k in os.environ} + result = subprocess.run(['kubectl', *arguments], input=payload, stdout=subprocess.PIPE, + stderr=subprocess.DEVNULL, timeout=60, env=env) + if result.returncode not in allowed or len(result.stdout) > 1024 * 1024: + raise Refused() + return result.stdout + + +def status(namespace, pod): + data = json.loads(run(['-n', namespace, 'exec', pod, '--', 'bao', 'status', '-format=json'], allowed=(0, 2))) + if not isinstance(data, dict) or type(data.get('initialized')) is not bool or type(data.get('sealed')) is not bool: + raise Refused() + return data + + +def init_material(raw): + value = json.loads(raw) + shares = value.get('unseal_keys_b64') + if (not isinstance(shares, list) or not shares or len(shares) > 16 + or any(not isinstance(x, str) or not re.fullmatch(r'[A-Za-z0-9+/=]{16,256}', x) for x in shares) + or len(set(shares)) != len(shares) + or not isinstance(value.get('root_token'), str) or not value['root_token']): + raise Refused() + return shares + + +def main(args): + if args == ['--dry-run']: + print('Plan only: validate explicit lab custody and cluster UID; retain private init material; unseal and verify. No files or cluster state inspected.') + return + if len(args) > 1 or (args and args[0].startswith('-')): + raise Refused() + root = Path(__file__).resolve().parents[2] + metadata = Path(os.environ.get('OPENBAO_CONSOLE_METADATA', str(root / '.local/security-bootstrap.json'))) + model = json.loads(metadata.read_bytes()) + if model.get('deployment_profile') != 'lab' or model.get('openbao_unseal_custody_model') != 'sops-held-automation': + raise Refused() + expected = os.environ.get('OPENBAO_EXPECTED_CLUSTER_UID', '') + if not re.fullmatch(r'[a-zA-Z0-9-]{1,128}', expected) or not os.environ.get('KUBECONFIG'): + raise Refused() + if os.environ.get('OPENBAO_RUN_CONFIGURE_INITIAL', '0') != '0': + raise Refused() # Platform configuration is a separate owner operation. + namespace = os.environ.get('OPENBAO_NAMESPACE', 'openbao') + release = os.environ.get('OPENBAO_RELEASE', 'openbao') + if any(not re.fullmatch(r'[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?', x) for x in (namespace, release)): + raise Refused() + shares = int(os.environ.get('OPENBAO_KEY_SHARES', '3')) + threshold = int(os.environ.get('OPENBAO_KEY_THRESHOLD', '2')) + if not 2 <= threshold <= shares <= 16: + raise Refused() + # Pin the actual target before credential input or any mutation. + obj = json.loads(run(['get', 'namespace', 'kube-system', '-o', 'json'])) + if obj.get('metadata', {}).get('uid') != expected: + raise Refused() + pod = release + '-0' + target = json.loads(run(['-n', namespace, 'get', 'pod', pod, '-o', 'json'])) + pod_uid = target.get('metadata', {}).get('uid') + if (not isinstance(pod_uid, str) or not pod_uid or target.get('metadata', {}).get('namespace') != namespace + or target.get('metadata', {}).get('name') != pod): + raise Refused() + directory = args[0] if args else str(Path(__file__).parent / 'secrets') + did_init = False + with private.custody(directory, ()) as _: + parent = Path(directory) / 'openbao' + try: + parent.mkdir(mode=0o700) + except FileExistsError: + pass + with private.custody(directory, ('openbao',)) as fds: + fd = fds['openbao'] + before = status(namespace, pod) + if not before['initialized']: + # Refuse prior custody material before sending init. An exclusive + # owner window is necessary; Kubernetes exec cannot CAS pod UID. + try: + os.stat('init.json', dir_fd=fd, follow_symlinks=False) + except FileNotFoundError: + pass + else: + raise Refused() + pinned = json.loads(run(['-n', namespace, 'get', 'pod', pod, '-o', 'json'])) + if pinned.get('metadata', {}).get('uid') != pod_uid: + raise Refused() + raw = run(['-n', namespace, 'exec', pod, '--', 'bao', 'operator', 'init', + '-key-shares=' + str(shares), '-key-threshold=' + str(threshold), '-format=json']) + # Preserve even a malformed successful response privately: the + # remote initialization cannot be undone or blindly retried. + output = os.open('init.json', os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW, + 0o600, dir_fd=fd) + with os.fdopen(output, 'wb') as dest: + dest.write(raw); dest.flush(); os.fsync(dest.fileno()) + os.fsync(fd) + init_material(raw) + did_init = True + observed = status(namespace, pod) + if observed['sealed']: + material = init_material(private.read_private(fd, 'init.json')) + for share in material: + pinned = json.loads(run(['-n', namespace, 'get', 'pod', pod, '-o', 'json'])) + if pinned.get('metadata', {}).get('uid') != pod_uid: + raise Refused() + run(['-n', namespace, 'exec', '-i', pod, '--', 'bao', 'write', 'sys/unseal', 'key=-'], + (share + '\n').encode()) + observed = status(namespace, pod) + if not observed['sealed']: + break + if not observed['initialized'] or observed['sealed']: + raise Refused() + print('EVIDENCE ' + json.dumps({'custody_model': 'sops-held-automation', 'namespace': namespace, + 'pod': pod, 'cluster_uid': expected, 'pod_uid': pod_uid, 'openbao_initialized': True, + 'openbao_did_init_this_run': did_init, 'openbao_post_unseal_verified': True, + 'recovery_custody_verified': False, 'dry_run': False}, sort_keys=True)) + print('Retain init material in protected custody; encrypt and independently verify recovery. Platform configuration remains a separate owner operation.') + + +if __name__ == '__main__': + try: + main(sys.argv[1:]) + except (Refused, private.Refused, OSError, ValueError, KeyError, TypeError, subprocess.SubprocessError): + print('ERROR: lab init/unseal refused or failed; diagnostics suppressed. Remote outcome may be unknown. Preserve private init material and reconcile through the owner before retry.', file=sys.stderr) + raise SystemExit(1) diff --git a/sso-mfa/bootstrap/pack-bundle.sh b/sso-mfa/bootstrap/pack-bundle.sh index e935555..ab39929 100755 --- a/sso-mfa/bootstrap/pack-bundle.sh +++ b/sso-mfa/bootstrap/pack-bundle.sh @@ -1,51 +1,61 @@ #!/usr/bin/env bash -# pack-bundle.sh — create an age-encrypted ops bundle from the secrets directory -# -# Usage: -# ./pack-bundle.sh SECRETS_DIR RECIPIENT_PUBKEY [OUTPUT_FILE] -# -# RECIPIENT_PUBKEY is the age public key to encrypt to. -# Generate one with: age-keygen -o ops-bundle.key (store key safely) -# The public key appears on stdout and in the .key file header. -# -# OUTPUT_FILE defaults to ops-bundle-.tar.age in the current directory. -# -# Example: -# age-keygen -o ~/ops-bundle.key -# ./pack-bundle.sh ./secrets "age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p" - +# Encrypt a private service-file snapshot without a plaintext tar intermediate. +# Usage: pack-bundle.sh SECRETS_DIR PUBLIC_AGE_RECIPIENT [NEW_OUTPUT_FILE] +# Use the independently registered recipient; private age keys never belong in +# the payload. Run under owner-exclusive custody maintenance (no same-UID writers). +set +x set -euo pipefail - -if [[ $# -lt 2 ]]; then - echo "Usage: $0 SECRETS_DIR RECIPIENT_PUBKEY [OUTPUT_FILE]" >&2 - exit 1 -fi - +umask 077 +fail() { printf '%s\n' 'ERROR: bundle refused or failed; no new recovery artifact published.' >&2; exit 1; } +[[ $# -ge 2 && $# -le 3 ]] || fail SECRETS_DIR="${1%/}" RECIPIENT="$2" -TIMESTAMP="$(date +%Y%m%dT%H%M%S)" -OUTPUT="${3:-ops-bundle-${TIMESTAMP}.tar.age}" - -if [[ ! -d "$SECRETS_DIR" ]]; then - echo "ERROR: $SECRETS_DIR is not a directory." >&2 - exit 1 -fi - -if ! command -v age &>/dev/null; then - echo "ERROR: age is not installed. Install with: apt install age" >&2 - exit 1 -fi - -TMPTAR="$(mktemp --suffix=.tar)" -trap 'shred -u "$TMPTAR" 2>/dev/null || rm -f "$TMPTAR"' EXIT - -echo "Packing: $SECRETS_DIR → (tar) → (age encrypt) → $OUTPUT" -tar -C "$(dirname "$SECRETS_DIR")" -cf "$TMPTAR" "$(basename "$SECRETS_DIR")" -age -r "$RECIPIENT" -o "$OUTPUT" "$TMPTAR" - -echo "Bundle written: $OUTPUT" -echo "Recipient key : $RECIPIENT" -echo "" -echo "Store $OUTPUT offsite (cloud, external drive, second location)." -echo "Decrypt with:" -echo " age -d -i -o secrets.tar $OUTPUT && tar xf secrets.tar" +OUTPUT="${3:-ops-bundle-$(date +%Y%m%dT%H%M%S)-$RANDOM.tar.age}" +[[ "$RECIPIENT" == age1* ]] || fail +printf '' | age -r "$RECIPIENT" >/dev/null 2>&1 || fail +[[ ! -e "$OUTPUT" && ! -L "$OUTPUT" ]] || fail +PARENT="$(dirname -- "$OUTPUT")" +[[ -d "$PARENT" ]] || fail +STAGING=$(mktemp -d "$PARENT/.netkingdom-bundle-XXXXXXXX") || fail +trap 'rm -rf -- "$STAGING"' EXIT +trap 'exit 129' HUP +trap 'exit 130' INT +trap 'exit 143' TERM +# Emit only member names to a private local list. Never follow links or include +# special/hardlinked files, permissive custody inputs or age private material. +if ! python3 - "$SECRETS_DIR" "$OUTPUT" > "$STAGING/members" 2>/dev/null <<'PY' +import os, pathlib, stat, sys +root = pathlib.Path(os.path.abspath(sys.argv[1])) +output = pathlib.Path(os.path.abspath(sys.argv[2])) +for path in (root, *root.parents, output.parent, *output.parent.parents): + if path.is_symlink(): raise ValueError() +if root == output or root in output.parents: raise ValueError() +count = 0 +for path in [root, *sorted(root.rglob('*'))]: + info = path.lstat() + if info.st_uid != os.getuid() or info.st_mode & 0o077: raise ValueError() + if stat.S_ISDIR(info.st_mode): continue + if not stat.S_ISREG(info.st_mode) or info.st_nlink != 1 or info.st_size > 32 * 1024 * 1024: + raise ValueError() + with path.open('rb') as source: + if b'AGE-SECRET-KEY-' in source.read(): raise ValueError() + name = str(path.relative_to(root.parent)) + if any(ord(c) < 32 for c in name): raise ValueError() + sys.stdout.buffer.write(name.encode() + b'\0') + count += 1 +if not count: raise ValueError() +PY +then fail; fi +# Source/root are data arguments; member names are NUL delimited and verbatim. +if ! tar -C "$(dirname -- "$SECRETS_DIR")" --null --verbatim-files-from \ + --no-recursion -T "$STAGING/members" -cf - 2>/dev/null \ + | age -r "$RECIPIENT" -o "$STAGING/ciphertext" 2>/dev/null; then fail; fi +python3 - "$STAGING/ciphertext" 2>/dev/null <<'PYSYNC' || fail +import os, sys +fd=os.open(sys.argv[1], os.O_RDONLY) +try: os.fsync(fd) +finally: os.close(fd) +PYSYNC +# Hard-link publication atomically refuses an output created concurrently. +ln -T -- "$STAGING/ciphertext" "$OUTPUT" 2>/dev/null || fail +printf '%s\n' 'Encrypted bundle published privately; retain it off-host and independently verify recovery before removing source custody.' diff --git a/sso-mfa/k8s/backup/DR-RUNBOOK.md b/sso-mfa/k8s/backup/DR-RUNBOOK.md index cd95c34..b1d8225 100644 --- a/sso-mfa/k8s/backup/DR-RUNBOOK.md +++ b/sso-mfa/k8s/backup/DR-RUNBOOK.md @@ -1,6 +1,10 @@ # Disaster Recovery Runbook — net-kingdom SSO/MFA Platform -Exercise status: unknown for the procedures in this runbook; no per-procedure successful-run receipt with operator attribution was established in the 2026-09-05 review. See [procedure inventory](../../../docs/attended-procedure-inventory.md). +Procedure status: legacy examples were withdrawn in the 2026-10-15 custody +review. Use current owner procedures below. Historical exercise receipts do not +prove current recovery or independent custody. See the +[procedure inventory](../../../docs/attended-procedure-inventory.md) and +[NK-WP-0049 recovery ceremony](../../../docs/bootstrap-recovery-ceremony.md). **Stack:** LLDAP + Authelia + KeyCape (sso namespace) + privacyIDEA (mfa namespace) **PostgreSQL:** Managed separately by CNPG (`postgresql/scheduled-backup.yaml`) @@ -13,34 +17,21 @@ Exercise status: unknown for the procedures in this runbook; no per-procedure su |----------|--------|----------| | Pod crash / OOM | Stateless pods (KeyCape) recover automatically. Stateful pods (LLDAP, Authelia, PI) restart and reload from PVC. | K8s self-heals. Verify with `verify-t05.sh`. | | PVC data corruption | Users/sessions/tokens lost. | Restore from SQLite backup (see below). | -| Node failure (single-node K3s) | All pods lost. PVCs intact on host. | Re-apply all manifests (idempotent). Pods re-attach to PVCs. | +| Node failure (single-node K3s) | All pods lost; PVC survival must be checked. | Owner-reviewed host/workload recovery with exact retained data and current declarations. | | Node total loss (disk gone) | Everything lost. | Full restore from backup + KeePassXC. | | Stack locked out (SSO broken, can't log in) | No user access to OIDC-protected apps. | Use break-glass account. | -| enckey lost (privacyIDEA) | All enrolled MFA tokens invalid. Users must re-enroll. | Restore from enckey backup or re-enroll all tokens. | +| enckey lost (privacyIDEA) | Enrolled token secrets cannot be decrypted without the matching key. | Restore the original encryption/audit key set and prove native token/audit continuity; never auto-generate a substitute. | --- -## Break-glass access +## Independent operator access -When the SSO stack is broken and no user can authenticate: - -```bash -# 1. Access LLDAP admin UI directly (requires VPN / IP-allowlisted access) -# URL: https://lldap.coulomb.social -# Username: break-glass -# Password: from KeePassXC → net-kingdom/Break-glass/break-glass -# -# 2. Or access LLDAP via kubectl exec (no network required) -kubectl exec -n sso deployment/lldap -- /bin/sh -# Inside container: use ldapwhoami / ldapsearch to verify directory state - -# 3. Access privacyIDEA admin UI -# URL: https://pink.coulomb.social -# Username: pi-admin -# Password: from KeePassXC → net-kingdom/privacyIDEA/pi-admin -# NOTE: pi-admin has MFA enrolled — if privacyIDEA MFA is down, use: -kubectl exec -n mfa deployment/privacyidea -- pi-manage admin list -``` +Confirm an existing provider console/rescue path and independently held custody +before any recovery change. An LLDAP account or `kubectl exec` depends on the +service or cluster being reachable; neither alone is independent recovery. +Preserve existing break-glass credentials. Use the IAM owner's approved procedure +and record actual login/MFA results, without passwords, tokens or OTP values. +Do not rerun the retired account/password helpers or disable MFA as a test. --- @@ -59,51 +50,16 @@ at startup: privacyIDEA needs PostgreSQL, KeyCape needs all three. --- -## Restore from SQLite backup (PVC data corruption) +## SQLite recovery -### LLDAP - -```bash -# 1. Scale down LLDAP -kubectl scale deployment/lldap -n sso --replicas=0 - -# 2. Start a restore pod on the lldap-data PVC -kubectl run -n sso lldap-restore --image=nouchka/sqlite3:latest \ - --restart=Never \ - --overrides='{"spec":{"volumes":[{"name":"data","persistentVolumeClaim":{"claimName":"lldap-data"}}],"containers":[{"name":"lldap-restore","image":"nouchka/sqlite3:latest","command":["sleep","3600"],"volumeMounts":[{"name":"data","mountPath":"/data"}]}]}}' - -# 3. Copy backup file into the pod (or it's already on the PVC under /data/backups/) -kubectl exec -n sso lldap-restore -- ls /data/backups/ - -# 4. Restore from the chosen backup -kubectl exec -n sso lldap-restore -- \ - sqlite3 /data/backups/users.backup.YYYY-MM-DD ".dump" | \ - sqlite3 /data/users.db - -# 5. Clean up and restart -kubectl delete pod -n sso lldap-restore -kubectl scale deployment/lldap -n sso --replicas=1 -kubectl rollout status deployment/lldap -n sso --timeout=120s -``` - -### Authelia - -```bash -# On single-node k3s (local-path PVCs are hostPath-backed), a restore pod can mount -# authelia-data alongside the running Authelia pod. Scale down only if you need to -# replace the live db.sqlite3 in-place (Authelia must be stopped to avoid corruption). -kubectl scale deployment/authelia -n sso --replicas=0 -kubectl run -n sso authelia-restore --image=nouchka/sqlite3:latest \ - --restart=Never \ - --overrides='{"spec":{"volumes":[{"name":"data","persistentVolumeClaim":{"claimName":"authelia-data"}}],"containers":[{"name":"authelia-restore","image":"nouchka/sqlite3:latest","command":["sleep","3600"],"volumeMounts":[{"name":"data","mountPath":"/data"}]}]}}' -kubectl exec -n sso authelia-restore -- ls /data/backups/ -kubectl exec -n sso authelia-restore -- \ - sqlite3 /data/backups/authelia.backup.YYYY-MM-DD ".dump" | \ - sqlite3 /data/db.sqlite3 -kubectl delete pod -n sso authelia-restore -kubectl scale deployment/authelia -n sso --replicas=1 -kubectl rollout status deployment/authelia -n sso --timeout=120s -``` +The former unpinned `sqlite3:latest` examples piped an in-pod dump into a local +shell's SQLite process; they were not a valid in-PVC restore procedure. Do not +run them. Under the exact owner-reviewed isolated ceremony, retrieve an identified +off-host database copy, verify its checksum and SQLite integrity in private +staging, stop only an explicitly approved target, and restore into its exact +pinned helper/PVC. Preserve the original files and their UIDs/ownership for +reconciliation. Restart and verify actual service behavior. Production restore +is a separate maintenance operation, not part of the local transport check. ### privacyIDEA encryption and audit keys @@ -119,83 +75,27 @@ a fresh encryption key as an automatic response to a missing recovery key. ## Full node restore (new host) -```bash -# Prerequisites on new host: -# - K3s installed -# - Traefik ingress (bundled with K3s) -# - cert-manager installed (helm install cert-manager ...) -# - DNS records pointing to new node IP -# - KeePassXC vault accessible (offline copy or age-encrypted bundle) +Prepare with the Railiance infrastructure, database and IAM owners. Obtain an +exact target inventory, immutable recovery images independent of the failed +registry, matching off-host database/key/config snapshots and retained original +state. Follow current owner deployment declarations, restore order above and +[privacyIDEA key recovery](../privacyidea/KEY-RECOVERY.md). Do not recursively +apply this historical tree or use fresh generated values in place of matching +recovery material. KeyCape's clients, issuer/signing keys and scoped factor +reader delivery must be preserved. Current package boundaries are recorded in +[the deployment guide](../README.md). -# 1. Restore PostgreSQL from CNPG backup -# (See CNPG documentation for cluster restore from barmanObjectStore) - -# 2. Re-apply all manifests in order -cd sso-mfa/k8s -kubectl apply -f namespaces/namespaces.yaml -kubectl apply -f network-policies/ -kubectl apply -f cert-manager/issuers.yaml - -# 3. Restore secrets from KeePassXC -# Run each create-secrets.sh in order: -cd postgresql && ./create-secrets.sh && cd .. -cd privacyidea && ./create-secrets.sh && cd .. -cd lldap && ./create-secrets.sh && cd .. -cd authelia && ./create-secrets.sh && cd .. -# Restore owner-managed RPF-WP-0040 factor delivery before first KeyCape creation. -# If keycape-config already exists, use the reviewed configuration-preserving restore. -cd keycape && ./create-secrets.sh && cd .. - -# 4. Apply workloads in restore order -kubectl apply -f postgresql/cluster.yaml -kubectl apply -f privacyidea/{pvc.yaml,configmap.yaml,deployment.yaml,middleware.yaml,ingress.yaml} -kubectl apply -f lldap/{pvc.yaml,deployment.yaml,middleware.yaml,ingress.yaml} -kubectl apply -f authelia/{pvc.yaml,configmap.yaml,deployment.yaml,ingress.yaml} -kubectl apply -f keycape/{deployment.yaml,middleware.yaml,ingress.yaml} - -# 5. Wait for everything to be Ready -kubectl rollout status deployment/privacyidea -n mfa --timeout=300s -kubectl rollout status deployment/lldap -n sso --timeout=120s -kubectl rollout status deployment/authelia -n sso --timeout=120s -kubectl rollout status deployment/keycape -n sso --timeout=60s - -# 6. Re-run bootstrap scripts if PVC data was lost -# Restore the original key set first using privacyidea/KEY-RECOVERY.md. -# Only then follow the reviewed account/realm reconciliation procedure. -cd privacyidea && ./enckey-bootstrap.sh /absolute/private/custody -# Restore and verify existing LLDAP identities from the owner-reviewed backup. -# Legacy break-glass creation is retired; do not overwrite existing passwords. -# For group preview/apply and structural diagnostics, follow lldap/GROUP-MAINTENANCE.md. -# Those checks do not prove native recovery or broader T07 completion. -# See ../../../docs/legacy-lldap-lifecycle-retirement.md for recovery limits. -# Verify owner-managed factor renewal, ESO delivery and mounted reload (RPF-WP-0040). -# Do not mint a personal admin token or replace existing KeyCape configuration. - -# 7. Verify -./verify-t04.sh && ./verify-t05.sh && ./verify-t06.sh --user platform-root && ./verify-t08.sh -# Separately, owner-invoked LDAP structure observation (exit 2 means incomplete acceptance): -./verify-t07.sh --root /absolute/private/custody -# Actual lifecycle, login and independent recovery receipts remain required. -``` - ---- +The whole-node restart proof and historical migration restore are supporting +receipts with their own scope; neither proves a fresh full identity restore. ## Backup offsite export -The SQLite backup files land on the PVCs but are not offsite until exported. -Run this on the node host to pull them out and encrypt for offsite storage: - -```bash -# Pull backup files from pods -kubectl exec -n sso deployment/lldap -- \ - cat /data/backups/users.backup.$(date +%Y-%m-%d) > /tmp/lldap-backup.db -kubectl exec -n sso deployment/authelia -- \ - cat /data/backups/authelia.backup.$(date +%Y-%m-%d) > /tmp/authelia-backup.db - -# Encrypt with age and send offsite (same key as the ops bundle) -age -r "$(cat ~/net-kingdom-ops-bundle.key | grep 'public key' | awk '{print $NF}')" \ - -o /tmp/lldap-backup.db.age /tmp/lldap-backup.db - -# Shred plaintext copies -shred -u /tmp/lldap-backup.db /tmp/authelia-backup.db -``` +Use the existing Platform/backup owner's reviewed encrypted transport with the +registered public recipient. No private age key is needed to encrypt. The former +example read a permanent HOME key just to find its public recipient, wrote +plaintext databases into predictable `/tmp` paths, and encrypted only one of its +two exports; it is withdrawn. Verify actual transfer and retrieval for every +required asset, preserve prior good points and independently held key custody. +Scaleway and Storage Share provider presence alone does not establish coverage +of LLDAP, Authelia, privacyIDEA keys/database, KeyCape configuration or a current +bootstrap bundle. Do not use an upload-only share as a retrieval credential. diff --git a/sso-mfa/k8s/keycape/README.md b/sso-mfa/k8s/keycape/README.md index 155bc46..c3e9253 100644 --- a/sso-mfa/k8s/keycape/README.md +++ b/sso-mfa/k8s/keycape/README.md @@ -135,7 +135,8 @@ existing live `keycape-config` Secret in place: ```bash cd sso-mfa/k8s/keycape -bash ./patch-openbao-client.sh +# Legacy patch-openbao-client.sh is retired; use the owner-reviewed +# KeyCape registration/rollout adapter with exact revision and object guards. kubectl rollout restart deployment/keycape -n sso kubectl rollout status deployment/keycape -n sso --timeout=60s bash ./verify-openbao-client.sh @@ -150,7 +151,9 @@ After the live KeyCape client is present, configure Railiance OpenBao to trust KeyCape: ```bash -bash ./configure-openbao-oidc.sh +# Legacy configure-openbao-oidc.sh is retired. +# Prepare the exact railiance-platform-owned change and run its governed admin operation +# through warden access openbao-platform-admin-login; do not supply a root token here. ``` That script enables and configures the OpenBao `netkingdom` and compatibility diff --git a/sso-mfa/k8s/keycape/configure-openbao-oidc.sh b/sso-mfa/k8s/keycape/configure-openbao-oidc.sh index 4a4b4be..1780b40 100644 --- a/sso-mfa/k8s/keycape/configure-openbao-oidc.sh +++ b/sso-mfa/k8s/keycape/configure-openbao-oidc.sh @@ -1,63 +1,5 @@ #!/usr/bin/env bash -# Configure Railiance OpenBao to trust KeyCape for platform-admin OIDC login. -# The OpenBao token is prompted inside the pod TTY and is never placed on the -# local command line or stored by this script. - -set -euo pipefail - -KUBECTL="${KUBECTL:-kubectl}" -OPENBAO_NAMESPACE="${OPENBAO_NAMESPACE:-openbao}" -OPENBAO_POD="${OPENBAO_POD:-openbao-0}" - -"$KUBECTL" exec -it -n "$OPENBAO_NAMESPACE" "$OPENBAO_POD" -- sh -lc ' - set -eu - - restore_tty() { stty echo 2>/dev/null || true; } - trap restore_tty EXIT INT TERM - - printf "OpenBao root/sudo token: " >&2 - stty -echo - read -r BAO_TOKEN - stty echo - printf "\n" >&2 - export BAO_TOKEN - - # OpenBao requires oidc_client_secret for OIDC auth config. The current - # KeyCape openbao-admin profile is public PKCE and does not validate this - # downstream client-secret field, so this compatibility value is not a - # protected secret. Replace this with a real managed client secret when - # KeyCape supports confidential downstream clients. - OPENBAO_OIDC_MOUNTS="netkingdom keycape" - - # The platform-admin role (policies, callbacks, bound claims) is owned and - # declared by railiance-platform at - # openbao/auth/netkingdom-platform-admin-role.json. This script never writes - # the role: it configures the OIDC mount and requires the declared role to - # exist. Additional allowances belong in that central declaration. - for mount in $OPENBAO_OIDC_MOUNTS; do - bao auth enable -path="$mount" oidc >/tmp/openbao-${mount}-auth-enable.out 2>/tmp/openbao-${mount}-auth-enable.err || { - if grep -q "path is already in use" /tmp/openbao-${mount}-auth-enable.err; then - printf "auth/%s already exists\n" "$mount" >&2 - else - cat /tmp/openbao-${mount}-auth-enable.err >&2 - exit 1 - fi - } - - bao write "auth/${mount}/config" \ - oidc_discovery_url="https://kc.coulomb.social" \ - oidc_client_id="openbao-admin" \ - oidc_client_secret="keycape-public-pkce-compatibility-value" \ - default_role="platform-admin" - - if ! bao read "auth/${mount}/role/platform-admin" >/dev/null 2>&1; then - printf "auth/%s/role/platform-admin is missing; apply the railiance-platform declaration (openbao/auth/netkingdom-platform-admin-role.json)\n" "$mount" >&2 - exit 1 - fi - bao write "sys/auth/${mount}/tune" listing_visibility=unauth - printf "configured auth/%s (role preserved) and listing_visibility=unauth\n" "$mount" >&2 - done - - rm -f /tmp/openbao-*-auth-enable.out /tmp/openbao-*-auth-enable.err - unset BAO_TOKEN -' +# Legacy root-token pod prompt withdrawn under NK-WP-0049. +set +x +printf '%s\n' 'Retired: use railiance-platform declarations and its governed OpenBao administration operation through warden access openbao-platform-admin-login. Prepare the exact owner-reviewed config change first. No token, auth mount or role is read or changed here.' >&2 +exit 2 diff --git a/sso-mfa/k8s/keycape/openbao-client-config.py b/sso-mfa/k8s/keycape/openbao-client-config.py index 9ed3562..b123627 100644 --- a/sso-mfa/k8s/keycape/openbao-client-config.py +++ b/sso-mfa/k8s/keycape/openbao-client-config.py @@ -198,7 +198,7 @@ def load_config() -> dict[str, Any]: try: config_text = base64.b64decode(encoded_config).decode("utf-8") except Exception as exc: # noqa: BLE001 - concise operator error - raise SystemExit(f"could not decode data.config.yaml: {exc}") from exc + raise SystemExit("could not decode private config; diagnostics suppressed") from None config = yaml.safe_load(config_text) or {} if not isinstance(config, dict): raise SystemExit("KeyCape config.yaml must decode to a YAML mapping") @@ -271,11 +271,7 @@ def enforce_lldap_defaults(config: dict[str, Any]) -> dict[str, Any]: def render_patch(config: dict[str, Any]) -> None: - updated = enforce_lldap_defaults(upsert_client(upsert_client(config, OPENBAO_CLIENT), RAPP_QONTO_CLIENT)) - config_text = yaml.safe_dump(updated, sort_keys=False) - encoded = base64.b64encode(config_text.encode("utf-8")).decode("ascii") - json.dump({"data": {"config.yaml": encoded}}, sys.stdout, separators=(",", ":")) - sys.stdout.write("\n") + raise SystemExit("Retired: raw registry patch output is disabled; use the reviewed owner rollout adapter.") def verify(config: dict[str, Any]) -> None: @@ -295,11 +291,13 @@ def main() -> None: args = parser.parse_args() if args.mode.startswith('issuer-'): raise SystemExit(issuer_main(args)) - config = load_config() if args.mode == "patch": - render_patch(config) - else: + render_patch({}) # Refuse before reading secret-bearing stdin. + try: + config = load_config() verify(config) + except (ValueError, TypeError, KeyError, AttributeError, yaml.YAMLError): + raise SystemExit("KeyCape verification failed; private input diagnostics suppressed.") from None if __name__ == "__main__": diff --git a/sso-mfa/k8s/keycape/patch-openbao-client.sh b/sso-mfa/k8s/keycape/patch-openbao-client.sh index 3ce2ac3..d007f38 100644 --- a/sso-mfa/k8s/keycape/patch-openbao-client.sh +++ b/sso-mfa/k8s/keycape/patch-openbao-client.sh @@ -1,19 +1,5 @@ #!/usr/bin/env bash -# Patch the live KeyCape config Secret with non-secret code-defined settings: -# the OpenBao admin client, browser auth mount callbacks, and LLDAP OU lookup -# paths. -# This does not require decrypted bootstrap secrets and does not print existing -# Secret values. - -set -euo pipefail - -NAMESPACE="${KEYCAPE_NAMESPACE:-sso}" -SECRET="${KEYCAPE_CONFIG_SECRET:-keycape-config}" -KUBECTL="${KUBECTL:-kubectl}" -SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" - -"$KUBECTL" get secret "$SECRET" -n "$NAMESPACE" -o json \ - | python3 "$SCRIPT_DIR/openbao-client-config.py" patch \ - | "$KUBECTL" patch secret "$SECRET" -n "$NAMESPACE" --type merge --patch-file /dev/stdin - -echo "Patched $NAMESPACE/$SECRET with the openbao-admin client and LLDAP OU lookup settings." +# Legacy unguarded registry mutation withdrawn under NK-WP-0049. +set +x +printf '%s\n' 'Retired: use the reviewed KeyCape client registration/rollout adapter with exact source, object/version checks, contained diagnostics and owner authorization. No registry or credential is read or changed here.' >&2 +exit 2 diff --git a/sso-mfa/k8s/keycape/register-coulomb-social.py b/sso-mfa/k8s/keycape/register-coulomb-social.py index 88ffae1..624fac5 100755 --- a/sso-mfa/k8s/keycape/register-coulomb-social.py +++ b/sso-mfa/k8s/keycape/register-coulomb-social.py @@ -36,34 +36,8 @@ CLIENT = { def main() -> None: - secret = json.load(sys.stdin) - encoded = secret.get("data", {}).get("config.yaml") - if not encoded: - raise SystemExit("keycape-config does not contain config.yaml") - config = yaml.safe_load(base64.b64decode(encoded)) - clients = config.setdefault("clients", []) - existing = next( - ( - index - for index, client in enumerate(clients) - if client.get("clientId") == CLIENT_ID - ), - None, - ) - if existing is None: - clients.append(CLIENT) - else: - # Preserve any extra fields operators may have set; overwrite known keys. - merged = dict(clients[existing]) - merged.update(CLIENT) - clients[existing] = merged - rendered = yaml.safe_dump(config, sort_keys=False).encode() - secret["data"]["config.yaml"] = base64.b64encode(rendered).decode() - secret.pop("status", None) - metadata = secret.get("metadata", {}) - for key in ("creationTimestamp", "managedFields", "resourceVersion", "uid"): - metadata.pop(key, None) - json.dump(secret, sys.stdout, separators=(",", ":")) + print("Retired: use the owner-reviewed KeyCape client registration adapter; raw registry output is disabled.", file=sys.stderr) + raise SystemExit(2) if __name__ == "__main__": diff --git a/sso-mfa/k8s/keycape/register-coulomb-social.sh b/sso-mfa/k8s/keycape/register-coulomb-social.sh index 09c6864..d007f38 100755 --- a/sso-mfa/k8s/keycape/register-coulomb-social.sh +++ b/sso-mfa/k8s/keycape/register-coulomb-social.sh @@ -1,20 +1,5 @@ #!/usr/bin/env bash -# Register coulomb.social as a public KeyCape OIDC client (idempotent). -# Never prints Secret values. -set -euo pipefail - -NAMESPACE="${KEYCAPE_NAMESPACE:-sso}" -SECRET="${KEYCAPE_CONFIG_SECRET:-keycape-config}" -KUBECTL="${KUBECTL:-kubectl}" -SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" - -"$KUBECTL" get secret "$SECRET" -n "$NAMESPACE" -o json \ - | python3 "$SCRIPT_DIR/register-coulomb-social.py" \ - | "$KUBECTL" apply -f - - -echo "Registered coulomb-social client in $NAMESPACE/$SECRET" - -# KeyCape loads config at process start. -"$KUBECTL" -n "$NAMESPACE" rollout restart deploy/keycape -"$KUBECTL" -n "$NAMESPACE" rollout status deploy/keycape --timeout=120s -echo "KeyCape rolled out." +# Legacy unguarded registry mutation withdrawn under NK-WP-0049. +set +x +printf '%s\n' 'Retired: use the reviewed KeyCape client registration/rollout adapter with exact source, object/version checks, contained diagnostics and owner authorization. No registry or credential is read or changed here.' >&2 +exit 2 diff --git a/sso-mfa/k8s/keycape/verify-openbao-client.sh b/sso-mfa/k8s/keycape/verify-openbao-client.sh index c4ee19f..962ec6d 100755 --- a/sso-mfa/k8s/keycape/verify-openbao-client.sh +++ b/sso-mfa/k8s/keycape/verify-openbao-client.sh @@ -2,7 +2,9 @@ # Verify the live KeyCape config carries the OpenBao admin client and KeyCape # is serving OIDC discovery after rollout. +set +x set -euo pipefail +umask 077 NAMESPACE="${KEYCAPE_NAMESPACE:-sso}" SECRET="${KEYCAPE_CONFIG_SECRET:-keycape-config}" @@ -40,8 +42,7 @@ probe_redirect() { if ! grep -qE '^HTTP/[0-9.]+ 302 ' <<<"$output"; then echo "[FAIL] $PUBLIC_AUTHORIZE_URL did not accept the $label redirect URI for openbao-admin" >&2 echo " Redirect URI: $redirect_uri" >&2 - echo " First response:" >&2 - sed -n '1,12p' <<<"$output" >&2 + echo " Provider response withheld; use owner-scoped metadata diagnostics." >&2 exit 1 fi echo "[PASS] public KeyCape authorize endpoint accepts $label redirect" @@ -60,8 +61,7 @@ if [[ -z "$KC_POD" ]]; then exit 1 fi -PF_LOG="${TMPDIR:-/tmp}/netkingdom-keycape-openbao-client-port-forward.log" -rm -f "$PF_LOG" +PF_LOG=$(mktemp "${TMPDIR:-/tmp}/netkingdom-keycape-verify-XXXXXXXX") "$KUBECTL" -n "$NAMESPACE" port-forward --address 127.0.0.1 svc/keycape "$PORT:8080" \ >"$PF_LOG" 2>&1 & @@ -70,6 +70,7 @@ PF_PID=$! cleanup() { kill "$PF_PID" 2>/dev/null || true wait "$PF_PID" 2>/dev/null || true + rm -f -- "$PF_LOG" } trap cleanup EXIT @@ -91,12 +92,12 @@ PY fi if ! kill -0 "$PF_PID" 2>/dev/null; then echo "[FAIL] KeyCape port-forward exited before discovery responded" >&2 - cat "$PF_LOG" >&2 + echo "Port-forward diagnostics withheld." >&2 exit 1 fi sleep 1 done echo "[FAIL] KeyCape discovery endpoint did not respond via local port-forward" >&2 -cat "$PF_LOG" >&2 +echo "Port-forward diagnostics withheld." >&2 exit 1 diff --git a/sso-mfa/k8s/user-engine/README.md b/sso-mfa/k8s/user-engine/README.md index 2eb59e3..76e52bb 100644 --- a/sso-mfa/k8s/user-engine/README.md +++ b/sso-mfa/k8s/user-engine/README.md @@ -4,8 +4,8 @@ Exercise status: unknown for the procedures in this runbook; no per-procedure su **Apply home moved.** The managed package is `rapp-user-engine`. Render, deploy, verify, and rollback from that repo (`make deploy`, -`make verify-live`). These files remain migration input until that -package is the only checkout operators apply. +`make verify-live`). These files are historical migration input, not a current apply or rollback +procedure. In particular, the old runtime mail binding must not be replayed. This is a stateful `rail-kubernetes` platform workload. It intentionally has no public Ingress until the KeyCape authorization-code/PKCE edge and @@ -25,12 +25,13 @@ service token at `platform/workloads/user-engine/runtime`. The `openbao-runtime.yaml` stores and ExternalSecrets deliver those values into the existing namespaced Secret names without changing application interfaces. -The audit-core sender and email-connect ingest credentials stay authoritative -in their provider namespaces. Run -`tools/sync-user-engine-delivery-secret.sh` from the repository root to copy -only those scoped values into `user-engine-delivery` without printing them. -Rerun it after either provider rotates its token, then restart user-engine -because the application receives these two values as environment variables. +Email delivery now uses email-connect's scoped `user-engine-scoped-mail` ESO +contract owned by the provider; the former shared `mail-token` is revoked and +must not be restored. The legacy `tools/sync-user-engine-delivery-secret.sh` +now refuses before credential or cluster access. Audit sender maintenance must +use the audit-core/Platform owner-reviewed delivery path; the existing audit +Secret is preserved, not rotated or revoked by source retirement. The managed +package's `docs/runtime-contract.md` is the release authority. The desired runtime also projects a short-lived ServiceAccount token with audience `flex-auth` at `/var/run/secrets/flex-auth-caller/token`. user-engine @@ -40,12 +41,10 @@ image that understands `USER_ENGINE_FLEX_AUTH_TOKEN_FILE` and flex-auth desired state that binds `system:serviceaccount:user-engine:user-engine`. The current pinned digest predates that coordinated rollout. -```bash -kubectl apply -f openbao-runtime.yaml -kubectl apply -f runtime.yaml -kubectl -n user-engine rollout status deployment/user-engine -kubectl -n user-engine get cluster,pod,service,networkpolicy -``` +Use `make verify-live` from `rapp-user-engine` for current verification. +The former `verify-operability.sh` is retired: its rollback reapplied this +migration tree and its fixed July job names could not prove a current restore. + Rollback sets the Deployment image to the preceding immutable digest. Database migrations are additive and run before serving; restore uses the diff --git a/sso-mfa/k8s/user-engine/verify-operability.sh b/sso-mfa/k8s/user-engine/verify-operability.sh index 5961d9c..a938831 100755 --- a/sso-mfa/k8s/user-engine/verify-operability.sh +++ b/sso-mfa/k8s/user-engine/verify-operability.sh @@ -1,83 +1,5 @@ #!/usr/bin/env bash -set -euo pipefail - -namespace=user-engine -deployment=user-engine -manifest="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/runtime.yaml" -rollback_target="${USER_ENGINE_ROLLBACK_IMAGE:-forgejo.coulomb.social/coulomb/user-engine@sha256:d25037e61c0c71dd9fbf89f653797932061b0be1fc5b4b23223821e5c2e3d6d0}" -exercise_rollback=false -if [[ "${1:-}" == "--exercise-rollback" ]]; then - exercise_rollback=true -elif [[ $# -gt 0 ]]; then - echo "usage: $0 [--exercise-rollback]" >&2 - exit 2 -fi - -current_image="$(kubectl -n "$namespace" get deployment "$deployment" \ - -o jsonpath='{.spec.template.spec.containers[0].image}')" - -restore_current() { - if $exercise_rollback; then - kubectl apply -f "$manifest" >/dev/null - kubectl -n "$namespace" rollout status deployment/"$deployment" \ - --timeout=180s >/dev/null - fi -} -trap restore_current EXIT - -kubectl -n "$namespace" wait --for=condition=Ready \ - cluster/user-engine-pg --timeout=60s >/dev/null -kubectl -n "$namespace" get cronjob user-engine-backup >/dev/null -kubectl -n "$namespace" wait --for=condition=complete \ - job/user-engine-backup-manual-20260729b --timeout=30s >/dev/null -kubectl -n "$namespace" wait --for=condition=complete \ - job/user-engine-restore-drill --timeout=30s >/dev/null - -metrics="$( - kubectl -n "$namespace" exec deployment/"$deployment" -- python3 -c \ - 'import os,urllib.error,urllib.request -u="http://127.0.0.1:8080/metrics"; denied=0 -try: urllib.request.urlopen(u) -except urllib.error.HTTPError as error: denied=error.code -request=urllib.request.Request(u,headers={"X-User-Engine-Proxy-Secret":os.environ["USER_ENGINE_PROXY_SECRET"].strip()}) -response=urllib.request.urlopen(request); body=response.read().decode() -print("%s %s %s" % (denied, response.status, int("user_engine_ready 1" in body)))' -)" -[[ "$metrics" == "403 200 1" ]] - -rollback_image="" -rollback_restored=true -if $exercise_rollback; then - kubectl -n "$namespace" set image deployment/"$deployment" \ - "portal=${rollback_target}" >/dev/null - kubectl -n "$namespace" rollout status deployment/"$deployment" \ - --timeout=180s >/dev/null - rollback_image="$(kubectl -n "$namespace" get deployment "$deployment" \ - -o jsonpath='{.spec.template.spec.containers[0].image}')" - [[ "$rollback_image" == "$rollback_target" ]] - kubectl apply -f "$manifest" >/dev/null - kubectl -n "$namespace" rollout status deployment/"$deployment" \ - --timeout=180s >/dev/null - restored_image="$(kubectl -n "$namespace" get deployment "$deployment" \ - -o jsonpath='{.spec.template.spec.containers[0].image}')" - [[ "$restored_image" == "$current_image" ]] -fi - -python3 - "$current_image" "$rollback_image" "$rollback_restored" <<'PY' -import json -import sys - -print(json.dumps({ - "backup_schedule_present": True, - "database_ready": True, - "latest_backup_complete": True, - "metrics_authorized": True, - "metrics_unauthorized_denied": True, - "restore_drill_complete": True, - "rollback_exercised": bool(sys.argv[2]), - "rollback_image": sys.argv[2] or None, - "rollforward_image": sys.argv[1], - "rollforward_restored": sys.argv[3] == "true", - "secret_values_observed": False, -}, sort_keys=True)) -PY +# Managed verification/rollback belongs to rapp-user-engine, not this migration tree. +set +x +printf '%s\n' 'Retired: use rapp-user-engine make verify-live and its reviewed release/rollback contract. This legacy helper cannot reapply old manifests, exercise rollback or certify historical backup jobs.' >&2 +exit 2 diff --git a/sso-mfa/k8s/verify-identity-cutover-dependencies.sh b/sso-mfa/k8s/verify-identity-cutover-dependencies.sh index e4e6086..f073c36 100755 --- a/sso-mfa/k8s/verify-identity-cutover-dependencies.sh +++ b/sso-mfa/k8s/verify-identity-cutover-dependencies.sh @@ -2,7 +2,9 @@ # Verify that the active identity stack has no runtime dependency on the # retired/source host. Secret values are decoded only in process memory and # never printed; failures report resource and key locators only. +set +x set -euo pipefail +umask 077 KUBECTL="${KUBECTL:-kubectl}" SOURCE_HOST_MARKER="${SOURCE_HOST_MARKER:-coulombcore}" diff --git a/tests/test_bootstrap_recovery_transport.py b/tests/test_bootstrap_recovery_transport.py new file mode 100644 index 0000000..e11eea4 --- /dev/null +++ b/tests/test_bootstrap_recovery_transport.py @@ -0,0 +1,87 @@ +"""Actual age/archive checks using disposable custody, never estate material.""" +import hashlib +import io +import json +import os +from pathlib import Path +import shutil +import subprocess +import tarfile + +import pytest + +ROOT=Path(__file__).resolve().parents[1] + + +@pytest.fixture +def transport(tmp_path): + checkout=tmp_path/'checkout' + for relative in ['tools/bootstrap-recovery/verify-bundle.py', 'sso-mfa/bootstrap/sops-private.py']: + target=checkout/relative;target.parent.mkdir(parents=True, exist_ok=True) + shutil.copy(ROOT/relative,target) + key=tmp_path/'fixture.key' + subprocess.run(['age-keygen','-o',str(key)],capture_output=True,check=True) + recipient=subprocess.check_output(['age-keygen','-y',str(key)],text=True).strip() + (checkout/'keys').mkdir();(checkout/'keys/age.pub').write_text(recipient+'\n') + env={'PATH':os.environ['PATH'],'HOME':str(tmp_path),'NETKINGDOM_CUSTODY_AGE_KEY_FILE':str(key), + 'SOPS_AGE_KEY':'fixture-ambient-ignored','TMPDIR':str(checkout)} + return checkout,key,recipient,env,tmp_path + + +def archive_bytes(members): + stream=io.BytesIO() + with tarfile.open(fileobj=stream,mode='w') as tar: + for name,content,kind in members: + info=tarfile.TarInfo(name) + if kind=='link':info.type=tarfile.SYMTYPE;info.linkname='/outside' + elif kind=='hardlink':info.type=tarfile.LNKTYPE;info.linkname='secrets/lldap/secrets.env' + else:info.size=len(content) + tar.addfile(info,io.BytesIO(content) if kind=='file' else None) + return stream.getvalue() + + +def verify(transport, members, *, required='lldap/secrets.env', sha=None): + checkout,_,recipient,env,root=transport + cipher=subprocess.check_output(['age','-r',recipient],input=archive_bytes(members)) + bundle=root/'input.tar.age';bundle.write_bytes(cipher) + receipt=root/'receipt.json' + result=subprocess.run(['python3',str(checkout/'tools/bootstrap-recovery/verify-bundle.py'), + '--bundle',str(bundle),'--bundle-sha256',sha or hashlib.sha256(cipher).hexdigest(), + '--required-file',required,'--receipt',str(receipt)],capture_output=True,env=env,timeout=20) + assert b'fixture-private-payload' not in result.stdout+result.stderr + assert not list(checkout.glob('netkingdom-recovery-*')) + return result,receipt + + +def test_native_ciphertext_is_restored_checked_and_removed_with_explicit_limits(transport): + r,p=verify(transport,[('secrets/lldap/secrets.env',b'VALUE=fixture-private-payload\n','file')]) + assert r.returncode==0 + receipt=json.loads(p.read_text()) + assert receipt['wrong_key_refused'] and receipt['registered_recipient_match'] + assert receipt['isolated_restore_readback_passed'] and receipt['temporary_plaintext_cleanup_verified'] + assert not receipt['native_consumer_recovery_proven'] + assert not receipt['independent_custody_proven'] + assert receipt['restored_files']==['lldap/secrets.env'] + + +@pytest.mark.parametrize('members',[ + [('../escape',b'fixture-private-payload','file')], + [('secrets/lldap/secrets.env',b'', 'link')], + [('secrets/lldap/secrets.env',b'', 'hardlink')], + [('secrets/lldap/secrets.env',b'AGE-SECRET-KEY-fixture-private-payload','file')], + [('secrets/lldap/secrets.env',b'one','file'),('secrets/lldap/secrets.env',b'two','file')], + [('secrets/unreviewed/secrets.env',b'fixture-private-payload','file')], +]) +def test_unsafe_archives_never_publish_acceptance(transport,members): + r,p=verify(transport,members) + assert r.returncode!=0 and not p.exists() + + +def test_ciphertext_digest_is_bound_before_custody_use(transport): + r,p=verify(transport,[('secrets/lldap/secrets.env',b'fixture-private-payload','file')],sha='0'*64) + assert r.returncode!=0 and not p.exists() + + +def test_missing_required_service_file_refuses_acceptance(transport): + r,p=verify(transport,[('secrets/lldap/secrets.env',b'fixture-private-payload','file')],required='privacyidea/pi.enc') + assert r.returncode!=0 and not p.exists() diff --git a/tests/test_bootstrap_secret_output.py b/tests/test_bootstrap_secret_output.py index 2fac268..1c40138 100644 --- a/tests/test_bootstrap_secret_output.py +++ b/tests/test_bootstrap_secret_output.py @@ -257,3 +257,27 @@ def test_status_keeps_historical_receipt_distinct_from_current_handoff(): assert 'historical emergency delivery' in result.stdout assert 'current recovery acknowledgment' in result.stdout assert 'does not establish native recovery readiness' in result.stdout + + +def test_interrupted_generation_never_erases_existing_custody(bootstrap_fixture): + bootstrap, _, _, env, _ = bootstrap_fixture + source=bootstrap/'secrets';source.mkdir(mode=0o700) + retained=source/'retained';retained.write_bytes(b'synthetic-retained-recovery-material') + code, transcript=run_attended(bootstrap/'creds-bootstrap-agent.sh', env) + assert code != 0 + assert retained.read_bytes()==b'synthetic-retained-recovery-material' + assert b'synthetic-retained-recovery-material' not in transcript + + +def test_failed_openbao_phase_preserves_private_files_and_refuses_completion(bootstrap_fixture): + bootstrap, state, _, env, _ = bootstrap_fixture + binary=Path(env['PATH'].split(':')[0])/'kubectl' + binary.write_text(binary.read_text().replace("'get namespace openbao' ]]; then exit 1", "'get namespace openbao' ]]; then exit 0")) + # Existing component booleans make the fixture skip creation; the optional + # owner helper returns failure. No real OpenBao or cluster is involved. + (bootstrap/'openbao-init-unseal.sh').write_text('#!/bin/bash\nexit 1\n') + code, _=run_attended(bootstrap/'creds-bootstrap-agent.sh', env) + assert code != 0 + assert (bootstrap/'secrets').is_dir() + assert 'bootstrap_complete: false' in state.read_text() + assert not list(bootstrap.parents[1].glob('ops-bundle-*.tar.age')) diff --git a/tests/test_openbao_lab_boundary.py b/tests/test_openbao_lab_boundary.py new file mode 100644 index 0000000..b1e5947 --- /dev/null +++ b/tests/test_openbao_lab_boundary.py @@ -0,0 +1,94 @@ +"""Private lab transport regression fixtures; never contacts a real cluster.""" +import json +import os +from pathlib import Path +import subprocess + +import pytest + +ROOT = Path(__file__).resolve().parents[1] +SCRIPT = ROOT / 'sso-mfa/bootstrap/openbao-init-unseal.sh' + + +@pytest.fixture +def lab(tmp_path): + metadata = tmp_path / 'metadata.json' + metadata.write_text(json.dumps({'deployment_profile': 'lab', 'openbao_unseal_custody_model': 'sops-held-automation'})) + source = tmp_path / 'secrets'; source.mkdir(mode=0o700) + executable = tmp_path / 'kubectl' + executable.write_text('''#!/usr/bin/python3 +import json,os,pathlib,sys +root=pathlib.Path(os.environ['KUBECONFIG']).parent +state=root/'fixture-state.json' +x=json.loads(state.read_text()) if state.exists() else {'initialized':False,'sealed':True,'applied':0} +args=sys.argv[1:] +with (root/'calls.jsonl').open('a') as log: log.write(json.dumps(args)+'\\n') +assert not any(k in os.environ for k in ['VAULT_TOKEN','BAO_TOKEN','BASH_ENV']) +if args[:3]==['get','namespace','kube-system']: + print(json.dumps({'metadata':{'uid':'fixture-cluster'}})) +elif 'get' in args: + print(json.dumps({'metadata':{'uid':'fixture-pod','name':'openbao-0','namespace':'openbao'}})) +elif 'status' in args: + print(json.dumps(x));sys.exit(2 if x['sealed'] else 0) +elif 'init' in args: + x['initialized']=True;state.write_text(json.dumps(x)) + print(json.dumps({'unseal_keys_b64':['A'*32,'B'*32,'C'*32], 'root_token':'fixture-private-root-marker'})) +elif 'sys/unseal' in args: + assert sys.stdin.read().strip() in ['A'*32,'B'*32,'C'*32] + x['applied']+=1;x['sealed']=x['applied']<2;state.write_text(json.dumps(x)) + print('fixture-private-provider-response') +else: sys.exit(1) +print('fixture-private-provider-diagnostic',file=sys.stderr) +''') + executable.chmod(0o755) + env = {'PATH': str(tmp_path) + ':' + os.environ['PATH'], 'HOME': str(tmp_path), + 'KUBECONFIG': str(tmp_path / 'fixture-kubeconfig'), + 'OPENBAO_CONSOLE_METADATA': str(metadata), 'OPENBAO_EXPECTED_CLUSTER_UID': 'fixture-cluster', + 'VAULT_TOKEN': 'fixture-ambient-token', 'BAO_TOKEN': 'fixture-ambient-token'} + return source, env, tmp_path + + +def invoke(lab): + source, env, _ = lab + return subprocess.run(['bash', '-x', str(SCRIPT), str(source)], env=env, capture_output=True, timeout=10) + + +def test_lab_init_unseal_keeps_material_private_and_never_uses_secret_argv(lab): + source, _, root = lab + r = invoke(lab) + assert r.returncode == 0 + transcript = r.stdout + r.stderr + assert not any(x in transcript for x in [b'fixture-private-root-marker', b'fixture-private-provider-response', + b'fixture-private-provider-diagnostic', b'A'*32, b'B'*32]) + material = source / 'openbao/init.json' + assert material.stat().st_mode & 0o077 == 0 + calls = (root / 'calls.jsonl').read_bytes() + assert b'fixture-private-root-marker' not in calls + assert b'A'*32 not in calls and b'B'*32 not in calls + assert json.loads((root / 'fixture-state.json').read_text())['applied'] == 2 + + +def test_prior_init_material_refuses_new_init_without_overwrite(lab): + source, _, root = lab + directory = source / 'openbao'; directory.mkdir(mode=0o700) + file = directory / 'init.json'; file.write_bytes(b'fixture-existing-custody'); file.chmod(0o600) + r = invoke(lab) + assert r.returncode != 0 + assert file.read_bytes() == b'fixture-existing-custody' + assert not (root / 'fixture-state.json').exists() + assert b'"init"' not in (root / 'calls.jsonl').read_bytes() + + +@pytest.mark.parametrize('mutation', ['wrong-cluster', 'missing-profile', 'production', 'configure-initial']) +def test_lab_rejects_unadmitted_targets_and_automatic_root_followthrough(lab, mutation): + _, env, root = lab + if mutation == 'wrong-cluster': env['OPENBAO_EXPECTED_CLUSTER_UID'] = 'different-cluster' + elif mutation == 'configure-initial': env['OPENBAO_RUN_CONFIGURE_INITIAL'] = '1' + else: + data=json.loads(Path(env['OPENBAO_CONSOLE_METADATA']).read_text()) + if mutation == 'missing-profile': data.pop('deployment_profile') + else: data['deployment_profile']='production' + Path(env['OPENBAO_CONSOLE_METADATA']).write_text(json.dumps(data)) + r=invoke(lab) + assert r.returncode != 0 + assert not (root/'fixture-state.json').exists() diff --git a/tests/test_retirement_and_bundle_boundaries.py b/tests/test_retirement_and_bundle_boundaries.py new file mode 100644 index 0000000..7c5cfb4 --- /dev/null +++ b/tests/test_retirement_and_bundle_boundaries.py @@ -0,0 +1,134 @@ +"""Final caller audit regressions; only disposable inputs and fake cluster tools.""" +import os +from pathlib import Path +import subprocess + +import pytest + +ROOT = Path(__file__).resolve().parents[1] +BOOT = ROOT / 'sso-mfa/bootstrap' + + +@pytest.fixture +def bundle(tmp_path): + key = tmp_path / 'fixture.key' + subprocess.run(['age-keygen', '-o', str(key)], capture_output=True, check=True) + recipient = subprocess.check_output(['age-keygen', '-y', str(key)], text=True).strip() + source = tmp_path / 'secrets'; source.mkdir(mode=0o700) + component = source / 'service'; component.mkdir(mode=0o700) + value = component / 'secrets.env' + value.write_text('VALUE=synthetic-recovery-payload\n'); value.chmod(0o600) + return source, recipient, key, tmp_path / 'bundle.tar.age' + + +def pack(bundle): + source, recipient, _, output = bundle + return subprocess.run(['bash', '-x', str(BOOT / 'pack-bundle.sh'), str(source), recipient, str(output)], + capture_output=True, timeout=10) + + +def test_bundle_does_not_replace_existing_output(bundle): + output = bundle[3]; output.write_bytes(b'previous-recovery-artifact') + r = pack(bundle) + assert r.returncode != 0 + assert output.read_bytes() == b'previous-recovery-artifact' + + +def test_bundle_refuses_links_to_outside_custody(bundle, tmp_path): + source, _, _, output = bundle + outside = tmp_path / 'outside'; outside.write_text('synthetic-outside-marker') + (source / 'service/link').symlink_to(outside) + r = pack(bundle) + assert r.returncode != 0 + assert not output.exists() + + +def test_bundle_ciphertext_is_private_and_has_no_plaintext_staging(bundle, tmp_path): + r = pack(bundle) + assert r.returncode == 0 + assert bundle[3].stat().st_mode & 0o077 == 0 + assert b'synthetic-recovery-payload' not in r.stdout + r.stderr + assert not list(tmp_path.glob('.netkingdom-bundle-*')) + clear = subprocess.check_output(['age', '-d', '-i', str(bundle[2]), str(bundle[3])]) + assert b'synthetic-recovery-payload' in clear + + +def test_bundle_refuses_private_age_key_in_payload(bundle): + source, _, key, output = bundle + (source / 'service/custody.key').write_bytes(key.read_bytes()) + (source / 'service/custody.key').chmod(0o600) + r = pack(bundle) + assert r.returncode != 0 + assert not output.exists() + + +def test_legacy_delivery_does_not_contact_cluster_or_inspect_arguments(tmp_path): + binary = tmp_path / 'kubectl'; marker = tmp_path / 'called' + binary.write_text('#!/bin/sh\ntouch "$CALLED"\necho synthetic-provider-marker >&2\nexit 1\n') + binary.chmod(0o755) + env = {**os.environ, 'PATH': str(tmp_path) + ':' + os.environ['PATH'], 'CALLED': str(marker)} + r = subprocess.run(['bash', '-x', str(ROOT / 'tools/sync-user-engine-delivery-secret.sh'), + 'synthetic-private-argument'], capture_output=True, env=env) + assert r.returncode == 2 + assert not marker.exists() + assert b'synthetic-private-argument' not in r.stdout + r.stderr + + +def test_openbao_plan_is_nonmutating_and_does_not_contact_cluster(tmp_path): + binary = tmp_path / 'kubectl'; marker = tmp_path / 'called' + binary.write_text('#!/bin/sh\ntouch "$CALLED"\nexit 1\n'); binary.chmod(0o755) + env = {**os.environ, 'PATH': str(tmp_path) + ':' + os.environ['PATH'], 'CALLED': str(marker), + 'OPENBAO_CONSOLE_METADATA': str(tmp_path / 'no-metadata')} + r = subprocess.run(['bash', '-x', str(BOOT / 'openbao-init-unseal.sh'), '--dry-run'], + capture_output=True, env=env) + assert r.returncode == 0 + assert not marker.exists() + assert not (tmp_path / 'secrets').exists() + + +@pytest.mark.parametrize('relative', ['sso-mfa/k8s/keycape/configure-openbao-oidc.sh', + 'sso-mfa/k8s/user-engine/verify-operability.sh']) +def test_retired_root_and_stale_rollout_paths_refuse_before_external_calls(tmp_path, relative): + marker = tmp_path / 'called'; binary = tmp_path / 'kubectl' + binary.write_text('#!/bin/sh\ntouch "$CALLED"\nexit 1\n'); binary.chmod(0o755) + env = {**os.environ, 'PATH': str(tmp_path) + ':' + os.environ['PATH'], 'CALLED': str(marker)} + r = subprocess.run(['bash', '-x', str(ROOT / relative), 'synthetic-private-argument'], + env=env, capture_output=True) + assert r.returncode == 2 + assert not marker.exists() + assert b'synthetic-private-argument' not in r.stdout + r.stderr + + +def test_secret_presence_does_not_rewrite_historical_acceptance(tmp_path): + import shutil + script = tmp_path / 'creds-verify.sh'; shutil.copy(BOOT / script.name, script) + state = tmp_path / 'creds-state.yaml' + original = b'secrets_applied:\n keycape: false\nenckey_bootstrapped: false\n' + state.write_bytes(original) + binary = tmp_path / 'kubectl' + binary.write_text('#!/bin/sh\nif [ "$1" = get ]; then echo secret/fixture; fi\nexit 0\n') + binary.chmod(0o755) + env = {**os.environ, 'PATH': str(tmp_path) + ':' + os.environ['PATH']} + r = subprocess.run(['bash', str(script)], env=env, capture_output=True) + assert r.returncode == 0 + assert state.read_bytes() == original + assert b'keycape-pi-token' not in r.stdout + r.stderr + + +@pytest.mark.parametrize('name', ['patch-openbao-client.sh', 'register-coulomb-social.sh']) +def test_legacy_registry_mutation_refuses_without_cluster_calls(tmp_path, name): + marker = tmp_path / 'called'; executable = tmp_path / 'kubectl' + executable.write_text('#!/bin/sh\ntouch "$CALLED"\nexit 1\n'); executable.chmod(0o755) + env = {**os.environ, 'PATH': str(tmp_path) + ':' + os.environ['PATH'], 'CALLED': str(marker)} + r = subprocess.run(['bash', '-x', str(ROOT / 'sso-mfa/k8s/keycape' / name)], env=env, capture_output=True) + assert r.returncode == 2 + assert not marker.exists() + + +@pytest.mark.parametrize('name,args', [('register-coulomb-social.py', []), ('openbao-client-config.py', ['patch'])]) +def test_legacy_python_registry_output_is_disabled(name, args): + r = subprocess.run(['python3', str(ROOT / 'sso-mfa/k8s/keycape' / name), *args], + input=b'synthetic-private-input', capture_output=True) + assert r.returncode != 0 + assert b'synthetic-private-input' not in r.stdout + r.stderr + assert not r.stdout diff --git a/tools/bootstrap-recovery/README.md b/tools/bootstrap-recovery/README.md new file mode 100644 index 0000000..dc4a010 --- /dev/null +++ b/tools/bootstrap-recovery/README.md @@ -0,0 +1,23 @@ +# Bootstrap recovery transport check + +Owner work: NK-WP-0049. Follow +[the exact recovery ceremony](../../docs/bootstrap-recovery-ceremony.md). + +`verify-bundle.py` validates a digest-bound age-encrypted `secrets/` tar archive +using an explicit private temporary key and registered `keys/age.pub`. It checks +an exact required-file list, unrelated wrong-key refusal, safe isolated file +publication/readback and staging removal. No service or cluster is contacted. + +It refuses links, special/duplicate/unreviewed entries, oversized content, private +age keys in the bundle, wrong digest/recipient and existing receipt destinations. +Only a closed metadata receipt leaves the process. Current service credentials, +independent/off-host custody, production acceptance and revocation remain separate +owner evidence. Same-UID/process-memory isolation and forensic secure erasure are +not provided by this local checker. + +Tests use real age encryption/decryption with disposable keys and malicious +archive fixtures, never production credentials: + +```bash +python3 -m pytest -q tests/test_bootstrap_recovery_transport.py +``` diff --git a/tools/bootstrap-recovery/verify-bundle.py b/tools/bootstrap-recovery/verify-bundle.py new file mode 100644 index 0000000..e7b3cfb --- /dev/null +++ b/tools/bootstrap-recovery/verify-bundle.py @@ -0,0 +1,139 @@ +#!/usr/bin/env python3 +"""Isolated ciphertext recovery check; no cluster, credentials or values emitted. + +This verifies the handoff transport and exact requested file coverage. Current +service validity, independent source custody and native consumer recovery need +separate owner evidence; this tool cannot infer them from a decrypted archive. +""" +import argparse +import hashlib +import importlib.util +import io +import json +import os +from pathlib import Path, PurePosixPath +import re +import signal +import subprocess +import sys +import tarfile +import tempfile +from datetime import datetime, timezone + +ROOT = Path(__file__).resolve().parents[2] +spec = importlib.util.spec_from_file_location('private_age_recovery', ROOT / 'sso-mfa/bootstrap/sops-private.py') +custody = importlib.util.module_from_spec(spec); spec.loader.exec_module(custody) +LIMIT = 64 * 1024 * 1024 +ALLOWED = {f'{c}/secrets.env' for c in ('privacyidea', 'postgres', 'lldap', 'authelia', 'keycape', 'breakglass')} | { + 'privacyidea/pi.enc', 'privacyidea/private.pem', 'privacyidea/public.pem', + 'keycape/key.pem', 'authelia/oidc_private_key.pem', 'openbao/init.json'} + + +class Refused(Exception): + pass + + +def recover(ciphertext, keyfd, required, work): + # Exclude ambient age keys/plugins and unrelated credential environments. + env = {'PATH': os.environ.get('PATH', os.defpath), 'HOME': str(work), 'XDG_CONFIG_HOME': str(work)} + result = subprocess.run(['age', '-d', '-i', f'/proc/self/fd/{keyfd}'], input=ciphertext, + stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, pass_fds=(keyfd,), + timeout=60, env=env, cwd=work) + if result.returncode or not result.stdout or len(result.stdout) > LIMIT: + raise Refused() + restored = set(); names = set(); total = 0 + with tarfile.open(fileobj=io.BytesIO(result.stdout), mode='r:') as archive: + for index, member in enumerate(archive): + if index >= 128 or member.name in names: + raise Refused() + names.add(member.name) + path = PurePosixPath(member.name) + if (path.is_absolute() or '..' in path.parts or '.' in path.parts + or not path.parts or path.parts[0] != 'secrets'): + raise Refused() + if member.isdir(): + if len(path.parts) > 2 or (len(path.parts) == 2 and path.parts[1] not in {p.split('/')[0] for p in ALLOWED}): + raise Refused() + continue + relative = '/'.join(path.parts[1:]) + if not member.isfile() or relative not in ALLOWED or not 0 <= member.size <= 16 * 1024 * 1024: + raise Refused() + total += member.size + if total > LIMIT or member.size == 0: + raise Refused() + content = archive.extractfile(member).read() + if len(content) != member.size or b'AGE-SECRET-KEY-' in content: + raise Refused() + target = work / 'restored' / relative + target.parent.mkdir(mode=0o700, parents=True, exist_ok=True) + fd = os.open(target, os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW, 0o600) + with os.fdopen(fd, 'wb') as dest: + dest.write(content); dest.flush(); os.fsync(dest.fileno()) + if target.read_bytes() != content: + raise Refused() + restored.add(relative) + if not required.issubset(restored): + raise Refused() + return sorted(restored) + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('--bundle', required=True) + parser.add_argument('--bundle-sha256', required=True) + parser.add_argument('--required-file', action='append', required=True) + parser.add_argument('--receipt', required=True) + args = parser.parse_args() + if not re.fullmatch('[a-f0-9]{64}', args.bundle_sha256): raise Refused() + required = set(args.required_file) + if not required or required - ALLOWED or len(required) != len(args.required_file): raise Refused() + path = Path(args.bundle).absolute() + if any(p.is_symlink() for p in (path, *path.parents)): raise Refused() + with path.open('rb') as file: + data = file.read(LIMIT + 1) + if len(data) > LIMIT or hashlib.sha256(data).hexdigest() != args.bundle_sha256: raise Refused() + recipient = custody.public_recipient() + keyfd = custody.private_key_fd() + try: + derived = subprocess.run(['age-keygen', '-y', f'/proc/self/fd/{keyfd}'], + pass_fds=(keyfd,), stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, timeout=10) + if derived.returncode or derived.stdout.strip() != recipient.encode(): raise Refused() + # Check the negative path with a disposable unrelated key. It never + # replaces the registered key, leaves temporary custody, or contacts IAM. + with tempfile.TemporaryDirectory(prefix='netkingdom-recovery-', dir='/tmp') as temporary: + work = Path(temporary) + disposable = work / 'wrong.key' + created = subprocess.run(['age-keygen', '-o', str(disposable)], stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, timeout=10) + if created.returncode: raise Refused() + wrong = subprocess.run(['age', '-d', '-i', str(disposable)], input=data, + stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, timeout=60, + env={'PATH': os.environ.get('PATH', os.defpath), 'HOME': temporary, + 'XDG_CONFIG_HOME': temporary}, cwd=temporary) + if wrong.returncode == 0: raise Refused() + restored = recover(data, keyfd, required, work) + if work.exists(): raise Refused() + finally: + os.close(keyfd) + receipt = {'schema': 'netkingdom.bootstrap-recovery-transport.v1', + 'observed_at': datetime.now(timezone.utc).isoformat(), 'bundle_sha256': args.bundle_sha256, + 'registered_recipient_match': True, 'wrong_key_refused': True, + 'required_files': sorted(required), 'restored_files': restored, + 'isolated_restore_readback_passed': True, 'temporary_plaintext_cleanup_verified': True, + 'secret_values_exported': False, 'cluster_contacted': False, + 'independent_custody_proven': False, 'current_service_validity_proven': False, + 'native_consumer_recovery_proven': False} + # Publish only the closed, safe receipt shape; preserve existing receipts. + fd = os.open(args.receipt, os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW, 0o600) + with os.fdopen(fd, 'w') as dest: + json.dump(receipt, dest, indent=2); dest.write('\n') + print('Ciphertext transport/isolated restore checks passed. Independent custody and current service recovery remain separate owner evidence.') + + +if __name__ == '__main__': + try: + for sig in (signal.SIGHUP, signal.SIGTERM, signal.SIGINT): signal.signal(sig, custody.interrupted) + main() + except (Refused, custody.Refused, OSError, ValueError, TypeError, KeyError, tarfile.TarError, subprocess.SubprocessError): + print('ERROR: recovery transport check refused or failed; no acceptance receipt. Private diagnostics suppressed.', file=sys.stderr) + raise SystemExit(1) diff --git a/tools/sync-user-engine-delivery-secret.sh b/tools/sync-user-engine-delivery-secret.sh index 854bc11..33e2c38 100755 --- a/tools/sync-user-engine-delivery-secret.sh +++ b/tools/sync-user-engine-delivery-secret.sh @@ -1,19 +1,5 @@ #!/usr/bin/env bash -set -euo pipefail - -# Copy only the two scoped receiver credentials into user-engine. Values stay -# in process memory and Kubernetes API requests; stdout contains metadata only. -audit_json="$(kubectl -n audit-core get secret audit-core-senders -o jsonpath='{.data.senders\.json}' | base64 -d)" -event_token="$(python3 -c 'import json,sys; rows=json.load(sys.stdin); print(next(row["tokens"][0] for row in rows if row["name"] == "user-engine"))' <<<"$audit_json")" -mail_token="$(kubectl -n email-connect get secret email-connect-runtime -o jsonpath='{.data.EMAIL_CONNECT_INGEST_TOKEN}' | base64 -d)" - -test -n "$event_token" -test -n "$mail_token" - -kubectl -n user-engine create secret generic user-engine-delivery \ - --from-literal=event-token="$event_token" \ - --from-literal=mail-token="$mail_token" \ - --dry-run=client -o yaml \ - | kubectl apply -f - >/dev/null - -echo "user-engine delivery Secret synchronized (values withheld)" +# Historical provider-token copier withdrawn under NK-WP-0049. +set +x +printf '%s\n' 'Retired: legacy User Engine delivery copying is unavailable. Email uses email-connect scoped ESO delivery; audit sender maintenance requires its owner-reviewed custody path. Existing credentials and Secrets are preserved.' >&2 +exit 2 diff --git a/workplans/NK-WP-0049-bootstrap-secret-free-recovery-handoff.md b/workplans/NK-WP-0049-bootstrap-secret-free-recovery-handoff.md index a04b9b3..7b3903f 100644 --- a/workplans/NK-WP-0049-bootstrap-secret-free-recovery-handoff.md +++ b/workplans/NK-WP-0049-bootstrap-secret-free-recovery-handoff.md @@ -8,7 +8,7 @@ status: active owner: codex topic_slug: netkingdom created: "2026-10-12" -updated: "2026-10-14" +updated: "2026-10-15" flavor: implementation depends_on: [] state_hub_workstream_id: "17d315f0-829e-57c7-85cc-3a7914152804" @@ -385,6 +385,59 @@ This completes the bounded two-helper source repair; the full caller/root-output inventory and owner-native recovery/consumer acceptance still keep T02 progress and T03 wait. Founder-deferred exposure rotation remains deferred. +2026-10-15 continuation: expanded the source inventory to all 87 current +non-test shell/Python callers under bootstrap/k8s plus Make, console, delivery +copier and the recovery transport checker. Reconciled the attended inventory; +source enumeration is not native acceptance. Six new regressions failed against +the old bundle/delivery/preview paths, then passed after repair. Bundle packing +now streams directly to age, refuses links/private age keys/non-private inputs, +publishes a private new output without overwriting an existing artifact, and +creates no plaintext tar intermediate. Bootstrap no longer erases retained +service custody to regenerate or continues through failed OpenBao verification. + +The lab OpenBao adapter now requires an explicit lab profile, admitted kubeconfig +and exact cluster UID, contains child diagnostics, retains private no-overwrite +init output and sends shares only on stdin. Production and automatic post-init +root configuration are refused. Existing model selection is not authorization; +new transport tests use only a fake cluster. The historical greenfield receipt +does not prove this revised adapter was exercised natively. + +Retired the unsafe User Engine shared-token copier, root-token pod prompt, +unguarded legacy registry patchers/raw Python registry output and stale managed +workload rollback. Current package/provider procedures remain authoritative. +Presence diagnostics now check the scoped factor reader and preserve historical +applied/native state. Dependency/client diagnostics suppress decoded-config +tracing and provider response dumps. Withdrew old runbook recipes that generated +replacement PI keys, wrote predictable plaintext backups, used a HOME private key +for public metadata, encrypted only one export, or piped a pod dump into local +SQLite. No existing credential/Secret/workload was rotated, deleted or reapplied. + +Read-only primary metadata: all six identity-related Deployments are Ready; +local backup jobs ran October 15. Personal-token Secret `sso/keycape-pi-token` +and `mfa/privacyidea-trigger-admin` still exist; this is not proof of validity +or provider revocation. Current job commands target PVCs, and the PI job does not +mention the audit pair; off-host coverage must be established from actual +artifacts. Founder identified Scaleway and Storage Share as backup destinations; +exact artifact/manifest and independently held matching-key availability remain +pending. No recovery key or backup content was retrieved for this review. + +Prepared `tools/bootstrap-recovery/verify-bundle.py` and +`docs/bootstrap-recovery-ceremony.md`: digest-bound ciphertext, explicit private +key/registered recipient, strict file coverage and archive checks, unrelated +wrong-key refusal, isolated file restore/readback and cleanup, with a closed +non-secret receipt. Nine actual age/archive tests use disposable material. +A successful transport check explicitly does not prove independent custody, +current credential validity or native service recovery. This is preparation for +the owner ceremony, not a completed production restore. + +T02 remains progress for current owner/runtime acceptance, legacy provider +credential disposition and retained coordinated-rotation requirements. The +founder-deferred exposure rotation stays CUST-WP-0073-T05; this request does not +silently reopen it. T03 remains wait for the exact off-host artifact/key lineage, +reviewed isolated target/procedure and actual owner/native acceptance. Evidence: +`docs/evidence/2026-10-15-bootstrap-recovery/`; full verification results are +recorded there. No workplan completion is asserted. + ## Verify owner-approved recovery and current native callers ```task @@ -392,7 +445,7 @@ id: NK-WP-0049-T03 status: wait priority: high depends_on: [NK-WP-0049-T02] -blocking_reason: "Source caller reconciliation and exact owner-scoped isolated recovery ceremony are required; no live bootstrap or root retrieval is authorized by this source receipt." +blocking_reason: "Exact off-host artifact/coverage and independently held matching key remain unconfirmed after Scaleway/Storage Share destination identification. Requires completed T02 owner/runtime and legacy-credential disposition, then the exact reviewed isolated recovery ceremony and native consumer acceptance. No live bootstrap/root retrieval or deferred exposure rotation is authorized by source tests." state_hub_task_id: "d68cd4d3-1ba3-530e-88f9-26ebb8dd6eac" ```