From dca999222f21dfc8a27d2fcd2624631f6863ebcc Mon Sep 17 00:00:00 2001 From: tegwick Date: Fri, 16 Oct 2026 00:32:50 +0200 Subject: [PATCH] Record off-host database recovery and prepare identity backup coverage Assistant: codex Assistant-Model: gpt-6-astra Assistant-Session: 01a13eab-1e6a-7b33-9c37-372e90ada3a2 --- docs/bootstrap-recovery-ceremony.md | 85 ++++++++-- .../caller-source-inventory.json | 16 +- .../database-drill-preparation.json | 7 +- .../database-drill-review.json | 15 ++ .../identity-backup-preparation-r1.json | 106 ++++++++++++ .../offsite-discovery-summary.json | 9 +- .../privacyidea-database-drill-r1.json | 28 ++++ .../readiness.json | 24 ++- tests/test_identity_backup_preparation.py | 54 +++++++ tools/bootstrap-recovery/README.md | 9 ++ .../identity-backup-profile.json | 152 ++++++++++++++++++ .../prepare-identity-coverage.py | 94 +++++++++++ ...-bootstrap-secret-free-recovery-handoff.md | 26 ++- 13 files changed, 596 insertions(+), 29 deletions(-) create mode 100644 docs/evidence/2026-10-15-bootstrap-recovery/database-drill-review.json create mode 100644 docs/evidence/2026-10-15-bootstrap-recovery/identity-backup-preparation-r1.json create mode 100644 docs/evidence/2026-10-15-bootstrap-recovery/privacyidea-database-drill-r1.json create mode 100644 tests/test_identity_backup_preparation.py create mode 100644 tools/bootstrap-recovery/identity-backup-profile.json create mode 100644 tools/bootstrap-recovery/prepare-identity-coverage.py diff --git a/docs/bootstrap-recovery-ceremony.md b/docs/bootstrap-recovery-ceremony.md index c206033..c3787dc 100644 --- a/docs/bootstrap-recovery-ceremony.md +++ b/docs/bootstrap-recovery-ceremony.md @@ -1,7 +1,8 @@ # NK-WP-0049 recovery ceremony and closure packet -Status: preparation; no live restore, credential read/grant, revocation, rotation, -root retrieval or production bootstrap is authorized by this document. +Status: actual off-host privacyIDEA database drill passed; full identity backup +coverage remains in preparation. This document grants no credential export, +provider access, revocation, rotation, root retrieval or production bootstrap. ## What has been established @@ -41,12 +42,12 @@ owner custody and pass the decryption key through attended temporary unlock. | Current consumer source | Reconcile the deployed revisions with the repaired caller/provider source. The October 14 reserved platform-group fix must be deployed and independently checked before claiming live enforcement. | | Legacy credential disposition | IAM owner identifies the exact historical personal-token/trigger principals, actual consumers and revocation method before any deletion. Deleting a Secret does not revoke an issued provider credential. | -The operator subsequently reported holding the `railiance-backup` age key. -Key material has not been received or read. Exact artifact/manifest selection -and verification that this independently held key matches the selected -ciphertext remain pending. The backup lane recipient and bootstrap -`keys/age.pub` must not be assumed to be the same. -Do not draft a blanket restore or grant approval while these remain unknown. +The operator supplied the independently held `railiance-backup` age key only to +the attended local custody helper. The actual off-host PI ciphertext decrypted +and restored successfully; the agent received no private key material. The full +identity asset manifest remains unfulfilled. The backup lane recipient and +bootstrap `keys/age.pub` must not be assumed to be the same. +Do not draft a blanket restore or grant approval while full coverage is unknown. Once fixed, bind the source revisions, target identities, allowed operations, recovery/cleanup sequence and maintenance window to the reviewed live packet. Existing eligibility or another client's approval is not this ceremony's grant. @@ -137,8 +138,8 @@ self-revoked. Existing scoped backup credentials were reused without changes; the Scaleway bootstrap API key was not used. The operator subsequently reported the hidden-terminal check passed and the -temporary key file was removed. This is operator-reported recipient matching; -the private key was not provided to the agent. Actual decryption remains pending. +temporary key file was removed. The later actual database drill also passed; +the private key was not provided to the agent. ## Prepared bounded privacyIDEA database drill — 2026-10-16 @@ -164,7 +165,9 @@ ciphertext remains for reproducibility. This proves database recovery only; token decryption, audit signature continuity and full identity service recovery require matching key/config files and further native owner checks. -Run from the repository in the operator's own terminal. Paste the private key +Executed successfully on 2026-10-16 (local time). The command is retained for +provenance; its r1 receipt now exists and must not be overwritten or rerun merely +to repeat evidence. Run future reviewed drills with a new receipt. Paste the key only into the hidden prompt, then share the success/error line or safe JSON receipt. The receipt destination must be new; do not overwrite a failed receipt. @@ -183,3 +186,63 @@ refusal, native restore/readback and cleanup. Those tests use synthetic data and fresh fixture keys. They are not evidence of actual backup decryption. If the real drill fails, review its closed receipt; do not expose PostgreSQL/age stderr or dump contents in chat. T02/T03 remain open. + +## Actual database recovery result — 2026-10-16 + +The operator reported success and the actual receipt was reviewed against the +exact checker source, pinned image and fetched ciphertext SHA256. The retained +ciphertext digest was independently rechecked. Decryption, unrelated wrong-key +refusal, native PostgreSQL restore, fixed aggregate readback and container/plaintext +cleanup all passed. The restored database has five tokens, two resolvers and +two realms. The local Docker inventory contains no remaining NK49 containers. +Evidence: `privacyidea-database-drill-r1.json` and `database-drill-review.json` +in the October15 evidence directory. No production connection occurred. + +This proves recovery of the selected database using the operator's independently +held key. It does not prove the recoverability of token plaintext, old audit +signatures, directory accounts, issuer keys or a full identity environment. + +## Missing coverage prepared at the operator's request + +The operator confirmed there is no known full identity bundle and requested +preparation of the missing coverage. The machine-readable proposal is +[`identity-backup-profile.json`](../tools/bootstrap-recovery/identity-backup-profile.json). +The read-only preparer checked the primary cluster UID, four current Deployment +UIDs/readiness/observed generations, five Bound PVC UIDs and the complete current +Secret/ConfigMap reference sets. It captured names and identities only; it did +not read Secret values or execute in pods. Its exact result is +`identity-backup-preparation-r1.json`. Fourteen tests cover foreign/stale targets, +new unreviewed dependencies, projected Secrets and private-value suppression. + +| Component | Required new coverage | Required recovery proof | +| --- | --- | --- | +| LLDAP | Consistent `users.db` snapshot, remaining durable data and current credentials/configuration | User, group and membership preservation; existing-user authentication | +| Authelia | Consistent storage DB, original encryption/OIDC/session keys and current configuration | Storage decryption, signing identity and isolated authentication | +| privacyIDEA | Original enckey plus matching audit pair, configuration/scripts and a linked native database dump | Existing token decryption/MFA, old audit signature verification and new isolated audit entry | +| KeyCape | Current registry/config and signing key, policy/receipts, every referenced credential's recovery or independent reissue disposition | Issuer/JWKS identity, registry/policy continuity and positive/negative consumer acceptance | + +The proposed primary prefix is +`platform-pg/application-archives/net-kingdom/`; the proposed secondary folder is +`Backup/railiance-backups/net-kingdom-identity/`. These are **not admitted**. +Platform's current generic application archive adapter only admits +`coulomb-social` and `email-connect`, and their publisher custody profiles are +application-bound. A new NetKingdom profile and scoped publisher require owner +admission; existing credentials must not be borrowed. The operator's Scaleway +bootstrap API key is unnecessary for preparation. + +The initial goal is one verified generation, with a measured encrypted size and +fresh provider quota/upload-overlap checks. A proposed 512 MiB ceiling is a +refusal bound, not evidence that the data fits the secondary budget. Seven daily +and two weekly points remain a later measured proposal. No retention deletion +or recurring schedule was installed. Capture must preserve original keys and +use SQLite online-backup/PostgreSQL logical snapshots, record each capture +interval and refuse changed source identities/configuration. It must not claim +global atomicity or substitute a fresh export for the already recovered old dump. + +Implementation remains: owner-admitted capture/sealing and new-profile transfer +adapters, exact credential disposition, immutable create plus retrieved readback, +then the isolated full-service recovery checks above. The current bootstrap +secrets-only tar checker cannot verify this new data-bearing profile. Approval +of preparation does not make these adapters implemented or the proposed +destinations active. The profile lists these gates explicitly; no live grant or +deployment approval is requested from this incomplete execution packet. diff --git a/docs/evidence/2026-10-15-bootstrap-recovery/caller-source-inventory.json b/docs/evidence/2026-10-15-bootstrap-recovery/caller-source-inventory.json index 0b25571..71fab68 100644 --- a/docs/evidence/2026-10-15-bootstrap-recovery/caller-source-inventory.json +++ b/docs/evidence/2026-10-15-bootstrap-recovery/caller-source-inventory.json @@ -1,8 +1,8 @@ { "schema": "netkingdom.bootstrap-caller-inventory.v1", - "observed_at": "2026-10-15T22:08:27.470032+00:00", - "scope": "all current .sh/.py non-test files under sso-mfa/bootstrap and sso-mfa/k8s, Makefile, console, delivery copier and recovery transport tool plus two bounded provider metadata inventory helpers plus bounded fixed-object ciphertext fetch and isolated PostgreSQL recovery check", - "entry_count": 91, + "observed_at": "2026-10-15T22:29:09.131605+00:00", + "scope": "all current .sh/.py non-test files under sso-mfa/bootstrap and sso-mfa/k8s, Makefile, console, delivery copier and recovery transport tool plus two bounded provider metadata inventory helpers plus bounded fixed-object ciphertext fetch and isolated PostgreSQL recovery check plus proposed full identity backup profile and metadata-only preparer", + "entry_count": 93, "entries": [ { "path": "Makefile", @@ -458,6 +458,16 @@ "path": "tools/bootstrap-recovery/check-privacyidea-database.py", "sha256": "90f454c87998d7f5d625e3c6656c3418f8f56275dc1879a9d19ed605431b94a4", "classification": "bounded off-host transfer / isolated database drill; actual recovery acceptance requires operator execution" + }, + { + "path": "tools/bootstrap-recovery/prepare-identity-coverage.py", + "sha256": "cd5f3067d28b4c7cc93ed08b8cf3869218bdecf7c54b5dffed22d697de71dbb1", + "classification": "missing identity backup coverage proposal and metadata-only preparation; no live capture/grant/delivery capability" + }, + { + "path": "tools/bootstrap-recovery/identity-backup-profile.json", + "sha256": "176c1f2f9111c2c51503212deb162274609d5e53bfe03d7319e19e44c5e09500", + "classification": "missing identity backup coverage proposal and metadata-only preparation; no live capture/grant/delivery capability" } ], "source_inventory_complete_for_declared_scope": true, diff --git a/docs/evidence/2026-10-15-bootstrap-recovery/database-drill-preparation.json b/docs/evidence/2026-10-15-bootstrap-recovery/database-drill-preparation.json index 1320105..26b661b 100644 --- a/docs/evidence/2026-10-15-bootstrap-recovery/database-drill-preparation.json +++ b/docs/evidence/2026-10-15-bootstrap-recovery/database-drill-preparation.json @@ -2,7 +2,7 @@ "observed_at": "2026-10-15T22:08:27.470032+00:00", "operator_reported_key_check_passed": true, "private_key_received_by_agent": false, - "actual_ciphertext_decryption_verified": false, + "actual_ciphertext_decryption_verified": true, "transfer_receipt": "privacyidea-transfer-r1.json", "transfer_envelope_exit_code": 0, "transfer_session_revoked_and_cleaned": true, @@ -13,7 +13,7 @@ "data": "synthetic database and fresh disposable age keys only", "includes_native_postgres_restore_and_cleanup": true }, - "pending_human_command": "docs/bootstrap-recovery-ceremony.md: Prepared bounded privacyIDEA database drill", + "pending_human_command": null, "native_identity_recovery_proven": false, "sources": [ { @@ -28,5 +28,6 @@ "path": "tests/test_privacyidea_offhost_recovery.py", "sha256": "ea977b0d3368fb14f6a7a252c24cc5cb9915022afa3dca61c80952e7489bd0e5" } - ] + ], + "execution_receipt": "privacyidea-database-drill-r1.json" } diff --git a/docs/evidence/2026-10-15-bootstrap-recovery/database-drill-review.json b/docs/evidence/2026-10-15-bootstrap-recovery/database-drill-review.json new file mode 100644 index 0000000..85754ca --- /dev/null +++ b/docs/evidence/2026-10-15-bootstrap-recovery/database-drill-review.json @@ -0,0 +1,15 @@ +{ + "observed_at": "2026-10-15T22:24:49.717396+00:00", + "receipt": "privacyidea-database-drill-r1.json", + "receipt_sha256": "c72dc3ababcc3ef3d7485a434128daf71e0ee58a8118e633fb09fed516f4b980", + "source_and_transfer_binding_verified": true, + "retained_ciphertext_digest_rechecked": true, + "operator_reported_success": true, + "independent_custody_basis": "Operator reports independently held railiance-backup key; attended temporary-key drill decrypted the fetched ciphertext. Agent never received private key material.", + "docker_nk49_container_inventory_empty": true, + "full_identity_recovery_proven": false, + "operator_reports_no_known_full_identity_bundle": true, + "next_work_authorized": "Prepare missing backup coverage; no blanket production restore or credential rotation.", + "coverage_preparation": "identity-backup-preparation-r1.json", + "operator_coverage_decision_id": "ed20023f-0168-421d-be86-460391339876" +} diff --git a/docs/evidence/2026-10-15-bootstrap-recovery/identity-backup-preparation-r1.json b/docs/evidence/2026-10-15-bootstrap-recovery/identity-backup-preparation-r1.json new file mode 100644 index 0000000..8a26627 --- /dev/null +++ b/docs/evidence/2026-10-15-bootstrap-recovery/identity-backup-preparation-r1.json @@ -0,0 +1,106 @@ +{ + "schema": "netkingdom.identity-backup-preparation.v1", + "status": "prepared_not_admitted", + "observed_at": "2026-10-15T22:32:41.149409+00:00", + "cluster_uid": "a553c742-0115-43d4-99a4-a5ca56fe0786", + "profile_sha256": "176c1f2f9111c2c51503212deb162274609d5e53bfe03d7319e19e44c5e09500", + "source_sha256": "cd5f3067d28b4c7cc93ed08b8cf3869218bdecf7c54b5dffed22d697de71dbb1", + "bindings": [ + { + "namespace": "sso", + "deployment": "lldap", + "uid": "743a86bd-5cb0-4ca0-9c56-a3438f4746a8", + "generation": 7, + "pvc_bindings": [ + { + "name": "lldap-data", + "uid": "0a06b65a-e7b3-42d3-916e-e74dbe5fc7da" + } + ], + "secret_names_only": [ + "lldap-secrets" + ], + "configmap_names_only": [] + }, + { + "namespace": "sso", + "deployment": "authelia", + "uid": "dc40a925-2306-4a3b-89c0-e714187503c3", + "generation": 28, + "pvc_bindings": [ + { + "name": "authelia-data", + "uid": "86826554-b014-4df4-8b28-866aa64f6413" + } + ], + "secret_names_only": [ + "authelia-secrets", + "authelia-smtp" + ], + "configmap_names_only": [ + "authelia-config" + ] + }, + { + "namespace": "sso", + "deployment": "keycape", + "uid": "99ddd83c-cb3f-4847-bcf8-35f1aa87627f", + "generation": 65, + "pvc_bindings": [ + { + "name": "keycape-authentication-policy", + "uid": "423f558f-8c11-4edf-956b-243559df4a8e" + } + ], + "secret_names_only": [ + "keycape-approval-engine-operator-client", + "keycape-config", + "keycape-factor-read", + "keycape-informed-decision-sitting-requester-client", + "keycape-rapp-qonto-client", + "keycape-secrets-engine-approval-client", + "keycape-secrets-engine-openbao-client", + "keycape-secrets-engine-requester-client" + ], + "configmap_names_only": [] + }, + { + "namespace": "mfa", + "deployment": "privacyidea", + "uid": "58c7f96d-61cb-4dd4-bca2-54661c0ac375", + "generation": 11, + "pvc_bindings": [ + { + "name": "privacyidea-data", + "uid": "65650fc1-02d8-4c0f-84fc-e0015c8f3ed9" + }, + { + "name": "privacyidea-logs", + "uid": "39fa90d9-bc54-4a03-a0d6-ae85165e0cb9" + } + ], + "secret_names_only": [ + "privacyidea-config" + ], + "configmap_names_only": [ + "factor-recovery-01119187d90d7b76", + "keycape-onboarding-9465e5e0b98f7986", + "privacyidea-cfg" + ] + } + ], + "secret_values_read": false, + "backup_captured": false, + "uploaded": false, + "runtime_changed": false, + "credential_grant_created": false, + "owner_admission_required": [ + "Railiance owner accepts exact read/capture executor, immutable helper images, PVC access and task-scoped caller attribution; no production stop/restart inferred", + "NetKingdom/service owners accept full asset manifest, original key preservation, consistency interval, bounded recovery environment and native acceptance", + "Platform/reef-storage admits both new archive prefixes, immutable create/readback, scoped publisher custody and current quota/headroom; current application archive profiles do not include net-kingdom", + "Resolve every referenced credential through encrypted recovery or an independent owner reissue path; no undocumented grant propagation", + "Implement and test capture/sealing/transfer adapters against this new profile before live execution; existing bootstrap tar checker is insufficient", + "Retrieve by immutable object/version from each admitted provider; digest check, attended independent-key decryption, native service recovery, failed-key case and verified cleanup", + "Only then enable recurring capture/alerts and measured retention; preserve T02 legacy provider-credential and deferred-rotation boundaries" + ] +} diff --git a/docs/evidence/2026-10-15-bootstrap-recovery/offsite-discovery-summary.json b/docs/evidence/2026-10-15-bootstrap-recovery/offsite-discovery-summary.json index 1d29f3b..894e30f 100644 --- a/docs/evidence/2026-10-15-bootstrap-recovery/offsite-discovery-summary.json +++ b/docs/evidence/2026-10-15-bootstrap-recovery/offsite-discovery-summary.json @@ -2,7 +2,7 @@ "observed_at": "2026-10-15T22:08:27.470032+00:00", "operator_reports_holding_railiance_backup_age_key": true, "private_key_received_or_read": false, - "candidate_key_match_verified": false, + "candidate_key_match_verified": true, "storage_share": { "inventory": "storage-share-inventory-r1.json", "status": "metadata_inventory_complete", @@ -14,7 +14,9 @@ "bootstrap_identity_file_bundle_found_in_inspected_scope": false, "transfer_receipt": "privacyidea-transfer-r1.json", "attended_transfer_envelope_exit_code": 0, - "transfer_session_self_revocation_and_cleanup_confirmed": true + "transfer_session_self_revocation_and_cleanup_confirmed": true, + "candidate_decryption_and_database_restore_verified": true, + "database_drill_receipt": "privacyidea-database-drill-r1.json" }, "scaleway": { "inventory": "scaleway-inventory-r1.json", @@ -38,5 +40,6 @@ "command": "python3 -m pytest -q tests/test_scaleway_backup_inventory.py tests/test_storage_share_inventory.py" }, "native_recovery_proven": false, - "production_mutations_performed": false + "production_mutations_performed": false, + "private_key_received_or_read_scope": "Agent custody: false; attended human recovery child used the temporary explicit key." } diff --git a/docs/evidence/2026-10-15-bootstrap-recovery/privacyidea-database-drill-r1.json b/docs/evidence/2026-10-15-bootstrap-recovery/privacyidea-database-drill-r1.json new file mode 100644 index 0000000..55ce9ff --- /dev/null +++ b/docs/evidence/2026-10-15-bootstrap-recovery/privacyidea-database-drill-r1.json @@ -0,0 +1,28 @@ +{ + "schema": "netkingdom.privacyidea-database-drill.v1", + "status": "isolated_database_drill_passed", + "started_at": "2026-10-15T22:18:55.858354+00:00", + "source_sha256": "90f454c87998d7f5d625e3c6656c3418f8f56275dc1879a9d19ed605431b94a4", + "production_contacted": false, + "secret_values_exported": false, + "native_identity_recovery_proven": false, + "token_decryption_proven": false, + "audit_signature_continuity_proven": false, + "full_identity_file_coverage_proven": false, + "container_cleanup_verified": true, + "temporary_plaintext_cleanup_verified": true, + "ciphertext_sha256": "cc81da442755928171d2c51bce93eda7de4392a0a1aa9d0ffb83a25ba34e5ead", + "registered_recipient_match": true, + "wrong_key_refused": true, + "offhost_ciphertext_decryption_verified": true, + "image": "postgres@sha256:ff23cdce56cac62ada6f66013e1a50864c0abbe79d132d40b6e05bd80f378a70", + "image_id": "sha256:ff23cdce56cac62ada6f66013e1a50864c0abbe79d132d40b6e05bd80f378a70", + "pg_restore_version": "pg_restore (PostgreSQL) 16.14 (Debian 16.14-1.pgdg13+1)", + "aggregate_row_counts": { + "token": 5, + "resolver": 2, + "realm": 2 + }, + "database_restore_and_readback_passed": true, + "finished_at": "2026-10-15T22:19:05.279736+00:00" +} diff --git a/docs/evidence/2026-10-15-bootstrap-recovery/readiness.json b/docs/evidence/2026-10-15-bootstrap-recovery/readiness.json index 572f096..d164adf 100644 --- a/docs/evidence/2026-10-15-bootstrap-recovery/readiness.json +++ b/docs/evidence/2026-10-15-bootstrap-recovery/readiness.json @@ -1,13 +1,13 @@ { "workplan": "NK-WP-0049", - "observed_at": "2026-10-15T22:08:27.470032+00:00", - "status": "source_prepared_pending_owner_inputs_and_native_acceptance", + "observed_at": "2026-10-15T22:24:49.717396+00:00", + "status": "offhost_privacyidea_database_recovered_full_identity_coverage_open", "backup_destinations_operator_reported": [ "Scaleway", "Storage Share" ], "exact_offhost_artifact_confirmed": true, - "independently_held_matching_key_confirmed": false, + "independently_held_matching_key_confirmed": true, "recovery_key_or_backup_contents_retrieved": true, "legacy_token_secret_present": true, "legacy_trigger_secret_present": true, @@ -18,15 +18,14 @@ "verification": "source-verification.json", "ceremony": "../../bootstrap-recovery-ceremony.md", "outstanding": [ - "Matching off-host identity data/key/config set remains unlocated in inspected scopes; selected existing PI database ciphertext is now fetched and digest-bound", - "Operator reports matching-recipient key check passed; actual ciphertext decryption/native database drill is prepared and pending operator hidden-key invocation", - "Full identity service recovery, token decryption, audit signature continuity and cleanup evidence", + "Prepare and admit missing encrypted identity data/key/config backup coverage; operator confirms no known full identity bundle", + "Retrieve new off-host bundle and prove original token decryption, audit continuity, directory and issuer recovery with safe negative cases and cleanup", "T02 current caller/provider source admission and exact legacy provider credential disposition; retain existing deferred-rotation boundary" ], "operator_reports_holding_railiance_backup_age_key": true, "operator_report_recorded_at": "2026-10-15T21:52:22.549430+00:00", "key_material_received": false, - "key_matches_selected_ciphertext_verified": false, + "key_matches_selected_ciphertext_verified": true, "offsite_discovery": "offsite-discovery-summary.json", "privacyidea_database_ciphertext_identified": true, "required_full_asset_coverage_confirmed": false, @@ -36,5 +35,14 @@ "ciphertext_fetched": true, "transfer_receipt": "privacyidea-transfer-r1.json", "prepared_database_drill": "../../bootstrap-recovery-ceremony.md", - "age_private_key_retrieved_by_agent": false + "age_private_key_retrieved_by_agent": false, + "offhost_database_restore_proven": true, + "database_drill_receipt": "privacyidea-database-drill-r1.json", + "database_drill_review": "database-drill-review.json", + "operator_reports_no_known_full_identity_bundle": true, + "identity_backup_profile": "../../../tools/bootstrap-recovery/identity-backup-profile.json", + "identity_coverage_preparation": "identity-backup-preparation-r1.json", + "coverage_preparation_status": "prepared_not_admitted", + "coverage_preparation_tests_passed": 14, + "coverage_decision_id": "ed20023f-0168-421d-be86-460391339876" } diff --git a/tests/test_identity_backup_preparation.py b/tests/test_identity_backup_preparation.py new file mode 100644 index 0000000..efec56a --- /dev/null +++ b/tests/test_identity_backup_preparation.py @@ -0,0 +1,54 @@ +"""Preparation must never promote stale metadata or expand its data-read scope.""" +import importlib.util +import json +from pathlib import Path +import pytest + +ROOT=Path(__file__).resolve().parents[1] +spec=importlib.util.spec_from_file_location('prepare',ROOT/'tools/bootstrap-recovery/prepare-identity-coverage.py') +m=importlib.util.module_from_spec(spec);spec.loader.exec_module(m) + + +def fixture(): + profile=json.loads(m.PROFILE.read_text());objects={} + objects[(None,'namespace','kube-system')]={'metadata':{'uid':m.CLUSTER_UID}} + for c in profile['components']: + ns=c['namespace'];name=c['deployment'] + objects[(ns,'deployment',name)]={'metadata':{'uid':c['deployment_uid'],'generation':9}, + 'status':{'observedGeneration':9,'readyReplicas':1},'spec':{'template':{'spec':{ + 'containers':[{'name':'service','env':[{'name':'DO_NOT_EXPORT','value':'private-value-canary'}], + 'envFrom':[{'secretRef':{'name':n}} for n in c['secret_refs']]}], + 'volumes':[{'configMap':{'name':n}} for n in c['configmap_refs']]+ + [{'persistentVolumeClaim':{'claimName':p['name']}} for p in c['pvc_refs']]}}}} + for pvc in c['pvc_refs']:objects[(ns,'pvc',pvc['name'])]={'metadata':{'uid':pvc['uid']},'status':{'phase':'Bound'}} + calls=[] + def get(ns,kind,name):calls.append((ns,kind,name));return objects[(ns,kind,name)] + return profile,objects,calls,get + + +def test_only_safe_metadata_leaves_preparation_and_no_secret_api_is_called(): + p,objects,calls,get=fixture();r=m.prepare(p,get) + assert r['status']=='prepared_not_admitted' + assert not any(r[k] for k in ['secret_values_read','backup_captured','uploaded','runtime_changed','credential_grant_created']) + assert 'private-value-canary' not in json.dumps(r) + assert all(kind in {'namespace','deployment','pvc'} for ns,kind,name in calls) + + +@pytest.mark.parametrize('change',['cluster','pvc','deployment','not-ready','unobserved','terminating','new-secret','projected-secret','new-pvc']) +def test_stale_or_expanded_live_state_requires_fresh_owner_review(change): + p,o,calls,get=fixture();d=o[('sso','deployment','lldap')];s=d['spec']['template']['spec'] + if change=='cluster':o[(None,'namespace','kube-system')]['metadata']['uid']='other-cluster' + if change=='pvc':o[('sso','pvc','lldap-data')]['metadata']['uid']='replacement' + if change=='deployment':d['metadata']['uid']='replacement' + if change=='not-ready':d['status']['readyReplicas']=0 + if change=='unobserved':d['status']['observedGeneration']=8 + if change=='terminating':d['metadata']['deletionTimestamp']='2026-10-16' + if change=='new-secret':s['containers'][0]['envFrom'].append({'secretRef':{'name':'unreviewed-secret'}}) + if change=='projected-secret':s['volumes'].append({'projected':{'sources':[{'secret':{'name':'unreviewed-secret'}}]}}) + if change=='new-pvc':s['volumes'].append({'persistentVolumeClaim':{'claimName':'unreviewed-pvc'}}) + with pytest.raises(m.Refused):m.prepare(p,get) + + +@pytest.mark.parametrize('kind',['secret','configmap','pods/exec','clusterrolebinding']) +def test_preparation_transport_has_no_private_data_or_mutation_route(kind): + with pytest.raises(m.Refused):m.query('sso',kind,'anything') diff --git a/tools/bootstrap-recovery/README.md b/tools/bootstrap-recovery/README.md index 18a0a58..6b38587 100644 --- a/tools/bootstrap-recovery/README.md +++ b/tools/bootstrap-recovery/README.md @@ -52,3 +52,12 @@ NK49_RUN_DISPOSABLE_POSTGRES_TEST=1 python3 -m pytest -q \ tests/test_privacyidea_offhost_recovery.py \ tests/test_storage_share_inventory.py tests/test_scaleway_backup_inventory.py ``` + +`identity-backup-profile.json` prepares missing full service coverage and names +the exact current workload/PVC/credential dependencies, proposed provider paths +and remaining implementation/admission gates. It is a proposal, not a new live +publisher grant. `prepare-identity-coverage.py --receipt NEW_PATH` verifies the +primary cluster and current deployment/PVC/reference bindings using metadata +only. It refuses replacement resources, unobserved/unready deployments and new +unreviewed references. It has no Secret-data, pod-exec, mutation or upload route. +Its receipt says `prepared_not_admitted`; it cannot prove a backup exists. diff --git a/tools/bootstrap-recovery/identity-backup-profile.json b/tools/bootstrap-recovery/identity-backup-profile.json new file mode 100644 index 0000000..93a550c --- /dev/null +++ b/tools/bootstrap-recovery/identity-backup-profile.json @@ -0,0 +1,152 @@ +{ + "schema": "netkingdom.identity-backup-profile.v1", + "status": "proposed", + "owner": "net-kingdom with railiance-platform/rapp-postgres runtime and storage admission", + "public_recipient": "age1zvryunvjhvpkmasskauga2heeg0ztnte9ymgppvjge36ekumk50syr3tsz", + "archive_profile": "netkingdom-identity-v1; new profile, not the bootstrap secrets-only tar profile", + "components": [ + { + "namespace": "sso", + "deployment": "lldap", + "deployment_uid": "743a86bd-5cb0-4ca0-9c56-a3438f4746a8", + "pvc_refs": [ + { + "name": "lldap-data", + "uid": "0a06b65a-e7b3-42d3-916e-e74dbe5fc7da" + } + ], + "secret_refs": [ + "lldap-secrets" + ], + "configmap_refs": [], + "required_assets": [ + "SQLite online-backup snapshot of /data/users.db with integrity check", + "remaining non-backup durable /data files, including any private identity material", + "current deployment configuration and exact lldap-secrets fields" + ], + "native_acceptance": [ + "directory user/group/membership preservation", + "existing-user password authentication in isolated environment" + ] + }, + { + "namespace": "sso", + "deployment": "authelia", + "deployment_uid": "dc40a925-2306-4a3b-89c0-e714187503c3", + "pvc_refs": [ + { + "name": "authelia-data", + "uid": "86826554-b014-4df4-8b28-866aa64f6413" + } + ], + "secret_refs": [ + "authelia-secrets", + "authelia-smtp" + ], + "configmap_refs": [ + "authelia-config" + ], + "required_assets": [ + "SQLite online-backup snapshot of /var/authelia/data/db.sqlite3 with integrity check", + "current configuration.yml and all Secret-mounted encryption/OIDC/session fields", + "explicit disposition for notifier credentials and remaining durable files" + ], + "native_acceptance": [ + "storage decryption", + "same OIDC signing identity and preserved client configuration", + "isolated authentication flow with outbound notification disabled" + ] + }, + { + "namespace": "sso", + "deployment": "keycape", + "deployment_uid": "99ddd83c-cb3f-4847-bcf8-35f1aa87627f", + "pvc_refs": [ + { + "name": "keycape-authentication-policy", + "uid": "423f558f-8c11-4edf-956b-243559df4a8e" + } + ], + "secret_refs": [ + "keycape-approval-engine-operator-client", + "keycape-config", + "keycape-factor-read", + "keycape-informed-decision-sitting-requester-client", + "keycape-rapp-qonto-client", + "keycape-secrets-engine-approval-client", + "keycape-secrets-engine-openbao-client", + "keycape-secrets-engine-requester-client" + ], + "configmap_refs": [], + "required_assets": [ + "exact current keycape-config config.yaml and key.pem", + "complete stable policy/receipt PVC snapshot", + "current deployment and owner disposition for every referenced credential: encrypted recovery or independently recoverable provider reissue path" + ], + "native_acceptance": [ + "same issuer/JWKS signing identity", + "registry, authentication policy and receipt continuity", + "current consumer positive/negative authentication under isolated test endpoints" + ] + }, + { + "namespace": "mfa", + "deployment": "privacyidea", + "deployment_uid": "58c7f96d-61cb-4dd4-bca2-54661c0ac375", + "pvc_refs": [ + { + "name": "privacyidea-data", + "uid": "65650fc1-02d8-4c0f-84fc-e0015c8f3ed9" + }, + { + "name": "privacyidea-logs", + "uid": "39fa90d9-bc54-4a03-a0d6-ae85165e0cb9" + } + ], + "secret_refs": [ + "privacyidea-config" + ], + "configmap_refs": [ + "factor-recovery-01119187d90d7b76", + "keycape-onboarding-9465e5e0b98f7986", + "privacyidea-cfg" + ], + "required_assets": [ + "original /etc/privacyidea/enckey, private.pem and public.pem with matching RSA pair validation", + "current privacyidea-config and mounted configuration/scripts", + "bounded native PostgreSQL custom dump with exact database identity and key-generation linkage", + "explicit disposition for service logs; audit database/signatures must be preserved" + ], + "native_acceptance": [ + "decrypt existing retained token material without exporting it", + "verify pre-backup audit signatures and create/verify a new isolated audit entry", + "existing-token MFA and negative-key refusal" + ] + } + ], + "capture_contract": { + "plaintext": "private task staging outside Git; never process output, State Hub or logs; verified cleanup after encryption", + "consistency": "native SQLite online-backup and PostgreSQL logical snapshot; explicit capture interval per asset; no global atomic snapshot claim; before/after source identities/config versions and directory inventories must match", + "key_policy": "preserve original service keys; no age private key in the payload, no replacement generation, no root/unseal collection", + "encryption": "public-recipient-only streaming archive to age; private fresh ciphertext, no overwrite; manifest inside ciphertext with source identities, times, complete required paths, digests and omissions", + "limits": "measure first; proposed 512 MiB encrypted admission ceiling, no silent truncation/exclusions", + "historical_database": "existing verified October15 dump remains protected; new bundle must link its own exact database/key capture generation and recovery receipt" + }, + "proposed_destinations": { + "primary": "s3://railiance-platform-pg-backup/platform-pg/application-archives/net-kingdom/", + "secondary": "Storage Share Backup/railiance-backups/net-kingdom-identity/", + "admitted": false, + "publisher": "new exact prefix-scoped owner-delivered publisher; do not borrow another workload reader or bootstrap API key", + "retention": "initial single verified generation; later seven daily/two weekly only after measured quota/overlap/headroom and separately admitted retention executor; never expire existing good backups", + "schedule": "not installed; propose daily capture after logical database backup with per-generation linkage, separate failure alert and operator-approved RPO/RTO" + }, + "admission_gates": [ + "Railiance owner accepts exact read/capture executor, immutable helper images, PVC access and task-scoped caller attribution; no production stop/restart inferred", + "NetKingdom/service owners accept full asset manifest, original key preservation, consistency interval, bounded recovery environment and native acceptance", + "Platform/reef-storage admits both new archive prefixes, immutable create/readback, scoped publisher custody and current quota/headroom; current application archive profiles do not include net-kingdom", + "Resolve every referenced credential through encrypted recovery or an independent owner reissue path; no undocumented grant propagation", + "Implement and test capture/sealing/transfer adapters against this new profile before live execution; existing bootstrap tar checker is insufficient", + "Retrieve by immutable object/version from each admitted provider; digest check, attended independent-key decryption, native service recovery, failed-key case and verified cleanup", + "Only then enable recurring capture/alerts and measured retention; preserve T02 legacy provider-credential and deferred-rotation boundaries" + ] +} diff --git a/tools/bootstrap-recovery/prepare-identity-coverage.py b/tools/bootstrap-recovery/prepare-identity-coverage.py new file mode 100644 index 0000000..4d946ef --- /dev/null +++ b/tools/bootstrap-recovery/prepare-identity-coverage.py @@ -0,0 +1,94 @@ +#!/usr/bin/env python3 +"""Prepare exact identity backup coverage from read-only workload/PVC metadata. + +This produces a proposal, never reads Secrets, executes in pods, changes cluster +resources, captures private files, uploads, schedules or grants credentials. +""" +import argparse +from datetime import datetime,timezone +import hashlib +import json +import os +from pathlib import Path +import subprocess + +ROOT=Path(__file__).resolve().parents[2] +CLUSTER_UID='a553c742-0115-43d4-99a4-a5ca56fe0786' +PROFILE=ROOT/'tools/bootstrap-recovery/identity-backup-profile.json' + + +class Refused(Exception):pass + + +def query(namespace,kind,name): + # Closed read-only types; Secret/ConfigMap data and arbitrary API paths have + # no transport in this preparation command. + if kind not in {'namespace','deployment','pvc'}:raise Refused() + args=['kubectl','--request-timeout=20s'] + if namespace:args+=['-n',namespace] + r=subprocess.run(args+['get',kind,name,'-o','json'],capture_output=True,timeout=25) + if r.returncode:raise Refused() + return json.loads(r.stdout) + + +def references(spec): + secrets=set();configmaps=set();pvcs=set() + for c in spec.get('containers',[])+spec.get('initContainers',[]): + for e in c.get('env',[]): + v=e.get('valueFrom',{}) + if 'secretKeyRef' in v:secrets.add(v['secretKeyRef']['name']) + if 'configMapKeyRef' in v:configmaps.add(v['configMapKeyRef']['name']) + for e in c.get('envFrom',[]): + if 'secretRef' in e:secrets.add(e['secretRef']['name']) + if 'configMapRef' in e:configmaps.add(e['configMapRef']['name']) + for v in spec.get('volumes',[]): + if 'secret' in v:secrets.add(v['secret']['secretName']) + if 'configMap' in v:configmaps.add(v['configMap']['name']) + if 'persistentVolumeClaim' in v:pvcs.add(v['persistentVolumeClaim']['claimName']) + for p in v.get('projected',{}).get('sources',[]): + if 'secret' in p:secrets.add(p['secret']['name']) + if 'configMap' in p:configmaps.add(p['configMap']['name']) + return secrets,configmaps,pvcs + + +def prepare(profile,get=query): + if get(None,'namespace','kube-system')['metadata']['uid']!=CLUSTER_UID:raise Refused() + bindings=[] + for component in profile['components']: + ns=component['namespace'];name=component['deployment'] + d=get(ns,'deployment',name);m=d['metadata'] + if (m['uid']!=component['deployment_uid'] or m.get('deletionTimestamp') or + d['status'].get('observedGeneration')!=m['generation'] or d['status'].get('readyReplicas')!=1):raise Refused() + refs=references(d['spec']['template']['spec']) + expected_pvcs={p['name']:p['uid'] for p in component['pvc_refs']} + if refs[0]!=set(component['secret_refs']) or refs[1]!=set(component['configmap_refs']) or refs[2]!=set(expected_pvcs):raise Refused() + pvcs=[] + for pvc,expected in expected_pvcs.items(): + obj=get(ns,'pvc',pvc) + if obj['metadata']['uid']!=expected or obj['metadata'].get('deletionTimestamp') or obj['status'].get('phase')!='Bound':raise Refused() + pvcs.append({'name':pvc,'uid':expected}) + bindings.append({'namespace':ns,'deployment':name,'uid':m['uid'],'generation':m['generation'], + 'pvc_bindings':pvcs,'secret_names_only':sorted(refs[0]),'configmap_names_only':sorted(refs[1])}) + if get(None,'namespace','kube-system')['metadata']['uid']!=CLUSTER_UID:raise Refused() + return {'schema':'netkingdom.identity-backup-preparation.v1','status':'prepared_not_admitted', + 'observed_at':datetime.now(timezone.utc).isoformat(),'cluster_uid':CLUSTER_UID, + 'profile_sha256':hashlib.sha256(PROFILE.read_bytes()).hexdigest(), + 'source_sha256':hashlib.sha256(Path(__file__).read_bytes()).hexdigest(), + 'bindings':bindings,'secret_values_read':False,'backup_captured':False,'uploaded':False, + 'runtime_changed':False,'credential_grant_created':False, + 'owner_admission_required':profile['admission_gates']} + + +def main(): + p=argparse.ArgumentParser(description=__doc__);p.add_argument('--receipt',required=True) + a=p.parse_args() + try: + r=prepare(json.loads(PROFILE.read_text())) + fd=os.open(a.receipt,os.O_WRONLY|os.O_CREAT|os.O_EXCL|os.O_NOFOLLOW,0o600) + with os.fdopen(fd,'w') as out:json.dump(r,out,indent=2);out.write('\n') + except Exception: + print('ERROR: coverage preparation refused; private diagnostics suppressed.');return 1 + print('OK: exact coverage proposal prepared; capture, delivery and owner admission remain pending.');return 0 + + +if __name__=='__main__':raise SystemExit(main()) diff --git a/workplans/NK-WP-0049-bootstrap-secret-free-recovery-handoff.md b/workplans/NK-WP-0049-bootstrap-secret-free-recovery-handoff.md index b6857ee..5134049 100644 --- a/workplans/NK-WP-0049-bootstrap-secret-free-recovery-handoff.md +++ b/workplans/NK-WP-0049-bootstrap-secret-free-recovery-handoff.md @@ -476,6 +476,30 @@ recovery preparation; full matching identity key/config coverage, service continuity, T02 owner acceptance and legacy credential disposition remain open. Evidence: `privacyidea-transfer-r1.json`, `database-drill-preparation.json`. +2026-10-16 actual recovery and next coverage: the operator ran the prepared +hidden-key command successfully. Reviewed the actual receipt against exact +source/image/transfer digest and rehashed retained ciphertext. Original off-host +PI ciphertext decrypted, unrelated wrong key refused, native PostgreSQL restore +and aggregate readback passed (5 tokens, 2 resolvers, 2 realms), container and +plaintext cleanup passed; no NK49 Docker containers remain. The agent received +no private key or restored values. Full identity/token decryption/audit continuity +remain unproven; this is actual database recovery, not full service recovery. + +The operator confirmed no known full identity bundle and requested preparation +of missing coverage (decision `ed20023f-0168-421d-be86-460391339876`). Prepared +`identity-backup-profile.json` and a metadata-only preparer. Current exact four +Deployment and five Bound PVC identities and all Secret/ConfigMap references +were checked on the primary cluster without Secret-value reads or pod exec. +Fourteen source tests pass for drift/unreviewed dependencies/private-data scope. +The profile specifies consistent data snapshots, original key/config capture, +credential recovery/reissue disposition, proposed primary/secondary archive +paths, per-generation lineage, immutable readback, measured size/quota, eventual +schedule/retention and native service acceptance. It is `prepared_not_admitted`: +new capture/sealing/transfer adapters and scoped publisher owner admission remain +necessary. No capture/upload/grant/schedule/rotation occurred in preparation. +T02 remains progress and T03 wait. Evidence: actual database drill/review and +`identity-backup-preparation-r1.json` under the October15 recovery evidence. + ## Verify owner-approved recovery and current native callers ```task @@ -483,7 +507,7 @@ id: NK-WP-0049-T03 status: wait priority: high depends_on: [NK-WP-0049-T02] -blocking_reason: "A recent off-host PI database ciphertext is identified, but matching full identity data/key/config coverage and actual independent-key decryption remain unverified after bounded Storage Share/Scaleway inventory. Requires completed T02 owner/runtime and legacy-credential disposition, then the exact reviewed isolated recovery ceremony and native consumer acceptance. No live bootstrap/root retrieval or deferred exposure rotation is authorized by source tests." +blocking_reason: "The actual off-host PI database decrypted and restored with the independently held key; wrong-key refusal and cleanup passed. Operator reports no known full identity bundle and requested missing coverage preparation. Exact current workload/PVC/reference profile is prepared, but full data/key/config capture, new provider profile/publisher admission and native token/audit/directory/issuer recovery remain open, along with T02 current-owner and legacy-credential disposition. No live bootstrap/root retrieval or deferred exposure rotation is authorized." state_hub_task_id: "d68cd4d3-1ba3-530e-88f9-26ebb8dd6eac" ```