coulomb/net-kingdom
5
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Record off-host database recovery and prepare identity backup coverage
All checks were successful
CI Smoke / host-smoke (push) Successful in 0s
Details
CI Smoke / container-smoke (push) Successful in 1s
Details

Browse source 
Assistant: codex
Assistant-Model: gpt-6-astra
Assistant-Session: 01a13eab-1e6a-7b33-9c37-372e90ada3a2
This commit is contained in:
tegwick 2026-10-16 00:32:50 +02:00
parent a1939a7d06
commit dca999222f
13 changed files with 596 additions and 29 deletions
Download patch file Download diff file Expand all files Collapse all files

85
docs/bootstrap-recovery-ceremony.md
View file

@ -1,7 +1,8 @@
# NK-WP-0049 recovery ceremony and closure packet
Status: preparation; no live restore, credential read/grant, revocation, rotation,
root retrieval or production bootstrap is authorized by this document.
Status: actual off-host privacyIDEA database drill passed; full identity backup
coverage remains in preparation. This document grants no credential export,
provider access, revocation, rotation, root retrieval or production bootstrap.
## What has been established
@ -41,12 +42,12 @@ owner custody and pass the decryption key through attended temporary unlock.
| Current consumer source | Reconcile the deployed revisions with the repaired caller/provider source. The October 14 reserved platform-group fix must be deployed and independently checked before claiming live enforcement. |
| Legacy credential disposition | IAM owner identifies the exact historical personal-token/trigger principals, actual consumers and revocation method before any deletion. Deleting a Secret does not revoke an issued provider credential. |
The operator subsequently reported holding the `railiance-backup` age key.
Key material has not been received or read. Exact artifact/manifest selection
and verification that this independently held key matches the selected
ciphertext remain pending. The backup lane recipient and bootstrap
`keys/age.pub` must not be assumed to be the same.
Do not draft a blanket restore or grant approval while these remain unknown.
The operator supplied the independently held `railiance-backup` age key only to
the attended local custody helper. The actual off-host PI ciphertext decrypted
and restored successfully; the agent received no private key material. The full
identity asset manifest remains unfulfilled. The backup lane recipient and
bootstrap `keys/age.pub` must not be assumed to be the same.
Do not draft a blanket restore or grant approval while full coverage is unknown.
Once fixed, bind the source revisions, target identities, allowed operations,
recovery/cleanup sequence and maintenance window to the reviewed live packet.
Existing eligibility or another client's approval is not this ceremony's grant.
@ -137,8 +138,8 @@ self-revoked. Existing scoped backup credentials were reused without changes;
the Scaleway bootstrap API key was not used.
The operator subsequently reported the hidden-terminal check passed and the
temporary key file was removed. This is operator-reported recipient matching;
the private key was not provided to the agent. Actual decryption remains pending.
temporary key file was removed. The later actual database drill also passed;
the private key was not provided to the agent.
## Prepared bounded privacyIDEA database drill — 2026-10-16
@ -164,7 +165,9 @@ ciphertext remains for reproducibility. This proves database recovery only;
token decryption, audit signature continuity and full identity service recovery
require matching key/config files and further native owner checks.
Run from the repository in the operator's own terminal. Paste the private key
Executed successfully on 2026-10-16 (local time). The command is retained for
provenance; its r1 receipt now exists and must not be overwritten or rerun merely
to repeat evidence. Run future reviewed drills with a new receipt. Paste the key
only into the hidden prompt, then share the success/error line or safe JSON
receipt. The receipt destination must be new; do not overwrite a failed receipt.
@ -183,3 +186,63 @@ refusal, native restore/readback and cleanup. Those tests use synthetic data and
fresh fixture keys. They are not evidence of actual backup decryption. If the
real drill fails, review its closed receipt; do not expose PostgreSQL/age stderr
or dump contents in chat. T02/T03 remain open.
## Actual database recovery result — 2026-10-16
The operator reported success and the actual receipt was reviewed against the
exact checker source, pinned image and fetched ciphertext SHA256. The retained
ciphertext digest was independently rechecked. Decryption, unrelated wrong-key
refusal, native PostgreSQL restore, fixed aggregate readback and container/plaintext
cleanup all passed. The restored database has five tokens, two resolvers and
two realms. The local Docker inventory contains no remaining NK49 containers.
Evidence: `privacyidea-database-drill-r1.json` and `database-drill-review.json`
in the October15 evidence directory. No production connection occurred.
This proves recovery of the selected database using the operator's independently
held key. It does not prove the recoverability of token plaintext, old audit
signatures, directory accounts, issuer keys or a full identity environment.
## Missing coverage prepared at the operator's request
The operator confirmed there is no known full identity bundle and requested
preparation of the missing coverage. The machine-readable proposal is
[`identity-backup-profile.json`](../tools/bootstrap-recovery/identity-backup-profile.json).
The read-only preparer checked the primary cluster UID, four current Deployment
UIDs/readiness/observed generations, five Bound PVC UIDs and the complete current
Secret/ConfigMap reference sets. It captured names and identities only; it did
not read Secret values or execute in pods. Its exact result is
`identity-backup-preparation-r1.json`. Fourteen tests cover foreign/stale targets,
new unreviewed dependencies, projected Secrets and private-value suppression.
| Component | Required new coverage | Required recovery proof |
| --- | --- | --- |
| LLDAP | Consistent `users.db` snapshot, remaining durable data and current credentials/configuration | User, group and membership preservation; existing-user authentication |
| Authelia | Consistent storage DB, original encryption/OIDC/session keys and current configuration | Storage decryption, signing identity and isolated authentication |
| privacyIDEA | Original enckey plus matching audit pair, configuration/scripts and a linked native database dump | Existing token decryption/MFA, old audit signature verification and new isolated audit entry |
| KeyCape | Current registry/config and signing key, policy/receipts, every referenced credential's recovery or independent reissue disposition | Issuer/JWKS identity, registry/policy continuity and positive/negative consumer acceptance |
The proposed primary prefix is
`platform-pg/application-archives/net-kingdom/`; the proposed secondary folder is
`Backup/railiance-backups/net-kingdom-identity/`. These are **not admitted**.
Platform's current generic application archive adapter only admits
`coulomb-social` and `email-connect`, and their publisher custody profiles are
application-bound. A new NetKingdom profile and scoped publisher require owner
admission; existing credentials must not be borrowed. The operator's Scaleway
bootstrap API key is unnecessary for preparation.
The initial goal is one verified generation, with a measured encrypted size and
fresh provider quota/upload-overlap checks. A proposed 512 MiB ceiling is a
refusal bound, not evidence that the data fits the secondary budget. Seven daily
and two weekly points remain a later measured proposal. No retention deletion
or recurring schedule was installed. Capture must preserve original keys and
use SQLite online-backup/PostgreSQL logical snapshots, record each capture
interval and refuse changed source identities/configuration. It must not claim
global atomicity or substitute a fresh export for the already recovered old dump.
Implementation remains: owner-admitted capture/sealing and new-profile transfer
adapters, exact credential disposition, immutable create plus retrieved readback,
then the isolated full-service recovery checks above. The current bootstrap
secrets-only tar checker cannot verify this new data-bearing profile. Approval
of preparation does not make these adapters implemented or the proposed
destinations active. The profile lists these gates explicitly; no live grant or
deployment approval is requested from this incomplete execution packet.

16
docs/evidence/2026-10-15-bootstrap-recovery/caller-source-inventory.json
View file

@ -1,8 +1,8 @@
{
"schema": "netkingdom.bootstrap-caller-inventory.v1",
"observed_at": "2026-10-15T22:08:27.470032+00:00",
"scope": "all current .sh/.py non-test files under sso-mfa/bootstrap and sso-mfa/k8s, Makefile, console, delivery copier and recovery transport tool plus two bounded provider metadata inventory helpers plus bounded fixed-object ciphertext fetch and isolated PostgreSQL recovery check",
"entry_count": 91,
"observed_at": "2026-10-15T22:29:09.131605+00:00",
"scope": "all current .sh/.py non-test files under sso-mfa/bootstrap and sso-mfa/k8s, Makefile, console, delivery copier and recovery transport tool plus two bounded provider metadata inventory helpers plus bounded fixed-object ciphertext fetch and isolated PostgreSQL recovery check plus proposed full identity backup profile and metadata-only preparer",
"entry_count": 93,
"entries": [
{
"path": "Makefile",
@ -458,6 +458,16 @@
"path": "tools/bootstrap-recovery/check-privacyidea-database.py",
"sha256": "90f454c87998d7f5d625e3c6656c3418f8f56275dc1879a9d19ed605431b94a4",
"classification": "bounded off-host transfer / isolated database drill; actual recovery acceptance requires operator execution"
},
{
"path": "tools/bootstrap-recovery/prepare-identity-coverage.py",
"sha256": "cd5f3067d28b4c7cc93ed08b8cf3869218bdecf7c54b5dffed22d697de71dbb1",
"classification": "missing identity backup coverage proposal and metadata-only preparation; no live capture/grant/delivery capability"
},
{
"path": "tools/bootstrap-recovery/identity-backup-profile.json",
"sha256": "176c1f2f9111c2c51503212deb162274609d5e53bfe03d7319e19e44c5e09500",
"classification": "missing identity backup coverage proposal and metadata-only preparation; no live capture/grant/delivery capability"
}
],
"source_inventory_complete_for_declared_scope": true,

7
docs/evidence/2026-10-15-bootstrap-recovery/database-drill-preparation.json
View file

@ -2,7 +2,7 @@
"observed_at": "2026-10-15T22:08:27.470032+00:00",
"operator_reported_key_check_passed": true,
"private_key_received_by_agent": false,
"actual_ciphertext_decryption_verified": false,
"actual_ciphertext_decryption_verified": true,
"transfer_receipt": "privacyidea-transfer-r1.json",
"transfer_envelope_exit_code": 0,
"transfer_session_revoked_and_cleaned": true,
@ -13,7 +13,7 @@
"data": "synthetic database and fresh disposable age keys only",
"includes_native_postgres_restore_and_cleanup": true
},
"pending_human_command": "docs/bootstrap-recovery-ceremony.md: Prepared bounded privacyIDEA database drill",
"pending_human_command": null,
"native_identity_recovery_proven": false,
"sources": [
{
@ -28,5 +28,6 @@
"path": "tests/test_privacyidea_offhost_recovery.py",
"sha256": "ea977b0d3368fb14f6a7a252c24cc5cb9915022afa3dca61c80952e7489bd0e5"
}
]
],
"execution_receipt": "privacyidea-database-drill-r1.json"
}

15
docs/evidence/2026-10-15-bootstrap-recovery/database-drill-review.json Normal file
View file

@ -0,0 +1,15 @@
{
"observed_at": "2026-10-15T22:24:49.717396+00:00",
"receipt": "privacyidea-database-drill-r1.json",
"receipt_sha256": "c72dc3ababcc3ef3d7485a434128daf71e0ee58a8118e633fb09fed516f4b980",
"source_and_transfer_binding_verified": true,
"retained_ciphertext_digest_rechecked": true,
"operator_reported_success": true,
"independent_custody_basis": "Operator reports independently held railiance-backup key; attended temporary-key drill decrypted the fetched ciphertext. Agent never received private key material.",
"docker_nk49_container_inventory_empty": true,
"full_identity_recovery_proven": false,
"operator_reports_no_known_full_identity_bundle": true,
"next_work_authorized": "Prepare missing backup coverage; no blanket production restore or credential rotation.",
"coverage_preparation": "identity-backup-preparation-r1.json",
"operator_coverage_decision_id": "ed20023f-0168-421d-be86-460391339876"
}

106
docs/evidence/2026-10-15-bootstrap-recovery/identity-backup-preparation-r1.json Normal file
View file

@ -0,0 +1,106 @@
{
"schema": "netkingdom.identity-backup-preparation.v1",
"status": "prepared_not_admitted",
"observed_at": "2026-10-15T22:32:41.149409+00:00",
"cluster_uid": "a553c742-0115-43d4-99a4-a5ca56fe0786",
"profile_sha256": "176c1f2f9111c2c51503212deb162274609d5e53bfe03d7319e19e44c5e09500",
"source_sha256": "cd5f3067d28b4c7cc93ed08b8cf3869218bdecf7c54b5dffed22d697de71dbb1",
"bindings": [
{
"namespace": "sso",
"deployment": "lldap",
"uid": "743a86bd-5cb0-4ca0-9c56-a3438f4746a8",
"generation": 7,
"pvc_bindings": [
{
"name": "lldap-data",
"uid": "0a06b65a-e7b3-42d3-916e-e74dbe5fc7da"
}
],
"secret_names_only": [
"lldap-secrets"
],
"configmap_names_only": []
},
{
"namespace": "sso",
"deployment": "authelia",
"uid": "dc40a925-2306-4a3b-89c0-e714187503c3",
"generation": 28,
"pvc_bindings": [
{
"name": "authelia-data",
"uid": "86826554-b014-4df4-8b28-866aa64f6413"
}
],
"secret_names_only": [
"authelia-secrets",
"authelia-smtp"
],
"configmap_names_only": [
"authelia-config"
]
},
{
"namespace": "sso",
"deployment": "keycape",
"uid": "99ddd83c-cb3f-4847-bcf8-35f1aa87627f",
"generation": 65,
"pvc_bindings": [
{
"name": "keycape-authentication-policy",
"uid": "423f558f-8c11-4edf-956b-243559df4a8e"
}
],
"secret_names_only": [
"keycape-approval-engine-operator-client",
"keycape-config",
"keycape-factor-read",
"keycape-informed-decision-sitting-requester-client",
"keycape-rapp-qonto-client",
"keycape-secrets-engine-approval-client",
"keycape-secrets-engine-openbao-client",
"keycape-secrets-engine-requester-client"
],
"configmap_names_only": []
},
{
"namespace": "mfa",
"deployment": "privacyidea",
"uid": "58c7f96d-61cb-4dd4-bca2-54661c0ac375",
"generation": 11,
"pvc_bindings": [
{
"name": "privacyidea-data",
"uid": "65650fc1-02d8-4c0f-84fc-e0015c8f3ed9"
},
{
"name": "privacyidea-logs",
"uid": "39fa90d9-bc54-4a03-a0d6-ae85165e0cb9"
}
],
"secret_names_only": [
"privacyidea-config"
],
"configmap_names_only": [
"factor-recovery-01119187d90d7b76",
"keycape-onboarding-9465e5e0b98f7986",
"privacyidea-cfg"
]
}
],
"secret_values_read": false,
"backup_captured": false,
"uploaded": false,
"runtime_changed": false,
"credential_grant_created": false,
"owner_admission_required": [
"Railiance owner accepts exact read/capture executor, immutable helper images, PVC access and task-scoped caller attribution; no production stop/restart inferred",
"NetKingdom/service owners accept full asset manifest, original key preservation, consistency interval, bounded recovery environment and native acceptance",
"Platform/reef-storage admits both new archive prefixes, immutable create/readback, scoped publisher custody and current quota/headroom; current application archive profiles do not include net-kingdom",
"Resolve every referenced credential through encrypted recovery or an independent owner reissue path; no undocumented grant propagation",
"Implement and test capture/sealing/transfer adapters against this new profile before live execution; existing bootstrap tar checker is insufficient",
"Retrieve by immutable object/version from each admitted provider; digest check, attended independent-key decryption, native service recovery, failed-key case and verified cleanup",
"Only then enable recurring capture/alerts and measured retention; preserve T02 legacy provider-credential and deferred-rotation boundaries"
]
}

9
docs/evidence/2026-10-15-bootstrap-recovery/offsite-discovery-summary.json
View file

@ -2,7 +2,7 @@
"observed_at": "2026-10-15T22:08:27.470032+00:00",
"operator_reports_holding_railiance_backup_age_key": true,
"private_key_received_or_read": false,
"candidate_key_match_verified": false,
"candidate_key_match_verified": true,
"storage_share": {
"inventory": "storage-share-inventory-r1.json",
"status": "metadata_inventory_complete",
@ -14,7 +14,9 @@
"bootstrap_identity_file_bundle_found_in_inspected_scope": false,
"transfer_receipt": "privacyidea-transfer-r1.json",
"attended_transfer_envelope_exit_code": 0,
"transfer_session_self_revocation_and_cleanup_confirmed": true
"transfer_session_self_revocation_and_cleanup_confirmed": true,
"candidate_decryption_and_database_restore_verified": true,
"database_drill_receipt": "privacyidea-database-drill-r1.json"
},
"scaleway": {
"inventory": "scaleway-inventory-r1.json",
@ -38,5 +40,6 @@
"command": "python3 -m pytest -q tests/test_scaleway_backup_inventory.py tests/test_storage_share_inventory.py"
},
"native_recovery_proven": false,
"production_mutations_performed": false
"production_mutations_performed": false,
"private_key_received_or_read_scope": "Agent custody: false; attended human recovery child used the temporary explicit key."
}

28
docs/evidence/2026-10-15-bootstrap-recovery/privacyidea-database-drill-r1.json Normal file
View file

@ -0,0 +1,28 @@
{
"schema": "netkingdom.privacyidea-database-drill.v1",
"status": "isolated_database_drill_passed",
"started_at": "2026-10-15T22:18:55.858354+00:00",
"source_sha256": "90f454c87998d7f5d625e3c6656c3418f8f56275dc1879a9d19ed605431b94a4",
"production_contacted": false,
"secret_values_exported": false,
"native_identity_recovery_proven": false,
"token_decryption_proven": false,
"audit_signature_continuity_proven": false,
"full_identity_file_coverage_proven": false,
"container_cleanup_verified": true,
"temporary_plaintext_cleanup_verified": true,
"ciphertext_sha256": "cc81da442755928171d2c51bce93eda7de4392a0a1aa9d0ffb83a25ba34e5ead",
"registered_recipient_match": true,
"wrong_key_refused": true,
"offhost_ciphertext_decryption_verified": true,
"image": "postgres@sha256:ff23cdce56cac62ada6f66013e1a50864c0abbe79d132d40b6e05bd80f378a70",
"image_id": "sha256:ff23cdce56cac62ada6f66013e1a50864c0abbe79d132d40b6e05bd80f378a70",
"pg_restore_version": "pg_restore (PostgreSQL) 16.14 (Debian 16.14-1.pgdg13+1)",
"aggregate_row_counts": {
"token": 5,
"resolver": 2,
"realm": 2
},
"database_restore_and_readback_passed": true,
"finished_at": "2026-10-15T22:19:05.279736+00:00"
}

24
docs/evidence/2026-10-15-bootstrap-recovery/readiness.json
View file

@ -1,13 +1,13 @@
{
"workplan": "NK-WP-0049",
"observed_at": "2026-10-15T22:08:27.470032+00:00",
"status": "source_prepared_pending_owner_inputs_and_native_acceptance",
"observed_at": "2026-10-15T22:24:49.717396+00:00",
"status": "offhost_privacyidea_database_recovered_full_identity_coverage_open",
"backup_destinations_operator_reported": [
"Scaleway",
"Storage Share"
],
"exact_offhost_artifact_confirmed": true,
"independently_held_matching_key_confirmed": false,
"independently_held_matching_key_confirmed": true,
"recovery_key_or_backup_contents_retrieved": true,
"legacy_token_secret_present": true,
"legacy_trigger_secret_present": true,
@ -18,15 +18,14 @@
"verification": "source-verification.json",
"ceremony": "../../bootstrap-recovery-ceremony.md",
"outstanding": [
"Matching off-host identity data/key/config set remains unlocated in inspected scopes; selected existing PI database ciphertext is now fetched and digest-bound",
"Operator reports matching-recipient key check passed; actual ciphertext decryption/native database drill is prepared and pending operator hidden-key invocation",
"Full identity service recovery, token decryption, audit signature continuity and cleanup evidence",
"Prepare and admit missing encrypted identity data/key/config backup coverage; operator confirms no known full identity bundle",
"Retrieve new off-host bundle and prove original token decryption, audit continuity, directory and issuer recovery with safe negative cases and cleanup",
"T02 current caller/provider source admission and exact legacy provider credential disposition; retain existing deferred-rotation boundary"
],
"operator_reports_holding_railiance_backup_age_key": true,
"operator_report_recorded_at": "2026-10-15T21:52:22.549430+00:00",
"key_material_received": false,
"key_matches_selected_ciphertext_verified": false,
"key_matches_selected_ciphertext_verified": true,
"offsite_discovery": "offsite-discovery-summary.json",
"privacyidea_database_ciphertext_identified": true,
"required_full_asset_coverage_confirmed": false,
@ -36,5 +35,14 @@
"ciphertext_fetched": true,
"transfer_receipt": "privacyidea-transfer-r1.json",
"prepared_database_drill": "../../bootstrap-recovery-ceremony.md",
"age_private_key_retrieved_by_agent": false
"age_private_key_retrieved_by_agent": false,
"offhost_database_restore_proven": true,
"database_drill_receipt": "privacyidea-database-drill-r1.json",
"database_drill_review": "database-drill-review.json",
"operator_reports_no_known_full_identity_bundle": true,
"identity_backup_profile": "../../../tools/bootstrap-recovery/identity-backup-profile.json",
"identity_coverage_preparation": "identity-backup-preparation-r1.json",
"coverage_preparation_status": "prepared_not_admitted",
"coverage_preparation_tests_passed": 14,
"coverage_decision_id": "ed20023f-0168-421d-be86-460391339876"
}

54
tests/test_identity_backup_preparation.py Normal file
View file

@ -0,0 +1,54 @@
"""Preparation must never promote stale metadata or expand its data-read scope."""
import importlib.util
import json
from pathlib import Path
import pytest
ROOT=Path(__file__).resolve().parents[1]
spec=importlib.util.spec_from_file_location('prepare',ROOT/'tools/bootstrap-recovery/prepare-identity-coverage.py')
m=importlib.util.module_from_spec(spec);spec.loader.exec_module(m)
def fixture():
profile=json.loads(m.PROFILE.read_text());objects={}
objects[(None,'namespace','kube-system')]={'metadata':{'uid':m.CLUSTER_UID}}
for c in profile['components']:
ns=c['namespace'];name=c['deployment']
objects[(ns,'deployment',name)]={'metadata':{'uid':c['deployment_uid'],'generation':9},
'status':{'observedGeneration':9,'readyReplicas':1},'spec':{'template':{'spec':{
'containers':[{'name':'service','env':[{'name':'DO_NOT_EXPORT','value':'private-value-canary'}],
'envFrom':[{'secretRef':{'name':n}} for n in c['secret_refs']]}],
'volumes':[{'configMap':{'name':n}} for n in c['configmap_refs']]+
[{'persistentVolumeClaim':{'claimName':p['name']}} for p in c['pvc_refs']]}}}}
for pvc in c['pvc_refs']:objects[(ns,'pvc',pvc['name'])]={'metadata':{'uid':pvc['uid']},'status':{'phase':'Bound'}}
calls=[]
def get(ns,kind,name):calls.append((ns,kind,name));return objects[(ns,kind,name)]
return profile,objects,calls,get
def test_only_safe_metadata_leaves_preparation_and_no_secret_api_is_called():
p,objects,calls,get=fixture();r=m.prepare(p,get)
assert r['status']=='prepared_not_admitted'
assert not any(r[k] for k in ['secret_values_read','backup_captured','uploaded','runtime_changed','credential_grant_created'])
assert 'private-value-canary' not in json.dumps(r)
assert all(kind in {'namespace','deployment','pvc'} for ns,kind,name in calls)
@pytest.mark.parametrize('change',['cluster','pvc','deployment','not-ready','unobserved','terminating','new-secret','projected-secret','new-pvc'])
def test_stale_or_expanded_live_state_requires_fresh_owner_review(change):
p,o,calls,get=fixture();d=o[('sso','deployment','lldap')];s=d['spec']['template']['spec']
if change=='cluster':o[(None,'namespace','kube-system')]['metadata']['uid']='other-cluster'
if change=='pvc':o[('sso','pvc','lldap-data')]['metadata']['uid']='replacement'
if change=='deployment':d['metadata']['uid']='replacement'
if change=='not-ready':d['status']['readyReplicas']=0
if change=='unobserved':d['status']['observedGeneration']=8
if change=='terminating':d['metadata']['deletionTimestamp']='2026-10-16'
if change=='new-secret':s['containers'][0]['envFrom'].append({'secretRef':{'name':'unreviewed-secret'}})
if change=='projected-secret':s['volumes'].append({'projected':{'sources':[{'secret':{'name':'unreviewed-secret'}}]}})
if change=='new-pvc':s['volumes'].append({'persistentVolumeClaim':{'claimName':'unreviewed-pvc'}})
with pytest.raises(m.Refused):m.prepare(p,get)
@pytest.mark.parametrize('kind',['secret','configmap','pods/exec','clusterrolebinding'])
def test_preparation_transport_has_no_private_data_or_mutation_route(kind):
with pytest.raises(m.Refused):m.query('sso',kind,'anything')

9
tools/bootstrap-recovery/README.md
View file

@ -52,3 +52,12 @@ NK49_RUN_DISPOSABLE_POSTGRES_TEST=1 python3 -m pytest -q \
tests/test_privacyidea_offhost_recovery.py \
tests/test_storage_share_inventory.py tests/test_scaleway_backup_inventory.py
```
`identity-backup-profile.json` prepares missing full service coverage and names
the exact current workload/PVC/credential dependencies, proposed provider paths
and remaining implementation/admission gates. It is a proposal, not a new live
publisher grant. `prepare-identity-coverage.py --receipt NEW_PATH` verifies the
primary cluster and current deployment/PVC/reference bindings using metadata
only. It refuses replacement resources, unobserved/unready deployments and new
unreviewed references. It has no Secret-data, pod-exec, mutation or upload route.
Its receipt says `prepared_not_admitted`; it cannot prove a backup exists.

152
tools/bootstrap-recovery/identity-backup-profile.json Normal file
View file

@ -0,0 +1,152 @@
{
"schema": "netkingdom.identity-backup-profile.v1",
"status": "proposed",
"owner": "net-kingdom with railiance-platform/rapp-postgres runtime and storage admission",
"public_recipient": "age1zvryunvjhvpkmasskauga2heeg0ztnte9ymgppvjge36ekumk50syr3tsz",
"archive_profile": "netkingdom-identity-v1; new profile, not the bootstrap secrets-only tar profile",
"components": [
{
"namespace": "sso",
"deployment": "lldap",
"deployment_uid": "743a86bd-5cb0-4ca0-9c56-a3438f4746a8",
"pvc_refs": [
{
"name": "lldap-data",
"uid": "0a06b65a-e7b3-42d3-916e-e74dbe5fc7da"
}
],
"secret_refs": [
"lldap-secrets"
],
"configmap_refs": [],
"required_assets": [
"SQLite online-backup snapshot of /data/users.db with integrity check",
"remaining non-backup durable /data files, including any private identity material",
"current deployment configuration and exact lldap-secrets fields"
],
"native_acceptance": [
"directory user/group/membership preservation",
"existing-user password authentication in isolated environment"
]
},
{
"namespace": "sso",
"deployment": "authelia",
"deployment_uid": "dc40a925-2306-4a3b-89c0-e714187503c3",
"pvc_refs": [
{
"name": "authelia-data",
"uid": "86826554-b014-4df4-8b28-866aa64f6413"
}
],
"secret_refs": [
"authelia-secrets",
"authelia-smtp"
],
"configmap_refs": [
"authelia-config"
],
"required_assets": [
"SQLite online-backup snapshot of /var/authelia/data/db.sqlite3 with integrity check",
"current configuration.yml and all Secret-mounted encryption/OIDC/session fields",
"explicit disposition for notifier credentials and remaining durable files"
],
"native_acceptance": [
"storage decryption",
"same OIDC signing identity and preserved client configuration",
"isolated authentication flow with outbound notification disabled"
]
},
{
"namespace": "sso",
"deployment": "keycape",
"deployment_uid": "99ddd83c-cb3f-4847-bcf8-35f1aa87627f",
"pvc_refs": [
{
"name": "keycape-authentication-policy",
"uid": "423f558f-8c11-4edf-956b-243559df4a8e"
}
],
"secret_refs": [
"keycape-approval-engine-operator-client",
"keycape-config",
"keycape-factor-read",
"keycape-informed-decision-sitting-requester-client",
"keycape-rapp-qonto-client",
"keycape-secrets-engine-approval-client",
"keycape-secrets-engine-openbao-client",
"keycape-secrets-engine-requester-client"
],
"configmap_refs": [],
"required_assets": [
"exact current keycape-config config.yaml and key.pem",
"complete stable policy/receipt PVC snapshot",
"current deployment and owner disposition for every referenced credential: encrypted recovery or independently recoverable provider reissue path"
],
"native_acceptance": [
"same issuer/JWKS signing identity",
"registry, authentication policy and receipt continuity",
"current consumer positive/negative authentication under isolated test endpoints"
]
},
{
"namespace": "mfa",
"deployment": "privacyidea",
"deployment_uid": "58c7f96d-61cb-4dd4-bca2-54661c0ac375",
"pvc_refs": [
{
"name": "privacyidea-data",
"uid": "65650fc1-02d8-4c0f-84fc-e0015c8f3ed9"
},
{
"name": "privacyidea-logs",
"uid": "39fa90d9-bc54-4a03-a0d6-ae85165e0cb9"
}
],
"secret_refs": [
"privacyidea-config"
],
"configmap_refs": [
"factor-recovery-01119187d90d7b76",
"keycape-onboarding-9465e5e0b98f7986",
"privacyidea-cfg"
],
"required_assets": [
"original /etc/privacyidea/enckey, private.pem and public.pem with matching RSA pair validation",
"current privacyidea-config and mounted configuration/scripts",
"bounded native PostgreSQL custom dump with exact database identity and key-generation linkage",
"explicit disposition for service logs; audit database/signatures must be preserved"
],
"native_acceptance": [
"decrypt existing retained token material without exporting it",
"verify pre-backup audit signatures and create/verify a new isolated audit entry",
"existing-token MFA and negative-key refusal"
]
}
],
"capture_contract": {
"plaintext": "private task staging outside Git; never process output, State Hub or logs; verified cleanup after encryption",
"consistency": "native SQLite online-backup and PostgreSQL logical snapshot; explicit capture interval per asset; no global atomic snapshot claim; before/after source identities/config versions and directory inventories must match",
"key_policy": "preserve original service keys; no age private key in the payload, no replacement generation, no root/unseal collection",
"encryption": "public-recipient-only streaming archive to age; private fresh ciphertext, no overwrite; manifest inside ciphertext with source identities, times, complete required paths, digests and omissions",
"limits": "measure first; proposed 512 MiB encrypted admission ceiling, no silent truncation/exclusions",
"historical_database": "existing verified October15 dump remains protected; new bundle must link its own exact database/key capture generation and recovery receipt"
},
"proposed_destinations": {
"primary": "s3://railiance-platform-pg-backup/platform-pg/application-archives/net-kingdom/",
"secondary": "Storage Share Backup/railiance-backups/net-kingdom-identity/",
"admitted": false,
"publisher": "new exact prefix-scoped owner-delivered publisher; do not borrow another workload reader or bootstrap API key",
"retention": "initial single verified generation; later seven daily/two weekly only after measured quota/overlap/headroom and separately admitted retention executor; never expire existing good backups",
"schedule": "not installed; propose daily capture after logical database backup with per-generation linkage, separate failure alert and operator-approved RPO/RTO"
},
"admission_gates": [
"Railiance owner accepts exact read/capture executor, immutable helper images, PVC access and task-scoped caller attribution; no production stop/restart inferred",
"NetKingdom/service owners accept full asset manifest, original key preservation, consistency interval, bounded recovery environment and native acceptance",
"Platform/reef-storage admits both new archive prefixes, immutable create/readback, scoped publisher custody and current quota/headroom; current application archive profiles do not include net-kingdom",
"Resolve every referenced credential through encrypted recovery or an independent owner reissue path; no undocumented grant propagation",
"Implement and test capture/sealing/transfer adapters against this new profile before live execution; existing bootstrap tar checker is insufficient",
"Retrieve by immutable object/version from each admitted provider; digest check, attended independent-key decryption, native service recovery, failed-key case and verified cleanup",
"Only then enable recurring capture/alerts and measured retention; preserve T02 legacy provider-credential and deferred-rotation boundaries"
]
}

94
tools/bootstrap-recovery/prepare-identity-coverage.py Normal file
View file

@ -0,0 +1,94 @@
#!/usr/bin/env python3
"""Prepare exact identity backup coverage from read-only workload/PVC metadata.
This produces a proposal, never reads Secrets, executes in pods, changes cluster
resources, captures private files, uploads, schedules or grants credentials.
"""
import argparse
from datetime import datetime,timezone
import hashlib
import json
import os
from pathlib import Path
import subprocess
ROOT=Path(__file__).resolve().parents[2]
CLUSTER_UID='a553c742-0115-43d4-99a4-a5ca56fe0786'
PROFILE=ROOT/'tools/bootstrap-recovery/identity-backup-profile.json'
class Refused(Exception):pass
def query(namespace,kind,name):
# Closed read-only types; Secret/ConfigMap data and arbitrary API paths have
# no transport in this preparation command.
if kind not in {'namespace','deployment','pvc'}:raise Refused()
args=['kubectl','--request-timeout=20s']
if namespace:args+=['-n',namespace]
r=subprocess.run(args+['get',kind,name,'-o','json'],capture_output=True,timeout=25)
if r.returncode:raise Refused()
return json.loads(r.stdout)
def references(spec):
secrets=set();configmaps=set();pvcs=set()
for c in spec.get('containers',[])+spec.get('initContainers',[]):
for e in c.get('env',[]):
v=e.get('valueFrom',{})
if 'secretKeyRef' in v:secrets.add(v['secretKeyRef']['name'])
if 'configMapKeyRef' in v:configmaps.add(v['configMapKeyRef']['name'])
for e in c.get('envFrom',[]):
if 'secretRef' in e:secrets.add(e['secretRef']['name'])
if 'configMapRef' in e:configmaps.add(e['configMapRef']['name'])
for v in spec.get('volumes',[]):
if 'secret' in v:secrets.add(v['secret']['secretName'])
if 'configMap' in v:configmaps.add(v['configMap']['name'])
if 'persistentVolumeClaim' in v:pvcs.add(v['persistentVolumeClaim']['claimName'])
for p in v.get('projected',{}).get('sources',[]):
if 'secret' in p:secrets.add(p['secret']['name'])
if 'configMap' in p:configmaps.add(p['configMap']['name'])
return secrets,configmaps,pvcs
def prepare(profile,get=query):
if get(None,'namespace','kube-system')['metadata']['uid']!=CLUSTER_UID:raise Refused()
bindings=[]
for component in profile['components']:
ns=component['namespace'];name=component['deployment']
d=get(ns,'deployment',name);m=d['metadata']
if (m['uid']!=component['deployment_uid'] or m.get('deletionTimestamp') or
d['status'].get('observedGeneration')!=m['generation'] or d['status'].get('readyReplicas')!=1):raise Refused()
refs=references(d['spec']['template']['spec'])
expected_pvcs={p['name']:p['uid'] for p in component['pvc_refs']}
if refs[0]!=set(component['secret_refs']) or refs[1]!=set(component['configmap_refs']) or refs[2]!=set(expected_pvcs):raise Refused()
pvcs=[]
for pvc,expected in expected_pvcs.items():
obj=get(ns,'pvc',pvc)
if obj['metadata']['uid']!=expected or obj['metadata'].get('deletionTimestamp') or obj['status'].get('phase')!='Bound':raise Refused()
pvcs.append({'name':pvc,'uid':expected})
bindings.append({'namespace':ns,'deployment':name,'uid':m['uid'],'generation':m['generation'],
'pvc_bindings':pvcs,'secret_names_only':sorted(refs[0]),'configmap_names_only':sorted(refs[1])})
if get(None,'namespace','kube-system')['metadata']['uid']!=CLUSTER_UID:raise Refused()
return {'schema':'netkingdom.identity-backup-preparation.v1','status':'prepared_not_admitted',
'observed_at':datetime.now(timezone.utc).isoformat(),'cluster_uid':CLUSTER_UID,
'profile_sha256':hashlib.sha256(PROFILE.read_bytes()).hexdigest(),
'source_sha256':hashlib.sha256(Path(__file__).read_bytes()).hexdigest(),
'bindings':bindings,'secret_values_read':False,'backup_captured':False,'uploaded':False,
'runtime_changed':False,'credential_grant_created':False,
'owner_admission_required':profile['admission_gates']}
def main():
p=argparse.ArgumentParser(description=__doc__);p.add_argument('--receipt',required=True)
a=p.parse_args()
try:
r=prepare(json.loads(PROFILE.read_text()))
fd=os.open(a.receipt,os.O_WRONLY|os.O_CREAT|os.O_EXCL|os.O_NOFOLLOW,0o600)
with os.fdopen(fd,'w') as out:json.dump(r,out,indent=2);out.write('\n')
except Exception:
print('ERROR: coverage preparation refused; private diagnostics suppressed.');return 1
print('OK: exact coverage proposal prepared; capture, delivery and owner admission remain pending.');return 0
if __name__=='__main__':raise SystemExit(main())

26
workplans/NK-WP-0049-bootstrap-secret-free-recovery-handoff.md
View file

@ -476,6 +476,30 @@ recovery preparation; full matching identity key/config coverage, service
continuity, T02 owner acceptance and legacy credential disposition remain open.
Evidence: `privacyidea-transfer-r1.json`, `database-drill-preparation.json`.
2026-10-16 actual recovery and next coverage: the operator ran the prepared
hidden-key command successfully. Reviewed the actual receipt against exact
source/image/transfer digest and rehashed retained ciphertext. Original off-host
PI ciphertext decrypted, unrelated wrong key refused, native PostgreSQL restore
and aggregate readback passed (5 tokens, 2 resolvers, 2 realms), container and
plaintext cleanup passed; no NK49 Docker containers remain. The agent received
no private key or restored values. Full identity/token decryption/audit continuity
remain unproven; this is actual database recovery, not full service recovery.
The operator confirmed no known full identity bundle and requested preparation
of missing coverage (decision `ed20023f-0168-421d-be86-460391339876`). Prepared
`identity-backup-profile.json` and a metadata-only preparer. Current exact four
Deployment and five Bound PVC identities and all Secret/ConfigMap references
were checked on the primary cluster without Secret-value reads or pod exec.
Fourteen source tests pass for drift/unreviewed dependencies/private-data scope.
The profile specifies consistent data snapshots, original key/config capture,
credential recovery/reissue disposition, proposed primary/secondary archive
paths, per-generation lineage, immutable readback, measured size/quota, eventual
schedule/retention and native service acceptance. It is `prepared_not_admitted`:
new capture/sealing/transfer adapters and scoped publisher owner admission remain
necessary. No capture/upload/grant/schedule/rotation occurred in preparation.
T02 remains progress and T03 wait. Evidence: actual database drill/review and
`identity-backup-preparation-r1.json` under the October15 recovery evidence.
## Verify owner-approved recovery and current native callers
```task
@ -483,7 +507,7 @@ id: NK-WP-0049-T03
status: wait
priority: high
depends_on: [NK-WP-0049-T02]
blocking_reason: "A recent off-host PI database ciphertext is identified, but matching full identity data/key/config coverage and actual independent-key decryption remain unverified after bounded Storage Share/Scaleway inventory. Requires completed T02 owner/runtime and legacy-credential disposition, then the exact reviewed isolated recovery ceremony and native consumer acceptance. No live bootstrap/root retrieval or deferred exposure rotation is authorized by source tests."
blocking_reason: "The actual off-host PI database decrypted and restored with the independently held key; wrong-key refusal and cleanup passed. Operator reports no known full identity bundle and requested missing coverage preparation. Exact current workload/PVC/reference profile is prepared, but full data/key/config capture, new provider profile/publisher admission and native token/audit/directory/issuer recovery remain open, along with T02 current-owner and legacy-credential disposition. No live bootstrap/root retrieval or deferred exposure rotation is authorized."
state_hub_task_id: "d68cd4d3-1ba3-530e-88f9-26ebb8dd6eac"
```