diff --git a/docs/bootstrap-recovery-ceremony.md b/docs/bootstrap-recovery-ceremony.md index 6b89814..c206033 100644 --- a/docs/bootstrap-recovery-ceremony.md +++ b/docs/bootstrap-recovery-ceremony.md @@ -136,9 +136,50 @@ No object contents or age keys were read; the operator envelope exited0 and self-revoked. Existing scoped backup credentials were reused without changes; the Scaleway bootstrap API key was not used. -The next human act is to run the existing custody helper in their own terminal -with `--expected-recipient` set to the public recipient in the discovery summary, -`--prompt --check-only`. Paste private material only into the hidden prompt. -This verifies the independently held key against the backup owner's declared -recipient and removes its temporary copy. It does not by itself prove that the -selected backup decrypts or includes the full matching identity recovery set. +The operator subsequently reported the hidden-terminal check passed and the +temporary key file was removed. This is operator-reported recipient matching; +the private key was not provided to the agent. Actual decryption remains pending. + +## Prepared bounded privacyIDEA database drill — 2026-10-16 + +The identified existing off-host ciphertext was downloaded through the same +attended Platform Backup-account procedure. The before/GET/after strong-ETag +binding and exact byte count passed. Ciphertext SHA256: +`cc81da442755928171d2c51bce93eda7de4392a0a1aa9d0ffb83a25ba34e5ead`. +See `privacyidea-transfer-r1.json` in the October15 evidence folder. The owner +envelope exited0 and self-revoked; no age key or plaintext was read by the fetch. +Ciphertext is retained in the fresh private `/tmp/nk49-offhost-j9dzl0ge/` directory. + +The following exact human command performs only an isolated local database +drill. It checks the transfer receipt/digest before opening the temporary key, +checks the backup recipient, refuses an unrelated wrong key, decrypts to private +temporary storage and restores into disposable PostgreSQL. Image +`postgres@sha256:ff23cdce56cac62ada6f66013e1a50864c0abbe79d132d40b6e05bd80f378a70` +is already cached locally (pg_restore16.14). There is no image pull, network, +published port, host data mount, production connection or IAM/client execution. +Database storage is a bounded temporary memory filesystem. Only fixed aggregate +counts for `token`, `resolver`, and `realm` leave the database. The container, +volumes and temporary plaintext are removed and cleanup is checked. The original +ciphertext remains for reproducibility. This proves database recovery only; +token decryption, audit signature continuity and full identity service recovery +require matching key/config files and further native owner checks. + +Run from the repository in the operator's own terminal. Paste the private key +only into the hidden prompt, then share the success/error line or safe JSON +receipt. The receipt destination must be new; do not overwrite a failed receipt. + +```bash +cd /home/worsch/net-kingdom +bash sso-mfa/bootstrap/sops-custody-unlock.sh \ + --expected-recipient age1zvryunvjhvpkmasskauga2heeg0ztnte9ymgppvjge36ekumk50syr3tsz \ + --prompt -- python3 tools/bootstrap-recovery/check-privacyidea-database.py \ + --transfer-receipt docs/evidence/2026-10-15-bootstrap-recovery/privacyidea-transfer-r1.json \ + --receipt docs/evidence/2026-10-15-bootstrap-recovery/privacyidea-database-drill-r1.json +``` + +Source verification:23 focused checks passed, including a real local Docker +custom-format dump, age encryption, temporary fixture-key unlock, wrong-key +refusal, native restore/readback and cleanup. Those tests use synthetic data and +fresh fixture keys. They are not evidence of actual backup decryption. If the +real drill fails, review its closed receipt; do not expose PostgreSQL/age stderr +or dump contents in chat. T02/T03 remain open. diff --git a/docs/evidence/2026-10-15-bootstrap-recovery/caller-source-inventory.json b/docs/evidence/2026-10-15-bootstrap-recovery/caller-source-inventory.json index 825a132..0b25571 100644 --- a/docs/evidence/2026-10-15-bootstrap-recovery/caller-source-inventory.json +++ b/docs/evidence/2026-10-15-bootstrap-recovery/caller-source-inventory.json @@ -1,8 +1,8 @@ { "schema": "netkingdom.bootstrap-caller-inventory.v1", - "observed_at": "2026-10-15T20:54:09.856047+00:00", - "scope": "all current .sh/.py non-test files under sso-mfa/bootstrap and sso-mfa/k8s, Makefile, console, delivery copier and recovery transport tool plus two bounded provider metadata inventory helpers", - "entry_count": 89, + "observed_at": "2026-10-15T22:08:27.470032+00:00", + "scope": "all current .sh/.py non-test files under sso-mfa/bootstrap and sso-mfa/k8s, Makefile, console, delivery copier and recovery transport tool plus two bounded provider metadata inventory helpers plus bounded fixed-object ciphertext fetch and isolated PostgreSQL recovery check", + "entry_count": 91, "entries": [ { "path": "Makefile", @@ -431,7 +431,7 @@ }, { "path": "tools/bootstrap-recovery/inventory-storage-share.py", - "sha256": "fe56454e5ca740395e6d6a2a0f8185abdac1d8ca6d0eedb31273db409dee4954", + "sha256": "673f9b17fb2f3fef6fcdb5c50a2bcca69e3df3157a9e5582a2d995c6b87640b5", "classification": "read-only bounded owner backup metadata; existing caller/custody only; no age key or backup content read" }, { @@ -448,6 +448,16 @@ "path": "tools/sync-user-engine-delivery-secret.sh", "sha256": "7c0fb6b066e4ad8b103e75551beecd5de65a4bbb22570d95b42e5da6395e2dfc", "classification": "retired entry point; refuses before credential/cluster access" + }, + { + "path": "tools/bootstrap-recovery/fetch-privacyidea-backup.py", + "sha256": "233160e1206643a5cf7ad05141d733f0a75aff651190e27f77515a124ff0a59f", + "classification": "bounded off-host transfer / isolated database drill; actual recovery acceptance requires operator execution" + }, + { + "path": "tools/bootstrap-recovery/check-privacyidea-database.py", + "sha256": "90f454c87998d7f5d625e3c6656c3418f8f56275dc1879a9d19ed605431b94a4", + "classification": "bounded off-host transfer / isolated database drill; actual recovery acceptance requires operator execution" } ], "source_inventory_complete_for_declared_scope": true, diff --git a/docs/evidence/2026-10-15-bootstrap-recovery/database-drill-preparation.json b/docs/evidence/2026-10-15-bootstrap-recovery/database-drill-preparation.json new file mode 100644 index 0000000..1320105 --- /dev/null +++ b/docs/evidence/2026-10-15-bootstrap-recovery/database-drill-preparation.json @@ -0,0 +1,32 @@ +{ + "observed_at": "2026-10-15T22:08:27.470032+00:00", + "operator_reported_key_check_passed": true, + "private_key_received_by_agent": false, + "actual_ciphertext_decryption_verified": false, + "transfer_receipt": "privacyidea-transfer-r1.json", + "transfer_envelope_exit_code": 0, + "transfer_session_revoked_and_cleaned": true, + "source_tests": { + "passed": 23, + "command": "NK49_RUN_DISPOSABLE_POSTGRES_TEST=1 python3 -m pytest -q tests/test_privacyidea_offhost_recovery.py tests/test_storage_share_inventory.py tests/test_scaleway_backup_inventory.py", + "duration_seconds": 8.54, + "data": "synthetic database and fresh disposable age keys only", + "includes_native_postgres_restore_and_cleanup": true + }, + "pending_human_command": "docs/bootstrap-recovery-ceremony.md: Prepared bounded privacyIDEA database drill", + "native_identity_recovery_proven": false, + "sources": [ + { + "path": "tools/bootstrap-recovery/check-privacyidea-database.py", + "sha256": "90f454c87998d7f5d625e3c6656c3418f8f56275dc1879a9d19ed605431b94a4" + }, + { + "path": "tools/bootstrap-recovery/fetch-privacyidea-backup.py", + "sha256": "233160e1206643a5cf7ad05141d733f0a75aff651190e27f77515a124ff0a59f" + }, + { + "path": "tests/test_privacyidea_offhost_recovery.py", + "sha256": "ea977b0d3368fb14f6a7a252c24cc5cb9915022afa3dca61c80952e7489bd0e5" + } + ] +} diff --git a/docs/evidence/2026-10-15-bootstrap-recovery/offsite-discovery-summary.json b/docs/evidence/2026-10-15-bootstrap-recovery/offsite-discovery-summary.json index 3a4e8fd..1d29f3b 100644 --- a/docs/evidence/2026-10-15-bootstrap-recovery/offsite-discovery-summary.json +++ b/docs/evidence/2026-10-15-bootstrap-recovery/offsite-discovery-summary.json @@ -1,5 +1,5 @@ { - "observed_at": "2026-10-15T21:58:20.278564+00:00", + "observed_at": "2026-10-15T22:08:27.470032+00:00", "operator_reports_holding_railiance_backup_age_key": true, "private_key_received_or_read": false, "candidate_key_match_verified": false, @@ -10,8 +10,11 @@ "session_self_revocation_and_cleanup_confirmed": true, "selected_candidate": "r01-net-kingdom-pg/net-kingdom-pg-privacyidea_db-20261015T023000Z.dump.age", "candidate_bytes": 89129309, - "candidate_contents_downloaded": false, - "bootstrap_identity_file_bundle_found_in_inspected_scope": false + "candidate_contents_downloaded": true, + "bootstrap_identity_file_bundle_found_in_inspected_scope": false, + "transfer_receipt": "privacyidea-transfer-r1.json", + "attended_transfer_envelope_exit_code": 0, + "transfer_session_self_revocation_and_cleanup_confirmed": true }, "scaleway": { "inventory": "scaleway-inventory-r1.json", @@ -26,7 +29,9 @@ "source_sha256": "09f349ae815339732db505d6896fb1ca384be7c66b40574750a3ba30d4d922e0", "declared_default_public_recipient": "age1zvryunvjhvpkmasskauga2heeg0ztnte9ymgppvjge36ekumk50syr3tsz", "operator_hidden_terminal_check_requested": true, - "scope": "A successful key match checks the declared public recipient, not actual ciphertext decryption or whole-stack recovery." + "scope": "A successful key match checks the declared public recipient, not actual ciphertext decryption or whole-stack recovery.", + "operator_reports_check_passed": true, + "operator_reports_temporary_key_removed": true }, "source_tests": { "passed": 13, diff --git a/docs/evidence/2026-10-15-bootstrap-recovery/privacyidea-transfer-r1.json b/docs/evidence/2026-10-15-bootstrap-recovery/privacyidea-transfer-r1.json new file mode 100644 index 0000000..3ff5fb2 --- /dev/null +++ b/docs/evidence/2026-10-15-bootstrap-recovery/privacyidea-transfer-r1.json @@ -0,0 +1,19 @@ +{ + "schema": "netkingdom.privacyidea-offhost-transfer.v1", + "status": "ciphertext_fetched", + "started_at": "2026-10-15T22:07:08.615448+00:00", + "provider": "Storage Share", + "object": "r01-net-kingdom-pg/net-kingdom-pg-privacyidea_db-20261015T023000Z.dump.age", + "age_keys_read": false, + "plaintext_created": false, + "production_writes": false, + "source_sha256": "233160e1206643a5cf7ad05141d733f0a75aff651190e27f77515a124ff0a59f", + "owner_accessor_sha256": "673f9b17fb2f3fef6fcdb5c50a2bcca69e3df3157a9e5582a2d995c6b87640b5", + "caller_auth_path": "auth/netkingdom/oidc/callback", + "caller_entity_id": "4a1f663c-0621-0483-edd0-f852274f4b6b", + "ciphertext_bytes": 89129309, + "ciphertext_sha256": "cc81da442755928171d2c51bce93eda7de4392a0a1aa9d0ffb83a25ba34e5ead", + "object_unchanged_during_transfer": true, + "ciphertext_path": "/tmp/nk49-offhost-j9dzl0ge/privacyidea.dump.age", + "finished_at": "2026-10-15T22:07:16.363645+00:00" +} diff --git a/docs/evidence/2026-10-15-bootstrap-recovery/readiness.json b/docs/evidence/2026-10-15-bootstrap-recovery/readiness.json index 0bca3fb..572f096 100644 --- a/docs/evidence/2026-10-15-bootstrap-recovery/readiness.json +++ b/docs/evidence/2026-10-15-bootstrap-recovery/readiness.json @@ -1,14 +1,14 @@ { "workplan": "NK-WP-0049", - "observed_at": "2026-10-15T20:56:58.471680+00:00", + "observed_at": "2026-10-15T22:08:27.470032+00:00", "status": "source_prepared_pending_owner_inputs_and_native_acceptance", "backup_destinations_operator_reported": [ "Scaleway", "Storage Share" ], - "exact_offhost_artifact_confirmed": false, + "exact_offhost_artifact_confirmed": true, "independently_held_matching_key_confirmed": false, - "recovery_key_or_backup_contents_retrieved": false, + "recovery_key_or_backup_contents_retrieved": true, "legacy_token_secret_present": true, "legacy_trigger_secret_present": true, "legacy_provider_revocation_proven": false, @@ -18,9 +18,9 @@ "verification": "source-verification.json", "ceremony": "../../bootstrap-recovery-ceremony.md", "outstanding": [ - "Matching off-host identity data/key/config set is not located in inspected provider scopes; a recent PI database ciphertext is identified, not downloaded/decrypted", - "Independent age key availability/custody confirmation", - "Owner admitted isolated recovery target/procedure and native service continuity/cleanup evidence", + "Matching off-host identity data/key/config set remains unlocated in inspected scopes; selected existing PI database ciphertext is now fetched and digest-bound", + "Operator reports matching-recipient key check passed; actual ciphertext decryption/native database drill is prepared and pending operator hidden-key invocation", + "Full identity service recovery, token decryption, audit signature continuity and cleanup evidence", "T02 current caller/provider source admission and exact legacy provider credential disposition; retain existing deferred-rotation boundary" ], "operator_reports_holding_railiance_backup_age_key": true, @@ -29,5 +29,12 @@ "key_matches_selected_ciphertext_verified": false, "offsite_discovery": "offsite-discovery-summary.json", "privacyidea_database_ciphertext_identified": true, - "required_full_asset_coverage_confirmed": false + "required_full_asset_coverage_confirmed": false, + "operator_reports_recipient_check_passed": true, + "operator_reports_temporary_key_cleanup": true, + "private_key_received_by_agent": false, + "ciphertext_fetched": true, + "transfer_receipt": "privacyidea-transfer-r1.json", + "prepared_database_drill": "../../bootstrap-recovery-ceremony.md", + "age_private_key_retrieved_by_agent": false } diff --git a/tests/test_privacyidea_offhost_recovery.py b/tests/test_privacyidea_offhost_recovery.py new file mode 100644 index 0000000..76c87a3 --- /dev/null +++ b/tests/test_privacyidea_offhost_recovery.py @@ -0,0 +1,129 @@ +"""Transfer binding failures and a real disposable PostgreSQL fixture restore.""" +import hashlib +import importlib.util +import io +import json +import os +from pathlib import Path +import subprocess +import tempfile +import time +import uuid +import pytest + +ROOT=Path(__file__).resolve().parents[1] + + +def load(name): + spec=importlib.util.spec_from_file_location(name,ROOT/f'tools/bootstrap-recovery/{name}.py') + module=importlib.util.module_from_spec(spec);spec.loader.exec_module(module) + return module + + +fetch=load('fetch-privacyidea-backup') +drill=load('check-privacyidea-database') + + +class Response(io.BytesIO): + def __init__(self,body=b'abc',etag='"same"',length='3',status=200): + super().__init__(body);self.status=status + self.url=fetch.owner.HOST+fetch.owner.BASE+fetch.OBJECT + self.headers={'ETag':etag,'Content-Length':length} + + +class Provider: + def __init__(self,responses):self.responses=iter(responses);self.requests=[] + def open(self,req,timeout):self.requests.append(req);return next(self.responses) + + +def test_transfer_binds_head_get_and_recheck(monkeypatch,tmp_path): + monkeypatch.setattr(fetch,'SIZE',3) + provider=Provider([Response(),Response(),Response()]);target=tmp_path/'ciphertext' + result=fetch.transfer(provider,'disposable-test-auth',target) + assert result['ciphertext_sha256']==hashlib.sha256(b'abc').hexdigest() + assert target.stat().st_mode&0o077==0 + assert provider.requests[1].get_header('If-match')=='"same"' + + +@pytest.mark.parametrize('responses',[ + [Response(etag='W/"weak"')], + [Response(),Response(etag='"changed"')], + [Response(),Response(body=b'ab')], + [Response(),Response(body=b'abcd')], + [Response(),Response(),Response(etag='"changed"')], + [Response(status=302)], +]) +def test_provider_changes_truncation_oversize_and_redirect_refuse(monkeypatch,tmp_path,responses): + monkeypatch.setattr(fetch,'SIZE',3) + with pytest.raises(fetch.owner.Refused):fetch.transfer(Provider(responses),'fixture',tmp_path/'ciphertext') + + +def test_existing_ciphertext_is_preserved(monkeypatch,tmp_path): + monkeypatch.setattr(fetch,'SIZE',3);target=tmp_path/'ciphertext';target.write_bytes(b'existing') + with pytest.raises(FileExistsError):fetch.transfer(Provider([Response(),Response()]),'fixture',target) + assert target.read_bytes()==b'existing' + + +def test_transfer_digest_and_permissions_gate_private_key_use(monkeypatch,tmp_path): + monkeypatch.setattr(drill,'SIZE',3) + with tempfile.TemporaryDirectory(prefix='nk49-offhost-',dir='/tmp') as directory: + target=Path(directory)/'privacyidea.dump.age';target.write_bytes(b'abc');target.chmod(0o600) + r={'schema':'netkingdom.privacyidea-offhost-transfer.v1','status':'ciphertext_fetched', + 'object':drill.OBJECT,'ciphertext_bytes':3,'object_unchanged_during_transfer':True, + 'ciphertext_path':str(target),'ciphertext_sha256':hashlib.sha256(b'abc').hexdigest()} + receipt=tmp_path/'receipt.json';receipt.write_text(json.dumps(r)) + assert drill.transfer_input(receipt)[0]==b'abc' + target.chmod(0o644) + with pytest.raises(drill.Refused):drill.transfer_input(receipt) + target.chmod(0o600);target.write_bytes(b'bad') + with pytest.raises(drill.Refused):drill.transfer_input(receipt) + + +@pytest.mark.skipif(os.environ.get('NK49_RUN_DISPOSABLE_POSTGRES_TEST')!='1',reason='explicit local Docker fixture opt-in') +def test_real_disposable_custom_dump_restore_and_readback(tmp_path,monkeypatch,capsys): + """No provider/key/production access: three synthetic PI-shaped tables.""" + fixture='nk49-fixture-'+uuid.uuid4().hex;restored='nk49-fixture-'+uuid.uuid4().hex + def command(args,**kwargs): + return subprocess.run(args,check=True,stdout=subprocess.PIPE,stderr=subprocess.PIPE,timeout=120,**kwargs).stdout + try: + command(['docker','run','--detach','--pull=never','--name',fixture,'--network','none', + '--tmpfs','/var/lib/postgresql/data:rw,size=268435456', + '--env','POSTGRES_HOST_AUTH_METHOD=trust',drill.IMAGE,'postgres','-c','listen_addresses=']) + for _ in range(60): + probe=subprocess.run(['docker','exec',fixture,'pg_isready','-U','postgres'],capture_output=True) + if probe.returncode==0:break + time.sleep(1) + command(['docker','exec','-i',fixture,'psql','-X','-U','postgres','-v','ON_ERROR_STOP=1'], + input=b'CREATE TABLE token(id integer); INSERT INTO token VALUES (1),(2); CREATE TABLE resolver(id integer); CREATE TABLE realm(id integer); INSERT INTO realm VALUES (1);') + dump=tmp_path/'fixture.dump' + dump.write_bytes(command(['docker','exec',fixture,'pg_dump','-U','postgres','-Fc','postgres'])) + result=drill.restore(dump,restored) + assert result['aggregate_row_counts']=={'token':2,'resolver':0,'realm':1} + assert '16.' in result['pg_restore_version'] + # Exercise the actual custody/decryption/negative/restore/cleanup path + # with a freshly created fixture key, never an operator key. + key=tmp_path/'fixture.key' + command(['age-keygen','-o',str(key)]) + recipient=command(['age-keygen','-y',str(key)]).decode().strip() + blob=command(['age','-r',recipient],input=dump.read_bytes()) + monkeypatch.setenv('NETKINGDOM_CUSTODY_AGE_KEY_FILE',str(key)) + monkeypatch.setattr(drill,'RECIPIENT',recipient) + monkeypatch.setattr(drill,'SIZE',len(blob)) + monkeypatch.setattr(drill,'transfer_input',lambda path:(blob,hashlib.sha256(blob).hexdigest())) + receipt=tmp_path/'drill.json' + monkeypatch.setattr('sys.argv',['check','--transfer-receipt','fixture','--receipt',str(receipt)]) + assert drill.main()==0 + r=json.loads(receipt.read_text()) + assert r['wrong_key_refused'] and r['offhost_ciphertext_decryption_verified'] + assert r['aggregate_row_counts']=={'token':2,'resolver':0,'realm':1} + assert r['container_cleanup_verified'] and r['temporary_plaintext_cleanup_verified'] + assert r['native_identity_recovery_proven'] is False + assert 'AGE-SECRET-KEY-' not in capsys.readouterr().out + failed=tmp_path/'wrong-recipient.json' + monkeypatch.setattr(drill,'RECIPIENT','age1invalid') + monkeypatch.setattr('sys.argv',['check','--transfer-receipt','fixture','--receipt',str(failed)]) + assert drill.main()==1 + assert json.loads(failed.read_text())['status']=='failed' + finally: + for name in (fixture,restored): + subprocess.run(['docker','rm','--force','--volumes',name],stdout=subprocess.DEVNULL,stderr=subprocess.DEVNULL,timeout=30) diff --git a/tools/bootstrap-recovery/README.md b/tools/bootstrap-recovery/README.md index 0f86602..18a0a58 100644 --- a/tools/bootstrap-recovery/README.md +++ b/tools/bootstrap-recovery/README.md @@ -35,3 +35,20 @@ neither the bootstrap API key nor object GET/write/delete operations. Run throug the already admitted caller with cached boto3 1.40.18; it emits only its safe receipt. Both inventories explicitly state their bounds and cannot prove asset absence outside them, ciphertext integrity, decryption or native recovery. + +`fetch-privacyidea-backup.py` retrieves only the fixed reviewed PI ciphertext +under the same attended Platform Backup-account procedure. Strong ETag binding, +exact length, digest and private no-overwrite output are required. It reads no +age key. `check-privacyidea-database.py` then runs through temporary human custody, +checks the transfer lineage/digest and recipient, tests wrong-key refusal and +restores the decrypted custom-format dump into pinned local PostgreSQL with no +network, ports or host data mounts. Only three fixed table counts are exported. +It removes the disposable database and plaintext and records cleanup. It does +not prove token decryption, audit signing continuity or full identity recovery. +See the exact prepared command in the ceremony, rather than inventing new inputs. + +```bash +NK49_RUN_DISPOSABLE_POSTGRES_TEST=1 python3 -m pytest -q \ + tests/test_privacyidea_offhost_recovery.py \ + tests/test_storage_share_inventory.py tests/test_scaleway_backup_inventory.py +``` diff --git a/tools/bootstrap-recovery/check-privacyidea-database.py b/tools/bootstrap-recovery/check-privacyidea-database.py new file mode 100644 index 0000000..d00361b --- /dev/null +++ b/tools/bootstrap-recovery/check-privacyidea-database.py @@ -0,0 +1,156 @@ +#!/usr/bin/env python3 +"""Restore the fetched PI database in an isolated, disposable local PostgreSQL. + +Run only through the attended custody helper. No cluster, production database, +network, ports, host data mounts, IAM login or token decryption is involved. +""" +import argparse +from datetime import datetime, timezone +import hashlib +import importlib.util +import json +import os +from pathlib import Path +import signal +import subprocess +import sys +import tempfile +import time +import uuid + +ROOT=Path(__file__).resolve().parents[2] +spec=importlib.util.spec_from_file_location('custody',ROOT/'sso-mfa/bootstrap/sops-private.py') +custody=importlib.util.module_from_spec(spec);spec.loader.exec_module(custody) +RECIPIENT='age1zvryunvjhvpkmasskauga2heeg0ztnte9ymgppvjge36ekumk50syr3tsz' +IMAGE='postgres@sha256:ff23cdce56cac62ada6f66013e1a50864c0abbe79d132d40b6e05bd80f378a70' +OBJECT='r01-net-kingdom-pg/net-kingdom-pg-privacyidea_db-20261015T023000Z.dump.age' +SIZE=89129309 + + +class Refused(Exception):pass + + +def run(args,**kwargs): + result=subprocess.run(args,stdout=subprocess.PIPE,stderr=subprocess.DEVNULL,timeout=600,**kwargs) + if result.returncode:raise Refused() + return result.stdout + + +def transfer_input(receipt): + r=json.loads(receipt.read_text()) + if (r.get('schema')!='netkingdom.privacyidea-offhost-transfer.v1' or + r.get('status')!='ciphertext_fetched' or r.get('object')!=OBJECT or + r.get('ciphertext_bytes')!=SIZE or r.get('object_unchanged_during_transfer') is not True):raise Refused() + path=Path(r['ciphertext_path']) + if not path.is_absolute() or path.parent.parent!=Path('/tmp') or not path.parent.name.startswith('nk49-offhost-'):raise Refused() + if any(p.is_symlink() for p in (path,*path.parents)):raise Refused() + st=path.stat() + if st.st_uid!=os.getuid() or st.st_mode&0o077 or st.st_nlink!=1 or not path.is_file() or st.st_size!=SIZE:raise Refused() + blob=path.read_bytes() + if len(blob)!=SIZE or hashlib.sha256(blob).hexdigest()!=r['ciphertext_sha256']:raise Refused() + return blob,r['ciphertext_sha256'] + + +def restore(dump,container): + # Require the exact image already present; do not pull through a mutable tag. + image_id=run(['docker','image','inspect','--format','{{.Id}}',IMAGE]).decode().strip() + run(['docker','run','--detach','--pull=never','--name',container,'--network','none', + '--security-opt','no-new-privileges','--memory','2g','--cpus','2', + '--tmpfs','/var/lib/postgresql/data:rw,nosuid,nodev,size=1073741824', + '--env','POSTGRES_HOST_AUTH_METHOD=trust','--env','POSTGRES_DB=recovery', + IMAGE,'postgres','-c','listen_addresses=']) + for _ in range(60): + # The image starts a temporary server during initdb. Wait until the + # entrypoint has exec'd the final server before attempting the restore. + pid1=subprocess.run(['docker','exec',container,'cat','/proc/1/comm'], + stdout=subprocess.PIPE,stderr=subprocess.DEVNULL,timeout=10) + if pid1.returncode or pid1.stdout.strip()!=b'postgres': + time.sleep(1);continue + ready=subprocess.run(['docker','exec',container,'pg_isready','-U','postgres','-d','recovery'], + stdout=subprocess.DEVNULL,stderr=subprocess.DEVNULL,timeout=10) + if ready.returncode==0:break + time.sleep(1) + else:raise Refused() + version=run(['docker','exec',container,'pg_restore','--version']).decode().strip() + with dump.open('rb') as source: + run(['docker','exec','-i',container,'pg_restore','-U','postgres','-d','recovery', + '--no-owner','--no-privileges','--exit-on-error'],stdin=source) + # Fixed aggregate queries only. No identity, resolver config or token values. + counts={} + for table in ('token','resolver','realm'): + raw=run(['docker','exec',container,'psql','-X','-U','postgres','-d','recovery','-At', + '-v','ON_ERROR_STOP=1','-c',f'SELECT count(*) FROM public.{table};']).strip() + if not raw.isdigit():raise Refused() + counts[table]=int(raw) + return {'image':IMAGE,'image_id':image_id,'pg_restore_version':version,'aggregate_row_counts':counts} + + +def main(): + p=argparse.ArgumentParser(description=__doc__) + p.add_argument('--transfer-receipt',required=True);p.add_argument('--receipt',required=True) + a=p.parse_args();os.umask(0o077) + # Reserve the receipt before any recovery action; never overwrite evidence. + fd=os.open(a.receipt,os.O_WRONLY|os.O_CREAT|os.O_EXCL|os.O_NOFOLLOW,0o600) + r={'schema':'netkingdom.privacyidea-database-drill.v1','status':'failed', + 'started_at':datetime.now(timezone.utc).isoformat(),'source_sha256':hashlib.sha256(Path(__file__).read_bytes()).hexdigest(), + 'production_contacted':False,'secret_values_exported':False,'native_identity_recovery_proven':False, + 'token_decryption_proven':False,'audit_signature_continuity_proven':False, + 'full_identity_file_coverage_proven':False,'container_cleanup_verified':False, + 'temporary_plaintext_cleanup_verified':False} + container='nk49-pi-recovery-'+uuid.uuid4().hex + keyfd=None;work=None + try: + blob,digest=transfer_input(Path(a.transfer_receipt));r['ciphertext_sha256']=digest + keyfd=custody.private_key_fd() + derived=run(['age-keygen','-y',f'/proc/self/fd/{keyfd}'],pass_fds=(keyfd,)).strip() + if derived!=RECIPIENT.encode():raise Refused() + r['registered_recipient_match']=True + with tempfile.TemporaryDirectory(prefix='nk49-pi-drill-',dir='/tmp') as temporary: + work=Path(temporary) + env={'PATH':os.environ.get('PATH',os.defpath),'HOME':temporary,'XDG_CONFIG_HOME':temporary} + wrong=work/'wrong.key' + run(['age-keygen','-o',str(wrong)],env=env) + negative=subprocess.run(['age','-d','-i',str(wrong)],input=blob,stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL,timeout=60,env=env) + if negative.returncode==0:raise Refused() + r['wrong_key_refused']=True + dump=work/'privacyidea.dump' + with dump.open('xb') as output: + result=subprocess.run(['age','-d','-i',f'/proc/self/fd/{keyfd}'],input=blob, + stdout=output,stderr=subprocess.DEVNULL,timeout=120,pass_fds=(keyfd,),env=env) + if result.returncode or not 5200 or not etag.startswith('"') or not etag.endswith('"') or + any(ord(c)<32 for c in etag)): + raise owner.Refused() + return etag + + +def transfer(opener,auth,destination): + url=owner.HOST+owner.BASE+OBJECT + headers={'Authorization':'Basic '+auth,'Accept-Encoding':'identity'} + with opener.open(urllib.request.Request(url,method='HEAD',headers=headers),timeout=30) as response: + etag=metadata(response,url) + headers['If-Match']=etag + digest=hashlib.sha256();count=0 + with opener.open(urllib.request.Request(url,headers=headers),timeout=60) as response: + if metadata(response,url)!=etag:raise owner.Refused() + fd=os.open(destination,os.O_WRONLY|os.O_CREAT|os.O_EXCL|os.O_NOFOLLOW,0o600) + with os.fdopen(fd,'wb') as out: + while chunk:=response.read(1024*1024): + count+=len(chunk) + if count>SIZE:raise owner.Refused() + digest.update(chunk);out.write(chunk) + out.flush();os.fsync(out.fileno()) + if count!=SIZE:raise owner.Refused() + with opener.open(urllib.request.Request(url,method='HEAD',headers=headers),timeout=30) as response: + if metadata(response,url)!=etag:raise owner.Refused() + # ETag used for request binding only; it is not asserted to be a content hash. + return {'ciphertext_bytes':count,'ciphertext_sha256':digest.hexdigest(), + 'object_unchanged_during_transfer':True} + + +def main(): + p=argparse.ArgumentParser(description=__doc__);p.add_argument('--receipt',required=True) + a=p.parse_args();os.umask(0o077) + fd=os.open(a.receipt,os.O_WRONLY|os.O_CREAT|os.O_EXCL|os.O_NOFOLLOW,0o600) + r={'schema':'netkingdom.privacyidea-offhost-transfer.v1','status':'failed', + 'started_at':datetime.now(timezone.utc).isoformat(),'provider':'Storage Share', + 'object':OBJECT,'age_keys_read':False,'plaintext_created':False, + 'production_writes':False,'source_sha256':hashlib.sha256(Path(__file__).read_bytes()).hexdigest(), + 'owner_accessor_sha256':hashlib.sha256((HERE/'inventory-storage-share.py').read_bytes()).hexdigest()} + destination=None + try: + identity,credentials=owner.caller_credentials() + r['caller_auth_path']=identity['path'];r['caller_entity_id']=identity.get('entity_id') + auth=base64.b64encode((credentials['BACKUP_USERNAME']+':'+credentials['BACKUP_PASSWORD']).encode()).decode() + destination=Path(tempfile.mkdtemp(prefix='nk49-offhost-',dir='/tmp'))/'privacyidea.dump.age' + opener=urllib.request.build_opener(urllib.request.ProxyHandler({}),owner.NoRedirect()) + r.update(transfer(opener,auth,destination)) + r.update(status='ciphertext_fetched',ciphertext_path=str(destination)) + except Exception: + r['error']='bounded_transfer_failed_diagnostics_suppressed' + if destination is not None: + destination.unlink(missing_ok=True);destination.parent.rmdir() + finally: + r['finished_at']=datetime.now(timezone.utc).isoformat() + with os.fdopen(fd,'w') as out:json.dump(r,out,indent=2);out.write('\n') + return int(r['status']=='failed') + + +if __name__=='__main__':raise SystemExit(main()) diff --git a/tools/bootstrap-recovery/inventory-storage-share.py b/tools/bootstrap-recovery/inventory-storage-share.py index adde0df..fdcdfb6 100644 --- a/tools/bootstrap-recovery/inventory-storage-share.py +++ b/tools/bootstrap-recovery/inventory-storage-share.py @@ -74,13 +74,18 @@ def listing(relative,credentials): return parse_listing(raw,relative) -def inventory(): +def caller_credentials(): # Lookup payload contains a token: retain only safe caller/role facts. identity=data(bao(['token','lookup','-format=json']))['data'] if 'platform-admin' not in identity.get('policies',[]) or 'root' in identity.get('policies',[]):raise Refused() if not str(identity.get('path','')).startswith('auth/netkingdom/'):raise Refused() credentials=data(bao(['read','-format=json',OPERATOR_PATH]))['data']['data'] if credentials.get('BACKUP_USERNAME')!='Backup' or not isinstance(credentials.get('BACKUP_PASSWORD'),str) or not credentials['BACKUP_PASSWORD']:raise Refused() + return identity,credentials + + +def inventory(): + identity,credentials=caller_credentials() root=listing('',credentials) folders={} for row in root: diff --git a/workplans/NK-WP-0049-bootstrap-secret-free-recovery-handoff.md b/workplans/NK-WP-0049-bootstrap-secret-free-recovery-handoff.md index 46e8df4..b6857ee 100644 --- a/workplans/NK-WP-0049-bootstrap-secret-free-recovery-handoff.md +++ b/workplans/NK-WP-0049-bootstrap-secret-free-recovery-handoff.md @@ -8,7 +8,7 @@ status: active owner: codex topic_slug: netkingdom created: "2026-10-12" -updated: "2026-10-15" +updated: "2026-10-16" flavor: implementation depends_on: [] state_hub_workstream_id: "17d315f0-829e-57c7-85cc-3a7914152804" @@ -459,6 +459,23 @@ bootstrap `keys/age.pub` is not assumed to be the same recipient. T02/T03 remain open. Evidence: `offsite-discovery-summary.json` and the two exact inventory receipts in the existing October15 evidence folder. +2026-10-16 custody/retrieval checkpoint: operator reports the hidden check +matched the backup recipient and removed the temporary key. No private key was +provided to the agent. The exact existing PI database ciphertext was fetched +through the attended Platform Backup-account procedure, with strong ETag binding, +89,129,309 bytes and SHA256 +`cc81da442755928171d2c51bce93eda7de4392a0a1aa9d0ffb83a25ba34e5ead`. +The session exited0 and self-revoked. Ciphertext is private temporary storage; +no plaintext or age key was read by the fetch. Prepared the exact human custody +command in the recovery ceremony for an isolated, pinned PostgreSQL restore +with no network, ports or host mounts; fixed aggregate readback and verified +container/plaintext cleanup.23 focused tests passed including actual local +PostgreSQL restore using synthetic data and a fresh fixture age key. Actual +backup decryption remains pending the operator command. This is partial database +recovery preparation; full matching identity key/config coverage, service +continuity, T02 owner acceptance and legacy credential disposition remain open. +Evidence: `privacyidea-transfer-r1.json`, `database-drill-preparation.json`. + ## Verify owner-approved recovery and current native callers ```task