Prepare bounded off-host privacyIDEA recovery drill
Assistant: codex Assistant-Model: gpt-6-astra Assistant-Session: 01a13eab-1e6a-7b33-9c37-372e90ada3a2
This commit is contained in:
parent
c262470089
commit
a1939a7d06
12 changed files with 549 additions and 23 deletions
|
|
@ -136,9 +136,50 @@ No object contents or age keys were read; the operator envelope exited0 and
|
|||
self-revoked. Existing scoped backup credentials were reused without changes;
|
||||
the Scaleway bootstrap API key was not used.
|
||||
|
||||
The next human act is to run the existing custody helper in their own terminal
|
||||
with `--expected-recipient` set to the public recipient in the discovery summary,
|
||||
`--prompt --check-only`. Paste private material only into the hidden prompt.
|
||||
This verifies the independently held key against the backup owner's declared
|
||||
recipient and removes its temporary copy. It does not by itself prove that the
|
||||
selected backup decrypts or includes the full matching identity recovery set.
|
||||
The operator subsequently reported the hidden-terminal check passed and the
|
||||
temporary key file was removed. This is operator-reported recipient matching;
|
||||
the private key was not provided to the agent. Actual decryption remains pending.
|
||||
|
||||
## Prepared bounded privacyIDEA database drill — 2026-10-16
|
||||
|
||||
The identified existing off-host ciphertext was downloaded through the same
|
||||
attended Platform Backup-account procedure. The before/GET/after strong-ETag
|
||||
binding and exact byte count passed. Ciphertext SHA256:
|
||||
`cc81da442755928171d2c51bce93eda7de4392a0a1aa9d0ffb83a25ba34e5ead`.
|
||||
See `privacyidea-transfer-r1.json` in the October15 evidence folder. The owner
|
||||
envelope exited0 and self-revoked; no age key or plaintext was read by the fetch.
|
||||
Ciphertext is retained in the fresh private `/tmp/nk49-offhost-j9dzl0ge/` directory.
|
||||
|
||||
The following exact human command performs only an isolated local database
|
||||
drill. It checks the transfer receipt/digest before opening the temporary key,
|
||||
checks the backup recipient, refuses an unrelated wrong key, decrypts to private
|
||||
temporary storage and restores into disposable PostgreSQL. Image
|
||||
`postgres@sha256:ff23cdce56cac62ada6f66013e1a50864c0abbe79d132d40b6e05bd80f378a70`
|
||||
is already cached locally (pg_restore16.14). There is no image pull, network,
|
||||
published port, host data mount, production connection or IAM/client execution.
|
||||
Database storage is a bounded temporary memory filesystem. Only fixed aggregate
|
||||
counts for `token`, `resolver`, and `realm` leave the database. The container,
|
||||
volumes and temporary plaintext are removed and cleanup is checked. The original
|
||||
ciphertext remains for reproducibility. This proves database recovery only;
|
||||
token decryption, audit signature continuity and full identity service recovery
|
||||
require matching key/config files and further native owner checks.
|
||||
|
||||
Run from the repository in the operator's own terminal. Paste the private key
|
||||
only into the hidden prompt, then share the success/error line or safe JSON
|
||||
receipt. The receipt destination must be new; do not overwrite a failed receipt.
|
||||
|
||||
```bash
|
||||
cd /home/worsch/net-kingdom
|
||||
bash sso-mfa/bootstrap/sops-custody-unlock.sh \
|
||||
--expected-recipient age1zvryunvjhvpkmasskauga2heeg0ztnte9ymgppvjge36ekumk50syr3tsz \
|
||||
--prompt -- python3 tools/bootstrap-recovery/check-privacyidea-database.py \
|
||||
--transfer-receipt docs/evidence/2026-10-15-bootstrap-recovery/privacyidea-transfer-r1.json \
|
||||
--receipt docs/evidence/2026-10-15-bootstrap-recovery/privacyidea-database-drill-r1.json
|
||||
```
|
||||
|
||||
Source verification:23 focused checks passed, including a real local Docker
|
||||
custom-format dump, age encryption, temporary fixture-key unlock, wrong-key
|
||||
refusal, native restore/readback and cleanup. Those tests use synthetic data and
|
||||
fresh fixture keys. They are not evidence of actual backup decryption. If the
|
||||
real drill fails, review its closed receipt; do not expose PostgreSQL/age stderr
|
||||
or dump contents in chat. T02/T03 remain open.
|
||||
|
|
|
|||
|
|
@ -1,8 +1,8 @@
|
|||
{
|
||||
"schema": "netkingdom.bootstrap-caller-inventory.v1",
|
||||
"observed_at": "2026-10-15T20:54:09.856047+00:00",
|
||||
"scope": "all current .sh/.py non-test files under sso-mfa/bootstrap and sso-mfa/k8s, Makefile, console, delivery copier and recovery transport tool plus two bounded provider metadata inventory helpers",
|
||||
"entry_count": 89,
|
||||
"observed_at": "2026-10-15T22:08:27.470032+00:00",
|
||||
"scope": "all current .sh/.py non-test files under sso-mfa/bootstrap and sso-mfa/k8s, Makefile, console, delivery copier and recovery transport tool plus two bounded provider metadata inventory helpers plus bounded fixed-object ciphertext fetch and isolated PostgreSQL recovery check",
|
||||
"entry_count": 91,
|
||||
"entries": [
|
||||
{
|
||||
"path": "Makefile",
|
||||
|
|
@ -431,7 +431,7 @@
|
|||
},
|
||||
{
|
||||
"path": "tools/bootstrap-recovery/inventory-storage-share.py",
|
||||
"sha256": "fe56454e5ca740395e6d6a2a0f8185abdac1d8ca6d0eedb31273db409dee4954",
|
||||
"sha256": "673f9b17fb2f3fef6fcdb5c50a2bcca69e3df3157a9e5582a2d995c6b87640b5",
|
||||
"classification": "read-only bounded owner backup metadata; existing caller/custody only; no age key or backup content read"
|
||||
},
|
||||
{
|
||||
|
|
@ -448,6 +448,16 @@
|
|||
"path": "tools/sync-user-engine-delivery-secret.sh",
|
||||
"sha256": "7c0fb6b066e4ad8b103e75551beecd5de65a4bbb22570d95b42e5da6395e2dfc",
|
||||
"classification": "retired entry point; refuses before credential/cluster access"
|
||||
},
|
||||
{
|
||||
"path": "tools/bootstrap-recovery/fetch-privacyidea-backup.py",
|
||||
"sha256": "233160e1206643a5cf7ad05141d733f0a75aff651190e27f77515a124ff0a59f",
|
||||
"classification": "bounded off-host transfer / isolated database drill; actual recovery acceptance requires operator execution"
|
||||
},
|
||||
{
|
||||
"path": "tools/bootstrap-recovery/check-privacyidea-database.py",
|
||||
"sha256": "90f454c87998d7f5d625e3c6656c3418f8f56275dc1879a9d19ed605431b94a4",
|
||||
"classification": "bounded off-host transfer / isolated database drill; actual recovery acceptance requires operator execution"
|
||||
}
|
||||
],
|
||||
"source_inventory_complete_for_declared_scope": true,
|
||||
|
|
|
|||
|
|
@ -0,0 +1,32 @@
|
|||
{
|
||||
"observed_at": "2026-10-15T22:08:27.470032+00:00",
|
||||
"operator_reported_key_check_passed": true,
|
||||
"private_key_received_by_agent": false,
|
||||
"actual_ciphertext_decryption_verified": false,
|
||||
"transfer_receipt": "privacyidea-transfer-r1.json",
|
||||
"transfer_envelope_exit_code": 0,
|
||||
"transfer_session_revoked_and_cleaned": true,
|
||||
"source_tests": {
|
||||
"passed": 23,
|
||||
"command": "NK49_RUN_DISPOSABLE_POSTGRES_TEST=1 python3 -m pytest -q tests/test_privacyidea_offhost_recovery.py tests/test_storage_share_inventory.py tests/test_scaleway_backup_inventory.py",
|
||||
"duration_seconds": 8.54,
|
||||
"data": "synthetic database and fresh disposable age keys only",
|
||||
"includes_native_postgres_restore_and_cleanup": true
|
||||
},
|
||||
"pending_human_command": "docs/bootstrap-recovery-ceremony.md: Prepared bounded privacyIDEA database drill",
|
||||
"native_identity_recovery_proven": false,
|
||||
"sources": [
|
||||
{
|
||||
"path": "tools/bootstrap-recovery/check-privacyidea-database.py",
|
||||
"sha256": "90f454c87998d7f5d625e3c6656c3418f8f56275dc1879a9d19ed605431b94a4"
|
||||
},
|
||||
{
|
||||
"path": "tools/bootstrap-recovery/fetch-privacyidea-backup.py",
|
||||
"sha256": "233160e1206643a5cf7ad05141d733f0a75aff651190e27f77515a124ff0a59f"
|
||||
},
|
||||
{
|
||||
"path": "tests/test_privacyidea_offhost_recovery.py",
|
||||
"sha256": "ea977b0d3368fb14f6a7a252c24cc5cb9915022afa3dca61c80952e7489bd0e5"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -1,5 +1,5 @@
|
|||
{
|
||||
"observed_at": "2026-10-15T21:58:20.278564+00:00",
|
||||
"observed_at": "2026-10-15T22:08:27.470032+00:00",
|
||||
"operator_reports_holding_railiance_backup_age_key": true,
|
||||
"private_key_received_or_read": false,
|
||||
"candidate_key_match_verified": false,
|
||||
|
|
@ -10,8 +10,11 @@
|
|||
"session_self_revocation_and_cleanup_confirmed": true,
|
||||
"selected_candidate": "r01-net-kingdom-pg/net-kingdom-pg-privacyidea_db-20261015T023000Z.dump.age",
|
||||
"candidate_bytes": 89129309,
|
||||
"candidate_contents_downloaded": false,
|
||||
"bootstrap_identity_file_bundle_found_in_inspected_scope": false
|
||||
"candidate_contents_downloaded": true,
|
||||
"bootstrap_identity_file_bundle_found_in_inspected_scope": false,
|
||||
"transfer_receipt": "privacyidea-transfer-r1.json",
|
||||
"attended_transfer_envelope_exit_code": 0,
|
||||
"transfer_session_self_revocation_and_cleanup_confirmed": true
|
||||
},
|
||||
"scaleway": {
|
||||
"inventory": "scaleway-inventory-r1.json",
|
||||
|
|
@ -26,7 +29,9 @@
|
|||
"source_sha256": "09f349ae815339732db505d6896fb1ca384be7c66b40574750a3ba30d4d922e0",
|
||||
"declared_default_public_recipient": "age1zvryunvjhvpkmasskauga2heeg0ztnte9ymgppvjge36ekumk50syr3tsz",
|
||||
"operator_hidden_terminal_check_requested": true,
|
||||
"scope": "A successful key match checks the declared public recipient, not actual ciphertext decryption or whole-stack recovery."
|
||||
"scope": "A successful key match checks the declared public recipient, not actual ciphertext decryption or whole-stack recovery.",
|
||||
"operator_reports_check_passed": true,
|
||||
"operator_reports_temporary_key_removed": true
|
||||
},
|
||||
"source_tests": {
|
||||
"passed": 13,
|
||||
|
|
|
|||
|
|
@ -0,0 +1,19 @@
|
|||
{
|
||||
"schema": "netkingdom.privacyidea-offhost-transfer.v1",
|
||||
"status": "ciphertext_fetched",
|
||||
"started_at": "2026-10-15T22:07:08.615448+00:00",
|
||||
"provider": "Storage Share",
|
||||
"object": "r01-net-kingdom-pg/net-kingdom-pg-privacyidea_db-20261015T023000Z.dump.age",
|
||||
"age_keys_read": false,
|
||||
"plaintext_created": false,
|
||||
"production_writes": false,
|
||||
"source_sha256": "233160e1206643a5cf7ad05141d733f0a75aff651190e27f77515a124ff0a59f",
|
||||
"owner_accessor_sha256": "673f9b17fb2f3fef6fcdb5c50a2bcca69e3df3157a9e5582a2d995c6b87640b5",
|
||||
"caller_auth_path": "auth/netkingdom/oidc/callback",
|
||||
"caller_entity_id": "4a1f663c-0621-0483-edd0-f852274f4b6b",
|
||||
"ciphertext_bytes": 89129309,
|
||||
"ciphertext_sha256": "cc81da442755928171d2c51bce93eda7de4392a0a1aa9d0ffb83a25ba34e5ead",
|
||||
"object_unchanged_during_transfer": true,
|
||||
"ciphertext_path": "/tmp/nk49-offhost-j9dzl0ge/privacyidea.dump.age",
|
||||
"finished_at": "2026-10-15T22:07:16.363645+00:00"
|
||||
}
|
||||
|
|
@ -1,14 +1,14 @@
|
|||
{
|
||||
"workplan": "NK-WP-0049",
|
||||
"observed_at": "2026-10-15T20:56:58.471680+00:00",
|
||||
"observed_at": "2026-10-15T22:08:27.470032+00:00",
|
||||
"status": "source_prepared_pending_owner_inputs_and_native_acceptance",
|
||||
"backup_destinations_operator_reported": [
|
||||
"Scaleway",
|
||||
"Storage Share"
|
||||
],
|
||||
"exact_offhost_artifact_confirmed": false,
|
||||
"exact_offhost_artifact_confirmed": true,
|
||||
"independently_held_matching_key_confirmed": false,
|
||||
"recovery_key_or_backup_contents_retrieved": false,
|
||||
"recovery_key_or_backup_contents_retrieved": true,
|
||||
"legacy_token_secret_present": true,
|
||||
"legacy_trigger_secret_present": true,
|
||||
"legacy_provider_revocation_proven": false,
|
||||
|
|
@ -18,9 +18,9 @@
|
|||
"verification": "source-verification.json",
|
||||
"ceremony": "../../bootstrap-recovery-ceremony.md",
|
||||
"outstanding": [
|
||||
"Matching off-host identity data/key/config set is not located in inspected provider scopes; a recent PI database ciphertext is identified, not downloaded/decrypted",
|
||||
"Independent age key availability/custody confirmation",
|
||||
"Owner admitted isolated recovery target/procedure and native service continuity/cleanup evidence",
|
||||
"Matching off-host identity data/key/config set remains unlocated in inspected scopes; selected existing PI database ciphertext is now fetched and digest-bound",
|
||||
"Operator reports matching-recipient key check passed; actual ciphertext decryption/native database drill is prepared and pending operator hidden-key invocation",
|
||||
"Full identity service recovery, token decryption, audit signature continuity and cleanup evidence",
|
||||
"T02 current caller/provider source admission and exact legacy provider credential disposition; retain existing deferred-rotation boundary"
|
||||
],
|
||||
"operator_reports_holding_railiance_backup_age_key": true,
|
||||
|
|
@ -29,5 +29,12 @@
|
|||
"key_matches_selected_ciphertext_verified": false,
|
||||
"offsite_discovery": "offsite-discovery-summary.json",
|
||||
"privacyidea_database_ciphertext_identified": true,
|
||||
"required_full_asset_coverage_confirmed": false
|
||||
"required_full_asset_coverage_confirmed": false,
|
||||
"operator_reports_recipient_check_passed": true,
|
||||
"operator_reports_temporary_key_cleanup": true,
|
||||
"private_key_received_by_agent": false,
|
||||
"ciphertext_fetched": true,
|
||||
"transfer_receipt": "privacyidea-transfer-r1.json",
|
||||
"prepared_database_drill": "../../bootstrap-recovery-ceremony.md",
|
||||
"age_private_key_retrieved_by_agent": false
|
||||
}
|
||||
|
|
|
|||
129
tests/test_privacyidea_offhost_recovery.py
Normal file
129
tests/test_privacyidea_offhost_recovery.py
Normal file
|
|
@ -0,0 +1,129 @@
|
|||
"""Transfer binding failures and a real disposable PostgreSQL fixture restore."""
|
||||
import hashlib
|
||||
import importlib.util
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import tempfile
|
||||
import time
|
||||
import uuid
|
||||
import pytest
|
||||
|
||||
ROOT=Path(__file__).resolve().parents[1]
|
||||
|
||||
|
||||
def load(name):
|
||||
spec=importlib.util.spec_from_file_location(name,ROOT/f'tools/bootstrap-recovery/{name}.py')
|
||||
module=importlib.util.module_from_spec(spec);spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
fetch=load('fetch-privacyidea-backup')
|
||||
drill=load('check-privacyidea-database')
|
||||
|
||||
|
||||
class Response(io.BytesIO):
|
||||
def __init__(self,body=b'abc',etag='"same"',length='3',status=200):
|
||||
super().__init__(body);self.status=status
|
||||
self.url=fetch.owner.HOST+fetch.owner.BASE+fetch.OBJECT
|
||||
self.headers={'ETag':etag,'Content-Length':length}
|
||||
|
||||
|
||||
class Provider:
|
||||
def __init__(self,responses):self.responses=iter(responses);self.requests=[]
|
||||
def open(self,req,timeout):self.requests.append(req);return next(self.responses)
|
||||
|
||||
|
||||
def test_transfer_binds_head_get_and_recheck(monkeypatch,tmp_path):
|
||||
monkeypatch.setattr(fetch,'SIZE',3)
|
||||
provider=Provider([Response(),Response(),Response()]);target=tmp_path/'ciphertext'
|
||||
result=fetch.transfer(provider,'disposable-test-auth',target)
|
||||
assert result['ciphertext_sha256']==hashlib.sha256(b'abc').hexdigest()
|
||||
assert target.stat().st_mode&0o077==0
|
||||
assert provider.requests[1].get_header('If-match')=='"same"'
|
||||
|
||||
|
||||
@pytest.mark.parametrize('responses',[
|
||||
[Response(etag='W/"weak"')],
|
||||
[Response(),Response(etag='"changed"')],
|
||||
[Response(),Response(body=b'ab')],
|
||||
[Response(),Response(body=b'abcd')],
|
||||
[Response(),Response(),Response(etag='"changed"')],
|
||||
[Response(status=302)],
|
||||
])
|
||||
def test_provider_changes_truncation_oversize_and_redirect_refuse(monkeypatch,tmp_path,responses):
|
||||
monkeypatch.setattr(fetch,'SIZE',3)
|
||||
with pytest.raises(fetch.owner.Refused):fetch.transfer(Provider(responses),'fixture',tmp_path/'ciphertext')
|
||||
|
||||
|
||||
def test_existing_ciphertext_is_preserved(monkeypatch,tmp_path):
|
||||
monkeypatch.setattr(fetch,'SIZE',3);target=tmp_path/'ciphertext';target.write_bytes(b'existing')
|
||||
with pytest.raises(FileExistsError):fetch.transfer(Provider([Response(),Response()]),'fixture',target)
|
||||
assert target.read_bytes()==b'existing'
|
||||
|
||||
|
||||
def test_transfer_digest_and_permissions_gate_private_key_use(monkeypatch,tmp_path):
|
||||
monkeypatch.setattr(drill,'SIZE',3)
|
||||
with tempfile.TemporaryDirectory(prefix='nk49-offhost-',dir='/tmp') as directory:
|
||||
target=Path(directory)/'privacyidea.dump.age';target.write_bytes(b'abc');target.chmod(0o600)
|
||||
r={'schema':'netkingdom.privacyidea-offhost-transfer.v1','status':'ciphertext_fetched',
|
||||
'object':drill.OBJECT,'ciphertext_bytes':3,'object_unchanged_during_transfer':True,
|
||||
'ciphertext_path':str(target),'ciphertext_sha256':hashlib.sha256(b'abc').hexdigest()}
|
||||
receipt=tmp_path/'receipt.json';receipt.write_text(json.dumps(r))
|
||||
assert drill.transfer_input(receipt)[0]==b'abc'
|
||||
target.chmod(0o644)
|
||||
with pytest.raises(drill.Refused):drill.transfer_input(receipt)
|
||||
target.chmod(0o600);target.write_bytes(b'bad')
|
||||
with pytest.raises(drill.Refused):drill.transfer_input(receipt)
|
||||
|
||||
|
||||
@pytest.mark.skipif(os.environ.get('NK49_RUN_DISPOSABLE_POSTGRES_TEST')!='1',reason='explicit local Docker fixture opt-in')
|
||||
def test_real_disposable_custom_dump_restore_and_readback(tmp_path,monkeypatch,capsys):
|
||||
"""No provider/key/production access: three synthetic PI-shaped tables."""
|
||||
fixture='nk49-fixture-'+uuid.uuid4().hex;restored='nk49-fixture-'+uuid.uuid4().hex
|
||||
def command(args,**kwargs):
|
||||
return subprocess.run(args,check=True,stdout=subprocess.PIPE,stderr=subprocess.PIPE,timeout=120,**kwargs).stdout
|
||||
try:
|
||||
command(['docker','run','--detach','--pull=never','--name',fixture,'--network','none',
|
||||
'--tmpfs','/var/lib/postgresql/data:rw,size=268435456',
|
||||
'--env','POSTGRES_HOST_AUTH_METHOD=trust',drill.IMAGE,'postgres','-c','listen_addresses='])
|
||||
for _ in range(60):
|
||||
probe=subprocess.run(['docker','exec',fixture,'pg_isready','-U','postgres'],capture_output=True)
|
||||
if probe.returncode==0:break
|
||||
time.sleep(1)
|
||||
command(['docker','exec','-i',fixture,'psql','-X','-U','postgres','-v','ON_ERROR_STOP=1'],
|
||||
input=b'CREATE TABLE token(id integer); INSERT INTO token VALUES (1),(2); CREATE TABLE resolver(id integer); CREATE TABLE realm(id integer); INSERT INTO realm VALUES (1);')
|
||||
dump=tmp_path/'fixture.dump'
|
||||
dump.write_bytes(command(['docker','exec',fixture,'pg_dump','-U','postgres','-Fc','postgres']))
|
||||
result=drill.restore(dump,restored)
|
||||
assert result['aggregate_row_counts']=={'token':2,'resolver':0,'realm':1}
|
||||
assert '16.' in result['pg_restore_version']
|
||||
# Exercise the actual custody/decryption/negative/restore/cleanup path
|
||||
# with a freshly created fixture key, never an operator key.
|
||||
key=tmp_path/'fixture.key'
|
||||
command(['age-keygen','-o',str(key)])
|
||||
recipient=command(['age-keygen','-y',str(key)]).decode().strip()
|
||||
blob=command(['age','-r',recipient],input=dump.read_bytes())
|
||||
monkeypatch.setenv('NETKINGDOM_CUSTODY_AGE_KEY_FILE',str(key))
|
||||
monkeypatch.setattr(drill,'RECIPIENT',recipient)
|
||||
monkeypatch.setattr(drill,'SIZE',len(blob))
|
||||
monkeypatch.setattr(drill,'transfer_input',lambda path:(blob,hashlib.sha256(blob).hexdigest()))
|
||||
receipt=tmp_path/'drill.json'
|
||||
monkeypatch.setattr('sys.argv',['check','--transfer-receipt','fixture','--receipt',str(receipt)])
|
||||
assert drill.main()==0
|
||||
r=json.loads(receipt.read_text())
|
||||
assert r['wrong_key_refused'] and r['offhost_ciphertext_decryption_verified']
|
||||
assert r['aggregate_row_counts']=={'token':2,'resolver':0,'realm':1}
|
||||
assert r['container_cleanup_verified'] and r['temporary_plaintext_cleanup_verified']
|
||||
assert r['native_identity_recovery_proven'] is False
|
||||
assert 'AGE-SECRET-KEY-' not in capsys.readouterr().out
|
||||
failed=tmp_path/'wrong-recipient.json'
|
||||
monkeypatch.setattr(drill,'RECIPIENT','age1invalid')
|
||||
monkeypatch.setattr('sys.argv',['check','--transfer-receipt','fixture','--receipt',str(failed)])
|
||||
assert drill.main()==1
|
||||
assert json.loads(failed.read_text())['status']=='failed'
|
||||
finally:
|
||||
for name in (fixture,restored):
|
||||
subprocess.run(['docker','rm','--force','--volumes',name],stdout=subprocess.DEVNULL,stderr=subprocess.DEVNULL,timeout=30)
|
||||
|
|
@ -35,3 +35,20 @@ neither the bootstrap API key nor object GET/write/delete operations. Run throug
|
|||
the already admitted caller with cached boto3 1.40.18; it emits only its safe
|
||||
receipt. Both inventories explicitly state their bounds and cannot prove asset
|
||||
absence outside them, ciphertext integrity, decryption or native recovery.
|
||||
|
||||
`fetch-privacyidea-backup.py` retrieves only the fixed reviewed PI ciphertext
|
||||
under the same attended Platform Backup-account procedure. Strong ETag binding,
|
||||
exact length, digest and private no-overwrite output are required. It reads no
|
||||
age key. `check-privacyidea-database.py` then runs through temporary human custody,
|
||||
checks the transfer lineage/digest and recipient, tests wrong-key refusal and
|
||||
restores the decrypted custom-format dump into pinned local PostgreSQL with no
|
||||
network, ports or host data mounts. Only three fixed table counts are exported.
|
||||
It removes the disposable database and plaintext and records cleanup. It does
|
||||
not prove token decryption, audit signing continuity or full identity recovery.
|
||||
See the exact prepared command in the ceremony, rather than inventing new inputs.
|
||||
|
||||
```bash
|
||||
NK49_RUN_DISPOSABLE_POSTGRES_TEST=1 python3 -m pytest -q \
|
||||
tests/test_privacyidea_offhost_recovery.py \
|
||||
tests/test_storage_share_inventory.py tests/test_scaleway_backup_inventory.py
|
||||
```
|
||||
|
|
|
|||
156
tools/bootstrap-recovery/check-privacyidea-database.py
Normal file
156
tools/bootstrap-recovery/check-privacyidea-database.py
Normal file
|
|
@ -0,0 +1,156 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Restore the fetched PI database in an isolated, disposable local PostgreSQL.
|
||||
|
||||
Run only through the attended custody helper. No cluster, production database,
|
||||
network, ports, host data mounts, IAM login or token decryption is involved.
|
||||
"""
|
||||
import argparse
|
||||
from datetime import datetime, timezone
|
||||
import hashlib
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import signal
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
import uuid
|
||||
|
||||
ROOT=Path(__file__).resolve().parents[2]
|
||||
spec=importlib.util.spec_from_file_location('custody',ROOT/'sso-mfa/bootstrap/sops-private.py')
|
||||
custody=importlib.util.module_from_spec(spec);spec.loader.exec_module(custody)
|
||||
RECIPIENT='age1zvryunvjhvpkmasskauga2heeg0ztnte9ymgppvjge36ekumk50syr3tsz'
|
||||
IMAGE='postgres@sha256:ff23cdce56cac62ada6f66013e1a50864c0abbe79d132d40b6e05bd80f378a70'
|
||||
OBJECT='r01-net-kingdom-pg/net-kingdom-pg-privacyidea_db-20261015T023000Z.dump.age'
|
||||
SIZE=89129309
|
||||
|
||||
|
||||
class Refused(Exception):pass
|
||||
|
||||
|
||||
def run(args,**kwargs):
|
||||
result=subprocess.run(args,stdout=subprocess.PIPE,stderr=subprocess.DEVNULL,timeout=600,**kwargs)
|
||||
if result.returncode:raise Refused()
|
||||
return result.stdout
|
||||
|
||||
|
||||
def transfer_input(receipt):
|
||||
r=json.loads(receipt.read_text())
|
||||
if (r.get('schema')!='netkingdom.privacyidea-offhost-transfer.v1' or
|
||||
r.get('status')!='ciphertext_fetched' or r.get('object')!=OBJECT or
|
||||
r.get('ciphertext_bytes')!=SIZE or r.get('object_unchanged_during_transfer') is not True):raise Refused()
|
||||
path=Path(r['ciphertext_path'])
|
||||
if not path.is_absolute() or path.parent.parent!=Path('/tmp') or not path.parent.name.startswith('nk49-offhost-'):raise Refused()
|
||||
if any(p.is_symlink() for p in (path,*path.parents)):raise Refused()
|
||||
st=path.stat()
|
||||
if st.st_uid!=os.getuid() or st.st_mode&0o077 or st.st_nlink!=1 or not path.is_file() or st.st_size!=SIZE:raise Refused()
|
||||
blob=path.read_bytes()
|
||||
if len(blob)!=SIZE or hashlib.sha256(blob).hexdigest()!=r['ciphertext_sha256']:raise Refused()
|
||||
return blob,r['ciphertext_sha256']
|
||||
|
||||
|
||||
def restore(dump,container):
|
||||
# Require the exact image already present; do not pull through a mutable tag.
|
||||
image_id=run(['docker','image','inspect','--format','{{.Id}}',IMAGE]).decode().strip()
|
||||
run(['docker','run','--detach','--pull=never','--name',container,'--network','none',
|
||||
'--security-opt','no-new-privileges','--memory','2g','--cpus','2',
|
||||
'--tmpfs','/var/lib/postgresql/data:rw,nosuid,nodev,size=1073741824',
|
||||
'--env','POSTGRES_HOST_AUTH_METHOD=trust','--env','POSTGRES_DB=recovery',
|
||||
IMAGE,'postgres','-c','listen_addresses='])
|
||||
for _ in range(60):
|
||||
# The image starts a temporary server during initdb. Wait until the
|
||||
# entrypoint has exec'd the final server before attempting the restore.
|
||||
pid1=subprocess.run(['docker','exec',container,'cat','/proc/1/comm'],
|
||||
stdout=subprocess.PIPE,stderr=subprocess.DEVNULL,timeout=10)
|
||||
if pid1.returncode or pid1.stdout.strip()!=b'postgres':
|
||||
time.sleep(1);continue
|
||||
ready=subprocess.run(['docker','exec',container,'pg_isready','-U','postgres','-d','recovery'],
|
||||
stdout=subprocess.DEVNULL,stderr=subprocess.DEVNULL,timeout=10)
|
||||
if ready.returncode==0:break
|
||||
time.sleep(1)
|
||||
else:raise Refused()
|
||||
version=run(['docker','exec',container,'pg_restore','--version']).decode().strip()
|
||||
with dump.open('rb') as source:
|
||||
run(['docker','exec','-i',container,'pg_restore','-U','postgres','-d','recovery',
|
||||
'--no-owner','--no-privileges','--exit-on-error'],stdin=source)
|
||||
# Fixed aggregate queries only. No identity, resolver config or token values.
|
||||
counts={}
|
||||
for table in ('token','resolver','realm'):
|
||||
raw=run(['docker','exec',container,'psql','-X','-U','postgres','-d','recovery','-At',
|
||||
'-v','ON_ERROR_STOP=1','-c',f'SELECT count(*) FROM public.{table};']).strip()
|
||||
if not raw.isdigit():raise Refused()
|
||||
counts[table]=int(raw)
|
||||
return {'image':IMAGE,'image_id':image_id,'pg_restore_version':version,'aggregate_row_counts':counts}
|
||||
|
||||
|
||||
def main():
|
||||
p=argparse.ArgumentParser(description=__doc__)
|
||||
p.add_argument('--transfer-receipt',required=True);p.add_argument('--receipt',required=True)
|
||||
a=p.parse_args();os.umask(0o077)
|
||||
# Reserve the receipt before any recovery action; never overwrite evidence.
|
||||
fd=os.open(a.receipt,os.O_WRONLY|os.O_CREAT|os.O_EXCL|os.O_NOFOLLOW,0o600)
|
||||
r={'schema':'netkingdom.privacyidea-database-drill.v1','status':'failed',
|
||||
'started_at':datetime.now(timezone.utc).isoformat(),'source_sha256':hashlib.sha256(Path(__file__).read_bytes()).hexdigest(),
|
||||
'production_contacted':False,'secret_values_exported':False,'native_identity_recovery_proven':False,
|
||||
'token_decryption_proven':False,'audit_signature_continuity_proven':False,
|
||||
'full_identity_file_coverage_proven':False,'container_cleanup_verified':False,
|
||||
'temporary_plaintext_cleanup_verified':False}
|
||||
container='nk49-pi-recovery-'+uuid.uuid4().hex
|
||||
keyfd=None;work=None
|
||||
try:
|
||||
blob,digest=transfer_input(Path(a.transfer_receipt));r['ciphertext_sha256']=digest
|
||||
keyfd=custody.private_key_fd()
|
||||
derived=run(['age-keygen','-y',f'/proc/self/fd/{keyfd}'],pass_fds=(keyfd,)).strip()
|
||||
if derived!=RECIPIENT.encode():raise Refused()
|
||||
r['registered_recipient_match']=True
|
||||
with tempfile.TemporaryDirectory(prefix='nk49-pi-drill-',dir='/tmp') as temporary:
|
||||
work=Path(temporary)
|
||||
env={'PATH':os.environ.get('PATH',os.defpath),'HOME':temporary,'XDG_CONFIG_HOME':temporary}
|
||||
wrong=work/'wrong.key'
|
||||
run(['age-keygen','-o',str(wrong)],env=env)
|
||||
negative=subprocess.run(['age','-d','-i',str(wrong)],input=blob,stdout=subprocess.DEVNULL,
|
||||
stderr=subprocess.DEVNULL,timeout=60,env=env)
|
||||
if negative.returncode==0:raise Refused()
|
||||
r['wrong_key_refused']=True
|
||||
dump=work/'privacyidea.dump'
|
||||
with dump.open('xb') as output:
|
||||
result=subprocess.run(['age','-d','-i',f'/proc/self/fd/{keyfd}'],input=blob,
|
||||
stdout=output,stderr=subprocess.DEVNULL,timeout=120,pass_fds=(keyfd,),env=env)
|
||||
if result.returncode or not 5<dump.stat().st_size<SIZE:raise Refused()
|
||||
with dump.open('rb') as source:
|
||||
if source.read(5)!=b'PGDMP':raise Refused()
|
||||
r['offhost_ciphertext_decryption_verified']=True
|
||||
try:
|
||||
r.update(restore(dump,container));r['database_restore_and_readback_passed']=True
|
||||
finally:
|
||||
removed=subprocess.run(['docker','rm','--force','--volumes',container],stdout=subprocess.DEVNULL,
|
||||
stderr=subprocess.DEVNULL,timeout=60)
|
||||
# A removal error is a failed ceremony even if the restore passed.
|
||||
if removed.returncode:raise Refused()
|
||||
absent=subprocess.run(['docker','container','inspect',container],stdout=subprocess.DEVNULL,
|
||||
stderr=subprocess.DEVNULL,timeout=15)
|
||||
# Confirm the daemon remains reachable, so inspect failure means absent.
|
||||
run(['docker','info','--format','{{.ServerVersion}}'])
|
||||
if absent.returncode==0:raise Refused()
|
||||
r['container_cleanup_verified']=True
|
||||
if work.exists():raise Refused()
|
||||
r['temporary_plaintext_cleanup_verified']=True;r['status']='isolated_database_drill_passed'
|
||||
except Exception:
|
||||
r['error']='database_drill_failed_diagnostics_suppressed'
|
||||
finally:
|
||||
if keyfd is not None:os.close(keyfd)
|
||||
if work is not None and not work.exists():r['temporary_plaintext_cleanup_verified']=True
|
||||
r['finished_at']=datetime.now(timezone.utc).isoformat()
|
||||
with os.fdopen(fd,'w') as out:json.dump(r,out,indent=2);out.write('\n')
|
||||
if r['status']=='isolated_database_drill_passed':
|
||||
print('OK: off-host database decrypted, isolated restore/readback passed, disposable data removed. Full identity recovery remains open.')
|
||||
return 0
|
||||
print('ERROR: database drill failed; private diagnostics suppressed. Retain the safe receipt for review.',file=sys.stderr)
|
||||
return 1
|
||||
|
||||
|
||||
if __name__=='__main__':
|
||||
for sig in (signal.SIGHUP,signal.SIGTERM,signal.SIGINT):signal.signal(sig,custody.interrupted)
|
||||
raise SystemExit(main())
|
||||
88
tools/bootstrap-recovery/fetch-privacyidea-backup.py
Normal file
88
tools/bootstrap-recovery/fetch-privacyidea-backup.py
Normal file
|
|
@ -0,0 +1,88 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Fetch one fixed ciphertext through Platform's attended Backup-account lane.
|
||||
|
||||
No key read, plaintext, redirects, proxies, grants, deletion or production writes.
|
||||
The object and byte count are bound to the reviewed inventory. Provider diagnostics
|
||||
and credentials remain private. Output is a fresh private directory under /tmp.
|
||||
"""
|
||||
import argparse
|
||||
import base64
|
||||
from datetime import datetime, timezone
|
||||
import hashlib
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import tempfile
|
||||
import urllib.request
|
||||
|
||||
HERE=Path(__file__).resolve().parent
|
||||
spec=importlib.util.spec_from_file_location('inventory',HERE/'inventory-storage-share.py')
|
||||
owner=importlib.util.module_from_spec(spec);spec.loader.exec_module(owner)
|
||||
OBJECT='r01-net-kingdom-pg/net-kingdom-pg-privacyidea_db-20261015T023000Z.dump.age'
|
||||
SIZE=89129309
|
||||
|
||||
|
||||
def metadata(response,url):
|
||||
etag=response.headers.get('ETag','')
|
||||
if (response.status!=200 or response.url!=url or
|
||||
response.headers.get('Content-Length')!=str(SIZE) or
|
||||
len(etag)>200 or not etag.startswith('"') or not etag.endswith('"') or
|
||||
any(ord(c)<32 for c in etag)):
|
||||
raise owner.Refused()
|
||||
return etag
|
||||
|
||||
|
||||
def transfer(opener,auth,destination):
|
||||
url=owner.HOST+owner.BASE+OBJECT
|
||||
headers={'Authorization':'Basic '+auth,'Accept-Encoding':'identity'}
|
||||
with opener.open(urllib.request.Request(url,method='HEAD',headers=headers),timeout=30) as response:
|
||||
etag=metadata(response,url)
|
||||
headers['If-Match']=etag
|
||||
digest=hashlib.sha256();count=0
|
||||
with opener.open(urllib.request.Request(url,headers=headers),timeout=60) as response:
|
||||
if metadata(response,url)!=etag:raise owner.Refused()
|
||||
fd=os.open(destination,os.O_WRONLY|os.O_CREAT|os.O_EXCL|os.O_NOFOLLOW,0o600)
|
||||
with os.fdopen(fd,'wb') as out:
|
||||
while chunk:=response.read(1024*1024):
|
||||
count+=len(chunk)
|
||||
if count>SIZE:raise owner.Refused()
|
||||
digest.update(chunk);out.write(chunk)
|
||||
out.flush();os.fsync(out.fileno())
|
||||
if count!=SIZE:raise owner.Refused()
|
||||
with opener.open(urllib.request.Request(url,method='HEAD',headers=headers),timeout=30) as response:
|
||||
if metadata(response,url)!=etag:raise owner.Refused()
|
||||
# ETag used for request binding only; it is not asserted to be a content hash.
|
||||
return {'ciphertext_bytes':count,'ciphertext_sha256':digest.hexdigest(),
|
||||
'object_unchanged_during_transfer':True}
|
||||
|
||||
|
||||
def main():
|
||||
p=argparse.ArgumentParser(description=__doc__);p.add_argument('--receipt',required=True)
|
||||
a=p.parse_args();os.umask(0o077)
|
||||
fd=os.open(a.receipt,os.O_WRONLY|os.O_CREAT|os.O_EXCL|os.O_NOFOLLOW,0o600)
|
||||
r={'schema':'netkingdom.privacyidea-offhost-transfer.v1','status':'failed',
|
||||
'started_at':datetime.now(timezone.utc).isoformat(),'provider':'Storage Share',
|
||||
'object':OBJECT,'age_keys_read':False,'plaintext_created':False,
|
||||
'production_writes':False,'source_sha256':hashlib.sha256(Path(__file__).read_bytes()).hexdigest(),
|
||||
'owner_accessor_sha256':hashlib.sha256((HERE/'inventory-storage-share.py').read_bytes()).hexdigest()}
|
||||
destination=None
|
||||
try:
|
||||
identity,credentials=owner.caller_credentials()
|
||||
r['caller_auth_path']=identity['path'];r['caller_entity_id']=identity.get('entity_id')
|
||||
auth=base64.b64encode((credentials['BACKUP_USERNAME']+':'+credentials['BACKUP_PASSWORD']).encode()).decode()
|
||||
destination=Path(tempfile.mkdtemp(prefix='nk49-offhost-',dir='/tmp'))/'privacyidea.dump.age'
|
||||
opener=urllib.request.build_opener(urllib.request.ProxyHandler({}),owner.NoRedirect())
|
||||
r.update(transfer(opener,auth,destination))
|
||||
r.update(status='ciphertext_fetched',ciphertext_path=str(destination))
|
||||
except Exception:
|
||||
r['error']='bounded_transfer_failed_diagnostics_suppressed'
|
||||
if destination is not None:
|
||||
destination.unlink(missing_ok=True);destination.parent.rmdir()
|
||||
finally:
|
||||
r['finished_at']=datetime.now(timezone.utc).isoformat()
|
||||
with os.fdopen(fd,'w') as out:json.dump(r,out,indent=2);out.write('\n')
|
||||
return int(r['status']=='failed')
|
||||
|
||||
|
||||
if __name__=='__main__':raise SystemExit(main())
|
||||
|
|
@ -74,13 +74,18 @@ def listing(relative,credentials):
|
|||
return parse_listing(raw,relative)
|
||||
|
||||
|
||||
def inventory():
|
||||
def caller_credentials():
|
||||
# Lookup payload contains a token: retain only safe caller/role facts.
|
||||
identity=data(bao(['token','lookup','-format=json']))['data']
|
||||
if 'platform-admin' not in identity.get('policies',[]) or 'root' in identity.get('policies',[]):raise Refused()
|
||||
if not str(identity.get('path','')).startswith('auth/netkingdom/'):raise Refused()
|
||||
credentials=data(bao(['read','-format=json',OPERATOR_PATH]))['data']['data']
|
||||
if credentials.get('BACKUP_USERNAME')!='Backup' or not isinstance(credentials.get('BACKUP_PASSWORD'),str) or not credentials['BACKUP_PASSWORD']:raise Refused()
|
||||
return identity,credentials
|
||||
|
||||
|
||||
def inventory():
|
||||
identity,credentials=caller_credentials()
|
||||
root=listing('',credentials)
|
||||
folders={}
|
||||
for row in root:
|
||||
|
|
|
|||
|
|
@ -8,7 +8,7 @@ status: active
|
|||
owner: codex
|
||||
topic_slug: netkingdom
|
||||
created: "2026-10-12"
|
||||
updated: "2026-10-15"
|
||||
updated: "2026-10-16"
|
||||
flavor: implementation
|
||||
depends_on: []
|
||||
state_hub_workstream_id: "17d315f0-829e-57c7-85cc-3a7914152804"
|
||||
|
|
@ -459,6 +459,23 @@ bootstrap `keys/age.pub` is not assumed to be the same recipient. T02/T03 remain
|
|||
open. Evidence: `offsite-discovery-summary.json` and the two exact inventory
|
||||
receipts in the existing October15 evidence folder.
|
||||
|
||||
2026-10-16 custody/retrieval checkpoint: operator reports the hidden check
|
||||
matched the backup recipient and removed the temporary key. No private key was
|
||||
provided to the agent. The exact existing PI database ciphertext was fetched
|
||||
through the attended Platform Backup-account procedure, with strong ETag binding,
|
||||
89,129,309 bytes and SHA256
|
||||
`cc81da442755928171d2c51bce93eda7de4392a0a1aa9d0ffb83a25ba34e5ead`.
|
||||
The session exited0 and self-revoked. Ciphertext is private temporary storage;
|
||||
no plaintext or age key was read by the fetch. Prepared the exact human custody
|
||||
command in the recovery ceremony for an isolated, pinned PostgreSQL restore
|
||||
with no network, ports or host mounts; fixed aggregate readback and verified
|
||||
container/plaintext cleanup.23 focused tests passed including actual local
|
||||
PostgreSQL restore using synthetic data and a fresh fixture age key. Actual
|
||||
backup decryption remains pending the operator command. This is partial database
|
||||
recovery preparation; full matching identity key/config coverage, service
|
||||
continuity, T02 owner acceptance and legacy credential disposition remain open.
|
||||
Evidence: `privacyidea-transfer-r1.json`, `database-drill-preparation.json`.
|
||||
|
||||
## Verify owner-approved recovery and current native callers
|
||||
|
||||
```task
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue