Diagnose attended custody finish failures with safe staged receipts
Assistant: codex Assistant-Model: gpt-6.1-sol Assistant-Session: 01a1029d-2c93-7500-a0e0-e3e089668b29
This commit is contained in:
parent
b8f8a7ada6
commit
33ff82210e
8 changed files with 397 additions and 34 deletions
|
|
@ -139,3 +139,25 @@ approval applies; operator OIDC/MFA remains execution-only. Fresh receipt must
|
|||
show success. If the exposed historical local grant matches the current lane,
|
||||
coordinated existing credential recovery is required before provider proof;
|
||||
there is no automatic credential rotation or root authority change.
|
||||
|
||||
## Failed attended finish and diagnostic correction
|
||||
|
||||
Attempt185421Z333977 completed and revoked its operator login, but the finish
|
||||
child failed without a receipt. Owner audit proves custody, backup-lane and
|
||||
operator-backup reads all succeeded. The exact external-provider failure is
|
||||
unknown because the previous catch-all discarded diagnostics; no IAM widening
|
||||
or speculative protocol/status relaxation is warranted. Safe audit receipt:
|
||||
2026-10-03-root-custody-failed-finish-audit.json.
|
||||
|
||||
The corrected silent wrapper runs a stdlib bootstrap that reserves a fresh
|
||||
private receipt before importing helper dependencies. Full main emits fixed
|
||||
allowlisted stage/error codes and safe HTTP/artifact metadata on failure,
|
||||
including partial owner-baseline success and any attempted created artifact.
|
||||
It never emits exception text, raw response, credential or provider coordinates.
|
||||
Private clean-environment full-main tests use the actual attended envelope
|
||||
predicate, real validator/system age and fake owner/provider boundaries; they
|
||||
cover success, denied owner read, quota, missing binary, upload, download,
|
||||
ciphertext/decrypt failure, startup imports, stale receipt and incident stop.
|
||||
A new fresh-path attended attempt can now identify the actual provider boundary
|
||||
without guessing or repeating opaque failure. Actual provider acceptance remains
|
||||
pending; simulated successful provider tests do not establish it.
|
||||
|
|
|
|||
|
|
@ -0,0 +1,28 @@
|
|||
{
|
||||
"events": [
|
||||
{
|
||||
"error_present": false,
|
||||
"permission_denied": false,
|
||||
"request_id": "b29f3261-7fd3-84e9-7aa4-493205e469a4",
|
||||
"stage": "custody_read",
|
||||
"time": "2026-10-03T18:54:37.880715204Z"
|
||||
},
|
||||
{
|
||||
"error_present": false,
|
||||
"permission_denied": false,
|
||||
"request_id": "e474bc4b-9265-f3d9-d0f3-cdf6ec7eb038",
|
||||
"stage": "backup_lane_read",
|
||||
"time": "2026-10-03T18:54:38.14371374Z"
|
||||
},
|
||||
{
|
||||
"error_present": false,
|
||||
"permission_denied": false,
|
||||
"request_id": "d4c3df09-30e0-36f7-c66f-8d000e23dfe2",
|
||||
"stage": "operator_backup_read",
|
||||
"time": "2026-10-03T18:54:38.4374245Z"
|
||||
}
|
||||
],
|
||||
"lookup_succeeded": true,
|
||||
"raw_logs_emitted": false,
|
||||
"scope": "failed-attended-finish-stage-audit"
|
||||
}
|
||||
|
|
@ -192,3 +192,27 @@ upload and requires coordinated existing credential incident/cutover recovery,
|
|||
preserving consumers; it never rotates an uninformed predecessor. If different,
|
||||
the historical owner's revocation status remains unverified separately from
|
||||
root-grant authority. No credential-bearing coordinates are copied to evidence.
|
||||
|
||||
## Finish failure receipts
|
||||
|
||||
The first finish attempt failed after successful authenticated owner reads but
|
||||
before a receipt. Its exact provider cause was lost by the old catch-all; it is
|
||||
not evidence of missing IAM. The corrected wrapper reserves a fresh receipt
|
||||
before helper imports and always records safe stage/error codes. Inspect that
|
||||
receipt after execution even if the child returns nonzero. Failed upload or
|
||||
readback includes safe numeric HTTP status and generated artifact name when
|
||||
available; do not rerun with the same receipt path or weaken expected provider
|
||||
contract/status checks blindly. No token, share coordinates, private key, raw
|
||||
response or exception message is written. `helper_import` means startup failed;
|
||||
`backup_operator_read` means current owner credential retrieval failed;
|
||||
`provider_contract`, crypto, quota, upload/download or ciphertext/decrypt stages
|
||||
identify the remaining provider proof boundary. Login cleanup and task success
|
||||
remain separate, and only a verified fresh success receipt completes T05.
|
||||
|
||||
The finish helper also uses the established independent operator escrow at
|
||||
/home/worsch/.config/age/railiance-backup.key. It derives that key's public
|
||||
recipient in a captured subprocess, compares it with the current governed lane
|
||||
and decrypts the downloaded artifact with the independent key file. Only match
|
||||
and decryption booleans are recorded; the private key is never read into logs
|
||||
or replaced. A missing/different escrow is an explicit fixed-code recovery
|
||||
failure, not a request to create or reset a key implicitly.
|
||||
|
|
|
|||
|
|
@ -1,3 +1,3 @@
|
|||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
exec python3 "$(dirname "$0")/root_custody_finish.py" "$@" >/dev/null 2>&1
|
||||
exec /usr/bin/python3 "$(dirname "$0")/root_custody_finish_entry.py" "$@" >/dev/null 2>&1
|
||||
|
|
|
|||
|
|
@ -19,22 +19,23 @@ from migrate_nextcloud_backup_account import HOST, OPERATOR_PATH, LANE, quota, r
|
|||
|
||||
VALIDATOR_DIGEST='9156e399c8d92a45b817bfc3738c89843487ce57e1718ed747f804f95cc719bb'
|
||||
SCHEMA_DIGEST='bb6cd6da5dc48eddcf118ae78c2cacfafd5f860098fd8a8368fe6bc845776d07'
|
||||
INDEPENDENT_ESCROW=Path('/home/worsch/.config/age/railiance-backup.key')
|
||||
|
||||
|
||||
def checked_bundle(state_path, validator):
|
||||
state=json.loads(state_path.read_text())
|
||||
if state.get('pending') or validator.validate(state['document']) or state['record_id']!=RECORD_ID or digest(state['document'])!=state['digest']:
|
||||
raise ValueError('retained_state_invalid')
|
||||
raise FinishFailure('retained_state_invalid')
|
||||
history=[]
|
||||
for path in sorted((state_path.parent/'history').glob('*.json')):
|
||||
if path.is_symlink() or path.stat().st_size>262144:
|
||||
raise ValueError('history_invalid')
|
||||
raise FinishFailure('history_invalid')
|
||||
receipt=json.loads(path.read_text())
|
||||
if validator.validate(receipt['document']) or receipt['record_id']!=RECORD_ID or receipt['digest']!=digest(receipt['document']):
|
||||
raise ValueError('history_invalid')
|
||||
raise FinishFailure('history_invalid')
|
||||
history.append(receipt)
|
||||
if state not in history:
|
||||
raise ValueError('latest_history_missing')
|
||||
raise FinishFailure('latest_history_missing')
|
||||
return state,json.dumps({'baseline':state,'history':history},sort_keys=True,separators=(',',':')).encode()
|
||||
|
||||
|
||||
|
|
@ -54,78 +55,169 @@ def historical_config_comparison(config_path, values):
|
|||
"historical_credential_status":"comparison-unavailable"}
|
||||
|
||||
|
||||
def roundtrip(bundle, lane, operator, network=request, get_quota=quota):
|
||||
def roundtrip(bundle, lane, operator, network=None, get_quota=None, progress=lambda stage, **details: None):
|
||||
network=request if network is None else network
|
||||
get_quota=quota if get_quota is None else get_quota
|
||||
progress('provider_contract')
|
||||
values=lane['data']
|
||||
if operator['BACKUP_USERNAME']!='Backup':
|
||||
raise ValueError('governed_owner_mismatch')
|
||||
raise FinishFailure('governed_owner_mismatch')
|
||||
auth=(operator['BACKUP_USERNAME'],operator['BACKUP_PASSWORD'])
|
||||
upload=values['NC_WEBDAV_URL']
|
||||
if not upload.startswith(HOST+'/public.php/dav/filesdrop/') or values['NC_WEBDAV_TOKEN']!=upload.rsplit('/',1)[-1]:
|
||||
raise ValueError('current_lane_contract_mismatch')
|
||||
raise FinishFailure('current_lane_contract_mismatch')
|
||||
key=(values['AGE_PRIVATE_KEY'].strip()+'\n').encode()
|
||||
def process(command,payload):
|
||||
result=subprocess.run(command,input=payload,capture_output=True,timeout=30)
|
||||
try:
|
||||
result=subprocess.run(command,input=payload,capture_output=True,timeout=30)
|
||||
except FileNotFoundError:
|
||||
raise FinishFailure('crypto_binary_unavailable') from None
|
||||
except subprocess.TimeoutExpired:
|
||||
raise FinishFailure('crypto_command_timeout') from None
|
||||
if result.returncode:
|
||||
raise ValueError('protected_process_failed')
|
||||
raise FinishFailure('crypto_command_failed')
|
||||
return result.stdout
|
||||
progress('derive_age_recipient')
|
||||
recipient=process(['age-keygen','-y','/dev/stdin'],key).decode().strip()
|
||||
progress('encrypt_history')
|
||||
encrypted=process(['age','-r',recipient],bundle)
|
||||
progress('independent_key_escrow')
|
||||
if not INDEPENDENT_ESCROW.is_file():
|
||||
raise FinishFailure('independent_key_escrow_missing')
|
||||
independent_recipient=process(['age-keygen','-y',str(INDEPENDENT_ESCROW)],b'').decode().strip()
|
||||
matched=hmac.compare_digest(recipient,independent_recipient)
|
||||
progress('independent_key_escrow',independent_escrow_recipient_matches_current_lane=matched)
|
||||
if not matched:
|
||||
raise FinishFailure('independent_key_escrow_mismatch')
|
||||
progress('provider_quota')
|
||||
if len(encrypted)>1048576 or get_quota(auth)['available_bytes']<len(encrypted):
|
||||
raise ValueError('backup_size_or_quota_invalid')
|
||||
raise FinishFailure('backup_size_or_quota_invalid')
|
||||
name='root-custody-history-'+datetime.now(timezone.utc).strftime('%Y%m%dT%H%M%SZ')+'-'+uuid.uuid4().hex+'.json.age'
|
||||
progress('provider_upload',artifact=name)
|
||||
status,_=network(upload+'/forgejo/'+name,'PUT',encrypted,(values['NC_WEBDAV_TOKEN'],''),{'Content-Type':'application/octet-stream','If-None-Match':'*'})
|
||||
progress('provider_upload',upload_http_status=status)
|
||||
if status!=201:
|
||||
raise ValueError('new_artifact_upload_failed')
|
||||
raise FinishFailure('new_artifact_upload_failed')
|
||||
progress('provider_download')
|
||||
status,fetched=network(HOST+'/remote.php/dav/files/Backup/railiance-backups/forgejo/'+name,'GET',auth=auth)
|
||||
if status!=200 or hashlib.sha256(fetched).digest()!=hashlib.sha256(encrypted).digest():
|
||||
raise ValueError('provider_readback_mismatch')
|
||||
progress('provider_download',readback_http_status=status)
|
||||
if status!=200:
|
||||
raise FinishFailure('provider_download_failed')
|
||||
progress('provider_ciphertext_check')
|
||||
if hashlib.sha256(fetched).digest()!=hashlib.sha256(encrypted).digest():
|
||||
raise FinishFailure('provider_ciphertext_mismatch')
|
||||
progress('decrypt_and_compare_history')
|
||||
with tempfile.TemporaryDirectory(prefix='root-custody-recovery-') as tmp:
|
||||
cipher=Path(tmp)/'history.json.age'
|
||||
cipher.write_bytes(fetched)
|
||||
cipher.chmod(0o600)
|
||||
restored=process(['age','-d','-i','/dev/stdin',str(cipher)],key)
|
||||
restored=process(['age','-d','-i',str(INDEPENDENT_ESCROW),str(cipher)],b'')
|
||||
if restored!=bundle:
|
||||
raise ValueError('decrypted_history_mismatch')
|
||||
return {'provider':'existing-governed-nextcloud-backup','artifact':name,'upload_http_status':201,'readback_http_status':200,'ciphertext_sha256':hashlib.sha256(encrypted).hexdigest(),'decrypted_history_matches':True,'lane_kv_version':lane['metadata']['version'],'local_historical_rclone_used':False}
|
||||
raise FinishFailure('decrypted_history_mismatch')
|
||||
return {'provider':'existing-governed-nextcloud-backup','artifact':name,'upload_http_status':201,'readback_http_status':200,'ciphertext_sha256':hashlib.sha256(encrypted).hexdigest(),'decrypted_history_matches':True,'lane_kv_version':lane['metadata']['version'],'local_historical_rclone_used':False,'independent_escrow_recipient_matches_current_lane':True,'independent_escrow_decryption_verified':True}
|
||||
|
||||
|
||||
def main():
|
||||
STAGES = {
|
||||
'startup', 'attended_envelope', 'operator_endpoint', 'canon_pins', 'validator_import',
|
||||
'publisher_lock', 'retained_history', 'current_source_read', 'current_baseline_check',
|
||||
'backup_lane_read', 'backup_operator_read', 'historical_share_comparison',
|
||||
'provider_contract', 'derive_age_recipient', 'encrypt_history', 'independent_key_escrow', 'provider_quota',
|
||||
'provider_upload', 'provider_download', 'provider_ciphertext_check', 'decrypt_and_compare_history', 'complete',
|
||||
}
|
||||
|
||||
|
||||
class FinishFailure(ValueError):
|
||||
def __init__(self, code):
|
||||
self.code = code
|
||||
|
||||
|
||||
ERROR_CODES = {'stage_failed', 'attended_envelope_required', 'approved_operator_endpoint_required',
|
||||
'canon_pin_mismatch', 'current_backup_lane_incident', 'unexpected_stage',
|
||||
'crypto_binary_unavailable','crypto_command_timeout','crypto_command_failed','independent_key_escrow_missing','independent_key_escrow_mismatch',
|
||||
'provider_download_failed','provider_ciphertext_mismatch','decrypted_history_mismatch',
|
||||
'retained_state_invalid','history_invalid','latest_history_missing',
|
||||
'governed_owner_mismatch','current_lane_contract_mismatch','backup_size_or_quota_invalid','new_artifact_upload_failed'}
|
||||
|
||||
|
||||
def main(receipt_reserved=False):
|
||||
p=argparse.ArgumentParser()
|
||||
p.add_argument('--state',type=Path,required=True)
|
||||
p.add_argument('--net-kingdom',type=Path,required=True)
|
||||
p.add_argument('--receipt',type=Path,required=True)
|
||||
args=p.parse_args()
|
||||
result={'ok':False,'stage':'startup','status':'started',
|
||||
'recorded_at':datetime.now(timezone.utc).isoformat()}
|
||||
def progress(stage, **details):
|
||||
if stage not in STAGES:
|
||||
raise FinishFailure('unexpected_stage')
|
||||
result['stage']=stage
|
||||
for key,value in details.items():
|
||||
if key in {'upload_http_status','readback_http_status'} and type(value) is int and 100 <= value <= 599:
|
||||
result[key]=value
|
||||
elif key=='independent_escrow_recipient_matches_current_lane' and type(value) is bool:
|
||||
result[key]=value
|
||||
elif key=='artifact' and isinstance(value,str) and value.startswith('root-custody-history-') and value.endswith('.json.age'):
|
||||
result[key]=value
|
||||
atomic_json(args.receipt,result)
|
||||
try:
|
||||
if args.receipt.exists() or args.receipt.is_symlink():
|
||||
if args.receipt.is_symlink():
|
||||
return 1
|
||||
require_attended()
|
||||
if not receipt_reserved:
|
||||
fd=os.open(args.receipt,os.O_CREAT|os.O_EXCL|os.O_WRONLY,0o600)
|
||||
os.close(fd)
|
||||
progress('attended_envelope')
|
||||
try:
|
||||
require_attended()
|
||||
except Exception:
|
||||
raise FinishFailure('attended_envelope_required') from None
|
||||
progress('operator_endpoint')
|
||||
if os.getenv('BAO_ADDR')!='http://127.0.0.1:18200':
|
||||
return 1
|
||||
raise FinishFailure('approved_operator_endpoint_required')
|
||||
progress('canon_pins')
|
||||
source=args.net_kingdom/'tools/custody-record/validate.py'
|
||||
schema=args.net_kingdom/'canon/schemas/custody-record_v0.1.schema.json'
|
||||
if hashlib.sha256(source.read_bytes()).hexdigest()!=VALIDATOR_DIGEST or hashlib.sha256(schema.read_bytes()).hexdigest()!=SCHEMA_DIGEST:
|
||||
return 1
|
||||
raise FinishFailure('canon_pin_mismatch')
|
||||
progress('validator_import')
|
||||
spec=importlib.util.spec_from_file_location('custody_validator',source)
|
||||
validator=importlib.util.module_from_spec(spec);spec.loader.exec_module(validator)
|
||||
import fcntl
|
||||
progress('publisher_lock')
|
||||
with (args.state.parent/'publisher.lock').open('a') as lock:
|
||||
fcntl.flock(lock,fcntl.LOCK_EX)
|
||||
progress('retained_history')
|
||||
state,bundle=checked_bundle(args.state,validator)
|
||||
bao=Bao()
|
||||
check_baseline(bao.read('platform/data/custody/net-kingdom/platform-root'),state,validator)
|
||||
# Only current owner-held lane; no cached rclone config or caller token.
|
||||
progress('current_source_read')
|
||||
actual=bao.read('platform/data/custody/net-kingdom/platform-root')
|
||||
progress('current_baseline_check')
|
||||
check_baseline(actual,state,validator)
|
||||
result.update(record_id=RECORD_ID,revision=state['revision'],kv_version=state['kv_version'],
|
||||
digest=state['digest'],current_live_baseline_matches=True)
|
||||
progress('backup_lane_read')
|
||||
lane=bao.read(LANE)
|
||||
progress('backup_operator_read')
|
||||
operator=bao.read(OPERATOR_PATH)['data']
|
||||
progress('historical_share_comparison')
|
||||
comparison=historical_config_comparison(Path('/home/worsch/.config/rclone/rclone.conf'),lane['data'])
|
||||
result.update(comparison)
|
||||
if comparison['historical_share_matches_current_lane'] is True:
|
||||
atomic_json(args.receipt,{'ok':False,'current_live_baseline_matches':True,**comparison,
|
||||
'error':'current_backup_lane_requires_coordinated_existing_incident_recovery','recorded_at':datetime.now(timezone.utc).isoformat()})
|
||||
return 1
|
||||
receipt={**roundtrip(bundle,lane,operator),**comparison}
|
||||
atomic_json(args.receipt,{'ok':True,'scope':'current-owner-read-and-governed-history-provider-roundtrip','record_id':RECORD_ID,'revision':state['revision'],'kv_version':state['kv_version'],'digest':state['digest'],'current_live_baseline_matches':True,'history_entries':len(json.loads(bundle)['history']),'recorded_at':datetime.now(timezone.utc).isoformat(),**receipt})
|
||||
raise FinishFailure('current_backup_lane_incident')
|
||||
receipt=roundtrip(bundle,lane,operator,progress=progress)
|
||||
result.update(ok=True,status='verified',stage='complete',finished_at=datetime.now(timezone.utc).isoformat(),
|
||||
scope='current-owner-read-and-governed-history-provider-roundtrip',
|
||||
history_entries=len(json.loads(bundle)['history']),**receipt)
|
||||
atomic_json(args.receipt,result)
|
||||
return 0
|
||||
except Exception:
|
||||
# Server/credential/HTTP/process diagnostics are never disclosed.
|
||||
except FileExistsError:
|
||||
# Never change an old success receipt or claim it belongs to this attempt.
|
||||
return 1
|
||||
except Exception as error:
|
||||
result.update(status='failed',finished_at=datetime.now(timezone.utc).isoformat(),error_code=error.code if isinstance(error,FinishFailure) and error.code in ERROR_CODES else 'stage_failed')
|
||||
try:
|
||||
atomic_json(args.receipt,result)
|
||||
except Exception:
|
||||
pass
|
||||
return 1
|
||||
|
||||
|
||||
|
|
|
|||
40
scripts/root_custody_finish_entry.py
Normal file
40
scripts/root_custody_finish_entry.py
Normal file
|
|
@ -0,0 +1,40 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Stdlib-only silent bootstrap; reserves a new safe receipt before imports."""
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import sys
|
||||
from datetime import datetime,timezone
|
||||
|
||||
|
||||
def main():
|
||||
receipt=None
|
||||
fd=None
|
||||
try:
|
||||
for index,arg in enumerate(sys.argv[1:],start=1):
|
||||
if arg=='--receipt' and index+1<len(sys.argv):
|
||||
receipt=Path(sys.argv[index+1])
|
||||
elif arg.startswith('--receipt='):
|
||||
receipt=Path(arg.split('=',1)[1])
|
||||
if receipt is None or receipt.is_symlink():
|
||||
return 1
|
||||
fd=os.open(receipt,os.O_CREAT|os.O_EXCL|os.O_WRONLY,0o600)
|
||||
result={'ok':False,'status':'started','stage':'helper_import','recorded_at':datetime.now(timezone.utc).isoformat()}
|
||||
os.write(fd,json.dumps(result).encode());os.fsync(fd);os.close(fd);fd=None
|
||||
spec=importlib.util.spec_from_file_location('custody_finish',Path(__file__).with_name('root_custody_finish.py'))
|
||||
helper=importlib.util.module_from_spec(spec);spec.loader.exec_module(helper)
|
||||
return helper.main(receipt_reserved=True)
|
||||
except FileExistsError:
|
||||
return 1
|
||||
except BaseException:
|
||||
if receipt is not None:
|
||||
try:
|
||||
if fd is not None:os.close(fd)
|
||||
result={'ok':False,'status':'failed','stage':'helper_import','error_code':'helper_startup_failed','recorded_at':datetime.now(timezone.utc).isoformat()}
|
||||
with receipt.open('w') as out:json.dump(result,out);out.flush();os.fsync(out.fileno())
|
||||
except Exception:pass
|
||||
return 1
|
||||
|
||||
|
||||
if __name__=='__main__':raise SystemExit(main())
|
||||
|
|
@ -5,6 +5,8 @@ from pathlib import Path
|
|||
import sys
|
||||
import tempfile
|
||||
import subprocess
|
||||
import os
|
||||
import shutil
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
|
|
@ -51,9 +53,13 @@ class FinishTests(unittest.TestCase):
|
|||
stored['encrypted']=body
|
||||
return 201,b''
|
||||
return 200,stored['encrypted']
|
||||
receipt=finish.roundtrip(b'{"nonsecret":"receipt-fixture"}',lane,
|
||||
{'BACKUP_USERNAME':'Backup','BACKUP_PASSWORD':'DISPOSABLE-PASSWORD'},
|
||||
network=memory_provider,get_quota=lambda auth:{'available_bytes':1048576})
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
escrow=Path(tmp)/'escrow.age.key'
|
||||
escrow.write_text(key);escrow.chmod(0o600)
|
||||
with patch.object(finish,'INDEPENDENT_ESCROW',escrow):
|
||||
receipt=finish.roundtrip(b'{"nonsecret":"receipt-fixture"}',lane,
|
||||
{'BACKUP_USERNAME':'Backup','BACKUP_PASSWORD':'DISPOSABLE-PASSWORD'},
|
||||
network=memory_provider,get_quota=lambda auth:{'available_bytes':1048576})
|
||||
self.assertTrue(receipt['decrypted_history_matches'])
|
||||
self.assertNotIn(key,json.dumps(receipt))
|
||||
self.assertNotIn('DISPOSABLE-SHARE',json.dumps(receipt))
|
||||
|
|
@ -66,4 +72,144 @@ class FinishTests(unittest.TestCase):
|
|||
with self.assertRaises(ValueError):
|
||||
finish.checked_bundle(path,validator)
|
||||
|
||||
class MainIntegrationTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.temp=tempfile.TemporaryDirectory()
|
||||
self.directory=Path(self.temp.name)
|
||||
shutil.copytree(ROOT/'docs/evidence/root-custody-history',self.directory/'holder')
|
||||
self.state=self.directory/'holder/baseline.json'
|
||||
self.receipt=self.directory/'receipt.json'
|
||||
self.args=['finish','--state',str(self.state),'--net-kingdom',str(ROOT.parent/'net-kingdom'),'--receipt',str(self.receipt)]
|
||||
self.baseline=json.loads(self.state.read_text())
|
||||
key=subprocess.run(['/usr/bin/age-keygen'],capture_output=True,check=True).stdout.decode()
|
||||
self.lane={'data':{'NC_WEBDAV_URL':finish.HOST+'/public.php/dav/filesdrop/TEST-SHARE',
|
||||
'NC_WEBDAV_TOKEN':'TEST-SHARE','AGE_PRIVATE_KEY':key},'metadata':{'version':4}}
|
||||
self.operator={'data':{'BACKUP_USERNAME':'Backup','BACKUP_PASSWORD':'TEST-PASSWORD'}}
|
||||
self.escrow=self.directory/'disposable-escrow.key'
|
||||
self.escrow.write_text(key);self.escrow.chmod(0o600)
|
||||
self.stored={}
|
||||
def tearDown(self):self.temp.cleanup()
|
||||
def read(self,path):
|
||||
if path==finish.LANE:return self.lane
|
||||
if path==finish.OPERATOR_PATH:return self.operator
|
||||
return {'data':{'document':self.baseline['document']},'metadata':{'version':1}}
|
||||
def provider(self,url,method='GET',body=None,auth=None,headers=None):
|
||||
if method=='PUT':self.stored['encrypted']=body;return 201,b''
|
||||
return 200,self.stored['encrypted']
|
||||
def invoke(self,reader=None,provider=None,quota_failure=None,comparison=None):
|
||||
# The actual attended-envelope predicate runs with private fake home;
|
||||
# only owner I/O boundaries are doubled. System deps and age stay real.
|
||||
actor=self.directory/'.warden-attended-login/operator'
|
||||
with patch.object(sys,'argv',self.args), patch.object(owner.Path,'home',return_value=actor), \
|
||||
patch.dict(os.environ,{'BAO_ADDR':'http://127.0.0.1:18200','PATH':'/usr/bin:/bin'},clear=True), \
|
||||
patch.object(finish.Bao,'read',side_effect=reader or self.read), \
|
||||
patch.object(finish,'request',side_effect=provider or self.provider), \
|
||||
patch.object(finish,'INDEPENDENT_ESCROW',self.escrow), \
|
||||
patch.object(finish,'quota',side_effect=quota_failure,return_value={'available_bytes':1048576}), \
|
||||
patch.object(finish,'historical_config_comparison',return_value=comparison or {'historical_config_present':False,'historical_share_matches_current_lane':None}):
|
||||
return finish.main()
|
||||
def test_full_attended_main_success_real_crypto_fake_owner_and_provider(self):
|
||||
self.assertEqual(self.invoke(),0)
|
||||
receipt=json.loads(self.receipt.read_text())
|
||||
self.assertTrue(receipt['ok']);self.assertEqual(receipt['stage'],'complete')
|
||||
self.assertTrue(receipt['current_live_baseline_matches'])
|
||||
self.assertTrue(receipt['decrypted_history_matches'])
|
||||
self.assertEqual(receipt['upload_http_status'],201)
|
||||
self.assertNotIn('TEST-PASSWORD',self.receipt.read_text())
|
||||
def test_permission_or_backend_error_writes_safe_exact_stage(self):
|
||||
def failure(path):
|
||||
if path==finish.OPERATOR_PATH:raise RuntimeError('DO-NOT-LOG-CREDENTIAL-OR-BACKEND-TEXT')
|
||||
return self.read(path)
|
||||
self.assertEqual(self.invoke(reader=failure),1)
|
||||
receipt=json.loads(self.receipt.read_text())
|
||||
self.assertFalse(receipt['ok']);self.assertEqual(receipt['stage'],'backup_operator_read')
|
||||
self.assertTrue(receipt['current_live_baseline_matches'])
|
||||
self.assertNotIn('DO-NOT-LOG',self.receipt.read_text())
|
||||
def test_upload_failure_has_status_and_artifact_without_raw_error(self):
|
||||
self.assertEqual(self.invoke(provider=lambda *args,**kwargs:(403,b'NEVER-LOG-RESPONSE')),1)
|
||||
receipt=json.loads(self.receipt.read_text())
|
||||
self.assertEqual(receipt['stage'],'provider_upload');self.assertEqual(receipt['upload_http_status'],403)
|
||||
self.assertIn('artifact',receipt);self.assertNotIn('NEVER-LOG',self.receipt.read_text())
|
||||
def test_download_failure_preserves_successful_created_artifact(self):
|
||||
def failure(url,method='GET',body=None,auth=None,headers=None):
|
||||
return (201,b'') if method=='PUT' else (401,b'RAW-SECRET')
|
||||
self.assertEqual(self.invoke(provider=failure),1)
|
||||
receipt=json.loads(self.receipt.read_text())
|
||||
self.assertEqual(receipt['stage'],'provider_download');self.assertEqual(receipt['upload_http_status'],201)
|
||||
self.assertEqual(receipt['readback_http_status'],401);self.assertIn('artifact',receipt)
|
||||
def test_quota_failure_has_safe_stage_and_no_secret(self):
|
||||
self.assertEqual(self.invoke(quota_failure=RuntimeError('PRIVATE-QUOTA-URL')),1)
|
||||
receipt=json.loads(self.receipt.read_text())
|
||||
self.assertEqual(receipt['stage'],'provider_quota')
|
||||
self.assertNotIn('PRIVATE-QUOTA-URL',self.receipt.read_text())
|
||||
def test_missing_crypto_binary_is_identified_without_exception_text(self):
|
||||
with patch.object(finish.subprocess,'run',side_effect=FileNotFoundError('SECRET-ENV-PATH')):
|
||||
self.assertEqual(self.invoke(),1)
|
||||
receipt=json.loads(self.receipt.read_text())
|
||||
self.assertEqual(receipt['stage'],'derive_age_recipient')
|
||||
self.assertEqual(receipt['error_code'],'crypto_binary_unavailable')
|
||||
self.assertNotIn('SECRET-ENV',self.receipt.read_text())
|
||||
def test_ciphertext_mismatch_never_decrypts_or_claims_backup(self):
|
||||
def bad_download(url,method='GET',body=None,auth=None,headers=None):
|
||||
return (201,b'') if method=='PUT' else (200,b'WRONG-CIPHERTEXT')
|
||||
self.assertEqual(self.invoke(provider=bad_download),1)
|
||||
receipt=json.loads(self.receipt.read_text())
|
||||
self.assertEqual(receipt['stage'],'provider_ciphertext_check')
|
||||
self.assertEqual(receipt['error_code'],'provider_ciphertext_mismatch')
|
||||
self.assertEqual(receipt['upload_http_status'],201)
|
||||
def test_decryption_failure_reports_fixed_code_after_matching_download(self):
|
||||
original=subprocess.run
|
||||
def protected_process(command,*args,**kwargs):
|
||||
if command[0]=='age' and command[1]=='-d':
|
||||
return subprocess.CompletedProcess(command,1,b'',b'RAW-KEY-MUST-NOT-LOG')
|
||||
return original(command,*args,**kwargs)
|
||||
with patch.object(finish.subprocess,'run',side_effect=protected_process):
|
||||
self.assertEqual(self.invoke(),1)
|
||||
receipt=json.loads(self.receipt.read_text())
|
||||
self.assertEqual(receipt['stage'],'decrypt_and_compare_history')
|
||||
self.assertEqual(receipt['error_code'],'crypto_command_failed')
|
||||
self.assertEqual(receipt['upload_http_status'],201)
|
||||
self.assertNotIn('RAW-KEY-MUST-NOT-LOG',self.receipt.read_text())
|
||||
|
||||
def test_current_incident_stops_before_any_provider_action(self):
|
||||
comparison={'historical_config_present':True,'historical_share_matches_current_lane':True,
|
||||
'historical_credential_status':'current-lane-incident'}
|
||||
provider=unittest.mock.Mock()
|
||||
self.assertEqual(self.invoke(provider=provider,comparison=comparison),1)
|
||||
receipt=json.loads(self.receipt.read_text())
|
||||
self.assertEqual(receipt['stage'],'historical_share_comparison')
|
||||
self.assertEqual(receipt['error_code'],'current_backup_lane_incident')
|
||||
provider.assert_not_called()
|
||||
def test_independent_escrow_missing_does_not_upload(self):
|
||||
self.escrow.unlink()
|
||||
provider=unittest.mock.Mock()
|
||||
self.assertEqual(self.invoke(provider=provider),1)
|
||||
receipt=json.loads(self.receipt.read_text())
|
||||
self.assertEqual(receipt['stage'],'independent_key_escrow')
|
||||
self.assertEqual(receipt['error_code'],'independent_key_escrow_missing')
|
||||
provider.assert_not_called()
|
||||
|
||||
def test_failed_attended_envelope_writes_safe_receipt(self):
|
||||
with patch.object(sys,'argv',self.args),patch.object(finish,'require_attended',side_effect=RuntimeError('TOKEN-DO-NOT-LOG')):
|
||||
self.assertEqual(finish.main(),1)
|
||||
receipt=json.loads(self.receipt.read_text())
|
||||
self.assertEqual(receipt['stage'],'attended_envelope')
|
||||
self.assertEqual(receipt['error_code'],'attended_envelope_required')
|
||||
self.assertNotIn('TOKEN-DO-NOT-LOG',self.receipt.read_text())
|
||||
def test_old_success_receipt_is_preserved(self):
|
||||
self.receipt.write_text('{"ok":true}')
|
||||
self.assertEqual(self.invoke(),1)
|
||||
self.assertEqual(self.receipt.read_text(),'{"ok":true}')
|
||||
def test_bootstrap_dependency_import_failure_still_writes_receipt(self):
|
||||
entry=self.directory/'root_custody_finish_entry.py'
|
||||
shutil.copy2(ROOT/'scripts/root_custody_finish_entry.py',entry)
|
||||
(self.directory/'root_custody_finish.py').write_text('raise RuntimeError("IMPORT-MUST-NOT-LOG")')
|
||||
result=subprocess.run(['/usr/bin/python3',str(entry),*self.args[1:]],capture_output=True)
|
||||
self.assertEqual(result.returncode,1)
|
||||
receipt=json.loads(self.receipt.read_text())
|
||||
self.assertEqual(receipt['stage'],'helper_import');self.assertEqual(receipt['error_code'],'helper_startup_failed')
|
||||
self.assertEqual(result.stdout,b'');self.assertEqual(result.stderr,b'')
|
||||
self.assertNotIn('IMPORT-MUST-NOT-LOG',self.receipt.read_text())
|
||||
|
||||
|
||||
if __name__=='__main__':unittest.main()
|
||||
|
|
|
|||
|
|
@ -190,3 +190,14 @@ owner baseline and direct governed history provider roundtrip receipt. No new
|
|||
generic approval, storage service or credential is requested. If the historical
|
||||
exposed grant matches current lane, existing coordinated owner incident flow
|
||||
must resolve it before backup proof; authority remains no-grant meanwhile.
|
||||
|
||||
## Failed finish diagnosed safely — 2026-10-03
|
||||
|
||||
First attended finish child failed without a receipt. Audit verifies all three
|
||||
OpenBao reads succeeded; provider failure boundary was not retained. Corrected
|
||||
stdlib startup bootstrap and full-main stage/error receipts eliminate that gap;
|
||||
closed diagnostic allowlist preserves secret suppression. No IAM change or
|
||||
provider protocol relaxation is made. Private full-main tests with actual age
|
||||
and fake owner/provider I/O verify success and each meaningful failure boundary.
|
||||
T05 still waits on actual governed provider roundtrip; exact command unchanged,
|
||||
with a fresh receipt path mandatory.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue