the-custodian/docs/changes/netkingdom-target-2026-10-16/protected-execution-recovery/implementation-r1-to-r2.patch
codex a192d24c9a
Some checks are pending
CI Smoke / host-smoke (push) Waiting to run
CI Smoke / container-smoke (push) Waiting to run
Reject generated-only commits / generated-commit-guard (push) Waiting to run
docs: preserve reviewed NetKingdom session artifacts
2026-11-01 23:14:27 +01:00

394 lines
23 KiB
Diff

diff --git a/docs/task-execution-native-fixture.md b/docs/task-execution-native-fixture.md
index 4fb60fe..f4ff070 100644
--- a/docs/task-execution-native-fixture.md
+++ b/docs/task-execution-native-fixture.md
@@ -174,3 +174,34 @@ revision drift, namespace policy-name reuse and private-file/binding refusals.
This is source fixture proof only. Native custody, isolation, exact recovery-role
permissions, retention/encryption, clock admission, audit delivery and human scope
approval remain missing. No generic broker, production reader, grant or CLI is added.
+
+
+### Recovery R2 exact absence and authenticated capsule refinement
+
+R1 overclaimed policy absence for arbitrary 404 bodies. R2 requires exactly
+`{"errors":[]}` for both initial GET and post-DELETE GET. Other 404 bodies,
+permission denials, extra data, malformed/duplicate-key/nonfinite JSON and failed
+readback cannot confirm policy cleanup. A successful DELETE status alone cannot
+confirm absence. Recovery native policy comparison admits precisely one optional
+terminal LF, matching the retained OpenBao HCL source/readback convention; spaces,
+tabs, CR, extra terminal LF and any other byte change are refused. This is a
+narrow declared normalization, not full whitespace or semantic equivalence.
+
+Private fixture capsules now carry HMAC-SHA256 authentication under an exactly
+32-byte integrity key supplied independently by the recovery owner. The key is
+never stored in the capsule or public journal, and is not a bearer credential.
+The fixture owner retains this key independently across executor death; a fresh
+owner custody object must supply the same key. Wrong-key or damaged accessor/HCL
+bytes refuse before admission. This is integrity protection, not encryption or
+same-UID isolation; production key custody, rotation/retention and recovery actor
+selection remain explicitly unadmitted.
+
+After authentication the reader enforces exact top-level and nested binding/
+cleanup key sets and types, validated Binding/Cleanup formats and the ordered
+source-derived cleanup_specs operation digests. Duplicate JSON keys, nonfinite
+numbers, malformed phase/revision combinations and type-coercible values refuse.
+The reader bounds bytes, rejects indirect/nonprivate files and detects stat/size
+changes during its single read. A valid older prepared capsule still cannot
+prove a task token absent when durable journal state/calls reached issuance:
+that remains unknown cleanup debt. Authentication alone is not anti-rollback.
+No business or consume admission is restored and no production scope is granted.
diff --git a/scripts/task_execution_recovery.py b/scripts/task_execution_recovery.py
index b4ce2b9..ab8b8cb 100644
--- a/scripts/task_execution_recovery.py
+++ b/scripts/task_execution_recovery.py
@@ -8,6 +8,7 @@ public request/journal data. No business adapter or resumed admission is built.
from __future__ import annotations
from dataclasses import asdict, dataclass, field
import hashlib
+import hmac
import http.client
import json
import os
@@ -17,17 +18,45 @@ import subprocess
import time
import uuid
-from task_execution_journal import Binding, Journal, Refused
+from task_execution_journal import Binding, Cleanup, Journal, Refused
+from task_execution_plan import cleanup_specs
def _encoded(value):
- return json.dumps(value, sort_keys=True, separators=(',', ':')).encode()
+ return json.dumps(value, sort_keys=True, separators=(',', ':'), allow_nan=False).encode()
def _digest(value):
return hashlib.sha256(_encoded(value)).hexdigest()
+def _json_unique(pairs):
+ result={}
+ for key,value in pairs:
+ if key in result: raise Refused('duplicate private custody field')
+ result[key]=value
+ return result
+
+
+def _invalid_constant(value):
+ raise Refused('nonfinite private custody value')
+
+
+def _policy_absent(status, response):
+ # Exactly the native OpenBao empty-errors 404. Denial/malformed/error-shaped
+ # bodies never prove absence, including after a successful delete response.
+ return (status==404 and type(response) is dict and set(response)=={'errors'}
+ and type(response['errors']) is list and not response['errors'])
+
+
+def _same_policy(actual, expected):
+ # OpenBao may omit one terminal LF from HCL source. Only that equivalence
+ # is admitted: spaces, tabs, CR and additional blank lines remain meaningful.
+ if type(actual) is not str or type(expected) is not str: return False
+ strip_one=lambda value:value[:-1] if value.endswith('\n') else value
+ return strip_one(actual)==strip_one(expected)
+
+
class FixtureRecoveryCustody:
"""Independent owner-private persistent fixture custody; no public API.
@@ -36,13 +65,17 @@ class FixtureRecoveryCustody:
enforced owner storage/executor authority; this implementation refuses estate
backends and is never evidence of a production grant.
"""
- def __init__(self, directory: Path):
+ def __init__(self, directory: Path, *, integrity_key: bytes):
+ if type(integrity_key) is not bytes or len(integrity_key)!=32:
+ raise Refused('independent owner capsule integrity key required')
+ self._integrity_key=integrity_key
self.directory = directory
if directory.is_symlink() or directory.absolute() != directory.resolve():
raise Refused('indirect recovery custody directory')
Journal._private(directory.stat(), directory=True)
def _path(self, binding):
+ if type(binding) is not Binding: raise Refused('exact recovery binding required')
binding.validate()
return self.directory / (binding.operation_id + '.private.json')
@@ -52,30 +85,59 @@ class FixtureRecoveryCustody:
finally: os.close(fd)
def read(self, binding):
- fd = os.open(self._path(binding), os.O_RDONLY | os.O_NOFOLLOW)
+ try: fd = os.open(self._path(binding), os.O_RDONLY | os.O_NOFOLLOW | os.O_NONBLOCK)
+ except OSError: raise Refused('private custody unavailable') from None
try:
- Journal._private(os.fstat(fd))
+ before=os.fstat(fd)
+ Journal._private(before)
with os.fdopen(fd, 'rb', closefd=False) as stream: raw = stream.read(65537)
+ after=os.fstat(fd)
+ identity=lambda value:(value.st_dev,value.st_ino,value.st_size,value.st_mtime_ns,value.st_ctime_ns,value.st_mode,value.st_nlink)
+ if identity(before)!=identity(after) or len(raw)!=before.st_size:
+ raise Refused('private custody changed during read')
finally: os.close(fd)
if len(raw) > 65536: raise Refused('oversize private custody record')
- try: value = json.loads(raw)
+ try: capsule = json.loads(raw,object_pairs_hook=_json_unique,parse_constant=_invalid_constant)
except Exception: raise Refused('invalid private custody record') from None
- if type(value) is not dict or value.get('binding') != asdict(binding):
- raise Refused('private custody binding mismatch')
+ if (type(capsule) is not dict or set(capsule)!={'payload','mac'} or type(capsule['mac']) is not str
+ or len(capsule['mac'])!=64 or any(x not in '0123456789abcdef' for x in capsule['mac'])):
+ raise Refused('invalid authenticated private capsule')
+ expected_mac=hmac.new(self._integrity_key,_encoded(capsule['payload']),hashlib.sha256).hexdigest()
+ if not hmac.compare_digest(capsule['mac'],expected_mac):
+ raise Refused('private capsule integrity mismatch')
+ value=capsule['payload']
expected={'binding','cleanup','port','policy_hcl','identity_accessor','task_accessor','phase','revision'}
- if (set(value)!=expected or type(value['port']) is not int or not 1<=value['port']<=65535
+ if type(value) is not dict or set(value)!=expected:
+ raise Refused('invalid private custody schema')
+ stored=value['binding']
+ if (type(stored) is not dict or set(stored)!=set(asdict(binding))
+ or any(type(v) is not str for v in stored.values())):
+ raise Refused('invalid private custody binding schema')
+ restored=Binding(**stored)
+ restored.validate()
+ if asdict(restored)!=asdict(binding): raise Refused('private custody binding mismatch')
+ if (type(value['port']) is not int or not 1<=value['port']<=65535
or type(value['policy_hcl']) is not str or not value['policy_hcl']
or type(value['identity_accessor']) is not str or not value['identity_accessor']
- or value['phase'] not in {'prepared','mint-intent','issued'}
+ or type(value['phase']) is not str or value['phase'] not in {'prepared','mint-intent','issued'}
or type(value['revision']) is not int or value['revision']!={'prepared':0,'mint-intent':1,'issued':2}[value['phase']]
or (value['phase']=='issued' and (type(value['task_accessor']) is not str or not value['task_accessor']))
or (value['phase']!='issued' and value['task_accessor'] is not None)
or type(value['cleanup']) is not list or len(value['cleanup'])!=3):
raise Refused('invalid private custody schema')
+ for item in value['cleanup']:
+ if type(item) is not dict or set(item)!={'kind','call_digest'}:
+ raise Refused('invalid private cleanup schema')
+ Cleanup(**item).validate()
+ expected_cleanup=[{'kind':item['kind'],'call_digest':item['call_digest']}
+ for item in cleanup_specs(binding.operation_id)]
+ if value['cleanup']!=expected_cleanup:
+ raise Refused('private cleanup differs from deterministic owner commitment')
return value
def _write(self, binding, value, initial=False):
- raw = _encoded(value)
+ capsule={'payload':value,'mac':hmac.new(self._integrity_key,_encoded(value),hashlib.sha256).hexdigest()}
+ raw = _encoded(capsule)
if len(raw) > 65536: raise Refused('oversize private custody record')
path = self._path(binding)
if initial:
@@ -159,6 +221,11 @@ class _Recorder:
if all(x['outcome'] == 'confirmed' for x in current):
raise Refused('cleanup already confirmed')
self.current = current
+ # An authenticated older prepared capsule must not erase later issuance
+ # uncertainty. Without an issued accessor, any native call or state at
+ # the grant boundary makes absence unprovable. Recovery is conservative.
+ self.no_mint_proved=(journal.observe(b)['state'] in {'reserved','consume-confirmed','authenticating'}
+ and journal.db.execute('SELECT COUNT(*) FROM calls WHERE operation=?',(b.operation_id,)).fetchone()[0]==0)
db = journal.db
try: db.execute('INSERT INTO recovery_attempts VALUES (?,?,?)', (permit.attempt_id,b.operation_id,json.dumps(current,sort_keys=True)))
except Exception: raise Refused('recovery admission already used') from None
@@ -233,7 +300,7 @@ def recover_fixture_cleanup(journal: Journal, permit: FixtureRecoveryPermit):
headers={'X-Vault-Token':permit.owner_token,'Content-Type':'application/json'})
response=conn.getresponse();raw=response.read(1048577)
if len(raw)>1048576: raise Refused('oversize recovery response')
- data=json.loads(raw) if raw else {}
+ data=json.loads(raw,object_pairs_hook=_json_unique,parse_constant=_invalid_constant) if raw else {}
if type(data) is not dict: raise Refused('invalid recovery response')
status=response.status
except Exception: raise Refused('recovery transport or response unconfirmed') from None
@@ -245,7 +312,7 @@ def recover_fixture_cleanup(journal: Journal, permit: FixtureRecoveryPermit):
if item['kind'] in {'task-token-revoke','identity-token-revoke'}:
accessor=value['task_accessor' if ordinal==0 else 'identity_accessor']
if accessor is None:
- confirmed=ordinal==0 and value['phase']=='prepared'
+ confirmed=ordinal==0 and value['phase']=='prepared' and record.no_mint_proved
else:
status,data=call('POST','auth/token/revoke-accessor',{'accessor':accessor})
# Revoke is idempotent for an already absent accessor;
@@ -256,11 +323,12 @@ def recover_fixture_cleanup(journal: Journal, permit: FixtureRecoveryPermit):
elif item['kind']=='grant-policy-delete':
path='sys/policies/acl/task-exec-'+permit.binding.operation_id
status,data=call('GET',path)
- if status==404:confirmed=True
- elif status==200 and data.get('data',{}).get('policy','').rstrip()==value['policy_hcl'].rstrip():
+ if _policy_absent(status,data):confirmed=True
+ elif (status==200 and not data.get('errors') and type(data.get('data')) is dict
+ and _same_policy(data['data'].get('policy'),value['policy_hcl'])):
status,data=call('DELETE',path)
if status in {200,204}:
- status,data=call('GET',path);confirmed=status==404
+ status,data=call('GET',path);confirmed=_policy_absent(status,data)
except Exception:
# No backend errors/response/handle hashes enter public evidence.
confirmed=False
diff --git a/tests/test_task_execution_recovery.py b/tests/test_task_execution_recovery.py
index e29d716..52b8b61 100644
--- a/tests/test_task_execution_recovery.py
+++ b/tests/test_task_execution_recovery.py
@@ -3,6 +3,7 @@ from dataclasses import replace
import hashlib
import json
import os
+import secrets
from pathlib import Path
import sys
import time
@@ -20,7 +21,7 @@ from task_execution_recovery import FixtureRecoveryCustody, recover_fixture_clea
def protected(owner,tmp_path):
native,journal,binding,call,compiler,descriptor=owner
directory=tmp_path/'separate-owner-custody';directory.mkdir(mode=0o700)
- custody=FixtureRecoveryCustody(directory)
+ custody=FixtureRecoveryCustody(directory,integrity_key=secrets.token_bytes(32))
native=NativeFixtureAdapter(compiler,native.plan,journal,binding,native.admission,recovery_custody=custody)
return native,journal,binding,call,custody
@@ -178,7 +179,7 @@ def test_private_custody_permissions_and_identity_binding_fail_closed(owner,tmp_
path=custody._path(binding);path.chmod(0o644)
with pytest.raises(Refused):custody.read(binding)
path.chmod(0o600)
- data=json.loads(path.read_bytes());data['binding']['operation_id']='other'
+ data=json.loads(path.read_bytes());data['payload']['binding']['operation_id']='other'
path.write_text(json.dumps(data))
with pytest.raises(Refused):custody.read(binding)
@@ -205,3 +206,115 @@ def test_lost_recovery_response_stays_unknown_until_new_independent_attempt(owne
# Reconciliation does not erase the historic uncertain invocation.
assert journal.observe(binding)['external_outcome_unknown']
assert journal.db.execute("SELECT COUNT(*) FROM recovery_calls WHERE outcome='pending'").fetchone()[0]==1
+
+
+@pytest.fixture
+def capsule(tmp_path):
+ from task_execution_journal import Binding,Cleanup
+ from task_execution_plan import cleanup_specs
+ binding=Binding('SYNTHETIC','sha256:'+'a'*64,'12345678-1234-1234-1234-123456789abc','b'*64)
+ directory=tmp_path/'private-capsule';directory.mkdir(mode=0o700)
+ custody=FixtureRecoveryCustody(directory,integrity_key=secrets.token_bytes(32))
+ cleanup=tuple(Cleanup(x['kind'],x['call_digest']) for x in cleanup_specs(binding.operation_id))
+ custody.prepare(binding,cleanup,12345,'path "fixture" {}\n','synthetic-private-accessor')
+ return custody,binding
+
+
+@pytest.mark.parametrize('mutate',[
+ lambda v:v.update(extra=True),lambda v:v.pop('phase'),lambda v:v.update(binding=True),
+ lambda v:v['binding'].update(approval_id=True),lambda v:v['binding'].update(extra='x'),
+ lambda v:v['binding'].update(request_digest='0'*64),lambda v:v.update(port=True),
+ lambda v:v.update(port=12345.0),lambda v:v.update(port=0),lambda v:v.update(phase=[]),
+ lambda v:v.update(phase='issued'),lambda v:v.update(revision=False),
+ lambda v:v.update(revision=0.0),lambda v:v.update(identity_accessor=False),
+ lambda v:v.update(task_accessor='unexpected-before-issue'),lambda v:v.update(policy_hcl=[]),
+ lambda v:v.update(cleanup={}),lambda v:v['cleanup'].pop(),
+ lambda v:v['cleanup'].__setitem__(0,False),lambda v:v['cleanup'][0].update(extra=True),
+ lambda v:v['cleanup'][0].update(kind=True),lambda v:v['cleanup'][0].update(call_digest=0),
+ lambda v:v['cleanup'][0].update(call_digest='f'*64),lambda v:v['cleanup'].reverse(),
+])
+def test_authenticated_but_damaged_capsule_schema_refuses(capsule,mutate):
+ custody,binding=capsule
+ value=custody.read(binding);mutate(value)
+ # Trusted test owner re-signs malformed payload to exercise schema defenses
+ # independently of MAC validation; untrusted corruption cannot do this.
+ custody._write(binding,value)
+ with pytest.raises(Refused):custody.read(binding)
+
+
+def test_capsule_corruption_wrong_key_duplicate_and_nonfinite_refuse(capsule):
+ custody,binding=capsule;path=custody._path(binding);original=path.read_bytes()
+ value=json.loads(original);value['payload']['identity_accessor']='different-synthetic-accessor'
+ path.write_text(json.dumps(value))
+ with pytest.raises(Refused):custody.read(binding)
+ path.write_bytes(original)
+ other=FixtureRecoveryCustody(custody.directory,integrity_key=b'X'*32)
+ with pytest.raises(Refused):other.read(binding)
+ for raw in [b'{',original[:-1]+b',"mac":"duplicate"}',original.replace(b'12345',b'NaN'),b'[]',b'{}',b'{"payload":{},"mac":"'+b'\xc3\xa9'*64+b'"}']:
+ path.write_bytes(raw)
+ with pytest.raises(Refused):custody.read(binding)
+ path.write_bytes(original)
+ assert custody.read(binding)['identity_accessor']=='synthetic-private-accessor'
+
+
+def test_exact_absence_and_single_terminal_lf_contract():
+ from task_execution_recovery import _policy_absent,_same_policy
+ assert _policy_absent(404,{'errors':[]})
+ for status,body in [(403,{'errors':[]}),(404,{}),(404,{'errors':['denied']}),
+ (404,{'errors':None}),(404,{'errors':[]} | {'data':{}}),
+ (404,{'errors':False}),(404,{'errors':0}),(404,{'errors':{}})]:
+ assert not _policy_absent(status,body)
+ assert _same_policy('fixture','fixture\n')
+ assert _same_policy('fixture\n','fixture\n')
+ for text in ['fixture \n','fixture\t','fixture\r\n','fixture\n\n',False,None]:
+ assert not _same_policy(text,'fixture\n')
+
+
+@pytest.mark.parametrize('after_delete',[False,True])
+def test_recovery_policy_absence_rejects_misleading_native_http_bodies(owner,tmp_path,monkeypatch,after_delete):
+ native,journal,binding,call,custody=protected(owner,tmp_path);crash(native,journal,binding)
+ import task_execution_recovery as module
+ malformed=[{}, {'errors':['permission denied']}, {'errors':None}, {'errors':False},
+ {'errors':0}, {'errors':[],'data':{}}, {'errors':['not really absent']},
+ b'{"errors":["permission denied"],"errors":[]}',b'{"errors":NaN}',b'not JSON']
+ path='sys/policies/acl/'+native.policy_name
+ for body in malformed:
+ sent=[];gets=[0]
+ class Response:
+ def __init__(self,status,data):self.status=status;self.data=data
+ def read(self,limit):return self.data if type(self.data) is bytes else json.dumps(self.data).encode()
+ class Connection:
+ def __init__(self,*args,**kwargs):pass
+ def request(self,method,url,**kwargs):sent.append((method,url))
+ def getresponse(self):
+ method,url=sent[-1]
+ if url=='/v1/'+path:
+ if method=='DELETE':return Response(204,{})
+ gets[0]+=1
+ if after_delete and gets[0]==1:
+ return Response(200,{'data':{'policy':native.hcl[:-1]}})
+ return Response(404,body)
+ return Response(403,{'errors':['synthetic refusal']})
+ def close(self):pass
+ with monkeypatch.context() as patch:
+ patch.setattr(module.http.client,'HTTPConnection',Connection)
+ result=recover_fixture_cleanup(journal,permit(native,custody,binding))
+ assert not result['confirmed'] and result['obligations'][2]['outcome']=='owed'
+ assert ('DELETE','/v1/'+path) in sent if after_delete else ('DELETE','/v1/'+path) not in sent
+ # The denied/malformed probes above emitted no real native calls.
+ assert call('GET',path)[0]==200
+ assert recover_fixture_cleanup(journal,permit(native,custody,binding))['confirmed']
+
+
+def test_authenticated_prepared_capsule_rollback_cannot_hide_later_mint(owner,tmp_path):
+ native,journal,binding,call,custody=protected(owner,tmp_path)
+ old=custody._path(binding).read_bytes()
+ crash(native,journal,binding)
+ current=custody.read(binding)
+ custody._path(binding).write_bytes(old)
+ # Still authentic under the owner key, but the durable journal proves the
+ # operation reached issuance. The stale prepared record cannot prove absence.
+ result=recover_fixture_cleanup(journal,permit(native,custody,binding))
+ assert not result['confirmed'] and result['obligations'][0]['outcome']=='owed'
+ assert call('POST','auth/token/lookup-accessor',{'accessor':current['task_accessor']})[0]==200
+ assert call('POST','auth/token/revoke-accessor',{'accessor':current['task_accessor']})[0] in {200,204}
diff --git a/workplans/RPF-WP-0035-credential-lane-implementation.md b/workplans/RPF-WP-0035-credential-lane-implementation.md
index 1b3a079..f58a6b9 100644
--- a/workplans/RPF-WP-0035-credential-lane-implementation.md
+++ b/workplans/RPF-WP-0035-credential-lane-implementation.md
@@ -1299,3 +1299,11 @@ without restoring business/consume admission. Separate durable attempt evidence
retains original unknown outcomes. Lost token-create response remains unresolved
when no accessor was committed. Source-only process-loss fixtures are not native
production custody or recovery-role admission; T12 remains progress.
+
+2026-10-16 T12 recovery R2 source refinement retains R1's overbroad 404 failure:
+policy cleanup now requires exact native empty-errors absence before/after delete;
+HCL uses only documented single optional terminal LF normalization. Private
+capsules are owner-key HMAC authenticated, strict nested schemas/ordered cleanup
+commitments reject damage, and journal history prevents stale prepared capsules
+from hiding issuance. Fixture process-loss source proof remains separate from
+unadmitted production custody, integrity-key storage and recovery-role scope.