861 lines
51 KiB
Diff
861 lines
51 KiB
Diff
diff --git a/docs/task-execution-native-fixture.md b/docs/task-execution-native-fixture.md
|
|
index 1dd6b91..f4ff070 100644
|
|
--- a/docs/task-execution-native-fixture.md
|
|
+++ b/docs/task-execution-native-fixture.md
|
|
@@ -121,3 +121,87 @@ retry cleanup. Native effects can survive process loss. Cleanup handles remain
|
|
memory-only: no independently authorized recovery mechanism is implemented here.
|
|
This closes disposable sequential apply acceptance only, not TSA03 or native
|
|
production recovery/admission.
|
|
+
|
|
+
|
|
+## Independent cleanup recovery source successor
|
|
+
|
|
+`task_execution_recovery.py` adds separately installed fixture-only owner custody
|
|
+and fresh recovery permits. Optional `recovery_custody` on NativeFixtureAdapter
|
|
+persists the original binding/cleanup commitments, exact operation policy, backend
|
|
+port and identity accessor before any grant effect; it commits mint intent before
|
|
+token creation and task accessor immediately after a valid issued response.
|
|
+Bearer credentials are never persisted. The custody directory is separate from
|
|
+the public journal, owner-only 0700/0600, bounded at 64KiB, no-follow, with atomic
|
|
+replacement and fsync. It is real process-independent persistence for synthetic
|
|
+fixtures. It does not isolate same-UID hostile execution, encrypt at rest, select
|
|
+production storage or establish a production secret policy. Those are explicit
|
|
+native custody/deployment gates, not supplied by Python types or file modes.
|
|
+
|
|
+The independent owner reads its private custody revision and installs a fresh
|
|
+bounded permit, current owner credential and retained live disposable Popen.
|
|
+Permit revocation is durably owner-recorded. The recovery runner has no mint,
|
|
+consume, policy-apply or business method. It dispatches only fixed exact-accessor
|
|
+revoke/readback and exact `task-exec-<operation>` policy read/delete/readback.
|
|
+A differently recreated policy is never deleted. Backend admission remains the
|
|
+same loopback-only owner-spawned dev process; no estate backend can be selected.
|
|
+Accessor-based recovery is an independently admitted alternate cleanup projection,
|
|
+not a reinterpretation of the original revoke-self call digest. Production needs
|
|
+exact owner-reviewed recovery scope; original business grant does not imply it.
|
|
+
|
|
+Fresh recovery holds the journal process lock but never sets business admission
|
|
+or consumed-admission fields. Separate recovery attempt/call/result tables retain
|
|
+original cleanup statuses before reconciliation, write native intent before calls,
|
|
+and preserve failed/pending outcomes. Proven cleanup alone updates the exact
|
|
+original cleanup obligation to confirmed. Business state and original pending
|
|
+call uncertainty remain unchanged. Reusing the original approval, continuing
|
|
+metadata calls and reusing a recovery permit refuse. Failure needs another
|
|
+independent owner permit; there is no automatic retry. Expired/revoked permits
|
|
+or changed private custody revision refuse before calls.
|
|
+
|
|
+A token-create response lost before accessor custody remains unresolved token
|
|
+cleanup debt: policy deletion/identity revocation can be independently confirmed,
|
|
+but cannot imply task-token revocation. Neither expiry guessed from wall clock
|
|
+nor policy absence fabricates token absence. A production successor must admit
|
|
+an independent exact ambiguous-issuance reconciliation mechanism or prove bounded
|
|
+expiry from owner backend evidence. No such new grant is implemented here.
|
|
+
|
|
+`test_task_execution_recovery.py` forks a real executor and terminates it without
|
|
+finally cleanup after issue, business effect/lost response and revoke/lost response.
|
|
+The independent parent, holding the backend and separately installed custody,
|
|
+recovers specific obligations and verifies actual token/policy absence. Additional
|
|
+cases cover lost-mint response, revoked credential, permit revocation/expiry,
|
|
+revision drift, namespace policy-name reuse and private-file/binding refusals.
|
|
+This is source fixture proof only. Native custody, isolation, exact recovery-role
|
|
+permissions, retention/encryption, clock admission, audit delivery and human scope
|
|
+approval remain missing. No generic broker, production reader, grant or CLI is added.
|
|
+
|
|
+
|
|
+### Recovery R2 exact absence and authenticated capsule refinement
|
|
+
|
|
+R1 overclaimed policy absence for arbitrary 404 bodies. R2 requires exactly
|
|
+`{"errors":[]}` for both initial GET and post-DELETE GET. Other 404 bodies,
|
|
+permission denials, extra data, malformed/duplicate-key/nonfinite JSON and failed
|
|
+readback cannot confirm policy cleanup. A successful DELETE status alone cannot
|
|
+confirm absence. Recovery native policy comparison admits precisely one optional
|
|
+terminal LF, matching the retained OpenBao HCL source/readback convention; spaces,
|
|
+tabs, CR, extra terminal LF and any other byte change are refused. This is a
|
|
+narrow declared normalization, not full whitespace or semantic equivalence.
|
|
+
|
|
+Private fixture capsules now carry HMAC-SHA256 authentication under an exactly
|
|
+32-byte integrity key supplied independently by the recovery owner. The key is
|
|
+never stored in the capsule or public journal, and is not a bearer credential.
|
|
+The fixture owner retains this key independently across executor death; a fresh
|
|
+owner custody object must supply the same key. Wrong-key or damaged accessor/HCL
|
|
+bytes refuse before admission. This is integrity protection, not encryption or
|
|
+same-UID isolation; production key custody, rotation/retention and recovery actor
|
|
+selection remain explicitly unadmitted.
|
|
+
|
|
+After authentication the reader enforces exact top-level and nested binding/
|
|
+cleanup key sets and types, validated Binding/Cleanup formats and the ordered
|
|
+source-derived cleanup_specs operation digests. Duplicate JSON keys, nonfinite
|
|
+numbers, malformed phase/revision combinations and type-coercible values refuse.
|
|
+The reader bounds bytes, rejects indirect/nonprivate files and detects stat/size
|
|
+changes during its single read. A valid older prepared capsule still cannot
|
|
+prove a task token absent when durable journal state/calls reached issuance:
|
|
+that remains unknown cleanup debt. Authentication alone is not anti-rollback.
|
|
+No business or consume admission is restored and no production scope is granted.
|
|
diff --git a/scripts/task_execution_journal.py b/scripts/task_execution_journal.py
|
|
index 8f1b9a2..1eaa583 100644
|
|
--- a/scripts/task_execution_journal.py
|
|
+++ b/scripts/task_execution_journal.py
|
|
@@ -104,6 +104,9 @@ class Journal:
|
|
self.db.execute("CREATE TABLE IF NOT EXISTS calls (operation TEXT NOT NULL, ordinal INTEGER NOT NULL, call_digest TEXT NOT NULL, outcome TEXT NOT NULL, evidence TEXT, PRIMARY KEY(operation,ordinal))")
|
|
self.db.execute("CREATE TABLE IF NOT EXISTS cleanup (operation TEXT NOT NULL, ordinal INTEGER NOT NULL, kind TEXT NOT NULL, call_digest TEXT NOT NULL, outcome TEXT NOT NULL, evidence TEXT, PRIMARY KEY(operation,ordinal), UNIQUE(operation,kind))")
|
|
self.db.execute("CREATE TABLE IF NOT EXISTS cleanup_calls (operation TEXT NOT NULL, obligation INTEGER NOT NULL, step INTEGER NOT NULL, call_digest TEXT NOT NULL, outcome TEXT NOT NULL, evidence TEXT, PRIMARY KEY(operation,obligation,step))")
|
|
+ self.db.execute("CREATE TABLE IF NOT EXISTS recovery_attempts (attempt TEXT PRIMARY KEY, operation TEXT NOT NULL, original TEXT NOT NULL)")
|
|
+ self.db.execute("CREATE TABLE IF NOT EXISTS recovery_calls (attempt TEXT NOT NULL, obligation INTEGER NOT NULL, step INTEGER NOT NULL, digest TEXT NOT NULL, outcome TEXT NOT NULL, evidence TEXT, PRIMARY KEY(attempt,obligation,step))")
|
|
+ self.db.execute("CREATE TABLE IF NOT EXISTS recovery_results (attempt TEXT NOT NULL, obligation INTEGER NOT NULL, confirmed INTEGER NOT NULL, evidence TEXT NOT NULL, PRIMARY KEY(attempt,obligation))")
|
|
self._held = False
|
|
self._admission = None
|
|
self._consumed_admission = None
|
|
@@ -228,6 +231,7 @@ class Journal:
|
|
pending = self.db.execute("SELECT COUNT(*) FROM calls WHERE operation=? AND outcome='pending'", (binding.operation_id,)).fetchone()[0]
|
|
pending += self.db.execute("SELECT COUNT(*) FROM cleanup WHERE operation=? AND outcome='pending'", (binding.operation_id,)).fetchone()[0]
|
|
pending += self.db.execute("SELECT COUNT(*) FROM cleanup_calls WHERE operation=? AND outcome='pending'", (binding.operation_id,)).fetchone()[0]
|
|
+ pending += self.db.execute("SELECT COUNT(*) FROM recovery_calls AS c JOIN recovery_attempts AS a ON c.attempt=a.attempt WHERE a.operation=? AND c.outcome='pending'", (binding.operation_id,)).fetchone()[0]
|
|
return {"state": row[3], "evidence_digest": row[4], "dispatch_allowed": False,
|
|
"external_outcome_unknown": bool(pending)}
|
|
|
|
diff --git a/scripts/task_execution_native_fixture.py b/scripts/task_execution_native_fixture.py
|
|
index caa7da1..31a822b 100644
|
|
--- a/scripts/task_execution_native_fixture.py
|
|
+++ b/scripts/task_execution_native_fixture.py
|
|
@@ -3,7 +3,8 @@
|
|
This adapter deliberately admits only a disposable loopback fixture. It supplies
|
|
real native primitive coverage while production grant schema, identity/consume and
|
|
origin/source admission are unfinished. It cannot open an estate backend. Owner
|
|
-credentials/handles stay in memory; no subprocess, redirects or ambient proxy.
|
|
+bearer credentials stay in memory; optional owner-private fixture custody persists
|
|
+accessors for independently admitted cleanup recovery; no redirects or ambient proxy.
|
|
Python privacy is not credential isolation; production needs the owner runtime.
|
|
"""
|
|
from __future__ import annotations
|
|
@@ -41,7 +42,7 @@ class FixtureAdmission:
|
|
|
|
class NativeFixtureAdapter:
|
|
def __init__(self, compiler: Compiler, plan: Plan, journal: Journal,
|
|
- binding: Binding, admission: FixtureAdmission):
|
|
+ binding: Binding, admission: FixtureAdmission, *, recovery_custody=None):
|
|
if type(admission) is not FixtureAdmission or type(admission.descriptor) is not bytes:
|
|
raise Refused('independent fixture admission required')
|
|
self.compiler,self.plan,self.journal,self.binding=compiler,plan,journal,binding
|
|
@@ -100,6 +101,12 @@ class NativeFixtureAdapter:
|
|
if self.meta['cleanup'] != cleanup_specs(binding.operation_id): raise Refused('cleanup plan mismatch')
|
|
self.task=None; self._mint_attempted=False; self._policy_attempted=False; self._ordinal=0; self._next_metadata=0
|
|
self._cleanup_steps={item.kind:0 for item in self.cleanup}
|
|
+ self.recovery_custody=recovery_custody
|
|
+ if recovery_custody is not None:
|
|
+ from task_execution_recovery import FixtureRecoveryCustody
|
|
+ if type(recovery_custody) is not FixtureRecoveryCustody or journal.observe(binding) is not None:
|
|
+ raise Refused('fresh independent fixture recovery custody required')
|
|
+ recovery_custody.prepare(binding,self.cleanup,self.port,self.hcl,admission.identity_session.accessor)
|
|
|
|
def _live(self):
|
|
if not self.journal._held or self.journal._admission!=self.binding or self.journal._consumed_admission!=self.binding:
|
|
@@ -177,6 +184,8 @@ class NativeFixtureAdapter:
|
|
'type':'service','display_name':'task-exec-'+self.binding.operation_id,'no_parent':False}
|
|
if body!=self.meta['execution_grant']['openbao']['token_create_body']:
|
|
raise Refused('native token body differs from bound owner projection')
|
|
+ if self.recovery_custody is not None:
|
|
+ self.recovery_custody.checkpoint(self.binding,'mint-intent')
|
|
self._mint_attempted=True
|
|
status,data=self._business('POST','auth/token/create',root,encoded(body));self._ok(status,data,{200})
|
|
auth=data.get('auth',{})
|
|
@@ -185,6 +194,8 @@ class NativeFixtureAdapter:
|
|
# Retain handles before validation so finally cleanup can revoke even a
|
|
# malformed overbroad issue response. Never return token-bearing metadata.
|
|
self.task=PrivateSession(auth['client_token'],auth['accessor'])
|
|
+ if self.recovery_custody is not None:
|
|
+ self.recovery_custody.checkpoint(self.binding,'issued',self.task.accessor)
|
|
status,data=self._business('POST','auth/token/lookup',root,encoded({'token':self.task.token}));self._ok(status,data,{200})
|
|
info=data.get('data',{})
|
|
if (info.get('policies')!=[self.policy_name] or info.get('identity_policies',[]) or info.get('renewable') is not False
|
|
diff --git a/scripts/task_execution_recovery.py b/scripts/task_execution_recovery.py
|
|
new file mode 100644
|
|
index 0000000..ab8b8cb
|
|
--- /dev/null
|
|
+++ b/scripts/task_execution_recovery.py
|
|
@@ -0,0 +1,336 @@
|
|
+"""Owner-only cleanup recovery for the disposable native fixture.
|
|
+
|
|
+Private, separately installed custody persists accessors, never bearer tokens.
|
|
+This is a fixture implementation, not production custody/isolation admission.
|
|
+The recovery owner supplies a fresh capability and credential independently of
|
|
+public request/journal data. No business adapter or resumed admission is built.
|
|
+"""
|
|
+from __future__ import annotations
|
|
+from dataclasses import asdict, dataclass, field
|
|
+import hashlib
|
|
+import hmac
|
|
+import http.client
|
|
+import json
|
|
+import os
|
|
+from pathlib import Path
|
|
+import secrets
|
|
+import subprocess
|
|
+import time
|
|
+import uuid
|
|
+
|
|
+from task_execution_journal import Binding, Cleanup, Journal, Refused
|
|
+from task_execution_plan import cleanup_specs
|
|
+
|
|
+
|
|
+def _encoded(value):
|
|
+ return json.dumps(value, sort_keys=True, separators=(',', ':'), allow_nan=False).encode()
|
|
+
|
|
+
|
|
+def _digest(value):
|
|
+ return hashlib.sha256(_encoded(value)).hexdigest()
|
|
+
|
|
+
|
|
+def _json_unique(pairs):
|
|
+ result={}
|
|
+ for key,value in pairs:
|
|
+ if key in result: raise Refused('duplicate private custody field')
|
|
+ result[key]=value
|
|
+ return result
|
|
+
|
|
+
|
|
+def _invalid_constant(value):
|
|
+ raise Refused('nonfinite private custody value')
|
|
+
|
|
+
|
|
+def _policy_absent(status, response):
|
|
+ # Exactly the native OpenBao empty-errors 404. Denial/malformed/error-shaped
|
|
+ # bodies never prove absence, including after a successful delete response.
|
|
+ return (status==404 and type(response) is dict and set(response)=={'errors'}
|
|
+ and type(response['errors']) is list and not response['errors'])
|
|
+
|
|
+
|
|
+def _same_policy(actual, expected):
|
|
+ # OpenBao may omit one terminal LF from HCL source. Only that equivalence
|
|
+ # is admitted: spaces, tabs, CR and additional blank lines remain meaningful.
|
|
+ if type(actual) is not str or type(expected) is not str: return False
|
|
+ strip_one=lambda value:value[:-1] if value.endswith('\n') else value
|
|
+ return strip_one(actual)==strip_one(expected)
|
|
+
|
|
+
|
|
+class FixtureRecoveryCustody:
|
|
+ """Independent owner-private persistent fixture custody; no public API.
|
|
+
|
|
+ Mode/ownership and no-follow checks protect against accidental exposure.
|
|
+ Same-UID hostile code is NOT isolated. Production must supply independently
|
|
+ enforced owner storage/executor authority; this implementation refuses estate
|
|
+ backends and is never evidence of a production grant.
|
|
+ """
|
|
+ def __init__(self, directory: Path, *, integrity_key: bytes):
|
|
+ if type(integrity_key) is not bytes or len(integrity_key)!=32:
|
|
+ raise Refused('independent owner capsule integrity key required')
|
|
+ self._integrity_key=integrity_key
|
|
+ self.directory = directory
|
|
+ if directory.is_symlink() or directory.absolute() != directory.resolve():
|
|
+ raise Refused('indirect recovery custody directory')
|
|
+ Journal._private(directory.stat(), directory=True)
|
|
+
|
|
+ def _path(self, binding):
|
|
+ if type(binding) is not Binding: raise Refused('exact recovery binding required')
|
|
+ binding.validate()
|
|
+ return self.directory / (binding.operation_id + '.private.json')
|
|
+
|
|
+ def _sync(self):
|
|
+ fd = os.open(self.directory, os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW)
|
|
+ try: os.fsync(fd)
|
|
+ finally: os.close(fd)
|
|
+
|
|
+ def read(self, binding):
|
|
+ try: fd = os.open(self._path(binding), os.O_RDONLY | os.O_NOFOLLOW | os.O_NONBLOCK)
|
|
+ except OSError: raise Refused('private custody unavailable') from None
|
|
+ try:
|
|
+ before=os.fstat(fd)
|
|
+ Journal._private(before)
|
|
+ with os.fdopen(fd, 'rb', closefd=False) as stream: raw = stream.read(65537)
|
|
+ after=os.fstat(fd)
|
|
+ identity=lambda value:(value.st_dev,value.st_ino,value.st_size,value.st_mtime_ns,value.st_ctime_ns,value.st_mode,value.st_nlink)
|
|
+ if identity(before)!=identity(after) or len(raw)!=before.st_size:
|
|
+ raise Refused('private custody changed during read')
|
|
+ finally: os.close(fd)
|
|
+ if len(raw) > 65536: raise Refused('oversize private custody record')
|
|
+ try: capsule = json.loads(raw,object_pairs_hook=_json_unique,parse_constant=_invalid_constant)
|
|
+ except Exception: raise Refused('invalid private custody record') from None
|
|
+ if (type(capsule) is not dict or set(capsule)!={'payload','mac'} or type(capsule['mac']) is not str
|
|
+ or len(capsule['mac'])!=64 or any(x not in '0123456789abcdef' for x in capsule['mac'])):
|
|
+ raise Refused('invalid authenticated private capsule')
|
|
+ expected_mac=hmac.new(self._integrity_key,_encoded(capsule['payload']),hashlib.sha256).hexdigest()
|
|
+ if not hmac.compare_digest(capsule['mac'],expected_mac):
|
|
+ raise Refused('private capsule integrity mismatch')
|
|
+ value=capsule['payload']
|
|
+ expected={'binding','cleanup','port','policy_hcl','identity_accessor','task_accessor','phase','revision'}
|
|
+ if type(value) is not dict or set(value)!=expected:
|
|
+ raise Refused('invalid private custody schema')
|
|
+ stored=value['binding']
|
|
+ if (type(stored) is not dict or set(stored)!=set(asdict(binding))
|
|
+ or any(type(v) is not str for v in stored.values())):
|
|
+ raise Refused('invalid private custody binding schema')
|
|
+ restored=Binding(**stored)
|
|
+ restored.validate()
|
|
+ if asdict(restored)!=asdict(binding): raise Refused('private custody binding mismatch')
|
|
+ if (type(value['port']) is not int or not 1<=value['port']<=65535
|
|
+ or type(value['policy_hcl']) is not str or not value['policy_hcl']
|
|
+ or type(value['identity_accessor']) is not str or not value['identity_accessor']
|
|
+ or type(value['phase']) is not str or value['phase'] not in {'prepared','mint-intent','issued'}
|
|
+ or type(value['revision']) is not int or value['revision']!={'prepared':0,'mint-intent':1,'issued':2}[value['phase']]
|
|
+ or (value['phase']=='issued' and (type(value['task_accessor']) is not str or not value['task_accessor']))
|
|
+ or (value['phase']!='issued' and value['task_accessor'] is not None)
|
|
+ or type(value['cleanup']) is not list or len(value['cleanup'])!=3):
|
|
+ raise Refused('invalid private custody schema')
|
|
+ for item in value['cleanup']:
|
|
+ if type(item) is not dict or set(item)!={'kind','call_digest'}:
|
|
+ raise Refused('invalid private cleanup schema')
|
|
+ Cleanup(**item).validate()
|
|
+ expected_cleanup=[{'kind':item['kind'],'call_digest':item['call_digest']}
|
|
+ for item in cleanup_specs(binding.operation_id)]
|
|
+ if value['cleanup']!=expected_cleanup:
|
|
+ raise Refused('private cleanup differs from deterministic owner commitment')
|
|
+ return value
|
|
+
|
|
+ def _write(self, binding, value, initial=False):
|
|
+ capsule={'payload':value,'mac':hmac.new(self._integrity_key,_encoded(value),hashlib.sha256).hexdigest()}
|
|
+ raw = _encoded(capsule)
|
|
+ if len(raw) > 65536: raise Refused('oversize private custody record')
|
|
+ path = self._path(binding)
|
|
+ if initial:
|
|
+ fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW, 0o600)
|
|
+ else:
|
|
+ # Existing owner-private record checked before replacement. Unique
|
|
+ # temp + fsync + replace + directory fsync never truncates old state.
|
|
+ self.read(binding)
|
|
+ tmp = self.directory / ('.pending-' + secrets.token_hex(16))
|
|
+ fd = os.open(tmp, os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW, 0o600)
|
|
+ try:
|
|
+ with os.fdopen(fd, 'wb', closefd=False) as stream:
|
|
+ stream.write(raw); stream.flush(); os.fsync(fd)
|
|
+ finally: os.close(fd)
|
|
+ if not initial: os.replace(tmp, path)
|
|
+ self._sync()
|
|
+
|
|
+ def prepare(self, binding, cleanup, port, policy_hcl, identity_accessor):
|
|
+ value = {'binding': asdict(binding), 'cleanup': [asdict(x) for x in cleanup],
|
|
+ 'port': port, 'policy_hcl': policy_hcl,
|
|
+ 'identity_accessor': identity_accessor, 'task_accessor': None,
|
|
+ 'phase': 'prepared', 'revision': 0}
|
|
+ self._write(binding, value, initial=True)
|
|
+
|
|
+ def checkpoint(self, binding, phase, task_accessor=None):
|
|
+ value = self.read(binding)
|
|
+ if (value['phase'], phase) not in {('prepared', 'mint-intent'), ('mint-intent', 'issued')}:
|
|
+ raise Refused('invalid protected recovery checkpoint')
|
|
+ if phase == 'issued' and (type(task_accessor) is not str or not task_accessor):
|
|
+ raise Refused('protected issued accessor required')
|
|
+ value.update(phase=phase, task_accessor=task_accessor, revision=value['revision'] + 1)
|
|
+ self._write(binding, value)
|
|
+
|
|
+ def revoke(self, permit):
|
|
+ if type(permit) is not FixtureRecoveryPermit or permit.custody is not self:
|
|
+ raise Refused('exact owner recovery permit required')
|
|
+ uuid.UUID(permit.attempt_id)
|
|
+ path=self.directory / (permit.attempt_id+'.revoked')
|
|
+ fd=os.open(path,os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW,0o600)
|
|
+ try:os.fsync(fd)
|
|
+ finally:os.close(fd)
|
|
+ self._sync()
|
|
+
|
|
+ def admit(self, binding, *, owner_token, server_process, valid_seconds=30):
|
|
+ """Trusted recovery-owner action, not deserialization of caller flags.
|
|
+
|
|
+ Fresh admission seals the current private custody revision. The owner
|
|
+ supplies its credential and retained backend process separately. Neither
|
|
+ token nor private capsule digest is emitted to the public journal.
|
|
+ """
|
|
+ value = self.read(binding)
|
|
+ if type(valid_seconds) is not int or not 1 <= valid_seconds <= 60:
|
|
+ raise Refused('bounded recovery admission required')
|
|
+ if type(owner_token) is not str or not owner_token:
|
|
+ raise Refused('independent recovery owner credential required')
|
|
+ return FixtureRecoveryPermit(str(uuid.uuid4()), binding, self, _digest(value),
|
|
+ owner_token, server_process, time.monotonic()+valid_seconds)
|
|
+
|
|
+
|
|
+@dataclass(frozen=True, repr=False)
|
|
+class FixtureRecoveryPermit:
|
|
+ attempt_id: str
|
|
+ binding: Binding
|
|
+ custody: FixtureRecoveryCustody
|
|
+ private_revision_digest: str = field(repr=False)
|
|
+ owner_token: str = field(repr=False)
|
|
+ server_process: subprocess.Popen = field(repr=False)
|
|
+ deadline: float
|
|
+
|
|
+
|
|
+class _Recorder:
|
|
+ def __init__(self, journal, permit, value):
|
|
+ self.journal, self.permit = journal, permit
|
|
+ b = permit.binding
|
|
+ if journal._admission is not None or journal._consumed_admission is not None:
|
|
+ raise Refused('recovery cannot share a business admission')
|
|
+ if journal.observe(b) is None: raise Refused('original operation absent')
|
|
+ current = journal.cleanup_status(b)['obligations']
|
|
+ if [(x['kind'], x['call_digest']) for x in current] != [(x['kind'], x['call_digest']) for x in value['cleanup']]:
|
|
+ raise Refused('original cleanup commitment mismatch')
|
|
+ if all(x['outcome'] == 'confirmed' for x in current):
|
|
+ raise Refused('cleanup already confirmed')
|
|
+ self.current = current
|
|
+ # An authenticated older prepared capsule must not erase later issuance
|
|
+ # uncertainty. Without an issued accessor, any native call or state at
|
|
+ # the grant boundary makes absence unprovable. Recovery is conservative.
|
|
+ self.no_mint_proved=(journal.observe(b)['state'] in {'reserved','consume-confirmed','authenticating'}
|
|
+ and journal.db.execute('SELECT COUNT(*) FROM calls WHERE operation=?',(b.operation_id,)).fetchone()[0]==0)
|
|
+ db = journal.db
|
|
+ try: db.execute('INSERT INTO recovery_attempts VALUES (?,?,?)', (permit.attempt_id,b.operation_id,json.dumps(current,sort_keys=True)))
|
|
+ except Exception: raise Refused('recovery admission already used') from None
|
|
+ journal._sync_directory()
|
|
+
|
|
+ def before(self, ordinal, step, method, path):
|
|
+ digest = _digest({'kind': self.current[ordinal]['kind'], 'commitment':self.current[ordinal]['call_digest'], 'method':method,'path':path,'step':step})
|
|
+ self.journal.db.execute("INSERT INTO recovery_calls VALUES (?,?,?,?,'pending',NULL)", (self.permit.attempt_id,ordinal,step,digest))
|
|
+ self.journal._sync_directory()
|
|
+ return digest
|
|
+
|
|
+ def after(self, ordinal, step, status):
|
|
+ self.journal.db.execute("UPDATE recovery_calls SET outcome='complete', evidence=? WHERE attempt=? AND obligation=? AND step=? AND outcome='pending'", (_digest({'http_status':status}),self.permit.attempt_id,ordinal,step))
|
|
+ self.journal._sync_directory()
|
|
+
|
|
+ def finish(self, ordinal, confirmed):
|
|
+ evidence = _digest({'attempt':self.permit.attempt_id,'obligation':ordinal,'confirmed':confirmed})
|
|
+ db = self.journal.db
|
|
+ try:
|
|
+ db.execute('BEGIN IMMEDIATE')
|
|
+ pending = db.execute("SELECT COUNT(*) FROM recovery_calls WHERE attempt=? AND obligation=? AND outcome='pending'",(self.permit.attempt_id,ordinal)).fetchone()[0]
|
|
+ if confirmed and pending: raise Refused('pending recovery cannot confirm')
|
|
+ db.execute('INSERT INTO recovery_results VALUES (?,?,?,?)',(self.permit.attempt_id,ordinal,int(confirmed),evidence))
|
|
+ if confirmed:
|
|
+ item=self.current[ordinal]
|
|
+ db.execute("UPDATE cleanup SET outcome='confirmed', evidence=? WHERE operation=? AND ordinal=? AND kind=? AND call_digest=?",(evidence,self.permit.binding.operation_id,ordinal,item['kind'],item['call_digest']))
|
|
+ db.execute('COMMIT');self.journal._sync_directory()
|
|
+ except BaseException:
|
|
+ if db.in_transaction: db.execute('ROLLBACK')
|
|
+ raise
|
|
+
|
|
+
|
|
+def recover_fixture_cleanup(journal: Journal, permit: FixtureRecoveryPermit):
|
|
+ """Fresh independent owner admission: fixed cleanup only, no business retry.
|
|
+
|
|
+ Each attempt preserves original pending/unconfirmed rows in recovery_attempts
|
|
+ and native pending call rows unchanged. A fresh owner permit is required for
|
|
+ another attempt after failure; never autonomously retry uncertain mutations.
|
|
+ """
|
|
+ if type(permit) is not FixtureRecoveryPermit or type(permit.custody) is not FixtureRecoveryCustody:
|
|
+ raise Refused('independent typed recovery admission required')
|
|
+ if str(uuid.UUID(permit.attempt_id)) != permit.attempt_id:
|
|
+ raise Refused('invalid recovery attempt identity')
|
|
+ value = permit.custody.read(permit.binding)
|
|
+ if _digest(value) != permit.private_revision_digest:
|
|
+ raise Refused('private custody revision changed')
|
|
+ proc=permit.server_process
|
|
+ if (type(proc) is not subprocess.Popen or proc.poll() is not None or type(proc.args) is not list
|
|
+ or '-dev' not in proc.args or '-dev-no-store-token' not in proc.args
|
|
+ or '-dev-listen-address=127.0.0.1:'+str(value['port']) not in proc.args):
|
|
+ raise Refused('same live disposable backend required')
|
|
+ def fresh():
|
|
+ if (permit.custody.directory / (permit.attempt_id+'.revoked')).exists():
|
|
+ raise Refused('recovery permit revoked')
|
|
+ if time.monotonic() >= permit.deadline: raise Refused('recovery admission expired')
|
|
+ if proc.poll() is not None: raise Refused('disposable backend lost')
|
|
+ if _digest(permit.custody.read(permit.binding)) != permit.private_revision_digest:
|
|
+ raise Refused('private custody changed during recovery')
|
|
+ fresh()
|
|
+ with journal.locked():
|
|
+ record = _Recorder(journal,permit,value)
|
|
+ for ordinal,item in enumerate(record.current):
|
|
+ if item['outcome']=='confirmed': continue
|
|
+ step=0
|
|
+ def call(method,path,body=None):
|
|
+ nonlocal step
|
|
+ fresh(); index=step; step+=1
|
|
+ record.before(ordinal,index,method,path)
|
|
+ conn=http.client.HTTPConnection('127.0.0.1',value['port'],timeout=2)
|
|
+ try:
|
|
+ conn.request(method,'/v1/'+path,body=_encoded(body) if body is not None else None,
|
|
+ headers={'X-Vault-Token':permit.owner_token,'Content-Type':'application/json'})
|
|
+ response=conn.getresponse();raw=response.read(1048577)
|
|
+ if len(raw)>1048576: raise Refused('oversize recovery response')
|
|
+ data=json.loads(raw,object_pairs_hook=_json_unique,parse_constant=_invalid_constant) if raw else {}
|
|
+ if type(data) is not dict: raise Refused('invalid recovery response')
|
|
+ status=response.status
|
|
+ except Exception: raise Refused('recovery transport or response unconfirmed') from None
|
|
+ finally:conn.close()
|
|
+ record.after(ordinal,index,status)
|
|
+ return status,data
|
|
+ confirmed=False
|
|
+ try:
|
|
+ if item['kind'] in {'task-token-revoke','identity-token-revoke'}:
|
|
+ accessor=value['task_accessor' if ordinal==0 else 'identity_accessor']
|
|
+ if accessor is None:
|
|
+ confirmed=ordinal==0 and value['phase']=='prepared' and record.no_mint_proved
|
|
+ else:
|
|
+ status,data=call('POST','auth/token/revoke-accessor',{'accessor':accessor})
|
|
+ # Revoke is idempotent for an already absent accessor;
|
|
+ # actual absence must still be independently read back.
|
|
+ if status in {200,204,400}:
|
|
+ status,data=call('POST','auth/token/lookup-accessor',{'accessor':accessor})
|
|
+ confirmed=status==400 and data.get('errors') in (['invalid accessor'],['1 error occurred:\n\t* invalid accessor\n\n'])
|
|
+ elif item['kind']=='grant-policy-delete':
|
|
+ path='sys/policies/acl/task-exec-'+permit.binding.operation_id
|
|
+ status,data=call('GET',path)
|
|
+ if _policy_absent(status,data):confirmed=True
|
|
+ elif (status==200 and not data.get('errors') and type(data.get('data')) is dict
|
|
+ and _same_policy(data['data'].get('policy'),value['policy_hcl'])):
|
|
+ status,data=call('DELETE',path)
|
|
+ if status in {200,204}:
|
|
+ status,data=call('GET',path);confirmed=_policy_absent(status,data)
|
|
+ except Exception:
|
|
+ # No backend errors/response/handle hashes enter public evidence.
|
|
+ confirmed=False
|
|
+ record.finish(ordinal,confirmed)
|
|
+ return journal.cleanup_status(permit.binding)
|
|
diff --git a/tests/test_task_execution_recovery.py b/tests/test_task_execution_recovery.py
|
|
new file mode 100644
|
|
index 0000000..52b8b61
|
|
--- /dev/null
|
|
+++ b/tests/test_task_execution_recovery.py
|
|
@@ -0,0 +1,320 @@
|
|
+"""Actual executor process loss against independent disposable OpenBao."""
|
|
+from dataclasses import replace
|
|
+import hashlib
|
|
+import json
|
|
+import os
|
|
+import secrets
|
|
+from pathlib import Path
|
|
+import sys
|
|
+import time
|
|
+
|
|
+import pytest
|
|
+
|
|
+sys.path.insert(0,str(Path(__file__).resolve().parents[1]/'scripts'))
|
|
+from test_task_execution_plan import compiler
|
|
+from test_task_execution_native_fixture import bao_dev, owner, admit
|
|
+from task_execution_journal import Journal, Refused
|
|
+from task_execution_native_fixture import NativeFixtureAdapter
|
|
+from task_execution_recovery import FixtureRecoveryCustody, recover_fixture_cleanup
|
|
+
|
|
+
|
|
+def protected(owner,tmp_path):
|
|
+ native,journal,binding,call,compiler,descriptor=owner
|
|
+ directory=tmp_path/'separate-owner-custody';directory.mkdir(mode=0o700)
|
|
+ custody=FixtureRecoveryCustody(directory,integrity_key=secrets.token_bytes(32))
|
|
+ native=NativeFixtureAdapter(compiler,native.plan,journal,binding,native.admission,recovery_custody=custody)
|
|
+ return native,journal,binding,call,custody
|
|
+
|
|
+
|
|
+def crash(native,journal,binding,stage='issued'):
|
|
+ pid=os.fork()
|
|
+ if pid==0:
|
|
+ try:
|
|
+ # The backend belongs to the independent parent, so this child cannot
|
|
+ # waitpid it. Parent retains the actual Popen and checks it on recovery.
|
|
+ native.admission.server_process.poll=lambda:None
|
|
+ native.journal=Journal(journal.directory)
|
|
+ with native.journal.locked():
|
|
+ admit(native,native.journal,binding)
|
|
+ if stage=='mint-response-lost':
|
|
+ original=native._http
|
|
+ def lost(method,path,token,body=b''):
|
|
+ result=original(method,path,token,body)
|
|
+ if path=='auth/token/create':os._exit(73)
|
|
+ return result
|
|
+ native._http=lost
|
|
+ native.materialize_grant()
|
|
+ native.journal.transition(binding,'grant-issued','applying')
|
|
+ if stage=='business-response-lost':
|
|
+ native.metadata_call(0)
|
|
+ original=native._http
|
|
+ def lost(method,path,token,body=b''):
|
|
+ result=original(method,path,token,body)
|
|
+ if method=='PUT':os._exit(73)
|
|
+ return result
|
|
+ native._http=lost
|
|
+ native.metadata_call(1)
|
|
+ if stage=='cleanup-response-lost':
|
|
+ from task_execution_cleanup import run_cleanup
|
|
+ original=native._http
|
|
+ def lost(method,path,token,body=b''):
|
|
+ result=original(method,path,token,body)
|
|
+ if path=='auth/token/revoke-self':os._exit(73)
|
|
+ return result
|
|
+ native._http=lost
|
|
+ run_cleanup(native.journal,binding,native)
|
|
+ os._exit(73)
|
|
+ except BaseException:os._exit(74)
|
|
+ _,status=os.waitpid(pid,0)
|
|
+ assert os.waitstatus_to_exitcode(status)==73
|
|
+ # Original admission never appears in the independent parent or reopened DB.
|
|
+ assert native.task is None
|
|
+ assert journal._admission is None and journal._consumed_admission is None
|
|
+
|
|
+
|
|
+def permit(native,custody,binding,**kwargs):
|
|
+ return custody.admit(binding,owner_token=kwargs.pop('owner_token',native.admission.owner_token),
|
|
+ server_process=native.admission.server_process,**kwargs)
|
|
+
|
|
+
|
|
+@pytest.mark.parametrize('stage',['issued','business-response-lost','cleanup-response-lost'])
|
|
+def test_process_loss_then_independent_cleanup_never_resumes_business(owner,tmp_path,stage,monkeypatch,request):
|
|
+ native,journal,binding,call,custody=protected(owner,tmp_path)
|
|
+ crash(native,journal,binding,stage)
|
|
+ journal=Journal(journal.directory)
|
|
+ request.addfinalizer(journal.close)
|
|
+ native.journal=journal
|
|
+ private=custody.read(binding)
|
|
+ assert private['phase']=='issued' and private['task_accessor']
|
|
+ target=native.compiler.before_call(native.plan,1).path
|
|
+ before=call('GET',target)
|
|
+ with journal.locked():
|
|
+ with pytest.raises(Refused):journal.reserve(binding,native.cleanup)
|
|
+ with pytest.raises(Refused):native.materialize_grant()
|
|
+ with pytest.raises(Refused):native.metadata_call(0)
|
|
+ with pytest.raises(Refused):native.revoke_task_token(native.cleanup[0].call_digest)
|
|
+ admitted=permit(native,custody,binding)
|
|
+ import task_execution_recovery as recovery_module
|
|
+ sent=[]
|
|
+ original_request=recovery_module.http.client.HTTPConnection.request
|
|
+ def tracked(connection,method,url,*args,**kwargs):
|
|
+ sent.append((method,url)) # metadata only; never capture body/headers
|
|
+ return original_request(connection,method,url,*args,**kwargs)
|
|
+ with monkeypatch.context() as patch:
|
|
+ patch.setattr(recovery_module.http.client.HTTPConnection,'request',tracked)
|
|
+ result=recover_fixture_cleanup(journal,admitted)
|
|
+ assert all((method,path) in {
|
|
+ ('POST','/v1/auth/token/revoke-accessor'),('POST','/v1/auth/token/lookup-accessor'),
|
|
+ ('GET','/v1/sys/policies/acl/'+native.policy_name),
|
|
+ ('DELETE','/v1/sys/policies/acl/'+native.policy_name)} for method,path in sent)
|
|
+ assert result['confirmed']
|
|
+ for accessor in [private['task_accessor'],private['identity_accessor']]:
|
|
+ assert call('POST','auth/token/lookup-accessor',{'accessor':accessor})[0]==400
|
|
+ assert call('GET','sys/policies/acl/'+native.policy_name)[0]==404
|
|
+ after=call('GET',target)
|
|
+ assert after[0]==before[0] and after[1].get('data')==before[1].get('data') # request IDs vary
|
|
+ assert journal.observe(binding)['dispatch_allowed'] is False
|
|
+ if stage!='issued':assert journal.observe(binding)['external_outcome_unknown']
|
|
+ if stage=='cleanup-response-lost':
|
|
+ original=json.loads(journal.db.execute('SELECT original FROM recovery_attempts').fetchone()[0])
|
|
+ assert original[0]['outcome']=='pending'
|
|
+ assert journal.db.execute("SELECT COUNT(*) FROM cleanup_calls WHERE outcome='pending'").fetchone()[0]==1
|
|
+ with pytest.raises(Refused):recover_fixture_cleanup(journal,admitted)
|
|
+ public=(journal.directory/'journal.sqlite').read_bytes()
|
|
+ for value in [private['task_accessor'],private['identity_accessor'],native.admission.owner_token]:
|
|
+ assert value.encode() not in public
|
|
+ assert 'client_token' not in json.dumps(private)
|
|
+ assert 'token' not in repr(admitted) and private['task_accessor'] not in repr(admitted)
|
|
+
|
|
+
|
|
+def test_process_loss_before_handle_commit_preserves_unknown_mint_debt(owner,tmp_path):
|
|
+ native,journal,binding,call,custody=protected(owner,tmp_path)
|
|
+ crash(native,journal,binding,'mint-response-lost')
|
|
+ assert custody.read(binding)['phase']=='mint-intent'
|
|
+ result=recover_fixture_cleanup(journal,permit(native,custody,binding))
|
|
+ assert not result['confirmed']
|
|
+ assert [x['outcome'] for x in result['obligations']]==['owed','confirmed','confirmed']
|
|
+ assert journal.observe(binding)['external_outcome_unknown']
|
|
+ assert journal.db.execute('SELECT confirmed FROM recovery_results ORDER BY obligation').fetchall()==[(0,),(1,),(1,)]
|
|
+
|
|
+
|
|
+def test_revoked_expired_wrong_revision_and_public_flag_refuse_before_calls(owner,tmp_path):
|
|
+ native,journal,binding,call,custody=protected(owner,tmp_path);crash(native,journal,binding)
|
|
+ p=permit(native,custody,binding)
|
|
+ custody.revoke(p)
|
|
+ for bad in [p,replace(permit(native,custody,binding),deadline=time.monotonic()-1),
|
|
+ replace(permit(native,custody,binding),private_revision_digest='0'*64),{'verified':True}]:
|
|
+ with pytest.raises(Refused):recover_fixture_cleanup(journal,bad)
|
|
+ assert call('GET','sys/policies/acl/'+native.policy_name)[0]==200
|
|
+ assert journal.cleanup_status(binding)['obligations'][0]['outcome']=='owed'
|
|
+
|
|
+
|
|
+def test_revoked_recovery_credential_cannot_fake_cleanup_success(owner,tmp_path):
|
|
+ native,journal,binding,call,custody=protected(owner,tmp_path);crash(native,journal,binding)
|
|
+ status,data=call('POST','auth/token/create',{'policies':['root'],'ttl':'30s'})
|
|
+ assert status==200
|
|
+ token=data['auth']['client_token']
|
|
+ assert call('POST','auth/token/revoke',{'token':token})[0] in {200,204}
|
|
+ result=recover_fixture_cleanup(journal,permit(native,custody,binding,owner_token=token))
|
|
+ assert not result['confirmed'] and all(x['outcome']=='owed' for x in result['obligations'])
|
|
+ assert call('GET','sys/policies/acl/'+native.policy_name)[0]==200
|
|
+ # A separately admitted owner attempt can recover after the failed attempt;
|
|
+ # neither the failed attempt nor its receipt grants an automatic retry.
|
|
+ assert recover_fixture_cleanup(journal,permit(native,custody,binding))['confirmed']
|
|
+ assert journal.db.execute('SELECT COUNT(*) FROM recovery_attempts').fetchone()[0]==2
|
|
+
|
|
+
|
|
+def test_policy_name_reuse_never_deletes_different_policy(owner,tmp_path):
|
|
+ native,journal,binding,call,custody=protected(owner,tmp_path);crash(native,journal,binding)
|
|
+ replacement='path "unrelated" { capabilities = ["read"] }'
|
|
+ assert call('PUT','sys/policies/acl/'+native.policy_name,{'policy':replacement})[0] in {200,204}
|
|
+ result=recover_fixture_cleanup(journal,permit(native,custody,binding))
|
|
+ assert not result['confirmed']
|
|
+ assert [x['outcome'] for x in result['obligations']]==['confirmed','confirmed','owed']
|
|
+ assert call('GET','sys/policies/acl/'+native.policy_name)[1]['data']['policy']==replacement
|
|
+
|
|
+
|
|
+def test_private_custody_permissions_and_identity_binding_fail_closed(owner,tmp_path):
|
|
+ native,journal,binding,call,custody=protected(owner,tmp_path)
|
|
+ path=custody._path(binding);path.chmod(0o644)
|
|
+ with pytest.raises(Refused):custody.read(binding)
|
|
+ path.chmod(0o600)
|
|
+ data=json.loads(path.read_bytes());data['payload']['binding']['operation_id']='other'
|
|
+ path.write_text(json.dumps(data))
|
|
+ with pytest.raises(Refused):custody.read(binding)
|
|
+
|
|
+
|
|
+def test_lost_recovery_response_stays_unknown_until_new_independent_attempt(owner,tmp_path,monkeypatch):
|
|
+ native,journal,binding,call,custody=protected(owner,tmp_path);crash(native,journal,binding)
|
|
+ import task_execution_recovery as recovery_module
|
|
+ original=recovery_module.http.client.HTTPConnection.getresponse
|
|
+ once=[True]
|
|
+ def lost(connection):
|
|
+ response=original(connection)
|
|
+ if once[0]:
|
|
+ once[0]=False
|
|
+ response.close()
|
|
+ raise Refused('synthetic lost recovery response after real revoke')
|
|
+ return response
|
|
+ first=permit(native,custody,binding)
|
|
+ with monkeypatch.context() as patch:
|
|
+ patch.setattr(recovery_module.http.client.HTTPConnection,'getresponse',lost)
|
|
+ assert not recover_fixture_cleanup(journal,first)['confirmed']
|
|
+ assert journal.observe(binding)['external_outcome_unknown']
|
|
+ with pytest.raises(Refused):recover_fixture_cleanup(journal,first)
|
|
+ assert recover_fixture_cleanup(journal,permit(native,custody,binding))['confirmed']
|
|
+ # Reconciliation does not erase the historic uncertain invocation.
|
|
+ assert journal.observe(binding)['external_outcome_unknown']
|
|
+ assert journal.db.execute("SELECT COUNT(*) FROM recovery_calls WHERE outcome='pending'").fetchone()[0]==1
|
|
+
|
|
+
|
|
+@pytest.fixture
|
|
+def capsule(tmp_path):
|
|
+ from task_execution_journal import Binding,Cleanup
|
|
+ from task_execution_plan import cleanup_specs
|
|
+ binding=Binding('SYNTHETIC','sha256:'+'a'*64,'12345678-1234-1234-1234-123456789abc','b'*64)
|
|
+ directory=tmp_path/'private-capsule';directory.mkdir(mode=0o700)
|
|
+ custody=FixtureRecoveryCustody(directory,integrity_key=secrets.token_bytes(32))
|
|
+ cleanup=tuple(Cleanup(x['kind'],x['call_digest']) for x in cleanup_specs(binding.operation_id))
|
|
+ custody.prepare(binding,cleanup,12345,'path "fixture" {}\n','synthetic-private-accessor')
|
|
+ return custody,binding
|
|
+
|
|
+
|
|
+@pytest.mark.parametrize('mutate',[
|
|
+ lambda v:v.update(extra=True),lambda v:v.pop('phase'),lambda v:v.update(binding=True),
|
|
+ lambda v:v['binding'].update(approval_id=True),lambda v:v['binding'].update(extra='x'),
|
|
+ lambda v:v['binding'].update(request_digest='0'*64),lambda v:v.update(port=True),
|
|
+ lambda v:v.update(port=12345.0),lambda v:v.update(port=0),lambda v:v.update(phase=[]),
|
|
+ lambda v:v.update(phase='issued'),lambda v:v.update(revision=False),
|
|
+ lambda v:v.update(revision=0.0),lambda v:v.update(identity_accessor=False),
|
|
+ lambda v:v.update(task_accessor='unexpected-before-issue'),lambda v:v.update(policy_hcl=[]),
|
|
+ lambda v:v.update(cleanup={}),lambda v:v['cleanup'].pop(),
|
|
+ lambda v:v['cleanup'].__setitem__(0,False),lambda v:v['cleanup'][0].update(extra=True),
|
|
+ lambda v:v['cleanup'][0].update(kind=True),lambda v:v['cleanup'][0].update(call_digest=0),
|
|
+ lambda v:v['cleanup'][0].update(call_digest='f'*64),lambda v:v['cleanup'].reverse(),
|
|
+])
|
|
+def test_authenticated_but_damaged_capsule_schema_refuses(capsule,mutate):
|
|
+ custody,binding=capsule
|
|
+ value=custody.read(binding);mutate(value)
|
|
+ # Trusted test owner re-signs malformed payload to exercise schema defenses
|
|
+ # independently of MAC validation; untrusted corruption cannot do this.
|
|
+ custody._write(binding,value)
|
|
+ with pytest.raises(Refused):custody.read(binding)
|
|
+
|
|
+
|
|
+def test_capsule_corruption_wrong_key_duplicate_and_nonfinite_refuse(capsule):
|
|
+ custody,binding=capsule;path=custody._path(binding);original=path.read_bytes()
|
|
+ value=json.loads(original);value['payload']['identity_accessor']='different-synthetic-accessor'
|
|
+ path.write_text(json.dumps(value))
|
|
+ with pytest.raises(Refused):custody.read(binding)
|
|
+ path.write_bytes(original)
|
|
+ other=FixtureRecoveryCustody(custody.directory,integrity_key=b'X'*32)
|
|
+ with pytest.raises(Refused):other.read(binding)
|
|
+ for raw in [b'{',original[:-1]+b',"mac":"duplicate"}',original.replace(b'12345',b'NaN'),b'[]',b'{}',b'{"payload":{},"mac":"'+b'\xc3\xa9'*64+b'"}']:
|
|
+ path.write_bytes(raw)
|
|
+ with pytest.raises(Refused):custody.read(binding)
|
|
+ path.write_bytes(original)
|
|
+ assert custody.read(binding)['identity_accessor']=='synthetic-private-accessor'
|
|
+
|
|
+
|
|
+def test_exact_absence_and_single_terminal_lf_contract():
|
|
+ from task_execution_recovery import _policy_absent,_same_policy
|
|
+ assert _policy_absent(404,{'errors':[]})
|
|
+ for status,body in [(403,{'errors':[]}),(404,{}),(404,{'errors':['denied']}),
|
|
+ (404,{'errors':None}),(404,{'errors':[]} | {'data':{}}),
|
|
+ (404,{'errors':False}),(404,{'errors':0}),(404,{'errors':{}})]:
|
|
+ assert not _policy_absent(status,body)
|
|
+ assert _same_policy('fixture','fixture\n')
|
|
+ assert _same_policy('fixture\n','fixture\n')
|
|
+ for text in ['fixture \n','fixture\t','fixture\r\n','fixture\n\n',False,None]:
|
|
+ assert not _same_policy(text,'fixture\n')
|
|
+
|
|
+
|
|
+@pytest.mark.parametrize('after_delete',[False,True])
|
|
+def test_recovery_policy_absence_rejects_misleading_native_http_bodies(owner,tmp_path,monkeypatch,after_delete):
|
|
+ native,journal,binding,call,custody=protected(owner,tmp_path);crash(native,journal,binding)
|
|
+ import task_execution_recovery as module
|
|
+ malformed=[{}, {'errors':['permission denied']}, {'errors':None}, {'errors':False},
|
|
+ {'errors':0}, {'errors':[],'data':{}}, {'errors':['not really absent']},
|
|
+ b'{"errors":["permission denied"],"errors":[]}',b'{"errors":NaN}',b'not JSON']
|
|
+ path='sys/policies/acl/'+native.policy_name
|
|
+ for body in malformed:
|
|
+ sent=[];gets=[0]
|
|
+ class Response:
|
|
+ def __init__(self,status,data):self.status=status;self.data=data
|
|
+ def read(self,limit):return self.data if type(self.data) is bytes else json.dumps(self.data).encode()
|
|
+ class Connection:
|
|
+ def __init__(self,*args,**kwargs):pass
|
|
+ def request(self,method,url,**kwargs):sent.append((method,url))
|
|
+ def getresponse(self):
|
|
+ method,url=sent[-1]
|
|
+ if url=='/v1/'+path:
|
|
+ if method=='DELETE':return Response(204,{})
|
|
+ gets[0]+=1
|
|
+ if after_delete and gets[0]==1:
|
|
+ return Response(200,{'data':{'policy':native.hcl[:-1]}})
|
|
+ return Response(404,body)
|
|
+ return Response(403,{'errors':['synthetic refusal']})
|
|
+ def close(self):pass
|
|
+ with monkeypatch.context() as patch:
|
|
+ patch.setattr(module.http.client,'HTTPConnection',Connection)
|
|
+ result=recover_fixture_cleanup(journal,permit(native,custody,binding))
|
|
+ assert not result['confirmed'] and result['obligations'][2]['outcome']=='owed'
|
|
+ assert ('DELETE','/v1/'+path) in sent if after_delete else ('DELETE','/v1/'+path) not in sent
|
|
+ # The denied/malformed probes above emitted no real native calls.
|
|
+ assert call('GET',path)[0]==200
|
|
+ assert recover_fixture_cleanup(journal,permit(native,custody,binding))['confirmed']
|
|
+
|
|
+
|
|
+def test_authenticated_prepared_capsule_rollback_cannot_hide_later_mint(owner,tmp_path):
|
|
+ native,journal,binding,call,custody=protected(owner,tmp_path)
|
|
+ old=custody._path(binding).read_bytes()
|
|
+ crash(native,journal,binding)
|
|
+ current=custody.read(binding)
|
|
+ custody._path(binding).write_bytes(old)
|
|
+ # Still authentic under the owner key, but the durable journal proves the
|
|
+ # operation reached issuance. The stale prepared record cannot prove absence.
|
|
+ result=recover_fixture_cleanup(journal,permit(native,custody,binding))
|
|
+ assert not result['confirmed'] and result['obligations'][0]['outcome']=='owed'
|
|
+ assert call('POST','auth/token/lookup-accessor',{'accessor':current['task_accessor']})[0]==200
|
|
+ assert call('POST','auth/token/revoke-accessor',{'accessor':current['task_accessor']})[0] in {200,204}
|
|
diff --git a/workplans/RPF-WP-0035-credential-lane-implementation.md b/workplans/RPF-WP-0035-credential-lane-implementation.md
|
|
index 8b09305..f58a6b9 100644
|
|
--- a/workplans/RPF-WP-0035-credential-lane-implementation.md
|
|
+++ b/workplans/RPF-WP-0035-credential-lane-implementation.md
|
|
@@ -1290,3 +1290,20 @@ CAS records consumed/unknown with zero native effects. Exact dependency/source
|
|
hashes, tests and remaining gates are retained with the source package and in
|
|
docs/task-execution-mediator-seam.md. No real grant/custody, policy default, owner
|
|
sync or native production acceptance is performed; T12 stays progress.
|
|
+
|
|
+2026-10-16 T12 recovery successor source preparation: optional independently
|
|
+installed fixture custody durably retains operation-bound private accessors and
|
|
+issuance checkpoints. Fresh cleanup-only owner permits can reconcile after actual
|
|
+executor process death using fixed revoke/readback and policy delete/readback,
|
|
+without restoring business/consume admission. Separate durable attempt evidence
|
|
+retains original unknown outcomes. Lost token-create response remains unresolved
|
|
+when no accessor was committed. Source-only process-loss fixtures are not native
|
|
+production custody or recovery-role admission; T12 remains progress.
|
|
+
|
|
+2026-10-16 T12 recovery R2 source refinement retains R1's overbroad 404 failure:
|
|
+policy cleanup now requires exact native empty-errors absence before/after delete;
|
|
+HCL uses only documented single optional terminal LF normalization. Private
|
|
+capsules are owner-key HMAC authenticated, strict nested schemas/ordered cleanup
|
|
+commitments reject damage, and journal history prevents stale prepared capsules
|
|
+from hiding issuance. Fixture process-loss source proof remains separate from
|
|
+unadmitted production custody, integrity-key storage and recovery-role scope.
|