From b9eaa200601f1226d13efe82472dfec81c2999aa Mon Sep 17 00:00:00 2001 From: codex Date: Fri, 16 Oct 2026 16:43:20 +0200 Subject: [PATCH] Recover exact task cleanup after executor process loss Assistant: codex Assistant-Model: gpt-6-astra Assistant-Session: 01a140c4-354a-7823-8f0d-8f717397f786 --- docs/task-execution-native-fixture.md | 84 +++++ scripts/task_execution_journal.py | 4 + scripts/task_execution_native_fixture.py | 15 +- scripts/task_execution_recovery.py | 336 ++++++++++++++++++ tests/test_task_execution_recovery.py | 320 +++++++++++++++++ ...-WP-0035-credential-lane-implementation.md | 17 + 6 files changed, 774 insertions(+), 2 deletions(-) create mode 100644 scripts/task_execution_recovery.py create mode 100644 tests/test_task_execution_recovery.py diff --git a/docs/task-execution-native-fixture.md b/docs/task-execution-native-fixture.md index 1dd6b91..f4ff070 100644 --- a/docs/task-execution-native-fixture.md +++ b/docs/task-execution-native-fixture.md @@ -121,3 +121,87 @@ retry cleanup. Native effects can survive process loss. Cleanup handles remain memory-only: no independently authorized recovery mechanism is implemented here. This closes disposable sequential apply acceptance only, not TSA03 or native production recovery/admission. + + +## Independent cleanup recovery source successor + +`task_execution_recovery.py` adds separately installed fixture-only owner custody +and fresh recovery permits. Optional `recovery_custody` on NativeFixtureAdapter +persists the original binding/cleanup commitments, exact operation policy, backend +port and identity accessor before any grant effect; it commits mint intent before +token creation and task accessor immediately after a valid issued response. +Bearer credentials are never persisted. The custody directory is separate from +the public journal, owner-only 0700/0600, bounded at 64KiB, no-follow, with atomic +replacement and fsync. It is real process-independent persistence for synthetic +fixtures. It does not isolate same-UID hostile execution, encrypt at rest, select +production storage or establish a production secret policy. Those are explicit +native custody/deployment gates, not supplied by Python types or file modes. + +The independent owner reads its private custody revision and installs a fresh +bounded permit, current owner credential and retained live disposable Popen. +Permit revocation is durably owner-recorded. The recovery runner has no mint, +consume, policy-apply or business method. It dispatches only fixed exact-accessor +revoke/readback and exact `task-exec-` policy read/delete/readback. +A differently recreated policy is never deleted. Backend admission remains the +same loopback-only owner-spawned dev process; no estate backend can be selected. +Accessor-based recovery is an independently admitted alternate cleanup projection, +not a reinterpretation of the original revoke-self call digest. Production needs +exact owner-reviewed recovery scope; original business grant does not imply it. + +Fresh recovery holds the journal process lock but never sets business admission +or consumed-admission fields. Separate recovery attempt/call/result tables retain +original cleanup statuses before reconciliation, write native intent before calls, +and preserve failed/pending outcomes. Proven cleanup alone updates the exact +original cleanup obligation to confirmed. Business state and original pending +call uncertainty remain unchanged. Reusing the original approval, continuing +metadata calls and reusing a recovery permit refuse. Failure needs another +independent owner permit; there is no automatic retry. Expired/revoked permits +or changed private custody revision refuse before calls. + +A token-create response lost before accessor custody remains unresolved token +cleanup debt: policy deletion/identity revocation can be independently confirmed, +but cannot imply task-token revocation. Neither expiry guessed from wall clock +nor policy absence fabricates token absence. A production successor must admit +an independent exact ambiguous-issuance reconciliation mechanism or prove bounded +expiry from owner backend evidence. No such new grant is implemented here. + +`test_task_execution_recovery.py` forks a real executor and terminates it without +finally cleanup after issue, business effect/lost response and revoke/lost response. +The independent parent, holding the backend and separately installed custody, +recovers specific obligations and verifies actual token/policy absence. Additional +cases cover lost-mint response, revoked credential, permit revocation/expiry, +revision drift, namespace policy-name reuse and private-file/binding refusals. +This is source fixture proof only. Native custody, isolation, exact recovery-role +permissions, retention/encryption, clock admission, audit delivery and human scope +approval remain missing. No generic broker, production reader, grant or CLI is added. + + +### Recovery R2 exact absence and authenticated capsule refinement + +R1 overclaimed policy absence for arbitrary 404 bodies. R2 requires exactly +`{"errors":[]}` for both initial GET and post-DELETE GET. Other 404 bodies, +permission denials, extra data, malformed/duplicate-key/nonfinite JSON and failed +readback cannot confirm policy cleanup. A successful DELETE status alone cannot +confirm absence. Recovery native policy comparison admits precisely one optional +terminal LF, matching the retained OpenBao HCL source/readback convention; spaces, +tabs, CR, extra terminal LF and any other byte change are refused. This is a +narrow declared normalization, not full whitespace or semantic equivalence. + +Private fixture capsules now carry HMAC-SHA256 authentication under an exactly +32-byte integrity key supplied independently by the recovery owner. The key is +never stored in the capsule or public journal, and is not a bearer credential. +The fixture owner retains this key independently across executor death; a fresh +owner custody object must supply the same key. Wrong-key or damaged accessor/HCL +bytes refuse before admission. This is integrity protection, not encryption or +same-UID isolation; production key custody, rotation/retention and recovery actor +selection remain explicitly unadmitted. + +After authentication the reader enforces exact top-level and nested binding/ +cleanup key sets and types, validated Binding/Cleanup formats and the ordered +source-derived cleanup_specs operation digests. Duplicate JSON keys, nonfinite +numbers, malformed phase/revision combinations and type-coercible values refuse. +The reader bounds bytes, rejects indirect/nonprivate files and detects stat/size +changes during its single read. A valid older prepared capsule still cannot +prove a task token absent when durable journal state/calls reached issuance: +that remains unknown cleanup debt. Authentication alone is not anti-rollback. +No business or consume admission is restored and no production scope is granted. diff --git a/scripts/task_execution_journal.py b/scripts/task_execution_journal.py index 8f1b9a2..1eaa583 100644 --- a/scripts/task_execution_journal.py +++ b/scripts/task_execution_journal.py @@ -104,6 +104,9 @@ class Journal: self.db.execute("CREATE TABLE IF NOT EXISTS calls (operation TEXT NOT NULL, ordinal INTEGER NOT NULL, call_digest TEXT NOT NULL, outcome TEXT NOT NULL, evidence TEXT, PRIMARY KEY(operation,ordinal))") self.db.execute("CREATE TABLE IF NOT EXISTS cleanup (operation TEXT NOT NULL, ordinal INTEGER NOT NULL, kind TEXT NOT NULL, call_digest TEXT NOT NULL, outcome TEXT NOT NULL, evidence TEXT, PRIMARY KEY(operation,ordinal), UNIQUE(operation,kind))") self.db.execute("CREATE TABLE IF NOT EXISTS cleanup_calls (operation TEXT NOT NULL, obligation INTEGER NOT NULL, step INTEGER NOT NULL, call_digest TEXT NOT NULL, outcome TEXT NOT NULL, evidence TEXT, PRIMARY KEY(operation,obligation,step))") + self.db.execute("CREATE TABLE IF NOT EXISTS recovery_attempts (attempt TEXT PRIMARY KEY, operation TEXT NOT NULL, original TEXT NOT NULL)") + self.db.execute("CREATE TABLE IF NOT EXISTS recovery_calls (attempt TEXT NOT NULL, obligation INTEGER NOT NULL, step INTEGER NOT NULL, digest TEXT NOT NULL, outcome TEXT NOT NULL, evidence TEXT, PRIMARY KEY(attempt,obligation,step))") + self.db.execute("CREATE TABLE IF NOT EXISTS recovery_results (attempt TEXT NOT NULL, obligation INTEGER NOT NULL, confirmed INTEGER NOT NULL, evidence TEXT NOT NULL, PRIMARY KEY(attempt,obligation))") self._held = False self._admission = None self._consumed_admission = None @@ -228,6 +231,7 @@ class Journal: pending = self.db.execute("SELECT COUNT(*) FROM calls WHERE operation=? AND outcome='pending'", (binding.operation_id,)).fetchone()[0] pending += self.db.execute("SELECT COUNT(*) FROM cleanup WHERE operation=? AND outcome='pending'", (binding.operation_id,)).fetchone()[0] pending += self.db.execute("SELECT COUNT(*) FROM cleanup_calls WHERE operation=? AND outcome='pending'", (binding.operation_id,)).fetchone()[0] + pending += self.db.execute("SELECT COUNT(*) FROM recovery_calls AS c JOIN recovery_attempts AS a ON c.attempt=a.attempt WHERE a.operation=? AND c.outcome='pending'", (binding.operation_id,)).fetchone()[0] return {"state": row[3], "evidence_digest": row[4], "dispatch_allowed": False, "external_outcome_unknown": bool(pending)} diff --git a/scripts/task_execution_native_fixture.py b/scripts/task_execution_native_fixture.py index caa7da1..31a822b 100644 --- a/scripts/task_execution_native_fixture.py +++ b/scripts/task_execution_native_fixture.py @@ -3,7 +3,8 @@ This adapter deliberately admits only a disposable loopback fixture. It supplies real native primitive coverage while production grant schema, identity/consume and origin/source admission are unfinished. It cannot open an estate backend. Owner -credentials/handles stay in memory; no subprocess, redirects or ambient proxy. +bearer credentials stay in memory; optional owner-private fixture custody persists +accessors for independently admitted cleanup recovery; no redirects or ambient proxy. Python privacy is not credential isolation; production needs the owner runtime. """ from __future__ import annotations @@ -41,7 +42,7 @@ class FixtureAdmission: class NativeFixtureAdapter: def __init__(self, compiler: Compiler, plan: Plan, journal: Journal, - binding: Binding, admission: FixtureAdmission): + binding: Binding, admission: FixtureAdmission, *, recovery_custody=None): if type(admission) is not FixtureAdmission or type(admission.descriptor) is not bytes: raise Refused('independent fixture admission required') self.compiler,self.plan,self.journal,self.binding=compiler,plan,journal,binding @@ -100,6 +101,12 @@ class NativeFixtureAdapter: if self.meta['cleanup'] != cleanup_specs(binding.operation_id): raise Refused('cleanup plan mismatch') self.task=None; self._mint_attempted=False; self._policy_attempted=False; self._ordinal=0; self._next_metadata=0 self._cleanup_steps={item.kind:0 for item in self.cleanup} + self.recovery_custody=recovery_custody + if recovery_custody is not None: + from task_execution_recovery import FixtureRecoveryCustody + if type(recovery_custody) is not FixtureRecoveryCustody or journal.observe(binding) is not None: + raise Refused('fresh independent fixture recovery custody required') + recovery_custody.prepare(binding,self.cleanup,self.port,self.hcl,admission.identity_session.accessor) def _live(self): if not self.journal._held or self.journal._admission!=self.binding or self.journal._consumed_admission!=self.binding: @@ -177,6 +184,8 @@ class NativeFixtureAdapter: 'type':'service','display_name':'task-exec-'+self.binding.operation_id,'no_parent':False} if body!=self.meta['execution_grant']['openbao']['token_create_body']: raise Refused('native token body differs from bound owner projection') + if self.recovery_custody is not None: + self.recovery_custody.checkpoint(self.binding,'mint-intent') self._mint_attempted=True status,data=self._business('POST','auth/token/create',root,encoded(body));self._ok(status,data,{200}) auth=data.get('auth',{}) @@ -185,6 +194,8 @@ class NativeFixtureAdapter: # Retain handles before validation so finally cleanup can revoke even a # malformed overbroad issue response. Never return token-bearing metadata. self.task=PrivateSession(auth['client_token'],auth['accessor']) + if self.recovery_custody is not None: + self.recovery_custody.checkpoint(self.binding,'issued',self.task.accessor) status,data=self._business('POST','auth/token/lookup',root,encoded({'token':self.task.token}));self._ok(status,data,{200}) info=data.get('data',{}) if (info.get('policies')!=[self.policy_name] or info.get('identity_policies',[]) or info.get('renewable') is not False diff --git a/scripts/task_execution_recovery.py b/scripts/task_execution_recovery.py new file mode 100644 index 0000000..ab8b8cb --- /dev/null +++ b/scripts/task_execution_recovery.py @@ -0,0 +1,336 @@ +"""Owner-only cleanup recovery for the disposable native fixture. + +Private, separately installed custody persists accessors, never bearer tokens. +This is a fixture implementation, not production custody/isolation admission. +The recovery owner supplies a fresh capability and credential independently of +public request/journal data. No business adapter or resumed admission is built. +""" +from __future__ import annotations +from dataclasses import asdict, dataclass, field +import hashlib +import hmac +import http.client +import json +import os +from pathlib import Path +import secrets +import subprocess +import time +import uuid + +from task_execution_journal import Binding, Cleanup, Journal, Refused +from task_execution_plan import cleanup_specs + + +def _encoded(value): + return json.dumps(value, sort_keys=True, separators=(',', ':'), allow_nan=False).encode() + + +def _digest(value): + return hashlib.sha256(_encoded(value)).hexdigest() + + +def _json_unique(pairs): + result={} + for key,value in pairs: + if key in result: raise Refused('duplicate private custody field') + result[key]=value + return result + + +def _invalid_constant(value): + raise Refused('nonfinite private custody value') + + +def _policy_absent(status, response): + # Exactly the native OpenBao empty-errors 404. Denial/malformed/error-shaped + # bodies never prove absence, including after a successful delete response. + return (status==404 and type(response) is dict and set(response)=={'errors'} + and type(response['errors']) is list and not response['errors']) + + +def _same_policy(actual, expected): + # OpenBao may omit one terminal LF from HCL source. Only that equivalence + # is admitted: spaces, tabs, CR and additional blank lines remain meaningful. + if type(actual) is not str or type(expected) is not str: return False + strip_one=lambda value:value[:-1] if value.endswith('\n') else value + return strip_one(actual)==strip_one(expected) + + +class FixtureRecoveryCustody: + """Independent owner-private persistent fixture custody; no public API. + + Mode/ownership and no-follow checks protect against accidental exposure. + Same-UID hostile code is NOT isolated. Production must supply independently + enforced owner storage/executor authority; this implementation refuses estate + backends and is never evidence of a production grant. + """ + def __init__(self, directory: Path, *, integrity_key: bytes): + if type(integrity_key) is not bytes or len(integrity_key)!=32: + raise Refused('independent owner capsule integrity key required') + self._integrity_key=integrity_key + self.directory = directory + if directory.is_symlink() or directory.absolute() != directory.resolve(): + raise Refused('indirect recovery custody directory') + Journal._private(directory.stat(), directory=True) + + def _path(self, binding): + if type(binding) is not Binding: raise Refused('exact recovery binding required') + binding.validate() + return self.directory / (binding.operation_id + '.private.json') + + def _sync(self): + fd = os.open(self.directory, os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW) + try: os.fsync(fd) + finally: os.close(fd) + + def read(self, binding): + try: fd = os.open(self._path(binding), os.O_RDONLY | os.O_NOFOLLOW | os.O_NONBLOCK) + except OSError: raise Refused('private custody unavailable') from None + try: + before=os.fstat(fd) + Journal._private(before) + with os.fdopen(fd, 'rb', closefd=False) as stream: raw = stream.read(65537) + after=os.fstat(fd) + identity=lambda value:(value.st_dev,value.st_ino,value.st_size,value.st_mtime_ns,value.st_ctime_ns,value.st_mode,value.st_nlink) + if identity(before)!=identity(after) or len(raw)!=before.st_size: + raise Refused('private custody changed during read') + finally: os.close(fd) + if len(raw) > 65536: raise Refused('oversize private custody record') + try: capsule = json.loads(raw,object_pairs_hook=_json_unique,parse_constant=_invalid_constant) + except Exception: raise Refused('invalid private custody record') from None + if (type(capsule) is not dict or set(capsule)!={'payload','mac'} or type(capsule['mac']) is not str + or len(capsule['mac'])!=64 or any(x not in '0123456789abcdef' for x in capsule['mac'])): + raise Refused('invalid authenticated private capsule') + expected_mac=hmac.new(self._integrity_key,_encoded(capsule['payload']),hashlib.sha256).hexdigest() + if not hmac.compare_digest(capsule['mac'],expected_mac): + raise Refused('private capsule integrity mismatch') + value=capsule['payload'] + expected={'binding','cleanup','port','policy_hcl','identity_accessor','task_accessor','phase','revision'} + if type(value) is not dict or set(value)!=expected: + raise Refused('invalid private custody schema') + stored=value['binding'] + if (type(stored) is not dict or set(stored)!=set(asdict(binding)) + or any(type(v) is not str for v in stored.values())): + raise Refused('invalid private custody binding schema') + restored=Binding(**stored) + restored.validate() + if asdict(restored)!=asdict(binding): raise Refused('private custody binding mismatch') + if (type(value['port']) is not int or not 1<=value['port']<=65535 + or type(value['policy_hcl']) is not str or not value['policy_hcl'] + or type(value['identity_accessor']) is not str or not value['identity_accessor'] + or type(value['phase']) is not str or value['phase'] not in {'prepared','mint-intent','issued'} + or type(value['revision']) is not int or value['revision']!={'prepared':0,'mint-intent':1,'issued':2}[value['phase']] + or (value['phase']=='issued' and (type(value['task_accessor']) is not str or not value['task_accessor'])) + or (value['phase']!='issued' and value['task_accessor'] is not None) + or type(value['cleanup']) is not list or len(value['cleanup'])!=3): + raise Refused('invalid private custody schema') + for item in value['cleanup']: + if type(item) is not dict or set(item)!={'kind','call_digest'}: + raise Refused('invalid private cleanup schema') + Cleanup(**item).validate() + expected_cleanup=[{'kind':item['kind'],'call_digest':item['call_digest']} + for item in cleanup_specs(binding.operation_id)] + if value['cleanup']!=expected_cleanup: + raise Refused('private cleanup differs from deterministic owner commitment') + return value + + def _write(self, binding, value, initial=False): + capsule={'payload':value,'mac':hmac.new(self._integrity_key,_encoded(value),hashlib.sha256).hexdigest()} + raw = _encoded(capsule) + if len(raw) > 65536: raise Refused('oversize private custody record') + path = self._path(binding) + if initial: + fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW, 0o600) + else: + # Existing owner-private record checked before replacement. Unique + # temp + fsync + replace + directory fsync never truncates old state. + self.read(binding) + tmp = self.directory / ('.pending-' + secrets.token_hex(16)) + fd = os.open(tmp, os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW, 0o600) + try: + with os.fdopen(fd, 'wb', closefd=False) as stream: + stream.write(raw); stream.flush(); os.fsync(fd) + finally: os.close(fd) + if not initial: os.replace(tmp, path) + self._sync() + + def prepare(self, binding, cleanup, port, policy_hcl, identity_accessor): + value = {'binding': asdict(binding), 'cleanup': [asdict(x) for x in cleanup], + 'port': port, 'policy_hcl': policy_hcl, + 'identity_accessor': identity_accessor, 'task_accessor': None, + 'phase': 'prepared', 'revision': 0} + self._write(binding, value, initial=True) + + def checkpoint(self, binding, phase, task_accessor=None): + value = self.read(binding) + if (value['phase'], phase) not in {('prepared', 'mint-intent'), ('mint-intent', 'issued')}: + raise Refused('invalid protected recovery checkpoint') + if phase == 'issued' and (type(task_accessor) is not str or not task_accessor): + raise Refused('protected issued accessor required') + value.update(phase=phase, task_accessor=task_accessor, revision=value['revision'] + 1) + self._write(binding, value) + + def revoke(self, permit): + if type(permit) is not FixtureRecoveryPermit or permit.custody is not self: + raise Refused('exact owner recovery permit required') + uuid.UUID(permit.attempt_id) + path=self.directory / (permit.attempt_id+'.revoked') + fd=os.open(path,os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW,0o600) + try:os.fsync(fd) + finally:os.close(fd) + self._sync() + + def admit(self, binding, *, owner_token, server_process, valid_seconds=30): + """Trusted recovery-owner action, not deserialization of caller flags. + + Fresh admission seals the current private custody revision. The owner + supplies its credential and retained backend process separately. Neither + token nor private capsule digest is emitted to the public journal. + """ + value = self.read(binding) + if type(valid_seconds) is not int or not 1 <= valid_seconds <= 60: + raise Refused('bounded recovery admission required') + if type(owner_token) is not str or not owner_token: + raise Refused('independent recovery owner credential required') + return FixtureRecoveryPermit(str(uuid.uuid4()), binding, self, _digest(value), + owner_token, server_process, time.monotonic()+valid_seconds) + + +@dataclass(frozen=True, repr=False) +class FixtureRecoveryPermit: + attempt_id: str + binding: Binding + custody: FixtureRecoveryCustody + private_revision_digest: str = field(repr=False) + owner_token: str = field(repr=False) + server_process: subprocess.Popen = field(repr=False) + deadline: float + + +class _Recorder: + def __init__(self, journal, permit, value): + self.journal, self.permit = journal, permit + b = permit.binding + if journal._admission is not None or journal._consumed_admission is not None: + raise Refused('recovery cannot share a business admission') + if journal.observe(b) is None: raise Refused('original operation absent') + current = journal.cleanup_status(b)['obligations'] + if [(x['kind'], x['call_digest']) for x in current] != [(x['kind'], x['call_digest']) for x in value['cleanup']]: + raise Refused('original cleanup commitment mismatch') + if all(x['outcome'] == 'confirmed' for x in current): + raise Refused('cleanup already confirmed') + self.current = current + # An authenticated older prepared capsule must not erase later issuance + # uncertainty. Without an issued accessor, any native call or state at + # the grant boundary makes absence unprovable. Recovery is conservative. + self.no_mint_proved=(journal.observe(b)['state'] in {'reserved','consume-confirmed','authenticating'} + and journal.db.execute('SELECT COUNT(*) FROM calls WHERE operation=?',(b.operation_id,)).fetchone()[0]==0) + db = journal.db + try: db.execute('INSERT INTO recovery_attempts VALUES (?,?,?)', (permit.attempt_id,b.operation_id,json.dumps(current,sort_keys=True))) + except Exception: raise Refused('recovery admission already used') from None + journal._sync_directory() + + def before(self, ordinal, step, method, path): + digest = _digest({'kind': self.current[ordinal]['kind'], 'commitment':self.current[ordinal]['call_digest'], 'method':method,'path':path,'step':step}) + self.journal.db.execute("INSERT INTO recovery_calls VALUES (?,?,?,?,'pending',NULL)", (self.permit.attempt_id,ordinal,step,digest)) + self.journal._sync_directory() + return digest + + def after(self, ordinal, step, status): + self.journal.db.execute("UPDATE recovery_calls SET outcome='complete', evidence=? WHERE attempt=? AND obligation=? AND step=? AND outcome='pending'", (_digest({'http_status':status}),self.permit.attempt_id,ordinal,step)) + self.journal._sync_directory() + + def finish(self, ordinal, confirmed): + evidence = _digest({'attempt':self.permit.attempt_id,'obligation':ordinal,'confirmed':confirmed}) + db = self.journal.db + try: + db.execute('BEGIN IMMEDIATE') + pending = db.execute("SELECT COUNT(*) FROM recovery_calls WHERE attempt=? AND obligation=? AND outcome='pending'",(self.permit.attempt_id,ordinal)).fetchone()[0] + if confirmed and pending: raise Refused('pending recovery cannot confirm') + db.execute('INSERT INTO recovery_results VALUES (?,?,?,?)',(self.permit.attempt_id,ordinal,int(confirmed),evidence)) + if confirmed: + item=self.current[ordinal] + db.execute("UPDATE cleanup SET outcome='confirmed', evidence=? WHERE operation=? AND ordinal=? AND kind=? AND call_digest=?",(evidence,self.permit.binding.operation_id,ordinal,item['kind'],item['call_digest'])) + db.execute('COMMIT');self.journal._sync_directory() + except BaseException: + if db.in_transaction: db.execute('ROLLBACK') + raise + + +def recover_fixture_cleanup(journal: Journal, permit: FixtureRecoveryPermit): + """Fresh independent owner admission: fixed cleanup only, no business retry. + + Each attempt preserves original pending/unconfirmed rows in recovery_attempts + and native pending call rows unchanged. A fresh owner permit is required for + another attempt after failure; never autonomously retry uncertain mutations. + """ + if type(permit) is not FixtureRecoveryPermit or type(permit.custody) is not FixtureRecoveryCustody: + raise Refused('independent typed recovery admission required') + if str(uuid.UUID(permit.attempt_id)) != permit.attempt_id: + raise Refused('invalid recovery attempt identity') + value = permit.custody.read(permit.binding) + if _digest(value) != permit.private_revision_digest: + raise Refused('private custody revision changed') + proc=permit.server_process + if (type(proc) is not subprocess.Popen or proc.poll() is not None or type(proc.args) is not list + or '-dev' not in proc.args or '-dev-no-store-token' not in proc.args + or '-dev-listen-address=127.0.0.1:'+str(value['port']) not in proc.args): + raise Refused('same live disposable backend required') + def fresh(): + if (permit.custody.directory / (permit.attempt_id+'.revoked')).exists(): + raise Refused('recovery permit revoked') + if time.monotonic() >= permit.deadline: raise Refused('recovery admission expired') + if proc.poll() is not None: raise Refused('disposable backend lost') + if _digest(permit.custody.read(permit.binding)) != permit.private_revision_digest: + raise Refused('private custody changed during recovery') + fresh() + with journal.locked(): + record = _Recorder(journal,permit,value) + for ordinal,item in enumerate(record.current): + if item['outcome']=='confirmed': continue + step=0 + def call(method,path,body=None): + nonlocal step + fresh(); index=step; step+=1 + record.before(ordinal,index,method,path) + conn=http.client.HTTPConnection('127.0.0.1',value['port'],timeout=2) + try: + conn.request(method,'/v1/'+path,body=_encoded(body) if body is not None else None, + headers={'X-Vault-Token':permit.owner_token,'Content-Type':'application/json'}) + response=conn.getresponse();raw=response.read(1048577) + if len(raw)>1048576: raise Refused('oversize recovery response') + data=json.loads(raw,object_pairs_hook=_json_unique,parse_constant=_invalid_constant) if raw else {} + if type(data) is not dict: raise Refused('invalid recovery response') + status=response.status + except Exception: raise Refused('recovery transport or response unconfirmed') from None + finally:conn.close() + record.after(ordinal,index,status) + return status,data + confirmed=False + try: + if item['kind'] in {'task-token-revoke','identity-token-revoke'}: + accessor=value['task_accessor' if ordinal==0 else 'identity_accessor'] + if accessor is None: + confirmed=ordinal==0 and value['phase']=='prepared' and record.no_mint_proved + else: + status,data=call('POST','auth/token/revoke-accessor',{'accessor':accessor}) + # Revoke is idempotent for an already absent accessor; + # actual absence must still be independently read back. + if status in {200,204,400}: + status,data=call('POST','auth/token/lookup-accessor',{'accessor':accessor}) + confirmed=status==400 and data.get('errors') in (['invalid accessor'],['1 error occurred:\n\t* invalid accessor\n\n']) + elif item['kind']=='grant-policy-delete': + path='sys/policies/acl/task-exec-'+permit.binding.operation_id + status,data=call('GET',path) + if _policy_absent(status,data):confirmed=True + elif (status==200 and not data.get('errors') and type(data.get('data')) is dict + and _same_policy(data['data'].get('policy'),value['policy_hcl'])): + status,data=call('DELETE',path) + if status in {200,204}: + status,data=call('GET',path);confirmed=_policy_absent(status,data) + except Exception: + # No backend errors/response/handle hashes enter public evidence. + confirmed=False + record.finish(ordinal,confirmed) + return journal.cleanup_status(permit.binding) diff --git a/tests/test_task_execution_recovery.py b/tests/test_task_execution_recovery.py new file mode 100644 index 0000000..52b8b61 --- /dev/null +++ b/tests/test_task_execution_recovery.py @@ -0,0 +1,320 @@ +"""Actual executor process loss against independent disposable OpenBao.""" +from dataclasses import replace +import hashlib +import json +import os +import secrets +from pathlib import Path +import sys +import time + +import pytest + +sys.path.insert(0,str(Path(__file__).resolve().parents[1]/'scripts')) +from test_task_execution_plan import compiler +from test_task_execution_native_fixture import bao_dev, owner, admit +from task_execution_journal import Journal, Refused +from task_execution_native_fixture import NativeFixtureAdapter +from task_execution_recovery import FixtureRecoveryCustody, recover_fixture_cleanup + + +def protected(owner,tmp_path): + native,journal,binding,call,compiler,descriptor=owner + directory=tmp_path/'separate-owner-custody';directory.mkdir(mode=0o700) + custody=FixtureRecoveryCustody(directory,integrity_key=secrets.token_bytes(32)) + native=NativeFixtureAdapter(compiler,native.plan,journal,binding,native.admission,recovery_custody=custody) + return native,journal,binding,call,custody + + +def crash(native,journal,binding,stage='issued'): + pid=os.fork() + if pid==0: + try: + # The backend belongs to the independent parent, so this child cannot + # waitpid it. Parent retains the actual Popen and checks it on recovery. + native.admission.server_process.poll=lambda:None + native.journal=Journal(journal.directory) + with native.journal.locked(): + admit(native,native.journal,binding) + if stage=='mint-response-lost': + original=native._http + def lost(method,path,token,body=b''): + result=original(method,path,token,body) + if path=='auth/token/create':os._exit(73) + return result + native._http=lost + native.materialize_grant() + native.journal.transition(binding,'grant-issued','applying') + if stage=='business-response-lost': + native.metadata_call(0) + original=native._http + def lost(method,path,token,body=b''): + result=original(method,path,token,body) + if method=='PUT':os._exit(73) + return result + native._http=lost + native.metadata_call(1) + if stage=='cleanup-response-lost': + from task_execution_cleanup import run_cleanup + original=native._http + def lost(method,path,token,body=b''): + result=original(method,path,token,body) + if path=='auth/token/revoke-self':os._exit(73) + return result + native._http=lost + run_cleanup(native.journal,binding,native) + os._exit(73) + except BaseException:os._exit(74) + _,status=os.waitpid(pid,0) + assert os.waitstatus_to_exitcode(status)==73 + # Original admission never appears in the independent parent or reopened DB. + assert native.task is None + assert journal._admission is None and journal._consumed_admission is None + + +def permit(native,custody,binding,**kwargs): + return custody.admit(binding,owner_token=kwargs.pop('owner_token',native.admission.owner_token), + server_process=native.admission.server_process,**kwargs) + + +@pytest.mark.parametrize('stage',['issued','business-response-lost','cleanup-response-lost']) +def test_process_loss_then_independent_cleanup_never_resumes_business(owner,tmp_path,stage,monkeypatch,request): + native,journal,binding,call,custody=protected(owner,tmp_path) + crash(native,journal,binding,stage) + journal=Journal(journal.directory) + request.addfinalizer(journal.close) + native.journal=journal + private=custody.read(binding) + assert private['phase']=='issued' and private['task_accessor'] + target=native.compiler.before_call(native.plan,1).path + before=call('GET',target) + with journal.locked(): + with pytest.raises(Refused):journal.reserve(binding,native.cleanup) + with pytest.raises(Refused):native.materialize_grant() + with pytest.raises(Refused):native.metadata_call(0) + with pytest.raises(Refused):native.revoke_task_token(native.cleanup[0].call_digest) + admitted=permit(native,custody,binding) + import task_execution_recovery as recovery_module + sent=[] + original_request=recovery_module.http.client.HTTPConnection.request + def tracked(connection,method,url,*args,**kwargs): + sent.append((method,url)) # metadata only; never capture body/headers + return original_request(connection,method,url,*args,**kwargs) + with monkeypatch.context() as patch: + patch.setattr(recovery_module.http.client.HTTPConnection,'request',tracked) + result=recover_fixture_cleanup(journal,admitted) + assert all((method,path) in { + ('POST','/v1/auth/token/revoke-accessor'),('POST','/v1/auth/token/lookup-accessor'), + ('GET','/v1/sys/policies/acl/'+native.policy_name), + ('DELETE','/v1/sys/policies/acl/'+native.policy_name)} for method,path in sent) + assert result['confirmed'] + for accessor in [private['task_accessor'],private['identity_accessor']]: + assert call('POST','auth/token/lookup-accessor',{'accessor':accessor})[0]==400 + assert call('GET','sys/policies/acl/'+native.policy_name)[0]==404 + after=call('GET',target) + assert after[0]==before[0] and after[1].get('data')==before[1].get('data') # request IDs vary + assert journal.observe(binding)['dispatch_allowed'] is False + if stage!='issued':assert journal.observe(binding)['external_outcome_unknown'] + if stage=='cleanup-response-lost': + original=json.loads(journal.db.execute('SELECT original FROM recovery_attempts').fetchone()[0]) + assert original[0]['outcome']=='pending' + assert journal.db.execute("SELECT COUNT(*) FROM cleanup_calls WHERE outcome='pending'").fetchone()[0]==1 + with pytest.raises(Refused):recover_fixture_cleanup(journal,admitted) + public=(journal.directory/'journal.sqlite').read_bytes() + for value in [private['task_accessor'],private['identity_accessor'],native.admission.owner_token]: + assert value.encode() not in public + assert 'client_token' not in json.dumps(private) + assert 'token' not in repr(admitted) and private['task_accessor'] not in repr(admitted) + + +def test_process_loss_before_handle_commit_preserves_unknown_mint_debt(owner,tmp_path): + native,journal,binding,call,custody=protected(owner,tmp_path) + crash(native,journal,binding,'mint-response-lost') + assert custody.read(binding)['phase']=='mint-intent' + result=recover_fixture_cleanup(journal,permit(native,custody,binding)) + assert not result['confirmed'] + assert [x['outcome'] for x in result['obligations']]==['owed','confirmed','confirmed'] + assert journal.observe(binding)['external_outcome_unknown'] + assert journal.db.execute('SELECT confirmed FROM recovery_results ORDER BY obligation').fetchall()==[(0,),(1,),(1,)] + + +def test_revoked_expired_wrong_revision_and_public_flag_refuse_before_calls(owner,tmp_path): + native,journal,binding,call,custody=protected(owner,tmp_path);crash(native,journal,binding) + p=permit(native,custody,binding) + custody.revoke(p) + for bad in [p,replace(permit(native,custody,binding),deadline=time.monotonic()-1), + replace(permit(native,custody,binding),private_revision_digest='0'*64),{'verified':True}]: + with pytest.raises(Refused):recover_fixture_cleanup(journal,bad) + assert call('GET','sys/policies/acl/'+native.policy_name)[0]==200 + assert journal.cleanup_status(binding)['obligations'][0]['outcome']=='owed' + + +def test_revoked_recovery_credential_cannot_fake_cleanup_success(owner,tmp_path): + native,journal,binding,call,custody=protected(owner,tmp_path);crash(native,journal,binding) + status,data=call('POST','auth/token/create',{'policies':['root'],'ttl':'30s'}) + assert status==200 + token=data['auth']['client_token'] + assert call('POST','auth/token/revoke',{'token':token})[0] in {200,204} + result=recover_fixture_cleanup(journal,permit(native,custody,binding,owner_token=token)) + assert not result['confirmed'] and all(x['outcome']=='owed' for x in result['obligations']) + assert call('GET','sys/policies/acl/'+native.policy_name)[0]==200 + # A separately admitted owner attempt can recover after the failed attempt; + # neither the failed attempt nor its receipt grants an automatic retry. + assert recover_fixture_cleanup(journal,permit(native,custody,binding))['confirmed'] + assert journal.db.execute('SELECT COUNT(*) FROM recovery_attempts').fetchone()[0]==2 + + +def test_policy_name_reuse_never_deletes_different_policy(owner,tmp_path): + native,journal,binding,call,custody=protected(owner,tmp_path);crash(native,journal,binding) + replacement='path "unrelated" { capabilities = ["read"] }' + assert call('PUT','sys/policies/acl/'+native.policy_name,{'policy':replacement})[0] in {200,204} + result=recover_fixture_cleanup(journal,permit(native,custody,binding)) + assert not result['confirmed'] + assert [x['outcome'] for x in result['obligations']]==['confirmed','confirmed','owed'] + assert call('GET','sys/policies/acl/'+native.policy_name)[1]['data']['policy']==replacement + + +def test_private_custody_permissions_and_identity_binding_fail_closed(owner,tmp_path): + native,journal,binding,call,custody=protected(owner,tmp_path) + path=custody._path(binding);path.chmod(0o644) + with pytest.raises(Refused):custody.read(binding) + path.chmod(0o600) + data=json.loads(path.read_bytes());data['payload']['binding']['operation_id']='other' + path.write_text(json.dumps(data)) + with pytest.raises(Refused):custody.read(binding) + + +def test_lost_recovery_response_stays_unknown_until_new_independent_attempt(owner,tmp_path,monkeypatch): + native,journal,binding,call,custody=protected(owner,tmp_path);crash(native,journal,binding) + import task_execution_recovery as recovery_module + original=recovery_module.http.client.HTTPConnection.getresponse + once=[True] + def lost(connection): + response=original(connection) + if once[0]: + once[0]=False + response.close() + raise Refused('synthetic lost recovery response after real revoke') + return response + first=permit(native,custody,binding) + with monkeypatch.context() as patch: + patch.setattr(recovery_module.http.client.HTTPConnection,'getresponse',lost) + assert not recover_fixture_cleanup(journal,first)['confirmed'] + assert journal.observe(binding)['external_outcome_unknown'] + with pytest.raises(Refused):recover_fixture_cleanup(journal,first) + assert recover_fixture_cleanup(journal,permit(native,custody,binding))['confirmed'] + # Reconciliation does not erase the historic uncertain invocation. + assert journal.observe(binding)['external_outcome_unknown'] + assert journal.db.execute("SELECT COUNT(*) FROM recovery_calls WHERE outcome='pending'").fetchone()[0]==1 + + +@pytest.fixture +def capsule(tmp_path): + from task_execution_journal import Binding,Cleanup + from task_execution_plan import cleanup_specs + binding=Binding('SYNTHETIC','sha256:'+'a'*64,'12345678-1234-1234-1234-123456789abc','b'*64) + directory=tmp_path/'private-capsule';directory.mkdir(mode=0o700) + custody=FixtureRecoveryCustody(directory,integrity_key=secrets.token_bytes(32)) + cleanup=tuple(Cleanup(x['kind'],x['call_digest']) for x in cleanup_specs(binding.operation_id)) + custody.prepare(binding,cleanup,12345,'path "fixture" {}\n','synthetic-private-accessor') + return custody,binding + + +@pytest.mark.parametrize('mutate',[ + lambda v:v.update(extra=True),lambda v:v.pop('phase'),lambda v:v.update(binding=True), + lambda v:v['binding'].update(approval_id=True),lambda v:v['binding'].update(extra='x'), + lambda v:v['binding'].update(request_digest='0'*64),lambda v:v.update(port=True), + lambda v:v.update(port=12345.0),lambda v:v.update(port=0),lambda v:v.update(phase=[]), + lambda v:v.update(phase='issued'),lambda v:v.update(revision=False), + lambda v:v.update(revision=0.0),lambda v:v.update(identity_accessor=False), + lambda v:v.update(task_accessor='unexpected-before-issue'),lambda v:v.update(policy_hcl=[]), + lambda v:v.update(cleanup={}),lambda v:v['cleanup'].pop(), + lambda v:v['cleanup'].__setitem__(0,False),lambda v:v['cleanup'][0].update(extra=True), + lambda v:v['cleanup'][0].update(kind=True),lambda v:v['cleanup'][0].update(call_digest=0), + lambda v:v['cleanup'][0].update(call_digest='f'*64),lambda v:v['cleanup'].reverse(), +]) +def test_authenticated_but_damaged_capsule_schema_refuses(capsule,mutate): + custody,binding=capsule + value=custody.read(binding);mutate(value) + # Trusted test owner re-signs malformed payload to exercise schema defenses + # independently of MAC validation; untrusted corruption cannot do this. + custody._write(binding,value) + with pytest.raises(Refused):custody.read(binding) + + +def test_capsule_corruption_wrong_key_duplicate_and_nonfinite_refuse(capsule): + custody,binding=capsule;path=custody._path(binding);original=path.read_bytes() + value=json.loads(original);value['payload']['identity_accessor']='different-synthetic-accessor' + path.write_text(json.dumps(value)) + with pytest.raises(Refused):custody.read(binding) + path.write_bytes(original) + other=FixtureRecoveryCustody(custody.directory,integrity_key=b'X'*32) + with pytest.raises(Refused):other.read(binding) + for raw in [b'{',original[:-1]+b',"mac":"duplicate"}',original.replace(b'12345',b'NaN'),b'[]',b'{}',b'{"payload":{},"mac":"'+b'\xc3\xa9'*64+b'"}']: + path.write_bytes(raw) + with pytest.raises(Refused):custody.read(binding) + path.write_bytes(original) + assert custody.read(binding)['identity_accessor']=='synthetic-private-accessor' + + +def test_exact_absence_and_single_terminal_lf_contract(): + from task_execution_recovery import _policy_absent,_same_policy + assert _policy_absent(404,{'errors':[]}) + for status,body in [(403,{'errors':[]}),(404,{}),(404,{'errors':['denied']}), + (404,{'errors':None}),(404,{'errors':[]} | {'data':{}}), + (404,{'errors':False}),(404,{'errors':0}),(404,{'errors':{}})]: + assert not _policy_absent(status,body) + assert _same_policy('fixture','fixture\n') + assert _same_policy('fixture\n','fixture\n') + for text in ['fixture \n','fixture\t','fixture\r\n','fixture\n\n',False,None]: + assert not _same_policy(text,'fixture\n') + + +@pytest.mark.parametrize('after_delete',[False,True]) +def test_recovery_policy_absence_rejects_misleading_native_http_bodies(owner,tmp_path,monkeypatch,after_delete): + native,journal,binding,call,custody=protected(owner,tmp_path);crash(native,journal,binding) + import task_execution_recovery as module + malformed=[{}, {'errors':['permission denied']}, {'errors':None}, {'errors':False}, + {'errors':0}, {'errors':[],'data':{}}, {'errors':['not really absent']}, + b'{"errors":["permission denied"],"errors":[]}',b'{"errors":NaN}',b'not JSON'] + path='sys/policies/acl/'+native.policy_name + for body in malformed: + sent=[];gets=[0] + class Response: + def __init__(self,status,data):self.status=status;self.data=data + def read(self,limit):return self.data if type(self.data) is bytes else json.dumps(self.data).encode() + class Connection: + def __init__(self,*args,**kwargs):pass + def request(self,method,url,**kwargs):sent.append((method,url)) + def getresponse(self): + method,url=sent[-1] + if url=='/v1/'+path: + if method=='DELETE':return Response(204,{}) + gets[0]+=1 + if after_delete and gets[0]==1: + return Response(200,{'data':{'policy':native.hcl[:-1]}}) + return Response(404,body) + return Response(403,{'errors':['synthetic refusal']}) + def close(self):pass + with monkeypatch.context() as patch: + patch.setattr(module.http.client,'HTTPConnection',Connection) + result=recover_fixture_cleanup(journal,permit(native,custody,binding)) + assert not result['confirmed'] and result['obligations'][2]['outcome']=='owed' + assert ('DELETE','/v1/'+path) in sent if after_delete else ('DELETE','/v1/'+path) not in sent + # The denied/malformed probes above emitted no real native calls. + assert call('GET',path)[0]==200 + assert recover_fixture_cleanup(journal,permit(native,custody,binding))['confirmed'] + + +def test_authenticated_prepared_capsule_rollback_cannot_hide_later_mint(owner,tmp_path): + native,journal,binding,call,custody=protected(owner,tmp_path) + old=custody._path(binding).read_bytes() + crash(native,journal,binding) + current=custody.read(binding) + custody._path(binding).write_bytes(old) + # Still authentic under the owner key, but the durable journal proves the + # operation reached issuance. The stale prepared record cannot prove absence. + result=recover_fixture_cleanup(journal,permit(native,custody,binding)) + assert not result['confirmed'] and result['obligations'][0]['outcome']=='owed' + assert call('POST','auth/token/lookup-accessor',{'accessor':current['task_accessor']})[0]==200 + assert call('POST','auth/token/revoke-accessor',{'accessor':current['task_accessor']})[0] in {200,204} diff --git a/workplans/RPF-WP-0035-credential-lane-implementation.md b/workplans/RPF-WP-0035-credential-lane-implementation.md index 8b09305..f58a6b9 100644 --- a/workplans/RPF-WP-0035-credential-lane-implementation.md +++ b/workplans/RPF-WP-0035-credential-lane-implementation.md @@ -1290,3 +1290,20 @@ CAS records consumed/unknown with zero native effects. Exact dependency/source hashes, tests and remaining gates are retained with the source package and in docs/task-execution-mediator-seam.md. No real grant/custody, policy default, owner sync or native production acceptance is performed; T12 stays progress. + +2026-10-16 T12 recovery successor source preparation: optional independently +installed fixture custody durably retains operation-bound private accessors and +issuance checkpoints. Fresh cleanup-only owner permits can reconcile after actual +executor process death using fixed revoke/readback and policy delete/readback, +without restoring business/consume admission. Separate durable attempt evidence +retains original unknown outcomes. Lost token-create response remains unresolved +when no accessor was committed. Source-only process-loss fixtures are not native +production custody or recovery-role admission; T12 remains progress. + +2026-10-16 T12 recovery R2 source refinement retains R1's overbroad 404 failure: +policy cleanup now requires exact native empty-errors absence before/after delete; +HCL uses only documented single optional terminal LF normalization. Private +capsules are owner-key HMAC authenticated, strict nested schemas/ordered cleanup +commitments reject damage, and journal history prevents stale prepared capsules +from hiding issuance. Fixture process-loss source proof remains separate from +unadmitted production custody, integrity-key storage and recovery-role scope.