641 lines
37 KiB
Diff
641 lines
37 KiB
Diff
diff --git a/docs/task-execution-native-fixture.md b/docs/task-execution-native-fixture.md
|
|
index 1dd6b91..4fb60fe 100644
|
|
--- a/docs/task-execution-native-fixture.md
|
|
+++ b/docs/task-execution-native-fixture.md
|
|
@@ -121,3 +121,56 @@ retry cleanup. Native effects can survive process loss. Cleanup handles remain
|
|
memory-only: no independently authorized recovery mechanism is implemented here.
|
|
This closes disposable sequential apply acceptance only, not TSA03 or native
|
|
production recovery/admission.
|
|
+
|
|
+
|
|
+## Independent cleanup recovery source successor
|
|
+
|
|
+`task_execution_recovery.py` adds separately installed fixture-only owner custody
|
|
+and fresh recovery permits. Optional `recovery_custody` on NativeFixtureAdapter
|
|
+persists the original binding/cleanup commitments, exact operation policy, backend
|
|
+port and identity accessor before any grant effect; it commits mint intent before
|
|
+token creation and task accessor immediately after a valid issued response.
|
|
+Bearer credentials are never persisted. The custody directory is separate from
|
|
+the public journal, owner-only 0700/0600, bounded at 64KiB, no-follow, with atomic
|
|
+replacement and fsync. It is real process-independent persistence for synthetic
|
|
+fixtures. It does not isolate same-UID hostile execution, encrypt at rest, select
|
|
+production storage or establish a production secret policy. Those are explicit
|
|
+native custody/deployment gates, not supplied by Python types or file modes.
|
|
+
|
|
+The independent owner reads its private custody revision and installs a fresh
|
|
+bounded permit, current owner credential and retained live disposable Popen.
|
|
+Permit revocation is durably owner-recorded. The recovery runner has no mint,
|
|
+consume, policy-apply or business method. It dispatches only fixed exact-accessor
|
|
+revoke/readback and exact `task-exec-<operation>` policy read/delete/readback.
|
|
+A differently recreated policy is never deleted. Backend admission remains the
|
|
+same loopback-only owner-spawned dev process; no estate backend can be selected.
|
|
+Accessor-based recovery is an independently admitted alternate cleanup projection,
|
|
+not a reinterpretation of the original revoke-self call digest. Production needs
|
|
+exact owner-reviewed recovery scope; original business grant does not imply it.
|
|
+
|
|
+Fresh recovery holds the journal process lock but never sets business admission
|
|
+or consumed-admission fields. Separate recovery attempt/call/result tables retain
|
|
+original cleanup statuses before reconciliation, write native intent before calls,
|
|
+and preserve failed/pending outcomes. Proven cleanup alone updates the exact
|
|
+original cleanup obligation to confirmed. Business state and original pending
|
|
+call uncertainty remain unchanged. Reusing the original approval, continuing
|
|
+metadata calls and reusing a recovery permit refuse. Failure needs another
|
|
+independent owner permit; there is no automatic retry. Expired/revoked permits
|
|
+or changed private custody revision refuse before calls.
|
|
+
|
|
+A token-create response lost before accessor custody remains unresolved token
|
|
+cleanup debt: policy deletion/identity revocation can be independently confirmed,
|
|
+but cannot imply task-token revocation. Neither expiry guessed from wall clock
|
|
+nor policy absence fabricates token absence. A production successor must admit
|
|
+an independent exact ambiguous-issuance reconciliation mechanism or prove bounded
|
|
+expiry from owner backend evidence. No such new grant is implemented here.
|
|
+
|
|
+`test_task_execution_recovery.py` forks a real executor and terminates it without
|
|
+finally cleanup after issue, business effect/lost response and revoke/lost response.
|
|
+The independent parent, holding the backend and separately installed custody,
|
|
+recovers specific obligations and verifies actual token/policy absence. Additional
|
|
+cases cover lost-mint response, revoked credential, permit revocation/expiry,
|
|
+revision drift, namespace policy-name reuse and private-file/binding refusals.
|
|
+This is source fixture proof only. Native custody, isolation, exact recovery-role
|
|
+permissions, retention/encryption, clock admission, audit delivery and human scope
|
|
+approval remain missing. No generic broker, production reader, grant or CLI is added.
|
|
diff --git a/scripts/task_execution_journal.py b/scripts/task_execution_journal.py
|
|
index 8f1b9a2..1eaa583 100644
|
|
--- a/scripts/task_execution_journal.py
|
|
+++ b/scripts/task_execution_journal.py
|
|
@@ -104,6 +104,9 @@ class Journal:
|
|
self.db.execute("CREATE TABLE IF NOT EXISTS calls (operation TEXT NOT NULL, ordinal INTEGER NOT NULL, call_digest TEXT NOT NULL, outcome TEXT NOT NULL, evidence TEXT, PRIMARY KEY(operation,ordinal))")
|
|
self.db.execute("CREATE TABLE IF NOT EXISTS cleanup (operation TEXT NOT NULL, ordinal INTEGER NOT NULL, kind TEXT NOT NULL, call_digest TEXT NOT NULL, outcome TEXT NOT NULL, evidence TEXT, PRIMARY KEY(operation,ordinal), UNIQUE(operation,kind))")
|
|
self.db.execute("CREATE TABLE IF NOT EXISTS cleanup_calls (operation TEXT NOT NULL, obligation INTEGER NOT NULL, step INTEGER NOT NULL, call_digest TEXT NOT NULL, outcome TEXT NOT NULL, evidence TEXT, PRIMARY KEY(operation,obligation,step))")
|
|
+ self.db.execute("CREATE TABLE IF NOT EXISTS recovery_attempts (attempt TEXT PRIMARY KEY, operation TEXT NOT NULL, original TEXT NOT NULL)")
|
|
+ self.db.execute("CREATE TABLE IF NOT EXISTS recovery_calls (attempt TEXT NOT NULL, obligation INTEGER NOT NULL, step INTEGER NOT NULL, digest TEXT NOT NULL, outcome TEXT NOT NULL, evidence TEXT, PRIMARY KEY(attempt,obligation,step))")
|
|
+ self.db.execute("CREATE TABLE IF NOT EXISTS recovery_results (attempt TEXT NOT NULL, obligation INTEGER NOT NULL, confirmed INTEGER NOT NULL, evidence TEXT NOT NULL, PRIMARY KEY(attempt,obligation))")
|
|
self._held = False
|
|
self._admission = None
|
|
self._consumed_admission = None
|
|
@@ -228,6 +231,7 @@ class Journal:
|
|
pending = self.db.execute("SELECT COUNT(*) FROM calls WHERE operation=? AND outcome='pending'", (binding.operation_id,)).fetchone()[0]
|
|
pending += self.db.execute("SELECT COUNT(*) FROM cleanup WHERE operation=? AND outcome='pending'", (binding.operation_id,)).fetchone()[0]
|
|
pending += self.db.execute("SELECT COUNT(*) FROM cleanup_calls WHERE operation=? AND outcome='pending'", (binding.operation_id,)).fetchone()[0]
|
|
+ pending += self.db.execute("SELECT COUNT(*) FROM recovery_calls AS c JOIN recovery_attempts AS a ON c.attempt=a.attempt WHERE a.operation=? AND c.outcome='pending'", (binding.operation_id,)).fetchone()[0]
|
|
return {"state": row[3], "evidence_digest": row[4], "dispatch_allowed": False,
|
|
"external_outcome_unknown": bool(pending)}
|
|
|
|
diff --git a/scripts/task_execution_native_fixture.py b/scripts/task_execution_native_fixture.py
|
|
index caa7da1..31a822b 100644
|
|
--- a/scripts/task_execution_native_fixture.py
|
|
+++ b/scripts/task_execution_native_fixture.py
|
|
@@ -3,7 +3,8 @@
|
|
This adapter deliberately admits only a disposable loopback fixture. It supplies
|
|
real native primitive coverage while production grant schema, identity/consume and
|
|
origin/source admission are unfinished. It cannot open an estate backend. Owner
|
|
-credentials/handles stay in memory; no subprocess, redirects or ambient proxy.
|
|
+bearer credentials stay in memory; optional owner-private fixture custody persists
|
|
+accessors for independently admitted cleanup recovery; no redirects or ambient proxy.
|
|
Python privacy is not credential isolation; production needs the owner runtime.
|
|
"""
|
|
from __future__ import annotations
|
|
@@ -41,7 +42,7 @@ class FixtureAdmission:
|
|
|
|
class NativeFixtureAdapter:
|
|
def __init__(self, compiler: Compiler, plan: Plan, journal: Journal,
|
|
- binding: Binding, admission: FixtureAdmission):
|
|
+ binding: Binding, admission: FixtureAdmission, *, recovery_custody=None):
|
|
if type(admission) is not FixtureAdmission or type(admission.descriptor) is not bytes:
|
|
raise Refused('independent fixture admission required')
|
|
self.compiler,self.plan,self.journal,self.binding=compiler,plan,journal,binding
|
|
@@ -100,6 +101,12 @@ class NativeFixtureAdapter:
|
|
if self.meta['cleanup'] != cleanup_specs(binding.operation_id): raise Refused('cleanup plan mismatch')
|
|
self.task=None; self._mint_attempted=False; self._policy_attempted=False; self._ordinal=0; self._next_metadata=0
|
|
self._cleanup_steps={item.kind:0 for item in self.cleanup}
|
|
+ self.recovery_custody=recovery_custody
|
|
+ if recovery_custody is not None:
|
|
+ from task_execution_recovery import FixtureRecoveryCustody
|
|
+ if type(recovery_custody) is not FixtureRecoveryCustody or journal.observe(binding) is not None:
|
|
+ raise Refused('fresh independent fixture recovery custody required')
|
|
+ recovery_custody.prepare(binding,self.cleanup,self.port,self.hcl,admission.identity_session.accessor)
|
|
|
|
def _live(self):
|
|
if not self.journal._held or self.journal._admission!=self.binding or self.journal._consumed_admission!=self.binding:
|
|
@@ -177,6 +184,8 @@ class NativeFixtureAdapter:
|
|
'type':'service','display_name':'task-exec-'+self.binding.operation_id,'no_parent':False}
|
|
if body!=self.meta['execution_grant']['openbao']['token_create_body']:
|
|
raise Refused('native token body differs from bound owner projection')
|
|
+ if self.recovery_custody is not None:
|
|
+ self.recovery_custody.checkpoint(self.binding,'mint-intent')
|
|
self._mint_attempted=True
|
|
status,data=self._business('POST','auth/token/create',root,encoded(body));self._ok(status,data,{200})
|
|
auth=data.get('auth',{})
|
|
@@ -185,6 +194,8 @@ class NativeFixtureAdapter:
|
|
# Retain handles before validation so finally cleanup can revoke even a
|
|
# malformed overbroad issue response. Never return token-bearing metadata.
|
|
self.task=PrivateSession(auth['client_token'],auth['accessor'])
|
|
+ if self.recovery_custody is not None:
|
|
+ self.recovery_custody.checkpoint(self.binding,'issued',self.task.accessor)
|
|
status,data=self._business('POST','auth/token/lookup',root,encoded({'token':self.task.token}));self._ok(status,data,{200})
|
|
info=data.get('data',{})
|
|
if (info.get('policies')!=[self.policy_name] or info.get('identity_policies',[]) or info.get('renewable') is not False
|
|
diff --git a/scripts/task_execution_recovery.py b/scripts/task_execution_recovery.py
|
|
new file mode 100644
|
|
index 0000000..b4ce2b9
|
|
--- /dev/null
|
|
+++ b/scripts/task_execution_recovery.py
|
|
@@ -0,0 +1,268 @@
|
|
+"""Owner-only cleanup recovery for the disposable native fixture.
|
|
+
|
|
+Private, separately installed custody persists accessors, never bearer tokens.
|
|
+This is a fixture implementation, not production custody/isolation admission.
|
|
+The recovery owner supplies a fresh capability and credential independently of
|
|
+public request/journal data. No business adapter or resumed admission is built.
|
|
+"""
|
|
+from __future__ import annotations
|
|
+from dataclasses import asdict, dataclass, field
|
|
+import hashlib
|
|
+import http.client
|
|
+import json
|
|
+import os
|
|
+from pathlib import Path
|
|
+import secrets
|
|
+import subprocess
|
|
+import time
|
|
+import uuid
|
|
+
|
|
+from task_execution_journal import Binding, Journal, Refused
|
|
+
|
|
+
|
|
+def _encoded(value):
|
|
+ return json.dumps(value, sort_keys=True, separators=(',', ':')).encode()
|
|
+
|
|
+
|
|
+def _digest(value):
|
|
+ return hashlib.sha256(_encoded(value)).hexdigest()
|
|
+
|
|
+
|
|
+class FixtureRecoveryCustody:
|
|
+ """Independent owner-private persistent fixture custody; no public API.
|
|
+
|
|
+ Mode/ownership and no-follow checks protect against accidental exposure.
|
|
+ Same-UID hostile code is NOT isolated. Production must supply independently
|
|
+ enforced owner storage/executor authority; this implementation refuses estate
|
|
+ backends and is never evidence of a production grant.
|
|
+ """
|
|
+ def __init__(self, directory: Path):
|
|
+ self.directory = directory
|
|
+ if directory.is_symlink() or directory.absolute() != directory.resolve():
|
|
+ raise Refused('indirect recovery custody directory')
|
|
+ Journal._private(directory.stat(), directory=True)
|
|
+
|
|
+ def _path(self, binding):
|
|
+ binding.validate()
|
|
+ return self.directory / (binding.operation_id + '.private.json')
|
|
+
|
|
+ def _sync(self):
|
|
+ fd = os.open(self.directory, os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW)
|
|
+ try: os.fsync(fd)
|
|
+ finally: os.close(fd)
|
|
+
|
|
+ def read(self, binding):
|
|
+ fd = os.open(self._path(binding), os.O_RDONLY | os.O_NOFOLLOW)
|
|
+ try:
|
|
+ Journal._private(os.fstat(fd))
|
|
+ with os.fdopen(fd, 'rb', closefd=False) as stream: raw = stream.read(65537)
|
|
+ finally: os.close(fd)
|
|
+ if len(raw) > 65536: raise Refused('oversize private custody record')
|
|
+ try: value = json.loads(raw)
|
|
+ except Exception: raise Refused('invalid private custody record') from None
|
|
+ if type(value) is not dict or value.get('binding') != asdict(binding):
|
|
+ raise Refused('private custody binding mismatch')
|
|
+ expected={'binding','cleanup','port','policy_hcl','identity_accessor','task_accessor','phase','revision'}
|
|
+ if (set(value)!=expected or type(value['port']) is not int or not 1<=value['port']<=65535
|
|
+ or type(value['policy_hcl']) is not str or not value['policy_hcl']
|
|
+ or type(value['identity_accessor']) is not str or not value['identity_accessor']
|
|
+ or value['phase'] not in {'prepared','mint-intent','issued'}
|
|
+ or type(value['revision']) is not int or value['revision']!={'prepared':0,'mint-intent':1,'issued':2}[value['phase']]
|
|
+ or (value['phase']=='issued' and (type(value['task_accessor']) is not str or not value['task_accessor']))
|
|
+ or (value['phase']!='issued' and value['task_accessor'] is not None)
|
|
+ or type(value['cleanup']) is not list or len(value['cleanup'])!=3):
|
|
+ raise Refused('invalid private custody schema')
|
|
+ return value
|
|
+
|
|
+ def _write(self, binding, value, initial=False):
|
|
+ raw = _encoded(value)
|
|
+ if len(raw) > 65536: raise Refused('oversize private custody record')
|
|
+ path = self._path(binding)
|
|
+ if initial:
|
|
+ fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW, 0o600)
|
|
+ else:
|
|
+ # Existing owner-private record checked before replacement. Unique
|
|
+ # temp + fsync + replace + directory fsync never truncates old state.
|
|
+ self.read(binding)
|
|
+ tmp = self.directory / ('.pending-' + secrets.token_hex(16))
|
|
+ fd = os.open(tmp, os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW, 0o600)
|
|
+ try:
|
|
+ with os.fdopen(fd, 'wb', closefd=False) as stream:
|
|
+ stream.write(raw); stream.flush(); os.fsync(fd)
|
|
+ finally: os.close(fd)
|
|
+ if not initial: os.replace(tmp, path)
|
|
+ self._sync()
|
|
+
|
|
+ def prepare(self, binding, cleanup, port, policy_hcl, identity_accessor):
|
|
+ value = {'binding': asdict(binding), 'cleanup': [asdict(x) for x in cleanup],
|
|
+ 'port': port, 'policy_hcl': policy_hcl,
|
|
+ 'identity_accessor': identity_accessor, 'task_accessor': None,
|
|
+ 'phase': 'prepared', 'revision': 0}
|
|
+ self._write(binding, value, initial=True)
|
|
+
|
|
+ def checkpoint(self, binding, phase, task_accessor=None):
|
|
+ value = self.read(binding)
|
|
+ if (value['phase'], phase) not in {('prepared', 'mint-intent'), ('mint-intent', 'issued')}:
|
|
+ raise Refused('invalid protected recovery checkpoint')
|
|
+ if phase == 'issued' and (type(task_accessor) is not str or not task_accessor):
|
|
+ raise Refused('protected issued accessor required')
|
|
+ value.update(phase=phase, task_accessor=task_accessor, revision=value['revision'] + 1)
|
|
+ self._write(binding, value)
|
|
+
|
|
+ def revoke(self, permit):
|
|
+ if type(permit) is not FixtureRecoveryPermit or permit.custody is not self:
|
|
+ raise Refused('exact owner recovery permit required')
|
|
+ uuid.UUID(permit.attempt_id)
|
|
+ path=self.directory / (permit.attempt_id+'.revoked')
|
|
+ fd=os.open(path,os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW,0o600)
|
|
+ try:os.fsync(fd)
|
|
+ finally:os.close(fd)
|
|
+ self._sync()
|
|
+
|
|
+ def admit(self, binding, *, owner_token, server_process, valid_seconds=30):
|
|
+ """Trusted recovery-owner action, not deserialization of caller flags.
|
|
+
|
|
+ Fresh admission seals the current private custody revision. The owner
|
|
+ supplies its credential and retained backend process separately. Neither
|
|
+ token nor private capsule digest is emitted to the public journal.
|
|
+ """
|
|
+ value = self.read(binding)
|
|
+ if type(valid_seconds) is not int or not 1 <= valid_seconds <= 60:
|
|
+ raise Refused('bounded recovery admission required')
|
|
+ if type(owner_token) is not str or not owner_token:
|
|
+ raise Refused('independent recovery owner credential required')
|
|
+ return FixtureRecoveryPermit(str(uuid.uuid4()), binding, self, _digest(value),
|
|
+ owner_token, server_process, time.monotonic()+valid_seconds)
|
|
+
|
|
+
|
|
+@dataclass(frozen=True, repr=False)
|
|
+class FixtureRecoveryPermit:
|
|
+ attempt_id: str
|
|
+ binding: Binding
|
|
+ custody: FixtureRecoveryCustody
|
|
+ private_revision_digest: str = field(repr=False)
|
|
+ owner_token: str = field(repr=False)
|
|
+ server_process: subprocess.Popen = field(repr=False)
|
|
+ deadline: float
|
|
+
|
|
+
|
|
+class _Recorder:
|
|
+ def __init__(self, journal, permit, value):
|
|
+ self.journal, self.permit = journal, permit
|
|
+ b = permit.binding
|
|
+ if journal._admission is not None or journal._consumed_admission is not None:
|
|
+ raise Refused('recovery cannot share a business admission')
|
|
+ if journal.observe(b) is None: raise Refused('original operation absent')
|
|
+ current = journal.cleanup_status(b)['obligations']
|
|
+ if [(x['kind'], x['call_digest']) for x in current] != [(x['kind'], x['call_digest']) for x in value['cleanup']]:
|
|
+ raise Refused('original cleanup commitment mismatch')
|
|
+ if all(x['outcome'] == 'confirmed' for x in current):
|
|
+ raise Refused('cleanup already confirmed')
|
|
+ self.current = current
|
|
+ db = journal.db
|
|
+ try: db.execute('INSERT INTO recovery_attempts VALUES (?,?,?)', (permit.attempt_id,b.operation_id,json.dumps(current,sort_keys=True)))
|
|
+ except Exception: raise Refused('recovery admission already used') from None
|
|
+ journal._sync_directory()
|
|
+
|
|
+ def before(self, ordinal, step, method, path):
|
|
+ digest = _digest({'kind': self.current[ordinal]['kind'], 'commitment':self.current[ordinal]['call_digest'], 'method':method,'path':path,'step':step})
|
|
+ self.journal.db.execute("INSERT INTO recovery_calls VALUES (?,?,?,?,'pending',NULL)", (self.permit.attempt_id,ordinal,step,digest))
|
|
+ self.journal._sync_directory()
|
|
+ return digest
|
|
+
|
|
+ def after(self, ordinal, step, status):
|
|
+ self.journal.db.execute("UPDATE recovery_calls SET outcome='complete', evidence=? WHERE attempt=? AND obligation=? AND step=? AND outcome='pending'", (_digest({'http_status':status}),self.permit.attempt_id,ordinal,step))
|
|
+ self.journal._sync_directory()
|
|
+
|
|
+ def finish(self, ordinal, confirmed):
|
|
+ evidence = _digest({'attempt':self.permit.attempt_id,'obligation':ordinal,'confirmed':confirmed})
|
|
+ db = self.journal.db
|
|
+ try:
|
|
+ db.execute('BEGIN IMMEDIATE')
|
|
+ pending = db.execute("SELECT COUNT(*) FROM recovery_calls WHERE attempt=? AND obligation=? AND outcome='pending'",(self.permit.attempt_id,ordinal)).fetchone()[0]
|
|
+ if confirmed and pending: raise Refused('pending recovery cannot confirm')
|
|
+ db.execute('INSERT INTO recovery_results VALUES (?,?,?,?)',(self.permit.attempt_id,ordinal,int(confirmed),evidence))
|
|
+ if confirmed:
|
|
+ item=self.current[ordinal]
|
|
+ db.execute("UPDATE cleanup SET outcome='confirmed', evidence=? WHERE operation=? AND ordinal=? AND kind=? AND call_digest=?",(evidence,self.permit.binding.operation_id,ordinal,item['kind'],item['call_digest']))
|
|
+ db.execute('COMMIT');self.journal._sync_directory()
|
|
+ except BaseException:
|
|
+ if db.in_transaction: db.execute('ROLLBACK')
|
|
+ raise
|
|
+
|
|
+
|
|
+def recover_fixture_cleanup(journal: Journal, permit: FixtureRecoveryPermit):
|
|
+ """Fresh independent owner admission: fixed cleanup only, no business retry.
|
|
+
|
|
+ Each attempt preserves original pending/unconfirmed rows in recovery_attempts
|
|
+ and native pending call rows unchanged. A fresh owner permit is required for
|
|
+ another attempt after failure; never autonomously retry uncertain mutations.
|
|
+ """
|
|
+ if type(permit) is not FixtureRecoveryPermit or type(permit.custody) is not FixtureRecoveryCustody:
|
|
+ raise Refused('independent typed recovery admission required')
|
|
+ if str(uuid.UUID(permit.attempt_id)) != permit.attempt_id:
|
|
+ raise Refused('invalid recovery attempt identity')
|
|
+ value = permit.custody.read(permit.binding)
|
|
+ if _digest(value) != permit.private_revision_digest:
|
|
+ raise Refused('private custody revision changed')
|
|
+ proc=permit.server_process
|
|
+ if (type(proc) is not subprocess.Popen or proc.poll() is not None or type(proc.args) is not list
|
|
+ or '-dev' not in proc.args or '-dev-no-store-token' not in proc.args
|
|
+ or '-dev-listen-address=127.0.0.1:'+str(value['port']) not in proc.args):
|
|
+ raise Refused('same live disposable backend required')
|
|
+ def fresh():
|
|
+ if (permit.custody.directory / (permit.attempt_id+'.revoked')).exists():
|
|
+ raise Refused('recovery permit revoked')
|
|
+ if time.monotonic() >= permit.deadline: raise Refused('recovery admission expired')
|
|
+ if proc.poll() is not None: raise Refused('disposable backend lost')
|
|
+ if _digest(permit.custody.read(permit.binding)) != permit.private_revision_digest:
|
|
+ raise Refused('private custody changed during recovery')
|
|
+ fresh()
|
|
+ with journal.locked():
|
|
+ record = _Recorder(journal,permit,value)
|
|
+ for ordinal,item in enumerate(record.current):
|
|
+ if item['outcome']=='confirmed': continue
|
|
+ step=0
|
|
+ def call(method,path,body=None):
|
|
+ nonlocal step
|
|
+ fresh(); index=step; step+=1
|
|
+ record.before(ordinal,index,method,path)
|
|
+ conn=http.client.HTTPConnection('127.0.0.1',value['port'],timeout=2)
|
|
+ try:
|
|
+ conn.request(method,'/v1/'+path,body=_encoded(body) if body is not None else None,
|
|
+ headers={'X-Vault-Token':permit.owner_token,'Content-Type':'application/json'})
|
|
+ response=conn.getresponse();raw=response.read(1048577)
|
|
+ if len(raw)>1048576: raise Refused('oversize recovery response')
|
|
+ data=json.loads(raw) if raw else {}
|
|
+ if type(data) is not dict: raise Refused('invalid recovery response')
|
|
+ status=response.status
|
|
+ except Exception: raise Refused('recovery transport or response unconfirmed') from None
|
|
+ finally:conn.close()
|
|
+ record.after(ordinal,index,status)
|
|
+ return status,data
|
|
+ confirmed=False
|
|
+ try:
|
|
+ if item['kind'] in {'task-token-revoke','identity-token-revoke'}:
|
|
+ accessor=value['task_accessor' if ordinal==0 else 'identity_accessor']
|
|
+ if accessor is None:
|
|
+ confirmed=ordinal==0 and value['phase']=='prepared'
|
|
+ else:
|
|
+ status,data=call('POST','auth/token/revoke-accessor',{'accessor':accessor})
|
|
+ # Revoke is idempotent for an already absent accessor;
|
|
+ # actual absence must still be independently read back.
|
|
+ if status in {200,204,400}:
|
|
+ status,data=call('POST','auth/token/lookup-accessor',{'accessor':accessor})
|
|
+ confirmed=status==400 and data.get('errors') in (['invalid accessor'],['1 error occurred:\n\t* invalid accessor\n\n'])
|
|
+ elif item['kind']=='grant-policy-delete':
|
|
+ path='sys/policies/acl/task-exec-'+permit.binding.operation_id
|
|
+ status,data=call('GET',path)
|
|
+ if status==404:confirmed=True
|
|
+ elif status==200 and data.get('data',{}).get('policy','').rstrip()==value['policy_hcl'].rstrip():
|
|
+ status,data=call('DELETE',path)
|
|
+ if status in {200,204}:
|
|
+ status,data=call('GET',path);confirmed=status==404
|
|
+ except Exception:
|
|
+ # No backend errors/response/handle hashes enter public evidence.
|
|
+ confirmed=False
|
|
+ record.finish(ordinal,confirmed)
|
|
+ return journal.cleanup_status(permit.binding)
|
|
diff --git a/tests/test_task_execution_recovery.py b/tests/test_task_execution_recovery.py
|
|
new file mode 100644
|
|
index 0000000..e29d716
|
|
--- /dev/null
|
|
+++ b/tests/test_task_execution_recovery.py
|
|
@@ -0,0 +1,207 @@
|
|
+"""Actual executor process loss against independent disposable OpenBao."""
|
|
+from dataclasses import replace
|
|
+import hashlib
|
|
+import json
|
|
+import os
|
|
+from pathlib import Path
|
|
+import sys
|
|
+import time
|
|
+
|
|
+import pytest
|
|
+
|
|
+sys.path.insert(0,str(Path(__file__).resolve().parents[1]/'scripts'))
|
|
+from test_task_execution_plan import compiler
|
|
+from test_task_execution_native_fixture import bao_dev, owner, admit
|
|
+from task_execution_journal import Journal, Refused
|
|
+from task_execution_native_fixture import NativeFixtureAdapter
|
|
+from task_execution_recovery import FixtureRecoveryCustody, recover_fixture_cleanup
|
|
+
|
|
+
|
|
+def protected(owner,tmp_path):
|
|
+ native,journal,binding,call,compiler,descriptor=owner
|
|
+ directory=tmp_path/'separate-owner-custody';directory.mkdir(mode=0o700)
|
|
+ custody=FixtureRecoveryCustody(directory)
|
|
+ native=NativeFixtureAdapter(compiler,native.plan,journal,binding,native.admission,recovery_custody=custody)
|
|
+ return native,journal,binding,call,custody
|
|
+
|
|
+
|
|
+def crash(native,journal,binding,stage='issued'):
|
|
+ pid=os.fork()
|
|
+ if pid==0:
|
|
+ try:
|
|
+ # The backend belongs to the independent parent, so this child cannot
|
|
+ # waitpid it. Parent retains the actual Popen and checks it on recovery.
|
|
+ native.admission.server_process.poll=lambda:None
|
|
+ native.journal=Journal(journal.directory)
|
|
+ with native.journal.locked():
|
|
+ admit(native,native.journal,binding)
|
|
+ if stage=='mint-response-lost':
|
|
+ original=native._http
|
|
+ def lost(method,path,token,body=b''):
|
|
+ result=original(method,path,token,body)
|
|
+ if path=='auth/token/create':os._exit(73)
|
|
+ return result
|
|
+ native._http=lost
|
|
+ native.materialize_grant()
|
|
+ native.journal.transition(binding,'grant-issued','applying')
|
|
+ if stage=='business-response-lost':
|
|
+ native.metadata_call(0)
|
|
+ original=native._http
|
|
+ def lost(method,path,token,body=b''):
|
|
+ result=original(method,path,token,body)
|
|
+ if method=='PUT':os._exit(73)
|
|
+ return result
|
|
+ native._http=lost
|
|
+ native.metadata_call(1)
|
|
+ if stage=='cleanup-response-lost':
|
|
+ from task_execution_cleanup import run_cleanup
|
|
+ original=native._http
|
|
+ def lost(method,path,token,body=b''):
|
|
+ result=original(method,path,token,body)
|
|
+ if path=='auth/token/revoke-self':os._exit(73)
|
|
+ return result
|
|
+ native._http=lost
|
|
+ run_cleanup(native.journal,binding,native)
|
|
+ os._exit(73)
|
|
+ except BaseException:os._exit(74)
|
|
+ _,status=os.waitpid(pid,0)
|
|
+ assert os.waitstatus_to_exitcode(status)==73
|
|
+ # Original admission never appears in the independent parent or reopened DB.
|
|
+ assert native.task is None
|
|
+ assert journal._admission is None and journal._consumed_admission is None
|
|
+
|
|
+
|
|
+def permit(native,custody,binding,**kwargs):
|
|
+ return custody.admit(binding,owner_token=kwargs.pop('owner_token',native.admission.owner_token),
|
|
+ server_process=native.admission.server_process,**kwargs)
|
|
+
|
|
+
|
|
+@pytest.mark.parametrize('stage',['issued','business-response-lost','cleanup-response-lost'])
|
|
+def test_process_loss_then_independent_cleanup_never_resumes_business(owner,tmp_path,stage,monkeypatch,request):
|
|
+ native,journal,binding,call,custody=protected(owner,tmp_path)
|
|
+ crash(native,journal,binding,stage)
|
|
+ journal=Journal(journal.directory)
|
|
+ request.addfinalizer(journal.close)
|
|
+ native.journal=journal
|
|
+ private=custody.read(binding)
|
|
+ assert private['phase']=='issued' and private['task_accessor']
|
|
+ target=native.compiler.before_call(native.plan,1).path
|
|
+ before=call('GET',target)
|
|
+ with journal.locked():
|
|
+ with pytest.raises(Refused):journal.reserve(binding,native.cleanup)
|
|
+ with pytest.raises(Refused):native.materialize_grant()
|
|
+ with pytest.raises(Refused):native.metadata_call(0)
|
|
+ with pytest.raises(Refused):native.revoke_task_token(native.cleanup[0].call_digest)
|
|
+ admitted=permit(native,custody,binding)
|
|
+ import task_execution_recovery as recovery_module
|
|
+ sent=[]
|
|
+ original_request=recovery_module.http.client.HTTPConnection.request
|
|
+ def tracked(connection,method,url,*args,**kwargs):
|
|
+ sent.append((method,url)) # metadata only; never capture body/headers
|
|
+ return original_request(connection,method,url,*args,**kwargs)
|
|
+ with monkeypatch.context() as patch:
|
|
+ patch.setattr(recovery_module.http.client.HTTPConnection,'request',tracked)
|
|
+ result=recover_fixture_cleanup(journal,admitted)
|
|
+ assert all((method,path) in {
|
|
+ ('POST','/v1/auth/token/revoke-accessor'),('POST','/v1/auth/token/lookup-accessor'),
|
|
+ ('GET','/v1/sys/policies/acl/'+native.policy_name),
|
|
+ ('DELETE','/v1/sys/policies/acl/'+native.policy_name)} for method,path in sent)
|
|
+ assert result['confirmed']
|
|
+ for accessor in [private['task_accessor'],private['identity_accessor']]:
|
|
+ assert call('POST','auth/token/lookup-accessor',{'accessor':accessor})[0]==400
|
|
+ assert call('GET','sys/policies/acl/'+native.policy_name)[0]==404
|
|
+ after=call('GET',target)
|
|
+ assert after[0]==before[0] and after[1].get('data')==before[1].get('data') # request IDs vary
|
|
+ assert journal.observe(binding)['dispatch_allowed'] is False
|
|
+ if stage!='issued':assert journal.observe(binding)['external_outcome_unknown']
|
|
+ if stage=='cleanup-response-lost':
|
|
+ original=json.loads(journal.db.execute('SELECT original FROM recovery_attempts').fetchone()[0])
|
|
+ assert original[0]['outcome']=='pending'
|
|
+ assert journal.db.execute("SELECT COUNT(*) FROM cleanup_calls WHERE outcome='pending'").fetchone()[0]==1
|
|
+ with pytest.raises(Refused):recover_fixture_cleanup(journal,admitted)
|
|
+ public=(journal.directory/'journal.sqlite').read_bytes()
|
|
+ for value in [private['task_accessor'],private['identity_accessor'],native.admission.owner_token]:
|
|
+ assert value.encode() not in public
|
|
+ assert 'client_token' not in json.dumps(private)
|
|
+ assert 'token' not in repr(admitted) and private['task_accessor'] not in repr(admitted)
|
|
+
|
|
+
|
|
+def test_process_loss_before_handle_commit_preserves_unknown_mint_debt(owner,tmp_path):
|
|
+ native,journal,binding,call,custody=protected(owner,tmp_path)
|
|
+ crash(native,journal,binding,'mint-response-lost')
|
|
+ assert custody.read(binding)['phase']=='mint-intent'
|
|
+ result=recover_fixture_cleanup(journal,permit(native,custody,binding))
|
|
+ assert not result['confirmed']
|
|
+ assert [x['outcome'] for x in result['obligations']]==['owed','confirmed','confirmed']
|
|
+ assert journal.observe(binding)['external_outcome_unknown']
|
|
+ assert journal.db.execute('SELECT confirmed FROM recovery_results ORDER BY obligation').fetchall()==[(0,),(1,),(1,)]
|
|
+
|
|
+
|
|
+def test_revoked_expired_wrong_revision_and_public_flag_refuse_before_calls(owner,tmp_path):
|
|
+ native,journal,binding,call,custody=protected(owner,tmp_path);crash(native,journal,binding)
|
|
+ p=permit(native,custody,binding)
|
|
+ custody.revoke(p)
|
|
+ for bad in [p,replace(permit(native,custody,binding),deadline=time.monotonic()-1),
|
|
+ replace(permit(native,custody,binding),private_revision_digest='0'*64),{'verified':True}]:
|
|
+ with pytest.raises(Refused):recover_fixture_cleanup(journal,bad)
|
|
+ assert call('GET','sys/policies/acl/'+native.policy_name)[0]==200
|
|
+ assert journal.cleanup_status(binding)['obligations'][0]['outcome']=='owed'
|
|
+
|
|
+
|
|
+def test_revoked_recovery_credential_cannot_fake_cleanup_success(owner,tmp_path):
|
|
+ native,journal,binding,call,custody=protected(owner,tmp_path);crash(native,journal,binding)
|
|
+ status,data=call('POST','auth/token/create',{'policies':['root'],'ttl':'30s'})
|
|
+ assert status==200
|
|
+ token=data['auth']['client_token']
|
|
+ assert call('POST','auth/token/revoke',{'token':token})[0] in {200,204}
|
|
+ result=recover_fixture_cleanup(journal,permit(native,custody,binding,owner_token=token))
|
|
+ assert not result['confirmed'] and all(x['outcome']=='owed' for x in result['obligations'])
|
|
+ assert call('GET','sys/policies/acl/'+native.policy_name)[0]==200
|
|
+ # A separately admitted owner attempt can recover after the failed attempt;
|
|
+ # neither the failed attempt nor its receipt grants an automatic retry.
|
|
+ assert recover_fixture_cleanup(journal,permit(native,custody,binding))['confirmed']
|
|
+ assert journal.db.execute('SELECT COUNT(*) FROM recovery_attempts').fetchone()[0]==2
|
|
+
|
|
+
|
|
+def test_policy_name_reuse_never_deletes_different_policy(owner,tmp_path):
|
|
+ native,journal,binding,call,custody=protected(owner,tmp_path);crash(native,journal,binding)
|
|
+ replacement='path "unrelated" { capabilities = ["read"] }'
|
|
+ assert call('PUT','sys/policies/acl/'+native.policy_name,{'policy':replacement})[0] in {200,204}
|
|
+ result=recover_fixture_cleanup(journal,permit(native,custody,binding))
|
|
+ assert not result['confirmed']
|
|
+ assert [x['outcome'] for x in result['obligations']]==['confirmed','confirmed','owed']
|
|
+ assert call('GET','sys/policies/acl/'+native.policy_name)[1]['data']['policy']==replacement
|
|
+
|
|
+
|
|
+def test_private_custody_permissions_and_identity_binding_fail_closed(owner,tmp_path):
|
|
+ native,journal,binding,call,custody=protected(owner,tmp_path)
|
|
+ path=custody._path(binding);path.chmod(0o644)
|
|
+ with pytest.raises(Refused):custody.read(binding)
|
|
+ path.chmod(0o600)
|
|
+ data=json.loads(path.read_bytes());data['binding']['operation_id']='other'
|
|
+ path.write_text(json.dumps(data))
|
|
+ with pytest.raises(Refused):custody.read(binding)
|
|
+
|
|
+
|
|
+def test_lost_recovery_response_stays_unknown_until_new_independent_attempt(owner,tmp_path,monkeypatch):
|
|
+ native,journal,binding,call,custody=protected(owner,tmp_path);crash(native,journal,binding)
|
|
+ import task_execution_recovery as recovery_module
|
|
+ original=recovery_module.http.client.HTTPConnection.getresponse
|
|
+ once=[True]
|
|
+ def lost(connection):
|
|
+ response=original(connection)
|
|
+ if once[0]:
|
|
+ once[0]=False
|
|
+ response.close()
|
|
+ raise Refused('synthetic lost recovery response after real revoke')
|
|
+ return response
|
|
+ first=permit(native,custody,binding)
|
|
+ with monkeypatch.context() as patch:
|
|
+ patch.setattr(recovery_module.http.client.HTTPConnection,'getresponse',lost)
|
|
+ assert not recover_fixture_cleanup(journal,first)['confirmed']
|
|
+ assert journal.observe(binding)['external_outcome_unknown']
|
|
+ with pytest.raises(Refused):recover_fixture_cleanup(journal,first)
|
|
+ assert recover_fixture_cleanup(journal,permit(native,custody,binding))['confirmed']
|
|
+ # Reconciliation does not erase the historic uncertain invocation.
|
|
+ assert journal.observe(binding)['external_outcome_unknown']
|
|
+ assert journal.db.execute("SELECT COUNT(*) FROM recovery_calls WHERE outcome='pending'").fetchone()[0]==1
|
|
diff --git a/workplans/RPF-WP-0035-credential-lane-implementation.md b/workplans/RPF-WP-0035-credential-lane-implementation.md
|
|
index 8b09305..1b3a079 100644
|
|
--- a/workplans/RPF-WP-0035-credential-lane-implementation.md
|
|
+++ b/workplans/RPF-WP-0035-credential-lane-implementation.md
|
|
@@ -1290,3 +1290,12 @@ CAS records consumed/unknown with zero native effects. Exact dependency/source
|
|
hashes, tests and remaining gates are retained with the source package and in
|
|
docs/task-execution-mediator-seam.md. No real grant/custody, policy default, owner
|
|
sync or native production acceptance is performed; T12 stays progress.
|
|
+
|
|
+2026-10-16 T12 recovery successor source preparation: optional independently
|
|
+installed fixture custody durably retains operation-bound private accessors and
|
|
+issuance checkpoints. Fresh cleanup-only owner permits can reconcile after actual
|
|
+executor process death using fixed revoke/readback and policy delete/readback,
|
|
+without restoring business/consume admission. Separate durable attempt evidence
|
|
+retains original unknown outcomes. Lost token-create response remains unresolved
|
|
+when no accessor was committed. Source-only process-loss fixtures are not native
|
|
+production custody or recovery-role admission; T12 remains progress.
|