Recover exact task cleanup after executor process loss
All checks were successful
CI Smoke / host-smoke (push) Successful in 0s
CI Smoke / container-smoke (push) Successful in 1s

Assistant: codex
Assistant-Model: gpt-6-astra
Assistant-Session: 01a140c4-354a-7823-8f0d-8f717397f786
This commit is contained in:
codex 2026-10-16 16:43:20 +02:00
parent d870596cc7
commit b9eaa20060
6 changed files with 774 additions and 2 deletions

View file

@ -121,3 +121,87 @@ retry cleanup. Native effects can survive process loss. Cleanup handles remain
memory-only: no independently authorized recovery mechanism is implemented here.
This closes disposable sequential apply acceptance only, not TSA03 or native
production recovery/admission.
## Independent cleanup recovery source successor
`task_execution_recovery.py` adds separately installed fixture-only owner custody
and fresh recovery permits. Optional `recovery_custody` on NativeFixtureAdapter
persists the original binding/cleanup commitments, exact operation policy, backend
port and identity accessor before any grant effect; it commits mint intent before
token creation and task accessor immediately after a valid issued response.
Bearer credentials are never persisted. The custody directory is separate from
the public journal, owner-only 0700/0600, bounded at 64KiB, no-follow, with atomic
replacement and fsync. It is real process-independent persistence for synthetic
fixtures. It does not isolate same-UID hostile execution, encrypt at rest, select
production storage or establish a production secret policy. Those are explicit
native custody/deployment gates, not supplied by Python types or file modes.
The independent owner reads its private custody revision and installs a fresh
bounded permit, current owner credential and retained live disposable Popen.
Permit revocation is durably owner-recorded. The recovery runner has no mint,
consume, policy-apply or business method. It dispatches only fixed exact-accessor
revoke/readback and exact `task-exec-<operation>` policy read/delete/readback.
A differently recreated policy is never deleted. Backend admission remains the
same loopback-only owner-spawned dev process; no estate backend can be selected.
Accessor-based recovery is an independently admitted alternate cleanup projection,
not a reinterpretation of the original revoke-self call digest. Production needs
exact owner-reviewed recovery scope; original business grant does not imply it.
Fresh recovery holds the journal process lock but never sets business admission
or consumed-admission fields. Separate recovery attempt/call/result tables retain
original cleanup statuses before reconciliation, write native intent before calls,
and preserve failed/pending outcomes. Proven cleanup alone updates the exact
original cleanup obligation to confirmed. Business state and original pending
call uncertainty remain unchanged. Reusing the original approval, continuing
metadata calls and reusing a recovery permit refuse. Failure needs another
independent owner permit; there is no automatic retry. Expired/revoked permits
or changed private custody revision refuse before calls.
A token-create response lost before accessor custody remains unresolved token
cleanup debt: policy deletion/identity revocation can be independently confirmed,
but cannot imply task-token revocation. Neither expiry guessed from wall clock
nor policy absence fabricates token absence. A production successor must admit
an independent exact ambiguous-issuance reconciliation mechanism or prove bounded
expiry from owner backend evidence. No such new grant is implemented here.
`test_task_execution_recovery.py` forks a real executor and terminates it without
finally cleanup after issue, business effect/lost response and revoke/lost response.
The independent parent, holding the backend and separately installed custody,
recovers specific obligations and verifies actual token/policy absence. Additional
cases cover lost-mint response, revoked credential, permit revocation/expiry,
revision drift, namespace policy-name reuse and private-file/binding refusals.
This is source fixture proof only. Native custody, isolation, exact recovery-role
permissions, retention/encryption, clock admission, audit delivery and human scope
approval remain missing. No generic broker, production reader, grant or CLI is added.
### Recovery R2 exact absence and authenticated capsule refinement
R1 overclaimed policy absence for arbitrary 404 bodies. R2 requires exactly
`{"errors":[]}` for both initial GET and post-DELETE GET. Other 404 bodies,
permission denials, extra data, malformed/duplicate-key/nonfinite JSON and failed
readback cannot confirm policy cleanup. A successful DELETE status alone cannot
confirm absence. Recovery native policy comparison admits precisely one optional
terminal LF, matching the retained OpenBao HCL source/readback convention; spaces,
tabs, CR, extra terminal LF and any other byte change are refused. This is a
narrow declared normalization, not full whitespace or semantic equivalence.
Private fixture capsules now carry HMAC-SHA256 authentication under an exactly
32-byte integrity key supplied independently by the recovery owner. The key is
never stored in the capsule or public journal, and is not a bearer credential.
The fixture owner retains this key independently across executor death; a fresh
owner custody object must supply the same key. Wrong-key or damaged accessor/HCL
bytes refuse before admission. This is integrity protection, not encryption or
same-UID isolation; production key custody, rotation/retention and recovery actor
selection remain explicitly unadmitted.
After authentication the reader enforces exact top-level and nested binding/
cleanup key sets and types, validated Binding/Cleanup formats and the ordered
source-derived cleanup_specs operation digests. Duplicate JSON keys, nonfinite
numbers, malformed phase/revision combinations and type-coercible values refuse.
The reader bounds bytes, rejects indirect/nonprivate files and detects stat/size
changes during its single read. A valid older prepared capsule still cannot
prove a task token absent when durable journal state/calls reached issuance:
that remains unknown cleanup debt. Authentication alone is not anti-rollback.
No business or consume admission is restored and no production scope is granted.