---
id: RPF-WP-0015
type: workplan
title: "Coordinate audit-core temporary custody and recovery exercises"
domain: financials
repo: railiance-platform
status: active
owner: codex
topic_slug: railiance
created: "2026-08-22"
updated: "2026-08-22"
related:
  - AUDIT-WP-0008
  - WH-ENG-20260822-AUDIT-E2-01
  - WH-ENG-20260822-AUDIT-E2-02
  - RAILIANCE-WP-0025
origin: routed
origin_ref: "State Hub messages 10f80080-4c83-42ba-8590-f23c582d9f05 and a93fa88f-a9c5-4539-93ae-0c8f8490f53d"
state_hub_workstream_id: "f4640325-e89c-591d-b58e-ec6b087900ac"
---

# RPF-WP-0015 — audit-core custody and recovery coordination

## Goal

Define value-safe, approval-gated execution paths for audit-core's proposed
Whitehat E2 sender exercise, runtime database-lease recovery exercise, and
coordinated `railiance01` reboot. This workplan produces preflight contracts;
it does not authorize a live token mint, OpenBao mutation, workload restart,
lease revocation, firewall change, or host reboot.

## Boundaries

- Never put bearer values, database credentials, Secret payloads, or OpenBao
  response bodies in Git, State Hub, command output, evidence, or chat.
- Use exact paths and non-secret handles only. Temporary files, if later
  approved, must be outside Git, mode `0600`, and securely removed.
- A sender token stored in the audit-core registry has no intrinsic OpenBao
  lease. Audit-core deployed boundary enforcement for timezone-aware
  `expires_at` at commit `abd22fa` / image digest
  `sha256:c2fe39a0185b99be3fc0cb14d2de69772b8e66e20490097c9d11d90cc39719a6`.
  Reverify that exact prerequisite before projection; projection-credential
  expiry alone is still not sufficient.
- All live exercises require a separate engagement acknowledgement, attended
  operators, a named window, and explicit abort authority.

## T01 — Define the Whitehat E2 temporary sender lane

```task
id: RPF-WP-0015-T01
status: done
priority: high
state_hub_task_id: "1d0cd97e-bc65-5dfe-b57a-97649419a118"
```

Define the reviewed path for exactly two distinct bearer values with an
effective lifetime of at most 900 seconds. Each identity is bound to
`source=whitehat-security`, one exact synthetic tenant, `may_write=true`, and
`may_read=true`; neither reuses a production sender identity.

The design must keep the full sender registry out of namespace `whitehat`.
Use exact per-engagement KV paths and exact-key ExternalSecrets that project
only mounted token files. Define the readiness-gated audit-core rolling reload,
the audit-core `expires_at` prerequisite, the scheduled and attended cleanup,
and value-safe positive/negative evidence.

Done when audit-core and the engagement owner can review a complete path whose
failure mode is expiry or denial, not a long-lived accepted bearer.

**Window update (2026-08-22):** engagement `WH-ENG-20260822-AUDIT-E2-01` is
operator-approved and target-owner-acknowledged for 18:00Z–18:15Z. The task is
waiting for that attended window. Do not mint before approximately 17:59Z;
both identities must expire no later than 18:15Z and cleanup/reload must be
attended. No unattended scheduler or background credential job is authorized
by this record.

**Procedure implementation (2026-08-22):**
`scripts/audit-core-whitehat-e2-credentials.py` now provides fail-closed
`preflight`, `project`, and idempotent `cleanup` commands. The projection uses
two exact per-engagement OpenBao paths, an ephemeral least-privilege Kubernetes
auth role, a Whitehat-only ClusterSecretStore, and an ExternalSecret that emits
exactly `token-a` and `token-b`. Project is hard-gated to 18:00Z–18:03Z;
cleanup remains callable after interruption and removes only the named
temporary identities and exact engagement resources. Four unit tests, a live
value-safe preflight, policy formatting, and Kubernetes server-side dry-run
passed. Live projection and cleanup remain waiting for the attended window.

**Fresh-run readiness (2026-08-22):** the expired `-01` engagement is terminal
and clean. Its replacement, `WH-ENG-20260822-AUDIT-E2-02`, is operator-approved
and target-owner-acknowledged for 19:15Z–19:30Z. Commit `f135c0a` gives the new
run a dedicated `-02` policy, auth role, ClusterSecretStore, four exact KV
paths, and fail-closed procedure; no `-01` projection reference remains. The
projection gate is 19:15Z–19:18Z and cleanup must finish by 19:30Z. Platform
tests (114), Whitehat tests (53), both Kubernetes server-side dry-runs, and a
live value-safe preflight passed. At preflight there were no temporary token
paths, projection resources, runner pod, or temporary sender identities, and
no secret value was observed. At that pre-run point the task remained `wait`
until an attended operator invoked projection and completed the cleanup receipt.

**Terminal result (2026-08-22):** projection succeeded at 19:17:54Z with
exactly two bounded identities and the two declared mounted keys. Whitehat
failed closed before traffic because its live broker could not consume the
value-safe platform receipt; the runner sent zero target packets and was
deleted at 19:21:07Z. Exact cleanup completed at 19:21:39Z and independent
preflight proved no temporary identity, KV path, ESO resource, mounted Secret,
or runner remained; audit-core was `1/1` Ready and no secret value was
observed. The first attempt also proved automatic rollback when SSH shell
expansion corrupted a Go-template argument; commit `d239ed3` fixes that defect.

T01 is done because the mount-only projection, application expiry, reload,
abort, and cleanup lane is implemented and live-proven. The missing consumable
broker contract is not carried as prose in this completed task: versioned
receipts, the pre-projection broker gate, generic lifecycle, and future expired
cleanup interface are implemented under `RAILIANCE-WP-0025`. The dated `-01`
and `-02` procedures are terminal evidence/cleanup references and must not be
copied for another engagement.

## T02 — Define the runtime database lease recovery exercise

```task
id: RPF-WP-0015-T02
status: progress
priority: high
state_hub_task_id: "fda4262a-56f6-5f06-a070-8fe7e1297209"
```

Coordinate with `rapp-postgres` and audit-core on a value-safe exercise for
`database/creds/audit-core-runtime`: identify the lease only by handle, revoke
or expire it under synthetic load, observe `/readyz` and 503 retry behavior,
and prove ESO refresh plus mounted-file reread restores service without a pod
restart. Record preconditions, time bounds, abort conditions, and rollback.

Done when the procedure proves the property claimed by audit-core's credential
reader without exposing a username/password or affecting non-synthetic data.

**Procedure implementation (2026-08-22):**
`docs/audit-core-database-lease-recovery.md` fixes the exact consumer, lease,
database, synthetic-load, owner, time-bound, abort, and value-safe evidence
contract. It requires matching the current lease internally while retaining
only its non-usable handle, proving health 200/readiness 503/retryable request
503 after revocation, forcing only the database ExternalSecret, and proving
recovery with the same pod UID and restart count. The read-only helper
`scripts/audit-core-recovery-preflight.py database-lease` passed live automated
checks without observing Secret data. The live revocation still waits for an
approved window, audit-core/rapp-postgres acknowledgement, a separately
approved synthetic-load contract, and a named abort operator.

**Harness hardening (2026-08-22):**
`scripts/audit-core-database-lease-recovery.py` now implements the attended
orchestration but cannot run from repository access alone. It requires a
value-safe approved receipt based on
`docs/audit-core-database-lease-approval.example.json`, a revision-pinned
synthetic-load driver, exact confirmation, an open ≤15-minute window, all
three owner acknowledgements, and at least five minutes before the ordinary
lease refresh. OpenBao metadata inspection corrected the selection contract:
lease lookup exposes issue/expiry/TTL, not a database username. The harness
therefore aborts unless the exact runtime prefix has one live handle whose
issue time coheres with the ExternalSecret refresh. A live read-only check
observed exactly one handle, 0.073 seconds of issue/refresh skew, and no secret
values. It was not revoked.
The procedure's responsibility table now uses the same metadata-only selection
rule; it no longer claims lease metadata exposes or matches a mounted database
username.

**Owner receipts complete (2026-08-22):** the canonical direct-review contract
`f86d418f951f829f075de04dd825c6e2e185e577019ee23d6da1fa9040302d62`
is approved by audit-core (`0c0e3dff-c7fd-4fac-898b-1aec160f4753`),
rapp-postgres (`e958f75a-4e41-4103-9261-a4393eb6f1ef`), and
railiance-platform (`52ab4c51-0950-45e8-a592-564dc2a450bb`). T02 no longer
waits on procedure review. Its remaining gates are a revision-pinned approved
synthetic-load driver, a named abort operator, and a non-overlapping ≤15-minute
live window after the Whitehat engagement cleanup.

## T03 — Define the coordinated railiance01 reboot exercise

```task
id: RPF-WP-0015-T03
status: progress
priority: high
state_hub_task_id: "ba729d18-f053-539f-86ef-5cb8ad4cce45"
```

Coordinate the single-node outage with `railiance-infra`,
`railiance-cluster`, audit-core, and the OpenBao package owner. Require current
backup evidence, operator access, a declared restart order, OpenBao
seal/readiness handling, ESO reconciliation, audit-core database and sender
readiness, synthetic end-to-end verification, notifications, and explicit
abort/rollback conditions.

Done when an operator-approved window can execute from one checklist without
guessing or relying on credential values in the record.

**Procedure implementation (2026-08-22):**
`docs/railiance01-coordinated-reboot.md` now provides the single ordered
preflight, hold point, reboot, recovery, escalation, and evidence checklist
across host, cluster, OpenBao/ESO, platform-pg, and audit-core owners. The
read-only helper `scripts/audit-core-recovery-preflight.py node-reboot` passed
all automated live checks: node/k3s, platform-pg 1/1, continuous archiving,
fresh completed Barman backup, unsealed OpenBao, ESO Valid/SecretSynced, and
the reviewed audit-core digest 1/1 Ready. Live readiness remains fail-closed on
the intentionally non-automated gates: an approved window and owner set,
provider-console access, a current OpenBao Raft snapshot receipt, 2-of-3 Shamir
quorum availability, and a named abort operator.

**Snapshot gate hardening (2026-08-22):** reboot preflight no longer accepts a
file-existence assertion as snapshot evidence. A receipt based on
`docs/openbao-reboot-snapshot-receipt.example.json` must be no more than 24
hours old, match the live OpenBao cluster id and a possible Raft applied index,
carry distinct non-placeholder plain/encrypted hashes, and attest encryption,
off-host persistence, verification, and absence of secret material. The
pending example intentionally fails. Creating the secret-bearing snapshot and
its off-host copy remains an attended platform-custody action near the future
reboot window; no snapshot was created in this implementation step.

**Prior-evidence audit (2026-08-22):** rapp-postgres evidence from the
2026-08-16 reboot proves the same PVC returned without re-bootstrap, original
audit triples and the isolation fixture survived, platform-pg recovered, and
audit-core readiness lagged by roughly 40 seconds. It remains a baseline, not
T03 completion: that reboot was not driven by continuous synthetic accepts and
did not measure the complete post-unseal ESO, audit-chain, and accepted-event
recovery path required here. T03 therefore still needs one deliberate run, but
does not need to rediscover the expected boot order.

**Owner receipts complete (2026-08-22):** the same canonical contract is
approved by audit-core (`0c0e3dff-c7fd-4fac-898b-1aec160f4753`),
rapp-postgres (`e958f75a-4e41-4103-9261-a4393eb6f1ef`),
railiance-cluster (`f5919864-b7f1-40b5-b07e-d19055dffca8`),
railiance-infra (`9bbf986c-7385-42de-964f-9040eebdfee4`), and
railiance-platform (`52ab4c51-0950-45e8-a592-564dc2a450bb`). T03 no longer
waits on procedure review. Its remaining gates are the approved outage window,
fresh encrypted off-host OpenBao snapshot receipt, provider-console access,
2-of-3 Shamir quorum, and named abort operator.

**WP0027 preparation gate (2026-08-23):** under preparation-only scenario
`WARDEN-WP-0027-T02-DRILL-20260822-01`, platform-admin OIDC/MFA created a
fresh Raft snapshot, encrypted it to the established age recovery recipient,
verified a decrypt stream against the plaintext digest, and uploaded only the
ciphertext through the approved write-only off-host lane. The platform
validator accepted metadata receipt
`reviews/WARDEN-WP-0027-T02-DRILL-20260822-01-openbao-snapshot-receipt.json`
against the live cluster id and a possible applied index; automated recovery
checks remained green and no secret value was observed. Both local snapshot
files were securely deleted. The platform accepts the exact attended driver
role, but this receipt authorizes no seal, unseal, restore, reboot, or other
live execution; the final scenario still requires the independent quorum gate,
fully parameterized green preflight, and an exact human GO.

**WP0027 live attempt terminal NO-GO (2026-08-23):** State Hub decision
`449a697a-5303-4582-aa9e-b0bc8b35ab2d` opened the bounded live window and the
fully parameterized preflight reran green at `2026-08-22T22:55:11Z`. At the
live hold point, the governed attended-login adapter authenticated but could
not persist its token helper into the agent's read-only home and unexpectedly
emitted the short-lived credential into captured output despite the no-print
contract. The token was immediately self-revoked and the drill stopped before
seal. Post-abort status proved OpenBao remained initialized and unsealed with
zero unseal progress; no seal, unseal, reboot, restore, policy, PVC, or general
workload mutation occurred. Metadata-only NO-GO receipts were sent to
ops-warden (`b04e92bc-4523-4c3e-8d14-78de5da4e67e`) and railiance-infra
(`5264d499-0ee2-40b5-bffd-f1f875718153`). The decision is consumed and must not
be retried. A future window requires a login adapter that proves isolated
writable token-helper state and output suppression on persistence failure,
followed by fresh owner receipts and a fresh human decision.

## T04 — Route the contracts and approval gates

```task
id: RPF-WP-0015-T04
status: done
priority: medium
state_hub_task_id: "52f37fac-2449-5e65-ab5b-8c4889caee48"
```

Route the reviewed procedures to audit-core, `rapp-postgres`,
`railiance-cluster`, and `railiance-infra`. Identify each live mutation and its
owner, record outstanding approvals, and keep live tasks waiting until the
engagement or recovery window is explicitly acknowledged.

Done when every participant has the same value-safe procedure, responsibility
map, and stop conditions.

**Routing package (2026-08-22):** the owner-review package consists of
`docs/audit-core-whitehat-e2-credential-projection.md`,
`docs/audit-core-database-lease-recovery.md`, and
`docs/railiance01-coordinated-reboot.md`. T01's live credential window remains
separate from the new T02/T03 authorization gates. Outstanding approvals are
stated in each procedure and enforced by the read-only preflight output rather
than inferred from repository access.

Routed from immutable revision `bd25f7f` to audit-core
(`5ef8f06b-76f4-48ae-a09e-ba4eb87e6906`), rapp-postgres
(`8e80f166-267a-471e-ab1d-e0293e0781e7`), railiance-cluster
(`268d64c0-4b3c-408f-bdfb-e07de588111d`), and railiance-infra
(`3918db56-ca20-4cc7-8b63-001d840bc608`). Every participant received the same
procedure paths, responsibility boundary, stop conditions, passing automated
preflight state, and explicit statement that no live mutation was requested.

**Direct review interface (2026-08-22):**
`interfaces/RPF-WP-0015-owner-reviews.json` and
`scripts/wp0024-owner-review.py` replace follow-up prose-to-task translation
with owner-executable `show`, `verify`, `approve`, `request-changes`, and
aggregate `status` operations. Approvals are valid only for the canonical
contract digest and the owner's pinned artifact hashes, so a changed procedure
invalidates the old receipt. The interface runs only closed-set, read-only
checks and cannot perform any T02/T03 live mutation.
The task map requires audit-core, rapp-postgres, plus railiance-platform for
T02, and audit-core, rapp-postgres, railiance-cluster, railiance-infra, plus
railiance-platform for T03; aggregate status is computed independently for
each task.

## Acceptance

- [x] E2 cannot proceed without enforced sender expiry and exact-path projection.
- [ ] Database recovery is restart-free and evidenced without credential values.
- [ ] Reboot recovery has an ordered, owner-signed checklist and abort path.
- [x] No live action is implied by completing this design workplan.
