secrets-engine/workplans/SECRETS-WP-0007-production-lifecycle-hardening.md

---
id: SECRETS-WP-0007
type: workplan
title: "Production-safe provisioning, authorization, and lifecycle hardening"
domain: infotech
repo: secrets-engine
status: active
owner: codex
topic_slug: custodian
created: "2026-08-23"
updated: "2026-09-02"
state_hub_workstream_id: "68a39be1-bd9c-5133-ad64-e7bca892aaf3"
---

# SECRETS-WP-0007 - Production-safe provisioning, authorization, and lifecycle hardening

## Goal

Close the highest-risk gaps between the proven CLI MVP and the production
workflow described by `INTENT.md`, before expanding secrets-engine into a
service or moving additional high-risk lanes from interim routing to native
operation.

The source assessment is
`history/2026-08-23-scope-intent-capability-gap-assessment.md`. The most relevant
gaps are:

1. single-field `bao kv put` can replace siblings on a multi-field path and puts
   the value in the local subprocess argument vector;
2. KV `revoke` destroys metadata while leaving policy/AppRole access behind,
   and its dry-run does not match live behavior;
3. production approval checks are coarse status checks, accept local fixtures,
   and do not bind actor, action, lane, fields, expiry, or dual control;
4. AppRole delivery tokens are not explicitly revoked after exec/verification;
5. route and verification can attest only the first field and use a garbage
   token rather than a real unrelated identity for KV denial;
6. evidence is best-effort and incomplete on failure paths.

## Scope and sequencing

This workplan hardens the existing CLI contract first. It covers KV write
transport and merge semantics, lifecycle command semantics, decision binding,
short-lived delivery sessions, verification/readiness, and evidence. It then
provides the safe implementation gate for resuming `SECRETS-WP-0006-T05/T06`.

The first task is containment. No task may exercise live production values to
develop or test a primitive. Use fake values and a throwaway OpenBao instance
until the relevant behavior has passed unit and integration tests. Existing ESO,
Kubernetes/OIDC delivery, provider rotation, and application health remain with
their current owners.

## Non-goals

- A service API, daemon, UI, scheduler, or remote multi-user protocol.
- Runtime tenant authentication, quotas, or a tenant control plane.
- A second custody backend or backend-portability project.
- Provider-specific rotation automation.
- Replacing flex-auth, State Hub decisions, key-cape/user-engine identity,
  ops-warden routing, ops-bridge transport, or OpenBao enforcement.
- Unparking the standalone `warden-sign` AppRole as a break-glass path.

Service mode remains deferred until the exit criteria here are satisfied; it
must not expose the current bootstrap and lifecycle shortcuts as an API.

## Safety invariants

- No raw value in Git, State Hub, chat, prompts, workplans, evidence, normal
  logs, environment diagnostics, or a subprocess argument vector.
- No write to a shared multi-field path unless sibling preservation is proven.
- No destructive metadata deletion through a command presented as ordinary
  revoke/deactivate.
- Production actions require canonical, exact-action approval. Offline fixtures
  cannot authorize steady-state production mutations or delivery.
- Existing workload delivery stays in place until separately approved owner
  coordination says otherwise.
- Tests use fake values and throwaway OpenBao only; live checks are
  metadata/capability-safe and approval-gated.

## Contain unsafe lifecycle surfaces

```task
id: SECRETS-WP-0007-T01
status: done
priority: high
state_hub_task_id: "5ddba36d-bc86-5670-baf1-50c872e6577e"
```

Completed 2026-08-23. During containment, existing multi-field provisioning
failed before file read or backend access. Ordinary `revoke` builds one
lifecycle plan used by both
dry-run and live execution, deletes only engine-managed native AppRole/policy
objects, and explicitly preserves KV custody, external delivery auth, and
workload delivery. No destructive KV command remains on the ordinary CLI.
Focused lifecycle, admission, auth-capability, and throwaway OpenBao integration
tests pass. The temporary multi-field rejection was then superseded by the
merge-safe primitive completed in T02.

Add immediate fail-closed guards before implementing replacement primitives:

- reject `provision` on an existing multi-field lane;
- reject or rename the current destructive KV `revoke` path so it cannot be
  mistaken for soft deactivation;
- make KV revoke dry-run render exactly the operations the live path would run;
- document that `SECRETS-WP-0006` may apply/verify metadata but must not provision
  or revoke shared values through the current commands.

Acceptance:

- A regression test proves no backend call occurs for multi-field existing-lane
  provisioning.
- A regression test proves dry-run and live operation plans contain the same KV,
  AppRole, and policy targets.
- The destructive action requires an explicit name/flag and a separately
  approved action; ordinary revoke cannot destroy KV metadata.
- CLI/help/runbook language no longer describes metadata deletion as reversible
  deactivation.

## Implement merge-safe, non-argv provisioning

```task
id: SECRETS-WP-0007-T02
status: done
priority: high
state_hub_task_id: "ddf3ed54-9cbc-5e18-b15a-5da14cda5d9b"
```

Completed 2026-08-23. Replaced raw `field=value` argv writes with a CAS-aware
backend primitive: new paths use CAS-zero create; existing paths use server-side
HTTP patch with the observed metadata version. Values travel through a
mode-0600 temporary JSON reference removed in `finally`, and never appear in
argv. Unit tests inspect the full request/cleanup; throwaway OpenBao integration
proves sibling preservation and stale-CAS rejection. Stage policies now include
the narrow KV `patch` capability needed by this operation.

Replace the raw `field=value` subprocess call with a backend write primitive
that never places the value in argv and safely updates one or more declared
fields without removing unmentioned siblings. The design may use OpenBao CAS,
a merge/patch operation, an input file reference with strict cleanup, or direct
request input, but it must remain isolated behind `OpenBaoClient` and preserve
the catalog field allowlist.

Acceptance:

- Unit tests inspect the complete subprocess request and prove the fake value is
  absent from argv, environment, output, exceptions, and evidence.
- Throwaway OpenBao integration proves updating one field preserves every
  sibling field and fails closed on a CAS conflict.
- Multi-field input is atomic from the caller's perspective; a partial failure
  cannot silently leave an unreported mixed version.
- Source and generated inputs retain mode/path guards and guaranteed temporary
  cleanup.
- Production generation remains forbidden, and high-risk production
  provisioning is still decision- and dual-control-gated.

## Split suspend, deactivate, and destroy semantics

```task
id: SECRETS-WP-0007-T03
status: done
priority: high
state_hub_task_id: "bb6073e1-c952-5357-b707-5574cef50f2f"
```

Completed 2026-08-29. Suspend, deactivate, and destroy share one plan object
for dry-run and live execution. Ordinary `revoke` aliases native deactivation
and preserves KV custody plus externally managed workload delivery. Live
destroy stays fail-closed until T04's exact-action contract; that is the
designed containment, not unfinished T03 work. Issued delivery sessions
self-revoke in T05.

Residual from this task, now implemented: `secrets-engine session revoke`
`--accessor-file` or `--lease-id-file` (hardening H4a).

Progress 2026-08-23. Added one lifecycle plan model shared by dry-run and live
execution. Ordinary `revoke` safely aliases native deactivation; suspend and
deactivate preserve KV custody and externally managed workload delivery.
Destroy has an unmistakable dry-run plan and exact-id confirmation, but its live
handler is deliberately fail-closed before coarse lane approval or backend
access until T04 supplies distinct action authorization. Scoped issued-session
self-revocation is implemented in T05; a general known-accessor operator
command is `session revoke --accessor-file` (H4a).

Replace the overloaded `revoke` behavior with explicit lifecycle operations and
plans. Define at least:

- **suspend delivery**: stop minting new native delivery sessions while
  preserving custody and a documented rollback;
- **deactivate native access**: remove/disable the lane AppRole and consumer
  policy without deleting the KV value;
- **revoke issued sessions**: revoke known tokens/leases by non-secret accessor
  where the backend supports it;
- **destroy custody metadata**: irreversibly delete KV metadata/versions only
  through an unmistakable, separately approved action.

Preserve compatibility only where it cannot mislead. A legacy `revoke` alias
may map to a safe non-destructive action; it must never silently retain the
current destructive meaning.

Acceptance:

- Each operation has a guarded dry-run plan, exact targets, distinct approval
  action, non-secret evidence, and documented rollback/irreversibility.
- Dry-run and live use the same plan object rather than separate handwritten
  descriptions.
- Integration tests prove suspend/deactivate preserve KV data and destroy does
  not run without its explicit approval and confirmation contract.
- Auth-capability and KV lanes have coherent but kind-appropriate behavior.
- Existing workload delivery is reported as an external dependency and is not
  deleted by native lifecycle commands.

## Bind approvals to exact production actions

```task
id: SECRETS-WP-0007-T04
status: wait
priority: high
state_hub_task_id: "4b58edec-c705-55e5-9ece-362e1ff13079"
```

Wait 2026-08-29. The consumer validator and production fail-closed gate are
shipped. What remains is not local engine work: State Hub / `access-engine`
must serve the durable ActionAuthorization object. Paired with
`SECRETS-WP-0008-T02`. Do not reopen live production on hub status or local
fixtures.

Progress 2026-09-02. `GH-DEC-2026-003` consume-before-OpenBao is now a
shared production gate: `_require_lane_approval` CAS-consumes through
approval-engine before any OpenBao client is resolved. Tests prove 409,
unreachable, and missing-binding paths never call OpenBao. Live production
stays fail-closed until the durable object is served; this task remains
`wait`.

Progress 2026-08-23. flex-auth revision `c473f19` now supplies execute-time
structured binding and the proposed durable `ActionAuthorization` schema. The
engine builds and fail-closed validates the exact secrets-engine profile,
validity, State Hub authority, decision digest/binding, accepted policy
revision, and a caller-selected minimum distinct-approver threshold. Every live
production handler now carries its action into the shared gate and fails before
OpenBao while State Hub lacks the durable endpoint. Legacy local decisions are
reachable for prod-labeled lanes only with an explicit unsafe-demo switch,
disabled Hub URL, and loopback OpenBao. The throwaway scripts declare that
exception. Live destroy remains disabled independently. State Hub endpoint and
authenticated approval storage are still outstanding. flex-auth corrected its
example digest and added a complete binding regression assertion in `d402408`.

Define and enforce the decision contract needed by production commands. A
resolved approval must bind at least:

- canonical decision UUID and non-superseded status;
- catalog id and stage;
- action (`apply`, `provision`, `verify`, `exec`, `handoff`, `suspend`,
  `deactivate`, `destroy`, or later `rotate`);
- exact field set or policy/auth targets where applicable;
- requesting actor/purpose and expiry or bounded execution window;
- required approval count and distinct approvers for `dual-control`.

Coordinate the external object shape with State Hub/flex-auth rather than
inventing authorization ownership in this repo. Keep local fixtures for
throwaway demos and non-production tests only; require an explicit unsafe-demo
switch if they remain reachable at all.

Acceptance:

- Every privileged production handler passes an action-specific request into
  one shared authorization function.
- Tests reject wrong lane, stage, action, field set, actor/purpose, expiry,
  superseded status, insufficient approvals, duplicate approvers, and local
  production fixtures before any OpenBao call.
- `approval.model: dual-control` has enforced semantics rather than being a
  label.
- Catalogs use resolvable canonical decision UUIDs for live actions; legacy CCR
  references remain provenance, not executable authorization.
- State Hub outage fails production actions closed while dry-run/catalog
  inspection remains usable.

## Make delivery sessions short-lived and explicitly closed

```task
id: SECRETS-WP-0007-T05
status: done
priority: high
state_hub_task_id: "36bcd64d-8b27-542d-b8da-ede1a4d3c22f"
```

Completed 2026-08-29. Delivery AppRole sessions self-revoke in `finally` on
exec and both verification kinds. Bootstrap is a named provider with no
implicit fallback from the KeyCape scaffold. Wiring that scaffold as OpenBao
JWT login is `SECRETS-WP-0008-T06` / hardening H1, blocked on the
platform-owned role — not remaining T05 engine work.

Progress 2026-08-23. AppRole login material now travels through strict temporary
JSON input rather than argv. Exec and both verification kinds use a scoped
session that self-revokes in `finally`, drops the in-memory token even when
cleanup fails, and exposes only an accessor fingerprint plus cleanup booleans as
evidence. Tests cover success and exception cleanup, idempotence, failed revoke
visibility, and value/credential exclusion. Provider review requests were sent
to railiance-platform (`8f910aff-3a94-43c6-8805-eb9276e46fc0`) and key-cape
(`0627ca55-115e-43dc-b2f6-3195be3bc90d`); steady-state engine authentication
remains external-contract work.

KeyCape has accepted the `secrets-engine-openbao` service contract. An explicit
consumer scaffold now performs HTTPS client-credentials exchange, strict
out-of-repo client-secret input, exact claim/assurance/lifetime preflight,
three-minute renewal classification, and rejection of ID/refresh tokens. It has
no bootstrap/operator/AppRole fallback and is deliberately not wired into the
CLI until railiance-platform materializes the exact OpenBao JWT mount and role.

Refactor AppRole login into a scoped session object that retains only the
minimum non-secret lifecycle handle needed to revoke the issued token in a
`finally` path. Apply it consistently to exec, KV verification, auth-capability
verification, and any future file/wrapped delivery.

Define the steady-state OpenBao authentication interface separately from the
delivery AppRole. Bootstrap token files remain an explicit break-glass/setup
provider; add a bounded OIDC/service-auth provider only with railiance-platform
and identity-owner review.

Acceptance:

- Success, child non-zero exit, setup failure, verification failure, SIGINT, and
  SIGTERM tests all attempt token revocation and clean temporary material.
- Secret IDs remain single-use where configured; tokens retain TTL/use limits
  as backstops rather than their only cleanup.
- Evidence records session establishment/revocation outcome using non-secret
  handles or request ids only.
- A failed revocation is visible and actionable without printing token or
  accessor material.
- Bootstrap-file authentication is clearly separated from the reviewed
  steady-state provider and cannot become an implicit fallback.

## Strengthen verification, readiness, and evidence

```task
id: SECRETS-WP-0007-T06
status: done
priority: high
state_hub_task_id: "431bc91b-439e-595a-8a8a-7d4c82e68322"
```

Completed 2026-08-29. Acceptance is met: multi-field readiness, real unrelated
identity for KV denial, privileged attempt/terminal evidence, and `audit`.
Residuals belong elsewhere: exact-action authorization is T04 /
`SECRETS-WP-0008-T02`; OpenBao audit request-id correlation is hardening H5;
unrelated-identity *selection* for production remains an owner/ops concern,
not an engine mint.

Progress 2026-08-23. KV verification now checks every declared field by default,
with one path-level negative probe, and route readiness reads once and requires
all declared fields. Missing readiness evidence contains field names only. Tests
prove a missing sibling makes readiness false. Evidence delivery attempts now
receive append-only local success/failure receipts, and `audit` produces an
allowlisted per-lane summary of actions, results, decisions, session cleanup,
and delivery outcomes without echoing arbitrary detail. The real
unrelated-identity input is now enforced as a strict mode-0600 out-of-repo token
file; absence fails closed, and throwaway OpenBao integration deliberately adds
an overlapping unrelated read policy and proves the check fails. Production
identity selection/ownership, audit request correlation, queued/replayed State
Hub delivery, and exact-action authorization remain outstanding. All live
privileged handlers now share one attempt/terminal evidence guard: approval
rejection, backend/input exceptions, interruption, verification failure, and
success are recorded without exception prose. Tests prove decision and backend
failures stop before inappropriate backend work and exclude fake secret text.

State Hub evidence posts now carry stable idempotency, source-agent, and repo
headers. HTTP 202 edge-relay receipts are classified as `queued` and retain only
their non-secret outbox id in the companion evidence record and lane audit.
Outbox replay remains owned by State Hub/operator tooling.

The complete repository suite is re-run at each implementation slice, including
throwaway OpenBao integration coverage.

Make verification and routing truthful for multi-field and high-risk lanes:

- verify every declared field unless the exact approved subset is explicit;
- make route readiness require every required field, not only the first;
- replace the garbage-token negative check with a reviewed real unrelated
  identity/token or a capabilities-based denial proof that can detect policy
  overlap;
- record attempt, success/failure, cleanup, and authorization outcome for every
  privileged command;
- correlate engine evidence with OpenBao audit request ids where available;
- provide a non-secret lane audit summary over local/State Hub evidence.

Acceptance:

- Removing any required sibling field makes readiness false and identifies only
  the missing field name, never values.
- Negative verification fails when a deliberately overlapping unrelated policy
  is introduced in throwaway OpenBao.
- Evidence tests cover backend exceptions, child failures, State Hub delivery
  failures, and cleanup failures without leaking fake secret patterns.
- An audit/report command summarizes decisions, attempts, outcomes, and cleanup
  for a lane without becoming an authority or replacing OpenBao audit logs.
- The full test suite and throwaway OpenBao integration suite pass.

## Resume native production lane adoption

```task
id: SECRETS-WP-0007-T07
status: wait
priority: high
state_hub_task_id: "a0a1dd92-d703-5a95-b488-d895f34d5cf7"
```

Dependency: T01–T03, T05, T06 are done. T04 still waits on the durable
ActionAuthorization serving path. Also requires explicit per-lane approvals
and scoped production authority from railiance-platform.

Resume `SECRETS-WP-0006-T05` lane by lane. Apply only the exact native policy and
bounded AppRole, verify every field plus unrelated-consumer denial, confirm the
existing workload delivery remains healthy, exercise only an approved
value-safe child command, and record rollback/session-cleanup evidence.

After a lane passes, complete `SECRETS-WP-0006-T06` with ops-warden. Retire its
interim proxy only after both parties agree that native readiness evidence is
complete. Do not batch-retire all five lanes.

Acceptance per lane:

- Canonical action-bound approval is resolved and linked.
- Policy/AppRole server readback matches the reviewed exact-path plan.
- All declared fields pass presence/access checks; a real unrelated identity is
  denied; no value enters evidence.
- Issued delivery sessions are explicitly revoked and cleanup is evidenced.
- Existing ESO/OIDC delivery and application health remain good according to
  their owners.
- Rollback is rehearsed or capabilities-safe verified before proxy retirement.
- ops-warden routing reports native readiness and removes only that lane's
  interim proxy.

## Dependencies and coordination

- `SECRETS-WP-0006` owns the five-lane catalog admission and live cutover.
- railiance-platform owns production OpenBao authority, custody, and backend
  audit.
- State Hub/flex-auth own the canonical decision/authorization contract.
- key-cape/user-engine own human/service identity and claims.
- Workload owners own ESO/Kubernetes delivery and provider/application health.
- ops-warden owns credential routing and interim proxy disposition.

Any external contract change must be recorded as non-secret evidence and must
not transfer custody or authorization ownership into this repository.

## Exit criteria

- Current unsafe multi-field provisioning and misleading revoke behavior are
  contained and replaced.
- Raw values cannot enter subprocess argv or evidence through provisioning.
- Production approvals are canonical, exact-action, expiring, and genuinely
  dual-control-capable.
- Delivery tokens are explicitly revoked on every exit path.
- Readiness and verification cover all declared fields and meaningful unrelated
  identities.
- Lifecycle operations distinguish reversible access suspension from irreversible
  custody destruction.
- At least one of the five admitted production lanes completes the hardened
  native verification/cutover path; remaining lanes have explicit external
  blockers rather than engine-safety blockers.
- Service-mode design remains deferred until these guarantees are demonstrated.