secrets-engine/workplans/SECRETS-WP-0007-production-lifecycle-hardening.md

---
id: SECRETS-WP-0007
type: workplan
title: "Production-safe provisioning, authorization, and lifecycle hardening"
domain: infotech
repo: secrets-engine
status: active
owner: codex
topic_slug: custodian
created: "2026-08-23"
updated: "2026-08-23"
---

# SECRETS-WP-0007 - Production-safe provisioning, authorization, and lifecycle hardening

## Goal

Close the highest-risk gaps between the proven CLI MVP and the production
workflow described by `INTENT.md`, before expanding secrets-engine into a
service or moving additional high-risk lanes from interim routing to native
operation.

The source assessment is
`history/2026-08-23-scope-intent-capability-gap-assessment.md`. The most relevant
gaps are:

1. single-field `bao kv put` can replace siblings on a multi-field path and puts
   the value in the local subprocess argument vector;
2. KV `revoke` destroys metadata while leaving policy/AppRole access behind,
   and its dry-run does not match live behavior;
3. production approval checks are coarse status checks, accept local fixtures,
   and do not bind actor, action, lane, fields, expiry, or dual control;
4. AppRole delivery tokens are not explicitly revoked after exec/verification;
5. route and verification can attest only the first field and use a garbage
   token rather than a real unrelated identity for KV denial;
6. evidence is best-effort and incomplete on failure paths.

## Scope and sequencing

This workplan hardens the existing CLI contract first. It covers KV write
transport and merge semantics, lifecycle command semantics, decision binding,
short-lived delivery sessions, verification/readiness, and evidence. It then
provides the safe implementation gate for resuming `SECRETS-WP-0006-T05/T06`.

The first task is containment. No task may exercise live production values to
develop or test a primitive. Use fake values and a throwaway OpenBao instance
until the relevant behavior has passed unit and integration tests. Existing ESO,
Kubernetes/OIDC delivery, provider rotation, and application health remain with
their current owners.

## Non-goals

- A service API, daemon, UI, scheduler, or remote multi-user protocol.
- Runtime tenant authentication, quotas, or a tenant control plane.
- A second custody backend or backend-portability project.
- Provider-specific rotation automation.
- Replacing flex-auth, State Hub decisions, key-cape/user-engine identity,
  ops-warden routing, ops-bridge transport, or OpenBao enforcement.
- Unparking the standalone `warden-sign` AppRole as a break-glass path.

Service mode remains deferred until the exit criteria here are satisfied; it
must not expose the current bootstrap and lifecycle shortcuts as an API.

## Safety invariants

- No raw value in Git, State Hub, chat, prompts, workplans, evidence, normal
  logs, environment diagnostics, or a subprocess argument vector.
- No write to a shared multi-field path unless sibling preservation is proven.
- No destructive metadata deletion through a command presented as ordinary
  revoke/deactivate.
- Production actions require canonical, exact-action approval. Offline fixtures
  cannot authorize steady-state production mutations or delivery.
- Existing workload delivery stays in place until separately approved owner
  coordination says otherwise.
- Tests use fake values and throwaway OpenBao only; live checks are
  metadata/capability-safe and approval-gated.

## Contain unsafe lifecycle surfaces

```task
id: SECRETS-WP-0007-T01
status: done
priority: high
```

Completed 2026-08-23. During containment, existing multi-field provisioning
failed before file read or backend access. Ordinary `revoke` builds one
lifecycle plan used by both
dry-run and live execution, deletes only engine-managed native AppRole/policy
objects, and explicitly preserves KV custody, external delivery auth, and
workload delivery. No destructive KV command remains on the ordinary CLI.
Focused lifecycle, admission, auth-capability, and throwaway OpenBao integration
tests pass. The temporary multi-field rejection was then superseded by the
merge-safe primitive completed in T02.

Add immediate fail-closed guards before implementing replacement primitives:

- reject `provision` on an existing multi-field lane;
- reject or rename the current destructive KV `revoke` path so it cannot be
  mistaken for soft deactivation;
- make KV revoke dry-run render exactly the operations the live path would run;
- document that `SECRETS-WP-0006` may apply/verify metadata but must not provision
  or revoke shared values through the current commands.

Acceptance:

- A regression test proves no backend call occurs for multi-field existing-lane
  provisioning.
- A regression test proves dry-run and live operation plans contain the same KV,
  AppRole, and policy targets.
- The destructive action requires an explicit name/flag and a separately
  approved action; ordinary revoke cannot destroy KV metadata.
- CLI/help/runbook language no longer describes metadata deletion as reversible
  deactivation.

## Implement merge-safe, non-argv provisioning

```task
id: SECRETS-WP-0007-T02
status: done
priority: high
```

Completed 2026-08-23. Replaced raw `field=value` argv writes with a CAS-aware
backend primitive: new paths use CAS-zero create; existing paths use server-side
HTTP patch with the observed metadata version. Values travel through a
mode-0600 temporary JSON reference removed in `finally`, and never appear in
argv. Unit tests inspect the full request/cleanup; throwaway OpenBao integration
proves sibling preservation and stale-CAS rejection. Stage policies now include
the narrow KV `patch` capability needed by this operation.

Replace the raw `field=value` subprocess call with a backend write primitive
that never places the value in argv and safely updates one or more declared
fields without removing unmentioned siblings. The design may use OpenBao CAS,
a merge/patch operation, an input file reference with strict cleanup, or direct
request input, but it must remain isolated behind `OpenBaoClient` and preserve
the catalog field allowlist.

Acceptance:

- Unit tests inspect the complete subprocess request and prove the fake value is
  absent from argv, environment, output, exceptions, and evidence.
- Throwaway OpenBao integration proves updating one field preserves every
  sibling field and fails closed on a CAS conflict.
- Multi-field input is atomic from the caller's perspective; a partial failure
  cannot silently leave an unreported mixed version.
- Source and generated inputs retain mode/path guards and guaranteed temporary
  cleanup.
- Production generation remains forbidden, and high-risk production
  provisioning is still decision- and dual-control-gated.

## Split suspend, deactivate, and destroy semantics

```task
id: SECRETS-WP-0007-T03
status: progress
priority: high
```

Progress 2026-08-23. Added one lifecycle plan model shared by dry-run and live
execution. Ordinary `revoke` safely aliases native deactivation; suspend and
deactivate preserve KV custody and externally managed workload delivery.
Destroy has an unmistakable dry-run plan and exact-id confirmation, but its live
handler is deliberately fail-closed before coarse lane approval or backend
access until T04 supplies distinct action authorization. Scoped issued-session
self-revocation is implemented in T05; a general known-accessor operator command
remains outstanding.

Replace the overloaded `revoke` behavior with explicit lifecycle operations and
plans. Define at least:

- **suspend delivery**: stop minting new native delivery sessions while
  preserving custody and a documented rollback;
- **deactivate native access**: remove/disable the lane AppRole and consumer
  policy without deleting the KV value;
- **revoke issued sessions**: revoke known tokens/leases by non-secret accessor
  where the backend supports it;
- **destroy custody metadata**: irreversibly delete KV metadata/versions only
  through an unmistakable, separately approved action.

Preserve compatibility only where it cannot mislead. A legacy `revoke` alias
may map to a safe non-destructive action; it must never silently retain the
current destructive meaning.

Acceptance:

- Each operation has a guarded dry-run plan, exact targets, distinct approval
  action, non-secret evidence, and documented rollback/irreversibility.
- Dry-run and live use the same plan object rather than separate handwritten
  descriptions.
- Integration tests prove suspend/deactivate preserve KV data and destroy does
  not run without its explicit approval and confirmation contract.
- Auth-capability and KV lanes have coherent but kind-appropriate behavior.
- Existing workload delivery is reported as an external dependency and is not
  deleted by native lifecycle commands.

## Bind approvals to exact production actions

```task
id: SECRETS-WP-0007-T04
status: wait
priority: high
```

Waiting 2026-08-23 on the canonical external contract rather than parsing
decision prose or inventing authorization ownership locally. Contract requests
were sent to State Hub (`24663321-0263-43fe-8d48-e9c7e06d7bb9`) and flex-auth
(`ef8ff95d-6b4e-46f8-b1a9-497d06cf7c9a`). Until resolved, local fixtures cannot
unlock the new destroy path and that live operation remains disabled.

Define and enforce the decision contract needed by production commands. A
resolved approval must bind at least:

- canonical decision UUID and non-superseded status;
- catalog id and stage;
- action (`apply`, `provision`, `verify`, `exec`, `handoff`, `suspend`,
  `deactivate`, `destroy`, or later `rotate`);
- exact field set or policy/auth targets where applicable;
- requesting actor/purpose and expiry or bounded execution window;
- required approval count and distinct approvers for `dual-control`.

Coordinate the external object shape with State Hub/flex-auth rather than
inventing authorization ownership in this repo. Keep local fixtures for
throwaway demos and non-production tests only; require an explicit unsafe-demo
switch if they remain reachable at all.

Acceptance:

- Every privileged production handler passes an action-specific request into
  one shared authorization function.
- Tests reject wrong lane, stage, action, field set, actor/purpose, expiry,
  superseded status, insufficient approvals, duplicate approvers, and local
  production fixtures before any OpenBao call.
- `approval.model: dual-control` has enforced semantics rather than being a
  label.
- Catalogs use resolvable canonical decision UUIDs for live actions; legacy CCR
  references remain provenance, not executable authorization.
- State Hub outage fails production actions closed while dry-run/catalog
  inspection remains usable.

## Make delivery sessions short-lived and explicitly closed

```task
id: SECRETS-WP-0007-T05
status: progress
priority: high
```

Progress 2026-08-23. AppRole login material now travels through strict temporary
JSON input rather than argv. Exec and both verification kinds use a scoped
session that self-revokes in `finally`, drops the in-memory token even when
cleanup fails, and exposes only an accessor fingerprint plus cleanup booleans as
evidence. Tests cover success and exception cleanup, idempotence, failed revoke
visibility, and value/credential exclusion. Provider review requests were sent
to railiance-platform (`8f910aff-3a94-43c6-8805-eb9276e46fc0`) and key-cape
(`0627ca55-115e-43dc-b2f6-3195be3bc90d`); steady-state engine authentication
remains external-contract work.

Refactor AppRole login into a scoped session object that retains only the
minimum non-secret lifecycle handle needed to revoke the issued token in a
`finally` path. Apply it consistently to exec, KV verification, auth-capability
verification, and any future file/wrapped delivery.

Define the steady-state OpenBao authentication interface separately from the
delivery AppRole. Bootstrap token files remain an explicit break-glass/setup
provider; add a bounded OIDC/service-auth provider only with railiance-platform
and identity-owner review.

Acceptance:

- Success, child non-zero exit, setup failure, verification failure, SIGINT, and
  SIGTERM tests all attempt token revocation and clean temporary material.
- Secret IDs remain single-use where configured; tokens retain TTL/use limits
  as backstops rather than their only cleanup.
- Evidence records session establishment/revocation outcome using non-secret
  handles or request ids only.
- A failed revocation is visible and actionable without printing token or
  accessor material.
- Bootstrap-file authentication is clearly separated from the reviewed
  steady-state provider and cannot become an implicit fallback.

## Strengthen verification, readiness, and evidence

```task
id: SECRETS-WP-0007-T06
status: progress
priority: high
```

Progress 2026-08-23. KV verification now checks every declared field by default,
with one path-level negative probe, and route readiness reads once and requires
all declared fields. Missing readiness evidence contains field names only. Tests
prove a missing sibling makes readiness false. Evidence delivery attempts now
receive append-only local success/failure receipts, and `audit` produces an
allowlisted per-lane summary of actions, results, decisions, session cleanup,
and delivery outcomes without echoing arbitrary detail. The real
unrelated-identity input is now enforced as a strict mode-0600 out-of-repo token
file; absence fails closed, and throwaway OpenBao integration deliberately adds
an overlapping unrelated read policy and proves the check fails. Production
identity selection/ownership, audit request correlation, queued/replayed State
Hub delivery, and complete privileged failure-path evidence remain outstanding.

The complete repository suite passes with 111 tests after these changes,
including throwaway OpenBao integration coverage.

Make verification and routing truthful for multi-field and high-risk lanes:

- verify every declared field unless the exact approved subset is explicit;
- make route readiness require every required field, not only the first;
- replace the garbage-token negative check with a reviewed real unrelated
  identity/token or a capabilities-based denial proof that can detect policy
  overlap;
- record attempt, success/failure, cleanup, and authorization outcome for every
  privileged command;
- correlate engine evidence with OpenBao audit request ids where available;
- provide a non-secret lane audit summary over local/State Hub evidence.

Acceptance:

- Removing any required sibling field makes readiness false and identifies only
  the missing field name, never values.
- Negative verification fails when a deliberately overlapping unrelated policy
  is introduced in throwaway OpenBao.
- Evidence tests cover backend exceptions, child failures, State Hub delivery
  failures, and cleanup failures without leaking fake secret patterns.
- An audit/report command summarizes decisions, attempts, outcomes, and cleanup
  for a lane without becoming an authority or replacing OpenBao audit logs.
- The full test suite and throwaway OpenBao integration suite pass.

## Resume native production lane adoption

```task
id: SECRETS-WP-0007-T07
status: wait
priority: high
```

Dependency: T01-T06 complete, then explicit per-lane approvals and scoped
production authority from railiance-platform.

Resume `SECRETS-WP-0006-T05` lane by lane. Apply only the exact native policy and
bounded AppRole, verify every field plus unrelated-consumer denial, confirm the
existing workload delivery remains healthy, exercise only an approved
value-safe child command, and record rollback/session-cleanup evidence.

After a lane passes, complete `SECRETS-WP-0006-T06` with ops-warden. Retire its
interim proxy only after both parties agree that native readiness evidence is
complete. Do not batch-retire all five lanes.

Acceptance per lane:

- Canonical action-bound approval is resolved and linked.
- Policy/AppRole server readback matches the reviewed exact-path plan.
- All declared fields pass presence/access checks; a real unrelated identity is
  denied; no value enters evidence.
- Issued delivery sessions are explicitly revoked and cleanup is evidenced.
- Existing ESO/OIDC delivery and application health remain good according to
  their owners.
- Rollback is rehearsed or capabilities-safe verified before proxy retirement.
- ops-warden routing reports native readiness and removes only that lane's
  interim proxy.

## Dependencies and coordination

- `SECRETS-WP-0006` owns the five-lane catalog admission and live cutover.
- railiance-platform owns production OpenBao authority, custody, and backend
  audit.
- State Hub/flex-auth own the canonical decision/authorization contract.
- key-cape/user-engine own human/service identity and claims.
- Workload owners own ESO/Kubernetes delivery and provider/application health.
- ops-warden owns credential routing and interim proxy disposition.

Any external contract change must be recorded as non-secret evidence and must
not transfer custody or authorization ownership into this repository.

## Exit criteria

- Current unsafe multi-field provisioning and misleading revoke behavior are
  contained and replaced.
- Raw values cannot enter subprocess argv or evidence through provisioning.
- Production approvals are canonical, exact-action, expiring, and genuinely
  dual-control-capable.
- Delivery tokens are explicitly revoked on every exit path.
- Readiness and verification cover all declared fields and meaningful unrelated
  identities.
- Lifecycle operations distinguish reversible access suspension from irreversible
  custody destruction.
- At least one of the five admitted production lanes completes the hardened
  native verification/cutover path; remaining lanes have explicit external
  blockers rather than engine-safety blockers.
- Service-mode design remains deferred until these guarantees are demonstrated.