secrets-engine/workplans/SECRETS-WP-0007-production-lifecycle-hardening.md

id	type	title	domain	repo	status	owner	topic_slug	created	updated	state_hub_workstream_id
SECRETS-WP-0007	workplan	Production-safe provisioning, authorization, and lifecycle hardening	infotech	secrets-engine	active	codex	custodian	2026-08-23	2026-09-02	68a39be1-bd9c-5133-ad64-e7bca892aaf3

SECRETS-WP-0007 - Production-safe provisioning, authorization, and lifecycle hardening

Goal

Close the highest-risk gaps between the proven CLI MVP and the production workflow described by INTENT.md, before expanding secrets-engine into a service or moving additional high-risk lanes from interim routing to native operation.

The source assessment is history/2026-08-23-scope-intent-capability-gap-assessment.md. The most relevant gaps are:

1. single-field bao kv put can replace siblings on a multi-field path and puts the value in the local subprocess argument vector;
2. KV revoke destroys metadata while leaving policy/AppRole access behind, and its dry-run does not match live behavior;
3. production approval checks are coarse status checks, accept local fixtures, and do not bind actor, action, lane, fields, expiry, or dual control;
4. AppRole delivery tokens are not explicitly revoked after exec/verification;
5. route and verification can attest only the first field and use a garbage token rather than a real unrelated identity for KV denial;
6. evidence is best-effort and incomplete on failure paths.

Scope and sequencing

This workplan hardens the existing CLI contract first. It covers KV write transport and merge semantics, lifecycle command semantics, decision binding, short-lived delivery sessions, verification/readiness, and evidence. It then provides the safe implementation gate for resuming SECRETS-WP-0006-T05/T06.

The first task is containment. No task may exercise live production values to develop or test a primitive. Use fake values and a throwaway OpenBao instance until the relevant behavior has passed unit and integration tests. Existing ESO, Kubernetes/OIDC delivery, provider rotation, and application health remain with their current owners.

Non-goals

• A service API, daemon, UI, scheduler, or remote multi-user protocol.
• Runtime tenant authentication, quotas, or a tenant control plane.
• A second custody backend or backend-portability project.
• Provider-specific rotation automation.
• Replacing flex-auth, State Hub decisions, key-cape/user-engine identity, ops-warden routing, ops-bridge transport, or OpenBao enforcement.
• Unparking the standalone warden-sign AppRole as a break-glass path.

Service mode remains deferred until the exit criteria here are satisfied; it must not expose the current bootstrap and lifecycle shortcuts as an API.

Safety invariants

• No raw value in Git, State Hub, chat, prompts, workplans, evidence, normal logs, environment diagnostics, or a subprocess argument vector.
• No write to a shared multi-field path unless sibling preservation is proven.
• No destructive metadata deletion through a command presented as ordinary revoke/deactivate.
• Production actions require canonical, exact-action approval. Offline fixtures cannot authorize steady-state production mutations or delivery.
• Existing workload delivery stays in place until separately approved owner coordination says otherwise.
• Tests use fake values and throwaway OpenBao only; live checks are metadata/capability-safe and approval-gated.

Contain unsafe lifecycle surfaces

id: SECRETS-WP-0007-T01
status: done
priority: high
state_hub_task_id: "5ddba36d-bc86-5670-baf1-50c872e6577e"

Completed 2026-08-23. During containment, existing multi-field provisioning failed before file read or backend access. Ordinary revoke builds one lifecycle plan used by both dry-run and live execution, deletes only engine-managed native AppRole/policy objects, and explicitly preserves KV custody, external delivery auth, and workload delivery. No destructive KV command remains on the ordinary CLI. Focused lifecycle, admission, auth-capability, and throwaway OpenBao integration tests pass. The temporary multi-field rejection was then superseded by the merge-safe primitive completed in T02.

Add immediate fail-closed guards before implementing replacement primitives:

• reject provision on an existing multi-field lane;
• reject or rename the current destructive KV revoke path so it cannot be mistaken for soft deactivation;
• make KV revoke dry-run render exactly the operations the live path would run;
• document that SECRETS-WP-0006 may apply/verify metadata but must not provision or revoke shared values through the current commands.

Acceptance:

• A regression test proves no backend call occurs for multi-field existing-lane provisioning.
• A regression test proves dry-run and live operation plans contain the same KV, AppRole, and policy targets.
• The destructive action requires an explicit name/flag and a separately approved action; ordinary revoke cannot destroy KV metadata.
• CLI/help/runbook language no longer describes metadata deletion as reversible deactivation.

Implement merge-safe, non-argv provisioning

id: SECRETS-WP-0007-T02
status: done
priority: high
state_hub_task_id: "ddf3ed54-9cbc-5e18-b15a-5da14cda5d9b"

Completed 2026-08-23. Replaced raw field=value argv writes with a CAS-aware backend primitive: new paths use CAS-zero create; existing paths use server-side HTTP patch with the observed metadata version. Values travel through a mode-0600 temporary JSON reference removed in finally, and never appear in argv. Unit tests inspect the full request/cleanup; throwaway OpenBao integration proves sibling preservation and stale-CAS rejection. Stage policies now include the narrow KV patch capability needed by this operation.

Replace the raw field=value subprocess call with a backend write primitive that never places the value in argv and safely updates one or more declared fields without removing unmentioned siblings. The design may use OpenBao CAS, a merge/patch operation, an input file reference with strict cleanup, or direct request input, but it must remain isolated behind OpenBaoClient and preserve the catalog field allowlist.

Acceptance:

• Unit tests inspect the complete subprocess request and prove the fake value is absent from argv, environment, output, exceptions, and evidence.
• Throwaway OpenBao integration proves updating one field preserves every sibling field and fails closed on a CAS conflict.
• Multi-field input is atomic from the caller's perspective; a partial failure cannot silently leave an unreported mixed version.
• Source and generated inputs retain mode/path guards and guaranteed temporary cleanup.
• Production generation remains forbidden, and high-risk production provisioning is still decision- and dual-control-gated.

Split suspend, deactivate, and destroy semantics

id: SECRETS-WP-0007-T03
status: done
priority: high
state_hub_task_id: "bb6073e1-c952-5357-b707-5574cef50f2f"

Completed 2026-08-29. Suspend, deactivate, and destroy share one plan object for dry-run and live execution. Ordinary revoke aliases native deactivation and preserves KV custody plus externally managed workload delivery. Live destroy stays fail-closed until T04's exact-action contract; that is the designed containment, not unfinished T03 work. Issued delivery sessions self-revoke in T05.

Residual from this task, now implemented: secrets-engine session revoke --accessor-file (hardening H4a). Lease-id revoke remains outstanding.

Progress 2026-08-23. Added one lifecycle plan model shared by dry-run and live execution. Ordinary revoke safely aliases native deactivation; suspend and deactivate preserve KV custody and externally managed workload delivery. Destroy has an unmistakable dry-run plan and exact-id confirmation, but its live handler is deliberately fail-closed before coarse lane approval or backend access until T04 supplies distinct action authorization. Scoped issued-session self-revocation is implemented in T05; a general known-accessor operator command is session revoke --accessor-file (H4a).

Replace the overloaded revoke behavior with explicit lifecycle operations and plans. Define at least:

• suspend delivery: stop minting new native delivery sessions while preserving custody and a documented rollback;
• deactivate native access: remove/disable the lane AppRole and consumer policy without deleting the KV value;
• revoke issued sessions: revoke known tokens/leases by non-secret accessor where the backend supports it;
• destroy custody metadata: irreversibly delete KV metadata/versions only through an unmistakable, separately approved action.

Preserve compatibility only where it cannot mislead. A legacy revoke alias may map to a safe non-destructive action; it must never silently retain the current destructive meaning.

Acceptance:

• Each operation has a guarded dry-run plan, exact targets, distinct approval action, non-secret evidence, and documented rollback/irreversibility.
• Dry-run and live use the same plan object rather than separate handwritten descriptions.
• Integration tests prove suspend/deactivate preserve KV data and destroy does not run without its explicit approval and confirmation contract.
• Auth-capability and KV lanes have coherent but kind-appropriate behavior.
• Existing workload delivery is reported as an external dependency and is not deleted by native lifecycle commands.

Bind approvals to exact production actions

id: SECRETS-WP-0007-T04
status: wait
priority: high
state_hub_task_id: "4b58edec-c705-55e5-9ece-362e1ff13079"

Wait 2026-08-29. The consumer validator and production fail-closed gate are shipped. What remains is not local engine work: State Hub / access-engine must serve the durable ActionAuthorization object. Paired with SECRETS-WP-0008-T02. Do not reopen live production on hub status or local fixtures.

Progress 2026-09-02. GH-DEC-2026-003 consume-before-OpenBao is now a shared production gate: _require_lane_approval CAS-consumes through approval-engine before any OpenBao client is resolved. Tests prove 409, unreachable, and missing-binding paths never call OpenBao. Live production stays fail-closed until the durable object is served; this task remains wait.

Progress 2026-08-23. flex-auth revision c473f19 now supplies execute-time structured binding and the proposed durable ActionAuthorization schema. The engine builds and fail-closed validates the exact secrets-engine profile, validity, State Hub authority, decision digest/binding, accepted policy revision, and a caller-selected minimum distinct-approver threshold. Every live production handler now carries its action into the shared gate and fails before OpenBao while State Hub lacks the durable endpoint. Legacy local decisions are reachable for prod-labeled lanes only with an explicit unsafe-demo switch, disabled Hub URL, and loopback OpenBao. The throwaway scripts declare that exception. Live destroy remains disabled independently. State Hub endpoint and authenticated approval storage are still outstanding. flex-auth corrected its example digest and added a complete binding regression assertion in d402408.

Define and enforce the decision contract needed by production commands. A resolved approval must bind at least:

• canonical decision UUID and non-superseded status;
• catalog id and stage;
• action (apply, provision, verify, exec, handoff, suspend, deactivate, destroy, or later rotate);
• exact field set or policy/auth targets where applicable;
• requesting actor/purpose and expiry or bounded execution window;
• required approval count and distinct approvers for dual-control.

Coordinate the external object shape with State Hub/flex-auth rather than inventing authorization ownership in this repo. Keep local fixtures for throwaway demos and non-production tests only; require an explicit unsafe-demo switch if they remain reachable at all.

Acceptance:

• Every privileged production handler passes an action-specific request into one shared authorization function.
• Tests reject wrong lane, stage, action, field set, actor/purpose, expiry, superseded status, insufficient approvals, duplicate approvers, and local production fixtures before any OpenBao call.
• approval.model: dual-control has enforced semantics rather than being a label.
• Catalogs use resolvable canonical decision UUIDs for live actions; legacy CCR references remain provenance, not executable authorization.
• State Hub outage fails production actions closed while dry-run/catalog inspection remains usable.

Make delivery sessions short-lived and explicitly closed

id: SECRETS-WP-0007-T05
status: done
priority: high
state_hub_task_id: "36bcd64d-8b27-542d-b8da-ede1a4d3c22f"

Completed 2026-08-29. Delivery AppRole sessions self-revoke in finally on exec and both verification kinds. Bootstrap is a named provider with no implicit fallback from the KeyCape scaffold. Wiring that scaffold as OpenBao JWT login is SECRETS-WP-0008-T06 / hardening H1, blocked on the platform-owned role — not remaining T05 engine work.

Progress 2026-08-23. AppRole login material now travels through strict temporary JSON input rather than argv. Exec and both verification kinds use a scoped session that self-revokes in finally, drops the in-memory token even when cleanup fails, and exposes only an accessor fingerprint plus cleanup booleans as evidence. Tests cover success and exception cleanup, idempotence, failed revoke visibility, and value/credential exclusion. Provider review requests were sent to railiance-platform (8f910aff-3a94-43c6-8805-eb9276e46fc0) and key-cape (0627ca55-115e-43dc-b2f6-3195be3bc90d); steady-state engine authentication remains external-contract work.

KeyCape has accepted the secrets-engine-openbao service contract. An explicit consumer scaffold now performs HTTPS client-credentials exchange, strict out-of-repo client-secret input, exact claim/assurance/lifetime preflight, three-minute renewal classification, and rejection of ID/refresh tokens. It has no bootstrap/operator/AppRole fallback and is deliberately not wired into the CLI until railiance-platform materializes the exact OpenBao JWT mount and role.

Refactor AppRole login into a scoped session object that retains only the minimum non-secret lifecycle handle needed to revoke the issued token in a finally path. Apply it consistently to exec, KV verification, auth-capability verification, and any future file/wrapped delivery.

Define the steady-state OpenBao authentication interface separately from the delivery AppRole. Bootstrap token files remain an explicit break-glass/setup provider; add a bounded OIDC/service-auth provider only with railiance-platform and identity-owner review.

Acceptance:

• Success, child non-zero exit, setup failure, verification failure, SIGINT, and SIGTERM tests all attempt token revocation and clean temporary material.
• Secret IDs remain single-use where configured; tokens retain TTL/use limits as backstops rather than their only cleanup.
• Evidence records session establishment/revocation outcome using non-secret handles or request ids only.
• A failed revocation is visible and actionable without printing token or accessor material.
• Bootstrap-file authentication is clearly separated from the reviewed steady-state provider and cannot become an implicit fallback.

Strengthen verification, readiness, and evidence

id: SECRETS-WP-0007-T06
status: done
priority: high
state_hub_task_id: "431bc91b-439e-595a-8a8a-7d4c82e68322"

Completed 2026-08-29. Acceptance is met: multi-field readiness, real unrelated identity for KV denial, privileged attempt/terminal evidence, and audit. Residuals belong elsewhere: exact-action authorization is T04 / SECRETS-WP-0008-T02; OpenBao audit request-id correlation is hardening H5; unrelated-identity selection for production remains an owner/ops concern, not an engine mint.

Progress 2026-08-23. KV verification now checks every declared field by default, with one path-level negative probe, and route readiness reads once and requires all declared fields. Missing readiness evidence contains field names only. Tests prove a missing sibling makes readiness false. Evidence delivery attempts now receive append-only local success/failure receipts, and audit produces an allowlisted per-lane summary of actions, results, decisions, session cleanup, and delivery outcomes without echoing arbitrary detail. The real unrelated-identity input is now enforced as a strict mode-0600 out-of-repo token file; absence fails closed, and throwaway OpenBao integration deliberately adds an overlapping unrelated read policy and proves the check fails. Production identity selection/ownership, audit request correlation, queued/replayed State Hub delivery, and exact-action authorization remain outstanding. All live privileged handlers now share one attempt/terminal evidence guard: approval rejection, backend/input exceptions, interruption, verification failure, and success are recorded without exception prose. Tests prove decision and backend failures stop before inappropriate backend work and exclude fake secret text.

State Hub evidence posts now carry stable idempotency, source-agent, and repo headers. HTTP 202 edge-relay receipts are classified as queued and retain only their non-secret outbox id in the companion evidence record and lane audit. Outbox replay remains owned by State Hub/operator tooling.

The complete repository suite is re-run at each implementation slice, including throwaway OpenBao integration coverage.

Make verification and routing truthful for multi-field and high-risk lanes:

• verify every declared field unless the exact approved subset is explicit;
• make route readiness require every required field, not only the first;
• replace the garbage-token negative check with a reviewed real unrelated identity/token or a capabilities-based denial proof that can detect policy overlap;
• record attempt, success/failure, cleanup, and authorization outcome for every privileged command;
• correlate engine evidence with OpenBao audit request ids where available;
• provide a non-secret lane audit summary over local/State Hub evidence.

Acceptance:

• Removing any required sibling field makes readiness false and identifies only the missing field name, never values.
• Negative verification fails when a deliberately overlapping unrelated policy is introduced in throwaway OpenBao.
• Evidence tests cover backend exceptions, child failures, State Hub delivery failures, and cleanup failures without leaking fake secret patterns.
• An audit/report command summarizes decisions, attempts, outcomes, and cleanup for a lane without becoming an authority or replacing OpenBao audit logs.
• The full test suite and throwaway OpenBao integration suite pass.

Resume native production lane adoption

id: SECRETS-WP-0007-T07
status: wait
priority: high
state_hub_task_id: "a0a1dd92-d703-5a95-b488-d895f34d5cf7"

Dependency: T01–T03, T05, T06 are done. T04 still waits on the durable ActionAuthorization serving path. Also requires explicit per-lane approvals and scoped production authority from railiance-platform.

Resume SECRETS-WP-0006-T05 lane by lane. Apply only the exact native policy and bounded AppRole, verify every field plus unrelated-consumer denial, confirm the existing workload delivery remains healthy, exercise only an approved value-safe child command, and record rollback/session-cleanup evidence.

After a lane passes, complete SECRETS-WP-0006-T06 with ops-warden. Retire its interim proxy only after both parties agree that native readiness evidence is complete. Do not batch-retire all five lanes.

Acceptance per lane:

• Canonical action-bound approval is resolved and linked.
• Policy/AppRole server readback matches the reviewed exact-path plan.
• All declared fields pass presence/access checks; a real unrelated identity is denied; no value enters evidence.
• Issued delivery sessions are explicitly revoked and cleanup is evidenced.
• Existing ESO/OIDC delivery and application health remain good according to their owners.
• Rollback is rehearsed or capabilities-safe verified before proxy retirement.
• ops-warden routing reports native readiness and removes only that lane's interim proxy.

Dependencies and coordination

• SECRETS-WP-0006 owns the five-lane catalog admission and live cutover.
• railiance-platform owns production OpenBao authority, custody, and backend audit.
• State Hub/flex-auth own the canonical decision/authorization contract.
• key-cape/user-engine own human/service identity and claims.
• Workload owners own ESO/Kubernetes delivery and provider/application health.
• ops-warden owns credential routing and interim proxy disposition.

Any external contract change must be recorded as non-secret evidence and must not transfer custody or authorization ownership into this repository.

Exit criteria

• Current unsafe multi-field provisioning and misleading revoke behavior are contained and replaced.
• Raw values cannot enter subprocess argv or evidence through provisioning.
• Production approvals are canonical, exact-action, expiring, and genuinely dual-control-capable.
• Delivery tokens are explicitly revoked on every exit path.
• Readiness and verification cover all declared fields and meaningful unrelated identities.
• Lifecycle operations distinguish reversible access suspension from irreversible custody destruction.
• At least one of the five admitted production lanes completes the hardened native verification/cutover path; remaining lanes have explicit external blockers rather than engine-safety blockers.
• Service-mode design remains deferred until these guarantees are demonstrated.