railiance-platform/docs/credential-change-approval.md

Credential Change Approval Workflow

This document sketches the operator workflow we want for it-sec and credential changes. The goal is to remove raw OpenBao command authoring from routine human operation while preserving explicit human approval, auditability, and safe handling of secret values.

Problem

The current workflow still asks operators to translate a reviewed intent into OpenBao commands by hand:

• create or update policies;
• create auth roles with the right bound claims;
• create or rotate secret paths and fields;
• verify positive and negative access;
• tell ops-warden or another access front door when a lane may become active.

That is inefficient and easy to get wrong. It is also hard to review because the actual unit of work is spread across chat, workplans, OpenBao UI screens, State Hub notes, and shell commands.

Direction

Treat OpenBao as the enforcement and audit engine, not the primary review UI. Add a small approval control plane in front of it:

1. an agent or CLI creates a structured, non-secret credential change request;
2. humans review the rendered proposal, risk notes, generated OpenBao plan, and verification plan;
3. a human approves or denies with a comment;
4. only approved requests can be applied by an operator-controlled helper;
5. the helper records non-secret evidence and marks the request active, rejected, deactivated, rotated, or compromised.

This can be implemented with repo files, State Hub, and CLI/chat integration first. An OpenBao UI extension can come later if the workflow proves itself.

Core Object

The canonical unit is a credential change request, abbreviated CCR.

The CCR must be non-secret. It may contain:

• stable request id and title;
• requester, reviewer, approver, and applier identities;
• target domain, tenant, workload, environment, and purpose;
• OpenBao mount, path, field names, policy names, and auth role names;
• exact non-secret policy HCL or generated policy references;
• proposed auth bindings and bound claims;
• delivery surface such as ops-warden, External Secrets, CSI, or direct caller fetch;
• machine-readable front-door readiness, including readiness and resolvable;
• risk classification and approval requirements;
• generated apply plan;
• verification plan;
• rollback, deactivate, rotate, and compromise response plan;
• comments, approvals, denials, and timestamps;
• non-secret OpenBao audit request ids or timestamps after execution.

It must not contain:

• secret values;
• wrapped token values;
• root, platform-admin, or issuer tokens;
• passwords, API keys, private keys, OTP seeds, unseal shares, or recovery codes;
• command output that includes secret values.

State Machine

Suggested states:

draft
in_flight
proposed
needs_changes
approved
denied
apply_pending
applied
verified
active
deactivated
rotated
compromised
superseded
cancelled

in_flight is the only nonterminal state that may explicitly omit completion-only fields. It must declare every omission in in_flight.missing_fields, name the owner and blocking reason, and remain non-resolvable. The validator still checks every other field. This is not an applyable state and must never be used to hide a malformed active lane.

An incomplete request withdrawn by its owner may instead be cancelled, retaining its in_flight declaration as history. This requires cancellation.owner, cancellation.reason and a non-secret cancellation.source_ref, with front-door readiness disabled and resolvable: false. Only the same declared omissions are allowed; all other validation remains. Cancellation cannot authorize apply or resolve a reader. Moving back to an approval/apply state requires completing the binding; keeping the historical declaration does not bypass that requirement.

Only approved requests may be applied. Only verified requests may become active.

Emergency break-glass work may create a request after the fact, but it must be marked as break-glass, reviewed retrospectively, and linked to audit evidence.

Review Surface

A reviewer should see a concise rendered proposal:

Request: whynot-design npm publish token lane
Type: workload-kv-read
Mount/path/field:
  platform/workloads/coulomb/whynot-design/npm-publish
  NPM_AUTH_TOKEN
Policy:
  workload-kv-read-whynot-design-npm-publish
Auth binding:
  netkingdom OIDC role whynot-design-workload-kv-read
  bound claim: groups includes whynot-design
Access front door:
  ops-warden whynot-design-npm-publish
  readiness: template
  resolvable: false
Risk:
  grants read access to npm publish credential
Checks:
  positive whynot fetch, negative non-whynot denial, OpenBao audit evidence
Decision:
  approve | deny | needs changes
Comment:
  free text

The reviewer should not need to know the exact bao write syntax. They should be able to discuss the proposal in chat, request changes, and then make a formal decision.

Minimal Implementation

Version 1 should be boring:

• store CCR files under credential-change-requests/;
• validate CCR schema offline;
• render a human-readable review summary;
• generate OpenBao apply plans from approved CCRs;
• require an approval record before apply;
• apply only non-secret policy/auth/path metadata;
• prompt or delegate separately for secret value entry;
• record non-secret evidence in State Hub.

When the schema adds or strengthens a required field, the same change must include a migration pass over every existing CCR. Active declarations must describe the live authentication path; incomplete requests must move to the explicit in_flight state rather than relying on a filename exception or a loosened repository-wide assertion.

The first implemented CLI slice is:

make credential-change-validate
make credential-change-render CREDENTIAL_CHANGE=CCR-2026-0001
make credential-change-plan CREDENTIAL_CHANGE=CCR-2026-0001
make credential-change-status CREDENTIAL_CHANGE=CCR-2026-0001
make credential-change-status-json CREDENTIAL_CHANGE=CCR-2026-0001
scripts/credential-change.py confirm-binding CCR-2026-0001 --reviewer <name> --comment "..."
scripts/credential-change.py approve CCR-2026-0001 --reviewer <name> --comment "..."
scripts/credential-change.py deny CCR-2026-0001 --reviewer <name> --comment "..."
scripts/credential-change.py needs-changes CCR-2026-0001 --reviewer <name> --comment "..."
make credential-change-sync-decision CREDENTIAL_CHANGE=CCR-2026-0001
make credential-change-apply-plan CREDENTIAL_CHANGE=CCR-2026-0001
make credential-change-operator-commands CREDENTIAL_CHANGE=CCR-2026-0001
make credential-change-runbook CREDENTIAL_CHANGE=CCR-2026-0001
scripts/credential-change.py runbook CCR-2026-0001 --execute-metadata --actor <operator> --confirm "APPLY CCR-2026-0001"
scripts/credential-change.py record-evidence CCR-2026-0001 --actor <operator> --kind positive_verification --result passed --detail "<non-secret audit reference>" --record-state-hub
make credential-change-lifecycle-plan CREDENTIAL_CHANGE=CCR-2026-0001 CREDENTIAL_CHANGE_LIFECYCLE_ACTION=deactivate
scripts/credential-change.py lifecycle-event CCR-2026-0001 --action compromise --actor <operator> --reason "<non-secret reason>" --detail "<non-secret evidence>" --blast-radius "<non-secret scope>" --follow-up "<task/ref>" --record-state-hub
scripts/credential-change.py import-inventory CCR-YYYY-NNNN --title "existing lane" --tenant <tenant> --workload <workload> --environment production --purpose "<purpose>" --kv-path platform/workloads/<tenant>/<workload>/<purpose> --field <FIELD_NAME> --auth-method oidc --auth-mount netkingdom --auth-role <role> --bound-claim groups=<group> --bound-claims-confirmed --frontdoor-type ops-warden --catalog-id <catalog-id> --reason "Imported existing lane without secret values"

apply-plan and operator-commands are intentionally guarded: they refuse anything not approved and refuse unconfirmed auth bindings. operator-commands renders the reviewed non-secret bao policy write and bao write auth/.../role commands for a platform operator; the actual secret value is still provisioned through approved OpenBao/operator custody only.

The same operations can be exposed through chat by having the agent create the proposal, show the rendered summary, then call the CLI only after the human gives an explicit approval phrase.

State Hub Role

State Hub should hold:

• request lifecycle events;
• review comments;
• approval/denial decisions;
• non-secret apply and verification evidence;
• links to workplans and CCR files.

State Hub should not hold secret values. It can be the first review UI because it already supports messages, progress, task status, and cross-repo coordination.

For CCR review, create a pending State Hub decision that links to the CCR and contains only non-secret coordinates. Operators can inspect it in the dashboard at http://127.0.0.1:3000/decisions and resolve it with a rationale beginning with APPROVE:, DENY:, or NEEDS_CHANGES:. Then run make credential-change-sync-decision CREDENTIAL_CHANGE=<CCR> to copy the resolved decision back into the CCR file-backed state.

OpenBao Role

OpenBao remains authoritative for:

• policy enforcement;
• auth method configuration;
• token issuance and revocation;
• secret storage;
• audit logs.

Where OpenBao supports non-secret metadata on secret paths or auth roles, we can mirror CCR ids and status labels. The workflow must not depend on OpenBao being the only index, because operators need to see proposed, rejected, deactivated, rotated, and compromised items across repos and access front doors.

ops-warden Role

ops-warden should consume only approved and active access lanes.

For draft requests, ops-warden may create a draft catalog entry that points to the CCR, but it should not activate the entry until the CCR is verified.

For warden access --fetch / --exec, the catalog should include the CCR id and refuse active use when the CCR state is not active, readiness is not ready, or resolvable is not true.

Interactive Runbook Role

The interactive runbook is the operator bridge:

1. load a CCR;
2. show the rendered summary and exact generated plan;
3. confirm the request is approved;
4. acquire operator authority through an approved path;
5. apply the plan;
6. ask for attended secret entry when needed;
7. run positive and negative verification;
8. record non-secret evidence;
9. notify downstream front doors such as ops-warden.

credential-change.py runbook <CCR> renders the checklist and exact final confirmation phrase. --execute-metadata is intentionally opt-in and requires that phrase; it uses the local bao CLI with ambient approved operator authority, writes only policy/auth metadata, and records a non-secret metadata_apply evidence entry. Secret value provisioning stays outside the script through approved OpenBao/operator custody. Verification, activation, and manual custody events are recorded with record-evidence, whose comments are scanned for known secret markers before the CCR file or State Hub is updated.

This lets operators safely drive privileged work without needing to remember every OpenBao command.

Compromise And Deactivation

Every active CCR needs a deactivate and rotate path:

• deactivated: access intentionally disabled but not necessarily compromised;
• rotated: secret value replaced and old value no longer valid;
• compromised: emergency state requiring immediate disablement, rotation, blast-radius notes, and incident follow-up.

lifecycle-plan renders the attended checklist for each case, including the front-door state change and OpenBao metadata disable commands for deactivation or compromise. lifecycle-event records the non-secret lifecycle event in the CCR, sets the CCR status, and marks the access front door disabled, pending verification, or compromised as appropriate. For compromise events it accepts non-secret blast-radius notes and follow-up task references. Existing lanes that predate CCRs can be imported with import-inventory, which writes a CCR and matching read-policy artifact from metadata only; it never asks for or stores the secret value.

The workflow must support marking an existing credential or lane as compromised even when the original request predates this system.

Near-Term Target

Use the whynot-design npm token lane as the pilot:

1. encode the existing non-secret lane as a CCR;
2. render it for review;
3. approve or request changes from chat;
4. generate/apply the OpenBao policy and auth role only after approval;
5. provision the secret value by attended operator custody;
6. verify and activate the ops-warden catalog entry.

Once that path feels good, reuse it for the sibling workload-KV lanes and the credential broker's OpenBao token-role gates.

Self-Service Intake (RPF-WP-0061)

Clients file standard CCRs without a railiance-platform worker session. The platform's judgement is in registry/ccr-standard-profile.yaml; review-standard applies it, and .forgejo/workflows/ccr-intake.yaml re-checks it in CI. Allowed submitters: ops-mason only (profile submitters).

In profile means: a workload-kv-read lane on mount platform; one exact path platform/workloads/<a>/<b>[/<c>] or platform/suppliers/<supplier>/<account>; metadata_read: false; a policy file equal to the generated HCL; Kubernetes auth with role external-secrets-*, audience openbao, all ttls 15m or less, token_no_default_policy: true; one service account in one non-platform namespace; risk low or medium; and no other live CCR reading the path. Everything else (OIDC roles, AppRole, token roles, second readers, high risk) still goes to the railiance-platform inbox.

Client steps

Run these in a railiance-platform checkout that is up to date with main:

# 0. Generate the policy instead of writing it, so it is byte-equal by construction.
python3 scripts/credential-change.py render-policy --kv-path <platform/...> > <policy.hcl>
#    Optional: plan_ref: {repo, path, commit} in the draft links the client's
#    construction plan; submit keeps it and review-standard only checks its shape.

# 1. File the draft. Allocates the CCR id, copies the policy, runs the review.
python3 scripts/credential-change.py submit \
  --ccr <draft.yaml> --policy <policy.hcl> --slug <slug> \
  --requester ops-mason --push --notify
#    [NEEDS-PLATFORM] lines mean out of profile: stop. --notify messaged
#    railiance-platform. Do not edit the review comment to get past it.

# 2. Wait for the "CCR intake gate" run on ccr/<id>-<slug> to pass, then
#    fast-forward main to it. Never merge a red or unchecked ccr/* branch.
git switch main && git merge --ff-only ccr/<id>-<slug> && git push origin main

# 3. Open the founder decision and commit the decision id.
python3 scripts/credential-change.py request-decision <id>
git commit -m "<id>: request founder decision" -- credential-change-requests/<id>-<slug>.yaml
git push origin main

After step 3, the client waits and does not edit status or review.comments itself. Approvals reach the file only through sync-decision, even when the founder approved the client's plan elsewhere. The hub decision is the single approval record for both the client plan and the CCR; clients copy their plan approval from it after the sync. The founder resolves the State Hub decision. Then a platform session or the founder runs sync-decision <id>. Clients do not run sync-decision on their own CCRs. State Hub does not authenticate who resolved a decision (decided_by is free text). Until founder approval can be verified (RPF-WP-0061 follow-up B), the sync must come from someone other than the requester. After the CCR is approved, the usual steps apply: confirm-binding against native state, apply (ops-mason under ops-mason-construct for in-profile lanes, otherwise the applier), then record-evidence.

Known limits

• All workstation agents push to Forgejo as the same account, so ccr/* cannot yet be restricted to ops-mason. The submitter list and the merge-only-green rule are enforced by tooling and convention, not by Forgejo. The main-branch check-ids job and check-branch detect violations after the fact (follow-up C).
• The automated review is a source review only. It never sets approved and never touches OpenBao.