railiance-platform/workplans/RPF-WP-0001-credential-request-and-lease-broker.md
codex e4af2a1d10
All checks were successful
CI Smoke / host-smoke (push) Successful in 0s
CI Smoke / container-smoke (push) Successful in 1s
fix(workplans): migrate active workplans off the retired RAILIANCE-WP prefix
RAILIANCE-WP is a family name, not a repository (ADR-007, and the prefix
registry already lists it retired). Three repositories independently used one
number space for unrelated work — RAILIANCE-WP-0012 was openbao extraction here,
a cnpg backup in railiance-apps and a deploy-verify in railiance-cluster. This
repository also carried two files both numbered 0016.

Active workplans move to the successor prefix and are renumbered from 0001 in
historical order. Archived workplans keep their historical identifiers.

Projection UUIDs are re-derived from the new canonical ids. Records already
registered under the old identifiers leave orphaned hub rows behind; that debt
is recorded in CUST-WP-0068 and clears when ADR-012's reset-from-forge lands.

Refs CUST-WP-0068-T03

Assistant: claude-code
Assistant-Model: opus
Assistant-Process: 2583210@bnt-lap001
Assistant-Session: f2bff2d5-e9b2-4338-92ca-10282a927006
2026-08-25 22:58:35 +02:00

481 lines
23 KiB
Markdown

---
id: RPF-WP-0001
type: workplan
title: "Credential Request and Lease Broker"
domain: financials
repo: railiance-platform
status: finished
owner: codex
topic_slug: railiance
planning_priority: high
planning_order: 5
created: "2026-06-24"
updated: "2026-07-02"
depends_on_workplans:
- RAIL-PL-WP-0002
state_hub_workstream_id: "bcc11cf9-d008-5e42-8fff-e2b33a1d0ff2"
---
# RPF-WP-0001 - Credential Request and Lease Broker
## Goal
Provide a clean, secure, low-friction way for operators, agents, and approved
automations to request, generate, receive, use, renew, and revoke short-lived
credentials such as the OpenBao token needed for ops-warden vault-backed SSH
signing smoke.
The target experience is self-service for routine, policy-approved leases and
explicit human approval for high-risk grants, without ever pasting secret values
into Git, State Hub, chat, prompts, workplans, or shell history.
## Repository Decision
The primary owner is railiance-platform because OpenBao is the canonical
runtime secret custody service and this repo owns platform secrets, identity
integration, and shared credential delivery contracts.
Cross-repo responsibilities:
| Concern | Owner | Boundary |
| --- | --- | --- |
| OpenBao policies, token roles, lease broker, audit | railiance-platform | Owns secret custody and credential generation. |
| Login, OIDC, MFA, IAM profile claims | key-cape | Authenticates humans and service identities. |
| Authorization decision for requested grants | flex-auth | May decide whether actor X may request grant Y for purpose Z. |
| SSH certificate signing | ops-warden | Issues SSH certs only; does not vend OpenBao tokens. |
| Request tracking and progress | state-hub | Stores non-secret request metadata, status, decision ids, and audit pointers only. |
| Agent inference/runtime | llm-connect and callers | Never place secrets in prompts; consume via local env injection or wrapped lease handles. |
This work should update the ops-warden routing catalog when complete, but the
implementation belongs here. If the broker later becomes a general NetKingdom
service, code can split to a dedicated credential-broker repo while OpenBao
policies and grants remain owned by railiance-platform.
## Design Principles
- Prefer dynamic or short-lived leases over static secrets.
- Use response wrapping or local exec-time injection; do not print raw tokens by default.
- Store non-secret lease metadata only: actor, grant, TTL, purpose, lease id or accessor, decision id, timestamps, and revocation state.
- Keep OpenBao audit logs as the source of truth for secret access.
- Make the common path easy: one command to run a task with the right credential.
- Keep high-risk paths explicit: human approval and MFA for elevated grants.
- Every grant has a catalog entry, max TTL, allowed actors/subjects, delivery mode, audit expectations, and revocation behavior.
## Proposed User Experience
Initial pilot command shapes:
credential request vault-token --grant ops-warden/warden-sign --purpose flex-auth-openbao-smoke --ttl 15m
credential exec --grant ops-warden/warden-sign --ttl 15m -- SMOKE_VAULT=1 /home/worsch/ops-warden/scripts/policy_gate_production_smoke.sh
credential status <lease-handle>
credential revoke <lease-handle>
For the ops-warden smoke, the preferred path is credential exec. It obtains a
bounded OpenBao token with the warden-sign policy, injects it as VAULT_TOKEN
only into the child process environment, redacts logs, and revokes or lets the
lease expire after the command finishes.
## Threat Model Summary
Primary risks:
- token leakage through shell history, logs, prompts, chat, State Hub, or Git;
- confused-deputy issuance where an agent requests a broader token than needed;
- stale leases surviving after a task completes;
- bypassing KeyCape identity or flex-auth authorization checks;
- replacing one manual secret-handling ritual with another brittle one.
Mitigations required by this workplan:
- no raw token in command-line arguments, State Hub payloads, workplans, or logs;
- bounded OpenBao token roles and policies;
- response wrapping for copy/paste or remote handoff flows;
- exec-time environment injection for local command execution;
- default TTLs measured in minutes, with explicit max TTLs per grant;
- revocation by lease handle/accessor;
- OpenBao audit verification and non-secret State Hub progress events.
## Tasks
## T01 - Record ownership and architecture decision
```task
id: RPF-WP-0001-T01
status: done
priority: high
state_hub_task_id: "19795f6b-bd41-54cb-a782-846814417704"
```
Write an ADR or docs section confirming railiance-platform as the owner for
OpenBao credential request/generation/delivery, with key-cape, flex-auth,
ops-warden, state-hub, and llm-connect boundaries.
Acceptance:
- Docs state that ops-warden routes SSH certs only and must not vend OpenBao tokens.
- Docs state that State Hub stores request metadata only, never secret values.
- Ops-warden credential routing can point OpenBao token requests here.
**2026-06-25:** Added `docs/credential-broker.md` as the ownership and architecture decision. It records that railiance-platform owns OpenBao credential request/generation/delivery, ops-warden owns SSH certificate signing only, State Hub stores non-secret request metadata only, and llm-connect/callers must not place secrets in prompts.
## T02 - Define credential grant catalog
```task
id: RPF-WP-0001-T02
status: done
priority: high
state_hub_task_id: "ce3913d6-ab7d-56e0-a067-7c5b6d4d5850"
```
Add a non-secret grant catalog schema and initial grant entries.
Initial grant:
- id: ops-warden/warden-sign
- credential type: openbao-token
- policies: warden-sign
- default TTL: 15 minutes
- max TTL: 1 hour unless a human approves more
- purpose examples: flex-auth OpenBao smoke, ops-warden production sign smoke
- allowed delivery: exec-env, response-wrap, local-token-file mode 0600
- denied delivery: chat, State Hub body, Git, command-line token argument
Acceptance:
- Catalog can be validated in CI.
- The catalog distinguishes self-service, approval-required, and break-glass grants.
- No grant entry contains a secret.
**2026-06-25:** Added the non-secret grant catalog at `credential-grants/catalog.yaml` with the initial `ops-warden/warden-sign` pilot grant, plus `scripts/credential-grants-validate.py` and `make credential-grants-validate`. The validator enforces required fields, TTL bounds, denied delivery modes, disallowed OpenBao policies, audit/revocation expectations, and secret-looking marker rejection.
## T03 - Configure bounded OpenBao token roles and policies
```task
id: RPF-WP-0001-T03
status: done
priority: high
state_hub_task_id: "98819394-390d-5a1b-8fe6-cf80f471cffc"
```
Create idempotent scripts/manifests for OpenBao token roles or equivalent lease
issuance paths that can generate child tokens only for approved policies and
TTLs. Start with warden-sign.
Acceptance:
- A non-root issuer path can create a warden-sign token with bounded TTL.
- The resulting token cannot administer OpenBao and can only call the SSH sign paths allowed by openbao/policies/warden-sign.hcl.
- Verification proves the token can run ops-warden vault signing and cannot list unrelated secrets.
**2026-06-26:** Added the source-side OpenBao token-grant implementation for
the `ops-warden/warden-sign` pilot: issuer policy
`openbao/policies/credential-broker-warden-sign-issuer.hcl`, idempotent apply
and verify scripts, Make targets for dry-run/live apply/live verification, and
catalog validation for `openbao.issuer_policy`. Dry-run validation is expected
to work offline. Live closure still requires an approved OpenBao operator token
path and successful runs of `make openbao-configure-token-grants` and
`make openbao-verify-token-grants-smoke`, so T03 remains `progress`.
**2026-06-27:** Attempted the live idempotent apply with
`make openbao-configure-token-grants OPENBAO_TOKEN_GRANT_ARGS=--use-token-helper`.
OpenBao was reachable and unsealed, but the pod token helper received
`403 permission denied` while writing
`sys/policies/acl/credential-broker-warden-sign-issuer`. T03 is now `wait`
until an approved OpenBao issuer/platform-admin path applies the policy and
role, or the pod token helper is granted that narrow capability.
**2026-07-01:** Operator unsealed OpenBao. Live apply succeeded with
`OPENBAO_TOKEN_FILE=~/.local/openbao/platform-admin.token make openbao-configure-token-grants`:
`credential-broker-warden-sign-issuer` policy and `warden-sign` token role are
configured. T03 is `done`.
**2026-07-01 follow-up:** Live smoke succeeded with openbao-verify-token-grants-smoke: a child token minted from role warden-sign signed a throwaway SSH public key through ssh/sign/agt-role, was denied policy metadata read, and was revoked by accessor.
## T04 - Build credential helper MVP
```task
id: RPF-WP-0001-T04
status: done
priority: high
state_hub_task_id: "a339fdb2-c593-5b9e-9080-d08b3061dab9"
```
Build a small CLI/helper in this repo first, for example credential or
openbao-lease, with request, exec, status, and revoke commands.
Acceptance:
- credential exec can run the ops-warden production smoke with VAULT_TOKEN only in the child process environment.
- request returns a wrapped token or lease handle by default, not the raw token.
- status and revoke work by non-secret lease handle/accessor.
- The helper redacts token-looking values from logs and refuses to run in verbose modes that would print secrets.
**2026-06-26:** Added `scripts/credential.py` as the source helper MVP with
`request`, `exec`, `status`, and `revoke` subcommands. The helper validates the
grant catalog, enforces purpose and TTL bounds, defaults `request` to a local
mode-0600 token file plus non-secret accessor metadata, supports response-wrap
handoff, injects `VAULT_TOKEN` only into the child process for `exec`, redacts
token-looking child output, rejects caller-supplied token env assignments, and
revokes exec tokens by accessor in a `finally` block. Added Make dry-run and
ops-warden smoke targets. T04 remains `progress` until a live OpenBao issuer
token is available to prove `credential-exec-ops-warden-smoke` end to end.
**2026-06-27:** Extended the helper with optional flex-auth preflight,
non-secret State Hub lifecycle metadata, actor/subject binding fields,
`--decision-id` support, and Kubernetes-auth delegation output. Fixed the Make
surface so global helper flags such as `--use-token-helper` are passed before
the subcommand. T04 is now `wait` on the same OpenBao live gate as T03 before
ops-warden smoke can be proven end to end.
**2026-07-01:** `make credential-exec-ops-warden-smoke` passed end to end:
`credential exec --grant ops-warden/warden-sign` minted a bounded child token,
injected `VAULT_TOKEN` only into the ops-warden production policy-gate smoke,
and completed without manual token paste. T04 is `done`.
**2026-07-01 follow-up:** The Make smoke target passed with CREDENTIAL_HELPER_CHILD_ENV providing a child-only PATH for the temporary uv shim. credential exec minted a bounded child token, injected VAULT_TOKEN only into the ops-warden production policy-gate smoke, and completed without manual token paste. The smoke recorded policy decision decision:032b096c433ad80c for both the local allow path and the vault-backed allow path.
## T05 - Implement secure delivery modes
```task
id: RPF-WP-0001-T05
status: done
priority: high
state_hub_task_id: "f4299d05-5365-5903-a1db-982356fc2791"
```
Support safe delivery modes for different runtime contexts.
Required modes:
- exec-env: inject credential into one child process, then forget it;
- response-wrap: produce a single-use OpenBao wrapping token for attended handoff;
- local-token-file: write mode 0600 under an ignored local state directory, with TTL metadata and cleanup;
- kubernetes-auth: use service-account-bound auth for in-cluster workloads instead of handing them tokens manually.
Acceptance:
- No delivery mode requires pasting the secret into chat or State Hub.
- local-token-file paths are gitignored and rejected by secret scans if accidentally staged.
- response-wrap unwraps once and fails on second use.
**2026-06-27:** Source support now covers all four delivery modes: `exec-env`,
`response-wrap`, `local-token-file`, and `kubernetes-auth`. The helper refuses
caller-supplied token env assignments, writes local leases under the ignored
`.local/credential-leases/` path with mode `0600`, and emits only service
account auth metadata for Kubernetes-auth. T05 is `wait` until live response-wrap
single-use behavior and the OpenBao-backed exec path are verified with an
approved issuer token.
**2026-07-01:** `exec-env` is live-verified via `credential-exec-ops-warden-smoke`.
`response-wrap`, `local-token-file`, and `kubernetes-auth` still need live
evidence. T05 is `progress`.
**2026-07-01 follow-up:** Completed the remaining delivery-mode proof. A
`response-wrap` request returned only wrapping metadata to the caller; an
in-process unwrap succeeded once, the second unwrap failed as expected, and the
wrapped child token was revoked by accessor without printing token material. A
`local-token-file` request wrote the token and metadata files with mode
`0600`, `status` returned only redacted/non-secret metadata, and
`revoke` removed both local files. `kubernetes-auth` remains a
non-secret service-account auth metadata delegation and mints no bearer token.
T05 is `done`.
## T06 - Integrate KeyCape identity and agent subject binding
```task
id: RPF-WP-0001-T06
status: done
priority: medium
state_hub_task_id: "affee57b-4bb8-5ca0-bf62-33acbbcaaf4a"
```
Define how humans and agents authenticate to request grants.
Acceptance:
- Human operator path uses KeyCape/OIDC with MFA where required.
- Agent/service path has a documented subject id shape compatible with IAM profile claims and existing actor naming.
- Headless automation uses Kubernetes auth or an explicitly approved non-interactive identity; it does not reuse a human token.
**2026-06-27:** Documented the identity contract in `docs/credential-broker.md`:
KeyCape/OIDC with MFA for human operators, stable IAM-compatible subjects for
agents and CI, and Kubernetes service-account subjects for headless workloads.
The helper now exposes `--actor`, `--actor-type`, and `--subject`, and validates
actor type against the grant catalog. T06 is done source-side.
## T07 - Add flex-auth preflight authorization and State Hub request metadata
```task
id: RPF-WP-0001-T07
status: done
priority: medium
state_hub_task_id: "9c415813-a842-5f50-8990-7d0a4f85931f"
```
Before issuing a lease, optionally call flex-auth with actor, subject, grant,
purpose, TTL, audience, and requested delivery mode. Record non-secret request
metadata and decision ids in State Hub when available.
Acceptance:
- flex-auth can deny overbroad TTL, wrong actor type, wrong purpose, or disallowed delivery mode.
- State Hub records request lifecycle without token values.
- The helper works in offline/degraded mode only for pre-authorized local flows; it never caches new secret material in State Hub.
**2026-06-27:** Added optional flex-auth preflight via `--flex-auth-url` /
`FLEX_AUTH_URL`, strict `--require-flex-auth`, provided decision ids via
`--decision-id`, and opt-in State Hub lifecycle notes via `--record-state-hub`.
The helper records only non-secret metadata. T07 is `wait` until a live flex-auth
credential authorization endpoint is available and the OpenBao live gate is
cleared.
**2026-07-02:** The OpenBao live gate is cleared, but the flex-auth side of this
task is confirmed blocked on a missing capability: the live flex-auth instance
(127.0.0.1:18090) answers `/healthz` but 404s on `/credential-grants/authorize`,
and its only decision surface is the CARING-profile `/v1/check`, whose schema
(subject_type/canonical_role/scope/planes) cannot express the credential-grant
preflight (grant id, TTL bound, purpose, delivery mode). No FLEX-WP workplan
covers this endpoint. Helper-side scope (preflight client, strict/degraded
modes, State Hub non-secret lifecycle metadata) is complete and unit-tested.
Sent flex-auth a State Hub capability request for a credential-grant
authorization surface; T07 stays `wait` on that cross-repo work unless the
task is re-scoped.
**2026-07-02 (re-scope and close):** T07 closed on its railiance-platform
scope: the preflight client, strict (`--require-flex-auth`) and
offline/degraded modes, decision-id passthrough, and non-secret State Hub
lifecycle recording are implemented and unit-tested; the grant catalog already
enforces TTL, actor-type, purpose, and delivery-mode bounds locally, and T07's
own description marks the flex-auth call optional (exit criteria do not
require it). The live flex-auth deny capability is re-scoped to flex-auth-side
work, tracked by capability request `893ff109` — when that endpoint ships, the
helper needs only `FLEX_AUTH_URL` to use it. Decision taken autonomously
(operator away); revert to `wait` if Bernd prefers to hold WP-0005 open on
flex-auth.
## T08 - Integrate ops-warden smoke and routing catalog
```task
id: RPF-WP-0001-T08
status: done
priority: high
state_hub_task_id: "39ba556f-59d9-5696-8295-91b347b8a82c"
```
Replace the manual VAULT_TOKEN step in ops-warden smoke docs with the credential
helper flow and update the credential routing catalog.
Acceptance:
- FLEX-WP-0007 T4 can be run with one command once the grant is configured:
credential exec --grant ops-warden/warden-sign --ttl 15m -- SMOKE_VAULT=1 /home/worsch/ops-warden/scripts/policy_gate_production_smoke.sh
- ops-warden docs still make clear it owns SSH cert signing, not OpenBao token vending.
- warden route find VAULT_TOKEN points to this railiance-platform flow.
**2026-06-27:** Added `make credential-exec-ops-warden-smoke` for the intended
one-command smoke and confirmed credential routing locally with
`uv run warden route show openbao-api-key --json`: OpenBao/API/dynamic lease
needs belong to `railiance-platform`; ops-warden executes SSH cert issuance
only. T08 is `wait` because this workspace cannot update the external
ops-warden routing catalog and the live OpenBao grant apply is still denied.
**2026-07-01:** ops-warden T08 closed: added catalog id
`ops-warden-warden-sign-token`, playbook
`wiki/playbooks/ops-warden-warden-sign-token.md`, and updated
`operator-openbao-token-hygiene.md`, `PolicyGatedSigning.md`, and
`CredentialRouting.md`. `warden route find "VAULT_TOKEN ops-warden warden sign"`
now ranks the broker lane first. Live smoke already proven via
`make credential-exec-ops-warden-smoke`. T08 is `done`.
## T09 - Verification, audit, and red-team checks
```task
id: RPF-WP-0001-T09
status: done
priority: high
state_hub_task_id: "17d02492-6f0e-52c9-8747-03c2131d03e3"
```
Add tests and operator verification for the complete flow.
Acceptance:
- Unit tests cover grant validation, TTL bounds, redaction, and delivery-mode restrictions.
- Dry-run tests require no secrets.
- Live smoke proves OpenBao audit logs record issuance and use.
- Negative tests prove denied grants do not mint tokens.
- Documentation includes emergency revocation and cleanup commands.
**2026-06-27:** Added `tests/test_credential_helper.py` and `make credential-tests`
covering TTL bounds, actor-type restrictions, token redaction, unsafe env
rejection, local lease mode/cleanup, Kubernetes-auth delegation, and gitignore
coverage for local lease files. Offline validation is passing. T09 is `wait`
until live OpenBao audit evidence, response-wrap unwrap-once evidence, and
negative live mint checks can be collected.
**2026-07-02:** T09 closed. Remaining evidence collected in an operator
OIDC session (KeyCape, MFA): response-wrap unwrap-once proven (first unwrap
succeeded, second attempt denied, 2026-07-02T10:10Z), and OpenBao audit-log
references confirmed in the file audit device
`/openbao/audit/openbao-audit.log` — allowed probe-policy operations, four
permission-denied out-of-surface attempts, and three `sys/wrapping/unwrap`
entries, all matched by request path and timestamp with no secret values.
Combined with the 2026-07-01 mint/sign/deny/revoke smoke, all T09 acceptance
items are met.
**2026-07-01:** Live verification moved forward. make credential-tests passed 50 tests. make openbao-verify-token-grants-smoke minted a child token with policy warden-sign, proved it can sign via ssh/sign/agt-role, proved it cannot read policy metadata, and revoked it by accessor. make credential-exec-ops-warden-smoke passed with the child-only PATH hook, proving the flex-auth allow/deny smoke and vault-backed ops-warden signing path without manual VAULT_TOKEN paste. T09 is progress; remaining evidence is OpenBao audit-log reference collection plus response-wrap unwrap-once verification.
## T10 - Rollout and migration
```task
id: RPF-WP-0001-T10
status: done
priority: medium
state_hub_task_id: "678b105d-1b3e-5f91-86c5-a5e6960814bd"
```
Roll out in phases.
Phases:
1. warden-sign VAULT_TOKEN pilot for flex-auth/ops-warden smoke.
2. Platform-readonly token helper for diagnostics.
3. Workload-specific grants for app repositories.
4. Optional split to a dedicated credential-broker repo if code grows beyond railiance-platform ownership.
Acceptance:
- The VAULT_TOKEN blocker from FLEX-WP-0007 is cleared without manual token paste.
- Operators have a documented fast path and a break-glass path.
- State Hub, ops-warden, key-cape, and flex-auth docs link to the same routing truth.
**2026-06-27:** Documented rollout phases, emergency revocation, delivery modes,
identity binding, flex-auth preflight, State Hub metadata, and routing ownership
in `docs/credential-broker.md`. T10 is `wait` on the live warden-sign pilot and
external routing-doc/catalog updates.
**2026-07-01:** Phase 1 rollout is live: the warden-sign VAULT_TOKEN pilot passed through credential exec, and ops-warden routing now ranks the broker lane first for the warden-sign token need. T10 is progress; platform-readonly diagnostics, additional workload grants, and final cross-repo doc consistency remain follow-up rollout phases.
**2026-07-02:** T10 closed on its acceptance criteria. (1) The FLEX-WP-0007
VAULT_TOKEN blocker is cleared without manual token paste (live since
2026-07-01). (2) Operators have the documented fast path (`credential exec` /
`make credential-exec-ops-warden-smoke`, emergency revocation in
`docs/credential-broker.md`) and break-glass path (root-token/unseal ceremony
in `docs/openbao.md`). (3) Routing truth is consistent: ops-warden
`CredentialRouting.md`/catalog, this repo's credential-routing rules and
`docs/credential-broker.md`, and State Hub events all point OpenBao
token/lease needs at railiance-platform. Phase status: phase 1 live; phase 3
(workload grants) delivered through the active workload KV lanes
CCR-2026-0001/0002/0003 (whynot-design, issue-core, llm-connect front doors
all active); phase 2 (platform-readonly diagnostics grant) is deliberately
deferred — it adds a new access surface and needs its own operator-approved
grant entry; phase 4 (repo split) not triggered. Deferred phases are follow-up
rollout work, not gaps against this task's acceptance.
## Exit Criteria
- A policy-approved actor can request or exec with a short-lived OpenBao token without seeing or pasting the raw token.
- The ops-warden vault-backed smoke can run without manual VAULT_TOKEN handling.
- All issued credentials are bounded, auditable, and revocable.
- State Hub and workplans contain only non-secret metadata.
- The credential routing catalog points token/dynamic-lease requests to railiance-platform.