railiance-platform/docs/credential-lane-designs/state-hub-preflight-activation.md

State Hub preflight signing activation — RPF-WP-0035-T04

CCR-2026-0015 is the exact read grant. The user requested this task on 2026-09-05; FLEX-WP-0020-T05 remains its consuming demand. A fresh non-mutating live preflight for flex-auth -> access-engine returned 200 with exactly one blocker: preflight_signing_unavailable. No rename or operation journal is authorized.

Observed target and access boundary

Primary/railiance01, namespace/release/deployment state-hub, API container and service account state-hub, one replica, existing image main-cdff3b7. The dedicated ESO identity is state-hub/state-hub-preflight-eso, audience openbao; installed CRDs accept serviceAccountRef.audiences. ESO is already a trusted cluster-wide controller with TokenRequest/Secret permissions; this lane adds no broad RBAC. The API SA cannot create ExternalSecrets or TokenRequests for the delivery identity (live kubectl auth can-i returned no for both). Only trusted namespace administrators may create pod/ExternalSecret/TokenRequest resources; no workload RBAC grant is added. Namespace-scoped SecretStore prevents cross-namespace store references. Coding-agent data AND metadata are denied.

Review and apply

1. Review CCR-2026-0015, its generated policy/auth payload, delivery.yaml, and State Hub chart diff together. Confirm the dedicated SA binding and record CCR approval from the user's task authorization. Keep the lane non-resolvable.

2. Commit/push the reviewed source in both repositories before live apply. Verify the selected kubeconfig reaches kube-system UID a553c742-0115-43d4-99a4-a5ca56fe0786 using a metadata-only namespace GET. The writer enforces this identity before OpenBao access. The workstation's default config uses a local port forward; if that listener is unavailable, establish the approved cluster access path before starting attended login. Do not substitute another cluster's context to make the command succeed.

3. Run the silent writer through the contained attended login envelope:

   warden access openbao-platform-admin-login --exec -- \
     /usr/bin/python3 /home/worsch/railiance-platform/scripts/state_hub_preflight_lane.py \
     provision --expected-version 0 --kubeconfig /home/worsch/.kube/config-railiance01 \
     --receipt /tmp/statehub-preflight-provision.json --confirm 'APPLY CCR-2026-0015'
   

   The identity act requires operator OIDC/MFA. The expired ambient token is not usable. Warden captures both streams and revokes/removes the contained helper; the writer emits only a mode-0600 metadata receipt. Never fall back to root, a workload token or another service's secret. Generation is CSPRNG 32 bytes, CAS zero, with no raw key on argv/disk/output. Subsequent CAS rotation is a separately named writer action, never granted to ESO or API.

4. Verify SecretStore Ready and ExternalSecret SecretSynced. Compare accepted KV version with the Secret's safe metadata and check key presence without output. Check shared state-hub-env has no same-name key; do not overwrite that Secret.

5. Enable renamePreflight.enabled in production values and deploy the reviewed chart with the current image. Only API gets the required explicit key ref; MCP and migration jobs remain excluded. Verify all API replicas' process key matches the materialized version using in-memory equality and boolean output.

6. Call only the non-mutating preflight endpoint for the observed fixture, keep its token private, and record blocker codes and signing-verification booleans. Do not call start/apply/rollback mutation endpoints or print the full response.

Executable rotation fence and forward recovery

The single-key application has no hot reload. Use a controlled API outage; it fences issuance and every rename mutation equally, including direct pod callers. Retain the live replica count and Helm revision. Confirm no HPA or other reconciler is changing the deployment during this attended window.

kubectl -n state-hub scale deployment/state-hub --replicas=0
kubectl -n state-hub wait --for=delete pod -l app=state-hub --timeout=120s
kubectl -n state-hub get pods -l app=state-hub

No API process may remain, including terminating pods. Explicit invalidation of outstanding preflight tokens replaces a TTL drain: do not wait while issuance continues. Preserve an old signed fixture only in protected memory/file, never in receipts. The rotation helper independently refuses nonzero replicas, any matching pod, or an HPA before generating a key.

Run the same attended envelope with rotate --expected-version <current> and a fresh receipt path. Wait for ESO to project the new version, then scale to the retained replica count, wait for rollout and verify health, one accepted key across all replicas, new-token validity and old-token rejection. Never reopen with a mixed-key population. Update production source and record exact Helm, image and KV versions. No repository rename is performed by this rehearsal.

On failure after CAS, leave the API fenced until ESO and forward recovery pass. The receipt retains the new KV version even if a later check fails. The verify action rechecks custody without regenerating or overwriting a key. Initial partial metadata apply is deliberately not silently overwritten: inspect the exact role/policy and receipt before continuing. Do not replay CAS zero against an existing lane. Restore availability without signing only by disabling the chart binding after confirming no signing key remains in shared env; retain custody for reviewed recovery. Never restore a compromised predecessor key.

Completion gate

T04 remains open until live custody/auth negatives, ESO delivery, API-only exposure, all-replica equality, signed preflight, rotation/invalidation and recovery receipts pass. Source tests and server dry-run alone are preparation.

Live policy drift review

The first contained apply refused before mutation because the live shared boundary lacks the source-only Policy Nexus data/metadata denies. A separate contained metadata read confirmed that exact difference. The reviewed writer now pins the observed live baseline and appends only agent-deny-overlay.hcl, preserving every unrelated live rule. It does not deploy the pending Policy Nexus source-only change as part of this lane. The global source policy retains its desired rules; live parity for that other lane remains a platform assurance observation, not an acceptance claim here.

Accepted activation — 2026-09-05

T04 completed; CCR-2026-0015 is active. Version 1 was generated with CAS zero, then version 2 was generated with CAS 1 during the all-API-replicas outage. Both custody receipts passed exact read and scope/identity denials. ESO is Ready; Helm revision 59 enabled the API-only binding with the existing image. The post-rotation API probe passed key equality, primary health, new signed preflight and predecessor signature rejection. No repository rename was executed.

ESO v0.16.1 required token self-validation and cleanup despite the deliberate absence of the default policy. The exact read policy now additionally permits auth/token/lookup-self read and auth/token/revoke-self update. It adds no KV metadata, parent list or secret write. Source evidence: https://github.com/external-secrets/external-secrets/blob/v0.16.1/pkg/provider/vault/auth.go The bounded repair-policy action refuses drift and does not regenerate a key.

Use the dedicated kubeconfig above, whose port-forward endpoint is 16444; the default config's 16443 listener was unavailable. Always verify the pinned cluster UID. Protected predecessor fixture was removed after acceptance. Receipts: docs/evidence/RPF-WP-0035-T04-signing-activation-2026-09-05.json.