railiance-platform/docs/credential-lane-designs/state-hub-preflight-activation.md

# State Hub preflight signing activation — RPF-WP-0035-T04

CCR-2026-0015 is the exact read grant. The user requested this task on 2026-09-05;
FLEX-WP-0020-T05 remains its consuming demand. A fresh non-mutating live preflight
for flex-auth -> access-engine returned 200 with exactly one blocker:
`preflight_signing_unavailable`. No rename or operation journal is authorized.

## Observed target and access boundary

Primary/railiance01, namespace/release/deployment `state-hub`, API container and
service account `state-hub`, one replica, existing image `main-cdff3b7`.
The dedicated ESO identity is `state-hub/state-hub-preflight-eso`, audience
`openbao`; installed CRDs accept `serviceAccountRef.audiences`. ESO is already a
trusted cluster-wide controller with TokenRequest/Secret permissions; this lane
adds no broad RBAC. The API SA cannot create ExternalSecrets or TokenRequests
for the delivery identity (live `kubectl auth can-i` returned no for both).
Only trusted namespace administrators may create pod/ExternalSecret/TokenRequest
resources; no workload RBAC grant is added. Namespace-scoped SecretStore prevents
cross-namespace store references. Coding-agent data AND metadata are denied.

## Review and apply

1. Review CCR-2026-0015, its generated policy/auth payload, delivery.yaml, and
   State Hub chart diff together. Confirm the dedicated SA binding and record
   CCR approval from the user's task authorization. Keep the lane non-resolvable.
2. Commit/push the reviewed source in both repositories before live apply.
   Verify the selected kubeconfig reaches kube-system UID
   `a553c742-0115-43d4-99a4-a5ca56fe0786` using a metadata-only namespace GET.
   The writer enforces this identity before OpenBao access. The workstation's
   default config uses a local port forward; if that listener is unavailable,
   establish the approved cluster access path before starting attended login.
   Do not substitute another cluster's context to make the command succeed.
3. Run the silent writer through the contained attended login envelope:

   ```sh
   warden access openbao-platform-admin-login --exec -- \
     /usr/bin/python3 /home/worsch/railiance-platform/scripts/state_hub_preflight_lane.py \
     provision --expected-version 0 --kubeconfig /home/worsch/.kube/config-railiance01 \
     --receipt /tmp/statehub-preflight-provision.json --confirm 'APPLY CCR-2026-0015'
   ```

   The identity act requires operator OIDC/MFA. The expired ambient token is not
   usable. Warden captures both streams and revokes/removes the contained helper;
   the writer emits only a mode-0600 metadata receipt. Never fall back to root,
   a workload token or another service's secret. Generation is CSPRNG 32 bytes,
   CAS zero, with no raw key on argv/disk/output. Subsequent CAS rotation is a
   separately named writer action, never granted to ESO or API.
4. Verify SecretStore Ready and ExternalSecret SecretSynced. Compare accepted KV
   version with the Secret's safe metadata and check key presence without output.
   Check shared `state-hub-env` has no same-name key; do not overwrite that Secret.
5. Enable `renamePreflight.enabled` in production values and deploy the reviewed
   chart with the current image. Only API gets the required explicit key ref;
   MCP and migration jobs remain excluded. Verify all API replicas' process key
   matches the materialized version using in-memory equality and boolean output.
6. Call only the non-mutating preflight endpoint for the observed fixture, keep
   its token private, and record blocker codes and signing-verification booleans.
   Do not call start/apply/rollback mutation endpoints or print the full response.

## Executable rotation fence and forward recovery

The single-key application has no hot reload. Use a controlled API outage; it
fences issuance and every rename mutation equally, including direct pod callers.
Retain the live replica count and Helm revision. Confirm no HPA or other
reconciler is changing the deployment during this attended window.

```sh
kubectl -n state-hub scale deployment/state-hub --replicas=0
kubectl -n state-hub wait --for=delete pod -l app=state-hub --timeout=120s
kubectl -n state-hub get pods -l app=state-hub
```

No API process may remain, including terminating pods. Explicit invalidation of
outstanding preflight tokens replaces a TTL drain: do not wait while issuance
continues. Preserve an old signed fixture only in protected memory/file, never
in receipts. The rotation helper independently refuses nonzero replicas, any
matching pod, or an HPA before generating a key.

Run the same attended envelope with `rotate --expected-version <current>` and
a fresh receipt path. Wait for ESO to project the new version, then scale to
the retained replica count, wait for rollout and verify health, one accepted key
across all replicas, new-token validity and old-token rejection. Never reopen
with a mixed-key population. Update production source and record exact Helm,
image and KV versions. No repository rename is performed by this rehearsal.

On failure after CAS, leave the API fenced until ESO and forward recovery pass.
The receipt retains the new KV version even if a later check fails. The `verify`
action rechecks custody without regenerating or overwriting a key. Initial
partial metadata apply is deliberately not silently overwritten: inspect the
exact role/policy and receipt before continuing. Do not replay CAS zero against
an existing lane. Restore availability without signing only by disabling the
chart binding after confirming no signing key remains in shared env; retain
custody for reviewed recovery. Never restore a compromised predecessor key.

## Completion gate

T04 remains open until live custody/auth negatives, ESO delivery, API-only
exposure, all-replica equality, signed preflight, rotation/invalidation and
recovery receipts pass. Source tests and server dry-run alone are preparation.

## Live policy drift review

The first contained apply refused before mutation because the live shared
boundary lacks the source-only Policy Nexus data/metadata denies. A separate
contained metadata read confirmed that exact difference. The reviewed writer
now pins the observed live baseline and appends only `agent-deny-overlay.hcl`,
preserving every unrelated live rule. It does not deploy the pending Policy
Nexus source-only change as part of this lane. The global source policy retains
its desired rules; live parity for that other lane remains a platform assurance
observation, not an acceptance claim here.


## Accepted activation — 2026-09-05

T04 completed; CCR-2026-0015 is active. Version 1 was generated with CAS zero,
then version 2 was generated with CAS 1 during the all-API-replicas outage.
Both custody receipts passed exact read and scope/identity denials. ESO is Ready;
Helm revision 59 enabled the API-only binding with the existing image. The
post-rotation API probe passed key equality, primary health, new signed preflight
and predecessor signature rejection. No repository rename was executed.

ESO v0.16.1 required token self-validation and cleanup despite the deliberate
absence of the default policy. The exact read policy now additionally permits
`auth/token/lookup-self` read and `auth/token/revoke-self` update. It adds no KV
metadata, parent list or secret write. Source evidence:
https://github.com/external-secrets/external-secrets/blob/v0.16.1/pkg/provider/vault/auth.go
The bounded `repair-policy` action refuses drift and does not regenerate a key.

Use the dedicated kubeconfig above, whose port-forward endpoint is 16444;
the default config's 16443 listener was unavailable. Always verify the pinned
cluster UID. Protected predecessor fixture was removed after acceptance.
Receipts: `docs/evidence/RPF-WP-0035-T04-signing-activation-2026-09-05.json`.