feat: prepare scoped State Hub preflight signing custody and rotation

Assistant: codex
Assistant-Model: gpt-6-astra
Assistant-Session: 01a06ed7-828d-7ca0-a8d4-0c3e5a0c4102

docs/credential-lane-designs/state-hub-preflight-activation.md

@@ -0,0 +1,92 @@
+# State Hub preflight signing activation — RPF-WP-0035-T04
+
+CCR-2026-0015 is the exact read grant. The user requested this task on 2026-09-05;
+FLEX-WP-0020-T05 remains its consuming demand. A fresh non-mutating live preflight
+for flex-auth -> access-engine returned 200 with exactly one blocker:
+`preflight_signing_unavailable`. No rename or operation journal is authorized.
+
+## Observed target and access boundary
+
+Primary/railiance01, namespace/release/deployment `state-hub`, API container and
+service account `state-hub`, one replica, existing image `main-cdff3b7`.
+The dedicated ESO identity is `state-hub/state-hub-preflight-eso`, audience
+`openbao`; installed CRDs accept `serviceAccountRef.audiences`. ESO is already a
+trusted cluster-wide controller with TokenRequest/Secret permissions; this lane
+adds no broad RBAC. The API SA cannot create ExternalSecrets or TokenRequests
+for the delivery identity (live `kubectl auth can-i` returned no for both).
+Only trusted namespace administrators may create pod/ExternalSecret/TokenRequest
+resources; no workload RBAC grant is added. Namespace-scoped SecretStore prevents
+cross-namespace store references. Coding-agent data AND metadata are denied.
+
+## Review and apply
+
+1. Review CCR-2026-0015, its generated policy/auth payload, delivery.yaml, and
+   State Hub chart diff together. Confirm the dedicated SA binding and record
+   CCR approval from the user's task authorization. Keep the lane non-resolvable.
+2. Commit/push the reviewed source in both repositories before live apply.
+3. Run the silent writer through the contained attended login envelope:
+
+   ```sh
+   warden access openbao-platform-admin-login --exec -- \
+     /usr/bin/python3 /home/worsch/railiance-platform/scripts/state_hub_preflight_lane.py \
+     provision --expected-version 0 --kubeconfig /home/worsch/.kube/config \
+     --receipt /tmp/statehub-preflight-provision.json --confirm 'APPLY CCR-2026-0015'
+   ```
+
+   The identity act requires operator OIDC/MFA. The expired ambient token is not
+   usable. Warden captures both streams and revokes/removes the contained helper;
+   the writer emits only a mode-0600 metadata receipt. Never fall back to root,
+   a workload token or another service's secret. Generation is CSPRNG 32 bytes,
+   CAS zero, with no raw key on argv/disk/output. Subsequent CAS rotation is a
+   separately named writer action, never granted to ESO or API.
+4. Verify SecretStore Ready and ExternalSecret SecretSynced. Compare accepted KV
+   version with the Secret's safe metadata and check key presence without output.
+   Check shared `state-hub-env` has no same-name key; do not overwrite that Secret.
+5. Enable `renamePreflight.enabled` in production values and deploy the reviewed
+   chart with the current image. Only API gets the required explicit key ref;
+   MCP and migration jobs remain excluded. Verify all API replicas' process key
+   matches the materialized version using in-memory equality and boolean output.
+6. Call only the non-mutating preflight endpoint for the observed fixture, keep
+   its token private, and record blocker codes and signing-verification booleans.
+   Do not call start/apply/rollback mutation endpoints or print the full response.
+
+## Executable rotation fence and forward recovery
+
+The single-key application has no hot reload. Use a controlled API outage; it
+fences issuance and every rename mutation equally, including direct pod callers.
+Retain the live replica count and Helm revision. Confirm no HPA or other
+reconciler is changing the deployment during this attended window.
+
+```sh
+kubectl -n state-hub scale deployment/state-hub --replicas=0
+kubectl -n state-hub wait --for=delete pod -l app=state-hub --timeout=120s
+kubectl -n state-hub get pods -l app=state-hub
+```
+
+No API process may remain, including terminating pods. Explicit invalidation of
+outstanding preflight tokens replaces a TTL drain: do not wait while issuance
+continues. Preserve an old signed fixture only in protected memory/file, never
+in receipts. The rotation helper independently refuses nonzero replicas, any
+matching pod, or an HPA before generating a key.
+
+Run the same attended envelope with `rotate --expected-version <current>` and
+a fresh receipt path. Wait for ESO to project the new version, then scale to
+the retained replica count, wait for rollout and verify health, one accepted key
+across all replicas, new-token validity and old-token rejection. Never reopen
+with a mixed-key population. Update production source and record exact Helm,
+image and KV versions. No repository rename is performed by this rehearsal.
+
+On failure after CAS, leave the API fenced until ESO and forward recovery pass.
+The receipt retains the new KV version even if a later check fails. The `verify`
+action rechecks custody without regenerating or overwriting a key. Initial
+partial metadata apply is deliberately not silently overwritten: inspect the
+exact role/policy and receipt before continuing. Do not replay CAS zero against
+an existing lane. Restore availability without signing only by disabling the
+chart binding after confirming no signing key remains in shared env; retain
+custody for reviewed recovery. Never restore a compromised predecessor key.
+
+## Completion gate
+
+T04 remains open until live custody/auth negatives, ESO delivery, API-only
+exposure, all-replica equality, signed preflight, rotation/invalidation and
+recovery receipts pass. Source tests and server dry-run alone are preparation.

docs/credential-lane-designs/state-hub-preflight-signing.md

@@ -1,6 +1,6 @@
 # State Hub repository-rename preflight signing lane
 
-Status: proposed, not provisioned. Owner: railiance-platform, RPF-WP-0034.
+Status: implementation staged, not provisioned. Owner: railiance-platform, RPF-WP-0035-T04.
 Demand: State Hub message `cd52ba10-de41-46ce-aa8b-9b44050da8f7`,
 STATE-WP-0085-T09. Provisioning this lane does not authorize any repository rename.
 
@@ -112,3 +112,12 @@ and the migration owner must confirm the target runtime and continued need
 before provisioning this design. RPF-WP-0035-T04 is the current platform task;
 RPF-WP-0034 is the archived design record. No demand withdrawal or activation
 is inferred from retirement planning alone.
+
+## Implementation preparation — 2026-09-05
+
+CCR-2026-0015, the exact data-only policy and bounded role, namespace-scoped
+ESO manifests, silent CAS writer and API-only chart binding are staged.
+The fresh live fixture has only the signing-unavailable blocker.
+`state-hub-preflight-activation.md` supplies the controlled-outage rotation
+fence and recovery sequence. Live acceptance still requires attended OIDC/MFA;
+the existing operator session returned 403.