railiance-platform/docs/credential-lane-designs/nk-bind-executor-go-live.md

# Bounded executor credential go-live (NK-WP-0056-T06; RPF-WP-0069-T05/T09)

Founder runbook for the platform side of the bounded executor's two credentials. Source only:
nothing here has been run. Every command is founder-run and attended, from
`/home/worsch/railiance-platform` at the reviewed commit. No command prints a credential value.

| Request | What it grants | Approvers |
| --- | --- | --- |
| CCR-2026-0060 | KeyCape's ESO reader of `platform/workloads/keycape/nk-bind-executor-approval` (`CLIENT_SECRET`), role `external-secrets-keycape-nk-bind-executor-approval` | platform-owner |
| CCR-2026-0061 | The executor's own reader of the same path, role `keycape-nk-bind-executor-approval-bounded-executor-workload-kv-read`, bound to `sso/nk-bind-executor`, audience `openbao`, 15m | platform-owner, net-kingdom-owner |
| CCR-2026-0063 | Audit-core sender bearer custody `platform/workloads/net-kingdom/bounded-executor/audit-sender`, ESO reader role `external-secrets-nk-bind-executor-audit` | platform-owner, audit-core-owner, net-kingdom-owner |
| CCR-2026-0058 | Window-scoped svc-authelia write role (already approved; created per admission window by net-kingdom `tools/service-binds/executor-lanes.py`, not here) | done |

The KeyCape client is SecurityCase `nk-bind-executor-approval` of type `oidc-service-client` 0.2
(acknowledged 2026-11-01). Its client secret is minted straight into OpenBao by the credential broker,
delivered to KeyCape by ESO, and read natively by the executor through CCR-2026-0061's role.

## Tools

- `scripts/credential-change.py approve | confirm-binding | applier-apply`: approvals and the delegated
  metadata applier (policies and roles). Since RPF-WP-0069-T07, `approve` records each approval in turn;
  for `workload-kv-read` no hand-recorded second approval is needed.
- `scripts/credential.py generate --grant keycape/client-secret-generate`: the broker mint
  (`scripts/keycape_client_secret_generate.py`, admits `oidc-service-client@0.2` since RPF-WP-0069-T08).
  Silent, CAS 0, create-only, metadata-only verification, mode-0600 receipt.
- `scripts/nk_bind_executor_custody.py` (new): `plan` (offline), `fence` (coding-agent deny for both
  custody paths, append-only on the live boundary) and `verify` (metadata, capabilities, login results
  and in-memory hashes only; never reads the KeyCape client secret). Silent; results in the receipt.
  Tests: `tests/test_nk_bind_executor_custody.py`.
- audit-core `scripts/attended-seed-nk-bounded-executor-sender.sh`: mints the audit bearer CAS 0 into
  the CCR-2026-0063 path, merges the `nk-bounded-executor` registry row with compare-and-set, force-syncs
  `audit-core-senders`, restarts the receiver and proves the bearer (audit-core
  `docs/bounded-executor-receipts.md`, "Founder runbook").
- `manifests/nk-bind-executor-eso.yaml`: two ClusterSecretStores (restricted to `sso`, audience
  `openbao`) and two ExternalSecrets: `sso/keycape-nk-bind-executor-approval-client` (key
  `client-secret`, rendered from `manifests/templates/keycape-issuer-eso.yaml`) and
  `sso/bounded-executor-tokens` (key `audit-core`, the Secret the rapp run Job mounts).

## Gates owned elsewhere (before the steps that need them)

- **G1, before step 5 (mint).** The founder accepts case `nk-bind-executor-approval` (owner and
  founder), and net-kingdom commits
  `security-cases/nk-bind-executor-approval.yaml` with
  `status: materialized` and `acceptance: {at, record, by: [{role: owner}, {role: founder}]}`. The
  broker refuses anything else (`case_not_materialized`, `case_source_validation_failed`); a synthetic
  copy with exactly that shape passed the broker's case check on 2026-11-04.
- **G2, before step 4 (audit seed).** audit-core founder runbook steps 1 and 2: the ingress
  `audit-core-nk-bounded-executor-ingress` is applied and the live scope ConfigMap carries
  `nk-bounded-executor`.

## Order relative to KeyCape

KeyCape resolves a confidential client's secret at start and fails closed when the binding is missing.
Its Deployment runs one replica with strategy Recreate. So:

1. platform steps 1 to 7 below finish first, and step 7 reports
   `sso/keycape-nk-bind-executor-approval-client` Ready=True;
2. only then does net-kingdom patch Deployment `sso/keycape` with env
   `KEYCAPE_NK_BIND_EXECUTOR_APPROVAL_CLIENT_SECRET` from Secret
   `keycape-nk-bind-executor-approval-client` key `client-secret` (bind-env), **and** register client
   `nk-bind-executor-approval` (`keycape-client.py apply`), in one rollout, bind-env first;
3. then the executor's request phase can exchange tokens.

If the ExternalSecret is not Ready when KeyCape restarts, the new pod cannot start and every client
loses sign-in.

## Step 0: session

```bash
cd /home/worsch/railiance-platform && git pull --ff-only
bridge up openbao-ui-railiance01
mkdir -p -m 700 /home/worsch/.local/state/railiance/receipts
python3 scripts/nk_bind_executor_custody.py plan
python3 -m pytest -q tests/test_nk_bind_executor_custody.py tests/test_keycape_client_secret_generate.py
```

Expected: the plan lists the three CCRs with the roles, stores and ExternalSecrets above; tests pass.

## Step 1: approvals and binding confirmations

```bash
cd /home/worsch/railiance-platform
python3 scripts/credential-change.py approve CCR-2026-0060 --reviewer platform-owner --comment "Approved by the founder 2026-11-04: KeyCape ESO reader for SecurityCase nk-bind-executor-approval (oidc-service-client 0.2), NK-WP-0056-T06."
python3 scripts/credential-change.py confirm-binding CCR-2026-0060 --reviewer platform-owner --comment "Binding confirmed: external-secrets/external-secrets, audience openbao, as CCR-2026-0033/0041."
python3 scripts/credential-change.py approve CCR-2026-0061 --reviewer platform-owner --comment "Approved by the founder 2026-11-04: executor reader of the nk-bind-executor-approval client secret, NK-WP-0056-T06."
python3 scripts/credential-change.py approve CCR-2026-0061 --reviewer net-kingdom-owner --comment "Approved by the founder 2026-11-04 for NK-WP-0056-T06."
python3 scripts/credential-change.py confirm-binding CCR-2026-0061 --reviewer platform-owner --comment "Binding confirmed: ServiceAccount nk-bind-executor in namespace sso only (uid bd15569c-8a85-4625-b001-d9f799cb857b), audience openbao, from rapp-bounded-executor."
python3 scripts/credential-change.py approve CCR-2026-0063 --reviewer platform-owner --comment "Approved by the founder 2026-11-04: bounded-executor audit sender custody, ESO to sso/bounded-executor-tokens, NK-WP-0056-T15/T06."
python3 scripts/credential-change.py approve CCR-2026-0063 --reviewer audit-core-owner --comment "Approved by the founder 2026-11-04: sender nk-bounded-executor as registered in AUDIT-WP-0018-T02."
python3 scripts/credential-change.py approve CCR-2026-0063 --reviewer net-kingdom-owner --comment "Approved by the founder 2026-11-04: ESO as drafted (NetKingdom answer recorded 2026-11-04)."
python3 scripts/credential-change.py confirm-binding CCR-2026-0063 --reviewer platform-owner --comment "Binding confirmed: external-secrets/external-secrets, audience openbao; store restricted to namespace sso."
for c in CCR-2026-0060 CCR-2026-0061 CCR-2026-0063; do python3 scripts/credential-change.py validate $c; python3 scripts/credential-change.py applier-dry-run $c | tail -3; done
git add credential-change-requests/CCR-2026-0060-*.yaml credential-change-requests/CCR-2026-0061-*.yaml credential-change-requests/CCR-2026-0063-*.yaml
git commit -m "CCR-2026-0060/0061/0063: founder approvals and binding confirmations (NK-WP-0056-T06)"
git push origin main
```

Expected: `[OK]` with no "bound claim is not confirmed" warning; each dry-run ends with `(matches)`.
The CCRs must be committed before step 2: `nk_bind_executor_custody.py` refuses uncommitted CCRs.

## Step 2: coding-agent fence (before any role exists)

```bash
cd /home/worsch/railiance-platform
BAO_ADDR=http://127.0.0.1:18200 VAULT_ADDR=http://127.0.0.1:18200 warden access openbao-platform-admin-login --exec -- python3 /home/worsch/railiance-platform/scripts/nk_bind_executor_custody.py fence --receipt /home/worsch/.local/state/railiance/receipts/nk-bind-executor-fence-r1.json
python3 -c 'import json; r=json.load(open("/home/worsch/.local/state/railiance/receipts/nk-bind-executor-fence-r1.json")); print(r["status"], r.get("error"), r.get("fence_lines_added"))'
```

Expected: `fenced None 2` (the audit-sender lines may already be live; then fewer, never a rewrite of
existing lines). A rerun needs a new receipt name (`-r2.json`).

## Step 3: policies and roles (delegated applier)

```bash
cd /home/worsch/railiance-platform
BAO_ADDR=http://127.0.0.1:18200 VAULT_ADDR=http://127.0.0.1:18200 warden access openbao-platform-admin-login --exec -- python3 /home/worsch/railiance-platform/scripts/credential-change.py applier-apply CCR-2026-0060 --confirm "DELEGATED APPLY CCR-2026-0060" --actor founder --record-state-hub --quiet
BAO_ADDR=http://127.0.0.1:18200 VAULT_ADDR=http://127.0.0.1:18200 warden access openbao-platform-admin-login --exec -- python3 /home/worsch/railiance-platform/scripts/credential-change.py applier-apply CCR-2026-0061 --confirm "DELEGATED APPLY CCR-2026-0061" --actor founder --record-state-hub --quiet
BAO_ADDR=http://127.0.0.1:18200 VAULT_ADDR=http://127.0.0.1:18200 warden access openbao-platform-admin-login --exec -- python3 /home/worsch/railiance-platform/scripts/credential-change.py applier-apply CCR-2026-0063 --confirm "DELEGATED APPLY CCR-2026-0063" --actor founder --record-state-hub --quiet
git add credential-change-requests/CCR-2026-0060-*.yaml credential-change-requests/CCR-2026-0061-*.yaml credential-change-requests/CCR-2026-0063-*.yaml
git commit -m "CCR-2026-0060/0061/0063: delegated metadata apply (NK-WP-0056-T06)"
git push origin main
```

Expected: each CCR moves to `applied` with a `delegated_metadata_apply` evidence entry; the applier
reads back the policy and role and refuses on drift.

## Step 4: audit sender seed (audit-core's founder runbook, step 3; needs G2)

```bash
BAO_ADDR=http://127.0.0.1:18200 VAULT_ADDR=http://127.0.0.1:18200 warden access openbao-platform-admin-login --exec -- /home/worsch/audit-core/scripts/attended-seed-nk-bounded-executor-sender.sh /home/worsch/audit-core/docs/evidence/nk-bounded-executor-sender-seed-r1.json
python3 -c 'import json; r=json.load(open("/home/worsch/audit-core/docs/evidence/nk-bounded-executor-sender-seed-r1.json")); print(r["status"], r.get("error"), r.get("receiver_probe"), r.get("registry_secret_hash_match"), r["credential_values_emitted"])'
```

Expected: `seeded_pending_sso_delivery None {"read": 403, "own_tenant": 200, "other_tenant": 403,
"other_source": 403} True False`. The seed refuses unless CCR-2026-0063 carries all three approvals and
the fence from step 2 is live.

## Step 5: mint the KeyCape client secret into custody (needs G1)

The caller is the founder's own OIDC login to the broker issuer role (`net-kingdom-admins` + aal2,
CCR-2026-0043), not the platform-admin envelope.

```bash
cd /home/worsch/railiance-platform
export BAO_ADDR=http://127.0.0.1:18200 VAULT_ADDR=http://127.0.0.1:18200
unset BAO_TOKEN VAULT_TOKEN
CASE=/home/worsch/net-kingdom/security-cases/nk-bind-executor-approval.yaml
grep -q '^  status: materialized$' "$CASE" && echo case-materialized
REF="nk-bind-executor-approval@sha256:$(sha256sum "$CASE" | cut -d' ' -f1)"
bao login -no-print -method=oidc -path=netkingdom role=credential-broker-keycape-client-secret-generate
python3 scripts/credential.py generate --grant keycape/client-secret-generate --client-id nk-bind-executor-approval --case-file "$CASE" --case-ref "$REF" --dry-run --receipt /home/worsch/.local/state/railiance/receipts/nk-bind-executor-approval-mint-dry.json
python3 scripts/credential.py generate --grant keycape/client-secret-generate --client-id nk-bind-executor-approval --case-file "$CASE" --case-ref "$REF" --record-state-hub --receipt /home/worsch/.local/state/railiance/receipts/nk-bind-executor-approval-mint-1.json
bao token revoke -self
python3 -c 'import json; [print(r["outcome"], r["exit_code"], r.get("kv_version"), r.get("grant_token_revoked"), r["credential_values_emitted"]) for r in (json.load(open("/home/worsch/.local/state/railiance/receipts/nk-bind-executor-approval-mint-"+n+".json")) for n in ("dry","1"))]'
```

Expected: `case-materialized`; then `dry_run_would_generate 0 None True False` and
`generated 0 1 True False`. Exit 3 (`already_present`) means an entry exists and was not touched; stop
and reconcile, never rerun with another tool. No value is printed or returned.

## Step 6: ESO delivery to `sso`

```bash
cd /home/worsch/railiance-platform
KUBECONFIG=$HOME/.kube/config-railiance01 kubectl apply --server-side --field-manager=railiance-platform --dry-run=server -f manifests/nk-bind-executor-eso.yaml
KUBECONFIG=$HOME/.kube/config-railiance01 kubectl apply --server-side --field-manager=railiance-platform -f manifests/nk-bind-executor-eso.yaml
KUBECONFIG=$HOME/.kube/config-railiance01 kubectl wait --for=condition=Ready clustersecretstore/openbao-keycape-nk-bind-executor-approval-issuer clustersecretstore/openbao-nk-bind-executor-audit --timeout=120s
KUBECONFIG=$HOME/.kube/config-railiance01 kubectl -n sso wait --for=condition=Ready externalsecret/keycape-nk-bind-executor-approval-client externalsecret/bounded-executor-tokens --timeout=180s
```

Expected: four objects created; both stores and both ExternalSecrets `condition met`. These are status
reads only.

Use server-side apply only. A client-side `kubectl apply` stamps each ExternalSecret with the
`kubectl.kubernetes.io/last-applied-configuration` annotation, and ESO copies annotations to the target object. The
`reject-secret-last-applied` ValidatingAdmissionPolicy then denies every write (`SecretSyncedError: could not
update secret`). This happened on the first go-live run on 2026-11-04. To recover: remove the annotation from both
ExternalSecrets and both stores, re-apply server-side, then force-sync.

## Step 7: verify (metadata, capabilities, logins and hashes only)

```bash
cd /home/worsch/railiance-platform
KUBECONFIG=$HOME/.kube/config-railiance01 BAO_ADDR=http://127.0.0.1:18200 VAULT_ADDR=http://127.0.0.1:18200 warden access openbao-platform-admin-login --exec -- python3 /home/worsch/railiance-platform/scripts/nk_bind_executor_custody.py verify --kubeconfig /home/worsch/.kube/config-railiance01 --receipt /home/worsch/.local/state/railiance/receipts/nk-bind-executor-verify-r1.json
python3 -c 'import json; r=json.load(open("/home/worsch/.local/state/railiance/receipts/nk-bind-executor-verify-r1.json")); print(r["status"], r.get("error"), r.get("phase"), r.get("keycape_client_secret_kv_version"), r.get("audit_sender_kv_version"), r.get("audit_copies_agree_by_hash"), r.get("keycape_client_secret_value_read"))'
BAO_ADDR=http://127.0.0.1:18200 VAULT_ADDR=http://127.0.0.1:18200 warden access openbao-platform-admin-login --exec -- /home/worsch/audit-core/scripts/attended-seed-nk-bounded-executor-sender.sh /home/worsch/audit-core/docs/evidence/nk-bounded-executor-sender-seed-r2.json
```

Expected: `custody_verified_pending_keycape_and_executor None done 1 1 True False`. The verify proves:

- policies and roles equal the CCR-generated bodies; the coding-agent fence beats each lane policy; a
  default-only token cannot read either path;
- custody metadata: both paths at a live version 1;
- `sso/nk-bind-executor` logs in to its role with exactly one policy and a lease of at most 15m, reads
  only its path (sibling KeyCape clients, the audit path, the svc-authelia bind, metadata and
  `auth/token/create` denied); the `default` ServiceAccount, a wrong audience and the two ESO roles are
  refused; `kubectl auth can-i get secrets -n sso` as the executor answers `no`;
- both stores and ExternalSecrets are Ready with the exact role, audience and one field; the
  delivered Secrets hold exactly one key each;
- the audit bearer in custody, in `sso/bounded-executor-tokens` and in the registry row agree by
  SHA-256 in memory; the registry row equals the declared scope.

The KeyCape client secret is never read: its delivery is proven by ExternalSecret status and the
executor's own first token exchange (type verification `service-token-claims`, net-kingdom).

The audit-core rerun (`-r2`) writes nothing; its `sso` hash comparison looks for the older Secret name
`nk-bind-executor-audit` and is skipped until audit-core aligns it to `bounded-executor-tokens`/`audit-core` (requested 2026-11-04). Step 7's hash check covers it either way.

## Then (not platform)

- net-kingdom: bind-env and register-client in one KeyCape rollout (see "Order relative to KeyCape"),
  then `service-token-claims` and `excess-scope-refused` from the executor.
- net-kingdom: the window-scoped CCR-2026-0058 role per admission (`executor-lanes.py`), the founder's
  admission, then the pilot run (NK-WP-0056-T06).
- railiance-platform: record the receipts as CCR evidence (`credential-change.py record-evidence`) and
  move the three CCRs to `verified` after the first executor run.

## Rollback

- Executor reader: delete role `keycape-nk-bind-executor-approval-bounded-executor-workload-kv-read`,
  then its policy.
- KeyCape delivery: only after net-kingdom has removed the client and the env binding, delete
  ExternalSecret `sso/keycape-nk-bind-executor-approval-client`, the store, role
  `external-secrets-keycape-nk-bind-executor-approval` and its policy; then the custody metadata.
- Audit sender: stop the executor, remove only its registry row with CAS and reload audit-core, then
  delete `sso/bounded-executor-tokens`, its store, role and policy; keep the KV versions.
- The coding-agent fence lines stay.