82 lines
5.2 KiB
Markdown
82 lines
5.2 KiB
Markdown
|
# CUST-WP-0073 — attended cutover runbook (2026-10-20)
|
|||
|
|
|
|||
|
|
Founder interview 2026-10-20 (Bernd Worsch) fixed: one isolated workstation
|
|||
|
|
account hosts **Codex, Claude Code and Grok Build**; attended execution with
|
|||
|
|
per-step confirmation; owner workers commit and push to main; privileged
|
|||
|
|
actions go through the new `gated-executor` repo; curated read RBAC;
|
|||
|
|
short-lived, re-issued credentials; **open egress** (builder mode — re-arm an
|
|||
|
|
egress allowlist at production close-down); Forgejo **bot user without write to
|
|||
|
|
GitOps-sourced repos**, scoped per tenant (current work = tenant zero, the
|
|||
|
|
platform building itself); transport mixed (Codex desktop via loopback SSH,
|
|||
|
|
Claude Code/Grok Build via `wsl -d custodian-agents`).
|
|||
|
|
|
|||
|
|
Shared names: workstation distro `custodian-agents`, account `custodian-agt`
|
|||
|
|
(workstation and railiance01), SSH principal `agt-custodian`, ServiceAccount
|
|||
|
|
`custodian-agents/custodian-agent`, executor account `gexec`.
|
|||
|
|
|
|||
|
|
## Owner deliveries (source, pushed; nothing live)
|
|||
|
|
|
|||
|
|
| Owner | Workplan | Commit | Attended step |
|
|||
|
|
|---|---|---|---|
|
|||
|
|
| railiance-platform | RPF-WP-0062 | 567d2c6, ded5922 | T03 Argo sync |
|
|||
|
|
| railiance-platform | (OpenBao agt-role) | e415da9 | apply ssh roles |
|
|||
|
|
| railiance-cluster | RCLUSTER-WP-0019 | 1d8d11d | T03 mode-600 playbook |
|
|||
|
|
| railiance-infra | RAIL-HO-WP-0014 | c62b187 | T02 account/principal |
|
|||
|
|
| ops-warden | WARDEN-WP-0045 | e62763c | T03 inventory + first issuance |
|
|||
|
|
| gated-executor | GEXEC-WP-0001 | 80b1973 | T05 register, T06 install |
|
|||
|
|
| the-custodian | this dir | workstation-agents-r4/ | distro creation |
|
|||
|
|
|
|||
|
|
## Order (each step: check/diff first, then apply, then verify)
|
|||
|
|
|
|||
|
|
0. **Recovery access**: keep an open `ssh tegwick@railiance01` session through
|
|||
|
|
steps 3–4. Migrate `railiance-platform/scripts/application_backup_credentials.py`
|
|||
|
|
(reads admin kubeconfig as non-root) to `sudo` or gexec before step 3.
|
|||
|
|
1. **OpenBao agt-role**: `railiance-platform/scripts/openbao-apply-ssh-engine.sh`
|
|||
|
|
(dry-run first). Warden: copy the `agt-custodian` actor block from
|
|||
|
|
`ops-warden/examples/inventory.seed.yaml` into `~/.config/warden/inventory.yaml`.
|
|||
|
|
2. **RBAC**: in railiance-platform `make argocd-pin-drift`; then
|
|||
|
|
`argocd app sync railiance-apps-root --revision ded5922 --resource argoproj.io:Application:custodian-agents`;
|
|||
|
|
`argocd app sync custodian-agents`.
|
|||
|
|
3. **k3s mode 600** (railiance-cluster):
|
|||
|
|
`ansible-playbook ansible/k3s-kubeconfig-mode.yml -e target=<railiance01> --check --diff`, then without `--check`.
|
|||
|
|
API pause ~30–90 s. Rollback in playbook header.
|
|||
|
|
4. **Host account** (railiance-infra/ansible):
|
|||
|
|
`ansible-playbook playbooks/bootstrap-ssh-ca.yaml --limit Railiance01 --tags restricted_agent_account --check --diff`, then apply.
|
|||
|
|
Confirm a fresh `ssh tegwick@railiance01` still works before closing step 0's session.
|
|||
|
|
5. **Workstation distro** (operator distro, this repo):
|
|||
|
|
```
|
|||
|
|
powershell.exe -NoProfile -ExecutionPolicy Bypass -File "$(wslpath -w workstation-agents-r4/import.ps1)" -IAmAttended
|
|||
|
|
tar -C workstation-agents-r4 -cf - . | wsl.exe -d custodian-agents -u root -- bash -c 'mkdir -p /root/r4 && tar -C /root/r4 -xf - && bash /root/r4/guest-setup.sh --i-am-attended'
|
|||
|
|
wsl.exe --terminate custodian-agents
|
|||
|
|
```
|
|||
|
|
Never `wsl --shutdown`, never touch another distro.
|
|||
|
|
6. **Harness install + login inside the distro** (as custodian-agt, interactive
|
|||
|
|
by Bernd): Codex, Claude Code, Grok Build installers; each logs in with its
|
|||
|
|
own account. Codex desktop: add the Windows-side public key to
|
|||
|
|
`/etc/ssh/authorized_keys/custodian-agt` (root), connect to
|
|||
|
|
`custodian-agt@127.0.0.1:22273`.
|
|||
|
|
7. **Credentials into the distro** (from the operator account, daily):
|
|||
|
|
- `kubectl -n custodian-agents create token custodian-agent --duration=24h` →
|
|||
|
|
kubeconfig at `/home/custodian-agt/.kube/config` (server URL, CA, token only).
|
|||
|
|
- agent keypair generated inside the distro; operator runs
|
|||
|
|
`warden sign agt-custodian --pubkey <pub> > <cert>`; place cert beside key.
|
|||
|
|
- Forgejo bot token (tenant-zero bot, no GitOps-repo write).
|
|||
|
|
8. **gated-executor**: register project (`custodian register-project`), then
|
|||
|
|
GEXEC-WP-0001-T06 attended install on railiance01.
|
|||
|
|
9. **Proof (T02 acceptance)**, run inside `custodian-agents` as custodian-agt:
|
|||
|
|
- `kubectl auth whoami` → `system:serviceaccount:custodian-agents:custodian-agent`.
|
|||
|
|
- `credential-boundary-r1/verify_agent_denials.py` with supervisor namespace inventory.
|
|||
|
|
- `ssh custodian-agt@railiance01`: `sudo -n true` fails; `cat /etc/rancher/k3s/k3s.yaml` fails.
|
|||
|
|
- `ssh tegwick@railiance01` refused (no cert for that principal); no `~/.kube` admin
|
|||
|
|
config, no `/mnt/c`, `wsl.exe`/`powershell.exe` absent, no `SSH_AUTH_SOCK`,
|
|||
|
|
no OpenBao token.
|
|||
|
|
- gexec: one approved no-op executes; tampered/replayed/expired refused; sanitized output only.
|
|||
|
|
Record sanitized output under `docs/evidence/`.
|
|||
|
|
|
|||
|
|
## Re-arm list (builder-mode conveniences to tighten at production close-down)
|
|||
|
|
|
|||
|
|
- egress allowlist for `custodian-agents`; loopback forwards from the operator distro are reachable (shared WSL2 network namespace)
|
|||
|
|
- retire `agt-task-bridge` / `agt-interhub-bootstrap` → tegwick (RAIL-HO-WP-0014-T03)
|
|||
|
|
- gexec credential fetch via custody owner (GEXEC-WP-0001-T07) instead of a pre-placed file
|
|||
|
|
- T05 rotation of the 2026-09-23 exposed Secrets stays founder-triggered
|