net-kingdom/docs/smooth-bootstrap-guide.md

# NET-WP-0018 Smooth Bootstrap Guide

Exercise status: unknown for the procedures in this runbook; no per-procedure successful-run receipt with operator attribution was established in the 2026-09-05 review. See [procedure inventory](attended-procedure-inventory.md).

**Status:** draft (initial consolidation for T05)
**Date:** 2026-06-03
**Purpose:** The single operator guide for a smooth, repeatable NetKingdom security bootstrap. An operator knows what to do, in what order, and what (non-secret) evidence proves each step complete. Covers the full sequence from the T05 spec + inputs from T02 runtime architecture, T03 retrospective + gap matrix, existing UX contracts (operator-journey, user-lifecycle), console lifecycle-guide (incl. 0019 T06-adjacent polish), evidence templates/validators, and make targets.

This replaces piecemeal reliance on separate docs. It makes wrong-order execution visibly hard via "next safe action" and blocked gates. Links to concrete commands, scripts, console subcommands, validate targets, and evidence.

See also:
- docs/NetkingdomRuntimeArchitecture.md (T02 – what exists)
- docs/security-bootstrap-retrospective.md (T03 – what was bumpy, now automated, gaps)
- tools/security-bootstrap-console/security_bootstrap_console.py + make security-bootstrap-* (control surface, evidence, validators)
- [Current lifecycle owner routes](legacy-lldap-lifecycle-retirement.md); the 0019 scripts are retired.
- .local/security-bootstrap.json + console status (current gates)

**Pragmatic note (per 0018 Coordination):** Track your progress through this guide using State Hub /progress/ (with workstream/task), dated notes in NET-WP-0018 workplan, git, console evidence/validators, /tmp evidence. This feeds future retrospectives.

## Overall Model and Principles

From platform architecture and UX contracts:
- **Stages:** S1 Low-trust assembly → ... → S6 Reopen under custody (see console status).
- **Shell / First screen always answers:** Current stage, Next safe action, Blocked gates (why), Evidence (non-secret records).
- **UI posture (console/web):** Calm field notebook; black/white + hi accents; panels; sentence case; no hype. Shows effective access before any save/action. Blocked conditions explicit (e.g., no platform-root for non-king, MFA required for privileged).
- **Evidence discipline:** All steps produce/require non-secret evidence.json or metadata flags matching exact templates/validators (no secret markers). 12+ bools for user lifecycle (effective preview, no root grant, actor checks, verified identity/claims, reversible, no secrets recorded, etc.).
- **Actor classes & previews:** Always distinguish (setup operator, platform admin, tenant admin, reviewer, king). Show effective privileges before create/save. Never grant platform-root except via explicit king path.
- **Secret boundary:** Console/UI never collects/stores secrets. Use password-safe, k8s secrets, or operator memory. Use the existing governed credential-delivery path; legacy automatic admin-secret extraction is withdrawn.
- **Reversible where possible; human custody gates explicit.**
- **Handoff to production readiness:** After S6, move to 0017 production items (audit durability, etc. – not duplicated here).

**Sequence overview (high-level; details per section):**
1. Prerequisites & cluster foundation.
2. Credential bundle / king kit (SOPS/age, custody).
3. PrivacyIDEA bootstrap + realm.
4. LLDAP/bootstrap user (platform-root/king) + MFA self-enroll + verify.
5. KeyCape deployment + client registration + OIDC.
6. OpenBao init/unseal/config (OIDC admin binding via KeyCape).
7. Token cleanup, root disposition, restore drill, escrow/custody.
8. User lifecycle (onboard/lock/offboard/review through the current User Engine and IAM owner contracts).
9. State Hub sync, audit posture, cleanup/rotation.
10. Platform reopen (S6) + handoff.

Each step has: commands/scripts, evidence required, blocked conditions, links to validators/console.

## Step 1: Prerequisites, Cluster Foundation, Credential Bundle

**Prerequisites:**
- Live Railiance/k3s cluster with ingress, cert-manager, NetworkPolicies, operators as per T02.
- Operator access: kubectl to sso/openbao ns; password safe entries (net-kingdom/LLDAP/admin, etc.); age keys/custodian public.
- No live OpenBao init in console (attended only).

**Credential bundle / king kit:**
- Use `make security-bootstrap-king-kit` or console `king-kit`.
- Dedicated king credential (platform-root@lldap, separate from personal).
- MFA: privacyIDEA self-service TOTP.
- Storage: password-safe + offline packet (age encrypted).
- Evidence: custodian_age_*_confirmed, king_credential_ready, mfa_enrolled_confirmed, password_safe_confirmed, storage_classes, custody_packet_prepared.
- Validate: `make security-bootstrap-validate-kit` or console `validate-king-kit`.
- Custody roster (for 2of3 target): `make security-bootstrap-custody-roster-template`, sign, validate.

**Blocked if:** No king kit or custody approval.

**Next safe after:** Approve custody mode (console `approve-custody-mode` or make with ARGS for flags like --mfa-enrolled-confirmed).

## Step 2: PrivacyIDEA Bootstrap + Realm

- Repair missing realm/resolver configuration; preserve current owner-managed self-service/MFA policies.
- Use `sso-mfa/k8s/privacyidea/repair-realm-live.sh` or console action "Repair privacyIDEA realm and resolver". See the [configuration-preserving contract](../sso-mfa/k8s/privacyidea/README.md#configuration-preserving-realm-repair); missing onboarding policies use their existing owner procedure.
- Enroll platform-root TOTP in repaired realm (pi-admin for setup).
- Evidence: related to t02 validate (realm healthy).
- Validate: `make security-bootstrap-validate-t02` (covers audit/recovery gates incl. this).

**Blocked if:** Realm not correct for LLDAP users (MFA/self-enroll fails).

See T03 retrospective for past realm drift bumps (now partially automated via runbook + validate).

## Step 3: LLDAP / Bootstrap User Creation (platform-root / king)

- Establish or recover platform-root only through the exact owner-reviewed IAM bootstrap/recovery ceremony; ordinary public registration cannot grant platform-root.
- The legacy account-creation and break-glass scripts are retired; see [recovery limits](legacy-lldap-lifecycle-retirement.md). Preserve existing account/password custody pending owner verification.
- Self-enroll TOTP in privacyIDEA.
- Verify MFA state: `bash sso-mfa/k8s/privacyidea/check-user-mfa-state.sh platform-root --token-file /absolute/private/diagnostic.jwt` after owner-admitted protected input. See the [diagnostic contract](../sso-mfa/k8s/privacyidea/README.md#scoped-factor-state-diagnostic); the KeyCape workload credential is not a general diagnostic delivery lane. Counts do not verify user existence or an MFA login.
- Verify OIDC/KeyCape path: `cd ../keycape && ./verify-openbao-client.sh`.
- Groups: net-kingdom-admins for platform-root.
- Evidence: identity_account_created, identity_group_confirmed, mfa_*, oidc_login_verified.
- For non-root trials: follow Step 7 with exact owner scope and actual consumer evidence.

**Blocked if:** Actor requests net-kingdom-admins without king path; no MFA for privileged.

**Console:** `lifecycle-guide` for current owner routes. Legacy execution and inferred-claims commands exit with failure.

## Step 4: KeyCape Deployment + Client Registration + OIDC

- KeyCape as lightweight IAM (conforms to IAM Profile v0.2).
- Deploy client config (sso-mfa/k8s/keycape/create-secrets.sh).
- Apply keycape-config Secret, restart KeyCape.
- Register bootstrap clients (netkingdom-bootstrap-console, openbao-admin).
- OpenBao admin redirects: localhost CLI callbacks plus
  `http://127.0.0.1:18200/ui/vault/auth/netkingdom/oidc/callback` for the
  preferred operator-tunneled browser UI login. The former public callback is
  retained only for the bounded listener rollback window; `keycape` remains a
  compatibility auth mount.
- Verify OIDC admin login: platform-root obtains OpenBao platform-admin via KeyCape/MFA.
- Evidence: keycape client gates, openbao_oidc_* , oidc_login_verified.
- Validate related in t02 / console.

**Issuer:** https://kc.coulomb.social (see T02 for claims: tenant, groups, roles, assurance, etc.).

See T03 for past callback/registration bumps (now gated).

## Step 5: OpenBao Init / Unseal / Config + OIDC Admin Binding

**Unseal custody model (select first):** see `docs/openbao-unseal-custody-models.md`.

| Model | When | Console |
| --- | --- | --- |
| `sops-held-automation` | **Default** — fast rebuild test cycles | Selectable; gate `automation` |
| `attended-ceremony` | Production trust | Blocked until implemented |
| `auto-unseal-transit` | Production HA | Blocked until implemented |

```bash
make security-bootstrap-openbao-unseal-custody-models
make security-bootstrap-select-openbao-unseal-custody-model MODEL=sops-held-automation
```

Development strategy: prove automation path first (SSH engine, warden, host CA),
then add attended and auto-unseal gates for production profiles.

**Console refuses live init in all models** — automation runs via `creds-bootstrap-agent.sh`
(sops-held) or future attended/auto-unseal playbooks.

- Preflight: `make security-bootstrap-openbao-preflight --run` or console.
- Init/unseal: **sops-held-automation** → `sso-mfa/bootstrap/creds-bootstrap-agent.sh`;
  **attended-ceremony** → human ceremony (planned automation hooks).
- Post-unseal: apply initial config (auth, mounts, policies, audit).
- OIDC auth config against KeyCape (maps claims/groups to policies e.g. net-kingdom-admins → platform-admin).
- Key material handling: trial exposure taint, rotate unseal keys, emergency lockdown, restore drill (snapshot, isolate, verify, destroy).
- Root token disposition: revoked (evidence: root_token_disposition).
- Evidence: openbao_initialized, initial_config_applied, trial_exposed + response_complete, keys_rotated, post_unseal_verified, openbao_oidc_*, restore_drill_passed, etc.
- Validate: multiple t02-related + console gates (taint logic in runbooks).

**Blocked if:** Not unsealed, no OIDC binding, no restore drill, trial material not responded to.

**Console runbooks:** Key material compromised, generate new unseal, emergency lock-down, restore drill, token revocation.

See T02 for secret/credential path details; T03 for past claim shape / token issues (now partially automated via gates/runbooks).

## Step 6: Token Cleanup, State Hub Sync, Audit Posture, Cleanup/Rotation

- Revoke short-lived/bootstrap tokens (use console revoke helpers or runbooks; no plaintext on CLI).
- State Hub: sync work (POST /progress/, decisions, etc.); ensure .custodian-brief reflects.
- Audit posture: record audit_core_bootstrap_risk_accepted (or production_sink_ready), owner, review_date (2026-07-02), note. Use console approve or metadata update. Validate: console audit_core_posture.
- Cleanup/rotation: review bootstrap-era creds/databases/paths; rotate as needed. Evidence: cleanup_complete.
- Custody packet / roster final.

**Evidence:** root_token_disposition, audit_core_*, cleanup_complete, etc.

**Console:** `cleanup-evidence-template`, `validate-cleanup`, metadata updates.

See T03 for operator-state / token bumps (now in metadata + validators).

## Step 7: User lifecycle through its owners

Follow [the current lifecycle routes and retirement notice](legacy-lldap-lifecycle-retirement.md).
User Engine owns ordinary onboarding, account lifecycle and product memberships;
the existing identity-provisioner and IAM provider own identity and password setup.
Use only admitted clients/tenants. Platform-root bootstrap has a separate owner
ceremony and cannot be performed through public registration.

Preview the intended scope, obtain the existing authorization, and retain actual
identity/claims/access, denial and exact cleanup receipts. A group list cannot
verify issued claims or prove all sessions revoked. Never delete test accounts
by a name pattern. Preserve other tenants and global identity state.

`lifecycle-flow-template` and `onboarding-dry-run-template` are blank evidence
worksheets. Their validators check structure; passing is not native acceptance.
The old Make/console execution commands and “User lifecycle dry-run (T06)” runbook
are retired. Group administration and independent recovery remain open in
NK-WP-0049; do not infer readiness from the earlier 0019 receipts.

## Step 8: Platform Reopen + Handoff

- Final gates: all prior evidence + platform_reopened flag.
- Approve custody if needed.
- Console: status shows S6; "Review related workplans".
- Handoff: produce handover checklist (console `handover-checklist`); transfer to production readiness (audit durability, escrow, etc. per 0017 – not duplicated here).
- Evidence: platform_reopened, review_date, notes.

**Validate:** related custody/kit + full status.

## Step 9: Post-Reopen / Optimization

- Use T03 retrospective + T02 arch + this guide for future drills.
- Address gaps (UE adapters per assessment, full Audit Core correlation, more T08 validators).
- Rehearse rebuild per T09 (T09 complete; see docs/security-bootstrap-rebuild-risk-and-rehearsal.md for risk classification + non-destructive scripted/ns/parallel plan using 0019+creds+T08 validators; scripted/namespace first; use current owner lifecycle contracts and exact trial provenance).

## Evidence Summary Table (Core Non-Secret)

- King/custody: age keys, roster, packet, approval.
- Identity/MFA: account/group created, mfa_enrolled, oidc_verified.
- OpenBao: initialized, config_applied, oidc_bound, root_disposition, restore_drill, keys_rotated, taint/response.
- Lifecycle (0019): dry_run_date, actor_class, groups, effective_access_summary, lock_offboard_result, *_verified, no_secret_material_recorded, prevents_*, shows_effective_before_save.
- General: cleanup_complete, platform_reopened, audit_core_* (risk accepted or sink ready), metadata_updated_at.

All validated via console `validate-*` or make targets. Templates in console.

## References and Updates

- Full list in T02/T03 docs.
- Console `lifecycle-guide`, `status`, `web-ui`.
- Update this guide + console guide section as T06/T08 work proceeds (e.g., more validators, control surface alignment).
- For web-ui exposure of this guide: see T06.

This guide + the runtime architecture + retrospective turn the first bootstrap into a repeatable, auditable (pragmatically), low-diagnosis path. Use it; record evidence; improve via T07+.

**Next after this guide:** Align control surface (T06), add tests (T07), integrate validations (T08), assess rebuild risk (T09). 

See NET-WP-0018 workplan for full acceptance.