**Scope:** Captures the first successful NetKingdom bootstrap (through 0015-0017 "Reopen under custody" + 0019 T06-adjacent polish) and automation gaps. Input from T02 runtime architecture doc, prior workplan notes/evidence (0017/0019), console metadata/.local flags, operator experience, assessment, and pragmatic records (progress events, evidence.json, git, console validators).
This feeds T05 (smooth guide with evidence per step), T06/T08 (control surface + validations), T09 (rebuild risk/rehearsal), and future optimization.
Not a full historical dump; focused on bumps, diagnosis, current automation status, and remaining fragile/manual assumptions.
## Executive Summary
The first bootstrap proved the stack (LLDAP + Authelia + privacyIDEA + KeyCape + OpenBao + local bootstrap console + State Hub) can deliver a working IAM/security bootstrap path with platform-root custody, MFA, OIDC admin bridging to OpenBao, and non-root user lifecycle.
It also proved the system is easy to derail interactively: realm drift, OIDC callback bridging, LLDAP lookup assumptions, OpenBao claim/policy shape, token expiry/revocation, operator-state persistence (.local metadata), secret taint hygiene, and audit correlation.
**Post-0017/0019 state (S6 Reopen under custody):**
- Many gates now durable/non-secret in .local/security-bootstrap.json + console (platform_reopened, cleanup_complete, oidc_login_verified, audit_core_bootstrap_risk_accepted with owner/review 2026-07-02, etc.).
- Control surface (console + make + web-ui) + evidence templates/validators provide repeatable safe checks and runbooks.
- 0019 added repeatable non-root dry-run orchestration ( /tmp hygiene + k8s fallback, create --test, verifs, lock/offboard, evidence with 12+ exact bools, validate) + claims helper + cleanup + runbook exposure. Makes T06 gate automatable and less manual/taint-prone.
- Pragmatic audit (local-identity TSV + OpenBao PVC+mock + State Hub/console evidence) in use; production Audit Core deferred (risk accepted).
- T02 arch doc now exists as specific-as-deployed baseline (including pragmatic audit paths and UE gaps).
- Evidence discipline for cleanup (T03/T04 0017), lifecycle flow (T05), onboarding dry-run (T06 0017/0019) — all validate exact bools + no secrets.
- Dry-run user lifecycle repeatable/safe (0019 orchestrator + console/make).
- Non-secret progress in console metadata + State Hub /progress/ (used for tracking 0018 impl itself).
**Remaining gaps / fragile assumptions (see matrix):**
- Consolidated smooth bootstrap guide (T05; piecemeal docs + console guide exist but not one operator sequence with per-step evidence).
- Full automated tests/validations for all UI sections/runbooks (T07; 0019 pieces are ready for coverage; more gates needed in T08).
- UE integration (adapters, claims_enrichment routing, membership sync, audit correlation, bootstrap-to-governed transition) — biggest per assessment; current paths direct LLDAP/Keycloak (valid for bootstrap per contract but drift risk).
- Production Audit Core + full correlation (deferred; pragmatic sufficient for now).
- Scratch rebuild risk rehearsal plan (T09; needs T03/T05/T07/T08 complete).
- Some interactive repairs still required for drift cases (realm, callbacks, claims); console exposes repair but not fully preventive.
A scratch rebuild should only happen after the guide + validations + T09 say which human interactions are unavoidable.
## Bumps Encountered, Diagnosis, and Current Status
(Compiled from 0017/0019 notes, T02 doc, console, metadata, assessment, operator history. Each: bump, root cause/diagnosis, was it automated now?, remaining manual/fragile?)
- Bump: Realm/resolver not correctly set for LLDAP users; self-enroll or admin MFA broken.
- Diagnosis: Manual inspection of privacyIDEA admin + LLDAP queries; repair-realm-live.sh or similar.
- Now automated?: Console has "privacyIDEA realm repair" runbook (template + attended steps); validate-t02 etc. check related. repair script exists.
- Remaining: Still attended (not fully declarative/CI); needs live cluster + operator. Gap in T05 guide + T08 validator for "realm healthy for bootstrap users".
- Bump: Redirects (localhost:8250 etc.) or client config not matching; login fails for console/OpenBao.
- Diagnosis: KeyCape client definition in code (create-secrets.sh); apply + restart; verify via console OIDC login.
- Now automated?: KeyCape OpenBao client definition (non-secret in source) + "KeyCape OpenBao client deployed" gate; console preflight + status checks.
- Remaining: Manual apply/restart cycle for drift; documented in T02 but no preventive validator yet (T08 target).
3.**LLDAP lookup assumptions / group membership for claims:**
- Bump: Groups (net-kingdom-admins/users) not reflected in OIDC claims or policy; platform-admin not granted or over-granted.
- Diagnosis: Direct LLDAP queries (GraphQL or scripts); inventory scripts; console claims verification helper (0019).
- Now automated?: 0019 dry-run-nonroot-user.sh + create-user.sh --test + verify-openbao-client + onboarding-dry-run-claims (infers from groups + T01 role; warns on root/admins); evidence "lldap_identity_verified", "keycape_oidc_claims_verified"; console status.
- Remaining: Direct LLDAP in paths (not yet via UE claims_enrichment adapter per assessment gap 6); for non-dry-run, still some manual verification. UE adapters missing.
- Now automated?: Console metadata-template + approve-custody-mode + save_progress_metadata; validate-* targets; .local updated only via console (non-secret); S6 "platform_reopened" + "cleanup_complete" set.
- Remaining: Still file-based (not in cluster secret or State Hub durable for multi-op); risk of manual tamper. T08 should compute more from validators.
7.**Secret taint / hygiene during user lifecycle (pre-0019):**
- Bump: Temporary secrets written to sso-mfa/bootstrap/secrets/ for dry-run/test users; not cleaned; plaintext exposure risk.
- Diagnosis: Manual steps in early T06; inventory + evidence checks.
- Now automated?: 0019 dry-run-nonroot-user.sh (/tmp WORKSPACE + trap EXIT rm; k8s fallback in create-user.sh never touches persistent bootstrap/secrets for --test); evidence "no_secret_material_recorded":true + validator; make security-bootstrap-onboarding-dry-run + cleanup targets; console subcmd; rm in guide updated conceptually.
- Remaining: Old manual path in lifecycle_guide T06 section still lists secret-mkdir (minor doc staleness; prefer orchestrator). Good model for other secret handling.
8.**Audit / correlation gaps (ongoing):**
- Bump: Audit events not correlated across bootstrap (local-identity TSV, OpenBao PVC, State Hub progress, console evidence) vs. UE audit records or flex-auth decisions.
- Diagnosis: Separate systems; assessment gap 7; contract requires shared IDs (request/actor/decision/user_engine_audit/outbox).
- Now automated?: Pragmatic layer working (progress events with workstream/task/decision correlation used for 0018 impl tracking itself; console evidence; local audit.py); audit_core posture gate in console (risk accepted).
- Remaining: No production Audit Core sink (deferred per metadata 2026-07-02); no UE adapters for audit writer/outbox; bootstrap audit separate. T02/T03 document it; T09 risk item.
| User lifecycle (onboard/lock/offboard) | Manual, no preview, no evidence, taint risk | 0019 dry-run + claims + cleanup + console + make + evidence (effective before save, actor checks, reversible) | Transition to UE-backed (adapters) | High (T05/T08 use as model) | T02 + 0019; prevents platform-root grant |
| Restore drill / escrow | No proof of recovery before trust | restore drill passed gate + evidence; custody roster (2of3 planned) | Attended; low-friction upgrade path to escrow | Medium (T09) | 0017 T02 evidence; T02 custody |
| UE integration (adapters, claims_enrichment, memberships, app onboarding, audit correlation) | Direct LLDAP in bootstrap paths; no adapters | Documented in T02 + assessment; 0019 dry-run proves IAM contract | Adapters missing (biggest gap); claims still direct; memberships not synced with owning semantics | High (T03/T09 classify; T07/T08 testbed) | assessment 7 gaps; T02 UE section; boundary contract |
| Consolidated guide + per-step evidence | Operator must rediscover sequence | Piecemeal docs + console lifecycle-guide (T05/T06 flows + 0019 dry-run) + evidence templates | No single "smooth bootstrap guide" with evidence per step + wrong-order hard | High (T05 primary) | T02 feeds it; link validate-* |
| Tests / validations for UI/runbooks | No coverage; sections can regress | Layered plan in T07; 0019 pieces (orchestrator, console cmds, claims, validators) ready | Most sections lack unit/fixture/live tests; live gated | High (T07) | Use T02 doc + 0019 artifacts as fixtures |
| Rebuild risk / rehearsal | Unknown residual human interactions | T02 specific doc + 0019 dry-run model + S6 evidence | Full T03/T05/T07/T08 needed before T09 assessment | High (T09 at end) | Recommend isolated/namespace/scripted first (non-goal: destructive) |
## Recommendations / Next Steps (from this retrospective)
- **T05 priority:** Consolidate into one smooth guide (update operator-journey or new) with explicit evidence per step (link the validate-* and 0019 templates). Update console lifecycle_guide T06 section to prefer orchestrator.
- **T07/T08:** Use 0019 dry-run + new T02 arch doc + evidence as concrete test cases/fixtures. Add validators for realm health, KeyCape client, audit_core posture (already partial), token revocation success, etc. Static tests for runbook presence.
- **T03 complete:** Expand this doc with any new bumps from T05-T08 work. Output the matrix as table in final.
- **T09:** After above, classify risks (esp. UE integration as high, with mitigation via adapters + updated dry-run). Recommend rehearsal strategy (scripted dry + namespace first).
- **Cross:** Feed pragmatic records (this retrospective process itself used progress events + file notes + T02 doc) back into T03. Document current audit in T02 (done).
- **UE:** Per assessment, do not block 0018 on adapters (NK orchestration role), but use T07/T08 + 0019 tooling to prepare integration tests. Create follow-up for UE-side adapter stub if needed.
