Plan identity stack cutover to railiance01

workplans/NET-WP-0021-railiance01-identity-cutover-and-coulombcore-retirement.md

@@ -0,0 +1,180 @@
+---
+id: NET-WP-0021
+type: workplan
+title: "Cut over NetKingdom identity to railiance01 and retire CoulombCore"
+domain: infotech
+repo: net-kingdom
+status: ready
+owner: codex
+topic_slug: netkingdom
+created: "2026-07-27"
+updated: "2026-07-27"
+---
+
+# NET-WP-0021 - railiance01 identity cutover and CoulombCore retirement
+
+Move the authoritative lightweight NetKingdom identity stack from
+CoulombCore (`92.205.130.254`) to railiance01 (`92.205.62.239`) without
+losing users, groups, MFA enrollments, signing/encryption material, or the
+ability to roll back.
+
+The two servers currently run independent copies of KeyCape, Authelia, LLDAP,
+privacyIDEA, and `net-kingdom-pg`. Public KeyCape DNS already points to
+railiance01, while Authelia, LLDAP, and privacyIDEA DNS still points to
+CoulombCore. Retirement is forbidden until state equivalence, end-to-end
+login, backup restoration, and an observed rollback window pass.
+
+## T01 - Freeze the migration contract and inventory both stacks
+
+```task
+id: NET-WP-0021-T01
+status: todo
+priority: high
+```
+
+Capture secret-free inventories of deployments, image digests, configuration
+keys, database schemas/counts, PVCs, ingress, certificates, backup schedules,
+DNS, and external consumers on both clusters. Identify the authoritative
+source for every state set and record expected maintenance/write-freeze
+behavior. Compare public JWKS fingerprints and all client registrations
+without printing private configuration.
+
+Done when every component and state owner has a source, destination,
+verification method, rollback source, and explicit decision on whether
+railiance01 is already equivalent or requires migration.
+
+## T02 - Prove recoverable backups before changing state
+
+```task
+id: NET-WP-0021-T02
+status: wait
+priority: high
+```
+
+Create fresh, encrypted backups for CoulombCore LLDAP, Authelia,
+privacyIDEA, `net-kingdom-pg`, KeyCape configuration/signing key, and required
+Kubernetes Secrets. Preserve privacyIDEA encryption material with its
+database. Validate backup manifests, checksums, custody, retention, and a
+non-production restore drill. Never emit secret values into Git, State Hub,
+logs, or chat.
+
+Done when restore evidence proves the migration can be reversed from
+independent custody.
+
+## T03 - Reconcile persistent identity state onto railiance01
+
+```task
+id: NET-WP-0021-T03
+status: wait
+priority: high
+```
+
+Under a bounded write freeze, compare and migrate the authoritative
+`net-kingdom-pg` data and any non-database LLDAP, Authelia, or privacyIDEA
+state. Preserve user/group identifiers, password hashes, MFA token serials,
+realms, resolver mappings, encryption keys, and audit-relevant timestamps.
+Do not blindly overwrite newer railiance01-only configuration such as the
+`rapp-qonto-client`; merge and verify it explicitly.
+
+Done when source/destination counts and semantic probes agree and restart
+tests pass on railiance01.
+
+## T04 - Align configuration, secrets, and internal dependencies
+
+```task
+id: NET-WP-0021-T04
+status: wait
+priority: high
+```
+
+Reconcile KeyCape, Authelia, LLDAP, and privacyIDEA configuration and secret
+references on railiance01 through approved custody lanes. Verify KeyCape uses
+only railiance01 cluster-local dependencies, signing-key continuity is
+preserved, service clients remain registered, and no runtime dependency
+points back to CoulombCore.
+
+Done when an automated dependency graph and negative search prove there are
+no hidden CoulombCore runtime dependencies.
+
+## T05 - Run full pre-cutover identity conformance
+
+```task
+id: NET-WP-0021-T05
+status: wait
+priority: high
+```
+
+Using TLS-preserving direct resolution to railiance01, verify health,
+discovery/JWKS, authorization-code + PKCE login, MFA, tenant/group/role
+claims, OpenBao OIDC login, `rapp-qonto` client credentials, wrong-secret and
+wrong-scope denial, logout/session behavior, and restart recovery. Include
+the platform-root and Binky tenant paths without exposing tokens.
+
+Done when all critical human and service flows pass and rollback criteria are
+machine-readable.
+
+## T06 - Cut over remaining DNS with monitored rollback
+
+```task
+id: NET-WP-0021-T06
+status: wait
+priority: high
+```
+
+Lower TTL where needed and move `auth.coulomb.social`,
+`lldap.coulomb.social`, `pink.coulomb.social`, and
+`pink-account.coulomb.social` to `92.205.62.239`. Confirm certificate,
+ingress, resolver, and normal-DNS flows from multiple resolvers. Keep
+CoulombCore intact but prevent divergent writes during a defined observation
+window. Roll back DNS if critical authentication or MFA gates fail.
+
+Done when normal DNS consistently resolves to railiance01 and the observation
+window passes without fallback traffic or state divergence.
+
+## T07 - Retire CoulombCore identity workloads reversibly
+
+```task
+id: NET-WP-0021-T07
+status: wait
+priority: high
+```
+
+After T01-T06, scale the old KeyCape, Authelia, LLDAP, and privacyIDEA
+deployments to zero and suspend old backup jobs/ingress. Preserve manifests,
+PVCs, database, Secrets, and encrypted final backup during a rollback
+retention period. Confirm no DNS, ingress, client, monitor, or traffic still
+targets the old services.
+
+Deletion of old workloads, PVCs, database clusters, or secret material is a
+separate destructive action permitted only after the retention period and an
+explicit recorded approval.
+
+Done when CoulombCore serves no identity traffic and remains recoverable
+during the retention window.
+
+## T08 - Final deletion and closure
+
+```task
+id: NET-WP-0021-T08
+status: wait
+priority: medium
+```
+
+After explicit approval, expiry of the rollback retention window, and one
+successful railiance01 restore/restart drill, remove the retired CoulombCore
+identity resources and stale DNS/monitoring/configuration. Record what was
+deleted, backup retention, recovery limits, final topology, and operational
+runbooks. Run `statehub fix-consistency`.
+
+Done when railiance01 is the sole authoritative identity stack, all evidence
+is reconciled, and the workplan is marked finished.
+
+## Safety gates
+
+- No retirement before a tested backup and restore path exists.
+- No database overwrite before authoritative-source and divergence analysis.
+- No privacyIDEA restore without its matching encryption material.
+- No normal-DNS cutover before direct-resolution end-to-end verification.
+- No PVC/database/Secret deletion as part of the reversible retirement step.
+- Final deletion always requires an explicit human approval distinct from DNS
+  cutover approval.