net-kingdom/workplans/NK-WP-0022-railiance01-identity-cutover-and-coulombcore-retirement.md

id	type	title	domain	repo	status	owner	topic_slug	created	updated	depends_on	state_hub_workstream_id
NK-WP-0022	workplan	Cut over NetKingdom identity to railiance01 and retire CoulombCore	infotech	net-kingdom	active	codex	netkingdom	2026-07-27	2026-07-28	USER-WP-0020 NK-WP-0023 KEY-WP-0004	3dadddeb-4cec-48e2-9c2c-788b26dd6e93

NK-WP-0022 - railiance01 identity cutover and CoulombCore retirement

Move the authoritative lightweight NetKingdom identity stack from CoulombCore (92.205.130.254) to railiance01 (92.205.62.239) without losing users, groups, MFA enrollments, signing/encryption material, or the ability to roll back.

The two servers currently run independent copies of KeyCape, Authelia, LLDAP, privacyIDEA, and net-kingdom-pg. Public KeyCape DNS already points to railiance01, while Authelia, LLDAP, and privacyIDEA DNS still points to CoulombCore. Retirement is forbidden until state equivalence, end-to-end login, backup restoration, and an observed rollback window pass.

This cutover intentionally waits until the reusable user onboarding portal completes the Binky tenant-admin flow. That supplies the human login/MFA and lifecycle evidence needed to judge which identity stack is authoritative before state migration or retirement begins.

T01 - Freeze the migration contract and inventory both stacks

id: NK-WP-0022-T01
status: progress
priority: high
state_hub_task_id: "fe55a2c2-8613-4190-a0b9-07eff4a58146"

Capture secret-free inventories of deployments, image digests, configuration keys, database schemas/counts, PVCs, ingress, certificates, backup schedules, DNS, and external consumers on both clusters. Identify the authoritative source for every state set and record expected maintenance/write-freeze behavior. Compare public JWKS fingerprints and all client registrations without printing private configuration.

Done when every component and state owner has a source, destination, verification method, rollback source, and explicit decision on whether railiance01 is already equivalent or requires migration.

2026-07-28 DNS inventory: public resolvers show auth.coulomb.social and the new login.coulomb.social alias at railiance01 (92.205.62.239). The following names remain explicitly on CoulombCore (92.205.130.254): bao, gitea, hub, lldap, pink-account, and pink under coulomb.social. The last three are identity-cutover scope; Bao, Gitea, and Hub must be handed to their owning workload cutovers before CoulombCore host retirement. Local resolver caches may temporarily retain the former auth.coulomb.social address.

The read-only comparison is recorded in docs/railiance01-coulombcore-cutover-inventory-2026-07-28.md. It confirms matching public KeyCape JWKS fingerprints and matching declared Authelia/LLDAP/privacyIDEA images, but independent PVCs and divergent LLDAP database fingerprints. CoulombCore exposes PostgreSQL logical backups but no matching LLDAP, Authelia, or privacyIDEA backup CronJobs; fresh protected source backups remain a hard gate.

T02 - Prove recoverable backups before changing state

id: NK-WP-0022-T02
status: done
priority: high
state_hub_task_id: "fdc7b492-855f-471e-a7df-ee67a291d7a1"

Create fresh, encrypted backups for CoulombCore LLDAP, Authelia, privacyIDEA, net-kingdom-pg, KeyCape configuration/signing key, and required Kubernetes Secrets. Preserve privacyIDEA encryption material with its database. Validate backup manifests, checksums, custody, retention, and a non-production restore drill. Never emit secret values into Git, State Hub, logs, or chat.

Done when restore evidence proves the migration can be reversed from independent custody.

2026-07-28: created the age-encrypted source set documented in docs/railiance01-coulombcore-cutover-inventory-2026-07-28.md. Checksums, age decryption, both SQLite integrity checks, PostgreSQL dump readability, privacyIDEA archive readability, permissions, and plaintext cleanup passed. The encrypted set was then copied to mode-0700 independent custody on railiance01 with mode-0600 files and matching source hashes. A temporary, non-networked PostgreSQL 16 pod restored the full dump and reproduced the expected interhub, privacyidea_db, token, owner, realm, resolver, and policy counts. The pod and transient restore log were removed automatically.

T03 - Reconcile persistent identity state onto railiance01

id: NK-WP-0022-T03
status: done
priority: high
state_hub_task_id: "737b2210-92c7-45c2-a37a-76951e459c83"

Under a bounded write freeze, compare and migrate the authoritative net-kingdom-pg data and any non-database LLDAP, Authelia, or privacyIDEA state. Preserve user/group identifiers, password hashes, MFA token serials, realms, resolver mappings, encryption keys, and audit-relevant timestamps. Do not blindly overwrite newer railiance01-only configuration such as the rapp-qonto-client; merge and verify it explicitly.

Done when source/destination counts and semantic probes agree and restart tests pass on railiance01.

Pre-migration comparison proves this task is required: LLDAP is 4/7/7 users/groups/memberships on CoulombCore versus 3/7/3 on railiance01. CoulombCore privacyIDEA has one enrolled token, two realms, two resolvers, and two policies; railiance01 is empty. All privacyIDEA custody fingerprints differ, so its database and encryption/signing material must migrate together.

2026-07-28: completed the bounded reconciliation documented in docs/railiance01-coulombcore-cutover-inventory-2026-07-28.md. Fresh encrypted source and destination snapshots preceded all changes. LLDAP now contains the authoritative platform-root identity and memberships plus preserved railiance01-only users and tenant groups; disposable source lifecycle users were excluded. privacyIDEA database counts and all matching custody fingerprints agree. Twelve nonempty interhub source-table digests agree while the newer destination schema, migration ledger, and destination-only user were preserved. SQLite integrity, privacyIDEA HTTP, deployment rollout, and post-restart semantic probes passed. Transient artifacts were removed and CoulombCore writers remain frozen against divergence.

T04 - Align configuration, secrets, and internal dependencies

id: NK-WP-0022-T04
status: done
priority: high
state_hub_task_id: "33591ee3-586e-4ac6-82e6-0ab4642cea5e"

Reconcile KeyCape, Authelia, LLDAP, and privacyIDEA configuration and secret references on railiance01 through approved custody lanes. Verify KeyCape uses only railiance01 cluster-local dependencies, signing-key continuity is preserved, service clients remain registered, and no runtime dependency points back to CoulombCore.

Done when an automated dependency graph and negative search prove there are no hidden CoulombCore runtime dependencies.

2026-07-28: added sso-mfa/k8s/verify-identity-cutover-dependencies.sh. Its live railiance01 run found no CoulombCore IP/host or public cross-cluster endpoint in decoded identity Secrets, ConfigMaps, Deployments, or StatefulSets. It verified cluster-local KeyCape and privacyIDEA dependencies, retained user-engine and rapp-qonto clients, and Ready deployments. KeyCape signing fingerprints match. TLS-preserving direct resolution returned HTTP 200 with valid certificates for auth, login, lldap, pink, and pink-account.

T05 - Run full pre-cutover identity conformance

id: NK-WP-0022-T05
status: progress
priority: high
state_hub_task_id: "5b01e471-afd0-479d-8b35-66eef7820cf6"

Using TLS-preserving direct resolution to railiance01, verify health, discovery/JWKS, authorization-code + PKCE login, MFA, tenant/group/role claims, OpenBao OIDC login, rapp-qonto client credentials, wrong-secret and wrong-scope denial, logout/session behavior, and restart recovery. Include the platform-root and Binky tenant paths without exposing tokens.

Done when all critical human and service flows pass and rollback criteria are machine-readable.

2026-07-28 progress: the portal starts a KeyCape authorization-code flow with S256 PKCE, and KeyCape discovery/TLS is healthy at its canonical kc.coulomb.social issuer. rapp-qonto-client obtains a token only for qonto:read; a wrong secret returns 401 and excessive qonto:write returns 400 without a token. The Binky administrator was created and provisioned through user-engine's deployed versioned API, producing a linked LLDAP identity and password_setup_required state. This exposed and fixed newline normalization for the portal's trusted proxy secret in user-engine 0ef2ae5; 105 tests pass and the fixed image is live.

Human acceptance remains open: LLDAP has no SMTP configuration, so its native self-service reset cannot deliver the first-password handoff. No operator-set password was substituted. A NetKingdom-owned password-setup or transactional mail handoff plus Binky MFA enrollment and final claim/denial checks remain.

T06 - Cut over remaining DNS with monitored rollback

id: NK-WP-0022-T06
status: progress
priority: high
state_hub_task_id: "00bfa48b-28de-4b8e-b3da-715c797beae5"

Lower TTL where needed and move auth.coulomb.social, lldap.coulomb.social, pink.coulomb.social, and pink-account.coulomb.social to 92.205.62.239. Confirm certificate, ingress, resolver, and normal-DNS flows from multiple resolvers. Keep CoulombCore intact but prevent divergent writes during a defined observation window. Roll back DNS if critical authentication or MFA gates fail.

Done when normal DNS consistently resolves to railiance01 and the observation window passes without fallback traffic or state divergence.

2026-07-28: authoritative/public recursive DNS now returns railiance01 for auth.coulomb.social; login.coulomb.social was added as an Authelia ingress and trusted certificate alias. The complete portal → KeyCape → Authelia redirect reaches railiance01. LLDAP and both privacyIDEA names remain intentionally on CoulombCore pending the state and conformance gates above.

T07 - Retire CoulombCore identity workloads reversibly

id: NK-WP-0022-T07
status: wait
priority: high
state_hub_task_id: "75d160e6-ed58-4b89-b870-4a755e1ca5ce"

After T01-T06, scale the old KeyCape, Authelia, LLDAP, and privacyIDEA deployments to zero and suspend old backup jobs/ingress. Preserve manifests, PVCs, database, Secrets, and encrypted final backup during a rollback retention period. Confirm no DNS, ingress, client, monitor, or traffic still targets the old services.

Deletion of old workloads, PVCs, database clusters, or secret material is a separate destructive action permitted only after the retention period and an explicit recorded approval.

Done when CoulombCore serves no identity traffic and remains recoverable during the retention window.

T08 - Final deletion and closure

id: NK-WP-0022-T08
status: wait
priority: medium
state_hub_task_id: "9f209f15-64a9-4324-bfa9-3ff21c52fc19"

After explicit approval, expiry of the rollback retention window, and one successful railiance01 restore/restart drill, remove the retired CoulombCore identity resources and stale DNS/monitoring/configuration. Record what was deleted, backup retention, recovery limits, final topology, and operational runbooks. Run statehub fix-consistency.

Done when railiance01 is the sole authoritative identity stack, all evidence is reconciled, and the workplan is marked finished.

Safety gates

• No retirement before a tested backup and restore path exists.
• No database overwrite before authoritative-source and divergence analysis.
• No privacyIDEA restore without its matching encryption material.
• No normal-DNS cutover before direct-resolution end-to-end verification.
• No PVC/database/Secret deletion as part of the reversible retirement step.
• Final deletion always requires an explicit human approval distinct from DNS cutover approval.