net-kingdom/workplans/NK-WP-0022-railiance01-identity-cutover-and-coulombcore-retirement.md

id	type	title	domain	repo	status	owner	topic_slug	created	updated	depends_on	state_hub_workstream_id
NK-WP-0022	workplan	Cut over NetKingdom identity to railiance01 and retire CoulombCore	infotech	net-kingdom	blocked	codex	netkingdom	2026-07-27	2026-08-08	USER-WP-0020 NK-WP-0023 KEY-WP-0004	d76ddccc-00c8-548a-b141-2cd660fa38da

NK-WP-0022 - railiance01 identity cutover and CoulombCore retirement

Move the authoritative lightweight NetKingdom identity stack from CoulombCore (92.205.130.254) to railiance01 (92.205.62.239) without losing users, groups, MFA enrollments, signing/encryption material, or the ability to roll back.

The two servers currently run independent copies of KeyCape, Authelia, LLDAP, privacyIDEA, and net-kingdom-pg. Public KeyCape DNS already points to railiance01, while Authelia, LLDAP, and privacyIDEA DNS still points to CoulombCore. Retirement is forbidden until state equivalence, end-to-end login, backup restoration, and an observed rollback window pass.

This cutover intentionally waits until the reusable user onboarding portal completes the Binky tenant-admin flow. That supplies the human login/MFA and lifecycle evidence needed to judge which identity stack is authoritative before state migration or retirement begins.

T01 - Freeze the migration contract and inventory both stacks

id: NK-WP-0022-T01
status: done
priority: high
state_hub_task_id: "244ef874-8ee4-5969-bcbf-42a7cb47f246"

Capture secret-free inventories of deployments, image digests, configuration keys, database schemas/counts, PVCs, ingress, certificates, backup schedules, DNS, and external consumers on both clusters. Identify the authoritative source for every state set and record expected maintenance/write-freeze behavior. Compare public JWKS fingerprints and all client registrations without printing private configuration.

Done when every component and state owner has a source, destination, verification method, rollback source, and explicit decision on whether railiance01 is already equivalent or requires migration.

2026-07-28 DNS inventory: public resolvers show auth.coulomb.social and the new login.coulomb.social alias at railiance01 (92.205.62.239). The following names remain explicitly on CoulombCore (92.205.130.254): bao, gitea, hub, lldap, pink-account, and pink under coulomb.social. The last three are identity-cutover scope; Bao, Gitea, and Hub must be handed to their owning workload cutovers before CoulombCore host retirement. Local resolver caches may temporarily retain the former auth.coulomb.social address.

The read-only comparison is recorded in docs/railiance01-coulombcore-cutover-inventory-2026-07-28.md. It confirms matching public KeyCape JWKS fingerprints and matching declared Authelia/LLDAP/privacyIDEA images, but independent PVCs and divergent LLDAP database fingerprints. CoulombCore exposes PostgreSQL logical backups but no matching LLDAP, Authelia, or privacyIDEA backup CronJobs; fresh protected source backups remain a hard gate.

2026-07-29 closure review: the subsequent backup, migration, dependency and conformance tasks used this inventory successfully. Every identity state set now has an authority, protected rollback source and verification method; the remaining public names are tracked explicitly under T06 or their owning non-identity cutovers. Inventory contract complete.

T02 - Prove recoverable backups before changing state

id: NK-WP-0022-T02
status: done
priority: high
state_hub_task_id: "12658d0f-eee9-5553-92b6-956ccde094bf"

Create fresh, encrypted backups for CoulombCore LLDAP, Authelia, privacyIDEA, net-kingdom-pg, KeyCape configuration/signing key, and required Kubernetes Secrets. Preserve privacyIDEA encryption material with its database. Validate backup manifests, checksums, custody, retention, and a non-production restore drill. Never emit secret values into Git, State Hub, logs, or chat.

Done when restore evidence proves the migration can be reversed from independent custody.

2026-07-28: created the age-encrypted source set documented in docs/railiance01-coulombcore-cutover-inventory-2026-07-28.md. Checksums, age decryption, both SQLite integrity checks, PostgreSQL dump readability, privacyIDEA archive readability, permissions, and plaintext cleanup passed. The encrypted set was then copied to mode-0700 independent custody on railiance01 with mode-0600 files and matching source hashes. A temporary, non-networked PostgreSQL 16 pod restored the full dump and reproduced the expected interhub, privacyidea_db, token, owner, realm, resolver, and policy counts. The pod and transient restore log were removed automatically.

T03 - Reconcile persistent identity state onto railiance01

id: NK-WP-0022-T03
status: done
priority: high
state_hub_task_id: "642cec3f-fd96-5c05-a37a-8696ed118811"

Under a bounded write freeze, compare and migrate the authoritative net-kingdom-pg data and any non-database LLDAP, Authelia, or privacyIDEA state. Preserve user/group identifiers, password hashes, MFA token serials, realms, resolver mappings, encryption keys, and audit-relevant timestamps. Do not blindly overwrite newer railiance01-only configuration such as the rapp-qonto-client; merge and verify it explicitly.

Done when source/destination counts and semantic probes agree and restart tests pass on railiance01.

Pre-migration comparison proves this task is required: LLDAP is 4/7/7 users/groups/memberships on CoulombCore versus 3/7/3 on railiance01. CoulombCore privacyIDEA has one enrolled token, two realms, two resolvers, and two policies; railiance01 is empty. All privacyIDEA custody fingerprints differ, so its database and encryption/signing material must migrate together.

2026-07-28: completed the bounded reconciliation documented in docs/railiance01-coulombcore-cutover-inventory-2026-07-28.md. Fresh encrypted source and destination snapshots preceded all changes. LLDAP now contains the authoritative platform-root identity and memberships plus preserved railiance01-only users and tenant groups; disposable source lifecycle users were excluded. privacyIDEA database counts and all matching custody fingerprints agree. Twelve nonempty interhub source-table digests agree while the newer destination schema, migration ledger, and destination-only user were preserved. SQLite integrity, privacyIDEA HTTP, deployment rollout, and post-restart semantic probes passed. Transient artifacts were removed and CoulombCore writers remain frozen against divergence.

T04 - Align configuration, secrets, and internal dependencies

id: NK-WP-0022-T04
status: done
priority: high
state_hub_task_id: "8aab3b5d-ec9e-5bc0-a884-ee418e78d5f4"

Reconcile KeyCape, Authelia, LLDAP, and privacyIDEA configuration and secret references on railiance01 through approved custody lanes. Verify KeyCape uses only railiance01 cluster-local dependencies, signing-key continuity is preserved, service clients remain registered, and no runtime dependency points back to CoulombCore.

Done when an automated dependency graph and negative search prove there are no hidden CoulombCore runtime dependencies.

2026-07-28: added sso-mfa/k8s/verify-identity-cutover-dependencies.sh. Its live railiance01 run found no CoulombCore IP/host or public cross-cluster endpoint in decoded identity Secrets, ConfigMaps, Deployments, or StatefulSets. It verified cluster-local KeyCape and privacyIDEA dependencies, retained user-engine and rapp-qonto clients, and Ready deployments. KeyCape signing fingerprints match. TLS-preserving direct resolution returned HTTP 200 with valid certificates for auth, login, lldap, pink, and pink-account.

T05 - Run full pre-cutover identity conformance

id: NK-WP-0022-T05
status: done
priority: high
state_hub_task_id: "a66acc29-33a7-5c20-8816-c3acedd6595d"

Using TLS-preserving direct resolution to railiance01, verify health, discovery/JWKS, authorization-code + PKCE login, MFA, tenant/group/role claims, OpenBao OIDC login, rapp-qonto client credentials, wrong-secret and wrong-scope denial, logout/session behavior, and restart recovery. Include the platform-root and Binky tenant paths without exposing tokens.

Done when all critical human and service flows pass and rollback criteria are machine-readable.

2026-07-28 progress: the portal starts a KeyCape authorization-code flow with S256 PKCE, and KeyCape discovery/TLS is healthy at its canonical kc.coulomb.social issuer. rapp-qonto-client obtains a token only for qonto:read; a wrong secret returns 401 and excessive qonto:write returns 400 without a token. The Binky administrator was created and provisioned through user-engine's deployed versioned API, producing a linked LLDAP identity and password_setup_required state. This exposed and fixed newline normalization for the portal's trusted proxy secret in user-engine 0ef2ae5; 105 tests pass and the fixed image is live.

The missing SMTP dependency is now removed from the critical path by the deployed NetKingdom-owned password setup surface. It issues 15-minute, single-use opaque links, revokes an older link on renewal, consumes before calling LLDAP's official OPAQUE helper, and fails closed on restart. A disposable-user conformance run proved renewal, validation, setup, LLDAP login, replay denial, and cleanup without emitting credentials. Human acceptance remains open only for Binky's user-chosen password, MFA enrollment, and final claim/denial checks; no operator-set password was substituted.

2026-07-29 complete: platform-root completed password + OTP login and opened the portal and OpenBao paths. Binky then completed reusable password setup, TOTP enrollment, and a fresh authorization-code login after the KeyCape rollout. The session verified the canonical issuer/audience, Binky tenant and groups, tenant-admin role, AAL2 pwd + otp, and denial for platform and Coulomb administration. Together with the prior qonto positive/negative client tests and TLS/discovery evidence, pre-cutover conformance passes.

T06 - Cut over remaining DNS with monitored rollback

id: NK-WP-0022-T06
status: done
priority: high
state_hub_task_id: "fc9ed5fb-fd5c-5b04-990b-5a5b7f32fa0f"

Lower TTL where needed and move auth.coulomb.social, lldap.coulomb.social, pink.coulomb.social, and pink-account.coulomb.social to 92.205.62.239. Confirm certificate, ingress, resolver, and normal-DNS flows from multiple resolvers. Keep CoulombCore intact but prevent divergent writes during a defined observation window. Roll back DNS if critical authentication or MFA gates fail.

Done when normal DNS consistently resolves to railiance01 and the observation window passes without fallback traffic or state divergence.

2026-07-28: authoritative/public recursive DNS now returns railiance01 for auth.coulomb.social; login.coulomb.social was added as an Authelia ingress and trusted certificate alias. The complete portal → KeyCape → Authelia redirect reaches railiance01. LLDAP and both privacyIDEA names remain intentionally on CoulombCore pending the state and conformance gates above.

2026-07-30 recheck: normal resolution now sends auth, login, pink, pink-account, and kc to railiance01 (92.205.62.239). Only lldap.coulomb.social still resolves to CoulombCore (92.205.130.254); bao remains there intentionally and is outside this identity-workload cutover. No automated DNS credential lane is registered: warden route find returned generic OpenBao/login/SSH routes, not provider DNS authority. Consequently T06 remains in progress on one explicit operator DNS change and its observation window, rather than being closed cosmetically.

2026-07-30 completion: the operator moved lldap.coulomb.social to railiance01. Normal resolution now sends auth, login, lldap, pink, pink-account, and kc to 92.205.62.239; only the explicitly retained bao, retired gitea, and hub names resolve to CoulombCore. HTTPS probes for Authelia, LLDAP, both privacyIDEA surfaces, and KeyCape discovery pass on the public path. Direct TLS-preserving requests to the old IP return 404 for each identity hostname, proving the stale ingress has no functioning backend. The observation gate therefore passes without fallback traffic.

T07 - Retire CoulombCore identity workloads reversibly

id: NK-WP-0022-T07
status: done
priority: high
state_hub_task_id: "2a6d4c0b-2565-5bf5-9c04-1b63e992ce99"

After T01-T06, scale the old KeyCape, Authelia, LLDAP, and privacyIDEA deployments to zero and suspend old backup jobs/ingress. Preserve manifests, PVCs, database, Secrets, and encrypted final backup during a rollback retention period. Confirm no DNS, ingress, client, monitor, or traffic still targets the old services.

Deletion of old workloads, PVCs, database clusters, or secret material is a separate destructive action permitted only after the retention period and an explicit recorded approval.

Done when CoulombCore serves no identity traffic and remains recoverable during the retention window.

2026-07-30 inventory: legacy KeyCape, LLDAP, and privacyIDEA are already scaled to zero with their resources retained; only CoulombCore Authelia remains at one replica. It must not be scaled down until the final LLDAP DNS record and T06 observation gate pass. The reversible retirement is therefore reduced to one remaining workload plus stale ingress/backup checks.

2026-07-30 completion: after T06 passed, CoulombCore Authelia was scaled to zero. KeyCape, LLDAP, privacyIDEA, and Authelia are now all zero replicas. No legacy identity backup CronJobs remain present. Services, Ingresses, ConfigMaps, Secrets, PVCs, TLS material, and source manifests were deliberately retained. Public identity probes still reach healthy railiance01 endpoints, while direct old-IP identity requests return 404. The reversible rollback window begins 2026-07-30; no retained state may be deleted under this task.

T08 - Final deletion and closure

id: NK-WP-0022-T08
status: wait
priority: medium
state_hub_task_id: "42a3b4c0-3481-5cfa-bdc2-66d05e751829"

After explicit approval, expiry of the rollback retention window, and one successful railiance01 restore/restart drill, remove the retired CoulombCore identity resources and stale DNS/monitoring/configuration. Record what was deleted, backup retention, recovery limits, final topology, and operational runbooks. Run statehub fix-consistency.

Done when railiance01 is the sole authoritative identity stack, all evidence is reconciled, and the workplan is marked finished.

Retention gate: keep the reversible CoulombCore identity resources through at least 2026-08-29. The workplan is blocked until that review date, when T08 can be checked for readiness to finish. Reaching the date does not authorize deletion or completion: T08 still requires a successful railiance01 restore/restart drill and new explicit approval for destructive deletion.

Safety gates

• No retirement before a tested backup and restore path exists.
• No database overwrite before authoritative-source and divergence analysis.
• No privacyIDEA restore without its matching encryption material.
• No normal-DNS cutover before direct-resolution end-to-end verification.
• No PVC/database/Secret deletion as part of the reversible retirement step.
• Final deletion always requires an explicit human approval distinct from DNS cutover approval.