net-kingdom/workplans/NK-WP-0022-railiance01-identity-cutover-and-coulombcore-retirement.md

---
id: NK-WP-0022
type: workplan
title: "Cut over NetKingdom identity to railiance01 and retire CoulombCore"
domain: infotech
repo: net-kingdom
status: blocked
owner: codex
topic_slug: netkingdom
created: "2026-07-27"
updated: "2026-08-08"
depends_on:
  - USER-WP-0020
  - NK-WP-0023
  - KEY-WP-0004
state_hub_workstream_id: "3dadddeb-4cec-48e2-9c2c-788b26dd6e93"
---

# NK-WP-0022 - railiance01 identity cutover and CoulombCore retirement

Move the authoritative lightweight NetKingdom identity stack from
CoulombCore (`92.205.130.254`) to railiance01 (`92.205.62.239`) without
losing users, groups, MFA enrollments, signing/encryption material, or the
ability to roll back.

The two servers currently run independent copies of KeyCape, Authelia, LLDAP,
privacyIDEA, and `net-kingdom-pg`. Public KeyCape DNS already points to
railiance01, while Authelia, LLDAP, and privacyIDEA DNS still points to
CoulombCore. Retirement is forbidden until state equivalence, end-to-end
login, backup restoration, and an observed rollback window pass.

This cutover intentionally waits until the reusable user onboarding portal
completes the Binky tenant-admin flow. That supplies the human login/MFA and
lifecycle evidence needed to judge which identity stack is authoritative
before state migration or retirement begins.

## T01 - Freeze the migration contract and inventory both stacks

```task
id: NK-WP-0022-T01
status: done
priority: high
state_hub_task_id: "fe55a2c2-8613-4190-a0b9-07eff4a58146"
```

Capture secret-free inventories of deployments, image digests, configuration
keys, database schemas/counts, PVCs, ingress, certificates, backup schedules,
DNS, and external consumers on both clusters. Identify the authoritative
source for every state set and record expected maintenance/write-freeze
behavior. Compare public JWKS fingerprints and all client registrations
without printing private configuration.

Done when every component and state owner has a source, destination,
verification method, rollback source, and explicit decision on whether
railiance01 is already equivalent or requires migration.

2026-07-28 DNS inventory: public resolvers show `auth.coulomb.social` and the
new `login.coulomb.social` alias at railiance01 (`92.205.62.239`). The
following names remain explicitly on CoulombCore (`92.205.130.254`):
`bao`, `gitea`, `hub`, `lldap`, `pink-account`, and `pink` under
`coulomb.social`. The last three are identity-cutover scope; Bao, Gitea, and
Hub must be handed to their owning workload cutovers before CoulombCore host
retirement. Local resolver caches may temporarily retain the former
`auth.coulomb.social` address.

The read-only comparison is recorded in
`docs/railiance01-coulombcore-cutover-inventory-2026-07-28.md`. It confirms
matching public KeyCape JWKS fingerprints and matching declared
Authelia/LLDAP/privacyIDEA images, but independent PVCs and divergent LLDAP
database fingerprints. CoulombCore exposes PostgreSQL logical backups but no
matching LLDAP, Authelia, or privacyIDEA backup CronJobs; fresh protected
source backups remain a hard gate.

2026-07-29 closure review: the subsequent backup, migration, dependency and
conformance tasks used this inventory successfully. Every identity state set
now has an authority, protected rollback source and verification method; the
remaining public names are tracked explicitly under T06 or their owning
non-identity cutovers. Inventory contract complete.

## T02 - Prove recoverable backups before changing state

```task
id: NK-WP-0022-T02
status: done
priority: high
state_hub_task_id: "fdc7b492-855f-471e-a7df-ee67a291d7a1"
```

Create fresh, encrypted backups for CoulombCore LLDAP, Authelia,
privacyIDEA, `net-kingdom-pg`, KeyCape configuration/signing key, and required
Kubernetes Secrets. Preserve privacyIDEA encryption material with its
database. Validate backup manifests, checksums, custody, retention, and a
non-production restore drill. Never emit secret values into Git, State Hub,
logs, or chat.

Done when restore evidence proves the migration can be reversed from
independent custody.

2026-07-28: created the age-encrypted source set documented in
`docs/railiance01-coulombcore-cutover-inventory-2026-07-28.md`. Checksums,
age decryption, both SQLite integrity checks, PostgreSQL dump readability,
privacyIDEA archive readability, permissions, and plaintext cleanup passed.
The encrypted set was then copied to mode-`0700` independent custody on
railiance01 with mode-`0600` files and matching source hashes. A temporary,
non-networked PostgreSQL 16 pod restored the full dump and reproduced the
expected `interhub`, `privacyidea_db`, token, owner, realm, resolver, and policy
counts. The pod and transient restore log were removed automatically.

## T03 - Reconcile persistent identity state onto railiance01

```task
id: NK-WP-0022-T03
status: done
priority: high
state_hub_task_id: "737b2210-92c7-45c2-a37a-76951e459c83"
```

Under a bounded write freeze, compare and migrate the authoritative
`net-kingdom-pg` data and any non-database LLDAP, Authelia, or privacyIDEA
state. Preserve user/group identifiers, password hashes, MFA token serials,
realms, resolver mappings, encryption keys, and audit-relevant timestamps.
Do not blindly overwrite newer railiance01-only configuration such as the
`rapp-qonto-client`; merge and verify it explicitly.

Done when source/destination counts and semantic probes agree and restart
tests pass on railiance01.

Pre-migration comparison proves this task is required: LLDAP is 4/7/7
users/groups/memberships on CoulombCore versus 3/7/3 on railiance01.
CoulombCore privacyIDEA has one enrolled token, two realms, two resolvers, and
two policies; railiance01 is empty. All privacyIDEA custody fingerprints
differ, so its database and encryption/signing material must migrate together.

2026-07-28: completed the bounded reconciliation documented in
`docs/railiance01-coulombcore-cutover-inventory-2026-07-28.md`. Fresh encrypted
source and destination snapshots preceded all changes. LLDAP now contains the
authoritative `platform-root` identity and memberships plus preserved
railiance01-only users and tenant groups; disposable source lifecycle users
were excluded. privacyIDEA database counts and all matching custody
fingerprints agree. Twelve nonempty interhub source-table digests agree while
the newer destination schema, migration ledger, and destination-only user were
preserved. SQLite integrity, privacyIDEA HTTP, deployment rollout, and
post-restart semantic probes passed. Transient artifacts were removed and
CoulombCore writers remain frozen against divergence.

## T04 - Align configuration, secrets, and internal dependencies

```task
id: NK-WP-0022-T04
status: done
priority: high
state_hub_task_id: "33591ee3-586e-4ac6-82e6-0ab4642cea5e"
```

Reconcile KeyCape, Authelia, LLDAP, and privacyIDEA configuration and secret
references on railiance01 through approved custody lanes. Verify KeyCape uses
only railiance01 cluster-local dependencies, signing-key continuity is
preserved, service clients remain registered, and no runtime dependency
points back to CoulombCore.

Done when an automated dependency graph and negative search prove there are
no hidden CoulombCore runtime dependencies.

2026-07-28: added
`sso-mfa/k8s/verify-identity-cutover-dependencies.sh`. Its live railiance01
run found no CoulombCore IP/host or public cross-cluster endpoint in decoded
identity Secrets, ConfigMaps, Deployments, or StatefulSets. It verified
cluster-local KeyCape and privacyIDEA dependencies, retained user-engine and
rapp-qonto clients, and Ready deployments. KeyCape signing fingerprints
match. TLS-preserving direct resolution returned HTTP 200 with valid
certificates for `auth`, `login`, `lldap`, `pink`, and `pink-account`.

## T05 - Run full pre-cutover identity conformance

```task
id: NK-WP-0022-T05
status: done
priority: high
state_hub_task_id: "5b01e471-afd0-479d-8b35-66eef7820cf6"
```

Using TLS-preserving direct resolution to railiance01, verify health,
discovery/JWKS, authorization-code + PKCE login, MFA, tenant/group/role
claims, OpenBao OIDC login, `rapp-qonto` client credentials, wrong-secret and
wrong-scope denial, logout/session behavior, and restart recovery. Include
the platform-root and Binky tenant paths without exposing tokens.

Done when all critical human and service flows pass and rollback criteria are
machine-readable.

2026-07-28 progress: the portal starts a KeyCape authorization-code flow with
S256 PKCE, and KeyCape discovery/TLS is healthy at its canonical
`kc.coulomb.social` issuer. `rapp-qonto-client` obtains a token only for
`qonto:read`; a wrong secret returns 401 and excessive `qonto:write` returns
400 without a token. The Binky administrator was created and provisioned
through user-engine's deployed versioned API, producing a linked LLDAP
identity and `password_setup_required` state. This exposed and fixed newline
normalization for the portal's trusted proxy secret in user-engine
`0ef2ae5`; 105 tests pass and the fixed image is live.

The missing SMTP dependency is now removed from the critical path by the
deployed NetKingdom-owned password setup surface. It issues 15-minute,
single-use opaque links, revokes an older link on renewal, consumes before
calling LLDAP's official OPAQUE helper, and fails closed on restart. A
disposable-user conformance run proved renewal, validation, setup, LLDAP
login, replay denial, and cleanup without emitting credentials. Human
acceptance remains open only for Binky's user-chosen password, MFA enrollment,
and final claim/denial checks; no operator-set password was substituted.

2026-07-29 complete: platform-root completed password + OTP login and opened
the portal and OpenBao paths. Binky then completed reusable password setup,
TOTP enrollment, and a fresh authorization-code login after the KeyCape
rollout. The session verified the canonical issuer/audience, Binky tenant and
groups, tenant-admin role, AAL2 `pwd` + `otp`, and denial for platform and
Coulomb administration. Together with the prior qonto positive/negative
client tests and TLS/discovery evidence, pre-cutover conformance passes.

## T06 - Cut over remaining DNS with monitored rollback

```task
id: NK-WP-0022-T06
status: done
priority: high
state_hub_task_id: "00bfa48b-28de-4b8e-b3da-715c797beae5"
```

Lower TTL where needed and move `auth.coulomb.social`,
`lldap.coulomb.social`, `pink.coulomb.social`, and
`pink-account.coulomb.social` to `92.205.62.239`. Confirm certificate,
ingress, resolver, and normal-DNS flows from multiple resolvers. Keep
CoulombCore intact but prevent divergent writes during a defined observation
window. Roll back DNS if critical authentication or MFA gates fail.

Done when normal DNS consistently resolves to railiance01 and the observation
window passes without fallback traffic or state divergence.

2026-07-28: authoritative/public recursive DNS now returns railiance01 for
`auth.coulomb.social`; `login.coulomb.social` was added as an Authelia ingress
and trusted certificate alias. The complete portal → KeyCape → Authelia
redirect reaches railiance01. LLDAP and both privacyIDEA names remain
intentionally on CoulombCore pending the state and conformance gates above.

2026-07-30 recheck: normal resolution now sends `auth`, `login`, `pink`,
`pink-account`, and `kc` to railiance01 (`92.205.62.239`). Only
`lldap.coulomb.social` still resolves to CoulombCore (`92.205.130.254`);
`bao` remains there intentionally and is outside this identity-workload
cutover. No automated DNS credential lane is registered: `warden route find`
returned generic OpenBao/login/SSH routes, not provider DNS authority.
Consequently T06 remains in progress on one explicit operator DNS change and
its observation window, rather than being closed cosmetically.

2026-07-30 completion: the operator moved `lldap.coulomb.social` to
railiance01. Normal resolution now sends `auth`, `login`, `lldap`, `pink`,
`pink-account`, and `kc` to `92.205.62.239`; only the explicitly retained
`bao`, retired `gitea`, and `hub` names resolve to CoulombCore. HTTPS probes
for Authelia, LLDAP, both privacyIDEA surfaces, and KeyCape discovery pass on
the public path. Direct TLS-preserving requests to the old IP return 404 for
each identity hostname, proving the stale ingress has no functioning backend.
The observation gate therefore passes without fallback traffic.

## T07 - Retire CoulombCore identity workloads reversibly

```task
id: NK-WP-0022-T07
status: done
priority: high
state_hub_task_id: "75d160e6-ed58-4b89-b870-4a755e1ca5ce"
```

After T01-T06, scale the old KeyCape, Authelia, LLDAP, and privacyIDEA
deployments to zero and suspend old backup jobs/ingress. Preserve manifests,
PVCs, database, Secrets, and encrypted final backup during a rollback
retention period. Confirm no DNS, ingress, client, monitor, or traffic still
targets the old services.

Deletion of old workloads, PVCs, database clusters, or secret material is a
separate destructive action permitted only after the retention period and an
explicit recorded approval.

Done when CoulombCore serves no identity traffic and remains recoverable
during the retention window.

2026-07-30 inventory: legacy KeyCape, LLDAP, and privacyIDEA are already scaled
to zero with their resources retained; only CoulombCore Authelia remains at
one replica. It must not be scaled down until the final LLDAP DNS record and
T06 observation gate pass. The reversible retirement is therefore reduced to
one remaining workload plus stale ingress/backup checks.

2026-07-30 completion: after T06 passed, CoulombCore Authelia was scaled to
zero. KeyCape, LLDAP, privacyIDEA, and Authelia are now all zero replicas.
No legacy identity backup CronJobs remain present. Services, Ingresses,
ConfigMaps, Secrets, PVCs, TLS material, and source manifests were deliberately
retained. Public identity probes still reach healthy railiance01 endpoints,
while direct old-IP identity requests return 404. The reversible rollback
window begins 2026-07-30; no retained state may be deleted under this task.

## T08 - Final deletion and closure

```task
id: NK-WP-0022-T08
status: wait
priority: medium
state_hub_task_id: "9f209f15-64a9-4324-bfa9-3ff21c52fc19"
```

After explicit approval, expiry of the rollback retention window, and one
successful railiance01 restore/restart drill, remove the retired CoulombCore
identity resources and stale DNS/monitoring/configuration. Record what was
deleted, backup retention, recovery limits, final topology, and operational
runbooks. Run `statehub fix-consistency`.

Done when railiance01 is the sole authoritative identity stack, all evidence
is reconciled, and the workplan is marked finished.

Retention gate: keep the reversible CoulombCore identity resources through at
least 2026-08-29. The workplan is blocked until that review date, when T08 can
be checked for readiness to finish. Reaching the date does not authorize
deletion or completion: T08 still requires a successful railiance01
restore/restart drill and new explicit approval for destructive deletion.

## Safety gates

- No retirement before a tested backup and restore path exists.
- No database overwrite before authoritative-source and divergence analysis.
- No privacyIDEA restore without its matching encryption material.
- No normal-DNS cutover before direct-resolution end-to-end verification.
- No PVC/database/Secret deletion as part of the reversible retirement step.
- Final deletion always requires an explicit human approval distinct from DNS
  cutover approval.