273 lines
12 KiB
Markdown
273 lines
12 KiB
Markdown
---
|
|
id: NK-WP-0022
|
|
type: workplan
|
|
title: "Cut over NetKingdom identity to railiance01 and retire CoulombCore"
|
|
domain: infotech
|
|
repo: net-kingdom
|
|
status: active
|
|
owner: codex
|
|
topic_slug: netkingdom
|
|
created: "2026-07-27"
|
|
updated: "2026-07-28"
|
|
depends_on:
|
|
- USER-WP-0020
|
|
- NK-WP-0023
|
|
- KEY-WP-0004
|
|
state_hub_workstream_id: "3dadddeb-4cec-48e2-9c2c-788b26dd6e93"
|
|
---
|
|
|
|
# NK-WP-0022 - railiance01 identity cutover and CoulombCore retirement
|
|
|
|
Move the authoritative lightweight NetKingdom identity stack from
|
|
CoulombCore (`92.205.130.254`) to railiance01 (`92.205.62.239`) without
|
|
losing users, groups, MFA enrollments, signing/encryption material, or the
|
|
ability to roll back.
|
|
|
|
The two servers currently run independent copies of KeyCape, Authelia, LLDAP,
|
|
privacyIDEA, and `net-kingdom-pg`. Public KeyCape DNS already points to
|
|
railiance01, while Authelia, LLDAP, and privacyIDEA DNS still points to
|
|
CoulombCore. Retirement is forbidden until state equivalence, end-to-end
|
|
login, backup restoration, and an observed rollback window pass.
|
|
|
|
This cutover intentionally waits until the reusable user onboarding portal
|
|
completes the Binky tenant-admin flow. That supplies the human login/MFA and
|
|
lifecycle evidence needed to judge which identity stack is authoritative
|
|
before state migration or retirement begins.
|
|
|
|
## T01 - Freeze the migration contract and inventory both stacks
|
|
|
|
```task
|
|
id: NK-WP-0022-T01
|
|
status: progress
|
|
priority: high
|
|
state_hub_task_id: "fe55a2c2-8613-4190-a0b9-07eff4a58146"
|
|
```
|
|
|
|
Capture secret-free inventories of deployments, image digests, configuration
|
|
keys, database schemas/counts, PVCs, ingress, certificates, backup schedules,
|
|
DNS, and external consumers on both clusters. Identify the authoritative
|
|
source for every state set and record expected maintenance/write-freeze
|
|
behavior. Compare public JWKS fingerprints and all client registrations
|
|
without printing private configuration.
|
|
|
|
Done when every component and state owner has a source, destination,
|
|
verification method, rollback source, and explicit decision on whether
|
|
railiance01 is already equivalent or requires migration.
|
|
|
|
2026-07-28 DNS inventory: public resolvers show `auth.coulomb.social` and the
|
|
new `login.coulomb.social` alias at railiance01 (`92.205.62.239`). The
|
|
following names remain explicitly on CoulombCore (`92.205.130.254`):
|
|
`bao`, `gitea`, `hub`, `lldap`, `pink-account`, and `pink` under
|
|
`coulomb.social`. The last three are identity-cutover scope; Bao, Gitea, and
|
|
Hub must be handed to their owning workload cutovers before CoulombCore host
|
|
retirement. Local resolver caches may temporarily retain the former
|
|
`auth.coulomb.social` address.
|
|
|
|
The read-only comparison is recorded in
|
|
`docs/railiance01-coulombcore-cutover-inventory-2026-07-28.md`. It confirms
|
|
matching public KeyCape JWKS fingerprints and matching declared
|
|
Authelia/LLDAP/privacyIDEA images, but independent PVCs and divergent LLDAP
|
|
database fingerprints. CoulombCore exposes PostgreSQL logical backups but no
|
|
matching LLDAP, Authelia, or privacyIDEA backup CronJobs; fresh protected
|
|
source backups remain a hard gate.
|
|
|
|
## T02 - Prove recoverable backups before changing state
|
|
|
|
```task
|
|
id: NK-WP-0022-T02
|
|
status: done
|
|
priority: high
|
|
state_hub_task_id: "fdc7b492-855f-471e-a7df-ee67a291d7a1"
|
|
```
|
|
|
|
Create fresh, encrypted backups for CoulombCore LLDAP, Authelia,
|
|
privacyIDEA, `net-kingdom-pg`, KeyCape configuration/signing key, and required
|
|
Kubernetes Secrets. Preserve privacyIDEA encryption material with its
|
|
database. Validate backup manifests, checksums, custody, retention, and a
|
|
non-production restore drill. Never emit secret values into Git, State Hub,
|
|
logs, or chat.
|
|
|
|
Done when restore evidence proves the migration can be reversed from
|
|
independent custody.
|
|
|
|
2026-07-28: created the age-encrypted source set documented in
|
|
`docs/railiance01-coulombcore-cutover-inventory-2026-07-28.md`. Checksums,
|
|
age decryption, both SQLite integrity checks, PostgreSQL dump readability,
|
|
privacyIDEA archive readability, permissions, and plaintext cleanup passed.
|
|
The encrypted set was then copied to mode-`0700` independent custody on
|
|
railiance01 with mode-`0600` files and matching source hashes. A temporary,
|
|
non-networked PostgreSQL 16 pod restored the full dump and reproduced the
|
|
expected `interhub`, `privacyidea_db`, token, owner, realm, resolver, and policy
|
|
counts. The pod and transient restore log were removed automatically.
|
|
|
|
## T03 - Reconcile persistent identity state onto railiance01
|
|
|
|
```task
|
|
id: NK-WP-0022-T03
|
|
status: done
|
|
priority: high
|
|
state_hub_task_id: "737b2210-92c7-45c2-a37a-76951e459c83"
|
|
```
|
|
|
|
Under a bounded write freeze, compare and migrate the authoritative
|
|
`net-kingdom-pg` data and any non-database LLDAP, Authelia, or privacyIDEA
|
|
state. Preserve user/group identifiers, password hashes, MFA token serials,
|
|
realms, resolver mappings, encryption keys, and audit-relevant timestamps.
|
|
Do not blindly overwrite newer railiance01-only configuration such as the
|
|
`rapp-qonto-client`; merge and verify it explicitly.
|
|
|
|
Done when source/destination counts and semantic probes agree and restart
|
|
tests pass on railiance01.
|
|
|
|
Pre-migration comparison proves this task is required: LLDAP is 4/7/7
|
|
users/groups/memberships on CoulombCore versus 3/7/3 on railiance01.
|
|
CoulombCore privacyIDEA has one enrolled token, two realms, two resolvers, and
|
|
two policies; railiance01 is empty. All privacyIDEA custody fingerprints
|
|
differ, so its database and encryption/signing material must migrate together.
|
|
|
|
2026-07-28: completed the bounded reconciliation documented in
|
|
`docs/railiance01-coulombcore-cutover-inventory-2026-07-28.md`. Fresh encrypted
|
|
source and destination snapshots preceded all changes. LLDAP now contains the
|
|
authoritative `platform-root` identity and memberships plus preserved
|
|
railiance01-only users and tenant groups; disposable source lifecycle users
|
|
were excluded. privacyIDEA database counts and all matching custody
|
|
fingerprints agree. Twelve nonempty interhub source-table digests agree while
|
|
the newer destination schema, migration ledger, and destination-only user were
|
|
preserved. SQLite integrity, privacyIDEA HTTP, deployment rollout, and
|
|
post-restart semantic probes passed. Transient artifacts were removed and
|
|
CoulombCore writers remain frozen against divergence.
|
|
|
|
## T04 - Align configuration, secrets, and internal dependencies
|
|
|
|
```task
|
|
id: NK-WP-0022-T04
|
|
status: done
|
|
priority: high
|
|
state_hub_task_id: "33591ee3-586e-4ac6-82e6-0ab4642cea5e"
|
|
```
|
|
|
|
Reconcile KeyCape, Authelia, LLDAP, and privacyIDEA configuration and secret
|
|
references on railiance01 through approved custody lanes. Verify KeyCape uses
|
|
only railiance01 cluster-local dependencies, signing-key continuity is
|
|
preserved, service clients remain registered, and no runtime dependency
|
|
points back to CoulombCore.
|
|
|
|
Done when an automated dependency graph and negative search prove there are
|
|
no hidden CoulombCore runtime dependencies.
|
|
|
|
2026-07-28: added
|
|
`sso-mfa/k8s/verify-identity-cutover-dependencies.sh`. Its live railiance01
|
|
run found no CoulombCore IP/host or public cross-cluster endpoint in decoded
|
|
identity Secrets, ConfigMaps, Deployments, or StatefulSets. It verified
|
|
cluster-local KeyCape and privacyIDEA dependencies, retained user-engine and
|
|
rapp-qonto clients, and Ready deployments. KeyCape signing fingerprints
|
|
match. TLS-preserving direct resolution returned HTTP 200 with valid
|
|
certificates for `auth`, `login`, `lldap`, `pink`, and `pink-account`.
|
|
|
|
## T05 - Run full pre-cutover identity conformance
|
|
|
|
```task
|
|
id: NK-WP-0022-T05
|
|
status: progress
|
|
priority: high
|
|
state_hub_task_id: "5b01e471-afd0-479d-8b35-66eef7820cf6"
|
|
```
|
|
|
|
Using TLS-preserving direct resolution to railiance01, verify health,
|
|
discovery/JWKS, authorization-code + PKCE login, MFA, tenant/group/role
|
|
claims, OpenBao OIDC login, `rapp-qonto` client credentials, wrong-secret and
|
|
wrong-scope denial, logout/session behavior, and restart recovery. Include
|
|
the platform-root and Binky tenant paths without exposing tokens.
|
|
|
|
Done when all critical human and service flows pass and rollback criteria are
|
|
machine-readable.
|
|
|
|
2026-07-28 progress: the portal starts a KeyCape authorization-code flow with
|
|
S256 PKCE, and KeyCape discovery/TLS is healthy at its canonical
|
|
`kc.coulomb.social` issuer. `rapp-qonto-client` obtains a token only for
|
|
`qonto:read`; a wrong secret returns 401 and excessive `qonto:write` returns
|
|
400 without a token. The Binky administrator was created and provisioned
|
|
through user-engine's deployed versioned API, producing a linked LLDAP
|
|
identity and `password_setup_required` state. This exposed and fixed newline
|
|
normalization for the portal's trusted proxy secret in user-engine
|
|
`0ef2ae5`; 105 tests pass and the fixed image is live.
|
|
|
|
Human acceptance remains open: LLDAP has no SMTP configuration, so its native
|
|
self-service reset cannot deliver the first-password handoff. No operator-set
|
|
password was substituted. A NetKingdom-owned password-setup or transactional
|
|
mail handoff plus Binky MFA enrollment and final claim/denial checks remain.
|
|
|
|
## T06 - Cut over remaining DNS with monitored rollback
|
|
|
|
```task
|
|
id: NK-WP-0022-T06
|
|
status: progress
|
|
priority: high
|
|
state_hub_task_id: "00bfa48b-28de-4b8e-b3da-715c797beae5"
|
|
```
|
|
|
|
Lower TTL where needed and move `auth.coulomb.social`,
|
|
`lldap.coulomb.social`, `pink.coulomb.social`, and
|
|
`pink-account.coulomb.social` to `92.205.62.239`. Confirm certificate,
|
|
ingress, resolver, and normal-DNS flows from multiple resolvers. Keep
|
|
CoulombCore intact but prevent divergent writes during a defined observation
|
|
window. Roll back DNS if critical authentication or MFA gates fail.
|
|
|
|
Done when normal DNS consistently resolves to railiance01 and the observation
|
|
window passes without fallback traffic or state divergence.
|
|
|
|
2026-07-28: authoritative/public recursive DNS now returns railiance01 for
|
|
`auth.coulomb.social`; `login.coulomb.social` was added as an Authelia ingress
|
|
and trusted certificate alias. The complete portal → KeyCape → Authelia
|
|
redirect reaches railiance01. LLDAP and both privacyIDEA names remain
|
|
intentionally on CoulombCore pending the state and conformance gates above.
|
|
|
|
## T07 - Retire CoulombCore identity workloads reversibly
|
|
|
|
```task
|
|
id: NK-WP-0022-T07
|
|
status: wait
|
|
priority: high
|
|
state_hub_task_id: "75d160e6-ed58-4b89-b870-4a755e1ca5ce"
|
|
```
|
|
|
|
After T01-T06, scale the old KeyCape, Authelia, LLDAP, and privacyIDEA
|
|
deployments to zero and suspend old backup jobs/ingress. Preserve manifests,
|
|
PVCs, database, Secrets, and encrypted final backup during a rollback
|
|
retention period. Confirm no DNS, ingress, client, monitor, or traffic still
|
|
targets the old services.
|
|
|
|
Deletion of old workloads, PVCs, database clusters, or secret material is a
|
|
separate destructive action permitted only after the retention period and an
|
|
explicit recorded approval.
|
|
|
|
Done when CoulombCore serves no identity traffic and remains recoverable
|
|
during the retention window.
|
|
|
|
## T08 - Final deletion and closure
|
|
|
|
```task
|
|
id: NK-WP-0022-T08
|
|
status: wait
|
|
priority: medium
|
|
state_hub_task_id: "9f209f15-64a9-4324-bfa9-3ff21c52fc19"
|
|
```
|
|
|
|
After explicit approval, expiry of the rollback retention window, and one
|
|
successful railiance01 restore/restart drill, remove the retired CoulombCore
|
|
identity resources and stale DNS/monitoring/configuration. Record what was
|
|
deleted, backup retention, recovery limits, final topology, and operational
|
|
runbooks. Run `statehub fix-consistency`.
|
|
|
|
Done when railiance01 is the sole authoritative identity stack, all evidence
|
|
is reconciled, and the workplan is marked finished.
|
|
|
|
## Safety gates
|
|
|
|
- No retirement before a tested backup and restore path exists.
|
|
- No database overwrite before authoritative-source and divergence analysis.
|
|
- No privacyIDEA restore without its matching encryption material.
|
|
- No normal-DNS cutover before direct-resolution end-to-end verification.
|
|
- No PVC/database/Secret deletion as part of the reversible retirement step.
|
|
- Final deletion always requires an explicit human approval distinct from DNS
|
|
cutover approval.
|