net-kingdom/docs/railiance01-coulombcore-cutover-inventory-2026-07-28.md
tegwick 9497529ed2
All checks were successful
CI Smoke / host-smoke (push) Successful in 0s
CI Smoke / container-smoke (push) Successful in 1s
Reconcile identity state onto railiance01
2026-07-28 12:11:45 +02:00

8.7 KiB

Railiance01 / CoulombCore cutover inventory

Captured read-only on 2026-07-28. This inventory intentionally contains no Secret values, private keys, password hashes, tokens, or database contents.

Public routing

Name Current target Cutover owner/state
kc.coulomb.social railiance01 (92.205.62.239) KeyCape active
auth.coulomb.social railiance01 (92.205.62.239) Authelia active
login.coulomb.social railiance01 (92.205.62.239) Authelia alias active
lldap.coulomb.social CoulombCore (92.205.130.254) identity cutover pending
pink.coulomb.social CoulombCore (92.205.130.254) identity cutover pending
pink-account.coulomb.social CoulombCore (92.205.130.254) identity cutover pending
bao.coulomb.social CoulombCore (92.205.130.254) OpenBao-owned cutover
gitea.coulomb.social CoulombCore (92.205.130.254) Forgejo/Gitea-owned cutover
hub.coulomb.social CoulombCore (92.205.130.254) State Hub-owned cutover

Public recursive resolvers return railiance01 for both Authelia names. The certificate at railiance01 is trusted and covers both names. The portal → KeyCape → Authelia redirect chain terminates on railiance01.

During cutover, cert-manager's HTTP-01 self-check inherited stale node DNS. The live controller now uses 1.1.1.1:53,8.8.8.8:53 for ACME self-checks and the expanded certificate was renewed through cmctl. The cluster add-on configuration must codify that resolver flag so a future Helm reconciliation does not remove it.

Identity workload comparison

Component CoulombCore railiance01 Finding
KeyCape main-nonce-0601 key-cape:909bb32 New runtime is ahead; both healthy
Authelia authelia:4.38 authelia:4.38 Same declared image
LLDAP lldap/lldap:stable lldap/lldap:stable Independent PVCs; content differs
privacyIDEA 3.12.2 3.12.2 Independent data/log PVCs
NetKingdom PostgreSQL 10 Gi PVC 10 Gi PVC Independent clusters require semantic comparison
Identity provisioner absent identity-provisioner:dbf7cfd railiance01-only lifecycle adapter
User engine absent user-engine:portal-e23674d railiance01-only control surface

Both KeyCape endpoints serve the same public JWKS fingerprint: 8e3237da6030c6af91c5005d0112a149204bc549f67d2234939c5581fdb95a32. This proves current public signing-key continuity, not full configuration equivalence.

Persistent state and backup observations

  • Both LLDAP databases are 139264 bytes, but their SHA-256 fingerprints differ: CoulombCore b3be5eed...b91278; railiance01 fc538cc8...03dab. Equal size is not equivalence. Railiance01 also has newer writes from the provisioning conformance path.
  • railiance01 runs daily LLDAP, Authelia, and privacyIDEA backup CronJobs and exposes an LLDAP backup directory on its PVC.
  • The queried CoulombCore workload inventory exposed daily logical PostgreSQL backup CronJobs, but no matching LLDAP, Authelia, or privacyIDEA backup CronJobs. A fresh source-side protected backup is therefore a hard gate.
  • Both servers retain independent LLDAP, Authelia, privacyIDEA, and NetKingdom PostgreSQL PVCs. No DNS move for LLDAP/privacyIDEA is evidence of data migration.
  • CoulombCore still runs KeyCape and Authelia ingress even though their public DNS moved. Keep these intact only for the bounded rollback window; prevent new authoritative writes once state migration begins.

Encrypted source backup

A fresh source backup set was created on CoulombCore at ~/identity-cutover-backups/2026-07-28T0100Z, encrypted directly to the existing SOPS/age custody recipient. The directory is mode 0700; encrypted artifacts and SHA256SUMS are mode 0600.

The set contains:

  • hot SQLite backups for LLDAP and Authelia;
  • a complete pg_dumpall stream for net-kingdom-pg;
  • privacyIDEA encryption/signing custody files;
  • encrypted Kubernetes identity Secrets and ConfigMaps;
  • encrypted workload/recovery manifests.

All encrypted checksums pass. Every artifact decrypts with the custody key; both restored SQLite databases return pragma integrity_check = ok. Plaintext verification used /dev/shm or streams and was removed, and the temporary export pod was deleted.

The encrypted set was copied to independent custody on railiance01 at ~/identity-cutover-backups/coulombcore/2026-07-28T0100Z. The destination directory is mode 0700, every file is mode 0600, and all source hashes match. The original checksum manifest contains absolute CoulombCore paths, so the independent-copy check preserved the manifest and resolved each entry by basename.

An isolated restore drill started a temporary PostgreSQL 16 pod with no service, ingress, or persistent volume. The decrypted dump recreated interhub and privacyidea_db; the restored privacyIDEA database contained the expected one token, one token owner, two realms, two resolvers, and two policies. The pod and transient restore log were deleted automatically.

Semantic comparison

State CoulombCore railiance01 Result
LLDAP users 4 3 divergent
LLDAP groups 7 7 stable-identifier digest divergent
LLDAP memberships 7 3 divergent
privacyIDEA tokens 1 0 critical migration required
privacyIDEA token owners 1 0 critical migration required
privacyIDEA realms 2 0 configuration migration required
privacyIDEA resolvers 2 0 configuration migration required
privacyIDEA policies 2 0 configuration migration required

Authelia schemas match. Session, consent, access-token, authorization-code, and audit counts differ because both instances have been independently active; those ephemeral sessions should not be merged as authoritative identity data.

Every compared privacyIDEA custody fingerprint (enckey, signing keypair, and instance UUID) differs. Migrating only database rows would make the enrolled token unusable. The CoulombCore privacyIDEA database plus its encryption and signing custody must move as one rollback-tested unit.

Persistent-state reconciliation

T03 used a bounded write freeze covering KeyCape, LLDAP, privacyIDEA, and user-engine where present. Fresh encrypted frozen snapshots were created at:

  • CoulombCore: ~/identity-cutover-backups/2026-07-28T0925Z-pre-migration-source;
  • railiance01: ~/identity-cutover-backups/2026-07-28T0925Z-pre-migration-destination.

Both sets contain LLDAP, privacyIDEA database and custody, and interhub rollback artifacts with passing relative-path checksum manifests.

LLDAP was reconciled as a selective union. The CoulombCore platform-root row, attributes, and memberships are authoritative. railiance01's bootstrap admin, destination test user, and Binky tenant groups remain authoritative. Shared groups were mapped by normalized group name, and the source-only whynot-design and activity-core-operators groups were added with collision-free destination IDs. The known --cleanup-only and t06-final-* lifecycle-test identities were deliberately excluded.

Post-restart LLDAP evidence is:

  • SQLite integrity: ok;
  • users/groups/memberships: 3/9/6;
  • platform-root memberships: 4;
  • destination test user preserved: 1;
  • excluded source test artifacts present: 0.

The complete CoulombCore privacyIDEA database and its matching encryption key, audit signing keypair, and instance UUID were applied as one unit. All four custody fingerprints match the source. Source and destination now both report one token, one token owner, two realms, two resolvers, and two policies. privacyIDEA returned HTTP 200 after restart.

CoulombCore interhub data was loaded into railiance01's newer schema rather than replacing that schema. A legacy hubs.hub_family column was intentionally discarded after the first data-only restore rejected the mismatch and the encrypted destination snapshot was restored. All twelve nonempty source-table row counts and stable row digests match, including a common-column digest for hubs. railiance01's 16-row migration ledger and one destination-only user were preserved.

Temporary helper pods, shared-memory databases, dumps, archives, and restore logs were removed. CoulombCore identity writers remain scaled to zero to prevent divergence while the railiance01 direct-resolution conformance gates run; no PVC, Secret, database, or workload declaration was deleted.

Required next evidence

  1. Reconcile cluster-local configuration and prove that railiance01 has no runtime dependency on CoulombCore.
  2. Exercise platform-root and Binky login/MFA using TLS-preserving direct resolution to railiance01.
  3. Verify KeyCape service-client flows and negative authorization probes.
  4. Move LLDAP and privacyIDEA DNS only after those gates pass.