net-kingdom/docs/railiance01-coulombcore-cutover-inventory-2026-07-28.md
tegwick f0bbd0bfce
All checks were successful
CI Smoke / host-smoke (push) Successful in 0s
CI Smoke / container-smoke (push) Successful in 1s
Complete CoulombCore identity backup gate
2026-07-28 09:26:16 +02:00

130 lines
6.6 KiB
Markdown

# Railiance01 / CoulombCore cutover inventory
Captured read-only on 2026-07-28. This inventory intentionally contains no
Secret values, private keys, password hashes, tokens, or database contents.
## Public routing
| Name | Current target | Cutover owner/state |
| --- | --- | --- |
| `kc.coulomb.social` | railiance01 (`92.205.62.239`) | KeyCape active |
| `auth.coulomb.social` | railiance01 (`92.205.62.239`) | Authelia active |
| `login.coulomb.social` | railiance01 (`92.205.62.239`) | Authelia alias active |
| `lldap.coulomb.social` | CoulombCore (`92.205.130.254`) | identity cutover pending |
| `pink.coulomb.social` | CoulombCore (`92.205.130.254`) | identity cutover pending |
| `pink-account.coulomb.social` | CoulombCore (`92.205.130.254`) | identity cutover pending |
| `bao.coulomb.social` | CoulombCore (`92.205.130.254`) | OpenBao-owned cutover |
| `gitea.coulomb.social` | CoulombCore (`92.205.130.254`) | Forgejo/Gitea-owned cutover |
| `hub.coulomb.social` | CoulombCore (`92.205.130.254`) | State Hub-owned cutover |
Public recursive resolvers return railiance01 for both Authelia names. The
certificate at railiance01 is trusted and covers both names. The portal →
KeyCape → Authelia redirect chain terminates on railiance01.
During cutover, cert-manager's HTTP-01 self-check inherited stale node DNS.
The live controller now uses `1.1.1.1:53,8.8.8.8:53` for ACME self-checks and
the expanded certificate was renewed through `cmctl`. The cluster add-on
configuration must codify that resolver flag so a future Helm reconciliation
does not remove it.
## Identity workload comparison
| Component | CoulombCore | railiance01 | Finding |
| --- | --- | --- | --- |
| KeyCape | `main-nonce-0601` | `key-cape:909bb32` | New runtime is ahead; both healthy |
| Authelia | `authelia:4.38` | `authelia:4.38` | Same declared image |
| LLDAP | `lldap/lldap:stable` | `lldap/lldap:stable` | Independent PVCs; content differs |
| privacyIDEA | `3.12.2` | `3.12.2` | Independent data/log PVCs |
| NetKingdom PostgreSQL | 10 Gi PVC | 10 Gi PVC | Independent clusters require semantic comparison |
| Identity provisioner | absent | `identity-provisioner:dbf7cfd` | railiance01-only lifecycle adapter |
| User engine | absent | `user-engine:portal-e23674d` | railiance01-only control surface |
Both KeyCape endpoints serve the same public JWKS fingerprint:
`8e3237da6030c6af91c5005d0112a149204bc549f67d2234939c5581fdb95a32`.
This proves current public signing-key continuity, not full configuration
equivalence.
## Persistent state and backup observations
- Both LLDAP databases are 139264 bytes, but their SHA-256 fingerprints differ:
CoulombCore `b3be5eed...b91278`; railiance01 `fc538cc8...03dab`.
Equal size is not equivalence. Railiance01 also has newer writes from the
provisioning conformance path.
- railiance01 runs daily LLDAP, Authelia, and privacyIDEA backup CronJobs and
exposes an LLDAP backup directory on its PVC.
- The queried CoulombCore workload inventory exposed daily logical PostgreSQL
backup CronJobs, but no matching LLDAP, Authelia, or privacyIDEA backup
CronJobs. A fresh source-side protected backup is therefore a hard gate.
- Both servers retain independent LLDAP, Authelia, privacyIDEA, and
NetKingdom PostgreSQL PVCs. No DNS move for LLDAP/privacyIDEA is evidence of
data migration.
- CoulombCore still runs KeyCape and Authelia ingress even though their public
DNS moved. Keep these intact only for the bounded rollback window; prevent
new authoritative writes once state migration begins.
## Encrypted source backup
A fresh source backup set was created on CoulombCore at
`~/identity-cutover-backups/2026-07-28T0100Z`, encrypted directly to the
existing SOPS/age custody recipient. The directory is mode `0700`; encrypted
artifacts and `SHA256SUMS` are mode `0600`.
The set contains:
- hot SQLite backups for LLDAP and Authelia;
- a complete `pg_dumpall` stream for `net-kingdom-pg`;
- privacyIDEA encryption/signing custody files;
- encrypted Kubernetes identity Secrets and ConfigMaps;
- encrypted workload/recovery manifests.
All encrypted checksums pass. Every artifact decrypts with the custody key;
both restored SQLite databases return `pragma integrity_check = ok`. Plaintext
verification used `/dev/shm` or streams and was removed, and the temporary
export pod was deleted.
The encrypted set was copied to independent custody on railiance01 at
`~/identity-cutover-backups/coulombcore/2026-07-28T0100Z`. The destination
directory is mode `0700`, every file is mode `0600`, and all source hashes
match. The original checksum manifest contains absolute CoulombCore paths, so
the independent-copy check preserved the manifest and resolved each entry by
basename.
An isolated restore drill started a temporary PostgreSQL 16 pod with no
service, ingress, or persistent volume. The decrypted dump recreated
`interhub` and `privacyidea_db`; the restored privacyIDEA database contained
the expected one token, one token owner, two realms, two resolvers, and two
policies. The pod and transient restore log were deleted automatically.
## Semantic comparison
| State | CoulombCore | railiance01 | Result |
| --- | ---: | ---: | --- |
| LLDAP users | 4 | 3 | divergent |
| LLDAP groups | 7 | 7 | stable-identifier digest divergent |
| LLDAP memberships | 7 | 3 | divergent |
| privacyIDEA tokens | 1 | 0 | critical migration required |
| privacyIDEA token owners | 1 | 0 | critical migration required |
| privacyIDEA realms | 2 | 0 | configuration migration required |
| privacyIDEA resolvers | 2 | 0 | configuration migration required |
| privacyIDEA policies | 2 | 0 | configuration migration required |
Authelia schemas match. Session, consent, access-token, authorization-code,
and audit counts differ because both instances have been independently active;
those ephemeral sessions should not be merged as authoritative identity data.
Every compared privacyIDEA custody fingerprint (`enckey`, signing keypair, and
instance UUID) differs. Migrating only database rows would make the enrolled
token unusable. The CoulombCore privacyIDEA database plus its encryption and
signing custody must move as one rollback-tested unit.
## Required next evidence
1. Copy the encrypted backup set to independent custody and perform an
isolated PostgreSQL/privacyIDEA restore drill.
2. Decide the LLDAP merge rule for the one source-only user and four
source-only memberships while preserving railiance01-only lifecycle groups.
3. Freeze writes, migrate privacyIDEA database and custody as one unit, and
migrate/merge authoritative LLDAP state.
4. Rerun semantic counts/digests and restart recovery probes.
5. Exercise platform-root and Binky login/MFA.
6. Move LLDAP and privacyIDEA DNS only after those gates pass.