core-hub/docs/evidence/core-hub-railiance01-stabilization-2026-08-21.md
tegwick d38ab710c3
Some checks failed
CI Smoke / host-smoke (push) Successful in 0s
CI Smoke / pytest-smoke (push) Failing after 2s
docs: close railiance01 stabilization
2026-08-21 08:36:28 +02:00

4 KiB

Core Hub railiance01 stabilization closeout — 2026-08-21

Status: complete; CoulombCore runtime retired reversibly.

Closeout threshold and observations

The relocation used a bounded stabilization gate: recursive DNS had to remain fully drained to railiance01 for multiple original TTLs, two independent consumer snapshots had to pass, the target had to remain Ready without restarts or 5xx responses, governed secret delivery and TLS had to remain healthy, and both source and target backups had to be identified before the old runtime stopped.

  • Cloudflare, Google, Quad9 and the workstation resolver continued returning 92.205.62.239. Normal HTTPS health and readiness returned 200 from that address with successful certificate verification.
  • Both the cutover snapshot and closeout snapshot passed the ops-hub public gate and activity-core's deployed Core Hub resolver. The closeout resolver passed health, readiness, protected-route denial and the minimum widget-type count of 26.
  • The target request-log snapshot contained 989 HTTP 200 responses and four expected HTTP 401 responses, with no 5xx response or application error/exception/traceback match.
  • Helm revision 4 runs chart 0.3.2 at the immutable image digest. The API pod remained Ready with zero restarts; the package live verifier passed after source retirement.
  • All three ExternalSecrets remained SecretSynced=True. Certificate core-hub-tls remained Ready through 2026-11-18T22:41:01Z.

Retained data and rollback evidence

  • Target CloudNativePG backup platform-pg-daily-20260821021500 completed at 2026-08-21T02:15:11Z after the final restore and cutover write.
  • CoulombCore logical-backup Job cnpg-logical-backup-apps-pg-29787990 completed at 2026-08-21T02:30:17Z. Its encrypted object-store outputs include apps-pg/apps-pg-core_hub-20260821T023005Z.dump.age and the separate staging archive. No credential or archive content was read.
  • The frozen final-transfer archive SHA-256 remains recorded as 4015ad622303ba206a46627ba0d03513e1054b95b87247d61cce6cb6b94189e6. The unencrypted mode-0600 workstation copies were removed after the two external backup paths were verified.
  • Database core_hub remains present on CoulombCore with default_transaction_read_only=on. It is retained data for rollback, not an active production dependency.

Reversible source retirement

  • CoulombCore Deployments core-hub/core-hub-api and core-hub-staging/core-hub-api now both specify zero replicas. No source API pod remains. Services, Ingress, Secrets and databases were not deleted.
  • core-hub-staging-coulombcore was removed from the operator tunnel map. Its dedicated supervisor ignored graceful TERM, so only its confirmed supervisor and SSH child PIDs were force-stopped; local port 18010 is closed. No other tunnel was changed.
  • A forced request to the old public address returns 404, while normal public health/readiness and the hardened verifier continue passing on railiance01.

Rollback remains possible while retained artifacts exist: restore the IONOS A record to 92.205.130.254, scale core-hub/core-hub-api to one, reset database core_hub's default_transaction_read_only, restart the Deployment and verify authenticated service before disabling target Ingress. Never enable both writers. The staging Deployment and retired local tunnel are not required for production rollback.

Residual ownership

  • rapp-core-hub owns the verified railiance01 Helm package, live verifier, NetworkPolicies and rollback surface.
  • railiance-platform / rapp-postgres own platform-pg, dynamic database credentials and target backup/restore operations.
  • The CoulombCore shutdown owner may delete the inert namespaces and retained source database with the host after its retention decision; they no longer block shutdown.
  • CORE-WP-0010 remains proposed and independently dependent on HUB-WP-0004. This relocation does not authorize repository archival or runtime absorption.