docs: close railiance01 stabilization
Some checks failed
CI Smoke / host-smoke (push) Successful in 0s
CI Smoke / pytest-smoke (push) Failing after 2s

This commit is contained in:
tegwick 2026-08-21 08:36:28 +02:00
parent c13834e365
commit d38ab710c3
4 changed files with 102 additions and 7 deletions

View file

@ -1,6 +1,6 @@
# Core Hub railiance01 production cutover — 2026-08-21
Status: production cutover complete; stabilization in progress.
Status: production cutover and stabilization complete.
## Approved boundary
@ -109,3 +109,9 @@ After the DNS edit, rollback is:
Never allow both databases to accept production writes. During stabilization,
CoulombCore remains intact with database-default read-only mode and
railiance01 is the sole writer.
Stabilization closed successfully and the source runtime was retired
reversibly. See
`docs/evidence/core-hub-railiance01-stabilization-2026-08-21.md` for the
observation snapshots, retained backups, zero-replica source state and residual
ownership.

View file

@ -0,0 +1,77 @@
# Core Hub railiance01 stabilization closeout — 2026-08-21
Status: complete; CoulombCore runtime retired reversibly.
## Closeout threshold and observations
The relocation used a bounded stabilization gate: recursive DNS had to remain
fully drained to railiance01 for multiple original TTLs, two independent
consumer snapshots had to pass, the target had to remain Ready without
restarts or 5xx responses, governed secret delivery and TLS had to remain
healthy, and both source and target backups had to be identified before the
old runtime stopped.
- Cloudflare, Google, Quad9 and the workstation resolver continued returning
`92.205.62.239`. Normal HTTPS health and readiness returned 200 from that
address with successful certificate verification.
- Both the cutover snapshot and closeout snapshot passed the ops-hub public
gate and activity-core's deployed Core Hub resolver. The closeout resolver
passed health, readiness, protected-route denial and the minimum widget-type
count of 26.
- The target request-log snapshot contained 989 HTTP 200 responses and four
expected HTTP 401 responses, with no 5xx response or application
error/exception/traceback match.
- Helm revision 4 runs chart `0.3.2` at the immutable image digest. The API pod
remained Ready with zero restarts; the package live verifier passed after
source retirement.
- All three ExternalSecrets remained `SecretSynced=True`. Certificate
`core-hub-tls` remained Ready through 2026-11-18T22:41:01Z.
## Retained data and rollback evidence
- Target CloudNativePG backup `platform-pg-daily-20260821021500` completed at
2026-08-21T02:15:11Z after the final restore and cutover write.
- CoulombCore logical-backup Job
`cnpg-logical-backup-apps-pg-29787990` completed at
2026-08-21T02:30:17Z. Its encrypted object-store outputs include
`apps-pg/apps-pg-core_hub-20260821T023005Z.dump.age` and the separate staging
archive. No credential or archive content was read.
- The frozen final-transfer archive SHA-256 remains recorded as
`4015ad622303ba206a46627ba0d03513e1054b95b87247d61cce6cb6b94189e6`.
The unencrypted mode-0600 workstation copies were removed after the two
external backup paths were verified.
- Database `core_hub` remains present on CoulombCore with
`default_transaction_read_only=on`. It is retained data for rollback, not an
active production dependency.
## Reversible source retirement
- CoulombCore Deployments `core-hub/core-hub-api` and
`core-hub-staging/core-hub-api` now both specify zero replicas. No source API
pod remains. Services, Ingress, Secrets and databases were not deleted.
- `core-hub-staging-coulombcore` was removed from the operator tunnel map. Its
dedicated supervisor ignored graceful TERM, so only its confirmed supervisor
and SSH child PIDs were force-stopped; local port 18010 is closed. No other
tunnel was changed.
- A forced request to the old public address returns 404, while normal public
health/readiness and the hardened verifier continue passing on railiance01.
Rollback remains possible while retained artifacts exist: restore the IONOS A
record to `92.205.130.254`, scale `core-hub/core-hub-api` to one, reset database
`core_hub`'s `default_transaction_read_only`, restart the Deployment and verify
authenticated service before disabling target Ingress. Never enable both
writers. The staging Deployment and retired local tunnel are not required for
production rollback.
## Residual ownership
- `rapp-core-hub` owns the verified railiance01 Helm package, live verifier,
NetworkPolicies and rollback surface.
- `railiance-platform` / `rapp-postgres` own platform-pg, dynamic database
credentials and target backup/restore operations.
- The CoulombCore shutdown owner may delete the inert namespaces and retained
source database with the host after its retention decision; they no longer
block shutdown.
- `CORE-WP-0010` remains proposed and independently dependent on
`HUB-WP-0004`. This relocation does not authorize repository archival or
runtime absorption.