From d38ab710c30233d85c4970bc3bd60f1e9bf287c8 Mon Sep 17 00:00:00 2001 From: tegwick Date: Fri, 21 Aug 2026 08:36:28 +0200 Subject: [PATCH] docs: close railiance01 stabilization --- WORK-RECORDS.md | 4 +- ...core-hub-railiance01-cutover-2026-08-21.md | 8 +- ...ub-railiance01-stabilization-2026-08-21.md | 77 +++++++++++++++++++ ...-0011-railiance01-production-relocation.md | 20 ++++- 4 files changed, 102 insertions(+), 7 deletions(-) create mode 100644 docs/evidence/core-hub-railiance01-stabilization-2026-08-21.md diff --git a/WORK-RECORDS.md b/WORK-RECORDS.md index cb4e697..d80ec7a 100644 --- a/WORK-RECORDS.md +++ b/WORK-RECORDS.md @@ -16,7 +16,7 @@ | workplan | CORE-WP-0008 | finished | — | workplans/CORE-WP-0008-api-first-ops-evidence-continuation.md | | workplan | CORE-WP-0009 | finished | — | workplans/CORE-WP-0009-hub-core-adoption-and-integration.md | | workplan | CORE-WP-0010 | proposed | — | workplans/CORE-WP-0010-runtime-absorption-and-archive.md | -| workplan | CORE-WP-0011 | active | — | workplans/CORE-WP-0011-railiance01-production-relocation.md | +| workplan | CORE-WP-0011 | finished | — | workplans/CORE-WP-0011-railiance01-production-relocation.md | | task | CORE-WP-0001-T01 | done | — | workplans/CORE-WP-0001-statehub-bootstrap.md | | task | CORE-WP-0001-T02 | done | — | workplans/CORE-WP-0001-statehub-bootstrap.md | | task | CORE-WP-0001-T03 | done | — | workplans/CORE-WP-0001-statehub-bootstrap.md | @@ -58,4 +58,4 @@ | task | CORE-WP-0011-T02 | done | — | workplans/CORE-WP-0011-railiance01-production-relocation.md | | task | CORE-WP-0011-T03 | done | — | workplans/CORE-WP-0011-railiance01-production-relocation.md | | task | CORE-WP-0011-T04 | done | — | workplans/CORE-WP-0011-railiance01-production-relocation.md | -| task | CORE-WP-0011-T05 | progress | — | workplans/CORE-WP-0011-railiance01-production-relocation.md | +| task | CORE-WP-0011-T05 | done | — | workplans/CORE-WP-0011-railiance01-production-relocation.md | diff --git a/docs/evidence/core-hub-railiance01-cutover-2026-08-21.md b/docs/evidence/core-hub-railiance01-cutover-2026-08-21.md index ee5b71c..afef8d3 100644 --- a/docs/evidence/core-hub-railiance01-cutover-2026-08-21.md +++ b/docs/evidence/core-hub-railiance01-cutover-2026-08-21.md @@ -1,6 +1,6 @@ # Core Hub railiance01 production cutover — 2026-08-21 -Status: production cutover complete; stabilization in progress. +Status: production cutover and stabilization complete. ## Approved boundary @@ -109,3 +109,9 @@ After the DNS edit, rollback is: Never allow both databases to accept production writes. During stabilization, CoulombCore remains intact with database-default read-only mode and railiance01 is the sole writer. + +Stabilization closed successfully and the source runtime was retired +reversibly. See +`docs/evidence/core-hub-railiance01-stabilization-2026-08-21.md` for the +observation snapshots, retained backups, zero-replica source state and residual +ownership. diff --git a/docs/evidence/core-hub-railiance01-stabilization-2026-08-21.md b/docs/evidence/core-hub-railiance01-stabilization-2026-08-21.md new file mode 100644 index 0000000..72b7575 --- /dev/null +++ b/docs/evidence/core-hub-railiance01-stabilization-2026-08-21.md @@ -0,0 +1,77 @@ +# Core Hub railiance01 stabilization closeout — 2026-08-21 + +Status: complete; CoulombCore runtime retired reversibly. + +## Closeout threshold and observations + +The relocation used a bounded stabilization gate: recursive DNS had to remain +fully drained to railiance01 for multiple original TTLs, two independent +consumer snapshots had to pass, the target had to remain Ready without +restarts or 5xx responses, governed secret delivery and TLS had to remain +healthy, and both source and target backups had to be identified before the +old runtime stopped. + +- Cloudflare, Google, Quad9 and the workstation resolver continued returning + `92.205.62.239`. Normal HTTPS health and readiness returned 200 from that + address with successful certificate verification. +- Both the cutover snapshot and closeout snapshot passed the ops-hub public + gate and activity-core's deployed Core Hub resolver. The closeout resolver + passed health, readiness, protected-route denial and the minimum widget-type + count of 26. +- The target request-log snapshot contained 989 HTTP 200 responses and four + expected HTTP 401 responses, with no 5xx response or application + error/exception/traceback match. +- Helm revision 4 runs chart `0.3.2` at the immutable image digest. The API pod + remained Ready with zero restarts; the package live verifier passed after + source retirement. +- All three ExternalSecrets remained `SecretSynced=True`. Certificate + `core-hub-tls` remained Ready through 2026-11-18T22:41:01Z. + +## Retained data and rollback evidence + +- Target CloudNativePG backup `platform-pg-daily-20260821021500` completed at + 2026-08-21T02:15:11Z after the final restore and cutover write. +- CoulombCore logical-backup Job + `cnpg-logical-backup-apps-pg-29787990` completed at + 2026-08-21T02:30:17Z. Its encrypted object-store outputs include + `apps-pg/apps-pg-core_hub-20260821T023005Z.dump.age` and the separate staging + archive. No credential or archive content was read. +- The frozen final-transfer archive SHA-256 remains recorded as + `4015ad622303ba206a46627ba0d03513e1054b95b87247d61cce6cb6b94189e6`. + The unencrypted mode-0600 workstation copies were removed after the two + external backup paths were verified. +- Database `core_hub` remains present on CoulombCore with + `default_transaction_read_only=on`. It is retained data for rollback, not an + active production dependency. + +## Reversible source retirement + +- CoulombCore Deployments `core-hub/core-hub-api` and + `core-hub-staging/core-hub-api` now both specify zero replicas. No source API + pod remains. Services, Ingress, Secrets and databases were not deleted. +- `core-hub-staging-coulombcore` was removed from the operator tunnel map. Its + dedicated supervisor ignored graceful TERM, so only its confirmed supervisor + and SSH child PIDs were force-stopped; local port 18010 is closed. No other + tunnel was changed. +- A forced request to the old public address returns 404, while normal public + health/readiness and the hardened verifier continue passing on railiance01. + +Rollback remains possible while retained artifacts exist: restore the IONOS A +record to `92.205.130.254`, scale `core-hub/core-hub-api` to one, reset database +`core_hub`'s `default_transaction_read_only`, restart the Deployment and verify +authenticated service before disabling target Ingress. Never enable both +writers. The staging Deployment and retired local tunnel are not required for +production rollback. + +## Residual ownership + +- `rapp-core-hub` owns the verified railiance01 Helm package, live verifier, + NetworkPolicies and rollback surface. +- `railiance-platform` / `rapp-postgres` own platform-pg, dynamic database + credentials and target backup/restore operations. +- The CoulombCore shutdown owner may delete the inert namespaces and retained + source database with the host after its retention decision; they no longer + block shutdown. +- `CORE-WP-0010` remains proposed and independently dependent on + `HUB-WP-0004`. This relocation does not authorize repository archival or + runtime absorption. diff --git a/workplans/CORE-WP-0011-railiance01-production-relocation.md b/workplans/CORE-WP-0011-railiance01-production-relocation.md index 2cf866c..d53295b 100644 --- a/workplans/CORE-WP-0011-railiance01-production-relocation.md +++ b/workplans/CORE-WP-0011-railiance01-production-relocation.md @@ -4,7 +4,7 @@ type: workplan title: "Relocate Core Hub production from CoulombCore to railiance01" domain: infotech repo: core-hub -status: active +status: finished owner: codex topic_slug: infotech created: "2026-08-20" @@ -232,7 +232,7 @@ to T05. Current evidence and exact rollback state are in ```task id: CORE-WP-0011-T05 -status: progress +status: done priority: high ``` @@ -251,11 +251,23 @@ CoulombCore Deployment remains Ready for rollback, but database-level `default_transaction_read_only=on` was reverified. Keep the source frozen until the bounded observation window and operator sign-off close this task. +Completed 2026-08-21 after multiple original DNS TTLs and two independent +consumer snapshots. The target recorded 989 successful requests, four expected +401s and no 5xx/error match; its pod stayed Ready with zero restarts, all +ExternalSecrets and TLS remained Ready, and the post-retirement package +verifier passed. Completed target and encrypted source backups were identified +before both CoulombCore Core Hub Deployments were scaled to zero. The obsolete +staging tunnel was removed from configuration and its exact supervisor/SSH +processes were stopped; port 18010 is closed. Source Services, Ingresses, +Secrets and the read-only database remain only as reversible retention +artifacts and no longer block CoulombCore shutdown. Closeout evidence is in +`docs/evidence/core-hub-railiance01-stabilization-2026-08-21.md`. + ## Acceptance - [x] Decommission deadline and live source/target inventory are recorded - [x] Governed rapp package renders and passes policy checks - [x] Shadow restore/reconciliation and consumer smokes pass on railiance01 - [x] Production traffic and tunnel move with explicit approval and rollback -- [ ] Stabilization passes and CoulombCore no longer hosts Core Hub dependencies -- [ ] `CORE-WP-0010` remains independently sequenced behind `HUB-WP-0004` +- [x] Stabilization passes and CoulombCore no longer hosts Core Hub dependencies +- [x] `CORE-WP-0010` remains independently sequenced behind `HUB-WP-0004`