diff --git a/WORK-RECORDS.md b/WORK-RECORDS.md index ef9aa8f..cb4e697 100644 --- a/WORK-RECORDS.md +++ b/WORK-RECORDS.md @@ -57,5 +57,5 @@ | task | CORE-WP-0011-T01 | done | — | workplans/CORE-WP-0011-railiance01-production-relocation.md | | task | CORE-WP-0011-T02 | done | — | workplans/CORE-WP-0011-railiance01-production-relocation.md | | task | CORE-WP-0011-T03 | done | — | workplans/CORE-WP-0011-railiance01-production-relocation.md | -| task | CORE-WP-0011-T04 | progress | — | workplans/CORE-WP-0011-railiance01-production-relocation.md | -| task | CORE-WP-0011-T05 | todo | — | workplans/CORE-WP-0011-railiance01-production-relocation.md | +| task | CORE-WP-0011-T04 | done | — | workplans/CORE-WP-0011-railiance01-production-relocation.md | +| task | CORE-WP-0011-T05 | progress | — | workplans/CORE-WP-0011-railiance01-production-relocation.md | diff --git a/docs/evidence/core-hub-railiance01-cutover-2026-08-21.md b/docs/evidence/core-hub-railiance01-cutover-2026-08-21.md index 41b40b2..ee5b71c 100644 --- a/docs/evidence/core-hub-railiance01-cutover-2026-08-21.md +++ b/docs/evidence/core-hub-railiance01-cutover-2026-08-21.md @@ -1,6 +1,6 @@ # Core Hub railiance01 production cutover — 2026-08-21 -Status: in progress; waiting for the operator-owned IONOS DNS edit. +Status: production cutover complete; stabilization in progress. ## Approved boundary @@ -39,21 +39,73 @@ stabilization gate. - The railiance01 Deployment is Ready. The hardened package verifier passes. - Direct target resolution returns health/readiness 200, protected hubs 401 without authentication, and 200 with the governed token. -- The Ingress is present at `92.205.62.239`. Certificate issuance is pending - because the IONOS A record still resolves the HTTP-01 challenge to the - source host. +- The Ingress is present at `92.205.62.239`. + +## DNS, TLS and public verification + +- The operator changed the IONOS A record for `hub.coulomb.social` from + `92.205.130.254` to `92.205.62.239`. All four authoritative IONOS name + servers returned the target, as did Google and Quad9. Cloudflare still held + the source address under the previously observed 3600-second TTL at the + completion check; source therefore remains available but read-only while + residual caches drain. +- The namespace-wide default-deny policy initially denied Traefik access to + cert-manager's solver. A temporary NetworkPolicy selected only + `acme.cert-manager.io/http01-solver=true`, allowed only kube-system ingress + on TCP 8089, and was removed after issuance. Because Cloudflare was still + cached, a temporary CoulombCore Ingress forwarded only + `/.well-known/acme-challenge/` to the target solver; it too was removed after + issuance. No application route or write policy was opened. +- Certificate `core-hub-tls` is Ready. The Let's Encrypt certificate has SAN + `hub.coulomb.social`, verifies successfully, and is valid from + 2026-08-20 22:41:02 UTC through 2026-11-18 22:41:01 UTC. +- Target-forced HTTPS checks returned 200 for `/healthz`, `/readyz`, + `/api/v2/widget-types`, `/api/v2/event-types`, + `/api/v2/annotation-categories`, and `/api/v2/policy-scopes`. + `/api/v2/hubs` and `/console` returned 401 without authentication. +- Governed-token checks against the target returned 200 for hubs and the HTML + operator console. The first post-cutover interaction-event write returned + 201, then listed back successfully as + `e56bca14-51f8-4a73-bdab-f660e360fb14`; the event contains only cutover + metadata and references a migrated widget. +- The ops-hub `interhub-gate` probe passed from railiance01: protected hubs + returned 401 and all required OpenAPI paths were present. The deployed + activity-core `core-hub` resolver passed health, readiness, protected hubs, + widget-types status, and the minimum catalog count of 26 through the public + production hostname. +- Inspection found no active ops-bridge tunnel in the public request path. + Public traffic moves by the direct IONOS A record. The workstation's obsolete + `core-hub-staging-coulombcore` tunnel remains a T05 cleanup item. + +## Stabilization entry snapshot + +- Cloudflare (`1.1.1.1`, `1.0.0.1`), Google (`8.8.8.8`), Quad9 (`9.9.9.9`) + and the workstation resolver now all return `92.205.62.239` with TTL 3600. + A normal, non-forced HTTPS health request reaches that address with status + 200 and successful certificate verification. +- Package chart `0.3.2` was deployed as Helm revision 4 after 10 tests, Helm + lint, deterministic render and server dry-run passed. Its permanent + `core-hub-acme-http01` NetworkPolicy selects only cert-manager's solver label + and permits only kube-system ingress on TCP 8089. The hardened live verifier + and external TLS health check pass after rollout. +- Certificate `core-hub-tls` remains Ready through + 2026-11-18T22:41:01Z. CoulombCore's Deployment remains Ready as the rollback + runtime, and the `core_hub` database setting was reverified as + `default_transaction_read_only=on`. ## Current rollback -Before the DNS edit, rollback is: +After the DNS edit, rollback is: -1. Roll railiance01 Helm release `core-hub` back from revision 3 to revision 2 - to remove the target Ingress. +1. Restore the IONOS A record to `92.205.130.254` and verify authoritative plus + recursive resolution before enabling source writes. 2. On CoulombCore, reset database `core_hub`'s `default_transaction_read_only` setting and restart Deployment `core-hub-api` so the application reconnects writable. 3. Verify public health, readiness and an authenticated protected operation. +4. Only after source service is restored, roll railiance01 Helm release + `core-hub` back to the last private revision to remove target Ingress. -After DNS changes, restore `hub.coulomb.social` to `92.205.130.254` before -reenabling source writes. Never allow both databases to accept production -writes. +Never allow both databases to accept production writes. During stabilization, +CoulombCore remains intact with database-default read-only mode and +railiance01 is the sole writer. diff --git a/workplans/CORE-WP-0011-railiance01-production-relocation.md b/workplans/CORE-WP-0011-railiance01-production-relocation.md index 55442d2..2cf866c 100644 --- a/workplans/CORE-WP-0011-railiance01-production-relocation.md +++ b/workplans/CORE-WP-0011-railiance01-production-relocation.md @@ -184,7 +184,7 @@ evidence is in ```task id: CORE-WP-0011-T04 -status: progress +status: done priority: high ``` @@ -208,18 +208,31 @@ table counts and order-independent row-content digests match; Alembic `20260627_0002`, migration, the hardened verifier, direct target health/ readiness, unauthenticated denial, and authenticated protected read pass. -The cutover is paused only for the IONOS `hub.coulomb.social` A-record change -from `92.205.130.254` to `92.205.62.239`. Target cert-manager is waiting on -that HTTP-01 propagation. Do not restore source writes until the DNS/TLS/public -smoke decision either completes or rolls back. Current evidence and exact -rollback state are in +Completed 2026-08-21 after the operator changed the authoritative IONOS A +record from `92.205.130.254` to `92.205.62.239`. All four authoritative name +servers and Google/Quad9 returned railiance01; Cloudflare retained the old +address under its pre-change TTL, so CoulombCore remains read-only while that +cache drains. A temporary, challenge-path-only bridge on CoulombCore and a +solver-label/port-specific target NetworkPolicy allowed cert-manager to issue +the production certificate without reopening either application or database +access; both temporary resources were removed immediately afterward. + +The Let's Encrypt certificate verifies for `hub.coulomb.social`. Forced target +checks passed health/readiness, all four public catalogs, protected-route and +operator-console auth boundaries, authenticated reads, and the hardened live +verifier. The first post-cutover write returned 201 and read back as event +`e56bca14-51f8-4a73-bdab-f660e360fb14`. The ops-hub public gate and the +deployed activity-core stabilization resolver both passed against railiance01. +No active ops-bridge tunnel fronts the public hostname; it is a direct IONOS +DNS route. The obsolete local `core-hub-staging-coulombcore` tunnel is deferred +to T05. Current evidence and exact rollback state are in `docs/evidence/core-hub-railiance01-cutover-2026-08-21.md`. ## Stabilize and release CoulombCore dependencies ```task id: CORE-WP-0011-T05 -status: todo +status: progress priority: high ``` @@ -229,11 +242,20 @@ critical path, retire `core-hub-staging` and any obsolete tunnel there, and record retained backup/rollback evidence and residual owners. Do not archive this repo unless the separate absorption workplan has also completed. +Stabilization started 2026-08-21. Cloudflare, Google, Quad9 and the local +resolver now all return `92.205.62.239`; normal public HTTPS reaches +railiance01 with valid TLS. Helm revision 4 runs package chart `0.3.2`, which +permanently carries the solver-label/kube-system/TCP-8089 NetworkPolicy needed +for future certificate renewals, and the post-upgrade verifier passes. The +CoulombCore Deployment remains Ready for rollback, but database-level +`default_transaction_read_only=on` was reverified. Keep the source frozen until +the bounded observation window and operator sign-off close this task. + ## Acceptance - [x] Decommission deadline and live source/target inventory are recorded - [x] Governed rapp package renders and passes policy checks - [x] Shadow restore/reconciliation and consumer smokes pass on railiance01 -- [ ] Production traffic and tunnel move with explicit approval and rollback +- [x] Production traffic and tunnel move with explicit approval and rollback - [ ] Stabilization passes and CoulombCore no longer hosts Core Hub dependencies - [ ] `CORE-WP-0010` remains independently sequenced behind `HUB-WP-0004`