docs: complete railiance01 production cutover

WORK-RECORDS.md

@@ -57,5 +57,5 @@
 | task | CORE-WP-0011-T01 | done | — | workplans/CORE-WP-0011-railiance01-production-relocation.md |
 | task | CORE-WP-0011-T02 | done | — | workplans/CORE-WP-0011-railiance01-production-relocation.md |
 | task | CORE-WP-0011-T03 | done | — | workplans/CORE-WP-0011-railiance01-production-relocation.md |
-| task | CORE-WP-0011-T04 | progress | — | workplans/CORE-WP-0011-railiance01-production-relocation.md |
-| task | CORE-WP-0011-T05 | todo | — | workplans/CORE-WP-0011-railiance01-production-relocation.md |
+| task | CORE-WP-0011-T04 | done | — | workplans/CORE-WP-0011-railiance01-production-relocation.md |
+| task | CORE-WP-0011-T05 | progress | — | workplans/CORE-WP-0011-railiance01-production-relocation.md |

docs/evidence/core-hub-railiance01-cutover-2026-08-21.md

@@ -1,6 +1,6 @@
 # Core Hub railiance01 production cutover — 2026-08-21
 
-Status: in progress; waiting for the operator-owned IONOS DNS edit.
+Status: production cutover complete; stabilization in progress.
 
 ## Approved boundary
 
@@ -39,21 +39,73 @@ stabilization gate.
 - The railiance01 Deployment is Ready. The hardened package verifier passes.
 - Direct target resolution returns health/readiness 200, protected hubs 401
   without authentication, and 200 with the governed token.
-- The Ingress is present at `92.205.62.239`. Certificate issuance is pending
-  because the IONOS A record still resolves the HTTP-01 challenge to the
-  source host.
+- The Ingress is present at `92.205.62.239`.
+
+## DNS, TLS and public verification
+
+- The operator changed the IONOS A record for `hub.coulomb.social` from
+  `92.205.130.254` to `92.205.62.239`. All four authoritative IONOS name
+  servers returned the target, as did Google and Quad9. Cloudflare still held
+  the source address under the previously observed 3600-second TTL at the
+  completion check; source therefore remains available but read-only while
+  residual caches drain.
+- The namespace-wide default-deny policy initially denied Traefik access to
+  cert-manager's solver. A temporary NetworkPolicy selected only
+  `acme.cert-manager.io/http01-solver=true`, allowed only kube-system ingress
+  on TCP 8089, and was removed after issuance. Because Cloudflare was still
+  cached, a temporary CoulombCore Ingress forwarded only
+  `/.well-known/acme-challenge/` to the target solver; it too was removed after
+  issuance. No application route or write policy was opened.
+- Certificate `core-hub-tls` is Ready. The Let's Encrypt certificate has SAN
+  `hub.coulomb.social`, verifies successfully, and is valid from
+  2026-08-20 22:41:02 UTC through 2026-11-18 22:41:01 UTC.
+- Target-forced HTTPS checks returned 200 for `/healthz`, `/readyz`,
+  `/api/v2/widget-types`, `/api/v2/event-types`,
+  `/api/v2/annotation-categories`, and `/api/v2/policy-scopes`.
+  `/api/v2/hubs` and `/console` returned 401 without authentication.
+- Governed-token checks against the target returned 200 for hubs and the HTML
+  operator console. The first post-cutover interaction-event write returned
+  201, then listed back successfully as
+  `e56bca14-51f8-4a73-bdab-f660e360fb14`; the event contains only cutover
+  metadata and references a migrated widget.
+- The ops-hub `interhub-gate` probe passed from railiance01: protected hubs
+  returned 401 and all required OpenAPI paths were present. The deployed
+  activity-core `core-hub` resolver passed health, readiness, protected hubs,
+  widget-types status, and the minimum catalog count of 26 through the public
+  production hostname.
+- Inspection found no active ops-bridge tunnel in the public request path.
+  Public traffic moves by the direct IONOS A record. The workstation's obsolete
+  `core-hub-staging-coulombcore` tunnel remains a T05 cleanup item.
+
+## Stabilization entry snapshot
+
+- Cloudflare (`1.1.1.1`, `1.0.0.1`), Google (`8.8.8.8`), Quad9 (`9.9.9.9`)
+  and the workstation resolver now all return `92.205.62.239` with TTL 3600.
+  A normal, non-forced HTTPS health request reaches that address with status
+  200 and successful certificate verification.
+- Package chart `0.3.2` was deployed as Helm revision 4 after 10 tests, Helm
+  lint, deterministic render and server dry-run passed. Its permanent
+  `core-hub-acme-http01` NetworkPolicy selects only cert-manager's solver label
+  and permits only kube-system ingress on TCP 8089. The hardened live verifier
+  and external TLS health check pass after rollout.
+- Certificate `core-hub-tls` remains Ready through
+  2026-11-18T22:41:01Z. CoulombCore's Deployment remains Ready as the rollback
+  runtime, and the `core_hub` database setting was reverified as
+  `default_transaction_read_only=on`.
 
 ## Current rollback
 
-Before the DNS edit, rollback is:
+After the DNS edit, rollback is:
 
-1. Roll railiance01 Helm release `core-hub` back from revision 3 to revision 2
-   to remove the target Ingress.
+1. Restore the IONOS A record to `92.205.130.254` and verify authoritative plus
+   recursive resolution before enabling source writes.
 2. On CoulombCore, reset database `core_hub`'s
    `default_transaction_read_only` setting and restart Deployment
    `core-hub-api` so the application reconnects writable.
 3. Verify public health, readiness and an authenticated protected operation.
+4. Only after source service is restored, roll railiance01 Helm release
+   `core-hub` back to the last private revision to remove target Ingress.
 
-After DNS changes, restore `hub.coulomb.social` to `92.205.130.254` before
-reenabling source writes. Never allow both databases to accept production
-writes.
+Never allow both databases to accept production writes. During stabilization,
+CoulombCore remains intact with database-default read-only mode and
+railiance01 is the sole writer.

workplans/CORE-WP-0011-railiance01-production-relocation.md

@@ -184,7 +184,7 @@ evidence is in
 
 ```task
 id: CORE-WP-0011-T04
-status: progress
+status: done
 priority: high
 ```
 
@@ -208,18 +208,31 @@ table counts and order-independent row-content digests match; Alembic
 `20260627_0002`, migration, the hardened verifier, direct target health/
 readiness, unauthenticated denial, and authenticated protected read pass.
 
-The cutover is paused only for the IONOS `hub.coulomb.social` A-record change
-from `92.205.130.254` to `92.205.62.239`. Target cert-manager is waiting on
-that HTTP-01 propagation. Do not restore source writes until the DNS/TLS/public
-smoke decision either completes or rolls back. Current evidence and exact
-rollback state are in
+Completed 2026-08-21 after the operator changed the authoritative IONOS A
+record from `92.205.130.254` to `92.205.62.239`. All four authoritative name
+servers and Google/Quad9 returned railiance01; Cloudflare retained the old
+address under its pre-change TTL, so CoulombCore remains read-only while that
+cache drains. A temporary, challenge-path-only bridge on CoulombCore and a
+solver-label/port-specific target NetworkPolicy allowed cert-manager to issue
+the production certificate without reopening either application or database
+access; both temporary resources were removed immediately afterward.
+
+The Let's Encrypt certificate verifies for `hub.coulomb.social`. Forced target
+checks passed health/readiness, all four public catalogs, protected-route and
+operator-console auth boundaries, authenticated reads, and the hardened live
+verifier. The first post-cutover write returned 201 and read back as event
+`e56bca14-51f8-4a73-bdab-f660e360fb14`. The ops-hub public gate and the
+deployed activity-core stabilization resolver both passed against railiance01.
+No active ops-bridge tunnel fronts the public hostname; it is a direct IONOS
+DNS route. The obsolete local `core-hub-staging-coulombcore` tunnel is deferred
+to T05. Current evidence and exact rollback state are in
 `docs/evidence/core-hub-railiance01-cutover-2026-08-21.md`.
 
 ## Stabilize and release CoulombCore dependencies
 
 ```task
 id: CORE-WP-0011-T05
-status: todo
+status: progress
 priority: high
 ```
 
@@ -229,11 +242,20 @@ critical path, retire `core-hub-staging` and any obsolete tunnel there, and
 record retained backup/rollback evidence and residual owners. Do not archive
 this repo unless the separate absorption workplan has also completed.
 
+Stabilization started 2026-08-21. Cloudflare, Google, Quad9 and the local
+resolver now all return `92.205.62.239`; normal public HTTPS reaches
+railiance01 with valid TLS. Helm revision 4 runs package chart `0.3.2`, which
+permanently carries the solver-label/kube-system/TCP-8089 NetworkPolicy needed
+for future certificate renewals, and the post-upgrade verifier passes. The
+CoulombCore Deployment remains Ready for rollback, but database-level
+`default_transaction_read_only=on` was reverified. Keep the source frozen until
+the bounded observation window and operator sign-off close this task.
+
 ## Acceptance
 
 - [x] Decommission deadline and live source/target inventory are recorded
 - [x] Governed rapp package renders and passes policy checks
 - [x] Shadow restore/reconciliation and consumer smokes pass on railiance01
-- [ ] Production traffic and tunnel move with explicit approval and rollback
+- [x] Production traffic and tunnel move with explicit approval and rollback
 - [ ] Stabilization passes and CoulombCore no longer hosts Core Hub dependencies
 - [ ] `CORE-WP-0010` remains independently sequenced behind `HUB-WP-0004`