docs: record hub-core production cutover

INTENT.md

@@ -38,36 +38,35 @@ The framework must be usable by humans and agents. Human operators need a stable
 
 ## Ecosystem position
 
-`core-hub` is the **production framework layer** in the three-repo hub stack:
+`core-hub` is the **previous production service and temporary rollback** in the
+three-repo hub stack:
 
 | Repo | Role |
 | --- | --- |
 | `hub-core` | Surviving framework/runtime — contracts, ports, package, and primary image |
 | `state-hub` | Dev coordination — file-first workplans, dev MCP |
-| `core-hub` | Production service — `/api/v2`, hubs, manifests, operator console |
+| `core-hub` | Previous production service retained through cutover stabilization |
 
 Canon: `/home/worsch/the-custodian/docs/hub-ecosystem-architecture.md`
 
-**Transition naming:** `core-hub` remains the current production service and
+**Transition naming:** `hub-core` is the production service and surviving
-rollback authority while `hub-core` becomes the surviving framework/runtime.
+framework/runtime. `core-hub` is the tested rollback until archive.
 
 ## Current State
 
-Core Hub is the verified production `/api/v2` runtime at
+Hub-core is the verified production `/api/v2` runtime at
-`hub.coulomb.social`, now running on railiance01 through the `rapp-core-hub`
+`hub.coulomb.social`, running on railiance01 through the `rapp-core-hub`
-package. The 2026-08-21 relocation preserved the public contract and durable
+package. The 2026-08-21 cutover preserved the public contract and exact durable
-data, reverified ops-hub and activity-core consumers, and retired the active
+data, reverified ops-hub and activity-core consumers, enforced one writer per
-CoulombCore runtimes and obsolete tunnel. Haskell Inter-Hub remains retired
+group, and exercised rollback to Core Hub plus forward recovery. Core Hub is
-(`CORE-WP-0007`).
+still deployed with no writer groups through the seven-day stabilization
+window; the active CoulombCore runtimes and obsolete tunnel remain retired.
+Haskell Inter-Hub remains retired (`CORE-WP-0007`).
 
 This delivers the practical Gen3 production reset, but not the whole product
-intent. Messaging, progress/work projections, several durable evidence and
+intent. Fleet development coordination remains in state-hub; progress/work
-decision resources, and a single extension runtime remain outside Core Hub or
+projections and several durable evidence, decision, deployment, and outcome
-only partially represented. `CORE-WP-0010` tracks absorption into hub-core and
+resources are still external or only partially represented. `CORE-WP-0010`
-repository archive. Hub-core's packaging, contract, port, and conformance
+now tracks only stabilization, residual-owner handoff, and read-only archive.
-foundations are complete, and the joint route-group migration design is
+Hub-core revision `055cf49` is the production implementation. See `STATE.md`
-recorded. Hub-core revision `7e1ec03` publishes that foundation, so S0 is
+for current operational truth.
-active: durable PostgreSQL storage, legacy auth/health compatibility, an
-immutable image, and an internal candidate deployment come next. The healthy
-service remains the production and rollback authority until dual-run consumer
-gates prove a replacement. See `STATE.md` for current operational truth.

STATE.md

@@ -5,26 +5,28 @@
 
 ## One-line posture
 
-**Core Hub remains the verified production `/api/v2` authority on railiance01.
+**Hub-core is the verified production `/api/v2` authority on railiance01.
-Hub-core S0–S5 implementation is published at `8ab1d0c` with immutable image
+Core Hub remains deployed and healthy with no writer groups as the tested
-digest `sha256:2a8b396c5295476d5ce865927b8309a6e054e50eac4e92175ef32cdeaba962fb`;
+rollback until the stabilization window closes on 2026-08-28.**
-the next gate is the private production candidate and data import.**
 
 ## Production truth
 
 | Surface | Current state |
 | --- | --- |
 | Public endpoint | `https://hub.coulomb.social` → `92.205.62.239` |
-| Runtime | railiance01, Helm revision 4, chart `0.3.2` |
+| Runtime | railiance01, Helm revision 20, chart `0.4.0` |
 | Package | `rapp-core-hub`, verified and bound to `reef-railiance` |
-| Image | Forgejo digest `sha256:0db6f7c646c6d4c1fbc844dc266fcf883584637e1c1f47802efe1c6a98558d3c` |
+| Authoritative image | hub-core `055cf49`, digest `sha256:adf580d09a4a9139b1663c41d59d46903a007e03a3ae567e09b8a6ec23708ab8` |
-| Database | railiance01 `platform-pg`, database `core_hub`, schema `core_hub` |
+| Rollback image | Core Hub digest `sha256:388a94ec752e13a0031e329a45e970480c41691e969d4e40c31924669e8d0005` |
+| Database | railiance01 `platform-pg`, database `core_hub`, schemas `core_hub` and `hub_runtime` |
 | TLS | Let's Encrypt certificate Ready through 2026-11-18 |
 | Consumers | ops-hub gate and activity-core resolver/evidence sink verified |
-| Source fallback | CoulombCore production and staging Deployments at zero replicas; source DB read-only |
+| Live rollback | Core Hub Deployment Ready, public routes removed, writer set empty |
 
 The relocation and retirement evidence is in
 [`docs/evidence/core-hub-railiance01-stabilization-2026-08-21.md`](docs/evidence/core-hub-railiance01-stabilization-2026-08-21.md).
+The hub-core migration and cutover evidence is in
+[`docs/evidence/core-hub-hub-core-cutover-2026-08-21.md`](docs/evidence/core-hub-hub-core-cutover-2026-08-21.md).
 Deployment packaging and rollout truth lives in the `rapp-core-hub` repository.
 
 ## Workplans
@@ -33,7 +35,7 @@ Deployment packaging and rollout truth lives in the `rapp-core-hub` repository.
 | --- | --- | --- |
 | `CORE-WP-0001`–`CORE-WP-0009` | finished / archived | Gen3 runtime, contracts, persistence, consumer gates, and hardening delivered |
 | `CORE-WP-0011` | finished | Production relocated to railiance01 and active CoulombCore runtime retired |
-| `CORE-WP-0010` | active | S0–S5 code done; private candidate/data gates next |
+| `CORE-WP-0010` | active | Production cutover done; stabilization/archive gate remains |
 
 ### Open work
 
@@ -41,19 +43,21 @@ Deployment packaging and rollout truth lives in the `rapp-core-hub` repository.
 | --- | --- | --- |
 | `CORE-WP-0010-T01` inventory | done | Inventory recorded in `docs/specs/runtime-absorption-inventory.md` |
 | `CORE-WP-0010-T02` dual-run design | done | Route-group plan recorded in `docs/specs/runtime-absorption-plan.md` |
-| `CORE-WP-0010-T03` absorption slices | progress | Deploy private candidate, import/compare data, run consumers |
+| `CORE-WP-0010-T03` absorption slices | done | Seven-table parity, conformance, restart, and consumer gates passed |
-| `CORE-WP-0010-T04` production cutover | wait | Absorption evidence and operator approval |
+| `CORE-WP-0010-T04` production cutover | done | Hub-core authoritative; rollback and forward recovery exercised |
-| `CORE-WP-0010-T05` archive | wait | Cutover complete and residual ownership recorded |
+| `CORE-WP-0010-T05` archive | wait | Seven-day stabilization through 2026-08-28T20:49:50+02:00 |
 
 ## Intent position
 
-The operational reset promised by `INTENT.md` is real: the service is small,
+The operational and consolidation goals in `INTENT.md` are now materially
-containerized, contract-tested, credential-safe, observable, and deployed on
+realized: hub-core is the single production runtime, remains contract-tested
-ordinary Kubernetes/PostgreSQL infrastructure. The larger Gen3 product intent
+and credential-safe, and runs on ordinary Kubernetes/PostgreSQL infrastructure
-is only partly realized. Core Hub does not yet own the fleet's messaging,
+with a reversible migration. The larger Gen3 product intent is still partial.
-progress/work projections, durable decision/deployment/outcome resources, or a
+Fleet development coordination remains in state-hub, and durable
-single extension runtime. Those gaps are not a reason to disturb the healthy
+decision/deployment/outcome resources plus some progress/work projections are
-production runtime; they are inputs to the hub-core consolidation decision.
+not yet unified behind hub-core. Those residual capabilities need explicit
+owners after this repository is archived; they are not reasons to reverse the
+healthy production cutover.
 
 ## Verify
 
@@ -69,12 +73,11 @@ curl -fsS https://hub.coulomb.social/readyz
 - `CORE-WP-0011` is finished but has no State Hub UUID; the registrar warning is
   retained rather than inventing an identifier.
 - `.custodian-brief.md` is generated and may lag file-backed workplan truth.
-- Hub-core runtime/compatibility/migration implementation is published at
+- Hub-core production authority is published at `055cf49`; exact migration,
-  `8ab1d0c`; its candidate image digest is recorded above. Public authority
+  conformance, consumer, writer-fence, and rollback gates passed.
-  stays here until the private data, conformance, consumer, and rollback gates
-  pass.
 - `make lint` has baseline style findings in unchanged Python files; the full
   suite passes all 26 tests after adding migration export and write fencing.
 - The stopped CoulombCore workloads and read-only source database are retained
   only as controlled fallback evidence.
-- Do not archive this repository before `CORE-WP-0010` completes.
+- Do not archive this repository before the 2026-08-28 stabilization review
+  completes `CORE-WP-0010-T05`.

WORK-RECORDS.md

@@ -51,8 +51,8 @@
 | task | CORE-WP-0009-T06 | done | — | workplans/CORE-WP-0009-hub-core-adoption-and-integration.md |
 | task | CORE-WP-0010-T01 | done | — | workplans/CORE-WP-0010-runtime-absorption-and-archive.md |
 | task | CORE-WP-0010-T02 | done | — | workplans/CORE-WP-0010-runtime-absorption-and-archive.md |
-| task | CORE-WP-0010-T03 | progress | — | workplans/CORE-WP-0010-runtime-absorption-and-archive.md |
+| task | CORE-WP-0010-T03 | done | — | workplans/CORE-WP-0010-runtime-absorption-and-archive.md |
-| task | CORE-WP-0010-T04 | wait | — | workplans/CORE-WP-0010-runtime-absorption-and-archive.md |
+| task | CORE-WP-0010-T04 | done | — | workplans/CORE-WP-0010-runtime-absorption-and-archive.md |
 | task | CORE-WP-0010-T05 | wait | — | workplans/CORE-WP-0010-runtime-absorption-and-archive.md |
 | task | CORE-WP-0011-T01 | done | — | workplans/CORE-WP-0011-railiance01-production-relocation.md |
 | task | CORE-WP-0011-T02 | done | — | workplans/CORE-WP-0011-railiance01-production-relocation.md |

docs/evidence/core-hub-hub-core-cutover-2026-08-21.md

@@ -0,0 +1,70 @@
+# Core Hub to hub-core production cutover evidence — 2026-08-21
+
+## Outcome
+
+Hub-core became the public `https://hub.coulomb.social` authority at
+2026-08-21T20:49:50+02:00. The current release is Helm revision 20 of chart
+`rapp-core-hub` 0.4.0. Core Hub remains Ready with no public route and no
+writer groups as the live rollback through the stabilization window.
+
+## Immutable identities
+
+- hub-core source revision: `055cf49`
+- authoritative hub-core image digest:
+  `sha256:adf580d09a4a9139b1663c41d59d46903a007e03a3ae567e09b8a6ec23708ab8`
+- rollback Core Hub image digest:
+  `sha256:388a94ec752e13a0031e329a45e970480c41691e969d4e40c31924669e8d0005`
+- database: `core_hub` on railiance01 `platform-pg`
+- source/target schemas: `core_hub` / `hub_runtime`
+
+## Data gates
+
+The full import, idempotent replay, restart-persistence check, delta replay,
+and reverse export passed. Source and target had identical record identities
+and the following canonical counts and content hashes:
+
+| Resource | Count | SHA-256 |
+| --- | ---: | --- |
+| `hubs` | 2 | `1903171468a9ec1e19afc8e6d6c3dccd8ec96abbb2eb7a6d43cb0694a27d1bed` |
+| `hubCapabilityManifests` | 2 | `71cc6cf72176558ce656a31f786b81032976355436f4207ef267e079890f0ba5` |
+| `apiConsumers` | 3 | `ee909a15b21c514cab56b35219fb86f39ee092c61a2b05e24827b316e5b47710` |
+| `apiKeys` | 10 | `a6a5123f410e996f8c40922ab8289ff8a1d91a0324234e25d91197ceabf9f6c0` |
+| `widgets` | 16 | `f9b0abfc236e16264b0b02cc22ee680e4a71da58d990db1edae67dfe16a372a1` |
+| `interactionEvents` | 4 | `32d617ff66f67d6a9da516b449145c66a92d7f9e0e2ecede3916957f7402213c` |
+| `migrationRuns` | 2 | `47318fe8f2fa51c66a8fadf2d548ef1d45d1429c365ea97a50d4620d8e6c9c61` |
+
+No raw key or credential value was exported or recorded as evidence.
+
+## Runtime and consumer gates
+
+- All five route groups (`system`, `registry`, `credentials`, `interaction`,
+  `deferred`) route to hub-core.
+- All four mutable writer groups belong only to hub-core. Core Hub rejects
+  writes and advertises an empty writer set.
+- Hub-core internal conformance passed 8/8.
+- `ops-hub` `make interhub-gate` passed against the public host.
+- The activity-core stabilization resolver passed all five checks.
+- Both Deployments are Ready, dependency-aware readiness reports PostgreSQL
+  and authentication healthy, and `make verify-live` passed.
+
+During the registry fence probe, the still-running legacy process briefly
+accepted one exact synthetic `must-not-write` hub before its configuration
+checksum restart. The identified row was removed after an exact identity/name
+check; the restarted legacy runtime then returned 503 and the target/public
+dataset remained at the expected two hubs. No user record was affected.
+
+## Rollback rehearsal
+
+The final cutover was rolled back with Helm to the pre-cutover configuration,
+creating revision 19. Core Hub served healthy public health/readiness checks.
+The final source delta replay preserved all counts and hashes. Forward deploy
+then created current revision 20, after which image, readiness, writer-fence,
+conformance, ops-hub, and activity-core gates all passed again.
+
+## Stabilization and residual owner
+
+The seven-day stabilization interval is
+2026-08-21T20:49:50+02:00 through no earlier than
+2026-08-28T20:49:50+02:00. `CORE-WP-0010-T05` owns the final review, Forgejo
+read-only archive, and handoff of any remaining product-intent gaps. Core Hub
+must remain available as the no-writer rollback until that task closes.

workplans/CORE-WP-0010-runtime-absorption-and-archive.md

@@ -102,7 +102,7 @@ aligned with hub-core's accepted canonical plan; `HUB-WP-0004-T06` is done.
 
 ```task
 id: CORE-WP-0010-T03
-status: progress
+status: done
 priority: high
 state_hub_task_id: "c01ddec1-5737-4115-ac73-c402ffb5bdc5"
 ```
@@ -110,25 +110,20 @@ state_hub_task_id: "c01ddec1-5737-4115-ac73-c402ffb5bdc5"
 Migrate in slices with consumer smokes (ops-hub, activity-core) after each.
 Record row counts and fixture replays per slice.
 
-Started 2026-08-21 after hub-core revision `7e1ec03` closed the joint planning
+Completed 2026-08-21. Hub-core revision `055cf49` and immutable image
-and publication gate. S0 now owns the durable PostgreSQL backend, legacy
+`sha256:adf580d09a4a9139b1663c41d59d46903a007e03a3ae567e09b8a6ec23708ab8`
-auth/health compatibility, immutable image, and internal candidate deployment.
+provide the durable PostgreSQL store, audit ledger, complete compatibility
-Do not change public traffic.
+surface, fail-closed writer controls, and lossless migration tooling. The live
-
+seven-table import was repeatable and preserved exact counts, identity sets,
-Implementation advancement 2026-08-21: hub-core revision `8ab1d0c` adds the
+and canonical content hashes; restart persistence and reverse export also
-durable PostgreSQL store, audit ledger, all `/api/v2` compatibility groups,
+matched. Hub-core conformance passed 8/8, and the public ops-hub and
-fail-closed writer controls, seven-table idempotent import/export, and reverse
+activity-core consumer gates passed before and after rollback recovery.
-bundle support. Its Forgejo image is pinned at
-`sha256:2a8b396c5295476d5ce865927b8309a6e054e50eac4e92175ef32cdeaba962fb`.
-This source now exports all seven tables without raw keys and fences writes by
-route group. The remaining T03 work is live candidate migration, comparison,
-and named consumer evidence.
 
 ## Cutover production traffic
 
 ```task
 id: CORE-WP-0010-T04
-status: wait
+status: done
 priority: high
 state_hub_task_id: "dc414053-a896-4bf0-af05-944b7a7cf750"
 ```
@@ -136,7 +131,13 @@ state_hub_task_id: "dc414053-a896-4bf0-af05-944b7a7cf750"
 With operator approval, point production at hub-core runtime; verify consumer
 smokes and interaction evidence path.
 
-Waiting on completed absorption evidence and explicit operator approval.
+Completed 2026-08-21 with operator approval. All five route groups and all
+four mutable writer groups now belong to hub-core; Core Hub remains Ready with
+an empty writer set as rollback. Helm rollback to the pre-cutover runtime was
+exercised at revision 19, followed by successful forward recovery to revision
+20 and repeat conformance, consumer, image, readiness, and writer-fence gates.
+Evidence is recorded in
+`docs/evidence/core-hub-hub-core-cutover-2026-08-21.md`.
 
 ## Archive repository
 
@@ -150,12 +151,16 @@ state_hub_task_id: "16525001-f85e-4dae-9dc4-d36cb1eb4fbe"
 Write completion record (revisions, PRs, residual owners), set read-only archive
 on forge, and hand residual work as live records outside this repo.
 
-Waiting on production cutover and recorded residual ownership.
+Production cutover is complete. Wait through the stabilization interval that
+started at 2026-08-21T20:49:50+02:00 and ends no earlier than
+2026-08-28T20:49:50+02:00. Then record the final review, archive this
+repository read-only on Forgejo, and hand any remaining product-intent gaps to
+live records outside this repository.
 
 ## Acceptance
 
 - [x] Absorption inventory complete
 - [x] Dual-run design approved/recorded
-- [ ] Consumer smokes pass on hub-core host
+- [x] Consumer smokes pass on hub-core host
-- [ ] Production traffic cut over with evidence
+- [x] Production traffic cut over with evidence
 - [ ] core-hub archived read-only; residuals owned elsewhere