hub-core/docs/core-hub-absorption-plan.md

Core Hub absorption and /api/v2 cutover plan

Status: accepted joint input for HUB-WP-0004-T06 and CORE-WP-0010-T02 Date: 2026-08-21 Runtime source: hub-core Rollback runtime: core-hub on railiance01

Decision

Absorb Core Hub into hub-core by capability and data boundary. There is no big-bang route move and no period with two production writers for the same boundary.

Hub-core owns the surviving wheel, runtime image, extension contracts, compatibility routes, durable framework store, and migrations. Core Hub stays deployable as the rollback runtime until every /api/v2 route group, durable record family, consumer gate, and rollback exercise has passed. Its Python package is never imported by hub-core.

The current production baseline is Core Hub chart 0.3.2 on railiance01, backed by the core_hub PostgreSQL schema. Its source database on CoulombCore is already frozen and is historical rollback evidence, not a participant in this absorption.

Non-negotiable invariants

1. A route group has exactly one production writer at any moment.
2. A write freeze, final export, count/content comparison, and explicit operator approval precede every production writer change.
3. Shadow traffic consists of reads and synthetic fixture writes only. Do not mirror normal writes to both runtimes.
4. Hub-core and Core Hub keep separate SQLAlchemy metadata and migration lineages. Hub-core may copy legacy rows but must not import core_hub models or migrations.
5. IDs, timestamps, relationships, source revision, source table, and import bundle hash survive migration as provenance.
6. Only API-key hashes and prefixes move. Raw keys, static operator tokens, database credentials, and secret values never enter bundles or evidence.
7. Authorization fails closed when its dependency is unavailable. The compatibility API-key validator is temporary; authorization decisions remain owned through the identity/policy boundary.
8. Core Hub remains a tested rollback target until the stabilization and rollback-retirement gates close.

Runtime and database shape

The hub-core image selected by ADR-0001 is the candidate runtime. Its current in-memory PortStore is not a migration target. The first absorption slice must add a PostgreSQL implementation, production readiness checks, an audit ledger, and migrations owned entirely by hub-core.

Use the existing platform PostgreSQL allocation currently assigned to Core Hub; do not consume a fifth application slot. Add a distinct target schema (provisionally hub_runtime) beside retained legacy schema core_hub, then transfer the consumer declaration and dynamic credential identity as part of the final package transition. Separate schemas make source/target counts and rollback observable without mixing ORM metadata.

The deployment package currently named rapp-core-hub owns the verified railiance01 chart, policies, secret projections, live verifier, and rollback surface. During dual-run it must deploy both immutable image revisions or a candidate release beside the legacy release. Before Core Hub is archived, the package must either be renamed/replaced by a hub-core-owned rapp package or recorded explicitly as the surviving package with updated ownership and intent. No application source moves into the deployment package.

Traffic controls

Implement route-group controls before moving data:

• HUB_CORE_V2_GROUPS: compatibility groups served by hub-core;
• HUB_CORE_V2_WRITE_GROUPS: subset allowed to mutate hub-core authority;
• CORE_HUB_V2_WRITE_GROUPS: legacy groups still allowed to mutate Core Hub;
• an ingress/gateway route map selecting core-hub or hub-core per group;
• a shadow comparator that calls both read targets, normalizes ordering and volatile fields, and records hashes without response bodies containing sensitive data.

The exact configuration carrier may be Helm values rather than environment variables, but the three states are mandatory and independently observable:

State	Public owner	Candidate behavior	Write rule
legacy	Core Hub	health plus synthetic fixtures	Core Hub only
shadow	Core Hub	normalized read comparison	Core Hub only
candidate	hub-core	public read/write for the group	hub-core only

Direct service access is network-policy restricted. The route map is the normal traffic authority; application write-group checks are defense in depth. A configuration assertion must reject overlap between legacy and candidate write sets.

Route and module move order

Each slice is independently buildable, reversible, and evidenced. Later slices do not excuse a failed earlier gate.

S0 — Durable runtime, compatibility shell, and migration tooling

No public route moves.

• Add hub-core PostgreSQL PortStore, session lifecycle, readiness, audit ledger, import/export commands, and a migration bundle schema.
• Add /api/v2 router composition, legacy response serializers, bearer-token dependency, per-group write controls, and normalized shadow comparison.
• Import the static operator token through the existing OpenBao/ExternalSecret lane and copy API-key hashes into the isolated compatibility auth store.
• Preserve current error/status semantics and unprefixed OpenAPI aliases.
• Extend the conformance suite with durable restart/rebuild, authorization allow/deny/unavailable, metadata isolation, and migration idempotence.

Gate: frozen source fixture imports twice without duplicates; all seven source table counts can be represented; production readiness fails when PostgreSQL or authorization is unavailable; no traffic has moved.

S1 — System and read-only discovery

Routes:

• /healthz, /readyz;
• /api/v2/widget-types, /api/v2/event-types;
• /api/v2/annotation-categories, /api/v2/policy-scopes;
• /api/v2/openapi.json, /api/v2/openapi.yaml, /api/v2/docs;
• compatibility aliases published by the current OpenAPI surface.

Move the catalogs, schemas, OpenAPI snapshot/export behavior, and public consumer fixtures. Static catalog responses are compared as canonical JSON. Readiness must reflect the dependencies required by enabled groups rather than reporting healthy from process liveness alone.

Gate: public catalog and documentation smoke passes against both targets; normalized payload hashes match; ops-hub public gate and activity-core resolver pass; rollback route-map change is rehearsed.

S2 — Registry and manifest authority

Routes:

• GET/POST /api/v2/hubs;
• GET/POST /api/v2/hub-capability-manifests;
• PATCH /api/v2/hub-capability-manifests/{manifest_id};
• POST /api/v2/hub-capability-manifests/{manifest_id}/activate;
• GET /api/v2/hub-registry.

Modules/data:

• copy hubs and hub_capability_manifests into hub-core registry authority;
• preserve legacy UUIDs and camel-case compatibility serialization;
• validate new writes against helixforge.hub-extension, while a named legacy adapter reads migrated 0.x records without silently rewriting them;
• add activation audit records and expose registry projections with provenance.

Gate: table counts and order-independent row hashes match; manifest activation is idempotent and audited; protected allow/deny checks pass; ops-hub bootstrap can resolve and update its manifest; rollback delta replay is proven.

S3 — API consumers and compatibility credentials

Routes:

• GET/POST /api/v2/api-consumers;
• POST /api/v2/api-consumers/{consumer_id}/api-keys;
• POST /api/v2/token.

Modules/data:

• copy api_consumers and api_keys, including status, scopes, quotas, hash, and prefix but never raw key material;
• preserve one-time key issuance semantics and ensure generated secrets are absent from logs, projections, migration bundles, and evidence;
• keep the static operator credential and dynamic consumer keys in distinct custody lanes;
• record a residual owner and follow-on task for replacing compatibility keys with the platform identity/authorization ports. The shim may outlive Core Hub briefly, but it is not a permanent hub-core authority.

Gate: existing ops-hub and activity-core credentials authenticate against the candidate; invalid, revoked, missing, and dependency-unavailable cases deny; rate/quota behavior is either preserved or explicitly residual-owned; reverse delta replay for newly issued key hashes is rehearsed without exposing a key.

S4 — Widgets and interaction evidence

Routes:

• GET/POST /api/v2/widgets;
• GET/POST /api/v2/interaction-events.

Modules/data:

• copy widgets into the operator-surface registry/projection boundary;
• copy interaction_events into the physically separate interaction-event authority while retaining legacy widgetId, event type, context, metadata, IDs, and timestamps in the compatibility view;
• route new writes through port.events.interaction and build /api/v2 reads from the same authority rather than a second compatibility-only store.

Gate: counts and hashes match; projection rebuild fixture and migrated sample produce provenance-equivalent views; progress events never appear in the interaction family; ops-hub and activity-core each post and read back unique correlated evidence; rollback delta replay is proven.

S5 — Empty compatibility collections and operator console

Routes:

• GET/POST /api/v2/annotations;
• GET/POST /api/v2/requirement-candidates;
• GET/POST /api/v2/decision-records;
• GET/POST /api/v2/deployment-records;
• GET/POST /api/v2/outcome-signals;
• /console.

The five collections have no Core Hub durable models and currently return empty data. Preserve that behavior only as a compatibility adapter; do not claim or create authority. Each endpoint needs an explicit retirement or future-owner record before the adapter is removed. Rebuild the console from hub-core projections and public APIs; it must not query the retained core_hub schema.

Gate: exact empty/error compatibility passes, the console authorization boundary and visual smoke pass, and every deferred collection has a residual disposition.

S6 — Whole-host cutover and Core Hub retirement

After S1–S5 are in candidate state:

1. Freeze all remaining Core Hub writes and capture a final encrypted backup, table counts, content hashes, schema revision, image digest, and correlation ID.
2. Import any remaining deltas and require zero unexplained count/hash drift.
3. Run hub-core conformance, the Core Hub deployed smoke, ops-hub bootstrap, and activity-core resolver/write-readback against the candidate image.
4. With explicit operator approval, point hub.coulomb.social entirely at hub-core and set every Core Hub write group empty.
5. Observe at least seven consecutive days with no unexplained 5xx, no failed consumer gate, no data drift, and no normal request to Core Hub. Record any exception with an owner and expiry.
6. Exercise rollback once during rehearsal. After the stabilization window and explicit retirement approval, scale Core Hub to zero while retaining its immutable image, database backup, deployment revision, and route-map rollback instructions until the agreed expiry.
7. Move the final OpenAPI/contracts/smoke fixtures into hub-core history, record residual owners, archive the core-hub repository read-only, and update the deployment package identity.

Evidence required for every durable slice

Evidence	Pass condition
Source identity	Core Hub image/source revision and schema revision recorded
Target identity	hub-core wheel/image version, revision, and contract version recorded
Backup	Encrypted location and checksum recorded; no secret value logged
Counts	Per-table source/import/target counts with zero unexplained delta
Content	Order-independent canonical row hashes match after documented transforms
Provenance	Source table/ID/revision, import bundle hash, and target record IDs queryable
Contract	Static OpenAPI diff has no unexplained removed route/status/schema behavior
Authorization	Missing/invalid/revoked deny; allowed roles pass; dependency outage denies
Failure mode	Database, policy/auth, and candidate unavailability behavior recorded
Consumers	Owned ops-hub and activity-core gates pass against the candidate
Rollback	Reverse delta export/import and route-map reversal exercised

An unexplained row, response, authorization, or provenance mismatch fails the slice. A waiver requires a human progress note and a live residual task.

Smoke and change ownership

Surface	Owner during absorption
Hub-core unit, contract, conformance, migration, and image tests	hub-core
Legacy OpenAPI and deployed-smoke oracle	core-hub until archive, then historical fixture in hub-core
ops-hub public/bootstrap gate	ops-hub
activity-core resolver and interaction write/readback	activity-core
Helm render, policy, secret projection, live verification, traffic map, rollback	rapp package (currently rapp-core-hub) + platform owners
Production writer flip, rollback, scale-to-zero, archive	operator approval required

Rollback procedure

For a failed candidate slice:

1. Stop candidate writes for the affected group; do not enable legacy writes yet.
2. Export the candidate-only delta with IDs and hashes, validate it against the reverse adapter, and import it into the retained legacy schema.
3. Compare counts/content and run the legacy group smoke.
4. Route the group to Core Hub, enable only its legacy writer, and confirm the candidate writer remains disabled.
5. Record the failure correlation ID, data interval, revisions, evidence, and next owner. Do not continue to the next slice.

If candidate data cannot be replayed safely, keep both writers disabled and invoke the operator recovery path. Availability pressure does not authorize discarding acknowledged writes or enabling concurrent writers.

Completion criteria

HUB-WP-0004-T06 is complete when this plan is linked to CORE-WP-0010 and its handoff is recorded. HUB-WP-0004 may then finish; it does not claim production absorption.

CORE-WP-0010 completes only when all slices have evidence, production is on hub-core, the stabilization and rollback gates close, residuals have live owners, the deployment package names the surviving runtime, and Core Hub is archived read-only.