docs: prepare core-hub archive closeout
Some checks failed
CI Smoke / host-smoke (push) Successful in 0s
CI Smoke / pytest-smoke (push) Failing after 2s
Build and Publish Container Image / build-and-push (push) Successful in 19s

This commit is contained in:
tegwick 2026-08-21 21:23:57 +02:00
parent 52bb84fff5
commit 7a95a8345c
9 changed files with 226 additions and 84 deletions

View file

@ -6,9 +6,9 @@ Core Hub is the 3rd-generation Production Interaction Framework for Coulomb / He
| Name | Role | | Name | Role |
| --- | --- | | --- | --- |
| **hub-core** | Shared library — `hub_core` utils, models, router factories | | **hub-core** | Production runtime and shared framework/library |
| **state-hub** | Dev coordination — file-first workplans, agent MCP | | **state-hub** | Dev coordination — file-first workplans, agent MCP |
| **core-hub** | This repo — production `/api/v2` framework | | **core-hub** | This repo — previous `/api/v2` runtime, retained as rollback until archive |
Ecosystem architecture: `/home/worsch/the-custodian/docs/hub-ecosystem-architecture.md` Ecosystem architecture: `/home/worsch/the-custodian/docs/hub-ecosystem-architecture.md`
@ -42,11 +42,12 @@ It carries forward the lessons from State Hub and Inter-Hub: workplans and progr
## Current Work ## Current Work
The contract, FastAPI/Postgres foundation, Inter-Hub migration, Haskell/IHP The contract, FastAPI/Postgres foundation, Inter-Hub migration, Haskell/IHP
retirement, and relocation from CoulombCore to railiance01 are complete. retirement, relocation from CoulombCore, and production absorption into
`CORE-WP-0011` is finished. The only open local workplan is runtime absorption hub-core are complete. Hub-core revision `055cf49` is public authority. This
and eventual archive (`CORE-WP-0010`). Its dual-run design is recorded and S0 repository remains a no-writer rollback through the stabilization window
is active against published hub-core revision `7e1ec03`, building the durable ending no earlier than 2026-08-28T20:49:50+02:00. New product work belongs in
backend and internal candidate before any public traffic changes. hub-core; the only open local task is the prepared read-only archive closeout
in [`docs/deployment/archive-closeout.md`](docs/deployment/archive-closeout.md).
Operational migration bundles are emitted without raw key material: Operational migration bundles are emitted without raw key material:

View file

@ -35,7 +35,7 @@ Deployment packaging and rollout truth lives in the `rapp-core-hub` repository.
| --- | --- | --- | | --- | --- | --- |
| `CORE-WP-0001``CORE-WP-0009` | finished / archived | Gen3 runtime, contracts, persistence, consumer gates, and hardening delivered | | `CORE-WP-0001``CORE-WP-0009` | finished / archived | Gen3 runtime, contracts, persistence, consumer gates, and hardening delivered |
| `CORE-WP-0011` | finished | Production relocated to railiance01 and active CoulombCore runtime retired | | `CORE-WP-0011` | finished | Production relocated to railiance01 and active CoulombCore runtime retired |
| `CORE-WP-0010` | active | Production cutover done; stabilization/archive gate remains | | `CORE-WP-0010` | active | Archive packet prepared; timed stabilization/closeout remains |
### Open work ### Open work
@ -45,7 +45,10 @@ Deployment packaging and rollout truth lives in the `rapp-core-hub` repository.
| `CORE-WP-0010-T02` dual-run design | done | Route-group plan recorded in `docs/specs/runtime-absorption-plan.md` | | `CORE-WP-0010-T02` dual-run design | done | Route-group plan recorded in `docs/specs/runtime-absorption-plan.md` |
| `CORE-WP-0010-T03` absorption slices | done | Seven-table parity, conformance, restart, and consumer gates passed | | `CORE-WP-0010-T03` absorption slices | done | Seven-table parity, conformance, restart, and consumer gates passed |
| `CORE-WP-0010-T04` production cutover | done | Hub-core authoritative; rollback and forward recovery exercised | | `CORE-WP-0010-T04` production cutover | done | Hub-core authoritative; rollback and forward recovery exercised |
| `CORE-WP-0010-T05` archive | wait | Seven-day stabilization through 2026-08-28T20:49:50+02:00 | | `CORE-WP-0010-T05` archive | progress | Closure packet ready; stabilization through 2026-08-28T20:49:50+02:00 |
The executable residual-owner map, retention boundary, and ordered closeout
checklist are in `docs/deployment/archive-closeout.md`.
## Intent position ## Intent position

View file

@ -53,7 +53,7 @@
| task | CORE-WP-0010-T02 | done | — | workplans/CORE-WP-0010-runtime-absorption-and-archive.md | | task | CORE-WP-0010-T02 | done | — | workplans/CORE-WP-0010-runtime-absorption-and-archive.md |
| task | CORE-WP-0010-T03 | done | — | workplans/CORE-WP-0010-runtime-absorption-and-archive.md | | task | CORE-WP-0010-T03 | done | — | workplans/CORE-WP-0010-runtime-absorption-and-archive.md |
| task | CORE-WP-0010-T04 | done | — | workplans/CORE-WP-0010-runtime-absorption-and-archive.md | | task | CORE-WP-0010-T04 | done | — | workplans/CORE-WP-0010-runtime-absorption-and-archive.md |
| task | CORE-WP-0010-T05 | wait | — | workplans/CORE-WP-0010-runtime-absorption-and-archive.md | | task | CORE-WP-0010-T05 | progress | — | workplans/CORE-WP-0010-runtime-absorption-and-archive.md |
| task | CORE-WP-0011-T01 | done | — | workplans/CORE-WP-0011-railiance01-production-relocation.md | | task | CORE-WP-0011-T01 | done | — | workplans/CORE-WP-0011-railiance01-production-relocation.md |
| task | CORE-WP-0011-T02 | done | — | workplans/CORE-WP-0011-railiance01-production-relocation.md | | task | CORE-WP-0011-T02 | done | — | workplans/CORE-WP-0011-railiance01-production-relocation.md |
| task | CORE-WP-0011-T03 | done | — | workplans/CORE-WP-0011-railiance01-production-relocation.md | | task | CORE-WP-0011-T03 | done | — | workplans/CORE-WP-0011-railiance01-production-relocation.md |

View file

@ -3,11 +3,12 @@ id: "c5d9f3a2-7b4e-5f6c-0a1d-3e8f9b2c4d5e"
name: "Core Hub Stabilization Closeout Check" name: "Core Hub Stabilization Closeout Check"
type: activity-definition type: activity-definition
version: "1.0" version: "1.0"
enabled: true enabled: false
owner: core-hub owner: core-hub
governance: core-hub governance: core-hub
status: active status: archived
created: "2026-07-07" created: "2026-07-07"
updated: "2026-08-21"
trigger: trigger:
type: scheduled type: scheduled
at: "2026-07-10T17:35:00+00:00" at: "2026-07-10T17:35:00+00:00"
@ -34,6 +35,10 @@ context_sources:
# ActivityDefinition: Core Hub Stabilization Closeout Check # ActivityDefinition: Core Hub Stabilization Closeout Check
> Historical definition for the completed July 2026 Core Hub/Inter-Hub
> stabilization window. It is disabled and must not be reused for the August
> hub-core cutover window.
## Purpose ## Purpose
One-shot closeout gate at the end of the Core Hub stabilization window. When One-shot closeout gate at the end of the Core Hub stabilization window. When

View file

@ -3,11 +3,12 @@ id: "b4c8e2f1-6a3d-4e5b-9f0c-2d7e8a1b3c4d"
name: "Core Hub Stabilization Daily Check" name: "Core Hub Stabilization Daily Check"
type: activity-definition type: activity-definition
version: "1.0" version: "1.0"
enabled: true enabled: false
owner: core-hub owner: core-hub
governance: core-hub governance: core-hub
status: active status: archived
created: "2026-07-07" created: "2026-07-07"
updated: "2026-08-21"
trigger: trigger:
type: cron type: cron
cron_expression: "0 9 * * *" cron_expression: "0 9 * * *"
@ -35,6 +36,10 @@ context_sources:
# ActivityDefinition: Core Hub Stabilization Daily Check # ActivityDefinition: Core Hub Stabilization Daily Check
> Historical definition for the completed July 2026 Core Hub/Inter-Hub
> stabilization window. It is disabled and must not be reused for the August
> hub-core cutover window.
## Purpose ## Purpose
Scheduled hub-visible health gate for `CORE-WP-0007-T02` during the post-cutover Scheduled hub-visible health gate for `CORE-WP-0007-T02` during the post-cutover

View file

@ -0,0 +1,85 @@
# Core Hub archive closeout
Prepared 2026-08-21 for `CORE-WP-0010-T05`. This document is an executable
closure packet, not evidence that the time-gated archive has happened.
## Freeze boundary
Core Hub is no longer a product-development target. Until archive:
- accept only stabilization, security, rollback, evidence, or archive changes;
- implement new runtime and compatibility behavior in hub-core;
- keep Core Hub Ready with an empty writer set;
- never enable legacy and hub-core writers concurrently; and
- do not delete the legacy schema, immutable image, Helm history, or retained
migration evidence during this task.
The absence of an approved retention expiry blocks destructive cleanup. It
does not block scaling the rollback workload to zero or archiving this Git
repository read-only after operator approval.
## Residual ownership
| Residual | Live owner / record | Archive condition |
| --- | --- | --- |
| Production runtime, compatibility API, migration code, and future product work | `hub-core`; completed baseline `HUB-WP-0005`, new changes require hub-core workplans | No Core Hub implementation remains authoritative |
| Deployment, route map, immutable images, live verification, rollback retirement | `rapp-core-hub`; `RAPPCOREHUB-WP-0002-T05` | Stabilization closes and rollback is scaled down through the package |
| `hub_runtime` ownership, grants, and lease behavior | `rapp-postgres`; `RAPP-POSTGRES-WP-0004-T03` | Rotation evidence is closed or remains explicitly owned there |
| OpenBao/ESO runtime and migration projections | `railiance-platform`; `RAILIANCE-WP-0023-T02` | Rotation evidence is closed or remains explicitly owned there |
| Consumer evidence and hub-port alignment | `activity-core`; `ACTIVITY-WP-0029` | Resolver/evidence path passes on hub-core |
| State Hub strangler, remaining development coordination, and retirement | `state-hub`; `STATE-WP-0079` | No State Hub work is silently absorbed into this archive |
| Registrar/workstation coupling that prevents authoritative UUID allocation | `state-hub`; `STATE-WP-0081` | Existing missing-UUID warnings remain owned outside this repo |
| ops-hub compatibility probe | `ops-hub`; `make interhub-gate` | Public gate passes at closeout |
The broader intent gaps recorded in `STATE.md`—progress/work projections and
durable decision, deployment, and outcome resources—are future hub-core and
State Hub retirement scope. They are not unfinished Core Hub migration code
and must not be restarted here.
## Current immutable anchors
- Public hub-core source/image: `055cf49` /
`sha256:adf580d09a4a9139b1663c41d59d46903a007e03a3ae567e09b8a6ec23708ab8`
- Core Hub rollback image:
`sha256:388a94ec752e13a0031e329a45e970480c41691e969d4e40c31924669e8d0005`
- Deployment package: chart 0.4.0, current Helm revision 20
- Rehearsed rollback revision: 19
- Data and rollback evidence:
`docs/evidence/core-hub-hub-core-cutover-2026-08-21.md`
The final repository revision is intentionally filled in only by the closeout
commit immediately before Forgejo is made read-only.
## Closeout checklist
Do not check the time- or approval-dependent items early.
- [ ] Current time is no earlier than `2026-08-28T20:49:50+02:00`
- [ ] No unresolved stabilization anomaly or rollback occurred after final promotion
- [ ] `make test` passes
- [ ] `make stabilization-check` passes
- [ ] activity-core public resolver/evidence gate passes
- [ ] Counts, identity sets, and canonical hashes remain explained and matching
- [ ] Credential rotation tasks are done or retain explicit external owners
- [ ] State Hub inbox contains no unhandled Core Hub retirement message
- [ ] Final Core Hub Git revision and evidence links are recorded
- [ ] Operator approves rollback retirement and read-only archive
## Ordered closeout
1. Run the checklist and append dated results to the cutover evidence record.
2. Record the operator decision and completion progress in State Hub against
`CORE-WP-0010-T05`.
3. Through `rapp-core-hub`, scale the Core Hub rollback workload to zero while
retaining its immutable image, Helm history, schema, and recovery record.
4. Re-run the public consumer and deployment-package gates against hub-core.
5. Set `CORE-WP-0010-T05` to `done`, the workplan to `finished`, and update
`STATE.md`/`README.md` to point all active development to hub-core.
6. Run `statehub fix-consistency`, commit, and push the final revision.
7. Route the Forgejo admin credential through `warden route` and set
`coulomb/core-hub` read-only/archived. Do not expose the credential.
8. Verify anonymous clone/history remains available and pushes are refused.
If any closeout gate fails, leave the repository writable for evidence fixes,
keep the rollback workload Ready with no writers, and record the failed gate
and owner. Do not improvise concurrent writers or destructive cleanup.

View file

@ -1,65 +1,64 @@
# Core Hub production stabilization # hub-core production stabilization
Post-cutover checks for `CORE-WP-0007-T02` operator sign-off after This is the stabilization and closeout gate for `CORE-WP-0010-T05` after
`CORE-WP-0005` moved `hub.coulomb.social` to Core Hub (2026-07-03). hub-core replaced Core Hub as the public production runtime.
## Window ## Window
- **Start:** 2026-07-03 (production cutover) - Start: `2026-08-21T20:49:50+02:00`
- **End:** 2026-07-10T17:35:00Z (closeout job) - Earliest closeout: `2026-08-28T20:49:50+02:00`
- **Daily job:** 09:00 Europe/Berlin via activity-core - Public endpoint: `https://hub.coulomb.social`
- Production package: `rapp-core-hub` chart 0.4.0, Helm revision 20 at start
- Rollback: Core Hub remains Ready with no writer groups
## Automated checks The July Core Hub/Inter-Hub activities in `activity-definitions/` are retained
as historical evidence. They must not be treated as the schedule for this
window. Current fleet-side observation is owned by activity-core under
`ACTIVITY-WP-0029`; this repository's manual gate is authoritative for archive
closeout until that projection is confirmed.
The `core-hub` context resolver query `stabilization_check` verifies: ## Daily/manual gate
| Check | Expected | Run from this repository:
| --- | --- |
| `GET /healthz` | 200 |
| `GET /readyz` | 200 |
| `GET /api/v2/widget-types` | 200, count ≥ 26 |
| `GET /api/v2/hubs` (unauthenticated) | 401 |
State Hub progress events:
- `core_hub_stabilization_check` — daily while the window is open
- `core_hub_stabilization_closeout` — one-shot at window end
Activity definitions live in `activity-definitions/` and are projected into
`activity-core/k8s/railiance/20-runtime.yaml`.
## Manual companion script
```bash ```bash
CORE_HUB_BASE_URL=https://hub.coulomb.social \ make stabilization-check
STATE_HUB_URL=http://127.0.0.1:8000 \
./scripts/core_hub_stabilization_check.sh
``` ```
Optional full bootstrap smoke (needs approved operator token): The command fails closed unless all of the following pass:
```bash - public health reports runtime version 0.2.0;
CORE_HUB_BASE_URL=https://hub.coulomb.social \ - readiness identifies hub-core, production PostgreSQL, database and
CORE_HUB_OPERATOR_TOKEN_FILE=/secure/path/to/token \ authorization health, all four candidate writer groups, and no legacy
make deployed-smoke writer group;
``` - the public widget-type contract has at least 26 entries;
- an unauthenticated protected request returns 401;
- the ops-hub public consumer gate passes; and
- the deployment package verifies both immutable workloads, routing, writer
ownership, credential metadata, policies, and authentication.
## Operator sign-off for T02 The protected deployed smoke is optional because it needs an approved token
file. If it is used, the value must remain outside Git, State Hub, logs, and
chat.
Sign-off is a human decision recorded in State Hub after: ## Closeout gate
1. Daily checks (or closeout) show `overall_pass=true` through the window After the earliest closeout time, require all of these before archive:
2. No rollback was invoked since cutover
3. Operator approves retiring the Inter-Hub k8s rollback deployment
Post a progress note or decision referencing workstream 1. The manual stabilization gate passes.
`a8d66822-e435-4b1e-ad81-37a298d1795e` / task `16eb7ce3-b574-4667-a189-c14ff5d0502b`, 2. The activity-core resolver/evidence path passes against the public host.
then proceed with `CORE-WP-0007-T02`. 3. No unexplained data, contract, authorization, or writer-ownership drift is
open.
4. The final source/target comparison or retained reverse-export evidence has
matching counts, identities, and canonical hashes.
5. The runtime/database credential rotation observations in
`RAPP-POSTGRES-WP-0004-T03` and `RAILIANCE-WP-0023-T02` are resolved or
carry an explicit live residual owner.
6. The operator approves rollback retirement and repository archive.
## Deploy / enable Record the decision and final progress against workplan
`f5a90aa2-e63f-4f94-b052-8be0d0ee55ea`, task
`16525001-f85e-4dae-9dc4-d36cb1eb4fbe`.
1. Rebuild and deploy `activity-core` on railiance01 with the new `core-hub` The complete procedure and residual ownership map are in
context resolver. `docs/deployment/archive-closeout.md`.
2. Apply the updated `activity-core/k8s/railiance/20-runtime.yaml` ConfigMap
projection for the two new ActivityDefinitions.
3. Confirm events appear at `GET /progress/?event_type=core_hub_stabilization_check`.

View file

@ -1,28 +1,59 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# Core Hub production stabilization snapshot for CORE-WP-0007-T02. # hub-core production stabilization gate for CORE-WP-0010-T05.
# Complements the scheduled activity-core job (core_hub_stabilization_check).
set -euo pipefail set -euo pipefail
BASE_URL="${CORE_HUB_BASE_URL:-https://hub.coulomb.social}" BASE_URL="${CORE_HUB_BASE_URL:-https://hub.coulomb.social}"
STATE_HUB_URL="${STATE_HUB_URL:-http://127.0.0.1:8000}" STATE_HUB_URL="${STATE_HUB_URL:-http://127.0.0.1:8000}"
OPS_HUB_ROOT="${OPS_HUB_ROOT:-/home/worsch/ops-hub}"
RAPP_CORE_HUB_ROOT="${RAPP_CORE_HUB_ROOT:-/home/worsch/rapp-core-hub}"
EXPECTED_RUNTIME_VERSION="${EXPECTED_RUNTIME_VERSION:-0.2.0}"
export EXPECTED_RUNTIME_VERSION
echo "=== hub-core public authority (${BASE_URL}) ==="
curl -fsS "${BASE_URL}/healthz" | python3 -c '
import json, os, sys
payload = json.load(sys.stdin)
assert payload.get("status") == "ok", payload
assert payload.get("version") == os.environ["EXPECTED_RUNTIME_VERSION"], payload
'
echo "/healthz -> ok (version ${EXPECTED_RUNTIME_VERSION})"
curl -fsS "${BASE_URL}/readyz" | python3 -c '
import json, sys
payload = json.load(sys.stdin)
checks = payload.get("checks") or {}
assert payload.get("service") == "hub-core", payload
assert payload.get("status") == "ok", payload
assert checks.get("environment") == "production", checks
assert checks.get("configured_backend") == "postgresql", checks
assert checks.get("active_backend") == "postgresql", checks
assert checks.get("database") == "ok", checks
assert checks.get("authorization") == "ok", checks
assert checks.get("legacy_write_groups") == "none", checks
assert set(filter(None, checks.get("v2_write_groups", "").split(","))) == {
"credentials", "deferred", "interaction", "registry"
}, checks
assert {"credentials", "deferred", "interaction", "operator", "registry", "system"} <= set(
filter(None, checks.get("v2_groups", "").split(","))
), checks
'
echo "/readyz -> ok (PostgreSQL/auth healthy; hub-core is sole writer)"
echo "=== Core Hub public surface (${BASE_URL}) ==="
for path in /healthz /readyz /api/v2/widget-types; do
code=$(curl -sS -o /dev/null -w '%{http_code}' "${BASE_URL}${path}")
echo "${path} -> ${code}"
done
hubs_code=$(curl -sS -o /dev/null -w '%{http_code}' "${BASE_URL}/api/v2/hubs") hubs_code=$(curl -sS -o /dev/null -w '%{http_code}' "${BASE_URL}/api/v2/hubs")
echo "/api/v2/hubs -> ${hubs_code} (expect 401 hardened contract)" echo "/api/v2/hubs -> ${hubs_code} (expect 401 hardened contract)"
widget_count=$(curl -fsS "${BASE_URL}/api/v2/widget-types" | python3 -c " test "${hubs_code}" = "401"
import json,sys
payload=json.load(sys.stdin) widget_count=$(curl -fsS "${BASE_URL}/api/v2/widget-types" | python3 -c '
items=payload if isinstance(payload,list) else payload.get('data',[]) import json, sys
payload = json.load(sys.stdin)
items = payload if isinstance(payload, list) else payload.get("data", [])
assert len(items) >= 26, len(items)
print(len(items)) print(len(items))
") ')
echo "widget_types count=${widget_count} (expect >= 26)" echo "widget_types count=${widget_count} (expect >= 26)"
echo echo
echo "=== optional deployed smoke (needs approved token) ===" echo "=== optional protected deployed smoke (needs approved token) ==="
if [[ -n "${CORE_HUB_OPERATOR_TOKEN_FILE:-}" && -f "${CORE_HUB_OPERATOR_TOKEN_FILE}" ]]; then if [[ -n "${CORE_HUB_OPERATOR_TOKEN_FILE:-}" && -f "${CORE_HUB_OPERATOR_TOKEN_FILE}" ]]; then
make deployed-smoke make deployed-smoke
else else
@ -30,12 +61,14 @@ else
fi fi
echo echo
echo "=== optional ops-hub gate probe ===" echo "=== required ops-hub consumer gate ==="
if command -v make >/dev/null && [[ -d "${HOME}/ops-hub" ]]; then test -d "${OPS_HUB_ROOT}"
(cd "${HOME}/ops-hub" && make interhub-gate IHUB_BASE="${BASE_URL}") || true (cd "${OPS_HUB_ROOT}" && make interhub-gate IHUB_BASE="${BASE_URL}")
else
echo "skip: ops-hub checkout unavailable" echo
fi echo "=== required deployment-package gate ==="
test -d "${RAPP_CORE_HUB_ROOT}"
(cd "${RAPP_CORE_HUB_ROOT}" && make verify-live)
echo echo
echo "=== last 6 core_hub_stabilization_check events ===" echo "=== last 6 core_hub_stabilization_check events ==="
@ -47,3 +80,6 @@ for e in json.load(sys.stdin):
print(e['created_at'][:19], (e.get('summary') or '')[:72], print(e['created_at'][:19], (e.get('summary') or '')[:72],
'pass='+str(probe.get('overall_pass'))) 'pass='+str(probe.get('overall_pass')))
" 2>/dev/null || echo "no stabilization events yet" " 2>/dev/null || echo "no stabilization events yet"
echo
echo "ok: hub-core stabilization gate passed"

View file

@ -143,7 +143,7 @@ Evidence is recorded in
```task ```task
id: CORE-WP-0010-T05 id: CORE-WP-0010-T05
status: wait status: progress
priority: medium priority: medium
state_hub_task_id: "16525001-f85e-4dae-9dc4-d36cb1eb4fbe" state_hub_task_id: "16525001-f85e-4dae-9dc4-d36cb1eb4fbe"
``` ```
@ -157,6 +157,14 @@ started at 2026-08-21T20:49:50+02:00 and ends no earlier than
repository read-only on Forgejo, and hand any remaining product-intent gaps to repository read-only on Forgejo, and hand any remaining product-intent gaps to
live records outside this repository. live records outside this repository.
Archive preparation started 2026-08-21. `docs/deployment/archive-closeout.md`
now freezes feature development here, records live residual owners, preserves
non-destructive retention boundaries, and defines the ordered final checklist.
`docs/deployment/stabilization-signoff.md` and
`make stabilization-check` define the current fail-closed production gate.
Only elapsed stabilization, final observations, operator approval, rollback
scale-down, and the read-only Forgejo action remain.
## Acceptance ## Acceptance
- [x] Absorption inventory complete - [x] Absorption inventory complete