From 6d53e707e21971b53ba9bce4b6ed389433b3d37b Mon Sep 17 00:00:00 2001 From: tegwick Date: Tue, 13 Oct 2026 23:18:36 +0200 Subject: [PATCH] STATE-WP-0098: A5 comparison packet, A4 owner map, inbox change-feed contract draft Co-Authored-By: Claude Opus 5.5 Assistant: claude-code Assistant-Model: opus Assistant-Process: 356991@bnt-lap001 Assistant-Session: 5d99ea69-ed55-4a1e-8d97-f4f312db247f --- docs/a5-topic-spine-comparison-packet.md | 203 +++++++++++ .../a5-topic-spine-comparison-cases.json | 124 +++++++ docs/inbox-change-feed-contract.md | 328 ++++++++++++++++++ .../STATE-WP-0098-hub-core-receipt-inputs.md | 25 +- 4 files changed, 677 insertions(+), 3 deletions(-) create mode 100644 docs/a5-topic-spine-comparison-packet.md create mode 100644 docs/evidence/a5-topic-spine-comparison-cases.json create mode 100644 docs/inbox-change-feed-contract.md diff --git a/docs/a5-topic-spine-comparison-packet.md b/docs/a5-topic-spine-comparison-packet.md new file mode 100644 index 0000000..aca1d25 --- /dev/null +++ b/docs/a5-topic-spine-comparison-packet.md @@ -0,0 +1,203 @@ +# A5 topic-spine comparison packet (draft) + +**Task:** STATE-WP-0098-T01 (A5) and STATE-WP-0098-T02 (A4 section) +**Receiver:** hub-core `HUB-WP-0006-T06` +**Captured:** 2026-10-13, read-only GETs against State Hub `primary/railiance01` +**Cases:** `docs/evidence/a5-topic-spine-comparison-cases.json` (10 cases) +**Status:** draft. Not yet sent to hub-core. No route, flag or deployment was changed. + +## 1. Where the A5 and A4 item lists come from + +The counts "A4 / 22" and "A5 / 8" in `docs/retirement-cutover-slice-plan.md` +come from the original slice plan (commit `613477b`). That plan grouped +inventory items by capability: A5 is `topic-spine` (8); A4 is +`work-coordination-ui` (14) plus `repo-ui` (8). The item list is +`prj-state-hub-retirement/inventory/state-hub-disposition.yaml` +(SHR-INV-0001). Items were matched on their `capability` field; none of them +were guessed. + +**Correction to the framing.** The 8 A5 items are not 8 read endpoints: + +| Inventory id | Kind | Read? | +| --- | --- | --- | +| `route:GET:/topics/` | REST | read | +| `route:GET:/topics/{topic_id}` | REST | read | +| `mcp:get_topic` | MCP tool (wraps the two GETs plus `/progress`) | read | +| `route:POST:/topics/` | REST | write | +| `route:PATCH:/topics/{topic_id}` | REST | write | +| `route:DELETE:/topics/{topic_id}` | REST | write | +| `mcp:create_topic` | MCP tool | write | +| `table:topics` | dataset | data | + +That leaves three read surfaces to compare. One more read surface is not in +the inventory: the MCP resource `state://topics`. The three writes are out of +scope for a read switch. They should be retired or frozen, not redirected, +because the receiver accepts no classification writes. Classification fields +on `/repos/` (`domain_slug`, `category`, tags, stake, mechanics, `topic_id`) +belong to A1 in the inventory. They are still the real counterpart of the +hub-core projection, so A5-C05 to A5-C07 compare them. + +## 2. Callers + +**Legacy meter.** None of the topic routes is metered. `GET /legacy-meter/interfaces` +returns 16 interfaces, all for workstreams, SBOM or messages. The weekly reviews +up to `legacy-meter-weekly-review-20261013.json` do not mention topics. State +Hub has no other per-route usage telemetry. **So there are no volume or +last-seen figures.** To get them, register `rest_api:GET /topics/` and +`rest_api:GET /topics/{topic_id}` in the meter before the switch window. +That is a State Hub write and needs approval. + +**Code callers** (ripgrep over `/home/worsch/*`, excluding archives and +workplans). Every direct caller is inside `state-hub`: + +| Caller | Route | Use | +| --- | --- | --- | +| `custodian_cli.py` (`statehub`) l.178, 481 | `GET /topics/?status=active` (+ `POST` l.184) | resolve topic for register/brief | +| `statehub_register.py` l.851, 900 | `GET /topics/?status=active`, `POST /topics/` | repo registration | +| `scripts/consistency_check.py` l.3398, 3784, 3969 | `GET /topics/{id}`, `GET /topics` | fix-consistency (C-06 etc.) | +| `scripts/promote_intake.py`, `validate_repo_adr.py`, `normalize_attached_repo_workplans.py`, `update_agent_instruction_files.py`, `register_project.sh` | `GET /topics[/?status=active]` | slug/id/domain lookup | +| `mcp_server/server.py` (`state://topics`, `get_topic`, l.208, 263, 2402, 2757) | `GET /topics`, `GET /topics/{id}` | MCP clients (agents) | +| Dashboard: `tasks`, `decisions`, `dependencies`, `extensions`, `interventions`, `techdept`, `todo`, `workstreams` pages | `GET /topics/` | topic label lookup | + +The only hit outside `state-hub` is `kontextual-engine/core/cmis.py`. It builds +a CMIS path, not a State Hub call, so it is not a caller. + +**Indirect alias consumers.** 126 repos carry a `topic_id` in `AGENTS.md` or +`.claude/rules/repo-identity.md`. They use it in +`/workplans/?topic_id=` (A2) and in progress events (B3). They never call +`/topics/`, but they need the topic UUIDs to stay stable until those slices +drop `topic_id`. + +## 3. Legacy response shapes (captured live) + +- `GET /topics/?status=active` → 200, a list of 16 rows. Each row has + `{id: uuid, slug, title, description, domain_slug, status, created_at, updated_at}`. + All 16 are `active`. Headers: `Cache-Control: max-age=60`, `ETag`. +- `GET /topics/{id}` → 200, the same fields plus `workstreams: [WorkstreamStub]`. + An unknown id → 404. **Defect:** `workstreams` is always `[]`. The model + relation was renamed `workplans`, but the schema still reads `workstreams`. + No consumer can depend on its content. +- MCP `get_topic(slug)` → `{topic: , recent_progress: []}`. +- Repo classification (`GET /repos/`, 154 rows: 149 active, 5 archived) includes + `id, slug, canonical_slug, domain_slug, secondary_domains, category, + capability_tags, business_stake, business_mechanics, topic_id, classified_at, + classified_by`. 58 repos have a `topic_id` and 96 do not; 23 have no + `category`. +- The topic ids and slugs are listed in the JSON case file (A5-C01/C02 use all 16). + +## 4. Hub-core side and alias expectations + +- **Routes** (`hub_core/runtime/repository_navigation_routes.py`, GET only): + `GET /ports/projections/repository-navigation/repositories` (filters + `primary_domain`, `secondary_domain`, `category`, `capability_tag`, + `business_stake`, `business_mechanic`; `cursor`; `limit` ≤ 500) and + `GET /ports/projections/repository-navigation/facets/{kind}/{value}`. + MCP: `query_repository_navigation`, `get_repository_navigation_facet`. + 503 means no accepted projection; 409 means a stale cursor. +- **Contract:** `helixforge.repository-navigation` 1.0.0 + (`hub_core/contracts/repository_navigation/v1_0_0/`, `docs/repository-navigation-contract.md`). + The output carries `repositories[]` keyed by registrar `repository_id`, + `facets[]`, `legacy_topic_aliases[]`, provenance and `content_hash`. +- **Identity.** Join on legacy repo `id` == `repository_id`. Repo Manager's + registry import takes the UUID from State Hub `/repos/{slug}` + (`repo_manager/repository_publisher.py`). That this holds for all 123 rows + in the deployed registry is **unverified**. Slugs are labels, not identity. +- **Topic aliases.** Each alias maps one State Hub topic UUID/slug to exactly + one facet and must carry `expires_at` and a `residual_owner`. The expected + mapping is `facet_kind = primary_domain` and `facet_value = topic.domain_slug`, + with residual owner `STATE-WP-0079`. This mapping is lossy: + - several topics share a domain (7 topics are `infotech`); + - 4 repos sit under a topic of a different domain (`adaptive-pricing`, + `commerce-canon`, `informed-decision`, `whynot-design`); + - 96 repos have no topic. + + A topic is a grouping label, not a facet, so exact topic membership has no + hub-core counterpart (A5-C04). +- **Legacy fields with no counterpart:** topic `title`, `description`, + `status`, `created_at`, `updated_at`, `workstreams`; in `get_topic`, + `recent_progress` (B3) and workplans (A2); in repos, `classified_at` and + `classified_by` (provenance replaces them). + +## 5. Comparison cases + +Ten cases are defined in the JSON file: + +| Id | What it compares | Expected result | +| --- | --- | --- | +| C01/C02 | Topic list and topic-by-id against the aliases | equal once aliases exist | +| C03 | Unknown topic | 404 vs empty facet, a documented difference | +| C04 | Topic membership vs domain facet | subset, a documented difference | +| C05 | Full per-repo classification parity on the id join | equal; cardinality gap 154 vs 123 to be listed | +| C06 | `state-hub` spot check | equal | +| C07 | Domain facet counts | equal | +| C08 | MCP `get_topic` | partial | +| C09 | Fields with no counterpart | none to compare | +| C10 | Rollback read | equal to the 2026-10-13 capture | + +## 6. Fallback and rollback + +The fallback is the current State Hub read routes, which stay unchanged +throughout: `GET /topics/`, `GET /topics/{topic_id}`, and MCP +`get_topic`/`state://topics`. Rollback means setting `RM_SLICE_TOPICSPINE` +back to `off` and confirming with A5-C10. + +**Caveat:** `RM_SLICE_TOPICSPINE` does not exist in any code in State Hub, +hub-core, rapp-core-hub or repo-manager. The slice plan calls the `RM_SLICE_*` +names "proposed controls, not proof of deployed flags". So today the rollback +is "callers keep or restore the legacy base URL". Someone must implement the +flag, or name a per-caller switch instead, before the switch window. + +## 7. Gaps blocking hub-core from running the comparison + +1. **No alias source.** The hub-core runtime always emits + `legacy_topic_aliases: []` (`repository_navigation.py` l.74). The only + alias in the fixtures expired on 2026-09-30. C01, C02 and C08 cannot pass + until someone owns alias seeding: hub-core config, or a State Hub export of + the 16 topics. +2. **No flag.** `RM_SLICE_TOPICSPINE` is not implemented (see §6). +3. **No usage figures.** The topic routes are unmetered, so there is no + volume or last-seen data. +4. **Candidate access not verified.** The base URL and auth for the + candidate routes have not been checked from State Hub. The only source is + the 2026-10-03 receipt (Helm rev 31, 123/123 accepted). Nobody has made a + live candidate GET this session. +5. **Cardinality gap.** 154 legacy repos vs 123 published. The 31-row + difference needs an explanation before C05 can be judged. +6. **The real switch targets are State Hub's own tools.** All direct callers + are State Hub tools (CLI, consistency check, MCP, dashboard). They use + topics for id/slug/domain lookup and for writes. A read switch to + navigation facets only covers the domain lookup. + +## A4 work/repo UI callers + +The 22 A4 items are Observable dashboard views in `state-hub/dashboard`, not +API endpoints. They are served by `make dashboard` on the workstation +(`:3000`) and viewed by the operator (Bernd). The inventory lists +`repo-manager` as their owner, but RMGR-ADR-003 excludes a Repo Manager +dashboard. The slice plan names "hub-core projection + UI clients" as the +receiver. No hub-core page equivalents were checked (unverified). The backing +reads below come from each page's `apiFetch`/`fetch`/`FileAttachment` calls. + +| View | Backing reads | Proposed owner to ask | +| --- | --- | --- | +| `/tasks`, `/tasks/*`, `/todo`, `/interventions`, `/workstreams`, `/workstreams/*`, `/dependencies` | `/tasks/`, `/workplans/`, `/workplans/index`, `/state/deps`, `/repos/`, `/topics/`, `/contributions/`, `/technical-debt/` | hub-core (work-record projection, A2) with repo-manager (file authority) | +| `/decisions` | `/decisions/`, `/topics/` | hub-core; external link from **railiance-platform** (`docs/credential-change-approval.md`, CCR yaml `decision_dashboard_url`) | +| `/goals` | `/domain-goals/`, `/repo-goals/`, `/domains/`, `/repos/` | hub-core / repo-manager (unverified owner) | +| `/wsjf-triage` | `/progress/`, `/workplans/index` | hub-core (B3 progress) | +| `/workplan-queue` | `/execution/*` | activity-core (C1) | +| `/repos`, `/repos/*`, `/repo-sync`, `/contributions`, `/extensions`, `/techdept` | `/repos/`, `/domains/`, `/extension-points/`, `/technical-debt/`, `/contributions/`, `/workplans/`, build-time `data/repos.json`, `gitea-inventory.json` | repo-manager (A1 registry) via hub-core navigation | +| `/sbom` | `/sbom/*` (Nexus-backed) | sbom-nexus (rapp-sbom-nexus); archived railiance-cluster/bootstrap docs link it | +| `/policy/intake-doc`, `/policy/workstream-dod`, `/policy/repo-doi` | static policy text | policy-nexus (B4) | +| `/suggestions` | page already removed (E1) | none; already retired | + +That makes 21 live views plus `/suggestions`, which is already gone. Pending +approval, ask each owner for two things: + +- **A candidate read smoke.** The owner's endpoint returns the same rows for + the backing reads above, compared on ids and counts. +- **A reversible switch plan.** The page reads the candidate behind a toggle. + Rollback re-points the page, or removes the redirect, to the retained State + Hub UI, which stays unchanged. + +railiance-platform should also be asked where `decision_dashboard_url` should +point after cutover. No messages have been sent. diff --git a/docs/evidence/a5-topic-spine-comparison-cases.json b/docs/evidence/a5-topic-spine-comparison-cases.json new file mode 100644 index 0000000..6523618 --- /dev/null +++ b/docs/evidence/a5-topic-spine-comparison-cases.json @@ -0,0 +1,124 @@ +{ + "packet": "STATE-WP-0098-T01", + "receiver_task": "HUB-WP-0006-T06", + "status": "draft", + "captured_at": "2026-10-13", + "legacy_base": "State Hub primary/railiance01 (workstation forward http://127.0.0.1:8000; in-cluster http://10.43.68.154:8000)", + "candidate_base": "UNVERIFIED: hub-core candidate runtime (Deployment core-hub-api-candidate, Helm rev 31). Route prefix /ports/projections/repository-navigation. Auth requirement not verified from State Hub.", + "candidate_contract": "helixforge.repository-navigation 1.0.0", + "legacy_snapshot": { + "topics_total": 16, + "topics_active": 16, + "repos_total": 154, + "repos_active": 149, + "repos_archived": 5, + "repos_with_topic_id": 58, + "repos_without_topic_id": 96, + "repos_without_category": 23, + "domains_total": 14, + "repos_whose_domain_differs_from_their_topic_domain": ["adaptive-pricing", "commerce-canon", "informed-decision", "whynot-design"] + }, + "global_normalisation": [ + "Join repositories on legacy /repos/ id == candidate repository_id (Repo Manager imports the registrar UUID from State Hub /repos/{slug} id; verify on the deployed registry).", + "Compare list fields as sorted sets; treat null and [] as equal.", + "Ignore ordering, pagination, cursors, ETag/Cache-Control headers, timestamps and candidate provenance/content_hash fields.", + "Follow candidate next_cursor until null before comparing (limit 500).", + "Record candidate projection_status, source_snapshot.source_revision, rebuilt_at and content_hash in the receipt." + ], + "cases": [ + { + "id": "A5-C01", + "title": "Topic list maps to legacy topic aliases", + "legacy_request": "GET /topics/?status=active", + "candidate_request": "GET /ports/projections/repository-navigation/repositories?limit=500", + "compare": {"legacy": ["id", "slug", "domain_slug"], "candidate": ["legacy_topic_aliases[].legacy_topic_id", "legacy_topic_aliases[].legacy_topic_slug", "legacy_topic_aliases[].facet_kind", "legacy_topic_aliases[].facet_value"]}, + "normalisation": ["Set comparison on legacy id == legacy_topic_id", "facet_kind must be primary_domain and facet_value == legacy domain_slug", "every alias must carry expires_at and residual_owner"], + "expected": "expected-equal (16 aliases) once aliases are seeded", + "current_known_state": "BLOCKER: hub-core runtime returns legacy_topic_aliases = [] (hub_core/runtime/repository_navigation.py line 74); no alias seeding path found. Case fails until an alias source exists." + }, + { + "id": "A5-C02", + "title": "Single topic by id resolves to one alias", + "legacy_request": "GET /topics/cee7bedf-2b48-46ef-8601-006474f2ad7a", + "candidate_request": "GET /ports/projections/repository-navigation/repositories?limit=500 (select alias with legacy_topic_id = cee7bedf-...)", + "compare": {"legacy": ["id", "slug", "domain_slug"], "candidate": ["legacy_topic_id", "legacy_topic_slug", "facet_value"]}, + "normalisation": ["Repeat for all 16 topic ids from A5-C01"], + "expected": "expected-equal: custodian -> primary_domain infotech", + "current_known_state": "Blocked by A5-C01." + }, + { + "id": "A5-C03", + "title": "Unknown topic", + "legacy_request": "GET /topics/00000000-0000-0000-0000-000000000000", + "candidate_request": "No alias lookup route; GET /ports/projections/repository-navigation/facets/primary_domain/no-such-domain", + "compare": {"legacy": ["HTTP status"], "candidate": ["HTTP status", "repositories length"]}, + "normalisation": [], + "expected": "documented-difference: legacy returns 404; candidate has no per-alias route, an unknown facet value is expected to return 200 with zero repositories (unverified). Expired aliases return 410 per contract." + }, + { + "id": "A5-C04", + "title": "Topic membership versus domain facet membership", + "legacy_request": "GET /repos/ then filter topic_id == (instances: custodian cee7bedf, helix-forge f39fa2a3, netkingdom a6c6e745)", + "candidate_request": "GET /ports/projections/repository-navigation/facets/primary_domain/", + "compare": {"legacy": ["id set"], "candidate": ["repositories[].repository_id set"]}, + "normalisation": ["Exclude repos whose own domain differs from the topic domain (adaptive-pricing, commerce-canon, informed-decision, whynot-design)"], + "expected": "documented-difference: a topic is a grouping label, a facet is the repository's own domain. After exclusion, legacy topic members must be a subset of the candidate facet; candidate may contain more repos. Consumers that need exact topic membership have no counterpart and must keep reading topic_id from A1/A2 until owners drop it." + }, + { + "id": "A5-C05", + "title": "Per-repository classification parity (full set)", + "legacy_request": "GET /repos/", + "candidate_request": "GET /ports/projections/repository-navigation/repositories?limit=500 (all pages)", + "compare": { + "pairs": [["domain_slug", "primary_domain"], ["secondary_domains", "secondary_domains"], ["category", "category"], ["capability_tags", "capability_tags"], ["business_stake", "business_stake"], ["business_mechanics", "business_mechanics"], ["status", "lifecycle"], ["canonical_slug", "slug"]] + }, + "normalisation": ["Join on id == repository_id", "sorted-set compare for list fields", "Report legacy-only and candidate-only ids separately"], + "expected": "expected-equal on the joined set. Documented-difference on cardinality: legacy 154 rows vs candidate 123 accepted (2026-10-03 receipt); the 31-row gap must be listed and explained (archived, unregistered, private). Note: repository files are the authority, so a value mismatch is a State Hub staleness finding unless the candidate source revision is older." + }, + { + "id": "A5-C06", + "title": "Single repository spot check", + "legacy_request": "GET /repos/state-hub", + "candidate_request": "GET /ports/projections/repository-navigation/repositories?limit=500 (select repository_id 8e9ffb56-7abc-47b5-84b2-5ad392b37dcd)", + "compare": {"pairs": [["domain_slug", "primary_domain"], ["secondary_domains", "secondary_domains"], ["category", "category"], ["capability_tags", "capability_tags"], ["business_stake", "business_stake"], ["business_mechanics", "business_mechanics"]]}, + "normalisation": ["sorted sets"], + "expected": "expected-equal: infotech / [agents] / tooling / tags [coordination, governance, knowledge, observability, platform] / stake [automation, intelligence, operations, product, technology] / mechanics [adaptation, control, coordination, operation]" + }, + { + "id": "A5-C07", + "title": "Domain facet counts", + "legacy_request": "GET /domains/ and GET /repos/ (count by domain_slug, joined set only)", + "candidate_request": "GET /ports/projections/repository-navigation/repositories?limit=500 (facets[] where kind = primary_domain)", + "compare": {"legacy": ["count per domain_slug"], "candidate": ["facets[].repository_count"]}, + "normalisation": ["Count only repositories present in both sides (A5-C05 join)", "Legacy domains with zero repos have no candidate facet"], + "expected": "expected-equal on the joined set" + }, + { + "id": "A5-C08", + "title": "MCP get_topic(slug)", + "legacy_request": "MCP get_topic(slug='markitect') = GET /topics/ + GET /topics/{id} + GET /progress?topic_id=...&limit=10", + "candidate_request": "MCP get_repository_navigation_facet(kind='primary_domain', value=) plus alias lookup", + "compare": {"legacy": ["topic.id", "topic.slug", "topic.domain_slug"], "candidate": ["alias legacy_topic_id/slug", "facet_value"]}, + "normalisation": [], + "expected": "documented-difference: recent_progress and workplans have no navigation counterpart (B3 progress and A2 work records own them). Only the identity/domain part is comparable." + }, + { + "id": "A5-C09", + "title": "Legacy topic fields without counterpart", + "legacy_request": "GET /topics/{id}", + "candidate_request": "n/a", + "compare": {"legacy_only": ["title", "description", "status", "created_at", "updated_at", "workstreams"]}, + "normalisation": [], + "expected": "documented-difference: no counterpart. workstreams is always [] in the live API (schema field name 'workstreams' does not match model relation 'workplans'), so no consumer can depend on its content. Titles/descriptions must stay in State Hub (or the alias owner) until the alias expires." + }, + { + "id": "A5-C10", + "title": "Rollback read", + "legacy_request": "GET /topics/?status=active and GET /topics/{id} after switching back", + "candidate_request": "n/a", + "compare": {"legacy": ["HTTP 200", "row count 16", "id/slug/domain_slug set unchanged from capture"]}, + "normalisation": [], + "expected": "expected-equal to the 2026-10-13 capture (ETag may differ only if topics changed)." + } + ] +} diff --git a/docs/inbox-change-feed-contract.md b/docs/inbox-change-feed-contract.md new file mode 100644 index 0000000..561e7d5 --- /dev/null +++ b/docs/inbox-change-feed-contract.md @@ -0,0 +1,328 @@ +# State Hub inbox change-feed contract (draft) + +Status: **draft for review**. Not implemented. Deliverable of `STATE-WP-0098-T03`. +It is the State Hub input for `HUB-WP-0011-T01` (the first row of its +"Dependency review — 2026-10-04" table). Date: 2026-10-13. + +Scope: how hub-core keeps a *derived*, read-only copy of the State Hub inbox +fresh. Hub-core would replace its one-shot snapshot (`import_snapshot` in +`hub_core/runtime/inbox_projection.py`) with this feed. This document does not +switch any reader or writer. Reader switches remain under `STATE-WP-0079-T04` and +`HUB-WP-0011-T03`. + +Sources read: `api/models/agent_message.py`, `api/routers/messages.py`, +`api/services/message_receipts.py`, migrations `f3a4b5c6d7e8` and `d7e8f9a0b1c2`, +`docs/evidence/statehub-inbox-pilot-20260905.{md,json}`, +`STATE-WP-0079` T08/T09, `HUB-WP-0011`, and hub-core `inbox_projection.py` and +`schemas/agent_message.py`. + +## 1. Identity, transport and auth + +- **Contract id:** `statehub.inbox-feed`. **Version:** `v1`. Every response + carries `"contract": "statehub.inbox-feed.v1"`. Additive fields do not bump + the version. A changed meaning, a removed field or a changed cursor encoding + bumps it to `v2`, and State Hub then serves both versions for the parity + window. +- **Endpoints (read-only, GET only):** + - `GET /messages/feed/snapshot` returns a consistent full image and the + cursor that goes with it. + - `GET /messages/feed?after=&limit=` returns changes after the + cursor. + - Both send `Cache-Control: no-store`. Any other method returns 405. +- **Direction: pull.** Hub-core's refresh runner calls State Hub over the + in-cluster ClusterIP. Today the State Hub API has no application-level auth; + access is bounded by the network (ClusterIP, the workstation SSH forward). The + feed adds **no new credentials**. It inherits that boundary. +- **Admission (unverified):** the pilot evidence proves the *opposite* + direction. NetworkPolicy admits State Hub API pods to hub-core, and + `transport_from_state_hub_without_token` returned 401. It does not show + whether hub-core pods can reach the State Hub API. If they cannot, the + rapp-core-hub packaging owner adds a scoped ingress/egress rule for the + refresh runner only. That is a packaging change, not a credential, and it + belongs to the second T01 row of HUB-WP-0011. +- **Message bodies:** the feed includes `body`, because the inbox cannot be + served without it. Evidence and receipts must keep excluding bodies, as the + pilot did. + +## 2. Consistent snapshot and monotonic cursor + +### What the schema has today + +`agent_messages` has the columns `id`, `from_agent`, `to_agent`, `subject`, +`body`, `thread_id`, `read_at`, `archived_at`, `kind`, `expires_at`, +`supersedes_id` and `created_at`. `message_receipts` has `(message_id, agent)`, +`delivered_at`, `read_at` and `acknowledged_at`. + +Neither table has `updated_at`, a sequence, a revision column or a tombstone +table. No existing column can serve as a cursor: + +- **Timestamps are application-clock values.** They come from `datetime.now()` + and `utcnow()`, not from commit time. Concurrent transactions can commit them + out of order, and they collide on ties. +- **Some changes leave no timestamp.** A receipt upsert can change one field + among several. An archive call on a row that is already archived *overwrites* + `archived_at` (see `archive_message`). A delete leaves nothing behind. + +### Required migration (described here, not written) + +One Alembic revision, roughly 80 lines: + +1. **Revision sequence.** Create the sequence `inbox_revision_seq` (bigint). +2. **Revision columns.** Add `revision bigint NOT NULL` to `agent_messages` and + to `message_receipts`. Backfill both with one value from the sequence, so + that one snapshot equals one revision. Index `revision` on both tables. +3. **Tombstone table.** Add `message_tombstones` with the columns + `message_id uuid PK`, `revision bigint NOT NULL`, + `deleted_at timestamptz NOT NULL` and `reason text`. +4. **Write trigger.** A `BEFORE INSERT OR UPDATE` trigger on both tables. It + - skips the update when `NEW IS NOT DISTINCT FROM OLD`, so that no-op + COALESCE upserts and repeated reads do not bump the revision; + - takes `pg_advisory_xact_lock()`; + - assigns **one revision per transaction**. On the first row it calls + `nextval`, caches the value with `set_config('statehub.inbox_rev', …, true)`, + and reuses it for every later row in the same transaction. +5. **Delete trigger.** A `BEFORE DELETE` trigger on `agent_messages`. It writes + a tombstone row under the same transaction revision. + +**Why the advisory lock is needed.** A sequence value is drawn at write time, +not at commit time. Without the lock, transaction A can draw revision 41, +transaction B can draw 42 and commit first, and a reader at cursor 42 would +never see 41. With the lock, inbox writes are serialized, so commit order +equals revision order. Any visible maximum revision is then a safe watermark. + +Inbox write volume is low: sends, read marks, receipts, archives. The lock cost +is negligible. *Unverified:* there is no load measurement. If the lock is +rejected, the alternative is an `xid8` column plus a +`pg_snapshot_xmin(pg_current_snapshot())` watermark. That avoids the lock but +is more complex to serve. + +**The cursor** is an opaque string that encodes `v1:`. Consumers +must not do arithmetic on it. They may compare the decoded revisions only for +the out-of-order checks in §5. + +### Snapshot + +`GET /messages/feed/snapshot` runs in one `REPEATABLE READ READ ONLY` +transaction and returns: + +``` +contract, source_instance (from /state/health instance_label), +source_time (DB now() of the snapshot txn), cursor (max visible revision), +messages[] (all rows, archived included), receipts[], tombstones[] (within horizon), +count, content_hash (sha256 over the normalized rows, as in the pilot) +``` + +It is returned as a single response. The pilot captured 20 rows. *Unverified:* +the current production row count. If the response grows past a few MB, the +snapshot is paged inside one exported snapshot (`pg_export_snapshot`). That is +a v1 addition, not a new version. + +### No-migration interim (not recommended) + +Hub-core could poll the snapshot endpoint and diff it, with +`content_hash` + `source_time` as the revision. It gives freshness but no +ordering: hashes cannot be compared for order, and `source_time` is only as +monotonic as the DB clock. It also cannot distinguish a delete from a +filtering bug. Acceptable only as a stop-gap behind the existing +`snapshot-pilot` label. + +## 3. Change records and event types + +The wire carries **state, not deltas**. Each change is the full post-image of +one entity at its revision: + +``` +{revision, entity: "message"|"receipt"|"tombstone", key, op: "upsert"|"delete", state} +``` + +Hub-core works out the event type by comparing the post-image with its +current row. This keeps replay idempotent (§6). A page never splits a +revision: the server extends the page past `limit` until the last revision is +complete. + +Only the transitions below can occur in the current State Hub model. + +| Event | Source mechanics (today) | Appears on the feed as | +| --- | --- | --- | +| `created` | `POST /messages/`, `POST /{id}/reply` | `message` upsert with no prior row; `thread_id`, `supersedes_id`, `kind` and `expires_at` set at creation | +| `read` (direct) | `PATCH /{id}/read` without a reader; a reply marks the original as read; archiving a direct message stamps `read_at` if it is null | `message` upsert where `read_at` goes from null to a value. There is one global `read_at` and no reader identity | +| `receipt` (broadcast, per reader) | `PATCH /{id}/read?reader=`, `POST /{id}/ack`, a reply to a broadcast, and **`GET /messages/?unread_only=true` (stamps `delivered_at`)** | `receipt` upsert keyed `(message_id, agent)`. The fields are first-write-wins (`COALESCE`) | +| `archived` | `PATCH /{id}/archive`, and supersession of a broadcast | `message` upsert where `archived_at` is set. Re-archiving today *changes* `archived_at`, which arrives as a normal higher-revision upsert | +| `superseded` | sending a broadcast with `supersedes_id` archives its predecessor in the same transaction | one revision that holds the new `message` and the predecessor's archive upsert | +| reply / thread link | `thread_id` points at the root; a reply inherits the root | carried in `created`. There is no separate link event | +| `deleted` | **no API route deletes messages.** Only a manual DB delete can | `tombstone` record. The delete cascades: receipts are dropped, and `thread_id`/`supersedes_id` on other rows become NULL. Those referencing rows arrive as upserts in the same revision | + +**State that does not exist, so no event is emitted:** unarchive (no route +does it), message edits, per-reader read state for *direct* messages, who +archived a message, and per-reader archive. + +**Expiry is not an event.** `expires_at` is a stored value, and visibility +(`not_expired`) is evaluated against the clock at read time. Hub-core must +evaluate it the same way. + +**Message identity aliases** are resolved by State Hub reads +(`MessageIdentityAlias`), but no State Hub code writes them. *Unverified:* +whether that table is populated in the State Hub DB. Aliases are not in v1. + +### Gaps against hub-core's pilot + +- **Nothing the pilot carries is missing from State Hub.** Its `MessageRead` + (`id`, `from_agent`, `to_agent`, `subject`, `body`, `thread_id`, `read_at`, + `archived_at`, `created_at`) is a strict subset. +- **The reverse gap is hub-core's to close.** Hub-core's model lacks `kind`, + `expires_at`, `supersedes_id` and receipts. Without receipts, its + `unread_only` filter on `read_at` gives wrong answers for broadcasts: since + STATE-WP-0093, broadcast read state lives in `message_receipts`. +- **Delivery stamps stop for switched readers.** A State Hub + `GET ?unread_only=true` writes `delivered_at`. A reader moved to hub-core + would stop producing those stamps unless hub-core writes them back, and that + would make hub-core a second writer. See founder decision D3. +- **Recipient slugs are stored as written.** `to_agent` may hold a historical + repository slug, and State Hub canonicalizes it when reading + (`_reader_values`). The feed emits stored values. Resolving aliases is + HUB-WP-0011-T02 scope. + +## 4. Deletion versus retention + +### Settled by the STATE-WP-0079 T08/T09 record + +- **State Hub is the only message writer** throughout the read cutover. Sends, + read marks, acks, archives and replies all stay on State Hub. A switched + reader that changes state calls the State Hub routes. +- **The hub-core copy is derived and read-only.** It is never authoritative, + and it never mutates source history. +- **Rollback is State Hub `/messages/`.** +- **The snapshot pilot must not be presented as a fresh inbox.** +- **Writer cutover and the B2/B3 route migration are out of scope here.** + +### Proposed by this contract (to be accepted with the contract) + +- **Archive is not deletion.** Archived rows stay in the feed and in the + snapshot. Hub-core keeps them so that thread closure holds, as the pilot + already requires. +- **A State Hub delete always produces a tombstone** (the trigger in §2), even + if the delete is a manual DB operation. +- **On a tombstone, hub-core deletes its derived row** and keeps only + `(message_id, revision, deleted_at)`. It must not keep the body. +- **Hub-core never emits its own tombstones.** + +### Open founder decisions + +These are not settled by T08/T09. They must be answered before hub-core +implements against this contract. + +- **D1 — Tombstone retention horizon.** *Recommend:* keep tombstones for 90 + days, and at least twice the maximum consumer lag. A cursor older than the + horizon gets 410, and the consumer re-snapshots. +- **D2 — Purging during the retirement window.** Should State Hub purge any + messages before it retires, such as expired `news` (the default TTL is 30 + days) or old archived rows? *Recommend:* no purge. Use archive only until + State Hub retires. Retention is then decided once, by the receiving owner. +- **D3 — Delivery stamps.** Delivery stamps (`delivered_at`) are lost for a + reader that moves to hub-core. *Recommend:* accept the loss, and pick the + first switched reader (`STATE-WP-0098-T04`) from readers that do not rely on + broadcast delivery tracking. Do not add a write-back path. +- **D4 — Approve the migration.** Approve the §2 migration (revision columns, + trigger, tombstone table) as a retirement-enabling change to a frozen + service. *Recommend:* approve. It is bounded and additive, and it is the + only way to give a true cursor. + +## 5. Recovery, ordering and freshness + +**Feed response.** + +``` +contract, source_instance, source_time (DB now() at serve time), +high_water (cursor of the max committed revision), next_cursor, has_more, +compacted_before (oldest cursor still servable), changes[] +``` + +**Compaction is implicit.** Rows hold only their latest revision, so +intermediate states are never replayable. A consumer that resumes from cursor +`c` gets the latest post-image of every entity changed after `c`. Converging +on the final state is guaranteed; seeing every step is not. Tombstones are the +only history kept, and they are bounded by D1. + +**Recovery.** + +| Situation | Behaviour | +| --- | --- | +| `after` is older than `compacted_before` | **410 Gone** with `{"resnapshot_required": true}` | +| Cursor from another `source_instance` or another contract version | **409**. The consumer re-snapshots | +| Consumer restarts | It resumes from its persisted cursor. Replaying pages it already applied is a no-op (§6) | +| Re-snapshot | Applies the snapshot as a full image under the §6 rules. Rows absent from the snapshot and not tombstoned are a consistency error: refuse and alert. They are not silently deleted | + +**Out-of-order refusal (consumer side).** + +- Reject a page whose `changes[0].revision` is at or below the stored cursor, + unless it is an exact replay. +- Reject a page whose `high_water` is lower than one already seen from the + same `source_instance`. That indicates a source restore or rollback. The + consumer stops serving as fresh and re-snapshots. + +**Freshness.** + +- State Hub provides `source_time`, `high_water` and `source_instance` on + every response, including empty ones. An empty page is a heartbeat: "nothing + changed as of `source_time`". +- Lag = consumer clock − last successful `source_time`. +- Maximum lag, stale-source behaviour and the read-mode header are hub-core's + and rapp-core-hub's to set. This contract only supplies the signals. + +## 6. Idempotency without an unconditional overwrite + +Hub-core's importer refuses any receiver that differs from the bundle. The +contract replaces "refuse any difference" with "refuse any difference that a +higher source revision does not justify". Hub-core stores `source_revision` on +each derived row and applies these rules per entity key: + +| Incoming revision `r` vs stored `s` | Outcome | +| --- | --- | +| No stored row | Insert | +| `r < s` | Ignore (stale replay). Log it | +| `r == s`, identical normalized hash | No-op | +| `r == s`, different content | **Refuse.** Integrity error; stop and re-snapshot | +| `r > s` | Replace with the post-image, after the field checks below | + +**Field checks (for `r > s`).** These catch source bugs instead of copying +them: + +- Immutable fields must not change: `id`, `from_agent`, `to_agent`, `subject`, + `body`, `kind`, `created_at`, `expires_at`. +- `thread_id` and `supersedes_id` may change only from a value to NULL, and + only in a revision that tombstones the referenced message. +- `read_at` and the receipt fields may change only from null to a value. +- `archived_at` may change from null to a value, or to a later value + (re-archive). It never returns to null. + +Anything else is refused, and hub-core re-snapshots. + +The whole page is applied in one destination transaction, in revision order, +and the cursor is updated in the same transaction. Thread roots arrive in the +same or an earlier revision, because a reply cannot exist before its root, so +the pilot's thread-closure check still holds. + +Small State Hub fix (part of the implementation): make `archive_message` +first-write-wins for `archived_at`. The re-archive rule can then tighten to +"null to value only". + +## 7. State Hub implementation sketch (not implemented) + +| File | Change | Size | +| --- | --- | --- | +| `migrations/versions/_inbox_feed_revision.py` | §2 sequence, columns, backfill, triggers, tombstone table | ~80 lines | +| `api/models/agent_message.py` | `revision` on both models; `MessageTombstone` model | ~20 | +| `api/schemas/inbox_feed.py` | snapshot/page/change envelopes; cursor encode and decode | ~70 | +| `api/routers/message_feed.py` | the two GET routes; repeatable-read transaction; page extension to a revision boundary; 409/410 handling | ~130 | +| `api/main.py` | include the router before `messages.router` | ~2 | +| `api/routers/messages.py` | first-write-wins `archived_at` | ~3 | +| `tests/test_message_feed.py` | snapshot/feed parity, per-transaction revisions, no bump on no-op upsert, tombstone on delete, 410 past horizon, page boundary not split, replay no-op | ~250 | + +The total is about 550 lines, roughly half of them tests. Expect one or two +sessions, plus a deploy through the normal State Hub migration path. + +The trigger needs PostgreSQL. *Unverified:* whether the test suite runs on +SQLite. If it does, the trigger tests need the Postgres test fixture. + +The hub-core side (`source_revision` column, page applier, cursor store, +staleness) is not sketched here. It belongs to HUB-WP-0011-T01. diff --git a/workplans/STATE-WP-0098-hub-core-receipt-inputs.md b/workplans/STATE-WP-0098-hub-core-receipt-inputs.md index 1c53ccb..2f21aec 100644 --- a/workplans/STATE-WP-0098-hub-core-receipt-inputs.md +++ b/workplans/STATE-WP-0098-hub-core-receipt-inputs.md @@ -25,7 +25,7 @@ acceptance retained by State Hub. ```task id: STATE-WP-0098-T01 -status: todo +status: done priority: high state_hub_task_id: "2197b8ae-40d1-52ff-b9d5-3244b9e5e8a9" ``` @@ -42,7 +42,7 @@ plus a JSON case list, and notify hub-core. ```task id: STATE-WP-0098-T02 -status: todo +status: progress priority: medium state_hub_task_id: "e6ef3381-9867-52f6-8c9c-2aa24f9a16b3" ``` @@ -56,7 +56,9 @@ companion section. A4 redirects stay behind the retained UI as rollback. ```task id: STATE-WP-0098-T03 -status: todo +status: wait +needs_human: true +blocking_reason: "Founder decisions on tombstone retention, pre-retirement purge, broadcast delivery stamps and the additive migration (docs/inbox-change-feed-contract.md)" priority: high state_hub_task_id: "17c424c9-4089-56d2-9057-42e34f13827b" ``` @@ -82,3 +84,20 @@ Input for HUB-WP-0011-T03. Name the one inbox reader to switch first. Give its parity sample/window and the rollback endpoint (current State Hub `/messages/`). Get the reader owner's agreement. This depends on T03 being accepted by hub-core. + +2026-10-13 delivery: +- T01: `docs/a5-topic-spine-comparison-packet.md` and + `docs/evidence/a5-topic-spine-comparison-cases.json` (10 cases). + - A5 is 3 topic reads, 4 writes and the `topics` table, not 8 read + endpoints. + - All direct callers are State Hub's own tooling. 126 repos keep a + `topic_id` that must stay stable. + - Gaps that block hub-core: `legacy_topic_aliases` is hard-coded empty in + hub-core `repository_navigation.py`; `RM_SLICE_TOPICSPINE` is not + implemented anywhere; topic routes are not metered; the 154 legacy vs 123 + published repository gap is unexplained. +- T02: 22 A4 dashboard views mapped to owners (same packet). Owner requests + sent 2026-10-13. +- T03: `docs/inbox-change-feed-contract.md` drafted. A migration is required + (no revision column on `agent_messages`/`message_receipts`, no tombstones). + Four founder decisions are open.