# State Hub inbox change-feed contract Status: **implemented (not deployed)** (2026-10-14, `STATE-WP-0098-T05`; migration `c4d5e6f7a8b9`). Accepted by the founder 2026-10-13: D1-D4 as recommended. Deliverable of `STATE-WP-0098-T03`. Deploy is a separate confirmed step. Implementation details and deviations are in §8. It is the State Hub input for `HUB-WP-0011-T01` (the first row of its "Dependency review — 2026-10-04" table). Date: 2026-10-13. Scope: how hub-core keeps a *derived*, read-only copy of the State Hub inbox fresh. Hub-core would replace its one-shot snapshot (`import_snapshot` in `hub_core/runtime/inbox_projection.py`) with this feed. This document does not switch any reader or writer. Reader switches remain under `STATE-WP-0079-T04` and `HUB-WP-0011-T03`. Sources read: `api/models/agent_message.py`, `api/routers/messages.py`, `api/services/message_receipts.py`, migrations `f3a4b5c6d7e8` and `d7e8f9a0b1c2`, `docs/evidence/statehub-inbox-pilot-20260905.{md,json}`, `STATE-WP-0079` T08/T09, `HUB-WP-0011`, and hub-core `inbox_projection.py` and `schemas/agent_message.py`. ## 1. Identity, transport and auth - **Contract id:** `statehub.inbox-feed`. **Version:** `v1`. Every response carries `"contract": "statehub.inbox-feed.v1"`. Additive fields do not bump the version. A changed meaning, a removed field or a changed cursor encoding bumps it to `v2`, and State Hub then serves both versions for the parity window. - **Endpoints (read-only, GET only):** - `GET /messages/feed/snapshot` returns a consistent full image and the cursor that goes with it. - `GET /messages/feed?after=&limit=` returns changes after the cursor. - Both send `Cache-Control: no-store`. Any other method returns 405. - **Direction: pull.** Hub-core's refresh runner calls State Hub over the in-cluster ClusterIP. Today the State Hub API has no application-level auth; access is bounded by the network (ClusterIP, the workstation SSH forward). The feed adds **no new credentials**. It inherits that boundary. - **Admission (unverified):** the pilot evidence proves the *opposite* direction. NetworkPolicy admits State Hub API pods to hub-core, and `transport_from_state_hub_without_token` returned 401. It does not show whether hub-core pods can reach the State Hub API. If they cannot, the rapp-core-hub packaging owner adds a scoped ingress/egress rule for the refresh runner only. That is a packaging change, not a credential, and it belongs to the second T01 row of HUB-WP-0011. - **Message bodies:** the feed includes `body`, because the inbox cannot be served without it. Evidence and receipts must keep excluding bodies, as the pilot did. ## 2. Consistent snapshot and monotonic cursor ### What the schema has today `agent_messages` has the columns `id`, `from_agent`, `to_agent`, `subject`, `body`, `thread_id`, `read_at`, `archived_at`, `kind`, `expires_at`, `supersedes_id` and `created_at`. `message_receipts` has `(message_id, agent)`, `delivered_at`, `read_at` and `acknowledged_at`. Neither table has `updated_at`, a sequence, a revision column or a tombstone table. No existing column can serve as a cursor: - **Timestamps are application-clock values.** They come from `datetime.now()` and `utcnow()`, not from commit time. Concurrent transactions can commit them out of order, and they collide on ties. - **Some changes leave no timestamp.** A receipt upsert can change one field among several. An archive call on a row that is already archived *overwrites* `archived_at` (see `archive_message`). A delete leaves nothing behind. ### Required migration (described here, not written) One Alembic revision, roughly 80 lines: 1. **Revision sequence.** Create the sequence `inbox_revision_seq` (bigint). 2. **Revision columns.** Add `revision bigint NOT NULL` to `agent_messages` and to `message_receipts`. Backfill both with one value from the sequence, so that one snapshot equals one revision. Index `revision` on both tables. 3. **Tombstone table.** Add `message_tombstones` with the columns `message_id uuid PK`, `revision bigint NOT NULL`, `deleted_at timestamptz NOT NULL` and `reason text`. 4. **Write trigger.** A `BEFORE INSERT OR UPDATE` trigger on both tables. It - skips the update when `NEW IS NOT DISTINCT FROM OLD`, so that no-op COALESCE upserts and repeated reads do not bump the revision; - takes `pg_advisory_xact_lock()`; - assigns **one revision per transaction**. On the first row it calls `nextval`, caches the value with `set_config('statehub.inbox_rev', …, true)`, and reuses it for every later row in the same transaction. 5. **Delete trigger.** A `BEFORE DELETE` trigger on `agent_messages`. It writes a tombstone row under the same transaction revision. **Why the advisory lock is needed.** A sequence value is drawn at write time, not at commit time. Without the lock, transaction A can draw revision 41, transaction B can draw 42 and commit first, and a reader at cursor 42 would never see 41. With the lock, inbox writes are serialized, so commit order equals revision order. Any visible maximum revision is then a safe watermark. Inbox write volume is low: sends, read marks, receipts, archives. The lock cost is negligible. *Unverified:* there is no load measurement. If the lock is rejected, the alternative is an `xid8` column plus a `pg_snapshot_xmin(pg_current_snapshot())` watermark. That avoids the lock but is more complex to serve. **The cursor** is an opaque string that encodes `v1:`. Consumers must not do arithmetic on it. They may compare the decoded revisions only for the out-of-order checks in §5. ### Snapshot `GET /messages/feed/snapshot` runs in one `REPEATABLE READ READ ONLY` transaction and returns: ``` contract, source_instance (from /state/health instance_label), source_time (DB now() of the snapshot txn), cursor (max visible revision), messages[] (all rows, archived included), receipts[], tombstones[] (within horizon), count, content_hash (sha256 over the normalized rows, as in the pilot) ``` It is returned as a single response. The pilot captured 20 rows. *Unverified:* the current production row count. If the response grows past a few MB, the snapshot is paged inside one exported snapshot (`pg_export_snapshot`). That is a v1 addition, not a new version. ### No-migration interim (not recommended) Hub-core could poll the snapshot endpoint and diff it, with `content_hash` + `source_time` as the revision. It gives freshness but no ordering: hashes cannot be compared for order, and `source_time` is only as monotonic as the DB clock. It also cannot distinguish a delete from a filtering bug. Acceptable only as a stop-gap behind the existing `snapshot-pilot` label. ## 3. Change records and event types The wire carries **state, not deltas**. Each change is the full post-image of one entity at its revision: ``` {revision, entity: "message"|"receipt"|"tombstone", key, op: "upsert"|"delete", state} ``` Hub-core works out the event type by comparing the post-image with its current row. This keeps replay idempotent (§6). A page never splits a revision: the server extends the page past `limit` until the last revision is complete. Only the transitions below can occur in the current State Hub model. | Event | Source mechanics (today) | Appears on the feed as | | --- | --- | --- | | `created` | `POST /messages/`, `POST /{id}/reply` | `message` upsert with no prior row; `thread_id`, `supersedes_id`, `kind` and `expires_at` set at creation | | `read` (direct) | `PATCH /{id}/read` without a reader; a reply marks the original as read; archiving a direct message stamps `read_at` if it is null | `message` upsert where `read_at` goes from null to a value. There is one global `read_at` and no reader identity | | `receipt` (broadcast, per reader) | `PATCH /{id}/read?reader=`, `POST /{id}/ack`, a reply to a broadcast, and **`GET /messages/?unread_only=true` (stamps `delivered_at`)** | `receipt` upsert keyed `(message_id, agent)`. The fields are first-write-wins (`COALESCE`) | | `archived` | `PATCH /{id}/archive`, and supersession of a broadcast | `message` upsert where `archived_at` is set. Since T05, re-archiving keeps the first `archived_at` and produces no new revision | | `superseded` | sending a broadcast with `supersedes_id` archives its predecessor in the same transaction | one revision that holds the new `message` and the predecessor's archive upsert | | reply / thread link | `thread_id` points at the root; a reply inherits the root | carried in `created`. There is no separate link event | | `deleted` | **no API route deletes messages.** Only a manual DB delete can | `tombstone` record. The delete cascades: receipts are dropped, and `thread_id`/`supersedes_id` on other rows become NULL. Those referencing rows arrive as upserts in the same revision | **State that does not exist, so no event is emitted:** unarchive (no route does it), message edits, per-reader read state for *direct* messages, who archived a message, and per-reader archive. **Expiry is not an event.** `expires_at` is a stored value, and visibility (`not_expired`) is evaluated against the clock at read time. Hub-core must evaluate it the same way. **Message identity aliases** are resolved by State Hub reads (`MessageIdentityAlias`), but no State Hub code writes them. *Unverified:* whether that table is populated in the State Hub DB. Aliases are not in v1. ### Gaps against hub-core's pilot - **Nothing the pilot carries is missing from State Hub.** Its `MessageRead` (`id`, `from_agent`, `to_agent`, `subject`, `body`, `thread_id`, `read_at`, `archived_at`, `created_at`) is a strict subset. - **The reverse gap is hub-core's to close.** Hub-core's model lacks `kind`, `expires_at`, `supersedes_id` and receipts. Without receipts, its `unread_only` filter on `read_at` gives wrong answers for broadcasts: since STATE-WP-0093, broadcast read state lives in `message_receipts`. - **Delivery stamps stop for switched readers.** A State Hub `GET ?unread_only=true` writes `delivered_at`. A reader moved to hub-core would stop producing those stamps unless hub-core writes them back, and that would make hub-core a second writer. See founder decision D3. - **Recipient slugs are stored as written.** `to_agent` may hold a historical repository slug, and State Hub canonicalizes it when reading (`_reader_values`). The feed emits stored values. Resolving aliases is HUB-WP-0011-T02 scope. ## 4. Deletion versus retention ### Settled by the STATE-WP-0079 T08/T09 record - **State Hub is the only message writer** throughout the read cutover. Sends, read marks, acks, archives and replies all stay on State Hub. A switched reader that changes state calls the State Hub routes. - **The hub-core copy is derived and read-only.** It is never authoritative, and it never mutates source history. - **Rollback is State Hub `/messages/`.** - **The snapshot pilot must not be presented as a fresh inbox.** - **Writer cutover and the B2/B3 route migration are out of scope here.** ### Proposed by this contract (to be accepted with the contract) - **Archive is not deletion.** Archived rows stay in the feed and in the snapshot. Hub-core keeps them so that thread closure holds, as the pilot already requires. - **A State Hub delete always produces a tombstone** (the trigger in §2), even if the delete is a manual DB operation. - **On a tombstone, hub-core deletes its derived row** and keeps only `(message_id, revision, deleted_at)`. It must not keep the body. - **Hub-core never emits its own tombstones.** ### Founder decisions (resolved 2026-10-13) The T08/T09 record did not settle these. The founder accepted all four recommendations on 2026-10-13: 90-day tombstones (D1), no purge (D2), delivery-stamp loss accepted (D3), migration approved (D4). - **D1 — Tombstone retention horizon.** *Recommend:* keep tombstones for 90 days, and at least twice the maximum consumer lag. A cursor older than the horizon gets 410, and the consumer re-snapshots. - **D2 — Purging during the retirement window.** Should State Hub purge any messages before it retires, such as expired `news` (the default TTL is 30 days) or old archived rows? *Recommend:* no purge. Use archive only until State Hub retires. Retention is then decided once, by the receiving owner. - **D3 — Delivery stamps.** Delivery stamps (`delivered_at`) are lost for a reader that moves to hub-core. *Recommend:* accept the loss, and pick the first switched reader (`STATE-WP-0098-T04`) from readers that do not rely on broadcast delivery tracking. Do not add a write-back path. - **D4 — Approve the migration.** Approve the §2 migration (revision columns, trigger, tombstone table) as a retirement-enabling change to a frozen service. *Recommend:* approve. It is bounded and additive, and it is the only way to give a true cursor. ## 5. Recovery, ordering and freshness **Feed response.** ``` contract, source_instance, source_time (DB now() at serve time), high_water (cursor of the max committed revision), next_cursor, has_more, compacted_before (oldest cursor still servable), changes[] ``` **Compaction is implicit.** Rows hold only their latest revision, so intermediate states are never replayable. A consumer that resumes from cursor `c` gets the latest post-image of every entity changed after `c`. Converging on the final state is guaranteed; seeing every step is not. Tombstones are the only history kept, and they are bounded by D1. **Recovery.** | Situation | Behaviour | | --- | --- | | `after` is older than `compacted_before` | **410 Gone** with `{"resnapshot_required": true}` | | Cursor from another `source_instance` or another contract version | **409**. The consumer re-snapshots | | Consumer restarts | It resumes from its persisted cursor. Replaying pages it already applied is a no-op (§6) | | Re-snapshot | Applies the snapshot as a full image under the §6 rules. Rows absent from the snapshot and not tombstoned are a consistency error: refuse and alert. They are not silently deleted | **Out-of-order refusal (consumer side).** - Reject a page whose `changes[0].revision` is at or below the stored cursor, unless it is an exact replay. - Reject a page whose `high_water` is lower than one already seen from the same `source_instance`. That indicates a source restore or rollback. The consumer stops serving as fresh and re-snapshots. **Freshness.** - State Hub provides `source_time`, `high_water` and `source_instance` on every response, including empty ones. An empty page is a heartbeat: "nothing changed as of `source_time`". - Lag = consumer clock − last successful `source_time`. - Maximum lag, stale-source behaviour and the read-mode header are hub-core's and rapp-core-hub's to set. This contract only supplies the signals. ## 6. Idempotency without an unconditional overwrite Hub-core's importer refuses any receiver that differs from the bundle. The contract replaces "refuse any difference" with "refuse any difference that a higher source revision does not justify". Hub-core stores `source_revision` on each derived row and applies these rules per entity key: | Incoming revision `r` vs stored `s` | Outcome | | --- | --- | | No stored row | Insert | | `r < s` | Ignore (stale replay). Log it | | `r == s`, identical normalized hash | No-op | | `r == s`, different content | **Refuse.** Integrity error; stop and re-snapshot | | `r > s` | Replace with the post-image, after the field checks below | **Field checks (for `r > s`).** These catch source bugs instead of copying them: - Immutable fields must not change: `id`, `from_agent`, `to_agent`, `subject`, `body`, `kind`, `created_at`, `expires_at`. - `thread_id` and `supersedes_id` may change only from a value to NULL, and only in a revision that tombstones the referenced message. - `read_at` and the receipt fields may change only from null to a value. - `archived_at` may change only from null to a value (first-write-wins since T05). It never returns to null. Anything else is refused, and hub-core re-snapshots. The whole page is applied in one destination transaction, in revision order, and the cursor is updated in the same transaction. Thread roots arrive in the same or an earlier revision, because a reply cannot exist before its root, so the pilot's thread-closure check still holds. Small State Hub fix (done in T05): `archive_message` is first-write-wins for `archived_at`, so the re-archive rule above is "null to value only". ## 7. State Hub implementation sketch (implemented 2026-10-14, see §8) | File | Change | Size | | --- | --- | --- | | `migrations/versions/_inbox_feed_revision.py` | §2 sequence, columns, backfill, triggers, tombstone table | ~80 lines | | `api/models/agent_message.py` | `revision` on both models; `MessageTombstone` model | ~20 | | `api/schemas/inbox_feed.py` | snapshot/page/change envelopes; cursor encode and decode | ~70 | | `api/routers/message_feed.py` | the two GET routes; repeatable-read transaction; page extension to a revision boundary; 409/410 handling | ~130 | | `api/main.py` | include the router before `messages.router` | ~2 | | `api/routers/messages.py` | first-write-wins `archived_at` | ~3 | | `tests/test_message_feed.py` | snapshot/feed parity, per-transaction revisions, no bump on no-op upsert, tombstone on delete, 410 past horizon, page boundary not split, replay no-op | ~250 | The total is about 550 lines, roughly half of them tests. Expect one or two sessions, plus a deploy through the normal State Hub migration path. The trigger needs PostgreSQL. *Unverified:* whether the test suite runs on SQLite. If it does, the trigger tests need the Postgres test fixture. The hub-core side (`source_revision` column, page applier, cursor store, staleness) is not sketched here. It belongs to HUB-WP-0011-T01. ## 8. Implementation notes (2026-10-14) Implemented as `STATE-WP-0098-T05`. Files: migration `migrations/versions/c4d5e6f7a8b9_inbox_feed_revision.py`, `api/models/agent_message.py`, `api/schemas/inbox_feed.py`, `api/routers/message_feed.py`, `api/main.py`, `api/routers/messages.py`, `tests/test_message_feed.py`. Where this section and §1-§7 differ, this section describes what the code does. - **Revision cache.** `inbox_feed_revision()` caches `":"` in the transaction-local GUC `statehub.inbox_rev` and reuses it only when the txid matches. A savepoint that rolls back after drawing a revision can make a transaction use two revisions; both stay ordered under the advisory lock (key `5348490098`). - **Keys and state.** `key` is the message UUID for `message` and `tombstone`, and `":"` for `receipt`. `state` is the full row and includes `revision`. Snapshot rows have the same shape as `state`. Within a revision the order is messages, receipts, tombstones. - **Foreign cursor (409).** The cursor stays `v1:` and cannot carry the instance. The server answers 409 (`resnapshot_required: true`) when the cursor has another version prefix (for example `v2:`), when the optional query parameter `source_instance` differs from this instance's `instance_label`, or when the cursor is ahead of `high_water` (another instance or a restored source). A malformed cursor gets 400. - **Horizon (410).** `compacted_before` is `v1:`, where `R` is the highest revision of any tombstone older than 90 days (`v1:0` if there is none). Those tombstones are no longer served, and `after < R` gets 410 with `resnapshot_required: true`. The horizon is enforced logically. Nothing purges tombstones physically (D2). The snapshot also carries `compacted_before` (additive). - **Paging.** `limit` defaults to 500, maximum 5000. A page may exceed it to finish a revision. An empty page returns `next_cursor` equal to `after`. - **content_hash** is sha256 over canonical JSON (`sort_keys`, compact separators) of `{messages, receipts, tombstones}`. Rows are sorted by key and timestamps are ISO strings. It is deterministic but not byte-identical to the pilot's hash. - **Delete reason.** The tombstone `reason` comes from `SET LOCAL statehub.delete_reason = '...'` if the deleting session sets it, and is NULL otherwise. - **Not covered.** A direct `DELETE` on `message_receipts` writes no tombstone, and no API route does this. Receipts disappear only through the message delete cascade, which the message tombstone implies. - **Tests and create_all.** The test suite builds its schema with `Base.metadata.create_all`. The models attach the same trigger DDL via `after_create`. The migration keeps its own frozen copy, and a test asserts the two are identical. It also runs the migration's downgrade and upgrade on the test database in a rolled-back transaction.