state-hub/docs/inbox-change-feed-contract.md

State Hub inbox change-feed contract

Status: implemented (not deployed) (2026-10-14, STATE-WP-0098-T05; migration c4d5e6f7a8b9). Accepted by the founder 2026-10-13: D1-D4 as recommended. Deliverable of STATE-WP-0098-T03. Deploy is a separate confirmed step. Implementation details and deviations are in §8. It is the State Hub input for HUB-WP-0011-T01 (the first row of its "Dependency review — 2026-10-04" table). Date: 2026-10-13.

Scope: how hub-core keeps a derived, read-only copy of the State Hub inbox fresh. Hub-core would replace its one-shot snapshot (import_snapshot in hub_core/runtime/inbox_projection.py) with this feed. This document does not switch any reader or writer. Reader switches remain under STATE-WP-0079-T04 and HUB-WP-0011-T03.

Sources read: api/models/agent_message.py, api/routers/messages.py, api/services/message_receipts.py, migrations f3a4b5c6d7e8 and d7e8f9a0b1c2, docs/evidence/statehub-inbox-pilot-20260905.{md,json}, STATE-WP-0079 T08/T09, HUB-WP-0011, and hub-core inbox_projection.py and schemas/agent_message.py.

1. Identity, transport and auth

• Contract id: statehub.inbox-feed. Version: v1. Every response carries "contract": "statehub.inbox-feed.v1". Additive fields do not bump the version. A changed meaning, a removed field or a changed cursor encoding bumps it to v2, and State Hub then serves both versions for the parity window.
• Endpoints (read-only, GET only):
  • GET /messages/feed/snapshot returns a consistent full image and the cursor that goes with it.
  • GET /messages/feed?after=<cursor>&limit=<n> returns changes after the cursor.
  • Both send Cache-Control: no-store. Any other method returns 405.
• Direction: pull. Hub-core's refresh runner calls State Hub over the in-cluster ClusterIP. Today the State Hub API has no application-level auth; access is bounded by the network (ClusterIP, the workstation SSH forward). The feed adds no new credentials. It inherits that boundary.
• Admission (unverified): the pilot evidence proves the opposite direction. NetworkPolicy admits State Hub API pods to hub-core, and transport_from_state_hub_without_token returned 401. It does not show whether hub-core pods can reach the State Hub API. If they cannot, the rapp-core-hub packaging owner adds a scoped ingress/egress rule for the refresh runner only. That is a packaging change, not a credential, and it belongs to the second T01 row of HUB-WP-0011.
• Message bodies: the feed includes body, because the inbox cannot be served without it. Evidence and receipts must keep excluding bodies, as the pilot did.

2. Consistent snapshot and monotonic cursor

What the schema has today

agent_messages has the columns id, from_agent, to_agent, subject, body, thread_id, read_at, archived_at, kind, expires_at, supersedes_id and created_at. message_receipts has (message_id, agent), delivered_at, read_at and acknowledged_at.

Neither table has updated_at, a sequence, a revision column or a tombstone table. No existing column can serve as a cursor:

• Timestamps are application-clock values. They come from datetime.now() and utcnow(), not from commit time. Concurrent transactions can commit them out of order, and they collide on ties.
• Some changes leave no timestamp. A receipt upsert can change one field among several. An archive call on a row that is already archived overwrites archived_at (see archive_message). A delete leaves nothing behind.

Required migration (described here, not written)

One Alembic revision, roughly 80 lines:

1. Revision sequence. Create the sequence inbox_revision_seq (bigint).
2. Revision columns. Add revision bigint NOT NULL to agent_messages and to message_receipts. Backfill both with one value from the sequence, so that one snapshot equals one revision. Index revision on both tables.
3. Tombstone table. Add message_tombstones with the columns message_id uuid PK, revision bigint NOT NULL, deleted_at timestamptz NOT NULL and reason text.
4. Write trigger. A BEFORE INSERT OR UPDATE trigger on both tables. It
   • skips the update when NEW IS NOT DISTINCT FROM OLD, so that no-op COALESCE upserts and repeated reads do not bump the revision;
   • takes pg_advisory_xact_lock(<inbox-feed constant>);
   • assigns one revision per transaction. On the first row it calls nextval, caches the value with set_config('statehub.inbox_rev', …, true), and reuses it for every later row in the same transaction.
5. Delete trigger. A BEFORE DELETE trigger on agent_messages. It writes a tombstone row under the same transaction revision.

Why the advisory lock is needed. A sequence value is drawn at write time, not at commit time. Without the lock, transaction A can draw revision 41, transaction B can draw 42 and commit first, and a reader at cursor 42 would never see 41. With the lock, inbox writes are serialized, so commit order equals revision order. Any visible maximum revision is then a safe watermark.

Inbox write volume is low: sends, read marks, receipts, archives. The lock cost is negligible. Unverified: there is no load measurement. If the lock is rejected, the alternative is an xid8 column plus a pg_snapshot_xmin(pg_current_snapshot()) watermark. That avoids the lock but is more complex to serve.

The cursor is an opaque string that encodes v1:<revision>. Consumers must not do arithmetic on it. They may compare the decoded revisions only for the out-of-order checks in §5.

Snapshot

GET /messages/feed/snapshot runs in one REPEATABLE READ READ ONLY transaction and returns:

contract, source_instance (from /state/health instance_label),
source_time (DB now() of the snapshot txn), cursor (max visible revision),
messages[] (all rows, archived included), receipts[], tombstones[] (within horizon),
count, content_hash (sha256 over the normalized rows, as in the pilot)

It is returned as a single response. The pilot captured 20 rows. Unverified: the current production row count. If the response grows past a few MB, the snapshot is paged inside one exported snapshot (pg_export_snapshot). That is a v1 addition, not a new version.

No-migration interim (not recommended)

Hub-core could poll the snapshot endpoint and diff it, with content_hash + source_time as the revision. It gives freshness but no ordering: hashes cannot be compared for order, and source_time is only as monotonic as the DB clock. It also cannot distinguish a delete from a filtering bug. Acceptable only as a stop-gap behind the existing snapshot-pilot label.

3. Change records and event types

The wire carries state, not deltas. Each change is the full post-image of one entity at its revision:

{revision, entity: "message"|"receipt"|"tombstone", key, op: "upsert"|"delete", state}

Hub-core works out the event type by comparing the post-image with its current row. This keeps replay idempotent (§6). A page never splits a revision: the server extends the page past limit until the last revision is complete.

Only the transitions below can occur in the current State Hub model.

Event	Source mechanics (today)	Appears on the feed as
created	POST /messages/, POST /{id}/reply	message upsert with no prior row; thread_id, supersedes_id, kind and expires_at set at creation
read (direct)	PATCH /{id}/read without a reader; a reply marks the original as read; archiving a direct message stamps read_at if it is null	message upsert where read_at goes from null to a value. There is one global read_at and no reader identity
receipt (broadcast, per reader)	PATCH /{id}/read?reader=, POST /{id}/ack, a reply to a broadcast, and GET /messages/?unread_only=true (stamps delivered_at)	receipt upsert keyed (message_id, agent). The fields are first-write-wins (COALESCE)
archived	PATCH /{id}/archive, and supersession of a broadcast	message upsert where archived_at is set. Since T05, re-archiving keeps the first archived_at and produces no new revision
superseded	sending a broadcast with supersedes_id archives its predecessor in the same transaction	one revision that holds the new message and the predecessor's archive upsert
reply / thread link	thread_id points at the root; a reply inherits the root	carried in created. There is no separate link event
deleted	no API route deletes messages. Only a manual DB delete can	tombstone record. The delete cascades: receipts are dropped, and thread_id/supersedes_id on other rows become NULL. Those referencing rows arrive as upserts in the same revision

State that does not exist, so no event is emitted: unarchive (no route does it), message edits, per-reader read state for direct messages, who archived a message, and per-reader archive.

Expiry is not an event. expires_at is a stored value, and visibility (not_expired) is evaluated against the clock at read time. Hub-core must evaluate it the same way.

Message identity aliases are resolved by State Hub reads (MessageIdentityAlias), but no State Hub code writes them. Unverified: whether that table is populated in the State Hub DB. Aliases are not in v1.

Gaps against hub-core's pilot

• Nothing the pilot carries is missing from State Hub. Its MessageRead (id, from_agent, to_agent, subject, body, thread_id, read_at, archived_at, created_at) is a strict subset.
• The reverse gap is hub-core's to close. Hub-core's model lacks kind, expires_at, supersedes_id and receipts. Without receipts, its unread_only filter on read_at gives wrong answers for broadcasts: since STATE-WP-0093, broadcast read state lives in message_receipts.
• Delivery stamps stop for switched readers. A State Hub GET ?unread_only=true writes delivered_at. A reader moved to hub-core would stop producing those stamps unless hub-core writes them back, and that would make hub-core a second writer. See founder decision D3.
• Recipient slugs are stored as written. to_agent may hold a historical repository slug, and State Hub canonicalizes it when reading (_reader_values). The feed emits stored values. Resolving aliases is HUB-WP-0011-T02 scope.

4. Deletion versus retention

Settled by the STATE-WP-0079 T08/T09 record

• State Hub is the only message writer throughout the read cutover. Sends, read marks, acks, archives and replies all stay on State Hub. A switched reader that changes state calls the State Hub routes.
• The hub-core copy is derived and read-only. It is never authoritative, and it never mutates source history.
• Rollback is State Hub /messages/.
• The snapshot pilot must not be presented as a fresh inbox.
• Writer cutover and the B2/B3 route migration are out of scope here.

Proposed by this contract (to be accepted with the contract)

• Archive is not deletion. Archived rows stay in the feed and in the snapshot. Hub-core keeps them so that thread closure holds, as the pilot already requires.
• A State Hub delete always produces a tombstone (the trigger in §2), even if the delete is a manual DB operation.
• On a tombstone, hub-core deletes its derived row and keeps only (message_id, revision, deleted_at). It must not keep the body.
• Hub-core never emits its own tombstones.

Founder decisions (resolved 2026-10-13)

The T08/T09 record did not settle these. The founder accepted all four recommendations on 2026-10-13: 90-day tombstones (D1), no purge (D2), delivery-stamp loss accepted (D3), migration approved (D4).

• D1 — Tombstone retention horizon. Recommend: keep tombstones for 90 days, and at least twice the maximum consumer lag. A cursor older than the horizon gets 410, and the consumer re-snapshots.
• D2 — Purging during the retirement window. Should State Hub purge any messages before it retires, such as expired news (the default TTL is 30 days) or old archived rows? Recommend: no purge. Use archive only until State Hub retires. Retention is then decided once, by the receiving owner.
• D3 — Delivery stamps. Delivery stamps (delivered_at) are lost for a reader that moves to hub-core. Recommend: accept the loss, and pick the first switched reader (STATE-WP-0098-T04) from readers that do not rely on broadcast delivery tracking. Do not add a write-back path.
• D4 — Approve the migration. Approve the §2 migration (revision columns, trigger, tombstone table) as a retirement-enabling change to a frozen service. Recommend: approve. It is bounded and additive, and it is the only way to give a true cursor.

5. Recovery, ordering and freshness

Feed response.

contract, source_instance, source_time (DB now() at serve time),
high_water (cursor of the max committed revision), next_cursor, has_more,
compacted_before (oldest cursor still servable), changes[]

Compaction is implicit. Rows hold only their latest revision, so intermediate states are never replayable. A consumer that resumes from cursor c gets the latest post-image of every entity changed after c. Converging on the final state is guaranteed; seeing every step is not. Tombstones are the only history kept, and they are bounded by D1.

Recovery.

Situation	Behaviour
after is older than compacted_before	410 Gone with {"resnapshot_required": true}
Cursor from another source_instance or another contract version	409. The consumer re-snapshots
Consumer restarts	It resumes from its persisted cursor. Replaying pages it already applied is a no-op (§6)
Re-snapshot	Applies the snapshot as a full image under the §6 rules. Rows absent from the snapshot and not tombstoned are a consistency error: refuse and alert. They are not silently deleted

Out-of-order refusal (consumer side).

• Reject a page whose changes[0].revision is at or below the stored cursor, unless it is an exact replay.
• Reject a page whose high_water is lower than one already seen from the same source_instance. That indicates a source restore or rollback. The consumer stops serving as fresh and re-snapshots.

Freshness.

• State Hub provides source_time, high_water and source_instance on every response, including empty ones. An empty page is a heartbeat: "nothing changed as of source_time".
• Lag = consumer clock − last successful source_time.
• Maximum lag, stale-source behaviour and the read-mode header are hub-core's and rapp-core-hub's to set. This contract only supplies the signals.

6. Idempotency without an unconditional overwrite

Hub-core's importer refuses any receiver that differs from the bundle. The contract replaces "refuse any difference" with "refuse any difference that a higher source revision does not justify". Hub-core stores source_revision on each derived row and applies these rules per entity key:

Incoming revision r vs stored s	Outcome
No stored row	Insert
r < s	Ignore (stale replay). Log it
r == s, identical normalized hash	No-op
r == s, different content	Refuse. Integrity error; stop and re-snapshot
r > s	Replace with the post-image, after the field checks below

Field checks (for r > s). These catch source bugs instead of copying them:

• Immutable fields must not change: id, from_agent, to_agent, subject, body, kind, created_at, expires_at.
• thread_id and supersedes_id may change only from a value to NULL, and only in a revision that tombstones the referenced message.
• read_at and the receipt fields may change only from null to a value.
• archived_at may change only from null to a value (first-write-wins since T05). It never returns to null.

Anything else is refused, and hub-core re-snapshots.

The whole page is applied in one destination transaction, in revision order, and the cursor is updated in the same transaction. Thread roots arrive in the same or an earlier revision, because a reply cannot exist before its root, so the pilot's thread-closure check still holds.

Small State Hub fix (done in T05): archive_message is first-write-wins for archived_at, so the re-archive rule above is "null to value only".

7. State Hub implementation sketch (implemented 2026-10-14, see §8)

File	Change	Size
migrations/versions/<new>_inbox_feed_revision.py	§2 sequence, columns, backfill, triggers, tombstone table	~80 lines
api/models/agent_message.py	revision on both models; MessageTombstone model	~20
api/schemas/inbox_feed.py	snapshot/page/change envelopes; cursor encode and decode	~70
api/routers/message_feed.py	the two GET routes; repeatable-read transaction; page extension to a revision boundary; 409/410 handling	~130
api/main.py	include the router before messages.router	~2
api/routers/messages.py	first-write-wins archived_at	~3
tests/test_message_feed.py	snapshot/feed parity, per-transaction revisions, no bump on no-op upsert, tombstone on delete, 410 past horizon, page boundary not split, replay no-op	~250

The total is about 550 lines, roughly half of them tests. Expect one or two sessions, plus a deploy through the normal State Hub migration path.

The trigger needs PostgreSQL. Unverified: whether the test suite runs on SQLite. If it does, the trigger tests need the Postgres test fixture.

The hub-core side (source_revision column, page applier, cursor store, staleness) is not sketched here. It belongs to HUB-WP-0011-T01.

8. Implementation notes (2026-10-14)

Implemented as STATE-WP-0098-T05. Files: migration migrations/versions/c4d5e6f7a8b9_inbox_feed_revision.py, api/models/agent_message.py, api/schemas/inbox_feed.py, api/routers/message_feed.py, api/main.py, api/routers/messages.py, tests/test_message_feed.py. Where this section and §1-§7 differ, this section describes what the code does.

• Revision cache. inbox_feed_revision() caches "<txid>:<revision>" in the transaction-local GUC statehub.inbox_rev and reuses it only when the txid matches. A savepoint that rolls back after drawing a revision can make a transaction use two revisions; both stay ordered under the advisory lock (key 5348490098).
• Keys and state. key is the message UUID for message and tombstone, and "<message_id>:<agent>" for receipt. state is the full row and includes revision. Snapshot rows have the same shape as state. Within a revision the order is messages, receipts, tombstones.
• Foreign cursor (409). The cursor stays v1:<revision> and cannot carry the instance. The server answers 409 (resnapshot_required: true) when the cursor has another version prefix (for example v2:), when the optional query parameter source_instance differs from this instance's instance_label, or when the cursor is ahead of high_water (another instance or a restored source). A malformed cursor gets 400.
• Horizon (410). compacted_before is v1:<R>, where R is the highest revision of any tombstone older than 90 days (v1:0 if there is none). Those tombstones are no longer served, and after < R gets 410 with resnapshot_required: true. The horizon is enforced logically. Nothing purges tombstones physically (D2). The snapshot also carries compacted_before (additive).
• Paging. limit defaults to 500, maximum 5000. A page may exceed it to finish a revision. An empty page returns next_cursor equal to after.
• content_hash is sha256 over canonical JSON (sort_keys, compact separators) of {messages, receipts, tombstones}. Rows are sorted by key and timestamps are ISO strings. It is deterministic but not byte-identical to the pilot's hash.
• Delete reason. The tombstone reason comes from SET LOCAL statehub.delete_reason = '...' if the deleting session sets it, and is NULL otherwise.
• Not covered. A direct DELETE on message_receipts writes no tombstone, and no API route does this. Receipts disappear only through the message delete cascade, which the message tombstone implies.
• Tests and create_all. The test suite builds its schema with Base.metadata.create_all. The models attach the same trigger DDL via after_create. The migration keeps its own frozen copy, and a test asserts the two are identical. It also runs the migration's downgrade and upgrade on the test database in a rolled-back transaction.