state-hub/docs/inbox-change-feed-contract.md

# State Hub inbox change-feed contract

Status: **implemented (not deployed)** (2026-10-14, `STATE-WP-0098-T05`;
migration `c4d5e6f7a8b9`). Accepted by the founder 2026-10-13: D1-D4 as
recommended. Deliverable of `STATE-WP-0098-T03`. Deploy is a separate confirmed
step. Implementation details and deviations are in §8.
It is the State Hub input for `HUB-WP-0011-T01` (the first row of its
"Dependency review — 2026-10-04" table). Date: 2026-10-13.

Scope: how hub-core keeps a *derived*, read-only copy of the State Hub inbox
fresh. Hub-core would replace its one-shot snapshot (`import_snapshot` in
`hub_core/runtime/inbox_projection.py`) with this feed. This document does not
switch any reader or writer. Reader switches remain under `STATE-WP-0079-T04` and
`HUB-WP-0011-T03`.

Sources read: `api/models/agent_message.py`, `api/routers/messages.py`,
`api/services/message_receipts.py`, migrations `f3a4b5c6d7e8` and `d7e8f9a0b1c2`,
`docs/evidence/statehub-inbox-pilot-20260905.{md,json}`,
`STATE-WP-0079` T08/T09, `HUB-WP-0011`, and hub-core `inbox_projection.py` and
`schemas/agent_message.py`.

## 1. Identity, transport and auth

- **Contract id:** `statehub.inbox-feed`. **Version:** `v1`. Every response
  carries `"contract": "statehub.inbox-feed.v1"`. Additive fields do not bump
  the version. A changed meaning, a removed field or a changed cursor encoding
  bumps it to `v2`, and State Hub then serves both versions for the parity
  window.
- **Endpoints (read-only, GET only):**
  - `GET /messages/feed/snapshot` returns a consistent full image and the
    cursor that goes with it.
  - `GET /messages/feed?after=<cursor>&limit=<n>` returns changes after the
    cursor.
  - Both send `Cache-Control: no-store`. Any other method returns 405.
- **Direction: pull.** Hub-core's refresh runner calls State Hub over the
  in-cluster ClusterIP. Today the State Hub API has no application-level auth;
  access is bounded by the network (ClusterIP, the workstation SSH forward). The
  feed adds **no new credentials**. It inherits that boundary.
- **Admission (unverified):** the pilot evidence proves the *opposite*
  direction. NetworkPolicy admits State Hub API pods to hub-core, and
  `transport_from_state_hub_without_token` returned 401. It does not show
  whether hub-core pods can reach the State Hub API. If they cannot, the
  rapp-core-hub packaging owner adds a scoped ingress/egress rule for the
  refresh runner only. That is a packaging change, not a credential, and it
  belongs to the second T01 row of HUB-WP-0011.
- **Message bodies:** the feed includes `body`, because the inbox cannot be
  served without it. Evidence and receipts must keep excluding bodies, as the
  pilot did.

## 2. Consistent snapshot and monotonic cursor

### What the schema has today

`agent_messages` has the columns `id`, `from_agent`, `to_agent`, `subject`,
`body`, `thread_id`, `read_at`, `archived_at`, `kind`, `expires_at`,
`supersedes_id` and `created_at`. `message_receipts` has `(message_id, agent)`,
`delivered_at`, `read_at` and `acknowledged_at`.

Neither table has `updated_at`, a sequence, a revision column or a tombstone
table. No existing column can serve as a cursor:

- **Timestamps are application-clock values.** They come from `datetime.now()`
  and `utcnow()`, not from commit time. Concurrent transactions can commit them
  out of order, and they collide on ties.
- **Some changes leave no timestamp.** A receipt upsert can change one field
  among several. An archive call on a row that is already archived *overwrites*
  `archived_at` (see `archive_message`). A delete leaves nothing behind.

### Required migration (described here, not written)

One Alembic revision, roughly 80 lines:

1. **Revision sequence.** Create the sequence `inbox_revision_seq` (bigint).
2. **Revision columns.** Add `revision bigint NOT NULL` to `agent_messages` and
   to `message_receipts`. Backfill both with one value from the sequence, so
   that one snapshot equals one revision. Index `revision` on both tables.
3. **Tombstone table.** Add `message_tombstones` with the columns
   `message_id uuid PK`, `revision bigint NOT NULL`,
   `deleted_at timestamptz NOT NULL` and `reason text`.
4. **Write trigger.** A `BEFORE INSERT OR UPDATE` trigger on both tables. It
   - skips the update when `NEW IS NOT DISTINCT FROM OLD`, so that no-op
     COALESCE upserts and repeated reads do not bump the revision;
   - takes `pg_advisory_xact_lock(<inbox-feed constant>)`;
   - assigns **one revision per transaction**. On the first row it calls
     `nextval`, caches the value with `set_config('statehub.inbox_rev', …, true)`,
     and reuses it for every later row in the same transaction.
5. **Delete trigger.** A `BEFORE DELETE` trigger on `agent_messages`. It writes
   a tombstone row under the same transaction revision.

**Why the advisory lock is needed.** A sequence value is drawn at write time,
not at commit time. Without the lock, transaction A can draw revision 41,
transaction B can draw 42 and commit first, and a reader at cursor 42 would
never see 41. With the lock, inbox writes are serialized, so commit order
equals revision order. Any visible maximum revision is then a safe watermark.

Inbox write volume is low: sends, read marks, receipts, archives. The lock cost
is negligible. *Unverified:* there is no load measurement. If the lock is
rejected, the alternative is an `xid8` column plus a
`pg_snapshot_xmin(pg_current_snapshot())` watermark. That avoids the lock but
is more complex to serve.

**The cursor** is an opaque string that encodes `v1:<revision>`. Consumers
must not do arithmetic on it. They may compare the decoded revisions only for
the out-of-order checks in §5.

### Snapshot

`GET /messages/feed/snapshot` runs in one `REPEATABLE READ READ ONLY`
transaction and returns:

```
contract, source_instance (from /state/health instance_label),
source_time (DB now() of the snapshot txn), cursor (max visible revision),
messages[] (all rows, archived included), receipts[], tombstones[] (within horizon),
count, content_hash (sha256 over the normalized rows, as in the pilot)
```

It is returned as a single response. The pilot captured 20 rows. *Unverified:*
the current production row count. If the response grows past a few MB, the
snapshot is paged inside one exported snapshot (`pg_export_snapshot`). That is
a v1 addition, not a new version.

### No-migration interim (not recommended)

Hub-core could poll the snapshot endpoint and diff it, with
`content_hash` + `source_time` as the revision. It gives freshness but no
ordering: hashes cannot be compared for order, and `source_time` is only as
monotonic as the DB clock. It also cannot distinguish a delete from a
filtering bug. Acceptable only as a stop-gap behind the existing
`snapshot-pilot` label.

## 3. Change records and event types

The wire carries **state, not deltas**. Each change is the full post-image of
one entity at its revision:

```
{revision, entity: "message"|"receipt"|"tombstone", key, op: "upsert"|"delete", state}
```

Hub-core works out the event type by comparing the post-image with its
current row. This keeps replay idempotent (§6). A page never splits a
revision: the server extends the page past `limit` until the last revision is
complete.

Only the transitions below can occur in the current State Hub model.

| Event | Source mechanics (today) | Appears on the feed as |
| --- | --- | --- |
| `created` | `POST /messages/`, `POST /{id}/reply` | `message` upsert with no prior row; `thread_id`, `supersedes_id`, `kind` and `expires_at` set at creation |
| `read` (direct) | `PATCH /{id}/read` without a reader; a reply marks the original as read; archiving a direct message stamps `read_at` if it is null | `message` upsert where `read_at` goes from null to a value. There is one global `read_at` and no reader identity |
| `receipt` (broadcast, per reader) | `PATCH /{id}/read?reader=`, `POST /{id}/ack`, a reply to a broadcast, and **`GET /messages/?unread_only=true` (stamps `delivered_at`)** | `receipt` upsert keyed `(message_id, agent)`. The fields are first-write-wins (`COALESCE`) |
| `archived` | `PATCH /{id}/archive`, and supersession of a broadcast | `message` upsert where `archived_at` is set. Since T05, re-archiving keeps the first `archived_at` and produces no new revision |
| `superseded` | sending a broadcast with `supersedes_id` archives its predecessor in the same transaction | one revision that holds the new `message` and the predecessor's archive upsert |
| reply / thread link | `thread_id` points at the root; a reply inherits the root | carried in `created`. There is no separate link event |
| `deleted` | **no API route deletes messages.** Only a manual DB delete can | `tombstone` record. The delete cascades: receipts are dropped, and `thread_id`/`supersedes_id` on other rows become NULL. Those referencing rows arrive as upserts in the same revision |

**State that does not exist, so no event is emitted:** unarchive (no route
does it), message edits, per-reader read state for *direct* messages, who
archived a message, and per-reader archive.

**Expiry is not an event.** `expires_at` is a stored value, and visibility
(`not_expired`) is evaluated against the clock at read time. Hub-core must
evaluate it the same way.

**Message identity aliases** are resolved by State Hub reads
(`MessageIdentityAlias`), but no State Hub code writes them. *Unverified:*
whether that table is populated in the State Hub DB. Aliases are not in v1.

### Gaps against hub-core's pilot

- **Nothing the pilot carries is missing from State Hub.** Its `MessageRead`
  (`id`, `from_agent`, `to_agent`, `subject`, `body`, `thread_id`, `read_at`,
  `archived_at`, `created_at`) is a strict subset.
- **The reverse gap is hub-core's to close.** Hub-core's model lacks `kind`,
  `expires_at`, `supersedes_id` and receipts. Without receipts, its
  `unread_only` filter on `read_at` gives wrong answers for broadcasts: since
  STATE-WP-0093, broadcast read state lives in `message_receipts`.
- **Delivery stamps stop for switched readers.** A State Hub
  `GET ?unread_only=true` writes `delivered_at`. A reader moved to hub-core
  would stop producing those stamps unless hub-core writes them back, and that
  would make hub-core a second writer. See founder decision D3.
- **Recipient slugs are stored as written.** `to_agent` may hold a historical
  repository slug, and State Hub canonicalizes it when reading
  (`_reader_values`). The feed emits stored values. Resolving aliases is
  HUB-WP-0011-T02 scope.

## 4. Deletion versus retention

### Settled by the STATE-WP-0079 T08/T09 record

- **State Hub is the only message writer** throughout the read cutover. Sends,
  read marks, acks, archives and replies all stay on State Hub. A switched
  reader that changes state calls the State Hub routes.
- **The hub-core copy is derived and read-only.** It is never authoritative,
  and it never mutates source history.
- **Rollback is State Hub `/messages/`.**
- **The snapshot pilot must not be presented as a fresh inbox.**
- **Writer cutover and the B2/B3 route migration are out of scope here.**

### Proposed by this contract (to be accepted with the contract)

- **Archive is not deletion.** Archived rows stay in the feed and in the
  snapshot. Hub-core keeps them so that thread closure holds, as the pilot
  already requires.
- **A State Hub delete always produces a tombstone** (the trigger in §2), even
  if the delete is a manual DB operation.
- **On a tombstone, hub-core deletes its derived row** and keeps only
  `(message_id, revision, deleted_at)`. It must not keep the body.
- **Hub-core never emits its own tombstones.**

### Founder decisions (resolved 2026-10-13)

The T08/T09 record did not settle these. The founder accepted all four
recommendations on 2026-10-13: 90-day tombstones (D1), no purge (D2),
delivery-stamp loss accepted (D3), migration approved (D4).

- **D1 — Tombstone retention horizon.** *Recommend:* keep tombstones for 90
  days, and at least twice the maximum consumer lag. A cursor older than the
  horizon gets 410, and the consumer re-snapshots.
- **D2 — Purging during the retirement window.** Should State Hub purge any
  messages before it retires, such as expired `news` (the default TTL is 30
  days) or old archived rows? *Recommend:* no purge. Use archive only until
  State Hub retires. Retention is then decided once, by the receiving owner.
- **D3 — Delivery stamps.** Delivery stamps (`delivered_at`) are lost for a
  reader that moves to hub-core. *Recommend:* accept the loss, and pick the
  first switched reader (`STATE-WP-0098-T04`) from readers that do not rely on
  broadcast delivery tracking. Do not add a write-back path.
- **D4 — Approve the migration.** Approve the §2 migration (revision columns,
  trigger, tombstone table) as a retirement-enabling change to a frozen
  service. *Recommend:* approve. It is bounded and additive, and it is the
  only way to give a true cursor.

## 5. Recovery, ordering and freshness

**Feed response.**

```
contract, source_instance, source_time (DB now() at serve time),
high_water (cursor of the max committed revision), next_cursor, has_more,
compacted_before (oldest cursor still servable), changes[]
```

**Compaction is implicit.** Rows hold only their latest revision, so
intermediate states are never replayable. A consumer that resumes from cursor
`c` gets the latest post-image of every entity changed after `c`. Converging
on the final state is guaranteed; seeing every step is not. Tombstones are the
only history kept, and they are bounded by D1.

**Recovery.**

| Situation | Behaviour |
| --- | --- |
| `after` is older than `compacted_before` | **410 Gone** with `{"resnapshot_required": true}` |
| Cursor from another `source_instance` or another contract version | **409**. The consumer re-snapshots |
| Consumer restarts | It resumes from its persisted cursor. Replaying pages it already applied is a no-op (§6) |
| Re-snapshot | Applies the snapshot as a full image under the §6 rules. Rows absent from the snapshot and not tombstoned are a consistency error: refuse and alert. They are not silently deleted |

**Out-of-order refusal (consumer side).**

- Reject a page whose `changes[0].revision` is at or below the stored cursor,
  unless it is an exact replay.
- Reject a page whose `high_water` is lower than one already seen from the
  same `source_instance`. That indicates a source restore or rollback. The
  consumer stops serving as fresh and re-snapshots.

**Freshness.**

- State Hub provides `source_time`, `high_water` and `source_instance` on
  every response, including empty ones. An empty page is a heartbeat: "nothing
  changed as of `source_time`".
- Lag = consumer clock − last successful `source_time`.
- Maximum lag, stale-source behaviour and the read-mode header are hub-core's
  and rapp-core-hub's to set. This contract only supplies the signals.

## 6. Idempotency without an unconditional overwrite

Hub-core's importer refuses any receiver that differs from the bundle. The
contract replaces "refuse any difference" with "refuse any difference that a
higher source revision does not justify". Hub-core stores `source_revision` on
each derived row and applies these rules per entity key:

| Incoming revision `r` vs stored `s` | Outcome |
| --- | --- |
| No stored row | Insert |
| `r < s` | Ignore (stale replay). Log it |
| `r == s`, identical normalized hash | No-op |
| `r == s`, different content | **Refuse.** Integrity error; stop and re-snapshot |
| `r > s` | Replace with the post-image, after the field checks below |

**Field checks (for `r > s`).** These catch source bugs instead of copying
them:

- Immutable fields must not change: `id`, `from_agent`, `to_agent`, `subject`,
  `body`, `kind`, `created_at`, `expires_at`.
- `thread_id` and `supersedes_id` may change only from a value to NULL, and
  only in a revision that tombstones the referenced message.
- `read_at` and the receipt fields may change only from null to a value.
- `archived_at` may change only from null to a value (first-write-wins since
  T05). It never returns to null.

Anything else is refused, and hub-core re-snapshots.

The whole page is applied in one destination transaction, in revision order,
and the cursor is updated in the same transaction. Thread roots arrive in the
same or an earlier revision, because a reply cannot exist before its root, so
the pilot's thread-closure check still holds.

Small State Hub fix (done in T05): `archive_message` is first-write-wins for
`archived_at`, so the re-archive rule above is "null to value only".

## 7. State Hub implementation sketch (implemented 2026-10-14, see §8)

| File | Change | Size |
| --- | --- | --- |
| `migrations/versions/<new>_inbox_feed_revision.py` | §2 sequence, columns, backfill, triggers, tombstone table | ~80 lines |
| `api/models/agent_message.py` | `revision` on both models; `MessageTombstone` model | ~20 |
| `api/schemas/inbox_feed.py` | snapshot/page/change envelopes; cursor encode and decode | ~70 |
| `api/routers/message_feed.py` | the two GET routes; repeatable-read transaction; page extension to a revision boundary; 409/410 handling | ~130 |
| `api/main.py` | include the router before `messages.router` | ~2 |
| `api/routers/messages.py` | first-write-wins `archived_at` | ~3 |
| `tests/test_message_feed.py` | snapshot/feed parity, per-transaction revisions, no bump on no-op upsert, tombstone on delete, 410 past horizon, page boundary not split, replay no-op | ~250 |

The total is about 550 lines, roughly half of them tests. Expect one or two
sessions, plus a deploy through the normal State Hub migration path.

The trigger needs PostgreSQL. *Unverified:* whether the test suite runs on
SQLite. If it does, the trigger tests need the Postgres test fixture.

The hub-core side (`source_revision` column, page applier, cursor store,
staleness) is not sketched here. It belongs to HUB-WP-0011-T01.

## 8. Implementation notes (2026-10-14)

Implemented as `STATE-WP-0098-T05`. Files: migration
`migrations/versions/c4d5e6f7a8b9_inbox_feed_revision.py`,
`api/models/agent_message.py`, `api/schemas/inbox_feed.py`,
`api/routers/message_feed.py`, `api/main.py`, `api/routers/messages.py`,
`tests/test_message_feed.py`. Where this section and §1-§7 differ, this
section describes what the code does.

- **Revision cache.** `inbox_feed_revision()` caches `"<txid>:<revision>"` in
  the transaction-local GUC `statehub.inbox_rev` and reuses it only when the
  txid matches. A savepoint that rolls back after drawing a revision can make a
  transaction use two revisions; both stay ordered under the advisory lock
  (key `5348490098`).
- **Keys and state.** `key` is the message UUID for `message` and `tombstone`,
  and `"<message_id>:<agent>"` for `receipt`. `state` is the full row and
  includes `revision`. Snapshot rows have the same shape as `state`. Within a
  revision the order is messages, receipts, tombstones.
- **Foreign cursor (409).** The cursor stays `v1:<revision>` and cannot carry
  the instance. The server answers 409 (`resnapshot_required: true`) when the
  cursor has another version prefix (for example `v2:`), when the optional
  query parameter `source_instance` differs from this instance's
  `instance_label`, or when the cursor is ahead of `high_water` (another
  instance or a restored source). A malformed cursor gets 400.
- **Horizon (410).** `compacted_before` is `v1:<R>`, where `R` is the highest
  revision of any tombstone older than 90 days (`v1:0` if there is none).
  Those tombstones are no longer served, and `after < R` gets 410 with
  `resnapshot_required: true`. The horizon is enforced logically. Nothing
  purges tombstones physically (D2). The snapshot also carries
  `compacted_before` (additive).
- **Paging.** `limit` defaults to 500, maximum 5000. A page may exceed it to
  finish a revision. An empty page returns `next_cursor` equal to `after`.
- **content_hash** is sha256 over canonical JSON (`sort_keys`, compact
  separators) of `{messages, receipts, tombstones}`. Rows are sorted by key and
  timestamps are ISO strings. It is deterministic but not byte-identical to the
  pilot's hash.
- **Delete reason.** The tombstone `reason` comes from
  `SET LOCAL statehub.delete_reason = '...'` if the deleting session sets it,
  and is NULL otherwise.
- **Not covered.** A direct `DELETE` on `message_receipts` writes no tombstone,
  and no API route does this. Receipts disappear only through the message
  delete cascade, which the message tombstone implies.
- **Tests and create_all.** The test suite builds its schema with
  `Base.metadata.create_all`. The models attach the same trigger DDL via
  `after_create`. The migration keeps its own frozen copy, and a test asserts
  the two are identical. It also runs the migration's downgrade and upgrade on
  the test database in a rolled-back transaction.