94 lines
3.5 KiB
Markdown
94 lines
3.5 KiB
Markdown
|
# Data contracts — schema `inbound-library/v1`
|
||
|
|
|
||
|
|
Consumers may rely on every field listed here. Any change that breaks one
|
||
|
|
of them bumps the schema version.
|
||
|
|
|
||
|
|
## Layout
|
||
|
|
|
||
|
|
| Path | Mutability | Versioned in git |
|
||
|
|
| --- | --- | --- |
|
||
|
|
| `registry/topics.json` | edited by commit | yes |
|
||
|
|
| `registry/sources/<id>.json` | edited by commit | yes |
|
||
|
|
| `registry/subscriptions/<repo>.json` | edited by commit | yes |
|
||
|
|
| `config/library.json` | edited by commit, no secrets | yes |
|
||
|
|
| `data/items/<sha256>` | immutable original bytes | no |
|
||
|
|
| `data/deliveries/<sha256>.json` | immutable | no |
|
||
|
|
| `data/notices/<sha256>.json` | immutable | no |
|
||
|
|
| `data/derived/<item>.json` | rebuildable, replaceable | no |
|
||
|
|
| `data/digests/<repo>/<cadence>/<period>.md` | rebuildable, replaceable | no |
|
||
|
|
| `data/state/*.json` | cursor and hub-sent state | no |
|
||
|
|
|
||
|
|
`data/` is gitignored because mail can be private. A record's id is the SHA-256
|
||
|
|
of its canonical JSON (sorted keys, no whitespace). An item's id is the SHA-256
|
||
|
|
of its bytes. `check` rehashes both.
|
||
|
|
|
||
|
|
## Item
|
||
|
|
|
||
|
|
The original bytes, exactly as delivered. For mail, the item is the full
|
||
|
|
RFC 822 message. Each attachment is also an item of its own.
|
||
|
|
|
||
|
|
## Delivery (provenance)
|
||
|
|
|
||
|
|
One record per delivery of an item.
|
||
|
|
|
||
|
|
| Field | Meaning |
|
||
|
|
| --- | --- |
|
||
|
|
| `schema` | `inbound-library/v1` |
|
||
|
|
| `item` | item id |
|
||
|
|
| `channel` | `imap` or `attachment` |
|
||
|
|
| `status` | `routed` (a source matched) or `quarantined` (no source) |
|
||
|
|
| `source` | source id, or `null` |
|
||
|
|
| `received_at` | IMAP INTERNALDATE (UTC ISO 8601) |
|
||
|
|
| `mailbox` | `<user>/<folder>`; for `imap` only |
|
||
|
|
| `message_id`, `sender`, `subject`, `list_id` | for `imap` only; `sender` is lowercased |
|
||
|
|
| `authentication_results` | raw headers, recorded as provenance, not as trust |
|
||
|
|
| `parent_item`, `filename`, `content_type` | for `attachment` only |
|
||
|
|
|
||
|
|
IMAP UIDs are kept out of the delivery record on purpose. After a UIDVALIDITY
|
||
|
|
change, the rescan reproduces identical records. When a source is registered
|
||
|
|
later, a quarantined mail gets an additional `routed` delivery. The quarantined
|
||
|
|
record stays as history.
|
||
|
|
|
||
|
|
## Derived
|
||
|
|
|
||
|
|
`subject`, `sender`, `date`, `text` (text/plain, otherwise stripped HTML) and
|
||
|
|
`attachments` (`item`, `filename`, `content_type`). It can be rebuilt from the
|
||
|
|
item at any time.
|
||
|
|
|
||
|
|
## Topic
|
||
|
|
|
||
|
|
In `topics.json`: `{"version": n, "topics": {"<name>": "<description>"}}`.
|
||
|
|
Names use dots, from domain to specific, for example
|
||
|
|
`intelligence.daily-brief`. Add new topics here first; sources and
|
||
|
|
subscriptions may only use registered topics.
|
||
|
|
|
||
|
|
## Source
|
||
|
|
|
||
|
|
| Field | Meaning |
|
||
|
|
| --- | --- |
|
||
|
|
| `id` | the file name without `.json` |
|
||
|
|
| `owner_repo` | the repository that registered it |
|
||
|
|
| `channel` | `email` or `url` |
|
||
|
|
| `match` | for `email`: `from` (required), `subject` (substring), `list_id` |
|
||
|
|
| `url` | for `url` |
|
||
|
|
| `topics` | list of registered topics |
|
||
|
|
| `cadence` | `daily`, `weekly` or `irregular`; daily and weekly email sources are watched for silence (36 h or 9 days) |
|
||
|
|
| `visibility` | `shared` (any matching repo) or `restricted` (owner only) |
|
||
|
|
|
||
|
|
## Subscription
|
||
|
|
|
||
|
|
| Field | Meaning |
|
||
|
|
| --- | --- |
|
||
|
|
| `repo` | the file name without `.json` |
|
||
|
|
| `agent` | State Hub agent name that receives notices |
|
||
|
|
| `topics`, `sources` | matched by either |
|
||
|
|
| `channels` | a subset of `hub`, `pull`, `digest` |
|
||
|
|
| `cadence` | `daily` or `weekly`; required for `digest` |
|
||
|
|
|
||
|
|
## Notice
|
||
|
|
|
||
|
|
`{schema, repo, delivery, item, source, received_at}`. It is deterministic, so
|
||
|
|
each (repo, delivery) pair gets exactly one notice. Hub delivery sends one
|
||
|
|
message per subscriber per run. A failed send stays pending and is retried on
|
||
|
|
the next run.
|