diff --git a/.custodian-brief.md b/.custodian-brief.md new file mode 100644 index 0000000..a0dda78 --- /dev/null +++ b/.custodian-brief.md @@ -0,0 +1,27 @@ + +# Custodian Brief - inbound-library + +**Project:** inbound-library +**Domain:** infotech +**State Hub:** http://127.0.0.1:8000 +**Topic ID:** `cee7bedf-2b48-46ef-8601-006474f2ad7a` + +## Open Workplans + +### Bootstrap State Hub integration + +Workplan file: `workplans/ILIB-WP-0001-statehub-bootstrap.md` + +Open tasks: +- T01 - Review generated integration files +- T02 - Verify local developer workflow +- T03 - Seed first real workplan + +## Session Start + +1. Read `INTENT.md`, `SCOPE.md`, and `AGENTS.md`. +2. Check inbox: `GET /messages/?to_agent=inbound-library&unread_only=true`. +3. Scan `workplans/`. +4. Update task statuses in workplan files as work progresses. + +Last generated: 2026-10-05 diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..e4e0199 --- /dev/null +++ b/.gitignore @@ -0,0 +1,5 @@ +# state-hub: track .claude/rules +# Claude Code local state (track shared rules; ignore machine-specific files) +.claude/* +!.claude/rules/ +!.claude/rules/*.md diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..f4df499 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,205 @@ +# inbound-library — Agent Instructions + +## Repo Identity + +**Purpose:** Keeps material delivered to the ecosystem and routes it by topic to the repositories that asked to see it. + +**Domain:** infotech +**Repo slug:** inbound-library +**Topic ID:** `cee7bedf-2b48-46ef-8601-006474f2ad7a` +**Workplan prefix:** `ILIB-WP-` + +--- + +## State Hub Integration + +The Custodian State Hub tracks work across all domains. Codex uses HTTP REST and +the `statehub` CLI by default. MCP is opt-in because the current Codex MCP bridge +adds severe call latency; the full administrative MCP surface remains available +to clients that need it. + +| Context | URL | +|---------|-----| +| Local workstation | `http://127.0.0.1:8000` | +| Remote via tunnel | `http://127.0.0.1:18000` | +| Optional local edge relay | http://127.0.0.1:18080 | + +When an operator has enabled the edge relay, set API_BASE to the relay URL. +Queueable writes return an explicit queued receipt if the central hub is +unreachable. Treat that as pending local evidence, then ask the operator to run +statehub outbox status/replay after connectivity returns. + +Codex workspace-write sandboxes need network access enabled to reach the host's +loopback listener. Bootstrap this once with `make -C ~/state-hub configure-codex` +and restart Codex. The canonical REST health endpoint is `/state/health`, not +`/health`. If a sandboxed loopback probe fails, retry it with escalated execution +before declaring State Hub unavailable; a managed Codex permission profile may +still enforce isolated networking. Experimental MCP can be enabled explicitly +with `make -C ~/state-hub configure-codex WITH_MCP=1`. + +### Orient at session start + +```bash +# Offline brief — works without hub connection +cat .custodian-brief.md + +# Active workplans for this domain +curl -s "http://127.0.0.1:8000/workplans/?topic_id=cee7bedf-2b48-46ef-8601-006474f2ad7a&status=active" \ + | python3 -m json.tool + +# Check inbox +curl -s "http://127.0.0.1:8000/messages/?to_agent=inbound-library&unread_only=true" \ + | python3 -m json.tool +``` + +Mark a message read: +```bash +curl -s -X PATCH "http://127.0.0.1:8000/messages//read" \ + -H "Content-Type: application/json" -d '{}' +``` + +### Log progress (required at session close) + +```bash +curl -s -X POST http://127.0.0.1:8000/progress/ \ + -H "Content-Type: application/json" \ + -d '{ + "summary": "what was done", + "event_type": "note", + "author": "codex", + "workplan_id": "", + "task_id": "" + }' +``` + +Omit `workplan_id` / `task_id` when not applicable. + +### Update task status + +```bash +curl -s -X PATCH "http://127.0.0.1:8000/tasks/" \ + -H "Content-Type: application/json" \ + -d '{"status": "progress"}' +# values: wait | todo | progress | done | cancel +``` + +### Flag a task for human review + +```bash +curl -s -X PATCH "http://127.0.0.1:8000/tasks/" \ + -H "Content-Type: application/json" \ + -d '{"needs_human": true, "intervention_note": "reason"}' +``` + +--- + +## Session Protocol + +**Start:** +1. `cat .custodian-brief.md` — domain goal and open workplans (offline-safe) +2. Check inbox: `GET /messages/?to_agent=inbound-library&unread_only=true`; mark read +3. Scan workplans: `ls workplans/` — note `status: ready`, `active`, or `blocked` files and open tasks +4. Check human-needed tasks: `GET /tasks/?needs_human=true` + +**During work:** +- Update task statuses in workplan files as tasks progress +- Record significant decisions via `POST /decisions/` + +**Close:** +1. Update workplan file task statuses to reflect progress +2. If finishing a workplan: hand off **residuals** as live work records first + (intake with `origin: residual` + `origin_ref: `, or a next workplan / + decision / engagement). Do not park leftovers only in prose or `SCOPE.md`. + Canon: `the-custodian/canon/standards/work-record-types_v0.1.md` § Residuals. +3. Log: `POST /progress/` with a summary of what changed (name handoff ids) +4. After workplan file changes, run: + ```bash + uv run --project ~/repo-manager rmgr sync --path . --push + ``` + This assigns only missing deterministic identifiers, verifies the pushed + Forgejo commit and `primary/railliance01`, then requests one central + reconciliation. A queued receipt is pending evidence; rerun after + connectivity returns. Use `statehub fix-consistency` for a separate deep audit. + +--- + +{CREDENTIAL_ROUTING} + + + + +--- + +## Workplan Convention (ADR-001) + +Work items originate as files in this repo — not in the hub. The hub is a +read/cache/index layer that rebuilds from files. + +**File location:** `workplans/ILIB-WP-NNNN-.md` + +**Archived location:** finished workplans may move to +`workplans/archived/YYMMDD-ILIB-WP-NNNN-.md`. The `YYMMDD` prefix is +the completion/archive date; the frontmatter `id` does not change. + +**Ad Hoc Tasks:** small opportunistic fixes discovered during a session use +`workplans/ADHOC-YYYY-MM-DD.md`, workplan id +`ILIB-WP-ADHOC-YYYY-MM-DD`, and task ids +`ILIB-WP-ADHOC-YYYY-MM-DD-T01`, etc. `ILIB-WP` includes its final `-WP` +token. Unqualified historic `ADHOC-*` ids are grandfathered and must not be +copied into new records. Use this only for low-risk work completed directly; +create a normal workplan for anything needing analysis, design, approval, +dependencies, or multiple phases. + +**Frontmatter:** + +```yaml +--- +id: ILIB-WP-NNNN +type: workplan +title: "..." +domain: infotech +repo: inbound-library +status: proposed | ready | active | blocked | backlog | finished | archived +owner: codex +topic_slug: ... +created: "YYYY-MM-DD" +updated: "YYYY-MM-DD" +state_hub_workstream_id: "" # deterministic UUIDv5; managed by Repo Manager +--- +``` + +Use `proposed` for a new draft, `ready` after review against current repo +state, and `finished` after implementation. `stalled` and `needs_review` are +derived health labels, not frontmatter statuses. + +**Terminology:** workplan is the fleet term; `workstream` appears only in legacy +API/MCP/frontmatter bridges until `STATE-WP-0069` retires them — see +`the-custodian/canon/standards/workplan-terminology-fleet_v0.1.md`. + +**Task block format** (one per `##` section): + +``` +## Task Title + +` ` `task +id: ILIB-WP-NNNN-T01 +status: wait | todo | progress | done | cancel +priority: high | medium | low +state_hub_task_id: "" # deterministic UUIDv5; managed by Repo Manager +` ` ` + +Task description text. +``` + +Status progression: `todo` → `progress` → `done`; use `wait` for waiting/blocked work and `cancel` for stopped work. + +**Residuals when finishing:** actionable leftovers become live work records +before `status: finished` — usually an intake (`origin: residual`, +`origin_ref: ILIB-WP-NNNN`) or a spawned workplan. Residual is a *role*, +not a kind. Fleet list lives on State Hub, not in `SCOPE.md`. + +To create a new workplan: +1. Write the file following the format above +2. Run `uv run --project ~/repo-manager rmgr sync --path . --push`. +3. Run `statehub fix-consistency` only when a separate deep audit is needed. diff --git a/INTENT.md b/INTENT.md new file mode 100644 index 0000000..7c56c15 --- /dev/null +++ b/INTENT.md @@ -0,0 +1,153 @@ +# INTENT — inbound-library + +**Status:** Draft baseline 0.1 +**Date:** 2026-10-05 + +## 1. Intent + +inbound-library collects material that may matter to the ecosystem. Consuming +repositories can find it, review it, and decide what it means for them. + +It answers three questions: + +- **What has arrived?** Mail and registered sources, kept with their original + bytes and their provenance. +- **Who should look at it?** Material is matched to the topics that repositories + have registered. +- **How do they get it?** Notification, then access when the consumer chooses. + +The library keeps and routes material. It does not judge whether material is +true, relevant or good. That review belongs to the consuming repository. + +## 2. First use case + +The daily and weekly intelligence briefs that Grok and ChatGPT automations +deliver by email. They are consumed by intelligence-radar. + +intelligence-radar used to receive them only through a drop folder. Pulling them +from the signed-in websites was rejected: it would break the providers' terms of +service. Email is a delivery the providers offer, so collecting it is +legitimate. + +## 3. Vocabulary + +### Item + +One piece of retained material, for example an email, an attachment or a +captured document. It has: +- its original bytes, content-addressed by SHA-256; +- a provenance record: channel, sender, received-at, Message-ID or URL, and + source; +- derived text and metadata, which never replace the original bytes. + +### Source + +A registered origin of items, for example a sender address, a feed or a URL. A +source has an owner (the repository that registered it), topics, and an expected +cadence. A consuming repository may register sources from its own research. The +library then makes those sources available to every repository whose topics +match. + +### Topic + +A shared name for an area of interest, such as `intelligence.daily-brief`. +Topics form a small, versioned vocabulary. They are not free tags per item. + +### Subscription + +A repository's registered interest. It has: +- topics, and optionally sources; +- delivery channels; +- cadence; +- the agent name that receives notices. + +### Notice + +A pointer sent to a subscriber: "these items arrived for your topics". A notice +refers to items. It does not copy them. + +## 4. Principles + +**L1 — Original bytes are kept.** Items are append-only and content-addressed. +Derived text, parsing and classification can be redone. The original cannot. + +**L2 — Provenance is part of the item.** An item without a known channel and +sender or origin is quarantined. It is not routed. + +**L3 — Only material that was delivered or published.** Items come from mail +sent to the library's mailbox, or from sources published for access. No account +sessions are used, no scraping happens behind sign-ins, and no provider's terms +of service are worked around. + +**L4 — Read-only toward the mailbox.** The mailbox is read without changing it. +The scan cursor lives in the library, not in mail flags. + +**L5 — The library routes. It does not judge.** Every item enters as +`unreviewed`. Relevance and credibility are decided by the consumers, and their +review outcomes may be recorded back as annotations. + +**L6 — Notices are pointers and idempotent.** A subscriber is told about an item +once per subscription. Retries never create duplicate notices. + +**L7 — Registries are files.** Topics, sources and subscriptions live as +versioned files in this repository. Consumers change them with a commit. The +State Hub is a read model and receives notices only as messages. + +**L8 — Shared by default, restricted when declared.** Sources registered by one +repository are visible to every repository with matching topics. The exception +is a source the registering repository marks `restricted`. + +**L9 — Standard library first.** Python 3.11+ standard library. It runs on the +workstation under a systemd user timer and moves to the cluster once the +contracts are stable. + +## 5. Service contract + +### Intake + +- an IMAP mailbox, polled read-only; +- registered sources; +- items submitted by a consumer through the CLI. + +### Delivery to subscribers + +- **State Hub agent messages:** one notice per subscriber per run that has new + matches. +- **Pull CLI/API:** `new --topic … --since …`, `show ` and + `get ` (original bytes). +- **Periodic digest file:** a Markdown digest for each subscriber and cadence. + +### Operation + +- a health report: last scan, items per source, sources that missed their + expected cadence, and quarantine size; +- an integrity check over every retained item. + +## 6. Success criteria + +inbound-library succeeds when: + +- a brief emailed by Grok or ChatGPT reaches intelligence-radar within one + timer interval, without an operator; +- every item can be traced back to its original bytes and delivery; +- a source registered by one repository is found by another repository through + topics; +- a missed delivery (a source that stays silent past its cadence) is visible + rather than silent; +- adding a consumer is a commit, not a code change. + +## 7. Non-goals + +inbound-library is not: + +- a mail client, or a sender of mail other than notices; +- a scraper of signed-in websites; +- an editorial or fact-checking system; +- a news-content farm; +- a replacement for a consumer's own evidence store. Consumers retain what they + decide to use. + +## 8. One-line description + +> **inbound-library keeps material delivered to the ecosystem and routes it, by +> topic, to the repositories that asked to see it.** diff --git a/SCOPE.md b/SCOPE.md new file mode 100644 index 0000000..f708bf1 --- /dev/null +++ b/SCOPE.md @@ -0,0 +1,38 @@ +# SCOPE + +## One-liner + +Keeps material delivered to the ecosystem and routes it by topic to the repositories that asked to see it. + +## Core Idea + +See INTENT.md. Email and registered sources go into an append-only, content-addressed +item store. File-backed registries of topics, sources and subscriptions decide +who is told about each item. Notices reach consumers as State Hub agent messages, +through a pull CLI, and in periodic digest files. + +## In Scope + +- Read-only IMAP intake from one dedicated mailbox. +- Items with original bytes, provenance and derived text; integrity checking. +- Registries of topics, sources (including sources contributed by consumers) and subscriptions. +- Matching, idempotent notices, pull access, digest files. +- Scheduling, health reporting and quarantine on the workstation, later in the cluster. + +## Out of Scope + +- Sending mail other than notices. email-connect owns email delivery and its evidence. +- Scraping signed-in websites, or any intake that breaks a provider's terms of service. +- Judging whether material is relevant or credible. Consumers review it. +- Replacing a consumer's own evidence store. + +## Current State + +- Status: inception. INTENT drafted 2026-10-05, no code yet. +- First consumer: intelligence-radar (Grok and ChatGPT daily and weekly briefs). + +## Getting Oriented + +- Start with: INTENT.md +- Agent instructions: AGENTS.md +- Workplans: workplans/ diff --git a/workplans/ILIB-WP-0001-statehub-bootstrap.md b/workplans/ILIB-WP-0001-statehub-bootstrap.md new file mode 100644 index 0000000..c820f47 --- /dev/null +++ b/workplans/ILIB-WP-0001-statehub-bootstrap.md @@ -0,0 +1,62 @@ +--- +id: ILIB-WP-0001 +type: workplan +title: "Bootstrap State Hub integration" +domain: infotech +repo: inbound-library +status: finished +owner: codex +topic_slug: custodian +created: "2026-10-05" +updated: "2026-10-05" +--- + +# Bootstrap State Hub integration + +Keeps material delivered to the ecosystem and routes it by topic to the repositories that asked to see it. + +## Review Generated Integration Files + +```task +id: ILIB-WP-0001-T01 +status: done +priority: high +``` + +Review `INTENT.md`, `SCOPE.md`, `AGENTS.md`, and `.custodian-brief.md`. +Replace generated placeholders with repo-specific facts where needed. + +Done 2026-10-05: INTENT.md was written by hand and SCOPE.md was refined. + +## Verify Local Developer Workflow + +```task +id: ILIB-WP-0001-T02 +status: cancel +priority: high +``` + +Identify the repo's install, test, lint, build, and run commands. Add or refine +those commands in the agent instructions so future coding sessions can verify +changes confidently. + +Cancelled 2026-10-05: there is no code yet. This moves to ILIB-WP-0002-T01. + +## Seed First Real Workplan + +```task +id: ILIB-WP-0001-T03 +status: done +priority: medium +``` + +Create the first implementation workplan for the repository's most important +next change. After workplan file updates, run the sync locally from this repo +checkout: + +```bash +statehub fix-consistency +``` + +Done 2026-10-05: ILIB-WP-0002 holds the infrastructure. intelligence-radar +IR-WP-0010 is the first consumer. diff --git a/workplans/ILIB-WP-0002-library-infrastructure.md b/workplans/ILIB-WP-0002-library-infrastructure.md new file mode 100644 index 0000000..af28724 --- /dev/null +++ b/workplans/ILIB-WP-0002-library-infrastructure.md @@ -0,0 +1,153 @@ +--- +id: ILIB-WP-0002 +type: workplan +title: "Establish the inbound-library infrastructure" +domain: infotech +repo: inbound-library +status: active +flavor: implementation +owner: claude +topic_slug: custodian +created: "2026-10-05" +updated: "2026-10-05" +--- + +# Establish the inbound-library infrastructure + +Build the first working library, following INTENT.md: +- read-only IMAP intake; +- an append-only item store; +- file-backed topics, sources and subscriptions; +- notices through State Hub messages, a pull CLI and digest files; +- a systemd user timer on the workstation. + +Python 3.11+ standard library only. The first consumer is intelligence-radar +(IR-WP-0010). + +## Repository baseline and data contracts + +```task +id: ILIB-WP-0002-T01 +status: todo +priority: high +``` + +Create the `inbound_library` package, `python3 -m inbound_library` with +`--help`, a Makefile (`check`, `test`, `demo`, `sync`) and `docs/data-contracts.md`. +The contracts define item, provenance, source, topic, subscription and notice, +with schema versions. Record the install, test and run commands in AGENTS.md +(taken over from ILIB-WP-0001-T02). + +Acceptance: `make check` runs the unit tests on synthetic fixtures. The contracts +document every field that a consumer may rely on. + +## Append-only item store + +```task +id: ILIB-WP-0002-T02 +status: todo +priority: high +``` + +Store original bytes under `data/items/`, with a provenance record for +each delivery. Identical bytes delivered twice give one item with two +deliveries. Derive plain text and metadata into separate, rebuildable records. +Add an integrity check that rehashes every retained item. Never overwrite and +never delete. Items without provenance go to quarantine. + +Acceptance: the tests prove there is no overwrite, that deduplication works, +that tampering is detected and that quarantine routing works. + +## Read-only IMAP intake + +```task +id: ILIB-WP-0002-T03 +status: todo +priority: high +``` + +Poll one dedicated mailbox with `imaplib`, over TLS, using SELECT read-only and +BODY.PEEK[]. Credentials come only from environment variable names set in the +config. Keep a UIDVALIDITY/UID cursor in the library. Retain the full RFC 822 +message as the item, and record each attachment as its own item linked to the +message. Map the sender (and optionally a List-Id or subject pattern) to a +registered source. Unknown senders are quarantined and not routed. Record +DKIM/SPF authentication results as provenance, not as trust. Reuse the +read-only IMAP approach of email-connect EMAIL-WP-0002 where that helps. + +Acceptance: the tests run against a fixture mailbox. A live run is part of T07. + +## File-backed registries + +```task +id: ILIB-WP-0002-T04 +status: todo +priority: high +``` + +Add `registry/topics.json` (a versioned vocabulary), `registry/sources/*.json` +(owner repo, channel, match rules, topics, expected cadence, +`visibility: shared|restricted`) and `registry/subscriptions/.json` +(topics, sources, channels, cadence, agent name, digest path). A consumer +registers a source from its own research with a commit. The commit is checked +for unknown topics, duplicate match rules and restricted sources that leak to +other repos. + +Acceptance: `check` rejects invalid registries. A source registered by repo A +is matched for repo B through a shared topic. + +## Matching and notification + +```task +id: ILIB-WP-0002-T05 +status: todo +priority: high +``` + +Match new items to subscriptions by source and topic. Then deliver: +- a State Hub agent message (`POST /messages/`) per subscriber and run, listing + the new item ids, sources and pull commands; +- a pull CLI: `new --repo … --topic … --since …`, `show ` and `get `; +- a Markdown digest file per subscriber and cadence. + +Record each notice in `data/notices/`, so a retry never notifies twice. If the +hub is unreachable, keep the notice pending and retry on the next run. + +Acceptance: the tests prove idempotent delivery and recovery when the hub is +unreachable. + +## Scheduling, health and source silence + +```task +id: ILIB-WP-0002-T06 +status: todo +priority: medium +``` + +Add a `run-due` that does the intake, then matching and delivery, under a lock. +Add a systemd user service and timer in `ops/systemd/`, and a `health` command. +Health reports: +- the last scan; +- items per source; +- sources that have stayed silent past their expected cadence; +- quarantine size; +- pending notices. + +Acceptance: a silent source is reported, and repeated timer wakes are safe. + +## Provision the dedicated mailbox + +```task +id: ILIB-WP-0002-T07 +status: todo +priority: high +``` + +Operator decisions: +- which mailbox (a new address, or a Gmail alias plus label or folder); +- how its IMAP credential is delivered (an app password through the + secrets-engine path, never committed). + +Then do a live read-only scan. + +Acceptance: one real mail retained with provenance. No mailbox state changed.