diff --git a/.custodian-brief.md b/.custodian-brief.md new file mode 100644 index 0000000..9105d9d --- /dev/null +++ b/.custodian-brief.md @@ -0,0 +1,27 @@ + +# Custodian Brief - web-fetcher + +**Project:** web-fetcher +**Domain:** infotech +**State Hub:** http://127.0.0.1:8000 +**Topic ID:** `595afc64-bd28-47bf-aafb-ba230b28371b` + +## Open Workplans + +### Bootstrap State Hub integration + +Workplan file: `workplans/WEBFETCH-WP-0001-statehub-bootstrap.md` + +Open tasks: +- T01 - Review generated integration files +- T02 - Verify local developer workflow +- T03 - Seed first real workplan + +## Session Start + +1. Read `INTENT.md`, `SCOPE.md`, and `AGENTS.md`. +2. Check inbox: `GET /messages/?to_agent=web-fetcher&unread_only=true`. +3. Scan `workplans/`. +4. Update task statuses in workplan files as work progresses. + +Last generated: 2026-10-13 diff --git a/.gitignore b/.gitignore index f5d1254..3bb735b 100644 --- a/.gitignore +++ b/.gitignore @@ -1,3 +1,9 @@ __pycache__/ *.pyc .cache/ + +# state-hub: track .claude/rules +# Claude Code local state (track shared rules; ignore machine-specific files) +.claude/* +!.claude/rules/ +!.claude/rules/*.md diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..a14b9a2 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,261 @@ +# web-fetcher — Agent Instructions + +## Repo Identity + +**Purpose:** Provenance-first, standard-library fetching of data from public APIs and web sources for research repositories. + +**Domain:** infotech +**Repo slug:** web-fetcher +**Topic ID:** `595afc64-bd28-47bf-aafb-ba230b28371b` +**Workplan prefix:** `WEBFETCH-WP-` + +--- + +## State Hub Integration + +The Custodian State Hub tracks work across all domains. Codex uses HTTP REST and +the `statehub` CLI by default. MCP is opt-in because the current Codex MCP bridge +adds severe call latency; the full administrative MCP surface remains available +to clients that need it. + +| Context | URL | +|---------|-----| +| Local workstation | `http://127.0.0.1:8000` | +| Remote via tunnel | `http://127.0.0.1:18000` | +| Optional local edge relay | http://127.0.0.1:18080 | + +When an operator has enabled the edge relay, set API_BASE to the relay URL. +Queueable writes return an explicit queued receipt if the central hub is +unreachable. Treat that as pending local evidence, then ask the operator to run +statehub outbox status/replay after connectivity returns. + +Codex workspace-write sandboxes need network access enabled to reach the host's +loopback listener. Bootstrap this once with `make -C ~/state-hub configure-codex` +and restart Codex. The canonical REST health endpoint is `/state/health`, not +`/health`. If a sandboxed loopback probe fails, retry it with escalated execution +before declaring State Hub unavailable; a managed Codex permission profile may +still enforce isolated networking. Experimental MCP can be enabled explicitly +with `make -C ~/state-hub configure-codex WITH_MCP=1`. + +Repository files cross machine boundaries through committed Forge revisions: +workstation agents push to Forgejo and cluster infrastructure reads a verified +commit into ephemeral storage. Neither side reads or writes the other's +checkout. `local_path` and `host_paths` describe host-local observations only. + +### Orient at session start + +```bash +# Offline brief — works without hub connection +cat .custodian-brief.md + +# Active workplans for this domain +curl -s "http://127.0.0.1:8000/workplans/?topic_id=595afc64-bd28-47bf-aafb-ba230b28371b&status=active" \ + | python3 -m json.tool + +# Check inbox +curl -s "http://127.0.0.1:8000/messages/?to_agent=web-fetcher&unread_only=true" \ + | python3 -m json.tool +``` + +Mark a message read: +```bash +curl -s -X PATCH "http://127.0.0.1:8000/messages//read?reader=web-fetcher" \ + -H "Content-Type: application/json" -d '{}' +# Standing notices (to_agent "broadcast", kind "standing") stay until acknowledged: +# curl -s -X POST "http://127.0.0.1:8000/messages//ack" -H "Content-Type: application/json" -d '{"agent":"web-fetcher"}' +``` + +### Log progress (required at session close) + +```bash +curl -s -X POST http://127.0.0.1:8000/progress/ \ + -H "Content-Type: application/json" \ + -d '{ + "summary": "what was done", + "event_type": "note", + "author": "codex", + "workplan_id": "", + "task_id": "" + }' +``` + +Omit `workplan_id` / `task_id` when not applicable. + +### Update task status + +```bash +curl -s -X PATCH "http://127.0.0.1:8000/tasks/" \ + -H "Content-Type: application/json" \ + -d '{"status": "progress"}' +# values: wait | todo | progress | done | cancel +``` + +### Flag a task for human review + +```bash +curl -s -X PATCH "http://127.0.0.1:8000/tasks/" \ + -H "Content-Type: application/json" \ + -d '{"needs_human": true, "intervention_note": "reason"}' +``` + +--- + +## Session Protocol + +**Start:** +1. `cat .custodian-brief.md` — domain goal and open workplans (offline-safe) +2. Check inbox: `GET /messages/?to_agent=web-fetcher&unread_only=true`; mark read +3. Scan workplans: `ls workplans/` — note `status: ready`, `active`, or `blocked` files and open tasks +4. Check human-needed tasks: `GET /tasks/?needs_human=true` + +**During work:** +- Update task statuses in workplan files as tasks progress +- Record significant decisions via `POST /decisions/` + +**Close:** +1. Update workplan file task statuses to reflect progress +2. If finishing a workplan: hand off **residuals** as live work records first + (intake with `origin: residual` + `origin_ref: `, or a next workplan / + decision / engagement). Do not park leftovers only in prose or `SCOPE.md`. + Canon: `the-custodian/canon/standards/work-record-types_v0.1.md` § Residuals. +3. Log: `POST /progress/` with a summary of what changed (name handoff ids) +4. After workplan file changes, run: + ```bash + uv run --project ~/repo-manager rmgr sync --path . --push + ``` + This assigns only missing deterministic identifiers, verifies the pushed + Forgejo commit and `primary/railliance01`, then requests one central + reconciliation. A queued receipt is pending evidence; rerun after + connectivity returns. Use `statehub fix-consistency` for a separate deep audit. + +--- + +## Credential and access routing + +**Audience:** Codex, Claude Code, Grok, and custodian agents that call **llm-connect** +for inference. Run this check **before** requesting secrets, API keys, SSH access, +login tokens, or database passwords — in any repo, not only `ops-warden`. + +ops-warden **issues SSH certificates only** (`warden sign`, `cert_command`). Every +other credential need belongs to another subsystem. **Do not** message +`ops-warden` on State Hub expecting a secret value; the reply is a pointer, not a key. + +### Lookup (do this first) + +```bash +warden route find "" --json +warden route show --json +``` + +Requires the `warden` CLI from `~/ops-warden` (`uv tool install .` or `uv run warden`). + +| Agent runtime | How to orient | +| --- | --- | +| **Codex / Grok** (shell, HTTP State Hub) | `warden route` commands above; inbox `to_agent=web-fetcher` is for coordination, not secret vending | +| **Claude Code** (MCP when available) | `get_domain_summary("custodian")` for workplans; **still** use `warden route` for credential ownership | +| **llm-connect** (inference service) | Never put secret retrieval in prompts; route custody to OpenBao/operator paths surfaced by `warden route` | + +### Quick routing table + +| I need… | Owner | ops-warden executes? | +| --- | --- | --- | +| SSH cert (`adm`/`agt`/`atm`) | ops-warden | **Yes** — `warden sign` | +| API key, DB password, provider token | OpenBao (`railiance-platform`) | No — route only | +| Login / OIDC / MFA | key-cape / Keycloak | No — route only | +| Authorization decision | flex-auth | No — route only | +| activity-core → issue-core emission | activity-core + issue-core | No — `warden route show activity-core-issue-sink` | +| SSH tunnel | ops-bridge (+ `cert_command` from warden) | No — route only | + +### Anti-patterns (do not do these) + +- `POST /messages/` to `ops-warden` asking for `ISSUE_CORE_API_KEY`, `OPENROUTER_API_KEY`, etc. +- Inventing `warden secret`, `warden login`, `warden bao`, `warden tunnel` — they do not exist +- Pasting secrets into Git, State Hub, workplans, logs, or chat + +### Other capabilities (reuse-surface) + +Non-credential capabilities are usually discovered through **reuse-surface** federation +(`reuse-surface` registry / `capability.*` indexes). Credential routing is inlined in +every repo's agent instructions because it is high-frequency, high-risk, and easy to +get wrong. + +**Canon:** `~/ops-warden/wiki/CredentialRouting.md` · catalog `~/ops-warden/registry/routing/catalog.yaml` + + + + +--- + +## Workplan Convention (ADR-001) + +Work items originate as files in this repo — not in the hub. The hub is a +read/cache/index layer that rebuilds from files. + +**File location:** `workplans/WEBFETCH-WP-NNNN-.md` + +**Archived location:** finished workplans may move to +`workplans/archived/YYMMDD-WEBFETCH-WP-NNNN-.md`. The `YYMMDD` prefix is +the completion/archive date; the frontmatter `id` does not change. + +**Ad Hoc Tasks:** small opportunistic fixes discovered during a session use +`workplans/ADHOC-YYYY-MM-DD.md`, workplan id +`WEBFETCH-WP-ADHOC-YYYY-MM-DD`, and task ids +`WEBFETCH-WP-ADHOC-YYYY-MM-DD-T01`, etc. `WEBFETCH-WP` includes its final `-WP` +token. Unqualified historic `ADHOC-*` ids are grandfathered and must not be +copied into new records. Use this only for low-risk work completed directly; +create a normal workplan for anything needing analysis, design, approval, +dependencies, or multiple phases. + +**Frontmatter:** + +```yaml +--- +id: WEBFETCH-WP-NNNN +type: workplan +title: "..." +domain: infotech +repo: web-fetcher +status: proposed | ready | active | blocked | backlog | finished | archived +owner: codex +topic_slug: ... +created: "YYYY-MM-DD" +updated: "YYYY-MM-DD" +state_hub_workstream_id: "" # deterministic UUIDv5; managed by Repo Manager +--- +``` + +Use `proposed` for a new draft, `ready` after review against current repo +state, and `finished` after implementation. `stalled` and `needs_review` are +derived health labels, not frontmatter statuses. + +**Terminology:** workplan is the fleet term; `workstream` appears only in legacy +API/MCP/frontmatter bridges until `STATE-WP-0069` retires them — see +`the-custodian/canon/standards/workplan-terminology-fleet_v0.1.md`. + +**Task block format** (one per `##` section): + +``` +## Task Title + +` ` `task +id: WEBFETCH-WP-NNNN-T01 +status: wait | todo | progress | done | cancel +priority: high | medium | low +state_hub_task_id: "" # deterministic UUIDv5; managed by Repo Manager +` ` ` + +Task description text. +``` + +Status progression: `todo` → `progress` → `done`; use `wait` for waiting/blocked work and `cancel` for stopped work. + +**Residuals when finishing:** actionable leftovers become live work records +before `status: finished` — usually an intake (`origin: residual`, +`origin_ref: WEBFETCH-WP-NNNN`) or a spawned workplan. Residual is a *role*, +not a kind. Fleet list lives on State Hub, not in `SCOPE.md`. + +To create a new workplan: +1. Write the file following the format above +2. Run `uv run --project ~/repo-manager rmgr sync --path . --push`. +3. Run `statehub fix-consistency` only when a separate deep audit is needed. diff --git a/SCOPE.md b/SCOPE.md new file mode 100644 index 0000000..071fa12 --- /dev/null +++ b/SCOPE.md @@ -0,0 +1,32 @@ +# SCOPE + +> This file was generated by `statehub register`. Refine it as the repository +> boundaries become clearer. + +## One-liner + +Provenance-first, standard-library fetching of data from public APIs and web sources for research repositories. + +## Core Idea + +web-fetcher exists to provide the capability described in INTENT.md. + +## In Scope + +- Maintain the repository's primary implementation. +- Keep docs, tests, and operational metadata current. + +## Out of Scope + +- Own unrelated adjacent systems. +- Make irreversible operational decisions without human approval. + +## Current State + +- Status: active; implementation and stability should be verified by the repo agent. + +## Getting Oriented + +- Start with: INTENT.md +- Agent instructions: AGENTS.md +- Workplans: workplans/ diff --git a/workplans/WEBFETCH-WP-0001-statehub-bootstrap.md b/workplans/WEBFETCH-WP-0001-statehub-bootstrap.md new file mode 100644 index 0000000..d517b1a --- /dev/null +++ b/workplans/WEBFETCH-WP-0001-statehub-bootstrap.md @@ -0,0 +1,57 @@ +--- +id: WEBFETCH-WP-0001 +type: workplan +title: "Bootstrap State Hub integration" +domain: infotech +repo: web-fetcher +status: ready +flavor: planning +depends_on: [] +owner: codex +topic_slug: stack +created: "2026-10-13" +updated: "2026-10-13" +--- + +# Bootstrap State Hub integration + +Provenance-first, standard-library fetching of data from public APIs and web sources for research repositories. + +## Review Generated Integration Files + +```task +id: WEBFETCH-WP-0001-T01 +status: todo +priority: high +``` + +Review `INTENT.md`, `SCOPE.md`, `AGENTS.md`, and `.custodian-brief.md`. +Replace generated placeholders with repo-specific facts where needed. + +## Verify Local Developer Workflow + +```task +id: WEBFETCH-WP-0001-T02 +status: todo +priority: high +``` + +Identify the repo's install, test, lint, build, and run commands. Add or refine +those commands in the agent instructions so future coding sessions can verify +changes confidently. + +## Seed First Real Workplan + +```task +id: WEBFETCH-WP-0001-T03 +status: todo +priority: medium +``` + +Create the first implementation workplan for the repository's most important +next change. After workplan file updates, run the sync locally from this repo +checkout: + +```bash +statehub fix-consistency +```