web-fetcher/AGENTS.md

web-fetcher — Agent Instructions

Repo Identity

Purpose: Provenance-first, standard-library fetching of data from public APIs and web sources for research repositories.

Domain: infotech Repo slug: web-fetcher Topic ID: 595afc64-bd28-47bf-aafb-ba230b28371b Workplan prefix: WEBFETCH-WP-

---

State Hub Integration

The Custodian State Hub tracks work across all domains. Codex uses HTTP REST and the statehub CLI by default. MCP is opt-in because the current Codex MCP bridge adds severe call latency; the full administrative MCP surface remains available to clients that need it.

Context	URL
Local workstation	http://127.0.0.1:8000
Remote via tunnel	http://127.0.0.1:18000
Optional local edge relay	http://127.0.0.1:18080

When an operator has enabled the edge relay, set API_BASE to the relay URL. Queueable writes return an explicit queued receipt if the central hub is unreachable. Treat that as pending local evidence, then ask the operator to run statehub outbox status/replay after connectivity returns.

Codex workspace-write sandboxes need network access enabled to reach the host's loopback listener. Bootstrap this once with make -C ~/state-hub configure-codex and restart Codex. The canonical REST health endpoint is /state/health, not /health. If a sandboxed loopback probe fails, retry it with escalated execution before declaring State Hub unavailable; a managed Codex permission profile may still enforce isolated networking. Experimental MCP can be enabled explicitly with make -C ~/state-hub configure-codex WITH_MCP=1.

Repository files cross machine boundaries through committed Forge revisions: workstation agents push to Forgejo and cluster infrastructure reads a verified commit into ephemeral storage. Neither side reads or writes the other's checkout. local_path and host_paths describe host-local observations only.

Orient at session start

# Offline brief — works without hub connection
cat .custodian-brief.md

# Active workplans for this domain
curl -s "http://127.0.0.1:8000/workplans/?topic_id=595afc64-bd28-47bf-aafb-ba230b28371b&status=active" \
  | python3 -m json.tool

# Check inbox
curl -s "http://127.0.0.1:8000/messages/?to_agent=web-fetcher&unread_only=true" \
  | python3 -m json.tool

Mark a message read:

curl -s -X PATCH "http://127.0.0.1:8000/messages/<id>/read?reader=web-fetcher" \
  -H "Content-Type: application/json" -d '{}'
# Standing notices (to_agent "broadcast", kind "standing") stay until acknowledged:
# curl -s -X POST "http://127.0.0.1:8000/messages/<id>/ack" -H "Content-Type: application/json" -d '{"agent":"web-fetcher"}'

Log progress (required at session close)

curl -s -X POST http://127.0.0.1:8000/progress/ \
  -H "Content-Type: application/json" \
  -d '{
    "summary": "what was done",
    "event_type": "note",
    "author": "codex",
    "workplan_id": "<uuid>",
    "task_id": "<uuid>"
  }'

Omit workplan_id / task_id when not applicable.

Update task status

curl -s -X PATCH "http://127.0.0.1:8000/tasks/<task_id>" \
  -H "Content-Type: application/json" \
  -d '{"status": "progress"}'
# values: wait | todo | progress | done | cancel

Flag a task for human review

curl -s -X PATCH "http://127.0.0.1:8000/tasks/<task_id>" \
  -H "Content-Type: application/json" \
  -d '{"needs_human": true, "intervention_note": "reason"}'

---

Session Protocol

Start:

1. cat .custodian-brief.md — domain goal and open workplans (offline-safe)
2. Check inbox: GET /messages/?to_agent=web-fetcher&unread_only=true; mark read
3. Scan workplans: ls workplans/ — note status: ready, active, or blocked files and open tasks
4. Check human-needed tasks: GET /tasks/?needs_human=true

During work:

• Update task statuses in workplan files as tasks progress
• Record significant decisions via POST /decisions/

Close:

1. Update workplan file task statuses to reflect progress
2. If finishing a workplan: hand off residuals as live work records first (intake with origin: residual + origin_ref: <WP-id>, or a next workplan / decision / engagement). Do not park leftovers only in prose or SCOPE.md. Canon: the-custodian/canon/standards/work-record-types_v0.1.md § Residuals.
3. Log: POST /progress/ with a summary of what changed (name handoff ids)
4. After workplan file changes, run:

   uv run --project ~/repo-manager rmgr sync --path . --push
   

   This assigns only missing deterministic identifiers, verifies the pushed Forgejo commit and primary/railliance01, then requests one central reconciliation. A queued receipt is pending evidence; rerun after connectivity returns. Use statehub fix-consistency for a separate deep audit.

---

Credential and access routing

Audience: Codex, Claude Code, Grok, and custodian agents that call llm-connect for inference. Run this check before requesting secrets, API keys, SSH access, login tokens, or database passwords — in any repo, not only ops-warden.

ops-warden issues SSH certificates only (warden sign, cert_command). Every other credential need belongs to another subsystem. Do not message ops-warden on State Hub expecting a secret value; the reply is a pointer, not a key.

Lookup (do this first)

warden route find "<describe your need>" --json
warden route show <catalog-id> --json

Requires the warden CLI from ~/ops-warden (uv tool install . or uv run warden).

Agent runtime	How to orient
Codex / Grok (shell, HTTP State Hub)	warden route commands above; inbox to_agent=web-fetcher is for coordination, not secret vending
Claude Code (MCP when available)	get_domain_summary("custodian") for workplans; still use warden route for credential ownership
llm-connect (inference service)	Never put secret retrieval in prompts; route custody to OpenBao/operator paths surfaced by warden route

Quick routing table

I need…	Owner	ops-warden executes?
SSH cert (adm/agt/atm)	ops-warden	Yes — warden sign
API key, DB password, provider token	OpenBao (railiance-platform)	No — route only
Login / OIDC / MFA	key-cape / Keycloak	No — route only
Authorization decision	flex-auth	No — route only
activity-core → issue-core emission	activity-core + issue-core	No — warden route show activity-core-issue-sink
SSH tunnel	ops-bridge (+ cert_command from warden)	No — route only

Anti-patterns (do not do these)

• POST /messages/ to ops-warden asking for ISSUE_CORE_API_KEY, OPENROUTER_API_KEY, etc.
• Inventing warden secret, warden login, warden bao, warden tunnel — they do not exist
• Pasting secrets into Git, State Hub, workplans, logs, or chat

Other capabilities (reuse-surface)

Non-credential capabilities are usually discovered through reuse-surface federation (reuse-surface registry / capability.* indexes). Credential routing is inlined in every repo's agent instructions because it is high-frequency, high-risk, and easy to get wrong.

Canon: ~/ops-warden/wiki/CredentialRouting.md · catalog ~/ops-warden/registry/routing/catalog.yaml

---

Workplan Convention (ADR-001)

Work items originate as files in this repo — not in the hub. The hub is a read/cache/index layer that rebuilds from files.

File location: workplans/WEBFETCH-WP-NNNN-<slug>.md

Archived location: finished workplans may move to workplans/archived/YYMMDD-WEBFETCH-WP-NNNN-<slug>.md. The YYMMDD prefix is the completion/archive date; the frontmatter id does not change.

Ad Hoc Tasks: small opportunistic fixes discovered during a session use workplans/ADHOC-YYYY-MM-DD.md, workplan id WEBFETCH-WP-ADHOC-YYYY-MM-DD, and task ids WEBFETCH-WP-ADHOC-YYYY-MM-DD-T01, etc. WEBFETCH-WP includes its final -WP token. Unqualified historic ADHOC-* ids are grandfathered and must not be copied into new records. Use this only for low-risk work completed directly; create a normal workplan for anything needing analysis, design, approval, dependencies, or multiple phases.

Frontmatter:

---
id: WEBFETCH-WP-NNNN
type: workplan
title: "..."
domain: infotech
repo: web-fetcher
status: proposed | ready | active | blocked | backlog | finished | archived
owner: codex
topic_slug: ...
created: "YYYY-MM-DD"
updated: "YYYY-MM-DD"
state_hub_workstream_id: "<uuid>"   # deterministic UUIDv5; managed by Repo Manager
---

Use proposed for a new draft, ready after review against current repo state, and finished after implementation. stalled and needs_review are derived health labels, not frontmatter statuses.

Terminology: workplan is the fleet term; workstream appears only in legacy API/MCP/frontmatter bridges until STATE-WP-0069 retires them — see the-custodian/canon/standards/workplan-terminology-fleet_v0.1.md.

Task block format (one per ## section):

## Task Title

` ` `task
id: WEBFETCH-WP-NNNN-T01
status: wait | todo | progress | done | cancel
priority: high | medium | low
state_hub_task_id: "<uuid>"         # deterministic UUIDv5; managed by Repo Manager
` ` `

Task description text.

Status progression: todo → progress → done; use wait for waiting/blocked work and cancel for stopped work.

Residuals when finishing: actionable leftovers become live work records before status: finished — usually an intake (origin: residual, origin_ref: WEBFETCH-WP-NNNN) or a spawned workplan. Residual is a role, not a kind. Fleet list lives on State Hub, not in SCOPE.md.

To create a new workplan:

1. Write the file following the format above
2. Run uv run --project ~/repo-manager rmgr sync --path . --push.
3. Run statehub fix-consistency only when a separate deep audit is needed.

Developer Workflow

Standard library only (Python 3.10+); there is no build or lint toolchain and no dependency install step.

Step	Command
Install (optional, editable)	pip install -e . — or use WEB_FETCHER_PATH/PYTHONPATH from consumers
Test (offline, fixtures)	python3 -B -m unittest discover -s tests
Run	python3 -m web_fetcher fetch specs.json [--cache .cache] [--only ID] > observations.jsonl
Byte-compile check	python3 -m compileall -q web_fetcher
Validate capability registry	uv run --project ~/reuse-surface reuse-surface validate --root .
Live smoke test (network)	WEB_FETCHER_LIVE=1 python3 -B -m unittest tests.test_live -v
CI	.forgejo/workflows/test.yaml runs the offline tests on Python 3.10 and 3.13 for every push to main and every v* tag

Rules: every new adapter ships with a recorded fixture in tests/fixtures/ and an offline test; tests must not hit the network. Do not add third-party dependencies. When behaviour, adapters or maturity change, update registry/capabilities/capability.infotech.provenance-fetch.md and its index row (keep summary/tags identical, or validate reports drift).