Implement ACTIVITY-WP-0022/0023: safe sink default and gap closures

Default ISSUE_SINK_TYPE to state-hub (no silent Forgejo issues), hard-fail prune apply without live-images protection, refresh-live-images script, disable TaskExecutor stub by default, and document consumer/sink contracts.
2026-07-21 21:40:08 +02:00 · 2026-07-21 21:40:08 +02:00 · 4f5399df84
commit 4f5399df84
parent 5c7a90ce7c
19 changed files with 525 additions and 155 deletions
--- a/docs/conventions.md
+++ b/docs/conventions.md
@ -18,7 +18,7 @@ extension point `af654abb`).
 | Queue name | Registered workers |
 |---|---|
 | `orchestrator-tq` | `RunActivityWorkflow` and all its activities (`load_activity_definition`, `resolve_context`, `log_run`) |
-| `task-execution-tq` | `TaskExecutorWorkflow` compatibility stub only; real execution belongs in per-repo workers |
+| `task-execution-tq` | Legacy `TaskExecutorWorkflow` stub only when `ACTIVITY_CORE_ENABLE_TASK_EXECUTOR_STUB=true`; real execution belongs in per-repo workers / agent-harness |

 **Rule:** a workflow and its activities must be registered on the same task queue.
 Cross-queue activity calls require an explicit `task_queue` argument on
--- a/docs/issue-core-emission-boundary.md
+++ b/docs/issue-core-emission-boundary.md
@ -1,19 +1,30 @@
-# Issue-Core Emission Boundary
+# Task emission boundary (IssueSink)

 activity-core owns the decision to spawn a task and the audit trail that says
 why it spawned. It does not own downstream task lifecycle state after emission.

-## Current authoritative endpoint
+## Sink matrix (ACTIVITY-WP-0022)

-The current authoritative boundary is the issue-core REST API:
+| `ISSUE_SINK_TYPE` | Destination | Default? |
+| --- | --- | --- |
+| **`state-hub`** | State Hub progress `activity_task_spawn` | **Yes** (safe for internal findings) |
+| **`null`** | Synthetic `null-*` refs only | Dry-run / tests |
+| **`rest`** | issue-core `POST /issues/` (may → Forgejo) | **Explicit opt-in only** |
+
+Code default when unset: **`state-hub`** (`DEFAULT_ISSUE_SINK_TYPE`).
+
+Unknown values fall back to `state-hub` and log a warning.
+
+Consumer contract (fields, idempotency, Binky completion):  
+`docs/task-emission-consumer-contract.md`.
+
+## REST endpoint (opt-in)

 ```text
 POST {ISSUE_CORE_URL}/issues/
 ```

-`IssueCoreRestSink` authenticates with the shared `ISSUE_CORE_API_KEY` env var
-(same value as the issue-core server) via `Authorization: Bearer <key>` and
-sends this payload:
+`IssueCoreRestSink` uses `ISSUE_CORE_API_KEY` via `Authorization: Bearer <key>`:

 ```json
 {
@ -30,110 +41,57 @@ sends this payload:
 }
 ```

-The expected response contains `issue_id` and may include `issue_url` and
-`backend`. activity-core stores only the returned task reference in
-`task_spawn_log`; issue-core remains authoritative for task status, assignment,
-comments, closure, and cancellation.
+Response: `issue_id` (required), optional `issue_url`, `backend`. Stored in
+`task_spawn_log` only as a reference.
+
+## State Hub sink (default)
+
+Each TaskSpec becomes a progress event (`activity_task_spawn` by default).
+No Forgejo issue is created. Suitable for Binky rhythm, internal hygiene, and
+any definition that should not spam trackers.

 ## REST versus NATS

-Keep REST as the active emission contract until issue-core publishes and owns a
-durable NATS consumer for task-creation commands. NATS is still appropriate for
-event intake into activity-core, but task creation needs an acknowledged,
-idempotent command boundary. A future NATS sink must return or later correlate a
-task reference before it can replace `IssueCoreRestSink`.
+Keep REST as the issue-core transport until issue-core owns a durable NATS
+command consumer. NATS remains appropriate for **event intake** into
+activity-core.

-## Safe operating modes
+## Operating modes

- `ISSUE_SINK_TYPE=null`: dry-run/audit mode. Task specs are rendered and the
-  workflow records synthetic `null-*` references. Use this for contract review
-  and emergency rollback.
- `ISSUE_SINK_TYPE=rest`: live task creation. Sink failures raise out of
-  `emit_tasks`, so Temporal retries and the workflow history make failures
-  visible. Railiance runtime ConfigMap uses this mode once
-  `ISSUE_CORE_API_KEY` is present in `actcore-runtime-secret`.
- `ISSUE_SINK_TYPE=state-hub`: ACTIVITY-WP-0021 path B. Each TaskSpec is posted
-  as a State Hub progress event (`activity_task_spawn` by default) instead of
-  creating a Forgejo issue. Use when issue-core→Forgejo is down or automated
-  Forgejo issues are policy-blocked.
+- **`state-hub` (default):** fleet-visible spawns without Forgejo.
+- **`null`:** dry-run / audit.
+- **`rest`:** live issue-core; requires healthy backend and intentional policy.
+  Railiance production uses `state-hub` unless an overlay explicitly sets
+  `rest` for an experiment.

-### Known production failure (2026-07-21)
+### Known production failure (2026-07-21) — rest path

-`POST /issues/` returned **HTTP 503** with:
+`POST /issues/` returned **HTTP 503**:

 ```text
 Failed to connect to backend 'forgejo-inbox': Failed to connect to Gitea API
 ```

-Root cause on coulombcore issue-core: `GITEA_BACKEND_TOKEN` is rejected by
-Forgejo (`/api/v1/user` → 401 user does not exist). Healthz stays 200.
+`GITEA_BACKEND_TOKEN` on issue-core rejected by Forgejo. Fix is issue-core
+token rotation (`warden route show issue-core-ingestion-api-key`), not
+activity-core defaults.

-**Operator fix (path A):** rotate `GITEA_BACKEND_TOKEN` in OpenBao path
-`platform/workloads/issue-core/issue-core/issue-core-runtime` (see
-`warden route show issue-core-ingestion-api-key`) using a valid Forgejo PAT for
-the issue-core service identity, then restart `issue-core` so the entrypoint
-rewrites backends.json. Smoke from the worker:
+## Promotion to rest (one definition at a time)

-```bash
-# From actcore-worker (does not print secrets)
-python -c "import os,httpx; r=httpx.post(os.environ['ISSUE_CORE_URL']+'/issues/',
-  json={...full TaskSpec payload...},
-  headers={'Authorization':'Bearer '+os.environ['ISSUE_CORE_API_KEY']}, timeout=30);
-  print(r.status_code, r.text[:200])"
-```
-
-Expect **201**, not 503.
-
-Weekly SBOM staleness now posts a deterministic `sbom_staleness` progress report
-even when task emission is disabled.
-
-## Promotion and rollback
-
-### Promote one definition safely
-
-1. Keep `ISSUE_SINK_TYPE=null` and run or wait for the target definition.
-2. Review rendered task specs in `task_spawn_log` (source id, condition,
-   target repo, synthetic `null-*` reference).
-3. Confirm `ISSUE_CORE_URL` reachability and a populated `ISSUE_CORE_API_KEY`
-   on both activity-core and issue-core (same value). Credential custody:
-   `warden route show issue-core-ingestion-api-key --json`.
-4. Run the repo smoke:
-   ```bash
-   uv run python scripts/smoke_issue_core_emission.py
-   ISSUE_CORE_URL=http://127.0.0.1:8765 ISSUE_CORE_API_KEY=... \
-     uv run python scripts/smoke_issue_core_emission.py --live
-   ```
-5. Set `ISSUE_SINK_TYPE=rest` in `actcore-runtime-config`, apply
-   `k8s/railiance/15-externalsecret-issue-core.yaml` so External Secrets merges
-   `ISSUE_CORE_API_KEY` into `actcore-runtime-secret`, and restart
-   `actcore-worker` / `actcore-event-router` after the ExternalSecret is Ready.
-6. Trigger one known-safe run (weekly SBOM staleness on a stale fixture or
-   manual `/activity-definitions/<id>/trigger`) and confirm `task_spawn_log`
-   stores the real `issue_id` returned by issue-core.
-
-### Roll back to null-sink
-
-1. Set `ISSUE_SINK_TYPE=null` in `actcore-runtime-config`.
-2. `kubectl -n activity-core rollout restart deploy/actcore-worker deploy/actcore-event-router`
-3. Verify the next run records synthetic `null-*` references again.
-4. Leave issue-core tasks already created in place; activity-core does not own
-   downstream task lifecycle. Close or cancel duplicates in issue-core if a
-   promotion experiment created unexpected tasks.
-
-Duplicate handling today: issue-core REST ingest does not yet dedupe on
-`triggering_event_id`; Temporal retry visibility is the current guardrail.
-Treat promotion as one-definition-at-a-time until server-side idempotency ships.
+1. Dry-run with `null` or observe `state-hub` spawns.
+2. Confirm issue-core smoke returns **201**.
+3. Temporarily set `ISSUE_SINK_TYPE=rest` only if policy allows external issues.
+4. Prefer per-definition future opt-in over global rest (WP-0022).
+5. Roll back: `ISSUE_SINK_TYPE=state-hub` or `null` + worker restart.

 ## Verification

-Local contract tests cover the rendered weekly SBOM task path and the REST
-payload shape:
-
 ```bash
-uv run pytest tests/test_integration_event_bridge.py tests/test_issue_sink.py
+uv run pytest tests/test_issue_sink.py tests/test_integration_event_bridge.py -q
 ```

-For a live environment, run with `ISSUE_SINK_TYPE=null` first and confirm
-`task_spawn_log` contains the expected source id, condition, triggering event id,
-and synthetic task reference. Then switch to `ISSUE_SINK_TYPE=rest` only after a
-single known-safe rule match creates one issue-core task with the same fields.
+## Side-effect resolvers (not IssueSink)
+
+`forgejo_package_prune` with `apply: true` is a **declared platform side-effect**,
+not task emission. It requires a non-empty `live_images_file` (ACTIVITY-WP-0023-T03).
+See runbook § Weekly maintenance.
--- a/docs/runbook.md
+++ b/docs/runbook.md
@ -238,6 +238,43 @@ kubectl -n activity-core exec deploy/actcore-worker -- /app/.venv/bin/python3 -c
 '
 ```

+### Where progress evidence lives (edge vs workstation)
+
+Prod activations post to **`http://actcore-statehub-edge-relay:8000`** on
+railiance01 (upstream in-cluster `state-hub`). That feed is **not always** the
+same history as workstation `http://127.0.0.1:8000` (local primary vs tunnel).
+
+After a fire, query the edge from the worker:
+
+```bash
+ssh railiance01 'export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
+kubectl -n activity-core exec deploy/actcore-worker -- /app/.venv/bin/python3 -c "
+import urllib.request, json
+for et in [\"daily_triage\",\"forgejo_package_prune\",\"activity_task_spawn\",\"sbom_staleness\"]:
+  d=json.loads(urllib.request.urlopen(
+    f\"http://actcore-statehub-edge-relay:8000/progress/?event_type={et}&limit=3\").read())
+  print(et, d[0][\"created_at\"] if d else None, (d[0].get(\"summary\") or \"\")[:80] if d else \"\")
+"'
+```
+
+## IssueSink / task emission
+
+Default: **`ISSUE_SINK_TYPE=state-hub`** (ACTIVITY-WP-0022). See
+`docs/issue-core-emission-boundary.md` and
+`docs/task-emission-consumer-contract.md`.
+
+| Mode | Use |
+| --- | --- |
+| `state-hub` | Internal findings (default) |
+| `null` | Dry-run |
+| `rest` | Intentional issue-core / external tracker only |
+
+`TaskExecutorWorkflow` is **disabled** unless
+`ACTIVITY_CORE_ENABLE_TASK_EXECUTOR_STUB=true` (legacy tests only).
+
+`review_required` on instructions is **metadata only** until a downstream
+review queue exists (issue-core / work-record lane) — see ACTIVITY-WP-0023-T09.
+
 Example distinction from the June 2026 daily triage evidence:

 ```text
@ -332,14 +369,31 @@ mount of `~/railiance-platform` on the worker) with `apply: true` and posts

 **Enabled 2026-07-21** after dry-run + first apply evidence
 (`railiance-platform/docs/evidence/forgejo-package-prune-apply-20260721.json`:
-38 deleted, 0 errors). Manual apply:
+38 deleted, 0 errors).
+
+**Apply safety (ACTIVITY-WP-0023-T03):** `apply: true` **refuses** to run without
+a non-empty `live_images_file` (or `FORGEJO_LIVE_IMAGES_FILE`). Allowed
+side-effect shell query with apply today: **`forgejo_package_prune` only**.
+
+Refresh protection list after cluster image rollouts:
+
+```bash
+# From workstation with both contexts, or merge scp'd exports on railiance01:
+./scripts/refresh_live_images.sh
+# railiance01 worker hostPath target:
+OUT=~/railiance-platform/docs/evidence/live-images-all.txt \
+  EXTRA_LIVE_FILES=/path/to/coulombcore-export.txt \
+  ./scripts/refresh_live_images.sh
+```
+
+Manual apply:

 ```bash
 cd ~/railiance-platform
 export VAULT_ADDR=https://bao.coulomb.social
 # OIDC or platform token — never paste PAT into chat
 export FORGEJO_TOKEN=$(bao kv get -field=API_TOKEN platform/workloads/forgejo/forgejo-admin)
-./tools/cmd/forgejo-package-prune --apply --live-images-file /path/to/live-images.txt
+./tools/cmd/forgejo-package-prune --apply --live-images-file docs/evidence/live-images-all.txt
 ```

 `weekly-sbom-staleness` is the canonical rule-only weekly maintenance schedule.
--- a/docs/task-emission-consumer-contract.md
+++ b/docs/task-emission-consumer-contract.md
@ -0,0 +1,78 @@
+# Task emission consumer contract
+
+**Audience:** agent-harness, per-repo Temporal workers, operators.  
+**Owners:** activity-core (producer), consumers (executors).  
+**Related:** ACTIVITY-WP-0022 (sink policy), ACTIVITY-WP-0023-T02 (executor gap).
+
+activity-core answers **when / what / where**. It does **not** execute work.
+Consumers must pick up emitted tasks and produce domain evidence.
+
+## Sink matrix (ACTIVITY-WP-0022)
+
+| `ISSUE_SINK_TYPE` | Behaviour | When to use |
+| --- | --- | --- |
+| **`state-hub` (default)** | POST State Hub `/progress/` with `event_type=activity_task_spawn` (override via `STATE_HUB_TASK_EVENT_TYPE`) | Internal fleet findings; no Forgejo issues |
+| **`null`** | Synthetic `null-*` refs in `task_spawn_log` only | Dry-run / contract review |
+| **`rest`** | POST issue-core `/issues/` (may project to Forgejo) | **Explicit opt-in** only when external tracker issues are intended and backend is healthy |
+
+Unset or unknown values fall back to **`state-hub`** (safe default).
+
+## Payload: `activity_task_spawn` (State Hub)
+
+Produced by `StateHubProgressSink`. Consumers should treat `detail` as the
+authoritative task spec.
+
+```json
+{
+  "event_type": "activity_task_spawn",
+  "author": "activity-core",
+  "summary": "<task title, max ~240 chars>",
+  "detail": {
+    "task_ref": "sh-<uuid>",
+    "title": "Run Binky daily rhythm (daily_brief) for 2026-07-21",
+    "description": "...",
+    "target_repo": "binky-control",
+    "priority": "medium",
+    "labels": ["binky", "rhythm", "automated"],
+    "source_type": "rule",
+    "source_id": "emit-daily-rhythm-task",
+    "triggering_event_id": "manual-… or scheduled",
+    "activity_definition_id": "<activity uuid>",
+    "backend": "state-hub-progress"
+  }
+}
+```
+
+### Required consumer behaviour
+
+1. **Idempotency:** key on `detail.task_ref` or
+   `(activity_definition_id, triggering_event_id, source_id, title)`.
+2. **Routing:** use `target_repo` (and labels) to select checkout / lane.
+3. **Completion evidence:** post a domain progress event when work finishes
+   (e.g. Binky: `event_type=binky_daily_brief` with `detail.repo=binky-control`
+   and date), so rhythm resolvers can set `due=false`.
+4. **Do not** re-implement task lifecycle in activity-core.
+
+## Payload: issue-core REST (`ISSUE_SINK_TYPE=rest`)
+
+See `docs/issue-core-emission-boundary.md`. `task_spawn_log.task_ref` is the
+issue-core `issue_id`. Prefer this only for intentional external projection.
+
+## Binky daily brief path (reference)
+
+1. Schedule / one-shot: `Binky Daily Operating Rhythm`.
+2. Context: `binky_rhythm_status` → `due=true` for `daily_brief`.
+3. Rule emit → sink (`state-hub` recommended).
+4. **Consumer (out of repo):** agent-harness or operator session runs
+   `binky-control` OperatingRhythm / brief scripts.
+5. Completion: post `binky_daily_brief` progress so the next fire is not due.
+
+Until a harness consumer is wired, operators may complete the brief manually
+and still post the progress event — spawn without completion leaves `due=true`.
+
+## Anti-patterns
+
+- Using `TaskExecutorWorkflow` in activity-core for real work (disabled by
+  default; ACTIVITY-WP-0023-T08).
+- Global `ISSUE_SINK_TYPE=rest` for all definitions (reintroduces Forgejo spam).
+- Treating `task_spawn_log` as task status authority.