activity-core/docs/issue-core-emission-boundary.md
tegwick 98e8aa83bd
All checks were successful
CI Smoke / host-smoke (push) Successful in 0s
CI Smoke / container-smoke (push) Successful in 2s
Build and Publish Container Image / build-and-push (push) Successful in 47s
Implement ACTIVITY-WP-0021 production automation reliability
Root-cause IssueSink 503 (dead Forgejo PAT on issue-core), add state-hub
task sink path B, log runs before emit, harden sync_schedules, deterministic
SBOM/triage reports, DB probe thrash fix, and prod automation-status helper.
2026-07-21 04:21:55 +02:00

139 lines
5.7 KiB
Markdown

# Issue-Core Emission Boundary
activity-core owns the decision to spawn a task and the audit trail that says
why it spawned. It does not own downstream task lifecycle state after emission.
## Current authoritative endpoint
The current authoritative boundary is the issue-core REST API:
```text
POST {ISSUE_CORE_URL}/issues/
```
`IssueCoreRestSink` authenticates with the shared `ISSUE_CORE_API_KEY` env var
(same value as the issue-core server) via `Authorization: Bearer <key>` and
sends this payload:
```json
{
"title": "Run SBOM rescan for activity-core",
"description": "",
"target_repo": "activity-core",
"priority": "medium",
"labels": ["sbom", "security", "automated"],
"due_in_days": null,
"source_type": "rule",
"source_id": "flag-stale-sbom",
"triggering_event_id": "event-or-schedule-key",
"activity_definition_id": "activity-definition-uuid"
}
```
The expected response contains `issue_id` and may include `issue_url` and
`backend`. activity-core stores only the returned task reference in
`task_spawn_log`; issue-core remains authoritative for task status, assignment,
comments, closure, and cancellation.
## REST versus NATS
Keep REST as the active emission contract until issue-core publishes and owns a
durable NATS consumer for task-creation commands. NATS is still appropriate for
event intake into activity-core, but task creation needs an acknowledged,
idempotent command boundary. A future NATS sink must return or later correlate a
task reference before it can replace `IssueCoreRestSink`.
## Safe operating modes
- `ISSUE_SINK_TYPE=null`: dry-run/audit mode. Task specs are rendered and the
workflow records synthetic `null-*` references. Use this for contract review
and emergency rollback.
- `ISSUE_SINK_TYPE=rest`: live task creation. Sink failures raise out of
`emit_tasks`, so Temporal retries and the workflow history make failures
visible. Railiance runtime ConfigMap uses this mode once
`ISSUE_CORE_API_KEY` is present in `actcore-runtime-secret`.
- `ISSUE_SINK_TYPE=state-hub`: ACTIVITY-WP-0021 path B. Each TaskSpec is posted
as a State Hub progress event (`activity_task_spawn` by default) instead of
creating a Forgejo issue. Use when issue-core→Forgejo is down or automated
Forgejo issues are policy-blocked.
### Known production failure (2026-07-21)
`POST /issues/` returned **HTTP 503** with:
```text
Failed to connect to backend 'forgejo-inbox': Failed to connect to Gitea API
```
Root cause on coulombcore issue-core: `GITEA_BACKEND_TOKEN` is rejected by
Forgejo (`/api/v1/user` → 401 user does not exist). Healthz stays 200.
**Operator fix (path A):** rotate `GITEA_BACKEND_TOKEN` in OpenBao path
`platform/workloads/issue-core/issue-core/issue-core-runtime` (see
`warden route show issue-core-ingestion-api-key`) using a valid Forgejo PAT for
the issue-core service identity, then restart `issue-core` so the entrypoint
rewrites backends.json. Smoke from the worker:
```bash
# From actcore-worker (does not print secrets)
python -c "import os,httpx; r=httpx.post(os.environ['ISSUE_CORE_URL']+'/issues/',
json={...full TaskSpec payload...},
headers={'Authorization':'Bearer '+os.environ['ISSUE_CORE_API_KEY']}, timeout=30);
print(r.status_code, r.text[:200])"
```
Expect **201**, not 503.
Weekly SBOM staleness now posts a deterministic `sbom_staleness` progress report
even when task emission is disabled.
## Promotion and rollback
### Promote one definition safely
1. Keep `ISSUE_SINK_TYPE=null` and run or wait for the target definition.
2. Review rendered task specs in `task_spawn_log` (source id, condition,
target repo, synthetic `null-*` reference).
3. Confirm `ISSUE_CORE_URL` reachability and a populated `ISSUE_CORE_API_KEY`
on both activity-core and issue-core (same value). Credential custody:
`warden route show issue-core-ingestion-api-key --json`.
4. Run the repo smoke:
```bash
uv run python scripts/smoke_issue_core_emission.py
ISSUE_CORE_URL=http://127.0.0.1:8765 ISSUE_CORE_API_KEY=... \
uv run python scripts/smoke_issue_core_emission.py --live
```
5. Set `ISSUE_SINK_TYPE=rest` in `actcore-runtime-config`, apply
`k8s/railiance/15-externalsecret-issue-core.yaml` so External Secrets merges
`ISSUE_CORE_API_KEY` into `actcore-runtime-secret`, and restart
`actcore-worker` / `actcore-event-router` after the ExternalSecret is Ready.
6. Trigger one known-safe run (weekly SBOM staleness on a stale fixture or
manual `/activity-definitions/<id>/trigger`) and confirm `task_spawn_log`
stores the real `issue_id` returned by issue-core.
### Roll back to null-sink
1. Set `ISSUE_SINK_TYPE=null` in `actcore-runtime-config`.
2. `kubectl -n activity-core rollout restart deploy/actcore-worker deploy/actcore-event-router`
3. Verify the next run records synthetic `null-*` references again.
4. Leave issue-core tasks already created in place; activity-core does not own
downstream task lifecycle. Close or cancel duplicates in issue-core if a
promotion experiment created unexpected tasks.
Duplicate handling today: issue-core REST ingest does not yet dedupe on
`triggering_event_id`; Temporal retry visibility is the current guardrail.
Treat promotion as one-definition-at-a-time until server-side idempotency ships.
## Verification
Local contract tests cover the rendered weekly SBOM task path and the REST
payload shape:
```bash
uv run pytest tests/test_integration_event_bridge.py tests/test_issue_sink.py
```
For a live environment, run with `ISSUE_SINK_TYPE=null` first and confirm
`task_spawn_log` contains the expected source id, condition, triggering event id,
and synthetic task reference. Then switch to `ISSUE_SINK_TYPE=rest` only after a
single known-safe rule match creates one issue-core task with the same fields.