Root-cause IssueSink 503 (dead Forgejo PAT on issue-core), add state-hub task sink path B, log runs before emit, harden sync_schedules, deterministic SBOM/triage reports, DB probe thrash fix, and prod automation-status helper.
139 lines
5.7 KiB
Markdown
139 lines
5.7 KiB
Markdown
# Issue-Core Emission Boundary
|
|
|
|
activity-core owns the decision to spawn a task and the audit trail that says
|
|
why it spawned. It does not own downstream task lifecycle state after emission.
|
|
|
|
## Current authoritative endpoint
|
|
|
|
The current authoritative boundary is the issue-core REST API:
|
|
|
|
```text
|
|
POST {ISSUE_CORE_URL}/issues/
|
|
```
|
|
|
|
`IssueCoreRestSink` authenticates with the shared `ISSUE_CORE_API_KEY` env var
|
|
(same value as the issue-core server) via `Authorization: Bearer <key>` and
|
|
sends this payload:
|
|
|
|
```json
|
|
{
|
|
"title": "Run SBOM rescan for activity-core",
|
|
"description": "",
|
|
"target_repo": "activity-core",
|
|
"priority": "medium",
|
|
"labels": ["sbom", "security", "automated"],
|
|
"due_in_days": null,
|
|
"source_type": "rule",
|
|
"source_id": "flag-stale-sbom",
|
|
"triggering_event_id": "event-or-schedule-key",
|
|
"activity_definition_id": "activity-definition-uuid"
|
|
}
|
|
```
|
|
|
|
The expected response contains `issue_id` and may include `issue_url` and
|
|
`backend`. activity-core stores only the returned task reference in
|
|
`task_spawn_log`; issue-core remains authoritative for task status, assignment,
|
|
comments, closure, and cancellation.
|
|
|
|
## REST versus NATS
|
|
|
|
Keep REST as the active emission contract until issue-core publishes and owns a
|
|
durable NATS consumer for task-creation commands. NATS is still appropriate for
|
|
event intake into activity-core, but task creation needs an acknowledged,
|
|
idempotent command boundary. A future NATS sink must return or later correlate a
|
|
task reference before it can replace `IssueCoreRestSink`.
|
|
|
|
## Safe operating modes
|
|
|
|
- `ISSUE_SINK_TYPE=null`: dry-run/audit mode. Task specs are rendered and the
|
|
workflow records synthetic `null-*` references. Use this for contract review
|
|
and emergency rollback.
|
|
- `ISSUE_SINK_TYPE=rest`: live task creation. Sink failures raise out of
|
|
`emit_tasks`, so Temporal retries and the workflow history make failures
|
|
visible. Railiance runtime ConfigMap uses this mode once
|
|
`ISSUE_CORE_API_KEY` is present in `actcore-runtime-secret`.
|
|
- `ISSUE_SINK_TYPE=state-hub`: ACTIVITY-WP-0021 path B. Each TaskSpec is posted
|
|
as a State Hub progress event (`activity_task_spawn` by default) instead of
|
|
creating a Forgejo issue. Use when issue-core→Forgejo is down or automated
|
|
Forgejo issues are policy-blocked.
|
|
|
|
### Known production failure (2026-07-21)
|
|
|
|
`POST /issues/` returned **HTTP 503** with:
|
|
|
|
```text
|
|
Failed to connect to backend 'forgejo-inbox': Failed to connect to Gitea API
|
|
```
|
|
|
|
Root cause on coulombcore issue-core: `GITEA_BACKEND_TOKEN` is rejected by
|
|
Forgejo (`/api/v1/user` → 401 user does not exist). Healthz stays 200.
|
|
|
|
**Operator fix (path A):** rotate `GITEA_BACKEND_TOKEN` in OpenBao path
|
|
`platform/workloads/issue-core/issue-core/issue-core-runtime` (see
|
|
`warden route show issue-core-ingestion-api-key`) using a valid Forgejo PAT for
|
|
the issue-core service identity, then restart `issue-core` so the entrypoint
|
|
rewrites backends.json. Smoke from the worker:
|
|
|
|
```bash
|
|
# From actcore-worker (does not print secrets)
|
|
python -c "import os,httpx; r=httpx.post(os.environ['ISSUE_CORE_URL']+'/issues/',
|
|
json={...full TaskSpec payload...},
|
|
headers={'Authorization':'Bearer '+os.environ['ISSUE_CORE_API_KEY']}, timeout=30);
|
|
print(r.status_code, r.text[:200])"
|
|
```
|
|
|
|
Expect **201**, not 503.
|
|
|
|
Weekly SBOM staleness now posts a deterministic `sbom_staleness` progress report
|
|
even when task emission is disabled.
|
|
|
|
## Promotion and rollback
|
|
|
|
### Promote one definition safely
|
|
|
|
1. Keep `ISSUE_SINK_TYPE=null` and run or wait for the target definition.
|
|
2. Review rendered task specs in `task_spawn_log` (source id, condition,
|
|
target repo, synthetic `null-*` reference).
|
|
3. Confirm `ISSUE_CORE_URL` reachability and a populated `ISSUE_CORE_API_KEY`
|
|
on both activity-core and issue-core (same value). Credential custody:
|
|
`warden route show issue-core-ingestion-api-key --json`.
|
|
4. Run the repo smoke:
|
|
```bash
|
|
uv run python scripts/smoke_issue_core_emission.py
|
|
ISSUE_CORE_URL=http://127.0.0.1:8765 ISSUE_CORE_API_KEY=... \
|
|
uv run python scripts/smoke_issue_core_emission.py --live
|
|
```
|
|
5. Set `ISSUE_SINK_TYPE=rest` in `actcore-runtime-config`, apply
|
|
`k8s/railiance/15-externalsecret-issue-core.yaml` so External Secrets merges
|
|
`ISSUE_CORE_API_KEY` into `actcore-runtime-secret`, and restart
|
|
`actcore-worker` / `actcore-event-router` after the ExternalSecret is Ready.
|
|
6. Trigger one known-safe run (weekly SBOM staleness on a stale fixture or
|
|
manual `/activity-definitions/<id>/trigger`) and confirm `task_spawn_log`
|
|
stores the real `issue_id` returned by issue-core.
|
|
|
|
### Roll back to null-sink
|
|
|
|
1. Set `ISSUE_SINK_TYPE=null` in `actcore-runtime-config`.
|
|
2. `kubectl -n activity-core rollout restart deploy/actcore-worker deploy/actcore-event-router`
|
|
3. Verify the next run records synthetic `null-*` references again.
|
|
4. Leave issue-core tasks already created in place; activity-core does not own
|
|
downstream task lifecycle. Close or cancel duplicates in issue-core if a
|
|
promotion experiment created unexpected tasks.
|
|
|
|
Duplicate handling today: issue-core REST ingest does not yet dedupe on
|
|
`triggering_event_id`; Temporal retry visibility is the current guardrail.
|
|
Treat promotion as one-definition-at-a-time until server-side idempotency ships.
|
|
|
|
## Verification
|
|
|
|
Local contract tests cover the rendered weekly SBOM task path and the REST
|
|
payload shape:
|
|
|
|
```bash
|
|
uv run pytest tests/test_integration_event_bridge.py tests/test_issue_sink.py
|
|
```
|
|
|
|
For a live environment, run with `ISSUE_SINK_TYPE=null` first and confirm
|
|
`task_spawn_log` contains the expected source id, condition, triggering event id,
|
|
and synthetic task reference. Then switch to `ISSUE_SINK_TYPE=rest` only after a
|
|
single known-safe rule match creates one issue-core task with the same fields.
|