Root-cause IssueSink 503 (dead Forgejo PAT on issue-core), add state-hub task sink path B, log runs before emit, harden sync_schedules, deterministic SBOM/triage reports, DB probe thrash fix, and prod automation-status helper.
5.7 KiB
Issue-Core Emission Boundary
activity-core owns the decision to spawn a task and the audit trail that says why it spawned. It does not own downstream task lifecycle state after emission.
Current authoritative endpoint
The current authoritative boundary is the issue-core REST API:
POST {ISSUE_CORE_URL}/issues/
IssueCoreRestSink authenticates with the shared ISSUE_CORE_API_KEY env var
(same value as the issue-core server) via Authorization: Bearer <key> and
sends this payload:
{
"title": "Run SBOM rescan for activity-core",
"description": "",
"target_repo": "activity-core",
"priority": "medium",
"labels": ["sbom", "security", "automated"],
"due_in_days": null,
"source_type": "rule",
"source_id": "flag-stale-sbom",
"triggering_event_id": "event-or-schedule-key",
"activity_definition_id": "activity-definition-uuid"
}
The expected response contains issue_id and may include issue_url and
backend. activity-core stores only the returned task reference in
task_spawn_log; issue-core remains authoritative for task status, assignment,
comments, closure, and cancellation.
REST versus NATS
Keep REST as the active emission contract until issue-core publishes and owns a
durable NATS consumer for task-creation commands. NATS is still appropriate for
event intake into activity-core, but task creation needs an acknowledged,
idempotent command boundary. A future NATS sink must return or later correlate a
task reference before it can replace IssueCoreRestSink.
Safe operating modes
ISSUE_SINK_TYPE=null: dry-run/audit mode. Task specs are rendered and the workflow records syntheticnull-*references. Use this for contract review and emergency rollback.ISSUE_SINK_TYPE=rest: live task creation. Sink failures raise out ofemit_tasks, so Temporal retries and the workflow history make failures visible. Railiance runtime ConfigMap uses this mode onceISSUE_CORE_API_KEYis present inactcore-runtime-secret.ISSUE_SINK_TYPE=state-hub: ACTIVITY-WP-0021 path B. Each TaskSpec is posted as a State Hub progress event (activity_task_spawnby default) instead of creating a Forgejo issue. Use when issue-core→Forgejo is down or automated Forgejo issues are policy-blocked.
Known production failure (2026-07-21)
POST /issues/ returned HTTP 503 with:
Failed to connect to backend 'forgejo-inbox': Failed to connect to Gitea API
Root cause on coulombcore issue-core: GITEA_BACKEND_TOKEN is rejected by
Forgejo (/api/v1/user → 401 user does not exist). Healthz stays 200.
Operator fix (path A): rotate GITEA_BACKEND_TOKEN in OpenBao path
platform/workloads/issue-core/issue-core/issue-core-runtime (see
warden route show issue-core-ingestion-api-key) using a valid Forgejo PAT for
the issue-core service identity, then restart issue-core so the entrypoint
rewrites backends.json. Smoke from the worker:
# From actcore-worker (does not print secrets)
python -c "import os,httpx; r=httpx.post(os.environ['ISSUE_CORE_URL']+'/issues/',
json={...full TaskSpec payload...},
headers={'Authorization':'Bearer '+os.environ['ISSUE_CORE_API_KEY']}, timeout=30);
print(r.status_code, r.text[:200])"
Expect 201, not 503.
Weekly SBOM staleness now posts a deterministic sbom_staleness progress report
even when task emission is disabled.
Promotion and rollback
Promote one definition safely
- Keep
ISSUE_SINK_TYPE=nulland run or wait for the target definition. - Review rendered task specs in
task_spawn_log(source id, condition, target repo, syntheticnull-*reference). - Confirm
ISSUE_CORE_URLreachability and a populatedISSUE_CORE_API_KEYon both activity-core and issue-core (same value). Credential custody:warden route show issue-core-ingestion-api-key --json. - Run the repo smoke:
uv run python scripts/smoke_issue_core_emission.py ISSUE_CORE_URL=http://127.0.0.1:8765 ISSUE_CORE_API_KEY=... \ uv run python scripts/smoke_issue_core_emission.py --live - Set
ISSUE_SINK_TYPE=restinactcore-runtime-config, applyk8s/railiance/15-externalsecret-issue-core.yamlso External Secrets mergesISSUE_CORE_API_KEYintoactcore-runtime-secret, and restartactcore-worker/actcore-event-routerafter the ExternalSecret is Ready. - Trigger one known-safe run (weekly SBOM staleness on a stale fixture or
manual
/activity-definitions/<id>/trigger) and confirmtask_spawn_logstores the realissue_idreturned by issue-core.
Roll back to null-sink
- Set
ISSUE_SINK_TYPE=nullinactcore-runtime-config. kubectl -n activity-core rollout restart deploy/actcore-worker deploy/actcore-event-router- Verify the next run records synthetic
null-*references again. - Leave issue-core tasks already created in place; activity-core does not own downstream task lifecycle. Close or cancel duplicates in issue-core if a promotion experiment created unexpected tasks.
Duplicate handling today: issue-core REST ingest does not yet dedupe on
triggering_event_id; Temporal retry visibility is the current guardrail.
Treat promotion as one-definition-at-a-time until server-side idempotency ships.
Verification
Local contract tests cover the rendered weekly SBOM task path and the REST payload shape:
uv run pytest tests/test_integration_event_bridge.py tests/test_issue_sink.py
For a live environment, run with ISSUE_SINK_TYPE=null first and confirm
task_spawn_log contains the expected source id, condition, triggering event id,
and synthetic task reference. Then switch to ISSUE_SINK_TYPE=rest only after a
single known-safe rule match creates one issue-core task with the same fields.