activity-core/docs/issue-core-emission-boundary.md
tegwick 98e8aa83bd
All checks were successful
CI Smoke / host-smoke (push) Successful in 0s
CI Smoke / container-smoke (push) Successful in 2s
Build and Publish Container Image / build-and-push (push) Successful in 47s
Implement ACTIVITY-WP-0021 production automation reliability
Root-cause IssueSink 503 (dead Forgejo PAT on issue-core), add state-hub
task sink path B, log runs before emit, harden sync_schedules, deterministic
SBOM/triage reports, DB probe thrash fix, and prod automation-status helper.
2026-07-21 04:21:55 +02:00

5.7 KiB

Issue-Core Emission Boundary

activity-core owns the decision to spawn a task and the audit trail that says why it spawned. It does not own downstream task lifecycle state after emission.

Current authoritative endpoint

The current authoritative boundary is the issue-core REST API:

POST {ISSUE_CORE_URL}/issues/

IssueCoreRestSink authenticates with the shared ISSUE_CORE_API_KEY env var (same value as the issue-core server) via Authorization: Bearer <key> and sends this payload:

{
  "title": "Run SBOM rescan for activity-core",
  "description": "",
  "target_repo": "activity-core",
  "priority": "medium",
  "labels": ["sbom", "security", "automated"],
  "due_in_days": null,
  "source_type": "rule",
  "source_id": "flag-stale-sbom",
  "triggering_event_id": "event-or-schedule-key",
  "activity_definition_id": "activity-definition-uuid"
}

The expected response contains issue_id and may include issue_url and backend. activity-core stores only the returned task reference in task_spawn_log; issue-core remains authoritative for task status, assignment, comments, closure, and cancellation.

REST versus NATS

Keep REST as the active emission contract until issue-core publishes and owns a durable NATS consumer for task-creation commands. NATS is still appropriate for event intake into activity-core, but task creation needs an acknowledged, idempotent command boundary. A future NATS sink must return or later correlate a task reference before it can replace IssueCoreRestSink.

Safe operating modes

  • ISSUE_SINK_TYPE=null: dry-run/audit mode. Task specs are rendered and the workflow records synthetic null-* references. Use this for contract review and emergency rollback.
  • ISSUE_SINK_TYPE=rest: live task creation. Sink failures raise out of emit_tasks, so Temporal retries and the workflow history make failures visible. Railiance runtime ConfigMap uses this mode once ISSUE_CORE_API_KEY is present in actcore-runtime-secret.
  • ISSUE_SINK_TYPE=state-hub: ACTIVITY-WP-0021 path B. Each TaskSpec is posted as a State Hub progress event (activity_task_spawn by default) instead of creating a Forgejo issue. Use when issue-core→Forgejo is down or automated Forgejo issues are policy-blocked.

Known production failure (2026-07-21)

POST /issues/ returned HTTP 503 with:

Failed to connect to backend 'forgejo-inbox': Failed to connect to Gitea API

Root cause on coulombcore issue-core: GITEA_BACKEND_TOKEN is rejected by Forgejo (/api/v1/user → 401 user does not exist). Healthz stays 200.

Operator fix (path A): rotate GITEA_BACKEND_TOKEN in OpenBao path platform/workloads/issue-core/issue-core/issue-core-runtime (see warden route show issue-core-ingestion-api-key) using a valid Forgejo PAT for the issue-core service identity, then restart issue-core so the entrypoint rewrites backends.json. Smoke from the worker:

# From actcore-worker (does not print secrets)
python -c "import os,httpx; r=httpx.post(os.environ['ISSUE_CORE_URL']+'/issues/',
  json={...full TaskSpec payload...},
  headers={'Authorization':'Bearer '+os.environ['ISSUE_CORE_API_KEY']}, timeout=30);
  print(r.status_code, r.text[:200])"

Expect 201, not 503.

Weekly SBOM staleness now posts a deterministic sbom_staleness progress report even when task emission is disabled.

Promotion and rollback

Promote one definition safely

  1. Keep ISSUE_SINK_TYPE=null and run or wait for the target definition.
  2. Review rendered task specs in task_spawn_log (source id, condition, target repo, synthetic null-* reference).
  3. Confirm ISSUE_CORE_URL reachability and a populated ISSUE_CORE_API_KEY on both activity-core and issue-core (same value). Credential custody: warden route show issue-core-ingestion-api-key --json.
  4. Run the repo smoke:
    uv run python scripts/smoke_issue_core_emission.py
    ISSUE_CORE_URL=http://127.0.0.1:8765 ISSUE_CORE_API_KEY=... \
      uv run python scripts/smoke_issue_core_emission.py --live
    
  5. Set ISSUE_SINK_TYPE=rest in actcore-runtime-config, apply k8s/railiance/15-externalsecret-issue-core.yaml so External Secrets merges ISSUE_CORE_API_KEY into actcore-runtime-secret, and restart actcore-worker / actcore-event-router after the ExternalSecret is Ready.
  6. Trigger one known-safe run (weekly SBOM staleness on a stale fixture or manual /activity-definitions/<id>/trigger) and confirm task_spawn_log stores the real issue_id returned by issue-core.

Roll back to null-sink

  1. Set ISSUE_SINK_TYPE=null in actcore-runtime-config.
  2. kubectl -n activity-core rollout restart deploy/actcore-worker deploy/actcore-event-router
  3. Verify the next run records synthetic null-* references again.
  4. Leave issue-core tasks already created in place; activity-core does not own downstream task lifecycle. Close or cancel duplicates in issue-core if a promotion experiment created unexpected tasks.

Duplicate handling today: issue-core REST ingest does not yet dedupe on triggering_event_id; Temporal retry visibility is the current guardrail. Treat promotion as one-definition-at-a-time until server-side idempotency ships.

Verification

Local contract tests cover the rendered weekly SBOM task path and the REST payload shape:

uv run pytest tests/test_integration_event_bridge.py tests/test_issue_sink.py

For a live environment, run with ISSUE_SINK_TYPE=null first and confirm task_spawn_log contains the expected source id, condition, triggering event id, and synthetic task reference. Then switch to ISSUE_SINK_TYPE=rest only after a single known-safe rule match creates one issue-core task with the same fields.