Implement ACTIVITY-WP-0021 production automation reliability
All checks were successful
CI Smoke / host-smoke (push) Successful in 0s
CI Smoke / container-smoke (push) Successful in 2s
Build and Publish Container Image / build-and-push (push) Successful in 47s

Root-cause IssueSink 503 (dead Forgejo PAT on issue-core), add state-hub
task sink path B, log runs before emit, harden sync_schedules, deterministic
SBOM/triage reports, DB probe thrash fix, and prod automation-status helper.
This commit is contained in:
tegwick 2026-07-21 04:21:55 +02:00
parent 1209ff6973
commit 98e8aa83bd
15 changed files with 638 additions and 63 deletions

View file

@ -52,10 +52,40 @@ task reference before it can replace `IssueCoreRestSink`.
`emit_tasks`, so Temporal retries and the workflow history make failures
visible. Railiance runtime ConfigMap uses this mode once
`ISSUE_CORE_API_KEY` is present in `actcore-runtime-secret`.
- `ISSUE_SINK_TYPE=state-hub`: ACTIVITY-WP-0021 path B. Each TaskSpec is posted
as a State Hub progress event (`activity_task_spawn` by default) instead of
creating a Forgejo issue. Use when issue-core→Forgejo is down or automated
Forgejo issues are policy-blocked.
Weekly SBOM staleness is the canonical promotion candidate because the rule
contract is deterministic and tested. Promote it only after a null-sink dry-run
review and one live `IssueCoreRestSink` smoke against the target endpoint.
### Known production failure (2026-07-21)
`POST /issues/` returned **HTTP 503** with:
```text
Failed to connect to backend 'forgejo-inbox': Failed to connect to Gitea API
```
Root cause on coulombcore issue-core: `GITEA_BACKEND_TOKEN` is rejected by
Forgejo (`/api/v1/user` → 401 user does not exist). Healthz stays 200.
**Operator fix (path A):** rotate `GITEA_BACKEND_TOKEN` in OpenBao path
`platform/workloads/issue-core/issue-core/issue-core-runtime` (see
`warden route show issue-core-ingestion-api-key`) using a valid Forgejo PAT for
the issue-core service identity, then restart `issue-core` so the entrypoint
rewrites backends.json. Smoke from the worker:
```bash
# From actcore-worker (does not print secrets)
python -c "import os,httpx; r=httpx.post(os.environ['ISSUE_CORE_URL']+'/issues/',
json={...full TaskSpec payload...},
headers={'Authorization':'Bearer '+os.environ['ISSUE_CORE_API_KEY']}, timeout=30);
print(r.status_code, r.text[:200])"
```
Expect **201**, not 503.
Weekly SBOM staleness now posts a deterministic `sbom_staleness` progress report
even when task emission is disabled.
## Promotion and rollback