Implement ACTIVITY-WP-0021 production automation reliability
All checks were successful
CI Smoke / host-smoke (push) Successful in 0s
CI Smoke / container-smoke (push) Successful in 2s
Build and Publish Container Image / build-and-push (push) Successful in 47s

Root-cause IssueSink 503 (dead Forgejo PAT on issue-core), add state-hub
task sink path B, log runs before emit, harden sync_schedules, deterministic
SBOM/triage reports, DB probe thrash fix, and prod automation-status helper.
This commit is contained in:
tegwick 2026-07-21 04:21:55 +02:00
parent 1209ff6973
commit 98e8aa83bd
15 changed files with 638 additions and 63 deletions

View file

@ -52,10 +52,40 @@ task reference before it can replace `IssueCoreRestSink`.
`emit_tasks`, so Temporal retries and the workflow history make failures
visible. Railiance runtime ConfigMap uses this mode once
`ISSUE_CORE_API_KEY` is present in `actcore-runtime-secret`.
- `ISSUE_SINK_TYPE=state-hub`: ACTIVITY-WP-0021 path B. Each TaskSpec is posted
as a State Hub progress event (`activity_task_spawn` by default) instead of
creating a Forgejo issue. Use when issue-core→Forgejo is down or automated
Forgejo issues are policy-blocked.
Weekly SBOM staleness is the canonical promotion candidate because the rule
contract is deterministic and tested. Promote it only after a null-sink dry-run
review and one live `IssueCoreRestSink` smoke against the target endpoint.
### Known production failure (2026-07-21)
`POST /issues/` returned **HTTP 503** with:
```text
Failed to connect to backend 'forgejo-inbox': Failed to connect to Gitea API
```
Root cause on coulombcore issue-core: `GITEA_BACKEND_TOKEN` is rejected by
Forgejo (`/api/v1/user` → 401 user does not exist). Healthz stays 200.
**Operator fix (path A):** rotate `GITEA_BACKEND_TOKEN` in OpenBao path
`platform/workloads/issue-core/issue-core/issue-core-runtime` (see
`warden route show issue-core-ingestion-api-key`) using a valid Forgejo PAT for
the issue-core service identity, then restart `issue-core` so the entrypoint
rewrites backends.json. Smoke from the worker:
```bash
# From actcore-worker (does not print secrets)
python -c "import os,httpx; r=httpx.post(os.environ['ISSUE_CORE_URL']+'/issues/',
json={...full TaskSpec payload...},
headers={'Authorization':'Bearer '+os.environ['ISSUE_CORE_API_KEY']}, timeout=30);
print(r.status_code, r.text[:200])"
```
Expect **201**, not 503.
Weekly SBOM staleness now posts a deterministic `sbom_staleness` progress report
even when task emission is disabled.
## Promotion and rollback

View file

@ -195,6 +195,49 @@ make automation-status SINCE=2026-06-26 FORMAT=json
make automation-status SINCE=2026-06-26 UNTIL=2026-06-27 ACTCORE_DB_URL=
```
### Production evidence path (railiance01)
The workstation `make automation-status` is **degraded** without a live
`ACTCORE_DB_URL` / `TEMPORAL_HOST` to the railiance01 stack (docker hostnames
in `.env` are not the production DBs). For operator questions about live
schedules, use the prod helper (SSH to the host; no k3s API tunnel is required):
```bash
# Human summary since last Sunday (Europe/Berlin window handled client-side)
./scripts/prod_automation_status.sh sunday
# Explicit UTC lower bound
./scripts/prod_automation_status.sh 2026-07-18T22:00:00+00:00
```
The script SSHes to `railiance01` (see `~/.ssh/config`), queries
`activity_runs` via `kubectl -n activity-core exec actcore-app-db-0`, and prints
per-activity counts plus non-high-frequency fire rows. It never prints secrets.
Manual equivalent:
```bash
ssh railiance01 'export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
kubectl -n activity-core exec actcore-app-db-0 -- psql -U actcore -d actcore -c "
SELECT d.name, count(*) AS runs, max(r.fired_at) AS last_fire,
sum(r.tasks_spawned) AS tasks
FROM activity_runs r
JOIN activity_definitions d ON d.id = r.activity_id
WHERE coalesce(r.scheduled_for, r.fired_at) >= timestamptz '\''2026-07-18 22:00:00+00'\''
GROUP BY d.name ORDER BY runs DESC;"
'
```
Temporal schedule / workflow status (from the worker pod):
```bash
ssh railiance01 'export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
kubectl -n activity-core exec deploy/actcore-worker -- /app/.venv/bin/python3 -c "
# describe ScheduleHandle for activity-schedule-<uuid>
..."
'
```
Example distinction from the June 2026 daily triage evidence:
```text