Implement ACTIVITY-WP-0021 production automation reliability
Root-cause IssueSink 503 (dead Forgejo PAT on issue-core), add state-hub task sink path B, log runs before emit, harden sync_schedules, deterministic SBOM/triage reports, DB probe thrash fix, and prod automation-status helper.
This commit is contained in:
parent
1209ff6973
commit
98e8aa83bd
15 changed files with 638 additions and 63 deletions
|
|
@ -52,10 +52,40 @@ task reference before it can replace `IssueCoreRestSink`.
|
|||
`emit_tasks`, so Temporal retries and the workflow history make failures
|
||||
visible. Railiance runtime ConfigMap uses this mode once
|
||||
`ISSUE_CORE_API_KEY` is present in `actcore-runtime-secret`.
|
||||
- `ISSUE_SINK_TYPE=state-hub`: ACTIVITY-WP-0021 path B. Each TaskSpec is posted
|
||||
as a State Hub progress event (`activity_task_spawn` by default) instead of
|
||||
creating a Forgejo issue. Use when issue-core→Forgejo is down or automated
|
||||
Forgejo issues are policy-blocked.
|
||||
|
||||
Weekly SBOM staleness is the canonical promotion candidate because the rule
|
||||
contract is deterministic and tested. Promote it only after a null-sink dry-run
|
||||
review and one live `IssueCoreRestSink` smoke against the target endpoint.
|
||||
### Known production failure (2026-07-21)
|
||||
|
||||
`POST /issues/` returned **HTTP 503** with:
|
||||
|
||||
```text
|
||||
Failed to connect to backend 'forgejo-inbox': Failed to connect to Gitea API
|
||||
```
|
||||
|
||||
Root cause on coulombcore issue-core: `GITEA_BACKEND_TOKEN` is rejected by
|
||||
Forgejo (`/api/v1/user` → 401 user does not exist). Healthz stays 200.
|
||||
|
||||
**Operator fix (path A):** rotate `GITEA_BACKEND_TOKEN` in OpenBao path
|
||||
`platform/workloads/issue-core/issue-core/issue-core-runtime` (see
|
||||
`warden route show issue-core-ingestion-api-key`) using a valid Forgejo PAT for
|
||||
the issue-core service identity, then restart `issue-core` so the entrypoint
|
||||
rewrites backends.json. Smoke from the worker:
|
||||
|
||||
```bash
|
||||
# From actcore-worker (does not print secrets)
|
||||
python -c "import os,httpx; r=httpx.post(os.environ['ISSUE_CORE_URL']+'/issues/',
|
||||
json={...full TaskSpec payload...},
|
||||
headers={'Authorization':'Bearer '+os.environ['ISSUE_CORE_API_KEY']}, timeout=30);
|
||||
print(r.status_code, r.text[:200])"
|
||||
```
|
||||
|
||||
Expect **201**, not 503.
|
||||
|
||||
Weekly SBOM staleness now posts a deterministic `sbom_staleness` progress report
|
||||
even when task emission is disabled.
|
||||
|
||||
## Promotion and rollback
|
||||
|
||||
|
|
|
|||
|
|
@ -195,6 +195,49 @@ make automation-status SINCE=2026-06-26 FORMAT=json
|
|||
make automation-status SINCE=2026-06-26 UNTIL=2026-06-27 ACTCORE_DB_URL=
|
||||
```
|
||||
|
||||
### Production evidence path (railiance01)
|
||||
|
||||
The workstation `make automation-status` is **degraded** without a live
|
||||
`ACTCORE_DB_URL` / `TEMPORAL_HOST` to the railiance01 stack (docker hostnames
|
||||
in `.env` are not the production DBs). For operator questions about live
|
||||
schedules, use the prod helper (SSH to the host; no k3s API tunnel is required):
|
||||
|
||||
```bash
|
||||
# Human summary since last Sunday (Europe/Berlin window handled client-side)
|
||||
./scripts/prod_automation_status.sh sunday
|
||||
|
||||
# Explicit UTC lower bound
|
||||
./scripts/prod_automation_status.sh 2026-07-18T22:00:00+00:00
|
||||
```
|
||||
|
||||
The script SSHes to `railiance01` (see `~/.ssh/config`), queries
|
||||
`activity_runs` via `kubectl -n activity-core exec actcore-app-db-0`, and prints
|
||||
per-activity counts plus non-high-frequency fire rows. It never prints secrets.
|
||||
|
||||
Manual equivalent:
|
||||
|
||||
```bash
|
||||
ssh railiance01 'export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
|
||||
kubectl -n activity-core exec actcore-app-db-0 -- psql -U actcore -d actcore -c "
|
||||
SELECT d.name, count(*) AS runs, max(r.fired_at) AS last_fire,
|
||||
sum(r.tasks_spawned) AS tasks
|
||||
FROM activity_runs r
|
||||
JOIN activity_definitions d ON d.id = r.activity_id
|
||||
WHERE coalesce(r.scheduled_for, r.fired_at) >= timestamptz '\''2026-07-18 22:00:00+00'\''
|
||||
GROUP BY d.name ORDER BY runs DESC;"
|
||||
'
|
||||
```
|
||||
|
||||
Temporal schedule / workflow status (from the worker pod):
|
||||
|
||||
```bash
|
||||
ssh railiance01 'export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
|
||||
kubectl -n activity-core exec deploy/actcore-worker -- /app/.venv/bin/python3 -c "
|
||||
# describe ScheduleHandle for activity-schedule-<uuid>
|
||||
..."
|
||||
'
|
||||
```
|
||||
|
||||
Example distinction from the June 2026 daily triage evidence:
|
||||
|
||||
```text
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue