coulomb/activity-core
4
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Implement ACTIVITY-WP-0021 production automation reliability
All checks were successful
CI Smoke / host-smoke (push) Successful in 0s
Details
CI Smoke / container-smoke (push) Successful in 2s
Details
Build and Publish Container Image / build-and-push (push) Successful in 47s
Details

Browse source 
Root-cause IssueSink 503 (dead Forgejo PAT on issue-core), add state-hub
task sink path B, log runs before emit, harden sync_schedules, deterministic
SBOM/triage reports, DB probe thrash fix, and prod automation-status helper.
This commit is contained in:
tegwick 2026-07-21 04:21:55 +02:00
parent 1209ff6973
commit 98e8aa83bd
15 changed files with 638 additions and 63 deletions
Download patch file Download diff file Expand all files Collapse all files

43
docs/runbook.md
View file

@ -195,6 +195,49 @@ make automation-status SINCE=2026-06-26 FORMAT=json
make automation-status SINCE=2026-06-26 UNTIL=2026-06-27 ACTCORE_DB_URL=
```
### Production evidence path (railiance01)
The workstation `make automation-status` is **degraded** without a live
`ACTCORE_DB_URL` / `TEMPORAL_HOST` to the railiance01 stack (docker hostnames
in `.env` are not the production DBs). For operator questions about live
schedules, use the prod helper (SSH to the host; no k3s API tunnel is required):
```bash
# Human summary since last Sunday (Europe/Berlin window handled client-side)
./scripts/prod_automation_status.sh sunday
# Explicit UTC lower bound
./scripts/prod_automation_status.sh 2026-07-18T22:00:00+00:00
```
The script SSHes to `railiance01` (see `~/.ssh/config`), queries
`activity_runs` via `kubectl -n activity-core exec actcore-app-db-0`, and prints
per-activity counts plus non-high-frequency fire rows. It never prints secrets.
Manual equivalent:
```bash
ssh railiance01 'export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
kubectl -n activity-core exec actcore-app-db-0 -- psql -U actcore -d actcore -c "
SELECT d.name, count(*) AS runs, max(r.fired_at) AS last_fire,
sum(r.tasks_spawned) AS tasks
FROM activity_runs r
JOIN activity_definitions d ON d.id = r.activity_id
WHERE coalesce(r.scheduled_for, r.fired_at) >= timestamptz '\''2026-07-18 22:00:00+00'\''
GROUP BY d.name ORDER BY runs DESC;"
'
```
Temporal schedule / workflow status (from the worker pod):
```bash
ssh railiance01 'export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
kubectl -n activity-core exec deploy/actcore-worker -- /app/.venv/bin/python3 -c "
# describe ScheduleHandle for activity-schedule-<uuid>
..."
'
```
Example distinction from the June 2026 daily triage evidence:
```text