fix(ops): surface pre-record schedule failures
All checks were successful
CI Smoke / host-smoke (push) Successful in 0s
CI Smoke / container-smoke (push) Successful in 1s
Build and Publish Container Image / build-and-push (push) Successful in 26s

Assistant: codex
Assistant-Model: gpt-5.6-sol
Assistant-Session: 01a06bfe-2a55-7ed3-bacd-879977b099bf
This commit is contained in:
tegwick 2026-09-04 21:09:35 +02:00
parent fc38ca0fdf
commit 9394c40e3a
5 changed files with 45 additions and 3 deletions

View file

@ -368,9 +368,12 @@ schedules, use the prod helper (SSH to the host; no k3s API tunnel is required):
./scripts/prod_automation_status.sh 2026-07-18T22:00:00+00:00
```
The script SSHes to `railiance01` (see `~/.ssh/config`), queries
`activity_runs` via `kubectl -n activity-core exec actcore-app-db-0`, and prints
per-activity counts plus non-high-frequency fire rows. It never prints secrets.
The script SSHes to `railiance01` (see `~/.ssh/config`), runs the repository's
deterministic status engine inside the worker, queries `activity_runs` via
`kubectl -n activity-core exec actcore-app-db-0`, and prints per-activity counts
plus non-high-frequency fire rows. The deterministic verdict catches an
expected schedule fire that failed before `activity_runs` was written instead
of letting it disappear from the SQL-only table. It never prints secrets.
Manual equivalent:

View file

@ -69,6 +69,18 @@ ssh -o BatchMode=yes -o ConnectTimeout=15 "${SSH_HOST}" "export KUBECONFIG=/etc/
echo '--- API health ---'
kubectl -n ${NS} exec deploy/actcore-api -- /app/.venv/bin/python3 -c 'import urllib.request; print(urllib.request.urlopen(\"http://127.0.0.1:8010/health\").read().decode())'
echo
echo '--- deterministic schedule verdicts ---'
# Run the repository-owned status engine where DB, Temporal, State Hub, and
# working-memory sources are reachable. Exit 1 means the report found a
# missed/failed automation and is evidence, not a transport failure; continue
# with the bounded SQL detail below. Exit >1 means the report itself failed.
status_rc=0
kubectl -n ${NS} exec deploy/actcore-worker -- /app/.venv/bin/python3 -m activity_core.automation_status \
--since '${SINCE_UTC}' --timezone UTC --format human || status_rc=\$?
if [ \"\$status_rc\" -gt 1 ]; then
exit \"\$status_rc\"
fi
echo
echo '--- runs by activity ---'
kubectl -n ${NS} exec actcore-app-db-0 -- psql -U actcore -d actcore -c \"
SELECT d.name, d.enabled, count(*) AS runs,

View file

@ -559,6 +559,10 @@ async def build_report(args: argparse.Namespace) -> tuple[dict[str, Any], int]:
all_evidence = progress_evidence + wm_evidence + spawn_evidence
definitions = add_evidence_only_definitions(definitions, all_evidence)
# The first filter limits DB work to selected definitions. Evidence-only
# synthetic definitions are added afterwards, so reapply it to prevent an
# activity-specific report from leaking unrelated State Hub evidence rows.
definitions = filter_definitions(definitions, args.activity_id, args.activity_name)
activities = []
runs_available = sources["activity_runs"].get("status") == "ok"
for definition in definitions:

View file

@ -99,6 +99,20 @@ def test_last_run_picks_newest_fired_at() -> None:
assert summary["tasks_spawned"] == 2
def test_activity_filter_excludes_unrelated_evidence_only_definitions() -> None:
evidence = [
{
"activity_id": "00000000-0000-0000-0000-000000000999",
"source": "state_hub_progress",
}
]
augmented = status.add_evidence_only_definitions([_definition()], evidence)
selected = status.filter_definitions(augmented, [], ["Daily Check"])
assert [item["id"] for item in selected] == [ACTIVITY_ID]
def test_validation_failure_wins_over_completed_run() -> None:
run = {"run_id": "run-1", "activity_id": ACTIVITY_ID, "scheduled_for": None, "fired_at": "2026-06-26T07:00:10+00:00"}

View file

@ -347,6 +347,15 @@ Depends on T02T07 as applicable.
- Temporal reports the schedule enabled and unpaused. T08 remains `progress`
until the next natural apply fire (2026-09-06 03:30 UTC) produces successful
State Hub evidence; the schedule was not manually fired.
- Hardened the production status helper so it runs the deterministic
multi-source schedule verdict before its SQL detail. Live validation for the
window since 2026-08-30 now reports Weekly Forgejo Package Prune as `missed`
(`expected=1`, `runs=0`) instead of omitting a failure that occurred before an
`activity_runs` row was written. The helper treats a findings exit code as
evidence and still prints its remaining bounded diagnostics. The full source
suite passes (`494 passed, 1 skipped`), and an activity-filter regression test
prevents unrelated evidence-only definitions from leaking into targeted
reports.
- State Hub message `868a2326-cdad-4256-be4c-7cd6c8ec4b5b` asks
railiance-platform to move the generated protection inventory onto a durable
checkout-independent path and refresh it through the platform image-rollout