fix(ops): surface pre-record schedule failures
Assistant: codex Assistant-Model: gpt-5.6-sol Assistant-Session: 01a06bfe-2a55-7ed3-bacd-879977b099bf
This commit is contained in:
parent
fc38ca0fdf
commit
9394c40e3a
5 changed files with 45 additions and 3 deletions
|
|
@ -368,9 +368,12 @@ schedules, use the prod helper (SSH to the host; no k3s API tunnel is required):
|
|||
./scripts/prod_automation_status.sh 2026-07-18T22:00:00+00:00
|
||||
```
|
||||
|
||||
The script SSHes to `railiance01` (see `~/.ssh/config`), queries
|
||||
`activity_runs` via `kubectl -n activity-core exec actcore-app-db-0`, and prints
|
||||
per-activity counts plus non-high-frequency fire rows. It never prints secrets.
|
||||
The script SSHes to `railiance01` (see `~/.ssh/config`), runs the repository's
|
||||
deterministic status engine inside the worker, queries `activity_runs` via
|
||||
`kubectl -n activity-core exec actcore-app-db-0`, and prints per-activity counts
|
||||
plus non-high-frequency fire rows. The deterministic verdict catches an
|
||||
expected schedule fire that failed before `activity_runs` was written instead
|
||||
of letting it disappear from the SQL-only table. It never prints secrets.
|
||||
|
||||
Manual equivalent:
|
||||
|
||||
|
|
|
|||
|
|
@ -69,6 +69,18 @@ ssh -o BatchMode=yes -o ConnectTimeout=15 "${SSH_HOST}" "export KUBECONFIG=/etc/
|
|||
echo '--- API health ---'
|
||||
kubectl -n ${NS} exec deploy/actcore-api -- /app/.venv/bin/python3 -c 'import urllib.request; print(urllib.request.urlopen(\"http://127.0.0.1:8010/health\").read().decode())'
|
||||
echo
|
||||
echo '--- deterministic schedule verdicts ---'
|
||||
# Run the repository-owned status engine where DB, Temporal, State Hub, and
|
||||
# working-memory sources are reachable. Exit 1 means the report found a
|
||||
# missed/failed automation and is evidence, not a transport failure; continue
|
||||
# with the bounded SQL detail below. Exit >1 means the report itself failed.
|
||||
status_rc=0
|
||||
kubectl -n ${NS} exec deploy/actcore-worker -- /app/.venv/bin/python3 -m activity_core.automation_status \
|
||||
--since '${SINCE_UTC}' --timezone UTC --format human || status_rc=\$?
|
||||
if [ \"\$status_rc\" -gt 1 ]; then
|
||||
exit \"\$status_rc\"
|
||||
fi
|
||||
echo
|
||||
echo '--- runs by activity ---'
|
||||
kubectl -n ${NS} exec actcore-app-db-0 -- psql -U actcore -d actcore -c \"
|
||||
SELECT d.name, d.enabled, count(*) AS runs,
|
||||
|
|
|
|||
|
|
@ -559,6 +559,10 @@ async def build_report(args: argparse.Namespace) -> tuple[dict[str, Any], int]:
|
|||
|
||||
all_evidence = progress_evidence + wm_evidence + spawn_evidence
|
||||
definitions = add_evidence_only_definitions(definitions, all_evidence)
|
||||
# The first filter limits DB work to selected definitions. Evidence-only
|
||||
# synthetic definitions are added afterwards, so reapply it to prevent an
|
||||
# activity-specific report from leaking unrelated State Hub evidence rows.
|
||||
definitions = filter_definitions(definitions, args.activity_id, args.activity_name)
|
||||
activities = []
|
||||
runs_available = sources["activity_runs"].get("status") == "ok"
|
||||
for definition in definitions:
|
||||
|
|
|
|||
|
|
@ -99,6 +99,20 @@ def test_last_run_picks_newest_fired_at() -> None:
|
|||
assert summary["tasks_spawned"] == 2
|
||||
|
||||
|
||||
def test_activity_filter_excludes_unrelated_evidence_only_definitions() -> None:
|
||||
evidence = [
|
||||
{
|
||||
"activity_id": "00000000-0000-0000-0000-000000000999",
|
||||
"source": "state_hub_progress",
|
||||
}
|
||||
]
|
||||
|
||||
augmented = status.add_evidence_only_definitions([_definition()], evidence)
|
||||
selected = status.filter_definitions(augmented, [], ["Daily Check"])
|
||||
|
||||
assert [item["id"] for item in selected] == [ACTIVITY_ID]
|
||||
|
||||
|
||||
def test_validation_failure_wins_over_completed_run() -> None:
|
||||
run = {"run_id": "run-1", "activity_id": ACTIVITY_ID, "scheduled_for": None, "fired_at": "2026-06-26T07:00:10+00:00"}
|
||||
|
||||
|
|
|
|||
|
|
@ -347,6 +347,15 @@ Depends on T02–T07 as applicable.
|
|||
- Temporal reports the schedule enabled and unpaused. T08 remains `progress`
|
||||
until the next natural apply fire (2026-09-06 03:30 UTC) produces successful
|
||||
State Hub evidence; the schedule was not manually fired.
|
||||
- Hardened the production status helper so it runs the deterministic
|
||||
multi-source schedule verdict before its SQL detail. Live validation for the
|
||||
window since 2026-08-30 now reports Weekly Forgejo Package Prune as `missed`
|
||||
(`expected=1`, `runs=0`) instead of omitting a failure that occurred before an
|
||||
`activity_runs` row was written. The helper treats a findings exit code as
|
||||
evidence and still prints its remaining bounded diagnostics. The full source
|
||||
suite passes (`494 passed, 1 skipped`), and an activity-filter regression test
|
||||
prevents unrelated evidence-only definitions from leaking into targeted
|
||||
reports.
|
||||
- State Hub message `868a2326-cdad-4256-be4c-7cd6c8ec4b5b` asks
|
||||
railiance-platform to move the generated protection inventory onto a durable
|
||||
checkout-independent path and refresh it through the platform image-rollout
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue