Add ACTIVITY-WP-0021 production automation reliability workplan
Captures the Sunday–Monday railiance01 schedule review: IssueSink 503s, Binky cutover failures, edge-relay/ROS issues, silent daily triage, sync_schedules bug, and DB restart thrash — with tasks to fix and verify.
This commit is contained in:
parent
114866ef1a
commit
5afd2f347d
1 changed files with 302 additions and 0 deletions
302
workplans/ACTIVITY-WP-0021-prod-automation-reliability.md
Normal file
302
workplans/ACTIVITY-WP-0021-prod-automation-reliability.md
Normal file
|
|
@ -0,0 +1,302 @@
|
|||
---
|
||||
id: ACTIVITY-WP-0021
|
||||
type: workplan
|
||||
title: "Production automation reliability after Sunday–Monday fire review"
|
||||
domain: infotech
|
||||
repo: activity-core
|
||||
status: ready
|
||||
owner: codex
|
||||
topic_slug: activity-core
|
||||
created: "2026-07-21"
|
||||
updated: "2026-07-21"
|
||||
---
|
||||
|
||||
# Production automation reliability after Sunday–Monday fire review
|
||||
|
||||
## Problem
|
||||
|
||||
Operator review of railiance01 activity-core schedules for
|
||||
**2026-07-19 00:00 → 2026-07-21 ~01:30 Europe/Berlin** found that **Temporal
|
||||
schedules fire on time**, but several high-value outcomes failed or were
|
||||
invisible:
|
||||
|
||||
| Finding | Evidence |
|
||||
| --- | --- |
|
||||
| **IssueSink emit path broken** | Binky daily, Binky mail intake, SBOM weekly, and TODO-stale workflows **FAILED** at `emit_tasks` with **HTTP 503** from `actcore-issue-core-bridge…:8765/issues/` (retries exhausted). No `activity_runs` row for failed Binky/SBOM fires. |
|
||||
| **Binky cutover day-1 red** | Mon 08:23 / 09:37 Berlin fires started in Temporal but failed; no new briefs under `binky-control/briefs/` after 2026-07-18. Mon–Wed verification window from cutover is at risk. |
|
||||
| **Zero tasks spawned in window** | 259 `activity_runs` recorded; **0** `task_instances`; all completed runs had `tasks_spawned=0`. |
|
||||
| **Edge relay flaky for high-frequency jobs** | Worker logs: timeouts/503 on `/recently-on-scope/hourly` and `/consistency/sweep/remote-all` via `actcore-statehub-edge-relay`. |
|
||||
| **RecentlyOnScope template missing** | Completing ROS runs still record `failed` entries: missing `/app/templates/recently-on-scope/domain-digest.md` in the worker/image path used by the relay or resolver. |
|
||||
| **Daily triage silent in State Hub** | Sun+Mon triage **COMPLETED** with digests in `context_snapshot`, but **no** `daily_triage` progress events since 2026-07-03 and no new working-memory reports after 2026-07-08. |
|
||||
| **`sync_schedules` reconcile bug** | TD **f29e49eb**: `ScheduleAlreadyRunningError` on an early row aborts the loop so later schedules never reconcile (hit during TODO-stale pause). |
|
||||
| **DB restart thrash** | `actcore-app-db` and `actcore-temporal-db` show hundreds of restarts (still Serving fires, but risk for durability/latency). |
|
||||
| **Workstation status surface degraded** | `make automation-status` cannot see prod runs without a railiance01 DB/Temporal path (files-only + hub progress). |
|
||||
|
||||
Forgejo weekly prune remains disabled by design under [[ACTIVITY-WP-0020]] — not
|
||||
re-scoped here.
|
||||
|
||||
## Goal
|
||||
|
||||
Restore **observable, successful scheduled outcomes** on railiance01:
|
||||
|
||||
1. Task emission works for definitions that intentionally emit (or they use a
|
||||
non-Forgejo sink aligned with current policy).
|
||||
2. Binky Mon–Wed verification can complete with clean scheduled runs.
|
||||
3. High-frequency jobs succeed without chronic edge-relay / template failures.
|
||||
4. Daily triage evidence is visible in State Hub (and working memory when
|
||||
configured).
|
||||
5. Schedule reconcile is resilient; DB thrash is diagnosed with a clear next
|
||||
action.
|
||||
6. Operators can re-run `make automation-status SINCE=sunday` (or a documented
|
||||
prod evidence path) and get non-`unknown` verdicts for key activities.
|
||||
|
||||
## Out of scope
|
||||
|
||||
- Finishing Forgejo package prune enablement ([[ACTIVITY-WP-0020]]).
|
||||
- Re-enabling Daily TODO.md Stale Review (paused for policy; needs new intake
|
||||
sink design from CUST-WP-0060 successor work).
|
||||
- New activity definitions unrelated to the reliability gaps above.
|
||||
|
||||
## Dependencies
|
||||
|
||||
- railiance01 SSH + in-cluster kubectl (no k3s tunnel; host path as in WP-0014).
|
||||
- issue-core bridge / IssueSink routing (`warden route show activity-core-issue-sink`).
|
||||
- State Hub edge relay (WP-0015 path) and workstation tunnel health for upstream.
|
||||
- agent-harness / binky-control only for Binky verification evidence.
|
||||
|
||||
## Tasks
|
||||
|
||||
## Task: Root-cause and fix IssueSink / issue-core bridge 503
|
||||
|
||||
```task
|
||||
id: ACTIVITY-WP-0021-T01
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
**Why first:** Shared failure mode for Binky daily, Binky mail, SBOM weekly, and
|
||||
any definition whose rules call `emit_tasks` → IssueSink.
|
||||
|
||||
1. Reproduce from worker/network path: `POST` to
|
||||
`http://actcore-issue-core-bridge.activity-core.svc.cluster.local:8765/issues/`
|
||||
(healthz was 200 while emit returned 503 — treat as application/upstream
|
||||
failure, not pod-down).
|
||||
2. Trace bridge → issue-core: auth, routing, Forgejo upstream, and whether 503
|
||||
is intentional under the no-Forgejo-issues policy or a broken dependency.
|
||||
3. Choose and implement a fix path:
|
||||
- **A.** Restore a healthy emit path if IssueSink remains canonical, or
|
||||
- **B.** Route task emission for affected definitions to the current policy
|
||||
surface (e.g. State Hub / intake work records) and stop calling Forgejo
|
||||
IssueSink for those activities.
|
||||
4. Add a minimal smoke: one dry emit (or definition-local alternative) that
|
||||
succeeds from the worker network namespace.
|
||||
5. Record non-secret evidence (status codes, chosen path A/B, commit SHAs).
|
||||
|
||||
**Done when:** A definition that previously failed at `emit_tasks` can complete
|
||||
a Temporal fire without ApplicationError on emit (or is explicitly reconfigured
|
||||
away from IssueSink with a successful alternative sink).
|
||||
|
||||
## Task: Recover Binky cutover verification (daily + mail)
|
||||
|
||||
```task
|
||||
id: ACTIVITY-WP-0021-T02
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
Depends on T01 if Binky still uses IssueSink/task emission.
|
||||
|
||||
1. After emit path is fixed (or Binky definitions use harness-safe sinks),
|
||||
confirm Temporal schedules for:
|
||||
- `activity-schedule-ecdfadd3-…` (Binky Daily, `23 8 * * 1-5` Europe/Berlin)
|
||||
- `activity-schedule-878fbb78-…` (Binky mail, `37 9 * * 1` Europe/Berlin)
|
||||
2. Accept either the next scheduled business-day fires **or** a controlled
|
||||
one-shot trigger with the same definition versions as prod.
|
||||
3. Evidence required:
|
||||
- Temporal workflow **COMPLETED**
|
||||
- `activity_runs` row present
|
||||
- Expected Binky side-effect (brief/commit/harness task) **or** explicit
|
||||
documented no-op with `tasks_spawned` and context snapshot explaining why
|
||||
4. Update cutover notes / BINKY verification if needed so Mon–Wed gate is
|
||||
honest about remaining days.
|
||||
|
||||
**Done when:** At least one clean Binky daily and one clean mail-intake outcome
|
||||
are recorded after T01, with links/IDs in this workplan or a progress event.
|
||||
|
||||
## Task: Fix Weekly SBOM emit path (or sink policy)
|
||||
|
||||
```task
|
||||
id: ACTIVITY-WP-0021-T03
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
Mon 2026-07-20 07:00 UTC SBOM fire **FAILED** after resolver work when
|
||||
`flag-stale-sbom` hit issue-core bridge 503. Last successful task-spawning SBOM
|
||||
run was 2026-07-06 (71 tasks).
|
||||
|
||||
1. Reuse T01 emit/sink decision.
|
||||
2. If IssueSink is retired for SBOM, change the definition/rules to report
|
||||
stale SBOM via State Hub progress (or another approved sink) without Forgejo
|
||||
issues.
|
||||
3. Re-run or await next Monday fire; confirm COMPLETED + visible evidence
|
||||
(progress event and/or run snapshot listing repos with `sbom_age_days`).
|
||||
|
||||
**Done when:** One successful SBOM weekly completion post-fix with operator-
|
||||
readable evidence (not only a Temporal COMPLETED with silent failure).
|
||||
|
||||
## Task: Stabilize State Hub edge relay + RecentlyOnScope template
|
||||
|
||||
```task
|
||||
id: ACTIVITY-WP-0021-T04
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
1. **Template gap:** locate or add
|
||||
`/app/templates/recently-on-scope/domain-digest.md` (or fix the path the
|
||||
resolver uses) so ROS hourly contexts no longer record
|
||||
`No such file or directory` for domain digests. Ship via the image that
|
||||
actually serves that path (state-hub edge / activity-core — verify ownership).
|
||||
2. **Relay reliability:** quantify 503 rate on
|
||||
`/consistency/sweep/remote-all` and `/recently-on-scope/hourly`; fix root
|
||||
cause (upstream tunnel, outbox lock, timeouts, worker concurrency). Prefer
|
||||
durable queue/retry over silent workflow failure.
|
||||
3. Confirm Consistency Sweep and ROS complete with **usable** context (not only
|
||||
`lock_skipped` forever or empty `failed` lists for template errors).
|
||||
|
||||
**Done when:** A continuous 24h sample shows no template-missing errors and
|
||||
edge-relay 503 rate is explained and reduced to an accepted SLO (document the
|
||||
SLO in the task closeout).
|
||||
|
||||
## Task: Restore daily triage evidence visibility
|
||||
|
||||
```task
|
||||
id: ACTIVITY-WP-0021-T05
|
||||
status: todo
|
||||
priority: medium
|
||||
```
|
||||
|
||||
Sun+Mon 2026-07-19/20 triage runs **completed** (`run_id` present,
|
||||
`tasks_spawned=0`) with digests in `context_snapshot`, but:
|
||||
|
||||
- State Hub has **no** `event_type=daily_triage` since 2026-07-03
|
||||
- working memory last triage note is 2026-07-08
|
||||
|
||||
1. Trace report/sink instructions for Daily State Hub WSJF Triage in prod
|
||||
(ConfigMap/external defs vs repo files).
|
||||
2. Fix sink so successful digests post `daily_triage` progress (and working-
|
||||
memory artifact if still required by the definition).
|
||||
3. Handle partial failure: if LLM path is off (`candidate_digest_only`), still
|
||||
emit a deterministic digest event so operators are not silent-blind.
|
||||
4. Verify with next morning fire or one-shot trigger.
|
||||
|
||||
**Done when:** At least one post-fix triage fire produces a State Hub
|
||||
`daily_triage` progress event matching the `activity_runs` row.
|
||||
|
||||
## Task: Harden sync_schedules against ScheduleAlreadyRunningError
|
||||
|
||||
```task
|
||||
id: ACTIVITY-WP-0021-T06
|
||||
status: todo
|
||||
priority: medium
|
||||
```
|
||||
|
||||
Tracks TD **f29e49eb** (registered 2026-07-20): reconcile aborts mid-loop when
|
||||
one schedule is already running, so later definitions never pause/update.
|
||||
|
||||
1. Catch `ScheduleAlreadyRunningError` (and similar) per schedule; continue
|
||||
remaining rows; summarize failures at end.
|
||||
2. Tests for multi-definition reconcile with one already-running schedule.
|
||||
3. Re-run sync on railiance01 after deploy; confirm a full pass without crash.
|
||||
|
||||
**Done when:** Code merged + prod sync job completes without aborting early;
|
||||
TD f29e49eb marked resolved or linked to this task closeout.
|
||||
|
||||
## Task: Diagnose app-db and temporal-db restart thrash
|
||||
|
||||
```task
|
||||
id: ACTIVITY-WP-0021-T07
|
||||
status: todo
|
||||
priority: medium
|
||||
```
|
||||
|
||||
Observed on 2026-07-21: `actcore-app-db` ~384 restarts, `actcore-temporal-db`
|
||||
~500 restarts over long age — pods still Ready but thrash is a durability risk.
|
||||
|
||||
1. Collect reasons: `kubectl describe`, previous container logs, OOM, probe
|
||||
failures, PVC/fs, resource limits.
|
||||
2. Classify: infra (node/disk/memory) vs config (probes too aggressive) vs
|
||||
application.
|
||||
3. Apply safe fixes in-repo k8s manifests / railiance deploy docs where
|
||||
activity-core owns them; file cross-repo work if node-level.
|
||||
4. Record baseline restart rate before/after (24h).
|
||||
|
||||
**Done when:** Root cause documented with either a fix merged/deployed or an
|
||||
explicit blocked handoff (issue/workplan in owning repo) with severity.
|
||||
|
||||
## Task: Production evidence path for automation-status
|
||||
|
||||
```task
|
||||
id: ACTIVITY-WP-0021-T08
|
||||
status: todo
|
||||
priority: low
|
||||
```
|
||||
|
||||
Workstation `make automation-status SINCE=sunday` reported all key activities
|
||||
`unknown` because prod DB/Temporal are not on the laptop.
|
||||
|
||||
1. Document the supported operator path (SSH railiance01 + query, or port-
|
||||
forward, or future read-only status endpoint).
|
||||
2. Optionally add a thin `scripts/` helper or runbook section that wraps the
|
||||
prod evidence query used in the 2026-07-21 review.
|
||||
3. Do **not** put secrets in git; use existing host access patterns.
|
||||
|
||||
**Done when:** `docs/runbook.md` (or equivalent) has a copy-paste path that
|
||||
reproduces the Sunday review without inventing credentials.
|
||||
|
||||
## Task: Acceptance window — clean status after fixes
|
||||
|
||||
```task
|
||||
id: ACTIVITY-WP-0021-T09
|
||||
status: todo
|
||||
priority: medium
|
||||
```
|
||||
|
||||
After T01–T06 (and T07 if unblocked):
|
||||
|
||||
1. Run a bounded window status (prod evidence path from T08).
|
||||
2. Confirm for a 24–48h window:
|
||||
- No FAILED Temporal fires for Binky daily (business day), SBOM if due,
|
||||
triage, legacy-meter 8h/weekly
|
||||
- Edge ROS/consistency not dominated by template/503 failures
|
||||
- At least one fresh `daily_triage` progress event if triage fired
|
||||
3. Paste summary into this workplan closeout and log a State Hub progress
|
||||
event on this workplan.
|
||||
|
||||
**Done when:** Operator can answer "how did automations go?" with completed/
|
||||
failed verdicts backed by prod data, not all-`unknown`.
|
||||
|
||||
## Success criteria
|
||||
|
||||
- [ ] Emit/sink path fixed or intentionally replaced; Binky + SBOM no longer
|
||||
fail solely on issue-core 503
|
||||
- [ ] At least one clean Binky daily + mail-intake post-fix
|
||||
- [ ] ROS template present; edge-relay failure rate reduced and documented
|
||||
- [ ] Daily triage visible again in State Hub progress
|
||||
- [ ] `sync_schedules` continues past already-running schedules
|
||||
- [ ] DB restart thrash root-caused with fix or handoff
|
||||
- [ ] Runbook/status path for prod evidence documented
|
||||
- [ ] Acceptance window green enough to close this workplan
|
||||
|
||||
## Evidence anchor (2026-07-21 review)
|
||||
|
||||
- Window: 2026-07-19T00:00+02:00 → ~2026-07-21T01:30+02:00
|
||||
- Prod: `activity-core` ns on railiance01; API health ok; 18 cron defs
|
||||
- 259 `activity_runs`; 0 `task_instances` in window
|
||||
- Temporal: Binky daily/mail, SBOM Mon, TODO Mon **status=FAILED**; triage,
|
||||
legacy weekly, core-hub daily **COMPLETED**
|
||||
- Worker: `emit_tasks` → issue-core bridge 503; ROS/consistency edge-relay
|
||||
timeouts/503; ROS missing `domain-digest.md`
|
||||
- Related TD: f29e49eb (`sync_schedules` ScheduleAlreadyRunningError)
|
||||
- Related active work: [[ACTIVITY-WP-0020]] (Forgejo prune still disabled)
|
||||
Loading…
Add table
Add a link
Reference in a new issue