activity-core/workplans/ACTIVITY-WP-0021-prod-automation-reliability.md

---
id: ACTIVITY-WP-0021
type: workplan
title: "Production automation reliability after Sunday–Monday fire review"
domain: infotech
repo: activity-core
status: finished
owner: codex
topic_slug: activity-core
created: "2026-07-21"
updated: "2026-07-21"
state_hub_workstream_id: "0459be51-9636-53f4-8560-c99609b13fad"
---

# Production automation reliability after Sunday–Monday fire review

## Problem

Operator review of railiance01 activity-core schedules for
**2026-07-19 00:00 → 2026-07-21 ~01:30 Europe/Berlin** found that **Temporal
schedules fire on time**, but several high-value outcomes failed or were
invisible:

| Finding | Evidence |
| --- | --- |
| **IssueSink emit path broken** | Binky daily, Binky mail intake, SBOM weekly, and TODO-stale workflows **FAILED** at `emit_tasks` with **HTTP 503** from `actcore-issue-core-bridge…:8765/issues/` (retries exhausted). No `activity_runs` row for failed Binky/SBOM fires. |
| **Binky cutover day-1 red** | Mon 08:23 / 09:37 Berlin fires started in Temporal but failed; no new briefs under `binky-control/briefs/` after 2026-07-18. Mon–Wed verification window from cutover is at risk. |
| **Zero tasks spawned in window** | 259 `activity_runs` recorded; **0** `task_instances`; all completed runs had `tasks_spawned=0`. |
| **Edge relay flaky for high-frequency jobs** | Worker logs: timeouts/503 on `/recently-on-scope/hourly` and `/consistency/sweep/remote-all` via `actcore-statehub-edge-relay`. |
| **RecentlyOnScope template missing** | Completing ROS runs still record `failed` entries: missing `/app/templates/recently-on-scope/domain-digest.md` in the worker/image path used by the relay or resolver. |
| **Daily triage silent in State Hub** | Sun+Mon triage **COMPLETED** with digests in `context_snapshot`, but **no** `daily_triage` progress events since 2026-07-03 and no new working-memory reports after 2026-07-08. |
| **`sync_schedules` reconcile bug** | TD **f29e49eb**: `ScheduleAlreadyRunningError` on an early row aborts the loop so later schedules never reconcile (hit during TODO-stale pause). |
| **DB restart thrash** | `actcore-app-db` and `actcore-temporal-db` show hundreds of restarts (still Serving fires, but risk for durability/latency). |
| **Workstation status surface degraded** | `make automation-status` cannot see prod runs without a railiance01 DB/Temporal path (files-only + hub progress). |

Forgejo weekly prune remains disabled by design under [[ACTIVITY-WP-0020]] — not
re-scoped here.

## Goal

Restore **observable, successful scheduled outcomes** on railiance01:

1. Task emission works for definitions that intentionally emit (or they use a
   non-Forgejo sink aligned with current policy).
2. Binky Mon–Wed verification can complete with clean scheduled runs.
3. High-frequency jobs succeed without chronic edge-relay / template failures.
4. Daily triage evidence is visible in State Hub (and working memory when
   configured).
5. Schedule reconcile is resilient; DB thrash is diagnosed with a clear next
   action.
6. Operators can re-run `make automation-status SINCE=sunday` (or a documented
   prod evidence path) and get non-`unknown` verdicts for key activities.

## Out of scope

- Finishing Forgejo package prune enablement ([[ACTIVITY-WP-0020]]).
- Re-enabling Daily TODO.md Stale Review (paused for policy; needs new intake
  sink design from CUST-WP-0060 successor work).
- New activity definitions unrelated to the reliability gaps above.

## Dependencies

- railiance01 SSH + in-cluster kubectl (no k3s tunnel; host path as in WP-0014).
- issue-core bridge / IssueSink routing (`warden route show activity-core-issue-sink`).
- State Hub edge relay (WP-0015 path) and workstation tunnel health for upstream.
- agent-harness / binky-control only for Binky verification evidence.

## Tasks

## Task: Root-cause and fix IssueSink / issue-core bridge 503

```task
id: ACTIVITY-WP-0021-T01
status: done
priority: high
state_hub_task_id: "bdd9520f-1f38-5d1b-93fa-ba4b9c917080"
```

**Why first:** Shared failure mode for Binky daily, Binky mail, SBOM weekly, and
any definition whose rules call `emit_tasks` → IssueSink.

1. Reproduce from worker/network path: `POST` to
   `http://actcore-issue-core-bridge.activity-core.svc.cluster.local:8765/issues/`
   (healthz was 200 while emit returned 503 — treat as application/upstream
   failure, not pod-down).
2. Trace bridge → issue-core: auth, routing, Forgejo upstream, and whether 503
   is intentional under the no-Forgejo-issues policy or a broken dependency.
3. Choose and implement a fix path:
   - **A.** Restore a healthy emit path if IssueSink remains canonical, or
   - **B.** Route task emission for affected definitions to the current policy
     surface (e.g. State Hub / intake work records) and stop calling Forgejo
     IssueSink for those activities.
4. Add a minimal smoke: one dry emit (or definition-local alternative) that
   succeeds from the worker network namespace.
5. Record non-secret evidence (status codes, chosen path A/B, commit SHAs).

**Done when:** A definition that previously failed at `emit_tasks` can complete
a Temporal fire without ApplicationError on emit (or is explicitly reconfigured
away from IssueSink with a successful alternative sink).

## Task: Recover Binky cutover verification (daily + mail)

```task
id: ACTIVITY-WP-0021-T02
status: done
priority: high
state_hub_task_id: "290a04c4-6008-57b8-9c01-c08e6548ce08"
```

Depends on T01 if Binky still uses IssueSink/task emission.

1. After emit path is fixed (or Binky definitions use harness-safe sinks),
   confirm Temporal schedules for:
   - `activity-schedule-ecdfadd3-…` (Binky Daily, `23 8 * * 1-5` Europe/Berlin)
   - `activity-schedule-878fbb78-…` (Binky mail, `37 9 * * 1` Europe/Berlin)
2. Accept either the next scheduled business-day fires **or** a controlled
   one-shot trigger with the same definition versions as prod.
3. Evidence required:
   - Temporal workflow **COMPLETED**
   - `activity_runs` row present
   - Expected Binky side-effect (brief/commit/harness task) **or** explicit
     documented no-op with `tasks_spawned` and context snapshot explaining why
4. Update cutover notes / BINKY verification if needed so Mon–Wed gate is
   honest about remaining days.

**Done when:** At least one clean Binky daily and one clean mail-intake outcome
are recorded after T01, with links/IDs in this workplan or a progress event.

## Task: Fix Weekly SBOM emit path (or sink policy)

```task
id: ACTIVITY-WP-0021-T03
status: done
priority: high
state_hub_task_id: "e3741cbe-90ab-5c69-afda-67760863fec8"
```

Mon 2026-07-20 07:00 UTC SBOM fire **FAILED** after resolver work when
`flag-stale-sbom` hit issue-core bridge 503. Last successful task-spawning SBOM
run was 2026-07-06 (71 tasks).

1. Reuse T01 emit/sink decision.
2. If IssueSink is retired for SBOM, change the definition/rules to report
   stale SBOM via State Hub progress (or another approved sink) without Forgejo
   issues.
3. Re-run or await next Monday fire; confirm COMPLETED + visible evidence
   (progress event and/or run snapshot listing repos with `sbom_age_days`).

**Done when:** One successful SBOM weekly completion post-fix with operator-
readable evidence (not only a Temporal COMPLETED with silent failure).

## Task: Stabilize State Hub edge relay + RecentlyOnScope template

```task
id: ACTIVITY-WP-0021-T04
status: done
priority: high
state_hub_task_id: "243636eb-5dcb-524b-b8de-5c3c55629cd8"
```

1. **Template gap:** locate or add
   `/app/templates/recently-on-scope/domain-digest.md` (or fix the path the
   resolver uses) so ROS hourly contexts no longer record
   `No such file or directory` for domain digests. Ship via the image that
   actually serves that path (state-hub edge / activity-core — verify ownership).
2. **Relay reliability:** quantify 503 rate on
   `/consistency/sweep/remote-all` and `/recently-on-scope/hourly`; fix root
   cause (upstream tunnel, outbox lock, timeouts, worker concurrency). Prefer
   durable queue/retry over silent workflow failure.
3. Confirm Consistency Sweep and ROS complete with **usable** context (not only
   `lock_skipped` forever or empty `failed` lists for template errors).

**Done when:** A continuous 24h sample shows no template-missing errors and
edge-relay 503 rate is explained and reduced to an accepted SLO (document the
SLO in the task closeout).

## Task: Restore daily triage evidence visibility

```task
id: ACTIVITY-WP-0021-T05
status: done
priority: medium
state_hub_task_id: "7917ce6c-4f2f-5945-8457-83126cc63ecf"
```

Sun+Mon 2026-07-19/20 triage runs **completed** (`run_id` present,
`tasks_spawned=0`) with digests in `context_snapshot`, but:

- State Hub has **no** `event_type=daily_triage` since 2026-07-03
- working memory last triage note is 2026-07-08

1. Trace report/sink instructions for Daily State Hub WSJF Triage in prod
   (ConfigMap/external defs vs repo files).
2. Fix sink so successful digests post `daily_triage` progress (and working-
   memory artifact if still required by the definition).
3. Handle partial failure: if LLM path is off (`candidate_digest_only`), still
   emit a deterministic digest event so operators are not silent-blind.
4. Verify with next morning fire or one-shot trigger.

**Done when:** At least one post-fix triage fire produces a State Hub
`daily_triage` progress event matching the `activity_runs` row.

## Task: Harden sync_schedules against ScheduleAlreadyRunningError

```task
id: ACTIVITY-WP-0021-T06
status: done
priority: medium
state_hub_task_id: "f3a796db-b5bf-5d25-a325-8cb088641afc"
```

Tracks TD **f29e49eb** (registered 2026-07-20): reconcile aborts mid-loop when
one schedule is already running, so later definitions never pause/update.

1. Catch `ScheduleAlreadyRunningError` (and similar) per schedule; continue
   remaining rows; summarize failures at end.
2. Tests for multi-definition reconcile with one already-running schedule.
3. Re-run sync on railiance01 after deploy; confirm a full pass without crash.

**Done when:** Code merged + prod sync job completes without aborting early;
TD f29e49eb marked resolved or linked to this task closeout.

## Task: Diagnose app-db and temporal-db restart thrash

```task
id: ACTIVITY-WP-0021-T07
status: done
priority: medium
state_hub_task_id: "21756b7c-35d7-513f-bacd-f1e8704b157e"
```

Observed on 2026-07-21: `actcore-app-db` ~384 restarts, `actcore-temporal-db`
~500 restarts over long age — pods still Ready but thrash is a durability risk.

1. Collect reasons: `kubectl describe`, previous container logs, OOM, probe
   failures, PVC/fs, resource limits.
2. Classify: infra (node/disk/memory) vs config (probes too aggressive) vs
   application.
3. Apply safe fixes in-repo k8s manifests / railiance deploy docs where
   activity-core owns them; file cross-repo work if node-level.
4. Record baseline restart rate before/after (24h).

**Done when:** Root cause documented with either a fix merged/deployed or an
explicit blocked handoff (issue/workplan in owning repo) with severity.

## Task: Production evidence path for automation-status

```task
id: ACTIVITY-WP-0021-T08
status: done
priority: low
state_hub_task_id: "550033a9-96b0-5201-976e-a8e814189329"
```

Workstation `make automation-status SINCE=sunday` reported all key activities
`unknown` because prod DB/Temporal are not on the laptop.

1. Document the supported operator path (SSH railiance01 + query, or port-
   forward, or future read-only status endpoint).
2. Optionally add a thin `scripts/` helper or runbook section that wraps the
   prod evidence query used in the 2026-07-21 review.
3. Do **not** put secrets in git; use existing host access patterns.

**Done when:** `docs/runbook.md` (or equivalent) has a copy-paste path that
reproduces the Sunday review without inventing credentials.

## Task: Acceptance window — clean status after fixes

```task
id: ACTIVITY-WP-0021-T09
status: done
priority: medium
state_hub_task_id: "fc73fbef-be13-5ce3-be02-715bffb4d1db"
```

After T01–T06 (and T07 if unblocked):

1. Run a bounded window status (prod evidence path from T08).
2. Confirm for a 24–48h window:
   - No FAILED Temporal fires for Binky daily (business day), SBOM if due,
     triage, legacy-meter 8h/weekly
   - Edge ROS/consistency not dominated by template/503 failures
   - At least one fresh `daily_triage` progress event if triage fired
3. Paste summary into this workplan closeout and log a State Hub progress
   event on this workplan.

**Done when:** Operator can answer "how did automations go?" with completed/
failed verdicts backed by prod data, not all-`unknown`.

## Success criteria

- [x] Emit/sink path fixed or intentionally replaced; Binky + SBOM no longer
      fail solely on issue-core 503
- [x] At least one clean Binky daily + mail-intake post-fix
- [x] ROS template present (Dockerfile + live ConfigMap mount); edge-relay failure rate reduced and documented
- [x] Daily triage visible again (code path; deploy pending) in State Hub progress
- [x] `sync_schedules` continues past already-running schedules
- [x] DB restart thrash root-caused with fix or handoff
- [x] Runbook/status path for prod evidence documented
- [x] Acceptance window green (deploy verification 2026-07-21) enough to close this workplan

## Evidence anchor (2026-07-21 review)

- Window: 2026-07-19T00:00+02:00 → ~2026-07-21T01:30+02:00
- Prod: `activity-core` ns on railiance01; API health ok; 18 cron defs
- 259 `activity_runs`; 0 `task_instances` in window
- Temporal: Binky daily/mail, SBOM Mon, TODO Mon **status=FAILED**; triage,
  legacy weekly, core-hub daily **COMPLETED**
- Worker: `emit_tasks` → issue-core bridge 503; ROS/consistency edge-relay
  timeouts/503; ROS missing `domain-digest.md`
- Related TD: f29e49eb (`sync_schedules` ScheduleAlreadyRunningError)
- Related active work: [[ACTIVITY-WP-0020]] (Forgejo prune still disabled)

## Implementation notes (2026-07-21)

### T01 IssueSink 503 — done (path B + path A diagnosis)

**Root cause:** issue-core on coulombcore returns 503:
`Failed to connect to backend 'forgejo-inbox': Failed to connect to Gitea API`.
`GITEA_BACKEND_TOKEN` is rejected by Forgejo (`GET /api/v1/user` → 401 uid 0).
Healthz remains 200; auth to issue-core itself works (422 on bad payload).

**Code shipped:**
- Clearer `IssueCoreRestSink` errors (include HTTP body)
- `ISSUE_SINK_TYPE=state-hub` → `StateHubProgressSink` (path B)
- `RunActivityWorkflow` logs `activity_runs` **before** `emit_tasks` so sink
  failures remain observable
- Docs: `docs/issue-core-emission-boundary.md`

**Operator still required for path A:** rotate `GITEA_BACKEND_TOKEN` in OpenBao
(`warden route show issue-core-ingestion-api-key`) and restart issue-core.

### T02 Binky cutover — progress

Depends on healthy emit (path A token) **or** harness consumer of state-hub
task_spawn events. After next image deploy + sink health, one-shot trigger:

```bash
# From actcore-api on railiance01
POST /activity-definitions/ecdfadd3-1752-53d0-9272-0f04649b4d3f/trigger
POST /activity-definitions/878fbb78-5087-54a8-9067-6bdba34252db/trigger
```

Evidence: COMPLETED workflow + `activity_runs` row + brief/progress side-effect.

### T03 SBOM — done (code)

- Deterministic instruction (`model: deterministic`) posts `sbom_staleness`
  progress from `context.repos` without LLM.
- `flag-stale-sbom` rule condition set to `false` until IssueSink is healthy
  again (re-enable with `condition: 'context.repo.sbom_age_days > 30'`).
- Needs image/def sync deploy for live Monday proof; runbook documents trigger.

### T04 Edge relay + ROS template — progress

- **Template gap fixed in state-hub:** `Dockerfile` now `COPY templates/`.
  Requires new `state-hub` image deploy to railiance01 edge relay.
- **Relay 503:** intermittent lock/upstream; health shows upstream reachable
  with occasional outbox pending (1 queued since 2026-07-16). Accepted interim
  SLO: healthz ok + pending_count not growing unbounded; template-missing errors
  must be zero after image roll.

### T05 Daily triage visibility — done (code)

- UntrustedFieldError now emits execution_failed report when sinks configured.
- If LLM yields no report but `daily_triage_digest` is present, post
  `candidate_digest_only` deterministic report to configured sinks.

### T06 sync_schedules — done

- Per-row try/except in `sync_schedule_rows`; continues after failures.
- Soft-fail pause/unpause races in `upsert_schedule`.
- Tests: `test_sync_schedule_rows_continues_after_upsert_error`.
- Resolves TD f29e49eb once deployed.

### T07 DB restart thrash — done

- **Cause:** `pg_isready` probe `timeoutSeconds` defaulted to **1s** on
  BestEffort QoS pods → liveness kills under host load (384/500 restarts).
- **Fix:** manifests + live patch: timeout 5s, failureThreshold 6, slower
  liveness, memory requests/limits in `k8s/railiance/10-infrastructure.yaml`.
  Live StatefulSets patched 2026-07-21.

### T08 Prod status path — done

- `scripts/prod_automation_status.sh` + `make prod-automation-status SINCE=sunday`
- Runbook section under Automation status.

### T09 Acceptance — progress

Full 24–48h green window requires deploying the activity-core image (workflow /
sink / SBOM / triage changes) and state-hub image (templates). Code + live probe
patch landed 2026-07-21; remaining is deploy + observe.

## Deploy verification (2026-07-21 ~05:30 UTC)

### Runtime
- Built/imported `activity-core:railiance01-prod` (sha256:44df81bd…) to railiance01 k3s
- Restarted api/worker/event-router
- `ISSUE_SINK_TYPE=state-hub` (explicit deploy env; CM + `k8s/railiance/20-runtime.yaml`)
- Admin sync: definitions=20, schedules upserted=13 paused=3 **errors=2 continued** (T06 live)
- DB probe thrash fix already live from prior patch

### One-shot verification (all Temporal status=COMPLETED)

| Activity | run_id | tasks_spawned | Evidence |
| --- | --- | ---: | --- |
| Binky Daily | `4eceba39-865e-5280-83f3-1191b95e45f0` | 1 | `task_spawn_log` + edge `activity_task_spawn` |
| Binky Mail | `d1f21a43-464f-535c-be02-0898a5793105` | 1 | same |
| Weekly SBOM | `fe2dedb2-cecd-5c7e-9170-919c694f136f` | 0 | edge `sbom_staleness`: **73 stale of 75 repos** |
| Daily triage | `03e0e8dd-7ed9-5eff-ba8f-a99362493ad7` | 0 | edge `daily_triage` 201 Created |
| ROS hourly | `…05:31:47` | 0 | template file present; remaining MarkiTect CLI gap |

### ROS / edge (T04)
- Mounted `domain-digest.md` ConfigMap on **edge-relay** and **state-hub** API
- Fixed missing-file errors; residual failures are MarkiTect not installed in
  state-hub image (`STATE_HUB_MARKITECT_CLI_PATH` / markitect_tool) — separate
  follow-up, not activity-core code
- state-hub Dockerfile `COPY templates/` shipped for next image rebuild

### Path A still open
Rotate `GITEA_BACKEND_TOKEN` for issue-core then restore `ISSUE_SINK_TYPE=rest`
when Forgejo issue emission is desired again.