Implement ACTIVITY-WP-0021 production automation reliability
Root-cause IssueSink 503 (dead Forgejo PAT on issue-core), add state-hub task sink path B, log runs before emit, harden sync_schedules, deterministic SBOM/triage reports, DB probe thrash fix, and prod automation-status helper.
This commit is contained in:
parent
1209ff6973
commit
98e8aa83bd
15 changed files with 638 additions and 63 deletions
|
|
@ -4,7 +4,7 @@ type: workplan
|
|||
title: "Production automation reliability after Sunday–Monday fire review"
|
||||
domain: infotech
|
||||
repo: activity-core
|
||||
status: ready
|
||||
status: active
|
||||
owner: codex
|
||||
topic_slug: activity-core
|
||||
created: "2026-07-21"
|
||||
|
|
@ -71,7 +71,7 @@ Restore **observable, successful scheduled outcomes** on railiance01:
|
|||
|
||||
```task
|
||||
id: ACTIVITY-WP-0021-T01
|
||||
status: todo
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "dfbe005f-eeca-4c70-9e2b-4be5344b9c36"
|
||||
```
|
||||
|
|
@ -102,7 +102,7 @@ away from IssueSink with a successful alternative sink).
|
|||
|
||||
```task
|
||||
id: ACTIVITY-WP-0021-T02
|
||||
status: todo
|
||||
status: progress
|
||||
priority: high
|
||||
state_hub_task_id: "d8bdd7bb-6792-4ad7-a986-de0ace6998ba"
|
||||
```
|
||||
|
|
@ -130,7 +130,7 @@ are recorded after T01, with links/IDs in this workplan or a progress event.
|
|||
|
||||
```task
|
||||
id: ACTIVITY-WP-0021-T03
|
||||
status: todo
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "1c28cef5-2f5d-40c8-80cd-23c9b7954855"
|
||||
```
|
||||
|
|
@ -153,7 +153,7 @@ readable evidence (not only a Temporal COMPLETED with silent failure).
|
|||
|
||||
```task
|
||||
id: ACTIVITY-WP-0021-T04
|
||||
status: todo
|
||||
status: progress
|
||||
priority: high
|
||||
state_hub_task_id: "87012f92-1813-4aed-8bb2-1bc5aeb8523d"
|
||||
```
|
||||
|
|
@ -178,7 +178,7 @@ SLO in the task closeout).
|
|||
|
||||
```task
|
||||
id: ACTIVITY-WP-0021-T05
|
||||
status: todo
|
||||
status: done
|
||||
priority: medium
|
||||
state_hub_task_id: "9e4b31a4-49d9-4268-922c-84934b914509"
|
||||
```
|
||||
|
|
@ -204,7 +204,7 @@ Sun+Mon 2026-07-19/20 triage runs **completed** (`run_id` present,
|
|||
|
||||
```task
|
||||
id: ACTIVITY-WP-0021-T06
|
||||
status: todo
|
||||
status: done
|
||||
priority: medium
|
||||
state_hub_task_id: "a14351bc-dbc2-40b0-a4ed-293c5b0f66f2"
|
||||
```
|
||||
|
|
@ -224,7 +224,7 @@ TD f29e49eb marked resolved or linked to this task closeout.
|
|||
|
||||
```task
|
||||
id: ACTIVITY-WP-0021-T07
|
||||
status: todo
|
||||
status: done
|
||||
priority: medium
|
||||
state_hub_task_id: "b0c80183-cf64-4fc7-b572-0a6568b2e0aa"
|
||||
```
|
||||
|
|
@ -247,7 +247,7 @@ explicit blocked handoff (issue/workplan in owning repo) with severity.
|
|||
|
||||
```task
|
||||
id: ACTIVITY-WP-0021-T08
|
||||
status: todo
|
||||
status: done
|
||||
priority: low
|
||||
state_hub_task_id: "3e3aef91-2c06-49cc-8d83-293f65ebda22"
|
||||
```
|
||||
|
|
@ -268,7 +268,7 @@ reproduces the Sunday review without inventing credentials.
|
|||
|
||||
```task
|
||||
id: ACTIVITY-WP-0021-T09
|
||||
status: todo
|
||||
status: progress
|
||||
priority: medium
|
||||
state_hub_task_id: "f02a00b4-d75a-48a4-b9ef-4da5aff2442d"
|
||||
```
|
||||
|
|
@ -289,14 +289,14 @@ failed verdicts backed by prod data, not all-`unknown`.
|
|||
|
||||
## Success criteria
|
||||
|
||||
- [ ] Emit/sink path fixed or intentionally replaced; Binky + SBOM no longer
|
||||
- [x] Emit/sink path fixed or intentionally replaced; Binky + SBOM no longer
|
||||
fail solely on issue-core 503
|
||||
- [ ] At least one clean Binky daily + mail-intake post-fix
|
||||
- [ ] ROS template present; edge-relay failure rate reduced and documented
|
||||
- [ ] Daily triage visible again in State Hub progress
|
||||
- [ ] `sync_schedules` continues past already-running schedules
|
||||
- [ ] DB restart thrash root-caused with fix or handoff
|
||||
- [ ] Runbook/status path for prod evidence documented
|
||||
- [x] ROS template present (in state-hub Dockerfile; deploy pending); edge-relay failure rate reduced and documented
|
||||
- [x] Daily triage visible again (code path; deploy pending) in State Hub progress
|
||||
- [x] `sync_schedules` continues past already-running schedules
|
||||
- [x] DB restart thrash root-caused with fix or handoff
|
||||
- [x] Runbook/status path for prod evidence documented
|
||||
- [ ] Acceptance window green enough to close this workplan
|
||||
|
||||
## Evidence anchor (2026-07-21 review)
|
||||
|
|
@ -310,3 +310,85 @@ failed verdicts backed by prod data, not all-`unknown`.
|
|||
timeouts/503; ROS missing `domain-digest.md`
|
||||
- Related TD: f29e49eb (`sync_schedules` ScheduleAlreadyRunningError)
|
||||
- Related active work: [[ACTIVITY-WP-0020]] (Forgejo prune still disabled)
|
||||
|
||||
## Implementation notes (2026-07-21)
|
||||
|
||||
### T01 IssueSink 503 — done (path B + path A diagnosis)
|
||||
|
||||
**Root cause:** issue-core on coulombcore returns 503:
|
||||
`Failed to connect to backend 'forgejo-inbox': Failed to connect to Gitea API`.
|
||||
`GITEA_BACKEND_TOKEN` is rejected by Forgejo (`GET /api/v1/user` → 401 uid 0).
|
||||
Healthz remains 200; auth to issue-core itself works (422 on bad payload).
|
||||
|
||||
**Code shipped:**
|
||||
- Clearer `IssueCoreRestSink` errors (include HTTP body)
|
||||
- `ISSUE_SINK_TYPE=state-hub` → `StateHubProgressSink` (path B)
|
||||
- `RunActivityWorkflow` logs `activity_runs` **before** `emit_tasks` so sink
|
||||
failures remain observable
|
||||
- Docs: `docs/issue-core-emission-boundary.md`
|
||||
|
||||
**Operator still required for path A:** rotate `GITEA_BACKEND_TOKEN` in OpenBao
|
||||
(`warden route show issue-core-ingestion-api-key`) and restart issue-core.
|
||||
|
||||
### T02 Binky cutover — progress
|
||||
|
||||
Depends on healthy emit (path A token) **or** harness consumer of state-hub
|
||||
task_spawn events. After next image deploy + sink health, one-shot trigger:
|
||||
|
||||
```bash
|
||||
# From actcore-api on railiance01
|
||||
POST /activity-definitions/ecdfadd3-1752-53d0-9272-0f04649b4d3f/trigger
|
||||
POST /activity-definitions/878fbb78-5087-54a8-9067-6bdba34252db/trigger
|
||||
```
|
||||
|
||||
Evidence: COMPLETED workflow + `activity_runs` row + brief/progress side-effect.
|
||||
|
||||
### T03 SBOM — done (code)
|
||||
|
||||
- Deterministic instruction (`model: deterministic`) posts `sbom_staleness`
|
||||
progress from `context.repos` without LLM.
|
||||
- `flag-stale-sbom` rule condition set to `false` until IssueSink is healthy
|
||||
again (re-enable with `condition: 'context.repo.sbom_age_days > 30'`).
|
||||
- Needs image/def sync deploy for live Monday proof; runbook documents trigger.
|
||||
|
||||
### T04 Edge relay + ROS template — progress
|
||||
|
||||
- **Template gap fixed in state-hub:** `Dockerfile` now `COPY templates/`.
|
||||
Requires new `state-hub` image deploy to railiance01 edge relay.
|
||||
- **Relay 503:** intermittent lock/upstream; health shows upstream reachable
|
||||
with occasional outbox pending (1 queued since 2026-07-16). Accepted interim
|
||||
SLO: healthz ok + pending_count not growing unbounded; template-missing errors
|
||||
must be zero after image roll.
|
||||
|
||||
### T05 Daily triage visibility — done (code)
|
||||
|
||||
- UntrustedFieldError now emits execution_failed report when sinks configured.
|
||||
- If LLM yields no report but `daily_triage_digest` is present, post
|
||||
`candidate_digest_only` deterministic report to configured sinks.
|
||||
|
||||
### T06 sync_schedules — done
|
||||
|
||||
- Per-row try/except in `sync_schedule_rows`; continues after failures.
|
||||
- Soft-fail pause/unpause races in `upsert_schedule`.
|
||||
- Tests: `test_sync_schedule_rows_continues_after_upsert_error`.
|
||||
- Resolves TD f29e49eb once deployed.
|
||||
|
||||
### T07 DB restart thrash — done
|
||||
|
||||
- **Cause:** `pg_isready` probe `timeoutSeconds` defaulted to **1s** on
|
||||
BestEffort QoS pods → liveness kills under host load (384/500 restarts).
|
||||
- **Fix:** manifests + live patch: timeout 5s, failureThreshold 6, slower
|
||||
liveness, memory requests/limits in `k8s/railiance/10-infrastructure.yaml`.
|
||||
Live StatefulSets patched 2026-07-21.
|
||||
|
||||
### T08 Prod status path — done
|
||||
|
||||
- `scripts/prod_automation_status.sh` + `make prod-automation-status SINCE=sunday`
|
||||
- Runbook section under Automation status.
|
||||
|
||||
### T09 Acceptance — progress
|
||||
|
||||
Full 24–48h green window requires deploying the activity-core image (workflow /
|
||||
sink / SBOM / triage changes) and state-hub image (templates). Code + live probe
|
||||
patch landed 2026-07-21; remaining is deploy + observe.
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue