Repair production automation truth and schedule cleanup
All checks were successful
CI Smoke / host-smoke (push) Successful in 0s
CI Smoke / container-smoke (push) Successful in 1s
Build and Publish Container Image / build-and-push (push) Successful in 18s

This commit is contained in:
tegwick 2026-08-20 11:20:23 +02:00
parent 8bcb416285
commit 944fd158de
19 changed files with 441 additions and 64 deletions

View file

@ -8,7 +8,7 @@ status: active
owner: grok
topic_slug: infotech
created: "2026-08-18"
updated: "2026-08-18"
updated: "2026-08-20"
parent_workplan: CUST-WP-0062
related:
- CUST-WP-0062
@ -92,16 +92,17 @@ oldest never-scanned repo.
```task
id: ACTIVITY-WP-0030-T03
status: wait
status: progress
priority: high
state_hub_task_id: "18bcfc21-616a-48a2-aac3-cdcef7579913"
```
After the daily job is proven on railiance01: set
`weekly-sbom-staleness` `enabled: false` (or remove `flag-stale-sbom`), keep
or drop the weekly summary report only if it no longer emits one task per
repo. Update runbook / playbook. Prove one Monday window with zero SBOM task
spawns and a successful daily catch-up the same week.
Emergency containment began 2026-08-20 under ACTIVITY-WP-0031: set
`weekly-sbom-staleness` `enabled: false` in source and production before the
bounded replacement is available. Keep or drop the weekly summary report only
if it no longer emits one task per repo. After the daily job is proven on
railiance01, update the runbook / playbook and prove one Monday window with zero
SBOM task spawns and a successful daily catch-up the same week.
### Prod enable and evidence

View file

@ -0,0 +1,120 @@
---
id: ACTIVITY-WP-0031
type: workplan
title: "Production automation execution reliability cleanup"
domain: infotech
repo: activity-core
status: active
owner: codex
topic_slug: activity-core
created: "2026-08-20"
updated: "2026-08-20"
related:
- ACTIVITY-WP-0021
- ACTIVITY-WP-0022
- ACTIVITY-WP-0026
- ACTIVITY-WP-0030
---
# Production automation execution reliability cleanup
## Goal
Make production automation health reflect completed execution rather than only
successful schedule fires, stop known task floods, and repair the repository
contracts found during the 2026-08-20 activity-core review.
## Restore provider-backed execution
```task
id: ACTIVITY-WP-0031-T01
status: wait
priority: high
```
Production FI, Binky, and daily-triage runs reach `llm-connect` but fail after
three attempts. An in-cluster fixture returned a sanitized upstream OpenRouter
HTTP 401. Credential ownership is the `railiance-platform` OpenBao lane
(`openrouter-llm-connect`); activity-core must not bypass that custody boundary.
Track rotation/reconciliation, restart, and a successful fixture smoke as the
exit evidence.
## Emergency-pause weekly SBOM fan-out
```task
id: ACTIVITY-WP-0031-T02
status: done
priority: high
```
Disable `weekly-sbom-staleness` in source and on railiance01 before the next
Monday window. The bounded daily replacement remains owned by ACTIVITY-WP-0030;
this task is the immediate containment measure for the observed 75-task fire.
Completed 2026-08-20: source, live DB, and Temporal schedule are disabled. The
new image and ConfigMap projection were deployed; schedule reconciliation
reports the definition as paused.
## Report execution outcomes in production status
```task
id: ACTIVITY-WP-0031-T03
status: done
priority: high
```
Extend `scripts/prod_automation_status.sh` with ops-run outcome counts, bounded
failed/open detail, and daily-triage progress validation evidence. Preserve
sanitized error output and keep the command deterministic and read-only.
Completed 2026-08-20. The live report now shows ops-run state counts, bounded
failure/open detail, and bounded State Hub `daily_triage` validation evidence.
It exposed today's FI/Binky 502 failures and triage `output_validated=false`
beside otherwise healthy schedule-fire counts.
## Reconcile implementation and test contracts
```task
id: ACTIVITY-WP-0031-T04
status: done
priority: medium
```
Repair the four reproducible suite failures, keep runtime-path resolution out of
the rules package boundary, and document that `dedupe_key_strategy` is retained
metadata rather than repeated-content suppression.
Completed 2026-08-20. Runtime URI resolution now happens before the pure rules
boundary; stale assertions/test doubles are reconciled; disabled one-shots are
deleted idempotently during schedule sync; and the dedupe contract is truthful.
Verification: `347 passed, 1 skipped`; a subsequent focused warning regression
passed `10 passed` without warnings.
## Repair consumer routing and State Hub metadata
```task
id: ACTIVITY-WP-0031-T05
status: wait
priority: medium
```
Make the OpenBao retention reactivation task explicitly claimable without
authorizing destructive cleanup. Route Binky's missing workload credential
front door and the duplicate legacy workstream identifiers to their canonical
owners. Clear stale human-needed flags only for already-terminal activity-core
tasks.
Activity-core cleanup is complete: three terminal tasks no longer retain stale
`needs_human` flags. External handoffs were sent to binky-control and
railiance-platform for mail credential materialization, repo-manager for the
duplicate registrar IDs, and railiance-master for the unclaimed retention
closeout. The expired retention one-shot is disabled; its existing open ops run
remains operator-visible and cannot authorize destructive cleanup.
## Acceptance
- [x] Weekly SBOM fan-out is disabled in source and production
- [x] Production status distinguishes schedule fires from execution failures
- [x] Full test suite is green
- [ ] Provider and consumer prerequisites have named owners and evidence
- [x] Workplan and State Hub task state are consistent

View file

@ -0,0 +1,43 @@
---
id: ADHOC-2026-08-20
type: workplan
title: "Ad hoc — activity-core opportunistic fixes 2026-08-20"
domain: infotech
repo: activity-core
status: finished
owner: codex
topic_slug: activity-core
created: "2026-08-20"
updated: "2026-08-20"
---
# ADHOC-2026-08-20 — activity-core opportunistic fixes
## Reconcile the daily TODO review sink warning
```task
id: ADHOC-2026-08-20-T01
status: done
priority: low
```
The projected `daily-todo-md-stale-review` definition still said that
`IssueSink` defaulted to `rest` and that re-enabling the definition would
unconditionally recreate Forgejo issues. That warning described the state before
ACTIVITY-WP-0022 and contradicted the current code, deployment configuration,
and `weekly-sbom-staleness` definition.
Correct the warning to state that `ISSUE_SINK_TYPE=state-hub` is the safe fleet
default, `rest` is explicit opt-in, and State Hub progress is visibility rather
than claim authority. Keep the schedule disabled so a documentation correction
does not silently change production cadence, and document the operational checks
needed before it is deliberately re-enabled. Add a projection test so the stale
`default rest` claim cannot recur unnoticed.
Verification:
- Focused regression and IssueSink tests: `7 passed`.
- Full suite: `342 passed, 1 skipped, 4 failed`. The four failures are unrelated
existing inconsistencies: rules boundary allowlisting, the State Hub image pin
assertion, a schedule-health response double missing `status_code`, and the
sync-service expectation omitting the returned `errors` count.