Repair production automation truth and schedule cleanup
This commit is contained in:
parent
8bcb416285
commit
944fd158de
19 changed files with 441 additions and 64 deletions
|
|
@ -8,7 +8,7 @@ status: active
|
|||
owner: grok
|
||||
topic_slug: infotech
|
||||
created: "2026-08-18"
|
||||
updated: "2026-08-18"
|
||||
updated: "2026-08-20"
|
||||
parent_workplan: CUST-WP-0062
|
||||
related:
|
||||
- CUST-WP-0062
|
||||
|
|
@ -92,16 +92,17 @@ oldest never-scanned repo.
|
|||
|
||||
```task
|
||||
id: ACTIVITY-WP-0030-T03
|
||||
status: wait
|
||||
status: progress
|
||||
priority: high
|
||||
state_hub_task_id: "18bcfc21-616a-48a2-aac3-cdcef7579913"
|
||||
```
|
||||
|
||||
After the daily job is proven on railiance01: set
|
||||
`weekly-sbom-staleness` `enabled: false` (or remove `flag-stale-sbom`), keep
|
||||
or drop the weekly summary report only if it no longer emits one task per
|
||||
repo. Update runbook / playbook. Prove one Monday window with zero SBOM task
|
||||
spawns and a successful daily catch-up the same week.
|
||||
Emergency containment began 2026-08-20 under ACTIVITY-WP-0031: set
|
||||
`weekly-sbom-staleness` `enabled: false` in source and production before the
|
||||
bounded replacement is available. Keep or drop the weekly summary report only
|
||||
if it no longer emits one task per repo. After the daily job is proven on
|
||||
railiance01, update the runbook / playbook and prove one Monday window with zero
|
||||
SBOM task spawns and a successful daily catch-up the same week.
|
||||
|
||||
### Prod enable and evidence
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,120 @@
|
|||
---
|
||||
id: ACTIVITY-WP-0031
|
||||
type: workplan
|
||||
title: "Production automation execution reliability cleanup"
|
||||
domain: infotech
|
||||
repo: activity-core
|
||||
status: active
|
||||
owner: codex
|
||||
topic_slug: activity-core
|
||||
created: "2026-08-20"
|
||||
updated: "2026-08-20"
|
||||
related:
|
||||
- ACTIVITY-WP-0021
|
||||
- ACTIVITY-WP-0022
|
||||
- ACTIVITY-WP-0026
|
||||
- ACTIVITY-WP-0030
|
||||
---
|
||||
|
||||
# Production automation execution reliability cleanup
|
||||
|
||||
## Goal
|
||||
|
||||
Make production automation health reflect completed execution rather than only
|
||||
successful schedule fires, stop known task floods, and repair the repository
|
||||
contracts found during the 2026-08-20 activity-core review.
|
||||
|
||||
## Restore provider-backed execution
|
||||
|
||||
```task
|
||||
id: ACTIVITY-WP-0031-T01
|
||||
status: wait
|
||||
priority: high
|
||||
```
|
||||
|
||||
Production FI, Binky, and daily-triage runs reach `llm-connect` but fail after
|
||||
three attempts. An in-cluster fixture returned a sanitized upstream OpenRouter
|
||||
HTTP 401. Credential ownership is the `railiance-platform` OpenBao lane
|
||||
(`openrouter-llm-connect`); activity-core must not bypass that custody boundary.
|
||||
Track rotation/reconciliation, restart, and a successful fixture smoke as the
|
||||
exit evidence.
|
||||
|
||||
## Emergency-pause weekly SBOM fan-out
|
||||
|
||||
```task
|
||||
id: ACTIVITY-WP-0031-T02
|
||||
status: done
|
||||
priority: high
|
||||
```
|
||||
|
||||
Disable `weekly-sbom-staleness` in source and on railiance01 before the next
|
||||
Monday window. The bounded daily replacement remains owned by ACTIVITY-WP-0030;
|
||||
this task is the immediate containment measure for the observed 75-task fire.
|
||||
|
||||
Completed 2026-08-20: source, live DB, and Temporal schedule are disabled. The
|
||||
new image and ConfigMap projection were deployed; schedule reconciliation
|
||||
reports the definition as paused.
|
||||
|
||||
## Report execution outcomes in production status
|
||||
|
||||
```task
|
||||
id: ACTIVITY-WP-0031-T03
|
||||
status: done
|
||||
priority: high
|
||||
```
|
||||
|
||||
Extend `scripts/prod_automation_status.sh` with ops-run outcome counts, bounded
|
||||
failed/open detail, and daily-triage progress validation evidence. Preserve
|
||||
sanitized error output and keep the command deterministic and read-only.
|
||||
|
||||
Completed 2026-08-20. The live report now shows ops-run state counts, bounded
|
||||
failure/open detail, and bounded State Hub `daily_triage` validation evidence.
|
||||
It exposed today's FI/Binky 502 failures and triage `output_validated=false`
|
||||
beside otherwise healthy schedule-fire counts.
|
||||
|
||||
## Reconcile implementation and test contracts
|
||||
|
||||
```task
|
||||
id: ACTIVITY-WP-0031-T04
|
||||
status: done
|
||||
priority: medium
|
||||
```
|
||||
|
||||
Repair the four reproducible suite failures, keep runtime-path resolution out of
|
||||
the rules package boundary, and document that `dedupe_key_strategy` is retained
|
||||
metadata rather than repeated-content suppression.
|
||||
|
||||
Completed 2026-08-20. Runtime URI resolution now happens before the pure rules
|
||||
boundary; stale assertions/test doubles are reconciled; disabled one-shots are
|
||||
deleted idempotently during schedule sync; and the dedupe contract is truthful.
|
||||
Verification: `347 passed, 1 skipped`; a subsequent focused warning regression
|
||||
passed `10 passed` without warnings.
|
||||
|
||||
## Repair consumer routing and State Hub metadata
|
||||
|
||||
```task
|
||||
id: ACTIVITY-WP-0031-T05
|
||||
status: wait
|
||||
priority: medium
|
||||
```
|
||||
|
||||
Make the OpenBao retention reactivation task explicitly claimable without
|
||||
authorizing destructive cleanup. Route Binky's missing workload credential
|
||||
front door and the duplicate legacy workstream identifiers to their canonical
|
||||
owners. Clear stale human-needed flags only for already-terminal activity-core
|
||||
tasks.
|
||||
|
||||
Activity-core cleanup is complete: three terminal tasks no longer retain stale
|
||||
`needs_human` flags. External handoffs were sent to binky-control and
|
||||
railiance-platform for mail credential materialization, repo-manager for the
|
||||
duplicate registrar IDs, and railiance-master for the unclaimed retention
|
||||
closeout. The expired retention one-shot is disabled; its existing open ops run
|
||||
remains operator-visible and cannot authorize destructive cleanup.
|
||||
|
||||
## Acceptance
|
||||
|
||||
- [x] Weekly SBOM fan-out is disabled in source and production
|
||||
- [x] Production status distinguishes schedule fires from execution failures
|
||||
- [x] Full test suite is green
|
||||
- [ ] Provider and consumer prerequisites have named owners and evidence
|
||||
- [x] Workplan and State Hub task state are consistent
|
||||
43
workplans/ADHOC-2026-08-20.md
Normal file
43
workplans/ADHOC-2026-08-20.md
Normal file
|
|
@ -0,0 +1,43 @@
|
|||
---
|
||||
id: ADHOC-2026-08-20
|
||||
type: workplan
|
||||
title: "Ad hoc — activity-core opportunistic fixes 2026-08-20"
|
||||
domain: infotech
|
||||
repo: activity-core
|
||||
status: finished
|
||||
owner: codex
|
||||
topic_slug: activity-core
|
||||
created: "2026-08-20"
|
||||
updated: "2026-08-20"
|
||||
---
|
||||
|
||||
# ADHOC-2026-08-20 — activity-core opportunistic fixes
|
||||
|
||||
## Reconcile the daily TODO review sink warning
|
||||
|
||||
```task
|
||||
id: ADHOC-2026-08-20-T01
|
||||
status: done
|
||||
priority: low
|
||||
```
|
||||
|
||||
The projected `daily-todo-md-stale-review` definition still said that
|
||||
`IssueSink` defaulted to `rest` and that re-enabling the definition would
|
||||
unconditionally recreate Forgejo issues. That warning described the state before
|
||||
ACTIVITY-WP-0022 and contradicted the current code, deployment configuration,
|
||||
and `weekly-sbom-staleness` definition.
|
||||
|
||||
Correct the warning to state that `ISSUE_SINK_TYPE=state-hub` is the safe fleet
|
||||
default, `rest` is explicit opt-in, and State Hub progress is visibility rather
|
||||
than claim authority. Keep the schedule disabled so a documentation correction
|
||||
does not silently change production cadence, and document the operational checks
|
||||
needed before it is deliberately re-enabled. Add a projection test so the stale
|
||||
`default rest` claim cannot recur unnoticed.
|
||||
|
||||
Verification:
|
||||
|
||||
- Focused regression and IssueSink tests: `7 passed`.
|
||||
- Full suite: `342 passed, 1 skipped, 4 failed`. The four failures are unrelated
|
||||
existing inconsistencies: rules boundary allowlisting, the State Hub image pin
|
||||
assertion, a schedule-health response double missing `status_code`, and the
|
||||
sync-service expectation omitting the returned `errors` count.
|
||||
Loading…
Add table
Add a link
Reference in a new issue