activity-core/workplans/ACTIVITY-WP-0031-production-execution-reliability-cleanup.md

---
id: ACTIVITY-WP-0031
type: workplan
title: "Production automation execution reliability cleanup"
domain: infotech
repo: activity-core
status: active
owner: codex
topic_slug: activity-core
created: "2026-08-20"
updated: "2026-08-21"
related:
  - ACTIVITY-WP-0021
  - ACTIVITY-WP-0022
  - ACTIVITY-WP-0026
  - ACTIVITY-WP-0030
---

# Production automation execution reliability cleanup

## Goal

Make production automation health reflect completed execution rather than only
successful schedule fires, stop known task floods, and repair the repository
contracts found during the 2026-08-20 activity-core review.

## Restore provider-backed execution

```task
id: ACTIVITY-WP-0031-T01
status: wait
priority: high
```

Production FI, Binky, and daily-triage runs reach `llm-connect` but fail after
three attempts. An in-cluster fixture returned a sanitized upstream OpenRouter
HTTP 401. Credential ownership is the `railiance-platform` OpenBao lane
(`openrouter-llm-connect`); activity-core must not bypass that custody boundary.
Track rotation/reconciliation, restart, and a successful fixture smoke as the
exit evidence.

Progress 2026-08-20: railiance-platform repaired the delivery half. The
activity-core ESO bootstrap default now includes the approved exact-path
llm-connect policy, its replacement token reports read capability, and the
reviewed `ExternalSecret` is `Ready=True` / `SecretSynced` on railiance01. It
took ownership of the one-key Secret, llm-connect rolled out, and the real
actcore-worker reaches `/health` through the Service. The post-restart fixture
still returns sanitized OpenRouter HTTP 401, proving the canonical key itself
is rejected. T01 remains `wait` on an attended OpenRouter account owner to mint
and safely provision a replacement key; no key value was read or printed.

Progress 2026-08-21 (activity-core half, no custody boundary crossed): live
evidence confirms a single fault, not two. FI, Binky, and daily triage still
fail today (`05:20` / `05:30` / `06:23` UTC) with `502 Bad Gateway` from
llm-connect — **after** yesterday's rollout (`llm-connect` pod is `Running`
1/1, endpoints healthy). That 502 is not a dead gateway:
`llm_connect/server.py::_error_response` maps **every** `LLMAPIError` onto 502
and carries the real cause in the body (`error`, `provider_status`). So the
502s and the reported upstream 401 are the same rejected key.

`LLMConnectClient.complete` called `raise_for_status()` and discarded that
body, so four days of production evidence read as "llm-connect is down" when it
meant "the provider rejected the key". Fixed: the client now raises with
`error`, `provider_status`, `provider`, `model`, and a bounded copy of
llm-connect's already-sanitized `message`, under a field allowlist so no
provider blob or key material can reach the run artefact. Five tests cover the
401-behind-502 case, the allowlist, bounding, and unusable bodies.

The identical body-discarding call exists in
`rein-aharness/rein_aharness/llm_connect_client.py:65`, which is what produces
the opaque ops_run failure text in the status table. Not ours to edit —
handed to rein-aharness.

T01 still `wait`: the key rotation itself remains attended and
railiance-platform-owned. What changed is that the next failure will name its
own cause.

Diagnosis verified 2026-08-21 (previously relayed, now tested). Direct probe
from the llm-connect pod, no secret read or printed:

- `POST /execute` → `502 {"error":"provider_api_error","provider_status":401,
  "message":"HTTP 401 from https://openrouter.ai/api/v1/chat/completions"}`
- `GET https://openrouter.ai/api/v1/key` with the delivered key →
  **`401 {"error":{"message":"User not found.","code":401}}`**

"User not found" is OpenRouter's wording for a key that resolves to **no
account**. It is not insufficient credits (402) and not a model-permission
problem (403), so **no new OpenRouter account is required** — the existing
account needs a new key issued, because this key value is dead.
Key fingerprint for identification: `sha256[:12] = ab938241a2ec`, length 73
(+1, see below), prefix `sk-or-v1-`.

Ruled out along the way: provider/model config is correct
(`LLM_CONNECT_PROVIDER=openrouter`, `google/gemini-2.5-flash`); the Service and
pod are healthy; the ExternalSecret delivers exactly one key under the expected
name.

**Separate latent defect, railiance-platform lane:** the delivered secret has a
trailing newline (74 bytes for a 73-char key; `urllib` refuses it as a header
value outright). It is currently harmless only because
`llm_connect/config.py:53` calls `.strip()` on the env value. Any consumer that
does not strip would fail on a perfectly valid key. Worth fixing in the
OpenBao→ESO lane while the key is being replaced.

## Emergency-pause weekly SBOM fan-out

```task
id: ACTIVITY-WP-0031-T02
status: done
priority: high
```

Disable `weekly-sbom-staleness` in source and on railiance01 before the next
Monday window. The bounded daily replacement remains owned by ACTIVITY-WP-0030;
this task is the immediate containment measure for the observed 75-task fire.

Completed 2026-08-20: source, live DB, and Temporal schedule are disabled. The
new image and ConfigMap projection were deployed; schedule reconciliation
reports the definition as paused.

## Report execution outcomes in production status

```task
id: ACTIVITY-WP-0031-T03
status: done
priority: high
```

Extend `scripts/prod_automation_status.sh` with ops-run outcome counts, bounded
failed/open detail, and daily-triage progress validation evidence. Preserve
sanitized error output and keep the command deterministic and read-only.

Completed 2026-08-20. The live report now shows ops-run state counts, bounded
failure/open detail, and bounded State Hub `daily_triage` validation evidence.
It exposed today's FI/Binky 502 failures and triage `output_validated=false`
beside otherwise healthy schedule-fire counts.

## Reconcile implementation and test contracts

```task
id: ACTIVITY-WP-0031-T04
status: done
priority: medium
```

Repair the four reproducible suite failures, keep runtime-path resolution out of
the rules package boundary, and document that `dedupe_key_strategy` is retained
metadata rather than repeated-content suppression.

Completed 2026-08-20. Runtime URI resolution now happens before the pure rules
boundary; stale assertions/test doubles are reconciled; disabled one-shots are
deleted idempotently during schedule sync; and the dedupe contract is truthful.
Verification: `347 passed, 1 skipped`; a subsequent focused warning regression
passed `10 passed` without warnings.

## Repair consumer routing and State Hub metadata

```task
id: ACTIVITY-WP-0031-T05
status: wait
priority: medium
```

Make the OpenBao retention reactivation task explicitly claimable without
authorizing destructive cleanup. Route Binky's missing workload credential
front door and the duplicate legacy workstream identifiers to their canonical
owners. Clear stale human-needed flags only for already-terminal activity-core
tasks.

Activity-core cleanup is complete: three terminal tasks no longer retain stale
`needs_human` flags. External handoffs were sent to binky-control and
railiance-platform for mail credential materialization, repo-manager for the
duplicate registrar IDs, and railiance-master for the unclaimed retention
closeout. The expired retention one-shot is disabled; its existing open ops run
remains operator-visible and cannot authorize destructive cleanup.

## Acceptance

- [x] Weekly SBOM fan-out is disabled in source and production
- [x] Production status distinguishes schedule fires from execution failures
- [x] Full test suite is green
- [ ] Provider and consumer prerequisites have named owners and evidence
- [x] Workplan and State Hub task state are consistent