activity-core/workplans/ACTIVITY-WP-0031-production-execution-reliability-cleanup.md

---
id: ACTIVITY-WP-0031
type: workplan
title: "Production automation execution reliability cleanup"
domain: infotech
repo: activity-core
status: active
owner: codex
topic_slug: activity-core
created: "2026-08-20"
updated: "2026-09-04"
related:
  - ACTIVITY-WP-0021
  - ACTIVITY-WP-0022
  - ACTIVITY-WP-0026
  - ACTIVITY-WP-0030
state_hub_workstream_id: "93b37937-a29b-505e-adc2-390980cd8ceb"
---

# Production automation execution reliability cleanup

## Goal

Make production automation health reflect completed execution rather than only
successful schedule fires, stop known task floods, and repair the repository
contracts found during the 2026-08-20 activity-core review.

## Restore provider-backed execution

```task
id: ACTIVITY-WP-0031-T01
status: wait
priority: high
state_hub_task_id: "d84116ae-8219-569a-8d7b-e4eb83c192c3"
```

Production FI, Binky, and daily-triage runs reach `llm-connect` but fail after
three attempts. An in-cluster fixture returned a sanitized upstream OpenRouter
HTTP 401. Credential ownership is the `railiance-platform` OpenBao lane
(`openrouter-llm-connect`); activity-core must not bypass that custody boundary.
Track rotation/reconciliation, restart, and a successful fixture smoke as the
exit evidence.

Progress 2026-08-20: railiance-platform repaired the delivery half. The
activity-core ESO bootstrap default now includes the approved exact-path
llm-connect policy, its replacement token reports read capability, and the
reviewed `ExternalSecret` is `Ready=True` / `SecretSynced` on railiance01. It
took ownership of the one-key Secret, llm-connect rolled out, and the real
actcore-worker reaches `/health` through the Service. The post-restart fixture
still returns sanitized OpenRouter HTTP 401, proving the canonical key itself
is rejected. T01 remains `wait` on an attended OpenRouter account owner to mint
and safely provision a replacement key; no key value was read or printed.

Progress 2026-08-21 (activity-core half, no custody boundary crossed): live
evidence confirms a single fault, not two. FI, Binky, and daily triage still
fail today (`05:20` / `05:30` / `06:23` UTC) with `502 Bad Gateway` from
llm-connect — **after** yesterday's rollout (`llm-connect` pod is `Running`
1/1, endpoints healthy). That 502 is not a dead gateway:
`llm_connect/server.py::_error_response` maps **every** `LLMAPIError` onto 502
and carries the real cause in the body (`error`, `provider_status`). So the
502s and the reported upstream 401 are the same rejected key.

`LLMConnectClient.complete` called `raise_for_status()` and discarded that
body, so four days of production evidence read as "llm-connect is down" when it
meant "the provider rejected the key". Fixed: the client now raises with
`error`, `provider_status`, `provider`, `model`, and a bounded copy of
llm-connect's already-sanitized `message`, under a field allowlist so no
provider blob or key material can reach the run artefact. Five tests cover the
401-behind-502 case, the allowlist, bounding, and unusable bodies.

The identical body-discarding call exists in
`rein-aharness/rein_aharness/llm_connect_client.py:65`, which is what produces
the opaque ops_run failure text in the status table. Not ours to edit —
handed to rein-aharness.

T01 still `wait`: the key rotation itself remains attended and
railiance-platform-owned. What changed is that the next failure will name its
own cause.

Diagnosis verified 2026-08-21 (previously relayed, now tested). Direct probe
from the llm-connect pod, no secret read or printed:

- `POST /execute` → `502 {"error":"provider_api_error","provider_status":401,
  "message":"HTTP 401 from https://openrouter.ai/api/v1/chat/completions"}`
- `GET https://openrouter.ai/api/v1/key` with the delivered key →
  **`401 {"error":{"message":"User not found.","code":401}}`**

"User not found" is OpenRouter's wording for a key that resolves to **no
account**. It is not insufficient credits (402) and not a model-permission
problem (403), so **no new OpenRouter account is required** — the existing
account needs a new key issued, because this key value is dead.
Key fingerprint for identification: `sha256[:12] = ab938241a2ec`, length 73
(+1, see below), prefix `sk-or-v1-`.

Ruled out along the way: provider/model config is correct
(`LLM_CONNECT_PROVIDER=openrouter`, `google/gemini-2.5-flash`); the Service and
pod are healthy; the ExternalSecret delivers exactly one key under the expected
name.

**Separate latent defect, railiance-platform lane:** the delivered secret has a
trailing newline (74 bytes for a 73-char key; `urllib` refuses it as a header
value outright). It is currently harmless only because
`llm_connect/config.py:53` calls `.strip()` on the env value. Any consumer that
does not strip would fail on a perfectly valid key. Worth fixing in the
OpenBao→ESO lane while the key is being replaced.

Rechecked 2026-09-04 without reading or printing the value. The canonical
OpenBao record is still KV version 1 created 2026-07-02. Its ESO-managed
Kubernetes Secret and the running `llm-connect` process have the same 74-byte
fingerprint `ab938241a2ec`; production continues to return sanitized upstream
401. Thus no replacement is waiting for a pod restart. The only other cataloged
OpenRouter lane is scoped to `rein-openweights` and is not transferable without
an ownership decision. Reconciliation request
`c32901ff-2072-4b93-81cd-db93dddfa5f3` asks railiance-platform to install a
newly issued, newline-free key at the canonical path, allow ESO to sync,
restart `llm-connect`, and return value-free fixture evidence.

## Emergency-pause weekly SBOM fan-out

```task
id: ACTIVITY-WP-0031-T02
status: done
priority: high
state_hub_task_id: "53c16356-b80d-50f7-a4a1-b70b67eb0178"
```

Disable `weekly-sbom-staleness` in source and on railiance01 before the next
Monday window. The bounded daily replacement remains owned by ACTIVITY-WP-0030;
this task is the immediate containment measure for the observed 75-task fire.

Completed 2026-08-20: source, live DB, and Temporal schedule are disabled. The
new image and ConfigMap projection were deployed; schedule reconciliation
reports the definition as paused.

## Report execution outcomes in production status

```task
id: ACTIVITY-WP-0031-T03
status: done
priority: high
state_hub_task_id: "d70cfca0-3c3b-5bfd-abb9-3f07b52b2c9b"
```

Extend `scripts/prod_automation_status.sh` with ops-run outcome counts, bounded
failed/open detail, and daily-triage progress validation evidence. Preserve
sanitized error output and keep the command deterministic and read-only.

Completed 2026-08-20. The live report now shows ops-run state counts, bounded
failure/open detail, and bounded State Hub `daily_triage` validation evidence.
It exposed today's FI/Binky 502 failures and triage `output_validated=false`
beside otherwise healthy schedule-fire counts.

## Reconcile implementation and test contracts

```task
id: ACTIVITY-WP-0031-T04
status: done
priority: medium
state_hub_task_id: "4f44ec1b-65fb-56bb-9fdf-a7705434e09b"
```

Repair the four reproducible suite failures, keep runtime-path resolution out of
the rules package boundary, and document that `dedupe_key_strategy` is retained
metadata rather than repeated-content suppression.

Completed 2026-08-20. Runtime URI resolution now happens before the pure rules
boundary; stale assertions/test doubles are reconciled; disabled one-shots are
deleted idempotently during schedule sync; and the dedupe contract is truthful.
Verification: `347 passed, 1 skipped`; a subsequent focused warning regression
passed `10 passed` without warnings.

## Repair consumer routing and State Hub metadata

```task
id: ACTIVITY-WP-0031-T05
status: wait
priority: medium
state_hub_task_id: "a7eb3615-dd11-54c9-80ec-a147d100f8b7"
```

Make the OpenBao retention reactivation task explicitly claimable without
authorizing destructive cleanup. Route Binky's missing workload credential
front door and the duplicate legacy workstream identifiers to their canonical
owners. Clear stale human-needed flags only for already-terminal activity-core
tasks.

Activity-core cleanup is complete: three terminal tasks no longer retain stale
`needs_human` flags. External handoffs were sent to binky-control and
railiance-platform for mail credential materialization, repo-manager for the
duplicate registrar IDs, and railiance-master for the unclaimed retention
closeout. The expired retention one-shot is disabled; its existing open ops run
remains operator-visible and cannot authorize destructive cleanup.

Progress 2026-09-04: the Binky mail lane was already provisioned; its runtime
discovery was broken. OpenBao was healthy, OpenBao v2.5.4 existed at
`/home/tegwick/.local/bin/bao`, and the mode-0600 AppRole files existed under
`~/.local/rein-aharness/approle-binky-mail`, but the rein claim-loop omitted
both locations. `rein-aharness` commit `aedfadd` now supplies the non-secret
OpenBao/AppRole defaults and includes `~/.local/bin` in `PATH`. The corrected
unit is live and healthy. A value-free smoke proved `read` on the exact Binky
IMAP path, `deny` on a sibling path, and presence of both required IMAP fields;
the rein suite passed 224 tests with one skip. T05 remains `wait` only for its
other named external metadata/retention handoffs.

## Acceptance

- [x] Weekly SBOM fan-out is disabled in source and production
- [x] Production status distinguishes schedule fires from execution failures
- [x] Full test suite is green
- [ ] Provider and consumer prerequisites have named owners and evidence
- [x] Workplan and State Hub task state are consistent