activity-core/workplans/ACTIVITY-WP-0031-production-execution-reliability-cleanup.md

id	type	title	domain	repo	status	owner	topic_slug	created	updated	related	state_hub_workstream_id
ACTIVITY-WP-0031	workplan	Production automation execution reliability cleanup	infotech	activity-core	active	codex	activity-core	2026-08-20	2026-09-04	ACTIVITY-WP-0021 ACTIVITY-WP-0022 ACTIVITY-WP-0026 ACTIVITY-WP-0030	93b37937-a29b-505e-adc2-390980cd8ceb

Production automation execution reliability cleanup

Goal

Make production automation health reflect completed execution rather than only successful schedule fires, stop known task floods, and repair the repository contracts found during the 2026-08-20 activity-core review.

Restore provider-backed execution

id: ACTIVITY-WP-0031-T01
status: wait
priority: high
state_hub_task_id: "d84116ae-8219-569a-8d7b-e4eb83c192c3"

Production FI, Binky, and daily-triage runs reach llm-connect but fail after three attempts. An in-cluster fixture returned a sanitized upstream OpenRouter HTTP 401. Credential ownership is the railiance-platform OpenBao lane (openrouter-llm-connect); activity-core must not bypass that custody boundary. Track rotation/reconciliation, restart, and a successful fixture smoke as the exit evidence.

Progress 2026-08-20: railiance-platform repaired the delivery half. The activity-core ESO bootstrap default now includes the approved exact-path llm-connect policy, its replacement token reports read capability, and the reviewed ExternalSecret is Ready=True / SecretSynced on railiance01. It took ownership of the one-key Secret, llm-connect rolled out, and the real actcore-worker reaches /health through the Service. The post-restart fixture still returns sanitized OpenRouter HTTP 401, proving the canonical key itself is rejected. T01 remains wait on an attended OpenRouter account owner to mint and safely provision a replacement key; no key value was read or printed.

Progress 2026-08-21 (activity-core half, no custody boundary crossed): live evidence confirms a single fault, not two. FI, Binky, and daily triage still fail today (05:20 / 05:30 / 06:23 UTC) with 502 Bad Gateway from llm-connect — after yesterday's rollout (llm-connect pod is Running 1/1, endpoints healthy). That 502 is not a dead gateway: llm_connect/server.py::_error_response maps every LLMAPIError onto 502 and carries the real cause in the body (error, provider_status). So the 502s and the reported upstream 401 are the same rejected key.

LLMConnectClient.complete called raise_for_status() and discarded that body, so four days of production evidence read as "llm-connect is down" when it meant "the provider rejected the key". Fixed: the client now raises with error, provider_status, provider, model, and a bounded copy of llm-connect's already-sanitized message, under a field allowlist so no provider blob or key material can reach the run artefact. Five tests cover the 401-behind-502 case, the allowlist, bounding, and unusable bodies.

The identical body-discarding call exists in rein-aharness/rein_aharness/llm_connect_client.py:65, which is what produces the opaque ops_run failure text in the status table. Not ours to edit — handed to rein-aharness.

T01 still wait: the key rotation itself remains attended and railiance-platform-owned. What changed is that the next failure will name its own cause.

Diagnosis verified 2026-08-21 (previously relayed, now tested). Direct probe from the llm-connect pod, no secret read or printed:

• POST /execute → 502 {"error":"provider_api_error","provider_status":401, "message":"HTTP 401 from https://openrouter.ai/api/v1/chat/completions"}
• GET https://openrouter.ai/api/v1/key with the delivered key → 401 {"error":{"message":"User not found.","code":401}}

"User not found" is OpenRouter's wording for a key that resolves to no account. It is not insufficient credits (402) and not a model-permission problem (403), so no new OpenRouter account is required — the existing account needs a new key issued, because this key value is dead. Key fingerprint for identification: sha256[:12] = ab938241a2ec, length 73 (+1, see below), prefix sk-or-v1-.

Ruled out along the way: provider/model config is correct (LLM_CONNECT_PROVIDER=openrouter, google/gemini-2.5-flash); the Service and pod are healthy; the ExternalSecret delivers exactly one key under the expected name.

Separate latent defect, railiance-platform lane: the delivered secret has a trailing newline (74 bytes for a 73-char key; urllib refuses it as a header value outright). It is currently harmless only because llm_connect/config.py:53 calls .strip() on the env value. Any consumer that does not strip would fail on a perfectly valid key. Worth fixing in the OpenBao→ESO lane while the key is being replaced.

Rechecked 2026-09-04 without reading or printing the value. The canonical OpenBao record is still KV version 1 created 2026-07-02. Its ESO-managed Kubernetes Secret and the running llm-connect process have the same 74-byte fingerprint ab938241a2ec; production continues to return sanitized upstream 401. Thus no replacement is waiting for a pod restart. The only other cataloged OpenRouter lane is scoped to rein-openweights and is not transferable without an ownership decision. Reconciliation request c32901ff-2072-4b93-81cd-db93dddfa5f3 asks railiance-platform to install a newly issued, newline-free key at the canonical path, allow ESO to sync, restart llm-connect, and return value-free fixture evidence.

Emergency-pause weekly SBOM fan-out

id: ACTIVITY-WP-0031-T02
status: done
priority: high
state_hub_task_id: "53c16356-b80d-50f7-a4a1-b70b67eb0178"

Disable weekly-sbom-staleness in source and on railiance01 before the next Monday window. The bounded daily replacement remains owned by ACTIVITY-WP-0030; this task is the immediate containment measure for the observed 75-task fire.

Completed 2026-08-20: source, live DB, and Temporal schedule are disabled. The new image and ConfigMap projection were deployed; schedule reconciliation reports the definition as paused.

Report execution outcomes in production status

id: ACTIVITY-WP-0031-T03
status: done
priority: high
state_hub_task_id: "d70cfca0-3c3b-5bfd-abb9-3f07b52b2c9b"

Extend scripts/prod_automation_status.sh with ops-run outcome counts, bounded failed/open detail, and daily-triage progress validation evidence. Preserve sanitized error output and keep the command deterministic and read-only.

Completed 2026-08-20. The live report now shows ops-run state counts, bounded failure/open detail, and bounded State Hub daily_triage validation evidence. It exposed today's FI/Binky 502 failures and triage output_validated=false beside otherwise healthy schedule-fire counts.

Reconcile implementation and test contracts

id: ACTIVITY-WP-0031-T04
status: done
priority: medium
state_hub_task_id: "4f44ec1b-65fb-56bb-9fdf-a7705434e09b"

Repair the four reproducible suite failures, keep runtime-path resolution out of the rules package boundary, and document that dedupe_key_strategy is retained metadata rather than repeated-content suppression.

Completed 2026-08-20. Runtime URI resolution now happens before the pure rules boundary; stale assertions/test doubles are reconciled; disabled one-shots are deleted idempotently during schedule sync; and the dedupe contract is truthful. Verification: 347 passed, 1 skipped; a subsequent focused warning regression passed 10 passed without warnings.

Repair consumer routing and State Hub metadata

id: ACTIVITY-WP-0031-T05
status: wait
priority: medium
state_hub_task_id: "a7eb3615-dd11-54c9-80ec-a147d100f8b7"

Make the OpenBao retention reactivation task explicitly claimable without authorizing destructive cleanup. Route Binky's missing workload credential front door and the duplicate legacy workstream identifiers to their canonical owners. Clear stale human-needed flags only for already-terminal activity-core tasks.

Activity-core cleanup is complete: three terminal tasks no longer retain stale needs_human flags. External handoffs were sent to binky-control and railiance-platform for mail credential materialization, repo-manager for the duplicate registrar IDs, and railiance-master for the unclaimed retention closeout. The expired retention one-shot is disabled; its existing open ops run remains operator-visible and cannot authorize destructive cleanup.

Progress 2026-09-04: the Binky mail lane was already provisioned; its runtime discovery was broken. OpenBao was healthy, OpenBao v2.5.4 existed at /home/tegwick/.local/bin/bao, and the mode-0600 AppRole files existed under ~/.local/rein-aharness/approle-binky-mail, but the rein claim-loop omitted both locations. rein-aharness commit aedfadd now supplies the non-secret OpenBao/AppRole defaults and includes ~/.local/bin in PATH. The corrected unit is live and healthy. A value-free smoke proved read on the exact Binky IMAP path, deny on a sibling path, and presence of both required IMAP fields; the rein suite passed 224 tests with one skip. T05 remains wait only for its other named external metadata/retention handoffs.

Acceptance

• Weekly SBOM fan-out is disabled in source and production
• Production status distinguishes schedule fires from execution failures
• Full test suite is green
• Provider and consumer prerequisites have named owners and evidence
• Workplan and State Hub task state are consistent