fix(edge): retry+degrade side-effect POSTs; finish ACTIVITY-WP-0027
All checks were successful
CI Smoke / host-smoke (push) Successful in 0s
CI Smoke / container-smoke (push) Successful in 1s
Build and Publish Container Image / build-and-push (push) Successful in 33s

Consistency sweep and recently-on-scope retry transient 502/503/504 then
return a degraded context snapshot instead of thrashing Temporal.
Document edge-relay resilience. Retire Binky dual-clock host timers after
claim-loop smoke.
This commit is contained in:
tegwick 2026-08-05 17:53:00 +02:00
parent 2b3fc6a556
commit 4e605a6839
6 changed files with 334 additions and 20 deletions

View file

@ -0,0 +1,77 @@
# Edge relay resilience (ACTIVITY-WP-0027-T06)
**Audience:** operators and agents debugging high-frequency activity-core jobs
**Scope:** railiance01 `actcore-statehub-edge-relay` + worker `STATE_HUB_URL`
## Topology
```text
actcore-worker
STATE_HUB_URL=http://actcore-statehub-edge-relay:8000
edge relay (cluster)
· GET allowlist → upstream or stale read-cache
· POST side-effects → upstream (or 503 if busy/unreachable)
state-hub service (cluster DNS state-hub.state-hub.svc …)
· may depend on tunnel / central hub posture
```
Host rein-aharness should prefer the **same edge ClusterIP** (or `k8s://…edge-relay:8000`)
so completion events outbox when central hub is flaky — see
`docs/llm-connect-host-access.md`.
## Resolver classes
| Class | Examples | Hard-fail? | Notes |
| ----- | -------- | ---------- | ----- |
| **GET reads** | `fi_brief_status`, `domain_summary`, `state_summary` | Soft by default (`{}` / empty) unless `required: true` | Edge may serve **stale cache** (`X-StateHub-Edge-Cache: stale`) |
| **Side-effect POSTs** | `consistency_sweep_remote_all`, `recently_on_scope_hourly` | After retries: **degrade** by default | See env below |
| **Required pure reads** | `todo_md_staleness` when enabled | Yes if `required: true` | Prefer not to make hub-only reads required without cache |
## Side-effect POST behaviour (worker)
Implemented in `context_resolvers/state_hub.py` (`_post_json`):
1. **Retry** transient `502/503/504` and network/timeout errors
(`STATE_HUB_POST_RETRIES`, default **3**; backoff
`STATE_HUB_POST_RETRY_BACKOFF_SECONDS`, default **2s** × attempt).
2. If still failing and degrade is on (`STATE_HUB_SIDE_EFFECT_DEGRADE` default
**true**, or per-source `params.degrade_on_unavailable`):
- **consistency_sweep:** returns
`{exit_code: 75, lock_skipped: true, repos_processed: [], degraded: true, …}`
so the workflow **completes** without Temporal ApplicationError storms.
- **recently_on_scope:** returns
`{generated: [], failed: [{reason: edge_unavailable}], degraded: true, …}`.
3. Set `degrade_on_unavailable: false` on a definition (or env
`STATE_HUB_SIDE_EFFECT_DEGRADE=false`) to keep hard-fail + Temporal retry
after in-resolver attempts.
## Operator checks
```bash
# Edge health (from host with ClusterIP or kubectl exec)
curl -sS http://<edge-cluster-ip>:8000/edge/health | python3 -m json.tool
# Worker still pointing at edge
kubectl -n activity-core exec deploy/actcore-worker -- env | grep STATE_HUB
# Recent consistency fires should COMPLETE even when edge was briefly 503
# (look for degraded:true in context_snapshot if all retries failed)
```
## What edge 503 usually means
- Upstream hub slow / lock held / temporary overload on
`/consistency/sweep/remote-all` (large fleet).
- Upstream unreachable; edge refuses to invent a false success for POSTs.
- Outbox backlog (see `edge/health``outbox.pending_count`) is **write**
backlog for queued progress — separate from sweep POST 503.
## Non-goals
- Making the edge invent successful sweep results when nothing ran.
- Public exposure of State Hub or edge.
- Replacing Temporal schedules with workstation heartbeats.

View file

@ -50,7 +50,9 @@ activity-core also schedules it — dual cadence and forgotten design.
**Primary (Binky / FI):** activity-core schedule → `ops_run` → rein-aharness
**claim loop** on railiance01. Host user-systemd timers are **break-glass only**
(not cadence authority). FI dual-clock timer was disabled 2026-08-05 after the
ops_run per-fire idempotency fix landed; re-enable only if claim loop is down.
ops_run per-fire idempotency fix; **Binky dual-clock timers** (daily/mail/review)
were disabled 2026-08-05 after claim-loop smoke (ACTIVITY-WP-0027-T07). Re-enable
only if claim loop is down.
---

View file

@ -736,6 +736,11 @@ unreachable, which keeps daily triage context resolution alive during outages.
> receipts as successful sink delivery pending replay, and consumes stale cached
> reads transparently.
Side-effect POSTs (`consistency_sweep_remote_all`, `recently_on_scope_hourly`)
retry transient 502/503/504 then **degrade** by default so required workflows do
not thrash Temporal when the edge is briefly unavailable. Details:
`docs/edge-relay-resilience.md` (ACTIVITY-WP-0027-T06).
## Troubleshooting
### Worker fails to start: "ACTCORE_DB_URL is required"