coulomb/activity-core
4
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
activity-core/docs/edge-relay-resilience.md
tegwick 4e605a6839
All checks were successful
CI Smoke / host-smoke (push) Successful in 0s
Details
CI Smoke / container-smoke (push) Successful in 1s
Details
Build and Publish Container Image / build-and-push (push) Successful in 33s
Details
fix(edge): retry+degrade side-effect POSTs; finish ACTIVITY-WP-0027 
Consistency sweep and recently-on-scope retry transient 502/503/504 then
return a degraded context snapshot instead of thrashing Temporal.
Document edge-relay resilience. Retire Binky dual-clock host timers after
claim-loop smoke.
2026-08-05 17:53:00 +02:00

3.2 KiB
Raw Blame History

Edge relay resilience (ACTIVITY-WP-0027-T06)

Audience: operators and agents debugging high-frequency activity-core jobs
Scope: railiance01 actcore-statehub-edge-relay + worker STATE_HUB_URL

Topology

actcore-worker
  STATE_HUB_URL=http://actcore-statehub-edge-relay:8000
        │
        ▼
  edge relay (cluster)
    · GET allowlist → upstream or stale read-cache
    · POST side-effects → upstream (or 503 if busy/unreachable)
        │
        ▼
  state-hub service (cluster DNS state-hub.state-hub.svc …)
    · may depend on tunnel / central hub posture

Host rein-aharness should prefer the same edge ClusterIP (or k8s://…edge-relay:8000) so completion events outbox when central hub is flaky — see docs/llm-connect-host-access.md.

Resolver classes

Class Examples Hard-fail? Notes
GET reads fi_brief_status, domain_summary, state_summary Soft by default ({} / empty) unless required: true Edge may serve stale cache (X-StateHub-Edge-Cache: stale)
Side-effect POSTs consistency_sweep_remote_all, recently_on_scope_hourly After retries: degrade by default See env below
Required pure reads todo_md_staleness when enabled Yes if required: true Prefer not to make hub-only reads required without cache

Side-effect POST behaviour (worker)

Implemented in context_resolvers/state_hub.py (_post_json):

  1. Retry transient 502/503/504 and network/timeout errors
    (STATE_HUB_POST_RETRIES, default 3; backoff STATE_HUB_POST_RETRY_BACKOFF_SECONDS, default 2s × attempt).
  2. If still failing and degrade is on (STATE_HUB_SIDE_EFFECT_DEGRADE default true, or per-source params.degrade_on_unavailable):
    • consistency_sweep: returns {exit_code: 75, lock_skipped: true, repos_processed: [], degraded: true, …} so the workflow completes without Temporal ApplicationError storms.
    • recently_on_scope: returns {generated: [], failed: [{reason: edge_unavailable}], degraded: true, …}.
  3. Set degrade_on_unavailable: false on a definition (or env STATE_HUB_SIDE_EFFECT_DEGRADE=false) to keep hard-fail + Temporal retry after in-resolver attempts.

Operator checks

# Edge health (from host with ClusterIP or kubectl exec)
curl -sS http://<edge-cluster-ip>:8000/edge/health | python3 -m json.tool

# Worker still pointing at edge
kubectl -n activity-core exec deploy/actcore-worker -- env | grep STATE_HUB

# Recent consistency fires should COMPLETE even when edge was briefly 503
# (look for degraded:true in context_snapshot if all retries failed)

What edge 503 usually means

  • Upstream hub slow / lock held / temporary overload on /consistency/sweep/remote-all (large fleet).
  • Upstream unreachable; edge refuses to invent a false success for POSTs.
  • Outbox backlog (see edge/health → outbox.pending_count) is write backlog for queued progress — separate from sweep POST 503.

Non-goals

  • Making the edge invent successful sweep results when nothing ran.
  • Public exposure of State Hub or edge.
  • Replacing Temporal schedules with workstation heartbeats.