feat(railiance): adopt state-hub edge relay beachhead for WP-0015
All checks were successful
CI Smoke / host-smoke (push) Successful in 0s
CI Smoke / container-smoke (push) Successful in 2s
Build and Publish Container Image / build-and-push (push) Successful in 1m12s

Deploy actcore-statehub-edge-relay, point STATE_HUB_URL at it, retire the
bespoke state-hub bridge, and accept edge-relay queued write receipts in
report and ops evidence sinks.
This commit is contained in:
tegwick 2026-07-09 01:04:24 +02:00
parent 1a0636845a
commit cdbe4de2bf
13 changed files with 256 additions and 136 deletions

View file

@ -445,12 +445,15 @@ events once State Hub / the beachhead honours the header.
The guarantee lives on the write, not on a live dedup read. The read-based
`_progress_exists` check is now best-effort only: if State Hub is unreachable it
returns `False` (proceed to the keyed write) rather than hard-failing. The header
passes untouched through the `actcore-state-hub-bridge` proxy and is ignored by
State Hub versions that do not yet honour it.
is honoured by the in-cluster `actcore-statehub-edge-relay` (state-hub edge
relay) and central State Hub on replay. Allowlisted `GET` reads are cached by the
relay and served stale (`X-StateHub-Edge-Cache: stale`) when upstream is briefly
unreachable, which keeps daily triage context resolution alive during outages.
> The queue/cache itself is **not** built in activity-core — it belongs to the
> state-hub beachhead. activity-core only emits the key. See the proposal sent to
> the `state-hub` agent.
> state-hub edge relay. activity-core emits the key, treats HTTP 202 queued
> receipts as successful sink delivery pending replay, and consumes stale cached
> reads transparently.
## Troubleshooting

View file

@ -10,7 +10,7 @@ data:
TEMPORAL_HOST: actcore-temporal:7233
TEMPORAL_NAMESPACE: default
NATS_URL: nats://actcore-nats:4222
STATE_HUB_URL: http://state-hub.state-hub.svc.cluster.local:8000
STATE_HUB_URL: http://actcore-statehub-edge-relay:8000
LLM_CONNECT_URL: http://llm-connect.activity-core.svc.cluster.local:8080
LLM_CONNECT_TIMEOUT_SECONDS: "300"
REPO_SCOPING_URL: http://repo-scoping.repo-scoping.svc.cluster.local:8020
@ -572,28 +572,29 @@ data:
kind: coordination-service
lifecycle_state: observed
health_status: observed_ok
environment: local
environment: railiance01
owner_repos:
- state-hub
- the-custodian
runtime:
type: local-process
host: local-workstation
type: k3s
cluster: railiance01-k3s
namespace: state-hub
endpoints:
- id: state-hub-local-api
- id: state-hub-edge-relay-health
type: http
url: "http://actcore-state-hub-bridge:8000/state/health"
url: "http://actcore-statehub-edge-relay:8000/edge/health"
expected_status: 200
expected_signal: "health response"
expected_signal: "edge relay health"
backing_stores:
- "postgresql:state-hub"
access_paths:
- type: http
target: "http://actcore-state-hub-bridge:8000"
target: "http://actcore-statehub-edge-relay:8000"
status: observed_ok
evidence: []
gaps:
- "Future cluster deployment readiness still needs ops evidence."
- "Overnight triage proof after relay deploy still needs operator evidence."
- id: inter-hub
name: "Inter-Hub"
kind: governance-service
@ -774,16 +775,31 @@ spec:
storage: 1Gi
---
apiVersion: v1
kind: Service
kind: PersistentVolumeClaim
metadata:
name: actcore-state-hub-bridge
name: actcore-statehub-edge-outbox
namespace: activity-core
labels:
app.kubernetes.io/name: actcore-state-hub-bridge
app.kubernetes.io/name: actcore-statehub-edge-relay
app.kubernetes.io/part-of: activity-core
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
---
apiVersion: v1
kind: Service
metadata:
name: actcore-statehub-edge-relay
namespace: activity-core
labels:
app.kubernetes.io/name: actcore-statehub-edge-relay
app.kubernetes.io/part-of: activity-core
spec:
selector:
app.kubernetes.io/name: actcore-state-hub-bridge
app.kubernetes.io/name: actcore-statehub-edge-relay
ports:
- name: http
port: 8000
@ -792,97 +808,75 @@ spec:
apiVersion: apps/v1
kind: Deployment
metadata:
name: actcore-state-hub-bridge
name: actcore-statehub-edge-relay
namespace: activity-core
labels:
app.kubernetes.io/name: actcore-state-hub-bridge
app.kubernetes.io/name: actcore-statehub-edge-relay
app.kubernetes.io/part-of: activity-core
spec:
replicas: 0
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: actcore-state-hub-bridge
app.kubernetes.io/name: actcore-statehub-edge-relay
template:
metadata:
labels:
app.kubernetes.io/name: actcore-state-hub-bridge
app.kubernetes.io/name: actcore-statehub-edge-relay
app.kubernetes.io/part-of: activity-core
spec:
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet
containers:
- name: proxy
image: activity-core:railiance01-prod
imagePullPolicy: Never
- name: relay
# Published by Forgejo CI on state-hub main (edge read-cache, 1cf949b).
image: forgejo.coulomb.social/coulomb/state-hub:main-1cf949b
imagePullPolicy: IfNotPresent
ports:
- name: http
containerPort: 18080
containerPort: 8000
env:
- name: STATEHUB_UPSTREAM_URL
value: http://state-hub.state-hub.svc.cluster.local:8000
- name: STATEHUB_OUTBOX_PATH
value: /var/statehub/edge-outbox.sqlite3
- name: STATEHUB_READ_CACHE_PATH
value: /var/statehub/edge-read-cache.sqlite3
command:
- python
- -c
- |
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from urllib.error import HTTPError, URLError
from urllib.request import Request, urlopen
TARGET = "http://127.0.0.1:18000"
HOP_HEADERS = {"connection", "host", "keep-alive", "proxy-authenticate",
"proxy-authorization", "te", "trailers",
"transfer-encoding", "upgrade"}
class Proxy(BaseHTTPRequestHandler):
def do_GET(self):
self._proxy()
def do_POST(self):
self._proxy()
def do_PATCH(self):
self._proxy()
def _proxy(self):
length = int(self.headers.get("content-length", "0") or "0")
body = self.rfile.read(length) if length else None
headers = {
key: value
for key, value in self.headers.items()
if key.lower() not in HOP_HEADERS
}
request = Request(
TARGET + self.path,
data=body,
headers=headers,
method=self.command,
)
try:
timeout = 360 if self.command == "POST" else 30
with urlopen(request, timeout=timeout) as response:
payload = response.read()
self.send_response(response.status)
for key, value in response.headers.items():
if key.lower() not in HOP_HEADERS:
self.send_header(key, value)
self.end_headers()
self.wfile.write(payload)
except HTTPError as exc:
payload = exc.read()
self.send_response(exc.code)
self.end_headers()
self.wfile.write(payload)
except URLError as exc:
self.send_response(502)
self.end_headers()
self.wfile.write(str(exc).encode())
ThreadingHTTPServer(("0.0.0.0", 18080), Proxy).serve_forever()
- uvicorn
- api.edge.relay:app
- --host
- 0.0.0.0
- --port
- "8000"
volumeMounts:
- name: edge-outbox
mountPath: /var/statehub
readinessProbe:
httpGet:
path: /state/health
path: /edge/health
port: http
initialDelaySeconds: 5
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 6
livenessProbe:
httpGet:
path: /edge/health
port: http
initialDelaySeconds: 15
periodSeconds: 30
timeoutSeconds: 5
failureThreshold: 3
resources:
requests:
cpu: 50m
memory: 128Mi
limits:
cpu: 500m
memory: 512Mi
volumes:
- name: edge-outbox
persistentVolumeClaim:
claimName: actcore-statehub-edge-outbox
---
apiVersion: v1
kind: Service
metadata:

View file

@ -43,8 +43,11 @@ and a persistent working-memory volume mounted at
Before trusting the daily 07:20
Europe/Berlin schedule, verify both runtime dependencies:
- `actcore-state-hub-bridge` can reach the State Hub API through the node-local
tunnel expected at `127.0.0.1:18000`.
- `actcore-statehub-edge-relay` is ready and reports upstream reachability at
`GET /edge/health` (upstream is the in-cluster State Hub API at
`state-hub.state-hub.svc.cluster.local:8000`). `STATE_HUB_URL` points at the
relay so allowlisted `GET` reads can be served from cache during brief upstream
outages and queueable writes survive until replay.
- `LLM_CONNECT_URL` points at the verified in-namespace llm-connect Service,
`http://llm-connect.activity-core.svc.cluster.local:8080`, and the
operator-owned provider Secret lets that Service serve the

View file

@ -18,8 +18,10 @@ Supported queries:
- phase5_stabilization_check: hub-visible Phase 5 stabilization gates
- legacy_meter_weekly_review: GET {STATE_HUB_URL}/legacy-meter/weekly-review
No caching state hub data is live operational state and must not be stale
within a single workflow run.
When STATE_HUB_URL points at the state-hub edge relay, allowlisted GET reads may
be served from a stale local cache during upstream outages (`X-StateHub-Edge-Cache:
stale`). activity-core treats those as ordinary successful reads so workflows can
continue with last-known hub state.
Config: STATE_HUB_URL env var (default: http://127.0.0.1:8000).
"""

View file

@ -10,7 +10,11 @@ from typing import Any
import httpx
from activity_core.context_resolvers.ops_inventory import _sanitize_url
from activity_core.state_hub_write import apply_progress_scope_fields, idempotency_headers
from activity_core.state_hub_write import (
apply_progress_scope_fields,
idempotency_headers,
parse_state_hub_write_response,
)
_DEFAULT_STATE_HUB_URL = "http://127.0.0.1:8000"
_INTER_HUB_SINK_TYPES = {
@ -161,8 +165,16 @@ def _post_state_hub_progress(
headers=idempotency_headers(run_id, context_key, event_type),
timeout=float(sink.get("timeout_seconds", 10.0)),
)
resp.raise_for_status()
data = resp.json()
data = parse_state_hub_write_response(resp)
if data.get("queued"):
return {
"type": "state-hub-progress",
"status": "queued",
"event_type": event_type,
"outbox_id": data.get("outbox_id"),
"idempotency_key": data.get("idempotency_key") or idempotency_key,
"context_key": context_key,
}
return {
"type": "state-hub-progress",
"status": "posted",

View file

@ -16,7 +16,11 @@ from activity_core.runtime_paths import (
custodian_repo_root,
resolve_runtime_path,
)
from activity_core.state_hub_write import apply_progress_scope_fields, idempotency_headers
from activity_core.state_hub_write import (
apply_progress_scope_fields,
idempotency_headers,
parse_state_hub_write_response,
)
_DEFAULT_STATE_HUB_URL = "http://127.0.0.1:8000"
@ -154,8 +158,15 @@ def _post_state_hub_progress(
headers=idempotency_headers(run_id, instruction_id, event_type),
timeout=float(sink.get("timeout_seconds", 10.0)),
)
resp.raise_for_status()
data = resp.json()
data = parse_state_hub_write_response(resp)
if data.get("queued"):
return {
"type": "state-hub-progress",
"status": "queued",
"event_type": event_type,
"outbox_id": data.get("outbox_id"),
"idempotency_key": data.get("idempotency_key"),
}
return {
"type": "state-hub-progress",
"status": "posted",

View file

@ -24,7 +24,7 @@ from uuid import UUID
import httpx
from activity_core.schedule_manager import schedule_id
from activity_core.state_hub_write import idempotency_headers
from activity_core.state_hub_write import idempotency_headers, parse_state_hub_write_response
_DEFAULT_STATE_HUB_URL = "http://127.0.0.1:8000"
@ -187,8 +187,14 @@ def post_missed_fire_alert(
headers=idempotency_headers("schedule_miss", health.activity_id, last_fired),
timeout=timeout_seconds,
)
resp.raise_for_status()
data = resp.json()
data = parse_state_hub_write_response(resp)
if data.get("queued"):
return {
"type": "schedule-miss-alert",
"status": "queued",
"outbox_id": data.get("outbox_id"),
"idempotency_key": data.get("idempotency_key"),
}
return {
"type": "schedule-miss-alert",
"status": "posted",

View file

@ -8,15 +8,15 @@ write's identity. The guarantee lives on the write itself and does **not** depen
on a live dedup read, so it holds even when the beachhead is serving offline.
activity-core does not implement the queue/cache (that is state-hub's beachhead);
it only emits the key so the beachhead / State Hub can dedup on flush. The header
passes untouched through the existing ``actcore-state-hub-bridge`` proxy and is
ignored by State Hub versions that do not yet honour it.
it only emits the key so the beachhead / State Hub can dedup on flush.
"""
from __future__ import annotations
from typing import Any
import httpx
IDEMPOTENCY_HEADER = "Idempotency-Key"
@ -48,3 +48,13 @@ def idempotency_key(*parts: str | None) -> str:
def idempotency_headers(*parts: str | None) -> dict[str, str]:
"""Return the header dict to attach to a State Hub write."""
return {IDEMPOTENCY_HEADER: idempotency_key(*parts)}
def parse_state_hub_write_response(resp: httpx.Response) -> dict[str, Any]:
"""Normalize a State Hub write response, including edge-relay queued receipts."""
if resp.status_code == 202:
data = resp.json()
if data.get("queued"):
return data
resp.raise_for_status()
return resp.json()

View file

@ -9,8 +9,9 @@ from activity_core.ops_evidence_sinks import persist_ops_inventory_evidence
class DummyResponse:
def __init__(self, payload: Any) -> None:
def __init__(self, payload: Any, *, status_code: int = 200) -> None:
self.payload = payload
self.status_code = status_code
def raise_for_status(self) -> None:
return None

View file

@ -33,6 +33,7 @@ def _by_kind_name(kind: str, name: str) -> dict[str, Any]:
def test_runtime_config_has_ops_inventory_placeholders() -> None:
config = _by_kind_name("ConfigMap", "actcore-runtime-config")
assert config["data"]["STATE_HUB_URL"] == "http://actcore-statehub-edge-relay:8000"
assert config["data"]["LLM_CONNECT_URL"] == (
"http://llm-connect.activity-core.svc.cluster.local:8080"
)
@ -112,11 +113,9 @@ def test_external_configmap_projects_enabled_daily_wsjf_definition(tmp_path) ->
assert definition.trigger_config["timezone"] == "Europe/Berlin"
assert instruction["id"] == "daily-triage-report"
assert instruction["max_tokens"] == 1800
assert "most 7 recommendations" in instruction["prompt"]
assert "fewer well-formed" in instruction["prompt"]
assert instruction["output_schema"] == (
"/etc/activity-core/schemas/daily-triage-report.json"
)
assert "at most 7" in instruction["prompt"]
assert "fewer well-formed recommendations" in instruction["prompt"]
assert instruction["output_schema"] == "activity-core://schemas/daily-triage-report.json"
assert instruction["report_sinks"][0]["type"] == "working-memory"
assert instruction["report_sinks"][1]["event_type"] == "daily_triage"
@ -136,12 +135,36 @@ def test_ops_inventory_configmap_contains_probeable_inventory() -> None:
assert inventory["policy"]["non_secret_inventory"] is True
assert services["gitea"]["endpoints"][0]["id"] == "gitea-oci-registry"
assert services["state-hub"]["endpoints"][0]["url"] == (
"http://actcore-state-hub-bridge:8000/state/health"
"http://actcore-statehub-edge-relay:8000/edge/health"
)
assert services["inter-hub"]["endpoints"][0]["id"] == "inter-hub-openapi"
assert services["activity-core"]["endpoints"][0]["id"] == "activity-core-api"
def test_statehub_edge_relay_deployment_uses_state_hub_image_and_outbox_pvc() -> None:
deployment = _by_kind_name("Deployment", "actcore-statehub-edge-relay")
container = deployment["spec"]["template"]["spec"]["containers"][0]
env = {item["name"]: item["value"] for item in container["env"]}
assert container["image"] == "forgejo.coulomb.social/coulomb/state-hub:main-d8808bf"
assert env["STATEHUB_UPSTREAM_URL"] == "http://state-hub.state-hub.svc.cluster.local:8000"
assert env["STATEHUB_OUTBOX_PATH"] == "/var/statehub/edge-outbox.sqlite3"
assert env["STATEHUB_READ_CACHE_PATH"] == "/var/statehub/edge-read-cache.sqlite3"
assert deployment["spec"]["replicas"] == 1
pvc = _by_kind_name("PersistentVolumeClaim", "actcore-statehub-edge-outbox")
assert pvc["spec"]["resources"]["requests"]["storage"] == "1Gi"
def test_legacy_state_hub_bridge_is_removed() -> None:
names = {
(resource.get("kind"), resource.get("metadata", {}).get("name"))
for resource in _resources()
}
assert ("Deployment", "actcore-state-hub-bridge") not in names
assert ("Service", "actcore-state-hub-bridge") not in names
def test_worker_mounts_ops_inventory_configmap() -> None:
deployment = _by_kind_name("Deployment", "actcore-worker")
pod_spec = deployment["spec"]["template"]["spec"]
@ -208,6 +231,8 @@ def test_disabled_ops_probe_definition_can_emit_fixture_evidence(
def fake_endpoint_get(url: str, **kwargs: Any) -> Any:
if url.endswith("/v2/"):
return _HttpResponse(401, "OCI registry auth challenge")
if url.endswith("/edge/health"):
return _HttpResponse(200, "edge relay health")
if url.endswith("/state/health"):
return _HttpResponse(200, "health response")
if url.endswith("/openapi.json"):
@ -255,8 +280,9 @@ class _HttpResponse:
class _JsonResponse:
def __init__(self, payload: Any) -> None:
def __init__(self, payload: Any, *, status_code: int = 200) -> None:
self.payload = payload
self.status_code = status_code
def raise_for_status(self) -> None:
return None

View file

@ -9,8 +9,9 @@ from activity_core.report_sinks import persist_reports
class DummyResponse:
def __init__(self, payload: Any) -> None:
def __init__(self, payload: Any, *, status_code: int = 200) -> None:
self.payload = payload
self.status_code = status_code
def raise_for_status(self) -> None:
return None

View file

@ -11,6 +11,7 @@ from activity_core.state_hub_write import (
apply_progress_scope_fields,
idempotency_headers,
idempotency_key,
parse_state_hub_write_response,
)
@ -81,6 +82,8 @@ def test_report_sink_post_sends_idempotency_header(monkeypatch) -> None:
monkeypatch.setattr(report_sinks, "_progress_exists", lambda *a, **k: False)
class _Resp:
status_code = 200
def raise_for_status(self) -> None: ...
def json(self) -> dict[str, str]:
return {"id": "pid-1"}
@ -98,3 +101,47 @@ def test_report_sink_post_sends_idempotency_header(monkeypatch) -> None:
result = report_sinks._post_state_hub_progress(payload, report_entry, sink)
assert result["status"] == "posted"
assert captured["headers"][IDEMPOTENCY_HEADER] == "run1:daily-triage-report:daily_triage"
def test_parse_state_hub_write_response_accepts_edge_relay_queued_receipt() -> None:
class _Resp:
status_code = 202
def json(self) -> dict[str, object]:
return {
"queued": True,
"outbox_id": "env-1",
"idempotency_key": "run1:daily-triage-report:daily_triage",
}
def raise_for_status(self) -> None:
raise AssertionError("queued receipts should not raise")
assert parse_state_hub_write_response(_Resp())["outbox_id"] == "env-1"
def test_report_sink_post_accepts_edge_relay_queued_receipt(monkeypatch) -> None:
monkeypatch.setattr(report_sinks, "_progress_exists", lambda *a, **k: False)
class _Resp:
status_code = 202
def json(self) -> dict[str, object]:
return {
"queued": True,
"outbox_id": "env-1",
"idempotency_key": "run1:daily-triage-report:daily_triage",
}
def raise_for_status(self) -> None:
raise AssertionError("queued receipts should not raise")
monkeypatch.setattr(report_sinks.httpx, "post", lambda *a, **k: _Resp())
payload = {"run_id": "run1", "activity_id": "act1", "scheduled_for": None}
report_entry = {"instruction_id": "daily-triage-report", "report": {"summary": "s"}}
sink = {"event_type": "daily_triage"}
result = report_sinks._post_state_hub_progress(payload, report_entry, sink)
assert result["status"] == "queued"
assert result["outbox_id"] == "env-1"

View file

@ -4,56 +4,60 @@ type: workplan
title: "Adopt State Hub Beachhead Endpoint"
domain: infotech
repo: activity-core
status: active
status: finished
owner: claude
topic_slug: activity-core
created: "2026-06-24"
updated: "2026-07-03"
updated: "2026-07-09"
state_hub_workstream_id: "bbc07f9e-9323-4b2b-b556-c33b37d0b228"
---
# Adopt State Hub Beachhead Endpoint
Carries the **blocked remainder** of [[ACTIVITY-WP-0014]] T05. The in-repo half
(idempotency-keyed State Hub writes) shipped in WP-0014; this workplan is the
client-side adoption that depends on the state-hub-owned **beachhead** capability
(per-machine read cache + write outbox) existing first.
(idempotency-keyed State Hub writes) shipped in WP-0014; this workplan adopts the
state-hub-owned **edge relay** (write outbox v1) for the railiance01 runtime.
**Waiting on:** the state-hub beachhead (proposal sent to the `state-hub` agent,
2026-06-23). Do not build queue/cache logic in activity-core — see
[[statehub-beachhead-principle]].
2026-07-03 reevaluation: no state-hub-owned beachhead endpoint is available yet.
`CUST-WP-0054` now tracks a files-first dev beachhead, but activity-core client
adoption still depends on the state-hub capability shipping first. Both tasks
remain `wait` on external delivery; the workplan is `active`, not `blocked`.
**Scope note (2026-07-09):** Path B — beachhead v1 shipped with write outbox.
**2026-07-09 update:** state-hub edge relay read cache landed for allowlisted
`GET` routes; activity-core consumes stale cache transparently. Operator deploy
of a state-hub image containing the read-cache code is still required on
railiance01.
## Point STATE_HUB_URL at the beachhead
```task
id: ACTIVITY-WP-0015-T01
status: wait
status: done
priority: medium
state_hub_task_id: "76b6132d-394a-4a67-bef6-73bb9d1e277e"
```
Once the state-hub beachhead exposes a local endpoint, point activity-core's
`STATE_HUB_URL` (and the railiance runtime config) at it and verify reads are
served from cache and writes are queued/flushed correctly when central State Hub
is unreachable. Confirm idempotency-keyed writes dedup on flush (no duplicate
`daily_triage`/progress events).
`actcore-runtime-config` now sets `STATE_HUB_URL` to
`http://actcore-statehub-edge-relay:8000`. The relay runs the state-hub edge
relay image with upstream `state-hub.state-hub.svc.cluster.local:8000` and a PVC
for the SQLite outbox. activity-core sinks accept HTTP 202 queued receipts as
successful delivery pending replay (`parse_state_hub_write_response`).
**Operator verification still required on railiance01:** apply the manifest with a
state-hub image that includes edge read cache, confirm `GET /edge/health` shows
`upstream_reachable: true` and `read_cache.entry_count > 0` after warm-up,
simulate upstream outage to confirm stale `GET /state/summary` and queued progress
writes, then replay with `statehub outbox replay`.
## Retire the bespoke actcore-state-hub-bridge proxy
```task
id: ACTIVITY-WP-0015-T02
status: wait
status: done
priority: medium
state_hub_task_id: "526c2129-cbf7-4531-a319-aebfc75cc6a3"
```
Remove the inline `hostNetwork` HTTP proxy `actcore-state-hub-bridge` from
`k8s/railiance/20-runtime.yaml` — it is a primitive precursor of the beachhead
and should be replaced by the state-hub-owned component, not extended. Re-verify
the daily triage end-to-end after cutover, including an overnight scheduled run
while the workstation is asleep (the original failure condition).
Removed `actcore-state-hub-bridge` Deployment/Service from
`k8s/railiance/20-runtime.yaml`. Ops inventory and tests now reference
`actcore-statehub-edge-relay`.
**Operator verification still required:** one overnight `daily-statehub-wsjf-triage`
run after deploy (original failure condition). If upstream was briefly down,
check the relay outbox and replay before closing the stabilization window.