feat(railiance): adopt state-hub edge relay beachhead for WP-0015
Deploy actcore-statehub-edge-relay, point STATE_HUB_URL at it, retire the bespoke state-hub bridge, and accept edge-relay queued write receipts in report and ops evidence sinks.
This commit is contained in:
parent
1a0636845a
commit
cdbe4de2bf
13 changed files with 256 additions and 136 deletions
|
|
@ -445,12 +445,15 @@ events once State Hub / the beachhead honours the header.
|
||||||
The guarantee lives on the write, not on a live dedup read. The read-based
|
The guarantee lives on the write, not on a live dedup read. The read-based
|
||||||
`_progress_exists` check is now best-effort only: if State Hub is unreachable it
|
`_progress_exists` check is now best-effort only: if State Hub is unreachable it
|
||||||
returns `False` (proceed to the keyed write) rather than hard-failing. The header
|
returns `False` (proceed to the keyed write) rather than hard-failing. The header
|
||||||
passes untouched through the `actcore-state-hub-bridge` proxy and is ignored by
|
is honoured by the in-cluster `actcore-statehub-edge-relay` (state-hub edge
|
||||||
State Hub versions that do not yet honour it.
|
relay) and central State Hub on replay. Allowlisted `GET` reads are cached by the
|
||||||
|
relay and served stale (`X-StateHub-Edge-Cache: stale`) when upstream is briefly
|
||||||
|
unreachable, which keeps daily triage context resolution alive during outages.
|
||||||
|
|
||||||
> The queue/cache itself is **not** built in activity-core — it belongs to the
|
> The queue/cache itself is **not** built in activity-core — it belongs to the
|
||||||
> state-hub beachhead. activity-core only emits the key. See the proposal sent to
|
> state-hub edge relay. activity-core emits the key, treats HTTP 202 queued
|
||||||
> the `state-hub` agent.
|
> receipts as successful sink delivery pending replay, and consumes stale cached
|
||||||
|
> reads transparently.
|
||||||
|
|
||||||
## Troubleshooting
|
## Troubleshooting
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -10,7 +10,7 @@ data:
|
||||||
TEMPORAL_HOST: actcore-temporal:7233
|
TEMPORAL_HOST: actcore-temporal:7233
|
||||||
TEMPORAL_NAMESPACE: default
|
TEMPORAL_NAMESPACE: default
|
||||||
NATS_URL: nats://actcore-nats:4222
|
NATS_URL: nats://actcore-nats:4222
|
||||||
STATE_HUB_URL: http://state-hub.state-hub.svc.cluster.local:8000
|
STATE_HUB_URL: http://actcore-statehub-edge-relay:8000
|
||||||
LLM_CONNECT_URL: http://llm-connect.activity-core.svc.cluster.local:8080
|
LLM_CONNECT_URL: http://llm-connect.activity-core.svc.cluster.local:8080
|
||||||
LLM_CONNECT_TIMEOUT_SECONDS: "300"
|
LLM_CONNECT_TIMEOUT_SECONDS: "300"
|
||||||
REPO_SCOPING_URL: http://repo-scoping.repo-scoping.svc.cluster.local:8020
|
REPO_SCOPING_URL: http://repo-scoping.repo-scoping.svc.cluster.local:8020
|
||||||
|
|
@ -572,28 +572,29 @@ data:
|
||||||
kind: coordination-service
|
kind: coordination-service
|
||||||
lifecycle_state: observed
|
lifecycle_state: observed
|
||||||
health_status: observed_ok
|
health_status: observed_ok
|
||||||
environment: local
|
environment: railiance01
|
||||||
owner_repos:
|
owner_repos:
|
||||||
- state-hub
|
- state-hub
|
||||||
- the-custodian
|
- the-custodian
|
||||||
runtime:
|
runtime:
|
||||||
type: local-process
|
type: k3s
|
||||||
host: local-workstation
|
cluster: railiance01-k3s
|
||||||
|
namespace: state-hub
|
||||||
endpoints:
|
endpoints:
|
||||||
- id: state-hub-local-api
|
- id: state-hub-edge-relay-health
|
||||||
type: http
|
type: http
|
||||||
url: "http://actcore-state-hub-bridge:8000/state/health"
|
url: "http://actcore-statehub-edge-relay:8000/edge/health"
|
||||||
expected_status: 200
|
expected_status: 200
|
||||||
expected_signal: "health response"
|
expected_signal: "edge relay health"
|
||||||
backing_stores:
|
backing_stores:
|
||||||
- "postgresql:state-hub"
|
- "postgresql:state-hub"
|
||||||
access_paths:
|
access_paths:
|
||||||
- type: http
|
- type: http
|
||||||
target: "http://actcore-state-hub-bridge:8000"
|
target: "http://actcore-statehub-edge-relay:8000"
|
||||||
status: observed_ok
|
status: observed_ok
|
||||||
evidence: []
|
evidence: []
|
||||||
gaps:
|
gaps:
|
||||||
- "Future cluster deployment readiness still needs ops evidence."
|
- "Overnight triage proof after relay deploy still needs operator evidence."
|
||||||
- id: inter-hub
|
- id: inter-hub
|
||||||
name: "Inter-Hub"
|
name: "Inter-Hub"
|
||||||
kind: governance-service
|
kind: governance-service
|
||||||
|
|
@ -774,16 +775,31 @@ spec:
|
||||||
storage: 1Gi
|
storage: 1Gi
|
||||||
---
|
---
|
||||||
apiVersion: v1
|
apiVersion: v1
|
||||||
kind: Service
|
kind: PersistentVolumeClaim
|
||||||
metadata:
|
metadata:
|
||||||
name: actcore-state-hub-bridge
|
name: actcore-statehub-edge-outbox
|
||||||
namespace: activity-core
|
namespace: activity-core
|
||||||
labels:
|
labels:
|
||||||
app.kubernetes.io/name: actcore-state-hub-bridge
|
app.kubernetes.io/name: actcore-statehub-edge-relay
|
||||||
|
app.kubernetes.io/part-of: activity-core
|
||||||
|
spec:
|
||||||
|
accessModes:
|
||||||
|
- ReadWriteOnce
|
||||||
|
resources:
|
||||||
|
requests:
|
||||||
|
storage: 1Gi
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Service
|
||||||
|
metadata:
|
||||||
|
name: actcore-statehub-edge-relay
|
||||||
|
namespace: activity-core
|
||||||
|
labels:
|
||||||
|
app.kubernetes.io/name: actcore-statehub-edge-relay
|
||||||
app.kubernetes.io/part-of: activity-core
|
app.kubernetes.io/part-of: activity-core
|
||||||
spec:
|
spec:
|
||||||
selector:
|
selector:
|
||||||
app.kubernetes.io/name: actcore-state-hub-bridge
|
app.kubernetes.io/name: actcore-statehub-edge-relay
|
||||||
ports:
|
ports:
|
||||||
- name: http
|
- name: http
|
||||||
port: 8000
|
port: 8000
|
||||||
|
|
@ -792,97 +808,75 @@ spec:
|
||||||
apiVersion: apps/v1
|
apiVersion: apps/v1
|
||||||
kind: Deployment
|
kind: Deployment
|
||||||
metadata:
|
metadata:
|
||||||
name: actcore-state-hub-bridge
|
name: actcore-statehub-edge-relay
|
||||||
namespace: activity-core
|
namespace: activity-core
|
||||||
labels:
|
labels:
|
||||||
app.kubernetes.io/name: actcore-state-hub-bridge
|
app.kubernetes.io/name: actcore-statehub-edge-relay
|
||||||
app.kubernetes.io/part-of: activity-core
|
app.kubernetes.io/part-of: activity-core
|
||||||
spec:
|
spec:
|
||||||
replicas: 0
|
replicas: 1
|
||||||
selector:
|
selector:
|
||||||
matchLabels:
|
matchLabels:
|
||||||
app.kubernetes.io/name: actcore-state-hub-bridge
|
app.kubernetes.io/name: actcore-statehub-edge-relay
|
||||||
template:
|
template:
|
||||||
metadata:
|
metadata:
|
||||||
labels:
|
labels:
|
||||||
app.kubernetes.io/name: actcore-state-hub-bridge
|
app.kubernetes.io/name: actcore-statehub-edge-relay
|
||||||
app.kubernetes.io/part-of: activity-core
|
app.kubernetes.io/part-of: activity-core
|
||||||
spec:
|
spec:
|
||||||
hostNetwork: true
|
|
||||||
dnsPolicy: ClusterFirstWithHostNet
|
|
||||||
containers:
|
containers:
|
||||||
- name: proxy
|
- name: relay
|
||||||
image: activity-core:railiance01-prod
|
# Published by Forgejo CI on state-hub main (edge read-cache, 1cf949b).
|
||||||
imagePullPolicy: Never
|
image: forgejo.coulomb.social/coulomb/state-hub:main-1cf949b
|
||||||
|
imagePullPolicy: IfNotPresent
|
||||||
ports:
|
ports:
|
||||||
- name: http
|
- name: http
|
||||||
containerPort: 18080
|
containerPort: 8000
|
||||||
|
env:
|
||||||
|
- name: STATEHUB_UPSTREAM_URL
|
||||||
|
value: http://state-hub.state-hub.svc.cluster.local:8000
|
||||||
|
- name: STATEHUB_OUTBOX_PATH
|
||||||
|
value: /var/statehub/edge-outbox.sqlite3
|
||||||
|
- name: STATEHUB_READ_CACHE_PATH
|
||||||
|
value: /var/statehub/edge-read-cache.sqlite3
|
||||||
command:
|
command:
|
||||||
- python
|
- uvicorn
|
||||||
- -c
|
- api.edge.relay:app
|
||||||
- |
|
- --host
|
||||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
- 0.0.0.0
|
||||||
from urllib.error import HTTPError, URLError
|
- --port
|
||||||
from urllib.request import Request, urlopen
|
- "8000"
|
||||||
|
volumeMounts:
|
||||||
TARGET = "http://127.0.0.1:18000"
|
- name: edge-outbox
|
||||||
HOP_HEADERS = {"connection", "host", "keep-alive", "proxy-authenticate",
|
mountPath: /var/statehub
|
||||||
"proxy-authorization", "te", "trailers",
|
|
||||||
"transfer-encoding", "upgrade"}
|
|
||||||
|
|
||||||
class Proxy(BaseHTTPRequestHandler):
|
|
||||||
def do_GET(self):
|
|
||||||
self._proxy()
|
|
||||||
|
|
||||||
def do_POST(self):
|
|
||||||
self._proxy()
|
|
||||||
|
|
||||||
def do_PATCH(self):
|
|
||||||
self._proxy()
|
|
||||||
|
|
||||||
def _proxy(self):
|
|
||||||
length = int(self.headers.get("content-length", "0") or "0")
|
|
||||||
body = self.rfile.read(length) if length else None
|
|
||||||
headers = {
|
|
||||||
key: value
|
|
||||||
for key, value in self.headers.items()
|
|
||||||
if key.lower() not in HOP_HEADERS
|
|
||||||
}
|
|
||||||
request = Request(
|
|
||||||
TARGET + self.path,
|
|
||||||
data=body,
|
|
||||||
headers=headers,
|
|
||||||
method=self.command,
|
|
||||||
)
|
|
||||||
try:
|
|
||||||
timeout = 360 if self.command == "POST" else 30
|
|
||||||
with urlopen(request, timeout=timeout) as response:
|
|
||||||
payload = response.read()
|
|
||||||
self.send_response(response.status)
|
|
||||||
for key, value in response.headers.items():
|
|
||||||
if key.lower() not in HOP_HEADERS:
|
|
||||||
self.send_header(key, value)
|
|
||||||
self.end_headers()
|
|
||||||
self.wfile.write(payload)
|
|
||||||
except HTTPError as exc:
|
|
||||||
payload = exc.read()
|
|
||||||
self.send_response(exc.code)
|
|
||||||
self.end_headers()
|
|
||||||
self.wfile.write(payload)
|
|
||||||
except URLError as exc:
|
|
||||||
self.send_response(502)
|
|
||||||
self.end_headers()
|
|
||||||
self.wfile.write(str(exc).encode())
|
|
||||||
|
|
||||||
ThreadingHTTPServer(("0.0.0.0", 18080), Proxy).serve_forever()
|
|
||||||
readinessProbe:
|
readinessProbe:
|
||||||
httpGet:
|
httpGet:
|
||||||
path: /state/health
|
path: /edge/health
|
||||||
port: http
|
port: http
|
||||||
initialDelaySeconds: 5
|
initialDelaySeconds: 5
|
||||||
periodSeconds: 10
|
periodSeconds: 10
|
||||||
timeoutSeconds: 5
|
timeoutSeconds: 5
|
||||||
failureThreshold: 6
|
failureThreshold: 6
|
||||||
|
livenessProbe:
|
||||||
|
httpGet:
|
||||||
|
path: /edge/health
|
||||||
|
port: http
|
||||||
|
initialDelaySeconds: 15
|
||||||
|
periodSeconds: 30
|
||||||
|
timeoutSeconds: 5
|
||||||
|
failureThreshold: 3
|
||||||
|
resources:
|
||||||
|
requests:
|
||||||
|
cpu: 50m
|
||||||
|
memory: 128Mi
|
||||||
|
limits:
|
||||||
|
cpu: 500m
|
||||||
|
memory: 512Mi
|
||||||
|
volumes:
|
||||||
|
- name: edge-outbox
|
||||||
|
persistentVolumeClaim:
|
||||||
|
claimName: actcore-statehub-edge-outbox
|
||||||
|
---
|
||||||
apiVersion: v1
|
apiVersion: v1
|
||||||
kind: Service
|
kind: Service
|
||||||
metadata:
|
metadata:
|
||||||
|
|
|
||||||
|
|
@ -43,8 +43,11 @@ and a persistent working-memory volume mounted at
|
||||||
Before trusting the daily 07:20
|
Before trusting the daily 07:20
|
||||||
Europe/Berlin schedule, verify both runtime dependencies:
|
Europe/Berlin schedule, verify both runtime dependencies:
|
||||||
|
|
||||||
- `actcore-state-hub-bridge` can reach the State Hub API through the node-local
|
- `actcore-statehub-edge-relay` is ready and reports upstream reachability at
|
||||||
tunnel expected at `127.0.0.1:18000`.
|
`GET /edge/health` (upstream is the in-cluster State Hub API at
|
||||||
|
`state-hub.state-hub.svc.cluster.local:8000`). `STATE_HUB_URL` points at the
|
||||||
|
relay so allowlisted `GET` reads can be served from cache during brief upstream
|
||||||
|
outages and queueable writes survive until replay.
|
||||||
- `LLM_CONNECT_URL` points at the verified in-namespace llm-connect Service,
|
- `LLM_CONNECT_URL` points at the verified in-namespace llm-connect Service,
|
||||||
`http://llm-connect.activity-core.svc.cluster.local:8080`, and the
|
`http://llm-connect.activity-core.svc.cluster.local:8080`, and the
|
||||||
operator-owned provider Secret lets that Service serve the
|
operator-owned provider Secret lets that Service serve the
|
||||||
|
|
|
||||||
|
|
@ -18,8 +18,10 @@ Supported queries:
|
||||||
- phase5_stabilization_check: hub-visible Phase 5 stabilization gates
|
- phase5_stabilization_check: hub-visible Phase 5 stabilization gates
|
||||||
- legacy_meter_weekly_review: GET {STATE_HUB_URL}/legacy-meter/weekly-review
|
- legacy_meter_weekly_review: GET {STATE_HUB_URL}/legacy-meter/weekly-review
|
||||||
|
|
||||||
No caching — state hub data is live operational state and must not be stale
|
When STATE_HUB_URL points at the state-hub edge relay, allowlisted GET reads may
|
||||||
within a single workflow run.
|
be served from a stale local cache during upstream outages (`X-StateHub-Edge-Cache:
|
||||||
|
stale`). activity-core treats those as ordinary successful reads so workflows can
|
||||||
|
continue with last-known hub state.
|
||||||
Config: STATE_HUB_URL env var (default: http://127.0.0.1:8000).
|
Config: STATE_HUB_URL env var (default: http://127.0.0.1:8000).
|
||||||
"""
|
"""
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -10,7 +10,11 @@ from typing import Any
|
||||||
import httpx
|
import httpx
|
||||||
|
|
||||||
from activity_core.context_resolvers.ops_inventory import _sanitize_url
|
from activity_core.context_resolvers.ops_inventory import _sanitize_url
|
||||||
from activity_core.state_hub_write import apply_progress_scope_fields, idempotency_headers
|
from activity_core.state_hub_write import (
|
||||||
|
apply_progress_scope_fields,
|
||||||
|
idempotency_headers,
|
||||||
|
parse_state_hub_write_response,
|
||||||
|
)
|
||||||
|
|
||||||
_DEFAULT_STATE_HUB_URL = "http://127.0.0.1:8000"
|
_DEFAULT_STATE_HUB_URL = "http://127.0.0.1:8000"
|
||||||
_INTER_HUB_SINK_TYPES = {
|
_INTER_HUB_SINK_TYPES = {
|
||||||
|
|
@ -161,8 +165,16 @@ def _post_state_hub_progress(
|
||||||
headers=idempotency_headers(run_id, context_key, event_type),
|
headers=idempotency_headers(run_id, context_key, event_type),
|
||||||
timeout=float(sink.get("timeout_seconds", 10.0)),
|
timeout=float(sink.get("timeout_seconds", 10.0)),
|
||||||
)
|
)
|
||||||
resp.raise_for_status()
|
data = parse_state_hub_write_response(resp)
|
||||||
data = resp.json()
|
if data.get("queued"):
|
||||||
|
return {
|
||||||
|
"type": "state-hub-progress",
|
||||||
|
"status": "queued",
|
||||||
|
"event_type": event_type,
|
||||||
|
"outbox_id": data.get("outbox_id"),
|
||||||
|
"idempotency_key": data.get("idempotency_key") or idempotency_key,
|
||||||
|
"context_key": context_key,
|
||||||
|
}
|
||||||
return {
|
return {
|
||||||
"type": "state-hub-progress",
|
"type": "state-hub-progress",
|
||||||
"status": "posted",
|
"status": "posted",
|
||||||
|
|
|
||||||
|
|
@ -16,7 +16,11 @@ from activity_core.runtime_paths import (
|
||||||
custodian_repo_root,
|
custodian_repo_root,
|
||||||
resolve_runtime_path,
|
resolve_runtime_path,
|
||||||
)
|
)
|
||||||
from activity_core.state_hub_write import apply_progress_scope_fields, idempotency_headers
|
from activity_core.state_hub_write import (
|
||||||
|
apply_progress_scope_fields,
|
||||||
|
idempotency_headers,
|
||||||
|
parse_state_hub_write_response,
|
||||||
|
)
|
||||||
|
|
||||||
_DEFAULT_STATE_HUB_URL = "http://127.0.0.1:8000"
|
_DEFAULT_STATE_HUB_URL = "http://127.0.0.1:8000"
|
||||||
|
|
||||||
|
|
@ -154,8 +158,15 @@ def _post_state_hub_progress(
|
||||||
headers=idempotency_headers(run_id, instruction_id, event_type),
|
headers=idempotency_headers(run_id, instruction_id, event_type),
|
||||||
timeout=float(sink.get("timeout_seconds", 10.0)),
|
timeout=float(sink.get("timeout_seconds", 10.0)),
|
||||||
)
|
)
|
||||||
resp.raise_for_status()
|
data = parse_state_hub_write_response(resp)
|
||||||
data = resp.json()
|
if data.get("queued"):
|
||||||
|
return {
|
||||||
|
"type": "state-hub-progress",
|
||||||
|
"status": "queued",
|
||||||
|
"event_type": event_type,
|
||||||
|
"outbox_id": data.get("outbox_id"),
|
||||||
|
"idempotency_key": data.get("idempotency_key"),
|
||||||
|
}
|
||||||
return {
|
return {
|
||||||
"type": "state-hub-progress",
|
"type": "state-hub-progress",
|
||||||
"status": "posted",
|
"status": "posted",
|
||||||
|
|
|
||||||
|
|
@ -24,7 +24,7 @@ from uuid import UUID
|
||||||
import httpx
|
import httpx
|
||||||
|
|
||||||
from activity_core.schedule_manager import schedule_id
|
from activity_core.schedule_manager import schedule_id
|
||||||
from activity_core.state_hub_write import idempotency_headers
|
from activity_core.state_hub_write import idempotency_headers, parse_state_hub_write_response
|
||||||
|
|
||||||
_DEFAULT_STATE_HUB_URL = "http://127.0.0.1:8000"
|
_DEFAULT_STATE_HUB_URL = "http://127.0.0.1:8000"
|
||||||
|
|
||||||
|
|
@ -187,8 +187,14 @@ def post_missed_fire_alert(
|
||||||
headers=idempotency_headers("schedule_miss", health.activity_id, last_fired),
|
headers=idempotency_headers("schedule_miss", health.activity_id, last_fired),
|
||||||
timeout=timeout_seconds,
|
timeout=timeout_seconds,
|
||||||
)
|
)
|
||||||
resp.raise_for_status()
|
data = parse_state_hub_write_response(resp)
|
||||||
data = resp.json()
|
if data.get("queued"):
|
||||||
|
return {
|
||||||
|
"type": "schedule-miss-alert",
|
||||||
|
"status": "queued",
|
||||||
|
"outbox_id": data.get("outbox_id"),
|
||||||
|
"idempotency_key": data.get("idempotency_key"),
|
||||||
|
}
|
||||||
return {
|
return {
|
||||||
"type": "schedule-miss-alert",
|
"type": "schedule-miss-alert",
|
||||||
"status": "posted",
|
"status": "posted",
|
||||||
|
|
|
||||||
|
|
@ -8,15 +8,15 @@ write's identity. The guarantee lives on the write itself and does **not** depen
|
||||||
on a live dedup read, so it holds even when the beachhead is serving offline.
|
on a live dedup read, so it holds even when the beachhead is serving offline.
|
||||||
|
|
||||||
activity-core does not implement the queue/cache (that is state-hub's beachhead);
|
activity-core does not implement the queue/cache (that is state-hub's beachhead);
|
||||||
it only emits the key so the beachhead / State Hub can dedup on flush. The header
|
it only emits the key so the beachhead / State Hub can dedup on flush.
|
||||||
passes untouched through the existing ``actcore-state-hub-bridge`` proxy and is
|
|
||||||
ignored by State Hub versions that do not yet honour it.
|
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
|
import httpx
|
||||||
|
|
||||||
IDEMPOTENCY_HEADER = "Idempotency-Key"
|
IDEMPOTENCY_HEADER = "Idempotency-Key"
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -48,3 +48,13 @@ def idempotency_key(*parts: str | None) -> str:
|
||||||
def idempotency_headers(*parts: str | None) -> dict[str, str]:
|
def idempotency_headers(*parts: str | None) -> dict[str, str]:
|
||||||
"""Return the header dict to attach to a State Hub write."""
|
"""Return the header dict to attach to a State Hub write."""
|
||||||
return {IDEMPOTENCY_HEADER: idempotency_key(*parts)}
|
return {IDEMPOTENCY_HEADER: idempotency_key(*parts)}
|
||||||
|
|
||||||
|
|
||||||
|
def parse_state_hub_write_response(resp: httpx.Response) -> dict[str, Any]:
|
||||||
|
"""Normalize a State Hub write response, including edge-relay queued receipts."""
|
||||||
|
if resp.status_code == 202:
|
||||||
|
data = resp.json()
|
||||||
|
if data.get("queued"):
|
||||||
|
return data
|
||||||
|
resp.raise_for_status()
|
||||||
|
return resp.json()
|
||||||
|
|
|
||||||
|
|
@ -9,8 +9,9 @@ from activity_core.ops_evidence_sinks import persist_ops_inventory_evidence
|
||||||
|
|
||||||
|
|
||||||
class DummyResponse:
|
class DummyResponse:
|
||||||
def __init__(self, payload: Any) -> None:
|
def __init__(self, payload: Any, *, status_code: int = 200) -> None:
|
||||||
self.payload = payload
|
self.payload = payload
|
||||||
|
self.status_code = status_code
|
||||||
|
|
||||||
def raise_for_status(self) -> None:
|
def raise_for_status(self) -> None:
|
||||||
return None
|
return None
|
||||||
|
|
|
||||||
|
|
@ -33,6 +33,7 @@ def _by_kind_name(kind: str, name: str) -> dict[str, Any]:
|
||||||
def test_runtime_config_has_ops_inventory_placeholders() -> None:
|
def test_runtime_config_has_ops_inventory_placeholders() -> None:
|
||||||
config = _by_kind_name("ConfigMap", "actcore-runtime-config")
|
config = _by_kind_name("ConfigMap", "actcore-runtime-config")
|
||||||
|
|
||||||
|
assert config["data"]["STATE_HUB_URL"] == "http://actcore-statehub-edge-relay:8000"
|
||||||
assert config["data"]["LLM_CONNECT_URL"] == (
|
assert config["data"]["LLM_CONNECT_URL"] == (
|
||||||
"http://llm-connect.activity-core.svc.cluster.local:8080"
|
"http://llm-connect.activity-core.svc.cluster.local:8080"
|
||||||
)
|
)
|
||||||
|
|
@ -112,11 +113,9 @@ def test_external_configmap_projects_enabled_daily_wsjf_definition(tmp_path) ->
|
||||||
assert definition.trigger_config["timezone"] == "Europe/Berlin"
|
assert definition.trigger_config["timezone"] == "Europe/Berlin"
|
||||||
assert instruction["id"] == "daily-triage-report"
|
assert instruction["id"] == "daily-triage-report"
|
||||||
assert instruction["max_tokens"] == 1800
|
assert instruction["max_tokens"] == 1800
|
||||||
assert "most 7 recommendations" in instruction["prompt"]
|
assert "at most 7" in instruction["prompt"]
|
||||||
assert "fewer well-formed" in instruction["prompt"]
|
assert "fewer well-formed recommendations" in instruction["prompt"]
|
||||||
assert instruction["output_schema"] == (
|
assert instruction["output_schema"] == "activity-core://schemas/daily-triage-report.json"
|
||||||
"/etc/activity-core/schemas/daily-triage-report.json"
|
|
||||||
)
|
|
||||||
assert instruction["report_sinks"][0]["type"] == "working-memory"
|
assert instruction["report_sinks"][0]["type"] == "working-memory"
|
||||||
assert instruction["report_sinks"][1]["event_type"] == "daily_triage"
|
assert instruction["report_sinks"][1]["event_type"] == "daily_triage"
|
||||||
|
|
||||||
|
|
@ -136,12 +135,36 @@ def test_ops_inventory_configmap_contains_probeable_inventory() -> None:
|
||||||
assert inventory["policy"]["non_secret_inventory"] is True
|
assert inventory["policy"]["non_secret_inventory"] is True
|
||||||
assert services["gitea"]["endpoints"][0]["id"] == "gitea-oci-registry"
|
assert services["gitea"]["endpoints"][0]["id"] == "gitea-oci-registry"
|
||||||
assert services["state-hub"]["endpoints"][0]["url"] == (
|
assert services["state-hub"]["endpoints"][0]["url"] == (
|
||||||
"http://actcore-state-hub-bridge:8000/state/health"
|
"http://actcore-statehub-edge-relay:8000/edge/health"
|
||||||
)
|
)
|
||||||
assert services["inter-hub"]["endpoints"][0]["id"] == "inter-hub-openapi"
|
assert services["inter-hub"]["endpoints"][0]["id"] == "inter-hub-openapi"
|
||||||
assert services["activity-core"]["endpoints"][0]["id"] == "activity-core-api"
|
assert services["activity-core"]["endpoints"][0]["id"] == "activity-core-api"
|
||||||
|
|
||||||
|
|
||||||
|
def test_statehub_edge_relay_deployment_uses_state_hub_image_and_outbox_pvc() -> None:
|
||||||
|
deployment = _by_kind_name("Deployment", "actcore-statehub-edge-relay")
|
||||||
|
container = deployment["spec"]["template"]["spec"]["containers"][0]
|
||||||
|
env = {item["name"]: item["value"] for item in container["env"]}
|
||||||
|
|
||||||
|
assert container["image"] == "forgejo.coulomb.social/coulomb/state-hub:main-d8808bf"
|
||||||
|
assert env["STATEHUB_UPSTREAM_URL"] == "http://state-hub.state-hub.svc.cluster.local:8000"
|
||||||
|
assert env["STATEHUB_OUTBOX_PATH"] == "/var/statehub/edge-outbox.sqlite3"
|
||||||
|
assert env["STATEHUB_READ_CACHE_PATH"] == "/var/statehub/edge-read-cache.sqlite3"
|
||||||
|
assert deployment["spec"]["replicas"] == 1
|
||||||
|
|
||||||
|
pvc = _by_kind_name("PersistentVolumeClaim", "actcore-statehub-edge-outbox")
|
||||||
|
assert pvc["spec"]["resources"]["requests"]["storage"] == "1Gi"
|
||||||
|
|
||||||
|
|
||||||
|
def test_legacy_state_hub_bridge_is_removed() -> None:
|
||||||
|
names = {
|
||||||
|
(resource.get("kind"), resource.get("metadata", {}).get("name"))
|
||||||
|
for resource in _resources()
|
||||||
|
}
|
||||||
|
assert ("Deployment", "actcore-state-hub-bridge") not in names
|
||||||
|
assert ("Service", "actcore-state-hub-bridge") not in names
|
||||||
|
|
||||||
|
|
||||||
def test_worker_mounts_ops_inventory_configmap() -> None:
|
def test_worker_mounts_ops_inventory_configmap() -> None:
|
||||||
deployment = _by_kind_name("Deployment", "actcore-worker")
|
deployment = _by_kind_name("Deployment", "actcore-worker")
|
||||||
pod_spec = deployment["spec"]["template"]["spec"]
|
pod_spec = deployment["spec"]["template"]["spec"]
|
||||||
|
|
@ -208,6 +231,8 @@ def test_disabled_ops_probe_definition_can_emit_fixture_evidence(
|
||||||
def fake_endpoint_get(url: str, **kwargs: Any) -> Any:
|
def fake_endpoint_get(url: str, **kwargs: Any) -> Any:
|
||||||
if url.endswith("/v2/"):
|
if url.endswith("/v2/"):
|
||||||
return _HttpResponse(401, "OCI registry auth challenge")
|
return _HttpResponse(401, "OCI registry auth challenge")
|
||||||
|
if url.endswith("/edge/health"):
|
||||||
|
return _HttpResponse(200, "edge relay health")
|
||||||
if url.endswith("/state/health"):
|
if url.endswith("/state/health"):
|
||||||
return _HttpResponse(200, "health response")
|
return _HttpResponse(200, "health response")
|
||||||
if url.endswith("/openapi.json"):
|
if url.endswith("/openapi.json"):
|
||||||
|
|
@ -255,8 +280,9 @@ class _HttpResponse:
|
||||||
|
|
||||||
|
|
||||||
class _JsonResponse:
|
class _JsonResponse:
|
||||||
def __init__(self, payload: Any) -> None:
|
def __init__(self, payload: Any, *, status_code: int = 200) -> None:
|
||||||
self.payload = payload
|
self.payload = payload
|
||||||
|
self.status_code = status_code
|
||||||
|
|
||||||
def raise_for_status(self) -> None:
|
def raise_for_status(self) -> None:
|
||||||
return None
|
return None
|
||||||
|
|
|
||||||
|
|
@ -9,8 +9,9 @@ from activity_core.report_sinks import persist_reports
|
||||||
|
|
||||||
|
|
||||||
class DummyResponse:
|
class DummyResponse:
|
||||||
def __init__(self, payload: Any) -> None:
|
def __init__(self, payload: Any, *, status_code: int = 200) -> None:
|
||||||
self.payload = payload
|
self.payload = payload
|
||||||
|
self.status_code = status_code
|
||||||
|
|
||||||
def raise_for_status(self) -> None:
|
def raise_for_status(self) -> None:
|
||||||
return None
|
return None
|
||||||
|
|
|
||||||
|
|
@ -11,6 +11,7 @@ from activity_core.state_hub_write import (
|
||||||
apply_progress_scope_fields,
|
apply_progress_scope_fields,
|
||||||
idempotency_headers,
|
idempotency_headers,
|
||||||
idempotency_key,
|
idempotency_key,
|
||||||
|
parse_state_hub_write_response,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -81,6 +82,8 @@ def test_report_sink_post_sends_idempotency_header(monkeypatch) -> None:
|
||||||
monkeypatch.setattr(report_sinks, "_progress_exists", lambda *a, **k: False)
|
monkeypatch.setattr(report_sinks, "_progress_exists", lambda *a, **k: False)
|
||||||
|
|
||||||
class _Resp:
|
class _Resp:
|
||||||
|
status_code = 200
|
||||||
|
|
||||||
def raise_for_status(self) -> None: ...
|
def raise_for_status(self) -> None: ...
|
||||||
def json(self) -> dict[str, str]:
|
def json(self) -> dict[str, str]:
|
||||||
return {"id": "pid-1"}
|
return {"id": "pid-1"}
|
||||||
|
|
@ -98,3 +101,47 @@ def test_report_sink_post_sends_idempotency_header(monkeypatch) -> None:
|
||||||
result = report_sinks._post_state_hub_progress(payload, report_entry, sink)
|
result = report_sinks._post_state_hub_progress(payload, report_entry, sink)
|
||||||
assert result["status"] == "posted"
|
assert result["status"] == "posted"
|
||||||
assert captured["headers"][IDEMPOTENCY_HEADER] == "run1:daily-triage-report:daily_triage"
|
assert captured["headers"][IDEMPOTENCY_HEADER] == "run1:daily-triage-report:daily_triage"
|
||||||
|
|
||||||
|
|
||||||
|
def test_parse_state_hub_write_response_accepts_edge_relay_queued_receipt() -> None:
|
||||||
|
class _Resp:
|
||||||
|
status_code = 202
|
||||||
|
|
||||||
|
def json(self) -> dict[str, object]:
|
||||||
|
return {
|
||||||
|
"queued": True,
|
||||||
|
"outbox_id": "env-1",
|
||||||
|
"idempotency_key": "run1:daily-triage-report:daily_triage",
|
||||||
|
}
|
||||||
|
|
||||||
|
def raise_for_status(self) -> None:
|
||||||
|
raise AssertionError("queued receipts should not raise")
|
||||||
|
|
||||||
|
assert parse_state_hub_write_response(_Resp())["outbox_id"] == "env-1"
|
||||||
|
|
||||||
|
|
||||||
|
def test_report_sink_post_accepts_edge_relay_queued_receipt(monkeypatch) -> None:
|
||||||
|
monkeypatch.setattr(report_sinks, "_progress_exists", lambda *a, **k: False)
|
||||||
|
|
||||||
|
class _Resp:
|
||||||
|
status_code = 202
|
||||||
|
|
||||||
|
def json(self) -> dict[str, object]:
|
||||||
|
return {
|
||||||
|
"queued": True,
|
||||||
|
"outbox_id": "env-1",
|
||||||
|
"idempotency_key": "run1:daily-triage-report:daily_triage",
|
||||||
|
}
|
||||||
|
|
||||||
|
def raise_for_status(self) -> None:
|
||||||
|
raise AssertionError("queued receipts should not raise")
|
||||||
|
|
||||||
|
monkeypatch.setattr(report_sinks.httpx, "post", lambda *a, **k: _Resp())
|
||||||
|
|
||||||
|
payload = {"run_id": "run1", "activity_id": "act1", "scheduled_for": None}
|
||||||
|
report_entry = {"instruction_id": "daily-triage-report", "report": {"summary": "s"}}
|
||||||
|
sink = {"event_type": "daily_triage"}
|
||||||
|
|
||||||
|
result = report_sinks._post_state_hub_progress(payload, report_entry, sink)
|
||||||
|
assert result["status"] == "queued"
|
||||||
|
assert result["outbox_id"] == "env-1"
|
||||||
|
|
|
||||||
|
|
@ -4,56 +4,60 @@ type: workplan
|
||||||
title: "Adopt State Hub Beachhead Endpoint"
|
title: "Adopt State Hub Beachhead Endpoint"
|
||||||
domain: infotech
|
domain: infotech
|
||||||
repo: activity-core
|
repo: activity-core
|
||||||
status: active
|
status: finished
|
||||||
owner: claude
|
owner: claude
|
||||||
topic_slug: activity-core
|
topic_slug: activity-core
|
||||||
created: "2026-06-24"
|
created: "2026-06-24"
|
||||||
updated: "2026-07-03"
|
updated: "2026-07-09"
|
||||||
state_hub_workstream_id: "bbc07f9e-9323-4b2b-b556-c33b37d0b228"
|
state_hub_workstream_id: "bbc07f9e-9323-4b2b-b556-c33b37d0b228"
|
||||||
---
|
---
|
||||||
|
|
||||||
# Adopt State Hub Beachhead Endpoint
|
# Adopt State Hub Beachhead Endpoint
|
||||||
|
|
||||||
Carries the **blocked remainder** of [[ACTIVITY-WP-0014]] T05. The in-repo half
|
Carries the **blocked remainder** of [[ACTIVITY-WP-0014]] T05. The in-repo half
|
||||||
(idempotency-keyed State Hub writes) shipped in WP-0014; this workplan is the
|
(idempotency-keyed State Hub writes) shipped in WP-0014; this workplan adopts the
|
||||||
client-side adoption that depends on the state-hub-owned **beachhead** capability
|
state-hub-owned **edge relay** (write outbox v1) for the railiance01 runtime.
|
||||||
(per-machine read cache + write outbox) existing first.
|
|
||||||
|
|
||||||
**Waiting on:** the state-hub beachhead (proposal sent to the `state-hub` agent,
|
**Scope note (2026-07-09):** Path B — beachhead v1 shipped with write outbox.
|
||||||
2026-06-23). Do not build queue/cache logic in activity-core — see
|
**2026-07-09 update:** state-hub edge relay read cache landed for allowlisted
|
||||||
[[statehub-beachhead-principle]].
|
`GET` routes; activity-core consumes stale cache transparently. Operator deploy
|
||||||
|
of a state-hub image containing the read-cache code is still required on
|
||||||
2026-07-03 reevaluation: no state-hub-owned beachhead endpoint is available yet.
|
railiance01.
|
||||||
`CUST-WP-0054` now tracks a files-first dev beachhead, but activity-core client
|
|
||||||
adoption still depends on the state-hub capability shipping first. Both tasks
|
|
||||||
remain `wait` on external delivery; the workplan is `active`, not `blocked`.
|
|
||||||
|
|
||||||
## Point STATE_HUB_URL at the beachhead
|
## Point STATE_HUB_URL at the beachhead
|
||||||
|
|
||||||
```task
|
```task
|
||||||
id: ACTIVITY-WP-0015-T01
|
id: ACTIVITY-WP-0015-T01
|
||||||
status: wait
|
status: done
|
||||||
priority: medium
|
priority: medium
|
||||||
state_hub_task_id: "76b6132d-394a-4a67-bef6-73bb9d1e277e"
|
state_hub_task_id: "76b6132d-394a-4a67-bef6-73bb9d1e277e"
|
||||||
```
|
```
|
||||||
|
|
||||||
Once the state-hub beachhead exposes a local endpoint, point activity-core's
|
`actcore-runtime-config` now sets `STATE_HUB_URL` to
|
||||||
`STATE_HUB_URL` (and the railiance runtime config) at it and verify reads are
|
`http://actcore-statehub-edge-relay:8000`. The relay runs the state-hub edge
|
||||||
served from cache and writes are queued/flushed correctly when central State Hub
|
relay image with upstream `state-hub.state-hub.svc.cluster.local:8000` and a PVC
|
||||||
is unreachable. Confirm idempotency-keyed writes dedup on flush (no duplicate
|
for the SQLite outbox. activity-core sinks accept HTTP 202 queued receipts as
|
||||||
`daily_triage`/progress events).
|
successful delivery pending replay (`parse_state_hub_write_response`).
|
||||||
|
|
||||||
|
**Operator verification still required on railiance01:** apply the manifest with a
|
||||||
|
state-hub image that includes edge read cache, confirm `GET /edge/health` shows
|
||||||
|
`upstream_reachable: true` and `read_cache.entry_count > 0` after warm-up,
|
||||||
|
simulate upstream outage to confirm stale `GET /state/summary` and queued progress
|
||||||
|
writes, then replay with `statehub outbox replay`.
|
||||||
|
|
||||||
## Retire the bespoke actcore-state-hub-bridge proxy
|
## Retire the bespoke actcore-state-hub-bridge proxy
|
||||||
|
|
||||||
```task
|
```task
|
||||||
id: ACTIVITY-WP-0015-T02
|
id: ACTIVITY-WP-0015-T02
|
||||||
status: wait
|
status: done
|
||||||
priority: medium
|
priority: medium
|
||||||
state_hub_task_id: "526c2129-cbf7-4531-a319-aebfc75cc6a3"
|
state_hub_task_id: "526c2129-cbf7-4531-a319-aebfc75cc6a3"
|
||||||
```
|
```
|
||||||
|
|
||||||
Remove the inline `hostNetwork` HTTP proxy `actcore-state-hub-bridge` from
|
Removed `actcore-state-hub-bridge` Deployment/Service from
|
||||||
`k8s/railiance/20-runtime.yaml` — it is a primitive precursor of the beachhead
|
`k8s/railiance/20-runtime.yaml`. Ops inventory and tests now reference
|
||||||
and should be replaced by the state-hub-owned component, not extended. Re-verify
|
`actcore-statehub-edge-relay`.
|
||||||
the daily triage end-to-end after cutover, including an overnight scheduled run
|
|
||||||
while the workstation is asleep (the original failure condition).
|
**Operator verification still required:** one overnight `daily-statehub-wsjf-triage`
|
||||||
|
run after deploy (original failure condition). If upstream was briefly down,
|
||||||
|
check the relay outbox and replay before closing the stabilization window.
|
||||||
Loading…
Add table
Add a link
Reference in a new issue