activity-core/docs/runbook.md

activity-core Operational Runbook

Dev environment — quick start

# 1. Start the full stack (Temporal + PostgreSQL + Elasticsearch + NATS)
docker compose -f docker-compose.dev.yml up -d

# 2. Apply DB migrations
uv run alembic upgrade head

# 3. Seed initial ActivityDefinitions
uv run python src/activity_core/seed.py

# 4. Register custom Temporal search attributes (one-time per namespace)
docker exec temporal temporal operator search-attribute create \
  --name ActivityId   --type Keyword \
  --name ActivityName --type Keyword \
  --address temporal:7233

# 5. Start the worker (syncs schedules automatically on startup)
TEMPORAL_HOST=localhost:7233 \
ACTCORE_DB_URL=postgresql+asyncpg://actcore:actcore@localhost:5433/actcore \
  uv run python -m activity_core.worker

# 6. Start the Event Router (in a second terminal)
TEMPORAL_HOST=localhost:7233 \
ACTCORE_DB_URL=postgresql+asyncpg://actcore:actcore@localhost:5433/actcore \
NATS_URL=nats://localhost:4222 \
  uv run python -m activity_core.event_router

# 7. Start the REST API (in a third terminal)
TEMPORAL_HOST=localhost:7233 \
ACTCORE_DB_URL=postgresql+asyncpg://actcore:actcore@localhost:5433/actcore \
  uv run uvicorn activity_core.api:app --port 8010 --reload

---

Endpoints

Service	URL
Temporal Web UI	http://localhost:8080
REST API docs (Swagger)	http://localhost:8010/docs
Operator console UI	http://localhost:8010/ops/ui
Operator status JSON	http://localhost:8010/ops/automations/status?since=sunday
NATS monitoring	http://localhost:8222
Prometheus metrics (worker)	http://localhost:9090/metrics

---

Operator automation console (ACTIVITY-WP-0024)

Prefer the ops console over ad-hoc SSH/SQL for “did automations run?” and “run this now”.

Auth

Mode	When	How
SSO (primary)	Browser via activity.coulomb.social	Authelia session; app trusts Remote-User / Remote-Email from Traefik ForwardAuth
Break-glass token	Port-forward / emergency / scripts	X-Operator-Token or Authorization: Bearer
Local dev	No token configured	ACTIVITY_CORE_OPS_ALLOW_UNAUTH_MUTATIONS=1 only

Env	Purpose
ACTIVITY_CORE_OPERATOR_TOKEN	Shared operator token (break-glass); custody in actcore-runtime-secret
ACTIVITY_CORE_OPS_ALLOW_UNAUTH_MUTATIONS	1 only for local dev without a token

Mutations: POST /ops/automations/{id}/trigger|enable|disable|pause|unpause.

Fail-closed: without SSO headers and without a valid token (and unauth not allowed), mutations return 401/403. Reads (GET /ops/...) do not require auth at the app layer (ingress still gates browser access via Authelia). Do not put the token in git, chat, or workplans.

Daily checklist

# How did automations go since Sunday?
curl -sS "http://localhost:8010/ops/automations/status?since=sunday" | python3 -m json.tool
# or CLI equivalent:
make automation-status SINCE=sunday

# Inventory
curl -sS "http://localhost:8010/ops/automations" | python3 -m json.tool

# Run now (requires token)
curl -sS -X POST "http://localhost:8010/ops/automations/<id>/trigger" \
  -H "X-Operator-Token: $ACTIVITY_CORE_OPERATOR_TOKEN" \
  -H "Content-Type: application/json" -d '{}'

# Side-effect activities (e.g. forgejo prune) need explicit confirm:
curl -sS -X POST "http://localhost:8010/ops/automations/<id>/trigger" \
  -H "X-Operator-Token: $ACTIVITY_CORE_OPERATOR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"confirm_side_effect": true}'

# Pause / disable schedule (token required)
curl -sS -X POST "http://localhost:8010/ops/automations/<id>/pause" \
  -H "X-Operator-Token: $ACTIVITY_CORE_OPERATOR_TOKEN"
curl -sS -X POST "http://localhost:8010/ops/automations/<id>/disable" \
  -H "X-Operator-Token: $ACTIVITY_CORE_OPERATOR_TOKEN"

Thin UI: open https://activity.coulomb.social/ops/ui (SSO). Break-glass UI still accepts a pasted operator token (localStorage only). Cron edits are not in the UI — change definition files and sync.

Production access (railiance01)

Primary (SSO — ACTIVITY-WP-0025, live):

UI	URL
Ops console	https://activity.coulomb.social/ops/ui
Temporal Web UI	https://temporal.coulomb.social

Login via Authelia (auth.coulomb.social). Design: docs/ops-sso-access.md. Mutations use SSO identity; shared token is break-glass only.

Who may log in: LLDAP group activity-core-operators (Authelia domain rules — net-kingdom NK-WP-0021). Add/remove members:

# from net-kingdom checkout
cd sso-mfa/k8s/lldap
./manage-group-members.sh add <uid> activity-core-operators
./manage-group-members.sh list activity-core-operators
# full runbook: OPERATOR-GROUPS.md

DNS (already set for TLS):

activity.coulomb.social  A  92.205.62.239
temporal.coulomb.social  A  92.205.62.239

Break-glass port-forward (hosteurope kubeconfig):

export KUBECONFIG=~/.kube/config-hosteurope
kubectl -n activity-core port-forward svc/actcore-api 8010:8010
kubectl -n activity-core port-forward svc/actcore-temporal-ui 8080:8080
# http://127.0.0.1:8010/ops/ui  and  http://127.0.0.1:8080

Env overrides:

ACTIVITY_CORE_TEMPORAL_UI_URL=https://temporal.coulomb.social
ACTIVITY_CORE_OPERATOR_TOKEN=…   # break-glass; in actcore-runtime-secret

Bootstrap token (operator workstation; never commit the value):

# Generate and inject (example — adjust secret key name to match cluster)
TOKEN=$(openssl rand -hex 24)
kubectl -n activity-core create secret generic actcore-runtime-secret \
  --from-literal=ACTIVITY_CORE_OPERATOR_TOKEN="$TOKEN" \
  --dry-run=client -o yaml | kubectl apply -f -   # only if creating fresh;
# Prefer: kubectl patch / edit to merge the key into existing secret, then
kubectl -n activity-core rollout restart deploy/actcore-api
unset TOKEN

---

REST API — common operations

# List all ActivityDefinitions
curl http://localhost:8010/activity-definitions/

# Create a cron ActivityDefinition (fires every weekday at 09:00 Berlin time)
curl -s -X POST http://localhost:8010/activity-definitions/ \
  -H "Content-Type: application/json" -d '{
    "name": "daily-report",
    "trigger_config": {
      "trigger_type": "cron",
      "cron_expression": "0 9 * * 1-5",
      "timezone": "Europe/Berlin",
      "misfire_policy": "skip"
    }
  }'

# Create an event-triggered ActivityDefinition
curl -s -X POST http://localhost:8010/activity-definitions/ \
  -H "Content-Type: application/json" -d '{
    "name": "user-onboarding",
    "trigger_config": {
      "trigger_type": "event",
      "event_type": "user.created",
      "filters": {"tier": "pro"}
    }
  }'

# Manually trigger a one-shot run
curl -s -X POST http://localhost:8010/activity-definitions/<id>/trigger

# Disable an activity (pauses its schedule)
curl -s -X PUT http://localhost:8010/activity-definitions/<id> \
  -H "Content-Type: application/json" -d '{"enabled": false}'

---

Publishing events to the Event Router

The Event Router subscribes to the activity.> NATS subject on the ACTIVITY_EVENTS stream.

import asyncio, json, nats
from datetime import datetime, timezone
import uuid

async def publish():
    nc = await nats.connect("nats://localhost:4222")
    js = nc.jetstream()
    envelope = {
        "event_id": str(uuid.uuid4()),
        "type": "user.created",
        "source": "user-service",
        "occurred_at": datetime.now(tz=timezone.utc).isoformat(),
        "subject": "user/42",
        "trace_id": str(uuid.uuid4()),
        "payload": {"tier": "pro", "region": "eu"},
    }
    await js.publish("activity.user.created", json.dumps(envelope).encode())
    await nc.drain()

asyncio.run(publish())

---

Syncing definitions and schedules manually

When the API is running, prefer the admin sync endpoint for definition or schedule changes. It refreshes file-backed ActivityDefinitions and reconciles Temporal Schedules without restarting the worker:

curl -s -X POST \
  'http://localhost:8010/admin/sync?definitions=true&schedules=true'

The response reports:

• definitions.synced
• event_types.synced
• schedules.upserted
• schedules.paused
• schedules.deleted_orphans
• bounded errors[]

Automation inventory

Use the repo-native inventory command to answer "what automations are scheduled at all?" before checking whether a recent window succeeded. The command is read-only: it loads ActivityDefinition rows or files and, when TEMPORAL_HOST is configured, describes Temporal schedules for visibility. It does not sync, upsert, pause, delete, or enqueue schedules.

# Human-readable configured automation inventory.
make automation-list

# JSON for scripts or assistant summarization.
make automation-list-json

# Common filters.
make automation-list ENABLED=true TRIGGER=cron
make automation-list ACTIVITY_ID=6fca51fa-387a-4fd0-bc4e-d62c29eb859a

Inventory answers what is configured; make automation-status answers what happened in a time window. Missing optional live sources are warnings, not silent omissions, so a degraded local run still lists repo definition files.

Compact human output looks like:

- Daily State Hub WSJF Triage [enabled cron] schedule=activity-schedule-... trigger=20 7 * * * tz=Europe/Berlin source=files temporal=not_checked

Repo-scoped review CLI (activity)

From a consumer repo (e.g. Freedom Intelligence), use the activity CLI for morning review without AI tooling. Canon: docs/repo-automation-review-cli.md (ACTIVITY-WP-0028).

# Install once (from activity-core checkout)
uv tool install -e .
# or: uv run activity …

cd ~/freedom-intelligence
activity status
activity inbox
activity runs --since today          # needs ACTIVITY_CORE_URL for live ops API
activity ack briefs/2026/08/2026-08-06.md

Env	Purpose
ACTIVITY_CORE_URL	Ops API (e.g. https://activity.coulomb.social or ClusterIP)
STATE_HUB_URL	Hub progress for completion events
ACTIVITY_REVIEW_STATE_DIR	Override local checkpoint dir

Org-wide tools (make automation-status, prod SSH helper) remain for fleet view.

Automation status

Use the repo-native status command to answer operator questions such as "how did our automations go since Friday?". This is the baseline evidence surface; LLMs or coding assistants may summarize the output, but they are not the scheduler or source of truth.

# Human-readable status. `friday` resolves in Europe/Berlin by default.
make automation-status SINCE=friday

# JSON for scripts or assistant summarization.
make automation-status-json SINCE=2026-06-26

The command reads activity-core owned evidence only: ActivityDefinition files or DB rows, activity_runs, State Hub progress, working-memory report notes, and Temporal visibility when TEMPORAL_HOST is configured. Missing live sources are reported as warnings rather than hidden. It exits non-zero for real automation failures such as missed, validation_failed, or sink_failed.

Useful knobs:

AUTOMATION_STATUS_TIMEOUT_SECONDS=10 make automation-status SINCE=friday
make automation-status SINCE=2026-06-26 FORMAT=json
make automation-status SINCE=2026-06-26 UNTIL=2026-06-27 ACTCORE_DB_URL=

Production evidence path (railiance01)

The workstation make automation-status is degraded without a live ACTCORE_DB_URL / TEMPORAL_HOST to the railiance01 stack (docker hostnames in .env are not the production DBs). For operator questions about live schedules, use the prod helper (SSH to the host; no k3s API tunnel is required):

# Human summary since last Sunday (Europe/Berlin window handled client-side)
./scripts/prod_automation_status.sh sunday

# Explicit UTC lower bound
./scripts/prod_automation_status.sh 2026-07-18T22:00:00+00:00

The script SSHes to railiance01 (see ~/.ssh/config), queries activity_runs via kubectl -n activity-core exec actcore-app-db-0, and prints per-activity counts plus non-high-frequency fire rows. It never prints secrets.

Manual equivalent:

ssh railiance01 'export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
kubectl -n activity-core exec actcore-app-db-0 -- psql -U actcore -d actcore -c "
SELECT d.name, count(*) AS runs, max(r.fired_at) AS last_fire,
       sum(r.tasks_spawned) AS tasks
FROM activity_runs r
JOIN activity_definitions d ON d.id = r.activity_id
WHERE coalesce(r.scheduled_for, r.fired_at) >= timestamptz '\''2026-07-18 22:00:00+00'\''
GROUP BY d.name ORDER BY runs DESC;"
'

Temporal schedule / workflow status (from the worker pod):

ssh railiance01 'export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
kubectl -n activity-core exec deploy/actcore-worker -- /app/.venv/bin/python3 -c "
# describe ScheduleHandle for activity-schedule-<uuid>
..."
'

Where progress evidence lives (edge vs workstation)

Prod activations post to http://actcore-statehub-edge-relay:8000 on railiance01 (upstream in-cluster state-hub). That feed is not always the same history as workstation http://127.0.0.1:8000 (local primary vs tunnel).

After a fire, query the edge from the worker:

ssh railiance01 'export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
kubectl -n activity-core exec deploy/actcore-worker -- /app/.venv/bin/python3 -c "
import urllib.request, json
for et in [\"daily_triage\",\"forgejo_package_prune\",\"activity_task_spawn\",\"sbom_staleness\"]:
  d=json.loads(urllib.request.urlopen(
    f\"http://actcore-statehub-edge-relay:8000/progress/?event_type={et}&limit=3\").read())
  print(et, d[0][\"created_at\"] if d else None, (d[0].get(\"summary\") or \"\")[:80] if d else \"\")
"'

IssueSink / task emission

Default: ISSUE_SINK_TYPE=state-hub (ACTIVITY-WP-0022). See docs/issue-core-emission-boundary.md and docs/task-emission-consumer-contract.md.

Mode	Use
state-hub	Internal findings (default)
null	Dry-run
rest	Intentional issue-core / external tracker only

TaskExecutorWorkflow is disabled unless ACTIVITY_CORE_ENABLE_TASK_EXECUTOR_STUB=true (legacy tests only).

review_required on instructions is metadata only until a downstream review queue exists (issue-core / work-record lane) — see ACTIVITY-WP-0023-T09.

Ops run claim queue (ACTIVITY-WP-0026)

Durable claimable instances for scheduled automation. Spec: docs/ops-run-queue.md. Architecture: ACT-ADR-005.

# Visibility (counts + stuck SLA in status)
curl -sS "http://localhost:8010/ops/automations/status?since=today" \
  | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('ops_runs'))"

# Open runs
curl -sS "http://localhost:8010/ops-runs?state=open" | python3 -m json.tool

# Claim (worker)
curl -sS -X POST "http://localhost:8010/ops-runs/claim" \
  -H "Content-Type: application/json" \
  -H "X-Worker-Token: $ACTIVITY_CORE_WORKER_TOKEN" \
  -d '{"worker_id":"rein-aharness@railiance01","labels":["automated"],"limit":1}'

# Reopen stale leases
curl -sS -X POST "http://localhost:8010/ops-runs/expire-leases" \
  -H "X-Worker-Token: $ACTIVITY_CORE_WORKER_TOKEN"

Env	Default	Meaning
OPS_RUN_QUEUE_ENABLED	true	Insert ops_run on emit
OPS_RUN_LEASE_SECONDS	900	Claim lease
OPS_RUN_MAX_ATTEMPTS	3	Fail permanently after N claims
OPS_RUN_SLA_HOURS	1	Stuck threshold in status
ACTIVITY_CORE_WORKER_TOKEN	unset	Harness claim auth

Railiance rollout (T07): full checklist with image import, migrate job, smoke trigger, claim test, and dual-path residual:

→ docs/deploy-ops-run-queue-railiance.md

Example distinction from the June 2026 daily triage evidence:

- Activity 6fca51fa-387a-4fd0-bc4e-d62c29eb859a [validation_failed] expected=0 runs=0 evidence=2
  evidence state_hub_progress event_type=daily_triage run=ebec6e41... output_validated=false validation_error=Unterminated string...
  evidence state_hub_progress event_type=daily_triage run=c7370f9c... output_validated=false validation_error=Expecting ',' delimiter...

That means the schedule/report path left evidence, but the report was not a clean validated output. Disabled schedules, such as the gated weekly coding retro, are reported as disabled and are not counted as missed runs.

event_types defaults to false for this endpoint because event-triggered definitions already reload from the DB in the event router path; opt in when the operator intentionally changed event type definition files:

curl -s -X POST \
  'http://localhost:8010/admin/sync?definitions=true&schedules=true&event_types=true'

The v1 posture is manual/operator-triggered sync. A periodic background loop is deferred until live use shows it is needed; this keeps customer definition changes explicit and avoids background repo scanning from the worker.

Railiance01 no-restart smoke

After changing a projected definition in k8s/railiance/20-runtime.yaml, apply the ConfigMap and wait for the API pod volume to refresh (up to ~60s), then reconcile without restarting actcore-worker:

export KUBECONFIG=~/.kube/config-hosteurope
kubectl apply -f k8s/railiance/20-runtime.yaml
sleep 60
kubectl -n activity-core exec deploy/actcore-api -- \
  python3 -c 'import urllib.request; req=urllib.request.Request("http://localhost:8010/admin/sync?definitions=true&schedules=true", method="POST"); print(urllib.request.urlopen(req).read().decode())'

Automated regression for the disabled ops-service-inventory-probes projection (enable/cadence flip, idempotent repeat sync, rollback) lives in scripts/smoke_admin_sync_no_restart.py.

If the API is unavailable, the schedule-only CLI remains available:

TEMPORAL_HOST=localhost:7233 \
ACTCORE_DB_URL=postgresql+asyncpg://actcore:actcore@localhost:5433/actcore \
  uv run python -m activity_core.sync_schedules

This reconciles all Temporal Schedules with the activity_definitions table:

• Upserts schedules for every enabled cron definition
• Creates paused schedules for disabled cron or one-shot scheduled definitions
• Deletes orphaned schedules with no matching DB row

After adding or changing a recurring ActivityDefinition or workflow activity wiring, run a smoke schedule before trusting the next real fire:

ACTCORE_DB_URL=postgresql+asyncpg://actcore:actcore@localhost:5433/actcore \
TEMPORAL_HOST=localhost:7233 \
  uv run python scripts/smoke_test_schedule.py \
    --activity-id <activity-definition-uuid> \
    --recreate-recurring

The smoke command deletes and recreates the recurring Temporal Schedule when --recreate-recurring is set, creates a distinct one-shot smoke Schedule one minute in the future, waits for the smoke workflow to complete, and exits non-zero if the workflow fails or times out. Use this after worker deployments that add workflow imports or new activities; it catches stale-worker and missing activity registration issues before the next scheduled run.

---

Weekly maintenance definitions

weekly-forgejo-package-prune runs Sundays at 03:30 UTC (after 02:15 forgejo-backup). It invokes the shell context query forgejo_package_prune, which runs /opt/railiance-platform/tools/cmd/forgejo-package-prune (hostPath mount of ~/railiance-platform on the worker) with apply: true and posts forgejo_package_prune progress to State Hub.

Item	Value
Retention	newest 3 versions per package (container, pypi, npm, generic)
Org	coulomb
Protected	live cluster image tags + Helm values (--live-images-file / live scan)
Credential	FORGEJO_TOKEN in actcore-runtime-secret via ExternalSecret actcore-forgejo-admin (OpenBao platform/workloads/forgejo/forgejo-admin field API_TOKEN; warden route show forgejo-admin-api-token). ESO token: scripts/openbao-eso-token-apply.sh (includes workload-kv-read-forgejo-admin).
Rollback	restore package versions from Nextcloud forgejo dump if a needed tag was removed

Enabled 2026-07-21 after dry-run + first apply evidence (railiance-platform/docs/evidence/forgejo-package-prune-apply-20260721.json: 38 deleted, 0 errors).

Apply safety (ACTIVITY-WP-0023-T03): apply: true refuses to run without a non-empty live_images_file (or FORGEJO_LIVE_IMAGES_FILE). Allowed side-effect shell query with apply today: forgejo_package_prune only.

Refresh protection list after cluster image rollouts:

# From workstation with both contexts, or merge scp'd exports on railiance01:
./scripts/refresh_live_images.sh
# railiance01 worker hostPath target:
OUT=~/railiance-platform/docs/evidence/live-images-all.txt \
  EXTRA_LIVE_FILES=/path/to/coulombcore-export.txt \
  ./scripts/refresh_live_images.sh

Manual apply:

cd ~/railiance-platform
export VAULT_ADDR=https://bao.coulomb.social
# OIDC or platform token — never paste PAT into chat
export FORGEJO_TOKEN=$(bao kv get -field=API_TOKEN platform/workloads/forgejo/forgejo-admin)
./tools/cmd/forgejo-package-prune --apply --live-images-file docs/evidence/live-images-all.txt

weekly-sbom-staleness is the canonical rule-only weekly maintenance schedule. It runs Mondays at 09:00 Europe/Berlin, resolves State Hub SBOM status for all repos, and emits one automated task per stale repo through explicit for_each: context.repos.repos.

weekly-coding-retro follows the same cron -> context resolver -> per-repo task pattern for coding-session retrospection. It runs Saturdays at 19:00 Europe/Berlin and resolves the latest State Hub /progress/ item with event_type=coding_retro and a matching window_days into context.retro.suggestions. Each positive-score suggestion emits one task to context.s.repo with labels coding-retro, improvement, and automated. The weekly schedule intentionally ignores broader retro windows such as 30-day catch-up reports.

Keep weekly-coding-retro disabled until Helix Forge publishes the coding_retro read model and a live dry-run confirms the resolver returns the expected weekly window with correct routing and no duplicate target tasks on re-run. A zero-suggestion weekly read model is an acceptable enablement proof when the workflow completes cleanly twice in a row.

Ops inventory evidence posture

The current accepted live backend for activity-core ops inventory probes is State Hub progress with event_type=ops_inventory_probe.

Inter-Hub / ops-hub per-entity submission remains intentionally deferred until all of these are true:

• OPS_HUB_KEY is provisioned through an operator-owned secret path, never Git, chat, or State Hub detail.
• Widget or capability mapping is configured for the target ops-hub entities.
• Production Inter-Hub intake is deployed and smoke-tested for the relevant authenticated routes.

Until then, missing Inter-Hub configuration should produce an explicit skipped sink result, not a failed probe. This posture was recorded in State Hub decision 7c235bbb-ee6f-4c3e-b1dd-74717eac9082.

---

Temporal UI — filtering by activity

With search attributes registered, you can filter in the Temporal Web UI:

ActivityId = "your-activity-uuid"

Or via tctl:

docker exec temporal-admin-tools temporal workflow list \
  --query 'ActivityId="<uuid>"' \
  --address temporal:7233

---

Daily State Hub WSJF triage verification

Use this when answering: "did today's daily triage run happen?"

Set the ActivityDefinition id when known. If it is not known, pass the definition name used in the environment and let the live helper resolve it from Postgres.

export DAILY_TRIAGE_ACTIVITY_ID=<daily-triage-activity-definition-uuid>

# Dry-run checklist; safe from any shell because it only prints checks.
uv run python scripts/verify_daily_triage.py \
  --activity-id "$DAILY_TRIAGE_ACTIVITY_ID" \
  --date "$(date -u +%F)"

# Live check from a shell with Temporal, DB, State Hub, and working-memory access.
ACTCORE_DB_URL=postgresql+asyncpg://actcore:actcore@localhost:5433/actcore \
TEMPORAL_HOST=localhost:7233 \
STATE_HUB_URL=http://127.0.0.1:8000 \
  uv run python scripts/verify_daily_triage.py \
    --activity-id "$DAILY_TRIAGE_ACTIVITY_ID" \
    --working-memory-dir /home/worsch/the-custodian/memory/working \
    --live

The verification is complete when all of these agree:

• Temporal schedule activity-schedule-$DAILY_TRIAGE_ACTIVITY_ID exists, is not paused, and uses the skip overlap policy.
• The latest workflow found with ActivityId="$DAILY_TRIAGE_ACTIVITY_ID" either completed or is visibly retrying a failed activity in history.
• activity_runs has a row for the daily triage ActivityDefinition with today's scheduled_for or fired_at date.
• State Hub /progress/ contains a daily_triage event whose detail includes the same activity_core_run_id and its output_validated flag.
• The working-memory sink wrote daily-triage-YYYY-MM-DD-<run>.md and its frontmatter contains the same activity_core_run_id and validation metadata.
• The ActivityDefinition's instruction model, token budget, and sink timeouts fit under ACTIVITY_TIMEOUT_SECONDS (default 900 seconds). Temporal retries each activity up to 10 attempts, so a slow LLM or sink failure should show as workflow retry history rather than a silent missing report.

Expected missed-run behavior: the daily triage definition should use misfire_policy: skip. Planned downtime does not catch up missed daily reports; the next scheduled fire is the next authoritative run.

---

Scale-out

Multiple worker replicas

Temporal workers are stateless and horizontally scalable. Run additional worker processes to increase throughput on orchestrator-tq and task-execution-tq.

Each worker registers the same workflows/activities — Temporal distributes tasks across all pollers automatically.

Important: Only one process should call sync_schedules at startup to avoid race conditions. Consider disabling the startup sync on secondary worker replicas via an env var:

SKIP_SCHEDULE_SYNC=true uv run python -m activity_core.worker

(Implement the SKIP_SCHEDULE_SYNC check in worker.py when needed.)

Multiple Event Router replicas

The durable NATS consumer (activity-core-event-router) ensures that only one subscriber processes each message. Running multiple event_router processes with the same durable consumer name provides automatic failover.

---

Run-miss recovery policies (cron triggers)

A cron fire is missed when the worker or Temporal is unavailable at trigger time. trigger_config.misfire_policy selects what happens when the system recovers. Each policy combines a Temporal catchup window (how far back missed fires are recovered) with an overlap policy (what to do if a recovered fire would start while a prior run is still executing):

misfire_policy	Behaviour	Default catchup window	Overlap
skip	Run on trigger or skip — a missed fire is never recovered	60s grace	SKIP
catchup_all	Recover every fire missed during the outage	365 days	BUFFER_ALL
catchup_latest	Recover only the most recent missed fire; no backlog	24h	BUFFER_ONE

Set trigger_config.catchup_window_seconds to override the per-policy default (e.g. an hourly definition using catchup_latest should set it to ~3600 so a single missed hour is recovered but older ones are not).

Legacy values are still accepted: catchup → catchup_all, compress → catchup_latest.

Why this exists: before ACTIVITY-WP-0014 no catchup window was set, so a brief outage at trigger time silently dropped the fire with no recovery and no log line. The daily-statehub-wsjf-triage definition now uses catchup_latest.

State Hub write idempotency (ACTIVITY-WP-0014 T05)

Every State Hub write from activity-core (report-sink progress, ops-evidence progress, schedule-miss alerts) carries a stable Idempotency-Key header derived deterministically from the write's identity (run_id:instruction_id:event_type, or schedule_miss:activity_id:last_fired for miss alerts). This makes writes safe to buffer and replay under the planned State Hub beachhead (per-machine read cache + write outbox): a flush — possibly retried after an outage — cannot create duplicate progress/triage events once State Hub / the beachhead honours the header.

The guarantee lives on the write, not on a live dedup read. The read-based _progress_exists check is now best-effort only: if State Hub is unreachable it returns False (proceed to the keyed write) rather than hard-failing. The header is honoured by the in-cluster actcore-statehub-edge-relay (state-hub edge relay) and central State Hub on replay. Allowlisted GET reads are cached by the relay and served stale (X-StateHub-Edge-Cache: stale) when upstream is briefly unreachable, which keeps daily triage context resolution alive during outages.

The queue/cache itself is not built in activity-core — it belongs to the state-hub edge relay. activity-core emits the key, treats HTTP 202 queued receipts as successful sink delivery pending replay, and consumes stale cached reads transparently.

Side-effect POSTs (consistency_sweep_remote_all, recently_on_scope_hourly) retry transient 502/503/504 then degrade by default so required workflows do not thrash Temporal when the edge is briefly unavailable. Details: docs/edge-relay-resilience.md (ACTIVITY-WP-0027-T06).

Troubleshooting

Worker fails to start: "ACTCORE_DB_URL is required"

Set the environment variable before running the worker.

Schedule not firing

1. Check Temporal UI → Schedules tab for the schedule status.
2. Ensure enabled=True on the ActivityDefinition (paused schedules don't fire).
3. Verify the cron expression with: docker exec temporal-admin-tools temporal schedule describe --schedule-id activity-schedule-<uuid>
4. If a fire was missed entirely (no run, no failure event) during an outage, check misfire_policy — under skip missed fires are dropped by design. Use catchup_all or catchup_latest to recover them. See Run-miss recovery policies.

Event not routing

1. Check NATS monitoring: http://localhost:8222/jsz to verify the ACTIVITY_EVENTS stream exists.
2. Verify the consumer is active: http://localhost:8222/jsz?consumers=true
3. Check Event Router logs for "matched no definitions" — the event type may not match any enabled ActivityDefinition.
4. Check trigger_config.filters — all key/value pairs must match the event payload exactly.

Workflow stuck / not completing

1. Open Temporal UI → find the workflow by ID or ActivityId search attribute.
2. Check the workflow history for failed activities.
3. Common causes:
   • DB connection lost during load_activity_definition or log_run
   • Activity retry exhausted (check maximum_attempts=10)
   • ActivityDefinition row was deleted while workflow was running

Prometheus metrics not appearing

1. Confirm the worker is running with PROMETHEUS_BIND_ADDR set.
2. curl http://localhost:9090/metrics should return Temporal SDK metrics.
3. If port 9090 conflicts with Prometheus server, set PROMETHEUS_BIND_ADDR=0.0.0.0:9091.

Production alerting and failure modes

Kubernetes health expectations:

kubectl -n activity-core get deploy actcore-worker actcore-api actcore-event-router
kubectl -n activity-core get pods -l app.kubernetes.io/part-of=activity-core
kubectl -n activity-core port-forward svc/actcore-worker-metrics 9090:9090
curl -sf http://127.0.0.1:9090/metrics

Page an operator when:

• actcore-worker has no ready pod, cannot connect to Temporal, or cannot reach Postgres.
• The daily triage schedule is missing or paused outside an approved maintenance window.
• The expected daily triage run is absent from Temporal and activity_runs after the retry window.
• Both State Hub progress and working-memory report sinks are missing for a completed run.
• Report sink or task emission failures repeat across Temporal retries.

Leave a State Hub progress note, but do not page, when:

• A planned outage caused one skipped run and the schedule is healthy again.
• A sink idempotency check reports exists for the expected run id.
• An instruction report has output_validated=false but still emitted a validation-failure note preserving partial model output for review.
• The report completed but calibration feedback says the recommendations were noisy, too long, or under-sensitive.

Handle in the next operator session:

• Prompt/schema tuning, loose-end sensitivity, and stale-but-parked work calibration.
• Non-urgent schedule jitter or timeout adjustments.
• Moving a task sink from ISSUE_SINK_TYPE=null to the real issue-core endpoint after a dry-run contract check has passed. See docs/issue-core-emission-boundary.md for the promotion/rollback steps and scripts/smoke_issue_core_emission.py for the weekly SBOM staleness smoke.

DB migration drift

uv run alembic current    # show current revision
uv run alembic upgrade head  # apply pending migrations
uv run alembic history    # show full migration history

---

Railiance Deployment

Production API access posture

The FastAPI admin surface remains ClusterIP-only in production. Do not publish it through an external ingress until a separate access-policy work item chooses the hostname, authentication layer, allowed users/agents, and audit expectations. This posture was recorded in State Hub decision 9ffaf7a9-227a-4e39-92e3-cd93d8cda1f2.

Pre-requisites

• Docker ≥ 24 with Compose v2 (docker compose not docker-compose)
• ≥ 4 GB RAM available (Temporal server takes ~1 GB)
• Ports available: 4222 (NATS), 7233 (Temporal gRPC), 8010 (API), 8080 (Temporal UI), 9090 (Prometheus metrics)

First-time setup

# 1. Copy and edit the env file — fill in all secrets and URLs
cp .env.example .env

# 2. Build the image and start all services
make railiance-up

# 3. Wait for health (retry until 200)
curl -sf http://localhost:8010/health   # → {"status":"ok","db":true,"temporal":true}

# 4. Register Temporal search attributes (one-time per namespace)
docker exec actcore-temporal temporal operator search-attribute create \
  --name ActivityId   --type Keyword \
  --name ActivityName --type Keyword \
  --address temporal:7233

# 5. Load event types and activity definitions
make sync-all

Upgrade procedure

git pull
make railiance-up   # rebuilds image, restarts changed services
make migrate        # apply any new migrations (safe to run when none pending)
curl -sf http://localhost:8010/health

Health verification

# API health (db + temporal probes)
curl -s http://localhost:8010/health | python3 -m json.tool

# Temporal UI
open http://localhost:8080

# Prometheus metrics
curl -s http://localhost:9090/metrics | head -20

Common ops

# Follow logs for one service
docker compose -f docker-compose.railiance.yml logs -f actcore-worker

# Restart one service without bringing down others
docker compose -f docker-compose.railiance.yml restart actcore-api

# Re-run migrations manually
docker compose -f docker-compose.railiance.yml run --rm actcore-migrate

# Wipe and reset (DESTRUCTIVE — deletes all volumes including DB data)
make railiance-down
docker volume rm activity-core_temporal-db-data activity-core_app-db-data activity-core_nats-data
make railiance-up

---

Kaizen fleet resolver (coulomb-loop)

Dry-run scheduled agent discovery against State Hub + pilot roster:

export STATE_HUB_URL=http://127.0.0.1:8000
export KAIZEN_RUNNER_HOST=$(hostname)
export ACTIVITY_DEFINITION_DIRS=/home/worsch/coulomb-loop

uv run python -c "
from activity_core.context_resolvers.kaizen import discover_kaizen_scheduled_repos
print(discover_kaizen_scheduled_repos({
    'roster': '/home/worsch/coulomb-loop/loops/kaizen-stack/roster.yaml',
    'cadence': 'daily',
}))
"

make sync-activity-definitions   # requires ACTCORE_DB_URL + stack up

Source types: kaizen, resolver, or shell (alias). Queries: discover_kaizen_scheduled_repos, discover_kaizen_projects.

---

Wipe and restart dev stack

docker compose -f docker-compose.dev.yml down -v   # removes all volumes
docker compose -f docker-compose.dev.yml up -d
uv run alembic upgrade head
uv run python src/activity_core/seed.py
# Re-register search attributes (see Dev environment step 4)