activity-core/docs/runbook.md

# activity-core Operational Runbook

## Dev environment — quick start

```bash
# 1. Start the full stack (Temporal + PostgreSQL + Elasticsearch + NATS)
docker compose -f docker-compose.dev.yml up -d

# 2. Apply DB migrations
uv run alembic upgrade head

# 3. Seed initial ActivityDefinitions
uv run python src/activity_core/seed.py

# 4. Register custom Temporal search attributes (one-time per namespace)
docker exec temporal temporal operator search-attribute create \
  --name ActivityId   --type Keyword \
  --name ActivityName --type Keyword \
  --address temporal:7233

# 5. Start the worker (syncs schedules automatically on startup)
TEMPORAL_HOST=localhost:7233 \
ACTCORE_DB_URL=postgresql+asyncpg://actcore:actcore@localhost:5433/actcore \
  uv run python -m activity_core.worker

# 6. Start the Event Router (in a second terminal)
TEMPORAL_HOST=localhost:7233 \
ACTCORE_DB_URL=postgresql+asyncpg://actcore:actcore@localhost:5433/actcore \
NATS_URL=nats://localhost:4222 \
  uv run python -m activity_core.event_router

# 7. Start the REST API (in a third terminal)
TEMPORAL_HOST=localhost:7233 \
ACTCORE_DB_URL=postgresql+asyncpg://actcore:actcore@localhost:5433/actcore \
  uv run uvicorn activity_core.api:app --port 8010 --reload
```

---

## Endpoints

| Service | URL |
|---------|-----|
| Temporal Web UI | http://localhost:8080 |
| REST API docs (Swagger) | http://localhost:8010/docs |
| Operator console UI | http://localhost:8010/ops/ui |
| Operator status JSON | http://localhost:8010/ops/automations/status?since=sunday |
| NATS monitoring | http://localhost:8222 |
| Prometheus metrics (worker) | http://localhost:9090/metrics |

---

## Operator automation console (ACTIVITY-WP-0024)

Prefer the **ops console** over ad-hoc SSH/SQL for “did automations run?” and
“run this now”.

### Auth

| Mode | When | How |
| --- | --- | --- |
| **SSO (primary)** | Browser via `activity.coulomb.social` | Authelia session; app trusts `Remote-User` / `Remote-Email` from Traefik ForwardAuth |
| **Break-glass token** | Port-forward / emergency / scripts | `X-Operator-Token` or `Authorization: Bearer` |
| **Local dev** | No token configured | `ACTIVITY_CORE_OPS_ALLOW_UNAUTH_MUTATIONS=1` only |

| Env | Purpose |
| --- | --- |
| `ACTIVITY_CORE_OPERATOR_TOKEN` | Shared operator token (break-glass); custody in `actcore-runtime-secret` |
| `ACTIVITY_CORE_OPS_ALLOW_UNAUTH_MUTATIONS` | `1` only for local dev without a token |

Mutations: `POST /ops/automations/{id}/trigger|enable|disable|pause|unpause`.

Fail-closed: without SSO headers and without a valid token (and unauth not
allowed), mutations return **401/403**. Reads (`GET /ops/...`) do not require
auth at the app layer (ingress still gates browser access via Authelia).
**Do not** put the token in git, chat, or workplans.

### Daily checklist

```bash
# How did automations go since Sunday?
curl -sS "http://localhost:8010/ops/automations/status?since=sunday" | python3 -m json.tool
# or CLI equivalent:
make automation-status SINCE=sunday

# Inventory
curl -sS "http://localhost:8010/ops/automations" | python3 -m json.tool

# Run now (requires token)
curl -sS -X POST "http://localhost:8010/ops/automations/<id>/trigger" \
  -H "X-Operator-Token: $ACTIVITY_CORE_OPERATOR_TOKEN" \
  -H "Content-Type: application/json" -d '{}'

# Side-effect activities (e.g. forgejo prune) need explicit confirm:
curl -sS -X POST "http://localhost:8010/ops/automations/<id>/trigger" \
  -H "X-Operator-Token: $ACTIVITY_CORE_OPERATOR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"confirm_side_effect": true}'

# Pause / disable schedule (token required)
curl -sS -X POST "http://localhost:8010/ops/automations/<id>/pause" \
  -H "X-Operator-Token: $ACTIVITY_CORE_OPERATOR_TOKEN"
curl -sS -X POST "http://localhost:8010/ops/automations/<id>/disable" \
  -H "X-Operator-Token: $ACTIVITY_CORE_OPERATOR_TOKEN"
```

Thin UI: open https://activity.coulomb.social/ops/ui (SSO). Break-glass UI still
accepts a pasted operator token (localStorage only). **Cron edits are not in
the UI** — change definition files and sync.

### Production access (railiance01)

**Primary (SSO — ACTIVITY-WP-0025, live):**

| UI | URL |
| --- | --- |
| Ops console | https://activity.coulomb.social/ops/ui |
| Temporal Web UI | https://temporal.coulomb.social |

Login via Authelia (`auth.coulomb.social`). Design: `docs/ops-sso-access.md`.
Mutations use SSO identity; shared token is break-glass only.

**Who may log in:** LLDAP group `activity-core-operators` (Authelia domain
rules — net-kingdom NK-WP-0021). Add/remove members:

```bash
# from net-kingdom checkout
cd sso-mfa/k8s/lldap
./manage-group-members.sh add <uid> activity-core-operators
./manage-group-members.sh list activity-core-operators
# full runbook: OPERATOR-GROUPS.md
```

**DNS (already set for TLS):**

```text
activity.coulomb.social  A  92.205.62.239
temporal.coulomb.social  A  92.205.62.239
```

**Break-glass port-forward** (hosteurope kubeconfig):

```bash
export KUBECONFIG=~/.kube/config-hosteurope
kubectl -n activity-core port-forward svc/actcore-api 8010:8010
kubectl -n activity-core port-forward svc/actcore-temporal-ui 8080:8080
# http://127.0.0.1:8010/ops/ui  and  http://127.0.0.1:8080
```

Env overrides:

```bash
ACTIVITY_CORE_TEMPORAL_UI_URL=https://temporal.coulomb.social
ACTIVITY_CORE_OPERATOR_TOKEN=…   # break-glass; in actcore-runtime-secret
```

Bootstrap token (operator workstation; never commit the value):

```bash
# Generate and inject (example — adjust secret key name to match cluster)
TOKEN=$(openssl rand -hex 24)
kubectl -n activity-core create secret generic actcore-runtime-secret \
  --from-literal=ACTIVITY_CORE_OPERATOR_TOKEN="$TOKEN" \
  --dry-run=client -o yaml | kubectl apply -f -   # only if creating fresh;
# Prefer: kubectl patch / edit to merge the key into existing secret, then
kubectl -n activity-core rollout restart deploy/actcore-api
unset TOKEN
```

---

## REST API — common operations

```bash
# List all ActivityDefinitions
curl http://localhost:8010/activity-definitions/

# Create a cron ActivityDefinition (fires every weekday at 09:00 Berlin time)
curl -s -X POST http://localhost:8010/activity-definitions/ \
  -H "Content-Type: application/json" -d '{
    "name": "daily-report",
    "trigger_config": {
      "trigger_type": "cron",
      "cron_expression": "0 9 * * 1-5",
      "timezone": "Europe/Berlin",
      "misfire_policy": "skip"
    }
  }'

# Create an event-triggered ActivityDefinition
curl -s -X POST http://localhost:8010/activity-definitions/ \
  -H "Content-Type: application/json" -d '{
    "name": "user-onboarding",
    "trigger_config": {
      "trigger_type": "event",
      "event_type": "user.created",
      "filters": {"tier": "pro"}
    }
  }'

# Manually trigger a one-shot run
curl -s -X POST http://localhost:8010/activity-definitions/<id>/trigger

# Disable an activity (pauses its schedule)
curl -s -X PUT http://localhost:8010/activity-definitions/<id> \
  -H "Content-Type: application/json" -d '{"enabled": false}'
```

---

## Publishing events to the Event Router

The Event Router subscribes to the `activity.>` NATS subject on the `ACTIVITY_EVENTS` stream.

```python
import asyncio, json, nats
from datetime import datetime, timezone
import uuid

async def publish():
    nc = await nats.connect("nats://localhost:4222")
    js = nc.jetstream()
    envelope = {
        "event_id": str(uuid.uuid4()),
        "type": "user.created",
        "source": "user-service",
        "occurred_at": datetime.now(tz=timezone.utc).isoformat(),
        "subject": "user/42",
        "trace_id": str(uuid.uuid4()),
        "payload": {"tier": "pro", "region": "eu"},
    }
    await js.publish("activity.user.created", json.dumps(envelope).encode())
    await nc.drain()

asyncio.run(publish())
```

---

## Syncing definitions and schedules manually

When the API is running, prefer the admin sync endpoint for definition or
schedule changes. It refreshes file-backed ActivityDefinitions and reconciles
Temporal Schedules without restarting the worker:

```bash
curl -s -X POST \
  'http://localhost:8010/admin/sync?definitions=true&schedules=true'
```

The response reports:

- `definitions.synced`
- `event_types.synced`
- `schedules.upserted`
- `schedules.paused`
- `schedules.deleted_orphans`
- bounded `errors[]`

## Automation inventory

Use the repo-native inventory command to answer "what automations are scheduled
at all?" before checking whether a recent window succeeded. The command is
read-only: it loads ActivityDefinition rows or files and, when `TEMPORAL_HOST`
is configured, describes Temporal schedules for visibility. It does not sync,
upsert, pause, delete, or enqueue schedules.

```bash
# Human-readable configured automation inventory.
make automation-list

# JSON for scripts or assistant summarization.
make automation-list-json

# Common filters.
make automation-list ENABLED=true TRIGGER=cron
make automation-list ACTIVITY_ID=6fca51fa-387a-4fd0-bc4e-d62c29eb859a
```

Inventory answers what is configured; `make automation-status` answers what
happened in a time window. Missing optional live sources are warnings, not
silent omissions, so a degraded local run still lists repo definition files.

Compact human output looks like:

```text
- Daily State Hub WSJF Triage [enabled cron] schedule=activity-schedule-... trigger=20 7 * * * tz=Europe/Berlin source=files temporal=not_checked
```

## Repo-scoped review CLI (`activity`)

From a **consumer repo** (e.g. Freedom Intelligence), use the `activity` CLI for
morning review without AI tooling. Canon: `docs/repo-automation-review-cli.md`
(ACTIVITY-WP-0028).

```bash
# From activity-core checkout — list make targets with bare `make`
make
make install-cli          # installs `activity` via uv tool (userspace)

# Point the CLI at one or more activity-core deployments
activity service add railiance \
  --url https://activity.coulomb.social \
  --hub-url http://127.0.0.1:18000 \
  --make-default
activity service list
activity service use local                 # switch default
activity runs -s railiance --since today   # one-shot, no default change

cd ~/freedom-intelligence
activity status
activity inbox
activity ack briefs/2026/08/2026-08-06.md
```

| Mechanism | Purpose |
| --------- | ------- |
| `activity service …` | Named backends in `~/.config/activity/services.json` |
| `-s` / `--service` | Use a named backend for this call only |
| `--activity-url` | One-shot API URL (highest precedence) |
| `ACTIVITY_CORE_URL` | Env fallback when no service/flag |
| `STATE_HUB_URL` | Hub progress (env or per-service `state_hub_url`) |
| `ACTIVITY_REVIEW_STATE_DIR` | Override local checkpoint dir |

Org-wide tools (`make automation-status`, prod SSH helper) remain for fleet view.

## Automation status

Use the repo-native status command to answer operator questions such as "how did
our automations go since Friday?". This is the baseline evidence surface; LLMs
or coding assistants may summarize the output, but they are not the scheduler or
source of truth.

```bash
# Human-readable status. `friday` resolves in Europe/Berlin by default.
make automation-status SINCE=friday

# JSON for scripts or assistant summarization.
make automation-status-json SINCE=2026-06-26
```

The command reads activity-core owned evidence only: ActivityDefinition files or
DB rows, `activity_runs`, State Hub progress, working-memory report notes, and
Temporal visibility when `TEMPORAL_HOST` is configured. Missing live sources are
reported as warnings rather than hidden. It exits non-zero for real automation
failures such as `missed`, `validation_failed`, or `sink_failed`.

Useful knobs:

```bash
AUTOMATION_STATUS_TIMEOUT_SECONDS=10 make automation-status SINCE=friday
make automation-status SINCE=2026-06-26 FORMAT=json
make automation-status SINCE=2026-06-26 UNTIL=2026-06-27 ACTCORE_DB_URL=
```

### Production evidence path (railiance01)

The workstation `make automation-status` is **degraded** without a live
`ACTCORE_DB_URL` / `TEMPORAL_HOST` to the railiance01 stack (docker hostnames
in `.env` are not the production DBs). For operator questions about live
schedules, use the prod helper (SSH to the host; no k3s API tunnel is required):

```bash
# Human summary since last Sunday (Europe/Berlin window handled client-side)
./scripts/prod_automation_status.sh sunday

# Explicit UTC lower bound
./scripts/prod_automation_status.sh 2026-07-18T22:00:00+00:00
```

The script SSHes to `railiance01` (see `~/.ssh/config`), queries
`activity_runs` via `kubectl -n activity-core exec actcore-app-db-0`, and prints
per-activity counts plus non-high-frequency fire rows. It never prints secrets.

Manual equivalent:

```bash
ssh railiance01 'export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
kubectl -n activity-core exec actcore-app-db-0 -- psql -U actcore -d actcore -c "
SELECT d.name, count(*) AS runs, max(r.fired_at) AS last_fire,
       sum(r.tasks_spawned) AS tasks
FROM activity_runs r
JOIN activity_definitions d ON d.id = r.activity_id
WHERE coalesce(r.scheduled_for, r.fired_at) >= timestamptz '\''2026-07-18 22:00:00+00'\''
GROUP BY d.name ORDER BY runs DESC;"
'
```

Temporal schedule / workflow status (from the worker pod):

```bash
ssh railiance01 'export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
kubectl -n activity-core exec deploy/actcore-worker -- /app/.venv/bin/python3 -c "
# describe ScheduleHandle for activity-schedule-<uuid>
..."
'
```

### Where progress evidence lives (edge vs workstation)

Prod activations post to **`http://actcore-statehub-edge-relay:8000`** on
railiance01 (upstream in-cluster `state-hub`). That feed is **not always** the
same history as workstation `http://127.0.0.1:8000` (local primary vs tunnel).

After a fire, query the edge from the worker:

```bash
ssh railiance01 'export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
kubectl -n activity-core exec deploy/actcore-worker -- /app/.venv/bin/python3 -c "
import urllib.request, json
for et in [\"daily_triage\",\"forgejo_package_prune\",\"activity_task_spawn\",\"sbom_staleness\"]:
  d=json.loads(urllib.request.urlopen(
    f\"http://actcore-statehub-edge-relay:8000/progress/?event_type={et}&limit=3\").read())
  print(et, d[0][\"created_at\"] if d else None, (d[0].get(\"summary\") or \"\")[:80] if d else \"\")
"'
```

## IssueSink / task emission

Default: **`ISSUE_SINK_TYPE=state-hub`** (ACTIVITY-WP-0022). See
`docs/issue-core-emission-boundary.md` and
`docs/task-emission-consumer-contract.md`.

| Mode | Use |
| --- | --- |
| `state-hub` | Internal findings (default) |
| `null` | Dry-run |
| `rest` | Intentional issue-core / external tracker only |

`TaskExecutorWorkflow` is **disabled** unless
`ACTIVITY_CORE_ENABLE_TASK_EXECUTOR_STUB=true` (legacy tests only).

`review_required` on instructions is **metadata only** until a downstream
review queue exists (issue-core / work-record lane) — see ACTIVITY-WP-0023-T09.

## Ops run claim queue (ACTIVITY-WP-0026)

Durable claimable instances for scheduled automation. Spec:
`docs/ops-run-queue.md`. Architecture: ACT-ADR-005.

```bash
# Visibility (counts + stuck SLA in status)
curl -sS "http://localhost:8010/ops/automations/status?since=today" \
  | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('ops_runs'))"

# Open runs
curl -sS "http://localhost:8010/ops-runs?state=open" | python3 -m json.tool

# Claim (worker)
curl -sS -X POST "http://localhost:8010/ops-runs/claim" \
  -H "Content-Type: application/json" \
  -H "X-Worker-Token: $ACTIVITY_CORE_WORKER_TOKEN" \
  -d '{"worker_id":"rein-aharness@railiance01","labels":["automated"],"limit":1}'

# Reopen stale leases
curl -sS -X POST "http://localhost:8010/ops-runs/expire-leases" \
  -H "X-Worker-Token: $ACTIVITY_CORE_WORKER_TOKEN"
```

| Env | Default | Meaning |
| --- | --- | --- |
| `OPS_RUN_QUEUE_ENABLED` | `true` | Insert ops_run on emit |
| `OPS_RUN_LEASE_SECONDS` | `900` | Claim lease |
| `OPS_RUN_MAX_ATTEMPTS` | `3` | Fail permanently after N claims |
| `OPS_RUN_SLA_HOURS` | `1` | Stuck threshold in status |
| `ACTIVITY_CORE_WORKER_TOKEN` | unset | Harness claim auth |

**Railiance rollout (T07):** full checklist with image import, migrate job,
smoke trigger, claim test, and dual-path residual:

→ **`docs/deploy-ops-run-queue-railiance.md`**

Example distinction from the June 2026 daily triage evidence:

```text
- Activity 6fca51fa-387a-4fd0-bc4e-d62c29eb859a [validation_failed] expected=0 runs=0 evidence=2
  evidence state_hub_progress event_type=daily_triage run=ebec6e41... output_validated=false validation_error=Unterminated string...
  evidence state_hub_progress event_type=daily_triage run=c7370f9c... output_validated=false validation_error=Expecting ',' delimiter...
```

That means the schedule/report path left evidence, but the report was not a
clean validated output. Disabled schedules, such as the gated weekly coding
retro, are reported as `disabled` and are not counted as missed runs.

`event_types` defaults to `false` for this endpoint because event-triggered
definitions already reload from the DB in the event router path; opt in when
the operator intentionally changed event type definition files:

```bash
curl -s -X POST \
  'http://localhost:8010/admin/sync?definitions=true&schedules=true&event_types=true'
```

The v1 posture is manual/operator-triggered sync. A periodic background loop is
deferred until live use shows it is needed; this keeps customer definition
changes explicit and avoids background repo scanning from the worker.

### Railiance01 no-restart smoke

After changing a projected definition in `k8s/railiance/20-runtime.yaml`,
apply the ConfigMap and wait for the API pod volume to refresh (up to ~60s),
then reconcile without restarting `actcore-worker`:

```bash
export KUBECONFIG=~/.kube/config-hosteurope
kubectl apply -f k8s/railiance/20-runtime.yaml
sleep 60
kubectl -n activity-core exec deploy/actcore-api -- \
  python3 -c 'import urllib.request; req=urllib.request.Request("http://localhost:8010/admin/sync?definitions=true&schedules=true", method="POST"); print(urllib.request.urlopen(req).read().decode())'
```

Automated regression for the disabled `ops-service-inventory-probes`
projection (enable/cadence flip, idempotent repeat sync, rollback) lives in
`scripts/smoke_admin_sync_no_restart.py`.

If the API is unavailable, the schedule-only CLI remains available:

```bash
TEMPORAL_HOST=localhost:7233 \
ACTCORE_DB_URL=postgresql+asyncpg://actcore:actcore@localhost:5433/actcore \
  uv run python -m activity_core.sync_schedules
```

This reconciles all Temporal Schedules with the `activity_definitions` table:
- Upserts schedules for every enabled cron definition
- Creates paused schedules for disabled cron or one-shot scheduled definitions
- Deletes orphaned schedules with no matching DB row

After adding or changing a recurring ActivityDefinition or workflow activity
wiring, run a smoke schedule before trusting the next real fire:

```bash
ACTCORE_DB_URL=postgresql+asyncpg://actcore:actcore@localhost:5433/actcore \
TEMPORAL_HOST=localhost:7233 \
  uv run python scripts/smoke_test_schedule.py \
    --activity-id <activity-definition-uuid> \
    --recreate-recurring
```

The smoke command deletes and recreates the recurring Temporal Schedule when
`--recreate-recurring` is set, creates a distinct one-shot smoke Schedule one
minute in the future, waits for the smoke workflow to complete, and exits
non-zero if the workflow fails or times out. Use this after worker deployments
that add workflow imports or new activities; it catches stale-worker and missing
activity registration issues before the next scheduled run.

---

## Weekly maintenance definitions

`weekly-forgejo-package-prune` runs Sundays at **03:30 UTC** (after 02:15
`forgejo-backup`). It invokes the `shell` context query `forgejo_package_prune`,
which runs `/opt/railiance-platform/tools/cmd/forgejo-package-prune` (hostPath
mount of `~/railiance-platform` on the worker) with `apply: true` and posts
`forgejo_package_prune` progress to State Hub.

| Item | Value |
| --- | --- |
| Retention | newest **3** versions per package (`container`, `pypi`, `npm`, `generic`) |
| Org | `coulomb` |
| Protected | live cluster image tags + Helm values (`--live-images-file` / live scan) |
| Credential | `FORGEJO_TOKEN` in `actcore-runtime-secret` via ExternalSecret `actcore-forgejo-admin` (OpenBao `platform/workloads/forgejo/forgejo-admin` field `API_TOKEN`; `warden route show forgejo-admin-api-token`). ESO token: `scripts/openbao-eso-token-apply.sh` (includes `workload-kv-read-forgejo-admin`). |
| Rollback | restore package versions from Nextcloud `forgejo dump` if a needed tag was removed |

**Enabled 2026-07-21** after dry-run + first apply evidence
(`railiance-platform/docs/evidence/forgejo-package-prune-apply-20260721.json`:
38 deleted, 0 errors).

**Apply safety (ACTIVITY-WP-0023-T03):** `apply: true` **refuses** to run without
a non-empty `live_images_file` (or `FORGEJO_LIVE_IMAGES_FILE`). Allowed
side-effect shell query with apply today: **`forgejo_package_prune` only**.

Refresh protection list after cluster image rollouts:

```bash
# From workstation with both contexts, or merge scp'd exports on railiance01:
./scripts/refresh_live_images.sh
# railiance01 worker hostPath target:
OUT=~/railiance-platform/docs/evidence/live-images-all.txt \
  EXTRA_LIVE_FILES=/path/to/coulombcore-export.txt \
  ./scripts/refresh_live_images.sh
```

Manual apply:

```bash
cd ~/railiance-platform
export VAULT_ADDR=https://bao.coulomb.social
# OIDC or platform token — never paste PAT into chat
export FORGEJO_TOKEN=$(bao kv get -field=API_TOKEN platform/workloads/forgejo/forgejo-admin)
./tools/cmd/forgejo-package-prune --apply --live-images-file docs/evidence/live-images-all.txt
```

`weekly-sbom-staleness` is emergency-paused in source and production. Its
fleet-wide `for_each: context.repos.repos` emitted 75 tasks on 2026-08-17 and
must not be re-enabled. The bounded replacement is `daily-sbom-catchup`, active
weekdays at 09:15 Europe/Berlin. It makes one ranked sbom-nexus request,
records that read-only selection in workflow history, and processes at most
three terminal ingest/skip outcomes in a dedicated heartbeat-enabled activity.
Retries retain the same fixed targets and resume acknowledged outcomes. Each
write carries a stable per-run/per-repository `Idempotency-Key`; an ambiguous
response fails the activity and is not converted into a synthetic skip. Until
sbom-nexus enforces that key, a worker crash after a committed POST but before
the next heartbeat remains a possible duplicate-write window. The automation
emits no tasks. Two production proof fires on 2026-08-22 processed six distinct
repositories as documented `no-checkout` skips with zero task spawns; see
`docs/evidence/ACTIVITY-WP-0030-daily-sbom-catchup-2026-08-22.md`.

`weekly-coding-retro` follows the same cron -> context resolver -> per-repo task
pattern for coding-session retrospection. It runs Saturdays at 19:00
Europe/Berlin and resolves the latest State Hub `/progress/` item with
`event_type=coding_retro` and a matching `window_days` into
`context.retro.suggestions`. Each positive-score suggestion emits one task to
`context.s.repo` with labels `coding-retro`, `improvement`, and `automated`.
The weekly schedule intentionally ignores broader retro windows such as 30-day
catch-up reports.

Keep `weekly-coding-retro` disabled until Helix Forge publishes the
`coding_retro` read model and a live dry-run confirms the resolver returns the
expected weekly window with correct routing and no duplicate target tasks on
re-run. A zero-suggestion weekly read model is an acceptable enablement proof
when the workflow completes cleanly twice in a row.

## Ops inventory evidence posture

Hub-core has been the production runtime authority since 2026-08-21. The
Railiance projection of the (still disabled) ops-inventory probe now writes
sanitized evidence through canonical `port.events.interaction` using sink type
`hub-core-interaction-event`. `HUB_CORE_BASE_URL` targets the in-cluster
hub-core runtime. No widget mapping or Core Hub runtime token is required.

The sink wraps the observation as catalog event `hub.interaction.recorded`,
keeps the logical observation name in `payload.reported_event_type`, and reads
the `interaction_events` projection after append to verify persistence. The old
`core-hub-interaction-event` `/api/v2` sink remains implemented for rollback
compatibility during CORE-WP-0010 stabilization, but is no longer the Railiance
probe configuration.

Inter-Hub / ops-hub per-entity submission remains intentionally deferred until
all of these are true:

- `OPS_HUB_KEY` is provisioned through an operator-owned secret path, never Git,
  chat, or State Hub detail.
- Widget or capability mapping is configured for the target ops-hub entities.
- Production Inter-Hub intake is deployed and smoke-tested for the relevant
  authenticated routes.

Until then, missing Inter-Hub configuration should produce an explicit skipped
sink result, not a failed probe. This posture was recorded in State Hub decision
`7c235bbb-ee6f-4c3e-b1dd-74717eac9082`.

---

## Temporal UI — filtering by activity

With search attributes registered, you can filter in the Temporal Web UI:
```
ActivityId = "your-activity-uuid"
```

Or via `tctl`:
```bash
docker exec temporal-admin-tools temporal workflow list \
  --query 'ActivityId="<uuid>"' \
  --address temporal:7233
```

---

## Daily State Hub WSJF triage verification

Use this when answering: "did today's daily triage run happen?"

Set the ActivityDefinition id when known. If it is not known, pass the
definition name used in the environment and let the live helper resolve it from
Postgres.

```bash
export DAILY_TRIAGE_ACTIVITY_ID=<daily-triage-activity-definition-uuid>

# Dry-run checklist; safe from any shell because it only prints checks.
uv run python scripts/verify_daily_triage.py \
  --activity-id "$DAILY_TRIAGE_ACTIVITY_ID" \
  --date "$(date -u +%F)"

# Live check from a shell with Temporal, DB, State Hub, and working-memory access.
ACTCORE_DB_URL=postgresql+asyncpg://actcore:actcore@localhost:5433/actcore \
TEMPORAL_HOST=localhost:7233 \
STATE_HUB_URL=http://127.0.0.1:8000 \
  uv run python scripts/verify_daily_triage.py \
    --activity-id "$DAILY_TRIAGE_ACTIVITY_ID" \
    --working-memory-dir /home/worsch/the-custodian/memory/working \
    --live
```

The verification is complete when all of these agree:

- Temporal schedule `activity-schedule-$DAILY_TRIAGE_ACTIVITY_ID` exists, is not
  paused, and uses the `skip` overlap policy.
- The latest workflow found with `ActivityId="$DAILY_TRIAGE_ACTIVITY_ID"` either
  completed or is visibly retrying a failed activity in history.
- `activity_runs` has a row for the daily triage ActivityDefinition with today's
  `scheduled_for` or `fired_at` date.
- State Hub `/progress/` contains a `daily_triage` event whose detail includes
  the same `activity_core_run_id` and its `output_validated` flag.
- The working-memory sink wrote `daily-triage-YYYY-MM-DD-<run>.md` and its
  frontmatter contains the same `activity_core_run_id` and validation metadata.
- The ActivityDefinition's instruction model, token budget, and sink timeouts fit
  under `ACTIVITY_TIMEOUT_SECONDS` (default 900 seconds). Temporal retries each
  activity up to 10 attempts, so a slow LLM or sink failure should show as
  workflow retry history rather than a silent missing report.

Expected missed-run behavior: the daily triage definition should use
`misfire_policy: skip`. Planned downtime does not catch up missed daily reports;
the next scheduled fire is the next authoritative run.

---

## Scale-out

### Multiple worker replicas

Temporal workers are stateless and horizontally scalable. Run additional worker
processes to increase throughput on `orchestrator-tq` and `task-execution-tq`.

Each worker registers the same workflows/activities — Temporal distributes tasks
across all pollers automatically.

**Important:** Only one process should call `sync_schedules` at startup to avoid
race conditions. Consider disabling the startup sync on secondary worker replicas
via an env var:

```bash
SKIP_SCHEDULE_SYNC=true uv run python -m activity_core.worker
```

(Implement the `SKIP_SCHEDULE_SYNC` check in `worker.py` when needed.)

### Multiple Event Router replicas

The durable NATS consumer (`activity-core-event-router`) ensures that only one
subscriber processes each message. Running multiple `event_router` processes with
the same durable consumer name provides automatic failover.

---

## Run-miss recovery policies (cron triggers)

A cron fire is **missed** when the worker or Temporal is unavailable at trigger
time. `trigger_config.misfire_policy` selects what happens when the system
recovers. Each policy combines a Temporal **catchup window** (how far back missed
fires are recovered) with an **overlap policy** (what to do if a recovered fire
would start while a prior run is still executing):

| `misfire_policy` | Behaviour | Default catchup window | Overlap |
| --- | --- | --- | --- |
| `skip` | Run on trigger or skip — a missed fire is never recovered | 60s grace | `SKIP` |
| `catchup_all` | Recover **every** fire missed during the outage | 365 days | `BUFFER_ALL` |
| `catchup_latest` | Recover only the **most recent** missed fire; no backlog | 24h | `BUFFER_ONE` |

Set `trigger_config.catchup_window_seconds` to override the per-policy default
(e.g. an hourly definition using `catchup_latest` should set it to ~3600 so a
single missed hour is recovered but older ones are not).

Legacy values are still accepted: `catchup` → `catchup_all`,
`compress` → `catchup_latest`.

> **Why this exists:** before ACTIVITY-WP-0014 no catchup window was set, so a
> brief outage at trigger time silently dropped the fire with no recovery and no
> log line. The `daily-statehub-wsjf-triage` definition now uses `catchup_latest`.

## State Hub write idempotency (ACTIVITY-WP-0014 T05)

Every State Hub write from activity-core (report-sink progress, ops-evidence
progress, schedule-miss alerts) carries a stable **`Idempotency-Key`** header
derived deterministically from the write's identity
(`run_id:instruction_id:event_type`, or `schedule_miss:activity_id:last_fired`
for miss alerts). This makes writes safe to **buffer and replay** under the
planned State Hub *beachhead* (per-machine read cache + write outbox): a flush —
possibly retried after an outage — cannot create duplicate progress/triage
events once State Hub / the beachhead honours the header.

The guarantee lives on the write, not on a live dedup read. The read-based
`_progress_exists` check is now best-effort only: if State Hub is unreachable it
returns `False` (proceed to the keyed write) rather than hard-failing. The header
is honoured by the in-cluster `actcore-statehub-edge-relay` (state-hub edge
relay) and central State Hub on replay. Allowlisted `GET` reads are cached by the
relay and served stale (`X-StateHub-Edge-Cache: stale`) when upstream is briefly
unreachable, which keeps daily triage context resolution alive during outages.

> The queue/cache itself is **not** built in activity-core — it belongs to the
> state-hub edge relay. activity-core emits the key, treats HTTP 202 queued
> receipts as successful sink delivery pending replay, and consumes stale cached
> reads transparently.

Side-effect POSTs (`consistency_sweep_remote_all`, `recently_on_scope_hourly`)
retry transient 502/503/504 then **degrade** by default so required workflows do
not thrash Temporal when the edge is briefly unavailable. Details:
`docs/edge-relay-resilience.md` (ACTIVITY-WP-0027-T06).

The consistency sweep schedule stays in activity-core; the C-rule engine is
repo-manager. Default POST still uses the State Hub dual-run adapter. Override
with `CONSISTENCY_SWEEP_URL` or `REPO_MANAGER_URL` (`docs/state-hub-caller-map.md`).

## Troubleshooting

### Worker fails to start: "ACTCORE_DB_URL is required"
Set the environment variable before running the worker.

### Schedule not firing
1. Check Temporal UI → Schedules tab for the schedule status.
2. Ensure `enabled=True` on the ActivityDefinition (paused schedules don't fire).
3. Verify the cron expression with: `docker exec temporal-admin-tools temporal schedule describe --schedule-id activity-schedule-<uuid>`
4. If a fire was **missed entirely** (no run, no failure event) during an outage,
   check `misfire_policy` — under `skip` missed fires are dropped by design. Use
   `catchup_all` or `catchup_latest` to recover them. See *Run-miss recovery policies*.

### Event not routing
1. Check NATS monitoring: http://localhost:8222/jsz to verify the `ACTIVITY_EVENTS` stream exists.
2. Verify the consumer is active: http://localhost:8222/jsz?consumers=true
3. Check Event Router logs for "matched no definitions" — the event type may not match any enabled ActivityDefinition.
4. Check `trigger_config.filters` — all key/value pairs must match the event payload exactly.

### Workflow stuck / not completing
1. Open Temporal UI → find the workflow by ID or ActivityId search attribute.
2. Check the workflow history for failed activities.
3. Common causes:
   - DB connection lost during `load_activity_definition` or `log_run`
   - Activity retry exhausted (check `maximum_attempts=10`)
   - `ActivityDefinition` row was deleted while workflow was running

### Prometheus metrics not appearing
1. Confirm the worker is running with `PROMETHEUS_BIND_ADDR` set.
2. `curl http://localhost:9090/metrics` should return Temporal SDK metrics.
3. If port 9090 conflicts with Prometheus server, set `PROMETHEUS_BIND_ADDR=0.0.0.0:9091`.

### Production alerting and failure modes

Kubernetes health expectations:

```bash
kubectl -n activity-core get deploy actcore-worker actcore-api actcore-event-router
kubectl -n activity-core get pods -l app.kubernetes.io/part-of=activity-core
kubectl -n activity-core port-forward svc/actcore-worker-metrics 9090:9090
curl -sf http://127.0.0.1:9090/metrics
```

Page an operator when:

- `actcore-worker` has no ready pod, cannot connect to Temporal, or cannot reach
  Postgres.
- The daily triage schedule is missing or paused outside an approved maintenance
  window.
- The expected daily triage run is absent from Temporal and `activity_runs`
  after the retry window.
- Both State Hub progress and working-memory report sinks are missing for a
  completed run.
- Report sink or task emission failures repeat across Temporal retries.

Leave a State Hub progress note, but do not page, when:

- A planned outage caused one skipped run and the schedule is healthy again.
- A sink idempotency check reports `exists` for the expected run id.
- An instruction report has `output_validated=false` but still emitted a
  validation-failure note preserving partial model output for review.
- The report completed but calibration feedback says the recommendations were
  noisy, too long, or under-sensitive.

Handle in the next operator session:

- Prompt/schema tuning, loose-end sensitivity, and stale-but-parked work
  calibration.
- Non-urgent schedule jitter or timeout adjustments.
- Moving a task sink from `ISSUE_SINK_TYPE=null` to the real issue-core endpoint
  after a dry-run contract check has passed. See
  `docs/issue-core-emission-boundary.md` for the promotion/rollback steps and
  `scripts/smoke_issue_core_emission.py` for the weekly SBOM staleness smoke.

### DB migration drift
```bash
uv run alembic current    # show current revision
uv run alembic upgrade head  # apply pending migrations
uv run alembic history    # show full migration history
```

---

## Railiance Deployment

### Production API access posture

The FastAPI admin surface remains ClusterIP-only in production. Do not publish
it through an external ingress until a separate access-policy work item chooses
the hostname, authentication layer, allowed users/agents, and audit
expectations. This posture was recorded in State Hub decision
`9ffaf7a9-227a-4e39-92e3-cd93d8cda1f2`.

### Pre-requisites
- Docker ≥ 24 with Compose v2 (`docker compose` not `docker-compose`)
- ≥ 4 GB RAM available (Temporal server takes ~1 GB)
- Ports available: 4222 (NATS), 7233 (Temporal gRPC), 8010 (API), 8080 (Temporal UI),
  9090 (Prometheus metrics)

### First-time setup

```bash
# 1. Copy and edit the env file — fill in all secrets and URLs
cp .env.example .env

# 2. Build the image and start all services
make railiance-up

# 3. Wait for health (retry until 200)
curl -sf http://localhost:8010/health   # → {"status":"ok","db":true,"temporal":true}

# 4. Register Temporal search attributes (one-time per namespace)
docker exec actcore-temporal temporal operator search-attribute create \
  --name ActivityId   --type Keyword \
  --name ActivityName --type Keyword \
  --address temporal:7233

# 5. Load event types and activity definitions
make sync-all
```

### Upgrade procedure

```bash
git pull
make railiance-up   # rebuilds image, restarts changed services
make migrate        # apply any new migrations (safe to run when none pending)
curl -sf http://localhost:8010/health
```

### Health verification

```bash
# API health (db + temporal probes)
curl -s http://localhost:8010/health | python3 -m json.tool

# Temporal UI
open http://localhost:8080

# Prometheus metrics
curl -s http://localhost:9090/metrics | head -20
```

### Common ops

```bash
# Follow logs for one service
docker compose -f docker-compose.railiance.yml logs -f actcore-worker

# Restart one service without bringing down others
docker compose -f docker-compose.railiance.yml restart actcore-api

# Re-run migrations manually
docker compose -f docker-compose.railiance.yml run --rm actcore-migrate

# Wipe and reset (DESTRUCTIVE — deletes all volumes including DB data)
make railiance-down
docker volume rm activity-core_temporal-db-data activity-core_app-db-data activity-core_nats-data
make railiance-up
```

---

## Kaizen fleet resolver (coulomb-loop)

Dry-run scheduled agent discovery against State Hub + pilot roster:

```bash
export STATE_HUB_URL=http://127.0.0.1:8000
export KAIZEN_RUNNER_HOST=$(hostname)
export ACTIVITY_DEFINITION_DIRS=/home/worsch/coulomb-loop

uv run python -c "
from activity_core.context_resolvers.kaizen import discover_kaizen_scheduled_repos
print(discover_kaizen_scheduled_repos({
    'roster': '/home/worsch/coulomb-loop/loops/kaizen-stack/roster.yaml',
    'cadence': 'daily',
}))
"

make sync-activity-definitions   # requires ACTCORE_DB_URL + stack up
```

Source types: `kaizen`, `resolver`, or `shell` (alias). Queries:
`discover_kaizen_scheduled_repos`, `discover_kaizen_projects`.

---

## Wipe and restart dev stack

```bash
docker compose -f docker-compose.dev.yml down -v   # removes all volumes
docker compose -f docker-compose.dev.yml up -d
uv run alembic upgrade head
uv run python src/activity_core/seed.py
# Re-register search attributes (see Dev environment step 4)
```