tegwick
4e605a6839
Consistency sweep and recently-on-scope retry transient 502/503/504 then return a degraded context snapshot instead of thrashing Temporal. Document edge-relay resilience. Retire Binky dual-clock host timers after claim-loop smoke.
177 lines
8.3 KiB
Markdown
177 lines
8.3 KiB
Markdown
# Recurring automations playbook — activity-core as the organizer
|
||
|
||
**Audience:** operators and coding agents adding or fixing recurring work
|
||
**Principle:** activity-core answers **when / what / where** only. It does **not** execute.
|
||
**Architecture decision:** [ACT-ADR-005](adr/adr-005-ops-runs-vs-dev-work-records.md) —
|
||
ops runs (claimable DB) vs development work records (repo files + State Hub).
|
||
**Implementation stack:** ACTIVITY-WP-0026 · REIN-A-0002 · ISSUE-WP-0006 · STATE-WP-0078.
|
||
|
||
This is the org standard. Prefer this path over workstation cron, ad-hoc shell
|
||
timers, or “I’ll remember to run it.”
|
||
|
||
**Self-hosted forge:** **Forgejo** only (not Gitea). External tickets via
|
||
issue-core projection — never as the default ops claim queue.
|
||
|
||
---
|
||
|
||
## The three layers (do not collapse them)
|
||
|
||
```text
|
||
┌─────────────────────────────────────────────────────────────┐
|
||
│ 1. activity-core — WHEN / WHAT / WHERE │
|
||
│ Temporal schedule · context resolvers · rules │
|
||
│ → INSERT ops_run (open) + dual-write activity_task_spawn │
|
||
│ Claim API: POST /ops-runs/claim (docs/ops-run-queue.md) │
|
||
└────────────────────────────┬────────────────────────────────┘
|
||
│
|
||
┌────────────────────────────▼────────────────────────────────┐
|
||
│ 2. rein-aharness (or domain executor) — DOES THE WORK │
|
||
│ claim ops_run · llm-connect · checkout · complete/fail │
|
||
│ Host timers on railiance are *interim* until REIN-A-0002 │
|
||
│ claim loop is live (state-hub spawn alone is not claimable)│
|
||
└────────────────────────────┬────────────────────────────────┘
|
||
│
|
||
┌────────────────────────────▼────────────────────────────────┐
|
||
│ 3. domain repo — INSTANCE STATE │
|
||
│ activity-definitions/ · playbooks · briefs/ · catalogs │
|
||
│ completion event types (fi_daily_brief, binky_daily_brief)│
|
||
└─────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
| Layer | Owns | Must not own |
|
||
| ----- | ---- | ------------ |
|
||
| **activity-core** | Triggers, due checks, task specs, audit of spawns | Brief writing, git commits, LLM prompts for domain content |
|
||
| **rein-aharness** | Execution session, credentials at runtime, hub completion | Inventing schedules, domain policy |
|
||
| **domain repo** | Definition markdown, playbook, artifacts | Scattered crons that bypass activity-core |
|
||
|
||
**Anti-pattern:** workstation `crontab` that “does the daily brief” while
|
||
activity-core also schedules it — dual cadence and forgotten design.
|
||
|
||
**Primary (Binky / FI):** activity-core schedule → `ops_run` → rein-aharness
|
||
**claim loop** on railiance01. Host user-systemd timers are **break-glass only**
|
||
(not cadence authority). FI dual-clock timer was disabled 2026-08-05 after the
|
||
ops_run per-fire idempotency fix; **Binky dual-clock timers** (daily/mail/review)
|
||
were disabled 2026-08-05 after claim-loop smoke (ACTIVITY-WP-0027-T07). Re-enable
|
||
only if claim loop is down.
|
||
|
||
---
|
||
|
||
## How to add a recurring automation (checklist)
|
||
|
||
### A. Domain definition (source of truth)
|
||
|
||
1. Create `{domain-repo}/activity-definitions/{slug}.md`
|
||
- `trigger.type: cron` + timezone + `misfire_policy`
|
||
- `context_sources` with a **due resolver** when work is once-per-day
|
||
- `rule` with `target_repo`, labels, description pointing at the playbook
|
||
- Start with `enabled: false` until dry-run
|
||
2. Implement completion event shape in domain playbook
|
||
(e.g. `fi_daily_brief` with `detail.repo` + `detail.date`)
|
||
3. If needed, add resolver in activity-core (`context_resolvers/`) + tests
|
||
|
||
### B. Register with activity-core
|
||
|
||
| Environment | How |
|
||
| ----------- | --- |
|
||
| **Dev** | `ACTIVITY_DEFINITION_DIRS=/path/to/domain-repo` + `make sync-activity-definitions` |
|
||
| **Railiance** | Project definition into ConfigMap `actcore-external-activity-definitions` (and keep `k8s/railiance/20-runtime.yaml` in git) |
|
||
|
||
Then:
|
||
|
||
```bash
|
||
# after ConfigMap / files update
|
||
kubectl -n activity-core exec deploy/actcore-api -- python3 -c \
|
||
'import urllib.request as u; print(u.urlopen(u.Request(
|
||
"http://localhost:8010/admin/sync?definitions=true&schedules=true",
|
||
method="POST")).read().decode())'
|
||
```
|
||
|
||
Verify:
|
||
|
||
```bash
|
||
# definition enabled + cron
|
||
# Temporal schedule: activity-schedule-<definition-uuid> not paused
|
||
kubectl -n activity-core exec deploy/actcore-worker -- python3 -c '...' # describe schedule
|
||
```
|
||
|
||
Ops UI: `activity.coulomb.social` → automations status / trigger (SSO).
|
||
|
||
### C. Executor (rein-aharness) — target vs interim
|
||
|
||
**Target (ACT-ADR-005 / REIN-A-0002):** continuous **claim loop** on
|
||
`ops_run` rows from activity-core; approach table selects adapter
|
||
(`fi-research-brief`, `brief-daily`, mail-scan, …).
|
||
|
||
**Interim (until WP-0026 + REIN-A-0002 cutover):** railiance user-systemd
|
||
timer may call rein-aharness wall-clock — **break-glass / dual-clock**, not
|
||
cadence authority.
|
||
|
||
1. Prefer a **named rein-aharness command** + **llm-connect**, not Claude CLI on railiance01
|
||
2. On success, post the **domain completion event** so the resolver sets `due=false`
|
||
3. Close the **ops_run** (when claim queue is live)
|
||
|
||
### D. Prove the loop
|
||
|
||
1. `POST /activity-definitions/<id>/trigger` (ops UI or API)
|
||
2. Confirm `activity_task_spawn` progress event (when sink=state-hub)
|
||
3. Run executor once (timer or manual rein-aharness)
|
||
4. Confirm domain completion event → resolver `due=false` for that day
|
||
5. Set `enabled: true` if still false; commit domain + activity-core CM projection
|
||
|
||
---
|
||
|
||
## Where definitions live
|
||
|
||
| Kind | Location |
|
||
| ---- | -------- |
|
||
| Org-wide (SBOM, consistency, inventory) | `activity-core/activity-definitions/` and/or ConfigMap |
|
||
| Domain-specific (Binky, FI, …) | `{domain}/activity-definitions/` + ConfigMap projection |
|
||
|
||
ADR-002: domain repos own domain automations; activity-core discovers them via
|
||
`ACTIVITY_DEFINITION_DIRS` or the external ConfigMap mount.
|
||
|
||
---
|
||
|
||
## Sinks (ACTIVITY-WP-0022)
|
||
|
||
| `ISSUE_SINK_TYPE` | Use |
|
||
| ---------------- | --- |
|
||
| **`state-hub`** (default) | Internal fleet work → `activity_task_spawn` progress |
|
||
| **`null`** | Dry-run |
|
||
| **`rest`** | Explicit external issue-core / Forgejo only |
|
||
|
||
Do not flip production to `rest` globally.
|
||
|
||
---
|
||
|
||
## Reference implementations
|
||
|
||
| Domain | Definition | Executor |
|
||
| ------ | ---------- | -------- |
|
||
| **Binky** | `binky-daily-rhythm` in activity-core / binky | `rein-aharness brief-daily` + `binky-control/scripts/railiance-rhythm/` |
|
||
| **Freedom Intelligence** | `fi-daily-research-brief` | `rein-aharness fi-research-brief` + `freedom-intelligence/scripts/railiance-rhythm/` |
|
||
|
||
---
|
||
|
||
## Operator daily habit
|
||
|
||
1. Ops console: did automations run? (`/ops/automations/status`) — includes
|
||
`ops_runs.counts` and `stuck_open_or_claimed` (SLA, default 1h)
|
||
2. Claim queue: `GET /ops-runs?state=open` (or failed) — harness backlog
|
||
3. State Hub: `activity_task_spawn` + domain completion events
|
||
4. Domain repo: brief/artifact files + git history
|
||
5. If something is missing, **fix activity-core first** (schedule paused?
|
||
definition disabled? resolver due stuck? open ops_run with no claim?) —
|
||
do not invent a new cron
|
||
|
||
---
|
||
|
||
## Related
|
||
|
||
- `INTENT.md` — when/what/where boundary
|
||
- `docs/adr/adr-005-ops-runs-vs-dev-work-records.md` — ops vs dev work
|
||
- `docs/ops-run-queue.md` — claim API
|
||
- `docs/deploy-ops-run-queue-railiance.md` — T07 railiance rollout checklist
|
||
- `docs/task-emission-consumer-contract.md` — spawn payload + consumer duties
|
||
- `docs/runbook.md` — sync, trigger, ops UI
|
||
- `docs/adr/adr-002-definition-format.md` — definition files
|