activity-core/SCOPE.md
tegwick 9ef75e0e18
Some checks failed
CI Smoke / host-smoke (push) Successful in 0s
CI Smoke / container-smoke (push) Successful in 1s
Build and Publish Container Image / build-and-push (push) Failing after 2m10s
feat: checkpoint standalone Custodian snapshot receiver
Assistant: codex
Assistant-Model: gpt-6.1-sol
Assistant-Session: 01a15374-f59f-7a31-86d2-d23b5d2ed181
2026-10-19 17:56:32 +02:00

397 lines
20 KiB
Markdown

---
domain: capabilities
repo: activity-core
updated: "2026-08-23"
---
# SCOPE
> This file helps you quickly understand what this repository is about,
> when it is relevant, and when it is not.
---
## One-liner
activity-core is the org-wide Event Bridge for the Coulomb organization — a
rule-governed event loop that receives time-based and domain events, evaluates
declarative rules and LLM instructions against current org context, and emits
structured task, report, and evidence outputs without owning downstream task
lifecycle.
---
## Core Idea
An `ActivityDefinition` (a markdown file checked into a repo) declares a trigger
(cron, one-off scheduled datetime, or named event type), context sources to
resolve before evaluation, and a set of rules and instructions that determine
what work or evidence to produce. When triggered, a durable Temporal workflow
loads the definition, resolves context, evaluates the rule/instruction set, and
emits outputs via configured sinks:
- **Coordination outputs** — task specs through `IssueSink` (`rest` →
issue-core, `state-hub` progress, or `null` dry-run), plus claimable
`ops_runs` for automation executors when the queue is enabled.
- **Reports** — instruction reports to working memory and/or State Hub progress.
- **Evidence** — compact non-secret summaries from selected resolvers (ops
inventory, package prune, legacy-meter, etc.).
Everything is auditable at the orchestration boundary: `activity_runs` records
the activation and context snapshot; `task_spawn_log` records emission facts;
`ops_runs` records claim/lease/outcome evidence; report sinks leave
progress/working-memory artifacts. LLM audit stores the model and prompt hash,
not raw prompts or provider payloads.
The two evaluation modes:
- **Rule** — deterministic condition (sandboxed Python-like DSL) → fixed task
templates. Fast, testable, no LLM cost.
- **Instruction** — optional pre-filter → LLM or **deterministic** report path
(`model: deterministic` / `none` / `unused`) → structured tasks and/or
reports. LLM reserved for genuine judgement; deterministic for pure context
summaries.
---
## In Scope
- **ActivityDefinition model**: trigger config (cron / scheduled / event),
context sources, rules (condition + action), instructions (trusted-field
prompt + model + output schema + report sinks).
- **Event type registry**: publisher-declared markdown definitions with
attribute schemas, example payloads, and intent documentation.
Curator-gating configurable per runtime environment.
- **Trigger types**: 5-field cron with timezone and misfire policy; one-off
scheduled datetime; event-type subscription via NATS; manual one-shot API
trigger; one-shot schedule smoke tests for recurring definitions.
- **Schedule reconcile**: Temporal Schedule upsert/pause with per-row error
isolation (`sync_schedules` continues after `ScheduleAlreadyRunningError`).
- **Context resolution adapters**: repo-scoping, State Hub, hub-core,
reuse-surface, event payloads, Kaizen agent metadata, SBOM Nexus, shell
queries, CNPG backup, Forgejo package prune, and ops inventory probes. The
resolver registry is extensible.
- **Rule evaluator**: sandboxed AST walker; `for_each` binding; no `exec()`.
- **Instruction executor**: trusted-field rendering, llm-connect, structured
validation, item-granular recovery, deterministic report path, report sinks.
- **Task emission adapter**: `IssueCoreRestSink`, `StateHubProgressSink`,
`NullSink` selected by `ISSUE_SINK_TYPE`. Production railiance01 currently
uses **`state-hub`** as the safe default for internal findings (see
ACTIVITY-WP-0021/0022); restore `rest` only when issue-core→Forgejo is healthy
**and** policy allows external issues.
- **Report / evidence sinks**: working memory, State Hub progress
(`daily_triage`, `sbom_staleness`, `forgejo_package_prune`,
`ops_inventory_probe`, `activity_task_spawn`, etc.).
- **Spawn audit log**: `task_spawn_log` with rule/instruction id and sink ref.
- **Automation execution queue**: durable `ops_runs` with idempotent insertion,
concurrent claim (`FOR UPDATE SKIP LOCKED`), leases, heartbeat, retry/reopen,
terminal outcome, and operator visibility. This is machine-run delivery
state, not the authoritative lifecycle of a workplan task or tracker issue.
- **Profile-driven execution contract**: structurally validate a pinned
`harness_profile_ref`, pass through allowlisted attribution refs, and retain
normalized non-secret Glas execution evidence. Glas selects the rein/model/
sandbox constellation; activity-core does not.
- **Webhook receiver**: Gitea/GitHub → EventEnvelope → NATS, with HMAC
verification when the corresponding shared secret is configured.
- **Worker and workflow infrastructure**: Temporal `RunActivityWorkflow`
(load → resolve → evaluate → report → **log_run** → emit). Emit failures no
longer skip `activity_runs` rows.
- **REST admin API** (FastAPI): basic ActivityDefinition row CRUD, manual
trigger, admin sync, event-type approval, and dependency health. Full
rules/instructions authoring remains markdown-and-sync driven; the row CRUD
schema is not a complete replacement for the file format.
- **Operator automation console** (ACTIVITY-WP-0024): `/ops` inventory, status
window, run history, Run now / enable / disable / pause with operator-token
auth (fail-closed), plus thin HTML UI at `/ops/ui`.
- **Automation status surface**: `make automation-status` and
`make prod-automation-status` / `scripts/prod_automation_status.sh` for
railiance01 evidence without LLM authority (same contract as `/ops/automations/status`).
- **Review CLI**: the `activity` command merges repository commits with
activity/ops-run evidence across configured services, supports checkpoints,
and reports automation definitions and recent runs. It is an evidence review
client, not a task tracker.
- **Operational runbook**: `docs/runbook.md`, emission boundary docs.
---
## Out of Scope
- **Task lifecycle** — create/assign/track/close remains issue-core (or fleet
work-record connectors). activity-core holds spawn evidence and machine-run
delivery state, not the authoritative human/work-item record.
- **Project and initiative management** — project-core (future).
- **General execution workers** — agent loops, repository changes, arbitrary
scans, and privileged API writes belong to Glas/reins, per-repo workers, or
platform tools. The `ops_run` API dispatches and observes this work; it does
not perform it.
- **General ops control plane** — Kubernetes, SSH, tunnels, secret custody,
OpenBao policy administration.
- **Event broker / Temporal server hosting** — consume, do not own lifecycle.
- **End-user task UI** — tracking human work items. The **operator automation
console** (`/ops`) is in scope; it manages ActivityDefinitions and schedules,
not task lifecycle.
- **Coding assistant schedulers** as production authority.
**Boundary note (ACT-ADR-007):** Context resolution is read-only. A small
code-owned registry permits explicit bounded operations (currently controlled
SBOM ingest, Forgejo package prune, and CNPG backup) in a separate workflow
phase. Unknown operations or incomplete mutation/evidence declarations are not
an extension mechanism and must fail definition admission.
---
## Relevant When
- Org-wide recurring maintenance (SBOM, package prune, legacy-meter, Binky
rhythm, consistency sweeps) without bespoke per-service cron.
- Reactive task/report generation from org events and context.
- One-off future scheduling.
- Auditable activation history (runs, spawn log, progress).
- Answering “how did automations go since Sunday?” from deterministic
repo-native / prod evidence.
## Not Relevant When
- Tracking task completion/reassignment → issue-core / work records.
- Multi-phase project orchestration → project-core.
- Simple host cron with no durability or evidence needs.
---
## Current State (2026-08-23)
### Status
Production runs on **railiance01** in the `activity-core` namespace with
PostgreSQL, Temporal Schedules, a Temporal worker, API, and NATS event router.
The checked-in surface contains 12 ActivityDefinitions plus publisher-declared
event types. The schema is at migration `0008`, including the ops-run queue and
Glas profile/attribution fields.
No open task is independently actionable on 2026-08-23:
| Workplan | State | Current gate |
| --- | --- | --- |
| ACTIVITY-WP-0031 | wait | OpenRouter account owner must replace the rejected provider key; external routing handoffs remain with their named owners |
| ACTIVITY-WP-0032 | wait | `GLAS-IN-0002`: managed consumer namespace plus rein/model runtime and egress contract before a successful commit proof |
| ACTIVITY-WP-0034 | progress | Observe the first unassisted weekday controlled-source SBOM fire; do not substitute another manual trigger |
### Live posture (railiance01)
- Image: `activity-core:railiance01-prod` (locally built/imported deployment).
- `ISSUE_SINK_TYPE=state-hub` (internal findings → State Hub progress, not
Forgejo issues).
- The daily SBOM catch-up is enabled at weekday 09:15 Europe/Berlin, limit 3.
It freezes Repo Manager's full-SHA `forgejo-archive-v1` source reference,
reuses it across Nexus retries, and reports only allowlisted provenance.
- The old weekly SBOM fan-out is disabled in source, DB, and Temporal.
- `ops_runs` are claimable by rein-aharness and carry pinned Glas profile refs;
production has proven normalized fail-closed evidence and sandbox teardown,
but not yet a successful profiled commit.
- Weekly prune: worker has `FORGEJO_TOKEN` + hostPath `/opt/railiance-platform`
+ `live_images_file` (kubectl absent in pod).
- The deterministic status surface distinguishes schedule fires, emitted
tasks, ops-run states, and bounded failures.
### Definitions (repo files + prod DB)
| Definition | Role |
| --- | --- |
| `daily-sbom-catchup` | Bounded oldest-N Nexus ingestion with immutable controlled source refs |
| `weekly-sbom-staleness` | Disabled legacy fan-out; retained as deterministic report definition |
| `weekly-forgejo-package-prune` | Sunday 03:30 UTC apply prune + evidence |
| `weekly-legacy-meter-review` / `legacy-meter-8h-capture` | Meter evidence |
| `weekly-coding-retro` | Present; enablement depends on `coding_retro` feed |
| Binky daily / mail / review-prep | Rhythm tasks via IssueSink |
| `daily-cnpg-option-a-backup` | Declared bounded CNPG backup plus evidence |
| `monthly-secrets-elevation-review` | Scheduled governance review task |
| `glas-profile-pilot` | Disabled low-risk profile-contract proof definition |
### Task emission posture
- **Canonical REST sink** to issue-core remains implemented and documented.
- **Production default is `state-hub`** after Forgejo PAT/backend failures and
fleet policy against silent Forgejo issues (WP-0021 path B, WP-0022).
- Restoring `rest` requires healthy issue-core→Forgejo **and** explicit policy
opt-in per definition.
---
## Assessment Against Intent
The detailed review is preserved in
`history/2026-08-23-scope-against-intent-assessment.md`.
### Alignment
| INTENT promise | Status |
| --- | --- |
| Answers **when / what / where** | Met — Temporal schedules/NATS/webhooks, rules/instructions, and explicit sinks/targets |
| Markdown definitions | Met — `activity-definitions/`, event-types, external ConfigMaps |
| Rules before instructions | Met — deterministic rule/report paths dominate; LLM is optional |
| Durable recurring automation | Met — Temporal schedules plus deterministic status evidence |
| Audit trail | Met as a bounded, non-secret audit projection — activation, definition/version, context projection, prompt hash/model, validation, emission, queue, and normalized execution evidence; raw prompts/provider payloads are intentionally absent |
| Does not own task lifecycle | Met — `ops_run` claim/lease/outcome state is runtime delivery state, not work-item lifecycle |
| Does not own project phases | Met |
### Gaps and tensions (INTENT ↔ practice)
| Gap | Severity | Notes |
| --- | --- | --- |
| **G3. Live LLM execution** | High (operational, external) | The implementation is present, but production provider requests fail with a sanitized upstream 401 until the account owner replaces the key. |
| **G4. Profiled execution proof** | Medium (operational, external) | Profile selection, failure evidence, and teardown are proven; a successful commit waits on `GLAS-IN-0002`. |
| **G5. State Hub retirement** | Medium (integration) | hub-core adapters exist, but production task/report progress still defaults to the State Hub compatibility path. |
| **G6. Definition REST parity** | Low | Basic REST row administration intentionally cannot author or round-trip full rules/instructions; markdown source sync is authoritative. |
### Drift risks
1. **Convenience execution** — growing the bounded side-effect allowlist into a
general ops runner.
2. **Silent sink restore** — flipping `ISSUE_SINK_TYPE=rest` without policy +
healthy Forgejo backend reintroduces issue spam / 503s.
3. **Protection file staleness** — weekly prune with `apply: true` depends on
fresh live-image lists.
4. **Queue/task conflation** — treating an `ops_run` as the authoritative human
task record rather than a delivery instance.
---
## How It Fits
```
[NATS JetStream / Temporal Schedules / manual trigger]
↓
[activity-core] Event type registry · rules · instructions · context resolvers
↓
[IssueSink: rest | state-hub | null] → issue-core or hub-core progress (State Hub until cutover)
[report/evidence sinks] → hub-core / working memory
[ops_run claim queue + Glas profile] → rein-aharness / other approved reins
[bounded operation stage] → code-registered platform tools
```
- **Upstream**: NATS, Temporal, PostgreSQL, repo-manager / hub-core (State Hub
until retirement), repo-scoping, llm-connect.
- **Downstream**: issue-core (optional), State Hub/hub-core progress, SBOM
Nexus and bounded platform tools, Glas/reins claiming `ops_runs`.
- **Orientation (not runtime)**: info-tech-canon (task/org/governance
vocabulary), policy-nexus (publication of ADRs/canon — not authored here).
- **Ops**: railiance01 k3s; workstation status via SSH helper.
---
## Terminology
- **ActivityDefinition** — markdown policy unit (trigger + context + rules/instructions).
- **EventEnvelope** — canonical inbound event shape.
- **Rule / Instruction** — deterministic vs judgement (or deterministic report) paths.
- **IssueSink** — task emission adapter (`rest` / `state-hub` / `null`).
- **Report sink / Evidence sink** — non-task activation outputs.
- **Spawn audit trail** — local emission record; not authoritative task state.
- **Activity** (Temporal) vs **ActivityDefinition** (app policy) — different concepts.
---
## Related / Overlapping
- `info-tech-canon` — task / organization / governance vocabulary; repository layout.
- `policy-nexus` — publishes ADRs/canon; does not author them.
- `repo-manager` — repo registry, consistency engine, work-record index.
- `hub-core` — messages, progress projections, extension ports (core-hub absorbed later).
- `state-hub` — compatibility read model until STATE-WP-0079 retirement.
- `issue-core` — task connector / external tracker mapping (not fleet work-record origin).
- `repo-scoping`, `reuse-surface`, `sbom-nexus`, `llm-connect`, `glas-harness`,
`rein-aharness`, `sand-boxer`.
- `railiance-platform` — Forgejo package prune tool and credential lanes.
- `ops-warden` — credential routing (not secret vending for non-SSH).
- `ops-bridge` tunnels.
---
## Architecture Decisions
- `docs/adr/adr-001-event-bridge-architecture.md`
- `docs/adr/adr-002-definition-format.md`
- `docs/adr/adr-003-rule-instruction-model.md`
- `docs/adr/adr-004-producer-trust-boundary.md`
- `docs/adr/adr-005-ops-runs-vs-dev-work-records.md`
- `docs/adr/adr-006-glas-profile-execution.md`
- `docs/adr/adr-007-bounded-operations.md`
---
## Getting Oriented
- Start: `INTENT.md`, this file, `docs/adr/`, `docs/runbook.md`.
- Key code: `workflows.py`, `activities.py`, `issue_sink.py`, `report_sinks.py`,
`context_resolvers/`, `automation_status.py`, `sync_schedules.py`.
- Definitions: `activity-definitions/`, `event-types/`, Railiance ConfigMaps.
- Status: `make automation-status SINCE=sunday` ·
`make prod-automation-status SINCE=sunday`.
---
## Provided Capabilities
```capability
type: data
title: Durable event-triggered task factory
description: >
Org-wide Event Bridge that receives time-based and domain events, evaluates
declarative rules and LLM instructions against current org context, emits
task/report/evidence outputs, and offers a leased ops-run delivery queue with
normalized execution evidence while leaving work-item lifecycle downstream.
keywords: [temporal, workflow, event-bridge, ops-run, glas, task, report, evidence, cron, event, rule, instruction, org-automation]
```
## Daily security-flow consumer
`activity-definitions/daily-security-flow.md` supplies a fixed-command daily
schedule for the Custodian security-flow runner (CUST-WP-0079-T03,
ACTIVITY-WP-0047). Activity Core owns scheduling and ops_run evidence; the
claiming workstation rein owns execution, and the Custodian script owns report
and recipient checkpoint delivery. The definition carries no repository grant.
The checked-in definition remains disabled pending workstation/hub acceptance,
owner activation and the first unassisted fire. This source integration does
not establish an active automation. Consumers can reuse the existing schedule,
execution queue and profile contract rather than add another scheduler.
## Staged work-record generator adoption
The tracked `.generated-work-records.json` opts this repository into the shared
pre-commit generation lane. `make generated-work-records` uses the isolated
`.generated-work-records.mk` recipe and the Custodian adapter to render from
staged Markdown with the existing State Hub renderer. `make
generated-work-records-check` verifies the staged result without writes.
Unstaged source edits are preserved; unstaged generated-file edits and unresolved
conflicts stop the commit. The existing generated-only guard and secret scan
remain in place. This adoption covers WORK-RECORDS.md only; other owner indexes
retain their own generators. It makes no schedule or public-release change.
## Custodian deterministic operations
The existing bounded-operation stage registers custodian_security_flow and
custodian_allocation_review: fixed owner commands with trusted operator-bound
source/fleet/private-state inputs, explicit definition parameters, timeout
containment and compact counted/hash receipts. Both definitions remain disabled.
No coding-agent task or green-commit profile executes these routines. Runtime
placement, accepted mounts/network, delivery and first unattended proof remain
unverified; see docs/custodian-bounded-operations.md. This adds no scheduler or
generic shell. The generated .custodian-brief.md is now an ignored local
orientation snapshot; its existing bytes remain local.
Custodian delivery preparation now includes an image-baked, hash-verified four-file
owner bundle, tunable input policy, and a required daily pre-execution gate validating
completion markers, exact snapshot hashes and independent per-repo Forge
freshness receipts. The reviewed candidate in `k8s/custodian-release/` is outside
active kustomization; placeholders prevent claiming an accepted image/snapshot.
Both definitions stay disabled. No runtime publication, retention deletion or
schedule activation is implemented by these source artifacts.
Standalone `custodian_capture.py` prepares bounded publisher response reception
and retained descriptor advancement. It is not registered as another bounded
operation and is not integrated into the daily command. The checkpoint and
remaining timeout/transport/access work are in `k8s/custodian-release/RESTART.md`.