rein-aharness/docs/ops-run-claim-loop.md

# Ops run claim loop (REIN-A-0002)

**Primary intake** for activity-core scheduled automation.  
**Architecture:** ACT-ADR-005 · activity-core [ops-run-queue.md](../../activity-core/docs/ops-run-queue.md)

Does **not** require issue-core or Forgejo for FI / Binky-style ops.

## Flow

```text
activity-core emit_tasks
  → INSERT ops_run (open, optional profile + repository_grant) + task_spawn
        │
        ▼
rein-aharness claim-loop
  POST /ops-runs/claim
  POST /ops-runs/{id}/heartbeat (immediate ownership proof; fail closed)
  harness_profile_ref present? ─ yes → Glas ExecutionRequest → gateway
        │ no
        └─→ select_approach(labels, definition, hint) → legacy executor
  validate granted repository result + persist external metrics
  enqueue exact terminal close → POST complete | fail
  replay pending close evidence before another claim
```

`harness_profile_ref` is authoritative under ACT-ADR-006. A profiled row never
consults `approach_hint`, `select_approach`, or a legacy executor, including
after profile resolution refusal. The claim worker sends bounded
`ExecutionEvidence`; direct tool output/error and unknown provider fields are
not persisted in its close outbox. A separately typed grant activates commit,
path, clean-state, metadata, and remote-ref acceptance plus external metrics.
Profile-absent rows keep the legacy coexistence path, but cannot carry a grant.

## Environment

| Variable | Default | Meaning |
| -------- | ------- | ------- |
| `ACTIVITY_CORE_URL` | `http://127.0.0.1:8010` | actcore-api base |
| `ACTIVITY_CORE_WORKER_TOKEN` | unset | `X-Worker-Token` / Bearer |
| `AGENT_HARNESS_WORKER_ID` | `rein-aharness@hostname` | claim owner |
| `AGENT_HARNESS_OPS_LABELS` | `automated` | claim label filter |
| `AGENT_HARNESS_OPS_LABELS_MODE` | `any` | `any` or `all` |
| `AGENT_HARNESS_OPS_LEASE_SECONDS` | `900` | claim lease |
| `AGENT_HARNESS_CLAIM_INTERVAL` | `30` | empty-queue poll interval |
| `AGENT_HARNESS_REQUIRED_PROFILE_REFS` | unset | comma-separated exact production `profile@version` refs checked at startup |
| `AGENT_HARNESS_LEGACY_APPROACHES_UNTIL` | unset | ISO expiry date required for profile-absent compatibility routing |
| `AGENT_HARNESS_REPO_MAP` | `{}` | JSON slug→checkout path |
| `AGENT_HARNESS_REPO_ROOTS` | `~:~/work` | slug search roots |
| `REIN_AHARNESS_STATE_DIR` | user state directory | private repository-lock, metrics, and close-outbox state |

Example map on railiance01:

```bash
export AGENT_HARNESS_REPO_MAP='{
  "freedom-intelligence":"/home/tegwick/freedom-intelligence",
  "binky-control":"/home/tegwick/binky-control"
}'
export AGENT_HARNESS_LEGACY_APPROACHES_UNTIL=2026-12-31
export ACTIVITY_CORE_URL=http://127.0.0.1:8010   # or ClusterIP via tunnel
export ACTIVITY_CORE_WORKER_TOKEN=…               # from actcore-runtime-secret
```

## Approach registry

| Match | Approach |
| ----- | -------- |
| labels `research-brief` / `freedom-intelligence` or fi-daily* | `fi-research-brief` |
| labels `rhythm` / `binky-daily` or binky-daily* | `brief-daily` |
| labels `binky` + `weekly-review` or binky-weekly-review* | `brief-weekly` |
| labels `mail-intake` | `mail-scan` then `mail-triage` |
| labels `agent-session` | agentic `run` (TaskSpec) |
| none | fail ops_run (no silent drop) |

For a profile-absent legacy row, optional `approach_hint` overrides matching.
It has no effect when `harness_profile_ref` is present.

Add a row: edit `rein_aharness/approaches.py` (`APPROACH_RULES` + `execute_approach`)
and tests in `tests/test_approaches.py`. Prefer cheapest correct path
(structured llm-connect before open-ended agent session).

## CLI

```bash
# Peek open ops_runs + selected approach
rein-aharness poll --source=ops-run --no-claim

# Claim one, execute approach, complete/fail
rein-aharness run --from-ops-run
# or
rein-aharness poll --source=ops-run

# Continuous worker (systemd)
rein-aharness claim-loop
rein-aharness claim-loop --once --verbose

# Read-only startup readiness (never claims)
rein-aharness preflight

# Required close evidence and recovery
rein-aharness close-outbox status
rein-aharness close-outbox replay

# Dry-run: claim then fail+reopen without domain work
rein-aharness poll --source=ops-run --dry-run
```

Legacy issue-core (external tickets only):

```bash
rein-aharness poll --source=issue-core
rein-aharness run --from-issue-core
```

## Install claim-loop on railiance01 (user systemd)

```bash
# From rein-aharness checkout on railiance01
./deploy/scripts/install-pinned-runtime.sh
./deploy/scripts/install-claim-loop-user.sh

# Or manually:
mkdir -p ~/.config/systemd/user
cp deploy/systemd/rein-aharness-claim-loop.service ~/.config/systemd/user/
# Edit Environment= paths / token in the unit or env file
systemctl --user daemon-reload
systemctl --user enable --now rein-aharness-claim-loop.service
journalctl --user -u rein-aharness-claim-loop -f
```

The unit runs `rein-aharness preflight` through the same environment-loading
wrapper before every start. It probes Activity Core with a read-only open-run
list, verifies configured repository workspaces and private runtime state, and
refuses unresolved close-evidence quarantine without claiming or executing
work. Pending close evidence is allowed through startup because the first
claim-loop cycle replays it before any new claim; quarantine requires operator
review.

Set `AGENT_HARNESS_REQUIRED_PROFILE_REFS` to the comma-separated exact
`profile@version` references enabled for production. Every named profile must
resolve through the installed Glas catalog with `operational_readiness: ready`.
A ready `profile.bwrap-local` also executes a minimal bwrap namespace probe,
which detects missing bubblewrap and host user-namespace/AppArmor refusal.
Leaving the variable empty declares that no profiled production definition is
enabled; this does not make a blocked profile executable or introduce fallback.

Use `rein-aharness preflight --offline` only while installing or diagnosing
local files. It skips the Activity Core probe and is not the systemd gate.

`install-pinned-runtime.sh` first verifies clean sibling checkouts against
`deploy/runtime-contract-lock.json`, then applies the checked-in `uv.lock` with
`--frozen --no-editable`. This prevents the authoritative service from running
against whichever mutable editable sibling happened to be present. Updating a
runtime contract requires reviewing and committing both locks together.

`make contract-test` is the non-skippable cross-package release gate. It imports
Glas, sand-boxer, and llm-connect before running the contract/claim suites, so
the optional `pytest.importorskip` development behavior cannot turn a missing
production dependency into a green release result.

Forgejo runs the same gate from
`.forgejo/workflows/runtime-contract.yaml` on every main push. It follows the
estate runner contract (direct clone, no `actions/checkout`), detaches every
sibling at the revision declared in the runtime lock, applies the frozen uv
resolution, and needs no deployment or cluster credential.

`make recovery-test` is the deterministic pre-deploy recovery gate. It covers
initial and periodic lease rejection, adapter cancellation, repository-lock
release, response-lost close reconciliation, pending-close claim blocking,
every permanent Activity Core close code (including `expired_lease`), and Glas
cancellation. It does not replace the isolated expired-row API smoke or the
natural-run and sandbox-cleanup observations required from the deployed host.

### Installed-runtime recovery drill

Run `scripts/prove-runtime-recovery.py` with the selected artifact's interpreter:

```bash
"${RUNTIME}/bin/python3" -I -B scripts/prove-runtime-recovery.py \
  --runtime "${RUNTIME}" --sha256 "${RUNTIME_SHA256}" \
  --profile-ref harness.agent-dev-local@1.1.1
```

The script verifies the complete artifact digest and that all four runtime
packages load from that artifact. It uses private temporary repositories,
sandbox stores and worker state. The queue and rein authoring are fixtures;
the installed worker, Glas lifecycle, bwrap execution, repository acceptance,
external metrics and close outbox are real. No provider or production queue
client is invoked. The selected profile is made ready only in an in-memory
fixture catalog; no installed profile or standing service is changed.

The receipt covers response-lost close replay with one execution/commit,
execution failure, periodic lease rejection, a real SIGTERM to the disposable
worker, a killed lock holder, and SIGKILL of a disposable worker after sandbox
creation. Cancellation deliberately returns a late sandbox commit to prove
that it is not imported into the source. Every case checks lock reacquisition,
source state, sandbox/workspace cleanup and applicable durable evidence.

After abrupt worker death, cleanup is **explicit owner recovery**, not an
automatic startup sweep. Recover only the sandbox id attributed to the failed
run, using sand-boxer's `get`/`destroy` with the same owner state directory.
Verify destruction and workspace removal before resuming. Do not delete lock
files to release a kernel lock, erase pending close evidence, or replay the
workload to reconcile a close. Use `close-outbox status|replay` for that last
step. Activity Core remains responsible for expiring the abandoned queue lease.

Railiance receipt: `docs/evidence/2026-09-27-installed-runtime-recovery.json`.
It proves these host mechanisms against runtime `b6e4e8a4`; it does not prove
live Activity Core expiry, credential delivery, or a paid production run.

### Host access to cluster services (no port-forward)

On railiance01 (single-node k3s), set **k8s://** pseudo-URLs in
`claim-loop.env` (see `deploy/systemd/claim-loop.env.example`). The
`~/bin/rein-aharness-claim` wrapper resolves them to Service ClusterIPs at
start. Canon: activity-core `docs/llm-connect-host-access.md`.

```bash
# migrate existing env
./deploy/scripts/refresh-claim-loop-k8s-urls.sh
# reinstall wrapper
./deploy/scripts/install-claim-loop-user.sh
systemctl --user restart rein-aharness-claim-loop.service
```

**Deprecated:** long-lived `kubectl port-forward` to 127.0.0.1:8010 / :8080.

Legacy note (break-glass only) if ClusterIP routing is broken:

```bash
kubectl -n activity-core port-forward svc/actcore-api 8010:8010
export ACTIVITY_CORE_URL=http://127.0.0.1:8010
```

## Cutover from host timers (T05)

Host oneshot timers (FI 07:35, Binky rhythm) remain **break-glass** until:

1. claim-loop is running and healthy  
2. **5 clean weekday cycles** (or 5 forced triggers) with open→claimed→succeeded  
3. Then disable or demote timers — see `deploy/docs/timer-cutover.md`

Until then dual-path is intentional: schedule truth is activity-core; timers are
a safety net if claim-loop is down.

## Labels required on definitions

| Domain | Labels (min) |
| ------ | ------------ |
| Freedom Intelligence | `automated`, `research-brief` (already on FI rule) |
| Binky daily | `automated`, `rhythm` |
| Binky mail | `automated`, `mail-intake` |