rein-aharness/docs/ops-run-claim-loop.md

Ops run claim loop (REIN-A-0002)

Primary intake for activity-core scheduled automation.
Architecture: ACT-ADR-005 · activity-core ops-run-queue.md

Does not require issue-core or Forgejo for FI / Binky-style ops.

Flow

activity-core emit_tasks
  → INSERT ops_run (open, optional profile + repository_grant) + task_spawn
        │
        ▼
rein-aharness claim-loop
  POST /ops-runs/claim
  POST /ops-runs/{id}/heartbeat (immediate ownership proof; fail closed)
  harness_profile_ref present? ─ yes → Glas ExecutionRequest → gateway
        │ no
        └─→ select_approach(labels, definition, hint) → legacy executor
  validate granted repository result + persist external metrics
  enqueue exact terminal close → POST complete | fail
  replay pending close evidence before another claim

harness_profile_ref is authoritative under ACT-ADR-006. A profiled row never consults approach_hint, select_approach, or a legacy executor, including after profile resolution refusal. The claim worker sends bounded ExecutionEvidence; direct tool output/error and unknown provider fields are not persisted in its close outbox. A separately typed grant activates commit, path, clean-state, metadata, and remote-ref acceptance plus external metrics. Profile-absent rows keep the legacy coexistence path, but cannot carry a grant.

Environment

Variable	Default	Meaning
ACTIVITY_CORE_URL	http://127.0.0.1:8010	actcore-api base
ACTIVITY_CORE_WORKER_TOKEN	unset	X-Worker-Token / Bearer
AGENT_HARNESS_WORKER_ID	rein-aharness@hostname	claim owner
AGENT_HARNESS_OPS_LABELS	automated	claim label filter
AGENT_HARNESS_OPS_LABELS_MODE	any	any or all
AGENT_HARNESS_OPS_LEASE_SECONDS	900	claim lease
AGENT_HARNESS_CLAIM_INTERVAL	30	empty-queue poll interval
AGENT_HARNESS_REQUIRED_PROFILE_REFS	unset	comma-separated exact production profile@version refs checked at startup
AGENT_HARNESS_LEGACY_APPROACHES_UNTIL	unset	ISO expiry date required for profile-absent compatibility routing
AGENT_HARNESS_REPO_MAP	{}	JSON slug→checkout path
AGENT_HARNESS_REPO_ROOTS	~:~/work	slug search roots
REIN_AHARNESS_STATE_DIR	user state directory	private repository-lock, metrics, and close-outbox state

Example map on railiance01:

export AGENT_HARNESS_REPO_MAP='{
  "freedom-intelligence":"/home/tegwick/freedom-intelligence",
  "binky-control":"/home/tegwick/binky-control"
}'
export AGENT_HARNESS_LEGACY_APPROACHES_UNTIL=2026-12-31
export ACTIVITY_CORE_URL=http://127.0.0.1:8010   # or ClusterIP via tunnel
export ACTIVITY_CORE_WORKER_TOKEN=…               # from actcore-runtime-secret

Approach registry

Match	Approach
labels research-brief / freedom-intelligence or fi-daily*	fi-research-brief
labels rhythm / binky-daily or binky-daily*	brief-daily
labels binky + weekly-review or binky-weekly-review*	brief-weekly
labels mail-intake	mail-scan then mail-triage
labels agent-session	agentic run (TaskSpec)
none	fail ops_run (no silent drop)

For a profile-absent legacy row, optional approach_hint overrides matching. It has no effect when harness_profile_ref is present.

Add a row: edit rein_aharness/approaches.py (APPROACH_RULES + execute_approach) and tests in tests/test_approaches.py. Prefer cheapest correct path (structured llm-connect before open-ended agent session).

CLI

# Peek open ops_runs + selected approach
rein-aharness poll --source=ops-run --no-claim

# Claim one, execute approach, complete/fail
rein-aharness run --from-ops-run
# or
rein-aharness poll --source=ops-run

# Continuous worker (systemd)
rein-aharness claim-loop
rein-aharness claim-loop --once --verbose

# Read-only startup readiness (never claims)
rein-aharness preflight

# Required close evidence and recovery
rein-aharness close-outbox status
rein-aharness close-outbox replay

# Dry-run: claim then fail+reopen without domain work
rein-aharness poll --source=ops-run --dry-run

Legacy issue-core (external tickets only):

rein-aharness poll --source=issue-core
rein-aharness run --from-issue-core

Install claim-loop on railiance01 (user systemd)

# From rein-aharness checkout on railiance01
./deploy/scripts/install-pinned-runtime.sh
./deploy/scripts/install-claim-loop-user.sh

# Or manually:
mkdir -p ~/.config/systemd/user
cp deploy/systemd/rein-aharness-claim-loop.service ~/.config/systemd/user/
# Edit Environment= paths / token in the unit or env file
systemctl --user daemon-reload
systemctl --user enable --now rein-aharness-claim-loop.service
journalctl --user -u rein-aharness-claim-loop -f

The unit runs rein-aharness preflight through the same environment-loading wrapper before every start. It probes Activity Core with a read-only open-run list, verifies configured repository workspaces and private runtime state, and refuses unresolved close-evidence quarantine without claiming or executing work. Pending close evidence is allowed through startup because the first claim-loop cycle replays it before any new claim; quarantine requires operator review.

Set AGENT_HARNESS_REQUIRED_PROFILE_REFS to the comma-separated exact profile@version references enabled for production. Every named profile must resolve through the installed Glas catalog with operational_readiness: ready. A ready profile.bwrap-local also executes a minimal bwrap namespace probe, which detects missing bubblewrap and host user-namespace/AppArmor refusal. Leaving the variable empty declares that no profiled production definition is enabled; this does not make a blocked profile executable or introduce fallback.

Use rein-aharness preflight --offline only while installing or diagnosing local files. It skips the Activity Core probe and is not the systemd gate.

install-pinned-runtime.sh first verifies clean sibling checkouts against deploy/runtime-contract-lock.json, then applies the checked-in uv.lock with --frozen --no-editable. This prevents the authoritative service from running against whichever mutable editable sibling happened to be present. Updating a runtime contract requires reviewing and committing both locks together.

make contract-test is the non-skippable cross-package release gate. It imports Glas, sand-boxer, and llm-connect before running the contract/claim suites, so the optional pytest.importorskip development behavior cannot turn a missing production dependency into a green release result.

Forgejo runs the same gate from .forgejo/workflows/runtime-contract.yaml on every main push. It follows the estate runner contract (direct clone, no actions/checkout), detaches every sibling at the revision declared in the runtime lock, applies the frozen uv resolution, and needs no deployment or cluster credential.

make recovery-test is the deterministic pre-deploy recovery gate. It covers initial and periodic lease rejection, adapter cancellation, repository-lock release, response-lost close reconciliation, pending-close claim blocking, every permanent Activity Core close code (including expired_lease), and Glas cancellation. It does not replace the isolated expired-row API smoke or the natural-run and sandbox-cleanup observations required from the deployed host.

Installed-runtime recovery drill

Run scripts/prove-runtime-recovery.py with the selected artifact's interpreter:

"${RUNTIME}/bin/python3" -I -B scripts/prove-runtime-recovery.py \
  --runtime "${RUNTIME}" --sha256 "${RUNTIME_SHA256}" \
  --profile-ref harness.agent-dev-local@1.1.1

The script verifies the complete artifact digest and that all four runtime packages load from that artifact. It uses private temporary repositories, sandbox stores and worker state. The queue and rein authoring are fixtures; the installed worker, Glas lifecycle, bwrap execution, repository acceptance, external metrics and close outbox are real. No provider or production queue client is invoked. The selected profile is made ready only in an in-memory fixture catalog; no installed profile or standing service is changed.

The receipt covers response-lost close replay with one execution/commit, execution failure, periodic lease rejection, a real SIGTERM to the disposable worker, a killed lock holder, and SIGKILL of a disposable worker after sandbox creation. Cancellation deliberately returns a late sandbox commit to prove that it is not imported into the source. Every case checks lock reacquisition, source state, sandbox/workspace cleanup and applicable durable evidence.

After abrupt worker death, cleanup is explicit owner recovery, not an automatic startup sweep. Recover only the sandbox id attributed to the failed run, using sand-boxer's get/destroy with the same owner state directory. Verify destruction and workspace removal before resuming. Do not delete lock files to release a kernel lock, erase pending close evidence, or replay the workload to reconcile a close. Use close-outbox status|replay for that last step. Activity Core remains responsible for expiring the abandoned queue lease.

Railiance receipt: docs/evidence/2026-09-27-installed-runtime-recovery.json. It proves these host mechanisms against runtime b6e4e8a4; it does not prove live Activity Core expiry, credential delivery, or a paid production run.

Host access to cluster services (no port-forward)

On railiance01 (single-node k3s), set k8s:// pseudo-URLs in claim-loop.env (see deploy/systemd/claim-loop.env.example). The ~/bin/rein-aharness-claim wrapper resolves them to Service ClusterIPs at start. Canon: activity-core docs/llm-connect-host-access.md.

# migrate existing env
./deploy/scripts/refresh-claim-loop-k8s-urls.sh
# reinstall wrapper
./deploy/scripts/install-claim-loop-user.sh
systemctl --user restart rein-aharness-claim-loop.service

Deprecated: long-lived kubectl port-forward to 127.0.0.1:8010 / :8080.

Legacy note (break-glass only) if ClusterIP routing is broken:

kubectl -n activity-core port-forward svc/actcore-api 8010:8010
export ACTIVITY_CORE_URL=http://127.0.0.1:8010

Cutover from host timers (T05)

Host oneshot timers (FI 07:35, Binky rhythm) remain break-glass until:

1. claim-loop is running and healthy
2. 5 clean weekday cycles (or 5 forced triggers) with open→claimed→succeeded
3. Then disable or demote timers — see deploy/docs/timer-cutover.md

Until then dual-path is intentional: schedule truth is activity-core; timers are a safety net if claim-loop is down.

Labels required on definitions

Domain	Labels (min)
Freedom Intelligence	automated, research-brief (already on FI rule)
Binky daily	automated, rhythm
Binky mail	automated, mail-intake