coordination-engine/spec/worker-coordination-service-v0.1.md

# Worker Coordination Service v0.1

Status: local-alpha implementation; operational defaults approved in WP-0003-T02.
Owner: coordination-engine. Transport owner: tmux-amq. State owner: repositories
and State Hub's file-backed projections.

This revision reconciles the original proposal with TAMQ's implemented
`spec/coordination-engine-adapter-v0.1.md`. The original requirements are retained
in `history/260907-worker-coordination-service-v0.1-proposal.md`; TAMQ-specific
CLI, history, replay, endpoint registration and logging requirements belong to
that repository. This package installs `coordination-engine`, never `tamq`.

## Observation and actionability

Cross-owner waits that used to become stranded `wait` notes are specified in
`spec/cross-owner-wait-mode-v0.1.md`. That mode **reuses** this service's
observation, leases, safety gates and receipts; it does not add a second
wakeup path or enable live injection.

The runtime polls State Hub's repos, workplans, tasks, dependency edges and unread
messages. It dispatches only after a complete successful snapshot, only for
explicitly configured gita repositories. It refreshes gita before addressing a
TAMQ target and on worker updates; it never clones or registers repositories.

Task triggers require ready/active workplans, todo/progress task status, and
satisfied workplan/workplan or workplan/task dependencies. Wait tasks remain
waiting until their authoritative status changes. Unknown dependency types or
missing targets wait. The Hub's edge direction is `from_workplan_id` depends on
`to_workplan_id` or `to_task_id`.

The current Hub has no action-request flag on messages. An unread, unarchived
message whose subject starts with `[action]` requests inspection. Other messages
do not wake workers. Message text never grants authority or changes workflow
state. Generic free-text progress records are not interpreted as checkpoints;
workers publish structured checkpoints through the local control socket.

A normalized trigger has source (`task`, `message`, `checkpoint`), source ID,
source revision, repository and trigger ID. The ID hashes the tuple
`(source, source_id, source_revision, repo)`. Revisions hash the relevant observed
record, plan and dependency state, so dependency completion is reconsidered even
when the task timestamp does not change. Payload text is not retained locally
as trigger data and is never copied to a wake prompt.

## Policy and safety

The named `default` policy permits repository inspection, authorized edits,
local checks, state updates and tmux wakes; configuration may narrow this set.
Unknown profiles/action classes are refused. Human flags, intervention notes,
blocking reasons and conservative sensitive-action text detection stop a
trigger. Sensitive retries are always zero in v0.1. Text detection adds stops;
it does not authorize arbitrary source text. Workers must independently recheck
current repository instructions, scope, dependency state and authority.

Authentication failures, unregistered/mismatched targets, incompatible protocol
and ambiguous endpoints stop the lease. Missing endpoints and ordinary transport
failures use bounded backoff. No safety stop silently reroutes to another worker.
A new source revision may be evaluated normally; the same stopped revision is
never automatically reset. A changed runtime configuration stops pending leases
created under the old configuration rather than silently changing their policy.

## Leases, checkpoints and completion

SQLite stores unique trigger and coordination lease IDs, source/revision/repo,
state, attempts, expiry, retry deadline, policy/configuration versions, checkpoint,
and TAMQ endpoint/message/transport state. State transitions and audit receipts
are committed together. One unexpired offered/acknowledged/running lease per
repository prevents overlapping automatic wakes, including across task revisions.

States are pending, waiting, offered, acknowledged, running, checkpointed,
retry_wait, completed and stopped. Every transition emits a sanitized receipt.
Workers acknowledge before work, renew while active, and checkpoint or complete.
Unknown, expired, terminal and wrong-repository lease updates are rejected.

Transport recovery repeats the same coordination lease ID and identical prompt.
TAMQ deduplicates admission and returns its original message ID. Expiry/failure
uses configurable bounded backoff and a 0–9 attempt ceiling. Already delivered
TAMQ messages are not forcibly reinjected by coordination retry; an unresponsive
worker eventually stops for review. Terminal TAMQ failure stops coordination.

A normal checkpoint ends its lease and creates a new checkpoint trigger/lease,
linked through `source_id` to the parent trigger. This continuation is a new wake,
not a transport retry. The worker reads its parent's local checkpoint before
resuming. Checkpoints allow summary, files_changed, next_action and blocked_reason
within 8 KiB; contents stay local. Any blocked_reason stops automatic continuation.

TAMQ injected/acknowledged means transport delivery, never task completion.
Completion requires a worker completion operation or authoritative task done.
Source removal or other superseding changes stop stale leases. A safe status
revision does not revoke an unexpired active worker merely to issue another wake.

## TAMQ boundary

Coordination-engine connects to the configured TAMQ Unix socket as the same
approved local user. It checks socket ownership/mode 0600 and Linux SO_PEERCRED,
negotiates protocol 0.1 through ping, and requires `bounded_delivery_ack_v1` and
`idempotent_send_v1`. Each operation opens a fresh authenticated connection.

Exactly one live endpoint must advertise the selected gita repository. Multiple
matches require configured endpoint_id; zero is unavailable. TAMQ validates its
registered repository paths and owns endpoint lifecycle. Coordination-engine
checks any advertised path and never controls tmux directly.

The send envelope uses client_id/sender_repo `coordination-engine`, lease ID as
idempotency_key, trigger ID as correlation_id, exact target/endpoint, provenance
`coordination_engine`, structured lease/trigger metadata and a standard plain-text
prompt. TAMQ owns its 8 KiB body limit, queue, delivery acknowledgement profile,
local leases, retry timing and durable admission. The coordination runtime never
resets a failed TAMQ message automatically. Receipt recovery uses `message` after
restart; endpoint ID and message ID are preserved as the correlated pair.

The original inbound attach/handshake proposal is superseded: no second TAMQ
registration socket is introduced. The separate coordination control socket
serves ping, status, stop and structured worker ack/renew/checkpoint/complete
operations over newline-delimited JSON with an `op` field. It accepts same-user
peers only, enforces a 16 KiB request limit, validates selected/gita repository
claims, and returns JSON `ok` responses. It is a same-user trust boundary.

## Storage, lifecycle and configuration

Python 3.11+, standard-library runtime dependencies, root pyproject.toml, uv
install/editable support. See `docs/worker-runtime.md` for the exact TOML schema,
XDG paths, overrides, CLI, worker examples, backups and restore procedure.

Operator-approved operational defaults: 15-second polling, 30-second leases, 10-second
renewal, four attempts, 5/15/60/300-second delays, five-second adapter/busy timeout,
private mode-0700 state and mode-0600 sockets/databases/backups. Explicit repo
selection is mandatory; package installation never enables a service.

SQLite uses WAL and foreign keys. Schema upgrades are transactional, snapshot
existing tables before migrations, and reject newer versions. History is never
automatically removed. Backup uses SQLite's consistent backup API. An exclusive
writer lock prevents concurrent service/once dispatch. SIGINT/SIGTERM stops new
work, preserves durable leases and removes the control socket. The synchronous
local-alpha polling implementation is intended for small local worker sets;
control calls can wait for an observation pass. See the runbook's timing caveat.

Hub outages suspend new wakes and back polling off to 300 seconds. TAMQ local
messaging remains independent. Receipt projection retries from a local outbox;
projection is at least once, with a stable receipt ID in `detail.id`. Edge-relay
queued responses remain pending evidence. Credentials are not configuration and
must follow the approved credential-routing subsystem if access is unavailable.

## Diagnostics and verification

Default errors omit arbitrary remote exception text. Audit receipts contain only
IDs, state/reason codes, timestamps, retry metadata and policy/configuration
versions. No source bodies, checkpoint text, credentials or terminal output are
projected. Local status/history may expose checkpoints to the approved user.

TAMQ owns unsafe `--orwell` diagnostics. Coordination-engine has no unsafe sink.
`docs/orwell-logging-diagnostics-candidate.md` records the candidate practice;
canonical review/registration remains with info-tech-canon, tracked separately
in COORDINATION-WP-0004 with operator approval.

Default checks: `uv run pytest`, `git diff --check`, compileall, CLI help/version.
Tests exercise SQLite durability and locking, fake State Hub and gita, actual
local sockets with fake TAMQ peers, CLI behavior, service stop/restart, safety,
dependencies and checkpoint continuation. Package builds and explicitly enabled
read-only live adapter discovery are separate checks. No credential or live
worker mutation is required by the test suite.