coordination-engine/spec/worker-coordination-service-v0.1.md

Worker Coordination Service v0.1

Status: local-alpha implementation; operational defaults approved in WP-0003-T02. Owner: coordination-engine. Transport owner: tmux-amq. State owner: repositories and State Hub's file-backed projections.

This revision reconciles the original proposal with TAMQ's implemented spec/coordination-engine-adapter-v0.1.md. The original requirements are retained in history/260907-worker-coordination-service-v0.1-proposal.md; TAMQ-specific CLI, history, replay, endpoint registration and logging requirements belong to that repository. This package installs coordination-engine, never tamq.

Observation and actionability

The runtime polls State Hub's repos, workplans, tasks, dependency edges and unread messages. It dispatches only after a complete successful snapshot, only for explicitly configured gita repositories. It refreshes gita before addressing a TAMQ target and on worker updates; it never clones or registers repositories.

Task triggers require ready/active workplans, todo/progress task status, and satisfied workplan/workplan or workplan/task dependencies. Wait tasks remain waiting until their authoritative status changes. Unknown dependency types or missing targets wait. The Hub's edge direction is from_workplan_id depends on to_workplan_id or to_task_id.

The current Hub has no action-request flag on messages. An unread, unarchived message whose subject starts with [action] requests inspection. Other messages do not wake workers. Message text never grants authority or changes workflow state. Generic free-text progress records are not interpreted as checkpoints; workers publish structured checkpoints through the local control socket.

A normalized trigger has source (task, message, checkpoint), source ID, source revision, repository and trigger ID. The ID hashes the tuple (source, source_id, source_revision, repo). Revisions hash the relevant observed record, plan and dependency state, so dependency completion is reconsidered even when the task timestamp does not change. Payload text is not retained locally as trigger data and is never copied to a wake prompt.

Policy and safety

The named default policy permits repository inspection, authorized edits, local checks, state updates and tmux wakes; configuration may narrow this set. Unknown profiles/action classes are refused. Human flags, intervention notes, blocking reasons and conservative sensitive-action text detection stop a trigger. Sensitive retries are always zero in v0.1. Text detection adds stops; it does not authorize arbitrary source text. Workers must independently recheck current repository instructions, scope, dependency state and authority.

Authentication failures, unregistered/mismatched targets, incompatible protocol and ambiguous endpoints stop the lease. Missing endpoints and ordinary transport failures use bounded backoff. No safety stop silently reroutes to another worker. A new source revision may be evaluated normally; the same stopped revision is never automatically reset. A changed runtime configuration stops pending leases created under the old configuration rather than silently changing their policy.

Leases, checkpoints and completion

SQLite stores unique trigger and coordination lease IDs, source/revision/repo, state, attempts, expiry, retry deadline, policy/configuration versions, checkpoint, and TAMQ endpoint/message/transport state. State transitions and audit receipts are committed together. One unexpired offered/acknowledged/running lease per repository prevents overlapping automatic wakes, including across task revisions.

States are pending, waiting, offered, acknowledged, running, checkpointed, retry_wait, completed and stopped. Every transition emits a sanitized receipt. Workers acknowledge before work, renew while active, and checkpoint or complete. Unknown, expired, terminal and wrong-repository lease updates are rejected.

Transport recovery repeats the same coordination lease ID and identical prompt. TAMQ deduplicates admission and returns its original message ID. Expiry/failure uses configurable bounded backoff and a 0–9 attempt ceiling. Already delivered TAMQ messages are not forcibly reinjected by coordination retry; an unresponsive worker eventually stops for review. Terminal TAMQ failure stops coordination.

A normal checkpoint ends its lease and creates a new checkpoint trigger/lease, linked through source_id to the parent trigger. This continuation is a new wake, not a transport retry. The worker reads its parent's local checkpoint before resuming. Checkpoints allow summary, files_changed, next_action and blocked_reason within 8 KiB; contents stay local. Any blocked_reason stops automatic continuation.

TAMQ injected/acknowledged means transport delivery, never task completion. Completion requires a worker completion operation or authoritative task done. Source removal or other superseding changes stop stale leases. A safe status revision does not revoke an unexpired active worker merely to issue another wake.

TAMQ boundary

Coordination-engine connects to the configured TAMQ Unix socket as the same approved local user. It checks socket ownership/mode 0600 and Linux SO_PEERCRED, negotiates protocol 0.1 through ping, and requires bounded_delivery_ack_v1 and idempotent_send_v1. Each operation opens a fresh authenticated connection.

Exactly one live endpoint must advertise the selected gita repository. Multiple matches require configured endpoint_id; zero is unavailable. TAMQ validates its registered repository paths and owns endpoint lifecycle. Coordination-engine checks any advertised path and never controls tmux directly.

The send envelope uses client_id/sender_repo coordination-engine, lease ID as idempotency_key, trigger ID as correlation_id, exact target/endpoint, provenance coordination_engine, structured lease/trigger metadata and a standard plain-text prompt. TAMQ owns its 8 KiB body limit, queue, delivery acknowledgement profile, local leases, retry timing and durable admission. The coordination runtime never resets a failed TAMQ message automatically. Receipt recovery uses message after restart; endpoint ID and message ID are preserved as the correlated pair.

The original inbound attach/handshake proposal is superseded: no second TAMQ registration socket is introduced. The separate coordination control socket serves ping, status, stop and structured worker ack/renew/checkpoint/complete operations over newline-delimited JSON with an op field. It accepts same-user peers only, enforces a 16 KiB request limit, validates selected/gita repository claims, and returns JSON ok responses. It is a same-user trust boundary.

Storage, lifecycle and configuration

Python 3.11+, standard-library runtime dependencies, root pyproject.toml, uv install/editable support. See docs/worker-runtime.md for the exact TOML schema, XDG paths, overrides, CLI, worker examples, backups and restore procedure.

Operator-approved operational defaults: 15-second polling, 30-second leases, 10-second renewal, four attempts, 5/15/60/300-second delays, five-second adapter/busy timeout, private mode-0700 state and mode-0600 sockets/databases/backups. Explicit repo selection is mandatory; package installation never enables a service.

SQLite uses WAL and foreign keys. Schema upgrades are transactional, snapshot existing tables before migrations, and reject newer versions. History is never automatically removed. Backup uses SQLite's consistent backup API. An exclusive writer lock prevents concurrent service/once dispatch. SIGINT/SIGTERM stops new work, preserves durable leases and removes the control socket. The synchronous local-alpha polling implementation is intended for small local worker sets; control calls can wait for an observation pass. See the runbook's timing caveat.

Hub outages suspend new wakes and back polling off to 300 seconds. TAMQ local messaging remains independent. Receipt projection retries from a local outbox; projection is at least once, with a stable receipt ID in detail.id. Edge-relay queued responses remain pending evidence. Credentials are not configuration and must follow the approved credential-routing subsystem if access is unavailable.

Diagnostics and verification

Default errors omit arbitrary remote exception text. Audit receipts contain only IDs, state/reason codes, timestamps, retry metadata and policy/configuration versions. No source bodies, checkpoint text, credentials or terminal output are projected. Local status/history may expose checkpoints to the approved user.

TAMQ owns unsafe --orwell diagnostics. Coordination-engine has no unsafe sink. docs/orwell-logging-diagnostics-candidate.md records the candidate practice; canonical review/registration remains with info-tech-canon, tracked separately in COORDINATION-WP-0004 with operator approval.

Default checks: uv run pytest, git diff --check, compileall, CLI help/version. Tests exercise SQLite durability and locking, fake State Hub and gita, actual local sockets with fake TAMQ peers, CLI behavior, service stop/restart, safety, dependencies and checkpoint continuation. Package builds and explicitly enabled read-only live adapter discovery are separate checks. No credential or live worker mutation is required by the test suite.