coordination-engine/spec/worker-coordination-service-v0.1.md

Worker Coordination Service v0.1

Status: Proposed contract Owner: coordination-engine (contract and runtime); State Hub and tmux-amq are external local adapters. Request: net-kingdom, “automatic worker wake-up and dependency coordination”

1. Problem and outcome

Several repository terminals run independent worker agents. Today an operator must repeatedly inspect each repository, relay results, and issue prompts that are effectively a scheduler. The service defined here turns State Hub changes into safe, deduplicated wake-ups while keeping repository agents responsible for publishing workplan/task/message state.

The desired loop is:

State Hub change
  -> observe and normalize
  -> determine actionable task/message
  -> acquire idempotent lease
  -> wake owning repo session
  -> agent resumes, checkpoints, and updates State Hub
  -> dependent work becomes actionable and is evaluated again

The service does not execute repository code, invent task state, or replace State Hub as the source of truth. It does own the observation, lease, policy, checkpoint, and wake orchestration runtime.

2. Inputs and normalized trigger

The observer MUST support polling initially and MAY support a State Hub event subscription later. It watches:

• workplan status and task status/dependencies;
• unread coordination messages and message read/ack state;
• needs_human and intervention notes;
• progress/checkpoint receipts.

Each source change is normalized to a trigger with an idempotency key:

WorkerTrigger:
  id: string
  source: workplan | task | message | checkpoint
  source_id: string
  source_revision: string
  target_repo: string
  target_agent: string?
  reason: task_actionable | dependency_satisfied | inbox_message | retry_due | checkpoint_resume
  observed_at: timestamp

The idempotency key is (source, source_id, source_revision, target_repo). Repeated observations MUST collapse to one pending trigger.

3. Actionability and safety gates

A trigger is actionable only when its task/workplan is active or ready, its dependencies are satisfied (or the message explicitly requests action), and no terminal safety gate applies. The service MUST stop and leave a durable receipt when any of these are true:

• needs_human is set or an intervention note requests review;
• credentials, secrets, access tokens, or private key material are needed;
• destructive, production, live, financial, or externally visible action is proposed without an approved policy/authority;
• a policy decision, scope expansion, or ambiguous ownership requires a human;
• the target session cannot be authenticated or its repository identity does not match the trigger.

Safety-gated work is never silently retried or rerouted to another agent. Retry behavior is configuration-driven: ordinary triggers use bounded exponential backoff, while safety-gated retries default to zero and may only be enabled by an explicit policy. No retry setting may bypass the safety gate.

4. Lease and deduplication protocol

When connected, coordination-engine atomically claims a short-lived coordination lease before waking an agent. Independently, tmux-amq maintains its own local queue lease so messages remain deliverable while coordination-engine is offline. The two leases are correlated but neither is the other's storage dependency:

Coordination-engine stores coordination leases, deduplication keys, and receipts in a local SQLite database. State Hub receives sanitized projections when it is available; it is not required for local queue delivery or lease recovery.

WorkerLease:
  trigger_id: string
  lease_id: string
  owner: coordination-engine
  target_repo: string
  target_agent: string?
  state: offered | acknowledged | running | checkpointed | completed | retry_wait | stopped
  attempt: integer
  acquired_at: timestamp
  expires_at: timestamp
  last_checkpoint_at: timestamp?
  coordination_endpoint_id: string?
  local_lease_id: string?

Only one non-expired coordination lease may exist for a trigger. A wake request is retried with the same lease_id; the receiver MUST treat duplicate requests as no-ops. The agent acknowledges the lease before work begins, renews it while active, and releases it with a checkpoint or terminal result. Expiry moves the lease to retry_wait with bounded exponential backoff and an attempt limit. Exhaustion produces a stopped receipt for operator review.

5. Session wake contract

The runtime integrates with tmux-amq, a separate local repository that owns tmux session lifecycle, durable queueing, delivery leases, acknowledgement state, and PTY interaction. tmux-amq remains functional when coordination-engine is unavailable. The implemented transport contract is tmux-amq/spec/coordination-engine-adapter-v0.1.md; this section summarizes its coordination-engine-facing obligations.

Only repositories registered in gita may be addressed. A session may be started with, for example, tamq --command codex --mode trigger net-kingdom railiance-platform; TAMQ opens one terminal per repository and runs the command only because the operator selected it explicitly. Operator or worker messages use To:repo: text and arrive in non-routable From:sender[/o]: text envelopes. Coordination-engine calls TAMQ through the same-user Unix-socket adapter and never manipulates tmux directly.

Each tmux-amq endpoint is identified as tmux-amq-<PID>, where <PID> is the tmux server PID. Within one endpoint, a repository slug maps to exactly one tmux window; duplicate windows for the same repo are rejected or reconciled before attach. Different endpoint PIDs may independently host the same repo.

Outbound messages are observed by TAMQ's explicit PTY broker. A worker To:<registered-repo>: line begins a block that closes on an empty line, a new address, or worker exit; operator input is one addressed line. TAMQ preserves ordinary terminal bytes and records distinct worker/operator provenance.

If observation is unavailable, TAMQ leaves ordinary terminal use unchanged and does not fall back to pane scraping. Operators may explicitly use tamq send 'To:repo: message' as a degraded path.

tmux-amq MUST retain a local message history sufficient to inspect delivery state and export it to a messages file. It MUST also support replaying a messages file into a selected endpoint. Replay creates a new batch identifier, passes each entry through normal gita-target validation, policy checks, and deduplication, and records the original message ID as provenance. Replay MUST not bypass safety gates or silently overwrite an existing message.

History files use newline-delimited JSON (JSONL): one complete message envelope per line, including message ID, sender/target, body, timestamps, state, and replay provenance. JSONL exports are appendable and replayable as streams.

The repository is tmux-amq, but its executable is tamq (Tmux Agentic Message Queueing). It MUST provide conventional Unix CLI behavior, including --help/-h, --version/-V, non-zero exit status on errors, diagnostics on stderr, and machine-readable output options where commands produce records.

Default filesystem locations follow XDG conventions:

config:  ${XDG_CONFIG_HOME:-$HOME/.config}/tamq/config.toml
state:   ${XDG_STATE_HOME:-$HOME/.local/state}/tamq/tamq.sqlite3
socket:  ${XDG_RUNTIME_DIR:-/tmp}/tamq.sock

TAMQ_CONFIG, TAMQ_STATE_DIR, TAMQ_SOCKET, and TAMQ_POLICY_PROFILE may override these defaults. The CLI MUST provide tab completion through tamq completion bash|zsh|fish and expose completion for subcommands, options, and known gita repository targets.

Version 0.1 supports direct addressing only. Unknown or unregistered targets are rejected; there is no broadcast or wildcard routing. Coordination-engine resolves exactly one currently registered TAMQ endpoint for the target; zero is unavailable and multiple matches require an explicit endpoint ID. It does not implicitly create a session or choose a pane occupant. The sender label is derived from the structured request or authenticated source window, not message text.

Delivery acknowledgment is configurable and defaults to injected, meaning the message is considered transport-delivered once TAMQ successfully writes it to the target terminal. acknowledged enters awaiting_ack and redelivers the same message ID after its deadline. Attempts, write failures, lease expiry, backoff, and terminal failed state are durable and bounded by the selected profile. A late acknowledgement wins; terminal retry is explicit. Duplicate terminal presentation is possible and neither delivery mode proves agent comprehension or task completion.

Coordination-engine uses its coordination lease ID as TAMQ's idempotency key and the trigger revision as correlation ID. Repeating an identical wake returns the original TAMQ message ID; reusing the lease ID for a different request fails. The client opens a new authenticated socket connection per operation, so receipt recovery after broker restart uses message or history rather than a connection session. If the bound endpoint disappears, repeating the identical wake may rebind the same durable message ID to its one replacement endpoint; TAMQ refuses implicit movement while the old endpoint remains live.

Message bodies are human-readable UTF-8 plain text, limited to 8 KiB and preserving multiline content. The transport envelope carries structured IDs, sender/target slugs, timestamps, and policy metadata. Binary payloads and oversized bodies are rejected. Message text is never interpreted as task or workplan state.

On attach, tmux-amq connects to a coordination-engine Unix domain socket and performs a handshake, receiving a unique coordination_endpoint_id. That identifier namespaces all messages and leases originating from the connected endpoint, so local IDs can safely be reused across different machines or sessions without cross-endpoint collision. Coordination-engine MUST treat (coordination_endpoint_id, local_message_id) as the message identity and preserve the endpoint ID in receipts.

The socket protocol MUST use structured messages and authenticate the peer by filesystem ownership/permissions. The default deployment runs both processes under the same approved local user, uses socket mode 0600, and validates Unix peer credentials; bearer tokens in command lines are not required. The initial handshake includes the gita registered repository slugs, tmux session name, and agent command; the response contains the endpoint ID and negotiated lease/receipt capabilities. No TCP listener is required for the local integration.

coordination-engine MUST refresh the local gita registry before accepting an attach or addressing a target. It accepts only exact registered repository slugs, requires the attached session to report the matching slug/path, and never auto-clones or auto-registers a repository.

The adapter interface is:

WakeRequest:
  lease_id: string
  target_repo: string  # must be gita-registered and an active tmux-amq target
  target_agent: string?
  trigger: WorkerTrigger
  checkpoint: Checkpoint?
  prompt: string

WakeAck:
  lease_id: string
  session_id: string
  accepted_at: timestamp
  capability: resumed | started | unavailable

The prompt is generated from the trigger and the standard continuation policy; it MUST contain no secret values. The target agent remains the authority for editing files, calling tools, and updating State Hub.

6. Checkpoints and standard continuation policy

Agents SHOULD publish a checkpoint before yielding or stopping:

Checkpoint:
  lease_id: string
  task_id: string?
  summary: string
  files_changed: [string]
  next_action: string?
  blocked_reason: human | secret | destructive | policy | dependency | unavailable | null
  recorded_at: timestamp

When a lease is resumed, the service supplies the latest checkpoint and asks the agent to continue. The following policy templates are recognized, in order:

1. inspect open work and implement the highest-priority actionable item;
2. if no work is open, reconcile SCOPE.md with INTENT.md, record gaps under history/, and create/register a workplan for the most relevant gap;
3. review pending work, refine it where needed, and implement the next safe item;
4. if no safe work remains, close the session and write a perspective entry to the configured hall-of-helix destination.

Templates are suggestions, not permission to bypass safety gates or invent scope. A repository may override them through its agent instructions.

6.1 Policy profiles

Automatic behavior is selected through named policy profiles rather than code changes. The default profile is conservative:

policy:
  profile: default
  profiles:
    default:
      allow: [statehub_read, repo_inspect, repo_edit, local_checks,
              state_updates, coordination_messages, tmux_wake, transient_retry]
      require_human: [secrets, destructive, live_external, scope_change,
                      ambiguous_ownership, external_publish, financial, legal]
      safety_gated_max_attempts: 0

Profiles MUST be selected by configuration, validated at startup, and included by name/version in every lease and receipt. A profile cannot grant access to credentials that are unavailable through the configured credential-routing system, and every profile remains subject to authentication and repository boundary checks.

7. Receipts and observability

Every trigger, lease transition, wake attempt, checkpoint, stop, retry, and completion MUST emit a sanitized progress/audit receipt containing identifiers, status, reason, timestamps, and retry metadata. Receipts MUST exclude prompt secrets, credentials, payload contents, and unrestricted terminal output.

The service SHOULD publish receipts through State Hub progress/events and retain enough correlation data to explain why a dependent task was (or was not) woken.

8. Ownership boundary

Repositories publish workplan/task/message state and consume wake requests. They MUST NOT add independent polling daemons for this feature. Coordination-engine owns the service process, observation, lease/deduplication, retry policy, and wake dispatch. State Hub owns durable task/message state. Terminal/session control is a pluggable adapter owned by coordination-engine and deployed with the runtime; its underlying bridge remains an external dependency.

9. Operator decisions required before implementation

The following choices cannot be inferred safely from repository documents:

1. Which terminal/session bridge is authorized to wake or resume workers?
2. What local SQLite retention and backup policy should be used for leases and receipts?
3. Default polling interval, lease duration, renewal interval, and retry limit?
4. Which actions are pre-approved policies versus always human-gated?
5. What service identity and authentication path may write State Hub and invoke the wake adapter?
6. Which Unix socket path and filesystem ownership should the local deployment use?

9.1 Runtime configuration

The following values MUST be configurable without code changes:

coordination:
  statehub_poll_interval_seconds: 15
  wake_ack_timeout_seconds: 30
  lease_renew_interval_seconds: 10
  retry_backoff_seconds: [5, 15, 60, 300]
  max_attempts: 4
  safety_gated_max_attempts: 0
  delivery_ack_mode: injected # injected | acknowledged

Deployments may choose different values, including up to nine attempts. The service MUST validate max_attempts and safety_gated_max_attempts in the range 0..9 and record the effective configuration version in receipts.

State Hub observation uses the configured polling interval (15 seconds by default), backs off exponentially while unavailable, and resumes promptly when connectivity returns. Local tmux-amq messaging is never blocked by State Hub outages; receipt projections are queued for asynchronous publication.

The State Hub endpoint is configurable through STATEHUB_API_BASE or statehub.api_base, defaulting to http://127.0.0.1:8000. Credentials MUST be resolved through the approved credential-routing mechanism and never stored in tamq configuration, SQLite, JSONL exports, tmux commands, or receipts. Missing credentials leave local messaging operational while State Hub projections stay pending.

tamq status exposes live endpoint/session, queue, lease, and State Hub connectivity information. tamq history exposes full local delivery history; State Hub receives sanitized receipts and correlation IDs only. Message bodies remain local unless explicitly exported to JSONL.

History is never deleted automatically. When invoked without explicit filters, tamq purge uses dynamic defaults equivalent to --before 365d and --max-size 100MB; purge requires confirmation and may be preceded by JSONL export. On startup, if history exceeds 100 MB, tamq prints an advisory such as: Size of history is NNN MB since YYMMDD, exceeding 100MB. Consider using 'tamq purge'.

tamq purge performs a dry-run summary by default, reporting the matching message count, date range, and estimated bytes. Interactive execution requires explicit confirmation; non-interactive execution requires --yes. Combined filters are allowed, and each purge emits a receipt containing filters and counts without message bodies.

tamq start creates or reuses the tmux server/windows and starts the local service; tamq serve runs the Unix-socket service in the foreground; tamq stop stops the service without killing worker terminals or deleting queues. The initial repo-opening command MUST ensure the service is alive and start it automatically when absent. Startup is idempotent and lock-protected so two concurrent repo opens cannot create duplicate services. --detach is required to daemonize; foreground operation is the default for diagnostics.

If automatic startup fails, tamq reports the error on stderr, returns non-zero, does not register an endpoint or claim delivery, and leaves existing sessions and SQLite data untouched. --no-service may explicitly open terminals without messaging, with a visible warning.

10. Logging and diagnostics

Foreground tamq serve logs to stderr. Detached operation logs to ${XDG_STATE_HOME:-$HOME/.local/state}/tamq/tamq.log. The default level is info; --quiet suppresses non-errors, --verbose enables debug diagnostics, and --json requests structured output where applicable. Message bodies, credentials, tokens, and unrestricted PTY output are omitted by default.

--orwell is an explicitly unsafe, non-production diagnostic mode that logs otherwise-omitted diagnostic fields, including message bodies and credential material when available. It MUST be supplied per invocation, emit a prominent warning, write only to a local mode-0600 sink, never project to State Hub, and be rejected by the default production policy profile. Operators are responsible for purging Orwell logs after diagnosis.

This practice is proposed for canonization as the InfoTechCanon PracticePattern practice-pattern/orwell-logging-diagnostics (OrwellLoggingDiagnostics). The canonical owner is info-tech-canon; tamq is a concrete known use, not the owner of the general practice.

The first runtime implementation uses Python 3.11+ and should prefer standard library components (sqlite3, asyncio, Unix sockets, and CLI primitives) so local installation remains small and reliable.

The project is packaged through a root pyproject.toml with a tamq console entry point. It must support uv tool install ., editable development installs, and package-derived tamq --version output.

Verification uses layered tests: pytest unit tests for state/config/protocol logic; Unix-socket integration tests with fake tmux-amq peers; fake State Hub and gita registry tests; CLI behavior tests; and opt-in live smoke tests only when explicitly configured. Tests must not require credentials or mutate live systems.

The Unix-socket protocol is versioned independently from package versions, starting at 0.1. Attach handshakes negotiate protocol version and capabilities; incompatible major versions are rejected, compatible minor versions may interoperate, and the negotiated values are retained in endpoint/receipt data.

SQLite stores an explicit schema version. Startup applies forward migrations in transactions, creates a timestamped backup before altering message/lease tables, and refuses to run against a newer schema. Message history is never silently dropped; tamq db-version reports the current schema.

SQLite uses WAL mode, foreign-key enforcement, and a configurable busy timeout (5 seconds by default). Queue and lease transitions are transactional, with a single coordination-engine writer path and concurrent readers. Transient busy errors may retry without creating duplicate messages.

On SIGTERM or SIGINT, the service stops accepting socket requests and new leases, finishes the current transaction, marks the endpoint disconnected, closes the socket, preserves queues/leases for recovery, emits a sanitized shutdown receipt, and exits within a configurable grace period (10 seconds by default).

Default CI gates are uv run pytest, git diff --check, python -m compileall src, tamq --help, and tamq --version; package install/build and live smoke checks are separate opt-in jobs.

Health is local-only: tamq status is human-readable, tamq status --json is machine-readable, and tamq ping performs a lightweight Unix-socket liveness check. Status separates process/socket, SQLite, gita registry, State Hub, queue, and lease health; no TCP health port is exposed.