coordination-engine/spec/worker-coordination-service-v0.1.md

# Worker Coordination Service v0.1

**Status:** Proposed contract
**Owner:** coordination-engine (contract and runtime); State Hub and
`tmux-amq` are external local adapters.
**Request:** net-kingdom, “automatic worker wake-up and dependency coordination”

## 1. Problem and outcome

Several repository terminals run independent worker agents. Today an operator
must repeatedly inspect each repository, relay results, and issue prompts that
are effectively a scheduler. The service defined here turns State Hub changes
into safe, deduplicated wake-ups while keeping repository agents responsible for
publishing workplan/task/message state.

The desired loop is:

```text
State Hub change
  -> observe and normalize
  -> determine actionable task/message
  -> acquire idempotent lease
  -> wake owning repo session
  -> agent resumes, checkpoints, and updates State Hub
  -> dependent work becomes actionable and is evaluated again
```

The service does not execute repository code, invent task state, or replace
State Hub as the source of truth. It does own the observation, lease, policy,
checkpoint, and wake orchestration runtime.

## 2. Inputs and normalized trigger

The observer MUST support polling initially and MAY support a State Hub event
subscription later. It watches:

* workplan status and task status/dependencies;
* unread coordination messages and message read/ack state;
* `needs_human` and intervention notes;
* progress/checkpoint receipts.

Each source change is normalized to a trigger with an idempotency key:

```yaml
WorkerTrigger:
  id: string
  source: workplan | task | message | checkpoint
  source_id: string
  source_revision: string
  target_repo: string
  target_agent: string?
  reason: task_actionable | dependency_satisfied | inbox_message | retry_due | checkpoint_resume
  observed_at: timestamp
```

The idempotency key is `(source, source_id, source_revision, target_repo)`.
Repeated observations MUST collapse to one pending trigger.

## 3. Actionability and safety gates

A trigger is actionable only when its task/workplan is active or ready, its
dependencies are satisfied (or the message explicitly requests action), and no
terminal safety gate applies. The service MUST stop and leave a durable receipt
when any of these are true:

* `needs_human` is set or an intervention note requests review;
* credentials, secrets, access tokens, or private key material are needed;
* destructive, production, live, financial, or externally visible action is
  proposed without an approved policy/authority;
* a policy decision, scope expansion, or ambiguous ownership requires a human;
* the target session cannot be authenticated or its repository identity does not
  match the trigger.

Safety-gated work is never silently retried or rerouted to another agent.
Retry behavior is configuration-driven: ordinary triggers use bounded
exponential backoff, while safety-gated retries default to zero and may only be
enabled by an explicit policy. No retry setting may bypass the safety gate.

## 4. Lease and deduplication protocol

When connected, coordination-engine atomically claims a short-lived coordination
lease before waking an agent. Independently, `tmux-amq` maintains its own local
queue lease so messages remain deliverable while coordination-engine is offline.
The two leases are correlated but neither is the other's storage dependency:

Coordination-engine stores coordination leases, deduplication keys, and receipts
in a local SQLite database. State Hub receives sanitized projections when it is
available; it is not required for local queue delivery or lease recovery.

```yaml
WorkerLease:
  trigger_id: string
  lease_id: string
  owner: coordination-engine
  target_repo: string
  target_agent: string?
  state: offered | acknowledged | running | checkpointed | completed | retry_wait | stopped
  attempt: integer
  acquired_at: timestamp
  expires_at: timestamp
  last_checkpoint_at: timestamp?
  coordination_endpoint_id: string?
  local_lease_id: string?
```

Only one non-expired coordination lease may exist for a trigger. A wake request is retried
with the same `lease_id`; the receiver MUST treat duplicate requests as no-ops.
The agent acknowledges the lease before work begins, renews it while active,
and releases it with a checkpoint or terminal result. Expiry moves the lease to
`retry_wait` with bounded exponential backoff and an attempt limit. Exhaustion
produces a stopped receipt for operator review.

## 5. Session wake contract

The runtime integrates with `tmux-amq`, a separate local repository that owns
tmux session lifecycle, durable queueing, delivery leases, acknowledgement
state, and PTY interaction. `tmux-amq` remains functional when
coordination-engine is unavailable. The implemented transport contract is
`tmux-amq/spec/coordination-engine-adapter-v0.1.md`; this section summarizes its
coordination-engine-facing obligations.

Only repositories registered in gita may be addressed. A session may be started
with, for example, `tamq --command codex --mode trigger net-kingdom
railiance-platform`; TAMQ opens one terminal per repository and runs the command
only because the operator selected it explicitly. Operator or worker messages
use `To:repo: text` and arrive in non-routable `From:sender[/o]: text`
envelopes. Coordination-engine calls TAMQ through the same-user Unix-socket
adapter and never manipulates tmux directly.

Each tmux-amq endpoint is identified as `tmux-amq-<PID>`, where `<PID>` is the
tmux server PID. Within one endpoint, a repository slug maps to exactly one
tmux window; duplicate windows for the same repo are rejected or reconciled
before attach. Different endpoint PIDs may independently host the same repo.

Outbound messages are observed by TAMQ's explicit PTY broker. A worker
`To:<registered-repo>:` line begins a block that closes on an empty line, a new
address, or worker exit; operator input is one addressed line. TAMQ preserves
ordinary terminal bytes and records distinct worker/operator provenance.

If observation is unavailable, TAMQ leaves ordinary terminal use unchanged and
does not fall back to pane scraping. Operators may explicitly use
`tamq send 'To:repo: message'` as a degraded path.

tmux-amq MUST retain a local message history sufficient to inspect delivery
state and export it to a messages file. It MUST also support replaying a
messages file into a selected endpoint. Replay creates a new batch identifier,
passes each entry through normal gita-target validation, policy checks, and
deduplication, and records the original message ID as provenance. Replay MUST
not bypass safety gates or silently overwrite an existing message.

History files use newline-delimited JSON (JSONL): one complete message envelope
per line, including message ID, sender/target, body, timestamps, state, and
replay provenance. JSONL exports are appendable and replayable as streams.

The repository is `tmux-amq`, but its executable is `tamq` (Tmux Agentic Message
Queueing). It MUST provide conventional Unix CLI behavior, including
`--help`/`-h`, `--version`/`-V`, non-zero exit status on errors, diagnostics on
stderr, and machine-readable output options where commands produce records.

Default filesystem locations follow XDG conventions:

```text
config:  ${XDG_CONFIG_HOME:-$HOME/.config}/tamq/config.toml
state:   ${XDG_STATE_HOME:-$HOME/.local/state}/tamq/tamq.sqlite3
socket:  ${XDG_RUNTIME_DIR:-/tmp}/tamq.sock
```

`TAMQ_CONFIG`, `TAMQ_STATE_DIR`, `TAMQ_SOCKET`, and `TAMQ_POLICY_PROFILE` may
override these defaults. The CLI MUST provide tab completion through
`tamq completion bash|zsh|fish` and expose completion for subcommands,
options, and known gita repository targets.

Version 0.1 supports direct addressing only. Unknown or unregistered targets
are rejected; there is no broadcast or wildcard routing. Coordination-engine
resolves exactly one currently registered TAMQ endpoint for the target; zero is
unavailable and multiple matches require an explicit endpoint ID. It does not
implicitly create a session or choose a pane occupant. The sender label is
derived from the structured request or authenticated source window, not message
text.

Delivery acknowledgment is configurable and defaults to `injected`, meaning the
message is considered transport-delivered once TAMQ successfully writes it to
the target terminal. `acknowledged` enters `awaiting_ack` and redelivers the same
message ID after its deadline. Attempts, write failures, lease expiry, backoff,
and terminal `failed` state are durable and bounded by the selected profile. A
late acknowledgement wins; terminal retry is explicit. Duplicate terminal
presentation is possible and neither delivery mode proves agent comprehension
or task completion.

Coordination-engine uses its coordination lease ID as TAMQ's idempotency key and
the trigger revision as correlation ID. Repeating an identical wake returns the
original TAMQ message ID; reusing the lease ID for a different request fails.
The client opens a new authenticated socket connection per operation, so
receipt recovery after broker restart uses `message` or `history` rather than a
connection session. If the bound endpoint disappears, repeating the identical
wake may rebind the same durable message ID to its one replacement endpoint;
TAMQ refuses implicit movement while the old endpoint remains live.

Message bodies are human-readable UTF-8 plain text, limited to 8 KiB and
preserving multiline content. The transport envelope carries structured IDs,
sender/target slugs, timestamps, and policy metadata. Binary payloads and
oversized bodies are rejected. Message text is never interpreted as task or
workplan state.

On attach, `tmux-amq` connects to a coordination-engine Unix domain socket and
performs a handshake, receiving a unique
`coordination_endpoint_id`. That identifier namespaces all messages and leases
originating from the connected endpoint, so local IDs can safely be reused
across different machines or sessions without cross-endpoint collision.
Coordination-engine MUST treat `(coordination_endpoint_id, local_message_id)`
as the message identity and preserve the endpoint ID in receipts.

The socket protocol MUST use structured messages and authenticate the peer by
filesystem ownership/permissions. The default deployment runs both processes
under the same approved local user, uses socket mode `0600`, and validates Unix
peer credentials; bearer tokens in command lines are not required. The initial
handshake includes the gita
registered repository slugs, tmux session name, and agent command; the response
contains the endpoint ID and negotiated lease/receipt capabilities. No TCP
listener is required for the local integration.

coordination-engine MUST refresh the local gita registry before accepting an
attach or addressing a target. It accepts only exact registered repository
slugs, requires the attached session to report the matching slug/path, and
never auto-clones or auto-registers a repository.

The adapter interface is:

```yaml
WakeRequest:
  lease_id: string
  target_repo: string  # must be gita-registered and an active tmux-amq target
  target_agent: string?
  trigger: WorkerTrigger
  checkpoint: Checkpoint?
  prompt: string

WakeAck:
  lease_id: string
  session_id: string
  accepted_at: timestamp
  capability: resumed | started | unavailable
```

The prompt is generated from the trigger and the standard continuation policy;
it MUST contain no secret values. The target agent remains the authority for
editing files, calling tools, and updating State Hub.

## 6. Checkpoints and standard continuation policy

Agents SHOULD publish a checkpoint before yielding or stopping:

```yaml
Checkpoint:
  lease_id: string
  task_id: string?
  summary: string
  files_changed: [string]
  next_action: string?
  blocked_reason: human | secret | destructive | policy | dependency | unavailable | null
  recorded_at: timestamp
```

When a lease is resumed, the service supplies the latest checkpoint and asks
the agent to continue. The following policy templates are recognized, in order:

1. inspect open work and implement the highest-priority actionable item;
2. if no work is open, reconcile `SCOPE.md` with `INTENT.md`, record gaps under
   `history/`, and create/register a workplan for the most relevant gap;
3. review pending work, refine it where needed, and implement the next safe item;
4. if no safe work remains, close the session and write a perspective entry to
   the configured `hall-of-helix` destination.

Templates are suggestions, not permission to bypass safety gates or invent
scope. A repository may override them through its agent instructions.

### 6.1 Policy profiles

Automatic behavior is selected through named policy profiles rather than code
changes. The default profile is conservative:

```yaml
policy:
  profile: default
  profiles:
    default:
      allow: [statehub_read, repo_inspect, repo_edit, local_checks,
              state_updates, coordination_messages, tmux_wake, transient_retry]
      require_human: [secrets, destructive, live_external, scope_change,
                      ambiguous_ownership, external_publish, financial, legal]
      safety_gated_max_attempts: 0
```

Profiles MUST be selected by configuration, validated at startup, and included
by name/version in every lease and receipt. A profile cannot grant access to
credentials that are unavailable through the configured credential-routing
system, and every profile remains subject to authentication and repository
boundary checks.

## 7. Receipts and observability

Every trigger, lease transition, wake attempt, checkpoint, stop, retry, and
completion MUST emit a sanitized progress/audit receipt containing identifiers,
status, reason, timestamps, and retry metadata. Receipts MUST exclude prompt
secrets, credentials, payload contents, and unrestricted terminal output.

The service SHOULD publish receipts through State Hub progress/events and retain
enough correlation data to explain why a dependent task was (or was not) woken.

## 8. Ownership boundary

Repositories publish workplan/task/message state and consume wake requests. They
MUST NOT add independent polling daemons for this feature. Coordination-engine
owns the service process, observation, lease/deduplication, retry policy, and
wake dispatch. State Hub owns durable task/message state. Terminal/session
control is a pluggable adapter owned by coordination-engine and deployed with
the runtime; its underlying bridge remains an external dependency.

## 9. Operator decisions required before implementation

The following choices cannot be inferred safely from repository documents:

1. Which terminal/session bridge is authorized to wake or resume workers?
2. What local SQLite retention and backup policy should be used for leases and
   receipts?
3. Default polling interval, lease duration, renewal interval, and retry limit?
4. Which actions are pre-approved policies versus always human-gated?
5. What service identity and authentication path may write State Hub and invoke
   the wake adapter?
6. Which Unix socket path and filesystem ownership should the local deployment
   use?

### 9.1 Runtime configuration

The following values MUST be configurable without code changes:

```yaml
coordination:
  statehub_poll_interval_seconds: 15
  wake_ack_timeout_seconds: 30
  lease_renew_interval_seconds: 10
  retry_backoff_seconds: [5, 15, 60, 300]
  max_attempts: 4
  safety_gated_max_attempts: 0
  delivery_ack_mode: injected # injected | acknowledged
```

Deployments may choose different values, including up to nine attempts. The
service MUST validate `max_attempts` and `safety_gated_max_attempts` in the
range 0..9 and record the effective configuration version in receipts.

State Hub observation uses the configured polling interval (15 seconds by
default), backs off exponentially while unavailable, and resumes promptly when
connectivity returns. Local tmux-amq messaging is never blocked by State Hub
outages; receipt projections are queued for asynchronous publication.

The State Hub endpoint is configurable through `STATEHUB_API_BASE` or
`statehub.api_base`, defaulting to `http://127.0.0.1:8000`. Credentials MUST be
resolved through the approved credential-routing mechanism and never stored in
tamq configuration, SQLite, JSONL exports, tmux commands, or receipts. Missing
credentials leave local messaging operational while State Hub projections stay
pending.

`tamq status` exposes live endpoint/session, queue, lease, and State Hub
connectivity information. `tamq history` exposes full local delivery history;
State Hub receives sanitized receipts and correlation IDs only. Message bodies
remain local unless explicitly exported to JSONL.

History is never deleted automatically. When invoked without explicit filters,
`tamq purge` uses dynamic defaults equivalent to `--before 365d` and
`--max-size 100MB`; purge requires confirmation and may be preceded by JSONL
export. On startup, if history exceeds 100 MB, tamq prints an advisory such as:
`Size of history is NNN MB since YYMMDD, exceeding 100MB. Consider using 'tamq purge'.`

`tamq purge` performs a dry-run summary by default, reporting the matching
message count, date range, and estimated bytes. Interactive execution requires
explicit confirmation; non-interactive execution requires `--yes`. Combined
filters are allowed, and each purge emits a receipt containing filters and
counts without message bodies.

`tamq start` creates or reuses the tmux server/windows and starts the local
service; `tamq serve` runs the Unix-socket service in the foreground; `tamq stop`
stops the service without killing worker terminals or deleting queues. The
initial repo-opening command MUST ensure the service is alive and start it
automatically when absent. Startup is idempotent and lock-protected so two
concurrent repo opens cannot create duplicate services. `--detach` is required
to daemonize; foreground operation is the default for diagnostics.

If automatic startup fails, tamq reports the error on stderr, returns non-zero,
does not register an endpoint or claim delivery, and leaves existing sessions
and SQLite data untouched. `--no-service` may explicitly open terminals without
messaging, with a visible warning.

## 10. Logging and diagnostics

Foreground `tamq serve` logs to stderr. Detached operation logs to
`${XDG_STATE_HOME:-$HOME/.local/state}/tamq/tamq.log`. The default level is
`info`; `--quiet` suppresses non-errors, `--verbose` enables debug diagnostics,
and `--json` requests structured output where applicable. Message bodies,
credentials, tokens, and unrestricted PTY output are omitted by default.

`--orwell` is an explicitly unsafe, non-production diagnostic mode that logs
otherwise-omitted diagnostic fields, including message bodies and credential
material when available. It MUST be supplied per invocation, emit a prominent
warning, write only to a local mode-`0600` sink, never project to State Hub, and
be rejected by the default production policy profile. Operators are responsible
for purging Orwell logs after diagnosis.

This practice is proposed for canonization as the InfoTechCanon PracticePattern
`practice-pattern/orwell-logging-diagnostics` (`OrwellLoggingDiagnostics`). The
canonical owner is info-tech-canon; tamq is a concrete known use, not the owner
of the general practice.

The first runtime implementation uses Python 3.11+ and should prefer standard
library components (`sqlite3`, `asyncio`, Unix sockets, and CLI primitives) so
local installation remains small and reliable.

The project is packaged through a root `pyproject.toml` with a `tamq` console
entry point. It must support `uv tool install .`, editable development installs,
and package-derived `tamq --version` output.

Verification uses layered tests: pytest unit tests for state/config/protocol
logic; Unix-socket integration tests with fake tmux-amq peers; fake State Hub
and gita registry tests; CLI behavior tests; and opt-in live smoke tests only
when explicitly configured. Tests must not require credentials or mutate live
systems.

The Unix-socket protocol is versioned independently from package versions,
starting at `0.1`. Attach handshakes negotiate protocol version and capabilities;
incompatible major versions are rejected, compatible minor versions may
interoperate, and the negotiated values are retained in endpoint/receipt data.

SQLite stores an explicit schema version. Startup applies forward migrations in
transactions, creates a timestamped backup before altering message/lease
tables, and refuses to run against a newer schema. Message history is never
silently dropped; `tamq db-version` reports the current schema.

SQLite uses WAL mode, foreign-key enforcement, and a configurable busy timeout
(5 seconds by default). Queue and lease transitions are transactional, with a
single coordination-engine writer path and concurrent readers. Transient busy
errors may retry without creating duplicate messages.

On `SIGTERM` or `SIGINT`, the service stops accepting socket requests and new
leases, finishes the current transaction, marks the endpoint disconnected,
closes the socket, preserves queues/leases for recovery, emits a sanitized
shutdown receipt, and exits within a configurable grace period (10 seconds by
default).

Default CI gates are `uv run pytest`, `git diff --check`,
`python -m compileall src`, `tamq --help`, and `tamq --version`; package
install/build and live smoke checks are separate opt-in jobs.

Health is local-only: `tamq status` is human-readable, `tamq status --json` is
machine-readable, and `tamq ping` performs a lightweight Unix-socket liveness
check. Status separates process/socket, SQLite, gita registry, State Hub, queue,
and lease health; no TCP health port is exposed.