coulomb/coordination-engine
4
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Implement worker coordination runtime and finish WP-0003
All checks were successful
check / test (push) Successful in 7m8s
Details
CI Smoke / host-smoke (push) Successful in 0s
Details
CI Smoke / container-smoke (push) Successful in 1s
Details

Browse source 
Assistant: codex
Assistant-Model: gpt-6-astra
Assistant-Session: 01a07b5b-ea58-7ad2-bdbb-0b1c995cfc35
This commit is contained in:
tegwick 2026-09-07 23:19:52 +02:00
parent 214964ccb8
commit 628f984a10
23 changed files with 3025 additions and 544 deletions
Download patch file Download diff file Expand all files Collapse all files

607
spec/worker-coordination-service-v0.1.md
View file

@ -1,456 +1,155 @@
# Worker Coordination Service v0.1
**Status:** Proposed contract
**Owner:** coordination-engine (contract and runtime); State Hub and
`tmux-amq` are external local adapters.
**Request:** net-kingdom, “automatic worker wake-up and dependency coordination”
## 1. Problem and outcome
Several repository terminals run independent worker agents. Today an operator
must repeatedly inspect each repository, relay results, and issue prompts that
are effectively a scheduler. The service defined here turns State Hub changes
into safe, deduplicated wake-ups while keeping repository agents responsible for
publishing workplan/task/message state.
The desired loop is:
```text
State Hub change
-> observe and normalize
-> determine actionable task/message
-> acquire idempotent lease
-> wake owning repo session
-> agent resumes, checkpoints, and updates State Hub
-> dependent work becomes actionable and is evaluated again
```
The service does not execute repository code, invent task state, or replace
State Hub as the source of truth. It does own the observation, lease, policy,
checkpoint, and wake orchestration runtime.
## 2. Inputs and normalized trigger
The observer MUST support polling initially and MAY support a State Hub event
subscription later. It watches:
* workplan status and task status/dependencies;
* unread coordination messages and message read/ack state;
* `needs_human` and intervention notes;
* progress/checkpoint receipts.
Each source change is normalized to a trigger with an idempotency key:
```yaml
WorkerTrigger:
id: string
source: workplan | task | message | checkpoint
source_id: string
source_revision: string
target_repo: string
target_agent: string?
reason: task_actionable | dependency_satisfied | inbox_message | retry_due | checkpoint_resume
observed_at: timestamp
```
The idempotency key is `(source, source_id, source_revision, target_repo)`.
Repeated observations MUST collapse to one pending trigger.
## 3. Actionability and safety gates
A trigger is actionable only when its task/workplan is active or ready, its
dependencies are satisfied (or the message explicitly requests action), and no
terminal safety gate applies. The service MUST stop and leave a durable receipt
when any of these are true:
* `needs_human` is set or an intervention note requests review;
* credentials, secrets, access tokens, or private key material are needed;
* destructive, production, live, financial, or externally visible action is
proposed without an approved policy/authority;
* a policy decision, scope expansion, or ambiguous ownership requires a human;
* the target session cannot be authenticated or its repository identity does not
match the trigger.
Safety-gated work is never silently retried or rerouted to another agent.
Retry behavior is configuration-driven: ordinary triggers use bounded
exponential backoff, while safety-gated retries default to zero and may only be
enabled by an explicit policy. No retry setting may bypass the safety gate.
## 4. Lease and deduplication protocol
When connected, coordination-engine atomically claims a short-lived coordination
lease before waking an agent. Independently, `tmux-amq` maintains its own local
queue lease so messages remain deliverable while coordination-engine is offline.
The two leases are correlated but neither is the other's storage dependency:
Coordination-engine stores coordination leases, deduplication keys, and receipts
in a local SQLite database. State Hub receives sanitized projections when it is
available; it is not required for local queue delivery or lease recovery.
```yaml
WorkerLease:
trigger_id: string
lease_id: string
owner: coordination-engine
target_repo: string
target_agent: string?
state: offered | acknowledged | running | checkpointed | completed | retry_wait | stopped
attempt: integer
acquired_at: timestamp
expires_at: timestamp
last_checkpoint_at: timestamp?
coordination_endpoint_id: string?
local_lease_id: string?
```
Only one non-expired coordination lease may exist for a trigger. A wake request is retried
with the same `lease_id`; the receiver MUST treat duplicate requests as no-ops.
The agent acknowledges the lease before work begins, renews it while active,
and releases it with a checkpoint or terminal result. Expiry moves the lease to
`retry_wait` with bounded exponential backoff and an attempt limit. Exhaustion
produces a stopped receipt for operator review.
## 5. Session wake contract
The runtime integrates with `tmux-amq`, a separate local repository that owns
tmux session lifecycle, durable queueing, delivery leases, acknowledgement
state, and PTY interaction. `tmux-amq` remains functional when
coordination-engine is unavailable. The implemented transport contract is
`tmux-amq/spec/coordination-engine-adapter-v0.1.md`; this section summarizes its
coordination-engine-facing obligations.
Only repositories registered in gita may be addressed. A session may be started
with, for example, `tamq --command codex --mode trigger net-kingdom
railiance-platform`; TAMQ opens one terminal per repository and runs the command
only because the operator selected it explicitly. Operator or worker messages
use `To:repo: text` and arrive in non-routable `From:sender[/o]: text`
envelopes. Coordination-engine calls TAMQ through the same-user Unix-socket
adapter and never manipulates tmux directly.
Each tmux-amq endpoint is identified as `tmux-amq-<PID>`, where `<PID>` is the
tmux server PID. Within one endpoint, a repository slug maps to exactly one
tmux window; duplicate windows for the same repo are rejected or reconciled
before attach. Different endpoint PIDs may independently host the same repo.
Outbound messages are observed by TAMQ's explicit PTY broker. A worker
`To:<registered-repo>:` line begins a block that closes on an empty line, a new
address, or worker exit; operator input is one addressed line. TAMQ preserves
ordinary terminal bytes and records distinct worker/operator provenance.
If observation is unavailable, TAMQ leaves ordinary terminal use unchanged and
does not fall back to pane scraping. Operators may explicitly use
`tamq send 'To:repo: message'` as a degraded path.
tmux-amq MUST retain a local message history sufficient to inspect delivery
state and export it to a messages file. It MUST also support replaying a
messages file into a selected endpoint. Replay creates a new batch identifier,
passes each entry through normal gita-target validation, policy checks, and
deduplication, and records the original message ID as provenance. Replay MUST
not bypass safety gates or silently overwrite an existing message.
History files use newline-delimited JSON (JSONL): one complete message envelope
per line, including message ID, sender/target, body, timestamps, state, and
replay provenance. JSONL exports are appendable and replayable as streams.
The repository is `tmux-amq`, but its executable is `tamq` (Tmux Agentic Message
Queueing). It MUST provide conventional Unix CLI behavior, including
`--help`/`-h`, `--version`/`-V`, non-zero exit status on errors, diagnostics on
stderr, and machine-readable output options where commands produce records.
Default filesystem locations follow XDG conventions:
```text
config: ${XDG_CONFIG_HOME:-$HOME/.config}/tamq/config.toml
state: ${XDG_STATE_HOME:-$HOME/.local/state}/tamq/tamq.sqlite3
socket: ${XDG_RUNTIME_DIR:-/tmp}/tamq.sock
```
`TAMQ_CONFIG`, `TAMQ_STATE_DIR`, `TAMQ_SOCKET`, and `TAMQ_POLICY_PROFILE` may
override these defaults. The CLI MUST provide tab completion through
`tamq completion bash|zsh|fish` and expose completion for subcommands,
options, and known gita repository targets.
Version 0.1 supports direct addressing only. Unknown or unregistered targets
are rejected; there is no broadcast or wildcard routing. Coordination-engine
resolves exactly one currently registered TAMQ endpoint for the target; zero is
unavailable and multiple matches require an explicit endpoint ID. It does not
implicitly create a session or choose a pane occupant. The sender label is
derived from the structured request or authenticated source window, not message
text.
Delivery acknowledgment is configurable and defaults to `injected`, meaning the
message is considered transport-delivered once TAMQ successfully writes it to
the target terminal. `acknowledged` enters `awaiting_ack` and redelivers the same
message ID after its deadline. Attempts, write failures, lease expiry, backoff,
and terminal `failed` state are durable and bounded by the selected profile. A
late acknowledgement wins; terminal retry is explicit. Duplicate terminal
presentation is possible and neither delivery mode proves agent comprehension
or task completion.
Coordination-engine uses its coordination lease ID as TAMQ's idempotency key and
the trigger revision as correlation ID. Repeating an identical wake returns the
original TAMQ message ID; reusing the lease ID for a different request fails.
The client opens a new authenticated socket connection per operation, so
receipt recovery after broker restart uses `message` or `history` rather than a
connection session. If the bound endpoint disappears, repeating the identical
wake may rebind the same durable message ID to its one replacement endpoint;
TAMQ refuses implicit movement while the old endpoint remains live.
Message bodies are human-readable UTF-8 plain text, limited to 8 KiB and
preserving multiline content. The transport envelope carries structured IDs,
sender/target slugs, timestamps, and policy metadata. Binary payloads and
oversized bodies are rejected. Message text is never interpreted as task or
workplan state.
On attach, `tmux-amq` connects to a coordination-engine Unix domain socket and
performs a handshake, receiving a unique
`coordination_endpoint_id`. That identifier namespaces all messages and leases
originating from the connected endpoint, so local IDs can safely be reused
across different machines or sessions without cross-endpoint collision.
Coordination-engine MUST treat `(coordination_endpoint_id, local_message_id)`
as the message identity and preserve the endpoint ID in receipts.
The socket protocol MUST use structured messages and authenticate the peer by
filesystem ownership/permissions. The default deployment runs both processes
under the same approved local user, uses socket mode `0600`, and validates Unix
peer credentials; bearer tokens in command lines are not required. The initial
handshake includes the gita
registered repository slugs, tmux session name, and agent command; the response
contains the endpoint ID and negotiated lease/receipt capabilities. No TCP
listener is required for the local integration.
coordination-engine MUST refresh the local gita registry before accepting an
attach or addressing a target. It accepts only exact registered repository
slugs, requires the attached session to report the matching slug/path, and
never auto-clones or auto-registers a repository.
The adapter interface is:
```yaml
WakeRequest:
lease_id: string
target_repo: string # must be gita-registered and an active tmux-amq target
target_agent: string?
trigger: WorkerTrigger
checkpoint: Checkpoint?
prompt: string
WakeAck:
lease_id: string
session_id: string
accepted_at: timestamp
capability: resumed | started | unavailable
```
The prompt is generated from the trigger and the standard continuation policy;
it MUST contain no secret values. The target agent remains the authority for
editing files, calling tools, and updating State Hub.
## 6. Checkpoints and standard continuation policy
Agents SHOULD publish a checkpoint before yielding or stopping:
```yaml
Checkpoint:
lease_id: string
task_id: string?
summary: string
files_changed: [string]
next_action: string?
blocked_reason: human | secret | destructive | policy | dependency | unavailable | null
recorded_at: timestamp
```
When a lease is resumed, the service supplies the latest checkpoint and asks
the agent to continue. The following policy templates are recognized, in order:
1. inspect open work and implement the highest-priority actionable item;
2. if no work is open, reconcile `SCOPE.md` with `INTENT.md`, record gaps under
`history/`, and create/register a workplan for the most relevant gap;
3. review pending work, refine it where needed, and implement the next safe item;
4. if no safe work remains, close the session and write a perspective entry to
the configured `hall-of-helix` destination.
Templates are suggestions, not permission to bypass safety gates or invent
scope. A repository may override them through its agent instructions.
### 6.1 Policy profiles
Automatic behavior is selected through named policy profiles rather than code
changes. The default profile is conservative:
```yaml
policy:
profile: default
profiles:
default:
allow: [statehub_read, repo_inspect, repo_edit, local_checks,
state_updates, coordination_messages, tmux_wake, transient_retry]
require_human: [secrets, destructive, live_external, scope_change,
ambiguous_ownership, external_publish, financial, legal]
safety_gated_max_attempts: 0
```
Profiles MUST be selected by configuration, validated at startup, and included
by name/version in every lease and receipt. A profile cannot grant access to
credentials that are unavailable through the configured credential-routing
system, and every profile remains subject to authentication and repository
boundary checks.
## 7. Receipts and observability
Every trigger, lease transition, wake attempt, checkpoint, stop, retry, and
completion MUST emit a sanitized progress/audit receipt containing identifiers,
status, reason, timestamps, and retry metadata. Receipts MUST exclude prompt
secrets, credentials, payload contents, and unrestricted terminal output.
The service SHOULD publish receipts through State Hub progress/events and retain
enough correlation data to explain why a dependent task was (or was not) woken.
## 8. Ownership boundary
Repositories publish workplan/task/message state and consume wake requests. They
MUST NOT add independent polling daemons for this feature. Coordination-engine
owns the service process, observation, lease/deduplication, retry policy, and
wake dispatch. State Hub owns durable task/message state. Terminal/session
control is a pluggable adapter owned by coordination-engine and deployed with
the runtime; its underlying bridge remains an external dependency.
## 9. Operator decisions required before implementation
The following choices cannot be inferred safely from repository documents:
1. Which terminal/session bridge is authorized to wake or resume workers?
2. What local SQLite retention and backup policy should be used for leases and
receipts?
3. Default polling interval, lease duration, renewal interval, and retry limit?
4. Which actions are pre-approved policies versus always human-gated?
5. What service identity and authentication path may write State Hub and invoke
the wake adapter?
6. Which Unix socket path and filesystem ownership should the local deployment
use?
### 9.1 Runtime configuration
The following values MUST be configurable without code changes:
```yaml
coordination:
statehub_poll_interval_seconds: 15
wake_ack_timeout_seconds: 30
lease_renew_interval_seconds: 10
retry_backoff_seconds: [5, 15, 60, 300]
max_attempts: 4
safety_gated_max_attempts: 0
delivery_ack_mode: injected # injected | acknowledged
```
Deployments may choose different values, including up to nine attempts. The
service MUST validate `max_attempts` and `safety_gated_max_attempts` in the
range 0..9 and record the effective configuration version in receipts.
State Hub observation uses the configured polling interval (15 seconds by
default), backs off exponentially while unavailable, and resumes promptly when
connectivity returns. Local tmux-amq messaging is never blocked by State Hub
outages; receipt projections are queued for asynchronous publication.
The State Hub endpoint is configurable through `STATEHUB_API_BASE` or
`statehub.api_base`, defaulting to `http://127.0.0.1:8000`. Credentials MUST be
resolved through the approved credential-routing mechanism and never stored in
tamq configuration, SQLite, JSONL exports, tmux commands, or receipts. Missing
credentials leave local messaging operational while State Hub projections stay
pending.
`tamq status` exposes live endpoint/session, queue, lease, and State Hub
connectivity information. `tamq history` exposes full local delivery history;
State Hub receives sanitized receipts and correlation IDs only. Message bodies
remain local unless explicitly exported to JSONL.
History is never deleted automatically. When invoked without explicit filters,
`tamq purge` uses dynamic defaults equivalent to `--before 365d` and
`--max-size 100MB`; purge requires confirmation and may be preceded by JSONL
export. On startup, if history exceeds 100 MB, tamq prints an advisory such as:
`Size of history is NNN MB since YYMMDD, exceeding 100MB. Consider using 'tamq purge'.`
`tamq purge` performs a dry-run summary by default, reporting the matching
message count, date range, and estimated bytes. Interactive execution requires
explicit confirmation; non-interactive execution requires `--yes`. Combined
filters are allowed, and each purge emits a receipt containing filters and
counts without message bodies.
`tamq start` creates or reuses the tmux server/windows and starts the local
service; `tamq serve` runs the Unix-socket service in the foreground; `tamq stop`
stops the service without killing worker terminals or deleting queues. The
initial repo-opening command MUST ensure the service is alive and start it
automatically when absent. Startup is idempotent and lock-protected so two
concurrent repo opens cannot create duplicate services. `--detach` is required
to daemonize; foreground operation is the default for diagnostics.
If automatic startup fails, tamq reports the error on stderr, returns non-zero,
does not register an endpoint or claim delivery, and leaves existing sessions
and SQLite data untouched. `--no-service` may explicitly open terminals without
messaging, with a visible warning.
## 10. Logging and diagnostics
Foreground `tamq serve` logs to stderr. Detached operation logs to
`${XDG_STATE_HOME:-$HOME/.local/state}/tamq/tamq.log`. The default level is
`info`; `--quiet` suppresses non-errors, `--verbose` enables debug diagnostics,
and `--json` requests structured output where applicable. Message bodies,
credentials, tokens, and unrestricted PTY output are omitted by default.
`--orwell` is an explicitly unsafe, non-production diagnostic mode that logs
otherwise-omitted diagnostic fields, including message bodies and credential
material when available. It MUST be supplied per invocation, emit a prominent
warning, write only to a local mode-`0600` sink, never project to State Hub, and
be rejected by the default production policy profile. Operators are responsible
for purging Orwell logs after diagnosis.
This practice is proposed for canonization as the InfoTechCanon PracticePattern
`practice-pattern/orwell-logging-diagnostics` (`OrwellLoggingDiagnostics`). The
canonical owner is info-tech-canon; tamq is a concrete known use, not the owner
of the general practice.
The first runtime implementation uses Python 3.11+ and should prefer standard
library components (`sqlite3`, `asyncio`, Unix sockets, and CLI primitives) so
local installation remains small and reliable.
The project is packaged through a root `pyproject.toml` with a `tamq` console
entry point. It must support `uv tool install .`, editable development installs,
and package-derived `tamq --version` output.
Verification uses layered tests: pytest unit tests for state/config/protocol
logic; Unix-socket integration tests with fake tmux-amq peers; fake State Hub
and gita registry tests; CLI behavior tests; and opt-in live smoke tests only
when explicitly configured. Tests must not require credentials or mutate live
systems.
The Unix-socket protocol is versioned independently from package versions,
starting at `0.1`. Attach handshakes negotiate protocol version and capabilities;
incompatible major versions are rejected, compatible minor versions may
interoperate, and the negotiated values are retained in endpoint/receipt data.
SQLite stores an explicit schema version. Startup applies forward migrations in
transactions, creates a timestamped backup before altering message/lease
tables, and refuses to run against a newer schema. Message history is never
silently dropped; `tamq db-version` reports the current schema.
SQLite uses WAL mode, foreign-key enforcement, and a configurable busy timeout
(5 seconds by default). Queue and lease transitions are transactional, with a
single coordination-engine writer path and concurrent readers. Transient busy
errors may retry without creating duplicate messages.
On `SIGTERM` or `SIGINT`, the service stops accepting socket requests and new
leases, finishes the current transaction, marks the endpoint disconnected,
closes the socket, preserves queues/leases for recovery, emits a sanitized
shutdown receipt, and exits within a configurable grace period (10 seconds by
default).
Default CI gates are `uv run pytest`, `git diff --check`,
`python -m compileall src`, `tamq --help`, and `tamq --version`; package
install/build and live smoke checks are separate opt-in jobs.
Health is local-only: `tamq status` is human-readable, `tamq status --json` is
machine-readable, and `tamq ping` performs a lightweight Unix-socket liveness
check. Status separates process/socket, SQLite, gita registry, State Hub, queue,
and lease health; no TCP health port is exposed.
Status: local-alpha implementation; operational defaults approved in WP-0003-T02.
Owner: coordination-engine. Transport owner: tmux-amq. State owner: repositories
and State Hub's file-backed projections.
This revision reconciles the original proposal with TAMQ's implemented
`spec/coordination-engine-adapter-v0.1.md`. The original requirements are retained
in `history/260907-worker-coordination-service-v0.1-proposal.md`; TAMQ-specific
CLI, history, replay, endpoint registration and logging requirements belong to
that repository. This package installs `coordination-engine`, never `tamq`.
## Observation and actionability
The runtime polls State Hub's repos, workplans, tasks, dependency edges and unread
messages. It dispatches only after a complete successful snapshot, only for
explicitly configured gita repositories. It refreshes gita before addressing a
TAMQ target and on worker updates; it never clones or registers repositories.
Task triggers require ready/active workplans, todo/progress task status, and
satisfied workplan/workplan or workplan/task dependencies. Wait tasks remain
waiting until their authoritative status changes. Unknown dependency types or
missing targets wait. The Hub's edge direction is `from_workplan_id` depends on
`to_workplan_id` or `to_task_id`.
The current Hub has no action-request flag on messages. An unread, unarchived
message whose subject starts with `[action]` requests inspection. Other messages
do not wake workers. Message text never grants authority or changes workflow
state. Generic free-text progress records are not interpreted as checkpoints;
workers publish structured checkpoints through the local control socket.
A normalized trigger has source (`task`, `message`, `checkpoint`), source ID,
source revision, repository and trigger ID. The ID hashes the tuple
`(source, source_id, source_revision, repo)`. Revisions hash the relevant observed
record, plan and dependency state, so dependency completion is reconsidered even
when the task timestamp does not change. Payload text is not retained locally
as trigger data and is never copied to a wake prompt.
## Policy and safety
The named `default` policy permits repository inspection, authorized edits,
local checks, state updates and tmux wakes; configuration may narrow this set.
Unknown profiles/action classes are refused. Human flags, intervention notes,
blocking reasons and conservative sensitive-action text detection stop a
trigger. Sensitive retries are always zero in v0.1. Text detection adds stops;
it does not authorize arbitrary source text. Workers must independently recheck
current repository instructions, scope, dependency state and authority.
Authentication failures, unregistered/mismatched targets, incompatible protocol
and ambiguous endpoints stop the lease. Missing endpoints and ordinary transport
failures use bounded backoff. No safety stop silently reroutes to another worker.
A new source revision may be evaluated normally; the same stopped revision is
never automatically reset. A changed runtime configuration stops pending leases
created under the old configuration rather than silently changing their policy.
## Leases, checkpoints and completion
SQLite stores unique trigger and coordination lease IDs, source/revision/repo,
state, attempts, expiry, retry deadline, policy/configuration versions, checkpoint,
and TAMQ endpoint/message/transport state. State transitions and audit receipts
are committed together. One unexpired offered/acknowledged/running lease per
repository prevents overlapping automatic wakes, including across task revisions.
States are pending, waiting, offered, acknowledged, running, checkpointed,
retry_wait, completed and stopped. Every transition emits a sanitized receipt.
Workers acknowledge before work, renew while active, and checkpoint or complete.
Unknown, expired, terminal and wrong-repository lease updates are rejected.
Transport recovery repeats the same coordination lease ID and identical prompt.
TAMQ deduplicates admission and returns its original message ID. Expiry/failure
uses configurable bounded backoff and a 0–9 attempt ceiling. Already delivered
TAMQ messages are not forcibly reinjected by coordination retry; an unresponsive
worker eventually stops for review. Terminal TAMQ failure stops coordination.
A normal checkpoint ends its lease and creates a new checkpoint trigger/lease,
linked through `source_id` to the parent trigger. This continuation is a new wake,
not a transport retry. The worker reads its parent's local checkpoint before
resuming. Checkpoints allow summary, files_changed, next_action and blocked_reason
within 8 KiB; contents stay local. Any blocked_reason stops automatic continuation.
TAMQ injected/acknowledged means transport delivery, never task completion.
Completion requires a worker completion operation or authoritative task done.
Source removal or other superseding changes stop stale leases. A safe status
revision does not revoke an unexpired active worker merely to issue another wake.
## TAMQ boundary
Coordination-engine connects to the configured TAMQ Unix socket as the same
approved local user. It checks socket ownership/mode 0600 and Linux SO_PEERCRED,
negotiates protocol 0.1 through ping, and requires `bounded_delivery_ack_v1` and
`idempotent_send_v1`. Each operation opens a fresh authenticated connection.
Exactly one live endpoint must advertise the selected gita repository. Multiple
matches require configured endpoint_id; zero is unavailable. TAMQ validates its
registered repository paths and owns endpoint lifecycle. Coordination-engine
checks any advertised path and never controls tmux directly.
The send envelope uses client_id/sender_repo `coordination-engine`, lease ID as
idempotency_key, trigger ID as correlation_id, exact target/endpoint, provenance
`coordination_engine`, structured lease/trigger metadata and a standard plain-text
prompt. TAMQ owns its 8 KiB body limit, queue, delivery acknowledgement profile,
local leases, retry timing and durable admission. The coordination runtime never
resets a failed TAMQ message automatically. Receipt recovery uses `message` after
restart; endpoint ID and message ID are preserved as the correlated pair.
The original inbound attach/handshake proposal is superseded: no second TAMQ
registration socket is introduced. The separate coordination control socket
serves ping, status, stop and structured worker ack/renew/checkpoint/complete
operations over newline-delimited JSON with an `op` field. It accepts same-user
peers only, enforces a 16 KiB request limit, validates selected/gita repository
claims, and returns JSON `ok` responses. It is a same-user trust boundary.
## Storage, lifecycle and configuration
Python 3.11+, standard-library runtime dependencies, root pyproject.toml, uv
install/editable support. See `docs/worker-runtime.md` for the exact TOML schema,
XDG paths, overrides, CLI, worker examples, backups and restore procedure.
Operator-approved operational defaults: 15-second polling, 30-second leases, 10-second
renewal, four attempts, 5/15/60/300-second delays, five-second adapter/busy timeout,
private mode-0700 state and mode-0600 sockets/databases/backups. Explicit repo
selection is mandatory; package installation never enables a service.
SQLite uses WAL and foreign keys. Schema upgrades are transactional, snapshot
existing tables before migrations, and reject newer versions. History is never
automatically removed. Backup uses SQLite's consistent backup API. An exclusive
writer lock prevents concurrent service/once dispatch. SIGINT/SIGTERM stops new
work, preserves durable leases and removes the control socket. The synchronous
local-alpha polling implementation is intended for small local worker sets;
control calls can wait for an observation pass. See the runbook's timing caveat.
Hub outages suspend new wakes and back polling off to 300 seconds. TAMQ local
messaging remains independent. Receipt projection retries from a local outbox;
projection is at least once, with a stable receipt ID in `detail.id`. Edge-relay
queued responses remain pending evidence. Credentials are not configuration and
must follow the approved credential-routing subsystem if access is unavailable.
## Diagnostics and verification
Default errors omit arbitrary remote exception text. Audit receipts contain only
IDs, state/reason codes, timestamps, retry metadata and policy/configuration
versions. No source bodies, checkpoint text, credentials or terminal output are
projected. Local status/history may expose checkpoints to the approved user.
TAMQ owns unsafe `--orwell` diagnostics. Coordination-engine has no unsafe sink.
`docs/orwell-logging-diagnostics-candidate.md` records the candidate practice;
canonical review/registration remains with info-tech-canon, tracked separately
in COORDINATION-WP-0004 with operator approval.
Default checks: `uv run pytest`, `git diff --check`, compileall, CLI help/version.
Tests exercise SQLite durability and locking, fake State Hub and gita, actual
local sockets with fake TAMQ peers, CLI behavior, service stop/restart, safety,
dependencies and checkpoint continuation. Package builds and explicitly enabled
read-only live adapter discovery are separate checks. No credential or live
worker mutation is required by the test suite.