feat: complete reliable coordination adapter
Some checks failed
tamq-ci / test (push) Failing after 5s
Some checks failed
tamq-ci / test (push) Failing after 5s
Assistant: codex Assistant-Model: gpt-5.6-sol Assistant-Session: 01a03397-4d51-7fd1-8ff2-946eb22ea2bc
This commit is contained in:
parent
25113f463e
commit
6d2ccc7760
30 changed files with 2553 additions and 144 deletions
|
|
@ -16,6 +16,7 @@ jobs:
|
|||
- run: sudo apt-get install -y tmux
|
||||
- run: python -m pip install uv
|
||||
- run: python -m pip install -e '.[dev]'
|
||||
- run: pytest -q
|
||||
- run: make check
|
||||
- run: python -m tamq.cli --help
|
||||
- run: python -m tamq.cli --version
|
||||
- run: python -m tamq.cli capture --help
|
||||
|
|
|
|||
112
README.md
112
README.md
|
|
@ -4,8 +4,8 @@ Tmux Agentic Message Queueing (`tamq`) provides repository-aware tmux sessions
|
|||
and a local, durable message queue. It does not assume who or what uses a pane.
|
||||
|
||||
The local alpha provides tmux endpoint lifecycle, local SQLite history and
|
||||
leases, readable `To:` routing, JSONL export/replay, and a Unix-socket protocol
|
||||
for a later coordination-engine adapter.
|
||||
leases, readable `To:` routing, JSONL export/replay, bounded delivery state, and
|
||||
a versioned Unix-socket coordination-engine adapter.
|
||||
|
||||
Agents and operators should start with the concise
|
||||
[TAMQ messaging introduction](TamqMessagingIntroduction.md).
|
||||
|
|
@ -182,6 +182,34 @@ the durable id needed by `ack`:
|
|||
From:flex-auth/o: please review the auth boundary
|
||||
```
|
||||
|
||||
### Reviewing the communication protocol
|
||||
|
||||
TAMQ automatically appends structured protocol events to the local database.
|
||||
The ledger includes addressed-message acceptance, operator/worker provenance,
|
||||
worker block boundaries and termination reasons, allowlisted command outcomes,
|
||||
line-limit blocks, endpoint lifecycle, and delivery attempts or failures. It
|
||||
does not capture unrelated pane output or ordinary shell input.
|
||||
|
||||
Create a review artifact with:
|
||||
|
||||
```bash
|
||||
tamq capture --output tamq-protocol.md
|
||||
tamq capture --repo audit-core --output audit-core-protocol.md
|
||||
tamq capture --message-id m-... --format jsonl
|
||||
tamq capture --event delivery.failed
|
||||
```
|
||||
|
||||
Markdown is the human-oriented default and starts with aggregate counts useful
|
||||
for spotting protocol friction. JSONL is intended for scripts. The default is
|
||||
the newest 500 matching events; `--limit` changes that bound. Repository filters
|
||||
match either side of an exchange. A message body is included only on its
|
||||
acceptance event, not repeated for every delivery event. Captures are durable
|
||||
communication evidence, so never put credentials or secrets in TAMQ messages.
|
||||
When upgrading an already-running alpha, recreate the managed session once so
|
||||
its long-lived pane observers use the capture-aware build; existing durable
|
||||
message lifecycle is backfilled, but old observers cannot reconstruct block or
|
||||
command events retroactively.
|
||||
|
||||
To explicitly consume pending messages through a command, use an inbox filter:
|
||||
|
||||
```bash
|
||||
|
|
@ -195,12 +223,26 @@ it and all later messages pending. Filters never run in the background and
|
|||
cannot be combined with `--all` or `--json`.
|
||||
|
||||
Outside a managed window, use `tamq inbox --repo audit-core` and optionally
|
||||
`--json`. Output-displayed messages remain durable and pending until
|
||||
acknowledged. Neither `output` nor `inbox` mode injects terminal keystrokes.
|
||||
Pushy places input without Enter. Trigger waits for terminal paste detection to
|
||||
settle and then adds exactly one Enter. A short endpoint-startup grace protects
|
||||
the first input delivery while the foreground program initializes. Both input
|
||||
modes record accepted placement as `injected`, not recipient acknowledgement.
|
||||
`--json`. Neither `output` nor `inbox` mode injects terminal keystrokes. Pushy
|
||||
places input without Enter. Trigger waits for terminal paste detection to settle
|
||||
and then adds exactly one Enter. A short endpoint-startup grace protects the
|
||||
first input delivery while the foreground program initializes.
|
||||
|
||||
The policy profile controls completion. The default `injected` policy completes
|
||||
after a successful terminal write. `acknowledged` keeps the message in
|
||||
`awaiting_ack`, visible in the inbox, and redelivers the same ID after the ack
|
||||
deadline. Delivery failures and expired leases use bounded backoff; exhaustion
|
||||
becomes `failed`. Inspect the attempt count and reason, then explicitly reset a
|
||||
terminal message if retry is safe:
|
||||
|
||||
```bash
|
||||
tamq inspect m-...
|
||||
tamq retry m-...
|
||||
```
|
||||
|
||||
Acknowledged-mode redelivery can create duplicate terminal presentation. A late
|
||||
ack wins even after exhaustion. Neither state proves that an agent completed the
|
||||
requested work.
|
||||
|
||||
Every session window has independent, session-lifetime running counters for
|
||||
accepted outbound messages, operator input lines, and normalized worker output
|
||||
|
|
@ -296,17 +338,16 @@ later phase.
|
|||
This keeps tmux-specific topology concerns separate from reusable terminal I/O
|
||||
observation and message identity.
|
||||
|
||||
Normal endpoints use terminal-output delivery: each message is written once to
|
||||
the target pane's PTY output and stays pending in the durable inbox until
|
||||
acknowledged. `--mode inbox` selects inbox-only manual mode. Neither becomes
|
||||
pane input. Experimental `--mode pushy` places a sanitized `From:` block
|
||||
Normal endpoints use terminal-output delivery. `--mode inbox` selects
|
||||
inbox-only manual mode. Neither becomes pane input. Experimental `--mode pushy`
|
||||
places a sanitized `From:` block
|
||||
without Enter; `--mode trigger` performs the same placement, lets paste
|
||||
detection settle, and submits once.
|
||||
The PTY observer records explicit operator/worker provenance and suppresses
|
||||
echoed operator lines before they can route again.
|
||||
The older pane-delivery experiment remains available only with the explicit
|
||||
`tamq start --tap --command ...` opt-in. Both input paths remain subject to the
|
||||
retry and acknowledgement limitations tracked by `TAMQ-WP-0003`.
|
||||
`tamq start --tap --command ...` opt-in. Both input paths use the same bounded
|
||||
retry and acknowledgement state machine.
|
||||
The visible endpoint label is `tmux-amq-<PID>`; each boot also receives an
|
||||
instance nonce so PID reuse cannot collide with prior leases or receipts.
|
||||
|
||||
|
|
@ -324,6 +365,9 @@ purge_before = "365d"
|
|||
purge_max_size = "100MB"
|
||||
history_max_size = "100MB"
|
||||
delivery_poll_interval = "0.5"
|
||||
delivery_lease_seconds = "30"
|
||||
delivery_retry_backoff_seconds = "5,15,60,300"
|
||||
delivery_ack_timeout_seconds = "30"
|
||||
maxmsg = 8
|
||||
maxin = 1024
|
||||
maxout = 32768
|
||||
|
|
@ -331,15 +375,43 @@ maxout = 32768
|
|||
[policy.profiles.diagnostics]
|
||||
safety_gated_max_attempts = 2
|
||||
delivery_ack_mode = "acknowledged"
|
||||
delivery_max_attempts = 4
|
||||
```
|
||||
|
||||
Policy profiles accept a safety-gated retry cap of at most nine attempts, but
|
||||
the alpha delivery path does not enforce attempt counting yet. Explicit
|
||||
acknowledgement exists, while `delivery_ack_mode` enforcement and bounded retry
|
||||
state are tracked in `TAMQ-WP-0003`. Use `--policy-profile` to select a profile;
|
||||
`--orwell` enables explicitly unsafe local diagnostics.
|
||||
Policy profiles accept one to nine delivery attempts. Lease expiry and write
|
||||
failure consume an attempt; retry delay uses the configured sequence and then
|
||||
repeats its last value. `delivery_ack_mode` is either `injected` or
|
||||
`acknowledged`. Use `--policy-profile` to select a profile; `--orwell` enables
|
||||
explicitly unsafe local diagnostics.
|
||||
|
||||
The Unix socket service supports structured `ping`, `register`, `send`,
|
||||
`history`, `ack`, `endpoints`, and `disconnect` operations. Endpoint
|
||||
`message`, `history`, `ack`, `retry`, `endpoints`, and `disconnect` operations. Endpoint
|
||||
registrations and messages are persisted in the same local SQLite database;
|
||||
delivery remains delegated to the control-mode adapter.
|
||||
|
||||
## Coordination-engine adapter
|
||||
|
||||
`tamq.client.CoordinationEngineAdapter` is a transport-only async adapter. It
|
||||
does not import tmux/control-mode code and opens a fresh authenticated
|
||||
Unix-socket connection per operation. A wake resolves exactly one live endpoint,
|
||||
validates the gita target, and uses the coordination lease ID as its idempotency
|
||||
key:
|
||||
|
||||
```python
|
||||
from tamq.client import CoordinationEngineAdapter, WakeRequest
|
||||
|
||||
receipt = await CoordinationEngineAdapter().wake(
|
||||
WakeRequest(
|
||||
lease_id="lease-123",
|
||||
trigger_id="task-456:r7",
|
||||
target_repo="audit-core",
|
||||
prompt="Resume the actionable task and publish a checkpoint.",
|
||||
)
|
||||
)
|
||||
```
|
||||
|
||||
Repeating that wake returns the same message ID. Reusing its lease ID for a
|
||||
different payload is rejected. Receipt recovery, explicit acknowledgement, and
|
||||
terminal retry use `receipt`, `acknowledge`, and `retry_failed`. The normative
|
||||
state and ownership contract is
|
||||
[`spec/coordination-engine-adapter-v0.1.md`](spec/coordination-engine-adapter-v0.1.md).
|
||||
|
|
|
|||
37
SCOPE.md
37
SCOPE.md
|
|
@ -14,7 +14,8 @@ tamq does not choose or infer them.
|
|||
## In Scope
|
||||
|
||||
- Local SQLite message history, leases, endpoint registrations, delivery state,
|
||||
acknowledgements, replay, export, and bounded purging.
|
||||
acknowledgements, replay, export, bounded purging, and a structured
|
||||
communication-protocol ledger for review.
|
||||
- Managed neutral-shell tmux lifecycle and explicit initial commands.
|
||||
- Durable send/inbox/acknowledgement with per-window repository identity,
|
||||
readable `To:`/`From:` framing, explicit operator/worker provenance, and
|
||||
|
|
@ -30,8 +31,8 @@ tamq does not choose or infer them.
|
|||
an empty line.
|
||||
- Operator-only allowlisted `Cmd:` runtime changes and atomic, session-lifetime
|
||||
per-window message/input/output line budgets.
|
||||
- Unix-socket operations for local clients and a future coordination-engine
|
||||
adapter.
|
||||
- Versioned Unix-socket operations and a transport-only coordination-engine
|
||||
client adapter with correlation and idempotent wake admission.
|
||||
- Policy profiles, safety-gated retries, local diagnostics, tests, packaging,
|
||||
shell completion, and operator documentation.
|
||||
- Dry-run emergency cleanup for verified services, tamq-marked sessions,
|
||||
|
|
@ -63,13 +64,14 @@ transport.
|
|||
| Local socket service | Implemented | Peer-credential checks and structured ping/register/send/history/ack/endpoints/disconnect operations are tested. |
|
||||
| Neutral tmux session lifecycle | Implemented for local alpha | Repository-first startup opens untouched shells at exact gita paths, exports per-window identity, and runs no initial command unless `--command` is explicit. Stable reuse, service restart, and cleanup are covered by the installed-package test. |
|
||||
| Emergency local cleanup | Implemented for local alpha | `tamq cleanup` dry-runs by default; confirmed cleanup verifies ownership before stopping the broker or marked session, clears only transient DB state, removes configured runtime files/generated shims/owned stale tmux sockets, and preserves history. |
|
||||
| Safe output messaging | Implemented for local alpha | Normal endpoints write one sanitized, non-routable `From:` block above a stable shell input row without injecting stdin; `/o` marks operator origin. Messages remain pending until acknowledgement. Inbox-only mode is explicit. |
|
||||
| Safe output messaging | Implemented for local alpha | Normal endpoints write one sanitized, non-routable `From:` block above a stable shell input row without injecting stdin; `/o` marks operator origin. Inbox-only mode is explicit. The selected acknowledgement policy determines completion. |
|
||||
| Experimental pushy and trigger delivery | Explicit opt-in | Pushy places a non-routable `From:` block without Enter; trigger waits past paste detection and adds exactly one Enter. A startup grace protects first delivery. Both are capability-gated and mark accepted delivery `injected`, but cannot identify pane occupants or protect input already being edited. |
|
||||
| Full-duplex observation | Implemented for managed messaging | Every messaging-enabled new window runs its neutral shell or explicit command behind the PTY observer. It preserves geometry, resize, mouse input, and raw forwarding; collects worker `To:` blocks through an empty row, recognizes conservative full-screen worker-output gutters and row gaps, deduplicates redraws, and fails closed on exact recent operator echoes. |
|
||||
| Runtime commands and generation budgets | Implemented for local alpha | Operator-only `Cmd:` changes mode or per-window limits and resets the current ledger. Defaults are 8 message, 1024 input, and 32768 output lines. Admission and counter increments are atomic and survive service/tap restarts in one session generation. |
|
||||
| Bounded retry behavior | Not enforced | Failed output or injection remains pending and becomes claimable after lease expiry, but no attempt counter or terminal failure state applies the configured cap. |
|
||||
| Acknowledgement policy | Partially implemented | Terminal output remains pending until explicit acknowledgement, while legacy pane injection becomes `injected`; the configured `delivery_ack_mode` does not yet govern both paths. |
|
||||
| Coordination-engine interoperability | Not implemented | The adapter contract and implementation remain in `TAMQ-WP-0002`. |
|
||||
| Protocol review capture | Implemented for local alpha | An append-only SQLite ledger records addressed-message acceptance, provenance, worker block boundaries, allowlisted command outcomes, endpoint changes, limit blocks, and delivery outcomes. `tamq capture` renders filtered Markdown or JSONL without recording unrelated pane output or shell input. |
|
||||
| Bounded retry behavior | Implemented for local alpha | Lease acquisition increments a persistent attempt count. Failures and lease expiry schedule bounded backoff; cap exhaustion becomes inspectable `failed`, and `tamq retry` explicitly resets a terminal message. |
|
||||
| Acknowledgement policy | Implemented for local alpha | `injected` completes on successful terminal write. `acknowledged` enters `awaiting_ack`, redelivers the same message ID after a deadline, exhausts with `ack_timeout`, accepts late acknowledgements, and exposes duplicate-delivery semantics. |
|
||||
| Coordination-engine interoperability | Implemented transport boundary | The versioned same-user Unix-socket client negotiates capabilities, resolves an exact live endpoint, admits idempotent correlated wakes, recovers receipts after restart, and exposes ack/retry without importing tmux control code. Coordination-engine still owns its orchestration runtime. |
|
||||
|
||||
## Practical Usability
|
||||
|
||||
|
|
@ -89,22 +91,24 @@ Suitable today:
|
|||
gita-registered repositories.
|
||||
- Durable message exchange between managed repository windows, including
|
||||
explicit pull-time loggers and filters.
|
||||
- Reviewing repository-to-repository exchanges and onboarding friction through
|
||||
structured Markdown or JSONL protocol captures.
|
||||
- Controlled experiments with pushy placement or trigger submission to
|
||||
interfaces known to queue asynchronous prompts.
|
||||
- Developing and testing the future coordination-engine adapter against the
|
||||
local socket boundary.
|
||||
- Integrating coordination-engine through the implemented transport adapter and
|
||||
durable receipt boundary.
|
||||
|
||||
Not yet suitable:
|
||||
|
||||
- Unattended pane injection where bounded retries and positive recipient
|
||||
acknowledgement are required.
|
||||
- Treating terminal delivery or acknowledgement as proof of completed work;
|
||||
workflow completion remains coordination-engine/State Hub state.
|
||||
- Pushy delivery to arbitrary shells, editors, or panes with unknown input
|
||||
state.
|
||||
- Production-style operation without longer-running crash/terminal soak tests
|
||||
and stronger process-supervision evidence.
|
||||
- Cross-host messaging or use as a general-purpose broker.
|
||||
|
||||
The suite currently has 166 passing tests. It includes atomic counter races,
|
||||
The suite currently has 182 passing tests. It includes atomic counter races,
|
||||
pseudo-terminal normalization, real tmux pushy/trigger behavior, operator-echo
|
||||
suppression, and an isolated installed-package workflow with deterministic gita
|
||||
fixtures. Forgejo CI installs tmux and uv and retains CLI help/version smoke
|
||||
|
|
@ -112,10 +116,11 @@ checks on Python 3.11.
|
|||
|
||||
## Next Usability Gates
|
||||
|
||||
- `TAMQ-WP-0003-T01` and `T02` still own bounded retry state and acknowledgement
|
||||
enforcement. Its real tmux/PTY lifecycle gate (`T03`) is complete.
|
||||
- `TAMQ-WP-0002` owns the coordination-engine adapter after the local delivery
|
||||
contract is sufficiently reliable.
|
||||
- Operational soak evidence is still needed before treating input delivery as
|
||||
production-grade or unattended across arbitrary terminal applications.
|
||||
- Coordination-engine owns implementing its trigger observer, coordination
|
||||
leases, safety gates, checkpoints, and State Hub projections on top of this
|
||||
completed local transport boundary.
|
||||
|
||||
## Getting Oriented
|
||||
|
||||
|
|
|
|||
|
|
@ -75,8 +75,9 @@ The endpoint mode controls what delivery means:
|
|||
New pushy/trigger endpoints have a short startup grace so their foreground
|
||||
program can initialize before the first delivery. These modes still cannot
|
||||
prove that an arbitrary program accepted or understood the input. A durable
|
||||
state of `injected` records successful terminal placement, not agent
|
||||
acknowledgement.
|
||||
state of `injected` records completion under the default terminal-write policy,
|
||||
not agent comprehension or task completion. An `acknowledged` policy instead
|
||||
retains the message in `awaiting_ack` until explicit acknowledgement.
|
||||
|
||||
An operator can change the live mode:
|
||||
|
||||
|
|
@ -112,6 +113,38 @@ tamq inbox --repo audit-core
|
|||
tamq inspect <message-id>
|
||||
tamq ack <message-id>
|
||||
tamq send --from audit-core 'To:flex-auth: Direct CLI message.'
|
||||
tamq capture --repo audit-core --output tamq-protocol.md
|
||||
```
|
||||
|
||||
TAMQ automatically keeps a structured protocol ledger for review. It records
|
||||
addressed-message acceptance, operator/worker provenance, worker block start
|
||||
and close reasons, allowlisted `Cmd:` outcomes, limit blocks, endpoint changes,
|
||||
and delivery attempts or failures. It does **not** record unrelated shell input
|
||||
or arbitrary pane output. Message bodies appear once, at their durable
|
||||
acceptance event.
|
||||
|
||||
Use Markdown for a human review or JSONL for analysis:
|
||||
|
||||
```bash
|
||||
tamq capture --output tamq-protocol.md
|
||||
tamq capture --repo audit-core --format jsonl --output audit-core-protocol.jsonl
|
||||
tamq capture --message-id m-... --output one-message.md
|
||||
tamq capture --event delivery.failed
|
||||
```
|
||||
|
||||
The default report contains the newest 500 matching events and a summary of
|
||||
message provenance, attempts, failures, blocks, and outcomes. Filters match a
|
||||
repository as sender or recipient. Do not put secrets in TAMQ messages: durable
|
||||
history and captures intentionally retain addressed message content.
|
||||
|
||||
After first installing capture support over an older running alpha, recreate
|
||||
the managed session once when convenient. Existing durable message lifecycle
|
||||
is backfilled, but old already-running pane observers cannot emit worker-block
|
||||
and command events retroactively:
|
||||
|
||||
```bash
|
||||
make cleanup
|
||||
tamq --command codex --mode trigger flex-auth audit-core
|
||||
```
|
||||
|
||||
If no reply arrives, distinguish these states before resending:
|
||||
|
|
@ -119,9 +152,12 @@ If no reply arrives, distinguish these states before resending:
|
|||
- no durable history row: the source line or completed worker block was not
|
||||
observed, the slug was invalid, or a line budget blocked it;
|
||||
- `pending`: no eligible live endpoint has completed the selected delivery;
|
||||
- `displayed`: output mode showed it, but did not submit it to the worker;
|
||||
- `injected`: pushy/trigger placement succeeded, but the worker has not
|
||||
necessarily processed it;
|
||||
- `awaiting_ack`: terminal delivery succeeded, but explicit acknowledgement is
|
||||
outstanding and same-ID redelivery may occur after the deadline;
|
||||
- `injected`: the selected terminal-write policy completed, but the worker has
|
||||
not necessarily processed it;
|
||||
- `failed`: bounded delivery or acknowledgement attempts were exhausted; inspect
|
||||
the reason and use `tamq retry <message-id>` only when redelivery is safe;
|
||||
- explicit reply or acknowledgement: the recipient confirmed receipt.
|
||||
|
||||
Use a correlation token when duplicate requests would be harmful. TAMQ is an
|
||||
|
|
|
|||
|
|
@ -11,8 +11,8 @@
|
|||
| workplan | TAMQ-WP-ADHOC-2026-08-24 | finished | — | workplans/ADHOC-2026-08-24.md |
|
||||
| workplan | TAMQ-WP-ADHOC-2026-08-25 | finished | — | workplans/ADHOC-2026-08-25.md |
|
||||
| workplan | TAMQ-WP-0001 | finished | — | workplans/TAMQ-WP-0001-statehub-bootstrap.md |
|
||||
| workplan | TAMQ-WP-0002 | active | — | workplans/TAMQ-WP-0002-coordination-engine-adapter.md |
|
||||
| workplan | TAMQ-WP-0003 | active | — | workplans/TAMQ-WP-0003-delivery-reliability.md |
|
||||
| workplan | TAMQ-WP-0002 | finished | — | workplans/TAMQ-WP-0002-coordination-engine-adapter.md |
|
||||
| workplan | TAMQ-WP-0003 | finished | — | workplans/TAMQ-WP-0003-delivery-reliability.md |
|
||||
| workplan | TAMQ-WP-0004 | finished | — | workplans/TAMQ-WP-0004-operator-installable-local-alpha.md |
|
||||
| workplan | TAMQ-WP-0005 | finished | — | workplans/TAMQ-WP-0005-terminal-neutral-manual-messaging.md |
|
||||
| workplan | TAMQ-WP-0006 | finished | — | workplans/TAMQ-WP-0006-shell-native-message-routing.md |
|
||||
|
|
@ -25,6 +25,7 @@
|
|||
| workplan | TAMQ-WP-0013 | finished | — | workplans/TAMQ-WP-0013-emergency-cleanup.md |
|
||||
| workplan | TAMQ-WP-0014 | finished | — | workplans/TAMQ-WP-0014-readable-duplex-messaging-and-limits.md |
|
||||
| workplan | TAMQ-WP-0015 | finished | — | workplans/TAMQ-WP-0015-reliable-trigger-and-worker-blocks.md |
|
||||
| workplan | TAMQ-WP-0016 | finished | — | workplans/TAMQ-WP-0016-structured-protocol-capture.md |
|
||||
| task | TAMQ-WP-ADHOC-2026-08-24-T01 | done | — | workplans/ADHOC-2026-08-24.md |
|
||||
| task | TAMQ-WP-ADHOC-2026-08-25-T01 | done | — | workplans/ADHOC-2026-08-25.md |
|
||||
| task | TAMQ-WP-ADHOC-2026-08-25-T02 | done | — | workplans/ADHOC-2026-08-25.md |
|
||||
|
|
@ -33,13 +34,13 @@
|
|||
| task | TAMQ-WP-0001-T01 | done | — | workplans/TAMQ-WP-0001-statehub-bootstrap.md |
|
||||
| task | TAMQ-WP-0001-T02 | done | — | workplans/TAMQ-WP-0001-statehub-bootstrap.md |
|
||||
| task | TAMQ-WP-0001-T03 | done | — | workplans/TAMQ-WP-0001-statehub-bootstrap.md |
|
||||
| task | TAMQ-WP-0002-T01 | todo | — | workplans/TAMQ-WP-0002-coordination-engine-adapter.md |
|
||||
| task | TAMQ-WP-0002-T02 | wait | — | workplans/TAMQ-WP-0002-coordination-engine-adapter.md |
|
||||
| task | TAMQ-WP-0002-T03 | wait | — | workplans/TAMQ-WP-0002-coordination-engine-adapter.md |
|
||||
| task | TAMQ-WP-0003-T01 | todo | — | workplans/TAMQ-WP-0003-delivery-reliability.md |
|
||||
| task | TAMQ-WP-0003-T02 | wait | — | workplans/TAMQ-WP-0003-delivery-reliability.md |
|
||||
| task | TAMQ-WP-0002-T01 | done | — | workplans/TAMQ-WP-0002-coordination-engine-adapter.md |
|
||||
| task | TAMQ-WP-0002-T02 | done | — | workplans/TAMQ-WP-0002-coordination-engine-adapter.md |
|
||||
| task | TAMQ-WP-0002-T03 | done | — | workplans/TAMQ-WP-0002-coordination-engine-adapter.md |
|
||||
| task | TAMQ-WP-0003-T01 | done | — | workplans/TAMQ-WP-0003-delivery-reliability.md |
|
||||
| task | TAMQ-WP-0003-T02 | done | — | workplans/TAMQ-WP-0003-delivery-reliability.md |
|
||||
| task | TAMQ-WP-0003-T03 | done | — | workplans/TAMQ-WP-0003-delivery-reliability.md |
|
||||
| task | TAMQ-WP-0003-T04 | wait | — | workplans/TAMQ-WP-0003-delivery-reliability.md |
|
||||
| task | TAMQ-WP-0003-T04 | done | — | workplans/TAMQ-WP-0003-delivery-reliability.md |
|
||||
| task | TAMQ-WP-0004-T01 | done | — | workplans/TAMQ-WP-0004-operator-installable-local-alpha.md |
|
||||
| task | TAMQ-WP-0004-T02 | done | — | workplans/TAMQ-WP-0004-operator-installable-local-alpha.md |
|
||||
| task | TAMQ-WP-0004-T03 | done | — | workplans/TAMQ-WP-0004-operator-installable-local-alpha.md |
|
||||
|
|
@ -89,3 +90,7 @@
|
|||
| task | TAMQ-WP-0015-T02 | done | — | workplans/TAMQ-WP-0015-reliable-trigger-and-worker-blocks.md |
|
||||
| task | TAMQ-WP-0015-T03 | done | — | workplans/TAMQ-WP-0015-reliable-trigger-and-worker-blocks.md |
|
||||
| task | TAMQ-WP-0015-T04 | done | — | workplans/TAMQ-WP-0015-reliable-trigger-and-worker-blocks.md |
|
||||
| task | TAMQ-WP-0016-T01 | done | — | workplans/TAMQ-WP-0016-structured-protocol-capture.md |
|
||||
| task | TAMQ-WP-0016-T02 | done | — | workplans/TAMQ-WP-0016-structured-protocol-capture.md |
|
||||
| task | TAMQ-WP-0016-T03 | done | — | workplans/TAMQ-WP-0016-structured-protocol-capture.md |
|
||||
| task | TAMQ-WP-0016-T04 | done | — | workplans/TAMQ-WP-0016-structured-protocol-capture.md |
|
||||
|
|
|
|||
110
spec/coordination-engine-adapter-v0.1.md
Normal file
110
spec/coordination-engine-adapter-v0.1.md
Normal file
|
|
@ -0,0 +1,110 @@
|
|||
# Coordination-engine ↔ TAMQ adapter contract v0.1
|
||||
|
||||
Status: implemented local-alpha contract. TAMQ owns this transport contract;
|
||||
coordination-engine owns trigger policy, coordination leases, checkpoints, and
|
||||
workflow state.
|
||||
|
||||
## Boundary and authentication
|
||||
|
||||
The adapter connects to TAMQ's configured Unix-domain socket. The socket is
|
||||
mode `0600`, and TAMQ accepts only the same Unix user through `SO_PEERCRED` when
|
||||
the platform exposes it. There is no TCP listener or command-line bearer token.
|
||||
The client opens a new connection for every operation, so restarting TAMQ needs
|
||||
no reconnect handshake or client-side session state.
|
||||
|
||||
The client first sends `ping` with protocol `0.1`. Major versions must match and
|
||||
the service must advertise `bounded_delivery_ack_v1` and `idempotent_send_v1`.
|
||||
Unknown minor capabilities are ignored. An incompatible major version or a
|
||||
missing required capability stops the wake before message admission.
|
||||
|
||||
## Wake mapping
|
||||
|
||||
`CoordinationEngineAdapter.wake(WakeRequest)` resolves exactly one live endpoint
|
||||
whose registered repository list includes `target_repo`. Zero matches returns
|
||||
unavailable. Multiple matches require an explicit endpoint ID; TAMQ never
|
||||
chooses an arbitrary worker session.
|
||||
|
||||
The wake becomes a `send` request:
|
||||
|
||||
```json
|
||||
{
|
||||
"op": "send",
|
||||
"protocol": "0.1",
|
||||
"client_id": "coordination-engine",
|
||||
"idempotency_key": "<coordination lease id>",
|
||||
"correlation_id": "<trigger id>",
|
||||
"sender_repo": "coordination-engine",
|
||||
"target_repo": "<exact gita slug>",
|
||||
"endpoint_id": "<resolved TAMQ endpoint instance>",
|
||||
"provenance": "coordination_engine",
|
||||
"metadata": {"lease_id": "...", "trigger_id": "..."},
|
||||
"body": "<human-readable wake prompt>"
|
||||
}
|
||||
```
|
||||
|
||||
Targets and non-local senders are validated against the current `gita`
|
||||
registry. Bodies are UTF-8 text up to 8 KiB; metadata is a JSON object up to 4
|
||||
KiB. TAMQ transports the prompt but does not interpret it as task state or
|
||||
grant authority.
|
||||
|
||||
`(client_id, idempotency_key)` is unique. Repeating an identical request returns
|
||||
the original message ID with `deduplicated: true`. Reusing the key with a
|
||||
different sender, target, endpoint, or body is an error. The durable identity
|
||||
used by coordination-engine is therefore the pair of TAMQ endpoint instance
|
||||
and local message ID, correlated to its own lease and trigger IDs.
|
||||
|
||||
## Message and delivery states
|
||||
|
||||
| State | Meaning | Coordination interpretation |
|
||||
| --- | --- | --- |
|
||||
| `pending` | Accepted and eligible now or after `next_attempt_at` | Transport owns retry timing. |
|
||||
| `awaiting_ack` | Written successfully under explicit-ack policy | Recipient comprehension is still unproven. |
|
||||
| `injected` | Written successfully under injected policy | Transport delivery is complete, not task completion. |
|
||||
| `acknowledged` | Explicit acknowledgement received | Strongest TAMQ receipt; still not workflow completion. |
|
||||
| `failed` | Delivery or acknowledgement attempts exhausted | Terminal until an operator/client explicitly retries or a late ack arrives. |
|
||||
|
||||
Every delivery lease acquisition increments the persistent `attempt_count`.
|
||||
Failure releases the lease and sets `last_failure_reason` plus a bounded
|
||||
`next_attempt_at`. Lease expiry is itself a failed attempt. Retry delays default
|
||||
to 5, 15, 60, and 300 seconds and use the last value thereafter. The selected
|
||||
endpoint policy supplies `delivery_max_attempts` in the range 1–9.
|
||||
|
||||
With `delivery_ack_mode=injected`, a successful terminal write completes the
|
||||
message immediately. With `acknowledged`, success enters `awaiting_ack`; the
|
||||
default 30-second deadline then redelivers the same message ID. Duplicate visual
|
||||
or input delivery is therefore possible and recipients must correlate by
|
||||
message ID when the operation is not naturally idempotent. Exhausting the cap
|
||||
while waiting produces `failed` with reason `ack_timeout`. A late explicit ack
|
||||
wins even after terminal failure and cancels any outstanding retry. `retry`
|
||||
resets only a `failed` message's attempts and returns it to `pending`.
|
||||
|
||||
## Operations and failure semantics
|
||||
|
||||
- `endpoints`: discover live repository attachment and disambiguate a wake.
|
||||
- `send`: admit an idempotent durable message.
|
||||
- `message`: retrieve one receipt including state, attempts, deadlines, and
|
||||
correlation metadata.
|
||||
- `history`: recover receipts after either process restarts.
|
||||
- `ack`: record explicit or late acknowledgement and cancel pending delivery.
|
||||
- `retry`: operator/policy-authorized reset of terminal delivery failure.
|
||||
|
||||
Endpoint disappearance before admission is `unavailable` and creates no
|
||||
message. Disappearance after admission leaves the message durable. Repeating the
|
||||
identical wake after a replacement endpoint appears rebinds that same message
|
||||
ID, but only when its previous endpoint is no longer live; TAMQ will not move a
|
||||
message between two live sessions implicitly. A process crash after claiming a
|
||||
message is recovered through lease expiry and bounded backoff.
|
||||
|
||||
JSONL replay uses the original message ID (or a deterministic content digest)
|
||||
as a `tamq-replay` idempotency key. Repeating the same replay reports it as
|
||||
deduplicated rather than creating another message. Replay still uses normal
|
||||
gita validation and never changes coordination-engine workflow state.
|
||||
|
||||
## Ownership
|
||||
|
||||
TAMQ owns local queue rows, endpoint registration, delivery leases, retry and
|
||||
ack state, terminal interaction, and protocol capture. Coordination-engine owns
|
||||
trigger deduplication, coordination leases, actionability and safety policy,
|
||||
checkpoints, State Hub projection, and the decision to issue a wake or request
|
||||
an explicit retry. Neither system treats terminal injection as proof that an
|
||||
agent completed work.
|
||||
|
|
@ -2,6 +2,7 @@ from __future__ import annotations
|
|||
|
||||
from collections.abc import Callable
|
||||
from dataclasses import dataclass
|
||||
import json
|
||||
|
||||
from .registry import RegistryError, validate_targets
|
||||
from .routing import (
|
||||
|
|
@ -38,8 +39,30 @@ class InputBroker:
|
|||
self.notify = notify
|
||||
self._worker_target: str | None = None
|
||||
self._worker_lines: list[str] = []
|
||||
self._last_admitted_message_id: str | None = None
|
||||
self.store.configure_window(identity.endpoint_id, identity.source_repo, limits)
|
||||
|
||||
def _event(
|
||||
self,
|
||||
event_type: str,
|
||||
*,
|
||||
peer_repo: str | None = None,
|
||||
message_id: str | None = None,
|
||||
provenance: str | None = None,
|
||||
outcome: str = "observed",
|
||||
detail: dict | None = None,
|
||||
) -> None:
|
||||
self.store.record_protocol_event(
|
||||
event_type,
|
||||
endpoint_id=self.identity.endpoint_id,
|
||||
repo=self.identity.source_repo,
|
||||
peer_repo=peer_repo,
|
||||
message_id=message_id,
|
||||
provenance=provenance,
|
||||
outcome=outcome,
|
||||
detail=detail,
|
||||
)
|
||||
|
||||
def _notice(self, text: str) -> None:
|
||||
if self.notify is not None:
|
||||
self.notify(f"From:tamq: {text}")
|
||||
|
|
@ -52,9 +75,16 @@ class InputBroker:
|
|||
)
|
||||
|
||||
def _admit(self, routed: RoutedMessage, provenance: str) -> RoutedMessage | None:
|
||||
self._last_admitted_message_id = None
|
||||
try:
|
||||
validate_targets([routed.target_repo])
|
||||
except RegistryError:
|
||||
self._event(
|
||||
"message.rejected",
|
||||
peer_repo=routed.target_repo,
|
||||
provenance=provenance,
|
||||
outcome="invalid_target",
|
||||
)
|
||||
return None
|
||||
result = self.store.admit_message(
|
||||
self.identity.source_repo,
|
||||
|
|
@ -67,6 +97,7 @@ class InputBroker:
|
|||
if isinstance(result, LimitBlock):
|
||||
self._blocked(result)
|
||||
return None
|
||||
self._last_admitted_message_id = result
|
||||
return routed
|
||||
|
||||
def _route(self, line: str, provenance: str) -> RoutedMessage | None:
|
||||
|
|
@ -77,21 +108,44 @@ class InputBroker:
|
|||
try:
|
||||
validate_targets([routed.target_repo])
|
||||
except RegistryError:
|
||||
self._event(
|
||||
"worker.block_rejected",
|
||||
peer_repo=routed.target_repo,
|
||||
provenance="worker_output",
|
||||
outcome="invalid_target",
|
||||
)
|
||||
return
|
||||
self._worker_target = routed.target_repo
|
||||
self._worker_lines = [routed.body]
|
||||
self._event(
|
||||
"worker.block_started",
|
||||
peer_repo=routed.target_repo,
|
||||
provenance="worker_output",
|
||||
outcome="collecting",
|
||||
)
|
||||
|
||||
def flush_worker_message(self) -> RoutedMessage | None:
|
||||
def flush_worker_message(self, reason: str = "explicit_flush") -> RoutedMessage | None:
|
||||
"""Admit the worker block accumulated through its empty terminator."""
|
||||
if self._worker_target is None:
|
||||
return None
|
||||
target = self._worker_target
|
||||
line_count = len(self._worker_lines)
|
||||
routed = RoutedMessage(
|
||||
target_repo=self._worker_target,
|
||||
target_repo=target,
|
||||
body="\n".join(self._worker_lines),
|
||||
)
|
||||
self._worker_target = None
|
||||
self._worker_lines = []
|
||||
return self._admit(routed, "worker_output")
|
||||
admitted = self._admit(routed, "worker_output")
|
||||
self._event(
|
||||
"worker.block_closed",
|
||||
peer_repo=target,
|
||||
message_id=self._last_admitted_message_id,
|
||||
provenance="worker_output",
|
||||
outcome="accepted" if admitted is not None else "not_accepted",
|
||||
detail={"reason": reason, "line_count": line_count},
|
||||
)
|
||||
return admitted
|
||||
|
||||
def inspect_operator_line(self, line: str) -> RoutedMessage | None:
|
||||
"""Observe one submitted operator line; it is still forwarded unchanged."""
|
||||
|
|
@ -117,14 +171,14 @@ class InputBroker:
|
|||
)
|
||||
routed = parse_worker_address_line(line)
|
||||
if routed is not None:
|
||||
previous = self.flush_worker_message()
|
||||
previous = self.flush_worker_message("new_address")
|
||||
self._start_worker_message(routed)
|
||||
return previous
|
||||
if self._worker_target is None:
|
||||
return None
|
||||
content = worker_content_line(line).strip()
|
||||
if not content:
|
||||
return self.flush_worker_message()
|
||||
return self.flush_worker_message("empty_line")
|
||||
self._worker_lines.append(content)
|
||||
return None
|
||||
|
||||
|
|
@ -137,9 +191,20 @@ class InputBroker:
|
|||
self.store.reset_limits(
|
||||
self.identity.endpoint_id, self.identity.source_repo
|
||||
)
|
||||
self._event(
|
||||
"command.applied",
|
||||
provenance="operator_input",
|
||||
outcome="applied",
|
||||
detail={"command": "reset-limits"},
|
||||
)
|
||||
self._notice("Limits reset for this terminal.")
|
||||
return
|
||||
if "=" not in command:
|
||||
self._event(
|
||||
"command.rejected",
|
||||
provenance="operator_input",
|
||||
outcome="unknown_command",
|
||||
)
|
||||
self._notice(f"Unknown command: {command}")
|
||||
return
|
||||
name, value = (part.strip() for part in command.split("=", 1))
|
||||
|
|
@ -147,11 +212,32 @@ class InputBroker:
|
|||
if name == "mode":
|
||||
value = value.casefold()
|
||||
if value not in {"inbox", "output", "pushy", "trigger"}:
|
||||
self._event(
|
||||
"command.rejected",
|
||||
provenance="operator_input",
|
||||
outcome="invalid_value",
|
||||
detail={"command": "mode"},
|
||||
)
|
||||
self._notice(f"Invalid mode: {value}")
|
||||
return
|
||||
if not self.store.set_endpoint_mode(self.identity.endpoint_id, value):
|
||||
self._event(
|
||||
"command.rejected",
|
||||
provenance="operator_input",
|
||||
outcome="endpoint_missing",
|
||||
detail={"command": "mode"},
|
||||
)
|
||||
self._notice("Cannot change mode before endpoint registration.")
|
||||
return
|
||||
self.store.record_protocol_event(
|
||||
"command.applied",
|
||||
endpoint_id=self.identity.endpoint_id,
|
||||
repo=self.identity.source_repo,
|
||||
provenance="operator_input",
|
||||
delivery_mode=value,
|
||||
outcome="applied",
|
||||
detail={"command": "mode", "value": value},
|
||||
)
|
||||
self._notice(f"Mode set to {value}.")
|
||||
return
|
||||
if name in {"maxmsg", "maxin", "maxout"}:
|
||||
|
|
@ -164,21 +250,50 @@ class InputBroker:
|
|||
parsed,
|
||||
)
|
||||
except ValueError as exc:
|
||||
self._event(
|
||||
"command.rejected",
|
||||
provenance="operator_input",
|
||||
outcome="invalid_value",
|
||||
detail={"command": name},
|
||||
)
|
||||
self._notice(str(exc))
|
||||
return
|
||||
self._event(
|
||||
"command.applied",
|
||||
provenance="operator_input",
|
||||
outcome="applied",
|
||||
detail={"command": name, "value": parsed},
|
||||
)
|
||||
self._notice(f"{name} set to {parsed} for this terminal.")
|
||||
return
|
||||
self._event(
|
||||
"command.rejected",
|
||||
provenance="operator_input",
|
||||
outcome="unknown_command",
|
||||
)
|
||||
self._notice(f"Unknown command: {name}")
|
||||
|
||||
def deliver_pending(self, control, *, window_for_repo):
|
||||
"""Place pending messages through the authoritative control client."""
|
||||
delivered = 0
|
||||
for row in self.store.list(state="pending"):
|
||||
endpoint = self.store.endpoint(self.identity.endpoint_id)
|
||||
repos = (
|
||||
json.loads(endpoint["repos"])
|
||||
if endpoint is not None
|
||||
else [row["target_repo"] for row in self.store.list()]
|
||||
)
|
||||
for row in self.store.deliverable(self.identity.endpoint_id, repos):
|
||||
if row["endpoint_id"] not in (None, self.identity.endpoint_id):
|
||||
continue
|
||||
target = window_for_repo(row["target_repo"])
|
||||
lease_id = self.store.claim(row["message_id"], self.identity.endpoint_id)
|
||||
if lease_id is None:
|
||||
claim = self.store.claim_delivery(
|
||||
row["message_id"],
|
||||
self.identity.endpoint_id,
|
||||
attempt_limit=(int(endpoint["delivery_max_attempts"]) if endpoint is not None else 4),
|
||||
retry_delay=5.0,
|
||||
delivery_mode="pane",
|
||||
)
|
||||
if claim is None:
|
||||
continue
|
||||
try:
|
||||
control.inject(
|
||||
|
|
@ -187,8 +302,20 @@ class InputBroker:
|
|||
row["sender_repo"], row["body"], row["provenance"]
|
||||
),
|
||||
)
|
||||
except Exception:
|
||||
except Exception as exc:
|
||||
self.store.fail_delivery(
|
||||
row["message_id"],
|
||||
claim.lease_id,
|
||||
type(exc).__name__,
|
||||
delivery_mode="pane",
|
||||
)
|
||||
continue
|
||||
self.store.release(row["message_id"], lease_id, "injected")
|
||||
delivered += 1
|
||||
if self.store.complete_delivery(
|
||||
row["message_id"],
|
||||
claim.lease_id,
|
||||
delivery_mode="pane",
|
||||
ack_mode=(endpoint["delivery_ack_mode"] if endpoint is not None else "injected"),
|
||||
ack_timeout=(float(endpoint["ack_timeout_seconds"]) if endpoint is not None else 30.0),
|
||||
):
|
||||
delivered += 1
|
||||
return delivered
|
||||
|
|
|
|||
163
src/tamq/capture.py
Normal file
163
src/tamq/capture.py
Normal file
|
|
@ -0,0 +1,163 @@
|
|||
"""Render the structured TAMQ protocol ledger for review."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from collections import Counter
|
||||
from datetime import datetime, timezone
|
||||
from typing import Iterable, Mapping
|
||||
|
||||
|
||||
def _timestamp(value: float) -> str:
|
||||
return datetime.fromtimestamp(value, timezone.utc).isoformat(timespec="milliseconds")
|
||||
|
||||
|
||||
def _safe_text(value: str) -> str:
|
||||
"""Keep report structure readable without treating terminal controls as evidence."""
|
||||
return "".join(
|
||||
character
|
||||
for character in value
|
||||
if character in "\n\t" or ord(character) >= 32
|
||||
)
|
||||
|
||||
|
||||
def event_record(row: Mapping) -> dict:
|
||||
"""Build a stable JSON record and include a body only at queue acceptance."""
|
||||
record = {
|
||||
"sequence": row["sequence"],
|
||||
"event_id": row["event_id"],
|
||||
"occurred_at": _timestamp(float(row["occurred_at"])),
|
||||
"event_type": row["event_type"],
|
||||
"endpoint_id": row["endpoint_id"],
|
||||
"repo": row["repo"],
|
||||
"peer_repo": row["peer_repo"],
|
||||
"message_id": row["message_id"],
|
||||
"provenance": row["provenance"],
|
||||
"delivery_mode": row["delivery_mode"],
|
||||
"outcome": row["outcome"],
|
||||
"detail": json.loads(row["detail"]),
|
||||
}
|
||||
if row["event_type"] == "message.accepted" and row["message_body"] is not None:
|
||||
record["message_body"] = _safe_text(row["message_body"])
|
||||
return record
|
||||
|
||||
|
||||
def render_jsonl(rows: Iterable[Mapping]) -> str:
|
||||
return "".join(
|
||||
json.dumps(event_record(row), sort_keys=True) + "\n" for row in rows
|
||||
)
|
||||
|
||||
|
||||
def render_markdown(rows: Iterable[Mapping], *, scope: str = "all repositories") -> str:
|
||||
events = list(rows)
|
||||
types = Counter(row["event_type"] for row in events)
|
||||
outcomes = Counter(row["outcome"] for row in events)
|
||||
provenances = Counter(
|
||||
row["provenance"] for row in events if row["provenance"] is not None
|
||||
)
|
||||
generated = datetime.now(timezone.utc).isoformat(timespec="seconds")
|
||||
lines = [
|
||||
"# TAMQ protocol capture",
|
||||
"",
|
||||
f"Generated: `{generated}` ",
|
||||
f"Scope: {scope} ",
|
||||
f"Events: {len(events)}",
|
||||
"",
|
||||
"This report contains addressed TAMQ messages and structured lifecycle",
|
||||
"metadata. It does not contain unrelated pane output or shell input.",
|
||||
"",
|
||||
"## Summary",
|
||||
"",
|
||||
f"- Accepted messages: {types['message.accepted']}",
|
||||
f"- Delivery attempts: {types['delivery.attempted']}",
|
||||
f"- Delivery failures: {types['delivery.failed']}",
|
||||
f"- Limit blocks: {types['message.blocked']}",
|
||||
]
|
||||
if provenances:
|
||||
lines.append(
|
||||
"- Message provenance: "
|
||||
+ ", ".join(f"{name}={count}" for name, count in sorted(provenances.items()))
|
||||
)
|
||||
if outcomes:
|
||||
lines.append(
|
||||
"- Outcomes: "
|
||||
+ ", ".join(f"{name}={count}" for name, count in sorted(outcomes.items()))
|
||||
)
|
||||
close_reasons = Counter()
|
||||
for row in events:
|
||||
if row["event_type"] != "worker.block_closed":
|
||||
continue
|
||||
reason = json.loads(row["detail"]).get("reason")
|
||||
if reason:
|
||||
close_reasons[reason] += 1
|
||||
lines.extend(["", "## Review cues", ""])
|
||||
cues: list[str] = []
|
||||
open_blocks = types["worker.block_started"] - types["worker.block_closed"]
|
||||
if open_blocks > 0:
|
||||
cues.append(
|
||||
f"{open_blocks} worker block(s) started without a captured close; review "
|
||||
"blank-line termination and worker shutdown handling."
|
||||
)
|
||||
non_blank_closes = close_reasons["new_address"] + close_reasons["worker_exit"]
|
||||
if non_blank_closes:
|
||||
cues.append(
|
||||
f"{non_blank_closes} worker block(s) closed on a new address or worker exit; "
|
||||
"an explicit blank-line example may improve onboarding."
|
||||
)
|
||||
if types["command.rejected"]:
|
||||
cues.append(
|
||||
f"{types['command.rejected']} operator command(s) were rejected; review command discoverability."
|
||||
)
|
||||
if types["message.rejected"] + types["worker.block_rejected"]:
|
||||
cues.append(
|
||||
f"{types['message.rejected'] + types['worker.block_rejected']} address(es) were rejected; "
|
||||
"review repository-slug discovery."
|
||||
)
|
||||
if types["message.blocked"]:
|
||||
cues.append(
|
||||
f"{types['message.blocked']} message(s) hit a line budget; review limits and reset guidance."
|
||||
)
|
||||
if types["delivery.failed"]:
|
||||
cues.append(
|
||||
f"{types['delivery.failed']} delivery attempt(s) failed; group JSONL by mode and failure reason."
|
||||
)
|
||||
if not cues:
|
||||
cues.append(
|
||||
"No rejected commands, invalid addresses, limit blocks, unclosed worker blocks, "
|
||||
"or delivery failures are visible in this selection."
|
||||
)
|
||||
lines.extend(f"- {cue}" for cue in cues)
|
||||
lines.extend(["", "## Timeline", ""])
|
||||
if not events:
|
||||
lines.extend(["No matching protocol events.", ""])
|
||||
return "\n".join(lines)
|
||||
|
||||
for row in events:
|
||||
lines.append(
|
||||
f"### {_timestamp(float(row['occurred_at']))} — `{row['event_type']}`"
|
||||
)
|
||||
lines.append("")
|
||||
fields = (
|
||||
("Outcome", row["outcome"]),
|
||||
("Repository", row["repo"]),
|
||||
("Counterparty", row["peer_repo"]),
|
||||
("Provenance", row["provenance"]),
|
||||
("Mode", row["delivery_mode"]),
|
||||
("Endpoint", row["endpoint_id"]),
|
||||
("Message", row["message_id"]),
|
||||
("Event", row["event_id"]),
|
||||
)
|
||||
for label, value in fields:
|
||||
if value is not None:
|
||||
lines.append(f"- {label}: `{_safe_text(str(value))}`")
|
||||
detail = json.loads(row["detail"])
|
||||
if detail:
|
||||
lines.append(
|
||||
"- Detail: `" + _safe_text(json.dumps(detail, sort_keys=True)) + "`"
|
||||
)
|
||||
if row["event_type"] == "message.accepted" and row["message_body"] is not None:
|
||||
lines.extend(["", "Message body:", ""])
|
||||
body_lines = _safe_text(row["message_body"]).splitlines() or [""]
|
||||
lines.extend(f" {line}" for line in body_lines)
|
||||
lines.append("")
|
||||
return "\n".join(lines)
|
||||
142
src/tamq/cli.py
142
src/tamq/cli.py
|
|
@ -9,6 +9,7 @@ import os
|
|||
import signal
|
||||
import sqlite3
|
||||
import fcntl
|
||||
import hashlib
|
||||
from uuid import uuid4
|
||||
import subprocess
|
||||
import sys
|
||||
|
|
@ -20,6 +21,9 @@ from . import __version__
|
|||
from .config import db_path, pid_path, lock_path, config_path, socket_path, state_dir, setting
|
||||
from .service import (
|
||||
LINE_LIMITS_CAPABILITY,
|
||||
PROTOCOL_CAPTURE_CAPABILITY,
|
||||
DELIVERY_RELIABILITY_CAPABILITY,
|
||||
IDEMPOTENT_SEND_CAPABILITY,
|
||||
PUSHY_FRAMING_CAPABILITY,
|
||||
TRIGGER_CAPABILITY,
|
||||
Service,
|
||||
|
|
@ -37,10 +41,11 @@ from .policy import load_profile
|
|||
from .registry import RegistryError, validate_targets
|
||||
from .routing import parse_address_line
|
||||
from .terminal import format_delivery
|
||||
from .capture import render_jsonl, render_markdown
|
||||
|
||||
|
||||
SUBCOMMANDS = frozenset(
|
||||
"start attach serve stop cleanup status ping inbox history inspect ack send export replay purge completion db-version config tap".split()
|
||||
"start attach serve stop cleanup status ping inbox history capture inspect ack retry send export replay purge completion db-version config tap".split()
|
||||
)
|
||||
START_OPTIONS = frozenset({"--command", "--cmd", "--tap", "--detach", "--no-service", "--no-display", "--mode", "--maxmsg", "--maxin", "--maxout"})
|
||||
GLOBAL_FLAGS = frozenset({"--orwell", "--verbose"})
|
||||
|
|
@ -131,13 +136,28 @@ def ensure_service_capabilities(required: set[str]) -> bool:
|
|||
|
||||
def ensure_manual_service() -> bool:
|
||||
return ensure_service_capabilities(
|
||||
{"manual_delivery", PUSHY_FRAMING_CAPABILITY, LINE_LIMITS_CAPABILITY}
|
||||
{
|
||||
"manual_delivery",
|
||||
PUSHY_FRAMING_CAPABILITY,
|
||||
LINE_LIMITS_CAPABILITY,
|
||||
PROTOCOL_CAPTURE_CAPABILITY,
|
||||
DELIVERY_RELIABILITY_CAPABILITY,
|
||||
IDEMPOTENT_SEND_CAPABILITY,
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def ensure_output_service() -> bool:
|
||||
return ensure_service_capabilities(
|
||||
{"manual_delivery", "terminal_output", PUSHY_FRAMING_CAPABILITY, LINE_LIMITS_CAPABILITY}
|
||||
{
|
||||
"manual_delivery",
|
||||
"terminal_output",
|
||||
PUSHY_FRAMING_CAPABILITY,
|
||||
LINE_LIMITS_CAPABILITY,
|
||||
PROTOCOL_CAPTURE_CAPABILITY,
|
||||
DELIVERY_RELIABILITY_CAPABILITY,
|
||||
IDEMPOTENT_SEND_CAPABILITY,
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
|
|
@ -149,6 +169,9 @@ def ensure_pushy_service() -> bool:
|
|||
PUSHY_FRAMING_CAPABILITY,
|
||||
TRIGGER_CAPABILITY,
|
||||
LINE_LIMITS_CAPABILITY,
|
||||
PROTOCOL_CAPTURE_CAPABILITY,
|
||||
DELIVERY_RELIABILITY_CAPABILITY,
|
||||
IDEMPOTENT_SEND_CAPABILITY,
|
||||
}
|
||||
)
|
||||
|
||||
|
|
@ -202,6 +225,7 @@ duplex messaging from a managed terminal:
|
|||
Cmd: mode=trigger operator-only runtime command
|
||||
Cmd: reset-limits reset this window's running counters
|
||||
tamq inbox [--filter COMMAND]
|
||||
tamq capture --output tamq-protocol.md
|
||||
""",
|
||||
)
|
||||
parser.add_argument("--version", "-V", action="version", version=__version__)
|
||||
|
|
@ -241,6 +265,27 @@ duplex messaging from a managed terminal:
|
|||
history = subparsers.add_parser("history", help="inspect local message history")
|
||||
history.add_argument("--repo", dest="target_repo")
|
||||
history.add_argument("--state")
|
||||
capture = subparsers.add_parser(
|
||||
"capture",
|
||||
help="render the structured communication protocol ledger for review",
|
||||
)
|
||||
capture.add_argument("--repo", help="include events where this repository is either party")
|
||||
capture.add_argument("--endpoint-id", help="include only one endpoint instance")
|
||||
capture.add_argument("--message-id", help="include only one durable message lifecycle")
|
||||
capture.add_argument("--event", dest="event_type", help="include only one event type")
|
||||
capture.add_argument(
|
||||
"--format",
|
||||
choices=("markdown", "jsonl"),
|
||||
default="markdown",
|
||||
help="review format (default: markdown)",
|
||||
)
|
||||
capture.add_argument(
|
||||
"--limit",
|
||||
type=int,
|
||||
default=500,
|
||||
help="newest matching events to include (default: 500)",
|
||||
)
|
||||
capture.add_argument("--output", type=Path, help="write to a file instead of stdout")
|
||||
inbox = subparsers.add_parser("inbox", help="show durable messages for a repository without injecting terminal input")
|
||||
inbox.add_argument("--repo", dest="target_repo", help="target repository (default: TAMQ_REPO in a managed window)")
|
||||
inbox.add_argument("--all", action="store_true", help="include non-pending messages")
|
||||
|
|
@ -254,6 +299,8 @@ duplex messaging from a managed terminal:
|
|||
inspect.add_argument("message_id")
|
||||
ack = subparsers.add_parser("ack", help="acknowledge one durable message")
|
||||
ack.add_argument("message_id")
|
||||
retry = subparsers.add_parser("retry", help="reset one terminally failed message for bounded redelivery")
|
||||
retry.add_argument("message_id")
|
||||
send = subparsers.add_parser("send", help="queue a direct message")
|
||||
send.add_argument("address", help="To:repo: message")
|
||||
send.add_argument("body", nargs="*", help="message body when address is a repo slug")
|
||||
|
|
@ -366,7 +413,7 @@ def main(argv: list[str] | None = None) -> int:
|
|||
except ValueError as exc:
|
||||
print(f"tamq: {exc}", file=sys.stderr)
|
||||
return 2
|
||||
print(json.dumps({"config": str(config_path()), "state_dir": str(state_dir()), "database": str(db_path()), "socket": str(socket_path()), "pidfile": str(pid_path()), "lockfile": str(lock_path()), "policy_profile": profile.name, "delivery_ack_mode": profile.delivery_ack_mode, **configured_limits}, sort_keys=True))
|
||||
print(json.dumps({"config": str(config_path()), "state_dir": str(state_dir()), "database": str(db_path()), "socket": str(socket_path()), "pidfile": str(pid_path()), "lockfile": str(lock_path()), "policy_profile": profile.name, "delivery_ack_mode": profile.delivery_ack_mode, "delivery_max_attempts": profile.delivery_max_attempts, "ack_timeout_seconds": float(setting("delivery_ack_timeout_seconds", "30")), "delivery_retry_backoff_seconds": setting("delivery_retry_backoff_seconds", "5,15,60,300"), **configured_limits}, sort_keys=True))
|
||||
return 0
|
||||
if args.command == "tap":
|
||||
command = list(args.wrapped_command)
|
||||
|
|
@ -465,7 +512,7 @@ def main(argv: list[str] | None = None) -> int:
|
|||
if not args.no_service:
|
||||
try:
|
||||
delivery_mode = "pane" if args.tap else endpoint_delivery_mode
|
||||
registration = asyncio.run(request({"op": "register", "endpoint_id": endpoint.endpoint_id, "instance_id": endpoint.instance_key, "pid": endpoint.pid, "session": endpoint.session, "repos": endpoint.repos, "delivery_mode": delivery_mode, "maxmsg": limits[0], "maxin": limits[1], "maxout": limits[2]}))
|
||||
registration = asyncio.run(request({"op": "register", "endpoint_id": endpoint.endpoint_id, "instance_id": endpoint.instance_key, "pid": endpoint.pid, "session": endpoint.session, "repos": endpoint.repos, "delivery_mode": delivery_mode, "maxmsg": limits[0], "maxin": limits[1], "maxout": limits[2], "policy_profile": profile.name, "delivery_ack_mode": profile.delivery_ack_mode, "delivery_max_attempts": profile.delivery_max_attempts, "ack_timeout_seconds": float(setting("delivery_ack_timeout_seconds", "30"))}))
|
||||
if not registration.get("ok"):
|
||||
raise RuntimeError(f"endpoint registration failed: {registration.get('error', 'unknown error')}")
|
||||
registered = True
|
||||
|
|
@ -553,7 +600,7 @@ def main(argv: list[str] | None = None) -> int:
|
|||
if args.filter_command and (args.all or args.json):
|
||||
print("tamq: --filter cannot be combined with --all or --json", file=sys.stderr)
|
||||
return 2
|
||||
rows = store.list(target, None if args.all else "pending")
|
||||
rows = store.inbox(target, include_terminal=args.all)
|
||||
for row in rows:
|
||||
if args.json:
|
||||
print(json.dumps(dict(row), sort_keys=True))
|
||||
|
|
@ -587,6 +634,43 @@ def main(argv: list[str] | None = None) -> int:
|
|||
if args.command == "history":
|
||||
for row in store.list(args.target_repo, args.state): print(json.dumps(dict(row), sort_keys=True))
|
||||
return 0
|
||||
if args.command == "capture":
|
||||
try:
|
||||
rows = store.protocol_events(
|
||||
repo=args.repo,
|
||||
endpoint_id=args.endpoint_id,
|
||||
message_id=args.message_id,
|
||||
event_type=args.event_type,
|
||||
limit=args.limit,
|
||||
)
|
||||
except ValueError as exc:
|
||||
print(f"tamq: {exc}", file=sys.stderr)
|
||||
return 2
|
||||
scope_parts = []
|
||||
if args.repo:
|
||||
scope_parts.append(f"repository `{args.repo}`")
|
||||
if args.endpoint_id:
|
||||
scope_parts.append(f"endpoint `{args.endpoint_id}`")
|
||||
if args.message_id:
|
||||
scope_parts.append(f"message `{args.message_id}`")
|
||||
if args.event_type:
|
||||
scope_parts.append(f"event `{args.event_type}`")
|
||||
rendered = (
|
||||
render_jsonl(rows)
|
||||
if args.format == "jsonl"
|
||||
else render_markdown(rows, scope=", ".join(scope_parts) or "all repositories")
|
||||
)
|
||||
if args.output is None:
|
||||
print(rendered, end="" if rendered.endswith("\n") else "\n")
|
||||
return 0
|
||||
try:
|
||||
args.output.parent.mkdir(parents=True, exist_ok=True)
|
||||
args.output.write_text(rendered, encoding="utf-8")
|
||||
except OSError as exc:
|
||||
print(f"tamq: cannot write capture: {exc}", file=sys.stderr)
|
||||
return 1
|
||||
print(f"Wrote {len(rows)} protocol event(s) to {args.output}")
|
||||
return 0
|
||||
if args.command == "inspect":
|
||||
rows = [row for row in store.list() if row["message_id"] == args.message_id]
|
||||
if not rows:
|
||||
|
|
@ -597,11 +681,16 @@ def main(argv: list[str] | None = None) -> int:
|
|||
if not store.acknowledge(args.message_id):
|
||||
print("tamq: message not found", file=sys.stderr); return 1
|
||||
print(args.message_id); return 0
|
||||
if args.command == "retry":
|
||||
if not store.retry_message(args.message_id):
|
||||
print("tamq: message is not failed or was not found", file=sys.stderr); return 1
|
||||
print(args.message_id); return 0
|
||||
if args.command == "export":
|
||||
store.export(store.list(args.target_repo, args.state), Path(args.output)); return 0
|
||||
if args.command == "replay":
|
||||
batch_id = f"replay-{uuid4()}"
|
||||
count = 0
|
||||
deduplicated = 0
|
||||
if args.endpoint_id and store.endpoint(args.endpoint_id) is None:
|
||||
print(f"tamq: endpoint is not registered: {args.endpoint_id}", file=sys.stderr)
|
||||
return 1
|
||||
|
|
@ -611,14 +700,46 @@ def main(argv: list[str] | None = None) -> int:
|
|||
if not line.strip():
|
||||
continue
|
||||
item = json.loads(line)
|
||||
validate_targets([item["target_repo"]])
|
||||
if item.get("sender_repo", "replay") != "local":
|
||||
validate_targets([item.get("sender_repo", "replay")])
|
||||
provenance = json.dumps({"batch_id": batch_id, "original_message_id": item.get("message_id")}, sort_keys=True)
|
||||
endpoint = args.endpoint_id or item.get("endpoint_id")
|
||||
store.add(item.get("sender_repo", "replay"), item["target_repo"], item["body"], endpoint=endpoint, provenance=provenance)
|
||||
replay_key = str(
|
||||
item.get("message_id")
|
||||
or hashlib.sha256(
|
||||
json.dumps(
|
||||
{
|
||||
"sender_repo": item.get("sender_repo", "replay"),
|
||||
"target_repo": item["target_repo"],
|
||||
"body": item["body"],
|
||||
},
|
||||
sort_keys=True,
|
||||
).encode()
|
||||
).hexdigest()
|
||||
)
|
||||
if store.message_by_idempotency("tamq-replay", replay_key):
|
||||
deduplicated += 1
|
||||
continue
|
||||
store.add(
|
||||
item.get("sender_repo", "replay"),
|
||||
item["target_repo"],
|
||||
item["body"],
|
||||
endpoint=endpoint,
|
||||
provenance=provenance,
|
||||
client_id="tamq-replay",
|
||||
idempotency_key=replay_key,
|
||||
correlation_id=item.get("correlation_id"),
|
||||
metadata={
|
||||
"batch_id": batch_id,
|
||||
"original_message_id": item.get("message_id"),
|
||||
},
|
||||
)
|
||||
count += 1
|
||||
except (OSError, KeyError, TypeError, ValueError, json.JSONDecodeError) as exc:
|
||||
except (OSError, KeyError, TypeError, ValueError, RegistryError, json.JSONDecodeError) as exc:
|
||||
print(f"tamq: cannot replay {args.file}: {exc}", file=sys.stderr)
|
||||
return 2
|
||||
print(json.dumps({"batch_id": batch_id, "count": count})); return 0
|
||||
print(json.dumps({"batch_id": batch_id, "count": count, "deduplicated": deduplicated})); return 0
|
||||
if args.command == "purge":
|
||||
if args.feedback_chain:
|
||||
if args.before or args.max_size:
|
||||
|
|
@ -653,7 +774,8 @@ def main(argv: list[str] | None = None) -> int:
|
|||
if args.command == "status":
|
||||
live = asyncio.run(ping())
|
||||
endpoints = asyncio.run(request({"op": "endpoints"})) if live else {"endpoints": []}
|
||||
print(json.dumps({"version": __version__, "db": str(db_path()), "messages": len(store.list()), "pending": len(store.list(state="pending")), "leases": store.db.execute("SELECT COUNT(*) FROM leases").fetchone()[0], "service": live, "policy_profile": profile.name, "delivery_ack_mode": profile.delivery_ack_mode, "endpoints": endpoints.get("endpoints", []), "window_counters": [dict(row) for row in store.counters()]}, sort_keys=True)); return 0
|
||||
states = {state: count for state, count in store.db.execute("SELECT state,COUNT(*) FROM messages GROUP BY state")}
|
||||
print(json.dumps({"version": __version__, "db": str(db_path()), "messages": len(store.list()), "pending": states.get("pending", 0), "awaiting_ack": states.get("awaiting_ack", 0), "failed": states.get("failed", 0), "message_states": states, "leases": store.db.execute("SELECT COUNT(*) FROM leases").fetchone()[0], "service": live, "policy_profile": profile.name, "delivery_ack_mode": profile.delivery_ack_mode, "delivery_max_attempts": profile.delivery_max_attempts, "endpoints": endpoints.get("endpoints", []), "window_counters": [dict(row) for row in store.counters()]}, sort_keys=True)); return 0
|
||||
print(f"tamq {__version__}: command '{args.command}' is not implemented yet", file=sys.stderr)
|
||||
return 2
|
||||
finally:
|
||||
|
|
|
|||
223
src/tamq/client.py
Normal file
223
src/tamq/client.py
Normal file
|
|
@ -0,0 +1,223 @@
|
|||
"""Transport-only Unix-socket client for TAMQ integrations."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from .config import socket_path
|
||||
from .protocol import (
|
||||
DELIVERY_RELIABILITY_CAPABILITY,
|
||||
IDEMPOTENT_SEND_CAPABILITY,
|
||||
PROTOCOL_VERSION,
|
||||
)
|
||||
|
||||
|
||||
class TamqClientError(RuntimeError):
|
||||
"""The local TAMQ service rejected or could not complete an operation."""
|
||||
|
||||
|
||||
class TamqProtocolError(TamqClientError):
|
||||
"""The service and client do not share a compatible protocol contract."""
|
||||
|
||||
|
||||
class TamqTargetUnavailable(TamqClientError):
|
||||
"""No unambiguous live endpoint owns the requested repository."""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class WakeRequest:
|
||||
lease_id: str
|
||||
target_repo: str
|
||||
prompt: str
|
||||
trigger_id: str
|
||||
source_repo: str = "coordination-engine"
|
||||
endpoint_id: str | None = None
|
||||
target_agent: str | None = None
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class WakeReceipt:
|
||||
lease_id: str
|
||||
trigger_id: str
|
||||
endpoint_id: str
|
||||
message_id: str
|
||||
state: str
|
||||
deduplicated: bool
|
||||
|
||||
|
||||
class TamqClient:
|
||||
"""Small async client with no tmux/control-mode dependency."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
path: Path | None = None,
|
||||
*,
|
||||
client_id: str = "coordination-engine",
|
||||
protocol: str = PROTOCOL_VERSION,
|
||||
timeout: float = 5.0,
|
||||
):
|
||||
self.path = path or socket_path()
|
||||
self.client_id = client_id
|
||||
self.protocol = protocol
|
||||
self.timeout = timeout
|
||||
|
||||
async def request(self, operation: str, **fields: Any) -> dict:
|
||||
payload = {"op": operation, "protocol": self.protocol, **fields}
|
||||
try:
|
||||
reader, writer = await asyncio.wait_for(
|
||||
asyncio.open_unix_connection(str(self.path), limit=1024 * 1024),
|
||||
timeout=self.timeout,
|
||||
)
|
||||
try:
|
||||
writer.write((json.dumps(payload, sort_keys=True) + "\n").encode())
|
||||
await asyncio.wait_for(writer.drain(), timeout=self.timeout)
|
||||
line = await asyncio.wait_for(reader.readline(), timeout=self.timeout)
|
||||
finally:
|
||||
writer.close()
|
||||
await writer.wait_closed()
|
||||
except (OSError, asyncio.TimeoutError) as exc:
|
||||
raise TamqClientError(f"TAMQ service unavailable: {exc}") from exc
|
||||
try:
|
||||
response = json.loads(line)
|
||||
except (json.JSONDecodeError, UnicodeDecodeError) as exc:
|
||||
raise TamqProtocolError("TAMQ returned an invalid response") from exc
|
||||
if not response.get("ok"):
|
||||
error = str(response.get("error") or "operation failed")
|
||||
if error.startswith("incompatible protocol"):
|
||||
raise TamqProtocolError(error)
|
||||
raise TamqClientError(error)
|
||||
return response
|
||||
|
||||
async def negotiate(self) -> dict:
|
||||
response = await self.request("ping")
|
||||
server_protocol = str(response.get("protocol", ""))
|
||||
if server_protocol.split(".")[0] != self.protocol.split(".")[0]:
|
||||
raise TamqProtocolError(
|
||||
f"incompatible protocol: client {self.protocol}, server {server_protocol}"
|
||||
)
|
||||
required = {DELIVERY_RELIABILITY_CAPABILITY, IDEMPOTENT_SEND_CAPABILITY}
|
||||
missing = required.difference(response.get("capabilities", []))
|
||||
if missing:
|
||||
raise TamqProtocolError(
|
||||
"TAMQ lacks required capabilities: " + ", ".join(sorted(missing))
|
||||
)
|
||||
return response
|
||||
|
||||
async def send(
|
||||
self,
|
||||
*,
|
||||
sender_repo: str,
|
||||
target_repo: str,
|
||||
body: str,
|
||||
idempotency_key: str,
|
||||
endpoint_id: str | None = None,
|
||||
correlation_id: str | None = None,
|
||||
metadata: dict | None = None,
|
||||
provenance: str = "coordination_engine",
|
||||
) -> dict:
|
||||
fields: dict[str, Any] = {
|
||||
"sender_repo": sender_repo,
|
||||
"target_repo": target_repo,
|
||||
"body": body,
|
||||
"client_id": self.client_id,
|
||||
"idempotency_key": idempotency_key,
|
||||
"provenance": provenance,
|
||||
}
|
||||
if endpoint_id is not None:
|
||||
fields["endpoint_id"] = endpoint_id
|
||||
if correlation_id is not None:
|
||||
fields["correlation_id"] = correlation_id
|
||||
if metadata is not None:
|
||||
fields["metadata"] = metadata
|
||||
return await self.request("send", **fields)
|
||||
|
||||
async def message(self, message_id: str) -> dict:
|
||||
return (await self.request("message", message_id=message_id))["message"]
|
||||
|
||||
async def history(
|
||||
self, *, target_repo: str | None = None, state: str | None = None
|
||||
) -> list[dict]:
|
||||
fields = {
|
||||
key: value
|
||||
for key, value in {"target_repo": target_repo, "state": state}.items()
|
||||
if value is not None
|
||||
}
|
||||
return (await self.request("history", **fields))["messages"]
|
||||
|
||||
async def endpoints(self) -> list[dict]:
|
||||
return (await self.request("endpoints"))["endpoints"]
|
||||
|
||||
async def acknowledge(self, message_id: str) -> dict:
|
||||
return await self.request("ack", message_id=message_id)
|
||||
|
||||
async def retry(self, message_id: str) -> dict:
|
||||
return await self.request("retry", message_id=message_id)
|
||||
|
||||
|
||||
class CoordinationEngineAdapter:
|
||||
"""Map coordination-engine wake requests onto durable TAMQ messages."""
|
||||
|
||||
def __init__(self, client: TamqClient | None = None):
|
||||
self.client = client or TamqClient()
|
||||
|
||||
async def _resolve_endpoint(self, target_repo: str, requested: str | None) -> str:
|
||||
endpoints = await self.client.endpoints()
|
||||
matching = [
|
||||
endpoint
|
||||
for endpoint in endpoints
|
||||
if target_repo in json.loads(endpoint["repos"])
|
||||
and (requested is None or endpoint["endpoint_id"] == requested)
|
||||
]
|
||||
if not matching:
|
||||
raise TamqTargetUnavailable(
|
||||
f"no live TAMQ endpoint is attached to {target_repo}"
|
||||
)
|
||||
if len(matching) > 1:
|
||||
raise TamqTargetUnavailable(
|
||||
f"multiple TAMQ endpoints are attached to {target_repo}; endpoint_id is required"
|
||||
)
|
||||
return str(matching[0]["endpoint_id"])
|
||||
|
||||
async def wake(self, request: WakeRequest) -> WakeReceipt:
|
||||
await self.client.negotiate()
|
||||
endpoint_id = await self._resolve_endpoint(
|
||||
request.target_repo, request.endpoint_id
|
||||
)
|
||||
response = await self.client.send(
|
||||
sender_repo=request.source_repo,
|
||||
target_repo=request.target_repo,
|
||||
body=request.prompt,
|
||||
idempotency_key=request.lease_id,
|
||||
endpoint_id=endpoint_id,
|
||||
correlation_id=request.trigger_id,
|
||||
metadata={
|
||||
"lease_id": request.lease_id,
|
||||
"trigger_id": request.trigger_id,
|
||||
**(
|
||||
{"target_agent": request.target_agent}
|
||||
if request.target_agent is not None
|
||||
else {}
|
||||
),
|
||||
},
|
||||
)
|
||||
return WakeReceipt(
|
||||
lease_id=request.lease_id,
|
||||
trigger_id=request.trigger_id,
|
||||
endpoint_id=endpoint_id,
|
||||
message_id=response["message_id"],
|
||||
state=response["state"],
|
||||
deduplicated=bool(response.get("deduplicated")),
|
||||
)
|
||||
|
||||
async def receipt(self, message_id: str) -> dict:
|
||||
return await self.client.message(message_id)
|
||||
|
||||
async def acknowledge(self, message_id: str) -> dict:
|
||||
return await self.client.acknowledge(message_id)
|
||||
|
||||
async def retry_failed(self, message_id: str) -> dict:
|
||||
return await self.client.retry(message_id)
|
||||
|
|
@ -14,6 +14,7 @@ class PolicyProfile:
|
|||
require_human: frozenset[str]
|
||||
safety_gated_max_attempts: int = 0
|
||||
delivery_ack_mode: str = "injected"
|
||||
delivery_max_attempts: int = 4
|
||||
|
||||
|
||||
DEFAULT = PolicyProfile(
|
||||
|
|
@ -40,4 +41,14 @@ def load_profile(path: Path | None = None, selected: str = "default") -> PolicyP
|
|||
ack_mode = profile.get("delivery_ack_mode", "injected")
|
||||
if ack_mode not in {"injected", "acknowledged"}:
|
||||
raise ValueError("delivery_ack_mode must be injected or acknowledged")
|
||||
return PolicyProfile(selected, frozenset(profile.get("allow", [])), frozenset(profile.get("require_human", [])), attempts, ack_mode)
|
||||
delivery_attempts = int(profile.get("delivery_max_attempts", 4))
|
||||
if not 1 <= delivery_attempts <= 9:
|
||||
raise ValueError("delivery_max_attempts must be between 1 and 9")
|
||||
return PolicyProfile(
|
||||
selected,
|
||||
frozenset(profile.get("allow", [])),
|
||||
frozenset(profile.get("require_human", [])),
|
||||
attempts,
|
||||
ack_mode,
|
||||
delivery_attempts,
|
||||
)
|
||||
|
|
|
|||
29
src/tamq/protocol.py
Normal file
29
src/tamq/protocol.py
Normal file
|
|
@ -0,0 +1,29 @@
|
|||
"""Stable local socket protocol identifiers shared by service and clients."""
|
||||
|
||||
PROTOCOL_VERSION = "0.1"
|
||||
PUSHY_FRAMING_CAPABILITY = "readable_duplex_framing_v2"
|
||||
TRIGGER_CAPABILITY = "trigger_input_v2"
|
||||
LINE_LIMITS_CAPABILITY = "session_window_line_limits_v1"
|
||||
PROTOCOL_CAPTURE_CAPABILITY = "structured_protocol_capture_v1"
|
||||
DELIVERY_RELIABILITY_CAPABILITY = "bounded_delivery_ack_v1"
|
||||
IDEMPOTENT_SEND_CAPABILITY = "idempotent_send_v1"
|
||||
|
||||
SERVICE_CAPABILITIES = [
|
||||
"register",
|
||||
"send",
|
||||
"history",
|
||||
"message",
|
||||
"ack",
|
||||
"retry",
|
||||
"endpoints",
|
||||
"disconnect",
|
||||
"manual_delivery",
|
||||
"terminal_output",
|
||||
"pushy_input",
|
||||
PUSHY_FRAMING_CAPABILITY,
|
||||
TRIGGER_CAPABILITY,
|
||||
LINE_LIMITS_CAPABILITY,
|
||||
PROTOCOL_CAPTURE_CAPABILITY,
|
||||
DELIVERY_RELIABILITY_CAPABILITY,
|
||||
IDEMPOTENT_SEND_CAPABILITY,
|
||||
]
|
||||
|
|
@ -300,7 +300,7 @@ class PtyTap:
|
|||
self.output_observer.flush()
|
||||
flush_worker = getattr(self.broker, "flush_worker_message", None)
|
||||
if flush_worker is not None:
|
||||
flush_worker()
|
||||
flush_worker("worker_exit")
|
||||
for signum, handler in saved_handlers.items():
|
||||
signal.signal(signum, handler)
|
||||
if saved_terminal is not None:
|
||||
|
|
|
|||
|
|
@ -15,22 +15,16 @@ from .store import LimitBlock, Store
|
|||
from .registry import RegistryError, validate_targets
|
||||
from .control import ControlModeClient
|
||||
from .terminal import format_delivery, terminal_frame, write_terminal_output
|
||||
|
||||
PROTOCOL_VERSION = "0.1"
|
||||
PUSHY_FRAMING_CAPABILITY = "readable_duplex_framing_v2"
|
||||
TRIGGER_CAPABILITY = "trigger_input_v2"
|
||||
LINE_LIMITS_CAPABILITY = "session_window_line_limits_v1"
|
||||
SERVICE_CAPABILITIES = [
|
||||
"register",
|
||||
"send",
|
||||
"history",
|
||||
"manual_delivery",
|
||||
"terminal_output",
|
||||
"pushy_input",
|
||||
PUSHY_FRAMING_CAPABILITY,
|
||||
TRIGGER_CAPABILITY,
|
||||
from .protocol import (
|
||||
DELIVERY_RELIABILITY_CAPABILITY,
|
||||
IDEMPOTENT_SEND_CAPABILITY,
|
||||
LINE_LIMITS_CAPABILITY,
|
||||
]
|
||||
PROTOCOL_CAPTURE_CAPABILITY,
|
||||
PROTOCOL_VERSION,
|
||||
PUSHY_FRAMING_CAPABILITY,
|
||||
SERVICE_CAPABILITIES,
|
||||
TRIGGER_CAPABILITY,
|
||||
)
|
||||
|
||||
|
||||
class Service:
|
||||
|
|
@ -40,6 +34,8 @@ class Service:
|
|||
store: Store | None = None,
|
||||
poll_interval: float | None = None,
|
||||
input_grace: float | None = None,
|
||||
retry_backoff: tuple[float, ...] | None = None,
|
||||
lease_ttl: float | None = None,
|
||||
):
|
||||
self.path = path or socket_path()
|
||||
self.store = store or Store(db_path())
|
||||
|
|
@ -59,6 +55,22 @@ class Service:
|
|||
)
|
||||
except ValueError:
|
||||
self.input_grace = max(0.0, input_grace or 1.0)
|
||||
configured_backoff = setting("delivery_retry_backoff_seconds", "5,15,60,300")
|
||||
try:
|
||||
parsed_backoff = tuple(float(item.strip()) for item in configured_backoff.split(","))
|
||||
if not parsed_backoff or any(value < 0 for value in parsed_backoff):
|
||||
raise ValueError
|
||||
self.retry_backoff = retry_backoff or parsed_backoff
|
||||
except ValueError:
|
||||
self.retry_backoff = retry_backoff or (5.0, 15.0, 60.0, 300.0)
|
||||
try:
|
||||
self.lease_ttl = (
|
||||
max(0.05, lease_ttl)
|
||||
if lease_ttl is not None
|
||||
else max(0.05, float(setting("delivery_lease_seconds", "30")))
|
||||
)
|
||||
except ValueError:
|
||||
self.lease_ttl = max(0.05, lease_ttl or 30.0)
|
||||
|
||||
async def run(self) -> None:
|
||||
self.path.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
|
@ -103,6 +115,7 @@ class Service:
|
|||
continue
|
||||
|
||||
def _deliver_once(self) -> None:
|
||||
self.store.expire_delivery_leases()
|
||||
for endpoint in self.store.endpoints():
|
||||
repos = json.loads(endpoint["repos"])
|
||||
control = ControlModeClient(endpoint["session"])
|
||||
|
|
@ -117,20 +130,26 @@ class Service:
|
|||
and time.time() - float(endpoint["connected_at"]) < self.input_grace
|
||||
):
|
||||
continue
|
||||
pending = [
|
||||
row for row in self.store.list(state="pending")
|
||||
if row["endpoint_id"] in (None, endpoint["endpoint_id"])
|
||||
and row["target_repo"] in repos
|
||||
and (delivery_mode == "pane" or row["displayed_at"] is None)
|
||||
]
|
||||
pending = self.store.deliverable(endpoint["endpoint_id"], repos)
|
||||
if not pending:
|
||||
continue
|
||||
try:
|
||||
if delivery_mode == "pane":
|
||||
control.start()
|
||||
for row in pending:
|
||||
lease = self.store.claim(row["message_id"], endpoint["endpoint_id"])
|
||||
if lease is None:
|
||||
attempt = int(row["attempt_count"]) + 1
|
||||
retry_delay = self.retry_backoff[
|
||||
min(attempt - 1, len(self.retry_backoff) - 1)
|
||||
]
|
||||
claim = self.store.claim_delivery(
|
||||
row["message_id"],
|
||||
endpoint["endpoint_id"],
|
||||
attempt_limit=int(endpoint["delivery_max_attempts"]),
|
||||
retry_delay=retry_delay,
|
||||
delivery_mode=delivery_mode,
|
||||
ttl=self.lease_ttl,
|
||||
)
|
||||
if claim is None:
|
||||
continue
|
||||
try:
|
||||
target = f'{endpoint["session"]}:{row["target_repo"]}'
|
||||
|
|
@ -172,12 +191,21 @@ class Service:
|
|||
row["sender_repo"], row["body"], row["provenance"]
|
||||
),
|
||||
)
|
||||
except Exception:
|
||||
except Exception as exc:
|
||||
self.store.fail_delivery(
|
||||
row["message_id"],
|
||||
claim.lease_id,
|
||||
type(exc).__name__,
|
||||
delivery_mode=delivery_mode,
|
||||
)
|
||||
continue
|
||||
if delivery_mode == "output":
|
||||
self.store.mark_displayed(row["message_id"], lease)
|
||||
else:
|
||||
self.store.release(row["message_id"], lease, "injected")
|
||||
self.store.complete_delivery(
|
||||
row["message_id"],
|
||||
claim.lease_id,
|
||||
delivery_mode=delivery_mode,
|
||||
ack_mode=endpoint["delivery_ack_mode"],
|
||||
ack_timeout=float(endpoint["ack_timeout_seconds"]),
|
||||
)
|
||||
finally:
|
||||
control.close()
|
||||
|
||||
|
|
@ -219,6 +247,10 @@ class Service:
|
|||
int(request.get("maxin", 1024)),
|
||||
int(request.get("maxout", 32768)),
|
||||
),
|
||||
policy_profile=str(request.get("policy_profile", "default")),
|
||||
delivery_ack_mode=str(request.get("delivery_ack_mode", "injected")),
|
||||
delivery_max_attempts=int(request.get("delivery_max_attempts", 4)),
|
||||
ack_timeout_seconds=float(request.get("ack_timeout_seconds", 30)),
|
||||
)
|
||||
response = {"ok": True, "endpoint_id": request["endpoint_id"], "instance_id": endpoint_id, "protocol": PROTOCOL_VERSION}
|
||||
except (RegistryError, ValueError) as exc:
|
||||
|
|
@ -242,7 +274,60 @@ class Service:
|
|||
raise RegistryError("target is not attached to endpoint")
|
||||
endpoint_id = endpoint["endpoint_id"]
|
||||
provenance = request.get("provenance", "operator_input")
|
||||
if endpoint_id and request["sender_repo"] != "local":
|
||||
client_id = request.get("client_id")
|
||||
idempotency_key = request.get("idempotency_key")
|
||||
correlation_id = request.get("correlation_id")
|
||||
metadata = request.get("metadata")
|
||||
if bool(client_id) != bool(idempotency_key):
|
||||
raise ValueError(
|
||||
"client_id and idempotency_key must be provided together"
|
||||
)
|
||||
if client_id and (
|
||||
len(str(client_id)) > 128 or len(str(idempotency_key)) > 256
|
||||
):
|
||||
raise ValueError("client or idempotency identity is too long")
|
||||
if metadata is not None and not isinstance(metadata, dict):
|
||||
raise ValueError("metadata must be a JSON object")
|
||||
existing = (
|
||||
self.store.message_by_idempotency(
|
||||
str(client_id), str(idempotency_key)
|
||||
)
|
||||
if client_id
|
||||
else None
|
||||
)
|
||||
if existing is not None:
|
||||
if (
|
||||
existing["sender_repo"] != request["sender_repo"]
|
||||
or existing["target_repo"] != request["target_repo"]
|
||||
or existing["body"] != request["body"]
|
||||
):
|
||||
raise ValueError(
|
||||
"idempotency key was already used for a different message"
|
||||
)
|
||||
if existing["endpoint_id"] != endpoint_id and existing["state"] in {
|
||||
"pending", "awaiting_ack", "failed"
|
||||
}:
|
||||
old_endpoint = (
|
||||
self.store.endpoint(existing["endpoint_id"])
|
||||
if existing["endpoint_id"]
|
||||
else None
|
||||
)
|
||||
if old_endpoint is not None:
|
||||
raise ValueError(
|
||||
"idempotent message is still bound to another live endpoint"
|
||||
)
|
||||
self.store.rebind_message_endpoint(
|
||||
existing["message_id"], endpoint_id
|
||||
)
|
||||
existing = self.store.message(existing["message_id"])
|
||||
assert existing is not None
|
||||
response = {
|
||||
"ok": True,
|
||||
"message_id": existing["message_id"],
|
||||
"state": existing["state"],
|
||||
"deduplicated": True,
|
||||
}
|
||||
elif endpoint_id and request["sender_repo"] != "local":
|
||||
result = self.store.admit_message(
|
||||
request["sender_repo"],
|
||||
request["target_repo"],
|
||||
|
|
@ -250,6 +335,10 @@ class Service:
|
|||
endpoint=endpoint_id,
|
||||
source_repo=request["sender_repo"],
|
||||
provenance=provenance,
|
||||
client_id=str(client_id) if client_id else None,
|
||||
idempotency_key=str(idempotency_key) if idempotency_key else None,
|
||||
correlation_id=str(correlation_id) if correlation_id else None,
|
||||
metadata=metadata,
|
||||
)
|
||||
if isinstance(result, LimitBlock):
|
||||
response = {
|
||||
|
|
@ -260,7 +349,7 @@ class Service:
|
|||
"blocked": result.dimension,
|
||||
}
|
||||
else:
|
||||
response = {"ok": True, "message_id": result, "state": "pending"}
|
||||
response = {"ok": True, "message_id": result, "state": "pending", "deduplicated": False}
|
||||
else:
|
||||
message_id = self.store.add(
|
||||
request["sender_repo"],
|
||||
|
|
@ -268,18 +357,39 @@ class Service:
|
|||
request["body"],
|
||||
endpoint=endpoint_id,
|
||||
provenance=provenance,
|
||||
client_id=str(client_id) if client_id else None,
|
||||
idempotency_key=str(idempotency_key) if idempotency_key else None,
|
||||
correlation_id=str(correlation_id) if correlation_id else None,
|
||||
metadata=metadata,
|
||||
)
|
||||
response = {"ok": True, "message_id": message_id, "state": "pending"}
|
||||
response = {"ok": True, "message_id": message_id, "state": "pending", "deduplicated": False}
|
||||
except (RegistryError, ValueError) as exc:
|
||||
response = {"ok": False, "error": str(exc)}
|
||||
elif op == "history":
|
||||
response = {"ok": True, "messages": [dict(row) for row in self.store.list(request.get("target_repo"), request.get("state"))]}
|
||||
elif op == "message":
|
||||
row = self.store.message(str(request.get("message_id", "")))
|
||||
response = {
|
||||
"ok": row is not None,
|
||||
"message": dict(row) if row is not None else None,
|
||||
"error": None if row is not None else "message not found",
|
||||
}
|
||||
elif op == "ack":
|
||||
if not request.get("message_id"):
|
||||
response = {"ok": False, "error": "ack requires message_id"}
|
||||
else:
|
||||
ok = self.store.acknowledge(request["message_id"])
|
||||
response = {"ok": ok, "message_id": request["message_id"], "state": "acknowledged" if ok else "missing"}
|
||||
elif op == "retry":
|
||||
if not request.get("message_id"):
|
||||
response = {"ok": False, "error": "retry requires message_id"}
|
||||
else:
|
||||
ok = self.store.retry_message(request["message_id"])
|
||||
response = {
|
||||
"ok": ok,
|
||||
"message_id": request["message_id"],
|
||||
"state": "pending" if ok else "not_failed_or_missing",
|
||||
}
|
||||
elif op == "endpoints":
|
||||
response = {"ok": True, "endpoints": [dict(row) for row in self.store.endpoints()]}
|
||||
elif op == "disconnect":
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@ from pathlib import Path
|
|||
from typing import Iterable
|
||||
from uuid import uuid4
|
||||
|
||||
SCHEMA_VERSION = 4
|
||||
SCHEMA_VERSION = 6
|
||||
RECEIPT_SUFFIX = re.compile(
|
||||
r"\s+\[(m-[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12})\]$",
|
||||
re.IGNORECASE,
|
||||
|
|
@ -23,6 +23,12 @@ class LimitBlock:
|
|||
limit: int
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class DeliveryClaim:
|
||||
lease_id: str
|
||||
attempt: int
|
||||
|
||||
|
||||
class Store:
|
||||
def __init__(self, path: Path):
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
|
@ -44,7 +50,16 @@ class Store:
|
|||
provenance TEXT,
|
||||
injected_at REAL,
|
||||
acknowledged_at REAL,
|
||||
displayed_at REAL
|
||||
displayed_at REAL,
|
||||
delivered_at REAL,
|
||||
attempt_count INTEGER NOT NULL DEFAULT 0,
|
||||
last_attempt_at REAL,
|
||||
last_failure_reason TEXT,
|
||||
next_attempt_at REAL,
|
||||
client_id TEXT,
|
||||
idempotency_key TEXT,
|
||||
correlation_id TEXT,
|
||||
envelope_metadata TEXT NOT NULL DEFAULT '{}'
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS messages_target_state ON messages(target_repo, state);
|
||||
CREATE TABLE IF NOT EXISTS endpoints (
|
||||
|
|
@ -53,6 +68,10 @@ class Store:
|
|||
session TEXT NOT NULL,
|
||||
repos TEXT NOT NULL,
|
||||
delivery_mode TEXT NOT NULL DEFAULT 'manual',
|
||||
policy_profile TEXT NOT NULL DEFAULT 'default',
|
||||
delivery_ack_mode TEXT NOT NULL DEFAULT 'injected',
|
||||
delivery_max_attempts INTEGER NOT NULL DEFAULT 4,
|
||||
ack_timeout_seconds REAL NOT NULL DEFAULT 30,
|
||||
connected_at REAL NOT NULL,
|
||||
disconnected_at REAL
|
||||
);
|
||||
|
|
@ -73,8 +92,30 @@ class Store:
|
|||
lease_id TEXT NOT NULL,
|
||||
endpoint_id TEXT NOT NULL,
|
||||
acquired_at REAL NOT NULL,
|
||||
expires_at REAL NOT NULL
|
||||
expires_at REAL NOT NULL,
|
||||
attempt_limit INTEGER NOT NULL DEFAULT 1,
|
||||
retry_delay REAL NOT NULL DEFAULT 5
|
||||
);
|
||||
CREATE TABLE IF NOT EXISTS protocol_events (
|
||||
sequence INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
event_id TEXT NOT NULL UNIQUE,
|
||||
occurred_at REAL NOT NULL,
|
||||
event_type TEXT NOT NULL,
|
||||
endpoint_id TEXT,
|
||||
repo TEXT,
|
||||
peer_repo TEXT,
|
||||
message_id TEXT,
|
||||
provenance TEXT,
|
||||
delivery_mode TEXT,
|
||||
outcome TEXT NOT NULL,
|
||||
detail TEXT NOT NULL DEFAULT '{}'
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS protocol_events_time
|
||||
ON protocol_events(occurred_at, sequence);
|
||||
CREATE INDEX IF NOT EXISTS protocol_events_message
|
||||
ON protocol_events(message_id, occurred_at);
|
||||
CREATE INDEX IF NOT EXISTS protocol_events_endpoint
|
||||
ON protocol_events(endpoint_id, occurred_at);
|
||||
""")
|
||||
endpoint_columns = {
|
||||
row["name"] for row in self.db.execute("PRAGMA table_info(endpoints)")
|
||||
|
|
@ -83,11 +124,47 @@ class Store:
|
|||
self.db.execute(
|
||||
"ALTER TABLE endpoints ADD COLUMN delivery_mode TEXT NOT NULL DEFAULT 'manual'"
|
||||
)
|
||||
for name, definition in (
|
||||
("policy_profile", "TEXT NOT NULL DEFAULT 'default'"),
|
||||
("delivery_ack_mode", "TEXT NOT NULL DEFAULT 'injected'"),
|
||||
("delivery_max_attempts", "INTEGER NOT NULL DEFAULT 4"),
|
||||
("ack_timeout_seconds", "REAL NOT NULL DEFAULT 30"),
|
||||
):
|
||||
if name not in endpoint_columns:
|
||||
self.db.execute(f"ALTER TABLE endpoints ADD COLUMN {name} {definition}")
|
||||
message_columns = {
|
||||
row["name"] for row in self.db.execute("PRAGMA table_info(messages)")
|
||||
}
|
||||
if "displayed_at" not in message_columns:
|
||||
self.db.execute("ALTER TABLE messages ADD COLUMN displayed_at REAL")
|
||||
for name, definition in (
|
||||
("delivered_at", "REAL"),
|
||||
("attempt_count", "INTEGER NOT NULL DEFAULT 0"),
|
||||
("last_attempt_at", "REAL"),
|
||||
("last_failure_reason", "TEXT"),
|
||||
("next_attempt_at", "REAL"),
|
||||
("client_id", "TEXT"),
|
||||
("idempotency_key", "TEXT"),
|
||||
("correlation_id", "TEXT"),
|
||||
("envelope_metadata", "TEXT NOT NULL DEFAULT '{}'"),
|
||||
):
|
||||
if name not in message_columns:
|
||||
self.db.execute(f"ALTER TABLE messages ADD COLUMN {name} {definition}")
|
||||
lease_columns = {
|
||||
row["name"] for row in self.db.execute("PRAGMA table_info(leases)")
|
||||
}
|
||||
for name, definition in (
|
||||
("attempt_limit", "INTEGER NOT NULL DEFAULT 1"),
|
||||
("retry_delay", "REAL NOT NULL DEFAULT 5"),
|
||||
):
|
||||
if name not in lease_columns:
|
||||
self.db.execute(f"ALTER TABLE leases ADD COLUMN {name} {definition}")
|
||||
self.db.execute(
|
||||
"CREATE UNIQUE INDEX IF NOT EXISTS messages_client_idempotency "
|
||||
"ON messages(client_id,idempotency_key) "
|
||||
"WHERE client_id IS NOT NULL AND idempotency_key IS NOT NULL"
|
||||
)
|
||||
self._backfill_protocol_events()
|
||||
self.db.execute(
|
||||
"INSERT INTO metadata(key,value) VALUES('schema_version',?) "
|
||||
"ON CONFLICT(key) DO UPDATE SET value=excluded.value",
|
||||
|
|
@ -95,6 +172,131 @@ class Store:
|
|||
)
|
||||
self.db.commit()
|
||||
|
||||
def _backfill_protocol_events(self) -> None:
|
||||
"""Represent pre-v5 message history without inventing terminal activity."""
|
||||
self.db.execute(
|
||||
"INSERT OR IGNORE INTO protocol_events("
|
||||
"event_id,occurred_at,event_type,endpoint_id,repo,peer_repo,message_id,"
|
||||
"provenance,delivery_mode,outcome,detail) "
|
||||
"SELECT 'pe-accepted-' || message_id,created_at,'message.accepted',"
|
||||
"endpoint_id,sender_repo,target_repo,message_id,provenance,NULL,'accepted',"
|
||||
"'{\"backfilled\":true}' FROM messages m WHERE NOT EXISTS ("
|
||||
"SELECT 1 FROM protocol_events e WHERE e.message_id=m.message_id "
|
||||
"AND e.event_type='message.accepted')"
|
||||
)
|
||||
self.db.execute(
|
||||
"INSERT OR IGNORE INTO protocol_events("
|
||||
"event_id,occurred_at,event_type,endpoint_id,repo,peer_repo,message_id,"
|
||||
"provenance,delivery_mode,outcome,detail) "
|
||||
"SELECT 'pe-displayed-' || message_id,displayed_at,'delivery.displayed',"
|
||||
"endpoint_id,target_repo,sender_repo,message_id,provenance,'output','displayed',"
|
||||
"'{\"backfilled\":true}' FROM messages m WHERE displayed_at IS NOT NULL "
|
||||
"AND NOT EXISTS (SELECT 1 FROM protocol_events e WHERE "
|
||||
"e.message_id=m.message_id AND e.event_type='delivery.displayed')"
|
||||
)
|
||||
self.db.execute(
|
||||
"INSERT OR IGNORE INTO protocol_events("
|
||||
"event_id,occurred_at,event_type,endpoint_id,repo,peer_repo,message_id,"
|
||||
"provenance,delivery_mode,outcome,detail) "
|
||||
"SELECT 'pe-injected-' || message_id,injected_at,'delivery.injected',"
|
||||
"endpoint_id,target_repo,sender_repo,message_id,provenance,NULL,'injected',"
|
||||
"'{\"backfilled\":true}' FROM messages m WHERE injected_at IS NOT NULL "
|
||||
"AND NOT EXISTS (SELECT 1 FROM protocol_events e WHERE "
|
||||
"e.message_id=m.message_id AND e.event_type='delivery.injected')"
|
||||
)
|
||||
self.db.execute(
|
||||
"INSERT OR IGNORE INTO protocol_events("
|
||||
"event_id,occurred_at,event_type,endpoint_id,repo,peer_repo,message_id,"
|
||||
"provenance,delivery_mode,outcome,detail) "
|
||||
"SELECT 'pe-acknowledged-' || message_id,acknowledged_at,'message.acknowledged',"
|
||||
"endpoint_id,target_repo,sender_repo,message_id,provenance,NULL,'acknowledged',"
|
||||
"'{\"backfilled\":true}' FROM messages m WHERE acknowledged_at IS NOT NULL "
|
||||
"AND NOT EXISTS (SELECT 1 FROM protocol_events e WHERE "
|
||||
"e.message_id=m.message_id AND e.event_type='message.acknowledged')"
|
||||
)
|
||||
|
||||
def record_protocol_event(
|
||||
self,
|
||||
event_type: str,
|
||||
*,
|
||||
endpoint_id: str | None = None,
|
||||
repo: str | None = None,
|
||||
peer_repo: str | None = None,
|
||||
message_id: str | None = None,
|
||||
provenance: str | None = None,
|
||||
delivery_mode: str | None = None,
|
||||
outcome: str = "observed",
|
||||
detail: dict | None = None,
|
||||
occurred_at: float | None = None,
|
||||
commit: bool = True,
|
||||
) -> str:
|
||||
"""Append one structured TAMQ event; never record arbitrary pane content."""
|
||||
event_id = f"pe-{uuid4()}"
|
||||
self.db.execute(
|
||||
"INSERT INTO protocol_events("
|
||||
"event_id,occurred_at,event_type,endpoint_id,repo,peer_repo,message_id,"
|
||||
"provenance,delivery_mode,outcome,detail) VALUES(?,?,?,?,?,?,?,?,?,?,?)",
|
||||
(
|
||||
event_id,
|
||||
occurred_at if occurred_at is not None else time.time(),
|
||||
event_type,
|
||||
endpoint_id,
|
||||
repo,
|
||||
peer_repo,
|
||||
message_id,
|
||||
provenance,
|
||||
delivery_mode,
|
||||
outcome,
|
||||
json.dumps(detail or {}, sort_keys=True, separators=(",", ":")),
|
||||
),
|
||||
)
|
||||
if commit:
|
||||
self.db.commit()
|
||||
return event_id
|
||||
|
||||
def protocol_events(
|
||||
self,
|
||||
*,
|
||||
repo: str | None = None,
|
||||
endpoint_id: str | None = None,
|
||||
message_id: str | None = None,
|
||||
event_type: str | None = None,
|
||||
limit: int = 500,
|
||||
) -> list[sqlite3.Row]:
|
||||
"""Return the newest matching protocol events in chronological order."""
|
||||
if isinstance(limit, bool) or limit <= 0:
|
||||
raise ValueError("capture limit must be a positive integer")
|
||||
clauses: list[str] = []
|
||||
values: list[str | int] = []
|
||||
if repo:
|
||||
clauses.append(
|
||||
"(e.repo=? OR e.peer_repo=? OR m.sender_repo=? OR m.target_repo=?)"
|
||||
)
|
||||
values.extend([repo, repo, repo, repo])
|
||||
if endpoint_id:
|
||||
clauses.append("e.endpoint_id=?")
|
||||
values.append(endpoint_id)
|
||||
if message_id:
|
||||
clauses.append("e.message_id=?")
|
||||
values.append(message_id)
|
||||
if event_type:
|
||||
clauses.append("e.event_type=?")
|
||||
values.append(event_type)
|
||||
where = f"WHERE {' AND '.join(clauses)}" if clauses else ""
|
||||
values.append(limit)
|
||||
return list(
|
||||
self.db.execute(
|
||||
"SELECT * FROM (SELECT e.*,m.body AS message_body,"
|
||||
"m.sender_repo AS message_sender_repo,"
|
||||
"m.target_repo AS message_target_repo,m.state AS message_state "
|
||||
"FROM protocol_events e LEFT JOIN messages m "
|
||||
f"ON m.message_id=e.message_id {where} "
|
||||
"ORDER BY e.occurred_at DESC,e.sequence DESC LIMIT ?) "
|
||||
"ORDER BY occurred_at,sequence",
|
||||
values,
|
||||
)
|
||||
)
|
||||
|
||||
def close(self) -> None:
|
||||
self.db.close()
|
||||
|
||||
|
|
@ -106,23 +308,83 @@ class Store:
|
|||
repos: list[str],
|
||||
delivery_mode: str = "manual",
|
||||
limits: tuple[int, int, int] = (8, 1024, 32768),
|
||||
*,
|
||||
policy_profile: str = "default",
|
||||
delivery_ack_mode: str = "injected",
|
||||
delivery_max_attempts: int = 4,
|
||||
ack_timeout_seconds: float = 30.0,
|
||||
) -> None:
|
||||
import json
|
||||
if delivery_mode not in {"manual", "output", "pane", "pushy", "trigger"}:
|
||||
raise ValueError(f"invalid delivery mode: {delivery_mode}")
|
||||
if delivery_ack_mode not in {"injected", "acknowledged"}:
|
||||
raise ValueError("delivery_ack_mode must be injected or acknowledged")
|
||||
if isinstance(delivery_max_attempts, bool) or not 1 <= delivery_max_attempts <= 9:
|
||||
raise ValueError("delivery_max_attempts must be between 1 and 9")
|
||||
if ack_timeout_seconds <= 0:
|
||||
raise ValueError("ack_timeout_seconds must be positive")
|
||||
self._validate_limits(limits)
|
||||
replaced = [
|
||||
row["endpoint_id"]
|
||||
for row in self.db.execute(
|
||||
"SELECT endpoint_id FROM endpoints WHERE pid=? AND session=? "
|
||||
"AND endpoint_id<>? AND disconnected_at IS NULL",
|
||||
(pid, session, endpoint_id),
|
||||
)
|
||||
]
|
||||
self.db.execute(
|
||||
"UPDATE endpoints SET disconnected_at=strftime('%s','now') "
|
||||
"WHERE pid=? AND session=? AND endpoint_id<>? AND disconnected_at IS NULL",
|
||||
(pid, session, endpoint_id),
|
||||
)
|
||||
self.db.execute(
|
||||
"INSERT INTO endpoints(endpoint_id,pid,session,repos,delivery_mode,connected_at,disconnected_at) VALUES(?,?,?,?,?,?,NULL) "
|
||||
"ON CONFLICT(endpoint_id) DO UPDATE SET pid=excluded.pid, session=excluded.session, repos=excluded.repos, delivery_mode=excluded.delivery_mode, connected_at=excluded.connected_at, disconnected_at=NULL",
|
||||
(endpoint_id, pid, session, json.dumps(repos), delivery_mode, time.time()),
|
||||
"INSERT INTO endpoints(endpoint_id,pid,session,repos,delivery_mode,"
|
||||
"policy_profile,delivery_ack_mode,delivery_max_attempts,ack_timeout_seconds,"
|
||||
"connected_at,disconnected_at) VALUES(?,?,?,?,?,?,?,?,?,?,NULL) "
|
||||
"ON CONFLICT(endpoint_id) DO UPDATE SET pid=excluded.pid, "
|
||||
"session=excluded.session, repos=excluded.repos, "
|
||||
"delivery_mode=excluded.delivery_mode, policy_profile=excluded.policy_profile, "
|
||||
"delivery_ack_mode=excluded.delivery_ack_mode, "
|
||||
"delivery_max_attempts=excluded.delivery_max_attempts, "
|
||||
"ack_timeout_seconds=excluded.ack_timeout_seconds, "
|
||||
"connected_at=excluded.connected_at, disconnected_at=NULL",
|
||||
(
|
||||
endpoint_id,
|
||||
pid,
|
||||
session,
|
||||
json.dumps(repos),
|
||||
delivery_mode,
|
||||
policy_profile,
|
||||
delivery_ack_mode,
|
||||
delivery_max_attempts,
|
||||
ack_timeout_seconds,
|
||||
time.time(),
|
||||
),
|
||||
)
|
||||
for repo in repos:
|
||||
self.configure_window(endpoint_id, repo, limits, commit=False)
|
||||
for replaced_id in replaced:
|
||||
self.record_protocol_event(
|
||||
"endpoint.disconnected",
|
||||
endpoint_id=replaced_id,
|
||||
outcome="replaced",
|
||||
commit=False,
|
||||
)
|
||||
self.record_protocol_event(
|
||||
"endpoint.registered",
|
||||
endpoint_id=endpoint_id,
|
||||
delivery_mode=delivery_mode,
|
||||
outcome="registered",
|
||||
detail={
|
||||
"repos": repos,
|
||||
"limits": {"messages": limits[0], "input": limits[1], "output": limits[2]},
|
||||
"policy_profile": policy_profile,
|
||||
"delivery_ack_mode": delivery_ack_mode,
|
||||
"delivery_max_attempts": delivery_max_attempts,
|
||||
"ack_timeout_seconds": ack_timeout_seconds,
|
||||
},
|
||||
commit=False,
|
||||
)
|
||||
self.db.commit()
|
||||
|
||||
@staticmethod
|
||||
|
|
@ -229,11 +491,30 @@ class Store:
|
|||
return result.rowcount == 1
|
||||
|
||||
def disconnect_endpoint(self, endpoint_id: str) -> None:
|
||||
self.db.execute("UPDATE endpoints SET disconnected_at=strftime('%s','now') WHERE endpoint_id=?", (endpoint_id,))
|
||||
result = self.db.execute(
|
||||
"UPDATE endpoints SET disconnected_at=strftime('%s','now') "
|
||||
"WHERE endpoint_id=? AND disconnected_at IS NULL",
|
||||
(endpoint_id,),
|
||||
)
|
||||
if result.rowcount:
|
||||
self.record_protocol_event(
|
||||
"endpoint.disconnected",
|
||||
endpoint_id=endpoint_id,
|
||||
outcome="disconnected",
|
||||
commit=False,
|
||||
)
|
||||
self.db.commit()
|
||||
|
||||
def disconnect_all(self) -> None:
|
||||
endpoint_ids = [row["endpoint_id"] for row in self.endpoints()]
|
||||
self.db.execute("UPDATE endpoints SET disconnected_at=strftime('%s','now') WHERE disconnected_at IS NULL")
|
||||
for endpoint_id in endpoint_ids:
|
||||
self.record_protocol_event(
|
||||
"endpoint.disconnected",
|
||||
endpoint_id=endpoint_id,
|
||||
outcome="service_stopped",
|
||||
commit=False,
|
||||
)
|
||||
self.db.commit()
|
||||
|
||||
def clear_runtime_state(self) -> tuple[int, int]:
|
||||
|
|
@ -244,6 +525,7 @@ class Store:
|
|||
"SELECT COUNT(*) FROM endpoints WHERE disconnected_at IS NULL"
|
||||
).fetchone()[0]
|
||||
)
|
||||
endpoint_ids = [row["endpoint_id"] for row in self.endpoints()]
|
||||
with self.db:
|
||||
self.db.execute("DELETE FROM leases")
|
||||
self.db.execute("DELETE FROM window_counters")
|
||||
|
|
@ -251,6 +533,13 @@ class Store:
|
|||
"UPDATE endpoints SET disconnected_at=strftime('%s','now') "
|
||||
"WHERE disconnected_at IS NULL"
|
||||
)
|
||||
for endpoint_id in endpoint_ids:
|
||||
self.record_protocol_event(
|
||||
"endpoint.disconnected",
|
||||
endpoint_id=endpoint_id,
|
||||
outcome="runtime_cleanup",
|
||||
commit=False,
|
||||
)
|
||||
return endpoints, leases
|
||||
|
||||
def endpoints(self) -> list[sqlite3.Row]:
|
||||
|
|
@ -281,14 +570,59 @@ class Store:
|
|||
oldest = self.db.execute("SELECT MIN(created_at) FROM messages").fetchone()[0]
|
||||
return size, oldest
|
||||
|
||||
def add(self, sender: str, target: str, body: str, *, endpoint: str | None = None, provenance: str | None = None) -> str:
|
||||
@staticmethod
|
||||
def _encode_envelope_metadata(metadata: dict | None) -> str:
|
||||
encoded = json.dumps(metadata or {}, sort_keys=True, separators=(",", ":"))
|
||||
if len(encoded.encode("utf-8")) > 4096:
|
||||
raise ValueError("message metadata exceeds 4 KiB limit")
|
||||
return encoded
|
||||
|
||||
def add(
|
||||
self,
|
||||
sender: str,
|
||||
target: str,
|
||||
body: str,
|
||||
*,
|
||||
endpoint: str | None = None,
|
||||
provenance: str | None = None,
|
||||
client_id: str | None = None,
|
||||
idempotency_key: str | None = None,
|
||||
correlation_id: str | None = None,
|
||||
metadata: dict | None = None,
|
||||
) -> str:
|
||||
if len(body.encode("utf-8")) > 8192:
|
||||
raise ValueError("message body exceeds 8 KiB limit")
|
||||
if bool(client_id) != bool(idempotency_key):
|
||||
raise ValueError("client_id and idempotency_key must be provided together")
|
||||
envelope_metadata = self._encode_envelope_metadata(metadata)
|
||||
message_id = f"m-{uuid4()}"
|
||||
created_at = time.time()
|
||||
self.db.execute(
|
||||
"INSERT INTO messages(message_id,sender_repo,target_repo,body,created_at,state,endpoint_id,provenance,injected_at,acknowledged_at,displayed_at) "
|
||||
"VALUES(?,?,?,?,?,?,?,?,?,?,?)",
|
||||
(message_id, sender, target, body, time.time(), "pending", endpoint, provenance, None, None, None),
|
||||
"INSERT INTO messages(message_id,sender_repo,target_repo,body,created_at,"
|
||||
"state,endpoint_id,provenance,injected_at,acknowledged_at,displayed_at,"
|
||||
"client_id,idempotency_key,correlation_id,envelope_metadata) "
|
||||
"VALUES(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)",
|
||||
(
|
||||
message_id, sender, target, body, created_at, "pending", endpoint,
|
||||
provenance, None, None, None, client_id, idempotency_key,
|
||||
correlation_id, envelope_metadata,
|
||||
),
|
||||
)
|
||||
self.record_protocol_event(
|
||||
"message.accepted",
|
||||
endpoint_id=endpoint,
|
||||
repo=sender,
|
||||
peer_repo=target,
|
||||
message_id=message_id,
|
||||
provenance=provenance,
|
||||
outcome="accepted",
|
||||
detail={
|
||||
"line_count": max(1, len(body.splitlines())),
|
||||
**({"client_id": client_id} if client_id else {}),
|
||||
**({"correlation_id": correlation_id} if correlation_id else {}),
|
||||
},
|
||||
occurred_at=created_at,
|
||||
commit=False,
|
||||
)
|
||||
self.db.commit()
|
||||
return message_id
|
||||
|
|
@ -302,12 +636,20 @@ class Store:
|
|||
endpoint: str,
|
||||
source_repo: str,
|
||||
provenance: str,
|
||||
client_id: str | None = None,
|
||||
idempotency_key: str | None = None,
|
||||
correlation_id: str | None = None,
|
||||
metadata: dict | None = None,
|
||||
) -> str | LimitBlock:
|
||||
"""Atomically check source-window budgets and create one message."""
|
||||
if len(body.encode("utf-8")) > 8192:
|
||||
raise ValueError("message body exceeds 8 KiB limit")
|
||||
if bool(client_id) != bool(idempotency_key):
|
||||
raise ValueError("client_id and idempotency_key must be provided together")
|
||||
envelope_metadata = self._encode_envelope_metadata(metadata)
|
||||
self.db.execute("BEGIN IMMEDIATE")
|
||||
try:
|
||||
created_at = time.time()
|
||||
self.db.execute(
|
||||
"INSERT INTO window_counters(endpoint_id,repo,updated_at) VALUES(?,?,?) "
|
||||
"ON CONFLICT(endpoint_id,repo) DO NOTHING",
|
||||
|
|
@ -324,18 +666,56 @@ class Store:
|
|||
("output", "maxout", "output"),
|
||||
):
|
||||
if int(state[count_name]) >= int(state[limit_name]):
|
||||
blocked = LimitBlock(
|
||||
label, int(state[count_name]), int(state[limit_name])
|
||||
)
|
||||
self.db.rollback()
|
||||
return LimitBlock(label, int(state[count_name]), int(state[limit_name]))
|
||||
self.record_protocol_event(
|
||||
"message.blocked",
|
||||
endpoint_id=endpoint,
|
||||
repo=sender,
|
||||
peer_repo=target,
|
||||
provenance=provenance,
|
||||
outcome="blocked",
|
||||
detail={
|
||||
"dimension": blocked.dimension,
|
||||
"count": blocked.count,
|
||||
"limit": blocked.limit,
|
||||
},
|
||||
)
|
||||
return blocked
|
||||
message_id = f"m-{uuid4()}"
|
||||
self.db.execute(
|
||||
"INSERT INTO messages(message_id,sender_repo,target_repo,body,created_at,state,endpoint_id,provenance,injected_at,acknowledged_at,displayed_at) "
|
||||
"VALUES(?,?,?,?,?,?,?,?,?,?,?)",
|
||||
(message_id, sender, target, body, time.time(), "pending", endpoint, provenance, None, None, None),
|
||||
"INSERT INTO messages(message_id,sender_repo,target_repo,body,created_at,"
|
||||
"state,endpoint_id,provenance,injected_at,acknowledged_at,displayed_at,"
|
||||
"client_id,idempotency_key,correlation_id,envelope_metadata) "
|
||||
"VALUES(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)",
|
||||
(
|
||||
message_id, sender, target, body, created_at, "pending", endpoint,
|
||||
provenance, None, None, None, client_id, idempotency_key,
|
||||
correlation_id, envelope_metadata,
|
||||
),
|
||||
)
|
||||
self.db.execute(
|
||||
"UPDATE window_counters SET messages=messages+1,updated_at=? WHERE endpoint_id=? AND repo=?",
|
||||
(time.time(), endpoint, source_repo),
|
||||
)
|
||||
self.record_protocol_event(
|
||||
"message.accepted",
|
||||
endpoint_id=endpoint,
|
||||
repo=sender,
|
||||
peer_repo=target,
|
||||
message_id=message_id,
|
||||
provenance=provenance,
|
||||
outcome="accepted",
|
||||
detail={
|
||||
"line_count": max(1, len(body.splitlines())),
|
||||
**({"client_id": client_id} if client_id else {}),
|
||||
**({"correlation_id": correlation_id} if correlation_id else {}),
|
||||
},
|
||||
occurred_at=created_at,
|
||||
commit=False,
|
||||
)
|
||||
self.db.commit()
|
||||
return message_id
|
||||
except BaseException:
|
||||
|
|
@ -351,11 +731,54 @@ class Store:
|
|||
where = f" WHERE {' AND '.join(clauses)}" if clauses else ""
|
||||
return list(self.db.execute(f"SELECT * FROM messages{where} ORDER BY created_at", values))
|
||||
|
||||
def inbox(self, target: str, *, include_terminal: bool = False) -> list[sqlite3.Row]:
|
||||
if include_terminal:
|
||||
return self.list(target)
|
||||
return list(
|
||||
self.db.execute(
|
||||
"SELECT * FROM messages WHERE target_repo=? "
|
||||
"AND state IN ('pending','awaiting_ack') ORDER BY created_at",
|
||||
(target,),
|
||||
)
|
||||
)
|
||||
|
||||
def message(self, message_id: str) -> sqlite3.Row | None:
|
||||
return self.db.execute(
|
||||
"SELECT * FROM messages WHERE message_id=?", (message_id,)
|
||||
).fetchone()
|
||||
|
||||
def message_by_idempotency(
|
||||
self, client_id: str, idempotency_key: str
|
||||
) -> sqlite3.Row | None:
|
||||
return self.db.execute(
|
||||
"SELECT * FROM messages WHERE client_id=? AND idempotency_key=?",
|
||||
(client_id, idempotency_key),
|
||||
).fetchone()
|
||||
|
||||
def rebind_message_endpoint(
|
||||
self, message_id: str, endpoint_id: str
|
||||
) -> bool:
|
||||
row = self.message(message_id)
|
||||
if row is None or row["state"] not in {"pending", "awaiting_ack", "failed"}:
|
||||
return False
|
||||
with self.db:
|
||||
self.db.execute(
|
||||
"UPDATE messages SET endpoint_id=? WHERE message_id=?",
|
||||
(endpoint_id, message_id),
|
||||
)
|
||||
self.record_protocol_event(
|
||||
"message.rebound",
|
||||
endpoint_id=endpoint_id,
|
||||
repo=row["sender_repo"],
|
||||
peer_repo=row["target_repo"],
|
||||
message_id=message_id,
|
||||
provenance=row["provenance"],
|
||||
outcome="rebound",
|
||||
detail={"previous_endpoint_id": row["endpoint_id"]},
|
||||
commit=False,
|
||||
)
|
||||
return True
|
||||
|
||||
def feedback_chain(self, root_id: str) -> list[sqlite3.Row]:
|
||||
"""Resolve a reflected-delivery chain using receipt and direction evidence."""
|
||||
rows = self.list()
|
||||
|
|
@ -405,10 +828,323 @@ class Store:
|
|||
self.db.commit()
|
||||
|
||||
def acknowledge(self, message_id: str) -> bool:
|
||||
row = self.db.execute("SELECT 1 FROM messages WHERE message_id=?", (message_id,)).fetchone()
|
||||
row = self.db.execute("SELECT * FROM messages WHERE message_id=?", (message_id,)).fetchone()
|
||||
if row is None:
|
||||
return False
|
||||
self.set_state(message_id, "acknowledged")
|
||||
with self.db:
|
||||
self.db.execute("DELETE FROM leases WHERE message_id=?", (message_id,))
|
||||
self.db.execute(
|
||||
"UPDATE messages SET state='acknowledged',acknowledged_at=?,"
|
||||
"next_attempt_at=NULL,last_failure_reason=NULL WHERE message_id=?",
|
||||
(time.time(), message_id),
|
||||
)
|
||||
self.record_protocol_event(
|
||||
"message.acknowledged",
|
||||
endpoint_id=row["endpoint_id"],
|
||||
repo=row["target_repo"],
|
||||
peer_repo=row["sender_repo"],
|
||||
message_id=message_id,
|
||||
provenance=row["provenance"],
|
||||
outcome="late_acknowledged" if row["state"] == "failed" else "acknowledged",
|
||||
commit=False,
|
||||
)
|
||||
return True
|
||||
|
||||
def deliverable(
|
||||
self, endpoint_id: str, repos: list[str], *, now: float | None = None
|
||||
) -> list[sqlite3.Row]:
|
||||
if not repos:
|
||||
return []
|
||||
when = time.time() if now is None else now
|
||||
placeholders = ",".join("?" for _ in repos)
|
||||
return list(
|
||||
self.db.execute(
|
||||
"SELECT * FROM messages WHERE state IN ('pending','awaiting_ack') "
|
||||
"AND (endpoint_id IS NULL OR endpoint_id=?) "
|
||||
f"AND target_repo IN ({placeholders}) "
|
||||
"AND (next_attempt_at IS NULL OR next_attempt_at<=?) "
|
||||
"ORDER BY created_at",
|
||||
[endpoint_id, *repos, when],
|
||||
)
|
||||
)
|
||||
|
||||
def expire_delivery_leases(self, *, now: float | None = None) -> int:
|
||||
"""Turn expired delivery ownership into bounded retry or terminal failure."""
|
||||
when = time.time() if now is None else now
|
||||
rows = list(
|
||||
self.db.execute(
|
||||
"SELECT l.*,m.sender_repo,m.target_repo,m.provenance,m.attempt_count "
|
||||
"FROM leases l JOIN messages m USING(message_id) WHERE l.expires_at<?",
|
||||
(when,),
|
||||
)
|
||||
)
|
||||
with self.db:
|
||||
for row in rows:
|
||||
terminal = int(row["attempt_count"]) >= int(row["attempt_limit"])
|
||||
self.db.execute(
|
||||
"DELETE FROM leases WHERE message_id=? AND lease_id=?",
|
||||
(row["message_id"], row["lease_id"]),
|
||||
)
|
||||
self.db.execute(
|
||||
"UPDATE messages SET state=?,last_failure_reason='lease_expired',"
|
||||
"next_attempt_at=? WHERE message_id=? AND state!='acknowledged'",
|
||||
(
|
||||
"failed" if terminal else "pending",
|
||||
None if terminal else when + float(row["retry_delay"]),
|
||||
row["message_id"],
|
||||
),
|
||||
)
|
||||
self.record_protocol_event(
|
||||
"delivery.failed",
|
||||
endpoint_id=row["endpoint_id"],
|
||||
repo=row["target_repo"],
|
||||
peer_repo=row["sender_repo"],
|
||||
message_id=row["message_id"],
|
||||
provenance=row["provenance"],
|
||||
outcome="exhausted" if terminal else "retry_scheduled",
|
||||
detail={
|
||||
"reason": "lease_expired",
|
||||
"attempt": int(row["attempt_count"]),
|
||||
"attempt_limit": int(row["attempt_limit"]),
|
||||
},
|
||||
commit=False,
|
||||
)
|
||||
return len(rows)
|
||||
|
||||
def claim_delivery(
|
||||
self,
|
||||
message_id: str,
|
||||
endpoint_id: str,
|
||||
*,
|
||||
attempt_limit: int,
|
||||
retry_delay: float,
|
||||
delivery_mode: str,
|
||||
ttl: float = 30.0,
|
||||
now: float | None = None,
|
||||
) -> DeliveryClaim | None:
|
||||
if not 1 <= attempt_limit <= 9:
|
||||
raise ValueError("delivery attempt limit must be between 1 and 9")
|
||||
if retry_delay < 0 or ttl <= 0:
|
||||
raise ValueError("delivery retry delay must be non-negative and lease TTL positive")
|
||||
when = time.time() if now is None else now
|
||||
self.expire_delivery_leases(now=when)
|
||||
lease_id = f"lease-{uuid4()}"
|
||||
with self.db:
|
||||
row = self.db.execute(
|
||||
"SELECT * FROM messages WHERE message_id=? AND "
|
||||
"state IN ('pending','awaiting_ack') AND "
|
||||
"(next_attempt_at IS NULL OR next_attempt_at<=?)",
|
||||
(message_id, when),
|
||||
).fetchone()
|
||||
if row is None:
|
||||
return None
|
||||
if int(row["attempt_count"]) >= attempt_limit:
|
||||
reason = (
|
||||
"ack_timeout"
|
||||
if row["state"] == "awaiting_ack"
|
||||
else "attempt_limit_exhausted"
|
||||
)
|
||||
self.db.execute(
|
||||
"UPDATE messages SET state='failed',last_failure_reason=?,"
|
||||
"next_attempt_at=NULL WHERE message_id=?",
|
||||
(reason, message_id),
|
||||
)
|
||||
self.record_protocol_event(
|
||||
"delivery.failed",
|
||||
endpoint_id=endpoint_id,
|
||||
repo=row["target_repo"],
|
||||
peer_repo=row["sender_repo"],
|
||||
message_id=message_id,
|
||||
provenance=row["provenance"],
|
||||
delivery_mode=delivery_mode,
|
||||
outcome="exhausted",
|
||||
detail={
|
||||
"reason": reason,
|
||||
"attempt": int(row["attempt_count"]),
|
||||
"attempt_limit": attempt_limit,
|
||||
},
|
||||
commit=False,
|
||||
)
|
||||
return None
|
||||
if row["state"] == "awaiting_ack":
|
||||
self.record_protocol_event(
|
||||
"delivery.failed",
|
||||
endpoint_id=endpoint_id,
|
||||
repo=row["target_repo"],
|
||||
peer_repo=row["sender_repo"],
|
||||
message_id=message_id,
|
||||
provenance=row["provenance"],
|
||||
delivery_mode=delivery_mode,
|
||||
outcome="retrying",
|
||||
detail={
|
||||
"reason": "ack_timeout",
|
||||
"attempt": int(row["attempt_count"]),
|
||||
"attempt_limit": attempt_limit,
|
||||
},
|
||||
commit=False,
|
||||
)
|
||||
try:
|
||||
self.db.execute(
|
||||
"INSERT INTO leases(message_id,lease_id,endpoint_id,acquired_at,"
|
||||
"expires_at,attempt_limit,retry_delay) VALUES(?,?,?,?,?,?,?)",
|
||||
(
|
||||
message_id, lease_id, endpoint_id, when, when + ttl,
|
||||
attempt_limit, retry_delay,
|
||||
),
|
||||
)
|
||||
except sqlite3.IntegrityError:
|
||||
return None
|
||||
attempt = int(row["attempt_count"]) + 1
|
||||
self.db.execute(
|
||||
"UPDATE messages SET state='pending',attempt_count=?,last_attempt_at=?,"
|
||||
"last_failure_reason=NULL,next_attempt_at=NULL WHERE message_id=?",
|
||||
(attempt, when, message_id),
|
||||
)
|
||||
self.record_protocol_event(
|
||||
"delivery.attempted",
|
||||
endpoint_id=endpoint_id,
|
||||
repo=row["target_repo"],
|
||||
peer_repo=row["sender_repo"],
|
||||
message_id=message_id,
|
||||
provenance=row["provenance"],
|
||||
delivery_mode=delivery_mode,
|
||||
outcome="attempted",
|
||||
detail={"attempt": attempt, "attempt_limit": attempt_limit},
|
||||
commit=False,
|
||||
)
|
||||
return DeliveryClaim(lease_id, attempt)
|
||||
|
||||
def fail_delivery(
|
||||
self,
|
||||
message_id: str,
|
||||
lease_id: str,
|
||||
reason: str,
|
||||
*,
|
||||
delivery_mode: str,
|
||||
now: float | None = None,
|
||||
) -> bool:
|
||||
when = time.time() if now is None else now
|
||||
row = self.db.execute(
|
||||
"SELECT l.*,m.sender_repo,m.target_repo,m.provenance,m.attempt_count "
|
||||
"FROM leases l JOIN messages m USING(message_id) "
|
||||
"WHERE l.message_id=? AND l.lease_id=?",
|
||||
(message_id, lease_id),
|
||||
).fetchone()
|
||||
if row is None:
|
||||
return False
|
||||
terminal = int(row["attempt_count"]) >= int(row["attempt_limit"])
|
||||
with self.db:
|
||||
self.db.execute(
|
||||
"DELETE FROM leases WHERE message_id=? AND lease_id=?",
|
||||
(message_id, lease_id),
|
||||
)
|
||||
self.db.execute(
|
||||
"UPDATE messages SET state=?,last_failure_reason=?,next_attempt_at=? "
|
||||
"WHERE message_id=? AND state!='acknowledged'",
|
||||
(
|
||||
"failed" if terminal else "pending",
|
||||
reason,
|
||||
None if terminal else when + float(row["retry_delay"]),
|
||||
message_id,
|
||||
),
|
||||
)
|
||||
self.record_protocol_event(
|
||||
"delivery.failed",
|
||||
endpoint_id=row["endpoint_id"],
|
||||
repo=row["target_repo"],
|
||||
peer_repo=row["sender_repo"],
|
||||
message_id=message_id,
|
||||
provenance=row["provenance"],
|
||||
delivery_mode=delivery_mode,
|
||||
outcome="exhausted" if terminal else "retry_scheduled",
|
||||
detail={
|
||||
"reason": reason,
|
||||
"attempt": int(row["attempt_count"]),
|
||||
"attempt_limit": int(row["attempt_limit"]),
|
||||
"retry_at": None if terminal else when + float(row["retry_delay"]),
|
||||
},
|
||||
commit=False,
|
||||
)
|
||||
return True
|
||||
|
||||
def complete_delivery(
|
||||
self,
|
||||
message_id: str,
|
||||
lease_id: str,
|
||||
*,
|
||||
delivery_mode: str,
|
||||
ack_mode: str,
|
||||
ack_timeout: float,
|
||||
now: float | None = None,
|
||||
) -> bool:
|
||||
if ack_mode not in {"injected", "acknowledged"}:
|
||||
raise ValueError("invalid acknowledgement mode")
|
||||
if ack_timeout <= 0:
|
||||
raise ValueError("acknowledgement timeout must be positive")
|
||||
when = time.time() if now is None else now
|
||||
row = self.db.execute(
|
||||
"SELECT l.endpoint_id,m.* FROM leases l JOIN messages m USING(message_id) "
|
||||
"WHERE l.message_id=? AND l.lease_id=?",
|
||||
(message_id, lease_id),
|
||||
).fetchone()
|
||||
if row is None:
|
||||
return False
|
||||
state = "injected" if ack_mode == "injected" else "awaiting_ack"
|
||||
event_type = "delivery.displayed" if delivery_mode == "output" else "delivery.injected"
|
||||
with self.db:
|
||||
self.db.execute(
|
||||
"DELETE FROM leases WHERE message_id=? AND lease_id=?",
|
||||
(message_id, lease_id),
|
||||
)
|
||||
self.db.execute(
|
||||
"UPDATE messages SET state=?,delivered_at=?,displayed_at=CASE "
|
||||
"WHEN ?='output' THEN ? ELSE displayed_at END,injected_at=CASE "
|
||||
"WHEN ?!='output' THEN ? ELSE injected_at END,next_attempt_at=?,"
|
||||
"last_failure_reason=NULL WHERE message_id=?",
|
||||
(
|
||||
state, when, delivery_mode, when, delivery_mode, when,
|
||||
None if ack_mode == "injected" else when + ack_timeout,
|
||||
message_id,
|
||||
),
|
||||
)
|
||||
self.record_protocol_event(
|
||||
event_type,
|
||||
endpoint_id=row["endpoint_id"],
|
||||
repo=row["target_repo"],
|
||||
peer_repo=row["sender_repo"],
|
||||
message_id=message_id,
|
||||
provenance=row["provenance"],
|
||||
delivery_mode=delivery_mode,
|
||||
outcome=state,
|
||||
detail={
|
||||
"attempt": int(row["attempt_count"]),
|
||||
"ack_mode": ack_mode,
|
||||
"ack_deadline": None if ack_mode == "injected" else when + ack_timeout,
|
||||
},
|
||||
commit=False,
|
||||
)
|
||||
return True
|
||||
|
||||
def retry_message(self, message_id: str) -> bool:
|
||||
row = self.message(message_id)
|
||||
if row is None or row["state"] != "failed":
|
||||
return False
|
||||
with self.db:
|
||||
self.db.execute(
|
||||
"UPDATE messages SET state='pending',attempt_count=0,last_attempt_at=NULL,"
|
||||
"last_failure_reason=NULL,next_attempt_at=NULL WHERE message_id=?",
|
||||
(message_id,),
|
||||
)
|
||||
self.record_protocol_event(
|
||||
"delivery.retry_reset",
|
||||
endpoint_id=row["endpoint_id"],
|
||||
repo=row["target_repo"],
|
||||
peer_repo=row["sender_repo"],
|
||||
message_id=message_id,
|
||||
provenance=row["provenance"],
|
||||
outcome="pending",
|
||||
commit=False,
|
||||
)
|
||||
return True
|
||||
|
||||
def mark_displayed(self, message_id: str, lease_id: str) -> bool:
|
||||
|
|
@ -427,10 +1163,14 @@ class Store:
|
|||
def claim(self, message_id: str, endpoint_id: str, ttl: float = 30.0) -> str | None:
|
||||
now = time.time()
|
||||
lease_id = f"lease-{uuid4()}"
|
||||
self.expire_delivery_leases(now=now)
|
||||
with self.db:
|
||||
self.db.execute("DELETE FROM leases WHERE expires_at < ?", (now,))
|
||||
try:
|
||||
self.db.execute("INSERT INTO leases VALUES(?,?,?,?,?)", (message_id, lease_id, endpoint_id, now, now + ttl))
|
||||
self.db.execute(
|
||||
"INSERT INTO leases(message_id,lease_id,endpoint_id,acquired_at,expires_at) "
|
||||
"VALUES(?,?,?,?,?)",
|
||||
(message_id, lease_id, endpoint_id, now, now + ttl),
|
||||
)
|
||||
except sqlite3.IntegrityError:
|
||||
return None
|
||||
return lease_id
|
||||
|
|
|
|||
|
|
@ -47,6 +47,15 @@ def test_broker_collects_worker_followup_lines_until_empty(tmp_path, monkeypatch
|
|||
assert row["body"] == (
|
||||
"Review this\nContext: AUTH-WP-4.\nReply with accepted or blocked."
|
||||
)
|
||||
events = store.protocol_events()
|
||||
assert [event["event_type"] for event in events] == [
|
||||
"worker.block_started",
|
||||
"message.accepted",
|
||||
"worker.block_closed",
|
||||
]
|
||||
assert events[-1]["message_id"] == row["message_id"]
|
||||
assert events[-1]["outcome"] == "accepted"
|
||||
assert events[-1]["detail"] == '{"line_count":3,"reason":"empty_line"}'
|
||||
|
||||
|
||||
def test_new_worker_address_flushes_previous_block(tmp_path, monkeypatch):
|
||||
|
|
@ -81,6 +90,10 @@ def test_worker_cmd_is_inert_but_operator_cmd_changes_mode(tmp_path):
|
|||
broker.inspect_operator_line("cMD: MODE=TRIGGER")
|
||||
assert store.endpoint("ep")["delivery_mode"] == "trigger"
|
||||
assert notices[-1] == "From:tamq: Mode set to trigger."
|
||||
command_events = store.protocol_events(event_type="command.applied")
|
||||
assert len(command_events) == 1
|
||||
assert command_events[0]["delivery_mode"] == "trigger"
|
||||
assert command_events[0]["provenance"] == "operator_input"
|
||||
|
||||
|
||||
def test_operator_command_names_are_case_insensitive(tmp_path):
|
||||
|
|
|
|||
56
tests/test_capture.py
Normal file
56
tests/test_capture.py
Normal file
|
|
@ -0,0 +1,56 @@
|
|||
import json
|
||||
|
||||
from tamq.capture import render_jsonl, render_markdown
|
||||
from tamq.cli import main
|
||||
from tamq.store import Store
|
||||
|
||||
|
||||
def test_capture_renderers_expose_protocol_but_not_duplicate_message_bodies(tmp_path):
|
||||
store = Store(tmp_path / "queue.sqlite3")
|
||||
message_id = store.add(
|
||||
"source", "target", "Please review.\nReply accepted.", provenance="worker_output"
|
||||
)
|
||||
store.record_protocol_event(
|
||||
"delivery.attempted",
|
||||
repo="target",
|
||||
peer_repo="source",
|
||||
message_id=message_id,
|
||||
provenance="worker_output",
|
||||
delivery_mode="trigger",
|
||||
outcome="attempted",
|
||||
)
|
||||
rows = store.protocol_events(repo="source")
|
||||
|
||||
markdown = render_markdown(rows, scope="repository `source`")
|
||||
assert "# TAMQ protocol capture" in markdown
|
||||
assert "Accepted messages: 1" in markdown
|
||||
assert "## Review cues" in markdown
|
||||
assert markdown.count("Please review.") == 1
|
||||
assert "does not contain unrelated pane output" in markdown
|
||||
|
||||
records = [json.loads(line) for line in render_jsonl(rows).splitlines()]
|
||||
assert records[0]["message_body"] == "Please review.\nReply accepted."
|
||||
assert "message_body" not in records[1]
|
||||
store.close()
|
||||
|
||||
|
||||
def test_capture_cli_filters_and_writes_review_file(tmp_path, monkeypatch, capsys):
|
||||
database = tmp_path / "queue.sqlite3"
|
||||
monkeypatch.setattr("tamq.cli.db_path", lambda: database)
|
||||
store = Store(database)
|
||||
store.add("one", "two", "hello", provenance="operator_input")
|
||||
store.add("three", "four", "ignore", provenance="worker_output")
|
||||
store.close()
|
||||
output = tmp_path / "review" / "protocol.md"
|
||||
|
||||
assert main(["capture", "--repo", "one", "--output", str(output)]) == 0
|
||||
assert "Wrote 1 protocol event" in capsys.readouterr().out
|
||||
report = output.read_text()
|
||||
assert "hello" in report
|
||||
assert "ignore" not in report
|
||||
|
||||
|
||||
def test_capture_cli_rejects_non_positive_limit(tmp_path, monkeypatch, capsys):
|
||||
monkeypatch.setattr("tamq.cli.db_path", lambda: tmp_path / "queue.sqlite3")
|
||||
assert main(["capture", "--limit", "0"]) == 2
|
||||
assert "capture limit must be a positive integer" in capsys.readouterr().err
|
||||
|
|
@ -5,7 +5,10 @@ from tamq.cli import (
|
|||
parse_size,
|
||||
)
|
||||
from tamq.service import (
|
||||
DELIVERY_RELIABILITY_CAPABILITY,
|
||||
IDEMPOTENT_SEND_CAPABILITY,
|
||||
LINE_LIMITS_CAPABILITY,
|
||||
PROTOCOL_CAPTURE_CAPABILITY,
|
||||
PUSHY_FRAMING_CAPABILITY,
|
||||
TRIGGER_CAPABILITY,
|
||||
)
|
||||
|
|
@ -17,7 +20,7 @@ def test_parse_size():
|
|||
|
||||
|
||||
def test_manual_service_restarts_legacy_broker(monkeypatch):
|
||||
capabilities = iter([[], ["manual_delivery", PUSHY_FRAMING_CAPABILITY, LINE_LIMITS_CAPABILITY]])
|
||||
capabilities = iter([[], ["manual_delivery", PUSHY_FRAMING_CAPABILITY, LINE_LIMITS_CAPABILITY, PROTOCOL_CAPTURE_CAPABILITY, DELIVERY_RELIABILITY_CAPABILITY, IDEMPOTENT_SEND_CAPABILITY]])
|
||||
starts = []
|
||||
stops = []
|
||||
|
||||
|
|
@ -37,7 +40,7 @@ def test_manual_service_restarts_legacy_broker(monkeypatch):
|
|||
def test_output_service_requires_terminal_output_capability(monkeypatch):
|
||||
capabilities = iter([
|
||||
["manual_delivery"],
|
||||
["manual_delivery", "terminal_output", PUSHY_FRAMING_CAPABILITY, LINE_LIMITS_CAPABILITY],
|
||||
["manual_delivery", "terminal_output", PUSHY_FRAMING_CAPABILITY, LINE_LIMITS_CAPABILITY, PROTOCOL_CAPTURE_CAPABILITY, DELIVERY_RELIABILITY_CAPABILITY, IDEMPOTENT_SEND_CAPABILITY],
|
||||
])
|
||||
starts = []
|
||||
stops = []
|
||||
|
|
@ -64,6 +67,9 @@ def test_pushy_service_requires_pushy_input_capability(monkeypatch):
|
|||
PUSHY_FRAMING_CAPABILITY,
|
||||
TRIGGER_CAPABILITY,
|
||||
LINE_LIMITS_CAPABILITY,
|
||||
PROTOCOL_CAPTURE_CAPABILITY,
|
||||
DELIVERY_RELIABILITY_CAPABILITY,
|
||||
IDEMPOTENT_SEND_CAPABILITY,
|
||||
],
|
||||
])
|
||||
starts = []
|
||||
|
|
|
|||
202
tests/test_client_adapter.py
Normal file
202
tests/test_client_adapter.py
Normal file
|
|
@ -0,0 +1,202 @@
|
|||
import asyncio
|
||||
import json
|
||||
|
||||
import pytest
|
||||
|
||||
from tamq.client import (
|
||||
CoordinationEngineAdapter,
|
||||
TamqClient,
|
||||
TamqClientError,
|
||||
TamqProtocolError,
|
||||
TamqTargetUnavailable,
|
||||
WakeRequest,
|
||||
)
|
||||
from tamq.service import Service
|
||||
from tamq.store import Store
|
||||
|
||||
|
||||
async def _start_service(path, store):
|
||||
service = Service(path, store)
|
||||
server = await asyncio.start_unix_server(service.handle, path=str(path))
|
||||
return service, server
|
||||
|
||||
|
||||
def test_coordination_adapter_negotiates_and_deduplicates_wake(tmp_path, monkeypatch):
|
||||
monkeypatch.setattr("tamq.service.validate_targets", lambda repos: None)
|
||||
|
||||
async def run():
|
||||
path = tmp_path / "tamq.sock"
|
||||
store = Store(tmp_path / "queue.sqlite3")
|
||||
store.register_endpoint(
|
||||
"ep-1", 42, "tamq", ["target"], "manual",
|
||||
delivery_ack_mode="acknowledged",
|
||||
)
|
||||
service, server = await _start_service(path, store)
|
||||
adapter = CoordinationEngineAdapter(TamqClient(path))
|
||||
wake = WakeRequest(
|
||||
lease_id="lease-7",
|
||||
trigger_id="trigger-9",
|
||||
target_repo="target",
|
||||
prompt="Continue task T1.",
|
||||
)
|
||||
try:
|
||||
first = await adapter.wake(wake)
|
||||
repeated = await adapter.wake(wake)
|
||||
assert repeated.message_id == first.message_id
|
||||
assert first.deduplicated is False
|
||||
assert repeated.deduplicated is True
|
||||
row = await adapter.receipt(first.message_id)
|
||||
assert row["client_id"] == "coordination-engine"
|
||||
assert row["idempotency_key"] == "lease-7"
|
||||
assert row["correlation_id"] == "trigger-9"
|
||||
assert json.loads(row["envelope_metadata"])["trigger_id"] == "trigger-9"
|
||||
assert len(store.list()) == 1
|
||||
assert store.line_state("ep-1", "coordination-engine")["messages"] == 1
|
||||
await adapter.acknowledge(first.message_id)
|
||||
assert (await adapter.receipt(first.message_id))["state"] == "acknowledged"
|
||||
finally:
|
||||
server.close()
|
||||
await server.wait_closed()
|
||||
service.close()
|
||||
|
||||
asyncio.run(run())
|
||||
|
||||
|
||||
def test_client_reconnects_and_recovers_receipt_from_durable_store(tmp_path, monkeypatch):
|
||||
monkeypatch.setattr("tamq.service.validate_targets", lambda repos: None)
|
||||
|
||||
async def run():
|
||||
path = tmp_path / "tamq.sock"
|
||||
database = tmp_path / "queue.sqlite3"
|
||||
store = Store(database)
|
||||
store.register_endpoint("ep", 42, "tamq", ["target"], "manual")
|
||||
first_service, first_server = await _start_service(path, store)
|
||||
client = TamqClient(path)
|
||||
sent = await client.send(
|
||||
sender_repo="coordination-engine",
|
||||
target_repo="target",
|
||||
body="wake",
|
||||
endpoint_id="ep",
|
||||
idempotency_key="lease-restart",
|
||||
)
|
||||
first_server.close()
|
||||
await first_server.wait_closed()
|
||||
first_service.close()
|
||||
path.unlink(missing_ok=True)
|
||||
|
||||
reopened = Store(database)
|
||||
second_service, second_server = await _start_service(path, reopened)
|
||||
try:
|
||||
receipt = await client.message(sent["message_id"])
|
||||
assert receipt["state"] == "pending"
|
||||
repeated = await client.send(
|
||||
sender_repo="coordination-engine",
|
||||
target_repo="target",
|
||||
body="wake",
|
||||
endpoint_id="ep",
|
||||
idempotency_key="lease-restart",
|
||||
)
|
||||
assert repeated["deduplicated"] is True
|
||||
finally:
|
||||
second_server.close()
|
||||
await second_server.wait_closed()
|
||||
second_service.close()
|
||||
|
||||
asyncio.run(run())
|
||||
|
||||
|
||||
def test_adapter_rejects_incompatible_protocol_and_disappeared_endpoint(
|
||||
tmp_path, monkeypatch
|
||||
):
|
||||
monkeypatch.setattr("tamq.service.validate_targets", lambda repos: None)
|
||||
|
||||
async def run():
|
||||
path = tmp_path / "tamq.sock"
|
||||
store = Store(tmp_path / "queue.sqlite3")
|
||||
store.register_endpoint("ep", 42, "tamq", ["target"], "manual")
|
||||
service, server = await _start_service(path, store)
|
||||
try:
|
||||
with pytest.raises(TamqProtocolError):
|
||||
await TamqClient(path, protocol="2.0").negotiate()
|
||||
store.disconnect_endpoint("ep")
|
||||
adapter = CoordinationEngineAdapter(TamqClient(path))
|
||||
with pytest.raises(TamqTargetUnavailable):
|
||||
await adapter.wake(
|
||||
WakeRequest("lease", "target", "wake", "trigger")
|
||||
)
|
||||
finally:
|
||||
server.close()
|
||||
await server.wait_closed()
|
||||
service.close()
|
||||
|
||||
asyncio.run(run())
|
||||
|
||||
|
||||
def test_repeated_wake_rebinds_same_message_after_endpoint_replacement(
|
||||
tmp_path, monkeypatch
|
||||
):
|
||||
monkeypatch.setattr("tamq.service.validate_targets", lambda repos: None)
|
||||
|
||||
async def run():
|
||||
path = tmp_path / "tamq.sock"
|
||||
store = Store(tmp_path / "queue.sqlite3")
|
||||
store.register_endpoint("old", 41, "tamq-old", ["target"], "manual")
|
||||
service, server = await _start_service(path, store)
|
||||
adapter = CoordinationEngineAdapter(TamqClient(path))
|
||||
wake = WakeRequest("lease-rebind", "target", "wake", "trigger")
|
||||
try:
|
||||
first = await adapter.wake(wake)
|
||||
store.disconnect_endpoint("old")
|
||||
store.register_endpoint("new", 42, "tamq-new", ["target"], "manual")
|
||||
repeated = await adapter.wake(wake)
|
||||
assert repeated.message_id == first.message_id
|
||||
assert repeated.endpoint_id == "new"
|
||||
assert repeated.deduplicated is True
|
||||
assert store.message(first.message_id)["endpoint_id"] == "new"
|
||||
rebound = store.protocol_events(
|
||||
message_id=first.message_id, event_type="message.rebound"
|
||||
)
|
||||
assert len(rebound) == 1
|
||||
finally:
|
||||
server.close()
|
||||
await server.wait_closed()
|
||||
service.close()
|
||||
|
||||
asyncio.run(run())
|
||||
|
||||
|
||||
def test_idempotency_key_conflict_is_rejected(tmp_path, monkeypatch):
|
||||
monkeypatch.setattr("tamq.service.validate_targets", lambda repos: None)
|
||||
|
||||
async def run():
|
||||
path = tmp_path / "tamq.sock"
|
||||
store = Store(tmp_path / "queue.sqlite3")
|
||||
service, server = await _start_service(path, store)
|
||||
client = TamqClient(path)
|
||||
try:
|
||||
first = await client.send(
|
||||
sender_repo="coordination-engine",
|
||||
target_repo="one",
|
||||
body="first",
|
||||
idempotency_key="same",
|
||||
)
|
||||
with pytest.raises(TamqClientError, match="different message"):
|
||||
await client.send(
|
||||
sender_repo="coordination-engine",
|
||||
target_repo="two",
|
||||
body="second",
|
||||
idempotency_key="same",
|
||||
)
|
||||
store.db.execute(
|
||||
"UPDATE messages SET state='failed',attempt_count=4 WHERE message_id=?",
|
||||
(first["message_id"],),
|
||||
)
|
||||
store.db.commit()
|
||||
retried = await client.retry(first["message_id"])
|
||||
assert retried["state"] == "pending"
|
||||
finally:
|
||||
server.close()
|
||||
await server.wait_closed()
|
||||
service.close()
|
||||
|
||||
asyncio.run(run())
|
||||
|
|
@ -12,3 +12,6 @@ def test_config_command(monkeypatch, capsys, tmp_path):
|
|||
assert result["maxout"] == 32768
|
||||
assert result["database"].endswith("tamq.sqlite3")
|
||||
assert result["policy_profile"] == "default"
|
||||
assert result["delivery_ack_mode"] == "injected"
|
||||
assert result["delivery_max_attempts"] == 4
|
||||
assert result["ack_timeout_seconds"] == 30
|
||||
|
|
|
|||
|
|
@ -27,6 +27,7 @@ def test_isolated_installed_tool_session_smoke(tmp_path):
|
|||
bin_dir = tmp_path / "bin"
|
||||
runtime_dir = tmp_path / "runtime"
|
||||
state_dir = tmp_path / "state"
|
||||
config = tmp_path / "config.toml"
|
||||
repo_a = tmp_path / "railiance-platform"
|
||||
repo_b = tmp_path / "activity-core"
|
||||
for path in (bin_dir, runtime_dir, repo_a, repo_b):
|
||||
|
|
@ -52,6 +53,12 @@ def test_isolated_installed_tool_session_smoke(tmp_path):
|
|||
)
|
||||
gita.chmod(0o755)
|
||||
agent.chmod(0o755)
|
||||
config.write_text(
|
||||
"[policy.profiles.integration]\n"
|
||||
"delivery_ack_mode='acknowledged'\n"
|
||||
"delivery_max_attempts=3\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
socket_name = f"tamq-installed-{os.getpid()}"
|
||||
env = os.environ.copy()
|
||||
|
|
@ -63,6 +70,8 @@ def test_isolated_installed_tool_session_smoke(tmp_path):
|
|||
"UV_CACHE_DIR": str(tmp_path / "uv-cache"),
|
||||
"XDG_RUNTIME_DIR": str(runtime_dir),
|
||||
"TAMQ_STATE_DIR": str(state_dir),
|
||||
"TAMQ_CONFIG": str(config),
|
||||
"TAMQ_POLICY_PROFILE": "integration",
|
||||
"TAMQ_TMUX_SOCKET": socket_name,
|
||||
}
|
||||
)
|
||||
|
|
@ -202,7 +211,7 @@ def test_isolated_installed_tool_session_smoke(tmp_path):
|
|||
).stdout
|
||||
assert inbox[-1]["sender_repo"] == "railiance-platform"
|
||||
assert inbox[-1]["body"] == "installed-message"
|
||||
assert inbox[-1]["state"] == "pending"
|
||||
assert inbox[-1]["state"] == "awaiting_ack"
|
||||
assert inbox[-1]["displayed_at"] is not None
|
||||
message_id = inbox[-1]["message_id"]
|
||||
assert run(str(tamq), "inbox", "--repo", "activity-core").stdout == (
|
||||
|
|
|
|||
|
|
@ -9,6 +9,18 @@ def test_default_policy(tmp_path):
|
|||
|
||||
def test_configured_policy(tmp_path):
|
||||
path = tmp_path / "config.toml"
|
||||
path.write_text("[policy.profiles.dev]\nallow=['repo_inspect']\nrequire_human=['destructive']\nsafety_gated_max_attempts=2\n")
|
||||
path.write_text("[policy.profiles.dev]\nallow=['repo_inspect']\nrequire_human=['destructive']\nsafety_gated_max_attempts=2\ndelivery_max_attempts=3\n")
|
||||
profile = load_profile(path, "dev")
|
||||
assert profile.safety_gated_max_attempts == 2
|
||||
assert profile.delivery_max_attempts == 3
|
||||
|
||||
|
||||
def test_delivery_attempt_cap_is_bounded(tmp_path):
|
||||
path = tmp_path / "config.toml"
|
||||
path.write_text("[policy.profiles.bad]\ndelivery_max_attempts=10\n")
|
||||
try:
|
||||
load_profile(path, "bad")
|
||||
except ValueError as exc:
|
||||
assert str(exc) == "delivery_max_attempts must be between 1 and 9"
|
||||
else:
|
||||
raise AssertionError("invalid delivery cap was accepted")
|
||||
|
|
|
|||
|
|
@ -1,7 +1,13 @@
|
|||
import asyncio
|
||||
import json
|
||||
|
||||
from tamq.service import PUSHY_FRAMING_CAPABILITY, Service
|
||||
from tamq.service import (
|
||||
DELIVERY_RELIABILITY_CAPABILITY,
|
||||
IDEMPOTENT_SEND_CAPABILITY,
|
||||
PROTOCOL_CAPTURE_CAPABILITY,
|
||||
PUSHY_FRAMING_CAPABILITY,
|
||||
Service,
|
||||
)
|
||||
from tamq.store import Store
|
||||
|
||||
|
||||
|
|
@ -32,6 +38,9 @@ def test_ping_advertises_non_routable_pushy_framing(tmp_path):
|
|||
await writer.drain()
|
||||
response = json.loads(await reader.readline())
|
||||
assert PUSHY_FRAMING_CAPABILITY in response["capabilities"]
|
||||
assert PROTOCOL_CAPTURE_CAPABILITY in response["capabilities"]
|
||||
assert DELIVERY_RELIABILITY_CAPABILITY in response["capabilities"]
|
||||
assert IDEMPOTENT_SEND_CAPABILITY in response["capabilities"]
|
||||
writer.close()
|
||||
await writer.wait_closed()
|
||||
finally:
|
||||
|
|
|
|||
|
|
@ -8,16 +8,24 @@ def test_replay_reports_batch(tmp_path, monkeypatch, capsys):
|
|||
source = tmp_path / "messages.jsonl"
|
||||
source.write_text(json.dumps({"message_id": "old-1", "sender_repo": "a", "target_repo": "b", "body": "hello"}) + "\n")
|
||||
monkeypatch.setenv("TAMQ_STATE_DIR", str(tmp_path / "state"))
|
||||
monkeypatch.setattr("tamq.cli.validate_targets", lambda targets: None)
|
||||
assert main(["replay", str(source)]) == 0
|
||||
output = json.loads(capsys.readouterr().out)
|
||||
assert output["batch_id"].startswith("replay-")
|
||||
assert output["count"] == 1
|
||||
assert output["deduplicated"] == 0
|
||||
assert main(["replay", str(source)]) == 0
|
||||
repeated = json.loads(capsys.readouterr().out)
|
||||
assert repeated["count"] == 0
|
||||
assert repeated["deduplicated"] == 1
|
||||
assert len(Store(tmp_path / "state" / "tamq.sqlite3").list()) == 1
|
||||
|
||||
|
||||
def test_replay_preserves_endpoint(tmp_path, monkeypatch, capsys):
|
||||
source = tmp_path / "messages.jsonl"
|
||||
source.write_text(json.dumps({"sender_repo": "a", "target_repo": "b", "body": "hello", "endpoint_id": "tmux-amq-42"}) + "\n")
|
||||
monkeypatch.setenv("TAMQ_STATE_DIR", str(tmp_path / "state"))
|
||||
monkeypatch.setattr("tamq.cli.validate_targets", lambda targets: None)
|
||||
assert main(["replay", str(source)]) == 0
|
||||
row = Store(tmp_path / "state" / "tamq.sqlite3").list()[0]
|
||||
assert row["endpoint_id"] == "tmux-amq-42"
|
||||
|
|
@ -27,5 +35,6 @@ def test_replay_rejects_bad_json(tmp_path, monkeypatch, capsys):
|
|||
source = tmp_path / "bad.jsonl"
|
||||
source.write_text("not-json\n")
|
||||
monkeypatch.setenv("TAMQ_STATE_DIR", str(tmp_path / "state"))
|
||||
monkeypatch.setattr("tamq.cli.validate_targets", lambda targets: None)
|
||||
assert main(["replay", str(source)]) == 2
|
||||
assert "cannot replay" in capsys.readouterr().err
|
||||
|
|
|
|||
27
tests/test_retry.py
Normal file
27
tests/test_retry.py
Normal file
|
|
@ -0,0 +1,27 @@
|
|||
from tamq.cli import main
|
||||
from tamq.store import Store
|
||||
|
||||
|
||||
def test_retry_cli_resets_only_terminal_failures(tmp_path, monkeypatch, capsys):
|
||||
state = tmp_path / "state"
|
||||
monkeypatch.setenv("TAMQ_STATE_DIR", str(state))
|
||||
store = Store(state / "tamq.sqlite3")
|
||||
failed = store.add("a", "b", "failed")
|
||||
pending = store.add("a", "b", "pending")
|
||||
store.db.execute(
|
||||
"UPDATE messages SET state='failed',attempt_count=4,last_failure_reason='OSError' "
|
||||
"WHERE message_id=?",
|
||||
(failed,),
|
||||
)
|
||||
store.db.commit()
|
||||
store.close()
|
||||
|
||||
assert main(["retry", failed]) == 0
|
||||
assert capsys.readouterr().out.strip() == failed
|
||||
reopened = Store(state / "tamq.sqlite3")
|
||||
assert reopened.message(failed)["state"] == "pending"
|
||||
assert reopened.message(failed)["attempt_count"] == 0
|
||||
reopened.close()
|
||||
|
||||
assert main(["retry", pending]) == 1
|
||||
assert "not failed" in capsys.readouterr().err
|
||||
|
|
@ -1,3 +1,5 @@
|
|||
import json
|
||||
|
||||
from tamq.control import PaneDisplay
|
||||
from tamq.service import Service
|
||||
from tamq.store import Store
|
||||
|
|
@ -96,6 +98,13 @@ def test_service_trigger_mode_submits_exactly_once(tmp_path, monkeypatch):
|
|||
service._deliver_once()
|
||||
assert control.submitted == [("tamq:repo-b", "From:repo-a/o: go")]
|
||||
assert control.placed == []
|
||||
lifecycle = store.protocol_events(message_id=store.list()[0]["message_id"])
|
||||
assert [event["event_type"] for event in lifecycle] == [
|
||||
"message.accepted",
|
||||
"delivery.attempted",
|
||||
"delivery.injected",
|
||||
]
|
||||
assert lifecycle[-1]["delivery_mode"] == "trigger"
|
||||
|
||||
|
||||
def test_failed_pushy_submission_remains_pending(tmp_path, monkeypatch):
|
||||
|
|
@ -111,6 +120,12 @@ def test_failed_pushy_submission_remains_pending(tmp_path, monkeypatch):
|
|||
Service(store=store, input_grace=0)._deliver_once()
|
||||
|
||||
assert store.list()[0]["state"] == "pending"
|
||||
failure = store.protocol_events(event_type="delivery.failed")[0]
|
||||
assert failure["delivery_mode"] == "pushy"
|
||||
detail = json.loads(failure["detail"])
|
||||
assert detail["reason"] == "RuntimeError"
|
||||
assert detail["attempt"] == 1
|
||||
assert detail["attempt_limit"] == 4
|
||||
|
||||
|
||||
def test_input_delivery_waits_for_new_endpoint_grace(tmp_path, monkeypatch):
|
||||
|
|
@ -126,7 +141,7 @@ def test_input_delivery_waits_for_new_endpoint_grace(tmp_path, monkeypatch):
|
|||
assert store.list()[0]["state"] == "pending"
|
||||
|
||||
|
||||
def test_service_writes_output_once_and_retains_pending_ack(tmp_path, monkeypatch):
|
||||
def test_default_injected_policy_completes_output_delivery_once(tmp_path, monkeypatch):
|
||||
store = Store(tmp_path / "queue.sqlite3")
|
||||
store.register_endpoint("tmux-amq-9-boot", 9, "tamq", ["repo-b"], "output")
|
||||
message_id = store.add("repo-a", "repo-b", "continue")
|
||||
|
|
@ -148,7 +163,7 @@ def test_service_writes_output_once_and_retains_pending_ack(tmp_path, monkeypatc
|
|||
)
|
||||
]
|
||||
row = store.list()[0]
|
||||
assert row["state"] == "pending"
|
||||
assert row["state"] == "injected"
|
||||
assert row["displayed_at"] is not None
|
||||
assert store.db.execute("SELECT COUNT(*) FROM leases").fetchone()[0] == 0
|
||||
|
||||
|
|
@ -166,18 +181,125 @@ def test_failed_terminal_output_remains_undisplayed_and_retryable(tmp_path, monk
|
|||
raise OSError("temporary failure")
|
||||
|
||||
monkeypatch.setattr("tamq.service.write_terminal_output", fail_once)
|
||||
service = Service(store=store)
|
||||
service = Service(store=store, retry_backoff=(0,))
|
||||
|
||||
service._deliver_once()
|
||||
assert store.list()[0]["displayed_at"] is None
|
||||
store.db.execute("UPDATE leases SET expires_at=0")
|
||||
store.db.commit()
|
||||
service._deliver_once()
|
||||
|
||||
assert len(attempts) == 2
|
||||
assert store.list()[0]["displayed_at"] is not None
|
||||
|
||||
|
||||
def test_acknowledged_policy_waits_then_redelivers_same_message(tmp_path, monkeypatch):
|
||||
store = Store(tmp_path / "queue.sqlite3")
|
||||
store.register_endpoint(
|
||||
"ep", 9, "tamq", ["repo-b"], "trigger",
|
||||
delivery_ack_mode="acknowledged",
|
||||
delivery_max_attempts=3,
|
||||
ack_timeout_seconds=10,
|
||||
)
|
||||
message_id = store.add("repo-a", "repo-b", "confirm")
|
||||
monkeypatch.setattr("tamq.service.ControlModeClient", FakeControl)
|
||||
service = Service(store=store, input_grace=0, retry_backoff=(0,))
|
||||
|
||||
service._deliver_once()
|
||||
first_control = FakeControl.instances[-1]
|
||||
assert store.message(message_id)["state"] == "awaiting_ack"
|
||||
assert store.message(message_id)["attempt_count"] == 1
|
||||
assert len(first_control.submitted) == 1
|
||||
service._deliver_once()
|
||||
assert FakeControl.instances[-1].submitted == []
|
||||
|
||||
store.db.execute(
|
||||
"UPDATE messages SET next_attempt_at=0 WHERE message_id=?", (message_id,)
|
||||
)
|
||||
store.db.commit()
|
||||
service._deliver_once()
|
||||
second_control = FakeControl.instances[-1]
|
||||
assert store.message(message_id)["attempt_count"] == 2
|
||||
assert len(second_control.submitted) == 1
|
||||
assert store.acknowledge(message_id) is True
|
||||
service._deliver_once()
|
||||
assert FakeControl.instances[-1].submitted == []
|
||||
|
||||
|
||||
def test_delivery_attempt_cap_is_terminal_and_operator_retry_resets(tmp_path, monkeypatch):
|
||||
store = Store(tmp_path / "queue.sqlite3")
|
||||
store.register_endpoint(
|
||||
"ep", 9, "tamq", ["repo-b"], "pushy", delivery_max_attempts=2
|
||||
)
|
||||
message_id = store.add("repo-a", "repo-b", "fail")
|
||||
|
||||
class FailingControl(FakeControl):
|
||||
def place(self, window, text):
|
||||
raise OSError("no pane")
|
||||
|
||||
monkeypatch.setattr("tamq.service.ControlModeClient", FailingControl)
|
||||
service = Service(store=store, input_grace=0, retry_backoff=(0,))
|
||||
service._deliver_once()
|
||||
service._deliver_once()
|
||||
service._deliver_once()
|
||||
|
||||
row = store.message(message_id)
|
||||
assert row["state"] == "failed"
|
||||
assert row["attempt_count"] == 2
|
||||
assert row["last_failure_reason"] == "OSError"
|
||||
assert store.retry_message(message_id) is True
|
||||
assert store.message(message_id)["state"] == "pending"
|
||||
assert store.message(message_id)["attempt_count"] == 0
|
||||
|
||||
|
||||
def test_ack_timeout_exhaustion_is_terminal_but_late_ack_wins(tmp_path, monkeypatch):
|
||||
store = Store(tmp_path / "queue.sqlite3")
|
||||
store.register_endpoint(
|
||||
"ep", 9, "tamq", ["repo-b"], "trigger",
|
||||
delivery_ack_mode="acknowledged",
|
||||
delivery_max_attempts=2,
|
||||
ack_timeout_seconds=10,
|
||||
)
|
||||
message_id = store.add("repo-a", "repo-b", "confirm")
|
||||
monkeypatch.setattr("tamq.service.ControlModeClient", FakeControl)
|
||||
service = Service(store=store, input_grace=0, retry_backoff=(0,))
|
||||
|
||||
service._deliver_once()
|
||||
for _ in range(2):
|
||||
store.db.execute(
|
||||
"UPDATE messages SET next_attempt_at=0 WHERE message_id=?", (message_id,)
|
||||
)
|
||||
store.db.commit()
|
||||
service._deliver_once()
|
||||
|
||||
row = store.message(message_id)
|
||||
assert row["state"] == "failed"
|
||||
assert row["attempt_count"] == 2
|
||||
assert row["last_failure_reason"] == "ack_timeout"
|
||||
assert store.acknowledge(message_id) is True
|
||||
assert store.message(message_id)["state"] == "acknowledged"
|
||||
assert store.protocol_events(message_id=message_id)[-1]["outcome"] == "late_acknowledged"
|
||||
|
||||
|
||||
def test_expired_delivery_lease_consumes_attempt_and_schedules_retry(tmp_path):
|
||||
store = Store(tmp_path / "queue.sqlite3")
|
||||
message_id = store.add("repo-a", "repo-b", "lease")
|
||||
claim = store.claim_delivery(
|
||||
message_id,
|
||||
"ep",
|
||||
attempt_limit=2,
|
||||
retry_delay=7,
|
||||
delivery_mode="output",
|
||||
ttl=1,
|
||||
now=10,
|
||||
)
|
||||
assert claim is not None
|
||||
assert store.expire_delivery_leases(now=12) == 1
|
||||
row = store.message(message_id)
|
||||
assert row["state"] == "pending"
|
||||
assert row["attempt_count"] == 1
|
||||
assert row["last_failure_reason"] == "lease_expired"
|
||||
assert row["next_attempt_at"] == 19
|
||||
|
||||
|
||||
def test_service_disconnects_disappeared_tmux_endpoint(tmp_path, monkeypatch):
|
||||
store = Store(tmp_path / "queue.sqlite3")
|
||||
store.register_endpoint("tmux-amq-9-boot", 9, "tamq", ["repo-b"])
|
||||
|
|
|
|||
|
|
@ -41,10 +41,58 @@ def test_store_migrates_legacy_endpoint_rows_to_manual_delivery(tmp_path):
|
|||
assert store.endpoints()[0]["delivery_mode"] == "manual"
|
||||
assert store.db.execute(
|
||||
"SELECT value FROM metadata WHERE key='schema_version'"
|
||||
).fetchone()[0] == "4"
|
||||
).fetchone()[0] == "6"
|
||||
assert "displayed_at" in {
|
||||
row["name"] for row in store.db.execute("PRAGMA table_info(messages)")
|
||||
}
|
||||
assert store.db.execute(
|
||||
"SELECT COUNT(*) FROM protocol_events"
|
||||
).fetchone()[0] == 0
|
||||
|
||||
|
||||
def test_protocol_ledger_is_atomic_and_not_duplicated_on_reopen(tmp_path):
|
||||
path = tmp_path / "queue.sqlite3"
|
||||
store = Store(path)
|
||||
message_id = store.add(
|
||||
"source", "target", "first\nsecond", provenance="worker_output"
|
||||
)
|
||||
event = store.protocol_events(message_id=message_id)[0]
|
||||
assert event["event_type"] == "message.accepted"
|
||||
assert event["repo"] == "source"
|
||||
assert event["peer_repo"] == "target"
|
||||
assert json.loads(event["detail"]) == {"line_count": 2}
|
||||
store.close()
|
||||
|
||||
reopened = Store(path)
|
||||
assert len(reopened.protocol_events(message_id=message_id)) == 1
|
||||
reopened.close()
|
||||
|
||||
|
||||
def test_schema_v5_backfills_existing_message_lifecycle(tmp_path):
|
||||
path = tmp_path / "legacy-messages.sqlite3"
|
||||
store = Store(path)
|
||||
message_id = store.add("a", "b", "old")
|
||||
store.db.execute("DELETE FROM protocol_events")
|
||||
store.db.execute(
|
||||
"UPDATE messages SET displayed_at=2,injected_at=3,acknowledged_at=4 "
|
||||
"WHERE message_id=?",
|
||||
(message_id,),
|
||||
)
|
||||
store.db.commit()
|
||||
store.close()
|
||||
|
||||
migrated = Store(path)
|
||||
assert [row["event_type"] for row in migrated.protocol_events()] == [
|
||||
"delivery.displayed",
|
||||
"delivery.injected",
|
||||
"message.acknowledged",
|
||||
"message.accepted",
|
||||
]
|
||||
assert all(
|
||||
json.loads(row["detail"])["backfilled"] is True
|
||||
for row in migrated.protocol_events()
|
||||
)
|
||||
migrated.close()
|
||||
|
||||
|
||||
def test_mark_displayed_releases_lease_without_acknowledging(tmp_path):
|
||||
|
|
|
|||
|
|
@ -4,11 +4,11 @@ type: workplan
|
|||
title: "Coordination-engine adapter boundary"
|
||||
domain: communication
|
||||
repo: tmux-amq
|
||||
status: active
|
||||
status: finished
|
||||
owner: codex
|
||||
topic_slug: coulomb-social
|
||||
created: "2026-08-24"
|
||||
updated: "2026-08-24"
|
||||
updated: "2026-08-26"
|
||||
state_hub_workstream_id: "48da5fa7-b7bf-5d13-8edc-b90f266e241c"
|
||||
---
|
||||
|
||||
|
|
@ -22,7 +22,7 @@ workflow/orchestration policy in coordination-engine.
|
|||
|
||||
```task
|
||||
id: TAMQ-WP-0002-T01
|
||||
status: todo
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "a222dab7-edea-58c6-9dab-d684eb65c412"
|
||||
```
|
||||
|
|
@ -31,11 +31,17 @@ Define request, response, identity, delivery, acknowledgement, retry, and
|
|||
failure semantics for a coordination-engine client. Resolve protocol-version
|
||||
negotiation and document which system owns each state transition.
|
||||
|
||||
The implemented contract is `spec/coordination-engine-adapter-v0.1.md`. It
|
||||
defines same-user socket authentication, major-version and capability
|
||||
negotiation, exact endpoint resolution, message/correlation identity,
|
||||
idempotency conflicts, bounded delivery states, ack timeouts, late
|
||||
acknowledgement, recovery, and the TAMQ/coordination ownership boundary.
|
||||
|
||||
## Implement the coordination-engine client adapter
|
||||
|
||||
```task
|
||||
id: TAMQ-WP-0002-T02
|
||||
status: wait
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "195db613-05a3-58d8-9e94-2db34920576f"
|
||||
```
|
||||
|
|
@ -43,11 +49,18 @@ state_hub_task_id: "195db613-05a3-58d8-9e94-2db34920576f"
|
|||
Implement the adapter against the stable socket contract without importing
|
||||
tmux/control-mode concerns into coordination-engine. This task waits for T01.
|
||||
|
||||
`tamq.client` provides a transport-only async client and
|
||||
`CoordinationEngineAdapter`. It opens a fresh socket connection per operation,
|
||||
resolves one live repository endpoint, maps the coordination lease ID to a
|
||||
durable idempotency key, preserves trigger correlation metadata, and exposes
|
||||
receipt, acknowledgement, and terminal retry operations. Protocol constants
|
||||
are separate from the service and terminal implementation.
|
||||
|
||||
## Prove interoperability and recovery
|
||||
|
||||
```task
|
||||
id: TAMQ-WP-0002-T03
|
||||
status: wait
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "23c7f6c7-be48-5b6c-b85c-ae05d82cc277"
|
||||
```
|
||||
|
|
@ -55,3 +68,18 @@ state_hub_task_id: "23c7f6c7-be48-5b6c-b85c-ae05d82cc277"
|
|||
Add end-to-end coverage for delivery, acknowledgement, reconnect, replay,
|
||||
deduplication, incompatible protocol versions, and endpoint disappearance.
|
||||
Update both repositories' operator documentation with the verified workflow.
|
||||
|
||||
Unix-socket integration tests cover negotiation, identical-wake deduplication,
|
||||
idempotency conflict, durable receipt recovery after service restart,
|
||||
incompatible protocol, and disappeared endpoints. Reliability tests cover
|
||||
delivery, explicit and late acknowledgement, same-ID ack-timeout redelivery,
|
||||
lease expiry, exhaustion, and retry reset. JSONL replay now uses deterministic
|
||||
idempotency while retaining normal gita validation. TAMQ documentation and the
|
||||
coordination-engine worker-service specification link the verified contract.
|
||||
|
||||
## Residuals
|
||||
|
||||
This workplan does not implement coordination-engine's trigger observer,
|
||||
coordination leases, actionability decisions, checkpoints, or State Hub
|
||||
projection. Those remain owned by `COORDINATION-WP-0003` and consume this
|
||||
adapter rather than expanding TAMQ's scope.
|
||||
|
|
|
|||
|
|
@ -4,11 +4,11 @@ type: workplan
|
|||
title: "Delivery reliability and practical integration evidence"
|
||||
domain: communication
|
||||
repo: tmux-amq
|
||||
status: active
|
||||
status: finished
|
||||
owner: codex
|
||||
topic_slug: coulomb-social
|
||||
created: "2026-08-24"
|
||||
updated: "2026-08-24"
|
||||
updated: "2026-08-26"
|
||||
state_hub_workstream_id: "7961275b-8f1f-5827-b9fc-46b3ac35fb73"
|
||||
---
|
||||
|
||||
|
|
@ -22,7 +22,7 @@ stable coordination-engine dependency.
|
|||
|
||||
```task
|
||||
id: TAMQ-WP-0003-T01
|
||||
status: todo
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "52c43745-5c28-599e-b74c-0386d5b51e68"
|
||||
```
|
||||
|
|
@ -31,11 +31,18 @@ Persist attempt counts and failure reasons, apply the selected policy profile's
|
|||
retry cap, define lease-expiry behavior, and introduce an inspectable terminal
|
||||
failure state. Prove restart-safe behavior and avoid tight retry loops.
|
||||
|
||||
Completed with schema-v6 persistent attempt, deadline, and failure fields.
|
||||
Lease acquisition consumes an attempt; write failure or lease expiry releases
|
||||
ownership into configured bounded backoff. The selected endpoint policy caps
|
||||
attempts from one to nine, exhaustion becomes `failed`, and `tamq retry` is the
|
||||
explicit recovery action. Protocol events retain attempt, cap, reason, and
|
||||
retry timing without raw terminal output.
|
||||
|
||||
## Enforce acknowledgement semantics
|
||||
|
||||
```task
|
||||
id: TAMQ-WP-0003-T02
|
||||
status: wait
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "ff88d6c9-f4d7-52c7-9645-7c3a7a34ccb4"
|
||||
```
|
||||
|
|
@ -44,6 +51,13 @@ Make `delivery_ack_mode` control whether injection completes delivery or waits
|
|||
for explicit recipient acknowledgement. Specify timeout, redelivery, duplicate,
|
||||
and late-acknowledgement behavior. This task follows the state model from T01.
|
||||
|
||||
Completed for every automatic delivery mode. `injected` completes after a
|
||||
successful terminal write. `acknowledged` enters `awaiting_ack`, schedules
|
||||
same-message-ID redelivery after the configured deadline, records `ack_timeout`
|
||||
on retries and exhaustion, and accepts a late acknowledgement even after
|
||||
`failed`. The contract warns that redelivery can duplicate terminal
|
||||
presentation and never represents task completion.
|
||||
|
||||
## Add real tmux and PTY lifecycle coverage
|
||||
|
||||
```task
|
||||
|
|
@ -70,7 +84,7 @@ entirely as documented.
|
|||
|
||||
```task
|
||||
id: TAMQ-WP-0003-T04
|
||||
status: wait
|
||||
status: done
|
||||
priority: medium
|
||||
state_hub_task_id: "44939829-a52e-582a-b06b-60337b36080e"
|
||||
```
|
||||
|
|
@ -78,3 +92,14 @@ state_hub_task_id: "44939829-a52e-582a-b06b-60337b36080e"
|
|||
Run the integration suite in Forgejo CI with explicit tmux and gita setup. Update
|
||||
README claims about retries, acknowledgement, prerequisites, and maturity from
|
||||
verified behavior. This task follows T01-T03.
|
||||
|
||||
Forgejo CI now runs `make check`, including generated-gita and isolated tmux
|
||||
installation fixtures, compile/diff checks, and CLI/capture help smokes on
|
||||
Python 3.11. README, SCOPE, configuration examples, and the agent introduction
|
||||
describe the implemented state machine and terminal-comprehension boundary.
|
||||
|
||||
## Residuals
|
||||
|
||||
Long-duration and arbitrary-terminal soak coverage remains a maturity gate in
|
||||
SCOPE rather than a missing part of this bounded reliability contract. The
|
||||
coordination runtime above TAMQ remains owned by coordination-engine.
|
||||
|
|
|
|||
86
workplans/TAMQ-WP-0016-structured-protocol-capture.md
Normal file
86
workplans/TAMQ-WP-0016-structured-protocol-capture.md
Normal file
|
|
@ -0,0 +1,86 @@
|
|||
---
|
||||
id: TAMQ-WP-0016
|
||||
type: workplan
|
||||
title: "Structured communication protocol capture"
|
||||
domain: communication
|
||||
repo: tmux-amq
|
||||
status: finished
|
||||
owner: codex
|
||||
topic_slug: structured-protocol-capture
|
||||
created: "2026-08-25"
|
||||
updated: "2026-08-25"
|
||||
---
|
||||
|
||||
# Structured communication protocol capture
|
||||
|
||||
Retain reviewable evidence of TAMQ exchanges so agent onboarding and protocol
|
||||
friction can be assessed without recording arbitrary terminal sessions.
|
||||
|
||||
## Define the evidence and privacy boundary
|
||||
|
||||
```task
|
||||
id: TAMQ-WP-0016-T01
|
||||
status: done
|
||||
priority: high
|
||||
```
|
||||
|
||||
Capture only TAMQ protocol facts: addressed messages, provenance, worker block
|
||||
boundaries, allowlisted commands, line-limit decisions, endpoint lifecycle, and
|
||||
delivery outcomes. Do not capture unrelated pane output or ordinary shell
|
||||
input. Retain a message body once at acceptance rather than duplicating it at
|
||||
each lifecycle event.
|
||||
|
||||
## Add an append-only protocol ledger
|
||||
|
||||
```task
|
||||
id: TAMQ-WP-0016-T02
|
||||
status: done
|
||||
priority: high
|
||||
```
|
||||
|
||||
Add structured protocol events to SQLite schema version 5. Make acceptance
|
||||
events atomic with message admission, represent existing message lifecycle
|
||||
history during migration, and retain stable message/endpoint/repository links
|
||||
for filtering.
|
||||
|
||||
## Instrument onboarding-sensitive transitions
|
||||
|
||||
```task
|
||||
id: TAMQ-WP-0016-T03
|
||||
status: done
|
||||
priority: high
|
||||
```
|
||||
|
||||
Record operator versus worker origin, worker block start and close reason,
|
||||
invalid targets, limit blocks, operator-only command results, delivery mode,
|
||||
attempt, success, and failure class. Never store unrecognized command text or
|
||||
raw terminal output in event details.
|
||||
|
||||
## Provide a review and analysis path
|
||||
|
||||
```task
|
||||
id: TAMQ-WP-0016-T04
|
||||
status: done
|
||||
priority: medium
|
||||
```
|
||||
|
||||
Add `tamq capture` with repository, endpoint, message, event-type, and bounded
|
||||
event-count filters. Render a summarized Markdown review by default and stable
|
||||
JSONL for analysis. Document the workflow and retention/privacy boundary in the
|
||||
agent introduction, README, CLI help, and scope assessment.
|
||||
|
||||
## Evidence
|
||||
|
||||
- Schema migration backfills existing acceptance/display/injection/ack events
|
||||
without duplicating newly captured events on later opens.
|
||||
- Focused store, broker, service, renderer, and CLI tests cover atomic capture,
|
||||
worker termination evidence, provenance, failure classes, filtering, and
|
||||
Markdown/JSONL output.
|
||||
- `make check` passes the complete suite.
|
||||
|
||||
## Residuals
|
||||
|
||||
The capture deliberately describes transport behavior, not recipient
|
||||
comprehension or authorization. Long-term event retention and automated
|
||||
protocol-quality recommendations can be added after real captures show which
|
||||
summaries are useful.
|
||||
Loading…
Add table
Add a link
Reference in a new issue