QONTO-WP-0003-T06: MCP smoke script + operator runbook update

Add scripts/smoke_mcp.py, same shape as smoke_rest_api.py (random port,
subprocess-launch against QONTO_FIXTURE_DIR, wait on /v1/health, assert,
clean teardown), but goes further: generates a fresh
QONTO_ASSISTANT_MCP_TOKEN per run and connects through it with the mcp
SDK's streamablehttp_client, so the smoke exercises T03's bearer-token gate
instead of bypassing it. Lists tools, calls all four, and confirms an
out-of-catalog tool name comes back as a normal isError result through the
real wire protocol rather than a crash.

Extend docs/operator-runbook.md with a "One-command MCP smoke" section next
to the REST one; cross-link from docs/mcp-integration.md's manual smoke
walkthrough so the two don't drift.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
tegwick 2026-07-23 11:04:39 +02:00
parent c1e33decd9
commit 0dc001bf53
4 changed files with 208 additions and 1 deletions

View file

@ -155,6 +155,11 @@ the harness allows, per the shared `PolicyEngine.decide()` path.
## Local smoke: start with auth enabled, connect with only this snippet
For a one-command version of everything below (random port, generated
token, all four tools, plus the out-of-catalog deny check), see
`scripts/smoke_mcp.py` in `docs/operator-runbook.md`. The manual walkthrough
here is for when you want to see each step explicitly.
```bash
export QONTO_FIXTURE_DIR=tests/fixtures/qonto
export QONTO_ASSISTANT_MCP_TOKEN=dev-local-token

View file

@ -109,6 +109,24 @@ This starts the service on a random local port against the fixture payloads,
checks `/v1/health`, `/v1/accounts`, a recent `31`-day snapshot, and a wider
`90`-day cost-review snapshot, then shuts the process down.
## One-command MCP smoke
```bash
../state-hub/.venv/bin/python scripts/smoke_mcp.py \
--python ../state-hub/.venv/bin/python
```
Same shape as the REST smoke: starts the service on a random local port
against the fixture payloads, this time with `QONTO_ASSISTANT_MCP_TOKEN` set
to a freshly generated token so the auth layer (`docs/mcp-integration.md`) is
exercised too, not bypassed. Connects with the `mcp` SDK's
`streamablehttp_client`, lists tools, calls `qonto_ping`,
`qonto_org_summary`, `qonto_list_transactions`, and
`qonto_cost_run_rate_hints`, then confirms an out-of-catalog tool name
(`qonto_transfer_funds`, never registered) comes back as a normal
`isError` result rather than a crash or a policy bypass. No real Qonto
credentials involved.
## Example calls
Minimal local call:

159
scripts/smoke_mcp.py Normal file
View file

@ -0,0 +1,159 @@
#!/usr/bin/env python3
from __future__ import annotations
import argparse
import asyncio
import json
import os
import secrets
import socket
import subprocess
import sys
import time
import urllib.error
import urllib.request
from pathlib import Path
from mcp import ClientSession
from mcp.client.streamable_http import streamablehttp_client
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description="Run a local MCP smoke test for qonto-assistant.")
parser.add_argument(
"--python",
default=sys.executable,
help="Python interpreter used to start qonto_assistant.main",
)
parser.add_argument(
"--fixture-dir",
default=str(Path(__file__).resolve().parents[1] / "tests" / "fixtures" / "qonto"),
help="Fixture directory containing organization.json and transactions.json",
)
parser.add_argument(
"--startup-timeout",
type=float,
default=15.0,
help="Seconds to wait for the local service to start",
)
return parser.parse_args()
def free_port() -> int:
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
sock.bind(("127.0.0.1", 0))
return int(sock.getsockname()[1])
def wait_for_health(base_url: str, *, timeout_seconds: float) -> None:
deadline = time.monotonic() + timeout_seconds
last_error: Exception | None = None
while time.monotonic() < deadline:
try:
request = urllib.request.Request(f"{base_url}/v1/health")
with urllib.request.urlopen(request, timeout=5) as response:
json.load(response)
return
except Exception as exc: # noqa: BLE001
last_error = exc
time.sleep(0.25)
raise RuntimeError(f"Timed out waiting for {base_url}/v1/health: {last_error}")
async def run_mcp_checks(base_url: str, *, token: str) -> dict[str, object]:
headers = {
"Authorization": f"Bearer {token}",
"X-Actor-ID": "smoke",
"X-Tenant-ID": "binky",
}
async with streamablehttp_client(f"{base_url}/mcp", headers=headers) as (read, write, _):
async with ClientSession(read, write) as session:
await session.initialize()
tools = {tool.name for tool in (await session.list_tools()).tools}
expected_tools = {
"qonto_ping",
"qonto_org_summary",
"qonto_list_transactions",
"qonto_cost_run_rate_hints",
}
assert tools == expected_tools, f"unexpected tool catalog: {tools}"
ping = await session.call_tool("qonto_ping", {})
assert ping.isError is False
assert ping.structuredContent["status"] == "ok"
org_summary = await session.call_tool("qonto_org_summary", {})
assert org_summary.isError is False
assert org_summary.structuredContent["organization"]["name"] == "Binky Hedgehog GmbH"
assert org_summary.structuredContent["accounts"][0]["iban_last4"] == "6810"
transactions = await session.call_tool(
"qonto_list_transactions", {"window_days": 31, "page_size": 50}
)
assert transactions.isError is False
cost_hints = await session.call_tool(
"qonto_cost_run_rate_hints", {"window_days": 90, "page_size": 50}
)
assert cost_hints.isError is False
assert cost_hints.structuredContent["cost_run_rate_hints"]["recurring_debits"][0]["label"] == "HUB31"
# Deny behavior for an out-of-catalog / spend-shaped tool name:
# this tool was never registered, so the low-level protocol
# handler must reject it as a normal error result, never a
# crash and never a policy bypass.
denied = await session.call_tool("qonto_transfer_funds", {})
assert denied.isError is True
return {
"tools": sorted(tools),
"org_summary": org_summary.structuredContent,
"cost_hints": cost_hints.structuredContent,
"out_of_catalog_denied": denied.isError,
}
def main() -> int:
args = parse_args()
repo_root = Path(__file__).resolve().parents[1]
port = free_port()
base_url = f"http://127.0.0.1:{port}"
token = secrets.token_urlsafe(16)
env = os.environ.copy()
env["PYTHONPATH"] = str(repo_root / "src")
env["QONTO_ASSISTANT_HOST"] = "127.0.0.1"
env["QONTO_ASSISTANT_PORT"] = str(port)
env["QONTO_FIXTURE_DIR"] = str(Path(args.fixture_dir).resolve())
env["QONTO_ASSISTANT_MCP_TOKEN"] = token
process = subprocess.Popen( # noqa: S603
[args.python, "-m", "qonto_assistant.main"],
cwd=repo_root,
env=env,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
text=True,
)
try:
wait_for_health(base_url, timeout_seconds=args.startup_timeout)
result = asyncio.run(run_mcp_checks(base_url, token=token))
print(json.dumps(result, indent=2))
return 0
finally:
process.terminate()
try:
process.wait(timeout=5)
except subprocess.TimeoutExpired:
process.kill()
process.wait(timeout=5)
if process.stdout is not None:
output = process.stdout.read().strip()
if output:
sys.stderr.write(output + "\n")
if __name__ == "__main__":
raise SystemExit(main())

View file

@ -256,7 +256,7 @@ write exists to accidentally wire up. `pytest` → `31 passed`.
```task
id: QONTO-WP-0003-T06
status: todo
status: done
priority: medium
state_hub_task_id: "1093ec79-eb16-4797-8c9c-5423a925db77"
```
@ -270,6 +270,31 @@ alongside the existing REST section.
Done when: smoke script passes locally against fixtures without real Qonto
credentials, matching the Phase 1 REST smoke pattern.
**Done 2026-07-23:** Added `scripts/smoke_mcp.py`, same shape as
`scripts/smoke_rest_api.py` (random free port, subprocess-launch the
service against `QONTO_FIXTURE_DIR`, wait on `/v1/health`, assert, tear
down cleanly). Goes one step further than the REST smoke since MCP now has
an auth layer REST doesn't: generates a fresh `QONTO_ASSISTANT_MCP_TOKEN`
per run and connects with the `mcp` SDK's `streamablehttp_client` using it,
so the smoke exercises T03's bearer-token gate rather than bypassing it.
Lists tools and asserts the exact expected catalog; calls `qonto_ping`,
`qonto_org_summary`, `qonto_list_transactions`, and
`qonto_cost_run_rate_hints` against fixture data with the same balance/
recurring-debit assertions the REST smoke makes; calls a never-registered
tool name (`qonto_transfer_funds`) and asserts it comes back as a normal
`isError` result through the real wire protocol, not a crash — confirming
the low-level MCP protocol handler rejects unknown tools cleanly rather
than falling through to anything policy-adjacent.
Extended `docs/operator-runbook.md` with a "One-command MCP smoke" section
next to the existing REST one, and cross-linked it from
`docs/mcp-integration.md`'s manual smoke walkthrough so the two don't drift
out of sync.
Verified: `python3 scripts/smoke_mcp.py --python ../state-hub/.venv/bin/python`
exits `0` against fixtures, no real Qonto credentials. `pytest``31 passed`
(unchanged); `python3 -m compileall src tests scripts`.
## Task: Closure review
```task