QONTO-WP-0003-T06: MCP smoke script + operator runbook update
Add scripts/smoke_mcp.py, same shape as smoke_rest_api.py (random port, subprocess-launch against QONTO_FIXTURE_DIR, wait on /v1/health, assert, clean teardown), but goes further: generates a fresh QONTO_ASSISTANT_MCP_TOKEN per run and connects through it with the mcp SDK's streamablehttp_client, so the smoke exercises T03's bearer-token gate instead of bypassing it. Lists tools, calls all four, and confirms an out-of-catalog tool name comes back as a normal isError result through the real wire protocol rather than a crash. Extend docs/operator-runbook.md with a "One-command MCP smoke" section next to the REST one; cross-link from docs/mcp-integration.md's manual smoke walkthrough so the two don't drift. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
parent
c1e33decd9
commit
0dc001bf53
4 changed files with 208 additions and 1 deletions
|
|
@ -155,6 +155,11 @@ the harness allows, per the shared `PolicyEngine.decide()` path.
|
||||||
|
|
||||||
## Local smoke: start with auth enabled, connect with only this snippet
|
## Local smoke: start with auth enabled, connect with only this snippet
|
||||||
|
|
||||||
|
For a one-command version of everything below (random port, generated
|
||||||
|
token, all four tools, plus the out-of-catalog deny check), see
|
||||||
|
`scripts/smoke_mcp.py` in `docs/operator-runbook.md`. The manual walkthrough
|
||||||
|
here is for when you want to see each step explicitly.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
export QONTO_FIXTURE_DIR=tests/fixtures/qonto
|
export QONTO_FIXTURE_DIR=tests/fixtures/qonto
|
||||||
export QONTO_ASSISTANT_MCP_TOKEN=dev-local-token
|
export QONTO_ASSISTANT_MCP_TOKEN=dev-local-token
|
||||||
|
|
|
||||||
|
|
@ -109,6 +109,24 @@ This starts the service on a random local port against the fixture payloads,
|
||||||
checks `/v1/health`, `/v1/accounts`, a recent `31`-day snapshot, and a wider
|
checks `/v1/health`, `/v1/accounts`, a recent `31`-day snapshot, and a wider
|
||||||
`90`-day cost-review snapshot, then shuts the process down.
|
`90`-day cost-review snapshot, then shuts the process down.
|
||||||
|
|
||||||
|
## One-command MCP smoke
|
||||||
|
|
||||||
|
```bash
|
||||||
|
../state-hub/.venv/bin/python scripts/smoke_mcp.py \
|
||||||
|
--python ../state-hub/.venv/bin/python
|
||||||
|
```
|
||||||
|
|
||||||
|
Same shape as the REST smoke: starts the service on a random local port
|
||||||
|
against the fixture payloads, this time with `QONTO_ASSISTANT_MCP_TOKEN` set
|
||||||
|
to a freshly generated token so the auth layer (`docs/mcp-integration.md`) is
|
||||||
|
exercised too, not bypassed. Connects with the `mcp` SDK's
|
||||||
|
`streamablehttp_client`, lists tools, calls `qonto_ping`,
|
||||||
|
`qonto_org_summary`, `qonto_list_transactions`, and
|
||||||
|
`qonto_cost_run_rate_hints`, then confirms an out-of-catalog tool name
|
||||||
|
(`qonto_transfer_funds`, never registered) comes back as a normal
|
||||||
|
`isError` result rather than a crash or a policy bypass. No real Qonto
|
||||||
|
credentials involved.
|
||||||
|
|
||||||
## Example calls
|
## Example calls
|
||||||
|
|
||||||
Minimal local call:
|
Minimal local call:
|
||||||
|
|
|
||||||
159
scripts/smoke_mcp.py
Normal file
159
scripts/smoke_mcp.py
Normal file
|
|
@ -0,0 +1,159 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import asyncio
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import secrets
|
||||||
|
import socket
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
|
import urllib.error
|
||||||
|
import urllib.request
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from mcp import ClientSession
|
||||||
|
from mcp.client.streamable_http import streamablehttp_client
|
||||||
|
|
||||||
|
|
||||||
|
def parse_args() -> argparse.Namespace:
|
||||||
|
parser = argparse.ArgumentParser(description="Run a local MCP smoke test for qonto-assistant.")
|
||||||
|
parser.add_argument(
|
||||||
|
"--python",
|
||||||
|
default=sys.executable,
|
||||||
|
help="Python interpreter used to start qonto_assistant.main",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--fixture-dir",
|
||||||
|
default=str(Path(__file__).resolve().parents[1] / "tests" / "fixtures" / "qonto"),
|
||||||
|
help="Fixture directory containing organization.json and transactions.json",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--startup-timeout",
|
||||||
|
type=float,
|
||||||
|
default=15.0,
|
||||||
|
help="Seconds to wait for the local service to start",
|
||||||
|
)
|
||||||
|
return parser.parse_args()
|
||||||
|
|
||||||
|
|
||||||
|
def free_port() -> int:
|
||||||
|
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
|
||||||
|
sock.bind(("127.0.0.1", 0))
|
||||||
|
return int(sock.getsockname()[1])
|
||||||
|
|
||||||
|
|
||||||
|
def wait_for_health(base_url: str, *, timeout_seconds: float) -> None:
|
||||||
|
deadline = time.monotonic() + timeout_seconds
|
||||||
|
last_error: Exception | None = None
|
||||||
|
while time.monotonic() < deadline:
|
||||||
|
try:
|
||||||
|
request = urllib.request.Request(f"{base_url}/v1/health")
|
||||||
|
with urllib.request.urlopen(request, timeout=5) as response:
|
||||||
|
json.load(response)
|
||||||
|
return
|
||||||
|
except Exception as exc: # noqa: BLE001
|
||||||
|
last_error = exc
|
||||||
|
time.sleep(0.25)
|
||||||
|
raise RuntimeError(f"Timed out waiting for {base_url}/v1/health: {last_error}")
|
||||||
|
|
||||||
|
|
||||||
|
async def run_mcp_checks(base_url: str, *, token: str) -> dict[str, object]:
|
||||||
|
headers = {
|
||||||
|
"Authorization": f"Bearer {token}",
|
||||||
|
"X-Actor-ID": "smoke",
|
||||||
|
"X-Tenant-ID": "binky",
|
||||||
|
}
|
||||||
|
async with streamablehttp_client(f"{base_url}/mcp", headers=headers) as (read, write, _):
|
||||||
|
async with ClientSession(read, write) as session:
|
||||||
|
await session.initialize()
|
||||||
|
|
||||||
|
tools = {tool.name for tool in (await session.list_tools()).tools}
|
||||||
|
expected_tools = {
|
||||||
|
"qonto_ping",
|
||||||
|
"qonto_org_summary",
|
||||||
|
"qonto_list_transactions",
|
||||||
|
"qonto_cost_run_rate_hints",
|
||||||
|
}
|
||||||
|
assert tools == expected_tools, f"unexpected tool catalog: {tools}"
|
||||||
|
|
||||||
|
ping = await session.call_tool("qonto_ping", {})
|
||||||
|
assert ping.isError is False
|
||||||
|
assert ping.structuredContent["status"] == "ok"
|
||||||
|
|
||||||
|
org_summary = await session.call_tool("qonto_org_summary", {})
|
||||||
|
assert org_summary.isError is False
|
||||||
|
assert org_summary.structuredContent["organization"]["name"] == "Binky Hedgehog GmbH"
|
||||||
|
assert org_summary.structuredContent["accounts"][0]["iban_last4"] == "6810"
|
||||||
|
|
||||||
|
transactions = await session.call_tool(
|
||||||
|
"qonto_list_transactions", {"window_days": 31, "page_size": 50}
|
||||||
|
)
|
||||||
|
assert transactions.isError is False
|
||||||
|
|
||||||
|
cost_hints = await session.call_tool(
|
||||||
|
"qonto_cost_run_rate_hints", {"window_days": 90, "page_size": 50}
|
||||||
|
)
|
||||||
|
assert cost_hints.isError is False
|
||||||
|
assert cost_hints.structuredContent["cost_run_rate_hints"]["recurring_debits"][0]["label"] == "HUB31"
|
||||||
|
|
||||||
|
# Deny behavior for an out-of-catalog / spend-shaped tool name:
|
||||||
|
# this tool was never registered, so the low-level protocol
|
||||||
|
# handler must reject it as a normal error result, never a
|
||||||
|
# crash and never a policy bypass.
|
||||||
|
denied = await session.call_tool("qonto_transfer_funds", {})
|
||||||
|
assert denied.isError is True
|
||||||
|
|
||||||
|
return {
|
||||||
|
"tools": sorted(tools),
|
||||||
|
"org_summary": org_summary.structuredContent,
|
||||||
|
"cost_hints": cost_hints.structuredContent,
|
||||||
|
"out_of_catalog_denied": denied.isError,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
args = parse_args()
|
||||||
|
repo_root = Path(__file__).resolve().parents[1]
|
||||||
|
port = free_port()
|
||||||
|
base_url = f"http://127.0.0.1:{port}"
|
||||||
|
token = secrets.token_urlsafe(16)
|
||||||
|
|
||||||
|
env = os.environ.copy()
|
||||||
|
env["PYTHONPATH"] = str(repo_root / "src")
|
||||||
|
env["QONTO_ASSISTANT_HOST"] = "127.0.0.1"
|
||||||
|
env["QONTO_ASSISTANT_PORT"] = str(port)
|
||||||
|
env["QONTO_FIXTURE_DIR"] = str(Path(args.fixture_dir).resolve())
|
||||||
|
env["QONTO_ASSISTANT_MCP_TOKEN"] = token
|
||||||
|
|
||||||
|
process = subprocess.Popen( # noqa: S603
|
||||||
|
[args.python, "-m", "qonto_assistant.main"],
|
||||||
|
cwd=repo_root,
|
||||||
|
env=env,
|
||||||
|
stdout=subprocess.PIPE,
|
||||||
|
stderr=subprocess.STDOUT,
|
||||||
|
text=True,
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
wait_for_health(base_url, timeout_seconds=args.startup_timeout)
|
||||||
|
result = asyncio.run(run_mcp_checks(base_url, token=token))
|
||||||
|
print(json.dumps(result, indent=2))
|
||||||
|
return 0
|
||||||
|
finally:
|
||||||
|
process.terminate()
|
||||||
|
try:
|
||||||
|
process.wait(timeout=5)
|
||||||
|
except subprocess.TimeoutExpired:
|
||||||
|
process.kill()
|
||||||
|
process.wait(timeout=5)
|
||||||
|
|
||||||
|
if process.stdout is not None:
|
||||||
|
output = process.stdout.read().strip()
|
||||||
|
if output:
|
||||||
|
sys.stderr.write(output + "\n")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
|
|
@ -256,7 +256,7 @@ write exists to accidentally wire up. `pytest` → `31 passed`.
|
||||||
|
|
||||||
```task
|
```task
|
||||||
id: QONTO-WP-0003-T06
|
id: QONTO-WP-0003-T06
|
||||||
status: todo
|
status: done
|
||||||
priority: medium
|
priority: medium
|
||||||
state_hub_task_id: "1093ec79-eb16-4797-8c9c-5423a925db77"
|
state_hub_task_id: "1093ec79-eb16-4797-8c9c-5423a925db77"
|
||||||
```
|
```
|
||||||
|
|
@ -270,6 +270,31 @@ alongside the existing REST section.
|
||||||
Done when: smoke script passes locally against fixtures without real Qonto
|
Done when: smoke script passes locally against fixtures without real Qonto
|
||||||
credentials, matching the Phase 1 REST smoke pattern.
|
credentials, matching the Phase 1 REST smoke pattern.
|
||||||
|
|
||||||
|
**Done 2026-07-23:** Added `scripts/smoke_mcp.py`, same shape as
|
||||||
|
`scripts/smoke_rest_api.py` (random free port, subprocess-launch the
|
||||||
|
service against `QONTO_FIXTURE_DIR`, wait on `/v1/health`, assert, tear
|
||||||
|
down cleanly). Goes one step further than the REST smoke since MCP now has
|
||||||
|
an auth layer REST doesn't: generates a fresh `QONTO_ASSISTANT_MCP_TOKEN`
|
||||||
|
per run and connects with the `mcp` SDK's `streamablehttp_client` using it,
|
||||||
|
so the smoke exercises T03's bearer-token gate rather than bypassing it.
|
||||||
|
Lists tools and asserts the exact expected catalog; calls `qonto_ping`,
|
||||||
|
`qonto_org_summary`, `qonto_list_transactions`, and
|
||||||
|
`qonto_cost_run_rate_hints` against fixture data with the same balance/
|
||||||
|
recurring-debit assertions the REST smoke makes; calls a never-registered
|
||||||
|
tool name (`qonto_transfer_funds`) and asserts it comes back as a normal
|
||||||
|
`isError` result through the real wire protocol, not a crash — confirming
|
||||||
|
the low-level MCP protocol handler rejects unknown tools cleanly rather
|
||||||
|
than falling through to anything policy-adjacent.
|
||||||
|
|
||||||
|
Extended `docs/operator-runbook.md` with a "One-command MCP smoke" section
|
||||||
|
next to the existing REST one, and cross-linked it from
|
||||||
|
`docs/mcp-integration.md`'s manual smoke walkthrough so the two don't drift
|
||||||
|
out of sync.
|
||||||
|
|
||||||
|
Verified: `python3 scripts/smoke_mcp.py --python ../state-hub/.venv/bin/python`
|
||||||
|
exits `0` against fixtures, no real Qonto credentials. `pytest` → `31 passed`
|
||||||
|
(unchanged); `python3 -m compileall src tests scripts`.
|
||||||
|
|
||||||
## Task: Closure review
|
## Task: Closure review
|
||||||
|
|
||||||
```task
|
```task
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue