Survive credential rotation: re-read the lease for every connection

The deployment ran for one lease window and then sat unready for eight hours.
Platform credentials are 30-minute leases, not passwords: the service read the
mounted URL once at start-up, so External Secrets kept the file current while
the engine held the URL it booted with, and every reconnection after the first
expiry used a credential the database had already revoked.

make_engine now takes an optional refresh callable, invoked by a do_connect
hook each time the pool opens a connection, and pool_recycle is 900s so a
pooled connection is retired well inside the lease. Only username and password
are taken from the refreshed URL — host, port and database come from the engine,
so a malformed refresh cannot silently redirect the service somewhere else.

Two things behaved correctly and are worth keeping. /readyz reported the real
cause, "database unreachable: OperationalError", rather than a generic failure.
And liveness stayed independent of the database, so the pod was never
restart-looped: it was alive, unable to serve, and said so. Pointing liveness at
a database-dependent path would have masked this as a crash loop.

Service tests 47 -> 49, including one asserting pool_recycle stays inside the
shortest lease the platform issues.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bjefh8NUiEiahN4JLwoSKM

Assistant: claude-code
Assistant-Model: opus
Assistant-Process: 388925@bnt-lap001
Assistant-Session: 3507023f-e0fd-4a1e-9d90-a0d4217d1502

service/tests/test_health.py

@@ -138,3 +138,29 @@ def test_absent_database_file_still_fails_loudly(tmp_path: Path) -> None:
     settings = Settings(database_url_file=str(tmp_path / "absent"))
     with pytest.raises(RuntimeError, match="cannot read secret file"):
         _ = settings.resolved_database_url
+
+
+def test_engine_rereads_credentials_on_each_connection(tmp_path: Path) -> None:
+    """Platform credentials are 30-minute leases, not passwords. Reading the
+    file once at start-up worked for one lease window and then failed
+    permanently — External Secrets kept the file current while the engine kept
+    the URL it booted with."""
+    secret = tmp_path / "url"
+    secret.write_text(f"sqlite:///{tmp_path / 'a.db'}", encoding="utf-8")
+    settings = Settings(database_url_file=str(secret))
+
+    seen: list[str] = []
+    engine = make_engine(
+        settings.resolved_database_url,
+        refresh=lambda: (seen.append(settings.resolved_database_url) or settings.resolved_database_url),
+    )
+    with engine.connect():
+        pass
+    assert seen, "the refresh hook must run when the pool opens a connection"
+
+
+def test_pool_recycle_is_shorter_than_the_shortest_lease() -> None:
+    """30-minute runtime lease; a pooled connection must be retired first."""
+    from canned_prompts_service.db import POOL_RECYCLE_SECONDS
+
+    assert POOL_RECYCLE_SECONDS < 30 * 60