The deployment ran for one lease window and then sat unready for eight hours.
Platform credentials are 30-minute leases, not passwords: the service read the
mounted URL once at start-up, so External Secrets kept the file current while
the engine held the URL it booted with, and every reconnection after the first
expiry used a credential the database had already revoked.
make_engine now takes an optional refresh callable, invoked by a do_connect
hook each time the pool opens a connection, and pool_recycle is 900s so a
pooled connection is retired well inside the lease. Only username and password
are taken from the refreshed URL — host, port and database come from the engine,
so a malformed refresh cannot silently redirect the service somewhere else.
Two things behaved correctly and are worth keeping. /readyz reported the real
cause, "database unreachable: OperationalError", rather than a generic failure.
And liveness stayed independent of the database, so the pod was never
restart-looped: it was alive, unable to serve, and said so. Pointing liveness at
a database-dependent path would have masked this as a crash loop.
Service tests 47 -> 49, including one asserting pool_recycle stays inside the
shortest lease the platform issues.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bjefh8NUiEiahN4JLwoSKM
Assistant: claude-code
Assistant-Model: opus
Assistant-Process: 388925@bnt-lap001
Assistant-Session: 3507023f-e0fd-4a1e-9d90-a0d4217d1502
tegwick
f6e20b5e0c