Give the runtime real readiness, graceful shutdown and stated limits
All checks were successful
Build and Publish Container Image / build-and-push (push) Successful in 41s

Closes gap G08. /healthz returned a constant without probing anything, the server
called ListenAndServe with no signal handling, and the operational limits of
in-memory state, startup-loaded keys and local-only logout lived in code comments
rather than anywhere an operator would look.

/readyz probes LLDAP, Authelia and privacyIDEA; /healthz stays liveness and
probes nothing. Keeping them distinct matters: wiring liveness to dependency
health means an orchestrator restarts KeyCape when a dependency blinks, and a
restart also discards every in-flight login, so the reaction is worse than the
condition it reacts to.

LLDAP is probed with a bind rather than a dial, since a rotated or revoked
service password leaves the port open and every lookup failing -- exactly what
readiness should catch and exactly what a dial would miss. The response names the
failing check but never the reason: the endpoint is unauthenticated and upstream
error text carries hostnames and sometimes credentials-in-URLs. Results are
cached for 2s so an unauthenticated endpoint cannot be used to drive unbounded
upstream traffic, and probes run concurrently under a 3s bound so a hung
dependency makes the endpoint answer rather than hang with it.

SIGTERM and SIGINT now drain in-flight requests for 15s, under the 30s read/write
timeouts so a stuck request cannot outlive the window before SIGKILL.

docs/operations.md states the single-replica topology and why, and three limits
easy to get wrong: the constant key-1 key ID makes same-kid rotation a trap for
consumers caching JWKS, removing a client does not revoke its issued tokens, and
/logout is local only. No throughput figures are given, since nothing here
benchmarks KeyCape. Shared storage and refresh tokens stay excluded, as G08
allows.

Verified in the running executable: 503 naming all three checks failed while
/healthz returned 200, the LLDAP check flipping to ok once started, and 40/40
requests succeeding across a SIGTERM.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NV9oijZukGyGbRQGGKnK4P

Assistant: claude-code
Assistant-Model: opus
Assistant-Process: 713576@bnt-lap001
Assistant-Session: 384c511d-9bce-4cb8-a676-2aef6c0c8df6
This commit is contained in:
tegwick 2026-09-08 09:43:49 +02:00
parent a9296fdf84
commit d568b79223
8 changed files with 622 additions and 4 deletions

View file

@ -0,0 +1,113 @@
---
id: KEY-WP-0025
type: workplan
title: "Give the runtime real readiness, graceful shutdown and stated operational limits"
domain: infotech
repo: key-cape
status: finished
owner: claude
topic_slug: runtime-lifecycle-and-readiness
created: "2026-09-08"
updated: "2026-09-08"
---
Closes gap G08. `/healthz` returned a constant without probing anything, the
server called `ListenAndServe` with no signal handling, and the operational
limits of in-memory state, startup-loaded keys and local-only logout were spread
across code comments rather than stated anywhere an operator would look.
G08 explicitly does not require shared storage or refresh tokens. The work is
therefore to make readiness real, make restarts survivable for in-flight
requests, and write the limits down honestly.
## Separate readiness from liveness
```task
id: KEY-WP-0025-T01
status: done
priority: medium
```
Added `internal/server/readiness` and a `/readyz` endpoint probing LLDAP,
Authelia and privacyIDEA. `/healthz` stays liveness and deliberately probes
nothing: wiring liveness to dependency health means an orchestrator restarts
KeyCape when a dependency blinks, and since a restart also discards every
in-flight login, the reaction is worse than the condition.
Design decisions worth keeping:
- **LLDAP is probed with a bind, not a dial.** A rotated or revoked service
password leaves the port open and every lookup failing — precisely the state
readiness exists to catch, and precisely what a TCP dial would miss.
- **The body names the failing check but never the reason.** The endpoint is
unauthenticated, and upstream error text carries hostnames and sometimes
credentials-in-URLs. Detail goes to the log. A test asserts a probe error
containing a credentialed URL does not reach the response.
- **Results are cached for 2s.** Not an optimisation: without it, anyone able to
reach an unauthenticated endpoint could drive one upstream request per
dependency per call.
- **Probes run concurrently, each bounded at 3s**, so readiness costs the slowest
probe rather than their sum, and a hung dependency makes the endpoint answer
rather than hang alongside it.
## Shut down without dropping requests
```task
id: KEY-WP-0025-T02
status: done
priority: medium
```
`SIGTERM`/`SIGINT` now drain in-flight requests for up to 15 seconds. The grace
period sits under the 30s read/write timeouts so a stuck request cannot outlive
the window an orchestrator allows before `SIGKILL`.
In-memory login state is deliberately not preserved: a browser mid-login must
start again after a restart. That is a property of the single-replica topology,
documented rather than papered over.
## State the operational limits
```task
id: KEY-WP-0025-T03
status: done
priority: medium
```
`docs/operations.md` states the supported topology — exactly one replica, because
authorization codes, login sessions and handoffs are process-local and no sticky
configuration makes a second replica safe — and the consequences: restarts drop
in-flight logins while issued tokens keep working, and rolling deploys have a
window where new logins fail.
It also records three limits that are easy to get wrong in operation:
- the key ID is the constant `key-1`, so rotating the key without changing it can
leave consumers caching JWKS by `kid` rejecting freshly issued tokens;
- removing a client registration does not revoke its issued tokens, since there
is no revocation or introspection endpoint — they stay valid until expiry;
- `/logout` clears the local session only, not the Authelia session or any token.
No throughput or resource figures are given: nothing here benchmarks KeyCape, so
any number would be invented.
## Verify in the running server
```task
id: KEY-WP-0025-T04
status: done
priority: medium
```
Checked against the built executable rather than the handlers, following the
lesson from KEY-WP-0023:
- with every dependency down, `/readyz` returns 503 naming all three as failed
while `/healthz` still returns 200 — the split doing its job;
- starting LLDAP flips its check to `ok` while the others stay failed, so the
probes are measuring what they claim;
- 40 requests issued across a `SIGTERM` all returned 200, and the log shows the
signal handled and shutdown completed.
Seven unit tests cover the reporter, including the disclosure, caching, timeout
and concurrency properties above.