Give the runtime real readiness, graceful shutdown and stated limits
All checks were successful
Build and Publish Container Image / build-and-push (push) Successful in 41s
All checks were successful
Build and Publish Container Image / build-and-push (push) Successful in 41s
Closes gap G08. /healthz returned a constant without probing anything, the server called ListenAndServe with no signal handling, and the operational limits of in-memory state, startup-loaded keys and local-only logout lived in code comments rather than anywhere an operator would look. /readyz probes LLDAP, Authelia and privacyIDEA; /healthz stays liveness and probes nothing. Keeping them distinct matters: wiring liveness to dependency health means an orchestrator restarts KeyCape when a dependency blinks, and a restart also discards every in-flight login, so the reaction is worse than the condition it reacts to. LLDAP is probed with a bind rather than a dial, since a rotated or revoked service password leaves the port open and every lookup failing -- exactly what readiness should catch and exactly what a dial would miss. The response names the failing check but never the reason: the endpoint is unauthenticated and upstream error text carries hostnames and sometimes credentials-in-URLs. Results are cached for 2s so an unauthenticated endpoint cannot be used to drive unbounded upstream traffic, and probes run concurrently under a 3s bound so a hung dependency makes the endpoint answer rather than hang with it. SIGTERM and SIGINT now drain in-flight requests for 15s, under the 30s read/write timeouts so a stuck request cannot outlive the window before SIGKILL. docs/operations.md states the single-replica topology and why, and three limits easy to get wrong: the constant key-1 key ID makes same-kid rotation a trap for consumers caching JWKS, removing a client does not revoke its issued tokens, and /logout is local only. No throughput figures are given, since nothing here benchmarks KeyCape. Shared storage and refresh tokens stay excluded, as G08 allows. Verified in the running executable: 503 naming all three checks failed while /healthz returned 200, the LLDAP check flipping to ok once started, and 40/40 requests succeeding across a SIGTERM. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NV9oijZukGyGbRQGGKnK4P Assistant: claude-code Assistant-Model: opus Assistant-Process: 713576@bnt-lap001 Assistant-Session: 384c511d-9bce-4cb8-a676-2aef6c0c8df6
This commit is contained in:
parent
a9296fdf84
commit
d568b79223
8 changed files with 622 additions and 4 deletions
113
workplans/KEY-WP-0025-runtime-lifecycle-and-readiness.md
Normal file
113
workplans/KEY-WP-0025-runtime-lifecycle-and-readiness.md
Normal file
|
|
@ -0,0 +1,113 @@
|
|||
---
|
||||
id: KEY-WP-0025
|
||||
type: workplan
|
||||
title: "Give the runtime real readiness, graceful shutdown and stated operational limits"
|
||||
domain: infotech
|
||||
repo: key-cape
|
||||
status: finished
|
||||
owner: claude
|
||||
topic_slug: runtime-lifecycle-and-readiness
|
||||
created: "2026-09-08"
|
||||
updated: "2026-09-08"
|
||||
---
|
||||
|
||||
Closes gap G08. `/healthz` returned a constant without probing anything, the
|
||||
server called `ListenAndServe` with no signal handling, and the operational
|
||||
limits of in-memory state, startup-loaded keys and local-only logout were spread
|
||||
across code comments rather than stated anywhere an operator would look.
|
||||
|
||||
G08 explicitly does not require shared storage or refresh tokens. The work is
|
||||
therefore to make readiness real, make restarts survivable for in-flight
|
||||
requests, and write the limits down honestly.
|
||||
|
||||
## Separate readiness from liveness
|
||||
|
||||
```task
|
||||
id: KEY-WP-0025-T01
|
||||
status: done
|
||||
priority: medium
|
||||
```
|
||||
|
||||
Added `internal/server/readiness` and a `/readyz` endpoint probing LLDAP,
|
||||
Authelia and privacyIDEA. `/healthz` stays liveness and deliberately probes
|
||||
nothing: wiring liveness to dependency health means an orchestrator restarts
|
||||
KeyCape when a dependency blinks, and since a restart also discards every
|
||||
in-flight login, the reaction is worse than the condition.
|
||||
|
||||
Design decisions worth keeping:
|
||||
|
||||
- **LLDAP is probed with a bind, not a dial.** A rotated or revoked service
|
||||
password leaves the port open and every lookup failing — precisely the state
|
||||
readiness exists to catch, and precisely what a TCP dial would miss.
|
||||
- **The body names the failing check but never the reason.** The endpoint is
|
||||
unauthenticated, and upstream error text carries hostnames and sometimes
|
||||
credentials-in-URLs. Detail goes to the log. A test asserts a probe error
|
||||
containing a credentialed URL does not reach the response.
|
||||
- **Results are cached for 2s.** Not an optimisation: without it, anyone able to
|
||||
reach an unauthenticated endpoint could drive one upstream request per
|
||||
dependency per call.
|
||||
- **Probes run concurrently, each bounded at 3s**, so readiness costs the slowest
|
||||
probe rather than their sum, and a hung dependency makes the endpoint answer
|
||||
rather than hang alongside it.
|
||||
|
||||
## Shut down without dropping requests
|
||||
|
||||
```task
|
||||
id: KEY-WP-0025-T02
|
||||
status: done
|
||||
priority: medium
|
||||
```
|
||||
|
||||
`SIGTERM`/`SIGINT` now drain in-flight requests for up to 15 seconds. The grace
|
||||
period sits under the 30s read/write timeouts so a stuck request cannot outlive
|
||||
the window an orchestrator allows before `SIGKILL`.
|
||||
|
||||
In-memory login state is deliberately not preserved: a browser mid-login must
|
||||
start again after a restart. That is a property of the single-replica topology,
|
||||
documented rather than papered over.
|
||||
|
||||
## State the operational limits
|
||||
|
||||
```task
|
||||
id: KEY-WP-0025-T03
|
||||
status: done
|
||||
priority: medium
|
||||
```
|
||||
|
||||
`docs/operations.md` states the supported topology — exactly one replica, because
|
||||
authorization codes, login sessions and handoffs are process-local and no sticky
|
||||
configuration makes a second replica safe — and the consequences: restarts drop
|
||||
in-flight logins while issued tokens keep working, and rolling deploys have a
|
||||
window where new logins fail.
|
||||
|
||||
It also records three limits that are easy to get wrong in operation:
|
||||
|
||||
- the key ID is the constant `key-1`, so rotating the key without changing it can
|
||||
leave consumers caching JWKS by `kid` rejecting freshly issued tokens;
|
||||
- removing a client registration does not revoke its issued tokens, since there
|
||||
is no revocation or introspection endpoint — they stay valid until expiry;
|
||||
- `/logout` clears the local session only, not the Authelia session or any token.
|
||||
|
||||
No throughput or resource figures are given: nothing here benchmarks KeyCape, so
|
||||
any number would be invented.
|
||||
|
||||
## Verify in the running server
|
||||
|
||||
```task
|
||||
id: KEY-WP-0025-T04
|
||||
status: done
|
||||
priority: medium
|
||||
```
|
||||
|
||||
Checked against the built executable rather than the handlers, following the
|
||||
lesson from KEY-WP-0023:
|
||||
|
||||
- with every dependency down, `/readyz` returns 503 naming all three as failed
|
||||
while `/healthz` still returns 200 — the split doing its job;
|
||||
- starting LLDAP flips its check to `ok` while the others stay failed, so the
|
||||
probes are measuring what they claim;
|
||||
- 40 requests issued across a `SIGTERM` all returned 200, and the log shows the
|
||||
signal handled and shutdown completed.
|
||||
|
||||
Seven unit tests cover the reporter, including the disclosure, caching, timeout
|
||||
and concurrency properties above.
|
||||
Loading…
Add table
Add a link
Reference in a new issue