key-cape/docs/operations.md
tegwick 7dda967c27
All checks were successful
Build and Publish Container Image / build-and-push (push) Successful in 45s
Establish the live state and find a rollout precondition for G10
G10 waits on custody and platform owners and cannot close from here. What was
doable: verify the handoffs actually went out, replace a remembered live state
with an observed one, and find out whether main is safe to deploy. The last
question found a defect in this repository's own recent work.

Handoffs verified independently rather than trusted: all seven messages are in
the hub with receipt ids. This gap was reopened once for claimed-but-unsent
delivery, so the claim deserved the same scrutiny.

Live state read from the cluster read-only: image main-153258b, only the Qonto
secret materialized so the approval clients remain unprovisioned, four registered
clients, no tenantEngine block. That also corrects an earlier claim of mine --
the deployed config sets userOU explicitly, so the KEY-WP-0023 default fix was
never a production issue.

The precondition: KEY-WP-0019 discovers the expected issuer from
authelia.tokenBaseURL, and the deployed Authelia derives its advertised issuer
from the request Host, advertising the in-cluster address to KeyCape and the
browser-facing one to browsers. Verification fails closed, so a mismatch breaks
every human login and looks like a broken login rather than a misconfiguration.
Which value the token carries needs a real login against production to settle and
was not determined here.

Two mitigations: docs/operations.md documents pinning authelia.issuer and
jwksUrl, with the curl that reveals what the provider advertises for a given
Host; and the authentication failure event now carries a specific reason, so
id_token_issuer_mismatch is distinguishable from a signature failure or an
unreachable key set. The browser still learns nothing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NV9oijZukGyGbRQGGKnK4P

Assistant: claude-code
Assistant-Model: opus
Assistant-Process: 713576@bnt-lap001
Assistant-Session: 384c511d-9bce-4cb8-a676-2aef6c0c8df6
2026-09-08 11:41:42 +02:00

132 lines
6.1 KiB
Markdown

# Operating KeyCape
The supported deployment topology and the limits that come with it. These are
deliberate boundaries of the current implementation, not defects awaiting a fix:
where a limit is a consequence of a design choice, the choice is named.
## Topology: exactly one replica
**Run one instance per issuer.** Authorization codes, login sessions and
registration/enrollment handoffs are held in process memory. There is no shared
store, and no sticky-session configuration makes this safe: a browser that starts
a login on one replica and returns from Authelia to another finds no session and
must start again. Two replicas do not halve the failure rate, they roughly double
the login failure rate.
This is a documented exclusion rather than a missing feature. Adding a shared
session store is a real option if the profile ever requires horizontal scale; it
is not required today.
Consequences to plan for:
- **A restart drops in-flight logins.** Anyone mid-login gets an error and must
start again. Issued tokens are unaffected — they are self-contained JWTs and
stay valid until they expire.
- **Rolling deployments are single-instance rollovers**, so expect a brief window
where new logins fail. Draining (below) protects requests already in flight, not
logins waiting on a human at Authelia's password prompt.
## Liveness and readiness
| Endpoint | Answers | Use it for |
| --- | --- | --- |
| `/healthz` | Is the process up? Probes nothing. | Liveness. |
| `/readyz` | Are LLDAP, Authelia and privacyIDEA reachable? | Readiness / traffic gating. |
Keep these distinct. Wiring liveness to `/readyz` means an orchestrator restarts
KeyCape whenever a dependency blinks, turning someone else's blip into an outage
of your own — and a restart also discards every in-flight login, making it worse
than the condition it reacted to.
`/readyz` returns 200 with `status: ready`, or 503 with `status: not_ready` and
the failing check named. It reports *which* check failed, never *why*: the
endpoint is unauthenticated and upstream error text carries hostnames and
occasionally credentials-in-URLs. The reason is in the server log.
Probes are reachability and credential checks, not functional tests. LLDAP is
probed with a bind, so a rotated or revoked service password is caught — that
leaves the port open and every lookup failing, which is exactly what readiness
should catch. Authelia and privacyIDEA are probed with a plain HTTP GET: any
response means something is listening and speaking HTTP.
Results are cached for 2 seconds and each probe is bounded at 3 seconds. The
cache is not an optimisation: the endpoint is necessarily unauthenticated, and
without it anyone able to reach it could drive one upstream request per
dependency per call.
## Shutdown
On `SIGTERM` or `SIGINT` the server stops accepting connections and gives
in-flight requests up to 15 seconds to finish. The grace period is deliberately
under the 30-second read/write timeouts, so a stuck request cannot outlive the
window an orchestrator typically allows before `SIGKILL`.
In-memory login and authorization state is not preserved across shutdown, by
design — see the topology section.
## Key and registration lifecycle
The signing key and all client registrations are read once at startup. Both are
changed by editing configuration and restarting; there is no reload signal and no
rotation service.
**The key ID is the constant `key-1`.** Rotating the signing key while keeping
that identifier is a trap: a consumer caching JWKS by `kid` may keep the old key
and reject freshly issued tokens until its cache expires. Plan rotation as
"publish new keys, let consumers refetch, then issue with the new key", and treat
a same-`kid` swap as a breaking change for anyone caching aggressively.
**Removing a client registration does not revoke tokens already issued to it.**
There is no introspection or revocation endpoint, so an issued token stays valid
until it expires — 15 minutes by default, or the client's `tokenLifetime`. To cut
off a compromised client, remove the registration *and* wait out the lifetime, or
rotate the signing key if you cannot.
**`/logout` clears the local KeyCape session only.** It does not end the Authelia
session and does not revoke any issued token. A user who logs out and back in may
not be prompted for credentials, because the upstream session is still valid.
## Transport
The server speaks plain HTTP. TLS termination belongs to the deployment. KeyCape
deliberately does not check or gate on the transport used to reach its upstream
providers either; upstream ID tokens are verified cryptographically instead, so
that assurance does not depend on the network being what we believe it is
(KEY-WP-0019).
## Upstream issuer pinning (read before rolling out KEY-WP-0019)
KeyCape verifies upstream ID tokens against the issuer the provider advertises,
discovered server-side from `authelia.tokenBaseURL`. Some providers — Authelia
among them — derive the advertised issuer from the **request Host**, so the value
KeyCape learns over an in-cluster service address is not the value minted into
tokens issued for the browser-facing host.
Where that is true, pin it explicitly:
```yaml
authelia:
tokenBaseURL: "http://authelia.sso.svc.cluster.local:9091"
issuer: "https://auth.coulomb.social" # exactly the iss claim in ID tokens
jwksUrl: "http://authelia.sso.svc.cluster.local:9091/jwks.json"
```
Verification fails closed, so a mismatch means **every human login fails** — and
it looks like a broken login rather than a configuration error. Check the
issuer before rolling out, not after. The failure is diagnosable: the
authentication failure event carries `error_type=id_token_issuer_mismatch`, as
distinct from `id_token_signature` or `provider_keys_unavailable`.
Confirm the value with the Host the provider will actually see:
```bash
curl -s -H "Host: auth.coulomb.social" \
http://authelia.sso.svc.cluster.local:9091/.well-known/openid-configuration \
| jq -r .issuer
```
## What is not claimed
No resource-efficiency or throughput bounds are asserted here. Nothing in this
repository benchmarks KeyCape, so any figure would be invention. Measure it in
your own deployment before sizing against it.