key-cape/docs/operations.md

Operating KeyCape

The supported deployment topology and the limits that come with it. These are deliberate boundaries of the current implementation, not defects awaiting a fix: where a limit is a consequence of a design choice, the choice is named.

Topology: exactly one replica

Run one instance per issuer. Authorization codes, login sessions and registration/enrollment handoffs are held in process memory. There is no shared store, and no sticky-session configuration makes this safe: a browser that starts a login on one replica and returns from Authelia to another finds no session and must start again. Two replicas do not halve the failure rate, they roughly double the login failure rate.

This is a documented exclusion rather than a missing feature. Adding a shared session store is a real option if the profile ever requires horizontal scale; it is not required today.

Consequences to plan for:

• A restart drops in-flight logins. Anyone mid-login gets an error and must start again. Issued tokens are unaffected — they are self-contained JWTs and stay valid until they expire.
• Rolling deployments are single-instance rollovers, so expect a brief window where new logins fail. Draining (below) protects requests already in flight, not logins waiting on a human at Authelia's password prompt.

Liveness and readiness

Endpoint	Answers	Use it for
/healthz	Is the process up? Probes nothing.	Liveness.
/readyz	Are LLDAP, Authelia and privacyIDEA reachable?	Readiness / traffic gating.

Keep these distinct. Wiring liveness to /readyz means an orchestrator restarts KeyCape whenever a dependency blinks, turning someone else's blip into an outage of your own — and a restart also discards every in-flight login, making it worse than the condition it reacted to.

/readyz returns 200 with status: ready, or 503 with status: not_ready and the failing check named. It reports which check failed, never why: the endpoint is unauthenticated and upstream error text carries hostnames and occasionally credentials-in-URLs. The reason is in the server log.

Probes are reachability and credential checks, not functional tests. LLDAP is probed with a bind, so a rotated or revoked service password is caught — that leaves the port open and every lookup failing, which is exactly what readiness should catch. Authelia and privacyIDEA are probed with a plain HTTP GET: any response means something is listening and speaking HTTP.

Results are cached for 2 seconds and each probe is bounded at 3 seconds. The cache is not an optimisation: the endpoint is necessarily unauthenticated, and without it anyone able to reach it could drive one upstream request per dependency per call.

Shutdown

On SIGTERM or SIGINT the server stops accepting connections and gives in-flight requests up to 15 seconds to finish. The grace period is deliberately under the 30-second read/write timeouts, so a stuck request cannot outlive the window an orchestrator typically allows before SIGKILL.

In-memory login and authorization state is not preserved across shutdown, by design — see the topology section.

Key and registration lifecycle

The signing key and all client registrations are read once at startup. Both are changed by editing configuration and restarting; there is no reload signal and no rotation service.

The key ID is the constant key-1. Rotating the signing key while keeping that identifier is a trap: a consumer caching JWKS by kid may keep the old key and reject freshly issued tokens until its cache expires. Plan rotation as "publish new keys, let consumers refetch, then issue with the new key", and treat a same-kid swap as a breaking change for anyone caching aggressively.

Removing a client registration does not revoke tokens already issued to it. There is no introspection or revocation endpoint, so an issued token stays valid until it expires — 15 minutes by default, or the client's tokenLifetime. To cut off a compromised client, remove the registration and wait out the lifetime, or rotate the signing key if you cannot.

/logout clears the local KeyCape session only. It does not end the Authelia session and does not revoke any issued token. A user who logs out and back in may not be prompted for credentials, because the upstream session is still valid.

Transport

The server speaks plain HTTP. TLS termination belongs to the deployment. KeyCape deliberately does not check or gate on the transport used to reach its upstream providers either; upstream ID tokens are verified cryptographically instead, so that assurance does not depend on the network being what we believe it is (KEY-WP-0019).

Upstream issuer pinning (read before rolling out KEY-WP-0019)

KeyCape verifies upstream ID tokens against the issuer the provider advertises, discovered server-side from authelia.tokenBaseURL. Some providers — Authelia among them — derive the advertised issuer from the request Host, so the value KeyCape learns over an in-cluster service address is not the value minted into tokens issued for the browser-facing host.

Where that is true, pin it explicitly:

authelia:
  tokenBaseURL: "http://authelia.sso.svc.cluster.local:9091"
  issuer: "https://auth.coulomb.social"   # exactly the iss claim in ID tokens
  jwksUrl: "http://authelia.sso.svc.cluster.local:9091/jwks.json"

Verification fails closed, so a mismatch means every human login fails — and it looks like a broken login rather than a configuration error. Check the issuer before rolling out, not after. The failure is diagnosable: the authentication failure event carries error_type=id_token_issuer_mismatch, as distinct from id_token_signature or provider_keys_unavailable.

Confirm the value with the Host the provider will actually see:

curl -s -H "Host: auth.coulomb.social" \
  http://authelia.sso.svc.cluster.local:9091/.well-known/openid-configuration \
  | jq -r .issuer

Before any live change

The estate-wide rules are in the-custodian/docs/agent-environment-orientation.md (revision 2026-09-21); read it first. The parts that apply to KeyCape's objects in sso and mfa:

• The founder's go-ahead comes first, recorded as ADMINISTER @ realm:kubernetes/railiance01, activation=APPROVED. Run changes as ssh railiance01 '…'. A harness block is a stop signal, not something to route around.

• Check CPU headroom before a restart or rollout (kubectl describe node | grep -A4 "Allocated resources"). A rolling update starts the new pod before stopping the old one.

• Diff first: client-side kubectl diff, then --dry-run=server.

• Never kubectl apply a Secret, and never read one's metadata. apply copies the whole Secret into the last-applied-configuration annotation, so even a metadata read prints the signing key and client secrets. Write Secrets with kubectl replace. To test for the annotation without printing it: kubectl get secret <n> -n sso -o go-template='{{ if index .metadata.annotations "kubectl.kubernetes.io/last-applied-configuration" }}HAS-ANNOTATION{{ else }}clean{{ end }}'.

• Never run any other go-template or jsonpath against a Secret. When a template fails, for example len on an absent field, kubectl prints the whole raw object as debugging output, .data included. The presence check above is the only tested template. This leaked sso/keycape-config into an agent session on 2026-09-23 (see below).

Open finding, 2026-09-23: sso/keycape-config, sso/authelia-secrets, sso/lldap-secrets and mfa/privacyidea-config all carry that annotation. The earlier rotation script wrote them with apply; it now uses replace. Anyone who can get these Secrets could read the data anyway, so this is not a new reader. What it breaks is the assumption that a metadata read is safe. The fix is a live change and waits for the founder's go-ahead: kubectl annotate secret <n> -n <ns> kubectl.kubernetes.io/last-applied-configuration- for each of the four. It prints no value and leaves .data untouched, so pods need no restart. Re-check with the template above afterwards.

Superseded the same day by an exposure. While checking these Secrets' owners before the cleanup, an agent ran an untested template. It failed and printed keycape-config in full into the session transcript: the signing key, the LLDAP bind password and the Authelia client secret. The other three Secrets were probably printed as well. All four must be treated as exposed and rotated; the annotation cleanup folds into that rotation.

Unreleased fail-closed startup changes (read before the next rollout)

The deployed image at the time of writing is sha256:7ff54c54e63ee172ae9e6e7fd2da96e427352f712343d74626ee6fe0f6f82611, built from dcebd46. The changes below on main postdate it, all affect startup or issuance, and all fail closed, so they land together on the next rollout and a mistake in either presents as a refusal rather than as a warning.

1. Browser clients reject service-identity fields (74b35b6, KEY-WP-0028). Config validation now rejects serviceSubject or roles on a client whose grant is not client_credentials; those fields were silently ignored there, so a registration could look effective and do nothing. This is a startup error: a stray field in the deployed config stops the process from booting rather than degrading. The live sso/keycape-config was checked on 2026-09-09 and none of its three browser clients carries either field, so it passes as deployed. The custody owner confirmed independently that the config was last written during the 2026-09-09 activation (resource version 58747126) and has not been edited since, so the inspected state is the state that will boot — and that they will re-check the browser clients if anything writes that Secret before the window. tenant is deliberately not rejected; see below.

2. A conflicting tenant binding refuses issuance (329e48f, KEY-WP-0013-T05). A client registration may declare a tenant, which is supplied when the directory has placed the user nowhere and enforced when it has. A declared zone that conflicts with a directory assignment refuses the token with 403 and error_type: tenant_binding. No deployed client declares a tenant on a browser grant today, so nothing changes on rollout; the refusal only becomes reachable once the approver client is registered. See the tenant contract.

3. KeyCape enforces login freshness itself (11ce29a, KEY-WP-0033). prompt=login is no longer forwarded to Authelia, because 4.38 refuses it for every real login. prompt=login and max_age=0 go upstream as max_age=10, and the callback refuses any upstream auth_time that misses the requirement, or is absent (stale_upstream_authentication). This changes nothing for clients that do not ask for freshness. The proof is a completed fresh-login journey, not a redirect check.

None of these has been exercised against a running issuer. The honest proof for both is a live boot with the new binary, which belongs to the attended rollout window and not to a session running against production on its own.

What is not claimed

No resource-efficiency or throughput bounds are asserted here. Nothing in this repository benchmarks KeyCape, so any figure would be invention. Measure it in your own deployment before sizing against it.

Account recovery and browser sign-out

Set KEYCAPE_ACCOUNT_PORTAL_URL=https://users.coulomb.social and KEYCAPE_BROWSER_LOGOUT_URL=https://auth.coulomb.social/logout together to enable central browser recovery. Both must be HTTPS, with no credentials, query or fragment. They override the corresponding YAML accountPortalURL/browserLogoutURL. Failed browser authorization and expired callbacks redirect to the fixed /access-recovery page with no state, code, claimed user or browser return URL. API authorization/token validation remains fail-closed.

GET /account/logout displays confirmation and creates a short-lived Secure, HttpOnly host-only CSRF cookie. POST checks that cookie, the form nonce and exact issuer Origin; only then does it invalidate the current KeyCape session and redirect to Authelia's browser /logout, with a fixed portal /logged-out return. Authelia owns and deletes its session cookie. Its v4.38 SignOut view validates the return destination and performs the same-origin logout API call. Reference: https://github.com/authelia/authelia/blob/v4.38.19/web/src/views/LoginPortal/SignOut/SignOut.tsx Existing RP sessions and issued JWTs are not revoked. The original /logout endpoint retains its local-only contract. Actual shared-session destruction requires browser execution; a redirect-only smoke is not acceptance evidence.