net-kingdom/workplans/NK-WP-0033-keycape-secret-exposure-rotation.md

id	type	title	domain	repo	status	owner	topic_slug	planning_priority	created	updated	origin	origin_ref	related	state_hub_workstream_id
NK-WP-0033	workplan	Contain and rotate the exposed KeyCape credential bundle	infotech	net-kingdom	active	codex	net-kingdom	P0	2026-08-23	2026-08-23	routed	State Hub message 8cc44a39-683c-4fab-80dd-b2275d0728e0	NK-WP-0032	63665674-6880-593e-96e6-bab3211b1352

NK-WP-0033 — KeyCape credential exposure containment and rotation

KeyCape reported that an agent tool rendered the base64 data map of the live sso/keycape-config Secret. Base64 is encoding, not protection. Treat the carried RS256 signing private key, LLDAP bind credential, Authelia client credential, and privacyIDEA admin token as exposed even though no value was decoded, reused, or intentionally repeated.

No task in this workplan authorizes a credential value in Git, State Hub, agent output, command arguments, or non-secret evidence. Live mutation requires the exact provider/custody owners, a reviewed sequence, abort and rollback conditions, and the authority stated by the owning execution path.

T01 — Contain, classify, and route the exposure

id: NK-WP-0033-T01
status: done
priority: high
state_hub_task_id: "28a655da-7cd6-58e2-8ca8-64d192de63a6"

Mark every credential class in the rendered bundle exposed, stop further rendering, consult credential routing, notify custody and issuer owners, and publish a sanitized risk notice.

Completed 2026-08-23. State Hub decision c43eef14-a209-41b1-ba67-781ca14c5ea2 requires governed rotation. Coordination was routed to railiance-platform in acf98be3-ff6b-4270-bd21-0193bebd806b and KeyCape in c774f748-6240-4733-89c7-518079a30f61; risk-nexus received sanitized notice c5d67801-1bc3-4e66-98c9-e08a58313970. No value was inspected or copied.

T02 — Agree the value-safe rotation and issuer-continuity contract

id: NK-WP-0033-T02
status: done
priority: high
state_hub_task_id: "c5ca817c-d601-5172-a106-1546c0743635"

KeyCape and railiance-platform must name the authoritative provider, custody, and consumer update for all four classes; select one revision-pinned bundle or field-safe sequence that cannot replay a stale exposed value; specify positive and predecessor-negative verification; and state abort, rollback, and evidence rules.

KeyCape currently loads one private key and signs under static kid=key-1 even though its JWKS type can serve multiple keys. The contract must either add and verify an active-key plus retained-public-key overlap or deliberately accept immediate token invalidation and verifier refresh. Evidence was routed in 2b24ccc2-c031-453e-ac70-52f0aaa127d5 and 9277d962-b511-4da7-9fc6-2c79bac448c1.

Railiance-platform published its forward-only owner contract at revision 453fed3. NetKingdom review is recorded in history/2026-08-23-keycape-exposure-dependency-map.md. The review found that the LLDAP credential also feeds identity-provisioner and privacyIDEA's persisted LDAP resolver. It also found that privacyIDEA /auth issues an expiring administrative session JWT with no individual revocation operation; /token/revoke applies to MFA authenticators, not that JWT. The owner recovery selected the wider privacyIDEA signing-secret invalidation path, so predecessor JWT denial is covered by the live cutover outcome.

Completed 2026-08-23. KeyCape reported that its owner-controlled recovery replaced all four exposed classes, including the privacyIDEA signing secret, and that positive/negative checks passed. The live outcome resolves the predecessor disposition; sanitized receipts remain in the owning workplan.

Warden routing has no concrete entries for these four KeyCape lanes. Catalog repair was requested in e5794329-cda5-4a6f-8234-a6eaa1fc318c; generic or unrelated routes do not authorize execution.

T03 — Implement a non-printing, non-stale rotation path

id: NK-WP-0033-T03
status: progress
priority: high
state_hub_task_id: "f0f6d6c3-9c45-56b7-9fd6-52fb0ea9054a"

After T02 review, implement or select an owner-controlled path that receives replacement material through hidden input or mode-0600 files, updates every provider and consumer in the agreed order, never reconstructs the bundle from stale local inputs, and retains no plaintext artifact after verification.

Provider-specific helpers must cover field-safe updates for lldap-secrets and authelia-secrets, the persistent LLDAP admin password, the identity-provisioner restart, privacyIDEA resolver lldap-coulomb, the Authelia client hash, and the privacyIDEA JWT expiry disposition. Static or dry-run tests must prove replacements do not enter argv or stdout.

The source-of-truth reconciliation completed in revision c956ceb: Authelia 4.38 OIDC client entries cannot consume *_FILE environment overrides because clients are a list, so the verifier now uses Authelia's supported template filter over the Secret-mounted file. The revision was applied live and Authelia returned Ready with startup complete.

NetKingdom now provides the supported one-command attended operation in sso-mfa/k8s/privacyidea/reconcile-lldap-resolver-live.sh. It is explicitly gated by --apply, requires an interactive terminal, uses protected 0600 files, updates only lldap-coulomb, proves replacement resolver lookup and privacyIDEA MFA, proves predecessor LLDAP denial with explicit 401/403 status, rechecks readiness/health, and emits one sanitized PASS/FAIL receipt after cleanup. It has not been run; the exact attended runbook is pinned in docs/keycape-exposure-resolver-reconciliation.md.

Do not use sso-mfa/bootstrap/creds-rotate.sh through an agent as currently written: it prints generated replacement values and its signing-key path causes immediate invalidation. Do not use a helper that reads the live Secret unless its output and temporary-state behavior have been independently reviewed for this incident.

T04 — Execute the governed replacement cutover

id: NK-WP-0033-T04
status: done
priority: high
state_hub_task_id: "751cca48-2bc9-5d44-80a0-60478717e99e"

Under the separately approved authority/window, replace the signing material, LLDAP bind credential, Authelia client credential, and privacyIDEA admin token; roll or reload every affected provider/consumer; preserve or explicitly end JWT continuity according to T02; and abort on partial-bundle or stale-source evidence. The task was gated on the owner-controlled T02 contract and is now complete.

Completed by the KeyCape owner-controlled recovery path on 2026-08-23. The owner reported all four affected deployments Ready and positive/negative checks passing. NetKingdom additionally restarted identity-provisioner after the LLDAP bind rotation and applied the Authelia template-filter fix. The privacyIDEA resolver still awaits attended provider-admin reconciliation.

T05 — Prove replacement, predecessor rejection, and cleanup

id: NK-WP-0033-T05
status: progress
priority: high
state_hub_task_id: "41e55d5c-ae28-5ee2-be25-b9a3758428f7"

Record only sanitized identifiers, public-key fingerprints, revisions, rollout status, positive authentication/MFA outcomes, predecessor rejection or expiry, cleanup receipts, and residual limitations. Close the incident only after all four credential classes have an authoritative disposition and no temporary plaintext or stale bundle remains.

Attended session 2026-08-27 — findings

The resolver reconciliation was attempted and the operator's platform-root login is restored. bao login -method=oidc succeeds, and GET /user/?realm=coulomb&username=platform-root returns the record. No green receipt has been produced yet; the run below is the state of record.

LLDAP bind credential — reconciled. The --apply run wrote the current LLDAP_LDAP_USER_PASS into resolver lldap-coulomb, and the replacement authenticated against LLDAP in the same run. The resolver had been holding the pre-cutover value since 2026-08-23, which is what T04 left outstanding.

Predecessor disposition — NOT PROVEN by receipt; observed twice by hand. The exposed value is dead: the operator was locked out of the LLDAP WebUI on 2026-08-27 after logging out of a persisted session, and the --check run the same day passed predecessor-denial with the real value before it was lost. Neither observation is carried by an emitted receipt. The value is unrecoverable — see the custody gap below.

Custody gap — the actual root cause of this session. T04 replaced the credential without a step that updates operator custody, and there was no custody store to update it into. platform-root-custody.md names a password safe entry net-kingdom/LLDAP/admin; no KeePassXC database exists and never did. The value lived only in a Firefox password-manager entry dated 2026-06-28, which was overwritten with the replacement on 2026-08-27, destroying the predecessor. Any rotation runbook that does not name where the outgoing value is retained will lose it the same way.

The reconciliation script had never completed a run. Four defects, fixed in 4a38511: Content-Type on bodyless GETs (HTTP 400 on every GET), /user/ parsed as a dict when it returns a list (traceback past the except clause), resolver writes dropping TIMEOUT/CACHE_TIMEOUT/SIZELIMIT (silently un-repairing a hand-fixed resolver), and no way to declare an unavailable predecessor (forcing a placeholder that records a passing denial proof).

Verification blind spot — open. verify-t06.sh reported success at bootstrap against a resolver created without the three tuning parameters, by a reconciliation path that could not execute. A verification that passes while its subject cannot run is not verification. This is the finding worth acting on beyond the four fixes, and it is not yet addressed.

Diagnosis correction. The "stale bind credential" reading in T04 was real but was not what blocked the resolver lookup; the HTTP 400 was our own request builder throughout. The two faults were independent and looked like one, which is why each partial fix appeared to change nothing.

Remaining before T05 can close: a green receipt from the repaired script, and a ruling on whether a predecessor disposition observed but not receipted is an acceptable close for this incident.