fix: preserve resolver configuration during attended password cutover
Assistant: codex Assistant-Model: gpt-6-astra Assistant-Session: 01a1346b-72a4-7d40-b8bd-8b8ca346cfa2
This commit is contained in:
parent
ab25dde80d
commit
4dffe2fe7a
11 changed files with 802 additions and 548 deletions
|
|
@ -30,9 +30,9 @@ runbook or a dedicated procedure. Missing runbooks are listed separately.
|
|||
| `privacyidea/check-user-mfa-state.sh` | [smooth-bootstrap-guide.md](../docs/smooth-bootstrap-guide.md) | source repaired: explicit admitted private JWT, scoped read-only counts; live diagnostic delivery not admitted by this repair |
|
||||
| `privacyidea/create-secrets.sh` | [README.md](../sso-mfa/k8s/privacyidea/README.md), [smooth-bootstrap-guide.md](../docs/smooth-bootstrap-guide.md) | unknown |
|
||||
| `privacyidea/enckey-bootstrap.sh` | [README.md](../sso-mfa/k8s/privacyidea/README.md) | unknown |
|
||||
| `privacyidea/reconcile-lldap-resolver-live.sh` | [keycape-exposure-resolver-reconciliation.md](../docs/keycape-exposure-resolver-reconciliation.md) | attempted by Bernd Worsch 2026-08-27; no completed PASS receipt |
|
||||
| `privacyidea/reconcile-lldap-resolver-live.sh` | [keycape-exposure-resolver-reconciliation.md](../docs/keycape-exposure-resolver-reconciliation.md) | historical check 2026-09-23 with predecessor unproven; current core exercised 2026-10-14 on disposable PI/LLDAP, full production procedure unexercised |
|
||||
| `privacyidea/repair-realm-live.sh` | [privacyIDEA repair contract](../sso-mfa/k8s/privacyidea/README.md#configuration-preserving-realm-repair), [verify-t06.md](../docs/verify-t06.md) | source repaired: attended in-memory input and propagated functional proof; live acceptance pending |
|
||||
| `privacyidea/update-lldap-resolver-live.sh` | None found | unknown |
|
||||
| `privacyidea/update-lldap-resolver-live.sh` | [keycape-exposure-resolver-reconciliation.md](keycape-exposure-resolver-reconciliation.md) | core exercised 2026-10-14 on disposable PI/LLDAP; production unexercised |
|
||||
| `user-engine/verify-operability.sh` | None found | unknown |
|
||||
| `verify-identity-cutover-dependencies.sh` | None found | unknown |
|
||||
| `verify-t02.sh` | [README.md](../sso-mfa/k8s/README.md) | unknown |
|
||||
|
|
@ -46,7 +46,6 @@ runbook or a dedicated procedure. Missing runbooks are listed separately.
|
|||
## Scripts without a matching runbook
|
||||
|
||||
- `sso-mfa/k8s/keycape/register-coulomb-social.sh` — header usage is available, but a runbook with prerequisites, verification and exercise history was not found in the reviewed runbook set.
|
||||
- `sso-mfa/k8s/privacyidea/update-lldap-resolver-live.sh` — header usage is available, but a runbook with prerequisites, verification and exercise history was not found in the reviewed runbook set.
|
||||
- `sso-mfa/k8s/user-engine/verify-operability.sh` — header usage is available, but a runbook with prerequisites, verification and exercise history was not found in the reviewed runbook set.
|
||||
- `sso-mfa/k8s/verify-identity-cutover-dependencies.sh` — header usage is available, but a runbook with prerequisites, verification and exercise history was not found in the reviewed runbook set.
|
||||
|
||||
|
|
|
|||
|
|
@ -17,9 +17,11 @@ The first execution, on 2026-08-27, found four defects:
|
|||
2. `GET /user/` parsed as a dict when privacyIDEA returns a list — an
|
||||
`AttributeError` outside the `except` clause, so the run died as a traceback
|
||||
rather than a receipt.
|
||||
3. Resolver writes omitting `TIMEOUT`, `CACHE_TIMEOUT`, `SIZELIMIT` — a resolver
|
||||
write replaces the whole object, so each run silently un-repaired a resolver
|
||||
an operator had just fixed by hand.
|
||||
3. Missing resolver tuning (`TIMEOUT`, `CACHE_TIMEOUT`, `SIZELIMIT`) was
|
||||
historically attributed to whole-object replacement. Correction, 2026-10-14:
|
||||
native privacyIDEA 3.12.2 preserves omitted fields; sending explicit defaults
|
||||
can overwrite custom tuning. The maintained cutover helper updates only the
|
||||
bind password and verifies preservation. See the current resolver runbook.
|
||||
4. No way to declare an unavailable predecessor — the prompt could not be left
|
||||
empty, so an operator had to type a placeholder, which also fails the bind
|
||||
and was recorded as a **passing** denial proof.
|
||||
|
|
|
|||
|
|
@ -1,138 +1,132 @@
|
|||
# KeyCape exposure: privacyIDEA resolver reconciliation
|
||||
|
||||
Incident: `KEYCAPE-EXPOSURE-20260823-01`
|
||||
Workplan: `NK-WP-0033`
|
||||
NetKingdom procedure: `sso-mfa/k8s/privacyidea/reconcile-lldap-resolver-live.sh` (repaired 2026-08-27 in `4a38511`; the previously pinned revision could not complete a run — see NK-WP-0033 T05)
|
||||
Platform recovery contract: railiance-platform `453fed3`
|
||||
Owner cutover receipt: State Hub message `45b236c8-052f-43d3-a472-44f8e9694da2`
|
||||
Exercise status: `--check --predecessor-unavailable` exercised to PASS on 2026-09-23 by Bernd Worsch (railiance01 checkout `6096c395`, receipt in NK-WP-0033). `--apply` last ran on 2026-08-27 before the fixes in `4a38511`. Run it from your own terminal with `ssh -t`: Claude Code's `!` prefix has no TTY, and the script refuses to run without one. Take the pi-admin and LLDAP values from OpenBao `operators` custody (`privacyidea/pi-admin`, `lldap/admin`). See `attended-procedure-standard.md`.
|
||||
Incident: `KEYCAPE-EXPOSURE-20260823-01`
|
||||
Historical workplan: `NK-WP-0033`; source maintenance: `NK-WP-0049-T02`
|
||||
Exercise status: core helper exercised 2026-10-14 by custodian-codex against
|
||||
local disposable privacyIDEA 3.12.2 and LLDAP, including actual LDAP bind,
|
||||
password update, preserved configuration, exact user lookup and HOTP. Terminal
|
||||
handling and failure paths have automated tests. The complete new attended
|
||||
procedure has not run against the production cluster or KeyCape.
|
||||
Evidence: Custodian `docs/evidence/2026-10-14-netkingdom-security/resolver-cutover.json`.
|
||||
|
||||
This is the remaining attended provider-admin operation after the owner-reported
|
||||
four-class cutover. It updates only privacyIDEA resolver `lldap-coulomb` so the
|
||||
resolver uses the replacement LLDAP bind credential. It does not modify realms,
|
||||
policies, tokens, KeyCape Secrets, or any other resolver.
|
||||
Both `sso-mfa/k8s/privacyidea/reconcile-lldap-resolver-live.sh` and
|
||||
`update-lldap-resolver-live.sh` delegate to `resolver_cutover.py`. The update
|
||||
helper changes only an existing `lldap-coulomb` resolver's bind password; the
|
||||
reconcile helper additionally checks cluster readiness, discovery, exact user
|
||||
resolution, token-backed MFA and the supplied predecessor's HTTP denial.
|
||||
Neither helper bootstraps a missing resolver or realm, repairs missing tuning,
|
||||
or changes policies, tokens, directory credentials or Kubernetes resources.
|
||||
|
||||
No password, hash, token, Secret payload, or manifest belongs in this document,
|
||||
State Hub, Git, chat, command arguments, or ordinary logs.
|
||||
## Authority and target pins
|
||||
|
||||
## Resolver update semantics — corrected 2026-10-13
|
||||
The historical platform recovery contract is railiance-platform `453fed3`;
|
||||
owner cutover receipt is State Hub message
|
||||
`45b236c8-052f-43d3-a472-44f8e9694da2`. These identify historical evidence,
|
||||
not a fresh live grant. Founder-deferred exposure rotation remains deferred.
|
||||
|
||||
Disposable native privacyIDEA 3.12.2 tests and its owner API implementation prove
|
||||
that omitted resolver fields are preserved. The historical general statement
|
||||
below that every POST replaces the whole object is superseded for this verified
|
||||
version. Sending explicit defaults can still overwrite custom tuning. Historical
|
||||
incident receipts and their target-version limitations remain unchanged.
|
||||
The current general realm repair sends only missing tuning fields on an existing
|
||||
resolver and never rotates its bind credential or replays MFA policies. The
|
||||
incident-specific attended cutover remains a separate approved procedure; do not
|
||||
substitute or combine the two.
|
||||
Before any live run, the existing owner/supervisor path must bind a private
|
||||
approval record to the current NetKingdom source revision and dependencies,
|
||||
incident, provider-approved replacement, owner receipt, exact start/end window,
|
||||
attended driver, independent abort operator and recovery plan. Confirm predecessor
|
||||
provenance separately. Do not restore an exposed predecessor or bundle.
|
||||
A preview digest and TTY are technical preconditions, not authorization.
|
||||
|
||||
## Authority and pinning gate
|
||||
Reconcile requires an explicit absolute kubeconfig, context, expected API server
|
||||
and `kube-system` namespace UID from the reviewed owner packet. It checks the
|
||||
server and UID before workload reads. It never reads Kubernetes Secrets. The
|
||||
five existing deployments must each have the current generation observed and
|
||||
exactly one desired, actual, updated, ready and available replica, without a
|
||||
deletion timestamp. No ambient default context is accepted.
|
||||
|
||||
The operator must record a private approval receipt containing, at minimum:
|
||||
Default service origins are `https://pink.coulomb.social`,
|
||||
`https://lldap.coulomb.social` and `https://kc.coulomb.social`. Explicit literal
|
||||
loopback URLs with ports are supported for approved forwards or fixtures.
|
||||
Environment URL/tuning overrides and the old `--note` option are removed.
|
||||
KeyCape discovery must identify the canonical `https://kc.coulomb.social`
|
||||
issuer and its own authorization, token and JWKS endpoints, even through a forward.
|
||||
The shared transport disables proxies and redirects and bounds strict JSON
|
||||
responses; each extra service profile permits only its one required route and
|
||||
never receives the privacyIDEA authorization token.
|
||||
|
||||
- incident `KEYCAPE-EXPOSURE-20260823-01`;
|
||||
- the NetKingdom revision containing `reconcile-lldap-resolver-live.sh` and its adjacent `pi_api.py` and platform contract `453fed3`;
|
||||
- the owner receipt `45b236c8-052f-43d3-a472-44f8e9694da2`;
|
||||
- the exact start/end window, attended driver, and independent abort operator;
|
||||
- confirmation that the replacement LLDAP credential is the provider-approved
|
||||
value and that no exposed predecessor will be restored.
|
||||
## Preview, apply and check
|
||||
|
||||
No live action is permitted if any revision, owner, cluster, or approval
|
||||
identifier differs from the receipt.
|
||||
Use an attended terminal from the reviewed checkout. These are command templates;
|
||||
replace public placeholders with the approved target values. No secret belongs
|
||||
in argv, environment, Git, State Hub, chat or ordinary logs.
|
||||
|
||||
## Preflight (metadata and health only)
|
||||
```bash
|
||||
bash sso-mfa/k8s/privacyidea/reconcile-lldap-resolver-live.sh --preview \
|
||||
--kubeconfig /absolute/approved/kubeconfig --context APPROVED_CONTEXT \
|
||||
--server https://APPROVED_SERVER:6443 --cluster-uid APPROVED_NAMESPACE_UID
|
||||
```
|
||||
|
||||
Run from the approved operator workstation, with the cluster context and
|
||||
provider endpoint already authorized. Do not render any Secret data.
|
||||
Preview prompts only for pi-admin, authenticates with an admin-role check,
|
||||
validates the existing resolver/core/tuning/realm, and returns a public-state
|
||||
`snapshot_sha256`. It performs no configuration write. It creates an auth session
|
||||
and can produce provider audit events. The digest excludes the bind password and
|
||||
binds the target settings and observed non-secret configuration. It is not a
|
||||
credential fingerprint.
|
||||
|
||||
1. Verify the approved checkout contains `reconcile-lldap-resolver-live.sh`
|
||||
with mode `0755`; inspect its source, not live credential material.
|
||||
2. Confirm LLDAP, privacyIDEA, KeyCape, Authelia, and
|
||||
identity-provisioner are Ready using deployment/pod status fields only.
|
||||
3. Confirm privacyIDEA availability via the documented unauthenticated
|
||||
`/token/` probe (401/403 is expected) and KeyCape discovery health,
|
||||
discarding response bodies. Do not use a command that prints a bearer token
|
||||
or configuration response.
|
||||
4. Confirm the approved window, driver, abort operator, provider custody, and
|
||||
cleanup workspace are ready. Stop on any drift or missing owner.
|
||||
Run the same command with `--apply --expected-digest REVIEWED_DIGEST` in place
|
||||
of `--preview`, retaining all target and user options. The helper prompts silently
|
||||
for pi-admin, the approved replacement and the actual predecessor. Inputs stay
|
||||
in Python memory; no temporary secret files are created. Inherited shell tracing
|
||||
is disabled before input handling. A fresh OTP/PIN is requested only after
|
||||
configuration and exact user lookup succeed. `--user` defaults to `platform-root`.
|
||||
|
||||
## Apply (one attended operation)
|
||||
Before mutation the helper authenticates the replacement to LLDAP and uses
|
||||
privacyIDEA `/resolver/test` with the candidate password to prove an actual LDAP
|
||||
bind. It tests supplied predecessor denial at LLDAP HTTP login and rechecks the
|
||||
observed configuration. Apply sends exactly one resolver update containing only
|
||||
`type` and `BINDPW`. Native privacyIDEA 3.12.2 preserves omitted fields. Readback
|
||||
must preserve all other resolver data and every realm; a native resolver test
|
||||
then proves the stored bind works. The API has no compare-and-swap: an exclusive
|
||||
owner window remains required, and ambiguous writes are never retried.
|
||||
|
||||
1. From the approved checkout, run exactly:
|
||||
`--check` replaces `--preview` to perform the proofs without a configuration
|
||||
write; it needs no digest. Check proves the stored bind works, not that its
|
||||
censored value equals the supplied replacement. Authentication, audit and OTP
|
||||
counter effects still occur. It is not an effect-free operation.
|
||||
|
||||
```text
|
||||
bash sso-mfa/k8s/privacyidea/reconcile-lldap-resolver-live.sh --apply
|
||||
```
|
||||
The smaller `update-lldap-resolver-live.sh` supports the same preview/apply/check
|
||||
sequence without cluster flags. It proves candidate and stored binds plus exact
|
||||
configuration preservation, but does not claim MFA, readiness or predecessor
|
||||
acceptance. It must not substitute for full incident acceptance.
|
||||
|
||||
The helper securely prompts for pi-admin, replacement and predecessor
|
||||
LLDAP passwords, and one approved MFA code; authenticates to privacyIDEA;
|
||||
performs exactly one `POST /resolver/lldap-coulomb`; proves replacement
|
||||
resolver lookup, privacyIDEA MFA, replacement LLDAP authentication, and
|
||||
predecessor denial; rechecks readiness/health; securely cleans up; and
|
||||
emits one sanitized PASS/FAIL receipt. It must not be combined with
|
||||
`repair-realm-live.sh`, `bootstrap-realm.sh`, `creds-rotate.sh`, or any
|
||||
full-bundle generator.
|
||||
3. Stop immediately on any non-success response, timeout, unexpected endpoint,
|
||||
or failed cleanup. Do not restore the exposed bundle or predecessor
|
||||
credential.
|
||||
## Receipts and recovery
|
||||
|
||||
## Postflight and predecessor denial
|
||||
Receipts contain fixed phase names, booleans, write counts and a public-state
|
||||
digest; no raw responses, exceptions or free-form notes. `configured` means the
|
||||
update helper completed its apply; `checked` means its no-write proof completed.
|
||||
Reconcile returns `verified` only when all helper proofs pass. Passing a wrong
|
||||
arbitrary password also produces denial: `supplied_predecessor_denied` is not
|
||||
proof of predecessor provenance. The owner must match the input to custody.
|
||||
|
||||
Record only status codes, readiness, timestamps, revision identifiers, and
|
||||
boolean results.
|
||||
Use `--predecessor-unavailable` honestly when it cannot be produced. Reconcile
|
||||
then returns `incomplete`, exit 2, even if the other proofs pass; never type a
|
||||
placeholder to obtain a passing receipt. Failures and handled interruptions
|
||||
return `unverified`, exit 1, with `write_attempted` showing whether a partial
|
||||
change may exist. Exit 0 still leaves `incident_acceptance: unverified`: the
|
||||
separate owner receipt and actual KeyCape MFA path are outside this helper.
|
||||
|
||||
1. Confirm privacyIDEA, LLDAP, KeyCape, Authelia, and identity-provisioner are
|
||||
Ready again. Confirm the privacyIDEA health endpoint succeeds and the
|
||||
resolver endpoint returns success without retaining its response body.
|
||||
2. Exercise one approved KeyCape MFA path that requires the `coulomb` realm.
|
||||
Record pass/fail only; never record the token or response body.
|
||||
3. Using protected file inputs, prove an LDAP bind with the replacement value
|
||||
succeeds and a bind with the exposed predecessor fails. The predecessor
|
||||
test must be a boolean result and must not put the password in argv or
|
||||
stdout. A failed predecessor bind is required evidence; do not retry it
|
||||
against another provider.
|
||||
4. Confirm the KeyCape owner’s existing four-class positive/negative receipt
|
||||
remains associated with this resolver update. If any class lacks a receipt,
|
||||
keep T05 open.
|
||||
5. Securely remove the temporary workspace and record only cleanup success.
|
||||
Stop on drift, wrong target, failed health/readiness or any unverified result.
|
||||
After an attempted write, reconcile observed state through the approved owner
|
||||
path before another attempt. Recover forward with an approved replacement; no
|
||||
automatic rollback to an exposed predecessor. Record only sanitized evidence,
|
||||
source revision, target references and operator outcome. NK-WP-0049 T03 remains
|
||||
the owner-native recovery/acceptance gate; this source repair does not close it.
|
||||
|
||||
## Abort and rollback
|
||||
## Historical receipts and corrections
|
||||
|
||||
Abort before mutation on revision drift, missing authority, unavailable health,
|
||||
uncertain workspace cleanup, or any unsafe helper output. Abort forward after
|
||||
mutation on a failed resolver response, failed readiness, failed replacement
|
||||
MFA, or missing predecessor denial. The exposed bundle and every exposed
|
||||
predecessor are never rollback material. Recovery after a partial write must
|
||||
use a newly approved replacement value and revision, not a stale local bundle.
|
||||
The old `--check --predecessor-unavailable` printed PASS on 2026-09-23 when run by
|
||||
Bernd Worsch from railiance01 checkout `6096c395` (receipt in NK-WP-0033).
|
||||
That remains historical check evidence, with predecessor denial unproven; it
|
||||
cannot establish the new revision's acceptance. Historical apply last ran on
|
||||
2026-08-27 before fixes in `4a38511`.
|
||||
|
||||
## Completion evidence
|
||||
|
||||
T03 may move to done only after the helper run and cleanup receipt are recorded
|
||||
by the attended operator. T05 may move to done only after the resolver’s
|
||||
replacement success, predecessor denial, owner cutover receipt, and all
|
||||
residual limitations are recorded as sanitized evidence.
|
||||
|
||||
## Revision note — 2026-08-27
|
||||
|
||||
The pinned revision of `reconcile-lldap-resolver-live.sh` had never completed a
|
||||
run. Four defects were found by running it and are fixed in `4a38511`; see
|
||||
`NK-WP-0033` T05 for the full findings. Two change how this procedure is
|
||||
invoked:
|
||||
|
||||
- `--predecessor-unavailable` — use when the exposed predecessor cannot be
|
||||
produced. The denial bind is not attempted and the receipt records
|
||||
`predecessor denial=NOT-PROVEN`. Do **not** type a placeholder at the
|
||||
predecessor prompt instead: a wrong value also fails the bind, and the run
|
||||
records it as a *passing* denial proof — a receipt asserting a test that never
|
||||
ran.
|
||||
- `--note TEXT` — operator context carried verbatim in the receipt line, so a
|
||||
claim and its caveat travel together. One line, 200 characters, no
|
||||
credentials.
|
||||
|
||||
`TIMEOUT`, `CACHE_TIMEOUT` and `SIZELIMIT` are now sent with the resolver body
|
||||
(default 5 / 120 / 500, overridable via the matching `LDAP_*` environment
|
||||
variables). Before this fix, `--apply` dropped them, because a resolver write
|
||||
replaces the whole object. A resolver with them unset still resolves users, but
|
||||
the WebUI refuses to save or test it — so a hand repair was silently reverted by
|
||||
the next run.
|
||||
The August account attributed missing tuning to whole-object replacement.
|
||||
Native privacyIDEA 3.12.2 and source inspection corrected that interpretation on
|
||||
2026-10-13: omitted fields are preserved, while sending explicit defaults can
|
||||
overwrite custom tuning. The October helper therefore sends only the bind
|
||||
password and type. Historic claims, default-overwriting commands, temporary-file
|
||||
handling and raw `--note` receipts are superseded by this procedure.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue