docs: name where operator credentials live, and label unexercised procedures
Three gaps from NK-WP-0033, none of them the fault we set out to fix. Operator custody (platform-root-custody.md). This document named a password-safe entry that had never been created, so the 2026-08-23 rotation had nowhere to put the new LLDAP admin value and the outgoing one survived only in a browser entry that was overwritten on 2026-08-27. Decided: operators/<subsystem>/<account> in OpenBao for everything reachable once logged in, plus a short closed offline list for what you need when OpenBao is not — platform-root's own password, unseal shares, break-glass. Anything on that list in OpenBao would be circular; anything not on it does not belong in the packet. The rule that closes it: every rotation names, as a numbered step, where the outgoing and incoming values go. NK-WP-0033-T04 rotated four classes with no such step, and that is the root cause of the session that followed — not the four software defects it also surfaced. Secret disclosure by trust stage, tied to the existing S0-S6 table. A fingerprint may be spoken to confirm two parties hold the same credential; the whole value never may. Used ad hoc throughout 2026-08-27 and worth writing down. Attended procedures (new standard). reconcile-lldap-resolver-live.sh was approved, documented, revision-pinned and had never run; its first execution found four defects, two of which meant it could never have completed. verify-t06.sh passed throughout. An unexecuted procedure is unfinished work and must say so in its runbook header. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Assistant: claude-code Assistant-Model: opus Assistant-Process: 3377672@bnt-lap001 Assistant-Session: 15463ccf-238f-4e13-b163-93aa25c6d166
This commit is contained in:
parent
fd86397f52
commit
c99b8da41f
3 changed files with 160 additions and 4 deletions
71
docs/attended-procedure-standard.md
Normal file
71
docs/attended-procedure-standard.md
Normal file
|
|
@ -0,0 +1,71 @@
|
||||||
|
# Attended procedures: a script that has never run is not a procedure
|
||||||
|
|
||||||
|
*Adopted 2026-08-28, after `NK-WP-0033`.*
|
||||||
|
|
||||||
|
## What happened
|
||||||
|
|
||||||
|
`sso-mfa/k8s/privacyidea/reconcile-lldap-resolver-live.sh` was written, reviewed,
|
||||||
|
documented in an approved runbook, pinned by revision, and referenced by an
|
||||||
|
incident workplan as *the* supported one-command attended operation. Its own
|
||||||
|
workplan entry said plainly: "It has not been run."
|
||||||
|
|
||||||
|
The first execution, on 2026-08-27, found four defects:
|
||||||
|
|
||||||
|
1. `Content-Type: application/json` on bodyless GETs — Werkzeug rejects those in
|
||||||
|
front of the application, so **every** GET the script made returned an HTML
|
||||||
|
400. POSTs worked, which made the failure look like a server-state problem.
|
||||||
|
2. `GET /user/` parsed as a dict when privacyIDEA returns a list — an
|
||||||
|
`AttributeError` outside the `except` clause, so the run died as a traceback
|
||||||
|
rather than a receipt.
|
||||||
|
3. Resolver writes omitting `TIMEOUT`, `CACHE_TIMEOUT`, `SIZELIMIT` — a resolver
|
||||||
|
write replaces the whole object, so each run silently un-repaired a resolver
|
||||||
|
an operator had just fixed by hand.
|
||||||
|
4. No way to declare an unavailable predecessor — the prompt could not be left
|
||||||
|
empty, so an operator had to type a placeholder, which also fails the bind
|
||||||
|
and was recorded as a **passing** denial proof.
|
||||||
|
|
||||||
|
The first two meant the script could never have completed a run under any
|
||||||
|
circumstances. It had been treated as ready for an incident for four days.
|
||||||
|
|
||||||
|
`verify-t06.sh` reported success throughout.
|
||||||
|
|
||||||
|
## The rule
|
||||||
|
|
||||||
|
**An unexecuted procedure is unfinished work, and must be labelled as such.**
|
||||||
|
|
||||||
|
A procedure is *exercised* when it has completed successfully at least once
|
||||||
|
against a real or scratch target, and the operator who ran it is recorded. Until
|
||||||
|
then it is *unexercised*, whatever its review status.
|
||||||
|
|
||||||
|
Every attended procedure carries this in its runbook, in the header:
|
||||||
|
|
||||||
|
```markdown
|
||||||
|
Exercise status: exercised 2026-08-27 by <operator> | unexercised
|
||||||
|
```
|
||||||
|
|
||||||
|
An unexercised procedure may still be the approved path — sometimes the incident
|
||||||
|
*is* the first run. What it may not do is present itself as routine. Naming the
|
||||||
|
status sets the operator's expectations correctly: run it expecting to debug it,
|
||||||
|
budget time accordingly, and do not schedule it as the last step of a window.
|
||||||
|
|
||||||
|
## Consequences for verification
|
||||||
|
|
||||||
|
A verification script that passes against a procedure that cannot execute is not
|
||||||
|
verifying anything. Where a `verify-*.sh` exists, it must assert the property the
|
||||||
|
procedure is supposed to establish — not that its objects exist, and not that its
|
||||||
|
API calls returned 200.
|
||||||
|
|
||||||
|
`verify-t06.sh` checked that a resolver and realm existed. It did not check that
|
||||||
|
`GET /user/?realm=coulomb&username=<user>` returns a user, which is the entire
|
||||||
|
point of a resolver and would have caught defects 1 and 3 on the day they landed.
|
||||||
|
That gap is open and tracked in `NK-WP-0033-T05`.
|
||||||
|
|
||||||
|
## Consequences for shared helpers
|
||||||
|
|
||||||
|
Defect 1 had already been found and fixed in `bootstrap-realm.sh`'s `pi_api`
|
||||||
|
helper, in the same directory, with a comment explaining the cause. The reconcile
|
||||||
|
script was written later and reimplemented the request path without the fix.
|
||||||
|
|
||||||
|
Two implementations of the same API call in one directory is one too many. New
|
||||||
|
scripts against an existing API extend the existing helper or import it; they do
|
||||||
|
not start a second one.
|
||||||
|
|
@ -5,6 +5,7 @@ Workplan: `NK-WP-0033`
|
||||||
NetKingdom procedure: `sso-mfa/k8s/privacyidea/reconcile-lldap-resolver-live.sh` (repaired 2026-08-27 in `4a38511`; the previously pinned revision could not complete a run — see NK-WP-0033 T05)
|
NetKingdom procedure: `sso-mfa/k8s/privacyidea/reconcile-lldap-resolver-live.sh` (repaired 2026-08-27 in `4a38511`; the previously pinned revision could not complete a run — see NK-WP-0033 T05)
|
||||||
Platform recovery contract: railiance-platform `453fed3`
|
Platform recovery contract: railiance-platform `453fed3`
|
||||||
Owner cutover receipt: State Hub message `45b236c8-052f-43d3-a472-44f8e9694da2`
|
Owner cutover receipt: State Hub message `45b236c8-052f-43d3-a472-44f8e9694da2`
|
||||||
|
Exercise status: exercised 2026-08-27 by Bernd Worsch — first run ever; four defects found and fixed (`4a38511`). See `attended-procedure-standard.md`.
|
||||||
|
|
||||||
This is the remaining attended provider-admin operation after the owner-reported
|
This is the remaining attended provider-admin operation after the owner-reported
|
||||||
four-class cutover. It updates only privacyIDEA resolver `lldap-coulomb` so the
|
four-class cutover. It updates only privacyIDEA resolver `lldap-coulomb` so the
|
||||||
|
|
|
||||||
|
|
@ -99,12 +99,15 @@ LLDAP deliberately has no public registration flow. The first-user process is
|
||||||
administrator-provisioned:
|
administrator-provisioned:
|
||||||
|
|
||||||
1. Log in to `https://lldap.coulomb.social` as `admin`.
|
1. Log in to `https://lldap.coulomb.social` as `admin`.
|
||||||
2. Retrieve `LLDAP_LDAP_USER_PASS` from the operator password safe entry
|
2. Retrieve `LLDAP_LDAP_USER_PASS` from operator custody —
|
||||||
`net-kingdom/LLDAP/admin`.
|
`operators/lldap/admin` in OpenBao (see *Operator Credential Custody*).
|
||||||
|
Until 2026-08-28 this step named a password-safe entry that did not exist;
|
||||||
|
see that section for what that cost.
|
||||||
3. Create the dedicated `platform-root` or `king` user.
|
3. Create the dedicated `platform-root` or `king` user.
|
||||||
4. Add the user to `net-kingdom-admins` for the current lightweight path.
|
4. Add the user to `net-kingdom-admins` for the current lightweight path.
|
||||||
5. Store the new account password only in the password safe or offline custody
|
5. Store the new account password in operator custody (`operators/`) or, for a
|
||||||
packet.
|
bootstrap identity, in the offline packet — never only in a browser
|
||||||
|
password manager.
|
||||||
6. Use `pi-admin` in privacyIDEA to confirm that the LLDAP resolver can see
|
6. Use `pi-admin` in privacyIDEA to confirm that the LLDAP resolver can see
|
||||||
the new user and that self-enrollment is allowed.
|
the new user and that self-enrollment is allowed.
|
||||||
7. Log in to privacyIDEA self-service as `platform-root` and enroll the TOTP
|
7. Log in to privacyIDEA self-service as `platform-root` and enroll the TOTP
|
||||||
|
|
@ -124,6 +127,65 @@ custody unless the recorded king credential was intentionally changed to
|
||||||
`platform-admin`. `platform-admin` is the later non-root operator path; it is
|
`platform-admin`. `platform-admin` is the later non-root operator path; it is
|
||||||
useful, but it is not the same as proving the platform-root custody identity.
|
useful, but it is not the same as proving the platform-root custody identity.
|
||||||
|
|
||||||
|
## Operator Credential Custody
|
||||||
|
|
||||||
|
*Decided 2026-08-28, after `NK-WP-0033`.*
|
||||||
|
|
||||||
|
Every credential a human operator holds has exactly one recorded home. Before
|
||||||
|
this was written, this document named a password-safe entry
|
||||||
|
(`net-kingdom/LLDAP/admin`) that had never been created. The consequence was not
|
||||||
|
theoretical: the LLDAP admin credential was rotated on 2026-08-23 with nowhere
|
||||||
|
to put the new value, the outgoing value survived only in a browser password
|
||||||
|
manager, and it was destroyed on 2026-08-27 when that entry was overwritten with
|
||||||
|
the replacement. A full working session went into diagnosing a system whose
|
||||||
|
predecessor credential no longer existed anywhere.
|
||||||
|
|
||||||
|
### Two stores, and the line between them
|
||||||
|
|
||||||
|
**`operators/` in OpenBao — everything reachable once you are logged in.**
|
||||||
|
Service account passwords, provider PATs, the LLDAP admin bind, privacyIDEA
|
||||||
|
`pi-admin`, machine identities like `state-hub-svc`. Auditable, backed up with
|
||||||
|
the rest of the platform, reachable from any machine, and writable through
|
||||||
|
`warden desk --act paste_once_provision` so the value never reaches a terminal
|
||||||
|
or shell history.
|
||||||
|
|
||||||
|
Path convention: `operators/<subsystem>/<account>` — for example
|
||||||
|
`operators/lldap/admin`, `operators/privacyidea/pi-admin`,
|
||||||
|
`operators/forgejo/state-hub-svc`. Distinct from `platform/workloads/…`, which
|
||||||
|
holds credentials that *workloads* consume; `operators/` holds credentials that
|
||||||
|
*people* use.
|
||||||
|
|
||||||
|
**The offline packet — what you need when OpenBao is unreachable.** This list is
|
||||||
|
deliberately short, closed, and enumerated here rather than left to judgement:
|
||||||
|
|
||||||
|
- `platform-root`'s own password and TOTP recovery codes
|
||||||
|
- OpenBao unseal shares and root token
|
||||||
|
- the `break-glass` account password (`sso-mfa/k8s/lldap/break-glass.sh`)
|
||||||
|
|
||||||
|
Anything on that list in `operators/` would be a circular dependency: you would
|
||||||
|
need the store to reach the credential that opens the store. Anything *not* on
|
||||||
|
that list belongs in `operators/`, not in the packet — the packet is not a
|
||||||
|
general-purpose safe and does not scale.
|
||||||
|
|
||||||
|
### The rule that closes the gap
|
||||||
|
|
||||||
|
**Every rotation procedure must name, as a numbered step, where the outgoing and
|
||||||
|
incoming values go.** A rotation that changes a credential without saying what
|
||||||
|
happens to custody will lose one of the two, and which one it loses is a matter
|
||||||
|
of luck. `NK-WP-0033-T04` rotated four credential classes and had no such step;
|
||||||
|
that is the root cause of the session that followed, not any of the four
|
||||||
|
software defects it also surfaced.
|
||||||
|
|
||||||
|
The outgoing value is retained until its successor is verified *and* the
|
||||||
|
predecessor is proven rejected. Only then is it destroyed, and its destruction
|
||||||
|
is recorded.
|
||||||
|
|
||||||
|
### Not yet built
|
||||||
|
|
||||||
|
The `operators/` mount and its policy do not exist yet. Until they do, this
|
||||||
|
section describes intent rather than practice, and the same failure remains
|
||||||
|
available. Building it is a `railiance-platform` act, not an ops-mason one.
|
||||||
|
|
||||||
## Trust Progression
|
## Trust Progression
|
||||||
|
|
||||||
The platform moves through explicit trust stages:
|
The platform moves through explicit trust stages:
|
||||||
|
|
@ -141,6 +203,28 @@ The platform moves through explicit trust stages:
|
||||||
This prevents early bootstrap convenience from silently becoming permanent
|
This prevents early bootstrap convenience from silently becoming permanent
|
||||||
platform sovereignty.
|
platform sovereignty.
|
||||||
|
|
||||||
|
### Secret disclosure by stage
|
||||||
|
|
||||||
|
*Decided 2026-08-27, during `NK-WP-0033`.* Diagnosis stalls when two people
|
||||||
|
cannot establish whether they are discussing the same credential. What follows
|
||||||
|
is what may be said aloud — in a terminal, a transcript, a chat message, or a
|
||||||
|
ticket — about a secret's *value*, by stage.
|
||||||
|
|
||||||
|
| Stage | Permitted about a value |
|
||||||
|
| --- | --- |
|
||||||
|
| S0–S4 | A fingerprint: length, last three characters, or `sha256 \| head -c 8`. Enough to confirm two parties hold the same credential and no more. |
|
||||||
|
| S5+ | Fingerprint by hash only (`sha256 \| head -c 8`). No plaintext characters, on the grounds that a live platform's credentials outlive the conversation about them. |
|
||||||
|
| Any | Never the whole value, never a working prefix, never a screenshot of one. |
|
||||||
|
|
||||||
|
Prefer the hash form at every stage: it disambiguates just as well, it is what
|
||||||
|
this document's own evidence rules already ask for elsewhere ("public-key
|
||||||
|
fingerprints"), and it does not become a liability when a transcript is read
|
||||||
|
back later by someone who was not in the room.
|
||||||
|
|
||||||
|
This is a disclosure rule, not a handling rule. It says what may be *spoken*
|
||||||
|
about a value; it does not license reading a value out of a store that a role is
|
||||||
|
not otherwise entitled to read.
|
||||||
|
|
||||||
## Required Bootstrap Use Cases
|
## Required Bootstrap Use Cases
|
||||||
|
|
||||||
The guided bootstrap experience must cover at least:
|
The guided bootstrap experience must cover at least:
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue