tegwick
0150c8c37a
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Assistant: claude-code Assistant-Model: sonnet Assistant-Process: 295952@bnt-lap001 Assistant-Session: e93f64ad-516c-46eb-9666-aad8d300c477
75 lines
3.6 KiB
Markdown
75 lines
3.6 KiB
Markdown
# Attended procedures: a script that has never run is not a procedure
|
|
|
|
*Adopted 2026-08-28, after `NK-WP-0033`.*
|
|
|
|
## What happened
|
|
|
|
`sso-mfa/k8s/privacyidea/reconcile-lldap-resolver-live.sh` was written, reviewed,
|
|
documented in an approved runbook, pinned by revision, and referenced by an
|
|
incident workplan as *the* supported one-command attended operation. Its own
|
|
workplan entry said plainly: "It has not been run."
|
|
|
|
The first execution, on 2026-08-27, found four defects:
|
|
|
|
1. `Content-Type: application/json` on bodyless GETs — Werkzeug rejects those in
|
|
front of the application, so **every** GET the script made returned an HTML
|
|
400. POSTs worked, which made the failure look like a server-state problem.
|
|
2. `GET /user/` parsed as a dict when privacyIDEA returns a list — an
|
|
`AttributeError` outside the `except` clause, so the run died as a traceback
|
|
rather than a receipt.
|
|
3. Resolver writes omitting `TIMEOUT`, `CACHE_TIMEOUT`, `SIZELIMIT` — a resolver
|
|
write replaces the whole object, so each run silently un-repaired a resolver
|
|
an operator had just fixed by hand.
|
|
4. No way to declare an unavailable predecessor — the prompt could not be left
|
|
empty, so an operator had to type a placeholder, which also fails the bind
|
|
and was recorded as a **passing** denial proof.
|
|
|
|
The first two meant the script could never have completed a run under any
|
|
circumstances. It had been treated as ready for an incident for four days.
|
|
|
|
`verify-t06.sh` reported success throughout.
|
|
|
|
## The rule
|
|
|
|
**An unexecuted procedure is unfinished work, and must be labelled as such.**
|
|
|
|
A procedure is *exercised* when it has completed successfully at least once
|
|
against a real or scratch target, and the operator who ran it is recorded. Until
|
|
then it is *unexercised*, whatever its review status.
|
|
|
|
Every attended procedure carries this in its runbook, in the header:
|
|
|
|
```markdown
|
|
Exercise status: exercised 2026-08-27 by <operator> | unexercised
|
|
```
|
|
|
|
> **Runbook packs (2026-09-29).** For runbook packs run by the `runbook-tutorials` engine this label is no longer a field in the
|
|
> pack: the same idea is *maturity*, derived from clean runs in named environments and bound to the pack revision
|
|
> (`runbook-tutorials/docs/maturity-and-projects.md`). Markdown procedures keep the header below.
|
|
|
|
An unexercised procedure may still be the approved path — sometimes the incident
|
|
*is* the first run. What it may not do is present itself as routine. Naming the
|
|
status sets the operator's expectations correctly: run it expecting to debug it,
|
|
budget time accordingly, and do not schedule it as the last step of a window.
|
|
|
|
## Consequences for verification
|
|
|
|
A verification script that passes against a procedure that cannot execute is not
|
|
verifying anything. Where a `verify-*.sh` exists, it must assert the property the
|
|
procedure is supposed to establish — not that its objects exist, and not that its
|
|
API calls returned 200.
|
|
|
|
`verify-t06.sh` checked that a resolver and realm existed. It did not check that
|
|
`GET /user/?realm=coulomb&username=<user>` returns a user, which is the entire
|
|
point of a resolver and would have caught defects 1 and 3 on the day they landed.
|
|
That gap is open and tracked in `NK-WP-0033-T05`.
|
|
|
|
## Consequences for shared helpers
|
|
|
|
Defect 1 had already been found and fixed in `bootstrap-realm.sh`'s `pi_api`
|
|
helper, in the same directory, with a comment explaining the cause. The reconcile
|
|
script was written later and reimplemented the request path without the fix.
|
|
|
|
Two implementations of the same API call in one directory is one too many. New
|
|
scripts against an existing API extend the existing helper or import it; they do
|
|
not start a second one.
|