coulomb/runbook-tutorials
5
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Retire exercise_status: maturity is the only notion; add CLI tests (RBT-WP-0009-T01)

Browse source 
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Assistant: claude-code
Assistant-Model: sonnet
Assistant-Process: 295952@bnt-lap001
Assistant-Session: e93f64ad-516c-46eb-9666-aad8d300c477
This commit is contained in:
tegwick 2026-09-29 21:28:56 +02:00
parent b89e8eaa74
commit 6d3414d8b7
29 changed files with 444 additions and 218 deletions
Download patch file Download diff file Expand all files Collapse all files

2
docs/maturity-and-projects.md
View file

@ -91,7 +91,7 @@ A **rehearsal** project is where maturity evidence comes from. "Exercised" then
- Runs record `project`, the environment class and the pack digest. Scores become per project as well as per
runbook. Run records stay non-secret and local (`.local/`), so shared history remains an open question.
- `exercise_status` in `pack.yaml` is superseded. Until this is decided it stays as the interim marker.
- `exercise_status` in `pack.yaml` is retired (RBT-WP-0009): the field is deprecated and ignored, and maturity is the only notion.
- Project files are declarative and non-secret. Candidate location: the repository that owns the system
(for example `net-kingdom/projects/<name>/project.yaml`), next to `runbooks/`.

4
docs/pack-spec-v0.2.md
View file

@ -187,7 +187,7 @@ Every override, waiver, push-through and change of a value that steps already us
`kind` (`override|waiver|push-through`), `item` (step or gate id), `mode`,
`unmet` (what was not satisfied), `risk_acknowledged`, `reason` (required for
overrides and waivers; automatic text for push-through), `who`, `at`. A run with
deviations is marked so in the overview and can never exercise a pack. A **waiver** is
deviations is marked so in the overview and never counts towards maturity. A **waiver** is
declared in advance (`waivable` on a gate); an **override** is not.
## Decisions (2026-09-29, Bernd)
@ -210,7 +210,7 @@ See `docs/maturity-and-projects.md`.
Not in v0.2. When a procedure needs another runbook (for example "OpenBao operating
path requires the SSH access pack"), packs will declare `requires_packs` with a
required result (exercised, or a named gate passed). Tracked in RBT-WP-0006.
required result (a maturity level, or a named gate passed). Tracked in RBT-WP-0006.
## Recommendation (carried out)

9
docs/pack-spec.md
View file

@ -14,7 +14,7 @@ under any root given on the command line.
| `id` | yes | Stable dotted id, lower-case, e.g. `nk.ssh-certificates` |
| `title`, `outcome` | title yes | What the operator is doing; what is true afterwards |
| `owner` | yes | The repository that owns the procedure |
| `exercise_status` | yes | `unexercised` or `exercised YYYY-MM-DD by <operator>` |
| `exercise_status` | no | **Deprecated and ignored.** Maturity is derived from run history; old packs that carry it still load |
| `engine` | yes | `native` or `legacy-command` |
| `parameters` | no | Operator-supplied, non-secret values |
| `prerequisites` | no | `{text, owner}` list |
@ -54,11 +54,10 @@ A step with `risk` of `changes-state`, `attended` or `destructive` must state a
Commands are relative to the root of the repository that owns the pack. Verify
commands run there, so a pack can call `make` targets and repo-relative scripts.
## Exercise status
## Maturity, not a status field
Following the attended-procedure standard, a pack is `unexercised` until an
operator has completed it, and the operator is recorded. The engine displays the
status prominently and only the engine's evidence writer changes it.
How far a pack can be trusted is earned, not declared: `draft`, `reviewed`, `rehearsed`, `proven`, or `stale`, derived from the
runs of the pack at its current revision (`docs/maturity-and-projects.md`). The earlier `exercise_status` field is retired.
## Secrets

29
docs/run-spec.md
View file

@ -18,14 +18,14 @@ One run of one native pack, stored as JSON in
- Steps run in order; a step starts only after its predecessors are done or skipped.
- A step completes only with a passing verification and its declared evidence.
- A verify command runs only when the operator confirms (`--yes`), with a 60 s timeout.
- Only read-only steps may be skipped, and a run with skips never exercises a pack.
- Only read-only steps may be skipped, and a run with skips can never succeed.
- Values matching the secret markers are refused everywhere they could be recorded.
- The engine never edits `pack.yaml`. `rtut run finish` shows the exercise-status
diff; `--apply` writes it, and only for a run in which every step is done.
- The engine never edits `pack.yaml`. How far a pack can be trusted is not a field in it: it is derived from run history
(`rtut maturity`, `docs/maturity-and-projects.md`).
CLI: `rtut run new|list|status|cmd|start|verify|done|fail|retry|skip|finish <pack.yaml> ...`
CLI: `rtut run new|list|status|cmd|start|verify|done|fail|retry|skip|feedback|finalize|capture <pack.yaml> ...`
## Outcome, score and exercise status
## Outcome and score
A run is unfinished (**stalled**) until finalized. `rtut run finalize` (or the UI's *Close this run*) records
an `outcome`:
@ -41,11 +41,10 @@ its run records: tried, succeeded, failed, abandoned, stalled (unfinished), plus
(`rtut score <root>`; shown in the UI). Run records live in `.local/` of the machine that ran
them, so the score is per machine until run histories are shared.
Finalizing never edits the pack by itself. For a `succeeded` run the UI offers **Record pack
as exercised**, which shows what will be written, and on click sets
`exercise_status: exercised <date> by <operator>` in `pack.yaml` (`rtut run finalize ... --outcome
succeeded --apply` does the same). The change is an ordinary edit in the owning repository, to
be reviewed and committed there.
Finalizing never edits the pack. A clean `succeeded` run becomes evidence for the pack's **maturity** at the
revision it ran (`docs/maturity-and-projects.md`); this replaces the earlier `exercise_status` field and the
"Record pack as exercised" step, both retired. Old packs that still carry `exercise_status` load unchanged; the
field is ignored.
## Feedback
@ -96,12 +95,12 @@ Rules the engine enforces:
with `evidence` waits for the operator. If checks keep failing until the item's `timeout` (or the pack
default), the run stops, or with `on_timeout: push-through` the item is marked done with
state `pushed` and a deviation is recorded.
- **Deviations and exercise**: a run with any deviation or pushed-through step can be finalized but never exercises a pack.
- **Deviations and maturity**: a run with any deviation or pushed-through step can be finalized but never counts as maturity evidence.
`finalize(succeeded)` also needs every gate passed or waived.
- **Pushed is its own state.** A step, gate or datapoint passed over by push-through is `pushed`: not `done`,
not `set`, not `err`. Dependents may proceed (that is the point of pushing through), but a pushed datapoint
has no value, so a command that needs it fails its check instead of running with nothing. A run with pushed
items can be finished as `failed` (never `succeeded`) and never exercises a pack.
items can be finished as `failed` (never `succeeded`) and never counts as maturity evidence.
- **Stalled**: an unfinished run is counted as `stalled` in the score, and as `abandoned` after a week.
- **Modes** are `manual` and `auto-confirm`. Auto-confirm applies confirmation only; running the step's own
command is a separate, later level ("auto-pilot with execution", RBT-WP-0006).
@ -110,12 +109,12 @@ Rules the engine enforces:
Closing a run (`finalize`) makes its record final:
- **Locked.** A closed record cannot be saved with changes: the engine refuses, naming what would have changed. The only later update it
accepts is `exercise_recorded`, written by the legacy "Record pack as exercised" step. This holds for runs closed before sealing existed too.
- **Locked.** A closed record cannot be saved with changes: the engine refuses, naming what would have changed. This holds for runs closed
before sealing existed too.
- **Feedback.** It locks with the run. Give it before closing, or with the closing form: `finalize(..., rating, comment)` records an optional
rating and comment together with the outcome (validated first, so a refused close changes nothing). This keeps abandoned and failed runs from
losing what the operator wanted to say.
- **Sealed.** The record carries a SHA-256 `seal` of its content (excluding `seal`, `updated` and `exercise_recorded`), and the file is made
- **Sealed.** The record carries a SHA-256 `seal` of its content (excluding `seal` and `updated`; a legacy `exercise_recorded` key on old records is ignored), and the file is made
read-only. `seal_state` reports `open`, `sealed`, `broken` (changed outside the app after closing) or `unsealed` (closed before sealing existed).
A broken seal is called out in the UI, and such a run no longer counts as maturity evidence. This is tamper-evidence, not prevention: someone
with file access can still edit a file, but not without being noticed.