coulomb/runbook-tutorials
5
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Rename open to stalled and autopilot to auto-confirm; add pushed state; declare idempotency; propose maturity and projects model (RBT-WP-0008)
All checks were successful
CI Smoke / host-smoke (push) Successful in 0s
Details
CI Smoke / container-smoke (push) Successful in 2s
Details
Work Records / validate (push) Successful in 14s
Details

Browse source 
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Assistant: claude-code
Assistant-Model: sonnet
Assistant-Process: 295952@bnt-lap001
Assistant-Session: e93f64ad-516c-46eb-9666-aad8d300c477
This commit is contained in:
tegwick 2026-09-29 12:19:58 +02:00
parent f39d5cec3c
commit f5619e1a52
19 changed files with 429 additions and 105 deletions
Download patch file Download diff file Expand all files Collapse all files

106
docs/maturity-and-projects.md Normal file
View file

@ -0,0 +1,106 @@
# Runbook maturity, projects and idempotency
Status: **proposal for decision** (RBT-WP-0008-T01). Raised by Bernd on 2026-09-29.
## The problem
`exercise_status: unexercised | exercised <date> by <operator>` mixes three different things:
1. **What the runbook is like** (does it change state, can it be repeated, can it be undone).
2. **How much the runbook can be trusted** (has it been rehearsed, and where).
3. **What state an environment is in** (has this runbook been applied to *this* system).
A run that succeeded on one system says little about another system, so "exercised" is not a
property of the runbook alone. Environments differ, and applying a runbook to an environment is a
state of that environment, not of the runbook.
## Terminology survey
From general DevOps, SRE, configuration-management and ITIL practice, written from memory, not from
a source review. Please treat the names as candidates.
| Area | Terms in common use | Relevance here |
| --- | --- | --- |
| Automation maturity | manual, semi-automated (guided), automated; the "runbook automation ladder"; toil reduction (SRE) | our modes: manual, auto-confirm, later auto-pilot with execution |
| Rehearsal | game day, fire drill, disaster-recovery drill, dry run, rehearsal in staging, `--check` mode (Ansible), `plan` before `apply` (Terraform) | how a runbook earns trust without touching production |
| Readiness | production readiness review (PRR), operational readiness, definition of done | a maturity gate before a runbook is used on production |
| Repeatability | idempotent, convergent (declare desired state), re-runnable, resumable, reversible; drift; "converged" | idempotency is a *declared characteristic*, not a state |
| Change control (ITIL) | standard change (pre-approved because proven repeatable), normal change, change model | maps to "proven" runbooks that need no fresh review |
| Environments | environment / stage (dev, test, staging, production), target, inventory and group variables (Ansible), workspace (Terraform), context (kubectl), installation, instance | the thing runbooks are applied to |
| Decay | runbook rot, stale, drift between docs and system | maturity must expire when the runbook or the system changes |
## Proposal: three separate axes
### 1. Characteristics: declared by the author, static
- **Idempotency**: `idempotent | non-idempotent | conditional` (with a note). Declared per step; the
runbook's value is derived (all idempotent, or any non-idempotent, or unknown if a step is undeclared).
**Implemented 2026-09-29** in the v0.2 schema and validator. Values are words, not yes/no, because
YAML reads unquoted `yes`/`no` as booleans.
- **Reversibility**: already declared per step (`reversible`), with rollback text.
- **Effect class** (candidate): read-only, changes-state, destructive.
- Consequences to build: retry of a failed non-idempotent step should require the operator to confirm
that the earlier attempt did not partly apply; re-running a whole idempotent runbook against a
converged environment is safe and is the way to verify it.
### 2. Maturity: earned, independent of any environment, bound to a pack revision
Derived from run history, not typed in. A run records a digest of the pack revision it used.
| Level | Meaning (candidate names) |
| --- | --- |
| `draft` | valid against the spec, never run |
| `reviewed` | a second person has read it (recorded) |
| `rehearsed` | completed cleanly (no deviation) in a **rehearsal** environment |
| `proven` | completed cleanly in at least two distinct environments by at least two operators |
| `stale` | the pack changed since it was proven: maturity is bound to a revision and expires when it changes |
This replaces `exercise_status`. Deviations (overrides, waivers, push-through) never count towards
maturity, which matches the current rule for exercising.
### 3. Projects: environments the runbooks are applied to
A **project** groups runbooks that share one environment. It holds what they have in common and provides
it as datapoint defaults (like Ansible group variables), so a value such as the target host or tenant is
entered once.
```yaml
project: railiance01-identity
environment: production # rehearsal | staging | production (class, not free text)
datapoints: {tenant: coulomb, target_host: railiance01} # non-secret; never secrets
runbooks:
- {pack: nk.ssh-certificates}
- {pack: nk.openbao-operating-path, after: [nk.ssh-certificates]}
```
Project state is derived from the latest run of each member at the current pack revision:
| State (candidate) | Meaning |
| --- | --- |
| `planned` | no member applied yet |
| `applying` | a run is in progress, or some members are applied and others not (the "running" state) |
| `applied` | every member's latest run succeeded cleanly at the current revision |
| `stale` | a member's pack changed, or a run failed or was abandoned, since it was applied |
| `drifted` (later) | verification against the live system no longer matches |
A **rehearsal** project is where maturity evidence comes from. "Exercised" then has a precise meaning:
*applied successfully in a rehearsal project*. A production project is `applied`, not "exercised".
## Consequences for what exists
- Runs record `project`, the environment class and the pack digest. Scores become per project as well as per
runbook. Run records stay non-secret and local (`.local/`), so shared history remains an open question.
- `exercise_status` in `pack.yaml` is superseded. Until this is decided it stays as the interim marker.
- Project files are declarative and non-secret. Candidate location: the repository that owns the system
(for example `net-kingdom/projects/<name>/project.yaml`), next to `runbooks/`.
## Open questions
1. **Names.** *Project* is Bernd's word. Alternatives: *target*, *engagement*, *installation*, *rollout*. Is
*environment class* (`rehearsal|staging|production`) the right way to separate rehearsal from real?
2. **Maturity thresholds.** Is "two environments, two operators" the right bar for `proven`? Should `reviewed`
be part of the ladder?
3. **Where projects live**, and whether one runbook may belong to several projects (yes, one project per
environment, so a runbook applies to many).
4. **Retry policy** for non-idempotent steps, as described above.
5. **Shared run history**: committing a redacted run summary versus keeping records local.

15
docs/pack-spec-v0.2.md
View file

@ -154,7 +154,7 @@ A **run** has a mode. The mode belongs to the run, not to the pack.
| Mode | Who confirms | Checks, gates, preconditions |
| --- | --- | --- |
| **manual** | the operator | shown with their state; the operator confirms each |
| **autopilot** | the engine, when every precondition is met | evaluated by the engine; confirmation is applied automatically |
| **auto-confirm** | the engine, when every precondition is met | evaluated by the engine; confirmation is applied automatically |
### Manual mode: override
@ -164,10 +164,10 @@ downstream risk (`downstream_risk` text declared on the step or gate, else a
generic warning), (2) require an explicit confirmation, and (3) record a
**deviation** in the run.
### Autopilot mode: timeout and push-through
### Auto-confirm mode: timeout and push-through
- A step or gate may declare `timeout: {seconds: N, on_timeout: stop|push-through}`;
a run may set a default. Without a timeout, autopilot waits.
a run may set a default. Without a timeout, auto-confirm waits.
- `stop` (default): on timeout the run stops and reports.
- `push-through`: on timeout the engine confirms the item, records a deviation,
and moves on, ignoring the problem. This gathers the most information about
@ -176,10 +176,13 @@ generic warning), (2) require an explicit confirmation, and (3) record a
with an explicit, recorded pre-authorization naming those steps (and the pack's
`confirm_phrase`). Push-through does not bypass that rule. This is a safety
default, open to review.
- A step may declare `autopilot: manual-only` to forbid automation entirely.
- A step may declare `auto_confirm: manual-only` to forbid automation entirely.
- Auto-confirm applies *confirmation* only. It does not run a step's own command; execution is a later, separate level (RBT-WP-0006).
### Deviations
An item passed over by push-through gets the distinct state `pushed` (steps, gates and datapoints): not done, not set, not an error.
Every override, waiver and push-through is a **deviation** in the run record:
`kind` (`override|waiver|push-through`), `item` (step or gate id), `mode`,
`unmet` (what was not satisfied), `risk_acknowledged`, `reason` (required for
@ -201,6 +204,10 @@ declared in advance (`waivable` on a gate); an **override** is not.
## Later: dependencies between runbooks
**Idempotency** is a declared characteristic of a step: `idempotent: idempotent | non-idempotent | conditional`
(`conditional` needs an `idempotent_note`). Undeclared means unknown. The runbook's overall value is derived.
See `docs/maturity-and-projects.md`.
Not in v0.2. When a procedure needs another runbook (for example "OpenBao operating
path requires the SSH access pack"), packs will declare `requires_packs` with a
required result (exercised, or a named gate passed). Tracked in RBT-WP-0006.

31
docs/run-spec.md
View file

@ -27,17 +27,17 @@ CLI: `rtut run new|list|status|cmd|start|verify|done|fail|retry|skip|finish <pac
## Outcome, score and exercise status
A run is **open** until finalized. `rtut run finalize` (or the UI's *Close this run*) records
A run is unfinished (**stalled**) until finalized. `rtut run finalize` (or the UI's *Close this run*) records
an `outcome`:
| Outcome | Rule |
| --- | --- |
| `succeeded` | every step is `done` (skipped steps do not count) |
| `failed` | at least one step has failed; otherwise abandon |
| `abandoned` | any time. An open run untouched for 7 days also counts as abandoned in the score |
| `abandoned` | any time. An unfinished run untouched for 7 days also counts as abandoned in the score |
A finished run is read-only, except for feedback. The **score** of a runbook is derived from
its run records: tried, succeeded, failed, abandoned, open, plus a tally of feedback ratings
its run records: tried, succeeded, failed, abandoned, stalled (unfinished), plus a tally of feedback ratings
(`rtut score <root>`; shown in the UI). Run records live in `.local/` of the machine that ran
them, so the score is per machine until run histories are shared.
@ -67,12 +67,12 @@ Implemented in `rtut/run2.py` (graph in `rtut/flow.py`). It adds to the v0.1 rec
| Field | Meaning |
| --- | --- |
| `mode` | `manual` or `autopilot` |
| `datapoints.<id>` | `state` nil, set, err or ok; `value`, `how` (initial, manual, produced, check, manual confirmation), `error` |
| `steps.<id>` | as v0.1, plus `alternative` (chosen `one_of` branch), `authorized_by`, `pushed_through` |
| `gates.<id>` | `state` open, passed, failed or waived; per-check `results`; `waiver` |
| `mode` | `manual` or `auto-confirm` |
| `datapoints.<id>` | `state` nil, set, err, ok or pushed; `value`, `how` (initial, manual, produced, check, manual confirmation), `error` |
| `steps.<id>` | as v0.1 (`pushed` is an additional state), plus `alternative` (chosen `one_of` branch), `authorized_by` |
| `gates.<id>` | `state` open, passed, failed, waived or pushed; per-check `results`; `waiver` or `pushed` |
| `deviations[]` | every override, waiver and push-through (see `docs/pack-spec-v0.2.md`) |
| `autopilot` | `preauthorized` irreversible steps, `authorized_by`, and `stopped` if a timeout stopped the run |
| `auto-confirm` | `preauthorized` irreversible steps, `authorized_by`, and `stopped` if a timeout stopped the run |
Rules the engine enforces:
@ -88,13 +88,20 @@ Rules the engine enforces:
- **Override** (manual mode only): starting an item with unmet preconditions needs a reason and an acknowledged
downstream risk; it is recorded as a deviation.
- **Waiver**: only for gates that declare `waivable`, with a reason; recorded as a deviation.
- **Confirmation**: irreversible steps need the exact confirmation phrase at start. In autopilot they run only
if the run pre-authorized them, and never if the step is `autopilot: manual-only`.
- **Autopilot** (`autopilot_tick`): the engine applies confirmation when preconditions are met: it starts the
- **Confirmation**: irreversible steps need the exact confirmation phrase at start. In auto-confirm they run only
if the run pre-authorized them, and never if the step is `auto_confirm: manual-only`.
- **Auto-confirm** (`auto-confirm_tick`): the engine applies confirmation when preconditions are met: it starts the
recommended item, runs its checks (manual checks are auto-confirmed and marked so) and completes it. It
never runs a step's own command; the work is done by the operator or, later, by opt-in execution. A step
with `evidence` waits for the operator. If checks keep failing until the item's `timeout` (or the pack
default), the run stops, or with `on_timeout: push-through` the item is marked done with
`pushed_through` and a deviation is recorded.
state `pushed` and a deviation is recorded.
- **Deviations and exercise**: a run with any deviation or pushed-through step can be finalized but never exercises a pack.
`finalize(succeeded)` also needs every gate passed or waived.
- **Pushed is its own state.** A step, gate or datapoint passed over by push-through is `pushed`: not `done`,
not `set`, not `err`. Dependents may proceed (that is the point of pushing through), but a pushed datapoint
has no value, so a command that needs it fails its check instead of running with nothing. A run with pushed
items can be finished as `failed` (never `succeeded`) and never exercises a pack.
- **Stalled**: an unfinished run is counted as `stalled` in the score, and as `abandoned` after a week.
- **Modes** are `manual` and `auto-confirm`. Auto-confirm applies confirmation only; running the step's own
command is a separate, later level ("auto-pilot with execution", RBT-WP-0006).