runbook-tutorials/docs/maturity-and-projects.md

# Runbook maturity, projects and idempotency

Status: **proposal for decision** (RBT-WP-0008-T01). Raised by Bernd on 2026-09-29.

## The problem

`exercise_status: unexercised | exercised <date> by <operator>` mixes three different things:

1. **What the runbook is like** (does it change state, can it be repeated, can it be undone).
2. **How much the runbook can be trusted** (has it been rehearsed, and where).
3. **What state an environment is in** (has this runbook been applied to *this* system).

A run that succeeded on one system says little about another system, so "exercised" is not a
property of the runbook alone. Environments differ, and applying a runbook to an environment is a
state of that environment, not of the runbook.

## Terminology survey

From general DevOps, SRE, configuration-management and ITIL practice, written from memory, not from
a source review. Please treat the names as candidates.

| Area | Terms in common use | Relevance here |
| --- | --- | --- |
| Automation maturity | manual, semi-automated (guided), automated; the "runbook automation ladder"; toil reduction (SRE) | our modes: manual, auto-confirm, later auto-pilot with execution |
| Rehearsal | game day, fire drill, disaster-recovery drill, dry run, rehearsal in staging, `--check` mode (Ansible), `plan` before `apply` (Terraform) | how a runbook earns trust without touching production |
| Readiness | production readiness review (PRR), operational readiness, definition of done | a maturity gate before a runbook is used on production |
| Repeatability | idempotent, convergent (declare desired state), re-runnable, resumable, reversible; drift; "converged" | idempotency is a *declared characteristic*, not a state |
| Change control (ITIL) | standard change (pre-approved because proven repeatable), normal change, change model | maps to "proven" runbooks that need no fresh review |
| Environments | environment / stage (dev, test, staging, production), target, inventory and group variables (Ansible), workspace (Terraform), context (kubectl), installation, instance | the thing runbooks are applied to |
| Decay | runbook rot, stale, drift between docs and system | maturity must expire when the runbook or the system changes |

## Proposal: three separate axes

### 1. Characteristics: declared by the author, static

- **Idempotency**: `idempotent | non-idempotent | conditional` (with a note). Declared per step; the
  runbook's value is derived (all idempotent, or any non-idempotent, or unknown if a step is undeclared).
  **Implemented 2026-09-29** in the v0.2 schema and validator. Values are words, not yes/no, because
  YAML reads unquoted `yes`/`no` as booleans.
- **Reversibility**: already declared per step (`reversible`), with rollback text.
- **Effect class** (candidate): read-only, changes-state, destructive.
- Consequences to build: retry of a failed non-idempotent step should require the operator to confirm
  that the earlier attempt did not partly apply; re-running a whole idempotent runbook against a
  converged environment is safe and is the way to verify it.

### 2. Maturity: earned, independent of any environment, bound to a pack revision

Derived from run history, not typed in. A run records a digest of the pack revision it used.

| Level | Meaning (candidate names) |
| --- | --- |
| `draft` | valid against the spec, never run |
| `reviewed` | a second person has read it (recorded) |
| `rehearsed` | completed cleanly (no deviation) in a **rehearsal** environment |
| `proven` | completed cleanly in at least two distinct environments by at least two operators |
| `stale` | the pack changed since it was proven: maturity is bound to a revision and expires when it changes |

This replaces `exercise_status`. Deviations (overrides, waivers, push-through) never count towards
maturity, which matches the current rule for exercising.

### 3. Projects: environments the runbooks are applied to

A **project** groups runbooks that share one environment. It holds what they have in common and provides
it as datapoint defaults (like Ansible group variables), so a value such as the target host or tenant is
entered once.

```yaml
project: railiance01-identity
environment: production          # rehearsal | staging | production (class, not free text)
datapoints: {tenant: coulomb, target_host: railiance01}     # non-secret; never secrets
runbooks:
  - {pack: nk.ssh-certificates}
  - {pack: nk.openbao-operating-path, after: [nk.ssh-certificates]}
```

Project state is derived from the latest run of each member at the current pack revision:

| State (candidate) | Meaning |
| --- | --- |
| `planned` | no member applied yet |
| `applying` | a run is in progress, or some members are applied and others not (the "running" state) |
| `applied` | every member's latest run succeeded cleanly at the current revision |
| `stale` | a member's pack changed, or a run failed or was abandoned, since it was applied |
| `drifted` (later) | verification against the live system no longer matches |

A **rehearsal** project is where maturity evidence comes from. "Exercised" then has a precise meaning:
*applied successfully in a rehearsal project*. A production project is `applied`, not "exercised".

## Consequences for what exists

- Runs record `project`, the environment class and the pack digest. Scores become per project as well as per
  runbook. Run records stay non-secret and local (`.local/`), so shared history remains an open question.
- `exercise_status` in `pack.yaml` is superseded. Until this is decided it stays as the interim marker.
- Project files are declarative and non-secret. Candidate location: the repository that owns the system
  (for example `net-kingdom/projects/<name>/project.yaml`), next to `runbooks/`.

## Open questions

1. **Names.** *Project* is Bernd's word. Alternatives: *target*, *engagement*, *installation*, *rollout*. Is
   *environment class* (`rehearsal|staging|production`) the right way to separate rehearsal from real?
2. **Maturity thresholds.** Is "two environments, two operators" the right bar for `proven`? Should `reviewed`
   be part of the ladder?
3. **Where projects live**, and whether one runbook may belong to several projects (yes, one project per
   environment, so a runbook applies to many).
4. **Retry policy** for non-idempotent steps, as described above.
5. **Shared run history**: committing a redacted run summary versus keeping records local.