runbook-tutorials/docs/maturity-and-projects.md

Runbook maturity, projects and idempotency

Status: proposal for decision (RBT-WP-0008-T01). Raised by Bernd on 2026-09-29.

The problem

exercise_status: unexercised | exercised <date> by <operator> mixes three different things:

1. What the runbook is like (does it change state, can it be repeated, can it be undone).
2. How much the runbook can be trusted (has it been rehearsed, and where).
3. What state an environment is in (has this runbook been applied to this system).

A run that succeeded on one system says little about another system, so "exercised" is not a property of the runbook alone. Environments differ, and applying a runbook to an environment is a state of that environment, not of the runbook.

Terminology survey

From general DevOps, SRE, configuration-management and ITIL practice, written from memory, not from a source review. Please treat the names as candidates.

Area	Terms in common use	Relevance here
Automation maturity	manual, semi-automated (guided), automated; the "runbook automation ladder"; toil reduction (SRE)	our modes: manual, auto-confirm, later auto-pilot with execution
Rehearsal	game day, fire drill, disaster-recovery drill, dry run, rehearsal in staging, --check mode (Ansible), plan before apply (Terraform)	how a runbook earns trust without touching production
Readiness	production readiness review (PRR), operational readiness, definition of done	a maturity gate before a runbook is used on production
Repeatability	idempotent, convergent (declare desired state), re-runnable, resumable, reversible; drift; "converged"	idempotency is a declared characteristic, not a state
Change control (ITIL)	standard change (pre-approved because proven repeatable), normal change, change model	maps to "proven" runbooks that need no fresh review
Environments	environment / stage (dev, test, staging, production), target, inventory and group variables (Ansible), workspace (Terraform), context (kubectl), installation, instance	the thing runbooks are applied to
Decay	runbook rot, stale, drift between docs and system	maturity must expire when the runbook or the system changes

Proposal: three separate axes

1. Characteristics: declared by the author, static

• Idempotency: idempotent | non-idempotent | conditional (with a note). Declared per step; the runbook's value is derived (all idempotent, or any non-idempotent, or unknown if a step is undeclared). Implemented 2026-09-29 in the v0.2 schema and validator. Values are words, not yes/no, because YAML reads unquoted yes/no as booleans.
• Reversibility: already declared per step (reversible), with rollback text.
• Effect class (candidate): read-only, changes-state, destructive.
• Consequences to build: retry of a failed non-idempotent step should require the operator to confirm that the earlier attempt did not partly apply; re-running a whole idempotent runbook against a converged environment is safe and is the way to verify it.

2. Maturity: earned, independent of any environment, bound to a pack revision

Derived from run history, not typed in. A run records a digest of the pack revision it used.

Level	Meaning (candidate names)
draft	valid against the spec, never run
reviewed	a second person has read it (recorded)
rehearsed	completed cleanly (no deviation) in a rehearsal environment
proven	completed cleanly in at least two distinct environments by at least two operators
stale	the pack changed since it was proven: maturity is bound to a revision and expires when it changes

This replaces exercise_status. Deviations (overrides, waivers, push-through) never count towards maturity, which matches the current rule for exercising.

3. Projects: environments the runbooks are applied to

A project groups runbooks that share one environment. It holds what they have in common and provides it as datapoint defaults (like Ansible group variables), so a value such as the target host or tenant is entered once.

project: railiance01-identity
environment: production          # rehearsal | staging | production (class, not free text)
datapoints: {tenant: coulomb, target_host: railiance01}     # non-secret; never secrets
runbooks:
  - {pack: nk.ssh-certificates}
  - {pack: nk.openbao-operating-path, after: [nk.ssh-certificates]}

Project state is derived from the latest run of each member at the current pack revision:

State (candidate)	Meaning
planned	no member applied yet
applying	a run is in progress, or some members are applied and others not (the "running" state)
applied	every member's latest run succeeded cleanly at the current revision
stale	a member's pack changed, or a run failed or was abandoned, since it was applied
drifted (later)	verification against the live system no longer matches

A rehearsal project is where maturity evidence comes from. "Exercised" then has a precise meaning: applied successfully in a rehearsal project. A production project is applied, not "exercised".

Consequences for what exists

• Runs record project, the environment class and the pack digest. Scores become per project as well as per runbook. Run records stay non-secret and local (.local/), so shared history remains an open question.
• exercise_status in pack.yaml is superseded. Until this is decided it stays as the interim marker.
• Project files are declarative and non-secret. Candidate location: the repository that owns the system (for example net-kingdom/projects/<name>/project.yaml), next to runbooks/.

Open questions

1. Names. Project is Bernd's word. Alternatives: target, engagement, installation, rollout. Is environment class (rehearsal|staging|production) the right way to separate rehearsal from real?
2. Maturity thresholds. Is "two environments, two operators" the right bar for proven? Should reviewed be part of the ladder?
3. Where projects live, and whether one runbook may belong to several projects (yes, one project per environment, so a runbook applies to many).
4. Retry policy for non-idempotent steps, as described above.
5. Shared run history: committing a redacted run summary versus keeping records local.