runbook-tutorials/docs/maturity-and-projects.md

# Runbook maturity, projects and idempotency

Status: **decided 2026-09-29** (Bernd: "go with your suggestions and keep project"), with the attribute-set and template
extension below. Tracked as RBT-WP-0008.

## The problem

`exercise_status: unexercised | exercised <date> by <operator>` mixes three different things:

1. **What the runbook is like** (does it change state, can it be repeated, can it be undone).
2. **How much the runbook can be trusted** (has it been rehearsed, and where).
3. **What state an environment is in** (has this runbook been applied to *this* system).

A run that succeeded on one system says little about another system, so "exercised" is not a
property of the runbook alone. Environments differ, and applying a runbook to an environment is a
state of that environment, not of the runbook.

## Terminology survey

From general DevOps, SRE, configuration-management and ITIL practice, written from memory, not from
a source review. Please treat the names as candidates.

| Area | Terms in common use | Relevance here |
| --- | --- | --- |
| Automation maturity | manual, semi-automated (guided), automated; the "runbook automation ladder"; toil reduction (SRE) | our modes: manual, auto-confirm, later auto-pilot with execution |
| Rehearsal | game day, fire drill, disaster-recovery drill, dry run, rehearsal in staging, `--check` mode (Ansible), `plan` before `apply` (Terraform) | how a runbook earns trust without touching production |
| Readiness | production readiness review (PRR), operational readiness, definition of done | a maturity gate before a runbook is used on production |
| Repeatability | idempotent, convergent (declare desired state), re-runnable, resumable, reversible; drift; "converged" | idempotency is a *declared characteristic*, not a state |
| Change control (ITIL) | standard change (pre-approved because proven repeatable), normal change, change model | maps to "proven" runbooks that need no fresh review |
| Environments | environment / stage (dev, test, staging, production), target, inventory and group variables (Ansible), workspace (Terraform), context (kubectl), installation, instance | the thing runbooks are applied to |
| Decay | runbook rot, stale, drift between docs and system | maturity must expire when the runbook or the system changes |

## Proposal: three separate axes

### 1. Characteristics: declared by the author, static

- **Idempotency**: `idempotent | non-idempotent | conditional` (with a note). Declared per step; the
  runbook's value is derived (all idempotent, or any non-idempotent, or unknown if a step is undeclared).
  **Implemented 2026-09-29** in the v0.2 schema and validator. Values are words, not yes/no, because
  YAML reads unquoted `yes`/`no` as booleans.
- **Reversibility**: already declared per step (`reversible`), with rollback text.
- **Effect class** (candidate): read-only, changes-state, destructive.
- Consequences to build: retry of a failed non-idempotent step should require the operator to confirm
  that the earlier attempt did not partly apply; re-running a whole idempotent runbook against a
  converged environment is safe and is the way to verify it.

### 2. Maturity: earned, independent of any environment, bound to a pack revision

Derived from run history, not typed in. A run records a digest of the pack revision it used.

| Level | Meaning (candidate names) |
| --- | --- |
| `draft` | valid against the spec, never run |
| `reviewed` | a second person has read it (recorded) |
| `rehearsed` | completed cleanly (no deviation) in a **rehearsal** environment |
| `proven` | completed cleanly in at least two distinct environments by at least two operators |
| `stale` | the pack changed since it was proven: maturity is bound to a revision and expires when it changes |

This replaces `exercise_status`. Deviations (overrides, waivers, push-through) never count towards
maturity, which matches the current rule for exercising.

### 3. Projects: environments the runbooks are applied to

A **project** groups runbooks that share one environment. It holds what they have in common and provides
it as datapoint defaults (like Ansible group variables), so a value such as the target host or tenant is
entered once.

```yaml
project: railiance01-identity
environment: production          # rehearsal | staging | production (class, not free text)
datapoints: {tenant: coulomb, target_host: railiance01}     # non-secret; never secrets
runbooks:
  - {pack: nk.ssh-certificates}
  - {pack: nk.openbao-operating-path, after: [nk.ssh-certificates]}
```

Project state is derived from the latest run of each member at the current pack revision:

| State (candidate) | Meaning |
| --- | --- |
| `planned` | no member applied yet |
| `applying` | a run is in progress, or some members are applied and others not (the "running" state) |
| `applied` | every member's latest run succeeded cleanly at the current revision |
| `stale` | a member's pack changed, or a run failed or was abandoned, since it was applied |
| `drifted` (later) | verification against the live system no longer matches |

A **rehearsal** project is where maturity evidence comes from. "Exercised" then has a precise meaning:
*applied successfully in a rehearsal project*. A production project is `applied`, not "exercised".

## Consequences for what exists

- Runs record `project`, the environment class and the pack digest. Scores become per project as well as per
  runbook. Run records stay non-secret and local (`.local/`), so shared history remains an open question.
- `exercise_status` in `pack.yaml` is superseded. Until this is decided it stays as the interim marker.
- Project files are declarative and non-secret. Candidate location: the repository that owns the system
  (for example `net-kingdom/projects/<name>/project.yaml`), next to `runbooks/`.

## Decisions (2026-09-29, Bernd)

1. **Name: project.** Environment class `rehearsal | staging | production`.
2. **Maturity:** `draft`, `reviewed`, `rehearsed`, `proven`, `stale`; `proven` = reviewed, rehearsed, clean in at least two
   environments by at least two operators. Evidence flags are independent; the level is the highest one achieved.
3. **Project files live in the repository that owns the system**, next to `runbooks/`.
4. **Retry of a failed non-idempotent step requires the operator to confirm the earlier attempt did not partly apply.**

## Extension: attribute sets, targets and project templates

A project may have a **goal**, several **targets** (installations, tenants, hosts) and shared facts. All of these are
**attribute sets**: named collections of non-secret attributes that runbooks and other attribute sets can reuse. This is what
makes runbooks and projects composable, and it is the basis for reusable **project templates**.

### Attribute sets

```yaml
spec: attribute-set/v0.1
id: railiance01-host
title: The railiance01 host
extends: [base-cluster]                # composition: later sets override earlier ones
attributes:
  target_host: railiance01             # a value
  tenant: {requirement: required}      # needed, no value yet
  motto: {requirement: optional}
```

Every attribute has:

- a **requirement**, declared: `required` (needed) or `optional`;
- a **state**, derived, using the same words as run datapoints: `nil` (no value), `set`, `err` (fails a consuming runbook's
  datapoint definition), `ok` (confirmed by a run's check), plus a **modified** flag when a later layer overrides a value that an
  earlier layer provided;
- a **source**: which layer gave the value (set, project, target or member). Provenance is what makes a wrong value traceable.

Values are never secret (`secret` is not allowed; secret markers are refused). Only references to secrets are allowed.

### Layers, in order (later wins)

attribute sets listed by the project (each after its own `extends`) then the project's own `attributes`, then the target's
sets and attributes, then the member's own `attributes`. The resolved attributes become the defaults of the member runbook's
datapoints, so a value shared by several runbooks is entered once.

### Projects and targets

```yaml
spec: project/v0.1
id: railiance01-identity
title: Identity on railiance01
goal: Lightweight identity is reachable and its custody is documented.
environment: production                # rehearsal | staging | production
template: identity-stack               # set when created from a template
attribute_sets: [railiance01-host]
attributes: {tenant: coulomb}
targets:
  coulomb: {attribute_sets: [], attributes: {tenant: coulomb}}
runbooks:
  - {pack: nk.ssh-certificates, target: coulomb}
  - {pack: nk.openbao-operating-path, after: [nk.ssh-certificates]}
```

Project state is derived from each member's latest run at the current pack revision (see the table above, extended by
member status `not-applied | applying | applied | stale | failed`).

### Templates

A **project template** (`spec: project-template/v0.1`) is a project skeleton: goal, members, attribute sets and attributes that
still need values. Instantiating a template creates a project file, records which template it came from, and shows which required
attributes are still needed. Templates make a set of runbooks reusable as a product.

### Modes

Any project run can use any mode: **manual** (operator confirms everything), **auto-confirm** (the engine confirms when
preconditions hold), and later **auto-pilot** (the engine also executes steps; RBT-WP-0006, opt-in and audited). Because the
project supplies the attributes, an auto-pilot run needs no operator input except what the template marks as `required` and unset.

## Still open

1. Sharing run history between machines (committing a redacted run summary versus keeping records local).
2. Whether attribute sets may declare validation of their own (types), independent of the runbooks that consume them.
3. The exact shape of member ordering (`after`) once runbooks can also depend on runbooks (`requires_packs`, RBT-WP-0006).