runbook-tutorials/docs/maturity-and-projects.md

# Runbook maturity, projects and idempotency

Status: **decided 2026-09-29** (Bernd: "go with your suggestions and keep project"), with the attribute-set and template
extension below. Tracked as RBT-WP-0008.

## The problem

`exercise_status: unexercised | exercised <date> by <operator>` mixes three different things:

1. **What the runbook is like** (does it change state, can it be repeated, can it be undone).
2. **How much the runbook can be trusted** (has it been rehearsed, and where).
3. **What state an environment is in** (has this runbook been applied to *this* system).

A run that succeeded on one system says little about another system, so "exercised" is not a
property of the runbook alone. Environments differ, and applying a runbook to an environment is a
state of that environment, not of the runbook.

## Terminology survey

From general DevOps, SRE, configuration-management and ITIL practice, written from memory, not from
a source review. Please treat the names as candidates.

| Area | Terms in common use | Relevance here |
| --- | --- | --- |
| Automation maturity | manual, semi-automated (guided), automated; the "runbook automation ladder"; toil reduction (SRE) | our modes: manual, auto-confirm, later auto-pilot with execution |
| Rehearsal | game day, fire drill, disaster-recovery drill, dry run, rehearsal in staging, `--check` mode (Ansible), `plan` before `apply` (Terraform) | how a runbook earns trust without touching production |
| Readiness | production readiness review (PRR), operational readiness, definition of done | a maturity gate before a runbook is used on production |
| Repeatability | idempotent, convergent (declare desired state), re-runnable, resumable, reversible; drift; "converged" | idempotency is a *declared characteristic*, not a state |
| Change control (ITIL) | standard change (pre-approved because proven repeatable), normal change, change model | maps to "proven" runbooks that need no fresh review |
| Environments | environment / stage (dev, test, staging, production), target, inventory and group variables (Ansible), workspace (Terraform), context (kubectl), installation, instance | the thing runbooks are applied to |
| Decay | runbook rot, stale, drift between docs and system | maturity must expire when the runbook or the system changes |

## Proposal: three separate axes

### 1. Characteristics: declared by the author, static

- **Idempotency**: `idempotent | non-idempotent | conditional` (with a note). Declared per step; the
  runbook's value is derived (all idempotent, or any non-idempotent, or unknown if a step is undeclared).
  **Implemented 2026-09-29** in the v0.2 schema and validator. Values are words, not yes/no, because
  YAML reads unquoted `yes`/`no` as booleans.
- **Reversibility**: already declared per step (`reversible`), with rollback text.
- **Effect class** (candidate): read-only, changes-state, destructive.
- Consequences to build: retry of a failed non-idempotent step should require the operator to confirm
  that the earlier attempt did not partly apply; re-running a whole idempotent runbook against a
  converged environment is safe and is the way to verify it.

### 2. Maturity: earned, independent of any environment, bound to a pack revision

Derived from run history, not typed in. A run records a digest of the pack revision it used.

| Level | Meaning (candidate names) |
| --- | --- |
| `draft` | valid against the spec, never run |
| `reviewed` | a second person has read it (recorded) |
| `rehearsed` | completed cleanly (no deviation) in a **rehearsal** environment |
| `proven` | completed cleanly in at least two distinct environments by at least two operators |
| `stale` | the pack changed since it was proven: maturity is bound to a revision and expires when it changes |

This replaces `exercise_status`. Deviations (overrides, waivers, push-through) never count towards
maturity, which matches the current rule for exercising.

### 3. Projects: environments the runbooks are applied to

A **project** groups runbooks that share one environment. It holds what they have in common and provides
it as datapoint defaults (like Ansible group variables), so a value such as the target host or tenant is
entered once.

```yaml
project: railiance01-identity
environment: production          # rehearsal | staging | production (class, not free text)
datapoints: {tenant: coulomb, target_host: railiance01}     # non-secret; never secrets
runbooks:
  - {pack: nk.ssh-certificates}
  - {pack: nk.openbao-operating-path, after: [nk.ssh-certificates]}
```

Project state is derived from the latest run of each member at the current pack revision:

| State (candidate) | Meaning |
| --- | --- |
| `planned` | no member applied yet |
| `applying` | a run is in progress, or some members are applied and others not (the "running" state) |
| `applied` | every member's latest run succeeded cleanly at the current revision |
| `stale` | a member's pack changed, or a run failed or was abandoned, since it was applied |
| `drifted` (later) | verification against the live system no longer matches |

A **rehearsal** project is where maturity evidence comes from. "Exercised" then has a precise meaning:
*applied successfully in a rehearsal project*. A production project is `applied`, not "exercised".

## Consequences for what exists

- Runs record `project`, the environment class and the pack digest. Scores become per project as well as per
  runbook. Run records stay non-secret and local (`.local/`), so shared history remains an open question.
- `exercise_status` in `pack.yaml` is retired (RBT-WP-0009): the field is deprecated and ignored, and maturity is the only notion.
- Project files are declarative and non-secret. Candidate location: the repository that owns the system
  (for example `net-kingdom/projects/<name>/project.yaml`), next to `runbooks/`.

## Decisions (2026-09-29, Bernd)

1. **Name: project.** Environment class `rehearsal | staging | production`.
2. **Maturity:** `draft`, `reviewed`, `rehearsed`, `proven`, `stale`; `proven` = reviewed, rehearsed, clean in at least two
   environments by at least two operators. Evidence flags are independent; the level is the highest one achieved.
3. **Project files live in the repository that owns the system**, next to `runbooks/`.
4. **Retry of a failed non-idempotent step requires the operator to confirm the earlier attempt did not partly apply.**

## Extension: attribute sets, targets and project templates

A project may have a **goal**, several **targets** (installations, tenants, hosts) and shared facts. All of these are
**attribute sets**: named collections of non-secret attributes that runbooks and other attribute sets can reuse. This is what
makes runbooks and projects composable, and it is the basis for reusable **project templates**.

### Attribute sets

```yaml
spec: attribute-set/v0.1
id: railiance01-host
title: The railiance01 host
extends: [base-cluster]                # composition: later sets override earlier ones
attributes:
  target_host: railiance01             # a value
  tenant: {requirement: required}      # needed, no value yet
  motto: {requirement: optional}
```

Every attribute has:

- a **requirement**, declared: `required` (needed) or `optional`;
- a **state**, derived, using the same words as run datapoints: `nil` (no value), `set`, `err` (fails a consuming runbook's
  datapoint definition), `ok` (confirmed by a run's check), plus a **modified** flag when a later layer overrides a value that an
  earlier layer provided;
- a **source**: which layer gave the value (set, project, target or member). Provenance is what makes a wrong value traceable.

Values are never secret (`secret` is not allowed; secret markers are refused). Only references to secrets are allowed.

### Managing attribute sets in the UI

The **Attribute sets** section lists every set with where it lives, what it extends and who reuses it (projects, templates, other sets). A set's
page shows what it resolves to, with the source of each value and a *modified* mark where a later layer replaced one. Sets stored in a repository
can be created and edited there; your own `user-attributes` set is edited on the settings page. Deleting sets is not offered yet.

### The active user: `user-attributes`

One attribute set lives in the app rather than in any project: `user-attributes`, holding the active user's `operator` (one word) and an
optional `display_name`. It comes from the `RTUT_USER` environment variable, else from the user's saved settings (`/user` page,
`~/.config/runbook-tutorials/user-attributes.yaml`), else from the operating-system login. Forms that used to ask "your name" (starting a
run, reviewing a runbook, starting a project run) show who you are running as and link to that page. Any project can reuse the set
(`attribute_sets: [user-attributes]`); a repository may define its own set of that id, which then takes precedence. If the UI later runs
behind a login, the login fills the same attribute.

### Layers, in order (later wins)

attribute sets listed by the project (each after its own `extends`) then the project's own `attributes`, then the target's
sets and attributes, then the member's own `attributes`. The resolved attributes become the defaults of the member runbook's
datapoints, so a value shared by several runbooks is entered once.

### Projects and targets

```yaml
spec: project/v0.1
id: railiance01-identity
title: Identity on railiance01
goal: Lightweight identity is reachable and its custody is documented.
environment: production                # rehearsal | staging | production
template: identity-stack               # set when created from a template
attribute_sets: [railiance01-host]
attributes: {tenant: coulomb}
targets:
  coulomb: {attribute_sets: [], attributes: {tenant: coulomb}}
runbooks:
  - {pack: nk.ssh-certificates, target: coulomb}
  - {pack: nk.openbao-operating-path, after: [nk.ssh-certificates]}
```

Project state is derived from each member's latest run at the current pack revision (see the table above, extended by
member status `not-applied | applying | applied | stale | failed`).

### Templates

A **project template** (`spec: project-template/v0.1`) is a project skeleton: members, attribute sets and attributes that
still need values. Instantiating a template creates a project file with **its own name, description and goal**, records which template it
came from, and shows which required attributes are still needed. Nothing of the template's own title, goal or description is copied into the
project: a project is not a copy of its template, it only remembers where it came from. Templates make a set of runbooks reusable as a product.

### Project runs

A **project run** runs every member runbook of a project once, in `after` order, in one mode. The record lives next to the project
(`.local/project-runs/`); its state is derived from the member runs, which carry it in their context.

- **Snapshot.** When it starts, the project's resolved attributes are snapshotted per member. Every member run uses the snapshot, so editing the
  project meanwhile cannot make member runs inconsistent; the page says when the project has changed since. A new project run picks up the new values.
- **Manual.** You start each member yourself, one at a time. Starting a member before what it waits for is an override: a reason and an
  acknowledged risk, recorded on the project run.
- **Auto-confirm.** The engine starts the next ready member, ticks its run, closes it when it finishes cleanly and moves on (`Run until it waits`).
  It stops wherever a person is needed: a v0.1 member (always manual), a manual-only or irreversible step, a check that does not pass. A member
  that finished only by pushing through problems counts as done for its dependents, but the project run can then no longer succeed.
- **Member status**: `waiting`, `ready`, `running`, `succeeded`, `pushed`, `failed`, `abandoned`. A failed or abandoned member blocks its dependents.
- **Closing.** `succeeded` needs every member run to have succeeded; `failed` needs a failed, abandoned or pushed member (or an open member run);
  `abandoned` any time. Closing abandons an open member run, locks the record and seals it like a run. Only one project run per project is open at a time.

### Modes

Any project run can use any mode: **manual** (operator confirms everything), **auto-confirm** (the engine confirms when
preconditions hold), and later **auto-pilot** (the engine also executes steps; RBT-WP-0006, opt-in and audited). Because the
project supplies the attributes, an auto-pilot run needs no operator input except what the template marks as `required` and unset.

## Still open

1. Sharing run history between machines (committing a redacted run summary versus keeping records local).
2. Whether attribute sets may declare validation of their own (types), independent of the runbooks that consume them.
3. The exact shape of member ordering (`after`) once runbooks can also depend on runbooks (`requires_packs`, RBT-WP-0006).