runbook-tutorials/docs/maturity-and-projects.md

Runbook maturity, projects and idempotency

Status: decided 2026-09-29 (Bernd: "go with your suggestions and keep project"), with the attribute-set and template extension below. Tracked as RBT-WP-0008.

The problem

exercise_status: unexercised | exercised <date> by <operator> mixes three different things:

1. What the runbook is like (does it change state, can it be repeated, can it be undone).
2. How much the runbook can be trusted (has it been rehearsed, and where).
3. What state an environment is in (has this runbook been applied to this system).

A run that succeeded on one system says little about another system, so "exercised" is not a property of the runbook alone. Environments differ, and applying a runbook to an environment is a state of that environment, not of the runbook.

Terminology survey

From general DevOps, SRE, configuration-management and ITIL practice, written from memory, not from a source review. Please treat the names as candidates.

Area	Terms in common use	Relevance here
Automation maturity	manual, semi-automated (guided), automated; the "runbook automation ladder"; toil reduction (SRE)	our modes: manual, auto-confirm, later auto-pilot with execution
Rehearsal	game day, fire drill, disaster-recovery drill, dry run, rehearsal in staging, --check mode (Ansible), plan before apply (Terraform)	how a runbook earns trust without touching production
Readiness	production readiness review (PRR), operational readiness, definition of done	a maturity gate before a runbook is used on production
Repeatability	idempotent, convergent (declare desired state), re-runnable, resumable, reversible; drift; "converged"	idempotency is a declared characteristic, not a state
Change control (ITIL)	standard change (pre-approved because proven repeatable), normal change, change model	maps to "proven" runbooks that need no fresh review
Environments	environment / stage (dev, test, staging, production), target, inventory and group variables (Ansible), workspace (Terraform), context (kubectl), installation, instance	the thing runbooks are applied to
Decay	runbook rot, stale, drift between docs and system	maturity must expire when the runbook or the system changes

Proposal: three separate axes

1. Characteristics: declared by the author, static

• Idempotency: idempotent | non-idempotent | conditional (with a note). Declared per step; the runbook's value is derived (all idempotent, or any non-idempotent, or unknown if a step is undeclared). Implemented 2026-09-29 in the v0.2 schema and validator. Values are words, not yes/no, because YAML reads unquoted yes/no as booleans.
• Reversibility: already declared per step (reversible), with rollback text.
• Effect class (candidate): read-only, changes-state, destructive.
• Consequences to build: retry of a failed non-idempotent step should require the operator to confirm that the earlier attempt did not partly apply; re-running a whole idempotent runbook against a converged environment is safe and is the way to verify it.

2. Maturity: earned, independent of any environment, bound to a pack revision

Derived from run history, not typed in. A run records a digest of the pack revision it used.

Level	Meaning (candidate names)
draft	valid against the spec, never run
reviewed	a second person has read it (recorded)
rehearsed	completed cleanly (no deviation) in a rehearsal environment
proven	completed cleanly in at least two distinct environments by at least two operators
stale	the pack changed since it was proven: maturity is bound to a revision and expires when it changes

This replaces exercise_status. Deviations (overrides, waivers, push-through) never count towards maturity, which matches the current rule for exercising.

3. Projects: environments the runbooks are applied to

A project groups runbooks that share one environment. It holds what they have in common and provides it as datapoint defaults (like Ansible group variables), so a value such as the target host or tenant is entered once.

project: railiance01-identity
environment: production          # rehearsal | staging | production (class, not free text)
datapoints: {tenant: coulomb, target_host: railiance01}     # non-secret; never secrets
runbooks:
  - {pack: nk.ssh-certificates}
  - {pack: nk.openbao-operating-path, after: [nk.ssh-certificates]}

Project state is derived from the latest run of each member at the current pack revision:

State (candidate)	Meaning
planned	no member applied yet
applying	a run is in progress, or some members are applied and others not (the "running" state)
applied	every member's latest run succeeded cleanly at the current revision
stale	a member's pack changed, or a run failed or was abandoned, since it was applied
drifted (later)	verification against the live system no longer matches

A rehearsal project is where maturity evidence comes from. "Exercised" then has a precise meaning: applied successfully in a rehearsal project. A production project is applied, not "exercised".

Consequences for what exists

• Runs record project, the environment class and the pack digest. Scores become per project as well as per runbook. Run records stay non-secret and local (.local/), so shared history remains an open question.
• exercise_status in pack.yaml is retired (RBT-WP-0009): the field is deprecated and ignored, and maturity is the only notion.
• Project files are declarative and non-secret. Candidate location: the repository that owns the system (for example net-kingdom/projects/<name>/project.yaml), next to runbooks/.

Decisions (2026-09-29, Bernd)

1. Name: project. Environment class rehearsal | staging | production.
2. Maturity: draft, reviewed, rehearsed, proven, stale; proven = reviewed, rehearsed, clean in at least two environments by at least two operators. Evidence flags are independent; the level is the highest one achieved.
3. Project files live in the repository that owns the system, next to runbooks/.
4. Retry of a failed non-idempotent step requires the operator to confirm the earlier attempt did not partly apply.

Extension: attribute sets, targets and project templates

A project may have a goal, several targets (installations, tenants, hosts) and shared facts. All of these are attribute sets: named collections of non-secret attributes that runbooks and other attribute sets can reuse. This is what makes runbooks and projects composable, and it is the basis for reusable project templates.

Attribute sets

spec: attribute-set/v0.1
id: railiance01-host
title: The railiance01 host
extends: [base-cluster]                # composition: later sets override earlier ones
attributes:
  target_host: railiance01             # a value
  tenant: {requirement: required}      # needed, no value yet
  motto: {requirement: optional}

Every attribute has:

• a requirement, declared: required (needed) or optional;
• a state, derived, using the same words as run datapoints: nil (no value), set, err (fails a consuming runbook's datapoint definition), ok (confirmed by a run's check), plus a modified flag when a later layer overrides a value that an earlier layer provided;
• a source: which layer gave the value (set, project, target or member). Provenance is what makes a wrong value traceable.

Values are never secret (secret is not allowed; secret markers are refused). Only references to secrets are allowed.

Managing attribute sets in the UI

The Attribute sets section lists every set with where it lives, what it extends and who reuses it (projects, templates, other sets). A set's page shows what it resolves to, with the source of each value and a modified mark where a later layer replaced one. Sets stored in a repository can be created and edited there; your own user-attributes set is edited on the settings page. Deleting sets is not offered yet.

The active user: user-attributes

One attribute set lives in the app rather than in any project: user-attributes, holding the active user's operator (one word) and an optional display_name. It comes from the RTUT_USER environment variable, else from the user's saved settings (/user page, ~/.config/runbook-tutorials/user-attributes.yaml), else from the operating-system login. Forms that used to ask "your name" (starting a run, reviewing a runbook, starting a project run) show who you are running as and link to that page. Any project can reuse the set (attribute_sets: [user-attributes]); a repository may define its own set of that id, which then takes precedence. If the UI later runs behind a login, the login fills the same attribute.

Layers, in order (later wins)

attribute sets listed by the project (each after its own extends) then the project's own attributes, then the target's sets and attributes, then the member's own attributes. The resolved attributes become the defaults of the member runbook's datapoints, so a value shared by several runbooks is entered once.

Projects and targets

spec: project/v0.1
id: railiance01-identity
title: Identity on railiance01
goal: Lightweight identity is reachable and its custody is documented.
environment: production                # rehearsal | staging | production
template: identity-stack               # set when created from a template
attribute_sets: [railiance01-host]
attributes: {tenant: coulomb}
targets:
  coulomb: {attribute_sets: [], attributes: {tenant: coulomb}}
runbooks:
  - {pack: nk.ssh-certificates, target: coulomb}
  - {pack: nk.openbao-operating-path, after: [nk.ssh-certificates]}

Project state is derived from each member's latest run at the current pack revision (see the table above, extended by member status not-applied | applying | applied | stale | failed).

Templates

A project template (spec: project-template/v0.1) is a project skeleton: members, attribute sets and attributes that still need values. Instantiating a template creates a project file with its own name, description and goal, records which template it came from, and shows which required attributes are still needed. Nothing of the template's own title, goal or description is copied into the project: a project is not a copy of its template, it only remembers where it came from. Templates make a set of runbooks reusable as a product.

Project runs

A project run runs every member runbook of a project once, in after order, in one mode. The record lives next to the project (.local/project-runs/); its state is derived from the member runs, which carry it in their context.

• Snapshot. When it starts, the project's resolved attributes are snapshotted per member. Every member run uses the snapshot, so editing the project meanwhile cannot make member runs inconsistent; the page says when the project has changed since. A new project run picks up the new values.
• Manual. You start each member yourself, one at a time. Starting a member before what it waits for is an override: a reason and an acknowledged risk, recorded on the project run.
• Auto-confirm. The engine starts the next ready member, ticks its run, closes it when it finishes cleanly and moves on (Run until it waits). It stops wherever a person is needed: a v0.1 member (always manual), a manual-only or irreversible step, a check that does not pass. A member that finished only by pushing through problems counts as done for its dependents, but the project run can then no longer succeed.
• Member status: waiting, ready, running, succeeded, pushed, failed, abandoned. A failed or abandoned member blocks its dependents.
• Closing. succeeded needs every member run to have succeeded; failed needs a failed, abandoned or pushed member (or an open member run); abandoned any time. Closing abandons an open member run, locks the record and seals it like a run. Only one project run per project is open at a time.

Modes

Any project run can use any mode: manual (operator confirms everything), auto-confirm (the engine confirms when preconditions hold), and later auto-pilot (the engine also executes steps; RBT-WP-0006, opt-in and audited). Because the project supplies the attributes, an auto-pilot run needs no operator input except what the template marks as required and unset.

Still open

1. Sharing run history between machines (committing a redacted run summary versus keeping records local).
2. Whether attribute sets may declare validation of their own (types), independent of the runbooks that consume them.
3. The exact shape of member ordering (after) once runbooks can also depend on runbooks (requires_packs, RBT-WP-0006).