diff --git a/docs/maturity-and-projects.md b/docs/maturity-and-projects.md new file mode 100644 index 0000000..29561a6 --- /dev/null +++ b/docs/maturity-and-projects.md @@ -0,0 +1,106 @@ +# Runbook maturity, projects and idempotency + +Status: **proposal for decision** (RBT-WP-0008-T01). Raised by Bernd on 2026-09-29. + +## The problem + +`exercise_status: unexercised | exercised by ` mixes three different things: + +1. **What the runbook is like** (does it change state, can it be repeated, can it be undone). +2. **How much the runbook can be trusted** (has it been rehearsed, and where). +3. **What state an environment is in** (has this runbook been applied to *this* system). + +A run that succeeded on one system says little about another system, so "exercised" is not a +property of the runbook alone. Environments differ, and applying a runbook to an environment is a +state of that environment, not of the runbook. + +## Terminology survey + +From general DevOps, SRE, configuration-management and ITIL practice, written from memory, not from +a source review. Please treat the names as candidates. + +| Area | Terms in common use | Relevance here | +| --- | --- | --- | +| Automation maturity | manual, semi-automated (guided), automated; the "runbook automation ladder"; toil reduction (SRE) | our modes: manual, auto-confirm, later auto-pilot with execution | +| Rehearsal | game day, fire drill, disaster-recovery drill, dry run, rehearsal in staging, `--check` mode (Ansible), `plan` before `apply` (Terraform) | how a runbook earns trust without touching production | +| Readiness | production readiness review (PRR), operational readiness, definition of done | a maturity gate before a runbook is used on production | +| Repeatability | idempotent, convergent (declare desired state), re-runnable, resumable, reversible; drift; "converged" | idempotency is a *declared characteristic*, not a state | +| Change control (ITIL) | standard change (pre-approved because proven repeatable), normal change, change model | maps to "proven" runbooks that need no fresh review | +| Environments | environment / stage (dev, test, staging, production), target, inventory and group variables (Ansible), workspace (Terraform), context (kubectl), installation, instance | the thing runbooks are applied to | +| Decay | runbook rot, stale, drift between docs and system | maturity must expire when the runbook or the system changes | + +## Proposal: three separate axes + +### 1. Characteristics: declared by the author, static + +- **Idempotency**: `idempotent | non-idempotent | conditional` (with a note). Declared per step; the + runbook's value is derived (all idempotent, or any non-idempotent, or unknown if a step is undeclared). + **Implemented 2026-09-29** in the v0.2 schema and validator. Values are words, not yes/no, because + YAML reads unquoted `yes`/`no` as booleans. +- **Reversibility**: already declared per step (`reversible`), with rollback text. +- **Effect class** (candidate): read-only, changes-state, destructive. +- Consequences to build: retry of a failed non-idempotent step should require the operator to confirm + that the earlier attempt did not partly apply; re-running a whole idempotent runbook against a + converged environment is safe and is the way to verify it. + +### 2. Maturity: earned, independent of any environment, bound to a pack revision + +Derived from run history, not typed in. A run records a digest of the pack revision it used. + +| Level | Meaning (candidate names) | +| --- | --- | +| `draft` | valid against the spec, never run | +| `reviewed` | a second person has read it (recorded) | +| `rehearsed` | completed cleanly (no deviation) in a **rehearsal** environment | +| `proven` | completed cleanly in at least two distinct environments by at least two operators | +| `stale` | the pack changed since it was proven: maturity is bound to a revision and expires when it changes | + +This replaces `exercise_status`. Deviations (overrides, waivers, push-through) never count towards +maturity, which matches the current rule for exercising. + +### 3. Projects: environments the runbooks are applied to + +A **project** groups runbooks that share one environment. It holds what they have in common and provides +it as datapoint defaults (like Ansible group variables), so a value such as the target host or tenant is +entered once. + +```yaml +project: railiance01-identity +environment: production # rehearsal | staging | production (class, not free text) +datapoints: {tenant: coulomb, target_host: railiance01} # non-secret; never secrets +runbooks: + - {pack: nk.ssh-certificates} + - {pack: nk.openbao-operating-path, after: [nk.ssh-certificates]} +``` + +Project state is derived from the latest run of each member at the current pack revision: + +| State (candidate) | Meaning | +| --- | --- | +| `planned` | no member applied yet | +| `applying` | a run is in progress, or some members are applied and others not (the "running" state) | +| `applied` | every member's latest run succeeded cleanly at the current revision | +| `stale` | a member's pack changed, or a run failed or was abandoned, since it was applied | +| `drifted` (later) | verification against the live system no longer matches | + +A **rehearsal** project is where maturity evidence comes from. "Exercised" then has a precise meaning: +*applied successfully in a rehearsal project*. A production project is `applied`, not "exercised". + +## Consequences for what exists + +- Runs record `project`, the environment class and the pack digest. Scores become per project as well as per + runbook. Run records stay non-secret and local (`.local/`), so shared history remains an open question. +- `exercise_status` in `pack.yaml` is superseded. Until this is decided it stays as the interim marker. +- Project files are declarative and non-secret. Candidate location: the repository that owns the system + (for example `net-kingdom/projects//project.yaml`), next to `runbooks/`. + +## Open questions + +1. **Names.** *Project* is Bernd's word. Alternatives: *target*, *engagement*, *installation*, *rollout*. Is + *environment class* (`rehearsal|staging|production`) the right way to separate rehearsal from real? +2. **Maturity thresholds.** Is "two environments, two operators" the right bar for `proven`? Should `reviewed` + be part of the ladder? +3. **Where projects live**, and whether one runbook may belong to several projects (yes, one project per + environment, so a runbook applies to many). +4. **Retry policy** for non-idempotent steps, as described above. +5. **Shared run history**: committing a redacted run summary versus keeping records local. diff --git a/docs/pack-spec-v0.2.md b/docs/pack-spec-v0.2.md index d4f1081..f9cf8c0 100644 --- a/docs/pack-spec-v0.2.md +++ b/docs/pack-spec-v0.2.md @@ -154,7 +154,7 @@ A **run** has a mode. The mode belongs to the run, not to the pack. | Mode | Who confirms | Checks, gates, preconditions | | --- | --- | --- | | **manual** | the operator | shown with their state; the operator confirms each | -| **autopilot** | the engine, when every precondition is met | evaluated by the engine; confirmation is applied automatically | +| **auto-confirm** | the engine, when every precondition is met | evaluated by the engine; confirmation is applied automatically | ### Manual mode: override @@ -164,10 +164,10 @@ downstream risk (`downstream_risk` text declared on the step or gate, else a generic warning), (2) require an explicit confirmation, and (3) record a **deviation** in the run. -### Autopilot mode: timeout and push-through +### Auto-confirm mode: timeout and push-through - A step or gate may declare `timeout: {seconds: N, on_timeout: stop|push-through}`; - a run may set a default. Without a timeout, autopilot waits. + a run may set a default. Without a timeout, auto-confirm waits. - `stop` (default): on timeout the run stops and reports. - `push-through`: on timeout the engine confirms the item, records a deviation, and moves on, ignoring the problem. This gathers the most information about @@ -176,10 +176,13 @@ generic warning), (2) require an explicit confirmation, and (3) record a with an explicit, recorded pre-authorization naming those steps (and the pack's `confirm_phrase`). Push-through does not bypass that rule. This is a safety default, open to review. -- A step may declare `autopilot: manual-only` to forbid automation entirely. +- A step may declare `auto_confirm: manual-only` to forbid automation entirely. +- Auto-confirm applies *confirmation* only. It does not run a step's own command; execution is a later, separate level (RBT-WP-0006). ### Deviations +An item passed over by push-through gets the distinct state `pushed` (steps, gates and datapoints): not done, not set, not an error. + Every override, waiver and push-through is a **deviation** in the run record: `kind` (`override|waiver|push-through`), `item` (step or gate id), `mode`, `unmet` (what was not satisfied), `risk_acknowledged`, `reason` (required for @@ -201,6 +204,10 @@ declared in advance (`waivable` on a gate); an **override** is not. ## Later: dependencies between runbooks +**Idempotency** is a declared characteristic of a step: `idempotent: idempotent | non-idempotent | conditional` +(`conditional` needs an `idempotent_note`). Undeclared means unknown. The runbook's overall value is derived. +See `docs/maturity-and-projects.md`. + Not in v0.2. When a procedure needs another runbook (for example "OpenBao operating path requires the SSH access pack"), packs will declare `requires_packs` with a required result (exercised, or a named gate passed). Tracked in RBT-WP-0006. diff --git a/docs/run-spec.md b/docs/run-spec.md index d8bffab..39c52cf 100644 --- a/docs/run-spec.md +++ b/docs/run-spec.md @@ -27,17 +27,17 @@ CLI: `rtut run new|list|status|cmd|start|verify|done|fail|retry|skip|finish `; shown in the UI). Run records live in `.local/` of the machine that ran them, so the score is per machine until run histories are shared. @@ -67,12 +67,12 @@ Implemented in `rtut/run2.py` (graph in `rtut/flow.py`). It adds to the v0.1 rec | Field | Meaning | | --- | --- | -| `mode` | `manual` or `autopilot` | -| `datapoints.` | `state` nil, set, err or ok; `value`, `how` (initial, manual, produced, check, manual confirmation), `error` | -| `steps.` | as v0.1, plus `alternative` (chosen `one_of` branch), `authorized_by`, `pushed_through` | -| `gates.` | `state` open, passed, failed or waived; per-check `results`; `waiver` | +| `mode` | `manual` or `auto-confirm` | +| `datapoints.` | `state` nil, set, err, ok or pushed; `value`, `how` (initial, manual, produced, check, manual confirmation), `error` | +| `steps.` | as v0.1 (`pushed` is an additional state), plus `alternative` (chosen `one_of` branch), `authorized_by` | +| `gates.` | `state` open, passed, failed, waived or pushed; per-check `results`; `waiver` or `pushed` | | `deviations[]` | every override, waiver and push-through (see `docs/pack-spec-v0.2.md`) | -| `autopilot` | `preauthorized` irreversible steps, `authorized_by`, and `stopped` if a timeout stopped the run | +| `auto-confirm` | `preauthorized` irreversible steps, `authorized_by`, and `stopped` if a timeout stopped the run | Rules the engine enforces: @@ -88,13 +88,20 @@ Rules the engine enforces: - **Override** (manual mode only): starting an item with unmet preconditions needs a reason and an acknowledged downstream risk; it is recorded as a deviation. - **Waiver**: only for gates that declare `waivable`, with a reason; recorded as a deviation. -- **Confirmation**: irreversible steps need the exact confirmation phrase at start. In autopilot they run only - if the run pre-authorized them, and never if the step is `autopilot: manual-only`. -- **Autopilot** (`autopilot_tick`): the engine applies confirmation when preconditions are met: it starts the +- **Confirmation**: irreversible steps need the exact confirmation phrase at start. In auto-confirm they run only + if the run pre-authorized them, and never if the step is `auto_confirm: manual-only`. +- **Auto-confirm** (`auto-confirm_tick`): the engine applies confirmation when preconditions are met: it starts the recommended item, runs its checks (manual checks are auto-confirmed and marked so) and completes it. It never runs a step's own command; the work is done by the operator or, later, by opt-in execution. A step with `evidence` waits for the operator. If checks keep failing until the item's `timeout` (or the pack default), the run stops, or with `on_timeout: push-through` the item is marked done with - `pushed_through` and a deviation is recorded. + state `pushed` and a deviation is recorded. - **Deviations and exercise**: a run with any deviation or pushed-through step can be finalized but never exercises a pack. `finalize(succeeded)` also needs every gate passed or waived. +- **Pushed is its own state.** A step, gate or datapoint passed over by push-through is `pushed`: not `done`, + not `set`, not `err`. Dependents may proceed (that is the point of pushing through), but a pushed datapoint + has no value, so a command that needs it fails its check instead of running with nothing. A run with pushed + items can be finished as `failed` (never `succeeded`) and never exercises a pack. +- **Stalled**: an unfinished run is counted as `stalled` in the score, and as `abandoned` after a week. +- **Modes** are `manual` and `auto-confirm`. Auto-confirm applies confirmation only; running the step's own + command is a separate, later level ("auto-pilot with execution", RBT-WP-0006). diff --git a/rtut/__main__.py b/rtut/__main__.py index 5ee1a5f..910f21d 100644 --- a/rtut/__main__.py +++ b/rtut/__main__.py @@ -114,7 +114,7 @@ def main(argv: list[str] | None = None) -> int: for p in _packs(a.root): if p.ok and p.data.get("engine") == "native" and p.spec == "runbook-pack/v0.1": c = R.scorecard(p) - print(f"{p.id}\ttried {c['tried']}\tsucceeded {c['succeeded']}\tfailed {c['failed']}\tabandoned {c['abandoned']}\topen {c['open']}\t[{p.data['exercise_status']}]") + print(f"{p.id}\ttried {c['tried']}\tsucceeded {c['succeeded']}\tfailed {c['failed']}\tabandoned {c['abandoned']}\tstalled {c['stalled']}\t[{p.data['exercise_status']}]") return 0 if a.cmd in ("list", "validate"): packs = _packs(a.root) diff --git a/rtut/flow.py b/rtut/flow.py index 4f1003c..844d6ef 100644 --- a/rtut/flow.py +++ b/rtut/flow.py @@ -67,3 +67,17 @@ class Flow: for a in self.ancestors(id_): refs += self.stages[a].get("requires", []) return [r for r in (self.resolve(x) for x in refs) if r] + + +def pack_idempotency(data: dict) -> str: + """Derived characteristic of the whole runbook: idempotent, non-idempotent, conditional or unknown.""" + fl = Flow(data) + vals = [] + for s in fl.steps.values(): + parts = s.get("one_of") or [s] + vals += [p.get("idempotent", s.get("idempotent", "unknown")) for p in parts] + if not vals or "unknown" in vals: + return "unknown" + if "non-idempotent" in vals: + return "non-idempotent" + return "conditional" if "conditional" in vals else "idempotent" diff --git a/rtut/run.py b/rtut/run.py index 353447d..a700a64 100644 --- a/rtut/run.py +++ b/rtut/run.py @@ -244,8 +244,6 @@ def exercise_update(run: Run, today: date | None = None) -> tuple[str, str]: raise RunError("every step must be done (not skipped) to exercise the pack") if run.data.get("deviations"): raise RunError("a run with deviations (overrides, waivers, push-through) cannot exercise the pack") - if any(v.get("pushed_through") for v in run.data["steps"].values()): - raise RunError("a run with pushed-through steps cannot exercise the pack") today = today or date.today() old = run.pack.path.read_text() new = STATUS_LINE.sub(f"exercise_status: exercised {today.isoformat()} by {run.data['operator']}", @@ -346,8 +344,9 @@ def finalize(run: Run, outcome: str, note: str = "") -> None: raise RunError("a run succeeds only when every step is done (skipped steps do not count)") if outcome == "succeeded" and any(g["state"] not in ("passed", "waived") for g in run.data.get("gates", {}).values()): raise RunError("a run succeeds only when every gate has passed or been waived") - if outcome == "failed" and "failed" not in states: - raise RunError("a run can be finished as failed only when a step has failed; otherwise abandon it") + gstates = [g["state"] for g in run.data.get("gates", {}).values()] + if outcome == "failed" and not ({"failed", "pushed"} & set(states) or {"failed", "pushed"} & set(gstates)): + raise RunError("a run can be finished as failed only when a step or gate has failed or was pushed through; otherwise abandon it") run.data["outcome"] = outcome run.data["finished"] = _now() if note.strip(): @@ -356,17 +355,17 @@ def finalize(run: Run, outcome: str, note: str = "") -> None: def effective_outcome(run: Run, now: datetime | None = None, stale_days: int = STALE_DAYS) -> str: - """finished outcome, or 'open', or 'abandoned' for an open run untouched for stale_days.""" + """finished outcome, or 'stalled' (unfinished), or 'abandoned' for an unfinished run untouched for stale_days.""" if run.data.get("outcome"): return run.data["outcome"] now = now or datetime.now(timezone.utc) last = datetime.fromisoformat(run.data.get("updated") or run.data["created"]) - return "abandoned" if (now - last).days >= stale_days else "open" + return "abandoned" if (now - last).days >= stale_days else "stalled" def scorecard(pack: Pack, now: datetime | None = None, stale_days: int = STALE_DAYS) -> dict: """How often a runbook was tried and how it went, derived from its run records.""" - card = {"tried": 0, "succeeded": 0, "failed": 0, "abandoned": 0, "open": 0, + card = {"tried": 0, "succeeded": 0, "failed": 0, "abandoned": 0, "stalled": 0, "feedback": {"good": 0, "neutral": 0, "poor": 0}} for rid in list_runs(pack): r = load(pack, rid) diff --git a/rtut/run2.py b/rtut/run2.py index c218443..899e9d2 100644 --- a/rtut/run2.py +++ b/rtut/run2.py @@ -1,7 +1,7 @@ """Runs of runbook-pack/v0.2 packs (record runbook-run/v0.2). Adds to the v0.1 run: datapoint states (nil/set/err/ok), a dependency graph with a recommended -next action, alternatives, QA gates with waivers, manual and autopilot modes with timeouts, and +next action, alternatives, QA gates with waivers, manual and auto-confirm modes with timeouts, and a deviation record. Shared helpers (feedback, finalize, captures, score) live in run.py. """ from __future__ import annotations @@ -18,8 +18,10 @@ from .run import Run, RunError, _clean, _now, runs_dir # noqa: F401 (re-export DEFAULT_PHRASE = "I understand this cannot be undone" VERIFY_TIMEOUT = 60 -DONE_STEP = ("done", "skipped") -DONE_GATE = ("passed", "waived") +# "pushed" is its own state: the item was passed over by auto-confirm push-through without being verified. +# It lets dependents proceed but is neither done nor an error, and it is tracked separately. +DONE_STEP = ("done", "skipped", "pushed") +DONE_GATE = ("passed", "waived", "pushed") def flow(run: Run) -> Flow: @@ -85,8 +87,8 @@ def create(pack: Pack, operator: str, values: dict[str, str] | None = None, mode preauthorize: list[str] | None = None) -> Run: if not pack.ok or pack.data.get("engine") != "native" or pack.spec != "runbook-pack/v0.2": raise RunError("only valid native runbook-pack/v0.2 packs can be run here") - if mode not in ("manual", "autopilot"): - raise RunError("mode must be manual or autopilot") + if mode not in ("manual", "auto-confirm"): + raise RunError("mode must be manual or auto-confirm") operator = _clean(operator.strip(), "operator") if not re.fullmatch(r"\S+", operator): raise RunError("operator must be one word (name or handle)") @@ -102,7 +104,7 @@ def create(pack: Pack, operator: str, values: dict[str, str] | None = None, mode "spec": "runbook-run/v0.2", "run_id": time.strftime("%Y%m%d-%H%M%S-") + uuid.uuid4().hex[:6], "pack_id": pack.id, "pack_path": str(pack.path), "operator": operator, "created": _now(), "mode": mode, "params": {}, "deviations": [], - "autopilot": {"preauthorized": pre, "authorized_by": operator if pre else None}, + "auto_confirm": {"preauthorized": pre, "authorized_by": operator if pre else None}, "datapoints": {i: {"state": "nil"} for i in fl.datapoints}, "steps": {i: {"state": "todo"} for i in fl.steps}, "gates": {i: {"state": "open", "results": {}} for i in fl.gates}, @@ -188,7 +190,7 @@ def unmet(run: Run, id_: str) -> list[str]: if fl.kind(id_) == "step": for dp in fl.steps[id_].get("uses", []): st = run.data["datapoints"][dp]["state"] - if st not in ("set", "ok"): + if st not in ("set", "ok", "pushed"): out.append(f"datapoint {dp} is {st}") elif fl.kind(id_) == "gate": stage = fl.parent[id_] @@ -266,9 +268,9 @@ def start(run: Run, step_id: str, alt: str | None = None, phrase: str | None = N raise RunError(f"step {step_id} has no alternatives") eff = effective_step(run.pack, fl, step_id, st.get("alternative")) if not eff.get("reversible", True): # irreversible: the typed phrase is enforced here - pre = run.data["autopilot"]["preauthorized"] - if step_id in pre and run.data["mode"] == "autopilot": - st["authorized_by"] = run.data["autopilot"]["authorized_by"] + pre = run.data["auto_confirm"]["preauthorized"] + if step_id in pre and run.data["mode"] == "auto-confirm": + st["authorized_by"] = run.data["auto_confirm"]["authorized_by"] elif (phrase or "") != phrase_for(run, step_id): raise RunError(f"step {step_id} is irreversible: type the confirmation phrase exactly: " f"{phrase_for(run, step_id)!r}") @@ -298,7 +300,10 @@ def _check_result(run: Run, ck: dict, confirm: bool, manual_ok: bool | None, aut elif exp in ("exit-0", "output-contains"): if not confirm: raise RunError("running a check command needs operator confirmation") - cmd = render(ck["command"], run.data["params"], run.pack) + try: + cmd = render(ck["command"], run.data["params"], run.pack) + except (KeyError, ValueError) as e: # e.g. a datapoint that was pushed through has no value + return {"passed": False, "kind": exp, "parts": {"command": False, "error": str(e)}, "at": _now()} t0 = time.monotonic() try: cp = subprocess.run(cmd, shell=True, capture_output=True, text=True, timeout=VERIFY_TIMEOUT, @@ -428,7 +433,7 @@ def waive(run: Run, gate_id: str, reason: str) -> None: run.save() -# ---- autopilot ---------------------------------------------------------------------------------------------------------- +# ---- auto-confirm ---------------------------------------------------------------------------------------------------------- def _timeout_for(run: Run, id_: str) -> dict | None: return flow(run).node(id_).get("timeout") or run.pack.data.get("timeout") @@ -439,18 +444,18 @@ def _elapsed(started: str, now: datetime) -> float: def _stop(run: Run, item: str, why: str) -> dict: - run.data["autopilot"]["stopped"] = {"item": item, "reason": why, "at": _now()} + run.data["auto_confirm"]["stopped"] = {"item": item, "reason": why, "at": _now()} run.save() return {"action": "stopped", "item": item, "detail": why} -def autopilot_tick(run: Run, now: datetime | None = None) -> dict: - """One autopilot decision. Confirmation is applied when preconditions are met; a timeout stops +def auto_confirm_tick(run: Run, now: datetime | None = None) -> dict: + """One auto-confirm decision. Confirmation is applied when preconditions are met; a timeout stops the run or pushes through, recording a deviation. The engine never runs a step's own command.""" _writable(run) - if run.data["mode"] != "autopilot": - raise RunError("autopilot_tick needs an autopilot run") - ap = run.data["autopilot"] + if run.data["mode"] != "auto-confirm": + raise RunError("auto_confirm_tick needs an auto-confirm run") + ap = run.data["auto_confirm"] if ap.get("stopped"): return {"action": "stopped", "item": ap["stopped"]["item"], "detail": ap["stopped"]["reason"]} now = now or datetime.now(timezone.utc) @@ -463,8 +468,8 @@ def autopilot_tick(run: Run, now: datetime | None = None) -> dict: first = na["blocked"][0] return _stop(run, first["id"], "nothing can advance: " + "; ".join(first["unmet"])) if fl.kind(item) == "step": - return _autopilot_step(run, item, now) - return _autopilot_gate(run, item, now) + return _auto_confirm_step(run, item, now) + return _auto_confirm_gate(run, item, now) def _on_timeout(run: Run, item: str, cfg: dict, detail: str) -> dict: @@ -473,17 +478,17 @@ def _on_timeout(run: Run, item: str, cfg: dict, detail: str) -> dict: return {"action": "push-through", "item": item, "detail": detail} -def _autopilot_step(run: Run, sid: str, now: datetime) -> dict: +def _auto_confirm_step(run: Run, sid: str, now: datetime) -> dict: fl = flow(run) s = fl.steps[sid] st = run.data["steps"][sid] - if s.get("autopilot") == "manual-only": + if s.get("auto_confirm") == "manual-only": return {"action": "awaiting-operator", "item": sid, "detail": "step is manual-only"} if st["state"] == "failed": return {"action": "awaiting-operator", "item": sid, "detail": "step failed; retry or stop"} alt = st.get("alternative") or recommended_alt(run, sid) eff = effective_step(run.pack, fl, sid, alt) - if not eff.get("reversible", True) and sid not in run.data["autopilot"]["preauthorized"]: + if not eff.get("reversible", True) and sid not in run.data["auto_confirm"]["preauthorized"]: return {"action": "awaiting-operator", "item": sid, "detail": "irreversible and not pre-authorized"} if s.get("one_of") and not alt: return {"action": "awaiting-operator", "item": sid, "detail": "no alternative is recommended; the operator must choose"} @@ -503,19 +508,20 @@ def _autopilot_step(run: Run, sid: str, now: datetime) -> dict: if cfg and _elapsed(st["started"], now) >= cfg["seconds"]: out = _on_timeout(run, sid, cfg, "verification did not pass") if out["action"] == "push-through": - st.update(state="done", finished=_now(), pushed_through=True) + st.update(state="pushed", finished=_now()) _deviate(run, "push-through", sid, ["verification did not pass"], - f"autopilot timeout after {cfg['seconds']}s", _risk_text(run, sid), False) + f"auto-confirm timeout after {cfg['seconds']}s", _risk_text(run, sid), False) for dp in s.get("produces", []): rec = run.data["datapoints"].get(dp) - if rec and rec["state"] == "nil": - rec.update(state="err", error="pushed through without verification", at=_now()) + if rec and rec["state"] in ("nil", "err"): + rec.pop("error", None) + rec.update(state="pushed", how="push-through", at=_now()) run.save() return out return {"action": "waiting", "item": sid, "detail": "verification did not pass yet"} -def _autopilot_gate(run: Run, gid: str, now: datetime) -> dict: +def _auto_confirm_gate(run: Run, gid: str, now: datetime) -> dict: g = flow(run).gates[gid] gs = run.data["gates"][gid] gs.setdefault("started", _now()) @@ -531,9 +537,9 @@ def _autopilot_gate(run: Run, gid: str, now: datetime) -> dict: out = _on_timeout(run, gid, cfg, "gate did not pass") if out["action"] == "push-through": failed = [f"check {i} not passed" for i, r in gs["results"].items() if not r["passed"]] - gs.update(state="waived", waiver={"by_actor": "autopilot", "who": run.data["operator"], + gs.update(state="pushed", pushed={"who": run.data["operator"], "reason": f"push-through after {cfg['seconds']}s", "at": _now()}) - _deviate(run, "push-through", gid, failed, f"autopilot timeout after {cfg['seconds']}s", _risk_text(run, gid), False) + _deviate(run, "push-through", gid, failed, f"auto-confirm timeout after {cfg['seconds']}s", _risk_text(run, gid), False) run.save() return out return {"action": "waiting", "item": gid, "detail": "gate did not pass yet"} diff --git a/rtut/ui.py b/rtut/ui.py index fb0f089..10837d1 100644 --- a/rtut/ui.py +++ b/rtut/ui.py @@ -47,7 +47,7 @@ SMILEYS = (("good", "馃檪", "Went well"), ("neutral", "馃槓", "So-so"), ("poor", def score_line(c: dict) -> str: return (f"tried {c['tried']} 路 succeeded {c['succeeded']} 路 failed {c['failed']} 路 " - f"abandoned {c['abandoned']} 路 open {c['open']}") + f"abandoned {c['abandoned']} 路 stalled {c['stalled']}") class App: @@ -147,7 +147,7 @@ class App: btns.append("") return (f"

Close this run

" "

Finishing records the outcome in the pack's score. A finished run is read-only. " - "Runs left open for a week count as abandoned.

" + "Unfinished runs count as stalled, and as abandoned after a week.

" f"
{self.hidden()}" "

" f"

{' '.join(btns)}

") diff --git a/rtut/v2.py b/rtut/v2.py index a36e8d5..3fc6463 100644 --- a/rtut/v2.py +++ b/rtut/v2.py @@ -128,6 +128,9 @@ def _check_refs(m: _Model, data: dict) -> None: for v in vals: if v not in dps[k]["values"]: m.errs.append(f"step {pid}: recommended_when {k}={v} not in values") + for part, pid in parts: + if part.get("idempotent", node.get("idempotent")) == "conditional" and not (part.get("idempotent_note") or node.get("idempotent_note")): + m.errs.append(f"step {pid}: idempotent: conditional needs an idempotent_note saying when") if node.get("one_of") and node.get("command"): m.errs.append(f"step {id_}: a step with one_of must not also have its own command") elif kind == "gate": diff --git a/runbooks/example-hello/pack.yaml b/runbooks/example-hello/pack.yaml index 0262814..7a00553 100644 --- a/runbooks/example-hello/pack.yaml +++ b/runbooks/example-hello/pack.yaml @@ -3,7 +3,7 @@ id: rtut.example-hello title: Example - check a host answers owner: runbook-tutorials outcome: You have proven a host answers on the network. -exercise_status: unexercised +exercise_status: exercised 2026-09-29 by Bernd engine: native parameters: - {id: host, label: Host name, type: string, default: localhost, pattern: "[a-zA-Z0-9.-]+"} diff --git a/runbooks/showcase-lighthouse/pack.v0.2.yaml b/runbooks/showcase-lighthouse/pack.v0.2.yaml index 1e11ae3..a9e7de4 100644 --- a/runbooks/showcase-lighthouse/pack.v0.2.yaml +++ b/runbooks/showcase-lighthouse/pack.v0.2.yaml @@ -41,6 +41,7 @@ stages: goal: The lamp is lit and the weather is known before dark. steps: # no requires among these: they are parallel; list order is the recommended order - id: read-the-sign + idempotent: idempotent title: Read the poem over the door actor: keeper owner: the-lighthouse @@ -48,6 +49,7 @@ stages: reversible: true verify: {done_when: You read the poem, expect: manual} - id: light-lamp + idempotent: idempotent title: Light the lamp actor: keeper owner: the-lighthouse @@ -58,6 +60,7 @@ stages: downstream_risk: Without a lit lamp the night watch has no purpose and ships are not warned. verify: {done_when: The lamp is lit, expect: manual} - id: read-the-sky + idempotent: idempotent title: Read the sky actor: keeper owner: the-lighthouse @@ -83,6 +86,7 @@ stages: requires: [gate:lamp-check] steps: - id: warn + idempotent: idempotent title: Warn the ships actor: keeper owner: the-lighthouse @@ -107,6 +111,8 @@ stages: reversible: true verify: {done_when: The door is barred, expect: manual} - id: write-logbook + idempotent: non-idempotent + idempotent_note: Appends a line every time it runs; a second run leaves a duplicate entry. title: Write tonight's entry actor: keeper owner: the-lighthouse @@ -124,8 +130,10 @@ stages: requires: [night] steps: - id: close-log + idempotent: conditional + idempotent_note: Safe to repeat only while the log file exists; a second run fails on the read. title: Close the log for good - autopilot: manual-only # never confirmed by the engine on its own + auto_confirm: manual-only # never confirmed by the engine on its own actor: harbourmaster owner: the-lighthouse uses: [logdir] diff --git a/schema/runbook-pack-v0.2.schema.json b/schema/runbook-pack-v0.2.schema.json index 77b89f4..61f1b6b 100644 --- a/schema/runbook-pack-v0.2.schema.json +++ b/schema/runbook-pack-v0.2.schema.json @@ -48,7 +48,7 @@ }, "timeout": { "$ref": "#/$defs/timeout", - "description": "Default for autopilot runs." + "description": "Default for auto-confirm runs." }, "legacy": { "type": "object", @@ -341,6 +341,16 @@ "type": "string" } } + }, + "idempotent": { + "enum": [ + "idempotent", + "non-idempotent", + "conditional" + ] + }, + "idempotent_note": { + "type": "string" } } }, @@ -417,18 +427,30 @@ "$ref": "#/$defs/alternative" } }, - "autopilot": { + "timeout": { + "$ref": "#/$defs/timeout" + }, + "downstream_risk": { + "type": "string" + }, + "auto_confirm": { "enum": [ "allowed", "manual-only" ], "default": "allowed" }, - "timeout": { - "$ref": "#/$defs/timeout" + "idempotent": { + "enum": [ + "idempotent", + "non-idempotent", + "conditional" + ], + "description": "Is re-running this step after a partial or complete earlier attempt safe? Omit if unknown. Values are words, never yes/no: YAML reads unquoted yes/no as booleans." }, - "downstream_risk": { - "type": "string" + "idempotent_note": { + "type": "string", + "description": "Required when idempotent is conditional: under what condition." } }, "anyOf": [ diff --git a/tests/test_run2.py b/tests/test_run2.py index 87fe20c..77092d2 100644 --- a/tests/test_run2.py +++ b/tests/test_run2.py @@ -196,25 +196,25 @@ def test_irreversible_step_needs_the_typed_phrase(tmp_path): def test_preauthorization_only_for_irreversible_steps(tmp_path): with pytest.raises(R.RunError, match="only irreversible"): - new(tmp_path, mode="autopilot", preauthorize=["light-lamp"]) + new(tmp_path, mode="auto-confirm", preauthorize=["light-lamp"]) -# ---- autopilot ------------------------------------------------------------------------------------------------ +# ---- auto-confirm ------------------------------------------------------------------------------------------------ def drive(r, ticks=60, now=None, operator=None): log = [] for _ in range(ticks): - out = R.autopilot_tick(r, now) + out = R.auto_confirm_tick(r, now) log.append(out) if out["action"] in ("finished", "stopped", "awaiting-operator", "waiting"): break if out["action"] == "started" and operator: - operator(out["item"]) # the operator runs the step's command while autopilot confirms + operator(out["item"]) # the operator runs the step's command while auto-confirm confirms return log -def test_autopilot_confirms_when_preconditions_are_met_and_waits_for_the_operator_on_manual_only(tmp_path): - r = new(tmp_path, mode="autopilot") +def test_auto_confirm_confirms_when_preconditions_are_met_and_waits_for_the_operator_on_manual_only(tmp_path): + r = new(tmp_path, mode="auto-confirm") def operator(sid): cmd = R.command_for(r, sid) @@ -232,37 +232,37 @@ def test_autopilot_confirms_when_preconditions_are_met_and_waits_for_the_operato R.start(r, "close-log", phrase=PHRASE) # the operator takes over the last step subprocess.run(R.command_for(r, "close-log"), shell=True, check=True, capture_output=True) R.verify(r, "close-log", confirm=True); R.complete(r, "close-log") - assert R.autopilot_tick(r)["action"] == "completed" # gate night-on-record - assert R.autopilot_tick(r)["action"] == "finished" + assert R.auto_confirm_tick(r)["action"] == "completed" # gate night-on-record + assert R.auto_confirm_tick(r)["action"] == "finished" R1.finalize(r, "succeeded") assert r.data["outcome"] == "succeeded" and not r.data["deviations"] -def test_autopilot_timeout_stop(tmp_path): +def test_auto_confirm_timeout_stop(tmp_path): def m(d): d["timeout"] = {"seconds": 30, "on_timeout": "stop"} - r = new(tmp_path, m, mode="autopilot") + r = new(tmp_path, m, mode="auto-confirm") log = drive(r) # write-logbook's file check cannot pass: nobody ran its command assert log[-1]["action"] == "waiting" and log[-1]["item"] == "write-logbook" late = datetime.now(timezone.utc) + timedelta(seconds=31) - out = R.autopilot_tick(r, late) + out = R.auto_confirm_tick(r, late) assert out["action"] == "stopped" and "timed out after 30s" in out["detail"] - assert R.autopilot_tick(r)["action"] == "stopped" # stays stopped + assert R.auto_confirm_tick(r)["action"] == "stopped" # stays stopped assert r.data["deviations"] == [] # stopping is not a deviation -def test_autopilot_push_through_records_deviation_and_moves_on(tmp_path): +def test_auto_confirm_push_through_records_deviation_and_moves_on(tmp_path): def m(d): d["timeout"] = {"seconds": 30, "on_timeout": "push-through"} - r = new(tmp_path, m, mode="autopilot") + r = new(tmp_path, m, mode="auto-confirm") drive(r) late = datetime.now(timezone.utc) + timedelta(seconds=31) - out = R.autopilot_tick(r, late) + out = R.auto_confirm_tick(r, late) assert out["action"] == "push-through" and out["item"] == "write-logbook" - assert r.data["steps"]["write-logbook"]["state"] == "done" and r.data["steps"]["write-logbook"]["pushed_through"] + assert r.data["steps"]["write-logbook"]["state"] == "pushed" # its own state: not done, not failed dev = r.data["deviations"][0] - assert dev["kind"] == "push-through" and dev["mode"] == "autopilot" and dev["risk_acknowledged"] is False - nxt = R.autopilot_tick(r, late) # continues to the manual-only step + assert dev["kind"] == "push-through" and dev["mode"] == "auto-confirm" and dev["risk_acknowledged"] is False + nxt = R.auto_confirm_tick(r, late) # continues to the manual-only step assert nxt["item"] == "close-log" and nxt["action"] == "awaiting-operator" with pytest.raises(R1.RunError, match="deviations"): for k in r.data["steps"]: @@ -270,37 +270,37 @@ def test_autopilot_push_through_records_deviation_and_moves_on(tmp_path): R1.exercise_update(r) -def test_autopilot_gate_push_through_and_stop(tmp_path): +def test_auto_confirm_gate_push_through_and_stop(tmp_path): def m(d): d["stages"][0]["gates"][0]["checks"].append({"done_when": "always fails", "command": "false", "expect": "exit-0"}) d["stages"][0]["gates"][0]["timeout"] = {"seconds": 10, "on_timeout": "push-through"} - r = new(tmp_path, m, mode="autopilot") + r = new(tmp_path, m, mode="auto-confirm") drive(r) assert r.data["gates"]["lamp-check"]["state"] == "failed" # all checks evaluated, one failing - out = R.autopilot_tick(r, datetime.now(timezone.utc) + timedelta(seconds=11)) - assert out["action"] == "push-through" and r.data["gates"]["lamp-check"]["state"] == "waived" + out = R.auto_confirm_tick(r, datetime.now(timezone.utc) + timedelta(seconds=11)) + assert out["action"] == "push-through" and r.data["gates"]["lamp-check"]["state"] == "pushed" # not "waived": nobody approved it assert r.data["deviations"][0]["item"] == "lamp-check" and r.data["deviations"][0]["kind"] == "push-through" -def test_autopilot_never_confirms_irreversible_steps_unless_preauthorized(tmp_path): +def test_auto_confirm_never_confirms_irreversible_steps_unless_preauthorized(tmp_path): def m(d): for st in d["stages"]: for s in st.get("steps", []): - s.pop("autopilot", None) # allow automation of close-log - r = new(tmp_path, m, mode="autopilot") + s.pop("auto_confirm", None) # allow automation of close-log + r = new(tmp_path, m, mode="auto-confirm") drive(r, operator=lambda sid: (subprocess.run(R.command_for(r, sid), shell=True, check=True, capture_output=True) if R.command_for(r, sid) else None)) - last = R.autopilot_tick(r) + last = R.auto_confirm_tick(r) assert last["action"] == "awaiting-operator" and "not pre-authorized" in last["detail"] - r2 = new(tmp_path, m, mode="autopilot", preauthorize=["close-log"]) + r2 = new(tmp_path, m, mode="auto-confirm", preauthorize=["close-log"]) drive(r2, operator=lambda sid: (subprocess.run(R.command_for(r2, sid), shell=True, check=True, capture_output=True) if R.command_for(r2, sid) else None)) assert r2.data["steps"]["close-log"]["state"] in ("running", "done") assert r2.data["steps"]["close-log"]["authorized_by"] == "tester" -def test_manual_mode_rejects_autopilot_and_autopilot_rejects_overrides(tmp_path): - with pytest.raises(R.RunError, match="autopilot run"): - R.autopilot_tick(new(tmp_path)) - r = new(tmp_path, mode="autopilot") +def test_manual_mode_rejects_auto_confirm_and_auto_confirm_rejects_overrides(tmp_path): + with pytest.raises(R.RunError, match="auto-confirm run"): + R.auto_confirm_tick(new(tmp_path)) + r = new(tmp_path, mode="auto-confirm") with pytest.raises(R.RunError, match="manual mode"): R.start(r, "warn", alt="sound-horn", override_reason="x", acknowledge_risk=True) @@ -314,3 +314,32 @@ def test_finished_v02_run_is_read_only_and_scores_count_it(tmp_path): R.set_datapoint(r, "weather", "clear") assert R1.scorecard(r.pack)["abandoned"] == 1 assert R1.load(r.pack, r.data["run_id"]).data["spec"] == "runbook-run/v0.2" # reloads through the shared loader + + +def test_pushed_items_are_a_distinct_state_that_lets_the_run_continue_but_never_succeed(tmp_path): + def m(d): + d["timeout"] = {"seconds": 5, "on_timeout": "push-through"} + light = next(s for st in d["stages"] for s in st.get("steps", []) if s["id"] == "light-lamp") + light["verify"] = {"done_when": "never true here", "command": "false", "expect": "exit-0"} + d["stages"][0]["gates"][0]["timeout"] = {"seconds": 5, "on_timeout": "push-through"} # the gate's own timeout wins + r = new(tmp_path, m, mode="auto-confirm") + late = datetime.now(timezone.utc) + timedelta(seconds=6) + seen = [] + for _ in range(40): + out = R.auto_confirm_tick(r, late) + seen.append(out["action"]) + if out["action"] in ("stopped", "awaiting-operator", "finished"): + break + late = late + timedelta(seconds=6) + assert r.data["steps"]["light-lamp"]["state"] == "pushed" + dp = r.data["datapoints"]["lamp_lit"] + assert dp["state"] == "pushed" and "value" not in dp and "error" not in dp # not set, not err + assert "lamp_lit" not in r.data["params"] # never usable as a value + assert R.is_complete(r, "light-lamp") # dependents may proceed + assert r.data["gates"]["lamp-check"]["state"] == "pushed" # datapoints_ok needs 'ok', so the gate was pushed too + kinds = [d["kind"] for d in r.data["deviations"]] + assert kinds.count("push-through") >= 2 and "override" not in kinds + with pytest.raises(R1.RunError, match="succeeds only when every step is done"): + R1.finalize(r, "succeeded") + R1.finalize(r, "failed", "pushed through to collect problems") # a clean-completion failure, finishable as failed + assert r.data["outcome"] == "failed" diff --git a/tests/test_score.py b/tests/test_score.py index a62c7df..dbbdbfd 100644 --- a/tests/test_score.py +++ b/tests/test_score.py @@ -54,12 +54,12 @@ def test_scorecard_counts_every_outcome_and_stale_open_runs(pk): ok = R.create(pk, "a"); play_all(ok); R.finalize(ok, "succeeded"); R.add_feedback(ok, "good") bad = R.create(pk, "b"); R.start(bad, "read-the-sign"); R.fail(bad, "read-the-sign", "x"); R.finalize(bad, "failed") gone = R.create(pk, "c"); R.finalize(gone, "abandoned") - open_ = R.create(pk, "d") + stalled = R.create(pk, "d") stale = R.create(pk, "e") stale.data["updated"] = (datetime.now(timezone.utc) - timedelta(days=9)).isoformat() stale.path.write_text(__import__("json").dumps(stale.data)) c = R.scorecard(pk) - assert (c["tried"], c["succeeded"], c["failed"], c["abandoned"], c["open"]) == (5, 1, 1, 2, 1) + assert (c["tried"], c["succeeded"], c["failed"], c["abandoned"], c["stalled"]) == (5, 1, 1, 2, 1) assert c["feedback"]["good"] == 1 diff --git a/tests/test_ui.py b/tests/test_ui.py index 9e8a132..06b614f 100644 --- a/tests/test_ui.py +++ b/tests/test_ui.py @@ -236,7 +236,7 @@ def test_score_runs_list_and_finalize_flow(srv): base = loc.split("#")[0] i = int(base.split("/")[2]) _, _, body = req(srv, "GET", f"/pack/{i}") - assert "tried 1 路 succeeded 0" in body and "open 1" in body and "Runs" in body + assert "tried 1 路 succeeded 0" in body and "stalled 1" in body and "Runs" in body _, _, body = req(srv, "GET", base) assert "Abandon run" in body and "Finish: succeeded" not in body # cannot succeed before every step is done _finish_showcase(srv, t, base) diff --git a/tests/test_v2.py b/tests/test_v2.py index f1b8f00..0f33d78 100644 --- a/tests/test_v2.py +++ b/tests/test_v2.py @@ -61,7 +61,7 @@ CASES = [ ("gate without goal", lambda d: gate(d, "lamp-check").pop("goal"), "goal"), ("waiver must need a reason", lambda d: gate(d, "lamp-check").update(waivable={"by": "harbourmaster", "needs_reason": False}), "needs_reason"), ("bad timeout action", lambda d: gate(d, "lamp-check").update(timeout={"seconds": 5, "on_timeout": "explode"}), "on_timeout"), - ("bad autopilot value", lambda d: step(d, "close-log").update(autopilot="always"), "autopilot"), + ("bad auto_confirm value", lambda d: step(d, "close-log").update(auto_confirm="always"), "auto_confirm"), ] @@ -110,3 +110,23 @@ def test_legacy_v02_needs_only_schema(): def test_gate_cannot_confirm_a_datapoint_it_requires(): out = errs(lambda d: next(x for x in d["datapoints"] if x["id"] == "lamp_lit").update(confirmed_by="lamp-check")) assert any("circular" in e for e in out), out + + +def test_idempotency_is_declared_validated_and_summarized(): + from rtut import flow + assert P.load(FILE).ok + assert flow.pack_idempotency(GOOD) == "non-idempotent" # write-logbook appends + d = copy.deepcopy(GOOD) + for st in d["stages"]: + for s in st.get("steps", []): + s["idempotent"] = "idempotent" + s.pop("idempotent_note", None) + assert flow.pack_idempotency(d) == "idempotent" and v2.validate(d) == [] + step(d, "close-log")["idempotent"] = "conditional" + assert any("idempotent_note" in e for e in v2.validate(d)) + step(d, "close-log")["idempotent_note"] = "only while the log file exists" + assert v2.validate(d) == [] and flow.pack_idempotency(d) == "conditional" + step(d, "close-log").pop("idempotent") + assert flow.pack_idempotency(d) == "unknown" # one undeclared step makes the whole unknown + assert errs(lambda x: step(x, "warn").update(idempotent="sometimes")) + assert errs(lambda x: step(x, "warn").update(idempotent=True)) # a YAML "yes" is refused, not silently accepted diff --git a/workplans/RBT-WP-0004-local-runbook-ui.md b/workplans/RBT-WP-0004-local-runbook-ui.md index a2e2c44..af6ec76 100644 --- a/workplans/RBT-WP-0004-local-runbook-ui.md +++ b/workplans/RBT-WP-0004-local-runbook-ui.md @@ -159,7 +159,7 @@ priority: high state_hub_task_id: "99320417-8261-556b-8932-4947cfd8ce11" ``` -Unblocked: RBT-WP-0007 T06-T07 are finished. Choose manual or autopilot per run (with timeout and +Unblocked: RBT-WP-0007 T06-T07 are finished. Choose manual or auto-confirm per run (with timeout and push-through controls). In manual mode, an override dialog lists unmet preconditions and the downstream risk before continuing; deviations show in the overview. diff --git a/workplans/RBT-WP-0007-engine-v0-2-graph-datapoints-and-gates.md b/workplans/RBT-WP-0007-engine-v0-2-graph-datapoints-and-gates.md index f53ab4b..b445e4a 100644 --- a/workplans/RBT-WP-0007-engine-v0-2-graph-datapoints-and-gates.md +++ b/workplans/RBT-WP-0007-engine-v0-2-graph-datapoints-and-gates.md @@ -78,7 +78,7 @@ state_hub_task_id: "0933ff4a-8478-50d9-aa99-991948bc2379" Derive confirm button versus typed phrase from `reversible`; enforce the phrase in the engine so no renderer can skip it. -## Operating modes: manual and autopilot +## Operating modes: manual and auto-confirm ```task id: RBT-WP-0007-T06 @@ -88,8 +88,8 @@ state_hub_task_id: "1656597c-3748-5480-b136-e2872436bab4" ``` A run has a mode (`docs/pack-spec-v0.2.md`). Manual: the operator confirms; every -check, gate and precondition is shown with its state. Autopilot: the engine confirms -when all preconditions are met. `autopilot: manual-only` steps and irreversible steps +check, gate and precondition is shown with its state. Auto-confirm: the engine confirms +when all preconditions are met. `auto_confirm: manual-only` steps and irreversible steps are never auto-confirmed without a recorded pre-authorization. ## Overrides, waivers, push-through and the deviation record @@ -103,7 +103,7 @@ state_hub_task_id: "0ef958ef-88b6-565b-8797-df730cf11ad3" Override in manual mode when preconditions are unmet: name the unmet items and the declared `downstream_risk`, require explicit confirmation and a reason, record a -deviation. Waivers only where a gate declares `waivable`. Autopilot timeouts +deviation. Waivers only where a gate declares `waivable`. Auto-confirm timeouts (`stop` or `push-through`) record a deviation and, for push-through, move on. A run with any deviation is flagged and can never exercise a pack. @@ -111,7 +111,7 @@ any deviation is flagged and can never exercise a pack. `rtut/flow.py` and `rtut/run2.py`, documented in `docs/run-spec.md` (run record v0.2) and covered by `tests/test_run2.py` (25 tests: readiness and recommendation, parallel items, alternatives, datapoint -states, gates, overrides, waivers, confirmation phrase, autopilot confirm / stop / push-through, +states, gates, overrides, waivers, confirmation phrase, auto-confirm confirm / stop / push-through, manual-only and pre-authorization, read-only finished runs). Two spec corrections came out of it: a gate cannot require a datapoint it is itself `confirmed_by` (validator rule), and `run.finalize` also requires gates passed or waived. Not done here: any UI or CLI for v0.2 runs (RBT-WP-0004). diff --git a/workplans/RBT-WP-0008-runbook-maturity-projects-and-idempotency.md b/workplans/RBT-WP-0008-runbook-maturity-projects-and-idempotency.md new file mode 100644 index 0000000..2126d29 --- /dev/null +++ b/workplans/RBT-WP-0008-runbook-maturity-projects-and-idempotency.md @@ -0,0 +1,103 @@ +--- +id: RBT-WP-0008 +type: workplan +title: "Runbook maturity, projects and idempotency" +domain: infotech +repo: runbook-tutorials +status: active +flavor: implementation +owner: claude +topic_slug: stack +created: "2026-09-29" +updated: "2026-09-29" +depends_on: [RBT-WP-0007] +--- + +## Goal + +Separate what a runbook is (characteristics), how far it can be trusted (maturity, earned and bound to a +pack revision), and what state an environment is in (projects). Replaces the environment-dependent +`exercise_status`. Model and candidate terminology: `docs/maturity-and-projects.md`. + +## Decide the terminology and model + +```task +id: RBT-WP-0008-T01 +status: progress +priority: high +``` + +Review `docs/maturity-and-projects.md` with Bernd: names (project or target, environment class), maturity +ladder and thresholds, project states, where project files live, retry policy for non-idempotent steps. + +## Idempotency as a declared characteristic + +```task +id: RBT-WP-0008-T02 +status: done +priority: high +``` + +Done 2026-09-29: `idempotent: idempotent | non-idempotent | conditional` (+ `idempotent_note`) per step in the v0.2 +schema, validator rule for `conditional`, derived runbook value (`flow.pack_idempotency`), declared honestly in the +showcase. Values are words because YAML reads unquoted yes/no as booleans. Follow-ups: show it in the UI, and use +it for the retry policy (T07). + +## Bind runs to a pack revision + +```task +id: RBT-WP-0008-T03 +status: wait +priority: high +``` + +Record a digest of the pack directory in every run so maturity and project state can be tied to the revision used, +and expire when it changes. + +## Derive maturity from run history + +```task +id: RBT-WP-0008-T04 +status: wait +priority: high +``` + +`draft`, `reviewed`, `rehearsed`, `proven`, `stale`, derived from clean runs (no deviations) in rehearsal and other +environments, replacing `exercise_status` and the "Record pack as exercised" click. + +## Project specification and shared datapoints + +```task +id: RBT-WP-0008-T05 +status: wait +priority: high +``` + +`project.yaml`: environment class, shared non-secret datapoints provided as defaults to member runbooks, member +runbooks and ordering. Validator, and runs that belong to a project. + +## Project state + +```task +id: RBT-WP-0008-T06 +status: wait +priority: medium +``` + +Derive `planned`, `applying`, `applied`, `stale` (later `drifted`) from each member's latest run at the current revision. + +## Retry policy for non-idempotent steps + +```task +id: RBT-WP-0008-T07 +status: wait +priority: medium +``` + +Retrying a failed non-idempotent step requires the operator to confirm the earlier attempt did not partly apply. + +## Acceptance Criteria + +- "Exercised" has a precise, environment-aware meaning; a production project is never called "exercised". +- Maturity is derived from runs, bound to a revision, and expires when the pack changes. +- A value shared by several runbooks is entered once, in the project.