Rename open to stalled and autopilot to auto-confirm; add pushed state; declare idempotency; propose maturity and projects model (RBT-WP-0008)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Assistant: claude-code Assistant-Model: sonnet Assistant-Process: 295952@bnt-lap001 Assistant-Session: e93f64ad-516c-46eb-9666-aad8d300c477
This commit is contained in:
parent
f39d5cec3c
commit
f5619e1a52
19 changed files with 429 additions and 105 deletions
106
docs/maturity-and-projects.md
Normal file
106
docs/maturity-and-projects.md
Normal file
|
|
@ -0,0 +1,106 @@
|
|||
# Runbook maturity, projects and idempotency
|
||||
|
||||
Status: **proposal for decision** (RBT-WP-0008-T01). Raised by Bernd on 2026-09-29.
|
||||
|
||||
## The problem
|
||||
|
||||
`exercise_status: unexercised | exercised <date> by <operator>` mixes three different things:
|
||||
|
||||
1. **What the runbook is like** (does it change state, can it be repeated, can it be undone).
|
||||
2. **How much the runbook can be trusted** (has it been rehearsed, and where).
|
||||
3. **What state an environment is in** (has this runbook been applied to *this* system).
|
||||
|
||||
A run that succeeded on one system says little about another system, so "exercised" is not a
|
||||
property of the runbook alone. Environments differ, and applying a runbook to an environment is a
|
||||
state of that environment, not of the runbook.
|
||||
|
||||
## Terminology survey
|
||||
|
||||
From general DevOps, SRE, configuration-management and ITIL practice, written from memory, not from
|
||||
a source review. Please treat the names as candidates.
|
||||
|
||||
| Area | Terms in common use | Relevance here |
|
||||
| --- | --- | --- |
|
||||
| Automation maturity | manual, semi-automated (guided), automated; the "runbook automation ladder"; toil reduction (SRE) | our modes: manual, auto-confirm, later auto-pilot with execution |
|
||||
| Rehearsal | game day, fire drill, disaster-recovery drill, dry run, rehearsal in staging, `--check` mode (Ansible), `plan` before `apply` (Terraform) | how a runbook earns trust without touching production |
|
||||
| Readiness | production readiness review (PRR), operational readiness, definition of done | a maturity gate before a runbook is used on production |
|
||||
| Repeatability | idempotent, convergent (declare desired state), re-runnable, resumable, reversible; drift; "converged" | idempotency is a *declared characteristic*, not a state |
|
||||
| Change control (ITIL) | standard change (pre-approved because proven repeatable), normal change, change model | maps to "proven" runbooks that need no fresh review |
|
||||
| Environments | environment / stage (dev, test, staging, production), target, inventory and group variables (Ansible), workspace (Terraform), context (kubectl), installation, instance | the thing runbooks are applied to |
|
||||
| Decay | runbook rot, stale, drift between docs and system | maturity must expire when the runbook or the system changes |
|
||||
|
||||
## Proposal: three separate axes
|
||||
|
||||
### 1. Characteristics: declared by the author, static
|
||||
|
||||
- **Idempotency**: `idempotent | non-idempotent | conditional` (with a note). Declared per step; the
|
||||
runbook's value is derived (all idempotent, or any non-idempotent, or unknown if a step is undeclared).
|
||||
**Implemented 2026-09-29** in the v0.2 schema and validator. Values are words, not yes/no, because
|
||||
YAML reads unquoted `yes`/`no` as booleans.
|
||||
- **Reversibility**: already declared per step (`reversible`), with rollback text.
|
||||
- **Effect class** (candidate): read-only, changes-state, destructive.
|
||||
- Consequences to build: retry of a failed non-idempotent step should require the operator to confirm
|
||||
that the earlier attempt did not partly apply; re-running a whole idempotent runbook against a
|
||||
converged environment is safe and is the way to verify it.
|
||||
|
||||
### 2. Maturity: earned, independent of any environment, bound to a pack revision
|
||||
|
||||
Derived from run history, not typed in. A run records a digest of the pack revision it used.
|
||||
|
||||
| Level | Meaning (candidate names) |
|
||||
| --- | --- |
|
||||
| `draft` | valid against the spec, never run |
|
||||
| `reviewed` | a second person has read it (recorded) |
|
||||
| `rehearsed` | completed cleanly (no deviation) in a **rehearsal** environment |
|
||||
| `proven` | completed cleanly in at least two distinct environments by at least two operators |
|
||||
| `stale` | the pack changed since it was proven: maturity is bound to a revision and expires when it changes |
|
||||
|
||||
This replaces `exercise_status`. Deviations (overrides, waivers, push-through) never count towards
|
||||
maturity, which matches the current rule for exercising.
|
||||
|
||||
### 3. Projects: environments the runbooks are applied to
|
||||
|
||||
A **project** groups runbooks that share one environment. It holds what they have in common and provides
|
||||
it as datapoint defaults (like Ansible group variables), so a value such as the target host or tenant is
|
||||
entered once.
|
||||
|
||||
```yaml
|
||||
project: railiance01-identity
|
||||
environment: production # rehearsal | staging | production (class, not free text)
|
||||
datapoints: {tenant: coulomb, target_host: railiance01} # non-secret; never secrets
|
||||
runbooks:
|
||||
- {pack: nk.ssh-certificates}
|
||||
- {pack: nk.openbao-operating-path, after: [nk.ssh-certificates]}
|
||||
```
|
||||
|
||||
Project state is derived from the latest run of each member at the current pack revision:
|
||||
|
||||
| State (candidate) | Meaning |
|
||||
| --- | --- |
|
||||
| `planned` | no member applied yet |
|
||||
| `applying` | a run is in progress, or some members are applied and others not (the "running" state) |
|
||||
| `applied` | every member's latest run succeeded cleanly at the current revision |
|
||||
| `stale` | a member's pack changed, or a run failed or was abandoned, since it was applied |
|
||||
| `drifted` (later) | verification against the live system no longer matches |
|
||||
|
||||
A **rehearsal** project is where maturity evidence comes from. "Exercised" then has a precise meaning:
|
||||
*applied successfully in a rehearsal project*. A production project is `applied`, not "exercised".
|
||||
|
||||
## Consequences for what exists
|
||||
|
||||
- Runs record `project`, the environment class and the pack digest. Scores become per project as well as per
|
||||
runbook. Run records stay non-secret and local (`.local/`), so shared history remains an open question.
|
||||
- `exercise_status` in `pack.yaml` is superseded. Until this is decided it stays as the interim marker.
|
||||
- Project files are declarative and non-secret. Candidate location: the repository that owns the system
|
||||
(for example `net-kingdom/projects/<name>/project.yaml`), next to `runbooks/`.
|
||||
|
||||
## Open questions
|
||||
|
||||
1. **Names.** *Project* is Bernd's word. Alternatives: *target*, *engagement*, *installation*, *rollout*. Is
|
||||
*environment class* (`rehearsal|staging|production`) the right way to separate rehearsal from real?
|
||||
2. **Maturity thresholds.** Is "two environments, two operators" the right bar for `proven`? Should `reviewed`
|
||||
be part of the ladder?
|
||||
3. **Where projects live**, and whether one runbook may belong to several projects (yes, one project per
|
||||
environment, so a runbook applies to many).
|
||||
4. **Retry policy** for non-idempotent steps, as described above.
|
||||
5. **Shared run history**: committing a redacted run summary versus keeping records local.
|
||||
|
|
@ -154,7 +154,7 @@ A **run** has a mode. The mode belongs to the run, not to the pack.
|
|||
| Mode | Who confirms | Checks, gates, preconditions |
|
||||
| --- | --- | --- |
|
||||
| **manual** | the operator | shown with their state; the operator confirms each |
|
||||
| **autopilot** | the engine, when every precondition is met | evaluated by the engine; confirmation is applied automatically |
|
||||
| **auto-confirm** | the engine, when every precondition is met | evaluated by the engine; confirmation is applied automatically |
|
||||
|
||||
### Manual mode: override
|
||||
|
||||
|
|
@ -164,10 +164,10 @@ downstream risk (`downstream_risk` text declared on the step or gate, else a
|
|||
generic warning), (2) require an explicit confirmation, and (3) record a
|
||||
**deviation** in the run.
|
||||
|
||||
### Autopilot mode: timeout and push-through
|
||||
### Auto-confirm mode: timeout and push-through
|
||||
|
||||
- A step or gate may declare `timeout: {seconds: N, on_timeout: stop|push-through}`;
|
||||
a run may set a default. Without a timeout, autopilot waits.
|
||||
a run may set a default. Without a timeout, auto-confirm waits.
|
||||
- `stop` (default): on timeout the run stops and reports.
|
||||
- `push-through`: on timeout the engine confirms the item, records a deviation,
|
||||
and moves on, ignoring the problem. This gathers the most information about
|
||||
|
|
@ -176,10 +176,13 @@ generic warning), (2) require an explicit confirmation, and (3) record a
|
|||
with an explicit, recorded pre-authorization naming those steps (and the pack's
|
||||
`confirm_phrase`). Push-through does not bypass that rule. This is a safety
|
||||
default, open to review.
|
||||
- A step may declare `autopilot: manual-only` to forbid automation entirely.
|
||||
- A step may declare `auto_confirm: manual-only` to forbid automation entirely.
|
||||
- Auto-confirm applies *confirmation* only. It does not run a step's own command; execution is a later, separate level (RBT-WP-0006).
|
||||
|
||||
### Deviations
|
||||
|
||||
An item passed over by push-through gets the distinct state `pushed` (steps, gates and datapoints): not done, not set, not an error.
|
||||
|
||||
Every override, waiver and push-through is a **deviation** in the run record:
|
||||
`kind` (`override|waiver|push-through`), `item` (step or gate id), `mode`,
|
||||
`unmet` (what was not satisfied), `risk_acknowledged`, `reason` (required for
|
||||
|
|
@ -201,6 +204,10 @@ declared in advance (`waivable` on a gate); an **override** is not.
|
|||
|
||||
## Later: dependencies between runbooks
|
||||
|
||||
**Idempotency** is a declared characteristic of a step: `idempotent: idempotent | non-idempotent | conditional`
|
||||
(`conditional` needs an `idempotent_note`). Undeclared means unknown. The runbook's overall value is derived.
|
||||
See `docs/maturity-and-projects.md`.
|
||||
|
||||
Not in v0.2. When a procedure needs another runbook (for example "OpenBao operating
|
||||
path requires the SSH access pack"), packs will declare `requires_packs` with a
|
||||
required result (exercised, or a named gate passed). Tracked in RBT-WP-0006.
|
||||
|
|
|
|||
|
|
@ -27,17 +27,17 @@ CLI: `rtut run new|list|status|cmd|start|verify|done|fail|retry|skip|finish <pac
|
|||
|
||||
## Outcome, score and exercise status
|
||||
|
||||
A run is **open** until finalized. `rtut run finalize` (or the UI's *Close this run*) records
|
||||
A run is unfinished (**stalled**) until finalized. `rtut run finalize` (or the UI's *Close this run*) records
|
||||
an `outcome`:
|
||||
|
||||
| Outcome | Rule |
|
||||
| --- | --- |
|
||||
| `succeeded` | every step is `done` (skipped steps do not count) |
|
||||
| `failed` | at least one step has failed; otherwise abandon |
|
||||
| `abandoned` | any time. An open run untouched for 7 days also counts as abandoned in the score |
|
||||
| `abandoned` | any time. An unfinished run untouched for 7 days also counts as abandoned in the score |
|
||||
|
||||
A finished run is read-only, except for feedback. The **score** of a runbook is derived from
|
||||
its run records: tried, succeeded, failed, abandoned, open, plus a tally of feedback ratings
|
||||
its run records: tried, succeeded, failed, abandoned, stalled (unfinished), plus a tally of feedback ratings
|
||||
(`rtut score <root>`; shown in the UI). Run records live in `.local/` of the machine that ran
|
||||
them, so the score is per machine until run histories are shared.
|
||||
|
||||
|
|
@ -67,12 +67,12 @@ Implemented in `rtut/run2.py` (graph in `rtut/flow.py`). It adds to the v0.1 rec
|
|||
|
||||
| Field | Meaning |
|
||||
| --- | --- |
|
||||
| `mode` | `manual` or `autopilot` |
|
||||
| `datapoints.<id>` | `state` nil, set, err or ok; `value`, `how` (initial, manual, produced, check, manual confirmation), `error` |
|
||||
| `steps.<id>` | as v0.1, plus `alternative` (chosen `one_of` branch), `authorized_by`, `pushed_through` |
|
||||
| `gates.<id>` | `state` open, passed, failed or waived; per-check `results`; `waiver` |
|
||||
| `mode` | `manual` or `auto-confirm` |
|
||||
| `datapoints.<id>` | `state` nil, set, err, ok or pushed; `value`, `how` (initial, manual, produced, check, manual confirmation), `error` |
|
||||
| `steps.<id>` | as v0.1 (`pushed` is an additional state), plus `alternative` (chosen `one_of` branch), `authorized_by` |
|
||||
| `gates.<id>` | `state` open, passed, failed, waived or pushed; per-check `results`; `waiver` or `pushed` |
|
||||
| `deviations[]` | every override, waiver and push-through (see `docs/pack-spec-v0.2.md`) |
|
||||
| `autopilot` | `preauthorized` irreversible steps, `authorized_by`, and `stopped` if a timeout stopped the run |
|
||||
| `auto-confirm` | `preauthorized` irreversible steps, `authorized_by`, and `stopped` if a timeout stopped the run |
|
||||
|
||||
Rules the engine enforces:
|
||||
|
||||
|
|
@ -88,13 +88,20 @@ Rules the engine enforces:
|
|||
- **Override** (manual mode only): starting an item with unmet preconditions needs a reason and an acknowledged
|
||||
downstream risk; it is recorded as a deviation.
|
||||
- **Waiver**: only for gates that declare `waivable`, with a reason; recorded as a deviation.
|
||||
- **Confirmation**: irreversible steps need the exact confirmation phrase at start. In autopilot they run only
|
||||
if the run pre-authorized them, and never if the step is `autopilot: manual-only`.
|
||||
- **Autopilot** (`autopilot_tick`): the engine applies confirmation when preconditions are met: it starts the
|
||||
- **Confirmation**: irreversible steps need the exact confirmation phrase at start. In auto-confirm they run only
|
||||
if the run pre-authorized them, and never if the step is `auto_confirm: manual-only`.
|
||||
- **Auto-confirm** (`auto-confirm_tick`): the engine applies confirmation when preconditions are met: it starts the
|
||||
recommended item, runs its checks (manual checks are auto-confirmed and marked so) and completes it. It
|
||||
never runs a step's own command; the work is done by the operator or, later, by opt-in execution. A step
|
||||
with `evidence` waits for the operator. If checks keep failing until the item's `timeout` (or the pack
|
||||
default), the run stops, or with `on_timeout: push-through` the item is marked done with
|
||||
`pushed_through` and a deviation is recorded.
|
||||
state `pushed` and a deviation is recorded.
|
||||
- **Deviations and exercise**: a run with any deviation or pushed-through step can be finalized but never exercises a pack.
|
||||
`finalize(succeeded)` also needs every gate passed or waived.
|
||||
- **Pushed is its own state.** A step, gate or datapoint passed over by push-through is `pushed`: not `done`,
|
||||
not `set`, not `err`. Dependents may proceed (that is the point of pushing through), but a pushed datapoint
|
||||
has no value, so a command that needs it fails its check instead of running with nothing. A run with pushed
|
||||
items can be finished as `failed` (never `succeeded`) and never exercises a pack.
|
||||
- **Stalled**: an unfinished run is counted as `stalled` in the score, and as `abandoned` after a week.
|
||||
- **Modes** are `manual` and `auto-confirm`. Auto-confirm applies confirmation only; running the step's own
|
||||
command is a separate, later level ("auto-pilot with execution", RBT-WP-0006).
|
||||
|
|
|
|||
|
|
@ -114,7 +114,7 @@ def main(argv: list[str] | None = None) -> int:
|
|||
for p in _packs(a.root):
|
||||
if p.ok and p.data.get("engine") == "native" and p.spec == "runbook-pack/v0.1":
|
||||
c = R.scorecard(p)
|
||||
print(f"{p.id}\ttried {c['tried']}\tsucceeded {c['succeeded']}\tfailed {c['failed']}\tabandoned {c['abandoned']}\topen {c['open']}\t[{p.data['exercise_status']}]")
|
||||
print(f"{p.id}\ttried {c['tried']}\tsucceeded {c['succeeded']}\tfailed {c['failed']}\tabandoned {c['abandoned']}\tstalled {c['stalled']}\t[{p.data['exercise_status']}]")
|
||||
return 0
|
||||
if a.cmd in ("list", "validate"):
|
||||
packs = _packs(a.root)
|
||||
|
|
|
|||
14
rtut/flow.py
14
rtut/flow.py
|
|
@ -67,3 +67,17 @@ class Flow:
|
|||
for a in self.ancestors(id_):
|
||||
refs += self.stages[a].get("requires", [])
|
||||
return [r for r in (self.resolve(x) for x in refs) if r]
|
||||
|
||||
|
||||
def pack_idempotency(data: dict) -> str:
|
||||
"""Derived characteristic of the whole runbook: idempotent, non-idempotent, conditional or unknown."""
|
||||
fl = Flow(data)
|
||||
vals = []
|
||||
for s in fl.steps.values():
|
||||
parts = s.get("one_of") or [s]
|
||||
vals += [p.get("idempotent", s.get("idempotent", "unknown")) for p in parts]
|
||||
if not vals or "unknown" in vals:
|
||||
return "unknown"
|
||||
if "non-idempotent" in vals:
|
||||
return "non-idempotent"
|
||||
return "conditional" if "conditional" in vals else "idempotent"
|
||||
|
|
|
|||
13
rtut/run.py
13
rtut/run.py
|
|
@ -244,8 +244,6 @@ def exercise_update(run: Run, today: date | None = None) -> tuple[str, str]:
|
|||
raise RunError("every step must be done (not skipped) to exercise the pack")
|
||||
if run.data.get("deviations"):
|
||||
raise RunError("a run with deviations (overrides, waivers, push-through) cannot exercise the pack")
|
||||
if any(v.get("pushed_through") for v in run.data["steps"].values()):
|
||||
raise RunError("a run with pushed-through steps cannot exercise the pack")
|
||||
today = today or date.today()
|
||||
old = run.pack.path.read_text()
|
||||
new = STATUS_LINE.sub(f"exercise_status: exercised {today.isoformat()} by {run.data['operator']}",
|
||||
|
|
@ -346,8 +344,9 @@ def finalize(run: Run, outcome: str, note: str = "") -> None:
|
|||
raise RunError("a run succeeds only when every step is done (skipped steps do not count)")
|
||||
if outcome == "succeeded" and any(g["state"] not in ("passed", "waived") for g in run.data.get("gates", {}).values()):
|
||||
raise RunError("a run succeeds only when every gate has passed or been waived")
|
||||
if outcome == "failed" and "failed" not in states:
|
||||
raise RunError("a run can be finished as failed only when a step has failed; otherwise abandon it")
|
||||
gstates = [g["state"] for g in run.data.get("gates", {}).values()]
|
||||
if outcome == "failed" and not ({"failed", "pushed"} & set(states) or {"failed", "pushed"} & set(gstates)):
|
||||
raise RunError("a run can be finished as failed only when a step or gate has failed or was pushed through; otherwise abandon it")
|
||||
run.data["outcome"] = outcome
|
||||
run.data["finished"] = _now()
|
||||
if note.strip():
|
||||
|
|
@ -356,17 +355,17 @@ def finalize(run: Run, outcome: str, note: str = "") -> None:
|
|||
|
||||
|
||||
def effective_outcome(run: Run, now: datetime | None = None, stale_days: int = STALE_DAYS) -> str:
|
||||
"""finished outcome, or 'open', or 'abandoned' for an open run untouched for stale_days."""
|
||||
"""finished outcome, or 'stalled' (unfinished), or 'abandoned' for an unfinished run untouched for stale_days."""
|
||||
if run.data.get("outcome"):
|
||||
return run.data["outcome"]
|
||||
now = now or datetime.now(timezone.utc)
|
||||
last = datetime.fromisoformat(run.data.get("updated") or run.data["created"])
|
||||
return "abandoned" if (now - last).days >= stale_days else "open"
|
||||
return "abandoned" if (now - last).days >= stale_days else "stalled"
|
||||
|
||||
|
||||
def scorecard(pack: Pack, now: datetime | None = None, stale_days: int = STALE_DAYS) -> dict:
|
||||
"""How often a runbook was tried and how it went, derived from its run records."""
|
||||
card = {"tried": 0, "succeeded": 0, "failed": 0, "abandoned": 0, "open": 0,
|
||||
card = {"tried": 0, "succeeded": 0, "failed": 0, "abandoned": 0, "stalled": 0,
|
||||
"feedback": {"good": 0, "neutral": 0, "poor": 0}}
|
||||
for rid in list_runs(pack):
|
||||
r = load(pack, rid)
|
||||
|
|
|
|||
66
rtut/run2.py
66
rtut/run2.py
|
|
@ -1,7 +1,7 @@
|
|||
"""Runs of runbook-pack/v0.2 packs (record runbook-run/v0.2).
|
||||
|
||||
Adds to the v0.1 run: datapoint states (nil/set/err/ok), a dependency graph with a recommended
|
||||
next action, alternatives, QA gates with waivers, manual and autopilot modes with timeouts, and
|
||||
next action, alternatives, QA gates with waivers, manual and auto-confirm modes with timeouts, and
|
||||
a deviation record. Shared helpers (feedback, finalize, captures, score) live in run.py.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
|
@ -18,8 +18,10 @@ from .run import Run, RunError, _clean, _now, runs_dir # noqa: F401 (re-export
|
|||
|
||||
DEFAULT_PHRASE = "I understand this cannot be undone"
|
||||
VERIFY_TIMEOUT = 60
|
||||
DONE_STEP = ("done", "skipped")
|
||||
DONE_GATE = ("passed", "waived")
|
||||
# "pushed" is its own state: the item was passed over by auto-confirm push-through without being verified.
|
||||
# It lets dependents proceed but is neither done nor an error, and it is tracked separately.
|
||||
DONE_STEP = ("done", "skipped", "pushed")
|
||||
DONE_GATE = ("passed", "waived", "pushed")
|
||||
|
||||
|
||||
def flow(run: Run) -> Flow:
|
||||
|
|
@ -85,8 +87,8 @@ def create(pack: Pack, operator: str, values: dict[str, str] | None = None, mode
|
|||
preauthorize: list[str] | None = None) -> Run:
|
||||
if not pack.ok or pack.data.get("engine") != "native" or pack.spec != "runbook-pack/v0.2":
|
||||
raise RunError("only valid native runbook-pack/v0.2 packs can be run here")
|
||||
if mode not in ("manual", "autopilot"):
|
||||
raise RunError("mode must be manual or autopilot")
|
||||
if mode not in ("manual", "auto-confirm"):
|
||||
raise RunError("mode must be manual or auto-confirm")
|
||||
operator = _clean(operator.strip(), "operator")
|
||||
if not re.fullmatch(r"\S+", operator):
|
||||
raise RunError("operator must be one word (name or handle)")
|
||||
|
|
@ -102,7 +104,7 @@ def create(pack: Pack, operator: str, values: dict[str, str] | None = None, mode
|
|||
"spec": "runbook-run/v0.2", "run_id": time.strftime("%Y%m%d-%H%M%S-") + uuid.uuid4().hex[:6],
|
||||
"pack_id": pack.id, "pack_path": str(pack.path), "operator": operator, "created": _now(),
|
||||
"mode": mode, "params": {}, "deviations": [],
|
||||
"autopilot": {"preauthorized": pre, "authorized_by": operator if pre else None},
|
||||
"auto_confirm": {"preauthorized": pre, "authorized_by": operator if pre else None},
|
||||
"datapoints": {i: {"state": "nil"} for i in fl.datapoints},
|
||||
"steps": {i: {"state": "todo"} for i in fl.steps},
|
||||
"gates": {i: {"state": "open", "results": {}} for i in fl.gates},
|
||||
|
|
@ -188,7 +190,7 @@ def unmet(run: Run, id_: str) -> list[str]:
|
|||
if fl.kind(id_) == "step":
|
||||
for dp in fl.steps[id_].get("uses", []):
|
||||
st = run.data["datapoints"][dp]["state"]
|
||||
if st not in ("set", "ok"):
|
||||
if st not in ("set", "ok", "pushed"):
|
||||
out.append(f"datapoint {dp} is {st}")
|
||||
elif fl.kind(id_) == "gate":
|
||||
stage = fl.parent[id_]
|
||||
|
|
@ -266,9 +268,9 @@ def start(run: Run, step_id: str, alt: str | None = None, phrase: str | None = N
|
|||
raise RunError(f"step {step_id} has no alternatives")
|
||||
eff = effective_step(run.pack, fl, step_id, st.get("alternative"))
|
||||
if not eff.get("reversible", True): # irreversible: the typed phrase is enforced here
|
||||
pre = run.data["autopilot"]["preauthorized"]
|
||||
if step_id in pre and run.data["mode"] == "autopilot":
|
||||
st["authorized_by"] = run.data["autopilot"]["authorized_by"]
|
||||
pre = run.data["auto_confirm"]["preauthorized"]
|
||||
if step_id in pre and run.data["mode"] == "auto-confirm":
|
||||
st["authorized_by"] = run.data["auto_confirm"]["authorized_by"]
|
||||
elif (phrase or "") != phrase_for(run, step_id):
|
||||
raise RunError(f"step {step_id} is irreversible: type the confirmation phrase exactly: "
|
||||
f"{phrase_for(run, step_id)!r}")
|
||||
|
|
@ -298,7 +300,10 @@ def _check_result(run: Run, ck: dict, confirm: bool, manual_ok: bool | None, aut
|
|||
elif exp in ("exit-0", "output-contains"):
|
||||
if not confirm:
|
||||
raise RunError("running a check command needs operator confirmation")
|
||||
cmd = render(ck["command"], run.data["params"], run.pack)
|
||||
try:
|
||||
cmd = render(ck["command"], run.data["params"], run.pack)
|
||||
except (KeyError, ValueError) as e: # e.g. a datapoint that was pushed through has no value
|
||||
return {"passed": False, "kind": exp, "parts": {"command": False, "error": str(e)}, "at": _now()}
|
||||
t0 = time.monotonic()
|
||||
try:
|
||||
cp = subprocess.run(cmd, shell=True, capture_output=True, text=True, timeout=VERIFY_TIMEOUT,
|
||||
|
|
@ -428,7 +433,7 @@ def waive(run: Run, gate_id: str, reason: str) -> None:
|
|||
run.save()
|
||||
|
||||
|
||||
# ---- autopilot ----------------------------------------------------------------------------------------------------------
|
||||
# ---- auto-confirm ----------------------------------------------------------------------------------------------------------
|
||||
|
||||
def _timeout_for(run: Run, id_: str) -> dict | None:
|
||||
return flow(run).node(id_).get("timeout") or run.pack.data.get("timeout")
|
||||
|
|
@ -439,18 +444,18 @@ def _elapsed(started: str, now: datetime) -> float:
|
|||
|
||||
|
||||
def _stop(run: Run, item: str, why: str) -> dict:
|
||||
run.data["autopilot"]["stopped"] = {"item": item, "reason": why, "at": _now()}
|
||||
run.data["auto_confirm"]["stopped"] = {"item": item, "reason": why, "at": _now()}
|
||||
run.save()
|
||||
return {"action": "stopped", "item": item, "detail": why}
|
||||
|
||||
|
||||
def autopilot_tick(run: Run, now: datetime | None = None) -> dict:
|
||||
"""One autopilot decision. Confirmation is applied when preconditions are met; a timeout stops
|
||||
def auto_confirm_tick(run: Run, now: datetime | None = None) -> dict:
|
||||
"""One auto-confirm decision. Confirmation is applied when preconditions are met; a timeout stops
|
||||
the run or pushes through, recording a deviation. The engine never runs a step's own command."""
|
||||
_writable(run)
|
||||
if run.data["mode"] != "autopilot":
|
||||
raise RunError("autopilot_tick needs an autopilot run")
|
||||
ap = run.data["autopilot"]
|
||||
if run.data["mode"] != "auto-confirm":
|
||||
raise RunError("auto_confirm_tick needs an auto-confirm run")
|
||||
ap = run.data["auto_confirm"]
|
||||
if ap.get("stopped"):
|
||||
return {"action": "stopped", "item": ap["stopped"]["item"], "detail": ap["stopped"]["reason"]}
|
||||
now = now or datetime.now(timezone.utc)
|
||||
|
|
@ -463,8 +468,8 @@ def autopilot_tick(run: Run, now: datetime | None = None) -> dict:
|
|||
first = na["blocked"][0]
|
||||
return _stop(run, first["id"], "nothing can advance: " + "; ".join(first["unmet"]))
|
||||
if fl.kind(item) == "step":
|
||||
return _autopilot_step(run, item, now)
|
||||
return _autopilot_gate(run, item, now)
|
||||
return _auto_confirm_step(run, item, now)
|
||||
return _auto_confirm_gate(run, item, now)
|
||||
|
||||
|
||||
def _on_timeout(run: Run, item: str, cfg: dict, detail: str) -> dict:
|
||||
|
|
@ -473,17 +478,17 @@ def _on_timeout(run: Run, item: str, cfg: dict, detail: str) -> dict:
|
|||
return {"action": "push-through", "item": item, "detail": detail}
|
||||
|
||||
|
||||
def _autopilot_step(run: Run, sid: str, now: datetime) -> dict:
|
||||
def _auto_confirm_step(run: Run, sid: str, now: datetime) -> dict:
|
||||
fl = flow(run)
|
||||
s = fl.steps[sid]
|
||||
st = run.data["steps"][sid]
|
||||
if s.get("autopilot") == "manual-only":
|
||||
if s.get("auto_confirm") == "manual-only":
|
||||
return {"action": "awaiting-operator", "item": sid, "detail": "step is manual-only"}
|
||||
if st["state"] == "failed":
|
||||
return {"action": "awaiting-operator", "item": sid, "detail": "step failed; retry or stop"}
|
||||
alt = st.get("alternative") or recommended_alt(run, sid)
|
||||
eff = effective_step(run.pack, fl, sid, alt)
|
||||
if not eff.get("reversible", True) and sid not in run.data["autopilot"]["preauthorized"]:
|
||||
if not eff.get("reversible", True) and sid not in run.data["auto_confirm"]["preauthorized"]:
|
||||
return {"action": "awaiting-operator", "item": sid, "detail": "irreversible and not pre-authorized"}
|
||||
if s.get("one_of") and not alt:
|
||||
return {"action": "awaiting-operator", "item": sid, "detail": "no alternative is recommended; the operator must choose"}
|
||||
|
|
@ -503,19 +508,20 @@ def _autopilot_step(run: Run, sid: str, now: datetime) -> dict:
|
|||
if cfg and _elapsed(st["started"], now) >= cfg["seconds"]:
|
||||
out = _on_timeout(run, sid, cfg, "verification did not pass")
|
||||
if out["action"] == "push-through":
|
||||
st.update(state="done", finished=_now(), pushed_through=True)
|
||||
st.update(state="pushed", finished=_now())
|
||||
_deviate(run, "push-through", sid, ["verification did not pass"],
|
||||
f"autopilot timeout after {cfg['seconds']}s", _risk_text(run, sid), False)
|
||||
f"auto-confirm timeout after {cfg['seconds']}s", _risk_text(run, sid), False)
|
||||
for dp in s.get("produces", []):
|
||||
rec = run.data["datapoints"].get(dp)
|
||||
if rec and rec["state"] == "nil":
|
||||
rec.update(state="err", error="pushed through without verification", at=_now())
|
||||
if rec and rec["state"] in ("nil", "err"):
|
||||
rec.pop("error", None)
|
||||
rec.update(state="pushed", how="push-through", at=_now())
|
||||
run.save()
|
||||
return out
|
||||
return {"action": "waiting", "item": sid, "detail": "verification did not pass yet"}
|
||||
|
||||
|
||||
def _autopilot_gate(run: Run, gid: str, now: datetime) -> dict:
|
||||
def _auto_confirm_gate(run: Run, gid: str, now: datetime) -> dict:
|
||||
g = flow(run).gates[gid]
|
||||
gs = run.data["gates"][gid]
|
||||
gs.setdefault("started", _now())
|
||||
|
|
@ -531,9 +537,9 @@ def _autopilot_gate(run: Run, gid: str, now: datetime) -> dict:
|
|||
out = _on_timeout(run, gid, cfg, "gate did not pass")
|
||||
if out["action"] == "push-through":
|
||||
failed = [f"check {i} not passed" for i, r in gs["results"].items() if not r["passed"]]
|
||||
gs.update(state="waived", waiver={"by_actor": "autopilot", "who": run.data["operator"],
|
||||
gs.update(state="pushed", pushed={"who": run.data["operator"],
|
||||
"reason": f"push-through after {cfg['seconds']}s", "at": _now()})
|
||||
_deviate(run, "push-through", gid, failed, f"autopilot timeout after {cfg['seconds']}s", _risk_text(run, gid), False)
|
||||
_deviate(run, "push-through", gid, failed, f"auto-confirm timeout after {cfg['seconds']}s", _risk_text(run, gid), False)
|
||||
run.save()
|
||||
return out
|
||||
return {"action": "waiting", "item": gid, "detail": "gate did not pass yet"}
|
||||
|
|
|
|||
|
|
@ -47,7 +47,7 @@ SMILEYS = (("good", "🙂", "Went well"), ("neutral", "😐", "So-so"), ("poor",
|
|||
|
||||
def score_line(c: dict) -> str:
|
||||
return (f"tried {c['tried']} · succeeded {c['succeeded']} · failed {c['failed']} · "
|
||||
f"abandoned {c['abandoned']} · open {c['open']}")
|
||||
f"abandoned {c['abandoned']} · stalled {c['stalled']}")
|
||||
|
||||
|
||||
class App:
|
||||
|
|
@ -147,7 +147,7 @@ class App:
|
|||
btns.append("<button name=outcome value=abandoned onclick=\"return confirm('Abandon this run?')\">Abandon run</button>")
|
||||
return (f"<div id=close class=step><h2>Close this run</h2>"
|
||||
"<p class=note>Finishing records the outcome in the pack's score. A finished run is read-only. "
|
||||
"Runs left open for a week count as abandoned.</p>"
|
||||
"Unfinished runs count as stalled, and as abandoned after a week.</p>"
|
||||
f"<form method=post action='/run/{i}/{e(rid)}/finalize'>{self.hidden()}"
|
||||
"<p><input name=note class=grow style='width:100%' placeholder='comment (optional)'></p>"
|
||||
f"<p class=row>{' '.join(btns)}</p></form></div>")
|
||||
|
|
|
|||
|
|
@ -128,6 +128,9 @@ def _check_refs(m: _Model, data: dict) -> None:
|
|||
for v in vals:
|
||||
if v not in dps[k]["values"]:
|
||||
m.errs.append(f"step {pid}: recommended_when {k}={v} not in values")
|
||||
for part, pid in parts:
|
||||
if part.get("idempotent", node.get("idempotent")) == "conditional" and not (part.get("idempotent_note") or node.get("idempotent_note")):
|
||||
m.errs.append(f"step {pid}: idempotent: conditional needs an idempotent_note saying when")
|
||||
if node.get("one_of") and node.get("command"):
|
||||
m.errs.append(f"step {id_}: a step with one_of must not also have its own command")
|
||||
elif kind == "gate":
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@ id: rtut.example-hello
|
|||
title: Example - check a host answers
|
||||
owner: runbook-tutorials
|
||||
outcome: You have proven a host answers on the network.
|
||||
exercise_status: unexercised
|
||||
exercise_status: exercised 2026-09-29 by Bernd
|
||||
engine: native
|
||||
parameters:
|
||||
- {id: host, label: Host name, type: string, default: localhost, pattern: "[a-zA-Z0-9.-]+"}
|
||||
|
|
|
|||
|
|
@ -41,6 +41,7 @@ stages:
|
|||
goal: The lamp is lit and the weather is known before dark.
|
||||
steps: # no requires among these: they are parallel; list order is the recommended order
|
||||
- id: read-the-sign
|
||||
idempotent: idempotent
|
||||
title: Read the poem over the door
|
||||
actor: keeper
|
||||
owner: the-lighthouse
|
||||
|
|
@ -48,6 +49,7 @@ stages:
|
|||
reversible: true
|
||||
verify: {done_when: You read the poem, expect: manual}
|
||||
- id: light-lamp
|
||||
idempotent: idempotent
|
||||
title: Light the lamp
|
||||
actor: keeper
|
||||
owner: the-lighthouse
|
||||
|
|
@ -58,6 +60,7 @@ stages:
|
|||
downstream_risk: Without a lit lamp the night watch has no purpose and ships are not warned.
|
||||
verify: {done_when: The lamp is lit, expect: manual}
|
||||
- id: read-the-sky
|
||||
idempotent: idempotent
|
||||
title: Read the sky
|
||||
actor: keeper
|
||||
owner: the-lighthouse
|
||||
|
|
@ -83,6 +86,7 @@ stages:
|
|||
requires: [gate:lamp-check]
|
||||
steps:
|
||||
- id: warn
|
||||
idempotent: idempotent
|
||||
title: Warn the ships
|
||||
actor: keeper
|
||||
owner: the-lighthouse
|
||||
|
|
@ -107,6 +111,8 @@ stages:
|
|||
reversible: true
|
||||
verify: {done_when: The door is barred, expect: manual}
|
||||
- id: write-logbook
|
||||
idempotent: non-idempotent
|
||||
idempotent_note: Appends a line every time it runs; a second run leaves a duplicate entry.
|
||||
title: Write tonight's entry
|
||||
actor: keeper
|
||||
owner: the-lighthouse
|
||||
|
|
@ -124,8 +130,10 @@ stages:
|
|||
requires: [night]
|
||||
steps:
|
||||
- id: close-log
|
||||
idempotent: conditional
|
||||
idempotent_note: Safe to repeat only while the log file exists; a second run fails on the read.
|
||||
title: Close the log for good
|
||||
autopilot: manual-only # never confirmed by the engine on its own
|
||||
auto_confirm: manual-only # never confirmed by the engine on its own
|
||||
actor: harbourmaster
|
||||
owner: the-lighthouse
|
||||
uses: [logdir]
|
||||
|
|
|
|||
|
|
@ -48,7 +48,7 @@
|
|||
},
|
||||
"timeout": {
|
||||
"$ref": "#/$defs/timeout",
|
||||
"description": "Default for autopilot runs."
|
||||
"description": "Default for auto-confirm runs."
|
||||
},
|
||||
"legacy": {
|
||||
"type": "object",
|
||||
|
|
@ -341,6 +341,16 @@
|
|||
"type": "string"
|
||||
}
|
||||
}
|
||||
},
|
||||
"idempotent": {
|
||||
"enum": [
|
||||
"idempotent",
|
||||
"non-idempotent",
|
||||
"conditional"
|
||||
]
|
||||
},
|
||||
"idempotent_note": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
},
|
||||
|
|
@ -417,18 +427,30 @@
|
|||
"$ref": "#/$defs/alternative"
|
||||
}
|
||||
},
|
||||
"autopilot": {
|
||||
"timeout": {
|
||||
"$ref": "#/$defs/timeout"
|
||||
},
|
||||
"downstream_risk": {
|
||||
"type": "string"
|
||||
},
|
||||
"auto_confirm": {
|
||||
"enum": [
|
||||
"allowed",
|
||||
"manual-only"
|
||||
],
|
||||
"default": "allowed"
|
||||
},
|
||||
"timeout": {
|
||||
"$ref": "#/$defs/timeout"
|
||||
"idempotent": {
|
||||
"enum": [
|
||||
"idempotent",
|
||||
"non-idempotent",
|
||||
"conditional"
|
||||
],
|
||||
"description": "Is re-running this step after a partial or complete earlier attempt safe? Omit if unknown. Values are words, never yes/no: YAML reads unquoted yes/no as booleans."
|
||||
},
|
||||
"downstream_risk": {
|
||||
"type": "string"
|
||||
"idempotent_note": {
|
||||
"type": "string",
|
||||
"description": "Required when idempotent is conditional: under what condition."
|
||||
}
|
||||
},
|
||||
"anyOf": [
|
||||
|
|
|
|||
|
|
@ -196,25 +196,25 @@ def test_irreversible_step_needs_the_typed_phrase(tmp_path):
|
|||
|
||||
def test_preauthorization_only_for_irreversible_steps(tmp_path):
|
||||
with pytest.raises(R.RunError, match="only irreversible"):
|
||||
new(tmp_path, mode="autopilot", preauthorize=["light-lamp"])
|
||||
new(tmp_path, mode="auto-confirm", preauthorize=["light-lamp"])
|
||||
|
||||
|
||||
# ---- autopilot ------------------------------------------------------------------------------------------------
|
||||
# ---- auto-confirm ------------------------------------------------------------------------------------------------
|
||||
|
||||
def drive(r, ticks=60, now=None, operator=None):
|
||||
log = []
|
||||
for _ in range(ticks):
|
||||
out = R.autopilot_tick(r, now)
|
||||
out = R.auto_confirm_tick(r, now)
|
||||
log.append(out)
|
||||
if out["action"] in ("finished", "stopped", "awaiting-operator", "waiting"):
|
||||
break
|
||||
if out["action"] == "started" and operator:
|
||||
operator(out["item"]) # the operator runs the step's command while autopilot confirms
|
||||
operator(out["item"]) # the operator runs the step's command while auto-confirm confirms
|
||||
return log
|
||||
|
||||
|
||||
def test_autopilot_confirms_when_preconditions_are_met_and_waits_for_the_operator_on_manual_only(tmp_path):
|
||||
r = new(tmp_path, mode="autopilot")
|
||||
def test_auto_confirm_confirms_when_preconditions_are_met_and_waits_for_the_operator_on_manual_only(tmp_path):
|
||||
r = new(tmp_path, mode="auto-confirm")
|
||||
|
||||
def operator(sid):
|
||||
cmd = R.command_for(r, sid)
|
||||
|
|
@ -232,37 +232,37 @@ def test_autopilot_confirms_when_preconditions_are_met_and_waits_for_the_operato
|
|||
R.start(r, "close-log", phrase=PHRASE) # the operator takes over the last step
|
||||
subprocess.run(R.command_for(r, "close-log"), shell=True, check=True, capture_output=True)
|
||||
R.verify(r, "close-log", confirm=True); R.complete(r, "close-log")
|
||||
assert R.autopilot_tick(r)["action"] == "completed" # gate night-on-record
|
||||
assert R.autopilot_tick(r)["action"] == "finished"
|
||||
assert R.auto_confirm_tick(r)["action"] == "completed" # gate night-on-record
|
||||
assert R.auto_confirm_tick(r)["action"] == "finished"
|
||||
R1.finalize(r, "succeeded")
|
||||
assert r.data["outcome"] == "succeeded" and not r.data["deviations"]
|
||||
|
||||
|
||||
def test_autopilot_timeout_stop(tmp_path):
|
||||
def test_auto_confirm_timeout_stop(tmp_path):
|
||||
def m(d):
|
||||
d["timeout"] = {"seconds": 30, "on_timeout": "stop"}
|
||||
r = new(tmp_path, m, mode="autopilot")
|
||||
r = new(tmp_path, m, mode="auto-confirm")
|
||||
log = drive(r) # write-logbook's file check cannot pass: nobody ran its command
|
||||
assert log[-1]["action"] == "waiting" and log[-1]["item"] == "write-logbook"
|
||||
late = datetime.now(timezone.utc) + timedelta(seconds=31)
|
||||
out = R.autopilot_tick(r, late)
|
||||
out = R.auto_confirm_tick(r, late)
|
||||
assert out["action"] == "stopped" and "timed out after 30s" in out["detail"]
|
||||
assert R.autopilot_tick(r)["action"] == "stopped" # stays stopped
|
||||
assert R.auto_confirm_tick(r)["action"] == "stopped" # stays stopped
|
||||
assert r.data["deviations"] == [] # stopping is not a deviation
|
||||
|
||||
|
||||
def test_autopilot_push_through_records_deviation_and_moves_on(tmp_path):
|
||||
def test_auto_confirm_push_through_records_deviation_and_moves_on(tmp_path):
|
||||
def m(d):
|
||||
d["timeout"] = {"seconds": 30, "on_timeout": "push-through"}
|
||||
r = new(tmp_path, m, mode="autopilot")
|
||||
r = new(tmp_path, m, mode="auto-confirm")
|
||||
drive(r)
|
||||
late = datetime.now(timezone.utc) + timedelta(seconds=31)
|
||||
out = R.autopilot_tick(r, late)
|
||||
out = R.auto_confirm_tick(r, late)
|
||||
assert out["action"] == "push-through" and out["item"] == "write-logbook"
|
||||
assert r.data["steps"]["write-logbook"]["state"] == "done" and r.data["steps"]["write-logbook"]["pushed_through"]
|
||||
assert r.data["steps"]["write-logbook"]["state"] == "pushed" # its own state: not done, not failed
|
||||
dev = r.data["deviations"][0]
|
||||
assert dev["kind"] == "push-through" and dev["mode"] == "autopilot" and dev["risk_acknowledged"] is False
|
||||
nxt = R.autopilot_tick(r, late) # continues to the manual-only step
|
||||
assert dev["kind"] == "push-through" and dev["mode"] == "auto-confirm" and dev["risk_acknowledged"] is False
|
||||
nxt = R.auto_confirm_tick(r, late) # continues to the manual-only step
|
||||
assert nxt["item"] == "close-log" and nxt["action"] == "awaiting-operator"
|
||||
with pytest.raises(R1.RunError, match="deviations"):
|
||||
for k in r.data["steps"]:
|
||||
|
|
@ -270,37 +270,37 @@ def test_autopilot_push_through_records_deviation_and_moves_on(tmp_path):
|
|||
R1.exercise_update(r)
|
||||
|
||||
|
||||
def test_autopilot_gate_push_through_and_stop(tmp_path):
|
||||
def test_auto_confirm_gate_push_through_and_stop(tmp_path):
|
||||
def m(d):
|
||||
d["stages"][0]["gates"][0]["checks"].append({"done_when": "always fails", "command": "false", "expect": "exit-0"})
|
||||
d["stages"][0]["gates"][0]["timeout"] = {"seconds": 10, "on_timeout": "push-through"}
|
||||
r = new(tmp_path, m, mode="autopilot")
|
||||
r = new(tmp_path, m, mode="auto-confirm")
|
||||
drive(r)
|
||||
assert r.data["gates"]["lamp-check"]["state"] == "failed" # all checks evaluated, one failing
|
||||
out = R.autopilot_tick(r, datetime.now(timezone.utc) + timedelta(seconds=11))
|
||||
assert out["action"] == "push-through" and r.data["gates"]["lamp-check"]["state"] == "waived"
|
||||
out = R.auto_confirm_tick(r, datetime.now(timezone.utc) + timedelta(seconds=11))
|
||||
assert out["action"] == "push-through" and r.data["gates"]["lamp-check"]["state"] == "pushed" # not "waived": nobody approved it
|
||||
assert r.data["deviations"][0]["item"] == "lamp-check" and r.data["deviations"][0]["kind"] == "push-through"
|
||||
|
||||
|
||||
def test_autopilot_never_confirms_irreversible_steps_unless_preauthorized(tmp_path):
|
||||
def test_auto_confirm_never_confirms_irreversible_steps_unless_preauthorized(tmp_path):
|
||||
def m(d):
|
||||
for st in d["stages"]:
|
||||
for s in st.get("steps", []):
|
||||
s.pop("autopilot", None) # allow automation of close-log
|
||||
r = new(tmp_path, m, mode="autopilot")
|
||||
s.pop("auto_confirm", None) # allow automation of close-log
|
||||
r = new(tmp_path, m, mode="auto-confirm")
|
||||
drive(r, operator=lambda sid: (subprocess.run(R.command_for(r, sid), shell=True, check=True, capture_output=True) if R.command_for(r, sid) else None))
|
||||
last = R.autopilot_tick(r)
|
||||
last = R.auto_confirm_tick(r)
|
||||
assert last["action"] == "awaiting-operator" and "not pre-authorized" in last["detail"]
|
||||
r2 = new(tmp_path, m, mode="autopilot", preauthorize=["close-log"])
|
||||
r2 = new(tmp_path, m, mode="auto-confirm", preauthorize=["close-log"])
|
||||
drive(r2, operator=lambda sid: (subprocess.run(R.command_for(r2, sid), shell=True, check=True, capture_output=True) if R.command_for(r2, sid) else None))
|
||||
assert r2.data["steps"]["close-log"]["state"] in ("running", "done")
|
||||
assert r2.data["steps"]["close-log"]["authorized_by"] == "tester"
|
||||
|
||||
|
||||
def test_manual_mode_rejects_autopilot_and_autopilot_rejects_overrides(tmp_path):
|
||||
with pytest.raises(R.RunError, match="autopilot run"):
|
||||
R.autopilot_tick(new(tmp_path))
|
||||
r = new(tmp_path, mode="autopilot")
|
||||
def test_manual_mode_rejects_auto_confirm_and_auto_confirm_rejects_overrides(tmp_path):
|
||||
with pytest.raises(R.RunError, match="auto-confirm run"):
|
||||
R.auto_confirm_tick(new(tmp_path))
|
||||
r = new(tmp_path, mode="auto-confirm")
|
||||
with pytest.raises(R.RunError, match="manual mode"):
|
||||
R.start(r, "warn", alt="sound-horn", override_reason="x", acknowledge_risk=True)
|
||||
|
||||
|
|
@ -314,3 +314,32 @@ def test_finished_v02_run_is_read_only_and_scores_count_it(tmp_path):
|
|||
R.set_datapoint(r, "weather", "clear")
|
||||
assert R1.scorecard(r.pack)["abandoned"] == 1
|
||||
assert R1.load(r.pack, r.data["run_id"]).data["spec"] == "runbook-run/v0.2" # reloads through the shared loader
|
||||
|
||||
|
||||
def test_pushed_items_are_a_distinct_state_that_lets_the_run_continue_but_never_succeed(tmp_path):
|
||||
def m(d):
|
||||
d["timeout"] = {"seconds": 5, "on_timeout": "push-through"}
|
||||
light = next(s for st in d["stages"] for s in st.get("steps", []) if s["id"] == "light-lamp")
|
||||
light["verify"] = {"done_when": "never true here", "command": "false", "expect": "exit-0"}
|
||||
d["stages"][0]["gates"][0]["timeout"] = {"seconds": 5, "on_timeout": "push-through"} # the gate's own timeout wins
|
||||
r = new(tmp_path, m, mode="auto-confirm")
|
||||
late = datetime.now(timezone.utc) + timedelta(seconds=6)
|
||||
seen = []
|
||||
for _ in range(40):
|
||||
out = R.auto_confirm_tick(r, late)
|
||||
seen.append(out["action"])
|
||||
if out["action"] in ("stopped", "awaiting-operator", "finished"):
|
||||
break
|
||||
late = late + timedelta(seconds=6)
|
||||
assert r.data["steps"]["light-lamp"]["state"] == "pushed"
|
||||
dp = r.data["datapoints"]["lamp_lit"]
|
||||
assert dp["state"] == "pushed" and "value" not in dp and "error" not in dp # not set, not err
|
||||
assert "lamp_lit" not in r.data["params"] # never usable as a value
|
||||
assert R.is_complete(r, "light-lamp") # dependents may proceed
|
||||
assert r.data["gates"]["lamp-check"]["state"] == "pushed" # datapoints_ok needs 'ok', so the gate was pushed too
|
||||
kinds = [d["kind"] for d in r.data["deviations"]]
|
||||
assert kinds.count("push-through") >= 2 and "override" not in kinds
|
||||
with pytest.raises(R1.RunError, match="succeeds only when every step is done"):
|
||||
R1.finalize(r, "succeeded")
|
||||
R1.finalize(r, "failed", "pushed through to collect problems") # a clean-completion failure, finishable as failed
|
||||
assert r.data["outcome"] == "failed"
|
||||
|
|
|
|||
|
|
@ -54,12 +54,12 @@ def test_scorecard_counts_every_outcome_and_stale_open_runs(pk):
|
|||
ok = R.create(pk, "a"); play_all(ok); R.finalize(ok, "succeeded"); R.add_feedback(ok, "good")
|
||||
bad = R.create(pk, "b"); R.start(bad, "read-the-sign"); R.fail(bad, "read-the-sign", "x"); R.finalize(bad, "failed")
|
||||
gone = R.create(pk, "c"); R.finalize(gone, "abandoned")
|
||||
open_ = R.create(pk, "d")
|
||||
stalled = R.create(pk, "d")
|
||||
stale = R.create(pk, "e")
|
||||
stale.data["updated"] = (datetime.now(timezone.utc) - timedelta(days=9)).isoformat()
|
||||
stale.path.write_text(__import__("json").dumps(stale.data))
|
||||
c = R.scorecard(pk)
|
||||
assert (c["tried"], c["succeeded"], c["failed"], c["abandoned"], c["open"]) == (5, 1, 1, 2, 1)
|
||||
assert (c["tried"], c["succeeded"], c["failed"], c["abandoned"], c["stalled"]) == (5, 1, 1, 2, 1)
|
||||
assert c["feedback"]["good"] == 1
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -236,7 +236,7 @@ def test_score_runs_list_and_finalize_flow(srv):
|
|||
base = loc.split("#")[0]
|
||||
i = int(base.split("/")[2])
|
||||
_, _, body = req(srv, "GET", f"/pack/{i}")
|
||||
assert "tried 1 · succeeded 0" in body and "open 1" in body and "Runs" in body
|
||||
assert "tried 1 · succeeded 0" in body and "stalled 1" in body and "Runs" in body
|
||||
_, _, body = req(srv, "GET", base)
|
||||
assert "Abandon run" in body and "Finish: succeeded" not in body # cannot succeed before every step is done
|
||||
_finish_showcase(srv, t, base)
|
||||
|
|
|
|||
|
|
@ -61,7 +61,7 @@ CASES = [
|
|||
("gate without goal", lambda d: gate(d, "lamp-check").pop("goal"), "goal"),
|
||||
("waiver must need a reason", lambda d: gate(d, "lamp-check").update(waivable={"by": "harbourmaster", "needs_reason": False}), "needs_reason"),
|
||||
("bad timeout action", lambda d: gate(d, "lamp-check").update(timeout={"seconds": 5, "on_timeout": "explode"}), "on_timeout"),
|
||||
("bad autopilot value", lambda d: step(d, "close-log").update(autopilot="always"), "autopilot"),
|
||||
("bad auto_confirm value", lambda d: step(d, "close-log").update(auto_confirm="always"), "auto_confirm"),
|
||||
]
|
||||
|
||||
|
||||
|
|
@ -110,3 +110,23 @@ def test_legacy_v02_needs_only_schema():
|
|||
def test_gate_cannot_confirm_a_datapoint_it_requires():
|
||||
out = errs(lambda d: next(x for x in d["datapoints"] if x["id"] == "lamp_lit").update(confirmed_by="lamp-check"))
|
||||
assert any("circular" in e for e in out), out
|
||||
|
||||
|
||||
def test_idempotency_is_declared_validated_and_summarized():
|
||||
from rtut import flow
|
||||
assert P.load(FILE).ok
|
||||
assert flow.pack_idempotency(GOOD) == "non-idempotent" # write-logbook appends
|
||||
d = copy.deepcopy(GOOD)
|
||||
for st in d["stages"]:
|
||||
for s in st.get("steps", []):
|
||||
s["idempotent"] = "idempotent"
|
||||
s.pop("idempotent_note", None)
|
||||
assert flow.pack_idempotency(d) == "idempotent" and v2.validate(d) == []
|
||||
step(d, "close-log")["idempotent"] = "conditional"
|
||||
assert any("idempotent_note" in e for e in v2.validate(d))
|
||||
step(d, "close-log")["idempotent_note"] = "only while the log file exists"
|
||||
assert v2.validate(d) == [] and flow.pack_idempotency(d) == "conditional"
|
||||
step(d, "close-log").pop("idempotent")
|
||||
assert flow.pack_idempotency(d) == "unknown" # one undeclared step makes the whole unknown
|
||||
assert errs(lambda x: step(x, "warn").update(idempotent="sometimes"))
|
||||
assert errs(lambda x: step(x, "warn").update(idempotent=True)) # a YAML "yes" is refused, not silently accepted
|
||||
|
|
|
|||
|
|
@ -159,7 +159,7 @@ priority: high
|
|||
state_hub_task_id: "99320417-8261-556b-8932-4947cfd8ce11"
|
||||
```
|
||||
|
||||
Unblocked: RBT-WP-0007 T06-T07 are finished. Choose manual or autopilot per run (with timeout and
|
||||
Unblocked: RBT-WP-0007 T06-T07 are finished. Choose manual or auto-confirm per run (with timeout and
|
||||
push-through controls). In manual mode, an override dialog lists unmet preconditions and
|
||||
the downstream risk before continuing; deviations show in the overview.
|
||||
|
||||
|
|
|
|||
|
|
@ -78,7 +78,7 @@ state_hub_task_id: "0933ff4a-8478-50d9-aa99-991948bc2379"
|
|||
Derive confirm button versus typed phrase from `reversible`; enforce the phrase in the
|
||||
engine so no renderer can skip it.
|
||||
|
||||
## Operating modes: manual and autopilot
|
||||
## Operating modes: manual and auto-confirm
|
||||
|
||||
```task
|
||||
id: RBT-WP-0007-T06
|
||||
|
|
@ -88,8 +88,8 @@ state_hub_task_id: "1656597c-3748-5480-b136-e2872436bab4"
|
|||
```
|
||||
|
||||
A run has a mode (`docs/pack-spec-v0.2.md`). Manual: the operator confirms; every
|
||||
check, gate and precondition is shown with its state. Autopilot: the engine confirms
|
||||
when all preconditions are met. `autopilot: manual-only` steps and irreversible steps
|
||||
check, gate and precondition is shown with its state. Auto-confirm: the engine confirms
|
||||
when all preconditions are met. `auto_confirm: manual-only` steps and irreversible steps
|
||||
are never auto-confirmed without a recorded pre-authorization.
|
||||
|
||||
## Overrides, waivers, push-through and the deviation record
|
||||
|
|
@ -103,7 +103,7 @@ state_hub_task_id: "0ef958ef-88b6-565b-8797-df730cf11ad3"
|
|||
|
||||
Override in manual mode when preconditions are unmet: name the unmet items and the
|
||||
declared `downstream_risk`, require explicit confirmation and a reason, record a
|
||||
deviation. Waivers only where a gate declares `waivable`. Autopilot timeouts
|
||||
deviation. Waivers only where a gate declares `waivable`. Auto-confirm timeouts
|
||||
(`stop` or `push-through`) record a deviation and, for push-through, move on. A run with
|
||||
any deviation is flagged and can never exercise a pack.
|
||||
|
||||
|
|
@ -111,7 +111,7 @@ any deviation is flagged and can never exercise a pack.
|
|||
|
||||
`rtut/flow.py` and `rtut/run2.py`, documented in `docs/run-spec.md` (run record v0.2) and covered by
|
||||
`tests/test_run2.py` (25 tests: readiness and recommendation, parallel items, alternatives, datapoint
|
||||
states, gates, overrides, waivers, confirmation phrase, autopilot confirm / stop / push-through,
|
||||
states, gates, overrides, waivers, confirmation phrase, auto-confirm confirm / stop / push-through,
|
||||
manual-only and pre-authorization, read-only finished runs). Two spec corrections came out of it:
|
||||
a gate cannot require a datapoint it is itself `confirmed_by` (validator rule), and `run.finalize`
|
||||
also requires gates passed or waived. Not done here: any UI or CLI for v0.2 runs (RBT-WP-0004).
|
||||
|
|
|
|||
|
|
@ -0,0 +1,103 @@
|
|||
---
|
||||
id: RBT-WP-0008
|
||||
type: workplan
|
||||
title: "Runbook maturity, projects and idempotency"
|
||||
domain: infotech
|
||||
repo: runbook-tutorials
|
||||
status: active
|
||||
flavor: implementation
|
||||
owner: claude
|
||||
topic_slug: stack
|
||||
created: "2026-09-29"
|
||||
updated: "2026-09-29"
|
||||
depends_on: [RBT-WP-0007]
|
||||
---
|
||||
|
||||
## Goal
|
||||
|
||||
Separate what a runbook is (characteristics), how far it can be trusted (maturity, earned and bound to a
|
||||
pack revision), and what state an environment is in (projects). Replaces the environment-dependent
|
||||
`exercise_status`. Model and candidate terminology: `docs/maturity-and-projects.md`.
|
||||
|
||||
## Decide the terminology and model
|
||||
|
||||
```task
|
||||
id: RBT-WP-0008-T01
|
||||
status: progress
|
||||
priority: high
|
||||
```
|
||||
|
||||
Review `docs/maturity-and-projects.md` with Bernd: names (project or target, environment class), maturity
|
||||
ladder and thresholds, project states, where project files live, retry policy for non-idempotent steps.
|
||||
|
||||
## Idempotency as a declared characteristic
|
||||
|
||||
```task
|
||||
id: RBT-WP-0008-T02
|
||||
status: done
|
||||
priority: high
|
||||
```
|
||||
|
||||
Done 2026-09-29: `idempotent: idempotent | non-idempotent | conditional` (+ `idempotent_note`) per step in the v0.2
|
||||
schema, validator rule for `conditional`, derived runbook value (`flow.pack_idempotency`), declared honestly in the
|
||||
showcase. Values are words because YAML reads unquoted yes/no as booleans. Follow-ups: show it in the UI, and use
|
||||
it for the retry policy (T07).
|
||||
|
||||
## Bind runs to a pack revision
|
||||
|
||||
```task
|
||||
id: RBT-WP-0008-T03
|
||||
status: wait
|
||||
priority: high
|
||||
```
|
||||
|
||||
Record a digest of the pack directory in every run so maturity and project state can be tied to the revision used,
|
||||
and expire when it changes.
|
||||
|
||||
## Derive maturity from run history
|
||||
|
||||
```task
|
||||
id: RBT-WP-0008-T04
|
||||
status: wait
|
||||
priority: high
|
||||
```
|
||||
|
||||
`draft`, `reviewed`, `rehearsed`, `proven`, `stale`, derived from clean runs (no deviations) in rehearsal and other
|
||||
environments, replacing `exercise_status` and the "Record pack as exercised" click.
|
||||
|
||||
## Project specification and shared datapoints
|
||||
|
||||
```task
|
||||
id: RBT-WP-0008-T05
|
||||
status: wait
|
||||
priority: high
|
||||
```
|
||||
|
||||
`project.yaml`: environment class, shared non-secret datapoints provided as defaults to member runbooks, member
|
||||
runbooks and ordering. Validator, and runs that belong to a project.
|
||||
|
||||
## Project state
|
||||
|
||||
```task
|
||||
id: RBT-WP-0008-T06
|
||||
status: wait
|
||||
priority: medium
|
||||
```
|
||||
|
||||
Derive `planned`, `applying`, `applied`, `stale` (later `drifted`) from each member's latest run at the current revision.
|
||||
|
||||
## Retry policy for non-idempotent steps
|
||||
|
||||
```task
|
||||
id: RBT-WP-0008-T07
|
||||
status: wait
|
||||
priority: medium
|
||||
```
|
||||
|
||||
Retrying a failed non-idempotent step requires the operator to confirm the earlier attempt did not partly apply.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- "Exercised" has a precise, environment-aware meaning; a production project is never called "exercised".
|
||||
- Maturity is derived from runs, bound to a revision, and expires when the pack changes.
|
||||
- A value shared by several runbooks is entered once, in the project.
|
||||
Loading…
Add table
Add a link
Reference in a new issue