From 6d3414d8b7c958b9cd8bfcb9b78b4d7691ec16e6 Mon Sep 17 00:00:00 2001 From: tegwick Date: Tue, 29 Sep 2026 21:28:56 +0200 Subject: [PATCH] Retire exercise_status: maturity is the only notion; add CLI tests (RBT-WP-0009-T01) Co-Authored-By: Claude Sonnet 5.5 Assistant: claude-code Assistant-Model: sonnet Assistant-Process: 295952@bnt-lap001 Assistant-Session: e93f64ad-516c-46eb-9666-aad8d300c477 --- INTENT.md | 4 +- README.md | 8 +- SCOPE.md | 2 +- docs/maturity-and-projects.md | 2 +- docs/pack-spec-v0.2.md | 4 +- docs/pack-spec.md | 9 +- docs/run-spec.md | 29 +- rtut/__main__.py | 21 +- rtut/pack.py | 2 +- rtut/run.py | 43 +-- rtut/ui.py | 26 +- runbooks/example-hello/pack.yaml | 1 - runbooks/showcase-lighthouse-v2/pack.yaml | 1 - runbooks/showcase-lighthouse/pack.yaml | 1 - schema/runbook-pack-v0.2.schema.json | 4 +- schema/runbook-pack.schema.json | 296 +++++++++++++++--- tests/test_cli.py | 99 ++++++ tests/test_legacy.py | 2 +- tests/test_maturity.py | 6 +- tests/test_pack.py | 14 +- tests/test_projectrun.py | 2 +- tests/test_run.py | 8 +- tests/test_run2.py | 29 +- tests/test_score.py | 18 +- tests/test_showcase.py | 4 +- tests/test_ui.py | 11 +- tests/test_ui_projectrun.py | 2 +- tests/test_v2.py | 9 +- ...inalize-the-runbook-tutorials-workplans.md | 5 +- 29 files changed, 444 insertions(+), 218 deletions(-) create mode 100644 tests/test_cli.py diff --git a/INTENT.md b/INTENT.md index 36b5d06..f837513 100644 --- a/INTENT.md +++ b/INTENT.md @@ -23,8 +23,8 @@ verification and rollback, and non-secret evidence of what actually happened. - **Parameters, not copies.** A runbook declares its parameters; the engine renders commands with the operator's values and never asks for secrets. - **Evidence over assertion.** A step is done when its declared check has - passed and its outcome is recorded. A runbook that was never run is labelled - unexercised, following the attended-procedure standard. + passed and its outcome is recorded. How far a runbook can be trusted (its + maturity) is earned from clean runs in named environments, not declared. - **Never a secret vault.** The engine does not collect, store, log or transmit secret values. It shows the command; the operator runs it. - **Guide, not executor by default.** Steps are run by the operator. Any diff --git a/README.md b/README.md index 6d0c37d..16a6ca5 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ can guide through actual IT administration and operation tasks. ```bash python3 -m rtut validate # validate every pack -python3 -m rtut list # id, engine, exercise status, title +python3 -m rtut list # id, engine, maturity, title python3 -m rtut show # steps with owners and done-when python3 -m rtut render -p name=value python3 -m rtut launch # legacy-command packs: start the existing console @@ -19,10 +19,8 @@ python3 -m rtut maturity # draft | reviewed | rehearsed | prove python3 -m rtut review --by you # record a review of the current revision python3 -m rtut project list|validate|status|attrs|new|run ... # projects, attribute sets, templates python3 -m rtut ui [ ...] # local web UI on 127.0.0.1:8877: projects, templates, runbooks; v0.1 and v0.2 runs (guided flow, data view, overview) -python3 -m rtut run new|start|verify|done|finish ... # the same run record from the terminal +python3 -m rtut run new|start|verify|done|finalize ... # the same run record from the terminal make test ``` -Status: specification, validator, legacy launch, run records with verification and -exercise-status diff, and a first local web UI work. Terminal walk-through mode and -the console migration are open (`workplans/`). +Status: see `SCOPE.md`; the workplans are in `workplans/`. diff --git a/SCOPE.md b/SCOPE.md index a5802c7..65d2421 100644 --- a/SCOPE.md +++ b/SCOPE.md @@ -20,7 +20,7 @@ bootstrap console) working unchanged. - Runbook pack specification, JSON Schema, and versioning rules - Pack discovery, loading, validation and conformance tests - The local UI and CLI that present a pack and record progress -- Evidence records (non-secret) and exercise-status tracking +- Evidence records (non-secret), run outcomes and scores, and maturity derived from run history - The legacy adapter for existing consoles - Example and conformance packs diff --git a/docs/maturity-and-projects.md b/docs/maturity-and-projects.md index ee9f408..8ab7062 100644 --- a/docs/maturity-and-projects.md +++ b/docs/maturity-and-projects.md @@ -91,7 +91,7 @@ A **rehearsal** project is where maturity evidence comes from. "Exercised" then - Runs record `project`, the environment class and the pack digest. Scores become per project as well as per runbook. Run records stay non-secret and local (`.local/`), so shared history remains an open question. -- `exercise_status` in `pack.yaml` is superseded. Until this is decided it stays as the interim marker. +- `exercise_status` in `pack.yaml` is retired (RBT-WP-0009): the field is deprecated and ignored, and maturity is the only notion. - Project files are declarative and non-secret. Candidate location: the repository that owns the system (for example `net-kingdom/projects//project.yaml`), next to `runbooks/`. diff --git a/docs/pack-spec-v0.2.md b/docs/pack-spec-v0.2.md index 58bb6b6..bb50483 100644 --- a/docs/pack-spec-v0.2.md +++ b/docs/pack-spec-v0.2.md @@ -187,7 +187,7 @@ Every override, waiver, push-through and change of a value that steps already us `kind` (`override|waiver|push-through`), `item` (step or gate id), `mode`, `unmet` (what was not satisfied), `risk_acknowledged`, `reason` (required for overrides and waivers; automatic text for push-through), `who`, `at`. A run with -deviations is marked so in the overview and can never exercise a pack. A **waiver** is +deviations is marked so in the overview and never counts towards maturity. A **waiver** is declared in advance (`waivable` on a gate); an **override** is not. ## Decisions (2026-09-29, Bernd) @@ -210,7 +210,7 @@ See `docs/maturity-and-projects.md`. Not in v0.2. When a procedure needs another runbook (for example "OpenBao operating path requires the SSH access pack"), packs will declare `requires_packs` with a -required result (exercised, or a named gate passed). Tracked in RBT-WP-0006. +required result (a maturity level, or a named gate passed). Tracked in RBT-WP-0006. ## Recommendation (carried out) diff --git a/docs/pack-spec.md b/docs/pack-spec.md index a5017cf..40a266e 100644 --- a/docs/pack-spec.md +++ b/docs/pack-spec.md @@ -14,7 +14,7 @@ under any root given on the command line. | `id` | yes | Stable dotted id, lower-case, e.g. `nk.ssh-certificates` | | `title`, `outcome` | title yes | What the operator is doing; what is true afterwards | | `owner` | yes | The repository that owns the procedure | -| `exercise_status` | yes | `unexercised` or `exercised YYYY-MM-DD by ` | +| `exercise_status` | no | **Deprecated and ignored.** Maturity is derived from run history; old packs that carry it still load | | `engine` | yes | `native` or `legacy-command` | | `parameters` | no | Operator-supplied, non-secret values | | `prerequisites` | no | `{text, owner}` list | @@ -54,11 +54,10 @@ A step with `risk` of `changes-state`, `attended` or `destructive` must state a Commands are relative to the root of the repository that owns the pack. Verify commands run there, so a pack can call `make` targets and repo-relative scripts. -## Exercise status +## Maturity, not a status field -Following the attended-procedure standard, a pack is `unexercised` until an -operator has completed it, and the operator is recorded. The engine displays the -status prominently and only the engine's evidence writer changes it. +How far a pack can be trusted is earned, not declared: `draft`, `reviewed`, `rehearsed`, `proven`, or `stale`, derived from the +runs of the pack at its current revision (`docs/maturity-and-projects.md`). The earlier `exercise_status` field is retired. ## Secrets diff --git a/docs/run-spec.md b/docs/run-spec.md index 6a44d6f..18d1b29 100644 --- a/docs/run-spec.md +++ b/docs/run-spec.md @@ -18,14 +18,14 @@ One run of one native pack, stored as JSON in - Steps run in order; a step starts only after its predecessors are done or skipped. - A step completes only with a passing verification and its declared evidence. - A verify command runs only when the operator confirms (`--yes`), with a 60 s timeout. -- Only read-only steps may be skipped, and a run with skips never exercises a pack. +- Only read-only steps may be skipped, and a run with skips can never succeed. - Values matching the secret markers are refused everywhere they could be recorded. -- The engine never edits `pack.yaml`. `rtut run finish` shows the exercise-status - diff; `--apply` writes it, and only for a run in which every step is done. +- The engine never edits `pack.yaml`. How far a pack can be trusted is not a field in it: it is derived from run history + (`rtut maturity`, `docs/maturity-and-projects.md`). -CLI: `rtut run new|list|status|cmd|start|verify|done|fail|retry|skip|finish ...` +CLI: `rtut run new|list|status|cmd|start|verify|done|fail|retry|skip|feedback|finalize|capture ...` -## Outcome, score and exercise status +## Outcome and score A run is unfinished (**stalled**) until finalized. `rtut run finalize` (or the UI's *Close this run*) records an `outcome`: @@ -41,11 +41,10 @@ its run records: tried, succeeded, failed, abandoned, stalled (unfinished), plus (`rtut score `; shown in the UI). Run records live in `.local/` of the machine that ran them, so the score is per machine until run histories are shared. -Finalizing never edits the pack by itself. For a `succeeded` run the UI offers **Record pack -as exercised**, which shows what will be written, and on click sets -`exercise_status: exercised by ` in `pack.yaml` (`rtut run finalize ... --outcome -succeeded --apply` does the same). The change is an ordinary edit in the owning repository, to -be reviewed and committed there. +Finalizing never edits the pack. A clean `succeeded` run becomes evidence for the pack's **maturity** at the +revision it ran (`docs/maturity-and-projects.md`); this replaces the earlier `exercise_status` field and the +"Record pack as exercised" step, both retired. Old packs that still carry `exercise_status` load unchanged; the +field is ignored. ## Feedback @@ -96,12 +95,12 @@ Rules the engine enforces: with `evidence` waits for the operator. If checks keep failing until the item's `timeout` (or the pack default), the run stops, or with `on_timeout: push-through` the item is marked done with state `pushed` and a deviation is recorded. -- **Deviations and exercise**: a run with any deviation or pushed-through step can be finalized but never exercises a pack. +- **Deviations and maturity**: a run with any deviation or pushed-through step can be finalized but never counts as maturity evidence. `finalize(succeeded)` also needs every gate passed or waived. - **Pushed is its own state.** A step, gate or datapoint passed over by push-through is `pushed`: not `done`, not `set`, not `err`. Dependents may proceed (that is the point of pushing through), but a pushed datapoint has no value, so a command that needs it fails its check instead of running with nothing. A run with pushed - items can be finished as `failed` (never `succeeded`) and never exercises a pack. + items can be finished as `failed` (never `succeeded`) and never counts as maturity evidence. - **Stalled**: an unfinished run is counted as `stalled` in the score, and as `abandoned` after a week. - **Modes** are `manual` and `auto-confirm`. Auto-confirm applies confirmation only; running the step's own command is a separate, later level ("auto-pilot with execution", RBT-WP-0006). @@ -110,12 +109,12 @@ Rules the engine enforces: Closing a run (`finalize`) makes its record final: -- **Locked.** A closed record cannot be saved with changes: the engine refuses, naming what would have changed. The only later update it - accepts is `exercise_recorded`, written by the legacy "Record pack as exercised" step. This holds for runs closed before sealing existed too. +- **Locked.** A closed record cannot be saved with changes: the engine refuses, naming what would have changed. This holds for runs closed + before sealing existed too. - **Feedback.** It locks with the run. Give it before closing, or with the closing form: `finalize(..., rating, comment)` records an optional rating and comment together with the outcome (validated first, so a refused close changes nothing). This keeps abandoned and failed runs from losing what the operator wanted to say. -- **Sealed.** The record carries a SHA-256 `seal` of its content (excluding `seal`, `updated` and `exercise_recorded`), and the file is made +- **Sealed.** The record carries a SHA-256 `seal` of its content (excluding `seal` and `updated`; a legacy `exercise_recorded` key on old records is ignored), and the file is made read-only. `seal_state` reports `open`, `sealed`, `broken` (changed outside the app after closing) or `unsealed` (closed before sealing existed). A broken seal is called out in the UI, and such a run no longer counts as maturity evidence. This is tamper-evidence, not prevention: someone with file access can still edit a file, but not without being noticed. diff --git a/rtut/__main__.py b/rtut/__main__.py index cdb59f7..0b65efe 100644 --- a/rtut/__main__.py +++ b/rtut/__main__.py @@ -5,6 +5,7 @@ import argparse import sys from pathlib import Path +from . import maturity as M from . import pack as P @@ -47,19 +48,12 @@ def _run(pk, a) -> int: R.skip(r, a.step, a.note); return 0 if a.action == "finalize": R.finalize(r, a.outcome or "", a.note) - if r.data["outcome"] == "succeeded" and a.apply: - print("recorded:", R.apply_exercise(r)) print(r.data["outcome"]); return 0 if a.action == "capture": R.add_capture(r, a.step, sys.stdin.read(), a.command); print("captured"); return 0 if a.action == "feedback": R.add_feedback(r, a.rating or "", a.note); print("recorded"); return 0 - old, new = R.exercise_update(r) - import difflib - print("".join(difflib.unified_diff(old.splitlines(True), new.splitlines(True), "pack.yaml", "pack.yaml"))) - if a.apply: - pk.path.write_text(new); print("written") - return 0 + raise ValueError(f"unknown action {a.action}") except (R.RunError, KeyError, ValueError) as e: print(f"error: {e}", file=sys.stderr) return 1 @@ -125,7 +119,7 @@ def main(argv: list[str] | None = None) -> int: sp = sub.add_parser(name) sp.add_argument("root", nargs="?", default=".") sp = sub.add_parser("run", help="record a run of a native pack") - sp.add_argument("action", choices=["new", "list", "status", "cmd", "start", "verify", "done", "fail", "retry", "skip", "finish", "feedback", "finalize", "capture"]) + sp.add_argument("action", choices=["new", "list", "status", "cmd", "start", "verify", "done", "fail", "retry", "skip", "feedback", "finalize", "capture"]) sp.add_argument("manifest") sp.add_argument("run_id", nargs="?") sp.add_argument("step", nargs="?") @@ -138,7 +132,6 @@ def main(argv: list[str] | None = None) -> int: sp.add_argument("--rating", choices=["good", "neutral", "poor"], help="feedback: how the runbook felt") sp.add_argument("--yes", action="store_true", help="confirm running a verify command") sp.add_argument("--pass", dest="passed", action="store_true", help="manual check: operator confirms it passed") - sp.add_argument("--apply", action="store_true", help="finish: write the exercise-status change") sp = sub.add_parser("ui", help="serve the local UI on 127.0.0.1") sp.add_argument("roots", nargs="*", default=["."]) sp.add_argument("--port", type=int, default=8877) @@ -182,7 +175,6 @@ def main(argv: list[str] | None = None) -> int: if a.cmd == "project": return _project(a) if a.cmd in ("maturity", "review"): - from . import maturity as M pk = P.load(Path(a.manifest)) if not pk.ok: print("\n".join(pk.errors), file=sys.stderr); return 1 @@ -202,7 +194,7 @@ def main(argv: list[str] | None = None) -> int: for p in _packs(a.root): if p.ok and p.data.get("engine") == "native" and p.spec == "runbook-pack/v0.1": c = R.scorecard(p) - print(f"{p.id}\ttried {c['tried']}\tsucceeded {c['succeeded']}\tfailed {c['failed']}\tabandoned {c['abandoned']}\tstalled {c['stalled']}\t[{p.data['exercise_status']}]") + print(f"{p.id}\ttried {c['tried']}\tsucceeded {c['succeeded']}\tfailed {c['failed']}\tabandoned {c['abandoned']}\tstalled {c['stalled']}\t[{M.maturity(p)['level']}]") return 0 if a.cmd in ("list", "validate"): packs = _packs(a.root) @@ -213,7 +205,8 @@ def main(argv: list[str] | None = None) -> int: for p in packs: d = p.data if a.cmd == "list": - print(f"{p.id}\t{d.get('engine')}\t{d.get('exercise_status')}\t{d.get('title')}") + level = M.maturity(p)["level"] if p.ok and d.get("engine") == "native" else "-" + print(f"{p.id}\t{d.get('engine')}\t{level}\t{d.get('title')}") else: print(f"{'ok ' if p.ok else 'FAIL'} {p.id}") for e in p.errors: @@ -246,7 +239,7 @@ def main(argv: list[str] | None = None) -> int: return 0 if a.cmd == "show": d = pk.data - print(f"{d['title']} [{d['exercise_status']}] engine={d['engine']}") + print(f"{d['title']} [{M.maturity(pk)['level']}] engine={d['engine']}") for i, s in enumerate(d.get("steps", []), 1): print(f"{i}. [{s['owner']}] {s['title']} ({s.get('risk', 'read-only')})") print(f" done when: {s['verify']['done_when']}") diff --git a/rtut/pack.py b/rtut/pack.py index 19bc899..02d718a 100644 --- a/rtut/pack.py +++ b/rtut/pack.py @@ -143,7 +143,7 @@ REVIEW_FILE = "REVIEWS.yaml" def digest(pack: "Pack") -> str: """Revision of a pack: hash of every file in its directory, except review records and local state. - `exercise_status:` lines are ignored, so recording an exercise never changes the revision. + `exercise_status:` lines (a deprecated field) are ignored, so dropping it from a pack is not a new revision. """ import hashlib h = hashlib.sha256() diff --git a/rtut/run.py b/rtut/run.py index 7f20534..0973403 100644 --- a/rtut/run.py +++ b/rtut/run.py @@ -44,19 +44,19 @@ class Run: def path(self) -> Path: return runs_dir(self.pack) / f"{self.data['run_id']}.json" - def save(self, allow: tuple = ()) -> None: - """Write the record. A closed run is locked: nothing may change except the keys named in `allow`.""" + def save(self) -> None: + """Write the record. A closed run is locked: nothing may change.""" disk = _read_json(self.path) if disk and disk.get("outcome"): keys = set(disk) | set(self.data) - changed = {k for k in keys if disk.get(k) != self.data.get(k)} - {"updated"} - set(allow) + changed = {k for k in keys if disk.get(k) != self.data.get(k)} - {"updated"} if changed: raise RunError(f"this run is closed and locked ({disk['outcome']}); it cannot be changed " f"({', '.join(sorted(changed))})") self.data["updated"] = _now() self.path.parent.mkdir(parents=True, exist_ok=True) if self.path.exists(): - self.path.chmod(0o644) # a sealed file is read-only; the allowed update reopens it briefly + self.path.chmod(0o644) # a sealed file is read-only; only the closing write reaches here self.path.write_text(json.dumps(self.data, indent=2, sort_keys=True) + "\n") if self.data.get("outcome"): self.path.chmod(0o444) @@ -270,25 +270,6 @@ def skip(run: Run, step_id: str, reason: str) -> None: run.save() -STATUS_LINE = re.compile(r"^exercise_status:.*$", re.M) - - -def exercise_update(run: Run, today: date | None = None) -> tuple[str, str]: - """Return (old, new) pack.yaml text with exercised status. Does not write. - - A run counts only if every step is done (skips do not exercise the pack). - """ - if any(v["state"] != "done" for v in run.data["steps"].values()): - raise RunError("every step must be done (not skipped) to exercise the pack") - if run.data.get("deviations"): - raise RunError("a run with deviations (overrides, waivers, push-through) cannot exercise the pack") - today = today or date.today() - old = run.pack.path.read_text() - new = STATUS_LINE.sub(f"exercise_status: exercised {today.isoformat()} by {run.data['operator']}", - old, count=1) - return old, new - - RATINGS = {"good": ":)", "neutral": ":|", "poor": ":("} @@ -379,7 +360,8 @@ STALE_DAYS = 7 def _seal_of(data: dict) -> str: - """Fingerprint of a closed record. Timestamps of allowed later updates and the exercise marker are not part of it.""" + """Fingerprint of a closed record. `updated` is not part of it, nor is `exercise_recorded`, which records closed + before exercise recording was retired may still carry.""" import hashlib body = {k: v for k, v in data.items() if k not in ("seal", "updated", "exercise_recorded")} return hashlib.sha256(json.dumps(body, sort_keys=True).encode()).hexdigest() @@ -503,16 +485,3 @@ def scorecard(pack: Pack, now: datetime | None = None, stale_days: int = STALE_D card["fastest_seconds"] = durations[0] if durations else None card["timed_runs"] = len(durations) return card - - -def apply_exercise(run: Run, today: date | None = None) -> str: - """Write 'exercised by ' into pack.yaml for a succeeded run. Returns the new status.""" - if run.data.get("outcome") != "succeeded": - raise RunError("only a run finished as succeeded can exercise the pack") - _, new = exercise_update(run, today) - run.pack.path.write_text(new) - status = STATUS_LINE.search(new).group(0).split(": ", 1)[1] - run.pack.data["exercise_status"] = status - run.data["exercise_recorded"] = status - run.save(allow=("exercise_recorded",)) # the one thing a closed run may still receive - return status diff --git a/rtut/ui.py b/rtut/ui.py index f604487..61b90ed 100644 --- a/rtut/ui.py +++ b/rtut/ui.py @@ -260,7 +260,7 @@ class App: if x.get("values") else f"") + f"
{e(x.get('help', ''))} Never enter secrets.

" for x in d.get("parameters", [])) - return page(d["title"], f"

{e(d['title'])}

{e(d['exercise_status'])}

" + return page(d["title"], f"

{e(d['title'])}

{UP.maturity_pill(p)}

" f"

{e(d.get('outcome', ''))}

{e(score_line(R.scorecard(p)))}

" f"{UP.maturity_block(self, i, p)}{self.runs_list(i, p)}

Prerequisites

    {prereq}
" f"
{self.hidden()}{fields}" @@ -302,17 +302,6 @@ class App: if outcome: note = f" - {e(r.data['outcome_note'])}" if r.data.get("outcome_note") else "" box = f"

Run finished: {e(outcome)}{note}

{self.locked_note(r)}" - if outcome == "succeeded": - if r.data.get("exercise_recorded"): - box += f"

Pack recorded as {e(r.data['exercise_recorded'])}. Commit the change in the owning repository.

" - else: - old, new = R.exercise_update(r) - line = new.split("exercise_status:", 1)[1].split("\n", 1)[0].strip() - box += (f"

Current pack status: {e(p.data['exercise_status'])}. " - f"Recording this run would set it to {e(line)} in {e(p.path)} " - "(a reviewable edit; commit it in the owning repository).

" - f"{self.hidden()}" - "
") return f"

Close this run

{box}
" states = [v["state"] for v in r.data["steps"].values()] btns = [] @@ -383,7 +372,7 @@ class App: where = (f" · project {e(ctx['project'])}" f"{' @' + e(ctx['target']) if ctx.get('target') else ''} · {e(ctx.get('environment', ''))}" f"{' · part of project run ' + e(ctx['project_run']) + '' if ctx.get('project_run') else ''}") - out = [f"

{e(d['title'])}

{e(d['exercise_status'])}

" + out = [f"

{e(d['title'])}

{UP.maturity_pill(p)}

" f"

run {e(rid)} · operator {e(r.data['operator'])}{where}

" f"

{UI2.times_line(r)}

"] step_secs = R.run_times(r)["step_seconds"] @@ -607,15 +596,10 @@ def make_handler(app: App, port_holder: list): i, rid = int(parts[1]), parts[2] R.add_feedback(R.load(app.packs[i], rid), form.get("rating", ""), form.get("comment", "")) return self._redirect(f"/run/{i}/{rid}#feedback") - if len(parts) == 4 and parts[0] == "run" and parts[3] in ("finalize", "exercise"): + if len(parts) == 4 and parts[0] == "run" and parts[3] == "finalize": i, rid = int(parts[1]), parts[2] - r = R.load(app.packs[i], rid) - if parts[3] == "finalize": - R.finalize(r, form.get("outcome", ""), form.get("note", ""), rating=form.get("rating") or None, - comment=form.get("fb_comment", "")) - else: - R.apply_exercise(r) - app.packs[i] = P.load(app.packs[i].path) # pick up the new exercise status + R.finalize(R.load(app.packs[i], rid), form.get("outcome", ""), form.get("note", ""), + rating=form.get("rating") or None, comment=form.get("fb_comment", "")) return self._redirect(f"/run/{i}/{rid}#close") i, rid, sid, action = int(parts[1]), parts[2], parts[3], parts[4] p = app.packs[i]; r = R.load(p, rid) diff --git a/runbooks/example-hello/pack.yaml b/runbooks/example-hello/pack.yaml index 7a00553..7b277b4 100644 --- a/runbooks/example-hello/pack.yaml +++ b/runbooks/example-hello/pack.yaml @@ -3,7 +3,6 @@ id: rtut.example-hello title: Example - check a host answers owner: runbook-tutorials outcome: You have proven a host answers on the network. -exercise_status: exercised 2026-09-29 by Bernd engine: native parameters: - {id: host, label: Host name, type: string, default: localhost, pattern: "[a-zA-Z0-9.-]+"} diff --git a/runbooks/showcase-lighthouse-v2/pack.yaml b/runbooks/showcase-lighthouse-v2/pack.yaml index f446b35..63dadc6 100644 --- a/runbooks/showcase-lighthouse-v2/pack.yaml +++ b/runbooks/showcase-lighthouse-v2/pack.yaml @@ -5,7 +5,6 @@ id: rtut.showcase-lighthouse-v2 title: "Showcase: a night at the lighthouse (v0.2)" owner: runbook-tutorials outcome: You have played a short interactive story that touches every feature of the runbook engine, using only harmless echo, printf and a scratch file under /tmp/rtut-. -exercise_status: unexercised engine: native confirm_phrase: I understand this cannot be undone diff --git a/runbooks/showcase-lighthouse/pack.yaml b/runbooks/showcase-lighthouse/pack.yaml index cc9520b..1ea7e08 100644 --- a/runbooks/showcase-lighthouse/pack.yaml +++ b/runbooks/showcase-lighthouse/pack.yaml @@ -3,7 +3,6 @@ id: rtut.showcase-lighthouse title: "Showcase: a night at the lighthouse" owner: runbook-tutorials outcome: You have played a short interactive story that touches every feature of the runbook engine, using only harmless echo, printf and a scratch file. Nothing outside /tmp/rtut-* is touched. -exercise_status: unexercised engine: native parameters: - {id: keeper, label: Your keeper's name, type: string, default: Ada, pattern: "[A-Za-z]+", help: One word. Used in the story.} diff --git a/schema/runbook-pack-v0.2.schema.json b/schema/runbook-pack-v0.2.schema.json index 61f1b6b..c7bf5f4 100644 --- a/schema/runbook-pack-v0.2.schema.json +++ b/schema/runbook-pack-v0.2.schema.json @@ -8,7 +8,6 @@ "id", "title", "owner", - "exercise_status", "engine" ], "additionalProperties": false, @@ -33,7 +32,8 @@ }, "exercise_status": { "type": "string", - "pattern": "^(unexercised|exercised [0-9]{4}-[0-9]{2}-[0-9]{2} by \\S+)$" + "deprecated": true, + "description": "Deprecated and ignored. Maturity is derived from run history (docs/maturity-and-projects.md); old packs that still carry this field keep loading." }, "engine": { "enum": [ diff --git a/schema/runbook-pack.schema.json b/schema/runbook-pack.schema.json index cada4c4..e551de6 100644 --- a/schema/runbook-pack.schema.json +++ b/schema/runbook-pack.schema.json @@ -3,96 +3,292 @@ "$id": "https://runbook-tutorials.invalid/schema/runbook-pack/v0.1", "title": "Runbook pack manifest", "type": "object", - "required": ["spec", "id", "title", "owner", "exercise_status", "engine"], + "required": [ + "spec", + "id", + "title", + "owner", + "engine" + ], "additionalProperties": false, "properties": { - "spec": {"const": "runbook-pack/v0.1"}, - "id": {"type": "string", "pattern": "^[a-z0-9]+([.-][a-z0-9]+)*$"}, - "title": {"type": "string", "minLength": 3}, - "owner": {"type": "string", "minLength": 1, "description": "Repository that owns the procedure"}, - "outcome": {"type": "string"}, + "spec": { + "const": "runbook-pack/v0.1" + }, + "id": { + "type": "string", + "pattern": "^[a-z0-9]+([.-][a-z0-9]+)*$" + }, + "title": { + "type": "string", + "minLength": 3 + }, + "owner": { + "type": "string", + "minLength": 1, + "description": "Repository that owns the procedure" + }, + "outcome": { + "type": "string" + }, "exercise_status": { "type": "string", - "pattern": "^(unexercised|exercised [0-9]{4}-[0-9]{2}-[0-9]{2} by \\S+)$" + "deprecated": true, + "description": "Deprecated and ignored. Maturity is derived from run history (docs/maturity-and-projects.md); old packs that still carry this field keep loading." + }, + "engine": { + "enum": [ + "native", + "legacy-command" + ] }, - "engine": {"enum": ["native", "legacy-command"]}, "legacy": { "type": "object", - "required": ["command"], + "required": [ + "command" + ], "additionalProperties": false, "properties": { - "command": {"type": "array", "items": {"type": "string"}, "minItems": 1}, - "cwd": {"type": "string"}, - "url": {"type": "string"}, - "note": {"type": "string"} + "command": { + "type": "array", + "items": { + "type": "string" + }, + "minItems": 1 + }, + "cwd": { + "type": "string" + }, + "url": { + "type": "string" + }, + "note": { + "type": "string" + } } }, - "parameters": {"type": "array", "items": {"$ref": "#/$defs/parameter"}}, - "prerequisites": {"type": "array", "items": {"$ref": "#/$defs/prerequisite"}}, - "steps": {"type": "array", "items": {"$ref": "#/$defs/step"}}, - "threat_checks": {"type": "array", "items": {"type": "string"}}, - "ownership": {"type": "array", "items": {"$ref": "#/$defs/ownership"}} + "parameters": { + "type": "array", + "items": { + "$ref": "#/$defs/parameter" + } + }, + "prerequisites": { + "type": "array", + "items": { + "$ref": "#/$defs/prerequisite" + } + }, + "steps": { + "type": "array", + "items": { + "$ref": "#/$defs/step" + } + }, + "threat_checks": { + "type": "array", + "items": { + "type": "string" + } + }, + "ownership": { + "type": "array", + "items": { + "$ref": "#/$defs/ownership" + } + } }, "allOf": [ - {"if": {"properties": {"engine": {"const": "native"}}}, - "then": {"required": ["steps", "threat_checks"]}}, - {"if": {"properties": {"engine": {"const": "legacy-command"}}}, - "then": {"required": ["legacy"]}} + { + "if": { + "properties": { + "engine": { + "const": "native" + } + } + }, + "then": { + "required": [ + "steps", + "threat_checks" + ] + } + }, + { + "if": { + "properties": { + "engine": { + "const": "legacy-command" + } + } + }, + "then": { + "required": [ + "legacy" + ] + } + } ], "$defs": { "parameter": { "type": "object", - "required": ["id", "label", "type"], + "required": [ + "id", + "label", + "type" + ], "additionalProperties": false, "properties": { - "id": {"type": "string", "pattern": "^[a-z][a-z0-9_]*$"}, - "label": {"type": "string"}, - "type": {"enum": ["string", "text", "enum", "path", "bool", "int"]}, - "values": {"type": "array", "items": {"type": "string"}}, - "default": {"type": ["string", "integer", "boolean"]}, - "pattern": {"type": "string"}, - "help": {"type": "string"}, - "secret": {"const": false, "description": "Parameters are never secret. Secrets are entered by the operator into the tool that owns them, never into the engine."} + "id": { + "type": "string", + "pattern": "^[a-z][a-z0-9_]*$" + }, + "label": { + "type": "string" + }, + "type": { + "enum": [ + "string", + "text", + "enum", + "path", + "bool", + "int" + ] + }, + "values": { + "type": "array", + "items": { + "type": "string" + } + }, + "default": { + "type": [ + "string", + "integer", + "boolean" + ] + }, + "pattern": { + "type": "string" + }, + "help": { + "type": "string" + }, + "secret": { + "const": false, + "description": "Parameters are never secret. Secrets are entered by the operator into the tool that owns them, never into the engine." + } } }, "prerequisite": { "type": "object", - "required": ["text", "owner"], + "required": [ + "text", + "owner" + ], "additionalProperties": false, - "properties": {"text": {"type": "string"}, "owner": {"type": "string"}} + "properties": { + "text": { + "type": "string" + }, + "owner": { + "type": "string" + } + } }, "check": { "type": "object", - "required": ["done_when"], + "required": [ + "done_when" + ], "additionalProperties": false, "properties": { - "done_when": {"type": "string", "minLength": 3}, - "command": {"type": "string"}, - "expect": {"enum": ["manual", "exit-0", "output-contains"]}, - "contains": {"type": "string"} + "done_when": { + "type": "string", + "minLength": 3 + }, + "command": { + "type": "string" + }, + "expect": { + "enum": [ + "manual", + "exit-0", + "output-contains" + ] + }, + "contains": { + "type": "string" + } } }, "step": { "type": "object", - "required": ["id", "title", "owner", "verify"], + "required": [ + "id", + "title", + "owner", + "verify" + ], "additionalProperties": false, "properties": { - "id": {"type": "string", "pattern": "^[a-z][a-z0-9-]*$"}, - "title": {"type": "string"}, - "owner": {"type": "string"}, - "description": {"type": "string"}, - "command": {"type": "string", "description": "Template; {{param}} placeholders only"}, - "risk": {"enum": ["read-only", "changes-state", "attended", "destructive"], "default": "read-only"}, - "verify": {"$ref": "#/$defs/check"}, - "rollback": {"type": "string"}, - "evidence": {"type": "array", "items": {"type": "string"}, "description": "Names of non-secret facts recorded when the step completes"} + "id": { + "type": "string", + "pattern": "^[a-z][a-z0-9-]*$" + }, + "title": { + "type": "string" + }, + "owner": { + "type": "string" + }, + "description": { + "type": "string" + }, + "command": { + "type": "string", + "description": "Template; {{param}} placeholders only" + }, + "risk": { + "enum": [ + "read-only", + "changes-state", + "attended", + "destructive" + ], + "default": "read-only" + }, + "verify": { + "$ref": "#/$defs/check" + }, + "rollback": { + "type": "string" + }, + "evidence": { + "type": "array", + "items": { + "type": "string" + }, + "description": "Names of non-secret facts recorded when the step completes" + } } }, "ownership": { "type": "object", - "required": ["concern", "owner"], + "required": [ + "concern", + "owner" + ], "additionalProperties": false, - "properties": {"concern": {"type": "string"}, "owner": {"type": "string"}} + "properties": { + "concern": { + "type": "string" + }, + "owner": { + "type": "string" + } + } } } } diff --git a/tests/test_cli.py b/tests/test_cli.py new file mode 100644 index 0000000..42d251d --- /dev/null +++ b/tests/test_cli.py @@ -0,0 +1,99 @@ +"""The command line, end to end. Each command runs through main() against a throwaway repository.""" +import shutil +import sys +from pathlib import Path + +import pytest +import yaml + +from rtut.__main__ import main + +ROOT = Path(__file__).resolve().parent.parent + + +@pytest.fixture +def repo(tmp_path): + for name, src in (("v1", "showcase-lighthouse"), ("v2", "showcase-lighthouse-v2")): + d = tmp_path / "runbooks" / name; d.mkdir(parents=True) + shutil.copy(ROOT / f"runbooks/{src}/pack.yaml", d / "pack.yaml") + (tmp_path / "attribute-sets").mkdir() + shutil.copy(ROOT / "attribute-sets/harbour-defaults.yaml", tmp_path / "attribute-sets/harbour-defaults.yaml") + (tmp_path / "project-templates").mkdir() + shutil.copy(ROOT / "project-templates/lighthouse-watch.yaml", tmp_path / "project-templates/lighthouse-watch.yaml") + return tmp_path + + +def run(capsys, *argv): + code = main([str(a) for a in argv]) + out = capsys.readouterr() + return code, out.out, out.err + + +def test_validate_and_list_show_maturity_not_exercise_status(repo, capsys): + code, out, _ = run(capsys, "validate", repo) + assert code == 0 and out.count("ok ") == 2 + code, out, _ = run(capsys, "list", repo) + assert code == 0 and "rtut.showcase-lighthouse\tnative\tdraft\t" in out and "exercise" not in out + code, out, _ = run(capsys, "show", repo / "runbooks/v1/pack.yaml") + assert code == 0 and "[draft]" in out and "engine=native" in out + + +def test_render_and_score_and_maturity(repo, capsys): + code, out, _ = run(capsys, "render", repo / "runbooks/v1/pack.yaml", "meet-the-keeper", "-p", "keeper=Bea") + assert code == 0 and out.strip().startswith('echo "Bea climbs the stairs') + code, out, _ = run(capsys, "score", repo) + assert code == 0 and "rtut.showcase-lighthouse\ttried 0" in out and out.rstrip().endswith("[draft]") + code, out, _ = run(capsys, "maturity", repo / "runbooks/v1/pack.yaml") + assert code == 0 and out.startswith("rtut.showcase-lighthouse: draft") and "reviewed: False" in out + + +def test_review_then_maturity(repo, capsys): + pack = repo / "runbooks/v1/pack.yaml" + assert run(capsys, "review", pack, "--by", "alice", "--note", "read it")[0] == 0 + code, out, _ = run(capsys, "maturity", pack) + assert "rtut.showcase-lighthouse: reviewed" in out and "reviewed: True" in out + code, _, err = run(capsys, "review", pack, "--by", "two words") + assert code == 1 and "one word" in err + + +def test_run_commands_walk_a_v01_pack(repo, capsys): + pack = repo / "runbooks/v1/pack.yaml" + code, rid, _ = run(capsys, "run", "new", pack, "-o", "bernd", "-p", "logdir=/tmp/rtut-cli-x") + rid = rid.strip() + assert code == 0 and rid + assert run(capsys, "run", "start", pack, rid, "read-the-sign")[0] == 0 + assert run(capsys, "run", "verify", pack, rid, "read-the-sign", "--pass")[0] == 0 + assert run(capsys, "run", "done", pack, rid, "read-the-sign")[0] == 0 + code, out, _ = run(capsys, "run", "status", pack, rid) + assert "done read-the-sign" in out and "todo meet-the-keeper" in out + code, out, _ = run(capsys, "run", "finalize", pack, rid, "--outcome", "abandoned", "--note", "cli test") + assert code == 0 and out.strip() == "abandoned" + code, _, err = run(capsys, "run", "start", pack, rid, "meet-the-keeper") + assert code == 1 and "finished" in err + assert run(capsys, "run", "feedback", pack, rid, "--rating", "poor", "--note", "x")[0] == 1 # closed runs are locked + + +def test_the_retired_finish_action_is_gone(repo): + with pytest.raises(SystemExit): + main(["run", "finish", str(repo / "runbooks/v1/pack.yaml"), "x"]) + + +def test_project_commands(repo, capsys): + code, out, _ = run(capsys, "project", "list", repo) + assert code == 0 and "lighthouse-watch\ttemplate" in out + code, out, _ = run(capsys, "project", "new", repo, "lighthouse-watch", "demo", "-e", "rehearsal", "--title", "Demo watch", + "--description", "A demo", "-s", "keeper=Ada", "-s", "logdir=/tmp/rtut-cli-p") + assert code == 0 and out.strip().endswith("projects/demo/project.yaml") + made = yaml.safe_load((repo / "projects/demo/project.yaml").read_text()) + assert made["title"] == "Demo watch" and made["description"] == "A demo" and made["template"] == "lighthouse-watch" + # the template's example members need packs this fixture does not have, so validation reports it rather than crashing + code, out, _ = run(capsys, "project", "validate", repo, "demo") + assert code == 1 and "FAIL demo" in out and "rtut.example-hello not found" in out + code, _, err = run(capsys, "project", "status", repo, "nope") + assert code == 1 and "no project nope" in err + + +def test_ui_and_launch_arguments_parse(repo): + from rtut.__main__ import main as m + with pytest.raises(SystemExit): + m(["ui", "--port", "not-a-port"]) diff --git a/tests/test_legacy.py b/tests/test_legacy.py index b067a46..8fdc417 100644 --- a/tests/test_legacy.py +++ b/tests/test_legacy.py @@ -17,7 +17,7 @@ def make_repo(tmp_path, command, url, cwd="."): d.mkdir(parents=True) (d / "pack.yaml").write_text(yaml.safe_dump({ "spec": "runbook-pack/v0.1", "id": "t.old", "title": "Old console", "owner": "t", - "exercise_status": "unexercised", "engine": "legacy-command", + "engine": "legacy-command", "legacy": {"command": command, "cwd": cwd, "url": url}})) return P.load(d / "pack.yaml") diff --git a/tests/test_maturity.py b/tests/test_maturity.py index 8d74665..e223daf 100644 --- a/tests/test_maturity.py +++ b/tests/test_maturity.py @@ -33,8 +33,10 @@ def test_digest_is_stable_ignores_exercise_status_and_reviews_but_changes_with_c d1 = P.digest(pk) assert d1 == P.digest(pk) text = pk.path.read_text() - pk.path.write_text(text.replace("exercise_status: unexercised", "exercise_status: exercised 2026-09-29 by x")) - assert P.digest(pk) == d1 # recording an exercise is not a new revision + assert "engine: native\n" in text + pk.path.write_text(text.replace("engine: native\n", "exercise_status: exercised 2026-09-29 by x\nengine: native\n", 1)) # as an old pack carries it + assert P.digest(pk) == d1 # the deprecated field is ignored: dropping it is not a new revision + pk.path.write_text(text) M.add_review(pk, "alice", "read it") assert P.digest(pk) == d1 # neither is a review pk.path.write_text(text + "\n# changed\n") diff --git a/tests/test_pack.py b/tests/test_pack.py index 3c4e86b..52c2983 100644 --- a/tests/test_pack.py +++ b/tests/test_pack.py @@ -29,8 +29,6 @@ def mut(fn): @pytest.mark.parametrize("fn,frag", [ - (lambda d: d.pop("exercise_status"), "exercise_status"), - (lambda d: d.update(exercise_status="done"), "exercise_status"), (lambda d: d["parameters"][0].update(secret=True), "secret"), (lambda d: d["steps"][0].pop("verify"), "verify"), (lambda d: d["steps"][0].update(command="x {{nope}}"), "undefined placeholder"), @@ -49,7 +47,7 @@ def test_invalid(tmp_path, fn, frag): def test_legacy_requires_command(tmp_path): d = {"spec": "runbook-pack/v0.1", "id": "x.y", "title": "Legacy x", "owner": "o", - "exercise_status": "unexercised", "engine": "legacy-command"} + "engine": "legacy-command"} assert not check(tmp_path, d).ok d["legacy"] = {"command": ["true"]} assert check(tmp_path, d).ok @@ -74,3 +72,13 @@ def test_text_parameter_allows_spaces_but_not_metacharacters(): with pytest.raises(ValueError): P.render(d["steps"][0]["command"], {"host": "two words"}, pk) + + +def test_the_deprecated_exercise_status_is_accepted_and_ignored_so_old_packs_still_load(tmp_path): + d = copy.deepcopy(GOOD) + d["exercise_status"] = "exercised 2026-01-01 by someone" + assert check(tmp_path, d).ok + d["exercise_status"] = "whatever text" # deprecated: not validated, not used + assert check(tmp_path, d).ok + d.pop("exercise_status") + assert check(tmp_path, d).ok # and it is not required any more diff --git a/tests/test_projectrun.py b/tests/test_projectrun.py index 55cb348..db4430b 100644 --- a/tests/test_projectrun.py +++ b/tests/test_projectrun.py @@ -19,7 +19,7 @@ def v2_pack(pid, extra_step=None): {"id": "second", "title": "Second step", "actor": "op", "owner": "t", "verify": {"done_when": "second done", "expect": "manual"}}] if extra_step: steps.append(extra_step) - return {"spec": "runbook-pack/v0.2", "id": pid, "title": f"Pack {pid}", "owner": "t", "exercise_status": "unexercised", "engine": "native", + return {"spec": "runbook-pack/v0.2", "id": pid, "title": f"Pack {pid}", "owner": "t", "engine": "native", "framing": ["keep in mind"], "actors": [{"id": "op", "role": "Operator", "description": "does it"}], "datapoints": [{"id": "site", "label": "Site", "type": "string", "pattern": "[a-z]+", "default": "north", "ask_in": "main"}], "stages": [{"id": "main", "title": "Main", "goal": "do the things", "steps": steps}]} diff --git a/tests/test_run.py b/tests/test_run.py index c6bb628..f3bc62a 100644 --- a/tests/test_run.py +++ b/tests/test_run.py @@ -51,9 +51,7 @@ def test_full_run_and_exercise(pk): pytest.skip("ping unavailable in this environment") R.complete(r, "ping") assert r.complete - old, new = R.exercise_update(r, date(2026, 9, 29)) - assert "exercise_status: exercised 2026-09-29 by bernd" in new and old != new - assert pk.path.read_text() == old # never written by the engine + assert pk.path.read_text() == P.load(pk.path).path.read_text() # the engine never writes a pack def test_order_fail_retry_skip(pk): @@ -69,8 +67,8 @@ def test_order_fail_retry_skip(pk): R.fail(r, "ping", "-----BEGIN RSA PRIVATE KEY-----") r2 = R.create(pk, "bernd") R.skip(r2, "ping", "not needed") - with pytest.raises(R.RunError): - R.exercise_update(r2) # skips never exercise a pack + with pytest.raises(R.RunError, match="succeeds only when every step is done"): + R.finalize(r2, "succeeded") # a skipped step never counts as done def test_manual_verification_needs_answer(pk): diff --git a/tests/test_run2.py b/tests/test_run2.py index e8f248e..4bb2ef3 100644 --- a/tests/test_run2.py +++ b/tests/test_run2.py @@ -60,6 +60,14 @@ def pass_gate(r, gid): R.gate_check(r, gid, i, confirm=True, manual_ok=True) +def counts_towards_maturity(r) -> bool: + """Would this run be clean maturity evidence? (succeeded, no deviation, with an environment)""" + from rtut import maturity as M + r.data["outcome"] = "succeeded" + r.data["context"] = {"project": "p", "environment": "rehearsal"} + return M._evidence(r.pack, r.data.get("pack_digest"), [r])["clean_runs"] == 1 + + # ---- datapoints ----------------------------------------------------------------------------------- def test_create_sets_datapoint_states(tmp_path): @@ -165,15 +173,11 @@ def test_waiver_only_where_declared_with_reason_and_recorded(tmp_path): R.waive(r, "night-on-record", "try") -def test_deviations_block_exercise_but_not_finalize(tmp_path): +def test_deviations_keep_a_run_out_of_maturity(tmp_path): r = new(tmp_path) R.start(r, "warn", alt="sound-horn", override_reason="probe", acknowledge_risk=True) - r.data["steps"]["warn"]["state"] = "failed" - assert r.data["deviations"] - with pytest.raises(R1.RunError, match="deviations"): - for k in r.data["steps"]: - r.data["steps"][k]["state"] = "done" - R1.exercise_update(r) + assert r.data["deviations"] and not counts_towards_maturity(r) + assert counts_towards_maturity(new(tmp_path)) # the same without a deviation would count # ---- confirmation styles ------------------------------------------------------------------------------ @@ -264,10 +268,7 @@ def test_auto_confirm_push_through_records_deviation_and_moves_on(tmp_path): assert dev["kind"] == "push-through" and dev["mode"] == "auto-confirm" and dev["risk_acknowledged"] is False nxt = R.auto_confirm_tick(r, late) # continues to the manual-only step assert nxt["item"] == "close-log" and nxt["action"] == "awaiting-operator" - with pytest.raises(R1.RunError, match="deviations"): - for k in r.data["steps"]: - r.data["steps"][k]["state"] = "done" - R1.exercise_update(r) + assert not counts_towards_maturity(r) # a pushed-through run is not evidence def test_auto_confirm_gate_push_through_and_stop(tmp_path): @@ -460,8 +461,4 @@ def test_a_run_that_changed_a_used_value_never_counts_towards_maturity(tmp_path) r = new(tmp_path) do_step(r, "read-the-sky") R.set_datapoint(r, "weather", "clear", override_reason="changed my mind", acknowledge_risk=True) - assert r.data["deviations"] - with pytest.raises(R1.RunError, match="deviations"): - for k in r.data["steps"]: - r.data["steps"][k]["state"] = "done" - R1.exercise_update(r) + assert r.data["deviations"] and not counts_towards_maturity(r) diff --git a/tests/test_score.py b/tests/test_score.py index 548974c..b2df03f 100644 --- a/tests/test_score.py +++ b/tests/test_score.py @@ -64,17 +64,6 @@ def test_scorecard_counts_every_outcome_and_stale_open_runs(pk): assert c["feedback"]["good"] == 1 -def test_finalizing_success_can_record_exercise_status(pk): - r = R.create(pk, "bernd"); play_all(r) - with pytest.raises(R.RunError): - R.apply_exercise(r) # not finalized yet - R.finalize(r, "succeeded") - status = R.apply_exercise(r) - assert status.startswith("exercised ") and status.endswith(" by bernd") - reloaded = P.load(pk.path) - assert reloaded.ok and reloaded.data["exercise_status"] == status - - def test_capture_documentation_and_limits(pk): r = R.create(pk, "bernd") with pytest.raises(R.RunError, match="start the step"): @@ -166,7 +155,7 @@ def test_closing_records_feedback_with_the_outcome_and_validates_before_changing assert R.load(pk, r.data["run_id"]).data["feedback"][0]["rating"] == "poor" -def test_a_closed_record_cannot_be_saved_with_changes_but_the_exercise_marker_is_allowed(pk): +def test_a_closed_record_cannot_be_saved_with_changes(pk): r = R.create(pk, "bernd"); R.finalize(r, "abandoned") r2 = R.load(pk, r.data["run_id"]) r2.data["steps"]["read-the-sign"]["state"] = "done" @@ -177,10 +166,7 @@ def test_a_closed_record_cannot_be_saved_with_changes_but_the_exercise_marker_is with pytest.raises(R.RunError, match="operator"): r3.save() assert R.load(pk, r.data["run_id"]).data["operator"] == "bernd" # nothing reached the disk - r4 = R.load(pk, r.data["run_id"]) - r4.data["exercise_recorded"] = "x" - r4.save(allow=("exercise_recorded",)) # the one permitted later update - assert R.load(pk, r.data["run_id"]).data["exercise_recorded"] == "x" and R.seal_state(R.load(pk, r.data["run_id"])) == "sealed" + assert R.seal_state(R.load(pk, r.data["run_id"])) == "sealed" # and the seal still holds def test_closing_seals_the_file_read_only_and_tampering_is_detected(pk): diff --git a/tests/test_showcase.py b/tests/test_showcase.py index 68cb533..c26080e 100644 --- a/tests/test_showcase.py +++ b/tests/test_showcase.py @@ -35,8 +35,8 @@ def test_scripted_playthrough(pk): R.complete(r, sid, {n: "1" for n in s.get("evidence", [])}) assert r.complete assert not Path(r.data["params"]["logdir"]).exists() - old, new = R.exercise_update(r) - assert "exercised" in new + R.finalize(r, "succeeded") # every step done: the run may succeed + assert r.data["outcome"] == "succeeded" def test_bad_logdir_rejected(pk): diff --git a/tests/test_ui.py b/tests/test_ui.py index 88bd957..200d48a 100644 --- a/tests/test_ui.py +++ b/tests/test_ui.py @@ -245,17 +245,11 @@ def test_score_runs_list_and_finalize_flow(srv): assert req(srv, "POST", base + "/finalize", {"_t": t, "outcome": "succeeded", "note": "smooth"})[0] == 303 _, _, body = req(srv, "GET", base) assert "Run finished: succeeded" in body and "smooth" in body - assert "Record pack as exercised" in body and "would set it to" in body # explicit, with the diff in words + assert "Record pack as exercised" not in body and "would set it to" not in body # retired: maturity is derived from runs assert "Start this step" not in body # read-only now _, _, body = req(srv, "GET", "/") assert "succeeded 1" in body - # exercised status is only written on the explicit click - pack_file = srv.app.packs[i].path - assert "exercise_status: unexercised" in pack_file.read_text() - assert req(srv, "POST", base + "/exercise", {"_t": t})[0] == 303 - assert "exercise_status: exercised" in pack_file.read_text() - _, _, body = req(srv, "GET", base) - assert "Pack recorded as" in body and "Record pack as exercised" not in body + assert req(srv, "POST", base + "/exercise", {"_t": t})[0] == 400 # the route is gone def test_abandon_and_actions_refused_after_finish(srv): @@ -264,7 +258,6 @@ def test_abandon_and_actions_refused_after_finish(srv): assert req(srv, "POST", base + "/finalize", {"_t": t, "outcome": "abandoned"})[0] == 303 st, _, body = req(srv, "POST", base + "/read-the-sign/start", {"_t": t}) assert st == 400 and "finished" in body - assert req(srv, "POST", base + "/exercise", {"_t": t})[0] == 400 # only a succeeded run may exercise the pack def test_capture_and_pasted_output_check(srv): diff --git a/tests/test_ui_projectrun.py b/tests/test_ui_projectrun.py index c2a949d..f2aaa1b 100644 --- a/tests/test_ui_projectrun.py +++ b/tests/test_ui_projectrun.py @@ -19,7 +19,7 @@ def v2_pack(pid, manual_only_last=False): if manual_only_last: steps.append({"id": "careful", "title": "Careful step", "actor": "op", "owner": "t", "auto_confirm": "manual-only", "verify": {"done_when": "you did it", "expect": "manual"}}) - return {"spec": "runbook-pack/v0.2", "id": pid, "title": f"Pack {pid}", "owner": "t", "exercise_status": "unexercised", "engine": "native", + return {"spec": "runbook-pack/v0.2", "id": pid, "title": f"Pack {pid}", "owner": "t", "engine": "native", "framing": ["keep in mind"], "actors": [{"id": "op", "role": "Operator", "description": "does it"}], "datapoints": [{"id": "site", "label": "Site", "type": "string", "pattern": "[a-z]+", "default": "north", "ask_in": "main"}], "stages": [{"id": "main", "title": "Main", "goal": "do the things", "steps": steps}]} diff --git a/tests/test_v2.py b/tests/test_v2.py index d27836e..72c1b52 100644 --- a/tests/test_v2.py +++ b/tests/test_v2.py @@ -106,7 +106,7 @@ def test_three_levels_allowed(): def test_legacy_v02_needs_only_schema(): d = {"spec": "runbook-pack/v0.2", "id": "x.y", "title": "Legacy x", "owner": "o", - "exercise_status": "unexercised", "engine": "legacy-command", "legacy": {"command": ["true"]}} + "engine": "legacy-command", "legacy": {"command": ["true"]}} assert v2.validate(d) == [] @@ -133,3 +133,10 @@ def test_idempotency_is_declared_validated_and_summarized(): assert flow.pack_idempotency(d) == "unknown" # one undeclared step makes the whole unknown assert errs(lambda x: step(x, "warn").update(idempotent="sometimes")) assert errs(lambda x: step(x, "warn").update(idempotent=True)) # a YAML "yes" is refused, not silently accepted + + +def test_v02_accepts_the_deprecated_exercise_status_and_does_not_require_it(): + d = copy.deepcopy(GOOD) + assert "exercise_status" not in d and v2.validate(d) == [] + d["exercise_status"] = "exercised 2026-01-01 by someone" + assert v2.validate(d) == [] diff --git a/workplans/RBT-WP-0009-finalize-the-runbook-tutorials-workplans.md b/workplans/RBT-WP-0009-finalize-the-runbook-tutorials-workplans.md index 853c22a..714c40a 100644 --- a/workplans/RBT-WP-0009-finalize-the-runbook-tutorials-workplans.md +++ b/workplans/RBT-WP-0009-finalize-the-runbook-tutorials-workplans.md @@ -27,13 +27,14 @@ backlog and do only a small JSON export. ```task id: RBT-WP-0009-T01 -status: todo +status: done priority: high ``` -`exercise_status` mixed three things (RBT-WP-0008). Maturity, derived from run history, replaces it. Remove the "Record pack as +Done 2026-09-29. `exercise_status` mixed three things (RBT-WP-0008). Maturity, derived from run history, replaces it. Remove the "Record pack as exercised" click and `apply_exercise` / `exercise_update`; drop the field from the shipped packs (both repositories) and from the UI headers; make it optional and deprecated in both schemas so old packs still load; show maturity where the status was shown. +Also found and fixed while doing it: the CLI had no tests, and `rtut list` crashed from a shadowed import; `tests/test_cli.py` now covers it. ## Packaging, checks and CI