From 0203bf553c99459effa251c71093224861e39c25 Mon Sep 17 00:00:00 2001 From: tegwick Date: Tue, 29 Sep 2026 12:36:34 +0200 Subject: [PATCH] Maturity from run history, pack revision digests, attribute sets, projects and templates (RBT-WP-0008) Also fixes run lists and scores mixing runs of different packs in one repository. Co-Authored-By: Claude Sonnet 5.5 Assistant: claude-code Assistant-Model: sonnet Assistant-Process: 295952@bnt-lap001 Assistant-Session: e93f64ad-516c-46eb-9666-aad8d300c477 --- README.md | 3 + attribute-sets/harbour-defaults.yaml | 8 + attribute-sets/stormy-coast.yaml | 8 + docs/maturity-and-projects.md | 92 ++++++- docs/run-spec.md | 10 + project-templates/lighthouse-watch.yaml | 13 + rtut/__main__.py | 85 ++++++ rtut/attrs.py | 142 ++++++++++ rtut/maturity.py | 83 ++++++ rtut/pack.py | 22 ++ rtut/project.py | 244 ++++++++++++++++++ rtut/run.py | 33 ++- rtut/run2.py | 7 +- schema/attribute-set.schema.json | 37 +++ schema/project.schema.json | 165 ++++++++++++ tests/test_maturity.py | 99 +++++++ tests/test_projects.py | 214 +++++++++++++++ tests/test_score.py | 13 + ...nbook-maturity-projects-and-idempotency.md | 66 +++-- 19 files changed, 1311 insertions(+), 33 deletions(-) create mode 100644 attribute-sets/harbour-defaults.yaml create mode 100644 attribute-sets/stormy-coast.yaml create mode 100644 project-templates/lighthouse-watch.yaml create mode 100644 rtut/attrs.py create mode 100644 rtut/maturity.py create mode 100644 rtut/project.py create mode 100644 schema/attribute-set.schema.json create mode 100644 schema/project.schema.json create mode 100644 tests/test_maturity.py create mode 100644 tests/test_projects.py diff --git a/README.md b/README.md index c4152c2..8515c34 100644 --- a/README.md +++ b/README.md @@ -15,6 +15,9 @@ python3 -m rtut list # id, engine, exercise status, title python3 -m rtut show # steps with owners and done-when python3 -m rtut render -p name=value python3 -m rtut launch # legacy-command packs: start the existing console +python3 -m rtut maturity # draft | reviewed | rehearsed | proven | stale, from run history +python3 -m rtut review --by you # record a review of the current revision +python3 -m rtut project list|validate|status|attrs|new|run ... # projects, attribute sets, templates python3 -m rtut ui # local web UI on 127.0.0.1:8877: walk a pack, record a run python3 -m rtut run new|start|verify|done|finish ... # the same run record from the terminal make test diff --git a/attribute-sets/harbour-defaults.yaml b/attribute-sets/harbour-defaults.yaml new file mode 100644 index 0000000..ae95a57 --- /dev/null +++ b/attribute-sets/harbour-defaults.yaml @@ -0,0 +1,8 @@ +spec: attribute-set/v0.1 +id: harbour-defaults +title: Defaults for any harbour lighthouse +attributes: + lamp: paraffin + signal: horn + motto: Steady light for those at sea + keeper: {requirement: required, note: Who keeps this light. Needed before the story can start.} diff --git a/attribute-sets/stormy-coast.yaml b/attribute-sets/stormy-coast.yaml new file mode 100644 index 0000000..ee5b783 --- /dev/null +++ b/attribute-sets/stormy-coast.yaml @@ -0,0 +1,8 @@ +spec: attribute-set/v0.1 +id: stormy-coast +title: A coast with bad weather +extends: [harbour-defaults] +attributes: + weather: storm + signal: flags + lamp: electric diff --git a/docs/maturity-and-projects.md b/docs/maturity-and-projects.md index 29561a6..3ee03d8 100644 --- a/docs/maturity-and-projects.md +++ b/docs/maturity-and-projects.md @@ -1,6 +1,7 @@ # Runbook maturity, projects and idempotency -Status: **proposal for decision** (RBT-WP-0008-T01). Raised by Bernd on 2026-09-29. +Status: **decided 2026-09-29** (Bernd: "go with your suggestions and keep project"), with the attribute-set and template +extension below. Tracked as RBT-WP-0008. ## The problem @@ -94,13 +95,84 @@ A **rehearsal** project is where maturity evidence comes from. "Exercised" then - Project files are declarative and non-secret. Candidate location: the repository that owns the system (for example `net-kingdom/projects//project.yaml`), next to `runbooks/`. -## Open questions +## Decisions (2026-09-29, Bernd) -1. **Names.** *Project* is Bernd's word. Alternatives: *target*, *engagement*, *installation*, *rollout*. Is - *environment class* (`rehearsal|staging|production`) the right way to separate rehearsal from real? -2. **Maturity thresholds.** Is "two environments, two operators" the right bar for `proven`? Should `reviewed` - be part of the ladder? -3. **Where projects live**, and whether one runbook may belong to several projects (yes, one project per - environment, so a runbook applies to many). -4. **Retry policy** for non-idempotent steps, as described above. -5. **Shared run history**: committing a redacted run summary versus keeping records local. +1. **Name: project.** Environment class `rehearsal | staging | production`. +2. **Maturity:** `draft`, `reviewed`, `rehearsed`, `proven`, `stale`; `proven` = reviewed, rehearsed, clean in at least two + environments by at least two operators. Evidence flags are independent; the level is the highest one achieved. +3. **Project files live in the repository that owns the system**, next to `runbooks/`. +4. **Retry of a failed non-idempotent step requires the operator to confirm the earlier attempt did not partly apply.** + +## Extension: attribute sets, targets and project templates + +A project may have a **goal**, several **targets** (installations, tenants, hosts) and shared facts. All of these are +**attribute sets**: named collections of non-secret attributes that runbooks and other attribute sets can reuse. This is what +makes runbooks and projects composable, and it is the basis for reusable **project templates**. + +### Attribute sets + +```yaml +spec: attribute-set/v0.1 +id: railiance01-host +title: The railiance01 host +extends: [base-cluster] # composition: later sets override earlier ones +attributes: + target_host: railiance01 # a value + tenant: {requirement: required} # needed, no value yet + motto: {requirement: optional} +``` + +Every attribute has: + +- a **requirement**, declared: `required` (needed) or `optional`; +- a **state**, derived, using the same words as run datapoints: `nil` (no value), `set`, `err` (fails a consuming runbook's + datapoint definition), `ok` (confirmed by a run's check), plus a **modified** flag when a later layer overrides a value that an + earlier layer provided; +- a **source**: which layer gave the value (set, project, target or member). Provenance is what makes a wrong value traceable. + +Values are never secret (`secret` is not allowed; secret markers are refused). Only references to secrets are allowed. + +### Layers, in order (later wins) + +attribute sets listed by the project (each after its own `extends`) then the project's own `attributes`, then the target's +sets and attributes, then the member's own `attributes`. The resolved attributes become the defaults of the member runbook's +datapoints, so a value shared by several runbooks is entered once. + +### Projects and targets + +```yaml +spec: project/v0.1 +id: railiance01-identity +title: Identity on railiance01 +goal: Lightweight identity is reachable and its custody is documented. +environment: production # rehearsal | staging | production +template: identity-stack # set when created from a template +attribute_sets: [railiance01-host] +attributes: {tenant: coulomb} +targets: + coulomb: {attribute_sets: [], attributes: {tenant: coulomb}} +runbooks: + - {pack: nk.ssh-certificates, target: coulomb} + - {pack: nk.openbao-operating-path, after: [nk.ssh-certificates]} +``` + +Project state is derived from each member's latest run at the current pack revision (see the table above, extended by +member status `not-applied | applying | applied | stale | failed`). + +### Templates + +A **project template** (`spec: project-template/v0.1`) is a project skeleton: goal, members, attribute sets and attributes that +still need values. Instantiating a template creates a project file, records which template it came from, and shows which required +attributes are still needed. Templates make a set of runbooks reusable as a product. + +### Modes + +Any project run can use any mode: **manual** (operator confirms everything), **auto-confirm** (the engine confirms when +preconditions hold), and later **auto-pilot** (the engine also executes steps; RBT-WP-0006, opt-in and audited). Because the +project supplies the attributes, an auto-pilot run needs no operator input except what the template marks as `required` and unset. + +## Still open + +1. Sharing run history between machines (committing a redacted run summary versus keeping records local). +2. Whether attribute sets may declare validation of their own (types), independent of the runbooks that consume them. +3. The exact shape of member ordering (`after`) once runbooks can also depend on runbooks (`requires_packs`, RBT-WP-0006). diff --git a/docs/run-spec.md b/docs/run-spec.md index 39c52cf..404e3d2 100644 --- a/docs/run-spec.md +++ b/docs/run-spec.md @@ -105,3 +105,13 @@ Rules the engine enforces: - **Stalled**: an unfinished run is counted as `stalled` in the score, and as `abandoned` after a week. - **Modes** are `manual` and `auto-confirm`. Auto-confirm applies confirmation only; running the step's own command is a separate, later level ("auto-pilot with execution", RBT-WP-0006). + +## Context and revision + +Every run (v0.1 and v0.2) records `pack_digest`, the revision of the pack directory it used (a hash of its files, +excluding review records, local state and `exercise_status:` lines), and `context`: the optional non-secret +`project`, `target` and `environment` class (`rehearsal | staging | production`) it was applied to. Maturity and project +state are derived from these (`docs/maturity-and-projects.md`). Run records of all packs in a repository share one +folder; a pack's runs are those whose `pack_id` matches. + +Run timestamps have microsecond precision, so runs and feedback made within the same second still sort correctly. diff --git a/project-templates/lighthouse-watch.yaml b/project-templates/lighthouse-watch.yaml new file mode 100644 index 0000000..86b1db4 --- /dev/null +++ b/project-templates/lighthouse-watch.yaml @@ -0,0 +1,13 @@ +spec: project-template/v0.1 +id: lighthouse-watch +title: A watch at a lighthouse +goal: The night watch is played through and its logbook entry is written and removed again. +attribute_sets: [harbour-defaults] +attributes: + logdir: {requirement: optional, note: Scratch folder; must start with /tmp/rtut-} +targets: + north-light: {title: The northern light, attributes: {weather: fog}} + south-light: {title: The southern light, attributes: {weather: clear}} +runbooks: + - {pack: rtut.showcase-lighthouse, target: north-light} + - {pack: rtut.example-hello, after: [rtut.showcase-lighthouse]} diff --git a/rtut/__main__.py b/rtut/__main__.py index 910f21d..281ceea 100644 --- a/rtut/__main__.py +++ b/rtut/__main__.py @@ -65,6 +65,59 @@ def _run(pk, a) -> int: return 1 +def _project(a) -> int: + from . import project as J + from .attrs import Registry + reg = Registry([Path(a.root)]) + try: + if a.action == "list": + for pid, d in reg.projects.items(): + st = J.state(reg, d)["state"] if d.ok else "invalid" + print(f"{pid}\t{d.data.get('environment', '?')}\t{st}\t{d.data.get('title', '')}") + for tid, d in reg.templates.items(): + print(f"{tid}\ttemplate\t-\t{d.data.get('title', '')}") + return 0 + if a.action == "new": + path = J.instantiate(reg, a.name, a.extra, a.environment, + dict(kv.split("=", 1) for kv in a.set)) + print(path); return 0 + d = reg.projects.get(a.name) + if d is None: + print(f"error: no project {a.name}", file=sys.stderr); return 1 + errs = J.validate(reg, d) + if a.action == "validate": + print(("ok " if not errs else "FAIL ") + d.id) + for e in errs: + print(" -", e) + return 1 if errs else 0 + if errs: + print("\n".join(errs), file=sys.stderr); return 1 + if a.action == "status": + st = J.state(reg, d) + print(f"{d.id}: {st['state']} [{d.data['environment']}] {d.data.get('goal', '')}") + for m in st["members"]: + print(f" {m['status']:12} {m['pack']}" + (f" @{m['target']}" if m["target"] else "")) + print(" next:", ", ".join(st["next"]) or "-") + for n in st["needs"]: + print(f" needs: {n['attribute']} ({n['pack']})") + return 0 + if a.action == "attrs": + member = next(m for m in d.data["runbooks"] if m["pack"] == a.extra) + res = J.member_attributes(reg, d.data, member) + for r in res["rows"]: + print(f"{r['state']:4} {r['requirement']:9} {r['id']} = {r['value']!r} <- {r['source'] or 'runbook default' if 'default' in r else r['source'] or '-'}" + + (" (modified)" if r["modified"] else "") + (f" ERROR: {r['error']}" if r.get("error") else "")) + if res["unused"]: + print("unused attributes:", ", ".join(res["unused"])) + return 0 + if a.action == "run": + r = J.start_run(reg, d, a.extra, a.operator, a.target, a.mode) + print(r.data["run_id"]); return 0 + except Exception as e: # user-facing command: report the problem, do not print a trace + print(f"error: {e}", file=sys.stderr); return 1 + return 0 + + def main(argv: list[str] | None = None) -> int: ap = argparse.ArgumentParser(prog="rtut") sub = ap.add_subparsers(dest="cmd", required=True) @@ -89,6 +142,20 @@ def main(argv: list[str] | None = None) -> int: sp = sub.add_parser("ui", help="serve the local UI on 127.0.0.1") sp.add_argument("roots", nargs="*", default=["."]) sp.add_argument("--port", type=int, default=8877) + sp = sub.add_parser("maturity", help="derived maturity of a pack at its current revision") + sp.add_argument("manifest") + sp = sub.add_parser("review", help="record that you reviewed the pack at its current revision") + sp.add_argument("manifest"); sp.add_argument("--by", required=True); sp.add_argument("--note", default="") + sp = sub.add_parser("project", help="projects: list, validate, status, attrs, new (from a template), run") + sp.add_argument("action", choices=["list", "validate", "status", "attrs", "new", "run"]) + sp.add_argument("root") + sp.add_argument("name", nargs="?", help="project id (new: template id)") + sp.add_argument("extra", nargs="?", help="attrs/run: pack id; new: the new project id") + sp.add_argument("--environment", "-e", default="") + sp.add_argument("--set", "-s", action="append", default=[], help="new: attribute=value") + sp.add_argument("--operator", "-o", default="") + sp.add_argument("--target", default=None) + sp.add_argument("--mode", default="manual", choices=["manual", "auto-confirm"]) sp = sub.add_parser("score", help="how often each runbook was tried and how it went") sp.add_argument("root", nargs="?", default=".") sp = sub.add_parser("feedback", help="list the latest feedback of every run of a pack") @@ -109,6 +176,24 @@ def main(argv: list[str] | None = None) -> int: except KeyboardInterrupt: pass return 0 + if a.cmd == "project": + return _project(a) + if a.cmd in ("maturity", "review"): + from . import maturity as M + pk = P.load(Path(a.manifest)) + if not pk.ok: + print("\n".join(pk.errors), file=sys.stderr); return 1 + if a.cmd == "review": + try: + M.add_review(pk, a.by, a.note) + except M.ReviewError as e: + print(f"error: {e}", file=sys.stderr); return 1 + print("review recorded for revision", P.digest(pk)); return 0 + m = M.maturity(pk); ev = m["evidence"] + print(f"{pk.id}: {m['level']}" + (f" (was {m['stale_from']})" if m["stale_from"] else "") + f" revision {m['digest']}") + print(f" reviewed: {ev['reviewed']} clean runs: {ev['clean_runs']} (rehearsal {ev['rehearsal_runs']}) " + f"environments: {', '.join(ev['environments']) or '-'} operators: {', '.join(ev['operators']) or '-'}") + return 0 if a.cmd == "score": from . import run as R for p in _packs(a.root): diff --git a/rtut/attrs.py b/rtut/attrs.py new file mode 100644 index 0000000..6d15d90 --- /dev/null +++ b/rtut/attrs.py @@ -0,0 +1,142 @@ +"""Attribute sets: named, composable, non-secret attribute collections with layered resolution. + +An attribute has a *requirement* (required | optional), declared, and a *state*, derived with the same +words as run datapoints: nil (no value), set, err (invalid for a consuming runbook). `modified` marks a +value that a later layer replaced. Every value keeps its source, so a wrong value is traceable. +""" +from __future__ import annotations + +import json +from dataclasses import dataclass, field +from pathlib import Path + +import yaml +from jsonschema import Draft202012Validator + +from . import pack as P +from .pack import SECRET_RE + +SCHEMA_DIR = Path(__file__).resolve().parent.parent / "schema" + + +def _schema_errors(data: dict, schema_file: str) -> list[str]: + v = Draft202012Validator(json.loads((SCHEMA_DIR / schema_file).read_text())) + return [f"{'/'.join(map(str, e.absolute_path)) or ''}: {e.message}" + for e in sorted(v.iter_errors(data), key=lambda e: list(map(str, e.absolute_path)))] + + +def normalize(attrs: dict | None) -> dict[str, dict]: + """scalar or {value, requirement, note} -> {value, requirement, note}""" + out = {} + for k, v in (attrs or {}).items(): + out[k] = dict(v) if isinstance(v, dict) else {"value": v} + out[k].setdefault("value", None) + out[k].setdefault("requirement", None) + return out + + +@dataclass +class Doc: + path: Path + data: dict + errors: list[str] = field(default_factory=list) + + @property + def id(self) -> str: + return str(self.data.get("id", self.path.stem)) + + @property + def ok(self) -> bool: + return not self.errors + + +def _load(path: Path, schema_file: str) -> Doc: + try: + data = yaml.safe_load(Path(path).read_text()) + except yaml.YAMLError as e: + return Doc(Path(path), {}, [f"invalid YAML: {e}"]) + if not isinstance(data, dict): + return Doc(Path(path), {}, ["must be a mapping"]) + errs = _schema_errors(data, schema_file) + if not errs and SECRET_RE.search(json.dumps(data)): + errs.append("contains secret-looking marker") + return Doc(Path(path), data, errs) + + +def load_set(path: Path) -> Doc: + return _load(path, "attribute-set.schema.json") + + +class Registry: + """Everything found under one or more repository roots.""" + + def __init__(self, roots: list[Path]): + self.roots = [Path(r) for r in roots] + self.sets: dict[str, Doc] = {} + self.projects: dict[str, Doc] = {} + self.templates: dict[str, Doc] = {} + self.packs: dict[str, P.Pack] = {} + for r in self.roots: + for f in sorted(r.glob("attribute-sets/*.yaml")): + d = load_set(f); self.sets[d.id] = d + for f in sorted(r.glob("projects/*/project.yaml")): + d = _load(f, "project.schema.json"); self.projects[d.id] = d + for f in sorted(r.glob("project-templates/*.yaml")): + d = _load(f, "project.schema.json"); self.templates[d.id] = d + for f in P.discover(r): + pk = P.load(f); self.packs[pk.id] = pk + + +def set_layers(reg: Registry, set_id: str, _seen: tuple = ()) -> list[tuple[str, dict]]: + """Layers of an attribute set, its `extends` first. Raises on unknown sets and cycles.""" + if set_id in _seen: + raise ValueError("attribute set cycle: " + " -> ".join(_seen + (set_id,))) + doc = reg.sets.get(set_id) + if doc is None: + raise ValueError(f"unknown attribute set {set_id}") + if not doc.ok: + raise ValueError(f"attribute set {set_id} is invalid: {doc.errors[0]}") + layers: list[tuple[str, dict]] = [] + for base in doc.data.get("extends", []): + layers += set_layers(reg, base, _seen + (set_id,)) + layers.append((f"set:{set_id}", normalize(doc.data["attributes"]))) + return layers + + +def fold(layers: list[tuple[str, dict]]) -> dict[str, dict]: + """Later layers win. Keeps provenance and marks a value replaced by a later layer as modified.""" + out: dict[str, dict] = {} + for source, attrs in layers: + for name, a in attrs.items(): + cur = out.setdefault(name, {"value": None, "requirement": None, "source": None, "modified": False, + "history": []}) + if a.get("requirement"): + cur["requirement"] = a["requirement"] + if a.get("value") is not None: + if cur["value"] is not None and cur["value"] != a["value"]: + cur["modified"] = True + cur["value"] = a["value"] + cur["source"] = source + cur["history"].append({"source": source, "value": a["value"]}) + for cur in out.values(): + cur["state"] = "set" if cur["value"] is not None else "nil" + return out + + +def check_value(pack: P.Pack, name: str, value) -> str | None: + """None if the value is valid for the pack's datapoint/parameter `name`, else the reason.""" + defs = {d["id"]: d for d in (pack.data.get("datapoints") or pack.data.get("parameters") or [])} + d = defs.get(name) + if d is None: + return None + v = str(value).lower() if isinstance(value, bool) else str(value) + import re + if d["type"] == "int" and not re.fullmatch(r"-?\d+", v): + return "not an integer" + if d["type"] == "bool" and v not in ("true", "false"): + return "must be true or false" + try: + P.render("{{%s}}" % name, {name: v}, pack) + except (ValueError, KeyError) as e: + return str(e) + return None diff --git a/rtut/maturity.py b/rtut/maturity.py new file mode 100644 index 0000000..b96577c --- /dev/null +++ b/rtut/maturity.py @@ -0,0 +1,83 @@ +"""Runbook maturity: earned from run history, independent of any one environment, bound to a pack revision. + + draft valid, no review and no clean rehearsal at the current revision + reviewed a review is recorded for the current revision + rehearsed at least one clean run in a rehearsal environment at the current revision + proven reviewed, rehearsed, and clean in at least two environments by at least two operators + stale nothing holds at the current revision, but a higher level was reached at an earlier one + +A run is *clean* when it succeeded with no deviation (no override, waiver or push-through). +Runs without an environment context are not attributable and never count. +""" +from __future__ import annotations + +from datetime import datetime, timezone + +import yaml + +from . import run as R +from .pack import REVIEW_FILE, Pack, digest + +LEVELS = ("draft", "reviewed", "rehearsed", "proven") + + +class ReviewError(Exception): + pass + + +def _reviews(pack: Pack) -> list[dict]: + f = pack.path.parent / REVIEW_FILE + if not f.exists(): + return [] + return list((yaml.safe_load(f.read_text()) or {}).get("reviews", [])) + + +def add_review(pack: Pack, by: str, note: str = "") -> dict: + """Record that `by` reviewed the pack at its current revision.""" + if not by.strip() or len(by.split()) != 1: + raise ReviewError("reviewer must be one word (name or handle)") + try: + R._clean(by, "reviewer"); R._clean(note, "review note") + except R.RunError as e: + raise ReviewError(str(e)) from e + entry = {"by": by.strip(), "at": datetime.now(timezone.utc).isoformat(timespec="seconds"), + "digest": digest(pack), "note": note.strip()} + revs = _reviews(pack) + [entry] + (pack.path.parent / REVIEW_FILE).write_text(yaml.safe_dump({"reviews": revs}, sort_keys=False)) + return entry + + +def _env_key(ctx: dict) -> str: + return f"{ctx.get('project', '-')}/{ctx.get('target', '-')}" + + +def _evidence(pack: Pack, dig: str, runs: list[R.Run]) -> dict: + clean = [r for r in runs if r.data.get("pack_digest") == dig and r.data.get("outcome") == "succeeded" + and not r.data.get("deviations") and r.data.get("context", {}).get("environment")] + reviewed = any(rv.get("digest") == dig for rv in _reviews(pack)) + rehearsal = [r for r in clean if r.data["context"]["environment"] == "rehearsal"] + envs = {_env_key(r.data["context"]) for r in clean} + ops = {r.data["operator"] for r in clean} + proven = reviewed and bool(rehearsal) and len(envs) >= 2 and len(ops) >= 2 + level = "proven" if proven else "rehearsed" if rehearsal else "reviewed" if reviewed else "draft" + return {"level": level, "reviewed": reviewed, "clean_runs": len(clean), "rehearsal_runs": len(rehearsal), + "environments": sorted(envs), "operators": sorted(ops)} + + +def maturity(pack: Pack) -> dict: + """{'level', 'stale_from', 'digest', 'evidence'} for the pack's current revision.""" + runs = [R.load(pack, rid) for rid in R.list_runs(pack)] + dig = digest(pack) + now = _evidence(pack, dig, runs) + stale_from = None + if now["level"] == "draft": + # a higher level reached at an earlier revision means the maturity has expired + older = {r.data.get("pack_digest") for r in runs} | {rv.get("digest") for rv in _reviews(pack)} + best = "draft" + for d in older - {dig, None}: + lv = _evidence(pack, d, runs)["level"] + if LEVELS.index(lv) > LEVELS.index(best): + best = lv + if best != "draft": + stale_from = best + return {"level": "stale" if stale_from else now["level"], "stale_from": stale_from, "digest": dig, "evidence": now} diff --git a/rtut/pack.py b/rtut/pack.py index a8afad1..19bc899 100644 --- a/rtut/pack.py +++ b/rtut/pack.py @@ -135,3 +135,25 @@ def render(command: str, values: dict[str, str], pack: Pack) -> str: return val return PLACEHOLDER.sub(sub, command) + + +REVIEW_FILE = "REVIEWS.yaml" + + +def digest(pack: "Pack") -> str: + """Revision of a pack: hash of every file in its directory, except review records and local state. + + `exercise_status:` lines are ignored, so recording an exercise never changes the revision. + """ + import hashlib + h = hashlib.sha256() + root = pack.path.resolve().parent + for f in sorted(root.rglob("*")): + rel = f.relative_to(root) + if not f.is_file() or f.name == REVIEW_FILE or ".local" in rel.parts or "__pycache__" in rel.parts: + continue + data = f.read_bytes() + if f.name == "pack.yaml" or f.name.startswith("pack."): + data = b"\n".join(x for x in data.split(b"\n") if not x.startswith(b"exercise_status:")) + h.update(str(rel).encode() + b"\0" + data + b"\0") + return h.hexdigest()[:16] diff --git a/rtut/project.py b/rtut/project.py new file mode 100644 index 0000000..dbeeefa --- /dev/null +++ b/rtut/project.py @@ -0,0 +1,244 @@ +"""Projects: runbooks that share an environment, with layered attributes, derived state and templates.""" +from __future__ import annotations + +import re +from pathlib import Path + +import yaml + +from . import maturity as _m # noqa: F401 (kept importable for callers) +from . import run as R +from .attrs import Doc, Registry, check_value, fold, normalize, set_layers +from .pack import digest + +PROJECT_STATES = ("planned", "applying", "applied", "stale") + + +# ---- validation -------------------------------------------------------------------------------------- + +def validate(reg: Registry, doc: Doc) -> list[str]: + """Structural errors plus references that must resolve in the registry.""" + if not doc.ok: + return list(doc.errors) + d, errs = doc.data, [] + for s in d.get("attribute_sets", []): + try: + set_layers(reg, s) + except ValueError as e: + errs.append(str(e)) + targets = d.get("targets", {}) + for tid, t in targets.items(): + for s in t.get("attribute_sets", []): + try: + set_layers(reg, s) + except ValueError as e: + errs.append(f"target {tid}: {e}") + packs = [m["pack"] for m in d["runbooks"]] + keys = [(m["pack"], m.get("target")) for m in d["runbooks"]] + if len(keys) != len(set(keys)): + errs.append("a runbook appears twice for the same target") + for m in d["runbooks"]: + if m["pack"] not in reg.packs: + errs.append(f"runbook {m['pack']} not found") + elif not reg.packs[m["pack"]].ok: + errs.append(f"runbook {m['pack']} is invalid") + if m.get("target") and m["target"] not in targets: + errs.append(f"runbook {m['pack']}: unknown target {m['target']}") + for a in m.get("after", []): + if a not in packs: + errs.append(f"runbook {m['pack']}: after names {a}, which is not a member") + # acyclic member order + deps = {m["pack"]: set(m.get("after", [])) for m in d["runbooks"]} + state: dict[str, int] = {} + + def visit(n: str, path: list[str]) -> None: + if state.get(n) == 1: + errs.append("member order cycle: " + " -> ".join(path[path.index(n):] + [n])); return + if state.get(n) == 2 or n not in deps: + return + state[n] = 1 + for x in sorted(deps[n]): + visit(x, path + [n]) + state[n] = 2 + + for n in sorted(deps): + visit(n, []) + return sorted(set(errs)) + + +# ---- attribute resolution -------------------------------------------------------------------------- + +def _member(d: dict, pack_id: str, target: str | None = None) -> dict: + for m in d["runbooks"]: + if m["pack"] == pack_id and (target is None or m.get("target") == target): + return m + raise KeyError(pack_id) + + +def layers_for(reg: Registry, d: dict, member: dict) -> list: + layers: list = [] + for s in d.get("attribute_sets", []): + layers += set_layers(reg, s) + layers.append(("project", normalize(d.get("attributes")))) + t = d.get("targets", {}).get(member.get("target") or "", None) + if t: + for s in t.get("attribute_sets", []): + layers += set_layers(reg, s) + layers.append((f"target:{member['target']}", normalize(t.get("attributes")))) + layers.append((f"member:{member['pack']}", normalize(member.get("attributes")))) + return layers + + +def member_attributes(reg: Registry, d: dict, member: dict) -> dict: + """What the member runbook's inputs look like in this project. + + {'rows': [{id, label, value, state, requirement, source, modified, error?}], 'unused': [names]} + state: nil (no value), set, err (invalid for the runbook). requirement: required unless the attribute + says optional or the runbook itself has a default. + """ + pack = reg.packs[member["pack"]] + folded = fold(layers_for(reg, d, member)) + defs = [x for x in (pack.data.get("datapoints") or pack.data.get("parameters") or []) if not x.get("produced_by")] + rows = [] + for x in defs: + a = folded.get(x["id"], {"value": None, "requirement": None, "source": None, "modified": False}) + value = a["value"] + row = {"id": x["id"], "label": x.get("label", x["id"]), "value": value, "source": a["source"], + "modified": a["modified"], "requirement": a["requirement"] or ("optional" if "default" in x else "required")} + if value is None: + row["state"] = "nil" + if "default" in x: + row["default"] = x["default"] + else: + err = check_value(pack, x["id"], value) + row["state"] = "err" if err else "set" + if err: + row["error"] = err + rows.append(row) + known = {x["id"] for x in defs} + return {"rows": rows, "unused": sorted(n for n in folded if n not in known)} + + +def missing_required(reg: Registry, d: dict) -> list[dict]: + """Attributes some member still needs: required, no value, no runbook default.""" + out = [] + for m in d["runbooks"]: + if m["pack"] not in reg.packs: + continue + for r in member_attributes(reg, d, m)["rows"]: + if r["state"] == "nil" and r["requirement"] == "required" and "default" not in r: + out.append({"pack": m["pack"], "attribute": r["id"], "label": r["label"]}) + return out + + +def member_values(reg: Registry, d: dict, member: dict) -> dict[str, str]: + """Values to hand to a run: every set attribute (invalid ones are left out and reported as err).""" + return {r["id"]: (str(r["value"]).lower() if isinstance(r["value"], bool) else str(r["value"])) + for r in member_attributes(reg, d, member)["rows"] if r["state"] == "set"} + + +# ---- state --------------------------------------------------------------------------------------------- + +def _latest_run(pack, project_id: str, target: str | None): + best = None + for rid in R.list_runs(pack): + r = R.load(pack, rid) + ctx = r.data.get("context", {}) + if ctx.get("project") == project_id and ctx.get("target") == target: + if best is None or r.data["created"] >= best.data["created"]: + best = r + return best + + +def member_status(reg: Registry, d: dict, member: dict) -> dict: + """not-applied | applying | applied | stale | failed, from the latest run at the current pack revision.""" + pack = reg.packs[member["pack"]] + r = _latest_run(pack, d["id"], member.get("target")) + if r is None: + return {"pack": member["pack"], "target": member.get("target"), "status": "not-applied", "run": None} + eff = R.effective_outcome(r) + current = r.data.get("pack_digest") == digest(pack) + if eff == "succeeded": + status = "applied" if current else "stale" + elif eff in ("failed", "abandoned"): + status = "failed" + else: + status = "applying" + return {"pack": member["pack"], "target": member.get("target"), "status": status, "run": r.data["run_id"], + "deviations": len(r.data.get("deviations", [])), "current_revision": current} + + +def state(reg: Registry, doc: Doc) -> dict: + """Project state: planned | applying | applied | stale, with per-member status and what can start next.""" + d = doc.data + members = [member_status(reg, d, m) for m in d["runbooks"]] + st = [m["status"] for m in members] + if "stale" in st: + project = "stale" + elif all(x == "applied" for x in st): + project = "applied" + elif all(x == "not-applied" for x in st): + project = "planned" + else: + project = "applying" + applied = {m["pack"] for m in members if m["status"] == "applied"} + nxt = [m["pack"] for m, spec in zip(members, d["runbooks"]) + if m["status"] in ("not-applied", "failed", "stale") and set(spec.get("after", [])) <= applied] + return {"state": project, "members": members, "next": nxt, "needs": missing_required(reg, d)} + + +# ---- runs from a project ------------------------------------------------------------------------------------ + +def start_run(reg: Registry, doc: Doc, pack_id: str, operator: str, target: str | None = None, mode: str = "manual"): + """Create a run of a member runbook with the project's attributes as defaults and the project as context.""" + d = doc.data + if d["spec"] != "project/v0.1": + raise R.RunError("a template is not run; instantiate it into a project first") + errs = validate(reg, doc) + if errs: + raise R.RunError(f"project {doc.id} is invalid: {errs[0]}") + member = _member(d, pack_id, target) + pack = reg.packs[pack_id] + ctx = {"project": doc.id, "environment": d["environment"], "target": member.get("target")} + values = member_values(reg, d, member) + if pack.spec == "runbook-pack/v0.2": + from . import run2 + return run2.create(pack, operator, values, mode=mode, context=ctx) + if mode != "manual": + raise R.RunError("v0.1 packs run in manual mode only") + return R.create(pack, operator, values, ctx) + + +# ---- templates ----------------------------------------------------------------------------------------------- + +def instantiate(reg: Registry, template_id: str, project_id: str, environment: str, + values: dict[str, str] | None = None, title: str | None = None, root: Path | None = None) -> Path: + """Write projects//project.yaml from a template. Required attributes may stay unset; `state()` lists them.""" + t = reg.templates.get(template_id) + if t is None or not t.ok: + raise ValueError(f"no valid template {template_id}") + if not re.fullmatch(r"[a-z0-9]+([.-][a-z0-9]+)*", project_id): + raise ValueError("project id must be lower-case letters, digits, dots and hyphens") + if environment not in R.ENVIRONMENTS: + raise ValueError(f"environment must be one of {', '.join(R.ENVIRONMENTS)}") + d = {k: v for k, v in t.data.items() if k not in ("spec", "id", "title")} + attrs = normalize(d.get("attributes")) + for k, v in (values or {}).items(): + attrs.setdefault(k, {"value": None, "requirement": None}) + attrs[k]["value"] = v + out_attrs = {} + for k, a in attrs.items(): + out_attrs[k] = a["value"] if a.get("value") is not None and not a.get("requirement") and not a.get("note") else \ + {kk: vv for kk, vv in a.items() if vv is not None} + project = {"spec": "project/v0.1", "id": project_id, "title": title or t.data["title"], + **{k: v for k, v in d.items() if k not in ("attributes", "environment")}, "environment": environment, + "template": template_id} + if out_attrs: + project["attributes"] = out_attrs + home = Path(root) if root else t.path.parent.parent + target = home / "projects" / project_id / "project.yaml" + if target.exists(): + raise ValueError(f"{target} already exists") + target.parent.mkdir(parents=True, exist_ok=True) + target.write_text(yaml.safe_dump(project, sort_keys=False)) + return target diff --git a/rtut/run.py b/rtut/run.py index a700a64..057e69d 100644 --- a/rtut/run.py +++ b/rtut/run.py @@ -11,7 +11,7 @@ from datetime import date, datetime, timezone from pathlib import Path from .legacy import repo_root -from .pack import PLACEHOLDER, SECRET_RE, Pack, render +from .pack import PLACEHOLDER, SECRET_RE, Pack, digest, render STATES = ("todo", "running", "done", "failed", "skipped") VERIFY_TIMEOUT = 60 @@ -22,7 +22,7 @@ class RunError(Exception): def _now() -> str: - return datetime.now(timezone.utc).isoformat(timespec="seconds") + return datetime.now(timezone.utc).isoformat(timespec="microseconds") # sortable even within one second def _clean(value: str, what: str) -> str: @@ -69,12 +69,23 @@ def _writable(run: Run) -> None: raise RunError(f"run is finished (outcome: {run.data['outcome']}); start a new run") -def create(pack: Pack, operator: str, params: dict[str, str] | None = None) -> Run: +ENVIRONMENTS = ("rehearsal", "staging", "production") + + +def _context(ctx: dict | None) -> dict: + """Where a run applies: project id, target and environment class. Non-secret; all optional.""" + ctx = dict(ctx or {}) + if ctx.get("environment") and ctx["environment"] not in ENVIRONMENTS: + raise RunError(f"environment must be one of {', '.join(ENVIRONMENTS)}") + return {k: _clean(str(v), f"context {k}") for k, v in ctx.items() if v and k in ("project", "target", "environment")} + + +def create(pack: Pack, operator: str, params: dict[str, str] | None = None, context: dict | None = None) -> Run: if not pack.ok or pack.data.get("engine") != "native": raise RunError("only valid native packs can be run") if pack.spec == "runbook-pack/v0.2": from . import run2 - return run2.create(pack, operator, params) + return run2.create(pack, operator, params, context=context) if pack.spec != "runbook-pack/v0.1": raise RunError(f"{pack.spec} packs cannot be run") operator = _clean(operator.strip(), "operator") @@ -102,6 +113,8 @@ def create(pack: Pack, operator: str, params: dict[str, str] | None = None) -> R "created": _now(), "params": resolved, "steps": {s["id"]: {"state": "todo"} for s in pack.data["steps"]}, + "pack_digest": digest(pack), + "context": _context(context), } run = Run(pack, data) run.save() @@ -116,8 +129,18 @@ def load(pack: Pack, run_id: str) -> Run: def list_runs(pack: Pack) -> list[str]: + """Run ids of this pack. All packs of a repository share one run folder, so filter by pack id.""" d = runs_dir(pack) - return sorted(p.stem for p in d.glob("*.json")) if d.exists() else [] + if not d.exists(): + return [] + out = [] + for f in sorted(d.glob("*.json")): + try: + if json.loads(f.read_text()).get("pack_id") == pack.id: + out.append(f.stem) + except (OSError, ValueError): + continue + return out def _order(run: Run, step_id: str) -> None: diff --git a/rtut/run2.py b/rtut/run2.py index 899e9d2..8e63fdd 100644 --- a/rtut/run2.py +++ b/rtut/run2.py @@ -13,8 +13,8 @@ import uuid from datetime import datetime, timezone from .flow import Flow -from .pack import Pack, render -from .run import Run, RunError, _clean, _now, runs_dir # noqa: F401 (re-exported for callers) +from .pack import Pack, digest, render +from .run import Run, RunError, _clean, _context, _now, runs_dir # noqa: F401 (re-exported for callers) DEFAULT_PHRASE = "I understand this cannot be undone" VERIFY_TIMEOUT = 60 @@ -84,7 +84,7 @@ def _confirm_datapoints(run: Run, by: str, how: str) -> None: # ---- creation ------------------------------------------------------------------------------------- def create(pack: Pack, operator: str, values: dict[str, str] | None = None, mode: str = "manual", - preauthorize: list[str] | None = None) -> Run: + preauthorize: list[str] | None = None, context: dict | None = None) -> Run: if not pack.ok or pack.data.get("engine") != "native" or pack.spec != "runbook-pack/v0.2": raise RunError("only valid native runbook-pack/v0.2 packs can be run here") if mode not in ("manual", "auto-confirm"): @@ -104,6 +104,7 @@ def create(pack: Pack, operator: str, values: dict[str, str] | None = None, mode "spec": "runbook-run/v0.2", "run_id": time.strftime("%Y%m%d-%H%M%S-") + uuid.uuid4().hex[:6], "pack_id": pack.id, "pack_path": str(pack.path), "operator": operator, "created": _now(), "mode": mode, "params": {}, "deviations": [], + "pack_digest": digest(pack), "context": _context(context), "auto_confirm": {"preauthorized": pre, "authorized_by": operator if pre else None}, "datapoints": {i: {"state": "nil"} for i in fl.datapoints}, "steps": {i: {"state": "todo"} for i in fl.steps}, diff --git a/schema/attribute-set.schema.json b/schema/attribute-set.schema.json new file mode 100644 index 0000000..2f9720b --- /dev/null +++ b/schema/attribute-set.schema.json @@ -0,0 +1,37 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://runbook-tutorials.invalid/schema/attribute-set/v0.1", + "title": "Attribute set", + "type": "object", + "required": ["spec", "id", "title", "attributes"], + "additionalProperties": false, + "properties": { + "spec": {"const": "attribute-set/v0.1"}, + "id": {"type": "string", "pattern": "^[a-z0-9]+([.-][a-z0-9]+)*$"}, + "title": {"type": "string", "minLength": 3}, + "extends": {"type": "array", "items": {"type": "string"}, "description": "Composed first; later entries and this set's own attributes override."}, + "attributes": {"$ref": "#/$defs/attributes"} + }, + "$defs": { + "attributes": { + "type": "object", + "propertyNames": {"pattern": "^[a-z][a-z0-9_]*$"}, + "additionalProperties": {"$ref": "#/$defs/attribute"} + }, + "attribute": { + "oneOf": [ + {"type": ["string", "integer", "boolean"]}, + { + "type": "object", + "additionalProperties": false, + "properties": { + "value": {"type": ["string", "integer", "boolean"]}, + "requirement": {"enum": ["required", "optional"]}, + "note": {"type": "string"} + }, + "minProperties": 1 + } + ] + } + } +} diff --git a/schema/project.schema.json b/schema/project.schema.json new file mode 100644 index 0000000..c0b7743 --- /dev/null +++ b/schema/project.schema.json @@ -0,0 +1,165 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://runbook-tutorials.invalid/schema/project/v0.1", + "title": "Project or project template", + "type": "object", + "required": [ + "spec", + "id", + "title", + "runbooks" + ], + "additionalProperties": false, + "properties": { + "spec": { + "enum": [ + "project/v0.1", + "project-template/v0.1" + ] + }, + "id": { + "type": "string", + "pattern": "^[a-z0-9]+([.-][a-z0-9]+)*$" + }, + "title": { + "type": "string", + "minLength": 3 + }, + "goal": { + "type": "string" + }, + "environment": { + "enum": [ + "rehearsal", + "staging", + "production" + ] + }, + "template": { + "type": "string", + "description": "Id of the template this project was created from." + }, + "attribute_sets": { + "type": "array", + "items": { + "type": "string" + } + }, + "attributes": { + "$ref": "#/$defs/attributes" + }, + "targets": { + "type": "object", + "propertyNames": { + "pattern": "^[a-z][a-z0-9-]*$" + }, + "additionalProperties": { + "type": "object", + "additionalProperties": false, + "properties": { + "title": { + "type": "string" + }, + "attribute_sets": { + "type": "array", + "items": { + "type": "string" + } + }, + "attributes": { + "$ref": "#/$defs/attributes" + } + } + } + }, + "runbooks": { + "type": "array", + "minItems": 1, + "items": { + "type": "object", + "required": [ + "pack" + ], + "additionalProperties": false, + "properties": { + "pack": { + "type": "string" + }, + "target": { + "type": "string" + }, + "after": { + "type": "array", + "items": { + "type": "string" + } + }, + "attributes": { + "$ref": "#/$defs/attributes" + } + } + } + } + }, + "allOf": [ + { + "if": { + "properties": { + "spec": { + "const": "project/v0.1" + } + } + }, + "then": { + "required": [ + "environment" + ] + } + } + ], + "$defs": { + "attributes": { + "type": "object", + "propertyNames": { + "pattern": "^[a-z][a-z0-9_]*$" + }, + "additionalProperties": { + "$ref": "#/$defs/attribute" + } + }, + "attribute": { + "oneOf": [ + { + "type": [ + "string", + "integer", + "boolean" + ] + }, + { + "type": "object", + "additionalProperties": false, + "properties": { + "value": { + "type": [ + "string", + "integer", + "boolean" + ] + }, + "requirement": { + "enum": [ + "required", + "optional" + ] + }, + "note": { + "type": "string" + } + }, + "minProperties": 1 + } + ] + } + } +} diff --git a/tests/test_maturity.py b/tests/test_maturity.py new file mode 100644 index 0000000..8d74665 --- /dev/null +++ b/tests/test_maturity.py @@ -0,0 +1,99 @@ +import shutil +import subprocess +from pathlib import Path + +import pytest + +from rtut import maturity as M, pack as P, run as R + +ROOT = Path(__file__).resolve().parent.parent + + +@pytest.fixture +def pk(tmp_path): + d = tmp_path / "runbooks" / "show"; d.mkdir(parents=True) + shutil.copy(ROOT / "runbooks/showcase-lighthouse/pack.yaml", d / "pack.yaml") + return P.load(d / "pack.yaml") + + +def succeed(pk, operator, context): + r = R.create(pk, operator, {"logdir": f"/tmp/rtut-mat-{operator}-{context.get('project', 'x')}".lower()}, context) + for s in pk.data["steps"]: + sid = s["id"]; R.start(r, sid) + cmd = R.command_for(r, sid) + if cmd: + subprocess.run(cmd, shell=True, check=True, capture_output=True) + R.verify(r, sid, confirm=True, manual_ok=True if s["verify"]["expect"] == "manual" else None) + R.complete(r, sid, {n: "1" for n in s.get("evidence", [])}) + R.finalize(r, "succeeded") + return r + + +def test_digest_is_stable_ignores_exercise_status_and_reviews_but_changes_with_content(pk): + d1 = P.digest(pk) + assert d1 == P.digest(pk) + text = pk.path.read_text() + pk.path.write_text(text.replace("exercise_status: unexercised", "exercise_status: exercised 2026-09-29 by x")) + assert P.digest(pk) == d1 # recording an exercise is not a new revision + M.add_review(pk, "alice", "read it") + assert P.digest(pk) == d1 # neither is a review + pk.path.write_text(text + "\n# changed\n") + assert P.digest(pk) != d1 + + +def test_runs_carry_digest_and_context_and_validate_environment(pk): + r = R.create(pk, "bob", None, {"project": "p1", "environment": "rehearsal", "target": "t1"}) + assert r.data["pack_digest"] == P.digest(pk) and r.data["context"] == {"project": "p1", "environment": "rehearsal", "target": "t1"} + with pytest.raises(R.RunError, match="environment must be one of"): + R.create(pk, "bob", None, {"environment": "prod"}) + assert R.create(pk, "bob").data["context"] == {} + + +def test_maturity_ladder_draft_reviewed_rehearsed_proven(pk): + assert M.maturity(pk)["level"] == "draft" + M.add_review(pk, "alice") + assert M.maturity(pk)["level"] == "reviewed" + succeed(pk, "bob", {"project": "lab", "environment": "rehearsal"}) + m = M.maturity(pk) + assert m["level"] == "rehearsed" and m["evidence"]["rehearsal_runs"] == 1 + succeed(pk, "bob", {"project": "stage", "environment": "staging"}) # second environment, same operator + assert M.maturity(pk)["level"] == "rehearsed" + succeed(pk, "carol", {"project": "prod", "environment": "production"}) # second operator + m = M.maturity(pk) + assert m["level"] == "proven" and m["evidence"]["operators"] == ["bob", "carol"] and len(m["evidence"]["environments"]) == 3 + + +def test_proven_needs_a_review_and_a_rehearsal(pk): + succeed(pk, "bob", {"project": "a", "environment": "staging"}) + succeed(pk, "carol", {"project": "b", "environment": "production"}) + assert M.maturity(pk)["level"] == "draft" # clean runs in two environments, but no rehearsal and no review + M.add_review(pk, "alice") + m = M.maturity(pk) + assert m["level"] == "reviewed" # reviewed, yet never rehearsed: not proven + assert m["evidence"]["clean_runs"] == 2 and m["evidence"]["rehearsal_runs"] == 0 + succeed(pk, "dave", {"project": "lab", "environment": "rehearsal"}) + assert M.maturity(pk)["level"] == "proven" # review + rehearsal + 3 environments + 3 operators + + +def test_runs_without_context_or_with_deviations_never_count(pk): + succeed(pk, "bob", {}) # no environment: not attributable + assert M.maturity(pk)["evidence"]["clean_runs"] == 0 + r = R.create(pk, "bob", {"logdir": "/tmp/rtut-mat-dev"}, {"project": "lab", "environment": "rehearsal"}) + r.data["outcome"] = "succeeded"; r.data["deviations"] = [{"kind": "override"}]; r.save() + assert M.maturity(pk)["evidence"]["clean_runs"] == 0 + + +def test_maturity_goes_stale_when_the_pack_changes(pk): + M.add_review(pk, "alice") + succeed(pk, "bob", {"project": "lab", "environment": "rehearsal"}) + assert M.maturity(pk)["level"] == "rehearsed" + pk.path.write_text(pk.path.read_text() + "\n# a change after the proof\n") + m = M.maturity(pk) + assert m["level"] == "stale" and m["stale_from"] == "rehearsed" and m["evidence"]["level"] == "draft" + + +def test_review_input_is_validated(pk): + with pytest.raises(M.ReviewError): + M.add_review(pk, "two words") + with pytest.raises(M.ReviewError): + M.add_review(pk, "alice", "hvs.ABCDEFGHIJKLMN") diff --git a/tests/test_projects.py b/tests/test_projects.py new file mode 100644 index 0000000..b41f00b --- /dev/null +++ b/tests/test_projects.py @@ -0,0 +1,214 @@ +import shutil +import subprocess +from pathlib import Path + +import pytest +import yaml + +from rtut import project as J, run as R +from rtut.attrs import Registry, fold, load_set, normalize, set_layers + +ROOT = Path(__file__).resolve().parent.parent + + +def write(path: Path, data: dict): + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(yaml.safe_dump(data, sort_keys=False)) + + +@pytest.fixture +def repo(tmp_path): + for name in ("showcase-lighthouse", "example-hello"): + (tmp_path / "runbooks" / name).mkdir(parents=True) + shutil.copy(ROOT / f"runbooks/{name}/pack.yaml", tmp_path / "runbooks" / name / "pack.yaml") + for f in (ROOT / "attribute-sets").glob("*.yaml"): + write(tmp_path / "attribute-sets" / f.name, yaml.safe_load(f.read_text())) + shutil.copy(ROOT / "project-templates/lighthouse-watch.yaml", tmp_path / "project-templates/lighthouse-watch.yaml") \ + if (tmp_path / "project-templates").mkdir(exist_ok=True) is None else None + return tmp_path + + +def project(repo, **over): + d = {"spec": "project/v0.1", "id": "north-watch", "title": "North watch", "environment": "rehearsal", + "attribute_sets": ["stormy-coast"], + "attributes": {"keeper": "Ada", "logdir": "/tmp/rtut-proj-north"}, + "runbooks": [{"pack": "rtut.showcase-lighthouse"}, {"pack": "rtut.example-hello", "after": ["rtut.showcase-lighthouse"]}]} + d.update(over) + write(repo / "projects" / d["id"] / "project.yaml", d) + return Registry([repo]) + + +# ---- attribute sets --------------------------------------------------------------------------------------- + +def test_shipped_examples_are_valid(): + reg = Registry([ROOT]) + assert all(d.ok for d in reg.sets.values()), [d.errors for d in reg.sets.values()] + assert reg.templates["lighthouse-watch"].ok and set(reg.sets) >= {"harbour-defaults", "stormy-coast"} + + +def test_extends_order_provenance_and_modified_flag(repo): + reg = Registry([repo]) + f = fold(set_layers(reg, "stormy-coast")) + assert f["weather"]["value"] == "storm" and f["weather"]["source"] == "set:stormy-coast" + assert f["signal"]["value"] == "flags" and f["signal"]["modified"] is True # replaced a base value + assert [h["source"] for h in f["signal"]["history"]] == ["set:harbour-defaults", "set:stormy-coast"] + assert f["motto"]["value"] and f["motto"]["modified"] is False + assert f["keeper"]["state"] == "nil" and f["keeper"]["requirement"] == "required" + + +def test_attribute_set_errors(tmp_path): + write(tmp_path / "attribute-sets/a.yaml", {"spec": "attribute-set/v0.1", "id": "a", "title": "Set A", "extends": ["b"], "attributes": {}}) + write(tmp_path / "attribute-sets/b.yaml", {"spec": "attribute-set/v0.1", "id": "b", "title": "Set B", "extends": ["a"], "attributes": {}}) + write(tmp_path / "attribute-sets/c.yaml", {"spec": "attribute-set/v0.1", "id": "c", "title": "Set C", "attributes": {"token": "hvs.ABCDEFGHIJKLMN"}}) + write(tmp_path / "attribute-sets/d.yaml", {"spec": "attribute-set/v0.1", "id": "d", "title": "Set D", "attributes": {"Bad-Name": "x"}}) + reg = Registry([tmp_path]) + with pytest.raises(ValueError, match="cycle"): + set_layers(reg, "a") + with pytest.raises(ValueError, match="unknown attribute set"): + set_layers(reg, "zzz") + assert not reg.sets["c"].ok and any("secret" in e for e in reg.sets["c"].errors) + assert not reg.sets["d"].ok + assert not load_set(tmp_path / "attribute-sets/d.yaml").ok + + +def test_normalize_scalar_and_object(): + n = normalize({"a": "x", "b": {"requirement": "required"}, "c": 3}) + assert n["a"] == {"value": "x", "requirement": None} and n["b"]["value"] is None and n["c"]["value"] == 3 + + +# ---- resolution against runbooks ----------------------------------------------------------------------------- + +def test_member_attribute_states_requirements_and_layers(repo): + reg = project(repo, attributes={"logdir": "/etc", "surplus": "x"}) # invalid for the runbook, and unused + d = reg.projects["north-watch"].data + rows = {r["id"]: r for r in J.member_attributes(reg, d, d["runbooks"][0])["rows"]} + assert rows["weather"]["state"] == "set" and rows["weather"]["source"] == "set:stormy-coast" + assert rows["logdir"]["state"] == "err" and "rtut-" in rows["logdir"]["error"] or rows["logdir"]["state"] == "err" + assert rows["keeper"]["requirement"] == "required" # the set declares it required; the pack has a default + assert rows["keeper"]["state"] == "nil" and rows["keeper"]["default"] == "Ada" + assert J.member_attributes(reg, d, d["runbooks"][0])["unused"] == ["surplus"] + + +def test_target_and_member_layers_override_the_project(repo): + reg = project(repo, targets={"cape": {"attributes": {"weather": "fog"}}}, + runbooks=[{"pack": "rtut.showcase-lighthouse", "target": "cape", "attributes": {"motto": "Hold fast"}}]) + d = reg.projects["north-watch"].data + rows = {r["id"]: r for r in J.member_attributes(reg, d, d["runbooks"][0])["rows"]} + assert rows["weather"]["value"] == "fog" and rows["weather"]["source"] == "target:cape" and rows["weather"]["modified"] + assert rows["motto"]["value"] == "Hold fast" and rows["motto"]["source"].startswith("member:") + + +def test_missing_required_lists_what_a_member_still_needs(repo): + reg = project(repo, attributes={"logdir": "/tmp/rtut-proj-north"}) # keeper has a pack default, so not missing + d = reg.projects["north-watch"].data + assert J.missing_required(reg, d) == [] + reg = project(repo, runbooks=[{"pack": "rtut.example-hello"}], attribute_sets=[], attributes={"host": "h"}) + write(repo / "runbooks/example-hello/pack.yaml", {**yaml.safe_load((ROOT / "runbooks/example-hello/pack.yaml").read_text()), + "parameters": [{"id": "host", "label": "Host", "type": "string"}, {"id": "count", "label": "Count", "type": "int", "default": 1}]}) + reg = Registry([repo]); d = reg.projects["north-watch"].data; d["attributes"] = {} + assert [m["attribute"] for m in J.missing_required(reg, d)] == ["host"] + + +# ---- validation ------------------------------------------------------------------------------------------------ + +@pytest.mark.parametrize("over,frag", [ + ({"runbooks": [{"pack": "nope.nothing"}]}, "not found"), + ({"attribute_sets": ["ghost"]}, "unknown attribute set"), + ({"runbooks": [{"pack": "rtut.showcase-lighthouse", "target": "ghost"}]}, "unknown target"), + ({"runbooks": [{"pack": "rtut.showcase-lighthouse", "after": ["rtut.example-hello"]}]}, "not a member"), + ({"runbooks": [{"pack": "rtut.showcase-lighthouse", "after": ["rtut.example-hello"]}, + {"pack": "rtut.example-hello", "after": ["rtut.showcase-lighthouse"]}]}, "cycle"), + ({"runbooks": [{"pack": "rtut.showcase-lighthouse"}, {"pack": "rtut.showcase-lighthouse"}]}, "twice"), +]) +def test_project_validation(repo, over, frag): + reg = project(repo, **over) + errs = J.validate(reg, reg.projects["north-watch"]) + assert any(frag in e for e in errs), errs + + +def test_project_needs_an_environment_and_valid_class(repo): + write(repo / "projects/x/project.yaml", {"spec": "project/v0.1", "id": "x", "title": "Project x", "runbooks": [{"pack": "rtut.example-hello"}]}) + write(repo / "projects/y/project.yaml", {"spec": "project/v0.1", "id": "y", "title": "Project y", "environment": "prod", "runbooks": [{"pack": "rtut.example-hello"}]}) + reg = Registry([repo]) + assert not reg.projects["x"].ok and not reg.projects["y"].ok + + +# ---- runs and state ---------------------------------------------------------------------------------------------- + +def finish(r): + for s in r.pack.data["steps"]: + sid = s["id"]; R.start(r, sid) + cmd = R.command_for(r, sid) + if cmd: + subprocess.run(cmd, shell=True, check=True, capture_output=True) + R.verify(r, sid, confirm=True, manual_ok=True if s["verify"]["expect"] == "manual" else None) + R.complete(r, sid, {n: "1" for n in s.get("evidence", [])}) + R.finalize(r, "succeeded") + + +def test_start_run_uses_attributes_and_records_context(repo): + reg = project(repo) + r = J.start_run(reg, reg.projects["north-watch"], "rtut.showcase-lighthouse", "bob") + assert r.data["params"]["weather"] == "storm" and r.data["params"]["signal"] == "flags" and r.data["params"]["logdir"] == "/tmp/rtut-proj-north" + assert r.data["context"] == {"project": "north-watch", "environment": "rehearsal"} + with pytest.raises(R.RunError, match="manual mode only"): + J.start_run(reg, reg.projects["north-watch"], "rtut.showcase-lighthouse", "bob", mode="auto-confirm") + + +def test_project_state_planned_applying_applied_and_next(repo): + reg = project(repo) + doc = reg.projects["north-watch"] + st = J.state(reg, doc) + assert st["state"] == "planned" and st["next"] == ["rtut.showcase-lighthouse"] # the second member waits for the first + r = J.start_run(reg, doc, "rtut.showcase-lighthouse", "bob") + st = J.state(reg, doc) + assert st["state"] == "applying" and st["members"][0]["status"] == "applying" and st["next"] == [] + finish(r) + st = J.state(reg, doc) + assert st["members"][0]["status"] == "applied" and st["state"] == "applying" and st["next"] == ["rtut.example-hello"] + finish(J.start_run(reg, doc, "rtut.example-hello", "bob")) + assert J.state(reg, doc)["state"] == "applied" + + +def test_project_goes_stale_when_a_member_changes_and_failed_members_are_shown(repo): + reg = project(repo, runbooks=[{"pack": "rtut.showcase-lighthouse"}]) + doc = reg.projects["north-watch"] + finish(J.start_run(reg, doc, "rtut.showcase-lighthouse", "bob")) + assert J.state(reg, doc)["state"] == "applied" + p = repo / "runbooks/showcase-lighthouse/pack.yaml" + p.write_text(p.read_text() + "\n# revised\n") + reg = Registry([repo]) + st = J.state(reg, reg.projects["north-watch"]) + assert st["state"] == "stale" and st["members"][0]["status"] == "stale" + r = J.start_run(reg, reg.projects["north-watch"], "rtut.showcase-lighthouse", "bob") + R.finalize(r, "abandoned") + assert J.state(reg, reg.projects["north-watch"])["members"][0]["status"] == "failed" + + +def test_runs_of_other_projects_do_not_count(repo): + reg = project(repo, runbooks=[{"pack": "rtut.showcase-lighthouse"}]) + write(repo / "projects/other/project.yaml", {**reg.projects["north-watch"].data, "id": "other", "title": "Other"}) + reg = Registry([repo]) + finish(J.start_run(reg, reg.projects["other"], "rtut.showcase-lighthouse", "bob")) + assert J.state(reg, reg.projects["north-watch"])["state"] == "planned" + + +# ---- templates ------------------------------------------------------------------------------------------------------ + +def test_instantiate_template_records_origin_merges_values_and_lists_needs(repo): + reg = Registry([repo]) + out = J.instantiate(reg, "lighthouse-watch", "harbour-north", "rehearsal", {"keeper": "Ada", "logdir": "/tmp/rtut-harbour"}) + made = yaml.safe_load(out.read_text()) + assert made["spec"] == "project/v0.1" and made["template"] == "lighthouse-watch" and made["environment"] == "rehearsal" + assert made["attributes"]["keeper"] == "Ada" and made["goal"].startswith("The night watch") + assert set(made["targets"]) == {"north-light", "south-light"} + reg = Registry([repo]) + assert reg.projects["harbour-north"].ok and J.validate(reg, reg.projects["harbour-north"]) == [] + r = J.start_run(reg, reg.projects["harbour-north"], "rtut.showcase-lighthouse", "bob", target="north-light") + assert r.data["params"]["weather"] == "fog" and r.data["context"]["target"] == "north-light" # target attributes applied + with pytest.raises(ValueError, match="already exists"): + J.instantiate(reg, "lighthouse-watch", "harbour-north", "rehearsal") + with pytest.raises(ValueError, match="environment"): + J.instantiate(reg, "lighthouse-watch", "harbour-x", "prod") + with pytest.raises(R.RunError, match="template is not run"): + J.start_run(reg, reg.templates["lighthouse-watch"], "rtut.showcase-lighthouse", "bob") diff --git a/tests/test_score.py b/tests/test_score.py index dbbdbfd..cff0836 100644 --- a/tests/test_score.py +++ b/tests/test_score.py @@ -102,3 +102,16 @@ def test_verify_against_pasted_output(pk): assert ok["passed"] and ok["source"] == "pasted" caps = R.load(pk, r.data["run_id"]).state("meet-the-keeper")["captures"] assert len(caps) == 2 and caps[0]["source"] == "pasted-for-verification" + + +def test_runs_of_different_packs_in_one_repository_are_not_mixed(tmp_path): + a = tmp_path / "runbooks" / "show"; a.mkdir(parents=True) + b = tmp_path / "runbooks" / "hello"; b.mkdir(parents=True) + shutil.copy(ROOT / "runbooks/showcase-lighthouse/pack.yaml", a / "pack.yaml") + shutil.copy(ROOT / "runbooks/example-hello/pack.yaml", b / "pack.yaml") + pa, pb = P.load(a / "pack.yaml"), P.load(b / "pack.yaml") + ra = R.create(pa, "x"); R.create(pa, "y"); R.create(pb, "z") + R.add_feedback(ra, "good", "only for the showcase") + assert R.scorecard(pa)["tried"] == 2 and R.scorecard(pb)["tried"] == 1 + assert R.list_runs(pb) != R.list_runs(pa) and len(R.list_runs(pb)) == 1 + assert R.scorecard(pb)["feedback"]["good"] == 0 and R.all_feedback(pb) == [] diff --git a/workplans/RBT-WP-0008-runbook-maturity-projects-and-idempotency.md b/workplans/RBT-WP-0008-runbook-maturity-projects-and-idempotency.md index 48619c6..794fb8a 100644 --- a/workplans/RBT-WP-0008-runbook-maturity-projects-and-idempotency.md +++ b/workplans/RBT-WP-0008-runbook-maturity-projects-and-idempotency.md @@ -24,13 +24,12 @@ pack revision), and what state an environment is in (projects). Replaces the env ```task id: RBT-WP-0008-T01 -status: progress +status: done priority: high state_hub_task_id: "904939c5-ed69-53a0-8fe1-2a0ae6d5337d" ``` -Review `docs/maturity-and-projects.md` with Bernd: names (project or target, environment class), maturity -ladder and thresholds, project states, where project files live, retry policy for non-idempotent steps. +Decided 2026-09-29 (Bernd: go with the suggestions, keep *project*): `docs/maturity-and-projects.md`. ## Idempotency as a declared characteristic @@ -50,60 +49,97 @@ it for the retry policy (T07). ```task id: RBT-WP-0008-T03 -status: wait +status: done priority: high state_hub_task_id: "a79e5b3d-221b-5ac6-a7a1-44737139c8f3" ``` -Record a digest of the pack directory in every run so maturity and project state can be tied to the revision used, -and expire when it changes. +Done 2026-09-29: `pack.digest()`; every run records `pack_digest` and `context` (project, target, environment class). ## Derive maturity from run history ```task id: RBT-WP-0008-T04 -status: wait +status: done priority: high state_hub_task_id: "53fe81cd-6b67-5508-bdbd-38073a0d77e7" ``` -`draft`, `reviewed`, `rehearsed`, `proven`, `stale`, derived from clean runs (no deviations) in rehearsal and other -environments, replacing `exercise_status` and the "Record pack as exercised" click. +Done 2026-09-29: `rtut/maturity.py`, review records (`REVIEWS.yaml`, bound to a revision), `rtut maturity`, `rtut review`. +`proven` = reviewed + rehearsed + clean in two environments by two operators. Not yet done: retiring `exercise_status` and the +"Record pack as exercised" click in the UI (T09). ## Project specification and shared datapoints ```task id: RBT-WP-0008-T05 -status: wait +status: done priority: high state_hub_task_id: "d022c7fb-1421-5738-a00f-5b18593fb1a2" ``` -`project.yaml`: environment class, shared non-secret datapoints provided as defaults to member runbooks, member -runbooks and ordering. Validator, and runs that belong to a project. +Done 2026-09-29: attribute sets (`attribute-set/v0.1`: composition with `extends`, requirement, derived state, provenance, +`modified` flag), projects with targets and members (`project/v0.1`), project templates (`project-template/v0.1`), +`rtut/attrs.py`, `rtut/project.py`, schemas, CLI (`rtut project ...`), examples under `attribute-sets/` and `project-templates/`. +Runs started from a project take the resolved attributes as defaults and record the project as context. ## Project state ```task id: RBT-WP-0008-T06 -status: wait +status: done priority: medium state_hub_task_id: "41f2b781-3699-5d21-bdfd-4ff3c2c0b777" ``` -Derive `planned`, `applying`, `applied`, `stale` (later `drifted`) from each member's latest run at the current revision. +Done 2026-09-29: `project.state()`: member status `not-applied | applying | applied | stale | failed`, project state +`planned | applying | applied | stale`, plus the members that can start next (`after`) and the attributes still needed. ## Retry policy for non-idempotent steps ```task id: RBT-WP-0008-T07 -status: wait +status: todo priority: medium state_hub_task_id: "8b34c57b-3380-5f7b-aaf5-6227a47dcf60" ``` Retrying a failed non-idempotent step requires the operator to confirm the earlier attempt did not partly apply. +## UI for projects, maturity and attributes + +```task +id: RBT-WP-0008-T08 +status: todo +priority: high +``` + +Show maturity and idempotency next to each runbook; a project view (goal, targets, member status, next member, needed +attributes with state, source and modified marks); start a run from a project so its attributes fill the form; a +template picker that instantiates a project. + +## Retire exercise_status + +```task +id: RBT-WP-0008-T09 +status: todo +priority: medium +``` + +Replace the "Record pack as exercised" click and the `exercise_status` field with derived maturity; migrate the packs. + +## Templates as products: auto-confirm and auto-pilot on a project + +```task +id: RBT-WP-0008-T10 +status: wait +priority: medium +``` + +Run a whole project in manual or auto-confirm mode, member by member in `after` order, with the attributes coming from the +project; auto-pilot with execution follows RBT-WP-0006. Attribute states to explore in use: needed, optional, modified, +checked; add what real use shows. + ## Acceptance Criteria - "Exercised" has a precise, environment-aware meaning; a production project is never called "exercised".