From 79996ab2494ec6e06001c6dd0facca7ecf7be87a Mon Sep 17 00:00:00 2001 From: tegwick Date: Tue, 29 Sep 2026 10:49:10 +0200 Subject: [PATCH] Adopt spec v0.2: schema, semantic validator, operating modes; workplans updated Co-Authored-By: Claude Sonnet 5.5 Assistant: claude-code Assistant-Model: sonnet Assistant-Process: 295952@bnt-lap001 Assistant-Session: e93f64ad-516c-46eb-9666-aad8d300c477 --- README.md | 2 +- ...pec-v0.2-proposal.md => pack-spec-v0.2.md} | 84 ++- rtut/__main__.py | 2 + rtut/pack.py | 7 + rtut/run.py | 2 + rtut/ui.py | 3 + rtut/v2.py | 213 ++++++ .../{pack.v0.2.draft.yaml => pack.v0.2.yaml} | 10 +- schema/runbook-pack-v0.2.schema.json | 621 ++++++++++++++++++ tests/test_showcase.py | 23 - tests/test_v2.py | 106 +++ ...WP-0002-runbook-pack-spec-and-validator.md | 18 +- workplans/RBT-WP-0004-local-runbook-ui.md | 14 +- ...T-WP-0006-execution-mode-and-pack-index.md | 12 + ...-engine-v0-2-graph-datapoints-and-gates.md | 37 +- 15 files changed, 1090 insertions(+), 64 deletions(-) rename docs/{pack-spec-v0.2-proposal.md => pack-spec-v0.2.md} (62%) create mode 100644 rtut/v2.py rename runbooks/showcase-lighthouse/{pack.v0.2.draft.yaml => pack.v0.2.yaml} (94%) create mode 100644 schema/runbook-pack-v0.2.schema.json create mode 100644 tests/test_v2.py diff --git a/README.md b/README.md index 6767de1..c4152c2 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ A user interface specialized to present runbook tutorials that with parameters can guide through actual IT administration and operation tasks. - **Intent:** `INTENT.md` ยท **Scope:** `SCOPE.md` -- **Pack specification:** `docs/pack-spec.md`, schema in `schema/` +- **Pack specification:** `docs/pack-spec.md` (v0.1, runnable) and `docs/pack-spec-v0.2.md` (v0.2: stages, gates, datapoints, actors, modes; validates, not yet runnable). Schemas in `schema/`. UX findings: `docs/ux-review-2026-09-29.md` - **Packs live where the procedure lives:** `runbooks//pack.yaml` in the owning repo. Examples: `net-kingdom/runbooks/`, and here `runbooks/example-hello/` and `runbooks/showcase-lighthouse/` (a harmless story to try the UI on). diff --git a/docs/pack-spec-v0.2-proposal.md b/docs/pack-spec-v0.2.md similarity index 62% rename from docs/pack-spec-v0.2-proposal.md rename to docs/pack-spec-v0.2.md index f0b33a6..ed8dc40 100644 --- a/docs/pack-spec-v0.2-proposal.md +++ b/docs/pack-spec-v0.2.md @@ -1,7 +1,9 @@ -# Runbook pack specification v0.2 - proposal +# Runbook pack specification v0.2 -Status: **proposed** (review with Bernd; RBT-WP-0002-T04). Derived from -`docs/ux-review-2026-09-29.md`. v0.1 stays valid; v0.2 is a superset. +Status: **adopted 2026-09-29** (Bernd's review answers are in the Decisions section). +Derived from `docs/ux-review-2026-09-29.md`. v0.1 stays valid; v0.2 is a superset. +Machine-checkable form: `schema/runbook-pack-v0.2.schema.json` plus the semantic +checks in `rtut/v2.py`. The engine and UI do not run v0.2 packs yet (RBT-WP-0007). ## Principle @@ -141,23 +143,65 @@ things to keep in mind, shown before the work starts. They are not gates. v0.1 - No embedded scripting, and no execution mode (still RBT-WP-0006). - No secret values anywhere. -## Recommendation +## Operating modes, overrides and deviations -Keep `runbook-pack/v0.1` as shipped (three NetKingdom packs use it). Adopt this -proposal as v0.2 and port the four in-repo packs, then use the console's seven -runbooks and 14 gates as the first real v0.2 test: if the model can express them -without special cases, it is expressive enough. +A **run** has a mode. The mode belongs to the run, not to the pack. -## Open questions +| Mode | Who confirms | Checks, gates, preconditions | +| --- | --- | --- | +| **manual** | the operator | shown with their state; the operator confirms each | +| **autopilot** | the engine, when every precondition is met | evaluated by the engine; confirmation is applied automatically | -1. **Waivers:** allowed at all, and by whom? Proposal: only where the pack declares - `waivable`, always with a recorded reason. -2. **Gate checks against live systems:** may a gate run a command (as steps' verify - do, operator-confirmed), or only read datapoint states and step results? - Proposal: both, with the same confirmation rule. -3. **Dimension vocabulary:** free tags first, or start from a canon list? Proposal: - free tags, with a `dimensions` registry file that packs may reference, promoted - to canon later. -4. **Nesting depth:** three levels enough? -5. **Datapoint `ok`:** who confirms it, the operator or a check? Proposal: either; - a check makes it `ok`, a manual confirmation is recorded as such. +### Manual mode: override + +If preconditions are not met the operator may **override** and continue anyway. +Doing so must (1) show a dialog that names the unmet preconditions and the +downstream risk (`downstream_risk` text declared on the step or gate, else a +generic warning), (2) require an explicit confirmation, and (3) record a +**deviation** in the run. + +### Autopilot mode: timeout and push-through + +- A step or gate may declare `timeout: {seconds: N, on_timeout: stop|push-through}`; + a run may set a default. Without a timeout, autopilot waits. +- `stop` (default): on timeout the run stops and reports. +- `push-through`: on timeout the engine confirms the item, records a deviation, + and moves on, ignoring the problem. This gathers the most information about + problems in one run and reduces refinement runs. +- Irreversible steps are never confirmed automatically unless the run was started + with an explicit, recorded pre-authorization naming those steps (and the pack's + `confirm_phrase`). Push-through does not bypass that rule. This is a safety + default, open to review. +- A step may declare `autopilot: manual-only` to forbid automation entirely. + +### Deviations + +Every override, waiver and push-through is a **deviation** in the run record: +`kind` (`override|waiver|push-through`), `item` (step or gate id), `mode`, +`unmet` (what was not satisfied), `risk_acknowledged`, `reason` (required for +overrides and waivers; automatic text for push-through), `who`, `at`. A run with +deviations is marked so in the overview and can never exercise a pack. A **waiver** is +declared in advance (`waivable` on a gate); an **override** is not. + +## Decisions (2026-09-29, Bernd) + +1. **Waivers:** allowed only where a gate declares `waivable`, always with a reason. + Undeclared departures are overrides (above), also recorded. +2. **Gate checks:** a gate may run a command (operator-confirmed in manual mode) or + read datapoint states and step results. +3. **Dimension vocabulary:** free tags first; a shared registry may be promoted to canon. +4. **Nesting:** at most three levels. If procedures grow beyond that, dependencies + between runbooks are needed (see below), not deeper nesting. +5. **Datapoint `ok`:** reached by a check, or by a recorded manual confirmation. + Handling of missing preconditions is defined by the operating modes above. + +## Later: dependencies between runbooks + +Not in v0.2. When a procedure needs another runbook (for example "OpenBao operating +path requires the SSH access pack"), packs will declare `requires_packs` with a +required result (exercised, or a named gate passed). Tracked in RBT-WP-0006. + +## Recommendation (carried out) + +Keep `runbook-pack/v0.1` as shipped (three NetKingdom packs use it). Use the old +console's seven runbooks and 14 gates as the first real v0.2 test. diff --git a/rtut/__main__.py b/rtut/__main__.py index 6ed3b39..4c31c72 100644 --- a/rtut/__main__.py +++ b/rtut/__main__.py @@ -9,6 +9,8 @@ from . import pack as P def _packs(root: str) -> list[P.Pack]: + if Path(root).is_file(): + return [P.load(Path(root))] return [P.load(p) for p in P.discover(Path(root))] diff --git a/rtut/pack.py b/rtut/pack.py index 0f4a41f..d703df0 100644 --- a/rtut/pack.py +++ b/rtut/pack.py @@ -26,6 +26,10 @@ class Pack: def id(self) -> str: return str(self.data.get("id", self.path.parent.name)) + @property + def spec(self) -> str: + return str(self.data.get("spec", "")) + @property def ok(self) -> bool: return not self.errors @@ -97,6 +101,9 @@ def load(path: Path) -> Pack: return Pack(path, {}, [f"invalid YAML: {e}"]) if not isinstance(data, dict): return Pack(path, {}, ["manifest must be a mapping"]) + if data.get("spec") == "runbook-pack/v0.2": + from . import v2 + return Pack(path, data, v2.validate(data)) errs = _schema_errors(data) if not errs: errs = _semantic_errors(data) diff --git a/rtut/run.py b/rtut/run.py index fb9f821..9e2b10e 100644 --- a/rtut/run.py +++ b/rtut/run.py @@ -66,6 +66,8 @@ class Run: def create(pack: Pack, operator: str, params: dict[str, str] | None = None) -> Run: if not pack.ok or pack.data.get("engine") != "native": raise RunError("only valid native packs can be run") + if pack.spec != "runbook-pack/v0.1": + raise RunError(f"{pack.spec} packs cannot be run yet (RBT-WP-0007)") operator = _clean(operator.strip(), "operator") if not re.fullmatch(r"\S+", operator): raise RunError("operator must be one word (name or handle)") diff --git a/rtut/ui.py b/rtut/ui.py index f208b1d..dbf0668 100644 --- a/rtut/ui.py +++ b/rtut/ui.py @@ -48,6 +48,9 @@ class App: d = p.data if not p.ok: rows.append(f"
  • {e(p.id)}: invalid ({e(p.errors[0])})
    • ") + elif d.get("spec") != "runbook-pack/v0.1" and d["engine"] == "native": + rows.append(f"
  • {e(d['title'])} {e(d.get('spec'))} " + "cannot be run yet (RBT-WP-0007)
    • ") elif d["engine"] != "native": rows.append(f"
  • {e(d['title'])} legacy console " f"start with: python -m rtut launch {e(p.path)}
    • ") diff --git a/rtut/v2.py b/rtut/v2.py new file mode 100644 index 0000000..0828f8b --- /dev/null +++ b/rtut/v2.py @@ -0,0 +1,213 @@ +"""Validation for runbook-pack/v0.2: JSON Schema plus semantic checks.""" +from __future__ import annotations + +import json +import re +from pathlib import Path + +from jsonschema import Draft202012Validator + +from .pack import PLACEHOLDER, SECRET_RE + +SCHEMA_PATH = Path(__file__).resolve().parent.parent / "schema" / "runbook-pack-v0.2.schema.json" +MAX_DEPTH = 3 + + +def schema_errors(data: dict) -> list[str]: + v = Draft202012Validator(json.loads(SCHEMA_PATH.read_text())) + return [f"{'/'.join(map(str, e.absolute_path)) or ''}: {e.message}" + for e in sorted(v.iter_errors(data), key=lambda e: list(map(str, e.absolute_path)))] + + +class _Model: + """Flattened view of a v0.2 pack used by the semantic checks.""" + + def __init__(self, data: dict): + self.data = data + self.errs: list[str] = [] + self.flow: dict[str, dict] = {} # stages, steps, gates share one id space + self.kind: dict[str, str] = {} + self.parent: dict[str, str | None] = {} + self.children: dict[str, list[str]] = {} + self.alts: dict[str, str] = {} # alternative id -> parent step id + self.depth_ok = True + + def add(self, id_: str, kind: str, node: dict, parent: str | None) -> None: + if id_ in self.flow or id_ in self.alts: + self.errs.append(f"duplicate id: {id_}") + return # keep the first definition; never overwrite the tree + self.flow[id_] = node + self.kind[id_] = kind + self.parent[id_] = parent + if parent: + self.children.setdefault(parent, []).append(id_) + + def walk(self, stages: list[dict], parent: str | None, depth: int) -> None: + for st in stages: + if depth > MAX_DEPTH: + self.errs.append(f"stage {st['id']}: nesting deeper than {MAX_DEPTH} levels") + self.add(st["id"], "stage", st, parent) + for s in st.get("steps", []): + self.add(s["id"], "step", s, st["id"]) + for a in s.get("one_of", []): + if a["id"] in self.flow or a["id"] in self.alts: + self.errs.append(f"duplicate id: {a['id']}") + self.alts[a["id"]] = s["id"] + for g in st.get("gates", []): + self.add(g["id"], "gate", g, st["id"]) + self.walk(st.get("stages", []), st["id"], depth + 1) + + def resolve(self, ref: str) -> str | None: + """requires reference -> flow id, or None if unresolved.""" + if ref.startswith("gate:"): + g = ref[5:] + return g if self.kind.get(g) == "gate" else None + return ref if self.kind.get(ref) in ("stage", "step") else None + + +def _check_refs(m: _Model, data: dict) -> None: + actors = {a["id"] for a in data.get("actors", [])} + dps = {d["id"]: d for d in data.get("datapoints", [])} + arts = {a["id"] for a in data.get("artefacts", [])} + steps = {i for i, k in m.kind.items() if k == "step"} | set(m.alts) + gates = {i for i, k in m.kind.items() if k == "gate"} + stages = {i for i, k in m.kind.items() if k == "stage"} + for label, ids in (("actor", [a["id"] for a in data.get("actors", [])]), + ("datapoint", list(dps)), ("artefact", [a["id"] for a in data.get("artefacts", [])])): + if len(ids) != len(set(ids)): + m.errs.append(f"duplicate {label} id") + + def check_ck(ck: dict, where: str, uses: set[str]) -> None: + exp = ck.get("expect", "manual") + if exp in ("exit-0", "output-contains") and not ck.get("command"): + m.errs.append(f"{where}: verify expect {exp} needs a command") + if exp == "output-contains" and not ck.get("contains"): + m.errs.append(f"{where}: verify output-contains needs 'contains'") + for d in ck.get("datapoints_ok", []): + if d not in dps: + m.errs.append(f"{where}: datapoints_ok names unknown datapoint {d}") + for t in (ck.get("command", ""), ck.get("contains", "")): + for name in PLACEHOLDER.findall(t or ""): + if name not in dps: + m.errs.append(f"{where}: undefined placeholder {{{{{name}}}}}") + elif uses and name not in uses: + m.errs.append(f"{where}: placeholder {name} not listed in uses") + + def check_text(text: str, where: str, uses: set[str]) -> None: + for name in PLACEHOLDER.findall(text or ""): + if name not in dps: + m.errs.append(f"{where}: undefined placeholder {{{{{name}}}}}") + elif name not in uses: + m.errs.append(f"{where}: placeholder {name} not listed in uses") + + for id_, node in m.flow.items(): + kind = m.kind[id_] + if kind == "step": + if node["actor"] not in actors: + m.errs.append(f"step {id_}: unknown actor {node['actor']}") + uses = set(node.get("uses", [])) + for u in uses: + if u not in dps: + m.errs.append(f"step {id_}: uses unknown datapoint {u}") + for p in node.get("produces", []): + if p not in dps and p not in arts: + m.errs.append(f"step {id_}: produces unknown datapoint or artefact {p}") + parts = [(node, id_)] + [(a, a["id"]) for a in node.get("one_of", [])] + for part, pid in parts: + check_text(part.get("command", ""), f"step {pid}", uses) + check_text(part.get("rollback", ""), f"step {pid}", uses) + if "verify" in part: + check_ck(part["verify"], f"step {pid}", uses) + if part.get("reversible", node.get("reversible", True)) is False: + if not (node.get("confirm_phrase") or data.get("confirm_phrase")): + m.errs.append(f"step {pid}: irreversible step needs a confirm_phrase (step or pack)") + for k, vals in (part.get("recommended_when") or {}).items(): + if k not in dps: + m.errs.append(f"step {pid}: recommended_when names unknown datapoint {k}") + elif dps[k].get("values"): + for v in vals: + if v not in dps[k]["values"]: + m.errs.append(f"step {pid}: recommended_when {k}={v} not in values") + if node.get("one_of") and node.get("command"): + m.errs.append(f"step {id_}: a step with one_of must not also have its own command") + elif kind == "gate": + by = (node.get("waivable") or {}).get("by") + if by and by not in actors: + m.errs.append(f"gate {id_}: waivable.by names unknown actor {by}") + for ck in node["checks"]: + check_ck(ck, f"gate {id_}", set()) + for d in dps.values(): + if d["type"] == "enum" and not d.get("values"): + m.errs.append(f"datapoint {d['id']}: enum without values") + dv = d.get("default") + if dv is not None: + if d.get("values") and dv not in d["values"]: + m.errs.append(f"datapoint {d['id']}: default not in values") + if d.get("pattern") and isinstance(dv, str) and not re.fullmatch(d["pattern"], dv): + m.errs.append(f"datapoint {d['id']}: default violates pattern") + if d.get("ask_in") and d["ask_in"] not in stages: + m.errs.append(f"datapoint {d['id']}: ask_in names unknown stage {d['ask_in']}") + if d.get("produced_by") and d["produced_by"] not in steps: + m.errs.append(f"datapoint {d['id']}: produced_by names unknown step {d['produced_by']}") + if d.get("confirmed_by") and d["confirmed_by"] not in steps | gates: + m.errs.append(f"datapoint {d['id']}: confirmed_by names unknown step or gate") + for a in data.get("artefacts", []): + if a["location"]["datapoint"] not in dps: + m.errs.append(f"artefact {a['id']}: location names unknown datapoint") + if a["produced_by"] not in steps: + m.errs.append(f"artefact {a['id']}: produced_by names unknown step") + if a.get("lives_until") and a["lives_until"] not in steps: + m.errs.append(f"artefact {a['id']}: lives_until names unknown step") + + +def _check_graph(m: _Model) -> None: + deps: dict[str, set[str]] = {i: set() for i in m.flow} + for id_, node in m.flow.items(): + for ref in node.get("requires", []): + r = m.resolve(ref) + if r is None: + m.errs.append(f"{m.kind[id_]} {id_}: requires unknown item {ref}") + else: + deps[id_].add(r) + # a node also waits for whatever its enclosing stages wait for + p, seen = m.parent[id_], set() + while p and p not in seen: + seen.add(p) + for ref in m.flow[p].get("requires", []): + r = m.resolve(ref) + if r: + deps[id_].add(r) + p = m.parent[p] + # a stage is finished when its contents are + for c in m.children.get(id_, []): + deps[id_].add(c) + state: dict[str, int] = {} + + def visit(n: str, path: list[str]) -> None: + if state.get(n) == 2: + return + if state.get(n) == 1: + m.errs.append("requires cycle: " + " -> ".join(path[path.index(n):] + [n])) + return + state[n] = 1 + for d in sorted(deps[n]): + visit(d, path + [n]) + state[n] = 2 + + for n in sorted(deps): + visit(n, []) + + +def validate(data: dict) -> list[str]: + errs = schema_errors(data) + if errs: + return errs + if data["engine"] == "legacy-command": + return [] + m = _Model(data) + m.walk(data["stages"], None, 1) + _check_refs(m, data) + _check_graph(m) + if SECRET_RE.search(json.dumps(data)): + m.errs.append("contains secret-looking marker") + return sorted(set(m.errs)) diff --git a/runbooks/showcase-lighthouse/pack.v0.2.draft.yaml b/runbooks/showcase-lighthouse/pack.v0.2.yaml similarity index 94% rename from runbooks/showcase-lighthouse/pack.v0.2.draft.yaml rename to runbooks/showcase-lighthouse/pack.v0.2.yaml index 376dd0e..dd4290c 100644 --- a/runbooks/showcase-lighthouse/pack.v0.2.draft.yaml +++ b/runbooks/showcase-lighthouse/pack.v0.2.yaml @@ -1,4 +1,4 @@ -# DRAFT - not loaded by the engine. Written against docs/pack-spec-v0.2-proposal.md. +# DRAFT - not loaded by the engine. Written against docs/pack-spec-v0.2.md. # Purpose: an acceptance test for the proposed model. If the model cannot express # this story without special cases, the proposal must change. When v0.2 is adopted # this file becomes pack.yaml (and the v0.1 file is retired or kept as a @@ -55,6 +55,7 @@ stages: produces: [lamp_lit] command: echo "{{keeper_name}} lights the {{lamp_fuel}} lamp." reversible: true + downstream_risk: Without a lit lamp the night watch has no purpose and ships are not warned. verify: {done_when: The lamp is lit, expect: manual} - id: read-the-sky title: Read the sky @@ -69,6 +70,8 @@ stages: title: The lamp burns steadily goal: safe-navigation # quality objective; a canon standard id where one exists blocking: true + timeout: {seconds: 120, on_timeout: stop} + downstream_risk: Ships may steer by a light that is not burning. checks: - {done_when: The light is visible from the water, expect: manual} - {datapoints_ok: [lamp_lit]} @@ -109,7 +112,7 @@ stages: owner: the-lighthouse requires: [warn] uses: [keeper_name, weather, lamp_fuel, motto, logdir] - produces: [logbook_file] + produces: [logbook-file] command: mkdir -p {{logdir}} && echo "Night watch by {{keeper_name}} - {{weather}}, {{lamp_fuel}} lamp. {{motto}}." >> {{logdir}}/logbook.txt reversible: true rollback: rm -rf {{logdir}} @@ -122,6 +125,7 @@ stages: steps: - id: close-log title: Close the log for good + autopilot: manual-only # never confirmed by the engine on its own actor: harbourmaster owner: the-lighthouse uses: [logdir] @@ -137,7 +141,7 @@ stages: - {done_when: No files remain under the scratch folder, command: 'test ! -e {{logdir}}', expect: exit-0} artefacts: - - id: logbook_file + - id: logbook-file title: The night's logbook entry location: {datapoint: logdir, path: logbook.txt} # a reference, never a value produced_by: write-logbook diff --git a/schema/runbook-pack-v0.2.schema.json b/schema/runbook-pack-v0.2.schema.json new file mode 100644 index 0000000..77b89f4 --- /dev/null +++ b/schema/runbook-pack-v0.2.schema.json @@ -0,0 +1,621 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://runbook-tutorials.invalid/schema/runbook-pack/v0.2", + "title": "Runbook pack manifest v0.2", + "type": "object", + "required": [ + "spec", + "id", + "title", + "owner", + "exercise_status", + "engine" + ], + "additionalProperties": false, + "properties": { + "spec": { + "const": "runbook-pack/v0.2" + }, + "id": { + "type": "string", + "pattern": "^[a-z0-9]+([.-][a-z0-9]+)*$" + }, + "title": { + "type": "string", + "minLength": 3 + }, + "owner": { + "type": "string", + "minLength": 1 + }, + "outcome": { + "type": "string" + }, + "exercise_status": { + "type": "string", + "pattern": "^(unexercised|exercised [0-9]{4}-[0-9]{2}-[0-9]{2} by \\S+)$" + }, + "engine": { + "enum": [ + "native", + "legacy-command" + ] + }, + "confirm_phrase": { + "type": "string", + "minLength": 8, + "description": "Typed by the operator to confirm an irreversible step." + }, + "timeout": { + "$ref": "#/$defs/timeout", + "description": "Default for autopilot runs." + }, + "legacy": { + "type": "object", + "required": [ + "command" + ], + "additionalProperties": false, + "properties": { + "command": { + "type": "array", + "items": { + "type": "string" + }, + "minItems": 1 + }, + "cwd": { + "type": "string" + }, + "url": { + "type": "string" + }, + "note": { + "type": "string" + } + } + }, + "framing": { + "type": "array", + "items": { + "type": "string" + }, + "description": "Things to keep in mind; shown first; not gates." + }, + "actors": { + "type": "array", + "items": { + "$ref": "#/$defs/actor" + } + }, + "datapoints": { + "type": "array", + "items": { + "$ref": "#/$defs/datapoint" + } + }, + "stages": { + "type": "array", + "minItems": 1, + "items": { + "$ref": "#/$defs/stage" + } + }, + "artefacts": { + "type": "array", + "items": { + "$ref": "#/$defs/artefact" + } + }, + "ownership": { + "type": "array", + "items": { + "$ref": "#/$defs/ownership" + } + } + }, + "allOf": [ + { + "if": { + "properties": { + "engine": { + "const": "native" + } + } + }, + "then": { + "required": [ + "stages", + "framing" + ] + } + }, + { + "if": { + "properties": { + "engine": { + "const": "legacy-command" + } + } + }, + "then": { + "required": [ + "legacy" + ] + } + } + ], + "$defs": { + "id": { + "type": "string", + "pattern": "^[a-z][a-z0-9-]*$" + }, + "dpid": { + "type": "string", + "pattern": "^[a-z][a-z0-9_]*$" + }, + "timeout": { + "type": "object", + "required": [ + "seconds" + ], + "additionalProperties": false, + "properties": { + "seconds": { + "type": "integer", + "minimum": 1 + }, + "on_timeout": { + "enum": [ + "stop", + "push-through" + ], + "default": "stop" + } + } + }, + "actor": { + "type": "object", + "required": [ + "id", + "role", + "description" + ], + "additionalProperties": false, + "properties": { + "id": { + "$ref": "#/$defs/id" + }, + "role": { + "type": "string" + }, + "description": { + "type": "string" + }, + "privileges": { + "type": "array", + "items": { + "type": "string" + } + } + } + }, + "datapoint": { + "type": "object", + "required": [ + "id", + "label", + "type" + ], + "additionalProperties": false, + "properties": { + "id": { + "$ref": "#/$defs/dpid" + }, + "label": { + "type": "string" + }, + "type": { + "enum": [ + "string", + "text", + "enum", + "path", + "bool", + "int" + ] + }, + "values": { + "type": "array", + "items": { + "type": "string" + } + }, + "default": { + "type": [ + "string", + "integer", + "boolean" + ] + }, + "pattern": { + "type": "string" + }, + "help": { + "type": "string" + }, + "dimensions": { + "type": "object", + "additionalProperties": { + "type": "string" + } + }, + "ask_in": { + "$ref": "#/$defs/id" + }, + "produced_by": { + "$ref": "#/$defs/id" + }, + "confirmed_by": { + "$ref": "#/$defs/id" + }, + "secret": { + "const": false + } + } + }, + "check": { + "type": "object", + "additionalProperties": false, + "properties": { + "done_when": { + "type": "string", + "minLength": 3 + }, + "command": { + "type": "string" + }, + "expect": { + "enum": [ + "manual", + "exit-0", + "output-contains" + ] + }, + "contains": { + "type": "string" + }, + "datapoints_ok": { + "type": "array", + "items": { + "$ref": "#/$defs/dpid" + }, + "minItems": 1 + } + }, + "anyOf": [ + { + "required": [ + "done_when" + ] + }, + { + "required": [ + "datapoints_ok" + ] + } + ] + }, + "alternative": { + "type": "object", + "required": [ + "id", + "title", + "verify" + ], + "additionalProperties": false, + "properties": { + "id": { + "$ref": "#/$defs/id" + }, + "title": { + "type": "string" + }, + "command": { + "type": "string" + }, + "reversible": { + "type": "boolean" + }, + "rollback": { + "type": "string" + }, + "verify": { + "$ref": "#/$defs/check" + }, + "recommended_when": { + "type": "object", + "additionalProperties": { + "type": "array", + "items": { + "type": "string" + } + } + } + } + }, + "step": { + "type": "object", + "required": [ + "id", + "title", + "actor", + "owner" + ], + "additionalProperties": false, + "properties": { + "id": { + "$ref": "#/$defs/id" + }, + "title": { + "type": "string" + }, + "actor": { + "$ref": "#/$defs/id" + }, + "owner": { + "type": "string" + }, + "description": { + "type": "string" + }, + "command": { + "type": "string" + }, + "reversible": { + "type": "boolean", + "default": true + }, + "rollback": { + "type": "string" + }, + "confirm_phrase": { + "type": "string", + "minLength": 8 + }, + "verify": { + "$ref": "#/$defs/check" + }, + "requires": { + "type": "array", + "items": { + "type": "string" + } + }, + "uses": { + "type": "array", + "items": { + "$ref": "#/$defs/dpid" + } + }, + "produces": { + "type": "array", + "items": { + "type": "string" + } + }, + "evidence": { + "type": "array", + "items": { + "type": "string" + } + }, + "one_of": { + "type": "array", + "minItems": 2, + "items": { + "$ref": "#/$defs/alternative" + } + }, + "autopilot": { + "enum": [ + "allowed", + "manual-only" + ], + "default": "allowed" + }, + "timeout": { + "$ref": "#/$defs/timeout" + }, + "downstream_risk": { + "type": "string" + } + }, + "anyOf": [ + { + "required": [ + "verify" + ] + }, + { + "required": [ + "one_of" + ] + } + ] + }, + "gate": { + "type": "object", + "required": [ + "id", + "title", + "goal", + "checks" + ], + "additionalProperties": false, + "properties": { + "id": { + "$ref": "#/$defs/id" + }, + "title": { + "type": "string" + }, + "goal": { + "type": "string", + "minLength": 3 + }, + "blocking": { + "type": "boolean", + "default": true + }, + "checks": { + "type": "array", + "minItems": 1, + "items": { + "$ref": "#/$defs/check" + } + }, + "waivable": { + "type": "object", + "required": [ + "by", + "needs_reason" + ], + "additionalProperties": false, + "properties": { + "by": { + "$ref": "#/$defs/id" + }, + "needs_reason": { + "const": true + } + } + }, + "timeout": { + "$ref": "#/$defs/timeout" + }, + "downstream_risk": { + "type": "string" + } + } + }, + "stage": { + "type": "object", + "required": [ + "id", + "title", + "goal" + ], + "additionalProperties": false, + "properties": { + "id": { + "$ref": "#/$defs/id" + }, + "title": { + "type": "string" + }, + "goal": { + "type": "string", + "minLength": 3 + }, + "framing": { + "type": "array", + "items": { + "type": "string" + } + }, + "requires": { + "type": "array", + "items": { + "type": "string" + } + }, + "steps": { + "type": "array", + "items": { + "$ref": "#/$defs/step" + } + }, + "gates": { + "type": "array", + "items": { + "$ref": "#/$defs/gate" + } + }, + "stages": { + "type": "array", + "items": { + "$ref": "#/$defs/stage" + } + } + }, + "anyOf": [ + { + "required": [ + "steps" + ] + }, + { + "required": [ + "stages" + ] + } + ] + }, + "artefact": { + "type": "object", + "required": [ + "id", + "title", + "location", + "produced_by" + ], + "additionalProperties": false, + "properties": { + "id": { + "$ref": "#/$defs/id" + }, + "title": { + "type": "string" + }, + "location": { + "type": "object", + "required": [ + "datapoint" + ], + "additionalProperties": false, + "properties": { + "datapoint": { + "$ref": "#/$defs/dpid" + }, + "path": { + "type": "string" + } + } + }, + "produced_by": { + "$ref": "#/$defs/id" + }, + "lives_until": { + "$ref": "#/$defs/id" + } + } + }, + "ownership": { + "type": "object", + "required": [ + "concern", + "owner" + ], + "additionalProperties": false, + "properties": { + "concern": { + "type": "string" + }, + "owner": { + "type": "string" + } + } + } + } +} diff --git a/tests/test_showcase.py b/tests/test_showcase.py index 58f9224..68cb533 100644 --- a/tests/test_showcase.py +++ b/tests/test_showcase.py @@ -3,7 +3,6 @@ import shutil import subprocess import pytest -import yaml from rtut import pack as P, run as R @@ -22,28 +21,6 @@ def test_showcase_is_valid(): assert P.load(SRC / "pack.yaml").ok -def test_v02_draft_is_wellformed_and_self_consistent(): - d = yaml.safe_load((SRC / "pack.v0.2.draft.yaml").read_text()) - assert d["spec"] == "runbook-pack/v0.2" - actors = {a["id"] for a in d["actors"]} - dps = {x["id"] for x in d["datapoints"]} - steps = {} - for st in d["stages"]: - for s in st["steps"]: - steps[s["id"]] = s - assert s["actor"] in actors - for u in s.get("uses", []): - assert u in dps, u - gates = {g["id"] for st in d["stages"] for g in st.get("gates", [])} - stages = {st["id"] for st in d["stages"]} - for st in d["stages"]: - for r in st.get("requires", []): - assert r.removeprefix("gate:") in gates | stages - assert any(s.get("reversible") is False for s in steps.values()) - assert any("one_of" in s for s in steps.values()) - assert not any(k.startswith("secret") for x in d["datapoints"] for k in x) - - def test_scripted_playthrough(pk): r = R.create(pk, "tester", {"keeper": "Ada", "weather": "storm", "motto": "Light on"}) for s in pk.data["steps"]: diff --git a/tests/test_v2.py b/tests/test_v2.py new file mode 100644 index 0000000..dc7c13e --- /dev/null +++ b/tests/test_v2.py @@ -0,0 +1,106 @@ +import copy +from pathlib import Path + +import pytest +import yaml + +from rtut import pack as P, run as R, v2 + +ROOT = Path(__file__).resolve().parent.parent +FILE = ROOT / "runbooks/showcase-lighthouse/pack.v0.2.yaml" +GOOD = yaml.safe_load(FILE.read_text()) + + +def errs(mutate): + d = copy.deepcopy(GOOD) + mutate(d) + return v2.validate(d) + + +def step(d, sid): + for st in d["stages"]: + for s in st.get("steps", []): + if s["id"] == sid: + return s + raise KeyError(sid) + + +def gate(d, gid): + return next(g for st in d["stages"] for g in st.get("gates", []) if g["id"] == gid) + + +def test_showcase_v02_is_valid_and_loads_through_pack_load(): + pk = P.load(FILE) + assert pk.ok, pk.errors + assert pk.spec == "runbook-pack/v0.2" + + +def test_v02_packs_cannot_be_run_yet(): + with pytest.raises(R.RunError, match="cannot be run yet"): + R.create(P.load(FILE), "tester") + + +CASES = [ + ("unknown actor", lambda d: step(d, "light-lamp").update(actor="nobody"), "unknown actor"), + ("unknown datapoint in uses", lambda d: step(d, "light-lamp").update(uses=["nope"]), "uses unknown datapoint"), + ("placeholder not in uses", lambda d: step(d, "light-lamp").update(uses=["keeper_name"]), "not listed in uses"), + ("undefined placeholder", lambda d: step(d, "light-lamp").update(command="echo {{ghost}}"), "undefined placeholder"), + ("requires unknown", lambda d: step(d, "write-logbook").update(requires=["gate:ghost"]), "requires unknown item"), + ("duplicate id", lambda d: d["stages"][0]["gates"][0].update(id="dusk"), "duplicate id"), + ("irreversible without phrase", lambda d: (d.pop("confirm_phrase"), None), "confirm_phrase"), + ("waivable by unknown actor", lambda d: gate(d, "lamp-check").update(waivable={"by": "ghost", "needs_reason": True}), "unknown actor"), + ("datapoints_ok unknown", lambda d: gate(d, "lamp-check")["checks"].append({"datapoints_ok": ["ghost"]}), "unknown datapoint"), + ("ask_in unknown stage", lambda d: d["datapoints"][0].update(ask_in="ghost"), "unknown stage"), + ("artefact produced_by unknown", lambda d: d["artefacts"][0].update(produced_by="ghost"), "unknown step"), + ("recommended_when bad value", lambda d: step(d, "warn")["one_of"][0].update(recommended_when={"weather": ["hail"]}), "not in values"), + ("step with one_of and command", lambda d: step(d, "warn").update(command="echo x"), "must not also have its own command"), + ("secret marker", lambda d: d["framing"].append("token hvs.ABCDEFGHIJKLMN"), "secret-looking"), + ("exit-0 without command", lambda d: step(d, "write-logbook")["verify"].pop("command"), "needs a command"), + ("secret datapoint", lambda d: d["datapoints"][0].update(secret=True), "secret"), + ("gate without goal", lambda d: gate(d, "lamp-check").pop("goal"), "goal"), + ("waiver must need a reason", lambda d: gate(d, "lamp-check").update(waivable={"by": "harbourmaster", "needs_reason": False}), "needs_reason"), + ("bad timeout action", lambda d: gate(d, "lamp-check").update(timeout={"seconds": 5, "on_timeout": "explode"}), "on_timeout"), + ("bad autopilot value", lambda d: step(d, "close-log").update(autopilot="always"), "autopilot"), +] + + +@pytest.mark.parametrize("name,fn,frag", CASES, ids=[c[0] for c in CASES]) +def test_invalid(name, fn, frag): + out = errs(fn) + assert any(frag in e for e in out), out + + +def test_cycle_detected(): + out = errs(lambda d: d["stages"][0].update(requires=["night"])) + assert any("cycle" in e for e in out), out + + +def test_child_requiring_its_parent_is_a_cycle(): + def m(d): + step(d, "read-the-sky").update(requires=["dusk"]) + assert any("cycle" in e for e in errs(m)) + + +def test_depth_limit(): + def m(d): + inner = {"id": "l4", "title": "Level four", "goal": "too deep", "steps": [copy.deepcopy(step(d, "read-the-sign"))]} + inner["steps"][0]["id"] = "deep-step" + l3 = {"id": "l3", "title": "Level three", "goal": "three", "stages": [inner]} + l2 = {"id": "l2", "title": "Level two", "goal": "two", "stages": [l3]} + d["stages"].append({"id": "l1", "title": "Level one", "goal": "one", "stages": [l2]}) + assert any("nesting deeper" in e for e in errs(m)) + + +def test_three_levels_allowed(): + def m(d): + l3 = {"id": "l3", "title": "Level three", "goal": "three", "steps": [copy.deepcopy(step(d, "read-the-sign"))]} + l3["steps"][0]["id"] = "deep-step" + l2 = {"id": "l2", "title": "Level two", "goal": "two", "stages": [l3]} + d["stages"].append({"id": "l1", "title": "Level one", "goal": "one", "stages": [l2]}) + assert errs(m) == [] + + +def test_legacy_v02_needs_only_schema(): + d = {"spec": "runbook-pack/v0.2", "id": "x.y", "title": "Legacy x", "owner": "o", + "exercise_status": "unexercised", "engine": "legacy-command", "legacy": {"command": ["true"]}} + assert v2.validate(d) == [] diff --git a/workplans/RBT-WP-0002-runbook-pack-spec-and-validator.md b/workplans/RBT-WP-0002-runbook-pack-spec-and-validator.md index 7fa9499..4fc7e70 100644 --- a/workplans/RBT-WP-0002-runbook-pack-spec-and-validator.md +++ b/workplans/RBT-WP-0002-runbook-pack-spec-and-validator.md @@ -61,27 +61,24 @@ unsafe characters. ```task id: RBT-WP-0002-T04 -status: progress +status: done priority: high ``` -`docs/pack-spec-v0.2-proposal.md` (stages and hierarchy, dependency graph with a +`docs/pack-spec-v0.2.md` (stages and hierarchy, dependency graph with a recommended serialization, datapoints with `nil/set/err/ok`, QA gates tied to stages and goals, actors and roles, artefacts, reversibility-based confirmation, framing). -Review with Bernd, resolve the five open questions, and decide: freeze v0.1 as -shipped and adopt v0.2. Supersedes the earlier plan to freeze v0.1 alone. +Done 2026-09-29: Bernd answered the five questions; v0.2 adopted as `docs/pack-spec-v0.2.md` (with operating modes, overrides, deviations); v0.1 stays as shipped. Supersedes the earlier plan to freeze v0.1 alone. ## Write the v0.2 schema and validator ```task id: RBT-WP-0002-T05 -status: wait +status: done priority: high ``` -After T04: `schema/runbook-pack-v0.2.schema.json`, semantic checks (acyclic -requires, gate and datapoint references resolve, one stage per gate, no secret -datapoints), v0.1 packs still load, tests. +Done 2026-09-29: `schema/runbook-pack-v0.2.schema.json` and `rtut/v2.py` (acyclic requires, references resolve, nesting depth 3, irreversible steps need a confirm phrase, no secret datapoints). v0.1 packs still load. 61 tests including one negative test per rule. ## Port the in-repo and NetKingdom packs @@ -106,10 +103,9 @@ state_hub_task_id: "6740457f-81a7-5761-a768-b9b0faee7552" `runbooks/showcase-lighthouse/` is a harmless interactive-fiction runbook (poem, story, scratch logbook under /tmp/rtut-*). `pack.yaml` (v0.1) runs today and is the test bed for -UI refinement. `pack.v0.2.draft.yaml` expresses the same story in the proposed v0.2 model +UI refinement. `pack.v0.2.yaml` expresses the same story in the proposed v0.2 model (stages, parallel steps, alternatives, gates, actors, datapoints, artefacts, irreversible -confirmation). The draft is checked for internal consistency by tests. When v0.2 is adopted -the validator must accept it unchanged, or the proposal changes. +confirmation). It now validates against the v0.2 schema and semantic checks. It cannot be run until RBT-WP-0007 lands. ## Acceptance Criteria diff --git a/workplans/RBT-WP-0004-local-runbook-ui.md b/workplans/RBT-WP-0004-local-runbook-ui.md index 432683f..efb24c5 100644 --- a/workplans/RBT-WP-0004-local-runbook-ui.md +++ b/workplans/RBT-WP-0004-local-runbook-ui.md @@ -29,7 +29,7 @@ priority: high state_hub_task_id: "8309d778-3688-5cb3-bb34-18f87b66dbf3" ``` -Done 2026-09-29 with Bernd. Findings and consequences are in `docs/ux-review-2026-09-29.md`; the model they imply is proposed in `docs/pack-spec-v0.2-proposal.md`. +Done 2026-09-29 with Bernd. Findings and consequences are in `docs/ux-review-2026-09-29.md`; the model they imply is proposed in `docs/pack-spec-v0.2.md`. ## Build the step view @@ -146,6 +146,18 @@ state_hub_task_id: "ddbca5e1-dcf6-5c2a-b3e2-497d2a82ba23" Confirm button for reversible steps, typed phrase for irreversible ones, per the v0.2 `reversible` field. +## Mode switch and override dialog + +```task +id: RBT-WP-0004-T11 +status: wait +priority: high +``` + +Depends on RBT-WP-0007 T06-T07. Choose manual or autopilot per run (with timeout and +push-through controls). In manual mode, an override dialog lists unmet preconditions and +the downstream risk before continuing; deviations show in the overview. + ## Acceptance Criteria - Bernd can complete the SSH tutorial pack in the UI without pasting outputs to an agent. diff --git a/workplans/RBT-WP-0006-execution-mode-and-pack-index.md b/workplans/RBT-WP-0006-execution-mode-and-pack-index.md index d4cd861..5b685df 100644 --- a/workplans/RBT-WP-0006-execution-mode-and-pack-index.md +++ b/workplans/RBT-WP-0006-execution-mode-and-pack-index.md @@ -56,3 +56,15 @@ state_hub_task_id: "ce82d1ae-76a3-59c8-9d26-4813e8dc9745" Operator feedback on a step feeds an assistant that knows the runbook and may diagnose the deployment (read-only access first), giving a feedback, retry, feedback, confirm loop. Needs decisions on access, identity and what is sent where. + +## Dependencies between runbooks + +```task +id: RBT-WP-0006-T04 +status: wait +priority: low +``` + +Packs declare `requires_packs` with a required result (exercised, or a named gate +passed), instead of deeper nesting (v0.2 caps nesting at three levels). Needs a decision +on how a pack in another repo is identified and trusted. diff --git a/workplans/RBT-WP-0007-engine-v0-2-graph-datapoints-and-gates.md b/workplans/RBT-WP-0007-engine-v0-2-graph-datapoints-and-gates.md index e662b24..e7b09db 100644 --- a/workplans/RBT-WP-0007-engine-v0-2-graph-datapoints-and-gates.md +++ b/workplans/RBT-WP-0007-engine-v0-2-graph-datapoints-and-gates.md @@ -23,7 +23,7 @@ a dimension view and an overview from the same run data. ```task id: RBT-WP-0007-T01 -status: wait +status: todo priority: high state_hub_task_id: "55dd86f9-5f89-59a2-8a0a-85fbedd95483" ``` @@ -35,7 +35,7 @@ canonical order; explain what a blocked item waits for. ```task id: RBT-WP-0007-T02 -status: wait +status: todo priority: high state_hub_task_id: "571e8685-6eeb-54ce-bb1a-4616ea951f81" ``` @@ -47,7 +47,7 @@ how `ok` is reached (check or recorded manual confirmation). ```task id: RBT-WP-0007-T03 -status: wait +status: todo priority: high state_hub_task_id: "707cca55-24a1-5355-8722-10740c5a2862" ``` @@ -59,7 +59,7 @@ exit, record waivers with who, when and why. ```task id: RBT-WP-0007-T04 -status: wait +status: todo priority: medium state_hub_task_id: "73307bb9-2739-563b-92c7-a034a360b444" ``` @@ -70,7 +70,7 @@ state_hub_task_id: "73307bb9-2739-563b-92c7-a034a360b444" ```task id: RBT-WP-0007-T05 -status: wait +status: todo priority: medium state_hub_task_id: "0933ff4a-8478-50d9-aa99-991948bc2379" ``` @@ -78,6 +78,33 @@ state_hub_task_id: "0933ff4a-8478-50d9-aa99-991948bc2379" Derive confirm button versus typed phrase from `reversible`; enforce the phrase in the engine so no renderer can skip it. +## Operating modes: manual and autopilot + +```task +id: RBT-WP-0007-T06 +status: todo +priority: high +``` + +A run has a mode (`docs/pack-spec-v0.2.md`). Manual: the operator confirms; every +check, gate and precondition is shown with its state. Autopilot: the engine confirms +when all preconditions are met. `autopilot: manual-only` steps and irreversible steps +are never auto-confirmed without a recorded pre-authorization. + +## Overrides, waivers, push-through and the deviation record + +```task +id: RBT-WP-0007-T07 +status: todo +priority: high +``` + +Override in manual mode when preconditions are unmet: name the unmet items and the +declared `downstream_risk`, require explicit confirmation and a reason, record a +deviation. Waivers only where a gate declares `waivable`. Autopilot timeouts +(`stop` or `push-through`) record a deviation and, for push-through, move on. A run with +any deviation is flagged and can never exercise a pack. + ## Acceptance Criteria - The old console's stages, gates and datapoints can be expressed and computed