coulomb/ops-mason
4
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Enforce readiness tiers and reconcile blocked workplans

Browse source 
Assistant: codex
Assistant-Model: gpt-6-astra
Assistant-Session: 01a0e75a-fc5c-7913-9dba-9846210c766d
This commit is contained in:
tegwick 2026-09-28 11:40:57 +02:00
parent 370e1f84c7
commit 36445ae679
14 changed files with 652 additions and 39 deletions
Download patch file Download diff file Expand all files Collapse all files

6
INTENT.md
View file

@ -130,13 +130,13 @@ tooling_contacts:
# gate on ADMINISTER @ realm:kubernetes/railiance01 is a quality gate, not
# an authorization decision, tiered by the target's railiance-master
# ADR-0006 readiness_state. The owner question above is answered: the
# Kubernetes API stays a Tooling contact owned by rail-kubernetes.
# contact is with realm:kubernetes/railiance01; no layer is assigned.
change_gate:
decision: the-custodian/docs/kubernetes-change-gate-decision.md
decided_by: "Bernd Worsch (founder), GOVERN @ estate"
decided_at: "2026-09-21"
engine_owner: none
tooling_owner: rail-kubernetes
realm: "realm:kubernetes/railiance01"
tiers:
- readiness_state: [declared, installed, verified]
path: "direct ADMINISTER @ realm:kubernetes by ops-mason"
@ -156,7 +156,7 @@ tooling_contacts:
until: "2026-12-21"
rule: "Direct ADMINISTER under activation=APPROVED, each change recorded as production-tier, until ArgoCD onboarding."
relies_on_limits: "One expected namespace per plan; Pod and Secret kinds refused; no data or stringData. Widening them is a new decision."
enforcement: "Not yet in code: phase 4 does not check readiness_state. Planned in workplans/MASON-WP-0006-readiness-tier-check.md."
enforcement: "src/ops_mason/readiness.py: inspect_readiness and resolve_tier; kubernetes_plane.apply refuses before Kubernetes writes. Source/history verification, explicit whitehat placement, dated policy-nexus transition, and recorded BREAK_GLASS."
- id: bao-session-grant
shape: "5.2"
module: scripts/bao-session.sh

7
bundles/whitehat-foundational-plane.yaml
View file

@ -59,3 +59,10 @@ dependencies:
equals: "true"
- path: /spec/ingress/0/from/0/podSelector/matchLabels/whitehat.security~1target
equals: audit-core
readiness:
target: {kind: namespace, namespace: whitehat}
source:
repo: reef-railiance
path: bindings/rapps.yaml
revision: e3c5ca2b74f6ae4f3b79f918708f3b573157a0c2
sha256: 796007c68f0eda1d50cebddf9e11a2c123a8814ebaf67f739ed5d2af2d6fa3f1

91
docs/evidence/2026-09-28-loose-end-review.md Normal file
View file

@ -0,0 +1,91 @@
# Loose-end review — 2026-09-28
Reviewed every local workplan. MASON-0001 and MASON-WP-0001–0003 are
finished; no open task in those files needs implementation. The three proposed
workplans, MASON-WP-0004–0006, contain the remaining work. No new task or
workplan was opened. Existing uncommitted intake, Telegram construction-plan,
and lockfile work was left outside this change.
## MASON-WP-0006
T01 is answered by founder decision in
`the-custodian/docs/kubernetes-change-gate-decision.md`, communicated by message
`b41bd54f-a7a4-41e5-a1ee-aa9b0efa7dc5`: whitehat is explicitly non-production.
The same decision and current agent orientation establish that ArgoCD has
since been installed; the blanket absence-of-ArgoCD transition is obsolete.
T02–T05 implemented in the readiness resolver, bundle, CLI, evidence and docs.
Validation: 54 tests pass (`.venv/bin/pytest -q`), including a real temporary
Git history that promotes then reverts a binding; `git diff --check` passes.
Tests exercise approved and refused apply, explicit whitehat placement,
binding supersession, stale/missing sources, namespace mapping, historical
production approval, deprecation, the December 21 boundary, and emergency
reason/evidence. Existing manifest and approval refusals remain covered.
Secret-presence verification now requests object names, never Secret JSON.
A read-only local source check resolves the actual whitehat bundle to
non-production under APPROVED, using reef-railiance commit
`e3c5ca2b74f6ae4f3b79f918708f3b573157a0c2`, bindings SHA-256
`796007c68f0eda1d50cebddf9e11a2c123a8814ebaf67f739ed5d2af2d6fa3f1`.
No Kubernetes command or deployment was needed for this implementation.
T06 stays wait: its review is due 2026-12-21 and requires platform confirmation
of policy-nexus GitOps adoption and review of the accepted plan-approval
exception. The workplan remains blocked, not finished; the existing task holds
this obligation.
## MASON-WP-0004
The session check at `http://127.0.0.1:18200` reports no valid caller session
and no scoped ops-mason grant. No credential value was requested. The original
21-path/17-undescribed inventory has no saved path-level snapshot in this repo;
source documents cannot prove the current live path set or completeness.
T01–T03 therefore remain wait for an attended scoped metadata grant, current
inventory, and owner confirmations. No ownership or recovery story is invented.
Source-backed candidates for the next T01 inventory comparison:
| Path or family | Proposed responsible repo | Evidence / unresolved question |
| --- | --- | --- |
| operators/lldap/admin | net-kingdom | platform-root-custody.md; confirm current consumers and recovery |
| operators/privacyidea/pi-admin | net-kingdom | platform-root-custody.md and verify-t06.md; confirm recovery ownership |
| operators/forgejo/state-hub-svc | railiance-platform | MASON-WP-0003 construction/delivery record; confirm active metadata |
| platform/workloads/railiance/backup/object-storage | railiance-platform | plans/backup-object-storage.md |
| platform/workloads/railiance/backup/offsite-lane | railiance-platform | ops-warden catalog railiance-backup-offsite-lane |
| platform/workloads/railiance/scaleway/bootstrap | railiance-platform | plans/reef-storage-scaleway-bootstrap.md |
| user-engine/runtime and rapp-qonto families | owning consumer plus railiance-platform custody | corresponding plans in this repo; exact current paths need live inventory |
| reuse-surface/runtime-secrets | reuse-surface plus railiance-platform custody | cited by T02; current owner procedure still needs confirmation |
These are candidates, not accepted custom_metadata, and do not enumerate the
missing seventeen. Every additional live path needs an owner or a named unknown
before T01 can close.
T04 preparation fixes an actual false-success defect: an unavailable/denied
metadata request previously returned an empty list and exit 0. The inventory
now exits 2 on missing grant, command failure or malformed response, exits 1
for missing descriptions, and reserves 0 for a completed inventory. It refuses
to fall back silently to the user's default token. The default Bao address in
both helpers now follows the private tunnel. Tests cover failure reporting and
metadata-only traversal. Running it without a grant returned the expected
explicit error, not a fictitious healthy inventory.
Scheduled reporting remains blocked on T03 and a named reader plus a
non-interactive metadata-only credential. No existing scheduler for this repo
was found; a 45-minute interactive grant cannot support a durable schedule.
## MASON-WP-0005
The September 9 platform reply (`c0977d84-9a63-421d-8ee1-98587fc60b1b`)
and `railiance-platform/docs/credential-lane-designs/fluid-telegram-operator-kv.md`
confirm a proposed matrix, not an accepted writer contract. Tenant/path,
field/capability acceptance, actual OIDC group/MFA and callbacks, named writer
authority, matching platform validator/renderer, and live survey remain open.
The existing ops-mason grant does not authorize auth/netkingdom role changes;
it must not be widened to bypass this boundary.
T01–T05 remain wait for those inputs and explicit construction approval.
T02 cannot be called done by shipping an unapproved engine shape: the owner
requires matching approved CCR and builder contracts before any writer.
T06 also waits for verification and routing-owner acceptance; no live lane or
resolvable pointer is claimed. Local draft preparation remains in the existing
construction plan. The separate adapter demand stays in its existing intake.

62
docs/kubernetes-plane.md
View file

@ -46,3 +46,65 @@ ops-mason plane rollback-plan --bundle bundles/<id>.yaml
Rollback output is a plan, never an action. Review live inventory immediately
before using it.
## Readiness gate
`apply` checks `ops_mason.readiness.inspect_readiness` after approval/digest
checks and before any Kubernetes command. `preflight` reports the result even
when direct apply would be refused.
| Verified target state | Direct apply under APPROVED |
| --- | --- |
| declared, installed, verified | Allowed with the existing approved-plan checks |
| production-approved, including a later evidence lapse | Refused; use the manifest repository and ArgoCD |
| deprecated | Retains the previous tier; missing history means production |
| missing, unknown, invalid or stale readiness | Production; refused |
| whitehat namespace, explicit founder placement of 2026-09-21 | Non-production, until a binding supersedes the placement |
| rapp-policy-nexus, production tier | Transition only before 2026-12-21; evidence labels it production |
ArgoCD now exists on railiance01, so the historical blanket transition for
all production targets is not enabled. The policy-nexus transition remains
bounded by its review date. The gate does not decide authorization or contact
an authorization engine.
Bundles include a reviewed namespace-to-binding mapping and a source pin:
```yaml
readiness:
target: {kind: rapp, rapp_id: rapp-user-engine, namespace: user-engine}
source:
repo: reef-railiance
path: bindings/rapps.yaml
revision: <full-40-character-commit-id>
sha256: <sha256-of-file-at-that-commit>
```
The approved bundle is the mapping record: the namespace must match its object
scope, and any namespace mapping in the source must agree. The only explicit
namespace placement is `kind: namespace, namespace: whitehat` for bundle
`whitehat-foundational-plane`, using the same pinned source so a newly declared
whitehat binding invalidates the placement. Other namespace-only targets and
platform objects remain production-tier.
Use `--readiness-repo /path/to/reef-railiance` on `preflight` or `apply` to
locate the source; the default is the sibling checkout. The gate verifies its
file digest, commit ancestry, unchanged content through local HEAD, clean source
file, and complete Git history. Refresh that checkout before review: the gate
checks local HEAD, not an unfetched remote. Missing source/history fails closed.
Any production approval in the reachable file history retains production tier;
a deliberate re-scope needs a separately reviewed gate change, not just lowering
the readiness field. Source pins and mapping changes alter the bundle digest
and need fresh review/confirmation. Existing live evidence remains historical.
For emergency direct apply, keep all normal plan/digest requirements and add:
```bash
ops-mason plane apply --bundle bundles/<id>.yaml \
--confirm <approved-plan-id> --expect-digest <digest> \
--activation BREAK_GLASS --break-glass-reason '<incident and justification>'
```
The JSON evidence and CLI output record activation, local executing account,
time, reason, resolved tier/source, and the obligation to commit the same
change to the manifest repository ArgoCD reconciles. The command does not open
that change or grant emergency authority itself. An empty reason is refused.

2
scripts/bao-session.sh
View file

@ -28,7 +28,7 @@
set -euo pipefail
export BAO_ADDR="${BAO_ADDR:-https://bao.coulomb.social}"
export BAO_ADDR="${BAO_ADDR:-http://127.0.0.1:18200}"
GRANT_FILE="${GRANT_FILE:-$HOME/.claude-bao-token}"
GRANT_POLICY="${GRANT_POLICY:-ops-mason-build}"
GRANT_TTL="${GRANT_TTL:-45m}"

39
scripts/custody-inventory.py
View file

@ -24,31 +24,42 @@ REQUIRED = ("description", "owner", "used_by", "rotation", "on_loss")
DEFAULT_ROOTS = ("operators", "platform/workloads")
def bao(*args: str) -> dict | list | None:
class InventoryError(RuntimeError):
"""Inventory could not be completed; never report this as an empty store."""
def bao(*args: str) -> dict | list:
env = dict(os.environ)
env.setdefault("BAO_ADDR", "https://bao.coulomb.social")
env.setdefault("BAO_ADDR", "http://127.0.0.1:18200")
grant = os.path.expanduser("~/.claude-bao-token")
if "BAO_TOKEN" not in env and os.path.exists(grant):
with open(grant) as f:
env["BAO_TOKEN"] = f.read().strip()
if not env.get("BAO_TOKEN"):
raise InventoryError("No scoped BAO_TOKEN or ops-mason grant; inventory was not run.")
# The Vault/OpenBao CLI rejects flags placed after a positional argument,
# so -format=json goes immediately before the path, not at the end.
argv = ["bao", *args[:-1], "-format=json", args[-1]]
p = subprocess.run(argv, capture_output=True, text=True, timeout=30, env=env)
try:
p = subprocess.run(argv, capture_output=True, text=True, timeout=30, env=env)
except (OSError, subprocess.TimeoutExpired) as exc:
raise InventoryError("OpenBao metadata command unavailable or timed out") from exc
if p.returncode != 0:
return None
raise InventoryError(f"OpenBao metadata request failed for {args[-1]}; inventory incomplete")
try:
return json.loads(p.stdout)
except json.JSONDecodeError:
return None
except json.JSONDecodeError as exc:
raise InventoryError("Invalid OpenBao metadata response; inventory incomplete") from exc
def walk(prefix: str) -> list[str]:
keys = bao("kv", "list", prefix)
if not isinstance(keys, list):
return []
raise InventoryError(f"Invalid metadata listing for {prefix}")
out: list[str] = []
for k in keys:
if not isinstance(k, str) or not k.rstrip("/") or "/" in k.rstrip("/") or k.rstrip("/") in {".", ".."}:
raise InventoryError(f"Invalid child in metadata listing for {prefix}")
child = f"{prefix.rstrip('/')}/{k.rstrip('/')}"
out.extend(walk(child) if k.endswith("/") else [child])
return out
@ -62,9 +73,13 @@ def main() -> int:
for root in roots:
for path in walk(root):
total += 1
meta = bao("kv", "metadata", "get", path) or {}
d = meta.get("data", {}) if isinstance(meta, dict) else {}
meta = bao("kv", "metadata", "get", path)
if not isinstance(meta, dict) or not isinstance(meta.get("data"), dict):
raise InventoryError(f"Invalid metadata response for {path}")
d = meta["data"]
cm = d.get("custom_metadata") or {}
if not isinstance(cm, dict):
raise InventoryError(f"Invalid custom metadata for {path}")
missing = [k for k in REQUIRED if not cm.get(k)]
if missing:
incomplete += 1
@ -90,4 +105,8 @@ def main() -> int:
if __name__ == "__main__":
raise SystemExit(main())
try:
raise SystemExit(main())
except InventoryError as exc:
print(f"ERROR: {exc}", file=sys.stderr)
raise SystemExit(2)

11
src/ops_mason/cli.py
View file

@ -5,6 +5,7 @@ from __future__ import annotations
import argparse
import json
import sys
from pathlib import Path
from collections.abc import Sequence
from ops_mason.kubernetes_plane import (
@ -26,6 +27,8 @@ def _parser() -> argparse.ArgumentParser:
for name in ("render", "preflight", "verify", "rollback-plan"):
command = commands.add_parser(name)
command.add_argument("--bundle", required=True)
if name == "preflight":
command.add_argument("--readiness-repo", type=Path)
apply_parser = commands.add_parser("apply")
apply_parser.add_argument("--bundle", required=True)
@ -33,6 +36,9 @@ def _parser() -> argparse.ArgumentParser:
apply_parser.add_argument(
"--expect-digest", required=True, help="exact digest returned by preflight"
)
apply_parser.add_argument("--readiness-repo", type=Path)
apply_parser.add_argument("--activation", choices=("APPROVED", "BREAK_GLASS"), default="APPROVED")
apply_parser.add_argument("--break-glass-reason")
return parser
@ -43,7 +49,7 @@ def main(argv: Sequence[str] | None = None) -> int:
if args.command == "render":
result = bundle.render()
elif args.command == "preflight":
result = preflight(bundle)
result = preflight(bundle, readiness_repo=args.readiness_repo)
elif args.command == "verify":
result = verify(bundle)
elif args.command == "rollback-plan":
@ -53,6 +59,9 @@ def main(argv: Sequence[str] | None = None) -> int:
bundle,
confirm_plan_id=args.confirm,
expected_digest=args.expect_digest,
readiness_repo=args.readiness_repo,
activation=args.activation,
break_glass_reason=args.break_glass_reason,
)
else: # pragma: no cover - argparse enforces the command set
raise AssertionError(args.command)

46
src/ops_mason/kubernetes_plane.py
View file

@ -8,18 +8,20 @@ and records metadata-only evidence.
from __future__ import annotations
import getpass
import hashlib
import json
import subprocess
from collections.abc import Callable, Mapping, Sequence
from dataclasses import asdict, dataclass
from datetime import UTC, datetime
from datetime import UTC, date, datetime
from pathlib import Path
from typing import Any
import yaml
from ops_mason.plan import ConstructionPlan
from ops_mason.readiness import inspect_readiness
class PlaneError(RuntimeError):
@ -93,6 +95,7 @@ class PlaneBundle:
dependencies: tuple[Dependency, ...]
evidence_path: Path
documents: tuple[dict[str, Any], ...]
readiness: dict[str, Any] | None = None
@classmethod
def load(cls, path: str | Path) -> "PlaneBundle":
@ -172,6 +175,7 @@ class PlaneBundle:
dependencies=dependencies,
evidence_path=local_path(str(raw["evidence_path"])),
documents=tuple(documents),
readiness=raw.get("readiness"),
)
bundle.validate()
return bundle
@ -185,6 +189,12 @@ class PlaneBundle:
return digest.hexdigest()
def validate(self) -> None:
if self.readiness is not None and (
not isinstance(self.readiness, dict)
or not isinstance(self.readiness.get("target"), dict)
or not isinstance(self.readiness.get("source"), dict)
):
raise PlaneError("readiness needs target and source mappings")
actual_refs = tuple(_document_ref(doc, self.allowed_objects) for doc in self.documents)
if len(set(actual_refs)) != len(actual_refs):
raise PlaneRefused("bundle contains duplicate Kubernetes object identities")
@ -233,6 +243,7 @@ class PlaneBundle:
"source_revision": self.source_revision,
"implementation_revision": self.implementation_revision,
"expected_context": self.expected_context,
"readiness": self.readiness,
"objects": [asdict(ref) | {"display": ref.display} for ref in self.allowed_objects],
"forbidden_kinds": sorted(self.forbidden_kinds),
"plan_id": self.plan().id,
@ -363,7 +374,10 @@ def _check_inputs_clean(bundle: PlaneBundle, runner: Runner) -> None:
raise PlaneRefused("repository must be committed and clean before Kubernetes mutation")
def preflight(bundle: PlaneBundle, runner: Runner = subprocess_runner) -> dict[str, Any]:
def preflight(
bundle: PlaneBundle, runner: Runner = subprocess_runner, *,
readiness_repo: Path | None = None, activation: str = "APPROVED", today: date | None = None,
) -> dict[str, Any]:
context = _run(runner, ["kubectl", "config", "current-context"]).stdout.strip()
if context != bundle.expected_context:
raise PlaneRefused(
@ -464,6 +478,7 @@ def preflight(bundle: PlaneBundle, runner: Runner = subprocess_runner) -> dict[s
if str(item.path.relative_to(bundle.repo_root)) not in server_validated
],
"dependencies": dependency_evidence,
"readiness": inspect_readiness(bundle, runner, readiness_repo, activation, today),
}
@ -489,9 +504,9 @@ def verify(bundle: PlaneBundle, runner: Runner = subprocess_runner) -> dict[str,
for resource in ("pods", "secrets"):
result = _run(
runner,
["kubectl", "-n", bundle.expected_namespace, "get", resource, "-o", "json"],
["kubectl", "-n", bundle.expected_namespace, "get", resource, "-o", "name"],
)
count = len(json.loads(result.stdout).get("items", []))
count = len(result.stdout.splitlines())
if count:
raise PlaneError(
f"negative-scope check failed: {count} {resource} exist in "
@ -538,6 +553,10 @@ def apply(
confirm_plan_id: str,
expected_digest: str,
runner: Runner = subprocess_runner,
readiness_repo: Path | None = None,
activation: str = "APPROVED",
break_glass_reason: str | None = None,
today: date | None = None,
) -> dict[str, Any]:
plan = bundle.plan()
if not plan.is_approved():
@ -552,8 +571,17 @@ def apply(
raise PlaneRefused(
f"bundle digest confirmation mismatch: expected {bundle.digest}"
)
if activation not in {"APPROVED", "BREAK_GLASS"}:
raise PlaneRefused("unknown activation")
if activation == "BREAK_GLASS" and not (break_glass_reason or "").strip():
raise PlaneRefused("BREAK_GLASS requires a non-empty reason")
_check_inputs_clean(bundle, runner)
before = preflight(bundle, runner)
readiness = inspect_readiness(bundle, runner, readiness_repo, activation, today)
if not readiness["direct_apply_allowed"]:
raise PlaneRefused(f"production tier requires GitOps or BREAK_GLASS: {readiness['reason']}")
before = preflight(bundle, runner, readiness_repo=readiness_repo, activation=activation, today=today)
if not before["readiness"]["direct_apply_allowed"]:
raise PlaneRefused("readiness changed during preflight; refusing mutation")
server_validated = list(before["server_validated_manifests"])
persisted: list[str] = []
@ -607,6 +635,14 @@ def apply(
"server_validated_manifests": server_validated,
"persisted_manifests": persisted,
},
"readiness": before["readiness"],
"activation": activation,
"break_glass": {
"reason": break_glass_reason.strip(),
"actor": getpass.getuser(),
"recorded_at": datetime.now(UTC).isoformat(),
"follow_up": "Commit the same change to the manifest repository that ArgoCD reconciles.",
} if activation == "BREAK_GLASS" else None,
"preflight": before,
"verification": verified,
"rollback": rollback_plan(bundle),

145
src/ops_mason/readiness.py Normal file
View file

@ -0,0 +1,145 @@
"""Readiness quality gate; no credential or Kubernetes access."""
from __future__ import annotations
import hashlib
import re
import subprocess
from datetime import date
from pathlib import Path
from typing import Any
import yaml
TRANSITION_END = date(2026, 12, 21)
NON_PRODUCTION = {"declared", "installed", "verified"}
def resolve_tier(
state: str | None, target: dict[str, Any], activation: str, today: date,
) -> dict[str, Any]:
"""Resolve already verified source facts. Unknown facts stay production."""
tier = "non-production" if state in NON_PRODUCTION or state == "explicit-whitehat" else "production"
transition = (
tier == "production" and state == "production-approved"
and target.get("kind") == "rapp"
and target.get("rapp_id") == "rapp-policy-nexus"
and today < TRANSITION_END and activation == "APPROVED"
)
return {
"tier": tier,
"direct_apply_allowed": activation in {"APPROVED", "BREAK_GLASS"}
and (tier == "non-production" or activation == "BREAK_GLASS" or transition),
"transition": transition,
}
def inspect_readiness(
bundle, runner, repo: Path | None, activation: str = "APPROVED",
today: date | None = None,
) -> dict[str, Any]:
"""Verify pinned source, local freshness and history before trusting a tier.
A reviewed bundle supplies the namespace-to-rApp mapping. It must identify
the namespace explicitly; source mappings, when present, must agree.
"""
today = today or date.today()
block = bundle.readiness or {}
target = block.get("target", {})
source = block.get("source", {})
evidence: dict[str, Any] = {
"target": target, "source": source, "activation": activation,
"state": None, "reason": "missing or unverifiable readiness",
}
def finish(state=None, reason="unverifiable readiness"):
evidence.update(state=state, reason=reason)
evidence.update(resolve_tier(state, target, activation, today))
return evidence
if not block or target.get("namespace") != bundle.expected_namespace:
return finish(reason="missing readiness or namespace mapping")
# A namespace placement never covers other cluster-scoped objects.
if any(not ref.namespace and (ref.kind != "Namespace" or ref.name != bundle.expected_namespace)
for ref in bundle.allowed_objects):
return finish(reason="target includes objects outside the namespace mapping")
if target.get("kind") not in {"rapp", "namespace"}:
return finish(reason="unmapped platform target")
if source.get("repo") != "reef-railiance" or source.get("path") != "bindings/rapps.yaml":
return finish(reason="unsupported readiness source")
revision = source.get("revision", "")
digest = source.get("sha256", "")
if not isinstance(revision, str) or not re.fullmatch(r"[0-9a-f]{40}", revision):
return finish(reason="source needs a full commit id")
if not isinstance(digest, str) or not re.fullmatch(r"[0-9a-f]{64}", digest):
return finish(reason="source needs a SHA-256 digest")
root = repo or bundle.repo_root.parent / "reef-railiance"
prefix = ["git", "-C", str(root)]
path = source["path"]
def git(*args):
result = runner([*prefix, *args])
if result.returncode:
raise ValueError("readiness git verification failed")
return result.stdout
def rows(content):
data = yaml.safe_load(content)
if not isinstance(data, dict) or not isinstance(data.get("bound_rapps"), list):
raise ValueError("invalid readiness document")
result = data["bound_rapps"]
if any(not isinstance(row, dict) or not isinstance(row.get("rapp_id"), str) for row in result):
raise ValueError("invalid readiness binding")
if len({row["rapp_id"] for row in result}) != len(result):
raise ValueError("duplicate readiness binding")
return result
try:
if git("rev-parse", "--is-shallow-repository").strip() != "false":
return finish(reason="complete readiness history is required")
content = git("show", f"{revision}:{path}")
if hashlib.sha256(content.encode()).hexdigest() != digest:
return finish(reason="readiness digest mismatch")
git("merge-base", "--is-ancestor", revision, "HEAD")
git("diff", "--exit-code", revision, "HEAD", "--", path)
if git("status", "--porcelain", "--", path).strip():
return finish(reason="readiness source has uncommitted changes")
bindings = rows(content)
if target["kind"] == "namespace":
# Only the founder's one named placement is compiled into this gate.
if bundle.id != "whitehat-foundational-plane" or bundle.expected_namespace != "whitehat":
return finish(reason="no explicit tier placement for target")
if any(row.get("namespace") == "whitehat" or "whitehat" in row.get("namespaces", [])
or row["rapp_id"] == "rapp-whitehat" for row in bindings):
return finish(reason="whitehat has a binding; repin using the binding target")
return finish("explicit-whitehat", "founder placement 2026-09-21")
rapp_id = target.get("rapp_id")
matches = [row for row in bindings if row["rapp_id"] == rapp_id]
if len(matches) != 1:
return finish(reason="rApp target is not listed")
row = matches[0]
if (row.get("namespace") and row["namespace"] != bundle.expected_namespace) or (
"namespaces" in row and bundle.expected_namespace not in row["namespaces"]
):
return finish(reason="source namespace mapping disagrees with bundle")
state = row.get("readiness_state")
# Scan all reachable history, including intervening promotions later
# reverted. An evidence lapse must never silently lower the tier.
history = git("log", "--format=%H", "HEAD", "--", path).splitlines()
if not history:
return finish(reason="readiness history is missing")
previous = []
for commit in history:
if not re.fullmatch(r"[0-9a-f]{40}", commit):
return finish(reason="invalid readiness history")
for old in rows(git("show", f"{commit}:{path}")):
if old["rapp_id"] == rapp_id:
previous.append(old.get("readiness_state"))
if "production-approved" in previous:
return finish("production-approved", "production tier retained from binding history")
if state == "deprecated":
state = next((s for s in previous if s != "deprecated"), None)
if state not in NON_PRODUCTION | {"production-approved"}:
return finish(reason="unknown readiness state or prior tier")
return finish(state, "verified pinned binding and history")
except (OSError, ValueError, TypeError, subprocess.TimeoutExpired, yaml.YAMLError):
return finish(reason="readiness source or history unavailable or invalid")

40
tests/test_custody_inventory.py Normal file
View file

@ -0,0 +1,40 @@
"""Inventory failures must not be reported as a healthy empty store."""
import importlib.util
from pathlib import Path
from types import SimpleNamespace
import pytest
spec = importlib.util.spec_from_file_location("inventory", Path(__file__).parents[1] / "scripts/custody-inventory.py")
inventory = importlib.util.module_from_spec(spec)
spec.loader.exec_module(inventory)
def test_no_scoped_grant_refuses_instead_of_using_operator_token(monkeypatch):
monkeypatch.delenv("BAO_TOKEN", raising=False)
monkeypatch.setattr(inventory.os.path, "exists", lambda _: False)
with pytest.raises(inventory.InventoryError, match="No scoped"):
inventory.walk("operators")
@pytest.mark.parametrize("rc,output", [(1, ""), (0, "not-json"), (0, '{}')])
def test_listing_errors_never_become_empty_success(monkeypatch, rc, output):
monkeypatch.setenv("BAO_TOKEN", "synthetic-token")
monkeypatch.setattr(inventory.subprocess, "run", lambda *a, **kw: SimpleNamespace(returncode=rc, stdout=output))
with pytest.raises(inventory.InventoryError):
inventory.walk("operators")
def test_inventory_reads_only_metadata_and_reports_incomplete(monkeypatch, capsys):
calls = []
responses = {("kv", "list", "operators"): ["example/"],
("kv", "list", "operators/example"): ["admin"],
("kv", "metadata", "get", "operators/example/admin"): {"data": {"custom_metadata": {}}}}
def bao(*args):
calls.append(args)
return responses[args]
monkeypatch.setattr(inventory, "bao", bao)
monkeypatch.setattr(inventory.sys, "argv", ["inventory", "operators", "--undescribed"])
assert inventory.main() == 1
assert "1 credential path(s), 1 missing" in capsys.readouterr().out
assert all(c[:2] == ("kv", "list") or c[:3] == ("kv", "metadata", "get") for c in calls)

166
tests/test_kubernetes_plane.py
View file

@ -1,10 +1,13 @@
import hashlib
import json
from pathlib import Path
from datetime import date
import pytest
import yaml
from ops_mason.readiness import inspect_readiness, resolve_tier
from ops_mason.kubernetes_plane import (
CommandResult,
PlaneBundle,
@ -16,6 +19,9 @@ from ops_mason.kubernetes_plane import (
)
READINESS = "bound_rapps:\n - rapp_id: rapp-test\n readiness_state: verified\n"
REVISION = "a" * 40
ROOT = Path(__file__).resolve().parents[1]
@ -72,6 +78,11 @@ def _fixture(tmp_path: Path, *, approved: bool = True, kind: str = "Namespace")
descriptor = {
"schema_version": "ops-mason.kubernetes-plane/v1",
"id": "plane",
"readiness": {
"target": {"kind": "rapp", "rapp_id": "rapp-test", "namespace": "whitehat"},
"source": {"repo": "reef-railiance", "path": "bindings/rapps.yaml",
"revision": REVISION, "sha256": hashlib.sha256(READINESS.encode()).hexdigest()},
},
"plan": "../plans/plane.md",
"expected_context": "default",
"expected_namespace": "whitehat",
@ -148,6 +159,14 @@ class FakeCluster:
self.calls.append(command)
if command[:4] == ["git", "-C", command[2], "status"]:
return CommandResult(0, " M src/ops_mason/kubernetes_plane.py\n" if self.dirty else "")
if command[0] == "git":
if command[3] == "rev-parse":
return CommandResult(0, "false\n")
if command[3] == "show":
return CommandResult(0, READINESS)
if command[3] == "log":
return CommandResult(0, REVISION + "\n")
return CommandResult(0)
if command == ["kubectl", "config", "current-context"]:
return CommandResult(0, self.context + "\n")
if command[:4] == ["kubectl", "auth", "can-i", "create"]:
@ -176,7 +195,8 @@ class FakeCluster:
),
)
if command[:4] == ["kubectl", "-n", "whitehat", "get"]:
return CommandResult(0, json.dumps({"items": []}))
assert command[-2:] == ["-o", "name"]
return CommandResult(0, "")
raise AssertionError(f"unexpected command: {command}")
@ -264,3 +284,147 @@ def test_rollback_is_generated_but_never_executed(tmp_path: Path) -> None:
result = rollback_plan(bundle)
assert result["object_scoped_commands"] == []
assert result["conditional_namespace_commands"] == ["kubectl delete namespaces whitehat"]
class ReadinessCluster(FakeCluster):
def __init__(self, content=READINESS, *, old=None, changed=False, shallow=False):
super().__init__()
self.content, self.old, self.changed, self.shallow = content, old, changed, shallow
def __call__(self, args):
if args[0] == "git" and args[3] != "status":
self.calls.append(list(args))
if args[3] == "show":
return CommandResult(0, self.old if args[4].startswith("b" * 40) else self.content)
if args[3] == "log":
return CommandResult(0, REVISION + "\n" + ("b" * 40 + "\n" if self.old else ""))
if args[3] == "diff":
return CommandResult(1 if self.changed else 0)
if args[3] == "rev-parse":
return CommandResult(0, "true" if self.shallow else "false")
return CommandResult(0)
return super().__call__(args)
def readiness_bundle(tmp_path, content=READINESS):
bundle = PlaneBundle.load(_fixture(tmp_path))
bundle.readiness["source"]["sha256"] = hashlib.sha256(content.encode()).hexdigest()
return bundle
@pytest.mark.parametrize("case", ["missing", "platform", "unlisted", "digest", "changed", "unknown", "shallow", "namespace"])
def test_unverifiable_readiness_refuses_before_kubectl(tmp_path, case):
content = READINESS.replace("verified", "mystery") if case == "unknown" else READINESS
bundle = readiness_bundle(tmp_path, content)
cluster = ReadinessCluster(content, changed=case == "changed", shallow=case == "shallow")
if case == "missing":
bundle.readiness = None
elif case == "platform":
bundle.readiness["target"]["kind"] = "platform"
elif case == "unlisted":
bundle.readiness["target"]["rapp_id"] = "absent"
elif case == "digest":
bundle.readiness["source"]["sha256"] = "0" * 64
elif case == "namespace":
bundle.readiness["target"]["namespace"] = "other"
with pytest.raises(PlaneRefused, match="production tier"):
apply(bundle, confirm_plan_id="plane", expected_digest=bundle.digest, runner=cluster)
assert not any(c[0] == "kubectl" for c in cluster.calls)
def test_production_preflight_reports_but_apply_refuses(tmp_path):
content = READINESS.replace("verified", "production-approved")
bundle = readiness_bundle(tmp_path, content)
cluster = ReadinessCluster(content)
assert preflight(bundle, cluster)["readiness"]["tier"] == "production"
with pytest.raises(PlaneRefused, match="production tier"):
apply(bundle, confirm_plan_id="plane", expected_digest=bundle.digest, runner=cluster)
assert not cluster.applied
def test_break_glass_requires_reason_and_records_reconciliation(tmp_path):
content = READINESS.replace("verified", "production-approved")
bundle = readiness_bundle(tmp_path, content)
cluster = ReadinessCluster(content)
with pytest.raises(PlaneRefused, match="non-empty reason"):
apply(bundle, confirm_plan_id="plane", expected_digest=bundle.digest,
runner=cluster, activation="BREAK_GLASS", break_glass_reason=" ")
assert not cluster.calls
result = apply(bundle, confirm_plan_id="plane", expected_digest=bundle.digest,
runner=cluster, activation="BREAK_GLASS", break_glass_reason="Incident test")
assert result["readiness"]["tier"] == "production"
assert result["break_glass"]["reason"] == "Incident test"
assert result["break_glass"]["actor"] and result["break_glass"]["recorded_at"]
assert "ArgoCD" in result["break_glass"]["follow_up"]
@pytest.mark.parametrize("day,allowed", [(20, True), (21, False), (22, False)])
def test_policy_nexus_transition_expires_at_review_date(tmp_path, day, allowed):
content = READINESS.replace("rapp-test", "rapp-policy-nexus").replace("verified", "production-approved")
bundle = readiness_bundle(tmp_path, content)
bundle.readiness["target"]["rapp_id"] = "rapp-policy-nexus"
cluster = ReadinessCluster(content)
kwargs = dict(confirm_plan_id="plane", expected_digest=bundle.digest, runner=cluster, today=date(2026, 12, day))
if allowed:
result = apply(bundle, **kwargs)
assert result["readiness"]["transition"] and result["readiness"]["tier"] == "production"
else:
with pytest.raises(PlaneRefused, match="production tier"):
apply(bundle, **kwargs)
assert not cluster.applied
@pytest.mark.parametrize("state,old,tier", [("verified", "production-approved", "production"),
("deprecated", "production-approved", "production"), ("deprecated", "verified", "non-production")])
def test_lapse_and_deprecation_retain_previous_tier(tmp_path, state, old, tier):
content = READINESS.replace("verified", state)
bundle = readiness_bundle(tmp_path, content)
cluster = ReadinessCluster(content, old=READINESS.replace("verified", old))
assert inspect_readiness(bundle, cluster, None)["tier"] == tier
def test_whitehat_explicit_placement_and_binding_supersession(tmp_path):
bundle = readiness_bundle(tmp_path)
bundle.id = "whitehat-foundational-plane"
bundle.readiness["target"] = {"kind": "namespace", "namespace": "whitehat"}
assert inspect_readiness(bundle, ReadinessCluster(), None)["tier"] == "non-production"
content = READINESS.replace("rapp-test", "rapp-whitehat")
bundle.readiness["source"]["sha256"] = hashlib.sha256(content.encode()).hexdigest()
assert inspect_readiness(bundle, ReadinessCluster(content), None)["tier"] == "production"
def test_unknown_activation_never_allows_apply():
assert not resolve_tier("verified", {}, "anything", date.today())["direct_apply_allowed"]
def test_real_git_history_retains_promotion_even_after_file_reverted(tmp_path):
"""A clean file equal to its pin is not proof it was never production."""
import subprocess
from ops_mason.kubernetes_plane import subprocess_runner
bundle = readiness_bundle(tmp_path)
repo = tmp_path / "reef"
repo.mkdir()
def git(*args):
return subprocess.run(["git", "-C", str(repo), *args], check=True,
capture_output=True, text=True).stdout.strip()
git("init")
git("config", "user.name", "Test")
git("config", "user.email", "test@example.invalid")
(repo / "bindings").mkdir()
source = repo / "bindings/rapps.yaml"
def commit(content, message):
source.write_text(content)
git("add", "bindings/rapps.yaml")
git("commit", "-m", message)
commit(READINESS, "verified")
bundle.readiness["source"]["revision"] = git("rev-parse", "HEAD")
assert inspect_readiness(bundle, subprocess_runner, repo)["tier"] == "non-production"
commit(READINESS.replace("verified", "production-approved"), "promote")
commit(READINESS, "evidence lapse")
result = inspect_readiness(bundle, subprocess_runner, repo)
assert result["tier"] == "production"
assert result["reason"] == "production tier retained from binding history"
source.write_text(READINESS + "# uncommitted\n")
assert "uncommitted" in inspect_readiness(bundle, subprocess_runner, repo)["reason"]

13
workplans/MASON-WP-0004-credential-inventory-descriptions.md
View file

@ -4,11 +4,12 @@ type: workplan
title: "Describe every stored credential so the store is navigable"
domain: infotech
repo: ops-mason
status: proposed
status: blocked
flavor: planning
owner: codex
topic_slug: custodian
created: "2026-08-28"
updated: "2026-09-28"
related:
- MASON-WP-0003
- NK-WP-0033
@ -48,7 +49,7 @@ read, and `ops-mason-build` cannot read one.
```task
id: MASON-WP-0004-T01
status: todo
status: wait
priority: medium
state_hub_task_id: "5bda75f2-f780-579e-9d44-cc4011662b9a"
```
@ -68,7 +69,7 @@ listed as unknown with the question that would settle it.
```task
id: MASON-WP-0004-T02
status: todo
status: wait
priority: medium
state_hub_task_id: "a6a944d7-21c5-5043-a78d-b3809de0ee64"
```
@ -121,3 +122,9 @@ the report has a reader.
Acceptance: an undescribed path added today surfaces without anyone
remembering to look.
## Loose-end review — 2026-09-28
T01–T04 remain wait: no valid scoped metadata grant/current inventory or complete owner/recovery confirmations are available. Inventory failure handling and the private-tunnel defaults are fixed and tested; scheduled reporting still needs its reader and credential.
Evidence and precise resumption conditions: [review](../docs/evidence/2026-09-28-loose-end-review.md). Existing tasks retain all remaining obligations; no new task or workplan was opened.

21
workplans/MASON-WP-0005-fluid-telegram-operator-credential-lane.md
View file

@ -4,13 +4,12 @@ type: workplan
title: "Construct the fluid-telegram operator credential lane"
domain: infotech
repo: ops-mason
status: proposed
status: blocked
flavor: planning
owner: codex
topic_slug: helix-forge
created: "2026-09-04"
updated: "2026-09-04"
state_hub_workstream_id: "483e56d6-6601-5377-9066-66225214c046"
updated: "2026-09-28"
state_hub_workstream_id: "483e56d6-6601-5377-9066-66225214c046"
---
@ -65,7 +64,7 @@ from the current disk-only survey.
```task
id: MASON-WP-0005-T02
status: todo
status: wait
priority: high
state_hub_task_id: "69c558d9-664f-5ce0-80b2-d2e7544353a3"
```
@ -86,7 +85,7 @@ parent access, and the create-only salt shape.
```task
id: MASON-WP-0005-T03
status: todo
status: wait
priority: high
state_hub_task_id: "48a2b4ec-8e66-5b98-b039-c17fd8d820ab"
```
@ -103,7 +102,7 @@ adapter separation for explicit human approval. Only the plan's
```task
id: MASON-WP-0005-T04
status: todo
status: wait
priority: high
state_hub_task_id: "0e3d1333-ffc3-5f67-8528-50a9551c4949"
```
@ -122,7 +121,7 @@ a provisioner capability.
```task
id: MASON-WP-0005-T05
status: todo
status: wait
priority: high
state_hub_task_id: "4c205be7-4f1f-5dd1-aafa-fc112fdd640e"
```
@ -147,7 +146,7 @@ operator/platform flow and return metadata-only evidence to ops-mason.
```task
id: MASON-WP-0005-T06
status: todo
status: wait
priority: medium
state_hub_task_id: "09eae088-808d-5342-b100-292e13958ee3"
```
@ -163,3 +162,9 @@ The adapter lane is tracked separately as `MASON-IN-0003`, tied to
`redaction-salt`; it must be denied `operator-app` and `operator-session` and
must use the adapter's own runtime identity rather than this attended OIDC
role.
## Loose-end review — 2026-09-28
T01–T06 remain wait: accepted tenant/matrix, confirmed identity/MFA/callbacks, approved matching writer contracts, live survey, explicit construction approval and verification are outstanding. The platform design remains proposed. No lane was built or activated.
Evidence and precise resumption conditions: [review](../docs/evidence/2026-09-28-loose-end-review.md). Existing tasks retain all remaining obligations; no new task or workplan was opened.

42
workplans/MASON-WP-0006-readiness-tier-check.md
View file

@ -4,12 +4,12 @@ type: workplan
title: "Check the target's readiness tier before a direct Kubernetes apply"
domain: infotech
repo: ops-mason
status: proposed
status: blocked
flavor: implementation
owner: claude
topic_slug: ops-mason
created: "2026-09-21"
updated: "2026-09-21"
updated: "2026-09-28"
state_hub_workstream_id: "1b900161-51ff-544f-8412-ba7fcab64bb5"
---
@ -97,7 +97,7 @@ policy-nexus rule was used.
```task
id: MASON-WP-0006-T01
status: wait
status: done
priority: high
state_hub_task_id: "7b6fe7b4-08f7-5ad0-899d-a4862b5ddb00"
```
@ -120,11 +120,15 @@ question ops-mason can answer for itself. The founder chooses one of:
T03 must not merge before this is answered.
**Resolved 2026-09-21; implemented 2026-09-28.** The founder chose explicit
non-production placement for namespace whitehat. Decision and inbox receipt
are recorded in the September 28 review. This unblocks T03.
## Extend the bundle schema with the readiness block
```task
id: MASON-WP-0006-T02
status: todo
status: done
priority: high
state_hub_task_id: "89d07bd5-5a92-5a06-9ecb-441799c14dd7"
```
@ -135,11 +139,15 @@ is included in the bundle digest, as the bundle file already is. Add a
`APPROVED`, and a `--readiness-repo` option. `BREAK_GLASS` requires a
`--break-glass-reason` string. No existing refusal is relaxed.
**Done 2026-09-28.** Schema/CLI implemented; the mapping also requires an
explicit namespace matching the bundle. Whitehat now pins the source used to
check whether a binding supersedes its explicit placement.
## Enforce the tier in preflight and apply
```task
id: MASON-WP-0006-T03
status: todo
status: done
priority: high
state_hub_task_id: "163a807b-29aa-5eb7-a9a8-ef1ac70c89d3"
```
@ -163,11 +171,17 @@ Tests, all against the injected runner:
- the existing forbidden-kind, `data`/`stringData` and namespace tests still pass
unchanged.
**Done 2026-09-28.** Source digest, ancestry, local freshness and complete
binding history are checked. Historical production approval remains production;
deprecation retains the last known tier. Unknowns fail closed. The policy-nexus
transition stops on December 21 itself. Preflight reports; apply refuses before
Kubernetes commands. Covered by injected-runner regression tests.
## Record BREAK_GLASS and its reconcile-back obligation
```task
id: MASON-WP-0006-T04
status: todo
status: done
priority: medium
state_hub_task_id: "66349531-09ee-55b9-be47-537842c80f3b"
```
@ -177,11 +191,15 @@ record and prints the follow-up that is owed: the same change, committed to the
manifest repository that ArgoCD reconciles. ops-mason does not open that change
itself.
**Done 2026-09-28.** Apply evidence and printed JSON include the emergency
reason, local account, UTC time and GitOps reconciliation obligation. Empty
reasons are refused without invoking the runner.
## Update the docs and the declaration
```task
id: MASON-WP-0006-T05
status: todo
status: done
priority: medium
state_hub_task_id: "6d64e7f5-9cff-5bf9-9851-97d318d1df0a"
```
@ -190,6 +208,10 @@ Update `docs/kubernetes-plane.md` with the tier table and the readiness block,
and change the `enforcement` line of the `kubernetes-plane-apply` entry in
`INTENT.md` from "not yet in code" to point at the check.
**Done 2026-09-28.** Documented tiers, pins, mapping, history, checkout
freshness limits and emergency procedure; removed the withdrawn
rail-kubernetes ownership/layer assertion from INTENT.md.
## Review on 2026-12-21
```task
@ -203,3 +225,9 @@ On 2026-12-21 the policy-nexus transition ends and the plan-approval exception
comes up for review. Confirm with railiance-platform that policy-nexus is
onboarded to ArgoCD. The date check in T03 ends the transition in code on that
date either way.
## Loose-end review — 2026-09-28
T01–T05 are done: the founder resolved the whitehat placement and the guarded readiness implementation, tests, emergency evidence and docs are complete. T06 remains wait until the 2026-12-21 platform/exception review; this workplan is blocked on that dated review.
Evidence and precise resumption conditions: [review](../docs/evidence/2026-09-28-loose-end-review.md). Existing tasks retain all remaining obligations; no new task or workplan was opened.