NK-WP-0056-T12 done: pilot operations; synthetic rotation and revert pass
operations.py: OpenBao KV v2 CAS, LLDAP self-rotation (forward and revert), ExternalSecret force-sync and status, rollout restart (RolloutFailed is a definite failure), pilot checks. synthetic_pilot.py on the T08 stack plus OpenBao 2.5.4 with CCR-2026-0058's policy: rotation completed; a one-off wrong delivery reverted in declared order with the earlier value restored. Runner: revert steps run in declared order (BEA-10), definite failures revert, unknown outcomes stop, receipts carry error types. Found and corrected (in railiance-platform) CCR-2026-0058's required_parameters refusing reads. Packaging moved to new T13. 29 executor tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Assistant: claude-code Assistant-Model: opus Assistant-Process: 994874@bnt-lap001 Assistant-Session: 6017c416-d619-47eb-bd1a-f670a02d7d87
This commit is contained in:
parent
9b7701a32a
commit
db56d8468e
11 changed files with 1088 additions and 33 deletions
|
|
@ -354,7 +354,8 @@
|
|||
| task | NK-WP-0056-T09 | done | — | workplans/NK-WP-0056-bounded-executor-path.md |
|
||||
| task | NK-WP-0056-T10 | todo | — | workplans/NK-WP-0056-bounded-executor-path.md |
|
||||
| task | NK-WP-0056-T11 | done | — | workplans/NK-WP-0056-bounded-executor-path.md |
|
||||
| task | NK-WP-0056-T12 | todo | — | workplans/NK-WP-0056-bounded-executor-path.md |
|
||||
| task | NK-WP-0056-T12 | done | — | workplans/NK-WP-0056-bounded-executor-path.md |
|
||||
| task | NK-WP-0056-T13 | todo | — | workplans/NK-WP-0056-bounded-executor-path.md |
|
||||
| intake | NK-IN-0001 | closed | blue | docs/intakes/activity-core-ops-sso-operators.md |
|
||||
| intake | NK-IN-0002 | closed | blue | docs/intakes/activity-core-ops-sso-operators.md |
|
||||
| intake | NET-IN-0001 | open | — | intakes/intakes.md |
|
||||
|
|
|
|||
|
|
@ -28,9 +28,23 @@ Job per admission in `sso` (T12).
|
|||
- **receipt:** one `execution.attribution.receipt` per run, with PDP `not-evaluated` in the pilot. It never holds a
|
||||
value that an operation handled.
|
||||
|
||||
Operations, checks, the approval carrier, the trust oracle, the audit sink, alerts and clocks are injected. T12 adds
|
||||
the pilot's real operations (OpenBao CAS write, LLDAP self-rotation, ExternalSecret sync, rollout, synthetic
|
||||
journey), the CLI, the Containerfile and the image workflow.
|
||||
Operations, checks, the approval carrier, the trust oracle, the audit sink, alerts and clocks are injected.
|
||||
|
||||
`operations.py` (T12) holds the pilot's real operations:
|
||||
|
||||
- OpenBao KV v2 with CAS;
|
||||
- LLDAP self-rotation;
|
||||
- ExternalSecret force-sync and its status;
|
||||
- rollout restart, where a rollout that never becomes ready is a definite failure and is reverted;
|
||||
- the synthetic journey check.
|
||||
|
||||
An operation signals a known-bad outcome with `definite_failure`, and the runner reverts. Any other error stops
|
||||
without guessing.
|
||||
|
||||
`synthetic_pilot.py` runs a rotation and a deliberate revert on the T08 stack plus OpenBao:
|
||||
`uv run bounded-executor/synthetic_pilot.py`.
|
||||
|
||||
T13 adds the CLI, the Containerfile, the image workflow and the rapp package.
|
||||
|
||||
```bash
|
||||
python3 -m pytest bounded-executor/tests # or: make bounded-executor-test
|
||||
|
|
|
|||
|
|
@ -12,9 +12,10 @@ Runs one Bounded Execution Admission (canon/standards/bounded-execution-admissio
|
|||
started `execution.run.started` to the audit sink; no custody, no run
|
||||
steps in plan order, one consumer at a time; checks after each mutating step, after
|
||||
their settle time; the kill switch and the budget are checked between steps
|
||||
failure a failed check or an exhausted budget reverts the mutated steps in reverse
|
||||
order (or stops, as the admission says) and re-runs their checks; an error of
|
||||
unknown outcome stops without guessing; a failed revert stops and alerts
|
||||
failure a failed check, an exhausted budget or the kill switch runs the revert steps of
|
||||
the mutated steps in the admission's declared order (or stops, as it says), each
|
||||
proving the restored state with its own checks; an error of unknown outcome stops
|
||||
without guessing; a failed revert stops and alerts
|
||||
receipt one `execution.attribution.receipt` per run (PDP `not-evaluated` in the pilot)
|
||||
|
||||
Every effect goes through an injected interface: Carrier, operation and check callables, TrustOracle,
|
||||
|
|
@ -52,6 +53,12 @@ class Unknown(Exception):
|
|||
"""An operation's outcome is unknown: stop, do not guess, do not revert blindly."""
|
||||
|
||||
|
||||
class StepFailed(Exception):
|
||||
"""A definite, known-bad outcome (e.g. a rollout that never became ready): handled like a failed check.
|
||||
Operations outside this module may instead set `definite_failure = True` on their exception class."""
|
||||
definite_failure = True
|
||||
|
||||
|
||||
class Carrier(Protocol):
|
||||
def get(self, approval_id: str) -> dict: ...
|
||||
def consume(self, approval_id: str, request_digest: str, decision_id: str) -> dict: ...
|
||||
|
|
@ -203,9 +210,13 @@ class Run:
|
|||
def _budget_left(self) -> float:
|
||||
return self.spec["budget"]["wall_time_seconds"] - (self.d.monotonic() - self.start_mono)
|
||||
|
||||
def _record(self, step: dict, kind: str, outcome: str, checks: list[dict] | None = None) -> None:
|
||||
self.steps.append({"id": step["id"], "kind": kind, "operation": step["operation"], "target": step["target"],
|
||||
"consumer": step["consumer"], "outcome": outcome, "checks": checks or []})
|
||||
def _record(self, step: dict, kind: str, outcome: str, checks: list[dict] | None = None,
|
||||
error: BaseException | None = None) -> None:
|
||||
entry = {"id": step["id"], "kind": kind, "operation": step["operation"], "target": step["target"],
|
||||
"consumer": step["consumer"], "outcome": outcome, "checks": checks or []}
|
||||
if error is not None:
|
||||
entry["error_type"] = type(error).__name__ # the type only: messages can carry values
|
||||
self.steps.append(entry)
|
||||
|
||||
def _run_checks(self, ids: list[str], ctx: Context) -> tuple[bool, list[dict]]:
|
||||
results = []
|
||||
|
|
@ -230,18 +241,20 @@ class Run:
|
|||
return True, results
|
||||
|
||||
def _revert(self, done: list[dict], ctx: Context) -> bool:
|
||||
"""Undo the mutated steps in reverse order; each revert step must pass its own checks."""
|
||||
reverts = {s["id"]: s for s in self.spec["plan"].get("revert_steps", [])}
|
||||
for step in reversed(done):
|
||||
if not step["mutates"]:
|
||||
"""Undo the mutated steps. Revert steps run in the order the admission declares them (the author knows
|
||||
the recovery order: data before delivery before restart), limited to those whose forward step ran.
|
||||
Each must pass its own checks; an irreversible step that ran makes the revert fail."""
|
||||
mutated = [s for s in done if s["mutates"]]
|
||||
if any(s.get("irreversible") for s in mutated):
|
||||
return False
|
||||
wanted = {s["revert"] for s in mutated}
|
||||
for revert in self.spec["plan"].get("revert_steps", []):
|
||||
if revert["id"] not in wanted:
|
||||
continue
|
||||
if step.get("irreversible"):
|
||||
return False
|
||||
revert = reverts[step["revert"]]
|
||||
try:
|
||||
self.d.operations[revert["operation"]](revert, ctx)
|
||||
except Exception:
|
||||
self._record(revert, "revert", "error")
|
||||
except Exception as error:
|
||||
self._record(revert, "revert", "error", error=error)
|
||||
return False
|
||||
ok, results = self._run_checks(revert.get("checks_after", []), ctx)
|
||||
self._record(revert, "revert", "done" if ok else "check_failed", results)
|
||||
|
|
@ -262,11 +275,15 @@ class Run:
|
|||
return reason
|
||||
try:
|
||||
self.d.operations[step["operation"]](step, ctx)
|
||||
except Unknown:
|
||||
self._record(step, "step", "unknown")
|
||||
return "unknown"
|
||||
except Exception:
|
||||
self._record(step, "step", "unknown")
|
||||
except Exception as error:
|
||||
if getattr(error, "definite_failure", False) and step["mutates"]:
|
||||
# known-bad state: treat like a failed check, the step counts as done for the revert
|
||||
done.append(step)
|
||||
self._record(step, "step", "failed", error=error)
|
||||
if policy["on_check_failure"] == "revert":
|
||||
return "reverted" if self._revert(done, ctx) else "revert_failed"
|
||||
return "stopped"
|
||||
self._record(step, "step", "unknown", error=error) # outcome not established: stop
|
||||
return "unknown"
|
||||
done.append(step)
|
||||
ok, results = self._run_checks(step.get("checks_after", []), ctx) if step["mutates"] else (True, [])
|
||||
|
|
|
|||
332
bounded-executor/operations.py
Normal file
332
bounded-executor/operations.py
Normal file
|
|
@ -0,0 +1,332 @@
|
|||
"""Pilot operations and checks for the bounded executor (NK-WP-0056-T12).
|
||||
|
||||
The T06 pilot rotates one directory bind (svc-authelia) under one admission:
|
||||
|
||||
openbao.kv.write-next-version read version N, write a fresh value as N+1 with cas=N
|
||||
openbao.kv.write-previous-value-as-next-version revert: write the previous value as the next version (CAS)
|
||||
lldap.user.change-own-password-from-kv set the bind's LLDAP password to the current KV value,
|
||||
signing in as the bind with a value it still accepts
|
||||
(self-rotation, founder decision 3d0c8f86); forward and revert
|
||||
eso.force-sync ask the delivery to re-project the value (ExternalSecret)
|
||||
k8s.rollout-restart restart the consumer and wait for the rollout
|
||||
|
||||
Checks: kv-advanced, bind-signs-in, eso-synced, synthetic-journey.
|
||||
|
||||
Values live only in Context.data (memory) and in the requests that carry them; nothing here prints, logs or
|
||||
returns a value. Every adapter is stdlib HTTP so the image needs no client libraries.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import datetime as dt
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
import pathlib
|
||||
import secrets
|
||||
import ssl
|
||||
import subprocess
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
from typing import Callable
|
||||
|
||||
HERE = pathlib.Path(__file__).resolve().parent
|
||||
REPO = HERE.parent
|
||||
_spec = importlib.util.spec_from_file_location("journey", REPO / "sso-mfa/k8s/synthetic-journey/journey.py")
|
||||
journey = importlib.util.module_from_spec(_spec)
|
||||
_spec.loader.exec_module(journey)
|
||||
|
||||
FIELD = "LDAP_BIND_PASSWORD"
|
||||
|
||||
|
||||
class OperationError(Exception):
|
||||
"""A definite failure: the operation did not take effect."""
|
||||
|
||||
|
||||
class RolloutFailed(Exception):
|
||||
"""The restart was issued but the consumer never became ready: a definite, known-bad outcome that the
|
||||
executor reverts (executor.StepFailed semantics)."""
|
||||
definite_failure = True
|
||||
|
||||
|
||||
def _opener(ca_file: str | None):
|
||||
context = ssl.create_default_context(cafile=ca_file) if ca_file else ssl.create_default_context()
|
||||
return urllib.request.build_opener(urllib.request.ProxyHandler({}), urllib.request.HTTPSHandler(context=context))
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------------------------------
|
||||
# OpenBao KV v2
|
||||
# --------------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
class OpenBao:
|
||||
def __init__(self, addr: str, *, token: str | None = None, k8s_role: str | None = None,
|
||||
jwt_file: str | None = None, k8s_mount: str = "kubernetes", ca_file: str | None = None,
|
||||
timeout: float = 15):
|
||||
self.addr = addr.rstrip("/")
|
||||
self.open = _opener(ca_file).open
|
||||
self.timeout = timeout
|
||||
self.token = token
|
||||
if self.token is None:
|
||||
jwt = pathlib.Path(jwt_file).read_text().strip()
|
||||
auth = self._call("POST", f"/v1/auth/{k8s_mount}/login", {"role": k8s_role, "jwt": jwt}, auth=False)
|
||||
self.token = auth["auth"]["client_token"]
|
||||
|
||||
def _call(self, method: str, path: str, body: dict | None = None, auth: bool = True) -> dict:
|
||||
headers = {"Content-Type": "application/json"}
|
||||
if auth:
|
||||
headers["X-Vault-Token"] = self.token
|
||||
request = urllib.request.Request(self.addr + path, method=method, headers=headers,
|
||||
data=json.dumps(body).encode() if body is not None else None)
|
||||
try:
|
||||
with self.open(request, timeout=self.timeout) as response:
|
||||
raw = response.read()
|
||||
except urllib.error.HTTPError as error:
|
||||
error.close()
|
||||
raise OperationError(f"openbao {method} {error.code}") from None
|
||||
return json.loads(raw) if raw else {}
|
||||
|
||||
@staticmethod
|
||||
def _api(kv_path: str) -> str:
|
||||
mount, _, rest = kv_path.partition("/")
|
||||
return f"/v1/{mount}/data/{rest}"
|
||||
|
||||
def read(self, kv_path: str) -> tuple[dict, int]:
|
||||
body = self._call("GET", self._api(kv_path))["data"]
|
||||
return body["data"], int(body["metadata"]["version"])
|
||||
|
||||
def write(self, kv_path: str, data: dict, cas: int) -> int:
|
||||
return int(self._call("POST", self._api(kv_path), {"options": {"cas": cas}, "data": data})["data"]["version"])
|
||||
|
||||
def revoke_self(self) -> None:
|
||||
try:
|
||||
self._call("POST", "/v1/auth/token/revoke-self", {})
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------------------------------
|
||||
# LLDAP
|
||||
# --------------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
class Directory:
|
||||
"""Sign-in over LLDAP's HTTP API; password changes through the pinned image's lldap_set_password (OPAQUE)."""
|
||||
|
||||
def __init__(self, base_url: str, set_password_command: list[str], *, ca_file: str | None = None,
|
||||
tool_base_url: str | None = None, timeout: float = 15):
|
||||
self.base = base_url.rstrip("/")
|
||||
self.tool_base = (tool_base_url or base_url).rstrip("/")
|
||||
self.command = set_password_command
|
||||
self.open = _opener(ca_file).open
|
||||
self.timeout = timeout
|
||||
|
||||
def login(self, user: str, password: str) -> str | None:
|
||||
request = urllib.request.Request(self.base + "/auth/simple/login", method="POST",
|
||||
headers={"Content-Type": "application/json"},
|
||||
data=json.dumps({"username": user, "password": password}).encode())
|
||||
try:
|
||||
with self.open(request, timeout=self.timeout) as response:
|
||||
return json.load(response).get("token")
|
||||
except urllib.error.HTTPError as error:
|
||||
error.close()
|
||||
return None
|
||||
|
||||
def set_own_password(self, token: str, user: str, password: str) -> None:
|
||||
done = subprocess.run([*self.command, "--base-url", self.tool_base, "--token", token, "--username", user],
|
||||
env={"PATH": os.environ.get("PATH", "/usr/bin:/bin"), "LLDAP_USER_PASSWORD": password},
|
||||
capture_output=True, timeout=60, check=False)
|
||||
if done.returncode:
|
||||
raise OperationError("lldap_set_password refused")
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------------------------------
|
||||
# Kubernetes (stdlib client, in-cluster ServiceAccount)
|
||||
# --------------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
class Kube:
|
||||
def __init__(self, api: str = "https://kubernetes.default.svc",
|
||||
token_file: str = "/var/run/secrets/kubernetes.io/serviceaccount/token",
|
||||
ca_file: str | None = "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt", timeout: float = 15):
|
||||
self.api = api.rstrip("/")
|
||||
self.token_file = token_file
|
||||
self.open = _opener(ca_file).open
|
||||
self.timeout = timeout
|
||||
|
||||
def _call(self, method: str, path: str, body: dict | None = None) -> dict:
|
||||
headers = {"Authorization": "Bearer " + pathlib.Path(self.token_file).read_text().strip(),
|
||||
"Accept": "application/json"}
|
||||
data = None
|
||||
if body is not None:
|
||||
headers["Content-Type"] = "application/merge-patch+json"
|
||||
data = json.dumps(body).encode()
|
||||
request = urllib.request.Request(self.api + path, method=method, headers=headers, data=data)
|
||||
try:
|
||||
with self.open(request, timeout=self.timeout) as response:
|
||||
return json.load(response)
|
||||
except urllib.error.HTTPError as error:
|
||||
error.close()
|
||||
raise OperationError(f"kube {method} {error.code}") from None
|
||||
|
||||
def get(self, path: str) -> dict:
|
||||
return self._call("GET", path)
|
||||
|
||||
def patch(self, path: str, body: dict) -> dict:
|
||||
return self._call("PATCH", path, body)
|
||||
|
||||
|
||||
def now_utc() -> dt.datetime:
|
||||
return dt.datetime.now(dt.UTC).replace(microsecond=0)
|
||||
|
||||
|
||||
class KubeDelivery:
|
||||
"""eso.force-sync and its eso-synced check, through the ExternalSecret API (no Secret read)."""
|
||||
|
||||
def __init__(self, kube: Kube, namespace: str, external_secret: str, clock: Callable[[], dt.datetime] = now_utc):
|
||||
self.kube, self.clock = kube, clock
|
||||
self.path = f"/apis/external-secrets.io/v1/namespaces/{namespace}/externalsecrets/{external_secret}"
|
||||
|
||||
def force_sync(self) -> dt.datetime:
|
||||
at = self.clock()
|
||||
self.kube.patch(self.path, {"metadata": {"annotations": {"force-sync": str(int(at.timestamp()))}}})
|
||||
return at
|
||||
|
||||
def synced(self, since: dt.datetime) -> bool:
|
||||
status = self.kube.get(self.path).get("status", {})
|
||||
ready = any(c.get("type") == "Ready" and c.get("status") == "True" for c in status.get("conditions", []))
|
||||
refreshed = status.get("refreshTime")
|
||||
return ready and bool(refreshed) and dt.datetime.fromisoformat(refreshed.replace("Z", "+00:00")) >= since
|
||||
|
||||
|
||||
class KubeRestart:
|
||||
"""k8s.rollout-restart: patch the restartedAt annotation, then wait until the rollout is complete."""
|
||||
|
||||
def __init__(self, kube: Kube, namespace: str, deployment: str, timeout: float = 300, poll: float = 3,
|
||||
clock: Callable[[], dt.datetime] = now_utc, sleep: Callable[[float], None] = time.sleep,
|
||||
monotonic: Callable[[], float] = time.monotonic):
|
||||
self.kube, self.timeout, self.poll, self.clock, self.sleep, self.monotonic = kube, timeout, poll, clock, sleep, monotonic
|
||||
self.path = f"/apis/apps/v1/namespaces/{namespace}/deployments/{deployment}"
|
||||
|
||||
def restart(self) -> None:
|
||||
stamp = self.clock().isoformat().replace("+00:00", "Z")
|
||||
self.kube.patch(self.path, {"spec": {"template": {"metadata": {"annotations": {
|
||||
"kubectl.kubernetes.io/restartedAt": stamp}}}}})
|
||||
deadline = self.monotonic() + self.timeout
|
||||
while self.monotonic() < deadline:
|
||||
d = self.kube.get(self.path)
|
||||
spec, status = d.get("spec", {}), d.get("status", {})
|
||||
want = spec.get("replicas", 1)
|
||||
if (status.get("observedGeneration", 0) >= d.get("metadata", {}).get("generation", 0)
|
||||
and status.get("updatedReplicas") == want and status.get("availableReplicas") == want
|
||||
and status.get("replicas") == want and not status.get("unavailableReplicas")):
|
||||
return
|
||||
self.sleep(self.poll)
|
||||
raise RolloutFailed("rollout did not complete") # known-bad: the runner reverts
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------------------------------
|
||||
# The pilot registry
|
||||
# --------------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def pilot_registry(*, bao: OpenBao, directory: Directory, delivery, restart, bind_kv: str, bind_user: str,
|
||||
journey_kv: str, journey_config, journey_state: dict | None = None):
|
||||
"""Return (operations, checks) for the bind-rotation admission. `delivery` has force_sync()/synced(since);
|
||||
`restart` has restart(). Values stay in ctx.data. `journey_state` carries the last one-time-code time step
|
||||
across runs (privacyIDEA accepts each 30 s step once)."""
|
||||
shared = journey_state if journey_state is not None else {}
|
||||
|
||||
def kv_target(step: dict) -> str:
|
||||
target = step["target"]
|
||||
if not target.startswith("openbao:") or target[len("openbao:"):] != bind_kv:
|
||||
raise OperationError("target outside the configured bind")
|
||||
return bind_kv
|
||||
|
||||
def generate(step, ctx):
|
||||
path = kv_target(step)
|
||||
data, version = bao.read(path)
|
||||
if set(data) != {FIELD}:
|
||||
raise OperationError("kv shape")
|
||||
fresh = secrets.token_urlsafe(32)
|
||||
ctx.data.update(previous=data[FIELD], fresh=fresh)
|
||||
ctx.data["expect"] = (bao.write(path, {FIELD: fresh}, cas=version), "fresh")
|
||||
|
||||
def restore_kv(step, ctx):
|
||||
path = kv_target(step)
|
||||
_, version = bao.read(path)
|
||||
ctx.data["expect"] = (bao.write(path, {FIELD: ctx.data["previous"]}, cas=version), "previous")
|
||||
|
||||
def change_own_password(step, ctx):
|
||||
if step["target"] != "lldap:" + bind_user:
|
||||
raise OperationError("target outside the configured bind")
|
||||
want = bao.read(bind_kv)[0][FIELD]
|
||||
if directory.login(bind_user, want):
|
||||
return # already in effect
|
||||
for known in (ctx.data.get("previous"), ctx.data.get("fresh")):
|
||||
if known and known != want:
|
||||
token = directory.login(bind_user, known)
|
||||
if token:
|
||||
directory.set_own_password(token, bind_user, want)
|
||||
if not directory.login(bind_user, want):
|
||||
raise OperationError("password change not effective")
|
||||
return
|
||||
raise OperationError("no known value signs in as the bind")
|
||||
|
||||
def force_sync(step, ctx):
|
||||
ctx.data["delivered_at"] = delivery.force_sync()
|
||||
|
||||
def rollout_restart(step, ctx):
|
||||
restart.restart()
|
||||
|
||||
def check_kv_advanced(check, ctx):
|
||||
version, which = ctx.data["expect"]
|
||||
data, current = bao.read(bind_kv)
|
||||
return current == version and data.get(FIELD) == ctx.data[which]
|
||||
|
||||
def check_bind_signs_in(check, ctx):
|
||||
return directory.login(bind_user, bao.read(bind_kv)[0][FIELD]) is not None
|
||||
|
||||
def check_eso_synced(check, ctx):
|
||||
deadline = time.monotonic() + check["timeout_seconds"] * 0.8
|
||||
while True:
|
||||
if delivery.synced(ctx.data["delivered_at"]):
|
||||
return True
|
||||
if time.monotonic() > deadline:
|
||||
return False
|
||||
time.sleep(2)
|
||||
|
||||
def check_journey(check, ctx):
|
||||
creds, _ = bao.read(journey_kv)
|
||||
cfg = _load_config(journey_config() if callable(journey_config) else journey_config)
|
||||
last = shared.get("journey_step")
|
||||
receipt = journey.run_journey(cfg, creds["PASSWORD"], journey.decode_seed(creds["TOTP_SEED"]),
|
||||
min_time_step=None if last is None else last + 1)
|
||||
if receipt.get("otp_time_step") is not None:
|
||||
shared["journey_step"] = receipt["otp_time_step"]
|
||||
ctx.data.setdefault("journeys", []).append({k: receipt.get(k) for k in ("result", "failed_step", "reason")})
|
||||
return receipt["result"] == "pass"
|
||||
|
||||
operations = {
|
||||
"openbao.kv.write-next-version": generate,
|
||||
"openbao.kv.write-previous-value-as-next-version": restore_kv,
|
||||
"lldap.user.change-own-password-from-kv": change_own_password,
|
||||
"eso.force-sync": force_sync,
|
||||
"k8s.rollout-restart": rollout_restart,
|
||||
}
|
||||
checks = {"kv-advanced": check_kv_advanced, "bind-signs-in": check_bind_signs_in,
|
||||
"eso-synced": check_eso_synced, "synthetic-journey": check_journey}
|
||||
return operations, checks
|
||||
|
||||
|
||||
def _load_config(config: dict) -> dict:
|
||||
import tempfile
|
||||
fd, path = tempfile.mkstemp(suffix=".json", prefix="journey-")
|
||||
try:
|
||||
with os.fdopen(fd, "w") as out:
|
||||
json.dump(config, out)
|
||||
return journey.load_config(path)
|
||||
finally:
|
||||
os.unlink(path)
|
||||
280
bounded-executor/synthetic_pilot.py
Normal file
280
bounded-executor/synthetic_pilot.py
Normal file
|
|
@ -0,0 +1,280 @@
|
|||
#!/usr/bin/env python3
|
||||
# /// script
|
||||
# dependencies = ["PyYAML>=6,<7", "jsonschema>=4,<5"]
|
||||
# ///
|
||||
"""Synthetic pilot: the executor rotates the svc-authelia bind on the T08 stack (NK-WP-0056-T12).
|
||||
|
||||
Agent-runnable, local Docker only, generated values, everything removed afterwards. Starts the T08 identity
|
||||
stack (LLDAP, privacyIDEA, Authelia, KeyCape on the pinned images) and an OpenBao dev server on the platform's
|
||||
pinned 2.5.4 image (KV v2 at `platform`, `cas_required` on the bind). The executor gets its own token with
|
||||
exactly CCR-2026-0058's policy file plus a journey-read policy, never the root token.
|
||||
|
||||
Not on this stack: approval-engine (an in-process carrier with its consume semantics stands in), audit-core
|
||||
(an in-memory sink), and Kubernetes. Stand-ins replace the ExternalSecret delivery (re-project the KV value
|
||||
into Authelia's configuration) and the rollout (restart the Authelia container). The real Kubernetes adapters
|
||||
(operations.KubeDelivery, KubeRestart) are covered by unit tests and exercised live in T06.
|
||||
|
||||
Runs:
|
||||
rotation one admission, the bind rotates; afterwards the old value is refused and the journey passes
|
||||
revert a second admission whose first delivery projects a wrong value while reporting success:
|
||||
Authelia does not become ready after the restart (definite failure), the executor reverts in
|
||||
declared order, the previous value is back and the journey passes again
|
||||
|
||||
Writes a value-free report. Exit 0 when both runs end as expected.
|
||||
Run: uv run bounded-executor/synthetic_pilot.py [--report FILE]
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import base64
|
||||
import copy
|
||||
import datetime as dt
|
||||
import hashlib
|
||||
import importlib.util
|
||||
import json
|
||||
import pathlib
|
||||
import secrets
|
||||
import sys
|
||||
import tempfile
|
||||
|
||||
import yaml
|
||||
|
||||
HERE = pathlib.Path(__file__).resolve().parent
|
||||
REPO = HERE.parent
|
||||
|
||||
|
||||
def _load(name, path):
|
||||
spec = importlib.util.spec_from_file_location(name, path)
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
sys.modules[name] = module
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
X = _load("executor", HERE / "executor.py")
|
||||
OPS = _load("operations", HERE / "operations.py")
|
||||
stack_mod = _load("stack", REPO / "sso-mfa/k8s/synthetic-journey/stack.py")
|
||||
bea = X.bea
|
||||
|
||||
OPENBAO_IMAGE = "quay.io/openbao/openbao@sha256:436eaf9778cad75507ff70ea26ace30dcbe15606e619ac3823495663d7f7c115" # 2.5.4
|
||||
BIND_KV = "platform/workloads/net-kingdom/directory-binds/svc-authelia"
|
||||
JOURNEY_KV = "platform/workloads/net-kingdom/synthetic-journey/syn-journey"
|
||||
POLICY_FILE = REPO.parent / "railiance-platform/openbao/policies/net-kingdom-bind-executor-svc-authelia-write.hcl"
|
||||
JOURNEY_POLICY = 'path "platform/data/workloads/net-kingdom/synthetic-journey/syn-journey" { capabilities = ["read"] }\n'
|
||||
EXECUTOR = "service:nk-bind-executor"
|
||||
APPROVER_SUB = "synthetic-admin-sub"
|
||||
|
||||
|
||||
class OpenBaoDev:
|
||||
def __init__(self, network: str, tag: str):
|
||||
self.name = f"nk56-executor-{tag}-openbao"
|
||||
self.root = secrets.token_hex(16)
|
||||
stack_mod.docker("run", "--detach", "--pull=never", "--name", self.name, "--network", network,
|
||||
"--publish", "127.0.0.1::8200", "--cap-add", "IPC_LOCK", "--env", "BAO_DEV_ROOT_TOKEN_ID",
|
||||
OPENBAO_IMAGE, "server", "-dev", "-dev-listen-address=0.0.0.0:8200",
|
||||
env={"BAO_DEV_ROOT_TOKEN_ID": self.root})
|
||||
self.addr = f"http://127.0.0.1:{stack_mod.port(self.name, 8200)}"
|
||||
stack_mod.wait_http(self.addr + "/v1/sys/health", "openbao")
|
||||
self.admin = OPS.OpenBao(self.addr, token=self.root)
|
||||
|
||||
def call(self, method, path, body=None):
|
||||
return self.admin._call(method, path, body)
|
||||
|
||||
def setup(self, bind_value: str, journey_password: str, seed: bytes) -> str:
|
||||
self.call("POST", "/v1/sys/mounts/platform", {"type": "kv", "options": {"version": "2"}})
|
||||
self.admin.write(BIND_KV, {OPS.FIELD: bind_value}, cas=0)
|
||||
self.call("POST", "/v1/platform/metadata/workloads/net-kingdom/directory-binds/svc-authelia", {"cas_required": True})
|
||||
self.admin.write(JOURNEY_KV, {"PASSWORD": journey_password,
|
||||
"TOTP_SEED": base64.b32encode(seed).decode()}, cas=0)
|
||||
self.call("PUT", "/v1/sys/policies/acl/bind-executor-write", {"policy": POLICY_FILE.read_text()})
|
||||
self.call("PUT", "/v1/sys/policies/acl/journey-read", {"policy": JOURNEY_POLICY})
|
||||
token = self.call("POST", "/v1/auth/token/create", {"policies": ["bind-executor-write", "journey-read"],
|
||||
"no_default_policy": True, "ttl": "15m"})
|
||||
return token["auth"]["client_token"]
|
||||
|
||||
def remove(self):
|
||||
stack_mod.docker("rm", "--force", self.name, check=False)
|
||||
|
||||
|
||||
class Delivery:
|
||||
"""Stand-in for the ExternalSecret: projects the KV value into Authelia's configuration file."""
|
||||
|
||||
def __init__(self, stack, bao_admin):
|
||||
self.stack, self.bao, self.corrupt = stack, bao_admin, 0 # number of next deliveries to corrupt
|
||||
|
||||
def force_sync(self) -> dt.datetime:
|
||||
at = OPS.now_utc()
|
||||
value = self.bao.read(BIND_KV)[0][OPS.FIELD]
|
||||
if self.corrupt:
|
||||
self.corrupt -= 1
|
||||
value = secrets.token_urlsafe(32) # a one-off wrong projection that still reports success
|
||||
path = self.stack.workdir / "authelia.yml"
|
||||
config = yaml.safe_load(path.read_text())
|
||||
config["authentication_backend"]["ldap"]["password"] = value
|
||||
path.write_text(yaml.safe_dump(config)) # same inode: the bind mount sees it
|
||||
self.synced_at = OPS.now_utc()
|
||||
return at
|
||||
|
||||
def synced(self, since: dt.datetime) -> bool:
|
||||
return getattr(self, "synced_at", None) is not None and self.synced_at >= since
|
||||
|
||||
|
||||
class Restart:
|
||||
"""Stand-in for the rollout: restart the Authelia container and wait for it to answer."""
|
||||
|
||||
def __init__(self, stack):
|
||||
self.stack = stack
|
||||
|
||||
def restart(self):
|
||||
stack_mod.docker("restart", self.stack.names["authelia"])
|
||||
# docker assigns a new ephemeral host port on restart (an artifact of this stand-in)
|
||||
self.stack.ports["authelia"] = stack_mod.port(self.stack.names["authelia"], 9091)
|
||||
try:
|
||||
stack_mod.wait_http(f"http://127.0.0.1:{self.stack.ports['authelia']}/api/health", "authelia",
|
||||
{"Host": "auth.journey.test", "X-Forwarded-Proto": "https"}, attempts=60)
|
||||
except stack_mod.SetupError:
|
||||
raise OPS.RolloutFailed("authelia did not become ready") from None
|
||||
|
||||
|
||||
class MemoryCarrier:
|
||||
"""approval-engine's approval object and consume semantics, in memory."""
|
||||
|
||||
def __init__(self):
|
||||
self.approvals = {}
|
||||
|
||||
def approve(self, approval_id, admission):
|
||||
spec = admission["spec"]
|
||||
self.approvals[approval_id] = {
|
||||
"id": approval_id, "status": "approved",
|
||||
"binding": {"action": "bounded-execution", "actor": EXECUTOR, "principal": "admin", "purpose": spec["purpose"],
|
||||
"target": {"admission_id": spec["id"], "admission_digest": bea.admission_digest(admission)},
|
||||
"human_control": True, "pdp_path": False},
|
||||
"validity": {"not_before": spec["window"]["not_before"], "expires_at": spec["window"]["not_after"]},
|
||||
"entries": [{"subject_id": APPROVER_SUB, "assurance": "aal2", "principal_type": "human",
|
||||
"approved_at": OPS.now_utc().isoformat()}]}
|
||||
|
||||
def get(self, approval_id):
|
||||
return copy.deepcopy(self.approvals[approval_id])
|
||||
|
||||
def consume(self, approval_id, request_digest, decision_id):
|
||||
a = self.approvals[approval_id]
|
||||
if a["status"] != "approved":
|
||||
raise RuntimeError("409")
|
||||
a["status"] = "consumed"
|
||||
a["consumption"] = {"request_digest": request_digest, "decision_id": decision_id,
|
||||
"consumed_at": OPS.now_utc().isoformat()}
|
||||
return {"approval_id": approval_id, "status": "consumed", "request_digest": request_digest,
|
||||
"decision_id": decision_id, "consumed_at": a["consumption"]["consumed_at"], "idempotent": False}
|
||||
|
||||
|
||||
def admission(admission_id: str, attempts: int) -> dict:
|
||||
"""The pilot example, pinned to the current files, with a window around now."""
|
||||
doc = bea.load(REPO / "examples/bounded-execution-admission/bind-rotation-svc-authelia.yaml")
|
||||
spec = doc["spec"]
|
||||
spec["id"] = admission_id
|
||||
now = dt.datetime.now(dt.UTC).replace(microsecond=0)
|
||||
spec["window"] = {"not_before": (now - dt.timedelta(minutes=5)).isoformat().replace("+00:00", "Z"),
|
||||
"not_after": (now + dt.timedelta(hours=2)).isoformat().replace("+00:00", "Z")}
|
||||
spec["budget"]["attempts"] = attempts
|
||||
files = []
|
||||
for f in spec["plan"]["source"]["files"]:
|
||||
files.append({"path": f["path"], "sha256": hashlib.sha256((REPO / f["path"]).read_bytes()).hexdigest()})
|
||||
spec["plan"]["source"]["files"] = files
|
||||
pins = {f["path"]: f["sha256"] for f in files}
|
||||
for c in spec["checks"]:
|
||||
c["sha256"] = pins[c["ref"]]
|
||||
spec["plan_digest"] = bea.plan_digest(doc)
|
||||
report = bea.evaluate(doc, json.loads(bea.SCHEMA.read_text()))
|
||||
if not report["valid"]:
|
||||
raise SystemExit(f"synthetic admission invalid: {report['findings']}")
|
||||
return doc
|
||||
|
||||
|
||||
def lldap_command():
|
||||
return ["docker", "run", "--rm", "--pull=never", "--network", "host", "--env", "LLDAP_USER_PASSWORD",
|
||||
"--entrypoint", "/app/lldap_set_password", stack_mod.LLDAP_IMAGE]
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
parser = argparse.ArgumentParser(description=__doc__.split("\n", 1)[0])
|
||||
parser.add_argument("--report")
|
||||
args = parser.parse_args(argv)
|
||||
report = {"schema": "netkingdom.bounded-executor.synthetic-pilot/v1",
|
||||
"started_at": OPS.now_utc().isoformat(), "images": {"openbao": OPENBAO_IMAGE},
|
||||
"policy_file_sha256": hashlib.sha256(POLICY_FILE.read_bytes()).hexdigest(), "runs": []}
|
||||
ok = False
|
||||
with tempfile.TemporaryDirectory(prefix="nk56-executor-") as tmp:
|
||||
stack = stack_mod.Stack(pathlib.Path(tmp), pi_cache_timeout=120)
|
||||
bao = None
|
||||
try:
|
||||
stack.up()
|
||||
bao = OpenBaoDev(stack.network, stack.tag)
|
||||
executor_token = bao.setup(stack.pw["svc-authelia"], stack.pw[stack_mod.USER], stack.seed)
|
||||
executor_bao = OPS.OpenBao(bao.addr, token=executor_token)
|
||||
directory = OPS.Directory(stack.lldap, lldap_command())
|
||||
delivery = Delivery(stack, bao.admin)
|
||||
journey_state = {}
|
||||
operations, checks = OPS.pilot_registry(
|
||||
bao=executor_bao, directory=directory, delivery=delivery, restart=Restart(stack),
|
||||
bind_kv=BIND_KV, bind_user="svc-authelia", journey_kv=JOURNEY_KV,
|
||||
journey_config=stack.journey_config, journey_state=journey_state)
|
||||
carrier, events, alerts = MemoryCarrier(), [], []
|
||||
audit = type("Sink", (), {"emit": lambda _s, e: events.append(e)})()
|
||||
cfg = X.Config(executor_identity=EXECUTOR, runtime_principal="synthetic:executor-token",
|
||||
tenant="tenant:platform", environment="TEST", approver_subjects={"admin": APPROVER_SUB})
|
||||
deps = X.Deps(carrier=carrier, operations=operations, checks=checks, trust=lambda s: True,
|
||||
audit=audit, alert=lambda kind, info: alerts.append(kind))
|
||||
|
||||
def run(admission_id, attempts, corrupt):
|
||||
doc = admission(admission_id, attempts)
|
||||
approval = "apr-" + admission_id
|
||||
carrier.approve(approval, doc)
|
||||
before_value, before_version = bao.admin.read(BIND_KV)
|
||||
delivery.corrupt = 1 if corrupt else 0
|
||||
result = X.Run(doc, approval, cfg, deps).execute()
|
||||
delivery.corrupt = 0
|
||||
after_value, after_version = bao.admin.read(BIND_KV)
|
||||
changed = after_value != before_value
|
||||
old_refused = directory.login("svc-authelia", before_value[OPS.FIELD]) is None
|
||||
new_works = directory.login("svc-authelia", after_value[OPS.FIELD]) is not None
|
||||
receipt = result.get("receipt", {}).get("data", {})
|
||||
return {"admission": admission_id, "outcome": result["outcome"], "reason": result.get("reason"),
|
||||
"kv_version": [before_version, after_version], "kv_value_changed": changed,
|
||||
"previous_value_refused_by_lldap": old_refused, "current_kv_value_signs_in": new_works,
|
||||
"steps": [(s["id"], s["kind"], s["outcome"], [c["result"] for c in s.get("checks", [])],
|
||||
s.get("error_type"))
|
||||
for s in receipt.get("steps", [])],
|
||||
"reverted": receipt.get("reverted"), "pdp": receipt.get("pdp"),
|
||||
"events": [e["type"] for e in events if e["correlation_id"] == result.get("run_id")]}
|
||||
|
||||
rotation = run("BEA-2026-9001", 1, corrupt=False)
|
||||
rotation["met"] = (rotation["outcome"] == "completed" and rotation["kv_value_changed"]
|
||||
and rotation["previous_value_refused_by_lldap"] and rotation["current_kv_value_signs_in"])
|
||||
report["runs"].append(rotation)
|
||||
revert = run("BEA-2026-9002", 1, corrupt=True)
|
||||
revert["met"] = (revert["outcome"] == "reverted" and not revert["kv_value_changed"]
|
||||
and revert["current_kv_value_signs_in"]
|
||||
and any(s[0] == "restart" and s[2] in ("check_failed", "failed") for s in revert["steps"])
|
||||
and all(s[2] == "done" for s in revert["steps"] if s[1] == "revert"))
|
||||
report["runs"].append(revert)
|
||||
report["alerts"] = alerts
|
||||
secrets_in_report = [stack.pw["svc-authelia"], executor_token, bao.root]
|
||||
text = json.dumps(report)
|
||||
report["no_values_in_report"] = not any(v in text for v in secrets_in_report)
|
||||
ok = rotation["met"] and revert["met"] and report["no_values_in_report"]
|
||||
finally:
|
||||
if bao:
|
||||
bao.remove()
|
||||
stack.down()
|
||||
report["all_met"] = ok
|
||||
report["finished_at"] = OPS.now_utc().isoformat()
|
||||
text = json.dumps(report, indent=2, default=str)
|
||||
if args.report:
|
||||
pathlib.Path(args.report).write_text(text + "\n")
|
||||
print(text)
|
||||
return 0 if ok else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
|
|
@ -299,11 +299,12 @@ class FailureTests(Base):
|
|||
self.w.fail_check["eso-synced"] = 1
|
||||
result = self.run_it()
|
||||
self.assertEqual(result["outcome"], "reverted", result)
|
||||
self.assertEqual(self.ops(), ["generate", "set-directory", "deliver", "redeliver", "restore-directory", "restore-kv"])
|
||||
# declared recovery order, only for steps that ran (restart did not run, so no restart-again)
|
||||
self.assertEqual(self.ops(), ["generate", "set-directory", "deliver", "restore-kv", "restore-directory", "redeliver"])
|
||||
data = result["receipt"]["data"]
|
||||
self.assertTrue(data["reverted"])
|
||||
reverts = [x for x in data["steps"] if x["kind"] == "revert"]
|
||||
self.assertEqual([x["checks"][0]["id"] for x in reverts], ["eso-synced", "bind-signs-in", "kv-advanced"])
|
||||
self.assertEqual([x["checks"][0]["id"] for x in reverts], ["kv-advanced", "bind-signs-in", "eso-synced"])
|
||||
self.assertTrue(all(x["outcome"] == "done" for x in reverts))
|
||||
self.assertEqual(data["attempts"], 1)
|
||||
|
||||
|
|
@ -319,10 +320,21 @@ class FailureTests(Base):
|
|||
def test_revert_whose_check_fails_counts_as_failed(self):
|
||||
self.doc = synthetic_admission(self.root, lambda s: s["budget"].update(attempts=1))
|
||||
self.w = World(self.doc, self.clock)
|
||||
self.w.fail_check["eso-synced"] = 2 # the forward check and the revert's own check
|
||||
self.w.fail_check["eso-synced"] = 1
|
||||
self.w.fail_check["kv-advanced"] = 0
|
||||
original = self.w.check
|
||||
calls = {"kv": 0}
|
||||
def check(c, ctx): # kv-advanced passes forward, fails on the revert
|
||||
if c["id"] == "kv-advanced":
|
||||
calls["kv"] += 1
|
||||
if calls["kv"] == 2:
|
||||
self.w.calls.append("check:kv-advanced")
|
||||
return False
|
||||
return original(c, ctx)
|
||||
self.w.check = check
|
||||
result = self.run_it()
|
||||
self.assertEqual(result["outcome"], "revert_failed")
|
||||
self.assertNotIn("restore-directory", self.ops())
|
||||
self.assertNotIn("restore-directory", self.ops()) # stop at the failed revert
|
||||
|
||||
def test_failed_revert_stops_and_alerts(self):
|
||||
self.w.fail_check["eso-synced"] = 1
|
||||
|
|
@ -330,7 +342,7 @@ class FailureTests(Base):
|
|||
result = self.run_it()
|
||||
self.assertEqual(result["outcome"], "revert_failed")
|
||||
self.assertIn("revert_failed", self.w.alerts)
|
||||
self.assertNotIn("restore-kv", self.ops()) # stop at the failed revert, do not continue blindly
|
||||
self.assertNotIn("redeliver", self.ops()) # stop at the failed revert, do not continue blindly
|
||||
|
||||
def test_unknown_outcome_stops_without_revert(self):
|
||||
self.w.unknown_op = {"eso.force-sync"}
|
||||
|
|
@ -340,6 +352,23 @@ class FailureTests(Base):
|
|||
self.assertFalse(result["receipt"]["data"]["reverted"])
|
||||
self.assertIn("unknown", self.w.alerts)
|
||||
|
||||
def test_definite_failure_reverts_like_a_failed_check(self):
|
||||
self.doc = synthetic_admission(self.root, lambda s: s["budget"].update(attempts=1))
|
||||
self.w = World(self.doc, self.clock)
|
||||
original = self.w.op
|
||||
def op(step, ctx):
|
||||
if step["id"] == "restart":
|
||||
self.w.calls.append("restart")
|
||||
raise X.StepFailed("never became ready")
|
||||
return original(step, ctx)
|
||||
self.w.op = op
|
||||
result = self.run_it()
|
||||
self.assertEqual(result["outcome"], "reverted")
|
||||
self.assertEqual(self.ops(), ["generate", "set-directory", "deliver", "restart",
|
||||
"restore-kv", "restore-directory", "redeliver", "restart-again"])
|
||||
restart = next(s for s in result["receipt"]["data"]["steps"] if s["id"] == "restart")
|
||||
self.assertEqual((restart["outcome"], restart["error_type"]), ("failed", "StepFailed"))
|
||||
|
||||
def test_budget_exhausted_reverts(self):
|
||||
self.w.slow_op = 700.0 # wall_time_seconds is 1200
|
||||
result = self.run_it()
|
||||
|
|
|
|||
133
bounded-executor/tests/test_operations.py
Normal file
133
bounded-executor/tests/test_operations.py
Normal file
|
|
@ -0,0 +1,133 @@
|
|||
"""Kubernetes adapters of the pilot operations against a fake API server (NK-WP-0056-T12). No cluster."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import datetime as dt
|
||||
import importlib.util
|
||||
import json
|
||||
import pathlib
|
||||
import sys
|
||||
import tempfile
|
||||
import threading
|
||||
import unittest
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
|
||||
HERE = pathlib.Path(__file__).resolve().parent
|
||||
spec = importlib.util.spec_from_file_location("operations", HERE.parent / "operations.py")
|
||||
OPS = importlib.util.module_from_spec(spec)
|
||||
sys.modules["operations"] = OPS
|
||||
spec.loader.exec_module(OPS)
|
||||
|
||||
ES = "/apis/external-secrets.io/v1/namespaces/sso/externalsecrets/authelia-secrets"
|
||||
DEP = "/apis/apps/v1/namespaces/sso/deployments/authelia"
|
||||
T0 = dt.datetime(2026, 11, 3, 9, 30, tzinfo=dt.UTC)
|
||||
|
||||
|
||||
class Api:
|
||||
def __init__(self):
|
||||
self.patches = []
|
||||
self.auth = []
|
||||
self.es_status = {"conditions": [{"type": "Ready", "status": "True"}], "refreshTime": "2026-11-03T09:29:00Z"}
|
||||
self.dep = {"metadata": {"generation": 5}, "spec": {"replicas": 1},
|
||||
"status": {"observedGeneration": 5, "replicas": 1, "updatedReplicas": 1, "availableReplicas": 1}}
|
||||
self.gets_until_ready = 0
|
||||
self.forbidden = False
|
||||
|
||||
|
||||
def serve(api: Api):
|
||||
class Handler(BaseHTTPRequestHandler):
|
||||
def _send(self, code, body):
|
||||
raw = json.dumps(body).encode()
|
||||
self.send_response(code)
|
||||
self.send_header("Content-Type", "application/json")
|
||||
self.send_header("Content-Length", str(len(raw)))
|
||||
self.end_headers()
|
||||
self.wfile.write(raw)
|
||||
|
||||
def do_GET(self):
|
||||
api.auth.append(self.headers.get("Authorization"))
|
||||
if api.forbidden:
|
||||
return self._send(403, {"kind": "Status"})
|
||||
if self.path == ES:
|
||||
return self._send(200, {"status": api.es_status})
|
||||
if self.path == DEP:
|
||||
if api.gets_until_ready > 0:
|
||||
api.gets_until_ready -= 1
|
||||
return self._send(200, {**api.dep, "status": {**api.dep["status"], "updatedReplicas": 0,
|
||||
"unavailableReplicas": 1}})
|
||||
return self._send(200, api.dep)
|
||||
self._send(404, {})
|
||||
|
||||
def do_PATCH(self):
|
||||
api.patches.append((self.path, self.headers.get("Content-Type"),
|
||||
json.loads(self.rfile.read(int(self.headers["Content-Length"])))))
|
||||
if api.forbidden:
|
||||
return self._send(403, {"kind": "Status"})
|
||||
if self.path == DEP:
|
||||
api.dep["metadata"]["generation"] += 1
|
||||
api.dep["status"]["observedGeneration"] = api.dep["metadata"]["generation"]
|
||||
self._send(200, {})
|
||||
|
||||
def log_message(self, *a):
|
||||
return
|
||||
|
||||
server = ThreadingHTTPServer(("127.0.0.1", 0), Handler)
|
||||
threading.Thread(target=server.serve_forever, kwargs={"poll_interval": 0.02}, daemon=True).start()
|
||||
return server
|
||||
|
||||
|
||||
class KubeTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.api = Api()
|
||||
self.server = serve(self.api)
|
||||
self.addCleanup(self.server.shutdown)
|
||||
tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(tmp.cleanup)
|
||||
token = pathlib.Path(tmp.name) / "token"
|
||||
token.write_text("synthetic-sa-token\n")
|
||||
self.kube = OPS.Kube(api=f"http://127.0.0.1:{self.server.server_port}", token_file=str(token), ca_file=None)
|
||||
|
||||
def test_force_sync_patches_only_the_annotation(self):
|
||||
delivery = OPS.KubeDelivery(self.kube, "sso", "authelia-secrets", clock=lambda: T0)
|
||||
self.assertEqual(delivery.force_sync(), T0)
|
||||
((path, ctype, body),) = self.api.patches
|
||||
self.assertEqual((path, ctype), (ES, "application/merge-patch+json"))
|
||||
self.assertEqual(body, {"metadata": {"annotations": {"force-sync": str(int(T0.timestamp()))}}})
|
||||
self.assertEqual(self.api.auth, []) # no read before the patch
|
||||
|
||||
def test_synced_needs_ready_and_a_refresh_after_the_sync(self):
|
||||
delivery = OPS.KubeDelivery(self.kube, "sso", "authelia-secrets")
|
||||
self.assertFalse(delivery.synced(T0)) # refreshed before T0
|
||||
self.api.es_status["refreshTime"] = "2026-11-03T09:30:05Z"
|
||||
self.assertTrue(delivery.synced(T0))
|
||||
self.api.es_status["conditions"][0]["status"] = "False"
|
||||
self.assertFalse(delivery.synced(T0))
|
||||
|
||||
def test_restart_patches_restarted_at_and_waits_for_the_rollout(self):
|
||||
self.api.gets_until_ready = 2
|
||||
restart = OPS.KubeRestart(self.kube, "sso", "authelia", timeout=10, poll=0.01, clock=lambda: T0)
|
||||
restart.restart()
|
||||
((path, _, body),) = self.api.patches
|
||||
self.assertEqual(path, DEP)
|
||||
self.assertEqual(body["spec"]["template"]["metadata"]["annotations"]["kubectl.kubernetes.io/restartedAt"],
|
||||
"2026-11-03T09:30:00Z")
|
||||
self.assertEqual(self.api.gets_until_ready, 0)
|
||||
|
||||
def test_rollout_that_never_becomes_ready_is_a_definite_failure(self):
|
||||
self.api.gets_until_ready = 10 ** 6
|
||||
ticks = iter(range(10 ** 6))
|
||||
restart = OPS.KubeRestart(self.kube, "sso", "authelia", timeout=5, poll=0, monotonic=lambda: next(ticks),
|
||||
sleep=lambda s: None)
|
||||
with self.assertRaises(OPS.RolloutFailed) as caught:
|
||||
restart.restart()
|
||||
self.assertTrue(caught.exception.definite_failure)
|
||||
|
||||
def test_forbidden_is_an_operation_error_and_sends_the_bearer(self):
|
||||
self.api.forbidden = True
|
||||
with self.assertRaises(OPS.OperationError):
|
||||
OPS.KubeDelivery(self.kube, "sso", "authelia-secrets").synced(T0)
|
||||
self.assertEqual(self.api.auth, ["Bearer synthetic-sa-token"])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
|
@ -105,7 +105,7 @@ for up to 120 s, so a check run sooner proves nothing.
|
|||
| BEA-7 | No `human-intrinsic` step. A `root-class` step needs `high` risk |
|
||||
| BEA-8 | Every step's consumer is in the blast radius. `medium` and `high` risk change one consumer at a time |
|
||||
| BEA-9 | Every mutating step names checks, revert steps included: a revert must prove the restored state with its own checks. Every check is used and pinned: its `ref` and `sha256` appear in `plan.source.files` |
|
||||
| BEA-10 | Every mutating step has a revert step, or is `irreversible` at `high` risk. Revert steps mutate and do not nest. Step ids are unique |
|
||||
| BEA-10 | Every mutating step has a revert step, or is `irreversible` at `high` risk. Revert steps mutate and do not nest. Step ids are unique. The executor runs revert steps in their declared order (the recovery order: data, then delivery, then restart), limited to those whose forward step ran |
|
||||
| BEA-11 | A revertable plan reverts on check failure. A plan with nothing to revert stops |
|
||||
| BEA-12 | The wall time fits the window, the checks (timeouts plus settle times) fit the wall time, and attempts are at most 3 (1 at `high` risk) |
|
||||
| BEA-13 | The executed source is pinned by file and commit |
|
||||
|
|
@ -208,6 +208,8 @@ consume, and `execute-step` after it. The T06 pilot runs without the gate: its r
|
|||
gate as `not-evaluated` and claims no allow. Any further admission needs the gate, or an exception
|
||||
the founder records (founder decision 2026-10-31).
|
||||
|
||||
**Failure classes.** A failed check, or an operation that reports a definite failure (for example a rollout that never becomes ready), reverts per `on_check_failure`. An error whose outcome is not established stops (`on_unknown: stop`) without reverting. Receipts name the error type, never its message.
|
||||
|
||||
**Credentials during a run.** Write roles exist only during the admitted window: the platform creates
|
||||
them when an admission is approved and deletes them after the receipt (CCR-2026-0058). A directory
|
||||
bind rotates its own password, so no password-manager account is needed (founder decisions
|
||||
|
|
|
|||
41
docs/evidence/2026-10-31-nk56-t12-synthetic-pilot/README.md
Normal file
41
docs/evidence/2026-10-31-nk56-t12-synthetic-pilot/README.md
Normal file
|
|
@ -0,0 +1,41 @@
|
|||
# NK-WP-0056-T12: the executor rotates a bind on the synthetic stack (2026-10-31)
|
||||
|
||||
Agent-run, local Docker, generated values, everything removed afterwards: `uv run bounded-executor/synthetic_pilot.py`.
|
||||
|
||||
**Setup:**
|
||||
|
||||
- The T08 stack: LLDAP, privacyIDEA, Authelia and KeyCape on the pinned images.
|
||||
- An OpenBao dev server on the platform's pinned 2.5.4 image, with KV v2 at `platform` and `cas_required` on the
|
||||
`svc-authelia` bind.
|
||||
- The executor gets its own token with CCR-2026-0058's policy file plus a journey-read policy.
|
||||
- Stand-ins:
|
||||
- approval-engine: an in-process carrier with its consume semantics;
|
||||
- audit-core: an in-memory sink;
|
||||
- the ExternalSecret: re-projects the KV value into Authelia's configuration;
|
||||
- the rollout: restarts the Authelia container.
|
||||
- The Kubernetes adapters are unit-tested against a fake API server (`bounded-executor/tests/test_operations.py`).
|
||||
|
||||
Report: [`pilot-report.json`](pilot-report.json). It is value-free: scanned, and the harness asserts that no generated
|
||||
value is present.
|
||||
|
||||
| Run | Outcome | Evidence |
|
||||
| --- | --- | --- |
|
||||
| BEA-2026-9001 rotation | `completed` | KV v1→v2; LLDAP refuses the old value and accepts the new; delivery, restart and synthetic journey pass; started event and receipt emitted |
|
||||
| BEA-2026-9002 one-off wrong delivery | `reverted` | Authelia never becomes ready after the restart (`RolloutFailed`, a definite failure). Revert in declared order, each step passing its own check: `restore-kv` → `restore-directory` → `redeliver` → `restart-again`. The value from before the run is current again (KV v4); the journey passes |
|
||||
|
||||
## Findings that changed the design
|
||||
|
||||
1. **CCR-2026-0058's policy refused reads.** OpenBao 2.5.4 applies `required_parameters = ["options"]` to GET as well,
|
||||
so the executor could not read the version it must name for CAS. The executor token got 403 on every read.
|
||||
- Without that line, read and CAS write work.
|
||||
- A stale CAS and a write without `options` are refused, by `cas_required` on the key.
|
||||
- Metadata, delete, sibling reads and new paths are still denied.
|
||||
- The fix is in railiance-platform's renderer and policy file. CCR-2026-0058 needs re-approval.
|
||||
2. **Revert order.** Undoing in strict reverse order would restart Authelia before its password was restored. Revert
|
||||
steps now run in the order the admission declares (data, then delivery, then restart), limited to steps that ran.
|
||||
The standard's BEA-10 says so.
|
||||
3. **Definite failures revert.** A rollout that never becomes ready was first handled like an unknown outcome
|
||||
(stop, no revert). Operations can now signal a definite failure (`definite_failure`, `executor.StepFailed`),
|
||||
which the runner reverts like a failed check. Truly ambiguous errors still stop.
|
||||
4. **Error types in receipts.** Receipts now carry the error *type* of a failed or unknown step, never its message.
|
||||
5. **Stand-in artifact:** a restarted container gets a new host port. Fixed in the harness; not relevant to Kubernetes.
|
||||
|
|
@ -0,0 +1,162 @@
|
|||
{
|
||||
"schema": "netkingdom.bounded-executor.synthetic-pilot/v1",
|
||||
"started_at": "2026-10-31T15:21:55+00:00",
|
||||
"images": {
|
||||
"openbao": "quay.io/openbao/openbao@sha256:436eaf9778cad75507ff70ea26ace30dcbe15606e619ac3823495663d7f7c115"
|
||||
},
|
||||
"policy_file_sha256": "f14f0e0b885302b5931c1e84e258b5ee32a596fafab5b961d88becf7825eeeb0",
|
||||
"runs": [
|
||||
{
|
||||
"admission": "BEA-2026-9001",
|
||||
"outcome": "completed",
|
||||
"reason": null,
|
||||
"kv_version": [
|
||||
1,
|
||||
2
|
||||
],
|
||||
"kv_value_changed": true,
|
||||
"previous_value_refused_by_lldap": true,
|
||||
"current_kv_value_signs_in": true,
|
||||
"steps": [
|
||||
[
|
||||
"generate",
|
||||
"step",
|
||||
"done",
|
||||
[
|
||||
"pass"
|
||||
],
|
||||
null
|
||||
],
|
||||
[
|
||||
"set-directory",
|
||||
"step",
|
||||
"done",
|
||||
[
|
||||
"pass"
|
||||
],
|
||||
null
|
||||
],
|
||||
[
|
||||
"deliver",
|
||||
"step",
|
||||
"done",
|
||||
[
|
||||
"pass"
|
||||
],
|
||||
null
|
||||
],
|
||||
[
|
||||
"restart",
|
||||
"step",
|
||||
"done",
|
||||
[
|
||||
"pass"
|
||||
],
|
||||
null
|
||||
]
|
||||
],
|
||||
"reverted": false,
|
||||
"pdp": "not-evaluated",
|
||||
"events": [
|
||||
"execution.run.started",
|
||||
"execution.attribution.receipt"
|
||||
],
|
||||
"met": true
|
||||
},
|
||||
{
|
||||
"admission": "BEA-2026-9002",
|
||||
"outcome": "reverted",
|
||||
"reason": null,
|
||||
"kv_version": [
|
||||
2,
|
||||
4
|
||||
],
|
||||
"kv_value_changed": false,
|
||||
"previous_value_refused_by_lldap": false,
|
||||
"current_kv_value_signs_in": true,
|
||||
"steps": [
|
||||
[
|
||||
"generate",
|
||||
"step",
|
||||
"done",
|
||||
[
|
||||
"pass"
|
||||
],
|
||||
null
|
||||
],
|
||||
[
|
||||
"set-directory",
|
||||
"step",
|
||||
"done",
|
||||
[
|
||||
"pass"
|
||||
],
|
||||
null
|
||||
],
|
||||
[
|
||||
"deliver",
|
||||
"step",
|
||||
"done",
|
||||
[
|
||||
"pass"
|
||||
],
|
||||
null
|
||||
],
|
||||
[
|
||||
"restart",
|
||||
"step",
|
||||
"failed",
|
||||
[],
|
||||
"RolloutFailed"
|
||||
],
|
||||
[
|
||||
"restore-kv",
|
||||
"revert",
|
||||
"done",
|
||||
[
|
||||
"pass"
|
||||
],
|
||||
null
|
||||
],
|
||||
[
|
||||
"restore-directory",
|
||||
"revert",
|
||||
"done",
|
||||
[
|
||||
"pass"
|
||||
],
|
||||
null
|
||||
],
|
||||
[
|
||||
"redeliver",
|
||||
"revert",
|
||||
"done",
|
||||
[
|
||||
"pass"
|
||||
],
|
||||
null
|
||||
],
|
||||
[
|
||||
"restart-again",
|
||||
"revert",
|
||||
"done",
|
||||
[
|
||||
"pass"
|
||||
],
|
||||
null
|
||||
]
|
||||
],
|
||||
"reverted": true,
|
||||
"pdp": "not-evaluated",
|
||||
"events": [
|
||||
"execution.run.started",
|
||||
"execution.attribution.receipt"
|
||||
],
|
||||
"met": true
|
||||
}
|
||||
],
|
||||
"alerts": [],
|
||||
"no_values_in_report": true,
|
||||
"all_met": true,
|
||||
"finished_at": "2026-10-31T15:22:57+00:00"
|
||||
}
|
||||
|
|
@ -470,7 +470,7 @@ Standard and validator change: BEA-9 now applies to revert steps, which name the
|
|||
|
||||
```task
|
||||
id: NK-WP-0056-T12
|
||||
status: todo
|
||||
status: done
|
||||
priority: high
|
||||
assignee: claude-code
|
||||
depends_on: [NK-WP-0056-T11, NK-WP-0056-T08]
|
||||
|
|
@ -489,7 +489,51 @@ Run them on the T08 synthetic stack, including a revert. Add the Containerfile a
|
|||
net-kingdom. The `rapp-bounded-executor` package is created with `rmgr rapp wrap`, and the founder creates the
|
||||
Forgejo remote. It runs one Job per admission in `sso` with an egress-only NetworkPolicy.
|
||||
|
||||
**Done when:** a synthetic rotation completes and a deliberately failing one reverts, with receipts; the image builds.
|
||||
**Done when:** a synthetic rotation completes and a deliberately failing one reverts, with receipts.
|
||||
|
||||
**Done 2026-10-31** ([evidence](../docs/evidence/2026-10-31-nk56-t12-synthetic-pilot/README.md)).
|
||||
`bounded-executor/operations.py`:
|
||||
|
||||
- the OpenBao KV v2 CAS client;
|
||||
- LLDAP self-rotation, the same operation forward and in revert;
|
||||
- the Kubernetes ExternalSecret force-sync and status check, and the rollout restart that raises `RolloutFailed`;
|
||||
- the pilot checks.
|
||||
|
||||
`bounded-executor/synthetic_pilot.py`: rotation `completed`, and a one-off wrong delivery `reverted` in declared order
|
||||
with the earlier value restored. 29 tests.
|
||||
|
||||
Findings:
|
||||
|
||||
- CCR-2026-0058's `required_parameters` refused reads. Corrected in railiance-platform; needs re-approval.
|
||||
- Revert steps now run in declared order (BEA-10).
|
||||
- Definite failures revert; unknown outcomes stop.
|
||||
- Receipts carry error types.
|
||||
|
||||
The image part moved to T13 (split agreed 2026-10-31).
|
||||
|
||||
## Executor runtime: packaging
|
||||
|
||||
```task
|
||||
id: NK-WP-0056-T13
|
||||
status: todo
|
||||
priority: high
|
||||
assignee: claude-code
|
||||
depends_on: [NK-WP-0056-T12]
|
||||
```
|
||||
|
||||
The CLI entry point for one admission:
|
||||
|
||||
- reads the admission, the approval id and the configuration;
|
||||
- uses Kubernetes auth to OpenBao;
|
||||
- uses the in-cluster Kubernetes and audit sender;
|
||||
- ships the pinned `lldap_set_password` in the image.
|
||||
|
||||
Also: the Containerfile and the Forgejo image workflow in net-kingdom, and the `rapp-bounded-executor` package
|
||||
(`rmgr rapp wrap`, founder creates the Forgejo remote). The package holds one Job per admission in `sso` with the
|
||||
T03 ServiceAccount, Role and an egress-only NetworkPolicy.
|
||||
|
||||
**Done when:** the image builds in the workflow, the rapp renders and passes `server-dry-run` against a synthetic
|
||||
manifest check, and nothing is applied.
|
||||
|
||||
## Pilot: unattended bind rotation under one admission
|
||||
|
||||
|
|
@ -500,7 +544,7 @@ priority: high
|
|||
assignee: claude-code
|
||||
needs_human: true
|
||||
intervention_note: "Founder signs the admission for the single live pilot rotation."
|
||||
depends_on: [NK-WP-0056-T01, NK-WP-0056-T03, NK-WP-0056-T04, NK-WP-0056-T05, NK-WP-0056-T10, NK-WP-0056-T12]
|
||||
depends_on: [NK-WP-0056-T01, NK-WP-0056-T03, NK-WP-0056-T04, NK-WP-0056-T05, NK-WP-0056-T10, NK-WP-0056-T13]
|
||||
state_hub_task_id: "6c3939f8-23a9-5f70-a3b1-94f4b291706b"
|
||||
```
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue