Advance supervised agent records and close verified Secret annotation guard

docs/changes/CUST-WP-0073/README.md

@@ -0,0 +1,125 @@
+# Credential separation — reviewed change package
+
+2026-09-28. Prepared under the existing CUST-WP-0073 tasks. The initial admission rollout was rolled back, then corrected and re-enabled; see the
+[rollout receipt](../../evidence/2026-09-28-secret-annotation-rollout.md). The founder selected agent-specific supervised/autopilot modes in
+[the decision record](../../agent-autonomy-decision.md). The concrete supervised
+execution path and account cutover remain unimplemented.
+
+## Verified current state
+
+`ssh railiance01` runs as `tegwick` (uid 1000), with `(ALL) NOPASSWD: ALL`.
+`/etc/rancher/k3s/k3s.yaml` is `644 root root`.
+`kubectl auth whoami` reports `system:admin`, `system:masters`.
+No credential contents were read. The public principal inventory in
+`railiance-infra/ansible/inventory/ssh_principals.yaml` maps both `agt-*` and
+`adm-full` to `tegwick`. ops-warden issues certificates; railiance-infra owns
+host principal mapping. A different principal on this same account does not
+separate privilege.
+
+## Supervised starting identity and later scoped promotion (T01/T02)
+
+Use separate agent and admin OS identities on both the workstation and server.
+Agents must not inherit the admin SSH key/certificate, SSH agent socket, sudo,
+container-runtime socket, kubeconfig, OpenBao token or admin home directory.
+Moving a file or changing the default context under the same unrestricted
+account is insufficient. Keep an independently verified attended admin session
+open during cutover; verify recovery before withdrawing the old agent path.
+
+The supervised starting Kubernetes identity is outside `system:masters`, with an
+explicit reviewed observation allowlist. Do not simply bind built-in `view`:
+ConfigMaps, pod specs and logs can themselves contain credentials. Do not grant
+workload edits, arbitrary ConfigMap writes, exec/attach/portforward, proxy,
+logs, Secret verbs, token issuance, impersonation, RBAC writes or CSR approval.
+Broader action authority is a per-agent autopilot grant earned through evidence,
+with cost and risk limits in EUR; it is not unrestricted credential access.
+
+Deployment create/patch authority allows code or mounts to extract credentials.
+This is an upstream documented escalation route, not a missing deny rule:
+[Kubernetes RBAC good practices](https://kubernetes.io/docs/concepts/security/rbac-good-practices/).
+Agents prepare exact changes; in supervised-mode the supervisor approves or runs
+them through the privileged path. An agent-controlled GitOps write path must obey
+that same gate. Later autopilot may execute scoped actions without individual
+approval only through verified policy and budget/risk enforcement. Otherwise it
+recreates the bypass. Mode, supervisor, proposal dispositions and verified
+outcomes belong to the identified agent's existing records and receipts.
+
+Set k3s's persistent kubeconfig mode to `600` in railiance-enablement and fix
+the existing file, but do not mistake that step for OS-account separation.
+The final selected launcher/account setup must prove agents cannot regain the
+old admin account, including through workstation/Windows interoperability.
+No new credential broker or harness implementation is proposed.
+
+Acceptance uses the actual agent process/account, not only admin impersonation:
+whoami without masters; denied Secret get/list/watch; denied pod exec, workload
+writes, logs, token issuance and impersonation; denied admin-file reads and
+sudo; no accessible admin socket/key/token; approved observation succeeds;
+attended admin recovery succeeds. Record authorization booleans and file access
+results, never credential contents. Negative checks must not attempt to print
+an actual secret if access unexpectedly succeeds.
+
+## Secret annotation policy (T03)
+
+`reject-secret-last-applied.yaml` contains a native v1 policy and Deny binding.
+It rejects the annotation key even when its value is empty, on CREATE/UPDATE,
+cluster-wide. It introduces no controller or workload. Server dry-run on the
+verified v1.35.1+k3s1 cluster accepted both objects on September 28:
+
+```text
+validatingadmissionpolicy.admissionregistration.k8s.io/reject-secret-last-applied serverside-applied (server dry run)
+validatingadmissionpolicybinding.admissionregistration.k8s.io/reject-secret-last-applied serverside-applied (server dry run)
+```
+
+Subsequent live native tests passed, but actual ESO refresh failed and required
+rollback. Enforcement is now enabled after explicit metadata fixes and successful fresh
+refreshes of all 39 ExternalSecrets. The platform owner's pinned revision
+`7daf7e9` is authoritative; the original proposal file is retained for history.
+Reference: [ValidatingAdmissionPolicy](https://kubernetes.io/docs/reference/access-authn-authz/validating-admission-policy/).
+
+Before any retry, fix and verify the 31 ESO declarations described in the rollout
+receipt, then in one attended railiance-platform window: remove existing last-applied
+annotations without logging values; persist the manifest in that owner's
+deployment path; dry-run and diff; install policy and binding. Use only a
+synthetic, non-credential Secret in a scratch namespace to verify clean create
+and update succeed, annotated create/update (including empty annotation) fail,
+and client-side apply fails. Verify the policy type-check status, then remove
+the fixture. Record only names, booleans and counts. Do not use dry-run output
+of real Secret objects or print admission errors containing real values.
+
+If the policy interrupts a required write, the attended admin can delete its
+binding, fix the writer to use server-side apply/replace, and rebind. This
+temporarily reopens annotation leakage and must be recorded. The policy does
+not revoke any credential or stop other ways of reading secrets.
+
+## Guidance and deferred rotation (T04/T05)
+
+The September 24 standing notice exists. The raw presence template is now
+withdrawn: absent annotations can trigger a dump of the full Secret. Use only
+the maintenance helper, which captures and suppresses kubectl output on errors. Claude's recorded guard remains a
+stopgap. No equivalent Codex/Grok read-denial hook has been established by this
+work; this session has broad kubectl/SSH permissions, so instructions are the
+current protection. No claim is made that every Grok installation was inspected.
+OpenBao/Vault secret reads belong inside the same attended boundary, regardless
+of preapproved command prefixes.
+
+Rotation remains in existing T05 with the founder's September 24 trigger-based
+deferral. Do not silently cancel it or open another plan. At final closure,
+either execute the rotation in its attended window or explicitly resolve its
+existing disposition under the work-record rules. No rotation was performed.
+
+## Bounded implementation evidence
+
+The existing sandbox mechanism passed the synthetic checks in
+[the sandbox receipt](../../evidence/2026-09-28-supervised-sandbox-proof.json).
+It does not prove interactive agent migration. The per-agent record at
+`.kaizen/agents/custodian-codex/supervision.json` keeps this agent supervised,
+with no autopilot grant or EUR limits assigned. The descriptive summarizer
+`scripts/summarize_agent_supervision.py` cannot authorize or promote an agent.
+No eligible scoring sample exists; the failed annotation rollout and recovery
+are retained visibly rather than counted as unchanged success.
+
+Current T03 outcome: nine admission checks pass; all 39 ExternalSecrets refreshed
+successfully under Deny; the guard is Synced/Healthy. Source fixes and deployment
+receipts are in the linked rollout record. The absent-namespace orphan Secret was deleted after explicit founder approval,
+using its exact UID precondition; absence and unchanged 3 GiB PVC were verified.
+`orphan-secret-deletion.json` now contains the execution disposition. T03 is
+complete. Agent-runtime migration remains a separate unfinished task.

docs/changes/CUST-WP-0073/orphan-secret-deletion.json

@@ -0,0 +1,27 @@
+{
+  "reviewed_at": "2026-09-28T14:30:16.548657+00:00",
+  "resource": "Secret",
+  "namespace": "platform-pg-drill",
+  "name": "drill-minio",
+  "uid": "2fcb66df-d90f-4776-8ea0-8ca1a04bd307",
+  "created_at": "2026-08-13T11:09:33Z",
+  "namespace_exists": false,
+  "pods_in_namespace": 0,
+  "delete_options": {
+    "apiVersion": "v1",
+    "kind": "DeleteOptions",
+    "preconditions": {
+      "uid": "2fcb66df-d90f-4776-8ea0-8ca1a04bd307"
+    }
+  },
+  "proposal": "Delete only this orphan drill Secret, using the UID precondition. Do not recreate namespace, delete PVC or alter other resources.",
+  "reason": "Namespace is absent; API refuses annotation-only metadata update. Secret is an August 13 scratch drill artifact; referencing Deployment has zero Ready replicas and no pods.",
+  "risk": "Deletion removes the remaining credential copy in this orphan Secret. No Secret value has been inspected or archived.",
+  "storage": "Bound 3Gi platform-pg-drill-1 PVC and its PV retained unchanged.",
+  "approval": "Founder explicitly selected: Delete only the orphan Secret",
+  "executed": true,
+  "server_dry_run_passed": true,
+  "executed_at": "2026-09-28T16:07:36.040146+00:00",
+  "verified_absent": true,
+  "pvc_unchanged": true
+}

docs/changes/CUST-WP-0073/prove_secret_annotation_guard.py

@@ -0,0 +1,60 @@
+#!/usr/bin/env python3
+"""Positive/negative admission proof using only a uniquely named synthetic Secret."""
+import copy
+import json
+import subprocess
+import uuid
+from datetime import datetime, timezone
+
+KEY = "kubectl.kubernetes.io/last-applied-configuration"
+
+
+def run(args, obj=None):
+    return subprocess.run(["kubectl", "-n", "whitehat", *args],
+                          input=json.dumps(obj) if obj is not None else None,
+                          capture_output=True, text=True, timeout=30)
+
+
+def denied(result):
+    # Do not accept connectivity/RBAC failures as admission-policy success.
+    return result.returncode != 0 and "Secret last-applied annotations are forbidden" in result.stderr
+
+
+def main():
+    name = "cust-0073-proof-" + uuid.uuid4().hex[:12]
+    obj = {"apiVersion": "v1", "kind": "Secret", "metadata": {"name": name},
+           "type": "Opaque", "data": {"fixture": "c3ludGhldGlj"}}
+    report = {"captured_at": datetime.now(timezone.utc).isoformat(), "namespace": "whitehat",
+              "fixture": name, "synthetic_only": True, "checks": {}}
+    created = False
+    try:
+        result = run(["create", "--field-manager=cust-0073-proof", "-f", "-"], obj)
+        created = result.returncode == 0
+        report["checks"]["clean_create_allowed"] = created
+        if not created:
+            raise RuntimeError("synthetic create failed")
+        for label, value in [("empty", ""), ("populated", "synthetic")]:
+            annotated = copy.deepcopy(obj)
+            annotated["metadata"]["name"] = name + "-denied"
+            annotated["metadata"]["annotations"] = {KEY: value}
+            report["checks"][label + "_annotated_create_denied"] = denied(run(["create", "--dry-run=server", "-f", "-"], annotated))
+            patch = {"metadata": {"annotations": {KEY: value}}}
+            report["checks"][label + "_annotated_update_denied"] = denied(run(["patch", "secret", name, "--dry-run=server", "--type=merge", "-p", json.dumps(patch)]))
+        report["checks"]["client_apply_denied"] = denied(run(["apply", "--dry-run=server", "-f", "-"], obj))
+        report["checks"]["clean_server_apply_allowed"] = run(["apply", "--server-side", "--field-manager=cust-0073-proof", "-f", "-"], obj).returncode == 0
+        report["checks"]["clean_update_allowed"] = run(["patch", "secret", name, "--type=merge", "-p", json.dumps({"data": {"fixture": "c3ludGhldGljLXVwZGF0ZQ=="}})]).returncode == 0
+    except (RuntimeError, subprocess.SubprocessError, OSError):
+        report["error"] = "proof incomplete; raw output suppressed"
+    finally:
+        if created:
+            try:
+                report["checks"]["fixture_removed"] = run(["delete", "secret", name, "--wait=true"]).returncode == 0
+            except (subprocess.SubprocessError, OSError):
+                report["checks"]["fixture_removed"] = False
+    report["passed"] = "error" not in report and len(report["checks"]) == 9 and all(report["checks"].values())
+    print(json.dumps(report, indent=2))
+    return 0 if report["passed"] else 1
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

docs/changes/CUST-WP-0073/reject-secret-last-applied.yaml

@@ -0,0 +1,27 @@
+# Prepared under CUST-WP-0073-T03; not installed.
+# Owner: railiance-platform. Clean existing annotations in an attended session
+# before binding; otherwise subsequent updates to those Secrets are rejected.
+apiVersion: admissionregistration.k8s.io/v1
+kind: ValidatingAdmissionPolicy
+metadata:
+  name: reject-secret-last-applied
+spec:
+  failurePolicy: Fail
+  matchConstraints:
+    resourceRules:
+      - apiGroups: [""]
+        apiVersions: ["v1"]
+        operations: ["CREATE", "UPDATE"]
+        resources: ["secrets"]
+        scope: "*"
+  validations:
+    - expression: '!has(object.metadata.annotations) || !("kubectl.kubernetes.io/last-applied-configuration" in object.metadata.annotations)'
+      message: "Secret last-applied annotations are forbidden; use server-side apply or replace."
+---
+apiVersion: admissionregistration.k8s.io/v1
+kind: ValidatingAdmissionPolicyBinding
+metadata:
+  name: reject-secret-last-applied
+spec:
+  policyName: reject-secret-last-applied
+  validationActions: [Deny]

docs/changes/CUST-WP-0073/secret_annotation_maintenance.py

@@ -0,0 +1,91 @@
+#!/usr/bin/env python3
+"""Bounded CUST-WP-0073-T03 maintenance. Never emit kubectl output/errors.
+
+Run on railiance01 through the supervised admin path. Default is inspection;
+--clean removes only the last-applied annotation, leaving Secret data untouched.
+"""
+import argparse
+import json
+import re
+import subprocess
+from datetime import datetime, timezone
+
+KEY = "kubectl.kubernetes.io/last-applied-configuration"
+PRESENCE = ('{{ $found := false }}{{ range $key, $_ := .metadata.annotations }}'
+            '{{ if eq $key "' + KEY + '" }}{{ $found = true }}{{ end }}{{ end }}'
+            '{{ if $found }}HAS-ANNOTATION{{ else }}clean{{ end }}')
+NAME = re.compile(r"^[a-z0-9][a-z0-9.-]*$")
+
+
+def run(args):
+    # Even a template error can contain the entire Secret. Never forward it.
+    result = subprocess.run(["kubectl", *args], capture_output=True, text=True, timeout=30)
+    if result.returncode:
+        raise RuntimeError("kubectl operation failed; output suppressed")
+    return result.stdout.strip()
+
+
+def inspect(namespace, name):
+    value = run(["-n", namespace, "get", "secret", name, "-o", "go-template=" + PRESENCE])
+    if value not in ("clean", "HAS-ANNOTATION"):
+        raise RuntimeError("unexpected presence result; output suppressed")
+    return value == "HAS-ANNOTATION"
+
+
+def maintain(clean=False):
+    # Custom columns use fixed universally-present identity fields; no annotation
+    # or data output. Validate before using any returned text as an argument.
+    identities = run(["get", "secrets", "-A", "--no-headers", "-o",
+                      "custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name"])
+    rows = []
+    for line in identities.splitlines():
+        pair = line.split()
+        if len(pair) != 2 or not all(NAME.fullmatch(value) for value in pair):
+            raise RuntimeError("invalid Secret identity output; suppressed")
+        rows.append(pair)
+    namespaces = run(["get", "namespaces", "-o", "name"]).splitlines()
+    if not all(value.startswith("namespace/") and NAME.fullmatch(value.split("/", 1)[1]) for value in namespaces):
+        raise RuntimeError("invalid namespace inventory; suppressed")
+    active_namespaces = {value.split("/", 1)[1] for value in namespaces}
+    report = {"captured_at": datetime.now(timezone.utc).isoformat(),
+              "mode": "clean" if clean else "inspect", "checked": 0,
+              "annotated": [], "cleaned": [], "orphaned_namespace": [], "complete": False}
+    try:
+        for namespace, name in rows:
+            if namespace not in active_namespaces:
+                report["orphaned_namespace"].append(namespace + "/" + name)
+                continue
+            report["checked"] += 1
+            if not inspect(namespace, name):
+                continue
+            identity = namespace + "/" + name
+            report["annotated"].append(identity)
+            if clean:
+                # A single JSON patch operation cannot modify credential data.
+                patch = [{"op": "remove", "path": "/metadata/annotations/" + KEY.replace("/", "~1")}]
+                run(["-n", namespace, "patch", "secret", name, "--type=json",
+                     "-p", json.dumps(patch)])
+                if inspect(namespace, name):
+                    raise RuntimeError("annotation still present")
+                report["cleaned"].append(identity)
+        report["active_namespace_scan_complete"] = True
+        report["complete"] = not report["orphaned_namespace"]
+    except (RuntimeError, subprocess.SubprocessError, OSError):
+        report["error"] = "maintenance incomplete; raw output suppressed; inspect before retry"
+    return report
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--clean", action="store_true")
+    args = parser.parse_args()
+    try:
+        report = maintain(args.clean)
+    except (RuntimeError, subprocess.SubprocessError, OSError):
+        report = {"complete": False, "error": "inventory failed; raw output suppressed"}
+    print(json.dumps(report, indent=2))
+    return 0 if report["complete"] else 1
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())