the-custodian/workplans/CUST-WP-0073-agent-credential-separation.md

---
id: CUST-WP-0073
type: workplan
title: "Separate agent credentials and establish supervised privileged execution"
domain: infotech
repo: the-custodian
status: active
owner: the-custodian
topic_slug: custodian
flavor: implementation
created: "2026-09-24"
updated: "2026-09-28"
related:
  - KEY-WP-0033
  - RPF-WP-0044
origin: incident
origin_ref: key-cape/docs/operations.md#before-any-live-change
state_hub_workstream_id: "a98a9f34-83b4-5c8c-8107-e05f7806d50d"
---

# Separate agent credentials and establish supervised privileged execution

## Why

On 2026-09-23 a key-cape agent session meant to read only Secret metadata.
The go-template failed on an absent field, and kubectl printed
`sso/keycape-config` in full as debugging output: the signing key, the LLDAP
bind password and the Authelia client secret. Three more Secrets in `sso`/`mfa`
were probably printed as well.

The root cause is the credentials, not the command:

- Workstation `kubectl` and `ssh railiance01` both authenticate as
  `system:admin` in `system:masters`, which bypasses authorization. No RBAC
  rule can restrict it.
- k3s runs with `--write-kubeconfig-mode=644`, so `/etc/rancher/k3s/k3s.yaml`
  is world-readable, and anything running as `tegwick` is cluster-admin.
- The paths that leak a value are open-ended: template error dumps,
  `-o yaml`/`describe`, `helm get`, `kubectl logs`, `exec … env`, ConfigMaps
  holding credentials, last-applied annotations, and the kubeconfig file itself.
  A command denylist chases them one by one, and it only binds the harness that
  enforces it.

**Goal:** keep privileged credentials outside the agent's direct reach, and
mediate privileged actions according to the identified agent's autonomy mode.
Agents start supervised; proven agents may later receive scoped autopilot
permission with a cost budget and risk limit in EUR. Neither mode implies
unrestricted admin credentials. Founder decision, September 28:
`docs/agent-autonomy-decision.md`.

**Scope:** builder mode, founder decision 2026-09-24. Rotating the exposed
Secrets is deferred, not dropped (T05). This plan removes the problem class.

## Decide the identity model

```task
id: CUST-WP-0073-T01
status: done
priority: high
state_hub_task_id: "4781bb99-020f-59f6-be13-e3b8777d3fd8"
```

**Done 2026-09-28 — policy decision.** The founder chooses autonomy as a
characteristic of the agent: supervised-mode initially, promotion only on
successful proposals without adaptation/refinement, and autopilot bounded by
cost and risk limits in EUR. Decision: `docs/agent-autonomy-decision.md`.

The supervised starting identity is outside `system:masters` and has a reviewed
observation allowlist. Exact privileged actions go through supervisor approval
or supervisor execution, with unchanged-acceptance and verified unchanged-success
recorded separately. Admin credentials remain in the privileged execution path,
outside the agent's direct reach. Later autopilot removes per-action approval
only for an explicit grant within scope, cost and risk constraints; it does not
hand out unrestricted sudo or an admin kubeconfig.

Built-in `view` plus Deployment/ConfigMap writes cannot establish that boundary:
workload writes can extract credentials, and ConfigMaps/pod specs/logs can contain
values. Agent-visible results must be sanitized. T02 selects and proves the
actual account/profile/execution path. This decision resolves the identity and
autonomy policy, not the implementation, numeric promotion thresholds, euro
limits or authorization of a live cutover. Existing human-only lanes still apply.

## Build and hand out the agent identity

```task
id: CUST-WP-0073-T02
status: progress
priority: high
state_hub_task_id: "4b88b0b7-dc7e-5612-aa5d-1a08f0530c35"
```

Owner: railiance-platform (RBAC), railiance-enablement (k3s install),
railiance-infra (host principal mapping), ops-warden (certificate issuance).

- Bind the agent's stable identity and `supervised-mode` to its existing
  instance/assignment record and a restricted runtime profile. Name its
  supervisor and privileged execution path. No raw admin credentials in the
  agent process/account; no unrestricted sudo, socket or GitOps bypass.
- Create the ServiceAccount or client certificate, the ClusterRole and the
  binding in git, applied by the owner's documented path.
- Set `write-kubeconfig-mode` to `600` in the k3s install config. Attended admin
  use becomes `sudo k3s kubectl`.
- Workstation: `~/.kube/config` points at the agent kubeconfig. The admin
  kubeconfig moves out of the agent's reach (another account, or short-lived
  issuance through an attended login).
- Proof: as the agent identity, `kubectl auth can-i get secrets -A` and
  `can-i create pods/exec -A` both answer `no`, and `kubectl auth whoami`
  shows no `system:masters`. Record the output as evidence. Also verify that
  an exact supervisor-approved action can execute through the selected privileged
  path and return sanitized outcome evidence, while an unapproved/revised action
  cannot use that approval. Test from the actual agent account, including denial
  of the old admin SSH/sudo/credential paths.
- Autopilot remains disabled without a scoped promotion decision, concrete EUR
  cost/risk limits and verified enforcement. Implementing a general promotion or
  risk-scoring service is not required for this supervised credential boundary.

## Reject last-applied annotations on Secrets

```task
id: CUST-WP-0073-T03
status: done
priority: medium
needs_human: false
state_hub_task_id: "5abbfcae-eb65-5b62-a7fa-f4f698f95312"
```

Owner: railiance-platform.

**Done 2026-09-28.** Explicit metadata fixes landed in all 31 affected ESO
declarations. The guard is active and Synced/Healthy; all nine native admission
checks and 39/39 fresh ESO refreshes under Deny pass. All 257 active-namespace
Secrets were annotation-free. After explicit founder approval, the one orphan
drill Secret was deleted with its UID precondition; absence verified and the
3 GiB PVC unchanged. Full evidence: `docs/evidence/2026-09-28-secret-annotation-rollout.md`.

- Add a `ValidatingAdmissionPolicy` (v1.35 is available) with its binding. It
  rejects any Secret carrying `kubectl.kubernetes.io/last-applied-configuration`.
- Strip the annotation from existing Secrets first, cluster-wide, using the
  presence-check template, or the policy blocks their next update. Known:
  `sso/keycape-config`, `sso/authelia-secrets`, `sso/lldap-secrets`,
  `mfa/privacyidea-config`.
- Proof: a client-side `kubectl apply` of a test Secret in a scratch namespace
  is refused.

## Fleet guidance and harness guards

```task
id: CUST-WP-0073-T04
status: progress
priority: medium
state_hub_task_id: "90725511-4e31-549f-b567-47feff1a9ca4"
```

- Orientation doc §6: add the template-error dump; state that no template or
  jsonpath may run against a Secret except the tested presence check; point to
  the agent identity once T02 lands. Announce it as a standing notice that
  supersedes the 2026-09-21 one.
- Claude Code guard, **done on the workstation 2026-09-24**:
  `~/.claude/hooks/guard-secret-reads.py` (PreToolUse on Bash). It denies Secret
  reads, `helm get`, `config view --raw` and kubeconfig reads, and asks before
  exec/attach/cp/debug and `--raw`. The `kubectl * get secret *` allow rule and
  the expired 2026-07-07 temporary elevation are removed. Known false positive:
  any command naming both kubectl and "secret", including an `echo`. That is
  acceptable for a stopgap.
- Offer the same guard to the Codex and Grok harnesses, or record that they have
  none. Until T02 lands, those agents are protected by instructions only.
- OpenBao/Vault secret-reading permissions also belong behind the privileged
  boundary; preapproved command prefixes do not implement supervision.
- Document the agent-specific mode and supervisor. Reference exact proposal and
  execution receipts; track unchanged acceptance separately from verified
  unchanged success, including refinements, rejections and interventions. Reuse
  existing records and receipts per `docs/agent-autonomy-decision.md`; no new
  dashboard, supervisor service or automatic promotion machinery.

## Rotate what was exposed

```task
id: CUST-WP-0073-T05
status: wait
priority: low
state_hub_task_id: "1b41b532-38a2-5f39-b387-28737934811a"
```

Deferred by founder decision on 2026-09-24 (builder mode). Rotate
`sso/keycape-config` (signing key, LLDAP bind password, Authelia client secret),
`sso/authelia-secrets`, `sso/lldap-secrets` and `mfa/privacyidea-config`
together, with `replace` rather than `apply`, in one attended window.

Reopen on the first of:

- the first production workload or customer data passing through KeyCape
- any sign the 2026-09-23 transcript left the workstation beyond the model
  provider
- a planned key rotation

The passage of time alone reopens nothing.

## September 28 implementation review

The founder asks for minimal additional tasks/workplans/functionality. Keep
execution and all unresolved evidence in T01–T05. No new plan or task was opened.

Reviewable package: `docs/changes/CUST-WP-0073/README.md` and
`reject-secret-last-applied.yaml` beside it. Live read-only inspection confirms
`tegwick` has unrestricted passwordless sudo, k3s kubeconfig is still 644, and
Kubernetes uses `system:admin` / `system:masters`. The public host inventory maps
agent and admin principals to this same account. A kubeconfig switch or chmod
alone is insufficient; do not claim the agent boundary has landed.

T01's initial permanent observation-only proposal is superseded by the founder's
agent-specific supervised/autopilot decision in `docs/agent-autonomy-decision.md`.
T01 is done; T02 must
verify the actual agent execution environment has no route back through admin
SSH/sudo, tokens, sockets or automated deployment. This adds no new broker.

T02 in progress: the existing sand-boxer `profile.bwrap-local` passed a
synthetic supervised-process proof: admin homes, Kubernetes/container socket
paths and privileged environment variables absent; only loopback networking;
observation readable; proposal writable; wrong consumer identity rejected;
workspace destroyed. Receipt: `docs/evidence/2026-09-28-supervised-sandbox-proof.json`.
This was not an interactive agent or a credential migration. Existing GLAS
local-profile acceptance and actual admin-path denial remain required in T02.

T03 in progress: policy source `railiance-platform@800cbfa`, application `54885ac`
and nine passing native admission checks were followed by ESO refresh failures.
ESO v0.16.1 copies source metadata when an ExternalSecret has no target template;
31 declarations need explicit metadata before the strict guard is compatible.
The binding was removed and all 39 ExternalSecrets recovered. GitOps now pins
policy-only `6016f72` via application commit `c5d65b0`; the application is Synced
and Healthy, and enforcement is disabled. Detailed rollout and recovery receipt:
`docs/evidence/2026-09-28-secret-annotation-rollout.md`.

Cleanup removed the duplicate annotation from 49 distinct active-namespace
Secrets across the recorded passes, but ESO can regenerate it while enforcement
is off. An orphan `platform-pg-drill/drill-minio` Secret cannot be patched because
its namespace is absent; a referencing Deployment, PVC and Service remain. No
orphan was deleted. Neither stable cluster-wide cleanup nor T03 completion is
claimed. Keep remediation and integration proof in this existing task.

T04 in progress: orientation §6 withdraws the unsafe raw presence template;
only the capturing/sanitizing maintenance helper is allowed. A logical
per-agent supervised record lives at `.kaizen/agents/custodian-codex/supervision.json`.
Its summary separates unchanged acceptance from verified unchanged execution,
retains failed outcomes and rescue, and grants no authority. The rollout is an
unscored historical approval with a failed outcome and recovery, not promotion
evidence. There is no eligible acceptance-rate sample yet, no autopilot grant,
and no enforced interactive-runtime migration. Codex/Grok have no established
equivalent read-denial hook. Final guidance still needs the actual T02 path.
T05 remains the original trigger-based founder deferral, not cancelled or done.

## Corrected admission rollout — September 28 continuation

T03: all 31 affected ExternalSecrets now have explicit target metadata in their
owner sources (23 files, 12 repositories, committed and published). Server
dry-run verified only the target template changes; credential data mappings and
policies remain unchanged. All 39 ExternalSecrets refreshed successfully before
and after re-enabling Deny enforcement. All nine native admission checks pass.
The guard is Synced/Healthy at platform source `7daf7e9`, pinned by `db51ec8`.
The earlier rollback is historical, not the current live state. Detailed evidence:
`docs/evidence/2026-09-28-secret-annotation-rollout.md`.

A complete scan of all 257 Secrets in existing namespaces found no forbidden
annotation after the writer fixes. T03 now waits only for the orphan
`platform-pg-drill/drill-minio`: its namespace is absent, so an annotation patch
is refused. UID-bound deletion of that one Secret is prepared and awaits
explicit authorization; no PVC deletion or namespace recreation is proposed.
All remaining work stays in existing tasks; no new task, workplan, controller
or service was introduced. T02 still needs actual supervised-runtime admission;
T05 keeps its founder-deferred rotation triggers.

Post-enforcement scan: all 257 active-namespace Secrets remain annotation-free.
The exact orphan deletion also passed server-side dry-run; execution awaits
the founder response. Receipt: `docs/evidence/2026-09-28-secret-annotation-scan-enforced.json`.

Final orphan disposition: the founder explicitly selected “Delete only the
orphan Secret.” The UID-bound deletion succeeded and absence was verified;
the bound 3 GiB PVC retained the same UID, resourceVersion, volume and status.
No other resources were changed. T03 is done and its human-needed flag cleared.
Receipt: `docs/evidence/2026-09-28-orphan-secret-deletion.json`.