the-custodian/workplans/CUST-WP-0073-agent-credential-separation.md

id	type	title	domain	repo	status	owner	topic_slug	flavor	created	updated	related	origin	origin_ref	state_hub_workstream_id
CUST-WP-0073	workplan	Agents cannot read secret values: separate agent and admin credentials	infotech	the-custodian	proposed	claude-code	custodian	implementation	2026-09-24	2026-09-24	KEY-WP-0033 RPF-WP-0044	incident	key-cape/docs/operations.md#before-any-live-change	a98a9f34-83b4-5c8c-8107-e05f7806d50d

Agents cannot read secret values: separate agent and admin credentials

Why

On 2026-09-23 a key-cape agent session meant to read only Secret metadata. The go-template failed on an absent field, and kubectl printed sso/keycape-config in full as debugging output: the signing key, the LLDAP bind password and the Authelia client secret. Three more Secrets in sso/mfa were probably printed as well.

The root cause is the credentials, not the command:

• Workstation kubectl and ssh railiance01 both authenticate as system:admin in system:masters, which bypasses authorization. No RBAC rule can restrict it.
• k3s runs with --write-kubeconfig-mode=644, so /etc/rancher/k3s/k3s.yaml is world-readable, and anything running as tegwick is cluster-admin.
• The paths that leak a value are open-ended: template error dumps, -o yaml/describe, helm get, kubectl logs, exec … env, ConfigMaps holding credentials, last-applied annotations, and the kubeconfig file itself. A command denylist chases them one by one, and it only binds the harness that enforces it.

Goal: the identity an agent uses cannot read a secret value by any command. Then a leak needs a human's attended credential, not a mistake.

Scope: builder mode, founder decision 2026-09-24. Rotating the exposed Secrets is deferred, not dropped (T05). This plan removes the problem class.

Decide the identity model

id: CUST-WP-0073-T01
status: todo
priority: high
state_hub_task_id: "4781bb99-020f-59f6-be13-e3b8777d3fd8"

Decide, with railiance-platform and ops-warden:

• Agent identity: a dedicated kube identity outside system:masters, bound to the built-in view role plus the specific write verbs agents need (for example patch and rollout restart on Deployments, ConfigMap updates). No secrets verbs at all, since list and watch return data too. No pods/exec, pods/attach, pods/portforward or nodes/proxy. No helm, which stores its releases in Secrets.
• Admin identity: stays system:admin, reachable only by an attended step, never readable from the agent's Unix account.
• Where the agent credential lives on the workstation and on railiance01. ops-warden already distinguishes adm/agt/atm SSH principals. Mapping agt to a restricted account on railiance01 is the obvious candidate; how warden provisions those principals is still to be verified.
• Paths that stay attended: Secret writes, helm releases and break-glass.

Output: a decision record in the-custodian, resolved by the founder (GOVERN @ estate).

Build and hand out the agent identity

id: CUST-WP-0073-T02
status: todo
priority: high
state_hub_task_id: "4b88b0b7-dc7e-5612-aa5d-1a08f0530c35"

Owner: railiance-platform (RBAC), railiance-enablement (k3s install), ops-warden (principal mapping).

• Create the ServiceAccount or client certificate, the ClusterRole and the binding in git, applied by the owner's documented path.
• Set write-kubeconfig-mode to 600 in the k3s install config. Attended admin use becomes sudo k3s kubectl.
• Workstation: ~/.kube/config points at the agent kubeconfig. The admin kubeconfig moves out of the agent's reach (another account, or short-lived issuance through an attended login).
• Proof: as the agent identity, kubectl auth can-i get secrets -A and can-i create pods/exec -A both answer no, and kubectl auth whoami shows no system:masters. Record the output as evidence.

Reject last-applied annotations on Secrets

id: CUST-WP-0073-T03
status: todo
priority: medium
state_hub_task_id: "5abbfcae-eb65-5b62-a7fa-f4f698f95312"

Owner: railiance-platform.

• Add a ValidatingAdmissionPolicy (v1.35 is available) with its binding. It rejects any Secret carrying kubectl.kubernetes.io/last-applied-configuration.
• Strip the annotation from existing Secrets first, cluster-wide, using the presence-check template, or the policy blocks their next update. Known: sso/keycape-config, sso/authelia-secrets, sso/lldap-secrets, mfa/privacyidea-config.
• Proof: a client-side kubectl apply of a test Secret in a scratch namespace is refused.

Fleet guidance and harness guards

id: CUST-WP-0073-T04
status: todo
priority: medium
state_hub_task_id: "90725511-4e31-549f-b567-47feff1a9ca4"

• Orientation doc §6: add the template-error dump; state that no template or jsonpath may run against a Secret except the tested presence check; point to the agent identity once T02 lands. Announce it as a standing notice that supersedes the 2026-09-21 one.
• Claude Code guard, done on the workstation 2026-09-24: ~/.claude/hooks/guard-secret-reads.py (PreToolUse on Bash). It denies Secret reads, helm get, config view --raw and kubeconfig reads, and asks before exec/attach/cp/debug and --raw. The kubectl * get secret * allow rule and the expired 2026-07-07 temporary elevation are removed. Known false positive: any command naming both kubectl and "secret", including an echo. That is acceptable for a stopgap.
• Offer the same guard to the Codex and Grok harnesses, or record that they have none. Until T02 lands, those agents are protected by instructions only.
• Open question for the founder: Bash(bao read *), vault kv get and vault read are still pre-approved and print secret values the same way.

Rotate what was exposed

id: CUST-WP-0073-T05
status: wait
priority: low
state_hub_task_id: "1b41b532-38a2-5f39-b387-28737934811a"

Deferred by founder decision on 2026-09-24 (builder mode). Rotate sso/keycape-config (signing key, LLDAP bind password, Authelia client secret), sso/authelia-secrets, sso/lldap-secrets and mfa/privacyidea-config together, with replace rather than apply, in one attended window.

Reopen on the first of:

• the first production workload or customer data passing through KeyCape
• any sign the 2026-09-23 transcript left the workstation beyond the model provider
• a planned key rotation

The passage of time alone reopens nothing.