the-custodian/workplans/CUST-WP-0073-agent-credential-separation.md

id	type	title	domain	repo	status	owner	topic_slug	flavor	created	updated	related	origin	origin_ref	state_hub_workstream_id
CUST-WP-0073	workplan	Separate agent credentials and establish supervised privileged execution	infotech	the-custodian	active	the-custodian	custodian	implementation	2026-09-24	2026-09-28	KEY-WP-0033 RPF-WP-0044	incident	key-cape/docs/operations.md#before-any-live-change	a98a9f34-83b4-5c8c-8107-e05f7806d50d

Separate agent credentials and establish supervised privileged execution

Why

On 2026-09-23 a key-cape agent session meant to read only Secret metadata. The go-template failed on an absent field, and kubectl printed sso/keycape-config in full as debugging output: the signing key, the LLDAP bind password and the Authelia client secret. Three more Secrets in sso/mfa were probably printed as well.

The root cause is the credentials, not the command:

• Workstation kubectl and ssh railiance01 both authenticate as system:admin in system:masters, which bypasses authorization. No RBAC rule can restrict it.
• k3s runs with --write-kubeconfig-mode=644, so /etc/rancher/k3s/k3s.yaml is world-readable, and anything running as tegwick is cluster-admin.
• The paths that leak a value are open-ended: template error dumps, -o yaml/describe, helm get, kubectl logs, exec … env, ConfigMaps holding credentials, last-applied annotations, and the kubeconfig file itself. A command denylist chases them one by one, and it only binds the harness that enforces it.

Goal: keep privileged credentials outside the agent's direct reach, and mediate privileged actions according to the identified agent's autonomy mode. Agents start supervised; proven agents may later receive scoped autopilot permission with a cost budget and risk limit in EUR. Neither mode implies unrestricted admin credentials. Founder decision, September 28: docs/agent-autonomy-decision.md.

Scope: builder mode, founder decision 2026-09-24. Rotating the exposed Secrets is deferred, not dropped (T05). This plan removes the problem class.

Decide the identity model

id: CUST-WP-0073-T01
status: done
priority: high
state_hub_task_id: "4781bb99-020f-59f6-be13-e3b8777d3fd8"

Done 2026-09-28 — policy decision. The founder chooses autonomy as a characteristic of the agent: supervised-mode initially, promotion only on successful proposals without adaptation/refinement, and autopilot bounded by cost and risk limits in EUR. Decision: docs/agent-autonomy-decision.md.

The supervised starting identity is outside system:masters and has a reviewed observation allowlist. Exact privileged actions go through supervisor approval or supervisor execution, with unchanged-acceptance and verified unchanged-success recorded separately. Admin credentials remain in the privileged execution path, outside the agent's direct reach. Later autopilot removes per-action approval only for an explicit grant within scope, cost and risk constraints; it does not hand out unrestricted sudo or an admin kubeconfig.

Built-in view plus Deployment/ConfigMap writes cannot establish that boundary: workload writes can extract credentials, and ConfigMaps/pod specs/logs can contain values. Agent-visible results must be sanitized. T02 selects and proves the actual account/profile/execution path. This decision resolves the identity and autonomy policy, not the implementation, numeric promotion thresholds, euro limits or authorization of a live cutover. Existing human-only lanes still apply.

Build and hand out the agent identity

id: CUST-WP-0073-T02
status: progress
priority: high
state_hub_task_id: "4b88b0b7-dc7e-5612-aa5d-1a08f0530c35"

Owner: railiance-platform (RBAC), railiance-enablement (k3s install), railiance-infra (host principal mapping), ops-warden (certificate issuance).

• Bind the agent's stable identity and supervised-mode to its existing instance/assignment record and a restricted runtime profile. Name its supervisor and privileged execution path. No raw admin credentials in the agent process/account; no unrestricted sudo, socket or GitOps bypass.
• Create the ServiceAccount or client certificate, the ClusterRole and the binding in git, applied by the owner's documented path.
• Set write-kubeconfig-mode to 600 in the k3s install config. Attended admin use becomes sudo k3s kubectl.
• Workstation: ~/.kube/config points at the agent kubeconfig. The admin kubeconfig moves out of the agent's reach (another account, or short-lived issuance through an attended login).
• Proof: as the agent identity, kubectl auth can-i get secrets -A and can-i create pods/exec -A both answer no, and kubectl auth whoami shows no system:masters. Record the output as evidence. Also verify that an exact supervisor-approved action can execute through the selected privileged path and return sanitized outcome evidence, while an unapproved/revised action cannot use that approval. Test from the actual agent account, including denial of the old admin SSH/sudo/credential paths.
• Autopilot remains disabled without a scoped promotion decision, concrete EUR cost/risk limits and verified enforcement. Implementing a general promotion or risk-scoring service is not required for this supervised credential boundary.

Reject last-applied annotations on Secrets

id: CUST-WP-0073-T03
status: done
priority: medium
needs_human: false
state_hub_task_id: "5abbfcae-eb65-5b62-a7fa-f4f698f95312"

Owner: railiance-platform.

Done 2026-09-28. Explicit metadata fixes landed in all 31 affected ESO declarations. The guard is active and Synced/Healthy; all nine native admission checks and 39/39 fresh ESO refreshes under Deny pass. All 257 active-namespace Secrets were annotation-free. After explicit founder approval, the one orphan drill Secret was deleted with its UID precondition; absence verified and the 3 GiB PVC unchanged. Full evidence: docs/evidence/2026-09-28-secret-annotation-rollout.md.

• Add a ValidatingAdmissionPolicy (v1.35 is available) with its binding. It rejects any Secret carrying kubectl.kubernetes.io/last-applied-configuration.
• Strip the annotation from existing Secrets first, cluster-wide, using the output-suppressing maintenance helper, or the policy blocks their next update. Known: sso/keycape-config, sso/authelia-secrets, sso/lldap-secrets, mfa/privacyidea-config.
• Proof: a client-side kubectl apply of a test Secret in a scratch namespace is refused.

Fleet guidance and harness guards

id: CUST-WP-0073-T04
status: progress
priority: medium
state_hub_task_id: "90725511-4e31-549f-b567-47feff1a9ca4"

• Orientation doc §6 documents the template-error dump and requires the safe maintenance helper; standalone Secret templates/jsonpath are forbidden. The September 28 standing notice supersedes the September 24 inline-check exception. Add the verified agent identity/execution path once T02 lands.
• Claude Code guard, done on the workstation 2026-09-24: ~/.claude/hooks/guard-secret-reads.py (PreToolUse on Bash). It denies Secret reads, helm get, config view --raw and kubeconfig reads, and asks before exec/attach/cp/debug and --raw. The kubectl * get secret * allow rule and the expired 2026-07-07 temporary elevation are removed. Known false positive: any command naming both kubectl and "secret", including an echo. That is acceptable for a stopgap.
• Offer the same guard to the Codex and Grok harnesses, or record that they have none. Until T02 lands, those agents are protected by instructions only.
• OpenBao/Vault secret-reading permissions also belong behind the privileged boundary; preapproved command prefixes do not implement supervision.
• Document the agent-specific mode and supervisor. Reference exact proposal and execution receipts; track unchanged acceptance separately from verified unchanged success, including refinements, rejections and interventions. Reuse existing records and receipts per docs/agent-autonomy-decision.md; no new dashboard, supervisor service or automatic promotion machinery.

Rotate what was exposed

id: CUST-WP-0073-T05
status: wait
priority: low
state_hub_task_id: "1b41b532-38a2-5f39-b387-28737934811a"

Deferred by founder decision on 2026-09-24 (builder mode). Rotate sso/keycape-config (signing key, LLDAP bind password, Authelia client secret), sso/authelia-secrets, sso/lldap-secrets and mfa/privacyidea-config together, with replace rather than apply, in one attended window.

Reopen on the first of:

• the first production workload or customer data passing through KeyCape
• any sign the 2026-09-23 transcript left the workstation beyond the model provider
• a planned key rotation

The passage of time alone reopens nothing.

Current evidence and handoff — September 28

The founder requests minimal additional tasks, workplans and functionality. All remaining work stays in T02, T04 and the original deferred T05. No new workplan, task, controller or supervisor service was introduced.

T01 is done: autonomy is per agent and scope, supervised initially, with later promotion bounded by EUR cost and risk limits. The decision and descriptive per-agent record do not establish runtime enforcement or grant autopilot.

T02 remains in progress. The existing sand-boxer profile.bwrap-local passed a synthetic process-isolation proof: admin homes, privileged environment variables, Kubernetes/container sockets absent; only loopback networking; observation and proposal paths work; wrong consumer rejected; workspace destroyed. Receipt: docs/evidence/2026-09-28-supervised-sandbox-proof.json. It was not an interactive agent migration. The selected GLAS local-profile acceptance remains blocked; existing coordination receipt: docs/evidence/2026-09-28-supervised-runtime-coordination.json. Actual account/profile admission and denial of old admin SSH/sudo/credential paths are still required. The inspected legacy account has unrestricted sudo and k3s kubeconfig mode 644; changing its default kubeconfig alone is insufficient.

T03 is done. All 31 affected ESO declarations have explicit target metadata in 23 files across 12 owning repositories, committed and published. The guard is active and Synced/Healthy at platform source 7daf7e9, pinned by db51ec8. Nine native admission checks and 39/39 fresh ESO refreshes under Deny passed. The founder-approved UID-bound deletion removed only the orphan drill Secret; the 3 GiB PVC was unchanged. The final complete scan checked 257 Secrets with zero forbidden annotations and zero orphan exceptions. The earlier failed rollout and rollback remain historical evidence in docs/evidence/2026-09-28-secret-annotation-rollout.md; they are not the live state.

T04's local guidance and reporting are implemented. AGENTS.md now loads the supervision decision and per-agent record at startup and describes recording original proposals before approval and verified outcomes afterwards. The September 28 standing notice withdraws the unsafe inline presence check: 51e9eace-06d2-46d6-921a-4b92440df68f. Receipt: docs/evidence/2026-09-28-secret-guidance-notice.json. Codex/Grok have no equivalent read-denial hook established by this work. The remaining T04 step is to document the actual verified T02 execution path. The original rollout remains an unscored failed proposal with refinement and recovery; successful remediation does not become unchanged-success credit. There is no eligible acceptance-rate sample or autopilot grant yet.

T05 remains the founder's trigger-based rotation deferral, not cancelled or done. Neither workplan completion nor live credential separation is claimed.