the-custodian/workplans/CUST-WP-0073-agent-credential-separation.md

id	type	title	domain	repo	status	owner	topic_slug	flavor	created	updated	related	origin	origin_ref	state_hub_workstream_id
CUST-WP-0073	workplan	Separate agent credentials and establish supervised privileged execution	infotech	the-custodian	active	the-custodian	custodian	implementation	2026-09-24	2026-09-28	KEY-WP-0033 RPF-WP-0044	incident	key-cape/docs/operations.md#before-any-live-change	a98a9f34-83b4-5c8c-8107-e05f7806d50d

Separate agent credentials and establish supervised privileged execution

Why

On 2026-09-23 a key-cape agent session meant to read only Secret metadata. The go-template failed on an absent field, and kubectl printed sso/keycape-config in full as debugging output: the signing key, the LLDAP bind password and the Authelia client secret. Three more Secrets in sso/mfa were probably printed as well.

The root cause is the credentials, not the command:

• Workstation kubectl and ssh railiance01 both authenticate as system:admin in system:masters, which bypasses authorization. No RBAC rule can restrict it.
• k3s runs with --write-kubeconfig-mode=644, so /etc/rancher/k3s/k3s.yaml is world-readable, and anything running as tegwick is cluster-admin.
• The paths that leak a value are open-ended: template error dumps, -o yaml/describe, helm get, kubectl logs, exec … env, ConfigMaps holding credentials, last-applied annotations, and the kubeconfig file itself. A command denylist chases them one by one, and it only binds the harness that enforces it.

Goal: keep privileged credentials outside the agent's direct reach, and mediate privileged actions according to the identified agent's autonomy mode. Agents start supervised; proven agents may later receive scoped autopilot permission with a cost budget and risk limit in EUR. Neither mode implies unrestricted admin credentials. Founder decision, September 28: docs/agent-autonomy-decision.md.

Scope: builder mode, founder decision 2026-09-24. Rotating the exposed Secrets is deferred, not dropped (T05). This plan removes the problem class.

Decide the identity model

id: CUST-WP-0073-T01
status: done
priority: high
state_hub_task_id: "4781bb99-020f-59f6-be13-e3b8777d3fd8"

Done 2026-09-28 — policy decision. The founder chooses autonomy as a characteristic of the agent: supervised-mode initially, promotion only on successful proposals without adaptation/refinement, and autopilot bounded by cost and risk limits in EUR. Decision: docs/agent-autonomy-decision.md.

The supervised starting identity is outside system:masters and has a reviewed observation allowlist. Exact privileged actions go through supervisor approval or supervisor execution, with unchanged-acceptance and verified unchanged-success recorded separately. Admin credentials remain in the privileged execution path, outside the agent's direct reach. Later autopilot removes per-action approval only for an explicit grant within scope, cost and risk constraints; it does not hand out unrestricted sudo or an admin kubeconfig.

Built-in view plus Deployment/ConfigMap writes cannot establish that boundary: workload writes can extract credentials, and ConfigMaps/pod specs/logs can contain values. Agent-visible results must be sanitized. T02 selects and proves the actual account/profile/execution path. This decision resolves the identity and autonomy policy, not the implementation, numeric promotion thresholds, euro limits or authorization of a live cutover. Existing human-only lanes still apply.

Build and hand out the agent identity

id: CUST-WP-0073-T02
status: progress
priority: high
state_hub_task_id: "4b88b0b7-dc7e-5612-aa5d-1a08f0530c35"

Owner: railiance-platform (RBAC), railiance-enablement (k3s install), railiance-infra (host principal mapping), ops-warden (certificate issuance).

• Bind the agent's stable identity and supervised-mode to its existing instance/assignment record and a restricted runtime profile. Name its supervisor and privileged execution path. No raw admin credentials in the agent process/account; no unrestricted sudo, socket or GitOps bypass.
• Create the ServiceAccount or client certificate, the ClusterRole and the binding in git, applied by the owner's documented path.
• Set write-kubeconfig-mode to 600 in the k3s install config. Attended admin use becomes sudo k3s kubectl.
• Workstation: ~/.kube/config points at the agent kubeconfig. The admin kubeconfig moves out of the agent's reach (another account, or short-lived issuance through an attended login).
• Proof: as the agent identity, kubectl auth can-i get secrets -A and can-i create pods/exec -A both answer no, and kubectl auth whoami shows no system:masters. Record the output as evidence. Also verify that an exact supervisor-approved action can execute through the selected privileged path and return sanitized outcome evidence, while an unapproved/revised action cannot use that approval. Test from the actual agent account, including denial of the old admin SSH/sudo/credential paths.
• Autopilot remains disabled without a scoped promotion decision, concrete EUR cost/risk limits and verified enforcement. Implementing a general promotion or risk-scoring service is not required for this supervised credential boundary.

Reject last-applied annotations on Secrets

id: CUST-WP-0073-T03
status: done
priority: medium
needs_human: false
state_hub_task_id: "5abbfcae-eb65-5b62-a7fa-f4f698f95312"

Owner: railiance-platform.

Done 2026-09-28. Explicit metadata fixes landed in all 31 affected ESO declarations. The guard is active and Synced/Healthy; all nine native admission checks and 39/39 fresh ESO refreshes under Deny pass. All 257 active-namespace Secrets were annotation-free. After explicit founder approval, the one orphan drill Secret was deleted with its UID precondition; absence verified and the 3 GiB PVC unchanged. Full evidence: docs/evidence/2026-09-28-secret-annotation-rollout.md.

• Add a ValidatingAdmissionPolicy (v1.35 is available) with its binding. It rejects any Secret carrying kubectl.kubernetes.io/last-applied-configuration.
• Strip the annotation from existing Secrets first, cluster-wide, using the presence-check template, or the policy blocks their next update. Known: sso/keycape-config, sso/authelia-secrets, sso/lldap-secrets, mfa/privacyidea-config.
• Proof: a client-side kubectl apply of a test Secret in a scratch namespace is refused.

Fleet guidance and harness guards

id: CUST-WP-0073-T04
status: progress
priority: medium
state_hub_task_id: "90725511-4e31-549f-b567-47feff1a9ca4"

• Orientation doc §6: add the template-error dump; state that no template or jsonpath may run against a Secret except the tested presence check; point to the agent identity once T02 lands. Announce it as a standing notice that supersedes the 2026-09-21 one.
• Claude Code guard, done on the workstation 2026-09-24: ~/.claude/hooks/guard-secret-reads.py (PreToolUse on Bash). It denies Secret reads, helm get, config view --raw and kubeconfig reads, and asks before exec/attach/cp/debug and --raw. The kubectl * get secret * allow rule and the expired 2026-07-07 temporary elevation are removed. Known false positive: any command naming both kubectl and "secret", including an echo. That is acceptable for a stopgap.
• Offer the same guard to the Codex and Grok harnesses, or record that they have none. Until T02 lands, those agents are protected by instructions only.
• OpenBao/Vault secret-reading permissions also belong behind the privileged boundary; preapproved command prefixes do not implement supervision.
• Document the agent-specific mode and supervisor. Reference exact proposal and execution receipts; track unchanged acceptance separately from verified unchanged success, including refinements, rejections and interventions. Reuse existing records and receipts per docs/agent-autonomy-decision.md; no new dashboard, supervisor service or automatic promotion machinery.

Rotate what was exposed

id: CUST-WP-0073-T05
status: wait
priority: low
state_hub_task_id: "1b41b532-38a2-5f39-b387-28737934811a"

Deferred by founder decision on 2026-09-24 (builder mode). Rotate sso/keycape-config (signing key, LLDAP bind password, Authelia client secret), sso/authelia-secrets, sso/lldap-secrets and mfa/privacyidea-config together, with replace rather than apply, in one attended window.

Reopen on the first of:

• the first production workload or customer data passing through KeyCape
• any sign the 2026-09-23 transcript left the workstation beyond the model provider
• a planned key rotation

The passage of time alone reopens nothing.

September 28 implementation review

The founder asks for minimal additional tasks/workplans/functionality. Keep execution and all unresolved evidence in T01–T05. No new plan or task was opened.

Reviewable package: docs/changes/CUST-WP-0073/README.md and reject-secret-last-applied.yaml beside it. Live read-only inspection confirms tegwick has unrestricted passwordless sudo, k3s kubeconfig is still 644, and Kubernetes uses system:admin / system:masters. The public host inventory maps agent and admin principals to this same account. A kubeconfig switch or chmod alone is insufficient; do not claim the agent boundary has landed.

T01's initial permanent observation-only proposal is superseded by the founder's agent-specific supervised/autopilot decision in docs/agent-autonomy-decision.md. T01 is done; T02 must verify the actual agent execution environment has no route back through admin SSH/sudo, tokens, sockets or automated deployment. This adds no new broker.

T02 in progress: the existing sand-boxer profile.bwrap-local passed a synthetic supervised-process proof: admin homes, Kubernetes/container socket paths and privileged environment variables absent; only loopback networking; observation readable; proposal writable; wrong consumer identity rejected; workspace destroyed. Receipt: docs/evidence/2026-09-28-supervised-sandbox-proof.json. This was not an interactive agent or a credential migration. Existing GLAS local-profile acceptance and actual admin-path denial remain required in T02.

T03 in progress: policy source railiance-platform@800cbfa, application 54885ac and nine passing native admission checks were followed by ESO refresh failures. ESO v0.16.1 copies source metadata when an ExternalSecret has no target template; 31 declarations need explicit metadata before the strict guard is compatible. The binding was removed and all 39 ExternalSecrets recovered. GitOps now pins policy-only 6016f72 via application commit c5d65b0; the application is Synced and Healthy, and enforcement is disabled. Detailed rollout and recovery receipt: docs/evidence/2026-09-28-secret-annotation-rollout.md.

Cleanup removed the duplicate annotation from 49 distinct active-namespace Secrets across the recorded passes, but ESO can regenerate it while enforcement is off. An orphan platform-pg-drill/drill-minio Secret cannot be patched because its namespace is absent; a referencing Deployment, PVC and Service remain. No orphan was deleted. Neither stable cluster-wide cleanup nor T03 completion is claimed. Keep remediation and integration proof in this existing task.

T04 in progress: orientation §6 withdraws the unsafe raw presence template; only the capturing/sanitizing maintenance helper is allowed. A logical per-agent supervised record lives at .kaizen/agents/custodian-codex/supervision.json. Its summary separates unchanged acceptance from verified unchanged execution, retains failed outcomes and rescue, and grants no authority. The rollout is an unscored historical approval with a failed outcome and recovery, not promotion evidence. There is no eligible acceptance-rate sample yet, no autopilot grant, and no enforced interactive-runtime migration. Codex/Grok have no established equivalent read-denial hook. Final guidance still needs the actual T02 path. T05 remains the original trigger-based founder deferral, not cancelled or done.

Corrected admission rollout — September 28 continuation

T03: all 31 affected ExternalSecrets now have explicit target metadata in their owner sources (23 files, 12 repositories, committed and published). Server dry-run verified only the target template changes; credential data mappings and policies remain unchanged. All 39 ExternalSecrets refreshed successfully before and after re-enabling Deny enforcement. All nine native admission checks pass. The guard is Synced/Healthy at platform source 7daf7e9, pinned by db51ec8. The earlier rollback is historical, not the current live state. Detailed evidence: docs/evidence/2026-09-28-secret-annotation-rollout.md.

A complete scan of all 257 Secrets in existing namespaces found no forbidden annotation after the writer fixes. T03 now waits only for the orphan platform-pg-drill/drill-minio: its namespace is absent, so an annotation patch is refused. UID-bound deletion of that one Secret is prepared and awaits explicit authorization; no PVC deletion or namespace recreation is proposed. All remaining work stays in existing tasks; no new task, workplan, controller or service was introduced. T02 still needs actual supervised-runtime admission; T05 keeps its founder-deferred rotation triggers.

Post-enforcement scan: all 257 active-namespace Secrets remain annotation-free. The exact orphan deletion also passed server-side dry-run; execution awaits the founder response. Receipt: docs/evidence/2026-09-28-secret-annotation-scan-enforced.json.

Final orphan disposition: the founder explicitly selected “Delete only the orphan Secret.” The UID-bound deletion succeeded and absence was verified; the bound 3 GiB PVC retained the same UID, resourceVersion, volume and status. No other resources were changed. T03 is done and its human-needed flag cleared. Receipt: docs/evidence/2026-09-28-orphan-secret-deletion.json.