the-custodian/workplans/CUST-WP-0073-agent-credential-separation.md

id	type	title	domain	repo	status	owner	topic_slug	flavor	created	updated	related	origin	origin_ref	state_hub_workstream_id
CUST-WP-0073	workplan	Separate agent credentials and establish supervised privileged execution	infotech	the-custodian	active	the-custodian	custodian	implementation	2026-09-24	2026-10-19	KEY-WP-0033 RPF-WP-0044	incident	key-cape/docs/operations.md#before-any-live-change	a98a9f34-83b4-5c8c-8107-e05f7806d50d

Separate agent credentials and establish supervised privileged execution

Why

On 2026-09-23 a key-cape agent session meant to read only Secret metadata. The go-template failed on an absent field, and kubectl printed sso/keycape-config in full as debugging output: the signing key, the LLDAP bind password and the Authelia client secret. Three more Secrets in sso/mfa were probably printed as well.

The root cause is the credentials, not the command:

• Workstation kubectl and ssh railiance01 both authenticate as system:admin in system:masters, which bypasses authorization. No RBAC rule can restrict it.
• k3s runs with --write-kubeconfig-mode=644, so /etc/rancher/k3s/k3s.yaml is world-readable, and anything running as tegwick is cluster-admin.
• The paths that leak a value are open-ended: template error dumps, -o yaml/describe, helm get, kubectl logs, exec … env, ConfigMaps holding credentials, last-applied annotations, and the kubeconfig file itself. A command denylist chases them one by one, and it only binds the harness that enforces it.

Goal: keep privileged credentials outside the agent's direct reach, and mediate privileged actions according to the identified agent's autonomy mode. Agents start supervised; proven agents may later receive scoped autopilot permission with a cost budget and risk limit in EUR. Neither mode implies unrestricted admin credentials. Founder decision, September 28: docs/agent-autonomy-decision.md.

Scope: builder mode, founder decision 2026-09-24. Rotating the exposed Secrets is deferred, not dropped (T05). This plan removes the problem class.

Decide the identity model

id: CUST-WP-0073-T01
status: done
priority: high
state_hub_task_id: "4781bb99-020f-59f6-be13-e3b8777d3fd8"

Done 2026-09-28 — policy decision. The founder chooses autonomy as a characteristic of the agent: supervised-mode initially, promotion only on successful proposals without adaptation/refinement, and autopilot bounded by cost and risk limits in EUR. Decision: docs/agent-autonomy-decision.md.

The supervised starting identity is outside system:masters and has a reviewed observation allowlist. Exact privileged actions go through supervisor approval or supervisor execution, with unchanged-acceptance and verified unchanged-success recorded separately. Admin credentials remain in the privileged execution path, outside the agent's direct reach. Later autopilot removes per-action approval only for an explicit grant within scope, cost and risk constraints; it does not hand out unrestricted sudo or an admin kubeconfig.

Built-in view plus Deployment/ConfigMap writes cannot establish that boundary: workload writes can extract credentials, and ConfigMaps/pod specs/logs can contain values. Agent-visible results must be sanitized. T02 selects and proves the actual account/profile/execution path. This decision resolves the identity and autonomy policy, not the implementation, numeric promotion thresholds, euro limits or authorization of a live cutover. Existing human-only lanes still apply.

Build and hand out the agent identity

id: CUST-WP-0073-T02
status: progress
priority: high
depends_on: []
blocking_reason: "Interactive Codex assignment is the first target under the 2026-10-19 interview decision. Preparing a supported initial launch/tool-transport boundary and exact owner preflight; no actual account/profile admission yet. Native Claude-worker GLAS-WP-0012 proof is a distinct path, not a substitute."
state_hub_task_id: "4b88b0b7-dc7e-5612-aa5d-1a08f0530c35"

Owner: railiance-platform (RBAC), railiance-cluster (k3s install; ownership corrected 2026-10-19), railiance-infra (host principal mapping), ops-warden (certificate issuance).

• Bind the agent's stable identity and supervised-mode to its existing instance/assignment record and a restricted runtime profile. Name its supervisor and privileged execution path. No raw admin credentials in the agent process/account; no unrestricted sudo, socket or GitOps bypass.
• Create the ServiceAccount or client certificate, the ClusterRole and the binding in git, applied by the owner's documented path.
• Set write-kubeconfig-mode to 600 in the k3s install config. Attended admin use becomes sudo k3s kubectl.
• Workstation: ~/.kube/config points at the agent kubeconfig. The admin kubeconfig moves out of the agent's reach (another account, or short-lived issuance through an attended login).
• Proof: as the agent identity, kubectl auth can-i get secrets -A and can-i create pods/exec -A both answer no, and kubectl auth whoami shows no system:masters. Retain these aggregate checks and additionally test every namespaced forbidden permission in each namespace from a supervisor-verified complete current inventory; aggregate denial does not exclude a namespaced grant. Test cluster-scoped privileges separately. Record sanitized output as evidence. Also verify that an exact supervisor-approved action can execute through the selected privileged path and return sanitized outcome evidence, while an unapproved/revised action cannot use that approval. Test from the actual agent account, including denial of the old admin SSH/sudo/credential paths.
• Autopilot remains disabled without a scoped promotion decision, concrete EUR cost/risk limits and verified enforcement. Implementing a general promotion or risk-scoring service is not required for this supervised credential boundary.

Reject last-applied annotations on Secrets

id: CUST-WP-0073-T03
status: done
priority: medium
needs_human: false
state_hub_task_id: "5abbfcae-eb65-5b62-a7fa-f4f698f95312"

Owner: railiance-platform.

Done 2026-09-28. Explicit metadata fixes landed in all 31 affected ESO declarations. The guard is active and Synced/Healthy; all nine native admission checks and 39/39 fresh ESO refreshes under Deny pass. All 257 active-namespace Secrets were annotation-free. After explicit founder approval, the one orphan drill Secret was deleted with its UID precondition; absence verified and the 3 GiB PVC unchanged. Full evidence: docs/evidence/2026-09-28-secret-annotation-rollout.md.

• Add a ValidatingAdmissionPolicy (v1.35 is available) with its binding. It rejects any Secret carrying kubectl.kubernetes.io/last-applied-configuration.
• Strip the annotation from existing Secrets first, cluster-wide, using the output-suppressing maintenance helper, or the policy blocks their next update. Known: sso/keycape-config, sso/authelia-secrets, sso/lldap-secrets, mfa/privacyidea-config.
• Proof: a client-side kubectl apply of a test Secret in a scratch namespace is refused.

Fleet guidance and harness guards

id: CUST-WP-0073-T04
status: wait
priority: medium
depends_on: [CUST-WP-0073-T02]
blocking_reason: "Await CUST-WP-0073-T02: the verified agent identity and execution path must exist before it can be documented; Codex/Grok read-denial guard needs glas-harness or harness-owner delivery."
state_hub_task_id: "90725511-4e31-549f-b567-47feff1a9ca4"

• Orientation doc §6 documents the template-error dump and requires the safe maintenance helper; standalone Secret templates/jsonpath are forbidden. The September 28 standing notice supersedes the September 24 inline-check exception. Add the verified agent identity/execution path once T02 lands.
• Claude Code guard, done on the workstation 2026-09-24: ~/.claude/hooks/guard-secret-reads.py (PreToolUse on Bash). It denies Secret reads, helm get, config view --raw and kubeconfig reads, and asks before exec/attach/cp/debug and --raw. The kubectl * get secret * allow rule and the expired 2026-07-07 temporary elevation are removed. Known false positive: any command naming both kubectl and "secret", including an echo. That is acceptable for a stopgap.
• Offer the same guard to the Codex and Grok harnesses, or record that they have none. Until T02 lands, those agents are protected by instructions only.
• OpenBao/Vault secret-reading permissions also belong behind the privileged boundary; preapproved command prefixes do not implement supervision.
• Document the agent-specific mode and supervisor. Reference exact proposal and execution receipts; track unchanged acceptance separately from verified unchanged success, including refinements, rejections and interventions. Reuse existing records and receipts per docs/agent-autonomy-decision.md; no new dashboard, supervisor service or automatic promotion machinery.

Rotate what was exposed

id: CUST-WP-0073-T05
status: wait
priority: low
needs_human: true
decision_id: 1591daa1-6e85-4dfd-aadd-5a9fe1ef47fa
blocking_reason: "Deferred by founder decision 2026-09-24; reopened only by the founder on one of the three triggers below, in one attended rotation window."
state_hub_task_id: "1b41b532-38a2-5f39-b387-28737934811a"

Deferred by founder decision on 2026-09-24 (builder mode). Rotate sso/keycape-config (signing key, LLDAP bind password, Authelia client secret), sso/authelia-secrets, sso/lldap-secrets and mfa/privacyidea-config together, with replace rather than apply, in one attended window.

Reopen on the first of:

• the first production workload or customer data passing through KeyCape
• any sign the 2026-09-23 transcript left the workstation beyond the model provider
• a planned key rotation

The passage of time alone reopens nothing.

Current evidence and handoff — September 28

The founder requests minimal additional tasks, workplans and functionality. All remaining work stays in T02, T04 and the original deferred T05. No new workplan, task, controller or supervisor service was introduced.

T01 is done: autonomy is per agent and scope, supervised initially, with later promotion bounded by EUR cost and risk limits. The decision and descriptive per-agent record do not establish runtime enforcement or grant autopilot.

T02 remains in progress. The existing sand-boxer profile.bwrap-local passed a synthetic process-isolation proof: admin homes, privileged environment variables, Kubernetes/container sockets absent; only loopback networking; observation and proposal paths work; wrong consumer rejected; workspace destroyed. Receipt: docs/evidence/2026-09-28-supervised-sandbox-proof.json. It was not an interactive agent migration. The selected GLAS local-profile acceptance remains blocked; existing coordination receipt: docs/evidence/2026-09-28-supervised-runtime-coordination.json. Actual account/profile admission and denial of old admin SSH/sudo/credential paths are still required. The inspected legacy account has unrestricted sudo and k3s kubeconfig mode 644; changing its default kubeconfig alone is insufficient.

T03 is done. All 31 affected ESO declarations have explicit target metadata in 23 files across 12 owning repositories, committed and published. The guard is active and Synced/Healthy at platform source 7daf7e9, pinned by db51ec8. Nine native admission checks and 39/39 fresh ESO refreshes under Deny passed. The founder-approved UID-bound deletion removed only the orphan drill Secret; the 3 GiB PVC was unchanged. The final complete scan checked 257 Secrets with zero forbidden annotations and zero orphan exceptions. The earlier failed rollout and rollback remain historical evidence in docs/evidence/2026-09-28-secret-annotation-rollout.md; they are not the live state.

T04's local guidance and reporting are implemented. AGENTS.md now loads the supervision decision and per-agent record at startup and describes recording original proposals before approval and verified outcomes afterwards. The September 28 standing notice withdraws the unsafe inline presence check: 51e9eace-06d2-46d6-921a-4b92440df68f. Receipt: docs/evidence/2026-09-28-secret-guidance-notice.json. Codex/Grok have no equivalent read-denial hook established by this work. The remaining T04 step is to document the actual verified T02 execution path. The original rollout remains an unscored failed proposal with refinement and recovery; successful remediation does not become unchanged-success credit. There is no eligible acceptance-rate sample or autopilot grant yet.

T05 remains the founder's trigger-based rotation deferral, not cancelled or done. Neither workplan completion nor live credential separation is claimed.

Blocked — requirements on other repos (2026-09-28)

Nothing further is implementable in this repository; T01 and T03 are done and T04's local guidance is complete. Workplan set to blocked. Requirements:

Owner	Requirement	Gates
glas-harness (GLAS-WP-0012, blocked; T03 progress, T02/T04/T05 wait)	End-to-end production acceptance receipt for the local supervised profile, usable for an interactive agent	T02
railiance-platform	ServiceAccount/cert, ClusterRole and binding in git; no secrets/pods/exec	T02
railiance-enablement	k3s write-kubeconfig-mode: 600 in install config	T02
railiance-infra	Host principal mapping; remove unrestricted sudo/admin kubeconfig from the agent account	T02
ops-warden	Certificate issuance for the agent identity	T02
harness owners (Codex/Grok)	Read-denial guard equivalent to the Claude hook, or a recorded statement that none exists	T04

Resume when GLAS-WP-0012 records production acceptance: then T02 proof (auth can-i, whoami, approved/unapproved action, admin-path denial) and T04 documentation of the verified path. T05 stays trigger-based.

Exact-release knowledge distribution proposal — 2026-10-03

EMERGENT-WP-0002 T09 coordination produced docs/emergent-knowledge-distribution-draft.md: owner responsibilities, standing-notice/acknowledgement mechanics, proof conditions, 30-day review, source invalidation and withdrawal. It is unaccepted and unpublished. T04 remains wait and T02's interactive isolation remains unfinished. Local knowledge loading or model metadata checks do not satisfy runtime enforcement. No fleet notice, new authority record, credential access or deployment occurred.

Dependency review — 2026-10-04

Current file and Hub observations: docs/evidence/2026-10-04-custodian-blocked-review.json. GLAS-WP-0012 is still blocked; its T02, T03, T04 and T05 are all wait. The September 28 table's T03-progress description is historical. T02/T04 here remain wait, and this workplan remains blocked.

The glas-harness owner must return an exact accepted interactive runtime/profile revision, identity and supervisor, launch/admission receipt, approved-action execution receipt and refusal of unapproved or changed actions. The synthetic bwrap proof and launcher source tests do not replace end-to-end acceptance. GLAS-WP-0012-T02/T03 retain their sand-boxer and secrets-engine dependencies; the Custodian does not assume their custody or retry authority. The review's GLAS coordination case observes that owner commitment.

After that return, railiance-platform must deliver the versioned restricted identity/RBAC; railiance-enablement must return the installed k3s mode-600 configuration; railiance-infra must return actual account/principal isolation and old admin-path denials; ops-warden must return scoped SSH certificate issuance. Verify the combined path from the actual agent account: no system:masters, Secret reads or pod exec, no unrestricted sudo/admin credential access, and sanitized privileged execution with exact approval binding. Individual configuration proposals are not acceptance of that path.

T04 then documents the verified revision and gets a harness-owner receipt for Codex/Grok read denial, or explicitly records its absence. Existing local supervision guidance is complete; another local mode label cannot advance T02. T05 retains decision 1591daa1-6e85-4dfd-aadd-5a9fe1ef47fa and the founder's three original reopening triggers. This review grants no credential cutover, rotation or autopilot authority.

Implementation path review — 2026-10-19

The September/October dependency history above is historical. Reuse the existing Glas/sand-boxer/rein chain rather than introducing another supervisor service. Glas SCOPE.md was reconciled with its actual six-profile catalog and dated owner returns at 9ddf99c: no profile is ready; local 1.1.1 and FI research are unverified. The native credential lanes were activated and revoked in September; their old absence is not the remaining implementation problem.

The newer Custodian native-runtime receipt is authoritative over the October 16 Glas narrative: docs/changes/netkingdom-target-2026-10-17/native-runtime-install-r2/execution-receipt.json records installed artifact f59959b605a51abd2901becde14e7abd852c2bd3f848004b51d250d1cb81af6d and verified complete digest on Railiance. Offline rehearsal failed with phase_receipt_unavailable; process supervision verified quiescence after leader_left_descendants. That failed revision is retained. The subsequent docs/changes/netkingdom-target-2026-10-17/native-runtime-install-r3/execution-receipt.json records installed_runtime_offline_rehearsal_verified: rein, Glas and Claude startup all return zero, post-rehearsal full digest matches, durable receipt precedes cleanup, and no live/waitable descendants remain. Zero model calls and no credential/owner-configuration reads are explicit. Offline startup is done; actual interactive assignment admission and privileged-path acceptance remain.

Efficient remaining sequence within T02/T04:

1. Reconcile the existing HARNESS-WP-0008-T07 owner return with this installed artifact and successful revision-3 offline rehearsal (preserving revision-2 failure), then complete owner-admitted runtime/provider acceptance under the existing exact approval requirements. Do not reinstall or rebuild older candidates, or repeat the already completed offline rehearsal.
2. In parallel prepare restricted identity/RBAC, k3s mode-600 configuration and host principal/SSH issuance changes with their existing owners. Preparation does not need to wait for the model run; applying the combined cutover does require the exact reviewed proposal and privileged-path approval.
3. Admit the actual Custodian interactive assignment, including external tool paths. Prove Secret/exec denial, non-system:masters identity, denial of old admin SSH/sudo/socket/GitOps/credential paths, sanitized approved execution and refusal of changed/unapproved actions. A disposable sandbox or successful model task cannot replace these checks.
4. Document only that accepted execution revision in T04 and retain the explicit Codex/Grok guard availability statement. No equivalent hook was established by this review. T05 remains the original founder-triggered deferral.

Source cleanup inspection found only generated brief/work-record changes in Glas, platform, infra and warden, with sand-boxer, rein-aharness, rein-openweights and enablement clean. Glas's generated change matches GLAS-WP-0016-T02 already done in its source workplan; use the owning generator workflow to commit those projections. No privileged change, runtime retry, rotation or autonomy grant was performed by this review.

Concrete boundary source packet — 2026-10-19

Prepared docs/changes/CUST-WP-0073/credential-boundary-r1/PROPOSAL.md with SHA256SUMS-bound RBAC/identity candidate, host principal/account specification, cluster installer diff, source pins and a safe actual-account negative verifier. Seven counterexample/partial-proof tests pass after the same proposal revision-2 namespace-inventory correction; revision-1 digest inventory and refinement are retained. Installer patch applies to its pinned source. No live probe, identity issuance, account change or cutover ran.

Source ownership correction: railiance-cluster/ansible/bootstrap.yml owns the k3s installer and currently combines mode644 with unconditional admin kubeconfig fetch to the controller. railiance-enablement is S4, not this S2 installer owner. Route the persistent mode600 change to cluster; enablement is consulted only if distribution tooling requires it. The proposed installer diff also removes admin kubeconfig export. Existing hosts need an exact reviewed effective unit/config override because the installer creates guard skips their flags; chmod alone can be reverted by the old flag at restart.

Infra's current SSH inventory maps generic agent principals into tegwick. The packet proposes a new dedicated account/principal without altering existing bridge consumers. Actual assignment isolation must additionally exclude old principal credentials, external tools/connectors and writable privileged paths. The negative verifier uses auth reviews and access checks only, refuses the old account/identity and never upgrades a passing subset to assignment admission. Exact effective host/workstation inputs and privileged execution binding remain required before an execution approval packet is complete.

Existing-host read-only refinement — 2026-10-19

The next-review addendum is retained under docs/changes/CUST-WP-0073/existing-k3s-host-r1/; the original revision-2 packet is unchanged. Timestamped, host-bound observation used existing SSH with strict host-key checking and no sudo, issuance or credential-content reads. k3s is active with one effective mode644 flag, no unit drop-ins and no pending reload. Its regular root-owned unit preimage SHA256 is b5398c385430ce225faadc1f3724597fe4b05dd999a04b8abbca64d4b0d68c0e; admin kubeconfig metadata remains root:root0644. The pure guarded two-byte transformation yields candidate SHA256 866ea58bfab00215e5bd3352287ebb0230d4ddb7dd3ed4b1cd853e74c6698d7d. Five tests pass, and SHA256SUMS verifies the retained review inputs. No remote candidate, mutation, reload or restart occurred.

This replaces the generic missing-host-baseline gap with exact evidence; it still is not an execution-approved revision. The protected service environment was not read. Its owner must supply only nonsecret config-selector/conflict predicates, then bind the exact executor, interruption window and recovery to supervisor review. The addendum specifies guarded replacement, verification and compare-and-swap recovery without silently restoring world-readable credentials. Full workstation/assignment isolation, native credential/egress acceptance and actual-account denial proof remain required under T02/T04. No scored proposal success or runtime admission follows from this read-only preparation.

Interactive-assignment priority — 2026-10-19 interview

Decision 8a64ba65-1068-409c-86f8-b9414b29b7b4 selects isolation of this actual interactive Codex assignment first. T02 is progress for owner launch/transport work. Remove the blanket GLAS-WP-0012 prerequisite: that native Claude-worker proof cannot establish this assignment's boundary. Reuse any actual shared mechanism needed, but require supported initial launch before the first tool, no parent/external credential bypass, and the existing actual-account positive and negative proofs. This prioritization grants no privileged cutover or access scope change and does not weaken acceptance.

The separate read-only protected-input preflight is retained at docs/changes/CUST-WP-0073/privileged-preflight-r1/OPERATOR.md. Its eight fixture tests and retained SHA256SUMS pass independent review. It has not been executed with privileges. The historical interactive-choice field predates the interview; the decision above supersedes that field without rewriting the original evidence. Root-owned staging and an exact supervisor-approved execution revision remain required for a live protected-input receipt; this preparation grants no cutover.

Supported interactive transport and target preparation are retained at docs/changes/CUST-WP-0073/interactive-launch-r2/. Official documentation supports SSH-connected app-server and native existing-chat handoff; availability in this installed frontend remains a user check. The proposed destination is a new dedicated custodian-codex WSL2 distribution/account on bnt-lap001, with no operator home, Windows interop/drives or privileged shared sockets. WSL2 availability is verified; the guest, account and connection do not exist yet. An exact provisioning recipe and effective-denial proof remain implementation work, not acceptance inferred from configuration. Revision 1 is retained; revision 2 corrects the partial identity probe to reject the original host/UID pair rather than UID1000 globally. Six probe tests and packet hashes pass. Neither partial identity nor a child-process probe admits the actual assignment.

Revision 3 adds locked provisioning scripts and exact WSL/account/SSH/service/ network configuration at docs/changes/CUST-WP-0073/interactive-launch-r3/. Five source/negative tests and hashes pass independently. The signed Ubuntu rootfs and seven-package SSH/firewall dependency closure are verified; source, signed metadata and package manifests are retained in ubuntu-dependencies-r1/ beside that packet. Four tamper/installer-refusal tests pass against the complete downloaded bundle. Its RETENTION.md distinguishes the Git review copy from the complete local binary bundle; no large image or package cache is committed.

No distribution, account or service has been created. Scripts remain locked. Installed frontend handoff availability, Windows endpoint/storage checks, actual runtime binary and namespace behavior, scoped authentication/egress and full assignment denial proofs remain required before cutover. The package preparation removes an implementation gap without granting execution authority.