the-custodian/workplans/CUST-WP-0073-agent-credential-separation.md

463 lines
26 KiB
Markdown

---
id: CUST-WP-0073
type: workplan
title: "Separate agent credentials and establish supervised privileged execution"
domain: infotech
repo: the-custodian
status: active
owner: the-custodian
topic_slug: custodian
flavor: implementation
created: "2026-09-24"
updated: "2026-10-19"
related:
- KEY-WP-0033
- RPF-WP-0044
origin: incident
origin_ref: key-cape/docs/operations.md#before-any-live-change
state_hub_workstream_id: "a98a9f34-83b4-5c8c-8107-e05f7806d50d"
---
# Separate agent credentials and establish supervised privileged execution
## Why
On 2026-09-23 a key-cape agent session meant to read only Secret metadata.
The go-template failed on an absent field, and kubectl printed
`sso/keycape-config` in full as debugging output: the signing key, the LLDAP
bind password and the Authelia client secret. Three more Secrets in `sso`/`mfa`
were probably printed as well.
The root cause is the credentials, not the command:
- Workstation `kubectl` and `ssh railiance01` both authenticate as
`system:admin` in `system:masters`, which bypasses authorization. No RBAC
rule can restrict it.
- k3s runs with `--write-kubeconfig-mode=644`, so `/etc/rancher/k3s/k3s.yaml`
is world-readable, and anything running as `tegwick` is cluster-admin.
- The paths that leak a value are open-ended: template error dumps,
`-o yaml`/`describe`, `helm get`, `kubectl logs`, `exec … env`, ConfigMaps
holding credentials, last-applied annotations, and the kubeconfig file itself.
A command denylist chases them one by one, and it only binds the harness that
enforces it.
**Goal:** keep privileged credentials outside the agent's direct reach, and
mediate privileged actions according to the identified agent's autonomy mode.
Agents start supervised; proven agents may later receive scoped autopilot
permission with a cost budget and risk limit in EUR. Neither mode implies
unrestricted admin credentials. Founder decision, September 28:
`docs/agent-autonomy-decision.md`.
**Scope:** builder mode, founder decision 2026-09-24. Rotating the exposed
Secrets is deferred, not dropped (T05). This plan removes the problem class.
## Decide the identity model
```task
id: CUST-WP-0073-T01
status: done
priority: high
state_hub_task_id: "4781bb99-020f-59f6-be13-e3b8777d3fd8"
```
**Done 2026-09-28 — policy decision.** The founder chooses autonomy as a
characteristic of the agent: supervised-mode initially, promotion only on
successful proposals without adaptation/refinement, and autopilot bounded by
cost and risk limits in EUR. Decision: `docs/agent-autonomy-decision.md`.
The supervised starting identity is outside `system:masters` and has a reviewed
observation allowlist. Exact privileged actions go through supervisor approval
or supervisor execution, with unchanged-acceptance and verified unchanged-success
recorded separately. Admin credentials remain in the privileged execution path,
outside the agent's direct reach. Later autopilot removes per-action approval
only for an explicit grant within scope, cost and risk constraints; it does not
hand out unrestricted sudo or an admin kubeconfig.
Built-in `view` plus Deployment/ConfigMap writes cannot establish that boundary:
workload writes can extract credentials, and ConfigMaps/pod specs/logs can contain
values. Agent-visible results must be sanitized. T02 selects and proves the
actual account/profile/execution path. This decision resolves the identity and
autonomy policy, not the implementation, numeric promotion thresholds, euro
limits or authorization of a live cutover. Existing human-only lanes still apply.
## Build and hand out the agent identity
```task
id: CUST-WP-0073-T02
status: progress
priority: high
depends_on: []
blocking_reason: "Interactive Codex assignment is the first target under the 2026-10-19 interview decision. Preparing a supported initial launch/tool-transport boundary and exact owner preflight; no actual account/profile admission yet. Native Claude-worker GLAS-WP-0012 proof is a distinct path, not a substitute."
state_hub_task_id: "4b88b0b7-dc7e-5612-aa5d-1a08f0530c35"
```
Owner: railiance-platform (RBAC), railiance-cluster (k3s install; ownership corrected 2026-10-19),
railiance-infra (host principal mapping), ops-warden (certificate issuance).
- Bind the agent's stable identity and `supervised-mode` to its existing
instance/assignment record and a restricted runtime profile. Name its
supervisor and privileged execution path. No raw admin credentials in the
agent process/account; no unrestricted sudo, socket or GitOps bypass.
- Create the ServiceAccount or client certificate, the ClusterRole and the
binding in git, applied by the owner's documented path.
- Set `write-kubeconfig-mode` to `600` in the k3s install config. Attended admin
use becomes `sudo k3s kubectl`.
- Workstation: `~/.kube/config` points at the agent kubeconfig. The admin
kubeconfig moves out of the agent's reach (another account, or short-lived
issuance through an attended login).
- Proof: as the agent identity, `kubectl auth can-i get secrets -A` and
`can-i create pods/exec -A` both answer `no`, and `kubectl auth whoami`
shows no `system:masters`. Retain these aggregate checks and additionally
test every namespaced forbidden permission in each namespace from a supervisor-verified
complete current inventory; aggregate denial does not exclude a namespaced grant.
Test cluster-scoped privileges separately. Record sanitized output as evidence. Also verify that
an exact supervisor-approved action can execute through the selected privileged
path and return sanitized outcome evidence, while an unapproved/revised action
cannot use that approval. Test from the actual agent account, including denial
of the old admin SSH/sudo/credential paths.
- Autopilot remains disabled without a scoped promotion decision, concrete EUR
cost/risk limits and verified enforcement. Implementing a general promotion or
risk-scoring service is not required for this supervised credential boundary.
## Reject last-applied annotations on Secrets
```task
id: CUST-WP-0073-T03
status: done
priority: medium
needs_human: false
state_hub_task_id: "5abbfcae-eb65-5b62-a7fa-f4f698f95312"
```
Owner: railiance-platform.
**Done 2026-09-28.** Explicit metadata fixes landed in all 31 affected ESO
declarations. The guard is active and Synced/Healthy; all nine native admission
checks and 39/39 fresh ESO refreshes under Deny pass. All 257 active-namespace
Secrets were annotation-free. After explicit founder approval, the one orphan
drill Secret was deleted with its UID precondition; absence verified and the
3 GiB PVC unchanged. Full evidence: `docs/evidence/2026-09-28-secret-annotation-rollout.md`.
- Add a `ValidatingAdmissionPolicy` (v1.35 is available) with its binding. It
rejects any Secret carrying `kubectl.kubernetes.io/last-applied-configuration`.
- Strip the annotation from existing Secrets first, cluster-wide, using the
output-suppressing maintenance helper, or the policy blocks their next update. Known:
`sso/keycape-config`, `sso/authelia-secrets`, `sso/lldap-secrets`,
`mfa/privacyidea-config`.
- Proof: a client-side `kubectl apply` of a test Secret in a scratch namespace
is refused.
## Fleet guidance and harness guards
```task
id: CUST-WP-0073-T04
status: wait
priority: medium
depends_on: [CUST-WP-0073-T02]
blocking_reason: "Await CUST-WP-0073-T02: the verified agent identity and execution path must exist before it can be documented; Codex/Grok read-denial guard needs glas-harness or harness-owner delivery."
state_hub_task_id: "90725511-4e31-549f-b567-47feff1a9ca4"
```
- Orientation doc §6 documents the template-error dump and requires the safe
maintenance helper; standalone Secret templates/jsonpath are forbidden. The
September 28 standing notice supersedes the September 24 inline-check exception.
Add the verified agent identity/execution path once T02 lands.
- Claude Code guard, **done on the workstation 2026-09-24**:
`~/.claude/hooks/guard-secret-reads.py` (PreToolUse on Bash). It denies Secret
reads, `helm get`, `config view --raw` and kubeconfig reads, and asks before
exec/attach/cp/debug and `--raw`. The `kubectl * get secret *` allow rule and
the expired 2026-07-07 temporary elevation are removed. Known false positive:
any command naming both kubectl and "secret", including an `echo`. That is
acceptable for a stopgap.
- Offer the same guard to the Codex and Grok harnesses, or record that they have
none. Until T02 lands, those agents are protected by instructions only.
- OpenBao/Vault secret-reading permissions also belong behind the privileged
boundary; preapproved command prefixes do not implement supervision.
- Document the agent-specific mode and supervisor. Reference exact proposal and
execution receipts; track unchanged acceptance separately from verified
unchanged success, including refinements, rejections and interventions. Reuse
existing records and receipts per `docs/agent-autonomy-decision.md`; no new
dashboard, supervisor service or automatic promotion machinery.
## Rotate what was exposed
```task
id: CUST-WP-0073-T05
status: wait
priority: low
needs_human: true
decision_id: 1591daa1-6e85-4dfd-aadd-5a9fe1ef47fa
blocking_reason: "Deferred by founder decision 2026-09-24; reopened only by the founder on one of the three triggers below, in one attended rotation window."
state_hub_task_id: "1b41b532-38a2-5f39-b387-28737934811a"
```
Deferred by founder decision on 2026-09-24 (builder mode). Rotate
`sso/keycape-config` (signing key, LLDAP bind password, Authelia client secret),
`sso/authelia-secrets`, `sso/lldap-secrets` and `mfa/privacyidea-config`
together, with `replace` rather than `apply`, in one attended window.
Reopen on the first of:
- the first production workload or customer data passing through KeyCape
- any sign the 2026-09-23 transcript left the workstation beyond the model
provider
- a planned key rotation
The passage of time alone reopens nothing.
## Current evidence and handoff — September 28
The founder requests minimal additional tasks, workplans and functionality.
All remaining work stays in T02, T04 and the original deferred T05. No new
workplan, task, controller or supervisor service was introduced.
T01 is done: autonomy is per agent and scope, supervised initially, with later
promotion bounded by EUR cost and risk limits. The decision and descriptive
per-agent record do not establish runtime enforcement or grant autopilot.
T02 remains in progress. The existing sand-boxer `profile.bwrap-local` passed a
synthetic process-isolation proof: admin homes, privileged environment variables,
Kubernetes/container sockets absent; only loopback networking; observation and
proposal paths work; wrong consumer rejected; workspace destroyed. Receipt:
`docs/evidence/2026-09-28-supervised-sandbox-proof.json`. It was not an interactive
agent migration. The selected GLAS local-profile acceptance remains blocked;
existing coordination receipt: `docs/evidence/2026-09-28-supervised-runtime-coordination.json`.
Actual account/profile admission and denial of old admin SSH/sudo/credential
paths are still required. The inspected legacy account has unrestricted sudo
and k3s kubeconfig mode 644; changing its default kubeconfig alone is insufficient.
T03 is done. All 31 affected ESO declarations have explicit target metadata in
23 files across 12 owning repositories, committed and published. The guard is
active and Synced/Healthy at platform source `7daf7e9`, pinned by `db51ec8`.
Nine native admission checks and 39/39 fresh ESO refreshes under Deny passed.
The founder-approved UID-bound deletion removed only the orphan drill Secret;
the 3 GiB PVC was unchanged. The final complete scan checked 257 Secrets with
zero forbidden annotations and zero orphan exceptions. The earlier failed
rollout and rollback remain historical evidence in
`docs/evidence/2026-09-28-secret-annotation-rollout.md`; they are not the live state.
T04's local guidance and reporting are implemented. AGENTS.md now loads the
supervision decision and per-agent record at startup and describes recording
original proposals before approval and verified outcomes afterwards. The
September 28 standing notice withdraws the unsafe inline presence check:
51e9eace-06d2-46d6-921a-4b92440df68f. Receipt: `docs/evidence/2026-09-28-secret-guidance-notice.json`.
Codex/Grok have no equivalent read-denial hook established by this work. The
remaining T04 step is to document the actual verified T02 execution path.
The original rollout remains an unscored failed proposal with refinement and
recovery; successful remediation does not become unchanged-success credit.
There is no eligible acceptance-rate sample or autopilot grant yet.
T05 remains the founder's trigger-based rotation deferral, not cancelled or done.
Neither workplan completion nor live credential separation is claimed.
## Blocked — requirements on other repos (2026-09-28)
Nothing further is implementable in this repository; T01 and T03 are done and
T04's local guidance is complete. Workplan set to `blocked`. Requirements:
| Owner | Requirement | Gates |
|-------|-------------|-------|
| glas-harness (GLAS-WP-0012, blocked; T03 progress, T02/T04/T05 wait) | End-to-end production acceptance receipt for the local supervised profile, usable for an interactive agent | T02 |
| railiance-platform | ServiceAccount/cert, ClusterRole and binding in git; no `secrets`/`pods/exec` | T02 |
| railiance-enablement | k3s `write-kubeconfig-mode: 600` in install config | T02 |
| railiance-infra | Host principal mapping; remove unrestricted sudo/admin kubeconfig from the agent account | T02 |
| ops-warden | Certificate issuance for the agent identity | T02 |
| harness owners (Codex/Grok) | Read-denial guard equivalent to the Claude hook, or a recorded statement that none exists | T04 |
Resume when GLAS-WP-0012 records production acceptance: then T02 proof
(`auth can-i`, whoami, approved/unapproved action, admin-path denial) and T04
documentation of the verified path. T05 stays trigger-based.
## Exact-release knowledge distribution proposal — 2026-10-03
EMERGENT-WP-0002 T09 coordination produced
`docs/emergent-knowledge-distribution-draft.md`: owner responsibilities,
standing-notice/acknowledgement mechanics, proof conditions, 30-day review,
source invalidation and withdrawal. It is unaccepted and unpublished.
T04 remains wait and T02's interactive isolation remains unfinished. Local
knowledge loading or model metadata checks do not satisfy runtime enforcement.
No fleet notice, new authority record, credential access or deployment occurred.
## Dependency review — 2026-10-04
Current file and Hub observations:
`docs/evidence/2026-10-04-custodian-blocked-review.json`. GLAS-WP-0012 is
still blocked; its T02, T03, T04 and T05 are all `wait`. The September 28
table's T03-progress description is historical. T02/T04 here remain `wait`,
and this workplan remains `blocked`.
The glas-harness owner must return an exact accepted interactive runtime/profile
revision, identity and supervisor, launch/admission receipt, approved-action
execution receipt and refusal of unapproved or changed actions. The synthetic
bwrap proof and launcher source tests do not replace end-to-end acceptance.
GLAS-WP-0012-T02/T03 retain their sand-boxer and secrets-engine dependencies;
the Custodian does not assume their custody or retry authority. The review's
GLAS coordination case observes that owner commitment.
After that return, railiance-platform must deliver the versioned restricted
identity/RBAC; railiance-enablement must return the installed k3s mode-600
configuration; railiance-infra must return actual account/principal isolation
and old admin-path denials; ops-warden must return scoped SSH certificate
issuance. Verify the combined path from the actual agent account: no
`system:masters`, Secret reads or pod exec, no unrestricted sudo/admin
credential access, and sanitized privileged execution with exact approval
binding. Individual configuration proposals are not acceptance of that path.
T04 then documents the verified revision and gets a harness-owner receipt for
Codex/Grok read denial, or explicitly records its absence. Existing local
supervision guidance is complete; another local mode label cannot advance T02.
T05 retains decision `1591daa1-6e85-4dfd-aadd-5a9fe1ef47fa` and the founder's
three original reopening triggers. This review grants no credential cutover,
rotation or autopilot authority.
## Implementation path review — 2026-10-19
The September/October dependency history above is historical. Reuse the existing
Glas/sand-boxer/rein chain rather than introducing another supervisor service.
Glas `SCOPE.md` was reconciled with its actual six-profile catalog and dated
owner returns at `9ddf99c`: no profile is ready; local 1.1.1 and FI research are
unverified. The native credential lanes were activated and revoked in September;
their old absence is not the remaining implementation problem.
The newer Custodian native-runtime receipt is authoritative over the October 16
Glas narrative: `docs/changes/netkingdom-target-2026-10-17/native-runtime-install-r2/execution-receipt.json`
records installed artifact `f59959b605a51abd2901becde14e7abd852c2bd3f848004b51d250d1cb81af6d`
and verified complete digest on Railiance. Offline rehearsal failed with
`phase_receipt_unavailable`; process supervision verified quiescence after
`leader_left_descendants`. That failed revision is retained. The subsequent
`docs/changes/netkingdom-target-2026-10-17/native-runtime-install-r3/execution-receipt.json`
records `installed_runtime_offline_rehearsal_verified`: rein, Glas and Claude
startup all return zero, post-rehearsal full digest matches, durable receipt
precedes cleanup, and no live/waitable descendants remain. Zero model calls and
no credential/owner-configuration reads are explicit. Offline startup is done;
actual interactive assignment admission and privileged-path acceptance remain.
Efficient remaining sequence within T02/T04:
1. Reconcile the existing HARNESS-WP-0008-T07 owner return with this installed
artifact and successful revision-3 offline rehearsal (preserving revision-2
failure), then complete owner-admitted runtime/provider acceptance under the
existing exact approval requirements. Do not reinstall or rebuild older
candidates, or repeat the already completed offline rehearsal.
2. In parallel prepare restricted identity/RBAC, k3s mode-600 configuration and
host principal/SSH issuance changes with their existing owners. Preparation
does not need to wait for the model run; applying the combined cutover does
require the exact reviewed proposal and privileged-path approval.
3. Admit the actual Custodian interactive assignment, including external tool
paths. Prove Secret/exec denial, non-system:masters identity, denial of old
admin SSH/sudo/socket/GitOps/credential paths, sanitized approved execution
and refusal of changed/unapproved actions. A disposable sandbox or successful
model task cannot replace these checks.
4. Document only that accepted execution revision in T04 and retain the explicit
Codex/Grok guard availability statement. No equivalent hook was established
by this review. T05 remains the original founder-triggered deferral.
Source cleanup inspection found only generated brief/work-record changes in
Glas, platform, infra and warden, with sand-boxer, rein-aharness, rein-openweights
and enablement clean. Glas's generated change matches GLAS-WP-0016-T02 already
done in its source workplan; use the owning generator workflow to commit those
projections. No privileged change, runtime retry, rotation or autonomy grant
was performed by this review.
## Concrete boundary source packet — 2026-10-19
Prepared `docs/changes/CUST-WP-0073/credential-boundary-r1/PROPOSAL.md` with
SHA256SUMS-bound RBAC/identity candidate, host principal/account specification,
cluster installer diff, source pins and a safe actual-account negative verifier.
Seven counterexample/partial-proof tests pass after the same proposal revision-2
namespace-inventory correction; revision-1 digest inventory and refinement are
retained. Installer patch applies to its
pinned source. No live probe, identity issuance, account change or cutover ran.
Source ownership correction: `railiance-cluster/ansible/bootstrap.yml` owns the
k3s installer and currently combines mode644 with unconditional admin kubeconfig
fetch to the controller. `railiance-enablement` is S4, not this S2 installer
owner. Route the persistent mode600 change to cluster; enablement is consulted
only if distribution tooling requires it. The proposed installer diff also
removes admin kubeconfig export. Existing hosts need an exact reviewed effective
unit/config override because the installer `creates` guard skips their flags;
chmod alone can be reverted by the old flag at restart.
Infra's current SSH inventory maps generic agent principals into tegwick. The
packet proposes a new dedicated account/principal without altering existing
bridge consumers. Actual assignment isolation must additionally exclude old
principal credentials, external tools/connectors and writable privileged paths.
The negative verifier uses auth reviews and access checks only, refuses the
old account/identity and never upgrades a passing subset to assignment admission.
Exact effective host/workstation inputs and privileged execution binding remain
required before an execution approval packet is complete.
## Existing-host read-only refinement — 2026-10-19
The next-review addendum is retained under
`docs/changes/CUST-WP-0073/existing-k3s-host-r1/`; the original revision-2 packet
is unchanged. Timestamped, host-bound observation used existing SSH with strict
host-key checking and no sudo, issuance or credential-content reads. k3s is
active with one effective mode644 flag, no unit drop-ins and no pending reload.
Its regular root-owned unit preimage SHA256 is
`b5398c385430ce225faadc1f3724597fe4b05dd999a04b8abbca64d4b0d68c0e`;
admin kubeconfig metadata remains root:root0644. The pure guarded two-byte
transformation yields candidate SHA256
`866ea58bfab00215e5bd3352287ebb0230d4ddb7dd3ed4b1cd853e74c6698d7d`.
Five tests pass, and SHA256SUMS verifies the retained review inputs. No remote
candidate, mutation, reload or restart occurred.
This replaces the generic missing-host-baseline gap with exact evidence; it
still is not an execution-approved revision. The protected service environment
was not read. Its owner must supply only nonsecret config-selector/conflict
predicates, then bind the exact executor, interruption window and recovery to
supervisor review. The addendum specifies guarded replacement, verification and
compare-and-swap recovery without silently restoring world-readable credentials.
Full workstation/assignment isolation, native credential/egress acceptance and
actual-account denial proof remain required under T02/T04. No scored proposal
success or runtime admission follows from this read-only preparation.
## Interactive-assignment priority — 2026-10-19 interview
Decision `8a64ba65-1068-409c-86f8-b9414b29b7b4` selects isolation of this actual
interactive Codex assignment first. T02 is progress for owner launch/transport
work. Remove the blanket GLAS-WP-0012 prerequisite: that native Claude-worker
proof cannot establish this assignment's boundary. Reuse any actual shared
mechanism needed, but require supported initial launch before the first tool,
no parent/external credential bypass, and the existing actual-account positive
and negative proofs. This prioritization grants no privileged cutover or access
scope change and does not weaken acceptance.
The separate read-only protected-input preflight is retained at
`docs/changes/CUST-WP-0073/privileged-preflight-r1/OPERATOR.md`. Its eight fixture
tests and retained SHA256SUMS pass independent review. It has not been executed
with privileges. The historical interactive-choice field predates the interview;
the decision above supersedes that field without rewriting the original evidence.
Root-owned staging and an exact supervisor-approved execution revision remain
required for a live protected-input receipt; this preparation grants no cutover.
Supported interactive transport and target preparation are retained at
`docs/changes/CUST-WP-0073/interactive-launch-r2/`. Official documentation
supports SSH-connected app-server and native existing-chat handoff; availability
in this installed frontend remains a user check. The proposed destination is a
new dedicated `custodian-codex` WSL2 distribution/account on `bnt-lap001`, with
no operator home, Windows interop/drives or privileged shared sockets. WSL2
availability is verified; the guest, account and connection do not exist yet.
An exact provisioning recipe and effective-denial proof remain implementation
work, not acceptance inferred from configuration. Revision 1 is retained;
revision 2 corrects the partial identity probe to reject the original host/UID
pair rather than UID1000 globally. Six probe tests and packet hashes pass.
Neither partial identity nor a child-process probe admits the actual assignment.
Revision 3 adds locked provisioning scripts and exact WSL/account/SSH/service/
network configuration at `docs/changes/CUST-WP-0073/interactive-launch-r3/`.
Five source/negative tests and hashes pass independently. The signed Ubuntu
rootfs and seven-package SSH/firewall dependency closure are verified; source,
signed metadata and package manifests are retained in `ubuntu-dependencies-r1/`
beside that packet. Four tamper/installer-refusal tests pass against the complete
downloaded bundle. Its `RETENTION.md` distinguishes the Git review copy from
the complete local binary bundle; no large image or package cache is committed.
No distribution, account or service has been created. Scripts remain locked.
Installed frontend handoff availability, Windows endpoint/storage checks,
actual runtime binary and namespace behavior, scoped authentication/egress and
full assignment denial proofs remain required before cutover. The package
preparation removes an implementation gap without granting execution authority.