--- id: WARDEN-WP-0028 type: workplan title: "Tenant secret custody — NetKingdom pattern for client/tenant secrets" domain: infotech repo: ops-warden status: finished owner: grok topic_slug: custodian planning_priority: high planning_order: 28 created: "2026-07-16" updated: "2026-07-17" state_hub_workstream_id: "85b18255-d6a7-4c5e-bae0-e5b5bbc43757" --- # WARDEN-WP-0028 — Tenant secret custody (NetKingdom pattern) ## Origin Founder establishing **binky-control** as the control-plane repo for a Coulomb client/tenant (Binky-Hedgehog GmbH). Immediate need: store **tenant-specific** secrets in OpenBao (e.g. company mailbox usernames/passwords for IMAP scan) and consume them safely via email-connect / agents **without** putting values in git, State Hub, chat, or workplans. BINKY-WP-0003-T01 prepared a draft plan (`binky-control/integrations/company-email-openbao.md`) that targeted `secret/prod/binky/...` via secrets-engine. That draft is directionally right on *workflow*, but it does not match a production-ready custody layout on `bao.coulomb.social`. This workplan resolves the NetKingdom layout and makes **tenant** secrets first-class — **separate from** platform workload lanes (`platform/workloads/...`). ## Goal Define and land a **repeatable, multi-tenant-safe** pattern for: 1. Where tenant secrets live in OpenBao (path + mount convention) 2. Who owns policy, CCR, catalog, and front door 3. How agents and workloads obtain values (sanctioned transports only) 4. First concrete lane: **binky company email IMAP** (unblocks real-world event intake for binky-control) ## Architectural facts (as of 2026-07-16) ### Planes (ADR-0006) | Plane | Role | Example | | --- | --- | --- | | Bootstrap | First trust / recovery | unseal, platform-root | | Platform control | Shared identity, secrets, authz | OpenBao, key-cape, flex-auth, railiance-platform | | Tenant | Scoped client/workload authority | coulomb products, **binky**, future clients | **Binky is a tenant**, not platform-root. Tenant secrets must not require platform-admin for day-to-day read, and tenant operators must not gain platform-root grants. ### OpenBao mounts (live + planned) | Mount | Type | Role | | --- | --- | --- | | **`platform/`** | KV v2 | **Platform control-plane** secrets: `platform/workloads/...` (fleet CCRs, ESO) and `platform/operators/...` — *not* client/tenant commercial secrets | | **`secret/`** | KV v2 | secrets-engine stage / pilot paths (`secret/coulomb/...`) — not the tenant home | | **`tenants/`** | KV v2 (**new**) | **Client/tenant commercial secrets** — explicit multi-tenant namespace | Live under `platform/workloads/` today: `activity-core/`, `coulomb/`, `forgejo/`, `issue-core/`, `railiance/`, `reuse/`. Those stay where they are; we do **not** migrate them into `tenants/` in this workplan. ### Dual front-door systems (both real; don't invent a third) | Layer | Owns | Entry | | --- | --- | --- | | **OpenBao** | Custody, ACL, audit | `bao` CLI / API | | **railiance-platform CCR** | Approved metadata apply (policy + OIDC role + evidence) | `credential-change-requests/CCR-*.yaml` | | **ops-warden catalog** | Routing + `warden access` proxy (no custody) | `registry/routing/catalog.yaml` | | **secrets-engine catalog** | Workflow: decision → plan → provision → exec delivery | `secrets-engine/catalog/*.yaml` | **Binding rule (ops-warden):** never vend secret values; only route/proxy as the caller. **Binding rule (WP-0026):** capabilities-safe verify; agents on `risk: high` use `--out` / `--exec` / `--wrap` only. ### Why not bury tenants under `platform/workloads/` Earlier draft of this workplan used `platform/workloads//...`. Founder prefer a **shallower, explicit** namespace. That is sound: - **No conflict with Vault/OpenBao best practice.** Separate mounts (or top-level prefixes) for different security *domains* are common and good. `platform/workloads` is an *internal fleet convention* for platform services, not an industry mandate for every secret. - **Clearer mental model:** platform plane vs tenant plane (ADR-0006) maps to mount/prefix names operators can see at a glance. - **Isolation:** ACLs, list boundaries, and future audit queries stay simpler when tenant material is not mixed under the same path tree as forgejo/ESO lanes. ### Decision (2026-07-17) — use dedicated mount `tenants/` ```text mount: tenants # KV v2, enable once (platform-admin) logical path: // CLI: tenants/// API data path: tenants/data/// ``` **Rejected alternatives (kept for archaeology only):** | Option | Why not | | --- | --- | | `secret/prod/binky/...` | `secret/prod/` does not exist; stage model not ready as sole production path | | `platform/workloads/binky/...` | Too deep; conflates tenant clients with platform workloads | | `platform/tenants/binky/...` | Acceptable fallback if enabling a new mount is blocked; prefer full `tenants/` mount | ### Tooling debt to clear (not a reason to keep the deep path) `railiance-platform/scripts/credential-change.py` currently hard-fails unless: - `openbao.mount == "platform"` - `openbao.kv_path` starts with `platform/workloads/` That guard must be **extended** to allow `mount: tenants` and `kv_path` under `tenants/` (T03). It is an implementation constraint, not a security reason to force the deep path. ## Recommended path convention (tenant secrets) ```text tenants/// ``` | Segment | Rules | Binky email example | | --- | --- | --- | | `tenant_slug` | Stable kebab slug; company/client id | `binky` | | `workload` | Capability or system using the secret | `company-email` | | `bundle` | Single purpose unit (one CCR / one policy) | `imap` | Full CLI path: ```text tenants/binky/company-email/imap ``` Suggested fields (names only — never values in git): | Field | Purpose | | --- | --- | | `IMAP_USERNAME` | mailbox login (often the address) | | `IMAP_PASSWORD` | app password / mailbox password | | `IMAP_HOST` | optional if not fixed in consumer config | | `IMAP_PORT` | optional (default 993 in config) | Non-secret connection facts (provider host if stable) may live in `binky-control` config / email-connect yaml as **env names + host**, not passwords. ### Future multi-mailbox / multi-tenant ```text tenants//mailbox/ # e.g. tenants/binky/mailbox/founder-primary # tenants/acme/mailbox/billing ``` One CCR + one least-privilege policy **per bundle** (or per mailbox). Do not grant `tenants/binky/*` list/read to agents — exact-path policies only. ### Risk class Mailbox credentials and similar client secrets are **`risk: high`** (WP-0026 T04): agent identities get metadata/capabilities only; raw stream refused when `WARDEN_AGENT_ID` is set. ## Ownership split | Concern | Owner repo | Artifact | | --- | --- | --- | | Tenant business need, IMAP host facts, scan config, queues | **binky-control** | `integrations/*`, mailmeta (metadata only), activity defs | | OpenBao path, policy HCL, OIDC role, CCR lifecycle | **railiance-platform** | `CCR-YYYY-NNNN-…yaml`, `openbao/policies/…` | | Routing front door + rotation guidance | **ops-warden** | catalog entry, playbook, `warden access` / `rotate-guide` | | Exec-time delivery workflow (optional same path) | **secrets-engine** | catalog entry pointing at `tenants/...` (same path; no second copy) | | Identity groups for tenant operators | **net-kingdom / key-cape** | group e.g. `tenant-binky-operators` (near-term may use `net-kingdom-admins` for founder-only) | | Mailbox scan consumer | **email-connect** | config with `username_env` / `password_env` only | ## Auth model (near-term vs target) **Near-term (founder dogfood):** OIDC `netkingdom` role bound to `groups=net-kingdom-admins` (same pattern as backup/forgejo lanes). Acceptable while binky is founder-operated only. **Target (real multi-tenant):** - IAM group `tenant-binky-operators` (and later per-tenant groups) - OpenBao OIDC role `binky-company-email-imap-workload-kv-read` bound only to that group - flex-auth check `secret.read:binky-company-email` if pre-approval required - Agents **never** hold the workload-kv-read policy; use operator wrap / exec injection / AppRole for email-connect scanner only ## First worked lane (deliverable sketch) | Item | Value | | --- | --- | | Catalog id | `binky-company-email-imap` | | CCR | `CCR-2026-0007` (next free) — title: Binky company email IMAP | | Mount | `tenants` | | KV path | `tenants/binky/company-email/imap` | | Policy | `workload-kv-read-binky-company-email-imap` | | OIDC role | `binky-company-email-imap-workload-kv-read` | | Consumer | email-connect `scan-mailbox` via env inject | | Front door | `warden access binky-company-email-imap --out FILE` or `--exec -- …` | | Provision | **Red lane, founder once** — value via `bao kv put … @file` or secrets-engine provision; never chat | ## Out of scope - Mass rotation / lockdown machinery (WARDEN-WP-0027 Strand B) - Sending mail / SMTP from company address (separate lane if needed) - Migrating all `secret/` mount lanes onto `platform/` (separate secrets-engine / railiance work) - Full key-cape tenant group productization (may be a net-kingdom follow-on) ## Tasks ### T01 — Canon note: tenant secret path + ownership ```task id: WARDEN-WP-0028-T01 status: done priority: high state_hub_task_id: "b78cfae0-6863-4889-8a1c-81dc0b44ff3b" ``` Done 2026-07-17: `wiki/playbooks/tenant-secret-onboarding.md` + CredentialRouting tenant paragraph; catalog-lane-promotion draft table updated. Land a short ops-warden wiki page (and one paragraph in `wiki/CredentialRouting.md`) that freezes: - mount **`tenants`** + path `tenants///` - ownership table (binky-control / railiance-platform / ops-warden / secrets-engine) - link to WP-0026 hygiene (capabilities verify, high-risk, taint) - **do not** put new client/tenant commercial secrets under `platform/workloads/` or invent `secret/prod/...` as the production home **Done when:** wiki + CredentialRouting pointer exist; binky integration doc updated to match (or superseded with link). ### T02 — Align binky-control integration plan to production path ```task id: WARDEN-WP-0028-T02 status: done priority: high state_hub_task_id: "030c9e0d-cc25-44ea-93f1-ff575905dec4" ``` Done 2026-07-17: `binky-control/integrations/company-email-openbao.md` rewritten to `tenants/binky/company-email/imap` + checklist (host confirm + Red provision still open). **Cross-repo: binky-control.** Rewrite `integrations/company-email-openbao.md` to use `tenants/binky/company-email/imap`, CCR + ops-warden front door as primary, secrets-engine as optional exec wrapper on the same path. Confirm non-secret IMAP host/port with founder (Blue). Keep Red-lane value provision as the last step. **Done when:** binky doc matches this workplan; checklist items re-opened for implementation (T01 in BINKY was "prepared only"). ### T03 — Enable `tenants` mount + extend CCR tooling + policy/role ```task id: WARDEN-WP-0028-T03 status: done priority: high state_hub_task_id: "7ebc0a27-d2cc-4b71-985b-f793db583d91" ``` Done 2026-07-17: `tenants/` KV v2 mount live; CCR applier allowlist extended; CCR-2026-0007 + policy + OIDC role applied; lane-policy `read` / default `deny` / agent boundary on tenants path; `docs/workload-kv-access-lanes.md` tenant section. **Cross-repo: railiance-platform (+ OpenBao admin once).** 1. **Enable KV v2 mount** `tenants/` on `bao.coulomb.social` (platform-admin / attended; record non-secret evidence). Versioning + max versions policy aligned with `platform` if practical. 2. **Extend** `scripts/credential-change.py` allowlist: `mount: tenants` and `kv_path` under `tenants/` (keep existing `platform/workloads/` rules intact for fleet lanes). Update dry-run docs and tests. 3. Author `CCR-2026-0007` (or next id) for the binky IMAP path; policy HCL on exact data/metadata paths under `tenants/…`; OIDC role (founder group near-term). Dry-run; founder approves; apply metadata only. 4. Document fields without values; add a **tenant lanes** section to `docs/workload-kv-access-lanes.md` (or sibling doc). **Done when:** mount exists; CCR tool accepts tenant paths; CCR approved + policy/role applied; capabilities-safe positive/negative evidence (no value reads for verify). ### T04 — ops-warden catalog + playbook + rotation ```task id: WARDEN-WP-0028-T04 status: done priority: high state_hub_task_id: "bcb8b75c-9ed6-4fbd-a2d9-19fe127d6d86" ``` Done 2026-07-17: catalog draft `binky-company-email-imap` (`risk: high`, rotation, path `tenants/binky/...`); playbooks `binky-company-email-imap.md` + onboarding. Promote to active after T05. Add `binky-company-email-imap` to `registry/routing/catalog.yaml`: `status: draft` until verify, then promote; `risk: high`; `exec_capable: true`; concrete `fetch_command` for primary field; `rotation:` block (re-establish or rotate mailbox app-password). Playbook under `wiki/playbooks/`. Scorecard rotation coverage satisfied on promote. **Done when:** `warden route show binky-company-email-imap --json` shows expected pointers; after T03+T05, `resolvable: true`. ### T05 — Founder provision (Red) + first scan evidence ```task id: WARDEN-WP-0028-T05 status: done priority: high state_hub_task_id: "0f56a6ef-349b-41db-a6df-fe3f3cccdd13" ``` Done 2026-07-17: Founder provisioned via OpenBao UI (KV version 2; not placeholder). Capabilities-safe verify pass; catalog + CCR promoted active/resolvable. Optional follow-up: first email-connect read-only scan into `binky-control/mailmeta/` (OperatingRhythm), not a custody blocker. **Human-only value path.** Founder provisions IMAP username/password into OpenBao via approved tool (`bao kv put … @file` or secrets-engine provision). Agents never see the value. Run email-connect read-only scan with `warden access … --exec` or secrets-engine exec; store **metadata-only** evidence under `binky-control/mailmeta/`. **Done when:** one successful read-only scan evidence note exists; lane promoted active/resolvable; CCR verification evidence complete. ### T06 — Generalize "tenant secret onboarding" playbook ```task id: WARDEN-WP-0028-T06 status: done priority: medium state_hub_task_id: "913389da-72d8-4caf-a670-fdb74c20ee97" ``` Done 2026-07-17 with T01: `wiki/playbooks/tenant-secret-onboarding.md` + CredentialRouting link. From the binky lane, write a reusable playbook: `wiki/playbooks/tenant-secret-onboarding.md` — steps for any new tenant: slug, CCR template fields, policy naming, catalog entry, agent boundary, rotation block, promotion checklist. Intended for the next Coulomb client without redesign. **Done when:** playbook exists and is linked from CredentialRouting + first session notes for tenant repos. ### T07 — secrets-engine alignment decision (record only) ```task id: WARDEN-WP-0028-T07 status: done priority: low state_hub_task_id: "f65eafda-9e31-4b18-aeba-2e7a9d906ac6" ``` Done 2026-07-17: **Decision** — tenant production lanes use `mount: tenants` and the CCR path only; secrets-engine may wrap exec delivery against that same path later but must not store a second copy under `secret/`. Until stage roles gain an approved `tenants/` grant, production path is CCR + ops-warden + caller `bao`. Record a short decision: secrets-engine catalog entries for **tenant** production lanes **must** use `mount: tenants` and the same path as the CCR (no second copy under `secret/`). Stage roles may need a later extension to touch `tenants/` under an approved plan — until then, ops-warden + CCR + `bao` as caller are the production path. **Done when:** decision note in this workplan or `POST /decisions/` + wiki pointer; no dual-value storage for the IMAP password. ## Suggested implementation order 1. T01 canon (unblocks everyone) 2. T02 binky doc alignment 3. T03 mount + CCR tooling + CCR apply (metadata) 4. T04 catalog draft 5. T05 founder provision + verify + promote 6. T06 generalize playbook 7. T07 secrets-engine alignment (can parallel after T01) ## Acceptance - [ ] Documented path convention `tenants//…` used for ≥1 live lane (`binky`) - [ ] CCR applier accepts `tenants/` without weakening `platform/workloads/` rules - [ ] No secret values in any git/State Hub artifact - [ ] Lane is capabilities-verified and agent high-risk safe - [ ] email-connect can scan with env injection without printing password - [ ] Next tenant can copy the onboarding playbook without redesign ## References - ADR-0006 recursive multi-tenant identity (`net-kingdom/docs/adr/…`) - `net-kingdom/docs/secrets-engine-security-infrastructure-boundary.md` - `railiance-platform/docs/workload-kv-access-lanes.md` - `railiance-platform/docs/credential-change-approval.md` - `binky-control/integrations/company-email-openbao.md` (draft to align) - WARDEN-WP-0026 disclosure hygiene (capabilities, high-risk, taint, rotation) - WARDEN-WP-0027 Strand B (out of scope here)