WARDEN-WP-0028: tenant secrets on mount tenants/ (first lane draft)

Adopt tenants/<tenant>/… custody (not platform/workloads). Document onboarding, add draft binky-company-email-imap catalog entry, and mark T01–T04/T06–T07 done. Founder Red provision remains T05.
2026-07-17 00:09:28 +02:00 · 2026-07-17 00:09:28 +02:00 · 98a2339b81
commit 98a2339b81
parent b971403dad
7 changed files with 634 additions and 2 deletions
--- a/workplans/WARDEN-WP-0028-tenant-secret-custody.md
+++ b/workplans/WARDEN-WP-0028-tenant-secret-custody.md
@ -0,0 +1,408 @@
+---
+id: WARDEN-WP-0028
+type: workplan
+title: "Tenant secret custody — NetKingdom pattern for client/tenant secrets"
+domain: infotech
+repo: ops-warden
+status: active
+owner: grok
+topic_slug: custodian
+planning_priority: high
+planning_order: 28
+created: "2026-07-16"
+updated: "2026-07-17"
+---
+
+# WARDEN-WP-0028 — Tenant secret custody (NetKingdom pattern)
+
+## Origin
+
+Founder establishing **binky-control** as the control-plane repo for a Coulomb
+client/tenant (Binky-Hedgehog GmbH). Immediate need: store **tenant-specific**
+secrets in OpenBao (e.g. company mailbox usernames/passwords for IMAP scan) and
+consume them safely via email-connect / agents **without** putting values in
+git, State Hub, chat, or workplans.
+
+BINKY-WP-0003-T01 prepared a draft plan
+(`binky-control/integrations/company-email-openbao.md`) that targeted
+`secret/prod/binky/...` via secrets-engine. That draft is directionally right on
+*workflow*, but it does not match a production-ready custody layout on
+`bao.coulomb.social`. This workplan resolves the NetKingdom layout and makes
+**tenant** secrets first-class — **separate from** platform workload lanes
+(`platform/workloads/...`).
+
+## Goal
+
+Define and land a **repeatable, multi-tenant-safe** pattern for:
+
+1. Where tenant secrets live in OpenBao (path + mount convention)
+2. Who owns policy, CCR, catalog, and front door
+3. How agents and workloads obtain values (sanctioned transports only)
+4. First concrete lane: **binky company email IMAP** (unblocks real-world event
+   intake for binky-control)
+
+## Architectural facts (as of 2026-07-16)
+
+### Planes (ADR-0006)
+
+| Plane | Role | Example |
+| --- | --- | --- |
+| Bootstrap | First trust / recovery | unseal, platform-root |
+| Platform control | Shared identity, secrets, authz | OpenBao, key-cape, flex-auth, railiance-platform |
+| Tenant | Scoped client/workload authority | coulomb products, **binky**, future clients |
+
+**Binky is a tenant**, not platform-root. Tenant secrets must not require
+platform-admin for day-to-day read, and tenant operators must not gain
+platform-root grants.
+
+### OpenBao mounts (live + planned)
+
+| Mount | Type | Role |
+| --- | --- | --- |
+| **`platform/`** | KV v2 | **Platform control-plane** secrets: `platform/workloads/...` (fleet CCRs, ESO) and `platform/operators/...` — *not* client/tenant commercial secrets |
+| **`secret/`** | KV v2 | secrets-engine stage / pilot paths (`secret/coulomb/...`) — not the tenant home |
+| **`tenants/`** | KV v2 (**new**) | **Client/tenant commercial secrets** — explicit multi-tenant namespace |
+
+Live under `platform/workloads/` today: `activity-core/`, `coulomb/`, `forgejo/`,
+`issue-core/`, `railiance/`, `reuse/`. Those stay where they are; we do **not**
+migrate them into `tenants/` in this workplan.
+
+### Dual front-door systems (both real; don't invent a third)
+
+| Layer | Owns | Entry |
+| --- | --- | --- |
+| **OpenBao** | Custody, ACL, audit | `bao` CLI / API |
+| **railiance-platform CCR** | Approved metadata apply (policy + OIDC role + evidence) | `credential-change-requests/CCR-*.yaml` |
+| **ops-warden catalog** | Routing + `warden access` proxy (no custody) | `registry/routing/catalog.yaml` |
+| **secrets-engine catalog** | Workflow: decision → plan → provision → exec delivery | `secrets-engine/catalog/*.yaml` |
+
+**Binding rule (ops-warden):** never vend secret values; only route/proxy as the
+caller. **Binding rule (WP-0026):** capabilities-safe verify; agents on
+`risk: high` use `--out` / `--exec` / `--wrap` only.
+
+### Why not bury tenants under `platform/workloads/`
+
+Earlier draft of this workplan used `platform/workloads/<tenant>/...`. Founder
+prefer a **shallower, explicit** namespace. That is sound:
+
+- **No conflict with Vault/OpenBao best practice.** Separate mounts (or
+  top-level prefixes) for different security *domains* are common and good.
+  `platform/workloads` is an *internal fleet convention* for platform services,
+  not an industry mandate for every secret.
+- **Clearer mental model:** platform plane vs tenant plane (ADR-0006) maps to
+  mount/prefix names operators can see at a glance.
+- **Isolation:** ACLs, list boundaries, and future audit queries stay simpler
+  when tenant material is not mixed under the same path tree as forgejo/ESO
+  lanes.
+
+### Decision (2026-07-17) — use dedicated mount `tenants/`
+
+```text
+mount: tenants          # KV v2, enable once (platform-admin)
+logical path: <tenant_slug>/<workload>/<bundle>
+CLI: tenants/<tenant_slug>/<workload>/<bundle>
+API data path: tenants/data/<tenant_slug>/<workload>/<bundle>
+```
+
+**Rejected alternatives (kept for archaeology only):**
+
+| Option | Why not |
+| --- | --- |
+| `secret/prod/binky/...` | `secret/prod/` does not exist; stage model not ready as sole production path |
+| `platform/workloads/binky/...` | Too deep; conflates tenant clients with platform workloads |
+| `platform/tenants/binky/...` | Acceptable fallback if enabling a new mount is blocked; prefer full `tenants/` mount |
+
+### Tooling debt to clear (not a reason to keep the deep path)
+
+`railiance-platform/scripts/credential-change.py` currently hard-fails unless:
+
+- `openbao.mount == "platform"`
+- `openbao.kv_path` starts with `platform/workloads/`
+
+That guard must be **extended** to allow `mount: tenants` and
+`kv_path` under `tenants/` (T03). It is an implementation constraint, not a
+security reason to force the deep path.
+
+## Recommended path convention (tenant secrets)
+
+```text
+tenants/<tenant_slug>/<workload>/<bundle>
+```
+
+| Segment | Rules | Binky email example |
+| --- | --- | --- |
+| `tenant_slug` | Stable kebab slug; company/client id | `binky` |
+| `workload` | Capability or system using the secret | `company-email` |
+| `bundle` | Single purpose unit (one CCR / one policy) | `imap` |
+
+Full CLI path:
+
+```text
+tenants/binky/company-email/imap
+```
+
+Suggested fields (names only — never values in git):
+
+| Field | Purpose |
+| --- | --- |
+| `IMAP_USERNAME` | mailbox login (often the address) |
+| `IMAP_PASSWORD` | app password / mailbox password |
+| `IMAP_HOST` | optional if not fixed in consumer config |
+| `IMAP_PORT` | optional (default 993 in config) |
+
+Non-secret connection facts (provider host if stable) may live in
+`binky-control` config / email-connect yaml as **env names + host**, not
+passwords.
+
+### Future multi-mailbox / multi-tenant
+
+```text
+tenants/<tenant>/mailbox/<mailbox_id>
+# e.g. tenants/binky/mailbox/founder-primary
+#      tenants/acme/mailbox/billing
+```
+
+One CCR + one least-privilege policy **per bundle** (or per mailbox). Do not
+grant `tenants/binky/*` list/read to agents — exact-path policies only.
+
+### Risk class
+
+Mailbox credentials and similar client secrets are **`risk: high`** (WP-0026
+T04): agent identities get metadata/capabilities only; raw stream refused when
+`WARDEN_AGENT_ID` is set.
+
+## Ownership split
+
+| Concern | Owner repo | Artifact |
+| --- | --- | --- |
+| Tenant business need, IMAP host facts, scan config, queues | **binky-control** | `integrations/*`, mailmeta (metadata only), activity defs |
+| OpenBao path, policy HCL, OIDC role, CCR lifecycle | **railiance-platform** | `CCR-YYYY-NNNN-…yaml`, `openbao/policies/…` |
+| Routing front door + rotation guidance | **ops-warden** | catalog entry, playbook, `warden access` / `rotate-guide` |
+| Exec-time delivery workflow (optional same path) | **secrets-engine** | catalog entry pointing at `tenants/...` (same path; no second copy) |
+| Identity groups for tenant operators | **net-kingdom / key-cape** | group e.g. `tenant-binky-operators` (near-term may use `net-kingdom-admins` for founder-only) |
+| Mailbox scan consumer | **email-connect** | config with `username_env` / `password_env` only |
+
+## Auth model (near-term vs target)
+
+**Near-term (founder dogfood):** OIDC `netkingdom` role bound to
+`groups=net-kingdom-admins` (same pattern as backup/forgejo lanes). Acceptable
+while binky is founder-operated only.
+
+**Target (real multi-tenant):**
+
+- IAM group `tenant-binky-operators` (and later per-tenant groups)
+- OpenBao OIDC role `binky-company-email-imap-workload-kv-read` bound only to
+  that group
+- flex-auth check `secret.read:binky-company-email` if pre-approval required
+- Agents **never** hold the workload-kv-read policy; use operator wrap / exec
+  injection / AppRole for email-connect scanner only
+
+## First worked lane (deliverable sketch)
+
+| Item | Value |
+| --- | --- |
+| Catalog id | `binky-company-email-imap` |
+| CCR | `CCR-2026-0007` (next free) — title: Binky company email IMAP |
+| Mount | `tenants` |
+| KV path | `tenants/binky/company-email/imap` |
+| Policy | `workload-kv-read-binky-company-email-imap` |
+| OIDC role | `binky-company-email-imap-workload-kv-read` |
+| Consumer | email-connect `scan-mailbox` via env inject |
+| Front door | `warden access binky-company-email-imap --out FILE` or `--exec -- …` |
+| Provision | **Red lane, founder once** — value via `bao kv put … @file` or secrets-engine provision; never chat |
+
+## Out of scope
+
+- Mass rotation / lockdown machinery (WARDEN-WP-0027 Strand B)
+- Sending mail / SMTP from company address (separate lane if needed)
+- Migrating all `secret/` mount lanes onto `platform/` (separate secrets-engine
+  / railiance work)
+- Full key-cape tenant group productization (may be a net-kingdom follow-on)
+
+## Tasks
+
+### T01 — Canon note: tenant secret path + ownership
+
+```task
+id: WARDEN-WP-0028-T01
+status: done
+priority: high
+```
+
+Done 2026-07-17: `wiki/playbooks/tenant-secret-onboarding.md` + CredentialRouting
+tenant paragraph; catalog-lane-promotion draft table updated.
+
+Land a short ops-warden wiki page (and one paragraph in
+`wiki/CredentialRouting.md`) that freezes:
+
+- mount **`tenants`** + path `tenants/<tenant>/<workload>/<bundle>`
+- ownership table (binky-control / railiance-platform / ops-warden / secrets-engine)
+- link to WP-0026 hygiene (capabilities verify, high-risk, taint)
+- **do not** put new client/tenant commercial secrets under
+  `platform/workloads/` or invent `secret/prod/...` as the production home
+
+**Done when:** wiki + CredentialRouting pointer exist; binky integration doc
+updated to match (or superseded with link).
+
+### T02 — Align binky-control integration plan to production path
+
+```task
+id: WARDEN-WP-0028-T02
+status: done
+priority: high
+```
+
+Done 2026-07-17: `binky-control/integrations/company-email-openbao.md` rewritten
+to `tenants/binky/company-email/imap` + checklist (host confirm + Red provision
+still open).
+
+**Cross-repo: binky-control.** Rewrite
+`integrations/company-email-openbao.md` to use
+`tenants/binky/company-email/imap`, CCR + ops-warden front door as primary,
+secrets-engine as optional exec wrapper on the same path. Confirm non-secret
+IMAP host/port with founder (Blue). Keep Red-lane value provision as the last
+step.
+
+**Done when:** binky doc matches this workplan; checklist items re-opened for
+implementation (T01 in BINKY was "prepared only").
+
+### T03 — Enable `tenants` mount + extend CCR tooling + policy/role
+
+```task
+id: WARDEN-WP-0028-T03
+status: done
+priority: high
+```
+
+Done 2026-07-17: `tenants/` KV v2 mount live; CCR applier allowlist extended;
+CCR-2026-0007 + policy + OIDC role applied; lane-policy `read` / default
+`deny` / agent boundary on tenants path; `docs/workload-kv-access-lanes.md`
+tenant section.
+
+**Cross-repo: railiance-platform (+ OpenBao admin once).**
+
+1. **Enable KV v2 mount** `tenants/` on `bao.coulomb.social` (platform-admin /
+   attended; record non-secret evidence). Versioning + max versions policy
+   aligned with `platform` if practical.
+2. **Extend** `scripts/credential-change.py` allowlist: `mount: tenants` and
+   `kv_path` under `tenants/` (keep existing `platform/workloads/` rules
+   intact for fleet lanes). Update dry-run docs and tests.
+3. Author `CCR-2026-0007` (or next id) for the binky IMAP path; policy HCL on
+   exact data/metadata paths under `tenants/…`; OIDC role (founder group
+   near-term). Dry-run; founder approves; apply metadata only.
+4. Document fields without values; add a **tenant lanes** section to
+   `docs/workload-kv-access-lanes.md` (or sibling doc).
+
+**Done when:** mount exists; CCR tool accepts tenant paths; CCR approved +
+policy/role applied; capabilities-safe positive/negative evidence (no value
+reads for verify).
+
+### T04 — ops-warden catalog + playbook + rotation
+
+```task
+id: WARDEN-WP-0028-T04
+status: done
+priority: high
+```
+
+Done 2026-07-17: catalog draft `binky-company-email-imap` (`risk: high`,
+rotation, path `tenants/binky/...`); playbooks
+`binky-company-email-imap.md` + onboarding. Promote to active after T05.
+
+Add `binky-company-email-imap` to `registry/routing/catalog.yaml`:
+`status: draft` until verify, then promote; `risk: high`; `exec_capable: true`;
+concrete `fetch_command` for primary field; `rotation:` block (re-establish or
+rotate mailbox app-password). Playbook under `wiki/playbooks/`. Scorecard
+rotation coverage satisfied on promote.
+
+**Done when:** `warden route show binky-company-email-imap --json` shows
+expected pointers; after T03+T05, `resolvable: true`.
+
+### T05 — Founder provision (Red) + first scan evidence
+
+```task
+id: WARDEN-WP-0028-T05
+status: wait
+priority: high
+```
+
+**Human-only value path.** Founder provisions IMAP username/password into
+OpenBao via approved tool (`bao kv put … @file` or secrets-engine provision).
+Agents never see the value. Run email-connect read-only scan with
+`warden access … --exec` or secrets-engine exec; store **metadata-only**
+evidence under `binky-control/mailmeta/`.
+
+**Done when:** one successful read-only scan evidence note exists; lane
+promoted active/resolvable; CCR verification evidence complete.
+
+### T06 — Generalize "tenant secret onboarding" playbook
+
+```task
+id: WARDEN-WP-0028-T06
+status: done
+priority: medium
+```
+
+Done 2026-07-17 with T01: `wiki/playbooks/tenant-secret-onboarding.md` +
+CredentialRouting link.
+
+From the binky lane, write a reusable playbook:
+`wiki/playbooks/tenant-secret-onboarding.md` — steps for any new tenant:
+slug, CCR template fields, policy naming, catalog entry, agent boundary,
+rotation block, promotion checklist. Intended for the next Coulomb client
+without redesign.
+
+**Done when:** playbook exists and is linked from CredentialRouting + first
+session notes for tenant repos.
+
+### T07 — secrets-engine alignment decision (record only)
+
+```task
+id: WARDEN-WP-0028-T07
+status: done
+priority: low
+```
+
+Done 2026-07-17: **Decision** — tenant production lanes use `mount: tenants`
+and the CCR path only; secrets-engine may wrap exec delivery against that same
+path later but must not store a second copy under `secret/`. Until stage roles
+gain an approved `tenants/` grant, production path is CCR + ops-warden +
+caller `bao`.
+
+Record a short decision: secrets-engine catalog entries for **tenant**
+production lanes **must** use `mount: tenants` and the same path as the CCR
+(no second copy under `secret/`). Stage roles may need a later extension to
+touch `tenants/` under an approved plan — until then, ops-warden + CCR +
+`bao` as caller are the production path.
+
+**Done when:** decision note in this workplan or `POST /decisions/` + wiki
+pointer; no dual-value storage for the IMAP password.
+
+## Suggested implementation order
+
+1. T01 canon (unblocks everyone)
+2. T02 binky doc alignment
+3. T03 mount + CCR tooling + CCR apply (metadata)
+4. T04 catalog draft
+5. T05 founder provision + verify + promote
+6. T06 generalize playbook
+7. T07 secrets-engine alignment (can parallel after T01)
+
+## Acceptance
+
+- [ ] Documented path convention `tenants/<tenant>/…` used for ≥1 live lane (`binky`)
+- [ ] CCR applier accepts `tenants/` without weakening `platform/workloads/` rules
+- [ ] No secret values in any git/State Hub artifact
+- [ ] Lane is capabilities-verified and agent high-risk safe
+- [ ] email-connect can scan with env injection without printing password
+- [ ] Next tenant can copy the onboarding playbook without redesign
+
+## References
+
+- ADR-0006 recursive multi-tenant identity (`net-kingdom/docs/adr/…`)
+- `net-kingdom/docs/secrets-engine-security-infrastructure-boundary.md`
+- `railiance-platform/docs/workload-kv-access-lanes.md`
+- `railiance-platform/docs/credential-change-approval.md`
+- `binky-control/integrations/company-email-openbao.md` (draft to align)
+- WARDEN-WP-0026 disclosure hygiene (capabilities, high-risk, taint, rotation)
+- WARDEN-WP-0027 Strand B (out of scope here)