ops-warden/workplans/WARDEN-WP-0028-tenant-secret-custody.md

---
id: WARDEN-WP-0028
type: workplan
title: "Tenant secret custody — NetKingdom pattern for client/tenant secrets"
domain: infotech
repo: ops-warden
status: finished
owner: grok
topic_slug: custodian
planning_priority: high
planning_order: 28
created: "2026-07-16"
updated: "2026-07-17"
state_hub_workstream_id: "6b66228a-199a-5b85-b5e2-a7afeaab903b"
---

# WARDEN-WP-0028 — Tenant secret custody (NetKingdom pattern)

## Origin

Founder establishing **binky-control** as the control-plane repo for a Coulomb
client/tenant (Binky-Hedgehog GmbH). Immediate need: store **tenant-specific**
secrets in OpenBao (e.g. company mailbox usernames/passwords for IMAP scan) and
consume them safely via email-connect / agents **without** putting values in
git, State Hub, chat, or workplans.

BINKY-WP-0003-T01 prepared a draft plan
(`binky-control/integrations/company-email-openbao.md`) that targeted
`secret/prod/binky/...` via secrets-engine. That draft is directionally right on
*workflow*, but it does not match a production-ready custody layout on
`bao.coulomb.social`. This workplan resolves the NetKingdom layout and makes
**tenant** secrets first-class — **separate from** platform workload lanes
(`platform/workloads/...`).

## Goal

Define and land a **repeatable, multi-tenant-safe** pattern for:

1. Where tenant secrets live in OpenBao (path + mount convention)
2. Who owns policy, CCR, catalog, and front door
3. How agents and workloads obtain values (sanctioned transports only)
4. First concrete lane: **binky company email IMAP** (unblocks real-world event
   intake for binky-control)

## Architectural facts (as of 2026-07-16)

### Planes (ADR-0006)

| Plane | Role | Example |
| --- | --- | --- |
| Bootstrap | First trust / recovery | unseal, platform-root |
| Platform control | Shared identity, secrets, authz | OpenBao, key-cape, flex-auth, railiance-platform |
| Tenant | Scoped client/workload authority | coulomb products, **binky**, future clients |

**Binky is a tenant**, not platform-root. Tenant secrets must not require
platform-admin for day-to-day read, and tenant operators must not gain
platform-root grants.

### OpenBao mounts (live + planned)

| Mount | Type | Role |
| --- | --- | --- |
| **`platform/`** | KV v2 | **Platform control-plane** secrets: `platform/workloads/...` (fleet CCRs, ESO) and `platform/operators/...` — *not* client/tenant commercial secrets |
| **`secret/`** | KV v2 | secrets-engine stage / pilot paths (`secret/coulomb/...`) — not the tenant home |
| **`tenants/`** | KV v2 (**new**) | **Client/tenant commercial secrets** — explicit multi-tenant namespace |

Live under `platform/workloads/` today: `activity-core/`, `coulomb/`, `forgejo/`,
`issue-core/`, `railiance/`, `reuse/`. Those stay where they are; we do **not**
migrate them into `tenants/` in this workplan.

### Dual front-door systems (both real; don't invent a third)

| Layer | Owns | Entry |
| --- | --- | --- |
| **OpenBao** | Custody, ACL, audit | `bao` CLI / API |
| **railiance-platform CCR** | Approved metadata apply (policy + OIDC role + evidence) | `credential-change-requests/CCR-*.yaml` |
| **ops-warden catalog** | Routing + `warden access` proxy (no custody) | `registry/routing/catalog.yaml` |
| **secrets-engine catalog** | Workflow: decision → plan → provision → exec delivery | `secrets-engine/catalog/*.yaml` |

**Binding rule (ops-warden):** never vend secret values; only route/proxy as the
caller. **Binding rule (WP-0026):** capabilities-safe verify; agents on
`risk: high` use `--out` / `--exec` / `--wrap` only.

### Why not bury tenants under `platform/workloads/`

Earlier draft of this workplan used `platform/workloads/<tenant>/...`. Founder
prefer a **shallower, explicit** namespace. That is sound:

- **No conflict with Vault/OpenBao best practice.** Separate mounts (or
  top-level prefixes) for different security *domains* are common and good.
  `platform/workloads` is an *internal fleet convention* for platform services,
  not an industry mandate for every secret.
- **Clearer mental model:** platform plane vs tenant plane (ADR-0006) maps to
  mount/prefix names operators can see at a glance.
- **Isolation:** ACLs, list boundaries, and future audit queries stay simpler
  when tenant material is not mixed under the same path tree as forgejo/ESO
  lanes.

### Decision (2026-07-17) — use dedicated mount `tenants/`

```text
mount: tenants          # KV v2, enable once (platform-admin)
logical path: <tenant_slug>/<workload>/<bundle>
CLI: tenants/<tenant_slug>/<workload>/<bundle>
API data path: tenants/data/<tenant_slug>/<workload>/<bundle>
```

**Rejected alternatives (kept for archaeology only):**

| Option | Why not |
| --- | --- |
| `secret/prod/binky/...` | `secret/prod/` does not exist; stage model not ready as sole production path |
| `platform/workloads/binky/...` | Too deep; conflates tenant clients with platform workloads |
| `platform/tenants/binky/...` | Acceptable fallback if enabling a new mount is blocked; prefer full `tenants/` mount |

### Tooling debt to clear (not a reason to keep the deep path)

`railiance-platform/scripts/credential-change.py` currently hard-fails unless:

- `openbao.mount == "platform"`
- `openbao.kv_path` starts with `platform/workloads/`

That guard must be **extended** to allow `mount: tenants` and
`kv_path` under `tenants/` (T03). It is an implementation constraint, not a
security reason to force the deep path.

## Recommended path convention (tenant secrets)

```text
tenants/<tenant_slug>/<workload>/<bundle>
```

| Segment | Rules | Binky email example |
| --- | --- | --- |
| `tenant_slug` | Stable kebab slug; company/client id | `binky` |
| `workload` | Capability or system using the secret | `company-email` |
| `bundle` | Single purpose unit (one CCR / one policy) | `imap` |

Full CLI path:

```text
tenants/binky/company-email/imap
```

Suggested fields (names only — never values in git):

| Field | Purpose |
| --- | --- |
| `IMAP_USERNAME` | mailbox login (often the address) |
| `IMAP_PASSWORD` | app password / mailbox password |
| `IMAP_HOST` | optional if not fixed in consumer config |
| `IMAP_PORT` | optional (default 993 in config) |

Non-secret connection facts (provider host if stable) may live in
`binky-control` config / email-connect yaml as **env names + host**, not
passwords.

### Future multi-mailbox / multi-tenant

```text
tenants/<tenant>/mailbox/<mailbox_id>
# e.g. tenants/binky/mailbox/founder-primary
#      tenants/acme/mailbox/billing
```

One CCR + one least-privilege policy **per bundle** (or per mailbox). Do not
grant `tenants/binky/*` list/read to agents — exact-path policies only.

### Risk class

Mailbox credentials and similar client secrets are **`risk: high`** (WP-0026
T04): agent identities get metadata/capabilities only; raw stream refused when
`WARDEN_AGENT_ID` is set.

## Ownership split

| Concern | Owner repo | Artifact |
| --- | --- | --- |
| Tenant business need, IMAP host facts, scan config, queues | **binky-control** | `integrations/*`, mailmeta (metadata only), activity defs |
| OpenBao path, policy HCL, OIDC role, CCR lifecycle | **railiance-platform** | `CCR-YYYY-NNNN-…yaml`, `openbao/policies/…` |
| Routing front door + rotation guidance | **ops-warden** | catalog entry, playbook, `warden access` / `rotate-guide` |
| Exec-time delivery workflow (optional same path) | **secrets-engine** | catalog entry pointing at `tenants/...` (same path; no second copy) |
| Identity groups for tenant operators | **net-kingdom / key-cape** | group e.g. `tenant-binky-operators` (near-term may use `net-kingdom-admins` for founder-only) |
| Mailbox scan consumer | **email-connect** | config with `username_env` / `password_env` only |

## Auth model (near-term vs target)

**Near-term (founder dogfood):** OIDC `netkingdom` role bound to
`groups=net-kingdom-admins` (same pattern as backup/forgejo lanes). Acceptable
while binky is founder-operated only.

**Target (real multi-tenant):**

- IAM group `tenant-binky-operators` (and later per-tenant groups)
- OpenBao OIDC role `binky-company-email-imap-workload-kv-read` bound only to
  that group
- flex-auth check `secret.read:binky-company-email` if pre-approval required
- Agents **never** hold the workload-kv-read policy; use operator wrap / exec
  injection / AppRole for email-connect scanner only

## First worked lane (deliverable sketch)

| Item | Value |
| --- | --- |
| Catalog id | `binky-company-email-imap` |
| CCR | `CCR-2026-0007` (next free) — title: Binky company email IMAP |
| Mount | `tenants` |
| KV path | `tenants/binky/company-email/imap` |
| Policy | `workload-kv-read-binky-company-email-imap` |
| OIDC role | `binky-company-email-imap-workload-kv-read` |
| Consumer | email-connect `scan-mailbox` via env inject |
| Front door | `warden access binky-company-email-imap --out FILE` or `--exec -- …` |
| Provision | **Red lane, founder once** — value via `bao kv put … @file` or secrets-engine provision; never chat |

## Out of scope

- Mass rotation / lockdown machinery (WARDEN-WP-0027 Strand B)
- Sending mail / SMTP from company address (separate lane if needed)
- Migrating all `secret/` mount lanes onto `platform/` (separate secrets-engine
  / railiance work)
- Full key-cape tenant group productization (may be a net-kingdom follow-on)

## Tasks

### T01 — Canon note: tenant secret path + ownership

```task
id: WARDEN-WP-0028-T01
status: done
priority: high
state_hub_task_id: "9982a884-7a50-593e-861e-cc8d2343a0ba"
```

Done 2026-07-17: `wiki/playbooks/tenant-secret-onboarding.md` + CredentialRouting
tenant paragraph; catalog-lane-promotion draft table updated.

Land a short ops-warden wiki page (and one paragraph in
`wiki/CredentialRouting.md`) that freezes:

- mount **`tenants`** + path `tenants/<tenant>/<workload>/<bundle>`
- ownership table (binky-control / railiance-platform / ops-warden / secrets-engine)
- link to WP-0026 hygiene (capabilities verify, high-risk, taint)
- **do not** put new client/tenant commercial secrets under
  `platform/workloads/` or invent `secret/prod/...` as the production home

**Done when:** wiki + CredentialRouting pointer exist; binky integration doc
updated to match (or superseded with link).

### T02 — Align binky-control integration plan to production path

```task
id: WARDEN-WP-0028-T02
status: done
priority: high
state_hub_task_id: "d7d7ee9d-2ecf-5492-8a13-0b747d899456"
```

Done 2026-07-17: `binky-control/integrations/company-email-openbao.md` rewritten
to `tenants/binky/company-email/imap` + checklist (host confirm + Red provision
still open).

**Cross-repo: binky-control.** Rewrite
`integrations/company-email-openbao.md` to use
`tenants/binky/company-email/imap`, CCR + ops-warden front door as primary,
secrets-engine as optional exec wrapper on the same path. Confirm non-secret
IMAP host/port with founder (Blue). Keep Red-lane value provision as the last
step.

**Done when:** binky doc matches this workplan; checklist items re-opened for
implementation (T01 in BINKY was "prepared only").

### T03 — Enable `tenants` mount + extend CCR tooling + policy/role

```task
id: WARDEN-WP-0028-T03
status: done
priority: high
state_hub_task_id: "e0b675ef-9c08-5f89-8f13-ef4249ca2364"
```

Done 2026-07-17: `tenants/` KV v2 mount live; CCR applier allowlist extended;
CCR-2026-0007 + policy + OIDC role applied; lane-policy `read` / default
`deny` / agent boundary on tenants path; `docs/workload-kv-access-lanes.md`
tenant section.

**Cross-repo: railiance-platform (+ OpenBao admin once).**

1. **Enable KV v2 mount** `tenants/` on `bao.coulomb.social` (platform-admin /
   attended; record non-secret evidence). Versioning + max versions policy
   aligned with `platform` if practical.
2. **Extend** `scripts/credential-change.py` allowlist: `mount: tenants` and
   `kv_path` under `tenants/` (keep existing `platform/workloads/` rules
   intact for fleet lanes). Update dry-run docs and tests.
3. Author `CCR-2026-0007` (or next id) for the binky IMAP path; policy HCL on
   exact data/metadata paths under `tenants/…`; OIDC role (founder group
   near-term). Dry-run; founder approves; apply metadata only.
4. Document fields without values; add a **tenant lanes** section to
   `docs/workload-kv-access-lanes.md` (or sibling doc).

**Done when:** mount exists; CCR tool accepts tenant paths; CCR approved +
policy/role applied; capabilities-safe positive/negative evidence (no value
reads for verify).

### T04 — ops-warden catalog + playbook + rotation

```task
id: WARDEN-WP-0028-T04
status: done
priority: high
state_hub_task_id: "9405b13d-4045-519b-8e3f-79a891323397"
```

Done 2026-07-17: catalog draft `binky-company-email-imap` (`risk: high`,
rotation, path `tenants/binky/...`); playbooks
`binky-company-email-imap.md` + onboarding. Promote to active after T05.

Add `binky-company-email-imap` to `registry/routing/catalog.yaml`:
`status: draft` until verify, then promote; `risk: high`; `exec_capable: true`;
concrete `fetch_command` for primary field; `rotation:` block (re-establish or
rotate mailbox app-password). Playbook under `wiki/playbooks/`. Scorecard
rotation coverage satisfied on promote.

**Done when:** `warden route show binky-company-email-imap --json` shows
expected pointers; after T03+T05, `resolvable: true`.

### T05 — Founder provision (Red) + first scan evidence

```task
id: WARDEN-WP-0028-T05
status: done
priority: high
state_hub_task_id: "f17bba95-bc53-5c2a-8b44-7df9e42b2341"
```

Done 2026-07-17: Founder provisioned via OpenBao UI (KV version 2; not
placeholder). Capabilities-safe verify pass; catalog + CCR promoted
active/resolvable. Optional follow-up: first email-connect read-only scan into
`binky-control/mailmeta/` (OperatingRhythm), not a custody blocker.

**Human-only value path.** Founder provisions IMAP username/password into
OpenBao via approved tool (`bao kv put … @file` or secrets-engine provision).
Agents never see the value. Run email-connect read-only scan with
`warden access … --exec` or secrets-engine exec; store **metadata-only**
evidence under `binky-control/mailmeta/`.

**Done when:** one successful read-only scan evidence note exists; lane
promoted active/resolvable; CCR verification evidence complete.

### T06 — Generalize "tenant secret onboarding" playbook

```task
id: WARDEN-WP-0028-T06
status: done
priority: medium
state_hub_task_id: "76bd01a0-1d03-5ff9-8f59-a1b44763979d"
```

Done 2026-07-17 with T01: `wiki/playbooks/tenant-secret-onboarding.md` +
CredentialRouting link.

From the binky lane, write a reusable playbook:
`wiki/playbooks/tenant-secret-onboarding.md` — steps for any new tenant:
slug, CCR template fields, policy naming, catalog entry, agent boundary,
rotation block, promotion checklist. Intended for the next Coulomb client
without redesign.

**Done when:** playbook exists and is linked from CredentialRouting + first
session notes for tenant repos.

### T07 — secrets-engine alignment decision (record only)

```task
id: WARDEN-WP-0028-T07
status: done
priority: low
state_hub_task_id: "46c8f8a9-3bbe-568f-8a45-07b690874273"
```

Done 2026-07-17: **Decision** — tenant production lanes use `mount: tenants`
and the CCR path only; secrets-engine may wrap exec delivery against that same
path later but must not store a second copy under `secret/`. Until stage roles
gain an approved `tenants/` grant, production path is CCR + ops-warden +
caller `bao`.

Record a short decision: secrets-engine catalog entries for **tenant**
production lanes **must** use `mount: tenants` and the same path as the CCR
(no second copy under `secret/`). Stage roles may need a later extension to
touch `tenants/` under an approved plan — until then, ops-warden + CCR +
`bao` as caller are the production path.

**Done when:** decision note in this workplan or `POST /decisions/` + wiki
pointer; no dual-value storage for the IMAP password.

## Suggested implementation order

1. T01 canon (unblocks everyone)
2. T02 binky doc alignment
3. T03 mount + CCR tooling + CCR apply (metadata)
4. T04 catalog draft
5. T05 founder provision + verify + promote
6. T06 generalize playbook
7. T07 secrets-engine alignment (can parallel after T01)

## Acceptance

- [ ] Documented path convention `tenants/<tenant>/…` used for ≥1 live lane (`binky`)
- [ ] CCR applier accepts `tenants/` without weakening `platform/workloads/` rules
- [ ] No secret values in any git/State Hub artifact
- [ ] Lane is capabilities-verified and agent high-risk safe
- [ ] email-connect can scan with env injection without printing password
- [ ] Next tenant can copy the onboarding playbook without redesign

## References

- ADR-0006 recursive multi-tenant identity (`net-kingdom/docs/adr/…`)
- `net-kingdom/docs/secrets-engine-security-infrastructure-boundary.md`
- `railiance-platform/docs/workload-kv-access-lanes.md`
- `railiance-platform/docs/credential-change-approval.md`
- `binky-control/integrations/company-email-openbao.md` (draft to align)
- WARDEN-WP-0026 disclosure hygiene (capabilities, high-risk, taint, rotation)
- WARDEN-WP-0027 Strand B (out of scope here)