From d7d1a490f9f4fff6900ae63867f79f69573272ef Mon Sep 17 00:00:00 2001 From: codex Date: Mon, 17 Aug 2026 11:26:07 +0200 Subject: [PATCH] =?UTF-8?q?ADR-008:=20multi-tenancy=20model=20=E2=80=94=20?= =?UTF-8?q?four=20planes,=20one=20isolation=20ladder=20(proposed)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The estate has built multi-tenancy across five documents that each cover a slice and do not compose. This records the whole model and names what is missing. Core framing: multi-tenancy here is four orthogonal planes — identity, authorization, data isolation, placement — not one property. Identity and authorization are ratified and solid. Data isolation is stated but unverified. Placement is owned by nobody and is precedent-by-accident. Three findings that motivated the draft: - R0 -> R1 -> R2 does not improve tenant isolation at all. Those rungs buy consumer isolation and capacity. Only R3 makes the tenant boundary structural. A plan tier selling R2 as isolation would be false. - No fleet mechanism verifies that any consumer actually enforces the tenant boundary it is obliged to enforce. A missing WHERE tenant_id would be a cross-tenant breach that no probe fails and no log shows as an error. Highest-severity gap; needs an appointed owner. - platform-pg holds roughly six consumers (100 max_connections, 14 each) and memory likely binds before connections do. Two are provisioned. The runway is shorter than the ladder implies. Also reconciles two already-ratified and contradictory defaults: instance-per-client for business apps, pooled for platform services, with no rule for telling a new service which it is. Proposed, not ratified. Carries a ratification path so it does not join the shared-platform-relational-storage draft in limbo. Co-Authored-By: Claude Opus 5 --- .../adr-008-multi-tenancy-model.md | 382 ++++++++++++++++++ 1 file changed, 382 insertions(+) create mode 100644 canon/architecture/adr-008-multi-tenancy-model.md diff --git a/canon/architecture/adr-008-multi-tenancy-model.md b/canon/architecture/adr-008-multi-tenancy-model.md new file mode 100644 index 0000000..5cb4817 --- /dev/null +++ b/canon/architecture/adr-008-multi-tenancy-model.md @@ -0,0 +1,382 @@ +--- +id: ADR-008 +type: architecture-decision-record +title: "Multi-Tenancy Model: Four Planes, One Isolation Ladder" +status: proposed +decided_by: Bernd Worsch +date: "2026-08-17" +tags: ["architecture", "multi-tenancy", "isolation", "placement", "tenant-engine", "flex-auth", "rapp-postgres", "scaling"] +--- + +# ADR-008: Multi-Tenancy Model — Four Planes, One Isolation Ladder + +## Status + +**Proposed.** Drafted 2026-08-17 at the operator's request. This document +decides things currently owned by `tenant-engine`, `flex-auth`, +`rapp-postgres`, `adaptive-pricing`, and `railiance-platform`. It must be +reviewed by those owners before ratification; §12 lists what each is being +asked to accept. Until ratified it governs nothing, and the existing +per-repo documents remain authoritative for their own repos. + +## Context + +The estate has been building multi-tenancy for months and has never written +down what it is building. Five documents each cover a slice: + +| Document | Covers | Status | +|---|---|---| +| `iam-profile_v0.3` (NetKingdom) | Tenant identifier shape, `tenant_roles` claim, staleness rules | Ratified | +| `tenant-engine-boundary-contract_v0.1` (NetKingdom) | Who owns tenant records, roles, plan assignment | Ratified | +| `business-app-service-contract_v0.1` §1 (Custodian) | Business apps: instance-per-client, tenant-keyed data | Ratified | +| `rapp-postgres/docs/adr/ADR-0001` | Consumer + tenant isolation in PostgreSQL | Proposed, governs one repo | +| `rapp-postgres/docs/canon-drafts/shared-platform-relational-storage_v0.1` | The stacked-boundary gap | Routed 2026-08-10, **still unratified** | + +The slices do not compose, and three specific failures follow. + +**The gap has been diagnosed once and the fix stalled.** The +shared-platform-relational-storage draft was written precisely to fill this +hole. It has sat unratified for a week and appears in neither canon +directory. Producing a second orphaned draft would make things worse, not +better; §13 therefore attaches a ratification path to this one. + +**Placement is owned by nobody.** Whether a workload gets a dedicated +PostgreSQL cluster or a shared one is currently precedent-by-accident: +`user-engine-pg` and `target-revenue-pg` are dedicated; `apps-pg`, +`net-kingdom-pg`, `platform-pg`, `state-hub-db` and `forgejo-db` are shared. +Both patterns are live, neither is written down, and each new service copies +whichever neighbour it happened to consult. `tenant-engine` raised this with +`railiance-platform` on 2026-08-16 and it is unanswered. + +**Two contradictory tenancy defaults are already ratified.** +`business-app-service-contract` §1.2 makes *instance-per-client* the default +for business apps. Platform services such as `audit-core` and `tenant-engine` +pool all tenants into one database. Both are correct for their case, but +nothing states which shape a new service takes, and the estate has no +definition distinguishing "business app" from "platform service". A new +repo can read either document and be conformant with one while violating the +other. + +## Decision + +### 1. Multi-tenancy is four orthogonal planes, not one property + +The recurring confusion is that "is this multi-tenant?" is treated as a +single question. It is four, and they are independent: + +| Plane | Question | Owner | State | +|---|---|---|---| +| **Identity** | How is a tenant named and validated? | `tenant-engine` / IAM Profile | Ratified, solid | +| **Authorization** | How is a request bound to the tenants it may act for? | `flex-auth` | Contract ratified | +| **Data isolation** | Where does the boundary between two tenants' rows sit? | The consuming service | Stated in ADR-0001, unverified fleet-wide | +| **Placement** | Which physical substrate holds a tenant's data? | *unowned* | Precedent only | + +Treating them as one is not a theoretical tidiness problem; it is producing +concrete errors today. + +`rapp-postgres`'s `PostgresConsumer` declaration carries a field +`tenantIsolation: consumer-service-boundary`. That is a **data-isolation** +fact recorded in a **placement** artifact. It reads as though the storage +platform enforces something. It does not, and ADR-0001 §4 is explicit that it +must not pretend to. + +The "dedicated versus shared" argument routinely mixes placement (a capacity +and blast-radius decision) with data isolation (a correctness decision). +`tenant-engine` drew the right distinction unprompted on 2026-08-16 — +sharing a *cluster* is capacity and cheap to undo; sharing a *database* +entangles schemas and is not — and that distinction is adopted here as +binding. + +**Decision:** every document, declaration, and plan tier that says +"isolation" MUST name which plane it means. + +### 2. The identity and authorization planes are settled; this ADR does not reopen them + +Recorded here only so the model is complete and so no future document +re-derives them: + +- Tenant identifiers are `tenant::` per ADR-0013, with + `tenant:platform` and `tenant:coulomb` reserved and ungrouped. `tenant-engine` + owns which values exist; the IAM Profile owns the wire format. +- Capability roles `PLTF`/`IAM`/`VEN`/`CUS` are non-exclusive and are a + separate fact from grouping. `tenant-engine` is the source of truth; the + `tenant_roles` token claim is a point-in-time cache that MUST NOT be trusted + for privileged, destructive, credential-vending, or `aal2`-class decisions. +- `flex-auth` makes authorization decisions; `tenant-engine` is a data source + and makes none. +- Storage platforms MUST NOT mint, parse, or authorize on tenant identifiers. + To them these are opaque strings. + +### 3. Data isolation is enforced by the consuming service, and the platform must never imply otherwise + +This is ADR-0001 §4 promoted to fleet scope. + +A consumer connects through a pooled role shared across all of its tenants. +The database therefore sees one principal, not the requesting tenant, and has +no trustworthy basis on which to filter. Any database-level tenant filter +would depend on the consumer correctly setting a session variable — which is +the application-level check the consumer already owes, relocated somewhere it +*looks* like a database guarantee without being one. + +- **Mandatory:** consumer-owned data is tenant-keyed from the first + migration, even where the consumer serves exactly one tenant today. +- **Mandatory:** the consumer binds the authenticated identity to the tenants + it may act for, at its own service boundary. +- **Available, opt-in:** row-level security keyed on a session GUC set at pool + checkout, as defence in depth behind a correct service-side check — never + as a substitute for one. +- **Prohibited:** describing the opt-in layer as a platform guarantee. + +**This is the model's largest residual risk and §9 does not soften it.** + +### 4. The isolation ladder + +Placement decisions are expressed as rungs. Each rung states what it +actually buys, and — critically — what it does *not* change. + +| Rung | Shape | Data-isolation guarantee | Blast radius of a leaked runtime credential | Live occupants | +|---|---|---|---|---| +| **R0** | Shared tables, tenant column | App-enforced | All tenants of that consumer | Inside every consumer today | +| **R1** | Database per consumer, shared cluster | App-enforced *(unchanged)* | All tenants of that consumer | `audit-core`, `tenant-engine` on `platform-pg` | +| **R2** | Dedicated cluster per consumer | App-enforced *(unchanged)* | All tenants of that consumer | `user-engine-pg`, `target-revenue-pg` | +| **R3** | Dedicated cluster per tenant | Structural | One tenant | Business apps per `business-app-service-contract` §1.2 | +| **R4** | R3 + separate region/jurisdiction | Structural + residency | One tenant | None | + +The column that matters is the third one. **R0 → R1 → R2 does not improve +tenant isolation at all.** Those rungs buy consumer isolation, capacity +predictability, independent backup retention, and a smaller operational blast +radius. The tenant boundary stays exactly where §3 puts it: in application +code. Only R3 makes it structural. + +**Decision:** R1 is the default for platform services. R3 is the default for +client-facing business apps, as already ratified. A service that is unsure +which it is MUST resolve that before choosing a rung (§12.4). + +### 5. Movement between rungs must be an operational change, not a rebuild + +Adopted from `tenant-engine`'s TEN-WP-0009 design, which got this right: + +- Connect by injected credential only. No cluster, host, namespace, or + database name anywhere in source. +- Own a whole database, never tables inside someone else's. Sharing a cluster + is reversible; sharing a database makes relocation a schema merge. +- Idempotent schema creation, so a fresh target comes up correct. +- No cross-database joins or co-location assumptions. + +A service meeting these moves R1 → R2 by: create target, dump, restore, swap +the credential reference, restart. No rebuild, no release, no code review. + +**Decision:** these four properties are mandatory at every rung, including +R3. A business app at R3 that hard-codes its database name has forfeited +consolidation, which §1.3 of the business-app contract exists to preserve. + +### 6. Placement triggers are recorded in advance + +A rung change decided under pressure is decided badly. Each service records, +at provisioning time, what would move it up: + +- Noisy neighbour on a latency-critical path +- A compliance or residency requirement +- A plan tier that sells isolation (see §7) +- Backup retention or RTO no longer fitting +- Connection or memory ceiling reached (see §9) + +**Decision:** placement ownership is `railiance-platform`'s. This ADR +proposes the ladder and the triggers; it does not appoint per-workload +placement, which remains that repo's call and is the substance of the +2026-08-16 request still open there. + +### 7. Isolation as a product property — the commercial hazard + +Plan tiers that "sell isolation" are anticipated. The hazard is precise and +worth stating before revenue depends on it: + +**Selling a move from R1 to R2 as stronger tenant isolation would be false.** +Per §4 it changes nothing about the tenant boundary. A customer buying +"isolated" and receiving R2 gets better noisy-neighbour behaviour and an +independent restore path — real value, and not what they think they bought. +Only R3 changes the answer to "can another tenant's data reach mine". + +**Decision:** any plan tier naming isolation MUST cite the rung it delivers +and MUST NOT describe a lower rung in R3's language. `adaptive-pricing` owns +plan terms and `tenant-engine` owns which plan a tenant holds; neither +currently maps a tier to a rung, and §12.5 asks them to. + +## 8. Alternatives considered + +**Schema-per-consumer in one database.** Cheaper on connections, allows +cross-consumer queries. *Rejected on catalog leakage:* `pg_catalog` is +readable per-database, so every co-resident can enumerate every other's +table and column names regardless of `GRANT`s. Table names alone leak +business structure and no privilege setting removes that. + +**Row-level security as the consumer boundary.** *Rejected:* puts a +correctness-critical boundary inside query predicates, where one missing +policy or a `SECURITY DEFINER` function silently removes it. Retained as +opt-in defence in depth for the tenant boundary only (§3). + +**Platform-enforced tenant RLS by default.** *Rejected:* with a pooled +per-consumer role it depends on the consumer setting a session variable +correctly, producing the appearance of a database guarantee over what is +still an application check. A control that misrepresents where it is enforced +is worse than an honest absence. + +**Dedicated instance per consumer as the default (R2 everywhere).** +*Rejected as a default* on operational cost — every instance multiplies +backup, upgrade, and monitoring surface — but it is the honest answer for a +consumer that cannot accept §9's residual risk, and it stays available as an +escalation rather than being argued away. + +**Instance-per-tenant everywhere (R3 as the fleet default).** *Rejected:* at +the estate's tenant grouping taxonomy — which includes `consumer` (private +individuals) and `family` — R3 per tenant is economically impossible. The +taxonomy itself is evidence that pooling is required; you cannot run a +cluster per private individual. + +**One tenancy model for the whole estate.** *Rejected:* the ratified +business-app contract already commits client-facing apps to R3, and platform +services genuinely cannot follow it. The honest answer is two defaults with a +stated rule for which applies, not a single default that one half violates. + +## 9. Challenge against outside practice + +Held against the common industry framing (AWS's silo/pool/bridge model, +cell-based architectures), the estate lands as follows. + +**Where the model is stronger than typical.** Most SaaS multi-tenancy +literature models exactly one boundary — tenant-to-tenant. This estate has +**two stacked boundaries**: platform-service-to-platform-service (the +consumer boundary) and tenant-to-tenant (the princedom boundary). Naming them +separately and refusing to enforce both with one mechanism is a genuine +strength and is uncommon. + +The refusal to overstate the guarantee is also better than typical. Vendors +routinely describe pool-model RLS as tenant isolation. ADR-0001 §5 states the +blast radius in concrete terms instead, which is the practice worth keeping. + +**Where the model is weaker than typical, and this is the finding that +matters.** The pool model's standard mitigation is a *verified* enforcement +layer — a shared data-access library, or RLS, that every service is +demonstrably routed through. This estate has neither. §3 obliges every +consumer to enforce the tenant boundary in its own code, and there is +currently **no fleet mechanism that verifies any consumer actually does**. + +`audit-core` has AUDIT-WP-0004-T03; `tenant-engine` has its own checks; +`rapp-postgres` runs 15 adversarial probes against the *consumer* boundary +and none against the tenant boundary inside a consumer. A single missing +`WHERE tenant_id = ?` in any consumer is a cross-tenant data breach that no +platform control would catch, no probe would fail, and no log would show as +an error. This is the highest-severity known gap in the model and §12.3 +proposes an owner for it. + +**Where the estate has quietly adopted a good pattern without naming it.** +Short-lived leased credentials from OpenBao, re-read at connection checkout, +with overlap-first rotation, is stronger than the industry norm of a +long-lived per-service secret. It bounds §9's residual risk by credential +lifetime. It deserves to be stated as a tenancy control, not just a +credential-management one. + +**Cell-based architecture is the unexplored option.** R2 and R3 partition by +consumer or by tenant. Neither partitions by *cell* — a fixed-size slice of +infrastructure serving a bounded set of tenants, sized so that one cell's +failure is survivable and cell count scales linearly. Given the estate spans +railiance, NetKingdom, HelixForge and Coulomb with tenants layered on top, +cells may fit better than per-tenant clusters at the point where R1 runs out. +Not proposed here; flagged in §12.6 as worth evaluating before the ceiling in +§10 is hit rather than after. + +## 10. Scaling demands — the ceiling is closer than it looks + +Measured against the live `platform-pg` specification, not estimated. + +``` +instances: 1 (no HA; single-node rail) +max_connections: 100 +memory limit: 1Gi +cpu limit: 1 +storage: 20Gi +per consumer: 14 connections (12 runtime + 2 migration) +``` + +**Connection ceiling: roughly six consumers.** At 14 connections per consumer +declaration, seven consumers request 98 of 100 connections, before CNPG's +instance manager, the metrics exporter, and superuser-reserved slots. The +practical ceiling is about **six**. Two are provisioned today +(`audit-core`, `tenant-engine`) plus the isolation probe. **We are at +roughly a third of capacity, and the third consumer request will not feel +like a scaling event.** + +**Memory is likely to bind before connections do.** 100 PostgreSQL backends +against a 1Gi limit is roughly 10MB per backend for all of shared buffers, +work_mem, and per-backend overhead. Connection exhaustion produces a clear +error; memory pressure produces OOM kills and degraded latency across every +co-resident consumer at once. + +**Restore time couples all consumers.** Physical backup is instance-wide, so +restoring one consumer means restoring the whole instance to scratch, taking +a logical dump, and importing. A consumer's RTO is therefore a function of +**total instance size**, not its own. One large consumer silently degrades +every co-resident's recovery promise — and per the storage draft §7.3, a +consumer must not promise retention or recovery its platform cannot back. + +**`instances: 1` means no tenant on `platform-pg` has HA.** Deliberate on the +current rail and honestly documented, but it means every shared consumer's +availability story is "restart recovery", not high availability. A plan tier +promising uptime cannot be satisfied at R1 as currently built. + +**What this implies.** The ladder is not a long-term growth path on the +present substrate — it is a two-to-four-consumer runway. The triggers in §6 +must be monitored, not merely recorded, and connection and memory +utilisation belong on the alert list *before* the next consumer is accepted. + +## 11. Consequences + +- The estate gains one vocabulary for a conversation currently held in four + incompatible ones. +- `tenantIsolation` in `PostgresConsumer` is revealed as a mislabelled field + and should be renamed or moved once this ADR settles (§12.1). +- Two ratified defaults (R1 for platform services, R3 for business apps) are + reconciled by scope rather than by one overriding the other, at the cost of + needing a rule for classifying a new service. +- The unverified tenant boundary (§9) becomes a named, owned gap rather than + an assumption distributed across every repo. +- The `platform-pg` ceiling becomes a planning input rather than a discovery. +- Nothing in this ADR changes a running system. It is descriptive of what is + built, plus five decisions that need owner sign-off. + +## 12. Open questions for ratification + +1. **`tenantIsolation` field** — `rapp-postgres`: rename to name its plane + (`tenantBoundaryEnforcement`?), or move it out of the storage declaration + entirely. +2. **Placement ownership** — `railiance-platform`: accept the ladder and + triggers, and appoint a recorded placement owner per workload. This is the + 2026-08-16 request, still open. +3. **Tenant-boundary conformance** — *owner needed*. Who builds and runs a + fleet check that a consumer cannot leak tenant A's rows to tenant B? + `audit-core` and `tenant-engine` have both declined fleet-scope work on + correct boundary reasoning, so this needs appointing rather than + volunteering. Highest-severity gap in the model. +4. **Business app vs platform service** — *Custodian canon*: a definition, so + a new repo knows whether §1.2's R3 default or this ADR's R1 default + applies to it. +5. **Plan tier → rung mapping** — `adaptive-pricing` and `tenant-engine`: + no tier may name isolation without citing a rung (§7). +6. **Cell-based partitioning** — evaluate before the §10 ceiling is reached. +7. **Engine neutrality** — whether the ladder should be stated + engine-neutrally or admit that R1's guarantee rests on a PostgreSQL + property (per-database catalogs). Inherited unresolved from the storage + draft. + +## 13. Ratification path + +The predecessor draft stalled because it was routed without one. This ADR: + +1. Is reviewed by `tenant-engine`, `flex-auth`, `rapp-postgres`, + `railiance-platform`, and `adaptive-pricing` against §12. +2. On acceptance, **supersedes** the routing of + `rapp-postgres/docs/canon-drafts/shared-platform-relational-storage_v0.1-draft.md`, + whose §§3–8 are absorbed above. That draft should then be withdrawn rather + than left pending, so the estate does not carry two overlapping proposals. +3. On acceptance, `rapp-postgres/docs/adr/ADR-0001` moves from `proposed` to + `accepted` and is annotated as the PostgreSQL implementation of this model.