the-custodian/canon/architecture/adr-008-multi-tenancy-model.md

id	type	title	status	decided_by	date	tags
ADR-008	architecture-decision-record	Multi-Tenancy Model: Four Planes, One Isolation Ladder	proposed	Bernd Worsch	2026-08-17	architecture multi-tenancy isolation placement tenant-engine flex-auth rapp-postgres scaling

ADR-008: Multi-Tenancy Model — Four Planes, One Isolation Ladder

Status

Proposed. Drafted 2026-08-17 at the operator's request. This document decides things currently owned by tenant-engine, flex-auth, rapp-postgres, adaptive-pricing, and railiance-platform. It must be reviewed by those owners before ratification; §12 lists what each is being asked to accept. Until ratified it governs nothing, and the existing per-repo documents remain authoritative for their own repos.

Context

The estate has been building multi-tenancy for months and has never written down what it is building. Five documents each cover a slice:

Document	Covers	Status
iam-profile_v0.3 (NetKingdom)	Tenant identifier shape, tenant_roles claim, staleness rules	Ratified
tenant-engine-boundary-contract_v0.1 (NetKingdom)	Who owns tenant records, roles, plan assignment	Ratified
business-app-service-contract_v0.1 §1 (Custodian)	Business apps: instance-per-client, tenant-keyed data	Ratified
rapp-postgres/docs/adr/ADR-0001	Consumer + tenant isolation in PostgreSQL	Proposed, governs one repo
rapp-postgres/docs/canon-drafts/shared-platform-relational-storage_v0.1	The stacked-boundary gap	Routed 2026-08-10, still unratified

The slices do not compose, and three specific failures follow.

The gap has been diagnosed once and the fix stalled. The shared-platform-relational-storage draft was written precisely to fill this hole. It has sat unratified for a week and appears in neither canon directory. Producing a second orphaned draft would make things worse, not better; §13 therefore attaches a ratification path to this one.

Placement is owned by nobody. Whether a workload gets a dedicated PostgreSQL cluster or a shared one is currently precedent-by-accident: user-engine-pg and target-revenue-pg are dedicated; apps-pg, net-kingdom-pg, platform-pg, state-hub-db and forgejo-db are shared. Both patterns are live, neither is written down, and each new service copies whichever neighbour it happened to consult. tenant-engine raised this with railiance-platform on 2026-08-16 and it is unanswered.

Two contradictory tenancy defaults are already ratified. business-app-service-contract §1.2 makes instance-per-client the default for business apps. Platform services such as audit-core and tenant-engine pool all tenants into one database. Both are correct for their case, but nothing states which shape a new service takes, and the estate has no definition distinguishing "business app" from "platform service". A new repo can read either document and be conformant with one while violating the other.

Decision

1. Multi-tenancy is four orthogonal planes, not one property

The recurring confusion is that "is this multi-tenant?" is treated as a single question. It is four, and they are independent:

Plane	Question	Owner	State
Identity	How is a tenant named and validated?	tenant-engine / IAM Profile	Ratified, solid
Authorization	How is a request bound to the tenants it may act for?	flex-auth	Contract ratified
Data isolation	Where does the boundary between two tenants' rows sit?	The consuming service	Stated in ADR-0001, unverified fleet-wide
Placement	Which physical substrate holds a tenant's data?	unowned	Precedent only

Treating them as one is not a theoretical tidiness problem; it is producing concrete errors today.

rapp-postgres's PostgresConsumer declaration carries a field tenantIsolation: consumer-service-boundary. That is a data-isolation fact recorded in a placement artifact. It reads as though the storage platform enforces something. It does not, and ADR-0001 §4 is explicit that it must not pretend to.

The "dedicated versus shared" argument routinely mixes placement (a capacity and blast-radius decision) with data isolation (a correctness decision). tenant-engine drew the right distinction unprompted on 2026-08-16 — sharing a cluster is capacity and cheap to undo; sharing a database entangles schemas and is not — and that distinction is adopted here as binding.

Decision: every document, declaration, and plan tier that says "isolation" MUST name which plane it means.

2. The identity and authorization planes are settled; this ADR does not reopen them

Recorded here only so the model is complete and so no future document re-derives them:

• Tenant identifiers are tenant:<grouping>:<name> per ADR-0013, with tenant:platform and tenant:coulomb reserved and ungrouped. tenant-engine owns which values exist; the IAM Profile owns the wire format.
• Capability roles PLTF/IAM/VEN/CUS are non-exclusive and are a separate fact from grouping. tenant-engine is the source of truth; the tenant_roles token claim is a point-in-time cache that MUST NOT be trusted for privileged, destructive, credential-vending, or aal2-class decisions.
• flex-auth makes authorization decisions; tenant-engine is a data source and makes none.
• Storage platforms MUST NOT mint, parse, or authorize on tenant identifiers. To them these are opaque strings.

3. Data isolation is enforced by the consuming service, and the platform must never imply otherwise

This is ADR-0001 §4 promoted to fleet scope.

A consumer connects through a pooled role shared across all of its tenants. The database therefore sees one principal, not the requesting tenant, and has no trustworthy basis on which to filter. Any database-level tenant filter would depend on the consumer correctly setting a session variable — which is the application-level check the consumer already owes, relocated somewhere it looks like a database guarantee without being one.

• Mandatory: consumer-owned data is tenant-keyed from the first migration, even where the consumer serves exactly one tenant today.
• Mandatory: the consumer binds the authenticated identity to the tenants it may act for, at its own service boundary.
• Available, opt-in: row-level security keyed on a session GUC set at pool checkout, as defence in depth behind a correct service-side check — never as a substitute for one.
• Prohibited: describing the opt-in layer as a platform guarantee.

This is the model's largest residual risk and §9 does not soften it.

4. The isolation ladder

Placement decisions are expressed as rungs. Each rung states what it actually buys, and — critically — what it does not change.

Rung	Shape	Data-isolation guarantee	Blast radius of a leaked runtime credential	Live occupants
R0	Shared tables, tenant column	App-enforced	All tenants of that consumer	Inside every consumer today
R1	Database per consumer, shared cluster	App-enforced (unchanged)	All tenants of that consumer	audit-core, tenant-engine on platform-pg
R2	Dedicated cluster per consumer	App-enforced (unchanged)	All tenants of that consumer	user-engine-pg, target-revenue-pg
R3	Dedicated cluster per tenant	Structural	One tenant	Business apps per business-app-service-contract §1.2
R4	R3 + separate region/jurisdiction	Structural + residency	One tenant	None

The column that matters is the third one. R0 → R1 → R2 does not improve tenant isolation at all. Those rungs buy consumer isolation, capacity predictability, independent backup retention, and a smaller operational blast radius. The tenant boundary stays exactly where §3 puts it: in application code. Only R3 makes it structural.

Decision: R1 is the default for platform services. R3 is the default for client-facing business apps, as already ratified. A service that is unsure which it is MUST resolve that before choosing a rung (§12.4).

5. Movement between rungs must be an operational change, not a rebuild

Adopted from tenant-engine's TEN-WP-0009 design, which got this right:

• Connect by injected credential only. No cluster, host, namespace, or database name anywhere in source.
• Own a whole database, never tables inside someone else's. Sharing a cluster is reversible; sharing a database makes relocation a schema merge.
• Idempotent schema creation, so a fresh target comes up correct.
• No cross-database joins or co-location assumptions.

A service meeting these moves R1 → R2 by: create target, dump, restore, swap the credential reference, restart. No rebuild, no release, no code review.

Decision: these four properties are mandatory at every rung, including R3. A business app at R3 that hard-codes its database name has forfeited consolidation, which §1.3 of the business-app contract exists to preserve.

6. Placement triggers are recorded in advance

A rung change decided under pressure is decided badly. Each service records, at provisioning time, what would move it up:

• Noisy neighbour on a latency-critical path
• A compliance or residency requirement
• A plan tier that sells isolation (see §7)
• Backup retention or RTO no longer fitting
• Connection or memory ceiling reached (see §9)

Decision: placement ownership is railiance-platform's. This ADR proposes the ladder and the triggers; it does not appoint per-workload placement, which remains that repo's call and is the substance of the 2026-08-16 request still open there.

7. Isolation as a product property — the commercial hazard

Plan tiers that "sell isolation" are anticipated. The hazard is precise and worth stating before revenue depends on it:

Selling a move from R1 to R2 as stronger tenant isolation would be false. Per §4 it changes nothing about the tenant boundary. A customer buying "isolated" and receiving R2 gets better noisy-neighbour behaviour and an independent restore path — real value, and not what they think they bought. Only R3 changes the answer to "can another tenant's data reach mine".

Decision: any plan tier naming isolation MUST cite the rung it delivers and MUST NOT describe a lower rung in R3's language. adaptive-pricing owns plan terms and tenant-engine owns which plan a tenant holds; neither currently maps a tier to a rung, and §12.5 asks them to.

8. Alternatives considered

Schema-per-consumer in one database. Cheaper on connections, allows cross-consumer queries. Rejected on catalog leakage: pg_catalog is readable per-database, so every co-resident can enumerate every other's table and column names regardless of GRANTs. Table names alone leak business structure and no privilege setting removes that.

Row-level security as the consumer boundary. Rejected: puts a correctness-critical boundary inside query predicates, where one missing policy or a SECURITY DEFINER function silently removes it. Retained as opt-in defence in depth for the tenant boundary only (§3).

Platform-enforced tenant RLS by default. Rejected: with a pooled per-consumer role it depends on the consumer setting a session variable correctly, producing the appearance of a database guarantee over what is still an application check. A control that misrepresents where it is enforced is worse than an honest absence.

Dedicated instance per consumer as the default (R2 everywhere). Rejected as a default on operational cost — every instance multiplies backup, upgrade, and monitoring surface — but it is the honest answer for a consumer that cannot accept §9's residual risk, and it stays available as an escalation rather than being argued away.

Instance-per-tenant everywhere (R3 as the fleet default). Rejected: at the estate's tenant grouping taxonomy — which includes consumer (private individuals) and family — R3 per tenant is economically impossible. The taxonomy itself is evidence that pooling is required; you cannot run a cluster per private individual.

One tenancy model for the whole estate. Rejected: the ratified business-app contract already commits client-facing apps to R3, and platform services genuinely cannot follow it. The honest answer is two defaults with a stated rule for which applies, not a single default that one half violates.

9. Challenge against outside practice

Held against the common industry framing (AWS's silo/pool/bridge model, cell-based architectures), the estate lands as follows.

Where the model is stronger than typical. Most SaaS multi-tenancy literature models exactly one boundary — tenant-to-tenant. This estate has two stacked boundaries: platform-service-to-platform-service (the consumer boundary) and tenant-to-tenant (the princedom boundary). Naming them separately and refusing to enforce both with one mechanism is a genuine strength and is uncommon.

The refusal to overstate the guarantee is also better than typical. Vendors routinely describe pool-model RLS as tenant isolation. ADR-0001 §5 states the blast radius in concrete terms instead, which is the practice worth keeping.

Where the model is weaker than typical, and this is the finding that matters. The pool model's standard mitigation is a verified enforcement layer — a shared data-access library, or RLS, that every service is demonstrably routed through. This estate has neither. §3 obliges every consumer to enforce the tenant boundary in its own code, and there is currently no fleet mechanism that verifies any consumer actually does.

audit-core has AUDIT-WP-0004-T03; tenant-engine has its own checks; rapp-postgres runs 15 adversarial probes against the consumer boundary and none against the tenant boundary inside a consumer. A single missing WHERE tenant_id = ? in any consumer is a cross-tenant data breach that no platform control would catch, no probe would fail, and no log would show as an error. This is the highest-severity known gap in the model and §12.3 proposes an owner for it.

Where the estate has quietly adopted a good pattern without naming it. Short-lived leased credentials from OpenBao, re-read at connection checkout, with overlap-first rotation, is stronger than the industry norm of a long-lived per-service secret. It bounds §9's residual risk by credential lifetime. It deserves to be stated as a tenancy control, not just a credential-management one.

Cell-based architecture is the unexplored option. R2 and R3 partition by consumer or by tenant. Neither partitions by cell — a fixed-size slice of infrastructure serving a bounded set of tenants, sized so that one cell's failure is survivable and cell count scales linearly. Given the estate spans railiance, NetKingdom, HelixForge and Coulomb with tenants layered on top, cells may fit better than per-tenant clusters at the point where R1 runs out. Not proposed here; flagged in §12.6 as worth evaluating before the ceiling in §10 is hit rather than after.

10. Scaling demands — the ceiling is closer than it looks

Measured against the live platform-pg specification, not estimated.

instances:        1              (no HA; single-node rail)
max_connections:  100
memory limit:     1Gi
cpu limit:        1
storage:          20Gi
per consumer:     14 connections (12 runtime + 2 migration)

Connection ceiling: roughly six consumers. At 14 connections per consumer declaration, seven consumers request 98 of 100 connections, before CNPG's instance manager, the metrics exporter, and superuser-reserved slots. The practical ceiling is about six. Two are provisioned today (audit-core, tenant-engine) plus the isolation probe. We are at roughly a third of capacity, and the third consumer request will not feel like a scaling event.

Memory is likely to bind before connections do. 100 PostgreSQL backends against a 1Gi limit is roughly 10MB per backend for all of shared buffers, work_mem, and per-backend overhead. Connection exhaustion produces a clear error; memory pressure produces OOM kills and degraded latency across every co-resident consumer at once.

Restore time couples all consumers. Physical backup is instance-wide, so restoring one consumer means restoring the whole instance to scratch, taking a logical dump, and importing. A consumer's RTO is therefore a function of total instance size, not its own. One large consumer silently degrades every co-resident's recovery promise — and per the storage draft §7.3, a consumer must not promise retention or recovery its platform cannot back.

instances: 1 means no tenant on platform-pg has HA. Deliberate on the current rail and honestly documented, but it means every shared consumer's availability story is "restart recovery", not high availability. A plan tier promising uptime cannot be satisfied at R1 as currently built.

What this implies. The ladder is not a long-term growth path on the present substrate — it is a two-to-four-consumer runway. The triggers in §6 must be monitored, not merely recorded, and connection and memory utilisation belong on the alert list before the next consumer is accepted.

11. Consequences

• The estate gains one vocabulary for a conversation currently held in four incompatible ones.
• tenantIsolation in PostgresConsumer is revealed as a mislabelled field and should be renamed or moved once this ADR settles (§12.1).
• Two ratified defaults (R1 for platform services, R3 for business apps) are reconciled by scope rather than by one overriding the other, at the cost of needing a rule for classifying a new service.
• The unverified tenant boundary (§9) becomes a named, owned gap rather than an assumption distributed across every repo.
• The platform-pg ceiling becomes a planning input rather than a discovery.
• Nothing in this ADR changes a running system. It is descriptive of what is built, plus five decisions that need owner sign-off.

12. Open questions for ratification

1. tenantIsolation field — rapp-postgres: rename to name its plane (tenantBoundaryEnforcement?), or move it out of the storage declaration entirely.
2. Placement ownership — railiance-platform: accept the ladder and triggers, and appoint a recorded placement owner per workload. This is the 2026-08-16 request, still open.
3. Tenant-boundary conformance — owner needed. Who builds and runs a fleet check that a consumer cannot leak tenant A's rows to tenant B? audit-core and tenant-engine have both declined fleet-scope work on correct boundary reasoning, so this needs appointing rather than volunteering. Highest-severity gap in the model.
4. Business app vs platform service — Custodian canon: a definition, so a new repo knows whether §1.2's R3 default or this ADR's R1 default applies to it.
5. Plan tier → rung mapping — adaptive-pricing and tenant-engine: no tier may name isolation without citing a rung (§7).
6. Cell-based partitioning — evaluate before the §10 ceiling is reached.
7. Engine neutrality — whether the ladder should be stated engine-neutrally or admit that R1's guarantee rests on a PostgreSQL property (per-database catalogs). Inherited unresolved from the storage draft.

13. Ratification path

The predecessor draft stalled because it was routed without one. This ADR:

1. Is reviewed by tenant-engine, flex-auth, rapp-postgres, railiance-platform, and adaptive-pricing against §12.
2. On acceptance, supersedes the routing of rapp-postgres/docs/canon-drafts/shared-platform-relational-storage_v0.1-draft.md, whose §§3–8 are absorbed above. That draft should then be withdrawn rather than left pending, so the estate does not carry two overlapping proposals.
3. On acceptance, rapp-postgres/docs/adr/ADR-0001 moves from proposed to accepted and is annotated as the PostgreSQL implementation of this model.