coulomb/the-custodian
4
0
Fork 0
Code Issues 1 Pull requests Projects Releases Packages Wiki Activity Actions

ADR-008: multi-tenancy model — four planes, one isolation ladder (proposed)
All checks were successful
CI Smoke / host-smoke (push) Successful in 0s
Details
CI Smoke / container-smoke (push) Successful in 1s
Details

Browse source 
The estate has built multi-tenancy across five documents that each cover a
slice and do not compose. This records the whole model and names what is
missing.

Core framing: multi-tenancy here is four orthogonal planes — identity,
authorization, data isolation, placement — not one property. Identity and
authorization are ratified and solid. Data isolation is stated but
unverified. Placement is owned by nobody and is precedent-by-accident.

Three findings that motivated the draft:

- R0 -> R1 -> R2 does not improve tenant isolation at all. Those rungs buy
  consumer isolation and capacity. Only R3 makes the tenant boundary
  structural. A plan tier selling R2 as isolation would be false.
- No fleet mechanism verifies that any consumer actually enforces the tenant
  boundary it is obliged to enforce. A missing WHERE tenant_id would be a
  cross-tenant breach that no probe fails and no log shows as an error.
  Highest-severity gap; needs an appointed owner.
- platform-pg holds roughly six consumers (100 max_connections, 14 each) and
  memory likely binds before connections do. Two are provisioned. The runway
  is shorter than the ladder implies.

Also reconciles two already-ratified and contradictory defaults:
instance-per-client for business apps, pooled for platform services, with no
rule for telling a new service which it is.

Proposed, not ratified. Carries a ratification path so it does not join the
shared-platform-relational-storage draft in limbo.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
codex 2026-08-17 11:26:07 +02:00
parent 8253276bdd
commit d7d1a490f9
1 changed files with 382 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

382
canon/architecture/adr-008-multi-tenancy-model.md Normal file
View file

@ -0,0 +1,382 @@
---
id: ADR-008
type: architecture-decision-record
title: "Multi-Tenancy Model: Four Planes, One Isolation Ladder"
status: proposed
decided_by: Bernd Worsch
date: "2026-08-17"
tags: ["architecture", "multi-tenancy", "isolation", "placement", "tenant-engine", "flex-auth", "rapp-postgres", "scaling"]
---
# ADR-008: Multi-Tenancy Model — Four Planes, One Isolation Ladder
## Status
**Proposed.** Drafted 2026-08-17 at the operator's request. This document
decides things currently owned by `tenant-engine`, `flex-auth`,
`rapp-postgres`, `adaptive-pricing`, and `railiance-platform`. It must be
reviewed by those owners before ratification; §12 lists what each is being
asked to accept. Until ratified it governs nothing, and the existing
per-repo documents remain authoritative for their own repos.
## Context
The estate has been building multi-tenancy for months and has never written
down what it is building. Five documents each cover a slice:
| Document | Covers | Status |
|---|---|---|
| `iam-profile_v0.3` (NetKingdom) | Tenant identifier shape, `tenant_roles` claim, staleness rules | Ratified |
| `tenant-engine-boundary-contract_v0.1` (NetKingdom) | Who owns tenant records, roles, plan assignment | Ratified |
| `business-app-service-contract_v0.1` §1 (Custodian) | Business apps: instance-per-client, tenant-keyed data | Ratified |
| `rapp-postgres/docs/adr/ADR-0001` | Consumer + tenant isolation in PostgreSQL | Proposed, governs one repo |
| `rapp-postgres/docs/canon-drafts/shared-platform-relational-storage_v0.1` | The stacked-boundary gap | Routed 2026-08-10, **still unratified** |
The slices do not compose, and three specific failures follow.
**The gap has been diagnosed once and the fix stalled.** The
shared-platform-relational-storage draft was written precisely to fill this
hole. It has sat unratified for a week and appears in neither canon
directory. Producing a second orphaned draft would make things worse, not
better; §13 therefore attaches a ratification path to this one.
**Placement is owned by nobody.** Whether a workload gets a dedicated
PostgreSQL cluster or a shared one is currently precedent-by-accident:
`user-engine-pg` and `target-revenue-pg` are dedicated; `apps-pg`,
`net-kingdom-pg`, `platform-pg`, `state-hub-db` and `forgejo-db` are shared.
Both patterns are live, neither is written down, and each new service copies
whichever neighbour it happened to consult. `tenant-engine` raised this with
`railiance-platform` on 2026-08-16 and it is unanswered.
**Two contradictory tenancy defaults are already ratified.**
`business-app-service-contract` §1.2 makes *instance-per-client* the default
for business apps. Platform services such as `audit-core` and `tenant-engine`
pool all tenants into one database. Both are correct for their case, but
nothing states which shape a new service takes, and the estate has no
definition distinguishing "business app" from "platform service". A new
repo can read either document and be conformant with one while violating the
other.
## Decision
### 1. Multi-tenancy is four orthogonal planes, not one property
The recurring confusion is that "is this multi-tenant?" is treated as a
single question. It is four, and they are independent:
| Plane | Question | Owner | State |
|---|---|---|---|
| **Identity** | How is a tenant named and validated? | `tenant-engine` / IAM Profile | Ratified, solid |
| **Authorization** | How is a request bound to the tenants it may act for? | `flex-auth` | Contract ratified |
| **Data isolation** | Where does the boundary between two tenants' rows sit? | The consuming service | Stated in ADR-0001, unverified fleet-wide |
| **Placement** | Which physical substrate holds a tenant's data? | *unowned* | Precedent only |
Treating them as one is not a theoretical tidiness problem; it is producing
concrete errors today.
`rapp-postgres`'s `PostgresConsumer` declaration carries a field
`tenantIsolation: consumer-service-boundary`. That is a **data-isolation**
fact recorded in a **placement** artifact. It reads as though the storage
platform enforces something. It does not, and ADR-0001 §4 is explicit that it
must not pretend to.
The "dedicated versus shared" argument routinely mixes placement (a capacity
and blast-radius decision) with data isolation (a correctness decision).
`tenant-engine` drew the right distinction unprompted on 2026-08-16 —
sharing a *cluster* is capacity and cheap to undo; sharing a *database*
entangles schemas and is not — and that distinction is adopted here as
binding.
**Decision:** every document, declaration, and plan tier that says
"isolation" MUST name which plane it means.
### 2. The identity and authorization planes are settled; this ADR does not reopen them
Recorded here only so the model is complete and so no future document
re-derives them:
- Tenant identifiers are `tenant:<grouping>:<name>` per ADR-0013, with
`tenant:platform` and `tenant:coulomb` reserved and ungrouped. `tenant-engine`
owns which values exist; the IAM Profile owns the wire format.
- Capability roles `PLTF`/`IAM`/`VEN`/`CUS` are non-exclusive and are a
separate fact from grouping. `tenant-engine` is the source of truth; the
`tenant_roles` token claim is a point-in-time cache that MUST NOT be trusted
for privileged, destructive, credential-vending, or `aal2`-class decisions.
- `flex-auth` makes authorization decisions; `tenant-engine` is a data source
and makes none.
- Storage platforms MUST NOT mint, parse, or authorize on tenant identifiers.
To them these are opaque strings.
### 3. Data isolation is enforced by the consuming service, and the platform must never imply otherwise
This is ADR-0001 §4 promoted to fleet scope.
A consumer connects through a pooled role shared across all of its tenants.
The database therefore sees one principal, not the requesting tenant, and has
no trustworthy basis on which to filter. Any database-level tenant filter
would depend on the consumer correctly setting a session variable — which is
the application-level check the consumer already owes, relocated somewhere it
*looks* like a database guarantee without being one.
- **Mandatory:** consumer-owned data is tenant-keyed from the first
migration, even where the consumer serves exactly one tenant today.
- **Mandatory:** the consumer binds the authenticated identity to the tenants
it may act for, at its own service boundary.
- **Available, opt-in:** row-level security keyed on a session GUC set at pool
checkout, as defence in depth behind a correct service-side check — never
as a substitute for one.
- **Prohibited:** describing the opt-in layer as a platform guarantee.
**This is the model's largest residual risk and §9 does not soften it.**
### 4. The isolation ladder
Placement decisions are expressed as rungs. Each rung states what it
actually buys, and — critically — what it does *not* change.
| Rung | Shape | Data-isolation guarantee | Blast radius of a leaked runtime credential | Live occupants |
|---|---|---|---|---|
| **R0** | Shared tables, tenant column | App-enforced | All tenants of that consumer | Inside every consumer today |
| **R1** | Database per consumer, shared cluster | App-enforced *(unchanged)* | All tenants of that consumer | `audit-core`, `tenant-engine` on `platform-pg` |
| **R2** | Dedicated cluster per consumer | App-enforced *(unchanged)* | All tenants of that consumer | `user-engine-pg`, `target-revenue-pg` |
| **R3** | Dedicated cluster per tenant | Structural | One tenant | Business apps per `business-app-service-contract` §1.2 |
| **R4** | R3 + separate region/jurisdiction | Structural + residency | One tenant | None |
The column that matters is the third one. **R0 → R1 → R2 does not improve
tenant isolation at all.** Those rungs buy consumer isolation, capacity
predictability, independent backup retention, and a smaller operational blast
radius. The tenant boundary stays exactly where §3 puts it: in application
code. Only R3 makes it structural.
**Decision:** R1 is the default for platform services. R3 is the default for
client-facing business apps, as already ratified. A service that is unsure
which it is MUST resolve that before choosing a rung (§12.4).
### 5. Movement between rungs must be an operational change, not a rebuild
Adopted from `tenant-engine`'s TEN-WP-0009 design, which got this right:
- Connect by injected credential only. No cluster, host, namespace, or
database name anywhere in source.
- Own a whole database, never tables inside someone else's. Sharing a cluster
is reversible; sharing a database makes relocation a schema merge.
- Idempotent schema creation, so a fresh target comes up correct.
- No cross-database joins or co-location assumptions.
A service meeting these moves R1 → R2 by: create target, dump, restore, swap
the credential reference, restart. No rebuild, no release, no code review.
**Decision:** these four properties are mandatory at every rung, including
R3. A business app at R3 that hard-codes its database name has forfeited
consolidation, which §1.3 of the business-app contract exists to preserve.
### 6. Placement triggers are recorded in advance
A rung change decided under pressure is decided badly. Each service records,
at provisioning time, what would move it up:
- Noisy neighbour on a latency-critical path
- A compliance or residency requirement
- A plan tier that sells isolation (see §7)
- Backup retention or RTO no longer fitting
- Connection or memory ceiling reached (see §9)
**Decision:** placement ownership is `railiance-platform`'s. This ADR
proposes the ladder and the triggers; it does not appoint per-workload
placement, which remains that repo's call and is the substance of the
2026-08-16 request still open there.
### 7. Isolation as a product property — the commercial hazard
Plan tiers that "sell isolation" are anticipated. The hazard is precise and
worth stating before revenue depends on it:
**Selling a move from R1 to R2 as stronger tenant isolation would be false.**
Per §4 it changes nothing about the tenant boundary. A customer buying
"isolated" and receiving R2 gets better noisy-neighbour behaviour and an
independent restore path — real value, and not what they think they bought.
Only R3 changes the answer to "can another tenant's data reach mine".
**Decision:** any plan tier naming isolation MUST cite the rung it delivers
and MUST NOT describe a lower rung in R3's language. `adaptive-pricing` owns
plan terms and `tenant-engine` owns which plan a tenant holds; neither
currently maps a tier to a rung, and §12.5 asks them to.
## 8. Alternatives considered
**Schema-per-consumer in one database.** Cheaper on connections, allows
cross-consumer queries. *Rejected on catalog leakage:* `pg_catalog` is
readable per-database, so every co-resident can enumerate every other's
table and column names regardless of `GRANT`s. Table names alone leak
business structure and no privilege setting removes that.
**Row-level security as the consumer boundary.** *Rejected:* puts a
correctness-critical boundary inside query predicates, where one missing
policy or a `SECURITY DEFINER` function silently removes it. Retained as
opt-in defence in depth for the tenant boundary only (§3).
**Platform-enforced tenant RLS by default.** *Rejected:* with a pooled
per-consumer role it depends on the consumer setting a session variable
correctly, producing the appearance of a database guarantee over what is
still an application check. A control that misrepresents where it is enforced
is worse than an honest absence.
**Dedicated instance per consumer as the default (R2 everywhere).**
*Rejected as a default* on operational cost — every instance multiplies
backup, upgrade, and monitoring surface — but it is the honest answer for a
consumer that cannot accept §9's residual risk, and it stays available as an
escalation rather than being argued away.
**Instance-per-tenant everywhere (R3 as the fleet default).** *Rejected:* at
the estate's tenant grouping taxonomy — which includes `consumer` (private
individuals) and `family` — R3 per tenant is economically impossible. The
taxonomy itself is evidence that pooling is required; you cannot run a
cluster per private individual.
**One tenancy model for the whole estate.** *Rejected:* the ratified
business-app contract already commits client-facing apps to R3, and platform
services genuinely cannot follow it. The honest answer is two defaults with a
stated rule for which applies, not a single default that one half violates.
## 9. Challenge against outside practice
Held against the common industry framing (AWS's silo/pool/bridge model,
cell-based architectures), the estate lands as follows.
**Where the model is stronger than typical.** Most SaaS multi-tenancy
literature models exactly one boundary — tenant-to-tenant. This estate has
**two stacked boundaries**: platform-service-to-platform-service (the
consumer boundary) and tenant-to-tenant (the princedom boundary). Naming them
separately and refusing to enforce both with one mechanism is a genuine
strength and is uncommon.
The refusal to overstate the guarantee is also better than typical. Vendors
routinely describe pool-model RLS as tenant isolation. ADR-0001 §5 states the
blast radius in concrete terms instead, which is the practice worth keeping.
**Where the model is weaker than typical, and this is the finding that
matters.** The pool model's standard mitigation is a *verified* enforcement
layer — a shared data-access library, or RLS, that every service is
demonstrably routed through. This estate has neither. §3 obliges every
consumer to enforce the tenant boundary in its own code, and there is
currently **no fleet mechanism that verifies any consumer actually does**.
`audit-core` has AUDIT-WP-0004-T03; `tenant-engine` has its own checks;
`rapp-postgres` runs 15 adversarial probes against the *consumer* boundary
and none against the tenant boundary inside a consumer. A single missing
`WHERE tenant_id = ?` in any consumer is a cross-tenant data breach that no
platform control would catch, no probe would fail, and no log would show as
an error. This is the highest-severity known gap in the model and §12.3
proposes an owner for it.
**Where the estate has quietly adopted a good pattern without naming it.**
Short-lived leased credentials from OpenBao, re-read at connection checkout,
with overlap-first rotation, is stronger than the industry norm of a
long-lived per-service secret. It bounds §9's residual risk by credential
lifetime. It deserves to be stated as a tenancy control, not just a
credential-management one.
**Cell-based architecture is the unexplored option.** R2 and R3 partition by
consumer or by tenant. Neither partitions by *cell* — a fixed-size slice of
infrastructure serving a bounded set of tenants, sized so that one cell's
failure is survivable and cell count scales linearly. Given the estate spans
railiance, NetKingdom, HelixForge and Coulomb with tenants layered on top,
cells may fit better than per-tenant clusters at the point where R1 runs out.
Not proposed here; flagged in §12.6 as worth evaluating before the ceiling in
§10 is hit rather than after.
## 10. Scaling demands — the ceiling is closer than it looks
Measured against the live `platform-pg` specification, not estimated.
```
instances: 1 (no HA; single-node rail)
max_connections: 100
memory limit: 1Gi
cpu limit: 1
storage: 20Gi
per consumer: 14 connections (12 runtime + 2 migration)
```
**Connection ceiling: roughly six consumers.** At 14 connections per consumer
declaration, seven consumers request 98 of 100 connections, before CNPG's
instance manager, the metrics exporter, and superuser-reserved slots. The
practical ceiling is about **six**. Two are provisioned today
(`audit-core`, `tenant-engine`) plus the isolation probe. **We are at
roughly a third of capacity, and the third consumer request will not feel
like a scaling event.**
**Memory is likely to bind before connections do.** 100 PostgreSQL backends
against a 1Gi limit is roughly 10MB per backend for all of shared buffers,
work_mem, and per-backend overhead. Connection exhaustion produces a clear
error; memory pressure produces OOM kills and degraded latency across every
co-resident consumer at once.
**Restore time couples all consumers.** Physical backup is instance-wide, so
restoring one consumer means restoring the whole instance to scratch, taking
a logical dump, and importing. A consumer's RTO is therefore a function of
**total instance size**, not its own. One large consumer silently degrades
every co-resident's recovery promise — and per the storage draft §7.3, a
consumer must not promise retention or recovery its platform cannot back.
**`instances: 1` means no tenant on `platform-pg` has HA.** Deliberate on the
current rail and honestly documented, but it means every shared consumer's
availability story is "restart recovery", not high availability. A plan tier
promising uptime cannot be satisfied at R1 as currently built.
**What this implies.** The ladder is not a long-term growth path on the
present substrate — it is a two-to-four-consumer runway. The triggers in §6
must be monitored, not merely recorded, and connection and memory
utilisation belong on the alert list *before* the next consumer is accepted.
## 11. Consequences
- The estate gains one vocabulary for a conversation currently held in four
incompatible ones.
- `tenantIsolation` in `PostgresConsumer` is revealed as a mislabelled field
and should be renamed or moved once this ADR settles (§12.1).
- Two ratified defaults (R1 for platform services, R3 for business apps) are
reconciled by scope rather than by one overriding the other, at the cost of
needing a rule for classifying a new service.
- The unverified tenant boundary (§9) becomes a named, owned gap rather than
an assumption distributed across every repo.
- The `platform-pg` ceiling becomes a planning input rather than a discovery.
- Nothing in this ADR changes a running system. It is descriptive of what is
built, plus five decisions that need owner sign-off.
## 12. Open questions for ratification
1. **`tenantIsolation` field** — `rapp-postgres`: rename to name its plane
(`tenantBoundaryEnforcement`?), or move it out of the storage declaration
entirely.
2. **Placement ownership** — `railiance-platform`: accept the ladder and
triggers, and appoint a recorded placement owner per workload. This is the
2026-08-16 request, still open.
3. **Tenant-boundary conformance** — *owner needed*. Who builds and runs a
fleet check that a consumer cannot leak tenant A's rows to tenant B?
`audit-core` and `tenant-engine` have both declined fleet-scope work on
correct boundary reasoning, so this needs appointing rather than
volunteering. Highest-severity gap in the model.
4. **Business app vs platform service** — *Custodian canon*: a definition, so
a new repo knows whether §1.2's R3 default or this ADR's R1 default
applies to it.
5. **Plan tier → rung mapping** — `adaptive-pricing` and `tenant-engine`:
no tier may name isolation without citing a rung (§7).
6. **Cell-based partitioning** — evaluate before the §10 ceiling is reached.
7. **Engine neutrality** — whether the ladder should be stated
engine-neutrally or admit that R1's guarantee rests on a PostgreSQL
property (per-database catalogs). Inherited unresolved from the storage
draft.
## 13. Ratification path
The predecessor draft stalled because it was routed without one. This ADR:
1. Is reviewed by `tenant-engine`, `flex-auth`, `rapp-postgres`,
`railiance-platform`, and `adaptive-pricing` against §12.
2. On acceptance, **supersedes** the routing of
`rapp-postgres/docs/canon-drafts/shared-platform-relational-storage_v0.1-draft.md`,
whose §§3–8 are absorbed above. That draft should then be withdrawn rather
than left pending, so the estate does not carry two overlapping proposals.
3. On acceptance, `rapp-postgres/docs/adr/ADR-0001` moves from `proposed` to
`accepted` and is annotated as the PostgreSQL implementation of this model.