diff --git a/research/2026-08-17-adr008-multi-tenancy-research-index.md b/research/2026-08-17-adr008-multi-tenancy-research-index.md new file mode 100644 index 0000000..1712b3d --- /dev/null +++ b/research/2026-08-17-adr008-multi-tenancy-research-index.md @@ -0,0 +1,122 @@ +# ADR-008 multi-tenancy — external research index + +> Research artefact — 2026-08-17. Four deep-research digests, one per plane of +> `canon/architecture/adr-008-multi-tenancy-model.md` (draft-2), surveying prior +> knowledge and best practice and holding our ladders against it. +> +> - `2026-08-17-adr008-plane-i-identity.md` +> - `2026-08-17-adr008-plane-a-authorization.md` +> - `2026-08-17-adr008-plane-e-enforcement.md` +> - `2026-08-17-adr008-plane-p-placement.md` +> +> Purpose: feed draft-3. Nothing here amends the ADR by itself. + +## 1. Headline: the reframe is externally corroborated + +The operator's draft-1 → draft-2 instruction — stop specifying one model, build +a framework of graduated levels — matches the most rigorous public guidance +almost verbatim: + +> "Instead of viewing isolation as a discrete property, consider it a spectrum. +> You can deploy components of your architecture that are more isolated or less +> isolated than other components in the same architecture." +> — [Azure Architecture Center, Tenancy models](https://learn.microsoft.com/en-us/azure/architecture/guide/multitenant/considerations/tenancy-models) + +Microsoft also derives our E↔P coupling independently: shared deployment ⇒ +enforcement lives in application code; dedicated deployment ⇒ enforcement is +structural. Draft-2's shape is sound. What follows is about its details. + +## 2. Corrections to draft-2 — things it currently states wrongly + +Ranked by how badly they mislead. + +| # | Correction | Plane | Where | +|---|---|---|---| +| C1 | **E3 does not prevent an application from routing around it.** Any session can re-`SET` the GUC; SQL injection or a compromised process defeats it entirely. E3 is strong against *accident*, weak against *compromise*. The ADR's phrase "the application cannot trivially route around" overclaims — the exact fault §6 prohibits. | E | §4.3 | +| C2 | **The pooling claim in §16 is backwards.** Transaction-scoped context (`SET LOCAL`) is what makes E3 *safe* under pooling; **statement**-level pooling is what breaks it — returning other tenants' rows, only under production concurrency. The ADR says E3 "forecloses aggressive transaction-level pooling". | E | §16 | +| C3 | **"Set at pool checkout" is the wrong scope.** Should be set transaction-locally inside an explicit transaction. | E | §4.3 | +| C4 | **E3 does not supersede E2 — it layers on it.** Practice treats them as coexisting; a service dropping E2 on reaching E3 is worse off, because E3 fails open under injection. Claiming E3 should require the E2 artifact too. | E | §4.3, §13 | +| C5 | **P cannot express per-tenant placement, which §11 requires.** A tier requiring `E3 P2` bought by only some tenants puts the service at two P levels at once. The posture vector cannot represent it, forcing an over- or under-claim. | P | §5, §11 | + +## 3. Gaps — real concerns no plane covers + +| # | Gap | Suggested home | +|---|---|---| +| G1 | **Tenant offboarding and deletion.** OWASP lists complete data deletion on offboarding as a MUST. `tenant-engine` has reversible retirement and explicitly no hard-delete; `platform-pg` retains backups 30 days. Falls between two repos' declared scopes. | New open question; a lifecycle stage across all four planes, not a fifth plane. | +| G2 | **Non-database stores.** Caches, search indices and background jobs are named leak surfaces. Our ladders are datastore-shaped throughout. | Scope note on the E ladder, or widened evidence artifacts. | +| G3 | **Identity-provider placement.** Realm-per-tenant vs Keycloak Organizations is the same silo/pool decision, live in our estate, undecided. Realm-per-tenant has a stated ceiling of 5–20 tenants. | Note in §3 that P scopes to data substrate; identity placement is parallel and unowned. | +| G4 | **Noisy-neighbour evidence.** Azure requires testing both that data does not leak *and* that noisy-neighbour outcomes are acceptable. §13 covers only leakage. | Add a P1/P2 evidence artifact. | +| G5 | **Tenant→deployment mapping record.** A first-class component in the reference guidance. Our placement facts in §1 were assembled by inspection. | Follows from G3/C5. | +| G6 | **Cell sizing.** `platform-pg` is an uncapped cell with a computed ~6-consumer ceiling and no overflow target. | Reframes §18.8: ask what its declared maximum size is. | +| G7 | **Tenant identifier encodes mutable headcount bands.** `tenant::` embeds `small`/`medium`/`large`, which change. Contradicts identifier-design consensus. | Not ADR-008's; raise to `tenant-engine` / NetKingdom canon re ADR-0013. | +| G8 | **Logging.** OWASP MUST: include tenant context in all log entries. Unmodelled. | Minor; fold into evidence artifacts. | + +## 4. Opportunities — cheaper or better than we assumed + +| # | Finding | Effect | +|---|---|---| +| O1 | **pgTAP + pgrls** (67 RLS lint rules, semantic policy-diff for CI gating, pytest isolation plugin) exist off the shelf. `rapp-postgres` already runs a 15-probe harness of the same shape. | The E3 half of open question §18.3 is far cheaper than assumed. Splits the question into a tractable mechanical half and a human half. | +| O2 | **OWASP Multi-Tenant Security Cheat Sheet** is the closest external baseline to our framework and is referenced nowhere in the estate. | Adopt as the named external baseline in ADR §15. | +| O3 | **OpenID AuthZEN Authorization API 1.0** reached Final Specification January 2026; Keycloak shipped experimental support May 2026. Demonstrated PDP interoperability across 5+ engines. | `flex-auth`'s bespoke `CheckRequest` is pre-standard. Propose an `A4` rung for standard-interface PDP delegation. | +| O4 | **Token versioning** as a middle rung between cached claim and live re-query. | Cheap global invalidation without a hot-path lookup. | +| O5 | **BOLA / OWASP API1.** Our tenant-boundary gap is the industry's #1 API vulnerability, top of the list since it launched. | Cite by name. Converts an internal judgement into an external consensus for reviewers with a security background. | + +## 5. Specific technical risks for the E3 implementation contract (§18.6) + +Concrete requirements the contract must carry, each sourced in the enforcement +digest: + +- `FORCE ROW LEVEL SECURITY` on every tenant-owned table. Without it the table + owner bypasses policies **silently** — and ADR-0001 already established that + objects created by `_migrate` are owned by `_migrate`, so our + migration role would bypass every policy on tables it created. +- No `BYPASSRLS` attribute on any leased role. +- `SECURITY INVOKER` for ordinary logic; `SECURITY DEFINER` reserved for + deliberate, reviewed cross-tenant access. +- Context set with `SET LOCAL` inside an explicit transaction, never + session-scoped. +- An `EXPLAIN` comparison before and after enabling RLS: functional indexes + using non-leakproof functions (`lower`, `enum_eq`) stop being used under RLS. + Relevant to `tenant-engine`, which sits on the synchronous authorization path + and just requested a 5s statement timeout. + +## 6. Recommended shape of draft-3 + +1. Apply corrections C1–C4 to §4.3, §13 and §16. These are the ones a reviewer + will otherwise catch, and C1 is the ADR overclaiming in exactly the way it + forbids. +2. Resolve C5 by making placement expressible per tenant class — a default plus + exceptions in the posture vector — or by declaring per-tenant placement out + of scope and accepting that isolation tiers become unmodellable. +3. Split open question §18.3 into a mechanical half (E1/E3, tooling exists) and + a semantic half (E2, adversarial review on a recorded cadence). Cross-tenant + findings "require a human tester who can set up separate tenant contexts and + compare the responses" — a green CI run is not E2 evidence. +4. Add G1 (offboarding/deletion) and G3 (identity placement) as open questions; + add G2 as a scope note. +5. Cite the external baselines — OWASP Multi-Tenant Security Cheat Sheet, Azure + tenancy models, AWS SaaS Lens — in §15, and BOLA/API1 in §13.2. +6. Reframe §18.8 from "evaluate cells" to "declare `platform-pg`'s maximum + size and its overflow target". +7. Route G7 to `tenant-engine` separately. It is a critique of ratified canon, + not of this ADR, and folding it in would overreach. + +## 7. Method and limits + +Four search-and-read passes, one per plane, against vendor architecture +guidance (AWS Well-Architected SaaS Lens, Azure Architecture Center), standards +bodies (OpenID Foundation), security baselines (OWASP), PostgreSQL practitioner +literature, and IdP vendor documentation. Full source lists are in each digest. + +Limits worth stating: + +- No primary academic sources beyond the Zanzibar lineage; this is an + engineering-practice survey, not a literature review. +- Vendor guidance is not neutral — AWS and Azure both describe patterns their + platforms sell. The Azure tenancy-models page is the most rigorous taxonomy + found and is used heavily in the P digest; readers should discount its + Azure-service-specific recommendations accordingly. +- Nothing here was tested against our own systems. Every claim about our estate + is drawn from repo documents, not from probing running infrastructure. +- The E digest's corrections are the highest-confidence findings, being + PostgreSQL mechanics rather than architectural opinion. diff --git a/research/2026-08-17-adr008-plane-a-authorization.md b/research/2026-08-17-adr008-plane-a-authorization.md new file mode 100644 index 0000000..91e522c --- /dev/null +++ b/research/2026-08-17-adr008-plane-a-authorization.md @@ -0,0 +1,224 @@ +# Plane A — Authorization: prior knowledge and best practices + +> Research artefact — 2026-08-17. Supports `canon/architecture/adr-008-multi-tenancy-model.md` +> (draft-2), plane A of four. Surveys external practice for binding a request +> to the tenants it may act for, and holds our `A0–A3` ladder against it. +> Companions: plane I (identity), plane E (enforcement), plane P (placement). + +## 1. What the plane covers + +How a request is bound to the tenants its principal may act for, where that +decision is made, and how the decision point is kept honest. Our ladder: + +| Level | State | +|---|---| +| A0 | None, or tenant context not carried | +| A1 | Ad-hoc checks scattered through handlers | +| A2 | Single local authorization boundary; context bound once | +| A3 | Delegated to `flex-auth` as PDP, live re-query where the IAM Profile requires | + +## 2. This is the industry's #1 vulnerability class, by name + +The single most important external finding: what our framework calls a tenant +boundary failure, the security industry calls **Broken Object Level +Authorization**, and it is not a marginal concern. + +- BOLA is **API1** in the OWASP API Security Top 10 and "has held the top spot + since the list launched" + ([Decryption Digest — BOLA and IDOR testing guide](https://www.decryptiondigest.com/blog/bola-idor-api-authorization-test-remediation)). +- It is "consistently the most common, the most exploitable, and the most + damaging API vulnerability across bug bounty submissions, public disclosures, + and production penetration tests" (ibid.). +- The multi-tenant framing is explicit: "a single missing authorization check + can let one tenant access another's storage buckets, database records, or + configuration data" + ([Wiz — OWASP API security top 10](https://www.wiz.io/academy/api-security/owasp-api-security)). + +**Against our model:** ADR-008 §13.2 identifies the missing cross-tenant +evidence artifact as the framework's largest gap. This research upgrades that +from an internal judgement to an industry consensus: we have no test coverage +for the vulnerability class that ranks first globally. The ADR should cite +BOLA/API1 by name so the gap is legible to anyone who arrives with a security +background rather than an architecture one. + +## 3. OWASP has a Multi-Tenant Security Cheat Sheet, and we should adopt it wholesale + +This is the most directly applicable artefact found in the whole research +effort, and neither ADR-008 nor any repo document references it +([OWASP Multi-Tenant Security Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/Multi_Tenant_Security_Cheat_Sheet.html)). + +Its normative requirements, mapped to our ladders: + +| OWASP requirement | Our plane | Status | +|---|---|---| +| "Never trust client-supplied tenant IDs without validation" — derive from verified JWT claims | I2 | Implied, not stated. Tighten. | +| "Establish tenant context early in the request lifecycle (middleware/interceptor)"; bind to the authenticated session | A2 | **Matches A2 exactly.** | +| "Always validate that requested resources belong to the current tenant" | E1–E2 | Matches. | +| "Include tenant context in all log entries" | — | **Not covered by any plane.** | +| Tenant-isolated audit trails | — | `audit-core`'s concern; not in the framework. | +| "Ensure complete data deletion on tenant offboarding" | — | **Not covered. See §6.** | +| Prefix all cache keys with tenant identifiers | — | **Not covered** (see plane I §3). | +| Per-tenant rate limiting with tier-based quotas | — | Adjacent to `tenant-engine` guardrails. | +| Separate API keys per tenant | E4-adjacent | Our E4 is credential-based; consistent. | + +Explicitly rejected anti-patterns, two of which we should check ourselves +against: + +- **"Skipping tenant validation for internal services."** Our estate is mostly + internal service-to-service calls — `flex-auth` calls `tenant-engine` + synchronously on the authorization path, `user-engine` calls it for tenant + creation. Our A ladder describes a request arriving with a subject; it does + not say whether a *service* identity calling on behalf of a tenant carries + and revalidates tenant context. This is a real question for A3. +- **"Exposing sequential or guessable resource IDs."** Not assessed anywhere. + +**Proposed action:** reference the cheat sheet from ADR-008 as the external +baseline, and fold its uncovered requirements (logging, cache keys, offboarding +deletion) into either the ladders or an explicit scope exclusion. + +## 4. The PDP/PEP split is right, and there is now a standard for the wire + +Our A3 delegates decisions to `flex-auth` as PDP, which is the mainstream +architecture — a Policy Enforcement Point intercepts an action and queries a +Policy Decision Point +([DEV — RBAC vs ABAC vs ReBAC](https://dev.to/kanywst/rbac-vs-abac-vs-rebac-how-to-choose-and-implement-access-control-models-3i2d)). + +What is new since our contracts were written: + +- **OpenID AuthZEN Authorization API 1.0** was approved as an OpenID Final + Specification in January 2026 and published Standards Track in March 2026 + ([OpenID Foundation — Authorization API 1.0 final specification approved](https://openid.net/authorization-api-1-0-final-specification-approved/), + [spec](https://openid.github.io/authzen/)). +- Its purpose is exactly our seam: letting PDPs and PEPs "communicate + authorization requests and decisions to each other without requiring + knowledge of each other's inner workings" + ([Curity — introduction to AuthZEN](https://curity.io/resources/learn/authzen/)). +- Interoperability is demonstrated, not theoretical: one PEP implementation + driving 5+ different PDPs (Topaz, Axiomatics, OpenFGA) by switching endpoint + URLs alone + ([DEV — AuthZEN deep dive](https://dev.to/kanywst/authzen-authorization-api-10-deep-dive-the-standard-api-that-separates-authorization-decisions-1m2a)). +- Keycloak shipped experimental AuthZEN support in May 2026 + ([Keycloak — AuthZEN as experimental feature](https://www.keycloak.org/2026/05/authzen-as-experimental-feature)). + +**Against our model:** `flex-auth` uses a bespoke `CheckRequest` shape and a +bespoke action vocabulary (`tenant.update`, `tenant.guardrail.set`, …), with +action strings copied verbatim between repos to avoid re-derivation. That +coupling is exactly what AuthZEN standardises away. We are not wrong — we are +pre-standard, and the standard landed seven months ago. + +**Proposed amendment:** A3 currently says "delegated to `flex-auth` as PDP". +Consider distinguishing: + +- **A3** — externalised to a PDP (any protocol). +- **A4** — externalised over a standard interface (AuthZEN), so the PDP is + swappable and the PEP is not coupled to one engine's request shape. + +That gives the ladder somewhere to go and makes the current coupling visible +rather than invisible. It also reframes `key-cape`/Keycloak's AuthZEN support +as a fleet opportunity rather than an unrelated upstream feature. + +## 5. Model choice: our vocabulary is RBAC-shaped, our problem may be ReBAC-shaped + +- ReBAC originates in Google's **Zanzibar**, modelling authorization as a graph + of typed relationships, with authorization data stored as explicit tuples in + a dedicated datastore + ([AuthZed — introduction to Google Zanzibar](https://authzed.com/learn/google-zanzibar)). +- It is described as "well-suited to multi-tenant enterprise environments where + access patterns are relational rather than role-based", and specifically for + modelling "complex customer organizational structures while maintaining + strict data isolation" (ibid., + [AuthZed — ABAC vs ReBAC](https://authzed.com/learn/abac-vs-rebac-when-to-use-which)). +- Open implementations: SpiceDB, OpenFGA, Authzed + ([IAM Day by Day — Zanzibar & ReBAC](https://iamdaybyday.com/patterns/authorization/zanzibar/)). + +**Against our model:** our capability roles (`PLTF`/`IAM`/`VEN`/`CUS`) are +tenant-level RBAC, and `user-engine` owns memberships separately. The moment a +question like "may this user act for this tenant, given they are a member of an +organisation that administers it" arises, that is a relationship query, and +role-flattening will start to strain. Not a present problem — flagged because +the estate's own tenant taxonomy (`family`, `community`, `association`) is +overtly relational and will produce these questions. + +## 6. Finding: nothing in the framework covers tenant exit + +OWASP lists "ensure complete data deletion on tenant offboarding" as a **must** +([OWASP Multi-Tenant Security Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/Multi_Tenant_Security_Cheat_Sheet.html)). + +Our position today: `tenant-engine`'s lifecycle contract provides reversible +retirement and states plainly that **there is no hard-delete**. That is a +defensible design for an audit-bearing estate, but it means: + +- No plane in ADR-008 has anything to say about tenant exit, data deletion, or + the interaction with retention. +- Physical backups on `platform-pg` retain 30 days, so even a hypothetical + hard-delete leaves tenant data recoverable for a month — which is a GDPR + erasure question, not just an architecture one. +- `rapp-postgres` SCOPE explicitly excludes tenant identity, and + `tenant-engine` explicitly excludes storage. Deletion falls between them. + +**Proposed action:** this is a fifth concern the framework does not model, and +it is not obviously a fifth *plane* — it is a lifecycle stage cutting across +all four. Recommend adding it as an explicit open question rather than +inventing a plane for it. The estate's `gdpr_report` tooling in the hub +suggests someone has thought about this elsewhere; worth locating before +drafting. + +## 7. Finding: automated conformance testing may not be sufficient for this plane + +This challenges the recommendation I gave in the ADR review (§18.3, "build a +fleet conformance suite"). + +- "Finding cross-tenant vulnerabilities requires semantic understanding of what + data belongs to which context, and that understanding requires a human tester + who can set up separate tenant contexts and compare the responses." +- "Across 119 SaaS assessments, the highest-impact findings were concentrated + in access control, business logic, and tenant isolation, precisely the + categories that require context-aware human testing rather than automated + tool execution." + ([Bugstrix — multi-tenant SaaS security testing](https://bugstrix.com/blogs/multi-tenant-saas-security-testing-how-to-prevent-cross-tenant-data-leaks/)) + +**Against our model:** this does not invalidate the conformance suite — a +mechanical check that every tenant-owned table carries the key (E1), or that a +session without the GUC reads nothing (E3), is precisely the kind of structural +assertion automation does well. What it invalidates is the hope that a suite +*replaces* adversarial review at E2, where the boundary lives in business +logic and the failure is semantic. + +**Proposed amendment to §13:** split the evidence artifacts by what can be +automated. E1 and E3 are mechanical and belong in CI. E2 is semantic and needs +a periodic adversarial review with a recorded date, closer to a pen-test +cadence than a test suite. Claiming E2 on the strength of a green CI run would +be exactly the overclaim §6 of the ADR prohibits. + +## 8. Summary against our ladder + +| Finding | Verdict | +|---|---| +| Tenant boundary failure = BOLA = OWASP API1, top of the list since launch | **Reframe.** Cite by name; our largest gap is the industry's largest risk. | +| OWASP Multi-Tenant Security Cheat Sheet exists and is unreferenced | **Adopt.** Closest external baseline to our framework. | +| A2 matches OWASP's "establish context early, bind to session" | **Aligned.** | +| "Skipping tenant validation for internal services" is a named anti-pattern | **Check ourselves.** Our estate is mostly internal service-to-service. | +| PDP/PEP split is mainstream | **Aligned.** | +| AuthZEN 1.0 Final (Jan 2026) standardises the PDP wire; Keycloak experimental May 2026 | **Gap.** `flex-auth` is pre-standard and bespoke. Propose A4. | +| ReBAC/Zanzibar suits relational tenant structures | **Watch.** Our `family`/`community`/`association` groupings are relational. | +| Tenant offboarding deletion is a MUST we do not model | **Finding.** No plane covers exit; interacts with 30-day backup retention. | +| Cross-tenant testing needs human semantic review, not only automation | **Amend §13.** E1/E3 mechanical, E2 adversarial-review with a date. | + +## Sources + +- [OWASP — Multi-Tenant Security Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/Multi_Tenant_Security_Cheat_Sheet.html) +- [Wiz — OWASP API Security Top 10 risks and how to mitigate them](https://www.wiz.io/academy/api-security/owasp-api-security) +- [Decryption Digest — BOLA and IDOR testing guide](https://www.decryptiondigest.com/blog/bola-idor-api-authorization-test-remediation) +- [SecPortal — Broken Object Level Authorization (BOLA) guide](https://secportal.io/vulnerabilities/broken-object-level-authorization) +- [Bugstrix — Multi-tenant SaaS security testing: preventing cross-tenant data leaks](https://bugstrix.com/blogs/multi-tenant-saas-security-testing-how-to-prevent-cross-tenant-data-leaks/) +- [OpenID Foundation — Authorization API 1.0 Final Specification approved](https://openid.net/authorization-api-1-0-final-specification-approved/) +- [OpenID AuthZEN — Authorization API 1.0 specification](https://openid.github.io/authzen/) +- [OpenID Foundation — AuthZEN working group specifications](https://openid.net/wg/authzen/specifications/) +- [Curity — An introduction to Authorization Exchange (AuthZEN)](https://curity.io/resources/learn/authzen/) +- [DEV — AuthZEN Authorization API 1.0 deep dive](https://dev.to/kanywst/authzen-authorization-api-10-deep-dive-the-standard-api-that-separates-authorization-decisions-1m2a) +- [Keycloak — AuthZEN as an experimental feature (May 2026)](https://www.keycloak.org/2026/05/authzen-as-experimental-feature) +- [Auth0 — A guide to OpenID AuthZEN's Authorization API 1.0](https://auth0.com/blog/implementing-authzen-guide-openid-authorization-api/) +- [AuthZed — An introduction to Google Zanzibar and ReBAC](https://authzed.com/learn/google-zanzibar) +- [AuthZed — ABAC vs ReBAC: when to use which](https://authzed.com/learn/abac-vs-rebac-when-to-use-which) +- [IAM Day by Day — Zanzibar & relationship-based access control](https://iamdaybyday.com/patterns/authorization/zanzibar/) +- [DEV — RBAC vs ABAC vs ReBAC: how to choose](https://dev.to/kanywst/rbac-vs-abac-vs-rebac-how-to-choose-and-implement-access-control-models-3i2d) diff --git a/research/2026-08-17-adr008-plane-e-enforcement.md b/research/2026-08-17-adr008-plane-e-enforcement.md new file mode 100644 index 0000000..1968399 --- /dev/null +++ b/research/2026-08-17-adr008-plane-e-enforcement.md @@ -0,0 +1,228 @@ +# Plane E — Enforcement: prior knowledge and best practices + +> Research artefact — 2026-08-17. Supports `canon/architecture/adr-008-multi-tenancy-model.md` +> (draft-2), plane E of four. Surveys external practice for where the tenant +> boundary is mechanically enforced, and holds our `E0–E4` ladder against it. +> **This digest contains a correction to the ADR's own definition of E3.** +> Companions: plane I (identity), plane A (authorization), plane P (placement). + +## 1. What the plane covers + +Where the tenant boundary actually holds, and what defeats it. Our ladder: + +| Level | Mechanism | ADR claims it is defeated by | +|---|---|---| +| E0 | None | Anything | +| E1 | Tenant-keyed, filtered per query | One missing predicate | +| E2 | Single service-side choke point | Code bypassing the choke point | +| E3 | RLS keyed on a session GUC set at pool checkout | `SECURITY DEFINER`, missing policy, wrong GUC | +| E4 | Per-tenant credential and substrate | Platform-level compromise | + +## 2. Correction: E3's description is wrong, and the error matters + +ADR-008 draft-2 describes E3 as "platform-assisted: row-level security keyed on +a session GUC set at pool checkout, or an equivalent data-access layer **the +application cannot trivially route around**". + +That last clause is false for the GUC mechanism, and the literature is blunt +about it: + +> "Any session can `SET` any custom parameter. RLS policies that rely on +> `current_setting('app.tenant_id')` are only as secure as the application's +> control over SQL execution. Users with direct access can trivially +> `SET app.tenant_id = '999'` and bypass the policy." +> — [kmoppel — Yes, Postgres can do session vars, but should you use them?](https://kmoppel.github.io/2025-06-03-yes-postgres-can-do-session-variables-but-should-you-use-them/) + +And the corollary: "SQL injection could enable an attacker to issue their own +`SET` command, therefore accessing other customer's data. The session variable +based approach is only safe when you protect yourself against SQL injections" +(ibid.). + +**What this means for the framework.** Draft-1's instinct was not baseless +after all — but neither was draft-2's reversal. The honest resolution is that +E3's strength is **threat-model dependent**, and the ladder must say so: + +| Threat | Does E3 help? | +|---|---| +| Developer forgets a `WHERE tenant_id = ?` | **Yes, decisively.** This is the common case and the reason E3 exists. | +| A new code path bypasses the repository/choke point | **Yes.** The database still filters. | +| SQL injection reaching the connection | **No.** The attacker can re-`SET` the GUC. | +| Compromised application process | **No.** It holds a credential that can address all tenants. | + +So E3 is a strong control against **accident** and a weak one against +**compromise**. E4 is the only rung that holds against both, because the +credential itself cannot address another tenant's data. + +**Proposed amendment to §4.3 of the ADR:** replace "the application cannot +trivially route around" with an explicit threat-model row. The current wording +overclaims in precisely the way §6 of the ADR prohibits — an irony worth fixing +before reviewers find it. + +## 3. Three PostgreSQL mechanics that will bite our specific role model + +These are not generic warnings; each interacts with `rapp-postgres` ADR-0001 as +built. + +### 3.1 The table owner bypasses RLS silently — and our migration role owns tables + +> "By default, the table owner bypasses RLS policies unless the table is +> altered with `FORCE ROW LEVEL SECURITY`." +> — [pganalyze — RLS, security invoker views and LEAKPROOF functions](https://pganalyze.com/blog/5mins-postgres-row-level-security-bypassrls-security-invoker-views-leakproof-functions) + +> "The `FORCE` keyword is critical — without it, the table owner bypasses +> policies silently." +> — [Kawshik — securing pgvector with PostgreSQL RLS](https://kawshik.dev/blog/multi-tenant-rag-pgvector-postgres-rls.html) + +ADR-0001 already discovered — via `isolation-test.sh`, not review — that +**objects created by `_migrate` are owned by `_migrate`**, not by +`_owner`, because default privileges key on the creating role. That finding was +about grants. It has a second consequence nobody has drawn: under RLS, the +`_migrate` role would silently bypass every policy on every table it created. + +Add `BYPASSRLS` to the picture — roles with that attribute are exempt +([oneuptime — securing multi-tenant data with RLS](https://oneuptime.com/blog/post/2026-01-25-row-level-security-postgresql/view)) +— and the E3 provisioning contract needs to state explicitly: `FORCE ROW LEVEL +SECURITY` on every tenant-owned table, no `BYPASSRLS` on any leased role, and +the runtime role must not own the objects (which ADR-0001 already guarantees, +for a different reason). + +### 3.2 Connection pooling can silently serve the wrong tenant's rows + +This is the failure mode most likely to reach production undetected. + +> "RLS with `SET` or `SET LOCAL` will not work properly with pgbouncer in +> statement pooling mode; you will likely return rows for the wrong users and +> it may only happen in production when multiple people are hitting the app at +> once." — [Daniel Imfeld — PostgreSQL row level security](https://imfeld.dev/notes/postgresql_row_level_security) + +> "In highly concurrent environments with connection pooling (like pgbouncer), +> managing this context correctly is a major source of subtle, hard-to-debug +> bugs. If the context leaks between connections, so might the data." +> — [ShiftAsia — an intro to PostgreSQL's RLS](https://shiftasia.com/community/why-your-database-needs-boundaries-an-intro-to-postgresqls-row-level-security-rls/) + +The mitigation is consistent across sources: set the context **transaction-locally** +(`SET LOCAL` inside an explicit transaction), never session-locally, so pooling +mode cannot leak it. Our ADR says "set at pool checkout", which is the +session-scoped phrasing and the wrong one. + +**Against our model:** §16 of the ADR notes that E3 "forecloses aggressive +transaction-level pooling". That is backwards — transaction-scoped context is +what makes E3 *safe* under pooling; it is **statement**-level pooling that is +incompatible. Correct the scaling section as well as §4.3. + +This also sharpens the ADR's own §12 point about designing guarding for +invisible failure: an RLS context leak produces correct-looking rows for the +wrong tenant. No error, no log line, no crash. + +### 3.3 RLS silently disables some indexes + +> "Due to security concerns, functional indices using leakproof functions +> cannot be used for RLS-secured queries. Functions such as `lower` or +> `enum_eq` are not leakproof. This means indices such as +> `some_table(lower(email))` will not be used when RLS is active." +> — [sigmoid.at — notes on PostgreSQL row level security](https://sigmoid.at/post/2021/09/23/postgresql_row_level_security/) + +A service adopting E3 may see specific queries fall off an index with no schema +change. Relevant to `tenant-engine` in particular, which sits on `flex-auth`'s +synchronous authorization path and just requested a 5s statement timeout on +exactly that basis. + +**Proposed addition:** the E3 provisioning contract (ADR §18.6) should require +an EXPLAIN comparison before and after enabling RLS, not just a correctness +probe. + +## 4. The verification gap has off-the-shelf answers + +ADR §13.2 calls the missing E2/E3 evidence artifacts the framework's largest +live gap and asks who should build them. The research says: less building than +expected. + +- **pgTAP** is the established way to assert RLS behaviour in CI. "RLS policies + should be tested in CI with pgTAP, not just in dev, since policy regressions + are silent (no query error, just wrong data returned), and automated + cross-tenant isolation tests are the only reliable guard" + ([Blair Jordan — testing RLS policies with pgTAP](https://blair-devmode.medium.com/testing-row-level-security-rls-policies-in-postgresql-with-pgtap-a-supabase-example-b435c1852602), + [MakerKit — database testing with pgTAP](https://makerkit.dev/docs/next-supabase-turbo/development/database-tests)). +- **pgrls** is a purpose-built static analyzer: 67 lint rules for tenant and + per-user row-scoping bugs, performance traps and hygiene, 20 auto-fixable, a + semantic policy-diff command for CI gating, and a pytest plugin for isolation + tests ([pgrls on GitHub](https://github.com/pgrls/pgrls)). + +The named common failures match ours exactly: "a forgotten role, a pool reusing +connections with the wrong tenant context, or an admin path bypassing RLS +entirely" ([Kawshik](https://kawshik.dev/blog/multi-tenant-rag-pgvector-postgres-rls.html)). + +**Against our model:** the E3 evidence artifact in §13 ("RLS policies present on +every tenant-owned table; probe that a session without the GUC reads nothing; +probe that a wrong GUC reads nothing") is well specified and is roughly what +pgrls lints for plus what pgTAP asserts. `rapp-postgres` already runs +`scripts/isolation-test.sh` with 15 probes against a throwaway PostgreSQL 16 in +Docker — the same harness shape, pointed at the consumer boundary. Extending it +to the tenant boundary is incremental, not novel. + +**This materially changes the answer to open question §18.3.** The E3 half is +tooling-assisted and cheap. The E2 half remains semantic and human (see plane A +§7). Splitting the question that way makes it assignable. + +## 5. The layered consensus: RLS as defence in depth, not as the only layer + +Vendor and community guidance converges on the same layering our ADR describes, +which is reassuring for the ladder's shape even where the details need fixing. + +- AWS documents RLS as the mechanism for pooled multi-tenant isolation and + pairs it with a non-owner application role + ([AWS — multi-tenant data isolation with PostgreSQL RLS](https://aws.amazon.com/blogs/database/multi-tenant-data-isolation-with-postgresql-row-level-security/)). +- OWASP lists RLS under database-level enforcement and application-layer ORM + filters as *complementary*, recommending "database-level isolation as + defense-in-depth" rather than as a replacement + ([OWASP Multi-Tenant Security Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/Multi_Tenant_Security_Cheat_Sheet.html)). +- The application-layer half is a custom ORM/repository that automatically + appends tenant filters and injects `tenant_id` on write (ibid.) — which is + precisely our E2. +- "Use `SECURITY INVOKER` for typical multi-tenant logic so that the caller's + RLS context applies"; reserve `SECURITY DEFINER` for deliberate, controlled + cross-tenant access + ([oneuptime](https://oneuptime.com/blog/post/2026-01-25-row-level-security-postgresql/view)). + +**Against our model:** the E ladder is ordinal — each rung replacing the last — +but practice treats E2 and E3 as **layers that coexist**. A service at E3 that +dropped its E2 choke point would be worse off, not better, because E3 alone +fails open under SQL injection (§2). + +**Proposed amendment:** state that E3 *presupposes* E2 rather than superseding +it, and that a claim of E3 requires the E2 evidence artifact as well. That is a +small change with real consequences for how the ladder is read — it converts +"move up a rung" into "add a layer", which is what the sources actually +describe. + +## 6. Summary against our ladder + +| Finding | Verdict | +|---|---| +| Any session can re-`SET` the GUC; E3 fails against SQL injection and app compromise | **Correction.** E3's "cannot trivially route around" overclaims. Add a threat-model table. | +| E3 is decisive against forgotten predicates — the common case | **Confirms the reversal.** Draft-2's direction is right, its wording is not. | +| Table owner bypasses RLS silently without `FORCE ROW LEVEL SECURITY` | **Specific risk.** Our `_migrate` role owns the tables it creates (ADR-0001). | +| `BYPASSRLS` roles are exempt | Provisioning contract must forbid it on leased roles. | +| Statement-pooling + `SET` returns other tenants' rows, only under production concurrency | **Correction.** Use `SET LOCAL` in-transaction; ADR §16's pooling claim is backwards. | +| RLS disables functional indexes using non-leakproof functions | **New.** Require an EXPLAIN comparison in the E3 contract. | +| pgTAP + pgrls (67 lint rules, policy-diff CI gate, pytest plugin) exist | **Adopt.** §18.3's E3 half is far cheaper than assumed. | +| Policy regressions are silent — wrong data, no error | **Confirms §12.** Guarding must target invisible failure. | +| Practice treats E2 and E3 as coexisting layers, not successive rungs | **Amend.** E3 should presuppose E2, not replace it. | + +## Sources + +- [kmoppel — Yes, Postgres can do session vars, but should you use them?](https://kmoppel.github.io/2025-06-03-yes-postgres-can-do-session-variables-but-should-you-use-them/) +- [Daniel Imfeld — PostgreSQL row level security (notes)](https://imfeld.dev/notes/postgresql_row_level_security) +- [pganalyze — RLS in Postgres, security invoker views and why LEAKPROOF functions matter](https://pganalyze.com/blog/5mins-postgres-row-level-security-bypassrls-security-invoker-views-leakproof-functions) +- [AWS Database Blog — Multi-tenant data isolation with PostgreSQL Row Level Security](https://aws.amazon.com/blogs/database/multi-tenant-data-isolation-with-postgresql-row-level-security/) +- [oneuptime — How to secure multi-tenant data with RLS in PostgreSQL](https://oneuptime.com/blog/post/2026-01-25-row-level-security-postgresql/view) +- [sigmoid.at — Notes on PostgreSQL row level security](https://sigmoid.at/post/2021/09/23/postgresql_row_level_security/) +- [ShiftAsia — Why your database needs boundaries: an intro to PostgreSQL's RLS](https://shiftasia.com/community/why-your-database-needs-boundaries-an-intro-to-postgresqls-row-level-security-rls/) +- [pgEdge — It depends: using session variables in Postgres](https://www.pgedge.com/blog/it-depends-using-session-variables-in-postgres) +- [Kawshik — The multi-tenant RAG nightmare: securing pgvector with PostgreSQL RLS](https://kawshik.dev/blog/multi-tenant-rag-pgvector-postgres-rls.html) +- [pgrls — static analyzer for Postgres Row-Level Security](https://github.com/pgrls/pgrls) +- [Blair Jordan — Testing RLS policies in PostgreSQL with pgTAP](https://blair-devmode.medium.com/testing-row-level-security-rls-policies-in-postgresql-with-pgtap-a-supabase-example-b435c1852602) +- [MakerKit — Database testing with pgTAP](https://makerkit.dev/docs/next-supabase-turbo/development/database-tests) +- [Rico Fritzsche — Mastering PostgreSQL RLS for rock-solid multi-tenancy](https://ricofritzsche.me/mastering-postgresql-row-level-security-rls-for-rock-solid-multi-tenancy/) +- [pganalyze — Using Postgres row-level security in Ruby on Rails](https://pganalyze.com/blog/postgres-row-level-security-ruby-rails) +- [OWASP — Multi-Tenant Security Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/Multi_Tenant_Security_Cheat_Sheet.html) diff --git a/research/2026-08-17-adr008-plane-i-identity.md b/research/2026-08-17-adr008-plane-i-identity.md new file mode 100644 index 0000000..6d16a92 --- /dev/null +++ b/research/2026-08-17-adr008-plane-i-identity.md @@ -0,0 +1,234 @@ +# Plane I — Identity: prior knowledge and best practices + +> Research artefact — 2026-08-17. Supports `canon/architecture/adr-008-multi-tenancy-model.md` +> (draft-2), plane I of four. Surveys external practice for how multi-tenant +> systems name, carry, and validate tenant identity, and holds our `I0–I3` +> ladder against it. +> Companions: plane A (authorization), plane E (enforcement), plane P (placement). +> Index: `2026-08-17-adr008-multi-tenancy-research-index.md`. + +## 1. What the plane covers + +How a tenant is named, where that name is bound to a principal, how it travels +with a request, and how fresh the binding must be. Our ladder: + +| Level | State | +|---|---| +| I0 | No tenant concept | +| I1 | Local tenant notion, non-canonical | +| I2 | Canonical `tenant::`, validated; `tenant-engine` owns existence | +| I3 | I2 + capability roles honoured, live re-query for `aal2`-class decisions | + +## 2. Consensus practice: bind at registration, carry in the token + +The dominant pattern across vendor guidance is that tenant identity is bound +to the principal **at the identity provider**, not derived per-application. + +- AWS calls the unpacked token claims the **tenant context**, and frames it as + environment-like: it "influences how the SaaS application processes the + request" rather than being one parameter among many + ([SaaS Lens — preventing cross-tenant access](https://docs.aws.amazon.com/wellarchitected/latest/saas-lens/preventing-cross-tenant-access.html)). +- Binding is explicitly a **registration-time** responsibility: storing the + tenant identifier as a claim or attribute on the user in the IdP "shifts the + responsibility of mapping identities to tenants from each application to the + user registration process" + ([AWS Prescriptive Guidance — tenant onboarding and registration](https://docs.aws.amazon.com/prescriptive-guidance/latest/saas-multitenant-api-access-authorization/avp-design-onboarding-registration.html)). +- The recommendation is to define tenant context **at the IAM layer** rather + than relying on the application layer + ([AWS — 5 multi-tenant SaaS architecture best practices](https://aws.amazon.com/isv/resources/5-multi-tenant-saas-architecture/)). + +**Against our model:** this is what `I2` describes and what the IAM Profile +already mandates (`tenant` claim required on every token accepted by profile +consumers). We are aligned with consensus here, and `key-cape`/Keycloak +stamping the claim at issuance is the sanctioned shape. + +**Gap:** the ladder's I2 says "canonical identifiers used, validated". It does +not say *where the binding happens*. A service could satisfy I2 by parsing a +tenant id out of a request body. Consider tightening I2 to require the binding +be IdP-sourced, with request-supplied tenant ids explicitly at I1. + +## 3. Propagation is a separate discipline from naming + +Practice treats "the tenant id must reach every layer" as its own requirement, +not a consequence of having one: + +- A stable identifier should follow every request and stay consistent across + databases, services, logs, billing records, and background processes; without + a reliable tenant registry, other isolation controls are hard to enforce + consistently ([WorkOS — developer's guide to SaaS multi-tenant architecture](https://workos.com/blog/developers-guide-saas-multi-tenant-architecture)). +- Requests should execute in a tenant context established by middleware or + context variables, tagging each request with its owner + ([SSOJet — multi-tenant identity management](https://ssojet.com/blog/multi-tenant-identity-management)). +- Two leak surfaces get called out repeatedly and are **not** database + concerns: search indices (Elasticsearch queries that omit the tenant filter) + and cache keys that do not include the tenant id + ([WorkOS](https://workos.com/blog/developers-guide-saas-multi-tenant-architecture)). + +**Against our model:** ADR-008 has nothing to say about background jobs, +caches, or search indices. Our E ladder is described almost entirely in +database terms (tenant-keyed tables, RLS, per-tenant credentials). A service +could sit honestly at `E3` and still leak through a Redis key or an unfiltered +index. + +**Proposed amendment:** either widen the E-ladder evidence artifacts to cover +non-database stores, or add an explicit scope note that the E ladder describes +the *primary datastore* only and other stores are assessed separately. The +second is cheaper and more honest. + +## 4. Identity-plane placement is its own decision — and it has a known ceiling + +Identity providers face the same silo/pool question the P plane describes, and +the industry has converged on an answer with a number attached. + +- Keycloak offers **realm-per-tenant** (strong isolation, heavyweight) versus + **Organizations** (multiple tenants inside one realm, stable since Keycloak + 26, designed for the SaaS case) + ([Phase Two — multi-tenancy options in Keycloak](https://phasetwo.io/blog/multi-tenancy-options-keycloak/), + [Skycloak — multitenancy using Organizations](https://skycloak.io/blog/multitenancy-in-keycloak-using-the-organizations-feature/)). +- The operational verdict is blunt: "realm-per-tenant works for 5–20 tenants; + beyond that, you are managing an IAM platform, not building your product" + ([Izylife — Keycloak multi-tenancy: realm per tenant or Organizations?](https://izylifesolutions.com/articles/keycloak-multi-tenancy-b2b-iam/)). +- Auth0's equivalent is Organizations within a single Auth0 tenant, with + per-organization connections + ([KeycloakPro — Keycloak multi-tenancy with Organizations](https://keycloakpro.com/blog/keycloak-multi-tenancy-organizations-guide)). + +**Against our model:** ADR-008 treats placement as a property of *data +substrate* only. Identity has a placement axis too, it is live in our estate +(`key-cape`/Keycloak), and nobody has decided realm-per-tenant versus +Organizations. Given the tenant taxonomy includes `consumer` (private +individuals) and `family`, realm-per-tenant is as economically impossible here +as cluster-per-tenant — the 5–20 ceiling is well below our target tenant count. + +**Proposed amendment:** note in §3 of the ADR that the P plane, as written, +scopes to data substrate; identity-provider placement is a parallel decision +with the same silo/pool shape and should be recorded, probably by whoever owns +`key-cape`. + +## 5. Freshness: the tradeoff we already made, confirmed + +Our I3 requires live `tenant-engine` re-query for privileged, destructive, +credential-vending or `aal2`-class decisions, with the cached `tenant_roles` +claim acceptable for ordinary ones. External practice supports the split and +names the cost precisely. + +- The tradeoff is explicit: "tighter revocation controls often increase latency + and operational overhead, requiring organisations to balance immediate cutoff + against service reliability" + ([NHI Mgmt Group — JWT vs OAuth revocation](https://nhimg.org/faq/why-do-jwt-revocation-and-oauth-revocation-need-different-handling/)). +- For high-stakes decisions the sanctioned answer is opaque tokens plus + introspection (RFC 7662) — "a real kill switch at the cost of a lookup on the + hot path" ([oneuptime — how to handle JWT revocation](https://oneuptime.com/blog/post/2026-02-02-jwt-revocation/view)). +- Short lifetimes are the mainstream mitigation: "a 5-minute access token caps + how stale any resource server's view can be, which is the quiet reason the + default is 5 minutes and not an hour" (ibid.). Our IAM Profile's 5–30 minute + service/agent token lifetime sits inside this norm. +- Known failure mode directly relevant to us: controls "break down when + services cache JWT validation results" (ibid.). Our hybrid explicitly permits + caching for ordinary decisions, so the boundary between ordinary and + high-stakes is load-bearing and must be enforced, not documented. +- **Token versioning** is an alternative we do not use: a `token_version` per + subject, bumped on change, rejecting stale tokens service-side + ([TechNet Experts — JWT token versioning](https://www.technetexperts.com/jwt-token-versioning-stale-data/)). + It gives cheap global invalidation without a per-request lookup, at the cost + of a version read. Worth considering as a middle rung between our cached + claim and full live re-query. + +**Against our model:** we match best practice and, unusually, we wrote the +boundary down. The gap is evidential rather than architectural — our I3 +evidence artifact ("live re-query demonstrated on an `aal2`-class path; cached +path shown not to be used there") is exactly right and does not exist yet for +any service. + +## 6. Finding: our tenant identifier encodes a mutable attribute + +This is the significant new finding, and it is a critique of ratified canon +rather than of ADR-008. + +Our identifier is `tenant::`, where `` is an +onboarding-risk / entity-shape classification from ADR-0013: `trial`, +`friendly`, `single`, `small` (≤10 employees), `medium` (≤100), `large` +(≤1000), `enterprise` (1001+), `consumer`, `family`, `community`, +`association`, `agentic`. + +Several of those are **explicitly headcount bands**, and headcount changes. A +tenant onboarded as `tenant:small:acme` that grows past ten employees is +either misclassified forever or must be renamed — and renaming a tenant +identifier is a estate-wide data migration, since the identifier is the key +shared with `user-engine`, appears in OpenBao paths (`tenants//...`), +and is carried in every token. + +External practice is unambiguous: + +- "Avoid encoding mutable data or business logic in the identifier" — the + **ID Entanglement Effect**, where composite ids encode business relationships + or mutable state that is later parsed + ([Cogna — the unreasonable power of identifiers](https://cogna.co/blog/the-unreasonable-power-of-identifiers/)). +- Opaque identifiers "decouple identification from descriptive attributes, + thereby enhancing stability against entity changes" + ([Grokipedia — unique identifier](https://grokipedia.com/page/Unique_identifier)). +- The FAIR/persistent-identifier literature reaches the same conclusion: make + identifiers opaque from the start, so that when a name changes "just the + `rdfs:label` and some other properties may change — but not the URI" + ([Cool URIs for FAIR Knowledge Graphs](https://arxiv.org/pdf/2407.09237)). +- A structured id is acceptable *only if treated as opaque* — "the key is that + we treat it as an unchanging reference and don't try to parse meaning from + its structure" ([Cogna](https://cogna.co/blog/the-unreasonable-power-of-identifiers/)). + +The IAM Profile half-anticipates this. It states that capability roles were +deliberately separated from the identifier so "a tenant's roles may change +without renaming its identifier", and that grouping "does not gate which +capability roles a tenant may hold". The same reasoning applies to grouping +itself and was not carried through. + +**Two honest readings**, and this needs an owner decision rather than an +assertion from this document: + +1. **Grouping is onboarding-time and frozen by design.** It records the shape + of the entity *when it was onboarded*, not its current headcount, and is + never expected to change. If so, this should be stated explicitly in + ADR-0013 — because every reader will otherwise assume `small` means + currently-small, and some system will eventually make a decision on it. +2. **Grouping is current classification.** Then the identifier encodes a + mutable attribute, the estate has a rename problem it has not hit yet + because tenant counts are tiny, and the fix (opaque id + grouping as a + mutable property on the tenant record) gets more expensive every month. + +Either way, `rapp-postgres` is already correct to treat these as opaque +strings — ADR-0001 §4 prohibits the storage platform from parsing them. The +risk sits with any consumer that *does* parse the grouping segment. + +**Proposed action:** raise as a question to `tenant-engine` and NetKingdom +canon rather than folding into ADR-008. It is out of ADR-008's scope but +materially affects plane I's top rung. + +## 7. Summary against our ladder + +| Finding | Verdict | +|---|---| +| Bind tenant at IdP, carry in token, treat as context not parameter | **Aligned.** Matches I2 + IAM Profile. | +| I2 does not require the binding be IdP-sourced | **Gap.** Tighten wording; request-supplied ids belong at I1. | +| Propagation must cover caches, search indices, background jobs | **Gap.** ADR-008's ladders are datastore-shaped. Add a scope note. | +| Identity-provider placement is its own silo/pool decision, ceiling ~5–20 tenants for realm-per-tenant | **Missing.** P plane scopes to data only; identity placement is undecided and unowned. | +| Hybrid cached-claim / live-re-query for high-stakes | **Aligned and unusually well specified.** Evidence artifact does not exist yet. | +| Token versioning as a middle option | **Unconsidered.** Cheap global invalidation without hot-path lookup. | +| Identifier encodes mutable headcount bands | **Finding.** Contradicts identifier-design consensus; needs an ADR-0013 clarification. | + +## Sources + +- [AWS SaaS Lens — Preventing cross-tenant access](https://docs.aws.amazon.com/wellarchitected/latest/saas-lens/preventing-cross-tenant-access.html) +- [AWS Well-Architected SaaS Lens (PDF)](https://docs.aws.amazon.com/pdfs/wellarchitected/latest/saas-lens/wellarchitected-saas-lens.pdf) +- [AWS Prescriptive Guidance — Tenant onboarding and user tenant registration](https://docs.aws.amazon.com/prescriptive-guidance/latest/saas-multitenant-api-access-authorization/avp-design-onboarding-registration.html) +- [AWS — 5 multi-tenant SaaS architecture best practices](https://aws.amazon.com/isv/resources/5-multi-tenant-saas-architecture/) +- [AWS Security Blog — SaaS tenant isolation with ABAC using STS support for tags in JWT](https://aws.amazon.com/blogs/security/saas-tenant-isolation-with-abac-using-aws-sts-support-for-tags-in-jwt/) +- [WorkOS — The developer's guide to SaaS multi-tenant architecture](https://workos.com/blog/developers-guide-saas-multi-tenant-architecture) +- [SSOJet — Multi-tenant identity management for SaaS](https://ssojet.com/blog/multi-tenant-identity-management) +- [Phase Two — Understanding multi-tenancy options in Keycloak](https://phasetwo.io/blog/multi-tenancy-options-keycloak/) +- [Skycloak — Multitenancy in Keycloak using the Organizations feature](https://skycloak.io/blog/multitenancy-in-keycloak-using-the-organizations-feature/) +- [Izylife — Keycloak multi-tenancy: realm per tenant or Organizations?](https://izylifesolutions.com/articles/keycloak-multi-tenancy-b2b-iam/) +- [KeycloakPro — Keycloak multi-tenancy with Organizations](https://keycloakpro.com/blog/keycloak-multi-tenancy-organizations-guide) +- [oneuptime — How to handle JWT revocation](https://oneuptime.com/blog/post/2026-02-02-jwt-revocation/view) +- [NHI Management Group — Why JWT revocation and OAuth revocation need different handling](https://nhimg.org/faq/why-do-jwt-revocation-and-oauth-revocation-need-different-handling/) +- [TechNet Experts — How to fix stale JWTs via token versioning](https://www.technetexperts.com/jwt-token-versioning-stale-data/) +- [Cogna — The unreasonable power of identifiers](https://cogna.co/blog/the-unreasonable-power-of-identifiers/) +- [Grokipedia — Unique identifier](https://grokipedia.com/page/Unique_identifier) +- [Cool URIs for FAIR Knowledge Graphs (arXiv 2407.09237)](https://arxiv.org/pdf/2407.09237) diff --git a/research/2026-08-17-adr008-plane-p-placement.md b/research/2026-08-17-adr008-plane-p-placement.md new file mode 100644 index 0000000..7c6b064 --- /dev/null +++ b/research/2026-08-17-adr008-plane-p-placement.md @@ -0,0 +1,244 @@ +# Plane P — Placement: prior knowledge and best practices + +> Research artefact — 2026-08-17. Supports `canon/architecture/adr-008-multi-tenancy-model.md` +> (draft-2), plane P of four. Surveys external practice for which substrate +> holds a tenant's data, and holds our `P0–P4` ladder against it. +> **Contains strong external validation of draft-2's reframe, and one +> expressiveness failure in the P ladder itself.** +> Companions: plane I (identity), plane A (authorization), plane E (enforcement). + +## 1. What the plane covers + +Which physical substrate holds a tenant's data, who decides, and what moving +costs. Our ladder: + +| Level | Shape | +|---|---| +| P0 | Shares a database with another consumer | +| P1 | Database per consumer, shared cluster | +| P2 | Dedicated cluster per consumer | +| P3 | Dedicated cluster per tenant | +| P4 | P3 plus separate region or jurisdiction | + +## 2. Validation: "isolation is a spectrum, not a discrete property" + +The operator's instruction to reframe draft-1 from a fixed model into a +graduated framework is not merely defensible — it is Microsoft's stated +position, in almost the same words: + +> "Instead of viewing isolation as a discrete property, consider it a spectrum. +> You can deploy components of your architecture that are more isolated or less +> isolated than other components in the same architecture, depending on your +> requirements." +> — [Azure Architecture Center — Tenancy models for a multitenant solution](https://learn.microsoft.com/en-us/azure/architecture/guide/multitenant/considerations/tenancy-models) + +The same page decomposes isolation across tiers exactly as our planes do — +"you can use different levels of isolation for each tier", with a shared UI +tier, a shared middle tier, and isolated databases given as the worked example. + +**Against our model:** draft-2's core move is externally corroborated by the +most rigorous public taxonomy available. Worth citing in ADR §15, because it +converts "we decided to do it this way" into "this matches the reference +guidance", which matters for reviewers who will ask whether we invented a +private vocabulary. + +## 3. Validation: our E↔P coupling claim is independently confirmed + +ADR §3.2 asserts that the planes couple at the top — `E4` is only reachable at +`P3` or above — and §4.4 asserts that `P0 → P1 → P2` does not raise the E +level. Microsoft states the mechanism plainly: + +> "A key difference between tenants and deployments is how isolation is +> enforced. When multiple tenants share a single deployment (a set of +> infrastructure), you typically rely on your application code and a tenant +> identifier that's in a database to keep each tenant's data separate. When +> tenants have their own dedicated deployments, they have their own +> infrastructure, so it might be less important for your code to account for a +> multitenant environment." (ibid.) + +That is our E/P relationship derived independently: shared deployment ⇒ +enforcement is application code (our E1–E3); dedicated deployment ⇒ enforcement +is structural (our E4). + +## 4. Finding: our P ladder cannot express per-tenant placement, and our own commercial section requires it + +This is the significant gap. + +Azure names two partitioning axes that our single ladder collapses: + +- **Vertically partitioned deployments** — "a combination of single-tenant and + multitenant deployments. For example, you might have most of your customers' + data and application tiers on multitenant infrastructures, but you deploy + single-tenant infrastructures for customers who require higher performance or + data isolation." Explicitly monetisable: "You can even charge customers a + higher rate to use a single-tenant deployment." (ibid.) +- **Horizontally partitioned deployments** — shared application tier, dedicated + databases per tenant, targeted at the component that carries the load: "if + you identify that specific components cause most of the load on your system, + then you can deploy separate components for each tenant." (ibid.) + +Our P ladder assigns **one level per service**. `tenant-engine` is `P1`. +`audit-core` is `P1`. There is no way to say *"P1 for most tenants, P3 for the +three enterprise tenants who bought isolation"* — which is precisely the +vertically partitioned model, and precisely what ADR §11's minimum-level plan +tiers imply. + +The contradiction is internal to draft-2: §11.1 says "a tier may require +`E3 P2`", but if only some tenants hold that tier, the delivering service +occupies two P levels at once. The posture vector as specified cannot represent +that state, so a service in it would have to either overclaim (declare P2 for +everyone) or underclaim (declare P1 and misrepresent what the tier delivers). +Both are §6 violations. + +**Proposed amendment.** Either: + +1. **Make P a per-tenant-class level rather than a per-service one** — the + vector carries a default plus exceptions: + ```yaml + tenancy: + current: { I: 2, A: 3, E: 2, P: 1 } + placement_exceptions: + - tenants: ["tenant:enterprise:*"] + P: 3 + reason: "isolation tier" + ``` +2. Or **declare per-tenant placement out of scope** for the framework and say + so, accepting that plan tiers selling isolation are then unmodellable. + +Option 1 is more work and is the honest one, because the estate has already +anticipated tiered isolation. Azure also flags the cost: "your codebase needs +to be designed to support both multitenant and single-tenant deployments", and +migration between them must be planned for (ibid.) — which is exactly what our +§7 portability rules already provide, so we are better positioned for this than +most. + +## 5. Cell-based architecture: the pattern our ladder skips + +ADR §18.8 asks whether to evaluate cells. The literature gives enough to +sharpen the question. + +- A cell is "a complete, independent instance of the service" with **a fixed + maximum size** — compute, storage, supporting services and configuration for + a defined blast radius + ([Rackspace — cell-based architecture on AWS](https://www.rackspace.com/blog/cell-based-architecture-aws), + [AWS guidance for cell-based architecture](https://github.com/aws-solutions-library-samples/guidance-for-cell-based-architecture-on-aws)). +- Cells are logical partitions on a **partition key** — customer ID ranges or + tenant ID — and "the data required for or generated by each cell is also + partitioned, so that there is no replication of data between cells" (ibid.). +- The sizing argument is the point: "by capping the size of each cell… you can + potentially predict and address failures more easily. This leads to a higher + MTBF" (ibid.). And "it is important to identify the proper partition key to + minimize cross-cell interactions" (ibid.). +- Cells are the bulkhead pattern applied to tenancy — AWS files it under + fault isolation, REL10-BP04 + ([AWS Well-Architected — use bulkhead architectures to limit scope of impact](https://docs.aws.amazon.com/en_us/wellarchitected/2022-03-31/framework/rel_fault_isolation_use_bulkhead.html)). +- Azure's equivalent is the **Deployment Stamps pattern**: "each stamp serves a + predefined number of tenants, allowing you to scale almost linearly and serve + increasing numbers of tenants" + ([Azure — Deployment Stamps pattern](https://learn.microsoft.com/en-us/azure/architecture/patterns/deployment-stamp)). + Azure also calls deployments "supertenants or stamps" and describes the + tenant→deployment mapping table as a first-class component. + +**Against our model:** a cell is not a rung on our P ladder — it is a different +axis. P measures *how much is shared*; a cell measures *how large a shared unit +is allowed to get before you make another one*. Our `platform-pg` is, in cell +terms, an uncapped cell: §16 of the ADR computes a ceiling of roughly six +consumers but nothing enforces it, and there is no second cell to overflow +into. + +That reframes §18.8 usefully. The question is not "should we adopt cells" but +"**what is `platform-pg`'s declared maximum size, and what happens at the +boundary?**" — a question we can answer now, cheaply, and which the connection +ceiling already forces. + +Azure also names **bin packing** as the assignment strategy and stresses +maintaining "a record of your customers and the infrastructure where their data +and applications reside so that you can route their traffic to the correct +location" — a tenant→deployment mapping table. We have no such record; the +placement facts in ADR §1 were assembled by inspection. + +## 6. Placement is a commercial decision, and the guidance says so first + +Our §11 treats commercial expression as a constraint to be managed. Azure puts +it the other way round: + +> "Selecting a tenancy model isn't only a technical decision. It's also a +> commercial decision." — considering business objectives, compliance, scale, +> automation capacity, and SLAs (ibid.). + +And on the economics, matching our §14 rejection of R3/P3-everywhere: "if a +single tenant requires a specific infrastructure cost, 100 tenants probably +require 100 times that cost", with ongoing maintenance time-consuming at scale +(ibid.). Conversely single-tenant is appropriate "if you expect that your +business will have only a few customers… even though it's more costly". + +**Against our model:** ADR §8.2 proposes placement ownership to +`railiance-platform`, an operations-shaped repo. This guidance suggests +placement policy is *jointly* commercial and technical — which strengthens the +case that `adaptive-pricing` must be a co-signer on §18.2, not only on §18.5. + +## 7. Our tenant taxonomy is a recognised hard case + +Azure's B2C guidance describes our exact situation: + +> "If your customers are consumers, it's often more complicated to relate +> customers, tenants, and users… consider whether your solution might be used +> by families, groups of friends, clubs, associations, or other groups that +> might need to access and manage their data together. For example, a music +> streaming service might support both individual users and families, and it +> might treat each of these account types differently when it separates them +> into tenants." (ibid.) + +Our ADR-0013 taxonomy contains `consumer`, `family`, `community`, and +`association` verbatim. It also notes that if tenants are individuals, "you +might need to consider how you handle personal data and about the data +sovereignty laws in each jurisdiction" — which is our `P4` and, combined with +plane A §6, the tenant-deletion gap. + +## 8. Testing placement is a named obligation + +> "Whichever isolation model you choose, be sure to test your solution to verify +> that one tenant's data isn't accidentally leaked to another **and that any +> noisy neighbor outcomes are acceptable**." (ibid.) + +Two evidence artifacts, not one. ADR §13 covers the leak half (P1–P4: +"provisioning declaration plus the platform's own isolation probes") and says +nothing about noisy-neighbour behaviour. Azure suggests deliberate fault +injection (Chaos Studio) to verify resilience. + +**Proposed amendment to §13:** add a noisy-neighbour evidence artifact for P1 +and P2 — a demonstration that one consumer saturating its connection or CPU +allowance does not breach another's. `rapp-postgres` already sets +database-scoped `connection_limit`, `statement_timeout` and +`idle_in_transaction_session_timeout` precisely for this, and ADR-0001 §3 argues +resource exhaustion is an isolation failure — but nothing proves the limits +hold under load. + +## 9. Summary against our ladder + +| Finding | Verdict | +|---|---| +| "Isolation is a spectrum, not a discrete property"; different tiers at different levels | **Validates draft-2.** Cite in §15. | +| Shared deployment ⇒ app-code enforcement; dedicated ⇒ structural | **Confirms E↔P coupling** independently. | +| Vertical partitioning: some tenants pooled, some dedicated, explicitly monetised | **Gap.** P is per-service; cannot express per-tenant placement that §11 requires. | +| Horizontal partitioning: shared app tier, per-tenant DB for the hot component | **Unmodelled.** Our P assumes whole-service granularity. | +| A cell has a **fixed maximum size**; stamps scale near-linearly | **Reframes §18.8.** Ask "what is `platform-pg`'s declared max size?", not "should we adopt cells". | +| Tenant→deployment mapping table is a first-class component | **Missing.** Our placement facts were assembled by inspection. | +| Tenancy model is a commercial decision as much as technical | **Amend §8.2.** `adaptive-pricing` should co-sign placement ownership. | +| B2C/family/association tenancy is a recognised hard case | Our taxonomy hits it verbatim; jurisdiction concerns feed P4. | +| Test for leakage **and** for acceptable noisy-neighbour outcomes | **Amend §13.** No noisy-neighbour evidence artifact exists. | + +## Sources + +- [Azure Architecture Center — Tenancy models for a multitenant solution](https://learn.microsoft.com/en-us/azure/architecture/guide/multitenant/considerations/tenancy-models) +- [Azure Architecture Center — Architectural approaches for a multitenant solution](https://learn.microsoft.com/en-us/azure/architecture/guide/multitenant/approaches/overview) +- [Azure Architecture Center — Deployment Stamps pattern](https://learn.microsoft.com/en-us/azure/architecture/patterns/deployment-stamp) +- [Azure Architecture Center — Considerations for updating a multitenant solution](https://learn.microsoft.com/en-us/azure/architecture/guide/multitenant/considerations/updates) +- [AWS Well-Architected — REL10-BP04 Use bulkhead architectures to limit scope of impact](https://docs.aws.amazon.com/en_us/wellarchitected/2022-03-31/framework/rel_fault_isolation_use_bulkhead.html) +- [AWS Well-Architected — How do you use fault isolation to protect your workload?](https://wa.aws.amazon.com/wellarchitected/2020-07-02T19-33-23/wat.question.REL_10.en.html) +- [AWS Solutions Library — Guidance for cell-based architecture on AWS](https://github.com/aws-solutions-library-samples/guidance-for-cell-based-architecture-on-aws) +- [Rackspace — Cell-based architecture on AWS](https://www.rackspace.com/blog/cell-based-architecture-aws) +- [Ryan Harrison — Cell-based architecture engineering guidelines](https://guidelines.ryanharrison.co.uk/architecture/cell-architecture/) +- [Anudeep Balla — The guide to cell-based architecture](https://medium.com/@anudeepballa7/the-guide-to-cell-based-architecture-58cf607f9754) +- [AWS Well-Architected SaaS Lens (PDF)](https://docs.aws.amazon.com/pdfs/wellarchitected/latest/saas-lens/wellarchitected-saas-lens.pdf) +- [hidekazu-konishi — AWS SaaS multi-tenant architecture guide: tenant isolation, pool and silo models](https://hidekazu-konishi.com/entry/aws_saas_multi_tenant_architecture_guide.html)