diff --git a/canon/architecture/adr-007-workplan-identity-and-repo-worker-topology.md b/canon/architecture/adr-007-workplan-identity-and-repo-worker-topology.md index a80efb2..55f8cbc 100644 --- a/canon/architecture/adr-007-workplan-identity-and-repo-worker-topology.md +++ b/canon/architecture/adr-007-workplan-identity-and-repo-worker-topology.md @@ -82,7 +82,7 @@ nothing prevents number reuse. **1. A workplan identifier is globally unique.** `PREFIX-WP-NNNN` names exactly one workplan across the entire fleet, for all time. -> **Amended 2026-08-17 by `ADR-011` decision 2.** Uniqueness and forward-only +> **Amended 2026-08-17 by `ADR-009` decision 2.** Uniqueness and forward-only > allocation are **namespace-scoped**, not global; global identity is the pair > `(namespace, identifier)`, written `PREFIX-WP-NNNN@namespace` when foreign. > Global sequential allocation would require a central coordinator — the exact @@ -103,7 +103,7 @@ It also inverts ADR-001 — a file carrying a hub's private key is the file holding hub state. *Target state (C2).* `state_hub_workstream_id` and `state_hub_task_id` become -**deterministic**: UUIDv5 derived from the workplan identifier. Per `ADR-011` +**deterministic**: UUIDv5 derived from the workplan identifier. Per `ADR-009` decision 3 the derivation input is the pair `(namespace, identifier)`, not the identifier alone — deriving from the identifier alone would make two forks holding unrelated work under the same number compute the same UUID. Every instance computes the same value independently, writeback @@ -120,11 +120,11 @@ Decision 1 must therefore be enforced before derivation ships. identifiers into repository files. Other instances may read, project, and serve, but must not mint workplan or task UUIDs into git-tracked files. -> **Corrected 2026-08-17, superseded by `ADR-010` decisions 1–3.** This decision +> **Corrected 2026-08-17, superseded by `ADR-008` decisions 1–3.** This decision > originally described workstation hubs as "development read replicas". That was > wrong on both counts: the workstation instance was not a replica, and it was > the *larger* of the two, holding 306 more workplans than the primary. The two -> instances were peer databases. `ADR-010` establishes the central hub as +> instances were peer databases. `ADR-008` establishes the central hub as > authoritative and local instances as rebuildable caches, which is what makes > this interim rule coherent. diff --git a/canon/architecture/adr-010-hub-authority-and-local-cache-model.md b/canon/architecture/adr-008-hub-authority-and-local-cache-model.md similarity index 99% rename from canon/architecture/adr-010-hub-authority-and-local-cache-model.md rename to canon/architecture/adr-008-hub-authority-and-local-cache-model.md index a5ac00e..69801e0 100644 --- a/canon/architecture/adr-010-hub-authority-and-local-cache-model.md +++ b/canon/architecture/adr-008-hub-authority-and-local-cache-model.md @@ -1,5 +1,5 @@ --- -id: ADR-010 +id: ADR-008 type: architecture-decision-record title: "Hub Authority, Local Cache, and the Two Kinds of Hub Data" status: proposed @@ -8,7 +8,7 @@ date: "2026-08-17" tags: ["architecture", "state-hub", "repo-manager", "hub-core", "authority", "cache", "offline", "adr-001"] --- -# ADR-010: Hub Authority, Local Cache, and the Two Kinds of Hub Data +# ADR-008: Hub Authority, Local Cache, and the Two Kinds of Hub Data ## Status diff --git a/canon/architecture/adr-011-federated-namespaces-and-reconciliation-limits.md b/canon/architecture/adr-009-federated-namespaces-and-reconciliation-limits.md similarity index 97% rename from canon/architecture/adr-011-federated-namespaces-and-reconciliation-limits.md rename to canon/architecture/adr-009-federated-namespaces-and-reconciliation-limits.md index 907f8aa..ac4fe36 100644 --- a/canon/architecture/adr-011-federated-namespaces-and-reconciliation-limits.md +++ b/canon/architecture/adr-009-federated-namespaces-and-reconciliation-limits.md @@ -1,5 +1,5 @@ --- -id: ADR-011 +id: ADR-009 type: architecture-decision-record title: "Federated Namespaces, Qualified Identifiers, and the Limits of Reconciliation" status: proposed @@ -8,11 +8,11 @@ date: "2026-08-17" tags: ["architecture", "federation", "namespace", "identity", "reconciliation", "fos", "adr-007", "adr-008"] --- -# ADR-011: Federated Namespaces, Qualified Identifiers, and the Limits of Reconciliation +# ADR-009: Federated Namespaces, Qualified Identifiers, and the Limits of Reconciliation ## Status -Proposed. Amends `ADR-007` decision 1 and extends `ADR-010` decision 4. +Proposed. Amends `ADR-007` decision 1 and extends `ADR-008` decision 4. ## Context @@ -77,7 +77,7 @@ lineage, a shared identifier is ambiguous between *the same work, diverged* and *different work, colliding* — a distinction that cannot be reconstructed after the fact. -**5. Mutable state is derived from an append-only log.** Extending `ADR-010` +**5. Mutable state is derived from an append-only log.** Extending `ADR-008` decision 4: status and other mutable fields are a fold over append-only events rather than fields to be reconciled. Reconciliation then becomes union-of-events plus re-derivation — deterministic, order-independent, and requiring no @@ -167,6 +167,6 @@ absolute. Settle before implementation; the notation works either way. escalation, sovereignty by default, rebuildability - ADR-001 — workplans originate as repo files - ADR-007 — identifier uniqueness and derived identifiers (amended here) -- ADR-010 — hub authority, local cache, and the two kinds of hub data +- ADR-008 — hub authority, local cache, and the two kinds of hub data - `CUST-WP-0058` — instance-per-client tenancy - `SHR-INV-0001` — 425-item disposition inventory, T3 cost evidence diff --git a/research/2026-08-17-adr008-multi-tenancy-research-index.md b/research/2026-08-17-adr008-multi-tenancy-research-index.md deleted file mode 100644 index 1712b3d..0000000 --- a/research/2026-08-17-adr008-multi-tenancy-research-index.md +++ /dev/null @@ -1,122 +0,0 @@ -# ADR-008 multi-tenancy — external research index - -> Research artefact — 2026-08-17. Four deep-research digests, one per plane of -> `canon/architecture/adr-008-multi-tenancy-model.md` (draft-2), surveying prior -> knowledge and best practice and holding our ladders against it. -> -> - `2026-08-17-adr008-plane-i-identity.md` -> - `2026-08-17-adr008-plane-a-authorization.md` -> - `2026-08-17-adr008-plane-e-enforcement.md` -> - `2026-08-17-adr008-plane-p-placement.md` -> -> Purpose: feed draft-3. Nothing here amends the ADR by itself. - -## 1. Headline: the reframe is externally corroborated - -The operator's draft-1 → draft-2 instruction — stop specifying one model, build -a framework of graduated levels — matches the most rigorous public guidance -almost verbatim: - -> "Instead of viewing isolation as a discrete property, consider it a spectrum. -> You can deploy components of your architecture that are more isolated or less -> isolated than other components in the same architecture." -> — [Azure Architecture Center, Tenancy models](https://learn.microsoft.com/en-us/azure/architecture/guide/multitenant/considerations/tenancy-models) - -Microsoft also derives our E↔P coupling independently: shared deployment ⇒ -enforcement lives in application code; dedicated deployment ⇒ enforcement is -structural. Draft-2's shape is sound. What follows is about its details. - -## 2. Corrections to draft-2 — things it currently states wrongly - -Ranked by how badly they mislead. - -| # | Correction | Plane | Where | -|---|---|---|---| -| C1 | **E3 does not prevent an application from routing around it.** Any session can re-`SET` the GUC; SQL injection or a compromised process defeats it entirely. E3 is strong against *accident*, weak against *compromise*. The ADR's phrase "the application cannot trivially route around" overclaims — the exact fault §6 prohibits. | E | §4.3 | -| C2 | **The pooling claim in §16 is backwards.** Transaction-scoped context (`SET LOCAL`) is what makes E3 *safe* under pooling; **statement**-level pooling is what breaks it — returning other tenants' rows, only under production concurrency. The ADR says E3 "forecloses aggressive transaction-level pooling". | E | §16 | -| C3 | **"Set at pool checkout" is the wrong scope.** Should be set transaction-locally inside an explicit transaction. | E | §4.3 | -| C4 | **E3 does not supersede E2 — it layers on it.** Practice treats them as coexisting; a service dropping E2 on reaching E3 is worse off, because E3 fails open under injection. Claiming E3 should require the E2 artifact too. | E | §4.3, §13 | -| C5 | **P cannot express per-tenant placement, which §11 requires.** A tier requiring `E3 P2` bought by only some tenants puts the service at two P levels at once. The posture vector cannot represent it, forcing an over- or under-claim. | P | §5, §11 | - -## 3. Gaps — real concerns no plane covers - -| # | Gap | Suggested home | -|---|---|---| -| G1 | **Tenant offboarding and deletion.** OWASP lists complete data deletion on offboarding as a MUST. `tenant-engine` has reversible retirement and explicitly no hard-delete; `platform-pg` retains backups 30 days. Falls between two repos' declared scopes. | New open question; a lifecycle stage across all four planes, not a fifth plane. | -| G2 | **Non-database stores.** Caches, search indices and background jobs are named leak surfaces. Our ladders are datastore-shaped throughout. | Scope note on the E ladder, or widened evidence artifacts. | -| G3 | **Identity-provider placement.** Realm-per-tenant vs Keycloak Organizations is the same silo/pool decision, live in our estate, undecided. Realm-per-tenant has a stated ceiling of 5–20 tenants. | Note in §3 that P scopes to data substrate; identity placement is parallel and unowned. | -| G4 | **Noisy-neighbour evidence.** Azure requires testing both that data does not leak *and* that noisy-neighbour outcomes are acceptable. §13 covers only leakage. | Add a P1/P2 evidence artifact. | -| G5 | **Tenant→deployment mapping record.** A first-class component in the reference guidance. Our placement facts in §1 were assembled by inspection. | Follows from G3/C5. | -| G6 | **Cell sizing.** `platform-pg` is an uncapped cell with a computed ~6-consumer ceiling and no overflow target. | Reframes §18.8: ask what its declared maximum size is. | -| G7 | **Tenant identifier encodes mutable headcount bands.** `tenant::` embeds `small`/`medium`/`large`, which change. Contradicts identifier-design consensus. | Not ADR-008's; raise to `tenant-engine` / NetKingdom canon re ADR-0013. | -| G8 | **Logging.** OWASP MUST: include tenant context in all log entries. Unmodelled. | Minor; fold into evidence artifacts. | - -## 4. Opportunities — cheaper or better than we assumed - -| # | Finding | Effect | -|---|---|---| -| O1 | **pgTAP + pgrls** (67 RLS lint rules, semantic policy-diff for CI gating, pytest isolation plugin) exist off the shelf. `rapp-postgres` already runs a 15-probe harness of the same shape. | The E3 half of open question §18.3 is far cheaper than assumed. Splits the question into a tractable mechanical half and a human half. | -| O2 | **OWASP Multi-Tenant Security Cheat Sheet** is the closest external baseline to our framework and is referenced nowhere in the estate. | Adopt as the named external baseline in ADR §15. | -| O3 | **OpenID AuthZEN Authorization API 1.0** reached Final Specification January 2026; Keycloak shipped experimental support May 2026. Demonstrated PDP interoperability across 5+ engines. | `flex-auth`'s bespoke `CheckRequest` is pre-standard. Propose an `A4` rung for standard-interface PDP delegation. | -| O4 | **Token versioning** as a middle rung between cached claim and live re-query. | Cheap global invalidation without a hot-path lookup. | -| O5 | **BOLA / OWASP API1.** Our tenant-boundary gap is the industry's #1 API vulnerability, top of the list since it launched. | Cite by name. Converts an internal judgement into an external consensus for reviewers with a security background. | - -## 5. Specific technical risks for the E3 implementation contract (§18.6) - -Concrete requirements the contract must carry, each sourced in the enforcement -digest: - -- `FORCE ROW LEVEL SECURITY` on every tenant-owned table. Without it the table - owner bypasses policies **silently** — and ADR-0001 already established that - objects created by `_migrate` are owned by `_migrate`, so our - migration role would bypass every policy on tables it created. -- No `BYPASSRLS` attribute on any leased role. -- `SECURITY INVOKER` for ordinary logic; `SECURITY DEFINER` reserved for - deliberate, reviewed cross-tenant access. -- Context set with `SET LOCAL` inside an explicit transaction, never - session-scoped. -- An `EXPLAIN` comparison before and after enabling RLS: functional indexes - using non-leakproof functions (`lower`, `enum_eq`) stop being used under RLS. - Relevant to `tenant-engine`, which sits on the synchronous authorization path - and just requested a 5s statement timeout. - -## 6. Recommended shape of draft-3 - -1. Apply corrections C1–C4 to §4.3, §13 and §16. These are the ones a reviewer - will otherwise catch, and C1 is the ADR overclaiming in exactly the way it - forbids. -2. Resolve C5 by making placement expressible per tenant class — a default plus - exceptions in the posture vector — or by declaring per-tenant placement out - of scope and accepting that isolation tiers become unmodellable. -3. Split open question §18.3 into a mechanical half (E1/E3, tooling exists) and - a semantic half (E2, adversarial review on a recorded cadence). Cross-tenant - findings "require a human tester who can set up separate tenant contexts and - compare the responses" — a green CI run is not E2 evidence. -4. Add G1 (offboarding/deletion) and G3 (identity placement) as open questions; - add G2 as a scope note. -5. Cite the external baselines — OWASP Multi-Tenant Security Cheat Sheet, Azure - tenancy models, AWS SaaS Lens — in §15, and BOLA/API1 in §13.2. -6. Reframe §18.8 from "evaluate cells" to "declare `platform-pg`'s maximum - size and its overflow target". -7. Route G7 to `tenant-engine` separately. It is a critique of ratified canon, - not of this ADR, and folding it in would overreach. - -## 7. Method and limits - -Four search-and-read passes, one per plane, against vendor architecture -guidance (AWS Well-Architected SaaS Lens, Azure Architecture Center), standards -bodies (OpenID Foundation), security baselines (OWASP), PostgreSQL practitioner -literature, and IdP vendor documentation. Full source lists are in each digest. - -Limits worth stating: - -- No primary academic sources beyond the Zanzibar lineage; this is an - engineering-practice survey, not a literature review. -- Vendor guidance is not neutral — AWS and Azure both describe patterns their - platforms sell. The Azure tenancy-models page is the most rigorous taxonomy - found and is used heavily in the P digest; readers should discount its - Azure-service-specific recommendations accordingly. -- Nothing here was tested against our own systems. Every claim about our estate - is drawn from repo documents, not from probing running infrastructure. -- The E digest's corrections are the highest-confidence findings, being - PostgreSQL mechanics rather than architectural opinion. diff --git a/research/2026-08-17-adr008-plane-a-authorization.md b/research/2026-08-17-adr008-plane-a-authorization.md deleted file mode 100644 index 91e522c..0000000 --- a/research/2026-08-17-adr008-plane-a-authorization.md +++ /dev/null @@ -1,224 +0,0 @@ -# Plane A — Authorization: prior knowledge and best practices - -> Research artefact — 2026-08-17. Supports `canon/architecture/adr-008-multi-tenancy-model.md` -> (draft-2), plane A of four. Surveys external practice for binding a request -> to the tenants it may act for, and holds our `A0–A3` ladder against it. -> Companions: plane I (identity), plane E (enforcement), plane P (placement). - -## 1. What the plane covers - -How a request is bound to the tenants its principal may act for, where that -decision is made, and how the decision point is kept honest. Our ladder: - -| Level | State | -|---|---| -| A0 | None, or tenant context not carried | -| A1 | Ad-hoc checks scattered through handlers | -| A2 | Single local authorization boundary; context bound once | -| A3 | Delegated to `flex-auth` as PDP, live re-query where the IAM Profile requires | - -## 2. This is the industry's #1 vulnerability class, by name - -The single most important external finding: what our framework calls a tenant -boundary failure, the security industry calls **Broken Object Level -Authorization**, and it is not a marginal concern. - -- BOLA is **API1** in the OWASP API Security Top 10 and "has held the top spot - since the list launched" - ([Decryption Digest — BOLA and IDOR testing guide](https://www.decryptiondigest.com/blog/bola-idor-api-authorization-test-remediation)). -- It is "consistently the most common, the most exploitable, and the most - damaging API vulnerability across bug bounty submissions, public disclosures, - and production penetration tests" (ibid.). -- The multi-tenant framing is explicit: "a single missing authorization check - can let one tenant access another's storage buckets, database records, or - configuration data" - ([Wiz — OWASP API security top 10](https://www.wiz.io/academy/api-security/owasp-api-security)). - -**Against our model:** ADR-008 §13.2 identifies the missing cross-tenant -evidence artifact as the framework's largest gap. This research upgrades that -from an internal judgement to an industry consensus: we have no test coverage -for the vulnerability class that ranks first globally. The ADR should cite -BOLA/API1 by name so the gap is legible to anyone who arrives with a security -background rather than an architecture one. - -## 3. OWASP has a Multi-Tenant Security Cheat Sheet, and we should adopt it wholesale - -This is the most directly applicable artefact found in the whole research -effort, and neither ADR-008 nor any repo document references it -([OWASP Multi-Tenant Security Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/Multi_Tenant_Security_Cheat_Sheet.html)). - -Its normative requirements, mapped to our ladders: - -| OWASP requirement | Our plane | Status | -|---|---|---| -| "Never trust client-supplied tenant IDs without validation" — derive from verified JWT claims | I2 | Implied, not stated. Tighten. | -| "Establish tenant context early in the request lifecycle (middleware/interceptor)"; bind to the authenticated session | A2 | **Matches A2 exactly.** | -| "Always validate that requested resources belong to the current tenant" | E1–E2 | Matches. | -| "Include tenant context in all log entries" | — | **Not covered by any plane.** | -| Tenant-isolated audit trails | — | `audit-core`'s concern; not in the framework. | -| "Ensure complete data deletion on tenant offboarding" | — | **Not covered. See §6.** | -| Prefix all cache keys with tenant identifiers | — | **Not covered** (see plane I §3). | -| Per-tenant rate limiting with tier-based quotas | — | Adjacent to `tenant-engine` guardrails. | -| Separate API keys per tenant | E4-adjacent | Our E4 is credential-based; consistent. | - -Explicitly rejected anti-patterns, two of which we should check ourselves -against: - -- **"Skipping tenant validation for internal services."** Our estate is mostly - internal service-to-service calls — `flex-auth` calls `tenant-engine` - synchronously on the authorization path, `user-engine` calls it for tenant - creation. Our A ladder describes a request arriving with a subject; it does - not say whether a *service* identity calling on behalf of a tenant carries - and revalidates tenant context. This is a real question for A3. -- **"Exposing sequential or guessable resource IDs."** Not assessed anywhere. - -**Proposed action:** reference the cheat sheet from ADR-008 as the external -baseline, and fold its uncovered requirements (logging, cache keys, offboarding -deletion) into either the ladders or an explicit scope exclusion. - -## 4. The PDP/PEP split is right, and there is now a standard for the wire - -Our A3 delegates decisions to `flex-auth` as PDP, which is the mainstream -architecture — a Policy Enforcement Point intercepts an action and queries a -Policy Decision Point -([DEV — RBAC vs ABAC vs ReBAC](https://dev.to/kanywst/rbac-vs-abac-vs-rebac-how-to-choose-and-implement-access-control-models-3i2d)). - -What is new since our contracts were written: - -- **OpenID AuthZEN Authorization API 1.0** was approved as an OpenID Final - Specification in January 2026 and published Standards Track in March 2026 - ([OpenID Foundation — Authorization API 1.0 final specification approved](https://openid.net/authorization-api-1-0-final-specification-approved/), - [spec](https://openid.github.io/authzen/)). -- Its purpose is exactly our seam: letting PDPs and PEPs "communicate - authorization requests and decisions to each other without requiring - knowledge of each other's inner workings" - ([Curity — introduction to AuthZEN](https://curity.io/resources/learn/authzen/)). -- Interoperability is demonstrated, not theoretical: one PEP implementation - driving 5+ different PDPs (Topaz, Axiomatics, OpenFGA) by switching endpoint - URLs alone - ([DEV — AuthZEN deep dive](https://dev.to/kanywst/authzen-authorization-api-10-deep-dive-the-standard-api-that-separates-authorization-decisions-1m2a)). -- Keycloak shipped experimental AuthZEN support in May 2026 - ([Keycloak — AuthZEN as experimental feature](https://www.keycloak.org/2026/05/authzen-as-experimental-feature)). - -**Against our model:** `flex-auth` uses a bespoke `CheckRequest` shape and a -bespoke action vocabulary (`tenant.update`, `tenant.guardrail.set`, …), with -action strings copied verbatim between repos to avoid re-derivation. That -coupling is exactly what AuthZEN standardises away. We are not wrong — we are -pre-standard, and the standard landed seven months ago. - -**Proposed amendment:** A3 currently says "delegated to `flex-auth` as PDP". -Consider distinguishing: - -- **A3** — externalised to a PDP (any protocol). -- **A4** — externalised over a standard interface (AuthZEN), so the PDP is - swappable and the PEP is not coupled to one engine's request shape. - -That gives the ladder somewhere to go and makes the current coupling visible -rather than invisible. It also reframes `key-cape`/Keycloak's AuthZEN support -as a fleet opportunity rather than an unrelated upstream feature. - -## 5. Model choice: our vocabulary is RBAC-shaped, our problem may be ReBAC-shaped - -- ReBAC originates in Google's **Zanzibar**, modelling authorization as a graph - of typed relationships, with authorization data stored as explicit tuples in - a dedicated datastore - ([AuthZed — introduction to Google Zanzibar](https://authzed.com/learn/google-zanzibar)). -- It is described as "well-suited to multi-tenant enterprise environments where - access patterns are relational rather than role-based", and specifically for - modelling "complex customer organizational structures while maintaining - strict data isolation" (ibid., - [AuthZed — ABAC vs ReBAC](https://authzed.com/learn/abac-vs-rebac-when-to-use-which)). -- Open implementations: SpiceDB, OpenFGA, Authzed - ([IAM Day by Day — Zanzibar & ReBAC](https://iamdaybyday.com/patterns/authorization/zanzibar/)). - -**Against our model:** our capability roles (`PLTF`/`IAM`/`VEN`/`CUS`) are -tenant-level RBAC, and `user-engine` owns memberships separately. The moment a -question like "may this user act for this tenant, given they are a member of an -organisation that administers it" arises, that is a relationship query, and -role-flattening will start to strain. Not a present problem — flagged because -the estate's own tenant taxonomy (`family`, `community`, `association`) is -overtly relational and will produce these questions. - -## 6. Finding: nothing in the framework covers tenant exit - -OWASP lists "ensure complete data deletion on tenant offboarding" as a **must** -([OWASP Multi-Tenant Security Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/Multi_Tenant_Security_Cheat_Sheet.html)). - -Our position today: `tenant-engine`'s lifecycle contract provides reversible -retirement and states plainly that **there is no hard-delete**. That is a -defensible design for an audit-bearing estate, but it means: - -- No plane in ADR-008 has anything to say about tenant exit, data deletion, or - the interaction with retention. -- Physical backups on `platform-pg` retain 30 days, so even a hypothetical - hard-delete leaves tenant data recoverable for a month — which is a GDPR - erasure question, not just an architecture one. -- `rapp-postgres` SCOPE explicitly excludes tenant identity, and - `tenant-engine` explicitly excludes storage. Deletion falls between them. - -**Proposed action:** this is a fifth concern the framework does not model, and -it is not obviously a fifth *plane* — it is a lifecycle stage cutting across -all four. Recommend adding it as an explicit open question rather than -inventing a plane for it. The estate's `gdpr_report` tooling in the hub -suggests someone has thought about this elsewhere; worth locating before -drafting. - -## 7. Finding: automated conformance testing may not be sufficient for this plane - -This challenges the recommendation I gave in the ADR review (§18.3, "build a -fleet conformance suite"). - -- "Finding cross-tenant vulnerabilities requires semantic understanding of what - data belongs to which context, and that understanding requires a human tester - who can set up separate tenant contexts and compare the responses." -- "Across 119 SaaS assessments, the highest-impact findings were concentrated - in access control, business logic, and tenant isolation, precisely the - categories that require context-aware human testing rather than automated - tool execution." - ([Bugstrix — multi-tenant SaaS security testing](https://bugstrix.com/blogs/multi-tenant-saas-security-testing-how-to-prevent-cross-tenant-data-leaks/)) - -**Against our model:** this does not invalidate the conformance suite — a -mechanical check that every tenant-owned table carries the key (E1), or that a -session without the GUC reads nothing (E3), is precisely the kind of structural -assertion automation does well. What it invalidates is the hope that a suite -*replaces* adversarial review at E2, where the boundary lives in business -logic and the failure is semantic. - -**Proposed amendment to §13:** split the evidence artifacts by what can be -automated. E1 and E3 are mechanical and belong in CI. E2 is semantic and needs -a periodic adversarial review with a recorded date, closer to a pen-test -cadence than a test suite. Claiming E2 on the strength of a green CI run would -be exactly the overclaim §6 of the ADR prohibits. - -## 8. Summary against our ladder - -| Finding | Verdict | -|---|---| -| Tenant boundary failure = BOLA = OWASP API1, top of the list since launch | **Reframe.** Cite by name; our largest gap is the industry's largest risk. | -| OWASP Multi-Tenant Security Cheat Sheet exists and is unreferenced | **Adopt.** Closest external baseline to our framework. | -| A2 matches OWASP's "establish context early, bind to session" | **Aligned.** | -| "Skipping tenant validation for internal services" is a named anti-pattern | **Check ourselves.** Our estate is mostly internal service-to-service. | -| PDP/PEP split is mainstream | **Aligned.** | -| AuthZEN 1.0 Final (Jan 2026) standardises the PDP wire; Keycloak experimental May 2026 | **Gap.** `flex-auth` is pre-standard and bespoke. Propose A4. | -| ReBAC/Zanzibar suits relational tenant structures | **Watch.** Our `family`/`community`/`association` groupings are relational. | -| Tenant offboarding deletion is a MUST we do not model | **Finding.** No plane covers exit; interacts with 30-day backup retention. | -| Cross-tenant testing needs human semantic review, not only automation | **Amend §13.** E1/E3 mechanical, E2 adversarial-review with a date. | - -## Sources - -- [OWASP — Multi-Tenant Security Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/Multi_Tenant_Security_Cheat_Sheet.html) -- [Wiz — OWASP API Security Top 10 risks and how to mitigate them](https://www.wiz.io/academy/api-security/owasp-api-security) -- [Decryption Digest — BOLA and IDOR testing guide](https://www.decryptiondigest.com/blog/bola-idor-api-authorization-test-remediation) -- [SecPortal — Broken Object Level Authorization (BOLA) guide](https://secportal.io/vulnerabilities/broken-object-level-authorization) -- [Bugstrix — Multi-tenant SaaS security testing: preventing cross-tenant data leaks](https://bugstrix.com/blogs/multi-tenant-saas-security-testing-how-to-prevent-cross-tenant-data-leaks/) -- [OpenID Foundation — Authorization API 1.0 Final Specification approved](https://openid.net/authorization-api-1-0-final-specification-approved/) -- [OpenID AuthZEN — Authorization API 1.0 specification](https://openid.github.io/authzen/) -- [OpenID Foundation — AuthZEN working group specifications](https://openid.net/wg/authzen/specifications/) -- [Curity — An introduction to Authorization Exchange (AuthZEN)](https://curity.io/resources/learn/authzen/) -- [DEV — AuthZEN Authorization API 1.0 deep dive](https://dev.to/kanywst/authzen-authorization-api-10-deep-dive-the-standard-api-that-separates-authorization-decisions-1m2a) -- [Keycloak — AuthZEN as an experimental feature (May 2026)](https://www.keycloak.org/2026/05/authzen-as-experimental-feature) -- [Auth0 — A guide to OpenID AuthZEN's Authorization API 1.0](https://auth0.com/blog/implementing-authzen-guide-openid-authorization-api/) -- [AuthZed — An introduction to Google Zanzibar and ReBAC](https://authzed.com/learn/google-zanzibar) -- [AuthZed — ABAC vs ReBAC: when to use which](https://authzed.com/learn/abac-vs-rebac-when-to-use-which) -- [IAM Day by Day — Zanzibar & relationship-based access control](https://iamdaybyday.com/patterns/authorization/zanzibar/) -- [DEV — RBAC vs ABAC vs ReBAC: how to choose](https://dev.to/kanywst/rbac-vs-abac-vs-rebac-how-to-choose-and-implement-access-control-models-3i2d) diff --git a/research/2026-08-17-adr008-plane-e-enforcement.md b/research/2026-08-17-adr008-plane-e-enforcement.md deleted file mode 100644 index 1968399..0000000 --- a/research/2026-08-17-adr008-plane-e-enforcement.md +++ /dev/null @@ -1,228 +0,0 @@ -# Plane E — Enforcement: prior knowledge and best practices - -> Research artefact — 2026-08-17. Supports `canon/architecture/adr-008-multi-tenancy-model.md` -> (draft-2), plane E of four. Surveys external practice for where the tenant -> boundary is mechanically enforced, and holds our `E0–E4` ladder against it. -> **This digest contains a correction to the ADR's own definition of E3.** -> Companions: plane I (identity), plane A (authorization), plane P (placement). - -## 1. What the plane covers - -Where the tenant boundary actually holds, and what defeats it. Our ladder: - -| Level | Mechanism | ADR claims it is defeated by | -|---|---|---| -| E0 | None | Anything | -| E1 | Tenant-keyed, filtered per query | One missing predicate | -| E2 | Single service-side choke point | Code bypassing the choke point | -| E3 | RLS keyed on a session GUC set at pool checkout | `SECURITY DEFINER`, missing policy, wrong GUC | -| E4 | Per-tenant credential and substrate | Platform-level compromise | - -## 2. Correction: E3's description is wrong, and the error matters - -ADR-008 draft-2 describes E3 as "platform-assisted: row-level security keyed on -a session GUC set at pool checkout, or an equivalent data-access layer **the -application cannot trivially route around**". - -That last clause is false for the GUC mechanism, and the literature is blunt -about it: - -> "Any session can `SET` any custom parameter. RLS policies that rely on -> `current_setting('app.tenant_id')` are only as secure as the application's -> control over SQL execution. Users with direct access can trivially -> `SET app.tenant_id = '999'` and bypass the policy." -> — [kmoppel — Yes, Postgres can do session vars, but should you use them?](https://kmoppel.github.io/2025-06-03-yes-postgres-can-do-session-variables-but-should-you-use-them/) - -And the corollary: "SQL injection could enable an attacker to issue their own -`SET` command, therefore accessing other customer's data. The session variable -based approach is only safe when you protect yourself against SQL injections" -(ibid.). - -**What this means for the framework.** Draft-1's instinct was not baseless -after all — but neither was draft-2's reversal. The honest resolution is that -E3's strength is **threat-model dependent**, and the ladder must say so: - -| Threat | Does E3 help? | -|---|---| -| Developer forgets a `WHERE tenant_id = ?` | **Yes, decisively.** This is the common case and the reason E3 exists. | -| A new code path bypasses the repository/choke point | **Yes.** The database still filters. | -| SQL injection reaching the connection | **No.** The attacker can re-`SET` the GUC. | -| Compromised application process | **No.** It holds a credential that can address all tenants. | - -So E3 is a strong control against **accident** and a weak one against -**compromise**. E4 is the only rung that holds against both, because the -credential itself cannot address another tenant's data. - -**Proposed amendment to §4.3 of the ADR:** replace "the application cannot -trivially route around" with an explicit threat-model row. The current wording -overclaims in precisely the way §6 of the ADR prohibits — an irony worth fixing -before reviewers find it. - -## 3. Three PostgreSQL mechanics that will bite our specific role model - -These are not generic warnings; each interacts with `rapp-postgres` ADR-0001 as -built. - -### 3.1 The table owner bypasses RLS silently — and our migration role owns tables - -> "By default, the table owner bypasses RLS policies unless the table is -> altered with `FORCE ROW LEVEL SECURITY`." -> — [pganalyze — RLS, security invoker views and LEAKPROOF functions](https://pganalyze.com/blog/5mins-postgres-row-level-security-bypassrls-security-invoker-views-leakproof-functions) - -> "The `FORCE` keyword is critical — without it, the table owner bypasses -> policies silently." -> — [Kawshik — securing pgvector with PostgreSQL RLS](https://kawshik.dev/blog/multi-tenant-rag-pgvector-postgres-rls.html) - -ADR-0001 already discovered — via `isolation-test.sh`, not review — that -**objects created by `_migrate` are owned by `_migrate`**, not by -`_owner`, because default privileges key on the creating role. That finding was -about grants. It has a second consequence nobody has drawn: under RLS, the -`_migrate` role would silently bypass every policy on every table it created. - -Add `BYPASSRLS` to the picture — roles with that attribute are exempt -([oneuptime — securing multi-tenant data with RLS](https://oneuptime.com/blog/post/2026-01-25-row-level-security-postgresql/view)) -— and the E3 provisioning contract needs to state explicitly: `FORCE ROW LEVEL -SECURITY` on every tenant-owned table, no `BYPASSRLS` on any leased role, and -the runtime role must not own the objects (which ADR-0001 already guarantees, -for a different reason). - -### 3.2 Connection pooling can silently serve the wrong tenant's rows - -This is the failure mode most likely to reach production undetected. - -> "RLS with `SET` or `SET LOCAL` will not work properly with pgbouncer in -> statement pooling mode; you will likely return rows for the wrong users and -> it may only happen in production when multiple people are hitting the app at -> once." — [Daniel Imfeld — PostgreSQL row level security](https://imfeld.dev/notes/postgresql_row_level_security) - -> "In highly concurrent environments with connection pooling (like pgbouncer), -> managing this context correctly is a major source of subtle, hard-to-debug -> bugs. If the context leaks between connections, so might the data." -> — [ShiftAsia — an intro to PostgreSQL's RLS](https://shiftasia.com/community/why-your-database-needs-boundaries-an-intro-to-postgresqls-row-level-security-rls/) - -The mitigation is consistent across sources: set the context **transaction-locally** -(`SET LOCAL` inside an explicit transaction), never session-locally, so pooling -mode cannot leak it. Our ADR says "set at pool checkout", which is the -session-scoped phrasing and the wrong one. - -**Against our model:** §16 of the ADR notes that E3 "forecloses aggressive -transaction-level pooling". That is backwards — transaction-scoped context is -what makes E3 *safe* under pooling; it is **statement**-level pooling that is -incompatible. Correct the scaling section as well as §4.3. - -This also sharpens the ADR's own §12 point about designing guarding for -invisible failure: an RLS context leak produces correct-looking rows for the -wrong tenant. No error, no log line, no crash. - -### 3.3 RLS silently disables some indexes - -> "Due to security concerns, functional indices using leakproof functions -> cannot be used for RLS-secured queries. Functions such as `lower` or -> `enum_eq` are not leakproof. This means indices such as -> `some_table(lower(email))` will not be used when RLS is active." -> — [sigmoid.at — notes on PostgreSQL row level security](https://sigmoid.at/post/2021/09/23/postgresql_row_level_security/) - -A service adopting E3 may see specific queries fall off an index with no schema -change. Relevant to `tenant-engine` in particular, which sits on `flex-auth`'s -synchronous authorization path and just requested a 5s statement timeout on -exactly that basis. - -**Proposed addition:** the E3 provisioning contract (ADR §18.6) should require -an EXPLAIN comparison before and after enabling RLS, not just a correctness -probe. - -## 4. The verification gap has off-the-shelf answers - -ADR §13.2 calls the missing E2/E3 evidence artifacts the framework's largest -live gap and asks who should build them. The research says: less building than -expected. - -- **pgTAP** is the established way to assert RLS behaviour in CI. "RLS policies - should be tested in CI with pgTAP, not just in dev, since policy regressions - are silent (no query error, just wrong data returned), and automated - cross-tenant isolation tests are the only reliable guard" - ([Blair Jordan — testing RLS policies with pgTAP](https://blair-devmode.medium.com/testing-row-level-security-rls-policies-in-postgresql-with-pgtap-a-supabase-example-b435c1852602), - [MakerKit — database testing with pgTAP](https://makerkit.dev/docs/next-supabase-turbo/development/database-tests)). -- **pgrls** is a purpose-built static analyzer: 67 lint rules for tenant and - per-user row-scoping bugs, performance traps and hygiene, 20 auto-fixable, a - semantic policy-diff command for CI gating, and a pytest plugin for isolation - tests ([pgrls on GitHub](https://github.com/pgrls/pgrls)). - -The named common failures match ours exactly: "a forgotten role, a pool reusing -connections with the wrong tenant context, or an admin path bypassing RLS -entirely" ([Kawshik](https://kawshik.dev/blog/multi-tenant-rag-pgvector-postgres-rls.html)). - -**Against our model:** the E3 evidence artifact in §13 ("RLS policies present on -every tenant-owned table; probe that a session without the GUC reads nothing; -probe that a wrong GUC reads nothing") is well specified and is roughly what -pgrls lints for plus what pgTAP asserts. `rapp-postgres` already runs -`scripts/isolation-test.sh` with 15 probes against a throwaway PostgreSQL 16 in -Docker — the same harness shape, pointed at the consumer boundary. Extending it -to the tenant boundary is incremental, not novel. - -**This materially changes the answer to open question §18.3.** The E3 half is -tooling-assisted and cheap. The E2 half remains semantic and human (see plane A -§7). Splitting the question that way makes it assignable. - -## 5. The layered consensus: RLS as defence in depth, not as the only layer - -Vendor and community guidance converges on the same layering our ADR describes, -which is reassuring for the ladder's shape even where the details need fixing. - -- AWS documents RLS as the mechanism for pooled multi-tenant isolation and - pairs it with a non-owner application role - ([AWS — multi-tenant data isolation with PostgreSQL RLS](https://aws.amazon.com/blogs/database/multi-tenant-data-isolation-with-postgresql-row-level-security/)). -- OWASP lists RLS under database-level enforcement and application-layer ORM - filters as *complementary*, recommending "database-level isolation as - defense-in-depth" rather than as a replacement - ([OWASP Multi-Tenant Security Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/Multi_Tenant_Security_Cheat_Sheet.html)). -- The application-layer half is a custom ORM/repository that automatically - appends tenant filters and injects `tenant_id` on write (ibid.) — which is - precisely our E2. -- "Use `SECURITY INVOKER` for typical multi-tenant logic so that the caller's - RLS context applies"; reserve `SECURITY DEFINER` for deliberate, controlled - cross-tenant access - ([oneuptime](https://oneuptime.com/blog/post/2026-01-25-row-level-security-postgresql/view)). - -**Against our model:** the E ladder is ordinal — each rung replacing the last — -but practice treats E2 and E3 as **layers that coexist**. A service at E3 that -dropped its E2 choke point would be worse off, not better, because E3 alone -fails open under SQL injection (§2). - -**Proposed amendment:** state that E3 *presupposes* E2 rather than superseding -it, and that a claim of E3 requires the E2 evidence artifact as well. That is a -small change with real consequences for how the ladder is read — it converts -"move up a rung" into "add a layer", which is what the sources actually -describe. - -## 6. Summary against our ladder - -| Finding | Verdict | -|---|---| -| Any session can re-`SET` the GUC; E3 fails against SQL injection and app compromise | **Correction.** E3's "cannot trivially route around" overclaims. Add a threat-model table. | -| E3 is decisive against forgotten predicates — the common case | **Confirms the reversal.** Draft-2's direction is right, its wording is not. | -| Table owner bypasses RLS silently without `FORCE ROW LEVEL SECURITY` | **Specific risk.** Our `_migrate` role owns the tables it creates (ADR-0001). | -| `BYPASSRLS` roles are exempt | Provisioning contract must forbid it on leased roles. | -| Statement-pooling + `SET` returns other tenants' rows, only under production concurrency | **Correction.** Use `SET LOCAL` in-transaction; ADR §16's pooling claim is backwards. | -| RLS disables functional indexes using non-leakproof functions | **New.** Require an EXPLAIN comparison in the E3 contract. | -| pgTAP + pgrls (67 lint rules, policy-diff CI gate, pytest plugin) exist | **Adopt.** §18.3's E3 half is far cheaper than assumed. | -| Policy regressions are silent — wrong data, no error | **Confirms §12.** Guarding must target invisible failure. | -| Practice treats E2 and E3 as coexisting layers, not successive rungs | **Amend.** E3 should presuppose E2, not replace it. | - -## Sources - -- [kmoppel — Yes, Postgres can do session vars, but should you use them?](https://kmoppel.github.io/2025-06-03-yes-postgres-can-do-session-variables-but-should-you-use-them/) -- [Daniel Imfeld — PostgreSQL row level security (notes)](https://imfeld.dev/notes/postgresql_row_level_security) -- [pganalyze — RLS in Postgres, security invoker views and why LEAKPROOF functions matter](https://pganalyze.com/blog/5mins-postgres-row-level-security-bypassrls-security-invoker-views-leakproof-functions) -- [AWS Database Blog — Multi-tenant data isolation with PostgreSQL Row Level Security](https://aws.amazon.com/blogs/database/multi-tenant-data-isolation-with-postgresql-row-level-security/) -- [oneuptime — How to secure multi-tenant data with RLS in PostgreSQL](https://oneuptime.com/blog/post/2026-01-25-row-level-security-postgresql/view) -- [sigmoid.at — Notes on PostgreSQL row level security](https://sigmoid.at/post/2021/09/23/postgresql_row_level_security/) -- [ShiftAsia — Why your database needs boundaries: an intro to PostgreSQL's RLS](https://shiftasia.com/community/why-your-database-needs-boundaries-an-intro-to-postgresqls-row-level-security-rls/) -- [pgEdge — It depends: using session variables in Postgres](https://www.pgedge.com/blog/it-depends-using-session-variables-in-postgres) -- [Kawshik — The multi-tenant RAG nightmare: securing pgvector with PostgreSQL RLS](https://kawshik.dev/blog/multi-tenant-rag-pgvector-postgres-rls.html) -- [pgrls — static analyzer for Postgres Row-Level Security](https://github.com/pgrls/pgrls) -- [Blair Jordan — Testing RLS policies in PostgreSQL with pgTAP](https://blair-devmode.medium.com/testing-row-level-security-rls-policies-in-postgresql-with-pgtap-a-supabase-example-b435c1852602) -- [MakerKit — Database testing with pgTAP](https://makerkit.dev/docs/next-supabase-turbo/development/database-tests) -- [Rico Fritzsche — Mastering PostgreSQL RLS for rock-solid multi-tenancy](https://ricofritzsche.me/mastering-postgresql-row-level-security-rls-for-rock-solid-multi-tenancy/) -- [pganalyze — Using Postgres row-level security in Ruby on Rails](https://pganalyze.com/blog/postgres-row-level-security-ruby-rails) -- [OWASP — Multi-Tenant Security Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/Multi_Tenant_Security_Cheat_Sheet.html) diff --git a/research/2026-08-17-adr008-plane-i-identity.md b/research/2026-08-17-adr008-plane-i-identity.md deleted file mode 100644 index 6d16a92..0000000 --- a/research/2026-08-17-adr008-plane-i-identity.md +++ /dev/null @@ -1,234 +0,0 @@ -# Plane I — Identity: prior knowledge and best practices - -> Research artefact — 2026-08-17. Supports `canon/architecture/adr-008-multi-tenancy-model.md` -> (draft-2), plane I of four. Surveys external practice for how multi-tenant -> systems name, carry, and validate tenant identity, and holds our `I0–I3` -> ladder against it. -> Companions: plane A (authorization), plane E (enforcement), plane P (placement). -> Index: `2026-08-17-adr008-multi-tenancy-research-index.md`. - -## 1. What the plane covers - -How a tenant is named, where that name is bound to a principal, how it travels -with a request, and how fresh the binding must be. Our ladder: - -| Level | State | -|---|---| -| I0 | No tenant concept | -| I1 | Local tenant notion, non-canonical | -| I2 | Canonical `tenant::`, validated; `tenant-engine` owns existence | -| I3 | I2 + capability roles honoured, live re-query for `aal2`-class decisions | - -## 2. Consensus practice: bind at registration, carry in the token - -The dominant pattern across vendor guidance is that tenant identity is bound -to the principal **at the identity provider**, not derived per-application. - -- AWS calls the unpacked token claims the **tenant context**, and frames it as - environment-like: it "influences how the SaaS application processes the - request" rather than being one parameter among many - ([SaaS Lens — preventing cross-tenant access](https://docs.aws.amazon.com/wellarchitected/latest/saas-lens/preventing-cross-tenant-access.html)). -- Binding is explicitly a **registration-time** responsibility: storing the - tenant identifier as a claim or attribute on the user in the IdP "shifts the - responsibility of mapping identities to tenants from each application to the - user registration process" - ([AWS Prescriptive Guidance — tenant onboarding and registration](https://docs.aws.amazon.com/prescriptive-guidance/latest/saas-multitenant-api-access-authorization/avp-design-onboarding-registration.html)). -- The recommendation is to define tenant context **at the IAM layer** rather - than relying on the application layer - ([AWS — 5 multi-tenant SaaS architecture best practices](https://aws.amazon.com/isv/resources/5-multi-tenant-saas-architecture/)). - -**Against our model:** this is what `I2` describes and what the IAM Profile -already mandates (`tenant` claim required on every token accepted by profile -consumers). We are aligned with consensus here, and `key-cape`/Keycloak -stamping the claim at issuance is the sanctioned shape. - -**Gap:** the ladder's I2 says "canonical identifiers used, validated". It does -not say *where the binding happens*. A service could satisfy I2 by parsing a -tenant id out of a request body. Consider tightening I2 to require the binding -be IdP-sourced, with request-supplied tenant ids explicitly at I1. - -## 3. Propagation is a separate discipline from naming - -Practice treats "the tenant id must reach every layer" as its own requirement, -not a consequence of having one: - -- A stable identifier should follow every request and stay consistent across - databases, services, logs, billing records, and background processes; without - a reliable tenant registry, other isolation controls are hard to enforce - consistently ([WorkOS — developer's guide to SaaS multi-tenant architecture](https://workos.com/blog/developers-guide-saas-multi-tenant-architecture)). -- Requests should execute in a tenant context established by middleware or - context variables, tagging each request with its owner - ([SSOJet — multi-tenant identity management](https://ssojet.com/blog/multi-tenant-identity-management)). -- Two leak surfaces get called out repeatedly and are **not** database - concerns: search indices (Elasticsearch queries that omit the tenant filter) - and cache keys that do not include the tenant id - ([WorkOS](https://workos.com/blog/developers-guide-saas-multi-tenant-architecture)). - -**Against our model:** ADR-008 has nothing to say about background jobs, -caches, or search indices. Our E ladder is described almost entirely in -database terms (tenant-keyed tables, RLS, per-tenant credentials). A service -could sit honestly at `E3` and still leak through a Redis key or an unfiltered -index. - -**Proposed amendment:** either widen the E-ladder evidence artifacts to cover -non-database stores, or add an explicit scope note that the E ladder describes -the *primary datastore* only and other stores are assessed separately. The -second is cheaper and more honest. - -## 4. Identity-plane placement is its own decision — and it has a known ceiling - -Identity providers face the same silo/pool question the P plane describes, and -the industry has converged on an answer with a number attached. - -- Keycloak offers **realm-per-tenant** (strong isolation, heavyweight) versus - **Organizations** (multiple tenants inside one realm, stable since Keycloak - 26, designed for the SaaS case) - ([Phase Two — multi-tenancy options in Keycloak](https://phasetwo.io/blog/multi-tenancy-options-keycloak/), - [Skycloak — multitenancy using Organizations](https://skycloak.io/blog/multitenancy-in-keycloak-using-the-organizations-feature/)). -- The operational verdict is blunt: "realm-per-tenant works for 5–20 tenants; - beyond that, you are managing an IAM platform, not building your product" - ([Izylife — Keycloak multi-tenancy: realm per tenant or Organizations?](https://izylifesolutions.com/articles/keycloak-multi-tenancy-b2b-iam/)). -- Auth0's equivalent is Organizations within a single Auth0 tenant, with - per-organization connections - ([KeycloakPro — Keycloak multi-tenancy with Organizations](https://keycloakpro.com/blog/keycloak-multi-tenancy-organizations-guide)). - -**Against our model:** ADR-008 treats placement as a property of *data -substrate* only. Identity has a placement axis too, it is live in our estate -(`key-cape`/Keycloak), and nobody has decided realm-per-tenant versus -Organizations. Given the tenant taxonomy includes `consumer` (private -individuals) and `family`, realm-per-tenant is as economically impossible here -as cluster-per-tenant — the 5–20 ceiling is well below our target tenant count. - -**Proposed amendment:** note in §3 of the ADR that the P plane, as written, -scopes to data substrate; identity-provider placement is a parallel decision -with the same silo/pool shape and should be recorded, probably by whoever owns -`key-cape`. - -## 5. Freshness: the tradeoff we already made, confirmed - -Our I3 requires live `tenant-engine` re-query for privileged, destructive, -credential-vending or `aal2`-class decisions, with the cached `tenant_roles` -claim acceptable for ordinary ones. External practice supports the split and -names the cost precisely. - -- The tradeoff is explicit: "tighter revocation controls often increase latency - and operational overhead, requiring organisations to balance immediate cutoff - against service reliability" - ([NHI Mgmt Group — JWT vs OAuth revocation](https://nhimg.org/faq/why-do-jwt-revocation-and-oauth-revocation-need-different-handling/)). -- For high-stakes decisions the sanctioned answer is opaque tokens plus - introspection (RFC 7662) — "a real kill switch at the cost of a lookup on the - hot path" ([oneuptime — how to handle JWT revocation](https://oneuptime.com/blog/post/2026-02-02-jwt-revocation/view)). -- Short lifetimes are the mainstream mitigation: "a 5-minute access token caps - how stale any resource server's view can be, which is the quiet reason the - default is 5 minutes and not an hour" (ibid.). Our IAM Profile's 5–30 minute - service/agent token lifetime sits inside this norm. -- Known failure mode directly relevant to us: controls "break down when - services cache JWT validation results" (ibid.). Our hybrid explicitly permits - caching for ordinary decisions, so the boundary between ordinary and - high-stakes is load-bearing and must be enforced, not documented. -- **Token versioning** is an alternative we do not use: a `token_version` per - subject, bumped on change, rejecting stale tokens service-side - ([TechNet Experts — JWT token versioning](https://www.technetexperts.com/jwt-token-versioning-stale-data/)). - It gives cheap global invalidation without a per-request lookup, at the cost - of a version read. Worth considering as a middle rung between our cached - claim and full live re-query. - -**Against our model:** we match best practice and, unusually, we wrote the -boundary down. The gap is evidential rather than architectural — our I3 -evidence artifact ("live re-query demonstrated on an `aal2`-class path; cached -path shown not to be used there") is exactly right and does not exist yet for -any service. - -## 6. Finding: our tenant identifier encodes a mutable attribute - -This is the significant new finding, and it is a critique of ratified canon -rather than of ADR-008. - -Our identifier is `tenant::`, where `` is an -onboarding-risk / entity-shape classification from ADR-0013: `trial`, -`friendly`, `single`, `small` (≤10 employees), `medium` (≤100), `large` -(≤1000), `enterprise` (1001+), `consumer`, `family`, `community`, -`association`, `agentic`. - -Several of those are **explicitly headcount bands**, and headcount changes. A -tenant onboarded as `tenant:small:acme` that grows past ten employees is -either misclassified forever or must be renamed — and renaming a tenant -identifier is a estate-wide data migration, since the identifier is the key -shared with `user-engine`, appears in OpenBao paths (`tenants//...`), -and is carried in every token. - -External practice is unambiguous: - -- "Avoid encoding mutable data or business logic in the identifier" — the - **ID Entanglement Effect**, where composite ids encode business relationships - or mutable state that is later parsed - ([Cogna — the unreasonable power of identifiers](https://cogna.co/blog/the-unreasonable-power-of-identifiers/)). -- Opaque identifiers "decouple identification from descriptive attributes, - thereby enhancing stability against entity changes" - ([Grokipedia — unique identifier](https://grokipedia.com/page/Unique_identifier)). -- The FAIR/persistent-identifier literature reaches the same conclusion: make - identifiers opaque from the start, so that when a name changes "just the - `rdfs:label` and some other properties may change — but not the URI" - ([Cool URIs for FAIR Knowledge Graphs](https://arxiv.org/pdf/2407.09237)). -- A structured id is acceptable *only if treated as opaque* — "the key is that - we treat it as an unchanging reference and don't try to parse meaning from - its structure" ([Cogna](https://cogna.co/blog/the-unreasonable-power-of-identifiers/)). - -The IAM Profile half-anticipates this. It states that capability roles were -deliberately separated from the identifier so "a tenant's roles may change -without renaming its identifier", and that grouping "does not gate which -capability roles a tenant may hold". The same reasoning applies to grouping -itself and was not carried through. - -**Two honest readings**, and this needs an owner decision rather than an -assertion from this document: - -1. **Grouping is onboarding-time and frozen by design.** It records the shape - of the entity *when it was onboarded*, not its current headcount, and is - never expected to change. If so, this should be stated explicitly in - ADR-0013 — because every reader will otherwise assume `small` means - currently-small, and some system will eventually make a decision on it. -2. **Grouping is current classification.** Then the identifier encodes a - mutable attribute, the estate has a rename problem it has not hit yet - because tenant counts are tiny, and the fix (opaque id + grouping as a - mutable property on the tenant record) gets more expensive every month. - -Either way, `rapp-postgres` is already correct to treat these as opaque -strings — ADR-0001 §4 prohibits the storage platform from parsing them. The -risk sits with any consumer that *does* parse the grouping segment. - -**Proposed action:** raise as a question to `tenant-engine` and NetKingdom -canon rather than folding into ADR-008. It is out of ADR-008's scope but -materially affects plane I's top rung. - -## 7. Summary against our ladder - -| Finding | Verdict | -|---|---| -| Bind tenant at IdP, carry in token, treat as context not parameter | **Aligned.** Matches I2 + IAM Profile. | -| I2 does not require the binding be IdP-sourced | **Gap.** Tighten wording; request-supplied ids belong at I1. | -| Propagation must cover caches, search indices, background jobs | **Gap.** ADR-008's ladders are datastore-shaped. Add a scope note. | -| Identity-provider placement is its own silo/pool decision, ceiling ~5–20 tenants for realm-per-tenant | **Missing.** P plane scopes to data only; identity placement is undecided and unowned. | -| Hybrid cached-claim / live-re-query for high-stakes | **Aligned and unusually well specified.** Evidence artifact does not exist yet. | -| Token versioning as a middle option | **Unconsidered.** Cheap global invalidation without hot-path lookup. | -| Identifier encodes mutable headcount bands | **Finding.** Contradicts identifier-design consensus; needs an ADR-0013 clarification. | - -## Sources - -- [AWS SaaS Lens — Preventing cross-tenant access](https://docs.aws.amazon.com/wellarchitected/latest/saas-lens/preventing-cross-tenant-access.html) -- [AWS Well-Architected SaaS Lens (PDF)](https://docs.aws.amazon.com/pdfs/wellarchitected/latest/saas-lens/wellarchitected-saas-lens.pdf) -- [AWS Prescriptive Guidance — Tenant onboarding and user tenant registration](https://docs.aws.amazon.com/prescriptive-guidance/latest/saas-multitenant-api-access-authorization/avp-design-onboarding-registration.html) -- [AWS — 5 multi-tenant SaaS architecture best practices](https://aws.amazon.com/isv/resources/5-multi-tenant-saas-architecture/) -- [AWS Security Blog — SaaS tenant isolation with ABAC using STS support for tags in JWT](https://aws.amazon.com/blogs/security/saas-tenant-isolation-with-abac-using-aws-sts-support-for-tags-in-jwt/) -- [WorkOS — The developer's guide to SaaS multi-tenant architecture](https://workos.com/blog/developers-guide-saas-multi-tenant-architecture) -- [SSOJet — Multi-tenant identity management for SaaS](https://ssojet.com/blog/multi-tenant-identity-management) -- [Phase Two — Understanding multi-tenancy options in Keycloak](https://phasetwo.io/blog/multi-tenancy-options-keycloak/) -- [Skycloak — Multitenancy in Keycloak using the Organizations feature](https://skycloak.io/blog/multitenancy-in-keycloak-using-the-organizations-feature/) -- [Izylife — Keycloak multi-tenancy: realm per tenant or Organizations?](https://izylifesolutions.com/articles/keycloak-multi-tenancy-b2b-iam/) -- [KeycloakPro — Keycloak multi-tenancy with Organizations](https://keycloakpro.com/blog/keycloak-multi-tenancy-organizations-guide) -- [oneuptime — How to handle JWT revocation](https://oneuptime.com/blog/post/2026-02-02-jwt-revocation/view) -- [NHI Management Group — Why JWT revocation and OAuth revocation need different handling](https://nhimg.org/faq/why-do-jwt-revocation-and-oauth-revocation-need-different-handling/) -- [TechNet Experts — How to fix stale JWTs via token versioning](https://www.technetexperts.com/jwt-token-versioning-stale-data/) -- [Cogna — The unreasonable power of identifiers](https://cogna.co/blog/the-unreasonable-power-of-identifiers/) -- [Grokipedia — Unique identifier](https://grokipedia.com/page/Unique_identifier) -- [Cool URIs for FAIR Knowledge Graphs (arXiv 2407.09237)](https://arxiv.org/pdf/2407.09237) diff --git a/research/2026-08-17-adr008-plane-p-placement.md b/research/2026-08-17-adr008-plane-p-placement.md deleted file mode 100644 index 7c6b064..0000000 --- a/research/2026-08-17-adr008-plane-p-placement.md +++ /dev/null @@ -1,244 +0,0 @@ -# Plane P — Placement: prior knowledge and best practices - -> Research artefact — 2026-08-17. Supports `canon/architecture/adr-008-multi-tenancy-model.md` -> (draft-2), plane P of four. Surveys external practice for which substrate -> holds a tenant's data, and holds our `P0–P4` ladder against it. -> **Contains strong external validation of draft-2's reframe, and one -> expressiveness failure in the P ladder itself.** -> Companions: plane I (identity), plane A (authorization), plane E (enforcement). - -## 1. What the plane covers - -Which physical substrate holds a tenant's data, who decides, and what moving -costs. Our ladder: - -| Level | Shape | -|---|---| -| P0 | Shares a database with another consumer | -| P1 | Database per consumer, shared cluster | -| P2 | Dedicated cluster per consumer | -| P3 | Dedicated cluster per tenant | -| P4 | P3 plus separate region or jurisdiction | - -## 2. Validation: "isolation is a spectrum, not a discrete property" - -The operator's instruction to reframe draft-1 from a fixed model into a -graduated framework is not merely defensible — it is Microsoft's stated -position, in almost the same words: - -> "Instead of viewing isolation as a discrete property, consider it a spectrum. -> You can deploy components of your architecture that are more isolated or less -> isolated than other components in the same architecture, depending on your -> requirements." -> — [Azure Architecture Center — Tenancy models for a multitenant solution](https://learn.microsoft.com/en-us/azure/architecture/guide/multitenant/considerations/tenancy-models) - -The same page decomposes isolation across tiers exactly as our planes do — -"you can use different levels of isolation for each tier", with a shared UI -tier, a shared middle tier, and isolated databases given as the worked example. - -**Against our model:** draft-2's core move is externally corroborated by the -most rigorous public taxonomy available. Worth citing in ADR §15, because it -converts "we decided to do it this way" into "this matches the reference -guidance", which matters for reviewers who will ask whether we invented a -private vocabulary. - -## 3. Validation: our E↔P coupling claim is independently confirmed - -ADR §3.2 asserts that the planes couple at the top — `E4` is only reachable at -`P3` or above — and §4.4 asserts that `P0 → P1 → P2` does not raise the E -level. Microsoft states the mechanism plainly: - -> "A key difference between tenants and deployments is how isolation is -> enforced. When multiple tenants share a single deployment (a set of -> infrastructure), you typically rely on your application code and a tenant -> identifier that's in a database to keep each tenant's data separate. When -> tenants have their own dedicated deployments, they have their own -> infrastructure, so it might be less important for your code to account for a -> multitenant environment." (ibid.) - -That is our E/P relationship derived independently: shared deployment ⇒ -enforcement is application code (our E1–E3); dedicated deployment ⇒ enforcement -is structural (our E4). - -## 4. Finding: our P ladder cannot express per-tenant placement, and our own commercial section requires it - -This is the significant gap. - -Azure names two partitioning axes that our single ladder collapses: - -- **Vertically partitioned deployments** — "a combination of single-tenant and - multitenant deployments. For example, you might have most of your customers' - data and application tiers on multitenant infrastructures, but you deploy - single-tenant infrastructures for customers who require higher performance or - data isolation." Explicitly monetisable: "You can even charge customers a - higher rate to use a single-tenant deployment." (ibid.) -- **Horizontally partitioned deployments** — shared application tier, dedicated - databases per tenant, targeted at the component that carries the load: "if - you identify that specific components cause most of the load on your system, - then you can deploy separate components for each tenant." (ibid.) - -Our P ladder assigns **one level per service**. `tenant-engine` is `P1`. -`audit-core` is `P1`. There is no way to say *"P1 for most tenants, P3 for the -three enterprise tenants who bought isolation"* — which is precisely the -vertically partitioned model, and precisely what ADR §11's minimum-level plan -tiers imply. - -The contradiction is internal to draft-2: §11.1 says "a tier may require -`E3 P2`", but if only some tenants hold that tier, the delivering service -occupies two P levels at once. The posture vector as specified cannot represent -that state, so a service in it would have to either overclaim (declare P2 for -everyone) or underclaim (declare P1 and misrepresent what the tier delivers). -Both are §6 violations. - -**Proposed amendment.** Either: - -1. **Make P a per-tenant-class level rather than a per-service one** — the - vector carries a default plus exceptions: - ```yaml - tenancy: - current: { I: 2, A: 3, E: 2, P: 1 } - placement_exceptions: - - tenants: ["tenant:enterprise:*"] - P: 3 - reason: "isolation tier" - ``` -2. Or **declare per-tenant placement out of scope** for the framework and say - so, accepting that plan tiers selling isolation are then unmodellable. - -Option 1 is more work and is the honest one, because the estate has already -anticipated tiered isolation. Azure also flags the cost: "your codebase needs -to be designed to support both multitenant and single-tenant deployments", and -migration between them must be planned for (ibid.) — which is exactly what our -§7 portability rules already provide, so we are better positioned for this than -most. - -## 5. Cell-based architecture: the pattern our ladder skips - -ADR §18.8 asks whether to evaluate cells. The literature gives enough to -sharpen the question. - -- A cell is "a complete, independent instance of the service" with **a fixed - maximum size** — compute, storage, supporting services and configuration for - a defined blast radius - ([Rackspace — cell-based architecture on AWS](https://www.rackspace.com/blog/cell-based-architecture-aws), - [AWS guidance for cell-based architecture](https://github.com/aws-solutions-library-samples/guidance-for-cell-based-architecture-on-aws)). -- Cells are logical partitions on a **partition key** — customer ID ranges or - tenant ID — and "the data required for or generated by each cell is also - partitioned, so that there is no replication of data between cells" (ibid.). -- The sizing argument is the point: "by capping the size of each cell… you can - potentially predict and address failures more easily. This leads to a higher - MTBF" (ibid.). And "it is important to identify the proper partition key to - minimize cross-cell interactions" (ibid.). -- Cells are the bulkhead pattern applied to tenancy — AWS files it under - fault isolation, REL10-BP04 - ([AWS Well-Architected — use bulkhead architectures to limit scope of impact](https://docs.aws.amazon.com/en_us/wellarchitected/2022-03-31/framework/rel_fault_isolation_use_bulkhead.html)). -- Azure's equivalent is the **Deployment Stamps pattern**: "each stamp serves a - predefined number of tenants, allowing you to scale almost linearly and serve - increasing numbers of tenants" - ([Azure — Deployment Stamps pattern](https://learn.microsoft.com/en-us/azure/architecture/patterns/deployment-stamp)). - Azure also calls deployments "supertenants or stamps" and describes the - tenant→deployment mapping table as a first-class component. - -**Against our model:** a cell is not a rung on our P ladder — it is a different -axis. P measures *how much is shared*; a cell measures *how large a shared unit -is allowed to get before you make another one*. Our `platform-pg` is, in cell -terms, an uncapped cell: §16 of the ADR computes a ceiling of roughly six -consumers but nothing enforces it, and there is no second cell to overflow -into. - -That reframes §18.8 usefully. The question is not "should we adopt cells" but -"**what is `platform-pg`'s declared maximum size, and what happens at the -boundary?**" — a question we can answer now, cheaply, and which the connection -ceiling already forces. - -Azure also names **bin packing** as the assignment strategy and stresses -maintaining "a record of your customers and the infrastructure where their data -and applications reside so that you can route their traffic to the correct -location" — a tenant→deployment mapping table. We have no such record; the -placement facts in ADR §1 were assembled by inspection. - -## 6. Placement is a commercial decision, and the guidance says so first - -Our §11 treats commercial expression as a constraint to be managed. Azure puts -it the other way round: - -> "Selecting a tenancy model isn't only a technical decision. It's also a -> commercial decision." — considering business objectives, compliance, scale, -> automation capacity, and SLAs (ibid.). - -And on the economics, matching our §14 rejection of R3/P3-everywhere: "if a -single tenant requires a specific infrastructure cost, 100 tenants probably -require 100 times that cost", with ongoing maintenance time-consuming at scale -(ibid.). Conversely single-tenant is appropriate "if you expect that your -business will have only a few customers… even though it's more costly". - -**Against our model:** ADR §8.2 proposes placement ownership to -`railiance-platform`, an operations-shaped repo. This guidance suggests -placement policy is *jointly* commercial and technical — which strengthens the -case that `adaptive-pricing` must be a co-signer on §18.2, not only on §18.5. - -## 7. Our tenant taxonomy is a recognised hard case - -Azure's B2C guidance describes our exact situation: - -> "If your customers are consumers, it's often more complicated to relate -> customers, tenants, and users… consider whether your solution might be used -> by families, groups of friends, clubs, associations, or other groups that -> might need to access and manage their data together. For example, a music -> streaming service might support both individual users and families, and it -> might treat each of these account types differently when it separates them -> into tenants." (ibid.) - -Our ADR-0013 taxonomy contains `consumer`, `family`, `community`, and -`association` verbatim. It also notes that if tenants are individuals, "you -might need to consider how you handle personal data and about the data -sovereignty laws in each jurisdiction" — which is our `P4` and, combined with -plane A §6, the tenant-deletion gap. - -## 8. Testing placement is a named obligation - -> "Whichever isolation model you choose, be sure to test your solution to verify -> that one tenant's data isn't accidentally leaked to another **and that any -> noisy neighbor outcomes are acceptable**." (ibid.) - -Two evidence artifacts, not one. ADR §13 covers the leak half (P1–P4: -"provisioning declaration plus the platform's own isolation probes") and says -nothing about noisy-neighbour behaviour. Azure suggests deliberate fault -injection (Chaos Studio) to verify resilience. - -**Proposed amendment to §13:** add a noisy-neighbour evidence artifact for P1 -and P2 — a demonstration that one consumer saturating its connection or CPU -allowance does not breach another's. `rapp-postgres` already sets -database-scoped `connection_limit`, `statement_timeout` and -`idle_in_transaction_session_timeout` precisely for this, and ADR-0001 §3 argues -resource exhaustion is an isolation failure — but nothing proves the limits -hold under load. - -## 9. Summary against our ladder - -| Finding | Verdict | -|---|---| -| "Isolation is a spectrum, not a discrete property"; different tiers at different levels | **Validates draft-2.** Cite in §15. | -| Shared deployment ⇒ app-code enforcement; dedicated ⇒ structural | **Confirms E↔P coupling** independently. | -| Vertical partitioning: some tenants pooled, some dedicated, explicitly monetised | **Gap.** P is per-service; cannot express per-tenant placement that §11 requires. | -| Horizontal partitioning: shared app tier, per-tenant DB for the hot component | **Unmodelled.** Our P assumes whole-service granularity. | -| A cell has a **fixed maximum size**; stamps scale near-linearly | **Reframes §18.8.** Ask "what is `platform-pg`'s declared max size?", not "should we adopt cells". | -| Tenant→deployment mapping table is a first-class component | **Missing.** Our placement facts were assembled by inspection. | -| Tenancy model is a commercial decision as much as technical | **Amend §8.2.** `adaptive-pricing` should co-sign placement ownership. | -| B2C/family/association tenancy is a recognised hard case | Our taxonomy hits it verbatim; jurisdiction concerns feed P4. | -| Test for leakage **and** for acceptable noisy-neighbour outcomes | **Amend §13.** No noisy-neighbour evidence artifact exists. | - -## Sources - -- [Azure Architecture Center — Tenancy models for a multitenant solution](https://learn.microsoft.com/en-us/azure/architecture/guide/multitenant/considerations/tenancy-models) -- [Azure Architecture Center — Architectural approaches for a multitenant solution](https://learn.microsoft.com/en-us/azure/architecture/guide/multitenant/approaches/overview) -- [Azure Architecture Center — Deployment Stamps pattern](https://learn.microsoft.com/en-us/azure/architecture/patterns/deployment-stamp) -- [Azure Architecture Center — Considerations for updating a multitenant solution](https://learn.microsoft.com/en-us/azure/architecture/guide/multitenant/considerations/updates) -- [AWS Well-Architected — REL10-BP04 Use bulkhead architectures to limit scope of impact](https://docs.aws.amazon.com/en_us/wellarchitected/2022-03-31/framework/rel_fault_isolation_use_bulkhead.html) -- [AWS Well-Architected — How do you use fault isolation to protect your workload?](https://wa.aws.amazon.com/wellarchitected/2020-07-02T19-33-23/wat.question.REL_10.en.html) -- [AWS Solutions Library — Guidance for cell-based architecture on AWS](https://github.com/aws-solutions-library-samples/guidance-for-cell-based-architecture-on-aws) -- [Rackspace — Cell-based architecture on AWS](https://www.rackspace.com/blog/cell-based-architecture-aws) -- [Ryan Harrison — Cell-based architecture engineering guidelines](https://guidelines.ryanharrison.co.uk/architecture/cell-architecture/) -- [Anudeep Balla — The guide to cell-based architecture](https://medium.com/@anudeepballa7/the-guide-to-cell-based-architecture-58cf607f9754) -- [AWS Well-Architected SaaS Lens (PDF)](https://docs.aws.amazon.com/pdfs/wellarchitected/latest/saas-lens/wellarchitected-saas-lens.pdf) -- [hidekazu-konishi — AWS SaaS multi-tenant architecture guide: tenant isolation, pool and silo models](https://hidekazu-konishi.com/entry/aws_saas_multi_tenant_architecture_guide.html)