From c4f22a5e55d9220ec417918643048ae6932623be Mon Sep 17 00:00:00 2001 From: codex Date: Mon, 17 Aug 2026 14:49:00 +0200 Subject: [PATCH] ADR-008 draft-4: apply the eight amendments the gap research forced R4 now has a mechanism. It splits into horizon-elapsed (wait out a window somebody else's retention set - available to all, proves little) and key-destroyed (encrypt per entity, destroy the key, immediate). A service must name which route it uses. The regulatory position is stated rather than implied: DPAs have accepted key destruction as erasure where physical deletion is disproportionate, under conditions, and the EDPB has not formally endorsed it. Section 11.4 now says a service may make that claim but not in language implying a regulator blessed it. Overclaiming here would be worse than anywhere else in the document. Third coupling recorded: shredding a single tenant's data needs the application to encrypt under a per-tenant key before writing, so the top of the retention ladder is an enforcement-plane capability. Reaching R4 is not a retention project. The noisy-neighbour artifact was replaced. It had required proof that a saturating consumer does not breach another's allowance, which shared infrastructure cannot provide - an artifact that can only fail, or passes by being run gently, is an overclaim in the costume of evidence. It now measures: baseline, saturation run, proof the controls bind, recorded degradation. Generalised as decision 13.4. The connection ceiling is reframed. Seven consumers each politely inside a 14-connection allowance still exhaust the instance, so the number is the aggregate noisy-neighbour bound, and our per-consumer governance guards the other shape entirely. Also: quota transparency as a disclosure obligation (10.2) - a consumer learning its statement timeout by hitting it in production is our failure, not theirs; and two new open questions - crypto-shredding an audit trail destroys what audit-core exists to hold, and we have no QoS vocabulary despite a latency-critical consumer sharing an instance with a batch one. Co-Authored-By: Claude Opus 5 --- .../adr-008-multi-tenancy-model.md | 95 ++++++++++++++++--- 1 file changed, 82 insertions(+), 13 deletions(-) diff --git a/canon/architecture/adr-008-multi-tenancy-model.md b/canon/architecture/adr-008-multi-tenancy-model.md index 15bdc3f..6e4273a 100644 --- a/canon/architecture/adr-008-multi-tenancy-model.md +++ b/canon/architecture/adr-008-multi-tenancy-model.md @@ -5,7 +5,7 @@ title: "Tenancy Posture: Five Planes, Graduated Levels, Declared Conformance" status: proposed decided_by: Bernd Worsch date: "2026-08-17" -revision: "draft-3" +revision: "draft-4" tags: ["architecture", "multi-tenancy", "isolation", "placement", "retention", "maturity", "tenant-engine", "flex-auth", "rapp-postgres", "scaling"] --- @@ -13,16 +13,22 @@ tags: ["architecture", "multi-tenancy", "isolation", "placement", "retention", " ## Status -**Proposed, draft-3.** +**Proposed, draft-4.** - **draft-1** proposed a single model with fixed characteristics. Rejected: it could not describe a repo that is not there yet. - **draft-2** reframed to graduated levels per plane. Externally corroborated (§16), but four of its statements were wrong and one thing it needed was missing. -- **draft-3** applies those corrections, adds the retention plane, and records - an adoption stance. It is informed by four external research digests, one per - original plane, in `research/2026-08-17-adr008-*`. +- **draft-3** applied those corrections, added the retention plane, and + recorded an adoption stance. +- **draft-4** closes the two gaps draft-3 left open: `R4` had no mechanism + beyond waiting, and the noisy-neighbour evidence artifact asserted something + shared infrastructure cannot provide. Both were found by research, not by + review. + +Informed by five external research digests in `research/2026-08-17-adr008-*`, +which carry full citations for every external claim made here. Reviewed by nobody yet. §19 lists what each owner is being asked to accept. @@ -100,6 +106,11 @@ stated where they apply, not used to argue the planes are one: - `E4` is reachable only at `P3` or above. - `R`'s erasure horizon is bounded below by `P` — on shared substrate, a consumer's horizon is the instance maximum (§4.5). +- `R4` by key destruction is bounded by the **key boundary**, which is an + E-plane property. Shredding a single tenant's data requires the application + to encrypt under a per-tenant key before writing; the storage platform cannot + supply it. **Reaching the top of the retention ladder is not a retention + project.** **Decision 3.3 — scope.** The P and R ladders describe a service's **primary datastore**. Caches, search indices, message queues and background jobs are @@ -243,9 +254,25 @@ layer. Reference implementation: `rapp-postgres` ADR-0002. | **R1** | Platform default retention applies (N=30 days). The consumer has declared no requirement. | | **R2** | Retention declared as N days per dataset; the **erasure horizon** is published, and the consumer makes no promise shorter than it. | | **R3** | Policy-driven deletion: the consumer or its governance layer declares what is due, the platform sweeps whole datasets on that instruction and evidences each run. | -| **R4** | Verified erasure: deletion proven complete across live data, backups and derived copies within the published horizon. | +| **R4** | Verified erasure: data proven unrecoverable across live storage, backups and derived copies, by one of the two routes below. | -Three properties. +**R4 has two routes and a service MUST name which one it uses.** + +| Route | Mechanism | Cost | +|---|---|---| +| **Horizon-elapsed** | Wait out the published erasure horizon; the data ages out of every retained copy. | Available to everyone, proves little, and the wait is set by a co-resident's retention requirement rather than your own. | +| **Key-destroyed** | Encrypt per entity, then destroy the key. Retained copies survive but are unreadable. | Requires per-entity keys, strong encryption, and an auditable destruction record. Immediate. | + +**Regulatory standing of the key-destroyed route, stated carefully because +overclaiming here is worse than anywhere else in this framework.** Data +protection authorities have accepted key destruction as erasure where physical +deletion would be manifestly disproportionate, and the practice is recognised +under conditions — strong encryption, irreversible destruction, and an auditable +record of it. **The EDPB has not formally endorsed it as Article 17 erasure.** A +service reaching R4 by key destruction is making a defensible claim, not a +settled one, and must say so rather than reporting a clean "deleted". + +Three further properties. **The erasure horizon is the interval between deleting data and it ceasing to be recoverable from anything the platform holds.** Deleting a row does not @@ -380,6 +407,15 @@ levels it operates. `rapp-postgres` ADR-0001 §5 is the reference. Where the model cannot provide a guarantee, the platform says so and names the escalation. +**Decision 10.2 — quotas are disclosed, not discovered.** The same obligation +extends from what a leaked credential can reach to what the platform will +refuse to do for you. Every consumer MUST be told, at provisioning, the +throttles and quotas enforced against it — connection limits, statement +timeouts, idle-transaction timeouts — and told again when they change. A +consumer learning its statement timeout by hitting it in production is a +disclosure failure, not a consumer bug. This is how `tenant-engine` was +provisioned, by good practice rather than by rule; the rule now exists. + ## 11. Commercial expression - **11.1** Plan tiers are expressed *internally* as minimum levels. A tier may @@ -396,7 +432,9 @@ escalation. - A claim that another tenant **cannot** reach the customer's data requires **E4**. - A claim that deleted data **is gone** requires **R4**, or an erasure - horizon disclosed alongside it. + horizon disclosed alongside it. Where R4 is reached by key destruction, the + claim is defensible but not settled law (§4.5) — it may be made, and it may + not be made in language that implies a regulator has blessed it. ## 12. Methodology — analyze, establish, improve, guard @@ -424,6 +462,14 @@ Guarding must be designed for invisible failure, not for crashes. **Decision 13.1:** a level is claimed only with its evidence artifact present. This turns §6's accuracy rule from an honour system into a check. +**Decision 13.4 — an artifact must assert something achievable.** Draft-3's +noisy-neighbour evidence required proof that a saturating consumer "does not +breach" another's allowance. Shared infrastructure cannot provide that; the +risk is inherent and cannot be wholly removed. An artifact that can only fail, +or that passes by being run gently enough, is an overclaim wearing the costume +of evidence. Where a property cannot be guaranteed, the artifact measures and +records it instead. + **Decision 13.2 — evidence is of two kinds, and conflating them is an overclaim.** *Mechanical* evidence is a structural assertion a machine can make and belongs in CI. *Adversarial* evidence is semantic, requires setting up @@ -444,7 +490,7 @@ E2 evidence.** | **E3** | `FORCE ROW LEVEL SECURITY` on every tenant table; no `BYPASSRLS` on leased roles; probe that a session without the GUC reads nothing; probe that a wrong GUC reads nothing; `EXPLAIN` comparison | Mechanical | | **E4** | Per-tenant credential demonstrated unable to connect to another tenant's substrate | Mechanical | | **P1–P4** | Provisioning declaration plus the platform's isolation probes | Mechanical | -| **P1–P2 (noisy neighbour)** | One consumer saturating its connection or CPU allowance demonstrably does not breach another's | **Adversarial**, load-generated | +| **P1–P2 (noisy neighbour)** | A recorded baseline of per-consumer resource usage; a run in which one consumer saturates its declared allowance; evidence that the governance controls **bind** (the greedy consumer is held at its limits) and that the degradation co-residents experience is **measured, recorded and judged acceptable**; the aggregate headroom at time of measurement | **Adversarial**, load-generated, with a review date | | **R2** | Declared retention rendered; erasure horizon published and reported in the operator surface | Mechanical | | **R3** | Sweep evidence records: timestamp, dataset, identifiers removed, authorising policy reference | Mechanical | | **R4** | Erasure demonstrated across live data, backups and derived copies within the horizon | **Adversarial** | @@ -561,10 +607,19 @@ memory limit: 1Gi per consumer: 14 connections (12 runtime + 2 migration) ``` -**Connection ceiling: roughly six consumers.** Seven request 98 of 100 before -CNPG's instance manager, metrics exporter and reserved slots. Two are -provisioned. We are at roughly a third of capacity and the third request will -not feel like a scaling event. +**Connection ceiling: roughly six consumers — and this is the aggregate +noisy-neighbour bound, not a capacity statistic.** Seven consumers request 98 +of 100 before CNPG's instance manager, metrics exporter and reserved slots. +Every one of them is politely inside its declared 14-connection allowance; the +instance still fails. + +That distinction matters because our governance addresses the wrong shape. +Per-consumer `connection_limit`, `statement_timeout` and +`idle_in_transaction_session_timeout` guard well against **one greedy +consumer**. They do nothing about **the aggregate of many modest ones**, which +is the second and less intuitive noisy-neighbour failure and the one this +number describes. Two consumers are provisioned. We are at roughly a third of +the bound, and the third request will not feel like a scaling event. **Memory likely binds first.** 100 backends against 1Gi is ~10MB per backend. Connection exhaustion errors clearly; memory pressure OOM-kills and degrades @@ -634,6 +689,20 @@ minimum-level vocabulary alongside isolation. 10. **Engine neutrality** — the P ladder rests on a PostgreSQL property. State it engine-specifically and say so, or abstract it and risk a non-Postgres implementation that silently differs? +11. **Erasure versus audit** — `audit-core`: crypto-shredding a tenant's audit + records destroys the evidence the service exists to hold, and ADR-0001 §2 + deliberately built the role model so history could not be rewritten. The + usual resolution separates the *fact* of an event, retained, from its + *personal payload*, encrypted per subject and shreddable. Raised because a + naive "R4 everywhere" target would instruct the audit service to destroy + its own evidence. The answer is `audit-core`'s, not this framework's. +12. **Quality of service** — *owner needed.* The framework has no vocabulary + for saying one consumer's latency matters more than another's. + `tenant-engine` sits on `flex-auth`'s synchronous authorization path and + chose a 5s statement timeout for that reason; it shares an instance with + `audit-core`, which is not latency-critical. Nothing prioritises between + them. Either add a QoS dimension or state that all co-residents are equal + and latency-critical consumers must escalate to P2. **Routed elsewhere, deliberately.** The tenant identifier `tenant::` embeds headcount bands (`small`, `medium`, `large`)