diff --git a/research/2026-08-17-adr008-gap-research-erasure-and-noisy-neighbour.md b/research/2026-08-17-adr008-gap-research-erasure-and-noisy-neighbour.md new file mode 100644 index 0000000..8a95cf0 --- /dev/null +++ b/research/2026-08-17-adr008-gap-research-erasure-and-noisy-neighbour.md @@ -0,0 +1,265 @@ +# Gap research — erasure mechanism (R4) and noisy-neighbour evidence + +> Research artefact — 2026-08-17. Fifth digest supporting +> `canon/architecture/adr-008-multi-tenancy-model.md` (draft-3). Closes the two +> gaps the four plane digests left open: how `R4` (verified erasure) can be +> reached at all, and what a noisy-neighbour evidence artifact should actually +> assert. +> **Both findings change a level definition in draft-3.** +> Index: `2026-08-17-adr008-multi-tenancy-research-index.md`. + +--- + +# Part 1 — Erasure: R4 needs a mechanism it does not currently have + +## 1.1 The problem draft-3 left standing + +ADR-008 §4.5 defines `R4` as "deletion proven complete across live data, +backups and derived copies within the published horizon", and `rapp-postgres` +ADR-0002 establishes that on shared substrate the horizon is the instance +maximum — every co-resident consumer's retention window, whichever is longest. + +Read together, those say: **erasure completes by waiting.** A tenant exercising +a deletion right is told the data is gone in up to N days, where N is set by +somebody else's retention requirement. That is honest, and it is a weak answer. +It also makes `R4` unreachable in any useful sense, because "we waited" is not +a proof. + +## 1.2 Crypto-shredding is the established mechanism + +The industry answer to erasure under immutable or long-retention storage is to +make the data unreadable rather than absent. + +> "Crypto-shredding for backups encrypts each subject (or tenant) with a unique +> key derived from the subject identifier, and on erasure, destroys the key. +> The backups remain but are cryptographically inaccessible." +> — [oneuptime — setting up crypto-shredding for GDPR right-to-erasure](https://oneuptime.com/blog/post/2026-02-17-how-to-set-up-crypto-shredding-for-gdpr-right-to-erasure-compliance-in-google-cloud/view) + +It is aimed precisely at our situation: the named use cases are "append-only or +immutable storage (event sourcing, WORM, immutable backups) where physical +deletion is impossible" +([Granit — crypto-shredding: GDPR erasure without deleting rows](https://granit-fx.dev/blog/crypto-shredding-gdpr-erasure-without-deleting-rows/)). + +**Regulatory standing, stated carefully because overclaiming here would be +worse than in any other section of this framework.** The EDPB has *not* formally +endorsed crypto-shredding as Article 17 erasure. Several data protection +authorities have accepted it where physical deletion would be manifestly +disproportionate effort under Article 17(1), and EDPB Guidelines 5/2019, the UK +ICO and the French CNIL are cited as recognising it **provided three conditions +hold**: strong encryption (AES-256), irreversible key destruction, and auditable +destruction +([pg_hardstorage — GDPR Art. 17 crypto-shred](https://github.com/cybertec-postgresql/pg_hardstorage/blob/main/docs/compliance/gdpr-art-17-crypto-shred.md), +[Opsio — right to erasure in cloud systems](https://opsiocloud.com/blogs/right-to-erasure-article-17-gdpr-deletion-cloud-systems/)). + +The third condition is the one this estate is well placed for and the one most +implementations skip: **destruction must be auditable**. ADR-0002 already +requires every sweep to emit an evidence record. Key destruction is the same +shape. + +## 1.3 The finding: shredding granularity is bounded by the key boundary + +Crypto-shredding "requires per-entity key isolation: each entity instance gets +its own encryption key" (oneuptime, ibid.). That single requirement determines +what is available to us, and it is not uniform across the estate. + +| What we want to shred | Key boundary needed | Available today? | +|---|---|---| +| A consumer's logical exports | Per-consumer key | **Close.** The offsite lane already encrypts with `age`, and OpenBao holds key material. This is a key-management change, not an architecture change. | +| A consumer's physical backup | Per-consumer key on the Barman object store | **No.** One WAL stream, one credential, one encryption context for the whole instance. Same constraint that forced the derived maximum in ADR-0002 §2. | +| A single tenant's rows inside a consumer | Per-tenant key, applied by the application before the data reaches storage | **No, and not the platform's to build.** Requires application-level per-tenant encryption. | + +The third row is the architecturally interesting one. Per-tenant crypto-shred +means the consumer encrypts tenant data under a tenant-specific key before +writing it — which is an **enforcement-plane** property, not a retention one. It +also happens to be a strong one: data encrypted per tenant is unreadable across +the boundary even by a credential that can address the rows, which is E4-class +protection achieved without per-tenant substrate. + +**This is a new coupling and it belongs in §3.2.** Draft-3 records two +couplings (E4 needs P3; R's horizon is bounded by P). There is a third: +**R4 at tenant granularity requires per-tenant keys, which is an E-plane +capability.** Reaching the top of the retention ladder is not a retention +project. + +A reference implementation exists for the consumer-granularity case: +`pg_hardstorage` "implements crypto-shred by destroying the per-tenant KEK that +wraps every encrypted backup's DEK" (ibid.). Per the §14 adoption stance we +take the **structure** — envelope encryption with a per-consumer key-encryption +key wrapping each backup's data-encryption key, and destruction of the KEK as +the erasure act — and build it on our own OpenBao and `age` lane rather than +adopting the extension. + +## 1.4 The tension nobody should paper over: erasure versus audit + +`audit-core` exists to hold a tamper-evident trail. Crypto-shredding a tenant's +audit records destroys evidence, and ADR-0001 §2 deliberately built the role +model so that "a compromised runtime credential could not rewrite history at +the schema level". Erasure is a sanctioned way to do exactly that. + +The literature treats this as a genuine design tension rather than a solved +problem ([Remote Reason — balancing auditability and privacy with crypto-shredding](https://remotereason.com/blog/balancing-auditability-and-privacy-with-crypto-shredding)). +The usual resolution is to separate the *fact* of an event (retained, not +personal) from its *personal payload* (encrypted per subject, shreddable), so +the trail survives erasure with its personal content removed. + +**This is a question for `audit-core`, not an answer this framework should +supply.** Flagging it because a naive "R4 everywhere" target would silently +instruct the audit service to destroy its own evidence. + +## 1.5 Proposed amendments to draft-3 + +1. **`R4` admits two routes, and must name which one a service uses:** + - *Horizon-elapsed* — deletion is complete once the published erasure + horizon has passed. Available to everyone, proves little, slow. + - *Key-destroyed* — data is rendered unreadable immediately by destroying the + key that decrypts it. Requires per-entity keys, AES-256-class encryption, + and an auditable destruction record. +2. **Add the third coupling to §3.2:** R4 at tenant granularity requires an + E-plane capability (per-tenant encryption). R4 at consumer granularity + requires per-consumer keys on the retained artifacts. +3. **State the regulatory position honestly** wherever R4 is described: DPAs + have accepted key destruction as erasure under conditions; the EDPB has not + formally endorsed it. A plan tier claiming "your data is deleted" on the + strength of crypto-shredding is making a defensible but not settled claim, + and §11.4's hard line should say so rather than implying R4 is a clean yes. +4. **Raise the audit tension** as an open question addressed to `audit-core`. + +--- + +# Part 2 — Noisy neighbour: the evidence artifact asserts the wrong thing + +## 2.1 The problem draft-3 left standing + +ADR-008 §13 proposes a P1/P2 evidence artifact: "one consumer saturating its +connection or CPU allowance demonstrably **does not breach** another's." + +That is not achievable, and the reference guidance says so in its first line of +solution text: + +> "Sharing a single resource inherently carries the risk of noisy neighbor +> problems that you can't completely avoid." +> — [Azure Architecture Center — Noisy Neighbor antipattern](https://learn.microsoft.com/en-us/azure/architecture/antipatterns/noisy-neighbor/noisy-neighbor) + +An artifact that can only ever fail, or that passes by being run gently enough, +is worse than no artifact — it is the overclaim §6 prohibits, dressed as +evidence. + +## 2.2 There are two failure shapes, and we only guard against one + +Azure distinguishes them explicitly: + +- **One greedy tenant.** "At peak times, tenant A uses all of the system's + resources, which means that any requests that tenant B makes fail." (ibid.) +- **Aggregate of many modest tenants.** "The noisy neighbor problem also occurs + when each individual tenant consumes only a small portion of the system's + capacity. However, the combined resource usage of many tenants can result in + a peak in overall usage." (ibid.) + +Our per-consumer governance — `connection_limit`, `statement_timeout`, +`idle_in_transaction_session_timeout`, applied at database scope per ADR-0001 +§3 — addresses the first shape well. It does nothing about the second. + +**And the second shape is exactly what §17 computes.** Seven consumers, each +politely within its 14-connection allowance, request 98 of 100 connections. +Every one of them is well-behaved; the instance still fails. The connection +ceiling in §17 is not merely a capacity number — **it is the aggregate +noisy-neighbour bound**, and framing it as a scaling statistic understates what +it is. + +## 2.3 The detection signature we can actually build + +Azure's service-side diagnostic is specific and unintuitive: + +> "**Failures when performing an operation for a tenant:** Look for failures +> that occur when a tenant isn't consuming a large share of the system's +> resources. This pattern might indicate that the tenant is experiencing a +> noisy neighbor problem. Track resource consumption by tenant." (ibid.) + +The signal is a failure for a *quiet* consumer. That requires per-consumer +resource attribution, and we already have the raw material: ADR-0001 stamps +`application_name` per role (`/runtime`), PostgreSQL logs carry +database, user, application name and client address, and +`make postgres-ops-status` already reports sessions by application name. + +What is missing is a **baseline**. "It's important to clearly understand your +normal baseline resource usage and to configure monitoring and alerts to detect +spikes" (ibid.). No baseline for `platform-pg` has been recorded. Without one, +a spike is indistinguishable from a Tuesday. + +## 2.4 A normative requirement we are not meeting + +> "Be transparent with clients about any throttling mechanisms or usage quotas +> that you enforce. It's important that they handle failed requests gracefully +> and aren't caught off guard by limitations." (ibid.) + +`tenant-engine` was told its connection and timeout limits during provisioning, +which is good practice by accident rather than by rule. Nothing requires it. A +consumer discovering its `statement_timeout` by hitting it in production is the +failure this warns about. + +**Proposed:** fold quota transparency into §10 (blast radius must be published) +— the same disclosure obligation, extended from what a leaked credential can +reach to what the platform will refuse to do for you. + +## 2.5 Mitigations we have, and the ones we do not + +| Azure-recommended control | Our status | +|---|---| +| Query time limits / max returnable records | **Have.** `statement_timeout`, per-role since the tenant-engine change. | +| Connection quotas per tenant | **Have.** Database-scoped, per ADR-0001 §3. | +| Monitor overall *and* per-tenant resource usage | **Partial.** Attribution exists; no baseline, no alerting. | +| Throttling / rate limiting | **None.** No request-rate control at the database boundary. | +| Quality-of-service prioritisation | **None.** All consumers are equal, including on the synchronous authorization path where `tenant-engine` now sits. | +| Run heavy non-time-sensitive work off-peak | **Implicit.** Backups at 02:15; nothing else scheduled. | +| Rebalance tenants across stamps | **Not possible.** One instance, no second cell (§19.8). | + +The QoS absence is worth noting against a specific fact: `tenant-engine` chose +a 5s statement timeout because `flex-auth` calls it synchronously on the +authorization path. That is a latency-critical consumer sharing an instance +with `audit-core`, which is not. Nothing prioritises one over the other, and +the framework has no vocabulary for saying one consumer's latency matters more. + +## 2.6 Proposed amendment to draft-3 + +Replace the §13 P1/P2 artifact with one that asserts something true: + +> **Noisy-neighbour characterisation (P1/P2, adversarial, load-generated).** +> A recorded baseline of normal per-consumer resource usage; a load run in +> which one consumer saturates its declared allowance; evidence that (a) the +> governance controls bind — the greedy consumer is throttled at its declared +> limits rather than exceeding them, and (b) the measured degradation +> experienced by co-resident consumers is recorded and judged acceptable. +> Carries a review date and the aggregate headroom at time of measurement. + +That is achievable, it is falsifiable, and it produces a number — degradation +under contention — that a plan tier promising performance would need anyway. + +--- + +## 3. Summary of changes these two gaps force + +| # | Change | Where | +|---|---|---| +| A | `R4` splits into *horizon-elapsed* and *key-destroyed* routes; a service must name which | §4.5 | +| B | Third coupling: tenant-granularity R4 requires an E-plane capability (per-tenant encryption) | §3.2 | +| C | R4's regulatory standing stated honestly — accepted by DPAs under conditions, not formally endorsed by the EDPB | §4.5, §11.4 | +| D | Erasure-versus-audit tension raised as an open question to `audit-core` | §19 | +| E | Noisy-neighbour artifact reworded from "does not breach" to characterisation with a recorded baseline and measured degradation | §13 | +| F | §17's connection ceiling reframed as the aggregate noisy-neighbour bound, not only a capacity figure | §17 | +| G | Quota transparency added as a disclosure obligation | §10 | +| H | No QoS vocabulary despite a latency-critical consumer sharing with a batch one | §19, new | + +## Sources + +- [oneuptime — How to set up crypto-shredding for GDPR right-to-erasure compliance](https://oneuptime.com/blog/post/2026-02-17-how-to-set-up-crypto-shredding-for-gdpr-right-to-erasure-compliance-in-google-cloud/view) +- [cybertec-postgresql/pg_hardstorage — GDPR Art. 17 crypto-shred](https://github.com/cybertec-postgresql/pg_hardstorage/blob/main/docs/compliance/gdpr-art-17-crypto-shred.md) +- [Granit — Crypto-shredding: GDPR erasure without deleting a single row](https://granit-fx.dev/blog/crypto-shredding-gdpr-erasure-without-deleting-rows/) +- [Granit — Crypto-shredding (dotnet/compliance)](https://granit-fx.dev/dotnet/compliance/crypto-shredding/) +- [Opsio — Right to erasure, Article 17 GDPR, deletion in cloud systems](https://opsiocloud.com/blogs/right-to-erasure-article-17-gdpr-deletion-cloud-systems/) +- [Remote Reason — Balancing auditability and privacy with crypto-shredding](https://remotereason.com/blog/balancing-auditability-and-privacy-with-crypto-shredding) +- [Azure Architecture Center — Noisy Neighbor antipattern](https://learn.microsoft.com/en-us/azure/architecture/antipatterns/noisy-neighbor/noisy-neighbor) +- [Neon — The noisy neighbor problem in multitenant architectures](https://neon.com/blog/noisy-neighbor-multitenant) +- [PingCAP — Playbook: stop noisy neighbors, multi-tenant MySQL at scale](https://www.pingcap.com/playbook-noisy-neighbor-multi-tenant-mysql/) +- [Causal inference for quantifying noisy neighbor effects in multi-tenant cloud environments (arXiv 2604.03145)](https://arxiv.org/pdf/2604.03145) +- [Azure Architecture Center — Throttling pattern](https://learn.microsoft.com/en-us/azure/architecture/patterns/throttling) +- [Azure Architecture Center — Rate Limiting pattern](https://learn.microsoft.com/en-us/azure/architecture/patterns/rate-limiting-pattern)