the-custodian/research/2026-08-17-adr008-gap-research-erasure-and-noisy-neighbour.md

# Gap research — erasure mechanism (R4) and noisy-neighbour evidence

> Research artefact — 2026-08-17. Fifth digest supporting
> `canon/architecture/adr-008-multi-tenancy-model.md` (draft-3). Closes the two
> gaps the four plane digests left open: how `R4` (verified erasure) can be
> reached at all, and what a noisy-neighbour evidence artifact should actually
> assert.
> **Both findings change a level definition in draft-3.**
> Index: `2026-08-17-adr008-multi-tenancy-research-index.md`.

---

# Part 1 — Erasure: R4 needs a mechanism it does not currently have

## 1.1 The problem draft-3 left standing

ADR-008 §4.5 defines `R4` as "deletion proven complete across live data,
backups and derived copies within the published horizon", and `rapp-postgres`
ADR-0002 establishes that on shared substrate the horizon is the instance
maximum — every co-resident consumer's retention window, whichever is longest.

Read together, those say: **erasure completes by waiting.** A tenant exercising
a deletion right is told the data is gone in up to N days, where N is set by
somebody else's retention requirement. That is honest, and it is a weak answer.
It also makes `R4` unreachable in any useful sense, because "we waited" is not
a proof.

## 1.2 Crypto-shredding is the established mechanism

The industry answer to erasure under immutable or long-retention storage is to
make the data unreadable rather than absent.

> "Crypto-shredding for backups encrypts each subject (or tenant) with a unique
> key derived from the subject identifier, and on erasure, destroys the key.
> The backups remain but are cryptographically inaccessible."
> — [oneuptime — setting up crypto-shredding for GDPR right-to-erasure](https://oneuptime.com/blog/post/2026-02-17-how-to-set-up-crypto-shredding-for-gdpr-right-to-erasure-compliance-in-google-cloud/view)

It is aimed precisely at our situation: the named use cases are "append-only or
immutable storage (event sourcing, WORM, immutable backups) where physical
deletion is impossible"
([Granit — crypto-shredding: GDPR erasure without deleting rows](https://granit-fx.dev/blog/crypto-shredding-gdpr-erasure-without-deleting-rows/)).

**Regulatory standing, stated carefully because overclaiming here would be
worse than in any other section of this framework.** The EDPB has *not* formally
endorsed crypto-shredding as Article 17 erasure. Several data protection
authorities have accepted it where physical deletion would be manifestly
disproportionate effort under Article 17(1), and EDPB Guidelines 5/2019, the UK
ICO and the French CNIL are cited as recognising it **provided three conditions
hold**: strong encryption (AES-256), irreversible key destruction, and auditable
destruction
([pg_hardstorage — GDPR Art. 17 crypto-shred](https://github.com/cybertec-postgresql/pg_hardstorage/blob/main/docs/compliance/gdpr-art-17-crypto-shred.md),
[Opsio — right to erasure in cloud systems](https://opsiocloud.com/blogs/right-to-erasure-article-17-gdpr-deletion-cloud-systems/)).

The third condition is the one this estate is well placed for and the one most
implementations skip: **destruction must be auditable**. ADR-0002 already
requires every sweep to emit an evidence record. Key destruction is the same
shape.

## 1.3 The finding: shredding granularity is bounded by the key boundary

Crypto-shredding "requires per-entity key isolation: each entity instance gets
its own encryption key" (oneuptime, ibid.). That single requirement determines
what is available to us, and it is not uniform across the estate.

| What we want to shred | Key boundary needed | Available today? |
|---|---|---|
| A consumer's logical exports | Per-consumer key | **Close.** The offsite lane already encrypts with `age`, and OpenBao holds key material. This is a key-management change, not an architecture change. |
| A consumer's physical backup | Per-consumer key on the Barman object store | **No.** One WAL stream, one credential, one encryption context for the whole instance. Same constraint that forced the derived maximum in ADR-0002 §2. |
| A single tenant's rows inside a consumer | Per-tenant key, applied by the application before the data reaches storage | **No, and not the platform's to build.** Requires application-level per-tenant encryption. |

The third row is the architecturally interesting one. Per-tenant crypto-shred
means the consumer encrypts tenant data under a tenant-specific key before
writing it — which is an **enforcement-plane** property, not a retention one. It
also happens to be a strong one: data encrypted per tenant is unreadable across
the boundary even by a credential that can address the rows, which is E4-class
protection achieved without per-tenant substrate.

**This is a new coupling and it belongs in §3.2.** Draft-3 records two
couplings (E4 needs P3; R's horizon is bounded by P). There is a third:
**R4 at tenant granularity requires per-tenant keys, which is an E-plane
capability.** Reaching the top of the retention ladder is not a retention
project.

A reference implementation exists for the consumer-granularity case:
`pg_hardstorage` "implements crypto-shred by destroying the per-tenant KEK that
wraps every encrypted backup's DEK" (ibid.). Per the §14 adoption stance we
take the **structure** — envelope encryption with a per-consumer key-encryption
key wrapping each backup's data-encryption key, and destruction of the KEK as
the erasure act — and build it on our own OpenBao and `age` lane rather than
adopting the extension.

## 1.4 The tension nobody should paper over: erasure versus audit

`audit-core` exists to hold a tamper-evident trail. Crypto-shredding a tenant's
audit records destroys evidence, and ADR-0001 §2 deliberately built the role
model so that "a compromised runtime credential could not rewrite history at
the schema level". Erasure is a sanctioned way to do exactly that.

The literature treats this as a genuine design tension rather than a solved
problem ([Remote Reason — balancing auditability and privacy with crypto-shredding](https://remotereason.com/blog/balancing-auditability-and-privacy-with-crypto-shredding)).
The usual resolution is to separate the *fact* of an event (retained, not
personal) from its *personal payload* (encrypted per subject, shreddable), so
the trail survives erasure with its personal content removed.

**This is a question for `audit-core`, not an answer this framework should
supply.** Flagging it because a naive "R4 everywhere" target would silently
instruct the audit service to destroy its own evidence.

## 1.5 Proposed amendments to draft-3

1. **`R4` admits two routes, and must name which one a service uses:**
   - *Horizon-elapsed* — deletion is complete once the published erasure
     horizon has passed. Available to everyone, proves little, slow.
   - *Key-destroyed* — data is rendered unreadable immediately by destroying the
     key that decrypts it. Requires per-entity keys, AES-256-class encryption,
     and an auditable destruction record.
2. **Add the third coupling to §3.2:** R4 at tenant granularity requires an
   E-plane capability (per-tenant encryption). R4 at consumer granularity
   requires per-consumer keys on the retained artifacts.
3. **State the regulatory position honestly** wherever R4 is described: DPAs
   have accepted key destruction as erasure under conditions; the EDPB has not
   formally endorsed it. A plan tier claiming "your data is deleted" on the
   strength of crypto-shredding is making a defensible but not settled claim,
   and §11.4's hard line should say so rather than implying R4 is a clean yes.
4. **Raise the audit tension** as an open question addressed to `audit-core`.

---

# Part 2 — Noisy neighbour: the evidence artifact asserts the wrong thing

## 2.1 The problem draft-3 left standing

ADR-008 §13 proposes a P1/P2 evidence artifact: "one consumer saturating its
connection or CPU allowance demonstrably **does not breach** another's."

That is not achievable, and the reference guidance says so in its first line of
solution text:

> "Sharing a single resource inherently carries the risk of noisy neighbor
> problems that you can't completely avoid."
> — [Azure Architecture Center — Noisy Neighbor antipattern](https://learn.microsoft.com/en-us/azure/architecture/antipatterns/noisy-neighbor/noisy-neighbor)

An artifact that can only ever fail, or that passes by being run gently enough,
is worse than no artifact — it is the overclaim §6 prohibits, dressed as
evidence.

## 2.2 There are two failure shapes, and we only guard against one

Azure distinguishes them explicitly:

- **One greedy tenant.** "At peak times, tenant A uses all of the system's
  resources, which means that any requests that tenant B makes fail." (ibid.)
- **Aggregate of many modest tenants.** "The noisy neighbor problem also occurs
  when each individual tenant consumes only a small portion of the system's
  capacity. However, the combined resource usage of many tenants can result in
  a peak in overall usage." (ibid.)

Our per-consumer governance — `connection_limit`, `statement_timeout`,
`idle_in_transaction_session_timeout`, applied at database scope per ADR-0001
§3 — addresses the first shape well. It does nothing about the second.

**And the second shape is exactly what §17 computes.** Seven consumers, each
politely within its 14-connection allowance, request 98 of 100 connections.
Every one of them is well-behaved; the instance still fails. The connection
ceiling in §17 is not merely a capacity number — **it is the aggregate
noisy-neighbour bound**, and framing it as a scaling statistic understates what
it is.

## 2.3 The detection signature we can actually build

Azure's service-side diagnostic is specific and unintuitive:

> "**Failures when performing an operation for a tenant:** Look for failures
> that occur when a tenant isn't consuming a large share of the system's
> resources. This pattern might indicate that the tenant is experiencing a
> noisy neighbor problem. Track resource consumption by tenant." (ibid.)

The signal is a failure for a *quiet* consumer. That requires per-consumer
resource attribution, and we already have the raw material: ADR-0001 stamps
`application_name` per role (`<consumer>/runtime`), PostgreSQL logs carry
database, user, application name and client address, and
`make postgres-ops-status` already reports sessions by application name.

What is missing is a **baseline**. "It's important to clearly understand your
normal baseline resource usage and to configure monitoring and alerts to detect
spikes" (ibid.). No baseline for `platform-pg` has been recorded. Without one,
a spike is indistinguishable from a Tuesday.

## 2.4 A normative requirement we are not meeting

> "Be transparent with clients about any throttling mechanisms or usage quotas
> that you enforce. It's important that they handle failed requests gracefully
> and aren't caught off guard by limitations." (ibid.)

`tenant-engine` was told its connection and timeout limits during provisioning,
which is good practice by accident rather than by rule. Nothing requires it. A
consumer discovering its `statement_timeout` by hitting it in production is the
failure this warns about.

**Proposed:** fold quota transparency into §10 (blast radius must be published)
— the same disclosure obligation, extended from what a leaked credential can
reach to what the platform will refuse to do for you.

## 2.5 Mitigations we have, and the ones we do not

| Azure-recommended control | Our status |
|---|---|
| Query time limits / max returnable records | **Have.** `statement_timeout`, per-role since the tenant-engine change. |
| Connection quotas per tenant | **Have.** Database-scoped, per ADR-0001 §3. |
| Monitor overall *and* per-tenant resource usage | **Partial.** Attribution exists; no baseline, no alerting. |
| Throttling / rate limiting | **None.** No request-rate control at the database boundary. |
| Quality-of-service prioritisation | **None.** All consumers are equal, including on the synchronous authorization path where `tenant-engine` now sits. |
| Run heavy non-time-sensitive work off-peak | **Implicit.** Backups at 02:15; nothing else scheduled. |
| Rebalance tenants across stamps | **Not possible.** One instance, no second cell (§19.8). |

The QoS absence is worth noting against a specific fact: `tenant-engine` chose
a 5s statement timeout because `flex-auth` calls it synchronously on the
authorization path. That is a latency-critical consumer sharing an instance
with `audit-core`, which is not. Nothing prioritises one over the other, and
the framework has no vocabulary for saying one consumer's latency matters more.

## 2.6 Proposed amendment to draft-3

Replace the §13 P1/P2 artifact with one that asserts something true:

> **Noisy-neighbour characterisation (P1/P2, adversarial, load-generated).**
> A recorded baseline of normal per-consumer resource usage; a load run in
> which one consumer saturates its declared allowance; evidence that (a) the
> governance controls bind — the greedy consumer is throttled at its declared
> limits rather than exceeding them, and (b) the measured degradation
> experienced by co-resident consumers is recorded and judged acceptable.
> Carries a review date and the aggregate headroom at time of measurement.

That is achievable, it is falsifiable, and it produces a number — degradation
under contention — that a plan tier promising performance would need anyway.

---

## 3. Summary of changes these two gaps force

| # | Change | Where |
|---|---|---|
| A | `R4` splits into *horizon-elapsed* and *key-destroyed* routes; a service must name which | §4.5 |
| B | Third coupling: tenant-granularity R4 requires an E-plane capability (per-tenant encryption) | §3.2 |
| C | R4's regulatory standing stated honestly — accepted by DPAs under conditions, not formally endorsed by the EDPB | §4.5, §11.4 |
| D | Erasure-versus-audit tension raised as an open question to `audit-core` | §19 |
| E | Noisy-neighbour artifact reworded from "does not breach" to characterisation with a recorded baseline and measured degradation | §13 |
| F | §17's connection ceiling reframed as the aggregate noisy-neighbour bound, not only a capacity figure | §17 |
| G | Quota transparency added as a disclosure obligation | §10 |
| H | No QoS vocabulary despite a latency-critical consumer sharing with a batch one | §19, new |

## Sources

- [oneuptime — How to set up crypto-shredding for GDPR right-to-erasure compliance](https://oneuptime.com/blog/post/2026-02-17-how-to-set-up-crypto-shredding-for-gdpr-right-to-erasure-compliance-in-google-cloud/view)
- [cybertec-postgresql/pg_hardstorage — GDPR Art. 17 crypto-shred](https://github.com/cybertec-postgresql/pg_hardstorage/blob/main/docs/compliance/gdpr-art-17-crypto-shred.md)
- [Granit — Crypto-shredding: GDPR erasure without deleting a single row](https://granit-fx.dev/blog/crypto-shredding-gdpr-erasure-without-deleting-rows/)
- [Granit — Crypto-shredding (dotnet/compliance)](https://granit-fx.dev/dotnet/compliance/crypto-shredding/)
- [Opsio — Right to erasure, Article 17 GDPR, deletion in cloud systems](https://opsiocloud.com/blogs/right-to-erasure-article-17-gdpr-deletion-cloud-systems/)
- [Remote Reason — Balancing auditability and privacy with crypto-shredding](https://remotereason.com/blog/balancing-auditability-and-privacy-with-crypto-shredding)
- [Azure Architecture Center — Noisy Neighbor antipattern](https://learn.microsoft.com/en-us/azure/architecture/antipatterns/noisy-neighbor/noisy-neighbor)
- [Neon — The noisy neighbor problem in multitenant architectures](https://neon.com/blog/noisy-neighbor-multitenant)
- [PingCAP — Playbook: stop noisy neighbors, multi-tenant MySQL at scale](https://www.pingcap.com/playbook-noisy-neighbor-multi-tenant-mysql/)
- [Causal inference for quantifying noisy neighbor effects in multi-tenant cloud environments (arXiv 2604.03145)](https://arxiv.org/pdf/2604.03145)
- [Azure Architecture Center — Throttling pattern](https://learn.microsoft.com/en-us/azure/architecture/patterns/throttling)
- [Azure Architecture Center — Rate Limiting pattern](https://learn.microsoft.com/en-us/azure/architecture/patterns/rate-limiting-pattern)