coulomb/the-custodian
4
0
Fork 0
Code Issues 1 Pull requests Projects Releases Packages Wiki Activity Actions

research: close the two gaps — erasure mechanism and noisy-neighbour evidence

Browse source 
Both findings change a level definition in draft-3, so neither is decoration.

R4 had no mechanism. As drafted it meant erasure completes by waiting out a
horizon somebody else's retention set, which is honest and useless as proof.
Crypto-shredding is the established answer: encrypt per entity, destroy the
key, leave the immutable backup unreadable. DPAs have accepted it where
physical deletion is disproportionate; the EDPB has not formally endorsed it,
and R4 should say that rather than imply a clean yes.

The finding underneath it is that shredding granularity is bounded by the key
boundary. Per-consumer shred of logical exports is close - the age lane and
OpenBao already exist. Per-consumer shred of physical backups is not available,
for the same reason retention is not per-consumer. And per-TENANT shred needs
the application to encrypt under a tenant key before writing, which makes the
top of the retention ladder an enforcement-plane capability. That is a third
coupling the ADR does not record.

Also flagged: crypto-shredding an audit trail destroys the evidence audit-core
exists to hold. A naive R4-everywhere target would instruct it to do exactly
that. A question for audit-core, not an answer this framework should give.

The noisy-neighbour artifact asserted something unachievable - that one
consumer saturating its allowance does not breach another's. Azure's first line
of solution text is that the risk cannot be completely avoided. An artifact
that can only fail, or that passes by being run gently, is an overclaim dressed
as evidence. Replaced with characterisation: a recorded baseline, a saturation
run, proof the controls bind, and the measured degradation written down.

That research also reframes the connection ceiling. Seven consumers each
politely inside a 14-connection allowance still exhaust the instance, which is
the aggregate noisy-neighbour shape - so the number in the scaling section is
not a capacity statistic, it is the bound.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
codex 2026-08-17 14:37:02 +02:00
parent d0b57b1a50
commit e03c280f40
1 changed files with 265 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

265
research/2026-08-17-adr008-gap-research-erasure-and-noisy-neighbour.md Normal file
View file

@ -0,0 +1,265 @@
# Gap research — erasure mechanism (R4) and noisy-neighbour evidence
> Research artefact — 2026-08-17. Fifth digest supporting
> `canon/architecture/adr-008-multi-tenancy-model.md` (draft-3). Closes the two
> gaps the four plane digests left open: how `R4` (verified erasure) can be
> reached at all, and what a noisy-neighbour evidence artifact should actually
> assert.
> **Both findings change a level definition in draft-3.**
> Index: `2026-08-17-adr008-multi-tenancy-research-index.md`.
---
# Part 1 — Erasure: R4 needs a mechanism it does not currently have
## 1.1 The problem draft-3 left standing
ADR-008 §4.5 defines `R4` as "deletion proven complete across live data,
backups and derived copies within the published horizon", and `rapp-postgres`
ADR-0002 establishes that on shared substrate the horizon is the instance
maximum — every co-resident consumer's retention window, whichever is longest.
Read together, those say: **erasure completes by waiting.** A tenant exercising
a deletion right is told the data is gone in up to N days, where N is set by
somebody else's retention requirement. That is honest, and it is a weak answer.
It also makes `R4` unreachable in any useful sense, because "we waited" is not
a proof.
## 1.2 Crypto-shredding is the established mechanism
The industry answer to erasure under immutable or long-retention storage is to
make the data unreadable rather than absent.
> "Crypto-shredding for backups encrypts each subject (or tenant) with a unique
> key derived from the subject identifier, and on erasure, destroys the key.
> The backups remain but are cryptographically inaccessible."
> — [oneuptime — setting up crypto-shredding for GDPR right-to-erasure](https://oneuptime.com/blog/post/2026-02-17-how-to-set-up-crypto-shredding-for-gdpr-right-to-erasure-compliance-in-google-cloud/view)
It is aimed precisely at our situation: the named use cases are "append-only or
immutable storage (event sourcing, WORM, immutable backups) where physical
deletion is impossible"
([Granit — crypto-shredding: GDPR erasure without deleting rows](https://granit-fx.dev/blog/crypto-shredding-gdpr-erasure-without-deleting-rows/)).
**Regulatory standing, stated carefully because overclaiming here would be
worse than in any other section of this framework.** The EDPB has *not* formally
endorsed crypto-shredding as Article 17 erasure. Several data protection
authorities have accepted it where physical deletion would be manifestly
disproportionate effort under Article 17(1), and EDPB Guidelines 5/2019, the UK
ICO and the French CNIL are cited as recognising it **provided three conditions
hold**: strong encryption (AES-256), irreversible key destruction, and auditable
destruction
([pg_hardstorage — GDPR Art. 17 crypto-shred](https://github.com/cybertec-postgresql/pg_hardstorage/blob/main/docs/compliance/gdpr-art-17-crypto-shred.md),
[Opsio — right to erasure in cloud systems](https://opsiocloud.com/blogs/right-to-erasure-article-17-gdpr-deletion-cloud-systems/)).
The third condition is the one this estate is well placed for and the one most
implementations skip: **destruction must be auditable**. ADR-0002 already
requires every sweep to emit an evidence record. Key destruction is the same
shape.
## 1.3 The finding: shredding granularity is bounded by the key boundary
Crypto-shredding "requires per-entity key isolation: each entity instance gets
its own encryption key" (oneuptime, ibid.). That single requirement determines
what is available to us, and it is not uniform across the estate.
| What we want to shred | Key boundary needed | Available today? |
|---|---|---|
| A consumer's logical exports | Per-consumer key | **Close.** The offsite lane already encrypts with `age`, and OpenBao holds key material. This is a key-management change, not an architecture change. |
| A consumer's physical backup | Per-consumer key on the Barman object store | **No.** One WAL stream, one credential, one encryption context for the whole instance. Same constraint that forced the derived maximum in ADR-0002 §2. |
| A single tenant's rows inside a consumer | Per-tenant key, applied by the application before the data reaches storage | **No, and not the platform's to build.** Requires application-level per-tenant encryption. |
The third row is the architecturally interesting one. Per-tenant crypto-shred
means the consumer encrypts tenant data under a tenant-specific key before
writing it — which is an **enforcement-plane** property, not a retention one. It
also happens to be a strong one: data encrypted per tenant is unreadable across
the boundary even by a credential that can address the rows, which is E4-class
protection achieved without per-tenant substrate.
**This is a new coupling and it belongs in §3.2.** Draft-3 records two
couplings (E4 needs P3; R's horizon is bounded by P). There is a third:
**R4 at tenant granularity requires per-tenant keys, which is an E-plane
capability.** Reaching the top of the retention ladder is not a retention
project.
A reference implementation exists for the consumer-granularity case:
`pg_hardstorage` "implements crypto-shred by destroying the per-tenant KEK that
wraps every encrypted backup's DEK" (ibid.). Per the §14 adoption stance we
take the **structure** — envelope encryption with a per-consumer key-encryption
key wrapping each backup's data-encryption key, and destruction of the KEK as
the erasure act — and build it on our own OpenBao and `age` lane rather than
adopting the extension.
## 1.4 The tension nobody should paper over: erasure versus audit
`audit-core` exists to hold a tamper-evident trail. Crypto-shredding a tenant's
audit records destroys evidence, and ADR-0001 §2 deliberately built the role
model so that "a compromised runtime credential could not rewrite history at
the schema level". Erasure is a sanctioned way to do exactly that.
The literature treats this as a genuine design tension rather than a solved
problem ([Remote Reason — balancing auditability and privacy with crypto-shredding](https://remotereason.com/blog/balancing-auditability-and-privacy-with-crypto-shredding)).
The usual resolution is to separate the *fact* of an event (retained, not
personal) from its *personal payload* (encrypted per subject, shreddable), so
the trail survives erasure with its personal content removed.
**This is a question for `audit-core`, not an answer this framework should
supply.** Flagging it because a naive "R4 everywhere" target would silently
instruct the audit service to destroy its own evidence.
## 1.5 Proposed amendments to draft-3
1. **`R4` admits two routes, and must name which one a service uses:**
- *Horizon-elapsed* — deletion is complete once the published erasure
horizon has passed. Available to everyone, proves little, slow.
- *Key-destroyed* — data is rendered unreadable immediately by destroying the
key that decrypts it. Requires per-entity keys, AES-256-class encryption,
and an auditable destruction record.
2. **Add the third coupling to §3.2:** R4 at tenant granularity requires an
E-plane capability (per-tenant encryption). R4 at consumer granularity
requires per-consumer keys on the retained artifacts.
3. **State the regulatory position honestly** wherever R4 is described: DPAs
have accepted key destruction as erasure under conditions; the EDPB has not
formally endorsed it. A plan tier claiming "your data is deleted" on the
strength of crypto-shredding is making a defensible but not settled claim,
and §11.4's hard line should say so rather than implying R4 is a clean yes.
4. **Raise the audit tension** as an open question addressed to `audit-core`.
---
# Part 2 — Noisy neighbour: the evidence artifact asserts the wrong thing
## 2.1 The problem draft-3 left standing
ADR-008 §13 proposes a P1/P2 evidence artifact: "one consumer saturating its
connection or CPU allowance demonstrably **does not breach** another's."
That is not achievable, and the reference guidance says so in its first line of
solution text:
> "Sharing a single resource inherently carries the risk of noisy neighbor
> problems that you can't completely avoid."
> — [Azure Architecture Center — Noisy Neighbor antipattern](https://learn.microsoft.com/en-us/azure/architecture/antipatterns/noisy-neighbor/noisy-neighbor)
An artifact that can only ever fail, or that passes by being run gently enough,
is worse than no artifact — it is the overclaim §6 prohibits, dressed as
evidence.
## 2.2 There are two failure shapes, and we only guard against one
Azure distinguishes them explicitly:
- **One greedy tenant.** "At peak times, tenant A uses all of the system's
resources, which means that any requests that tenant B makes fail." (ibid.)
- **Aggregate of many modest tenants.** "The noisy neighbor problem also occurs
when each individual tenant consumes only a small portion of the system's
capacity. However, the combined resource usage of many tenants can result in
a peak in overall usage." (ibid.)
Our per-consumer governance — `connection_limit`, `statement_timeout`,
`idle_in_transaction_session_timeout`, applied at database scope per ADR-0001
§3 — addresses the first shape well. It does nothing about the second.
**And the second shape is exactly what §17 computes.** Seven consumers, each
politely within its 14-connection allowance, request 98 of 100 connections.
Every one of them is well-behaved; the instance still fails. The connection
ceiling in §17 is not merely a capacity number — **it is the aggregate
noisy-neighbour bound**, and framing it as a scaling statistic understates what
it is.
## 2.3 The detection signature we can actually build
Azure's service-side diagnostic is specific and unintuitive:
> "**Failures when performing an operation for a tenant:** Look for failures
> that occur when a tenant isn't consuming a large share of the system's
> resources. This pattern might indicate that the tenant is experiencing a
> noisy neighbor problem. Track resource consumption by tenant." (ibid.)
The signal is a failure for a *quiet* consumer. That requires per-consumer
resource attribution, and we already have the raw material: ADR-0001 stamps
`application_name` per role (`<consumer>/runtime`), PostgreSQL logs carry
database, user, application name and client address, and
`make postgres-ops-status` already reports sessions by application name.
What is missing is a **baseline**. "It's important to clearly understand your
normal baseline resource usage and to configure monitoring and alerts to detect
spikes" (ibid.). No baseline for `platform-pg` has been recorded. Without one,
a spike is indistinguishable from a Tuesday.
## 2.4 A normative requirement we are not meeting
> "Be transparent with clients about any throttling mechanisms or usage quotas
> that you enforce. It's important that they handle failed requests gracefully
> and aren't caught off guard by limitations." (ibid.)
`tenant-engine` was told its connection and timeout limits during provisioning,
which is good practice by accident rather than by rule. Nothing requires it. A
consumer discovering its `statement_timeout` by hitting it in production is the
failure this warns about.
**Proposed:** fold quota transparency into §10 (blast radius must be published)
— the same disclosure obligation, extended from what a leaked credential can
reach to what the platform will refuse to do for you.
## 2.5 Mitigations we have, and the ones we do not
| Azure-recommended control | Our status |
|---|---|
| Query time limits / max returnable records | **Have.** `statement_timeout`, per-role since the tenant-engine change. |
| Connection quotas per tenant | **Have.** Database-scoped, per ADR-0001 §3. |
| Monitor overall *and* per-tenant resource usage | **Partial.** Attribution exists; no baseline, no alerting. |
| Throttling / rate limiting | **None.** No request-rate control at the database boundary. |
| Quality-of-service prioritisation | **None.** All consumers are equal, including on the synchronous authorization path where `tenant-engine` now sits. |
| Run heavy non-time-sensitive work off-peak | **Implicit.** Backups at 02:15; nothing else scheduled. |
| Rebalance tenants across stamps | **Not possible.** One instance, no second cell (§19.8). |
The QoS absence is worth noting against a specific fact: `tenant-engine` chose
a 5s statement timeout because `flex-auth` calls it synchronously on the
authorization path. That is a latency-critical consumer sharing an instance
with `audit-core`, which is not. Nothing prioritises one over the other, and
the framework has no vocabulary for saying one consumer's latency matters more.
## 2.6 Proposed amendment to draft-3
Replace the §13 P1/P2 artifact with one that asserts something true:
> **Noisy-neighbour characterisation (P1/P2, adversarial, load-generated).**
> A recorded baseline of normal per-consumer resource usage; a load run in
> which one consumer saturates its declared allowance; evidence that (a) the
> governance controls bind — the greedy consumer is throttled at its declared
> limits rather than exceeding them, and (b) the measured degradation
> experienced by co-resident consumers is recorded and judged acceptable.
> Carries a review date and the aggregate headroom at time of measurement.
That is achievable, it is falsifiable, and it produces a number — degradation
under contention — that a plan tier promising performance would need anyway.
---
## 3. Summary of changes these two gaps force
| # | Change | Where |
|---|---|---|
| A | `R4` splits into *horizon-elapsed* and *key-destroyed* routes; a service must name which | §4.5 |
| B | Third coupling: tenant-granularity R4 requires an E-plane capability (per-tenant encryption) | §3.2 |
| C | R4's regulatory standing stated honestly — accepted by DPAs under conditions, not formally endorsed by the EDPB | §4.5, §11.4 |
| D | Erasure-versus-audit tension raised as an open question to `audit-core` | §19 |
| E | Noisy-neighbour artifact reworded from "does not breach" to characterisation with a recorded baseline and measured degradation | §13 |
| F | §17's connection ceiling reframed as the aggregate noisy-neighbour bound, not only a capacity figure | §17 |
| G | Quota transparency added as a disclosure obligation | §10 |
| H | No QoS vocabulary despite a latency-critical consumer sharing with a batch one | §19, new |
## Sources
- [oneuptime — How to set up crypto-shredding for GDPR right-to-erasure compliance](https://oneuptime.com/blog/post/2026-02-17-how-to-set-up-crypto-shredding-for-gdpr-right-to-erasure-compliance-in-google-cloud/view)
- [cybertec-postgresql/pg_hardstorage — GDPR Art. 17 crypto-shred](https://github.com/cybertec-postgresql/pg_hardstorage/blob/main/docs/compliance/gdpr-art-17-crypto-shred.md)
- [Granit — Crypto-shredding: GDPR erasure without deleting a single row](https://granit-fx.dev/blog/crypto-shredding-gdpr-erasure-without-deleting-rows/)
- [Granit — Crypto-shredding (dotnet/compliance)](https://granit-fx.dev/dotnet/compliance/crypto-shredding/)
- [Opsio — Right to erasure, Article 17 GDPR, deletion in cloud systems](https://opsiocloud.com/blogs/right-to-erasure-article-17-gdpr-deletion-cloud-systems/)
- [Remote Reason — Balancing auditability and privacy with crypto-shredding](https://remotereason.com/blog/balancing-auditability-and-privacy-with-crypto-shredding)
- [Azure Architecture Center — Noisy Neighbor antipattern](https://learn.microsoft.com/en-us/azure/architecture/antipatterns/noisy-neighbor/noisy-neighbor)
- [Neon — The noisy neighbor problem in multitenant architectures](https://neon.com/blog/noisy-neighbor-multitenant)
- [PingCAP — Playbook: stop noisy neighbors, multi-tenant MySQL at scale](https://www.pingcap.com/playbook-noisy-neighbor-multi-tenant-mysql/)
- [Causal inference for quantifying noisy neighbor effects in multi-tenant cloud environments (arXiv 2604.03145)](https://arxiv.org/pdf/2604.03145)
- [Azure Architecture Center — Throttling pattern](https://learn.microsoft.com/en-us/azure/architecture/patterns/throttling)
- [Azure Architecture Center — Rate Limiting pattern](https://learn.microsoft.com/en-us/azure/architecture/patterns/rate-limiting-pattern)