coulomb/the-custodian
4
0
Fork 0
Code Issues 1 Pull requests Projects Releases Packages Wiki Activity Actions

Compare commits

base: coulomb:d710888dc6ce84a2b533aee312b2ac27863c5f22
Branches Tags
coulomb:main
...
compare: coulomb:1674ea550d2012933077837fbe51ce5f5430328e
Branches Tags
coulomb:main

7 commits
d710888dc6 ... 1674ea550d

Author SHA1 Message Date
codex
1674ea550d ADR-008: relocate the multi-tenancy framework to NetKingdom canon
All checks were successful
CI Smoke / host-smoke (push) Successful in 0s
Details
CI Smoke / container-smoke (push) Successful in 1s
Details 
Multi-tenancy is part of the IT-security framework NetKingdom provides, so it
belongs beside the IAM Profile and the tenant-engine boundary contract rather
than in the work-factory canon. Operator decision.

Relocation surfaced two things a review would have caught embarrassingly late.

NetKingdom's accepted platform-identity-security-architecture has used the word
plane since July for a trust and deployment layer - bootstrap, platform
control, tenant. This framework was using the same word for an independent
dimension of concern. Two senses of one word in one canon is precisely the
concept-ownership collision the estate is careful about, and the newcomer
yields: they are now axes. The rename is also just better, since a posture
vector is a point in five-dimensional space.

That same document also disproves the framework's opening line. It has
described the trust model, the tenant model and a capability progression since
2026-07-23, so the claim that the estate had never written down what it was
building was wrong. The accurate and narrower claim is that nothing said how
far a given service had got, or could hold several answers at once.

Stub left behind so the ADR-008 identifier resolves. The renderer moved to
policy-nexus, which owns publication.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-08-17 15:40:59 +02:00
codex
1c65257352 tools: generate the artifact page from canon markdown 
The page and the ADR had already diverged once. They are now one source: the
markdown is authoritative and the page is generated, never hand-edited.

The renderer recognises conventions the document already uses rather than
requiring extra markup, so the source stays a readable document. A table whose
first column is **X0**/**X1** becomes a level ladder; a table headed Threat
becomes the threat matrix; a table with a Kind column gets mechanical and
adversarial chips; a table headed "E \ P" becomes the two-axis grid; ## N.
headings build the section rail.

Stdlib only, per the structure-not-tooling stance. A publishing step that needs
its own toolchain is a publishing step that stops being run - the same
reasoning tenant-engine used for its pin check.

One real consequence beyond deduplication: the E x P matrix had existed only on
the page, so the canonical document did not contain its own central diagram.
It is now a table in the markdown, which is where it should always have been.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-08-17 15:10:22 +02:00
codex
c4f22a5e55 ADR-008 draft-4: apply the eight amendments the gap research forced 
R4 now has a mechanism. It splits into horizon-elapsed (wait out a window
somebody else's retention set - available to all, proves little) and
key-destroyed (encrypt per entity, destroy the key, immediate). A service must
name which route it uses.

The regulatory position is stated rather than implied: DPAs have accepted key
destruction as erasure where physical deletion is disproportionate, under
conditions, and the EDPB has not formally endorsed it. Section 11.4 now says a
service may make that claim but not in language implying a regulator blessed
it. Overclaiming here would be worse than anywhere else in the document.

Third coupling recorded: shredding a single tenant's data needs the application
to encrypt under a per-tenant key before writing, so the top of the retention
ladder is an enforcement-plane capability. Reaching R4 is not a retention
project.

The noisy-neighbour artifact was replaced. It had required proof that a
saturating consumer does not breach another's allowance, which shared
infrastructure cannot provide - an artifact that can only fail, or passes by
being run gently, is an overclaim in the costume of evidence. It now measures:
baseline, saturation run, proof the controls bind, recorded degradation.
Generalised as decision 13.4.

The connection ceiling is reframed. Seven consumers each politely inside a
14-connection allowance still exhaust the instance, so the number is the
aggregate noisy-neighbour bound, and our per-consumer governance guards the
other shape entirely.

Also: quota transparency as a disclosure obligation (10.2) - a consumer
learning its statement timeout by hitting it in production is our failure, not
theirs; and two new open questions - crypto-shredding an audit trail destroys
what audit-core exists to hold, and we have no QoS vocabulary despite a
latency-critical consumer sharing an instance with a batch one.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-08-17 14:49:00 +02:00
codex
e03c280f40 research: close the two gaps — erasure mechanism and noisy-neighbour evidence 
Both findings change a level definition in draft-3, so neither is decoration.

R4 had no mechanism. As drafted it meant erasure completes by waiting out a
horizon somebody else's retention set, which is honest and useless as proof.
Crypto-shredding is the established answer: encrypt per entity, destroy the
key, leave the immutable backup unreadable. DPAs have accepted it where
physical deletion is disproportionate; the EDPB has not formally endorsed it,
and R4 should say that rather than imply a clean yes.

The finding underneath it is that shredding granularity is bounded by the key
boundary. Per-consumer shred of logical exports is close - the age lane and
OpenBao already exist. Per-consumer shred of physical backups is not available,
for the same reason retention is not per-consumer. And per-TENANT shred needs
the application to encrypt under a tenant key before writing, which makes the
top of the retention ladder an enforcement-plane capability. That is a third
coupling the ADR does not record.

Also flagged: crypto-shredding an audit trail destroys the evidence audit-core
exists to hold. A naive R4-everywhere target would instruct it to do exactly
that. A question for audit-core, not an answer this framework should give.

The noisy-neighbour artifact asserted something unachievable - that one
consumer saturating its allowance does not breach another's. Azure's first line
of solution text is that the risk cannot be completely avoided. An artifact
that can only fail, or that passes by being run gently, is an overclaim dressed
as evidence. Replaced with characterisation: a recorded baseline, a saturation
run, proof the controls bind, and the measured degradation written down.

That research also reframes the connection ceiling. Seven consumers each
politely inside a 14-connection allowance still exhaust the instance, which is
the aggregate noisy-neighbour shape - so the number in the scaling section is
not a capacity statistic, it is the bound.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-08-17 14:37:02 +02:00
codex
d0b57b1a50 ADR-008 draft-3: corrections, plane R, and a name 
Renamed to Tenancy Posture. Four Planes stopped being true and a name that
encodes a count breaks every time the framework grows; the posture vector is
the durable centre.

Corrections from the research digests:

- E3 no longer claims the application cannot route around it. Any session can
  re-SET the GUC, so E3 is decisive against a forgotten predicate and useless
  against SQL injection or a compromised process. Replaced the claim with a
  threat-model table. Draft-2 was overclaiming in exactly the way section 6
  prohibits, which is the correction I most wanted made before review.
- E3 layers on E2 rather than replacing it; claiming E3 now requires the E2
  artifact too, because E3 alone fails open under injection.
- The GUC is set transaction-locally, not at pool checkout, and the scaling
  section's pooling claim was backwards: SET LOCAL is what makes E3 safe under
  a pooler; statement-level pooling is what leaks.
- Placement is a default plus per-tenant exceptions. One P level per service
  could not express the isolation tiers section 11 already sells.

Added plane R for retention and erasure, implementing the rapp-postgres
ADR-0002 interface: N-day retention defaulting to 30, a published erasure
horizon, and dataset sweeps whose policy the consumer owns. R exists because
deleting a row does not remove it from yesterday's backup, and the estate had
never written that down.

Also: A4 for a standard PDP interface now that AuthZEN is Final; evidence split
into mechanical and adversarial, since a green CI run is not E2 evidence; the
tenant boundary named as OWASP API1; and section 14 recording the
structure-not-tooling stance.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-08-17 14:33:50 +02:00
codex
f09d3ef5d2 research: record the structure-not-tooling stance and the retention design 
Operator decision: use research as design input, adopt tooling only where it is
an established standard with broad application, otherwise build ground up so we
can refactor freely.

Settles two open recommendations. AuthZEN qualifies (OpenID Final Spec,
multi-vendor interop) and stays as the proposed A4 rung; OWASP stays as a
baseline; PostgreSQL RLS mechanics are facts, not tooling. pgrls and pgTAP are
out as dependencies - we take pgrls's rule taxonomy as a checklist for probes
we write into the existing isolation-test.sh harness, which already found four
real defects and has no dependency tree. Its SAFE/BREAKING policy-diff idea is
the structural contribution worth reimplementing.

Also records that gap G1 was designed rather than deferred, and the finding
that came out of it: per-consumer physical retention is not implementable on
CNPG, so the erasure horizon is the instance maximum and retention becomes a
placement trigger.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-08-17 14:20:33 +02:00
codex
7c69153da9 research: verify pgrls directly; add maturity caveat before recommending adoption 
Checked the repo rather than trusting the search snippet. Confirms the
capability claims (MIT, 67 rules, 54 of them security, Z3 isolation proofs,
SAFE/BREAKING policy-diff for CI gating, pytest plugin, PG 15-17) and corrects
the auto-fixable count from 20 to 19.

It is also 26 stars and self-described beta. That does not change the
recommendation to adopt the approach, but it does change the recommendation to
adopt the tool: it is a supply-chain decision, not a free win. pgTAP is
long-established and carries no such caveat. Saying so now is cheaper than
having a reviewer discover it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
 2026-08-17 13:07:41 +02:00
5 changed files with 382 additions and 547 deletions
Download patch file Download diff file Expand all files Collapse all files

7
Makefile
View file

@ -193,3 +193,10 @@ agents-update:
agents-validate:
@echo "Validating agents..."
@kaizen-agentic validate agents/
.PHONY: artifact-adr008
artifact-adr008:
python3 tools/render-artifact.py canon/architecture/adr-008-multi-tenancy-model.md \
--output build/adr-008.html \
--title "Tenancy Posture" \
--subtitle "A framework for describing, holding and improving multi-tenancy — including where we are not there yet."

562
canon/architecture/adr-008-multi-tenancy-model.md
View file

@ -1,554 +1,34 @@
---
id: ADR-008
type: architecture-decision-record
title: "Multi-Tenancy Framework: Four Planes, Graduated Levels, Declared Posture"
status: proposed
title: "Multi-Tenancy Framework — relocated to NetKingdom"
status: superseded
decided_by: Bernd Worsch
date: "2026-08-17"
revision: "draft-2"
tags: ["architecture", "multi-tenancy", "isolation", "placement", "maturity", "tenant-engine", "flex-auth", "rapp-postgres", "scaling"]
---
# ADR-008: Multi-Tenancy Framework — Four Planes, Graduated Levels, Declared Posture
# ADR-008 — relocated
## Status
Drafts 1–4 of the multi-tenancy framework were written here. On 2026-08-17 the
operator determined that multi-tenancy is part of the IT-security framework
NetKingdom provides, so the framework belongs in NetKingdom canon beside the
IAM Profile and the tenant-engine boundary contract.
**Proposed, draft-2.** Draft-1 (2026-08-17) proposed a single model with fixed
characteristics. It was rejected on the grounds that the estate does not need
one model of multi-tenancy — it needs a framework able to describe, hold, and
improve several situations at once, including repos that do not separate
tenants rigorously today and must be developed toward doing so. Draft-2 is
that reframe.
**It now lives at `net-kingdom/canon/standards/tenancy-posture_v0.1.md`.**
This document decides things currently owned by `tenant-engine`, `flex-auth`,
`rapp-postgres`, `adaptive-pricing`, and `railiance-platform`. It must be
reviewed by those owners before ratification; §18 lists what each is being
asked to accept.
Two things changed on relocation:
## 1. Context
1. The five dimensions were renamed from **planes** to **axes**. NetKingdom's
accepted `platform-identity-security-architecture` already uses *plane* for
a trust and deployment layer (bootstrap / platform control / tenant), and
two senses of one word in one canon is a concept-ownership collision.
2. The opening claim that the estate "has never written down what it is
building" was corrected. That architecture document has described the trust
and tenant model since 2026-07-23. What was actually missing is a way to say
how far a given service has got.
The estate has been building multi-tenancy for months and has never written
down what it is building. Five documents each cover a slice:
This stub remains so that the ADR-008 identifier resolves rather than
dangling. It is not a second copy and must not be edited as one.
| Document | Covers | Status |
|---|---|---|
| `iam-profile_v0.3` (NetKingdom) | Tenant identifier shape, `tenant_roles` claim, staleness rules | Ratified |
| `tenant-engine-boundary-contract_v0.1` (NetKingdom) | Who owns tenant records, roles, plan assignment | Ratified |
| `business-app-service-contract_v0.1` §1 (Custodian) | Business apps: instance-per-client, tenant-keyed data | Ratified |
| `rapp-postgres/docs/adr/ADR-0001` | Consumer + tenant isolation in PostgreSQL | Proposed, governs one repo |
| `rapp-postgres/docs/canon-drafts/shared-platform-relational-storage_v0.1` | The stacked-boundary gap | Routed 2026-08-10, **still unratified** |
Four failures follow.
**The gap has been diagnosed once and the fix stalled.** The
shared-platform-relational-storage draft was written to fill this hole. It has
sat unratified for a week and appears in neither canon directory. §19 attaches
a ratification path so this one does not join it.
**Placement is owned by nobody.** `user-engine-pg` and `target-revenue-pg` are
dedicated; `apps-pg`, `net-kingdom-pg`, `platform-pg`, `state-hub-db` and
`forgejo-db` are shared. Both patterns are live, neither is written down, and
each new service copies whichever neighbour it consulted. `tenant-engine`
raised this with `railiance-platform` on 2026-08-16; unanswered.
**Two contradictory tenancy defaults are already ratified.**
`business-app-service-contract` §1.2 makes instance-per-client the default for
business apps. Platform services pool all tenants into one database. Nothing
states which shape a new service takes, and no definition distinguishes the
two categories.
**There is no honest way to describe a repo that is not there yet.** This is
the failure draft-1 missed. The estate absorbs and will keep absorbing repos
with weak or absent tenant separation — app-local accounts, no tenant column,
ad-hoc filtering. Today such a repo is simply non-conformant, which gives it
two bad options: misrepresent its posture, or stay outside the framework
entirely. Both are worse than a framework that can say "this repo is at level
1, its target is level 3, here is the gap and the evidence."
## 2. What this document is
**A framework, not a model.** It does not specify one correct multi-tenancy
implementation with a fixed set of characteristics. It supplies:
- **Terminology** — four planes, and graduated levels within each (§3, §4).
- **A declaration** — the posture vector, so any repo can state where it
actually is (§5).
- **A conformance rule** — accuracy, not altitude (§6).
- **Methodology** — analyze, establish, improve, guard (§12).
- **Evidence definitions** — what proves a claimed level (§13).
A service is conformant when its declared posture is accurate and its
trajectory is recorded. A service is non-conformant when it claims a level it
cannot evidence — regardless of how high or low that level is.
## 3. Four orthogonal planes
"Is this multi-tenant?" is treated as one question. It is four, and they are
independent:
| Plane | Question | Vocabulary owner |
|---|---|---|
| **Identity (I)** | How is a tenant named and validated? | `tenant-engine` / IAM Profile |
| **Authorization (A)** | How is a request bound to the tenants it may act for? | `flex-auth` |
| **Enforcement (E)** | Where, mechanically, is the tenant boundary enforced? | This framework |
| **Placement (P)** | Which physical substrate holds a tenant's data? | `railiance-platform` |
Treating them as one produces concrete errors today. `rapp-postgres`'s
`PostgresConsumer` carries a field `tenantIsolation: consumer-service-boundary`
— an **E**-plane fact recorded in a **P**-plane artifact, reading as though the
storage platform enforces something it does not. The "dedicated versus shared"
argument routinely mixes P (capacity, blast radius) with E (correctness).
The planes are separated **precisely so that each may sit at a different
level**. A repo can be strong on identity and weak on enforcement, or the
reverse, and the framework must state that without flattening it to a single
verdict. That is the whole point, and draft-1 lost it by collapsing E into a
single fixed rule.
**Decision 3.1:** every document, declaration, and plan tier that says
"isolation" MUST name which plane it means.
**Decision 3.2:** the planes couple at their top levels — `E4` is only
reachable at `P3` or above — and that coupling MUST be stated where it applies
rather than used to argue the planes are one.
## 4. Graduated levels
Each plane carries an ordered ladder. Higher is stronger, not better: the
right level is the one a service can evidence and its risk warrants.
### 4.1 Identity (I) — how the tenant is named
| Level | State |
|---|---|
| **I0** | No tenant concept. Data is not attributable to a tenant. |
| **I1** | A local tenant notion exists but does not conform to `tenant:<grouping>:<name>`. |
| **I2** | Canonical identifiers used, validated against the ratified vocabulary; `tenant-engine` is the source of existence. |
| **I3** | I2 plus capability roles (`PLTF`/`IAM`/`VEN`/`CUS`) honoured, with live `tenant-engine` re-query for privileged, destructive, credential-vending or `aal2`-class decisions rather than trusting the cached `tenant_roles` claim. |
Note `business-app-service-contract` §2.1 sets app-local accounts as the v1
baseline for business apps — an explicitly sanctioned low level with recorded
triggers for moving up. That is the pattern this framework generalises.
### 4.2 Authorization (A) — how a request is bound to a tenant
| Level | State |
|---|---|
| **A0** | No authorization, or tenant context not carried on the request. |
| **A1** | Ad-hoc checks scattered through handlers. |
| **A2** | A single local authorization boundary; tenant context bound once, centrally. |
| **A3** | Decisions delegated to `flex-auth` as PDP, with live re-query where the IAM Profile requires it. |
### 4.3 Enforcement (E) — where the tenant boundary actually holds
This is the reframed core. Draft-1 asserted one rule for everyone; this is a
ladder whose top is platform-enforced.
| Level | Mechanism | Bypassed by |
|---|---|---|
| **E0** | None. Data not tenant-keyed; separation is incidental or absent. | Anything. |
| **E1** | Data tenant-keyed, filtering applied per query at call sites. | One missing predicate. |
| **E2** | Filtering centralised at a single service-side choke point that binds authenticated identity to permitted tenants. | Code that bypasses the choke point. |
| **E3** | Platform-assisted: row-level security keyed on a session GUC set at pool checkout, or an equivalent data-access layer the application cannot trivially route around. | A `SECURITY DEFINER` function, a missing policy, or a wrong GUC. |
| **E4** | Structural: the connection a workload holds cannot address another tenant's data at all. Requires per-tenant credentials and per-tenant substrate. | Only a platform-level compromise. |
Three properties of this ladder matter.
**E3 is real and is not forbidden.** Draft-1 rejected RLS on the grounds that
a control depending on the consumer setting a session variable "looks like a
database guarantee without being one". The observation is correct; the
conclusion was wrong. E3 is materially stronger than E2 — it converts "every
call site must remember" into "one checkout path must be right" — and it is
the standard mitigation in pool-model systems. What was actually wrong was
*calling E3 by E4's name*. The honesty requirement survives; the prohibition
does not.
**Higher enforcement is the direction of travel, not an optional extra.**
Services holding data whose cross-tenant exposure would be a reportable breach
SHOULD target E3 or above. The framework's default expectation for a new
platform service is **E2 at first serve, E3 recorded as target**.
**Platform enforcement is a platform obligation, not only a consumer one.**
Draft-1 placed the whole burden on consumers permanently. It is now stated
that reaching E3 requires the storage platform to *offer* the mechanism —
provisioned RLS policies, a documented GUC contract, and a probe.
`rapp-postgres` ADR-0001 §4 already commits to provisioning it on request;
that offer becomes an obligation with a defined contract (§18.6). Where a
consumer wants E3 and the platform has not supplied the mechanism, the gap is
the platform's, not the consumer's.
### 4.4 Placement (P) — which substrate holds the data
Draft-1's `R0` ("shared tables, tenant column") is removed: that was an
E-plane state mislabelled as placement. Corrected ladder:
| Level | Shape | Live occupants |
|---|---|---|
| **P0** | Shares a database with another consumer (schema or tables co-resident). | None sanctioned; the state absorbed repos often arrive in. |
| **P1** | Database per consumer, shared cluster. | `audit-core`, `tenant-engine` on `platform-pg` |
| **P2** | Dedicated cluster per consumer. | `user-engine-pg`, `target-revenue-pg` |
| **P3** | Dedicated cluster per tenant. | Business apps per `business-app-service-contract` §1.2 |
| **P4** | P3 plus separate region or jurisdiction. | None |
**P0 → P1 → P2 does not raise the E level.** Those steps buy consumer
isolation, capacity predictability, independent backup retention, and a
smaller operational blast radius. They leave the tenant boundary exactly where
the E level puts it. Only P3 makes E4 reachable. This is the single most
misusable fact in the framework and §11 governs how it may be described
commercially.
**Decision 4.5:** P1 is the default for platform services; P3 is the default
for client-facing business apps, as already ratified. A service unsure which
it is must resolve that before choosing (§18.4).
## 5. The posture vector
A service states its tenancy posture as one level per plane, plus a target and
a date:
```yaml
tenancy:
current: { I: 2, A: 3, E: 2, P: 1 }
target: { I: 2, A: 3, E: 3, P: 1 }
reviewed: "2026-08-17"
gap:
E: "Choke point exists and is tested; RLS not yet provisioned. Blocked on
rapp-postgres offering the GUC contract. Target Q4."
```
Worked examples, as best assessed today and subject to owner correction:
| Service | Current | Notes |
|---|---|---|
| `tenant-engine` | `I2 A3 E2 P1` | Moving to P1 under TEN-WP-0009; E2 via its own service boundary. |
| `audit-core` | `I2 A3 E2 P1` | Same shape; holds audit data, so E3 is the more urgent target. |
| A newly absorbed repo | `I1 A1 E1 P0` | Conformant **if declared**, with a recorded path. |
The vector is what makes the framework hold both situations honestly. A repo
at `I1 A1 E1 P0` is not a failure — it is a described state with a direction.
A repo claiming `E3` with no RLS policy is a failure regardless of how good the
rest of it is.
**Decision 5.1:** the posture vector is declared in the repo, not in the hub,
consistent with local-files-are-source-of-truth.
## 6. Honest absorption — conformance is accuracy, not altitude
The governing rule of this framework:
> **A service is conformant when its declared posture is accurate, its target
> is recorded, and it does not claim a level it cannot evidence. It is
> non-conformant when it overclaims — at any altitude.**
Consequences, stated deliberately:
- Declaring `E0` is conformant. Concealing `E0` is not.
- A repo may be absorbed at any posture. It may not be absorbed silently.
- No service is blocked from the estate for being low on a ladder. Services
MAY be blocked from *specific work* — serving a tenant grouping, holding a
data class, carrying a plan tier — by requirements expressed as minimum
levels (§11).
- Downgrading is permitted and must be declared. A regression discovered by
guarding (§12) is a defect; a regression declared in advance is a decision.
This is what the plane separation is *for*. Without it, "not rigorous about
tenant separation" is a single verdict that a repo either passes or fails.
With it, the same repo is `I1 A1 E1 P0` with a path — which is a plan, not an
indictment.
## 7. Portability across placement levels
Movement between P levels must be an operational change, not a rebuild.
Adopted from `tenant-engine`'s TEN-WP-0009 design:
- Connect by injected credential only — no cluster, host, namespace, or
database name in source.
- Own a whole database, never tables inside someone else's. Sharing a cluster
is reversible; sharing a database makes relocation a schema merge.
- Idempotent schema creation.
- No cross-database joins or co-location assumptions.
A service meeting these moves P1 → P2 by: create target, dump, restore, swap
the credential reference, restart.
**Decision 7.1:** mandatory at P1 and above. At P3, SHOULD rather than MUST —
the consolidation `business-app-service-contract` §1.3 preserves is a real
benefit, but a per-client instance that never moves is not misconformant for
naming its own database.
## 8. Placement triggers
A level change decided under pressure is decided badly. Each service records
at provisioning time what would move it:
- Noisy neighbour on a latency-critical path
- A compliance or residency requirement
- A plan tier requiring a higher minimum (§11)
- Backup retention or RTO no longer fitting
- Connection or memory ceiling reached (§16)
**Decision 8.1:** triggers MUST be *monitored*, not merely recorded. A trigger
in a YAML comment that nobody re-reads is documentation, not control. The
`platform-pg` ceiling in §16 is the first that requires an actual alert.
**Decision 8.2:** placement policy ownership is proposed to
`railiance-platform`. This framework supplies the ladder, the triggers, and
the declaration; it does not appoint per-workload placement. See §18.2 — this
remains their call and their open request.
## 9. Credentials as a tenancy control
Short-lived leased credentials from OpenBao, re-read at connection checkout,
with overlap-first rotation, bound the residual risk at every E level below
E4: a leaked credential expires rather than persisting. This is stronger than
the industry norm of a long-lived per-service secret and belongs in the
tenancy model, not only in credential-management canon.
**Decision 9.1:** static long-lived database credentials are not a sanctioned
path for any service above E0.
## 10. Blast radius must be published
**Decision 10.1:** every platform holding consumer data MUST publish, in
concrete terms, what a leaked runtime credential can and cannot reach at the
levels it operates. `rapp-postgres` ADR-0001 §5 is the reference
implementation. Where the model cannot provide a guarantee, the platform says
so and names the escalation, rather than describing a weaker control in
stronger language.
## 11. Commercial expression — room to be successful
Isolation will be sold. The framework must let commercial and presentation
work rather than obstruct them, while preventing a claim that cannot be
evidenced.
**Decision 11.1:** plan tiers are expressed internally as **minimum levels**,
not as implementation labels. A tier may require `E3 P2`; it need not print
that string anywhere customer-facing.
**Decision 11.2:** marketing and product language is free. There is no
requirement to expose level labels, ladder names, or this document to
customers. "Dedicated infrastructure", "isolated tenancy", "private instance"
all remain available.
**Decision 11.3:** the constraint is on **evidence, not vocabulary**. Any
customer-facing isolation or availability claim must map to a minimum level
the delivering service actually holds, recorded once when the tier is defined.
The review is internal and happens at tier definition — not per campaign, per
page, or per conversation.
**Decision 11.4:** the one hard line — a claim that another tenant *cannot*
reach the customer's data requires **E4**. Everything softer than that
absolute has room. This is the claim that reaches compliance questionnaires
and contracts, and it is the one that must be true.
`adaptive-pricing` owns plan terms; `tenant-engine` owns which plan a tenant
holds. Neither currently maps a tier to minimum levels; §18.5 asks them to,
for tiers making isolation or availability claims only.
## 12. Methodology — analyze, establish, improve, guard
The framework is operated as a cycle, not consulted as a reference.
**Analyze.** Assess a repo against the four ladders and produce its current
vector. Output: the `tenancy.current` block with the reasoning recorded.
Applies to new services and absorbed ones alike.
**Establish.** Declare the target vector and the gap. The target is set by the
data class held, the tenant groupings served, and any plan tier carried — not
by ambition. Output: `tenancy.target` plus a recorded gap per plane.
**Improve.** Move one plane at a time. Cross-plane moves are where mistakes
happen — raising P while leaving E untouched is exactly the misstep §4.4
warns about. Each step is ordinary repo work with ordinary evidence.
**Guard.** Verify continuously that the declared posture still holds. Guarding
checks a service **against its own declaration**, not against a universal
maximum. This is what makes the verification problem tractable: nobody has to
prove every service is at E4; the check is that nobody is below what they
declared, and nobody claims what they cannot evidence.
Regression found by guarding is a defect; regression declared in advance is a
decision. The estate has been bitten twice by silent pin rollbacks that
produced ordinary-looking 403s and 404s rather than errors. Posture regression
will look the same, so guarding must be designed for invisible failure rather
than for crashes.
## 13. Evidence per level
Guarding requires each level to have a defined evidence artifact. Proposed:
| Level | Evidence |
|---|---|
| **I2** | Identifiers validated against the ratified vocabulary; rejection test for a malformed tenant id. |
| **I3** | Live re-query demonstrated on an `aal2`-class path; cached-claim path shown not to be used there. |
| **A2** | Single choke point identified; test that an unbound request is refused. |
| **A3** | Live decision against `flex-auth` with a denial observed at the endpoint, not only at the decision surface. |
| **E1** | Every tenant-owned table carries the tenant key. Schema-level check, mechanical. |
| **E2** | Choke point identified; adversarial test that an identity bound to tenant A cannot read tenant B. |
| **E3** | RLS policies present on every tenant-owned table; probe that a session without the GUC set reads nothing; probe that a wrong GUC reads nothing. |
| **E4** | Per-tenant credential demonstrated unable to connect to another tenant's substrate. |
| **P1–P4** | Provisioning declaration plus the platform's own isolation probes. |
**Decision 13.1:** a level is claimed only with its evidence artifact present.
This turns §6's accuracy rule from an honour system into a check.
**Decision 13.2:** the E2 and E3 evidence artifacts do not exist anywhere in
the estate today. `rapp-postgres` runs 15 adversarial probes, all against the
*consumer* boundary, none against the tenant boundary inside a consumer. This
is the framework's largest live gap and §18.3 seeks an owner.
## 14. Alternatives considered
**One fixed model with a single set of characteristics** (draft-1). *Rejected
by the operator:* it cannot describe a repo that is not there yet, forcing
absorbed repos either to misrepresent their posture or to stay outside the
framework. A framework that can only describe its own end state is not a
framework.
**A maturity model with a single overall level** (bronze/silver/gold).
*Rejected:* collapses the plane separation that makes the framework useful. A
service strong on identity and weak on enforcement has a specific, actionable
gap; one composite score hides it and invites averaging.
**Schema-per-consumer in one database** (sanctioning P0). *Rejected:*
`pg_catalog` is readable per-database, so every co-resident enumerates every
other's table and column names regardless of `GRANT`s. Retained in the ladder
as a describable state, never as a sanctioned target.
**Row-level security as the *consumer* boundary.** *Rejected:* puts a
correctness-critical boundary between two services inside query predicates.
Retained as E3 for the *tenant* boundary, where the realistic alternative is
application code rather than a database.
**Mandating E4 for everyone.** *Rejected:* the tenant grouping taxonomy
includes `consumer` (private individuals) and `family`. A cluster per private
individual is economically impossible; the taxonomy is itself evidence that
pooling is required.
**Leaving each repo to its own model.** *Rejected:* this is the status quo,
and it produced two contradictory ratified defaults plus an unowned placement
question.
## 15. Challenge against outside practice
**Where the framework is stronger than typical.** Most SaaS multi-tenancy
literature (AWS's silo/pool/bridge framing among them) models exactly one
boundary — tenant-to-tenant. This estate has **two stacked boundaries**:
platform-service-to-platform-service, and tenant-to-tenant inside a consumer.
Naming them separately and refusing to enforce both with one mechanism is
uncommon and correct.
Graduated per-plane levels are also stronger than the usual silo/pool/bridge
trichotomy, which describes deployment shape only. Silo/pool/bridge is
approximately this framework's P plane with the other three missing — which is
why it cannot express "pooled infrastructure, structurally enforced
boundary", or the reverse.
**Where it is weaker, and this is the finding that matters.** The pool model's
standard mitigation is a *verified* enforcement layer that every service is
demonstrably routed through. The estate has the concept (E2/E3) and none of
the verification (§13.2). A single missing `WHERE tenant_id = ?` in any
consumer is a cross-tenant breach that no platform control catches, no probe
fails, and no log shows as an error.
**Where the estate has quietly adopted a good pattern without naming it.**
Short-lived leased credentials re-read at checkout, with overlap-first
rotation, beat the industry norm of a long-lived per-service secret. §9
promotes it to a tenancy control.
**Cell-based architecture remains unexplored.** Neither P nor E describes a
cell — a fixed-size slice of infrastructure serving a bounded set of tenants,
sized so one cell's failure is survivable and cell count scales linearly.
Given the estate spans railiance, NetKingdom, HelixForge and Coulomb, cells
may fit better than per-tenant clusters at the point P1 runs out (§18.8).
## 16. Scaling demands
Measured against the live `platform-pg` specification, not estimated.
```
instances: 1 (no HA; single-node rail)
max_connections: 100
memory limit: 1Gi
per consumer: 14 connections (12 runtime + 2 migration)
```
**Connection ceiling: roughly six consumers.** Seven consumers request 98 of
100 before CNPG's instance manager, metrics exporter, and reserved slots. Two
are provisioned today. **We are at roughly a third of capacity and the third
request will not feel like a scaling event.**
**Memory likely binds first.** 100 backends against 1Gi is ~10MB per backend
for shared buffers, work_mem and overhead. Connection exhaustion errors
clearly; memory pressure OOM-kills and degrades every co-resident at once.
**Raising E is not free on a shared substrate.** RLS with a per-checkout GUC
does not itself increase connection count, but it forecloses aggressive
transaction-level pooling for consumers that adopt it. E3 on P1 interacts with
the ceiling above and should be planned against it.
**Restore time couples all consumers.** Physical backup is instance-wide, so a
consumer's RTO is a function of *total* instance size, not its own. One large
consumer silently degrades every co-resident's recovery promise.
**`instances: 1` means no P1 tenant has HA.** A tier promising uptime cannot
be satisfied at P1 as currently built — an availability floor belongs in §11's
minimum-level vocabulary alongside isolation.
## 17. Consequences
- The estate gains one vocabulary and, more importantly, a way to be honest
about partial adoption.
- Absorbed repos get a described state and a path instead of a failing grade.
- `tenantIsolation` in `PostgresConsumer` is revealed as a mislabelled field
(§18.1).
- The two ratified defaults are reconciled by scope, at the cost of needing a
classification rule.
- The verification problem becomes tractable: guard against declaration, not
against a universal maximum.
- Draft-1's prohibition on RLS is reversed; `rapp-postgres` acquires an
obligation to define and offer the E3 mechanism.
- Nothing here changes a running system.
## 18. Open questions
1. **`tenantIsolation` field** — `rapp-postgres`: rename to name its plane and
carry a level (`tenancy.E: 2`), or move it out of the storage declaration.
2. **Placement ownership** — `railiance-platform`: accept the ladder, the
triggers, and the monitoring obligation in §8.1, and appoint a recorded
placement owner per workload.
3. **E2/E3 evidence artifacts** — *owner needed.* Who builds the adversarial
cross-tenant probes and the RLS conformance check? Both `audit-core` and
`tenant-engine` have declined fleet-scope work on correct boundary
reasoning, so this needs appointing. Highest-severity gap (§13.2).
4. **Business app vs platform service** — Custodian canon: a classification
rule, so a new repo knows whether P3 or P1 is its default. Candidate: reuse
`repo-classification-standard_v1.0` rather than invent a second taxonomy.
5. **Tier → minimum level mapping** — `adaptive-pricing` and `tenant-engine`:
required only for tiers making isolation or availability claims (§11).
6. **E3 mechanism** — `rapp-postgres`: what exactly is the GUC contract, and
is it provisioned per consumer on request or offered as a standard option?
7. **Engine neutrality** — the P ladder rests on a PostgreSQL property
(per-database catalogs). State it engine-specifically and say so, or
abstract it and risk a non-Postgres implementation that silently differs?
Inherited unresolved from the v0.1 draft.
8. **Cell-based partitioning** — evaluate at consumer #4 rather than now?
Neither P nor E describes a cell.
## 19. Ratification path
The predecessor draft stalled because it was routed without one.
1. Reviewed by `tenant-engine`, `flex-auth`, `rapp-postgres`,
`railiance-platform`, and `adaptive-pricing` against §18.
2. Each of those repos publishes its own posture vector (§5) as part of
review. **The framework is validated by whether it can describe them
accurately** — if a repo cannot express itself in these four ladders, the
ladders are wrong and this document changes, not the repo.
3. On acceptance, **supersedes** the routing of
`rapp-postgres/docs/canon-drafts/shared-platform-relational-storage_v0.1-draft.md`,
whose §§3–8 are absorbed here. That draft is then withdrawn rather than
left pending.
4. On acceptance, `rapp-postgres/docs/adr/ADR-0001` moves to `accepted` and is
annotated as the PostgreSQL implementation of the E and P ladders.
The publication renderer that lived in `tools/` moved to `policy-nexus`, which
owns publication.

265
research/2026-08-17-adr008-gap-research-erasure-and-noisy-neighbour.md Normal file
View file

@ -0,0 +1,265 @@
# Gap research — erasure mechanism (R4) and noisy-neighbour evidence
> Research artefact — 2026-08-17. Fifth digest supporting
> `canon/architecture/adr-008-multi-tenancy-model.md` (draft-3). Closes the two
> gaps the four plane digests left open: how `R4` (verified erasure) can be
> reached at all, and what a noisy-neighbour evidence artifact should actually
> assert.
> **Both findings change a level definition in draft-3.**
> Index: `2026-08-17-adr008-multi-tenancy-research-index.md`.
---
# Part 1 — Erasure: R4 needs a mechanism it does not currently have
## 1.1 The problem draft-3 left standing
ADR-008 §4.5 defines `R4` as "deletion proven complete across live data,
backups and derived copies within the published horizon", and `rapp-postgres`
ADR-0002 establishes that on shared substrate the horizon is the instance
maximum — every co-resident consumer's retention window, whichever is longest.
Read together, those say: **erasure completes by waiting.** A tenant exercising
a deletion right is told the data is gone in up to N days, where N is set by
somebody else's retention requirement. That is honest, and it is a weak answer.
It also makes `R4` unreachable in any useful sense, because "we waited" is not
a proof.
## 1.2 Crypto-shredding is the established mechanism
The industry answer to erasure under immutable or long-retention storage is to
make the data unreadable rather than absent.
> "Crypto-shredding for backups encrypts each subject (or tenant) with a unique
> key derived from the subject identifier, and on erasure, destroys the key.
> The backups remain but are cryptographically inaccessible."
> — [oneuptime — setting up crypto-shredding for GDPR right-to-erasure](https://oneuptime.com/blog/post/2026-02-17-how-to-set-up-crypto-shredding-for-gdpr-right-to-erasure-compliance-in-google-cloud/view)
It is aimed precisely at our situation: the named use cases are "append-only or
immutable storage (event sourcing, WORM, immutable backups) where physical
deletion is impossible"
([Granit — crypto-shredding: GDPR erasure without deleting rows](https://granit-fx.dev/blog/crypto-shredding-gdpr-erasure-without-deleting-rows/)).
**Regulatory standing, stated carefully because overclaiming here would be
worse than in any other section of this framework.** The EDPB has *not* formally
endorsed crypto-shredding as Article 17 erasure. Several data protection
authorities have accepted it where physical deletion would be manifestly
disproportionate effort under Article 17(1), and EDPB Guidelines 5/2019, the UK
ICO and the French CNIL are cited as recognising it **provided three conditions
hold**: strong encryption (AES-256), irreversible key destruction, and auditable
destruction
([pg_hardstorage — GDPR Art. 17 crypto-shred](https://github.com/cybertec-postgresql/pg_hardstorage/blob/main/docs/compliance/gdpr-art-17-crypto-shred.md),
[Opsio — right to erasure in cloud systems](https://opsiocloud.com/blogs/right-to-erasure-article-17-gdpr-deletion-cloud-systems/)).
The third condition is the one this estate is well placed for and the one most
implementations skip: **destruction must be auditable**. ADR-0002 already
requires every sweep to emit an evidence record. Key destruction is the same
shape.
## 1.3 The finding: shredding granularity is bounded by the key boundary
Crypto-shredding "requires per-entity key isolation: each entity instance gets
its own encryption key" (oneuptime, ibid.). That single requirement determines
what is available to us, and it is not uniform across the estate.
| What we want to shred | Key boundary needed | Available today? |
|---|---|---|
| A consumer's logical exports | Per-consumer key | **Close.** The offsite lane already encrypts with `age`, and OpenBao holds key material. This is a key-management change, not an architecture change. |
| A consumer's physical backup | Per-consumer key on the Barman object store | **No.** One WAL stream, one credential, one encryption context for the whole instance. Same constraint that forced the derived maximum in ADR-0002 §2. |
| A single tenant's rows inside a consumer | Per-tenant key, applied by the application before the data reaches storage | **No, and not the platform's to build.** Requires application-level per-tenant encryption. |
The third row is the architecturally interesting one. Per-tenant crypto-shred
means the consumer encrypts tenant data under a tenant-specific key before
writing it — which is an **enforcement-plane** property, not a retention one. It
also happens to be a strong one: data encrypted per tenant is unreadable across
the boundary even by a credential that can address the rows, which is E4-class
protection achieved without per-tenant substrate.
**This is a new coupling and it belongs in §3.2.** Draft-3 records two
couplings (E4 needs P3; R's horizon is bounded by P). There is a third:
**R4 at tenant granularity requires per-tenant keys, which is an E-plane
capability.** Reaching the top of the retention ladder is not a retention
project.
A reference implementation exists for the consumer-granularity case:
`pg_hardstorage` "implements crypto-shred by destroying the per-tenant KEK that
wraps every encrypted backup's DEK" (ibid.). Per the §14 adoption stance we
take the **structure** — envelope encryption with a per-consumer key-encryption
key wrapping each backup's data-encryption key, and destruction of the KEK as
the erasure act — and build it on our own OpenBao and `age` lane rather than
adopting the extension.
## 1.4 The tension nobody should paper over: erasure versus audit
`audit-core` exists to hold a tamper-evident trail. Crypto-shredding a tenant's
audit records destroys evidence, and ADR-0001 §2 deliberately built the role
model so that "a compromised runtime credential could not rewrite history at
the schema level". Erasure is a sanctioned way to do exactly that.
The literature treats this as a genuine design tension rather than a solved
problem ([Remote Reason — balancing auditability and privacy with crypto-shredding](https://remotereason.com/blog/balancing-auditability-and-privacy-with-crypto-shredding)).
The usual resolution is to separate the *fact* of an event (retained, not
personal) from its *personal payload* (encrypted per subject, shreddable), so
the trail survives erasure with its personal content removed.
**This is a question for `audit-core`, not an answer this framework should
supply.** Flagging it because a naive "R4 everywhere" target would silently
instruct the audit service to destroy its own evidence.
## 1.5 Proposed amendments to draft-3
1. **`R4` admits two routes, and must name which one a service uses:**
- *Horizon-elapsed* — deletion is complete once the published erasure
horizon has passed. Available to everyone, proves little, slow.
- *Key-destroyed* — data is rendered unreadable immediately by destroying the
key that decrypts it. Requires per-entity keys, AES-256-class encryption,
and an auditable destruction record.
2. **Add the third coupling to §3.2:** R4 at tenant granularity requires an
E-plane capability (per-tenant encryption). R4 at consumer granularity
requires per-consumer keys on the retained artifacts.
3. **State the regulatory position honestly** wherever R4 is described: DPAs
have accepted key destruction as erasure under conditions; the EDPB has not
formally endorsed it. A plan tier claiming "your data is deleted" on the
strength of crypto-shredding is making a defensible but not settled claim,
and §11.4's hard line should say so rather than implying R4 is a clean yes.
4. **Raise the audit tension** as an open question addressed to `audit-core`.
---
# Part 2 — Noisy neighbour: the evidence artifact asserts the wrong thing
## 2.1 The problem draft-3 left standing
ADR-008 §13 proposes a P1/P2 evidence artifact: "one consumer saturating its
connection or CPU allowance demonstrably **does not breach** another's."
That is not achievable, and the reference guidance says so in its first line of
solution text:
> "Sharing a single resource inherently carries the risk of noisy neighbor
> problems that you can't completely avoid."
> — [Azure Architecture Center — Noisy Neighbor antipattern](https://learn.microsoft.com/en-us/azure/architecture/antipatterns/noisy-neighbor/noisy-neighbor)
An artifact that can only ever fail, or that passes by being run gently enough,
is worse than no artifact — it is the overclaim §6 prohibits, dressed as
evidence.
## 2.2 There are two failure shapes, and we only guard against one
Azure distinguishes them explicitly:
- **One greedy tenant.** "At peak times, tenant A uses all of the system's
resources, which means that any requests that tenant B makes fail." (ibid.)
- **Aggregate of many modest tenants.** "The noisy neighbor problem also occurs
when each individual tenant consumes only a small portion of the system's
capacity. However, the combined resource usage of many tenants can result in
a peak in overall usage." (ibid.)
Our per-consumer governance — `connection_limit`, `statement_timeout`,
`idle_in_transaction_session_timeout`, applied at database scope per ADR-0001
§3 — addresses the first shape well. It does nothing about the second.
**And the second shape is exactly what §17 computes.** Seven consumers, each
politely within its 14-connection allowance, request 98 of 100 connections.
Every one of them is well-behaved; the instance still fails. The connection
ceiling in §17 is not merely a capacity number — **it is the aggregate
noisy-neighbour bound**, and framing it as a scaling statistic understates what
it is.
## 2.3 The detection signature we can actually build
Azure's service-side diagnostic is specific and unintuitive:
> "**Failures when performing an operation for a tenant:** Look for failures
> that occur when a tenant isn't consuming a large share of the system's
> resources. This pattern might indicate that the tenant is experiencing a
> noisy neighbor problem. Track resource consumption by tenant." (ibid.)
The signal is a failure for a *quiet* consumer. That requires per-consumer
resource attribution, and we already have the raw material: ADR-0001 stamps
`application_name` per role (`<consumer>/runtime`), PostgreSQL logs carry
database, user, application name and client address, and
`make postgres-ops-status` already reports sessions by application name.
What is missing is a **baseline**. "It's important to clearly understand your
normal baseline resource usage and to configure monitoring and alerts to detect
spikes" (ibid.). No baseline for `platform-pg` has been recorded. Without one,
a spike is indistinguishable from a Tuesday.
## 2.4 A normative requirement we are not meeting
> "Be transparent with clients about any throttling mechanisms or usage quotas
> that you enforce. It's important that they handle failed requests gracefully
> and aren't caught off guard by limitations." (ibid.)
`tenant-engine` was told its connection and timeout limits during provisioning,
which is good practice by accident rather than by rule. Nothing requires it. A
consumer discovering its `statement_timeout` by hitting it in production is the
failure this warns about.
**Proposed:** fold quota transparency into §10 (blast radius must be published)
— the same disclosure obligation, extended from what a leaked credential can
reach to what the platform will refuse to do for you.
## 2.5 Mitigations we have, and the ones we do not
| Azure-recommended control | Our status |
|---|---|
| Query time limits / max returnable records | **Have.** `statement_timeout`, per-role since the tenant-engine change. |
| Connection quotas per tenant | **Have.** Database-scoped, per ADR-0001 §3. |
| Monitor overall *and* per-tenant resource usage | **Partial.** Attribution exists; no baseline, no alerting. |
| Throttling / rate limiting | **None.** No request-rate control at the database boundary. |
| Quality-of-service prioritisation | **None.** All consumers are equal, including on the synchronous authorization path where `tenant-engine` now sits. |
| Run heavy non-time-sensitive work off-peak | **Implicit.** Backups at 02:15; nothing else scheduled. |
| Rebalance tenants across stamps | **Not possible.** One instance, no second cell (§19.8). |
The QoS absence is worth noting against a specific fact: `tenant-engine` chose
a 5s statement timeout because `flex-auth` calls it synchronously on the
authorization path. That is a latency-critical consumer sharing an instance
with `audit-core`, which is not. Nothing prioritises one over the other, and
the framework has no vocabulary for saying one consumer's latency matters more.
## 2.6 Proposed amendment to draft-3
Replace the §13 P1/P2 artifact with one that asserts something true:
> **Noisy-neighbour characterisation (P1/P2, adversarial, load-generated).**
> A recorded baseline of normal per-consumer resource usage; a load run in
> which one consumer saturates its declared allowance; evidence that (a) the
> governance controls bind — the greedy consumer is throttled at its declared
> limits rather than exceeding them, and (b) the measured degradation
> experienced by co-resident consumers is recorded and judged acceptable.
> Carries a review date and the aggregate headroom at time of measurement.
That is achievable, it is falsifiable, and it produces a number — degradation
under contention — that a plan tier promising performance would need anyway.
---
## 3. Summary of changes these two gaps force
| # | Change | Where |
|---|---|---|
| A | `R4` splits into *horizon-elapsed* and *key-destroyed* routes; a service must name which | §4.5 |
| B | Third coupling: tenant-granularity R4 requires an E-plane capability (per-tenant encryption) | §3.2 |
| C | R4's regulatory standing stated honestly — accepted by DPAs under conditions, not formally endorsed by the EDPB | §4.5, §11.4 |
| D | Erasure-versus-audit tension raised as an open question to `audit-core` | §19 |
| E | Noisy-neighbour artifact reworded from "does not breach" to characterisation with a recorded baseline and measured degradation | §13 |
| F | §17's connection ceiling reframed as the aggregate noisy-neighbour bound, not only a capacity figure | §17 |
| G | Quota transparency added as a disclosure obligation | §10 |
| H | No QoS vocabulary despite a latency-critical consumer sharing with a batch one | §19, new |
## Sources
- [oneuptime — How to set up crypto-shredding for GDPR right-to-erasure compliance](https://oneuptime.com/blog/post/2026-02-17-how-to-set-up-crypto-shredding-for-gdpr-right-to-erasure-compliance-in-google-cloud/view)
- [cybertec-postgresql/pg_hardstorage — GDPR Art. 17 crypto-shred](https://github.com/cybertec-postgresql/pg_hardstorage/blob/main/docs/compliance/gdpr-art-17-crypto-shred.md)
- [Granit — Crypto-shredding: GDPR erasure without deleting a single row](https://granit-fx.dev/blog/crypto-shredding-gdpr-erasure-without-deleting-rows/)
- [Granit — Crypto-shredding (dotnet/compliance)](https://granit-fx.dev/dotnet/compliance/crypto-shredding/)
- [Opsio — Right to erasure, Article 17 GDPR, deletion in cloud systems](https://opsiocloud.com/blogs/right-to-erasure-article-17-gdpr-deletion-cloud-systems/)
- [Remote Reason — Balancing auditability and privacy with crypto-shredding](https://remotereason.com/blog/balancing-auditability-and-privacy-with-crypto-shredding)
- [Azure Architecture Center — Noisy Neighbor antipattern](https://learn.microsoft.com/en-us/azure/architecture/antipatterns/noisy-neighbor/noisy-neighbor)
- [Neon — The noisy neighbor problem in multitenant architectures](https://neon.com/blog/noisy-neighbor-multitenant)
- [PingCAP — Playbook: stop noisy neighbors, multi-tenant MySQL at scale](https://www.pingcap.com/playbook-noisy-neighbor-multi-tenant-mysql/)
- [Causal inference for quantifying noisy neighbor effects in multi-tenant cloud environments (arXiv 2604.03145)](https://arxiv.org/pdf/2604.03145)
- [Azure Architecture Center — Throttling pattern](https://learn.microsoft.com/en-us/azure/architecture/patterns/throttling)
- [Azure Architecture Center — Rate Limiting pattern](https://learn.microsoft.com/en-us/azure/architecture/patterns/rate-limiting-pattern)

72
research/2026-08-17-adr008-multi-tenancy-research-index.md
View file

@ -55,7 +55,7 @@ Ranked by how badly they mislead.
| # | Finding | Effect |
|---|---|---|
| O1 | **pgTAP + pgrls** (67 RLS lint rules, semantic policy-diff for CI gating, pytest isolation plugin) exist off the shelf. `rapp-postgres` already runs a 15-probe harness of the same shape. | The E3 half of open question §18.3 is far cheaper than assumed. Splits the question into a tractable mechanical half and a human half. |
| O1 | **pgTAP + pgrls** exist off the shelf; `rapp-postgres` already runs a 15-probe harness of the same shape. pgrls **verified directly** 2026-08-17: MIT, 67 rules (54 security, incl. BYPASSRLS / LEAKPROOF / `SECURITY DEFINER` / cross-tenant leaks), 19 auto-fixable, Z3 isolation proofs, SAFE/BREAKING policy-diff for CI gating, pytest plugin, PG 15–17. **But it is a 26-star beta project** — adopt the approach with confidence, treat the dependency as a supply-chain decision. pgTAP is long-established and carries no such caveat. | The E3 half of open question §18.3 is far cheaper than assumed. Splits the question into a tractable mechanical half and a human half. |
| O2 | **OWASP Multi-Tenant Security Cheat Sheet** is the closest external baseline to our framework and is referenced nowhere in the estate. | Adopt as the named external baseline in ADR §15. |
| O3 | **OpenID AuthZEN Authorization API 1.0** reached Final Specification January 2026; Keycloak shipped experimental support May 2026. Demonstrated PDP interoperability across 5+ engines. | `flex-auth`'s bespoke `CheckRequest` is pre-standard. Propose an `A4` rung for standard-interface PDP delegation. |
| O4 | **Token versioning** as a middle rung between cached claim and live re-query. | Cheap global invalidation without a hot-path lookup. |
@ -101,7 +101,75 @@ digest:
7. Route G7 to `tenant-engine` separately. It is a critique of ratified canon,
not of this ADR, and folding it in would overreach.
## 7. Method and limits
## 7. Adoption stance — structure, not tooling (operator decision, 2026-08-17)
Research output is used as **design input**, not as a shopping list. The
operator's rule:
> Use the structure that supports, extends or challenges our design. Do not
> adopt the tooling unless it is an established industry standard with broad
> application. Otherwise build ground up, so we can optimize and refactor as we
> see fit.
Applied to this survey:
| Finding | Adopt? | How |
|---|---|---|
| OWASP Multi-Tenant Security Cheat Sheet | **Yes — baseline** | A published security baseline, not tooling. Cite as the external reference our ladders answer to. |
| OWASP API Security Top 10 (BOLA / API1) | **Yes — vocabulary** | Names our gap in terms a security reviewer already holds. |
| OpenID AuthZEN Authorization API 1.0 | **Yes — standard** | Final Specification from a standards body with demonstrated multi-vendor interop. Qualifies as broad and established. Propose as the `A4` rung. |
| Azure tenancy models / AWS SaaS Lens / cell architecture | **Yes — structure** | Taxonomies and patterns. Corroborate the spectrum reframe and the E↔P coupling. |
| PostgreSQL RLS mechanics (`FORCE`, `BYPASSRLS`, `SET LOCAL`, leakproof) | **Yes — facts** | Engine behaviour, not a vendor's product. |
| **pgrls** | **No — structure only** | Take its rule *taxonomy* (54 security rules over RLS enablement, BYPASSRLS, LEAKPROOF, `SECURITY DEFINER`, cross-tenant leak; the SAFE/BREAKING/REQUIRES_REVIEW/DANGEROUS policy-diff classification) as a checklist for probes we write ourselves. 26 stars and beta was already a supply-chain caveat; the stance settles it. |
| **pgTAP** | **No** | Established in the Postgres world but still tooling, and we already have the ground-up equivalent: `scripts/isolation-test.sh`, 15 probes, bash + psql, no dependency tree. Extend that harness to the tenant boundary. |
The practical consequence is small and good: `rapp-postgres` already owns a
probe harness that found four real defects in its own provisioning SQL. The E2
and E3 evidence artifacts become new probes in an existing tool, informed by
pgrls's taxonomy of what to look for, with no new dependency. The
policy-diff idea — classifying a policy change as safe or breaking before it
lands — is the genuinely novel structural contribution worth reimplementing.
## 8. Retention and erasure — designed, not just flagged
Gap G1 (tenant offboarding and deletion) has been taken up rather than left as
an open question. See `rapp-postgres/docs/adr/ADR-0002-data-retention-and-erasure.md`.
The shape, per operator direction: retention is declared per consumer as
`N` days defaulting to 30, `rapp-postgres` provides the abstract mechanism for
any dataset, and a consumer or its governance layer builds policy on top of
that interface.
The finding that emerged while designing it: **per-consumer physical retention
is not implementable.** CNPG has one WAL stream and one window, so the instance
retention is derived as the maximum across consumers and every consumer's
erasure horizon is that maximum. A consumer declaring 7 days beside one
declaring 90 gets 90. This is the retention analogue of ADR-0001 §5 — state the
coupling rather than imply an isolation that is not there — and it produces two
things the estate could not previously say:
- The **erasure horizon**: the interval between deleting data and it ceasing to
be recoverable from anything the platform holds. Deleted rows survive in
backups for the whole window; that is the difference between "deleted" and
"erased", and it had never been written down.
- **Retention as a placement trigger.** A consumer needing a shorter horizon
than the instance floor cannot have one at P1. It moves to a dedicated
cluster for a reason unrelated to performance — which is exactly why it needs
recording, since nobody looks for a retention argument when reviewing
placement.
Deletion splits mechanism from policy: the platform sweeps whole datasets and
records an opaque `deletionPolicyRef` it never interprets, so every deletion
traces to the policy that authorised it. Rows are not a dataset.
`consumerDatabase` is deliberately not sweepable.
**Open for draft-3:** whether this is a fifth plane (`R` — retention and
erasure, with its own ladder from "no policy" through "verified erasure within
a published horizon") or a lifecycle stage cutting across the existing four.
The interface is ladder-shaped, which argues for a plane — at the cost of the
framework's name.
## 9. Method and limits
Four search-and-read passes, one per plane, against vendor architecture
guidance (AWS Well-Architected SaaS Lens, Azure Architecture Center), standards

23
research/2026-08-17-adr008-plane-e-enforcement.md
View file

@ -143,10 +143,25 @@ expected.
cross-tenant isolation tests are the only reliable guard"
([Blair Jordan — testing RLS policies with pgTAP](https://blair-devmode.medium.com/testing-row-level-security-rls-policies-in-postgresql-with-pgtap-a-supabase-example-b435c1852602),
[MakerKit — database testing with pgTAP](https://makerkit.dev/docs/next-supabase-turbo/development/database-tests)).
- **pgrls** is a purpose-built static analyzer: 67 lint rules for tenant and
per-user row-scoping bugs, performance traps and hygiene, 20 auto-fixable, a
semantic policy-diff command for CI gating, and a pytest plugin for isolation
tests ([pgrls on GitHub](https://github.com/pgrls/pgrls)).
- **pgrls** is a purpose-built static analyzer, **verified directly 2026-08-17**
rather than taken from a search snippet
([pgrls on GitHub](https://github.com/pgrls/pgrls)). Confirmed: MIT licence;
67 rules across SEC (54), PERF (5), HYG (4) and VIEW (4); 19 auto-fixable;
Z3 SMT-solver verification of isolation properties; a policy-diff command
classifying changes SAFE / BREAKING / REQUIRES_REVIEW / DANGEROUS; a
`pgrls.testing` pytest plugin with role-switching and per-test transactions;
SARIF, JUnit XML and GitHub-annotation output; tested against PostgreSQL
15–17. Its SEC rules cover exactly the traps in §3 — BYPASSRLS, LEAKPROOF,
`SECURITY DEFINER`, cross-tenant leaks.
**Maturity caveat, stated because this digest recommends adoption.** It is
26 stars, 2 forks, 527 commits, self-described "Beta — actively maintained".
The engineering is serious (Z3 proofs, a precision corpus) but this is a
small project, not an established dependency. Adopt the *approach* with
confidence; treat the *tool* as a supply-chain decision requiring the estate's
normal SBOM and pinning discipline, and be prepared to reimplement its
handful of load-bearing rules if it goes unmaintained. pgTAP, by contrast, is
long-established and carries no such caveat.
The named common failures match ours exactly: "a forgotten role, a pool reusing
connections with the wrong tenant context, or an admin path bypassing RLS