coulomb/doc-store-pg
4
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
doc-store-pg/specs/ArchitectureBlueprint.md

1997 lines
44 KiB
Markdown
Raw Normal View History

Seeded intent research program and initial specs
2026-08-19 21:22:38 +02:00
# ArchitectureBlueprint.md
## Repository
`doc-store-pg`
## Status
Architecture blueprint / research baseline.
This document defines the initial technical architecture for `doc-store-pg`. It is intentionally a blueprint rather than a frozen implementation specification. The architecture should remain stable enough to support reproducible experimentation while leaving individual components replaceable.
The central design question is:
> Can document storage become a composable capability of a CloudNativePG/PostgreSQL platform while preserving MongoDB-compatible application surfaces, strong multitenancy choices, PostgreSQL-native access, and an evidence-driven path to long-term optimization?
---
## 1. Architectural Intent
`doc-store-pg` should provide a document-oriented data surface on top of PostgreSQL without turning PostgreSQL into an opaque implementation detail.
The architecture therefore preserves two first-class access paths:
1. **Document access** through MongoDB-compatible protocols and semantics.
2. **PostgreSQL access** through SQL and PostgreSQL-native tooling.
The project should prefer composition of upstream technologies over reimplementation.
The initial architecture is built around:
- Kubernetes as the runtime substrate
- CloudNativePG as the PostgreSQL lifecycle and HA layer
- PostgreSQL 18+ as the initial database baseline
- DocumentDB PostgreSQL extensions as the primary document engine candidate
- DocumentDB Gateway and FerretDB as interchangeable MongoDB-compatible gateway candidates
- PostgreSQL-native JSONB as a baseline and fallback document representation
- Kubernetes-native configuration and observability
- explicit multitenancy and tenant-placement policies
- reproducible benchmark and conformance workloads
The architecture is designed so that these choices can be challenged by research rather than becoming permanent assumptions.
---
## 2. Architecture Principles
### 2.1 PostgreSQL Remains Visible
Document functionality is added to PostgreSQL; it does not replace the PostgreSQL model.
Applications or services may intentionally use:
- MongoDB-compatible document APIs
- SQL
- JSON/JSONB queries
- BSON/document functions
- relational joins
- PostgreSQL transactions
- PostgreSQL extensions
This enables hybrid relational/document workloads to be treated as a core capability rather than an accidental escape hatch.
### 2.2 Composition Before Reimplementation
Existing open-source implementations should be evaluated before new equivalents are written.
In particular:
- BSON storage and document semantics should initially come from DocumentDB.
- MongoDB wire-protocol translation should initially come from DocumentDB Gateway or FerretDB.
- PostgreSQL HA, failover, replication, backup integration, and cluster lifecycle should remain CloudNativePG responsibilities.
`doc-store-pg` should only implement functionality itself where integration, compatibility, policy, observability, or research requirements cannot reasonably be satisfied upstream.
### 2.3 Gateways Are Replaceable
MongoDB compatibility is an interface, not a hard dependency on one gateway.
```text
MongoDB client
|
+--> DocumentDB Gateway --> PostgreSQL + DocumentDB
|
+--> FerretDB -----------> PostgreSQL + DocumentDB
```
Additional gateway implementations may later be evaluated using the same conformance and benchmark suites.
### 2.4 Tenant Placement Is Explicit
Multitenancy is not represented by a boolean flag.
A tenant should have an explicit placement/isolation model:
```text
POOLED
NAMESPACE
DATABASE
DEDICATED
```
Future versions may add distributed or federated placement modes.
### 2.5 Compatibility Is Measured
"MongoDB compatible" is not considered a binary architectural property.
Compatibility should be measured separately for:
- wire protocol
- CRUD operations
- query semantics
- aggregation semantics
- indexing
- drivers
- tools
- transactions
- sessions
- change streams
- errors and edge cases
- application-level behavior
### 2.6 Evidence Before Optimization
Architectural decisions that materially affect compatibility, tenancy, performance, or operations should be benchmarkable.
The system should make it possible to ask:
> Which configuration is better for this workload and why?
rather than:
> Which database do we generally prefer?
### 2.7 Upstream Escape Hatches Remain Available
DocStorePG should not create unnecessary lock-in to its own abstractions.
A deployment should remain understandable through normal:
- Kubernetes resources
- CloudNativePG resources
- PostgreSQL tools
- DocumentDB tooling
- MongoDB-compatible clients
The project should add useful abstraction, not hide the underlying systems.
---
# 3. System Context
```mermaid
flowchart TB
subgraph Clients
A[MongoDB-compatible applications]
B[PostgreSQL / SQL applications]
C[Operations and automation]
D[Benchmark and conformance clients]
end
subgraph DocStorePG["doc-store-pg capability"]
G[MongoDB-compatible Gateway]
P[PostgreSQL Access]
T[Tenant / Placement Policy]
O[Observability]
X[Benchmark & Conformance Hooks]
end
subgraph DataPlane["CloudNativePG Data Plane"]
CNPG[CloudNativePG Cluster]
PG[(PostgreSQL)]
DDB[DocumentDB Extensions]
JSONB[Native JSONB]
end
A --> G
B --> P
C --> T
D --> X
G --> PG
P --> PG
T --> CNPG
X --> G
X --> P
CNPG --> PG
PG --- DDB
PG --- JSONB
PG --> O
G --> O
```
The important architectural point is that **DocStorePG is not initially a database engine**.
It is a capability profile formed from:
```text
PostgreSQL
+ document engine
+ compatible gateway
+ CloudNativePG operations
+ tenancy model
+ policy
+ observability
+ conformance
+ benchmarks
```
---
# 4. Layer Model
The initial architecture is divided into seven layers.
```text
L7 Research & Evidence
benchmarks, conformance, experiments, results
L6 DocStorePG Control
profiles, tenancy policy, placement, configuration
L5 Application Interfaces
MongoDB wire protocol, PostgreSQL protocol, admin APIs
L4 Document Semantics
BSON, document CRUD, query, aggregation, indexing
L3 PostgreSQL Data Platform
SQL, JSONB, transactions, RLS, extensions
L2 CloudNativePG Operations
cluster lifecycle, HA, replication, backup hooks, services
L1 Kubernetes Infrastructure
scheduling, storage, networking, secrets, resources
```
A change at one layer should be measurable without requiring unrelated layers to change.
For example, replacing FerretDB with DocumentDB Gateway should not require changing the PostgreSQL cluster, tenant workload, or benchmark definitions.
---
# 5. Reference Runtime Architecture
## 5.1 Default Reference Deployment
```mermaid
flowchart LR
MC[Mongo Client]
SC[SQL Client]
subgraph GW["Gateway Tier"]
G1[DocumentDB Gateway]
end
subgraph K8S["Kubernetes"]
subgraph CNPG["CloudNativePG Cluster"]
RW[(Primary)]
R1[(Replica)]
R2[(Replica)]
end
end
MC -->|MongoDB wire protocol| G1
G1 -->|PostgreSQL operations| RW
SC -->|PostgreSQL protocol| RW
RW -->|streaming replication| R1
RW -->|streaming replication| R2
```
The default research deployment should use three PostgreSQL instances when testing HA behavior and may use a single PostgreSQL instance for local functional development.
The gateway tier should remain separately deployable from PostgreSQL so that:
- gateway resource consumption can be measured independently
- multiple gateway implementations can be compared
- gateway replicas can scale independently
- gateway failures can be distinguished from database failures
- protocol translation overhead can be isolated
---
## 5.2 PostgreSQL Baseline
The initial baseline should target PostgreSQL 18 or later.
This is particularly useful because CloudNativePG's ImageVolume extension mechanism uses PostgreSQL 18's `extension_control_path` support and permits compatible extension images to be mounted dynamically into cluster pods.
The implementation should pin exact versions in deployment profiles rather than using floating image tags.
Example conceptual lock information:
```yaml
platform:
postgresql: "18.x"
cloudnativepg: "<pinned>"
kubernetes: "<supported-version>"
extensions:
documentdb:
version: "<pinned>"
image: "<immutable-image-reference>"
gateway:
implementation: documentdb
version: "<pinned>"
```
The exact versions belong in machine-readable deployment profiles, not permanently in this blueprint.
---
# 6. Extension Architecture
## 6.1 Primary Document Engine
DocumentDB is the initial primary candidate because its architecture already separates document functionality into PostgreSQL-oriented components:
```text
pg_documentdb_core
|
+-- BSON storage and primitives
pg_documentdb
|
+-- document API
+-- CRUD
+-- queries
+-- indexes
+-- aggregation functionality
pg_documentdb_gw
|
+-- MongoDB wire protocol
+-- authentication/session gateway behavior
```
DocStorePG should treat these as upstream capabilities.
## 6.2 Extension Packaging
Preferred path:
```text
OCI extension image
|
v
CloudNativePG ImageVolume
|
v
PostgreSQL pod
|
v
CREATE/UPDATE EXTENSION
```
This should be preferred over building permanently customized PostgreSQL images where the required extension can be safely managed as an image-volume extension.
A custom PostgreSQL image remains an allowed fallback for:
- unsupported extension packaging
- experimental builds
- debugging
- compatibility tests
- extension combinations that cannot yet use ImageVolumes
## 6.3 Extension Lifecycle
Extension lifecycle must be treated separately from application schema lifecycle.
Required states:
```text
available
installed
configured
upgradable
upgraded
rollback-tested
```
The benchmark program should eventually include:
- extension install time
- upgrade behavior
- PostgreSQL major-upgrade interaction
- replica behavior
- recovery behavior
- incompatible-extension failure behavior
---
# 7. Gateway Architecture
## 7.1 Gateway Contract
A gateway accepts MongoDB-compatible client traffic and maps it to the PostgreSQL-backed document engine.
The gateway layer must be independently observable.
Required gateway metrics should eventually include:
- request count
- command type
- response latency
- upstream PostgreSQL latency
- translation latency where measurable
- active connections
- connection establishment rate
- rejected connections
- authentication failures
- protocol errors
- unsupported commands
- PostgreSQL errors
- memory
- CPU
- network throughput
## 7.2 Candidate A — DocumentDB Gateway
DocumentDB Gateway is the reference path closest to the DocumentDB implementation itself.
Use it as the initial compatibility baseline.
## 7.3 Candidate B — FerretDB
FerretDB 2.x uses PostgreSQL with the DocumentDB extension as its backend and translates MongoDB wire-protocol operations for clients.
It should be preserved as an alternative gateway because it permits a controlled experiment:
```text
same client
same workload
same PostgreSQL
same DocumentDB engine
different gateway
```
This isolates gateway behavior from storage-engine behavior.
## 7.4 Gateway Selection Profile
Conceptual configuration:
```yaml
gateway:
enabled: true
implementation: documentdb
replicas: 2
resources:
profile: standard
service:
exposure: cluster-internal
observability:
enabled: true
```
Alternative:
```yaml
gateway:
implementation: ferretdb
```
The actual configuration mechanism may initially be Helm/Kustomize values rather than a custom CRD.
---
# 8. Native PostgreSQL Surface
MongoDB compatibility should never be the only supported path.
DocStorePG should deliberately preserve a native SQL surface.
```mermaid
flowchart TD
DOC[(Document data)]
M[MongoDB-compatible API]
S[SQL]
H[Hybrid application]
M --> DOC
S --> DOC
H --> M
H --> S
```
This enables research into workloads such as:
```sql
SELECT
c.customer_id,
d.document
FROM customers c
JOIN device_documents d
ON d.customer_id = c.customer_id
WHERE c.contract_status = 'active'
AND document_matches_capability(d.document, 'cooling');
```
The precise document functions may differ, but the architectural requirement is that document data remain usable from SQL where the upstream engine supports it.
This hybrid access mode is one of the central differentiators to test.
---
# 9. Baseline Storage Profiles
DocStorePG should initially define stable comparable storage profiles.
## P0 — Native PostgreSQL JSONB
```text
PostgreSQL
+ JSONB
+ native PostgreSQL indexes
+ no Mongo gateway
```
Purpose:
- establish the minimum PostgreSQL document baseline
- measure what PostgreSQL already provides without a compatibility layer
## P1 — DocumentDB / Native PostgreSQL Interface
```text
PostgreSQL
+ DocumentDB extensions
+ direct PostgreSQL access
```
Purpose:
- isolate DocumentDB storage and document semantics
- measure document-engine overhead without Mongo wire translation
## P2 — DocumentDB Gateway
```text
Mongo client
|
DocumentDB Gateway
|
PostgreSQL + DocumentDB
```
Purpose:
- primary MongoDB compatibility candidate
## P3 — FerretDB Gateway
```text
Mongo client
|
FerretDB
|
PostgreSQL + DocumentDB
```
Purpose:
- alternative compatibility path
- gateway comparison
## M0 — Native MongoDB Reference
```text
Mongo client
|
MongoDB
```
Purpose:
- external reference implementation
- semantic compatibility oracle where appropriate
- performance and operational comparator
These profile identifiers should remain stable so benchmark results can be compared over time.
---
# 10. Multitenancy Architecture
Multitenancy is a first-class architecture dimension.
The initial tenant-placement ladder is:
```text
T0 POOLED
T1 NAMESPACE
T2 DATABASE
T3 DEDICATED
```
These identifiers should remain stable in benchmark datasets.
## 10.1 T0 — Pooled
Multiple tenants share document structures.
```text
PostgreSQL database
|
+-- shared document collection/table
|
+-- tenant_id = A
+-- tenant_id = B
+-- tenant_id = C
```
Required design goal:
> Tenant identity should be enforced as deeply as the selected interface permits.
For PostgreSQL-native access, this should include evaluating Row-Level Security.
Illustrative pattern:
```sql
ALTER TABLE documents ENABLE ROW LEVEL SECURITY;
CREATE POLICY tenant_isolation
ON documents
USING (
tenant_id = current_setting('docstorepg.tenant_id')::uuid
);
```
### Research questions
- Can tenant identity propagate safely through a MongoDB-compatible gateway?
- Does the document engine expose sufficient hooks for RLS-based enforcement?
- Can a gateway accidentally bypass RLS through privileged database roles?
- How can connection pooling safely preserve tenant context?
- What performance cost does RLS introduce?
- Can tenant-aware compound indexes prevent cross-tenant hot spots?
- How do backup and restore work for a single pooled tenant?
## 10.2 T1 — Namespace
Tenants receive PostgreSQL schemas or another logically separate namespace.
```text
cluster
|
+-- database
|
+-- tenant_a schema
+-- tenant_b schema
+-- tenant_c schema
```
Advantages to evaluate:
- stronger logical separation than pooled rows
- separate objects and indexes
- tenant-specific schema customization
- relatively low compute overhead compared with dedicated clusters
Risks to evaluate:
- object proliferation
- PostgreSQL catalog growth
- migration complexity
- gateway mapping complexity
- uncertain fit with document-engine assumptions
- backup/restore granularity
T1 must be treated as an experiment until compatibility with the selected document engine and gateway is proven.
## 10.3 T2 — Database
Each tenant receives a separate PostgreSQL database inside one CloudNativePG cluster.
```text
CloudNativePG cluster
|
+-- tenant_a database
+-- tenant_b database
+-- tenant_c database
```
CloudNativePG can declaratively manage additional databases and their extensions/schemas.
This mode should evaluate:
- per-tenant ownership
- per-database extensions
- migration lifecycle
- database count scaling
- connection overhead
- catalog/resource overhead
- backup/restore behavior
- gateway routing
- tenant-specific configuration
- noisy-neighbour effects
The compute and storage failure domain remains shared.
## 10.4 T3 — Dedicated
Each tenant receives a dedicated CloudNativePG-backed DocStorePG deployment.
```mermaid
flowchart LR
A[Tenant A] --> CA[DocStorePG Cluster A]
B[Tenant B] --> CB[DocStorePG Cluster B]
C[Tenant C] --> CC[DocStorePG Cluster C]
```
This mode provides the strongest initial isolation boundary.
Potential isolation includes:
- PostgreSQL process
- PostgreSQL database
- gateway
- CPU request/limit
- memory request/limit
- persistent storage
- Kubernetes scheduling
- secrets
- service identity
- network policy
- backup policy
- failure domain
The primary downside is fleet overhead.
T3 is expected to be most relevant to:
- large enterprise tenants
- regulated workloads
- tenants requiring special extensions
- tenants requiring independent maintenance windows
- high-volume "whale" tenants
- contractual isolation requirements
---
# 11. Tenant Placement as a Lifecycle
The important abstraction is not merely selecting one of T0–T3.
The longer-term architecture should explore tenant mobility:
```mermaid
stateDiagram-v2
[*] --> Pooled
Pooled --> Namespace
Pooled --> Database
Namespace --> Database
Database --> Dedicated
Pooled --> Dedicated
Dedicated --> Database
Database --> Pooled
```
Not all transitions need to be supported initially.
The strategic goal is:
> Keep the application-facing document contract as stable as possible while tenant placement changes underneath it.
This is DocStorePG's initial conceptual answer to some of the problems that MongoDB solves with shard placement.
It is not initially intended to reproduce MongoDB sharding.
---
# 12. Explicit Sharding Boundary
MongoDB provides native horizontal sharding in which data can be distributed across shards and routed through a unified cluster interface.
CloudNativePG's normal architecture is a PostgreSQL primary/standby cluster. It is not by itself a transparent horizontally sharded write architecture.
Therefore:
> DocStorePG V1 must not claim MongoDB-equivalent horizontal sharding.
The initial architecture deliberately stops here:
```text
single DocStorePG deployment
|
+-- primary
+-- replicas
```
and permits multiple independent placements:
```text
DocStorePG A
DocStorePG B
DocStorePG C
```
A future research track may investigate a placement/router layer:
```mermaid
flowchart TD
APP[Application]
R[Tenant Router]
A[DocStorePG Placement A]
B[DocStorePG Placement B]
C[DocStorePG Placement C]
APP --> R
R --> A
R --> B
R --> C
```
That work should only begin once there is evidence that it solves a problem not already better solved by:
- MongoDB sharding
- Citus or another PostgreSQL distribution technology
- application-level partitioning
- independent tenant clusters
- another existing distributed database
---
# 13. Proposed DocStorePG Control Surface
DocStorePG should not immediately create a Kubernetes operator.
The first implementation should prefer:
```text
Helm / Kustomize
+ CloudNativePG CRDs
+ standard Kubernetes resources
+ benchmark profiles
+ explicit configuration files
```
Only after recurring orchestration requirements emerge should a DocStorePG CRD/operator be justified.
A possible future API is documented here to establish architectural vocabulary.
## 13.1 Future `DocStore` Resource
```yaml
apiVersion: docstorepg.io/v1alpha1
kind: DocStore
metadata:
name: application-documents
spec:
postgres:
profile: ha-standard
documentEngine:
implementation: documentdb
gateway:
implementation: documentdb
replicas: 2
interfaces:
mongodb: true
postgresql: true
tenancy:
defaultPlacement: pooled
observability:
enabled: true
```
## 13.2 Future `DocStoreTenant` Resource
```yaml
apiVersion: docstorepg.io/v1alpha1
kind: DocStoreTenant
metadata:
name: acme
spec:
storeRef: application-documents
placement:
mode: database
resources:
class: standard
compatibility:
profile: mongodb-default
```
These resources are **design sketches**, not implementation commitments.
---
# 14. Configuration Profiles
Configuration should be grouped into composable profiles.
```text
profiles/
├── platform/
│ ├── local-single.yaml
│ ├── ha-small.yaml
│ └── ha-standard.yaml
├── engine/
│ ├── jsonb.yaml
│ └── documentdb.yaml
├── gateway/
│ ├── none.yaml
│ ├── documentdb.yaml
│ └── ferretdb.yaml
├── tenancy/
│ ├── pooled.yaml
│ ├── namespace.yaml
│ ├── database.yaml
│ └── dedicated.yaml
└── workload/
├── document-crud.yaml
├── aggregation.yaml
└── hybrid.yaml
```
A benchmark environment can then be expressed as composition:
```text
platform/ha-small
+ engine/documentdb
+ gateway/ferretdb
+ tenancy/pooled
+ workload/document-crud
```
This makes experiments easier to reproduce and diff.
---
# 15. Security Architecture
The security model should be layered.
```text
Kubernetes identity and RBAC
|
NetworkPolicy / service exposure
|
gateway authentication
|
PostgreSQL authentication
|
PostgreSQL role / database / schema privileges
|
Row-Level Security where applicable
|
document/application authorization
```
## 15.1 Security Rules
1. Gateway database credentials should not use PostgreSQL superuser privileges in production profiles.
2. `BYPASSRLS` roles must not be used for pooled tenant traffic where RLS is part of the tenant boundary.
3. Tenant context must be explicit and auditable.
4. Connection pooling must not leak tenant context between sessions.
5. PostgreSQL direct access must not silently bypass the intended Mongo-facing security model.
6. Mongo-compatible and SQL-facing identities must be mapped deliberately.
7. Secrets should be provided through Kubernetes-native secret mechanisms or external secret-management integration.
8. External service exposure should be opt-in.
9. NetworkPolicy should be part of production reference profiles.
10. Security tests belong in the conformance suite.
---
# 16. Identity Propagation
Tenant-aware pooled operation creates an important architecture problem:
```text
MongoDB client identity
|
v
Mongo gateway
|
v
PostgreSQL session identity
|
v
RLS / database policy
```
DocStorePG should explicitly test multiple strategies.
### Strategy A — Database Role Mapping
One PostgreSQL role per tenant or tenant class.
Potential strength:
- database-native identity
Potential weakness:
- large role counts
- connection-pool complexity
### Strategy B — Session Tenant Context
A trusted gateway sets a PostgreSQL session variable such as:
```sql
SET docstorepg.tenant_id = '...';
```
RLS reads that context.
Potential strength:
- scalable logical identity
Potential weakness:
- gateway becomes security-critical
- session reset must be flawless
### Strategy C — Database-per-Tenant Routing
Gateway selects the tenant database.
Potential strength:
- stronger logical boundary
Potential weakness:
- routing and connection-pool fragmentation
No strategy should be declared canonical until tested.
---
# 17. Network Architecture
Default production stance:
```text
Mongo endpoint: cluster-internal
PostgreSQL endpoint: cluster-internal
admin endpoints: cluster-internal
metrics endpoints: monitoring namespace / authorized collectors
external exposure: explicit profile only
```
Conceptual Kubernetes services:
```text
docstore-mongo-rw
docstore-postgres-rw
docstore-postgres-ro
docstore-metrics
```
The exact service structure should follow the selected upstream gateway and CloudNativePG conventions.
---
# 18. High Availability
CloudNativePG should own PostgreSQL HA.
```mermaid
flowchart LR
GW[Gateway replicas]
SVC[CNPG RW Service]
P[(Primary)]
R1[(Replica)]
R2[(Replica)]
GW --> SVC
SVC --> P
P --> R1
P --> R2
```
DocStorePG should test gateway behavior during PostgreSQL failover.
Important questions:
- Does the gateway reconnect automatically?
- What do clients observe?
- Which operations fail?
- Are retries safe?
- How do transactions behave?
- How quickly does service recover?
- Does MongoDB client retry behavior interact correctly with PostgreSQL failover?
- Does the gateway retain stale connections?
Failover testing belongs in the benchmark program.
---
# 19. Read Scaling
CloudNativePG provides read-only service patterns through replicas.
DocStorePG must not assume MongoDB-style read preferences map automatically to PostgreSQL read replicas.
Possible future mapping:
```text
Mongo read preference
|
v
Gateway policy
|
+--> primary
|
+--> CNPG read-only service
```
This requires explicit semantic research around:
- consistency
- replication lag
- transaction semantics
- session guarantees
- MongoDB driver expectations
- failover
Until proven, primary routing should be the correctness baseline.
---
# 20. Backup and Recovery
Backup architecture must distinguish between:
```text
cluster recovery
database recovery
tenant recovery
collection recovery
document recovery
```
CloudNativePG provides cluster-oriented PostgreSQL backup/recovery mechanisms; the DocumentDB Kubernetes Operator also has its own evolving backup abstractions.
DocStorePG should not assume that cluster-level backup automatically satisfies tenant-level recovery requirements.
Research must measure:
- backup duration
- backup size
- restore duration
- point-in-time recovery
- tenant-specific restore
- restoring one tenant from a pooled dataset
- restoring database-per-tenant
- dedicated-cluster restore
- Mongo-compatible application recovery behavior
Tenant recoverability is an important factor in choosing T0–T3.
---
# 21. Observability Architecture
Observability should expose each layer independently.
```mermaid
flowchart LR
C[Client]
G[Gateway]
P[PostgreSQL]
K[Kubernetes]
B[Benchmark Harness]
C --> G --> P
G --> M[Metrics]
P --> M
K --> M
B --> M
G --> L[Logs]
P --> L
K --> L
M --> O[Observability Backend]
L --> O
```
Minimum metric categories:
### PostgreSQL
- transactions
- query latency
- query count
- locks
- connections
- cache hit behavior
- WAL generation
- checkpoint behavior
- replication lag
- table/index size
- I/O
- CPU
- memory
### Gateway
- operations
- command types
- latency
- failures
- unsupported requests
- active connections
- authentication
- CPU
- memory
### Kubernetes
- pod restarts
- scheduling
- CPU requests/usage
- memory requests/usage
- storage
- network
- failovers
- PVC behavior
### Tenant
Where possible:
- requests per tenant
- storage per tenant
- latency per tenant
- error rate per tenant
- resource share
- noisy-neighbour indicators
### Benchmark
Every benchmark run must capture enough environment metadata to explain the result.
---
# 22. Benchmark and Conformance Plane
The benchmark plane is part of the architecture, not an external one-off tool.
```mermaid
flowchart TB
W[Workload Catalog]
R[Runner]
M0[MongoDB Reference]
P0[PostgreSQL JSONB]
P1[DocumentDB Native]
P2[DocumentDB Gateway]
P3[FerretDB]
C[Comparator]
E[(Evidence Store)]
W --> R
R --> M0
R --> P0
R --> P1
R --> P2
R --> P3
M0 --> C
P0 --> C
P1 --> C
P2 --> C
P3 --> C
C --> E
```
Benchmark code should support deterministic seeds where possible.
Every result should include:
```yaml
run:
id: ...
timestamp: ...
gitCommit: ...
workload: ...
seed: ...
environment:
kubernetes: ...
nodes: ...
cpu: ...
memory: ...
storageClass: ...
software:
mongodb: ...
postgresql: ...
cloudnativepg: ...
documentdb: ...
gateway: ...
profile:
storage: P2
tenancy: T0
dataset:
tenants: ...
documents: ...
averageDocumentSize: ...
result:
throughput: ...
latencyP50: ...
latencyP95: ...
latencyP99: ...
errors: ...
```
This structure should evolve into a formal benchmark schema.
---
# 23. Differential Conformance Testing
For compatible operations, DocStorePG should support differential testing.
```text
request
|
+------> MongoDB ----------> result A
|
+------> DocStorePG -------> result B
|
v
semantic comparator
```
Comparison dimensions:
- returned data
- BSON types
- null/missing behavior
- ordering
- update results
- aggregation results
- errors
- error categories/codes
- transaction outcome
- session behavior
- index behavior
Differences should be classified as:
```text
IDENTICAL
EQUIVALENT
DOCUMENTED_DIFFERENCE
UNSUPPORTED
INCORRECT
UNKNOWN
```
A compatibility score must not hide important individual differences.
---
# 24. Workload Classes
The architecture should support at least these benchmark workload classes.
## W0 — Basic CRUD
- insert
- find by ID
- update
- delete
- bulk operations
## W1 — Flexible Schema
- optional fields
- heterogeneous structures
- schema evolution
- large/small documents
## W2 — Nested Documents
- nested objects
- arrays
- array element updates
- nested predicates
## W3 — Indexing
- single-field
- compound
- nested
- wildcard/dynamic-field equivalents
- low/high cardinality
- many-index write cost
## W4 — Aggregation
- filtering
- projection
- grouping
- sorting
- unwind
- lookup/join-like behavior
- facets
## W5 — Transactions
- single-document
- multi-document
- concurrent updates
- abort/retry behavior
## W6 — Change/Event Workloads
- change streams or equivalents
- consumer lag
- reconnect behavior
- failover behavior
## W7 — Hybrid Relational/Document
- relational filters over document sets
- joins between relational and document data
- cross-model transactions
- reporting/analytics
## W8 — Multitenancy
- many small tenants
- mixed tenant sizes
- tenant provisioning
- whale tenant
- noisy neighbours
- placement changes
- isolation tests
## W9 — Operations
- failover
- backup
- restore
- upgrade
- extension upgrade
- scaling
- disaster simulation
---
# 25. Whale-Tenant Experiment
One canonical multitenancy experiment should be designed into the architecture from the start.
Example distribution:
```text
1000 tenants total
950 small tenants
49 medium tenants
1 whale tenant consuming roughly half of traffic
```
The exact distribution should be parameterized.
Questions:
- What happens to small-tenant p99 latency?
- Can the whale be identified operationally?
- Can the whale move from T0 to T2/T3?
- How long does migration take?
- How much downtime occurs?
- Does the application connection contract change?
- What happens to backup cost?
- At what point does dedicated placement become economically sensible?
This should eventually become a signature DocStorePG benchmark.
---
# 26. Tenant Migration Architecture
Tenant migration is initially an experimental subsystem.
```mermaid
flowchart LR
S[Source Placement]
E[Export / Replication]
V[Validate]
C[Cutover]
T[Target Placement]
R[Rollback Window]
S --> E --> V --> C --> T
C --> R
```
Desired migration properties:
- tenant-scoped
- auditable
- resumable
- consistency-aware
- verifiable
- rollback-aware
- application-transparent where possible
Potential mechanisms to investigate:
- logical export/import
- PostgreSQL logical replication
- document API replication
- change streams
- dual writes
- maintenance-window cutover
- backup/restore extraction
No mechanism is canonical yet.
---
# 27. DocumentDB Kubernetes Operator Relationship
The DocumentDB Kubernetes Operator is a relevant upstream project and should be actively compared with DocStorePG.
It already uses CloudNativePG underneath and provides Kubernetes-native DocumentDB lifecycle capabilities.
DocStorePG should therefore avoid duplicating its implementation without evidence.
Initial relationship:
```text
DocumentDB Operator
|
+-- upstream reference implementation
+-- optional deployment profile
+-- research comparator
+-- source of operational lessons
DocStorePG
|
+-- broader PostgreSQL document capability research
+-- interchangeable gateways
+-- native JSONB baseline
+-- multitenancy comparison
+-- MongoDB comparison
+-- conformance framework
+-- benchmark framework
+-- hybrid relational/document research
```
If the DocumentDB Operator eventually provides the best runtime deployment path, DocStorePG should be able to adopt it rather than compete with it.
---
# 28. Failure Domains
DocStorePG should model failure domains explicitly.
| Failure | Likely scope | Expected owner |
|---|---|---|
| Gateway pod | Mongo API traffic through pod | Kubernetes / gateway |
| Gateway deployment | Mongo-compatible surface | DocStorePG deployment |
| PostgreSQL instance | instance | CloudNativePG |
| PostgreSQL primary | writes until failover | CloudNativePG |
| PostgreSQL cluster | tenant set on cluster | operations |
| PVC/storage | instance or cluster | Kubernetes/storage |
| Namespace | contained deployments | Kubernetes |
| Kubernetes node | scheduled workloads | Kubernetes |
| Kubernetes cluster | all contained placements | infrastructure |
| Tenant credential leak | tenant/security boundary | security model |
| RLS policy error | pooled tenants | DocStorePG policy |
| Extension defect | databases using extension | upstream + operations |
Dedicated tenant placement should reduce blast radius at the cost of operational density.
---
# 29. Scaling Model
DocStorePG should distinguish four fundamentally different scaling mechanisms.
## Vertical
Increase:
- CPU
- memory
- storage performance
## Read Scale
Add PostgreSQL replicas where semantics permit.
## Gateway Scale
Increase stateless gateway replicas.
## Placement Scale
Move tenants or workloads onto additional independent DocStorePG deployments.
```text
scale
|
+-----------------+----------------+
| | |
vertical gateway placement
|
+--------+--------+
| |
shared dedicated
```
Transparent horizontal write sharding is explicitly outside the initial model.
---
# 30. Repository Architecture
Proposed repository structure:
```text
doc-store-pg/
├── INTENT.md
├── ArchitectureBlueprint.md
├── ResearchProgram.md
├── BenchmarkSpecification.md
├── CompatibilityModel.md
├── MultitenancyModel.md
├── WorkloadCatalog.md
├── README.md
│
├── docs/
│ ├── architecture/
│ ├── decisions/
│ ├── findings/
│ └── references/
│
├── deploy/
│ ├── cnpg/
│ ├── documentdb/
│ ├── gateways/
│ │ ├── documentdb/
│ │ └── ferretdb/
│ └── mongodb-reference/
│
├── profiles/
│ ├── platform/
│ ├── engine/
│ ├── gateway/
│ ├── tenancy/
│ └── workload/
│
├── experiments/
│ ├── compatibility/
│ ├── performance/
│ ├── multitenancy/
│ ├── operations/
│ └── migration/
│
├── benchmarks/
│ ├── runner/
│ ├── workloads/
│ ├── schemas/
│ └── analysis/
│
├── tests/
│ ├── conformance/
│ ├── security/
│ ├── failover/
│ └── integration/
│
├── results/
│ └── README.md
│
└── scripts/
```
Generated benchmark results should be governed carefully so the repository does not become filled with large ephemeral datasets.
Machine-readable summaries belong in Git where useful; raw large datasets may require external object storage with manifests/checksums committed to the repository.
---
# 31. Initial Implementation Stages
## Stage A — Functional Baseline
Establish:
```text
Kubernetes
+ CloudNativePG
+ PostgreSQL
```
Verify normal SQL operation.
## Stage B — Document Engine
Add:
```text
DocumentDB extensions
```
Verify document operations through PostgreSQL-native interfaces.
## Stage C — Mongo-Compatible Surface
Add:
```text
DocumentDB Gateway
```
Run basic official-driver CRUD tests.
## Stage D — Alternative Gateway
Add:
```text
FerretDB
```
Run the same tests against the same backend profile.
## Stage E — MongoDB Reference
Deploy native MongoDB as M0.
Run differential CRUD/query tests.
## Stage F — Tenant Baseline
Implement T0 pooled tenancy and evaluate PostgreSQL-enforced isolation.
## Stage G — Isolation Ladder
Add T2 database-per-tenant and T3 dedicated placement.
Treat T1 schema-per-tenant as experimental and implement only if document-engine behavior supports it cleanly.
## Stage H — Benchmark Plane
Add machine-readable benchmark runs and environment manifests.
## Stage I — Operational Experiments
Test:
- failover
- restart
- backup
- restore
- upgrade
- noisy neighbour
- tenant movement
---
# 32. Initial Architecture Decisions
The following decisions are provisional but form the starting baseline.
### AD-001 — CloudNativePG is the PostgreSQL operator
**Decision:** Use CloudNativePG rather than introducing a custom PostgreSQL lifecycle implementation.
**Reason:** PostgreSQL lifecycle and HA are not DocStorePG's differentiated problem.
### AD-002 — DocumentDB is the initial document engine
**Decision:** Use DocumentDB extensions as the first BSON/document compatibility implementation.
**Reason:** It already implements document semantics on PostgreSQL and exposes a MongoDB-compatible gateway architecture.
### AD-003 — Gateway implementation remains swappable
**Decision:** Support DocumentDB Gateway first; preserve FerretDB as an alternative profile.
**Reason:** This allows protocol-layer effects to be isolated experimentally.
### AD-004 — PostgreSQL access remains first-class
**Decision:** Do not hide SQL/PostgreSQL interfaces behind the document API.
**Reason:** Hybrid relational/document capability is a core research hypothesis.
### AD-005 — Multitenancy has explicit placement levels
**Decision:** Use T0–T3 as stable architectural vocabulary.
**Reason:** Isolation, cost, and scaling cannot be meaningfully represented by a single "multi-tenant" mode.
### AD-006 — No custom DocStorePG operator initially
**Decision:** Start with manifests/profiles and existing operators.
**Reason:** Operator behavior should only be built once recurring control-plane requirements are empirically understood.
### AD-007 — No claim of transparent write sharding
**Decision:** Treat MongoDB sharding as a distinct comparison point rather than something DocStorePG already provides.
**Reason:** CloudNativePG HA and MongoDB sharding solve different problems.
### AD-008 — Benchmarks are architecture
**Decision:** Reproducible benchmark and conformance hooks are part of the system design from the beginning.
**Reason:** Long-term architecture choices must remain evidence-driven.
---
# 33. Architecture Questions to Resolve Through Research
The blueprint intentionally leaves these open:
1. How complete is DocumentDB's real MongoDB semantic compatibility?
2. Does DocumentDB Gateway or FerretDB provide the better compatibility/performance trade-off?
3. How should Mongo identities map to PostgreSQL identities?
4. Can PostgreSQL RLS reliably enforce pooled tenant isolation through the Mongo-compatible surface?
5. What tenant count is practical for database-per-tenant within one CNPG cluster?
6. Is schema-per-tenant useful with DocumentDB or merely theoretical?
7. How much gateway overhead exists relative to direct document access?
8. How much document-engine overhead exists relative to native JSONB?
9. Where does native MongoDB outperform the PostgreSQL-backed architecture?
10. How do both architectures behave with a whale tenant?
11. What is the operational cost per tenant under T0, T2, and T3?
12. How should tenant migration work?
13. Can a stable logical endpoint survive tenant placement changes?
14. Which MongoDB read/write/session semantics can safely map to CNPG primary/replica topology?
15. What backup granularity is achievable for each tenant model?
16. How should extension versions be upgraded independently from PostgreSQL?
17. When does an existing distributed PostgreSQL technology become preferable to DocStorePG placement routing?
18. When should a workload simply remain on native MongoDB?
These questions should seed `ResearchProgram.md`.
---
# 34. Target Architecture
The long-term target, if supported by evidence, is:
```mermaid
flowchart TB
subgraph Applications
M[MongoDB clients]
S[SQL clients]
end
EP[Stable DocStorePG Endpoint]
TC[Tenant / Placement Control]
subgraph Shared["Shared Placement"]
SP[CNPG + DocumentDB]
end
subgraph DB["Database-Isolated Placement"]
DP[CNPG + Tenant Databases]
end
subgraph Dedicated["Dedicated Placements"]
D1[CNPG Tenant A]
D2[CNPG Tenant B]
end
OBS[Observability]
BENCH[Conformance & Benchmark Plane]
M --> EP
S --> EP
EP --> TC
TC --> SP
TC --> DP
TC --> D1
TC --> D2
SP --> OBS
DP --> OBS
D1 --> OBS
D2 --> OBS
BENCH --> EP
BENCH --> OBS
```
The important feature is not that all boxes necessarily become custom DocStorePG software.
The important feature is that the platform can express and test:
```text
interface
+ document semantics
+ tenant identity
+ placement
+ isolation
+ PostgreSQL capability
+ operations
+ evidence
```
as separable concerns.
---
# 35. Architectural North Star
`doc-store-pg` should aim to make this statement testable:
> A document store can be an operationally native PostgreSQL capability whose API, compatibility, tenant placement, and isolation level can evolve independently from the applications that use it.
The architecture succeeds if it allows the project to discover where that statement is true, where it is false, and where MongoDB or another architecture remains the better engineering choice.
The purpose is not to prove PostgreSQL wins.
The purpose is to create a system in which the trade-offs become measurable.
---
# 36. References and Upstream Baseline
The following upstream sources establish the initial technical baseline for this blueprint. Versions and capabilities should be revalidated when experiments are run.
1. **CloudNativePG — Image Volume Extensions**
https://cloudnative-pg.io/documentation/current/imagevolume_extensions/
2. **CloudNativePG — Operator Capability Levels**
https://cloudnative-pg.io/docs/current/operator_capability_levels/
3. **CloudNativePG — Declarative Database Management**
https://cloudnative-pg.io/docs/devel/declarative_database_management/
4. **CloudNativePG 1.30 Release**
https://cloudnative-pg.io/releases/cloudnative-pg-1-30.0-released/
5. **DocumentDB — Getting Started / Architecture Components**
https://documentdb.io/docs/getting-started
6. **DocumentDB — PostgreSQL API / Gateway**
https://documentdb.io/docs/postgres-api
7. **DocumentDB Kubernetes Operator**
https://documentdb.io/kubernetes-operator
8. **DocumentDB Kubernetes Operator — API Reference**
https://documentdb.io/documentdb-kubernetes-operator/latest/preview/api-reference/
9. **FerretDB — Introduction**
https://docs.ferretdb.io/
10. **FerretDB — Migration to v2 / DocumentDB Backend**
https://docs.ferretdb.io/migration/migrating-from-v1/
11. **PostgreSQL — Row Security Policies**
https://www.postgresql.org/docs/current/ddl-rowsecurity.html
12. **MongoDB — Multi-Tenant Architecture**
https://www.mongodb.com/docs/atlas/build-multi-tenant-arch/
13. **MongoDB — Sharding**
https://www.mongodb.com/docs/manual/sharding/
14. **MongoDB — Scaling Strategies / Collection Placement**
https://www.mongodb.com/docs/manual/core/sharding-scaling-strategies/
15. **MongoDB — Manage Unsharded Collections**
https://www.mongodb.com/docs/manual/core/sharding-manage-unsharded-collections/
Reference in a new issue Copy permalink