# ArchitectureBlueprint.md ## Repository `doc-store-pg` ## Status Architecture blueprint / research baseline. This document defines the initial technical architecture for `doc-store-pg`. It is intentionally a blueprint rather than a frozen implementation specification. The architecture should remain stable enough to support reproducible experimentation while leaving individual components replaceable. The central design question is: > Can document storage become a composable capability of a CloudNativePG/PostgreSQL platform while preserving MongoDB-compatible application surfaces, strong multitenancy choices, PostgreSQL-native access, and an evidence-driven path to long-term optimization? --- ## 1. Architectural Intent `doc-store-pg` should provide a document-oriented data surface on top of PostgreSQL without turning PostgreSQL into an opaque implementation detail. The architecture therefore preserves two first-class access paths: 1. **Document access** through MongoDB-compatible protocols and semantics. 2. **PostgreSQL access** through SQL and PostgreSQL-native tooling. The project should prefer composition of upstream technologies over reimplementation. The initial architecture is built around: - Kubernetes as the runtime substrate - CloudNativePG as the PostgreSQL lifecycle and HA layer - PostgreSQL 18+ as the initial database baseline - DocumentDB PostgreSQL extensions as the primary document engine candidate - DocumentDB Gateway and FerretDB as interchangeable MongoDB-compatible gateway candidates - PostgreSQL-native JSONB as a baseline and fallback document representation - Kubernetes-native configuration and observability - explicit multitenancy and tenant-placement policies - reproducible benchmark and conformance workloads The architecture is designed so that these choices can be challenged by research rather than becoming permanent assumptions. --- ## 2. Architecture Principles ### 2.1 PostgreSQL Remains Visible Document functionality is added to PostgreSQL; it does not replace the PostgreSQL model. Applications or services may intentionally use: - MongoDB-compatible document APIs - SQL - JSON/JSONB queries - BSON/document functions - relational joins - PostgreSQL transactions - PostgreSQL extensions This enables hybrid relational/document workloads to be treated as a core capability rather than an accidental escape hatch. ### 2.2 Composition Before Reimplementation Existing open-source implementations should be evaluated before new equivalents are written. In particular: - BSON storage and document semantics should initially come from DocumentDB. - MongoDB wire-protocol translation should initially come from DocumentDB Gateway or FerretDB. - PostgreSQL HA, failover, replication, backup integration, and cluster lifecycle should remain CloudNativePG responsibilities. `doc-store-pg` should only implement functionality itself where integration, compatibility, policy, observability, or research requirements cannot reasonably be satisfied upstream. ### 2.3 Gateways Are Replaceable MongoDB compatibility is an interface, not a hard dependency on one gateway. ```text MongoDB client | +--> DocumentDB Gateway --> PostgreSQL + DocumentDB | +--> FerretDB -----------> PostgreSQL + DocumentDB ``` Additional gateway implementations may later be evaluated using the same conformance and benchmark suites. ### 2.4 Tenant Placement Is Explicit Multitenancy is not represented by a boolean flag. A tenant should have an explicit placement/isolation model: ```text POOLED NAMESPACE DATABASE DEDICATED ``` Future versions may add distributed or federated placement modes. ### 2.5 Compatibility Is Measured "MongoDB compatible" is not considered a binary architectural property. Compatibility should be measured separately for: - wire protocol - CRUD operations - query semantics - aggregation semantics - indexing - drivers - tools - transactions - sessions - change streams - errors and edge cases - application-level behavior ### 2.6 Evidence Before Optimization Architectural decisions that materially affect compatibility, tenancy, performance, or operations should be benchmarkable. The system should make it possible to ask: > Which configuration is better for this workload and why? rather than: > Which database do we generally prefer? ### 2.7 Upstream Escape Hatches Remain Available DocStorePG should not create unnecessary lock-in to its own abstractions. A deployment should remain understandable through normal: - Kubernetes resources - CloudNativePG resources - PostgreSQL tools - DocumentDB tooling - MongoDB-compatible clients The project should add useful abstraction, not hide the underlying systems. --- # 3. System Context ```mermaid flowchart TB subgraph Clients A[MongoDB-compatible applications] B[PostgreSQL / SQL applications] C[Operations and automation] D[Benchmark and conformance clients] end subgraph DocStorePG["doc-store-pg capability"] G[MongoDB-compatible Gateway] P[PostgreSQL Access] T[Tenant / Placement Policy] O[Observability] X[Benchmark & Conformance Hooks] end subgraph DataPlane["CloudNativePG Data Plane"] CNPG[CloudNativePG Cluster] PG[(PostgreSQL)] DDB[DocumentDB Extensions] JSONB[Native JSONB] end A --> G B --> P C --> T D --> X G --> PG P --> PG T --> CNPG X --> G X --> P CNPG --> PG PG --- DDB PG --- JSONB PG --> O G --> O ``` The important architectural point is that **DocStorePG is not initially a database engine**. It is a capability profile formed from: ```text PostgreSQL + document engine + compatible gateway + CloudNativePG operations + tenancy model + policy + observability + conformance + benchmarks ``` --- # 4. Layer Model The initial architecture is divided into seven layers. ```text L7 Research & Evidence benchmarks, conformance, experiments, results L6 DocStorePG Control profiles, tenancy policy, placement, configuration L5 Application Interfaces MongoDB wire protocol, PostgreSQL protocol, admin APIs L4 Document Semantics BSON, document CRUD, query, aggregation, indexing L3 PostgreSQL Data Platform SQL, JSONB, transactions, RLS, extensions L2 CloudNativePG Operations cluster lifecycle, HA, replication, backup hooks, services L1 Kubernetes Infrastructure scheduling, storage, networking, secrets, resources ``` A change at one layer should be measurable without requiring unrelated layers to change. For example, replacing FerretDB with DocumentDB Gateway should not require changing the PostgreSQL cluster, tenant workload, or benchmark definitions. --- # 5. Reference Runtime Architecture ## 5.1 Default Reference Deployment ```mermaid flowchart LR MC[Mongo Client] SC[SQL Client] subgraph GW["Gateway Tier"] G1[DocumentDB Gateway] end subgraph K8S["Kubernetes"] subgraph CNPG["CloudNativePG Cluster"] RW[(Primary)] R1[(Replica)] R2[(Replica)] end end MC -->|MongoDB wire protocol| G1 G1 -->|PostgreSQL operations| RW SC -->|PostgreSQL protocol| RW RW -->|streaming replication| R1 RW -->|streaming replication| R2 ``` The default research deployment should use three PostgreSQL instances when testing HA behavior and may use a single PostgreSQL instance for local functional development. The gateway tier should remain separately deployable from PostgreSQL so that: - gateway resource consumption can be measured independently - multiple gateway implementations can be compared - gateway replicas can scale independently - gateway failures can be distinguished from database failures - protocol translation overhead can be isolated --- ## 5.2 PostgreSQL Baseline The initial baseline should target PostgreSQL 18 or later. This is particularly useful because CloudNativePG's ImageVolume extension mechanism uses PostgreSQL 18's `extension_control_path` support and permits compatible extension images to be mounted dynamically into cluster pods. The implementation should pin exact versions in deployment profiles rather than using floating image tags. Example conceptual lock information: ```yaml platform: postgresql: "18.x" cloudnativepg: "" kubernetes: "" extensions: documentdb: version: "" image: "" gateway: implementation: documentdb version: "" ``` The exact versions belong in machine-readable deployment profiles, not permanently in this blueprint. --- # 6. Extension Architecture ## 6.1 Primary Document Engine DocumentDB is the initial primary candidate because its architecture already separates document functionality into PostgreSQL-oriented components: ```text pg_documentdb_core | +-- BSON storage and primitives pg_documentdb | +-- document API +-- CRUD +-- queries +-- indexes +-- aggregation functionality pg_documentdb_gw | +-- MongoDB wire protocol +-- authentication/session gateway behavior ``` DocStorePG should treat these as upstream capabilities. ## 6.2 Extension Packaging Preferred path: ```text OCI extension image | v CloudNativePG ImageVolume | v PostgreSQL pod | v CREATE/UPDATE EXTENSION ``` This should be preferred over building permanently customized PostgreSQL images where the required extension can be safely managed as an image-volume extension. A custom PostgreSQL image remains an allowed fallback for: - unsupported extension packaging - experimental builds - debugging - compatibility tests - extension combinations that cannot yet use ImageVolumes ## 6.3 Extension Lifecycle Extension lifecycle must be treated separately from application schema lifecycle. Required states: ```text available installed configured upgradable upgraded rollback-tested ``` The benchmark program should eventually include: - extension install time - upgrade behavior - PostgreSQL major-upgrade interaction - replica behavior - recovery behavior - incompatible-extension failure behavior --- # 7. Gateway Architecture ## 7.1 Gateway Contract A gateway accepts MongoDB-compatible client traffic and maps it to the PostgreSQL-backed document engine. The gateway layer must be independently observable. Required gateway metrics should eventually include: - request count - command type - response latency - upstream PostgreSQL latency - translation latency where measurable - active connections - connection establishment rate - rejected connections - authentication failures - protocol errors - unsupported commands - PostgreSQL errors - memory - CPU - network throughput ## 7.2 Candidate A — DocumentDB Gateway DocumentDB Gateway is the reference path closest to the DocumentDB implementation itself. Use it as the initial compatibility baseline. ## 7.3 Candidate B — FerretDB FerretDB 2.x uses PostgreSQL with the DocumentDB extension as its backend and translates MongoDB wire-protocol operations for clients. It should be preserved as an alternative gateway because it permits a controlled experiment: ```text same client same workload same PostgreSQL same DocumentDB engine different gateway ``` This isolates gateway behavior from storage-engine behavior. ## 7.4 Gateway Selection Profile Conceptual configuration: ```yaml gateway: enabled: true implementation: documentdb replicas: 2 resources: profile: standard service: exposure: cluster-internal observability: enabled: true ``` Alternative: ```yaml gateway: implementation: ferretdb ``` The actual configuration mechanism may initially be Helm/Kustomize values rather than a custom CRD. --- # 8. Native PostgreSQL Surface MongoDB compatibility should never be the only supported path. DocStorePG should deliberately preserve a native SQL surface. ```mermaid flowchart TD DOC[(Document data)] M[MongoDB-compatible API] S[SQL] H[Hybrid application] M --> DOC S --> DOC H --> M H --> S ``` This enables research into workloads such as: ```sql SELECT c.customer_id, d.document FROM customers c JOIN device_documents d ON d.customer_id = c.customer_id WHERE c.contract_status = 'active' AND document_matches_capability(d.document, 'cooling'); ``` The precise document functions may differ, but the architectural requirement is that document data remain usable from SQL where the upstream engine supports it. This hybrid access mode is one of the central differentiators to test. --- # 9. Baseline Storage Profiles DocStorePG should initially define stable comparable storage profiles. ## P0 — Native PostgreSQL JSONB ```text PostgreSQL + JSONB + native PostgreSQL indexes + no Mongo gateway ``` Purpose: - establish the minimum PostgreSQL document baseline - measure what PostgreSQL already provides without a compatibility layer ## P1 — DocumentDB / Native PostgreSQL Interface ```text PostgreSQL + DocumentDB extensions + direct PostgreSQL access ``` Purpose: - isolate DocumentDB storage and document semantics - measure document-engine overhead without Mongo wire translation ## P2 — DocumentDB Gateway ```text Mongo client | DocumentDB Gateway | PostgreSQL + DocumentDB ``` Purpose: - primary MongoDB compatibility candidate ## P3 — FerretDB Gateway ```text Mongo client | FerretDB | PostgreSQL + DocumentDB ``` Purpose: - alternative compatibility path - gateway comparison ## M0 — Native MongoDB Reference ```text Mongo client | MongoDB ``` Purpose: - external reference implementation - semantic compatibility oracle where appropriate - performance and operational comparator These profile identifiers should remain stable so benchmark results can be compared over time. --- # 10. Multitenancy Architecture Multitenancy is a first-class architecture dimension. The initial tenant-placement ladder is: ```text T0 POOLED T1 NAMESPACE T2 DATABASE T3 DEDICATED ``` These identifiers should remain stable in benchmark datasets. ## 10.1 T0 — Pooled Multiple tenants share document structures. ```text PostgreSQL database | +-- shared document collection/table | +-- tenant_id = A +-- tenant_id = B +-- tenant_id = C ``` Required design goal: > Tenant identity should be enforced as deeply as the selected interface permits. For PostgreSQL-native access, this should include evaluating Row-Level Security. Illustrative pattern: ```sql ALTER TABLE documents ENABLE ROW LEVEL SECURITY; CREATE POLICY tenant_isolation ON documents USING ( tenant_id = current_setting('docstorepg.tenant_id')::uuid ); ``` ### Research questions - Can tenant identity propagate safely through a MongoDB-compatible gateway? - Does the document engine expose sufficient hooks for RLS-based enforcement? - Can a gateway accidentally bypass RLS through privileged database roles? - How can connection pooling safely preserve tenant context? - What performance cost does RLS introduce? - Can tenant-aware compound indexes prevent cross-tenant hot spots? - How do backup and restore work for a single pooled tenant? ## 10.2 T1 — Namespace Tenants receive PostgreSQL schemas or another logically separate namespace. ```text cluster | +-- database | +-- tenant_a schema +-- tenant_b schema +-- tenant_c schema ``` Advantages to evaluate: - stronger logical separation than pooled rows - separate objects and indexes - tenant-specific schema customization - relatively low compute overhead compared with dedicated clusters Risks to evaluate: - object proliferation - PostgreSQL catalog growth - migration complexity - gateway mapping complexity - uncertain fit with document-engine assumptions - backup/restore granularity T1 must be treated as an experiment until compatibility with the selected document engine and gateway is proven. ## 10.3 T2 — Database Each tenant receives a separate PostgreSQL database inside one CloudNativePG cluster. ```text CloudNativePG cluster | +-- tenant_a database +-- tenant_b database +-- tenant_c database ``` CloudNativePG can declaratively manage additional databases and their extensions/schemas. This mode should evaluate: - per-tenant ownership - per-database extensions - migration lifecycle - database count scaling - connection overhead - catalog/resource overhead - backup/restore behavior - gateway routing - tenant-specific configuration - noisy-neighbour effects The compute and storage failure domain remains shared. ## 10.4 T3 — Dedicated Each tenant receives a dedicated CloudNativePG-backed DocStorePG deployment. ```mermaid flowchart LR A[Tenant A] --> CA[DocStorePG Cluster A] B[Tenant B] --> CB[DocStorePG Cluster B] C[Tenant C] --> CC[DocStorePG Cluster C] ``` This mode provides the strongest initial isolation boundary. Potential isolation includes: - PostgreSQL process - PostgreSQL database - gateway - CPU request/limit - memory request/limit - persistent storage - Kubernetes scheduling - secrets - service identity - network policy - backup policy - failure domain The primary downside is fleet overhead. T3 is expected to be most relevant to: - large enterprise tenants - regulated workloads - tenants requiring special extensions - tenants requiring independent maintenance windows - high-volume "whale" tenants - contractual isolation requirements --- # 11. Tenant Placement as a Lifecycle The important abstraction is not merely selecting one of T0–T3. The longer-term architecture should explore tenant mobility: ```mermaid stateDiagram-v2 [*] --> Pooled Pooled --> Namespace Pooled --> Database Namespace --> Database Database --> Dedicated Pooled --> Dedicated Dedicated --> Database Database --> Pooled ``` Not all transitions need to be supported initially. The strategic goal is: > Keep the application-facing document contract as stable as possible while tenant placement changes underneath it. This is DocStorePG's initial conceptual answer to some of the problems that MongoDB solves with shard placement. It is not initially intended to reproduce MongoDB sharding. --- # 12. Explicit Sharding Boundary MongoDB provides native horizontal sharding in which data can be distributed across shards and routed through a unified cluster interface. CloudNativePG's normal architecture is a PostgreSQL primary/standby cluster. It is not by itself a transparent horizontally sharded write architecture. Therefore: > DocStorePG V1 must not claim MongoDB-equivalent horizontal sharding. The initial architecture deliberately stops here: ```text single DocStorePG deployment | +-- primary +-- replicas ``` and permits multiple independent placements: ```text DocStorePG A DocStorePG B DocStorePG C ``` A future research track may investigate a placement/router layer: ```mermaid flowchart TD APP[Application] R[Tenant Router] A[DocStorePG Placement A] B[DocStorePG Placement B] C[DocStorePG Placement C] APP --> R R --> A R --> B R --> C ``` That work should only begin once there is evidence that it solves a problem not already better solved by: - MongoDB sharding - Citus or another PostgreSQL distribution technology - application-level partitioning - independent tenant clusters - another existing distributed database --- # 13. Proposed DocStorePG Control Surface DocStorePG should not immediately create a Kubernetes operator. The first implementation should prefer: ```text Helm / Kustomize + CloudNativePG CRDs + standard Kubernetes resources + benchmark profiles + explicit configuration files ``` Only after recurring orchestration requirements emerge should a DocStorePG CRD/operator be justified. A possible future API is documented here to establish architectural vocabulary. ## 13.1 Future `DocStore` Resource ```yaml apiVersion: docstorepg.io/v1alpha1 kind: DocStore metadata: name: application-documents spec: postgres: profile: ha-standard documentEngine: implementation: documentdb gateway: implementation: documentdb replicas: 2 interfaces: mongodb: true postgresql: true tenancy: defaultPlacement: pooled observability: enabled: true ``` ## 13.2 Future `DocStoreTenant` Resource ```yaml apiVersion: docstorepg.io/v1alpha1 kind: DocStoreTenant metadata: name: acme spec: storeRef: application-documents placement: mode: database resources: class: standard compatibility: profile: mongodb-default ``` These resources are **design sketches**, not implementation commitments. --- # 14. Configuration Profiles Configuration should be grouped into composable profiles. ```text profiles/ ├── platform/ │ ├── local-single.yaml │ ├── ha-small.yaml │ └── ha-standard.yaml ├── engine/ │ ├── jsonb.yaml │ └── documentdb.yaml ├── gateway/ │ ├── none.yaml │ ├── documentdb.yaml │ └── ferretdb.yaml ├── tenancy/ │ ├── pooled.yaml │ ├── namespace.yaml │ ├── database.yaml │ └── dedicated.yaml └── workload/ ├── document-crud.yaml ├── aggregation.yaml └── hybrid.yaml ``` A benchmark environment can then be expressed as composition: ```text platform/ha-small + engine/documentdb + gateway/ferretdb + tenancy/pooled + workload/document-crud ``` This makes experiments easier to reproduce and diff. --- # 15. Security Architecture The security model should be layered. ```text Kubernetes identity and RBAC | NetworkPolicy / service exposure | gateway authentication | PostgreSQL authentication | PostgreSQL role / database / schema privileges | Row-Level Security where applicable | document/application authorization ``` ## 15.1 Security Rules 1. Gateway database credentials should not use PostgreSQL superuser privileges in production profiles. 2. `BYPASSRLS` roles must not be used for pooled tenant traffic where RLS is part of the tenant boundary. 3. Tenant context must be explicit and auditable. 4. Connection pooling must not leak tenant context between sessions. 5. PostgreSQL direct access must not silently bypass the intended Mongo-facing security model. 6. Mongo-compatible and SQL-facing identities must be mapped deliberately. 7. Secrets should be provided through Kubernetes-native secret mechanisms or external secret-management integration. 8. External service exposure should be opt-in. 9. NetworkPolicy should be part of production reference profiles. 10. Security tests belong in the conformance suite. --- # 16. Identity Propagation Tenant-aware pooled operation creates an important architecture problem: ```text MongoDB client identity | v Mongo gateway | v PostgreSQL session identity | v RLS / database policy ``` DocStorePG should explicitly test multiple strategies. ### Strategy A — Database Role Mapping One PostgreSQL role per tenant or tenant class. Potential strength: - database-native identity Potential weakness: - large role counts - connection-pool complexity ### Strategy B — Session Tenant Context A trusted gateway sets a PostgreSQL session variable such as: ```sql SET docstorepg.tenant_id = '...'; ``` RLS reads that context. Potential strength: - scalable logical identity Potential weakness: - gateway becomes security-critical - session reset must be flawless ### Strategy C — Database-per-Tenant Routing Gateway selects the tenant database. Potential strength: - stronger logical boundary Potential weakness: - routing and connection-pool fragmentation No strategy should be declared canonical until tested. --- # 17. Network Architecture Default production stance: ```text Mongo endpoint: cluster-internal PostgreSQL endpoint: cluster-internal admin endpoints: cluster-internal metrics endpoints: monitoring namespace / authorized collectors external exposure: explicit profile only ``` Conceptual Kubernetes services: ```text docstore-mongo-rw docstore-postgres-rw docstore-postgres-ro docstore-metrics ``` The exact service structure should follow the selected upstream gateway and CloudNativePG conventions. --- # 18. High Availability CloudNativePG should own PostgreSQL HA. ```mermaid flowchart LR GW[Gateway replicas] SVC[CNPG RW Service] P[(Primary)] R1[(Replica)] R2[(Replica)] GW --> SVC SVC --> P P --> R1 P --> R2 ``` DocStorePG should test gateway behavior during PostgreSQL failover. Important questions: - Does the gateway reconnect automatically? - What do clients observe? - Which operations fail? - Are retries safe? - How do transactions behave? - How quickly does service recover? - Does MongoDB client retry behavior interact correctly with PostgreSQL failover? - Does the gateway retain stale connections? Failover testing belongs in the benchmark program. --- # 19. Read Scaling CloudNativePG provides read-only service patterns through replicas. DocStorePG must not assume MongoDB-style read preferences map automatically to PostgreSQL read replicas. Possible future mapping: ```text Mongo read preference | v Gateway policy | +--> primary | +--> CNPG read-only service ``` This requires explicit semantic research around: - consistency - replication lag - transaction semantics - session guarantees - MongoDB driver expectations - failover Until proven, primary routing should be the correctness baseline. --- # 20. Backup and Recovery Backup architecture must distinguish between: ```text cluster recovery database recovery tenant recovery collection recovery document recovery ``` CloudNativePG provides cluster-oriented PostgreSQL backup/recovery mechanisms; the DocumentDB Kubernetes Operator also has its own evolving backup abstractions. DocStorePG should not assume that cluster-level backup automatically satisfies tenant-level recovery requirements. Research must measure: - backup duration - backup size - restore duration - point-in-time recovery - tenant-specific restore - restoring one tenant from a pooled dataset - restoring database-per-tenant - dedicated-cluster restore - Mongo-compatible application recovery behavior Tenant recoverability is an important factor in choosing T0–T3. --- # 21. Observability Architecture Observability should expose each layer independently. ```mermaid flowchart LR C[Client] G[Gateway] P[PostgreSQL] K[Kubernetes] B[Benchmark Harness] C --> G --> P G --> M[Metrics] P --> M K --> M B --> M G --> L[Logs] P --> L K --> L M --> O[Observability Backend] L --> O ``` Minimum metric categories: ### PostgreSQL - transactions - query latency - query count - locks - connections - cache hit behavior - WAL generation - checkpoint behavior - replication lag - table/index size - I/O - CPU - memory ### Gateway - operations - command types - latency - failures - unsupported requests - active connections - authentication - CPU - memory ### Kubernetes - pod restarts - scheduling - CPU requests/usage - memory requests/usage - storage - network - failovers - PVC behavior ### Tenant Where possible: - requests per tenant - storage per tenant - latency per tenant - error rate per tenant - resource share - noisy-neighbour indicators ### Benchmark Every benchmark run must capture enough environment metadata to explain the result. --- # 22. Benchmark and Conformance Plane The benchmark plane is part of the architecture, not an external one-off tool. ```mermaid flowchart TB W[Workload Catalog] R[Runner] M0[MongoDB Reference] P0[PostgreSQL JSONB] P1[DocumentDB Native] P2[DocumentDB Gateway] P3[FerretDB] C[Comparator] E[(Evidence Store)] W --> R R --> M0 R --> P0 R --> P1 R --> P2 R --> P3 M0 --> C P0 --> C P1 --> C P2 --> C P3 --> C C --> E ``` Benchmark code should support deterministic seeds where possible. Every result should include: ```yaml run: id: ... timestamp: ... gitCommit: ... workload: ... seed: ... environment: kubernetes: ... nodes: ... cpu: ... memory: ... storageClass: ... software: mongodb: ... postgresql: ... cloudnativepg: ... documentdb: ... gateway: ... profile: storage: P2 tenancy: T0 dataset: tenants: ... documents: ... averageDocumentSize: ... result: throughput: ... latencyP50: ... latencyP95: ... latencyP99: ... errors: ... ``` This structure should evolve into a formal benchmark schema. --- # 23. Differential Conformance Testing For compatible operations, DocStorePG should support differential testing. ```text request | +------> MongoDB ----------> result A | +------> DocStorePG -------> result B | v semantic comparator ``` Comparison dimensions: - returned data - BSON types - null/missing behavior - ordering - update results - aggregation results - errors - error categories/codes - transaction outcome - session behavior - index behavior Differences should be classified as: ```text IDENTICAL EQUIVALENT DOCUMENTED_DIFFERENCE UNSUPPORTED INCORRECT UNKNOWN ``` A compatibility score must not hide important individual differences. --- # 24. Workload Classes The architecture should support at least these benchmark workload classes. ## W0 — Basic CRUD - insert - find by ID - update - delete - bulk operations ## W1 — Flexible Schema - optional fields - heterogeneous structures - schema evolution - large/small documents ## W2 — Nested Documents - nested objects - arrays - array element updates - nested predicates ## W3 — Indexing - single-field - compound - nested - wildcard/dynamic-field equivalents - low/high cardinality - many-index write cost ## W4 — Aggregation - filtering - projection - grouping - sorting - unwind - lookup/join-like behavior - facets ## W5 — Transactions - single-document - multi-document - concurrent updates - abort/retry behavior ## W6 — Change/Event Workloads - change streams or equivalents - consumer lag - reconnect behavior - failover behavior ## W7 — Hybrid Relational/Document - relational filters over document sets - joins between relational and document data - cross-model transactions - reporting/analytics ## W8 — Multitenancy - many small tenants - mixed tenant sizes - tenant provisioning - whale tenant - noisy neighbours - placement changes - isolation tests ## W9 — Operations - failover - backup - restore - upgrade - extension upgrade - scaling - disaster simulation --- # 25. Whale-Tenant Experiment One canonical multitenancy experiment should be designed into the architecture from the start. Example distribution: ```text 1000 tenants total 950 small tenants 49 medium tenants 1 whale tenant consuming roughly half of traffic ``` The exact distribution should be parameterized. Questions: - What happens to small-tenant p99 latency? - Can the whale be identified operationally? - Can the whale move from T0 to T2/T3? - How long does migration take? - How much downtime occurs? - Does the application connection contract change? - What happens to backup cost? - At what point does dedicated placement become economically sensible? This should eventually become a signature DocStorePG benchmark. --- # 26. Tenant Migration Architecture Tenant migration is initially an experimental subsystem. ```mermaid flowchart LR S[Source Placement] E[Export / Replication] V[Validate] C[Cutover] T[Target Placement] R[Rollback Window] S --> E --> V --> C --> T C --> R ``` Desired migration properties: - tenant-scoped - auditable - resumable - consistency-aware - verifiable - rollback-aware - application-transparent where possible Potential mechanisms to investigate: - logical export/import - PostgreSQL logical replication - document API replication - change streams - dual writes - maintenance-window cutover - backup/restore extraction No mechanism is canonical yet. --- # 27. DocumentDB Kubernetes Operator Relationship The DocumentDB Kubernetes Operator is a relevant upstream project and should be actively compared with DocStorePG. It already uses CloudNativePG underneath and provides Kubernetes-native DocumentDB lifecycle capabilities. DocStorePG should therefore avoid duplicating its implementation without evidence. Initial relationship: ```text DocumentDB Operator | +-- upstream reference implementation +-- optional deployment profile +-- research comparator +-- source of operational lessons DocStorePG | +-- broader PostgreSQL document capability research +-- interchangeable gateways +-- native JSONB baseline +-- multitenancy comparison +-- MongoDB comparison +-- conformance framework +-- benchmark framework +-- hybrid relational/document research ``` If the DocumentDB Operator eventually provides the best runtime deployment path, DocStorePG should be able to adopt it rather than compete with it. --- # 28. Failure Domains DocStorePG should model failure domains explicitly. | Failure | Likely scope | Expected owner | |---|---|---| | Gateway pod | Mongo API traffic through pod | Kubernetes / gateway | | Gateway deployment | Mongo-compatible surface | DocStorePG deployment | | PostgreSQL instance | instance | CloudNativePG | | PostgreSQL primary | writes until failover | CloudNativePG | | PostgreSQL cluster | tenant set on cluster | operations | | PVC/storage | instance or cluster | Kubernetes/storage | | Namespace | contained deployments | Kubernetes | | Kubernetes node | scheduled workloads | Kubernetes | | Kubernetes cluster | all contained placements | infrastructure | | Tenant credential leak | tenant/security boundary | security model | | RLS policy error | pooled tenants | DocStorePG policy | | Extension defect | databases using extension | upstream + operations | Dedicated tenant placement should reduce blast radius at the cost of operational density. --- # 29. Scaling Model DocStorePG should distinguish four fundamentally different scaling mechanisms. ## Vertical Increase: - CPU - memory - storage performance ## Read Scale Add PostgreSQL replicas where semantics permit. ## Gateway Scale Increase stateless gateway replicas. ## Placement Scale Move tenants or workloads onto additional independent DocStorePG deployments. ```text scale | +-----------------+----------------+ | | | vertical gateway placement | +--------+--------+ | | shared dedicated ``` Transparent horizontal write sharding is explicitly outside the initial model. --- # 30. Repository Architecture Proposed repository structure: ```text doc-store-pg/ ├── INTENT.md ├── ArchitectureBlueprint.md ├── ResearchProgram.md ├── BenchmarkSpecification.md ├── CompatibilityModel.md ├── MultitenancyModel.md ├── WorkloadCatalog.md ├── README.md │ ├── docs/ │ ├── architecture/ │ ├── decisions/ │ ├── findings/ │ └── references/ │ ├── deploy/ │ ├── cnpg/ │ ├── documentdb/ │ ├── gateways/ │ │ ├── documentdb/ │ │ └── ferretdb/ │ └── mongodb-reference/ │ ├── profiles/ │ ├── platform/ │ ├── engine/ │ ├── gateway/ │ ├── tenancy/ │ └── workload/ │ ├── experiments/ │ ├── compatibility/ │ ├── performance/ │ ├── multitenancy/ │ ├── operations/ │ └── migration/ │ ├── benchmarks/ │ ├── runner/ │ ├── workloads/ │ ├── schemas/ │ └── analysis/ │ ├── tests/ │ ├── conformance/ │ ├── security/ │ ├── failover/ │ └── integration/ │ ├── results/ │ └── README.md │ └── scripts/ ``` Generated benchmark results should be governed carefully so the repository does not become filled with large ephemeral datasets. Machine-readable summaries belong in Git where useful; raw large datasets may require external object storage with manifests/checksums committed to the repository. --- # 31. Initial Implementation Stages ## Stage A — Functional Baseline Establish: ```text Kubernetes + CloudNativePG + PostgreSQL ``` Verify normal SQL operation. ## Stage B — Document Engine Add: ```text DocumentDB extensions ``` Verify document operations through PostgreSQL-native interfaces. ## Stage C — Mongo-Compatible Surface Add: ```text DocumentDB Gateway ``` Run basic official-driver CRUD tests. ## Stage D — Alternative Gateway Add: ```text FerretDB ``` Run the same tests against the same backend profile. ## Stage E — MongoDB Reference Deploy native MongoDB as M0. Run differential CRUD/query tests. ## Stage F — Tenant Baseline Implement T0 pooled tenancy and evaluate PostgreSQL-enforced isolation. ## Stage G — Isolation Ladder Add T2 database-per-tenant and T3 dedicated placement. Treat T1 schema-per-tenant as experimental and implement only if document-engine behavior supports it cleanly. ## Stage H — Benchmark Plane Add machine-readable benchmark runs and environment manifests. ## Stage I — Operational Experiments Test: - failover - restart - backup - restore - upgrade - noisy neighbour - tenant movement --- # 32. Initial Architecture Decisions The following decisions are provisional but form the starting baseline. ### AD-001 — CloudNativePG is the PostgreSQL operator **Decision:** Use CloudNativePG rather than introducing a custom PostgreSQL lifecycle implementation. **Reason:** PostgreSQL lifecycle and HA are not DocStorePG's differentiated problem. ### AD-002 — DocumentDB is the initial document engine **Decision:** Use DocumentDB extensions as the first BSON/document compatibility implementation. **Reason:** It already implements document semantics on PostgreSQL and exposes a MongoDB-compatible gateway architecture. ### AD-003 — Gateway implementation remains swappable **Decision:** Support DocumentDB Gateway first; preserve FerretDB as an alternative profile. **Reason:** This allows protocol-layer effects to be isolated experimentally. ### AD-004 — PostgreSQL access remains first-class **Decision:** Do not hide SQL/PostgreSQL interfaces behind the document API. **Reason:** Hybrid relational/document capability is a core research hypothesis. ### AD-005 — Multitenancy has explicit placement levels **Decision:** Use T0–T3 as stable architectural vocabulary. **Reason:** Isolation, cost, and scaling cannot be meaningfully represented by a single "multi-tenant" mode. ### AD-006 — No custom DocStorePG operator initially **Decision:** Start with manifests/profiles and existing operators. **Reason:** Operator behavior should only be built once recurring control-plane requirements are empirically understood. ### AD-007 — No claim of transparent write sharding **Decision:** Treat MongoDB sharding as a distinct comparison point rather than something DocStorePG already provides. **Reason:** CloudNativePG HA and MongoDB sharding solve different problems. ### AD-008 — Benchmarks are architecture **Decision:** Reproducible benchmark and conformance hooks are part of the system design from the beginning. **Reason:** Long-term architecture choices must remain evidence-driven. --- # 33. Architecture Questions to Resolve Through Research The blueprint intentionally leaves these open: 1. How complete is DocumentDB's real MongoDB semantic compatibility? 2. Does DocumentDB Gateway or FerretDB provide the better compatibility/performance trade-off? 3. How should Mongo identities map to PostgreSQL identities? 4. Can PostgreSQL RLS reliably enforce pooled tenant isolation through the Mongo-compatible surface? 5. What tenant count is practical for database-per-tenant within one CNPG cluster? 6. Is schema-per-tenant useful with DocumentDB or merely theoretical? 7. How much gateway overhead exists relative to direct document access? 8. How much document-engine overhead exists relative to native JSONB? 9. Where does native MongoDB outperform the PostgreSQL-backed architecture? 10. How do both architectures behave with a whale tenant? 11. What is the operational cost per tenant under T0, T2, and T3? 12. How should tenant migration work? 13. Can a stable logical endpoint survive tenant placement changes? 14. Which MongoDB read/write/session semantics can safely map to CNPG primary/replica topology? 15. What backup granularity is achievable for each tenant model? 16. How should extension versions be upgraded independently from PostgreSQL? 17. When does an existing distributed PostgreSQL technology become preferable to DocStorePG placement routing? 18. When should a workload simply remain on native MongoDB? These questions should seed `ResearchProgram.md`. --- # 34. Target Architecture The long-term target, if supported by evidence, is: ```mermaid flowchart TB subgraph Applications M[MongoDB clients] S[SQL clients] end EP[Stable DocStorePG Endpoint] TC[Tenant / Placement Control] subgraph Shared["Shared Placement"] SP[CNPG + DocumentDB] end subgraph DB["Database-Isolated Placement"] DP[CNPG + Tenant Databases] end subgraph Dedicated["Dedicated Placements"] D1[CNPG Tenant A] D2[CNPG Tenant B] end OBS[Observability] BENCH[Conformance & Benchmark Plane] M --> EP S --> EP EP --> TC TC --> SP TC --> DP TC --> D1 TC --> D2 SP --> OBS DP --> OBS D1 --> OBS D2 --> OBS BENCH --> EP BENCH --> OBS ``` The important feature is not that all boxes necessarily become custom DocStorePG software. The important feature is that the platform can express and test: ```text interface + document semantics + tenant identity + placement + isolation + PostgreSQL capability + operations + evidence ``` as separable concerns. --- # 35. Architectural North Star `doc-store-pg` should aim to make this statement testable: > A document store can be an operationally native PostgreSQL capability whose API, compatibility, tenant placement, and isolation level can evolve independently from the applications that use it. The architecture succeeds if it allows the project to discover where that statement is true, where it is false, and where MongoDB or another architecture remains the better engineering choice. The purpose is not to prove PostgreSQL wins. The purpose is to create a system in which the trade-offs become measurable. --- # 36. References and Upstream Baseline The following upstream sources establish the initial technical baseline for this blueprint. Versions and capabilities should be revalidated when experiments are run. 1. **CloudNativePG — Image Volume Extensions** https://cloudnative-pg.io/documentation/current/imagevolume_extensions/ 2. **CloudNativePG — Operator Capability Levels** https://cloudnative-pg.io/docs/current/operator_capability_levels/ 3. **CloudNativePG — Declarative Database Management** https://cloudnative-pg.io/docs/devel/declarative_database_management/ 4. **CloudNativePG 1.30 Release** https://cloudnative-pg.io/releases/cloudnative-pg-1-30.0-released/ 5. **DocumentDB — Getting Started / Architecture Components** https://documentdb.io/docs/getting-started 6. **DocumentDB — PostgreSQL API / Gateway** https://documentdb.io/docs/postgres-api 7. **DocumentDB Kubernetes Operator** https://documentdb.io/kubernetes-operator 8. **DocumentDB Kubernetes Operator — API Reference** https://documentdb.io/documentdb-kubernetes-operator/latest/preview/api-reference/ 9. **FerretDB — Introduction** https://docs.ferretdb.io/ 10. **FerretDB — Migration to v2 / DocumentDB Backend** https://docs.ferretdb.io/migration/migrating-from-v1/ 11. **PostgreSQL — Row Security Policies** https://www.postgresql.org/docs/current/ddl-rowsecurity.html 12. **MongoDB — Multi-Tenant Architecture** https://www.mongodb.com/docs/atlas/build-multi-tenant-arch/ 13. **MongoDB — Sharding** https://www.mongodb.com/docs/manual/sharding/ 14. **MongoDB — Scaling Strategies / Collection Placement** https://www.mongodb.com/docs/manual/core/sharding-scaling-strategies/ 15. **MongoDB — Manage Unsharded Collections** https://www.mongodb.com/docs/manual/core/sharding-manage-unsharded-collections/