diff --git a/INTENT.md b/INTENT.md new file mode 100755 index 0000000..9d69420 --- /dev/null +++ b/INTENT.md @@ -0,0 +1,234 @@ +# INTENT.md + +## Repository + +`doc-store-pg` + +## Intent + +`doc-store-pg` explores, integrates, and validates a CloudNativePG-native document-store capability built on PostgreSQL. + +The project investigates whether the capabilities that make MongoDB attractive as a document database can be provided as a composable PostgreSQL capability without giving up PostgreSQL's relational strengths, operational model, extensibility, and flexible multitenancy boundaries. + +The repository is intended to move this question from architectural theory into an inspectable, reproducible implementation and research program. + +## Core Hypothesis + +Document-oriented storage does not necessarily require a separate database architecture. + +A PostgreSQL platform, extended with document-oriented data types, query semantics, indexing, MongoDB-compatible interfaces, and suitable operational tooling, may provide a competitive document-store surface while retaining the advantages of PostgreSQL and CloudNativePG. + +`doc-store-pg` exists to test that hypothesis rather than assume it. + +## Objectives + +The project aims to: + +1. Establish a reproducible CloudNativePG-based document-store reference architecture. +2. Integrate suitable PostgreSQL document-store technologies such as DocumentDB and compatible gateway implementations. +3. Expose MongoDB-compatible APIs where useful while retaining direct PostgreSQL access. +4. Measure semantic compatibility with MongoDB rather than treating wire compatibility as sufficient. +5. Compare document-oriented workloads across native MongoDB and PostgreSQL-based implementations. +6. Investigate hybrid relational/document workloads that make use of PostgreSQL-specific strengths. +7. Explore multitenancy models ranging from highly pooled tenants to strongly isolated tenant deployments. +8. Measure operational characteristics including deployment, failover, backup, restore, upgrade, scaling, observability, and resource efficiency. +9. Identify which capabilities belong in reusable PostgreSQL extensions, gateway layers, CloudNativePG configuration, Kubernetes control-plane resources, or higher-level orchestration. +10. Produce evidence that can guide long-term architectural and product decisions. + +## Scope + +The repository may contain or coordinate work relating to: + +- CloudNativePG deployment profiles +- PostgreSQL extension packaging and lifecycle management +- DocumentDB integration +- MongoDB-compatible gateways +- FerretDB integration where useful +- BSON and JSONB storage models +- document query and aggregation semantics +- indexing strategies +- MongoDB driver and application compatibility +- transaction and change-stream behavior +- PostgreSQL-native access to document data +- relational/document hybrid queries +- schema validation +- observability and query diagnostics +- benchmark harnesses +- conformance tests +- workload generators +- operational tests +- multitenancy models +- tenant placement and isolation policies +- tenant migration experiments +- performance and cost analysis +- architecture research and decision records + +## Multitenancy Research + +Multitenancy is a first-class concern of the project. + +`doc-store-pg` should investigate a continuum of tenant isolation models, including: + +### Pooled + +Multiple tenants share document structures and storage, with tenant boundaries enforced through mechanisms such as tenant identifiers, PostgreSQL roles, Row-Level Security, gateway policy, and application identity propagation. + +### Namespace + +Tenants receive distinct PostgreSQL schemas or comparable logical namespaces while sharing a PostgreSQL cluster. + +### Database + +Tenants receive distinct PostgreSQL databases within a shared CloudNativePG cluster. + +### Dedicated + +Tenants receive dedicated CloudNativePG / document-store clusters and corresponding compute, storage, security, and failure boundaries. + +The project should evaluate whether tenants can move between these models as their requirements evolve without requiring applications to adopt different document APIs. + +This creates a central research question: + +> Can tenant placement and isolation become configurable properties of the data platform rather than permanent application architecture decisions? + +## Compatibility Model + +MongoDB compatibility must be treated as a spectrum rather than a binary property. + +The project should distinguish at least: + +- connection compatibility +- CRUD compatibility +- query compatibility +- index compatibility +- aggregation compatibility +- driver compatibility +- tooling compatibility +- transaction compatibility +- change-stream compatibility +- behavioral and error-semantics compatibility +- real-application compatibility + +Compatibility claims should be backed by reproducible tests. + +## Benchmarking Intent + +Benchmarking is a core product of the repository rather than an afterthought. + +The benchmark program should compare meaningful architectural configurations, potentially including: + +- native MongoDB +- PostgreSQL with JSONB +- PostgreSQL with document-store extensions +- PostgreSQL document APIs used directly +- MongoDB-compatible DocumentDB gateways +- FerretDB or other compatible gateways +- different CloudNativePG deployment profiles +- different multitenancy and isolation models + +Benchmarks should measure more than throughput. + +Relevant dimensions include: + +- latency +- throughput +- concurrency +- storage efficiency +- index size +- write amplification +- CPU efficiency +- memory efficiency +- connection overhead +- noisy-neighbour behavior +- tenant isolation +- failover behavior +- backup and restore +- recovery objectives +- upgrade complexity +- provisioning time +- tenant migration +- observability +- operational complexity +- cost per workload +- cost per tenant + +Benchmark results should remain reproducible, inspectable, and comparable over time. + +## Architectural Principle + +The project should prefer composition over reimplementation. + +Where mature open-source components already provide BSON handling, MongoDB protocol translation, query semantics, or other document-store functionality, `doc-store-pg` should integrate and evaluate them before implementing replacements. + +The repository's durable value should primarily come from: + +- architecture +- integration +- configuration +- compatibility profiles +- operational patterns +- multitenancy models +- conformance testing +- benchmark methodology +- comparative evidence +- automation +- reusable deployment abstractions + +rather than unnecessary duplication of upstream database functionality. + +## Non-Goals + +`doc-store-pg` does not initially aim to: + +- implement a MongoDB-compatible database engine from scratch +- reproduce every MongoDB feature +- claim drop-in compatibility without evidence +- replace CloudNativePG +- create a new PostgreSQL distribution +- build a distributed sharding system before simpler architectures have been evaluated +- optimize exclusively for synthetic benchmark scores +- assume PostgreSQL is superior to MongoDB +- assume MongoDB compatibility is the only valuable document-store interface + +The repository should remain evidence-driven. + +## Success Criteria + +The project is successful when it can provide clear, reproducible answers to questions such as: + +1. Which MongoDB workloads can run unchanged or with minimal adaptation? +2. What performance overhead is introduced by document abstractions and MongoDB protocol gateways? +3. Where does native MongoDB retain architectural or operational advantages? +4. Where does PostgreSQL provide equivalent or better behavior? +5. Which workloads benefit from combining relational and document models? +6. How do different multitenancy strategies affect density, isolation, performance, security, and cost? +7. How effectively can tenants move between pooled and dedicated deployment models? +8. Which configuration produces the best trade-off for a given workload? +9. Can a CloudNativePG platform offer document storage without introducing a second independent database operational stack? +10. Under which conditions should an operator deliberately choose MongoDB instead? + +## Expected Repository Outputs + +The repository is expected to evolve through artifacts such as: + +- `ArchitectureBlueprint.md` +- `ResearchProgram.md` +- `BenchmarkSpecification.md` +- `CompatibilityModel.md` +- `MultitenancyModel.md` +- `WorkloadCatalog.md` +- `BenchmarkResults/` +- `experiments/` +- `deploy/` +- `profiles/` +- `tests/` +- `docs/` +- architecture decision records +- reproducible benchmark environments +- machine-readable benchmark results + +## Guiding Principle + +> Treat document storage as a capability to be composed, measured, and governed — not as a database category to be assumed. + +`doc-store-pg` should make the trade-offs between PostgreSQL-based document storage and native document databases visible through working systems and reproducible evidence. diff --git a/ResearchProgram.md b/ResearchProgram.md new file mode 100755 index 0000000..03fdae0 --- /dev/null +++ b/ResearchProgram.md @@ -0,0 +1,2917 @@ +# ResearchProgram.md + +## Repository + +`doc-store-pg` + +## Status + +Initial research program / execution roadmap. + +## Purpose + +This document turns the hypotheses and open questions defined in `INTENT.md` and `ArchitectureBlueprint.md` into an ordered, reproducible research program. + +The goal is not to accumulate benchmarks. + +The goal is to produce enough evidence to make progressively stronger architectural decisions about whether, where, and how a PostgreSQL / CloudNativePG platform can provide document-store capabilities that are competitive with MongoDB while retaining PostgreSQL-native strengths. + +The research program should answer three classes of questions: + +1. **Can it work?** + Compatibility, correctness, integration, and operational feasibility. + +2. **How well does it work?** + Performance, efficiency, resilience, multitenancy, and cost. + +3. **Where should it be used?** + Architectural fit, workload boundaries, tenant placement, and long-term product direction. + +The program is deliberately staged so that expensive performance and scale research only begins after correctness and experimental reproducibility have been established. + +--- + +# 1. Research North Star + +The central research hypothesis is: + +> Document storage can be implemented as a composable PostgreSQL capability that is operationally native to CloudNativePG, sufficiently compatible with MongoDB for meaningful application reuse, and superior in selected hybrid or multitenant scenarios without requiring MongoDB's architecture to be reproduced wholesale. + +The program must be capable of disproving this hypothesis. + +A successful research outcome may therefore be any of the following: + +- PostgreSQL-based document storage is broadly competitive with MongoDB. +- PostgreSQL-based document storage is attractive only for specific workload classes. +- MongoDB remains clearly preferable for important document workloads. +- Different tenant classes should use different storage architectures. +- DocumentDB or FerretDB is useful only as a migration/compatibility surface. +- Native PostgreSQL JSONB is sufficient for many workloads and the MongoDB layer adds little value. +- A hybrid architecture is more compelling than a replacement architecture. +- The operational simplicity of one PostgreSQL platform outweighs moderate performance disadvantages. +- The complexity introduced by MongoDB compatibility is not justified. + +The research program exists to distinguish among these outcomes. + +--- + +# 2. Research Principles + +## 2.1 Correctness Before Performance + +A system that returns different results more quickly is not faster at the same task. + +Compatibility and semantic behavior must therefore be understood before comparative performance results are treated as meaningful. + +## 2.2 Reproducibility Before Optimization + +Every important result must be reproducible from: + +- a repository commit +- a machine-readable environment definition +- pinned software versions +- a workload definition +- a dataset definition +- a random seed where relevant +- benchmark configuration +- raw result references +- analysis code + +## 2.3 Compare Architectures, Not Brands + +The primary objects of comparison are deployment and access profiles. + +Stable profile identifiers from `ArchitectureBlueprint.md` are used: + +```text +M0 Native MongoDB reference + +P0 PostgreSQL + native JSONB +P1 PostgreSQL + DocumentDB, native PostgreSQL access +P2 PostgreSQL + DocumentDB + DocumentDB Gateway +P3 PostgreSQL + DocumentDB + FerretDB +``` + +Tenant placement identifiers: + +```text +T0 POOLED +T1 NAMESPACE +T2 DATABASE +T3 DEDICATED +``` + +Workload identifiers: + +```text +W0 Basic CRUD +W1 Flexible Schema +W2 Nested Documents +W3 Indexing +W4 Aggregation +W5 Transactions +W6 Change/Event Workloads +W7 Hybrid Relational/Document +W8 Multitenancy +W9 Operations +``` + +Additional profiles may be added, but existing identifiers should not be silently redefined. + +## 2.4 One Question Per Experiment Where Possible + +An experiment should isolate one major uncertainty. + +For example: + +```text +P1 vs P2 +``` + +is useful for estimating gateway/protocol overhead. + +By contrast: + +```text +different hardware ++ different storage ++ different gateway ++ different indexes ++ different dataset +``` + +does not provide a useful causal comparison. + +## 2.5 Negative Results Are Results + +Unsupported operations, poor performance, operational complexity, failed upgrades, and undesirable tenant behavior must be documented rather than optimized out of the published evidence. + +## 2.6 Benchmark Realistic Shapes + +The benchmark program should include both: + +- controlled synthetic workloads for causal isolation +- application-shaped workloads representing plausible production usage + +Neither is sufficient alone. + +## 2.7 Measure Tails, Not Only Means + +At minimum, performance work should consider: + +- median +- p95 +- p99 +- error rate +- saturation behavior + +Multitenant systems should additionally consider **per-tenant tail latency**. + +## 2.8 Separate Cold and Warm Behavior + +Relevant experiments should distinguish: + +- cold start +- cold cache +- warm cache +- steady state +- post-failover recovery + +## 2.9 Cost Is an Architectural Metric + +CPU, RAM, storage, IOPS, network, operational effort, and idle capacity are part of the outcome. + +A system that performs similarly but consumes twice the infrastructure is not equivalent. + +## 2.10 Upstream Changes Are Expected + +CloudNativePG, PostgreSQL, DocumentDB, FerretDB, MongoDB, Kubernetes, drivers, and related tooling will evolve. + +The research program must therefore support periodic regression runs against newer versions while retaining historical result comparability. + +--- + +# 3. Decision Questions + +The research program is organized around the following decision questions. + +## DQ-01 — Functional Viability + +Can a CloudNativePG PostgreSQL cluster run the selected document engine and expose a stable MongoDB-compatible interface? + +## DQ-02 — Semantic Compatibility + +Which MongoDB behaviors are identical, equivalent, different, unsupported, or incorrect? + +## DQ-03 — Application Compatibility + +Can representative MongoDB applications and official drivers operate without modification? + +## DQ-04 — Document Performance + +How does PostgreSQL-backed document access compare with native MongoDB under common document workloads? + +## DQ-05 — Abstraction Cost + +What performance and resource costs are introduced by: + +- DocumentDB compared with JSONB? +- Mongo-compatible gateways compared with native DocumentDB access? +- DocumentDB Gateway compared with FerretDB? + +## DQ-06 — PostgreSQL Advantage + +Which workloads benefit materially from retaining SQL, relational modeling, PostgreSQL transactions, RLS, and extension compatibility? + +## DQ-07 — Multitenancy + +How do T0–T3 compare in: + +- density +- isolation +- noisy-neighbour behavior +- security +- recoverability +- operational complexity +- cost? + +## DQ-08 — Tenant Mobility + +Can a tenant move between placement models without forcing application-level architectural changes? + +## DQ-09 — Operational Resilience + +How does the PostgreSQL-backed system behave under: + +- failover +- restart +- backup +- restore +- upgrade +- storage pressure +- node loss? + +## DQ-10 — Scaling Boundary + +At what point does MongoDB sharding or another distributed architecture become materially superior to DocStorePG placement-based scaling? + +## DQ-11 — Economic Boundary + +For which tenant and workload profiles does a shared PostgreSQL data plane offer lower total cost than MongoDB or dedicated placements? + +## DQ-12 — Product Boundary + +What should DocStorePG ultimately provide itself, and what should remain upstream configuration? + +--- + +# 4. Research Program Structure + +The research program is divided into ten phases. + +```text +R0 Research Foundation +R1 Functional Baseline +R2 Compatibility & Conformance +R3 Performance Anatomy +R4 PostgreSQL Differentiation +R5 Multitenancy +R6 Tenant Mobility +R7 Operations & Resilience +R8 Scaling & Economics +R9 Long-Term Regression Program +``` + +The phases are ordered. + +Not every experiment inside a phase must complete before later exploratory work begins, but each phase has a **research gate** that should be satisfied before major effort is committed to the next phase. + +--- + +# 5. R0 — Research Foundation + +## Goal + +Create a trustworthy experimental environment. + +## Primary Question + +Can two independently executed benchmark runs be meaningfully compared? + +## Hypotheses + +### H-R0-01 + +A benchmark environment can be fully described from version-controlled configuration plus a small amount of machine-discovered metadata. + +### H-R0-02 + +Repeated runs of the same workload on the same environment will exhibit bounded variance sufficient for architectural comparison. + +### H-R0-03 + +Results can be traced to exact software versions, infrastructure profiles, workload definitions, and repository commits. + +## Required Outputs + +```text +BenchmarkSpecification.md +WorkloadCatalog.md +CompatibilityModel.md +results schema +environment manifest schema +benchmark runner skeleton +MongoDB reference deployment +P0 baseline deployment +P1/P2 deployment skeleton +``` + +## Experiments + +### E-R0-001 — Environment Capture + +Capture: + +- Kubernetes version +- node type +- CPU +- memory +- kernel +- storage class +- volume type +- filesystem +- network configuration +- CNPG version +- PostgreSQL version +- MongoDB version +- DocumentDB version +- gateway version +- driver version +- repository commit + +**Pass condition:** a result can be reproduced without undocumented manual configuration. + +### E-R0-002 — Run Repeatability + +Execute a simple fixed CRUD workload at least 10 times under unchanged conditions. + +Measure: + +- throughput +- p50 +- p95 +- p99 +- CPU +- memory + +Determine variance. + +**Pass condition:** observed variance is characterized and acceptable experimental confidence rules can be defined. + +### E-R0-003 — Resource Contention Baseline + +Run the same benchmark: + +- on an idle cluster +- with unrelated CPU load +- with unrelated storage load + +Purpose: + +Identify how much environmental noise can distort results. + +### E-R0-004 — Result Provenance + +Given any benchmark summary, verify that a reviewer can identify: + +- exact workload +- exact dataset +- exact implementation profile +- exact tenant profile +- exact software versions +- raw result location +- analysis code + +## Research Gate G0 + +Proceed when: + +- environments are reproducible +- results are machine-readable +- variance is understood +- version pinning exists +- M0 and at least P0 can execute the same basic workload + +--- + +# 6. R1 — Functional Baseline + +## Goal + +Demonstrate that the proposed architecture works end to end before deep compatibility analysis. + +## Primary Question + +Can a MongoDB client successfully use a CloudNativePG-backed document store? + +## Hypotheses + +### H-R1-01 + +DocumentDB can run reliably within the selected CloudNativePG deployment model. + +### H-R1-02 + +DocumentDB Gateway can expose basic MongoDB CRUD through the CloudNativePG backend. + +### H-R1-03 + +FerretDB can operate against the same DocumentDB-backed PostgreSQL substrate as an alternative gateway. + +### H-R1-04 + +PostgreSQL-native access remains usable alongside the Mongo-compatible surface. + +## Experiments + +### E-R1-001 — P0 Native JSONB Baseline + +Deploy: + +```text +CloudNativePG ++ PostgreSQL ++ JSONB workload +``` + +Verify: + +- create +- insert +- read +- update +- delete +- index +- backup +- replica startup + +### E-R1-002 — P1 DocumentDB Native + +Deploy: + +```text +CloudNativePG ++ PostgreSQL ++ DocumentDB extensions +``` + +Verify document operations through PostgreSQL-native interfaces. + +### E-R1-003 — P2 Gateway Path + +Deploy DocumentDB Gateway. + +Test with a standard MongoDB client: + +- connect +- create collection +- insert +- find +- update +- delete +- index +- basic aggregation + +### E-R1-004 — P3 FerretDB Path + +Repeat E-R1-003 through FerretDB. + +### E-R1-005 — Concurrent SQL and Mongo Access + +Write through Mongo-compatible API and inspect/query through PostgreSQL. + +Then reverse where safely supported. + +Purpose: + +Confirm the document layer remains part of a visible PostgreSQL platform rather than an opaque silo. + +### E-R1-006 — Restart Survival + +Restart: + +- gateway +- PostgreSQL pod +- replica +- client + +Verify basic recovery. + +## Required Evidence + +For each profile: + +```text +deployment manifest +successful smoke test +known failures +startup time +idle CPU +idle memory +storage footprint +``` + +## Research Gate G1 + +Proceed when: + +- P1 and at least one Mongo-compatible gateway work end to end +- M0 and candidate profiles can execute a shared basic workload +- no unexplained data-corruption or fundamental consistency issue exists +- PostgreSQL-native access has been verified + +--- + +# 7. R2 — Compatibility & Conformance + +## Goal + +Determine how compatible the PostgreSQL-backed document surface actually is. + +## Primary Question + +What does "MongoDB compatible" mean for DocStorePG in practice? + +## Research Model + +Every test case should produce one of: + +```text +IDENTICAL +EQUIVALENT +DOCUMENTED_DIFFERENCE +UNSUPPORTED +INCORRECT +UNKNOWN +``` + +The reference behavior is M0 for MongoDB-specific semantics. + +## Research Clusters + +### C-R2-A — BSON and Type Semantics + +Test: + +- numbers +- strings +- booleans +- null +- missing fields +- arrays +- nested documents +- dates +- ObjectId +- binary +- decimal +- regex +- timestamps +- min/max key where relevant + +### C-R2-B — CRUD + +Test: + +- insertOne +- insertMany +- find +- findOne +- updateOne +- updateMany +- replaceOne +- deleteOne +- deleteMany +- findAndModify family +- bulk operations + +### C-R2-C — Query Operators + +Cover representative families: + +- equality +- comparison +- logical +- array +- element +- evaluation +- text +- expression + +### C-R2-D — Aggregation + +Cover: + +- `$match` +- `$project` +- `$group` +- `$sort` +- `$limit` +- `$skip` +- `$unwind` +- `$lookup` +- `$facet` +- expressions +- nested pipelines + +### C-R2-E — Index Semantics + +Test: + +- unique +- compound +- sparse behavior +- partial behavior +- nested fields +- arrays +- sort use +- index errors +- index lifecycle + +### C-R2-F — Transactions and Sessions + +Test: + +- single-document atomicity +- multi-document transactions +- abort +- commit +- retry +- concurrent writers +- session behavior + +### C-R2-G — Errors + +Compare: + +- duplicate key +- malformed query +- invalid update +- nonexistent namespace +- authorization failure +- transaction conflict +- unsupported operation + +Error compatibility should be evaluated independently from successful-response compatibility. + +### C-R2-H — Drivers + +At minimum investigate current official MongoDB drivers for: + +- Python +- Node.js +- Go +- Java + +Additional languages may be added based on real application demand. + +### C-R2-I — Tooling + +Where practical: + +- `mongosh` +- migration tools +- common ODM/ORM libraries +- backup/export tools +- schema tools + +### C-R2-J — Change Streams + +Determine: + +- availability +- semantics +- resume behavior +- ordering +- failover behavior +- unsupported edge cases + +## Experiments + +### E-R2-001 — Differential Primitive Suite + +Run identical request sets against: + +```text +M0 +P2 +P3 +``` + +Compare structured outcomes. + +### E-R2-002 — Driver Matrix + +Run the same functional suite through each target driver. + +### E-R2-003 — Error Matrix + +Generate controlled invalid operations and compare behavior. + +### E-R2-004 — Aggregation Corpus + +Create a growing corpus of aggregation pipelines ranging from trivial to complex. + +### E-R2-005 — Real Application Smoke Tests + +Select at least three open-source MongoDB-backed applications with distinct characteristics. + +Prefer applications that exercise: + +- normal CRUD +- nested data +- indexes +- aggregation +- transactions or change streams where possible + +Run with no code changes first. + +Classify required changes. + +## Outputs + +```text +CompatibilityModel.md +compatibility matrix +unsupported-operation registry +behavioral-difference registry +driver compatibility matrix +application compatibility reports +``` + +## Research Gate G2 + +Proceed when: + +- basic CRUD/query semantics are characterized +- known incompatibilities are explicit +- at least one realistic application has been exercised +- performance benchmarks can avoid comparing semantically different operations unknowingly + +No specific compatibility percentage is required to proceed. + +The purpose of G2 is **understanding**, not declaring victory. + +--- + +# 8. R3 — Performance Anatomy + +## Goal + +Understand where performance and resource costs originate. + +## Primary Question + +What is the cost of each architectural layer? + +## Core Comparisons + +```text +P0 vs P1 + document-engine cost/value + +P1 vs P2 + DocumentDB Gateway cost + +P1 vs P3 + FerretDB gateway cost + +P2 vs P3 + gateway implementation difference + +M0 vs P2 + MongoDB vs DocumentDB gateway architecture + +M0 vs P3 + MongoDB vs FerretDB architecture +``` + +## Hypotheses + +### H-R3-01 + +Gateway translation overhead is measurable but not dominant for typical networked application workloads. + +### H-R3-02 + +DocumentDB's storage/index model will outperform generic JSONB for some Mongo-shaped queries but may underperform native JSONB for other PostgreSQL-native access patterns. + +### H-R3-03 + +The relative performance ranking changes with document shape, index count, concurrency, and query complexity. + +### H-R3-04 + +Resource efficiency may differ substantially even when latency is similar. + +## Benchmark Dimensions + +### Dataset Size + +Example tiers: + +```text +S1 100k documents +S2 1m documents +S3 10m documents +S4 larger, where infrastructure permits +``` + +### Document Size + +Example distributions: + +```text +small ~1 KB +medium ~10 KB +large ~100 KB +mixed +``` + +### Concurrency + +Example: + +```text +1 +8 +32 +128 +saturation search +``` + +### Index Count + +```text +0 +1 +5 +20 +``` + +### Cache State + +```text +cold +warm +steady-state +``` + +## Experiments + +### E-R3-001 — CRUD Latency Curve + +For W0: + +- insert +- point read +- indexed read +- update +- delete + +Across concurrency levels. + +### E-R3-002 — Document Size Curve + +Measure throughput and tail latency as document size increases. + +### E-R3-003 — Nested Query Cost + +Use W2 with increasing nesting and array sizes. + +### E-R3-004 — Index Write Amplification + +Measure: + +- insert throughput +- update throughput +- index size +- WAL/write volume +- CPU + +as index count rises. + +### E-R3-005 — Aggregation Complexity + +Create graded pipelines: + +```text +A1 simple match/project +A2 match/group +A3 unwind/group +A4 lookup +A5 facet +A6 complex mixed pipeline +``` + +### E-R3-006 — Saturation Point + +Increase concurrency until: + +- latency rises nonlinearly +- throughput plateaus +- errors increase + +Record saturation behavior rather than only peak throughput. + +### E-R3-007 — Gateway Resource Isolation + +Measure gateway CPU/memory separately from PostgreSQL. + +### E-R3-008 — Storage Efficiency + +Compare: + +- raw dataset size +- table/collection size +- index size +- WAL/oplog-related write volume where comparable +- backup size + +### E-R3-009 — Connection Scaling + +Measure: + +- idle connection cost +- active connection cost +- pool behavior +- reconnect storms + +## Analysis Rule + +Do not publish one global "DocStorePG vs MongoDB" performance score. + +Results must remain workload-specific. + +## Research Gate G3 + +Proceed when: + +- performance anatomy is understood for basic workload classes +- layer overhead can be attributed reasonably +- at least one strength and one weakness of the PostgreSQL-backed approach are documented +- test variance remains controlled + +--- + +# 9. R4 — PostgreSQL Differentiation + +## Goal + +Test whether retaining PostgreSQL provides capabilities that justify the architecture beyond MongoDB emulation. + +## Primary Question + +What can DocStorePG do better because it is PostgreSQL? + +## Research Tracks + +### C-R4-A — Hybrid Relational/Document Queries + +Example model: + +```text +customers +contracts +billing +permissions + | + +---- document collections +``` + +Test queries that combine relational predicates with document predicates. + +### C-R4-B — Cross-Model Transactions + +Test transactions touching: + +- relational row +- document +- related audit record + +### C-R4-C — Row-Level Security + +Evaluate whether pooled document access can benefit from PostgreSQL-enforced tenant policy. + +### C-R4-D — SQL Analytics + +Compare operational document access with downstream SQL analysis. + +### C-R4-E — Extension Composition + +Potential future experiments may include compatibility with relevant extensions such as: + +- PostGIS +- pgvector +- full-text/search extensions +- observability extensions + +Only test extensions for which there is a plausible workload. + +## Hypotheses + +### H-R4-01 + +Hybrid relational/document queries can reduce duplication or synchronization between separate relational and document systems. + +### H-R4-02 + +Cross-model transactions provide a meaningful correctness advantage for some applications. + +### H-R4-03 + +Database-enforced tenant policy can provide a stronger pooled-security boundary than application-only tenant filtering. + +### H-R4-04 + +The operational value of one PostgreSQL platform may exceed raw document-query performance differences. + +## Experiments + +### E-R4-001 — Customer/Document Hybrid Query + +Create relational customer/contract data and document-oriented device/application data. + +Query: + +> Find documents for customers satisfying relational commercial conditions and document capability predicates. + +Compare implementation complexity and runtime behavior with M0-based alternatives. + +### E-R4-002 — Cross-Model Atomicity + +Update: + +```text +contract state ++ document state ++ audit row +``` + +in one PostgreSQL transaction. + +Document equivalent patterns required under M0. + +### E-R4-003 — SQL Reporting + +Run analytical queries over mixed relational/document data without ETL. + +### E-R4-004 — RLS Isolation + +Attempt cross-tenant access through: + +- direct SQL +- pooled application connection +- gateway path +- malicious tenant identifier manipulation + +### E-R4-005 — Operational Stack Reduction + +Compare required operational components for: + +```text +PostgreSQL + MongoDB +``` + +versus: + +```text +DocStorePG on PostgreSQL +``` + +Measure: + +- operators +- backup systems +- monitoring systems +- secrets +- upgrade procedures +- runbooks +- expertise requirements + +This is partly qualitative but should use explicit inventories. + +## Research Gate G4 + +Proceed when: + +- at least one PostgreSQL-specific benefit has been demonstrated or disproven +- hybrid architecture value can be discussed with evidence rather than assumption + +If no meaningful PostgreSQL-specific advantage exists, reconsider whether Mongo compatibility on PostgreSQL is worth pursuing beyond migration use cases. + +--- + +# 10. R5 — Multitenancy + +## Goal + +Compare tenant placement strategies and MongoDB multitenancy/sharding approaches. + +## Primary Question + +What is the best continuum between tenant density and tenant sovereignty? + +## Placement Profiles + +```text +T0 POOLED +T1 NAMESPACE +T2 DATABASE +T3 DEDICATED +``` + +MongoDB comparison profiles should distinguish at least: + +```text +MT0 shared collection with tenant key +MT1 database per tenant +MT2 sharded shared deployment +MT3 dedicated deployment +``` + +Exact MongoDB topology definitions belong in `BenchmarkSpecification.md`. + +## Research Dimensions + +### Isolation + +- data +- identity +- compute +- memory +- I/O +- failure +- backup +- maintenance +- configuration + +### Density + +- tenants per database +- tenants per cluster +- idle resource cost +- object/catalog overhead +- connection overhead + +### Operability + +- provisioning +- upgrade +- backup +- restore +- tenant deletion +- credential rotation +- migration + +### Security + +- database-enforced boundary +- gateway enforcement +- application enforcement +- privilege escape risk + +### Performance + +- average latency +- p95/p99 per tenant +- noisy-neighbour effects + +## Experiments + +### E-R5-001 — Tenant Density Curve + +Test: + +```text +10 tenants +100 tenants +1,000 tenants +10,000 tenants +``` + +where technically practical. + +Not every placement mode must reach every tenant count. + +Record failure/scaling boundaries. + +### E-R5-002 — Uniform Tenant Load + +All tenants generate similar load. + +Purpose: + +Establish baseline fairness. + +### E-R5-003 — Whale Tenant + +Canonical distribution: + +```text +many small tenants +some medium tenants +one tenant consuming ~50% of workload +``` + +Measure: + +- small-tenant p99 +- medium-tenant p99 +- whale latency +- CPU share +- I/O share +- connection share + +### E-R5-004 — Bursty Tenant + +One tenant periodically produces extreme short bursts. + +Measure recovery of other tenants after burst completion. + +### E-R5-005 — Tenant Security Boundary + +Attempt: + +- missing tenant filter +- manipulated tenant identifier +- reused connection context +- role escalation +- direct database access +- gateway bypass + +### E-R5-006 — Provisioning Cost + +Measure time and resources to provision: + +```text +new pooled tenant +new schema tenant +new database tenant +new dedicated tenant +``` + +### E-R5-007 — Idle Cost + +Measure per-tenant marginal cost when inactive. + +### E-R5-008 — Backup/Restore Granularity + +Attempt restoration of a single tenant under T0–T3. + +Measure: + +- time +- complexity +- collateral impact + +### E-R5-009 — Failure Blast Radius + +Inject failures and observe affected tenants. + +## SWOT Output + +Each placement model should receive a structured SWOT based on measured results. + +Example categories: + +```text +Strengths +Weaknesses +Opportunities +Threats +Recommended tenant profile +Known scale boundary +``` + +## Research Gate G5 + +Proceed when: + +- T0, T2, and T3 have measured characteristics +- T1 has either been validated or explicitly rejected +- noisy-neighbour behavior is known +- tenant-security enforcement has been exercised +- a first tenant-placement decision matrix can be written + +--- + +# 11. R6 — Tenant Mobility + +## Goal + +Determine whether tenant placement can evolve without redesigning the application. + +## Primary Question + +Can tenant isolation become a runtime/platform decision instead of a permanent application architecture decision? + +## Target Transitions + +Priority order: + +```text +T0 -> T2 +T2 -> T3 +T0 -> T3 +``` + +Optional later: + +```text +T1 -> T2 +T3 -> T2 +T2 -> T0 +``` + +Reverse transitions may be operationally useful but have lower initial priority. + +## Hypotheses + +### H-R6-01 + +A tenant can move from pooled storage to an isolated database while retaining the same logical document API. + +### H-R6-02 + +A high-volume tenant can move to dedicated infrastructure with bounded interruption. + +### H-R6-03 + +Migration complexity depends strongly on change capture and gateway routing capabilities. + +## Migration Correctness Model + +A migration must account for: + +```text +snapshot point +delta changes +validation +cutover +rollback +identity +indexes +metadata +permissions +``` + +## Experiments + +### E-R6-001 — Offline T0 -> T2 + +Stop tenant writes. + +Export. + +Import. + +Validate. + +Switch routing. + +Measure: + +- downtime +- data correctness +- operational steps + +This establishes the simplest baseline. + +### E-R6-002 — Online-ish T0 -> T2 + +Investigate: + +- snapshot +- delta capture +- short cutover + +### E-R6-003 — T2 -> T3 + +Move isolated database tenant to dedicated CNPG deployment. + +### E-R6-004 — Whale Evacuation + +Start E-R5-003 whale scenario. + +Trigger tenant migration when policy threshold is exceeded. + +Measure effect on smaller tenants before, during, and after migration. + +### E-R6-005 — Rollback + +Introduce a target validation failure. + +Return traffic to source safely. + +### E-R6-006 — Endpoint Stability + +Determine whether application connection details can remain unchanged through a stable routing abstraction. + +## Research Gate G6 + +Proceed when: + +- at least one tenant migration path is reliable and documented +- migration correctness can be validated automatically +- downtime and operational cost are measured +- routing requirements are understood + +If tenant mobility is prohibitively complex, treat placement as an initial provisioning decision rather than a dynamic property. + +--- + +# 12. R7 — Operations & Resilience + +## Goal + +Compare operational behavior under realistic failure and lifecycle events. + +## Primary Question + +Does DocStorePG inherit enough operational strength from CloudNativePG to justify the composed architecture? + +## Research Tracks + +### C-R7-A — Failover + +### C-R7-B — Backup and Recovery + +### C-R7-C — Upgrade + +### C-R7-D — Extension Lifecycle + +### C-R7-E — Gateway Lifecycle + +### C-R7-F — Disaster Scenarios + +## Experiments + +### E-R7-001 — Primary Failure + +Terminate PostgreSQL primary. + +Measure: + +- detection time +- promotion time +- Mongo client-visible errors +- SQL client-visible errors +- recovery time +- lost/ambiguous operations +- gateway reconnection behavior + +### E-R7-002 — Gateway Failure + +Terminate gateway instances progressively. + +Measure: + +- availability +- reconnect behavior +- load redistribution + +### E-R7-003 — Node Failure + +Remove Kubernetes node hosting database or gateway workload. + +### E-R7-004 — Planned PostgreSQL Restart + +Measure application-visible disruption. + +### E-R7-005 — Backup + +Measure: + +- duration +- CPU +- I/O +- storage size +- impact on workload latency + +### E-R7-006 — Point-in-Time Recovery + +Create known writes. + +Restore to a target point. + +Validate. + +### E-R7-007 — PostgreSQL Minor Upgrade + +Evaluate normal lifecycle. + +### E-R7-008 — PostgreSQL Major Upgrade + +Evaluate selected supported CNPG path and extension compatibility. + +### E-R7-009 — DocumentDB Extension Upgrade + +Measure: + +- downtime +- compatibility +- rollback options +- replica behavior + +### E-R7-010 — Gateway Upgrade + +Perform rolling upgrade where supported. + +### E-R7-011 — Storage Pressure + +Approach: + +- high disk usage +- reduced IOPS +- delayed storage + +Measure behavior. + +### E-R7-012 — Connection Storm + +Restart many clients or gateway pods simultaneously. + +### E-R7-013 — Recovery Runbook Test + +Have a clean environment execute recovery using only repository documentation. + +Purpose: + +Test operational documentation itself. + +## Comparative Operations Scorecard + +Do not collapse into one score unless weights are explicit. + +Compare: + +```text +availability +RTO +RPO +backup complexity +restore granularity +upgrade complexity +observability +failure transparency +staff/operator burden +``` + +## Research Gate G7 + +Proceed when: + +- common failures have reproducible outcomes +- backup/restore works +- upgrade path is understood +- client-visible behavior is documented +- operational weaknesses are explicit + +--- + +# 13. R8 — Scaling & Economics + +## Goal + +Define where DocStorePG is economically and architecturally preferable, and where MongoDB or another system is better. + +## Primary Question + +Where are the practical boundaries of the architecture? + +## Research Tracks + +### C-R8-A — Vertical Scaling + +Measure performance across compute sizes. + +### C-R8-B — Read Scaling + +Evaluate replica-based reads where semantics permit. + +### C-R8-C — Gateway Scaling + +Increase gateway replicas. + +### C-R8-D — Placement Scaling + +Increase independent DocStorePG deployments and distribute tenants. + +### C-R8-E — MongoDB Sharding Comparison + +Create MongoDB sharded reference scenarios. + +### C-R8-F — Cost Modeling + +Estimate total infrastructure and operational cost. + +## Experiments + +### E-R8-001 — Vertical Efficiency Curve + +For each system/profile: + +```text +small +medium +large +``` + +infrastructure sizes. + +Measure throughput per: + +- CPU +- GB RAM +- storage cost unit + +### E-R8-002 — Read Replica Experiment + +Test read-heavy workload with replicas. + +Validate consistency expectations explicitly. + +### E-R8-003 — Gateway Horizontal Scale + +Increase gateway replicas until database becomes bottleneck. + +### E-R8-004 — Multi-Placement Tenant Fleet + +Distribute tenants across independent CNPG clusters. + +Measure: + +- fleet overhead +- operational complexity +- routing +- cost + +### E-R8-005 — MongoDB Shard Scaling + +Compare a workload where MongoDB's native sharding should be advantaged. + +Purpose: + +Identify a genuine architectural boundary rather than avoiding it. + +### E-R8-006 — Whale Scaling Response + +Compare: + +```text +MongoDB shard placement +vs +DocStorePG tenant evacuation to T3 +``` + +Evaluate: + +- time to isolation +- client transparency +- operational complexity +- steady-state cost + +### E-R8-007 — Cost Per Tenant + +For each tenancy mode: + +```text +monthly infra cost +idle cost +active cost +backup cost +estimated operational cost +``` + +across tenant sizes. + +### E-R8-008 — Cost Per Million Operations + +Normalize infrastructure consumption where meaningful. + +### E-R8-009 — Dual-Database Avoidance Value + +Estimate architecture where an organization would otherwise operate: + +```text +PostgreSQL ++ MongoDB +``` + +versus: + +```text +PostgreSQL / DocStorePG only +``` + +Include: + +- compute +- storage +- backup +- monitoring +- engineering/operator effort +- failure modes +- data synchronization + +This is a model, not a benchmark; assumptions must be explicit. + +## Research Gate G8 + +Complete when a defensible architecture decision guide can answer: + +- use P0 +- use P1 +- use P2/P3 +- use native MongoDB +- use MongoDB sharding +- use dedicated placement +- use another architecture + +for representative workload profiles. + +--- + +# 14. R9 — Long-Term Regression Program + +## Goal + +Keep conclusions valid as upstream systems evolve. + +## Trigger Events + +A regression cycle should be considered when there is a meaningful release of: + +- PostgreSQL +- CloudNativePG +- DocumentDB +- FerretDB +- MongoDB +- Kubernetes +- major MongoDB drivers + +Not every patch release requires full reruns. + +## Regression Tiers + +### L0 — Smoke + +Run: + +- deployment +- connect +- W0 basic CRUD +- backup smoke test + +### L1 — Compatibility + +Run: + +- conformance core +- driver core +- known-difference regression + +### L2 — Performance + +Run: + +- representative CRUD +- aggregation +- index +- hybrid workload +- tenant benchmark + +### L3 — Full Program + +Run major benchmark suite including operations and multitenancy. + +## Regression Goals + +Track: + +```text +compatibility gained +compatibility lost +performance gained +performance lost +resource efficiency changes +operational behavior changes +new capabilities +deprecated assumptions +``` + +--- + +# 15. Research Workstream Map + +Research can proceed through semi-independent workstreams once foundational gates are satisfied. + +```mermaid +flowchart TD + R0[R0 Foundation] + R1[R1 Functional Baseline] + R2[R2 Compatibility] + R3[R3 Performance] + R4[R4 PostgreSQL Differentiation] + R5[R5 Multitenancy] + R6[R6 Tenant Mobility] + R7[R7 Operations] + R8[R8 Scaling & Economics] + R9[R9 Regression] + + R0 --> R1 + R1 --> R2 + R2 --> R3 + R2 --> R4 + R2 --> R5 + + R5 --> R6 + + R3 --> R7 + R5 --> R7 + + R3 --> R8 + R4 --> R8 + R5 --> R8 + R6 --> R8 + R7 --> R8 + + R8 --> R9 +``` + +This means R3, R4, and R5 may overlap after compatibility is sufficiently understood. + +--- + +# 16. Priority Order + +The default priority is: + +```text +P0 Reproducibility +P1 Functional viability +P2 Compatibility +P3 Multitenancy correctness +P4 Performance +P5 Hybrid PostgreSQL advantages +P6 Operations +P7 Tenant mobility +P8 Scaling/economics +P9 long-term optimization +``` + +A key reason multitenancy appears early is security: + +> A pooled architecture should not be performance-optimized before its tenant boundary is understood. + +--- + +# 17. Minimal First Research Cycle + +The first practical research cycle should deliberately be small. + +## Cycle 1 — "Can It Work?" + +Implement: + +```text +M0 +P0 +P1 +P2 +``` + +Run: + +```text +W0 basic CRUD +small W2 nested data +basic W4 aggregation +``` + +Outputs: + +```text +deployment manifests +environment manifest +first compatibility table +first performance table +known issues +``` + +## Cycle 2 — "Is It Really Compatible?" + +Add: + +```text +P3 +driver matrix +error behavior +index semantics +aggregation corpus +``` + +## Cycle 3 — "Does PostgreSQL Buy Us Anything?" + +Add: + +```text +W7 hybrid workload +RLS experiments +cross-model transactions +``` + +## Cycle 4 — "Can We Host Tenants Safely?" + +Add: + +```text +T0 +T2 +T3 +``` + +Run: + +```text +uniform tenants +whale tenant +security tests +backup/restore granularity +``` + +## Cycle 5 — "Can Placement Evolve?" + +Implement: + +```text +T0 -> T2 +T2 -> T3 +``` + +## Cycle 6 — "Where Are the Boundaries?" + +Add: + +```text +HA +upgrade +large datasets +MongoDB sharding +cost models +``` + +This gives the repository a practical path without requiring the whole program to exist before useful findings emerge. + +--- + +# 18. Experiment Definition Standard + +Each experiment should have a Markdown specification. + +Suggested format: + +```markdown +# E-R5-003 Whale Tenant + +## Question + +What effect does one dominant tenant have on other pooled tenants? + +## Hypothesis + +... + +## Profiles + +- P2/T0 +- P3/T0 +- M0/MT0 +- M0/MT2 + +## Environment + +... + +## Dataset + +... + +## Workload + +... + +## Controlled Variables + +... + +## Independent Variable + +... + +## Measurements + +... + +## Procedure + +... + +## Acceptance / Interpretation Rules + +... + +## Results + +... + +## Conclusion + +... + +## Follow-Up +... +``` + +Experiment definitions should be versioned before results are generated when practical. + +--- + +# 19. Evidence Hierarchy + +Not all evidence has equal strength. + +Use this hierarchy: + +```text +E0 Assumption +E1 Documentation evidence +E2 Manual observation +E3 Reproducible functional test +E4 Repeated controlled experiment +E5 Cross-environment replication +E6 Longitudinal production evidence +``` + +Architecture claims should state the strongest evidence level supporting them. + +Example: + +```text +Claim: +P2 supports operation X. + +Evidence: +E3 reproducible conformance test. +``` + +A benchmark finding based on one local run should not be described with production-level certainty. + +--- + +# 20. Result Classification + +Every significant finding should be classified. + +## CONFIRMED + +Evidence supports the hypothesis within the tested boundary. + +## REJECTED + +Evidence contradicts the hypothesis. + +## PARTIAL + +Hypothesis holds only under defined conditions. + +## INCONCLUSIVE + +Evidence is insufficient or contradictory. + +## BLOCKED + +Experiment cannot currently be run because of missing functionality or infrastructure. + +## OBSOLETE + +Upstream changes invalidate the experiment design or finding. + +--- + +# 21. Performance Statistics + +At minimum report: + +```text +operations +duration +throughput +p50 +p95 +p99 +maximum +error count +error rate +CPU +memory +storage growth +``` + +Where relevant: + +```text +WAL volume +network volume +index size +cache hit ratio +replication lag +gateway CPU +gateway memory +``` + +## Confidence + +For important comparisons: + +- repeat runs +- report dispersion +- avoid interpreting very small differences as meaningful +- establish a minimum practical effect size + +The exact statistical method should be defined in `BenchmarkSpecification.md`. + +--- + +# 22. Multitenant Metrics + +Per-tenant metrics are required for W8. + +At minimum: + +```text +tenant throughput +tenant p50 +tenant p95 +tenant p99 +tenant errors +tenant storage +``` + +System-level averages must not hide tenant starvation. + +Useful derived metrics may include: + +```text +latency fairness +throughput fairness +whale impact factor +isolation recovery time +marginal tenant cost +``` + +These should be formally defined before use. + +--- + +# 23. Compatibility Metrics + +A compatibility summary may report counts such as: + +```text +IDENTICAL +EQUIVALENT +DOCUMENTED_DIFFERENCE +UNSUPPORTED +INCORRECT +UNKNOWN +``` + +A percentage may be calculated for navigation, but must never replace the detailed matrix. + +Weighted compatibility scores are allowed only when: + +- weighting is documented +- use case is specified +- unsupported high-impact features remain visible + +--- + +# 24. Operational Metrics + +Operational experiments should record: + +```text +event start +failure detection +service interruption start +recovery start +service usable +full steady state +``` + +Derived: + +```text +detection time +failover time +client interruption +recovery time +RTO +data loss / RPO +ambiguous operation count +``` + +--- + +# 25. Cost Model + +Cost analysis should distinguish: + +## Infrastructure + +- compute +- memory +- block storage +- object storage +- network +- backup retention + +## Platform Overhead + +- Kubernetes +- gateways +- replicas +- idle capacity + +## Operational Overhead + +Estimate separately: + +- deployment +- upgrades +- monitoring +- backup +- recovery +- incident response +- expertise + +Operational estimates must be labeled as modeled assumptions unless measured from actual practice. + +--- + +# 26. Research Environments + +At least three environment classes are desirable over time. + +## DEV + +Purpose: + +- functional work +- rapid iteration +- local/small Kubernetes + +Not authoritative for performance. + +## LAB + +Purpose: + +- controlled benchmarks +- known hardware +- low environmental noise +- repeatability + +Primary environment for published benchmark results. + +## FIELD + +Purpose: + +- realistic long-running workload +- operational evidence +- future production-derived findings + +Not required initially. + +--- + +# 27. Dataset Strategy + +Use multiple dataset families. + +## Synthetic Canonical + +Purpose: + +- controlled variation +- deterministic generation + +Examples: + +- flat documents +- nested documents +- large arrays +- heterogeneous schemas + +## Application-Shaped + +Examples: + +- product catalog +- device/configuration model +- event metadata +- user/profile model +- content objects + +## Adversarial + +Purpose: + +Expose boundaries. + +Examples: + +- extreme nesting +- huge arrays +- large documents +- high-cardinality indexes +- low-cardinality indexes +- heavy update churn + +Synthetic datasets should be generated from seedable code rather than committed as large files. + +--- + +# 28. Workload Strategy + +Each workload should describe: + +```text +read/write ratio +operation distribution +document distribution +query distribution +index assumptions +concurrency +tenant distribution +duration +warmup +think time +retry behavior +``` + +Avoid hidden client behavior that changes results. + +MongoDB driver retries and PostgreSQL client retries should be understood and reported. + +--- + +# 29. Benchmark Anti-Patterns + +The research program should reject the following. + +## AP-01 — Peak TPS Marketing + +Publishing only maximum throughput. + +## AP-02 — Single Run Results + +Treating one benchmark run as definitive. + +## AP-03 — Different Hardware + +Comparing systems on materially different infrastructure without normalization. + +## AP-04 — Semantic Mismatch + +Comparing operations that produce different correctness guarantees. + +## AP-05 — Hidden Index Differences + +One system having more or different indexes than the other. + +## AP-06 — Cache Ambiguity + +Mixing cold- and warm-cache results. + +## AP-07 — Client Bottleneck + +Benchmark runner saturates before database. + +## AP-08 — Average-Only Multitenancy + +Hiding small-tenant degradation behind aggregate throughput. + +## AP-09 — Ignoring Errors + +Counting failed operations as successful throughput or omitting error rate. + +## AP-10 — Unpinned Versions + +Results that cannot be recreated. + +## AP-11 — Hand-Tuned Winner + +Aggressively tuning only one system. + +Tuning stages should be explicit: + +```text +default +reasonable production +expert optimized +``` + +## AP-12 — Benchmark-Only Architecture + +Optimizing implementation specifically for benchmark patterns that are not representative. + +--- + +# 30. Architecture Decision Integration + +Research outcomes should update architecture through ADRs. + +Suggested relationship: + +```text +Experiment + | +Finding + | +Architecture Decision Record + | +ArchitectureBlueprint.md update +``` + +Example: + +```text +E-R5-005 + | +RLS through gateway is unsafe under current connection model + | +ADR-012 Reject T0 gateway RLS strategy B + | +MultitenancyModel.md revised +``` + +This keeps the blueprint evidence-driven. + +--- + +# 31. Research Artifact Map + +The program should gradually produce: + +```text +INTENT.md +ArchitectureBlueprint.md +ResearchProgram.md +BenchmarkSpecification.md +CompatibilityModel.md +MultitenancyModel.md +WorkloadCatalog.md + +docs/ + research/ + findings/ + decisions/ + references/ + +experiments/ + R0/ + R1/ + R2/ + R3/ + R4/ + R5/ + R6/ + R7/ + R8/ + +results/ + manifests/ + summaries/ + reports/ +``` + +Possible future files: + +```text +TenantPlacementDecisionGuide.md +MongoCompatibilityReport.md +OperationsComparison.md +PerformanceReport.md +CostModel.md +ArchitectureDecisionGuide.md +``` + +--- + +# 32. Proposed Experiment Naming + +Use: + +```text +E-R- +``` + +Examples: + +```text +E-R2-001 +E-R5-003 +E-R7-001 +``` + +Sub-experiments may use: + +```text +E-R5-003-A +E-R5-003-B +``` + +Names must remain stable once results are published. + +--- + +# 33. Research Issue Labels + +Suggested repository labels: + +```text +research +experiment +benchmark +compatibility +performance +multitenancy +security +migration +operations +resilience +cost +architecture +upstream +blocked +finding +regression +``` + +Experiment lifecycle labels: + +```text +experiment:proposed +experiment:designed +experiment:ready +experiment:running +experiment:analyzed +experiment:closed +``` + +--- + +# 34. Initial Backlog + +## Foundation + +- [ ] Create `BenchmarkSpecification.md` +- [ ] Create `WorkloadCatalog.md` +- [ ] Create `CompatibilityModel.md` +- [ ] Define result JSON/YAML schema +- [ ] Define environment manifest +- [ ] Implement benchmark runner skeleton + +## Deployment + +- [ ] Deploy M0 MongoDB reference +- [ ] Deploy P0 PostgreSQL JSONB +- [ ] Deploy P1 DocumentDB native +- [ ] Deploy P2 DocumentDB Gateway +- [ ] Deploy P3 FerretDB +- [ ] Pin versions +- [ ] Capture deployment metadata + +## R0 + +- [ ] E-R0-001 Environment Capture +- [ ] E-R0-002 Run Repeatability +- [ ] E-R0-003 Resource Contention Baseline +- [ ] E-R0-004 Result Provenance + +## R1 + +- [ ] E-R1-001 P0 baseline +- [ ] E-R1-002 P1 DocumentDB native +- [ ] E-R1-003 P2 gateway +- [ ] E-R1-004 P3 FerretDB +- [ ] E-R1-005 SQL + Mongo coexistence +- [ ] E-R1-006 restart survival + +## R2 + +- [ ] BSON/type corpus +- [ ] CRUD corpus +- [ ] query corpus +- [ ] aggregation corpus +- [ ] index corpus +- [ ] transaction corpus +- [ ] error corpus +- [ ] driver matrix +- [ ] first real application test + +## R3 + +- [ ] CRUD latency curve +- [ ] document size curve +- [ ] nested query benchmark +- [ ] index amplification benchmark +- [ ] aggregation complexity benchmark +- [ ] saturation benchmark +- [ ] gateway isolation benchmark +- [ ] storage efficiency benchmark + +## R4 + +- [ ] hybrid customer/document workload +- [ ] cross-model transaction test +- [ ] SQL reporting test +- [ ] RLS tenant-security test +- [ ] operational stack inventory + +## R5 + +- [ ] pooled T0 deployment +- [ ] database T2 deployment +- [ ] dedicated T3 deployment +- [ ] assess T1 viability +- [ ] tenant density curve +- [ ] whale benchmark +- [ ] bursty tenant benchmark +- [ ] tenant isolation attack tests +- [ ] provisioning benchmark +- [ ] idle-cost benchmark +- [ ] tenant restore benchmark + +## R6 + +- [ ] T0 -> T2 offline migration +- [ ] T0 -> T2 delta migration +- [ ] T2 -> T3 migration +- [ ] whale evacuation test +- [ ] rollback test +- [ ] stable endpoint experiment + +## R7 + +- [ ] primary failure +- [ ] gateway failure +- [ ] node failure +- [ ] restart +- [ ] backup +- [ ] PITR +- [ ] minor upgrade +- [ ] major upgrade +- [ ] DocumentDB extension upgrade +- [ ] gateway upgrade +- [ ] storage pressure +- [ ] connection storm +- [ ] runbook validation + +## R8 + +- [ ] vertical scale curve +- [ ] read-replica study +- [ ] gateway horizontal scale +- [ ] multi-placement fleet +- [ ] MongoDB sharding reference +- [ ] whale scale response comparison +- [ ] cost-per-tenant model +- [ ] cost-per-million-operations model +- [ ] dual-database avoidance model + +--- + +# 35. First Milestone Definition + +## Milestone M1 — Document Store Baseline + +M1 is achieved when: + +- M0, P0, P1, and P2 deploy reproducibly +- basic W0 workload runs on all applicable profiles +- first differential compatibility results exist +- benchmark results are machine-readable +- environment provenance is complete +- repeated-run variance has been characterized + +M1 should not require performance optimization. + +--- + +# 36. Second Milestone Definition + +## Milestone M2 — Compatibility Characterized + +M2 is achieved when: + +- CRUD compatibility is well understood +- major query operators are classified +- index behavior is classified +- aggregation corpus exists +- at least three official drivers have been tested +- at least one real application has been tested +- P3 has been evaluated +- unsupported behaviors are explicit + +--- + +# 37. Third Milestone Definition + +## Milestone M3 — Architectural Value Demonstrated + +M3 is achieved when: + +- performance anatomy exists +- at least one hybrid relational/document workload has been evaluated +- T0, T2, and T3 exist +- first whale-tenant benchmark exists +- PostgreSQL-specific strengths and weaknesses have evidence + +At M3 the project should explicitly decide whether deeper investment is justified. + +--- + +# 38. Fourth Milestone Definition + +## Milestone M4 — Operationally Credible + +M4 is achieved when: + +- HA failover is tested +- backups are tested +- recovery is tested +- upgrades are tested +- tenant security has been exercised +- one tenant migration path works +- operational runbooks exist + +--- + +# 39. Fifth Milestone Definition + +## Milestone M5 — Architecture Decision Guide + +M5 is achieved when the evidence supports a practical guide answering: + +> For this workload and tenant profile, which architecture should be used? + +The guide should cover at least: + +```text +P0 JSONB +P1 DocumentDB native +P2/P3 Mongo-compatible PostgreSQL +M0 native MongoDB +MongoDB sharded +DocStorePG dedicated placement +``` + +M5 marks transition from exploratory research into long-term engineering/productization decisions. + +--- + +# 40. Stop / Pivot Criteria + +The research program should deliberately support stopping or changing direction. + +## Pivot P1 — Compatibility Layer Only + +Trigger: + +MongoDB compatibility is useful for migration but too incomplete for general runtime compatibility. + +Outcome: + +DocStorePG may become primarily: + +```text +migration +interop +document import/export +PostgreSQL convergence +``` + +## Pivot P2 — JSONB Wins + +Trigger: + +P0 provides equivalent application value with materially lower complexity than P1–P3 for target workloads. + +Outcome: + +Focus on PostgreSQL document patterns rather than MongoDB emulation. + +## Pivot P3 — MongoDB Remains Superior + +Trigger: + +MongoDB demonstrates material advantages in the workloads that actually matter and PostgreSQL-specific benefits do not offset them. + +Outcome: + +DocStorePG becomes a decision/benchmark framework rather than production alternative. + +## Pivot P4 — Multitenancy Becomes Core + +Trigger: + +Tenant placement and isolation prove substantially more valuable than MongoDB compatibility itself. + +Outcome: + +DocStorePG evolves toward a PostgreSQL tenant-placement/control-plane project. + +## Pivot P5 — Upstream Absorbs the Product + +Trigger: + +DocumentDB Operator or another upstream project provides nearly all runtime functionality needed. + +Outcome: + +DocStorePG concentrates on: + +```text +profiles +benchmarks +multitenancy research +conformance +decision support +``` + +rather than duplicating runtime orchestration. + +--- + +# 41. Long-Term Success Criteria + +`doc-store-pg` has long-term research value if it can maintain: + +1. A reproducible reference implementation. +2. A versioned compatibility corpus. +3. A versioned workload catalog. +4. Comparative benchmark history. +5. Multitenancy evidence. +6. Operational resilience evidence. +7. Architecture decision records tied to experiments. +8. A clear account of MongoDB's remaining strengths. +9. A clear account of PostgreSQL-specific strengths. +10. Regression tests that detect upstream improvements or regressions. + +The repository should become more useful as the ecosystem evolves, not obsolete after one benchmark campaign. + +--- + +# 42. Immediate Next Actions + +The recommended execution order from this document is: + +```text +1. BenchmarkSpecification.md +2. WorkloadCatalog.md +3. CompatibilityModel.md +4. M0/P0/P1/P2 reproducible deployments +5. R0 reproducibility experiments +6. R1 smoke tests +7. first R2 compatibility corpus +8. first T0 tenant-security experiment +9. only then broader performance benchmarking +``` + +This order intentionally puts experimental validity and correctness ahead of speed. + +--- + +# 43. Guiding Research Principle + +> Do not ask whether PostgreSQL can replace MongoDB in the abstract. + +Ask instead: + +> For a defined workload, tenant model, operational environment, and compatibility requirement, which architecture produces the best combination of correctness, performance, isolation, flexibility, and cost? + +That is the question `doc-store-pg` should become increasingly good at answering. diff --git a/specs/ArchitectureBlueprint.md b/specs/ArchitectureBlueprint.md new file mode 100755 index 0000000..c6c5f39 --- /dev/null +++ b/specs/ArchitectureBlueprint.md @@ -0,0 +1,1996 @@ +# ArchitectureBlueprint.md + +## Repository + +`doc-store-pg` + +## Status + +Architecture blueprint / research baseline. + +This document defines the initial technical architecture for `doc-store-pg`. It is intentionally a blueprint rather than a frozen implementation specification. The architecture should remain stable enough to support reproducible experimentation while leaving individual components replaceable. + +The central design question is: + +> Can document storage become a composable capability of a CloudNativePG/PostgreSQL platform while preserving MongoDB-compatible application surfaces, strong multitenancy choices, PostgreSQL-native access, and an evidence-driven path to long-term optimization? + +--- + +## 1. Architectural Intent + +`doc-store-pg` should provide a document-oriented data surface on top of PostgreSQL without turning PostgreSQL into an opaque implementation detail. + +The architecture therefore preserves two first-class access paths: + +1. **Document access** through MongoDB-compatible protocols and semantics. +2. **PostgreSQL access** through SQL and PostgreSQL-native tooling. + +The project should prefer composition of upstream technologies over reimplementation. + +The initial architecture is built around: + +- Kubernetes as the runtime substrate +- CloudNativePG as the PostgreSQL lifecycle and HA layer +- PostgreSQL 18+ as the initial database baseline +- DocumentDB PostgreSQL extensions as the primary document engine candidate +- DocumentDB Gateway and FerretDB as interchangeable MongoDB-compatible gateway candidates +- PostgreSQL-native JSONB as a baseline and fallback document representation +- Kubernetes-native configuration and observability +- explicit multitenancy and tenant-placement policies +- reproducible benchmark and conformance workloads + +The architecture is designed so that these choices can be challenged by research rather than becoming permanent assumptions. + +--- + +## 2. Architecture Principles + +### 2.1 PostgreSQL Remains Visible + +Document functionality is added to PostgreSQL; it does not replace the PostgreSQL model. + +Applications or services may intentionally use: + +- MongoDB-compatible document APIs +- SQL +- JSON/JSONB queries +- BSON/document functions +- relational joins +- PostgreSQL transactions +- PostgreSQL extensions + +This enables hybrid relational/document workloads to be treated as a core capability rather than an accidental escape hatch. + +### 2.2 Composition Before Reimplementation + +Existing open-source implementations should be evaluated before new equivalents are written. + +In particular: + +- BSON storage and document semantics should initially come from DocumentDB. +- MongoDB wire-protocol translation should initially come from DocumentDB Gateway or FerretDB. +- PostgreSQL HA, failover, replication, backup integration, and cluster lifecycle should remain CloudNativePG responsibilities. + +`doc-store-pg` should only implement functionality itself where integration, compatibility, policy, observability, or research requirements cannot reasonably be satisfied upstream. + +### 2.3 Gateways Are Replaceable + +MongoDB compatibility is an interface, not a hard dependency on one gateway. + +```text +MongoDB client + | + +--> DocumentDB Gateway --> PostgreSQL + DocumentDB + | + +--> FerretDB -----------> PostgreSQL + DocumentDB +``` + +Additional gateway implementations may later be evaluated using the same conformance and benchmark suites. + +### 2.4 Tenant Placement Is Explicit + +Multitenancy is not represented by a boolean flag. + +A tenant should have an explicit placement/isolation model: + +```text +POOLED +NAMESPACE +DATABASE +DEDICATED +``` + +Future versions may add distributed or federated placement modes. + +### 2.5 Compatibility Is Measured + +"MongoDB compatible" is not considered a binary architectural property. + +Compatibility should be measured separately for: + +- wire protocol +- CRUD operations +- query semantics +- aggregation semantics +- indexing +- drivers +- tools +- transactions +- sessions +- change streams +- errors and edge cases +- application-level behavior + +### 2.6 Evidence Before Optimization + +Architectural decisions that materially affect compatibility, tenancy, performance, or operations should be benchmarkable. + +The system should make it possible to ask: + +> Which configuration is better for this workload and why? + +rather than: + +> Which database do we generally prefer? + +### 2.7 Upstream Escape Hatches Remain Available + +DocStorePG should not create unnecessary lock-in to its own abstractions. + +A deployment should remain understandable through normal: + +- Kubernetes resources +- CloudNativePG resources +- PostgreSQL tools +- DocumentDB tooling +- MongoDB-compatible clients + +The project should add useful abstraction, not hide the underlying systems. + +--- + +# 3. System Context + +```mermaid +flowchart TB + subgraph Clients + A[MongoDB-compatible applications] + B[PostgreSQL / SQL applications] + C[Operations and automation] + D[Benchmark and conformance clients] + end + + subgraph DocStorePG["doc-store-pg capability"] + G[MongoDB-compatible Gateway] + P[PostgreSQL Access] + T[Tenant / Placement Policy] + O[Observability] + X[Benchmark & Conformance Hooks] + end + + subgraph DataPlane["CloudNativePG Data Plane"] + CNPG[CloudNativePG Cluster] + PG[(PostgreSQL)] + DDB[DocumentDB Extensions] + JSONB[Native JSONB] + end + + A --> G + B --> P + C --> T + D --> X + + G --> PG + P --> PG + T --> CNPG + X --> G + X --> P + + CNPG --> PG + PG --- DDB + PG --- JSONB + PG --> O + G --> O +``` + +The important architectural point is that **DocStorePG is not initially a database engine**. + +It is a capability profile formed from: + +```text +PostgreSQL ++ document engine ++ compatible gateway ++ CloudNativePG operations ++ tenancy model ++ policy ++ observability ++ conformance ++ benchmarks +``` + +--- + +# 4. Layer Model + +The initial architecture is divided into seven layers. + +```text +L7 Research & Evidence + benchmarks, conformance, experiments, results + +L6 DocStorePG Control + profiles, tenancy policy, placement, configuration + +L5 Application Interfaces + MongoDB wire protocol, PostgreSQL protocol, admin APIs + +L4 Document Semantics + BSON, document CRUD, query, aggregation, indexing + +L3 PostgreSQL Data Platform + SQL, JSONB, transactions, RLS, extensions + +L2 CloudNativePG Operations + cluster lifecycle, HA, replication, backup hooks, services + +L1 Kubernetes Infrastructure + scheduling, storage, networking, secrets, resources +``` + +A change at one layer should be measurable without requiring unrelated layers to change. + +For example, replacing FerretDB with DocumentDB Gateway should not require changing the PostgreSQL cluster, tenant workload, or benchmark definitions. + +--- + +# 5. Reference Runtime Architecture + +## 5.1 Default Reference Deployment + +```mermaid +flowchart LR + MC[Mongo Client] + SC[SQL Client] + + subgraph GW["Gateway Tier"] + G1[DocumentDB Gateway] + end + + subgraph K8S["Kubernetes"] + subgraph CNPG["CloudNativePG Cluster"] + RW[(Primary)] + R1[(Replica)] + R2[(Replica)] + end + end + + MC -->|MongoDB wire protocol| G1 + G1 -->|PostgreSQL operations| RW + SC -->|PostgreSQL protocol| RW + RW -->|streaming replication| R1 + RW -->|streaming replication| R2 +``` + +The default research deployment should use three PostgreSQL instances when testing HA behavior and may use a single PostgreSQL instance for local functional development. + +The gateway tier should remain separately deployable from PostgreSQL so that: + +- gateway resource consumption can be measured independently +- multiple gateway implementations can be compared +- gateway replicas can scale independently +- gateway failures can be distinguished from database failures +- protocol translation overhead can be isolated + +--- + +## 5.2 PostgreSQL Baseline + +The initial baseline should target PostgreSQL 18 or later. + +This is particularly useful because CloudNativePG's ImageVolume extension mechanism uses PostgreSQL 18's `extension_control_path` support and permits compatible extension images to be mounted dynamically into cluster pods. + +The implementation should pin exact versions in deployment profiles rather than using floating image tags. + +Example conceptual lock information: + +```yaml +platform: + postgresql: "18.x" + cloudnativepg: "" + kubernetes: "" + +extensions: + documentdb: + version: "" + image: "" + +gateway: + implementation: documentdb + version: "" +``` + +The exact versions belong in machine-readable deployment profiles, not permanently in this blueprint. + +--- + +# 6. Extension Architecture + +## 6.1 Primary Document Engine + +DocumentDB is the initial primary candidate because its architecture already separates document functionality into PostgreSQL-oriented components: + +```text +pg_documentdb_core + | + +-- BSON storage and primitives + +pg_documentdb + | + +-- document API + +-- CRUD + +-- queries + +-- indexes + +-- aggregation functionality + +pg_documentdb_gw + | + +-- MongoDB wire protocol + +-- authentication/session gateway behavior +``` + +DocStorePG should treat these as upstream capabilities. + +## 6.2 Extension Packaging + +Preferred path: + +```text +OCI extension image + | + v +CloudNativePG ImageVolume + | + v +PostgreSQL pod + | + v +CREATE/UPDATE EXTENSION +``` + +This should be preferred over building permanently customized PostgreSQL images where the required extension can be safely managed as an image-volume extension. + +A custom PostgreSQL image remains an allowed fallback for: + +- unsupported extension packaging +- experimental builds +- debugging +- compatibility tests +- extension combinations that cannot yet use ImageVolumes + +## 6.3 Extension Lifecycle + +Extension lifecycle must be treated separately from application schema lifecycle. + +Required states: + +```text +available +installed +configured +upgradable +upgraded +rollback-tested +``` + +The benchmark program should eventually include: + +- extension install time +- upgrade behavior +- PostgreSQL major-upgrade interaction +- replica behavior +- recovery behavior +- incompatible-extension failure behavior + +--- + +# 7. Gateway Architecture + +## 7.1 Gateway Contract + +A gateway accepts MongoDB-compatible client traffic and maps it to the PostgreSQL-backed document engine. + +The gateway layer must be independently observable. + +Required gateway metrics should eventually include: + +- request count +- command type +- response latency +- upstream PostgreSQL latency +- translation latency where measurable +- active connections +- connection establishment rate +- rejected connections +- authentication failures +- protocol errors +- unsupported commands +- PostgreSQL errors +- memory +- CPU +- network throughput + +## 7.2 Candidate A — DocumentDB Gateway + +DocumentDB Gateway is the reference path closest to the DocumentDB implementation itself. + +Use it as the initial compatibility baseline. + +## 7.3 Candidate B — FerretDB + +FerretDB 2.x uses PostgreSQL with the DocumentDB extension as its backend and translates MongoDB wire-protocol operations for clients. + +It should be preserved as an alternative gateway because it permits a controlled experiment: + +```text +same client +same workload +same PostgreSQL +same DocumentDB engine +different gateway +``` + +This isolates gateway behavior from storage-engine behavior. + +## 7.4 Gateway Selection Profile + +Conceptual configuration: + +```yaml +gateway: + enabled: true + implementation: documentdb + replicas: 2 + + resources: + profile: standard + + service: + exposure: cluster-internal + + observability: + enabled: true +``` + +Alternative: + +```yaml +gateway: + implementation: ferretdb +``` + +The actual configuration mechanism may initially be Helm/Kustomize values rather than a custom CRD. + +--- + +# 8. Native PostgreSQL Surface + +MongoDB compatibility should never be the only supported path. + +DocStorePG should deliberately preserve a native SQL surface. + +```mermaid +flowchart TD + DOC[(Document data)] + M[MongoDB-compatible API] + S[SQL] + H[Hybrid application] + + M --> DOC + S --> DOC + H --> M + H --> S +``` + +This enables research into workloads such as: + +```sql +SELECT + c.customer_id, + d.document +FROM customers c +JOIN device_documents d + ON d.customer_id = c.customer_id +WHERE c.contract_status = 'active' + AND document_matches_capability(d.document, 'cooling'); +``` + +The precise document functions may differ, but the architectural requirement is that document data remain usable from SQL where the upstream engine supports it. + +This hybrid access mode is one of the central differentiators to test. + +--- + +# 9. Baseline Storage Profiles + +DocStorePG should initially define stable comparable storage profiles. + +## P0 — Native PostgreSQL JSONB + +```text +PostgreSQL ++ JSONB ++ native PostgreSQL indexes ++ no Mongo gateway +``` + +Purpose: + +- establish the minimum PostgreSQL document baseline +- measure what PostgreSQL already provides without a compatibility layer + +## P1 — DocumentDB / Native PostgreSQL Interface + +```text +PostgreSQL ++ DocumentDB extensions ++ direct PostgreSQL access +``` + +Purpose: + +- isolate DocumentDB storage and document semantics +- measure document-engine overhead without Mongo wire translation + +## P2 — DocumentDB Gateway + +```text +Mongo client + | +DocumentDB Gateway + | +PostgreSQL + DocumentDB +``` + +Purpose: + +- primary MongoDB compatibility candidate + +## P3 — FerretDB Gateway + +```text +Mongo client + | +FerretDB + | +PostgreSQL + DocumentDB +``` + +Purpose: + +- alternative compatibility path +- gateway comparison + +## M0 — Native MongoDB Reference + +```text +Mongo client + | +MongoDB +``` + +Purpose: + +- external reference implementation +- semantic compatibility oracle where appropriate +- performance and operational comparator + +These profile identifiers should remain stable so benchmark results can be compared over time. + +--- + +# 10. Multitenancy Architecture + +Multitenancy is a first-class architecture dimension. + +The initial tenant-placement ladder is: + +```text +T0 POOLED +T1 NAMESPACE +T2 DATABASE +T3 DEDICATED +``` + +These identifiers should remain stable in benchmark datasets. + +## 10.1 T0 — Pooled + +Multiple tenants share document structures. + +```text +PostgreSQL database + | + +-- shared document collection/table + | + +-- tenant_id = A + +-- tenant_id = B + +-- tenant_id = C +``` + +Required design goal: + +> Tenant identity should be enforced as deeply as the selected interface permits. + +For PostgreSQL-native access, this should include evaluating Row-Level Security. + +Illustrative pattern: + +```sql +ALTER TABLE documents ENABLE ROW LEVEL SECURITY; + +CREATE POLICY tenant_isolation +ON documents +USING ( + tenant_id = current_setting('docstorepg.tenant_id')::uuid +); +``` + +### Research questions + +- Can tenant identity propagate safely through a MongoDB-compatible gateway? +- Does the document engine expose sufficient hooks for RLS-based enforcement? +- Can a gateway accidentally bypass RLS through privileged database roles? +- How can connection pooling safely preserve tenant context? +- What performance cost does RLS introduce? +- Can tenant-aware compound indexes prevent cross-tenant hot spots? +- How do backup and restore work for a single pooled tenant? + +## 10.2 T1 — Namespace + +Tenants receive PostgreSQL schemas or another logically separate namespace. + +```text +cluster + | + +-- database + | + +-- tenant_a schema + +-- tenant_b schema + +-- tenant_c schema +``` + +Advantages to evaluate: + +- stronger logical separation than pooled rows +- separate objects and indexes +- tenant-specific schema customization +- relatively low compute overhead compared with dedicated clusters + +Risks to evaluate: + +- object proliferation +- PostgreSQL catalog growth +- migration complexity +- gateway mapping complexity +- uncertain fit with document-engine assumptions +- backup/restore granularity + +T1 must be treated as an experiment until compatibility with the selected document engine and gateway is proven. + +## 10.3 T2 — Database + +Each tenant receives a separate PostgreSQL database inside one CloudNativePG cluster. + +```text +CloudNativePG cluster + | + +-- tenant_a database + +-- tenant_b database + +-- tenant_c database +``` + +CloudNativePG can declaratively manage additional databases and their extensions/schemas. + +This mode should evaluate: + +- per-tenant ownership +- per-database extensions +- migration lifecycle +- database count scaling +- connection overhead +- catalog/resource overhead +- backup/restore behavior +- gateway routing +- tenant-specific configuration +- noisy-neighbour effects + +The compute and storage failure domain remains shared. + +## 10.4 T3 — Dedicated + +Each tenant receives a dedicated CloudNativePG-backed DocStorePG deployment. + +```mermaid +flowchart LR + A[Tenant A] --> CA[DocStorePG Cluster A] + B[Tenant B] --> CB[DocStorePG Cluster B] + C[Tenant C] --> CC[DocStorePG Cluster C] +``` + +This mode provides the strongest initial isolation boundary. + +Potential isolation includes: + +- PostgreSQL process +- PostgreSQL database +- gateway +- CPU request/limit +- memory request/limit +- persistent storage +- Kubernetes scheduling +- secrets +- service identity +- network policy +- backup policy +- failure domain + +The primary downside is fleet overhead. + +T3 is expected to be most relevant to: + +- large enterprise tenants +- regulated workloads +- tenants requiring special extensions +- tenants requiring independent maintenance windows +- high-volume "whale" tenants +- contractual isolation requirements + +--- + +# 11. Tenant Placement as a Lifecycle + +The important abstraction is not merely selecting one of T0–T3. + +The longer-term architecture should explore tenant mobility: + +```mermaid +stateDiagram-v2 + [*] --> Pooled + Pooled --> Namespace + Pooled --> Database + Namespace --> Database + Database --> Dedicated + Pooled --> Dedicated + + Dedicated --> Database + Database --> Pooled +``` + +Not all transitions need to be supported initially. + +The strategic goal is: + +> Keep the application-facing document contract as stable as possible while tenant placement changes underneath it. + +This is DocStorePG's initial conceptual answer to some of the problems that MongoDB solves with shard placement. + +It is not initially intended to reproduce MongoDB sharding. + +--- + +# 12. Explicit Sharding Boundary + +MongoDB provides native horizontal sharding in which data can be distributed across shards and routed through a unified cluster interface. + +CloudNativePG's normal architecture is a PostgreSQL primary/standby cluster. It is not by itself a transparent horizontally sharded write architecture. + +Therefore: + +> DocStorePG V1 must not claim MongoDB-equivalent horizontal sharding. + +The initial architecture deliberately stops here: + +```text +single DocStorePG deployment + | + +-- primary + +-- replicas +``` + +and permits multiple independent placements: + +```text +DocStorePG A +DocStorePG B +DocStorePG C +``` + +A future research track may investigate a placement/router layer: + +```mermaid +flowchart TD + APP[Application] + R[Tenant Router] + + A[DocStorePG Placement A] + B[DocStorePG Placement B] + C[DocStorePG Placement C] + + APP --> R + R --> A + R --> B + R --> C +``` + +That work should only begin once there is evidence that it solves a problem not already better solved by: + +- MongoDB sharding +- Citus or another PostgreSQL distribution technology +- application-level partitioning +- independent tenant clusters +- another existing distributed database + +--- + +# 13. Proposed DocStorePG Control Surface + +DocStorePG should not immediately create a Kubernetes operator. + +The first implementation should prefer: + +```text +Helm / Kustomize ++ CloudNativePG CRDs ++ standard Kubernetes resources ++ benchmark profiles ++ explicit configuration files +``` + +Only after recurring orchestration requirements emerge should a DocStorePG CRD/operator be justified. + +A possible future API is documented here to establish architectural vocabulary. + +## 13.1 Future `DocStore` Resource + +```yaml +apiVersion: docstorepg.io/v1alpha1 +kind: DocStore +metadata: + name: application-documents + +spec: + postgres: + profile: ha-standard + + documentEngine: + implementation: documentdb + + gateway: + implementation: documentdb + replicas: 2 + + interfaces: + mongodb: true + postgresql: true + + tenancy: + defaultPlacement: pooled + + observability: + enabled: true +``` + +## 13.2 Future `DocStoreTenant` Resource + +```yaml +apiVersion: docstorepg.io/v1alpha1 +kind: DocStoreTenant +metadata: + name: acme + +spec: + storeRef: application-documents + + placement: + mode: database + + resources: + class: standard + + compatibility: + profile: mongodb-default +``` + +These resources are **design sketches**, not implementation commitments. + +--- + +# 14. Configuration Profiles + +Configuration should be grouped into composable profiles. + +```text +profiles/ +├── platform/ +│ ├── local-single.yaml +│ ├── ha-small.yaml +│ └── ha-standard.yaml +├── engine/ +│ ├── jsonb.yaml +│ └── documentdb.yaml +├── gateway/ +│ ├── none.yaml +│ ├── documentdb.yaml +│ └── ferretdb.yaml +├── tenancy/ +│ ├── pooled.yaml +│ ├── namespace.yaml +│ ├── database.yaml +│ └── dedicated.yaml +└── workload/ + ├── document-crud.yaml + ├── aggregation.yaml + └── hybrid.yaml +``` + +A benchmark environment can then be expressed as composition: + +```text +platform/ha-small ++ engine/documentdb ++ gateway/ferretdb ++ tenancy/pooled ++ workload/document-crud +``` + +This makes experiments easier to reproduce and diff. + +--- + +# 15. Security Architecture + +The security model should be layered. + +```text +Kubernetes identity and RBAC + | +NetworkPolicy / service exposure + | +gateway authentication + | +PostgreSQL authentication + | +PostgreSQL role / database / schema privileges + | +Row-Level Security where applicable + | +document/application authorization +``` + +## 15.1 Security Rules + +1. Gateway database credentials should not use PostgreSQL superuser privileges in production profiles. +2. `BYPASSRLS` roles must not be used for pooled tenant traffic where RLS is part of the tenant boundary. +3. Tenant context must be explicit and auditable. +4. Connection pooling must not leak tenant context between sessions. +5. PostgreSQL direct access must not silently bypass the intended Mongo-facing security model. +6. Mongo-compatible and SQL-facing identities must be mapped deliberately. +7. Secrets should be provided through Kubernetes-native secret mechanisms or external secret-management integration. +8. External service exposure should be opt-in. +9. NetworkPolicy should be part of production reference profiles. +10. Security tests belong in the conformance suite. + +--- + +# 16. Identity Propagation + +Tenant-aware pooled operation creates an important architecture problem: + +```text +MongoDB client identity + | + v +Mongo gateway + | + v +PostgreSQL session identity + | + v +RLS / database policy +``` + +DocStorePG should explicitly test multiple strategies. + +### Strategy A — Database Role Mapping + +One PostgreSQL role per tenant or tenant class. + +Potential strength: + +- database-native identity + +Potential weakness: + +- large role counts +- connection-pool complexity + +### Strategy B — Session Tenant Context + +A trusted gateway sets a PostgreSQL session variable such as: + +```sql +SET docstorepg.tenant_id = '...'; +``` + +RLS reads that context. + +Potential strength: + +- scalable logical identity + +Potential weakness: + +- gateway becomes security-critical +- session reset must be flawless + +### Strategy C — Database-per-Tenant Routing + +Gateway selects the tenant database. + +Potential strength: + +- stronger logical boundary + +Potential weakness: + +- routing and connection-pool fragmentation + +No strategy should be declared canonical until tested. + +--- + +# 17. Network Architecture + +Default production stance: + +```text +Mongo endpoint: cluster-internal +PostgreSQL endpoint: cluster-internal +admin endpoints: cluster-internal +metrics endpoints: monitoring namespace / authorized collectors +external exposure: explicit profile only +``` + +Conceptual Kubernetes services: + +```text +docstore-mongo-rw +docstore-postgres-rw +docstore-postgres-ro +docstore-metrics +``` + +The exact service structure should follow the selected upstream gateway and CloudNativePG conventions. + +--- + +# 18. High Availability + +CloudNativePG should own PostgreSQL HA. + +```mermaid +flowchart LR + GW[Gateway replicas] + SVC[CNPG RW Service] + P[(Primary)] + R1[(Replica)] + R2[(Replica)] + + GW --> SVC + SVC --> P + P --> R1 + P --> R2 +``` + +DocStorePG should test gateway behavior during PostgreSQL failover. + +Important questions: + +- Does the gateway reconnect automatically? +- What do clients observe? +- Which operations fail? +- Are retries safe? +- How do transactions behave? +- How quickly does service recover? +- Does MongoDB client retry behavior interact correctly with PostgreSQL failover? +- Does the gateway retain stale connections? + +Failover testing belongs in the benchmark program. + +--- + +# 19. Read Scaling + +CloudNativePG provides read-only service patterns through replicas. + +DocStorePG must not assume MongoDB-style read preferences map automatically to PostgreSQL read replicas. + +Possible future mapping: + +```text +Mongo read preference + | + v +Gateway policy + | + +--> primary + | + +--> CNPG read-only service +``` + +This requires explicit semantic research around: + +- consistency +- replication lag +- transaction semantics +- session guarantees +- MongoDB driver expectations +- failover + +Until proven, primary routing should be the correctness baseline. + +--- + +# 20. Backup and Recovery + +Backup architecture must distinguish between: + +```text +cluster recovery +database recovery +tenant recovery +collection recovery +document recovery +``` + +CloudNativePG provides cluster-oriented PostgreSQL backup/recovery mechanisms; the DocumentDB Kubernetes Operator also has its own evolving backup abstractions. + +DocStorePG should not assume that cluster-level backup automatically satisfies tenant-level recovery requirements. + +Research must measure: + +- backup duration +- backup size +- restore duration +- point-in-time recovery +- tenant-specific restore +- restoring one tenant from a pooled dataset +- restoring database-per-tenant +- dedicated-cluster restore +- Mongo-compatible application recovery behavior + +Tenant recoverability is an important factor in choosing T0–T3. + +--- + +# 21. Observability Architecture + +Observability should expose each layer independently. + +```mermaid +flowchart LR + C[Client] + G[Gateway] + P[PostgreSQL] + K[Kubernetes] + B[Benchmark Harness] + + C --> G --> P + G --> M[Metrics] + P --> M + K --> M + B --> M + + G --> L[Logs] + P --> L + K --> L + + M --> O[Observability Backend] + L --> O +``` + +Minimum metric categories: + +### PostgreSQL + +- transactions +- query latency +- query count +- locks +- connections +- cache hit behavior +- WAL generation +- checkpoint behavior +- replication lag +- table/index size +- I/O +- CPU +- memory + +### Gateway + +- operations +- command types +- latency +- failures +- unsupported requests +- active connections +- authentication +- CPU +- memory + +### Kubernetes + +- pod restarts +- scheduling +- CPU requests/usage +- memory requests/usage +- storage +- network +- failovers +- PVC behavior + +### Tenant + +Where possible: + +- requests per tenant +- storage per tenant +- latency per tenant +- error rate per tenant +- resource share +- noisy-neighbour indicators + +### Benchmark + +Every benchmark run must capture enough environment metadata to explain the result. + +--- + +# 22. Benchmark and Conformance Plane + +The benchmark plane is part of the architecture, not an external one-off tool. + +```mermaid +flowchart TB + W[Workload Catalog] + R[Runner] + M0[MongoDB Reference] + P0[PostgreSQL JSONB] + P1[DocumentDB Native] + P2[DocumentDB Gateway] + P3[FerretDB] + C[Comparator] + E[(Evidence Store)] + + W --> R + + R --> M0 + R --> P0 + R --> P1 + R --> P2 + R --> P3 + + M0 --> C + P0 --> C + P1 --> C + P2 --> C + P3 --> C + + C --> E +``` + +Benchmark code should support deterministic seeds where possible. + +Every result should include: + +```yaml +run: + id: ... + timestamp: ... + gitCommit: ... + workload: ... + seed: ... + +environment: + kubernetes: ... + nodes: ... + cpu: ... + memory: ... + storageClass: ... + +software: + mongodb: ... + postgresql: ... + cloudnativepg: ... + documentdb: ... + gateway: ... + +profile: + storage: P2 + tenancy: T0 + +dataset: + tenants: ... + documents: ... + averageDocumentSize: ... + +result: + throughput: ... + latencyP50: ... + latencyP95: ... + latencyP99: ... + errors: ... +``` + +This structure should evolve into a formal benchmark schema. + +--- + +# 23. Differential Conformance Testing + +For compatible operations, DocStorePG should support differential testing. + +```text +request + | + +------> MongoDB ----------> result A + | + +------> DocStorePG -------> result B + | + v + semantic comparator +``` + +Comparison dimensions: + +- returned data +- BSON types +- null/missing behavior +- ordering +- update results +- aggregation results +- errors +- error categories/codes +- transaction outcome +- session behavior +- index behavior + +Differences should be classified as: + +```text +IDENTICAL +EQUIVALENT +DOCUMENTED_DIFFERENCE +UNSUPPORTED +INCORRECT +UNKNOWN +``` + +A compatibility score must not hide important individual differences. + +--- + +# 24. Workload Classes + +The architecture should support at least these benchmark workload classes. + +## W0 — Basic CRUD + +- insert +- find by ID +- update +- delete +- bulk operations + +## W1 — Flexible Schema + +- optional fields +- heterogeneous structures +- schema evolution +- large/small documents + +## W2 — Nested Documents + +- nested objects +- arrays +- array element updates +- nested predicates + +## W3 — Indexing + +- single-field +- compound +- nested +- wildcard/dynamic-field equivalents +- low/high cardinality +- many-index write cost + +## W4 — Aggregation + +- filtering +- projection +- grouping +- sorting +- unwind +- lookup/join-like behavior +- facets + +## W5 — Transactions + +- single-document +- multi-document +- concurrent updates +- abort/retry behavior + +## W6 — Change/Event Workloads + +- change streams or equivalents +- consumer lag +- reconnect behavior +- failover behavior + +## W7 — Hybrid Relational/Document + +- relational filters over document sets +- joins between relational and document data +- cross-model transactions +- reporting/analytics + +## W8 — Multitenancy + +- many small tenants +- mixed tenant sizes +- tenant provisioning +- whale tenant +- noisy neighbours +- placement changes +- isolation tests + +## W9 — Operations + +- failover +- backup +- restore +- upgrade +- extension upgrade +- scaling +- disaster simulation + +--- + +# 25. Whale-Tenant Experiment + +One canonical multitenancy experiment should be designed into the architecture from the start. + +Example distribution: + +```text +1000 tenants total + +950 small tenants +49 medium tenants +1 whale tenant consuming roughly half of traffic +``` + +The exact distribution should be parameterized. + +Questions: + +- What happens to small-tenant p99 latency? +- Can the whale be identified operationally? +- Can the whale move from T0 to T2/T3? +- How long does migration take? +- How much downtime occurs? +- Does the application connection contract change? +- What happens to backup cost? +- At what point does dedicated placement become economically sensible? + +This should eventually become a signature DocStorePG benchmark. + +--- + +# 26. Tenant Migration Architecture + +Tenant migration is initially an experimental subsystem. + +```mermaid +flowchart LR + S[Source Placement] + E[Export / Replication] + V[Validate] + C[Cutover] + T[Target Placement] + R[Rollback Window] + + S --> E --> V --> C --> T + C --> R +``` + +Desired migration properties: + +- tenant-scoped +- auditable +- resumable +- consistency-aware +- verifiable +- rollback-aware +- application-transparent where possible + +Potential mechanisms to investigate: + +- logical export/import +- PostgreSQL logical replication +- document API replication +- change streams +- dual writes +- maintenance-window cutover +- backup/restore extraction + +No mechanism is canonical yet. + +--- + +# 27. DocumentDB Kubernetes Operator Relationship + +The DocumentDB Kubernetes Operator is a relevant upstream project and should be actively compared with DocStorePG. + +It already uses CloudNativePG underneath and provides Kubernetes-native DocumentDB lifecycle capabilities. + +DocStorePG should therefore avoid duplicating its implementation without evidence. + +Initial relationship: + +```text +DocumentDB Operator + | + +-- upstream reference implementation + +-- optional deployment profile + +-- research comparator + +-- source of operational lessons + +DocStorePG + | + +-- broader PostgreSQL document capability research + +-- interchangeable gateways + +-- native JSONB baseline + +-- multitenancy comparison + +-- MongoDB comparison + +-- conformance framework + +-- benchmark framework + +-- hybrid relational/document research +``` + +If the DocumentDB Operator eventually provides the best runtime deployment path, DocStorePG should be able to adopt it rather than compete with it. + +--- + +# 28. Failure Domains + +DocStorePG should model failure domains explicitly. + +| Failure | Likely scope | Expected owner | +|---|---|---| +| Gateway pod | Mongo API traffic through pod | Kubernetes / gateway | +| Gateway deployment | Mongo-compatible surface | DocStorePG deployment | +| PostgreSQL instance | instance | CloudNativePG | +| PostgreSQL primary | writes until failover | CloudNativePG | +| PostgreSQL cluster | tenant set on cluster | operations | +| PVC/storage | instance or cluster | Kubernetes/storage | +| Namespace | contained deployments | Kubernetes | +| Kubernetes node | scheduled workloads | Kubernetes | +| Kubernetes cluster | all contained placements | infrastructure | +| Tenant credential leak | tenant/security boundary | security model | +| RLS policy error | pooled tenants | DocStorePG policy | +| Extension defect | databases using extension | upstream + operations | + +Dedicated tenant placement should reduce blast radius at the cost of operational density. + +--- + +# 29. Scaling Model + +DocStorePG should distinguish four fundamentally different scaling mechanisms. + +## Vertical + +Increase: + +- CPU +- memory +- storage performance + +## Read Scale + +Add PostgreSQL replicas where semantics permit. + +## Gateway Scale + +Increase stateless gateway replicas. + +## Placement Scale + +Move tenants or workloads onto additional independent DocStorePG deployments. + +```text + scale + | + +-----------------+----------------+ + | | | + vertical gateway placement + | + +--------+--------+ + | | + shared dedicated +``` + +Transparent horizontal write sharding is explicitly outside the initial model. + +--- + +# 30. Repository Architecture + +Proposed repository structure: + +```text +doc-store-pg/ +├── INTENT.md +├── ArchitectureBlueprint.md +├── ResearchProgram.md +├── BenchmarkSpecification.md +├── CompatibilityModel.md +├── MultitenancyModel.md +├── WorkloadCatalog.md +├── README.md +│ +├── docs/ +│ ├── architecture/ +│ ├── decisions/ +│ ├── findings/ +│ └── references/ +│ +├── deploy/ +│ ├── cnpg/ +│ ├── documentdb/ +│ ├── gateways/ +│ │ ├── documentdb/ +│ │ └── ferretdb/ +│ └── mongodb-reference/ +│ +├── profiles/ +│ ├── platform/ +│ ├── engine/ +│ ├── gateway/ +│ ├── tenancy/ +│ └── workload/ +│ +├── experiments/ +│ ├── compatibility/ +│ ├── performance/ +│ ├── multitenancy/ +│ ├── operations/ +│ └── migration/ +│ +├── benchmarks/ +│ ├── runner/ +│ ├── workloads/ +│ ├── schemas/ +│ └── analysis/ +│ +├── tests/ +│ ├── conformance/ +│ ├── security/ +│ ├── failover/ +│ └── integration/ +│ +├── results/ +│ └── README.md +│ +└── scripts/ +``` + +Generated benchmark results should be governed carefully so the repository does not become filled with large ephemeral datasets. + +Machine-readable summaries belong in Git where useful; raw large datasets may require external object storage with manifests/checksums committed to the repository. + +--- + +# 31. Initial Implementation Stages + +## Stage A — Functional Baseline + +Establish: + +```text +Kubernetes ++ CloudNativePG ++ PostgreSQL +``` + +Verify normal SQL operation. + +## Stage B — Document Engine + +Add: + +```text +DocumentDB extensions +``` + +Verify document operations through PostgreSQL-native interfaces. + +## Stage C — Mongo-Compatible Surface + +Add: + +```text +DocumentDB Gateway +``` + +Run basic official-driver CRUD tests. + +## Stage D — Alternative Gateway + +Add: + +```text +FerretDB +``` + +Run the same tests against the same backend profile. + +## Stage E — MongoDB Reference + +Deploy native MongoDB as M0. + +Run differential CRUD/query tests. + +## Stage F — Tenant Baseline + +Implement T0 pooled tenancy and evaluate PostgreSQL-enforced isolation. + +## Stage G — Isolation Ladder + +Add T2 database-per-tenant and T3 dedicated placement. + +Treat T1 schema-per-tenant as experimental and implement only if document-engine behavior supports it cleanly. + +## Stage H — Benchmark Plane + +Add machine-readable benchmark runs and environment manifests. + +## Stage I — Operational Experiments + +Test: + +- failover +- restart +- backup +- restore +- upgrade +- noisy neighbour +- tenant movement + +--- + +# 32. Initial Architecture Decisions + +The following decisions are provisional but form the starting baseline. + +### AD-001 — CloudNativePG is the PostgreSQL operator + +**Decision:** Use CloudNativePG rather than introducing a custom PostgreSQL lifecycle implementation. + +**Reason:** PostgreSQL lifecycle and HA are not DocStorePG's differentiated problem. + +### AD-002 — DocumentDB is the initial document engine + +**Decision:** Use DocumentDB extensions as the first BSON/document compatibility implementation. + +**Reason:** It already implements document semantics on PostgreSQL and exposes a MongoDB-compatible gateway architecture. + +### AD-003 — Gateway implementation remains swappable + +**Decision:** Support DocumentDB Gateway first; preserve FerretDB as an alternative profile. + +**Reason:** This allows protocol-layer effects to be isolated experimentally. + +### AD-004 — PostgreSQL access remains first-class + +**Decision:** Do not hide SQL/PostgreSQL interfaces behind the document API. + +**Reason:** Hybrid relational/document capability is a core research hypothesis. + +### AD-005 — Multitenancy has explicit placement levels + +**Decision:** Use T0–T3 as stable architectural vocabulary. + +**Reason:** Isolation, cost, and scaling cannot be meaningfully represented by a single "multi-tenant" mode. + +### AD-006 — No custom DocStorePG operator initially + +**Decision:** Start with manifests/profiles and existing operators. + +**Reason:** Operator behavior should only be built once recurring control-plane requirements are empirically understood. + +### AD-007 — No claim of transparent write sharding + +**Decision:** Treat MongoDB sharding as a distinct comparison point rather than something DocStorePG already provides. + +**Reason:** CloudNativePG HA and MongoDB sharding solve different problems. + +### AD-008 — Benchmarks are architecture + +**Decision:** Reproducible benchmark and conformance hooks are part of the system design from the beginning. + +**Reason:** Long-term architecture choices must remain evidence-driven. + +--- + +# 33. Architecture Questions to Resolve Through Research + +The blueprint intentionally leaves these open: + +1. How complete is DocumentDB's real MongoDB semantic compatibility? +2. Does DocumentDB Gateway or FerretDB provide the better compatibility/performance trade-off? +3. How should Mongo identities map to PostgreSQL identities? +4. Can PostgreSQL RLS reliably enforce pooled tenant isolation through the Mongo-compatible surface? +5. What tenant count is practical for database-per-tenant within one CNPG cluster? +6. Is schema-per-tenant useful with DocumentDB or merely theoretical? +7. How much gateway overhead exists relative to direct document access? +8. How much document-engine overhead exists relative to native JSONB? +9. Where does native MongoDB outperform the PostgreSQL-backed architecture? +10. How do both architectures behave with a whale tenant? +11. What is the operational cost per tenant under T0, T2, and T3? +12. How should tenant migration work? +13. Can a stable logical endpoint survive tenant placement changes? +14. Which MongoDB read/write/session semantics can safely map to CNPG primary/replica topology? +15. What backup granularity is achievable for each tenant model? +16. How should extension versions be upgraded independently from PostgreSQL? +17. When does an existing distributed PostgreSQL technology become preferable to DocStorePG placement routing? +18. When should a workload simply remain on native MongoDB? + +These questions should seed `ResearchProgram.md`. + +--- + +# 34. Target Architecture + +The long-term target, if supported by evidence, is: + +```mermaid +flowchart TB + subgraph Applications + M[MongoDB clients] + S[SQL clients] + end + + EP[Stable DocStorePG Endpoint] + TC[Tenant / Placement Control] + + subgraph Shared["Shared Placement"] + SP[CNPG + DocumentDB] + end + + subgraph DB["Database-Isolated Placement"] + DP[CNPG + Tenant Databases] + end + + subgraph Dedicated["Dedicated Placements"] + D1[CNPG Tenant A] + D2[CNPG Tenant B] + end + + OBS[Observability] + BENCH[Conformance & Benchmark Plane] + + M --> EP + S --> EP + + EP --> TC + TC --> SP + TC --> DP + TC --> D1 + TC --> D2 + + SP --> OBS + DP --> OBS + D1 --> OBS + D2 --> OBS + + BENCH --> EP + BENCH --> OBS +``` + +The important feature is not that all boxes necessarily become custom DocStorePG software. + +The important feature is that the platform can express and test: + +```text +interface ++ document semantics ++ tenant identity ++ placement ++ isolation ++ PostgreSQL capability ++ operations ++ evidence +``` + +as separable concerns. + +--- + +# 35. Architectural North Star + +`doc-store-pg` should aim to make this statement testable: + +> A document store can be an operationally native PostgreSQL capability whose API, compatibility, tenant placement, and isolation level can evolve independently from the applications that use it. + +The architecture succeeds if it allows the project to discover where that statement is true, where it is false, and where MongoDB or another architecture remains the better engineering choice. + +The purpose is not to prove PostgreSQL wins. + +The purpose is to create a system in which the trade-offs become measurable. + +--- + +# 36. References and Upstream Baseline + +The following upstream sources establish the initial technical baseline for this blueprint. Versions and capabilities should be revalidated when experiments are run. + +1. **CloudNativePG — Image Volume Extensions** + https://cloudnative-pg.io/documentation/current/imagevolume_extensions/ + +2. **CloudNativePG — Operator Capability Levels** + https://cloudnative-pg.io/docs/current/operator_capability_levels/ + +3. **CloudNativePG — Declarative Database Management** + https://cloudnative-pg.io/docs/devel/declarative_database_management/ + +4. **CloudNativePG 1.30 Release** + https://cloudnative-pg.io/releases/cloudnative-pg-1-30.0-released/ + +5. **DocumentDB — Getting Started / Architecture Components** + https://documentdb.io/docs/getting-started + +6. **DocumentDB — PostgreSQL API / Gateway** + https://documentdb.io/docs/postgres-api + +7. **DocumentDB Kubernetes Operator** + https://documentdb.io/kubernetes-operator + +8. **DocumentDB Kubernetes Operator — API Reference** + https://documentdb.io/documentdb-kubernetes-operator/latest/preview/api-reference/ + +9. **FerretDB — Introduction** + https://docs.ferretdb.io/ + +10. **FerretDB — Migration to v2 / DocumentDB Backend** + https://docs.ferretdb.io/migration/migrating-from-v1/ + +11. **PostgreSQL — Row Security Policies** + https://www.postgresql.org/docs/current/ddl-rowsecurity.html + +12. **MongoDB — Multi-Tenant Architecture** + https://www.mongodb.com/docs/atlas/build-multi-tenant-arch/ + +13. **MongoDB — Sharding** + https://www.mongodb.com/docs/manual/sharding/ + +14. **MongoDB — Scaling Strategies / Collection Placement** + https://www.mongodb.com/docs/manual/core/sharding-scaling-strategies/ + +15. **MongoDB — Manage Unsharded Collections** + https://www.mongodb.com/docs/manual/core/sharding-manage-unsharded-collections/