From 488f029075e83aebb4c2661a328828c7dea7628a Mon Sep 17 00:00:00 2001 From: tegwick Date: Mon, 24 Aug 2026 01:16:55 +0200 Subject: [PATCH] Added search as additional facility --- .../DocStoreResearchProgram.md | 0 program/SearchCapabilitiesResearchProgram.md | 1691 +++++++++++++++++ 2 files changed, 1691 insertions(+) rename ResearchProgram.md => program/DocStoreResearchProgram.md (100%) create mode 100755 program/SearchCapabilitiesResearchProgram.md diff --git a/ResearchProgram.md b/program/DocStoreResearchProgram.md similarity index 100% rename from ResearchProgram.md rename to program/DocStoreResearchProgram.md diff --git a/program/SearchCapabilitiesResearchProgram.md b/program/SearchCapabilitiesResearchProgram.md new file mode 100755 index 0000000..ca9aa39 --- /dev/null +++ b/program/SearchCapabilitiesResearchProgram.md @@ -0,0 +1,1691 @@ +# SearchCapabilitiesResearchProgram.md + +**Project:** DocStorePG +**Research Track:** PostgreSQL-native Search Capabilities +**Working label:** SearchPG capability family +**Status:** Proposed +**Date:** 2026-08-23 + +## 1. Purpose + +DocStorePG started from the question of how far PostgreSQL can be extended toward a document-database role normally associated with MongoDB while retaining PostgreSQL as the underlying transactional substrate. + +This research track extends that question: + +> **How far can PostgreSQL also absorb the search responsibilities commonly delegated to Elasticsearch or OpenSearch, such that document storage, transactional access, structured filtering, lexical search, semantic retrieval, and tenant-aware application search can operate over one coherent data substrate?** + +The goal is not to prove that PostgreSQL should replace Elasticsearch everywhere. The goal is to establish a measured capability envelope, identify architectures that minimize unnecessary data-system proliferation, and determine where a dedicated distributed search engine remains justified. + +The research should produce both practical components for DocStorePG and reusable evidence for architectural decisions. + +--- + +## 2. Problem Statement + +A common modern application architecture uses several specialized data systems: + +```text +Application + | + +--> PostgreSQL / MongoDB -- system of record + | + +--> Elasticsearch -- search projection + ^ + | + CDC / ETL / queues +``` + +This creates a second representation of application reality. + +That duplication introduces: + +- synchronization delay; +- failed or incomplete index updates; +- schema and mapping duplication; +- reindexing workflows; +- CDC, queue, or ETL infrastructure; +- duplicated authorization and tenant filters; +- reconciliation requirements; +- separate backup and recovery procedures; +- separate observability and operational expertise; +- additional infrastructure cost; +- two query languages and often two application data-access paths; +- uncertainty about which representation is authoritative at a particular instant. + +For large-scale distributed search, these costs can be justified. For many application-search workloads, they may be accidental complexity. + +PostgreSQL already provides a substantial subset of search primitives through: + +- generalized inverted indexes (GIN); +- full-text search using `tsvector` / `tsquery`; +- `jsonb`; +- trigram similarity through `pg_trgm`; +- relational and structured filtering; +- Row Level Security; +- PostGIS for spatial search; +- extension mechanisms for alternative index access methods. + +The current extension ecosystem adds substantially more: + +- `pg_search` / ParadeDB for BM25, advanced lexical search, faceting, highlighting and search-oriented execution; +- `pgvector` for exact and approximate vector similarity search; +- PGroonga as an alternative full-text/search implementation with strong multilingual and JSONB capabilities; +- ZomboDB as a useful comparison architecture in which Elasticsearch is managed as a PostgreSQL index. + +The research problem is therefore no longer simply: + +> “Can PostgreSQL do full-text search?” + +It is: + +> **Can PostgreSQL provide a sufficiently complete, performant, operable and secure application-search substrate to eliminate a separate Elasticsearch/OpenSearch tier for a defined class of workloads?** + +--- + +## 3. Research Thesis + +The working thesis is: + +> **A PostgreSQL-centered architecture can cover most document-oriented application-search workloads when lexical search, structured filtering, vector retrieval and authorization operate over the same transactional objects. A separate distributed search engine should be introduced only when measured scale, ingestion, analytical, or distribution requirements exceed that capability envelope.** + +A stronger version of the thesis is relevant to DocStorePG: + +> **PostgreSQL may serve as a universal application-information substrate whose relational, document, search, vector and spatial capabilities are exposed through specialized access surfaces rather than implemented as separate authoritative databases.** + +This yields the conceptual target: + +```text + PostgreSQL + | + +------------------+------------------+ + | | | + SQL API Document API Search API + | | | + tables JSONB Query IR + | | | + +------------------+------------------+ + | + Common transactional objects + | + +------------------+------------------+ + | | | + B-tree/GIN lexical index vector index + BM25 / FTS HNSW / IVF +``` + +SearchPG is a working label for this capability family inside DocStorePG. It does not initially imply a separate repository or product. + +--- + +## 4. Research Goals + +The research program shall answer six principal questions. + +### G1 — Capability + +Determine which Elasticsearch/OpenSearch search capabilities can be provided by PostgreSQL itself or by mature PostgreSQL extensions. + +### G2 — Quality + +Determine whether result relevance and search behavior are competitive with Elasticsearch/OpenSearch for representative application workloads. + +### G3 — Performance and Scale + +Determine the performance envelope for: + +- ingestion; +- updates; +- top-K search; +- filtered search; +- faceting; +- fuzzy and prefix search; +- hybrid lexical/vector search; +- multi-tenant workloads; +- index construction and rebuilding. + +### G4 — Architecture and Operations + +Measure whether consolidating search into PostgreSQL materially reduces operational complexity and total cost without creating unacceptable OLTP interference or recovery risk. + +### G5 — Compatibility + +Determine whether a useful Elasticsearch/OpenSearch-compatible API subset can be implemented over PostgreSQL without attempting impractical full emulation. + +### G6 — DocStorePG Integration + +Determine how search becomes a first-class capability of the DocStorePG document model, API, multitenancy model and deployment architecture. + +--- + +## 5. Non-Goals + +The initial program will not attempt to: + +- prove that PostgreSQL replaces Elasticsearch for every workload; +- reproduce the complete Elasticsearch REST API; +- reproduce every Lucene analyzer or plugin; +- optimize initially for petabyte-scale observability; +- replace SIEM platforms; +- reproduce cross-region Elasticsearch cluster behavior; +- implement a new search engine from scratch before existing PostgreSQL extensions have been evaluated; +- conflate document storage compatibility with search API compatibility. + +The initial target is **application search**, not unrestricted distributed log analytics. + +--- + +## 6. Search Capability Canon + +The project should define a canonical capability model independent of any implementation. + +### 6.1 Structured Search + +- equality filters; +- ranges; +- boolean combinations; +- nested/document-field predicates; +- arrays; +- sorting; +- pagination; +- aggregation. + +### 6.2 Lexical Search + +- tokenization; +- stemming; +- stop words; +- language analyzers; +- field weighting; +- BM25-style ranking; +- phrase search; +- proximity search; +- fuzzy matching; +- prefix search; +- typo tolerance; +- synonyms; +- highlighting. + +### 6.3 Faceting and Analytics + +- term facets; +- range facets; +- counts; +- grouped aggregations; +- filtered aggregations; +- top-K plus facet evaluation. + +### 6.4 Semantic Search + +- vector storage; +- exact nearest-neighbor search; +- approximate nearest-neighbor search; +- cosine / inner-product / L2 distance; +- filtered vector search; +- vector index lifecycle. + +### 6.5 Hybrid Search + +- lexical + semantic retrieval; +- rank fusion; +- reranking; +- structured business signals; +- recency weighting; +- domain-specific ranking signals. + +### 6.6 Document Search + +- dynamic JSON fields; +- nested structures; +- field-specific indexing; +- indexing all textual JSON values; +- schema evolution; +- partial indexes; +- index configuration per document type. + +### 6.7 Security and Multitenancy + +- tenant filtering; +- Row Level Security interaction; +- tenant-specific indexes where appropriate; +- shared-index isolation; +- authorization-aware search; +- noisy-neighbor behavior; +- tenant-aware ranking and facets. + +### 6.8 Operational Search Capabilities + +- online index creation; +- rebuild/reindex; +- backup; +- restore; +- PITR; +- physical replication; +- logical replication; +- failover; +- rolling upgrade; +- extension upgrade; +- Kubernetes deployment; +- monitoring; +- resource governance. + +### 6.9 Compatibility Surface + +- search request; +- query clauses; +- filters; +- sort; +- pagination; +- highlighting; +- facets/aggregations; +- mappings/index definitions; +- bulk indexing semantics; +- selected Elasticsearch/OpenSearch response structures. + +--- + +## 7. Candidate Solution Families + +The program should evaluate implementations as competing or composable solution families. + +### A. Native PostgreSQL + +Components: + +```text +PostgreSQL ++ tsvector / tsquery ++ GIN ++ pg_trgm ++ JSONB ++ ordinary SQL indexes +``` + +Purpose: + +- establish the minimum-dependency baseline; +- determine how much search functionality requires no specialized engine at all; +- identify the point at which native ranking or performance becomes inadequate. + +This is the architectural simplicity baseline. + +--- + +### B. PostgreSQL + `pg_search` / ParadeDB + +Components: + +```text +PostgreSQL ++ pg_search ++ Tantivy-backed BM25 indexes ++ search-oriented aggregations +``` + +Investigate: + +- BM25 ranking; +- fuzzy search; +- phrase/proximity behavior; +- highlighting; +- facets; +- top-K performance; +- structured filters; +- joins; +- transaction semantics; +- index build and update behavior; +- resource consumption; +- Community vs Enterprise deployment characteristics; +- licensing implications; +- replication and recovery. + +This is currently the primary candidate for an Elasticsearch-class lexical search capability inside PostgreSQL. + +--- + +### C. PostgreSQL + PGroonga + +Investigate as both an alternative and a specialized capability. + +Particular areas of interest: + +- multilingual search; +- CJK language behavior; +- JSONB full-text search; +- similar-search functionality; +- Row Level Security interaction; +- performance; +- replication and recovery; +- permissive licensing. + +PGroonga should prevent the research program from prematurely equating “PostgreSQL search” with a single extension. + +--- + +### D. PostgreSQL + `pgvector` + +Components: + +```text +PostgreSQL ++ pgvector ++ HNSW ++ IVFFlat +``` + +Investigate: + +- exact retrieval; +- HNSW; +- IVFFlat; +- filtered vector search; +- recall/latency trade-offs; +- index size; +- write behavior; +- tenant filters; +- partitioning; +- hybrid integration. + +This represents the semantic-search layer. + +--- + +### E. Combined Search PostgreSQL + +The most important practical candidate is: + +```text +PostgreSQL ++ JSONB ++ pg_search ++ pgvector ++ pg_trgm where useful ++ PostGIS where useful +``` + +This configuration should be treated as a single integrated search substrate rather than a bag of extensions. + +Research must determine: + +- planner interaction; +- extension interaction; +- index selection; +- write amplification; +- memory pressure; +- vacuum behavior; +- transaction semantics; +- operational compatibility. + +--- + +### F. PostgreSQL Search Replica + +Search need not execute on the transactional primary. + +Candidate: + +```text +DocStorePG Primary + | + | logical replication + v +Search PostgreSQL ++ pg_search ++ pgvector +``` + +This topology attempts to obtain: + +- workload isolation; +- independent search scaling; +- PostgreSQL-compatible data model; +- reduced coupling between search and OLTP resource consumption. + +Measure the consistency/lag trade-off explicitly. + +--- + +### G. Distributed PostgreSQL Search + +Candidate: + +```text +PostgreSQL ++ Citus or comparable sharding layer ++ pg_search +``` + +Research questions: + +- tenant-key distribution; +- distributed top-K; +- distributed facets; +- cross-shard joins; +- query planning; +- rebalancing; +- replica placement; +- operational complexity relative to Elasticsearch/OpenSearch. + +This is important for finding the point at which “Postgres for search” remains viable after a single node is exhausted. + +--- + +### H. ZomboDB Control Architecture + +ZomboDB deliberately retains Elasticsearch as the physical search engine while exposing it through PostgreSQL index semantics. + +It should be included as a control because it answers a different question: + +> Can PostgreSQL own search consistency and query integration even when Elasticsearch remains the index engine? + +This helps separate three benefits: + +1. one application query surface; +2. transactional/index consistency; +3. elimination of Elasticsearch itself. + +--- + +### I. Elasticsearch and OpenSearch Baselines + +Both should be benchmark controls. + +The research is invalid if PostgreSQL configurations are measured only against one another. + +Reference configurations should include: + +```text +Application -> Elasticsearch +Application -> OpenSearch +``` + +and, where relevant: + +```text +PostgreSQL -> CDC -> Elasticsearch/OpenSearch +``` + +--- + +## 8. Proposed Search API Architecture + +Full Elasticsearch compatibility should not be the first implementation target. + +Instead, introduce a canonical **Search Query Intermediate Representation**. + +```text +Elasticsearch-like request + | +Mongo/DocStore search request + | +Native DocStorePG Search API + | + v + Search Query IR + | + +-----+------+---------+----------+ + | | | | +filters lexical vector facets + | | | | + +------------+---------+----------+ + | + PostgreSQL planner +``` + +The Search Query IR should model: + +```text +SearchRequest + query + lexical + semantic + filters + ranking + sort + pagination + facets + highlighting + projection +``` + +Backends can then compile the same request into: + +- native PostgreSQL FTS; +- `pg_search`; +- PGroonga; +- `pgvector`; +- hybrid SQL; +- optionally Elasticsearch/OpenSearch for comparison. + +This provides two strategic benefits. + +First, DocStorePG does not become coupled to one extension. + +Second, API compatibility can be implemented as adapters over the IR. + +--- + +## 9. Elasticsearch/OpenSearch Compatibility Strategy + +Compatibility should be **profile-based**, not claimed as universal. + +Suggested profiles: + +### ES-SEARCH-0 — Basic Search + +- `_search`; +- `match`; +- `term`; +- `terms`; +- `range`; +- `bool`; +- sorting; +- pagination. + +### ES-SEARCH-1 — Application Search + +Adds: + +- `multi_match`; +- phrase search; +- fuzzy; +- highlighting; +- selected aggregations; +- field boosting. + +### ES-SEARCH-2 — Hybrid Search + +Adds: + +- vector query primitives; +- lexical/vector rank fusion; +- selected reranking. + +### ES-SEARCH-X — Unsupported / Native Required + +Explicitly mark unsupported functionality rather than silently approximate it. + +Compatibility testing should be done against pinned Elasticsearch/OpenSearch versions because their APIs have diverged over time. + +The project should prefer **useful compatibility** over cosmetic wire compatibility. + +--- + +## 10. Research Hypotheses + +### H1 — Native PostgreSQL covers a large basic-search envelope + +For many application workloads, `tsvector`, GIN, `pg_trgm`, JSONB and SQL filters will be sufficient. + +### H2 — BM25 is the main threshold between database FTS and search-engine-quality lexical retrieval + +`pg_search` or another BM25-capable index should materially improve ranking quality and top-K performance over native PostgreSQL ranking. + +### H3 — Search over authoritative rows is an architectural advantage + +Removing an asynchronous external search projection will eliminate measurable classes of consistency and authorization defects. + +### H4 — Hybrid search becomes simpler inside one database + +Lexical relevance, vector similarity, structured filters and business ranking signals should be easier to compose when they operate over the same row identifiers. + +### H5 — Tenant-aware filtering benefits disproportionately from PostgreSQL integration + +Shared-table multitenancy, RLS, joins and application authorization should require less duplicated security logic than an external search index. + +### H6 — Search workload isolation remains necessary above moderate load + +A search replica or dedicated Search PostgreSQL node will likely become preferable before PostgreSQL's raw search capability is exhausted. + +### H7 — A PostgreSQL search topology can scale farther than the common “single database” mental model suggests + +Sharding and replication may extend the useful envelope considerably, but operational complexity may approach that of Elasticsearch. + +### H8 — A small Elasticsearch-compatible API subset provides most migration value + +A carefully selected 20–30% of API primitives may cover the majority of application-search integration use cases. + +### H9 — Full Elasticsearch compatibility is economically unattractive + +The long tail of mappings, analyzers, cluster APIs, plugins and Query DSL behavior is unlikely to justify reproduction. + +### H10 — Elasticsearch/OpenSearch will retain a clear advantage for some workloads + +Likely examples: + +- extremely high-volume event ingestion; +- very large append-only observability datasets; +- massive horizontal fan-out; +- mature Lucene-specific custom analysis; +- large distributed aggregation workloads. + +The research should locate this boundary rather than argue it away. + +--- + +## 11. Workstreams + +## WS0 — Capability Inventory and Baseline + +Deliver: + +- `SearchCapabilityCanon.md`; +- implementation capability matrix; +- version and licensing matrix; +- known deployment constraints. + +Candidates: + +- PostgreSQL native FTS; +- `pg_trgm`; +- `pg_search`; +- PGroonga; +- `pgvector`; +- ZomboDB; +- Elasticsearch; +- OpenSearch. + +--- + +## WS1 — Lexical Search Quality + +Evaluate: + +- tokenization; +- stemming; +- phrase search; +- fuzzy search; +- prefix behavior; +- field weighting; +- BM25; +- language support; +- synonyms; +- highlighting. + +Metrics: + +- NDCG@10; +- MRR; +- Recall@K; +- judged relevance; +- latency. + +Use at least one public relevance dataset plus one DocStorePG-style document corpus. + +--- + +## WS2 — Structured and JSON Document Search + +Evaluate: + +- dynamic fields; +- JSON path filters; +- nested structures; +- arrays; +- numeric/date ranges; +- combinations of full-text and JSON predicates; +- schema evolution; +- index configuration changes. + +This workstream directly connects search to the DocStorePG document model. + +--- + +## WS3 — Facets and Aggregations + +Evaluate: + +- term facets; +- range facets; +- top-K + facets; +- high-cardinality facets; +- filtered facets; +- multi-facet requests. + +Measure CPU, memory and latency separately from ordinary top-K search. + +--- + +## WS4 — Semantic and Hybrid Retrieval + +Compare: + +```text +BM25 +vector-only +BM25 + vector +BM25 + vector + structured business signals +``` + +Fusion methods should include at least: + +- Reciprocal Rank Fusion; +- weighted normalized scores; +- reranking of lexical candidates; +- reranking of vector candidates. + +Measure: + +- relevance; +- vector recall; +- latency; +- filter interaction; +- index memory; +- ingestion/update cost. + +--- + +## WS5 — Multitenancy and Security + +Test at least three tenant models. + +### Model A — Shared Table + +```text +documents( + tenant_id, + ... +) +``` + +### Model B — Partitioned Shared Database + +Partitions by tenant or tenant group. + +### Model C — Database / Cluster Isolation + +Dedicated PostgreSQL databases or clusters for high-isolation tenants. + +For each model measure: + +- tenant filter cost; +- RLS correctness; +- search ranking correctness; +- facet isolation; +- vector filtering; +- noisy-neighbor behavior; +- cross-tenant leakage attempts; +- operational cost. + +Search authorization tests should be adversarial, not merely functional. + +--- + +## WS6 — API Compatibility + +Implement a thin prototype: + +```text +HTTP Search Gateway + | +Elasticsearch/OpenSearch subset + | +Search Query IR + | +PostgreSQL +``` + +Test existing client libraries where practical. + +Record each feature as: + +```text +compatible +compatible-with-difference +translated +unsupported +``` + +Do not hide semantic differences. + +--- + +## WS7 — Consistency and Lifecycle + +Test: + +- insert visibility; +- update visibility; +- delete visibility; +- transaction rollback; +- concurrent updates; +- long-running transactions; +- index rebuild; +- schema migration; +- extension upgrade; +- backup/restore; +- failover; +- logical replication lag; +- crash recovery. + +This should explicitly compare synchronous in-database indexing with CDC-backed Elasticsearch/OpenSearch. + +--- + +## WS8 — Performance and Scale + +Test dataset sizes such as: + +```text +100 K +1 M +10 M +100 M +``` + +and larger where the environment makes the experiment meaningful. + +Measure: + +### Writes + +- documents/sec; +- updates/sec; +- deletes/sec; +- bulk load; +- WAL volume; +- write amplification. + +### Reads + +- p50; +- p95; +- p99; +- QPS; +- concurrent users. + +### Search Workloads + +- top-K lexical; +- phrase; +- fuzzy; +- filter + search; +- faceted; +- vector; +- hybrid; +- tenant-filtered; +- deep pagination where supported. + +### Resources + +- CPU; +- RAM; +- disk; +- index size; +- cache hit behavior; +- temporary I/O. + +--- + +## WS9 — Deployment and Operations + +Evaluate: + +```text +Single PostgreSQL +Primary + search replica +HA PostgreSQL cluster +CloudNativePG deployment +Sharded PostgreSQL +Elasticsearch/OpenSearch cluster +``` + +Measure: + +- installation; +- upgrade; +- backup; +- restore; +- failover; +- rebuild; +- monitoring; +- day-2 operations; +- operator support; +- extension packaging; +- Kubernetes compatibility. + +The benchmark should capture operator effort, not merely query latency. + +--- + +## WS10 — Economics and Complexity + +Create a normalized cost model. + +### Infrastructure + +- compute; +- memory; +- storage; +- network; +- backup; +- replicas. + +### Operational Complexity + +Score: + +- systems operated; +- data copies; +- synchronization mechanisms; +- schemas/mappings; +- recovery procedures; +- monitoring surfaces; +- specialist knowledge; +- failure modes. + +### Development Complexity + +Measure: + +- LOC for integration; +- number of APIs; +- authorization duplication; +- migration effort; +- test surface. + +The result should produce both: + +```text +technical capability envelope +``` + +and: + +```text +economic/operational capability envelope +``` + +--- + +## 12. Benchmark Dataset Families + +Use multiple dataset families rather than one synthetic benchmark. + +### D1 — Relevance Corpus + +Purpose: + +- lexical ranking; +- semantic ranking; +- hybrid search. + +Use a public IR/relevance dataset such as an appropriate BEIR/TREC-derived corpus. + +### D2 — Product Catalog + +Documents with: + +- title; +- description; +- brand; +- categories; +- attributes; +- price; +- stock; +- ratings. + +Purpose: + +- faceting; +- filtering; +- fuzzy search; +- ranking; +- high-cardinality attributes. + +### D3 — Document Management Corpus + +Documents with: + +- metadata; +- extracted text; +- participants; +- dates; +- document type; +- tags; +- tenant ACLs. + +Purpose: + +- realistic DocStorePG application search. + +### D4 — Multi-Tenant SaaS Corpus + +Generate tenants with a skewed distribution: + +```text +many small +some medium +few very large +``` + +Purpose: + +- tenant isolation; +- noisy-neighbor effects; +- partitioning; +- RLS; +- filtered vector search. + +### D5 — Event/Log Corpus + +Purpose: + +- intentionally stress the likely boundary; +- compare ingestion and time-range search against Elasticsearch/OpenSearch. + +This dataset is important because a good research program must contain workloads expected to falsify the PostgreSQL thesis. + +--- + +## 13. Benchmark Harness + +The benchmark harness should be reusable across backends. + +Suggested structure: + +```text +bench/search/ + datasets/ + workloads/ + adapters/ + postgres_native/ + pg_search/ + pgroonga/ + pgvector/ + postgres_hybrid/ + zombodb/ + elasticsearch/ + opensearch/ + runner/ + metrics/ + reports/ +``` + +Each workload should produce a machine-readable result record containing: + +```text +backend +version +dataset +dataset_size +query_class +concurrency +latency_p50 +latency_p95 +latency_p99 +throughput +cpu +memory +io +index_size +relevance_metrics +consistency_metrics +``` + +Benchmark definitions should be immutable/versioned so results remain comparable over time. + +--- + +## 14. Failure Injection + +Search systems are often selected on benchmark speed and regretted during failure. + +Inject: + +- database restart; +- hard process kill; +- search-node loss; +- network interruption; +- replication interruption; +- disk pressure; +- index corruption where safely reproducible; +- stalled replication; +- partially completed bulk load; +- extension upgrade failure. + +Measure: + +```text +RPO +RTO +search availability +data correctness +repair procedure +operator actions +``` + +--- + +## 15. Decision Metrics + +A configuration should not be declared successful merely because it is faster. + +Evaluate across six dimensions. + +### Q — Search Quality + +- relevance; +- recall; +- query expressiveness. + +### P — Performance + +- latency; +- throughput; +- ingestion. + +### C — Consistency + +- transactional visibility; +- lag; +- correctness. + +### O — Operability + +- deployment; +- recovery; +- upgrades; +- monitoring. + +### S — Security + +- tenant isolation; +- authorization integrity. + +### E — Economics + +- infrastructure cost; +- engineering cost; +- operational cost. + +The eventual decision model can be expressed as: + +```text +fitness = + f(Q, P, C, O, S, E) +``` + +with workload-specific weights rather than one universal winner. + +--- + +## 16. Milestones + +## M0 — Research Foundation + +Deliver: + +- research program; +- capability canon; +- candidate matrix; +- benchmark harness skeleton; +- pinned component versions. + +Exit criterion: + +> Every later experiment can be described through the same capability and metric vocabulary. + +--- + +## M1 — Native PostgreSQL Baseline + +Implement and benchmark: + +- `tsvector`; +- GIN; +- JSONB; +- `pg_trgm`. + +Exit criterion: + +> Native PostgreSQL search envelope is documented with measured limits. + +--- + +## M2 — Search Extension Shootout + +Add: + +- `pg_search`; +- PGroonga; +- ZomboDB control; +- Elasticsearch; +- OpenSearch. + +Exit criterion: + +> Lexical search quality, functionality, performance and operational differences are quantified. + +--- + +## M3 — Semantic and Hybrid Search + +Add: + +- `pgvector`; +- HNSW; +- IVFFlat; +- lexical/vector fusion. + +Exit criterion: + +> A recommended PostgreSQL-native hybrid search architecture is identified or rejected with evidence. + +--- + +## M4 — Multitenancy and Security + +Add: + +- RLS; +- tenant filters; +- partitions; +- authorization joins; +- adversarial leakage tests. + +Exit criterion: + +> At least one secure shared-database search pattern is validated. + +--- + +## M5 — Search API Prototype + +Implement: + +- Search Query IR; +- native DocStorePG search endpoint; +- ES-SEARCH-0 compatibility profile. + +Exit criterion: + +> A representative Elasticsearch-style application can query DocStorePG through the compatibility layer. + +--- + +## M6 — Isolation and Scale + +Evaluate: + +- dedicated search PostgreSQL; +- logical replication; +- sharded PostgreSQL where justified; +- OLTP/search interference. + +Exit criterion: + +> The transition point from embedded search to dedicated search nodes is characterized. + +--- + +## M7 — Failure, Recovery and Operations + +Evaluate: + +- backup; +- restore; +- crash; +- failover; +- reindex; +- upgrades; +- Kubernetes operations. + +Exit criterion: + +> Production-operability constraints and extension-specific risks are explicitly known. + +--- + +## M8 — Architecture Decision + +Produce: + +- benchmark report; +- capability envelope; +- cost model; +- architecture patterns; +- adoption guide; +- boundary conditions. + +Exit criterion: + +> DocStorePG has an evidence-backed answer to when PostgreSQL should replace, complement, or retain Elasticsearch/OpenSearch. + +--- + +## 17. Proposed Architecture Patterns to Validate + +### Pattern P1 — Embedded Search + +```text +Application + | +DocStorePG PostgreSQL + | +JSONB + native FTS +``` + +Target: + +- small/medium applications; +- minimal operations. + +--- + +### Pattern P2 — Enhanced Embedded Search + +```text +Application + | +DocStorePG PostgreSQL + | +JSONB + pg_search + pgvector +``` + +Target: + +- application search; +- knowledge systems; +- product search; +- document search. + +--- + +### Pattern P3 — Search-Isolated PostgreSQL + +```text + +--> OLTP reads + | +DocStorePG Primary + | + +--> logical replication --> Search PostgreSQL + pg_search + pgvector +``` + +Target: + +- higher search load; +- workload isolation. + +--- + +### Pattern P4 — Distributed PostgreSQL Search + +```text +Application + | +Search Gateway + | +Distributed PostgreSQL +Citus + pg_search +``` + +Target: + +- large tenant sets; +- large searchable corpora; +- horizontal scaling experiments. + +--- + +### Pattern P5 — External Search Remains Correct + +```text +DocStorePG + | +CDC / managed integration + | +Elasticsearch/OpenSearch +``` + +Target: + +- workloads outside the PostgreSQL capability envelope. + +This pattern must remain an acceptable research conclusion. + +--- + +## 18. Integration into DocStorePG Scope + +The DocStorePG project should expand from: + +```text +MongoDB-like document capabilities +on PostgreSQL +``` + +to: + +```text +Document-oriented application data capabilities +on PostgreSQL +including search and retrieval +``` + +The capability layers become: + +```text +DocStorePG +| ++-- Document Model +| +-- JSONB +| +-- schema/index management +| ++-- Document API +| +-- Mongo-compatible primitives +| ++-- Search Model +| +-- structured +| +-- lexical +| +-- semantic +| +-- hybrid +| +-- facets +| ++-- Search API +| +-- native DocStorePG +| +-- optional Elasticsearch/OpenSearch profile +| ++-- Multitenancy +| +-- tenant routing +| +-- RLS +| +-- partitioning +| ++-- Operations + +-- CloudNativePG + +-- replication + +-- backup/recovery + +-- observability +``` + +This keeps the research coherent: document storage and search are not independent features. Search is an alternative projection and access mode over the same information model. + +--- + +## 19. Proposed Repository Additions + +```text +docs/ + research/ + search/ + SearchCapabilitiesResearchProgram.md + SearchCapabilityCanon.md + SearchArchitectureOptions.md + SearchCompatibilityProfiles.md + SearchMultitenancyModel.md + SearchBenchmarkSpecification.md + SearchOperationalModel.md + SearchDecisionRecord.md + +bench/ + search/ + datasets/ + workloads/ + adapters/ + runner/ + reports/ + +src/ + search/ + ir/ + planner/ + api/ + adapters/ + +experiments/ + search/ + postgres-native/ + pg-search/ + pgroonga/ + pgvector/ + hybrid/ + replication/ + citus/ + zombodb/ + elasticsearch/ + opensearch/ +``` + +If the repository is still deliberately small, begin only with: + +```text +docs/research/search/SearchCapabilitiesResearchProgram.md +bench/search/ +experiments/search/ +``` + +and allow the remaining structure to emerge from evidence. + +--- + +## 20. Initial Priority + +The first practical experiment should be intentionally narrow. + +Build the same document corpus on: + +```text +A. PostgreSQL native FTS +B. PostgreSQL + pg_search +C. Elasticsearch +``` + +Run: + +```text +1. keyword top-K +2. phrase query +3. fuzzy query +4. structured filter + search +5. facet query +6. insert then immediate search +7. update then immediate search +8. delete then immediate search +``` + +Measure: + +```text +relevance +p50/p95/p99 latency +index size +ingest speed +freshness/consistency +CPU/RAM +implementation complexity +``` + +Then add `pgvector` and hybrid search. + +This sequence gives the project an early falsifiable answer before substantial compatibility work is attempted. + +--- + +## 21. Success Criteria + +The program succeeds even if PostgreSQL does **not** replace Elasticsearch in every tested scenario. + +Success means producing a dependable decision map such as: + +```text +Native PostgreSQL + sufficient here + | + v +PostgreSQL + search extensions + sufficient here + | + v +Dedicated PostgreSQL search nodes + sufficient here + | + v +Distributed PostgreSQL search + investigate here + | + v +Elasticsearch/OpenSearch + justified beyond here +``` + +The most valuable result is the location of the boundaries. + +--- + +## 22. Expected Strategic Outcome + +If the central hypotheses survive testing, DocStorePG becomes more than a MongoDB compatibility experiment. + +It becomes an exploration of: + +> **PostgreSQL as a unified application information substrate.** + +That substrate would combine: + +```text +relational transactions ++ +document storage ++ +structured retrieval ++ +lexical search ++ +semantic search ++ +hybrid ranking ++ +authorization ++ +multitenancy +``` + +while allowing specialized external systems to remain available when scale or workload characteristics genuinely require them. + +This is a stronger and more general research direction than “PostgreSQL as a MongoDB replacement,” because it tests whether several common application data-system boundaries are architectural necessities or historical artifacts. + +--- + +## 23. Current Technology Baseline to Track + +The following projects should be version-pinned in the research repository and periodically reevaluated: + +- PostgreSQL 18 documentation: GIN and Full Text Search; +- ParadeDB / `pg_search`: BM25, full-text search, facets, hybrid-search integration; +- `pgvector`: HNSW and IVFFlat vector indexes; +- PGroonga: multilingual full-text and JSONB search; +- ZomboDB: PostgreSQL-managed Elasticsearch indexes; +- Elasticsearch; +- OpenSearch. + +Important current research considerations include: + +- `pg_search` is actively evolving; +- ParadeDB Community uses AGPL-3.0 while commercial licensing is available separately; +- some recovery/replication characteristics differ between ParadeDB Community and Enterprise; +- PGroonga uses the permissive PostgreSQL license; +- OpenSearch and Elasticsearch APIs have diverged enough that compatibility must be version/profile specific; +- `pgvector` approximate indexes trade recall for speed and therefore require explicit quality measurement. + +These constraints belong in the benchmark and architecture decision process rather than being treated as implementation footnotes.