plenty-gobble/spec/ArchitectureBlueprint.md

plenty-gobble — Architecture Blueprint

Version: 0.1
Date: 2026-09-12
Status: Initial architecture blueprint
Companion document: INTENT.md

---

1. Architecture Goal

plenty-gobble implements a PostgreSQL-centered adaptive analytical datastore for large append-oriented datasets.

The architecture must provide one coherent logical data system while allowing different physical storage and execution mechanisms behind it.

The target shape is:

                         CLIENTS
                            |
                     SQL / API / OTLP
                            |
                            v
                  +--------------------+
                  | PostgreSQL         |
                  | control plane      |
                  +---------+----------+
                            |
             +--------------+--------------+
             |                             |
             v                             v
         hot data                    sealed segments
       PostgreSQL heap                    |
                                          v
                                   S3-compatible store
                                          |
                                          v
                                   Parquet objects
                                          |
                                          v
                                DuckDB / pg_duckdb
                                          |
                                          v
                                  query result merge

The architecture should remain useful at three scales:

1. local developer / single-node;
2. production single-node plus object storage;
3. distributed query execution over shared object storage.

---

2. Architectural Responsibilities

2.1 PostgreSQL

PostgreSQL is the authoritative control plane.

It owns:

• tenant metadata;
• dataset definitions;
• logical schema;
• field registry;
• segment catalog;
• segment lifecycle;
• policy state;
• hot-tail rows;
• optimization policies;
• query telemetry;
• optimization history;
• authorization;
• RLS;
• retention configuration;
• publication transactions.

PostgreSQL does not need to own all historical event payloads.

---

2.2 Object Storage

S3-compatible storage owns immutable bulk data.

Initial supported targets:

• local filesystem abstraction for tests;
• MinIO;
• Amazon S3 or compatible APIs.

Objects should be content-addressable or otherwise verifiable.

Sealed data should not require mutable in-place updates.

---

2.3 Parquet

Parquet is the initial sealed-data format.

It provides:

• columnar representation;
• compression;
• row groups;
• dictionary encoding;
• statistics;
• interoperability;
• efficient scans through DuckDB and other engines.

The architecture must isolate the segment-format interface so Parquet can later coexist with or be replaced by another format if required.

---

2.4 DuckDB / pg_duckdb

DuckDB is the initial cold-data execution engine.

Responsibilities:

• Parquet scanning;
• predicate pushdown;
• column projection;
• vectorized evaluation;
• local aggregation;
• Top-K;
• join execution where appropriate.

pg_duckdb should be evaluated as the shortest path to a unified PostgreSQL-facing query surface.

Standalone DuckDB workers should remain possible without redesigning the storage layer.

---

3. Core Domain Objects

3.1 Tenant

Represents an isolation boundary.

Tenant
  id
  name
  lifecycle_state
  storage_namespace
  policy_profile

All dataset, segment, query, cache, and optimization state must be tenant-scoped unless explicitly declared global.

---

3.2 Dataset

A logical append-oriented relation.

Dataset
  id
  tenant_id
  name
  logical_schema
  retention_policy
  segmentation_policy
  synopsis_policy
  optimization_policy

Examples:

logs
traces
audit_events
deployment_events
agent_events

---

3.3 Field

Represents a discovered or declared logical property.

Field
  id
  dataset_id
  logical_name
  logical_type
  semantic_role
  promotion_state
  first_seen
  last_seen

Possible semantic roles:

timestamp
identifier
dimension
metric
free_text
payload

---

3.4 Hot Row

A row not yet included in a sealed segment.

Hot rows live in PostgreSQL.

A minimal physical shape may include:

hot_event (
    tenant_id,
    dataset_id,
    event_time,
    ingest_time,
    core_fields ...,
    attributes jsonb,
    raw_payload jsonb
)

This schema is intentionally provisional.

The benchmark phase should determine how much structured promotion is beneficial before sealing.

---

3.5 Segment

The fundamental immutable cold-storage unit.

Segment
  id
  tenant_id
  dataset_id

  lower_time
  upper_time

  object_uri
  object_etag / content_hash

  row_count
  compressed_bytes
  uncompressed_bytes

  schema_version
  synopsis_version

  state
  created_at
  sealed_at
  published_at
  superseded_by

Suggested states:

BUILDING
SEALED
PUBLISHED
SUPERSEDED
DELETING
DELETED
FAILED

Only PUBLISHED segments are query-visible.

---

3.6 Segment Synopsis

A compact representation of segment contents.

SegmentSynopsis
  segment_id
  policy_id
  version
  generated_at

Containing per-field summaries such as:

occurrence_count
null_count
distinct_estimate
min
max
dictionary_summary
bloom
token_bloom
histogram
sketch

The synopsis may be:

• stored directly in PostgreSQL;
• split between PostgreSQL metadata and an object-store sidecar;
• embedded partly in Parquet metadata.

The first implementation should optimize for simplicity and observability rather than minimum bytes.

---

3.7 Query Record

Every meaningful query should optionally emit telemetry.

QueryRecord
  id
  tenant_id
  dataset_id
  normalized_query
  predicate_signature
  projection_signature
  started_at
  duration
  rows_returned

  segments_total
  segments_after_time
  segments_after_partition
  segments_after_synopsis

  row_groups_opened
  bytes_logical
  bytes_read

  execution_engine
  worker_count

This record becomes input to future optimization.

---

4. Main Components

4.1 Ingest Gateway

Responsibilities:

• authenticate tenant;
• identify dataset;
• validate basic input;
• normalize event envelope;
• assign ingest metadata;
• write to hot-tail path.

Initial interfaces:

• HTTP JSON;
• NDJSON;
• OpenTelemetry logs;
• file/batch ingest.

Future:

• OTLP traces;
• Kafka;
• Fluent Bit;
• Fluentd.

---

4.2 Schema Profiler

Observes incoming data and updates field knowledge.

Calculates approximate statistics such as:

type distribution
occurrence rate
null rate
cardinality estimate
value width
repetition rate
numeric range
text length

It must not block ingestion on expensive analysis.

Initial profiling may occur during segment construction rather than on every hot insert.

---

4.3 Segment Builder

Transforms hot rows into an immutable cold segment.

Pipeline:

select sealable hot range
        |
        v
materialize batch
        |
        v
normalize schema
        |
        v
profile fields
        |
        v
choose promoted/residual layout
        |
        v
choose encoding
        |
        v
write Parquet
        |
        v
build synopsis
        |
        v
upload object(s)
        |
        v
verify
        |
        v
publish segment transactionally
        |
        v
retire covered hot rows

The builder must be idempotent.

A failed build must never expose a partially published segment.

---

4.4 Synopsis Builder

Receives:

FieldProfile
+
SynopsisPolicy

and returns the selected summary structures.

Conceptual policy:

if numeric and stable:
    min/max
    maybe histogram

if low cardinality:
    dictionary summary

if high-cardinality equality lookup candidate:
    Bloom filter

if free text:
    token / n-gram synopsis

if rarely queried and low elimination value:
    no additional synopsis

The actual thresholds must be policy data rather than hard-coded architectural constants.

---

4.5 Segment Catalog

PostgreSQL relations representing:

• published segments;
• locations;
• schemas;
• time ranges;
• partition attributes;
• synopsis versions;
• lifecycle state.

The catalog must support transactional visibility.

The query planner must never infer visible objects by listing the bucket.

---

4.6 Query Frontend

Initial query surfaces:

1. internal logical query API;
2. SQL through PostgreSQL / pg_duckdb where practical.

The logical query model should express at least:

dataset
time range
predicates
projection
grouping
aggregations
order
limit

The architecture should avoid binding the planner to a single textual query syntax.

---

4.7 Poly Planner

The planner converts a logical query into a physical plan.

Stages:

logical query
    |
    v
authorization
    |
    v
dataset resolution
    |
    v
hot/cold split
    |
    v
time pruning
    |
    v
partition pruning
    |
    v
synopsis pruning
    |
    v
execution cost estimate
    |
    +---------------------------+
    |                           |
    v                           v
local execution            distributed execution

Initial planning can live in application code.

Later integration may use:

• PostgreSQL extension code;
• FDW;
• CustomScan;
• planner hooks.

---

4.8 Local Cold Executor

Initial implementation:

DuckDB / pg_duckdb

Receives a list of candidate segment URIs and a physical query.

Responsibilities:

• read only selected columns;
• exploit Parquet statistics;
• exploit row-group Bloom filters where available;
• evaluate predicates;
• aggregate locally;
• return compact result.

---

4.9 Distributed Worker

Introduced only after single-node limits are demonstrated.

Worker contract:

{
  "query_id": "...",
  "tenant_id": "...",
  "segments": ["..."],
  "projection": ["..."],
  "predicate": "...",
  "group_by": ["..."],
  "aggregations": ["..."],
  "order_by": ["..."],
  "limit": 100
}

Response should prefer reduced results:

partial aggregate
partial Top-K
row batch only when required

Workers are stateless.

No worker owns durable catalog state.

---

4.10 Result Merger

Combines:

• hot-tail result;
• one or more cold executor results;
• distributed partial aggregates.

Functions include:

UNION
GROUP reduction
Top-K merge
sort merge
limit
deduplication when required

---

4.11 Workload Profiler

Consumes query telemetry and computes:

field query frequency
predicate frequency
operator frequency
selectivity
bytes avoided
Bloom usefulness
promotion usefulness
aggregation recurrence

It produces recommendations rather than silent changes initially.

---

4.12 Optimizer Controller

Applies or recommends physical-design changes.

Possible actions:

promote field
demote field
add synopsis
remove synopsis
change segmentation target
change row-group target
recluster future segments
materialize aggregate
schedule re-compaction

Every action must record:

reason
policy
evidence
expected benefit
actual result
rollback path

---

5. PostgreSQL Catalog Sketch

Initial relational model:

tenant
dataset
dataset_field
schema_version
segment
segment_partition
segment_field_stats
segment_synopsis
synopsis_policy
segmentation_policy
optimization_policy
query_record
query_field_usage
optimization_action

A useful first schema relationship:

tenant
  |
  +-- dataset
        |
        +-- dataset_field
        |
        +-- schema_version
        |
        +-- segment
              |
              +-- segment_field_stats
              |
              +-- segment_synopsis

PostgreSQL RLS should be applied to catalog tables from the beginning.

---

6. Physical Segment Layout

Initial Parquet design should distinguish:

6.1 Core columns

Likely stable across most events:

event_time
ingest_time
event_id
tenant_id
dataset_id

For observability:

severity
service_name
trace_id
span_id
message

where available.

---

6.2 Promoted fields

Fields selected because they are:

• common;
• stable;
• frequently filtered;
• frequently grouped;
• operationally meaningful.

---

6.3 Residual attributes

Sparse dynamic data remains in a residual representation.

Candidates to benchmark:

• Parquet MAP;
• JSON string;
• structured key/value repeated field;
• separate sparse property table/segment.

No choice should be made without benchmark evidence.

---

7. Segment Sizing

Segment sizing is policy-driven.

Variables:

target rows
target compressed bytes
maximum time span
minimum time span
tenant activity
dataset activity
object-store request economics
query locality

Initial experiment candidates:

128 MB
256 MB
512 MB
1 GB

compressed target sizes.

The architecture must allow different datasets to use different policies.

---

8. Row-Group Sizing

Parquet row groups form a second pruning level.

Benchmark candidates:

16 MB
32 MB
64 MB
128 MB

Trade-off:

smaller groups
  + finer pruning
  - more metadata
  - potentially worse scan efficiency

larger groups
  + better scan throughput
  + less metadata
  - coarser pruning

The correct value is workload-dependent.

---

9. Query Lifecycle

Example query:

SELECT service_name, count(*)
FROM logs
WHERE event_time >= now() - interval '24 hours'
  AND status_code >= 500
GROUP BY service_name;

Execution:

1 authorize tenant
2 resolve dataset
3 determine hot/cold time split
4 query segment catalog for overlapping time
5 prune segments by partition tags
6 prune by status-code min/max synopsis
7 produce cold candidate set
8 execute hot subquery in PostgreSQL
9 execute cold subquery in DuckDB
10 partially aggregate
11 merge counts
12 record query telemetry
13 return result

An EXPLAIN surface should expose every pruning stage.

---

10. EXPLAIN Contract

Example:

PolyGobble Query Plan

Dataset: logs
Tenant: acme
Time range: 24h

Hot:
  rows considered: 184,221
  execution: PostgreSQL

Cold:
  segments total: 18,422
  removed by time: 17,901
  removed by partition: 311
  removed by synopsis: 183
  opened: 27

  row groups total in opened segments: 324
  row groups opened: 41

Columns:
  status_code
  service_name

Logical bytes covered: 8.4 TB
Physical bytes read: 812 MB

Execution:
  engine: DuckDB
  workers: 1

Data-skipping efficiency should be treated as a first-class metric.

---

11. Consistency Model

Hot data

Normal PostgreSQL transactional semantics.

Sealed segments

Immutable after publication.

Publication

Publication follows:

write object
    |
    v
verify object
    |
    v
BEGIN
    |
insert segment metadata
insert synopsis metadata
mark PUBLISHED
    |
COMMIT

Only after commit may queries see the segment.

Removal of covered hot rows may occur after publication and must be retryable.

Queries spanning the transition must avoid gaps or duplicates.

One approach:

• assign a deterministic coverage interval / batch ID;
• publish cold coverage;
• mark hot rows as covered;
• remove asynchronously.

---

12. Supersession

Because sealed segments are immutable, correction uses supersession.

segment A
   |
superseded by
   v
segment B

The catalog controls visibility.

This supports:

• re-compaction;
• re-encoding;
• schema normalization;
• corruption repair;
• optimization rewrites.

Garbage collection is separate from logical supersession.

---

13. Multi-Tenancy

Every relevant object carries tenant identity.

Isolation layers:

PostgreSQL RLS
object namespace
segment catalog
worker authorization
query routing
cache keying
metrics
optimization telemetry

A worker must never receive segment references outside the authorized tenant scope.

Object credentials should preferably be short-lived or scoped.

---

14. Security Boundaries

Trust zones:

client
  |
  v
ingest/query gateway
  |
  v
PostgreSQL control plane
  |
  +----> object storage
  |
  +----> execution workers

Required controls:

• authenticated tenant identity;
• authorization before segment selection;
• scoped storage access;
• no client-controlled arbitrary object URI execution;
• query resource limits;
• input limits;
• audit logging;
• secret isolation.

---

15. Caching

Caching should be layered.

Potential levels:

catalog metadata
segment synopsis
Parquet footer
object range
decoded row group
query result
materialized aggregate

Caching must remain tenant-safe.

The first implementation should cache metadata before data.

Good pruning reduces the need for large data caches.

---

16. Failure Handling

Segment build failure

• segment remains non-published;
• partial object can be garbage collected;
• hot data remains authoritative.

Object upload succeeds but PostgreSQL transaction fails

• object is orphaned;
• catalog scanner / GC may remove it later;
• it is not query-visible.

PostgreSQL publication succeeds

The referenced object must already have been verified.

Worker failure

Coordinator retries work on another stateless worker.

Query planner failure

Must not corrupt state.

---

17. Observability

plenty-gobble should expose metrics for:

Ingestion

events/sec
bytes/sec
hot-tail size
seal queue depth
segment build latency

Storage

segment count
average segment size
compression ratio
object count
orphan objects

Query

latency
segments considered
segments skipped
row groups skipped
logical bytes
physical bytes
bytes avoided
CPU time
object requests

Adaptive optimization

synopsis hit rate
Bloom usefulness
field promotion benefit
recompaction benefit
policy version
optimization success/failure

---

18. Repository Structure — Initial Proposal

plenty-gobble/
├── INTENT.md
├── ArchitectureBlueprint.md
├── research/
│   └── 2026-09-12T0933+0200-bronto-technical-core.md
│
├── docs/
│   ├── concepts/
│   ├── decisions/
│   └── benchmarks/
│
├── sql/
│   ├── catalog/
│   └── migrations/
│
├── src/
│   ├── ingest/
│   ├── catalog/
│   ├── segment/
│   ├── synopsis/
│   ├── planner/
│   ├── executor/
│   ├── optimizer/
│   └── common/
│
├── tests/
│   ├── unit/
│   ├── integration/
│   └── conformance/
│
├── benchmarks/
│   ├── datasets/
│   ├── workloads/
│   └── reports/
│
└── deploy/
    ├── docker/
    └── kubernetes/

The language should remain undecided until the first executable spike clarifies where performance-sensitive code actually lives.

A likely combination is:

• SQL for catalog;
• Python or Go for orchestration prototypes;
• Rust only where a systems component genuinely needs it;
• C/C++ only if PostgreSQL extension integration later requires it.

---

19. V0 Implementation Scope

V0 should prove the architecture with the fewest moving parts.

Required

• PostgreSQL;
• local filesystem or MinIO;
• Parquet writer;
• DuckDB or pg_duckdb;
• one dataset;
• one tenant;
• hot-tail table;
• segment catalog;
• segment builder;
• time pruning;
• simple min/max synopsis;
• union of hot + cold query results;
• benchmark harness.

Explicitly deferred

• Kafka;
• distributed workers;
• self-optimization;
• token Bloom;
• custom PostgreSQL extension;
• Iceberg;
• materialized aggregates;
• complex multi-tenant billing.

---

20. V0 Acceptance Test

Given:

10 million synthetic events

with:

timestamp
service_name
status_code
trace_id
duration_ms
message
dynamic attributes

the system should:

1. ingest events into the hot tail;
2. seal historical ranges into Parquet;
3. publish segments transactionally;
4. remove or retire covered hot rows;
5. execute a logical query across hot and cold data;
6. prune cold segments using timestamp metadata;
7. produce identical results to a reference full-scan query;
8. expose bytes read and segments skipped.

This is the first architecture milestone.

---

21. V1 Acceptance Test

Add adaptive synopsis capability.

For:

trace_id = X

the system should demonstrate that:

segments opened << segments overlapping time range

For:

status_code >= 500

min/max or dictionary summaries should eliminate irrelevant segments.

The benchmark report must show:

candidate segments
opened segments
bytes logical
bytes physical
query latency
index/synopsis bytes

---

22. Benchmark Workloads

Create stable workload classes.

W1 — exact high-cardinality lookup

trace_id = X

W2 — categorical lookup

service_name = "billing"

W3 — numeric range

duration_ms > 1000

W4 — time + predicate

last 24h AND status_code >= 500

W5 — grouped aggregation

count(*) GROUP BY service_name

W6 — Top-K

top endpoints by errors

W7 — free-text contains

message contains "authorization"

W8 — full scan

Intentional worst case to measure pure scan throughput.

---

23. Technology Decisions to Keep Open

The blueprint intentionally does not yet mandate:

• Kafka vs Redpanda;
• MinIO vs S3;
• pg_duckdb vs external DuckDB process;
• Python vs Go vs Rust orchestrator;
• Iceberg;
• exact residual sparse encoding;
• distributed execution protocol;
• caching engine;
• PostgreSQL extension depth.

These should be resolved by experiments and ADRs.

---

24. Architectural Invariants

The following should remain true unless the architecture is explicitly revised.

I1

PostgreSQL is authoritative for logical catalog state.

I2

Sealed historical segments are immutable.

I3

Object existence does not imply query visibility.

I4

Publication is transactional.

I5

All query planning is tenant-authorized before object access.

I6

Data-skipping metadata is versioned and explainable.

I7

Optimization policy is separate from logical data semantics.

I8

Single-node execution is preferred until scale-out is justified.

I9

Distributed workers are stateless.

I10

Every optimization should be measurable.

---

25. Primary Research Questions

The implementation program should answer:

1. What segment size minimizes total cost for representative workloads?
2. What row-group size gives the best pruning/scan balance?
3. How much value do Parquet-native statistics provide before custom synopsis data is added?
4. When are Bloom filters worth their size/build cost?
5. What sparse-field representation works best?
6. How well can pg_duckdb provide one logical SQL surface?
7. At what data/query scale does a dedicated worker pool outperform local execution?
8. How should free-text skipping work?
9. When should fields be promoted?
10. Can workload-guided physical adaptation produce measurable savings without operational instability?

---

26. Near-Term Build Sequence

Step 1
catalog schema

Step 2
synthetic event generator

Step 3
hot-tail ingestion

Step 4
Parquet segment writer

Step 5
segment publication

Step 6
DuckDB cold scan

Step 7
hot + cold logical query

Step 8
time pruning

Step 9
segment field statistics

Step 10
EXPLAIN / metrics

Step 11
Bloom synopsis prototype

Step 12
benchmark report

No distributed infrastructure is required to validate the core thesis.

---

27. Architecture Summary

plenty-gobble is built around a deliberate division of labor:

PostgreSQL
    = authority, catalog, policy, tenancy, SQL personality

Parquet
    = immutable columnar representation

Object storage
    = cheap durable capacity

Synopsis
    = proof that data need not be read

DuckDB
    = vectorized analytical execution

Workers
    = optional elastic parallelism

Workload telemetry
    = feedback for future physical optimization

The defining performance strategy is:

Prune first, project second, scan third, distribute only when necessary.

The defining systems strategy is:

Keep PostgreSQL as the coherent brain of the system while allowing physical storage and execution to specialize around append-only analytical data.