Architecture blueprint v0.1

Bounded context

content-explorer is a read/explore projection over one or more source dataspaces; not an authoritative universal knowledge graph. Source adapters produce candidate facts; a normalization and governance pipeline publishes immutable queryable snapshots. The interface sees only the snapshot contract and a declarative presentation manifest.

Git Markdown ─┐
JSON/NDJSON ──┼─> connector adapters ─> typed staging ─> identity/rights/evidence ─> graph validator ─> snapshot
future API ───┘                           |                       |               |
                                       quarantine              report        manifest
                                                                               |
                        +---------------- static JSON/read API ----------------+
                        |                              |                       |
                     Next.js UI                 search/graph views           future CLI/MCP

Canonical layers

Source artifact: exact immutable locator and content hash, acquired/observed timestamp, origin license, optional source-specific ID.
Assertion/Claim: typed subject–predicate–object or field value, linked evidence, status, confidence (if available), temporal validFrom/validTo, provenance and extractor version. Attribute displayed on entity may be an aggregation of claims; don't discard disagreement.
Entity/Relation projection: globally qualified stable identifiers, domain-local kind IDs, labels, TL;DR, summary, curated properties, typed relations and incoming derived backlinks. Nonsemantic related edges are allowed but distinguished.
Presentation: dataspace manifest defines allowed kinds, labels, facets, grouping, column specs, navigation, field layout, cards and optional widgets.
Transport/export: versioned snapshot JSON and OpenAPI later; no raw connector type reaches the UI.

Identity

Canonical IDs: urn:ce:<dataspace>:<kind>:<key> using normalized escaped segments or deterministic hashes when keys cannot meet the grammar. External IDs live separately as aliases. Two identical upstream IDs in different spaces cannot collide. Same-as merging is explicit with mapping confidence and curator decision; it must not happen from name similarity alone. Preserve source native identifiers for replay.

Relations

Each relation has its own ID, source, target, predicate (manifest-defined), labels for both directions, evidence and time. Store directed edges only once; derive reverse navigation. Allow many-to-many, multiple relation types between nodes, multi-parent hierarchies and cycles; selectively enforce acyclic property for declared taxonomic predicates only.

Claims, truth and evidence

Claims are source statements, not verified universal facts. Each projected display field optionally refers to one or more claim IDs; policies choose preferred, show-all, contested or withhold. Evidence includes source URI, document title, retrieved/verified timestamps, optional location/quote, license and attribution. UI exposes missing versus false versus stale versus disputed, without fake confidence scores.

Connector protocol

discover() -> SourceDescriptor[]; fetch(cursor?) -> SourceBatch; project(batch, mapping) -> ImportCandidate[]; checkpoint() for deterministic repeat. Adapter declares capabilities (full snapshot vs incremental, read-only, rights metadata, deletions), sync semantics, source authority and supported MIME/media types. Every run persists: connector version, input digest, output digest, insert/update/tombstone/quarantine counts and diagnostics. In v0.1 implement as build-time CLI jobs, no service dependency.

Ingestion modes: authoritative snapshot: missing source records tombstoned only within that source's ownership boundary; incremental: delete only on explicit tombstone; replay: same batch is idempotent. Namespace assertions per connector+source and preserve change history in report. Fail closed for forbidden rights and known dangling references, quarantine optional low-integrity records when policy allows.

Configurable views

Manifest owns home, kinds, relations, navigation, facets, columns, displayPanels, readingPaths, locale, theme. UI components know generic Entity, Facet, Claim, Link, Collection; any widget requiring domain-specific fields is an optional registered plugin with schema validation and fallback rendering.

Trust and deployment

Begin with static snapshots generated in CI for reproducibility, modest cost and safe public read. Source data is untrusted input: URL allowlist/scheme checks; HTML sanitization; bounded document size; no dynamic evaluation; prevent SSRF in future remote fetch adapters. No personal/sensitive data for early public demo. Record source copyright/attribution and policy before publishing. Signed/hashed snapshots and dependency scanning before production.

Quality measures

Model independence: 2 domain manifests, identical app build; zero code changes between.
Graph integrity: 100% IDs resolved or quarantined with diagnostics; no silent edge drops.
Provenance: 100% externally sourced displayed claims traceable to a source and rights record.
Navigation: browse, filter, search, detail, backlink, source, browser back and deep links pass e2e.
Performance budgets to baseline: static initial view and graph neighborhood size; set numeric SLOs after measuring actual upstream baseline.

Architectural trade-offs

Start with strongly typed core envelope and flexible per-kind properties; resist a rigid cross-domain ontology. Treat external RDF/JSON-LD as mapping targets, not prerequisite storage. Avoid graph DB until query or volume evidence demands one. Persist raw inputs separately from normalized snapshots. Never equate source trust with correctness of individual assertions.

5.9 KiB Raw Permalink Blame History Unescape Escape