coulomb/content-explorer
5
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
content-explorer/architecture/ConnectorContract.md
tegwick 072a7fa550 Freeze the v0.1 contract and the local ingest connector. 
The snapshot and manifest schemas, both example dataspaces, and the shared text renderer now pass the fixture validator and the negative contract tests. JSON, NDJSON, and Markdown frontmatter publish a snapshot only when rights, identity, and references allow it. CONTENT-WP-0001 and CONTENT-WP-0003 are marked finished against those artifacts. CONTENT-WP-0000-T04 stays open. SCOPE.md and the 19:05 scope-against-intent note record the stop.

Assistant: grok
Assistant-Session: 01a130f1-abfb-7d71-9cd6-a8b75ec7f95a
2026-10-12 19:07:32 +02:00

2 KiB
Raw Permalink Blame History

Connector protocol and governance

Adapter boundary: source format -> candidate entities, claims, relations, sources -> validated canonical snapshot. Connectors must not render or import React components.

Minimum connector metadata

id, version, sourceNamespace, capabilities, licensePolicy, mappingVersion, refreshMode, sourcePriority, lastCheckpoint.

Functions

  • discover(config) returns stable enumerations if supported.
  • fetch(config, checkpoint) yields batches with immutable locators and input digests.
  • map(batch) emits candidates and issues; preserve raw provenance.
  • validate(candidates, manifest) checks shape, IDs, rights, provenance, references.
  • commit(validated, priorSnapshot) atomically publishes snapshot/report or rolls back.

Candidate mapping

Input → source record → canonical id → kind → fields/claims → relations. Relationships may be forward references; resolve in a second pass. No fuzzy join without explicit mapping policy. Respect deletion semantics by source ownership. Retries + replay must converge.

First connectors

  1. JSON/NDJSON: local fixture ingestion. The build-time connector is content_explorer/ingest.py (load_json_records, load_ndjson_records). connectors/json_fixture.py only reads an already published snapshot.
  2. Markdown+frontmatter: repo text with typed metadata. The same module loads it with PyYAML (load_markdown_records). Remote static files and wiki-style references are not implemented yet.
  3. GitHub or Wikidata (P2): authenticated/rate-limited external API snapshot; mapping rules, pagination, ETag and license review before activation.

Required diagnostics

runId, connector/version, started/ended UTC, input digests, source counts, candidates, accepted, updated, tombstoned, quarantined, warning/error codes, rights blocks, unresolved target IDs and published snapshot digest. Generate a human-readable provenance/rights report with each snapshot.