Connector protocol and governance

Adapter boundary: source format -> candidate entities, claims, relations, sources -> validated canonical snapshot. Connectors must not render or import React components.

Minimum connector metadata

id, version, sourceNamespace, capabilities, licensePolicy, mappingVersion, refreshMode, sourcePriority, lastCheckpoint.

Functions

discover(config) returns stable enumerations if supported.
fetch(config, checkpoint) yields batches with immutable locators and input digests.
map(batch) emits candidates and issues; preserve raw provenance.
validate(candidates, manifest) checks shape, IDs, rights, provenance, references.
commit(validated, priorSnapshot) atomically publishes snapshot/report or rolls back.

Candidate mapping

Input → source record → canonical id → kind → fields/claims → relations. Relationships may be forward references; resolve in a second pass. No fuzzy join without explicit mapping policy. Respect deletion semantics by source ownership. Retries + replay must converge.

First connectors

JSON/NDJSON: local fixture and remote static file ingestion (local first). A simple reference mapping implementation is in connectors/json_fixture.py.
Markdown+frontmatter: repo text with typed metadata and wiki-style references; parse via a robust library, not ad-hoc regex in production. Preserve headings/source locations and content license.
GitHub or Wikidata (P2): authenticated/rate-limited external API snapshot; mapping rules, pagination, ETag and license review before activation.

Required diagnostics

runId, connector/version, started/ended UTC, input digests, source counts, candidates, accepted, updated, tombstoned, quarantined, warning/error codes, rights blocks, unresolved target IDs and published snapshot digest. Generate a human-readable provenance/rights report with each snapshot.

1.9 KiB Raw Blame History