Freeze the v0.1 contract and the local ingest connector.

The snapshot and manifest schemas, both example dataspaces, and the shared text renderer now pass the fixture validator and the negative contract tests. JSON, NDJSON, and Markdown frontmatter publish a snapshot only when rights, identity, and references allow it. CONTENT-WP-0001 and CONTENT-WP-0003 are marked finished against those artifacts. CONTENT-WP-0000-T04 stays open. SCOPE.md and the 19:05 scope-against-intent note record the stop.

Assistant: grok
Assistant-Session: 01a130f1-abfb-7d71-9cd6-a8b75ec7f95a
This commit is contained in:
tegwick 2026-10-12 19:07:32 +02:00
parent 2b1706559b
commit 072a7fa550
30 changed files with 1896 additions and 264 deletions

View file

@ -12,10 +12,10 @@ Infotech already has several topics. This repo is registered by slug and classif
| ID | Status | File |
|---|---|---|
| CONTENT-WP-0000 | active | `workplans/CONTENT-WP-0000-upstream-baseline.md` |
| CONTENT-WP-0001 | ready | `workplans/CONTENT-WP-0001-canonical-contracts.md` |
| CONTENT-WP-0000 | active (T04 still open) | `workplans/CONTENT-WP-0000-upstream-baseline.md` |
| CONTENT-WP-0001 | finished | `workplans/CONTENT-WP-0001-canonical-contracts.md` |
| CONTENT-WP-0002 | proposed | `workplans/CONTENT-WP-0002-ui-extraction.md` |
| CONTENT-WP-0003 | proposed | `workplans/CONTENT-WP-0003-ingestion-publishing.md` |
| CONTENT-WP-0003 | finished | `workplans/CONTENT-WP-0003-ingestion-publishing.md` |
| CONTENT-WP-0004 | proposed | `workplans/CONTENT-WP-0004-relations-journeys.md` |
| CONTENT-WP-0005 | proposed | `workplans/CONTENT-WP-0005-quality-parity.md` |
| CONTENT-WP-0006 | proposed | `workplans/CONTENT-WP-0006-external-pilot.md` |

2
.gitignore vendored
View file

@ -2,6 +2,8 @@ node_modules/
.next/
out/
upstream-onco/
__pycache__/
*.pyc
*.log
.env*

View file

@ -14,6 +14,7 @@ This checkout is a research seed, not a running application. There is no install
```sh
python3 scripts/validate_fixtures.py
python3 -m unittest tests.test_contract tests.test_ingest
```
## State Hub

View file

@ -18,6 +18,7 @@ Research grounded in publicly browsable upstream files, extraction plan, normati
```sh
python3 scripts/validate_fixtures.py
python3 -m unittest tests.test_contract tests.test_ingest
```
## Implementation start

42
SCOPE.md Normal file
View file

@ -0,0 +1,42 @@
# SCOPE — content-explorer
**As of:** 2026-10-12 19:05 CEST
**Stop:** Implementation stopped at 19:05 CEST, before 24:00. No later workplan had its prerequisites met. CONTENT-WP-0007 stays blocked.
This file describes the tree at that stop. It is the scope a reader can compare with `INTENT.md`. The comparison itself is `history/2026-10-12-1905-scope-against-intent.md`.
## In scope now
- A research seed for a domain-independent knowledge explorer. OnCo (onco.cc) is the experience reference. MIT-licensed upstream code may be adapted from the pinned checkout. The checkout stays gitignored at `upstream-onco/`.
- Pinned upstream baseline and reuse decision for commit `b594d7433bf4e408b44215e25e5304696d6f840e`, recorded under `research/baseline/`.
- Frozen snapshot and manifest contracts at `schemaVersion` 0.1, fact policy, and migration note under `contracts/`.
- Two independently authored example dataspaces, `examples/dataspaces/systems-thinking` and `examples/dataspaces/software-architecture`, each with at least three kinds and eight entities.
- Executable checks: `python3 scripts/validate_fixtures.py`, `python3 -m unittest tests.test_contract`, and `python3 -m unittest tests.test_ingest`.
- A shared text renderer, `content_explorer/render.py`. It takes headings and labels from the snapshot and manifest. It is not a browser UI.
- A build-time connector, `content_explorer/ingest.py` version `0.1.0`, for local JSON, NDJSON, and Markdown frontmatter. It publishes a snapshot, a manifest, and machine-readable and human import reports only when rights, identity, and references allow it.
## Out of scope
- The OnCo corpus, `src/data`, logos, portraits, trademarks, and CC BY-NC data. None of those are vendored in this repo.
- A production `next build` of upstream OnCo, pixel-perfect parity, MCP, and a third connector (GitHub issues or Wikidata).
- A browser explorer: landing, faceted index, detail page, search, graph view, and a mouse or keyboard journey. The workstation had no Chrome, Chromium, Firefox, or Playwright launcher, so those screens were not captured.
- Remote fetch, pagination, and authenticated APIs.
- New State Hub topics, or workplans posted by hand to the hub API. Local workplan files remain the source of truth.
- Unblocking CONTENT-WP-0007. Its first task still waits on a human data-use authorization.
## Workplan status
| ID | Status in the workplan file | What the tree shows |
|---|---|---|
| CONTENT-WP-0000 | active, partial | T01–T03 and T05–T09 are done and cite the baseline report, pin, and license notes. T04 is still `todo`: no headless browser, so there is no screenshot or keyboard log. |
| CONTENT-WP-0001 | finished | Schema, both fixtures, negative graph checks, fact policy, and the text renderer. Tests import `content_explorer/contract.py` and `content_explorer/render.py`. |
| CONTENT-WP-0002 | proposed, not started | Depends on 0000 and 0001. 0000 is not finished while T04 is open, so this plan was not started. |
| CONTENT-WP-0003 | finished | JSON, NDJSON, and Markdown connectors, source digest, second-pass links, tombstones, identity conflicts, rights quarantine, and atomic publish of snapshot, manifest, and both reports. Evidence is `content_explorer/ingest.py`, `tests/test_ingest.py`, and `workplans/evidence/CONTENT-WP-0003-*.md`. |
| CONTENT-WP-0004 | proposed, not started | Depends on 0002 and 0003. 0002 has not started. |
| CONTENT-WP-0005 | proposed, not started | Depends on 0004. |
| CONTENT-WP-0006 | proposed, not started | Depends on 0005. A third connector was a non-goal for this evening. |
| CONTENT-WP-0007 | blocked | Depends on 0000, 0001, 0002, 0003, and 0005. T01 remains `wait` with `needs_human: true`. No OnCo reference slice was imported. |
## Checks at the stop
`python3 -m unittest tests.test_ingest tests.test_contract` and `python3 scripts/validate_fixtures.py` exited 0 at 19:01 CEST and again after the ingest manifest and conflict work. The saved log for the closing run is outside the repo, in the session scratch file `fixture-check.txt`.

View file

@ -1,6 +1,6 @@
# Canonical data model — draft contract v0.1
The JSON Schema in `contracts/snapshot.schema.json` is a **first executable draft**. It intentionally models a compact display projection; Claim/Evidence and policy semantics in this document are richer than the initial schema and must be implemented in WP-03.
The JSON Schema in `contracts/snapshot.schema.json` is frozen at `schemaVersion` 0.1, including claims. Producing those claims from an external source is still CONTENT-WP-0003. Display policy is `contracts/fact-policy.md`.
## Records

View file

@ -20,8 +20,8 @@ Input → `source record` → `canonical id` → `kind` → `fields/claims` →
## First connectors
1. **JSON/NDJSON**: local fixture and remote static file ingestion (local first). A simple reference mapping implementation is in `connectors/json_fixture.py`.
2. **Markdown+frontmatter**: repo text with typed metadata and wiki-style references; parse via a robust library, not ad-hoc regex in production. Preserve headings/source locations and content license.
1. **JSON/NDJSON**: local fixture ingestion. The build-time connector is `content_explorer/ingest.py` (`load_json_records`, `load_ndjson_records`). `connectors/json_fixture.py` only reads an already published snapshot.
2. **Markdown+frontmatter**: repo text with typed metadata. The same module loads it with PyYAML (`load_markdown_records`). Remote static files and wiki-style references are not implemented yet.
3. **GitHub or Wikidata** (P2): authenticated/rate-limited external API snapshot; mapping rules, pagination, ETag and license review before activation.
## Required diagnostics

View file

@ -0,0 +1,3 @@
"""Domain-independent snapshot contracts and rendering."""
__version__ = "0.1.0"

View file

@ -0,0 +1,236 @@
"""Validate a v0.1 snapshot against its manifest.
Publication checks live here. Callers pass documents they already loaded.
`validate_dataspace` is the file entry point used by `scripts/validate_fixtures.py`.
"""
from __future__ import annotations
import json
from collections import defaultdict
from pathlib import Path
from typing import Any
import jsonschema
ROOT = Path(__file__).resolve().parent.parent
SNAPSHOT_SCHEMA_PATH = ROOT / "contracts" / "snapshot.schema.json"
MANIFEST_SCHEMA_PATH = ROOT / "contracts" / "manifest.schema.json"
SUPPORTED_VERSIONS = frozenset({"0.1"})
ENTITY_ID_PREFIX = "urn:ce:"
def load_json(path: Path) -> Any:
return json.loads(path.read_text(encoding="utf-8"))
def _validator(schema_path: Path) -> jsonschema.Draft202012Validator:
schema = load_json(schema_path)
return jsonschema.Draft202012Validator(
schema,
format_checker=jsonschema.Draft202012Validator.FORMAT_CHECKER,
)
def schema_errors(document: dict, schema_path: Path) -> list[str]:
validator = _validator(schema_path)
found = sorted(validator.iter_errors(document), key=lambda err: list(err.path))
messages = []
for err in found:
where = "/".join(str(part) for part in err.path) or "(root)"
messages.append(f"schema {where}: {err.message}")
return messages
def version_error(document: dict, label: str) -> str | None:
version = document.get("schemaVersion")
if version in SUPPORTED_VERSIONS:
return None
return (
f"{label} schemaVersion {version!r} is not supported; "
f"expected one of {', '.join(sorted(SUPPORTED_VERSIONS))}"
)
def resolve_claims(claims: list[dict], key: str, policy: str) -> dict[str, Any]:
"""Choose what a display field may show for one subject.
`missing` means no claim exists. `unknown` means a claim records that the
value is not known. Differing asserted values are a contradiction. The
manifest policy `show-all` keeps every asserted value, `withhold` keeps
none, and `preferred` keeps the claim marked preferred or withholds when
none is marked.
"""
relevant = [claim for claim in claims if claim.get("key") == key]
if not relevant:
return {"state": "missing", "values": []}
asserted = [claim for claim in relevant if claim.get("status") == "asserted"]
values = []
for claim in asserted:
value = claim.get("value")
if value not in values:
values.append(value)
if len(values) > 1:
if policy == "show-all":
return {"state": "contradicted", "values": values}
if policy == "preferred":
chosen = [claim.get("value") for claim in asserted if claim.get("preferred") is True]
if len(chosen) == 1:
return {"state": "present", "values": chosen}
return {"state": "contradicted", "values": []}
return {"state": "contradicted", "values": []}
if values:
return {"state": "present", "values": values}
if any(claim.get("status") == "unknown" for claim in relevant):
return {"state": "unknown", "values": []}
if any(claim.get("status") == "withheld" for claim in relevant):
return {"state": "withheld", "values": []}
return {"state": "missing", "values": []}
def _cycles(snapshot: dict, predicates: set[str]) -> list[str]:
if not predicates:
return []
adjacency: dict[str, list[tuple[str, str]]] = defaultdict(list)
for relation in snapshot.get("relations", []):
if relation.get("predicate") in predicates:
adjacency[relation["from"]].append((relation["to"], relation["id"]))
color: dict[str, int] = {}
errors: list[str] = []
def walk(node: str) -> None:
color[node] = 1
for nxt, relation_id in adjacency.get(node, []):
state = color.get(nxt, 0)
if state == 1:
errors.append(
f"taxonomic cycle via {relation_id}: {node} -> {nxt}"
)
elif state == 0:
walk(nxt)
color[node] = 2
for start in list(adjacency):
if color.get(start, 0) == 0:
walk(start)
return errors
def graph_errors(snapshot: dict, manifest: dict) -> list[str]:
errors: list[str] = []
version = version_error(snapshot, "snapshot")
if version:
errors.append(version)
manifest_version = version_error(manifest, "manifest")
if manifest_version:
errors.append(manifest_version)
kinds = [item.get("id") for item in snapshot.get("kinds", [])]
if len(kinds) != len(set(kinds)):
errors.append("duplicate kind id")
kind_ids = set(kinds)
entity_ids = [item.get("id") for item in snapshot.get("entities", [])]
if len(entity_ids) != len(set(entity_ids)):
errors.append("duplicate entity id")
entities = {item.get("id"): item for item in snapshot.get("entities", []) if item.get("id")}
sources = {item.get("id") for item in snapshot.get("sources", [])}
claim_ids = [item.get("id") for item in snapshot.get("claims", [])]
if len(claim_ids) != len(set(claim_ids)):
errors.append("duplicate claim id")
dataspace_id = (snapshot.get("dataspace") or {}).get("id")
if manifest.get("id") != dataspace_id:
errors.append(
f"manifest id {manifest.get('id')!r} does not match dataspace {dataspace_id!r}"
)
labels = manifest.get("relationLabels") or {}
views = manifest.get("kindViews") or {}
for kind_id in kind_ids:
if kind_id not in views:
errors.append(f"kind {kind_id} has no manifest kindView")
for featured in (manifest.get("home") or {}).get("featuredKinds", []):
if featured not in kind_ids:
errors.append(f"featured kind {featured} is not in the snapshot")
for entity in snapshot.get("entities", []):
entity_id = entity.get("id", "")
if not str(entity_id).startswith(ENTITY_ID_PREFIX):
errors.append(f"entity id {entity_id} is not a urn:ce identifier")
if entity.get("kindId") not in kind_ids:
errors.append(f"{entity_id}: undeclared kind {entity.get('kindId')}")
if not entity.get("sourceRefs"):
errors.append(f"{entity_id}: missing source")
for source_id in entity.get("sourceRefs", []):
if source_id not in sources:
errors.append(f"{entity_id}: missing source {source_id}")
for claim_id in entity.get("claimRefs", []):
if claim_id not in claim_ids:
errors.append(f"{entity_id}: missing claim {claim_id}")
known_entities = set(entities)
for relation in snapshot.get("relations", []):
relation_id = relation.get("id")
predicate = relation.get("predicate")
if predicate not in labels:
errors.append(f"relation {relation_id}: predicate {predicate} has no label")
for field in ("from", "to"):
target = relation.get(field)
if target not in known_entities:
errors.append(f"relation {relation_id}: dangling {field} {target}")
refs = relation.get("sourceRefs") or []
if not refs:
errors.append(f"relation {relation_id}: missing source")
for source_id in refs:
if source_id not in sources:
errors.append(f"relation {relation_id}: missing source {source_id}")
for claim in snapshot.get("claims", []):
claim_id = claim.get("id")
subject = claim.get("subjectId")
if subject not in known_entities:
errors.append(f"claim {claim_id}: dangling subject {subject}")
evidence = claim.get("evidenceRefs") or []
if not evidence:
errors.append(f"claim {claim_id}: missing source")
for source_id in evidence:
if source_id not in sources:
errors.append(f"claim {claim_id}: missing source {source_id}")
acyclic = set(manifest.get("acyclicPredicates") or [])
errors.extend(_cycles(snapshot, acyclic))
return errors
def size_errors(snapshot: dict, *, min_kinds: int = 3, min_entities: int = 8) -> list[str]:
errors = []
kind_count = len(snapshot.get("kinds") or [])
entity_count = len(snapshot.get("entities") or [])
if kind_count < min_kinds:
errors.append(f"needs at least {min_kinds} kinds, found {kind_count}")
if entity_count < min_entities:
errors.append(f"needs at least {min_entities} entities, found {entity_count}")
return errors
def validate_documents(
snapshot: dict,
manifest: dict,
*,
require_size: bool = False,
) -> list[str]:
errors = schema_errors(snapshot, SNAPSHOT_SCHEMA_PATH)
errors.extend(schema_errors(manifest, MANIFEST_SCHEMA_PATH))
errors.extend(graph_errors(snapshot, manifest))
if require_size:
errors.extend(size_errors(snapshot))
return errors
def validate_dataspace(directory: Path, *, require_size: bool = False) -> list[str]:
snapshot_path = directory / "snapshot.json"
manifest_path = directory / "manifest.json"
snapshot = load_json(snapshot_path)
manifest = load_json(manifest_path)
errors = validate_documents(snapshot, manifest, require_size=require_size)
return [f"{directory.name}: {message}" for message in errors]

371
content_explorer/ingest.py Normal file
View file

@ -0,0 +1,371 @@
"""Build a v0.1 snapshot from JSON, NDJSON, or Markdown frontmatter.
Connectors stop at a snapshot and a report. They do not render.
"""
from __future__ import annotations
import hashlib
import json
import os
import shutil
from pathlib import Path
from typing import Any
import yaml
CONNECTOR_VERSION = "0.1.0"
ALLOWED_LICENSES = frozenset({"CC0-1.0", "CC-BY-4.0", "MIT"})
def _digest(payload: bytes) -> str:
return hashlib.sha256(payload).hexdigest()
def snapshot_hash(snapshot: dict) -> str:
encoded = json.dumps(snapshot, sort_keys=True, separators=(",", ":")).encode("utf-8")
return _digest(encoded)
def load_json_records(path: Path) -> tuple[list[dict], str]:
raw = path.read_bytes()
document = json.loads(raw.decode("utf-8"))
if isinstance(document, dict) and "records" in document:
records = document["records"]
elif isinstance(document, list):
records = document
else:
raise ValueError(f"{path} is not a record list")
return records, _digest(raw)
def load_ndjson_records(path: Path) -> tuple[list[dict], str]:
raw = path.read_bytes()
records = []
for line in raw.decode("utf-8").splitlines():
if line.strip():
records.append(json.loads(line))
return records, _digest(raw)
def load_markdown_records(directory: Path) -> tuple[list[dict], str]:
parts: list[bytes] = []
records: list[dict] = []
for path in sorted(directory.glob("*.md")):
raw = path.read_bytes()
parts.append(raw)
text = raw.decode("utf-8")
if not text.startswith("---\n"):
raise ValueError(f"{path.name} has no frontmatter")
_, frontmatter, body = text.split("---", 2)
record = yaml.safe_load(frontmatter) or {}
if not isinstance(record, dict):
raise ValueError(f"{path.name} frontmatter is not a mapping")
if body.strip() and not record.get("summary"):
record["summary"] = body.strip()
records.append(record)
return records, _digest(b"".join(parts))
def _entity_id(dataspace: str, kind: str, key: str) -> str:
return f"urn:ce:{dataspace}:{kind}:{key}"
def _unpublished(report: dict) -> dict[str, Any]:
return {"snapshot": None, "manifest": None, "report": report}
def build_manifest(
*,
dataspace_id: str,
heading: str,
kinds: list[dict],
relation_labels: dict[str, dict[str, str]],
predicates: list[str],
fact_policy: str = "show-all",
acyclic_predicates: list[str] | None = None,
) -> tuple[dict[str, Any] | None, list[str]]:
"""Build a v0.1 manifest. Missing relation labels are returned and not invented."""
missing = sorted({predicate for predicate in predicates if predicate not in relation_labels})
if missing:
return None, missing
kind_ids = [kind["id"] for kind in kinds]
labels = {
predicate: {
"forward": relation_labels[predicate]["forward"],
"inverse": relation_labels[predicate]["inverse"],
}
for predicate in sorted(set(predicates))
}
manifest = {
"schemaVersion": "0.1",
"id": dataspace_id,
"factPolicy": fact_policy,
"home": {"heading": heading, "featuredKinds": kind_ids},
"navigation": [{"label": "Browse", "kinds": kind_ids}],
"kindViews": {
kind["id"]: {"columns": ["title"], "facets": ["tags"]}
for kind in kinds
},
"panels": ["summary", "relations", "evidence", "claims"],
"relationLabels": labels,
"acyclicPredicates": list(acyclic_predicates or []),
}
return manifest, []
def human_import_report(report: dict) -> str:
"""Provenance and rights summary a person can read next to the snapshot."""
source = report.get("source") or {}
lines = [
"# Import report",
"",
f"Connector version: {report.get('connectorVersion', '')}",
f"Source id: {source.get('id', '')}",
f"Source URI: {source.get('uri', '')}",
f"License: {source.get('license', '')}",
f"Attribution: {source.get('attribution') or '(none)'}",
f"Source digest: {report.get('sourceDigest') or '(none)'}",
f"Content hash: {report.get('contentHash') or '(not published)'}",
f"Published: {str(bool(report.get('published'))).lower()}",
f"Candidates: {report.get('candidates', 0)}",
f"Accepted: {report.get('accepted', 0)}",
"",
"## Tombstones",
]
tombstones = report.get("tombstoned") or []
lines.extend(f"- {key}" for key in tombstones)
if not tombstones:
lines.append("- (none)")
lines.extend(["", "## Quarantine"])
quarantined = report.get("quarantined") or []
if not quarantined:
lines.append("- (none)")
for item in quarantined:
lines.append(f"- {json.dumps(item, sort_keys=True)}")
lines.extend(["", "## Dangling references"])
dangling = report.get("dangling") or []
if not dangling:
lines.append("- (none)")
for item in dangling:
lines.append(f"- {item.get('from')} -{item.get('predicate')}-> {item.get('to')}")
lines.extend(["", "## Identity conflicts"])
conflicts = report.get("conflicts") or []
if not conflicts:
lines.append("- (none)")
for item in conflicts:
lines.append(f"- {item.get('key')}: {item.get('reason')}")
lines.append("")
return "\n".join(lines)
def project_records(
records: list[dict],
*,
dataspace_id: str,
title: str,
description: str,
source: dict,
kinds: list[dict],
observed_at: str,
source_digest: str = "",
relation_labels: dict[str, dict[str, str]] | None = None,
fact_policy: str = "show-all",
acyclic_predicates: list[str] | None = None,
) -> dict[str, Any]:
"""Return a report. `snapshot` is present only when publication is allowed.
Live keys are collected first. Relations resolve in a second pass against
that set. Duplicate keys and a live key that is also tombstoned are conflicts.
"""
license_name = str(source.get("license") or "")
report: dict[str, Any] = {
"connectorVersion": CONNECTOR_VERSION,
"candidates": len(records),
"accepted": 0,
"tombstoned": [],
"quarantined": [],
"dangling": [],
"conflicts": [],
"published": False,
"sourceDigest": source_digest,
"source": {
"id": source.get("id", ""),
"uri": source.get("uri", ""),
"title": source.get("title", ""),
"license": license_name,
"attribution": source.get("attribution") or "",
},
}
if license_name not in ALLOWED_LICENSES:
report["quarantined"].append({
"reason": "rights",
"license": license_name or "(missing)",
})
return _unpublished(report)
if license_name == "CC-BY-4.0" and not str(source.get("attribution") or "").strip():
report["quarantined"].append({
"reason": "rights",
"license": license_name,
"detail": "missing-attribution",
})
return _unpublished(report)
kind_ids = {kind["id"] for kind in kinds}
tombstoned_keys = set()
for record in records:
if record.get("deleted"):
report["tombstoned"].append(record.get("key"))
tombstoned_keys.add(f"{record.get('kind')}:{record.get('key')}")
grouped: dict[str, list[dict]] = {}
for record in records:
if record.get("deleted"):
continue
kind = record["kind"]
key = record["key"]
qualified = f"{kind}:{key}"
if kind not in kind_ids:
report["quarantined"].append({"reason": "unknown-kind", "key": key, "kind": kind})
continue
if qualified in tombstoned_keys:
report["conflicts"].append({"key": qualified, "reason": "tombstone-and-live"})
continue
grouped.setdefault(qualified, []).append(record)
accepted: dict[str, dict] = {}
for qualified, group in grouped.items():
if len(group) > 1:
report["conflicts"].append({
"key": qualified,
"reason": "duplicate-identity",
"count": len(group),
})
continue
accepted[qualified] = group[0]
entities = []
relations = []
claims = []
predicates: list[str] = []
# Second pass: links resolve only against keys accepted in the first pass.
for qualified, record in accepted.items():
kind, key = qualified.split(":", 1)
entity_id = _entity_id(dataspace_id, kind, key)
for link in record.get("relations") or []:
target = link.get("to")
predicate = link.get("predicate")
if target not in accepted:
report["dangling"].append({"from": qualified, "to": target, "predicate": predicate})
claim_id = f"claim:{kind}:{key}:tldr"
entities.append({
"id": entity_id,
"kindId": kind,
"title": record["title"],
"tldr": record["tldr"],
"summary": record.get("summary") or record["tldr"],
"tags": list(record.get("tags") or []),
"sourceRefs": [source["id"]],
"claimRefs": [claim_id],
"asOf": observed_at,
})
claims.append({
"id": claim_id,
"subjectId": entity_id,
"key": "tldr",
"value": record["tldr"],
"status": "asserted",
"evidenceRefs": [source["id"]],
"observedAt": observed_at,
})
for index, link in enumerate(record.get("relations") or [], start=1):
target = link["to"]
predicate = link["predicate"]
predicates.append(predicate)
target_kind, target_key = target.split(":", 1)
relations.append({
"id": f"{kind}-{key}-{index}",
"predicate": predicate,
"from": entity_id,
"to": _entity_id(dataspace_id, target_kind, target_key),
"sourceRefs": [source["id"]],
"asOf": observed_at,
})
blocked = (
report["dangling"]
or report["conflicts"]
or any(item["reason"] == "unknown-kind" for item in report["quarantined"])
)
if blocked:
return _unpublished(report)
manifest, missing_labels = build_manifest(
dataspace_id=dataspace_id,
heading=title,
kinds=kinds,
relation_labels=relation_labels or {},
predicates=predicates,
fact_policy=fact_policy,
acyclic_predicates=acyclic_predicates,
)
if missing_labels:
report["quarantined"].append({
"reason": "missing-relation-label",
"predicates": missing_labels,
})
return _unpublished(report)
entities.sort(key=lambda item: item["id"])
relations.sort(key=lambda item: item["id"])
claims.sort(key=lambda item: item["id"])
snapshot = {
"schemaVersion": "0.1",
"dataspace": {"id": dataspace_id, "title": title, "description": description},
"kinds": kinds,
"entities": entities,
"relations": relations,
"sources": [source],
"claims": claims,
}
for claim in claims:
if not claim["evidenceRefs"]:
report["quarantined"].append({"reason": "missing-evidence", "claim": claim["id"]})
return _unpublished(report)
report["accepted"] = len(entities)
report["published"] = True
report["contentHash"] = snapshot_hash(snapshot)
return {"snapshot": snapshot, "manifest": manifest, "report": report}
def publish_directory(destination: Path, result: dict[str, Any]) -> None:
"""Replace `destination` only after the snapshot, manifest, and both reports exist."""
if not result["report"]["published"] or result["snapshot"] is None or result.get("manifest") is None:
raise RuntimeError("refusing to publish a quarantined or dangling import")
destination.parent.mkdir(parents=True, exist_ok=True)
temporary = destination.parent / f".{destination.name}.incoming"
if temporary.exists():
shutil.rmtree(temporary)
temporary.mkdir()
(temporary / "snapshot.json").write_text(
json.dumps(result["snapshot"], indent=2) + "\n",
encoding="utf-8",
)
(temporary / "manifest.json").write_text(
json.dumps(result["manifest"], indent=2) + "\n",
encoding="utf-8",
)
(temporary / "import-report.json").write_text(
json.dumps(result["report"], indent=2) + "\n",
encoding="utf-8",
)
(temporary / "import-report.md").write_text(
human_import_report(result["report"]),
encoding="utf-8",
)
backup = destination.parent / f".{destination.name}.backup"
if backup.exists():
shutil.rmtree(backup)
if destination.exists():
os.rename(destination, backup)
os.rename(temporary, destination)

View file

@ -0,0 +1,60 @@
"""Render a snapshot with whatever labels the manifest supplies.
The text below is the shared explorer. It must not name a dataspace, a kind,
or a predicate. Those strings come from the snapshot and the manifest.
"""
from __future__ import annotations
from content_explorer.contract import resolve_claims
def render_explorer(snapshot: dict, manifest: dict) -> str:
kinds = {kind["id"]: kind for kind in snapshot.get("kinds", [])}
entities = {entity["id"]: entity for entity in snapshot.get("entities", [])}
claims = snapshot.get("claims", [])
policy = manifest.get("factPolicy", "show-all")
lines: list[str] = [manifest["home"]["heading"], ""]
for section in manifest.get("navigation", []):
names = [kinds[kind_id]["plural"] for kind_id in section["kinds"] if kind_id in kinds]
lines.append(f"{section['label']}: {', '.join(names)}")
lines.append("")
for kind_id in manifest["home"]["featuredKinds"]:
kind = kinds.get(kind_id)
if kind is None:
continue
lines.append(f"## {kind['plural']}")
members = [entity for entity in snapshot.get("entities", []) if entity.get("kindId") == kind_id]
if not members:
lines.append("(none)")
for entity in members:
field = resolve_claims(
[claim for claim in claims if claim.get("subjectId") == entity["id"]],
"tldr",
policy,
)
if field["state"] == "unknown":
shown = "unknown"
elif field["state"] == "contradicted":
shown = "contested: " + " | ".join(str(value) for value in field["values"])
elif field["state"] == "withheld":
shown = "withheld"
elif field["values"]:
shown = str(field["values"][0])
else:
shown = entity.get("tldr", "")
lines.append(f"- {entity['title']}: {shown}")
lines.append("")
lines.append("## Relations")
labels = manifest.get("relationLabels") or {}
for relation in snapshot.get("relations", []):
predicate = relation["predicate"]
wording = labels.get(predicate, {}).get("forward", predicate)
origin = entities.get(relation["from"], {}).get("title", relation["from"])
destination = entities.get(relation["to"], {}).get("title", relation["to"])
lines.append(f"- {origin} — {wording} → {destination}")
lines.append("")
return "\n".join(lines).rstrip() + "\n"

9
contracts/MIGRATIONS.md Normal file
View file

@ -0,0 +1,9 @@
# Snapshot and manifest migrations
## 0.1 — frozen 2026-10-12
This is the first supported `schemaVersion`. The seed draft omitted `claims`, required relation `sourceRefs`, manifest `schemaVersion`, `relationLabels`, and `factPolicy`. Those documents were edited in place. There is no 0.0 reader.
`content_explorer.contract.SUPPORTED_VERSIONS` accepts only `0.1`. A document with any other version fails validation. A later version must add a named migrator, for example `migrate_0_1_to_0_2`, and must not guess missing rights or claims.
Display text on an entity is not a second source of truth. A `tldr` claim with `status: asserted` is the statement the renderer prefers to show. `status: unknown` records that the field was considered and is not known.

12
contracts/SNAPSHOT.md Normal file
View file

@ -0,0 +1,12 @@
# Snapshot format 0.1
Normative files:
- `contracts/snapshot.schema.json` — entity, relation, source, and claim document.
- `contracts/manifest.schema.json` — presentation. Kind names, facet names, and relation wording live here.
- `contracts/MIGRATIONS.md` — version gate.
- `contracts/fact-policy.md` — missing, unknown, and contradiction.
Identifiers for entities are `urn:ce:<dataspace>:<kind>:<key>`. The same key in another dataspace is a different entity. Relation ids are local to the snapshot.
A published example must contain at least three kinds and eight entities. `scripts/validate_fixtures.py` enforces that, plus the schema, dangling references, duplicate ids, missing sources, and cycles on `acyclicPredicates` only.

19
contracts/fact-policy.md Normal file
View file

@ -0,0 +1,19 @@
# Fact policy v0.1
A claim is a statement about one entity field. It is not a proof. The snapshot keeps every claim. The manifest `factPolicy` only decides what a display may show.
| Situation | Claim shape | `resolve_claims` state |
|---|---|---|
| No claim for that key | absent | `missing` |
| The field was checked and has no value | `status: unknown` | `unknown` |
| One asserted value | `status: asserted` | `present` |
| The holder asked not to display it | `status: withheld` | `withheld` |
| Two asserted values | both `status: asserted` | `contradicted` |
Policies when values disagree:
- `show-all` returns every asserted value and state `contradicted`.
- `withhold` returns no values and state `contradicted`.
- `preferred` returns the single claim with `preferred: true`. If none or more than one is marked, it returns no values and state `contradicted`.
Absence is not the same as unknown, and unknown is not the same as false. A relation with no `sourceRefs` is invalid. It is not stored as an unsourced fact. Conflicting claims stay in the snapshot either way. The policy does not delete them.

View file

@ -1,7 +1,7 @@
{
"id": "systems-thinking",
"home": {
"heading": "Explore systems thinking",
"heading": "Explore Systems Thinking",
"featuredKinds": [
"concept",
"person",
@ -11,42 +11,97 @@
},
"navigation": [
{
"label": "Explore",
"label": "Browse",
"kinds": [
"concept",
"method",
"person",
"publication",
"person"
"method"
]
}
],
"kindViews": {
"concept": {
"facets": [
"discipline",
"tradition"
],
"columns": [
"name",
"tldr",
"tags"
],
"facets": [
"tags"
]
}
},
"relationLabels": {
"influences": {
"forward": "Influences",
"inverse": "Influenced by"
},
"discusses": {
"forward": "Discusses",
"inverse": "Discussed in"
"person": {
"columns": [
"name",
"tldr",
"tags"
],
"facets": [
"tags"
]
},
"publication": {
"columns": [
"name",
"tldr",
"tags"
],
"facets": [
"tags"
]
},
"method": {
"columns": [
"name",
"tldr",
"tags"
],
"facets": [
"tags"
]
}
},
"panels": [
"summary",
"relations",
"evidence"
"evidence",
"claims"
],
"schemaVersion": "0.1",
"factPolicy": "show-all",
"relationLabels": {
"represented-by": {
"forward": "Represented by",
"inverse": "Represents"
},
"scopes": {
"forward": "Scopes",
"inverse": "Scoped by"
},
"associated-with": {
"forward": "Associated with",
"inverse": "Associated with"
},
"authored-by": {
"forward": "Authored by",
"inverse": "Authored"
},
"related-to": {
"forward": "Related to",
"inverse": "Related to"
},
"models": {
"forward": "Models",
"inverse": "Modeled by"
},
"explores": {
"forward": "Explores",
"inverse": "Explored by"
}
},
"acyclicPredicates": [
"authored-by"
],
"theme": {
"accent": "indigo"

View file

@ -0,0 +1,73 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://content-explorer.local/schemas/manifest-0.1.json",
"title": "Content Explorer dataspace manifest v0.1",
"type": "object",
"additionalProperties": false,
"required": ["schemaVersion", "id", "home", "navigation", "kindViews", "panels", "relationLabels", "factPolicy"],
"properties": {
"schemaVersion": {"const": "0.1"},
"id": {"type": "string", "pattern": "^[a-z0-9-]+$"},
"home": {
"type": "object",
"additionalProperties": false,
"required": ["heading", "featuredKinds"],
"properties": {
"heading": {"type": "string", "minLength": 1},
"featuredKinds": {"type": "array", "items": {"type": "string"}, "minItems": 1}
}
},
"navigation": {
"type": "array",
"minItems": 1,
"items": {
"type": "object",
"additionalProperties": false,
"required": ["label", "kinds"],
"properties": {
"label": {"type": "string"},
"kinds": {"type": "array", "items": {"type": "string"}, "minItems": 1}
}
}
},
"kindViews": {
"type": "object",
"additionalProperties": {
"type": "object",
"additionalProperties": false,
"required": ["columns", "facets"],
"properties": {
"columns": {"type": "array", "items": {"type": "string"}, "minItems": 1},
"facets": {"type": "array", "items": {"type": "string"}}
}
}
},
"panels": {
"type": "array",
"items": {"enum": ["summary", "relations", "evidence", "claims"]}
},
"relationLabels": {
"type": "object",
"additionalProperties": {
"type": "object",
"additionalProperties": false,
"required": ["forward", "inverse"],
"properties": {
"forward": {"type": "string"},
"inverse": {"type": "string"}
}
}
},
"acyclicPredicates": {
"type": "array",
"items": {"type": "string"}
},
"factPolicy": {"enum": ["show-all", "preferred", "withhold"]},
"theme": {
"type": "object",
"additionalProperties": false,
"required": ["accent"],
"properties": {"accent": {"type": "string"}}
}
}
}

View file

@ -1,63 +1,33 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"title": "Content Explorer compact snapshot v0.1",
"$id": "https://content-explorer.local/schemas/snapshot-0.1.json",
"title": "Content Explorer snapshot v0.1",
"type": "object",
"required": [
"schemaVersion",
"dataspace",
"kinds",
"entities",
"relations",
"sources"
],
"additionalProperties": false,
"required": ["schemaVersion", "dataspace", "kinds", "entities", "relations", "sources", "claims"],
"properties": {
"schemaVersion": {
"const": "0.1"
},
"schemaVersion": {"const": "0.1"},
"dataspace": {
"type": "object",
"required": [
"id",
"title",
"description"
],
"additionalProperties": false,
"required": ["id", "title", "description"],
"properties": {
"id": {
"type": "string"
},
"title": {
"type": "string"
},
"description": {
"type": "string"
}
"id": {"type": "string", "pattern": "^[a-z0-9-]+$"},
"title": {"type": "string", "minLength": 1},
"description": {"type": "string", "minLength": 1}
}
},
"kinds": {
"type": "array",
"minItems": 1,
"items": {
"type": "object",
"required": [
"id",
"label",
"plural"
],
"additionalProperties": false,
"required": ["id", "label", "plural"],
"properties": {
"id": {
"type": "string"
},
"label": {
"type": "string"
},
"plural": {
"type": "string"
},
"facets": {
"type": "array",
"items": {
"type": "string"
}
}
"id": {"type": "string", "pattern": "^[a-z0-9-]+$"},
"label": {"type": "string"},
"plural": {"type": "string"}
}
}
},
@ -65,48 +35,22 @@
"type": "array",
"items": {
"type": "object",
"required": [
"id",
"kindId",
"title",
"tldr",
"summary",
"sourceRefs"
],
"additionalProperties": false,
"required": ["id", "kindId", "title", "tldr", "summary", "sourceRefs"],
"properties": {
"id": {
"type": "string"
},
"kindId": {
"type": "string"
},
"title": {
"type": "string"
},
"tldr": {
"type": "string"
},
"summary": {
"type": "string"
},
"tags": {
"type": "array",
"items": {
"type": "string"
}
},
"properties": {
"type": "object"
},
"sourceRefs": {
"type": "array",
"items": {
"type": "string"
}
},
"asOf": {
"type": "string",
"format": "date"
"id": {"type": "string", "pattern": "^urn:ce:[a-z0-9-]+:[a-z0-9-]+:[a-z0-9-]+$"},
"kindId": {"type": "string"},
"title": {"type": "string"},
"tldr": {"type": "string"},
"summary": {"type": "string"},
"aliases": {"type": "array", "items": {"type": "string"}},
"tags": {"type": "array", "items": {"type": "string"}},
"properties": {"type": "object"},
"sourceRefs": {"type": "array", "items": {"type": "string"}},
"claimRefs": {"type": "array", "items": {"type": "string"}},
"asOf": {"type": "string", "format": "date"},
"state": {
"enum": ["present", "missing", "unknown", "redacted", "stale", "contradicted", "not-applicable"]
}
}
}
@ -115,63 +59,52 @@
"type": "array",
"items": {
"type": "object",
"required": [
"id",
"predicate",
"from",
"to"
],
"additionalProperties": false,
"required": ["id", "predicate", "from", "to", "sourceRefs"],
"properties": {
"id": {
"type": "string"
},
"predicate": {
"type": "string"
},
"from": {
"type": "string"
},
"to": {
"type": "string"
},
"sourceRefs": {
"type": "array",
"items": {
"type": "string"
}
}
"id": {"type": "string"},
"predicate": {"type": "string", "pattern": "^[a-z0-9-]+$"},
"from": {"type": "string"},
"to": {"type": "string"},
"sourceRefs": {"type": "array", "items": {"type": "string"}, "minItems": 1},
"asOf": {"type": "string", "format": "date"}
}
}
},
"sources": {
"type": "array",
"minItems": 1,
"items": {
"type": "object",
"required": [
"id",
"uri",
"title",
"license"
],
"additionalProperties": false,
"required": ["id", "uri", "title", "license"],
"properties": {
"id": {
"type": "string"
},
"uri": {
"type": "string"
},
"title": {
"type": "string"
},
"license": {
"type": "string"
},
"attribution": {
"type": "string"
}
"id": {"type": "string"},
"uri": {"type": "string"},
"title": {"type": "string"},
"license": {"type": "string"},
"attribution": {"type": "string"},
"retrievedAt": {"type": "string", "format": "date"}
}
}
},
"claims": {
"type": "array",
"items": {
"type": "object",
"additionalProperties": false,
"required": ["id", "subjectId", "key", "status", "evidenceRefs"],
"properties": {
"id": {"type": "string"},
"subjectId": {"type": "string"},
"key": {"type": "string"},
"value": {},
"status": {"enum": ["asserted", "unknown", "contested", "withheld"]},
"preferred": {"type": "boolean"},
"evidenceRefs": {"type": "array", "items": {"type": "string"}, "minItems": 1},
"observedAt": {"type": "string", "format": "date"}
}
}
}
},
"additionalProperties": false
}
}

View file

@ -65,6 +65,42 @@
"panels": [
"summary",
"relations",
"evidence"
"evidence",
"claims"
],
"schemaVersion": "0.1",
"factPolicy": "show-all",
"relationLabels": {
"adapts": {
"forward": "Adapts",
"inverse": "Adapted by"
},
"uses": {
"forward": "Uses",
"inverse": "Used by"
},
"addressed-by": {
"forward": "Addressed by",
"inverse": "Addresses"
},
"related-to": {
"forward": "Related to",
"inverse": "Related to"
},
"documents": {
"forward": "Documents",
"inverse": "Documented by"
},
"visualizes": {
"forward": "Visualizes",
"inverse": "Visualized by"
},
"affects": {
"forward": "Affects",
"inverse": "Affected by"
}
},
"acyclicPredicates": [
"documents"
]
}

View file

@ -3,7 +3,7 @@
"dataspace": {
"id": "software-architecture",
"title": "Software Architecture",
"description": "A small illustrative collection of patterns and ideas in software design"
"description": "A small illustrative collection of patterns and ideas in software design. The sentences were written for this demo and are not quotations from another corpus."
},
"kinds": [
{
@ -42,108 +42,153 @@
"kindId": "concept",
"title": "Coupling",
"tldr": "The degree to which software elements depend on one another.",
"summary": "The degree to which software elements depend on one another. This fixture is illustrative and does not constitute a sourced scholarly claim.",
"tags": [],
"summary": "Coupling is how much one software element must know about another in order to work.",
"tags": [
"dependency"
],
"sourceRefs": [
"src:software-architecture:authored-demo"
],
"asOf": "2026-10-12"
"asOf": "2026-10-12",
"claimRefs": [
"claim:coupling:tldr"
]
},
{
"id": "urn:ce:software-architecture:concept:cohesion",
"kindId": "concept",
"title": "Cohesion",
"tldr": "The degree to which a module groups related responsibilities.",
"summary": "The degree to which a module groups related responsibilities. This fixture is illustrative and does not constitute a sourced scholarly claim.",
"tags": [],
"summary": "Cohesion is how closely the responsibilities inside one module belong together.",
"tags": [
"modularity"
],
"sourceRefs": [
"src:software-architecture:authored-demo"
],
"asOf": "2026-10-12"
"asOf": "2026-10-12",
"claimRefs": [
"claim:cohesion:tldr"
]
},
{
"id": "urn:ce:software-architecture:concept:interface",
"kindId": "concept",
"title": "Interface contract",
"tldr": "An agreed description of interaction between components.",
"summary": "An agreed description of interaction between components. This fixture is illustrative and does not constitute a sourced scholarly claim.",
"tags": [],
"summary": "An interface contract names the interactions two parts are allowed to rely on.",
"tags": [
"boundary"
],
"sourceRefs": [
"src:software-architecture:authored-demo"
],
"asOf": "2026-10-12"
"asOf": "2026-10-12",
"claimRefs": [
"claim:interface:tldr"
]
},
{
"id": "urn:ce:software-architecture:pattern:adapter",
"kindId": "pattern",
"title": "Adapter pattern",
"tldr": "An adapter translates between interfaces.",
"summary": "An adapter translates between interfaces. This fixture is illustrative and does not constitute a sourced scholarly claim.",
"tags": [],
"summary": "An adapter lets a caller use one interface while the callee offers another.",
"tags": [
"translation"
],
"sourceRefs": [
"src:software-architecture:authored-demo"
],
"asOf": "2026-10-12"
"asOf": "2026-10-12",
"claimRefs": [
"claim:adapter:tldr"
]
},
{
"id": "urn:ce:software-architecture:pattern:ports-adapters",
"kindId": "pattern",
"title": "Ports and adapters",
"tldr": "A pattern for isolating core logic from outside systems.",
"summary": "A pattern for isolating core logic from outside systems. This fixture is illustrative and does not constitute a sourced scholarly claim.",
"tags": [],
"summary": "Ports and adapters keep domain logic from importing the details of a database, a UI, or a network.",
"tags": [
"isolation"
],
"sourceRefs": [
"src:software-architecture:authored-demo"
],
"asOf": "2026-10-12"
"asOf": "2026-10-12",
"claimRefs": [
"claim:ports-adapters:tldr"
]
},
{
"id": "urn:ce:software-architecture:pattern:event-driven",
"kindId": "pattern",
"title": "Event-driven architecture",
"tldr": "Software components communicate through events.",
"summary": "Software components communicate through events. This fixture is illustrative and does not constitute a sourced scholarly claim.",
"tags": [],
"summary": "In an event-driven arrangement, parts announce facts and other parts react, instead of calling each other directly.",
"tags": [
"communication"
],
"sourceRefs": [
"src:software-architecture:authored-demo"
],
"asOf": "2026-10-12"
"asOf": "2026-10-12",
"claimRefs": [
"claim:event-driven:tldr"
]
},
{
"id": "urn:ce:software-architecture:person:alexander",
"kindId": "person",
"title": "Christopher Alexander",
"tldr": "Architect whose pattern-language ideas influenced software design.",
"summary": "Architect whose pattern-language ideas influenced software design. This fixture is illustrative and does not constitute a sourced scholarly claim.",
"tags": [],
"summary": "Christopher Alexander described pattern languages for buildings. Software designers later borrowed that way of naming recurring forms.",
"tags": [
"patterns"
],
"sourceRefs": [
"src:software-architecture:authored-demo"
],
"asOf": "2026-10-12"
"asOf": "2026-10-12",
"claimRefs": [
"claim:alexander:tldr"
]
},
{
"id": "urn:ce:software-architecture:artifact:adr",
"kindId": "artifact",
"title": "Architecture Decision Record",
"tldr": "A lightweight record of architectural decisions and rationale.",
"summary": "A lightweight record of architectural decisions and rationale. This fixture is illustrative and does not constitute a sourced scholarly claim.",
"tags": [],
"summary": "An architecture decision record states a decision, its context, and the consequences the writers accepted.",
"tags": [
"record"
],
"sourceRefs": [
"src:software-architecture:authored-demo"
],
"asOf": "2026-10-12"
"asOf": "2026-10-12",
"claimRefs": [
"claim:adr:tldr"
]
},
{
"id": "urn:ce:software-architecture:artifact:c4",
"kindId": "artifact",
"title": "C4 model diagram",
"tldr": "A set of views for communicating software architecture.",
"summary": "A set of views for communicating software architecture. This fixture is illustrative and does not constitute a sourced scholarly claim.",
"tags": [],
"summary": "The C4 model offers a small set of diagrams, from system context down to code, so different readers can stop at the level they need.",
"tags": [
"diagram"
],
"sourceRefs": [
"src:software-architecture:authored-demo"
],
"asOf": "2026-10-12"
"asOf": "2026-10-12",
"claimRefs": [
"claim:c4:tldr"
]
}
],
"relations": [
@ -210,5 +255,116 @@
"src:software-architecture:authored-demo"
]
}
],
"claims": [
{
"id": "claim:coupling:tldr",
"subjectId": "urn:ce:software-architecture:concept:coupling",
"key": "tldr",
"value": "The degree to which software elements depend on one another.",
"status": "asserted",
"evidenceRefs": [
"src:software-architecture:authored-demo"
],
"observedAt": "2026-10-12"
},
{
"id": "claim:cohesion:tldr",
"subjectId": "urn:ce:software-architecture:concept:cohesion",
"key": "tldr",
"value": "The degree to which a module groups related responsibilities.",
"status": "asserted",
"evidenceRefs": [
"src:software-architecture:authored-demo"
],
"observedAt": "2026-10-12"
},
{
"id": "claim:interface:tldr",
"subjectId": "urn:ce:software-architecture:concept:interface",
"key": "tldr",
"value": "An agreed description of interaction between components.",
"status": "asserted",
"evidenceRefs": [
"src:software-architecture:authored-demo"
],
"observedAt": "2026-10-12"
},
{
"id": "claim:adapter:tldr",
"subjectId": "urn:ce:software-architecture:pattern:adapter",
"key": "tldr",
"value": "An adapter translates between interfaces.",
"status": "asserted",
"evidenceRefs": [
"src:software-architecture:authored-demo"
],
"observedAt": "2026-10-12"
},
{
"id": "claim:ports-adapters:tldr",
"subjectId": "urn:ce:software-architecture:pattern:ports-adapters",
"key": "tldr",
"value": "A pattern for isolating core logic from outside systems.",
"status": "asserted",
"evidenceRefs": [
"src:software-architecture:authored-demo"
],
"observedAt": "2026-10-12"
},
{
"id": "claim:event-driven:tldr",
"subjectId": "urn:ce:software-architecture:pattern:event-driven",
"key": "tldr",
"value": "Software components communicate through events.",
"status": "asserted",
"evidenceRefs": [
"src:software-architecture:authored-demo"
],
"observedAt": "2026-10-12"
},
{
"id": "claim:alexander:tldr",
"subjectId": "urn:ce:software-architecture:person:alexander",
"key": "tldr",
"value": "Architect whose pattern-language ideas influenced software design.",
"status": "asserted",
"evidenceRefs": [
"src:software-architecture:authored-demo"
],
"observedAt": "2026-10-12"
},
{
"id": "claim:adr:tldr",
"subjectId": "urn:ce:software-architecture:artifact:adr",
"key": "tldr",
"value": "A lightweight record of architectural decisions and rationale.",
"status": "asserted",
"evidenceRefs": [
"src:software-architecture:authored-demo"
],
"observedAt": "2026-10-12"
},
{
"id": "claim:c4:tldr",
"subjectId": "urn:ce:software-architecture:artifact:c4",
"key": "tldr",
"value": "A set of views for communicating software architecture.",
"status": "asserted",
"evidenceRefs": [
"src:software-architecture:authored-demo"
],
"observedAt": "2026-10-12"
},
{
"id": "claim:unknown:numeric-scale",
"subjectId": "urn:ce:software-architecture:concept:coupling",
"key": "numeric-scale",
"status": "unknown",
"evidenceRefs": [
"src:software-architecture:authored-demo"
],
"observedAt": "2026-10-12"
}
]
}

View file

@ -65,6 +65,45 @@
"panels": [
"summary",
"relations",
"evidence"
]
"evidence",
"claims"
],
"schemaVersion": "0.1",
"factPolicy": "show-all",
"relationLabels": {
"represented-by": {
"forward": "Represented by",
"inverse": "Represents"
},
"scopes": {
"forward": "Scopes",
"inverse": "Scoped by"
},
"associated-with": {
"forward": "Associated with",
"inverse": "Associated with"
},
"authored-by": {
"forward": "Authored by",
"inverse": "Authored"
},
"related-to": {
"forward": "Related to",
"inverse": "Related to"
},
"models": {
"forward": "Models",
"inverse": "Modeled by"
},
"explores": {
"forward": "Explores",
"inverse": "Explored by"
}
},
"acyclicPredicates": [
"authored-by"
],
"theme": {
"accent": "indigo"
}
}

View file

@ -3,7 +3,7 @@
"dataspace": {
"id": "systems-thinking",
"title": "Systems Thinking",
"description": "A small illustrative collection of ideas in systems theory"
"description": "A small illustrative collection of ideas in systems theory. The sentences were written for this demo and are not quotations from another corpus."
},
"kinds": [
{
@ -42,108 +42,153 @@
"kindId": "concept",
"title": "Feedback",
"tldr": "Effects of actions change subsequent behavior.",
"summary": "Effects of actions change subsequent behavior. This fixture is illustrative and does not constitute a sourced scholarly claim.",
"tags": [],
"summary": "A feedback loop sends the result of an action back so the next action can change.",
"tags": [
"regulation"
],
"sourceRefs": [
"src:systems-thinking:authored-demo"
],
"asOf": "2026-10-12"
"asOf": "2026-10-12",
"claimRefs": [
"claim:feedback:tldr"
]
},
{
"id": "urn:ce:systems-thinking:concept:emergence",
"kindId": "concept",
"title": "Emergence",
"tldr": "Collective behavior arises through interactions.",
"summary": "Collective behavior arises through interactions. This fixture is illustrative and does not constitute a sourced scholarly claim.",
"tags": [],
"summary": "Emergence is behavior of a whole that is not obvious from one part looked at alone.",
"tags": [
"structure"
],
"sourceRefs": [
"src:systems-thinking:authored-demo"
],
"asOf": "2026-10-12"
"asOf": "2026-10-12",
"claimRefs": [
"claim:emergence:tldr"
]
},
{
"id": "urn:ce:systems-thinking:concept:variety",
"kindId": "concept",
"title": "Requisite variety",
"tldr": "Effective regulation requires sufficient response variety.",
"summary": "Effective regulation requires sufficient response variety. This fixture is illustrative and does not constitute a sourced scholarly claim.",
"tags": [],
"summary": "Requisite variety says a regulator needs enough responses to match the situations it faces.",
"tags": [
"regulation"
],
"sourceRefs": [
"src:systems-thinking:authored-demo"
],
"asOf": "2026-10-12"
"asOf": "2026-10-12",
"claimRefs": [
"claim:variety:tldr"
]
},
{
"id": "urn:ce:systems-thinking:concept:system-boundary",
"kindId": "concept",
"title": "System boundary",
"tldr": "A distinction determines what is examined as a system.",
"summary": "A distinction determines what is examined as a system. This fixture is illustrative and does not constitute a sourced scholarly claim.",
"tags": [],
"summary": "A system boundary is the distinction that decides which interactions count as inside the system.",
"tags": [
"boundary"
],
"sourceRefs": [
"src:systems-thinking:authored-demo"
],
"asOf": "2026-10-12"
"asOf": "2026-10-12",
"claimRefs": [
"claim:system-boundary:tldr"
]
},
{
"id": "urn:ce:systems-thinking:person:ashby",
"kindId": "person",
"title": "W. Ross Ashby",
"tldr": "Cybernetics researcher associated with requisite variety.",
"summary": "Cybernetics researcher associated with requisite variety. This fixture is illustrative and does not constitute a sourced scholarly claim.",
"tags": [],
"summary": "W. Ross Ashby wrote about regulation and variety in mid-century cybernetics.",
"tags": [
"cybernetics"
],
"sourceRefs": [
"src:systems-thinking:authored-demo"
],
"asOf": "2026-10-12"
"asOf": "2026-10-12",
"claimRefs": [
"claim:ashby:tldr"
]
},
{
"id": "urn:ce:systems-thinking:person:meadows",
"kindId": "person",
"title": "Donella Meadows",
"tldr": "Systems thinker known for work on leverage points.",
"summary": "Systems thinker known for work on leverage points. This fixture is illustrative and does not constitute a sourced scholarly claim.",
"tags": [],
"summary": "Donella Meadows described places in a system where a small change can have a large effect.",
"tags": [
"leverage"
],
"sourceRefs": [
"src:systems-thinking:authored-demo"
],
"asOf": "2026-10-12"
"asOf": "2026-10-12",
"claimRefs": [
"claim:meadows:tldr"
]
},
{
"id": "urn:ce:systems-thinking:publication:thinking-in-systems",
"kindId": "publication",
"title": "Thinking in Systems",
"tldr": "A widely read introductory systems-thinking book.",
"summary": "A widely read introductory systems-thinking book. This fixture is illustrative and does not constitute a sourced scholarly claim.",
"tags": [],
"summary": "Thinking in Systems is an introductory book that explains stocks, flows, and feedback with everyday examples.",
"tags": [
"introduction"
],
"sourceRefs": [
"src:systems-thinking:authored-demo"
],
"asOf": "2026-10-12"
"asOf": "2026-10-12",
"claimRefs": [
"claim:thinking-in-systems:tldr"
]
},
{
"id": "urn:ce:systems-thinking:method:causal-loop",
"kindId": "method",
"title": "Causal loop diagram",
"tldr": "A diagram representing causal influences and feedback.",
"summary": "A diagram representing causal influences and feedback. This fixture is illustrative and does not constitute a sourced scholarly claim.",
"tags": [],
"summary": "A causal loop diagram draws influence arrows and marks whether each link reinforces or balances.",
"tags": [
"diagram"
],
"sourceRefs": [
"src:systems-thinking:authored-demo"
],
"asOf": "2026-10-12"
"asOf": "2026-10-12",
"claimRefs": [
"claim:causal-loop:tldr"
]
},
{
"id": "urn:ce:systems-thinking:method:stock-flow",
"kindId": "method",
"title": "Stock and flow model",
"tldr": "A representation of accumulations and their rates of change.",
"summary": "A representation of accumulations and their rates of change. This fixture is illustrative and does not constitute a sourced scholarly claim.",
"tags": [],
"summary": "A stock and flow model separates accumulations from the rates that fill or drain them.",
"tags": [
"diagram"
],
"sourceRefs": [
"src:systems-thinking:authored-demo"
],
"asOf": "2026-10-12"
"asOf": "2026-10-12",
"claimRefs": [
"claim:stock-flow:tldr"
]
}
],
"relations": [
@ -210,5 +255,116 @@
"src:systems-thinking:authored-demo"
]
}
],
"claims": [
{
"id": "claim:feedback:tldr",
"subjectId": "urn:ce:systems-thinking:concept:feedback",
"key": "tldr",
"value": "Effects of actions change subsequent behavior.",
"status": "asserted",
"evidenceRefs": [
"src:systems-thinking:authored-demo"
],
"observedAt": "2026-10-12"
},
{
"id": "claim:emergence:tldr",
"subjectId": "urn:ce:systems-thinking:concept:emergence",
"key": "tldr",
"value": "Collective behavior arises through interactions.",
"status": "asserted",
"evidenceRefs": [
"src:systems-thinking:authored-demo"
],
"observedAt": "2026-10-12"
},
{
"id": "claim:variety:tldr",
"subjectId": "urn:ce:systems-thinking:concept:variety",
"key": "tldr",
"value": "Effective regulation requires sufficient response variety.",
"status": "asserted",
"evidenceRefs": [
"src:systems-thinking:authored-demo"
],
"observedAt": "2026-10-12"
},
{
"id": "claim:system-boundary:tldr",
"subjectId": "urn:ce:systems-thinking:concept:system-boundary",
"key": "tldr",
"value": "A distinction determines what is examined as a system.",
"status": "asserted",
"evidenceRefs": [
"src:systems-thinking:authored-demo"
],
"observedAt": "2026-10-12"
},
{
"id": "claim:ashby:tldr",
"subjectId": "urn:ce:systems-thinking:person:ashby",
"key": "tldr",
"value": "Cybernetics researcher associated with requisite variety.",
"status": "asserted",
"evidenceRefs": [
"src:systems-thinking:authored-demo"
],
"observedAt": "2026-10-12"
},
{
"id": "claim:meadows:tldr",
"subjectId": "urn:ce:systems-thinking:person:meadows",
"key": "tldr",
"value": "Systems thinker known for work on leverage points.",
"status": "asserted",
"evidenceRefs": [
"src:systems-thinking:authored-demo"
],
"observedAt": "2026-10-12"
},
{
"id": "claim:thinking-in-systems:tldr",
"subjectId": "urn:ce:systems-thinking:publication:thinking-in-systems",
"key": "tldr",
"value": "A widely read introductory systems-thinking book.",
"status": "asserted",
"evidenceRefs": [
"src:systems-thinking:authored-demo"
],
"observedAt": "2026-10-12"
},
{
"id": "claim:causal-loop:tldr",
"subjectId": "urn:ce:systems-thinking:method:causal-loop",
"key": "tldr",
"value": "A diagram representing causal influences and feedback.",
"status": "asserted",
"evidenceRefs": [
"src:systems-thinking:authored-demo"
],
"observedAt": "2026-10-12"
},
{
"id": "claim:stock-flow:tldr",
"subjectId": "urn:ce:systems-thinking:method:stock-flow",
"key": "tldr",
"value": "A representation of accumulations and their rates of change.",
"status": "asserted",
"evidenceRefs": [
"src:systems-thinking:authored-demo"
],
"observedAt": "2026-10-12"
},
{
"id": "claim:unknown:formal-measure",
"subjectId": "urn:ce:systems-thinking:concept:system-boundary",
"key": "formal-measure",
"status": "unknown",
"evidenceRefs": [
"src:systems-thinking:authored-demo"
],
"observedAt": "2026-10-12"
}
]
}

View file

@ -0,0 +1,59 @@
# SCOPE against INTENT — 2026-10-12 19:05 CEST
This note compares `SCOPE.md` (the tree at the stop below) with `INTENT.md` (initial intent v0.1, 2026-10-12). It can be read without the session transcript.
## Stop
Implementation stopped at **2026-10-12 19:05 CEST**, before 24:00 local. The reason is dependency order, not the clock. CONTENT-WP-0001 and CONTENT-WP-0003 are finished in the workplan files. CONTENT-WP-0002 depends on CONTENT-WP-0000 as well as 0001, and 0000 still has T04 open. CONTENT-WP-0004, 0005, and 0006 depend on plans that have not started. CONTENT-WP-0007 stays blocked on data-use authorization. No further registered plan was eligible.
CONTENT-WP-0000-T04 stays `todo`. At 18:52 CEST the workstation had no Chrome, Chromium, Firefox, or Playwright launcher (`command -v` found none). Live OnCo page titles were fetched over HTTP during the baseline. No screenshot and no keyboard walkthrough were produced, and T04 was not marked done. The same absence is why the v0.1 browser journey below is unmet. Chromium was not installed during this session.
## What already matches the mission
INTENT asks for a maintainable, domain-independent adaptation, with a connector projecting sources into a provenance-aware model and a manifest configuring presentation.
- Two unrelated authored dataspaces exist: systems thinking and software architecture. Their sentences were written for this demo. They are not an OnCo import. Both pass `python3 scripts/validate_fixtures.py`, including the size gate of at least three kinds and eight entities.
- The canonical types Dataspace, Kind, Entity, Relation, Claim, and Source are required by `contracts/snapshot.schema.json` at `schemaVersion` 0.1. Navigation, kind views, panels, relation labels, and fact policy live in `contracts/manifest.schema.json`.
- Domain wording is data. `content_explorer/render.py` reads the heading, kind plurals, and relation labels from the snapshot and the manifest. `tests/test_contract.py` switches the manifest while holding the systems snapshot fixed, and checks that the renderer source does not contain the fixture vocabulary.
- Fact policy distinguishes missing, unknown, one asserted value, withheld, and two asserted values. `show-all`, `preferred`, and `withhold` are specified in `contracts/fact-policy.md` and covered by tests. Absence is not stored as unknown or false.
- Ingestion is replaceable and does not render. `content_explorer/ingest.py` loads local JSON, NDJSON, and Markdown frontmatter (PyYAML). Equivalent records from those three inputs produce one snapshot hash. A second run of the same input repeats that hash. The report carries connector version `0.1.0` and a SHA-256 source digest.
- Every published tldr claim has `evidenceRefs` pointing at the source id. The source carries uri, title, license, and attribution.
- Dangling relations, duplicate identity, a key that is both live and tombstoned, an unknown kind, a missing relation label, a missing or disallowed license, and CC-BY-4.0 without attribution all refuse publication. Allowed licenses are CC0-1.0, CC-BY-4.0, and MIT. CC BY-NC is quarantined.
- Publish writes `snapshot.json`, `manifest.json`, `import-report.json`, and a human `import-report.md` by renaming a completed directory into place.
- Reuse limits match INTENT principle 10. The baseline record separates MIT code from CC BY-NC data, logos, and portraits. Those assets are not in this repo. CONTENT-WP-0007 was not advanced.
## v0.1 proof that is still unmet
INTENT's exit criteria are stronger than the seed that exists tonight.
- **No browser UI.** Switching dataspaces is proven for the text renderer, not for React components. There is still no landing page, faceted index, detail page, search, graph view, or route per entity. The browse, filter, entity, neighbor, and source journey has not been walked by mouse or keyboard.
- **CONTENT-WP-0000 is partial.** The pin, license matrix, import trace, and reuse decision exist. Screenshots and the keyboard walkthrough do not.
- **CONTENT-WP-0002 has not started**, so no OnCo table, facet control, or detail shell has been extracted. The earlier reuse note names ResultsTable, FacetSelect, browser sort, URL state, and a detail shell as later candidates. They were not copied in this session.
- **Collection** is named in INTENT's common types and is not a v0.1 schema object. Navigation and view metadata are on the manifest instead.
- **The connector is local only.** Remote static files, wiki-style references, refresh checkpoints, and the GitHub or Wikidata connector are not implemented. `connectors/json_fixture.py` only reads a snapshot that already exists.
- **Conflicting source values are not merged into a contested claim.** Two records with the same kind and key block publication as an identity conflict. The snapshot schema can store contested claims, and the fact policy can display them, but the connector does not produce that state.
- **No two-domain visual review** and no automated UI or end-to-end test. `research/UXParityMatrix.md` is still a research note, not a release-candidate parity result.
- **Regression coverage is the Python contract and ingest tests**, not an upstream `next build` or the OnCo test suite.
## Deferrals
Left for a later session, in `workplans/README.md` order:
- Finish CONTENT-WP-0000-T04 when a headless browser is available, then start CONTENT-WP-0002.
- CONTENT-WP-0004 (relations and journeys) after 0002. 0003, which it also needs, is finished.
- CONTENT-WP-0005 after 0004. Quality measurement was not opened early.
- CONTENT-WP-0006 (third connector, MCP is out of scope until then) after 0005.
- CONTENT-WP-0007 until a person authorizes OnCo data use. Do not vendor `src/data`, `public/logos`, or `public/portraits`.
## Status the files claim
| Workplan | File status | SCOPE verdict |
|---|---|---|
| CONTENT-WP-0000 | active | partial |
| CONTENT-WP-0001 | finished | finished for the contract and text renderer, not for a browser proof |
| CONTENT-WP-0002 | proposed | not started |
| CONTENT-WP-0003 | finished | finished for the local connector described in its evidence section |
| CONTENT-WP-0004 | proposed | not started |
| CONTENT-WP-0005 | proposed | not started |
| CONTENT-WP-0006 | proposed | not started |
| CONTENT-WP-0007 | blocked | blocked, not finished |

View file

@ -100,4 +100,4 @@ The detail page's first sections are Overview, State of the art, Anatomy and lym
- R2: default-heap `tsc` ran out of memory. The same command passed with a 6 GB heap.
- R2: production `next build` and the full test and lint runs were not executed.
- R1: `NOTICE` is absent. Preservation of the MIT notice applies when a file is copied, which this pass does not do.
- T04 screenshots and the keyboard walkthrough are not done.
- T04 screenshots and the keyboard walkthrough are not done. At 18:52 CEST on 2026-10-12 the workstation had no Chrome, Chromium, Firefox, or Playwright launcher (`command -v` found none). The task stays open. Live page titles were fetched over HTTP only.

View file

@ -1,27 +1,33 @@
#!/usr/bin/env python3
"""Portable smoke validator: structural and cross-reference checks; not full JSON Schema compliance."""
"""Validate every example dataspace against the v0.1 contract."""
from __future__ import annotations
import sys
from pathlib import Path
import json, sys
ROOT = Path(__file__).resolve().parent.parent
errors=[]
for path in sorted((ROOT/'examples/dataspaces').glob('*/snapshot.json')):
d=json.loads(path.read_text())
kinds={r['id'] for r in d['kinds']}; ids=[r['id'] for r in d['entities']]
sources={r['id'] for r in d['sources']}
if d['schemaVersion']!='0.1': errors.append(f'{path}: schemaVersion')
if len(ids)!=len(set(ids)): errors.append(f'{path}: duplicate entities')
for e in d['entities']:
if e['kindId'] not in kinds: errors.append(f'{path}: undeclared kind {e["kindId"]}')
for sid in e.get('sourceRefs',[]):
if sid not in sources: errors.append(f'{path}: missing source {sid}')
for r in d['relations']:
for field in ('from','to'):
if r[field] not in ids: errors.append(f'{path}: dangling {field} {r[field]}')
for sid in r.get('sourceRefs',[]):
if sid not in sources: errors.append(f'{path}: missing edge source {sid}')
manifest=json.loads((path.parent/'manifest.json').read_text())
if manifest['id']!=d['dataspace']['id']: errors.append(f'{path}: manifest ID mismatch')
print(f'{d["dataspace"]["id"]}: {len(ids)} entities, {len(d["relations"])} relations, {len(kinds)} kinds')
if errors:
print('\n'.join(errors),file=sys.stderr);sys.exit(1)
print('Smoke validation OK (full JSON Schema and claims pending).')
sys.path.insert(0, str(ROOT))
from content_explorer.contract import validate_dataspace # noqa: E402
def main() -> int:
errors: list[str] = []
directories = sorted((ROOT / "examples" / "dataspaces").glob("*/snapshot.json"))
if not directories:
print("no dataspace fixtures found", file=sys.stderr)
return 1
for snapshot_path in directories:
directory = snapshot_path.parent
found = validate_dataspace(directory, require_size=True)
errors.extend(found)
if not found:
print(f"{directory.name}: contract ok")
if errors:
print("\n".join(errors), file=sys.stderr)
return 1
print("Fixture validation OK")
return 0
if __name__ == "__main__":
raise SystemExit(main())

147
tests/test_contract.py Normal file
View file

@ -0,0 +1,147 @@
"""Contract checks against the shipped validator, not a copy of it."""
from __future__ import annotations
import copy
import json
import unittest
from pathlib import Path
ROOT = Path(__file__).resolve().parent.parent
import sys
sys.path.insert(0, str(ROOT))
from content_explorer.contract import ( # noqa: E402
resolve_claims,
validate_dataspace,
validate_documents,
)
from content_explorer.render import render_explorer # noqa: E402
EXAMPLES = ROOT / "examples" / "dataspaces"
RENDERER = ROOT / "content_explorer" / "render.py"
def load_pair(name: str) -> tuple[dict, dict]:
directory = EXAMPLES / name
snapshot = json.loads((directory / "snapshot.json").read_text(encoding="utf-8"))
manifest = json.loads((directory / "manifest.json").read_text(encoding="utf-8"))
return snapshot, manifest
class PublishedFixtures(unittest.TestCase):
def test_each_example_passes_the_file_entry_point(self) -> None:
names = sorted(path.name for path in EXAMPLES.iterdir() if (path / "snapshot.json").is_file())
self.assertGreaterEqual(len(names), 2)
for name in names:
errors = validate_dataspace(EXAMPLES / name, require_size=True)
self.assertEqual(errors, [], name)
def test_renderer_switches_with_the_manifest(self) -> None:
systems, systems_manifest = load_pair("systems-thinking")
software, software_manifest = load_pair("software-architecture")
systems_view = render_explorer(systems, systems_manifest)
software_view = render_explorer(software, software_manifest)
switched = render_explorer(systems, software_manifest)
self.assertIn(systems_manifest["home"]["heading"], systems_view)
self.assertIn(software_manifest["home"]["heading"], software_view)
self.assertIn(software_manifest["home"]["heading"], switched)
self.assertNotIn(systems_manifest["home"]["heading"], switched)
self.assertNotIn(software_manifest["home"]["heading"], systems_view)
def test_renderer_source_has_no_domain_vocabulary(self) -> None:
source = RENDERER.read_text(encoding="utf-8").lower()
for word in (
"cancer",
"oncolog",
"feedback",
"coupling",
"ashby",
"meadows",
"alexander",
"systems-thinking",
"software-architecture",
"causal",
"cohesion",
):
self.assertNotIn(word, source, word)
class NegativeGraph(unittest.TestCase):
def setUp(self) -> None:
self.snapshot, self.manifest = load_pair("systems-thinking")
def test_duplicate_entity_id(self) -> None:
snapshot = copy.deepcopy(self.snapshot)
snapshot["entities"].append(copy.deepcopy(snapshot["entities"][0]))
errors = validate_documents(snapshot, self.manifest)
self.assertTrue(any("duplicate entity id" in error for error in errors))
def test_dangling_relation(self) -> None:
snapshot = copy.deepcopy(self.snapshot)
snapshot["relations"][0]["to"] = "urn:ce:systems-thinking:concept:missing"
errors = validate_documents(snapshot, self.manifest)
self.assertTrue(any("dangling to" in error for error in errors))
def test_missing_source(self) -> None:
snapshot = copy.deepcopy(self.snapshot)
snapshot["entities"][0]["sourceRefs"] = ["src:missing"]
errors = validate_documents(snapshot, self.manifest)
self.assertTrue(any("missing source src:missing" in error for error in errors))
def test_taxonomic_cycle_fails_and_other_cycles_do_not(self) -> None:
snapshot = copy.deepcopy(self.snapshot)
left = snapshot["entities"][0]["id"]
right = snapshot["entities"][1]["id"]
source = snapshot["sources"][0]["id"]
snapshot["relations"].extend([
{
"id": "cycle-a",
"predicate": "related-to",
"from": left,
"to": right,
"sourceRefs": [source],
},
{
"id": "cycle-b",
"predicate": "related-to",
"from": right,
"to": left,
"sourceRefs": [source],
},
])
manifest = copy.deepcopy(self.manifest)
manifest["relationLabels"]["related-to"] = {"forward": "Related to", "inverse": "Related to"}
manifest["acyclicPredicates"] = []
open_cycle = validate_documents(snapshot, manifest)
self.assertFalse(any("taxonomic cycle" in error for error in open_cycle))
manifest["acyclicPredicates"] = ["related-to"]
closed = validate_documents(snapshot, manifest)
self.assertTrue(any("taxonomic cycle" in error for error in closed))
class FactPolicy(unittest.TestCase):
def test_missing_unknown_and_conflict(self) -> None:
claims = [
{"key": "measure", "status": "unknown", "value": None},
{"key": "name", "status": "asserted", "value": "one"},
{"key": "name", "status": "asserted", "value": "two"},
]
self.assertEqual(resolve_claims(claims, "absent", "show-all")["state"], "missing")
self.assertEqual(resolve_claims(claims, "measure", "show-all")["state"], "unknown")
shown = resolve_claims(claims, "name", "show-all")
self.assertEqual(shown["state"], "contradicted")
self.assertEqual(shown["values"], ["one", "two"])
withheld = resolve_claims(claims, "name", "withhold")
self.assertEqual(withheld["values"], [])
preferred_claims = [
{"key": "name", "status": "asserted", "value": "one", "preferred": True},
{"key": "name", "status": "asserted", "value": "two"},
]
chosen = resolve_claims(preferred_claims, "name", "preferred")
self.assertEqual(chosen, {"state": "present", "values": ["one"]})
if __name__ == "__main__":
unittest.main()

185
tests/test_ingest.py Normal file
View file

@ -0,0 +1,185 @@
"""Ingestion gates for JSON, NDJSON, and Markdown frontmatter."""
from __future__ import annotations
import json
import tempfile
import unittest
from pathlib import Path
import sys
ROOT = Path(__file__).resolve().parent.parent
sys.path.insert(0, str(ROOT))
from content_explorer.contract import validate_documents # noqa: E402
from content_explorer.ingest import ( # noqa: E402
load_json_records,
load_markdown_records,
load_ndjson_records,
project_records,
publish_directory,
)
LABELS = {"represented-by": {"forward": "Represented by", "inverse": "Represents"}}
KINDS = [
{"id": "concept", "label": "Concept", "plural": "Concepts"},
{"id": "method", "label": "Method", "plural": "Methods"},
]
SOURCE = {
"id": "src:demo:notes",
"uri": "https://example.invalid/content-explorer/ingest",
"title": "Demo notes",
"license": "CC0-1.0",
"attribution": "Content Explorer demo",
}
RECORDS = [
{
"kind": "concept",
"key": "feedback",
"title": "Feedback",
"tldr": "Results change the next action.",
"summary": "A loop carries an outcome back to the actor.",
"tags": ["regulation"],
"relations": [{"predicate": "represented-by", "to": "method:loop-diagram"}],
},
{
"kind": "method",
"key": "loop-diagram",
"title": "Loop diagram",
"tldr": "Arrows that show influence.",
"summary": "A drawing of influences among parts.",
"tags": ["diagram"],
"relations": [],
},
]
def project(records, source=SOURCE, relation_labels=None):
return project_records(
records,
dataspace_id="demo-notes",
title="Demo notes",
description="A connector test collection.",
source=source,
kinds=KINDS,
observed_at="2026-10-12",
relation_labels=LABELS if relation_labels is None else relation_labels,
)
class IngestGates(unittest.TestCase):
def test_json_ndjson_and_markdown_match_and_repeat(self) -> None:
with tempfile.TemporaryDirectory() as tmp:
root = Path(tmp)
(root / "records.json").write_text(json.dumps(RECORDS), encoding="utf-8")
(root / "records.ndjson").write_text(
"\n".join(json.dumps(record) for record in RECORDS) + "\n",
encoding="utf-8",
)
docs = root / "notes"
docs.mkdir()
for record in RECORDS:
body = "---\n" + _yaml(record) + "---\n\n" + record["summary"] + "\n"
(docs / f"{record['key']}.md").write_text(body, encoding="utf-8")
json_records, json_digest = load_json_records(root / "records.json")
ndjson_records, _ = load_ndjson_records(root / "records.ndjson")
markdown_records, _ = load_markdown_records(docs)
first = project_records(
json_records,
dataspace_id="demo-notes",
title="Demo notes",
description="A connector test collection.",
source=SOURCE,
kinds=KINDS,
observed_at="2026-10-12",
source_digest=json_digest,
relation_labels=LABELS,
)
self.assertEqual(first["report"]["sourceDigest"], json_digest)
second = project(ndjson_records)
third = project(markdown_records)
again = project(json_records)
self.assertTrue(first["report"]["published"])
self.assertEqual(first["report"]["contentHash"], second["report"]["contentHash"])
self.assertEqual(first["report"]["contentHash"], third["report"]["contentHash"])
self.assertEqual(first["report"]["contentHash"], again["report"]["contentHash"])
self.assertEqual(first["snapshot"], third["snapshot"])
for claim in first["snapshot"]["claims"]:
self.assertTrue(claim["evidenceRefs"])
errors = validate_documents(first["snapshot"], first["manifest"], require_size=False)
self.assertEqual(errors, [])
self.assertEqual(first["manifest"]["id"], "demo-notes")
self.assertIn("Represented by", first["manifest"]["relationLabels"]["represented-by"]["forward"])
def test_rights_and_dangling_refs_do_not_publish(self) -> None:
blocked = project(RECORDS, source={**SOURCE, "license": "CC-BY-NC-4.0"})
self.assertFalse(blocked["report"]["published"])
self.assertIsNone(blocked["snapshot"])
self.assertEqual(blocked["report"]["quarantined"][0]["reason"], "rights")
dangling = project([
{
**RECORDS[0],
"relations": [{"predicate": "represented-by", "to": "method:missing"}],
}
])
self.assertFalse(dangling["report"]["published"])
self.assertTrue(dangling["report"]["dangling"])
missing_credit = project(RECORDS, source={**SOURCE, "license": "CC-BY-4.0", "attribution": ""})
self.assertFalse(missing_credit["report"]["published"])
self.assertEqual(missing_credit["report"]["quarantined"][0]["detail"], "missing-attribution")
duplicate = project(RECORDS + [{**RECORDS[0], "title": "Other feedback"}])
self.assertFalse(duplicate["report"]["published"])
self.assertEqual(duplicate["report"]["conflicts"][0]["reason"], "duplicate-identity")
unlabeled = project(RECORDS, relation_labels={})
self.assertFalse(unlabeled["report"]["published"])
self.assertEqual(unlabeled["report"]["quarantined"][0]["reason"], "missing-relation-label")
def test_tombstone_is_omitted_and_publish_writes_both_files(self) -> None:
retired = project(RECORDS + [{
"kind": "concept",
"key": "retired",
"title": "Retired",
"tldr": "Gone.",
"deleted": True,
}])
self.assertEqual(retired["report"]["tombstoned"], ["retired"])
self.assertEqual(len(retired["snapshot"]["entities"]), 2)
with tempfile.TemporaryDirectory() as tmp:
destination = Path(tmp) / "out"
publish_directory(destination, retired)
self.assertTrue((destination / "snapshot.json").is_file())
self.assertTrue((destination / "manifest.json").is_file())
self.assertTrue((destination / "import-report.json").is_file())
human = (destination / "import-report.md").read_text(encoding="utf-8")
self.assertIn("Connector version: 0.1.0", human)
self.assertIn("License: CC0-1.0", human)
self.assertIn("retired", human)
published = json.loads((destination / "snapshot.json").read_text(encoding="utf-8"))
manifest = json.loads((destination / "manifest.json").read_text(encoding="utf-8"))
self.assertEqual(validate_documents(published, manifest, require_size=False), [])
with self.assertRaises(RuntimeError):
publish_directory(destination, project(RECORDS, source={**SOURCE, "license": ""}))
def _yaml(record: dict) -> str:
lines = [
f"kind: {record['kind']}",
f"key: {record['key']}",
f"title: {record['title']}",
f"tldr: {record['tldr']}",
f"summary: {record['summary']}",
"tags:",
*[f" - {tag}" for tag in record["tags"]],
"relations:",
]
if record["relations"]:
for link in record["relations"]:
lines.append(f" - predicate: {link['predicate']}")
lines.append(f" to: {link['to']}")
else:
lines.append(" []")
return "\n".join(lines) + "\n"
if __name__ == "__main__":
unittest.main()

View file

@ -4,7 +4,7 @@ type: workplan
title: "Canonical contracts and two-domain fixtures"
domain: infotech
repo: content-explorer
status: ready
status: finished
flavor: implementation
depends_on: []
created: "2026-10-12"
@ -14,7 +14,7 @@ state_hub_workstream_id: "833f4e24-1529-528f-8cc7-6caeeabc555f"
# Canonical contracts and two-domain fixtures
Establish a source-independent knowledge projection and the zero-UI-code-change experiment. The seed already contains a draft snapshot schema, two authored dataspaces, and a structural smoke validator. Those artifacts do not yet meet the gates below.
Establish a source-independent knowledge projection and the zero-UI-code-change experiment. The v0.1 schema, both fixtures, the negative checks, and the shared text renderer meet the gates below.
Coordinate with CONTENT-WP-0000. Neither plan blocks the other.
@ -24,7 +24,7 @@ Coordinate with CONTENT-WP-0000. Neither plan blocks the other.
```task
id: CONTENT-WP-0001-T01
status: todo
status: done
priority: high
state_hub_task_id: "c3de4ad2-7fae-51bc-9285-85d5cb2e5908"
```
@ -33,7 +33,7 @@ state_hub_task_id: "c3de4ad2-7fae-51bc-9285-85d5cb2e5908"
```task
id: CONTENT-WP-0001-T02
status: todo
status: done
priority: high
state_hub_task_id: "501807a7-f778-5799-88d6-e15bbdc42f3d"
```
@ -42,7 +42,7 @@ state_hub_task_id: "501807a7-f778-5799-88d6-e15bbdc42f3d"
```task
id: CONTENT-WP-0001-T03
status: todo
status: done
priority: high
state_hub_task_id: "f0e8a472-044e-5dfb-930f-d1afd0bcdd8d"
```
@ -51,7 +51,7 @@ state_hub_task_id: "f0e8a472-044e-5dfb-930f-d1afd0bcdd8d"
```task
id: CONTENT-WP-0001-T04
status: todo
status: done
priority: high
state_hub_task_id: "ec494de4-e634-5e8d-ab9e-cbb87f7d0ac1"
```
@ -60,7 +60,7 @@ state_hub_task_id: "ec494de4-e634-5e8d-ab9e-cbb87f7d0ac1"
```task
id: CONTENT-WP-0001-T05
status: todo
status: done
priority: high
state_hub_task_id: "8fc0c024-f179-55e0-a70d-173c94908cdd"
```
@ -71,7 +71,7 @@ state_hub_task_id: "8fc0c024-f179-55e0-a70d-173c94908cdd"
```task
id: CONTENT-WP-0001-T06
status: todo
status: done
priority: high
state_hub_task_id: "b0fbdf3b-f0cb-5a6e-bb29-593ee6c822c3"
```
@ -80,7 +80,7 @@ state_hub_task_id: "b0fbdf3b-f0cb-5a6e-bb29-593ee6c822c3"
```task
id: CONTENT-WP-0001-T07
status: todo
status: done
priority: high
state_hub_task_id: "7c2fd15b-60fa-5e7c-8b2b-8d3d6a22335e"
```
@ -89,7 +89,7 @@ state_hub_task_id: "7c2fd15b-60fa-5e7c-8b2b-8d3d6a22335e"
```task
id: CONTENT-WP-0001-T08
status: todo
status: done
priority: high
state_hub_task_id: "1847aa13-3369-577d-925d-3b08dcb5cb1a"
```
@ -100,4 +100,4 @@ R3: abstractions erase domain semantics. Log evidence and deviations in the work
## Evidence
Pull requests, test logs, pinned artifacts, screenshots, or a comparison report where applicable. Add `ImplementationNotes.md` and `Review.md` on completion. Do not mark a gate passed without a verifiable link.
Evidence: `contracts/snapshot.schema.json`, `contracts/manifest.schema.json`, `contracts/SNAPSHOT.md`, `contracts/MIGRATIONS.md`, `contracts/fact-policy.md`, `content_explorer/contract.py`, `content_explorer/render.py`, and `tests/test_contract.py`. `python3 -m unittest tests.test_contract` and `python3 scripts/validate_fixtures.py` both pass. The shared renderer is text, not a browser UI. It reads headings and labels only from the manifest and snapshot.

View file

@ -4,7 +4,7 @@ type: workplan
title: "Ingestion, provenance, and publishing"
domain: infotech
repo: content-explorer
status: proposed
status: finished
flavor: implementation
depends_on:
- CONTENT-WP-0001
@ -17,13 +17,15 @@ state_hub_workstream_id: "fd95d654-8696-5994-9643-bb45925963a6"
Turn raw inputs into trustworthy, reproducible snapshots. This gate is required before any externally ingested public content is published. It may proceed in parallel with CONTENT-WP-0002.
The v0.1 connector in `content_explorer/ingest.py` loads JSON, NDJSON, and Markdown frontmatter, resolves links in a second pass, and publishes a snapshot, a manifest, and two import reports only when rights, identity, and references allow it.
## Work packages
### Implement build-time JSON, NDJSON, and Markdown frontmatter connector APIs.
```task
id: CONTENT-WP-0003-T01
status: todo
status: done
priority: high
state_hub_task_id: "f9f8c59e-4bff-5dd4-82bb-c9f8985fd924"
```
@ -32,7 +34,7 @@ state_hub_task_id: "f9f8c59e-4bff-5dd4-82bb-c9f8985fd924"
```task
id: CONTENT-WP-0003-T02
status: todo
status: done
priority: high
state_hub_task_id: "406ae997-7b2c-5a8d-8009-a6ab53e37f4d"
```
@ -41,7 +43,7 @@ state_hub_task_id: "406ae997-7b2c-5a8d-8009-a6ab53e37f4d"
```task
id: CONTENT-WP-0003-T03
status: todo
status: done
priority: high
state_hub_task_id: "074caaf6-0501-5e02-beb3-f2c71e492f7a"
```
@ -50,7 +52,7 @@ state_hub_task_id: "074caaf6-0501-5e02-beb3-f2c71e492f7a"
```task
id: CONTENT-WP-0003-T04
status: todo
status: done
priority: high
state_hub_task_id: "0ad91201-4a1c-58b5-851f-b3e19a7eabee"
```
@ -59,7 +61,7 @@ state_hub_task_id: "0ad91201-4a1c-58b5-851f-b3e19a7eabee"
```task
id: CONTENT-WP-0003-T05
status: todo
status: done
priority: high
state_hub_task_id: "1ec567b1-2efd-558a-b790-e101b9087832"
```
@ -70,7 +72,7 @@ state_hub_task_id: "1ec567b1-2efd-558a-b790-e101b9087832"
```task
id: CONTENT-WP-0003-T06
status: todo
status: done
priority: high
state_hub_task_id: "76c69f07-3b7b-55f7-8556-3be717259348"
```
@ -79,7 +81,7 @@ state_hub_task_id: "76c69f07-3b7b-55f7-8556-3be717259348"
```task
id: CONTENT-WP-0003-T07
status: todo
status: done
priority: high
state_hub_task_id: "6fe89d04-4018-5491-85b6-76e26cb0ba4d"
```
@ -88,7 +90,7 @@ state_hub_task_id: "6fe89d04-4018-5491-85b6-76e26cb0ba4d"
```task
id: CONTENT-WP-0003-T08
status: todo
status: done
priority: high
state_hub_task_id: "d3988379-ee41-53ee-bb72-4f47c9fa37ef"
```
@ -97,7 +99,7 @@ state_hub_task_id: "d3988379-ee41-53ee-bb72-4f47c9fa37ef"
```task
id: CONTENT-WP-0003-T09
status: todo
status: done
priority: high
state_hub_task_id: "24ad02ea-75f5-5238-979c-c283072d85cb"
```
@ -108,4 +110,4 @@ R5: rights metadata missing. Quarantine instead of guessing. Log evidence and de
## Evidence
Pull requests, test logs, pinned artifacts, screenshots, or a comparison report where applicable. Add `ImplementationNotes.md` and `Review.md` on completion. Do not mark a gate passed without a verifiable link.
Evidence: `content_explorer/ingest.py` and `tests/test_ingest.py`. `python3 -m unittest tests.test_ingest` passes. JSON, NDJSON, and Markdown frontmatter projections share one content hash (`CONTENT-WP-0003-T01`, `T06`, `T07`). The report carries `connectorVersion` `0.1.0` and the source digest (`T02`). Dangling targets, duplicate keys, and a key that is both live and tombstoned block publication (`T03`). Allowed licenses are `CC0-1.0`, `CC-BY-4.0`, and `MIT`; `CC-BY-NC-4.0`, a missing license, and `CC-BY-4.0` without attribution quarantine and do not publish (`T04`, `T09`). `publish_directory` writes `snapshot.json`, `manifest.json`, `import-report.json`, and `import-report.md` by renaming a completed directory into place (`T05`). Every published tldr claim has `evidenceRefs` (`T08`). Notes: `workplans/evidence/CONTENT-WP-0003-ImplementationNotes.md` and `workplans/evidence/CONTENT-WP-0003-Review.md`. The connector does not fetch remote files and does not render.

View file

@ -0,0 +1,13 @@
# CONTENT-WP-0003 implementation notes
Date: 2026-10-12
The build-time connector is `content_explorer/ingest.py`, version `0.1.0`. It reads a JSON list or `{records}` object, NDJSON, or a directory of Markdown files with YAML frontmatter. PyYAML parses the frontmatter. The three loaders return the same record shape plus a SHA-256 digest of the bytes they read.
Projection collects live keys first, then resolves relations against that set. A target that is not an accepted key is dangling. Two live records with the same kind and key, or a key that is both tombstoned and live, are identity conflicts. Any dangling reference, identity conflict, unknown kind, or missing relation label leaves `snapshot` null and `published` false.
Publication allows only `CC0-1.0`, `CC-BY-4.0`, and `MIT`. `CC-BY-4.0` also requires a non-empty attribution. Other licenses, including `CC-BY-NC-4.0` and a missing license, are quarantined. The connector does not guess a license.
Each live entity gets one asserted `tldr` claim whose `evidenceRefs` is the source id. `publish_directory` writes `snapshot.json`, `manifest.json`, `import-report.json`, and `import-report.md` into a temporary directory and renames it into place. A quarantined result raises `RuntimeError` and does not replace an existing directory.
The manifest is generated from the caller-supplied kind list and relation labels. Predicate wording is not invented. `connectors/json_fixture.py` still only reads a snapshot that already exists. Remote fetch, pagination, and a third connector are not part of this module.

View file

@ -0,0 +1,16 @@
# CONTENT-WP-0003 review
Date: 2026-10-12
Risk R5 was rights metadata missing. The connector quarantines a missing license, a license outside `CC0-1.0`, `CC-BY-4.0`, and `MIT`, and `CC-BY-4.0` without attribution. It does not invent attribution or a rights statement.
Checked against `tests/test_ingest.py`:
- JSON, NDJSON, and Markdown projections of the same records share one content hash, including a second JSON run.
- The report records connector version `0.1.0` and the source digest.
- A dangling target, a duplicate key, and a missing relation label do not publish.
- A tombstoned key is listed and omitted from entities.
- `publish_directory` writes the snapshot, manifest, machine report, and human report. The written snapshot and manifest pass `validate_documents`.
- A rights failure raises before replacing the destination.
Not claimed: remote URLs, conflict merging into a contested claim, or a browser view of the published directory. Those stay with later workplans. CONTENT-WP-0007 was not used as a source.