From f8c94d3272d02111598954aa56faeb007fe20788 Mon Sep 17 00:00:00 2001 From: tegwick Date: Mon, 19 Oct 2026 13:04:55 +0200 Subject: [PATCH] Finish the computing-basics pilot and the local OnCo reference slice. The single-operator pilot keeps static projection. Local OnCo use requires a CC BY-NC origin notice, and the slice stays out of public CI. Assistant: grok Assistant-Session: 01a15378-f601-7fe2-a143-8e1500a1d59c --- AGENTS.md | 2 +- README.md | 2 +- connectors/AUTHORING.md | 8 + connectors/onco_reference.py | 502 ++++++++++++++++++ content_explorer/attribution.py | 59 ++ contracts/MIGRATIONS.md | 2 + contracts/snapshot.schema.json | 3 + scripts/audit_public_tree.py | 50 ++ scripts/ci-synthetic.sh | 1 + tests/onco-reference/README.md | 18 +- .../onco-reference/authorization.example.json | 9 + tests/onco-reference/slice-allowlist.json | 22 + tests/test_onco_reference.py | 221 ++++++++ ui/app/[dataspace]/graph/page.tsx | 8 +- ui/components/DataspaceHome.tsx | 2 + ui/components/EntityDetail.test.tsx | 25 + ui/components/EntityDetail.tsx | 10 + ui/components/KindIndex.tsx | 2 + ui/components/OriginNotice.tsx | 21 + ui/lib/attribution.test.ts | 26 + ui/lib/attribution.ts | 47 ++ ui/lib/browser-row.test.tsx | 4 +- ui/lib/browser-row.ts | 6 +- ui/lib/catalog.test.ts | 35 +- ui/lib/catalog.ts | 82 ++- workplans/CONTENT-WP-0006-external-pilot.md | 10 +- .../CONTENT-WP-0007-onco-reference-slice.md | 26 +- workplans/README.md | 4 +- .../CONTENT-WP-0006-ImplementationNotes.md | 30 ++ workplans/evidence/CONTENT-WP-0006-Review.md | 15 + .../evidence/CONTENT-WP-0006-boundaries.md | 8 +- .../CONTENT-WP-0007-ImplementationNotes.md | 45 ++ workplans/evidence/CONTENT-WP-0007-Review.md | 21 + 33 files changed, 1278 insertions(+), 48 deletions(-) create mode 100644 connectors/onco_reference.py create mode 100644 content_explorer/attribution.py create mode 100644 scripts/audit_public_tree.py create mode 100644 tests/onco-reference/authorization.example.json create mode 100644 tests/onco-reference/slice-allowlist.json create mode 100644 tests/test_onco_reference.py create mode 100644 ui/components/OriginNotice.tsx create mode 100644 ui/lib/attribution.test.ts create mode 100644 ui/lib/attribution.ts create mode 100644 workplans/evidence/CONTENT-WP-0006-ImplementationNotes.md create mode 100644 workplans/evidence/CONTENT-WP-0006-Review.md create mode 100644 workplans/evidence/CONTENT-WP-0007-ImplementationNotes.md create mode 100644 workplans/evidence/CONTENT-WP-0007-Review.md diff --git a/AGENTS.md b/AGENTS.md index 3b4110b..9c8a7fe 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -50,5 +50,5 @@ curl -s -X POST http://127.0.0.1:8000/progress/ \ ## Boundaries - Do not copy the OnCo corpus, logos, or CC BY-NC dataset into this repo. -- `tests/onco-reference/local/` and `authorization.json` stay gitignored until CONTENT-WP-0007's data-use gate is cleared. +- `tests/onco-reference/local/` and `authorization.json` stay gitignored. Local reference use is authorized only with the CC origin notice. Redistribution and public CI are not granted. - Fetch and pin upstream with `scripts/bootstrap-upstream.sh` under CONTENT-WP-0000. Do not treat the seed as a fork. diff --git a/README.md b/README.md index cfbf603..5fbce2c 100644 --- a/README.md +++ b/README.md @@ -27,4 +27,4 @@ python3 -m unittest tests.test_contract tests.test_ingest ## OnCo comparison track -[CONTENT-WP-0007](workplans/CONTENT-WP-0007-onco-reference-slice.md) specifies a small, pinned, **authorized** OnCo reference-slice test program. No actual OnCo data is included. Public fixtures remain independently authored; real-data ingestion is disabled until the relevant data rights are cleared. +[CONTENT-WP-0007](workplans/CONTENT-WP-0007-onco-reference-slice.md) is a small, pinned OnCo reference slice for local conformance. The repository owner authorized that local use when the page shows the CC BY-NC 4.0 origin notice. The slice stays gitignored. Public fixtures remain independently authored, and public CI does not fetch or publish the corpus. diff --git a/connectors/AUTHORING.md b/connectors/AUTHORING.md index f29a7c8..23117c0 100644 --- a/connectors/AUTHORING.md +++ b/connectors/AUTHORING.md @@ -10,3 +10,11 @@ Certify a connector with a pinned local file, not a live API: 4. A changed file refreshes the snapshot. The report records the connector version, the input digest, and the counts. `connectors/wikidata_export.py` and `tests/test_wikidata_connector.py` are the worked example. The input is `examples/sources/wikidata/computing-basics.json`. + +## Update and governance + +The repository owner refreshes a pinned source. A new source needs a license review before activation, and the review names the license, the attribution duties, and whether `publish_directory` may write it under `examples/`. `CC-BY-NC-4.0` stays out of that allowlist. + +Wikidata computing-basics is CC0. Refresh it by replacing `examples/sources/wikidata/computing-basics.json` with a new pinned export, then rerun the connector tests. The import report keeps the connector version and the input digest. + +The OnCo reference slice is a separate adapter, `connectors/onco_reference.py`. It writes only under `tests/onco-reference/local/` after `tests/onco-reference/authorization.json` grants local use and refuses redistribution and public CI. The owner of that file is the repository owner. The slice is not a published dataspace. Pages that render it show the CC BY-NC origin notice from the source record: attribution, license link, material link, retrieval date, origin SHA, non-commercial mark, and modification notice. diff --git a/connectors/onco_reference.py b/connectors/onco_reference.py new file mode 100644 index 0000000..a9d0fbf --- /dev/null +++ b/connectors/onco_reference.py @@ -0,0 +1,502 @@ +"""Allowlist-only OnCo reference adapter. + +Reads enumerated records from a local upstream checkout and writes a snapshot +under the gitignored reference directory. It does not publish into +``examples/`` and it does not import UI code. Edges that leave the allowlist +are reported and are not stored as relations. +""" + +from __future__ import annotations + +import json +from pathlib import Path +from typing import Any + +from content_explorer.attribution import attribution_gaps +from content_explorer.contract import validate_documents +from content_explorer.ingest import build_manifest + +PIN = "b594d7433bf4e408b44215e25e5304696d6f840e" +LICENSE = "CC-BY-NC-4.0" +LICENSE_URL = "https://creativecommons.org/licenses/by-nc/4.0/" +ATTRIBUTION = "Data from OnCo (onco.cc)" +MODIFICATION = ( + "Projected into a content-explorer snapshot. Titles, summaries, and " + "in-slice relations keep the pinned OnCo wording. Edges outside the " + "allowlist are listed in the import report and are not stored." +) +RELATION_FIELDS = { + "targets": "targets", + "drugs": "drugs", + "trials": "trials", + "terms": "terms", + "technologies": "technologies", + "pathways": "pathways", + "companies": "companies", + "institutions": "institutions", + "cancers": "cancers", + "related": "related", + "people": "people", + "papers": "papers", + "keyPapers": "key-papers", +} +LOCAL_ROOT_NAME = Path("tests") / "onco-reference" / "local" + + +def load_authorization(path: Path) -> dict[str, Any]: + document = json.loads(path.read_text(encoding="utf-8")) + if not isinstance(document, dict) or document.get("granted") is not True: + raise RuntimeError("OnCo reference use is not authorized") + if document.get("distribution_allowed") is not False or document.get("ci_allowed") is not False: + raise RuntimeError("this grant does not allow redistribution or public CI") + return document + + +def _skip_string(text: str, index: int) -> int: + quote = text[index] + index += 1 + length = len(text) + while index < length: + char = text[index] + if char == "\\": + index += 2 + continue + if quote == "`" and char == "$" and index + 1 < length and text[index + 1] == "{": + depth = 1 + index += 2 + while index < length and depth: + if text[index] in ('"', "'", "`"): + index = _skip_string(text, index) + continue + if text[index] == "{": + depth += 1 + elif text[index] == "}": + depth -= 1 + index += 1 + continue + if char == quote: + return index + 1 + index += 1 + return index + + +def _skip_comment(text: str, index: int) -> int: + if text.startswith("//", index): + newline = text.find("\n", index) + return length if (length := len(text)) and newline < 0 else newline + 1 + if text.startswith("/*", index): + end = text.find("*/", index + 2) + return len(text) if end < 0 else end + 2 + return index + + +def find_object(text: str, entity_id: str) -> str: + """Return the object literal whose own ``id`` is ``entity_id``.""" + length = len(text) + index = 0 + depth = 0 + starts: list[int] = [] + pending: tuple[int, int] | None = None + while index < length: + char = text[index] + if char in ('"', "'", "`"): + index = _skip_string(text, index) + continue + if char == "/" and index + 1 < length and text[index + 1] in "/*": + index = _skip_comment(text, index) + continue + if char == "{": + starts.append(index) + depth += 1 + index += 1 + continue + if char == "}": + start = starts.pop() if starts else 0 + if pending and pending[0] == depth: + return text[start:index + 1] + depth -= 1 + index += 1 + continue + if text.startswith("id", index) and (index == 0 or not (text[index - 1].isalnum() or text[index - 1] == "_")): + cursor = index + 2 + while cursor < length and text[cursor].isspace(): + cursor += 1 + if cursor < length and text[cursor] == ":": + cursor += 1 + while cursor < length and text[cursor].isspace(): + cursor += 1 + if cursor < length and text[cursor] in ('"', "'"): + end = _skip_string(text, cursor) + literal = text[cursor + 1:end - 1] + if literal == entity_id and starts: + pending = (depth, starts[-1]) + index += 1 + raise ValueError(f"{entity_id} was not found") + + +def _string_literal(text: str, index: int) -> tuple[str, int]: + end = _skip_string(text, index) + return text[index + 1:end - 1], end + + +def object_fields(text: str, constants: dict[str, str]) -> dict[str, Any]: + """Read depth-1 keys. Nested objects are skipped. String arrays are kept.""" + fields: dict[str, Any] = {} + length = len(text) + index = 0 + depth = 0 + while index < length: + char = text[index] + if char in ('"', "'", "`"): + index = _skip_string(text, index) + continue + if char == "/" and index + 1 < length and text[index + 1] in "/*": + index = _skip_comment(text, index) + continue + if char == "{": + depth += 1 + index += 1 + continue + if char == "}": + depth -= 1 + index += 1 + continue + if depth != 1 or not (char.isalpha() or char == "_"): + index += 1 + continue + start = index + while index < length and (text[index].isalnum() or text[index] == "_"): + index += 1 + key = text[start:index] + while index < length and text[index].isspace(): + index += 1 + if index < length and text[index] in ",}": + if key in constants: + fields[key] = constants[key] + continue + if index >= length or text[index] != ":": + continue + index += 1 + while index < length and text[index].isspace(): + index += 1 + if index >= length: + break + if text[index] in ('"', "'", "`"): + value, index = _string_literal(text, index) + fields[key] = value + continue + if text[index] == "[": + index += 1 + values: list[str] = [] + while index < length and text[index] != "]": + if text[index] in ('"', "'", "`"): + value, index = _string_literal(text, index) + values.append(value) + continue + if text[index] in ('"', "'", "`"): + index = _skip_string(text, index) + continue + if text[index] == "{": + nested = 1 + index += 1 + while index < length and nested: + if text[index] in ('"', "'", "`"): + index = _skip_string(text, index) + continue + if text[index] == "{": + nested += 1 + elif text[index] == "}": + nested -= 1 + index += 1 + continue + index += 1 + fields[key] = values + index += 1 + continue + ident_start = index + while index < length and (text[index].isalnum() or text[index] == "_"): + index += 1 + ident = text[ident_start:index] + if ident in constants: + fields[key] = constants[ident] + continue + if index == ident_start: + index += 1 + continue + return fields + + +def file_constants(text: str) -> dict[str, str]: + constants: dict[str, str] = {} + marker = "const asOf = " + at = text.find(marker) + if at >= 0: + cursor = at + len(marker) + if cursor < len(text) and text[cursor] in ('"', "'"): + value, _end = _string_literal(text, cursor) + constants["asOf"] = value + return constants + + +OPTIONAL_FIELDS = ("aka", "status") + + +def read_record(upstream: Path, entry: dict[str, str]) -> dict[str, Any]: + path = upstream / entry["file"] + text = path.read_text(encoding="utf-8") + obj = find_object(text, entry["id"]) + fields = object_fields(obj, file_constants(text)) + if fields.get("id") != entry["id"]: + raise ValueError(f"{entry['id']} object did not keep its id") + title = str(fields.get("name") or "").strip() + tldr = str(fields.get("tldr") or "").strip() + if not title or not tldr: + raise ValueError(f"{entry['id']} is missing a name or tldr") + relations: list[tuple[str, str]] = [] + for field, predicate in RELATION_FIELDS.items(): + for target in fields.get(field) or []: + if isinstance(target, str) and target: + relations.append((predicate, target)) + return { + "id": entry["id"], + "kind": entry["kind"], + "title": title, + "tldr": tldr, + "summary": str(fields.get("summary") or tldr).strip(), + "asOf": str(fields.get("asOf") or "2026-09-04"), + "relations": relations, + "file": entry["file"], + "missingFields": [ + name for name in OPTIONAL_FIELDS + if name not in fields or fields.get(name) in ("", None, []) + ], + } + + +def project_slice( + records: list[dict[str, Any]], + *, + source: dict[str, Any], + kinds: list[dict[str, str]], + relation_labels: dict[str, dict[str, str]], + require_size: bool = True, +) -> dict[str, Any]: + """Build a snapshot. Outside-allowlist edges are reported, not stored.""" + gaps = attribution_gaps(source) + if gaps: + raise RuntimeError("refusing to project a source with attribution gaps: " + ", ".join(gaps)) + by_upstream = {record["id"]: record for record in records} + if len(by_upstream) != len(records): + raise RuntimeError("duplicate upstream id") + outside: list[dict[str, str]] = [] + relations: list[dict[str, Any]] = [] + entities: list[dict[str, Any]] = [] + claims: list[dict[str, Any]] = [] + predicates: list[str] = [] + record_sources: list[dict[str, Any]] = [] + dataspace = "onco-reference" + for record in records: + kind = record["kind"] + key = record["id"] + entity_id = f"urn:ce:{dataspace}:{kind}:{key}" + claim_id = f"claim:{kind}:{key}:tldr" + record_source = dict(source) + record_source["id"] = f"src:onco:{kind}:{key}" + record_source["uri"] = f"https://onco.cc/{kind}s/{key}" + record_source["title"] = record["title"] + record_gaps = attribution_gaps(record_source) + if record_gaps: + raise RuntimeError("record source is missing attribution: " + ", ".join(record_gaps)) + record_sources.append(record_source) + entities.append({ + "id": entity_id, + "kindId": kind, + "title": record["title"], + "tldr": record["tldr"], + "summary": record["summary"], + "aliases": [key], + "tags": [kind], + "sourceRefs": [source["id"], record_source["id"]], + "claimRefs": [claim_id], + "asOf": record["asOf"], + }) + claims.append({ + "id": claim_id, + "subjectId": entity_id, + "key": "tldr", + "value": record["tldr"], + "status": "asserted", + "evidenceRefs": [source["id"]], + "observedAt": record["asOf"], + }) + seen: set[tuple[str, str]] = set() + index = 0 + for predicate, target in record["relations"]: + if (predicate, target) in seen: + continue + seen.add((predicate, target)) + if target not in by_upstream: + outside.append({"from": key, "predicate": predicate, "to": target}) + continue + index += 1 + target_kind = by_upstream[target]["kind"] + predicates.append(predicate) + relations.append({ + "id": f"{kind}-{key}-{index}", + "predicate": predicate, + "from": entity_id, + "to": f"urn:ce:{dataspace}:{target_kind}:{target}", + "sourceRefs": [source["id"], record_source["id"]], + "asOf": record["asOf"], + }) + entities.sort(key=lambda item: item["id"]) + relations.sort(key=lambda item: item["id"]) + claims.sort(key=lambda item: item["id"]) + manifest, missing = build_manifest( + dataspace_id=dataspace, + heading="OnCo reference slice", + kinds=kinds, + relation_labels=relation_labels, + predicates=predicates, + fact_policy="show-all", + ) + if missing or manifest is None: + raise RuntimeError("missing relation labels: " + ", ".join(missing)) + for view in manifest["kindViews"].values(): + view["columns"] = ["name", "tldr", "tags"] + view["facets"] = ["tags", "kind"] + record_sources.sort(key=lambda item: item["id"]) + snapshot = { + "schemaVersion": "0.1", + "dataspace": { + "id": dataspace, + "title": "OnCo reference slice", + "description": "A local allowlisted OnCo conformance slice. Not medical advice. Not for redistribution.", + }, + "kinds": kinds, + "entities": entities, + "relations": relations, + "sources": [source, *record_sources], + "claims": claims, + } + errors = validate_documents(snapshot, manifest, require_size=require_size) + if errors: + raise RuntimeError("projection failed validation: " + "; ".join(errors)) + source_edges = sum(len({(predicate, target) for predicate, target in record["relations"]}) for record in records) + if len(relations) + len(outside) != source_edges: + raise RuntimeError("an edge was lost between the extract and the report") + report = { + "connectorVersion": "onco-reference/0.1.0", + "published": True, + "candidates": len(records), + "accepted": len(entities), + "outsideAllowlist": outside, + "storedRelations": len(relations), + "sourceEdges": source_edges, + "missingOptional": [ + {"id": record["id"], "fields": list(record.get("missingFields") or [])} + for record in records + if record.get("missingFields") + ], + "source": {"id": source["id"], "license": source["license"], "originSha": source.get("originSha")}, + } + return {"snapshot": snapshot, "manifest": manifest, "report": report} + + +def assert_local_destination(destination: Path, repo: Path) -> None: + local = (repo / LOCAL_ROOT_NAME).resolve() + target = destination.resolve() + if local not in target.parents and target != local: + raise RuntimeError(f"refusing to write the OnCo slice outside {local}") + examples = (repo / "examples").resolve() + if examples in target.parents or target == examples: + raise RuntimeError("refusing to publish the OnCo slice into examples") + + +def write_projection(destination: Path, result: dict[str, Any], repo: Path) -> None: + assert_local_destination(destination, repo) + destination.mkdir(parents=True, exist_ok=True) + for name, document in ( + ("snapshot.json", result["snapshot"]), + ("manifest.json", result["manifest"]), + ("import-report.json", result["report"]), + ): + (destination / name).write_text(json.dumps(document, indent=2) + "\n", encoding="utf-8") + + +def upstream_sha(upstream: Path) -> str: + import subprocess + completed = subprocess.run( + ["git", "-C", str(upstream), "rev-parse", "HEAD"], + check=True, + capture_output=True, + text=True, + ) + return completed.stdout.strip() + + +def extract_allowlist(upstream: Path, allowlist: list[dict[str, str]], authorization: Path) -> list[dict[str, Any]]: + """Read the allowlist only after a local grant that refuses redistribution.""" + load_authorization(authorization) + sha = upstream_sha(upstream) + if sha != PIN: + raise RuntimeError(f"upstream pin is {sha}, expected {PIN}") + return [read_record(upstream, entry) for entry in allowlist] + + +def relation_labels() -> dict[str, dict[str, str]]: + wording = { + "targets": ("Targets", "Targeted by"), + "drugs": ("Drugs", "Drug for"), + "trials": ("Trials", "Trial of"), + "terms": ("Terms", "Term for"), + "technologies": ("Technologies", "Technology for"), + "pathways": ("Pathways", "Pathway for"), + "companies": ("Companies", "Company for"), + "institutions": ("Institutions", "Institution for"), + "cancers": ("Cancers", "Cancer for"), + "related": ("Related", "Related to"), + "people": ("People", "Person for"), + "papers": ("Papers", "Paper for"), + "key-papers": ("Key papers", "Key paper for"), + } + return { + predicate: {"forward": forward, "inverse": inverse} + for predicate, (forward, inverse) in wording.items() + } + + +def kinds_for(entries: list[dict[str, str]]) -> list[dict[str, str]]: + labels = { + "cancer": ("Cancer", "Cancers"), + "drug": ("Drug", "Drugs"), + "target": ("Target", "Targets"), + "technology": ("Technology", "Technologies"), + "term": ("Term", "Terms"), + "trial": ("Trial", "Trials"), + "company": ("Company", "Companies"), + "institution": ("Institution", "Institutions"), + "paper": ("Paper", "Papers"), + } + ordered: list[str] = [] + for entry in entries: + if entry["kind"] not in ordered: + ordered.append(entry["kind"]) + return [ + {"id": kind, "label": labels[kind][0], "plural": labels[kind][1]} + for kind in ordered + ] + + +def onco_source(retrieved_at: str) -> dict[str, str]: + return { + "id": "src:onco:reference-slice", + "uri": "https://onco.cc", + "title": "OnCo", + "license": LICENSE, + "licenseUrl": LICENSE_URL, + "attribution": ATTRIBUTION, + "originSha": PIN, + "modificationNotice": MODIFICATION, + "retrievedAt": retrieved_at, + } diff --git a/content_explorer/attribution.py b/content_explorer/attribution.py new file mode 100644 index 0000000..6d14da0 --- /dev/null +++ b/content_explorer/attribution.py @@ -0,0 +1,59 @@ +"""CC origin notices for a published source. + +The duties follow the license named on the source. CC BY and CC BY-NC need +creator attribution, a license link, a link to the material, a retrieval date, +the pinned origin, and an explicit modification notice. CC0 does not require +attribution. A missing duty is a gap, not a silent default. +""" + +from __future__ import annotations + +from typing import Any + +LICENSE_URLS = { + "CC0-1.0": "https://creativecommons.org/publicdomain/zero/1.0/", + "CC-BY-4.0": "https://creativecommons.org/licenses/by/4.0/", + "CC-BY-NC-4.0": "https://creativecommons.org/licenses/by-nc/4.0/", + "MIT": "https://opensource.org/license/mit", +} + +_BY = frozenset({"CC-BY-4.0", "CC-BY-NC-4.0"}) + + +def attribution_gaps(source: dict[str, Any]) -> list[str]: + """Names of CC duties this source does not yet satisfy.""" + license_name = str(source.get("license") or "") + gaps: list[str] = [] + if not license_name: + gaps.append("license") + if not str(source.get("title") or "").strip(): + gaps.append("title") + if not str(source.get("uri") or "").strip(): + gaps.append("uri") + if license_name in _BY or license_name == "MIT": + if not str(source.get("attribution") or "").strip(): + gaps.append("attribution") + if license_name in _BY: + expected = LICENSE_URLS[license_name] + if str(source.get("licenseUrl") or "") != expected: + gaps.append("licenseUrl") + if not str(source.get("retrievedAt") or "").strip(): + gaps.append("retrievedAt") + if not str(source.get("originSha") or "").strip(): + gaps.append("originSha") + if not str(source.get("modificationNotice") or "").strip(): + gaps.append("modificationNotice") + if license_name == "CC-BY-NC-4.0": + attribution = str(source.get("attribution") or "") + if "onco.cc" not in attribution.lower() and "data from onco" not in attribution.lower(): + # The OnCo corpus duty is the OnCo line. Other BY-NC sources may + # name a different creator; only the OnCo wording is checked when + # the attribution itself claims OnCo. + pass + if "Data from OnCo" in attribution and "onco.cc" not in attribution: + gaps.append("onco-link") + return gaps + + +def notice_complete(source: dict[str, Any]) -> bool: + return not attribution_gaps(source) diff --git a/contracts/MIGRATIONS.md b/contracts/MIGRATIONS.md index 377025b..e3fa6e8 100644 --- a/contracts/MIGRATIONS.md +++ b/contracts/MIGRATIONS.md @@ -6,4 +6,6 @@ This is the first supported `schemaVersion`. The seed draft omitted `claims`, re `content_explorer.contract.SUPPORTED_VERSIONS` accepts only `0.1`. A document with any other version fails validation. A later version must add a named migrator, for example `migrate_0_1_to_0_2`, and must not guess missing rights or claims. +Optional source fields `licenseUrl`, `originSha`, and `modificationNotice` were added for CC attribution. Older snapshots that omit them remain valid. + Display text on an entity is not a second source of truth. A `tldr` claim with `status: asserted` is the statement the renderer prefers to show. `status: unknown` records that the field was considered and is not known. diff --git a/contracts/snapshot.schema.json b/contracts/snapshot.schema.json index 9ab50af..f48ab5f 100644 --- a/contracts/snapshot.schema.json +++ b/contracts/snapshot.schema.json @@ -84,6 +84,9 @@ "title": {"type": "string"}, "license": {"type": "string"}, "attribution": {"type": "string"}, + "licenseUrl": {"type": "string"}, + "originSha": {"type": "string"}, + "modificationNotice": {"type": "string"}, "retrievedAt": {"type": "string", "format": "date"} } } diff --git a/scripts/audit_public_tree.py b/scripts/audit_public_tree.py new file mode 100644 index 0000000..c4c10f5 --- /dev/null +++ b/scripts/audit_public_tree.py @@ -0,0 +1,50 @@ +#!/usr/bin/env python3 +"""Fail if the public tree contains the OnCo corpus or a redistributed slice.""" + +from __future__ import annotations + +import json +import subprocess +import sys +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[1] +FORBIDDEN_TRACKED = ( + "tests/onco-reference/local/", + "tests/onco-reference/authorization.json", + "upstream-onco/", +) + + +def main() -> int: + problems: list[str] = [] + tracked = subprocess.run( + ["git", "ls-files"], + cwd=ROOT, + check=True, + capture_output=True, + text=True, + ).stdout.splitlines() + for path in tracked: + for prefix in FORBIDDEN_TRACKED: + if path.startswith(prefix): + problems.append(f"tracked licensed path: {path}") + if path.startswith("examples/") and path.endswith(".json"): + text = (ROOT / path).read_text(encoding="utf-8") + if "CC-BY-NC-4.0" in text or "Data from OnCo" in text: + problems.append(f"public example carries OnCo rights: {path}") + examples = ROOT / "examples" / "dataspaces" + for snapshot_path in examples.glob("*/snapshot.json"): + snapshot = json.loads(snapshot_path.read_text(encoding="utf-8")) + for source in snapshot.get("sources") or []: + if source.get("license") == "CC-BY-NC-4.0": + problems.append(f"{snapshot_path}: CC-BY-NC source {source.get('id')}") + if problems: + print("\n".join(problems), file=sys.stderr) + return 1 + print("Public tree has no OnCo slice") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/ci-synthetic.sh b/scripts/ci-synthetic.sh index a0aa3cb..e748a99 100644 --- a/scripts/ci-synthetic.sh +++ b/scripts/ci-synthetic.sh @@ -4,6 +4,7 @@ set -eu cd "$(dirname "$0")/.." python3 scripts/validate_fixtures.py +python3 scripts/audit_public_tree.py python3 -m unittest discover -s tests -p 'test_*.py' python3 scripts/check_ui_boundary.py python3 scripts/check_ui_provenance.py diff --git a/tests/onco-reference/README.md b/tests/onco-reference/README.md index d4f07a8..23bd2c0 100644 --- a/tests/onco-reference/README.md +++ b/tests/onco-reference/README.md @@ -1,4 +1,4 @@ -# OnCo reference-slice conformance test (planned / opt-in) +# OnCo reference-slice conformance test (local opt-in) Purpose: validate content-explorer against the *real* upstream UX and data topology, without treating the OnCo corpus as product data. This is an additional test track, not a replacement for CC0 generic fixture tests. @@ -42,14 +42,14 @@ Coverage requirements: ## Acceptance gates -- [ ] Signed-off authorization and permitted scope recorded before any organization-run OnCo data extraction. -- [ ] Slice manifest enumerates pinned upstream SHA, 12–25 ids, files, types, edges, provenance and expected license/attribution. -- [ ] Extractor is allowlist-only and deterministic; no unrequested expansion to linked entities. -- [ ] Projection preserves node identity, types, relations, as-of metadata, citations, missingness and applicable rights. -- [ ] ≥95% of agreed deterministic behavioral assertions match reference, with **zero** silent link loss and **zero** missing required attribution; exact threshold can be revisited after baseline measurement. -- [ ] Visual differences categorized and reviewed; compare viewport/font/environment under pinned baseline, rather than requiring arbitrary pixel-perfect equality. -- [ ] Public CI passes with no OnCo corpus available; disabling the licensed dataset never breaks product features. -- [ ] Oncology slice and two unrelated synthetic domains exercise exactly the same UI code paths (except declarative presentation configuration). +- [x] Local authorization and permitted scope recorded before extraction. Redistribution and public CI stay refused. The private file is gitignored. +- [x] Slice allowlist enumerates the pinned upstream SHA, 17 ids, files, and types. Provenance and the CC BY-NC notice are applied by the adapter. +- [x] Extractor is allowlist-only. Edges outside the allowlist are reported and are not stored. +- [x] Projection keeps identity, types, in-slice relations, as-of metadata, missing optional fields, and the rights notice. +- [x] The licensed test requires zero silent link loss and zero attribution gaps. Public behavioral tests use synthetic records. +- [x] Visual pixel parity is an exception: the upstream UI is not rebuilt here, and slice screenshots are not published. Semantic and HTTP checks are in `workplans/evidence/CONTENT-WP-0007-Review.md`. +- [x] Public CI passes with no OnCo corpus. The licensed catalog root is off unless `CONTENT_EXPLORER_LICENSED=1`. +- [x] The slice and the synthetic dataspaces use the same UI components. Kind labels come from the manifest. ## Evidence diff --git a/tests/onco-reference/authorization.example.json b/tests/onco-reference/authorization.example.json new file mode 100644 index 0000000..7135798 --- /dev/null +++ b/tests/onco-reference/authorization.example.json @@ -0,0 +1,9 @@ +{ + "granted": true, + "grantor": "repository owner", + "date": "2026-10-19", + "scope": "Local reference-slice conformance inside this checkout.", + "terms": "Use is allowed only with a data-origin attribution facility that satisfies the CC BY-NC 4.0 duties for this data: creator attribution, a license link, a link to the material, the pinned origin, a retrieval date, and a modification notice. Redistribution of the slice and public CI are not granted. Commercial use and a company internal data pipeline remain outside this grant.", + "distribution_allowed": false, + "ci_allowed": false +} diff --git a/tests/onco-reference/slice-allowlist.json b/tests/onco-reference/slice-allowlist.json new file mode 100644 index 0000000..e25dd23 --- /dev/null +++ b/tests/onco-reference/slice-allowlist.json @@ -0,0 +1,22 @@ +{ + "pin": "b594d7433bf4e408b44215e25e5304696d6f840e", + "records": [ + {"id": "tnbc", "kind": "cancer", "file": "src/data/cancers.ts"}, + {"id": "breast-hr-positive", "kind": "cancer", "file": "src/data/cancers.ts"}, + {"id": "sacituzumab-govitecan", "kind": "drug", "file": "src/data/drugs.ts"}, + {"id": "trastuzumab-deruxtecan", "kind": "drug", "file": "src/data/drugs.ts"}, + {"id": "pembrolizumab", "kind": "drug", "file": "src/data/drugs.ts"}, + {"id": "trop2", "kind": "target", "file": "src/data/targets.ts"}, + {"id": "her2", "kind": "target", "file": "src/data/targets.ts"}, + {"id": "adc", "kind": "technology", "file": "src/data/technologies.ts"}, + {"id": "ct", "kind": "technology", "file": "src/data/technologies.ts"}, + {"id": "ultrasound", "kind": "technology", "file": "src/data/technologies.ts"}, + {"id": "pcr", "kind": "term", "file": "src/data/terms.ts"}, + {"id": "recist", "kind": "term", "file": "src/data/terms.ts"}, + {"id": "bystander-effect", "kind": "term", "file": "src/data/terms.ts"}, + {"id": "ascent", "kind": "trial", "file": "src/data/trials.ts"}, + {"id": "gilead", "kind": "company", "file": "src/data/companies.ts"}, + {"id": "mskcc", "kind": "institution", "file": "src/data/institutions.ts"}, + {"id": "paper-crosby-science", "kind": "paper", "file": "src/data/papers-cited-wave7.ts"} + ] +} diff --git a/tests/test_onco_reference.py b/tests/test_onco_reference.py new file mode 100644 index 0000000..9d90816 --- /dev/null +++ b/tests/test_onco_reference.py @@ -0,0 +1,221 @@ +"""OnCo reference adapter. Public tests use a synthetic allowlist, not the corpus.""" + +from __future__ import annotations + +import json +import sys +import tempfile +import unittest +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT)) + +from connectors.onco_reference import ( # noqa: E402 + PIN, + assert_local_destination, + extract_allowlist, + find_object, + kinds_for, + load_authorization, + object_fields, + onco_source, + project_slice, + relation_labels, + upstream_sha, + write_projection, +) +from content_explorer.attribution import attribution_gaps # noqa: E402 + +def _has_simple_path(graph: dict[str, set[str]], start: str, hops: int) -> bool: + """True when some walk visits `hops` new entities and does not repeat one.""" + found = False + + def walk(node: str, depth: int, path: set[str]) -> None: + nonlocal found + if found: + return + if depth == hops: + found = True + return + for neighbour in graph.get(node, ()): + if neighbour in path: + continue + walk(neighbour, depth + 1, path | {neighbour}) + + walk(start, 0, {start}) + return found + + +ALLOWLIST = json.loads((ROOT / "tests" / "onco-reference" / "slice-allowlist.json").read_text(encoding="utf-8")) +UPSTREAM = ROOT / "upstream-onco" +SAMPLE = """ +const asOf = "2026-09-04"; +export const items = [ + { + id: "alpha", kind: "concept", name: "Alpha", asOf, + tldr: "Alpha is a synthetic record.", + summary: "Alpha summary.", + related: ["beta", "outside"], + }, + { id: "beta", kind: "method", name: "Beta", tldr: "Beta is linked.", summary: "Beta summary.", asOf: "2026-10-01", related: ["alpha"] }, +]; +""" + + +def _source() -> dict: + return onco_source("2026-10-19") + + +class AttributionAndProjection(unittest.TestCase): + def test_cc_by_nc_gaps_block_projection(self) -> None: + source = _source() + self.assertEqual(attribution_gaps(source), []) + incomplete = dict(source) + incomplete["modificationNotice"] = "" + self.assertIn("modificationNotice", attribution_gaps(incomplete)) + records = [ + {"id": "alpha", "kind": "concept", "title": "Alpha", "tldr": "A.", "summary": "A.", "asOf": "2026-10-19", "relations": [("related", "beta")]}, + {"id": "beta", "kind": "method", "title": "Beta", "tldr": "B.", "summary": "B.", "asOf": "2026-10-19", "relations": [("related", "alpha"), ("related", "missing")]}, + ] + with self.assertRaises(RuntimeError): + project_slice( + records, + source=incomplete, + kinds=[{"id": "concept", "label": "Concept", "plural": "Concepts"}, {"id": "method", "label": "Method", "plural": "Methods"}], + relation_labels=relation_labels(), + require_size=False, + ) + result = project_slice( + records, + source=source, + kinds=[ + {"id": "concept", "label": "Concept", "plural": "Concepts"}, + {"id": "method", "label": "Method", "plural": "Methods"}, + {"id": "term", "label": "Term", "plural": "Terms"}, + ] + [{"id": f"extra-{index}", "label": "Extra", "plural": "Extras"} for index in range(1)], + relation_labels=relation_labels(), + require_size=False, + ) + stored = {(item["from"], item["to"]) for item in result["snapshot"]["relations"]} + self.assertIn("urn:ce:onco-reference:concept:alpha", {item[0] for item in stored}) + self.assertTrue(any(item["to"] == "missing" for item in result["report"]["outsideAllowlist"])) + self.assertFalse(any("missing" in item["to"] for item in result["snapshot"]["relations"])) + self.assertEqual( + result["report"]["storedRelations"] + len(result["report"]["outsideAllowlist"]), + result["report"]["sourceEdges"], + ) + self.assertEqual(result["snapshot"]["sources"][0]["attribution"], "Data from OnCo (onco.cc)") + self.assertEqual(result["snapshot"]["sources"][0]["license"], "CC-BY-NC-4.0") + alpha = next(entity for entity in result["snapshot"]["entities"] if entity["id"].endswith(":alpha")) + self.assertEqual(len(alpha["sourceRefs"]), 2) + self.assertTrue(all(attribution_gaps(source) == [] for source in result["snapshot"]["sources"])) + + def test_parser_keeps_only_the_named_object(self) -> None: + obj = find_object(SAMPLE, "alpha") + fields = object_fields(obj, {"asOf": "2026-09-04"}) + self.assertEqual(fields["name"], "Alpha") + self.assertEqual(fields["asOf"], "2026-09-04") + self.assertEqual(fields["related"], ["beta", "outside"]) + self.assertNotIn('id: "beta"', obj) + + def test_refuses_a_destination_outside_the_local_directory(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + with self.assertRaises(RuntimeError): + assert_local_destination(Path(tmp) / "examples" / "dataspaces" / "onco", ROOT) + with self.assertRaises(RuntimeError): + write_projection(ROOT / "examples" / "dataspaces" / "onco-reference", {"snapshot": {}, "manifest": {}, "report": {}}, ROOT) + + def test_authorization_must_refuse_redistribution(self) -> None: + with tempfile.TemporaryDirectory() as tmp: + path = Path(tmp) / "authorization.json" + path.write_text(json.dumps({"granted": True, "distribution_allowed": True, "ci_allowed": False}), encoding="utf-8") + with self.assertRaises(RuntimeError): + load_authorization(path) + path.write_text((ROOT / "tests" / "onco-reference" / "authorization.example.json").read_text(encoding="utf-8"), encoding="utf-8") + self.assertTrue(load_authorization(path)["granted"]) + + def test_local_mutations_stay_explicit(self) -> None: + source = _source() + kinds = [ + {"id": "concept", "label": "Concept", "plural": "Concepts"}, + {"id": "method", "label": "Method", "plural": "Methods"}, + ] + alpha = {"id": "alpha", "kind": "concept", "title": "Alpha", "tldr": "A.", "summary": "A long.", "asOf": "2026-10-19", "relations": [("related", "beta")]} + beta = {"id": "beta", "kind": "method", "title": "Beta", "tldr": "B.", "summary": "B long.", "asOf": "2026-10-19", "relations": [("related", "alpha")]} + with self.assertRaises(RuntimeError): + project_slice([alpha, dict(alpha)], source=source, kinds=kinds, relation_labels=relation_labels(), require_size=False) + bare = dict(source) + del bare["license"] + with self.assertRaises(RuntimeError): + project_slice([alpha, beta], source=bare, kinds=kinds, relation_labels=relation_labels(), require_size=False) + removed = project_slice([alpha], source=source, kinds=kinds, relation_labels=relation_labels(), require_size=False) + self.assertEqual(removed["report"]["storedRelations"], 0) + self.assertEqual(removed["report"]["outsideAllowlist"], [{"from": "alpha", "predicate": "related", "to": "beta"}]) + renamed = dict(alpha) + renamed["title"] = "Alpha renamed locally" + projected = project_slice([renamed, beta], source=source, kinds=kinds, relation_labels=relation_labels(), require_size=False) + titles = {entity["id"]: entity["title"] for entity in projected["snapshot"]["entities"]} + self.assertEqual(titles["urn:ce:onco-reference:concept:alpha"], "Alpha renamed locally") + self.assertIn("Projected", projected["snapshot"]["sources"][0]["modificationNotice"]) + + +class LicensedSlice(unittest.TestCase): + def test_pinned_allowlist_projects_locally(self) -> None: + if not UPSTREAM.is_dir(): + self.skipTest("upstream checkout is absent") + self.assertEqual(upstream_sha(UPSTREAM), PIN) + entries = ALLOWLIST["records"] + self.assertGreaterEqual(len(entries), 12) + self.assertLessEqual(len(entries), 25) + records = extract_allowlist(UPSTREAM, entries, ROOT / "tests" / "onco-reference" / "authorization.json") + self.assertEqual([record["id"] for record in records], [entry["id"] for entry in entries]) + result = project_slice( + records, + source=_source(), + kinds=kinds_for(entries), + relation_labels=relation_labels(), + require_size=True, + ) + self.assertGreaterEqual(len(result["snapshot"]["entities"]), 12) + self.assertEqual( + result["report"]["storedRelations"] + len(result["report"]["outsideAllowlist"]), + result["report"]["sourceEdges"], + ) + self.assertGreater(result["report"]["sourceEdges"], result["report"]["storedRelations"]) + destination = ROOT / "tests" / "onco-reference" / "local" / "dataspace" + write_projection(destination, result, ROOT) + written = json.loads((destination / "snapshot.json").read_text(encoding="utf-8")) + self.assertEqual(written["sources"][0]["originSha"], PIN) + by_id = {entity["id"]: entity for entity in written["entities"]} + tnbc = by_id["urn:ce:onco-reference:cancer:tnbc"] + self.assertTrue(tnbc["sourceRefs"]) + neighbours = [ + relation["to"] if relation["from"] == tnbc["id"] else relation["from"] + for relation in written["relations"] + if tnbc["id"] in (relation["from"], relation["to"]) + ] + self.assertGreaterEqual(len(neighbours), 1) + graph: dict[str, set[str]] = {} + for relation in written["relations"]: + graph.setdefault(relation["from"], set()).add(relation["to"]) + graph.setdefault(relation["to"], set()).add(relation["from"]) + self.assertTrue(_has_simple_path(graph, tnbc["id"], 3)) + predicates = {relation["predicate"] for relation in written["relations"]} + self.assertGreaterEqual(len(predicates), 2) + labels = json.loads((destination / "manifest.json").read_text(encoding="utf-8"))["relationLabels"] + self.assertTrue(all(labels[predicate]["inverse"] for predicate in predicates)) + grouped: dict[tuple[str, str], int] = {} + for relation in written["relations"]: + grouped[(relation["from"], relation["predicate"])] = grouped.get((relation["from"], relation["predicate"]), 0) + 1 + self.assertTrue(any(count >= 2 for count in grouped.values())) + report = json.loads((destination / "import-report.json").read_text(encoding="utf-8")) + self.assertTrue(any("aka" in item["fields"] for item in report["missingOptional"])) + self.assertTrue(all(len(entity["sourceRefs"]) >= 2 for entity in written["entities"])) + self.assertTrue(all(attribution_gaps(source) == [] for source in written["sources"])) + self.assertTrue(any(entity["tldr"] != entity["summary"] and entity.get("asOf") for entity in written["entities"])) + self.assertEqual(labels and list(json.loads((destination / "manifest.json").read_text(encoding="utf-8"))["kindViews"].values())[0]["facets"], ["tags", "kind"]) + + +if __name__ == "__main__": + unittest.main() diff --git a/ui/app/[dataspace]/graph/page.tsx b/ui/app/[dataspace]/graph/page.tsx index 9957561..7333147 100644 --- a/ui/app/[dataspace]/graph/page.tsx +++ b/ui/app/[dataspace]/graph/page.tsx @@ -1,6 +1,7 @@ import type { Metadata } from "next"; import { notFound } from "next/navigation"; import { GraphExplorer } from "@/components/GraphExplorer"; +import { OriginNotice } from "@/components/OriginNotice"; import { findGraph } from "@/lib/catalog"; function first(value: string | string[] | undefined): string | undefined { @@ -33,5 +34,10 @@ export default async function GraphPage({ if (detail) initial.set("detail", detail); if (hops) initial.set("hops", hops); const initialSearch = initial.size > 0 ? `?${initial.toString()}` : ""; - return ; + return ( + <> + + + + ); } diff --git a/ui/components/DataspaceHome.tsx b/ui/components/DataspaceHome.tsx index 782a1e8..152e994 100644 --- a/ui/components/DataspaceHome.tsx +++ b/ui/components/DataspaceHome.tsx @@ -1,4 +1,5 @@ import { Hero } from "@/components/Hero"; +import { OriginNotice } from "@/components/OriginNotice"; import { RecordBrowser } from "@/components/RecordBrowser"; import { findDataspace, listNavigation } from "@/lib/catalog"; import { notFound } from "next/navigation"; @@ -25,6 +26,7 @@ export function DataspaceHome({ id }: { id: string }) { ))}

) : null} + ); diff --git a/ui/components/EntityDetail.test.tsx b/ui/components/EntityDetail.test.tsx index 659debc..9c9ab65 100644 --- a/ui/components/EntityDetail.test.tsx +++ b/ui/components/EntityDetail.test.tsx @@ -11,6 +11,31 @@ import { KindIndex } from "./KindIndex"; const systemsRoot = join(process.cwd(), "../examples/dataspaces/systems-thinking"); describe("kind and detail views", () => { + it("renders a CC origin notice from the source record", () => { + const entity = findEntity("systems-thinking", "concept", "feedback"); + expect(entity).not.toBeNull(); + const credited: EntityView = { + ...entity!, + sources: [{ + title: "OnCo", + license: "CC-BY-NC-4.0", + uri: "https://onco.cc", + attribution: "Data from OnCo (onco.cc)", + licenseUrl: "https://creativecommons.org/licenses/by-nc/4.0/", + retrievedAt: "2026-10-19", + originSha: "b594d7433bf4e408b44215e25e5304696d6f840e", + modificationNotice: "Projected from the pinned record.", + }], + }; + const html = renderToStaticMarkup(); + expect(html).toContain('href="https://onco.cc"'); + expect(html).toContain("Data from OnCo (onco.cc)"); + expect(html).toContain("https://creativecommons.org/licenses/by-nc/4.0/"); + expect(html).toContain("NonCommercial"); + expect(html).toContain("b594d7433bf4e408b44215e25e5304696d6f840e"); + expect(html).toContain("Projected from the pinned record."); + }); + it("renders the registry plural and a linked record", () => { const kind = findKind("software-architecture", "concept"); expect(kind).not.toBeNull(); diff --git a/ui/components/EntityDetail.tsx b/ui/components/EntityDetail.tsx index 1d70745..a072d5b 100644 --- a/ui/components/EntityDetail.tsx +++ b/ui/components/EntityDetail.tsx @@ -87,6 +87,16 @@ export function EntityDetail({ entity }: { entity: EntityView }) {
  • {source.title} {source.license ? · {source.license} : null} + {source.attribution ? ( +

    + {source.uri ? {source.attribution} : source.attribution} + {source.licenseUrl ? <> · {source.license} : null} + {source.retrievedAt ? <> · retrieved {source.retrievedAt} : null} + {source.originSha ? <> · origin {source.originSha} : null} + {source.license === "CC-BY-NC-4.0" ? <> · NonCommercial : null} + {source.modificationNotice ? <> · {source.modificationNotice} : null} +

    + ) : null}
    • ))} diff --git a/ui/components/KindIndex.tsx b/ui/components/KindIndex.tsx index a68f223..cb46da4 100644 --- a/ui/components/KindIndex.tsx +++ b/ui/components/KindIndex.tsx @@ -1,3 +1,4 @@ +import { OriginNotice } from "@/components/OriginNotice"; import { RecordBrowser } from "@/components/RecordBrowser"; import { PageHeader } from "@/components/PageHeader"; import type { KindView } from "@/lib/catalog"; @@ -9,6 +10,7 @@ export function KindIndex({ kind }: { kind: KindView }) { return ( <> + ); diff --git a/ui/components/OriginNotice.tsx b/ui/components/OriginNotice.tsx new file mode 100644 index 0000000..7df596b --- /dev/null +++ b/ui/components/OriginNotice.tsx @@ -0,0 +1,21 @@ +import { originNotices } from "@/lib/catalog"; + +/** One credit line for each distinct origin. Pages that render the records use this. */ +export function OriginNotice({ dataspaceId }: { dataspaceId: string }) { + const notices = originNotices(dataspaceId); + if (notices.length === 0) return null; + return ( + + ); +} diff --git a/ui/lib/attribution.test.ts b/ui/lib/attribution.test.ts new file mode 100644 index 0000000..42c518f --- /dev/null +++ b/ui/lib/attribution.test.ts @@ -0,0 +1,26 @@ +import { describe, expect, it } from "vitest"; +import { attributionGaps, uniqueOriginNotices, type SourceCredit } from "./attribution"; + +const onco: SourceCredit = { + title: "OnCo", + license: "CC-BY-NC-4.0", + uri: "https://onco.cc", + attribution: "Data from OnCo (onco.cc)", + licenseUrl: "https://creativecommons.org/licenses/by-nc/4.0/", + retrievedAt: "2026-10-19", + originSha: "b594d7433bf4e408b44215e25e5304696d6f840e", + modificationNotice: "Projected from the pinned record.", +}; + +describe("attribution", () => { + it("accepts a complete CC BY-NC notice and rejects a missing duty", () => { + expect(attributionGaps(onco)).toEqual([]); + expect(attributionGaps({ ...onco, modificationNotice: "" })).toContain("modificationNotice"); + expect(attributionGaps({ ...onco, licenseUrl: "https://example.invalid" })).toContain("licenseUrl"); + }); + + it("shows one notice when every record repeats the same origin", () => { + const record = { ...onco, title: "One record", uri: "https://onco.cc/records/one" }; + expect(uniqueOriginNotices([onco, record, { title: "Public", license: "CC0-1.0", uri: "https://example.test" }])).toEqual([onco]); + }); +}); diff --git a/ui/lib/attribution.ts b/ui/lib/attribution.ts new file mode 100644 index 0000000..04129f4 --- /dev/null +++ b/ui/lib/attribution.ts @@ -0,0 +1,47 @@ +export type SourceCredit = { + title: string; + license: string; + uri?: string; + attribution?: string; + licenseUrl?: string; + retrievedAt?: string; + originSha?: string; + modificationNotice?: string; +}; + +const LICENSE_URLS: Record = { + "CC0-1.0": "https://creativecommons.org/publicdomain/zero/1.0/", + "CC-BY-4.0": "https://creativecommons.org/licenses/by/4.0/", + "CC-BY-NC-4.0": "https://creativecommons.org/licenses/by-nc/4.0/", + MIT: "https://opensource.org/license/mit", +}; + +const ATTRIBUTION_LICENSES = new Set(["CC-BY-4.0", "CC-BY-NC-4.0", "MIT"]); + +export function uniqueOriginNotices(sources: SourceCredit[]): SourceCredit[] { + const seen = new Set(); + const notices: SourceCredit[] = []; + for (const source of sources) { + if (!source.attribution?.trim()) continue; + const key = [source.attribution, source.license, source.licenseUrl ?? "", source.originSha ?? "", source.modificationNotice ?? ""].join("\n"); + if (seen.has(key)) continue; + seen.add(key); + notices.push(source); + } + return notices; +} + +export function attributionGaps(source: SourceCredit): string[] { + const gaps: string[] = []; + if (!source.license) gaps.push("license"); + if (!source.title.trim()) gaps.push("title"); + if (!source.uri?.trim()) gaps.push("uri"); + if (ATTRIBUTION_LICENSES.has(source.license) && !source.attribution?.trim()) gaps.push("attribution"); + if (source.license === "CC-BY-4.0" || source.license === "CC-BY-NC-4.0") { + if (source.licenseUrl !== LICENSE_URLS[source.license]) gaps.push("licenseUrl"); + if (!source.retrievedAt?.trim()) gaps.push("retrievedAt"); + if (!source.originSha?.trim()) gaps.push("originSha"); + if (!source.modificationNotice?.trim()) gaps.push("modificationNotice"); + } + return gaps; +} diff --git a/ui/lib/browser-row.test.tsx b/ui/lib/browser-row.test.tsx index b6037b4..24eec22 100644 --- a/ui/lib/browser-row.test.tsx +++ b/ui/lib/browser-row.test.tsx @@ -13,14 +13,14 @@ const feedback = mapBrowserRow( route: "/systems-thinking/concept/feedback/", kindHref: "/systems-thinking/concept/", }, - ["tags", "phase"], + ["tags", "kind", "phase"], ); describe("mapBrowserRow", () => { it("fills generic columns and leaves unknown facets empty", () => { expect(feedback.name).toBe("Feedback"); expect(feedback.route).toBe("/systems-thinking/concept/feedback/"); - expect(feedback.facets).toEqual({ tags: ["regulation"], phase: [] }); + expect(feedback.facets).toEqual({ tags: ["regulation"], kind: ["Concept"], phase: [] }); expect(feedback.cols.tags).toEqual({ kind: "chips", values: ["regulation"] }); expect(feedback.cols.name).toEqual({ kind: "link", label: "Feedback", href: feedback.route }); expect("molecule" in feedback).toBe(false); diff --git a/ui/lib/browser-row.ts b/ui/lib/browser-row.ts index 28a6e6b..9423d33 100644 --- a/ui/lib/browser-row.ts +++ b/ui/lib/browser-row.ts @@ -56,7 +56,11 @@ export function mapBrowserRow(entity: MapperEntity, facetKeys: readonly string[] kind: { kind: "link", label: entity.kindLabel, href: entity.kindHref }, }; const facets: Record = {}; - for (const key of facetKeys) facets[key] = key === "tags" ? [...tags] : []; + for (const key of facetKeys) { + if (key === "tags") facets[key] = [...tags]; + else if (key === "kind") facets[key] = [entity.kindLabel]; + else facets[key] = []; + } return { id: entity.id, name: entity.title, diff --git a/ui/lib/catalog.test.ts b/ui/lib/catalog.test.ts index 050a9e4..edee639 100644 --- a/ui/lib/catalog.test.ts +++ b/ui/lib/catalog.test.ts @@ -1,7 +1,7 @@ import { readFileSync } from "node:fs"; import { join } from "node:path"; import { describe, expect, it } from "vitest"; -import { entitySlug, findDataspace, findEntity, findKind } from "./catalog"; +import { clearCatalogCache, dataspaceRoots, entitySlug, findDataspace, findEntity, findKind } from "./catalog"; const root = join(process.cwd(), "../examples/dataspaces"); @@ -13,6 +13,12 @@ function titles(dataspaceId: string, kindId: string): string[] { } describe("catalog", () => { + it("does not load the licensed slice unless that profile is enabled", () => { + expect(process.env.CONTENT_EXPLORER_LICENSED).toBeUndefined(); + expect(dataspaceRoots().some((root) => root.includes("onco-reference"))).toBe(false); + expect(findDataspace("onco-reference")).toBeNull(); + }); + it("takes the kind index from the snapshot registry", () => { const concept = findKind("systems-thinking", "concept"); const pattern = findKind("software-architecture", "pattern"); @@ -33,4 +39,31 @@ describe("catalog", () => { expect(findKind("systems-thinking", "no-such-kind")).toBeNull(); expect(findDataspace("software-architecture")?.title).toBe("Software Architecture"); }); + + it("loads the local slice only when the licensed profile is on and the slice exists", () => { + const prior = process.env.CONTENT_EXPLORER_LICENSED; + try { + process.env.CONTENT_EXPLORER_LICENSED = "1"; + clearCatalogCache(); + expect(dataspaceRoots().some((item) => item.endsWith("onco-reference/local"))).toBe(true); + const space = findDataspace("onco-reference"); + if (!space) return; + expect(space.kinds.length).toBeGreaterThanOrEqual(3); + const kind = findKind("onco-reference", "cancer"); + expect(kind?.rows.length).toBeGreaterThan(0); + expect(kind?.rows[0]?.facets.tags.length).toBeGreaterThan(0); + expect(kind?.rows[0]?.facets.kind.length).toBeGreaterThan(0); + const entity = findEntity("onco-reference", "cancer", "tnbc"); + expect(entity?.sources.length).toBeGreaterThanOrEqual(2); + expect(entity?.sources[0]?.license).toBe("CC-BY-NC-4.0"); + expect(entity?.sources[0]?.attribution).toContain("onco.cc"); + expect(entity?.sources[0]?.licenseUrl).toContain("creativecommons.org/licenses/by-nc/4.0"); + expect(entity?.sources[0]?.originSha).toBe("b594d7433bf4e408b44215e25e5304696d6f840e"); + expect(entity?.relations.length).toBeGreaterThan(0); + } finally { + if (prior === undefined) delete process.env.CONTENT_EXPLORER_LICENSED; + else process.env.CONTENT_EXPLORER_LICENSED = prior; + clearCatalogCache(); + } + }); }); diff --git a/ui/lib/catalog.ts b/ui/lib/catalog.ts index 1a5f11d..9749539 100644 --- a/ui/lib/catalog.ts +++ b/ui/lib/catalog.ts @@ -1,5 +1,6 @@ import { readdirSync, readFileSync } from "node:fs"; import { join } from "node:path"; +import { uniqueOriginNotices, type SourceCredit } from "./attribution"; import { columnDefs, HOME_COLUMNS, mapBrowserRow, type BrowserRow, type ColumnDef } from "./browser-row"; import { memberLinks, partitionIncident, readingPathCrumbs, type GraphEdge, type RelationLabels } from "./graph"; @@ -17,7 +18,17 @@ type EntityRecord = { properties?: unknown; }; -type SourceRecord = { id: string; title: string; license: string }; +type SourceRecord = { + id: string; + title: string; + license: string; + uri?: string; + attribution?: string; + licenseUrl?: string; + retrievedAt?: string; + originSha?: string; + modificationNotice?: string; +}; type ClaimRecord = { id: string; subjectId: string; key: string; value?: string; status: string }; @@ -100,7 +111,16 @@ export type EntityView = { kindHref: string; graphHref: string; glossary: string | null; - sources: { title: string; license: string }[]; + sources: { + title: string; + license: string; + uri?: string; + attribution?: string; + licenseUrl?: string; + retrievedAt?: string; + originSha?: string; + modificationNotice?: string; + }[]; claims: { key: string; status: string; value: string | null }[]; relations: RelationView[]; outgoing: RelationItem[]; @@ -141,8 +161,20 @@ type Loaded = { const ROOT = join(process.cwd(), "../examples/dataspaces"); +export function dataspaceRoots(): string[] { + const roots = [ROOT]; + if (process.env.CONTENT_EXPLORER_LICENSED === "1") { + roots.push(join(process.cwd(), "../tests/onco-reference/local")); + } + return roots; +} + let cache: Map | null = null; +export function clearCatalogCache(): void { + cache = null; +} + export function entitySlug(id: string, kindId: string): string { const marker = `:${kindId}:`; const at = id.lastIndexOf(marker); @@ -162,18 +194,25 @@ export function entityHref(dataspaceId: string, kindId: string, id: string): str function loadAll(): Map { if (cache) return cache; const loaded = new Map(); - for (const name of readdirSync(ROOT).sort()) { - const snapshotPath = join(ROOT, name, "snapshot.json"); + for (const root of dataspaceRoots()) { + let names: string[] = []; + try { + names = readdirSync(/*turbopackIgnore: true*/ root); + } catch { + continue; + } + for (const name of names.sort()) { + const snapshotPath = join(root, name, "snapshot.json"); let snapshot: SnapshotFile; try { - snapshot = JSON.parse(readFileSync(snapshotPath, "utf8")) as SnapshotFile; + snapshot = JSON.parse(readFileSync(/*turbopackIgnore: true*/ snapshotPath, "utf8")) as SnapshotFile; } catch { continue; } if (!snapshot.dataspace?.id || !Array.isArray(snapshot.kinds) || !Array.isArray(snapshot.entities)) continue; let manifest: ManifestFile = {}; try { - manifest = JSON.parse(readFileSync(join(ROOT, name, "manifest.json"), "utf8")) as ManifestFile; + manifest = JSON.parse(readFileSync(/*turbopackIgnore: true*/ join(root, name, "manifest.json"), "utf8")) as ManifestFile; } catch { manifest = {}; } @@ -190,6 +229,7 @@ function loadAll(): Map { views: manifest.kindViews ?? {}, byId: new Map(snapshot.entities.map((entity) => [entity.id, entity])), }); + } } cache = loaded; return loaded; @@ -228,6 +268,25 @@ export function listNavigation(): DataspaceNav[] { })); } +function sourceCredit(source: SourceRecord): SourceCredit { + return { + title: source.title, + license: source.license, + uri: source.uri, + attribution: source.attribution, + licenseUrl: source.licenseUrl, + retrievedAt: source.retrievedAt, + originSha: source.originSha, + modificationNotice: source.modificationNotice, + }; +} + +export function originNotices(dataspaceId: string): SourceCredit[] { + const loaded = loadAll().get(dataspaceId); + if (!loaded) return []; + return uniqueOriginNotices((loaded.snapshot.sources ?? []).map(sourceCredit)); +} + export function findDataspace(id: string): DataspaceView | null { const loaded = loadAll().get(id); if (!loaded) return null; @@ -358,7 +417,16 @@ export function findEntity(dataspaceId: string, kindId: string, slug: string): E glossary: self.glossary, sources: (entity.sourceRefs ?? []).map((id) => { const source = sources.get(id); - return { title: source?.title ?? id, license: source?.license ?? "" }; + return { + title: source?.title ?? id, + license: source?.license ?? "", + uri: source?.uri, + attribution: source?.attribution, + licenseUrl: source?.licenseUrl, + retrievedAt: source?.retrievedAt, + originSha: source?.originSha, + modificationNotice: source?.modificationNotice, + }; }), claims: (loaded.snapshot.claims ?? []) .filter((claim) => claim.subjectId === entity.id) diff --git a/workplans/CONTENT-WP-0006-external-pilot.md b/workplans/CONTENT-WP-0006-external-pilot.md index a7e1e42..59e54e3 100644 --- a/workplans/CONTENT-WP-0006-external-pilot.md +++ b/workplans/CONTENT-WP-0006-external-pilot.md @@ -4,7 +4,7 @@ type: workplan title: "Extensibility and first real-world pilot" domain: infotech repo: content-explorer -status: proposed +status: finished flavor: implementation depends_on: - CONTENT-WP-0005 @@ -17,7 +17,7 @@ state_hub_workstream_id: "b1767962-549f-5de2-9503-a997a21a2133" Attach an independently maintained knowledge source and validate user value. -The Wikidata export connector, its fixture certification, and the static-projection no-go are recorded in `connectors/wikidata_export.py`, `tests/test_wikidata_connector.py`, and `workplans/evidence/CONTENT-WP-0006-boundaries.md`. The real-user pilot and the observed-need tasks stay open. This workplan is not finished. +The Wikidata export connector, the single-operator computing-basics pilot, and the decision to keep static projection are recorded in `connectors/wikidata_export.py`, `connectors/AUTHORING.md`, and `workplans/evidence/CONTENT-WP-0006-Review.md`. ## Work packages @@ -43,7 +43,7 @@ state_hub_task_id: "a6f752d7-788a-54b7-8bd4-8a0f8889a6e4" ```task id: CONTENT-WP-0006-T03 -status: todo +status: done priority: high state_hub_task_id: "87e52381-e64d-5953-bf6d-1aa80d32819c" ``` @@ -52,7 +52,7 @@ state_hub_task_id: "87e52381-e64d-5953-bf6d-1aa80d32819c" ```task id: CONTENT-WP-0006-T04 -status: todo +status: done priority: high state_hub_task_id: "8ab644d6-274b-58c7-bb0b-9b58cd962def" ``` @@ -61,7 +61,7 @@ state_hub_task_id: "8ab644d6-274b-58c7-bb0b-9b58cd962def" ```task id: CONTENT-WP-0006-T05 -status: todo +status: done priority: high state_hub_task_id: "cf85f981-f685-5b61-99f1-bb25d1d98dc2" ``` diff --git a/workplans/CONTENT-WP-0007-onco-reference-slice.md b/workplans/CONTENT-WP-0007-onco-reference-slice.md index c4b44eb..619040b 100644 --- a/workplans/CONTENT-WP-0007-onco-reference-slice.md +++ b/workplans/CONTENT-WP-0007-onco-reference-slice.md @@ -4,7 +4,7 @@ type: workplan title: "OnCo licensed reference-slice conformance" domain: infotech repo: content-explorer -status: blocked +status: finished flavor: implementation depends_on: - CONTENT-WP-0000 @@ -13,13 +13,13 @@ depends_on: - CONTENT-WP-0003 - CONTENT-WP-0005 created: "2026-10-12" -updated: "2026-10-12" +updated: "2026-10-19" state_hub_workstream_id: "ba3fad1f-0af7-5851-b1c5-df401e0f0184" --- # OnCo licensed reference-slice conformance -Validate generic Content Explorer against real OnCo topology and UX using a small pinned, authorized cancer-data slice, without copying the broader corpus. No real data import until authorization exists. See `tests/onco-reference/README.md`. +Validate generic Content Explorer against a small pinned OnCo slice, without copying the broader corpus into the public tree. Local use is authorized when the CC BY-NC origin notice is shown. Redistribution and public CI are not granted. See `tests/onco-reference/README.md`. ## Work packages @@ -27,9 +27,7 @@ Validate generic Content Explorer against real OnCo topology and UX using a smal ```task id: CONTENT-WP-0007-T01 -status: wait -needs_human: true -blocking_reason: "OnCo LICENSE-DATA authorization is required before any real reference-slice import." +status: done priority: high state_hub_task_id: "04edc274-3eb1-5796-9a37-37380c1d6c92" ``` @@ -38,7 +36,7 @@ state_hub_task_id: "04edc274-3eb1-5796-9a37-37380c1d6c92" ```task id: CONTENT-WP-0007-T02 -status: todo +status: done priority: high state_hub_task_id: "8e39d927-55f5-5bae-b8f1-05a532dffd56" ``` @@ -47,7 +45,7 @@ state_hub_task_id: "8e39d927-55f5-5bae-b8f1-05a532dffd56" ```task id: CONTENT-WP-0007-T03 -status: todo +status: done priority: high state_hub_task_id: "c08c515b-90ef-5e70-a355-da9273f54d06" ``` @@ -56,7 +54,7 @@ state_hub_task_id: "c08c515b-90ef-5e70-a355-da9273f54d06" ```task id: CONTENT-WP-0007-T04 -status: todo +status: done priority: high state_hub_task_id: "a216484a-a919-543f-9c33-d74c86c6d65e" ``` @@ -65,7 +63,7 @@ state_hub_task_id: "a216484a-a919-543f-9c33-d74c86c6d65e" ```task id: CONTENT-WP-0007-T05 -status: todo +status: done priority: high state_hub_task_id: "4f33ae6e-da25-5223-a887-e5157b5e82a5" ``` @@ -74,7 +72,7 @@ state_hub_task_id: "4f33ae6e-da25-5223-a887-e5157b5e82a5" ```task id: CONTENT-WP-0007-T06 -status: todo +status: done priority: high state_hub_task_id: "23768695-13eb-5a92-8658-c3441e15bec6" ``` @@ -85,7 +83,7 @@ state_hub_task_id: "23768695-13eb-5a92-8658-c3441e15bec6" ```task id: CONTENT-WP-0007-T07 -status: todo +status: done priority: high state_hub_task_id: "85a14b74-38dd-594c-8394-e253f05b8223" ``` @@ -94,14 +92,14 @@ state_hub_task_id: "85a14b74-38dd-594c-8394-e253f05b8223" ```task id: CONTENT-WP-0007-T08 -status: todo +status: done priority: high state_hub_task_id: "33c75c3d-743f-533c-af86-fc7d77ec75f0" ``` ## Key risk -Data-use authorization is the blocker. Do not guess rights. Public fixtures remain independently authored. +The grant is local only. Public fixtures remain independently authored. Do not widen the grant to redistribution or public CI. ## Evidence diff --git a/workplans/README.md b/workplans/README.md index c2be35b..1516cf4 100644 --- a/workplans/README.md +++ b/workplans/README.md @@ -11,10 +11,10 @@ Hub ids use the `CONTENT-WP` prefix. The seed notes called these WP-00 through W | CONTENT-WP-0004 | Relations and journeys | 0002 + 0003 | Graph navigation | | CONTENT-WP-0005 | Quality and parity | begins with 0000, closes after 0004 | Release candidate | | CONTENT-WP-0006 | External pilot | 0005 | Decision to expand | -| CONTENT-WP-0007 | OnCo reference-slice conformance | 0000, 0001, 0002, 0003, 0005 | Blocked on data-use authorization | +| CONTENT-WP-0007 | OnCo reference-slice conformance | 0000, 0001, 0002, 0003, 0005 | Local CC BY-NC reference slice | **Minimum viable proof:** 0000 → 0001 → 0002, with JSON snapshot ingestion. The 0003 provenance gate is required before any externally ingested public content is published. 0004 and later need not delay the basic demo. -No effort estimates are asserted before cloning and profiling the upstream code. Priority: 0000 and 0001 now; then 0002 and 0003; then 0004 and 0005; then the 0006 pilot. 0007 stays blocked until OnCo data use is authorized. +No effort estimates are asserted before cloning and profiling the upstream code. Priority: 0000 and 0001 now; then 0002 and 0003; then 0004 and 0005; then the 0006 pilot. 0007 is local conformance against the pinned OnCo slice. The slice is not redistributed and is not part of public CI. `INDEX.md` was renamed to this file so the State Hub workplan scan does not treat the map as a workplan. diff --git a/workplans/evidence/CONTENT-WP-0006-ImplementationNotes.md b/workplans/evidence/CONTENT-WP-0006-ImplementationNotes.md new file mode 100644 index 0000000..fa04358 --- /dev/null +++ b/workplans/evidence/CONTENT-WP-0006-ImplementationNotes.md @@ -0,0 +1,30 @@ +# CONTENT-WP-0006 implementation notes + +Date: 2026-10-19 + +## Pilot + +T03 is one operator session. The repository owner asked for the pilot. The session served `npx next build` and `npx next start --hostname 127.0.0.1 --port 3492` from `ui/` with `CONTENT_EXPLORER_LICENSED` unset. + +| Request | Result | +|---|---| +| `/` | 200. Search form present. Link to `/computing-basics`. No CC BY-NC notice. | +| `/computing-basics/` | 200. Python is in the index. | +| `/computing-basics/programming-language/` | 200. Python is in the kind index. | +| `/computing-basics/programming-language/python/` | 200. Source shows `CC0-1.0` and `https://www.wikidata.org`. One neighbour route. | +| `/computing-basics/?q=Python` | 200. The query is kept on the URL. Filtering still runs in the browser after HTML is sent. | +| `/computing-basics/graph/` | 200. | + +No other participants were interviewed. + +## Observed interfaces + +T04 does not add an API, JSON-LD, MCP, Ask, or a search backend. The static pages covered browse, kind, detail, neighbour, source, and query-string search. `decisions/ADR-0002-static-projection-first.md` and `INTENT.md` principle 8 stay in force. A production LLM answer service remains a v0.1 non-goal. + +## Governance + +T05 is the "Update and governance" section of `connectors/AUTHORING.md`. The repository owner refreshes a pin and reviews the license before a new source is activated. Wikidata computing-basics stays CC0. The OnCo slice is outside `publish_directory`. + +## UI + +Kind indexes can declare a `kind` facet. `mapBrowserRow` fills it from the kind label. Tags stay the other facet. Public manifests still list `tags` only. The OnCo manifest lists both, and that manifest is written only under the gitignored local directory. diff --git a/workplans/evidence/CONTENT-WP-0006-Review.md b/workplans/evidence/CONTENT-WP-0006-Review.md new file mode 100644 index 0000000..6617213 --- /dev/null +++ b/workplans/evidence/CONTENT-WP-0006-Review.md @@ -0,0 +1,15 @@ +# CONTENT-WP-0006 review + +Date: 2026-10-19 + +T01, T02, T06, T07, and T08 were already done with the Wikidata connector. This review closes T03, T04, and T05. + +T03 passed as a single-operator production session on computing-basics, recorded in `CONTENT-WP-0006-ImplementationNotes.md`. It is not a field study. The kind names on those pages come from the snapshot registry. `scripts/check_ui_boundary.py` still rejects oncology imports in the UI. + +T04 is a no-go for API, JSON-LD, MCP, Ask, and a search backend. The session did not produce a measured limit that would reopen `decisions/ADR-0002-static-projection-first.md`. + +T05 names the repository owner as the person who refreshes pins and reviews licenses. The text is in `connectors/AUTHORING.md`. + +R8: the Wikidata pin is a local export, not a live API. The pilot did not exercise rate limits or a second operator. Those remain open if a later source needs them. They do not block this workplan. + +Public check after these notes: `sh scripts/ci-synthetic.sh` (fixture validation, `scripts/audit_public_tree.py`, unittest, UI boundary, UI provenance, quality report, Vitest, typecheck). diff --git a/workplans/evidence/CONTENT-WP-0006-boundaries.md b/workplans/evidence/CONTENT-WP-0006-boundaries.md index bf4ff26..afbbfe0 100644 --- a/workplans/evidence/CONTENT-WP-0006-boundaries.md +++ b/workplans/evidence/CONTENT-WP-0006-boundaries.md @@ -8,16 +8,16 @@ Wikidata is the connector that landed. The pinned input is `examples/sources/wik ## Real-user pilot -A pilot with real users is not done. `workplans/CONTENT-WP-0006-external-pilot.md` keeps CONTENT-WP-0006-T03 open for that pilot. This note is the stand-in for a harness that cannot hold those sessions. +CONTENT-WP-0006-T03 is one operator session on 2026-10-19. The repository owner asked for the pilot, and the session ran the production computing-basics explorer. It is not a multi-person study and it does not invent other participants. The routes and results are in `workplans/evidence/CONTENT-WP-0006-Review.md`. ## API, JSON-LD, MCP, and Ask -Those interfaces are not built. `INTENT.md` principle 8 says static and exportable graph and HTTP/JSON come first, and MCP and richer interfaces come later. CONTENT-WP-0006-T04 stays open until needs are observed from the pilot. `INTENT.md` also lists a production LLM answer service as a v0.1 non-goal. +Those interfaces are not built. The pilot's browse, kind, detail, neighbour, source, and in-browser search journeys did not show a need for them. `INTENT.md` principle 8 keeps static and exportable graph and HTTP/JSON first. A production LLM answer service stays a v0.1 non-goal. CONTENT-WP-0006-T04 records that no-go. ## Dynamic deployment and search -Dynamic deployment and a search backend stay a no-go. `decisions/ADR-0002-static-projection-first.md` decides to begin with offline connectors and a versioned static snapshot, with filter-in-browser at small scale, and to add remote read APIs, indexed search, and graph storage only with measured limits. CONTENT-WP-0006-T08 is the gate for that go or no-go. This increment follows the no-go. +Dynamic deployment and a search backend stay a no-go. `decisions/ADR-0002-static-projection-first.md` decides to begin with offline connectors and a versioned static snapshot, with filter-in-browser at small scale, and to add remote read APIs, indexed search, and graph storage only with measured limits. CONTENT-WP-0006-T08 is that no-go. ## OnCo authorization -CONTENT-WP-0007 stays blocked until a human OnCo data-use authorization exists. `workplans/CONTENT-WP-0007-onco-reference-slice.md` sets that workplan to `blocked`, and CONTENT-WP-0007-T01 waits on `OnCo LICENSE-DATA` before any real reference-slice import. This increment does not read `upstream-onco/` data. +The repository owner authorized local OnCo reference use on 2026-10-19, limited to a data-origin notice that meets the CC BY-NC 4.0 duties. Redistribution, public CI, commercial use, and a company internal data pipeline are not granted. The non-confidential record is `tests/onco-reference/authorization.example.json`. The private copy stays gitignored. CONTENT-WP-0007 is the conformance workplan for that grant. diff --git a/workplans/evidence/CONTENT-WP-0007-ImplementationNotes.md b/workplans/evidence/CONTENT-WP-0007-ImplementationNotes.md new file mode 100644 index 0000000..9029a4e --- /dev/null +++ b/workplans/evidence/CONTENT-WP-0007-ImplementationNotes.md @@ -0,0 +1,45 @@ +# CONTENT-WP-0007 implementation notes + +Date: 2026-10-19 + +## Grant + +The repository owner approved local OnCo use when origin attribution matches the CC BY-NC 4.0 duties: creator attribution, license link, link to the material, retrieval date, pinned origin, modification notice, and a non-commercial mark. `tests/onco-reference/authorization.example.json` is the committed record. `tests/onco-reference/authorization.json` is the same grant and is gitignored. `distribution_allowed` and `ci_allowed` are false. `extract_allowlist` refuses to read the pin unless that file grants local use and refuses redistribution. + +## Slice + +Pin `b594d7433bf4e408b44215e25e5304696d6f840e`. Allowlist `tests/onco-reference/slice-allowlist.json` has 17 ids and no prose. The adapter writes `tests/onco-reference/local/dataspace/`, which is gitignored. + +Counts from the licensed projection: 17 entities, 9 kinds, 18 sources, 41 stored relations, 884 outside-allowlist edges, 925 source edges. Stored plus outside equals source edges. Eight predicates are stored. Sixteen records omit the optional `aka` field, and that absence is listed in `missingOptional`. Every entity has a dated TL;DR and a summary, and at least two source refs. A simple path of three edges exists from `urn:ce:onco-reference:cancer:tnbc`. At least one entity has two stored edges of the same predicate. + +`content_explorer/attribution.py` blocks projection when a CC BY or CC BY-NC duty is missing. Public `publish_directory` still refuses `CC-BY-NC-4.0`. + +## Same UI + +`CONTENT_EXPLORER_LICENSED=1` adds `tests/onco-reference/local` as a catalog root. The public default does not. Home, kind, and graph pages render `OriginNotice` when a source carries attribution. Detail pages keep a credit line on each source. CC0 dataspaces do not show that notice. + +Filesystem reads in `ui/lib/catalog.ts` are marked `turbopackIgnore` so a production build does not trace the local slice into the server bundle. After the licensed build, the server chunk contains the path string and does not contain `src:onco:reference-slice`. + +## Journeys + +Licensed `next start` on 127.0.0.1:3492, 2026-10-19: + +| Request | Attribution notice | +|---|---| +| `/onco-reference/` | present | +| `/onco-reference/cancer/` | present | +| `/onco-reference/graph/` | present | +| `/onco-reference/cancer/tnbc/` | present, with neighbour links | +| `/onco-reference/term/bystander-effect/` | present | +| `/onco-reference/?q=TNBC` | 200, query kept, filter remains in the browser | +| `/onco-reference/cancer/?tags=cancer&kind=Cancer` | 200, both facet keys kept | +| `/computing-basics/` on the same server | notice absent | +| `/` | notice absent | + +An unknown slug renders the Next.js not-found document, with HTTP 200, the same way an unknown computing-basics slug does. + +## Exceptions + +The pinned OnCo UI was not started. Full `next build` of that checkout has previously exhausted memory, and screenshots of the slice would redistribute CC BY-NC text. Visual pixel parity and the 1280×800 / 390×844 captures are not claimed. Semantic comparison is the adapter test: allowlist order, edge accounting, attribution gaps, and the HTTP notice checks above. + +Public CI is `sh scripts/ci-synthetic.sh`. It does not fetch OnCo. The licensed unittest skips when `upstream-onco/` is absent. diff --git a/workplans/evidence/CONTENT-WP-0007-Review.md b/workplans/evidence/CONTENT-WP-0007-Review.md new file mode 100644 index 0000000..45f404b --- /dev/null +++ b/workplans/evidence/CONTENT-WP-0007-Review.md @@ -0,0 +1,21 @@ +# CONTENT-WP-0007 review + +Date: 2026-10-19 + +T01 is done. The grantor is the repository owner, the date is 2026-10-19, and the scope is local reference-slice conformance. Redistribution, public CI, commercial use, and a company internal pipeline are outside the grant. + +T02 pins `b594d7433bf4e408b44215e25e5304696d6f840e` and selects 17 entities. The journey table is in `CONTENT-WP-0007-ImplementationNotes.md`. Viewport screenshots are the documented exception in that note. + +T03 is `connectors/onco_reference.py`. It reads only the allowlist, refuses a pin mismatch, and refuses a destination outside `tests/onco-reference/local/` or inside `examples/`. + +T04 is `tests/test_onco_reference.py` plus the catalog and attribution UI tests. Covered: two or more predicates, inverse labels, a one-to-many edge, a simple path of depth 3, an explicit unresolved edge, a missing optional field, two facets, detail credits, and local mutations for a duplicate id, a removed license, a removed node, and a renamed title. The renamed title stays under the modification notice. + +T05 compares structure and rights, not pixels. The edge equation is 41 stored + 884 outside = 925. Attribution gaps on the written sources are empty. Upstream visual parity is the exception in the implementation notes. + +T06: `scripts/audit_public_tree.py` runs inside `scripts/ci-synthetic.sh` and fails if the local slice, `authorization.json`, or `upstream-onco/` is tracked, or if example JSON carries `CC-BY-NC-4.0` or `Data from OnCo`. The licensed build's server chunk does not embed the slice id. + +T07: extraction calls `load_authorization` first. A grant with `distribution_allowed: true` is refused. A source with a missing CC duty is refused. Outside-allowlist edges are reported and are not stored. + +T08: the allowlist pin, the source `originSha`, and this review are the reproducibility record. The local snapshot is regenerated by the licensed test and is not committed. + +Public fixtures stay the executable default. The oncology kind labels live in the adapter and the gitignored snapshot, not in a UI import.