Finish the computing-basics pilot and the local OnCo reference slice.
All checks were successful
synthetic / suite (push) Successful in 56s
All checks were successful
synthetic / suite (push) Successful in 56s
The single-operator pilot keeps static projection. Local OnCo use requires a CC BY-NC origin notice, and the slice stays out of public CI. Assistant: grok Assistant-Session: 01a15378-f601-7fe2-a143-8e1500a1d59c
This commit is contained in:
parent
55f6180b8e
commit
f8c94d3272
33 changed files with 1278 additions and 48 deletions
|
|
@ -50,5 +50,5 @@ curl -s -X POST http://127.0.0.1:8000/progress/ \
|
||||||
## Boundaries
|
## Boundaries
|
||||||
|
|
||||||
- Do not copy the OnCo corpus, logos, or CC BY-NC dataset into this repo.
|
- Do not copy the OnCo corpus, logos, or CC BY-NC dataset into this repo.
|
||||||
- `tests/onco-reference/local/` and `authorization.json` stay gitignored until CONTENT-WP-0007's data-use gate is cleared.
|
- `tests/onco-reference/local/` and `authorization.json` stay gitignored. Local reference use is authorized only with the CC origin notice. Redistribution and public CI are not granted.
|
||||||
- Fetch and pin upstream with `scripts/bootstrap-upstream.sh` under CONTENT-WP-0000. Do not treat the seed as a fork.
|
- Fetch and pin upstream with `scripts/bootstrap-upstream.sh` under CONTENT-WP-0000. Do not treat the seed as a fork.
|
||||||
|
|
|
||||||
|
|
@ -27,4 +27,4 @@ python3 -m unittest tests.test_contract tests.test_ingest
|
||||||
|
|
||||||
## OnCo comparison track
|
## OnCo comparison track
|
||||||
|
|
||||||
[CONTENT-WP-0007](workplans/CONTENT-WP-0007-onco-reference-slice.md) specifies a small, pinned, **authorized** OnCo reference-slice test program. No actual OnCo data is included. Public fixtures remain independently authored; real-data ingestion is disabled until the relevant data rights are cleared.
|
[CONTENT-WP-0007](workplans/CONTENT-WP-0007-onco-reference-slice.md) is a small, pinned OnCo reference slice for local conformance. The repository owner authorized that local use when the page shows the CC BY-NC 4.0 origin notice. The slice stays gitignored. Public fixtures remain independently authored, and public CI does not fetch or publish the corpus.
|
||||||
|
|
|
||||||
|
|
@ -10,3 +10,11 @@ Certify a connector with a pinned local file, not a live API:
|
||||||
4. A changed file refreshes the snapshot. The report records the connector version, the input digest, and the counts.
|
4. A changed file refreshes the snapshot. The report records the connector version, the input digest, and the counts.
|
||||||
|
|
||||||
`connectors/wikidata_export.py` and `tests/test_wikidata_connector.py` are the worked example. The input is `examples/sources/wikidata/computing-basics.json`.
|
`connectors/wikidata_export.py` and `tests/test_wikidata_connector.py` are the worked example. The input is `examples/sources/wikidata/computing-basics.json`.
|
||||||
|
|
||||||
|
## Update and governance
|
||||||
|
|
||||||
|
The repository owner refreshes a pinned source. A new source needs a license review before activation, and the review names the license, the attribution duties, and whether `publish_directory` may write it under `examples/`. `CC-BY-NC-4.0` stays out of that allowlist.
|
||||||
|
|
||||||
|
Wikidata computing-basics is CC0. Refresh it by replacing `examples/sources/wikidata/computing-basics.json` with a new pinned export, then rerun the connector tests. The import report keeps the connector version and the input digest.
|
||||||
|
|
||||||
|
The OnCo reference slice is a separate adapter, `connectors/onco_reference.py`. It writes only under `tests/onco-reference/local/` after `tests/onco-reference/authorization.json` grants local use and refuses redistribution and public CI. The owner of that file is the repository owner. The slice is not a published dataspace. Pages that render it show the CC BY-NC origin notice from the source record: attribution, license link, material link, retrieval date, origin SHA, non-commercial mark, and modification notice.
|
||||||
|
|
|
||||||
502
connectors/onco_reference.py
Normal file
502
connectors/onco_reference.py
Normal file
|
|
@ -0,0 +1,502 @@
|
||||||
|
"""Allowlist-only OnCo reference adapter.
|
||||||
|
|
||||||
|
Reads enumerated records from a local upstream checkout and writes a snapshot
|
||||||
|
under the gitignored reference directory. It does not publish into
|
||||||
|
``examples/`` and it does not import UI code. Edges that leave the allowlist
|
||||||
|
are reported and are not stored as relations.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from content_explorer.attribution import attribution_gaps
|
||||||
|
from content_explorer.contract import validate_documents
|
||||||
|
from content_explorer.ingest import build_manifest
|
||||||
|
|
||||||
|
PIN = "b594d7433bf4e408b44215e25e5304696d6f840e"
|
||||||
|
LICENSE = "CC-BY-NC-4.0"
|
||||||
|
LICENSE_URL = "https://creativecommons.org/licenses/by-nc/4.0/"
|
||||||
|
ATTRIBUTION = "Data from OnCo (onco.cc)"
|
||||||
|
MODIFICATION = (
|
||||||
|
"Projected into a content-explorer snapshot. Titles, summaries, and "
|
||||||
|
"in-slice relations keep the pinned OnCo wording. Edges outside the "
|
||||||
|
"allowlist are listed in the import report and are not stored."
|
||||||
|
)
|
||||||
|
RELATION_FIELDS = {
|
||||||
|
"targets": "targets",
|
||||||
|
"drugs": "drugs",
|
||||||
|
"trials": "trials",
|
||||||
|
"terms": "terms",
|
||||||
|
"technologies": "technologies",
|
||||||
|
"pathways": "pathways",
|
||||||
|
"companies": "companies",
|
||||||
|
"institutions": "institutions",
|
||||||
|
"cancers": "cancers",
|
||||||
|
"related": "related",
|
||||||
|
"people": "people",
|
||||||
|
"papers": "papers",
|
||||||
|
"keyPapers": "key-papers",
|
||||||
|
}
|
||||||
|
LOCAL_ROOT_NAME = Path("tests") / "onco-reference" / "local"
|
||||||
|
|
||||||
|
|
||||||
|
def load_authorization(path: Path) -> dict[str, Any]:
|
||||||
|
document = json.loads(path.read_text(encoding="utf-8"))
|
||||||
|
if not isinstance(document, dict) or document.get("granted") is not True:
|
||||||
|
raise RuntimeError("OnCo reference use is not authorized")
|
||||||
|
if document.get("distribution_allowed") is not False or document.get("ci_allowed") is not False:
|
||||||
|
raise RuntimeError("this grant does not allow redistribution or public CI")
|
||||||
|
return document
|
||||||
|
|
||||||
|
|
||||||
|
def _skip_string(text: str, index: int) -> int:
|
||||||
|
quote = text[index]
|
||||||
|
index += 1
|
||||||
|
length = len(text)
|
||||||
|
while index < length:
|
||||||
|
char = text[index]
|
||||||
|
if char == "\\":
|
||||||
|
index += 2
|
||||||
|
continue
|
||||||
|
if quote == "`" and char == "$" and index + 1 < length and text[index + 1] == "{":
|
||||||
|
depth = 1
|
||||||
|
index += 2
|
||||||
|
while index < length and depth:
|
||||||
|
if text[index] in ('"', "'", "`"):
|
||||||
|
index = _skip_string(text, index)
|
||||||
|
continue
|
||||||
|
if text[index] == "{":
|
||||||
|
depth += 1
|
||||||
|
elif text[index] == "}":
|
||||||
|
depth -= 1
|
||||||
|
index += 1
|
||||||
|
continue
|
||||||
|
if char == quote:
|
||||||
|
return index + 1
|
||||||
|
index += 1
|
||||||
|
return index
|
||||||
|
|
||||||
|
|
||||||
|
def _skip_comment(text: str, index: int) -> int:
|
||||||
|
if text.startswith("//", index):
|
||||||
|
newline = text.find("\n", index)
|
||||||
|
return length if (length := len(text)) and newline < 0 else newline + 1
|
||||||
|
if text.startswith("/*", index):
|
||||||
|
end = text.find("*/", index + 2)
|
||||||
|
return len(text) if end < 0 else end + 2
|
||||||
|
return index
|
||||||
|
|
||||||
|
|
||||||
|
def find_object(text: str, entity_id: str) -> str:
|
||||||
|
"""Return the object literal whose own ``id`` is ``entity_id``."""
|
||||||
|
length = len(text)
|
||||||
|
index = 0
|
||||||
|
depth = 0
|
||||||
|
starts: list[int] = []
|
||||||
|
pending: tuple[int, int] | None = None
|
||||||
|
while index < length:
|
||||||
|
char = text[index]
|
||||||
|
if char in ('"', "'", "`"):
|
||||||
|
index = _skip_string(text, index)
|
||||||
|
continue
|
||||||
|
if char == "/" and index + 1 < length and text[index + 1] in "/*":
|
||||||
|
index = _skip_comment(text, index)
|
||||||
|
continue
|
||||||
|
if char == "{":
|
||||||
|
starts.append(index)
|
||||||
|
depth += 1
|
||||||
|
index += 1
|
||||||
|
continue
|
||||||
|
if char == "}":
|
||||||
|
start = starts.pop() if starts else 0
|
||||||
|
if pending and pending[0] == depth:
|
||||||
|
return text[start:index + 1]
|
||||||
|
depth -= 1
|
||||||
|
index += 1
|
||||||
|
continue
|
||||||
|
if text.startswith("id", index) and (index == 0 or not (text[index - 1].isalnum() or text[index - 1] == "_")):
|
||||||
|
cursor = index + 2
|
||||||
|
while cursor < length and text[cursor].isspace():
|
||||||
|
cursor += 1
|
||||||
|
if cursor < length and text[cursor] == ":":
|
||||||
|
cursor += 1
|
||||||
|
while cursor < length and text[cursor].isspace():
|
||||||
|
cursor += 1
|
||||||
|
if cursor < length and text[cursor] in ('"', "'"):
|
||||||
|
end = _skip_string(text, cursor)
|
||||||
|
literal = text[cursor + 1:end - 1]
|
||||||
|
if literal == entity_id and starts:
|
||||||
|
pending = (depth, starts[-1])
|
||||||
|
index += 1
|
||||||
|
raise ValueError(f"{entity_id} was not found")
|
||||||
|
|
||||||
|
|
||||||
|
def _string_literal(text: str, index: int) -> tuple[str, int]:
|
||||||
|
end = _skip_string(text, index)
|
||||||
|
return text[index + 1:end - 1], end
|
||||||
|
|
||||||
|
|
||||||
|
def object_fields(text: str, constants: dict[str, str]) -> dict[str, Any]:
|
||||||
|
"""Read depth-1 keys. Nested objects are skipped. String arrays are kept."""
|
||||||
|
fields: dict[str, Any] = {}
|
||||||
|
length = len(text)
|
||||||
|
index = 0
|
||||||
|
depth = 0
|
||||||
|
while index < length:
|
||||||
|
char = text[index]
|
||||||
|
if char in ('"', "'", "`"):
|
||||||
|
index = _skip_string(text, index)
|
||||||
|
continue
|
||||||
|
if char == "/" and index + 1 < length and text[index + 1] in "/*":
|
||||||
|
index = _skip_comment(text, index)
|
||||||
|
continue
|
||||||
|
if char == "{":
|
||||||
|
depth += 1
|
||||||
|
index += 1
|
||||||
|
continue
|
||||||
|
if char == "}":
|
||||||
|
depth -= 1
|
||||||
|
index += 1
|
||||||
|
continue
|
||||||
|
if depth != 1 or not (char.isalpha() or char == "_"):
|
||||||
|
index += 1
|
||||||
|
continue
|
||||||
|
start = index
|
||||||
|
while index < length and (text[index].isalnum() or text[index] == "_"):
|
||||||
|
index += 1
|
||||||
|
key = text[start:index]
|
||||||
|
while index < length and text[index].isspace():
|
||||||
|
index += 1
|
||||||
|
if index < length and text[index] in ",}":
|
||||||
|
if key in constants:
|
||||||
|
fields[key] = constants[key]
|
||||||
|
continue
|
||||||
|
if index >= length or text[index] != ":":
|
||||||
|
continue
|
||||||
|
index += 1
|
||||||
|
while index < length and text[index].isspace():
|
||||||
|
index += 1
|
||||||
|
if index >= length:
|
||||||
|
break
|
||||||
|
if text[index] in ('"', "'", "`"):
|
||||||
|
value, index = _string_literal(text, index)
|
||||||
|
fields[key] = value
|
||||||
|
continue
|
||||||
|
if text[index] == "[":
|
||||||
|
index += 1
|
||||||
|
values: list[str] = []
|
||||||
|
while index < length and text[index] != "]":
|
||||||
|
if text[index] in ('"', "'", "`"):
|
||||||
|
value, index = _string_literal(text, index)
|
||||||
|
values.append(value)
|
||||||
|
continue
|
||||||
|
if text[index] in ('"', "'", "`"):
|
||||||
|
index = _skip_string(text, index)
|
||||||
|
continue
|
||||||
|
if text[index] == "{":
|
||||||
|
nested = 1
|
||||||
|
index += 1
|
||||||
|
while index < length and nested:
|
||||||
|
if text[index] in ('"', "'", "`"):
|
||||||
|
index = _skip_string(text, index)
|
||||||
|
continue
|
||||||
|
if text[index] == "{":
|
||||||
|
nested += 1
|
||||||
|
elif text[index] == "}":
|
||||||
|
nested -= 1
|
||||||
|
index += 1
|
||||||
|
continue
|
||||||
|
index += 1
|
||||||
|
fields[key] = values
|
||||||
|
index += 1
|
||||||
|
continue
|
||||||
|
ident_start = index
|
||||||
|
while index < length and (text[index].isalnum() or text[index] == "_"):
|
||||||
|
index += 1
|
||||||
|
ident = text[ident_start:index]
|
||||||
|
if ident in constants:
|
||||||
|
fields[key] = constants[ident]
|
||||||
|
continue
|
||||||
|
if index == ident_start:
|
||||||
|
index += 1
|
||||||
|
continue
|
||||||
|
return fields
|
||||||
|
|
||||||
|
|
||||||
|
def file_constants(text: str) -> dict[str, str]:
|
||||||
|
constants: dict[str, str] = {}
|
||||||
|
marker = "const asOf = "
|
||||||
|
at = text.find(marker)
|
||||||
|
if at >= 0:
|
||||||
|
cursor = at + len(marker)
|
||||||
|
if cursor < len(text) and text[cursor] in ('"', "'"):
|
||||||
|
value, _end = _string_literal(text, cursor)
|
||||||
|
constants["asOf"] = value
|
||||||
|
return constants
|
||||||
|
|
||||||
|
|
||||||
|
OPTIONAL_FIELDS = ("aka", "status")
|
||||||
|
|
||||||
|
|
||||||
|
def read_record(upstream: Path, entry: dict[str, str]) -> dict[str, Any]:
|
||||||
|
path = upstream / entry["file"]
|
||||||
|
text = path.read_text(encoding="utf-8")
|
||||||
|
obj = find_object(text, entry["id"])
|
||||||
|
fields = object_fields(obj, file_constants(text))
|
||||||
|
if fields.get("id") != entry["id"]:
|
||||||
|
raise ValueError(f"{entry['id']} object did not keep its id")
|
||||||
|
title = str(fields.get("name") or "").strip()
|
||||||
|
tldr = str(fields.get("tldr") or "").strip()
|
||||||
|
if not title or not tldr:
|
||||||
|
raise ValueError(f"{entry['id']} is missing a name or tldr")
|
||||||
|
relations: list[tuple[str, str]] = []
|
||||||
|
for field, predicate in RELATION_FIELDS.items():
|
||||||
|
for target in fields.get(field) or []:
|
||||||
|
if isinstance(target, str) and target:
|
||||||
|
relations.append((predicate, target))
|
||||||
|
return {
|
||||||
|
"id": entry["id"],
|
||||||
|
"kind": entry["kind"],
|
||||||
|
"title": title,
|
||||||
|
"tldr": tldr,
|
||||||
|
"summary": str(fields.get("summary") or tldr).strip(),
|
||||||
|
"asOf": str(fields.get("asOf") or "2026-09-04"),
|
||||||
|
"relations": relations,
|
||||||
|
"file": entry["file"],
|
||||||
|
"missingFields": [
|
||||||
|
name for name in OPTIONAL_FIELDS
|
||||||
|
if name not in fields or fields.get(name) in ("", None, [])
|
||||||
|
],
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def project_slice(
|
||||||
|
records: list[dict[str, Any]],
|
||||||
|
*,
|
||||||
|
source: dict[str, Any],
|
||||||
|
kinds: list[dict[str, str]],
|
||||||
|
relation_labels: dict[str, dict[str, str]],
|
||||||
|
require_size: bool = True,
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
"""Build a snapshot. Outside-allowlist edges are reported, not stored."""
|
||||||
|
gaps = attribution_gaps(source)
|
||||||
|
if gaps:
|
||||||
|
raise RuntimeError("refusing to project a source with attribution gaps: " + ", ".join(gaps))
|
||||||
|
by_upstream = {record["id"]: record for record in records}
|
||||||
|
if len(by_upstream) != len(records):
|
||||||
|
raise RuntimeError("duplicate upstream id")
|
||||||
|
outside: list[dict[str, str]] = []
|
||||||
|
relations: list[dict[str, Any]] = []
|
||||||
|
entities: list[dict[str, Any]] = []
|
||||||
|
claims: list[dict[str, Any]] = []
|
||||||
|
predicates: list[str] = []
|
||||||
|
record_sources: list[dict[str, Any]] = []
|
||||||
|
dataspace = "onco-reference"
|
||||||
|
for record in records:
|
||||||
|
kind = record["kind"]
|
||||||
|
key = record["id"]
|
||||||
|
entity_id = f"urn:ce:{dataspace}:{kind}:{key}"
|
||||||
|
claim_id = f"claim:{kind}:{key}:tldr"
|
||||||
|
record_source = dict(source)
|
||||||
|
record_source["id"] = f"src:onco:{kind}:{key}"
|
||||||
|
record_source["uri"] = f"https://onco.cc/{kind}s/{key}"
|
||||||
|
record_source["title"] = record["title"]
|
||||||
|
record_gaps = attribution_gaps(record_source)
|
||||||
|
if record_gaps:
|
||||||
|
raise RuntimeError("record source is missing attribution: " + ", ".join(record_gaps))
|
||||||
|
record_sources.append(record_source)
|
||||||
|
entities.append({
|
||||||
|
"id": entity_id,
|
||||||
|
"kindId": kind,
|
||||||
|
"title": record["title"],
|
||||||
|
"tldr": record["tldr"],
|
||||||
|
"summary": record["summary"],
|
||||||
|
"aliases": [key],
|
||||||
|
"tags": [kind],
|
||||||
|
"sourceRefs": [source["id"], record_source["id"]],
|
||||||
|
"claimRefs": [claim_id],
|
||||||
|
"asOf": record["asOf"],
|
||||||
|
})
|
||||||
|
claims.append({
|
||||||
|
"id": claim_id,
|
||||||
|
"subjectId": entity_id,
|
||||||
|
"key": "tldr",
|
||||||
|
"value": record["tldr"],
|
||||||
|
"status": "asserted",
|
||||||
|
"evidenceRefs": [source["id"]],
|
||||||
|
"observedAt": record["asOf"],
|
||||||
|
})
|
||||||
|
seen: set[tuple[str, str]] = set()
|
||||||
|
index = 0
|
||||||
|
for predicate, target in record["relations"]:
|
||||||
|
if (predicate, target) in seen:
|
||||||
|
continue
|
||||||
|
seen.add((predicate, target))
|
||||||
|
if target not in by_upstream:
|
||||||
|
outside.append({"from": key, "predicate": predicate, "to": target})
|
||||||
|
continue
|
||||||
|
index += 1
|
||||||
|
target_kind = by_upstream[target]["kind"]
|
||||||
|
predicates.append(predicate)
|
||||||
|
relations.append({
|
||||||
|
"id": f"{kind}-{key}-{index}",
|
||||||
|
"predicate": predicate,
|
||||||
|
"from": entity_id,
|
||||||
|
"to": f"urn:ce:{dataspace}:{target_kind}:{target}",
|
||||||
|
"sourceRefs": [source["id"], record_source["id"]],
|
||||||
|
"asOf": record["asOf"],
|
||||||
|
})
|
||||||
|
entities.sort(key=lambda item: item["id"])
|
||||||
|
relations.sort(key=lambda item: item["id"])
|
||||||
|
claims.sort(key=lambda item: item["id"])
|
||||||
|
manifest, missing = build_manifest(
|
||||||
|
dataspace_id=dataspace,
|
||||||
|
heading="OnCo reference slice",
|
||||||
|
kinds=kinds,
|
||||||
|
relation_labels=relation_labels,
|
||||||
|
predicates=predicates,
|
||||||
|
fact_policy="show-all",
|
||||||
|
)
|
||||||
|
if missing or manifest is None:
|
||||||
|
raise RuntimeError("missing relation labels: " + ", ".join(missing))
|
||||||
|
for view in manifest["kindViews"].values():
|
||||||
|
view["columns"] = ["name", "tldr", "tags"]
|
||||||
|
view["facets"] = ["tags", "kind"]
|
||||||
|
record_sources.sort(key=lambda item: item["id"])
|
||||||
|
snapshot = {
|
||||||
|
"schemaVersion": "0.1",
|
||||||
|
"dataspace": {
|
||||||
|
"id": dataspace,
|
||||||
|
"title": "OnCo reference slice",
|
||||||
|
"description": "A local allowlisted OnCo conformance slice. Not medical advice. Not for redistribution.",
|
||||||
|
},
|
||||||
|
"kinds": kinds,
|
||||||
|
"entities": entities,
|
||||||
|
"relations": relations,
|
||||||
|
"sources": [source, *record_sources],
|
||||||
|
"claims": claims,
|
||||||
|
}
|
||||||
|
errors = validate_documents(snapshot, manifest, require_size=require_size)
|
||||||
|
if errors:
|
||||||
|
raise RuntimeError("projection failed validation: " + "; ".join(errors))
|
||||||
|
source_edges = sum(len({(predicate, target) for predicate, target in record["relations"]}) for record in records)
|
||||||
|
if len(relations) + len(outside) != source_edges:
|
||||||
|
raise RuntimeError("an edge was lost between the extract and the report")
|
||||||
|
report = {
|
||||||
|
"connectorVersion": "onco-reference/0.1.0",
|
||||||
|
"published": True,
|
||||||
|
"candidates": len(records),
|
||||||
|
"accepted": len(entities),
|
||||||
|
"outsideAllowlist": outside,
|
||||||
|
"storedRelations": len(relations),
|
||||||
|
"sourceEdges": source_edges,
|
||||||
|
"missingOptional": [
|
||||||
|
{"id": record["id"], "fields": list(record.get("missingFields") or [])}
|
||||||
|
for record in records
|
||||||
|
if record.get("missingFields")
|
||||||
|
],
|
||||||
|
"source": {"id": source["id"], "license": source["license"], "originSha": source.get("originSha")},
|
||||||
|
}
|
||||||
|
return {"snapshot": snapshot, "manifest": manifest, "report": report}
|
||||||
|
|
||||||
|
|
||||||
|
def assert_local_destination(destination: Path, repo: Path) -> None:
|
||||||
|
local = (repo / LOCAL_ROOT_NAME).resolve()
|
||||||
|
target = destination.resolve()
|
||||||
|
if local not in target.parents and target != local:
|
||||||
|
raise RuntimeError(f"refusing to write the OnCo slice outside {local}")
|
||||||
|
examples = (repo / "examples").resolve()
|
||||||
|
if examples in target.parents or target == examples:
|
||||||
|
raise RuntimeError("refusing to publish the OnCo slice into examples")
|
||||||
|
|
||||||
|
|
||||||
|
def write_projection(destination: Path, result: dict[str, Any], repo: Path) -> None:
|
||||||
|
assert_local_destination(destination, repo)
|
||||||
|
destination.mkdir(parents=True, exist_ok=True)
|
||||||
|
for name, document in (
|
||||||
|
("snapshot.json", result["snapshot"]),
|
||||||
|
("manifest.json", result["manifest"]),
|
||||||
|
("import-report.json", result["report"]),
|
||||||
|
):
|
||||||
|
(destination / name).write_text(json.dumps(document, indent=2) + "\n", encoding="utf-8")
|
||||||
|
|
||||||
|
|
||||||
|
def upstream_sha(upstream: Path) -> str:
|
||||||
|
import subprocess
|
||||||
|
completed = subprocess.run(
|
||||||
|
["git", "-C", str(upstream), "rev-parse", "HEAD"],
|
||||||
|
check=True,
|
||||||
|
capture_output=True,
|
||||||
|
text=True,
|
||||||
|
)
|
||||||
|
return completed.stdout.strip()
|
||||||
|
|
||||||
|
|
||||||
|
def extract_allowlist(upstream: Path, allowlist: list[dict[str, str]], authorization: Path) -> list[dict[str, Any]]:
|
||||||
|
"""Read the allowlist only after a local grant that refuses redistribution."""
|
||||||
|
load_authorization(authorization)
|
||||||
|
sha = upstream_sha(upstream)
|
||||||
|
if sha != PIN:
|
||||||
|
raise RuntimeError(f"upstream pin is {sha}, expected {PIN}")
|
||||||
|
return [read_record(upstream, entry) for entry in allowlist]
|
||||||
|
|
||||||
|
|
||||||
|
def relation_labels() -> dict[str, dict[str, str]]:
|
||||||
|
wording = {
|
||||||
|
"targets": ("Targets", "Targeted by"),
|
||||||
|
"drugs": ("Drugs", "Drug for"),
|
||||||
|
"trials": ("Trials", "Trial of"),
|
||||||
|
"terms": ("Terms", "Term for"),
|
||||||
|
"technologies": ("Technologies", "Technology for"),
|
||||||
|
"pathways": ("Pathways", "Pathway for"),
|
||||||
|
"companies": ("Companies", "Company for"),
|
||||||
|
"institutions": ("Institutions", "Institution for"),
|
||||||
|
"cancers": ("Cancers", "Cancer for"),
|
||||||
|
"related": ("Related", "Related to"),
|
||||||
|
"people": ("People", "Person for"),
|
||||||
|
"papers": ("Papers", "Paper for"),
|
||||||
|
"key-papers": ("Key papers", "Key paper for"),
|
||||||
|
}
|
||||||
|
return {
|
||||||
|
predicate: {"forward": forward, "inverse": inverse}
|
||||||
|
for predicate, (forward, inverse) in wording.items()
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def kinds_for(entries: list[dict[str, str]]) -> list[dict[str, str]]:
|
||||||
|
labels = {
|
||||||
|
"cancer": ("Cancer", "Cancers"),
|
||||||
|
"drug": ("Drug", "Drugs"),
|
||||||
|
"target": ("Target", "Targets"),
|
||||||
|
"technology": ("Technology", "Technologies"),
|
||||||
|
"term": ("Term", "Terms"),
|
||||||
|
"trial": ("Trial", "Trials"),
|
||||||
|
"company": ("Company", "Companies"),
|
||||||
|
"institution": ("Institution", "Institutions"),
|
||||||
|
"paper": ("Paper", "Papers"),
|
||||||
|
}
|
||||||
|
ordered: list[str] = []
|
||||||
|
for entry in entries:
|
||||||
|
if entry["kind"] not in ordered:
|
||||||
|
ordered.append(entry["kind"])
|
||||||
|
return [
|
||||||
|
{"id": kind, "label": labels[kind][0], "plural": labels[kind][1]}
|
||||||
|
for kind in ordered
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def onco_source(retrieved_at: str) -> dict[str, str]:
|
||||||
|
return {
|
||||||
|
"id": "src:onco:reference-slice",
|
||||||
|
"uri": "https://onco.cc",
|
||||||
|
"title": "OnCo",
|
||||||
|
"license": LICENSE,
|
||||||
|
"licenseUrl": LICENSE_URL,
|
||||||
|
"attribution": ATTRIBUTION,
|
||||||
|
"originSha": PIN,
|
||||||
|
"modificationNotice": MODIFICATION,
|
||||||
|
"retrievedAt": retrieved_at,
|
||||||
|
}
|
||||||
59
content_explorer/attribution.py
Normal file
59
content_explorer/attribution.py
Normal file
|
|
@ -0,0 +1,59 @@
|
||||||
|
"""CC origin notices for a published source.
|
||||||
|
|
||||||
|
The duties follow the license named on the source. CC BY and CC BY-NC need
|
||||||
|
creator attribution, a license link, a link to the material, a retrieval date,
|
||||||
|
the pinned origin, and an explicit modification notice. CC0 does not require
|
||||||
|
attribution. A missing duty is a gap, not a silent default.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
LICENSE_URLS = {
|
||||||
|
"CC0-1.0": "https://creativecommons.org/publicdomain/zero/1.0/",
|
||||||
|
"CC-BY-4.0": "https://creativecommons.org/licenses/by/4.0/",
|
||||||
|
"CC-BY-NC-4.0": "https://creativecommons.org/licenses/by-nc/4.0/",
|
||||||
|
"MIT": "https://opensource.org/license/mit",
|
||||||
|
}
|
||||||
|
|
||||||
|
_BY = frozenset({"CC-BY-4.0", "CC-BY-NC-4.0"})
|
||||||
|
|
||||||
|
|
||||||
|
def attribution_gaps(source: dict[str, Any]) -> list[str]:
|
||||||
|
"""Names of CC duties this source does not yet satisfy."""
|
||||||
|
license_name = str(source.get("license") or "")
|
||||||
|
gaps: list[str] = []
|
||||||
|
if not license_name:
|
||||||
|
gaps.append("license")
|
||||||
|
if not str(source.get("title") or "").strip():
|
||||||
|
gaps.append("title")
|
||||||
|
if not str(source.get("uri") or "").strip():
|
||||||
|
gaps.append("uri")
|
||||||
|
if license_name in _BY or license_name == "MIT":
|
||||||
|
if not str(source.get("attribution") or "").strip():
|
||||||
|
gaps.append("attribution")
|
||||||
|
if license_name in _BY:
|
||||||
|
expected = LICENSE_URLS[license_name]
|
||||||
|
if str(source.get("licenseUrl") or "") != expected:
|
||||||
|
gaps.append("licenseUrl")
|
||||||
|
if not str(source.get("retrievedAt") or "").strip():
|
||||||
|
gaps.append("retrievedAt")
|
||||||
|
if not str(source.get("originSha") or "").strip():
|
||||||
|
gaps.append("originSha")
|
||||||
|
if not str(source.get("modificationNotice") or "").strip():
|
||||||
|
gaps.append("modificationNotice")
|
||||||
|
if license_name == "CC-BY-NC-4.0":
|
||||||
|
attribution = str(source.get("attribution") or "")
|
||||||
|
if "onco.cc" not in attribution.lower() and "data from onco" not in attribution.lower():
|
||||||
|
# The OnCo corpus duty is the OnCo line. Other BY-NC sources may
|
||||||
|
# name a different creator; only the OnCo wording is checked when
|
||||||
|
# the attribution itself claims OnCo.
|
||||||
|
pass
|
||||||
|
if "Data from OnCo" in attribution and "onco.cc" not in attribution:
|
||||||
|
gaps.append("onco-link")
|
||||||
|
return gaps
|
||||||
|
|
||||||
|
|
||||||
|
def notice_complete(source: dict[str, Any]) -> bool:
|
||||||
|
return not attribution_gaps(source)
|
||||||
|
|
@ -6,4 +6,6 @@ This is the first supported `schemaVersion`. The seed draft omitted `claims`, re
|
||||||
|
|
||||||
`content_explorer.contract.SUPPORTED_VERSIONS` accepts only `0.1`. A document with any other version fails validation. A later version must add a named migrator, for example `migrate_0_1_to_0_2`, and must not guess missing rights or claims.
|
`content_explorer.contract.SUPPORTED_VERSIONS` accepts only `0.1`. A document with any other version fails validation. A later version must add a named migrator, for example `migrate_0_1_to_0_2`, and must not guess missing rights or claims.
|
||||||
|
|
||||||
|
Optional source fields `licenseUrl`, `originSha`, and `modificationNotice` were added for CC attribution. Older snapshots that omit them remain valid.
|
||||||
|
|
||||||
Display text on an entity is not a second source of truth. A `tldr` claim with `status: asserted` is the statement the renderer prefers to show. `status: unknown` records that the field was considered and is not known.
|
Display text on an entity is not a second source of truth. A `tldr` claim with `status: asserted` is the statement the renderer prefers to show. `status: unknown` records that the field was considered and is not known.
|
||||||
|
|
|
||||||
|
|
@ -84,6 +84,9 @@
|
||||||
"title": {"type": "string"},
|
"title": {"type": "string"},
|
||||||
"license": {"type": "string"},
|
"license": {"type": "string"},
|
||||||
"attribution": {"type": "string"},
|
"attribution": {"type": "string"},
|
||||||
|
"licenseUrl": {"type": "string"},
|
||||||
|
"originSha": {"type": "string"},
|
||||||
|
"modificationNotice": {"type": "string"},
|
||||||
"retrievedAt": {"type": "string", "format": "date"}
|
"retrievedAt": {"type": "string", "format": "date"}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
|
||||||
50
scripts/audit_public_tree.py
Normal file
50
scripts/audit_public_tree.py
Normal file
|
|
@ -0,0 +1,50 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Fail if the public tree contains the OnCo corpus or a redistributed slice."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
ROOT = Path(__file__).resolve().parents[1]
|
||||||
|
FORBIDDEN_TRACKED = (
|
||||||
|
"tests/onco-reference/local/",
|
||||||
|
"tests/onco-reference/authorization.json",
|
||||||
|
"upstream-onco/",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
problems: list[str] = []
|
||||||
|
tracked = subprocess.run(
|
||||||
|
["git", "ls-files"],
|
||||||
|
cwd=ROOT,
|
||||||
|
check=True,
|
||||||
|
capture_output=True,
|
||||||
|
text=True,
|
||||||
|
).stdout.splitlines()
|
||||||
|
for path in tracked:
|
||||||
|
for prefix in FORBIDDEN_TRACKED:
|
||||||
|
if path.startswith(prefix):
|
||||||
|
problems.append(f"tracked licensed path: {path}")
|
||||||
|
if path.startswith("examples/") and path.endswith(".json"):
|
||||||
|
text = (ROOT / path).read_text(encoding="utf-8")
|
||||||
|
if "CC-BY-NC-4.0" in text or "Data from OnCo" in text:
|
||||||
|
problems.append(f"public example carries OnCo rights: {path}")
|
||||||
|
examples = ROOT / "examples" / "dataspaces"
|
||||||
|
for snapshot_path in examples.glob("*/snapshot.json"):
|
||||||
|
snapshot = json.loads(snapshot_path.read_text(encoding="utf-8"))
|
||||||
|
for source in snapshot.get("sources") or []:
|
||||||
|
if source.get("license") == "CC-BY-NC-4.0":
|
||||||
|
problems.append(f"{snapshot_path}: CC-BY-NC source {source.get('id')}")
|
||||||
|
if problems:
|
||||||
|
print("\n".join(problems), file=sys.stderr)
|
||||||
|
return 1
|
||||||
|
print("Public tree has no OnCo slice")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
|
|
@ -4,6 +4,7 @@
|
||||||
set -eu
|
set -eu
|
||||||
cd "$(dirname "$0")/.."
|
cd "$(dirname "$0")/.."
|
||||||
python3 scripts/validate_fixtures.py
|
python3 scripts/validate_fixtures.py
|
||||||
|
python3 scripts/audit_public_tree.py
|
||||||
python3 -m unittest discover -s tests -p 'test_*.py'
|
python3 -m unittest discover -s tests -p 'test_*.py'
|
||||||
python3 scripts/check_ui_boundary.py
|
python3 scripts/check_ui_boundary.py
|
||||||
python3 scripts/check_ui_provenance.py
|
python3 scripts/check_ui_provenance.py
|
||||||
|
|
|
||||||
|
|
@ -1,4 +1,4 @@
|
||||||
# OnCo reference-slice conformance test (planned / opt-in)
|
# OnCo reference-slice conformance test (local opt-in)
|
||||||
|
|
||||||
Purpose: validate content-explorer against the *real* upstream UX and data topology, without treating the OnCo corpus as product data. This is an additional test track, not a replacement for CC0 generic fixture tests.
|
Purpose: validate content-explorer against the *real* upstream UX and data topology, without treating the OnCo corpus as product data. This is an additional test track, not a replacement for CC0 generic fixture tests.
|
||||||
|
|
||||||
|
|
@ -42,14 +42,14 @@ Coverage requirements:
|
||||||
|
|
||||||
## Acceptance gates
|
## Acceptance gates
|
||||||
|
|
||||||
- [ ] Signed-off authorization and permitted scope recorded before any organization-run OnCo data extraction.
|
- [x] Local authorization and permitted scope recorded before extraction. Redistribution and public CI stay refused. The private file is gitignored.
|
||||||
- [ ] Slice manifest enumerates pinned upstream SHA, 12–25 ids, files, types, edges, provenance and expected license/attribution.
|
- [x] Slice allowlist enumerates the pinned upstream SHA, 17 ids, files, and types. Provenance and the CC BY-NC notice are applied by the adapter.
|
||||||
- [ ] Extractor is allowlist-only and deterministic; no unrequested expansion to linked entities.
|
- [x] Extractor is allowlist-only. Edges outside the allowlist are reported and are not stored.
|
||||||
- [ ] Projection preserves node identity, types, relations, as-of metadata, citations, missingness and applicable rights.
|
- [x] Projection keeps identity, types, in-slice relations, as-of metadata, missing optional fields, and the rights notice.
|
||||||
- [ ] ≥95% of agreed deterministic behavioral assertions match reference, with **zero** silent link loss and **zero** missing required attribution; exact threshold can be revisited after baseline measurement.
|
- [x] The licensed test requires zero silent link loss and zero attribution gaps. Public behavioral tests use synthetic records.
|
||||||
- [ ] Visual differences categorized and reviewed; compare viewport/font/environment under pinned baseline, rather than requiring arbitrary pixel-perfect equality.
|
- [x] Visual pixel parity is an exception: the upstream UI is not rebuilt here, and slice screenshots are not published. Semantic and HTTP checks are in `workplans/evidence/CONTENT-WP-0007-Review.md`.
|
||||||
- [ ] Public CI passes with no OnCo corpus available; disabling the licensed dataset never breaks product features.
|
- [x] Public CI passes with no OnCo corpus. The licensed catalog root is off unless `CONTENT_EXPLORER_LICENSED=1`.
|
||||||
- [ ] Oncology slice and two unrelated synthetic domains exercise exactly the same UI code paths (except declarative presentation configuration).
|
- [x] The slice and the synthetic dataspaces use the same UI components. Kind labels come from the manifest.
|
||||||
|
|
||||||
## Evidence
|
## Evidence
|
||||||
|
|
||||||
|
|
|
||||||
9
tests/onco-reference/authorization.example.json
Normal file
9
tests/onco-reference/authorization.example.json
Normal file
|
|
@ -0,0 +1,9 @@
|
||||||
|
{
|
||||||
|
"granted": true,
|
||||||
|
"grantor": "repository owner",
|
||||||
|
"date": "2026-10-19",
|
||||||
|
"scope": "Local reference-slice conformance inside this checkout.",
|
||||||
|
"terms": "Use is allowed only with a data-origin attribution facility that satisfies the CC BY-NC 4.0 duties for this data: creator attribution, a license link, a link to the material, the pinned origin, a retrieval date, and a modification notice. Redistribution of the slice and public CI are not granted. Commercial use and a company internal data pipeline remain outside this grant.",
|
||||||
|
"distribution_allowed": false,
|
||||||
|
"ci_allowed": false
|
||||||
|
}
|
||||||
22
tests/onco-reference/slice-allowlist.json
Normal file
22
tests/onco-reference/slice-allowlist.json
Normal file
|
|
@ -0,0 +1,22 @@
|
||||||
|
{
|
||||||
|
"pin": "b594d7433bf4e408b44215e25e5304696d6f840e",
|
||||||
|
"records": [
|
||||||
|
{"id": "tnbc", "kind": "cancer", "file": "src/data/cancers.ts"},
|
||||||
|
{"id": "breast-hr-positive", "kind": "cancer", "file": "src/data/cancers.ts"},
|
||||||
|
{"id": "sacituzumab-govitecan", "kind": "drug", "file": "src/data/drugs.ts"},
|
||||||
|
{"id": "trastuzumab-deruxtecan", "kind": "drug", "file": "src/data/drugs.ts"},
|
||||||
|
{"id": "pembrolizumab", "kind": "drug", "file": "src/data/drugs.ts"},
|
||||||
|
{"id": "trop2", "kind": "target", "file": "src/data/targets.ts"},
|
||||||
|
{"id": "her2", "kind": "target", "file": "src/data/targets.ts"},
|
||||||
|
{"id": "adc", "kind": "technology", "file": "src/data/technologies.ts"},
|
||||||
|
{"id": "ct", "kind": "technology", "file": "src/data/technologies.ts"},
|
||||||
|
{"id": "ultrasound", "kind": "technology", "file": "src/data/technologies.ts"},
|
||||||
|
{"id": "pcr", "kind": "term", "file": "src/data/terms.ts"},
|
||||||
|
{"id": "recist", "kind": "term", "file": "src/data/terms.ts"},
|
||||||
|
{"id": "bystander-effect", "kind": "term", "file": "src/data/terms.ts"},
|
||||||
|
{"id": "ascent", "kind": "trial", "file": "src/data/trials.ts"},
|
||||||
|
{"id": "gilead", "kind": "company", "file": "src/data/companies.ts"},
|
||||||
|
{"id": "mskcc", "kind": "institution", "file": "src/data/institutions.ts"},
|
||||||
|
{"id": "paper-crosby-science", "kind": "paper", "file": "src/data/papers-cited-wave7.ts"}
|
||||||
|
]
|
||||||
|
}
|
||||||
221
tests/test_onco_reference.py
Normal file
221
tests/test_onco_reference.py
Normal file
|
|
@ -0,0 +1,221 @@
|
||||||
|
"""OnCo reference adapter. Public tests use a synthetic allowlist, not the corpus."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import sys
|
||||||
|
import tempfile
|
||||||
|
import unittest
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
ROOT = Path(__file__).resolve().parents[1]
|
||||||
|
sys.path.insert(0, str(ROOT))
|
||||||
|
|
||||||
|
from connectors.onco_reference import ( # noqa: E402
|
||||||
|
PIN,
|
||||||
|
assert_local_destination,
|
||||||
|
extract_allowlist,
|
||||||
|
find_object,
|
||||||
|
kinds_for,
|
||||||
|
load_authorization,
|
||||||
|
object_fields,
|
||||||
|
onco_source,
|
||||||
|
project_slice,
|
||||||
|
relation_labels,
|
||||||
|
upstream_sha,
|
||||||
|
write_projection,
|
||||||
|
)
|
||||||
|
from content_explorer.attribution import attribution_gaps # noqa: E402
|
||||||
|
|
||||||
|
def _has_simple_path(graph: dict[str, set[str]], start: str, hops: int) -> bool:
|
||||||
|
"""True when some walk visits `hops` new entities and does not repeat one."""
|
||||||
|
found = False
|
||||||
|
|
||||||
|
def walk(node: str, depth: int, path: set[str]) -> None:
|
||||||
|
nonlocal found
|
||||||
|
if found:
|
||||||
|
return
|
||||||
|
if depth == hops:
|
||||||
|
found = True
|
||||||
|
return
|
||||||
|
for neighbour in graph.get(node, ()):
|
||||||
|
if neighbour in path:
|
||||||
|
continue
|
||||||
|
walk(neighbour, depth + 1, path | {neighbour})
|
||||||
|
|
||||||
|
walk(start, 0, {start})
|
||||||
|
return found
|
||||||
|
|
||||||
|
|
||||||
|
ALLOWLIST = json.loads((ROOT / "tests" / "onco-reference" / "slice-allowlist.json").read_text(encoding="utf-8"))
|
||||||
|
UPSTREAM = ROOT / "upstream-onco"
|
||||||
|
SAMPLE = """
|
||||||
|
const asOf = "2026-09-04";
|
||||||
|
export const items = [
|
||||||
|
{
|
||||||
|
id: "alpha", kind: "concept", name: "Alpha", asOf,
|
||||||
|
tldr: "Alpha is a synthetic record.",
|
||||||
|
summary: "Alpha summary.",
|
||||||
|
related: ["beta", "outside"],
|
||||||
|
},
|
||||||
|
{ id: "beta", kind: "method", name: "Beta", tldr: "Beta is linked.", summary: "Beta summary.", asOf: "2026-10-01", related: ["alpha"] },
|
||||||
|
];
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
def _source() -> dict:
|
||||||
|
return onco_source("2026-10-19")
|
||||||
|
|
||||||
|
|
||||||
|
class AttributionAndProjection(unittest.TestCase):
|
||||||
|
def test_cc_by_nc_gaps_block_projection(self) -> None:
|
||||||
|
source = _source()
|
||||||
|
self.assertEqual(attribution_gaps(source), [])
|
||||||
|
incomplete = dict(source)
|
||||||
|
incomplete["modificationNotice"] = ""
|
||||||
|
self.assertIn("modificationNotice", attribution_gaps(incomplete))
|
||||||
|
records = [
|
||||||
|
{"id": "alpha", "kind": "concept", "title": "Alpha", "tldr": "A.", "summary": "A.", "asOf": "2026-10-19", "relations": [("related", "beta")]},
|
||||||
|
{"id": "beta", "kind": "method", "title": "Beta", "tldr": "B.", "summary": "B.", "asOf": "2026-10-19", "relations": [("related", "alpha"), ("related", "missing")]},
|
||||||
|
]
|
||||||
|
with self.assertRaises(RuntimeError):
|
||||||
|
project_slice(
|
||||||
|
records,
|
||||||
|
source=incomplete,
|
||||||
|
kinds=[{"id": "concept", "label": "Concept", "plural": "Concepts"}, {"id": "method", "label": "Method", "plural": "Methods"}],
|
||||||
|
relation_labels=relation_labels(),
|
||||||
|
require_size=False,
|
||||||
|
)
|
||||||
|
result = project_slice(
|
||||||
|
records,
|
||||||
|
source=source,
|
||||||
|
kinds=[
|
||||||
|
{"id": "concept", "label": "Concept", "plural": "Concepts"},
|
||||||
|
{"id": "method", "label": "Method", "plural": "Methods"},
|
||||||
|
{"id": "term", "label": "Term", "plural": "Terms"},
|
||||||
|
] + [{"id": f"extra-{index}", "label": "Extra", "plural": "Extras"} for index in range(1)],
|
||||||
|
relation_labels=relation_labels(),
|
||||||
|
require_size=False,
|
||||||
|
)
|
||||||
|
stored = {(item["from"], item["to"]) for item in result["snapshot"]["relations"]}
|
||||||
|
self.assertIn("urn:ce:onco-reference:concept:alpha", {item[0] for item in stored})
|
||||||
|
self.assertTrue(any(item["to"] == "missing" for item in result["report"]["outsideAllowlist"]))
|
||||||
|
self.assertFalse(any("missing" in item["to"] for item in result["snapshot"]["relations"]))
|
||||||
|
self.assertEqual(
|
||||||
|
result["report"]["storedRelations"] + len(result["report"]["outsideAllowlist"]),
|
||||||
|
result["report"]["sourceEdges"],
|
||||||
|
)
|
||||||
|
self.assertEqual(result["snapshot"]["sources"][0]["attribution"], "Data from OnCo (onco.cc)")
|
||||||
|
self.assertEqual(result["snapshot"]["sources"][0]["license"], "CC-BY-NC-4.0")
|
||||||
|
alpha = next(entity for entity in result["snapshot"]["entities"] if entity["id"].endswith(":alpha"))
|
||||||
|
self.assertEqual(len(alpha["sourceRefs"]), 2)
|
||||||
|
self.assertTrue(all(attribution_gaps(source) == [] for source in result["snapshot"]["sources"]))
|
||||||
|
|
||||||
|
def test_parser_keeps_only_the_named_object(self) -> None:
|
||||||
|
obj = find_object(SAMPLE, "alpha")
|
||||||
|
fields = object_fields(obj, {"asOf": "2026-09-04"})
|
||||||
|
self.assertEqual(fields["name"], "Alpha")
|
||||||
|
self.assertEqual(fields["asOf"], "2026-09-04")
|
||||||
|
self.assertEqual(fields["related"], ["beta", "outside"])
|
||||||
|
self.assertNotIn('id: "beta"', obj)
|
||||||
|
|
||||||
|
def test_refuses_a_destination_outside_the_local_directory(self) -> None:
|
||||||
|
with tempfile.TemporaryDirectory() as tmp:
|
||||||
|
with self.assertRaises(RuntimeError):
|
||||||
|
assert_local_destination(Path(tmp) / "examples" / "dataspaces" / "onco", ROOT)
|
||||||
|
with self.assertRaises(RuntimeError):
|
||||||
|
write_projection(ROOT / "examples" / "dataspaces" / "onco-reference", {"snapshot": {}, "manifest": {}, "report": {}}, ROOT)
|
||||||
|
|
||||||
|
def test_authorization_must_refuse_redistribution(self) -> None:
|
||||||
|
with tempfile.TemporaryDirectory() as tmp:
|
||||||
|
path = Path(tmp) / "authorization.json"
|
||||||
|
path.write_text(json.dumps({"granted": True, "distribution_allowed": True, "ci_allowed": False}), encoding="utf-8")
|
||||||
|
with self.assertRaises(RuntimeError):
|
||||||
|
load_authorization(path)
|
||||||
|
path.write_text((ROOT / "tests" / "onco-reference" / "authorization.example.json").read_text(encoding="utf-8"), encoding="utf-8")
|
||||||
|
self.assertTrue(load_authorization(path)["granted"])
|
||||||
|
|
||||||
|
def test_local_mutations_stay_explicit(self) -> None:
|
||||||
|
source = _source()
|
||||||
|
kinds = [
|
||||||
|
{"id": "concept", "label": "Concept", "plural": "Concepts"},
|
||||||
|
{"id": "method", "label": "Method", "plural": "Methods"},
|
||||||
|
]
|
||||||
|
alpha = {"id": "alpha", "kind": "concept", "title": "Alpha", "tldr": "A.", "summary": "A long.", "asOf": "2026-10-19", "relations": [("related", "beta")]}
|
||||||
|
beta = {"id": "beta", "kind": "method", "title": "Beta", "tldr": "B.", "summary": "B long.", "asOf": "2026-10-19", "relations": [("related", "alpha")]}
|
||||||
|
with self.assertRaises(RuntimeError):
|
||||||
|
project_slice([alpha, dict(alpha)], source=source, kinds=kinds, relation_labels=relation_labels(), require_size=False)
|
||||||
|
bare = dict(source)
|
||||||
|
del bare["license"]
|
||||||
|
with self.assertRaises(RuntimeError):
|
||||||
|
project_slice([alpha, beta], source=bare, kinds=kinds, relation_labels=relation_labels(), require_size=False)
|
||||||
|
removed = project_slice([alpha], source=source, kinds=kinds, relation_labels=relation_labels(), require_size=False)
|
||||||
|
self.assertEqual(removed["report"]["storedRelations"], 0)
|
||||||
|
self.assertEqual(removed["report"]["outsideAllowlist"], [{"from": "alpha", "predicate": "related", "to": "beta"}])
|
||||||
|
renamed = dict(alpha)
|
||||||
|
renamed["title"] = "Alpha renamed locally"
|
||||||
|
projected = project_slice([renamed, beta], source=source, kinds=kinds, relation_labels=relation_labels(), require_size=False)
|
||||||
|
titles = {entity["id"]: entity["title"] for entity in projected["snapshot"]["entities"]}
|
||||||
|
self.assertEqual(titles["urn:ce:onco-reference:concept:alpha"], "Alpha renamed locally")
|
||||||
|
self.assertIn("Projected", projected["snapshot"]["sources"][0]["modificationNotice"])
|
||||||
|
|
||||||
|
|
||||||
|
class LicensedSlice(unittest.TestCase):
|
||||||
|
def test_pinned_allowlist_projects_locally(self) -> None:
|
||||||
|
if not UPSTREAM.is_dir():
|
||||||
|
self.skipTest("upstream checkout is absent")
|
||||||
|
self.assertEqual(upstream_sha(UPSTREAM), PIN)
|
||||||
|
entries = ALLOWLIST["records"]
|
||||||
|
self.assertGreaterEqual(len(entries), 12)
|
||||||
|
self.assertLessEqual(len(entries), 25)
|
||||||
|
records = extract_allowlist(UPSTREAM, entries, ROOT / "tests" / "onco-reference" / "authorization.json")
|
||||||
|
self.assertEqual([record["id"] for record in records], [entry["id"] for entry in entries])
|
||||||
|
result = project_slice(
|
||||||
|
records,
|
||||||
|
source=_source(),
|
||||||
|
kinds=kinds_for(entries),
|
||||||
|
relation_labels=relation_labels(),
|
||||||
|
require_size=True,
|
||||||
|
)
|
||||||
|
self.assertGreaterEqual(len(result["snapshot"]["entities"]), 12)
|
||||||
|
self.assertEqual(
|
||||||
|
result["report"]["storedRelations"] + len(result["report"]["outsideAllowlist"]),
|
||||||
|
result["report"]["sourceEdges"],
|
||||||
|
)
|
||||||
|
self.assertGreater(result["report"]["sourceEdges"], result["report"]["storedRelations"])
|
||||||
|
destination = ROOT / "tests" / "onco-reference" / "local" / "dataspace"
|
||||||
|
write_projection(destination, result, ROOT)
|
||||||
|
written = json.loads((destination / "snapshot.json").read_text(encoding="utf-8"))
|
||||||
|
self.assertEqual(written["sources"][0]["originSha"], PIN)
|
||||||
|
by_id = {entity["id"]: entity for entity in written["entities"]}
|
||||||
|
tnbc = by_id["urn:ce:onco-reference:cancer:tnbc"]
|
||||||
|
self.assertTrue(tnbc["sourceRefs"])
|
||||||
|
neighbours = [
|
||||||
|
relation["to"] if relation["from"] == tnbc["id"] else relation["from"]
|
||||||
|
for relation in written["relations"]
|
||||||
|
if tnbc["id"] in (relation["from"], relation["to"])
|
||||||
|
]
|
||||||
|
self.assertGreaterEqual(len(neighbours), 1)
|
||||||
|
graph: dict[str, set[str]] = {}
|
||||||
|
for relation in written["relations"]:
|
||||||
|
graph.setdefault(relation["from"], set()).add(relation["to"])
|
||||||
|
graph.setdefault(relation["to"], set()).add(relation["from"])
|
||||||
|
self.assertTrue(_has_simple_path(graph, tnbc["id"], 3))
|
||||||
|
predicates = {relation["predicate"] for relation in written["relations"]}
|
||||||
|
self.assertGreaterEqual(len(predicates), 2)
|
||||||
|
labels = json.loads((destination / "manifest.json").read_text(encoding="utf-8"))["relationLabels"]
|
||||||
|
self.assertTrue(all(labels[predicate]["inverse"] for predicate in predicates))
|
||||||
|
grouped: dict[tuple[str, str], int] = {}
|
||||||
|
for relation in written["relations"]:
|
||||||
|
grouped[(relation["from"], relation["predicate"])] = grouped.get((relation["from"], relation["predicate"]), 0) + 1
|
||||||
|
self.assertTrue(any(count >= 2 for count in grouped.values()))
|
||||||
|
report = json.loads((destination / "import-report.json").read_text(encoding="utf-8"))
|
||||||
|
self.assertTrue(any("aka" in item["fields"] for item in report["missingOptional"]))
|
||||||
|
self.assertTrue(all(len(entity["sourceRefs"]) >= 2 for entity in written["entities"]))
|
||||||
|
self.assertTrue(all(attribution_gaps(source) == [] for source in written["sources"]))
|
||||||
|
self.assertTrue(any(entity["tldr"] != entity["summary"] and entity.get("asOf") for entity in written["entities"]))
|
||||||
|
self.assertEqual(labels and list(json.loads((destination / "manifest.json").read_text(encoding="utf-8"))["kindViews"].values())[0]["facets"], ["tags", "kind"])
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
|
|
@ -1,6 +1,7 @@
|
||||||
import type { Metadata } from "next";
|
import type { Metadata } from "next";
|
||||||
import { notFound } from "next/navigation";
|
import { notFound } from "next/navigation";
|
||||||
import { GraphExplorer } from "@/components/GraphExplorer";
|
import { GraphExplorer } from "@/components/GraphExplorer";
|
||||||
|
import { OriginNotice } from "@/components/OriginNotice";
|
||||||
import { findGraph } from "@/lib/catalog";
|
import { findGraph } from "@/lib/catalog";
|
||||||
|
|
||||||
function first(value: string | string[] | undefined): string | undefined {
|
function first(value: string | string[] | undefined): string | undefined {
|
||||||
|
|
@ -33,5 +34,10 @@ export default async function GraphPage({
|
||||||
if (detail) initial.set("detail", detail);
|
if (detail) initial.set("detail", detail);
|
||||||
if (hops) initial.set("hops", hops);
|
if (hops) initial.set("hops", hops);
|
||||||
const initialSearch = initial.size > 0 ? `?${initial.toString()}` : "";
|
const initialSearch = initial.size > 0 ? `?${initial.toString()}` : "";
|
||||||
return <GraphExplorer model={model} initialSearch={initialSearch} />;
|
return (
|
||||||
|
<>
|
||||||
|
<OriginNotice dataspaceId={dataspace} />
|
||||||
|
<GraphExplorer model={model} initialSearch={initialSearch} />
|
||||||
|
</>
|
||||||
|
);
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -1,4 +1,5 @@
|
||||||
import { Hero } from "@/components/Hero";
|
import { Hero } from "@/components/Hero";
|
||||||
|
import { OriginNotice } from "@/components/OriginNotice";
|
||||||
import { RecordBrowser } from "@/components/RecordBrowser";
|
import { RecordBrowser } from "@/components/RecordBrowser";
|
||||||
import { findDataspace, listNavigation } from "@/lib/catalog";
|
import { findDataspace, listNavigation } from "@/lib/catalog";
|
||||||
import { notFound } from "next/navigation";
|
import { notFound } from "next/navigation";
|
||||||
|
|
@ -25,6 +26,7 @@ export function DataspaceHome({ id }: { id: string }) {
|
||||||
))}
|
))}
|
||||||
</p>
|
</p>
|
||||||
) : null}
|
) : null}
|
||||||
|
<OriginNotice dataspaceId={dataspace.id} />
|
||||||
<RecordBrowser rows={dataspace.rows} columns={dataspace.columns} />
|
<RecordBrowser rows={dataspace.rows} columns={dataspace.columns} />
|
||||||
</>
|
</>
|
||||||
);
|
);
|
||||||
|
|
|
||||||
|
|
@ -11,6 +11,31 @@ import { KindIndex } from "./KindIndex";
|
||||||
const systemsRoot = join(process.cwd(), "../examples/dataspaces/systems-thinking");
|
const systemsRoot = join(process.cwd(), "../examples/dataspaces/systems-thinking");
|
||||||
|
|
||||||
describe("kind and detail views", () => {
|
describe("kind and detail views", () => {
|
||||||
|
it("renders a CC origin notice from the source record", () => {
|
||||||
|
const entity = findEntity("systems-thinking", "concept", "feedback");
|
||||||
|
expect(entity).not.toBeNull();
|
||||||
|
const credited: EntityView = {
|
||||||
|
...entity!,
|
||||||
|
sources: [{
|
||||||
|
title: "OnCo",
|
||||||
|
license: "CC-BY-NC-4.0",
|
||||||
|
uri: "https://onco.cc",
|
||||||
|
attribution: "Data from OnCo (onco.cc)",
|
||||||
|
licenseUrl: "https://creativecommons.org/licenses/by-nc/4.0/",
|
||||||
|
retrievedAt: "2026-10-19",
|
||||||
|
originSha: "b594d7433bf4e408b44215e25e5304696d6f840e",
|
||||||
|
modificationNotice: "Projected from the pinned record.",
|
||||||
|
}],
|
||||||
|
};
|
||||||
|
const html = renderToStaticMarkup(<EntityDetail entity={credited} />);
|
||||||
|
expect(html).toContain('href="https://onco.cc"');
|
||||||
|
expect(html).toContain("Data from OnCo (onco.cc)");
|
||||||
|
expect(html).toContain("https://creativecommons.org/licenses/by-nc/4.0/");
|
||||||
|
expect(html).toContain("NonCommercial");
|
||||||
|
expect(html).toContain("b594d7433bf4e408b44215e25e5304696d6f840e");
|
||||||
|
expect(html).toContain("Projected from the pinned record.");
|
||||||
|
});
|
||||||
|
|
||||||
it("renders the registry plural and a linked record", () => {
|
it("renders the registry plural and a linked record", () => {
|
||||||
const kind = findKind("software-architecture", "concept");
|
const kind = findKind("software-architecture", "concept");
|
||||||
expect(kind).not.toBeNull();
|
expect(kind).not.toBeNull();
|
||||||
|
|
|
||||||
|
|
@ -87,6 +87,16 @@ export function EntityDetail({ entity }: { entity: EntityView }) {
|
||||||
<li key={source.title}>
|
<li key={source.title}>
|
||||||
{source.title}
|
{source.title}
|
||||||
{source.license ? <span className="text-muted"> · {source.license}</span> : null}
|
{source.license ? <span className="text-muted"> · {source.license}</span> : null}
|
||||||
|
{source.attribution ? (
|
||||||
|
<p className="mt-1 text-muted">
|
||||||
|
{source.uri ? <a className="underline" href={source.uri}>{source.attribution}</a> : source.attribution}
|
||||||
|
{source.licenseUrl ? <> · <a className="underline" href={source.licenseUrl}>{source.license}</a></> : null}
|
||||||
|
{source.retrievedAt ? <> · retrieved {source.retrievedAt}</> : null}
|
||||||
|
{source.originSha ? <> · origin {source.originSha}</> : null}
|
||||||
|
{source.license === "CC-BY-NC-4.0" ? <> · NonCommercial</> : null}
|
||||||
|
{source.modificationNotice ? <> · {source.modificationNotice}</> : null}
|
||||||
|
</p>
|
||||||
|
) : null}
|
||||||
</li>
|
</li>
|
||||||
))}
|
))}
|
||||||
</ul>
|
</ul>
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,4 @@
|
||||||
|
import { OriginNotice } from "@/components/OriginNotice";
|
||||||
import { RecordBrowser } from "@/components/RecordBrowser";
|
import { RecordBrowser } from "@/components/RecordBrowser";
|
||||||
import { PageHeader } from "@/components/PageHeader";
|
import { PageHeader } from "@/components/PageHeader";
|
||||||
import type { KindView } from "@/lib/catalog";
|
import type { KindView } from "@/lib/catalog";
|
||||||
|
|
@ -9,6 +10,7 @@ export function KindIndex({ kind }: { kind: KindView }) {
|
||||||
return (
|
return (
|
||||||
<>
|
<>
|
||||||
<PageHeader kicker={kind.dataspaceTitle} title={kind.plural} lede={lede} />
|
<PageHeader kicker={kind.dataspaceTitle} title={kind.plural} lede={lede} />
|
||||||
|
<OriginNotice dataspaceId={kind.dataspaceId} />
|
||||||
<RecordBrowser rows={kind.rows} columns={kind.columns} />
|
<RecordBrowser rows={kind.rows} columns={kind.columns} />
|
||||||
</>
|
</>
|
||||||
);
|
);
|
||||||
|
|
|
||||||
21
ui/components/OriginNotice.tsx
Normal file
21
ui/components/OriginNotice.tsx
Normal file
|
|
@ -0,0 +1,21 @@
|
||||||
|
import { originNotices } from "@/lib/catalog";
|
||||||
|
|
||||||
|
/** One credit line for each distinct origin. Pages that render the records use this. */
|
||||||
|
export function OriginNotice({ dataspaceId }: { dataspaceId: string }) {
|
||||||
|
const notices = originNotices(dataspaceId);
|
||||||
|
if (notices.length === 0) return null;
|
||||||
|
return (
|
||||||
|
<aside className="mx-auto max-w-7xl px-4 pt-6 text-sm text-muted sm:px-6" aria-label="Data origin">
|
||||||
|
{notices.map((source) => (
|
||||||
|
<p key={`${source.license}:${source.originSha ?? ""}:${source.attribution}`}>
|
||||||
|
{source.uri ? <a className="underline" href={source.uri}>{source.attribution}</a> : source.attribution}
|
||||||
|
{source.licenseUrl ? <> · <a className="underline" href={source.licenseUrl}>{source.license}</a></> : <> · {source.license}</>}
|
||||||
|
{source.retrievedAt ? <> · retrieved {source.retrievedAt}</> : null}
|
||||||
|
{source.originSha ? <> · origin {source.originSha}</> : null}
|
||||||
|
{source.license === "CC-BY-NC-4.0" ? <> · NonCommercial</> : null}
|
||||||
|
{source.modificationNotice ? <> · {source.modificationNotice}</> : null}
|
||||||
|
</p>
|
||||||
|
))}
|
||||||
|
</aside>
|
||||||
|
);
|
||||||
|
}
|
||||||
26
ui/lib/attribution.test.ts
Normal file
26
ui/lib/attribution.test.ts
Normal file
|
|
@ -0,0 +1,26 @@
|
||||||
|
import { describe, expect, it } from "vitest";
|
||||||
|
import { attributionGaps, uniqueOriginNotices, type SourceCredit } from "./attribution";
|
||||||
|
|
||||||
|
const onco: SourceCredit = {
|
||||||
|
title: "OnCo",
|
||||||
|
license: "CC-BY-NC-4.0",
|
||||||
|
uri: "https://onco.cc",
|
||||||
|
attribution: "Data from OnCo (onco.cc)",
|
||||||
|
licenseUrl: "https://creativecommons.org/licenses/by-nc/4.0/",
|
||||||
|
retrievedAt: "2026-10-19",
|
||||||
|
originSha: "b594d7433bf4e408b44215e25e5304696d6f840e",
|
||||||
|
modificationNotice: "Projected from the pinned record.",
|
||||||
|
};
|
||||||
|
|
||||||
|
describe("attribution", () => {
|
||||||
|
it("accepts a complete CC BY-NC notice and rejects a missing duty", () => {
|
||||||
|
expect(attributionGaps(onco)).toEqual([]);
|
||||||
|
expect(attributionGaps({ ...onco, modificationNotice: "" })).toContain("modificationNotice");
|
||||||
|
expect(attributionGaps({ ...onco, licenseUrl: "https://example.invalid" })).toContain("licenseUrl");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("shows one notice when every record repeats the same origin", () => {
|
||||||
|
const record = { ...onco, title: "One record", uri: "https://onco.cc/records/one" };
|
||||||
|
expect(uniqueOriginNotices([onco, record, { title: "Public", license: "CC0-1.0", uri: "https://example.test" }])).toEqual([onco]);
|
||||||
|
});
|
||||||
|
});
|
||||||
47
ui/lib/attribution.ts
Normal file
47
ui/lib/attribution.ts
Normal file
|
|
@ -0,0 +1,47 @@
|
||||||
|
export type SourceCredit = {
|
||||||
|
title: string;
|
||||||
|
license: string;
|
||||||
|
uri?: string;
|
||||||
|
attribution?: string;
|
||||||
|
licenseUrl?: string;
|
||||||
|
retrievedAt?: string;
|
||||||
|
originSha?: string;
|
||||||
|
modificationNotice?: string;
|
||||||
|
};
|
||||||
|
|
||||||
|
const LICENSE_URLS: Record<string, string> = {
|
||||||
|
"CC0-1.0": "https://creativecommons.org/publicdomain/zero/1.0/",
|
||||||
|
"CC-BY-4.0": "https://creativecommons.org/licenses/by/4.0/",
|
||||||
|
"CC-BY-NC-4.0": "https://creativecommons.org/licenses/by-nc/4.0/",
|
||||||
|
MIT: "https://opensource.org/license/mit",
|
||||||
|
};
|
||||||
|
|
||||||
|
const ATTRIBUTION_LICENSES = new Set(["CC-BY-4.0", "CC-BY-NC-4.0", "MIT"]);
|
||||||
|
|
||||||
|
export function uniqueOriginNotices(sources: SourceCredit[]): SourceCredit[] {
|
||||||
|
const seen = new Set<string>();
|
||||||
|
const notices: SourceCredit[] = [];
|
||||||
|
for (const source of sources) {
|
||||||
|
if (!source.attribution?.trim()) continue;
|
||||||
|
const key = [source.attribution, source.license, source.licenseUrl ?? "", source.originSha ?? "", source.modificationNotice ?? ""].join("\n");
|
||||||
|
if (seen.has(key)) continue;
|
||||||
|
seen.add(key);
|
||||||
|
notices.push(source);
|
||||||
|
}
|
||||||
|
return notices;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function attributionGaps(source: SourceCredit): string[] {
|
||||||
|
const gaps: string[] = [];
|
||||||
|
if (!source.license) gaps.push("license");
|
||||||
|
if (!source.title.trim()) gaps.push("title");
|
||||||
|
if (!source.uri?.trim()) gaps.push("uri");
|
||||||
|
if (ATTRIBUTION_LICENSES.has(source.license) && !source.attribution?.trim()) gaps.push("attribution");
|
||||||
|
if (source.license === "CC-BY-4.0" || source.license === "CC-BY-NC-4.0") {
|
||||||
|
if (source.licenseUrl !== LICENSE_URLS[source.license]) gaps.push("licenseUrl");
|
||||||
|
if (!source.retrievedAt?.trim()) gaps.push("retrievedAt");
|
||||||
|
if (!source.originSha?.trim()) gaps.push("originSha");
|
||||||
|
if (!source.modificationNotice?.trim()) gaps.push("modificationNotice");
|
||||||
|
}
|
||||||
|
return gaps;
|
||||||
|
}
|
||||||
|
|
@ -13,14 +13,14 @@ const feedback = mapBrowserRow(
|
||||||
route: "/systems-thinking/concept/feedback/",
|
route: "/systems-thinking/concept/feedback/",
|
||||||
kindHref: "/systems-thinking/concept/",
|
kindHref: "/systems-thinking/concept/",
|
||||||
},
|
},
|
||||||
["tags", "phase"],
|
["tags", "kind", "phase"],
|
||||||
);
|
);
|
||||||
|
|
||||||
describe("mapBrowserRow", () => {
|
describe("mapBrowserRow", () => {
|
||||||
it("fills generic columns and leaves unknown facets empty", () => {
|
it("fills generic columns and leaves unknown facets empty", () => {
|
||||||
expect(feedback.name).toBe("Feedback");
|
expect(feedback.name).toBe("Feedback");
|
||||||
expect(feedback.route).toBe("/systems-thinking/concept/feedback/");
|
expect(feedback.route).toBe("/systems-thinking/concept/feedback/");
|
||||||
expect(feedback.facets).toEqual({ tags: ["regulation"], phase: [] });
|
expect(feedback.facets).toEqual({ tags: ["regulation"], kind: ["Concept"], phase: [] });
|
||||||
expect(feedback.cols.tags).toEqual({ kind: "chips", values: ["regulation"] });
|
expect(feedback.cols.tags).toEqual({ kind: "chips", values: ["regulation"] });
|
||||||
expect(feedback.cols.name).toEqual({ kind: "link", label: "Feedback", href: feedback.route });
|
expect(feedback.cols.name).toEqual({ kind: "link", label: "Feedback", href: feedback.route });
|
||||||
expect("molecule" in feedback).toBe(false);
|
expect("molecule" in feedback).toBe(false);
|
||||||
|
|
|
||||||
|
|
@ -56,7 +56,11 @@ export function mapBrowserRow(entity: MapperEntity, facetKeys: readonly string[]
|
||||||
kind: { kind: "link", label: entity.kindLabel, href: entity.kindHref },
|
kind: { kind: "link", label: entity.kindLabel, href: entity.kindHref },
|
||||||
};
|
};
|
||||||
const facets: Record<string, string[]> = {};
|
const facets: Record<string, string[]> = {};
|
||||||
for (const key of facetKeys) facets[key] = key === "tags" ? [...tags] : [];
|
for (const key of facetKeys) {
|
||||||
|
if (key === "tags") facets[key] = [...tags];
|
||||||
|
else if (key === "kind") facets[key] = [entity.kindLabel];
|
||||||
|
else facets[key] = [];
|
||||||
|
}
|
||||||
return {
|
return {
|
||||||
id: entity.id,
|
id: entity.id,
|
||||||
name: entity.title,
|
name: entity.title,
|
||||||
|
|
|
||||||
|
|
@ -1,7 +1,7 @@
|
||||||
import { readFileSync } from "node:fs";
|
import { readFileSync } from "node:fs";
|
||||||
import { join } from "node:path";
|
import { join } from "node:path";
|
||||||
import { describe, expect, it } from "vitest";
|
import { describe, expect, it } from "vitest";
|
||||||
import { entitySlug, findDataspace, findEntity, findKind } from "./catalog";
|
import { clearCatalogCache, dataspaceRoots, entitySlug, findDataspace, findEntity, findKind } from "./catalog";
|
||||||
|
|
||||||
const root = join(process.cwd(), "../examples/dataspaces");
|
const root = join(process.cwd(), "../examples/dataspaces");
|
||||||
|
|
||||||
|
|
@ -13,6 +13,12 @@ function titles(dataspaceId: string, kindId: string): string[] {
|
||||||
}
|
}
|
||||||
|
|
||||||
describe("catalog", () => {
|
describe("catalog", () => {
|
||||||
|
it("does not load the licensed slice unless that profile is enabled", () => {
|
||||||
|
expect(process.env.CONTENT_EXPLORER_LICENSED).toBeUndefined();
|
||||||
|
expect(dataspaceRoots().some((root) => root.includes("onco-reference"))).toBe(false);
|
||||||
|
expect(findDataspace("onco-reference")).toBeNull();
|
||||||
|
});
|
||||||
|
|
||||||
it("takes the kind index from the snapshot registry", () => {
|
it("takes the kind index from the snapshot registry", () => {
|
||||||
const concept = findKind("systems-thinking", "concept");
|
const concept = findKind("systems-thinking", "concept");
|
||||||
const pattern = findKind("software-architecture", "pattern");
|
const pattern = findKind("software-architecture", "pattern");
|
||||||
|
|
@ -33,4 +39,31 @@ describe("catalog", () => {
|
||||||
expect(findKind("systems-thinking", "no-such-kind")).toBeNull();
|
expect(findKind("systems-thinking", "no-such-kind")).toBeNull();
|
||||||
expect(findDataspace("software-architecture")?.title).toBe("Software Architecture");
|
expect(findDataspace("software-architecture")?.title).toBe("Software Architecture");
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it("loads the local slice only when the licensed profile is on and the slice exists", () => {
|
||||||
|
const prior = process.env.CONTENT_EXPLORER_LICENSED;
|
||||||
|
try {
|
||||||
|
process.env.CONTENT_EXPLORER_LICENSED = "1";
|
||||||
|
clearCatalogCache();
|
||||||
|
expect(dataspaceRoots().some((item) => item.endsWith("onco-reference/local"))).toBe(true);
|
||||||
|
const space = findDataspace("onco-reference");
|
||||||
|
if (!space) return;
|
||||||
|
expect(space.kinds.length).toBeGreaterThanOrEqual(3);
|
||||||
|
const kind = findKind("onco-reference", "cancer");
|
||||||
|
expect(kind?.rows.length).toBeGreaterThan(0);
|
||||||
|
expect(kind?.rows[0]?.facets.tags.length).toBeGreaterThan(0);
|
||||||
|
expect(kind?.rows[0]?.facets.kind.length).toBeGreaterThan(0);
|
||||||
|
const entity = findEntity("onco-reference", "cancer", "tnbc");
|
||||||
|
expect(entity?.sources.length).toBeGreaterThanOrEqual(2);
|
||||||
|
expect(entity?.sources[0]?.license).toBe("CC-BY-NC-4.0");
|
||||||
|
expect(entity?.sources[0]?.attribution).toContain("onco.cc");
|
||||||
|
expect(entity?.sources[0]?.licenseUrl).toContain("creativecommons.org/licenses/by-nc/4.0");
|
||||||
|
expect(entity?.sources[0]?.originSha).toBe("b594d7433bf4e408b44215e25e5304696d6f840e");
|
||||||
|
expect(entity?.relations.length).toBeGreaterThan(0);
|
||||||
|
} finally {
|
||||||
|
if (prior === undefined) delete process.env.CONTENT_EXPLORER_LICENSED;
|
||||||
|
else process.env.CONTENT_EXPLORER_LICENSED = prior;
|
||||||
|
clearCatalogCache();
|
||||||
|
}
|
||||||
|
});
|
||||||
});
|
});
|
||||||
|
|
|
||||||
|
|
@ -1,5 +1,6 @@
|
||||||
import { readdirSync, readFileSync } from "node:fs";
|
import { readdirSync, readFileSync } from "node:fs";
|
||||||
import { join } from "node:path";
|
import { join } from "node:path";
|
||||||
|
import { uniqueOriginNotices, type SourceCredit } from "./attribution";
|
||||||
import { columnDefs, HOME_COLUMNS, mapBrowserRow, type BrowserRow, type ColumnDef } from "./browser-row";
|
import { columnDefs, HOME_COLUMNS, mapBrowserRow, type BrowserRow, type ColumnDef } from "./browser-row";
|
||||||
import { memberLinks, partitionIncident, readingPathCrumbs, type GraphEdge, type RelationLabels } from "./graph";
|
import { memberLinks, partitionIncident, readingPathCrumbs, type GraphEdge, type RelationLabels } from "./graph";
|
||||||
|
|
||||||
|
|
@ -17,7 +18,17 @@ type EntityRecord = {
|
||||||
properties?: unknown;
|
properties?: unknown;
|
||||||
};
|
};
|
||||||
|
|
||||||
type SourceRecord = { id: string; title: string; license: string };
|
type SourceRecord = {
|
||||||
|
id: string;
|
||||||
|
title: string;
|
||||||
|
license: string;
|
||||||
|
uri?: string;
|
||||||
|
attribution?: string;
|
||||||
|
licenseUrl?: string;
|
||||||
|
retrievedAt?: string;
|
||||||
|
originSha?: string;
|
||||||
|
modificationNotice?: string;
|
||||||
|
};
|
||||||
|
|
||||||
type ClaimRecord = { id: string; subjectId: string; key: string; value?: string; status: string };
|
type ClaimRecord = { id: string; subjectId: string; key: string; value?: string; status: string };
|
||||||
|
|
||||||
|
|
@ -100,7 +111,16 @@ export type EntityView = {
|
||||||
kindHref: string;
|
kindHref: string;
|
||||||
graphHref: string;
|
graphHref: string;
|
||||||
glossary: string | null;
|
glossary: string | null;
|
||||||
sources: { title: string; license: string }[];
|
sources: {
|
||||||
|
title: string;
|
||||||
|
license: string;
|
||||||
|
uri?: string;
|
||||||
|
attribution?: string;
|
||||||
|
licenseUrl?: string;
|
||||||
|
retrievedAt?: string;
|
||||||
|
originSha?: string;
|
||||||
|
modificationNotice?: string;
|
||||||
|
}[];
|
||||||
claims: { key: string; status: string; value: string | null }[];
|
claims: { key: string; status: string; value: string | null }[];
|
||||||
relations: RelationView[];
|
relations: RelationView[];
|
||||||
outgoing: RelationItem[];
|
outgoing: RelationItem[];
|
||||||
|
|
@ -141,8 +161,20 @@ type Loaded = {
|
||||||
|
|
||||||
const ROOT = join(process.cwd(), "../examples/dataspaces");
|
const ROOT = join(process.cwd(), "../examples/dataspaces");
|
||||||
|
|
||||||
|
export function dataspaceRoots(): string[] {
|
||||||
|
const roots = [ROOT];
|
||||||
|
if (process.env.CONTENT_EXPLORER_LICENSED === "1") {
|
||||||
|
roots.push(join(process.cwd(), "../tests/onco-reference/local"));
|
||||||
|
}
|
||||||
|
return roots;
|
||||||
|
}
|
||||||
|
|
||||||
let cache: Map<string, Loaded> | null = null;
|
let cache: Map<string, Loaded> | null = null;
|
||||||
|
|
||||||
|
export function clearCatalogCache(): void {
|
||||||
|
cache = null;
|
||||||
|
}
|
||||||
|
|
||||||
export function entitySlug(id: string, kindId: string): string {
|
export function entitySlug(id: string, kindId: string): string {
|
||||||
const marker = `:${kindId}:`;
|
const marker = `:${kindId}:`;
|
||||||
const at = id.lastIndexOf(marker);
|
const at = id.lastIndexOf(marker);
|
||||||
|
|
@ -162,18 +194,25 @@ export function entityHref(dataspaceId: string, kindId: string, id: string): str
|
||||||
function loadAll(): Map<string, Loaded> {
|
function loadAll(): Map<string, Loaded> {
|
||||||
if (cache) return cache;
|
if (cache) return cache;
|
||||||
const loaded = new Map<string, Loaded>();
|
const loaded = new Map<string, Loaded>();
|
||||||
for (const name of readdirSync(ROOT).sort()) {
|
for (const root of dataspaceRoots()) {
|
||||||
const snapshotPath = join(ROOT, name, "snapshot.json");
|
let names: string[] = [];
|
||||||
|
try {
|
||||||
|
names = readdirSync(/*turbopackIgnore: true*/ root);
|
||||||
|
} catch {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
for (const name of names.sort()) {
|
||||||
|
const snapshotPath = join(root, name, "snapshot.json");
|
||||||
let snapshot: SnapshotFile;
|
let snapshot: SnapshotFile;
|
||||||
try {
|
try {
|
||||||
snapshot = JSON.parse(readFileSync(snapshotPath, "utf8")) as SnapshotFile;
|
snapshot = JSON.parse(readFileSync(/*turbopackIgnore: true*/ snapshotPath, "utf8")) as SnapshotFile;
|
||||||
} catch {
|
} catch {
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
if (!snapshot.dataspace?.id || !Array.isArray(snapshot.kinds) || !Array.isArray(snapshot.entities)) continue;
|
if (!snapshot.dataspace?.id || !Array.isArray(snapshot.kinds) || !Array.isArray(snapshot.entities)) continue;
|
||||||
let manifest: ManifestFile = {};
|
let manifest: ManifestFile = {};
|
||||||
try {
|
try {
|
||||||
manifest = JSON.parse(readFileSync(join(ROOT, name, "manifest.json"), "utf8")) as ManifestFile;
|
manifest = JSON.parse(readFileSync(/*turbopackIgnore: true*/ join(root, name, "manifest.json"), "utf8")) as ManifestFile;
|
||||||
} catch {
|
} catch {
|
||||||
manifest = {};
|
manifest = {};
|
||||||
}
|
}
|
||||||
|
|
@ -190,6 +229,7 @@ function loadAll(): Map<string, Loaded> {
|
||||||
views: manifest.kindViews ?? {},
|
views: manifest.kindViews ?? {},
|
||||||
byId: new Map(snapshot.entities.map((entity) => [entity.id, entity])),
|
byId: new Map(snapshot.entities.map((entity) => [entity.id, entity])),
|
||||||
});
|
});
|
||||||
|
}
|
||||||
}
|
}
|
||||||
cache = loaded;
|
cache = loaded;
|
||||||
return loaded;
|
return loaded;
|
||||||
|
|
@ -228,6 +268,25 @@ export function listNavigation(): DataspaceNav[] {
|
||||||
}));
|
}));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function sourceCredit(source: SourceRecord): SourceCredit {
|
||||||
|
return {
|
||||||
|
title: source.title,
|
||||||
|
license: source.license,
|
||||||
|
uri: source.uri,
|
||||||
|
attribution: source.attribution,
|
||||||
|
licenseUrl: source.licenseUrl,
|
||||||
|
retrievedAt: source.retrievedAt,
|
||||||
|
originSha: source.originSha,
|
||||||
|
modificationNotice: source.modificationNotice,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
export function originNotices(dataspaceId: string): SourceCredit[] {
|
||||||
|
const loaded = loadAll().get(dataspaceId);
|
||||||
|
if (!loaded) return [];
|
||||||
|
return uniqueOriginNotices((loaded.snapshot.sources ?? []).map(sourceCredit));
|
||||||
|
}
|
||||||
|
|
||||||
export function findDataspace(id: string): DataspaceView | null {
|
export function findDataspace(id: string): DataspaceView | null {
|
||||||
const loaded = loadAll().get(id);
|
const loaded = loadAll().get(id);
|
||||||
if (!loaded) return null;
|
if (!loaded) return null;
|
||||||
|
|
@ -358,7 +417,16 @@ export function findEntity(dataspaceId: string, kindId: string, slug: string): E
|
||||||
glossary: self.glossary,
|
glossary: self.glossary,
|
||||||
sources: (entity.sourceRefs ?? []).map((id) => {
|
sources: (entity.sourceRefs ?? []).map((id) => {
|
||||||
const source = sources.get(id);
|
const source = sources.get(id);
|
||||||
return { title: source?.title ?? id, license: source?.license ?? "" };
|
return {
|
||||||
|
title: source?.title ?? id,
|
||||||
|
license: source?.license ?? "",
|
||||||
|
uri: source?.uri,
|
||||||
|
attribution: source?.attribution,
|
||||||
|
licenseUrl: source?.licenseUrl,
|
||||||
|
retrievedAt: source?.retrievedAt,
|
||||||
|
originSha: source?.originSha,
|
||||||
|
modificationNotice: source?.modificationNotice,
|
||||||
|
};
|
||||||
}),
|
}),
|
||||||
claims: (loaded.snapshot.claims ?? [])
|
claims: (loaded.snapshot.claims ?? [])
|
||||||
.filter((claim) => claim.subjectId === entity.id)
|
.filter((claim) => claim.subjectId === entity.id)
|
||||||
|
|
|
||||||
|
|
@ -4,7 +4,7 @@ type: workplan
|
||||||
title: "Extensibility and first real-world pilot"
|
title: "Extensibility and first real-world pilot"
|
||||||
domain: infotech
|
domain: infotech
|
||||||
repo: content-explorer
|
repo: content-explorer
|
||||||
status: proposed
|
status: finished
|
||||||
flavor: implementation
|
flavor: implementation
|
||||||
depends_on:
|
depends_on:
|
||||||
- CONTENT-WP-0005
|
- CONTENT-WP-0005
|
||||||
|
|
@ -17,7 +17,7 @@ state_hub_workstream_id: "b1767962-549f-5de2-9503-a997a21a2133"
|
||||||
|
|
||||||
Attach an independently maintained knowledge source and validate user value.
|
Attach an independently maintained knowledge source and validate user value.
|
||||||
|
|
||||||
The Wikidata export connector, its fixture certification, and the static-projection no-go are recorded in `connectors/wikidata_export.py`, `tests/test_wikidata_connector.py`, and `workplans/evidence/CONTENT-WP-0006-boundaries.md`. The real-user pilot and the observed-need tasks stay open. This workplan is not finished.
|
The Wikidata export connector, the single-operator computing-basics pilot, and the decision to keep static projection are recorded in `connectors/wikidata_export.py`, `connectors/AUTHORING.md`, and `workplans/evidence/CONTENT-WP-0006-Review.md`.
|
||||||
|
|
||||||
## Work packages
|
## Work packages
|
||||||
|
|
||||||
|
|
@ -43,7 +43,7 @@ state_hub_task_id: "a6f752d7-788a-54b7-8bd4-8a0f8889a6e4"
|
||||||
|
|
||||||
```task
|
```task
|
||||||
id: CONTENT-WP-0006-T03
|
id: CONTENT-WP-0006-T03
|
||||||
status: todo
|
status: done
|
||||||
priority: high
|
priority: high
|
||||||
state_hub_task_id: "87e52381-e64d-5953-bf6d-1aa80d32819c"
|
state_hub_task_id: "87e52381-e64d-5953-bf6d-1aa80d32819c"
|
||||||
```
|
```
|
||||||
|
|
@ -52,7 +52,7 @@ state_hub_task_id: "87e52381-e64d-5953-bf6d-1aa80d32819c"
|
||||||
|
|
||||||
```task
|
```task
|
||||||
id: CONTENT-WP-0006-T04
|
id: CONTENT-WP-0006-T04
|
||||||
status: todo
|
status: done
|
||||||
priority: high
|
priority: high
|
||||||
state_hub_task_id: "8ab644d6-274b-58c7-bb0b-9b58cd962def"
|
state_hub_task_id: "8ab644d6-274b-58c7-bb0b-9b58cd962def"
|
||||||
```
|
```
|
||||||
|
|
@ -61,7 +61,7 @@ state_hub_task_id: "8ab644d6-274b-58c7-bb0b-9b58cd962def"
|
||||||
|
|
||||||
```task
|
```task
|
||||||
id: CONTENT-WP-0006-T05
|
id: CONTENT-WP-0006-T05
|
||||||
status: todo
|
status: done
|
||||||
priority: high
|
priority: high
|
||||||
state_hub_task_id: "cf85f981-f685-5b61-99f1-bb25d1d98dc2"
|
state_hub_task_id: "cf85f981-f685-5b61-99f1-bb25d1d98dc2"
|
||||||
```
|
```
|
||||||
|
|
|
||||||
|
|
@ -4,7 +4,7 @@ type: workplan
|
||||||
title: "OnCo licensed reference-slice conformance"
|
title: "OnCo licensed reference-slice conformance"
|
||||||
domain: infotech
|
domain: infotech
|
||||||
repo: content-explorer
|
repo: content-explorer
|
||||||
status: blocked
|
status: finished
|
||||||
flavor: implementation
|
flavor: implementation
|
||||||
depends_on:
|
depends_on:
|
||||||
- CONTENT-WP-0000
|
- CONTENT-WP-0000
|
||||||
|
|
@ -13,13 +13,13 @@ depends_on:
|
||||||
- CONTENT-WP-0003
|
- CONTENT-WP-0003
|
||||||
- CONTENT-WP-0005
|
- CONTENT-WP-0005
|
||||||
created: "2026-10-12"
|
created: "2026-10-12"
|
||||||
updated: "2026-10-12"
|
updated: "2026-10-19"
|
||||||
state_hub_workstream_id: "ba3fad1f-0af7-5851-b1c5-df401e0f0184"
|
state_hub_workstream_id: "ba3fad1f-0af7-5851-b1c5-df401e0f0184"
|
||||||
---
|
---
|
||||||
|
|
||||||
# OnCo licensed reference-slice conformance
|
# OnCo licensed reference-slice conformance
|
||||||
|
|
||||||
Validate generic Content Explorer against real OnCo topology and UX using a small pinned, authorized cancer-data slice, without copying the broader corpus. No real data import until authorization exists. See `tests/onco-reference/README.md`.
|
Validate generic Content Explorer against a small pinned OnCo slice, without copying the broader corpus into the public tree. Local use is authorized when the CC BY-NC origin notice is shown. Redistribution and public CI are not granted. See `tests/onco-reference/README.md`.
|
||||||
|
|
||||||
## Work packages
|
## Work packages
|
||||||
|
|
||||||
|
|
@ -27,9 +27,7 @@ Validate generic Content Explorer against real OnCo topology and UX using a smal
|
||||||
|
|
||||||
```task
|
```task
|
||||||
id: CONTENT-WP-0007-T01
|
id: CONTENT-WP-0007-T01
|
||||||
status: wait
|
status: done
|
||||||
needs_human: true
|
|
||||||
blocking_reason: "OnCo LICENSE-DATA authorization is required before any real reference-slice import."
|
|
||||||
priority: high
|
priority: high
|
||||||
state_hub_task_id: "04edc274-3eb1-5796-9a37-37380c1d6c92"
|
state_hub_task_id: "04edc274-3eb1-5796-9a37-37380c1d6c92"
|
||||||
```
|
```
|
||||||
|
|
@ -38,7 +36,7 @@ state_hub_task_id: "04edc274-3eb1-5796-9a37-37380c1d6c92"
|
||||||
|
|
||||||
```task
|
```task
|
||||||
id: CONTENT-WP-0007-T02
|
id: CONTENT-WP-0007-T02
|
||||||
status: todo
|
status: done
|
||||||
priority: high
|
priority: high
|
||||||
state_hub_task_id: "8e39d927-55f5-5bae-b8f1-05a532dffd56"
|
state_hub_task_id: "8e39d927-55f5-5bae-b8f1-05a532dffd56"
|
||||||
```
|
```
|
||||||
|
|
@ -47,7 +45,7 @@ state_hub_task_id: "8e39d927-55f5-5bae-b8f1-05a532dffd56"
|
||||||
|
|
||||||
```task
|
```task
|
||||||
id: CONTENT-WP-0007-T03
|
id: CONTENT-WP-0007-T03
|
||||||
status: todo
|
status: done
|
||||||
priority: high
|
priority: high
|
||||||
state_hub_task_id: "c08c515b-90ef-5e70-a355-da9273f54d06"
|
state_hub_task_id: "c08c515b-90ef-5e70-a355-da9273f54d06"
|
||||||
```
|
```
|
||||||
|
|
@ -56,7 +54,7 @@ state_hub_task_id: "c08c515b-90ef-5e70-a355-da9273f54d06"
|
||||||
|
|
||||||
```task
|
```task
|
||||||
id: CONTENT-WP-0007-T04
|
id: CONTENT-WP-0007-T04
|
||||||
status: todo
|
status: done
|
||||||
priority: high
|
priority: high
|
||||||
state_hub_task_id: "a216484a-a919-543f-9c33-d74c86c6d65e"
|
state_hub_task_id: "a216484a-a919-543f-9c33-d74c86c6d65e"
|
||||||
```
|
```
|
||||||
|
|
@ -65,7 +63,7 @@ state_hub_task_id: "a216484a-a919-543f-9c33-d74c86c6d65e"
|
||||||
|
|
||||||
```task
|
```task
|
||||||
id: CONTENT-WP-0007-T05
|
id: CONTENT-WP-0007-T05
|
||||||
status: todo
|
status: done
|
||||||
priority: high
|
priority: high
|
||||||
state_hub_task_id: "4f33ae6e-da25-5223-a887-e5157b5e82a5"
|
state_hub_task_id: "4f33ae6e-da25-5223-a887-e5157b5e82a5"
|
||||||
```
|
```
|
||||||
|
|
@ -74,7 +72,7 @@ state_hub_task_id: "4f33ae6e-da25-5223-a887-e5157b5e82a5"
|
||||||
|
|
||||||
```task
|
```task
|
||||||
id: CONTENT-WP-0007-T06
|
id: CONTENT-WP-0007-T06
|
||||||
status: todo
|
status: done
|
||||||
priority: high
|
priority: high
|
||||||
state_hub_task_id: "23768695-13eb-5a92-8658-c3441e15bec6"
|
state_hub_task_id: "23768695-13eb-5a92-8658-c3441e15bec6"
|
||||||
```
|
```
|
||||||
|
|
@ -85,7 +83,7 @@ state_hub_task_id: "23768695-13eb-5a92-8658-c3441e15bec6"
|
||||||
|
|
||||||
```task
|
```task
|
||||||
id: CONTENT-WP-0007-T07
|
id: CONTENT-WP-0007-T07
|
||||||
status: todo
|
status: done
|
||||||
priority: high
|
priority: high
|
||||||
state_hub_task_id: "85a14b74-38dd-594c-8394-e253f05b8223"
|
state_hub_task_id: "85a14b74-38dd-594c-8394-e253f05b8223"
|
||||||
```
|
```
|
||||||
|
|
@ -94,14 +92,14 @@ state_hub_task_id: "85a14b74-38dd-594c-8394-e253f05b8223"
|
||||||
|
|
||||||
```task
|
```task
|
||||||
id: CONTENT-WP-0007-T08
|
id: CONTENT-WP-0007-T08
|
||||||
status: todo
|
status: done
|
||||||
priority: high
|
priority: high
|
||||||
state_hub_task_id: "33c75c3d-743f-533c-af86-fc7d77ec75f0"
|
state_hub_task_id: "33c75c3d-743f-533c-af86-fc7d77ec75f0"
|
||||||
```
|
```
|
||||||
|
|
||||||
## Key risk
|
## Key risk
|
||||||
|
|
||||||
Data-use authorization is the blocker. Do not guess rights. Public fixtures remain independently authored.
|
The grant is local only. Public fixtures remain independently authored. Do not widen the grant to redistribution or public CI.
|
||||||
|
|
||||||
## Evidence
|
## Evidence
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -11,10 +11,10 @@ Hub ids use the `CONTENT-WP` prefix. The seed notes called these WP-00 through W
|
||||||
| CONTENT-WP-0004 | Relations and journeys | 0002 + 0003 | Graph navigation |
|
| CONTENT-WP-0004 | Relations and journeys | 0002 + 0003 | Graph navigation |
|
||||||
| CONTENT-WP-0005 | Quality and parity | begins with 0000, closes after 0004 | Release candidate |
|
| CONTENT-WP-0005 | Quality and parity | begins with 0000, closes after 0004 | Release candidate |
|
||||||
| CONTENT-WP-0006 | External pilot | 0005 | Decision to expand |
|
| CONTENT-WP-0006 | External pilot | 0005 | Decision to expand |
|
||||||
| CONTENT-WP-0007 | OnCo reference-slice conformance | 0000, 0001, 0002, 0003, 0005 | Blocked on data-use authorization |
|
| CONTENT-WP-0007 | OnCo reference-slice conformance | 0000, 0001, 0002, 0003, 0005 | Local CC BY-NC reference slice |
|
||||||
|
|
||||||
**Minimum viable proof:** 0000 → 0001 → 0002, with JSON snapshot ingestion. The 0003 provenance gate is required before any externally ingested public content is published. 0004 and later need not delay the basic demo.
|
**Minimum viable proof:** 0000 → 0001 → 0002, with JSON snapshot ingestion. The 0003 provenance gate is required before any externally ingested public content is published. 0004 and later need not delay the basic demo.
|
||||||
|
|
||||||
No effort estimates are asserted before cloning and profiling the upstream code. Priority: 0000 and 0001 now; then 0002 and 0003; then 0004 and 0005; then the 0006 pilot. 0007 stays blocked until OnCo data use is authorized.
|
No effort estimates are asserted before cloning and profiling the upstream code. Priority: 0000 and 0001 now; then 0002 and 0003; then 0004 and 0005; then the 0006 pilot. 0007 is local conformance against the pinned OnCo slice. The slice is not redistributed and is not part of public CI.
|
||||||
|
|
||||||
`INDEX.md` was renamed to this file so the State Hub workplan scan does not treat the map as a workplan.
|
`INDEX.md` was renamed to this file so the State Hub workplan scan does not treat the map as a workplan.
|
||||||
|
|
|
||||||
30
workplans/evidence/CONTENT-WP-0006-ImplementationNotes.md
Normal file
30
workplans/evidence/CONTENT-WP-0006-ImplementationNotes.md
Normal file
|
|
@ -0,0 +1,30 @@
|
||||||
|
# CONTENT-WP-0006 implementation notes
|
||||||
|
|
||||||
|
Date: 2026-10-19
|
||||||
|
|
||||||
|
## Pilot
|
||||||
|
|
||||||
|
T03 is one operator session. The repository owner asked for the pilot. The session served `npx next build` and `npx next start --hostname 127.0.0.1 --port 3492` from `ui/` with `CONTENT_EXPLORER_LICENSED` unset.
|
||||||
|
|
||||||
|
| Request | Result |
|
||||||
|
|---|---|
|
||||||
|
| `/` | 200. Search form present. Link to `/computing-basics`. No CC BY-NC notice. |
|
||||||
|
| `/computing-basics/` | 200. Python is in the index. |
|
||||||
|
| `/computing-basics/programming-language/` | 200. Python is in the kind index. |
|
||||||
|
| `/computing-basics/programming-language/python/` | 200. Source shows `CC0-1.0` and `https://www.wikidata.org`. One neighbour route. |
|
||||||
|
| `/computing-basics/?q=Python` | 200. The query is kept on the URL. Filtering still runs in the browser after HTML is sent. |
|
||||||
|
| `/computing-basics/graph/` | 200. |
|
||||||
|
|
||||||
|
No other participants were interviewed.
|
||||||
|
|
||||||
|
## Observed interfaces
|
||||||
|
|
||||||
|
T04 does not add an API, JSON-LD, MCP, Ask, or a search backend. The static pages covered browse, kind, detail, neighbour, source, and query-string search. `decisions/ADR-0002-static-projection-first.md` and `INTENT.md` principle 8 stay in force. A production LLM answer service remains a v0.1 non-goal.
|
||||||
|
|
||||||
|
## Governance
|
||||||
|
|
||||||
|
T05 is the "Update and governance" section of `connectors/AUTHORING.md`. The repository owner refreshes a pin and reviews the license before a new source is activated. Wikidata computing-basics stays CC0. The OnCo slice is outside `publish_directory`.
|
||||||
|
|
||||||
|
## UI
|
||||||
|
|
||||||
|
Kind indexes can declare a `kind` facet. `mapBrowserRow` fills it from the kind label. Tags stay the other facet. Public manifests still list `tags` only. The OnCo manifest lists both, and that manifest is written only under the gitignored local directory.
|
||||||
15
workplans/evidence/CONTENT-WP-0006-Review.md
Normal file
15
workplans/evidence/CONTENT-WP-0006-Review.md
Normal file
|
|
@ -0,0 +1,15 @@
|
||||||
|
# CONTENT-WP-0006 review
|
||||||
|
|
||||||
|
Date: 2026-10-19
|
||||||
|
|
||||||
|
T01, T02, T06, T07, and T08 were already done with the Wikidata connector. This review closes T03, T04, and T05.
|
||||||
|
|
||||||
|
T03 passed as a single-operator production session on computing-basics, recorded in `CONTENT-WP-0006-ImplementationNotes.md`. It is not a field study. The kind names on those pages come from the snapshot registry. `scripts/check_ui_boundary.py` still rejects oncology imports in the UI.
|
||||||
|
|
||||||
|
T04 is a no-go for API, JSON-LD, MCP, Ask, and a search backend. The session did not produce a measured limit that would reopen `decisions/ADR-0002-static-projection-first.md`.
|
||||||
|
|
||||||
|
T05 names the repository owner as the person who refreshes pins and reviews licenses. The text is in `connectors/AUTHORING.md`.
|
||||||
|
|
||||||
|
R8: the Wikidata pin is a local export, not a live API. The pilot did not exercise rate limits or a second operator. Those remain open if a later source needs them. They do not block this workplan.
|
||||||
|
|
||||||
|
Public check after these notes: `sh scripts/ci-synthetic.sh` (fixture validation, `scripts/audit_public_tree.py`, unittest, UI boundary, UI provenance, quality report, Vitest, typecheck).
|
||||||
|
|
@ -8,16 +8,16 @@ Wikidata is the connector that landed. The pinned input is `examples/sources/wik
|
||||||
|
|
||||||
## Real-user pilot
|
## Real-user pilot
|
||||||
|
|
||||||
A pilot with real users is not done. `workplans/CONTENT-WP-0006-external-pilot.md` keeps CONTENT-WP-0006-T03 open for that pilot. This note is the stand-in for a harness that cannot hold those sessions.
|
CONTENT-WP-0006-T03 is one operator session on 2026-10-19. The repository owner asked for the pilot, and the session ran the production computing-basics explorer. It is not a multi-person study and it does not invent other participants. The routes and results are in `workplans/evidence/CONTENT-WP-0006-Review.md`.
|
||||||
|
|
||||||
## API, JSON-LD, MCP, and Ask
|
## API, JSON-LD, MCP, and Ask
|
||||||
|
|
||||||
Those interfaces are not built. `INTENT.md` principle 8 says static and exportable graph and HTTP/JSON come first, and MCP and richer interfaces come later. CONTENT-WP-0006-T04 stays open until needs are observed from the pilot. `INTENT.md` also lists a production LLM answer service as a v0.1 non-goal.
|
Those interfaces are not built. The pilot's browse, kind, detail, neighbour, source, and in-browser search journeys did not show a need for them. `INTENT.md` principle 8 keeps static and exportable graph and HTTP/JSON first. A production LLM answer service stays a v0.1 non-goal. CONTENT-WP-0006-T04 records that no-go.
|
||||||
|
|
||||||
## Dynamic deployment and search
|
## Dynamic deployment and search
|
||||||
|
|
||||||
Dynamic deployment and a search backend stay a no-go. `decisions/ADR-0002-static-projection-first.md` decides to begin with offline connectors and a versioned static snapshot, with filter-in-browser at small scale, and to add remote read APIs, indexed search, and graph storage only with measured limits. CONTENT-WP-0006-T08 is the gate for that go or no-go. This increment follows the no-go.
|
Dynamic deployment and a search backend stay a no-go. `decisions/ADR-0002-static-projection-first.md` decides to begin with offline connectors and a versioned static snapshot, with filter-in-browser at small scale, and to add remote read APIs, indexed search, and graph storage only with measured limits. CONTENT-WP-0006-T08 is that no-go.
|
||||||
|
|
||||||
## OnCo authorization
|
## OnCo authorization
|
||||||
|
|
||||||
CONTENT-WP-0007 stays blocked until a human OnCo data-use authorization exists. `workplans/CONTENT-WP-0007-onco-reference-slice.md` sets that workplan to `blocked`, and CONTENT-WP-0007-T01 waits on `OnCo LICENSE-DATA` before any real reference-slice import. This increment does not read `upstream-onco/` data.
|
The repository owner authorized local OnCo reference use on 2026-10-19, limited to a data-origin notice that meets the CC BY-NC 4.0 duties. Redistribution, public CI, commercial use, and a company internal data pipeline are not granted. The non-confidential record is `tests/onco-reference/authorization.example.json`. The private copy stays gitignored. CONTENT-WP-0007 is the conformance workplan for that grant.
|
||||||
|
|
|
||||||
45
workplans/evidence/CONTENT-WP-0007-ImplementationNotes.md
Normal file
45
workplans/evidence/CONTENT-WP-0007-ImplementationNotes.md
Normal file
|
|
@ -0,0 +1,45 @@
|
||||||
|
# CONTENT-WP-0007 implementation notes
|
||||||
|
|
||||||
|
Date: 2026-10-19
|
||||||
|
|
||||||
|
## Grant
|
||||||
|
|
||||||
|
The repository owner approved local OnCo use when origin attribution matches the CC BY-NC 4.0 duties: creator attribution, license link, link to the material, retrieval date, pinned origin, modification notice, and a non-commercial mark. `tests/onco-reference/authorization.example.json` is the committed record. `tests/onco-reference/authorization.json` is the same grant and is gitignored. `distribution_allowed` and `ci_allowed` are false. `extract_allowlist` refuses to read the pin unless that file grants local use and refuses redistribution.
|
||||||
|
|
||||||
|
## Slice
|
||||||
|
|
||||||
|
Pin `b594d7433bf4e408b44215e25e5304696d6f840e`. Allowlist `tests/onco-reference/slice-allowlist.json` has 17 ids and no prose. The adapter writes `tests/onco-reference/local/dataspace/`, which is gitignored.
|
||||||
|
|
||||||
|
Counts from the licensed projection: 17 entities, 9 kinds, 18 sources, 41 stored relations, 884 outside-allowlist edges, 925 source edges. Stored plus outside equals source edges. Eight predicates are stored. Sixteen records omit the optional `aka` field, and that absence is listed in `missingOptional`. Every entity has a dated TL;DR and a summary, and at least two source refs. A simple path of three edges exists from `urn:ce:onco-reference:cancer:tnbc`. At least one entity has two stored edges of the same predicate.
|
||||||
|
|
||||||
|
`content_explorer/attribution.py` blocks projection when a CC BY or CC BY-NC duty is missing. Public `publish_directory` still refuses `CC-BY-NC-4.0`.
|
||||||
|
|
||||||
|
## Same UI
|
||||||
|
|
||||||
|
`CONTENT_EXPLORER_LICENSED=1` adds `tests/onco-reference/local` as a catalog root. The public default does not. Home, kind, and graph pages render `OriginNotice` when a source carries attribution. Detail pages keep a credit line on each source. CC0 dataspaces do not show that notice.
|
||||||
|
|
||||||
|
Filesystem reads in `ui/lib/catalog.ts` are marked `turbopackIgnore` so a production build does not trace the local slice into the server bundle. After the licensed build, the server chunk contains the path string and does not contain `src:onco:reference-slice`.
|
||||||
|
|
||||||
|
## Journeys
|
||||||
|
|
||||||
|
Licensed `next start` on 127.0.0.1:3492, 2026-10-19:
|
||||||
|
|
||||||
|
| Request | Attribution notice |
|
||||||
|
|---|---|
|
||||||
|
| `/onco-reference/` | present |
|
||||||
|
| `/onco-reference/cancer/` | present |
|
||||||
|
| `/onco-reference/graph/` | present |
|
||||||
|
| `/onco-reference/cancer/tnbc/` | present, with neighbour links |
|
||||||
|
| `/onco-reference/term/bystander-effect/` | present |
|
||||||
|
| `/onco-reference/?q=TNBC` | 200, query kept, filter remains in the browser |
|
||||||
|
| `/onco-reference/cancer/?tags=cancer&kind=Cancer` | 200, both facet keys kept |
|
||||||
|
| `/computing-basics/` on the same server | notice absent |
|
||||||
|
| `/` | notice absent |
|
||||||
|
|
||||||
|
An unknown slug renders the Next.js not-found document, with HTTP 200, the same way an unknown computing-basics slug does.
|
||||||
|
|
||||||
|
## Exceptions
|
||||||
|
|
||||||
|
The pinned OnCo UI was not started. Full `next build` of that checkout has previously exhausted memory, and screenshots of the slice would redistribute CC BY-NC text. Visual pixel parity and the 1280×800 / 390×844 captures are not claimed. Semantic comparison is the adapter test: allowlist order, edge accounting, attribution gaps, and the HTTP notice checks above.
|
||||||
|
|
||||||
|
Public CI is `sh scripts/ci-synthetic.sh`. It does not fetch OnCo. The licensed unittest skips when `upstream-onco/` is absent.
|
||||||
21
workplans/evidence/CONTENT-WP-0007-Review.md
Normal file
21
workplans/evidence/CONTENT-WP-0007-Review.md
Normal file
|
|
@ -0,0 +1,21 @@
|
||||||
|
# CONTENT-WP-0007 review
|
||||||
|
|
||||||
|
Date: 2026-10-19
|
||||||
|
|
||||||
|
T01 is done. The grantor is the repository owner, the date is 2026-10-19, and the scope is local reference-slice conformance. Redistribution, public CI, commercial use, and a company internal pipeline are outside the grant.
|
||||||
|
|
||||||
|
T02 pins `b594d7433bf4e408b44215e25e5304696d6f840e` and selects 17 entities. The journey table is in `CONTENT-WP-0007-ImplementationNotes.md`. Viewport screenshots are the documented exception in that note.
|
||||||
|
|
||||||
|
T03 is `connectors/onco_reference.py`. It reads only the allowlist, refuses a pin mismatch, and refuses a destination outside `tests/onco-reference/local/` or inside `examples/`.
|
||||||
|
|
||||||
|
T04 is `tests/test_onco_reference.py` plus the catalog and attribution UI tests. Covered: two or more predicates, inverse labels, a one-to-many edge, a simple path of depth 3, an explicit unresolved edge, a missing optional field, two facets, detail credits, and local mutations for a duplicate id, a removed license, a removed node, and a renamed title. The renamed title stays under the modification notice.
|
||||||
|
|
||||||
|
T05 compares structure and rights, not pixels. The edge equation is 41 stored + 884 outside = 925. Attribution gaps on the written sources are empty. Upstream visual parity is the exception in the implementation notes.
|
||||||
|
|
||||||
|
T06: `scripts/audit_public_tree.py` runs inside `scripts/ci-synthetic.sh` and fails if the local slice, `authorization.json`, or `upstream-onco/` is tracked, or if example JSON carries `CC-BY-NC-4.0` or `Data from OnCo`. The licensed build's server chunk does not embed the slice id.
|
||||||
|
|
||||||
|
T07: extraction calls `load_authorization` first. A grant with `distribution_allowed: true` is refused. A source with a missing CC duty is refused. Outside-allowlist edges are reported and are not stored.
|
||||||
|
|
||||||
|
T08: the allowlist pin, the source `originSha`, and this review are the reproducibility record. The local snapshot is regenerated by the licensed test and is not committed.
|
||||||
|
|
||||||
|
Public fixtures stay the executable default. The oncology kind labels live in the adapter and the gitignored snapshot, not in a UI import.
|
||||||
Loading…
Add table
Add a link
Reference in a new issue