diff --git a/Makefile b/Makefile index 5a9ce85..c5deb64 100644 --- a/Makefile +++ b/Makefile @@ -20,6 +20,13 @@ AGENT_VARS_FILE := $(RAILIANCE_INFRA)/ansible/inventory/group_vars/all.yaml ops-inventory-view: ## Render the ops-hub service catalog now view python3 ops/render_service_inventory.py +.PHONY: classification-status classification-check +classification-status: ## Report active-fleet Repo Classification convergence + python3 tools/repo_classification_convergence.py + +classification-check: ## Fail until every active repo has a valid source and projection + python3 tools/repo_classification_convergence.py --require-converged + .PHONY: custodian-keygen custodian-keygen: ## Generate custodian agent SSH keypair (one-time setup) @if [ -f "$(CUSTODIAN_KEY)" ]; then \ diff --git a/SCOPE.md b/SCOPE.md index a6cb1f4..76b4ac2 100644 --- a/SCOPE.md +++ b/SCOPE.md @@ -1,7 +1,7 @@ --- domain: custodian repo: the-custodian -updated: "2026-07-08" +updated: "2026-08-23" --- # SCOPE @@ -87,9 +87,12 @@ their own repositories and are referenced here only as integration pointers. ## Current State -- Status: **stable maintenance** — governance substrate in daily ecosystem use. - No `active`/`ready`/`blocked` CUST-WP workplans; one `proposed` plan - (`CUST-WP-0055` terminology refactor) awaits review before pickup. +- Status: **stable maintenance with two active coordination workplans** — + `CUST-WP-0064` awaits the first unassisted weekday controlled-source SBOM + fire; `CUST-WP-0065` awaits 13 repo-owner classification decisions while its + live convergence check is automated in `make classification-status`. The + active registry contains 116 repositories after archiving the already-retired + Inter-Hub row. - Business platform enablement (`CUST-WP-0058`, finished 2026-07-10): DR-1/2/3 resolved (instance-per-client tenancy; coulomb.social as standalone app; app-local identity), business-app service contract diff --git a/WORK-RECORDS.md b/WORK-RECORDS.md index 89c0194..2d5a141 100644 --- a/WORK-RECORDS.md +++ b/WORK-RECORDS.md @@ -409,7 +409,7 @@ | task | CUST-WP-0063-T05 | done | — | workplans/CUST-WP-0063-inbox-governance-packets.md | | task | CUST-WP-0064-T01 | done | — | workplans/CUST-WP-0064-sbom-controlled-scan-inputs.md | | task | CUST-WP-0064-T02 | done | — | workplans/CUST-WP-0064-sbom-controlled-scan-inputs.md | -| task | CUST-WP-0064-T03 | progress | — | workplans/CUST-WP-0064-sbom-controlled-scan-inputs.md | +| task | CUST-WP-0064-T03 | done | — | workplans/CUST-WP-0064-sbom-controlled-scan-inputs.md | | task | CUST-WP-0064-T04 | progress | — | workplans/CUST-WP-0064-sbom-controlled-scan-inputs.md | | task | CUST-WP-0065-T01 | done | — | workplans/CUST-WP-0065-reclassify-repos-and-guidance-docs-to-the-new-sector-domain.md | | task | CUST-WP-0065-T02 | done | — | workplans/CUST-WP-0065-reclassify-repos-and-guidance-docs-to-the-new-sector-domain.md | diff --git a/docs/repo-classification-sector-migration-baseline-2026-08-23.md b/docs/repo-classification-sector-migration-baseline-2026-08-23.md index 77660cb..8116723 100644 --- a/docs/repo-classification-sector-migration-baseline-2026-08-23.md +++ b/docs/repo-classification-sector-migration-baseline-2026-08-23.md @@ -59,9 +59,12 @@ This table is a triage aid, not an automatic rewrite map. ## Reproduction -Read active repository records from `GET /repos/`, select records whose -`status` is `active` and whose `category` is null, then test the registered -`local_path` for `.repo-classification.yaml`. The observed counts are: +Run `make classification-status`. The command reads active repository records +from `GET /repos/`, compares every registered `local_path` with its +`.repo-classification.yaml`, validates present source files against the canon, +and reports owner-source gaps separately from projection gaps. Use +`make classification-check` as the eventual fail-closed convergence gate. +The initial observed counts were: ```text registered: 121 @@ -105,3 +108,12 @@ digest verification, and API projection passed. The current checkpoint is 104/117 classified, 13 missing-source/null-category repos, and zero present source files left unprojected. This is governed pending work rather than an unexplained projection failure. + +The first automated convergence run additionally found `inter-hub` still +marked active with a projected classification even though its registered +checkout had been removed. Custodian retirement evidence already records Core +Hub as the sole production surface from 2026-07-08. The supported State Hub +archive transition corrected the stale row while preserving its history. The +current active denominator is therefore 116: 103 classified, 13 +missing-source/null-category, zero present-but-unprojected, and zero invalid +source records. diff --git a/tests/test_repo_classification_convergence.py b/tests/test_repo_classification_convergence.py new file mode 100644 index 0000000..5563628 --- /dev/null +++ b/tests/test_repo_classification_convergence.py @@ -0,0 +1,83 @@ +from __future__ import annotations + +from pathlib import Path + +from tools.repo_classification_convergence import analyze_repositories, render_text + + +def _record(slug: str, path: Path | None, *, category: str | None) -> dict: + return { + "slug": slug, + "status": "active", + "category": category, + "local_path": str(path) if path else None, + } + + +def _write_valid(path: Path) -> None: + path.mkdir() + (path / ".repo-classification.yaml").write_text( + """repo_classification: + category: project + domain: infotech + secondary_domains: [] + capability_tags: [] + business_stake: [] + business_mechanics: [] +""", + encoding="utf-8", + ) + + +def test_report_separates_owner_gap_from_projection_gap(tmp_path: Path) -> None: + classified = tmp_path / "classified" + unprojected = tmp_path / "unprojected" + _write_valid(classified) + _write_valid(unprojected) + + report = analyze_repositories( + [ + _record("classified", classified, category="project"), + _record("owner-gap", tmp_path / "owner-gap", category=None), + _record("projection-gap", unprojected, category=None), + {"slug": "archived", "status": "archived", "category": None}, + ] + ) + + assert report["registered"] == 4 + assert report["active"] == 3 + assert report["active_classified"] == 1 + assert report["missing_source"] == ["owner-gap"] + assert report["present_unprojected"] == ["projection-gap"] + assert report["converged"] is False + + +def test_report_flags_stale_projection_and_invalid_source(tmp_path: Path) -> None: + invalid = tmp_path / "invalid" + invalid.mkdir() + (invalid / ".repo-classification.yaml").write_text( + "repo_classification:\n category: made-up\n domain: custodian\n", + encoding="utf-8", + ) + + report = analyze_repositories( + [ + _record("stale", tmp_path / "stale", category="project"), + _record("invalid", invalid, category="project"), + ] + ) + + assert report["projected_without_source"] == ["stale"] + assert "invalid" in report["invalid_source"] + assert report["converged"] is False + + +def test_clean_active_fleet_converges_and_renders(tmp_path: Path) -> None: + repo = tmp_path / "repo" + _write_valid(repo) + report = analyze_repositories([_record("repo", repo, category="project")]) + + assert report["converged"] is True + assert report["source_warning_count"] == 0 + assert "active classified: 1" in render_text(report) + assert "converged: yes" in render_text(report) diff --git a/tools/repo_classification_convergence.py b/tools/repo_classification_convergence.py new file mode 100644 index 0000000..4c64371 --- /dev/null +++ b/tools/repo_classification_convergence.py @@ -0,0 +1,204 @@ +#!/usr/bin/env python3 +"""Report active-fleet Repo Classification convergence from State Hub. + +The repository file remains authoritative. This command compares the live +State Hub projection with each active record's registered local checkout and +separates missing owner decisions from projection failures. +""" +from __future__ import annotations + +import argparse +import json +import os +import sys +import urllib.error +import urllib.request +from pathlib import Path +from typing import Any, Callable + +import yaml + +REPO_ROOT = Path(__file__).resolve().parent.parent +if str(REPO_ROOT) not in sys.path: + sys.path.insert(0, str(REPO_ROOT)) + +from tools.validate_repo_classification import load_allowed, validate # noqa: E402 + +DEFAULT_API_BASE = "http://127.0.0.1:8000" +UrlOpen = Callable[..., Any] + + +class FleetReadError(RuntimeError): + """State Hub did not return a usable repository projection.""" + + +def fetch_repositories( + api_base: str, + *, + timeout: float = 15.0, + opener: UrlOpen = urllib.request.urlopen, +) -> list[dict[str, Any]]: + url = f"{api_base.rstrip('/')}/repos/?limit=500" + request = urllib.request.Request(url, headers={"Accept": "application/json"}) + try: + with opener(request, timeout=timeout) as response: + payload = json.load(response) + except (OSError, urllib.error.URLError, json.JSONDecodeError) as exc: + raise FleetReadError(f"cannot read State Hub repositories: {exc}") from exc + if not isinstance(payload, list) or any(not isinstance(item, dict) for item in payload): + raise FleetReadError("State Hub /repos response is not a list of objects") + return payload + + +def _source_path(record: dict[str, Any]) -> Path | None: + local_path = record.get("local_path") + if not isinstance(local_path, str) or not local_path.strip(): + return None + return Path(local_path) / ".repo-classification.yaml" + + +def analyze_repositories( + records: list[dict[str, Any]], + *, + validate_sources: bool = True, +) -> dict[str, Any]: + active = [record for record in records if record.get("status") == "active"] + classified = [record for record in active if record.get("category") is not None] + null_records = [record for record in active if record.get("category") is None] + missing_source: list[str] = [] + present_unprojected: list[str] = [] + projected_without_source: list[str] = [] + invalid_source: dict[str, list[str]] = {} + source_warnings: dict[str, list[str]] = {} + duplicate_slugs: list[str] = [] + + slug_counts: dict[str, int] = {} + allowed = load_allowed() if validate_sources else None + for record in active: + slug = str(record.get("slug") or "") + slug_counts[slug] = slug_counts.get(slug, 0) + 1 + path = _source_path(record) + source_present = path is not None and path.is_file() + + if record.get("category") is None: + (present_unprojected if source_present else missing_source).append(slug) + elif not source_present: + projected_without_source.append(slug) + + if validate_sources and source_present and path is not None: + try: + document = yaml.safe_load(path.read_text(encoding="utf-8")) + errors, warnings = validate(document, allowed) + except (OSError, yaml.YAMLError) as exc: + errors, warnings = [f"cannot read/parse source: {exc}"], [] + if errors: + invalid_source[slug] = errors + if warnings: + source_warnings[slug] = warnings + + duplicate_slugs = sorted(slug for slug, count in slug_counts.items() if count > 1) + missing_source.sort() + present_unprojected.sort() + projected_without_source.sort() + + blocking = bool( + missing_source + or present_unprojected + or projected_without_source + or invalid_source + or duplicate_slugs + ) + return { + "registered": len(records), + "active": len(active), + "active_classified": len(classified), + "active_null_category": len(null_records), + "missing_source": missing_source, + "present_unprojected": present_unprojected, + "projected_without_source": projected_without_source, + "invalid_source": dict(sorted(invalid_source.items())), + "source_warnings": dict(sorted(source_warnings.items())), + "source_warning_count": sum(len(items) for items in source_warnings.values()), + "duplicate_active_slugs": duplicate_slugs, + "converged": not blocking, + } + + +def render_text(report: dict[str, Any], *, show_warnings: bool = False) -> str: + lines = [ + f"registered: {report['registered']}", + f"active: {report['active']}", + f"active classified: {report['active_classified']}", + f"active null category: {report['active_null_category']}", + f"missing source file: {len(report['missing_source'])}", + f"source present but unprojected: {len(report['present_unprojected'])}", + f"projected without source file: {len(report['projected_without_source'])}", + f"invalid source file: {len(report['invalid_source'])}", + f"source validation warnings: {report['source_warning_count']}", + f"duplicate active slug: {len(report['duplicate_active_slugs'])}", + f"converged: {'yes' if report['converged'] else 'no'}", + ] + for key in ( + "missing_source", + "present_unprojected", + "projected_without_source", + "duplicate_active_slugs", + ): + if report[key]: + lines.append(f"{key}: {', '.join(report[key])}") + for slug, errors in report["invalid_source"].items(): + lines.append(f"invalid_source[{slug}]: {'; '.join(errors)}") + if show_warnings: + for slug, warnings in report["source_warnings"].items(): + lines.append(f"source_warnings[{slug}]: {'; '.join(warnings)}") + return "\n".join(lines) + + +def build_parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--api-base", + default=os.environ.get("STATE_HUB_API_BASE", DEFAULT_API_BASE), + help="State Hub REST base URL (default: %(default)s)", + ) + parser.add_argument("--timeout", type=float, default=15.0) + parser.add_argument("--json", action="store_true", help="emit machine-readable JSON") + parser.add_argument( + "--show-warnings", + action="store_true", + help="include individual non-blocking source validation warnings", + ) + parser.add_argument( + "--skip-source-validation", + action="store_true", + help="only compare file presence and projection state", + ) + parser.add_argument( + "--require-converged", + action="store_true", + help="exit 1 while any active classification gap remains", + ) + return parser + + +def main(argv: list[str] | None = None) -> int: + args = build_parser().parse_args(argv) + try: + records = fetch_repositories(args.api_base, timeout=args.timeout) + report = analyze_repositories( + records, + validate_sources=not args.skip_source_validation, + ) + except FleetReadError as exc: + print(f"ERROR: {exc}", file=sys.stderr) + return 2 + + if args.json: + print(json.dumps(report, indent=2, sort_keys=True)) + else: + print(render_text(report, show_warnings=args.show_warnings)) + return 1 if args.require_converged and not report["converged"] else 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/workplans/CUST-WP-0064-sbom-controlled-scan-inputs.md b/workplans/CUST-WP-0064-sbom-controlled-scan-inputs.md index 966a788..dd6d00f 100644 --- a/workplans/CUST-WP-0064-sbom-controlled-scan-inputs.md +++ b/workplans/CUST-WP-0064-sbom-controlled-scan-inputs.md @@ -124,7 +124,7 @@ Process/database health, repository reads, and zero-restarter startup all pass. ```task id: CUST-WP-0064-T03 -status: progress +status: done priority: high state_hub_task_id: "6c645778-be59-57b1-bf44-d974a3a1e49f" ``` @@ -143,8 +143,16 @@ It then projected the live oldest-three target set (`can-you-assist`, The existing unpaused Temporal schedule was reconciled unchanged at limit 3 and operator-triggered: it froze exactly those targets, spawned zero tasks, created three provenance-bearing terminal `no-manifest` snapshots, and moved -`never_count` 94 to 91. Keep this task open for sustainable projection as each -new oldest-N batch is exposed. +`never_count` 94 to 91. + +**Done (2026-08-23):** after that controlled batch advanced, the next oldest +three (`citation-work`, `clay-borg`, and `config-atlas`) were selected with +matching `forgejo-archive-v1` references and full 40-character revisions. An +in-worker read-only report probe on deployed Activity Core digest +`sha256:9c611a394c117c8ccfe2fd813c0dda1943ea8a7444aa4bd4742d83712b2cf559` +returned `selected_count=3` and `controlled_source_count=3`. This proves +source-reference projection follows each newly exposed bounded batch rather +than the attended target set; T04 retains only the first unassisted fire. ## Prove real daily freshness improvement diff --git a/workplans/CUST-WP-0065-reclassify-repos-and-guidance-docs-to-the-new-sector-domain.md b/workplans/CUST-WP-0065-reclassify-repos-and-guidance-docs-to-the-new-sector-domain.md index bb8a904..6f4977c 100644 --- a/workplans/CUST-WP-0065-reclassify-repos-and-guidance-docs-to-the-new-sector-domain.md +++ b/workplans/CUST-WP-0065-reclassify-repos-and-guidance-docs-to-the-new-sector-domain.md @@ -141,6 +141,24 @@ unprojected. Project names and repository identities were not rewritten. After the first owner delivery, the live projection is 104/117 classified with 13 active null-category records and again zero present-but-unprojected files. +**Repeatable convergence gate (2026-08-23):** +`tools/repo_classification_convergence.py` now derives those counts directly +from the State Hub registry, checks each active record's registered checkout, +validates every present source file against the canonical vocabulary, and +separates missing owner records, present-but-unprojected records, stale +projections without a source file, invalid sources, and duplicate active slugs. +`make classification-status` reports progress; `make classification-check` +fails closed until all active gaps are resolved. + +Its first run exposed a stale active registry row hidden by the former +null-category-only check: retired `inter-hub` retained a projected category but +its registered checkout no longer exists. Existing Custodian retirement +evidence records Core Hub as the sole production surface since 2026-07-08, so +the supported State Hub archive endpoint corrected that row without altering +its historical classification. The active fleet is now 103/116 classified; +the same 13 owner-source gaps remain, with no projection or source-validity +failure. + ## Indexing note **Resolved (2026-08-23):** Repo Manager revisions `7a15f1d` and `7b9fdaa`