feat: add repository projection synchronization
All checks were successful
Build and Publish Container Image / build-and-push (push) Successful in 1m9s

Assistant: codex
Assistant-Model: gpt-5.6-sol
Assistant-Session: 01a028f0-a42f-7582-89a8-ebaad7343834
This commit is contained in:
tegwick 2026-08-22 16:52:47 +02:00
parent d3258a6ba0
commit 280da08455
7 changed files with 317 additions and 2 deletions

View file

@ -72,3 +72,26 @@ written to a repository. State Hub handoffs:
`T01` remains in progress until those gates are satisfied, the dark workload is
healthy on PostgreSQL, and a restore is verified against the deployed topology.
## Post-preflight status
Later on 2026-08-22, the Forgejo package gate was completed and the package was
registered. A namespace-first apply correction passed five package tests and a
server-side dry-run, then was published as `rapp-sbom-nexus` commit `dd110d1`.
The database owner accepted the declaration under `RAPP-POSTGRES-WP-0005`
(State Hub workplan `d6d36add-27d9-5b5d-b6fe-ce80eb7b6dab`). The distinct
`platform-pg-2` cell and bootstrap secret were created and the cell reached 1/1
Ready. Acceptance remains blocked because:
- Warden still requires the attended founder act for the OpenBao connection,
dynamic runtime/migration roles, exact two-path policy, and renewable parent
token;
- continuous archiving is `False` and the first base backup is
`walArchivingFailing`; the value-safe pod log reports S3 `HeadBucket` 403 for
the governed `platform-pg-2` prefix.
The backup diagnosis and request for a successful base backup plus scratch
restore were sent to `rapp-postgres` as message
`31b72f4b-fcf8-4454-82b7-2dbf325c573e`. The SBOM Nexus workload remains
undeployed and no caller ingress is enabled.

View file

@ -0,0 +1,48 @@
# SBOM-WP-0002-T02 repository projection rehearsal — 2026-08-22
## Result
The projection-only synchronization path is implemented and rehearsed against
an isolated SBOM Nexus database. It does not scan repositories, invoke an
ingest route, or create snapshots.
## Source and selection
The current Repo Manager compatibility projection was read from State Hub's
`GET /repos/` boundary using explicit host id `bnt-lap001`:
| Measure | Result |
| --- | ---: |
| Source repositories | 120 |
| Active | 116 |
| Inactive | 4 |
| Selected checkout path | 120 |
The dry-run used an intentionally unreachable target URL and still completed,
proving that dry-run does not contact or mutate the Nexus target.
## Apply rehearsal
An ephemeral SQLite-backed Nexus on `127.0.0.1:18010` received the projection:
- 120 upserts;
- 120 exact active/path matches on read-back;
- zero missing, mismatched, or extra target repositories;
- zero SBOM ingest calls and zero snapshots.
The bounded catch-up query with `limit=3` returned exactly three of 116 active,
never-attempted repositories, ordered deterministically by slug:
`activity-core`, `adaptive-pricing`, and `agent-harness`. The response reported
`selected_count=3`, `never_count=116`, `stale_count=116`, and
`total_count=116`.
Production apply remains gated on the dark PostgreSQL runtime in
`SBOM-WP-0002-T01`.
## Verification
- `uv run ruff check src tests scripts`: pass
- `uv run pytest -q`: 15 passed, 1 conditional PostgreSQL skip
- projection dry-run: 120 would-upsert, target not contacted
- projection apply/read-back: 120/120 matched
- bounded catch-up: 3/3 selected in deterministic order

View file

@ -52,6 +52,21 @@ operation.
## Import State Hub history
Synchronize the minimum repository identity/path projection first. This command
does not call an ingest endpoint or create an SBOM snapshot:
```bash
uv run python scripts/sync_repository_projections.py \
--source-url http://127.0.0.1:8000 \
--target-url http://127.0.0.1:8010 \
--host-id bnt-lap001 \
--dry-run
```
Remove `--dry-run` only after the target is deployed. The apply path upserts all
active and inactive records, then reads them back and fails on a missing or
mismatched active/path projection. Extra target rows are reported but retained.
Run a read-only preview first:
```bash

View file

@ -0,0 +1,35 @@
#!/usr/bin/env python3
"""Synchronize Repo Manager repository projections into SBOM Nexus."""
from __future__ import annotations
import argparse
import json
import urllib.error
from sbom_nexus.projection import sync_repository_projections
def main() -> None:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--source-url", default="http://127.0.0.1:8000")
parser.add_argument("--target-url", default="http://127.0.0.1:8010")
parser.add_argument("--host-id")
parser.add_argument("--dry-run", action="store_true")
args = parser.parse_args()
try:
result = sync_repository_projections(
args.source_url,
args.target_url,
dry_run=args.dry_run,
host_id=args.host_id,
)
except (urllib.error.URLError, json.JSONDecodeError) as exc:
raise SystemExit(f"Projection sync failed: {exc}") from exc
print(json.dumps(result, indent=2, sort_keys=True))
if not result["ok"]:
raise SystemExit(1)
if __name__ == "__main__":
main()

View file

@ -0,0 +1,102 @@
"""Synchronize the minimum repository projection used by SBOM Nexus."""
from __future__ import annotations
import socket
import urllib.parse
from collections import Counter
from typing import Any
from sbom_nexus.importer import request_json
def _checkout_path(repo: dict[str, Any], host_id: str | None) -> str | None:
host_paths = repo.get("host_paths") or {}
selected_host = host_id or socket.gethostname()
return host_paths.get(selected_host) or repo.get("local_path")
def sync_repository_projections(
source_url: str,
target_url: str,
*,
dry_run: bool,
host_id: str | None = None,
) -> dict[str, Any]:
"""Upsert repository identity/path projections without scanning or ingesting."""
repositories = request_json(source_url, "/repos/")
projections: dict[str, dict[str, Any]] = {}
statuses = Counter()
for repo in repositories:
slug = repo["slug"]
if slug in projections:
return {
"ok": False,
"dry_run": dry_run,
"source_repo_count": len(repositories),
"error": f"duplicate repository slug: {slug}",
}
active = repo.get("status", "active") == "active"
checkout_path = _checkout_path(repo, host_id)
projections[slug] = {
"slug": slug,
"active": active,
"checkout_path": checkout_path,
}
statuses["active" if active else "inactive"] += 1
statuses["with_checkout_path" if checkout_path else "without_checkout_path"] += 1
if dry_run:
return {
"ok": True,
"dry_run": True,
"host_id": host_id or socket.gethostname(),
"source_repo_count": len(projections),
"counts": dict(statuses),
"would_upsert": len(projections),
"reconciliation": None,
}
for slug, projection in sorted(projections.items()):
request_json(
target_url,
f"/repositories/{urllib.parse.quote(slug, safe='')}",
method="PUT",
body={
"active": projection["active"],
"checkout_path": projection["checkout_path"],
},
)
target_repositories = {
repo["slug"]: repo for repo in request_json(target_url, "/repositories/")
}
missing = sorted(set(projections) - set(target_repositories))
mismatched = []
for slug in sorted(set(projections) & set(target_repositories)):
expected = projections[slug]
actual = target_repositories[slug]
differences = {
field: {"source": expected[field], "target": actual.get(field)}
for field in ("active", "checkout_path")
if expected[field] != actual.get(field)
}
if differences:
mismatched.append({"repo_slug": slug, "differences": differences})
reconciliation = {
"ok": not missing and not mismatched,
"matched_repo_count": len(projections) - len(missing) - len(mismatched),
"missing_repo_slugs": missing,
"mismatched_repositories": mismatched,
"extra_target_repo_slugs": sorted(set(target_repositories) - set(projections)),
}
return {
"ok": reconciliation["ok"],
"dry_run": False,
"host_id": host_id or socket.gethostname(),
"source_repo_count": len(projections),
"counts": dict(statuses),
"upserted": len(projections),
"reconciliation": reconciliation,
}

82
tests/test_projection.py Normal file
View file

@ -0,0 +1,82 @@
from __future__ import annotations
from typing import Any
from sbom_nexus import projection
def test_projection_sync_is_projection_only_and_reconciles(monkeypatch) -> None:
source = [
{
"slug": "active-repo",
"status": "active",
"local_path": "/fallback/active",
"host_paths": {"build-host": "/repos/active"},
},
{
"slug": "retired-repo",
"status": "retired",
"local_path": "/repos/retired",
},
]
target: dict[str, dict[str, Any]] = {}
calls: list[tuple[str, str, str]] = []
def fake_request(
base_url: str,
path: str,
*,
method: str = "GET",
body: dict[str, Any] | None = None,
) -> Any:
calls.append((base_url, method, path))
if base_url == "source" and path == "/repos/":
return source
if base_url == "target" and method == "PUT":
assert body is not None
slug = path.rsplit("/", 1)[1]
target[slug] = {"slug": slug, **body}
return target[slug]
if base_url == "target" and path == "/repositories/":
return list(target.values())
raise AssertionError((base_url, method, path))
monkeypatch.setattr(projection, "request_json", fake_request)
result = projection.sync_repository_projections(
"source", "target", dry_run=False, host_id="build-host"
)
assert result["ok"] is True
assert result["source_repo_count"] == 2
assert result["counts"] == {
"active": 1,
"with_checkout_path": 2,
"inactive": 1,
}
assert result["reconciliation"]["matched_repo_count"] == 2
assert target["active-repo"]["checkout_path"] == "/repos/active"
assert target["retired-repo"]["active"] is False
assert all("/sbom/" not in path for _, _, path in calls)
def test_projection_dry_run_does_not_contact_target(monkeypatch) -> None:
def fake_request(
base_url: str,
path: str,
*,
method: str = "GET",
body: dict[str, Any] | None = None,
) -> Any:
assert base_url == "source"
assert method == "GET"
assert path == "/repos/"
return [{"slug": "demo", "status": "active", "local_path": "/repos/demo"}]
monkeypatch.setattr(projection, "request_json", fake_request)
result = projection.sync_repository_projections(
"source", "unreachable-target", dry_run=True, host_id="build-host"
)
assert result["ok"] is True
assert result["would_upsert"] == 1
assert result["reconciliation"] is None

View file

@ -38,7 +38,7 @@ id: SBOM-WP-0002-T01
status: progress
priority: high
needs_human: true
intervention_note: "Warden requires operator OIDC/MFA for first rapp-sbom-nexus Forgejo repository creation and one-time secret provisioning for platform-pg-2/OpenBao."
intervention_note: "Warden requires an attended founder act for the first platform-pg-2 OpenBao database connection, SBOM Nexus dynamic roles/policies, and renewable External Secrets parent token. The database owner must also repair the governed backup credential/policy after a live S3 HeadBucket 403."
state_hub_task_id: "95a520d4-30c2-5c87-8054-6bfe549c2686"
```
@ -47,12 +47,15 @@ deploy the API without callers, and capture health plus backup/restore evidence.
Image publication, package rendering, family validation, and server-side dry-run
are complete; see
`docs/evidence/SBOM-WP-0002-T01-dark-deployment-preflight-2026-08-22.md`.
The overflow cell is now 1/1 Ready, but continuous archiving and its first base
backup fail closed on an S3 `HeadBucket` 403. Database-owner work is tracked by
`RAPP-POSTGRES-WP-0005`; no caller or Nexus runtime has been enabled.
## Synchronize repository projections
```task
id: SBOM-WP-0002-T02
status: todo
status: progress
priority: high
state_hub_task_id: "22cbb75f-d82f-5b47-9fef-27bde3b410d5"
```
@ -60,6 +63,13 @@ state_hub_task_id: "22cbb75f-d82f-5b47-9fef-27bde3b410d5"
Populate active repository identity and host checkout paths from Repo Manager.
Verify fleet totals and catch-up ordering without performing ingest.
The projection-only synchronizer and reconciliation contract are implemented.
Its dry-run never contacts the Nexus target and no code path calls an SBOM
ingest route. Production apply and catch-up ordering proof wait for the dark
runtime from T01. The isolated rehearsal reconciled all 120 source projections
and selected exactly the oldest three of 116 active repositories; see
`docs/evidence/SBOM-WP-0002-T02-repository-projection-rehearsal-2026-08-22.md`.
## Import and reconcile State Hub history
```task