feat: add repository projection synchronization
All checks were successful
Build and Publish Container Image / build-and-push (push) Successful in 1m9s
All checks were successful
Build and Publish Container Image / build-and-push (push) Successful in 1m9s
Assistant: codex Assistant-Model: gpt-5.6-sol Assistant-Session: 01a028f0-a42f-7582-89a8-ebaad7343834
This commit is contained in:
parent
d3258a6ba0
commit
280da08455
7 changed files with 317 additions and 2 deletions
|
|
@ -72,3 +72,26 @@ written to a repository. State Hub handoffs:
|
|||
|
||||
`T01` remains in progress until those gates are satisfied, the dark workload is
|
||||
healthy on PostgreSQL, and a restore is verified against the deployed topology.
|
||||
|
||||
## Post-preflight status
|
||||
|
||||
Later on 2026-08-22, the Forgejo package gate was completed and the package was
|
||||
registered. A namespace-first apply correction passed five package tests and a
|
||||
server-side dry-run, then was published as `rapp-sbom-nexus` commit `dd110d1`.
|
||||
|
||||
The database owner accepted the declaration under `RAPP-POSTGRES-WP-0005`
|
||||
(State Hub workplan `d6d36add-27d9-5b5d-b6fe-ce80eb7b6dab`). The distinct
|
||||
`platform-pg-2` cell and bootstrap secret were created and the cell reached 1/1
|
||||
Ready. Acceptance remains blocked because:
|
||||
|
||||
- Warden still requires the attended founder act for the OpenBao connection,
|
||||
dynamic runtime/migration roles, exact two-path policy, and renewable parent
|
||||
token;
|
||||
- continuous archiving is `False` and the first base backup is
|
||||
`walArchivingFailing`; the value-safe pod log reports S3 `HeadBucket` 403 for
|
||||
the governed `platform-pg-2` prefix.
|
||||
|
||||
The backup diagnosis and request for a successful base backup plus scratch
|
||||
restore were sent to `rapp-postgres` as message
|
||||
`31b72f4b-fcf8-4454-82b7-2dbf325c573e`. The SBOM Nexus workload remains
|
||||
undeployed and no caller ingress is enabled.
|
||||
|
|
|
|||
|
|
@ -0,0 +1,48 @@
|
|||
# SBOM-WP-0002-T02 repository projection rehearsal — 2026-08-22
|
||||
|
||||
## Result
|
||||
|
||||
The projection-only synchronization path is implemented and rehearsed against
|
||||
an isolated SBOM Nexus database. It does not scan repositories, invoke an
|
||||
ingest route, or create snapshots.
|
||||
|
||||
## Source and selection
|
||||
|
||||
The current Repo Manager compatibility projection was read from State Hub's
|
||||
`GET /repos/` boundary using explicit host id `bnt-lap001`:
|
||||
|
||||
| Measure | Result |
|
||||
| --- | ---: |
|
||||
| Source repositories | 120 |
|
||||
| Active | 116 |
|
||||
| Inactive | 4 |
|
||||
| Selected checkout path | 120 |
|
||||
|
||||
The dry-run used an intentionally unreachable target URL and still completed,
|
||||
proving that dry-run does not contact or mutate the Nexus target.
|
||||
|
||||
## Apply rehearsal
|
||||
|
||||
An ephemeral SQLite-backed Nexus on `127.0.0.1:18010` received the projection:
|
||||
|
||||
- 120 upserts;
|
||||
- 120 exact active/path matches on read-back;
|
||||
- zero missing, mismatched, or extra target repositories;
|
||||
- zero SBOM ingest calls and zero snapshots.
|
||||
|
||||
The bounded catch-up query with `limit=3` returned exactly three of 116 active,
|
||||
never-attempted repositories, ordered deterministically by slug:
|
||||
`activity-core`, `adaptive-pricing`, and `agent-harness`. The response reported
|
||||
`selected_count=3`, `never_count=116`, `stale_count=116`, and
|
||||
`total_count=116`.
|
||||
|
||||
Production apply remains gated on the dark PostgreSQL runtime in
|
||||
`SBOM-WP-0002-T01`.
|
||||
|
||||
## Verification
|
||||
|
||||
- `uv run ruff check src tests scripts`: pass
|
||||
- `uv run pytest -q`: 15 passed, 1 conditional PostgreSQL skip
|
||||
- projection dry-run: 120 would-upsert, target not contacted
|
||||
- projection apply/read-back: 120/120 matched
|
||||
- bounded catch-up: 3/3 selected in deterministic order
|
||||
|
|
@ -52,6 +52,21 @@ operation.
|
|||
|
||||
## Import State Hub history
|
||||
|
||||
Synchronize the minimum repository identity/path projection first. This command
|
||||
does not call an ingest endpoint or create an SBOM snapshot:
|
||||
|
||||
```bash
|
||||
uv run python scripts/sync_repository_projections.py \
|
||||
--source-url http://127.0.0.1:8000 \
|
||||
--target-url http://127.0.0.1:8010 \
|
||||
--host-id bnt-lap001 \
|
||||
--dry-run
|
||||
```
|
||||
|
||||
Remove `--dry-run` only after the target is deployed. The apply path upserts all
|
||||
active and inactive records, then reads them back and fails on a missing or
|
||||
mismatched active/path projection. Extra target rows are reported but retained.
|
||||
|
||||
Run a read-only preview first:
|
||||
|
||||
```bash
|
||||
|
|
|
|||
35
scripts/sync_repository_projections.py
Normal file
35
scripts/sync_repository_projections.py
Normal file
|
|
@ -0,0 +1,35 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Synchronize Repo Manager repository projections into SBOM Nexus."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import urllib.error
|
||||
|
||||
from sbom_nexus.projection import sync_repository_projections
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--source-url", default="http://127.0.0.1:8000")
|
||||
parser.add_argument("--target-url", default="http://127.0.0.1:8010")
|
||||
parser.add_argument("--host-id")
|
||||
parser.add_argument("--dry-run", action="store_true")
|
||||
args = parser.parse_args()
|
||||
try:
|
||||
result = sync_repository_projections(
|
||||
args.source_url,
|
||||
args.target_url,
|
||||
dry_run=args.dry_run,
|
||||
host_id=args.host_id,
|
||||
)
|
||||
except (urllib.error.URLError, json.JSONDecodeError) as exc:
|
||||
raise SystemExit(f"Projection sync failed: {exc}") from exc
|
||||
print(json.dumps(result, indent=2, sort_keys=True))
|
||||
if not result["ok"]:
|
||||
raise SystemExit(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
102
src/sbom_nexus/projection.py
Normal file
102
src/sbom_nexus/projection.py
Normal file
|
|
@ -0,0 +1,102 @@
|
|||
"""Synchronize the minimum repository projection used by SBOM Nexus."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import socket
|
||||
import urllib.parse
|
||||
from collections import Counter
|
||||
from typing import Any
|
||||
|
||||
from sbom_nexus.importer import request_json
|
||||
|
||||
|
||||
def _checkout_path(repo: dict[str, Any], host_id: str | None) -> str | None:
|
||||
host_paths = repo.get("host_paths") or {}
|
||||
selected_host = host_id or socket.gethostname()
|
||||
return host_paths.get(selected_host) or repo.get("local_path")
|
||||
|
||||
|
||||
def sync_repository_projections(
|
||||
source_url: str,
|
||||
target_url: str,
|
||||
*,
|
||||
dry_run: bool,
|
||||
host_id: str | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""Upsert repository identity/path projections without scanning or ingesting."""
|
||||
repositories = request_json(source_url, "/repos/")
|
||||
projections: dict[str, dict[str, Any]] = {}
|
||||
statuses = Counter()
|
||||
for repo in repositories:
|
||||
slug = repo["slug"]
|
||||
if slug in projections:
|
||||
return {
|
||||
"ok": False,
|
||||
"dry_run": dry_run,
|
||||
"source_repo_count": len(repositories),
|
||||
"error": f"duplicate repository slug: {slug}",
|
||||
}
|
||||
active = repo.get("status", "active") == "active"
|
||||
checkout_path = _checkout_path(repo, host_id)
|
||||
projections[slug] = {
|
||||
"slug": slug,
|
||||
"active": active,
|
||||
"checkout_path": checkout_path,
|
||||
}
|
||||
statuses["active" if active else "inactive"] += 1
|
||||
statuses["with_checkout_path" if checkout_path else "without_checkout_path"] += 1
|
||||
|
||||
if dry_run:
|
||||
return {
|
||||
"ok": True,
|
||||
"dry_run": True,
|
||||
"host_id": host_id or socket.gethostname(),
|
||||
"source_repo_count": len(projections),
|
||||
"counts": dict(statuses),
|
||||
"would_upsert": len(projections),
|
||||
"reconciliation": None,
|
||||
}
|
||||
|
||||
for slug, projection in sorted(projections.items()):
|
||||
request_json(
|
||||
target_url,
|
||||
f"/repositories/{urllib.parse.quote(slug, safe='')}",
|
||||
method="PUT",
|
||||
body={
|
||||
"active": projection["active"],
|
||||
"checkout_path": projection["checkout_path"],
|
||||
},
|
||||
)
|
||||
|
||||
target_repositories = {
|
||||
repo["slug"]: repo for repo in request_json(target_url, "/repositories/")
|
||||
}
|
||||
missing = sorted(set(projections) - set(target_repositories))
|
||||
mismatched = []
|
||||
for slug in sorted(set(projections) & set(target_repositories)):
|
||||
expected = projections[slug]
|
||||
actual = target_repositories[slug]
|
||||
differences = {
|
||||
field: {"source": expected[field], "target": actual.get(field)}
|
||||
for field in ("active", "checkout_path")
|
||||
if expected[field] != actual.get(field)
|
||||
}
|
||||
if differences:
|
||||
mismatched.append({"repo_slug": slug, "differences": differences})
|
||||
|
||||
reconciliation = {
|
||||
"ok": not missing and not mismatched,
|
||||
"matched_repo_count": len(projections) - len(missing) - len(mismatched),
|
||||
"missing_repo_slugs": missing,
|
||||
"mismatched_repositories": mismatched,
|
||||
"extra_target_repo_slugs": sorted(set(target_repositories) - set(projections)),
|
||||
}
|
||||
return {
|
||||
"ok": reconciliation["ok"],
|
||||
"dry_run": False,
|
||||
"host_id": host_id or socket.gethostname(),
|
||||
"source_repo_count": len(projections),
|
||||
"counts": dict(statuses),
|
||||
"upserted": len(projections),
|
||||
"reconciliation": reconciliation,
|
||||
}
|
||||
82
tests/test_projection.py
Normal file
82
tests/test_projection.py
Normal file
|
|
@ -0,0 +1,82 @@
|
|||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from sbom_nexus import projection
|
||||
|
||||
|
||||
def test_projection_sync_is_projection_only_and_reconciles(monkeypatch) -> None:
|
||||
source = [
|
||||
{
|
||||
"slug": "active-repo",
|
||||
"status": "active",
|
||||
"local_path": "/fallback/active",
|
||||
"host_paths": {"build-host": "/repos/active"},
|
||||
},
|
||||
{
|
||||
"slug": "retired-repo",
|
||||
"status": "retired",
|
||||
"local_path": "/repos/retired",
|
||||
},
|
||||
]
|
||||
target: dict[str, dict[str, Any]] = {}
|
||||
calls: list[tuple[str, str, str]] = []
|
||||
|
||||
def fake_request(
|
||||
base_url: str,
|
||||
path: str,
|
||||
*,
|
||||
method: str = "GET",
|
||||
body: dict[str, Any] | None = None,
|
||||
) -> Any:
|
||||
calls.append((base_url, method, path))
|
||||
if base_url == "source" and path == "/repos/":
|
||||
return source
|
||||
if base_url == "target" and method == "PUT":
|
||||
assert body is not None
|
||||
slug = path.rsplit("/", 1)[1]
|
||||
target[slug] = {"slug": slug, **body}
|
||||
return target[slug]
|
||||
if base_url == "target" and path == "/repositories/":
|
||||
return list(target.values())
|
||||
raise AssertionError((base_url, method, path))
|
||||
|
||||
monkeypatch.setattr(projection, "request_json", fake_request)
|
||||
result = projection.sync_repository_projections(
|
||||
"source", "target", dry_run=False, host_id="build-host"
|
||||
)
|
||||
|
||||
assert result["ok"] is True
|
||||
assert result["source_repo_count"] == 2
|
||||
assert result["counts"] == {
|
||||
"active": 1,
|
||||
"with_checkout_path": 2,
|
||||
"inactive": 1,
|
||||
}
|
||||
assert result["reconciliation"]["matched_repo_count"] == 2
|
||||
assert target["active-repo"]["checkout_path"] == "/repos/active"
|
||||
assert target["retired-repo"]["active"] is False
|
||||
assert all("/sbom/" not in path for _, _, path in calls)
|
||||
|
||||
|
||||
def test_projection_dry_run_does_not_contact_target(monkeypatch) -> None:
|
||||
def fake_request(
|
||||
base_url: str,
|
||||
path: str,
|
||||
*,
|
||||
method: str = "GET",
|
||||
body: dict[str, Any] | None = None,
|
||||
) -> Any:
|
||||
assert base_url == "source"
|
||||
assert method == "GET"
|
||||
assert path == "/repos/"
|
||||
return [{"slug": "demo", "status": "active", "local_path": "/repos/demo"}]
|
||||
|
||||
monkeypatch.setattr(projection, "request_json", fake_request)
|
||||
result = projection.sync_repository_projections(
|
||||
"source", "unreachable-target", dry_run=True, host_id="build-host"
|
||||
)
|
||||
|
||||
assert result["ok"] is True
|
||||
assert result["would_upsert"] == 1
|
||||
assert result["reconciliation"] is None
|
||||
|
|
@ -38,7 +38,7 @@ id: SBOM-WP-0002-T01
|
|||
status: progress
|
||||
priority: high
|
||||
needs_human: true
|
||||
intervention_note: "Warden requires operator OIDC/MFA for first rapp-sbom-nexus Forgejo repository creation and one-time secret provisioning for platform-pg-2/OpenBao."
|
||||
intervention_note: "Warden requires an attended founder act for the first platform-pg-2 OpenBao database connection, SBOM Nexus dynamic roles/policies, and renewable External Secrets parent token. The database owner must also repair the governed backup credential/policy after a live S3 HeadBucket 403."
|
||||
state_hub_task_id: "95a520d4-30c2-5c87-8054-6bfe549c2686"
|
||||
```
|
||||
|
||||
|
|
@ -47,12 +47,15 @@ deploy the API without callers, and capture health plus backup/restore evidence.
|
|||
Image publication, package rendering, family validation, and server-side dry-run
|
||||
are complete; see
|
||||
`docs/evidence/SBOM-WP-0002-T01-dark-deployment-preflight-2026-08-22.md`.
|
||||
The overflow cell is now 1/1 Ready, but continuous archiving and its first base
|
||||
backup fail closed on an S3 `HeadBucket` 403. Database-owner work is tracked by
|
||||
`RAPP-POSTGRES-WP-0005`; no caller or Nexus runtime has been enabled.
|
||||
|
||||
## Synchronize repository projections
|
||||
|
||||
```task
|
||||
id: SBOM-WP-0002-T02
|
||||
status: todo
|
||||
status: progress
|
||||
priority: high
|
||||
state_hub_task_id: "22cbb75f-d82f-5b47-9fef-27bde3b410d5"
|
||||
```
|
||||
|
|
@ -60,6 +63,13 @@ state_hub_task_id: "22cbb75f-d82f-5b47-9fef-27bde3b410d5"
|
|||
Populate active repository identity and host checkout paths from Repo Manager.
|
||||
Verify fleet totals and catch-up ordering without performing ingest.
|
||||
|
||||
The projection-only synchronizer and reconciliation contract are implemented.
|
||||
Its dry-run never contacts the Nexus target and no code path calls an SBOM
|
||||
ingest route. Production apply and catch-up ordering proof wait for the dark
|
||||
runtime from T01. The isolated rehearsal reconciled all 120 source projections
|
||||
and selected exactly the oldest three of 116 active repositories; see
|
||||
`docs/evidence/SBOM-WP-0002-T02-repository-projection-rehearsal-2026-08-22.md`.
|
||||
|
||||
## Import and reconcile State Hub history
|
||||
|
||||
```task
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue