feat: add repository projection synchronization
All checks were successful
Build and Publish Container Image / build-and-push (push) Successful in 1m9s
All checks were successful
Build and Publish Container Image / build-and-push (push) Successful in 1m9s
Assistant: codex Assistant-Model: gpt-5.6-sol Assistant-Session: 01a028f0-a42f-7582-89a8-ebaad7343834
This commit is contained in:
parent
d3258a6ba0
commit
280da08455
7 changed files with 317 additions and 2 deletions
|
|
@ -72,3 +72,26 @@ written to a repository. State Hub handoffs:
|
||||||
|
|
||||||
`T01` remains in progress until those gates are satisfied, the dark workload is
|
`T01` remains in progress until those gates are satisfied, the dark workload is
|
||||||
healthy on PostgreSQL, and a restore is verified against the deployed topology.
|
healthy on PostgreSQL, and a restore is verified against the deployed topology.
|
||||||
|
|
||||||
|
## Post-preflight status
|
||||||
|
|
||||||
|
Later on 2026-08-22, the Forgejo package gate was completed and the package was
|
||||||
|
registered. A namespace-first apply correction passed five package tests and a
|
||||||
|
server-side dry-run, then was published as `rapp-sbom-nexus` commit `dd110d1`.
|
||||||
|
|
||||||
|
The database owner accepted the declaration under `RAPP-POSTGRES-WP-0005`
|
||||||
|
(State Hub workplan `d6d36add-27d9-5b5d-b6fe-ce80eb7b6dab`). The distinct
|
||||||
|
`platform-pg-2` cell and bootstrap secret were created and the cell reached 1/1
|
||||||
|
Ready. Acceptance remains blocked because:
|
||||||
|
|
||||||
|
- Warden still requires the attended founder act for the OpenBao connection,
|
||||||
|
dynamic runtime/migration roles, exact two-path policy, and renewable parent
|
||||||
|
token;
|
||||||
|
- continuous archiving is `False` and the first base backup is
|
||||||
|
`walArchivingFailing`; the value-safe pod log reports S3 `HeadBucket` 403 for
|
||||||
|
the governed `platform-pg-2` prefix.
|
||||||
|
|
||||||
|
The backup diagnosis and request for a successful base backup plus scratch
|
||||||
|
restore were sent to `rapp-postgres` as message
|
||||||
|
`31b72f4b-fcf8-4454-82b7-2dbf325c573e`. The SBOM Nexus workload remains
|
||||||
|
undeployed and no caller ingress is enabled.
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,48 @@
|
||||||
|
# SBOM-WP-0002-T02 repository projection rehearsal — 2026-08-22
|
||||||
|
|
||||||
|
## Result
|
||||||
|
|
||||||
|
The projection-only synchronization path is implemented and rehearsed against
|
||||||
|
an isolated SBOM Nexus database. It does not scan repositories, invoke an
|
||||||
|
ingest route, or create snapshots.
|
||||||
|
|
||||||
|
## Source and selection
|
||||||
|
|
||||||
|
The current Repo Manager compatibility projection was read from State Hub's
|
||||||
|
`GET /repos/` boundary using explicit host id `bnt-lap001`:
|
||||||
|
|
||||||
|
| Measure | Result |
|
||||||
|
| --- | ---: |
|
||||||
|
| Source repositories | 120 |
|
||||||
|
| Active | 116 |
|
||||||
|
| Inactive | 4 |
|
||||||
|
| Selected checkout path | 120 |
|
||||||
|
|
||||||
|
The dry-run used an intentionally unreachable target URL and still completed,
|
||||||
|
proving that dry-run does not contact or mutate the Nexus target.
|
||||||
|
|
||||||
|
## Apply rehearsal
|
||||||
|
|
||||||
|
An ephemeral SQLite-backed Nexus on `127.0.0.1:18010` received the projection:
|
||||||
|
|
||||||
|
- 120 upserts;
|
||||||
|
- 120 exact active/path matches on read-back;
|
||||||
|
- zero missing, mismatched, or extra target repositories;
|
||||||
|
- zero SBOM ingest calls and zero snapshots.
|
||||||
|
|
||||||
|
The bounded catch-up query with `limit=3` returned exactly three of 116 active,
|
||||||
|
never-attempted repositories, ordered deterministically by slug:
|
||||||
|
`activity-core`, `adaptive-pricing`, and `agent-harness`. The response reported
|
||||||
|
`selected_count=3`, `never_count=116`, `stale_count=116`, and
|
||||||
|
`total_count=116`.
|
||||||
|
|
||||||
|
Production apply remains gated on the dark PostgreSQL runtime in
|
||||||
|
`SBOM-WP-0002-T01`.
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
|
||||||
|
- `uv run ruff check src tests scripts`: pass
|
||||||
|
- `uv run pytest -q`: 15 passed, 1 conditional PostgreSQL skip
|
||||||
|
- projection dry-run: 120 would-upsert, target not contacted
|
||||||
|
- projection apply/read-back: 120/120 matched
|
||||||
|
- bounded catch-up: 3/3 selected in deterministic order
|
||||||
|
|
@ -52,6 +52,21 @@ operation.
|
||||||
|
|
||||||
## Import State Hub history
|
## Import State Hub history
|
||||||
|
|
||||||
|
Synchronize the minimum repository identity/path projection first. This command
|
||||||
|
does not call an ingest endpoint or create an SBOM snapshot:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv run python scripts/sync_repository_projections.py \
|
||||||
|
--source-url http://127.0.0.1:8000 \
|
||||||
|
--target-url http://127.0.0.1:8010 \
|
||||||
|
--host-id bnt-lap001 \
|
||||||
|
--dry-run
|
||||||
|
```
|
||||||
|
|
||||||
|
Remove `--dry-run` only after the target is deployed. The apply path upserts all
|
||||||
|
active and inactive records, then reads them back and fails on a missing or
|
||||||
|
mismatched active/path projection. Extra target rows are reported but retained.
|
||||||
|
|
||||||
Run a read-only preview first:
|
Run a read-only preview first:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
|
|
|
||||||
35
scripts/sync_repository_projections.py
Normal file
35
scripts/sync_repository_projections.py
Normal file
|
|
@ -0,0 +1,35 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Synchronize Repo Manager repository projections into SBOM Nexus."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import urllib.error
|
||||||
|
|
||||||
|
from sbom_nexus.projection import sync_repository_projections
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
parser = argparse.ArgumentParser(description=__doc__)
|
||||||
|
parser.add_argument("--source-url", default="http://127.0.0.1:8000")
|
||||||
|
parser.add_argument("--target-url", default="http://127.0.0.1:8010")
|
||||||
|
parser.add_argument("--host-id")
|
||||||
|
parser.add_argument("--dry-run", action="store_true")
|
||||||
|
args = parser.parse_args()
|
||||||
|
try:
|
||||||
|
result = sync_repository_projections(
|
||||||
|
args.source_url,
|
||||||
|
args.target_url,
|
||||||
|
dry_run=args.dry_run,
|
||||||
|
host_id=args.host_id,
|
||||||
|
)
|
||||||
|
except (urllib.error.URLError, json.JSONDecodeError) as exc:
|
||||||
|
raise SystemExit(f"Projection sync failed: {exc}") from exc
|
||||||
|
print(json.dumps(result, indent=2, sort_keys=True))
|
||||||
|
if not result["ok"]:
|
||||||
|
raise SystemExit(1)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
102
src/sbom_nexus/projection.py
Normal file
102
src/sbom_nexus/projection.py
Normal file
|
|
@ -0,0 +1,102 @@
|
||||||
|
"""Synchronize the minimum repository projection used by SBOM Nexus."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import socket
|
||||||
|
import urllib.parse
|
||||||
|
from collections import Counter
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from sbom_nexus.importer import request_json
|
||||||
|
|
||||||
|
|
||||||
|
def _checkout_path(repo: dict[str, Any], host_id: str | None) -> str | None:
|
||||||
|
host_paths = repo.get("host_paths") or {}
|
||||||
|
selected_host = host_id or socket.gethostname()
|
||||||
|
return host_paths.get(selected_host) or repo.get("local_path")
|
||||||
|
|
||||||
|
|
||||||
|
def sync_repository_projections(
|
||||||
|
source_url: str,
|
||||||
|
target_url: str,
|
||||||
|
*,
|
||||||
|
dry_run: bool,
|
||||||
|
host_id: str | None = None,
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
"""Upsert repository identity/path projections without scanning or ingesting."""
|
||||||
|
repositories = request_json(source_url, "/repos/")
|
||||||
|
projections: dict[str, dict[str, Any]] = {}
|
||||||
|
statuses = Counter()
|
||||||
|
for repo in repositories:
|
||||||
|
slug = repo["slug"]
|
||||||
|
if slug in projections:
|
||||||
|
return {
|
||||||
|
"ok": False,
|
||||||
|
"dry_run": dry_run,
|
||||||
|
"source_repo_count": len(repositories),
|
||||||
|
"error": f"duplicate repository slug: {slug}",
|
||||||
|
}
|
||||||
|
active = repo.get("status", "active") == "active"
|
||||||
|
checkout_path = _checkout_path(repo, host_id)
|
||||||
|
projections[slug] = {
|
||||||
|
"slug": slug,
|
||||||
|
"active": active,
|
||||||
|
"checkout_path": checkout_path,
|
||||||
|
}
|
||||||
|
statuses["active" if active else "inactive"] += 1
|
||||||
|
statuses["with_checkout_path" if checkout_path else "without_checkout_path"] += 1
|
||||||
|
|
||||||
|
if dry_run:
|
||||||
|
return {
|
||||||
|
"ok": True,
|
||||||
|
"dry_run": True,
|
||||||
|
"host_id": host_id or socket.gethostname(),
|
||||||
|
"source_repo_count": len(projections),
|
||||||
|
"counts": dict(statuses),
|
||||||
|
"would_upsert": len(projections),
|
||||||
|
"reconciliation": None,
|
||||||
|
}
|
||||||
|
|
||||||
|
for slug, projection in sorted(projections.items()):
|
||||||
|
request_json(
|
||||||
|
target_url,
|
||||||
|
f"/repositories/{urllib.parse.quote(slug, safe='')}",
|
||||||
|
method="PUT",
|
||||||
|
body={
|
||||||
|
"active": projection["active"],
|
||||||
|
"checkout_path": projection["checkout_path"],
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
target_repositories = {
|
||||||
|
repo["slug"]: repo for repo in request_json(target_url, "/repositories/")
|
||||||
|
}
|
||||||
|
missing = sorted(set(projections) - set(target_repositories))
|
||||||
|
mismatched = []
|
||||||
|
for slug in sorted(set(projections) & set(target_repositories)):
|
||||||
|
expected = projections[slug]
|
||||||
|
actual = target_repositories[slug]
|
||||||
|
differences = {
|
||||||
|
field: {"source": expected[field], "target": actual.get(field)}
|
||||||
|
for field in ("active", "checkout_path")
|
||||||
|
if expected[field] != actual.get(field)
|
||||||
|
}
|
||||||
|
if differences:
|
||||||
|
mismatched.append({"repo_slug": slug, "differences": differences})
|
||||||
|
|
||||||
|
reconciliation = {
|
||||||
|
"ok": not missing and not mismatched,
|
||||||
|
"matched_repo_count": len(projections) - len(missing) - len(mismatched),
|
||||||
|
"missing_repo_slugs": missing,
|
||||||
|
"mismatched_repositories": mismatched,
|
||||||
|
"extra_target_repo_slugs": sorted(set(target_repositories) - set(projections)),
|
||||||
|
}
|
||||||
|
return {
|
||||||
|
"ok": reconciliation["ok"],
|
||||||
|
"dry_run": False,
|
||||||
|
"host_id": host_id or socket.gethostname(),
|
||||||
|
"source_repo_count": len(projections),
|
||||||
|
"counts": dict(statuses),
|
||||||
|
"upserted": len(projections),
|
||||||
|
"reconciliation": reconciliation,
|
||||||
|
}
|
||||||
82
tests/test_projection.py
Normal file
82
tests/test_projection.py
Normal file
|
|
@ -0,0 +1,82 @@
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from sbom_nexus import projection
|
||||||
|
|
||||||
|
|
||||||
|
def test_projection_sync_is_projection_only_and_reconciles(monkeypatch) -> None:
|
||||||
|
source = [
|
||||||
|
{
|
||||||
|
"slug": "active-repo",
|
||||||
|
"status": "active",
|
||||||
|
"local_path": "/fallback/active",
|
||||||
|
"host_paths": {"build-host": "/repos/active"},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"slug": "retired-repo",
|
||||||
|
"status": "retired",
|
||||||
|
"local_path": "/repos/retired",
|
||||||
|
},
|
||||||
|
]
|
||||||
|
target: dict[str, dict[str, Any]] = {}
|
||||||
|
calls: list[tuple[str, str, str]] = []
|
||||||
|
|
||||||
|
def fake_request(
|
||||||
|
base_url: str,
|
||||||
|
path: str,
|
||||||
|
*,
|
||||||
|
method: str = "GET",
|
||||||
|
body: dict[str, Any] | None = None,
|
||||||
|
) -> Any:
|
||||||
|
calls.append((base_url, method, path))
|
||||||
|
if base_url == "source" and path == "/repos/":
|
||||||
|
return source
|
||||||
|
if base_url == "target" and method == "PUT":
|
||||||
|
assert body is not None
|
||||||
|
slug = path.rsplit("/", 1)[1]
|
||||||
|
target[slug] = {"slug": slug, **body}
|
||||||
|
return target[slug]
|
||||||
|
if base_url == "target" and path == "/repositories/":
|
||||||
|
return list(target.values())
|
||||||
|
raise AssertionError((base_url, method, path))
|
||||||
|
|
||||||
|
monkeypatch.setattr(projection, "request_json", fake_request)
|
||||||
|
result = projection.sync_repository_projections(
|
||||||
|
"source", "target", dry_run=False, host_id="build-host"
|
||||||
|
)
|
||||||
|
|
||||||
|
assert result["ok"] is True
|
||||||
|
assert result["source_repo_count"] == 2
|
||||||
|
assert result["counts"] == {
|
||||||
|
"active": 1,
|
||||||
|
"with_checkout_path": 2,
|
||||||
|
"inactive": 1,
|
||||||
|
}
|
||||||
|
assert result["reconciliation"]["matched_repo_count"] == 2
|
||||||
|
assert target["active-repo"]["checkout_path"] == "/repos/active"
|
||||||
|
assert target["retired-repo"]["active"] is False
|
||||||
|
assert all("/sbom/" not in path for _, _, path in calls)
|
||||||
|
|
||||||
|
|
||||||
|
def test_projection_dry_run_does_not_contact_target(monkeypatch) -> None:
|
||||||
|
def fake_request(
|
||||||
|
base_url: str,
|
||||||
|
path: str,
|
||||||
|
*,
|
||||||
|
method: str = "GET",
|
||||||
|
body: dict[str, Any] | None = None,
|
||||||
|
) -> Any:
|
||||||
|
assert base_url == "source"
|
||||||
|
assert method == "GET"
|
||||||
|
assert path == "/repos/"
|
||||||
|
return [{"slug": "demo", "status": "active", "local_path": "/repos/demo"}]
|
||||||
|
|
||||||
|
monkeypatch.setattr(projection, "request_json", fake_request)
|
||||||
|
result = projection.sync_repository_projections(
|
||||||
|
"source", "unreachable-target", dry_run=True, host_id="build-host"
|
||||||
|
)
|
||||||
|
|
||||||
|
assert result["ok"] is True
|
||||||
|
assert result["would_upsert"] == 1
|
||||||
|
assert result["reconciliation"] is None
|
||||||
|
|
@ -38,7 +38,7 @@ id: SBOM-WP-0002-T01
|
||||||
status: progress
|
status: progress
|
||||||
priority: high
|
priority: high
|
||||||
needs_human: true
|
needs_human: true
|
||||||
intervention_note: "Warden requires operator OIDC/MFA for first rapp-sbom-nexus Forgejo repository creation and one-time secret provisioning for platform-pg-2/OpenBao."
|
intervention_note: "Warden requires an attended founder act for the first platform-pg-2 OpenBao database connection, SBOM Nexus dynamic roles/policies, and renewable External Secrets parent token. The database owner must also repair the governed backup credential/policy after a live S3 HeadBucket 403."
|
||||||
state_hub_task_id: "95a520d4-30c2-5c87-8054-6bfe549c2686"
|
state_hub_task_id: "95a520d4-30c2-5c87-8054-6bfe549c2686"
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|
@ -47,12 +47,15 @@ deploy the API without callers, and capture health plus backup/restore evidence.
|
||||||
Image publication, package rendering, family validation, and server-side dry-run
|
Image publication, package rendering, family validation, and server-side dry-run
|
||||||
are complete; see
|
are complete; see
|
||||||
`docs/evidence/SBOM-WP-0002-T01-dark-deployment-preflight-2026-08-22.md`.
|
`docs/evidence/SBOM-WP-0002-T01-dark-deployment-preflight-2026-08-22.md`.
|
||||||
|
The overflow cell is now 1/1 Ready, but continuous archiving and its first base
|
||||||
|
backup fail closed on an S3 `HeadBucket` 403. Database-owner work is tracked by
|
||||||
|
`RAPP-POSTGRES-WP-0005`; no caller or Nexus runtime has been enabled.
|
||||||
|
|
||||||
## Synchronize repository projections
|
## Synchronize repository projections
|
||||||
|
|
||||||
```task
|
```task
|
||||||
id: SBOM-WP-0002-T02
|
id: SBOM-WP-0002-T02
|
||||||
status: todo
|
status: progress
|
||||||
priority: high
|
priority: high
|
||||||
state_hub_task_id: "22cbb75f-d82f-5b47-9fef-27bde3b410d5"
|
state_hub_task_id: "22cbb75f-d82f-5b47-9fef-27bde3b410d5"
|
||||||
```
|
```
|
||||||
|
|
@ -60,6 +63,13 @@ state_hub_task_id: "22cbb75f-d82f-5b47-9fef-27bde3b410d5"
|
||||||
Populate active repository identity and host checkout paths from Repo Manager.
|
Populate active repository identity and host checkout paths from Repo Manager.
|
||||||
Verify fleet totals and catch-up ordering without performing ingest.
|
Verify fleet totals and catch-up ordering without performing ingest.
|
||||||
|
|
||||||
|
The projection-only synchronizer and reconciliation contract are implemented.
|
||||||
|
Its dry-run never contacts the Nexus target and no code path calls an SBOM
|
||||||
|
ingest route. Production apply and catch-up ordering proof wait for the dark
|
||||||
|
runtime from T01. The isolated rehearsal reconciled all 120 source projections
|
||||||
|
and selected exactly the oldest three of 116 active repositories; see
|
||||||
|
`docs/evidence/SBOM-WP-0002-T02-repository-projection-rehearsal-2026-08-22.md`.
|
||||||
|
|
||||||
## Import and reconcile State Hub history
|
## Import and reconcile State Hub history
|
||||||
|
|
||||||
```task
|
```task
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue