coulomb/sbom-nexus
4
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat: establish sbom nexus extraction slice

Browse source 
Assistant: codex
Assistant-Model: gpt-5.6-sol
Assistant-Session: 01a028f0-a42f-7582-89a8-ebaad7343834
This commit is contained in:
tegwick 2026-08-22 12:50:08 +02:00
parent 79cd7dff06
commit d61698ea51
31 changed files with 3246 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

65
docs/operator-guide.md Normal file
View file

@ -0,0 +1,65 @@
# Operator guide
## Run locally
```bash
make install
make run
```
The API defaults to `127.0.0.1:8010` and `./sbom-nexus.db`. Set
`SBOM_NEXUS_DATABASE_PATH` to an explicit durable location for non-development
use.
## Register and ingest a repository
```bash
curl -X PUT http://127.0.0.1:8010/repositories/example \
-H 'Content-Type: application/json' \
-d '{"checkout_path":"/srv/repos/example","active":true}'
curl -X POST http://127.0.0.1:8010/sbom/example/ingest
```
The ingest response is terminal: `ingested`, or `skipped` with one of
`no-checkout`, `no-manifest`, or `ingest-error`. A skip advances queue fairness
but does not advance `last_success_at`.
## Inspect catch-up
```bash
curl -s 'http://127.0.0.1:8010/sbom/catch-up?limit=3' | python3 -m json.tool
```
The default stale threshold is 30 days. Override it globally with
`SBOM_NEXUS_STALE_DAYS` or per query with `stale_days` during controlled
operation.
## Import State Hub history
Run a read-only preview first:
```bash
uv run python scripts/import_state_hub.py --dry-run
```
Then run against a backed-up Nexus database:
```bash
uv run python scripts/import_state_hub.py \
--source-url http://127.0.0.1:8000 \
--target-url http://127.0.0.1:8010
```
Imports are idempotent on the State Hub snapshot UUID. Before cutover, compare
the source/target repository, snapshot, and entry counts described in the
extraction review. The current script reports counts but is not yet the complete
production reconciliation gate.
## Current production limitations
- The extraction store is SQLite and intended for local/single-node operation.
- Authentication and authorization are not yet integrated.
- Structured operational metrics, PostgreSQL migrations, backup/restore proof,
and retention policy are required before authority cutover.
- State Hub and Repo Manager callers have not yet been retargeted.

156
docs/state-hub-sbom-extraction-review.md Normal file
View file

@ -0,0 +1,156 @@
# State Hub SBOM extraction review
**Reviewed:** 2026-08-22
**Coordination:** `CUST-WP-0062-T01`, `STATE-WP-0079`, `RMGR-WP-0008`,
`ACTIVITY-WP-0030`
## Outcome
The SBOM capability is a coherent product embedded in State Hub, with a newer
derived-scanner copy in Repo Manager and an already-prepared Activity Core
consumer contract. The target ownership is:
- **SBOM Nexus:** scanning, ingest, snapshots, entries, history, reports,
freshness, catch-up, import/export;
- **Repo Manager:** repository identity, active status, and checkout/path
representation; invoke Nexus rather than retain a second scanner;
- **State Hub:** temporary compatibility façade and repository freshness
projection only;
- **Activity Core:** recurrence and the at-most-N side effect;
- **dashboard/MCP/CLI callers:** retarget to Nexus or the temporary façade.
This corrects a timing overlap in the retirement work: `RMGR-WP-0008` built a
good repository-derived scanner before `CUST-WP-0062` established the dedicated
Nexus. That scanner is the best extraction source, but Repo Manager is not the
durable SBOM product owner.
## Current behavior
State Hub stores an immutable snapshot container and entries keyed to a managed
repository. Manual ingest creates a snapshot, stores entries, and updates
`ManagedRepo.last_sbom_at`. Reads return snapshot history, snapshot detail,
latest entries per repository, a repository view, and a licence grouping with a
simple copyleft-family signal.
Its ingest script recursively recognizes eight source families and posts the
merged result to the API. MCP, the `statehub` CLI, Make targets, onboarding,
repository health, summary caches, and dashboard pages depend on these routes.
Repo Manager subsequently improved the derivation step with source-path and
SHA-256 provenance, Git revision, canonical generation time, structured parser
errors, and direct-production copyleft findings. Its snapshot schema is
`repo-manager.sbom-snapshot.v1`.
Activity Core already contains a disabled/test-double consumer for the desired
Nexus endpoint. It expects `GET /sbom/catch-up?limit=N`, defaults to 3, bounds N
to 1..25, and requires the Nexus to return an ordered `repos` list plus fleet
counts. The old weekly task-flood definition is disabled.
## Artifact and caller disposition
| Existing surface | Location | Disposition | Target/action |
| --- | --- | --- | --- |
| `SBOMSnapshot` model/table | `state-hub/api/models/sbom_snapshot.py`, migrations | move/import | Nexus snapshot store; retain legacy id/timestamp/source provenance |
| `SBOMEntry` model/table and ecosystem enum | `state-hub/api/models/sbom_entry.py`, migrations | move/import | Nexus entries; retain compatibility field names |
| manual ingest | `POST /sbom/ingest/` | move + proxy | Implement compat route in Nexus; State Hub proxies during cutover |
| snapshot list/detail | `GET /sbom/snapshots/*` | move + proxy | Nexus owns history and response contract |
| latest entry filters | `GET /sbom/` | move + proxy | Nexus owns current projections |
| repository view | `GET /sbom/{slug}` | move + proxy | Nexus owns inventory; join/sync minimal repo projection |
| licence report | `GET /sbom/report/licences/` | move + proxy | Nexus owns qualified signal and later SPDX policy integration |
| parser/detection script | `state-hub/scripts/ingest_sbom.py` | move, then retire | Extract improved Repo Manager variant into independent Nexus scanner |
| derived scanner | `repo-manager/src/repo_manager/sbom.py` | move/replace | Make Nexus the implementation; Repo Manager becomes caller/adapter |
| scanner CLI | `rmgr sbom scan`, `licence-report` | retarget | Preserve CLI UX by calling Nexus/local Nexus library during transition |
| State Hub Make target | `make ingest-sbom` | retarget | Invoke Nexus CLI/API; retain temporary alias with deprecation notice |
| `statehub ingest-sbom` | `custodian_cli.py` | retarget | Invoke Nexus, not State Hub-local script |
| tool capture script/prompt | `scripts/capture_sbom_tools.py`, `prompts/sbom-capture-agent.md` | move later | Nexus operator workflow; keep reviewed manifest as repository authority |
| MCP resources | `state://sbom/*` | retarget/proxy | Fetch Nexus; later publish Nexus-native MCP if useful |
| MCP ingest/report tools | `ingest_sbom_tool`, `get_licence_report` | retarget | Call Nexus contracts |
| dashboard SBOM page/data loader | `dashboard/src/sbom.md`, `data/sbom.json.py` | retarget | Read Nexus or façade; no new dashboard in extraction release |
| repo/dashboard coverage cards | repos/overview pages | projection | Consume Nexus freshness projection |
| `ManagedRepo.last_sbom_at`, `sbom_source` | State Hub repo model | strangle/projection | Keep compatible fields, written/synchronized from Nexus during cutover |
| summary cache revision | `api/services/summary_cache.py` | adapt | Revision from Nexus projection/event, not local SBOM tables |
| State summary licence scan | `api/routers/state.py` | adapt | Consume Nexus report/projection |
| DoI criterion C8 | `api/doi_engine.py`, `policies/repo-doi.md` | retarget | Query precise Nexus attempt/success state |
| edge cache `/sbom/` | `api/edge/read_cache.py` | retire/retarget | Cache façade only while compatibility route exists |
| repo registration optional ingest | `scripts/register_project.sh` and onboarding docs | retarget | Register projection then invoke Nexus ingest |
| repository synchronization view | `dashboard/src/repo-sync.md` | retarget | Use Nexus status |
| Activity Core bulk status resolver | old `repo_sbom_status` | retire | Replace with one Nexus catch-up query |
| daily bounded consumer | `activity_core/context_resolvers/sbom_nexus.py` | retain/activate later | Contract is accepted input for Nexus implementation |
| weekly task flood | `weekly-sbom-staleness` | retire | Keep disabled; no one-task-per-stale-repo behavior |
| historical State Hub rows | 22 snapshots/18 repos in RMGR evidence | import + retain | Import history; reconcile counts; delete only after retention gate |
## Extracted source coverage
The first Nexus scanner preserves the proven coverage common to State Hub and
Repo Manager:
| Source | Ecosystem | Important behavior |
| --- | --- | --- |
| `uv.lock` | Python | all locked packages |
| `requirements.txt` | Python | declared lines marked direct |
| `package-lock.json` | Node | package metadata, dev/direct hints, licence when present |
| `yarn.lock` | Node | locked package/version extraction |
| `Cargo.lock` | Rust | all locked packages |
| `go.sum` + `go.mod` | Go | deduplicated modules; direct hint from `go.mod` |
| `.terraform.lock.hcl` | Terraform | provider/version entries |
| `ansible/requirements.y[a]ml` | Ansible | collections and roles under `ansible` directories |
| root `sbom-tools.yaml` | Tool/declared | reviewed escape hatch including licence and direct/dev flags |
Excluded directories remain `.git`, VCS metadata, virtual environments,
`node_modules`, caches, distribution/build outputs, Rust `target`, tox, and
nox environments.
## Contract differences and improvements
1. Nexus records `last_attempt_at` separately from `last_success_at`. State
Hub's single `last_sbom_at` cannot truthfully express a skipped scan.
2. Every attempt has a terminal status. `no-checkout` and `no-manifest` are
successful control-loop outcomes but not successful inventories.
3. Parser errors are stored with the snapshot; they are not silently reduced to
empty entries.
4. Source paths, digests, and Git revision are retained from the Repo Manager
implementation.
5. Copyleft substring matching remains compatibility triage. It is explicitly
not an SPDX expression evaluator or legal conclusion.
6. Catch-up uses last attempt for queue fairness and last success for inventory
truth. Consumers can display both rather than infer one from the other.
## Historical migration requirements
The migration must enumerate State Hub snapshots by repository and oldest
first, fetch each snapshot detail, upsert the repository projection, and insert
the snapshot with its legacy UUID and timestamp. It must be idempotent on the
legacy UUID and compare:
- repository count;
- snapshot count per repository and total;
- entry count per snapshot and total;
- earliest/latest timestamps;
- sampled current-repository views;
- licence group and direct-production copyleft counts.
Fresh repository scans do not replace imported history. They become later
snapshots based on current source evidence.
## Cutover gates
- Nexus production store, migrations, backup, and restore evidence exist.
- Historical import reconciles before State Hub deletion is considered.
- State Hub compatibility tests pass through the façade.
- Repo Manager no longer presents its scanner as independent SBOM authority.
- Dashboard, MCP, CLI, DoI, summary, and onboarding callers are inventoried and
retargeted.
- Activity Core's ingest side effect is integrated but remains bounded by the
resolver's effective limit.
- Two production daily fires record at most N terminal outcomes and zero spawned
per-repository tasks.
- Weekly fleet task-flood behavior remains disabled.
## Review conclusion
Extraction is feasible without changing supported ecosystems. The principal
risk is not parser migration; it is authority and history cutover across three
temporarily overlapping implementations. The first Nexus slice therefore pins
the compatibility and catch-up contracts, while production ownership remains a
separate gated task requiring State Hub and Repo Manager adapter changes.