Harden SBOM retries and align hub evidence
Assistant: codex Assistant-Model: gpt-5.6-sol Assistant-Session: 01a028de-e2c8-7732-8521-46a7fc5db82f
This commit is contained in:
parent
0f573c4378
commit
3b3e1a1ff0
17 changed files with 941 additions and 140 deletions
|
|
@ -15,6 +15,9 @@ NATS_URL=nats://nats:4222
|
||||||
# ── Service integrations (gracefully degraded if unavailable) ─────────────────
|
# ── Service integrations (gracefully degraded if unavailable) ─────────────────
|
||||||
# State Hub — used by the state-hub context adapter. Binds {} on failure.
|
# State Hub — used by the state-hub context adapter. Binds {} on failure.
|
||||||
STATE_HUB_URL=http://127.0.0.1:8000
|
STATE_HUB_URL=http://127.0.0.1:8000
|
||||||
|
# Hub Core canonical extension ports (ACTIVITY-WP-0029). No Core Hub runtime
|
||||||
|
# token or widget mapping is needed for port.events.interaction.
|
||||||
|
HUB_CORE_BASE_URL=http://127.0.0.1:8010
|
||||||
# Consistency sweep engine is repo-manager (ACTIVITY-WP-0029). Default POST
|
# Consistency sweep engine is repo-manager (ACTIVITY-WP-0029). Default POST
|
||||||
# still hits State Hub dual-run. Set REPO_MANAGER_URL or CONSISTENCY_SWEEP_URL
|
# still hits State Hub dual-run. Set REPO_MANAGER_URL or CONSISTENCY_SWEEP_URL
|
||||||
# when repo-manager exposes the HTTP sweep.
|
# when repo-manager exposes the HTTP sweep.
|
||||||
|
|
|
||||||
|
|
@ -32,10 +32,11 @@
|
||||||
| workplan | ACTIVITY-WP-0026 | finished | — | workplans/ACTIVITY-WP-0026-ops-run-claim-queue.md |
|
| workplan | ACTIVITY-WP-0026 | finished | — | workplans/ACTIVITY-WP-0026-ops-run-claim-queue.md |
|
||||||
| workplan | ACTIVITY-WP-0027 | finished | — | workplans/ACTIVITY-WP-0027-llm-connect-and-run-artifacts.md |
|
| workplan | ACTIVITY-WP-0027 | finished | — | workplans/ACTIVITY-WP-0027-llm-connect-and-run-artifacts.md |
|
||||||
| workplan | ACTIVITY-WP-0028 | finished | — | workplans/ACTIVITY-WP-0028-activity-review-cli.md |
|
| workplan | ACTIVITY-WP-0028 | finished | — | workplans/ACTIVITY-WP-0028-activity-review-cli.md |
|
||||||
| workplan | ACTIVITY-WP-0029 | active | — | workplans/ACTIVITY-WP-0029-hub-port-alignment.md |
|
| workplan | ACTIVITY-WP-0029 | finished | — | workplans/ACTIVITY-WP-0029-hub-port-alignment.md |
|
||||||
| workplan | ACTIVITY-WP-0030 | done | — | workplans/ACTIVITY-WP-0030-daily-sbom-catchup.md |
|
| workplan | ACTIVITY-WP-0030 | finished | — | workplans/ACTIVITY-WP-0030-daily-sbom-catchup.md |
|
||||||
| workplan | ACTIVITY-WP-0031 | active | — | workplans/ACTIVITY-WP-0031-production-execution-reliability-cleanup.md |
|
| workplan | ACTIVITY-WP-0031 | active | — | workplans/ACTIVITY-WP-0031-production-execution-reliability-cleanup.md |
|
||||||
| workplan | ACTIVITY-WP-0032 | active | — | workplans/ACTIVITY-WP-0032-glas-profile-execution-contract.md |
|
| workplan | ACTIVITY-WP-0032 | active | — | workplans/ACTIVITY-WP-0032-glas-profile-execution-contract.md |
|
||||||
|
| workplan | ACTIVITY-WP-0033 | finished | — | workplans/ACTIVITY-WP-0033-sbom-catchup-retry-boundary.md |
|
||||||
| workplan | ADHOC-2026-06-01 | finished | — | workplans/ADHOC-2026-06-01.md |
|
| workplan | ADHOC-2026-06-01 | finished | — | workplans/ADHOC-2026-06-01.md |
|
||||||
| workplan | ADHOC-2026-08-20 | finished | — | workplans/ADHOC-2026-08-20.md |
|
| workplan | ADHOC-2026-08-20 | finished | — | workplans/ADHOC-2026-08-20.md |
|
||||||
| workplan | custodian-WP-0001 | done | — | workplans/custodian-WP-0001-temporal-backbone.md |
|
| workplan | custodian-WP-0001 | done | — | workplans/custodian-WP-0001-temporal-backbone.md |
|
||||||
|
|
@ -183,7 +184,7 @@
|
||||||
| task | ACTIVITY-WP-0028-T07 | done | — | workplans/ACTIVITY-WP-0028-activity-review-cli.md |
|
| task | ACTIVITY-WP-0028-T07 | done | — | workplans/ACTIVITY-WP-0028-activity-review-cli.md |
|
||||||
| task | ACTIVITY-WP-0029-T01 | done | — | workplans/ACTIVITY-WP-0029-hub-port-alignment.md |
|
| task | ACTIVITY-WP-0029-T01 | done | — | workplans/ACTIVITY-WP-0029-hub-port-alignment.md |
|
||||||
| task | ACTIVITY-WP-0029-T02 | done | — | workplans/ACTIVITY-WP-0029-hub-port-alignment.md |
|
| task | ACTIVITY-WP-0029-T02 | done | — | workplans/ACTIVITY-WP-0029-hub-port-alignment.md |
|
||||||
| task | ACTIVITY-WP-0029-T03 | wait | — | workplans/ACTIVITY-WP-0029-hub-port-alignment.md |
|
| task | ACTIVITY-WP-0029-T03 | done | — | workplans/ACTIVITY-WP-0029-hub-port-alignment.md |
|
||||||
| task | ACTIVITY-WP-0029-T04 | done | — | workplans/ACTIVITY-WP-0029-hub-port-alignment.md |
|
| task | ACTIVITY-WP-0029-T04 | done | — | workplans/ACTIVITY-WP-0029-hub-port-alignment.md |
|
||||||
| task | ACTIVITY-WP-0030-T01 | done | — | workplans/ACTIVITY-WP-0030-daily-sbom-catchup.md |
|
| task | ACTIVITY-WP-0030-T01 | done | — | workplans/ACTIVITY-WP-0030-daily-sbom-catchup.md |
|
||||||
| task | ACTIVITY-WP-0030-T02 | done | — | workplans/ACTIVITY-WP-0030-daily-sbom-catchup.md |
|
| task | ACTIVITY-WP-0030-T02 | done | — | workplans/ACTIVITY-WP-0030-daily-sbom-catchup.md |
|
||||||
|
|
@ -199,6 +200,9 @@
|
||||||
| task | ACTIVITY-WP-0032-T03 | todo | — | workplans/ACTIVITY-WP-0032-glas-profile-execution-contract.md |
|
| task | ACTIVITY-WP-0032-T03 | todo | — | workplans/ACTIVITY-WP-0032-glas-profile-execution-contract.md |
|
||||||
| task | ACTIVITY-WP-0032-T04 | wait | — | workplans/ACTIVITY-WP-0032-glas-profile-execution-contract.md |
|
| task | ACTIVITY-WP-0032-T04 | wait | — | workplans/ACTIVITY-WP-0032-glas-profile-execution-contract.md |
|
||||||
| task | ACTIVITY-WP-0032-T05 | wait | — | workplans/ACTIVITY-WP-0032-glas-profile-execution-contract.md |
|
| task | ACTIVITY-WP-0032-T05 | wait | — | workplans/ACTIVITY-WP-0032-glas-profile-execution-contract.md |
|
||||||
|
| task | ACTIVITY-WP-0033-T01 | done | — | workplans/ACTIVITY-WP-0033-sbom-catchup-retry-boundary.md |
|
||||||
|
| task | ACTIVITY-WP-0033-T02 | done | — | workplans/ACTIVITY-WP-0033-sbom-catchup-retry-boundary.md |
|
||||||
|
| task | ACTIVITY-WP-0033-T03 | done | — | workplans/ACTIVITY-WP-0033-sbom-catchup-retry-boundary.md |
|
||||||
| task | ADHOC-2026-06-01-T01 | done | — | workplans/ADHOC-2026-06-01.md |
|
| task | ADHOC-2026-06-01-T01 | done | — | workplans/ADHOC-2026-06-01.md |
|
||||||
| task | ADHOC-2026-06-01-T02 | done | — | workplans/ADHOC-2026-06-01.md |
|
| task | ADHOC-2026-06-01-T02 | done | — | workplans/ADHOC-2026-06-01.md |
|
||||||
| task | ADHOC-2026-06-01-T03 | done | — | workplans/ADHOC-2026-06-01.md |
|
| task | ADHOC-2026-06-01-T03 | done | — | workplans/ADHOC-2026-06-01.md |
|
||||||
|
|
|
||||||
|
|
@ -79,15 +79,24 @@ The progress event names the repos selected, updated, and skipped with a
|
||||||
reason (`no-checkout`, `no-manifest`, `ingest-error`), plus the fleet counters
|
reason (`no-checkout`, `no-manifest`, `ingest-error`), plus the fleet counters
|
||||||
so `never_count` can be watched declining day over day.
|
so `never_count` can be watched declining day over day.
|
||||||
|
|
||||||
## Bounded side-effect (ACTIVITY-WP-0030-T02)
|
## Bounded side-effect (ACTIVITY-WP-0030-T02 / ACTIVITY-WP-0033)
|
||||||
|
|
||||||
`params.apply: true` calls sbom-nexus for each of the already-truncated N
|
`params.apply: true` declares the write, but context resolution remains
|
||||||
targets. A target with an explicitly unavailable checkout is recorded through
|
read-only. The workflow records the already-truncated N targets in Temporal
|
||||||
|
history, then a dedicated activity applies exactly that fixed set. Completed
|
||||||
|
per-repository outcomes are heartbeated so an ordinary activity retry resumes
|
||||||
|
after them instead of selecting or writing a second batch.
|
||||||
|
|
||||||
|
A target with an explicitly unavailable checkout is recorded through
|
||||||
`POST /sbom/{slug}/skip` as `no-checkout`; other targets use the terminal ingest
|
`POST /sbom/{slug}/skip` as `no-checkout`; other targets use the terminal ingest
|
||||||
route, which returns `ingested`, `no-manifest`, or `ingest-error`. Transport and
|
route, which returns `ingested`, `no-manifest`, or `ingest-error`. Each request
|
||||||
contract failures are recorded as `ingest-error`. The resulting `updated` and
|
sends a stable per-run/per-repository `Idempotency-Key`. Transport failures,
|
||||||
`skipped` arrays are part of the run context and progress report. There is no
|
timeouts, and malformed responses fail visibly and are never converted into a
|
||||||
task or issue emission.
|
synthetic skip because the remote write may have committed. Nexus must enforce
|
||||||
|
the operation key to close the remaining crash window between a committed POST
|
||||||
|
and Activity Core's next heartbeat. The resulting `updated` and `skipped`
|
||||||
|
arrays are part of the run context and progress report. There is no task or
|
||||||
|
issue emission.
|
||||||
|
|
||||||
## Enable checklist
|
## Enable checklist
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -595,9 +595,15 @@ export FORGEJO_TOKEN=$(bao kv get -field=API_TOKEN platform/workloads/forgejo/fo
|
||||||
fleet-wide `for_each: context.repos.repos` emitted 75 tasks on 2026-08-17 and
|
fleet-wide `for_each: context.repos.repos` emitted 75 tasks on 2026-08-17 and
|
||||||
must not be re-enabled. The bounded replacement is `daily-sbom-catchup`, active
|
must not be re-enabled. The bounded replacement is `daily-sbom-catchup`, active
|
||||||
weekdays at 09:15 Europe/Berlin. It makes one ranked sbom-nexus request,
|
weekdays at 09:15 Europe/Berlin. It makes one ranked sbom-nexus request,
|
||||||
processes at most three terminal ingest/skip outcomes, and emits no tasks. Two
|
records that read-only selection in workflow history, and processes at most
|
||||||
production proof fires on 2026-08-22 processed six distinct repositories as
|
three terminal ingest/skip outcomes in a dedicated heartbeat-enabled activity.
|
||||||
documented `no-checkout` skips with zero task spawns; see
|
Retries retain the same fixed targets and resume acknowledged outcomes. Each
|
||||||
|
write carries a stable per-run/per-repository `Idempotency-Key`; an ambiguous
|
||||||
|
response fails the activity and is not converted into a synthetic skip. Until
|
||||||
|
sbom-nexus enforces that key, a worker crash after a committed POST but before
|
||||||
|
the next heartbeat remains a possible duplicate-write window. The automation
|
||||||
|
emits no tasks. Two production proof fires on 2026-08-22 processed six distinct
|
||||||
|
repositories as documented `no-checkout` skips with zero task spawns; see
|
||||||
`docs/evidence/ACTIVITY-WP-0030-daily-sbom-catchup-2026-08-22.md`.
|
`docs/evidence/ACTIVITY-WP-0030-daily-sbom-catchup-2026-08-22.md`.
|
||||||
|
|
||||||
`weekly-coding-retro` follows the same cron -> context resolver -> per-repo task
|
`weekly-coding-retro` follows the same cron -> context resolver -> per-repo task
|
||||||
|
|
@ -617,8 +623,18 @@ when the workflow completes cleanly twice in a row.
|
||||||
|
|
||||||
## Ops inventory evidence posture
|
## Ops inventory evidence posture
|
||||||
|
|
||||||
The current accepted live backend for activity-core ops inventory probes is
|
Hub-core has been the production runtime authority since 2026-08-21. The
|
||||||
State Hub progress with `event_type=ops_inventory_probe`.
|
Railiance projection of the (still disabled) ops-inventory probe now writes
|
||||||
|
sanitized evidence through canonical `port.events.interaction` using sink type
|
||||||
|
`hub-core-interaction-event`. `HUB_CORE_BASE_URL` targets the in-cluster
|
||||||
|
hub-core runtime. No widget mapping or Core Hub runtime token is required.
|
||||||
|
|
||||||
|
The sink wraps the observation as catalog event `hub.interaction.recorded`,
|
||||||
|
keeps the logical observation name in `payload.reported_event_type`, and reads
|
||||||
|
the `interaction_events` projection after append to verify persistence. The old
|
||||||
|
`core-hub-interaction-event` `/api/v2` sink remains implemented for rollback
|
||||||
|
compatibility during CORE-WP-0010 stabilization, but is no longer the Railiance
|
||||||
|
probe configuration.
|
||||||
|
|
||||||
Inter-Hub / ops-hub per-entity submission remains intentionally deferred until
|
Inter-Hub / ops-hub per-entity submission remains intentionally deferred until
|
||||||
all of these are true:
|
all of these are true:
|
||||||
|
|
|
||||||
|
|
@ -13,7 +13,9 @@ activity-core stays the scheduler even when it is not the engine.
|
||||||
|
|
||||||
Hub-core port names are from `helixforge.hub-extension` 0.1
|
Hub-core port names are from `helixforge.hub-extension` 0.1
|
||||||
(`prj-state-hub-retirement/architecture/hub-extension-contract_v0.1.yaml`).
|
(`prj-state-hub-retirement/architecture/hub-extension-contract_v0.1.yaml`).
|
||||||
HUB-WP-0004 has not shipped those ports yet.
|
HUB-WP-0004 shipped the named ports on 2026-08-21; hub-core became production
|
||||||
|
authority the same day. Core Hub remains a no-writer rollback only through the
|
||||||
|
stabilization window ending no earlier than 2026-08-28.
|
||||||
|
|
||||||
| Caller | Current URL | SHR-INV | Target owner | Port (when named) |
|
| Caller | Current URL | SHR-INV | Target owner | Port (when named) |
|
||||||
| --- | --- | --- | --- | --- |
|
| --- | --- | --- | --- | --- |
|
||||||
|
|
@ -51,11 +53,25 @@ HUB-WP-0004 has not shipped those ports yet.
|
||||||
(dual-run when `RM_RECONCILE` is on). Override with `CONSISTENCY_SWEEP_URL`
|
(dual-run when `RM_RECONCILE` is on). Override with `CONSISTENCY_SWEEP_URL`
|
||||||
or `REPO_MANAGER_URL` + optional `CONSISTENCY_SWEEP_PATH`.
|
or `REPO_MANAGER_URL` + optional `CONSISTENCY_SWEEP_PATH`.
|
||||||
|
|
||||||
## Evidence sink (T03 — wait)
|
## Evidence sink (T03 — aligned 2026-08-22)
|
||||||
|
|
||||||
All `POST /progress/` callers stay on the State Hub beachhead until
|
Ops-inventory evidence now has a canonical `hub-core-interaction-event` sink:
|
||||||
**HUB-WP-0004** publishes `port.events.progress` / `port.events.interaction`.
|
|
||||||
Do not add a second permanent progress host.
|
- `POST /ports/events/interaction` with catalog event
|
||||||
|
`hub.interaction.recorded`;
|
||||||
|
- the sanitized domain observation remains in
|
||||||
|
`payload.reported_event_type` (for example `ops_inventory_probe`);
|
||||||
|
- append verification reads
|
||||||
|
`GET /ports/projections/interaction_events`; and
|
||||||
|
- `HUB_CORE_BASE_URL` points at the surviving hub-core runtime. The Railiance
|
||||||
|
Service retains the compatibility name `core-hub-api` during stabilization,
|
||||||
|
but the process and authority behind it are hub-core.
|
||||||
|
|
||||||
|
The legacy `core-hub-interaction-event` `/api/v2` sink remains available only
|
||||||
|
for rollback compatibility. It is not the configured Railiance probe sink.
|
||||||
|
Unmigrated general progress callers can continue through the State Hub
|
||||||
|
beachhead during the retirement strangler; this task does not create a second
|
||||||
|
permanent progress authority.
|
||||||
|
|
||||||
## Execution (T04)
|
## Execution (T04)
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -29,6 +29,9 @@ data:
|
||||||
CUSTODIAN_REPO_ROOT: /var/custodian
|
CUSTODIAN_REPO_ROOT: /var/custodian
|
||||||
ACTIVITY_CORE_ROOT: /etc/activity-core
|
ACTIVITY_CORE_ROOT: /etc/activity-core
|
||||||
OPS_INVENTORY_PATH: /etc/activity-core/ops/service-inventory.yml
|
OPS_INVENTORY_PATH: /etc/activity-core/ops/service-inventory.yml
|
||||||
|
# Canonical hub-core runtime port; the Service name remains core-hub-api
|
||||||
|
# during the CORE-WP-0010 stabilization/rollback window.
|
||||||
|
HUB_CORE_BASE_URL: http://core-hub-api.core-hub.svc.cluster.local:8010
|
||||||
INTER_HUB_URL: ""
|
INTER_HUB_URL: ""
|
||||||
OPS_HUB_WIDGET_MAPPING: ""
|
OPS_HUB_WIDGET_MAPPING: ""
|
||||||
PROMETHEUS_BIND_ADDR: 0.0.0.0:9090
|
PROMETHEUS_BIND_ADDR: 0.0.0.0:9090
|
||||||
|
|
@ -68,8 +71,10 @@ data:
|
||||||
|
|
||||||
# Daily SBOM Catch-up
|
# Daily SBOM Catch-up
|
||||||
|
|
||||||
Ranked, bounded SBOM catch-up. Each fire processes at most three
|
Ranked, bounded SBOM catch-up. Each fire records one read-only selection,
|
||||||
repositories through sbom-nexus and deliberately contains no task rule.
|
then processes that fixed set of at most three repositories in a dedicated
|
||||||
|
heartbeat-enabled activity. Ambiguous writes fail visibly; stable operation
|
||||||
|
identity is sent to sbom-nexus. This definition contains no task rule.
|
||||||
|
|
||||||
```instruction
|
```instruction
|
||||||
id: daily-sbom-catchup-report
|
id: daily-sbom-catchup-report
|
||||||
|
|
@ -646,9 +651,8 @@ data:
|
||||||
- https
|
- https
|
||||||
allow_network: true
|
allow_network: true
|
||||||
evidence_sinks:
|
evidence_sinks:
|
||||||
- type: state-hub-progress
|
- type: hub-core-interaction-event
|
||||||
event_type: ops_inventory_probe
|
event_type: ops_inventory_probe
|
||||||
author: activity-core
|
|
||||||
bind_to: context.ops_inventory_probe
|
bind_to: context.ops_inventory_probe
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|
@ -656,7 +660,9 @@ data:
|
||||||
|
|
||||||
Disabled Railiance projection of the Custodian-owned definition in
|
Disabled Railiance projection of the Custodian-owned definition in
|
||||||
`/home/worsch/the-custodian/activity-definitions/ops-service-inventory-probes.md`.
|
`/home/worsch/the-custodian/activity-definitions/ops-service-inventory-probes.md`.
|
||||||
Keep disabled until ops-hub Inter-Hub evidence intake is active.
|
Keep disabled until the operator selects the desired probe cadence. Evidence
|
||||||
|
uses hub-core `port.events.interaction`; no widget mapping or runtime secret
|
||||||
|
is required.
|
||||||
fi-daily-research-brief.md: |
|
fi-daily-research-brief.md: |
|
||||||
---
|
---
|
||||||
id: fi-daily-research-brief
|
id: fi-daily-research-brief
|
||||||
|
|
|
||||||
|
|
@ -199,6 +199,81 @@ async def resolve_context(
|
||||||
return snapshot
|
return snapshot
|
||||||
|
|
||||||
|
|
||||||
|
def _sbom_heartbeat_state(run_id: str) -> dict[str, Any]:
|
||||||
|
try:
|
||||||
|
details = activity.info().heartbeat_details
|
||||||
|
except RuntimeError:
|
||||||
|
return {"run_id": run_id, "outcomes_by_bind": {}}
|
||||||
|
if not details or not isinstance(details[0], dict):
|
||||||
|
return {"run_id": run_id, "outcomes_by_bind": {}}
|
||||||
|
state = dict(details[0])
|
||||||
|
if state.get("run_id") != run_id:
|
||||||
|
return {"run_id": run_id, "outcomes_by_bind": {}}
|
||||||
|
if not isinstance(state.get("outcomes_by_bind"), dict):
|
||||||
|
state["outcomes_by_bind"] = {}
|
||||||
|
return state
|
||||||
|
|
||||||
|
|
||||||
|
def _heartbeat_sbom_state(state: dict[str, Any]) -> None:
|
||||||
|
try:
|
||||||
|
activity.heartbeat(state)
|
||||||
|
except RuntimeError:
|
||||||
|
# Direct unit invocation has no Temporal activity context.
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
@activity.defn
|
||||||
|
async def apply_sbom_catchup(payload: dict[str, Any]) -> dict[str, dict[str, Any]]:
|
||||||
|
"""Apply declared SBOM writes to the fixed selection in workflow history."""
|
||||||
|
from activity_core.context_resolvers.sbom_nexus import apply_bounded_ingest
|
||||||
|
|
||||||
|
run_id = str(payload["run_id"])
|
||||||
|
context_sources = payload.get("context_sources") or []
|
||||||
|
context = payload.get("context") or {}
|
||||||
|
heartbeat_state = _sbom_heartbeat_state(run_id)
|
||||||
|
outcomes_by_bind = heartbeat_state["outcomes_by_bind"]
|
||||||
|
patches: dict[str, dict[str, Any]] = {}
|
||||||
|
|
||||||
|
for source in context_sources:
|
||||||
|
if not isinstance(source, dict):
|
||||||
|
continue
|
||||||
|
params = source.get("params") or {}
|
||||||
|
if not (
|
||||||
|
source.get("type") == "sbom-nexus"
|
||||||
|
and source.get("query") == "catch_up"
|
||||||
|
and params.get("apply") is True
|
||||||
|
):
|
||||||
|
continue
|
||||||
|
|
||||||
|
raw_bind = source.get("bind_to") or source.get("name") or "sbom-nexus"
|
||||||
|
bind_key = str(raw_bind).removeprefix("context.")
|
||||||
|
selection = context.get(bind_key)
|
||||||
|
if not isinstance(selection, dict):
|
||||||
|
continue
|
||||||
|
repos = selection.get("repos")
|
||||||
|
if not isinstance(repos, list):
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
limit = int(selection.get("limit", params.get("limit", 3)))
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
limit = 3
|
||||||
|
limit = max(1, min(25, limit))
|
||||||
|
fixed_repos = [repo for repo in repos[:limit] if isinstance(repo, dict)]
|
||||||
|
|
||||||
|
def record_progress(outcomes: list[dict[str, Any]]) -> None:
|
||||||
|
outcomes_by_bind[bind_key] = outcomes
|
||||||
|
_heartbeat_sbom_state(heartbeat_state)
|
||||||
|
|
||||||
|
patches[bind_key] = apply_bounded_ingest(
|
||||||
|
fixed_repos,
|
||||||
|
operation_id=run_id,
|
||||||
|
completed=outcomes_by_bind.get(bind_key),
|
||||||
|
on_progress=record_progress,
|
||||||
|
)
|
||||||
|
|
||||||
|
return patches
|
||||||
|
|
||||||
|
|
||||||
@activity.defn
|
@activity.defn
|
||||||
async def log_run(run_payload: dict) -> str:
|
async def log_run(run_payload: dict) -> str:
|
||||||
"""Persist an ActivityRun record to Postgres and return its run_id.
|
"""Persist an ActivityRun record to Postgres and return its run_id.
|
||||||
|
|
|
||||||
|
|
@ -32,14 +32,10 @@ Ordering is the nexus's responsibility (never-scanned first, then oldest
|
||||||
``last_sbom_at``); this adapter validates the shape and normalises the entries
|
``last_sbom_at``); this adapter validates the shape and normalises the entries
|
||||||
so the deterministic report can render them without comprehensions.
|
so the deterministic report can render them without comprehensions.
|
||||||
|
|
||||||
Until CUST-WP-0062-T03 lands there is no live endpoint — the query is exercised
|
The ranked query is always read-only, including when a definition declares
|
||||||
against a test double (``tests/test_sbom_nexus_context_resolver.py``) and the
|
``params.apply: true``. The workflow records that result in Temporal history,
|
||||||
daily definition stays ``enabled: false``.
|
then a dedicated activity applies the already-truncated fixed target set. This
|
||||||
|
prevents a retry from querying and advancing into a second batch.
|
||||||
With ``params.apply: true`` the adapter performs the declared T02 side-effect:
|
|
||||||
each selected repository receives exactly one terminal ingest or skip outcome.
|
|
||||||
The ranked response is truncated before any write, so the number of processed
|
|
||||||
repositories can never exceed ``limit``. The default remains read-only.
|
|
||||||
|
|
||||||
Config: SBOM_NEXUS_URL env var (default: http://127.0.0.1:8010).
|
Config: SBOM_NEXUS_URL env var (default: http://127.0.0.1:8010).
|
||||||
"""
|
"""
|
||||||
|
|
@ -47,8 +43,10 @@ Config: SBOM_NEXUS_URL env var (default: http://127.0.0.1:8010).
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import os
|
import os
|
||||||
|
from collections.abc import Callable
|
||||||
from typing import Any
|
from typing import Any
|
||||||
from urllib.parse import quote
|
from urllib.parse import quote
|
||||||
|
from uuid import NAMESPACE_URL, uuid5
|
||||||
|
|
||||||
import httpx
|
import httpx
|
||||||
|
|
||||||
|
|
@ -86,10 +84,22 @@ def _fetch_json(path: str, params: dict[str, Any] | None = None) -> Any:
|
||||||
return response.json()
|
return response.json()
|
||||||
|
|
||||||
|
|
||||||
def _post_json(path: str, payload: dict[str, Any] | None = None) -> Any:
|
def _post_json(
|
||||||
|
path: str,
|
||||||
|
payload: dict[str, Any] | None = None,
|
||||||
|
*,
|
||||||
|
idempotency_key: str,
|
||||||
|
) -> Any:
|
||||||
url = f"{_base_url()}{path}"
|
url = f"{_base_url()}{path}"
|
||||||
with httpx.Client(timeout=_TIMEOUT_SECONDS) as client:
|
with httpx.Client(timeout=_TIMEOUT_SECONDS) as client:
|
||||||
response = client.post(url, json=payload)
|
response = client.post(
|
||||||
|
url,
|
||||||
|
json=payload,
|
||||||
|
headers={
|
||||||
|
"Idempotency-Key": idempotency_key,
|
||||||
|
"X-Activity-Core-Operation-ID": idempotency_key,
|
||||||
|
},
|
||||||
|
)
|
||||||
response.raise_for_status()
|
response.raise_for_status()
|
||||||
return response.json()
|
return response.json()
|
||||||
|
|
||||||
|
|
@ -145,10 +155,12 @@ def _catch_up(params: dict[str, Any]) -> dict[str, Any]:
|
||||||
raise RuntimeError("sbom-nexus catch_up response missing required key: repos")
|
raise RuntimeError("sbom-nexus catch_up response missing required key: repos")
|
||||||
|
|
||||||
repos: list[dict[str, Any]] = []
|
repos: list[dict[str, Any]] = []
|
||||||
|
seen_slugs: set[str] = set()
|
||||||
for raw in raw_repos:
|
for raw in raw_repos:
|
||||||
entry = _normalise_entry(raw)
|
entry = _normalise_entry(raw)
|
||||||
if entry is not None:
|
if entry is not None and entry["repo_slug"] not in seen_slugs:
|
||||||
repos.append(entry)
|
repos.append(entry)
|
||||||
|
seen_slugs.add(entry["repo_slug"])
|
||||||
# The nexus owns ranking, but the definition promises "at most N": never let
|
# The nexus owns ranking, but the definition promises "at most N": never let
|
||||||
# an over-long response widen the bounded side-effect in T02.
|
# an over-long response widen the bounded side-effect in T02.
|
||||||
repos = repos[:limit]
|
repos = repos[:limit]
|
||||||
|
|
@ -157,7 +169,7 @@ def _catch_up(params: dict[str, Any]) -> dict[str, Any]:
|
||||||
never_count = _int_or(payload.get("never_count"), 0)
|
never_count = _int_or(payload.get("never_count"), 0)
|
||||||
stale_count = _int_or(payload.get("stale_count"), len(repos))
|
stale_count = _int_or(payload.get("stale_count"), len(repos))
|
||||||
|
|
||||||
result = {
|
return {
|
||||||
"repos": repos,
|
"repos": repos,
|
||||||
"selected_count": len(repos),
|
"selected_count": len(repos),
|
||||||
"stale_count": stale_count,
|
"stale_count": stale_count,
|
||||||
|
|
@ -165,65 +177,135 @@ def _catch_up(params: dict[str, Any]) -> dict[str, Any]:
|
||||||
"total_count": total_count,
|
"total_count": total_count,
|
||||||
"limit": limit,
|
"limit": limit,
|
||||||
}
|
}
|
||||||
if params.get("apply") is True:
|
|
||||||
result.update(_apply_bounded_ingest(repos))
|
|
||||||
return result
|
|
||||||
|
|
||||||
|
|
||||||
def _skip(repo_slug: str, reason: str, detail: str | None = None) -> dict[str, Any]:
|
def _operation_key(operation_id: str, repo_slug: str) -> str:
|
||||||
|
return str(
|
||||||
|
uuid5(
|
||||||
|
NAMESPACE_URL,
|
||||||
|
f"activity-core:sbom-catchup:{operation_id}:{repo_slug}",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _skip(
|
||||||
|
repo_slug: str,
|
||||||
|
reason: str,
|
||||||
|
*,
|
||||||
|
operation_id: str,
|
||||||
|
detail: str | None = None,
|
||||||
|
) -> dict[str, Any]:
|
||||||
payload: dict[str, Any] = {"reason": reason}
|
payload: dict[str, Any] = {"reason": reason}
|
||||||
if detail:
|
if detail:
|
||||||
payload["detail"] = detail[:300]
|
payload["detail"] = detail[:300]
|
||||||
raw = _post_json(f"/sbom/{quote(repo_slug, safe='')}/skip", payload)
|
raw = _post_json(
|
||||||
if not isinstance(raw, dict) or raw.get("status") != "skipped":
|
f"/sbom/{quote(repo_slug, safe='')}/skip",
|
||||||
|
payload,
|
||||||
|
idempotency_key=_operation_key(operation_id, repo_slug),
|
||||||
|
)
|
||||||
|
if (
|
||||||
|
not isinstance(raw, dict)
|
||||||
|
or raw.get("status") != "skipped"
|
||||||
|
or raw.get("reason") not in {"no-checkout", "no-manifest", "ingest-error"}
|
||||||
|
):
|
||||||
raise RuntimeError(f"sbom-nexus skip returned an invalid outcome for {repo_slug}")
|
raise RuntimeError(f"sbom-nexus skip returned an invalid outcome for {repo_slug}")
|
||||||
return raw
|
return raw
|
||||||
|
|
||||||
|
|
||||||
def _ingest(repo_slug: str) -> dict[str, Any]:
|
def _ingest(repo_slug: str, *, operation_id: str) -> dict[str, Any]:
|
||||||
raw = _post_json(f"/sbom/{quote(repo_slug, safe='')}/ingest")
|
raw = _post_json(
|
||||||
if not isinstance(raw, dict) or raw.get("status") not in {"ingested", "skipped"}:
|
f"/sbom/{quote(repo_slug, safe='')}/ingest",
|
||||||
return _skip(repo_slug, "ingest-error", "invalid ingest outcome")
|
idempotency_key=_operation_key(operation_id, repo_slug),
|
||||||
|
)
|
||||||
|
valid = isinstance(raw, dict) and raw.get("status") in {"ingested", "skipped"}
|
||||||
|
if not valid:
|
||||||
|
raise RuntimeError(f"sbom-nexus ingest returned an invalid outcome for {repo_slug}")
|
||||||
|
if raw.get("status") == "skipped" and raw.get("reason") not in {
|
||||||
|
"no-checkout",
|
||||||
|
"no-manifest",
|
||||||
|
"ingest-error",
|
||||||
|
}:
|
||||||
|
raise RuntimeError(f"sbom-nexus ingest returned an invalid skip for {repo_slug}")
|
||||||
return raw
|
return raw
|
||||||
|
|
||||||
|
|
||||||
def _apply_bounded_ingest(repos: list[dict[str, Any]]) -> dict[str, Any]:
|
def _compact_outcome(repo_slug: str, outcome: dict[str, Any]) -> dict[str, Any]:
|
||||||
updated: list[dict[str, Any]] = []
|
compact = {
|
||||||
skipped: list[dict[str, Any]] = []
|
key: outcome.get(key)
|
||||||
|
for key in (
|
||||||
|
"repo_slug",
|
||||||
|
"status",
|
||||||
|
"reason",
|
||||||
|
"snapshot_id",
|
||||||
|
"entry_count",
|
||||||
|
"snapshot_at",
|
||||||
|
"source_revision",
|
||||||
|
)
|
||||||
|
if outcome.get(key) is not None
|
||||||
|
}
|
||||||
|
compact.setdefault("repo_slug", repo_slug)
|
||||||
|
return compact
|
||||||
|
|
||||||
|
|
||||||
|
def apply_bounded_ingest(
|
||||||
|
repos: list[dict[str, Any]],
|
||||||
|
*,
|
||||||
|
operation_id: str,
|
||||||
|
completed: list[dict[str, Any]] | None = None,
|
||||||
|
on_progress: Callable[[list[dict[str, Any]]], None] | None = None,
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
"""Apply one fixed target set, resuming outcomes acknowledged by heartbeat.
|
||||||
|
|
||||||
|
Transport errors and malformed responses are deliberately not converted to
|
||||||
|
synthetic skips. The remote write may have committed, so only Nexus can
|
||||||
|
safely resolve that ambiguity through operation-id enforcement.
|
||||||
|
"""
|
||||||
|
selected: list[dict[str, Any]] = []
|
||||||
|
selected_slugs: set[str] = set()
|
||||||
for repo in repos:
|
for repo in repos:
|
||||||
repo_slug = str(repo["repo_slug"])
|
repo_slug = str(repo["repo_slug"])
|
||||||
try:
|
if repo_slug not in selected_slugs:
|
||||||
if repo.get("checkout_available") is False:
|
selected.append(repo)
|
||||||
outcome = _skip(repo_slug, "no-checkout")
|
selected_slugs.add(repo_slug)
|
||||||
else:
|
|
||||||
outcome = _ingest(repo_slug)
|
|
||||||
except Exception as exc:
|
|
||||||
# A transport or contract failure still needs a terminal Nexus
|
|
||||||
# outcome so the same impossible repository cannot pin the queue.
|
|
||||||
outcome = _skip(repo_slug, "ingest-error", type(exc).__name__)
|
|
||||||
|
|
||||||
compact = {
|
outcomes_by_slug = {
|
||||||
key: outcome.get(key)
|
str(outcome.get("repo_slug")): dict(outcome)
|
||||||
for key in (
|
for outcome in completed or []
|
||||||
"repo_slug",
|
if isinstance(outcome, dict)
|
||||||
"status",
|
and outcome.get("repo_slug") in selected_slugs
|
||||||
"reason",
|
and outcome.get("status") in {"ingested", "skipped"}
|
||||||
"snapshot_id",
|
}
|
||||||
"entry_count",
|
|
||||||
"snapshot_at",
|
for repo in selected:
|
||||||
"source_revision",
|
repo_slug = str(repo["repo_slug"])
|
||||||
|
if repo_slug in outcomes_by_slug:
|
||||||
|
continue
|
||||||
|
if on_progress:
|
||||||
|
on_progress(list(outcomes_by_slug.values()))
|
||||||
|
if repo.get("checkout_available") is False:
|
||||||
|
outcome = _skip(
|
||||||
|
repo_slug,
|
||||||
|
"no-checkout",
|
||||||
|
operation_id=operation_id,
|
||||||
)
|
)
|
||||||
if outcome.get(key) is not None
|
|
||||||
}
|
|
||||||
compact.setdefault("repo_slug", repo_slug)
|
|
||||||
if outcome.get("status") == "ingested":
|
|
||||||
updated.append(compact)
|
|
||||||
else:
|
else:
|
||||||
skipped.append(compact)
|
outcome = _ingest(repo_slug, operation_id=operation_id)
|
||||||
|
outcomes_by_slug[repo_slug] = _compact_outcome(repo_slug, outcome)
|
||||||
|
if on_progress:
|
||||||
|
on_progress(list(outcomes_by_slug.values()))
|
||||||
|
|
||||||
|
ordered_outcomes = [
|
||||||
|
outcomes_by_slug[str(repo["repo_slug"])] for repo in selected
|
||||||
|
]
|
||||||
|
updated = [
|
||||||
|
outcome for outcome in ordered_outcomes if outcome.get("status") == "ingested"
|
||||||
|
]
|
||||||
|
skipped = [
|
||||||
|
outcome for outcome in ordered_outcomes if outcome.get("status") == "skipped"
|
||||||
|
]
|
||||||
|
|
||||||
return {
|
return {
|
||||||
"attempted_count": len(repos),
|
"attempted_count": len(ordered_outcomes),
|
||||||
"updated": updated,
|
"updated": updated,
|
||||||
"skipped": skipped,
|
"skipped": skipped,
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -4,8 +4,10 @@ from __future__ import annotations
|
||||||
|
|
||||||
import json
|
import json
|
||||||
import os
|
import os
|
||||||
|
from datetime import datetime, timezone
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Any
|
from typing import Any
|
||||||
|
from uuid import NAMESPACE_URL, UUID, uuid5
|
||||||
|
|
||||||
import httpx
|
import httpx
|
||||||
|
|
||||||
|
|
@ -22,10 +24,14 @@ _INTER_HUB_SINK_TYPES = {
|
||||||
"inter-hub-event",
|
"inter-hub-event",
|
||||||
"inter-hub-interaction-event",
|
"inter-hub-interaction-event",
|
||||||
}
|
}
|
||||||
_CORE_HUB_SINK_TYPES = {
|
_LEGACY_CORE_HUB_SINK_TYPES = {
|
||||||
"core-hub",
|
"core-hub",
|
||||||
"core-hub-interaction-event",
|
"core-hub-interaction-event",
|
||||||
}
|
}
|
||||||
|
_HUB_CORE_SINK_TYPES = {
|
||||||
|
"hub-core",
|
||||||
|
"hub-core-interaction-event",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
def persist_ops_inventory_evidence(payload: dict[str, Any]) -> list[dict[str, Any]]:
|
def persist_ops_inventory_evidence(payload: dict[str, Any]) -> list[dict[str, Any]]:
|
||||||
|
|
@ -77,7 +83,13 @@ def persist_ops_inventory_evidence(payload: dict[str, Any]) -> list[dict[str, An
|
||||||
results.append(
|
results.append(
|
||||||
_post_state_hub_progress(payload, bind_key, probe_result, sink)
|
_post_state_hub_progress(payload, bind_key, probe_result, sink)
|
||||||
)
|
)
|
||||||
elif sink_type in _CORE_HUB_SINK_TYPES:
|
elif sink_type in _HUB_CORE_SINK_TYPES:
|
||||||
|
results.append(
|
||||||
|
_post_hub_core_interaction_event(
|
||||||
|
payload, bind_key, probe_result, sink
|
||||||
|
)
|
||||||
|
)
|
||||||
|
elif sink_type in _LEGACY_CORE_HUB_SINK_TYPES:
|
||||||
results.append(
|
results.append(
|
||||||
_post_core_hub_interaction_event(
|
_post_core_hub_interaction_event(
|
||||||
payload, bind_key, probe_result, sink
|
payload, bind_key, probe_result, sink
|
||||||
|
|
@ -215,6 +227,131 @@ def _progress_exists(base_url: str, event_type: str, idempotency_key: str) -> bo
|
||||||
return False
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def _post_hub_core_interaction_event(
|
||||||
|
payload: dict[str, Any],
|
||||||
|
context_key: str,
|
||||||
|
probe_result: dict[str, Any],
|
||||||
|
sink: dict[str, Any],
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
"""Append sanitized evidence through hub-core's canonical interaction port."""
|
||||||
|
raw_base_url = (
|
||||||
|
sink.get("hub_core_url")
|
||||||
|
or sink.get("base_url")
|
||||||
|
or os.environ.get("HUB_CORE_BASE_URL")
|
||||||
|
or ""
|
||||||
|
)
|
||||||
|
base_url = str(raw_base_url).rstrip("/")
|
||||||
|
if not base_url:
|
||||||
|
return {
|
||||||
|
"type": sink.get("type"),
|
||||||
|
"status": "skipped",
|
||||||
|
"reason": "missing_hub_core_config",
|
||||||
|
"missing": ["HUB_CORE_BASE_URL"],
|
||||||
|
"context_key": context_key,
|
||||||
|
}
|
||||||
|
|
||||||
|
endpoint = _selected_endpoint(probe_result, sink)
|
||||||
|
correlation_id = _hub_core_correlation_id(payload, context_key)
|
||||||
|
reported_event_type = str(sink.get("event_type", "ops-endpoint-verified"))
|
||||||
|
body = {
|
||||||
|
"schema_version": str(sink.get("schema_version", "0.1.0")),
|
||||||
|
"correlation_id": correlation_id,
|
||||||
|
# The port catalog separates the interaction envelope from the
|
||||||
|
# domain-specific observation carried in its payload.
|
||||||
|
"event_type": "hub.interaction.recorded",
|
||||||
|
"occurred_at": _hub_core_occurred_at(payload, probe_result),
|
||||||
|
"subject_refs": _hub_core_subject_refs(payload, context_key, endpoint),
|
||||||
|
"payload": {
|
||||||
|
"reported_event_type": reported_event_type,
|
||||||
|
"view_context": _core_hub_view_context(
|
||||||
|
payload, context_key, endpoint, sink
|
||||||
|
),
|
||||||
|
**_core_hub_metadata(
|
||||||
|
payload, context_key, probe_result, endpoint
|
||||||
|
),
|
||||||
|
},
|
||||||
|
}
|
||||||
|
timeout = float(sink.get("timeout_seconds", 10.0))
|
||||||
|
headers = {
|
||||||
|
"Accept": "application/json",
|
||||||
|
"Content-Type": "application/json",
|
||||||
|
"User-Agent": "activity-core-ops-evidence/0.2",
|
||||||
|
}
|
||||||
|
resp = httpx.post(
|
||||||
|
f"{base_url}/ports/events/interaction",
|
||||||
|
json=body,
|
||||||
|
headers=headers,
|
||||||
|
timeout=timeout,
|
||||||
|
)
|
||||||
|
resp.raise_for_status()
|
||||||
|
data = resp.json()
|
||||||
|
event_id = data.get("id")
|
||||||
|
if not event_id:
|
||||||
|
raise RuntimeError("Hub Core interaction port response did not include an id")
|
||||||
|
if not _hub_core_event_exists(base_url, str(event_id), timeout, headers):
|
||||||
|
raise RuntimeError("Hub Core interaction event was not visible after append")
|
||||||
|
|
||||||
|
return {
|
||||||
|
"type": sink.get("type"),
|
||||||
|
"status": "posted",
|
||||||
|
"event_type": "hub.interaction.recorded",
|
||||||
|
"reported_event_type": reported_event_type,
|
||||||
|
"event_id": event_id,
|
||||||
|
"correlation_id": data.get("correlation_id", correlation_id),
|
||||||
|
"verified": True,
|
||||||
|
"context_key": context_key,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _hub_core_correlation_id(payload: dict[str, Any], context_key: str) -> str:
|
||||||
|
raw = str(payload.get("run_id") or "")
|
||||||
|
try:
|
||||||
|
return str(UUID(raw))
|
||||||
|
except ValueError:
|
||||||
|
return str(uuid5(NAMESPACE_URL, f"activity-core:{raw}:{context_key}"))
|
||||||
|
|
||||||
|
|
||||||
|
def _hub_core_occurred_at(
|
||||||
|
payload: dict[str, Any], probe_result: dict[str, Any]
|
||||||
|
) -> str:
|
||||||
|
value = payload.get("scheduled_for") or probe_result.get("generated_at")
|
||||||
|
if isinstance(value, str) and value:
|
||||||
|
return value
|
||||||
|
return datetime.now(timezone.utc).isoformat()
|
||||||
|
|
||||||
|
|
||||||
|
def _hub_core_subject_refs(
|
||||||
|
payload: dict[str, Any], context_key: str, endpoint: dict[str, Any]
|
||||||
|
) -> dict[str, str]:
|
||||||
|
refs = {
|
||||||
|
"activity": payload.get("activity_id"),
|
||||||
|
"activity_run": payload.get("run_id"),
|
||||||
|
"context": context_key,
|
||||||
|
"endpoint": endpoint.get("endpoint_id"),
|
||||||
|
}
|
||||||
|
return {key: str(value) for key, value in refs.items() if value}
|
||||||
|
|
||||||
|
|
||||||
|
def _hub_core_event_exists(
|
||||||
|
base_url: str,
|
||||||
|
event_id: str,
|
||||||
|
timeout: float,
|
||||||
|
headers: dict[str, str],
|
||||||
|
) -> bool:
|
||||||
|
resp = httpx.get(
|
||||||
|
f"{base_url}/ports/projections/interaction_events",
|
||||||
|
headers=headers,
|
||||||
|
timeout=timeout,
|
||||||
|
)
|
||||||
|
resp.raise_for_status()
|
||||||
|
body = resp.json()
|
||||||
|
data = body.get("data") if isinstance(body, dict) else None
|
||||||
|
items = data.get("items") if isinstance(data, dict) else None
|
||||||
|
if not isinstance(items, list):
|
||||||
|
return False
|
||||||
|
return any(isinstance(item, dict) and item.get("id") == event_id for item in items)
|
||||||
|
|
||||||
|
|
||||||
def _post_core_hub_interaction_event(
|
def _post_core_hub_interaction_event(
|
||||||
payload: dict[str, Any],
|
payload: dict[str, Any],
|
||||||
context_key: str,
|
context_key: str,
|
||||||
|
|
|
||||||
|
|
@ -33,6 +33,7 @@ from temporalio.runtime import PrometheusConfig, Runtime, TelemetryConfig
|
||||||
from temporalio.worker import Worker
|
from temporalio.worker import Worker
|
||||||
|
|
||||||
from activity_core.activities import (
|
from activity_core.activities import (
|
||||||
|
apply_sbom_catchup,
|
||||||
emit_tasks,
|
emit_tasks,
|
||||||
evaluate_instructions,
|
evaluate_instructions,
|
||||||
evaluate_rules,
|
evaluate_rules,
|
||||||
|
|
@ -104,6 +105,7 @@ async def run() -> None:
|
||||||
activities=[
|
activities=[
|
||||||
load_activity_definition,
|
load_activity_definition,
|
||||||
resolve_context,
|
resolve_context,
|
||||||
|
apply_sbom_catchup,
|
||||||
log_run,
|
log_run,
|
||||||
evaluate_rules,
|
evaluate_rules,
|
||||||
evaluate_instructions,
|
evaluate_instructions,
|
||||||
|
|
|
||||||
|
|
@ -20,6 +20,7 @@ from temporalio.common import RetryPolicy, SearchAttributeKey, TypedSearchAttrib
|
||||||
|
|
||||||
with workflow.unsafe.imports_passed_through():
|
with workflow.unsafe.imports_passed_through():
|
||||||
from activity_core.activities import (
|
from activity_core.activities import (
|
||||||
|
apply_sbom_catchup,
|
||||||
emit_tasks,
|
emit_tasks,
|
||||||
evaluate_rules,
|
evaluate_rules,
|
||||||
evaluate_instructions,
|
evaluate_instructions,
|
||||||
|
|
@ -57,10 +58,10 @@ class RunActivityWorkflow:
|
||||||
|
|
||||||
Sequence:
|
Sequence:
|
||||||
1. load_activity_definition(activity_id) → defn dict
|
1. load_activity_definition(activity_id) → defn dict
|
||||||
2. resolve_context(defn.context_sources) → context snapshot
|
2. resolve_context(defn.context_sources) → read-only context snapshot
|
||||||
3. evaluate_rules(rules, event, context) → matching rules → TaskSpec dicts
|
3. apply_sbom_catchup(fixed selection) → bounded outcome patch
|
||||||
4. emit_tasks(task_specs) → TaskRef list via IssueSink
|
4. evaluate rules/instructions → TaskSpec dicts and reports
|
||||||
5. log_run(...) → activity_runs row
|
5. log run, then emit tasks
|
||||||
"""
|
"""
|
||||||
|
|
||||||
@workflow.run
|
@workflow.run
|
||||||
|
|
@ -99,7 +100,13 @@ class RunActivityWorkflow:
|
||||||
])
|
])
|
||||||
)
|
)
|
||||||
|
|
||||||
# ── 2. Resolve context ────────────────────────────────────────────────
|
if trigger_key == SCHEDULED_TRIGGER_KEY:
|
||||||
|
dedup_source = workflow.info().workflow_id
|
||||||
|
else:
|
||||||
|
dedup_source = f"{activity_id}:{trigger_key}"
|
||||||
|
run_id = str(uuid.uuid5(uuid.NAMESPACE_URL, dedup_source))
|
||||||
|
|
||||||
|
# ── 2. Resolve context (read-only) ────────────────────────────────────
|
||||||
context_snapshot: dict = await workflow.execute_activity(
|
context_snapshot: dict = await workflow.execute_activity(
|
||||||
resolve_context,
|
resolve_context,
|
||||||
args=[defn["context_sources"], event_envelope_json],
|
args=[defn["context_sources"], event_envelope_json],
|
||||||
|
|
@ -107,11 +114,29 @@ class RunActivityWorkflow:
|
||||||
retry_policy=_RETRY_POLICY,
|
retry_policy=_RETRY_POLICY,
|
||||||
)
|
)
|
||||||
|
|
||||||
if trigger_key == SCHEDULED_TRIGGER_KEY:
|
# ── 3. Apply declared bounded side-effects to the fixed selection ────
|
||||||
dedup_source = workflow.info().workflow_id
|
if any(
|
||||||
else:
|
isinstance(source, dict)
|
||||||
dedup_source = f"{activity_id}:{trigger_key}"
|
and source.get("type") == "sbom-nexus"
|
||||||
run_id = str(uuid.uuid5(uuid.NAMESPACE_URL, dedup_source))
|
and source.get("query") == "catch_up"
|
||||||
|
and (source.get("params") or {}).get("apply") is True
|
||||||
|
for source in defn.get("context_sources", [])
|
||||||
|
) and workflow.patched("activity-wp-0033-sbom-retry-boundary"):
|
||||||
|
context_patches: dict = await workflow.execute_activity(
|
||||||
|
apply_sbom_catchup,
|
||||||
|
{
|
||||||
|
"context_sources": defn.get("context_sources", []),
|
||||||
|
"context": context_snapshot,
|
||||||
|
"run_id": run_id,
|
||||||
|
},
|
||||||
|
start_to_close_timeout=_ACTIVITY_TIMEOUT,
|
||||||
|
heartbeat_timeout=timedelta(seconds=60),
|
||||||
|
retry_policy=_RETRY_POLICY,
|
||||||
|
)
|
||||||
|
for bind_key, patch in context_patches.items():
|
||||||
|
current = context_snapshot.get(bind_key)
|
||||||
|
if isinstance(current, dict) and isinstance(patch, dict):
|
||||||
|
current.update(patch)
|
||||||
|
|
||||||
await workflow.execute_activity(
|
await workflow.execute_activity(
|
||||||
persist_ops_evidence,
|
persist_ops_evidence,
|
||||||
|
|
@ -127,7 +152,7 @@ class RunActivityWorkflow:
|
||||||
retry_policy=_RETRY_POLICY,
|
retry_policy=_RETRY_POLICY,
|
||||||
)
|
)
|
||||||
|
|
||||||
# ── 3. Evaluate rules ─────────────────────────────────────────────────
|
# ── 4. Evaluate rules ─────────────────────────────────────────────────
|
||||||
import json as _json
|
import json as _json
|
||||||
event_attrs: dict = {}
|
event_attrs: dict = {}
|
||||||
if event_envelope_json:
|
if event_envelope_json:
|
||||||
|
|
@ -162,7 +187,7 @@ class RunActivityWorkflow:
|
||||||
task_spec_dicts.extend(instruction_result.get("task_specs", []))
|
task_spec_dicts.extend(instruction_result.get("task_specs", []))
|
||||||
report_dicts.extend(instruction_result.get("reports", []))
|
report_dicts.extend(instruction_result.get("reports", []))
|
||||||
|
|
||||||
# ── 4. Persist reports ────────────────────────────────────────────────
|
# ── 5. Persist reports ────────────────────────────────────────────────
|
||||||
if report_dicts:
|
if report_dicts:
|
||||||
await workflow.execute_activity(
|
await workflow.execute_activity(
|
||||||
persist_instruction_reports,
|
persist_instruction_reports,
|
||||||
|
|
@ -177,7 +202,7 @@ class RunActivityWorkflow:
|
||||||
retry_policy=_RETRY_POLICY,
|
retry_policy=_RETRY_POLICY,
|
||||||
)
|
)
|
||||||
|
|
||||||
# ── 5. Log the run BEFORE emit ────────────────────────────────────────
|
# ── 6. Log the run BEFORE emit ────────────────────────────────────────
|
||||||
# ACTIVITY-WP-0021: emit_tasks sink failures used to abort the workflow
|
# ACTIVITY-WP-0021: emit_tasks sink failures used to abort the workflow
|
||||||
# before log_run, so failed Binky/SBOM fires left no activity_runs row
|
# before log_run, so failed Binky/SBOM fires left no activity_runs row
|
||||||
# and automation-status could not observe them. Always record the run;
|
# and automation-status could not observe them. Always record the run;
|
||||||
|
|
@ -196,7 +221,7 @@ class RunActivityWorkflow:
|
||||||
retry_policy=_RETRY_POLICY,
|
retry_policy=_RETRY_POLICY,
|
||||||
)
|
)
|
||||||
|
|
||||||
# ── 6. Emit tasks (may fail independently of run audit) ───────────────
|
# ── 7. Emit tasks (may fail independently of run audit) ───────────────
|
||||||
if task_spec_dicts:
|
if task_spec_dicts:
|
||||||
# Cron schedules pass trigger_key="scheduled" for *every* fire.
|
# Cron schedules pass trigger_key="scheduled" for *every* fire.
|
||||||
# ops_run idempotency is {def}:{source}:{triggering_event_id}, so
|
# ops_run idempotency is {def}:{source}:{triggering_event_id}, so
|
||||||
|
|
|
||||||
|
|
@ -367,6 +367,90 @@ def test_core_hub_interaction_event_sink_posts_and_verifies_compact_event(monkey
|
||||||
assert "token=secret" not in serialized
|
assert "token=secret" not in serialized
|
||||||
|
|
||||||
|
|
||||||
|
def test_hub_core_interaction_port_posts_and_verifies_sanitized_event(monkeypatch) -> None:
|
||||||
|
posts: list[dict[str, Any]] = []
|
||||||
|
|
||||||
|
def fake_post(url: str, **kwargs: Any) -> DummyResponse:
|
||||||
|
assert url == "http://hub-core.test/ports/events/interaction"
|
||||||
|
assert "Authorization" not in kwargs["headers"]
|
||||||
|
posts.append({"url": url, **kwargs})
|
||||||
|
return DummyResponse(
|
||||||
|
{
|
||||||
|
"id": "event-port-1",
|
||||||
|
"status": "accepted",
|
||||||
|
"correlation_id": _run_id(),
|
||||||
|
},
|
||||||
|
status_code=202,
|
||||||
|
)
|
||||||
|
|
||||||
|
def fake_get(url: str, **kwargs: Any) -> DummyResponse:
|
||||||
|
assert url == "http://hub-core.test/ports/projections/interaction_events"
|
||||||
|
return DummyResponse(
|
||||||
|
{
|
||||||
|
"id": "interaction_events",
|
||||||
|
"data": {"items": [{"id": "event-port-1"}]},
|
||||||
|
"provenance": {},
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
monkeypatch.setattr(httpx, "post", fake_post)
|
||||||
|
monkeypatch.setattr(httpx, "get", fake_get)
|
||||||
|
|
||||||
|
result = persist_ops_inventory_evidence(
|
||||||
|
_payload([
|
||||||
|
{
|
||||||
|
"type": "hub-core-interaction-event",
|
||||||
|
"hub_core_url": "http://hub-core.test",
|
||||||
|
"event_type": "ops-endpoint-verified",
|
||||||
|
}
|
||||||
|
])
|
||||||
|
)
|
||||||
|
|
||||||
|
assert result == [
|
||||||
|
{
|
||||||
|
"type": "hub-core-interaction-event",
|
||||||
|
"status": "posted",
|
||||||
|
"event_type": "hub.interaction.recorded",
|
||||||
|
"reported_event_type": "ops-endpoint-verified",
|
||||||
|
"event_id": "event-port-1",
|
||||||
|
"correlation_id": _run_id(),
|
||||||
|
"verified": True,
|
||||||
|
"context_key": "ops_probe",
|
||||||
|
}
|
||||||
|
]
|
||||||
|
body = posts[0]["json"]
|
||||||
|
assert body["schema_version"] == "0.1.0"
|
||||||
|
assert body["correlation_id"] == _run_id()
|
||||||
|
assert body["event_type"] == "hub.interaction.recorded"
|
||||||
|
assert body["payload"]["reported_event_type"] == "ops-endpoint-verified"
|
||||||
|
assert body["payload"]["endpoint"]["url"] == "http://state-hub.test/health"
|
||||||
|
assert body["subject_refs"]["endpoint"] == "state-hub-health"
|
||||||
|
|
||||||
|
serialized = json.dumps(body, sort_keys=True)
|
||||||
|
assert "secret response body" not in serialized
|
||||||
|
assert "Authorization" not in serialized
|
||||||
|
assert "user:pass" not in serialized
|
||||||
|
assert "token=secret" not in serialized
|
||||||
|
|
||||||
|
|
||||||
|
def test_hub_core_interaction_port_skips_when_base_url_missing(monkeypatch) -> None:
|
||||||
|
monkeypatch.delenv("HUB_CORE_BASE_URL", raising=False)
|
||||||
|
|
||||||
|
result = persist_ops_inventory_evidence(
|
||||||
|
_payload([{"type": "hub-core-interaction-event"}])
|
||||||
|
)
|
||||||
|
|
||||||
|
assert result == [
|
||||||
|
{
|
||||||
|
"type": "hub-core-interaction-event",
|
||||||
|
"status": "skipped",
|
||||||
|
"reason": "missing_hub_core_config",
|
||||||
|
"missing": ["HUB_CORE_BASE_URL"],
|
||||||
|
"context_key": "ops_probe",
|
||||||
|
}
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
def test_core_hub_sink_skips_cleanly_when_config_missing(monkeypatch) -> None:
|
def test_core_hub_sink_skips_cleanly_when_config_missing(monkeypatch) -> None:
|
||||||
monkeypatch.delenv("CORE_HUB_BASE_URL", raising=False)
|
monkeypatch.delenv("CORE_HUB_BASE_URL", raising=False)
|
||||||
monkeypatch.delenv("CORE_HUB_RUNTIME_TOKEN", raising=False)
|
monkeypatch.delenv("CORE_HUB_RUNTIME_TOKEN", raising=False)
|
||||||
|
|
|
||||||
|
|
@ -41,6 +41,9 @@ def test_runtime_config_has_ops_inventory_placeholders() -> None:
|
||||||
assert config["data"]["OPS_INVENTORY_PATH"] == (
|
assert config["data"]["OPS_INVENTORY_PATH"] == (
|
||||||
"/etc/activity-core/ops/service-inventory.yml"
|
"/etc/activity-core/ops/service-inventory.yml"
|
||||||
)
|
)
|
||||||
|
assert config["data"]["HUB_CORE_BASE_URL"] == (
|
||||||
|
"http://core-hub-api.core-hub.svc.cluster.local:8010"
|
||||||
|
)
|
||||||
assert config["data"]["INTER_HUB_URL"] == ""
|
assert config["data"]["INTER_HUB_URL"] == ""
|
||||||
assert config["data"]["OPS_HUB_WIDGET_MAPPING"] == ""
|
assert config["data"]["OPS_HUB_WIDGET_MAPPING"] == ""
|
||||||
|
|
||||||
|
|
@ -68,9 +71,8 @@ def test_external_configmap_projects_disabled_ops_probe_definition(tmp_path) ->
|
||||||
"allow_network": True,
|
"allow_network": True,
|
||||||
"evidence_sinks": [
|
"evidence_sinks": [
|
||||||
{
|
{
|
||||||
"type": "state-hub-progress",
|
"type": "hub-core-interaction-event",
|
||||||
"event_type": "ops_inventory_probe",
|
"event_type": "ops_inventory_probe",
|
||||||
"author": "activity-core",
|
|
||||||
}
|
}
|
||||||
],
|
],
|
||||||
},
|
},
|
||||||
|
|
@ -266,12 +268,23 @@ def test_disabled_ops_probe_definition_can_emit_fixture_evidence(
|
||||||
posts: list[dict[str, Any]] = []
|
posts: list[dict[str, Any]] = []
|
||||||
|
|
||||||
def fake_progress_get(url: str, **kwargs: Any) -> _JsonResponse:
|
def fake_progress_get(url: str, **kwargs: Any) -> _JsonResponse:
|
||||||
return _JsonResponse([])
|
assert url.endswith("/ports/projections/interaction_events")
|
||||||
|
return _JsonResponse({"data": {"items": [{"id": "event-1"}]}})
|
||||||
|
|
||||||
def fake_progress_post(url: str, **kwargs: Any) -> _JsonResponse:
|
def fake_progress_post(url: str, **kwargs: Any) -> _JsonResponse:
|
||||||
|
assert url.endswith("/ports/events/interaction")
|
||||||
posts.append({"url": url, **kwargs})
|
posts.append({"url": url, **kwargs})
|
||||||
return _JsonResponse({"id": "progress-1"})
|
return _JsonResponse(
|
||||||
|
{
|
||||||
|
"id": "event-1",
|
||||||
|
"status": "accepted",
|
||||||
|
"correlation_id": "12345678-aaaa-bbbb-cccc-123456789abc",
|
||||||
|
},
|
||||||
|
status_code=202,
|
||||||
|
)
|
||||||
|
|
||||||
|
runtime_config = _by_kind_name("ConfigMap", "actcore-runtime-config")
|
||||||
|
monkeypatch.setenv("HUB_CORE_BASE_URL", runtime_config["data"]["HUB_CORE_BASE_URL"])
|
||||||
monkeypatch.setattr(httpx, "get", fake_progress_get)
|
monkeypatch.setattr(httpx, "get", fake_progress_get)
|
||||||
monkeypatch.setattr(httpx, "post", fake_progress_post)
|
monkeypatch.setattr(httpx, "post", fake_progress_post)
|
||||||
|
|
||||||
|
|
@ -288,8 +301,9 @@ def test_disabled_ops_probe_definition_can_emit_fixture_evidence(
|
||||||
|
|
||||||
assert definition.enabled is False
|
assert definition.enabled is False
|
||||||
assert result[0]["status"] == "posted"
|
assert result[0]["status"] == "posted"
|
||||||
assert posts[0]["json"]["event_type"] == "ops_inventory_probe"
|
assert posts[0]["json"]["event_type"] == "hub.interaction.recorded"
|
||||||
assert posts[0]["json"]["detail"]["probe"]["summary"]["ok"] == 4
|
assert posts[0]["json"]["payload"]["reported_event_type"] == "ops_inventory_probe"
|
||||||
|
assert posts[0]["json"]["payload"]["probe"]["summary"]["ok"] == 4
|
||||||
|
|
||||||
|
|
||||||
class _HttpResponse:
|
class _HttpResponse:
|
||||||
|
|
|
||||||
|
|
@ -8,7 +8,10 @@ import httpx
|
||||||
import pytest
|
import pytest
|
||||||
|
|
||||||
from activity_core.context_resolvers.base import CONTEXT_RESOLVER_REGISTRY
|
from activity_core.context_resolvers.base import CONTEXT_RESOLVER_REGISTRY
|
||||||
from activity_core.context_resolvers.sbom_nexus import SbomNexusContextResolver
|
from activity_core.context_resolvers.sbom_nexus import (
|
||||||
|
SbomNexusContextResolver,
|
||||||
|
apply_bounded_ingest,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
class DummyResponse:
|
class DummyResponse:
|
||||||
|
|
@ -36,7 +39,7 @@ class DummyClient:
|
||||||
self._payload = payload
|
self._payload = payload
|
||||||
self._status_code = status_code
|
self._status_code = status_code
|
||||||
self.calls: list[tuple[str, dict[str, Any] | None]] = []
|
self.calls: list[tuple[str, dict[str, Any] | None]] = []
|
||||||
self.posts: list[tuple[str, dict[str, Any] | None]] = []
|
self.posts: list[tuple[str, dict[str, Any] | None, dict[str, str]]] = []
|
||||||
self.post_results: dict[str, list[DummyResponse]] = {}
|
self.post_results: dict[str, list[DummyResponse]] = {}
|
||||||
|
|
||||||
def __enter__(self) -> "DummyClient":
|
def __enter__(self) -> "DummyClient":
|
||||||
|
|
@ -49,8 +52,13 @@ class DummyClient:
|
||||||
self.calls.append((url, params))
|
self.calls.append((url, params))
|
||||||
return DummyResponse(self._payload, self._status_code)
|
return DummyResponse(self._payload, self._status_code)
|
||||||
|
|
||||||
def post(self, url: str, json: dict[str, Any] | None = None) -> DummyResponse:
|
def post(
|
||||||
self.posts.append((url, json))
|
self,
|
||||||
|
url: str,
|
||||||
|
json: dict[str, Any] | None = None,
|
||||||
|
headers: dict[str, str] | None = None,
|
||||||
|
) -> DummyResponse:
|
||||||
|
self.posts.append((url, json, headers or {}))
|
||||||
for suffix, results in self.post_results.items():
|
for suffix, results in self.post_results.items():
|
||||||
if url.endswith(suffix) and results:
|
if url.endswith(suffix) and results:
|
||||||
return results.pop(0)
|
return results.pop(0)
|
||||||
|
|
@ -160,7 +168,7 @@ def test_catch_up_truncates_an_over_long_response(monkeypatch) -> None:
|
||||||
assert result["selected_count"] == 2
|
assert result["selected_count"] == 2
|
||||||
|
|
||||||
|
|
||||||
def test_apply_processes_at_most_limit_and_records_terminal_outcomes(monkeypatch) -> None:
|
def test_declared_apply_remains_read_only_during_selection(monkeypatch) -> None:
|
||||||
repos = [
|
repos = [
|
||||||
{
|
{
|
||||||
"repo_slug": "no-checkout",
|
"repo_slug": "no-checkout",
|
||||||
|
|
@ -182,6 +190,31 @@ def test_apply_processes_at_most_limit_and_records_terminal_outcomes(monkeypatch
|
||||||
},
|
},
|
||||||
]
|
]
|
||||||
client = _install(monkeypatch, _payload(repos))
|
client = _install(monkeypatch, _payload(repos))
|
||||||
|
result = SbomNexusContextResolver().resolve(
|
||||||
|
"catch_up", None, {"limit": 2, "apply": True}
|
||||||
|
)
|
||||||
|
|
||||||
|
assert result["selected_count"] == 2
|
||||||
|
assert "attempted_count" not in result
|
||||||
|
assert client.posts == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_apply_processes_only_fixed_targets_and_records_terminal_outcomes(monkeypatch) -> None:
|
||||||
|
repos = [
|
||||||
|
{
|
||||||
|
"repo_slug": "no-checkout",
|
||||||
|
"last_sbom_at": None,
|
||||||
|
"has_sbom": False,
|
||||||
|
"checkout_available": False,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"repo_slug": "scan-me",
|
||||||
|
"last_sbom_at": None,
|
||||||
|
"has_sbom": False,
|
||||||
|
"checkout_available": True,
|
||||||
|
},
|
||||||
|
]
|
||||||
|
client = _install(monkeypatch, _payload([]))
|
||||||
client.post_results = {
|
client.post_results = {
|
||||||
"/sbom/no-checkout/skip": [
|
"/sbom/no-checkout/skip": [
|
||||||
DummyResponse(
|
DummyResponse(
|
||||||
|
|
@ -205,8 +238,9 @@ def test_apply_processes_at_most_limit_and_records_terminal_outcomes(monkeypatch
|
||||||
],
|
],
|
||||||
}
|
}
|
||||||
|
|
||||||
result = SbomNexusContextResolver().resolve(
|
result = apply_bounded_ingest(
|
||||||
"catch_up", None, {"limit": 2, "apply": True}
|
repos,
|
||||||
|
operation_id="run-1",
|
||||||
)
|
)
|
||||||
|
|
||||||
assert result["attempted_count"] == 2
|
assert result["attempted_count"] == 2
|
||||||
|
|
@ -227,48 +261,141 @@ def test_apply_processes_at_most_limit_and_records_terminal_outcomes(monkeypatch
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
assert len(client.posts) == 2
|
assert len(client.posts) == 2
|
||||||
assert all("not-selected" not in url for url, _body in client.posts)
|
first_key = client.posts[0][2]["Idempotency-Key"]
|
||||||
|
assert first_key == client.posts[0][2]["X-Activity-Core-Operation-ID"]
|
||||||
|
assert first_key != client.posts[1][2]["Idempotency-Key"]
|
||||||
|
|
||||||
|
|
||||||
def test_apply_records_ingest_error_when_ingest_fails(monkeypatch) -> None:
|
def test_ambiguous_ingest_failure_does_not_create_synthetic_skip(monkeypatch) -> None:
|
||||||
client = _install(
|
client = _install(
|
||||||
monkeypatch,
|
monkeypatch,
|
||||||
_payload(
|
_payload([]),
|
||||||
[
|
|
||||||
{
|
|
||||||
"repo_slug": "broken",
|
|
||||||
"last_sbom_at": None,
|
|
||||||
"has_sbom": False,
|
|
||||||
"checkout_available": True,
|
|
||||||
}
|
|
||||||
]
|
|
||||||
),
|
|
||||||
)
|
)
|
||||||
client.post_results = {
|
client.post_results = {
|
||||||
"/sbom/broken/ingest": [DummyResponse({}, status_code=503)],
|
"/sbom/broken/ingest": [DummyResponse({}, status_code=503)],
|
||||||
"/sbom/broken/skip": [
|
}
|
||||||
|
|
||||||
|
with pytest.raises(httpx.HTTPStatusError):
|
||||||
|
apply_bounded_ingest(
|
||||||
|
[{"repo_slug": "broken", "checkout_available": True}],
|
||||||
|
operation_id="run-ambiguous",
|
||||||
|
)
|
||||||
|
|
||||||
|
assert len(client.posts) == 1
|
||||||
|
assert client.posts[0][0].endswith("/sbom/broken/ingest")
|
||||||
|
|
||||||
|
|
||||||
|
def test_retry_resumes_heartbeat_outcomes(monkeypatch) -> None:
|
||||||
|
repos = [
|
||||||
|
{"repo_slug": "first", "checkout_available": True},
|
||||||
|
{"repo_slug": "second", "checkout_available": True},
|
||||||
|
]
|
||||||
|
first_client = _install(monkeypatch, _payload([]))
|
||||||
|
first_client.post_results = {
|
||||||
|
"/sbom/first/ingest": [
|
||||||
DummyResponse(
|
DummyResponse(
|
||||||
{
|
{
|
||||||
"repo_slug": "broken",
|
"repo_slug": "first",
|
||||||
"status": "skipped",
|
"status": "ingested",
|
||||||
"reason": "ingest-error",
|
"snapshot_id": "snapshot-first",
|
||||||
"snapshot_id": "skip-error-1",
|
}
|
||||||
|
)
|
||||||
|
],
|
||||||
|
"/sbom/second/ingest": [DummyResponse({}, status_code=503)],
|
||||||
|
}
|
||||||
|
heartbeats: list[list[dict[str, Any]]] = []
|
||||||
|
|
||||||
|
with pytest.raises(httpx.HTTPStatusError):
|
||||||
|
apply_bounded_ingest(
|
||||||
|
repos,
|
||||||
|
operation_id="run-retry",
|
||||||
|
on_progress=lambda outcomes: heartbeats.append(outcomes),
|
||||||
|
)
|
||||||
|
|
||||||
|
completed = heartbeats[-1]
|
||||||
|
retry_client = _install(monkeypatch, _payload([]))
|
||||||
|
retry_client.post_results = {
|
||||||
|
"/sbom/second/ingest": [
|
||||||
|
DummyResponse(
|
||||||
|
{
|
||||||
|
"repo_slug": "second",
|
||||||
|
"status": "ingested",
|
||||||
|
"snapshot_id": "snapshot-second",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
]
|
||||||
|
}
|
||||||
|
|
||||||
|
result = apply_bounded_ingest(
|
||||||
|
repos,
|
||||||
|
operation_id="run-retry",
|
||||||
|
completed=completed,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert [outcome["repo_slug"] for outcome in result["updated"]] == [
|
||||||
|
"first",
|
||||||
|
"second",
|
||||||
|
]
|
||||||
|
assert len(retry_client.posts) == 1
|
||||||
|
assert retry_client.posts[0][0].endswith("/sbom/second/ingest")
|
||||||
|
|
||||||
|
|
||||||
|
def test_retry_reuses_stable_identity_for_same_run_and_repo(monkeypatch) -> None:
|
||||||
|
repo = {"repo_slug": "ambiguous", "checkout_available": True}
|
||||||
|
first_client = _install(monkeypatch, _payload([]))
|
||||||
|
first_client.post_results = {
|
||||||
|
"/sbom/ambiguous/ingest": [DummyResponse({}, status_code=503)],
|
||||||
|
}
|
||||||
|
|
||||||
|
with pytest.raises(httpx.HTTPStatusError):
|
||||||
|
apply_bounded_ingest([repo], operation_id="run-stable")
|
||||||
|
|
||||||
|
retry_client = _install(monkeypatch, _payload([]))
|
||||||
|
retry_client.post_results = {
|
||||||
|
"/sbom/ambiguous/ingest": [
|
||||||
|
DummyResponse(
|
||||||
|
{
|
||||||
|
"repo_slug": "ambiguous",
|
||||||
|
"status": "ingested",
|
||||||
|
"snapshot_id": "snapshot-ambiguous",
|
||||||
}
|
}
|
||||||
)
|
)
|
||||||
],
|
],
|
||||||
}
|
}
|
||||||
|
apply_bounded_ingest([repo], operation_id="run-stable")
|
||||||
|
|
||||||
result = SbomNexusContextResolver().resolve(
|
first_headers = first_client.posts[0][2]
|
||||||
"catch_up", None, {"limit": 3, "apply": True}
|
retry_headers = retry_client.posts[0][2]
|
||||||
|
assert retry_headers["Idempotency-Key"] == first_headers["Idempotency-Key"]
|
||||||
|
assert retry_headers["X-Activity-Core-Operation-ID"] == first_headers[
|
||||||
|
"X-Activity-Core-Operation-ID"
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def test_apply_collapses_duplicate_targets(monkeypatch) -> None:
|
||||||
|
client = _install(monkeypatch, _payload([]))
|
||||||
|
client.post_results = {
|
||||||
|
"/sbom/duplicate/skip": [
|
||||||
|
DummyResponse(
|
||||||
|
{
|
||||||
|
"repo_slug": "duplicate",
|
||||||
|
"status": "skipped",
|
||||||
|
"reason": "no-checkout",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
]
|
||||||
|
}
|
||||||
|
|
||||||
|
result = apply_bounded_ingest(
|
||||||
|
[
|
||||||
|
{"repo_slug": "duplicate", "checkout_available": False},
|
||||||
|
{"repo_slug": "duplicate", "checkout_available": False},
|
||||||
|
],
|
||||||
|
operation_id="run-duplicates",
|
||||||
)
|
)
|
||||||
|
|
||||||
assert result["attempted_count"] == 1
|
assert result["attempted_count"] == 1
|
||||||
assert result["updated"] == []
|
assert len(client.posts) == 1
|
||||||
assert result["skipped"][0]["reason"] == "ingest-error"
|
|
||||||
assert client.posts[1][1] == {
|
|
||||||
"reason": "ingest-error",
|
|
||||||
"detail": "HTTPStatusError",
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def test_read_only_default_never_posts(monkeypatch) -> None:
|
def test_read_only_default_never_posts(monkeypatch) -> None:
|
||||||
|
|
@ -281,6 +408,106 @@ def test_read_only_default_never_posts(monkeypatch) -> None:
|
||||||
assert client.posts == []
|
assert client.posts == []
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_dedicated_activity_applies_truncated_workflow_selection(monkeypatch) -> None:
|
||||||
|
from activity_core import activities
|
||||||
|
|
||||||
|
captured: dict[str, Any] = {}
|
||||||
|
|
||||||
|
def fake_apply(repos, *, operation_id, completed, on_progress):
|
||||||
|
captured.update(
|
||||||
|
repos=repos,
|
||||||
|
operation_id=operation_id,
|
||||||
|
completed=completed,
|
||||||
|
)
|
||||||
|
return {"attempted_count": len(repos), "updated": [], "skipped": []}
|
||||||
|
|
||||||
|
monkeypatch.setattr(
|
||||||
|
"activity_core.context_resolvers.sbom_nexus.apply_bounded_ingest",
|
||||||
|
fake_apply,
|
||||||
|
)
|
||||||
|
patch = await activities.apply_sbom_catchup(
|
||||||
|
{
|
||||||
|
"run_id": "run-fixed",
|
||||||
|
"context_sources": [
|
||||||
|
{
|
||||||
|
"type": "sbom-nexus",
|
||||||
|
"query": "catch_up",
|
||||||
|
"bind_to": "context.catchup",
|
||||||
|
"params": {"limit": 2, "apply": True},
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"context": {
|
||||||
|
"catchup": {
|
||||||
|
"limit": 2,
|
||||||
|
"repos": [
|
||||||
|
{"repo_slug": "a"},
|
||||||
|
{"repo_slug": "b"},
|
||||||
|
{"repo_slug": "must-not-run"},
|
||||||
|
],
|
||||||
|
}
|
||||||
|
},
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
assert [repo["repo_slug"] for repo in captured["repos"]] == ["a", "b"]
|
||||||
|
assert captured["operation_id"] == "run-fixed"
|
||||||
|
assert patch["catchup"]["attempted_count"] == 2
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_dedicated_activity_forwards_heartbeat_outcomes(monkeypatch) -> None:
|
||||||
|
from activity_core import activities
|
||||||
|
|
||||||
|
completed = [
|
||||||
|
{
|
||||||
|
"repo_slug": "already-done",
|
||||||
|
"status": "skipped",
|
||||||
|
"reason": "no-checkout",
|
||||||
|
}
|
||||||
|
]
|
||||||
|
captured: dict[str, Any] = {}
|
||||||
|
|
||||||
|
monkeypatch.setattr(
|
||||||
|
activities,
|
||||||
|
"_sbom_heartbeat_state",
|
||||||
|
lambda run_id: {
|
||||||
|
"run_id": run_id,
|
||||||
|
"outcomes_by_bind": {"catchup": completed},
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
def fake_apply(repos, *, operation_id, completed, on_progress):
|
||||||
|
captured["completed"] = completed
|
||||||
|
return {"attempted_count": 1, "updated": [], "skipped": completed}
|
||||||
|
|
||||||
|
monkeypatch.setattr(
|
||||||
|
"activity_core.context_resolvers.sbom_nexus.apply_bounded_ingest",
|
||||||
|
fake_apply,
|
||||||
|
)
|
||||||
|
await activities.apply_sbom_catchup(
|
||||||
|
{
|
||||||
|
"run_id": "run-resumed",
|
||||||
|
"context_sources": [
|
||||||
|
{
|
||||||
|
"type": "sbom-nexus",
|
||||||
|
"query": "catch_up",
|
||||||
|
"bind_to": "context.catchup",
|
||||||
|
"params": {"limit": 1, "apply": True},
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"context": {
|
||||||
|
"catchup": {
|
||||||
|
"limit": 1,
|
||||||
|
"repos": [{"repo_slug": "already-done"}],
|
||||||
|
}
|
||||||
|
},
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
assert captured["completed"] == completed
|
||||||
|
|
||||||
|
|
||||||
def test_catch_up_normalises_partial_entries(monkeypatch) -> None:
|
def test_catch_up_normalises_partial_entries(monkeypatch) -> None:
|
||||||
_install(
|
_install(
|
||||||
monkeypatch,
|
monkeypatch,
|
||||||
|
|
|
||||||
|
|
@ -4,11 +4,11 @@ type: workplan
|
||||||
title: "Hub port and schedule alignment for State Hub retirement"
|
title: "Hub port and schedule alignment for State Hub retirement"
|
||||||
domain: infotech
|
domain: infotech
|
||||||
repo: activity-core
|
repo: activity-core
|
||||||
status: active
|
status: finished
|
||||||
owner: grok
|
owner: grok
|
||||||
topic_slug: infotech
|
topic_slug: infotech
|
||||||
created: "2026-08-09"
|
created: "2026-08-09"
|
||||||
updated: "2026-08-18"
|
updated: "2026-08-22"
|
||||||
parent_project: prj-state-hub-retirement
|
parent_project: prj-state-hub-retirement
|
||||||
parent_workplan: SHR-WP-0001
|
parent_workplan: SHR-WP-0001
|
||||||
stream: S5
|
stream: S5
|
||||||
|
|
@ -107,16 +107,27 @@ inside this repo.
|
||||||
|
|
||||||
```task
|
```task
|
||||||
id: ACTIVITY-WP-0029-T03
|
id: ACTIVITY-WP-0029-T03
|
||||||
status: wait
|
status: done
|
||||||
priority: medium
|
priority: medium
|
||||||
state_hub_task_id: "3fbadbfe-9107-4869-b53f-3b9b46e0555a"
|
state_hub_task_id: "3fbadbfe-9107-4869-b53f-3b9b46e0555a"
|
||||||
```
|
```
|
||||||
|
|
||||||
Wait on **HUB-WP-0004** (ports still `proposed`) and the CORE-WP-0010
|
HUB-WP-0004 finished and hub-core became production authority on 2026-08-21.
|
||||||
absorption path. Interaction-event / progress evidence follows the
|
Activity-core now implements `hub-core-interaction-event` against canonical
|
||||||
**hub-core** runtime, not a permanent core-hub or State Hub host. Update
|
`POST /ports/events/interaction`, then verifies the accepted id through
|
||||||
smokes when the port exists. Until then, keep the State Hub beachhead
|
`GET /ports/projections/interaction_events`. The Railiance projection points
|
||||||
(ACTIVITY-WP-0015) as the compatibility sink.
|
`HUB_CORE_BASE_URL` at the in-cluster hub-core runtime and uses this sink; it
|
||||||
|
does not require the legacy Core Hub widget mapping/runtime-token contract.
|
||||||
|
|
||||||
|
The old `core-hub-interaction-event` compatibility sink remains available for
|
||||||
|
the no-writer rollback window, but is no longer the configured probe path.
|
||||||
|
General progress callers not yet moved by the retirement strangler continue to
|
||||||
|
use the State Hub beachhead rather than creating another permanent authority.
|
||||||
|
Unit and deployment-wiring tests cover catalog envelope shape, readback,
|
||||||
|
sanitization, missing configuration, and the in-cluster target. A disposable
|
||||||
|
integration smoke against hub-core 0.2.0 returned `202 Accepted` from the
|
||||||
|
canonical interaction port and verified the new event through the interaction
|
||||||
|
projection.
|
||||||
|
|
||||||
## Execution queue boundary
|
## Execution queue boundary
|
||||||
|
|
||||||
|
|
@ -138,6 +149,6 @@ index, issue-core lifecycle, ITC Task Model types, policy-nexus publication.
|
||||||
(`docs/state-hub-caller-map.md`)
|
(`docs/state-hub-caller-map.md`)
|
||||||
- [x] Sweep still fires from activity-core and hits repo-manager (or dual-run)
|
- [x] Sweep still fires from activity-core and hits repo-manager (or dual-run)
|
||||||
(`CONSISTENCY_SWEEP_URL` / `REPO_MANAGER_URL`; default State Hub adapter)
|
(`CONSISTENCY_SWEEP_URL` / `REPO_MANAGER_URL`; default State Hub adapter)
|
||||||
- [x] Evidence sink plan names hub-core as the permanent host; T03 stays wait until that port exists
|
- [x] Ops evidence uses hub-core's canonical interaction port; the legacy Core Hub sink is rollback-only
|
||||||
- [x] Execution boundary is written so `/execution/*` cannot be read as "activity-core owns tasks"
|
- [x] Execution boundary is written so `/execution/*` cannot be read as "activity-core owns tasks"
|
||||||
(`docs/execution-queue-boundary.md`, `GET /execution/semantics`, 410 on workplan routes)
|
(`docs/execution-queue-boundary.md`, `GET /execution/semantics`, 410 on workplan routes)
|
||||||
|
|
|
||||||
|
|
@ -4,7 +4,7 @@ type: workplan
|
||||||
title: "Daily bounded SBOM catch-up via sbom-nexus"
|
title: "Daily bounded SBOM catch-up via sbom-nexus"
|
||||||
domain: infotech
|
domain: infotech
|
||||||
repo: activity-core
|
repo: activity-core
|
||||||
status: done
|
status: finished
|
||||||
owner: grok
|
owner: grok
|
||||||
topic_slug: infotech
|
topic_slug: infotech
|
||||||
created: "2026-08-18"
|
created: "2026-08-18"
|
||||||
|
|
|
||||||
90
workplans/ACTIVITY-WP-0033-sbom-catchup-retry-boundary.md
Normal file
90
workplans/ACTIVITY-WP-0033-sbom-catchup-retry-boundary.md
Normal file
|
|
@ -0,0 +1,90 @@
|
||||||
|
---
|
||||||
|
id: ACTIVITY-WP-0033
|
||||||
|
type: workplan
|
||||||
|
title: "Bound SBOM catch-up across Temporal retries"
|
||||||
|
domain: infotech
|
||||||
|
repo: activity-core
|
||||||
|
status: finished
|
||||||
|
owner: codex
|
||||||
|
topic_slug: infotech
|
||||||
|
created: "2026-08-22"
|
||||||
|
updated: "2026-08-22"
|
||||||
|
related:
|
||||||
|
- ACTIVITY-WP-0030
|
||||||
|
- CUST-WP-0062
|
||||||
|
- SBOM-WP-0002
|
||||||
|
---
|
||||||
|
|
||||||
|
# Bound SBOM catch-up across Temporal retries
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
|
||||||
|
Preserve ACTIVITY-WP-0030's oldest-N bound when Temporal retries an activity or
|
||||||
|
an SBOM Nexus response is ambiguous. Activity Core can independently guarantee
|
||||||
|
that one workflow fire retains one fixed set of at most N distinct repository
|
||||||
|
targets. Exact once-only snapshot creation remains a Nexus-side idempotency
|
||||||
|
follow-up.
|
||||||
|
|
||||||
|
## Separate selection from mutation
|
||||||
|
|
||||||
|
```task
|
||||||
|
id: ACTIVITY-WP-0033-T01
|
||||||
|
status: done
|
||||||
|
priority: high
|
||||||
|
```
|
||||||
|
|
||||||
|
Keep `resolve_context` and the `sbom-nexus / catch_up` resolver read-only even
|
||||||
|
when the definition declares `apply: true`. Compute the deterministic run id
|
||||||
|
before context resolution, then pass the already-truncated selection to a
|
||||||
|
dedicated Temporal activity. Retries must reuse that same selection rather than
|
||||||
|
querying the next batch.
|
||||||
|
|
||||||
|
## Resume bounded outcomes safely
|
||||||
|
|
||||||
|
```task
|
||||||
|
id: ACTIVITY-WP-0033-T02
|
||||||
|
status: done
|
||||||
|
priority: high
|
||||||
|
```
|
||||||
|
|
||||||
|
Heartbeat completed per-repository outcomes so an ordinary activity retry does
|
||||||
|
not repeat acknowledged targets. Send a stable per-run/per-repository
|
||||||
|
`Idempotency-Key` for forward compatibility with Nexus enforcement. Never turn
|
||||||
|
an HTTP timeout, transport error, or malformed success response into a second
|
||||||
|
synthetic skip: the result is ambiguous and must fail visibly against the same
|
||||||
|
fixed target set.
|
||||||
|
|
||||||
|
## Verification and handoff
|
||||||
|
|
||||||
|
```task
|
||||||
|
id: ACTIVITY-WP-0033-T03
|
||||||
|
status: done
|
||||||
|
priority: medium
|
||||||
|
```
|
||||||
|
|
||||||
|
Cover read-only selection, fixed-target retry, heartbeat resume, duplicate
|
||||||
|
target collapse, stable request identity, and ambiguous-response behavior.
|
||||||
|
Update the runbook and hand the remaining exactly-once enforcement requirement
|
||||||
|
to SBOM Nexus without claiming it is solved locally.
|
||||||
|
|
||||||
|
Completed 2026-08-22. `resolve_context` now only selects and normalises a
|
||||||
|
deduplicated oldest-N set. `RunActivityWorkflow` records that selection before
|
||||||
|
calling the registered `apply_sbom_catchup` activity, protected by Temporal
|
||||||
|
patch marker `activity-wp-0033-sbom-retry-boundary`. The apply activity resumes
|
||||||
|
terminal outcomes from heartbeat details and sends a deterministic UUID
|
||||||
|
operation key for each workflow-run/repository pair. Ambiguous HTTP and
|
||||||
|
contract failures remain failures against the same fixed target instead of
|
||||||
|
creating `ingest-error` skip snapshots.
|
||||||
|
|
||||||
|
Verification: 27 focused SBOM tests and 417 repository tests passed; the one
|
||||||
|
live NATS-to-Temporal bridge test was deselected because it requires the local
|
||||||
|
integration stack. Python compilation and `git diff --check` also passed.
|
||||||
|
|
||||||
|
## Acceptance
|
||||||
|
|
||||||
|
- [x] Context resolution performs no SBOM mutation
|
||||||
|
- [x] One workflow fire retains at most N distinct targets across retries
|
||||||
|
- [x] Heartbeat replay skips outcomes already acknowledged by the activity
|
||||||
|
- [x] Ambiguous ingest responses do not create synthetic skip snapshots
|
||||||
|
- [x] Stable operation identity is sent for future Nexus enforcement
|
||||||
|
- [x] Focused and repository-wide Activity Core verification pass
|
||||||
Loading…
Add table
Add a link
Reference in a new issue