feat: join audit-storage usage without inventing a booked Scaleway fact
Ingest resource-control's August technical usage, omit nulls, and reconcile forecast + usage + booked facts. Commissioning month reports missing_booked_fact instead of 0.00. T05 still waits on the first invoice.
This commit is contained in:
parent
75356ea19e
commit
ac0949afd0
7 changed files with 426 additions and 1 deletions
|
|
@ -26,6 +26,8 @@ uv run finhub import-ai-plan tests/fixtures/ai-plans.csv
|
|||
uv run finhub ledger set-balance 12000
|
||||
uv run finhub ledger import cloud tests/fixtures/cloud-costs.csv
|
||||
uv run finhub ledger import ai-plan tests/fixtures/ai-plans.csv
|
||||
uv run finhub ledger ingest-usage --resource resource:platform:audit-storage --source resource-control:data/actuals/2026-08.json tests/fixtures/platform-audit-storage-usage-2026-08.json
|
||||
uv run finhub ledger reconcile-resource --resource resource:platform:audit-storage --period 2026-08
|
||||
uv run finhub ledger commitments
|
||||
uv run finhub ledger set-entitlement --provider anthropic --plan claude-max --period 2026-08 --unit plan --plan-label "Max 20x" --source vendor-plan
|
||||
uv run finhub ledger plan-month --period 2026-08
|
||||
|
|
|
|||
45
docs/evidence/fin-wp-0004-t05-commissioning-2026-08.md
Normal file
45
docs/evidence/fin-wp-0004-t05-commissioning-2026-08.md
Normal file
|
|
@ -0,0 +1,45 @@
|
|||
# FIN-WP-0004-T05 — commissioning-period join, 2026-08
|
||||
|
||||
Date: 2026-08-15
|
||||
Resource: `resource:platform:audit-storage`
|
||||
Attribution key: `platform:audit-storage`
|
||||
|
||||
## What exists
|
||||
|
||||
| Record | Count | Authority |
|
||||
| --- | --- | --- |
|
||||
| Forecast (base, 2026-09 … 2027-08) | 12 rows | resource-control |
|
||||
| Technical usage 2026-08 | 1 row | resource-control |
|
||||
| Booked Scaleway fact 2026-08 | **0** | fin-hub |
|
||||
|
||||
The object store was commissioned 2026-08-14. resource-control’s usage
|
||||
record carries `infrastructure_eur: null` and names fin-hub as the
|
||||
owner of that gap. There is no invoice. A null is not booked as 0.00.
|
||||
|
||||
## Join result
|
||||
|
||||
`ingest_resource_usage` stores only measured quantities (`database_gb`,
|
||||
`stored_gb`, `backup_success_pct`, `restore_rto_minutes`). Null usage
|
||||
fields and the missing infrastructure amount remain gaps in
|
||||
`source_evidence`.
|
||||
|
||||
`reconcile_resource_period` for 2026-08 reports:
|
||||
|
||||
- usage present
|
||||
- `missing_booked_fact: true`
|
||||
- `booked_effective_amount: null`
|
||||
- `invented_zero: false`
|
||||
|
||||
`exchange_health` raises `usage_without_booked_fact` for this period.
|
||||
|
||||
## Why T05 stays open
|
||||
|
||||
The done criterion is: the same cost booked once, projected,
|
||||
joined to technical evidence, and returned without a second ledger.
|
||||
That cost does not exist yet. The first comparable Scaleway invoice
|
||||
period is 2026-09. This note is the commissioning baseline, not
|
||||
closure.
|
||||
|
||||
When a provider charge or credit lands, book it once, bind
|
||||
`financial_fact_id` → `resource:platform:audit-storage`, and rerun
|
||||
`ledger reconcile-resource --period 2026-09`.
|
||||
|
|
@ -20,6 +20,11 @@ from fin_hub.services.allocation import shared_cost_allocations
|
|||
from fin_hub.services.billing import build_billing_basis
|
||||
from fin_hub.services.effectiveness import effectiveness_report
|
||||
from fin_hub.services.evaluate import evaluate_runway
|
||||
from fin_hub.services.exchange import (
|
||||
ingest_resource_forecast,
|
||||
ingest_resource_usage,
|
||||
reconcile_resource_period,
|
||||
)
|
||||
from fin_hub.services.reporting_allocation import (
|
||||
allocate_ai_plan_report,
|
||||
list_ai_plan_allocations,
|
||||
|
|
@ -264,6 +269,40 @@ def _cmd_ledger_effectiveness(args: argparse.Namespace) -> int:
|
|||
return 0
|
||||
|
||||
|
||||
def _cmd_ledger_ingest_forecast(args: argparse.Namespace) -> int:
|
||||
payload = json.loads(Path(args.path).read_text(encoding="utf-8"))
|
||||
records = ingest_resource_forecast(
|
||||
payload,
|
||||
resource_id=args.resource,
|
||||
source_evidence_ref=args.source,
|
||||
ledger_path=_ledger_path(args),
|
||||
)
|
||||
print(json.dumps({"ingested": len(records)}, indent=2))
|
||||
return 0
|
||||
|
||||
|
||||
def _cmd_ledger_ingest_usage(args: argparse.Namespace) -> int:
|
||||
payload = json.loads(Path(args.path).read_text(encoding="utf-8"))
|
||||
records = ingest_resource_usage(
|
||||
payload,
|
||||
resource_id=args.resource,
|
||||
source_evidence_ref=args.source,
|
||||
ledger_path=_ledger_path(args),
|
||||
)
|
||||
print(json.dumps({"ingested": len(records)}, indent=2))
|
||||
return 0
|
||||
|
||||
|
||||
def _cmd_ledger_reconcile_resource(args: argparse.Namespace) -> int:
|
||||
report = reconcile_resource_period(
|
||||
resource_id=args.resource,
|
||||
period=args.period,
|
||||
ledger_path=_ledger_path(args),
|
||||
)
|
||||
print(json.dumps(report.as_dict(), indent=2))
|
||||
return 0
|
||||
|
||||
|
||||
def _cmd_ledger_allocations(args: argparse.Namespace) -> int:
|
||||
reports = shared_cost_allocations(ledger_path=_ledger_path(args))
|
||||
print(json.dumps([report.as_dict() for report in reports], indent=2, default=str))
|
||||
|
|
@ -364,6 +403,35 @@ def build_parser() -> argparse.ArgumentParser:
|
|||
ledger_import.add_argument("--force", action="store_true", help="Re-import even if file unchanged")
|
||||
ledger_import.set_defaults(func=_cmd_ledger_import)
|
||||
|
||||
ledger_ingest_forecast = ledger_sub.add_parser(
|
||||
"ingest-forecast",
|
||||
help="Ingest a resource-control monthly forecast as planning evidence",
|
||||
)
|
||||
ledger_ingest_forecast.add_argument("path")
|
||||
ledger_ingest_forecast.add_argument("--resource", required=True)
|
||||
ledger_ingest_forecast.add_argument("--source", required=True)
|
||||
ledger_ingest_forecast.add_argument("--ledger", help="Ledger database path")
|
||||
ledger_ingest_forecast.set_defaults(func=_cmd_ledger_ingest_forecast)
|
||||
|
||||
ledger_ingest_usage = ledger_sub.add_parser(
|
||||
"ingest-usage",
|
||||
help="Ingest a resource-control usage observation (nulls stay omitted)",
|
||||
)
|
||||
ledger_ingest_usage.add_argument("path")
|
||||
ledger_ingest_usage.add_argument("--resource", required=True)
|
||||
ledger_ingest_usage.add_argument("--source", required=True)
|
||||
ledger_ingest_usage.add_argument("--ledger", help="Ledger database path")
|
||||
ledger_ingest_usage.set_defaults(func=_cmd_ledger_ingest_usage)
|
||||
|
||||
ledger_reconcile = ledger_sub.add_parser(
|
||||
"reconcile-resource",
|
||||
help="Join forecast, usage, and booked facts for one resource period",
|
||||
)
|
||||
ledger_reconcile.add_argument("--resource", required=True)
|
||||
ledger_reconcile.add_argument("--period", required=True, help="YYYY-MM")
|
||||
ledger_reconcile.add_argument("--ledger", help="Ledger database path")
|
||||
ledger_reconcile.set_defaults(func=_cmd_ledger_reconcile_resource)
|
||||
|
||||
ledger_summary = ledger_sub.add_parser("summary", help="Monthly rollup summary from ledger")
|
||||
ledger_summary.add_argument("--ledger", help="Ledger database path (default: .fin-hub/ledger.db)")
|
||||
ledger_summary.set_defaults(func=_cmd_ledger_summary)
|
||||
|
|
|
|||
|
|
@ -438,6 +438,31 @@ def exchange_health(
|
|||
)
|
||||
)
|
||||
|
||||
for row in planning_rows:
|
||||
if row["record_type"] != "usage_observation":
|
||||
continue
|
||||
payload = json.loads(row["payload_json"])
|
||||
usage_resource = payload.get("resource_id")
|
||||
if not usage_resource:
|
||||
continue
|
||||
period = payload["period_start"][:7]
|
||||
if any(
|
||||
fact.resource_id == usage_resource and fact.accounting_period == period
|
||||
for fact in booked
|
||||
):
|
||||
continue
|
||||
issues.append(
|
||||
ExchangeQualityIssue(
|
||||
code="usage_without_booked_fact",
|
||||
severity="warning",
|
||||
record_id=row["record_id"],
|
||||
detail=(
|
||||
f"{usage_resource} {period} has technical usage and no booked "
|
||||
"fact; absence is not zero spend"
|
||||
),
|
||||
)
|
||||
)
|
||||
|
||||
for row in rejection_rows:
|
||||
issues.append(
|
||||
ExchangeQualityIssue(
|
||||
|
|
@ -525,3 +550,174 @@ def ingest_resource_forecast(
|
|||
)
|
||||
records.append(record)
|
||||
return records
|
||||
|
||||
|
||||
_USAGE_MEASURES = (
|
||||
("database_gb", "GB"),
|
||||
("stored_gb", "GB"),
|
||||
("wal_gb", "GB"),
|
||||
("restore_egress_gb", "GB"),
|
||||
("write_requests", "count"),
|
||||
("read_requests", "count"),
|
||||
("backup_success_pct", "percent"),
|
||||
("restore_rto_minutes", "minutes"),
|
||||
)
|
||||
|
||||
|
||||
def ingest_resource_usage(
|
||||
payload: dict,
|
||||
*,
|
||||
resource_id: str,
|
||||
source_evidence_ref: str,
|
||||
ledger_path: Path | None = None,
|
||||
) -> list[PlanningEvidence]:
|
||||
"""Adapt resource-control monthly usage. Nulls stay omitted, never zero."""
|
||||
|
||||
if payload.get("record_type") != "usage_observation":
|
||||
raise ValueError("resource usage payload must have record_type=usage_observation")
|
||||
if payload.get("schema_version") not in {"0.1", "0.2"}:
|
||||
raise ValueError("resource usage payload must be schema 0.1 or 0.2")
|
||||
if not resource_id.startswith("resource:"):
|
||||
raise ValueError("resource_id must use the resource: namespace")
|
||||
created_at = payload["created_at"]
|
||||
records: list[PlanningEvidence] = []
|
||||
for row in payload["rows"]:
|
||||
year, month = (int(part) for part in row["period"].split("-"))
|
||||
period_start = date(year, month, 1)
|
||||
period_end = date(year, month, monthrange(year, month)[1])
|
||||
measures = []
|
||||
for name, unit in _USAGE_MEASURES:
|
||||
value = row.get(name)
|
||||
if value is None:
|
||||
continue
|
||||
measures.append({"name": name, "value": value, "unit": unit})
|
||||
gaps = [f"gap:{gap}" for gap in row.get("measurement_gaps", [])]
|
||||
if row.get("infrastructure_eur") is None:
|
||||
gaps.append("gap:infrastructure_eur:no booked fact (owner: fin-hub)")
|
||||
record = ingest_planning_evidence(
|
||||
{
|
||||
"schema_version": "0.1",
|
||||
"record_type": "usage_observation",
|
||||
"record_id": (
|
||||
f"usage:{payload['provider_id']}:"
|
||||
f"{payload['cost_attribution_key']}:{row['period']}"
|
||||
),
|
||||
"revision_of": None,
|
||||
"resource_id": resource_id,
|
||||
"service_id": payload["provider_id"],
|
||||
"workload_id": payload["workload"],
|
||||
"tenant_id": None,
|
||||
"environment": "production",
|
||||
"cost_attribution_key": payload["cost_attribution_key"],
|
||||
"period_start": period_start.isoformat(),
|
||||
"period_end": period_end.isoformat(),
|
||||
"source_evidence": [
|
||||
source_evidence_ref,
|
||||
*payload.get("evidence", []),
|
||||
*gaps,
|
||||
],
|
||||
"created_at": created_at,
|
||||
"measures": measures,
|
||||
},
|
||||
ledger_path=ledger_path,
|
||||
)
|
||||
records.append(record)
|
||||
return records
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ResourcePeriodReconciliation:
|
||||
resource_id: str
|
||||
period: str
|
||||
forecast_record_ids: tuple[str, ...]
|
||||
usage_record_ids: tuple[str, ...]
|
||||
booked_financial_fact_ids: tuple[str, ...]
|
||||
booked_effective_amount: Decimal | None
|
||||
booked_currency: str | None
|
||||
usage_measures: tuple[tuple[str, str, str], ...]
|
||||
missing_booked_fact: bool
|
||||
invented_zero: bool
|
||||
|
||||
def as_dict(self) -> dict:
|
||||
payload = asdict(self)
|
||||
payload["booked_effective_amount"] = (
|
||||
None
|
||||
if self.booked_effective_amount is None
|
||||
else str(self.booked_effective_amount)
|
||||
)
|
||||
return payload
|
||||
|
||||
|
||||
def reconcile_resource_period(
|
||||
*,
|
||||
resource_id: str,
|
||||
period: str,
|
||||
ledger_path: Path | None = None,
|
||||
fact_resource_ids: Mapping[str, str] | None = None,
|
||||
) -> ResourcePeriodReconciliation:
|
||||
"""Join forecast, usage, and booked facts. Absence of a booked amount is not zero."""
|
||||
|
||||
if not resource_id.startswith("resource:"):
|
||||
raise ValueError("resource_id must use the resource: namespace")
|
||||
year, month = (int(part) for part in period.split("-"))
|
||||
period_start = date(year, month, 1)
|
||||
period_end = date(year, month, monthrange(year, month)[1])
|
||||
ledger = ledger_path or default_ledger_path()
|
||||
with _connect(ledger) as conn:
|
||||
_ensure_planning_schema(conn)
|
||||
planning_rows = conn.execute(
|
||||
"SELECT record_id, record_type, payload_json FROM planning_evidence "
|
||||
"WHERE is_current = 1 ORDER BY record_id"
|
||||
).fetchall()
|
||||
forecast_ids: list[str] = []
|
||||
usage_ids: list[str] = []
|
||||
usage_measures: list[tuple[str, str, str]] = []
|
||||
for row in planning_rows:
|
||||
payload = json.loads(row["payload_json"])
|
||||
if payload.get("resource_id") != resource_id:
|
||||
continue
|
||||
start = date.fromisoformat(payload["period_start"])
|
||||
end = date.fromisoformat(payload["period_end"])
|
||||
if end < period_start or start > period_end:
|
||||
continue
|
||||
if row["record_type"] == "forecast":
|
||||
forecast_ids.append(row["record_id"])
|
||||
if row["record_type"] == "usage_observation":
|
||||
usage_ids.append(row["record_id"])
|
||||
for measure in payload.get("measures", []):
|
||||
usage_measures.append(
|
||||
(measure["name"], str(measure["value"]), measure["unit"])
|
||||
)
|
||||
booked = booked_cost_projection(
|
||||
ledger_path=ledger, fact_resource_ids=fact_resource_ids
|
||||
)
|
||||
matching = [
|
||||
fact
|
||||
for fact in booked
|
||||
if fact.resource_id == resource_id and fact.accounting_period == period
|
||||
]
|
||||
amount = None
|
||||
currency = None
|
||||
if matching:
|
||||
currencies = {fact.currency for fact in matching}
|
||||
if len(currencies) != 1:
|
||||
raise ValueError("booked facts for this resource period mix currencies")
|
||||
currency = next(iter(currencies))
|
||||
amount = sum((fact.effective_amount for fact in matching), Decimal("0.00"))
|
||||
invented_zero = amount == Decimal("0.00") and all(
|
||||
fact.adjustment_kind == "charge" for fact in matching
|
||||
)
|
||||
else:
|
||||
invented_zero = False
|
||||
return ResourcePeriodReconciliation(
|
||||
resource_id=resource_id,
|
||||
period=period,
|
||||
forecast_record_ids=tuple(forecast_ids),
|
||||
usage_record_ids=tuple(usage_ids),
|
||||
booked_financial_fact_ids=tuple(fact.financial_fact_id for fact in matching),
|
||||
booked_effective_amount=amount,
|
||||
booked_currency=currency,
|
||||
usage_measures=tuple(usage_measures),
|
||||
missing_booked_fact=not matching,
|
||||
invented_zero=invented_zero,
|
||||
)
|
||||
|
|
|
|||
33
tests/fixtures/platform-audit-storage-usage-2026-08.json
vendored
Normal file
33
tests/fixtures/platform-audit-storage-usage-2026-08.json
vendored
Normal file
|
|
@ -0,0 +1,33 @@
|
|||
{
|
||||
"schema_version": "0.2",
|
||||
"record_type": "usage_observation",
|
||||
"workload": "rapp-postgres/platform-pg",
|
||||
"cost_attribution_key": "platform:audit-storage",
|
||||
"provider_id": "scaleway-standard-multi-az",
|
||||
"created_at": "2026-08-14T18:24:00Z",
|
||||
"scenario": "observed",
|
||||
"forecast_ref": "data/forecasts/platform-audit-storage-scaleway-base-2026-08.json",
|
||||
"rows": [
|
||||
{
|
||||
"period": "2026-08",
|
||||
"database_gb": 0.6365,
|
||||
"stored_gb": 0.0066,
|
||||
"wal_gb": null,
|
||||
"restore_egress_gb": null,
|
||||
"write_requests": null,
|
||||
"read_requests": null,
|
||||
"infrastructure_eur": null,
|
||||
"internal_labor_hours": null,
|
||||
"internal_labor_eur": null,
|
||||
"total_eur": null,
|
||||
"backup_success_pct": 100,
|
||||
"restore_rto_minutes": 1.08,
|
||||
"measurement_gaps": [
|
||||
"infrastructure_eur: no invoice for a resource commissioned 2026-08-14 (owner: fin-hub, FIN-WP-0004)"
|
||||
]
|
||||
}
|
||||
],
|
||||
"evidence": [
|
||||
"resource-control:data/actuals/2026-08.json"
|
||||
]
|
||||
}
|
||||
|
|
@ -1,3 +1,4 @@
|
|||
import json
|
||||
from datetime import date, datetime, timezone
|
||||
from decimal import Decimal
|
||||
from pathlib import Path
|
||||
|
|
@ -17,6 +18,8 @@ from fin_hub.services.exchange import (
|
|||
financial_constraint_projection,
|
||||
ingest_planning_evidence,
|
||||
ingest_resource_forecast,
|
||||
ingest_resource_usage,
|
||||
reconcile_resource_period,
|
||||
)
|
||||
from fin_hub.services.ledger import import_csv
|
||||
|
||||
|
|
@ -341,3 +344,72 @@ def test_exchange_health_exposes_rejection_stale_forecast_and_unattributed_cost(
|
|||
"stale_forecast",
|
||||
"unattributed_booked_cost",
|
||||
}
|
||||
|
||||
|
||||
FIXTURES = Path(__file__).parent / "fixtures"
|
||||
AUDIT_STORAGE = "resource:platform:audit-storage"
|
||||
|
||||
|
||||
def test_audit_storage_usage_omits_nulls_and_does_not_invent_zero_spend(tmp_path: Path):
|
||||
ledger = tmp_path / "ledger.db"
|
||||
payload = json.loads(
|
||||
(FIXTURES / "platform-audit-storage-usage-2026-08.json").read_text()
|
||||
)
|
||||
records = ingest_resource_usage(
|
||||
payload,
|
||||
resource_id=AUDIT_STORAGE,
|
||||
source_evidence_ref="resource-control:data/actuals/2026-08.json",
|
||||
ledger_path=ledger,
|
||||
)
|
||||
assert len(records) == 1
|
||||
names = {measure.name: measure for measure in records[0].measures}
|
||||
assert set(names) == {
|
||||
"database_gb",
|
||||
"stored_gb",
|
||||
"backup_success_pct",
|
||||
"restore_rto_minutes",
|
||||
}
|
||||
assert "infrastructure_eur" not in names
|
||||
assert names["stored_gb"].value == Decimal("0.0066")
|
||||
assert any("gap:infrastructure_eur" in item for item in records[0].source_evidence)
|
||||
|
||||
forecast = {
|
||||
"schema_version": "0.1",
|
||||
"record_type": "forecast",
|
||||
"workload": "platform-pg",
|
||||
"cost_attribution_key": "platform:audit-storage",
|
||||
"provider_id": "scaleway-standard-multi-az",
|
||||
"created_at": "2026-08-10T17:10:00Z",
|
||||
"scenario": "base",
|
||||
"forecast_ref": None,
|
||||
"rows": [
|
||||
{
|
||||
"period": "2026-09",
|
||||
"database_gb": 5,
|
||||
"stored_gb": 180,
|
||||
"wal_gb": 30,
|
||||
"restore_egress_gb": 5,
|
||||
"write_requests": 2500,
|
||||
"read_requests": 1000,
|
||||
"infrastructure_eur": 2.89,
|
||||
"internal_labor_hours": 1,
|
||||
"internal_labor_eur": 60,
|
||||
"total_eur": 62.89,
|
||||
}
|
||||
],
|
||||
}
|
||||
ingest_resource_forecast(
|
||||
forecast,
|
||||
resource_id=AUDIT_STORAGE,
|
||||
source_evidence_ref="resource-control:data/forecasts/platform-audit-storage",
|
||||
ledger_path=ledger,
|
||||
)
|
||||
august = reconcile_resource_period(
|
||||
resource_id=AUDIT_STORAGE, period="2026-08", ledger_path=ledger
|
||||
)
|
||||
assert august.usage_record_ids
|
||||
assert august.missing_booked_fact is True
|
||||
assert august.booked_effective_amount is None
|
||||
assert august.invented_zero is False
|
||||
health = exchange_health(ledger_path=ledger)
|
||||
assert "usage_without_booked_fact" in {issue.code for issue in health.issues}
|
||||
|
|
|
|||
|
|
@ -8,7 +8,7 @@ status: active
|
|||
owner: codex
|
||||
topic_slug: financials
|
||||
created: "2026-08-10"
|
||||
updated: "2026-08-11"
|
||||
updated: "2026-08-15"
|
||||
related:
|
||||
- FIN-WP-0001
|
||||
- RESOURCE-WP-0002
|
||||
|
|
@ -198,6 +198,15 @@ The task now waits on `RESOURCE-WP-0002` procurement and the first real
|
|||
provider fact attributable to `platform:audit-storage`. Synthetic records test
|
||||
the contract but do not satisfy the operational booked-cost round trip.
|
||||
|
||||
Progress 2026-08-15: resource-control commissioned the store on 2026-08-14
|
||||
and published technical usage for August (`data/actuals/2026-08.json`).
|
||||
`ingest_resource_usage` and `reconcile_resource_period` now join that
|
||||
usage to the 12-row forecast without inventing a booked amount. August
|
||||
reconciles as usage present, `missing_booked_fact: true`,
|
||||
`invented_zero: false`. Evidence:
|
||||
`docs/evidence/fin-wp-0004-t05-commissioning-2026-08.md`. T05 still
|
||||
waits on the first Scaleway charge or credit (expected 2026-09).
|
||||
|
||||
## T06 — Generalize and operate the contract
|
||||
|
||||
```task
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue