coulomb/resource-control
4
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(wp-0002): complete T07 — control loop on the live backup resource

Browse source 
The backup is procured and proven, so the loop runs on real evidence.

- data/actuals/2026-08.json: first real observation. database 0.6365 GB,
  stored 0.0066 GB over 8 objects, backup success 1/1, restore RTO 1.08 min.
  Five proxies null, each with a named owner in measurement_gaps.
- data/thresholds/platform-audit-storage.json + tools/thresholds.py: budget
  variance, abnormal growth, stale backup, unused commitment. Fail-closed —
  an unmeasured value is reported as unmeasured, never as within.
- financial_exchange.py gains a usage mode emitting technical_usage records to
  fin-hub, with measurement gaps carried through and no infrastructure amount:
  fin-hub owns the booked fact and a null is never sent as 0.00.
- observation schema 0.2 allows null cost and usage proxies; variance.py fails
  closed rather than reporting a 100% favourable variance on a missing amount.
- platform-audit-storage: ordered -> active, commissioned 2026-08-14, on
  operational fact rather than on the purchase.

The optimization case is now approved by the founder. That needed a schema
change: Host Europe never supplied written terms, so options gained
excluded/exclusion_reason. Previously an unevaluable alternative blocked its
case forever, leaving the record claiming no decision while the bucket was in
production. An excluded option keeps its unknowns and must say what would
bring it back.

August produces no variance and should not: the decision forecast starts at
2026-09, so August is a commissioning baseline. Threshold run is 2 within,
1 not applicable, 6 unmeasured, 0 breaches.

Also fixes a pre-existing test failure: reef-storage consumers_actual is now
rapp-postgres, which the assertion still expected to be empty.

136 tests pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
tegwick 2026-08-14 20:53:12 +02:00
parent 2704292d45
commit 10b988fa1c
17 changed files with 1312 additions and 122 deletions
Download patch file Download diff file Expand all files Collapse all files

46
data/actuals/2026-08.json Normal file
View file

@ -0,0 +1,46 @@
{
"schema_version": "0.2",
"record_type": "usage_observation",
"workload": "rapp-postgres/platform-pg",
"cost_attribution_key": "platform:audit-storage",
"provider_id": "scaleway-standard-multi-az",
"created_at": "2026-08-14T18:24:00Z",
"scenario": "observed",
"forecast_ref": "data/forecasts/platform-audit-storage-scaleway-base-2026-08.json",
"rows": [
{
"period": "2026-08",
"database_gb": 0.6365,
"stored_gb": 0.0066,
"wal_gb": null,
"restore_egress_gb": null,
"write_requests": null,
"read_requests": null,
"infrastructure_eur": null,
"internal_labor_hours": null,
"internal_labor_eur": null,
"total_eur": null,
"backup_success_pct": 100,
"restore_rto_minutes": 1.08,
"measurement_gaps": [
"wal_gb: the 8 objects in prefix platform-pg/ hold the first base backup and its WAL together; the T05 evidence does not separate them",
"restore_egress_gb: two scratch restores read from the bucket but bytes transferred were not recorded (owner: resource-control, add to the next drill)",
"write_requests, read_requests: no request metric is exposed to the scoped backup key (owner: resource-control, needs a Scaleway usage export)",
"infrastructure_eur: no invoice for a resource commissioned 2026-08-14; the Scaleway billing API is denied to this key by design (owner: fin-hub, FIN-WP-0004)",
"internal_labor_hours: procurement, credential custody, and two restore drills were performed in this period but no time record was kept (owner: resource-control)"
]
}
],
"evidence": [
"docs/evidence/RESOURCE-WP-0002-T05-restore-2026-08-14.json",
"data/resources/platform-audit-storage.json",
"reef:storage/substrate/object-stores/platform-audit-storage.yaml"
],
"notes": [
"First real observation for this resource. It went live mid-period: bucket created 2026-08-14T16:21:56Z, first backup 2026-08-14T18:14:21Z, so 2026-08 covers roughly four hours of operation, not a month.",
"Backup success is 1 of 1 completed backups, not a month of scheduled runs.",
"restore_rto_minutes is the full-restore wall time of 65 seconds; PITR measured the same 65 seconds on the second attempt after the recovery target was corrected.",
"Observed RPO: archive_timeout is 5 minutes and a WAL segment archived about 2 seconds after pg_switch_wal, so the measured RPO is well inside the 5-minute requirement — but on four hours of evidence.",
"Comparing this period against the 12-month decision forecast will show a very large negative stored_gb error. That is a model artifact, not underuse: the forecast assumes 30 days of retained base backups and WAL at steady state, and this period holds one backup."
]
}

100
data/optimization/platform-audit-storage-2026-08.json
View file

@ -18,8 +18,14 @@
"recurring_external_labor_eur_month": 0,
"utilization": {
"stored_gb": {
"provisioned": { "value": 457.968, "unit": "GB" },
"used": { "value": 457.968, "unit": "GB" }
"provisioned": {
"value": 457.968,
"unit": "GB"
},
"used": {
"value": 457.968,
"unit": "GB"
}
}
},
"uncertainty": {
@ -31,8 +37,14 @@
]
},
"service_constraints": {
"retention_days": { "value": 30, "unit": "days" },
"target_rpo_minutes": { "value": 5, "unit": "minutes" }
"retention_days": {
"value": 30,
"unit": "days"
},
"target_rpo_minutes": {
"value": 5,
"unit": "minutes"
}
},
"failure_domains": [
"provider:scaleway",
@ -51,8 +63,14 @@
"recurring_external_labor_eur_month": 0,
"utilization": {
"stored_gb": {
"provisioned": { "value": 1000, "unit": "GB" },
"used": { "value": 457.968, "unit": "GB" }
"provisioned": {
"value": 1000,
"unit": "GB"
},
"used": {
"value": 457.968,
"unit": "GB"
}
}
},
"uncertainty": {
@ -63,8 +81,14 @@
]
},
"service_constraints": {
"retention_days": { "value": 30, "unit": "days" },
"target_rpo_minutes": { "value": 5, "unit": "minutes" }
"retention_days": {
"value": 30,
"unit": "days"
},
"target_rpo_minutes": {
"value": 5,
"unit": "minutes"
}
},
"failure_domains": [
"provider:hetzner",
@ -82,8 +106,14 @@
"recurring_external_labor_eur_month": 0,
"utilization": {
"stored_gb": {
"provisioned": { "value": null, "unit": "GB" },
"used": { "value": 457.968, "unit": "GB" }
"provisioned": {
"value": null,
"unit": "GB"
},
"used": {
"value": 457.968,
"unit": "GB"
}
}
},
"uncertainty": {
@ -94,8 +124,14 @@
]
},
"service_constraints": {
"retention_days": { "value": 30, "unit": "days" },
"target_rpo_minutes": { "value": 5, "unit": "minutes" }
"retention_days": {
"value": 30,
"unit": "days"
},
"target_rpo_minutes": {
"value": 5,
"unit": "minutes"
}
},
"failure_domains": [
"provider:host-europe"
@ -106,27 +142,35 @@
"storage_eur_per_gb_month, monthly_minimum_eur, egress_eur_per_gb, operations_eur_per_month, support_eur_per_month",
"cancellation terms and minimum contract period",
"correlated-failure acceptance for same-provider placement (owner: human financial and risk authority)"
]
],
"excluded": true,
"exclusion_reason": "No written account confirmation of current S3 orderability, price, or cancellation terms was obtained within the decision window, and platform-pg needed off-host backup. The deciding authority excluded the option rather than delay recovery capability further. It returns to the comparison if Host Europe supplies written commercial terms; RESOURCE-WP-0002-T02 keeps that gap recorded. Note that even fully priced it would place the backup in the same provider:host-europe failure domain as the host it protects."
}
],
"decision": {
"state": "blocked_on_evidence",
"recommended_option_id": null,
"rationale": "This case exercises the optimization process against real cost-model output; it is not the procurement decision, which RESOURCE-WP-0002-T03 owns and human financial authority approves. Hetzner computes fully and is EUR 29.14 per month more expensive than the Scaleway baseline at month-12 base demand, driven by operator labour rather than storage price. Host Europe cannot be compared at all: its price, cancellation terms, and current orderability are unknown, and it shares a failure domain with the compute host it is meant to protect. No provider is recommended until Host Europe returns written account evidence.",
"approver": null,
"approved_on": null,
"delegated_to": []
"state": "approved",
"recommended_option_id": "scaleway-standard-multi-az",
"rationale": "Approved 2026-08-14 and executed. Scaleway Standard Multi-AZ in nl-ams is the primary; private bucket railiance-platform-pg-backup exists with versioning on, a 30-day current and noncurrent lifecycle, and owner-only ACL. Hetzner was fully costed and rejected: at month-12 base demand it is EUR 29.14 per month dearer, on operator labour rather than storage price, and its published quota pricing cannot evaluate the high scenario. Host Europe was excluded for lack of written commercial terms, not judged inferior on price. The choice is proven, not just purchased: first backup 48s, full restore 65s with audit_core.events 30=30, PITR 65s to a corrected target, 0 failures (RESOURCE-WP-0002-T05). Accepted risks recorded at decision time: Scaleway has no euro hard-stop, so the founder's email billing alert is the backstop; and the independent second copy required by the correlated-failure rule is still outstanding as RESOURCE-WP-0002-T06.",
"approver": "founder (human financial authority)",
"approved_on": "2026-08-14",
"delegated_to": [
"railiance-platform",
"rapp-postgres"
]
},
"financial_handoff": {
"cost_attribution_key": "platform:audit-storage",
"sent": false,
"reference": null
"sent": true,
"reference": "tools/financial_exchange.py usage data/actuals/2026-08.json resource:platform:audit-storage"
},
"outcome": {
"feeds_forecast": [
"data/forecasts/platform-audit-storage-scaleway-base-2026-08.json"
],
"actual_refs": []
"actual_refs": [
"data/actuals/2026-08.json",
"docs/evidence/RESOURCE-WP-0002-T05-restore-2026-08-14.json"
]
},
"evidence": [
{
@ -152,6 +196,18 @@
"ref": "RESOURCE-WP-0002-T03",
"authority": "resource-control",
"observed_at": null
},
{
"kind": "decision",
"ref": "docs/evidence/RESOURCE-WP-0002-primary-selection-2026-08-14.md",
"authority": "founder (human financial authority)",
"observed_at": "2026-08-14"
},
{
"kind": "telemetry",
"ref": "docs/evidence/RESOURCE-WP-0002-T05-restore-2026-08-14.json",
"authority": "reef-railiance CNPG / Scaleway object store",
"observed_at": "2026-08-14"
}
]
}

75
data/resources/platform-audit-storage.json
View file

@ -32,11 +32,15 @@
"secret:railiance-platform/backup"
],
"consumers": {
"potential": ["rapp-postgres"],
"actual": ["rapp-postgres"]
"potential": [
"rapp-postgres"
],
"actual": [
"rapp-postgres"
]
},
"resource_class": "storage",
"status": "ordered",
"status": "active",
"management_model": "provider_managed",
"provider": {
"name": "Scaleway",
@ -53,16 +57,28 @@
"location": {
"region": "nl-ams",
"country": "NL",
"failure_domains": ["provider:scaleway", "region:nl-ams", "reef:storage"],
"failure_domains": [
"provider:scaleway",
"region:nl-ams",
"reef:storage"
],
"residency": "European Union"
},
"capacity": [
{"metric": "stored_data", "value": null, "unit": "GB", "kind": "unknown", "observed_at": null}
{
"metric": "stored_data",
"value": null,
"unit": "GB",
"kind": "unknown",
"observed_at": null
}
],
"ownership": {
"owner": "resource-control",
"environment": "production",
"workload_ids": ["rapp-postgres/platform-pg"],
"workload_ids": [
"rapp-postgres/platform-pg"
],
"tenant_id": null,
"allocation": {
"mode": "dedicated",
@ -81,7 +97,7 @@
"lifecycle": {
"proposed_on": "2026-08-10",
"ordered_on": "2026-08-14",
"commissioned_on": null,
"commissioned_on": "2026-08-14",
"renews_on": null,
"cancel_by": null,
"retired_on": null,
@ -89,14 +105,45 @@
},
"relationships": [],
"requirements": [
{"kind": "recovery", "ref": "workplans/RESOURCE-WP-0002-procure-postgres-backup-storage.md#acceptance-requirements"},
{"kind": "retention", "ref": "data/demand/platform-audit-storage.json"}
{
"kind": "recovery",
"ref": "workplans/RESOURCE-WP-0002-procure-postgres-backup-storage.md#acceptance-requirements"
},
{
"kind": "retention",
"ref": "data/demand/platform-audit-storage.json"
}
],
"evidence": [
{"kind": "provider", "ref": "https://www.scaleway.com/en/pricing/storage/", "observed_at": "2026-08-10", "authority": "Scaleway"},
{"kind": "provider", "ref": "https://www.scaleway.com/en/object-storage/", "observed_at": "2026-08-10", "authority": "Scaleway"},
{"kind": "decision", "ref": "docs/evidence/RESOURCE-WP-0002-provider-due-diligence-2026-08-10.md", "observed_at": "2026-08-10", "authority": "resource-control"},
{"kind": "decision", "ref": "docs/evidence/RESOURCE-WP-0002-primary-selection-2026-08-14.md", "observed_at": "2026-08-14", "authority": "resource-control"},
{"kind": "provider", "ref": "reef:storage/substrate/object-stores/platform-audit-storage.yaml", "observed_at": "2026-08-14", "authority": "reef-storage"}
{
"kind": "provider",
"ref": "https://www.scaleway.com/en/pricing/storage/",
"observed_at": "2026-08-10",
"authority": "Scaleway"
},
{
"kind": "provider",
"ref": "https://www.scaleway.com/en/object-storage/",
"observed_at": "2026-08-10",
"authority": "Scaleway"
},
{
"kind": "decision",
"ref": "docs/evidence/RESOURCE-WP-0002-provider-due-diligence-2026-08-10.md",
"observed_at": "2026-08-10",
"authority": "resource-control"
},
{
"kind": "decision",
"ref": "docs/evidence/RESOURCE-WP-0002-primary-selection-2026-08-14.md",
"observed_at": "2026-08-14",
"authority": "resource-control"
},
{
"kind": "provider",
"ref": "reef:storage/substrate/object-stores/platform-audit-storage.yaml",
"observed_at": "2026-08-14",
"authority": "reef-storage"
}
]
}

93
data/thresholds/platform-audit-storage.json Normal file
View file

@ -0,0 +1,93 @@
{
"schema_version": "0.1",
"resource_id": "resource:platform:audit-storage",
"cost_attribution_key": "platform:audit-storage",
"effective_from": "2026-08-14",
"review": "quarterly, and before any contract or retention change",
"basis": "docs/forecast-actual-control.md error thresholds, plus the operating facts proven in RESOURCE-WP-0002-T05",
"thresholds": [
{
"id": "budget-variance-infrastructure",
"kind": "budget_variance",
"metric": "infrastructure_eur",
"comparison": "absolute_percentage_error",
"limit": 10,
"action": "investigate before accepting the period; separate usage error from rate, tax, or SKU change"
},
{
"id": "budget-variance-labor",
"kind": "budget_variance",
"metric": "internal_labor_hours",
"comparison": "absolute_error",
"limit": 1,
"unit": "hours",
"action": "investigate; recurring operator time above forecast is the usual reason a cheap provider is not cheap"
},
{
"id": "abnormal-growth-stored",
"kind": "abnormal_growth",
"metric": "stored_gb",
"comparison": "absolute_percentage_error",
"limit": 20,
"action": "check retention behaviour, incomplete multipart uploads, versioning, and base-backup frequency before assuming demand grew"
},
{
"id": "abnormal-growth-wal",
"kind": "abnormal_growth",
"metric": "wal_gb",
"comparison": "absolute_percentage_error",
"limit": 20,
"action": "check for a write-amplifying change in platform-pg before re-forecasting"
},
{
"id": "unplanned-egress",
"kind": "abnormal_growth",
"metric": "restore_egress_gb",
"comparison": "unplanned",
"limit": 0,
"action": "any egress not explained by a recorded restore drill is investigated as an access or exfiltration question, not only a cost question"
},
{
"id": "stale-backup",
"kind": "stale_backup",
"metric": "backup_success_pct",
"comparison": "minimum",
"limit": 100,
"action": "a single failed scheduled backup is investigated; backup success is not averaged across a month"
},
{
"id": "stale-archive",
"kind": "stale_backup",
"metric": "wal_archive_gap_minutes",
"comparison": "maximum",
"limit": 15,
"unit": "minutes",
"action": "three times the 5-minute archive_timeout; beyond this the 5-minute RPO requirement is no longer met",
"measured_baseline": "about 2 seconds after pg_switch_wal, observed 2026-08-14"
},
{
"id": "restore-rto-regression",
"kind": "stale_backup",
"metric": "restore_rto_minutes",
"comparison": "maximum",
"limit": 3.24,
"unit": "minutes",
"action": "three times the measured 1.08-minute full-restore baseline; a regression is investigated before the next drill is accepted",
"measured_baseline": "65 seconds full restore and 65 seconds PITR, observed 2026-08-14 on 636 MB of PGDATA"
},
{
"id": "unused-commitment",
"kind": "unused_commitment",
"metric": "commitment_utilization",
"comparison": "minimum",
"limit": null,
"status": "not_applicable",
"action": "Scaleway Multi-AZ is usage-based with no commitment (commitment_ref is null), so there is no unused commitment to detect. This threshold activates only if a term or reserved commitment is ever taken."
}
],
"known_gaps": [
"No euro budget ceiling exists for this resource. Scaleway has no hard spend stop and the founder's email alert is the accepted backstop; an authoritative ceiling belongs to fin-hub (FIN-WP-0004).",
"wal_archive_gap_minutes is not yet emitted by any monthly observation; it requires an archive-health metric from rapp-postgres.",
"restore_rto_minutes is populated only in periods that contain a drill; absence is not a pass."
]
}