From 234b1b559fe5f5fc328719c6e5ce4cdd3c5383cf Mon Sep 17 00:00:00 2001 From: codex Date: Sat, 5 Sep 2026 11:43:55 +0200 Subject: [PATCH] Implement S3 service assurance and admission checks Assistant: codex Assistant-Model: gpt-6-astra Assistant-Session: 01a06ecb-456a-71c2-b41e-0755d336e883 --- Makefile | 12 + SCOPE.md | 15 +- WORK-RECORDS.md | 14 +- assurance/admission-baseline.json | 95 +++++++ assurance/ownership-handoffs.json | 131 ++++++++++ assurance/placement-owners.json | 9 + assurance/service-contract.json | 96 +++++++ assurance/service-records.json | 198 ++++++++++++++ .../ADR-0004-service-assurance-disclosure.md | 64 +++++ ...ADR-0005-demand-gated-stateful-services.md | 55 ++++ .../RPF-WP-0036-assurance-2026-09-05.json | 242 ++++++++++++++++++ docs/placement-policy.md | 26 +- docs/platform-ownership-handoffs.md | 49 ++++ docs/s3-consumer-interfaces.md | 23 +- docs/service-assurance.md | 105 ++++++++ ...09-05-platform-assurance-implementation.md | 79 ++++++ scripts/capture_service_observation.py | 126 +++++++++ scripts/service_assurance.py | 183 +++++++++++++ tests/test_service_assurance.py | 162 ++++++++++++ workplans/README.md | 10 +- .../RPF-WP-0036-platform-service-assurance.md | 64 ++++- 21 files changed, 1728 insertions(+), 30 deletions(-) create mode 100644 assurance/admission-baseline.json create mode 100644 assurance/ownership-handoffs.json create mode 100644 assurance/placement-owners.json create mode 100644 assurance/service-contract.json create mode 100644 assurance/service-records.json create mode 100644 docs/adr/ADR-0004-service-assurance-disclosure.md create mode 100644 docs/adr/ADR-0005-demand-gated-stateful-services.md create mode 100644 docs/evidence/RPF-WP-0036-assurance-2026-09-05.json create mode 100644 docs/platform-ownership-handoffs.md create mode 100644 docs/service-assurance.md create mode 100644 history/2026-09-05-platform-assurance-implementation.md create mode 100644 scripts/capture_service_observation.py create mode 100644 scripts/service_assurance.py create mode 100644 tests/test_service_assurance.py diff --git a/Makefile b/Makefile index 5c8d2c1..b524cc0 100644 --- a/Makefile +++ b/Makefile @@ -550,3 +550,15 @@ wp0024-t02-driver-register: ## Register committed audit-core driver source witho .PHONY: audit-whitehat-e2-preflight audit-whitehat-e2-project audit-whitehat-e2-cleanup audit-database-lease-recovery-preflight audit-database-lease-approval-check audit-database-lease-recovery-exercise railiance01-reboot-preflight wp0024-owner-show wp0024-owner-verify wp0024-owner-status wp0024-t02-driver-show wp0024-t02-driver-status wp0024-t02-driver-register apps-pg-isolation-probe .PHONY: db-deploy db-status db-shell db-logs forgejo-db-deploy forgejo-db-status forgejo-db-shell apps-pg-deploy apps-pg-status apps-pg-shell apps-pg-logs net-kingdom-pg-inter-hub-networkpolicy-deploy pg-deploy pg-status pg-pgpool-check valkey-deploy valkey-status openbao-repo openbao-dry-run openbao-overlay-apply openbao-verify-login-overlay openbao-deploy openbao-status openbao-verify openbao-verify-post-unseal openbao-configure-initial openbao-configure-ssh openbao-verify-ssh openbao-verify-authenticated openbao-configure-external-secrets-issue-core openbao-configure-external-secrets-activity-core openbao-configure-external-secrets-forgejo openbao-validate-restore-evidence openbao-validate-emergency-evidence credential-grants-validate credential-change-applier-dry-run credential-change-applier-apply-plan credential-change-applier-apply credential-change-runbook credential-change-record-evidence credential-change-lifecycle-plan credential-change-lifecycle-event credential-change-import-inventory agent-high-risk-boundary-check openbao-credential-change-appliers-dry-run openbao-configure-credential-change-appliers openbao-token-grants-dry-run openbao-configure-token-grants openbao-verify-token-grants-dry-run openbao-verify-token-grants openbao-verify-token-grants-smoke credential-helper-dry-run credential-tests credential-exec-ops-warden-smoke argocd-bootstrap-dry-run argocd-bootstrap-deploy argocd-repo-apply argocd-status backup forgejo-backup forgejo-backup-dry-run forgejo-backup-status forgejo-package-prune forgejo-package-prune-dry-run reef-deployables apps-pg-evidence consumption-preflight help + +##@ S3 service assurance (read-only) +.PHONY: assurance-admission assurance-capture assurance-check +ASSURANCE_OBSERVATION ?= /tmp/railiance-platform-observation.json +assurance-admission: ## Verify owner-native admission and disclosure baseline + python3 scripts/service_assurance.py admission --check assurance/admission-baseline.json + +assurance-capture: ## Capture selected status metadata from the pinned railiance01 cluster + python3 scripts/capture_service_observation.py --output "$(ASSURANCE_OBSERVATION)" + +assurance-check: ## Classify freshness/failure; nonzero means incomplete or invalid evidence + python3 scripts/service_assurance.py evaluate "$(ASSURANCE_OBSERVATION)" diff --git a/SCOPE.md b/SCOPE.md index 897cc3b..731c415 100644 --- a/SCOPE.md +++ b/SCOPE.md @@ -22,7 +22,7 @@ or authorization authority. See [ADR-0001](docs/adr/ADR-0001-s3-platform-service | Service | Implemented / evidenced | Limit and authority | | --- | --- | --- | | `apps-pg` | CNPG source, per-consumer database ACLs, 20-connection allowances, 15-second timeouts; daily backup, continuous WAL and 30-day retention; 56-second scratch restore and 14/14 boundary probes on 2026-08-20 | One instance; restart recovery, no HA or measured latency SLO. Ceiling 3, overflow `apps-pg-2` provisionable but demand-gated. S3 owns this CR and its controls. | -| `platform-pg` | Package-operated CNPG service, governed dynamic credentials, backup/restore evidence, Core Hub admission and private shadow evidence | `rapp-postgres` owns CR/provisioning/restore. S3 owns custody, placement policy and consumer disclosure. Published ceiling 4 declarations, counting the probe; reconcile current occupancy before admission. One instance is not HA. | +| `platform-pg` | Package-operated CNPG service, governed dynamic credentials, backup/restore evidence, Core Hub admission and private shadow evidence | `rapp-postgres` owns CR/provisioning/restore. S3 owns custody, placement policy and consumer disclosure. Ceiling 4 declarations, counting the probe; platform-pg is 4/4 and the deployed platform-pg-2 hosts sbom-nexus at 1/4. One instance is not HA. | | OpenBao and credential delivery | CCR validation, policy/grant boundaries, approved apply interfaces, runtime KV/ESO and short-lived database lanes, ephemeral custody lifecycle | Package assets belong to `rapp-openbao`; custody/policy remains here. Existing helpers are not authority to mint arbitrary credentials. Operator-only access cutover and incident residuals remain open. | | Backup storage consumption | Governed off-cluster S3 backup destination and credential lane used by CNPG; encrypted upload tooling for other backup consumers | Resource procurement belongs to `resource-control`/`reef-storage`. This does not establish a general S3 service or deployed MinIO. One successful restore does not establish a recurring recovery guarantee. | | Cache / general object storage / messaging | Valkey deployment is gated; no supported consumer interface or deployed shared broker/general in-cluster object store is established by this review | Demand, reuse, ownership, service contract and recovery design must precede deployment. No engine/provider choice is made here. | @@ -69,7 +69,8 @@ records every reviewed plan and the consolidation mapping. | Next | Numeric availability/recovery promises, evidence freshness, recurring drills, emission and admission drift lack a complete S3 acceptance loop | RPF-WP-0036-T02–T05 | | Next | Compatibility ownership, stale Hub aliases, and undeployed capability demand need explicit disposition | RPF-WP-0036-T06/T07 | -The ready work in RPF-WP-0036 can proceed while attended operations wait. +RPF-WP-0036 now provides local assurance/admission checks and explicit service +disclosures; its remaining tasks wait for live proof and owner acceptance. It must define achievable promises for the existing single-node substrate, not declare HA through documentation. Any stronger availability target needs an S1/S2 dependency and a package-level failover proof. Fleet-wide Q3 ownership @@ -112,3 +113,13 @@ Read [AGENTS.md](AGENTS.md), [INTENT.md](INTENT.md), then `workplans/archived/`; [WORK-RECORDS.md](WORK-RECORDS.md) is the generated current index (archives omitted). Routine sync follows AGENTS.md. The dated `.custodian-brief.md` and Hub legacy aliases are not additional authoritative obligations. + +## Assurance implementation — 2026-09-05 + +`make assurance-admission` checks package-native capacity/retention and source +disclosure drift. `make assurance-capture` pins the cluster and collects only +status metadata; `make assurance-check` fails on incomplete/stale/failed evidence. +The first live run found all three database cells Ready with same-day backups +and archiving, but three failing ESO resources. Recurring restore acceptance, +validated restore adapters and Q2 delivery remain open. See +[service assurance](docs/service-assurance.md) and RPF-WP-0036. diff --git a/WORK-RECORDS.md b/WORK-RECORDS.md index 5aa346c..f4fafc4 100644 --- a/WORK-RECORDS.md +++ b/WORK-RECORDS.md @@ -13,7 +13,7 @@ | workplan | RPF-WP-0027 | blocked | — | workplans/RPF-WP-0027-keycape-live-secret-exposure-recovery.md | | workplan | RPF-WP-0029 | blocked | — | workplans/RPF-WP-0029-backup-credential-default-removal.md | | workplan | RPF-WP-0035 | blocked | — | workplans/RPF-WP-0035-credential-lane-implementation.md | -| workplan | RPF-WP-0036 | ready | — | workplans/RPF-WP-0036-platform-service-assurance.md | +| workplan | RPF-WP-0036 | blocked | — | workplans/RPF-WP-0036-platform-service-assurance.md | | task | RPF-WP-0015-T01 | done | — | workplans/RPF-WP-0015-audit-core-custody-and-recovery-coordination.md | | task | RPF-WP-0015-T02 | wait | — | workplans/RPF-WP-0015-audit-core-custody-and-recovery-coordination.md | | task | RPF-WP-0015-T03 | wait | — | workplans/RPF-WP-0015-audit-core-custody-and-recovery-coordination.md | @@ -34,9 +34,9 @@ | task | RPF-WP-0035-T03 | wait | — | workplans/RPF-WP-0035-credential-lane-implementation.md | | task | RPF-WP-0035-T04 | wait | — | workplans/RPF-WP-0035-credential-lane-implementation.md | | task | RPF-WP-0036-T01 | done | — | workplans/RPF-WP-0036-platform-service-assurance.md | -| task | RPF-WP-0036-T02 | todo | — | workplans/RPF-WP-0036-platform-service-assurance.md | -| task | RPF-WP-0036-T03 | todo | — | workplans/RPF-WP-0036-platform-service-assurance.md | -| task | RPF-WP-0036-T04 | todo | — | workplans/RPF-WP-0036-platform-service-assurance.md | -| task | RPF-WP-0036-T05 | todo | — | workplans/RPF-WP-0036-platform-service-assurance.md | -| task | RPF-WP-0036-T06 | todo | — | workplans/RPF-WP-0036-platform-service-assurance.md | -| task | RPF-WP-0036-T07 | todo | — | workplans/RPF-WP-0036-platform-service-assurance.md | +| task | RPF-WP-0036-T02 | done | — | workplans/RPF-WP-0036-platform-service-assurance.md | +| task | RPF-WP-0036-T03 | wait | — | workplans/RPF-WP-0036-platform-service-assurance.md | +| task | RPF-WP-0036-T04 | wait | — | workplans/RPF-WP-0036-platform-service-assurance.md | +| task | RPF-WP-0036-T05 | done | — | workplans/RPF-WP-0036-platform-service-assurance.md | +| task | RPF-WP-0036-T06 | wait | — | workplans/RPF-WP-0036-platform-service-assurance.md | +| task | RPF-WP-0036-T07 | done | — | workplans/RPF-WP-0036-platform-service-assurance.md | diff --git a/assurance/admission-baseline.json b/assurance/admission-baseline.json new file mode 100644 index 0000000..45b28bb --- /dev/null +++ b/assurance/admission-baseline.json @@ -0,0 +1,95 @@ +{ + "schema": "railiance-platform.admission.v1", + "basis": "source-declarations", + "cells": { + "apps-pg": { + "consumers": [ + "coulomb_social", + "vergabe" + ], + "ceiling": 3, + "retention_days": 30, + "instances": 1, + "memory_limit": "1Gi", + "max_connections": 100, + "service_classes": { + "coulomb_social": "interactive", + "vergabe": "interactive" + }, + "connection_limits": { + "vergabe": 20, + "coulomb_social": 20 + }, + "owners": { + "coulomb_social": "coulomb-social", + "vergabe": "vergabe-teilnahme" + } + }, + "apps-pg-2": { + "consumers": [], + "ceiling": 3, + "retention_days": 30, + "instances": 1, + "memory_limit": "1Gi", + "max_connections": 100, + "service_classes": {}, + "connection_limits": {}, + "owners": {} + }, + "platform-pg": { + "consumers": [ + "audit-core", + "core-hub", + "isolation-probe", + "tenant-engine" + ], + "ceiling": 4, + "retention_days": 30, + "instances": 1, + "memory_limit": "1Gi", + "max_connections": 100, + "service_classes": { + "audit-core": "batch", + "core-hub": "latency-critical", + "isolation-probe": "interactive", + "tenant-engine": "latency-critical" + }, + "owners": { + "audit-core": "audit-core", + "core-hub": "core-hub", + "isolation-probe": "rapp-postgres", + "tenant-engine": "tenant-engine" + } + }, + "platform-pg-2": { + "consumers": [ + "sbom-nexus" + ], + "ceiling": 4, + "retention_days": 30, + "instances": 1, + "memory_limit": "1Gi", + "max_connections": 100, + "service_classes": { + "sbom-nexus": "interactive" + }, + "owners": { + "sbom-nexus": "sbom-nexus" + } + } + }, + "sources": { + "railiance-platform/helm/apps-pg-cluster.yaml": "9885c81338e1c3bdb0558839552315720068c5d355819e4a1320fbd677d0ffcf", + "railiance-platform/helm/apps-pg-2-cluster.yaml": "e9e615a0b5de8597a5501b2d8740a763ba3622cbebb765ce651236c2175de942", + "rapp-postgres/consumers/audit-core.yaml": "a2db53f901df056546b2c028f287f8fc50486e6e6cd81026b9e926b6b64272b3", + "rapp-postgres/consumers/core-hub.yaml": "8fe3e8b4488548b2f5e01f1d46edd9fd5fe4f0e4f65a3821f172adbfb5ae1d84", + "rapp-postgres/consumers/isolation-probe.yaml": "8ea24d3bd04ab683b47ceff2586932ea4fdb0d45c49896167bb5b352a67a5cfb", + "rapp-postgres/consumers/sbom-nexus.yaml": "0b21d05fe7b9d21cbc9eb761f455329739ceb71f8dd3d1200971426962c11236", + "rapp-postgres/consumers/tenant-engine.yaml": "92f36d587f001c86ff401aedd25cb49fa171d49e000b0a6ab551e182e5c11448", + "railiance-platform/tools/verify_apps_pg_capacity.py": "b571ec63b0a7501dabba37a84af9d2cfcb7737ed891a7ea2ab05c54010b1893e", + "railiance-platform/assurance/placement-owners.json": "b9a52f755552dc4bb83536dca8aadca691a4f8bc47089e9e1962902abcec7aa2", + "rapp-postgres/scripts/render_consumers.py": "2dcfd68f94a3d6c7e1072c4a55a497b1697eab30acaecae53865c9b1a596773a", + "rapp-postgres/helm/platform-pg-cluster.yaml": "461da0d43e61a4ddca29d6e98080b8b1642bd3303f3395da1b0beb28775c2c5c", + "rapp-postgres/helm/platform-pg-2-cluster.yaml": "06b92a76189b1a33d1eee3f5c835a3e94269ad7b040383c04e8028c2c3fe766a" + } +} diff --git a/assurance/ownership-handoffs.json b/assurance/ownership-handoffs.json new file mode 100644 index 0000000..3e5ae36 --- /dev/null +++ b/assurance/ownership-handoffs.json @@ -0,0 +1,131 @@ +{ + "schema": "railiance-platform.ownership-handoffs.v1", + "reviewed": "2026-09-05", + "decision": "retain existing operational entry points through review; no bulk move", + "records": [ + { + "surface": "forge-runtime", + "proposed_accepting_owner": "railiance-forge", + "status": "retained-pending-acceptance", + "review_due": "2026-10-05", + "callers_to_verify": "activity-core command definitions and railiance-apps compatibility callers", + "source_files": [ + { + "path": "tools/cmd/forgejo-backup", + "sha256": "a20f0aebb22f0978c0f45f74e4ac55a927b080910844296acabec10f45110cb6" + }, + { + "path": "tools/cmd/forgejo-package-prune", + "sha256": "66bababca54931115831aa42b7ad95bb4ee348d4057b5bfd1f5a6ae13cfa8ea3" + }, + { + "path": "scripts/forgejo_package_prune.py", + "sha256": "63ba4df1c937a0067ea3bb4b1efbe4cfa47205f34e80f337ca71e4a0e6a31db4" + }, + { + "path": "tools/cmd/refresh-live-images", + "sha256": "341676396e2c2a85f4585fcbf5c7e085805c9228eae8961740e19960ef2ba5ea" + }, + { + "path": "scripts/refresh_live_images.py", + "sha256": "d1bcd88f3b757d98fa4cfa033329fe7b230f7bf1c6b466fc9cccc6804edf12b6" + }, + { + "path": "docs/forgejo-backup.md", + "sha256": "69f6f38de902b80c6de161087b1d5738d1dea7ea35b3d49816d2ea67a1a5329b" + }, + { + "path": "docs/forgejo-package-prune.md", + "sha256": "4f389876f52ba273f6a43c2d7d721a64c520ed9620e4a04899a423775224572a" + } + ], + "acceptance": "Owner names canonical replacements; update all callers and prove dry-run/smoke and rollback before source removal. No external acceptance asserted." + }, + { + "surface": "openbao-package", + "proposed_accepting_owner": "rapp-openbao", + "status": "retained-pending-acceptance", + "review_due": "2026-10-05", + "callers_to_verify": "Makefile package dispatch and existing operator entry points", + "source_files": [ + { + "path": "scripts/openbao-verify.sh", + "sha256": "1e118570c8682619a46413f55fa4690482db342f6b6c83e6644c908aa7a667ee" + }, + { + "path": "scripts/openbao-verify-login-overlay.sh", + "sha256": "57438714f1b7aac0c37b8bdf525c5387f812a6d30532bb98b395e5db5208ad6d" + }, + { + "path": "docs/rapp-openbao-compatibility-handoff.md", + "sha256": "65ff0b39ee7312c8a242d24cf3434f9d0636a830e841f1a70de560c96cd8d57f" + } + ], + "acceptance": "Owner names canonical replacements; update all callers and prove dry-run/smoke and rollback before source removal. No external acceptance asserted." + }, + { + "surface": "gitops-bootstrap", + "proposed_accepting_owner": "railiance-cluster / railiance-enablement", + "status": "retained-pending-acceptance", + "review_due": "2026-10-05", + "callers_to_verify": "cluster bootstrap and generic GitOps entry points", + "source_files": [ + { + "path": "argocd/bootstrap/00-railiance-bootstrap-project.yaml", + "sha256": "1fcdff98b1870e8867df65deeabd759e1357f0359af50ac6229f12ea3ec695ea" + }, + { + "path": "argocd/bootstrap/01-railiance-tenants-project.yaml", + "sha256": "cf1823ed2294b874730de6d908c0f471d9b67f9ac07bb51a1456de6f77154d07" + }, + { + "path": "argocd/bootstrap/02-railiance-platform-addons-project.yaml", + "sha256": "10b4596dcf6ac369329794612b2f43469a6f4ea1fa99759f92d1d66a50841cf8" + }, + { + "path": "argocd/bootstrap/10-railiance-apps-root.application.yaml", + "sha256": "0eadd51880488442b6c8ed31470fa27c123637b97d66c56a2489d06a03a3c0ba" + }, + { + "path": "argocd/bootstrap/kustomization.yaml", + "sha256": "5eaa17dc142ad134ef044fa9914eaea3ecaa5572b705be8e89660ecdb8a15b99" + } + ], + "acceptance": "Owner names canonical replacements; update all callers and prove dry-run/smoke and rollback before source removal. No external acceptance asserted." + }, + { + "surface": "application-releases", + "proposed_accepting_owner": "railiance-apps / individual package owners", + "status": "retained-pending-acceptance", + "review_due": "2026-10-05", + "callers_to_verify": "ArgoCD application reconciliation", + "source_files": [ + { + "path": "argocd/applications/target-revenue.application.yaml", + "sha256": "c59ff45f7cf5dae47cf39c32fbcde3a614aa79be6a9edb07079f833cde1fa38b" + }, + { + "path": "argocd/applications/external-secrets.application.yaml", + "sha256": "545d560c35ab1515f444099847730d7e9d8b7bd14beb30878ebc2f4bd99a5080" + }, + { + "path": "argocd/applications/openbao-secretstore.application.yaml", + "sha256": "01ca88f200be6c23926b0ccf17ef7a3ab0d247044469edd92e9571820119943b" + }, + { + "path": "argocd/applications/issue-core.application.yaml", + "sha256": "079fdb118996eed8e579f6a658dc11a153c8fc20998daf6379533d30eb339654" + } + ], + "acceptance": "Owner names canonical replacements; update all callers and prove dry-run/smoke and rollback before source removal. No external acceptance asserted." + }, + { + "surface": "derived-record-aliases", + "proposed_accepting_owner": "repo-manager / state-hub", + "status": "retained-pending-owner-repair", + "review_due": "2026-10-05", + "source_ref": "history/2026-09-05-platform-intent-workplan-assessment.md#derived-state-caveat", + "acceptance": "Scoped legacy alias retirement and generated brief agree with canonical source; no UUID edits or blanket retirement." + } + ] +} diff --git a/assurance/placement-owners.json b/assurance/placement-owners.json new file mode 100644 index 0000000..96e723d --- /dev/null +++ b/assurance/placement-owners.json @@ -0,0 +1,9 @@ +{ + "vergabe": {"owner": "vergabe-teilnahme", "cell": "apps-pg", "service_class": "interactive", "evidence": "docs/evidence/RPF-WP-0019-isolation-2026-08-20.md"}, + "coulomb_social": {"owner": "coulomb-social", "cell": "apps-pg", "service_class": "interactive", "evidence": "docs/evidence/RPF-WP-0019-isolation-2026-08-20.md"}, + "audit-core": {"owner": "audit-core", "cell": "platform-pg", "evidence": "rapp-postgres/consumers/audit-core.yaml"}, + "tenant-engine": {"owner": "tenant-engine", "cell": "platform-pg", "evidence": "rapp-postgres/docs/evidence/tenant-engine-postgres-cutover-2026-08-21.md"}, + "core-hub": {"owner": "core-hub", "cell": "platform-pg", "evidence": "docs/evidence/core-hub-postgres-capacity-admission-2026-08-20.md"}, + "isolation-probe": {"owner": "rapp-postgres", "cell": "platform-pg", "evidence": "rapp-postgres/consumers/isolation-probe.yaml"}, + "sbom-nexus": {"owner": "sbom-nexus", "cell": "platform-pg-2", "evidence": "rapp-postgres/docs/evidence/RAPP-POSTGRES-WP-0005-T04-boundary-restore-2026-08-22.md"} +} diff --git a/assurance/service-contract.json b/assurance/service-contract.json new file mode 100644 index 0000000..07a0346 --- /dev/null +++ b/assurance/service-contract.json @@ -0,0 +1,96 @@ +{ + "schema": "railiance-platform.assurance-contract.v1", + "cluster_uid": "a553c742-0115-43d4-99a4-a5ca56fe0786", + "capture_max_age_seconds": 900, + "threshold_status": "local-diagnostic-only", + "signals": { + "apps-pg.ready": { + "owner": "railiance-platform", + "max_age_seconds": 900 + }, + "apps-pg.backup": { + "owner": "railiance-platform", + "max_age_seconds": 129600 + }, + "apps-pg.wal": { + "owner": "railiance-platform", + "max_age_seconds": 900 + }, + "apps-pg.restore": { + "owner": "railiance-platform", + "max_age_seconds": 2592000 + }, + "apps-pg.headroom": { + "owner": "railiance-platform", + "max_age_seconds": 900 + }, + "platform-pg.ready": { + "owner": "rapp-postgres", + "max_age_seconds": 900 + }, + "platform-pg.backup": { + "owner": "rapp-postgres", + "max_age_seconds": 129600 + }, + "platform-pg.wal": { + "owner": "rapp-postgres", + "max_age_seconds": 900 + }, + "platform-pg.restore": { + "owner": "rapp-postgres", + "max_age_seconds": 2592000 + }, + "platform-pg.headroom": { + "owner": "rapp-postgres", + "max_age_seconds": 900 + }, + "platform-pg-2.ready": { + "owner": "rapp-postgres", + "max_age_seconds": 900 + }, + "platform-pg-2.backup": { + "owner": "rapp-postgres", + "max_age_seconds": 129600 + }, + "platform-pg-2.wal": { + "owner": "rapp-postgres", + "max_age_seconds": 900 + }, + "platform-pg-2.restore": { + "owner": "rapp-postgres", + "max_age_seconds": 2592000 + }, + "platform-pg-2.headroom": { + "owner": "rapp-postgres", + "max_age_seconds": 900 + }, + "openbao.seal": { + "owner": "railiance-platform", + "max_age_seconds": 900 + }, + "openbao.snapshot": { + "owner": "railiance-platform", + "max_age_seconds": 129600 + }, + "openbao.restore": { + "owner": "railiance-platform", + "max_age_seconds": 2592000 + }, + "offsite.upload": { + "owner": "railiance-platform", + "max_age_seconds": 129600 + }, + "offsite.restore": { + "owner": "railiance-platform", + "max_age_seconds": 2592000 + }, + "eso.ready": { + "owner": "railiance-platform", + "max_age_seconds": 900 + }, + "eso.refresh": { + "owner": "railiance-platform", + "max_age_seconds": 3600 + } + } +} diff --git a/assurance/service-records.json b/assurance/service-records.json new file mode 100644 index 0000000..2135e04 --- /dev/null +++ b/assurance/service-records.json @@ -0,0 +1,198 @@ +{ + "schema": "railiance-platform.service-records.v1", + "reviewed": "2026-09-05", + "review_owner": "railiance-platform", + "review_scope": "S3 disclosure of unsupported guarantees; not external package approval", + "services": [ + { + "service": "apps-pg", + "accountable_owner": "railiance-platform", + "package_or_consumer_owner": "railiance-platform", + "operator_role": "attended platform operator; application proof by consumer owner", + "consumers": [ + "vergabe", + "coulomb_social" + ], + "failure_domain": "single-node railiance01", + "availability": { + "status": "unsupported", + "target": null, + "decision_owner": "railiance-platform + railiance-platform" + }, + "rpo": { + "status": "unsupported", + "target_seconds": null, + "decision_owner": "railiance-platform + railiance-platform" + }, + "rto": { + "status": "unsupported", + "target_seconds": null, + "decision_owner": "railiance-platform + railiance-platform" + }, + "retention": "30 days", + "existing_evidence": "docs/evidence/RPF-WP-0019-backup-restore-2026-08-20.md", + "recovery_custody": "OpenBao: 2-of-3 operator quorum plus independent encrypted snapshot custody; database/offsite: governed backup lane and separately available restore key. Availability not verified in this task.", + "maintenance_abort": "docs/railiance01-coordinated-reboot.md; stop before mutation when freshness, quorum, consumer readiness or named abort operator is absent", + "freshness_policy": "assurance/service-contract.json; diagnostic thresholds only, no installed cadence approval", + "requirement_assessment": "No accepted numeric consumer availability/RPO/RTO requirement found in the reviewed contracts. Service classes inform placement, not guarantees. Refuse any request for guaranteed HA/node-loss recovery until matched to supported substrate and package proof." + }, + { + "service": "platform-pg", + "accountable_owner": "railiance-platform", + "package_or_consumer_owner": "rapp-postgres", + "operator_role": "attended platform operator; application proof by consumer owner", + "consumers": [ + "audit-core", + "tenant-engine", + "core-hub", + "isolation-probe" + ], + "failure_domain": "single-node railiance01", + "availability": { + "status": "unsupported", + "target": null, + "decision_owner": "railiance-platform + rapp-postgres" + }, + "rpo": { + "status": "unsupported", + "target_seconds": null, + "decision_owner": "railiance-platform + rapp-postgres" + }, + "rto": { + "status": "unsupported", + "target_seconds": null, + "decision_owner": "railiance-platform + rapp-postgres" + }, + "retention": "30 days", + "existing_evidence": "rapp-postgres/docs/evidence/backup-restore-2026-08-13.md", + "recovery_custody": "OpenBao: 2-of-3 operator quorum plus independent encrypted snapshot custody; database/offsite: governed backup lane and separately available restore key. Availability not verified in this task.", + "maintenance_abort": "docs/railiance01-coordinated-reboot.md; stop before mutation when freshness, quorum, consumer readiness or named abort operator is absent", + "freshness_policy": "assurance/service-contract.json; diagnostic thresholds only, no installed cadence approval", + "requirement_assessment": "No accepted numeric consumer availability/RPO/RTO requirement found in the reviewed contracts. Service classes inform placement, not guarantees. Refuse any request for guaranteed HA/node-loss recovery until matched to supported substrate and package proof." + }, + { + "service": "platform-pg-2", + "accountable_owner": "railiance-platform", + "package_or_consumer_owner": "rapp-postgres", + "operator_role": "attended platform operator; application proof by consumer owner", + "consumers": [ + "sbom-nexus" + ], + "failure_domain": "single-node railiance01", + "availability": { + "status": "unsupported", + "target": null, + "decision_owner": "railiance-platform + rapp-postgres" + }, + "rpo": { + "status": "unsupported", + "target_seconds": null, + "decision_owner": "railiance-platform + rapp-postgres" + }, + "rto": { + "status": "unsupported", + "target_seconds": null, + "decision_owner": "railiance-platform + rapp-postgres" + }, + "retention": "30 days", + "existing_evidence": "rapp-postgres/docs/evidence/RAPP-POSTGRES-WP-0005-T04-boundary-restore-2026-08-22.md", + "recovery_custody": "OpenBao: 2-of-3 operator quorum plus independent encrypted snapshot custody; database/offsite: governed backup lane and separately available restore key. Availability not verified in this task.", + "maintenance_abort": "docs/railiance01-coordinated-reboot.md; stop before mutation when freshness, quorum, consumer readiness or named abort operator is absent", + "freshness_policy": "assurance/service-contract.json; diagnostic thresholds only, no installed cadence approval", + "requirement_assessment": "No accepted numeric consumer availability/RPO/RTO requirement found in the reviewed contracts. Service classes inform placement, not guarantees. Refuse any request for guaranteed HA/node-loss recovery until matched to supported substrate and package proof." + }, + { + "service": "openbao", + "accountable_owner": "railiance-platform", + "package_or_consumer_owner": "rapp-openbao", + "operator_role": "attended platform operator; application proof by consumer owner", + "consumers": [ + "approved credential lanes" + ], + "failure_domain": "single-node railiance01", + "availability": { + "status": "unsupported", + "target": null, + "decision_owner": "railiance-platform + rapp-openbao" + }, + "rpo": { + "status": "unsupported", + "target_seconds": null, + "decision_owner": "railiance-platform + rapp-openbao" + }, + "rto": { + "status": "unsupported", + "target_seconds": null, + "decision_owner": "railiance-platform + rapp-openbao" + }, + "retention": "KV history/audit/snapshot retention not accepted", + "existing_evidence": "reviews/WARDEN-WP-0027-T02-DRILL-20260822-01-openbao-snapshot-receipt.json", + "recovery_custody": "OpenBao: 2-of-3 operator quorum plus independent encrypted snapshot custody; database/offsite: governed backup lane and separately available restore key. Availability not verified in this task.", + "maintenance_abort": "docs/railiance01-coordinated-reboot.md; stop before mutation when freshness, quorum, consumer readiness or named abort operator is absent", + "freshness_policy": "assurance/service-contract.json; diagnostic thresholds only, no installed cadence approval", + "requirement_assessment": "No accepted numeric consumer availability/RPO/RTO requirement found in the reviewed contracts. Service classes inform placement, not guarantees. Refuse any request for guaranteed HA/node-loss recovery until matched to supported substrate and package proof." + }, + { + "service": "forgejo-backup", + "accountable_owner": "railiance-platform", + "package_or_consumer_owner": "railiance-forge", + "operator_role": "attended platform operator; application proof by consumer owner", + "consumers": [ + "forgejo" + ], + "failure_domain": "source host plus separately encrypted Nextcloud copy", + "availability": { + "status": "unsupported", + "target": null, + "decision_owner": "railiance-platform + railiance-forge" + }, + "rpo": { + "status": "unsupported", + "target_seconds": null, + "decision_owner": "railiance-platform + railiance-forge" + }, + "rto": { + "status": "unsupported", + "target_seconds": null, + "decision_owner": "railiance-platform + railiance-forge" + }, + "retention": "14 daily + 4 weekly target; local 7/type", + "existing_evidence": "docs/forgejo-backup.md", + "recovery_custody": "OpenBao: 2-of-3 operator quorum plus independent encrypted snapshot custody; database/offsite: governed backup lane and separately available restore key. Availability not verified in this task.", + "maintenance_abort": "docs/railiance01-coordinated-reboot.md; stop before mutation when freshness, quorum, consumer readiness or named abort operator is absent", + "freshness_policy": "assurance/service-contract.json; diagnostic thresholds only, no installed cadence approval", + "requirement_assessment": "No accepted numeric consumer availability/RPO/RTO requirement found in the reviewed contracts. Service classes inform placement, not guarantees. Refuse any request for guaranteed HA/node-loss recovery until matched to supported substrate and package proof." + }, + { + "service": "cnpg-option-a", + "accountable_owner": "railiance-platform", + "package_or_consumer_owner": "railiance-platform", + "operator_role": "attended platform operator; application proof by consumer owner", + "consumers": [ + "production-of-record CNPG logical dumps" + ], + "failure_domain": "source host plus separately encrypted Nextcloud copy", + "availability": { + "status": "unsupported", + "target": null, + "decision_owner": "railiance-platform + railiance-platform" + }, + "rpo": { + "status": "unsupported", + "target_seconds": null, + "decision_owner": "railiance-platform + railiance-platform" + }, + "rto": { + "status": "unsupported", + "target_seconds": null, + "decision_owner": "railiance-platform + railiance-platform" + }, + "retention": "offsite retention not accepted", + "existing_evidence": "docs/cnpg-option-a-backup.md", + "recovery_custody": "OpenBao: 2-of-3 operator quorum plus independent encrypted snapshot custody; database/offsite: governed backup lane and separately available restore key. Availability not verified in this task.", + "maintenance_abort": "docs/railiance01-coordinated-reboot.md; stop before mutation when freshness, quorum, consumer readiness or named abort operator is absent", + "freshness_policy": "assurance/service-contract.json; diagnostic thresholds only, no installed cadence approval", + "requirement_assessment": "No accepted numeric consumer availability/RPO/RTO requirement found in the reviewed contracts. Service classes inform placement, not guarantees. Refuse any request for guaranteed HA/node-loss recovery until matched to supported substrate and package proof." + } + ] +} diff --git a/docs/adr/ADR-0004-service-assurance-disclosure.md b/docs/adr/ADR-0004-service-assurance-disclosure.md new file mode 100644 index 0000000..44a0b3e --- /dev/null +++ b/docs/adr/ADR-0004-service-assurance-disclosure.md @@ -0,0 +1,64 @@ +--- +id: RPLAT-ADR-0004 +title: "ADR-0004 — Disclose unsupported service guarantees and evidence freshness" +status: accepted +revision: "1.0" +owner: railiance-platform +date: "2026-09-05" +last_reviewed: "2026-09-05" +review_interval: 6m +--- + +# ADR-0004 — Service assurance disclosure + +## Context + +The platform has dated backup/restore evidence and three single-instance CNPG +cells, but no accepted numeric end-to-end availability/RPO/RTO commitments. +One successful scratch restore is not a recovery guarantee. A successful +source validator is not current live health. RPF-WP-0036 makes that distinction +executable and visible to consumers. + +## Decision + +Publish `assurance/service-records.json` as the S3-owned disclosure of supported +and unsupported guarantees. Availability, RPO and RTO remain explicitly +unsupported until the named service/package/consumer owners accept measurable +targets backed by evidence. This is a platform disclosure decision, not an +external owner's approval. A consumer requesting a stronger guarantee needs a +recorded placement/recovery decision before admission. In particular, CNPG on +one host does not provide host-loss HA; the dependency is S1 host failure +domains, S2 runtime placement and package-level failover proof. + +Use `assurance/service-contract.json` only for local diagnostics: 15-minute +capture/health age, 36-hour successful backup/upload age, 30-day restore age, +one-hour ESO refresh age, and 80% connection/memory occupancy for headroom. +These intentionally conservative defaults expose +missing evidence; they are neither SLAs nor approvals to install schedules. +Service-specific cadence and receiver acceptance remain RPF-WP-0036-T03/T04. +A different legitimate refresh interval needs an explicit service contract, +not automatic acceptance of an indefinitely stale observation. + +## Consequences + +The checker distinguishes healthy, stale, missing, unavailable and failed +samples. It rejects wrong clusters, unknown payload fields and future times. +Its receipt states `transport: unmonitored` and `guarantees: unsupported`. +Receipt syntax and freshness validation do not prove the truth of a supplied +observation or replace a signed native restore receipt. The live collector +queries only fixed metadata/status fields, a pg_stat_activity count and +unauthenticated seal status; it cannot fetch Secrets or run backup, restore or +seal/unseal mutations. + +Sources of restore proof remain the owning packages and their existing +validators. The platform accepts those artifacts after an approved isolated +exercise; it does not convert old Markdown or example JSON into fresh proof. +No unattended jobs or Q2 transport are deployed under this decision. + +## Alternatives considered + +Invent numeric targets from elapsed restore time: rejected because it creates +an untested promise. Keep all checks waiting for telemetry: rejected because +safe local diagnostics and unsupported-guarantee disclosure are useful now. +Treat every source declaration as live: rejected because desired placement +and actual deployment can diverge. diff --git a/docs/adr/ADR-0005-demand-gated-stateful-services.md b/docs/adr/ADR-0005-demand-gated-stateful-services.md new file mode 100644 index 0000000..9925d30 --- /dev/null +++ b/docs/adr/ADR-0005-demand-gated-stateful-services.md @@ -0,0 +1,55 @@ +--- +id: RPLAT-ADR-0005 +title: "ADR-0005 — Reuse existing storage and defer unrequested stateful services" +status: accepted +revision: "1.0" +owner: railiance-platform +date: "2026-09-05" +last_reviewed: "2026-09-05" +review_interval: 6m +--- + +# ADR-0005 — Demand-gated stateful services + +## Context + +INTENT includes cache, object storage and messaging. Valkey is gated in the +Makefile. CNPG already consumes an external governed S3 backup destination. +`artifact-store` has local/S3 backend code, retention/hold semantics and a +MinIO/STS assessment; its older SCOPE understates the code. The reviewed +platform queue has no accepted general cache or messaging consumer request. +Absence from this queue is not a claim that no future consumer could need one. + +## Decision + +- Cache: defer shared Valkey deployment. Reopen on a named consumer specifying + latency, eviction/durability, isolation, capacity and an operating owner. +- General object storage: reuse the existing backup destination only for its + approved backup scope. Evaluate artifact-store's existing S3 interface for + artifact-preservation demand before proposing another gateway or engine. + Defer a general platform S3/MinIO service until accepted demand specifies + retention, residency, capacity, recovery, credential scope and package owner. +- Messaging: defer shared broker deployment until a named consumer defines + delivery/replay/order guarantees, retention, throughput and recovery needs. + Do not replace application-specific queues by assumption. + +Review on a new accepted consumer request, a change in existing provider +capability, or 2027-03-05, whichever comes first. This is an S3 deployment +posture decision; it assigns no new obligation to potential consumers and +chooses no vendor or engine. The Q3 fleet-owner question stays with +railiance-master; local recovery work continues independently. + +## Consequences + +Keep deploy gates. Any accepted new demand creates a bounded delivery plan +covering service contract, custody, tenancy, capacity, backup/restore, telemetry, +package owner, costs and acceptance before provisioning. Existing backup +credentials are not general artifact credentials. Platform does not take over +artifact metadata lifecycle or application publication logic. + +## Alternatives considered + +Deploy Valkey/MinIO/RabbitMQ to satisfy the nouns in INTENT: rejected because +it adds unsupported operations without an accepted workload. Declare external +backup S3 a general object service: rejected because its scope and consumer +contract are narrower. diff --git a/docs/evidence/RPF-WP-0036-assurance-2026-09-05.json b/docs/evidence/RPF-WP-0036-assurance-2026-09-05.json new file mode 100644 index 0000000..50d7d1c --- /dev/null +++ b/docs/evidence/RPF-WP-0036-assurance-2026-09-05.json @@ -0,0 +1,242 @@ +{ + "schema": "railiance-platform.assurance-evidence.v1", + "date": "2026-09-05", + "observation": { + "schema": "railiance-platform.observation.v1", + "cluster_uid": "a553c742-0115-43d4-99a4-a5ca56fe0786", + "captured_at": "2026-09-05T09:37:05.118118+00:00", + "signals": { + "apps-pg.ready": { + "result": "pass", + "observed_at": "2026-09-05T09:36:57.163337+00:00" + }, + "apps-pg.backup": { + "result": "pass", + "observed_at": "2026-09-05T02:15:07Z" + }, + "apps-pg.wal": { + "result": "pass", + "observed_at": "2026-09-05T09:36:57.163365+00:00" + }, + "apps-pg.headroom": { + "result": "pass", + "observed_at": "2026-09-05T09:36:42Z" + }, + "platform-pg.ready": { + "result": "pass", + "observed_at": "2026-09-05T09:36:59.396739+00:00" + }, + "platform-pg.backup": { + "result": "pass", + "observed_at": "2026-09-05T02:15:11Z" + }, + "platform-pg.wal": { + "result": "pass", + "observed_at": "2026-09-05T09:36:59.396763+00:00" + }, + "platform-pg.headroom": { + "result": "pass", + "observed_at": "2026-09-05T09:36:44Z" + }, + "platform-pg-2.ready": { + "result": "pass", + "observed_at": "2026-09-05T09:37:01.824557+00:00" + }, + "platform-pg-2.backup": { + "result": "pass", + "observed_at": "2026-09-05T02:15:08Z" + }, + "platform-pg-2.wal": { + "result": "pass", + "observed_at": "2026-09-05T09:37:01.824580+00:00" + }, + "platform-pg-2.headroom": { + "result": "pass", + "observed_at": "2026-09-05T09:36:40Z" + }, + "openbao.seal": { + "result": "pass", + "observed_at": "2026-09-05T09:37:04.442236+00:00" + }, + "eso.ready": { + "result": "fail", + "observed_at": "2026-09-05T09:37:05.118098+00:00" + }, + "eso.refresh": { + "result": "pass", + "observed_at": "2026-08-06T12:33:40Z" + } + } + }, + "evaluation_at_capture": { + "schema": "railiance-platform.assurance-signal.v1", + "cluster_uid": "a553c742-0115-43d4-99a4-a5ca56fe0786", + "evaluated_at": "2026-09-05T09:37:05.118118+00:00", + "signals": { + "apps-pg.ready": { + "state": "healthy", + "owner": "railiance-platform" + }, + "apps-pg.backup": { + "state": "healthy", + "owner": "railiance-platform" + }, + "apps-pg.wal": { + "state": "healthy", + "owner": "railiance-platform" + }, + "apps-pg.restore": { + "state": "missing", + "owner": "railiance-platform" + }, + "apps-pg.headroom": { + "state": "healthy", + "owner": "railiance-platform" + }, + "platform-pg.ready": { + "state": "healthy", + "owner": "rapp-postgres" + }, + "platform-pg.backup": { + "state": "healthy", + "owner": "rapp-postgres" + }, + "platform-pg.wal": { + "state": "healthy", + "owner": "rapp-postgres" + }, + "platform-pg.restore": { + "state": "missing", + "owner": "rapp-postgres" + }, + "platform-pg.headroom": { + "state": "healthy", + "owner": "rapp-postgres" + }, + "platform-pg-2.ready": { + "state": "healthy", + "owner": "rapp-postgres" + }, + "platform-pg-2.backup": { + "state": "healthy", + "owner": "rapp-postgres" + }, + "platform-pg-2.wal": { + "state": "healthy", + "owner": "rapp-postgres" + }, + "platform-pg-2.restore": { + "state": "missing", + "owner": "rapp-postgres" + }, + "platform-pg-2.headroom": { + "state": "healthy", + "owner": "rapp-postgres" + }, + "openbao.seal": { + "state": "healthy", + "owner": "railiance-platform" + }, + "openbao.snapshot": { + "state": "missing", + "owner": "railiance-platform" + }, + "openbao.restore": { + "state": "missing", + "owner": "railiance-platform" + }, + "offsite.upload": { + "state": "missing", + "owner": "railiance-platform" + }, + "offsite.restore": { + "state": "missing", + "owner": "railiance-platform" + }, + "eso.ready": { + "state": "failed", + "owner": "railiance-platform" + }, + "eso.refresh": { + "state": "stale", + "owner": "railiance-platform" + } + }, + "transport": "unmonitored", + "guarantees": "unsupported", + "threshold_status": "local-diagnostic-only", + "healthy": false + }, + "live_admission": { + "captured_at": "2026-09-05T09:28:42.234951+00:00", + "cells": { + "apps-pg": { + "instances": 1, + "memory_limit": "1Gi", + "max_connections": "100", + "retention": "30d" + }, + "platform-pg": { + "instances": 1, + "memory_limit": "1Gi", + "max_connections": "100", + "retention": "30d" + }, + "platform-pg-2": { + "instances": 1, + "memory_limit": "1Gi", + "max_connections": "100", + "retention": "30d" + } + } + }, + "live_spec_matches_source": true, + "eso_failures": [ + { + "name": "forgejo-mailer", + "namespace": "forgejo", + "refresh": "2026-08-06T12:33:40Z", + "conditions": [ + { + "type": "Ready", + "status": "False", + "reason": "SecretSyncedError" + } + ] + }, + { + "name": "reuse-surface-runtime", + "namespace": "reuse", + "refresh": "2026-08-06T20:25:39Z", + "conditions": [ + { + "type": "Ready", + "status": "False", + "reason": "SecretSyncedError" + } + ] + }, + { + "name": "target-revenue-runtime", + "namespace": "target-revenue", + "refresh": "2026-09-04T14:52:08Z", + "conditions": [ + { + "type": "Ready", + "status": "False", + "reason": "SecretSyncedError" + } + ] + } + ], + "validation": { + "full_tests": 176, + "focused_assurance_tests": 14, + "admission_baseline": "pass" + }, + "live_mutations": false, + "secret_values_read": false, + "scheduling_installed": false, + "q2_delivery_proven": false, + "restore_executed": false +} diff --git a/docs/placement-policy.md b/docs/placement-policy.md index ae6f4ca..c5047e9 100644 --- a/docs/placement-policy.md +++ b/docs/placement-policy.md @@ -75,7 +75,9 @@ triggers. It is not the same role as the operator of the cluster. | Workload | Substrate | Provided level | Placement owner | Co-signer | §13 evidence | | --- | --- | --- | --- | --- | --- | | `audit-core` | `platform-pg` | P1 | `audit-core` | this repo | probes exist | -| `tenant-engine` | SQLite PVC current; `platform-pg` desired | off-ladder current; P1 desired | `tenant-engine` | this repo | source conformance; live cutover absent | +| `tenant-engine` | `platform-pg` | P1 | `tenant-engine` | this repo | owner cutover evidence 2026-08-21 | +| `core-hub` | `platform-pg` | P1 | `core-hub` | this repo | capacity admission 2026-08-20; private shadow 2026-08-21 | +| `sbom-nexus` | `platform-pg-2` | P1 | `sbom-nexus` | this repo | package boundary/restore evidence 2026-08-22 | | `user-engine` | `user-engine-pg` | P2 | `net-kingdom` | this repo | not assessed here | | `target-revenue` | `target-revenue-pg` | P2 | `target-revenue` | this repo | not assessed here | | `forgejo` | `forgejo-db` | P2 | this repo | — | single consumer | @@ -144,13 +146,16 @@ one". "Provision a second cluster" is an answer; "revisit at the time" is not. | Cluster | Ceiling declared | Overflow target | Standing | | --- | --- | --- | --- | -| `platform-pg` | 4 declarations (estimated; ADR-0004) | `platform-pg-2`, named but absent | 3/4 including the isolation probe; measure before the next workload | +| `platform-pg` | 4 declarations (package ADR-0004) | `platform-pg-2`, deployed | 4/4 including probe and Core Hub; fifth refused | +| `platform-pg-2` | 4 declarations (same package validator) | No third cell approved | 1/4: sbom-nexus; owner admission and restore completed 2026-08-22 | | `apps-pg` | 3 consumers; 20 connections each, 1Gi memory binds first | `apps-pg-2` | 2/3; overflow must exist before a 4th | | `forgejo-db` | n/a — single consumer | n/a | — | -Both live shared clusters are one consumer away from the trigger. This repo -owes `apps-pg`'s ceiling under its own rule, which is the correct asymmetry: a -policy whose author is exempt is not a policy. +Source and live metadata reviewed 2026-09-05: platform-pg is full; apps-pg +remains 2/3 and apps-pg-2 is still unapplied. The generated +`assurance/admission-baseline.json` binds these disclosures to the actual owner +validators/declarations. `make assurance-admission` refuses source drift. +Package acceptance, rather than this table, authorizes consumer provisioning. ## 5. Triggers, and who is watching them @@ -189,7 +194,8 @@ risk that nobody can see is strictly worse than one that is stated". | Cluster | Co-resident classes | Mismatch | | --- | --- | --- | -| `platform-pg` | `latency-critical` (`tenant-engine`) + `batch` (`audit-core`) | **yes — reported** | +| `platform-pg` | `latency-critical` (`tenant-engine`, `core-hub`) + `batch` (`audit-core`) | **yes — reported** | +| `platform-pg-2` | `interactive` (`sbom-nexus`) | no | | `apps-pg` | `interactive` + `interactive` | no | | `forgejo-db` | single consumer | n/a | @@ -256,3 +262,11 @@ onboarding enforce it. Six months, or on: a new shared-cluster consumer, a co-residency class change, `railiance-telemetry` going live, or the framework reaching `accepted`. Next review due **2027-02-17**. + +## Service assurance implementation — 2026-09-05 + +`docs/service-assurance.md` supplies the bounded local producer and admission +check. Capture proves database readiness/backup/controller status at a point in +time, and currently exposes failed/stale ESO resources. It does not implement +continuous capacity monitoring. Trigger 5 remains unmonitored until its native +headroom adapter and Q2 recipient are accepted (RPF-WP-0036-T03/T04). diff --git a/docs/platform-ownership-handoffs.md b/docs/platform-ownership-handoffs.md new file mode 100644 index 0000000..f5e2b39 --- /dev/null +++ b/docs/platform-ownership-handoffs.md @@ -0,0 +1,49 @@ +# Platform compatibility handoffs — 2026-09-05 + +RPF-WP-0036-T06 inventory: `assurance/ownership-handoffs.json` contains exact +paths/hashes, accepting owner candidates, callers, review dates and acceptance +criteria. S3 custody/CCR policies and shared backup mechanisms stay here. + +Retain each operational entry point until the proposed owner accepts its +canonical replacement, callers pass dry-run/smoke checks, and rollback is +recorded. Review by 2026-10-05 or on owner acceptance. No files were removed, +no ownership acceptance was invented, and no coordination messages were sent. +This is the dated platform retention decision; T06 remains waiting for accepted +handoffs and the separate derived-state repair. + +The forge contract already distinguishes forge requirements/verification from +shared backup mechanisms. Split those concerns; do not move all backup code +just because its filename mentions Forgejo. The platform OpenBao package +handoff is likewise already documented: preserve custody/governance here and +only adopt the package owner's authoritative replacement for its wrapper. +ArgoCD files need artifact-level ownership: runtime bootstrap to S2, generic +paved paths to S4, app releases to S5. ESO credential stores retain S3 custody +review even when another repo packages their deployment. + +## Prepared owner requests + +These are reviewable request content, not sent messages: + +- Forge/package/app owners: accept or amend the exact source inventory, name + canonical entry points and callers, and return revision-pinned dry-run, + smoke and rollback evidence before retirement. +- Repo-manager/State Hub: reconcile only the legacy UUID-to-canonical map in + the assessment's derived-state caveat; keep file-backed identities intact. + The installed brief generator reads open Hub workplans without excluding + legacy retired aliases, so rerunning it alone would reproduce duplicates. +- Railiance-telemetry: supply versioned receiving schema, transport and + retention contract, intended recipient, and a controlled stale-event/absence + acceptance path for the local S3 producer. It is currently unmonitored. +- Railiance-master: determine the accountable fleet Q3 recovery owner and + boundary. S3 continues its own service recovery work meanwhile. + +## Live ESO findings requiring scope confirmation + +The 2026-09-05 metadata capture found `SecretSyncedError` for +`forgejo/forgejo-mailer`, `reuse/reuse-surface-runtime`, and +`target-revenue/target-revenue-runtime`. Their last refreshes were August 6, +August 6 and September 4 respectively. Package/consumer and platform custody +owners must first classify current versus obsolete resources, then review +any exact lane repair or retirement. No Secret values were read and no failed +resource was simply excluded to make the aggregate pass. These results do not +by themselves identify a broken provider credential or authorize rotation. diff --git a/docs/s3-consumer-interfaces.md b/docs/s3-consumer-interfaces.md index 9d42ca1..02a4d8f 100644 --- a/docs/s3-consumer-interfaces.md +++ b/docs/s3-consumer-interfaces.md @@ -1,8 +1,8 @@ # S3 consumer service interfaces Work record: `RAILIANCE-WP-0016` item 17; `RPF-WP-0018` T04 -Date: 2026-08-15, revised 2026-08-17 -Contract version of this index: `1.1.0` +Date: 2026-08-15, revised 2026-09-05 +Contract version of this index: `1.2.0` Consumers bind to these contracts, not to Helm chart names or pod topology. Changing an engine behind a versioned interface does not bump the interface @@ -101,7 +101,7 @@ Consumer is a `consumers[].declaration`. Boundary unit is | Declared ceiling / overflow target | **4 declarations / `platform-pg-2`** | The isolation probe counts. `rapp-postgres` rejects a fifth declaration on one cell; ADR-0004 owns the number. | Service classes currently co-resident on `platform-pg`: `latency-critical` -(`tenant-engine`) and `batch` (`audit-core`). Nothing prioritises between them +(`tenant-engine`, `core-hub`) and `batch` (`audit-core`). Nothing prioritises between them — community PostgreSQL has no resource governor. Reported per Tenancy Posture §8.3.3. @@ -115,3 +115,20 @@ the lane without restating TTL, delivery, or revocation rules. Valkey and in-cluster MinIO have capability blocks in `SCOPE.md` and no live service. They do not receive a `v1` until a consumer exists. + +## Service assurance disclosure (1.2.0, additive) + +`assurance/service-records.json` explicitly discloses unsupported availability, +RPO and RTO guarantees, decision owners and existing recovery evidence; +`docs/service-assurance.md` documents the executable local checks. Diagnostic +freshness budgets are not SLAs or approval to deploy schedules. Consumers that +require stronger guarantees need a placement/recovery decision. + +Current source occupancy: platform-pg 4/4 including the isolation probe, +platform-pg-2 1/4 for sbom-nexus, apps-pg 2/3, apps-pg-2 0/3 and unapplied. +All three deployed cells declare one instance, 1Gi memory limit, 100 aggregate +connections and 30-day retention; live metadata matched on 2026-09-05. The +second platform cell's approved restore took 51 seconds on 2026-08-22; this +is dated evidence, not an RTO. Tenant-engine cut over from SQLite on 2026-08-21. +`assurance/admission-baseline.json` carries owner-native declarations, hashes +and limits. Do not refresh that baseline without reviewing changes. diff --git a/docs/service-assurance.md b/docs/service-assurance.md new file mode 100644 index 0000000..3bb51c0 --- /dev/null +++ b/docs/service-assurance.md @@ -0,0 +1,105 @@ +# S3 service assurance — v1 + +Workplan: RPF-WP-0036. Disclosure authority: +[ADR-0004](adr/ADR-0004-service-assurance-disclosure.md). + +## Operator entry points + +```bash +make assurance-admission +make assurance-capture +make assurance-check +``` + +Admission requires the adjacent reviewed `rapp-postgres` checkout and PyYAML, +which existing platform tooling already uses. Capture requires the existing +noninteractive SSH route to `railiance01`, with kubectl on that host. It pins +the kube-system UID from existing platform evidence and stops on mismatch. +It does not trust the default workstation kube context. Failure prints only a +fixed error code, never raw command output. No Secret, application row/log or credential query is run. A read-only +pg_stat_activity count and kubectl metrics supply aggregate headroom; +unauthenticated `bao status` supplies seal state. The observation is written to +`/tmp/railiance-platform-observation.json` by default; override with +`ASSURANCE_OBSERVATION`. Run the check only after capture succeeds. + +`evaluate` exits 0 when every required sample is healthy, 1 for incomplete, +stale, failed or unavailable samples, and 2 for invalid/unavailable input. +There is intentionally no green overall service claim while evidence is +missing. To replay a dated observation: + +```bash +python3 scripts/service_assurance.py evaluate observation.json --now 2026-09-05T12:00:00Z +``` + +`--now` is for replay/testing, not a way to freshen old evidence. The output +retains evaluation time, unsupported guarantees and unmonitored transport. +A `pass` sample is an attestation from its producer, not cryptographic proof. +Only approved owner receipts can close the workplan's live acceptance gates. + +## Supported source contract and missing adapters + +The closed observation envelope carries schema, pinned cluster UID, capture +time and a configured set of signals. Each sample has only `observed_at` and +`result` (`pass`, `fail`, `unavailable`). Free text, values, paths and unknown +signals are rejected. For backup/restore/upload signals the observed time is +the completion time of that action; it is not the time someone read the file. +A capture older than 15 minutes never passes even when a backup is recent. + +The collector supplies database readiness, last completed backup and continuous +archiving for apps-pg/platform-pg/platform-pg-2, plus fleet ESO Ready/oldest-refresh +status, OpenBao seal state, and aggregate connection/memory headroom. Headroom +uses a local diagnostic ceiling of 80% of source connection/memory limits; it +is not a workload latency guarantee. Metrics timestamps retain their age. ContinuousArchiving is a controller condition, not a measured WAL +lag guarantee. The fleet ESO aggregate is deliberately a conservative inventory +check; an obsolete resource must be explicitly classified by its owner before +it is excluded. One ESO failure cannot disappear inside an average. + +The following remain missing until a native value-safe adapter and acceptance +exist: validated isolated restore receipts, +OpenBao snapshot/restore proof, and offsite upload/restore receipts. Missing +adapters are not inferred healthy from pod readiness. The local producer is +not the Q2 standard; railiance-telemetry has no implemented receiving contract +in the reviewed checkout. Integration, routing and scheduled delivery remain +T04, and no notification was sent during implementation. + +## Service records and evidence inventory + +`assurance/service-records.json` covers all three database cells, OpenBao, +Forgejo backup and multi-host CNPG logical offsite delivery. It distinguishes +unsupported objectives, declared retention, recovery custody and decision +owners. The old Forgejo cron example mentions an RPO 24h target; that is not a +newly accepted or verified current service guarantee. RPF-WP-0029 still gates +provider invalidation/replacement recovery for the shared offsite lane. + +| Evidence | Existing authority | Current acceptance limit | +| --- | --- | --- | +| CNPG completed backup/WAL | Cluster controller status; live capture | Current status only; does not exercise restore | +| apps-pg restore | RPF-WP-0019 evidence, 2026-08-20 | Dated 56-second scratch restore; no numeric RTO commitment | +| platform-pg restore/PITR | rapp-postgres restore procedure and evidence | Recurring current proof and independent recovery access remain T03 | +| platform-pg-2 restore | RAPP-POSTGRES-WP-0005-T04, 2026-08-22 | 51-second isolated restore after SBOM migration; not a continuous guarantee | +| OpenBao snapshot | WARDEN-WP-0027 preparation receipt, 2026-08-23 | Snapshot/encrypted off-host preparation, not isolated restore | +| OpenBao restore | Existing `openbao-validate-restore-evidence.sh` and package procedure | Example receipt cannot pass as a fresh execution | +| Recovery exercise | RPF-WP-0015-T02/T03 | Separate windows, synthetic driver/quorum and abort operator required | +| Logical/Forgejo offsite | activity-core backup definitions, existing helper/runbooks | No new upload/restore performed; RPF-WP-0029 remains open | + +## Admission and disclosure drift + +`assurance/admission-baseline.json` is generated by the `admission` command from +local CNPG CRs, `placement-owners.json`, and the native rapp-postgres renderer. +The checker reuses both native capacity validators. It also rejects missing +owners, changed cell placement, unhonoured inherited retention and mismatch +between effective package retention and the actual CR. Hashes pin every +consumed source artifact; changing package code or declarations requires review +before refreshing the baseline. Do not blindly regenerate it in CI. + +Source occupancy: apps-pg 2/3, apps-pg-2 0/3 (not deployed), platform-pg 4/4 +including the probe, platform-pg-2 1/4 for sbom-nexus. Tenant-engine's PostgreSQL +cutover completed 2026-08-21; its old SQLite-current claim was stale. Platform-pg-2 +was accepted and restored on 2026-08-22; it is no longer merely named overflow. +Core Hub is latency-critical alongside audit-core batch work; no service-class +priority is implied. See the generated baseline for per-cell details. + +Source admission checks happen before deployment in the existing apps-pg and +package owner paths; this additional check detects disclosure drift. It does +not apply resources or become a second provisioning engine. Any new consumer +needs an owner entry and a reviewed baseline update, even within free capacity. diff --git a/history/2026-09-05-platform-assurance-implementation.md b/history/2026-09-05-platform-assurance-implementation.md new file mode 100644 index 0000000..22ecd12 --- /dev/null +++ b/history/2026-09-05-platform-assurance-implementation.md @@ -0,0 +1,79 @@ +# RPF-WP-0036 implementation — 2026-09-05 + +Implemented the ready plan's repository work and performed bounded live status +inspection using the existing SSH route to the pinned railiance01 cluster. +No credentials were fetched, no backup/restore or infrastructure mutation ran, +and no scheduler, external notification or owner acceptance was created. + +## Delivered + +- ADR-0004 and service records disclose unsupported availability/RPO/RTO + commitments, existing evidence, retention and accountable decision owners. + Local freshness/headroom thresholds are diagnostic, not SLAs. +- An admission checker reuses the apps-pg and rapp-postgres validators, verifies + effective retention and placement ownership, and refuses a changed source + hash/disclosure baseline. Five-consumer and mismatched-retention fixtures fail. +- A cluster-pinned collector supplies selected CNPG status, aggregate connection + counts, metrics and unauthenticated OpenBao seal status. It reads no Secret, + application data, session SQL text or logs. SSH argv are quoted; failures + suppress raw output; successful observation files publish atomically at 0600. +- A closed observation evaluator distinguishes healthy, stale, missing, failed + and unavailable samples, rejects wrong clusters/future/unknown payloads, and + makes missing emission detectable through capture expiry. Transport remains + explicitly unmonitored pending Q2 acceptance. +- Corrected placement/interface disclosures: tenant-engine's PostgreSQL cutover + completed; platform-pg is 4/4 and the deployed platform-pg-2 holds sbom-nexus. +- Exact compatibility path/hash/caller inventories and a dated retention + decision are ready for owner review. No functioning entry point was removed. +- ADR-0005 defers unrequested cache/general storage/broker deployments with + explicit review triggers and prefers existing storage interfaces for demand. + +## Live findings + +The default workstation kube context and the misleading config-railiance01 +file reached a different host; config-hosteurope depended on a down local +forwarder. The collector therefore uses existing SSH to railiance01 and checks +kube-system UID `a553c742-0115-43d4-99a4-a5ca56fe0786` before service reads. +No context was rewritten or tunnel deployed. + +All three deployed database cells matched one instance, 1Gi memory limit, +100 aggregate connections and 30-day retention. Each was Ready, had a completed +backup from 02:15 UTC that morning and reported continuous archiving. Aggregate +connection/memory occupancy passed the local 80% diagnostic threshold. OpenBao +was unsealed. These are point-in-time observations, not new guarantees. + +The checker also exposed three failed ESO resources (`SecretSyncedError`): +forgejo/forgejo-mailer, reuse/reuse-surface-runtime and +target-revenue/target-revenue-runtime. The earliest successful refresh was +2026-08-06. Their active/obsolete status and exact cause need consumer/platform +review before a lane repair or retirement; no credential diagnosis is inferred +from this condition alone. + +Evidence: `docs/evidence/RPF-WP-0036-assurance-2026-09-05.json` includes capture, +evaluation time, live limits and sanitized failing-resource metadata. It does +not manufacture isolated-restore receipts from historical prose. Missing +restore/snapshot/offsite inputs and failed/stale ESO keep overall assurance +non-green, which is the expected correct result. + +## Validation + +Full repository unittest discovery: 176 passed. Focused assurance tests after +collector output hardening: 14 passed. Admission baseline check passed; fresh +live capture completed and its specification limits matched the source. +Cases cover wrong clusters, future/naive times, stale/replayed captures, missing +and failed signals, unexpected payloads/value-safe errors, unavailable native +reads, memory units, missing ownership/overflow, fifth consumer admission, +and unhonoured retention. This is not a restore or alert-delivery test. + +## Remaining acceptance + +T02/T05/T07 are done. T03 waits on fresh owner-validated recovery/offsite +receipts, independent recovery access and approved recurring execution. +T04 waits on the Q2 receiving contract and controlled failure/absence delivery; +ESO findings remain visible for exact owner disposition. T06 waits on accepted +handoffs and scoped alias cleanup. The installed brief generator still reads +legacy open aliases; generating the brief alone would reproduce those rows. + +The prepared owner request content is in `docs/platform-ownership-handoffs.md`. +Its transmission is not claimed. The plan remains blocked on these explicit +acceptance gates; source implementation is not substituted for owner/live proof. diff --git a/scripts/capture_service_observation.py b/scripts/capture_service_observation.py new file mode 100644 index 0000000..2735191 --- /dev/null +++ b/scripts/capture_service_observation.py @@ -0,0 +1,126 @@ +#!/usr/bin/env python3 +"""Collect only selected status fields over the existing railiance01 SSH lane.""" +import argparse +from datetime import datetime, timezone +import json +import re +import os +import tempfile +from pathlib import Path +import shlex +import subprocess +import sys + +ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT / 'scripts')) +from service_assurance import read_contract, admission + + +def query(args, allowed_codes=(0,)): + result = subprocess.run(['ssh', '-o', 'BatchMode=yes', '-o', 'ConnectTimeout=10', + 'railiance01', shlex.join(['kubectl', '--request-timeout=10s', *args])], + capture_output=True, text=True, timeout=25) + if result.returncode not in allowed_codes: + raise ValueError('query unavailable') + return json.loads(result.stdout) + + +def memory_bytes(value): + match = re.fullmatch(r"([0-9]+(?:\.[0-9]+)?)(Ki|Mi|Gi|Ti|K|M|G|T)?", value) + if not match: + raise ValueError('invalid memory quantity') + unit = match.group(2) or '' + multipliers = {'': 1, 'Ki': 1024, 'Mi': 1024**2, 'Gi': 1024**3, + 'Ti': 1024**4, 'K': 1000, 'M': 1000**2, 'G': 1000**3, 'T': 1000**4} + return float(match.group(1)) * multipliers[unit] + + +def capture(): + contract = read_contract(ROOT / 'assurance/service-contract.json') + uid = query(['get', 'namespace', 'kube-system', '-o', 'go-template={{printf "%q" .metadata.uid}}']) + if uid != contract['cluster_uid']: + raise ValueError('wrong cluster') + signals = {} + def add(name, result, when=None): + signals[name] = {'result': result, 'observed_at': when or datetime.now(timezone.utc).isoformat()} + baseline = json.loads((ROOT / 'assurance/admission-baseline.json').read_text()) + if admission() != baseline: + raise ValueError('admission baseline drift') + for cell in ('apps-pg', 'platform-pg', 'platform-pg-2'): + names = [cell + '.' + suffix for suffix in ('ready', 'backup', 'wal')] + try: + d = query(['get', 'cluster', cell, '-n', 'databases', '-o', + 'go-template={"ready":{{.status.readyInstances}},"instances":{{.spec.instances}},"lastBackup":{{printf "%q" .status.lastSuccessfulBackup}},"conditions":[{{range $i,$v := .status.conditions}}{{if $i}},{{end}}{"type":{{printf "%q" $v.type}},"status":{{printf "%q" $v.status}}}{{end}}]}']) + add(names[0], 'pass' if d['ready'] == d['instances'] and d['ready'] > 0 else 'fail') + if d['lastBackup']: + add(names[1], 'pass', d['lastBackup']) + add(names[2], 'pass' if any(c == {'type': 'ContinuousArchiving', 'status': 'True'} for c in d['conditions']) else 'fail') + except (ValueError, KeyError, TypeError, subprocess.TimeoutExpired): + for name in names: + add(name, 'unavailable') + try: + metrics = query(['get', '--raw', '/apis/metrics.k8s.io/v1beta1/namespaces/databases/pods/' + cell + '-1']) + if not metrics['containers']: + raise ValueError('empty metrics') + memory = sum(memory_bytes(c['usage']['memory']) for c in metrics['containers']) + # Same read-only pg_stat_activity aggregate as the package ops surface, + # narrowed to a count: no SQL text, usernames, database rows or values. + count = query(['exec', '-n', 'databases', cell + '-1', '-c', 'postgres', '--', + 'psql', '-U', 'postgres', '-d', 'postgres', '-Atqc', + 'SELECT count(*) FROM pg_stat_activity']) + limits = baseline['cells'][cell] + add(cell + '.headroom', 'pass' if type(count) is int and count >= 0 + and count <= limits['max_connections'] * 0.8 + and memory <= memory_bytes(limits['memory_limit']) * 0.8 else 'fail', + metrics['timestamp']) + except (OSError, ValueError, KeyError, TypeError, subprocess.TimeoutExpired): + add(cell + '.headroom', 'unavailable') + try: + status = query(['exec', '-n', 'openbao', 'openbao-0', '--', + 'bao', 'status', '-format=json'], allowed_codes=(0, 2)) + add('openbao.seal', 'pass' if status.get('sealed') is False else 'fail') + except (OSError, ValueError, KeyError, TypeError, subprocess.TimeoutExpired): + add('openbao.seal', 'unavailable') + try: + rows = query(['get', 'externalsecrets', '-A', '-o', + 'go-template=[{{range $i,$v := .items}}{{if $i}},{{end}}{"refresh":{{printf "%q" $v.status.refreshTime}},"conditions":[{{range $j,$c := $v.status.conditions}}{{if $j}},{{end}}{"type":{{printf "%q" $c.type}},"status":{{printf "%q" $c.status}}}{{end}}]}{{end}}]']) + ready = bool(rows) and all(any(c == {'type': 'Ready', 'status': 'True'} for c in row['conditions']) for row in rows) + # Empty/missing refresh is a failure, never an apparently healthy zero age. + refresh = min((r['refresh'] for r in rows), default='') + add('eso.ready', 'pass' if ready else 'fail') + if refresh: + add('eso.refresh', 'pass', refresh) + except (ValueError, KeyError, TypeError, subprocess.TimeoutExpired): + add('eso.ready', 'unavailable') + add('eso.refresh', 'unavailable') + # No token, Secret, application data/logs or seal/unseal mutation. + # Native restore and offsite receipts remain separate attended evidence. + return {'schema': 'railiance-platform.observation.v1', 'cluster_uid': uid, + 'captured_at': datetime.now(timezone.utc).isoformat(), 'signals': signals} + + +if __name__ == '__main__': + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('--output', type=Path) + args = parser.parse_args() + try: + data = json.dumps(capture(), indent=2) + '\n' + if args.output: + temporary = None + try: + with tempfile.NamedTemporaryFile(mode='w', dir=args.output.parent, delete=False) as f: + temporary = f.name + f.write(data) + f.flush() + os.fsync(f.fileno()) + os.replace(temporary, args.output) + temporary = None + finally: + if temporary: + os.unlink(temporary) + print(json.dumps({'captured': True})) + else: + print(data, end='') + except (OSError, ValueError, KeyError, TypeError, subprocess.TimeoutExpired): + print(json.dumps({'schema': 'railiance-platform.assurance-error.v1', 'error': 'capture-unavailable-or-wrong-cluster'})) + sys.exit(2) diff --git a/scripts/service_assurance.py b/scripts/service_assurance.py new file mode 100644 index 0000000..89b82b2 --- /dev/null +++ b/scripts/service_assurance.py @@ -0,0 +1,183 @@ +#!/usr/bin/env python3 +"""Read-only S3 admission and bounded evidence evaluation. Never fetch Secrets.""" +from __future__ import annotations + +import argparse +from datetime import datetime, timezone +import hashlib +import importlib.util +import json +import math +from pathlib import Path +import sys + +import yaml + +ROOT = Path(__file__).resolve().parents[1] + + +def load_module(name, path): + spec = importlib.util.spec_from_file_location(name, path) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +def timestamp(value): + if not isinstance(value, str): + raise ValueError('invalid timestamp') + result = datetime.fromisoformat(value.replace('Z', '+00:00')) + if result.tzinfo is None: + raise ValueError('timezone required') + return result.astimezone(timezone.utc) + + +def digest(path): + return hashlib.sha256(path.read_bytes()).hexdigest() + + +def admission(root=ROOT, package=None): + package = package or root.parent / 'rapp-postgres' + local = load_module('apps_capacity', root / 'tools/verify_apps_pg_capacity.py') + native = load_module('postgres_consumers', package / 'scripts/render_consumers.py') + manifests = [root / 'helm' / (name + '-cluster.yaml') for name in local.REVIEWED_CELLS] + local.verify(manifests) + paths = sorted((package / 'consumers').glob('*.yaml')) + if not paths: + raise ValueError('package consumers missing') + consumers = [native.load(path) for path in paths] + native.validate_cluster_capacity(consumers) + registry = json.loads((root / 'assurance/placement-owners.json').read_text()) + cells = {} + source_files = manifests + paths + [root / 'tools/verify_apps_pg_capacity.py', + root / 'assurance/placement-owners.json', package / 'scripts/render_consumers.py'] + for manifest in manifests: + doc = yaml.safe_load(manifest.read_text()) + name, spec = doc['metadata']['name'], doc['spec'] + roles = spec.get('managed', {}).get('roles', []) + names = sorted(r['name'] for r in roles) + cells[name] = {'consumers': names, 'ceiling': local.MAX_CONSUMERS, + 'retention_days': int(spec['backup']['retentionPolicy'].removesuffix('d')), + 'instances': spec['instances'], 'memory_limit': spec['resources']['limits']['memory'], + 'max_connections': int(spec['postgresql']['parameters']['max_connections']), + 'service_classes': {n: registry[n]['service_class'] for n in names}, + 'connection_limits': {r['name']: r['connectionLimit'] for r in roles}} + if cells[name]['retention_days'] != 30: + raise ValueError('apps-pg retention differs from disclosed 30-day contract') + for name in sorted(native.ALLOWED_CLUSTERS): + manifest = package / 'helm' / (name + '-cluster.yaml') + spec = yaml.safe_load(manifest.read_text())['spec'] + source_files.append(manifest) + selected = [d for d in consumers if native.cluster_name(d) == name] + horizon = native.erasure_horizon(selected) + retention = int(spec['backup']['retentionPolicy'].removesuffix('d')) + if retention != horizon['instanceRetentionDays']: + raise ValueError('package retention differs from effective consumer horizon') + if any(not d['honoured'] for d in horizon['consumers'].values()): + raise ValueError('consumer retention request cannot be honoured on shared cell') + cells[name] = {'consumers': sorted(horizon['consumers']), + 'ceiling': native.MAX_CONSUMERS_PER_CLUSTER, 'retention_days': retention, + 'instances': spec['instances'], 'memory_limit': spec['resources']['limits']['memory'], + 'max_connections': int(spec['postgresql']['parameters']['max_connections']), + 'service_classes': horizon['serviceClasses']} + names = {n for c in cells.values() for n in c['consumers']} + if names != set(registry) or any(not r.get('owner') or not r.get('evidence') for r in registry.values()): + raise ValueError('missing or stale placement owner') + for name, cell in cells.items(): + for consumer in cell['consumers']: + if registry[consumer]['cell'] != name: + raise ValueError('placement owner cell mismatch') + cell['owners'] = {n: registry[n]['owner'] for n in cell['consumers']} + return {'schema': 'railiance-platform.admission.v1', 'basis': 'source-declarations', + 'cells': cells, 'sources': {str(p.relative_to(root.parent)): digest(p) for p in source_files}} + + +def evaluate(contract, observation, now): + """Classify a value-safe observation, not certify its underlying truth.""" + if set(observation) != {'schema', 'cluster_uid', 'captured_at', 'signals'}: + raise ValueError('invalid observation envelope') + if observation['schema'] != 'railiance-platform.observation.v1': + raise ValueError('unsupported observation version') + if observation['cluster_uid'] != contract['cluster_uid']: + raise ValueError('wrong cluster') + capture = timestamp(observation['captured_at']) + capture_age = (now - capture).total_seconds() + if capture_age < 0: + raise ValueError('future capture') + specs = contract['signals'] + if not isinstance(observation['signals'], dict) or set(observation['signals']) - set(specs): + raise ValueError('unknown signals') + results = {} + for name, spec in specs.items(): + sample = observation['signals'].get(name) + state = 'missing' + if sample is not None: + if not isinstance(sample, dict) or set(sample) != {'observed_at', 'result'}: + raise ValueError('invalid signal fields') + if sample['result'] not in ('pass', 'fail', 'unavailable'): + raise ValueError('invalid result') + observed = timestamp(sample['observed_at']) + age = (now - observed).total_seconds() + if observed > capture: + raise ValueError('signal newer than capture') + if sample['result'] == 'unavailable': + state = 'unavailable' + elif sample['result'] == 'fail': + state = 'failed' + elif capture_age > contract['capture_max_age_seconds']: + state = 'stale' + elif age > spec['max_age_seconds']: + state = 'stale' + else: + state = 'healthy' + results[name] = {'state': state, 'owner': spec['owner']} + return {'schema': 'railiance-platform.assurance-signal.v1', + 'cluster_uid': contract['cluster_uid'], 'evaluated_at': now.isoformat(), + 'signals': results, 'transport': 'unmonitored', + 'guarantees': 'unsupported', 'threshold_status': 'local-diagnostic-only', + 'healthy': all(r['state'] == 'healthy' for r in results.values())} + + +def read_contract(path): + doc = json.loads(path.read_text()) + if doc['schema'] != 'railiance-platform.assurance-contract.v1' or not doc['signals']: + raise ValueError('invalid contract') + for budget in [doc['capture_max_age_seconds'], *(s['max_age_seconds'] for s in doc['signals'].values())]: + if type(budget) not in (int, float) or not math.isfinite(budget) or budget <= 0: + raise ValueError('invalid freshness budget') + if any(not s.get('owner') for s in doc['signals'].values()): + raise ValueError('signal owner missing') + return doc + + +def main(argv=None): + parser = argparse.ArgumentParser(description=__doc__) + sub = parser.add_subparsers(dest='command', required=True) + a = sub.add_parser('admission') + a.add_argument('--package', type=Path, default=ROOT.parent / 'rapp-postgres') + a.add_argument('--check', type=Path) + e = sub.add_parser('evaluate') + e.add_argument('observation', type=Path) + e.add_argument('--contract', type=Path, default=ROOT / 'assurance/service-contract.json') + e.add_argument('--now', help='Explicit UTC time for deterministic replay') + args = parser.parse_args(argv) + try: + if args.command == 'admission': + report = admission(package=args.package) + if args.check and report != json.loads(args.check.read_text()): + raise ValueError('admission source or disclosure drift; review before refreshing') + code = 0 + else: + now = timestamp(args.now) if args.now else datetime.now(timezone.utc) + report = evaluate(read_contract(args.contract), json.loads(args.observation.read_text()), now) + code = 0 if report['healthy'] else 1 + except (OSError, ValueError, KeyError, TypeError, AttributeError, yaml.YAMLError): + # Do not print parser errors, raw input, values, source paths or subprocess stderr. + print(json.dumps({'schema': 'railiance-platform.assurance-error.v1', 'error': 'invalid-or-unavailable-input'})) + return 2 + print(json.dumps(report, indent=2, allow_nan=False)) + return code + + +if __name__ == '__main__': + sys.exit(main()) diff --git a/tests/test_service_assurance.py b/tests/test_service_assurance.py new file mode 100644 index 0000000..20abc90 --- /dev/null +++ b/tests/test_service_assurance.py @@ -0,0 +1,162 @@ +import copy +from datetime import datetime, timezone +import importlib.util +import json +from pathlib import Path +import shutil +import subprocess +import sys +import tempfile +import unittest +from unittest.mock import patch + +import yaml + +ROOT = Path(__file__).resolve().parents[1] +spec = importlib.util.spec_from_file_location('service_assurance', ROOT / 'scripts/service_assurance.py') +m = importlib.util.module_from_spec(spec) +spec.loader.exec_module(m) + + +class EvidenceTests(unittest.TestCase): + def setUp(self): + self.now = datetime(2026, 9, 5, 12, tzinfo=timezone.utc) + self.contract = {'cluster_uid': 'expected', 'capture_max_age_seconds': 60, + 'signals': {'db.backup': {'owner': 'platform', 'max_age_seconds': 3600}}} + self.observation = {'schema': 'railiance-platform.observation.v1', + 'cluster_uid': 'expected', 'captured_at': '2026-09-05T12:00:00Z', + 'signals': {'db.backup': {'observed_at': '2026-09-05T11:30:00Z', 'result': 'pass'}}} + + def state(self): + return m.evaluate(self.contract, self.observation, self.now)['signals']['db.backup']['state'] + + def test_current_old_missing_failure_and_unavailable_are_distinct(self): + self.assertEqual(self.state(), 'healthy') + self.observation['signals']['db.backup']['observed_at'] = '2026-09-05T10:00:00Z' + self.assertEqual(self.state(), 'stale') + self.observation['signals']['db.backup']['result'] = 'fail' + self.assertEqual(self.state(), 'failed') + self.observation['signals']['db.backup']['result'] = 'unavailable' + self.assertEqual(self.state(), 'unavailable') + self.observation['signals'].clear() + self.assertEqual(self.state(), 'missing') + + def test_replayed_capture_cannot_stay_healthy(self): + self.observation['captured_at'] = '2026-09-05T11:59:00Z' + self.assertEqual(self.state(), 'healthy') + self.observation['captured_at'] = '2026-09-05T11:58:59Z' + self.assertEqual(self.state(), 'stale') + + def test_wrong_cluster_future_and_naive_times_refused(self): + for field, value in [('cluster_uid', 'other'), ('captured_at', '2026-09-05T12:01:00Z'), + ('captured_at', '2026-09-05T12:00:00')]: + d = copy.deepcopy(self.observation); d[field] = value + with self.subTest(field=field, value=value), self.assertRaises(ValueError): + m.evaluate(self.contract, d, self.now) + self.observation['signals']['db.backup']['observed_at'] = '2026-09-06T00:00:00Z' + with self.assertRaises(ValueError): + self.state() + + def test_no_unrecognized_payload_can_escape_in_error(self): + with tempfile.TemporaryDirectory() as directory: + p = Path(directory) / 'input.json' + self.observation['token'] = 'CANARY_MUST_NOT_ESCAPE' + p.write_text(json.dumps(self.observation)) + result = subprocess.run([sys.executable, str(ROOT / 'scripts/service_assurance.py'), + 'evaluate', str(p)], capture_output=True, text=True) + self.assertEqual(result.returncode, 2) + self.assertNotIn('CANARY', result.stdout + result.stderr) + + def test_unknown_signal_and_extra_sample_fields_fail(self): + for signals in [{'arbitrary': {}}, {'db.backup': {'observed_at': '2026-09-05T12:00:00Z', + 'result': 'pass', 'value': 'secret'}}]: + self.observation['signals'] = signals + with self.assertRaises(ValueError): + self.state() + + def test_health_does_not_imply_monitoring_or_guarantees(self): + result = m.evaluate(self.contract, self.observation, self.now) + self.assertTrue(result['healthy']) + self.assertEqual(result['transport'], 'unmonitored') + self.assertEqual(result['guarantees'], 'unsupported') + + +@unittest.skipUnless((ROOT.parent / 'rapp-postgres/scripts/render_consumers.py').exists(), + 'owner package checkout required for integration checks') +class AdmissionTests(unittest.TestCase): + def setUp(self): + self.tmp = tempfile.TemporaryDirectory() + self.addCleanup(self.tmp.cleanup) + self.root = Path(self.tmp.name) / 'railiance-platform' + self.package = self.root.parent / 'rapp-postgres' + for folder in ['tools', 'helm', 'assurance']: + shutil.copytree(ROOT / folder, self.root / folder) + for folder in ['scripts', 'helm', 'consumers']: + shutil.copytree(ROOT.parent / 'rapp-postgres' / folder, self.package / folder) + + def check(self): + return m.admission(self.root, self.package) + + def test_current_occupancy_and_real_overflow(self): + result = self.check()['cells'] + self.assertEqual(len(result['platform-pg']['consumers']), 4) + self.assertEqual(result['platform-pg-2']['consumers'], ['sbom-nexus']) + self.assertEqual(result['apps-pg-2']['consumers'], []) + + def test_unowned_consumer_fails(self): + p = self.root / 'assurance/placement-owners.json' + data = json.loads(p.read_text()); del data['audit-core']; p.write_text(json.dumps(data)) + with self.assertRaises(ValueError): self.check() + + def test_native_validator_rejects_fifth_consumer(self): + p = self.package / 'consumers/sbom-nexus.yaml' + d = yaml.safe_load(p.read_text()); d['spec']['cluster'] = 'platform-pg'; p.write_text(yaml.safe_dump(d)) + with self.assertRaisesRegex(ValueError, 'ceiling'): self.check() + + def test_native_retention_and_effective_horizon_both_enforced(self): + p = self.package / 'consumers/audit-core.yaml' + for days in [1, 7, 60]: + d = yaml.safe_load(p.read_text()); d['spec']['retention']['backupRetentionDays'] = days + p.write_text(yaml.safe_dump(d)) + with self.subTest(days=days), self.assertRaises(ValueError): self.check() + + def test_missing_overflow_and_instance_policy_drift(self): + (self.package / 'helm/platform-pg-2-cluster.yaml').unlink() + with self.assertRaises(OSError): self.check() + + +class CollectorTests(unittest.TestCase): + @classmethod + def setUpClass(cls): + sys.path.insert(0, str(ROOT / 'scripts')) + cls.collector = __import__('capture_service_observation') + + def test_wrong_cluster_stops_before_other_reads(self): + with patch.object(self.collector, 'query', return_value='wrong') as query: + with self.assertRaises(ValueError): self.collector.capture() + self.assertEqual(query.call_count, 1) + + def test_failed_native_reads_are_unavailable_not_pass(self): + contract = m.read_contract(ROOT / 'assurance/service-contract.json') + calls = [] + def query(args, **kwargs): + calls.append(args) + if args[:3] == ['get', 'namespace', 'kube-system']: + return contract['cluster_uid'] + raise ValueError('unavailable') + baseline = json.loads((ROOT / 'assurance/admission-baseline.json').read_text()) + with patch.object(self.collector, 'query', side_effect=query), patch.object(self.collector, 'admission', return_value=baseline): + observation = self.collector.capture() + for sample in observation['signals'].values(): self.assertEqual(sample['result'], 'unavailable') + self.assertNotIn('secret', [str(a).lower() for call in calls for a in call]) + result = m.evaluate(contract, observation, datetime.now(timezone.utc)) + self.assertFalse(result['healthy']) + + def test_quantity_conversion_and_invalid_units(self): + self.assertEqual(self.collector.memory_bytes('1024Mi'), 1024**3) + for value in ['NaN', '-1Gi', '1password', '']: + with self.subTest(value=value), self.assertRaises(ValueError): self.collector.memory_bytes(value) + + +if __name__ == '__main__': + unittest.main() diff --git a/workplans/README.md b/workplans/README.md index ddf39f9..59f0a24 100644 --- a/workplans/README.md +++ b/workplans/README.md @@ -1,7 +1,7 @@ # Current platform work -Reviewed 2026-09-05. Six open workplans: five blocked on explicit owner/live -gates and one ready for local work. Completed designs and implementations are +Reviewed 2026-09-05. Six open workplans: six blocked on explicit owner/live +gates; RPF-WP-0036 now has its repository implementation. Completed designs and implementations are under `archived/`; their IDs and UUIDs are preserved. The number of blocked plans is not a count of missing implementations or independent incidents. @@ -12,10 +12,10 @@ plans is not a count of missing implementations or independent incidents. | [RPF-WP-0025](RPF-WP-0025-openbao-operator-only-access.md) | Private OpenBao access; fresh attended callback/login then guarded retraction | Coordinate package, issuer, tunnel and DNS owners; keep the window separate. | | [RPF-WP-0015](RPF-WP-0015-audit-core-custody-and-recovery-coordination.md) | Two prepared recovery exercises; fresh synthetic-load/outage approvals and custody readiness | S3 contributes lease/ESO and snapshot/unseal proof; S1/S2 and audit-core execute their parts. | | [RPF-WP-0035](RPF-WP-0035-credential-lane-implementation.md) | One implementation queue for secrets-engine JWT, Fluid operator KV and preflight signing | Three independent task gates; no new approval inherited from the completed designs. | -| [RPF-WP-0036](RPF-WP-0036-platform-service-assurance.md) | **Ready:** service guarantees, recurring evidence, S3 emission, admission consistency, ownership handoff and demand review | Local design/checking can progress while live work waits. | +| [RPF-WP-0036](RPF-WP-0036-platform-service-assurance.md) | Implemented local assurance/admission; waits for recurring restore evidence, Q2 reception and owner handoff | Run the assurance commands; live acceptance and external ownership remain gated. | -Start with RPF-WP-0036-T02/T05 for work that does not require a live credential -or outage. Treat credential exposure closure as the highest-priority attended +RPF-WP-0036-T02/T05/T07 are complete; T03/T04/T06 retain the remaining +acceptance gates. Treat credential exposure closure as the highest-priority attended work; task order does not combine or waive approvals. [Assessment and disposition of every plan](../history/2026-09-05-platform-intent-workplan-assessment.md) diff --git a/workplans/RPF-WP-0036-platform-service-assurance.md b/workplans/RPF-WP-0036-platform-service-assurance.md index 9e0a246..137cee2 100644 --- a/workplans/RPF-WP-0036-platform-service-assurance.md +++ b/workplans/RPF-WP-0036-platform-service-assurance.md @@ -4,7 +4,7 @@ type: workplan title: "Close S3 service assurance and ownership gaps" domain: financials repo: railiance-platform -status: ready +status: blocked owner: codex created: "2026-09-05" updated: "2026-09-05" @@ -39,7 +39,7 @@ This certifies the source review, not live service health or external acceptance ```task id: RPF-WP-0036-T02 -status: todo +status: done priority: high state_hub_task_id: "eebcd5c7-4cfd-5084-9a21-ca1e4e748bdd" ``` @@ -61,7 +61,7 @@ and package dependency rather than a blanket new-cluster project here. ```task id: RPF-WP-0036-T03 -status: todo +status: wait priority: high state_hub_task_id: "d64446fb-870b-5ae9-9628-f1fb9d06c5a4" ``` @@ -84,7 +84,7 @@ pass as restore proof. Record independent recovery-key access without values. ```task id: RPF-WP-0036-T04 -status: todo +status: wait priority: high state_hub_task_id: "5351e0e4-6263-58f0-afb2-7c78c4cd6f68" ``` @@ -105,7 +105,7 @@ finish before the receiver, but end-to-end acceptance cannot. ```task id: RPF-WP-0036-T05 -status: todo +status: done priority: high state_hub_task_id: "5cded2e9-7edd-57bd-9f7e-7977b75004dc" ``` @@ -127,7 +127,7 @@ under this task without its own owner-reviewed migration. ```task id: RPF-WP-0036-T06 -status: todo +status: wait priority: medium state_hub_task_id: "3b74f79c-7366-5d00-bf0e-31c0464a8f33" ``` @@ -154,7 +154,7 @@ the assessment and this task does not assert acceptance for another repo. ```task id: RPF-WP-0036-T07 -status: todo +status: done priority: medium state_hub_task_id: "b32709d2-581d-5750-a04c-e3494a42d27e" ``` @@ -171,3 +171,53 @@ it to S3 by implication. review trigger, or start a bounded consumer-backed delivery plan with explicit acceptance criteria. “No accepted demand; keep deploy gated” is a valid result. No Valkey, MinIO, RabbitMQ or new provider purchase is authorized by this plan. + +## Implementation and remaining acceptance — 2026-09-05 + +T02 completed with `assurance/service-records.json` and ADR-0004: all three +CNPG cells, OpenBao and both backup delivery surfaces disclose unsupported +availability/RPO/RTO guarantees, existing evidence, retention and named decision +owners. Numeric commitments have not been invented or approved for other owners. + +T05 completed with the owner-native admission checker, hash-bound baseline and +placement registry. It reuses local/package validators and rejects unowned +consumers, fifth consumers, unhonoured retention and source disclosure drift. +Corrected tenant-engine's completed PostgreSQL cutover and the deployed SBOM +overflow cell. Live metadata matches one instance, 1Gi limit, 100 connections +and 30-day retention on all three deployed cells. Dated owner evidence remains +the authority for database/application cutover; this run moved no workloads. + +T07 completed with ADR-0005: reuse approved backup storage only for backup; +evaluate artifact-store's existing S3 interface for artifact demand; defer +cache, general object store and broker deployment with concrete demand/review +triggers. The fleet Q3 question is retained in T06's prepared master handoff; +no external architecture assignment or provisioning occurred. + +T03/T04 have working read-only `assurance-capture`, `assurance-check` and +`assurance-admission` entry points plus adversarial tests. The collector pins +the cluster UID and reads selected status, native aggregate connection counts, +metrics and unauthenticated seal status. It captures no Secrets, SQL text from +sessions, application rows or logs. Healthy/stale/missing/unavailable/failed +are distinct; unknown fields, future timestamps and wrong clusters fail closed. +The output explicitly says unmonitored and unsupported guarantees. + +- **T03 waits:** fresh owner-validated isolated restore/snapshot/offsite receipts, + independent recovery access, approved cadence and execution evidence. Existing + live experiments stay in RPF-WP-0015; offsite exposure stays in RPF-WP-0029. + No restore, snapshot creation, upload, scheduler or seal/unseal was run. +- **T04 waits:** accepted Q2 receiving contract and controlled failure/absence + delivery to a named recipient. The local producer is not a ratified Q2 + contract. Live metadata also surfaced failed ESO resources: forgejo-mailer, + reuse-surface-runtime and target-revenue-runtime. Their current/obsolete scope + and exact repair require consumer/platform acceptance before lane mutation; + failures remain visible rather than excluded. +- **T06 waits:** ownership acceptance and scoped alias/brief repair. The exact + path/hash/caller inventory, dated retention decision through 2026-10-05 and + prepared owner requests are in `assurance/ownership-handoffs.json` and + `docs/platform-ownership-handoffs.md`. The installed brief generator still + includes open legacy aliases; rerunning it alone cannot satisfy acceptance. + No messages or external handoff acceptance were fabricated. + +Evidence: `docs/evidence/RPF-WP-0036-assurance-2026-09-05.json`. +The plan is blocked on these explicit live/owner gates, not finished merely +because its repository implementation and tests pass.