Implement S3 service assurance and admission checks
Assistant: codex Assistant-Model: gpt-6-astra Assistant-Session: 01a06ecb-456a-71c2-b41e-0755d336e883
This commit is contained in:
codex
parent
8f828c7609
commit
234b1b559f
21 changed files with 1728 additions and 30 deletions
12
Makefile
12
Makefile
|
|
@ -550,3 +550,15 @@ wp0024-t02-driver-register: ## Register committed audit-core driver source witho
|
|||
|
||||
.PHONY: audit-whitehat-e2-preflight audit-whitehat-e2-project audit-whitehat-e2-cleanup audit-database-lease-recovery-preflight audit-database-lease-approval-check audit-database-lease-recovery-exercise railiance01-reboot-preflight wp0024-owner-show wp0024-owner-verify wp0024-owner-status wp0024-t02-driver-show wp0024-t02-driver-status wp0024-t02-driver-register apps-pg-isolation-probe
|
||||
.PHONY: db-deploy db-status db-shell db-logs forgejo-db-deploy forgejo-db-status forgejo-db-shell apps-pg-deploy apps-pg-status apps-pg-shell apps-pg-logs net-kingdom-pg-inter-hub-networkpolicy-deploy pg-deploy pg-status pg-pgpool-check valkey-deploy valkey-status openbao-repo openbao-dry-run openbao-overlay-apply openbao-verify-login-overlay openbao-deploy openbao-status openbao-verify openbao-verify-post-unseal openbao-configure-initial openbao-configure-ssh openbao-verify-ssh openbao-verify-authenticated openbao-configure-external-secrets-issue-core openbao-configure-external-secrets-activity-core openbao-configure-external-secrets-forgejo openbao-validate-restore-evidence openbao-validate-emergency-evidence credential-grants-validate credential-change-applier-dry-run credential-change-applier-apply-plan credential-change-applier-apply credential-change-runbook credential-change-record-evidence credential-change-lifecycle-plan credential-change-lifecycle-event credential-change-import-inventory agent-high-risk-boundary-check openbao-credential-change-appliers-dry-run openbao-configure-credential-change-appliers openbao-token-grants-dry-run openbao-configure-token-grants openbao-verify-token-grants-dry-run openbao-verify-token-grants openbao-verify-token-grants-smoke credential-helper-dry-run credential-tests credential-exec-ops-warden-smoke argocd-bootstrap-dry-run argocd-bootstrap-deploy argocd-repo-apply argocd-status backup forgejo-backup forgejo-backup-dry-run forgejo-backup-status forgejo-package-prune forgejo-package-prune-dry-run reef-deployables apps-pg-evidence consumption-preflight help
|
||||
|
||||
##@ S3 service assurance (read-only)
|
||||
.PHONY: assurance-admission assurance-capture assurance-check
|
||||
ASSURANCE_OBSERVATION ?= /tmp/railiance-platform-observation.json
|
||||
assurance-admission: ## Verify owner-native admission and disclosure baseline
|
||||
python3 scripts/service_assurance.py admission --check assurance/admission-baseline.json
|
||||
|
||||
assurance-capture: ## Capture selected status metadata from the pinned railiance01 cluster
|
||||
python3 scripts/capture_service_observation.py --output "$(ASSURANCE_OBSERVATION)"
|
||||
|
||||
assurance-check: ## Classify freshness/failure; nonzero means incomplete or invalid evidence
|
||||
python3 scripts/service_assurance.py evaluate "$(ASSURANCE_OBSERVATION)"
|
||||
|
|
|
|||
15
SCOPE.md
15
SCOPE.md
|
|
@ -22,7 +22,7 @@ or authorization authority. See [ADR-0001](docs/adr/ADR-0001-s3-platform-service
|
|||
| Service | Implemented / evidenced | Limit and authority |
|
||||
| --- | --- | --- |
|
||||
| `apps-pg` | CNPG source, per-consumer database ACLs, 20-connection allowances, 15-second timeouts; daily backup, continuous WAL and 30-day retention; 56-second scratch restore and 14/14 boundary probes on 2026-08-20 | One instance; restart recovery, no HA or measured latency SLO. Ceiling 3, overflow `apps-pg-2` provisionable but demand-gated. S3 owns this CR and its controls. |
|
||||
| `platform-pg` | Package-operated CNPG service, governed dynamic credentials, backup/restore evidence, Core Hub admission and private shadow evidence | `rapp-postgres` owns CR/provisioning/restore. S3 owns custody, placement policy and consumer disclosure. Published ceiling 4 declarations, counting the probe; reconcile current occupancy before admission. One instance is not HA. |
|
||||
| `platform-pg` | Package-operated CNPG service, governed dynamic credentials, backup/restore evidence, Core Hub admission and private shadow evidence | `rapp-postgres` owns CR/provisioning/restore. S3 owns custody, placement policy and consumer disclosure. Ceiling 4 declarations, counting the probe; platform-pg is 4/4 and the deployed platform-pg-2 hosts sbom-nexus at 1/4. One instance is not HA. |
|
||||
| OpenBao and credential delivery | CCR validation, policy/grant boundaries, approved apply interfaces, runtime KV/ESO and short-lived database lanes, ephemeral custody lifecycle | Package assets belong to `rapp-openbao`; custody/policy remains here. Existing helpers are not authority to mint arbitrary credentials. Operator-only access cutover and incident residuals remain open. |
|
||||
| Backup storage consumption | Governed off-cluster S3 backup destination and credential lane used by CNPG; encrypted upload tooling for other backup consumers | Resource procurement belongs to `resource-control`/`reef-storage`. This does not establish a general S3 service or deployed MinIO. One successful restore does not establish a recurring recovery guarantee. |
|
||||
| Cache / general object storage / messaging | Valkey deployment is gated; no supported consumer interface or deployed shared broker/general in-cluster object store is established by this review | Demand, reuse, ownership, service contract and recovery design must precede deployment. No engine/provider choice is made here. |
|
||||
|
|
@ -69,7 +69,8 @@ records every reviewed plan and the consolidation mapping.
|
|||
| Next | Numeric availability/recovery promises, evidence freshness, recurring drills, emission and admission drift lack a complete S3 acceptance loop | RPF-WP-0036-T02–T05 |
|
||||
| Next | Compatibility ownership, stale Hub aliases, and undeployed capability demand need explicit disposition | RPF-WP-0036-T06/T07 |
|
||||
|
||||
The ready work in RPF-WP-0036 can proceed while attended operations wait.
|
||||
RPF-WP-0036 now provides local assurance/admission checks and explicit service
|
||||
disclosures; its remaining tasks wait for live proof and owner acceptance.
|
||||
It must define achievable promises for the existing single-node substrate,
|
||||
not declare HA through documentation. Any stronger availability target needs
|
||||
an S1/S2 dependency and a package-level failover proof. Fleet-wide Q3 ownership
|
||||
|
|
@ -112,3 +113,13 @@ Read [AGENTS.md](AGENTS.md), [INTENT.md](INTENT.md), then
|
|||
`workplans/archived/`; [WORK-RECORDS.md](WORK-RECORDS.md) is the generated current
|
||||
index (archives omitted). Routine sync follows AGENTS.md. The dated `.custodian-brief.md` and Hub
|
||||
legacy aliases are not additional authoritative obligations.
|
||||
|
||||
## Assurance implementation — 2026-09-05
|
||||
|
||||
`make assurance-admission` checks package-native capacity/retention and source
|
||||
disclosure drift. `make assurance-capture` pins the cluster and collects only
|
||||
status metadata; `make assurance-check` fails on incomplete/stale/failed evidence.
|
||||
The first live run found all three database cells Ready with same-day backups
|
||||
and archiving, but three failing ESO resources. Recurring restore acceptance,
|
||||
validated restore adapters and Q2 delivery remain open. See
|
||||
[service assurance](docs/service-assurance.md) and RPF-WP-0036.
|
||||
|
|
|
|||
|
|
@ -13,7 +13,7 @@
|
|||
| workplan | RPF-WP-0027 | blocked | — | workplans/RPF-WP-0027-keycape-live-secret-exposure-recovery.md |
|
||||
| workplan | RPF-WP-0029 | blocked | — | workplans/RPF-WP-0029-backup-credential-default-removal.md |
|
||||
| workplan | RPF-WP-0035 | blocked | — | workplans/RPF-WP-0035-credential-lane-implementation.md |
|
||||
| workplan | RPF-WP-0036 | ready | — | workplans/RPF-WP-0036-platform-service-assurance.md |
|
||||
| workplan | RPF-WP-0036 | blocked | — | workplans/RPF-WP-0036-platform-service-assurance.md |
|
||||
| task | RPF-WP-0015-T01 | done | — | workplans/RPF-WP-0015-audit-core-custody-and-recovery-coordination.md |
|
||||
| task | RPF-WP-0015-T02 | wait | — | workplans/RPF-WP-0015-audit-core-custody-and-recovery-coordination.md |
|
||||
| task | RPF-WP-0015-T03 | wait | — | workplans/RPF-WP-0015-audit-core-custody-and-recovery-coordination.md |
|
||||
|
|
@ -34,9 +34,9 @@
|
|||
| task | RPF-WP-0035-T03 | wait | — | workplans/RPF-WP-0035-credential-lane-implementation.md |
|
||||
| task | RPF-WP-0035-T04 | wait | — | workplans/RPF-WP-0035-credential-lane-implementation.md |
|
||||
| task | RPF-WP-0036-T01 | done | — | workplans/RPF-WP-0036-platform-service-assurance.md |
|
||||
| task | RPF-WP-0036-T02 | todo | — | workplans/RPF-WP-0036-platform-service-assurance.md |
|
||||
| task | RPF-WP-0036-T03 | todo | — | workplans/RPF-WP-0036-platform-service-assurance.md |
|
||||
| task | RPF-WP-0036-T04 | todo | — | workplans/RPF-WP-0036-platform-service-assurance.md |
|
||||
| task | RPF-WP-0036-T05 | todo | — | workplans/RPF-WP-0036-platform-service-assurance.md |
|
||||
| task | RPF-WP-0036-T06 | todo | — | workplans/RPF-WP-0036-platform-service-assurance.md |
|
||||
| task | RPF-WP-0036-T07 | todo | — | workplans/RPF-WP-0036-platform-service-assurance.md |
|
||||
| task | RPF-WP-0036-T02 | done | — | workplans/RPF-WP-0036-platform-service-assurance.md |
|
||||
| task | RPF-WP-0036-T03 | wait | — | workplans/RPF-WP-0036-platform-service-assurance.md |
|
||||
| task | RPF-WP-0036-T04 | wait | — | workplans/RPF-WP-0036-platform-service-assurance.md |
|
||||
| task | RPF-WP-0036-T05 | done | — | workplans/RPF-WP-0036-platform-service-assurance.md |
|
||||
| task | RPF-WP-0036-T06 | wait | — | workplans/RPF-WP-0036-platform-service-assurance.md |
|
||||
| task | RPF-WP-0036-T07 | done | — | workplans/RPF-WP-0036-platform-service-assurance.md |
|
||||
|
|
|
|||
95
assurance/admission-baseline.json
Normal file
95
assurance/admission-baseline.json
Normal file
|
|
@ -0,0 +1,95 @@
|
|||
{
|
||||
"schema": "railiance-platform.admission.v1",
|
||||
"basis": "source-declarations",
|
||||
"cells": {
|
||||
"apps-pg": {
|
||||
"consumers": [
|
||||
"coulomb_social",
|
||||
"vergabe"
|
||||
],
|
||||
"ceiling": 3,
|
||||
"retention_days": 30,
|
||||
"instances": 1,
|
||||
"memory_limit": "1Gi",
|
||||
"max_connections": 100,
|
||||
"service_classes": {
|
||||
"coulomb_social": "interactive",
|
||||
"vergabe": "interactive"
|
||||
},
|
||||
"connection_limits": {
|
||||
"vergabe": 20,
|
||||
"coulomb_social": 20
|
||||
},
|
||||
"owners": {
|
||||
"coulomb_social": "coulomb-social",
|
||||
"vergabe": "vergabe-teilnahme"
|
||||
}
|
||||
},
|
||||
"apps-pg-2": {
|
||||
"consumers": [],
|
||||
"ceiling": 3,
|
||||
"retention_days": 30,
|
||||
"instances": 1,
|
||||
"memory_limit": "1Gi",
|
||||
"max_connections": 100,
|
||||
"service_classes": {},
|
||||
"connection_limits": {},
|
||||
"owners": {}
|
||||
},
|
||||
"platform-pg": {
|
||||
"consumers": [
|
||||
"audit-core",
|
||||
"core-hub",
|
||||
"isolation-probe",
|
||||
"tenant-engine"
|
||||
],
|
||||
"ceiling": 4,
|
||||
"retention_days": 30,
|
||||
"instances": 1,
|
||||
"memory_limit": "1Gi",
|
||||
"max_connections": 100,
|
||||
"service_classes": {
|
||||
"audit-core": "batch",
|
||||
"core-hub": "latency-critical",
|
||||
"isolation-probe": "interactive",
|
||||
"tenant-engine": "latency-critical"
|
||||
},
|
||||
"owners": {
|
||||
"audit-core": "audit-core",
|
||||
"core-hub": "core-hub",
|
||||
"isolation-probe": "rapp-postgres",
|
||||
"tenant-engine": "tenant-engine"
|
||||
}
|
||||
},
|
||||
"platform-pg-2": {
|
||||
"consumers": [
|
||||
"sbom-nexus"
|
||||
],
|
||||
"ceiling": 4,
|
||||
"retention_days": 30,
|
||||
"instances": 1,
|
||||
"memory_limit": "1Gi",
|
||||
"max_connections": 100,
|
||||
"service_classes": {
|
||||
"sbom-nexus": "interactive"
|
||||
},
|
||||
"owners": {
|
||||
"sbom-nexus": "sbom-nexus"
|
||||
}
|
||||
}
|
||||
},
|
||||
"sources": {
|
||||
"railiance-platform/helm/apps-pg-cluster.yaml": "9885c81338e1c3bdb0558839552315720068c5d355819e4a1320fbd677d0ffcf",
|
||||
"railiance-platform/helm/apps-pg-2-cluster.yaml": "e9e615a0b5de8597a5501b2d8740a763ba3622cbebb765ce651236c2175de942",
|
||||
"rapp-postgres/consumers/audit-core.yaml": "a2db53f901df056546b2c028f287f8fc50486e6e6cd81026b9e926b6b64272b3",
|
||||
"rapp-postgres/consumers/core-hub.yaml": "8fe3e8b4488548b2f5e01f1d46edd9fd5fe4f0e4f65a3821f172adbfb5ae1d84",
|
||||
"rapp-postgres/consumers/isolation-probe.yaml": "8ea24d3bd04ab683b47ceff2586932ea4fdb0d45c49896167bb5b352a67a5cfb",
|
||||
"rapp-postgres/consumers/sbom-nexus.yaml": "0b21d05fe7b9d21cbc9eb761f455329739ceb71f8dd3d1200971426962c11236",
|
||||
"rapp-postgres/consumers/tenant-engine.yaml": "92f36d587f001c86ff401aedd25cb49fa171d49e000b0a6ab551e182e5c11448",
|
||||
"railiance-platform/tools/verify_apps_pg_capacity.py": "b571ec63b0a7501dabba37a84af9d2cfcb7737ed891a7ea2ab05c54010b1893e",
|
||||
"railiance-platform/assurance/placement-owners.json": "b9a52f755552dc4bb83536dca8aadca691a4f8bc47089e9e1962902abcec7aa2",
|
||||
"rapp-postgres/scripts/render_consumers.py": "2dcfd68f94a3d6c7e1072c4a55a497b1697eab30acaecae53865c9b1a596773a",
|
||||
"rapp-postgres/helm/platform-pg-cluster.yaml": "461da0d43e61a4ddca29d6e98080b8b1642bd3303f3395da1b0beb28775c2c5c",
|
||||
"rapp-postgres/helm/platform-pg-2-cluster.yaml": "06b92a76189b1a33d1eee3f5c835a3e94269ad7b040383c04e8028c2c3fe766a"
|
||||
}
|
||||
}
|
||||
131
assurance/ownership-handoffs.json
Normal file
131
assurance/ownership-handoffs.json
Normal file
|
|
@ -0,0 +1,131 @@
|
|||
{
|
||||
"schema": "railiance-platform.ownership-handoffs.v1",
|
||||
"reviewed": "2026-09-05",
|
||||
"decision": "retain existing operational entry points through review; no bulk move",
|
||||
"records": [
|
||||
{
|
||||
"surface": "forge-runtime",
|
||||
"proposed_accepting_owner": "railiance-forge",
|
||||
"status": "retained-pending-acceptance",
|
||||
"review_due": "2026-10-05",
|
||||
"callers_to_verify": "activity-core command definitions and railiance-apps compatibility callers",
|
||||
"source_files": [
|
||||
{
|
||||
"path": "tools/cmd/forgejo-backup",
|
||||
"sha256": "a20f0aebb22f0978c0f45f74e4ac55a927b080910844296acabec10f45110cb6"
|
||||
},
|
||||
{
|
||||
"path": "tools/cmd/forgejo-package-prune",
|
||||
"sha256": "66bababca54931115831aa42b7ad95bb4ee348d4057b5bfd1f5a6ae13cfa8ea3"
|
||||
},
|
||||
{
|
||||
"path": "scripts/forgejo_package_prune.py",
|
||||
"sha256": "63ba4df1c937a0067ea3bb4b1efbe4cfa47205f34e80f337ca71e4a0e6a31db4"
|
||||
},
|
||||
{
|
||||
"path": "tools/cmd/refresh-live-images",
|
||||
"sha256": "341676396e2c2a85f4585fcbf5c7e085805c9228eae8961740e19960ef2ba5ea"
|
||||
},
|
||||
{
|
||||
"path": "scripts/refresh_live_images.py",
|
||||
"sha256": "d1bcd88f3b757d98fa4cfa033329fe7b230f7bf1c6b466fc9cccc6804edf12b6"
|
||||
},
|
||||
{
|
||||
"path": "docs/forgejo-backup.md",
|
||||
"sha256": "69f6f38de902b80c6de161087b1d5738d1dea7ea35b3d49816d2ea67a1a5329b"
|
||||
},
|
||||
{
|
||||
"path": "docs/forgejo-package-prune.md",
|
||||
"sha256": "4f389876f52ba273f6a43c2d7d721a64c520ed9620e4a04899a423775224572a"
|
||||
}
|
||||
],
|
||||
"acceptance": "Owner names canonical replacements; update all callers and prove dry-run/smoke and rollback before source removal. No external acceptance asserted."
|
||||
},
|
||||
{
|
||||
"surface": "openbao-package",
|
||||
"proposed_accepting_owner": "rapp-openbao",
|
||||
"status": "retained-pending-acceptance",
|
||||
"review_due": "2026-10-05",
|
||||
"callers_to_verify": "Makefile package dispatch and existing operator entry points",
|
||||
"source_files": [
|
||||
{
|
||||
"path": "scripts/openbao-verify.sh",
|
||||
"sha256": "1e118570c8682619a46413f55fa4690482db342f6b6c83e6644c908aa7a667ee"
|
||||
},
|
||||
{
|
||||
"path": "scripts/openbao-verify-login-overlay.sh",
|
||||
"sha256": "57438714f1b7aac0c37b8bdf525c5387f812a6d30532bb98b395e5db5208ad6d"
|
||||
},
|
||||
{
|
||||
"path": "docs/rapp-openbao-compatibility-handoff.md",
|
||||
"sha256": "65ff0b39ee7312c8a242d24cf3434f9d0636a830e841f1a70de560c96cd8d57f"
|
||||
}
|
||||
],
|
||||
"acceptance": "Owner names canonical replacements; update all callers and prove dry-run/smoke and rollback before source removal. No external acceptance asserted."
|
||||
},
|
||||
{
|
||||
"surface": "gitops-bootstrap",
|
||||
"proposed_accepting_owner": "railiance-cluster / railiance-enablement",
|
||||
"status": "retained-pending-acceptance",
|
||||
"review_due": "2026-10-05",
|
||||
"callers_to_verify": "cluster bootstrap and generic GitOps entry points",
|
||||
"source_files": [
|
||||
{
|
||||
"path": "argocd/bootstrap/00-railiance-bootstrap-project.yaml",
|
||||
"sha256": "1fcdff98b1870e8867df65deeabd759e1357f0359af50ac6229f12ea3ec695ea"
|
||||
},
|
||||
{
|
||||
"path": "argocd/bootstrap/01-railiance-tenants-project.yaml",
|
||||
"sha256": "cf1823ed2294b874730de6d908c0f471d9b67f9ac07bb51a1456de6f77154d07"
|
||||
},
|
||||
{
|
||||
"path": "argocd/bootstrap/02-railiance-platform-addons-project.yaml",
|
||||
"sha256": "10b4596dcf6ac369329794612b2f43469a6f4ea1fa99759f92d1d66a50841cf8"
|
||||
},
|
||||
{
|
||||
"path": "argocd/bootstrap/10-railiance-apps-root.application.yaml",
|
||||
"sha256": "0eadd51880488442b6c8ed31470fa27c123637b97d66c56a2489d06a03a3c0ba"
|
||||
},
|
||||
{
|
||||
"path": "argocd/bootstrap/kustomization.yaml",
|
||||
"sha256": "5eaa17dc142ad134ef044fa9914eaea3ecaa5572b705be8e89660ecdb8a15b99"
|
||||
}
|
||||
],
|
||||
"acceptance": "Owner names canonical replacements; update all callers and prove dry-run/smoke and rollback before source removal. No external acceptance asserted."
|
||||
},
|
||||
{
|
||||
"surface": "application-releases",
|
||||
"proposed_accepting_owner": "railiance-apps / individual package owners",
|
||||
"status": "retained-pending-acceptance",
|
||||
"review_due": "2026-10-05",
|
||||
"callers_to_verify": "ArgoCD application reconciliation",
|
||||
"source_files": [
|
||||
{
|
||||
"path": "argocd/applications/target-revenue.application.yaml",
|
||||
"sha256": "c59ff45f7cf5dae47cf39c32fbcde3a614aa79be6a9edb07079f833cde1fa38b"
|
||||
},
|
||||
{
|
||||
"path": "argocd/applications/external-secrets.application.yaml",
|
||||
"sha256": "545d560c35ab1515f444099847730d7e9d8b7bd14beb30878ebc2f4bd99a5080"
|
||||
},
|
||||
{
|
||||
"path": "argocd/applications/openbao-secretstore.application.yaml",
|
||||
"sha256": "01ca88f200be6c23926b0ccf17ef7a3ab0d247044469edd92e9571820119943b"
|
||||
},
|
||||
{
|
||||
"path": "argocd/applications/issue-core.application.yaml",
|
||||
"sha256": "079fdb118996eed8e579f6a658dc11a153c8fc20998daf6379533d30eb339654"
|
||||
}
|
||||
],
|
||||
"acceptance": "Owner names canonical replacements; update all callers and prove dry-run/smoke and rollback before source removal. No external acceptance asserted."
|
||||
},
|
||||
{
|
||||
"surface": "derived-record-aliases",
|
||||
"proposed_accepting_owner": "repo-manager / state-hub",
|
||||
"status": "retained-pending-owner-repair",
|
||||
"review_due": "2026-10-05",
|
||||
"source_ref": "history/2026-09-05-platform-intent-workplan-assessment.md#derived-state-caveat",
|
||||
"acceptance": "Scoped legacy alias retirement and generated brief agree with canonical source; no UUID edits or blanket retirement."
|
||||
}
|
||||
]
|
||||
}
|
||||
9
assurance/placement-owners.json
Normal file
9
assurance/placement-owners.json
Normal file
|
|
@ -0,0 +1,9 @@
|
|||
{
|
||||
"vergabe": {"owner": "vergabe-teilnahme", "cell": "apps-pg", "service_class": "interactive", "evidence": "docs/evidence/RPF-WP-0019-isolation-2026-08-20.md"},
|
||||
"coulomb_social": {"owner": "coulomb-social", "cell": "apps-pg", "service_class": "interactive", "evidence": "docs/evidence/RPF-WP-0019-isolation-2026-08-20.md"},
|
||||
"audit-core": {"owner": "audit-core", "cell": "platform-pg", "evidence": "rapp-postgres/consumers/audit-core.yaml"},
|
||||
"tenant-engine": {"owner": "tenant-engine", "cell": "platform-pg", "evidence": "rapp-postgres/docs/evidence/tenant-engine-postgres-cutover-2026-08-21.md"},
|
||||
"core-hub": {"owner": "core-hub", "cell": "platform-pg", "evidence": "docs/evidence/core-hub-postgres-capacity-admission-2026-08-20.md"},
|
||||
"isolation-probe": {"owner": "rapp-postgres", "cell": "platform-pg", "evidence": "rapp-postgres/consumers/isolation-probe.yaml"},
|
||||
"sbom-nexus": {"owner": "sbom-nexus", "cell": "platform-pg-2", "evidence": "rapp-postgres/docs/evidence/RAPP-POSTGRES-WP-0005-T04-boundary-restore-2026-08-22.md"}
|
||||
}
|
||||
96
assurance/service-contract.json
Normal file
96
assurance/service-contract.json
Normal file
|
|
@ -0,0 +1,96 @@
|
|||
{
|
||||
"schema": "railiance-platform.assurance-contract.v1",
|
||||
"cluster_uid": "a553c742-0115-43d4-99a4-a5ca56fe0786",
|
||||
"capture_max_age_seconds": 900,
|
||||
"threshold_status": "local-diagnostic-only",
|
||||
"signals": {
|
||||
"apps-pg.ready": {
|
||||
"owner": "railiance-platform",
|
||||
"max_age_seconds": 900
|
||||
},
|
||||
"apps-pg.backup": {
|
||||
"owner": "railiance-platform",
|
||||
"max_age_seconds": 129600
|
||||
},
|
||||
"apps-pg.wal": {
|
||||
"owner": "railiance-platform",
|
||||
"max_age_seconds": 900
|
||||
},
|
||||
"apps-pg.restore": {
|
||||
"owner": "railiance-platform",
|
||||
"max_age_seconds": 2592000
|
||||
},
|
||||
"apps-pg.headroom": {
|
||||
"owner": "railiance-platform",
|
||||
"max_age_seconds": 900
|
||||
},
|
||||
"platform-pg.ready": {
|
||||
"owner": "rapp-postgres",
|
||||
"max_age_seconds": 900
|
||||
},
|
||||
"platform-pg.backup": {
|
||||
"owner": "rapp-postgres",
|
||||
"max_age_seconds": 129600
|
||||
},
|
||||
"platform-pg.wal": {
|
||||
"owner": "rapp-postgres",
|
||||
"max_age_seconds": 900
|
||||
},
|
||||
"platform-pg.restore": {
|
||||
"owner": "rapp-postgres",
|
||||
"max_age_seconds": 2592000
|
||||
},
|
||||
"platform-pg.headroom": {
|
||||
"owner": "rapp-postgres",
|
||||
"max_age_seconds": 900
|
||||
},
|
||||
"platform-pg-2.ready": {
|
||||
"owner": "rapp-postgres",
|
||||
"max_age_seconds": 900
|
||||
},
|
||||
"platform-pg-2.backup": {
|
||||
"owner": "rapp-postgres",
|
||||
"max_age_seconds": 129600
|
||||
},
|
||||
"platform-pg-2.wal": {
|
||||
"owner": "rapp-postgres",
|
||||
"max_age_seconds": 900
|
||||
},
|
||||
"platform-pg-2.restore": {
|
||||
"owner": "rapp-postgres",
|
||||
"max_age_seconds": 2592000
|
||||
},
|
||||
"platform-pg-2.headroom": {
|
||||
"owner": "rapp-postgres",
|
||||
"max_age_seconds": 900
|
||||
},
|
||||
"openbao.seal": {
|
||||
"owner": "railiance-platform",
|
||||
"max_age_seconds": 900
|
||||
},
|
||||
"openbao.snapshot": {
|
||||
"owner": "railiance-platform",
|
||||
"max_age_seconds": 129600
|
||||
},
|
||||
"openbao.restore": {
|
||||
"owner": "railiance-platform",
|
||||
"max_age_seconds": 2592000
|
||||
},
|
||||
"offsite.upload": {
|
||||
"owner": "railiance-platform",
|
||||
"max_age_seconds": 129600
|
||||
},
|
||||
"offsite.restore": {
|
||||
"owner": "railiance-platform",
|
||||
"max_age_seconds": 2592000
|
||||
},
|
||||
"eso.ready": {
|
||||
"owner": "railiance-platform",
|
||||
"max_age_seconds": 900
|
||||
},
|
||||
"eso.refresh": {
|
||||
"owner": "railiance-platform",
|
||||
"max_age_seconds": 3600
|
||||
}
|
||||
}
|
||||
}
|
||||
198
assurance/service-records.json
Normal file
198
assurance/service-records.json
Normal file
|
|
@ -0,0 +1,198 @@
|
|||
{
|
||||
"schema": "railiance-platform.service-records.v1",
|
||||
"reviewed": "2026-09-05",
|
||||
"review_owner": "railiance-platform",
|
||||
"review_scope": "S3 disclosure of unsupported guarantees; not external package approval",
|
||||
"services": [
|
||||
{
|
||||
"service": "apps-pg",
|
||||
"accountable_owner": "railiance-platform",
|
||||
"package_or_consumer_owner": "railiance-platform",
|
||||
"operator_role": "attended platform operator; application proof by consumer owner",
|
||||
"consumers": [
|
||||
"vergabe",
|
||||
"coulomb_social"
|
||||
],
|
||||
"failure_domain": "single-node railiance01",
|
||||
"availability": {
|
||||
"status": "unsupported",
|
||||
"target": null,
|
||||
"decision_owner": "railiance-platform + railiance-platform"
|
||||
},
|
||||
"rpo": {
|
||||
"status": "unsupported",
|
||||
"target_seconds": null,
|
||||
"decision_owner": "railiance-platform + railiance-platform"
|
||||
},
|
||||
"rto": {
|
||||
"status": "unsupported",
|
||||
"target_seconds": null,
|
||||
"decision_owner": "railiance-platform + railiance-platform"
|
||||
},
|
||||
"retention": "30 days",
|
||||
"existing_evidence": "docs/evidence/RPF-WP-0019-backup-restore-2026-08-20.md",
|
||||
"recovery_custody": "OpenBao: 2-of-3 operator quorum plus independent encrypted snapshot custody; database/offsite: governed backup lane and separately available restore key. Availability not verified in this task.",
|
||||
"maintenance_abort": "docs/railiance01-coordinated-reboot.md; stop before mutation when freshness, quorum, consumer readiness or named abort operator is absent",
|
||||
"freshness_policy": "assurance/service-contract.json; diagnostic thresholds only, no installed cadence approval",
|
||||
"requirement_assessment": "No accepted numeric consumer availability/RPO/RTO requirement found in the reviewed contracts. Service classes inform placement, not guarantees. Refuse any request for guaranteed HA/node-loss recovery until matched to supported substrate and package proof."
|
||||
},
|
||||
{
|
||||
"service": "platform-pg",
|
||||
"accountable_owner": "railiance-platform",
|
||||
"package_or_consumer_owner": "rapp-postgres",
|
||||
"operator_role": "attended platform operator; application proof by consumer owner",
|
||||
"consumers": [
|
||||
"audit-core",
|
||||
"tenant-engine",
|
||||
"core-hub",
|
||||
"isolation-probe"
|
||||
],
|
||||
"failure_domain": "single-node railiance01",
|
||||
"availability": {
|
||||
"status": "unsupported",
|
||||
"target": null,
|
||||
"decision_owner": "railiance-platform + rapp-postgres"
|
||||
},
|
||||
"rpo": {
|
||||
"status": "unsupported",
|
||||
"target_seconds": null,
|
||||
"decision_owner": "railiance-platform + rapp-postgres"
|
||||
},
|
||||
"rto": {
|
||||
"status": "unsupported",
|
||||
"target_seconds": null,
|
||||
"decision_owner": "railiance-platform + rapp-postgres"
|
||||
},
|
||||
"retention": "30 days",
|
||||
"existing_evidence": "rapp-postgres/docs/evidence/backup-restore-2026-08-13.md",
|
||||
"recovery_custody": "OpenBao: 2-of-3 operator quorum plus independent encrypted snapshot custody; database/offsite: governed backup lane and separately available restore key. Availability not verified in this task.",
|
||||
"maintenance_abort": "docs/railiance01-coordinated-reboot.md; stop before mutation when freshness, quorum, consumer readiness or named abort operator is absent",
|
||||
"freshness_policy": "assurance/service-contract.json; diagnostic thresholds only, no installed cadence approval",
|
||||
"requirement_assessment": "No accepted numeric consumer availability/RPO/RTO requirement found in the reviewed contracts. Service classes inform placement, not guarantees. Refuse any request for guaranteed HA/node-loss recovery until matched to supported substrate and package proof."
|
||||
},
|
||||
{
|
||||
"service": "platform-pg-2",
|
||||
"accountable_owner": "railiance-platform",
|
||||
"package_or_consumer_owner": "rapp-postgres",
|
||||
"operator_role": "attended platform operator; application proof by consumer owner",
|
||||
"consumers": [
|
||||
"sbom-nexus"
|
||||
],
|
||||
"failure_domain": "single-node railiance01",
|
||||
"availability": {
|
||||
"status": "unsupported",
|
||||
"target": null,
|
||||
"decision_owner": "railiance-platform + rapp-postgres"
|
||||
},
|
||||
"rpo": {
|
||||
"status": "unsupported",
|
||||
"target_seconds": null,
|
||||
"decision_owner": "railiance-platform + rapp-postgres"
|
||||
},
|
||||
"rto": {
|
||||
"status": "unsupported",
|
||||
"target_seconds": null,
|
||||
"decision_owner": "railiance-platform + rapp-postgres"
|
||||
},
|
||||
"retention": "30 days",
|
||||
"existing_evidence": "rapp-postgres/docs/evidence/RAPP-POSTGRES-WP-0005-T04-boundary-restore-2026-08-22.md",
|
||||
"recovery_custody": "OpenBao: 2-of-3 operator quorum plus independent encrypted snapshot custody; database/offsite: governed backup lane and separately available restore key. Availability not verified in this task.",
|
||||
"maintenance_abort": "docs/railiance01-coordinated-reboot.md; stop before mutation when freshness, quorum, consumer readiness or named abort operator is absent",
|
||||
"freshness_policy": "assurance/service-contract.json; diagnostic thresholds only, no installed cadence approval",
|
||||
"requirement_assessment": "No accepted numeric consumer availability/RPO/RTO requirement found in the reviewed contracts. Service classes inform placement, not guarantees. Refuse any request for guaranteed HA/node-loss recovery until matched to supported substrate and package proof."
|
||||
},
|
||||
{
|
||||
"service": "openbao",
|
||||
"accountable_owner": "railiance-platform",
|
||||
"package_or_consumer_owner": "rapp-openbao",
|
||||
"operator_role": "attended platform operator; application proof by consumer owner",
|
||||
"consumers": [
|
||||
"approved credential lanes"
|
||||
],
|
||||
"failure_domain": "single-node railiance01",
|
||||
"availability": {
|
||||
"status": "unsupported",
|
||||
"target": null,
|
||||
"decision_owner": "railiance-platform + rapp-openbao"
|
||||
},
|
||||
"rpo": {
|
||||
"status": "unsupported",
|
||||
"target_seconds": null,
|
||||
"decision_owner": "railiance-platform + rapp-openbao"
|
||||
},
|
||||
"rto": {
|
||||
"status": "unsupported",
|
||||
"target_seconds": null,
|
||||
"decision_owner": "railiance-platform + rapp-openbao"
|
||||
},
|
||||
"retention": "KV history/audit/snapshot retention not accepted",
|
||||
"existing_evidence": "reviews/WARDEN-WP-0027-T02-DRILL-20260822-01-openbao-snapshot-receipt.json",
|
||||
"recovery_custody": "OpenBao: 2-of-3 operator quorum plus independent encrypted snapshot custody; database/offsite: governed backup lane and separately available restore key. Availability not verified in this task.",
|
||||
"maintenance_abort": "docs/railiance01-coordinated-reboot.md; stop before mutation when freshness, quorum, consumer readiness or named abort operator is absent",
|
||||
"freshness_policy": "assurance/service-contract.json; diagnostic thresholds only, no installed cadence approval",
|
||||
"requirement_assessment": "No accepted numeric consumer availability/RPO/RTO requirement found in the reviewed contracts. Service classes inform placement, not guarantees. Refuse any request for guaranteed HA/node-loss recovery until matched to supported substrate and package proof."
|
||||
},
|
||||
{
|
||||
"service": "forgejo-backup",
|
||||
"accountable_owner": "railiance-platform",
|
||||
"package_or_consumer_owner": "railiance-forge",
|
||||
"operator_role": "attended platform operator; application proof by consumer owner",
|
||||
"consumers": [
|
||||
"forgejo"
|
||||
],
|
||||
"failure_domain": "source host plus separately encrypted Nextcloud copy",
|
||||
"availability": {
|
||||
"status": "unsupported",
|
||||
"target": null,
|
||||
"decision_owner": "railiance-platform + railiance-forge"
|
||||
},
|
||||
"rpo": {
|
||||
"status": "unsupported",
|
||||
"target_seconds": null,
|
||||
"decision_owner": "railiance-platform + railiance-forge"
|
||||
},
|
||||
"rto": {
|
||||
"status": "unsupported",
|
||||
"target_seconds": null,
|
||||
"decision_owner": "railiance-platform + railiance-forge"
|
||||
},
|
||||
"retention": "14 daily + 4 weekly target; local 7/type",
|
||||
"existing_evidence": "docs/forgejo-backup.md",
|
||||
"recovery_custody": "OpenBao: 2-of-3 operator quorum plus independent encrypted snapshot custody; database/offsite: governed backup lane and separately available restore key. Availability not verified in this task.",
|
||||
"maintenance_abort": "docs/railiance01-coordinated-reboot.md; stop before mutation when freshness, quorum, consumer readiness or named abort operator is absent",
|
||||
"freshness_policy": "assurance/service-contract.json; diagnostic thresholds only, no installed cadence approval",
|
||||
"requirement_assessment": "No accepted numeric consumer availability/RPO/RTO requirement found in the reviewed contracts. Service classes inform placement, not guarantees. Refuse any request for guaranteed HA/node-loss recovery until matched to supported substrate and package proof."
|
||||
},
|
||||
{
|
||||
"service": "cnpg-option-a",
|
||||
"accountable_owner": "railiance-platform",
|
||||
"package_or_consumer_owner": "railiance-platform",
|
||||
"operator_role": "attended platform operator; application proof by consumer owner",
|
||||
"consumers": [
|
||||
"production-of-record CNPG logical dumps"
|
||||
],
|
||||
"failure_domain": "source host plus separately encrypted Nextcloud copy",
|
||||
"availability": {
|
||||
"status": "unsupported",
|
||||
"target": null,
|
||||
"decision_owner": "railiance-platform + railiance-platform"
|
||||
},
|
||||
"rpo": {
|
||||
"status": "unsupported",
|
||||
"target_seconds": null,
|
||||
"decision_owner": "railiance-platform + railiance-platform"
|
||||
},
|
||||
"rto": {
|
||||
"status": "unsupported",
|
||||
"target_seconds": null,
|
||||
"decision_owner": "railiance-platform + railiance-platform"
|
||||
},
|
||||
"retention": "offsite retention not accepted",
|
||||
"existing_evidence": "docs/cnpg-option-a-backup.md",
|
||||
"recovery_custody": "OpenBao: 2-of-3 operator quorum plus independent encrypted snapshot custody; database/offsite: governed backup lane and separately available restore key. Availability not verified in this task.",
|
||||
"maintenance_abort": "docs/railiance01-coordinated-reboot.md; stop before mutation when freshness, quorum, consumer readiness or named abort operator is absent",
|
||||
"freshness_policy": "assurance/service-contract.json; diagnostic thresholds only, no installed cadence approval",
|
||||
"requirement_assessment": "No accepted numeric consumer availability/RPO/RTO requirement found in the reviewed contracts. Service classes inform placement, not guarantees. Refuse any request for guaranteed HA/node-loss recovery until matched to supported substrate and package proof."
|
||||
}
|
||||
]
|
||||
}
|
||||
64
docs/adr/ADR-0004-service-assurance-disclosure.md
Normal file
64
docs/adr/ADR-0004-service-assurance-disclosure.md
Normal file
|
|
@ -0,0 +1,64 @@
|
|||
---
|
||||
id: RPLAT-ADR-0004
|
||||
title: "ADR-0004 — Disclose unsupported service guarantees and evidence freshness"
|
||||
status: accepted
|
||||
revision: "1.0"
|
||||
owner: railiance-platform
|
||||
date: "2026-09-05"
|
||||
last_reviewed: "2026-09-05"
|
||||
review_interval: 6m
|
||||
---
|
||||
|
||||
# ADR-0004 — Service assurance disclosure
|
||||
|
||||
## Context
|
||||
|
||||
The platform has dated backup/restore evidence and three single-instance CNPG
|
||||
cells, but no accepted numeric end-to-end availability/RPO/RTO commitments.
|
||||
One successful scratch restore is not a recovery guarantee. A successful
|
||||
source validator is not current live health. RPF-WP-0036 makes that distinction
|
||||
executable and visible to consumers.
|
||||
|
||||
## Decision
|
||||
|
||||
Publish `assurance/service-records.json` as the S3-owned disclosure of supported
|
||||
and unsupported guarantees. Availability, RPO and RTO remain explicitly
|
||||
unsupported until the named service/package/consumer owners accept measurable
|
||||
targets backed by evidence. This is a platform disclosure decision, not an
|
||||
external owner's approval. A consumer requesting a stronger guarantee needs a
|
||||
recorded placement/recovery decision before admission. In particular, CNPG on
|
||||
one host does not provide host-loss HA; the dependency is S1 host failure
|
||||
domains, S2 runtime placement and package-level failover proof.
|
||||
|
||||
Use `assurance/service-contract.json` only for local diagnostics: 15-minute
|
||||
capture/health age, 36-hour successful backup/upload age, 30-day restore age,
|
||||
one-hour ESO refresh age, and 80% connection/memory occupancy for headroom.
|
||||
These intentionally conservative defaults expose
|
||||
missing evidence; they are neither SLAs nor approvals to install schedules.
|
||||
Service-specific cadence and receiver acceptance remain RPF-WP-0036-T03/T04.
|
||||
A different legitimate refresh interval needs an explicit service contract,
|
||||
not automatic acceptance of an indefinitely stale observation.
|
||||
|
||||
## Consequences
|
||||
|
||||
The checker distinguishes healthy, stale, missing, unavailable and failed
|
||||
samples. It rejects wrong clusters, unknown payload fields and future times.
|
||||
Its receipt states `transport: unmonitored` and `guarantees: unsupported`.
|
||||
Receipt syntax and freshness validation do not prove the truth of a supplied
|
||||
observation or replace a signed native restore receipt. The live collector
|
||||
queries only fixed metadata/status fields, a pg_stat_activity count and
|
||||
unauthenticated seal status; it cannot fetch Secrets or run backup, restore or
|
||||
seal/unseal mutations.
|
||||
|
||||
Sources of restore proof remain the owning packages and their existing
|
||||
validators. The platform accepts those artifacts after an approved isolated
|
||||
exercise; it does not convert old Markdown or example JSON into fresh proof.
|
||||
No unattended jobs or Q2 transport are deployed under this decision.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
Invent numeric targets from elapsed restore time: rejected because it creates
|
||||
an untested promise. Keep all checks waiting for telemetry: rejected because
|
||||
safe local diagnostics and unsupported-guarantee disclosure are useful now.
|
||||
Treat every source declaration as live: rejected because desired placement
|
||||
and actual deployment can diverge.
|
||||
55
docs/adr/ADR-0005-demand-gated-stateful-services.md
Normal file
55
docs/adr/ADR-0005-demand-gated-stateful-services.md
Normal file
|
|
@ -0,0 +1,55 @@
|
|||
---
|
||||
id: RPLAT-ADR-0005
|
||||
title: "ADR-0005 — Reuse existing storage and defer unrequested stateful services"
|
||||
status: accepted
|
||||
revision: "1.0"
|
||||
owner: railiance-platform
|
||||
date: "2026-09-05"
|
||||
last_reviewed: "2026-09-05"
|
||||
review_interval: 6m
|
||||
---
|
||||
|
||||
# ADR-0005 — Demand-gated stateful services
|
||||
|
||||
## Context
|
||||
|
||||
INTENT includes cache, object storage and messaging. Valkey is gated in the
|
||||
Makefile. CNPG already consumes an external governed S3 backup destination.
|
||||
`artifact-store` has local/S3 backend code, retention/hold semantics and a
|
||||
MinIO/STS assessment; its older SCOPE understates the code. The reviewed
|
||||
platform queue has no accepted general cache or messaging consumer request.
|
||||
Absence from this queue is not a claim that no future consumer could need one.
|
||||
|
||||
## Decision
|
||||
|
||||
- Cache: defer shared Valkey deployment. Reopen on a named consumer specifying
|
||||
latency, eviction/durability, isolation, capacity and an operating owner.
|
||||
- General object storage: reuse the existing backup destination only for its
|
||||
approved backup scope. Evaluate artifact-store's existing S3 interface for
|
||||
artifact-preservation demand before proposing another gateway or engine.
|
||||
Defer a general platform S3/MinIO service until accepted demand specifies
|
||||
retention, residency, capacity, recovery, credential scope and package owner.
|
||||
- Messaging: defer shared broker deployment until a named consumer defines
|
||||
delivery/replay/order guarantees, retention, throughput and recovery needs.
|
||||
Do not replace application-specific queues by assumption.
|
||||
|
||||
Review on a new accepted consumer request, a change in existing provider
|
||||
capability, or 2027-03-05, whichever comes first. This is an S3 deployment
|
||||
posture decision; it assigns no new obligation to potential consumers and
|
||||
chooses no vendor or engine. The Q3 fleet-owner question stays with
|
||||
railiance-master; local recovery work continues independently.
|
||||
|
||||
## Consequences
|
||||
|
||||
Keep deploy gates. Any accepted new demand creates a bounded delivery plan
|
||||
covering service contract, custody, tenancy, capacity, backup/restore, telemetry,
|
||||
package owner, costs and acceptance before provisioning. Existing backup
|
||||
credentials are not general artifact credentials. Platform does not take over
|
||||
artifact metadata lifecycle or application publication logic.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
Deploy Valkey/MinIO/RabbitMQ to satisfy the nouns in INTENT: rejected because
|
||||
it adds unsupported operations without an accepted workload. Declare external
|
||||
backup S3 a general object service: rejected because its scope and consumer
|
||||
contract are narrower.
|
||||
242
docs/evidence/RPF-WP-0036-assurance-2026-09-05.json
Normal file
242
docs/evidence/RPF-WP-0036-assurance-2026-09-05.json
Normal file
|
|
@ -0,0 +1,242 @@
|
|||
{
|
||||
"schema": "railiance-platform.assurance-evidence.v1",
|
||||
"date": "2026-09-05",
|
||||
"observation": {
|
||||
"schema": "railiance-platform.observation.v1",
|
||||
"cluster_uid": "a553c742-0115-43d4-99a4-a5ca56fe0786",
|
||||
"captured_at": "2026-09-05T09:37:05.118118+00:00",
|
||||
"signals": {
|
||||
"apps-pg.ready": {
|
||||
"result": "pass",
|
||||
"observed_at": "2026-09-05T09:36:57.163337+00:00"
|
||||
},
|
||||
"apps-pg.backup": {
|
||||
"result": "pass",
|
||||
"observed_at": "2026-09-05T02:15:07Z"
|
||||
},
|
||||
"apps-pg.wal": {
|
||||
"result": "pass",
|
||||
"observed_at": "2026-09-05T09:36:57.163365+00:00"
|
||||
},
|
||||
"apps-pg.headroom": {
|
||||
"result": "pass",
|
||||
"observed_at": "2026-09-05T09:36:42Z"
|
||||
},
|
||||
"platform-pg.ready": {
|
||||
"result": "pass",
|
||||
"observed_at": "2026-09-05T09:36:59.396739+00:00"
|
||||
},
|
||||
"platform-pg.backup": {
|
||||
"result": "pass",
|
||||
"observed_at": "2026-09-05T02:15:11Z"
|
||||
},
|
||||
"platform-pg.wal": {
|
||||
"result": "pass",
|
||||
"observed_at": "2026-09-05T09:36:59.396763+00:00"
|
||||
},
|
||||
"platform-pg.headroom": {
|
||||
"result": "pass",
|
||||
"observed_at": "2026-09-05T09:36:44Z"
|
||||
},
|
||||
"platform-pg-2.ready": {
|
||||
"result": "pass",
|
||||
"observed_at": "2026-09-05T09:37:01.824557+00:00"
|
||||
},
|
||||
"platform-pg-2.backup": {
|
||||
"result": "pass",
|
||||
"observed_at": "2026-09-05T02:15:08Z"
|
||||
},
|
||||
"platform-pg-2.wal": {
|
||||
"result": "pass",
|
||||
"observed_at": "2026-09-05T09:37:01.824580+00:00"
|
||||
},
|
||||
"platform-pg-2.headroom": {
|
||||
"result": "pass",
|
||||
"observed_at": "2026-09-05T09:36:40Z"
|
||||
},
|
||||
"openbao.seal": {
|
||||
"result": "pass",
|
||||
"observed_at": "2026-09-05T09:37:04.442236+00:00"
|
||||
},
|
||||
"eso.ready": {
|
||||
"result": "fail",
|
||||
"observed_at": "2026-09-05T09:37:05.118098+00:00"
|
||||
},
|
||||
"eso.refresh": {
|
||||
"result": "pass",
|
||||
"observed_at": "2026-08-06T12:33:40Z"
|
||||
}
|
||||
}
|
||||
},
|
||||
"evaluation_at_capture": {
|
||||
"schema": "railiance-platform.assurance-signal.v1",
|
||||
"cluster_uid": "a553c742-0115-43d4-99a4-a5ca56fe0786",
|
||||
"evaluated_at": "2026-09-05T09:37:05.118118+00:00",
|
||||
"signals": {
|
||||
"apps-pg.ready": {
|
||||
"state": "healthy",
|
||||
"owner": "railiance-platform"
|
||||
},
|
||||
"apps-pg.backup": {
|
||||
"state": "healthy",
|
||||
"owner": "railiance-platform"
|
||||
},
|
||||
"apps-pg.wal": {
|
||||
"state": "healthy",
|
||||
"owner": "railiance-platform"
|
||||
},
|
||||
"apps-pg.restore": {
|
||||
"state": "missing",
|
||||
"owner": "railiance-platform"
|
||||
},
|
||||
"apps-pg.headroom": {
|
||||
"state": "healthy",
|
||||
"owner": "railiance-platform"
|
||||
},
|
||||
"platform-pg.ready": {
|
||||
"state": "healthy",
|
||||
"owner": "rapp-postgres"
|
||||
},
|
||||
"platform-pg.backup": {
|
||||
"state": "healthy",
|
||||
"owner": "rapp-postgres"
|
||||
},
|
||||
"platform-pg.wal": {
|
||||
"state": "healthy",
|
||||
"owner": "rapp-postgres"
|
||||
},
|
||||
"platform-pg.restore": {
|
||||
"state": "missing",
|
||||
"owner": "rapp-postgres"
|
||||
},
|
||||
"platform-pg.headroom": {
|
||||
"state": "healthy",
|
||||
"owner": "rapp-postgres"
|
||||
},
|
||||
"platform-pg-2.ready": {
|
||||
"state": "healthy",
|
||||
"owner": "rapp-postgres"
|
||||
},
|
||||
"platform-pg-2.backup": {
|
||||
"state": "healthy",
|
||||
"owner": "rapp-postgres"
|
||||
},
|
||||
"platform-pg-2.wal": {
|
||||
"state": "healthy",
|
||||
"owner": "rapp-postgres"
|
||||
},
|
||||
"platform-pg-2.restore": {
|
||||
"state": "missing",
|
||||
"owner": "rapp-postgres"
|
||||
},
|
||||
"platform-pg-2.headroom": {
|
||||
"state": "healthy",
|
||||
"owner": "rapp-postgres"
|
||||
},
|
||||
"openbao.seal": {
|
||||
"state": "healthy",
|
||||
"owner": "railiance-platform"
|
||||
},
|
||||
"openbao.snapshot": {
|
||||
"state": "missing",
|
||||
"owner": "railiance-platform"
|
||||
},
|
||||
"openbao.restore": {
|
||||
"state": "missing",
|
||||
"owner": "railiance-platform"
|
||||
},
|
||||
"offsite.upload": {
|
||||
"state": "missing",
|
||||
"owner": "railiance-platform"
|
||||
},
|
||||
"offsite.restore": {
|
||||
"state": "missing",
|
||||
"owner": "railiance-platform"
|
||||
},
|
||||
"eso.ready": {
|
||||
"state": "failed",
|
||||
"owner": "railiance-platform"
|
||||
},
|
||||
"eso.refresh": {
|
||||
"state": "stale",
|
||||
"owner": "railiance-platform"
|
||||
}
|
||||
},
|
||||
"transport": "unmonitored",
|
||||
"guarantees": "unsupported",
|
||||
"threshold_status": "local-diagnostic-only",
|
||||
"healthy": false
|
||||
},
|
||||
"live_admission": {
|
||||
"captured_at": "2026-09-05T09:28:42.234951+00:00",
|
||||
"cells": {
|
||||
"apps-pg": {
|
||||
"instances": 1,
|
||||
"memory_limit": "1Gi",
|
||||
"max_connections": "100",
|
||||
"retention": "30d"
|
||||
},
|
||||
"platform-pg": {
|
||||
"instances": 1,
|
||||
"memory_limit": "1Gi",
|
||||
"max_connections": "100",
|
||||
"retention": "30d"
|
||||
},
|
||||
"platform-pg-2": {
|
||||
"instances": 1,
|
||||
"memory_limit": "1Gi",
|
||||
"max_connections": "100",
|
||||
"retention": "30d"
|
||||
}
|
||||
}
|
||||
},
|
||||
"live_spec_matches_source": true,
|
||||
"eso_failures": [
|
||||
{
|
||||
"name": "forgejo-mailer",
|
||||
"namespace": "forgejo",
|
||||
"refresh": "2026-08-06T12:33:40Z",
|
||||
"conditions": [
|
||||
{
|
||||
"type": "Ready",
|
||||
"status": "False",
|
||||
"reason": "SecretSyncedError"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "reuse-surface-runtime",
|
||||
"namespace": "reuse",
|
||||
"refresh": "2026-08-06T20:25:39Z",
|
||||
"conditions": [
|
||||
{
|
||||
"type": "Ready",
|
||||
"status": "False",
|
||||
"reason": "SecretSyncedError"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "target-revenue-runtime",
|
||||
"namespace": "target-revenue",
|
||||
"refresh": "2026-09-04T14:52:08Z",
|
||||
"conditions": [
|
||||
{
|
||||
"type": "Ready",
|
||||
"status": "False",
|
||||
"reason": "SecretSyncedError"
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"validation": {
|
||||
"full_tests": 176,
|
||||
"focused_assurance_tests": 14,
|
||||
"admission_baseline": "pass"
|
||||
},
|
||||
"live_mutations": false,
|
||||
"secret_values_read": false,
|
||||
"scheduling_installed": false,
|
||||
"q2_delivery_proven": false,
|
||||
"restore_executed": false
|
||||
}
|
||||
|
|
@ -75,7 +75,9 @@ triggers. It is not the same role as the operator of the cluster.
|
|||
| Workload | Substrate | Provided level | Placement owner | Co-signer | §13 evidence |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| `audit-core` | `platform-pg` | P1 | `audit-core` | this repo | probes exist |
|
||||
| `tenant-engine` | SQLite PVC current; `platform-pg` desired | off-ladder current; P1 desired | `tenant-engine` | this repo | source conformance; live cutover absent |
|
||||
| `tenant-engine` | `platform-pg` | P1 | `tenant-engine` | this repo | owner cutover evidence 2026-08-21 |
|
||||
| `core-hub` | `platform-pg` | P1 | `core-hub` | this repo | capacity admission 2026-08-20; private shadow 2026-08-21 |
|
||||
| `sbom-nexus` | `platform-pg-2` | P1 | `sbom-nexus` | this repo | package boundary/restore evidence 2026-08-22 |
|
||||
| `user-engine` | `user-engine-pg` | P2 | `net-kingdom` | this repo | not assessed here |
|
||||
| `target-revenue` | `target-revenue-pg` | P2 | `target-revenue` | this repo | not assessed here |
|
||||
| `forgejo` | `forgejo-db` | P2 | this repo | — | single consumer |
|
||||
|
|
@ -144,13 +146,16 @@ one". "Provision a second cluster" is an answer; "revisit at the time" is not.
|
|||
|
||||
| Cluster | Ceiling declared | Overflow target | Standing |
|
||||
| --- | --- | --- | --- |
|
||||
| `platform-pg` | 4 declarations (estimated; ADR-0004) | `platform-pg-2`, named but absent | 3/4 including the isolation probe; measure before the next workload |
|
||||
| `platform-pg` | 4 declarations (package ADR-0004) | `platform-pg-2`, deployed | 4/4 including probe and Core Hub; fifth refused |
|
||||
| `platform-pg-2` | 4 declarations (same package validator) | No third cell approved | 1/4: sbom-nexus; owner admission and restore completed 2026-08-22 |
|
||||
| `apps-pg` | 3 consumers; 20 connections each, 1Gi memory binds first | `apps-pg-2` | 2/3; overflow must exist before a 4th |
|
||||
| `forgejo-db` | n/a — single consumer | n/a | — |
|
||||
|
||||
Both live shared clusters are one consumer away from the trigger. This repo
|
||||
owes `apps-pg`'s ceiling under its own rule, which is the correct asymmetry: a
|
||||
policy whose author is exempt is not a policy.
|
||||
Source and live metadata reviewed 2026-09-05: platform-pg is full; apps-pg
|
||||
remains 2/3 and apps-pg-2 is still unapplied. The generated
|
||||
`assurance/admission-baseline.json` binds these disclosures to the actual owner
|
||||
validators/declarations. `make assurance-admission` refuses source drift.
|
||||
Package acceptance, rather than this table, authorizes consumer provisioning.
|
||||
|
||||
## 5. Triggers, and who is watching them
|
||||
|
||||
|
|
@ -189,7 +194,8 @@ risk that nobody can see is strictly worse than one that is stated".
|
|||
|
||||
| Cluster | Co-resident classes | Mismatch |
|
||||
| --- | --- | --- |
|
||||
| `platform-pg` | `latency-critical` (`tenant-engine`) + `batch` (`audit-core`) | **yes — reported** |
|
||||
| `platform-pg` | `latency-critical` (`tenant-engine`, `core-hub`) + `batch` (`audit-core`) | **yes — reported** |
|
||||
| `platform-pg-2` | `interactive` (`sbom-nexus`) | no |
|
||||
| `apps-pg` | `interactive` + `interactive` | no |
|
||||
| `forgejo-db` | single consumer | n/a |
|
||||
|
||||
|
|
@ -256,3 +262,11 @@ onboarding enforce it.
|
|||
Six months, or on: a new shared-cluster consumer, a co-residency class change,
|
||||
`railiance-telemetry` going live, or the framework reaching `accepted`. Next
|
||||
review due **2027-02-17**.
|
||||
|
||||
## Service assurance implementation — 2026-09-05
|
||||
|
||||
`docs/service-assurance.md` supplies the bounded local producer and admission
|
||||
check. Capture proves database readiness/backup/controller status at a point in
|
||||
time, and currently exposes failed/stale ESO resources. It does not implement
|
||||
continuous capacity monitoring. Trigger 5 remains unmonitored until its native
|
||||
headroom adapter and Q2 recipient are accepted (RPF-WP-0036-T03/T04).
|
||||
|
|
|
|||
49
docs/platform-ownership-handoffs.md
Normal file
49
docs/platform-ownership-handoffs.md
Normal file
|
|
@ -0,0 +1,49 @@
|
|||
# Platform compatibility handoffs — 2026-09-05
|
||||
|
||||
RPF-WP-0036-T06 inventory: `assurance/ownership-handoffs.json` contains exact
|
||||
paths/hashes, accepting owner candidates, callers, review dates and acceptance
|
||||
criteria. S3 custody/CCR policies and shared backup mechanisms stay here.
|
||||
|
||||
Retain each operational entry point until the proposed owner accepts its
|
||||
canonical replacement, callers pass dry-run/smoke checks, and rollback is
|
||||
recorded. Review by 2026-10-05 or on owner acceptance. No files were removed,
|
||||
no ownership acceptance was invented, and no coordination messages were sent.
|
||||
This is the dated platform retention decision; T06 remains waiting for accepted
|
||||
handoffs and the separate derived-state repair.
|
||||
|
||||
The forge contract already distinguishes forge requirements/verification from
|
||||
shared backup mechanisms. Split those concerns; do not move all backup code
|
||||
just because its filename mentions Forgejo. The platform OpenBao package
|
||||
handoff is likewise already documented: preserve custody/governance here and
|
||||
only adopt the package owner's authoritative replacement for its wrapper.
|
||||
ArgoCD files need artifact-level ownership: runtime bootstrap to S2, generic
|
||||
paved paths to S4, app releases to S5. ESO credential stores retain S3 custody
|
||||
review even when another repo packages their deployment.
|
||||
|
||||
## Prepared owner requests
|
||||
|
||||
These are reviewable request content, not sent messages:
|
||||
|
||||
- Forge/package/app owners: accept or amend the exact source inventory, name
|
||||
canonical entry points and callers, and return revision-pinned dry-run,
|
||||
smoke and rollback evidence before retirement.
|
||||
- Repo-manager/State Hub: reconcile only the legacy UUID-to-canonical map in
|
||||
the assessment's derived-state caveat; keep file-backed identities intact.
|
||||
The installed brief generator reads open Hub workplans without excluding
|
||||
legacy retired aliases, so rerunning it alone would reproduce duplicates.
|
||||
- Railiance-telemetry: supply versioned receiving schema, transport and
|
||||
retention contract, intended recipient, and a controlled stale-event/absence
|
||||
acceptance path for the local S3 producer. It is currently unmonitored.
|
||||
- Railiance-master: determine the accountable fleet Q3 recovery owner and
|
||||
boundary. S3 continues its own service recovery work meanwhile.
|
||||
|
||||
## Live ESO findings requiring scope confirmation
|
||||
|
||||
The 2026-09-05 metadata capture found `SecretSyncedError` for
|
||||
`forgejo/forgejo-mailer`, `reuse/reuse-surface-runtime`, and
|
||||
`target-revenue/target-revenue-runtime`. Their last refreshes were August 6,
|
||||
August 6 and September 4 respectively. Package/consumer and platform custody
|
||||
owners must first classify current versus obsolete resources, then review
|
||||
any exact lane repair or retirement. No Secret values were read and no failed
|
||||
resource was simply excluded to make the aggregate pass. These results do not
|
||||
by themselves identify a broken provider credential or authorize rotation.
|
||||
|
|
@ -1,8 +1,8 @@
|
|||
# S3 consumer service interfaces
|
||||
|
||||
Work record: `RAILIANCE-WP-0016` item 17; `RPF-WP-0018` T04
|
||||
Date: 2026-08-15, revised 2026-08-17
|
||||
Contract version of this index: `1.1.0`
|
||||
Date: 2026-08-15, revised 2026-09-05
|
||||
Contract version of this index: `1.2.0`
|
||||
|
||||
Consumers bind to these contracts, not to Helm chart names or pod topology.
|
||||
Changing an engine behind a versioned interface does not bump the interface
|
||||
|
|
@ -101,7 +101,7 @@ Consumer is a `consumers[].declaration`. Boundary unit is
|
|||
| Declared ceiling / overflow target | **4 declarations / `platform-pg-2`** | The isolation probe counts. `rapp-postgres` rejects a fifth declaration on one cell; ADR-0004 owns the number. |
|
||||
|
||||
Service classes currently co-resident on `platform-pg`: `latency-critical`
|
||||
(`tenant-engine`) and `batch` (`audit-core`). Nothing prioritises between them
|
||||
(`tenant-engine`, `core-hub`) and `batch` (`audit-core`). Nothing prioritises between them
|
||||
— community PostgreSQL has no resource governor. Reported per Tenancy Posture
|
||||
§8.3.3.
|
||||
|
||||
|
|
@ -115,3 +115,20 @@ the lane without restating TTL, delivery, or revocation rules.
|
|||
|
||||
Valkey and in-cluster MinIO have capability blocks in `SCOPE.md` and no
|
||||
live service. They do not receive a `v1` until a consumer exists.
|
||||
|
||||
## Service assurance disclosure (1.2.0, additive)
|
||||
|
||||
`assurance/service-records.json` explicitly discloses unsupported availability,
|
||||
RPO and RTO guarantees, decision owners and existing recovery evidence;
|
||||
`docs/service-assurance.md` documents the executable local checks. Diagnostic
|
||||
freshness budgets are not SLAs or approval to deploy schedules. Consumers that
|
||||
require stronger guarantees need a placement/recovery decision.
|
||||
|
||||
Current source occupancy: platform-pg 4/4 including the isolation probe,
|
||||
platform-pg-2 1/4 for sbom-nexus, apps-pg 2/3, apps-pg-2 0/3 and unapplied.
|
||||
All three deployed cells declare one instance, 1Gi memory limit, 100 aggregate
|
||||
connections and 30-day retention; live metadata matched on 2026-09-05. The
|
||||
second platform cell's approved restore took 51 seconds on 2026-08-22; this
|
||||
is dated evidence, not an RTO. Tenant-engine cut over from SQLite on 2026-08-21.
|
||||
`assurance/admission-baseline.json` carries owner-native declarations, hashes
|
||||
and limits. Do not refresh that baseline without reviewing changes.
|
||||
|
|
|
|||
105
docs/service-assurance.md
Normal file
105
docs/service-assurance.md
Normal file
|
|
@ -0,0 +1,105 @@
|
|||
# S3 service assurance — v1
|
||||
|
||||
Workplan: RPF-WP-0036. Disclosure authority:
|
||||
[ADR-0004](adr/ADR-0004-service-assurance-disclosure.md).
|
||||
|
||||
## Operator entry points
|
||||
|
||||
```bash
|
||||
make assurance-admission
|
||||
make assurance-capture
|
||||
make assurance-check
|
||||
```
|
||||
|
||||
Admission requires the adjacent reviewed `rapp-postgres` checkout and PyYAML,
|
||||
which existing platform tooling already uses. Capture requires the existing
|
||||
noninteractive SSH route to `railiance01`, with kubectl on that host. It pins
|
||||
the kube-system UID from existing platform evidence and stops on mismatch.
|
||||
It does not trust the default workstation kube context. Failure prints only a
|
||||
fixed error code, never raw command output. No Secret, application row/log or credential query is run. A read-only
|
||||
pg_stat_activity count and kubectl metrics supply aggregate headroom;
|
||||
unauthenticated `bao status` supplies seal state. The observation is written to
|
||||
`/tmp/railiance-platform-observation.json` by default; override with
|
||||
`ASSURANCE_OBSERVATION`. Run the check only after capture succeeds.
|
||||
|
||||
`evaluate` exits 0 when every required sample is healthy, 1 for incomplete,
|
||||
stale, failed or unavailable samples, and 2 for invalid/unavailable input.
|
||||
There is intentionally no green overall service claim while evidence is
|
||||
missing. To replay a dated observation:
|
||||
|
||||
```bash
|
||||
python3 scripts/service_assurance.py evaluate observation.json --now 2026-09-05T12:00:00Z
|
||||
```
|
||||
|
||||
`--now` is for replay/testing, not a way to freshen old evidence. The output
|
||||
retains evaluation time, unsupported guarantees and unmonitored transport.
|
||||
A `pass` sample is an attestation from its producer, not cryptographic proof.
|
||||
Only approved owner receipts can close the workplan's live acceptance gates.
|
||||
|
||||
## Supported source contract and missing adapters
|
||||
|
||||
The closed observation envelope carries schema, pinned cluster UID, capture
|
||||
time and a configured set of signals. Each sample has only `observed_at` and
|
||||
`result` (`pass`, `fail`, `unavailable`). Free text, values, paths and unknown
|
||||
signals are rejected. For backup/restore/upload signals the observed time is
|
||||
the completion time of that action; it is not the time someone read the file.
|
||||
A capture older than 15 minutes never passes even when a backup is recent.
|
||||
|
||||
The collector supplies database readiness, last completed backup and continuous
|
||||
archiving for apps-pg/platform-pg/platform-pg-2, plus fleet ESO Ready/oldest-refresh
|
||||
status, OpenBao seal state, and aggregate connection/memory headroom. Headroom
|
||||
uses a local diagnostic ceiling of 80% of source connection/memory limits; it
|
||||
is not a workload latency guarantee. Metrics timestamps retain their age. ContinuousArchiving is a controller condition, not a measured WAL
|
||||
lag guarantee. The fleet ESO aggregate is deliberately a conservative inventory
|
||||
check; an obsolete resource must be explicitly classified by its owner before
|
||||
it is excluded. One ESO failure cannot disappear inside an average.
|
||||
|
||||
The following remain missing until a native value-safe adapter and acceptance
|
||||
exist: validated isolated restore receipts,
|
||||
OpenBao snapshot/restore proof, and offsite upload/restore receipts. Missing
|
||||
adapters are not inferred healthy from pod readiness. The local producer is
|
||||
not the Q2 standard; railiance-telemetry has no implemented receiving contract
|
||||
in the reviewed checkout. Integration, routing and scheduled delivery remain
|
||||
T04, and no notification was sent during implementation.
|
||||
|
||||
## Service records and evidence inventory
|
||||
|
||||
`assurance/service-records.json` covers all three database cells, OpenBao,
|
||||
Forgejo backup and multi-host CNPG logical offsite delivery. It distinguishes
|
||||
unsupported objectives, declared retention, recovery custody and decision
|
||||
owners. The old Forgejo cron example mentions an RPO 24h target; that is not a
|
||||
newly accepted or verified current service guarantee. RPF-WP-0029 still gates
|
||||
provider invalidation/replacement recovery for the shared offsite lane.
|
||||
|
||||
| Evidence | Existing authority | Current acceptance limit |
|
||||
| --- | --- | --- |
|
||||
| CNPG completed backup/WAL | Cluster controller status; live capture | Current status only; does not exercise restore |
|
||||
| apps-pg restore | RPF-WP-0019 evidence, 2026-08-20 | Dated 56-second scratch restore; no numeric RTO commitment |
|
||||
| platform-pg restore/PITR | rapp-postgres restore procedure and evidence | Recurring current proof and independent recovery access remain T03 |
|
||||
| platform-pg-2 restore | RAPP-POSTGRES-WP-0005-T04, 2026-08-22 | 51-second isolated restore after SBOM migration; not a continuous guarantee |
|
||||
| OpenBao snapshot | WARDEN-WP-0027 preparation receipt, 2026-08-23 | Snapshot/encrypted off-host preparation, not isolated restore |
|
||||
| OpenBao restore | Existing `openbao-validate-restore-evidence.sh` and package procedure | Example receipt cannot pass as a fresh execution |
|
||||
| Recovery exercise | RPF-WP-0015-T02/T03 | Separate windows, synthetic driver/quorum and abort operator required |
|
||||
| Logical/Forgejo offsite | activity-core backup definitions, existing helper/runbooks | No new upload/restore performed; RPF-WP-0029 remains open |
|
||||
|
||||
## Admission and disclosure drift
|
||||
|
||||
`assurance/admission-baseline.json` is generated by the `admission` command from
|
||||
local CNPG CRs, `placement-owners.json`, and the native rapp-postgres renderer.
|
||||
The checker reuses both native capacity validators. It also rejects missing
|
||||
owners, changed cell placement, unhonoured inherited retention and mismatch
|
||||
between effective package retention and the actual CR. Hashes pin every
|
||||
consumed source artifact; changing package code or declarations requires review
|
||||
before refreshing the baseline. Do not blindly regenerate it in CI.
|
||||
|
||||
Source occupancy: apps-pg 2/3, apps-pg-2 0/3 (not deployed), platform-pg 4/4
|
||||
including the probe, platform-pg-2 1/4 for sbom-nexus. Tenant-engine's PostgreSQL
|
||||
cutover completed 2026-08-21; its old SQLite-current claim was stale. Platform-pg-2
|
||||
was accepted and restored on 2026-08-22; it is no longer merely named overflow.
|
||||
Core Hub is latency-critical alongside audit-core batch work; no service-class
|
||||
priority is implied. See the generated baseline for per-cell details.
|
||||
|
||||
Source admission checks happen before deployment in the existing apps-pg and
|
||||
package owner paths; this additional check detects disclosure drift. It does
|
||||
not apply resources or become a second provisioning engine. Any new consumer
|
||||
needs an owner entry and a reviewed baseline update, even within free capacity.
|
||||
79
history/2026-09-05-platform-assurance-implementation.md
Normal file
79
history/2026-09-05-platform-assurance-implementation.md
Normal file
|
|
@ -0,0 +1,79 @@
|
|||
# RPF-WP-0036 implementation — 2026-09-05
|
||||
|
||||
Implemented the ready plan's repository work and performed bounded live status
|
||||
inspection using the existing SSH route to the pinned railiance01 cluster.
|
||||
No credentials were fetched, no backup/restore or infrastructure mutation ran,
|
||||
and no scheduler, external notification or owner acceptance was created.
|
||||
|
||||
## Delivered
|
||||
|
||||
- ADR-0004 and service records disclose unsupported availability/RPO/RTO
|
||||
commitments, existing evidence, retention and accountable decision owners.
|
||||
Local freshness/headroom thresholds are diagnostic, not SLAs.
|
||||
- An admission checker reuses the apps-pg and rapp-postgres validators, verifies
|
||||
effective retention and placement ownership, and refuses a changed source
|
||||
hash/disclosure baseline. Five-consumer and mismatched-retention fixtures fail.
|
||||
- A cluster-pinned collector supplies selected CNPG status, aggregate connection
|
||||
counts, metrics and unauthenticated OpenBao seal status. It reads no Secret,
|
||||
application data, session SQL text or logs. SSH argv are quoted; failures
|
||||
suppress raw output; successful observation files publish atomically at 0600.
|
||||
- A closed observation evaluator distinguishes healthy, stale, missing, failed
|
||||
and unavailable samples, rejects wrong clusters/future/unknown payloads, and
|
||||
makes missing emission detectable through capture expiry. Transport remains
|
||||
explicitly unmonitored pending Q2 acceptance.
|
||||
- Corrected placement/interface disclosures: tenant-engine's PostgreSQL cutover
|
||||
completed; platform-pg is 4/4 and the deployed platform-pg-2 holds sbom-nexus.
|
||||
- Exact compatibility path/hash/caller inventories and a dated retention
|
||||
decision are ready for owner review. No functioning entry point was removed.
|
||||
- ADR-0005 defers unrequested cache/general storage/broker deployments with
|
||||
explicit review triggers and prefers existing storage interfaces for demand.
|
||||
|
||||
## Live findings
|
||||
|
||||
The default workstation kube context and the misleading config-railiance01
|
||||
file reached a different host; config-hosteurope depended on a down local
|
||||
forwarder. The collector therefore uses existing SSH to railiance01 and checks
|
||||
kube-system UID `a553c742-0115-43d4-99a4-a5ca56fe0786` before service reads.
|
||||
No context was rewritten or tunnel deployed.
|
||||
|
||||
All three deployed database cells matched one instance, 1Gi memory limit,
|
||||
100 aggregate connections and 30-day retention. Each was Ready, had a completed
|
||||
backup from 02:15 UTC that morning and reported continuous archiving. Aggregate
|
||||
connection/memory occupancy passed the local 80% diagnostic threshold. OpenBao
|
||||
was unsealed. These are point-in-time observations, not new guarantees.
|
||||
|
||||
The checker also exposed three failed ESO resources (`SecretSyncedError`):
|
||||
forgejo/forgejo-mailer, reuse/reuse-surface-runtime and
|
||||
target-revenue/target-revenue-runtime. The earliest successful refresh was
|
||||
2026-08-06. Their active/obsolete status and exact cause need consumer/platform
|
||||
review before a lane repair or retirement; no credential diagnosis is inferred
|
||||
from this condition alone.
|
||||
|
||||
Evidence: `docs/evidence/RPF-WP-0036-assurance-2026-09-05.json` includes capture,
|
||||
evaluation time, live limits and sanitized failing-resource metadata. It does
|
||||
not manufacture isolated-restore receipts from historical prose. Missing
|
||||
restore/snapshot/offsite inputs and failed/stale ESO keep overall assurance
|
||||
non-green, which is the expected correct result.
|
||||
|
||||
## Validation
|
||||
|
||||
Full repository unittest discovery: 176 passed. Focused assurance tests after
|
||||
collector output hardening: 14 passed. Admission baseline check passed; fresh
|
||||
live capture completed and its specification limits matched the source.
|
||||
Cases cover wrong clusters, future/naive times, stale/replayed captures, missing
|
||||
and failed signals, unexpected payloads/value-safe errors, unavailable native
|
||||
reads, memory units, missing ownership/overflow, fifth consumer admission,
|
||||
and unhonoured retention. This is not a restore or alert-delivery test.
|
||||
|
||||
## Remaining acceptance
|
||||
|
||||
T02/T05/T07 are done. T03 waits on fresh owner-validated recovery/offsite
|
||||
receipts, independent recovery access and approved recurring execution.
|
||||
T04 waits on the Q2 receiving contract and controlled failure/absence delivery;
|
||||
ESO findings remain visible for exact owner disposition. T06 waits on accepted
|
||||
handoffs and scoped alias cleanup. The installed brief generator still reads
|
||||
legacy open aliases; generating the brief alone would reproduce those rows.
|
||||
|
||||
The prepared owner request content is in `docs/platform-ownership-handoffs.md`.
|
||||
Its transmission is not claimed. The plan remains blocked on these explicit
|
||||
acceptance gates; source implementation is not substituted for owner/live proof.
|
||||
126
scripts/capture_service_observation.py
Normal file
126
scripts/capture_service_observation.py
Normal file
|
|
@ -0,0 +1,126 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Collect only selected status fields over the existing railiance01 SSH lane."""
|
||||
import argparse
|
||||
from datetime import datetime, timezone
|
||||
import json
|
||||
import re
|
||||
import os
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
import shlex
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0, str(ROOT / 'scripts'))
|
||||
from service_assurance import read_contract, admission
|
||||
|
||||
|
||||
def query(args, allowed_codes=(0,)):
|
||||
result = subprocess.run(['ssh', '-o', 'BatchMode=yes', '-o', 'ConnectTimeout=10',
|
||||
'railiance01', shlex.join(['kubectl', '--request-timeout=10s', *args])],
|
||||
capture_output=True, text=True, timeout=25)
|
||||
if result.returncode not in allowed_codes:
|
||||
raise ValueError('query unavailable')
|
||||
return json.loads(result.stdout)
|
||||
|
||||
|
||||
def memory_bytes(value):
|
||||
match = re.fullmatch(r"([0-9]+(?:\.[0-9]+)?)(Ki|Mi|Gi|Ti|K|M|G|T)?", value)
|
||||
if not match:
|
||||
raise ValueError('invalid memory quantity')
|
||||
unit = match.group(2) or ''
|
||||
multipliers = {'': 1, 'Ki': 1024, 'Mi': 1024**2, 'Gi': 1024**3,
|
||||
'Ti': 1024**4, 'K': 1000, 'M': 1000**2, 'G': 1000**3, 'T': 1000**4}
|
||||
return float(match.group(1)) * multipliers[unit]
|
||||
|
||||
|
||||
def capture():
|
||||
contract = read_contract(ROOT / 'assurance/service-contract.json')
|
||||
uid = query(['get', 'namespace', 'kube-system', '-o', 'go-template={{printf "%q" .metadata.uid}}'])
|
||||
if uid != contract['cluster_uid']:
|
||||
raise ValueError('wrong cluster')
|
||||
signals = {}
|
||||
def add(name, result, when=None):
|
||||
signals[name] = {'result': result, 'observed_at': when or datetime.now(timezone.utc).isoformat()}
|
||||
baseline = json.loads((ROOT / 'assurance/admission-baseline.json').read_text())
|
||||
if admission() != baseline:
|
||||
raise ValueError('admission baseline drift')
|
||||
for cell in ('apps-pg', 'platform-pg', 'platform-pg-2'):
|
||||
names = [cell + '.' + suffix for suffix in ('ready', 'backup', 'wal')]
|
||||
try:
|
||||
d = query(['get', 'cluster', cell, '-n', 'databases', '-o',
|
||||
'go-template={"ready":{{.status.readyInstances}},"instances":{{.spec.instances}},"lastBackup":{{printf "%q" .status.lastSuccessfulBackup}},"conditions":[{{range $i,$v := .status.conditions}}{{if $i}},{{end}}{"type":{{printf "%q" $v.type}},"status":{{printf "%q" $v.status}}}{{end}}]}'])
|
||||
add(names[0], 'pass' if d['ready'] == d['instances'] and d['ready'] > 0 else 'fail')
|
||||
if d['lastBackup']:
|
||||
add(names[1], 'pass', d['lastBackup'])
|
||||
add(names[2], 'pass' if any(c == {'type': 'ContinuousArchiving', 'status': 'True'} for c in d['conditions']) else 'fail')
|
||||
except (ValueError, KeyError, TypeError, subprocess.TimeoutExpired):
|
||||
for name in names:
|
||||
add(name, 'unavailable')
|
||||
try:
|
||||
metrics = query(['get', '--raw', '/apis/metrics.k8s.io/v1beta1/namespaces/databases/pods/' + cell + '-1'])
|
||||
if not metrics['containers']:
|
||||
raise ValueError('empty metrics')
|
||||
memory = sum(memory_bytes(c['usage']['memory']) for c in metrics['containers'])
|
||||
# Same read-only pg_stat_activity aggregate as the package ops surface,
|
||||
# narrowed to a count: no SQL text, usernames, database rows or values.
|
||||
count = query(['exec', '-n', 'databases', cell + '-1', '-c', 'postgres', '--',
|
||||
'psql', '-U', 'postgres', '-d', 'postgres', '-Atqc',
|
||||
'SELECT count(*) FROM pg_stat_activity'])
|
||||
limits = baseline['cells'][cell]
|
||||
add(cell + '.headroom', 'pass' if type(count) is int and count >= 0
|
||||
and count <= limits['max_connections'] * 0.8
|
||||
and memory <= memory_bytes(limits['memory_limit']) * 0.8 else 'fail',
|
||||
metrics['timestamp'])
|
||||
except (OSError, ValueError, KeyError, TypeError, subprocess.TimeoutExpired):
|
||||
add(cell + '.headroom', 'unavailable')
|
||||
try:
|
||||
status = query(['exec', '-n', 'openbao', 'openbao-0', '--',
|
||||
'bao', 'status', '-format=json'], allowed_codes=(0, 2))
|
||||
add('openbao.seal', 'pass' if status.get('sealed') is False else 'fail')
|
||||
except (OSError, ValueError, KeyError, TypeError, subprocess.TimeoutExpired):
|
||||
add('openbao.seal', 'unavailable')
|
||||
try:
|
||||
rows = query(['get', 'externalsecrets', '-A', '-o',
|
||||
'go-template=[{{range $i,$v := .items}}{{if $i}},{{end}}{"refresh":{{printf "%q" $v.status.refreshTime}},"conditions":[{{range $j,$c := $v.status.conditions}}{{if $j}},{{end}}{"type":{{printf "%q" $c.type}},"status":{{printf "%q" $c.status}}}{{end}}]}{{end}}]'])
|
||||
ready = bool(rows) and all(any(c == {'type': 'Ready', 'status': 'True'} for c in row['conditions']) for row in rows)
|
||||
# Empty/missing refresh is a failure, never an apparently healthy zero age.
|
||||
refresh = min((r['refresh'] for r in rows), default='')
|
||||
add('eso.ready', 'pass' if ready else 'fail')
|
||||
if refresh:
|
||||
add('eso.refresh', 'pass', refresh)
|
||||
except (ValueError, KeyError, TypeError, subprocess.TimeoutExpired):
|
||||
add('eso.ready', 'unavailable')
|
||||
add('eso.refresh', 'unavailable')
|
||||
# No token, Secret, application data/logs or seal/unseal mutation.
|
||||
# Native restore and offsite receipts remain separate attended evidence.
|
||||
return {'schema': 'railiance-platform.observation.v1', 'cluster_uid': uid,
|
||||
'captured_at': datetime.now(timezone.utc).isoformat(), 'signals': signals}
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument('--output', type=Path)
|
||||
args = parser.parse_args()
|
||||
try:
|
||||
data = json.dumps(capture(), indent=2) + '\n'
|
||||
if args.output:
|
||||
temporary = None
|
||||
try:
|
||||
with tempfile.NamedTemporaryFile(mode='w', dir=args.output.parent, delete=False) as f:
|
||||
temporary = f.name
|
||||
f.write(data)
|
||||
f.flush()
|
||||
os.fsync(f.fileno())
|
||||
os.replace(temporary, args.output)
|
||||
temporary = None
|
||||
finally:
|
||||
if temporary:
|
||||
os.unlink(temporary)
|
||||
print(json.dumps({'captured': True}))
|
||||
else:
|
||||
print(data, end='')
|
||||
except (OSError, ValueError, KeyError, TypeError, subprocess.TimeoutExpired):
|
||||
print(json.dumps({'schema': 'railiance-platform.assurance-error.v1', 'error': 'capture-unavailable-or-wrong-cluster'}))
|
||||
sys.exit(2)
|
||||
183
scripts/service_assurance.py
Normal file
183
scripts/service_assurance.py
Normal file
|
|
@ -0,0 +1,183 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Read-only S3 admission and bounded evidence evaluation. Never fetch Secrets."""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
from datetime import datetime, timezone
|
||||
import hashlib
|
||||
import importlib.util
|
||||
import json
|
||||
import math
|
||||
from pathlib import Path
|
||||
import sys
|
||||
|
||||
import yaml
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
|
||||
|
||||
def load_module(name, path):
|
||||
spec = importlib.util.spec_from_file_location(name, path)
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
def timestamp(value):
|
||||
if not isinstance(value, str):
|
||||
raise ValueError('invalid timestamp')
|
||||
result = datetime.fromisoformat(value.replace('Z', '+00:00'))
|
||||
if result.tzinfo is None:
|
||||
raise ValueError('timezone required')
|
||||
return result.astimezone(timezone.utc)
|
||||
|
||||
|
||||
def digest(path):
|
||||
return hashlib.sha256(path.read_bytes()).hexdigest()
|
||||
|
||||
|
||||
def admission(root=ROOT, package=None):
|
||||
package = package or root.parent / 'rapp-postgres'
|
||||
local = load_module('apps_capacity', root / 'tools/verify_apps_pg_capacity.py')
|
||||
native = load_module('postgres_consumers', package / 'scripts/render_consumers.py')
|
||||
manifests = [root / 'helm' / (name + '-cluster.yaml') for name in local.REVIEWED_CELLS]
|
||||
local.verify(manifests)
|
||||
paths = sorted((package / 'consumers').glob('*.yaml'))
|
||||
if not paths:
|
||||
raise ValueError('package consumers missing')
|
||||
consumers = [native.load(path) for path in paths]
|
||||
native.validate_cluster_capacity(consumers)
|
||||
registry = json.loads((root / 'assurance/placement-owners.json').read_text())
|
||||
cells = {}
|
||||
source_files = manifests + paths + [root / 'tools/verify_apps_pg_capacity.py',
|
||||
root / 'assurance/placement-owners.json', package / 'scripts/render_consumers.py']
|
||||
for manifest in manifests:
|
||||
doc = yaml.safe_load(manifest.read_text())
|
||||
name, spec = doc['metadata']['name'], doc['spec']
|
||||
roles = spec.get('managed', {}).get('roles', [])
|
||||
names = sorted(r['name'] for r in roles)
|
||||
cells[name] = {'consumers': names, 'ceiling': local.MAX_CONSUMERS,
|
||||
'retention_days': int(spec['backup']['retentionPolicy'].removesuffix('d')),
|
||||
'instances': spec['instances'], 'memory_limit': spec['resources']['limits']['memory'],
|
||||
'max_connections': int(spec['postgresql']['parameters']['max_connections']),
|
||||
'service_classes': {n: registry[n]['service_class'] for n in names},
|
||||
'connection_limits': {r['name']: r['connectionLimit'] for r in roles}}
|
||||
if cells[name]['retention_days'] != 30:
|
||||
raise ValueError('apps-pg retention differs from disclosed 30-day contract')
|
||||
for name in sorted(native.ALLOWED_CLUSTERS):
|
||||
manifest = package / 'helm' / (name + '-cluster.yaml')
|
||||
spec = yaml.safe_load(manifest.read_text())['spec']
|
||||
source_files.append(manifest)
|
||||
selected = [d for d in consumers if native.cluster_name(d) == name]
|
||||
horizon = native.erasure_horizon(selected)
|
||||
retention = int(spec['backup']['retentionPolicy'].removesuffix('d'))
|
||||
if retention != horizon['instanceRetentionDays']:
|
||||
raise ValueError('package retention differs from effective consumer horizon')
|
||||
if any(not d['honoured'] for d in horizon['consumers'].values()):
|
||||
raise ValueError('consumer retention request cannot be honoured on shared cell')
|
||||
cells[name] = {'consumers': sorted(horizon['consumers']),
|
||||
'ceiling': native.MAX_CONSUMERS_PER_CLUSTER, 'retention_days': retention,
|
||||
'instances': spec['instances'], 'memory_limit': spec['resources']['limits']['memory'],
|
||||
'max_connections': int(spec['postgresql']['parameters']['max_connections']),
|
||||
'service_classes': horizon['serviceClasses']}
|
||||
names = {n for c in cells.values() for n in c['consumers']}
|
||||
if names != set(registry) or any(not r.get('owner') or not r.get('evidence') for r in registry.values()):
|
||||
raise ValueError('missing or stale placement owner')
|
||||
for name, cell in cells.items():
|
||||
for consumer in cell['consumers']:
|
||||
if registry[consumer]['cell'] != name:
|
||||
raise ValueError('placement owner cell mismatch')
|
||||
cell['owners'] = {n: registry[n]['owner'] for n in cell['consumers']}
|
||||
return {'schema': 'railiance-platform.admission.v1', 'basis': 'source-declarations',
|
||||
'cells': cells, 'sources': {str(p.relative_to(root.parent)): digest(p) for p in source_files}}
|
||||
|
||||
|
||||
def evaluate(contract, observation, now):
|
||||
"""Classify a value-safe observation, not certify its underlying truth."""
|
||||
if set(observation) != {'schema', 'cluster_uid', 'captured_at', 'signals'}:
|
||||
raise ValueError('invalid observation envelope')
|
||||
if observation['schema'] != 'railiance-platform.observation.v1':
|
||||
raise ValueError('unsupported observation version')
|
||||
if observation['cluster_uid'] != contract['cluster_uid']:
|
||||
raise ValueError('wrong cluster')
|
||||
capture = timestamp(observation['captured_at'])
|
||||
capture_age = (now - capture).total_seconds()
|
||||
if capture_age < 0:
|
||||
raise ValueError('future capture')
|
||||
specs = contract['signals']
|
||||
if not isinstance(observation['signals'], dict) or set(observation['signals']) - set(specs):
|
||||
raise ValueError('unknown signals')
|
||||
results = {}
|
||||
for name, spec in specs.items():
|
||||
sample = observation['signals'].get(name)
|
||||
state = 'missing'
|
||||
if sample is not None:
|
||||
if not isinstance(sample, dict) or set(sample) != {'observed_at', 'result'}:
|
||||
raise ValueError('invalid signal fields')
|
||||
if sample['result'] not in ('pass', 'fail', 'unavailable'):
|
||||
raise ValueError('invalid result')
|
||||
observed = timestamp(sample['observed_at'])
|
||||
age = (now - observed).total_seconds()
|
||||
if observed > capture:
|
||||
raise ValueError('signal newer than capture')
|
||||
if sample['result'] == 'unavailable':
|
||||
state = 'unavailable'
|
||||
elif sample['result'] == 'fail':
|
||||
state = 'failed'
|
||||
elif capture_age > contract['capture_max_age_seconds']:
|
||||
state = 'stale'
|
||||
elif age > spec['max_age_seconds']:
|
||||
state = 'stale'
|
||||
else:
|
||||
state = 'healthy'
|
||||
results[name] = {'state': state, 'owner': spec['owner']}
|
||||
return {'schema': 'railiance-platform.assurance-signal.v1',
|
||||
'cluster_uid': contract['cluster_uid'], 'evaluated_at': now.isoformat(),
|
||||
'signals': results, 'transport': 'unmonitored',
|
||||
'guarantees': 'unsupported', 'threshold_status': 'local-diagnostic-only',
|
||||
'healthy': all(r['state'] == 'healthy' for r in results.values())}
|
||||
|
||||
|
||||
def read_contract(path):
|
||||
doc = json.loads(path.read_text())
|
||||
if doc['schema'] != 'railiance-platform.assurance-contract.v1' or not doc['signals']:
|
||||
raise ValueError('invalid contract')
|
||||
for budget in [doc['capture_max_age_seconds'], *(s['max_age_seconds'] for s in doc['signals'].values())]:
|
||||
if type(budget) not in (int, float) or not math.isfinite(budget) or budget <= 0:
|
||||
raise ValueError('invalid freshness budget')
|
||||
if any(not s.get('owner') for s in doc['signals'].values()):
|
||||
raise ValueError('signal owner missing')
|
||||
return doc
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
sub = parser.add_subparsers(dest='command', required=True)
|
||||
a = sub.add_parser('admission')
|
||||
a.add_argument('--package', type=Path, default=ROOT.parent / 'rapp-postgres')
|
||||
a.add_argument('--check', type=Path)
|
||||
e = sub.add_parser('evaluate')
|
||||
e.add_argument('observation', type=Path)
|
||||
e.add_argument('--contract', type=Path, default=ROOT / 'assurance/service-contract.json')
|
||||
e.add_argument('--now', help='Explicit UTC time for deterministic replay')
|
||||
args = parser.parse_args(argv)
|
||||
try:
|
||||
if args.command == 'admission':
|
||||
report = admission(package=args.package)
|
||||
if args.check and report != json.loads(args.check.read_text()):
|
||||
raise ValueError('admission source or disclosure drift; review before refreshing')
|
||||
code = 0
|
||||
else:
|
||||
now = timestamp(args.now) if args.now else datetime.now(timezone.utc)
|
||||
report = evaluate(read_contract(args.contract), json.loads(args.observation.read_text()), now)
|
||||
code = 0 if report['healthy'] else 1
|
||||
except (OSError, ValueError, KeyError, TypeError, AttributeError, yaml.YAMLError):
|
||||
# Do not print parser errors, raw input, values, source paths or subprocess stderr.
|
||||
print(json.dumps({'schema': 'railiance-platform.assurance-error.v1', 'error': 'invalid-or-unavailable-input'}))
|
||||
return 2
|
||||
print(json.dumps(report, indent=2, allow_nan=False))
|
||||
return code
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
sys.exit(main())
|
||||
162
tests/test_service_assurance.py
Normal file
162
tests/test_service_assurance.py
Normal file
|
|
@ -0,0 +1,162 @@
|
|||
import copy
|
||||
from datetime import datetime, timezone
|
||||
import importlib.util
|
||||
import json
|
||||
from pathlib import Path
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
import yaml
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
spec = importlib.util.spec_from_file_location('service_assurance', ROOT / 'scripts/service_assurance.py')
|
||||
m = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(m)
|
||||
|
||||
|
||||
class EvidenceTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.now = datetime(2026, 9, 5, 12, tzinfo=timezone.utc)
|
||||
self.contract = {'cluster_uid': 'expected', 'capture_max_age_seconds': 60,
|
||||
'signals': {'db.backup': {'owner': 'platform', 'max_age_seconds': 3600}}}
|
||||
self.observation = {'schema': 'railiance-platform.observation.v1',
|
||||
'cluster_uid': 'expected', 'captured_at': '2026-09-05T12:00:00Z',
|
||||
'signals': {'db.backup': {'observed_at': '2026-09-05T11:30:00Z', 'result': 'pass'}}}
|
||||
|
||||
def state(self):
|
||||
return m.evaluate(self.contract, self.observation, self.now)['signals']['db.backup']['state']
|
||||
|
||||
def test_current_old_missing_failure_and_unavailable_are_distinct(self):
|
||||
self.assertEqual(self.state(), 'healthy')
|
||||
self.observation['signals']['db.backup']['observed_at'] = '2026-09-05T10:00:00Z'
|
||||
self.assertEqual(self.state(), 'stale')
|
||||
self.observation['signals']['db.backup']['result'] = 'fail'
|
||||
self.assertEqual(self.state(), 'failed')
|
||||
self.observation['signals']['db.backup']['result'] = 'unavailable'
|
||||
self.assertEqual(self.state(), 'unavailable')
|
||||
self.observation['signals'].clear()
|
||||
self.assertEqual(self.state(), 'missing')
|
||||
|
||||
def test_replayed_capture_cannot_stay_healthy(self):
|
||||
self.observation['captured_at'] = '2026-09-05T11:59:00Z'
|
||||
self.assertEqual(self.state(), 'healthy')
|
||||
self.observation['captured_at'] = '2026-09-05T11:58:59Z'
|
||||
self.assertEqual(self.state(), 'stale')
|
||||
|
||||
def test_wrong_cluster_future_and_naive_times_refused(self):
|
||||
for field, value in [('cluster_uid', 'other'), ('captured_at', '2026-09-05T12:01:00Z'),
|
||||
('captured_at', '2026-09-05T12:00:00')]:
|
||||
d = copy.deepcopy(self.observation); d[field] = value
|
||||
with self.subTest(field=field, value=value), self.assertRaises(ValueError):
|
||||
m.evaluate(self.contract, d, self.now)
|
||||
self.observation['signals']['db.backup']['observed_at'] = '2026-09-06T00:00:00Z'
|
||||
with self.assertRaises(ValueError):
|
||||
self.state()
|
||||
|
||||
def test_no_unrecognized_payload_can_escape_in_error(self):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
p = Path(directory) / 'input.json'
|
||||
self.observation['token'] = 'CANARY_MUST_NOT_ESCAPE'
|
||||
p.write_text(json.dumps(self.observation))
|
||||
result = subprocess.run([sys.executable, str(ROOT / 'scripts/service_assurance.py'),
|
||||
'evaluate', str(p)], capture_output=True, text=True)
|
||||
self.assertEqual(result.returncode, 2)
|
||||
self.assertNotIn('CANARY', result.stdout + result.stderr)
|
||||
|
||||
def test_unknown_signal_and_extra_sample_fields_fail(self):
|
||||
for signals in [{'arbitrary': {}}, {'db.backup': {'observed_at': '2026-09-05T12:00:00Z',
|
||||
'result': 'pass', 'value': 'secret'}}]:
|
||||
self.observation['signals'] = signals
|
||||
with self.assertRaises(ValueError):
|
||||
self.state()
|
||||
|
||||
def test_health_does_not_imply_monitoring_or_guarantees(self):
|
||||
result = m.evaluate(self.contract, self.observation, self.now)
|
||||
self.assertTrue(result['healthy'])
|
||||
self.assertEqual(result['transport'], 'unmonitored')
|
||||
self.assertEqual(result['guarantees'], 'unsupported')
|
||||
|
||||
|
||||
@unittest.skipUnless((ROOT.parent / 'rapp-postgres/scripts/render_consumers.py').exists(),
|
||||
'owner package checkout required for integration checks')
|
||||
class AdmissionTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(self.tmp.cleanup)
|
||||
self.root = Path(self.tmp.name) / 'railiance-platform'
|
||||
self.package = self.root.parent / 'rapp-postgres'
|
||||
for folder in ['tools', 'helm', 'assurance']:
|
||||
shutil.copytree(ROOT / folder, self.root / folder)
|
||||
for folder in ['scripts', 'helm', 'consumers']:
|
||||
shutil.copytree(ROOT.parent / 'rapp-postgres' / folder, self.package / folder)
|
||||
|
||||
def check(self):
|
||||
return m.admission(self.root, self.package)
|
||||
|
||||
def test_current_occupancy_and_real_overflow(self):
|
||||
result = self.check()['cells']
|
||||
self.assertEqual(len(result['platform-pg']['consumers']), 4)
|
||||
self.assertEqual(result['platform-pg-2']['consumers'], ['sbom-nexus'])
|
||||
self.assertEqual(result['apps-pg-2']['consumers'], [])
|
||||
|
||||
def test_unowned_consumer_fails(self):
|
||||
p = self.root / 'assurance/placement-owners.json'
|
||||
data = json.loads(p.read_text()); del data['audit-core']; p.write_text(json.dumps(data))
|
||||
with self.assertRaises(ValueError): self.check()
|
||||
|
||||
def test_native_validator_rejects_fifth_consumer(self):
|
||||
p = self.package / 'consumers/sbom-nexus.yaml'
|
||||
d = yaml.safe_load(p.read_text()); d['spec']['cluster'] = 'platform-pg'; p.write_text(yaml.safe_dump(d))
|
||||
with self.assertRaisesRegex(ValueError, 'ceiling'): self.check()
|
||||
|
||||
def test_native_retention_and_effective_horizon_both_enforced(self):
|
||||
p = self.package / 'consumers/audit-core.yaml'
|
||||
for days in [1, 7, 60]:
|
||||
d = yaml.safe_load(p.read_text()); d['spec']['retention']['backupRetentionDays'] = days
|
||||
p.write_text(yaml.safe_dump(d))
|
||||
with self.subTest(days=days), self.assertRaises(ValueError): self.check()
|
||||
|
||||
def test_missing_overflow_and_instance_policy_drift(self):
|
||||
(self.package / 'helm/platform-pg-2-cluster.yaml').unlink()
|
||||
with self.assertRaises(OSError): self.check()
|
||||
|
||||
|
||||
class CollectorTests(unittest.TestCase):
|
||||
@classmethod
|
||||
def setUpClass(cls):
|
||||
sys.path.insert(0, str(ROOT / 'scripts'))
|
||||
cls.collector = __import__('capture_service_observation')
|
||||
|
||||
def test_wrong_cluster_stops_before_other_reads(self):
|
||||
with patch.object(self.collector, 'query', return_value='wrong') as query:
|
||||
with self.assertRaises(ValueError): self.collector.capture()
|
||||
self.assertEqual(query.call_count, 1)
|
||||
|
||||
def test_failed_native_reads_are_unavailable_not_pass(self):
|
||||
contract = m.read_contract(ROOT / 'assurance/service-contract.json')
|
||||
calls = []
|
||||
def query(args, **kwargs):
|
||||
calls.append(args)
|
||||
if args[:3] == ['get', 'namespace', 'kube-system']:
|
||||
return contract['cluster_uid']
|
||||
raise ValueError('unavailable')
|
||||
baseline = json.loads((ROOT / 'assurance/admission-baseline.json').read_text())
|
||||
with patch.object(self.collector, 'query', side_effect=query), patch.object(self.collector, 'admission', return_value=baseline):
|
||||
observation = self.collector.capture()
|
||||
for sample in observation['signals'].values(): self.assertEqual(sample['result'], 'unavailable')
|
||||
self.assertNotIn('secret', [str(a).lower() for call in calls for a in call])
|
||||
result = m.evaluate(contract, observation, datetime.now(timezone.utc))
|
||||
self.assertFalse(result['healthy'])
|
||||
|
||||
def test_quantity_conversion_and_invalid_units(self):
|
||||
self.assertEqual(self.collector.memory_bytes('1024Mi'), 1024**3)
|
||||
for value in ['NaN', '-1Gi', '1password', '']:
|
||||
with self.subTest(value=value), self.assertRaises(ValueError): self.collector.memory_bytes(value)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
|
|
@ -1,7 +1,7 @@
|
|||
# Current platform work
|
||||
|
||||
Reviewed 2026-09-05. Six open workplans: five blocked on explicit owner/live
|
||||
gates and one ready for local work. Completed designs and implementations are
|
||||
Reviewed 2026-09-05. Six open workplans: six blocked on explicit owner/live
|
||||
gates; RPF-WP-0036 now has its repository implementation. Completed designs and implementations are
|
||||
under `archived/`; their IDs and UUIDs are preserved. The number of blocked
|
||||
plans is not a count of missing implementations or independent incidents.
|
||||
|
||||
|
|
@ -12,10 +12,10 @@ plans is not a count of missing implementations or independent incidents.
|
|||
| [RPF-WP-0025](RPF-WP-0025-openbao-operator-only-access.md) | Private OpenBao access; fresh attended callback/login then guarded retraction | Coordinate package, issuer, tunnel and DNS owners; keep the window separate. |
|
||||
| [RPF-WP-0015](RPF-WP-0015-audit-core-custody-and-recovery-coordination.md) | Two prepared recovery exercises; fresh synthetic-load/outage approvals and custody readiness | S3 contributes lease/ESO and snapshot/unseal proof; S1/S2 and audit-core execute their parts. |
|
||||
| [RPF-WP-0035](RPF-WP-0035-credential-lane-implementation.md) | One implementation queue for secrets-engine JWT, Fluid operator KV and preflight signing | Three independent task gates; no new approval inherited from the completed designs. |
|
||||
| [RPF-WP-0036](RPF-WP-0036-platform-service-assurance.md) | **Ready:** service guarantees, recurring evidence, S3 emission, admission consistency, ownership handoff and demand review | Local design/checking can progress while live work waits. |
|
||||
| [RPF-WP-0036](RPF-WP-0036-platform-service-assurance.md) | Implemented local assurance/admission; waits for recurring restore evidence, Q2 reception and owner handoff | Run the assurance commands; live acceptance and external ownership remain gated. |
|
||||
|
||||
Start with RPF-WP-0036-T02/T05 for work that does not require a live credential
|
||||
or outage. Treat credential exposure closure as the highest-priority attended
|
||||
RPF-WP-0036-T02/T05/T07 are complete; T03/T04/T06 retain the remaining
|
||||
acceptance gates. Treat credential exposure closure as the highest-priority attended
|
||||
work; task order does not combine or waive approvals.
|
||||
|
||||
[Assessment and disposition of every plan](../history/2026-09-05-platform-intent-workplan-assessment.md)
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@ type: workplan
|
|||
title: "Close S3 service assurance and ownership gaps"
|
||||
domain: financials
|
||||
repo: railiance-platform
|
||||
status: ready
|
||||
status: blocked
|
||||
owner: codex
|
||||
created: "2026-09-05"
|
||||
updated: "2026-09-05"
|
||||
|
|
@ -39,7 +39,7 @@ This certifies the source review, not live service health or external acceptance
|
|||
|
||||
```task
|
||||
id: RPF-WP-0036-T02
|
||||
status: todo
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "eebcd5c7-4cfd-5084-9a21-ca1e4e748bdd"
|
||||
```
|
||||
|
|
@ -61,7 +61,7 @@ and package dependency rather than a blanket new-cluster project here.
|
|||
|
||||
```task
|
||||
id: RPF-WP-0036-T03
|
||||
status: todo
|
||||
status: wait
|
||||
priority: high
|
||||
state_hub_task_id: "d64446fb-870b-5ae9-9628-f1fb9d06c5a4"
|
||||
```
|
||||
|
|
@ -84,7 +84,7 @@ pass as restore proof. Record independent recovery-key access without values.
|
|||
|
||||
```task
|
||||
id: RPF-WP-0036-T04
|
||||
status: todo
|
||||
status: wait
|
||||
priority: high
|
||||
state_hub_task_id: "5351e0e4-6263-58f0-afb2-7c78c4cd6f68"
|
||||
```
|
||||
|
|
@ -105,7 +105,7 @@ finish before the receiver, but end-to-end acceptance cannot.
|
|||
|
||||
```task
|
||||
id: RPF-WP-0036-T05
|
||||
status: todo
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "5cded2e9-7edd-57bd-9f7e-7977b75004dc"
|
||||
```
|
||||
|
|
@ -127,7 +127,7 @@ under this task without its own owner-reviewed migration.
|
|||
|
||||
```task
|
||||
id: RPF-WP-0036-T06
|
||||
status: todo
|
||||
status: wait
|
||||
priority: medium
|
||||
state_hub_task_id: "3b74f79c-7366-5d00-bf0e-31c0464a8f33"
|
||||
```
|
||||
|
|
@ -154,7 +154,7 @@ the assessment and this task does not assert acceptance for another repo.
|
|||
|
||||
```task
|
||||
id: RPF-WP-0036-T07
|
||||
status: todo
|
||||
status: done
|
||||
priority: medium
|
||||
state_hub_task_id: "b32709d2-581d-5750-a04c-e3494a42d27e"
|
||||
```
|
||||
|
|
@ -171,3 +171,53 @@ it to S3 by implication.
|
|||
review trigger, or start a bounded consumer-backed delivery plan with explicit
|
||||
acceptance criteria. “No accepted demand; keep deploy gated” is a valid result.
|
||||
No Valkey, MinIO, RabbitMQ or new provider purchase is authorized by this plan.
|
||||
|
||||
## Implementation and remaining acceptance — 2026-09-05
|
||||
|
||||
T02 completed with `assurance/service-records.json` and ADR-0004: all three
|
||||
CNPG cells, OpenBao and both backup delivery surfaces disclose unsupported
|
||||
availability/RPO/RTO guarantees, existing evidence, retention and named decision
|
||||
owners. Numeric commitments have not been invented or approved for other owners.
|
||||
|
||||
T05 completed with the owner-native admission checker, hash-bound baseline and
|
||||
placement registry. It reuses local/package validators and rejects unowned
|
||||
consumers, fifth consumers, unhonoured retention and source disclosure drift.
|
||||
Corrected tenant-engine's completed PostgreSQL cutover and the deployed SBOM
|
||||
overflow cell. Live metadata matches one instance, 1Gi limit, 100 connections
|
||||
and 30-day retention on all three deployed cells. Dated owner evidence remains
|
||||
the authority for database/application cutover; this run moved no workloads.
|
||||
|
||||
T07 completed with ADR-0005: reuse approved backup storage only for backup;
|
||||
evaluate artifact-store's existing S3 interface for artifact demand; defer
|
||||
cache, general object store and broker deployment with concrete demand/review
|
||||
triggers. The fleet Q3 question is retained in T06's prepared master handoff;
|
||||
no external architecture assignment or provisioning occurred.
|
||||
|
||||
T03/T04 have working read-only `assurance-capture`, `assurance-check` and
|
||||
`assurance-admission` entry points plus adversarial tests. The collector pins
|
||||
the cluster UID and reads selected status, native aggregate connection counts,
|
||||
metrics and unauthenticated seal status. It captures no Secrets, SQL text from
|
||||
sessions, application rows or logs. Healthy/stale/missing/unavailable/failed
|
||||
are distinct; unknown fields, future timestamps and wrong clusters fail closed.
|
||||
The output explicitly says unmonitored and unsupported guarantees.
|
||||
|
||||
- **T03 waits:** fresh owner-validated isolated restore/snapshot/offsite receipts,
|
||||
independent recovery access, approved cadence and execution evidence. Existing
|
||||
live experiments stay in RPF-WP-0015; offsite exposure stays in RPF-WP-0029.
|
||||
No restore, snapshot creation, upload, scheduler or seal/unseal was run.
|
||||
- **T04 waits:** accepted Q2 receiving contract and controlled failure/absence
|
||||
delivery to a named recipient. The local producer is not a ratified Q2
|
||||
contract. Live metadata also surfaced failed ESO resources: forgejo-mailer,
|
||||
reuse-surface-runtime and target-revenue-runtime. Their current/obsolete scope
|
||||
and exact repair require consumer/platform acceptance before lane mutation;
|
||||
failures remain visible rather than excluded.
|
||||
- **T06 waits:** ownership acceptance and scoped alias/brief repair. The exact
|
||||
path/hash/caller inventory, dated retention decision through 2026-10-05 and
|
||||
prepared owner requests are in `assurance/ownership-handoffs.json` and
|
||||
`docs/platform-ownership-handoffs.md`. The installed brief generator still
|
||||
includes open legacy aliases; rerunning it alone cannot satisfy acceptance.
|
||||
No messages or external handoff acceptance were fabricated.
|
||||
|
||||
Evidence: `docs/evidence/RPF-WP-0036-assurance-2026-09-05.json`.
|
||||
The plan is blocked on these explicit live/owner gates, not finished merely
|
||||
because its repository implementation and tests pass.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue