Implement S3 service assurance and admission checks
Assistant: codex Assistant-Model: gpt-6-astra Assistant-Session: 01a06ecb-456a-71c2-b41e-0755d336e883
This commit is contained in:
parent
8f828c7609
commit
234b1b559f
21 changed files with 1728 additions and 30 deletions
64
docs/adr/ADR-0004-service-assurance-disclosure.md
Normal file
64
docs/adr/ADR-0004-service-assurance-disclosure.md
Normal file
|
|
@ -0,0 +1,64 @@
|
|||
---
|
||||
id: RPLAT-ADR-0004
|
||||
title: "ADR-0004 — Disclose unsupported service guarantees and evidence freshness"
|
||||
status: accepted
|
||||
revision: "1.0"
|
||||
owner: railiance-platform
|
||||
date: "2026-09-05"
|
||||
last_reviewed: "2026-09-05"
|
||||
review_interval: 6m
|
||||
---
|
||||
|
||||
# ADR-0004 — Service assurance disclosure
|
||||
|
||||
## Context
|
||||
|
||||
The platform has dated backup/restore evidence and three single-instance CNPG
|
||||
cells, but no accepted numeric end-to-end availability/RPO/RTO commitments.
|
||||
One successful scratch restore is not a recovery guarantee. A successful
|
||||
source validator is not current live health. RPF-WP-0036 makes that distinction
|
||||
executable and visible to consumers.
|
||||
|
||||
## Decision
|
||||
|
||||
Publish `assurance/service-records.json` as the S3-owned disclosure of supported
|
||||
and unsupported guarantees. Availability, RPO and RTO remain explicitly
|
||||
unsupported until the named service/package/consumer owners accept measurable
|
||||
targets backed by evidence. This is a platform disclosure decision, not an
|
||||
external owner's approval. A consumer requesting a stronger guarantee needs a
|
||||
recorded placement/recovery decision before admission. In particular, CNPG on
|
||||
one host does not provide host-loss HA; the dependency is S1 host failure
|
||||
domains, S2 runtime placement and package-level failover proof.
|
||||
|
||||
Use `assurance/service-contract.json` only for local diagnostics: 15-minute
|
||||
capture/health age, 36-hour successful backup/upload age, 30-day restore age,
|
||||
one-hour ESO refresh age, and 80% connection/memory occupancy for headroom.
|
||||
These intentionally conservative defaults expose
|
||||
missing evidence; they are neither SLAs nor approvals to install schedules.
|
||||
Service-specific cadence and receiver acceptance remain RPF-WP-0036-T03/T04.
|
||||
A different legitimate refresh interval needs an explicit service contract,
|
||||
not automatic acceptance of an indefinitely stale observation.
|
||||
|
||||
## Consequences
|
||||
|
||||
The checker distinguishes healthy, stale, missing, unavailable and failed
|
||||
samples. It rejects wrong clusters, unknown payload fields and future times.
|
||||
Its receipt states `transport: unmonitored` and `guarantees: unsupported`.
|
||||
Receipt syntax and freshness validation do not prove the truth of a supplied
|
||||
observation or replace a signed native restore receipt. The live collector
|
||||
queries only fixed metadata/status fields, a pg_stat_activity count and
|
||||
unauthenticated seal status; it cannot fetch Secrets or run backup, restore or
|
||||
seal/unseal mutations.
|
||||
|
||||
Sources of restore proof remain the owning packages and their existing
|
||||
validators. The platform accepts those artifacts after an approved isolated
|
||||
exercise; it does not convert old Markdown or example JSON into fresh proof.
|
||||
No unattended jobs or Q2 transport are deployed under this decision.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
Invent numeric targets from elapsed restore time: rejected because it creates
|
||||
an untested promise. Keep all checks waiting for telemetry: rejected because
|
||||
safe local diagnostics and unsupported-guarantee disclosure are useful now.
|
||||
Treat every source declaration as live: rejected because desired placement
|
||||
and actual deployment can diverge.
|
||||
55
docs/adr/ADR-0005-demand-gated-stateful-services.md
Normal file
55
docs/adr/ADR-0005-demand-gated-stateful-services.md
Normal file
|
|
@ -0,0 +1,55 @@
|
|||
---
|
||||
id: RPLAT-ADR-0005
|
||||
title: "ADR-0005 — Reuse existing storage and defer unrequested stateful services"
|
||||
status: accepted
|
||||
revision: "1.0"
|
||||
owner: railiance-platform
|
||||
date: "2026-09-05"
|
||||
last_reviewed: "2026-09-05"
|
||||
review_interval: 6m
|
||||
---
|
||||
|
||||
# ADR-0005 — Demand-gated stateful services
|
||||
|
||||
## Context
|
||||
|
||||
INTENT includes cache, object storage and messaging. Valkey is gated in the
|
||||
Makefile. CNPG already consumes an external governed S3 backup destination.
|
||||
`artifact-store` has local/S3 backend code, retention/hold semantics and a
|
||||
MinIO/STS assessment; its older SCOPE understates the code. The reviewed
|
||||
platform queue has no accepted general cache or messaging consumer request.
|
||||
Absence from this queue is not a claim that no future consumer could need one.
|
||||
|
||||
## Decision
|
||||
|
||||
- Cache: defer shared Valkey deployment. Reopen on a named consumer specifying
|
||||
latency, eviction/durability, isolation, capacity and an operating owner.
|
||||
- General object storage: reuse the existing backup destination only for its
|
||||
approved backup scope. Evaluate artifact-store's existing S3 interface for
|
||||
artifact-preservation demand before proposing another gateway or engine.
|
||||
Defer a general platform S3/MinIO service until accepted demand specifies
|
||||
retention, residency, capacity, recovery, credential scope and package owner.
|
||||
- Messaging: defer shared broker deployment until a named consumer defines
|
||||
delivery/replay/order guarantees, retention, throughput and recovery needs.
|
||||
Do not replace application-specific queues by assumption.
|
||||
|
||||
Review on a new accepted consumer request, a change in existing provider
|
||||
capability, or 2027-03-05, whichever comes first. This is an S3 deployment
|
||||
posture decision; it assigns no new obligation to potential consumers and
|
||||
chooses no vendor or engine. The Q3 fleet-owner question stays with
|
||||
railiance-master; local recovery work continues independently.
|
||||
|
||||
## Consequences
|
||||
|
||||
Keep deploy gates. Any accepted new demand creates a bounded delivery plan
|
||||
covering service contract, custody, tenancy, capacity, backup/restore, telemetry,
|
||||
package owner, costs and acceptance before provisioning. Existing backup
|
||||
credentials are not general artifact credentials. Platform does not take over
|
||||
artifact metadata lifecycle or application publication logic.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
Deploy Valkey/MinIO/RabbitMQ to satisfy the nouns in INTENT: rejected because
|
||||
it adds unsupported operations without an accepted workload. Declare external
|
||||
backup S3 a general object service: rejected because its scope and consumer
|
||||
contract are narrower.
|
||||
242
docs/evidence/RPF-WP-0036-assurance-2026-09-05.json
Normal file
242
docs/evidence/RPF-WP-0036-assurance-2026-09-05.json
Normal file
|
|
@ -0,0 +1,242 @@
|
|||
{
|
||||
"schema": "railiance-platform.assurance-evidence.v1",
|
||||
"date": "2026-09-05",
|
||||
"observation": {
|
||||
"schema": "railiance-platform.observation.v1",
|
||||
"cluster_uid": "a553c742-0115-43d4-99a4-a5ca56fe0786",
|
||||
"captured_at": "2026-09-05T09:37:05.118118+00:00",
|
||||
"signals": {
|
||||
"apps-pg.ready": {
|
||||
"result": "pass",
|
||||
"observed_at": "2026-09-05T09:36:57.163337+00:00"
|
||||
},
|
||||
"apps-pg.backup": {
|
||||
"result": "pass",
|
||||
"observed_at": "2026-09-05T02:15:07Z"
|
||||
},
|
||||
"apps-pg.wal": {
|
||||
"result": "pass",
|
||||
"observed_at": "2026-09-05T09:36:57.163365+00:00"
|
||||
},
|
||||
"apps-pg.headroom": {
|
||||
"result": "pass",
|
||||
"observed_at": "2026-09-05T09:36:42Z"
|
||||
},
|
||||
"platform-pg.ready": {
|
||||
"result": "pass",
|
||||
"observed_at": "2026-09-05T09:36:59.396739+00:00"
|
||||
},
|
||||
"platform-pg.backup": {
|
||||
"result": "pass",
|
||||
"observed_at": "2026-09-05T02:15:11Z"
|
||||
},
|
||||
"platform-pg.wal": {
|
||||
"result": "pass",
|
||||
"observed_at": "2026-09-05T09:36:59.396763+00:00"
|
||||
},
|
||||
"platform-pg.headroom": {
|
||||
"result": "pass",
|
||||
"observed_at": "2026-09-05T09:36:44Z"
|
||||
},
|
||||
"platform-pg-2.ready": {
|
||||
"result": "pass",
|
||||
"observed_at": "2026-09-05T09:37:01.824557+00:00"
|
||||
},
|
||||
"platform-pg-2.backup": {
|
||||
"result": "pass",
|
||||
"observed_at": "2026-09-05T02:15:08Z"
|
||||
},
|
||||
"platform-pg-2.wal": {
|
||||
"result": "pass",
|
||||
"observed_at": "2026-09-05T09:37:01.824580+00:00"
|
||||
},
|
||||
"platform-pg-2.headroom": {
|
||||
"result": "pass",
|
||||
"observed_at": "2026-09-05T09:36:40Z"
|
||||
},
|
||||
"openbao.seal": {
|
||||
"result": "pass",
|
||||
"observed_at": "2026-09-05T09:37:04.442236+00:00"
|
||||
},
|
||||
"eso.ready": {
|
||||
"result": "fail",
|
||||
"observed_at": "2026-09-05T09:37:05.118098+00:00"
|
||||
},
|
||||
"eso.refresh": {
|
||||
"result": "pass",
|
||||
"observed_at": "2026-08-06T12:33:40Z"
|
||||
}
|
||||
}
|
||||
},
|
||||
"evaluation_at_capture": {
|
||||
"schema": "railiance-platform.assurance-signal.v1",
|
||||
"cluster_uid": "a553c742-0115-43d4-99a4-a5ca56fe0786",
|
||||
"evaluated_at": "2026-09-05T09:37:05.118118+00:00",
|
||||
"signals": {
|
||||
"apps-pg.ready": {
|
||||
"state": "healthy",
|
||||
"owner": "railiance-platform"
|
||||
},
|
||||
"apps-pg.backup": {
|
||||
"state": "healthy",
|
||||
"owner": "railiance-platform"
|
||||
},
|
||||
"apps-pg.wal": {
|
||||
"state": "healthy",
|
||||
"owner": "railiance-platform"
|
||||
},
|
||||
"apps-pg.restore": {
|
||||
"state": "missing",
|
||||
"owner": "railiance-platform"
|
||||
},
|
||||
"apps-pg.headroom": {
|
||||
"state": "healthy",
|
||||
"owner": "railiance-platform"
|
||||
},
|
||||
"platform-pg.ready": {
|
||||
"state": "healthy",
|
||||
"owner": "rapp-postgres"
|
||||
},
|
||||
"platform-pg.backup": {
|
||||
"state": "healthy",
|
||||
"owner": "rapp-postgres"
|
||||
},
|
||||
"platform-pg.wal": {
|
||||
"state": "healthy",
|
||||
"owner": "rapp-postgres"
|
||||
},
|
||||
"platform-pg.restore": {
|
||||
"state": "missing",
|
||||
"owner": "rapp-postgres"
|
||||
},
|
||||
"platform-pg.headroom": {
|
||||
"state": "healthy",
|
||||
"owner": "rapp-postgres"
|
||||
},
|
||||
"platform-pg-2.ready": {
|
||||
"state": "healthy",
|
||||
"owner": "rapp-postgres"
|
||||
},
|
||||
"platform-pg-2.backup": {
|
||||
"state": "healthy",
|
||||
"owner": "rapp-postgres"
|
||||
},
|
||||
"platform-pg-2.wal": {
|
||||
"state": "healthy",
|
||||
"owner": "rapp-postgres"
|
||||
},
|
||||
"platform-pg-2.restore": {
|
||||
"state": "missing",
|
||||
"owner": "rapp-postgres"
|
||||
},
|
||||
"platform-pg-2.headroom": {
|
||||
"state": "healthy",
|
||||
"owner": "rapp-postgres"
|
||||
},
|
||||
"openbao.seal": {
|
||||
"state": "healthy",
|
||||
"owner": "railiance-platform"
|
||||
},
|
||||
"openbao.snapshot": {
|
||||
"state": "missing",
|
||||
"owner": "railiance-platform"
|
||||
},
|
||||
"openbao.restore": {
|
||||
"state": "missing",
|
||||
"owner": "railiance-platform"
|
||||
},
|
||||
"offsite.upload": {
|
||||
"state": "missing",
|
||||
"owner": "railiance-platform"
|
||||
},
|
||||
"offsite.restore": {
|
||||
"state": "missing",
|
||||
"owner": "railiance-platform"
|
||||
},
|
||||
"eso.ready": {
|
||||
"state": "failed",
|
||||
"owner": "railiance-platform"
|
||||
},
|
||||
"eso.refresh": {
|
||||
"state": "stale",
|
||||
"owner": "railiance-platform"
|
||||
}
|
||||
},
|
||||
"transport": "unmonitored",
|
||||
"guarantees": "unsupported",
|
||||
"threshold_status": "local-diagnostic-only",
|
||||
"healthy": false
|
||||
},
|
||||
"live_admission": {
|
||||
"captured_at": "2026-09-05T09:28:42.234951+00:00",
|
||||
"cells": {
|
||||
"apps-pg": {
|
||||
"instances": 1,
|
||||
"memory_limit": "1Gi",
|
||||
"max_connections": "100",
|
||||
"retention": "30d"
|
||||
},
|
||||
"platform-pg": {
|
||||
"instances": 1,
|
||||
"memory_limit": "1Gi",
|
||||
"max_connections": "100",
|
||||
"retention": "30d"
|
||||
},
|
||||
"platform-pg-2": {
|
||||
"instances": 1,
|
||||
"memory_limit": "1Gi",
|
||||
"max_connections": "100",
|
||||
"retention": "30d"
|
||||
}
|
||||
}
|
||||
},
|
||||
"live_spec_matches_source": true,
|
||||
"eso_failures": [
|
||||
{
|
||||
"name": "forgejo-mailer",
|
||||
"namespace": "forgejo",
|
||||
"refresh": "2026-08-06T12:33:40Z",
|
||||
"conditions": [
|
||||
{
|
||||
"type": "Ready",
|
||||
"status": "False",
|
||||
"reason": "SecretSyncedError"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "reuse-surface-runtime",
|
||||
"namespace": "reuse",
|
||||
"refresh": "2026-08-06T20:25:39Z",
|
||||
"conditions": [
|
||||
{
|
||||
"type": "Ready",
|
||||
"status": "False",
|
||||
"reason": "SecretSyncedError"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "target-revenue-runtime",
|
||||
"namespace": "target-revenue",
|
||||
"refresh": "2026-09-04T14:52:08Z",
|
||||
"conditions": [
|
||||
{
|
||||
"type": "Ready",
|
||||
"status": "False",
|
||||
"reason": "SecretSyncedError"
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"validation": {
|
||||
"full_tests": 176,
|
||||
"focused_assurance_tests": 14,
|
||||
"admission_baseline": "pass"
|
||||
},
|
||||
"live_mutations": false,
|
||||
"secret_values_read": false,
|
||||
"scheduling_installed": false,
|
||||
"q2_delivery_proven": false,
|
||||
"restore_executed": false
|
||||
}
|
||||
|
|
@ -75,7 +75,9 @@ triggers. It is not the same role as the operator of the cluster.
|
|||
| Workload | Substrate | Provided level | Placement owner | Co-signer | §13 evidence |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| `audit-core` | `platform-pg` | P1 | `audit-core` | this repo | probes exist |
|
||||
| `tenant-engine` | SQLite PVC current; `platform-pg` desired | off-ladder current; P1 desired | `tenant-engine` | this repo | source conformance; live cutover absent |
|
||||
| `tenant-engine` | `platform-pg` | P1 | `tenant-engine` | this repo | owner cutover evidence 2026-08-21 |
|
||||
| `core-hub` | `platform-pg` | P1 | `core-hub` | this repo | capacity admission 2026-08-20; private shadow 2026-08-21 |
|
||||
| `sbom-nexus` | `platform-pg-2` | P1 | `sbom-nexus` | this repo | package boundary/restore evidence 2026-08-22 |
|
||||
| `user-engine` | `user-engine-pg` | P2 | `net-kingdom` | this repo | not assessed here |
|
||||
| `target-revenue` | `target-revenue-pg` | P2 | `target-revenue` | this repo | not assessed here |
|
||||
| `forgejo` | `forgejo-db` | P2 | this repo | — | single consumer |
|
||||
|
|
@ -144,13 +146,16 @@ one". "Provision a second cluster" is an answer; "revisit at the time" is not.
|
|||
|
||||
| Cluster | Ceiling declared | Overflow target | Standing |
|
||||
| --- | --- | --- | --- |
|
||||
| `platform-pg` | 4 declarations (estimated; ADR-0004) | `platform-pg-2`, named but absent | 3/4 including the isolation probe; measure before the next workload |
|
||||
| `platform-pg` | 4 declarations (package ADR-0004) | `platform-pg-2`, deployed | 4/4 including probe and Core Hub; fifth refused |
|
||||
| `platform-pg-2` | 4 declarations (same package validator) | No third cell approved | 1/4: sbom-nexus; owner admission and restore completed 2026-08-22 |
|
||||
| `apps-pg` | 3 consumers; 20 connections each, 1Gi memory binds first | `apps-pg-2` | 2/3; overflow must exist before a 4th |
|
||||
| `forgejo-db` | n/a — single consumer | n/a | — |
|
||||
|
||||
Both live shared clusters are one consumer away from the trigger. This repo
|
||||
owes `apps-pg`'s ceiling under its own rule, which is the correct asymmetry: a
|
||||
policy whose author is exempt is not a policy.
|
||||
Source and live metadata reviewed 2026-09-05: platform-pg is full; apps-pg
|
||||
remains 2/3 and apps-pg-2 is still unapplied. The generated
|
||||
`assurance/admission-baseline.json` binds these disclosures to the actual owner
|
||||
validators/declarations. `make assurance-admission` refuses source drift.
|
||||
Package acceptance, rather than this table, authorizes consumer provisioning.
|
||||
|
||||
## 5. Triggers, and who is watching them
|
||||
|
||||
|
|
@ -189,7 +194,8 @@ risk that nobody can see is strictly worse than one that is stated".
|
|||
|
||||
| Cluster | Co-resident classes | Mismatch |
|
||||
| --- | --- | --- |
|
||||
| `platform-pg` | `latency-critical` (`tenant-engine`) + `batch` (`audit-core`) | **yes — reported** |
|
||||
| `platform-pg` | `latency-critical` (`tenant-engine`, `core-hub`) + `batch` (`audit-core`) | **yes — reported** |
|
||||
| `platform-pg-2` | `interactive` (`sbom-nexus`) | no |
|
||||
| `apps-pg` | `interactive` + `interactive` | no |
|
||||
| `forgejo-db` | single consumer | n/a |
|
||||
|
||||
|
|
@ -256,3 +262,11 @@ onboarding enforce it.
|
|||
Six months, or on: a new shared-cluster consumer, a co-residency class change,
|
||||
`railiance-telemetry` going live, or the framework reaching `accepted`. Next
|
||||
review due **2027-02-17**.
|
||||
|
||||
## Service assurance implementation — 2026-09-05
|
||||
|
||||
`docs/service-assurance.md` supplies the bounded local producer and admission
|
||||
check. Capture proves database readiness/backup/controller status at a point in
|
||||
time, and currently exposes failed/stale ESO resources. It does not implement
|
||||
continuous capacity monitoring. Trigger 5 remains unmonitored until its native
|
||||
headroom adapter and Q2 recipient are accepted (RPF-WP-0036-T03/T04).
|
||||
|
|
|
|||
49
docs/platform-ownership-handoffs.md
Normal file
49
docs/platform-ownership-handoffs.md
Normal file
|
|
@ -0,0 +1,49 @@
|
|||
# Platform compatibility handoffs — 2026-09-05
|
||||
|
||||
RPF-WP-0036-T06 inventory: `assurance/ownership-handoffs.json` contains exact
|
||||
paths/hashes, accepting owner candidates, callers, review dates and acceptance
|
||||
criteria. S3 custody/CCR policies and shared backup mechanisms stay here.
|
||||
|
||||
Retain each operational entry point until the proposed owner accepts its
|
||||
canonical replacement, callers pass dry-run/smoke checks, and rollback is
|
||||
recorded. Review by 2026-10-05 or on owner acceptance. No files were removed,
|
||||
no ownership acceptance was invented, and no coordination messages were sent.
|
||||
This is the dated platform retention decision; T06 remains waiting for accepted
|
||||
handoffs and the separate derived-state repair.
|
||||
|
||||
The forge contract already distinguishes forge requirements/verification from
|
||||
shared backup mechanisms. Split those concerns; do not move all backup code
|
||||
just because its filename mentions Forgejo. The platform OpenBao package
|
||||
handoff is likewise already documented: preserve custody/governance here and
|
||||
only adopt the package owner's authoritative replacement for its wrapper.
|
||||
ArgoCD files need artifact-level ownership: runtime bootstrap to S2, generic
|
||||
paved paths to S4, app releases to S5. ESO credential stores retain S3 custody
|
||||
review even when another repo packages their deployment.
|
||||
|
||||
## Prepared owner requests
|
||||
|
||||
These are reviewable request content, not sent messages:
|
||||
|
||||
- Forge/package/app owners: accept or amend the exact source inventory, name
|
||||
canonical entry points and callers, and return revision-pinned dry-run,
|
||||
smoke and rollback evidence before retirement.
|
||||
- Repo-manager/State Hub: reconcile only the legacy UUID-to-canonical map in
|
||||
the assessment's derived-state caveat; keep file-backed identities intact.
|
||||
The installed brief generator reads open Hub workplans without excluding
|
||||
legacy retired aliases, so rerunning it alone would reproduce duplicates.
|
||||
- Railiance-telemetry: supply versioned receiving schema, transport and
|
||||
retention contract, intended recipient, and a controlled stale-event/absence
|
||||
acceptance path for the local S3 producer. It is currently unmonitored.
|
||||
- Railiance-master: determine the accountable fleet Q3 recovery owner and
|
||||
boundary. S3 continues its own service recovery work meanwhile.
|
||||
|
||||
## Live ESO findings requiring scope confirmation
|
||||
|
||||
The 2026-09-05 metadata capture found `SecretSyncedError` for
|
||||
`forgejo/forgejo-mailer`, `reuse/reuse-surface-runtime`, and
|
||||
`target-revenue/target-revenue-runtime`. Their last refreshes were August 6,
|
||||
August 6 and September 4 respectively. Package/consumer and platform custody
|
||||
owners must first classify current versus obsolete resources, then review
|
||||
any exact lane repair or retirement. No Secret values were read and no failed
|
||||
resource was simply excluded to make the aggregate pass. These results do not
|
||||
by themselves identify a broken provider credential or authorize rotation.
|
||||
|
|
@ -1,8 +1,8 @@
|
|||
# S3 consumer service interfaces
|
||||
|
||||
Work record: `RAILIANCE-WP-0016` item 17; `RPF-WP-0018` T04
|
||||
Date: 2026-08-15, revised 2026-08-17
|
||||
Contract version of this index: `1.1.0`
|
||||
Date: 2026-08-15, revised 2026-09-05
|
||||
Contract version of this index: `1.2.0`
|
||||
|
||||
Consumers bind to these contracts, not to Helm chart names or pod topology.
|
||||
Changing an engine behind a versioned interface does not bump the interface
|
||||
|
|
@ -101,7 +101,7 @@ Consumer is a `consumers[].declaration`. Boundary unit is
|
|||
| Declared ceiling / overflow target | **4 declarations / `platform-pg-2`** | The isolation probe counts. `rapp-postgres` rejects a fifth declaration on one cell; ADR-0004 owns the number. |
|
||||
|
||||
Service classes currently co-resident on `platform-pg`: `latency-critical`
|
||||
(`tenant-engine`) and `batch` (`audit-core`). Nothing prioritises between them
|
||||
(`tenant-engine`, `core-hub`) and `batch` (`audit-core`). Nothing prioritises between them
|
||||
— community PostgreSQL has no resource governor. Reported per Tenancy Posture
|
||||
§8.3.3.
|
||||
|
||||
|
|
@ -115,3 +115,20 @@ the lane without restating TTL, delivery, or revocation rules.
|
|||
|
||||
Valkey and in-cluster MinIO have capability blocks in `SCOPE.md` and no
|
||||
live service. They do not receive a `v1` until a consumer exists.
|
||||
|
||||
## Service assurance disclosure (1.2.0, additive)
|
||||
|
||||
`assurance/service-records.json` explicitly discloses unsupported availability,
|
||||
RPO and RTO guarantees, decision owners and existing recovery evidence;
|
||||
`docs/service-assurance.md` documents the executable local checks. Diagnostic
|
||||
freshness budgets are not SLAs or approval to deploy schedules. Consumers that
|
||||
require stronger guarantees need a placement/recovery decision.
|
||||
|
||||
Current source occupancy: platform-pg 4/4 including the isolation probe,
|
||||
platform-pg-2 1/4 for sbom-nexus, apps-pg 2/3, apps-pg-2 0/3 and unapplied.
|
||||
All three deployed cells declare one instance, 1Gi memory limit, 100 aggregate
|
||||
connections and 30-day retention; live metadata matched on 2026-09-05. The
|
||||
second platform cell's approved restore took 51 seconds on 2026-08-22; this
|
||||
is dated evidence, not an RTO. Tenant-engine cut over from SQLite on 2026-08-21.
|
||||
`assurance/admission-baseline.json` carries owner-native declarations, hashes
|
||||
and limits. Do not refresh that baseline without reviewing changes.
|
||||
|
|
|
|||
105
docs/service-assurance.md
Normal file
105
docs/service-assurance.md
Normal file
|
|
@ -0,0 +1,105 @@
|
|||
# S3 service assurance — v1
|
||||
|
||||
Workplan: RPF-WP-0036. Disclosure authority:
|
||||
[ADR-0004](adr/ADR-0004-service-assurance-disclosure.md).
|
||||
|
||||
## Operator entry points
|
||||
|
||||
```bash
|
||||
make assurance-admission
|
||||
make assurance-capture
|
||||
make assurance-check
|
||||
```
|
||||
|
||||
Admission requires the adjacent reviewed `rapp-postgres` checkout and PyYAML,
|
||||
which existing platform tooling already uses. Capture requires the existing
|
||||
noninteractive SSH route to `railiance01`, with kubectl on that host. It pins
|
||||
the kube-system UID from existing platform evidence and stops on mismatch.
|
||||
It does not trust the default workstation kube context. Failure prints only a
|
||||
fixed error code, never raw command output. No Secret, application row/log or credential query is run. A read-only
|
||||
pg_stat_activity count and kubectl metrics supply aggregate headroom;
|
||||
unauthenticated `bao status` supplies seal state. The observation is written to
|
||||
`/tmp/railiance-platform-observation.json` by default; override with
|
||||
`ASSURANCE_OBSERVATION`. Run the check only after capture succeeds.
|
||||
|
||||
`evaluate` exits 0 when every required sample is healthy, 1 for incomplete,
|
||||
stale, failed or unavailable samples, and 2 for invalid/unavailable input.
|
||||
There is intentionally no green overall service claim while evidence is
|
||||
missing. To replay a dated observation:
|
||||
|
||||
```bash
|
||||
python3 scripts/service_assurance.py evaluate observation.json --now 2026-09-05T12:00:00Z
|
||||
```
|
||||
|
||||
`--now` is for replay/testing, not a way to freshen old evidence. The output
|
||||
retains evaluation time, unsupported guarantees and unmonitored transport.
|
||||
A `pass` sample is an attestation from its producer, not cryptographic proof.
|
||||
Only approved owner receipts can close the workplan's live acceptance gates.
|
||||
|
||||
## Supported source contract and missing adapters
|
||||
|
||||
The closed observation envelope carries schema, pinned cluster UID, capture
|
||||
time and a configured set of signals. Each sample has only `observed_at` and
|
||||
`result` (`pass`, `fail`, `unavailable`). Free text, values, paths and unknown
|
||||
signals are rejected. For backup/restore/upload signals the observed time is
|
||||
the completion time of that action; it is not the time someone read the file.
|
||||
A capture older than 15 minutes never passes even when a backup is recent.
|
||||
|
||||
The collector supplies database readiness, last completed backup and continuous
|
||||
archiving for apps-pg/platform-pg/platform-pg-2, plus fleet ESO Ready/oldest-refresh
|
||||
status, OpenBao seal state, and aggregate connection/memory headroom. Headroom
|
||||
uses a local diagnostic ceiling of 80% of source connection/memory limits; it
|
||||
is not a workload latency guarantee. Metrics timestamps retain their age. ContinuousArchiving is a controller condition, not a measured WAL
|
||||
lag guarantee. The fleet ESO aggregate is deliberately a conservative inventory
|
||||
check; an obsolete resource must be explicitly classified by its owner before
|
||||
it is excluded. One ESO failure cannot disappear inside an average.
|
||||
|
||||
The following remain missing until a native value-safe adapter and acceptance
|
||||
exist: validated isolated restore receipts,
|
||||
OpenBao snapshot/restore proof, and offsite upload/restore receipts. Missing
|
||||
adapters are not inferred healthy from pod readiness. The local producer is
|
||||
not the Q2 standard; railiance-telemetry has no implemented receiving contract
|
||||
in the reviewed checkout. Integration, routing and scheduled delivery remain
|
||||
T04, and no notification was sent during implementation.
|
||||
|
||||
## Service records and evidence inventory
|
||||
|
||||
`assurance/service-records.json` covers all three database cells, OpenBao,
|
||||
Forgejo backup and multi-host CNPG logical offsite delivery. It distinguishes
|
||||
unsupported objectives, declared retention, recovery custody and decision
|
||||
owners. The old Forgejo cron example mentions an RPO 24h target; that is not a
|
||||
newly accepted or verified current service guarantee. RPF-WP-0029 still gates
|
||||
provider invalidation/replacement recovery for the shared offsite lane.
|
||||
|
||||
| Evidence | Existing authority | Current acceptance limit |
|
||||
| --- | --- | --- |
|
||||
| CNPG completed backup/WAL | Cluster controller status; live capture | Current status only; does not exercise restore |
|
||||
| apps-pg restore | RPF-WP-0019 evidence, 2026-08-20 | Dated 56-second scratch restore; no numeric RTO commitment |
|
||||
| platform-pg restore/PITR | rapp-postgres restore procedure and evidence | Recurring current proof and independent recovery access remain T03 |
|
||||
| platform-pg-2 restore | RAPP-POSTGRES-WP-0005-T04, 2026-08-22 | 51-second isolated restore after SBOM migration; not a continuous guarantee |
|
||||
| OpenBao snapshot | WARDEN-WP-0027 preparation receipt, 2026-08-23 | Snapshot/encrypted off-host preparation, not isolated restore |
|
||||
| OpenBao restore | Existing `openbao-validate-restore-evidence.sh` and package procedure | Example receipt cannot pass as a fresh execution |
|
||||
| Recovery exercise | RPF-WP-0015-T02/T03 | Separate windows, synthetic driver/quorum and abort operator required |
|
||||
| Logical/Forgejo offsite | activity-core backup definitions, existing helper/runbooks | No new upload/restore performed; RPF-WP-0029 remains open |
|
||||
|
||||
## Admission and disclosure drift
|
||||
|
||||
`assurance/admission-baseline.json` is generated by the `admission` command from
|
||||
local CNPG CRs, `placement-owners.json`, and the native rapp-postgres renderer.
|
||||
The checker reuses both native capacity validators. It also rejects missing
|
||||
owners, changed cell placement, unhonoured inherited retention and mismatch
|
||||
between effective package retention and the actual CR. Hashes pin every
|
||||
consumed source artifact; changing package code or declarations requires review
|
||||
before refreshing the baseline. Do not blindly regenerate it in CI.
|
||||
|
||||
Source occupancy: apps-pg 2/3, apps-pg-2 0/3 (not deployed), platform-pg 4/4
|
||||
including the probe, platform-pg-2 1/4 for sbom-nexus. Tenant-engine's PostgreSQL
|
||||
cutover completed 2026-08-21; its old SQLite-current claim was stale. Platform-pg-2
|
||||
was accepted and restored on 2026-08-22; it is no longer merely named overflow.
|
||||
Core Hub is latency-critical alongside audit-core batch work; no service-class
|
||||
priority is implied. See the generated baseline for per-cell details.
|
||||
|
||||
Source admission checks happen before deployment in the existing apps-pg and
|
||||
package owner paths; this additional check detects disclosure drift. It does
|
||||
not apply resources or become a second provisioning engine. Any new consumer
|
||||
needs an owner entry and a reviewed baseline update, even within free capacity.
|
||||
Loading…
Add table
Add a link
Reference in a new issue