Complete local Prometheus mapping and record live acceptance blockers

Assistant: codex
Assistant-Model: gpt-6-astra
Assistant-Session: 01a0e6f1-443f-7783-9920-a16b2ffc467f

README.md

@@ -40,5 +40,7 @@ operator delivery proof.
 and Grafana for railiance01, with telemetry.coulomb.social as the intended
 Grafana hostname. This repo retains Q2 contracts and rule meaning. The local
 receiver and activity candidates are reference tooling; do not enable them as
-a second production monitoring plane. Runtime deployment remains pending in
+a second production monitoring plane. The package records private installation
-the package foundation workplan.
+and restore proof; delivery acceptance remains blocked in RTEL-WP-0002-T04.
+The [Prometheus mapping](docs/prometheus-mapping.md) supplies a tested report
+exporter and failure/absence rules for package integration.

WORK-RECORDS.md

@@ -8,7 +8,7 @@
 
 | Kind | ID | Status | Lane | Source |
 | --- | --- | --- | --- | --- |
-| workplan | RTEL-WP-0002 | active | — | workplans/RTEL-WP-0002-signal-contract.md |
+| workplan | RTEL-WP-0002 | blocked | — | workplans/RTEL-WP-0002-signal-contract.md |
 | workplan | RTELE-WP-0001 | finished | — | workplans/RTELE-WP-0001-statehub-bootstrap.md |
 | task | RTEL-WP-0002-T01 | done | — | workplans/RTEL-WP-0002-signal-contract.md |
 | task | RTEL-WP-0002-T02 | done | — | workplans/RTEL-WP-0002-signal-contract.md |

contracts/platform-assurance-2026-09-28.json

@@ -0,0 +1,35 @@
+{
+  "schema": "railiance-telemetry.stream.v1",
+  "stream": "railiance-platform.service-assurance",
+  "producer": "railiance-platform",
+  "recipient": "railiance-platform-operator",
+  "signals": [
+    "apps-pg.ready",
+    "apps-pg.backup",
+    "apps-pg.wal",
+    "apps-pg.restore",
+    "apps-pg.headroom",
+    "platform-pg.ready",
+    "platform-pg.backup",
+    "platform-pg.wal",
+    "platform-pg.restore",
+    "platform-pg.headroom",
+    "platform-pg-2.ready",
+    "platform-pg-2.backup",
+    "platform-pg-2.wal",
+    "platform-pg-2.restore",
+    "platform-pg-2.headroom",
+    "openbao.seal",
+    "openbao.snapshot",
+    "openbao.restore",
+    "offsite.upload",
+    "offsite.restore",
+    "eso.ready",
+    "eso.refresh",
+    "forgejo-db.restore",
+    "eso.token-renewal"
+  ],
+  "max_event_age_seconds": 900,
+  "heartbeat_seconds": 900,
+  "retention_days": 30
+}

docs/prometheus-mapping.md

@@ -0,0 +1,56 @@
+# Platform assurance Prometheus mapping
+
+RTEL-WP-0002-T04 supplies `scripts/prometheus_export.py` and
+`rules/platform-assurance.json` for integration by rapp-telemetry. These are local
+artifacts; no listener, scraper, schedule or Alertmanager route is installed.
+
+Use `contracts/platform-assurance-2026-09-28.json` with current platform reports.
+It adds the producer-owned `eso.token-renewal` classification to the original 23
+signals. The original contract stays unchanged for existing SQLite databases:
+do not change their binding without an explicit migration. Export consumes the
+evaluation report itself, not the evidence envelope's `observation`/`evaluation`
+wrapper. Unknown fields, signals, states and scope are rejected by the adapter.
+
+```bash
+python3 scripts/prometheus_export.py \
+  --contract contracts/platform-assurance-2026-09-28.json \
+  /path/to/fresh-platform-report.json
+promtool check rules rules/platform-assurance.json
+promtool test rules tests/prometheus-rules.json
+```
+
+The exporter prints Prometheus text to stdout only after validation; errors use
+stderr and exit 2. The package must publish via a temporary file and atomic rename
+only on success, or serve successful output through an admitted private exporter.
+Never redirect directly over the currently scraped file. No textfile collector
+is assumed to exist. The package must select and admit that transport and executor.
+
+Each signal has five `railiance_assurance_state` gauge series, exactly one set to
+1. Labels are bounded by the contract (`stream`, `producer`, `signal`, `state`).
+No logs, credentials or free-text producer details are exported. The separate
+`railiance_assurance_observed_timestamp_seconds` gauge holds original evaluation
+time; re-export does not refresh it. Reports older than 900 seconds or in the
+future are rejected. State and timestamp metric families must be scraped together
+from one complete export for this stream.
+
+`RailianceAssuranceUnhealthy` fires for each non-healthy classification.
+`RailianceAssuranceEmissionAbsent` fires for never-seen/disappeared timestamp
+series, observation age over 900 seconds, or future observation time. These rules
+have no additional `for` delay. They retain source semantics and the proposed
+15-minute transport budget. They do not reclassify backup/restore age.
+
+Platform's token-renewal signal currently uses a 36-hour source age threshold.
+It reaches the unhealthy rule with all other source classifications. The inbox
+request for failed CronJobs or no success for 48 hours (platform RPF-WP-0046-T06)
+is not closed by this mapping: exact Job-failure observation and actual delivery
+still need owner acceptance. Do not silently replace S3's threshold in Q2.
+
+The JSON rule file is valid Prometheus YAML input. Package integration must wrap
+its groups in the selected PrometheusRule/release configuration and verify live
+selectors, loading and routing. The warning/owner labels do not identify a
+confirmed human recipient. Prometheus loss cannot be detected by its own rules;
+an outside-node heartbeat receiver and acknowledged failure/absence drill remain
+mandatory. The SQLite inbox and local rule tests cannot satisfy that gate.
+
+Format and validation references: [Prometheus exposition format](https://prometheus.io/docs/instrumenting/exposition_formats/)
+and [native rule testing](https://prometheus.io/docs/prometheus/3.7/configuration/unit_testing_rules/).

docs/signal-contract.md

@@ -110,3 +110,8 @@ implementation. Do not activate its candidate activity definitions as a parallel
 production monitoring plane. Q2 now needs a reviewed Prometheus exporter/rule
 mapping preserving the existing state semantics, and actual Alertmanager delivery
 acceptance. Package runtime/admission is tracked by RAPP-TELEMETRY-WP-0001.
+
+September 28: the local [Prometheus mapping](prometheus-mapping.md) now supplies
+that exporter and tested failure/absence rules. It uses a separate dated contract
+for the added platform token-renewal signal. Package integration and actual
+delivery acceptance remain blocked under RTEL-WP-0002-T04.

history/2026-09-28-loose-end-review.md

@@ -0,0 +1,38 @@
+# Loose-end review — 2026-09-28
+
+Reviewed both repository workplans: RTELE-WP-0001 is finished with all three
+tasks done; RTEL-WP-0002 has T01–T03 done and T04 waiting. No proposed, ready or
+other unfinished workplans exist in this checkout. No whole task can honestly
+close without live acceptance, but T04's exporter/rule implementation is now done.
+
+Added a strict stdout Prometheus exporter, a dated 24-signal contract including
+`eso.token-renewal`, failure/absence rules, and native rule fixtures. Preserved
+the original contract for existing SQLite bindings and producer evaluation time
+for absence detection. Updated the README's stale package-deployment claim.
+
+Validation:
+
+- `python3 -m unittest discover -s tests -v`: 19 passed.
+- Prometheus 3.5.0 `promtool check rules rules/platform-assurance.json`: two valid rules.
+- `promtool test rules tests/prometheus-rules.json`: seven scenarios passed
+  (never seen, fresh, exact budget, stopped producer, future clock, disappeared
+  scrape and failed token renewal).
+- `promtool check metrics`: passed on exported platform September 27 evidence.
+  That compatibility check used the original evaluation time, not a claim that
+  yesterday's evidence is fresh today.
+
+Reviewed platform source at `c3607ff`, including
+`docs/evidence/2026-09-27-service-assurance.json` and the source contract, and
+rapp-telemetry records at `6d2e9fe`. The package records private install/restore
+as complete. Its T04 retains delivery, outside-node watchdog and recurring
+backup admission. No live infrastructure was modified or notification sent.
+
+The September 23 platform inbox request is acknowledged as read. Its existing
+RPF-WP-0046-T06 retains exact failed-Job/48-hour delivery acceptance; the current
+S3 renewal signal uses 36 hours and must not be silently redefined here.
+
+RTEL-WP-0002 is now blocked; T04 remains wait for accepted package transport,
+confirmed recipient/channel, actual acknowledged failure/absence, independent
+watchdog and recurring backup ownership. Bernd Worsch supplies recipient and
+admission decisions; rapp-telemetry/platform retain runtime/custody integration.
+No tasks or workplans were created, and no unfinished workplan was closed.

rules/platform-assurance.json

@@ -0,0 +1,25 @@
+{
+  "groups": [
+    {
+      "name": "railiance-platform-assurance",
+      "rules": [
+        {
+          "alert": "RailianceAssuranceUnhealthy",
+          "expr": "railiance_assurance_state{stream=\"railiance-platform.service-assurance\",producer=\"railiance-platform\",state!=\"healthy\"} == 1",
+          "labels": {
+            "severity": "warning",
+            "owner": "railiance-telemetry"
+          }
+        },
+        {
+          "alert": "RailianceAssuranceEmissionAbsent",
+          "expr": "absent(railiance_assurance_observed_timestamp_seconds{stream=\"railiance-platform.service-assurance\",producer=\"railiance-platform\"}) or (time() - railiance_assurance_observed_timestamp_seconds{stream=\"railiance-platform.service-assurance\",producer=\"railiance-platform\"} > 900) or (railiance_assurance_observed_timestamp_seconds{stream=\"railiance-platform.service-assurance\",producer=\"railiance-platform\"} > time())",
+          "labels": {
+            "severity": "warning",
+            "owner": "railiance-telemetry"
+          }
+        }
+      ]
+    }
+  ]
+}

scripts/prometheus_export.py

@@ -0,0 +1,55 @@
+#!/usr/bin/env python3
+"""Render a validated S3 report as Prometheus text; no listener or delivery."""
+import argparse
+from datetime import datetime, timezone
+import json
+from pathlib import Path
+import sys
+
+from platform_event import translate
+from receiver import STATES, contract_check, instant, read_json
+
+
+def labels(values):
+    def escape(value):
+        return value.replace('\\', '\\\\').replace('\n', '\\n').replace('"', '\\"')
+    return '{' + ','.join(f'{key}="{escape(value)}"' for key, value in values.items()) + '}'
+
+
+def render(report, contract, now):
+    contract_check(contract)
+    event = translate(report, contract)
+    observed = instant(event['observed_at']).timestamp()
+    if not 0 <= now.timestamp() - observed <= contract['max_event_age_seconds']:
+        raise ValueError('stale or future report')
+    identity = dict(stream=contract['stream'], producer=contract['producer'])
+    lines = ['# HELP railiance_assurance_state Producer classification, one hot per signal.',
+             '# TYPE railiance_assurance_state gauge']
+    for signal, state in sorted(event['states'].items()):
+        for candidate in sorted(STATES):
+            key = labels(dict(identity, signal=signal, state=candidate))
+            lines.append(f'railiance_assurance_state{key} {int(state == candidate)}')
+    lines.extend([
+        '# HELP railiance_assurance_observed_timestamp_seconds Original producer evaluation time.',
+        '# TYPE railiance_assurance_observed_timestamp_seconds gauge',
+        f'railiance_assurance_observed_timestamp_seconds{labels(identity)} {observed}',
+    ])
+    return '\n'.join(lines) + '\n'
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument('--contract', required=True, type=Path)
+    parser.add_argument('report', type=Path)
+    args = parser.parse_args()
+    try:
+        output = render(read_json(args.report), read_json(args.contract), datetime.now(timezone.utc))
+    except (OSError, ValueError, KeyError, TypeError, AttributeError):
+        print(json.dumps({'status': 'rejected', 'error': 'invalid-or-expired-report'}), file=sys.stderr)
+        return 2
+    sys.stdout.write(output)
+    return 0
+
+
+if __name__ == '__main__':
+    sys.exit(main())

tests/prometheus-rules.json

@@ -0,0 +1,207 @@
+{
+  "rule_files": [
+    "../rules/platform-assurance.json"
+  ],
+  "evaluation_interval": "1m",
+  "tests": [
+    {
+      "name": "never seen",
+      "interval": "1m",
+      "input_series": [],
+      "alert_rule_test": [
+        {
+          "eval_time": "0m",
+          "alertname": "RailianceAssuranceEmissionAbsent",
+          "exp_alerts": [
+            {
+              "exp_labels": {
+                "stream": "railiance-platform.service-assurance",
+                "producer": "railiance-platform",
+                "owner": "railiance-telemetry",
+                "severity": "warning"
+              }
+            }
+          ]
+        },
+        {
+          "eval_time": "0m",
+          "alertname": "RailianceAssuranceUnhealthy",
+          "exp_alerts": []
+        }
+      ]
+    },
+    {
+      "name": "healthy fresh",
+      "interval": "1m",
+      "input_series": [
+        {
+          "series": "railiance_assurance_observed_timestamp_seconds{stream=\"railiance-platform.service-assurance\",producer=\"railiance-platform\"}",
+          "values": "0+60x20"
+        }
+      ],
+      "alert_rule_test": [
+        {
+          "eval_time": "20m",
+          "alertname": "RailianceAssuranceEmissionAbsent",
+          "exp_alerts": []
+        },
+        {
+          "eval_time": "20m",
+          "alertname": "RailianceAssuranceUnhealthy",
+          "exp_alerts": []
+        }
+      ]
+    },
+    {
+      "name": "budget boundary",
+      "interval": "1m",
+      "input_series": [
+        {
+          "series": "railiance_assurance_observed_timestamp_seconds{stream=\"railiance-platform.service-assurance\",producer=\"railiance-platform\"}",
+          "values": "0x16"
+        }
+      ],
+      "alert_rule_test": [
+        {
+          "eval_time": "15m",
+          "alertname": "RailianceAssuranceEmissionAbsent",
+          "exp_alerts": []
+        },
+        {
+          "eval_time": "15m",
+          "alertname": "RailianceAssuranceUnhealthy",
+          "exp_alerts": []
+        }
+      ]
+    },
+    {
+      "name": "stopped producer still scraped",
+      "interval": "1m",
+      "input_series": [
+        {
+          "series": "railiance_assurance_observed_timestamp_seconds{stream=\"railiance-platform.service-assurance\",producer=\"railiance-platform\"}",
+          "values": "0x16"
+        }
+      ],
+      "alert_rule_test": [
+        {
+          "eval_time": "16m",
+          "alertname": "RailianceAssuranceEmissionAbsent",
+          "exp_alerts": [
+            {
+              "exp_labels": {
+                "stream": "railiance-platform.service-assurance",
+                "producer": "railiance-platform",
+                "owner": "railiance-telemetry",
+                "severity": "warning"
+              }
+            }
+          ]
+        },
+        {
+          "eval_time": "16m",
+          "alertname": "RailianceAssuranceUnhealthy",
+          "exp_alerts": []
+        }
+      ]
+    },
+    {
+      "name": "future clock",
+      "interval": "1m",
+      "input_series": [
+        {
+          "series": "railiance_assurance_observed_timestamp_seconds{stream=\"railiance-platform.service-assurance\",producer=\"railiance-platform\"}",
+          "values": "600"
+        }
+      ],
+      "alert_rule_test": [
+        {
+          "eval_time": "0m",
+          "alertname": "RailianceAssuranceEmissionAbsent",
+          "exp_alerts": [
+            {
+              "exp_labels": {
+                "stream": "railiance-platform.service-assurance",
+                "producer": "railiance-platform",
+                "owner": "railiance-telemetry",
+                "severity": "warning"
+              }
+            }
+          ]
+        },
+        {
+          "eval_time": "0m",
+          "alertname": "RailianceAssuranceUnhealthy",
+          "exp_alerts": []
+        }
+      ]
+    },
+    {
+      "name": "scrape disappears",
+      "interval": "1m",
+      "input_series": [
+        {
+          "series": "railiance_assurance_observed_timestamp_seconds{stream=\"railiance-platform.service-assurance\",producer=\"railiance-platform\"}",
+          "values": "0 stale"
+        }
+      ],
+      "alert_rule_test": [
+        {
+          "eval_time": "6m",
+          "alertname": "RailianceAssuranceEmissionAbsent",
+          "exp_alerts": [
+            {
+              "exp_labels": {
+                "stream": "railiance-platform.service-assurance",
+                "producer": "railiance-platform",
+                "owner": "railiance-telemetry",
+                "severity": "warning"
+              }
+            }
+          ]
+        },
+        {
+          "eval_time": "6m",
+          "alertname": "RailianceAssuranceUnhealthy",
+          "exp_alerts": []
+        }
+      ]
+    },
+    {
+      "name": "failed renewal",
+      "input_series": [
+        {
+          "series": "railiance_assurance_observed_timestamp_seconds{stream=\"railiance-platform.service-assurance\",producer=\"railiance-platform\"}",
+          "values": "0"
+        },
+        {
+          "series": "railiance_assurance_state{stream=\"railiance-platform.service-assurance\",producer=\"railiance-platform\",signal=\"eso.token-renewal\",state=\"failed\"}",
+          "values": "1"
+        }
+      ],
+      "alert_rule_test": [
+        {
+          "eval_time": "0m",
+          "alertname": "RailianceAssuranceEmissionAbsent",
+          "exp_alerts": []
+        },
+        {
+          "eval_time": "0m",
+          "alertname": "RailianceAssuranceUnhealthy",
+          "exp_alerts": [
+            {
+              "exp_labels": {
+                "stream": "railiance-platform.service-assurance",
+                "producer": "railiance-platform",
+                "owner": "railiance-telemetry",
+                "severity": "warning",
+                "signal": "eso.token-renewal",
+                "state": "failed"
+              }
+            }
+          ]
+        }
+      ]
+    }
+  ]
+}

tests/test_prometheus_export.py

@@ -0,0 +1,57 @@
+from datetime import datetime, timedelta, timezone
+import copy
+import json
+from pathlib import Path
+import subprocess
+import sys
+import unittest
+
+ROOT = Path(__file__).resolve().parents[1]
+sys.path.insert(0, str(ROOT / 'scripts'))
+from prometheus_export import labels, render
+
+
+class ExportTests(unittest.TestCase):
+    def setUp(self):
+        self.now = datetime(2026, 9, 28, tzinfo=timezone.utc)
+        self.contract = json.loads((ROOT / 'contracts/platform-assurance-2026-09-28.json').read_text())
+        self.report = dict(schema='railiance-platform.assurance-signal.v1',
+            cluster_uid='a553c742-0115-43d4-99a4-a5ca56fe0786',
+            evaluated_at=self.now.isoformat(),
+            signals={s: dict(state='healthy', owner='railiance-platform') for s in self.contract['signals']},
+            transport='unmonitored', guarantees='unsupported',
+            threshold_status='local-diagnostic-only', healthy=True)
+
+    def test_all_states_preserved_including_token_renewal(self):
+        for state in ('healthy', 'failed', 'missing', 'unavailable', 'stale'):
+            self.report['signals']['eso.token-renewal']['state'] = state
+            self.report['healthy'] = state == 'healthy'
+            output = render(self.report, self.contract, self.now)
+            samples = [line for line in output.splitlines() if line.startswith('railiance_assurance_state')]
+            self.assertEqual(len(samples), 120)
+            self.assertEqual(sum(line.endswith(' 1') for line in samples), 24)
+            self.assertIn(f'signal="eso.token-renewal",state="{state}"}} 1', output)
+
+    def test_retry_preserves_time_and_does_not_mutate_report(self):
+        before = copy.deepcopy(self.report)
+        self.assertEqual(render(self.report, self.contract, self.now),
+                         render(self.report, self.contract, self.now + timedelta(seconds=900)))
+        self.assertEqual(self.report, before)
+        for delta in (-1, 901):
+            with self.assertRaises(ValueError):
+                render(self.report, self.contract, self.now + timedelta(seconds=delta))
+
+    def test_scope_drift_is_rejected(self):
+        self.report['signals']['unexpected'] = dict(state='healthy', owner='railiance-platform')
+        with self.assertRaises(ValueError): render(self.report, self.contract, self.now)
+
+    def test_label_escaping(self):
+        self.assertEqual(labels({'signal': 'a"b\\c\nd'}), '{signal="a\\"b\\\\c\\nd"}')
+
+    def test_cli_failure_has_no_metrics_stdout(self):
+        result = subprocess.run([sys.executable, str(ROOT / 'scripts/prometheus_export.py'),
+            '--contract', str(ROOT / 'contracts/platform-assurance-2026-09-28.json'),
+            str(ROOT / 'tests/nonexistent-report.json')], capture_output=True, text=True)
+        self.assertEqual(result.returncode, 2)
+        self.assertEqual(result.stdout, '')
+        self.assertIn('invalid-or-expired-report', result.stderr)

workplans/RTEL-WP-0002-signal-contract.md

@@ -4,11 +4,11 @@ type: workplan
 title: "Provide the Q2 receiving contract and prove signal delivery"
 domain: financials
 repo: railiance-telemetry
-status: active
+status: blocked
 flavor: implementation
 owner: codex
 created: "2026-09-06"
-updated: "2026-09-06"
+updated: "2026-09-28"
 related:
   - RTELE-WP-0001
   - RPF-WP-0036
@@ -96,3 +96,19 @@ package install, readiness and authenticated exposure. This T04 retains Q2's
 exporter/rule mapping and actual delivery/absence acceptance. Existing SQLite
 runtime and disabled activities are reference work, not the production service.
 No package activation or Q2 delivery acceptance is claimed by repo creation.
+
+September 28 loose-end review: completed the local Prometheus text exporter and
+native-tested failure/absence rules under this task. A separate dated contract
+includes platform's added `eso.token-renewal` signal without silently migrating
+existing SQLite bindings. Source classifications and evaluation time survive
+export; stale/future reports fail. Nineteen Python tests, seven native Prometheus
+rule scenarios and metric lint pass. See `docs/prometheus-mapping.md` and
+`history/2026-09-28-loose-end-review.md`.
+
+Package T03 is already done (private installation and attended restore proof).
+T04 remains `wait` and this workplan is now `blocked`: rapp-telemetry T04 still
+owes accepted scrape/executor binding, a confirmed recipient/channel, actual
+failure/absence acknowledgments, an outside-node watchdog and recurring backup
+ownership. The founder, Bernd Worsch, supplies recipient/admission decisions;
+rapp-telemetry and platform own runtime/custody integration. No additional task
+or workplan was opened, and no live schedule or notification was enabled.