coulomb-social/docs/release/backup-alert-recurring-integration-2026-10-31.md

Recurring backup alert integration — source candidate

Both destinations are accepted and their one-off tests received: platform-root's State Hub inbox and bernd.worsch@gmail.com. The founder confirmed Gmail inbox receipt; the exact proof Job and SMTP receipt were independently verified. That does not activate recurring delivery.

The consumer now has a durable outbox and source wiring for recurring delivery. It reuses State Hub and the existing Alertmanager/SMTP projection; it adds no mail service, mailbox credential or public endpoint. Deployment remains pending.

Consumer behavior

scripts/backup_alerts.py locks a private journal, writes and fsyncs each event before a send, and checkpoints each channel separately. A failed State Hub call does not prevent the email handoff, and vice versa. Replays retain the UUID. State Hub delivery requires the exact central message receipt; a queue receipt is insufficient. A lost HTTP reply can produce a duplicate inbox message with the same event ID: this is at-least-once delivery, not exactly-once delivery.

Alertmanager submission uses its v2 API. The consumer reads back exact labels/times and the expected email receiver for active events. Millisecond timestamps match the native API's serialization. Acceptance is an email-transport handoff, never a recipient acknowledgment. Alertmanager owns SMTP retry/grouping/silencing. No SMTP password enters this consumer process, State Hub, output or repository.

observe-backup-alerts.py calls the existing complete health check. Pending and stale reasons create their respective alert episodes; other evidence/capacity failures create backup_failed. Repeated unhealthy observations refresh the same episode; a later recurrence after recovery gets a new UUID. The existing hourly freshness unit drives observation and replay. Its exit status measures whether monitoring/delivery succeeded; backup health remains explicit in its JSON.

Both the paired-backup and freshness units have an OnFailure template handler. The handler validates systemd's MONITOR_UNIT, MONITOR_SERVICE_RESULT and MONITOR_INVOCATION_ID, and uses the failed invocation UUID for replay. A later healthy report clears explicit failed-run events only if its fully verified paired capture is newer than the failure. An old healthy backup cannot erase a new failed-run alert. No automatic restore, capture replay or deletion is added.

Active alerts have a three-hour lease, refreshed hourly. Resolved emails explicitly say that lease expiry or an explicit clear is not recovery proof. Independent producer/notification-debt monitoring remains an operating requirement: a dead producer cannot monitor itself. Native delivery and receipt tests must include producer interruption, notification backlog and channel outage cases.

The separate private state directory is /var/lib/coulomb-social-backup-alerts. Records are limited to 16 KiB each and 1,024 events; pressure at 75% produces a health alert. Confirmed records are retained, not automatically deleted. This bounds committed record count, not filesystem allocation or abandoned temporary files after a crash. Reviewed archival/capacity handling is still required. A full journal refuses new events while explicit replay of existing events remains available. Processing rotates attempts to prevent bounded service executions from permanently starving later records.

Email route and ownership

deploy/backup/alertmanager-config.pending.json is a consumer route proposal, not a live Helm values file. It selects only this application's three backup conditions and platform-root owner, leaving unrelated alerts on the existing pending receiver. SMTP uses the existing telemetry-alert-smtp/password projection, sender platform@coulomb.social, verified TLS and the chosen Gmail recipient. It includes resolved notifications so short incidents can still be reported. The pending generic telemetry acknowledgment service is not required or activated by this backup route; recipient receipt is not audit acknowledgment.

The observed live service is telemetry-alertmanager, ClusterIP 10.43.37.119, port 9093; GET /api/v2/receivers returned only pending-operator-channel. The route is therefore not active. alerting.env.example records candidate private endpoints and contains no credential. Recheck their identities before installation. rapp-telemetry owns the exact Helm merge, existing-secret mount, current release diff, route activation and rollback. Preserve all other admitted routes and mounted secrets when preparing that exact owner operation.

Verification and remaining execution

Full make check: 460 passed, zero failures/errors/skips, including the mandatory browser journeys. The 14 new outbox/producer checks cover independent failures, queue receipts, replay, recovery, journal pressure and systemd invocation binding. backup-alert-recurring-sources-2026-10-31.json binds the checked sources.

Native amtool validated the exact route configuration and all three positive conditions plus four unrelated labelsets. Its container had no network. backup-alert-route-validation-2026-10-31.json binds the image and config hash. The actual target systemd 255 verified all five candidate units from temporary files, then removed those files. Return code was zero; warnings concerned two unrelated unreadable identity timers. Nothing was installed or enabled. backup-alert-unit-validation-2026-10-31.json binds the exact unit files.

An isolated real Alertmanager process accepted/read back a synthetic event and delivered an email to a loopback-only SMTP fixture. Its overrides replaced recipient/sender with example.invalid addresses, SMTP with a local listener without authentication/TLS, and grouping delays with short fixture values. This proves API, routing, template and message correlation, not live TLS/custody. The previous founder test proves the live SMTP projection separately. Receipt and reproduction source are backup-alert-isolated-smtp-2026-10-31.json and .rehearsal.txt. The local process, copied binary and state were removed.

Before activation, prepare and review the exact owner route/mount change and rebuilt operating bundle, install consumer units disabled, and verify live failed/pending/stale test events in both channels. Include channel failure, restart/replay and producer-death detection evidence. Preserve stable test IDs and do not replay the already completed one-off email Job. The existing capacity, rolling retention, historical-copy disposition and service-contract obligations remain required before the already approved schedule is enabled. WP-0012-T03 remains in progress; this candidate does not complete WP-0013/14/15 acceptance.