railiance-telemetry/docs/alert-acknowledgment.md

Email receipt acknowledgment

The founder authorized email to bernd.worsch@gmail.com, a Railiance admin role filled by that user, controlled failure/absence drills and acknowledgment audit records on September 28. This work remains under RTEL-WP-0002-T04; no new task or workplan is needed. The machine-readable record is contracts/alert-acknowledgment.json.

User flow

Alertmanager sends an email containing a link to one alert occurrence. The link opens a review page at https://telemetry.coulomb.social/ack/alerts. Sign-in uses the existing estate identity provider. A Railiance admin presses Acknowledge receipt; the page confirms local recording and separately reports whether audit-core has accepted its audit event. Clicking again returns the same record. An email scanner or preview GET never acknowledges anything. Acknowledgment does not resolve the fault, silence Alertmanager or suppress its repeat emails.

The link contains the Alertmanager fingerprint and original start timestamp, not a bearer credential. A fresh occurrence needs a fresh acknowledgment, even if its labels/fingerprint are identical. The timestamp template preserves RFC3339 nanoseconds. The corresponding webhook must arrive first; an early click reports that the alert has not arrived and can be retried.

Implemented component

scripts/alert_ack.py supplies a WSGI component with authenticated /webhook and protected GET/POST /ack/alerts routes. It requires server-side identity and authorization adapters at construction. alert_service.py supplies the Waitress runtime, alert_identity.py supplies OIDC code/PKCE sessions, and alert_policy.py validates native Flex Auth decisions. There is no anonymous mode, trusted-email/header fallback or default allow decision.

The private SQLite store commits the first human acknowledgment and exact audit envelope in one transaction. Ordinary updates/deletes of acknowledgments, alerts and audit payloads are refused. This does not protect against a database administrator. Duplicate webhook calls and clicks preserve the original fact. Only bounded alert name, fingerprint, start time and a digest of labels persist; annotations and arbitrary alert details are not copied into email or audit.

scripts/alert_audit.py supplies a bounded, redirect-refusing sender for POST /v1/events, with stable Idempotency-Key. Credential provision remains external. Exact 202 accepted or 200 duplicate and audit:<event-id> are required for delivery completion. Lost replies retain the event for replay; schema/auth/conflict refusals retain it as blocked. The package must expose pending/blocked debt and explicitly requeue a blocked row after repair. It must schedule bounded drains and per-class reconciliation/heartbeat before claiming live audit operation. The runtime drains every 30 seconds and reports readiness false while audit debt remains. Per-class reconciliation/heartbeat and independent readback remain activation gates. No retention deletion is installed.

Identity and authorization binding still required

The email address is the notification destination. NetKingdom records associate it with directory username tegwick; the explicit railiance-admins membership was applied using the native identity-provisioner, preserving other memberships. KeyCape commit 1164f65 was built, published and deployed successfully. A real login must still verify the signed (issuer, subject) and role claim. Do not grant on an email match, domain match or client-supplied header. Do not automatically equate net-kingdom-admins with railiance-admin. The requested role is consumed here for read and acknowledge on platform telemetry alerts; this integration grants no unrelated estate privileges.

The package must bind a registered OIDC Authorization Code + PKCE session with verified signature, issuer, audience, nonce, expiry, human provenance and MFA. Keep tokens server-side; use Secure/HttpOnly/SameSite cookies and logout/expiry. Populate Actor only from that verified session, with a random session-bound CSRF value. The component separately enforces human/platform/role constraints, session expiry, exact form Origin and CSRF. These are additional restrictions, not a substitute for the policy decision.

The Flex Auth adapter must request a fresh decision for every read or acknowledge, with system railiance-telemetry, resource type telemetry-alert, resource alert:<occurrence-uuid>, tenant tenant:platform and verified subject facts. It must validate the native decision contract, exact actor/resource/action binding, submitted request digest, admitted package/version/digest, lifetime and supported obligations before returning the component's bounded receipt. Both adapters are implemented. Signed RSA issuer fixtures exercise the browser flow; the native Flex Auth evaluator validates exact request digests and rejects wrong identities, missing roles/groups and stale MFA. These tests use synthetic credentials and do not prove live authentication. The package must still admit the OIDC client and enforced Kubernetes TokenReview caller.

Concrete activation packet for existing owners

Owner	Required binding
NetKingdom / key-cape	Register railiance-telemetry-admin and exact proposed callback /ack/auth/callback; verify Bernd's subject; apply and verify railiance-admin membership and MFA claims.
flex-auth	Admit workload caller, telemetry-alert resource and read/acknowledge package/assignments; return pinned package/version/digest and positive/negative fixtures.
railiance-platform	Provision dedicated SMTP, webhook and audit-sender custody through approved lanes; do not extract or copy email-connect's invitation credential into this app.
audit-core	Register source railiance-telemetry, exact tenant tenant:platform, write-only, secret_policy=redact, proposed load-bearing class; admit ingress and independent readback.
rapp-telemetry	Package session/PDP adapters, audit drain/debt monitoring and private persistent state; route /ack/ separately from Grafana, keep /webhook private; integrate SMTP/template/webhook and verify restart/restore.

Use Alertmanager's native email integration with recipient bernd.worsch@gmail.com, From platform@coulomb.social, template railiance.alert.email from templates/alert-email.tmpl and receiver name railiance-admin-email. The same receiver's webhook posts to the private acknowledgment service using a dedicated credential file. Route only alerts with owner=railiance-telemetry until other inventories are reviewed; the receiver rejects other scopes. The package overlay has this shape (the private service and mounted credential paths are candidates, not deployed resources):

templates:
  - /etc/alertmanager/templates/alert-email.tmpl
receivers:
  - name: railiance-admin-email
    email_configs:
      - to: bernd.worsch@gmail.com
        from: platform@coulomb.social
        require_tls: true
        text: '{{ template "railiance.alert.email" . }}'
        send_resolved: true
    webhook_configs:
      - url: http://telemetry-ack.telemetry.svc.cluster.local:8080/webhook
        send_resolved: true
        http_config:
          authorization:
            type: Bearer
            credentials_file: /etc/alertmanager/secrets/telemetry-ack/webhook-token

The overlay must retain existing routes and supply approved global SMTP smarthost/from/auth settings. Do not enable it with an unimplemented browser adapter or assume the credential file/service already exists. Use SMTP STARTTLS and mounted credential files through the package-owned secret references. The existing email-connect lane proves IONOS is available but does not admit a new SMTP consumer. No existing Secret was read during this work.

The first live drill must retain email identifiers, alert occurrences, Bernd's explicit acknowledgment events, exact audit references and independent readback. Run a controlled failure and stopped-producer case, plus unauthorized account, expired login, scanner GET, duplicate click and audit-unavailable checks. Outside-node monitoring and recurring backups remain the existing T04 gates.

Verification

RTEL_AUDIT_CORE_SOURCE=/home/worsch/audit-core python3 -m unittest discover -s tests -v
amtool template render --template.glob=templates/alert-email.tmpl \
  --template.text='{{ template "railiance.alert.email" . }}'

The optional receiver test uses actual audit-core ingestion and SQLite with synthetic credentials: first accepted, lost reply, then duplicate after restart. It is not production custody proof. The template follows the upstream notification data contract. The founder created platform@coulomb.social. The reviewed platform helper creates platform/workloads/railiance-telemetry/smtp without a password through attended OpenBao login. The founder then adds SMTP_PASSWORD as a new version, preserving existing public SMTP fields. Live activation remains blocked on password provisioning, dedicated credential custody, client/caller admission, runtime rollout and recipient/readback drills. No email or production acknowledgment is claimed.

Runtime validation (hash-pinned dependencies in requirements-runtime.lock):

RTEL_FLEX_AUTH_BINARY=/tmp/rtel-flex-auth /tmp/rtel-ack-venv/bin/python -m unittest discover -s tests_runtime -v

The candidate workload, container recipe and runtime settings are owned by rapp-telemetry/acknowledgment. The policy source and exact client registration are in integration/; they are not active registrations.