railiance-telemetry/docs/alert-acknowledgment.md

# Email receipt acknowledgment

The founder authorized email to `bernd.worsch@gmail.com`, a Railiance admin role
filled by that user, controlled failure/absence drills and acknowledgment audit
records on September 28. This work remains under RTEL-WP-0002-T04; no new task
or workplan is needed. The machine-readable record is
`contracts/alert-acknowledgment.json`.

## User flow

Alertmanager sends an email containing a link to one alert occurrence. The link
opens a review page at `https://telemetry.coulomb.social/ack/alerts`. Sign-in uses
the existing estate identity provider. A Railiance admin presses **Acknowledge
receipt**; the page confirms local recording and separately reports whether
audit-core has accepted its audit event. Clicking again returns the same record.
An email scanner or preview GET never acknowledges anything. Acknowledgment
does not resolve the fault, silence Alertmanager or suppress its repeat emails.

The link contains the Alertmanager fingerprint and original start timestamp,
not a bearer credential. A fresh occurrence needs a fresh acknowledgment, even
if its labels/fingerprint are identical. The timestamp template preserves
RFC3339 nanoseconds. The corresponding webhook must arrive first; an early
click reports that the alert has not arrived and can be retried.

## Implemented component

`scripts/alert_ack.py` supplies a WSGI component with authenticated `/webhook`
and protected GET/POST `/ack/alerts` routes. It requires server-side identity
and authorization adapters at construction. `alert_service.py` supplies the
Waitress runtime, `alert_identity.py` supplies OIDC code/PKCE sessions, and
`alert_policy.py` validates native Flex Auth decisions. There is no anonymous
mode, trusted-email/header fallback or default allow decision.

The private SQLite store commits the first human acknowledgment and exact
audit envelope in one transaction. Ordinary updates/deletes of acknowledgments,
alerts and audit payloads are refused. This does not protect against a database
administrator. Duplicate webhook calls and clicks preserve the original fact.
Only bounded alert name, fingerprint, start time and a digest of labels persist;
annotations and arbitrary alert details are not copied into email or audit.

`scripts/alert_audit.py` supplies a bounded, redirect-refusing sender for
`POST /v1/events`, with stable `Idempotency-Key`. Credential provision remains
external. Exact `202 accepted` or `200 duplicate` and `audit:<event-id>` are
required for delivery completion. Lost replies retain the event for replay;
schema/auth/conflict refusals retain it as blocked. The package must expose
pending/blocked debt and explicitly requeue a blocked row after repair. It must
schedule bounded drains and per-class reconciliation/heartbeat before claiming
live audit operation. The runtime drains every 30 seconds and reports readiness false while audit
debt remains. Per-class reconciliation/heartbeat and independent readback remain
activation gates. No retention deletion is installed.

## Identity and authorization binding still required

The email address is the notification destination. NetKingdom records associate
it with directory username `tegwick`; the explicit `railiance-admins` membership was applied using the native
identity-provisioner, preserving other memberships. KeyCape commit `1164f65`
was built, published and deployed successfully. A real login must still verify
the signed `(issuer, subject)` and role claim. Do not grant on an email match, domain match or client-supplied
header. Do not automatically equate `net-kingdom-admins` with `railiance-admin`.
The requested role is consumed here for `read` and `acknowledge` on platform
telemetry alerts; this integration grants no unrelated estate privileges.

The package must bind a registered OIDC Authorization Code + PKCE session with
verified signature, issuer, audience, nonce, expiry, human provenance and MFA.
Keep tokens server-side; use Secure/HttpOnly/SameSite cookies and logout/expiry.
Populate `Actor` only from that verified session, with a random session-bound
CSRF value. The component separately enforces human/platform/role constraints,
session expiry, exact form Origin and CSRF. These are additional restrictions,
not a substitute for the policy decision.

The Flex Auth adapter must request a fresh decision for every read or acknowledge,
with system `railiance-telemetry`, resource type `telemetry-alert`, resource
`alert:<occurrence-uuid>`, tenant `tenant:platform` and verified subject facts.
It must validate the native decision contract, exact actor/resource/action
binding, submitted request digest, admitted package/version/digest, lifetime
and supported obligations before returning the component's bounded receipt.
Both adapters are implemented. Signed RSA issuer fixtures exercise the browser
flow; the native Flex Auth evaluator validates exact request digests and rejects
wrong identities, missing roles/groups and stale MFA. These tests use synthetic
credentials and do not prove live authentication. The package must still admit
the OIDC client and enforced Kubernetes TokenReview caller.

## Concrete activation packet for existing owners

| Owner | Required binding |
| --- | --- |
| NetKingdom / key-cape | Register `railiance-telemetry-admin` and exact proposed callback `/ack/auth/callback`; verify Bernd's subject; apply and verify `railiance-admin` membership and MFA claims. |
| flex-auth | Admit workload caller, telemetry-alert resource and read/acknowledge package/assignments; return pinned package/version/digest and positive/negative fixtures. |
| railiance-platform | Provision dedicated SMTP, webhook and audit-sender custody through approved lanes; do not extract or copy email-connect's invitation credential into this app. |
| audit-core | Register source `railiance-telemetry`, exact tenant `tenant:platform`, write-only, `secret_policy=redact`, proposed load-bearing class; admit ingress and independent readback. |
| rapp-telemetry | Package session/PDP adapters, audit drain/debt monitoring and private persistent state; route `/ack/` separately from Grafana, keep `/webhook` private; integrate SMTP/template/webhook and verify restart/restore. |

Use Alertmanager's native email integration with recipient
`bernd.worsch@gmail.com`, From `platform@coulomb.social`, template `railiance.alert.email` from
`templates/alert-email.tmpl` and receiver name `railiance-admin-email`.
The same receiver's webhook posts to the private acknowledgment service using
a dedicated credential file. Route only alerts with `owner=railiance-telemetry`
until other inventories are reviewed; the receiver rejects other scopes.
The package overlay has this shape (the private service and mounted credential
paths are candidates, not deployed resources):

```yaml
templates:
  - /etc/alertmanager/templates/alert-email.tmpl
receivers:
  - name: railiance-admin-email
    email_configs:
      - to: bernd.worsch@gmail.com
        from: platform@coulomb.social
        require_tls: true
        text: '{{ template "railiance.alert.email" . }}'
        send_resolved: true
    webhook_configs:
      - url: http://telemetry-ack.telemetry.svc.cluster.local:8080/webhook
        send_resolved: true
        http_config:
          authorization:
            type: Bearer
            credentials_file: /etc/alertmanager/secrets/telemetry-ack/webhook-token
```

The overlay must retain existing routes and supply approved global SMTP
smarthost/from/auth settings. Do not enable it with an unimplemented browser
adapter or assume the credential file/service already exists.
Use SMTP STARTTLS and mounted credential files through the package-owned secret
references. The existing email-connect lane proves IONOS is available but does
not admit a new SMTP consumer. No existing Secret was read during this work.

The first live drill must retain email identifiers, alert occurrences, Bernd's
explicit acknowledgment events, exact audit references and independent readback.
Run a controlled failure and stopped-producer case, plus unauthorized account,
expired login, scanner GET, duplicate click and audit-unavailable checks.
Outside-node monitoring and recurring backups remain the existing T04 gates.

## Verification

```bash
RTEL_AUDIT_CORE_SOURCE=/home/worsch/audit-core python3 -m unittest discover -s tests -v
amtool template render --template.glob=templates/alert-email.tmpl \
  --template.text='{{ template "railiance.alert.email" . }}'
```

The optional receiver test uses actual audit-core ingestion and SQLite with
synthetic credentials: first accepted, lost reply, then duplicate after restart.
It is not production custody proof. The template follows the upstream
[notification data contract](https://prometheus.io/docs/alerting/latest/notifications/).
The founder created `platform@coulomb.social`. The reviewed platform helper
creates `platform/workloads/railiance-telemetry/smtp` without a password through
attended OpenBao login. The founder then adds `SMTP_PASSWORD` as a new version,
preserving existing public SMTP fields.
Live activation remains blocked on password provisioning, dedicated credential custody,
client/caller admission, runtime rollout and recipient/readback drills.
No email or production acknowledgment is claimed.

Runtime validation (hash-pinned dependencies in requirements-runtime.lock):

```bash
RTEL_FLEX_AUTH_BINARY=/tmp/rtel-flex-auth /tmp/rtel-ack-venv/bin/python -m unittest discover -s tests_runtime -v
```

The candidate workload, container recipe and runtime settings are owned by
`rapp-telemetry/acknowledgment`. The policy source and exact client registration
are in `integration/`; they are not active registrations.