railiance-telemetry/docs/acknowledgment-operations.md

Acknowledgment maintenance and recovery

These operations belong to RTEL-WP-0002-T04. Run with the database owner's OS identity, against a private owned directory (0700) and database (0600). No credential or browser login is needed for filesystem administration. This is not a public HTTP administration interface. Database access confers access to acknowledgment evidence; restrict it accordingly.

Inspect and retry audit delivery

python3 scripts/alert_admin.py --database /state/private/ack.sqlite status
python3 scripts/alert_admin.py --database /state/private/ack.sqlite list-blocked
python3 scripts/alert_admin.py --database /state/private/ack.sqlite requeue --event-id EVENT_UUID

Status reports pending, blocked and delivered counts. Listing returns at most 100 blocked event IDs (override with --limit, maximum 1000); it never prints actor identities, credentials or event payloads. Repair the receiver schema, authorization or custody fault before explicitly requeuing a particular event. Requeue only changes blocked → pending. The running worker retries within its next pass using the original event ID and exact payload. It cannot rewrite an acknowledgment or declare delivery successful. Exit 2 means the selected event was not blocked (including unknown or already delivered); exit 1 is a refusal.

Private GET /metrics exports:

• railiance_ack_audit_events{status="pending|blocked|delivered"} (all three series, including zero; the label takes one of the three individual values).
• railiance_ack_audit_last_drain_timestamp_seconds (zero before the first pass).

No actor, alert or event IDs are metric labels. A completed local drain does not prove an idle audit receiver is reachable. Keep this endpoint on the private Service: the public Ingress remains /ack/ only. Package integration must admit the Prometheus scraper through network policy and configure its scrape and alerts. Alert on blocked debt, sustained pending debt and a stale worker; independent receiver readback/reconciliation remains an activation gate.

Backup and restore

python3 scripts/alert_admin.py --database /state/private/ack.sqlite backup --destination /backup/private/ack-snapshot.sqlite
python3 scripts/alert_admin.py --database /backup/private/ack-snapshot.sqlite restore --destination /state/private/ack-restored.sqlite

Create destination directories beforehand with mode 0700 and the service's owner. Backup uses SQLite's online backup API, so it captures committed acknowledgments and their outbox atomically even while the service is running. A successful command validates SQLite integrity, foreign keys, required immutability triggers and acknowledgment/outbox correspondence, then fsyncs the snapshot. Existing destinations, symlinks and unsafe permissions are refused. Treat interrupted commands as unverified; retain and inspect their destination before choosing a new path. These checks detect structural problems, not malicious alteration by an administrator.

For recovery, stop the runtime and retain the current database first. Restore to a new path, change the package configuration to that path, then restart. Do not run original and restored databases as competing writers. Credentials and sessions are not in this database; users sign in again. Acknowledgments made after the selected backup are outside its recovery point: reconcile with independent audit evidence before opening the recovered service. A snapshot alone cannot reconstruct those later acknowledgments.

The restored store preserves pending, blocked and delivered states, original actors/decisions and event IDs. Already delivered events are not resent; an event accepted by audit-core whose reply was lost is replayed with its same ID and deduplicated by the receiver. Repeated human confirmation preserves the first acknowledgment. Tests exercise both cases, including real audit-core.

Off-host copying, retention, encryption, backup scheduling and a native restore drill remain package/platform responsibilities under the existing task.