# Acknowledgment maintenance and recovery

These operations belong to RTEL-WP-0002-T04. Run with the database owner's OS
identity, against a private owned directory (0700) and database (0600). No
credential or browser login is needed for filesystem administration. This is
not a public HTTP administration interface. Database access confers access to
acknowledgment evidence; restrict it accordingly.

## Inspect and retry audit delivery

```sh
python3 scripts/alert_admin.py --database /state/private/ack.sqlite status
python3 scripts/alert_admin.py --database /state/private/ack.sqlite list-blocked
python3 scripts/alert_admin.py --database /state/private/ack.sqlite requeue --event-id EVENT_UUID
```

Status reports pending, blocked and delivered counts. Listing returns at most
100 blocked event IDs (override with `--limit`, maximum 1000); it never prints
actor identities, credentials or event payloads. Repair the receiver schema,
authorization or custody fault before explicitly requeuing a particular event.
Requeue only changes blocked → pending. The running worker retries within its
next pass using the original event ID and exact payload. It cannot rewrite an
acknowledgment or declare delivery successful. Exit 2 means the selected event
was not blocked (including unknown or already delivered); exit 1 is a refusal.

Private GET `/metrics` exports:

- `railiance_ack_audit_events{status="pending|blocked|delivered"}` (all three
  series, including zero; the label takes one of the three individual values).
- `railiance_ack_audit_last_drain_timestamp_seconds` (zero before the first pass).

No actor, alert or event IDs are metric labels. A completed local drain does not
prove an idle audit receiver is reachable. Keep this endpoint on the private
Service: the public Ingress remains `/ack/` only. Package integration must admit
the Prometheus scraper through network policy and configure its scrape and
alerts. Alert on blocked debt, sustained pending debt and a stale worker;
independent receiver readback/reconciliation remains an activation gate.

## Backup and restore

```sh
python3 scripts/alert_admin.py --database /state/private/ack.sqlite backup --destination /backup/private/ack-snapshot.sqlite
python3 scripts/alert_admin.py --database /backup/private/ack-snapshot.sqlite restore --destination /state/private/ack-restored.sqlite
```

Create destination directories beforehand with mode 0700 and the service's
owner. Backup uses SQLite's online backup API, so it captures committed
acknowledgments and their outbox atomically even while the service is running.
A successful command validates SQLite integrity, foreign keys, required
immutability triggers and acknowledgment/outbox correspondence, then fsyncs the
snapshot. Existing destinations, symlinks and unsafe permissions are refused.
Treat interrupted commands as unverified; retain and inspect their destination
before choosing a new path. These checks detect structural problems, not
malicious alteration by an administrator.

For recovery, stop the runtime and retain the current database first. Restore
to a **new path**, change the package configuration to that path, then restart.
Do not run original and restored databases as competing writers. Credentials
and sessions are not in this database; users sign in again. Acknowledgments made
after the selected backup are outside its recovery point: reconcile with
independent audit evidence before opening the recovered service. A snapshot
alone cannot reconstruct those later acknowledgments.

The restored store preserves pending, blocked and delivered states, original
actors/decisions and event IDs. Already delivered events are not resent; an
event accepted by audit-core whose reply was lost is replayed with its same ID
and deduplicated by the receiver. Repeated human confirmation preserves the
first acknowledgment. Tests exercise both cases, including real audit-core.

Off-host copying, retention, encryption, backup scheduling and a native restore
drill remain package/platform responsibilities under the existing task.