Add audit maintenance, verified recovery and reproducible verification

Assistant: codex
Assistant-Model: gpt-6-astra
Assistant-Session: 01a0e6f1-443f-7783-9920-a16b2ffc467f

.gitignore

@@ -10,3 +10,6 @@ __pycache__/
 *.db-journal
 *.db-wal
 *.db-shm
+
+.venv-verify/
+.cache/

README.md

@@ -30,7 +30,15 @@ carries and surfaces their output.
 Quality dimension, Q2 Observability, as recorded in INTENT/SCOPE. Monitoring
 workload packaging remains outside this repo.
 
-Run `python3 -m unittest discover -s tests -v` (Python 3.11+, no dependencies).
+Run `python3 -m unittest discover -s tests -v` for the dependency-free core
+suite (native receiver check is optional there). For the full core/runtime and
+native owner-contract suite, run **`bash scripts/verify.sh`** with Python 3.11+,
+uv, Go and sibling `audit-core` / `flex-auth` checkouts. The script installs the
+hash-locked runtime dependencies in `.venv-verify`, builds Flex Auth into
+`.cache/verify`, and fails if either native source is absent. Override checkout
+locations with `RTEL_AUDIT_CORE_SOURCE` and `RTEL_FLEX_AUTH_SOURCE`. It reports
+the tested owner revisions; those source checkouts are not fetched or changed.
+This command tests local contracts, not live infrastructure or Prometheus rules.
 Live acceptance is tracked by RTEL-WP-0002-T04; local receipts are not external
 operator delivery proof.
 
@@ -51,3 +59,6 @@ not activated. Identity/policy adapters are implemented; native package admissio
 webhook/audit custody and audited alert confirmation remain under T04. SMTP
 custody and delivery are verified, including Bernd’s confirmed inbox receipt
 of the transport-test email.
+
+[Maintenance and recovery](docs/acknowledgment-operations.md) covers private audit
+metrics, explicit event requeue and verified backup/restore to a new database.

docs/acknowledgment-operations.md

@@ -0,0 +1,71 @@
+# Acknowledgment maintenance and recovery
+
+These operations belong to RTEL-WP-0002-T04. Run with the database owner's OS
+identity, against a private owned directory (0700) and database (0600). No
+credential or browser login is needed for filesystem administration. This is
+not a public HTTP administration interface. Database access confers access to
+acknowledgment evidence; restrict it accordingly.
+
+## Inspect and retry audit delivery
+
+```sh
+python3 scripts/alert_admin.py --database /state/private/ack.sqlite status
+python3 scripts/alert_admin.py --database /state/private/ack.sqlite list-blocked
+python3 scripts/alert_admin.py --database /state/private/ack.sqlite requeue --event-id EVENT_UUID
+```
+
+Status reports pending, blocked and delivered counts. Listing returns at most
+100 blocked event IDs (override with `--limit`, maximum 1000); it never prints
+actor identities, credentials or event payloads. Repair the receiver schema,
+authorization or custody fault before explicitly requeuing a particular event.
+Requeue only changes blocked → pending. The running worker retries within its
+next pass using the original event ID and exact payload. It cannot rewrite an
+acknowledgment or declare delivery successful. Exit 2 means the selected event
+was not blocked (including unknown or already delivered); exit 1 is a refusal.
+
+Private GET `/metrics` exports:
+
+- `railiance_ack_audit_events{status="pending|blocked|delivered"}` (all three
+  series, including zero; the label takes one of the three individual values).
+- `railiance_ack_audit_last_drain_timestamp_seconds` (zero before the first pass).
+
+No actor, alert or event IDs are metric labels. A completed local drain does not
+prove an idle audit receiver is reachable. Keep this endpoint on the private
+Service: the public Ingress remains `/ack/` only. Package integration must admit
+the Prometheus scraper through network policy and configure its scrape and
+alerts. Alert on blocked debt, sustained pending debt and a stale worker;
+independent receiver readback/reconciliation remains an activation gate.
+
+## Backup and restore
+
+```sh
+python3 scripts/alert_admin.py --database /state/private/ack.sqlite backup --destination /backup/private/ack-snapshot.sqlite
+python3 scripts/alert_admin.py --database /backup/private/ack-snapshot.sqlite restore --destination /state/private/ack-restored.sqlite
+```
+
+Create destination directories beforehand with mode 0700 and the service's
+owner. Backup uses SQLite's online backup API, so it captures committed
+acknowledgments and their outbox atomically even while the service is running.
+A successful command validates SQLite integrity, foreign keys, required
+immutability triggers and acknowledgment/outbox correspondence, then fsyncs the
+snapshot. Existing destinations, symlinks and unsafe permissions are refused.
+Treat interrupted commands as unverified; retain and inspect their destination
+before choosing a new path. These checks detect structural problems, not
+malicious alteration by an administrator.
+
+For recovery, stop the runtime and retain the current database first. Restore
+to a **new path**, change the package configuration to that path, then restart.
+Do not run original and restored databases as competing writers. Credentials
+and sessions are not in this database; users sign in again. Acknowledgments made
+after the selected backup are outside its recovery point: reconcile with
+independent audit evidence before opening the recovered service. A snapshot
+alone cannot reconstruct those later acknowledgments.
+
+The restored store preserves pending, blocked and delivered states, original
+actors/decisions and event IDs. Already delivered events are not resent; an
+event accepted by audit-core whose reply was lost is replayed with its same ID
+and deduplicated by the receiver. Repeated human confirmation preserves the
+first acknowledgment. Tests exercise both cases, including real audit-core.
+
+Off-host copying, retention, encryption, backup scheduling and a native restore
+drill remain package/platform responsibilities under the existing task.

docs/alert-acknowledgment.md

@@ -159,9 +159,14 @@ email does not establish those remaining facts.
 Runtime validation (hash-pinned dependencies in requirements-runtime.lock):
 
 ```bash
-RTEL_FLEX_AUTH_BINARY=/tmp/rtel-flex-auth /tmp/rtel-ack-venv/bin/python -m unittest discover -s tests_runtime -v
+bash scripts/verify.sh
 ```
 
 The candidate workload, container recipe and runtime settings are owned by
 `rapp-telemetry/acknowledgment`. The policy source and exact client registration
 are in `integration/`; they are not active registrations.
+
+See [maintenance and recovery](acknowledgment-operations.md) for aggregate
+metrics, explicit blocked-event retries, and backup/restore. The full verification
+command requires the documented sibling owner checkouts; it never silently skips
+the native contract tests.

history/2026-09-28-acknowledgment-operations.md

@@ -0,0 +1,23 @@
+# Acknowledgment operations closure — 2026-09-28
+
+Existing RTEL-WP-0002-T04, no new task/workplan. Added private Prometheus metrics
+for pending/blocked/delivered outbox counts and last completed drain. No event
+or actor labels. Database failures in the drain are retried instead of killing
+the worker. Idle receiver health is explicitly not inferred from a drain.
+
+Local admin CLI provides counts, bounded blocked-event IDs, explicit requeue and
+verified SQLite backup/restore to a new path. No overwrite or public admin API.
+Existing acknowledgment/event bytes are preserved by retry and restore.
+
+Full verification command: bash scripts/verify.sh. Hash-locked runtime packages
+installed into .venv-verify, native Flex Auth built in .cache/verify. Native owner
+checkouts required; missing dependencies fail rather than silently skip tests.
+37 core + 8 runtime tests pass. Actual audit-core receiver accepted an event,
+lost its response, then deduplicated replay after backup and restore. Snapshot
+tests retain all delivery states, repeated confirmation identity and immutable
+triggers; unsafe paths and overwrite are refused.
+
+Tested audit-core 3e42ca8ffb2ead14fd52f9a6f9f42800f1dc2771 and Flex Auth
+2dfee5782ec9010758af9ba5a6f72d9fa0fe6234. No new live acknowledgment activation;
+remaining native identity/PDP/audit, scrape, off-host and recipient gates stay
+in the existing blocked workplan. Package image updated separately.

scripts/alert_ack.py

@@ -214,6 +214,19 @@ class Store:
             return {row['status']: row['n'] for row in db.execute(
                 "SELECT status,COUNT(*) AS n FROM outbox WHERE status!='delivered' GROUP BY status")}
 
+    def audit_metrics(self, last_drain):
+        """Private aggregate metrics: no event, actor, or alert labels."""
+        with self.connect() as db:
+            counts = dict(db.execute('SELECT status,COUNT(*) FROM outbox GROUP BY status'))
+        lines = ['# HELP railiance_ack_audit_events Stored audit events by delivery state.',
+                 '# TYPE railiance_ack_audit_events gauge']
+        for status in ('pending', 'blocked', 'delivered'):
+            lines.append(f'railiance_ack_audit_events{{status="{status}"}} {counts.get(status, 0)}')
+        lines += ['# HELP railiance_ack_audit_last_drain_timestamp_seconds Last completed local outbox pass; not receiver reachability.',
+                  '# TYPE railiance_ack_audit_last_drain_timestamp_seconds gauge',
+                  f'railiance_ack_audit_last_drain_timestamp_seconds {float(last_drain)}']
+        return '\n'.join(lines) + '\n'
+
 
 class Application:
     """WSGI adapter. authenticate and authorize are required, never default-allow.

scripts/alert_admin.py

@@ -0,0 +1,130 @@
+#!/usr/bin/env python3
+"""Local maintenance of the private acknowledgment store; no HTTP admin route."""
+import argparse
+import json
+import os
+from pathlib import Path
+import sqlite3
+import stat
+
+from alert_ack import Store
+
+TRIGGERS = {'ack_immutable_update', 'ack_immutable_delete', 'outbox_body_immutable',
+            'outbox_immutable_delete', 'alert_immutable_update', 'alert_immutable_delete'}
+
+
+def private_path(path, existing=True):
+    path = Path(path).absolute()
+    parent = path.parent.lstat()
+    if not stat.S_ISDIR(parent.st_mode) or parent.st_uid != os.getuid() or parent.st_mode & 0o077:
+        raise ValueError('private owned directory required')
+    if existing:
+        info = path.lstat()
+        if (not stat.S_ISREG(info.st_mode) or info.st_uid != os.getuid()
+                or info.st_nlink != 1 or info.st_mode & 0o077):
+            raise ValueError('private owned database required')
+    return path
+
+
+def readonly(path):
+    return sqlite3.connect(private_path(path).as_uri() + '?mode=ro', uri=True)
+
+
+def validate(db):
+    if db.execute('PRAGMA integrity_check').fetchall() != [('ok',)] or db.execute('PRAGMA foreign_key_check').fetchall():
+        raise ValueError('database integrity check failed')
+    triggers = {r[0] for r in db.execute("SELECT name FROM sqlite_master WHERE type='trigger'")}
+    if not TRIGGERS <= triggers:
+        raise ValueError('acknowledgment schema required')
+    # Every durable acknowledgment has exactly its original outbox event.
+    if db.execute('''SELECT COUNT(*) FROM acknowledgments a LEFT JOIN outbox o
+                     ON a.event_id=o.event_id WHERE o.event_id IS NULL''').fetchone()[0]:
+        raise ValueError('incomplete audit outbox')
+    if db.execute('''SELECT COUNT(*) FROM outbox o LEFT JOIN acknowledgments a
+                     ON a.event_id=o.event_id WHERE a.event_id IS NULL
+                     OR o.status NOT IN ('pending','blocked','delivered')''').fetchone()[0]:
+        raise ValueError('invalid audit outbox')
+
+
+def snapshot(source, destination):
+    """SQLite online backup, never overwrite; restore uses the same fresh-target path.
+
+    Source may be live for backup. Restore to a NEW path, then switch the stopped
+    service to it. Neither operation rewinds or replaces an existing database.
+    """
+    target = private_path(destination, existing=False)
+    source_db = readonly(source)
+    created = False
+    try:
+        fd = os.open(target, os.O_CREAT | os.O_EXCL | os.O_WRONLY | os.O_NOFOLLOW, 0o600)
+        os.close(fd)
+        created = True
+        target_db = sqlite3.connect(target)
+        try:
+            source_db.backup(target_db)
+            validate(target_db)
+        finally:
+            target_db.close()
+        with target.open('rb') as saved:
+            os.fsync(saved.fileno())
+        directory = os.open(target.parent, os.O_RDONLY | os.O_DIRECTORY)
+        try:
+            os.fsync(directory)
+        finally:
+            os.close(directory)
+    except Exception:
+        if created:
+            target.unlink()
+        raise
+    finally:
+        source_db.close()
+
+
+def main(argv=None):
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument('--database', required=True, type=Path)
+    commands = parser.add_subparsers(dest='action', required=True)
+    commands.add_parser('status')
+    listing = commands.add_parser('list-blocked')
+    listing.add_argument('--limit', type=int, default=100)
+    retry = commands.add_parser('requeue')
+    retry.add_argument('--event-id', required=True)
+    for action in ('backup', 'restore'):
+        command = commands.add_parser(action)
+        command.add_argument('--destination', type=Path, required=True)
+    args = parser.parse_args(argv)
+    try:
+        if args.action in ('backup', 'restore'):
+            snapshot(args.database, args.destination)
+            result = {'status': 'verified', 'action': args.action}
+        else:
+            db = readonly(args.database)
+            try:
+                validate(db)
+                counts = dict(db.execute('SELECT status,COUNT(*) FROM outbox GROUP BY status'))
+                if args.action == 'status':
+                    result = {'outbox': {s: counts.get(s, 0) for s in ('pending','blocked','delivered')}}
+                elif args.action == 'list-blocked':
+                    if not 1 <= args.limit <= 1000:
+                        raise ValueError('limit must be 1..1000')
+                    result = {'event_ids': [r[0] for r in db.execute(
+                        "SELECT event_id FROM outbox WHERE status='blocked' ORDER BY event_id LIMIT ?", (args.limit,))]}
+                else:
+                    # Requeue only, never manufacture delivery or change evidence.
+                    store = Store(args.database)
+                    changed = store.requeue(args.event_id)
+                    result = {'event_id': args.event_id, 'requeued': changed}
+                    if not changed:
+                        print(json.dumps(result))
+                        return 2
+            finally:
+                db.close()
+        print(json.dumps(result, sort_keys=True))
+        return 0
+    except (OSError, ValueError, sqlite3.Error):
+        print(json.dumps({'status': 'refused', 'reason': 'maintenance failed; check paths, permissions and database integrity'}))
+        return 1
+
+
+if __name__ == '__main__':
+    raise SystemExit(main())

scripts/alert_service.py

@@ -7,6 +7,7 @@ import os
 from pathlib import Path
 import secrets
 import signal
+import sqlite3
 import threading
 import time
 from urllib.parse import parse_qs
@@ -61,6 +62,11 @@ class Router:
             return ('Set-Cookie', f'{name}={value}; Path=/; Secure; HttpOnly; SameSite=Lax; Max-Age={age}')
         path, method = env.get('PATH_INFO'), env.get('REQUEST_METHOD')
         try:
+            if method == 'GET' and path == '/metrics':
+                body = self.store.audit_metrics(self.last_drain)
+                start('200 OK', [('Content-Type', 'text/plain; version=0.0.4; charset=utf-8'),
+                                 ('Cache-Control', 'no-store')])
+                return [body.encode()]
             if method == 'GET' and path in ('/healthz', '/readyz'):
                 ready = path == '/healthz' or time.time() - self.last_drain < 90 and not self.store.audit_debt()
                 return reply('200 OK' if ready else '503 Service Unavailable', 'ready' if ready else 'audit delivery pending')
@@ -87,7 +93,7 @@ class Router:
             if path == '/webhook':
                 self.application.webhook_token = credential(self.config['webhook_token_file'])
             return self.application(env, start)
-        except (ValueError, OSError, KeyError, TypeError):
+        except (ValueError, OSError, KeyError, TypeError, sqlite3.Error):
             return reply('503 Service Unavailable', 'Identity or service unavailable. Retry later.')
 
 
@@ -109,7 +115,7 @@ def main():
             try:
                 store.drain(transport)
                 router.last_drain = time.time()
-            except (OSError, ValueError):
+            except (OSError, ValueError, sqlite3.Error):
                 router.last_drain = 0
             stop.wait(30)
     worker = threading.Thread(target=drain, daemon=True)

scripts/verify.sh

@@ -0,0 +1,26 @@
+#!/usr/bin/env bash
+# Full Python and native owner-contract suite; fail if native dependencies are absent.
+set -euo pipefail
+cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.."
+repo_root="$PWD"
+audit_source="${RTEL_AUDIT_CORE_SOURCE:-$repo_root/../audit-core}"
+policy_source="${RTEL_FLEX_AUTH_SOURCE:-$repo_root/../flex-auth}"
+if [[ ! -f "$audit_source/audit_core/ingestion.py" || ! -f "$policy_source/go.mod" ]]; then
+  echo 'Required: sibling audit-core and flex-auth checkouts, or RTEL_AUDIT_CORE_SOURCE / RTEL_FLEX_AUTH_SOURCE.' >&2
+  exit 1
+fi
+mkdir -p .cache/verify
+export UV_CACHE_DIR="$repo_root/.cache/verify/uv"
+export GOCACHE="$repo_root/.cache/verify/go-build"
+if [[ ! -x .venv-verify/bin/python ]]; then
+  uv venv --python python3 .venv-verify
+fi
+uv pip sync --python .venv-verify/bin/python --require-hashes requirements-runtime.lock
+go -C "$policy_source" build -mod=readonly -o "$repo_root/.cache/verify/flex-auth" ./cmd/flex-auth
+export RTEL_AUDIT_CORE_SOURCE="$audit_source"
+export RTEL_FLEX_AUTH_BINARY="$repo_root/.cache/verify/flex-auth"
+echo 'Native owner source revisions:'
+git -C "$audit_source" rev-parse HEAD
+git -C "$policy_source" rev-parse HEAD
+.venv-verify/bin/python -m unittest discover -s tests -v
+.venv-verify/bin/python -m unittest discover -s tests_runtime -v

tests/test_alert_admin.py

@@ -0,0 +1,92 @@
+from contextlib import redirect_stdout
+import io
+import json
+from pathlib import Path
+import sqlite3
+import tempfile
+import unittest
+
+from test_alert_ack import NOW, START
+from alert_ack import Actor, Store
+from alert_admin import main, snapshot
+
+
+class MaintenanceTests(unittest.TestCase):
+    def setUp(self):
+        self.temp = tempfile.TemporaryDirectory()
+        self.addCleanup(self.temp.cleanup)
+        self.root = Path(self.temp.name)
+        self.path = self.root / 'source.db'
+        self.store = Store(self.path)
+        self.actor = Actor('https://fixture.example', 'private-human', 'tenant:platform',
+                           ('railiance-admin',), NOW+900, 'c'*32)
+        self.ids = []
+        self.events = []
+        for i in range(3):
+            identity = self.store.receive({'version':'4', 'receiver':'railiance-admin-email', 'alerts':[
+                {'status':'firing','fingerprint':f'{i:016x}','startsAt':START,
+                 'labels':{'alertname':'Fixture','owner':'railiance-telemetry'}}]},NOW)[0]
+            self.ids.append(identity)
+            self.events.append(self.store.acknowledge(identity,self.actor,'decision',NOW))
+        def send(event):
+            if event['id'] == self.events[0]: return 202, {'status':'accepted','reference':'audit:'+event['id']}
+            if event['id'] == self.events[1]: return 403, {}
+            raise TimeoutError()
+        self.store.drain(send)
+
+    def cli(self,*args):
+        output=io.StringIO()
+        with redirect_stdout(output): code=main(['--database',str(self.path),*args])
+        return code,json.loads(output.getvalue())
+
+    def test_status_and_explicit_requeue_preserve_evidence(self):
+        self.assertEqual(self.cli('status'),(0,{'outbox':{'pending':1,'blocked':1,'delivered':1}}))
+        self.assertEqual(self.cli('list-blocked')[1]['event_ids'],[self.events[1]])
+        with self.store.connect() as db: before=db.execute('SELECT event_id,body FROM outbox ORDER BY event_id').fetchall()
+        self.assertTrue(self.cli('requeue','--event-id',self.events[1])[1]['requeued'])
+        self.assertEqual(self.cli('requeue','--event-id',self.events[0])[0],2)
+        with self.store.connect() as db: after=db.execute('SELECT event_id,body FROM outbox ORDER BY event_id').fetchall()
+        self.assertEqual([tuple(r) for r in before],[tuple(r) for r in after])
+
+    def test_backup_restore_preserves_all_states_identity_and_immutability(self):
+        backup=self.root/'snapshot.db'; restored=self.root/'restored.db'
+        snapshot(self.path,backup)
+        snapshot(backup,restored)
+        store=Store(restored)
+        for identity,event in zip(self.ids,self.events):
+            self.assertEqual(store.acknowledge(identity,self.actor,'new-decision',NOW+5),event)
+        self.assertEqual([store.get(i)['audit_status'] for i in self.ids],['delivered','blocked','pending'])
+        with store.connect() as db:
+            with self.assertRaises(sqlite3.IntegrityError):db.execute('DELETE FROM acknowledgments')
+            with self.assertRaises(sqlite3.IntegrityError):db.execute("UPDATE outbox SET body='{}'")
+        self.assertEqual(restored.stat().st_mode & 0o777,0o600)
+        self.assertTrue(store.requeue(self.events[1]))
+        sent=[]
+        def duplicate(event):
+            sent.append(event['id']);return 200,{'status':'duplicate','reference':'audit:'+event['id']}
+        store.drain(duplicate)
+        self.assertEqual(set(sent),set(self.events[1:]))
+        self.assertFalse(store.audit_debt())
+        self.assertEqual(self.store.audit_debt(),{'blocked':1,'pending':1})
+
+    def test_refuse_overwrite_missing_input_bad_schema_and_symlink(self):
+        with self.assertRaises(FileExistsError):snapshot(self.path,self.path)
+        missing=self.root/'missing.db'
+        with redirect_stdout(io.StringIO()):self.assertEqual(main(['--database',str(missing),'status']),1)
+        self.assertFalse(missing.exists())
+        invalid=self.root/'invalid.db';invalid.write_bytes(b'bad')
+        invalid.chmod(0o600)
+        with self.assertRaises(sqlite3.DatabaseError):snapshot(invalid,self.root/'bad-backup.db')
+        self.assertFalse((self.root/'bad-backup.db').exists())
+        link=self.root/'link.db';link.symlink_to(self.path)
+        with self.assertRaises(ValueError):snapshot(link,self.root/'link-backup.db')
+
+    def test_metrics_are_aggregate_and_show_empty_states(self):
+        metrics=self.store.audit_metrics(123)
+        for state in ('pending','blocked','delivered'):
+            self.assertIn(f'{{status="{state}"}} 1',metrics)
+        self.assertIn('timestamp_seconds 123.0',metrics)
+        self.assertNotIn(self.actor.subject,metrics)
+        for event in self.events:self.assertNotIn(event,metrics)
+        empty=Store(self.root/'empty.db').audit_metrics(0)
+        self.assertIn('{status="pending"} 0',empty)

tests/test_alert_audit_native.py

@@ -9,6 +9,7 @@ import unittest
 
 sys.path.insert(0, str(Path(__file__).resolve().parents[1] / 'scripts'))
 from alert_ack import Actor, Store
+from alert_admin import snapshot
 
 
 @unittest.skipUnless(os.environ.get('RTEL_AUDIT_CORE_SOURCE'), 'set RTEL_AUDIT_CORE_SOURCE for real receiver check')
@@ -42,7 +43,9 @@ class NativeAuditTests(unittest.TestCase):
                     raise TimeoutError('simulated lost response')
                 return response['status'], json.loads(body)
             store.drain(send)
-            store = Store(Path(tmp) / 'ack.db')
+            snapshot(store.path, Path(tmp) / 'backup.db')
+            snapshot(Path(tmp) / 'backup.db', Path(tmp) / 'restored.db')
+            store = Store(Path(tmp) / 'restored.db')
             store.drain(send)
             self.assertEqual(calls, [202, 200])
             self.assertEqual(store.get(identity)['audit_status'], 'delivered')

tests_runtime/test_service.py

@@ -55,3 +55,18 @@ class ServiceTests(unittest.TestCase):
             response, _ = call('/ack/logout', 'POST', session_cookie, query='', body=form)
             self.assertEqual(response['status'], '200 OK')
             self.assertIsNone(login.session(session_cookie.split('=', 1)[1]))
+
+    def test_private_metrics_route_without_identity_or_event_labels(self):
+        with tempfile.TemporaryDirectory() as tmp:
+            token=Path(tmp)/'token';token.write_text('w'*32)
+            store=Store(Path(tmp)/'state.db')
+            router=Router({'origin':'https://telemetry.example','webhook_token_file':str(token)},
+                          store,login=object(),policy=lambda *args: None)
+            router.last_drain=123
+            result={}
+            output=b''.join(router({'PATH_INFO':'/metrics','REQUEST_METHOD':'GET'},
+                                  lambda status,headers:result.update(status=status,headers=dict(headers))))
+            self.assertEqual(result['status'],'200 OK')
+            self.assertIn('version=0.0.4',result['headers']['Content-Type'])
+            self.assertIn(b'{status="blocked"} 0',output)
+            self.assertIn(b'timestamp_seconds 123.0',output)

workplans/RTEL-WP-0002-signal-contract.md

@@ -154,3 +154,13 @@ evidence, along with browser client/PDP/audit admission and other T04 gates.
 Bernd Worsch confirmed the transport-test email arrived in his inbox. SMTP
 credential custody, projection, authentication and recipient delivery now pass.
 This is a transport-test receipt, not an audit-core alert acknowledgment.
+
+September 28 local operations closure: private aggregate audit metrics and local
+status/list-blocked/requeue commands implemented. SQLite online backup and
+restore-to-new-path preserve original receipts, immutable event bytes and all
+delivery states; native audit-core deduplicates replay after lost reply and
+restore. `bash scripts/verify.sh` installs locked dependencies and builds the
+native policy test dependency from declared checkouts; all 45 tests pass without
+skips. See docs/acknowledgment-operations.md. No new task/workplan. T04 remains
+wait and workplan blocked for live identity/PDP/audit admission, scrape binding,
+recipient alert confirmation/readback, off-host watchdog and scheduled backups.