AUDIT-WP-0009 T04/T06/T07 — heartbeats, reconciliation, and a home for findings

The detection half audit-core argued up to a MUST and then could not support.
Two registered sources were waiting on it.

T04, heartbeats. A heartbeat is an ordinary event — same envelope, same
append-only custody, same chain, no special table. Deliberate: a heartbeat
stored outside the chain would be the one record here that could be back-dated.
Declared per class rather than per source, because a per-source heartbeat from
a mixed-volume emitter is satisfied by its chattiest class and says nothing
about the quiet, security-relevant one, which is the only reason heartbeats
exist. Not the §17 cadence schema T05 waits on: cadence describes expected
rate, this says how often a source promises to say "nothing to report" for a
class that may legitimately be silent.

no_heartbeat_since_registration is its own finding kind rather than a skip —
it is the case most likely to be a broken integration and the one a naive
"compare against last seen" implementation silently drops. Grace widens the
window so one late run does not flap; it never removes a finding.

T06, reconciliation. Counts, never payloads. The awkward part is that every
registered sender holds may_read: false, which taken literally makes the §9.6
reconciliation obligation undischargeable by every source actually registered.
Resolved by observing that a source asking how many of its own events we hold
is not reading the archive — it learns nothing it did not itself emit. So the
surface is scoped to the caller's own sources and tenants and returns no
payloads; anything wider stays behind may_read and full tenant scope. Another
source's counts return 403 rather than an empty count, because a zero would
read as "we hold none of yours" — a false answer to a question about
completeness. No default window, since a count whose bounds the caller did not
choose is not comparable to anything the caller computed.

T07, the findings surface. /v1/stream-findings, following the dead-letter and
secret-finding conventions: may_read plus full tenant scope, since findings
span every sender and carry no tenant key to filter on.

The bound is on every response rather than in a document nobody opens beside
it. A missing heartbeat is not proof of suppression, and agreement on counts
proves neither completeness nor that any event occurred. Both controls cover
loss, outage, drain failure and accident; neither covers a source lying about
itself, and where the emitter is compromised both agree with it. Closing that
needs an observer independent of the emitter, which §16 put outside our scope.

The scope overlay may shorten a heartbeat interval or add a class, never
lengthen or remove one — same asymmetry as evidence_kind, and for the same
reason: a ConfigMap refresh must not widen the window in which a suppressed
class goes unnoticed without anyone deciding to.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nb7Q6ZmXppNDkTWytfYqfv

Assistant: claude-code
Assistant-Model: opus
Assistant-Process: 2069992@bnt-lap001
Assistant-Session: 167dd7f8-2a25-4be1-aa46-3b6f1a5f94c6

audit_core/ingestion.py

@@ -46,7 +46,7 @@ from audit_core.redaction import (
     apply_policy,
     finding_from_path,
 )
-from audit_core.senders import SenderIdentity, SenderRegistry, development_registry
+from audit_core.senders import WILDCARD, SenderIdentity, SenderRegistry, development_registry
 from audit_core.sqlite_backend import SQLiteAuditBackend
 
 MAX_BODY_BYTES = 256 * 1024
@@ -164,10 +164,21 @@ class IngestionApplication:
                 "/v1/secret-findings",
                 "/v1/stats",
                 "/v1/integrity",
+                "/v1/stream-findings",
             )
         ):
             return self._read(start_response, environ, path, identity)
 
+        # Reconciliation is deliberately routed BEFORE the may_read gate.
+        # A source asking how many of its own events audit-core holds is not
+        # reading the archive — it learns nothing it did not itself emit — and
+        # §9.6 makes that comparison the source's own detection obligation. A
+        # writer with may_read: false must therefore be able to ask, or the
+        # obligation audit-core argued for is undischargeable by every sender
+        # actually registered. Cross-source counts stay behind may_read.
+        if method == "GET" and path == "/v1/reconciliation":
+            return self._reconciliation(start_response, environ, identity)
+
         if path != "/v1/events" or method != "POST":
             return self._json(start_response, HTTPStatus.NOT_FOUND, {"error": "not_found"})
 
@@ -264,6 +275,8 @@ class IngestionApplication:
                 )
             if path == "/v1/stats":
                 return self._json(start_response, HTTPStatus.OK, self.counters.snapshot())
+            if path == "/v1/stream-findings":
+                return self._stream_findings(start_response)
             if path == "/v1/secret-findings":
                 return self._json(
                     start_response, HTTPStatus.OK,
@@ -309,6 +322,97 @@ class IngestionApplication:
                 start_response, HTTPStatus.SERVICE_UNAVAILABLE, {"error": "backend_unavailable"}
             )
 
+    def _reconciliation(self, start_response, environ, identity):
+        """Per-class counts of one source's own events (AUDIT-WP-0009-T06).
+
+        Counts, never payloads, and never another source's. The comparison a
+        source makes against its own state transitions is the §9.6 detection
+        obligation; audit-core supplies the numerator and says nothing about
+        what the answer means.
+        """
+        query = parse_qs(environ.get("QUERY_STRING", ""))
+        source = (query.get("source") or [""])[0]
+        if not source:
+            return self._json(
+                start_response, HTTPStatus.BAD_REQUEST, {"error": "source_required"}
+            )
+        if not identity.permits_source(source):
+            # 403 rather than an empty count. A zero here would read as "we
+            # hold none of yours", which is a materially different and false
+            # answer to a question about completeness.
+            return self._json(
+                start_response, HTTPStatus.FORBIDDEN, {"error": "source_not_allowed"}
+            )
+        window = _window(query)
+        if window is None:
+            return self._json(
+                start_response, HTTPStatus.BAD_REQUEST,
+                {"error": "since_and_until_required"},
+            )
+        since, until = window
+        tenant = (query.get("tenant") or [""])[0] or None
+        if tenant is not None and not identity.permits_tenant(tenant):
+            return self._json(
+                start_response, HTTPStatus.FORBIDDEN, {"error": "tenant_not_allowed"}
+            )
+        if tenant is None and not identity.has_full_tenant_scope():
+            # A scoped credential must name the tenant it is counting, so the
+            # answer cannot silently aggregate across a boundary it may not see.
+            return self._json(
+                start_response, HTTPStatus.BAD_REQUEST, {"error": "tenant_required"}
+            )
+        counter = getattr(self.backend, "event_counts", None)
+        if not callable(counter):
+            return self._json(
+                start_response, HTTPStatus.NOT_FOUND,
+                {"error": "reconciliation_not_supported"},
+            )
+        try:
+            counts = counter(source, since, until, tenant)
+        except BackendUnavailableError as exc:
+            log.error("reconciliation failed: %s", exc)
+            return self._json(
+                start_response, HTTPStatus.SERVICE_UNAVAILABLE,
+                {"error": "backend_unavailable"},
+            )
+        return self._json(start_response, HTTPStatus.OK, {
+            "source": source,
+            "tenant": tenant,
+            "since": since,
+            "until": until,
+            "counts": counts,
+            # Said in the response because this is the number most likely to
+            # be quoted out of context in someone else's conformance argument.
+            "means": (
+                "the count of events audit-core accepted and stored in this "
+                "window. Divergence from the source's own count is a finding "
+                "for the source; agreement proves neither completeness nor "
+                "that any event occurred."
+            ),
+        })
+
+    def _stream_findings(self, start_response):
+        """Missing-heartbeat findings (AUDIT-WP-0009-T04, T07)."""
+        from audit_core.stream_findings import evaluate
+
+        reader = getattr(self.backend, "last_heartbeats", None)
+        if not callable(reader):
+            return self._json(
+                start_response, HTTPStatus.NOT_FOUND,
+                {"error": "stream_findings_not_supported"},
+            )
+        last: dict[tuple[str, str], str | None] = {}
+        for identity in self.senders.identities:
+            for source in identity.sources:
+                if source == WILDCARD:
+                    continue
+                for event_class, at in reader(source).items():
+                    last[(source, event_class)] = at
+        findings = evaluate(self.senders.identities, last)
+        return self._json(start_response, HTTPStatus.OK, {
+            "stream_findings": [finding.as_dict() for finding in findings],
+        })
+
     def _count_secrets(self, payload: Any, identity, outcome: str, findings) -> None:
         """Count secret-shaped fields by path, sender, source and action.
 
@@ -472,9 +576,32 @@ _UNSCOPED_READ_PATHS = frozenset({
     "/v1/stats",
     "/v1/secret-findings",
     "/v1/integrity",
+    # Findings span every registered sender and carry no tenant key, so there
+    # is nothing to filter on. A scoped reader is refused rather than served
+    # instance-wide facts — the same rule as the other three.
+    "/v1/stream-findings",
 })
 
 
+def _window(query: dict[str, list[str]]) -> tuple[str, str] | None:
+    """Require an explicit bounded window for a reconciliation query.
+
+    No default window. A count whose bounds the caller did not choose is not
+    comparable against anything the caller computed, and would be quoted as
+    though it were.
+    """
+    since = (query.get("since") or [""])[0]
+    until = (query.get("until") or [""])[0]
+    if not since or not until:
+        return None
+    try:
+        start = _normalize_timestamp(since)
+        end = _normalize_timestamp(until)
+    except ValueError:
+        return None
+    return (start, end) if start < end else None
+
+
 def _readable_by(identity, record: dict) -> bool:
     """Whether ``identity`` may read ``record``.