audit-core/docs/stream-completeness.md

# Stream completeness: heartbeats, reconciliation, and findings

`AUDIT-WP-0009` T04, T05, T06, T07. Statute §9.6.

The chain proves records held were not altered or truncated. It says nothing
about a record that never arrived — and §9.6 is explicit that **omission** is
the acute risk for exactly the rare negative classes (revocation, denial,
containment) where suppression is most valuable and least visible.

audit-core argued that obligation up to a MUST. A source cannot declare a
cadence to a system with nowhere to put it, so this is the surface owed.

## Three surfaces, and what each actually covers

| Surface | Covers | Does not cover |
| --- | --- | --- |
| Heartbeat (`POST /v1/events`, class `audit-core.heartbeat`) | A class going silent — including one that is *legitimately* silent, which rate monitoring can never distinguish | A compromised source emitting a truthful-looking heartbeat while suppressing the event |
| Reconciliation (`GET /v1/reconciliation`) | Loss, outage, drain failure — divergence between what a source emitted and what arrived | A compromised source suppressing the event and its own count together |
| Declared rate (`emission_cadence` on the registration) | A high-volume class falling below the rate its source declared | A class that may legitimately be silent — that is the heartbeat's job; and a rate held proves nothing about completeness |
| Findings (`GET /v1/stream-findings`) | Surfacing the above where an operator sees them | Anything the two above do not detect |

**The bound is the same in both rows and it is not a footnote.** Where the
emitter itself is compromised, both controls agree with it. They cover loss,
outage, drain failure and accident — most of what actually goes wrong — and not
a source lying about itself. Closing that needs an observer independent of the
emitter, which §16 placed outside audit-core's scope. No conformance claim may
read these surfaces as covering adversarial omission by the source.

## Heartbeats

A heartbeat is an **ordinary event**: same envelope, same append-only custody,
same chain, no special table. That is deliberate — a heartbeat stored outside
the chain would be the one record in this store that could be back-dated.

```json
{
  "id": "...", "type": "audit-core.heartbeat",
  "source": "approval-engine", "subject": "approval-engine",
  "tenant": "tenant:platform", "correlation_id": "...",
  "occurred_at": "2026-09-10T03:00:00+00:00",
  "data": {"class": "approval.revocation", "assertion": "nothing-to-report"}
}
```

### Declared per class, never per source

`heartbeat_classes` on the sender registration maps event class → the longest
gap in seconds that is not yet a finding.

Per class is the whole point. A per-source heartbeat from a mixed-volume
emitter is satisfied by its chattiest class and says nothing about the quiet,
security-relevant one — and the quiet class is the only reason heartbeats
exist. `informed-decision` raised this shape first for presentations versus
dispositions; it generalises.

### Heartbeat and cadence are complementary

A heartbeat is a registration property saying how often a source promises to
say *nothing to report* for a class that may legitimately be silent. Cadence
describes a stream's expected **rate**. They are not alternatives, and the rate
shape is not audit-core's — see [Declared emission cadence](#declared-emission-cadence).

### Findings

`GET /v1/stream-findings` requires `may_read` **and** full tenant scope:
findings span every registered sender and carry no tenant key, so there is
nothing to filter on and a scoped reader is refused rather than served
instance-wide facts. Same rule as dead letters, secret findings and integrity.

Two kinds:

- `no_heartbeat_since_registration` — declared a heartbeat, never sent one.
  Its own kind rather than skipped, because it is the case most likely to be a
  broken integration and the one a naive "compare against last seen"
  implementation silently drops.
- `missing_heartbeat` — one arrived once and is now overdue, with how late.

A grace factor of 1.5 widens the window so a single late run does not flap a
finding on and off — the same reasoning as the attestation freshness window. It
widens; it never removes.

Every finding carries a `means` field saying that absence of a heartbeat is not
proof of suppression. That travels on the finding rather than in a document
nobody opens alongside it.

## Reconciliation

`GET /v1/reconciliation?source=&tenant=&since=&until=` returns per-class counts
of a source's own events. **Counts, never payloads.**

### Why a writer with `may_read: false` may call it

Every registered sender holds `may_read: false` — a source does not gain a read
surface by emitting. Taken literally that would make the §9.6 reconciliation
obligation undischargeable by every source actually registered, which is a
strange place for a rule audit-core argued for to end up.

The resolution is that **a source asking how many of its own events audit-core
holds is not reading the archive**. It learns nothing it did not itself emit.
So the surface is scoped to the caller's own permitted sources and tenants, and
returns no payloads. Anything wider — another source's counts, or an unscoped
aggregate — stays behind `may_read` and full tenant scope.

Two refusals worth knowing:

- Another source's counts return **403, not an empty count**. A zero would read
  as "we hold none of yours", which is a materially different and false answer
  to a question about completeness.
- There is **no default window**. A count whose bounds the caller did not choose
  is not comparable against anything the caller computed, and would be quoted as
  though it were.

The response carries a `means` field stating that agreement proves neither
completeness nor that any event occurred — because this is the number most
likely to be quoted out of context in someone else's conformance argument.


The optional `stream_instance_id` query parameter restricts counts to the exact
string stored at `details.data.stream_instance_id`. The response echoes that
filter; no filter preserves the existing aggregate response. Empty or repeated
instance parameters are refused. Source and tenant authorization still apply,
and duplicate deliveries count once. Both SQLite and PostgreSQL implement it.
These are **acceptance-time** windows (`since` inclusive, `until` exclusive),
not the producer's occurrence-time or sequence windows. A collector/observer
must account for drain lag before comparing source-window counters, retain
`stream_sequence`, and never compare a restarted process's reset counters with
another instance or a lifetime total. Instance isolation alone does not detect
sequence gaps or determine whether an instance is active.

## Declared emission cadence

`AUDIT-WP-0009-T05`. The contract is info-tech-canon's EmissionCadence
declaration, **wire schema 0.1**, read at contract digest `b08b4d95fc4b0bd3`
(standard document 0.2.0, candidate). audit-core did not invent the shape; it
waited for it. `audit_core/emission_cadence.py` validates a declaration against
that schema by hand — the receiver stays stdlib-only — and refuses what the
schema refuses rather than relaxing a rule to admit a declaration.

The source owns and publishes its declaration. It reaches audit-core as
`emission_cadence` on the sender registration (the declaration object, as
JSON). An invalid declaration refuses the registration. The ConfigMap scope
overlay may not set or change it: a partial overlay has no sound merge with a
digest-pinned declaration, and a wholesale replacement could widen a window or
drop a class without anyone deciding to.

What each declared form gets here:

| Form | What audit-core does |
| --- | --- |
| `expected-rate` | Evaluated. For each permitted source, events of the class in the trailing `window` ending now are counted; fewer than `expected_min` is a `below_declared_cadence` finding on `GET /v1/stream-findings`. The count is the one `/v1/reconciliation` returns, so a finding and a count cannot disagree. |
| `heartbeat-or-reconciliation` | Validated and kept, not re-evaluated. The reconciliation half is `/v1/reconciliation`, whose count is the `compare_observed` side; `compare_local` belongs to the source. The heartbeat half is evaluated from `heartbeat_classes` (see below). |

Refusals, same as for heartbeats: a wildcard source is not held to a rate, and
two `expected-rate` entries for one class are refused — each would be satisfied
by the other's events.

Every cadence finding carries a `means` field: fewer events than declared says
the source may have stopped **or the declaration is wrong**, and is not proof of
suppression; a rate held would not prove completeness.

### Observer evaluation, 2026-09-22

info-tech-canon asked whether audit-core would evaluate the one published
source-owned declaration, net-kingdom `local-identity/emission-cadence.yaml`
(commit `116643f`, pinned to the earlier digest `972c0b6701d1693f`; the wire
schema is identical). The result, stated at no more than it is:

- **Structural: clean.** Both entries (`serve/token.token_issued`,
  `revoke-token`) validate as `heartbeat-or-reconciliation` with a
  reconciliation block. Asserted in `tests/test_emission_cadence.py`.
- **Operational: not evaluated.** audit-core registers no `local-identity`
  sender and holds none of its events, so there is no observed count behind
  `compare_observed` here either. This is not an observer result on the stream,
  and nobody should count it as one.
- **One incompatibility, recorded.** The contract's `heartbeat.event_class`
  names the heartbeat event's own class (e.g. `flex-auth.decision.heartbeat`).
  audit-core's heartbeat is a single class, `audit-core.heartbeat`, carrying
  the vouched-for class in `data.class`, declared per class in
  `heartbeat_classes`. A declaration's heartbeat block therefore does not map
  onto audit-core's evaluation by itself: a source needs `heartbeat_classes` on
  its registration too, and the two can drift apart. That is fixable on
  either side and has not been fixed on either.