audit-core/docs/stream-completeness.md

# Stream completeness: heartbeats, reconciliation, and findings

`AUDIT-WP-0009` T04, T05, T06, T07. Statute §9.6.

The chain proves records held were not altered or truncated. It says nothing
about a record that never arrived — and §9.6 is explicit that **omission** is
the acute risk for exactly the rare negative classes (revocation, denial,
containment) where suppression is most valuable and least visible.

audit-core argued that obligation up to a MUST. A source cannot declare a
cadence to a system with nowhere to put it, so this is the surface owed.

## Three surfaces, and what each actually covers

| Surface | Covers | Does not cover |
| --- | --- | --- |
| Heartbeat (`POST /v1/events`, class `audit-core.heartbeat`) | A class going silent — including one that is *legitimately* silent, which rate monitoring can never distinguish | A compromised source emitting a truthful-looking heartbeat while suppressing the event |
| Reconciliation (`GET /v1/reconciliation`) | Loss, outage, drain failure — divergence between what a source emitted and what arrived | A compromised source suppressing the event and its own count together |
| Declared rate (`emission_cadence` on the registration) | A high-volume class falling below the rate its source declared | A class that may legitimately be silent — that is the heartbeat's job; and a rate held proves nothing about completeness |
| Findings (`GET /v1/stream-findings`) | Surfacing the above where an operator sees them | Anything the two above do not detect |

**The bound is the same in both rows and it is not a footnote.** Where the
emitter itself is compromised, both controls agree with it. They cover loss,
outage, drain failure and accident — most of what actually goes wrong — and not
a source lying about itself. Closing that needs an observer independent of the
emitter, which §16 placed outside audit-core's scope. No conformance claim may
read these surfaces as covering adversarial omission by the source.

## Heartbeats

A heartbeat is an **ordinary event**: same envelope, same append-only custody,
same chain, no special table. That is deliberate — a heartbeat stored outside
the chain would be the one record in this store that could be back-dated.

```json
{
  "id": "...", "type": "audit-core.heartbeat",
  "source": "approval-engine", "subject": "approval-engine",
  "tenant": "tenant:platform", "correlation_id": "...",
  "occurred_at": "2026-09-10T03:00:00+00:00",
  "data": {"class": "approval.revocation", "assertion": "nothing-to-report"}
}
```

### Declared per class, never per source

`heartbeat_classes` on the sender registration maps event class → the longest
gap in seconds that is not yet a finding.

Per class is the whole point. A per-source heartbeat from a mixed-volume
emitter is satisfied by its chattiest class and says nothing about the quiet,
security-relevant one — and the quiet class is the only reason heartbeats
exist. `informed-decision` raised this shape first for presentations versus
dispositions; it generalises.

### Heartbeat and cadence are complementary

A heartbeat is a registration property saying how often a source promises to
say *nothing to report* for a class that may legitimately be silent. Cadence
describes a stream's expected **rate**. They are not alternatives, and the rate
shape is not audit-core's — see [Declared emission cadence](#declared-emission-cadence).

### Findings

`GET /v1/stream-findings` requires `may_read` **and** full tenant scope:
findings span every registered sender and carry no tenant key, so there is
nothing to filter on and a scoped reader is refused rather than served
instance-wide facts. Same rule as dead letters, secret findings and integrity.

Two kinds:

- `no_heartbeat_since_registration` — declared a heartbeat, never sent one.
  Its own kind rather than skipped, because it is the case most likely to be a
  broken integration and the one a naive "compare against last seen"
  implementation silently drops.
- `missing_heartbeat` — one arrived once and is now overdue, with how late.

A grace factor of 1.5 widens the window so a single late run does not flap a
finding on and off — the same reasoning as the attestation freshness window. It
widens; it never removes.

Every finding carries a `means` field saying that absence of a heartbeat is not
proof of suppression. That travels on the finding rather than in a document
nobody opens alongside it.

## Reconciliation

`GET /v1/reconciliation?source=&tenant=&since=&until=` returns per-class counts
of a source's own events. **Counts, never payloads.**

### Why a writer with `may_read: false` may call it

Every registered sender holds `may_read: false` — a source does not gain a read
surface by emitting. Taken literally that would make the §9.6 reconciliation
obligation undischargeable by every source actually registered, which is a
strange place for a rule audit-core argued for to end up.

The resolution is that **a source asking how many of its own events audit-core
holds is not reading the archive**. It learns nothing it did not itself emit.
So the surface is scoped to the caller's own permitted sources and tenants, and
returns no payloads. Anything wider — another source's counts, or an unscoped
aggregate — stays behind `may_read` and full tenant scope.

Two refusals worth knowing:

- Another source's counts return **403, not an empty count**. A zero would read
  as "we hold none of yours", which is a materially different and false answer
  to a question about completeness.
- There is **no default window**. A count whose bounds the caller did not choose
  is not comparable against anything the caller computed, and would be quoted as
  though it were.

The response carries a `means` field stating that agreement proves neither
completeness nor that any event occurred — because this is the number most
likely to be quoted out of context in someone else's conformance argument.

## Declared emission cadence

`AUDIT-WP-0009-T05`. The contract is info-tech-canon's EmissionCadence
declaration, **wire schema 0.1**, read at contract digest `b08b4d95fc4b0bd3`
(standard document 0.2.0, candidate). audit-core did not invent the shape; it
waited for it. `audit_core/emission_cadence.py` validates a declaration against
that schema by hand — the receiver stays stdlib-only — and refuses what the
schema refuses rather than relaxing a rule to admit a declaration.

The source owns and publishes its declaration. It reaches audit-core as
`emission_cadence` on the sender registration (the declaration object, as
JSON). An invalid declaration refuses the registration. The ConfigMap scope
overlay may not set or change it: a partial overlay has no sound merge with a
digest-pinned declaration, and a wholesale replacement could widen a window or
drop a class without anyone deciding to.

What each declared form gets here:

| Form | What audit-core does |
| --- | --- |
| `expected-rate` | Evaluated. For each permitted source, events of the class in the trailing `window` ending now are counted; fewer than `expected_min` is a `below_declared_cadence` finding on `GET /v1/stream-findings`. The count is the one `/v1/reconciliation` returns, so a finding and a count cannot disagree. |
| `heartbeat-or-reconciliation` | Validated and kept, not re-evaluated. The reconciliation half is `/v1/reconciliation`, whose count is the `compare_observed` side; `compare_local` belongs to the source. The heartbeat half is evaluated from `heartbeat_classes` (see below). |

Refusals, same as for heartbeats: a wildcard source is not held to a rate, and
two `expected-rate` entries for one class are refused — each would be satisfied
by the other's events.

Every cadence finding carries a `means` field: fewer events than declared says
the source may have stopped **or the declaration is wrong**, and is not proof of
suppression; a rate held would not prove completeness.

### Observer evaluation, 2026-09-22

info-tech-canon asked whether audit-core would evaluate the one published
source-owned declaration, net-kingdom `local-identity/emission-cadence.yaml`
(commit `116643f`, pinned to the earlier digest `972c0b6701d1693f`; the wire
schema is identical). The result, stated at no more than it is:

- **Structural: clean.** Both entries (`serve/token.token_issued`,
  `revoke-token`) validate as `heartbeat-or-reconciliation` with a
  reconciliation block. Asserted in `tests/test_emission_cadence.py`.
- **Operational: not evaluated.** audit-core registers no `local-identity`
  sender and holds none of its events, so there is no observed count behind
  `compare_observed` here either. This is not an observer result on the stream,
  and nobody should count it as one.
- **One incompatibility, recorded.** The contract's `heartbeat.event_class`
  names the heartbeat event's own class (e.g. `flex-auth.decision.heartbeat`).
  audit-core's heartbeat is a single class, `audit-core.heartbeat`, carrying
  the vouched-for class in `data.class`, declared per class in
  `heartbeat_classes`. A declaration's heartbeat block therefore does not map
  onto audit-core's evaluation by itself: a source needs `heartbeat_classes` on
  its registration too, and the two can drift apart. That is fixable on
  either side and has not been fixed on either.