audit-core/docs/stream-completeness.md

Stream completeness: heartbeats, reconciliation, and findings

AUDIT-WP-0009 T04, T05, T06, T07. Statute §9.6.

The chain proves records held were not altered or truncated. It says nothing about a record that never arrived — and §9.6 is explicit that omission is the acute risk for exactly the rare negative classes (revocation, denial, containment) where suppression is most valuable and least visible.

audit-core argued that obligation up to a MUST. A source cannot declare a cadence to a system with nowhere to put it, so this is the surface owed.

Three surfaces, and what each actually covers

Surface	Covers	Does not cover
Heartbeat (POST /v1/events, class audit-core.heartbeat)	A class going silent — including one that is legitimately silent, which rate monitoring can never distinguish	A compromised source emitting a truthful-looking heartbeat while suppressing the event
Reconciliation (GET /v1/reconciliation)	Loss, outage, drain failure — divergence between what a source emitted and what arrived	A compromised source suppressing the event and its own count together
Declared rate (emission_cadence on the registration)	A high-volume class falling below the rate its source declared	A class that may legitimately be silent — that is the heartbeat's job; and a rate held proves nothing about completeness
Findings (GET /v1/stream-findings)	Surfacing the above where an operator sees them	Anything the two above do not detect

The bound is the same in both rows and it is not a footnote. Where the emitter itself is compromised, both controls agree with it. They cover loss, outage, drain failure and accident — most of what actually goes wrong — and not a source lying about itself. Closing that needs an observer independent of the emitter, which §16 placed outside audit-core's scope. No conformance claim may read these surfaces as covering adversarial omission by the source.

Heartbeats

A heartbeat is an ordinary event: same envelope, same append-only custody, same chain, no special table. That is deliberate — a heartbeat stored outside the chain would be the one record in this store that could be back-dated.

{
  "id": "...", "type": "audit-core.heartbeat",
  "source": "approval-engine", "subject": "approval-engine",
  "tenant": "tenant:platform", "correlation_id": "...",
  "occurred_at": "2026-09-10T03:00:00+00:00",
  "data": {"class": "approval.revocation", "assertion": "nothing-to-report"}
}

Declared per class, never per source

heartbeat_classes on the sender registration maps event class → the longest gap in seconds that is not yet a finding.

Per class is the whole point. A per-source heartbeat from a mixed-volume emitter is satisfied by its chattiest class and says nothing about the quiet, security-relevant one — and the quiet class is the only reason heartbeats exist. informed-decision raised this shape first for presentations versus dispositions; it generalises.

Heartbeat and cadence are complementary

A heartbeat is a registration property saying how often a source promises to say nothing to report for a class that may legitimately be silent. Cadence describes a stream's expected rate. They are not alternatives, and the rate shape is not audit-core's — see Declared emission cadence.

Findings

GET /v1/stream-findings requires may_read and full tenant scope: findings span every registered sender and carry no tenant key, so there is nothing to filter on and a scoped reader is refused rather than served instance-wide facts. Same rule as dead letters, secret findings and integrity.

Two kinds:

• no_heartbeat_since_registration — declared a heartbeat, never sent one. Its own kind rather than skipped, because it is the case most likely to be a broken integration and the one a naive "compare against last seen" implementation silently drops.
• missing_heartbeat — one arrived once and is now overdue, with how late.

A grace factor of 1.5 widens the window so a single late run does not flap a finding on and off — the same reasoning as the attestation freshness window. It widens; it never removes.

Every finding carries a means field saying that absence of a heartbeat is not proof of suppression. That travels on the finding rather than in a document nobody opens alongside it.

Reconciliation

GET /v1/reconciliation?source=&tenant=&since=&until= returns per-class counts of a source's own events. Counts, never payloads.

Why a writer with may_read: false may call it

Every registered sender holds may_read: false — a source does not gain a read surface by emitting. Taken literally that would make the §9.6 reconciliation obligation undischargeable by every source actually registered, which is a strange place for a rule audit-core argued for to end up.

The resolution is that a source asking how many of its own events audit-core holds is not reading the archive. It learns nothing it did not itself emit. So the surface is scoped to the caller's own permitted sources and tenants, and returns no payloads. Anything wider — another source's counts, or an unscoped aggregate — stays behind may_read and full tenant scope.

Two refusals worth knowing:

• Another source's counts return 403, not an empty count. A zero would read as "we hold none of yours", which is a materially different and false answer to a question about completeness.
• There is no default window. A count whose bounds the caller did not choose is not comparable against anything the caller computed, and would be quoted as though it were.

The response carries a means field stating that agreement proves neither completeness nor that any event occurred — because this is the number most likely to be quoted out of context in someone else's conformance argument.

The optional stream_instance_id query parameter restricts counts to the exact string stored at details.data.stream_instance_id. The response echoes that filter; no filter preserves the existing aggregate response. Empty or repeated instance parameters are refused. Source and tenant authorization still apply, and duplicate deliveries count once. Both SQLite and PostgreSQL implement it. These are acceptance-time windows (since inclusive, until exclusive), not the producer's occurrence-time or sequence windows. A collector/observer must account for drain lag before comparing source-window counters, retain stream_sequence, and never compare a restarted process's reset counters with another instance or a lifetime total. Instance isolation alone does not detect sequence gaps or determine whether an instance is active.

Declared emission cadence

AUDIT-WP-0009-T05. The contract is info-tech-canon's EmissionCadence declaration, wire schema 0.1, read at contract digest b08b4d95fc4b0bd3 (standard document 0.2.0, candidate). audit-core did not invent the shape; it waited for it. audit_core/emission_cadence.py validates a declaration against that schema by hand — the receiver stays stdlib-only — and refuses what the schema refuses rather than relaxing a rule to admit a declaration.

The source owns and publishes its declaration. It reaches audit-core as emission_cadence on the sender registration (the declaration object, as JSON). An invalid declaration refuses the registration. The ConfigMap scope overlay may not set or change it: a partial overlay has no sound merge with a digest-pinned declaration, and a wholesale replacement could widen a window or drop a class without anyone deciding to.

What each declared form gets here:

Form	What audit-core does
expected-rate	Evaluated. For each permitted source, events of the class in the trailing window ending now are counted; fewer than expected_min is a below_declared_cadence finding on GET /v1/stream-findings. The count is the one /v1/reconciliation returns, so a finding and a count cannot disagree.
heartbeat-or-reconciliation	Validated and kept, not re-evaluated. The reconciliation half is /v1/reconciliation, whose count is the compare_observed side; compare_local belongs to the source. The heartbeat half is evaluated from heartbeat_classes (see below).

Refusals, same as for heartbeats: a wildcard source is not held to a rate, and two expected-rate entries for one class are refused — each would be satisfied by the other's events.

Every cadence finding carries a means field: fewer events than declared says the source may have stopped or the declaration is wrong, and is not proof of suppression; a rate held would not prove completeness.

Observer evaluation, 2026-09-22

info-tech-canon asked whether audit-core would evaluate the one published source-owned declaration, net-kingdom local-identity/emission-cadence.yaml (commit 116643f, pinned to the earlier digest 972c0b6701d1693f; the wire schema is identical). The result, stated at no more than it is:

• Structural: clean. Both entries (serve/token.token_issued, revoke-token) validate as heartbeat-or-reconciliation with a reconciliation block. Asserted in tests/test_emission_cadence.py.
• Operational: not evaluated. audit-core registers no local-identity sender and holds none of its events, so there is no observed count behind compare_observed here either. This is not an observer result on the stream, and nobody should count it as one.
• One incompatibility, recorded. The contract's heartbeat.event_class names the heartbeat event's own class (e.g. flex-auth.decision.heartbeat). audit-core's heartbeat is a single class, audit-core.heartbeat, carrying the vouched-for class in data.class, declared per class in heartbeat_classes. A declaration's heartbeat block therefore does not map onto audit-core's evaluation by itself: a source needs heartbeat_classes on its registration too, and the two can drift apart. That is fixable on either side and has not been fixed on either.

Opening events that never close (AUDIT-WP-0018-T03)

Some streams come in pairs. The bounded executor emits execution.run.started once it has consumed its single approval, and execution.attribution.receipt when the run ends. A crash in between leaves only the start in custody, and neither heartbeats nor rate counts can see that.

A sender's registration may therefore declare closing_classes:

"closing_classes": {
  "execution.run.started": {
    "closed_by": "execution.attribution.receipt",
    "within_seconds": 7200
  }
}

GET /v1/stream-findings then carries an opening_without_closing finding for every opening event that has no closing event with the same correlation_id from the same source by its deadline.

• Clock. The deadline counts from the receiver's own accepted_at, never from a source-supplied time.
• Bound. The deadline is the declared within_seconds. If the opening event carries data.window.not_after, the earlier of the two applies (bound names which one). A per-run window can tighten the declaration but never extend it.
• Grace. The same 1.5× allowance as heartbeats is added (within_seconds × 0.5), so a receipt accepted moments late does not flap the finding. For the executor that means a finding at 3 h at the latest.
• Lookback. Opening events accepted in the last 30 days, the operational custody window, are examined.
• No correlation id. An opening event without a correlation_id can never be closed and is always reported.
• Disclosure. A finding carries identifiers and timing only: event id, correlation id, accepted time, deadline. No payload is disclosed.
• Overlay. The scope overlay may add a pair or shorten its bound. It may never remove a pair, rebind its closing class, or lengthen the bound.

The finding's means says what it does not prove. The run may have partly executed, failed before emitting, or had its receipt lost or suppressed. Audit-core does not infer which, and does not infer what the run did. It is a finding for the verifier and the founder. As with heartbeats, a compromised emitter that suppresses its start and its receipt together is not covered.