audit-core/workplans/AUDIT-WP-0004-receiver-correctness-and-hardening.md

id	type	title	domain	repo	status	owner	topic_slug	created	updated
AUDIT-WP-0004	workplan	Make the event receiver correct and operable under load	infotech	audit-core	proposed	codex	netkingdom	2026-08-10	2026-08-10

AUDIT-WP-0004 - receiver correctness and hardening

Goal

Close the gap between what the receiver claims and what it does, and make it survive a real sender. This is the work that must land before audit-core is worth deploying.

A pre-deploy review of the WP-0003 implementation found that the ingestion path never calls the AuditBackend contract, that several failure modes escape the error handler and drop the connection, and that two isolation properties the workplan claims as done are not implemented. None of these depend on the choice of storage engine, so this workplan runs in parallel with RAPP-POSTGRES-WP-0002 and gates AUDIT-WP-0005.

Boundaries

This workplan owns the receiver's correctness, error semantics, and operable surface. It does not own deployment, the production storage engine, or the live failure matrix — those are AUDIT-WP-0005.

Work here stays storage-agnostic: the existing SQLite store remains the development and test backend, and the Postgres implementation lands in WP-0005 against the interface this workplan fixes.

T01 - Route ingestion through the audit backend contract

id: AUDIT-WP-0004-T01
status: todo
priority: high

ingestion.py writes records into SQLite directly and never calls AuditBackend.emit(). WP-0002 built a pluggable backend contract with a declared RetentionPolicy; WP-0003 built an ingestion path that bypasses it. A 202 currently means "a row exists in a file", against a store that declares no custody class at all.

Make the ingestion path write through the backend contract, so that accepting an event means a backend with a declared retention policy acknowledged it. Idempotency state and custody write must not be able to diverge — an event recorded as seen but not durably stored is the failure this whole service exists to prevent.

Add a readiness condition that refuses to serve if the configured backend declares durable=False, so the development file backend can never silently become the production sink.

Done when a successful response is backed by a durable backend acknowledgment, and the mock backend cannot be used to serve production traffic.

T02 - Fix error semantics and failure handling

id: AUDIT-WP-0004-T02
status: todo
priority: high

The handler catches ValueError, TypeError, KeyError, and JSONDecodeError. Storage errors do not inherit from those: sqlite3.OperationalError on lock contention and sqlite3.IntegrityError on the race between the existence check and the insert both escape with start_response never called. The auth check sits outside the try block, so a non-ASCII Authorization header raises TypeError from hmac.compare_digest and crashes the request before any handler runs.

Add a catch-all that always produces a response, map storage and unexpected failures to 503 or 500 rather than letting them surface as dropped connections, and move credential comparison inside the guarded path.

Correct the status codes: a duplicate event ID carrying a different payload is a conflict and must return 409, not 400. This distinction drives the sender's retry-versus-dead-letter decision, so the full set of responses — accepted, duplicate, conflict, rejected, unavailable — needs to be documented as a contract with the retry semantics each implies.

Done when no request path can terminate without a response, and every status code the receiver returns is deliberate and documented.

T03 - Enforce the isolation properties already claimed

id: AUDIT-WP-0004-T03
status: todo
priority: high

WP-0003 T01 and T02 both record tenant isolation and cross-tenant claim rejection as done. normalize() accepts whatever tenant string the caller sends and never checks it against the authenticated identity; no test covers it. Separately, source is hardcoded to "user-engine", so the allowlist is unbound to the credential — any holder of the token can claim to be user-engine, and a second sender would require a code change.

Bind the authenticated identity to the tenants and sources it may write for, and reject claims outside that binding. Drive the binding from configuration rather than literals.

Done when a credential scoped to one tenant is refused when it claims another, and that refusal is covered by a test.

T04 - Settle redaction policy

id: AUDIT-WP-0004-T04
status: todo
priority: medium

_contains_secret rejects the entire event when any key name in data contains a secret-shaped fragment, and never inspects the envelope. WP-0003 T01 describes "redacted data" and the failure matrix tests redaction, but the implementation only rejects.

Decide whether a secret-shaped field is a rejection or a redaction, and note the consequence: rejection means a legitimate outbox event containing a field named token_count is dead-lettered at the sender, and the audit trail loses the event entirely. Extend detection to the envelope, and consider value shape rather than key name alone.

Whatever is chosen, the contract document and the sender's expectations must agree with the code.

Done when the redaction rule is explicit, applied to the whole payload, and consistent between the contract and the implementation.

T05 - Provide the operator read surface

id: AUDIT-WP-0004-T05
status: todo
priority: high

There is no way to read anything back. The failure matrix requires dead-letter visibility, operator replay, and correlation lookup; none exist, so that evidence cannot be produced regardless of how the deployment goes.

Build lookup by event ID and by correlation ID, a dead-letter view of rejected events with their rejection reason, and an operator replay path. Replay must be idempotent against the same durable store, so a replayed event reconciles with the original rather than creating a second custody record.

Read access is a separate privilege from write access — a sender credential must not be able to read the audit trail back.

Done when an operator can trace one correlation ID through the system and replay a specific event without creating a duplicate.

T06 - Serving layer and observability

id: AUDIT-WP-0004-T06
status: todo
priority: high

wsgiref.simple_server is single-threaded with no request timeout, no graceful shutdown, and no access log. One slow client blocks every sender. It also makes the failure matrix meaningless: "receiver correctly reported unavailable" and "the server stalled" are indistinguishable.

Move to a production WSGI server with explicit worker and timeout configuration and graceful shutdown on SIGTERM, so in-flight events are not lost on rollout.

Add structured request logging with a request ID, propagate the correlation ID into logs, and expose counters for accepted, duplicate, conflicted, rejected, and failed events plus write latency. Logs and metrics must never carry event payloads.

Done when the receiver serves concurrent senders under a bounded timeout, sheds load predictably instead of stalling, and its behaviour is visible from outside.

T07 - Close the test gaps

id: AUDIT-WP-0004-T07
status: todo
priority: medium

Ingestion has two tests. Uncovered: oversized and zero-length bodies, absent Content-Length, malformed JSON, wrong path and method, the health endpoints, event ID conflict, naive and out-of-range timestamps, storage failure, and persistence across restart.

accepted_at is recorded in local time while every other timestamp is UTC; datetime.fromisoformat accepts naive timestamps and arbitrary dates. Fix both and cover them.

Done when each rejection reason and each failure mode above has a test that asserts the documented status code.