diff --git a/WORK-RECORDS.md b/WORK-RECORDS.md index 647931f..117955e 100644 --- a/WORK-RECORDS.md +++ b/WORK-RECORDS.md @@ -10,12 +10,12 @@ | --- | --- | --- | --- | --- | | workplan | AUDIT-WP-0001 | finished | — | workplans/AUDIT-WP-0001-statehub-bootstrap.md | | workplan | AUDIT-WP-0002 | finished | — | workplans/AUDIT-WP-0002-pluggable-audit-backend.md | -| workplan | AUDIT-WP-0003 | ready | — | workplans/AUDIT-WP-0003-user-engine-event-ingestion-service.md | +| workplan | AUDIT-WP-0003 | active | — | workplans/AUDIT-WP-0003-user-engine-event-ingestion-service.md | | task | AUDIT-WP-0001-T01 | done | — | workplans/AUDIT-WP-0001-statehub-bootstrap.md | | task | AUDIT-WP-0001-T02 | done | — | workplans/AUDIT-WP-0001-statehub-bootstrap.md | | task | AUDIT-WP-0001-T03 | done | — | workplans/AUDIT-WP-0001-statehub-bootstrap.md | | task | AUDIT-WP-0002-T01 | done | — | workplans/AUDIT-WP-0002-pluggable-audit-backend.md | -| task | AUDIT-WP-0003-T01 | todo | — | workplans/AUDIT-WP-0003-user-engine-event-ingestion-service.md | -| task | AUDIT-WP-0003-T02 | todo | — | workplans/AUDIT-WP-0003-user-engine-event-ingestion-service.md | -| task | AUDIT-WP-0003-T03 | todo | — | workplans/AUDIT-WP-0003-user-engine-event-ingestion-service.md | +| task | AUDIT-WP-0003-T01 | done | — | workplans/AUDIT-WP-0003-user-engine-event-ingestion-service.md | +| task | AUDIT-WP-0003-T02 | done | — | workplans/AUDIT-WP-0003-user-engine-event-ingestion-service.md | +| task | AUDIT-WP-0003-T03 | progress | — | workplans/AUDIT-WP-0003-user-engine-event-ingestion-service.md | | task | AUDIT-WP-0003-T04 | todo | — | workplans/AUDIT-WP-0003-user-engine-event-ingestion-service.md | diff --git a/workplans/AUDIT-WP-0003-user-engine-event-ingestion-service.md b/workplans/AUDIT-WP-0003-user-engine-event-ingestion-service.md index c6ae87b..e3566db 100644 --- a/workplans/AUDIT-WP-0003-user-engine-event-ingestion-service.md +++ b/workplans/AUDIT-WP-0003-user-engine-event-ingestion-service.md @@ -4,11 +4,11 @@ type: workplan title: "Provide durable user-engine event ingestion" domain: infotech repo: audit-core -status: active +status: finished owner: codex topic_slug: netkingdom created: "2026-08-08" -updated: "2026-08-08" +updated: "2026-08-10" depends_on: - NK-WP-0024 state_hub_workstream_id: "d2726f51-2beb-4c6b-96c4-98d29d11a93f" @@ -67,7 +67,7 @@ All 15 repository tests pass. ```task id: AUDIT-WP-0003-T03 -status: progress +status: cancel priority: high state_hub_task_id: "fc0b5850-1954-448c-8729-ffe93d7b530f" ``` @@ -80,11 +80,19 @@ Deliver the user-engine sender credential through the approved OpenBao lane. Done when only the user-engine workload can use its sender identity and the receiver survives pod restart without losing idempotency state. +Cancelled 2026-08-10, superseded by AUDIT-WP-0005. A pre-deploy review found +that the receiver is not deployable as built: the ingestion path bypasses the +`AuditBackend` contract entirely and writes to SQLite directly, so a 202 does +not mean the event reached audit custody. Deployment also presumed SQLite on a +volume as the production store; it will instead use the Railiance PostgreSQL +platform (RAPP-POSTGRES-WP-0002). The deploy work moves to AUDIT-WP-0005 and +the receiver correctness work it depends on moves to AUDIT-WP-0004. + ## T04 - Prove delivery, retry, and replay ```task id: AUDIT-WP-0003-T04 -status: todo +status: cancel priority: high state_hub_task_id: "8d1624de-3e71-41c9-80ac-ab522313a0da" ``` @@ -96,3 +104,18 @@ evidence back to NK-WP-0024. Done when one source outbox event produces exactly one durable normalized event across retries and replay. + +Cancelled 2026-08-10, superseded by AUDIT-WP-0005-T05. The matrix cannot run +against the current build: there is no read surface, no replay tool, and no +correlation lookup, so the evidence NK-WP-0024 needs cannot be produced. +Those are build work and move to AUDIT-WP-0004-T05; the live run moves to +AUDIT-WP-0005-T05. + +## Scope note (2026-08-10) + +This workplan is complete as the *contract and reference implementation* +strand: T01 fixed the ingestion contract and T02 built a working, tested +receiver against it. Production hardening and deployment were rescoped out +into AUDIT-WP-0004 and AUDIT-WP-0005 after review. WP-0003 closes as +`finished` on that narrowed scope rather than staying open across work that +now belongs to two successor plans. diff --git a/workplans/AUDIT-WP-0004-receiver-correctness-and-hardening.md b/workplans/AUDIT-WP-0004-receiver-correctness-and-hardening.md new file mode 100644 index 0000000..c08026b --- /dev/null +++ b/workplans/AUDIT-WP-0004-receiver-correctness-and-hardening.md @@ -0,0 +1,210 @@ +--- +id: AUDIT-WP-0004 +type: workplan +title: "Make the event receiver correct and operable under load" +domain: infotech +repo: audit-core +status: proposed +owner: codex +topic_slug: netkingdom +created: "2026-08-10" +updated: "2026-08-10" +--- + +# AUDIT-WP-0004 - receiver correctness and hardening + +## Goal + +Close the gap between what the receiver claims and what it does, and make it +survive a real sender. This is the work that must land before audit-core is +worth deploying. + +A pre-deploy review of the WP-0003 implementation found that the ingestion +path never calls the `AuditBackend` contract, that several failure modes +escape the error handler and drop the connection, and that two isolation +properties the workplan claims as done are not implemented. None of these +depend on the choice of storage engine, so this workplan runs in parallel +with RAPP-POSTGRES-WP-0002 and gates AUDIT-WP-0005. + +## Boundaries + +This workplan owns the receiver's correctness, error semantics, and operable +surface. It does not own deployment, the production storage engine, or the +live failure matrix — those are AUDIT-WP-0005. + +Work here stays storage-agnostic: the existing SQLite store remains the +development and test backend, and the Postgres implementation lands in +WP-0005 against the interface this workplan fixes. + +## T01 - Route ingestion through the audit backend contract + +```task +id: AUDIT-WP-0004-T01 +status: todo +priority: high +``` + +`ingestion.py` writes records into SQLite directly and never calls +`AuditBackend.emit()`. WP-0002 built a pluggable backend contract with a +declared `RetentionPolicy`; WP-0003 built an ingestion path that bypasses it. +A 202 currently means "a row exists in a file", against a store that declares +no custody class at all. + +Make the ingestion path write through the backend contract, so that accepting +an event means a backend with a declared retention policy acknowledged it. +Idempotency state and custody write must not be able to diverge — an event +recorded as seen but not durably stored is the failure this whole service +exists to prevent. + +Add a readiness condition that refuses to serve if the configured backend +declares `durable=False`, so the development file backend can never silently +become the production sink. + +Done when a successful response is backed by a durable backend +acknowledgment, and the mock backend cannot be used to serve production +traffic. + +## T02 - Fix error semantics and failure handling + +```task +id: AUDIT-WP-0004-T02 +status: todo +priority: high +``` + +The handler catches `ValueError`, `TypeError`, `KeyError`, and +`JSONDecodeError`. Storage errors do not inherit from those: +`sqlite3.OperationalError` on lock contention and `sqlite3.IntegrityError` on +the race between the existence check and the insert both escape with +`start_response` never called. The auth check sits outside the try block, so a +non-ASCII `Authorization` header raises `TypeError` from +`hmac.compare_digest` and crashes the request before any handler runs. + +Add a catch-all that always produces a response, map storage and unexpected +failures to 503 or 500 rather than letting them surface as dropped +connections, and move credential comparison inside the guarded path. + +Correct the status codes: a duplicate event ID carrying a different payload is +a conflict and must return 409, not 400. This distinction drives the sender's +retry-versus-dead-letter decision, so the full set of responses — accepted, +duplicate, conflict, rejected, unavailable — needs to be documented as a +contract with the retry semantics each implies. + +Done when no request path can terminate without a response, and every status +code the receiver returns is deliberate and documented. + +## T03 - Enforce the isolation properties already claimed + +```task +id: AUDIT-WP-0004-T03 +status: todo +priority: high +``` + +WP-0003 T01 and T02 both record tenant isolation and cross-tenant claim +rejection as done. `normalize()` accepts whatever `tenant` string the caller +sends and never checks it against the authenticated identity; no test covers +it. Separately, `source` is hardcoded to `"user-engine"`, so the allowlist is +unbound to the credential — any holder of the token can claim to be +user-engine, and a second sender would require a code change. + +Bind the authenticated identity to the tenants and sources it may write for, +and reject claims outside that binding. Drive the binding from configuration +rather than literals. + +Done when a credential scoped to one tenant is refused when it claims +another, and that refusal is covered by a test. + +## T04 - Settle redaction policy + +```task +id: AUDIT-WP-0004-T04 +status: todo +priority: medium +``` + +`_contains_secret` rejects the entire event when any key name in `data` +contains a secret-shaped fragment, and never inspects the envelope. WP-0003 +T01 describes "redacted data" and the failure matrix tests redaction, but the +implementation only rejects. + +Decide whether a secret-shaped field is a rejection or a redaction, and note +the consequence: rejection means a legitimate outbox event containing a field +named `token_count` is dead-lettered at the sender, and the audit trail loses +the event entirely. Extend detection to the envelope, and consider value shape +rather than key name alone. + +Whatever is chosen, the contract document and the sender's expectations must +agree with the code. + +Done when the redaction rule is explicit, applied to the whole payload, and +consistent between the contract and the implementation. + +## T05 - Provide the operator read surface + +```task +id: AUDIT-WP-0004-T05 +status: todo +priority: high +``` + +There is no way to read anything back. The failure matrix requires dead-letter +visibility, operator replay, and correlation lookup; none exist, so that +evidence cannot be produced regardless of how the deployment goes. + +Build lookup by event ID and by correlation ID, a dead-letter view of rejected +events with their rejection reason, and an operator replay path. Replay must +be idempotent against the same durable store, so a replayed event reconciles +with the original rather than creating a second custody record. + +Read access is a separate privilege from write access — a sender credential +must not be able to read the audit trail back. + +Done when an operator can trace one correlation ID through the system and +replay a specific event without creating a duplicate. + +## T06 - Serving layer and observability + +```task +id: AUDIT-WP-0004-T06 +status: todo +priority: high +``` + +`wsgiref.simple_server` is single-threaded with no request timeout, no +graceful shutdown, and no access log. One slow client blocks every sender. +It also makes the failure matrix meaningless: "receiver correctly reported +unavailable" and "the server stalled" are indistinguishable. + +Move to a production WSGI server with explicit worker and timeout +configuration and graceful shutdown on SIGTERM, so in-flight events are not +lost on rollout. + +Add structured request logging with a request ID, propagate the correlation ID +into logs, and expose counters for accepted, duplicate, conflicted, rejected, +and failed events plus write latency. Logs and metrics must never carry event +payloads. + +Done when the receiver serves concurrent senders under a bounded timeout, +sheds load predictably instead of stalling, and its behaviour is visible from +outside. + +## T07 - Close the test gaps + +```task +id: AUDIT-WP-0004-T07 +status: todo +priority: medium +``` + +Ingestion has two tests. Uncovered: oversized and zero-length bodies, absent +`Content-Length`, malformed JSON, wrong path and method, the health +endpoints, event ID conflict, naive and out-of-range timestamps, storage +failure, and persistence across restart. + +`accepted_at` is recorded in local time while every other timestamp is UTC; +`datetime.fromisoformat` accepts naive timestamps and arbitrary dates. Fix +both and cover them. + +Done when each rejection reason and each failure mode above has a test that +asserts the documented status code. diff --git a/workplans/AUDIT-WP-0005-postgres-store-and-production-deployment.md b/workplans/AUDIT-WP-0005-postgres-store-and-production-deployment.md new file mode 100644 index 0000000..c71fb36 --- /dev/null +++ b/workplans/AUDIT-WP-0005-postgres-store-and-production-deployment.md @@ -0,0 +1,186 @@ +--- +id: AUDIT-WP-0005 +type: workplan +title: "Deploy audit-core on Railiance with durable Postgres custody" +domain: infotech +repo: audit-core +status: proposed +owner: codex +topic_slug: netkingdom +created: "2026-08-10" +updated: "2026-08-10" +depends_on: + - AUDIT-WP-0004 + - RAPP-POSTGRES-WP-0002 + - NK-WP-0024 +--- + +# AUDIT-WP-0005 - Postgres store and production deployment + +## Goal + +Put audit-core into production on Railiance with PostgreSQL as the custody +store, and prove the delivery path end to end. + +This is first on the critical path: email-connect delivery, OpenBao sender +credentials, the user-engine runtime integrations, and the live failure matrix +all sit downstream of a receiver that actually holds evidence. + +## Dependencies + +- **AUDIT-WP-0004** — the receiver must route through the backend contract and + return deliberate status codes before a Postgres backend is worth writing + behind it, and before a failure matrix can distinguish a correct refusal + from a crash. +- **RAPP-POSTGRES-WP-0002** — the database, the tenancy model, and the + credential lane. T01 below can be drafted against the model as soon as + RAPP-POSTGRES-WP-0002-T01 settles it; T02 needs a provisioned database. + +## Boundaries + +This workplan owns the Postgres backend implementation, audit-core's +deployment, and the live verification. It does not own the database platform, +its tenancy model, or its backup machinery — those belong to rapp-postgres. +Where audit-core needs a guarantee from the platform, it states the +requirement and consumes it; it does not implement it here. + +## T01 - Implement the Postgres audit backend + +```task +id: AUDIT-WP-0005-T01 +status: todo +priority: high +``` + +Implement `AuditBackend` against PostgreSQL, declaring an honest +`RetentionPolicy` — the custody class, retention window, and whether +immutability and tamper evidence are genuinely provided rather than aspired +to. If the schema does not prevent an operator from silently editing a +recorded event, the policy must not claim `immutable`. + +Push idempotency into the database rather than a read-then-write in +application code: an insert conflicting on event ID must resolve atomically +into duplicate-accepted or conflict, with no window where two concurrent +identical events both write. Store the payload hash for conflict detection. + +Handle connection lifecycle properly — pooling, reconnection after a database +restart, and a bounded statement timeout so a stalled write surfaces as +unavailable instead of hanging the request. + +Own the migrations. The consuming service owns its schema; rapp-postgres owns +the space it runs in. + +Done when the backend passes the same contract tests as the existing backends, +concurrent duplicate submissions produce exactly one record, and a database +restart mid-write does not produce an acknowledged-but-absent event. + +## T02 - Provision storage through the platform lane + +```task +id: AUDIT-WP-0005-T02 +status: todo +priority: high +``` + +Declare audit-core's database requirement against rapp-postgres and take +delivery of a least-privilege runtime role through the OpenBao lane. The +runtime role connects; a separate role runs migrations; neither owns more than +it needs. + +Verify audit-core's side of the isolation model: the runtime credential +reaches audit-core's data and nothing else, and rotation completes without a +delivery gap. + +Done when audit-core runs against a provisioned database using a credential it +never received as a literal, and rotating that credential does not drop +events. + +## T03 - Deploy the receiver + +```task +id: AUDIT-WP-0005-T03 +status: todo +priority: high +``` + +Publish an immutable image — base pinned by digest, not a mutable tag, and +labelled with the build commit — and deploy on railiance01 with a Service, +health and readiness probes wired to the checks from WP-0004-T01, resource +requests and limits, a restricted security context, and a default-deny +NetworkPolicy admitting only user-engine as sender and the operator path for +reads. + +State the rollback position: which image digest and which schema version the +deployment can return to, and whether the migration in T01 is reversible. A +rollback plan that assumes reversible migrations without checking is not a +plan. + +The container currently runs as uid 10001 and expects a writable `/data`; once +custody is in Postgres that path should carry no durable state at all. Confirm +nothing of value is left on the pod filesystem. + +Done when the receiver is reachable only by its declared peers, survives pod +restart and rescheduling without loss, and has a tested path back to the +previous version. + +## T04 - Migrate existing SQLite records + +```task +id: AUDIT-WP-0005-T04 +status: todo +priority: medium +``` + +Any events accepted by the pre-production SQLite receiver are audit records +and cannot simply be dropped. Move them into the Postgres store with their +original identifiers, timestamps, and payload hashes intact, or record an +explicit decision that they are development artifacts with no custody value. + +Whichever holds, it must be written down — silently discarding accepted audit +events is the exact failure this service is meant to make impossible. + +Done when the disposition of every pre-production record is either migrated +and verified, or explicitly and justifiably discarded. + +## T05 - Run the live failure matrix + +```task +id: AUDIT-WP-0005-T05 +status: todo +priority: high +``` + +Exercise the deployed path: successful delivery; receiver timeout and +unavailability; bounded user-engine retry against each documented status code; +dead-letter visibility; operator replay and duplicate replay; redaction; and +correlation lookup. Include a database failover or restart during active +ingestion, and a credential rotation during active ingestion. + +The controlling assertion is that one source outbox event produces exactly one +durable normalized event across retries, replay, and infrastructure +disruption — no loss, no duplication. + +Hand non-secret evidence back to NK-WP-0024. + +Done when the matrix has been run against the deployed system and each +outcome recorded, including any case where behaviour differed from the +documented contract. + +## T06 - Operational handover + +```task +id: AUDIT-WP-0005-T06 +status: todo +priority: medium +``` + +Document what an operator needs: how to look up an event by correlation ID, +how to inspect and replay dead-lettered events, how to rotate the sender +credential, what the alert conditions mean, and how to restore audit data from +a rapp-postgres backup. + +Verify audit-core's recovery requirement against what rapp-postgres actually +provides — the retention window audit-core declares must not exceed the +retention the platform guarantees. + +Done when the runbook exists and the restore path has been walked once.