Rescope WP-0003 and open WP-0004/WP-0005 after pre-deploy review
A pre-deploy review found the receiver is not deployable as built. The ingestion path never calls AuditBackend.emit(), so a 202 means a SQLite row exists, not that a backend with a declared retention policy accepted the event. Storage exceptions escape the handler with start_response never called. Tenant isolation and source binding are recorded as done but are not implemented. There is no read, replay, or correlation-lookup surface, so the failure matrix cannot produce the evidence NK-WP-0024 needs. WP-0003 closes as finished on its narrowed scope (contract + reference implementation, T01/T02). T03 and T04 are cancelled with rationale. WP-0004 covers receiver correctness and hardening, storage-agnostic so it runs in parallel with the database platform work. WP-0005 covers the Postgres backend, deployment, SQLite record migration, and the live failure matrix. Production storage moves from SQLite-on-a-volume to the Railiance PostgreSQL platform (RAPP-POSTGRES-WP-0002). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
d17fdf8b26
commit
3e9ec78f91
4 changed files with 427 additions and 8 deletions
|
|
@ -10,12 +10,12 @@
|
|||
| --- | --- | --- | --- | --- |
|
||||
| workplan | AUDIT-WP-0001 | finished | — | workplans/AUDIT-WP-0001-statehub-bootstrap.md |
|
||||
| workplan | AUDIT-WP-0002 | finished | — | workplans/AUDIT-WP-0002-pluggable-audit-backend.md |
|
||||
| workplan | AUDIT-WP-0003 | ready | — | workplans/AUDIT-WP-0003-user-engine-event-ingestion-service.md |
|
||||
| workplan | AUDIT-WP-0003 | active | — | workplans/AUDIT-WP-0003-user-engine-event-ingestion-service.md |
|
||||
| task | AUDIT-WP-0001-T01 | done | — | workplans/AUDIT-WP-0001-statehub-bootstrap.md |
|
||||
| task | AUDIT-WP-0001-T02 | done | — | workplans/AUDIT-WP-0001-statehub-bootstrap.md |
|
||||
| task | AUDIT-WP-0001-T03 | done | — | workplans/AUDIT-WP-0001-statehub-bootstrap.md |
|
||||
| task | AUDIT-WP-0002-T01 | done | — | workplans/AUDIT-WP-0002-pluggable-audit-backend.md |
|
||||
| task | AUDIT-WP-0003-T01 | todo | — | workplans/AUDIT-WP-0003-user-engine-event-ingestion-service.md |
|
||||
| task | AUDIT-WP-0003-T02 | todo | — | workplans/AUDIT-WP-0003-user-engine-event-ingestion-service.md |
|
||||
| task | AUDIT-WP-0003-T03 | todo | — | workplans/AUDIT-WP-0003-user-engine-event-ingestion-service.md |
|
||||
| task | AUDIT-WP-0003-T01 | done | — | workplans/AUDIT-WP-0003-user-engine-event-ingestion-service.md |
|
||||
| task | AUDIT-WP-0003-T02 | done | — | workplans/AUDIT-WP-0003-user-engine-event-ingestion-service.md |
|
||||
| task | AUDIT-WP-0003-T03 | progress | — | workplans/AUDIT-WP-0003-user-engine-event-ingestion-service.md |
|
||||
| task | AUDIT-WP-0003-T04 | todo | — | workplans/AUDIT-WP-0003-user-engine-event-ingestion-service.md |
|
||||
|
|
|
|||
|
|
@ -4,11 +4,11 @@ type: workplan
|
|||
title: "Provide durable user-engine event ingestion"
|
||||
domain: infotech
|
||||
repo: audit-core
|
||||
status: active
|
||||
status: finished
|
||||
owner: codex
|
||||
topic_slug: netkingdom
|
||||
created: "2026-08-08"
|
||||
updated: "2026-08-08"
|
||||
updated: "2026-08-10"
|
||||
depends_on:
|
||||
- NK-WP-0024
|
||||
state_hub_workstream_id: "d2726f51-2beb-4c6b-96c4-98d29d11a93f"
|
||||
|
|
@ -67,7 +67,7 @@ All 15 repository tests pass.
|
|||
|
||||
```task
|
||||
id: AUDIT-WP-0003-T03
|
||||
status: progress
|
||||
status: cancel
|
||||
priority: high
|
||||
state_hub_task_id: "fc0b5850-1954-448c-8729-ffe93d7b530f"
|
||||
```
|
||||
|
|
@ -80,11 +80,19 @@ Deliver the user-engine sender credential through the approved OpenBao lane.
|
|||
Done when only the user-engine workload can use its sender identity and the
|
||||
receiver survives pod restart without losing idempotency state.
|
||||
|
||||
Cancelled 2026-08-10, superseded by AUDIT-WP-0005. A pre-deploy review found
|
||||
that the receiver is not deployable as built: the ingestion path bypasses the
|
||||
`AuditBackend` contract entirely and writes to SQLite directly, so a 202 does
|
||||
not mean the event reached audit custody. Deployment also presumed SQLite on a
|
||||
volume as the production store; it will instead use the Railiance PostgreSQL
|
||||
platform (RAPP-POSTGRES-WP-0002). The deploy work moves to AUDIT-WP-0005 and
|
||||
the receiver correctness work it depends on moves to AUDIT-WP-0004.
|
||||
|
||||
## T04 - Prove delivery, retry, and replay
|
||||
|
||||
```task
|
||||
id: AUDIT-WP-0003-T04
|
||||
status: todo
|
||||
status: cancel
|
||||
priority: high
|
||||
state_hub_task_id: "8d1624de-3e71-41c9-80ac-ab522313a0da"
|
||||
```
|
||||
|
|
@ -96,3 +104,18 @@ evidence back to NK-WP-0024.
|
|||
|
||||
Done when one source outbox event produces exactly one durable normalized
|
||||
event across retries and replay.
|
||||
|
||||
Cancelled 2026-08-10, superseded by AUDIT-WP-0005-T05. The matrix cannot run
|
||||
against the current build: there is no read surface, no replay tool, and no
|
||||
correlation lookup, so the evidence NK-WP-0024 needs cannot be produced.
|
||||
Those are build work and move to AUDIT-WP-0004-T05; the live run moves to
|
||||
AUDIT-WP-0005-T05.
|
||||
|
||||
## Scope note (2026-08-10)
|
||||
|
||||
This workplan is complete as the *contract and reference implementation*
|
||||
strand: T01 fixed the ingestion contract and T02 built a working, tested
|
||||
receiver against it. Production hardening and deployment were rescoped out
|
||||
into AUDIT-WP-0004 and AUDIT-WP-0005 after review. WP-0003 closes as
|
||||
`finished` on that narrowed scope rather than staying open across work that
|
||||
now belongs to two successor plans.
|
||||
|
|
|
|||
210
workplans/AUDIT-WP-0004-receiver-correctness-and-hardening.md
Normal file
210
workplans/AUDIT-WP-0004-receiver-correctness-and-hardening.md
Normal file
|
|
@ -0,0 +1,210 @@
|
|||
---
|
||||
id: AUDIT-WP-0004
|
||||
type: workplan
|
||||
title: "Make the event receiver correct and operable under load"
|
||||
domain: infotech
|
||||
repo: audit-core
|
||||
status: proposed
|
||||
owner: codex
|
||||
topic_slug: netkingdom
|
||||
created: "2026-08-10"
|
||||
updated: "2026-08-10"
|
||||
---
|
||||
|
||||
# AUDIT-WP-0004 - receiver correctness and hardening
|
||||
|
||||
## Goal
|
||||
|
||||
Close the gap between what the receiver claims and what it does, and make it
|
||||
survive a real sender. This is the work that must land before audit-core is
|
||||
worth deploying.
|
||||
|
||||
A pre-deploy review of the WP-0003 implementation found that the ingestion
|
||||
path never calls the `AuditBackend` contract, that several failure modes
|
||||
escape the error handler and drop the connection, and that two isolation
|
||||
properties the workplan claims as done are not implemented. None of these
|
||||
depend on the choice of storage engine, so this workplan runs in parallel
|
||||
with RAPP-POSTGRES-WP-0002 and gates AUDIT-WP-0005.
|
||||
|
||||
## Boundaries
|
||||
|
||||
This workplan owns the receiver's correctness, error semantics, and operable
|
||||
surface. It does not own deployment, the production storage engine, or the
|
||||
live failure matrix — those are AUDIT-WP-0005.
|
||||
|
||||
Work here stays storage-agnostic: the existing SQLite store remains the
|
||||
development and test backend, and the Postgres implementation lands in
|
||||
WP-0005 against the interface this workplan fixes.
|
||||
|
||||
## T01 - Route ingestion through the audit backend contract
|
||||
|
||||
```task
|
||||
id: AUDIT-WP-0004-T01
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
`ingestion.py` writes records into SQLite directly and never calls
|
||||
`AuditBackend.emit()`. WP-0002 built a pluggable backend contract with a
|
||||
declared `RetentionPolicy`; WP-0003 built an ingestion path that bypasses it.
|
||||
A 202 currently means "a row exists in a file", against a store that declares
|
||||
no custody class at all.
|
||||
|
||||
Make the ingestion path write through the backend contract, so that accepting
|
||||
an event means a backend with a declared retention policy acknowledged it.
|
||||
Idempotency state and custody write must not be able to diverge — an event
|
||||
recorded as seen but not durably stored is the failure this whole service
|
||||
exists to prevent.
|
||||
|
||||
Add a readiness condition that refuses to serve if the configured backend
|
||||
declares `durable=False`, so the development file backend can never silently
|
||||
become the production sink.
|
||||
|
||||
Done when a successful response is backed by a durable backend
|
||||
acknowledgment, and the mock backend cannot be used to serve production
|
||||
traffic.
|
||||
|
||||
## T02 - Fix error semantics and failure handling
|
||||
|
||||
```task
|
||||
id: AUDIT-WP-0004-T02
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
The handler catches `ValueError`, `TypeError`, `KeyError`, and
|
||||
`JSONDecodeError`. Storage errors do not inherit from those:
|
||||
`sqlite3.OperationalError` on lock contention and `sqlite3.IntegrityError` on
|
||||
the race between the existence check and the insert both escape with
|
||||
`start_response` never called. The auth check sits outside the try block, so a
|
||||
non-ASCII `Authorization` header raises `TypeError` from
|
||||
`hmac.compare_digest` and crashes the request before any handler runs.
|
||||
|
||||
Add a catch-all that always produces a response, map storage and unexpected
|
||||
failures to 503 or 500 rather than letting them surface as dropped
|
||||
connections, and move credential comparison inside the guarded path.
|
||||
|
||||
Correct the status codes: a duplicate event ID carrying a different payload is
|
||||
a conflict and must return 409, not 400. This distinction drives the sender's
|
||||
retry-versus-dead-letter decision, so the full set of responses — accepted,
|
||||
duplicate, conflict, rejected, unavailable — needs to be documented as a
|
||||
contract with the retry semantics each implies.
|
||||
|
||||
Done when no request path can terminate without a response, and every status
|
||||
code the receiver returns is deliberate and documented.
|
||||
|
||||
## T03 - Enforce the isolation properties already claimed
|
||||
|
||||
```task
|
||||
id: AUDIT-WP-0004-T03
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
WP-0003 T01 and T02 both record tenant isolation and cross-tenant claim
|
||||
rejection as done. `normalize()` accepts whatever `tenant` string the caller
|
||||
sends and never checks it against the authenticated identity; no test covers
|
||||
it. Separately, `source` is hardcoded to `"user-engine"`, so the allowlist is
|
||||
unbound to the credential — any holder of the token can claim to be
|
||||
user-engine, and a second sender would require a code change.
|
||||
|
||||
Bind the authenticated identity to the tenants and sources it may write for,
|
||||
and reject claims outside that binding. Drive the binding from configuration
|
||||
rather than literals.
|
||||
|
||||
Done when a credential scoped to one tenant is refused when it claims
|
||||
another, and that refusal is covered by a test.
|
||||
|
||||
## T04 - Settle redaction policy
|
||||
|
||||
```task
|
||||
id: AUDIT-WP-0004-T04
|
||||
status: todo
|
||||
priority: medium
|
||||
```
|
||||
|
||||
`_contains_secret` rejects the entire event when any key name in `data`
|
||||
contains a secret-shaped fragment, and never inspects the envelope. WP-0003
|
||||
T01 describes "redacted data" and the failure matrix tests redaction, but the
|
||||
implementation only rejects.
|
||||
|
||||
Decide whether a secret-shaped field is a rejection or a redaction, and note
|
||||
the consequence: rejection means a legitimate outbox event containing a field
|
||||
named `token_count` is dead-lettered at the sender, and the audit trail loses
|
||||
the event entirely. Extend detection to the envelope, and consider value shape
|
||||
rather than key name alone.
|
||||
|
||||
Whatever is chosen, the contract document and the sender's expectations must
|
||||
agree with the code.
|
||||
|
||||
Done when the redaction rule is explicit, applied to the whole payload, and
|
||||
consistent between the contract and the implementation.
|
||||
|
||||
## T05 - Provide the operator read surface
|
||||
|
||||
```task
|
||||
id: AUDIT-WP-0004-T05
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
There is no way to read anything back. The failure matrix requires dead-letter
|
||||
visibility, operator replay, and correlation lookup; none exist, so that
|
||||
evidence cannot be produced regardless of how the deployment goes.
|
||||
|
||||
Build lookup by event ID and by correlation ID, a dead-letter view of rejected
|
||||
events with their rejection reason, and an operator replay path. Replay must
|
||||
be idempotent against the same durable store, so a replayed event reconciles
|
||||
with the original rather than creating a second custody record.
|
||||
|
||||
Read access is a separate privilege from write access — a sender credential
|
||||
must not be able to read the audit trail back.
|
||||
|
||||
Done when an operator can trace one correlation ID through the system and
|
||||
replay a specific event without creating a duplicate.
|
||||
|
||||
## T06 - Serving layer and observability
|
||||
|
||||
```task
|
||||
id: AUDIT-WP-0004-T06
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
`wsgiref.simple_server` is single-threaded with no request timeout, no
|
||||
graceful shutdown, and no access log. One slow client blocks every sender.
|
||||
It also makes the failure matrix meaningless: "receiver correctly reported
|
||||
unavailable" and "the server stalled" are indistinguishable.
|
||||
|
||||
Move to a production WSGI server with explicit worker and timeout
|
||||
configuration and graceful shutdown on SIGTERM, so in-flight events are not
|
||||
lost on rollout.
|
||||
|
||||
Add structured request logging with a request ID, propagate the correlation ID
|
||||
into logs, and expose counters for accepted, duplicate, conflicted, rejected,
|
||||
and failed events plus write latency. Logs and metrics must never carry event
|
||||
payloads.
|
||||
|
||||
Done when the receiver serves concurrent senders under a bounded timeout,
|
||||
sheds load predictably instead of stalling, and its behaviour is visible from
|
||||
outside.
|
||||
|
||||
## T07 - Close the test gaps
|
||||
|
||||
```task
|
||||
id: AUDIT-WP-0004-T07
|
||||
status: todo
|
||||
priority: medium
|
||||
```
|
||||
|
||||
Ingestion has two tests. Uncovered: oversized and zero-length bodies, absent
|
||||
`Content-Length`, malformed JSON, wrong path and method, the health
|
||||
endpoints, event ID conflict, naive and out-of-range timestamps, storage
|
||||
failure, and persistence across restart.
|
||||
|
||||
`accepted_at` is recorded in local time while every other timestamp is UTC;
|
||||
`datetime.fromisoformat` accepts naive timestamps and arbitrary dates. Fix
|
||||
both and cover them.
|
||||
|
||||
Done when each rejection reason and each failure mode above has a test that
|
||||
asserts the documented status code.
|
||||
|
|
@ -0,0 +1,186 @@
|
|||
---
|
||||
id: AUDIT-WP-0005
|
||||
type: workplan
|
||||
title: "Deploy audit-core on Railiance with durable Postgres custody"
|
||||
domain: infotech
|
||||
repo: audit-core
|
||||
status: proposed
|
||||
owner: codex
|
||||
topic_slug: netkingdom
|
||||
created: "2026-08-10"
|
||||
updated: "2026-08-10"
|
||||
depends_on:
|
||||
- AUDIT-WP-0004
|
||||
- RAPP-POSTGRES-WP-0002
|
||||
- NK-WP-0024
|
||||
---
|
||||
|
||||
# AUDIT-WP-0005 - Postgres store and production deployment
|
||||
|
||||
## Goal
|
||||
|
||||
Put audit-core into production on Railiance with PostgreSQL as the custody
|
||||
store, and prove the delivery path end to end.
|
||||
|
||||
This is first on the critical path: email-connect delivery, OpenBao sender
|
||||
credentials, the user-engine runtime integrations, and the live failure matrix
|
||||
all sit downstream of a receiver that actually holds evidence.
|
||||
|
||||
## Dependencies
|
||||
|
||||
- **AUDIT-WP-0004** — the receiver must route through the backend contract and
|
||||
return deliberate status codes before a Postgres backend is worth writing
|
||||
behind it, and before a failure matrix can distinguish a correct refusal
|
||||
from a crash.
|
||||
- **RAPP-POSTGRES-WP-0002** — the database, the tenancy model, and the
|
||||
credential lane. T01 below can be drafted against the model as soon as
|
||||
RAPP-POSTGRES-WP-0002-T01 settles it; T02 needs a provisioned database.
|
||||
|
||||
## Boundaries
|
||||
|
||||
This workplan owns the Postgres backend implementation, audit-core's
|
||||
deployment, and the live verification. It does not own the database platform,
|
||||
its tenancy model, or its backup machinery — those belong to rapp-postgres.
|
||||
Where audit-core needs a guarantee from the platform, it states the
|
||||
requirement and consumes it; it does not implement it here.
|
||||
|
||||
## T01 - Implement the Postgres audit backend
|
||||
|
||||
```task
|
||||
id: AUDIT-WP-0005-T01
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
Implement `AuditBackend` against PostgreSQL, declaring an honest
|
||||
`RetentionPolicy` — the custody class, retention window, and whether
|
||||
immutability and tamper evidence are genuinely provided rather than aspired
|
||||
to. If the schema does not prevent an operator from silently editing a
|
||||
recorded event, the policy must not claim `immutable`.
|
||||
|
||||
Push idempotency into the database rather than a read-then-write in
|
||||
application code: an insert conflicting on event ID must resolve atomically
|
||||
into duplicate-accepted or conflict, with no window where two concurrent
|
||||
identical events both write. Store the payload hash for conflict detection.
|
||||
|
||||
Handle connection lifecycle properly — pooling, reconnection after a database
|
||||
restart, and a bounded statement timeout so a stalled write surfaces as
|
||||
unavailable instead of hanging the request.
|
||||
|
||||
Own the migrations. The consuming service owns its schema; rapp-postgres owns
|
||||
the space it runs in.
|
||||
|
||||
Done when the backend passes the same contract tests as the existing backends,
|
||||
concurrent duplicate submissions produce exactly one record, and a database
|
||||
restart mid-write does not produce an acknowledged-but-absent event.
|
||||
|
||||
## T02 - Provision storage through the platform lane
|
||||
|
||||
```task
|
||||
id: AUDIT-WP-0005-T02
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
Declare audit-core's database requirement against rapp-postgres and take
|
||||
delivery of a least-privilege runtime role through the OpenBao lane. The
|
||||
runtime role connects; a separate role runs migrations; neither owns more than
|
||||
it needs.
|
||||
|
||||
Verify audit-core's side of the isolation model: the runtime credential
|
||||
reaches audit-core's data and nothing else, and rotation completes without a
|
||||
delivery gap.
|
||||
|
||||
Done when audit-core runs against a provisioned database using a credential it
|
||||
never received as a literal, and rotating that credential does not drop
|
||||
events.
|
||||
|
||||
## T03 - Deploy the receiver
|
||||
|
||||
```task
|
||||
id: AUDIT-WP-0005-T03
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
Publish an immutable image — base pinned by digest, not a mutable tag, and
|
||||
labelled with the build commit — and deploy on railiance01 with a Service,
|
||||
health and readiness probes wired to the checks from WP-0004-T01, resource
|
||||
requests and limits, a restricted security context, and a default-deny
|
||||
NetworkPolicy admitting only user-engine as sender and the operator path for
|
||||
reads.
|
||||
|
||||
State the rollback position: which image digest and which schema version the
|
||||
deployment can return to, and whether the migration in T01 is reversible. A
|
||||
rollback plan that assumes reversible migrations without checking is not a
|
||||
plan.
|
||||
|
||||
The container currently runs as uid 10001 and expects a writable `/data`; once
|
||||
custody is in Postgres that path should carry no durable state at all. Confirm
|
||||
nothing of value is left on the pod filesystem.
|
||||
|
||||
Done when the receiver is reachable only by its declared peers, survives pod
|
||||
restart and rescheduling without loss, and has a tested path back to the
|
||||
previous version.
|
||||
|
||||
## T04 - Migrate existing SQLite records
|
||||
|
||||
```task
|
||||
id: AUDIT-WP-0005-T04
|
||||
status: todo
|
||||
priority: medium
|
||||
```
|
||||
|
||||
Any events accepted by the pre-production SQLite receiver are audit records
|
||||
and cannot simply be dropped. Move them into the Postgres store with their
|
||||
original identifiers, timestamps, and payload hashes intact, or record an
|
||||
explicit decision that they are development artifacts with no custody value.
|
||||
|
||||
Whichever holds, it must be written down — silently discarding accepted audit
|
||||
events is the exact failure this service is meant to make impossible.
|
||||
|
||||
Done when the disposition of every pre-production record is either migrated
|
||||
and verified, or explicitly and justifiably discarded.
|
||||
|
||||
## T05 - Run the live failure matrix
|
||||
|
||||
```task
|
||||
id: AUDIT-WP-0005-T05
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
Exercise the deployed path: successful delivery; receiver timeout and
|
||||
unavailability; bounded user-engine retry against each documented status code;
|
||||
dead-letter visibility; operator replay and duplicate replay; redaction; and
|
||||
correlation lookup. Include a database failover or restart during active
|
||||
ingestion, and a credential rotation during active ingestion.
|
||||
|
||||
The controlling assertion is that one source outbox event produces exactly one
|
||||
durable normalized event across retries, replay, and infrastructure
|
||||
disruption — no loss, no duplication.
|
||||
|
||||
Hand non-secret evidence back to NK-WP-0024.
|
||||
|
||||
Done when the matrix has been run against the deployed system and each
|
||||
outcome recorded, including any case where behaviour differed from the
|
||||
documented contract.
|
||||
|
||||
## T06 - Operational handover
|
||||
|
||||
```task
|
||||
id: AUDIT-WP-0005-T06
|
||||
status: todo
|
||||
priority: medium
|
||||
```
|
||||
|
||||
Document what an operator needs: how to look up an event by correlation ID,
|
||||
how to inspect and replay dead-lettered events, how to rotate the sender
|
||||
credential, what the alert conditions mean, and how to restore audit data from
|
||||
a rapp-postgres backup.
|
||||
|
||||
Verify audit-core's recovery requirement against what rapp-postgres actually
|
||||
provides — the retention window audit-core declares must not exceed the
|
||||
retention the platform guarantees.
|
||||
|
||||
Done when the runbook exists and the restore path has been walked once.
|
||||
Loading…
Add table
Add a link
Reference in a new issue