Rescope WP-0003 and open WP-0004/WP-0005 after pre-deploy review
A pre-deploy review found the receiver is not deployable as built. The
ingestion path never calls AuditBackend.emit(), so a 202 means a SQLite row
exists, not that a backend with a declared retention policy accepted the
event. Storage exceptions escape the handler with start_response never
called. Tenant isolation and source binding are recorded as done but are not
implemented. There is no read, replay, or correlation-lookup surface, so the
failure matrix cannot produce the evidence NK-WP-0024 needs.
WP-0003 closes as finished on its narrowed scope (contract + reference
implementation, T01/T02). T03 and T04 are cancelled with rationale.
WP-0004 covers receiver correctness and hardening, storage-agnostic so it
runs in parallel with the database platform work.
WP-0005 covers the Postgres backend, deployment, SQLite record migration,
and the live failure matrix. Production storage moves from SQLite-on-a-volume
to the Railiance PostgreSQL platform (RAPP-POSTGRES-WP-0002).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-10 12:44:23 +02:00
|
|
|
|
---
|
|
|
|
|
|
id: AUDIT-WP-0005
|
|
|
|
|
|
type: workplan
|
|
|
|
|
|
title: "Deploy audit-core on Railiance with durable Postgres custody"
|
|
|
|
|
|
domain: infotech
|
|
|
|
|
|
repo: audit-core
|
2026-08-13 14:14:22 +02:00
|
|
|
|
status: finished
|
Rescope WP-0003 and open WP-0004/WP-0005 after pre-deploy review
A pre-deploy review found the receiver is not deployable as built. The
ingestion path never calls AuditBackend.emit(), so a 202 means a SQLite row
exists, not that a backend with a declared retention policy accepted the
event. Storage exceptions escape the handler with start_response never
called. Tenant isolation and source binding are recorded as done but are not
implemented. There is no read, replay, or correlation-lookup surface, so the
failure matrix cannot produce the evidence NK-WP-0024 needs.
WP-0003 closes as finished on its narrowed scope (contract + reference
implementation, T01/T02). T03 and T04 are cancelled with rationale.
WP-0004 covers receiver correctness and hardening, storage-agnostic so it
runs in parallel with the database platform work.
WP-0005 covers the Postgres backend, deployment, SQLite record migration,
and the live failure matrix. Production storage moves from SQLite-on-a-volume
to the Railiance PostgreSQL platform (RAPP-POSTGRES-WP-0002).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-10 12:44:23 +02:00
|
|
|
|
owner: codex
|
|
|
|
|
|
topic_slug: netkingdom
|
|
|
|
|
|
created: "2026-08-10"
|
2026-08-13 00:58:49 +02:00
|
|
|
|
updated: "2026-08-13"
|
Rescope WP-0003 and open WP-0004/WP-0005 after pre-deploy review
A pre-deploy review found the receiver is not deployable as built. The
ingestion path never calls AuditBackend.emit(), so a 202 means a SQLite row
exists, not that a backend with a declared retention policy accepted the
event. Storage exceptions escape the handler with start_response never
called. Tenant isolation and source binding are recorded as done but are not
implemented. There is no read, replay, or correlation-lookup surface, so the
failure matrix cannot produce the evidence NK-WP-0024 needs.
WP-0003 closes as finished on its narrowed scope (contract + reference
implementation, T01/T02). T03 and T04 are cancelled with rationale.
WP-0004 covers receiver correctness and hardening, storage-agnostic so it
runs in parallel with the database platform work.
WP-0005 covers the Postgres backend, deployment, SQLite record migration,
and the live failure matrix. Production storage moves from SQLite-on-a-volume
to the Railiance PostgreSQL platform (RAPP-POSTGRES-WP-0002).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-10 12:44:23 +02:00
|
|
|
|
depends_on:
|
|
|
|
|
|
- AUDIT-WP-0004
|
|
|
|
|
|
- RAPP-POSTGRES-WP-0002
|
|
|
|
|
|
- NK-WP-0024
|
Route ingestion through the backend contract; fix error semantics
AUDIT-WP-0004 T01, T02, T07.
T01 - ingestion wrote to SQLite directly and never called the AuditBackend
contract, so a 202 meant a row existed rather than that a backend with a
declared retention policy had accepted the event. Adds IdempotentAuditBackend
to the contract: duplicate detection lives inside the backend so custody and
idempotency state share a transaction and cannot diverge. SQLiteAuditBackend
implements it with WAL, synchronous=FULL and a busy timeout. Ingestion now
refuses any backend declaring durable=False, so the development file backend
cannot silently become the production sink.
The atomicity claim was tested rather than asserted, and the first attempt
failed: with a single shared connection, 16 racing submissions of one event
told two callers they were first. Storage was correct but the response was
not. Fixed with per-thread connections and BEGIN IMMEDIATE around the
insert/read pair, and locked in by a test.
T02 - storage errors previously escaped the handler with start_response never
called, and the auth check sat outside the try block so a non-ASCII
Authorization header crashed the request. Adds a catch-all, maps conflict to
409, backend unavailability to 503 and unexpected faults to 500, and
documents the full response contract with the retry semantics each status
implies, since senders key their behaviour off it.
T07 - ingestion tests 2 -> 23, suite 15 -> 36. accepted_at is now UTC rather
than local time, and naive timestamps are rejected instead of silently
assumed.
Remaining in WP-0004: T03 tenant/source binding, T04 redaction policy,
T05 operator read surface, T06 production serving layer.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-10 14:30:20 +02:00
|
|
|
|
state_hub_workstream_id: "7b24a844-c9f2-4d2d-ac7d-20978bdf6b38"
|
Rescope WP-0003 and open WP-0004/WP-0005 after pre-deploy review
A pre-deploy review found the receiver is not deployable as built. The
ingestion path never calls AuditBackend.emit(), so a 202 means a SQLite row
exists, not that a backend with a declared retention policy accepted the
event. Storage exceptions escape the handler with start_response never
called. Tenant isolation and source binding are recorded as done but are not
implemented. There is no read, replay, or correlation-lookup surface, so the
failure matrix cannot produce the evidence NK-WP-0024 needs.
WP-0003 closes as finished on its narrowed scope (contract + reference
implementation, T01/T02). T03 and T04 are cancelled with rationale.
WP-0004 covers receiver correctness and hardening, storage-agnostic so it
runs in parallel with the database platform work.
WP-0005 covers the Postgres backend, deployment, SQLite record migration,
and the live failure matrix. Production storage moves from SQLite-on-a-volume
to the Railiance PostgreSQL platform (RAPP-POSTGRES-WP-0002).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-10 12:44:23 +02:00
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
|
|
# AUDIT-WP-0005 - Postgres store and production deployment
|
|
|
|
|
|
|
|
|
|
|
|
## Goal
|
|
|
|
|
|
|
|
|
|
|
|
Put audit-core into production on Railiance with PostgreSQL as the custody
|
|
|
|
|
|
store, and prove the delivery path end to end.
|
|
|
|
|
|
|
|
|
|
|
|
This is first on the critical path: email-connect delivery, OpenBao sender
|
|
|
|
|
|
credentials, the user-engine runtime integrations, and the live failure matrix
|
|
|
|
|
|
all sit downstream of a receiver that actually holds evidence.
|
|
|
|
|
|
|
|
|
|
|
|
## Dependencies
|
|
|
|
|
|
|
|
|
|
|
|
- **AUDIT-WP-0004** — the receiver must route through the backend contract and
|
|
|
|
|
|
return deliberate status codes before a Postgres backend is worth writing
|
|
|
|
|
|
behind it, and before a failure matrix can distinguish a correct refusal
|
|
|
|
|
|
from a crash.
|
|
|
|
|
|
- **RAPP-POSTGRES-WP-0002** — the database, the tenancy model, and the
|
|
|
|
|
|
credential lane. T01 below can be drafted against the model as soon as
|
|
|
|
|
|
RAPP-POSTGRES-WP-0002-T01 settles it; T02 needs a provisioned database.
|
|
|
|
|
|
|
|
|
|
|
|
## Boundaries
|
|
|
|
|
|
|
|
|
|
|
|
This workplan owns the Postgres backend implementation, audit-core's
|
|
|
|
|
|
deployment, and the live verification. It does not own the database platform,
|
|
|
|
|
|
its tenancy model, or its backup machinery — those belong to rapp-postgres.
|
|
|
|
|
|
Where audit-core needs a guarantee from the platform, it states the
|
|
|
|
|
|
requirement and consumes it; it does not implement it here.
|
|
|
|
|
|
|
|
|
|
|
|
## T01 - Implement the Postgres audit backend
|
|
|
|
|
|
|
|
|
|
|
|
```task
|
|
|
|
|
|
id: AUDIT-WP-0005-T01
|
2026-08-10 17:09:46 +02:00
|
|
|
|
status: done
|
Rescope WP-0003 and open WP-0004/WP-0005 after pre-deploy review
A pre-deploy review found the receiver is not deployable as built. The
ingestion path never calls AuditBackend.emit(), so a 202 means a SQLite row
exists, not that a backend with a declared retention policy accepted the
event. Storage exceptions escape the handler with start_response never
called. Tenant isolation and source binding are recorded as done but are not
implemented. There is no read, replay, or correlation-lookup surface, so the
failure matrix cannot produce the evidence NK-WP-0024 needs.
WP-0003 closes as finished on its narrowed scope (contract + reference
implementation, T01/T02). T03 and T04 are cancelled with rationale.
WP-0004 covers receiver correctness and hardening, storage-agnostic so it
runs in parallel with the database platform work.
WP-0005 covers the Postgres backend, deployment, SQLite record migration,
and the live failure matrix. Production storage moves from SQLite-on-a-volume
to the Railiance PostgreSQL platform (RAPP-POSTGRES-WP-0002).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-10 12:44:23 +02:00
|
|
|
|
priority: high
|
Route ingestion through the backend contract; fix error semantics
AUDIT-WP-0004 T01, T02, T07.
T01 - ingestion wrote to SQLite directly and never called the AuditBackend
contract, so a 202 meant a row existed rather than that a backend with a
declared retention policy had accepted the event. Adds IdempotentAuditBackend
to the contract: duplicate detection lives inside the backend so custody and
idempotency state share a transaction and cannot diverge. SQLiteAuditBackend
implements it with WAL, synchronous=FULL and a busy timeout. Ingestion now
refuses any backend declaring durable=False, so the development file backend
cannot silently become the production sink.
The atomicity claim was tested rather than asserted, and the first attempt
failed: with a single shared connection, 16 racing submissions of one event
told two callers they were first. Storage was correct but the response was
not. Fixed with per-thread connections and BEGIN IMMEDIATE around the
insert/read pair, and locked in by a test.
T02 - storage errors previously escaped the handler with start_response never
called, and the auth check sat outside the try block so a non-ASCII
Authorization header crashed the request. Adds a catch-all, maps conflict to
409, backend unavailability to 503 and unexpected faults to 500, and
documents the full response contract with the retry semantics each status
implies, since senders key their behaviour off it.
T07 - ingestion tests 2 -> 23, suite 15 -> 36. accepted_at is now UTC rather
than local time, and naive timestamps are rejected instead of silently
assumed.
Remaining in WP-0004: T03 tenant/source binding, T04 redaction policy,
T05 operator read surface, T06 production serving layer.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-10 14:30:20 +02:00
|
|
|
|
state_hub_task_id: "b1601d0b-922a-40f7-92c0-ea06af6c4468"
|
Rescope WP-0003 and open WP-0004/WP-0005 after pre-deploy review
A pre-deploy review found the receiver is not deployable as built. The
ingestion path never calls AuditBackend.emit(), so a 202 means a SQLite row
exists, not that a backend with a declared retention policy accepted the
event. Storage exceptions escape the handler with start_response never
called. Tenant isolation and source binding are recorded as done but are not
implemented. There is no read, replay, or correlation-lookup surface, so the
failure matrix cannot produce the evidence NK-WP-0024 needs.
WP-0003 closes as finished on its narrowed scope (contract + reference
implementation, T01/T02). T03 and T04 are cancelled with rationale.
WP-0004 covers receiver correctness and hardening, storage-agnostic so it
runs in parallel with the database platform work.
WP-0005 covers the Postgres backend, deployment, SQLite record migration,
and the live failure matrix. Production storage moves from SQLite-on-a-volume
to the Railiance PostgreSQL platform (RAPP-POSTGRES-WP-0002).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-10 12:44:23 +02:00
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
Implement `AuditBackend` against PostgreSQL, declaring an honest
|
|
|
|
|
|
`RetentionPolicy` — the custody class, retention window, and whether
|
|
|
|
|
|
immutability and tamper evidence are genuinely provided rather than aspired
|
|
|
|
|
|
to. If the schema does not prevent an operator from silently editing a
|
|
|
|
|
|
recorded event, the policy must not claim `immutable`.
|
|
|
|
|
|
|
|
|
|
|
|
Push idempotency into the database rather than a read-then-write in
|
|
|
|
|
|
application code: an insert conflicting on event ID must resolve atomically
|
|
|
|
|
|
into duplicate-accepted or conflict, with no window where two concurrent
|
|
|
|
|
|
identical events both write. Store the payload hash for conflict detection.
|
|
|
|
|
|
|
|
|
|
|
|
Handle connection lifecycle properly — pooling, reconnection after a database
|
|
|
|
|
|
restart, and a bounded statement timeout so a stalled write surfaces as
|
|
|
|
|
|
unavailable instead of hanging the request.
|
|
|
|
|
|
|
|
|
|
|
|
Own the migrations. The consuming service owns its schema; rapp-postgres owns
|
|
|
|
|
|
the space it runs in.
|
|
|
|
|
|
|
|
|
|
|
|
Done when the backend passes the same contract tests as the existing backends,
|
|
|
|
|
|
concurrent duplicate submissions produce exactly one record, and a database
|
|
|
|
|
|
restart mid-write does not produce an acknowledged-but-absent event.
|
|
|
|
|
|
|
2026-08-10 17:09:46 +02:00
|
|
|
|
Done 2026-08-10, built and verified against PostgreSQL 16 locally in Docker —
|
|
|
|
|
|
the Railiance cluster was not needed for any of it.
|
|
|
|
|
|
|
|
|
|
|
|
`tests/test_backend_conformance.py` is a single suite run against every
|
|
|
|
|
|
backend, so "the Postgres backend is done" means it satisfies the same
|
|
|
|
|
|
contract SQLite already does rather than having its own tests that happen to
|
|
|
|
|
|
be green. It skips cleanly when no server is reachable; `make pg-test-up` and
|
|
|
|
|
|
`make test-pg` run it. Suite 50 -> 71.
|
|
|
|
|
|
|
|
|
|
|
|
`RetentionPolicy` declares `immutable=True`, and that is earned: migration
|
|
|
|
|
|
0002 installs a trigger rejecting UPDATE and DELETE on the events table, so a
|
|
|
|
|
|
leaked runtime credential can append but cannot rewrite or erase the trail.
|
|
|
|
|
|
This materially narrows the residual risk ADR-0001 §5 called out — a leaked
|
|
|
|
|
|
credential could previously forge the audit record. `tamper_evidence` stays
|
|
|
|
|
|
False, because nothing here would *prove* a database owner had dropped the
|
|
|
|
|
|
trigger; hash-chaining or external anchoring would be needed and is not
|
|
|
|
|
|
implemented.
|
|
|
|
|
|
|
|
|
|
|
|
Migrations are ordered, recorded in `schema_migrations`, and idempotent.
|
|
|
|
|
|
Replay reconciles rather than duplicating — the piece deferred out of
|
|
|
|
|
|
WP-0004-T05 — and is tested to leave exactly one custody record.
|
|
|
|
|
|
|
|
|
|
|
|
Backend selection is by `AUDIT_CORE_DATABASE_URL`; falling back to SQLite logs
|
|
|
|
|
|
a warning, so a deployment that lost its URL is visible rather than quietly
|
|
|
|
|
|
running on the wrong store. End-to-end smoke through waitress against Postgres
|
|
|
|
|
|
confirmed accept, duplicate, cross-tenant refusal, read/write privilege
|
|
|
|
|
|
separation, visible redaction, secret-finding counters, and readiness
|
|
|
|
|
|
reporting `custody_class=archive`.
|
|
|
|
|
|
|
|
|
|
|
|
Also fixed: `audit_core.ingestion` had no `__main__` guard, so `python -m
|
|
|
|
|
|
audit_core.ingestion` silently did nothing.
|
|
|
|
|
|
|
|
|
|
|
|
Not covered here: behaviour across a real database failover, which needs the
|
|
|
|
|
|
cluster and belongs to T05.
|
|
|
|
|
|
|
Rescope WP-0003 and open WP-0004/WP-0005 after pre-deploy review
A pre-deploy review found the receiver is not deployable as built. The
ingestion path never calls AuditBackend.emit(), so a 202 means a SQLite row
exists, not that a backend with a declared retention policy accepted the
event. Storage exceptions escape the handler with start_response never
called. Tenant isolation and source binding are recorded as done but are not
implemented. There is no read, replay, or correlation-lookup surface, so the
failure matrix cannot produce the evidence NK-WP-0024 needs.
WP-0003 closes as finished on its narrowed scope (contract + reference
implementation, T01/T02). T03 and T04 are cancelled with rationale.
WP-0004 covers receiver correctness and hardening, storage-agnostic so it
runs in parallel with the database platform work.
WP-0005 covers the Postgres backend, deployment, SQLite record migration,
and the live failure matrix. Production storage moves from SQLite-on-a-volume
to the Railiance PostgreSQL platform (RAPP-POSTGRES-WP-0002).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-10 12:44:23 +02:00
|
|
|
|
## T02 - Provision storage through the platform lane
|
|
|
|
|
|
|
|
|
|
|
|
```task
|
|
|
|
|
|
id: AUDIT-WP-0005-T02
|
2026-08-13 12:36:25 +02:00
|
|
|
|
status: done
|
Rescope WP-0003 and open WP-0004/WP-0005 after pre-deploy review
A pre-deploy review found the receiver is not deployable as built. The
ingestion path never calls AuditBackend.emit(), so a 202 means a SQLite row
exists, not that a backend with a declared retention policy accepted the
event. Storage exceptions escape the handler with start_response never
called. Tenant isolation and source binding are recorded as done but are not
implemented. There is no read, replay, or correlation-lookup surface, so the
failure matrix cannot produce the evidence NK-WP-0024 needs.
WP-0003 closes as finished on its narrowed scope (contract + reference
implementation, T01/T02). T03 and T04 are cancelled with rationale.
WP-0004 covers receiver correctness and hardening, storage-agnostic so it
runs in parallel with the database platform work.
WP-0005 covers the Postgres backend, deployment, SQLite record migration,
and the live failure matrix. Production storage moves from SQLite-on-a-volume
to the Railiance PostgreSQL platform (RAPP-POSTGRES-WP-0002).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-10 12:44:23 +02:00
|
|
|
|
priority: high
|
Route ingestion through the backend contract; fix error semantics
AUDIT-WP-0004 T01, T02, T07.
T01 - ingestion wrote to SQLite directly and never called the AuditBackend
contract, so a 202 meant a row existed rather than that a backend with a
declared retention policy had accepted the event. Adds IdempotentAuditBackend
to the contract: duplicate detection lives inside the backend so custody and
idempotency state share a transaction and cannot diverge. SQLiteAuditBackend
implements it with WAL, synchronous=FULL and a busy timeout. Ingestion now
refuses any backend declaring durable=False, so the development file backend
cannot silently become the production sink.
The atomicity claim was tested rather than asserted, and the first attempt
failed: with a single shared connection, 16 racing submissions of one event
told two callers they were first. Storage was correct but the response was
not. Fixed with per-thread connections and BEGIN IMMEDIATE around the
insert/read pair, and locked in by a test.
T02 - storage errors previously escaped the handler with start_response never
called, and the auth check sat outside the try block so a non-ASCII
Authorization header crashed the request. Adds a catch-all, maps conflict to
409, backend unavailability to 503 and unexpected faults to 500, and
documents the full response contract with the retry semantics each status
implies, since senders key their behaviour off it.
T07 - ingestion tests 2 -> 23, suite 15 -> 36. accepted_at is now UTC rather
than local time, and naive timestamps are rejected instead of silently
assumed.
Remaining in WP-0004: T03 tenant/source binding, T04 redaction policy,
T05 operator read surface, T06 production serving layer.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-10 14:30:20 +02:00
|
|
|
|
state_hub_task_id: "831b2472-0d80-4369-a5e3-eb08ef3526b1"
|
Rescope WP-0003 and open WP-0004/WP-0005 after pre-deploy review
A pre-deploy review found the receiver is not deployable as built. The
ingestion path never calls AuditBackend.emit(), so a 202 means a SQLite row
exists, not that a backend with a declared retention policy accepted the
event. Storage exceptions escape the handler with start_response never
called. Tenant isolation and source binding are recorded as done but are not
implemented. There is no read, replay, or correlation-lookup surface, so the
failure matrix cannot produce the evidence NK-WP-0024 needs.
WP-0003 closes as finished on its narrowed scope (contract + reference
implementation, T01/T02). T03 and T04 are cancelled with rationale.
WP-0004 covers receiver correctness and hardening, storage-agnostic so it
runs in parallel with the database platform work.
WP-0005 covers the Postgres backend, deployment, SQLite record migration,
and the live failure matrix. Production storage moves from SQLite-on-a-volume
to the Railiance PostgreSQL platform (RAPP-POSTGRES-WP-0002).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-10 12:44:23 +02:00
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
Declare audit-core's database requirement against rapp-postgres and take
|
|
|
|
|
|
delivery of a least-privilege runtime role through the OpenBao lane. The
|
|
|
|
|
|
runtime role connects; a separate role runs migrations; neither owns more than
|
|
|
|
|
|
it needs.
|
|
|
|
|
|
|
|
|
|
|
|
Verify audit-core's side of the isolation model: the runtime credential
|
|
|
|
|
|
reaches audit-core's data and nothing else, and rotation completes without a
|
|
|
|
|
|
delivery gap.
|
|
|
|
|
|
|
|
|
|
|
|
Done when audit-core runs against a provisioned database using a credential it
|
|
|
|
|
|
never received as a literal, and rotating that credential does not drop
|
|
|
|
|
|
events.
|
|
|
|
|
|
|
Deliver database credentials as a rotatable mounted directory
AUDIT-WP-0005-T02 (progress). rapp-postgres has landed platform-pg with the
audit_core database, roles, and dynamic credential provisioning, so
audit-core's side is now built against it.
In-cluster delivery is a mounted directory rather than environment variables.
A dynamic lease rotates while the pod runs and an env var is fixed at process
start, so env delivery would force a restart on every rotation - and every
restart is a delivery gap, which is what this task forbids.
CredentialDirectory is re-read on every connection attempt via psycopg_pool's
callable kwargs, so a rotated lease takes effect with no restart. Rotation is
logged by password fingerprint, never by value.
deploy/externalsecrets.yaml follows the ClusterSecretStore -> ExternalSecret ->
Secret pattern already used by activity-core and rapp-qonto, at a 15m refresh
rather than the default 1h since the interval bounds how long a revoked lease
can stay mounted. All manifests validated --dry-run=server --validate=strict.
The rotation test was initially vacuous: it passed against a deliberately naive
implementation that read credentials once at startup, because pooled sessions
stay authenticated after a password change and nothing forced a reconnect. It
now terminates the role's sessions first, and is verified to fail against the
naive implementation and pass against the real one. Tests 82 -> 84.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-12 01:36:28 +02:00
|
|
|
|
Progress 2026-08-12: rapp-postgres has landed the platform — `platform-pg` is
|
|
|
|
|
|
healthy, the `audit_core` database exists with `audit_core_owner`/`_migrate`/
|
|
|
|
|
|
`_app`, and RAPP-POSTGRES-WP-0002-T04 delivered dynamic credential
|
|
|
|
|
|
provisioning. audit-core's side is now built against it.
|
|
|
|
|
|
|
|
|
|
|
|
**Two consumption paths, both supported.** The rapp-postgres playbook has the
|
|
|
|
|
|
`railiance-platform` broker inject `PGUSER`/`PGPASSWORD`/`PGHOST`/`PGPORT`/
|
|
|
|
|
|
`PGDATABASE` into a child process; audit-core previously accepted only
|
|
|
|
|
|
`AUDIT_CORE_DATABASE_URL`, which would have meant assembling a DSN by hand from
|
|
|
|
|
|
those variables and putting the credential back into audit-core's own
|
|
|
|
|
|
configuration — the thing the lane exists to prevent. An empty conninfo now
|
|
|
|
|
|
lets libpq read them directly.
|
|
|
|
|
|
|
|
|
|
|
|
**In-cluster delivery is a mounted directory, not environment variables**
|
|
|
|
|
|
(`AUDIT_CORE_CREDENTIAL_DIR`, decision by Bernd). This is the design point:
|
|
|
|
|
|
a dynamic lease rotates *while the pod runs*, and an environment variable is
|
|
|
|
|
|
fixed at process start. Env delivery would force a restart on every rotation,
|
|
|
|
|
|
and every restart is a delivery gap — precisely what this task forbids.
|
|
|
|
|
|
`audit_core.credentials.CredentialDirectory` is re-read on every connection
|
|
|
|
|
|
attempt, exploiting psycopg_pool's support for a callable `kwargs`, so a
|
|
|
|
|
|
rotated lease is picked up by the next connection with no restart.
|
|
|
|
|
|
|
|
|
|
|
|
Rotation is logged by password fingerprint, never by value.
|
|
|
|
|
|
|
|
|
|
|
|
`deploy/externalsecrets.yaml` follows the ClusterSecretStore → ExternalSecret →
|
|
|
|
|
|
Secret pattern already used by activity-core and rapp-qonto, with a 15m refresh
|
|
|
|
|
|
rather than the platform default 1h — the interval bounds how long a revoked
|
|
|
|
|
|
lease can stay mounted.
|
|
|
|
|
|
|
|
|
|
|
|
The rotation test was initially **vacuous** and was caught by running it
|
|
|
|
|
|
against a deliberately naive implementation that read credentials once at
|
|
|
|
|
|
startup: it passed. Existing pooled sessions stay authenticated after a
|
|
|
|
|
|
password change, so nothing forced a reconnect. The test now terminates the
|
|
|
|
|
|
role's sessions before checking, and has been verified to fail against the
|
|
|
|
|
|
naive implementation with an authentication error and pass against the real
|
|
|
|
|
|
one. Tests 82 -> 84.
|
|
|
|
|
|
|
|
|
|
|
|
**Security finding, fixed.** `scripts/isolation-test.sh` in rapp-postgres left
|
|
|
|
|
|
three login roles on the production cluster after its remote run —
|
|
|
|
|
|
`audit_login` (member of `audit_core_app`, so full read/write on the audit
|
|
|
|
|
|
trail), `audit_migrate_login`, and `probe_login` — with the committed literal
|
|
|
|
|
|
password `probe` and no expiry. Roles dropped from `platform-pg` on 2026-08-12
|
|
|
|
|
|
after confirming no active sessions; the group roles are untouched and remain
|
|
|
|
|
|
NOLOGIN. The harness now uses a per-run random password, `VALID UNTIL` one
|
|
|
|
|
|
hour, and an exit trap that drops the roles in every mode including on failure.
|
|
|
|
|
|
|
2026-08-13 00:58:49 +02:00
|
|
|
|
Progress 2026-08-13: the in-cluster delivery path is now honest about what
|
|
|
|
|
|
kind of secret it is pulling.
|
|
|
|
|
|
|
|
|
|
|
|
The previous ExternalSecret pointed at KV
|
|
|
|
|
|
`platform/workloads/audit-core/database/audit-core-runtime`. That path does
|
|
|
|
|
|
not exist, and copying a dynamic lease into KV would freeze it — the same
|
|
|
|
|
|
delivery-gap problem the mounted directory exists to avoid. Database
|
|
|
|
|
|
credentials now come from ESO `VaultDynamicSecret` against
|
|
|
|
|
|
`database/creds/audit-core-runtime` and `database/creds/audit-core-migration`,
|
|
|
|
|
|
refreshed every 15m into one-file-per-field Secrets. The sender registry
|
|
|
|
|
|
stays on ClusterSecretStore `openbao-audit-core` (KV), which is the right
|
|
|
|
|
|
tool for a document.
|
|
|
|
|
|
|
|
|
|
|
|
A second finding: `PostgresAuditBackend` applied schema on every process
|
|
|
|
|
|
start. The runtime role cannot `CREATE TABLE` (isolation-test, ADR-0001).
|
|
|
|
|
|
Production now sets `AUDIT_CORE_AUTO_MIGRATE=0`; `python -m audit_core
|
|
|
|
|
|
migrate` runs as Job `audit-core-migrate` with the migration lease.
|
|
|
|
|
|
|
|
|
|
|
|
Package side in this repo: `deploy/clustersecretstore.yaml`,
|
|
|
|
|
|
`deploy/vaultdynamicsecrets.yaml`, `deploy/migrate-job.yaml`,
|
|
|
|
|
|
`scripts/openbao-eso-token-apply.sh`. Platform side drafted in
|
|
|
|
|
|
`railiance-platform` (`openbao/policies/external-secrets-audit-core.hcl` and
|
|
|
|
|
|
the ClusterSecretStore add-on). The OpenBao database roles themselves were
|
|
|
|
|
|
already delivered by RAPP-POSTGRES-WP-0002-T04.
|
|
|
|
|
|
|
2026-08-13 10:27:13 +02:00
|
|
|
|
Progress 2026-08-13 (lane, not token-paste): founder `bao kv put` is retired
|
|
|
|
|
|
as the T02 delivery path. Senders were minted in-cluster into Secret
|
|
|
|
|
|
`audit-core-senders`. Database leases come from ClusterSecretStore
|
|
|
|
|
|
`openbao-audit-core-database` (one extract = one lease). Both ExternalSecrets
|
|
|
|
|
|
are `SecretSynced`. ops-mason plan
|
|
|
|
|
|
`audit-core-openbao-runtime-custody` is `reviewed` and waiting on the one
|
|
|
|
|
|
founder approve to replace the interim ESO token with an AppRole. Catalog
|
|
|
|
|
|
draft: `warden route find "audit-core senders" --all`.
|
|
|
|
|
|
|
2026-08-13 10:42:59 +02:00
|
|
|
|
Mason plan approved 2026-08-13 and built: AppRole
|
|
|
|
|
|
`external-secrets-audit-core`, Secret `openbao-audit-core-approle`, both
|
|
|
|
|
|
ClusterSecretStores on AppRole and `Valid`. Interim ESO token Secret
|
|
|
|
|
|
removed. Receiver stayed Ready.
|
|
|
|
|
|
|
2026-08-13 12:36:25 +02:00
|
|
|
|
Live rotation 2026-08-13: ESO refresh while Ready; accept 202 before and
|
|
|
|
|
|
after; both events readable; `/readyz` stayed `archive`. Fingerprint log
|
|
|
|
|
|
was not seen on that refresh (lease may have been reused).
|
|
|
|
|
|
|
|
|
|
|
|
Findings fixed on the way:
|
|
|
|
|
|
|
|
|
|
|
|
- ESO AppRole login+discard **revokes** `database/creds` leases. Working
|
|
|
|
|
|
client is a renewable orphan token. AppRole remains for a later
|
|
|
|
|
|
lease-aware generator.
|
|
|
|
|
|
- Mounted Secret reads must use Kubernetes `..data` or they can tear
|
|
|
|
|
|
across two leases.
|
|
|
|
|
|
- Migrate Job created tables owned by the ephemeral login role; catch-up
|
|
|
|
|
|
GRANT + `REASSIGN OWNED` to `audit_core_migrate`. `SET ROLE
|
|
|
|
|
|
audit_core_migrate` is now in migrate().
|
|
|
|
|
|
|
|
|
|
|
|
T02 done for the provisioned-lane + no-delivery-gap claim. Empty senders
|
|
|
|
|
|
KV wrap-migrate is follow-on, not a founder paste.
|
Deliver database credentials as a rotatable mounted directory
AUDIT-WP-0005-T02 (progress). rapp-postgres has landed platform-pg with the
audit_core database, roles, and dynamic credential provisioning, so
audit-core's side is now built against it.
In-cluster delivery is a mounted directory rather than environment variables.
A dynamic lease rotates while the pod runs and an env var is fixed at process
start, so env delivery would force a restart on every rotation - and every
restart is a delivery gap, which is what this task forbids.
CredentialDirectory is re-read on every connection attempt via psycopg_pool's
callable kwargs, so a rotated lease takes effect with no restart. Rotation is
logged by password fingerprint, never by value.
deploy/externalsecrets.yaml follows the ClusterSecretStore -> ExternalSecret ->
Secret pattern already used by activity-core and rapp-qonto, at a 15m refresh
rather than the default 1h since the interval bounds how long a revoked lease
can stay mounted. All manifests validated --dry-run=server --validate=strict.
The rotation test was initially vacuous: it passed against a deliberately naive
implementation that read credentials once at startup, because pooled sessions
stay authenticated after a password change and nothing forced a reconnect. It
now terminates the role's sessions first, and is verified to fail against the
naive implementation and pass against the real one. Tests 82 -> 84.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-12 01:36:28 +02:00
|
|
|
|
|
2026-08-14 20:12:33 +02:00
|
|
|
|
Follow-on completed 2026-08-14: the live registry was streamed directly from
|
|
|
|
|
|
Kubernetes into `platform/workloads/audit-core/senders` under an attended
|
|
|
|
|
|
KeyCape/MFA platform-admin session. ExternalSecret `audit-core-senders` is
|
|
|
|
|
|
`SecretSynced`, owns the derived Secret, and preserved its pre-migration
|
|
|
|
|
|
checksum. The receiver rolled successfully and `/readyz` remained durable
|
|
|
|
|
|
`archive`; no sender token was printed or staged.
|
|
|
|
|
|
|
Rescope WP-0003 and open WP-0004/WP-0005 after pre-deploy review
A pre-deploy review found the receiver is not deployable as built. The
ingestion path never calls AuditBackend.emit(), so a 202 means a SQLite row
exists, not that a backend with a declared retention policy accepted the
event. Storage exceptions escape the handler with start_response never
called. Tenant isolation and source binding are recorded as done but are not
implemented. There is no read, replay, or correlation-lookup surface, so the
failure matrix cannot produce the evidence NK-WP-0024 needs.
WP-0003 closes as finished on its narrowed scope (contract + reference
implementation, T01/T02). T03 and T04 are cancelled with rationale.
WP-0004 covers receiver correctness and hardening, storage-agnostic so it
runs in parallel with the database platform work.
WP-0005 covers the Postgres backend, deployment, SQLite record migration,
and the live failure matrix. Production storage moves from SQLite-on-a-volume
to the Railiance PostgreSQL platform (RAPP-POSTGRES-WP-0002).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-10 12:44:23 +02:00
|
|
|
|
## T03 - Deploy the receiver
|
|
|
|
|
|
|
|
|
|
|
|
```task
|
|
|
|
|
|
id: AUDIT-WP-0005-T03
|
2026-08-13 12:36:25 +02:00
|
|
|
|
status: done
|
Rescope WP-0003 and open WP-0004/WP-0005 after pre-deploy review
A pre-deploy review found the receiver is not deployable as built. The
ingestion path never calls AuditBackend.emit(), so a 202 means a SQLite row
exists, not that a backend with a declared retention policy accepted the
event. Storage exceptions escape the handler with start_response never
called. Tenant isolation and source binding are recorded as done but are not
implemented. There is no read, replay, or correlation-lookup surface, so the
failure matrix cannot produce the evidence NK-WP-0024 needs.
WP-0003 closes as finished on its narrowed scope (contract + reference
implementation, T01/T02). T03 and T04 are cancelled with rationale.
WP-0004 covers receiver correctness and hardening, storage-agnostic so it
runs in parallel with the database platform work.
WP-0005 covers the Postgres backend, deployment, SQLite record migration,
and the live failure matrix. Production storage moves from SQLite-on-a-volume
to the Railiance PostgreSQL platform (RAPP-POSTGRES-WP-0002).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-10 12:44:23 +02:00
|
|
|
|
priority: high
|
Route ingestion through the backend contract; fix error semantics
AUDIT-WP-0004 T01, T02, T07.
T01 - ingestion wrote to SQLite directly and never called the AuditBackend
contract, so a 202 meant a row existed rather than that a backend with a
declared retention policy had accepted the event. Adds IdempotentAuditBackend
to the contract: duplicate detection lives inside the backend so custody and
idempotency state share a transaction and cannot diverge. SQLiteAuditBackend
implements it with WAL, synchronous=FULL and a busy timeout. Ingestion now
refuses any backend declaring durable=False, so the development file backend
cannot silently become the production sink.
The atomicity claim was tested rather than asserted, and the first attempt
failed: with a single shared connection, 16 racing submissions of one event
told two callers they were first. Storage was correct but the response was
not. Fixed with per-thread connections and BEGIN IMMEDIATE around the
insert/read pair, and locked in by a test.
T02 - storage errors previously escaped the handler with start_response never
called, and the auth check sat outside the try block so a non-ASCII
Authorization header crashed the request. Adds a catch-all, maps conflict to
409, backend unavailability to 503 and unexpected faults to 500, and
documents the full response contract with the retry semantics each status
implies, since senders key their behaviour off it.
T07 - ingestion tests 2 -> 23, suite 15 -> 36. accepted_at is now UTC rather
than local time, and naive timestamps are rejected instead of silently
assumed.
Remaining in WP-0004: T03 tenant/source binding, T04 redaction policy,
T05 operator read surface, T06 production serving layer.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-10 14:30:20 +02:00
|
|
|
|
state_hub_task_id: "598af2ac-e772-4a4e-9a65-dde9d4ca167f"
|
Rescope WP-0003 and open WP-0004/WP-0005 after pre-deploy review
A pre-deploy review found the receiver is not deployable as built. The
ingestion path never calls AuditBackend.emit(), so a 202 means a SQLite row
exists, not that a backend with a declared retention policy accepted the
event. Storage exceptions escape the handler with start_response never
called. Tenant isolation and source binding are recorded as done but are not
implemented. There is no read, replay, or correlation-lookup surface, so the
failure matrix cannot produce the evidence NK-WP-0024 needs.
WP-0003 closes as finished on its narrowed scope (contract + reference
implementation, T01/T02). T03 and T04 are cancelled with rationale.
WP-0004 covers receiver correctness and hardening, storage-agnostic so it
runs in parallel with the database platform work.
WP-0005 covers the Postgres backend, deployment, SQLite record migration,
and the live failure matrix. Production storage moves from SQLite-on-a-volume
to the Railiance PostgreSQL platform (RAPP-POSTGRES-WP-0002).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-10 12:44:23 +02:00
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
Publish an immutable image — base pinned by digest, not a mutable tag, and
|
|
|
|
|
|
labelled with the build commit — and deploy on railiance01 with a Service,
|
|
|
|
|
|
health and readiness probes wired to the checks from WP-0004-T01, resource
|
|
|
|
|
|
requests and limits, a restricted security context, and a default-deny
|
|
|
|
|
|
NetworkPolicy admitting only user-engine as sender and the operator path for
|
|
|
|
|
|
reads.
|
|
|
|
|
|
|
|
|
|
|
|
State the rollback position: which image digest and which schema version the
|
|
|
|
|
|
deployment can return to, and whether the migration in T01 is reversible. A
|
|
|
|
|
|
rollback plan that assumes reversible migrations without checking is not a
|
|
|
|
|
|
plan.
|
|
|
|
|
|
|
|
|
|
|
|
The container currently runs as uid 10001 and expects a writable `/data`; once
|
|
|
|
|
|
custody is in Postgres that path should carry no durable state at all. Confirm
|
|
|
|
|
|
nothing of value is left on the pod filesystem.
|
|
|
|
|
|
|
|
|
|
|
|
Done when the receiver is reachable only by its declared peers, survives pod
|
|
|
|
|
|
restart and rescheduling without loss, and has a tested path back to the
|
|
|
|
|
|
previous version.
|
|
|
|
|
|
|
Add deployment manifests, custody-class guard and request counters
AUDIT-WP-0005-T03 (progress). Manifests validated --dry-run=server
--validate=strict against railiance01; not applied, since deployment is gated
on RAPP-POSTGRES-WP-0002 and T02 credentials. Nothing here mutates the cluster.
Conventions read off the deployed user-engine workload rather than invented:
digest-pinned image from forgejo.coulomb.social, runAsNonRoot with
RuntimeDefault seccomp, no privilege escalation, all capabilities dropped,
readOnlyRootFilesystem, probes on a named http port, same resource envelope.
The namespace carries railiance.io/postgres-client: platform-pg, which is what
platform-pg-consumer-ingress in rapp-postgres admits; without that label the
pod cannot reach the database at all.
NetworkPolicies default-deny both directions, then permit ingress from the
user-engine namespace only, a separately labelled operator read path, and
egress to PostgreSQL in databases plus DNS.
Three decisions worth naming. Liveness is /healthz while readiness is /readyz,
so a database outage drops the pod from the Service rather than restarting it
in a loop. readOnlyRootFilesystem enforces the empty-filesystem property rather
than trusting it, so the SQLite fallback physically cannot accumulate audit
records on ephemeral storage. AUDIT_CORE_REQUIRE_CUSTODY_CLASS=archive makes a
missing database URL a startup failure instead of a silent downgrade to the
development store.
Counters deferred from WP-0004-T06 are exposed as JSON at /v1/stats behind the
read privilege, not as Prometheus exposition format: the cluster runs no
Prometheus, no ServiceMonitor CRD and no other scrape target, so an exposition
endpoint would target a scrape path that does not exist. Usable with curl now
and a small step from /metrics later.
Tests 77 -> 80.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-10 17:42:43 +02:00
|
|
|
|
Progress 2026-08-10: manifests written and validated `--dry-run=server
|
|
|
|
|
|
--validate=strict` against railiance01. Not applied — deployment is gated on
|
|
|
|
|
|
RAPP-POSTGRES-WP-0002 and on T02's credentials, and nothing here mutates the
|
|
|
|
|
|
cluster.
|
|
|
|
|
|
|
|
|
|
|
|
Conventions were read off the deployed `user-engine` workload rather than
|
|
|
|
|
|
invented: digest-pinned image from `forgejo.coulomb.social/coulomb/<name>`,
|
|
|
|
|
|
`runAsNonRoot` + `RuntimeDefault` seccomp, `allowPrivilegeEscalation: false`,
|
|
|
|
|
|
all capabilities dropped, `readOnlyRootFilesystem: true`, probes on a named
|
|
|
|
|
|
`http` port, and the same resource envelope.
|
|
|
|
|
|
|
|
|
|
|
|
`deploy/audit-core.yaml` — namespace, Service, Deployment. The namespace
|
|
|
|
|
|
carries `railiance.io/postgres-client: platform-pg`, which is what
|
|
|
|
|
|
`platform-pg-consumer-ingress` in rapp-postgres admits; without that label the
|
|
|
|
|
|
pod cannot reach the database at all.
|
|
|
|
|
|
|
|
|
|
|
|
`deploy/networkpolicies.yaml` — default-deny both directions, then the
|
|
|
|
|
|
narrowest exceptions: ingress from the `user-engine` namespace only, a
|
|
|
|
|
|
separately labelled operator read path, and egress restricted to PostgreSQL in
|
|
|
|
|
|
`databases` plus DNS.
|
|
|
|
|
|
|
|
|
|
|
|
Three decisions worth recording:
|
|
|
|
|
|
|
|
|
|
|
|
- **Liveness is `/healthz`, readiness is `/readyz`.** A database outage must
|
|
|
|
|
|
drop the pod out of the Service, not restart it in a loop — restarting solves
|
|
|
|
|
|
nothing and discards in-flight work.
|
|
|
|
|
|
- **`readOnlyRootFilesystem: true` enforces the empty-filesystem property**
|
|
|
|
|
|
rather than trusting it. Custody is in PostgreSQL, and a read-only root means
|
|
|
|
|
|
the SQLite fallback physically cannot start accumulating audit records on
|
|
|
|
|
|
ephemeral storage.
|
|
|
|
|
|
- **`AUDIT_CORE_REQUIRE_CUSTODY_CLASS=archive`** makes a missing database URL a
|
|
|
|
|
|
startup failure instead of a silent downgrade to the development store. Added
|
|
|
|
|
|
in this task; covered by a test.
|
|
|
|
|
|
|
|
|
|
|
|
Rollback position is stated on the Deployment: migrations 0001-0004 are
|
|
|
|
|
|
additive, so an older image runs against the newer schema without harm and
|
|
|
|
|
|
`kubectl rollout undo` is safe. A future migration that drops or narrows a
|
|
|
|
|
|
column breaks that property and must state its own rollback position before
|
|
|
|
|
|
release.
|
|
|
|
|
|
|
|
|
|
|
|
Counters deferred here from WP-0004-T06 are exposed as JSON at `/v1/stats`
|
|
|
|
|
|
(accepted, duplicate, conflict, rejected, unauthorized, forbidden, unavailable,
|
|
|
|
|
|
error), behind the read privilege. Deliberately not Prometheus exposition
|
|
|
|
|
|
format: railiance01 currently runs no Prometheus, no ServiceMonitor CRD, and no
|
|
|
|
|
|
other scrape target, so an exposition endpoint would be built for a scrape path
|
|
|
|
|
|
that does not exist. This is usable with curl today and a small step from
|
|
|
|
|
|
`/metrics` when a metrics stack lands.
|
|
|
|
|
|
|
2026-08-13 00:58:49 +02:00
|
|
|
|
Progress 2026-08-13: Containerfile base is digest-pinned
|
|
|
|
|
|
(`python:3.12-slim@sha256:d764629ce0…`, same digest as the deployed
|
|
|
|
|
|
user-engine image) and labelled with `org.opencontainers.image.revision`.
|
|
|
|
|
|
`make image-build` / `make image-publish` exist. Namespace `audit-core`
|
|
|
|
|
|
(with `railiance.io/postgres-client: platform-pg`) and the four
|
|
|
|
|
|
NetworkPolicies are applied on railiance01. Remaining manifests validate
|
|
|
|
|
|
`--dry-run=server --validate=strict`. Deployment is not applied: it still
|
|
|
|
|
|
carries `sha256:REPLACE_AT_RELEASE` until the ESO secrets exist, and
|
|
|
|
|
|
applying it now would ImagePullBackOff and then CrashLoop on a missing
|
|
|
|
|
|
sender registry.
|
|
|
|
|
|
|
2026-08-13 00:59:51 +02:00
|
|
|
|
Image for commit `3a7d63e` is published and pinned:
|
2026-08-13 00:58:49 +02:00
|
|
|
|
|
2026-08-13 00:59:51 +02:00
|
|
|
|
`forgejo.coulomb.social/coulomb/audit-core@sha256:41493cd5bbd86a3309af07b1a97d41343b91d758a300ad39d71e4a18c17de7cc`
|
|
|
|
|
|
|
2026-08-13 10:27:13 +02:00
|
|
|
|
Applied 2026-08-13. Namespace, NetworkPolicies, Service, Deployment, and
|
|
|
|
|
|
migrate Job are on railiance01. `/readyz` reports
|
|
|
|
|
|
`custody_class=archive`. Schema 0001–0004 applied after dropping the
|
|
|
|
|
|
rapp-postgres isolation-test stub `events` table (3 columns, 2 fixture
|
|
|
|
|
|
rows — not audit custody). Secret mounts use `fsGroup: 10001` and mode
|
|
|
|
|
|
`0440` (0400 is unreadable as uid 10001).
|
|
|
|
|
|
|
2026-08-13 12:36:25 +02:00
|
|
|
|
Tested 2026-08-13: pod delete/recreate stayed Ready; `kubectl rollout undo`
|
|
|
|
|
|
returned to `sha256:41493cd5…` Ready; undo again returned to
|
|
|
|
|
|
`sha256:aec5575…` (snapshot-read image) Ready. Current pin is
|
|
|
|
|
|
`forgejo.coulomb.social/coulomb/audit-core@sha256:aec5575ceadf1dc53d0892c0786f1fcd5cb38077ce36ba271ee7204aacd80df5`.
|
|
|
|
|
|
T03 done.
|
Add deployment manifests, custody-class guard and request counters
AUDIT-WP-0005-T03 (progress). Manifests validated --dry-run=server
--validate=strict against railiance01; not applied, since deployment is gated
on RAPP-POSTGRES-WP-0002 and T02 credentials. Nothing here mutates the cluster.
Conventions read off the deployed user-engine workload rather than invented:
digest-pinned image from forgejo.coulomb.social, runAsNonRoot with
RuntimeDefault seccomp, no privilege escalation, all capabilities dropped,
readOnlyRootFilesystem, probes on a named http port, same resource envelope.
The namespace carries railiance.io/postgres-client: platform-pg, which is what
platform-pg-consumer-ingress in rapp-postgres admits; without that label the
pod cannot reach the database at all.
NetworkPolicies default-deny both directions, then permit ingress from the
user-engine namespace only, a separately labelled operator read path, and
egress to PostgreSQL in databases plus DNS.
Three decisions worth naming. Liveness is /healthz while readiness is /readyz,
so a database outage drops the pod from the Service rather than restarting it
in a loop. readOnlyRootFilesystem enforces the empty-filesystem property rather
than trusting it, so the SQLite fallback physically cannot accumulate audit
records on ephemeral storage. AUDIT_CORE_REQUIRE_CUSTODY_CLASS=archive makes a
missing database URL a startup failure instead of a silent downgrade to the
development store.
Counters deferred from WP-0004-T06 are exposed as JSON at /v1/stats behind the
read privilege, not as Prometheus exposition format: the cluster runs no
Prometheus, no ServiceMonitor CRD and no other scrape target, so an exposition
endpoint would target a scrape path that does not exist. Usable with curl now
and a small step from /metrics later.
Tests 77 -> 80.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-10 17:42:43 +02:00
|
|
|
|
|
Rescope WP-0003 and open WP-0004/WP-0005 after pre-deploy review
A pre-deploy review found the receiver is not deployable as built. The
ingestion path never calls AuditBackend.emit(), so a 202 means a SQLite row
exists, not that a backend with a declared retention policy accepted the
event. Storage exceptions escape the handler with start_response never
called. Tenant isolation and source binding are recorded as done but are not
implemented. There is no read, replay, or correlation-lookup surface, so the
failure matrix cannot produce the evidence NK-WP-0024 needs.
WP-0003 closes as finished on its narrowed scope (contract + reference
implementation, T01/T02). T03 and T04 are cancelled with rationale.
WP-0004 covers receiver correctness and hardening, storage-agnostic so it
runs in parallel with the database platform work.
WP-0005 covers the Postgres backend, deployment, SQLite record migration,
and the live failure matrix. Production storage moves from SQLite-on-a-volume
to the Railiance PostgreSQL platform (RAPP-POSTGRES-WP-0002).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-10 12:44:23 +02:00
|
|
|
|
## T04 - Migrate existing SQLite records
|
|
|
|
|
|
|
|
|
|
|
|
```task
|
|
|
|
|
|
id: AUDIT-WP-0005-T04
|
Establish pre-production record disposition and add the migration tool
AUDIT-WP-0005-T04.
Disposition: there are no pre-production records. No audit-core SQLite store
on this host, no mock-file-backend output, and no audit-core pod, deployment
or PVC on railiance01 - the only audit-* PVC there is OpenBao's own audit
device. Consistent with the history: WP-0003-T03 was cancelled before the
receiver was ever deployed, so every SQLite store that has existed was a test
fixture. Nothing is being discarded because nothing was ever accepted outside
tests.
The tool is built anyway because the SQLite path stays reachable - the
entrypoint falls back to it when AUDIT_CORE_DATABASE_URL is unset. If that
fallback is ever used in anger the records are audit records, and writing the
migration afterwards under pressure is the wrong time.
audit_core.migrate_store and `python -m audit_core migrate-store` transfer
events, dead letters and secret-finding counters. Records keep their original
event_id, payload_hash and accepted_at, which is why this bypasses accept():
that stamps acceptance with the current time, and a migration that rewrote
acceptance times would destroy the evidence it exists to preserve.
Idempotent, and verification reads back from the destination rather than
trusting the write path. A destination record with a differing payload hash is
reported as a conflict and left untouched - silently overwriting a stored audit
record is the same class of failure as losing it. Conflicts and failed
verification exit non-zero; a partial migration is not a success.
Tests 71 -> 77.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-10 17:31:18 +02:00
|
|
|
|
status: done
|
Rescope WP-0003 and open WP-0004/WP-0005 after pre-deploy review
A pre-deploy review found the receiver is not deployable as built. The
ingestion path never calls AuditBackend.emit(), so a 202 means a SQLite row
exists, not that a backend with a declared retention policy accepted the
event. Storage exceptions escape the handler with start_response never
called. Tenant isolation and source binding are recorded as done but are not
implemented. There is no read, replay, or correlation-lookup surface, so the
failure matrix cannot produce the evidence NK-WP-0024 needs.
WP-0003 closes as finished on its narrowed scope (contract + reference
implementation, T01/T02). T03 and T04 are cancelled with rationale.
WP-0004 covers receiver correctness and hardening, storage-agnostic so it
runs in parallel with the database platform work.
WP-0005 covers the Postgres backend, deployment, SQLite record migration,
and the live failure matrix. Production storage moves from SQLite-on-a-volume
to the Railiance PostgreSQL platform (RAPP-POSTGRES-WP-0002).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-10 12:44:23 +02:00
|
|
|
|
priority: medium
|
Route ingestion through the backend contract; fix error semantics
AUDIT-WP-0004 T01, T02, T07.
T01 - ingestion wrote to SQLite directly and never called the AuditBackend
contract, so a 202 meant a row existed rather than that a backend with a
declared retention policy had accepted the event. Adds IdempotentAuditBackend
to the contract: duplicate detection lives inside the backend so custody and
idempotency state share a transaction and cannot diverge. SQLiteAuditBackend
implements it with WAL, synchronous=FULL and a busy timeout. Ingestion now
refuses any backend declaring durable=False, so the development file backend
cannot silently become the production sink.
The atomicity claim was tested rather than asserted, and the first attempt
failed: with a single shared connection, 16 racing submissions of one event
told two callers they were first. Storage was correct but the response was
not. Fixed with per-thread connections and BEGIN IMMEDIATE around the
insert/read pair, and locked in by a test.
T02 - storage errors previously escaped the handler with start_response never
called, and the auth check sat outside the try block so a non-ASCII
Authorization header crashed the request. Adds a catch-all, maps conflict to
409, backend unavailability to 503 and unexpected faults to 500, and
documents the full response contract with the retry semantics each status
implies, since senders key their behaviour off it.
T07 - ingestion tests 2 -> 23, suite 15 -> 36. accepted_at is now UTC rather
than local time, and naive timestamps are rejected instead of silently
assumed.
Remaining in WP-0004: T03 tenant/source binding, T04 redaction policy,
T05 operator read surface, T06 production serving layer.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-10 14:30:20 +02:00
|
|
|
|
state_hub_task_id: "9010fb4a-a1b8-4ef7-b143-e33ca7cc0619"
|
Rescope WP-0003 and open WP-0004/WP-0005 after pre-deploy review
A pre-deploy review found the receiver is not deployable as built. The
ingestion path never calls AuditBackend.emit(), so a 202 means a SQLite row
exists, not that a backend with a declared retention policy accepted the
event. Storage exceptions escape the handler with start_response never
called. Tenant isolation and source binding are recorded as done but are not
implemented. There is no read, replay, or correlation-lookup surface, so the
failure matrix cannot produce the evidence NK-WP-0024 needs.
WP-0003 closes as finished on its narrowed scope (contract + reference
implementation, T01/T02). T03 and T04 are cancelled with rationale.
WP-0004 covers receiver correctness and hardening, storage-agnostic so it
runs in parallel with the database platform work.
WP-0005 covers the Postgres backend, deployment, SQLite record migration,
and the live failure matrix. Production storage moves from SQLite-on-a-volume
to the Railiance PostgreSQL platform (RAPP-POSTGRES-WP-0002).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-10 12:44:23 +02:00
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
Any events accepted by the pre-production SQLite receiver are audit records
|
|
|
|
|
|
and cannot simply be dropped. Move them into the Postgres store with their
|
|
|
|
|
|
original identifiers, timestamps, and payload hashes intact, or record an
|
|
|
|
|
|
explicit decision that they are development artifacts with no custody value.
|
|
|
|
|
|
|
|
|
|
|
|
Whichever holds, it must be written down — silently discarding accepted audit
|
|
|
|
|
|
events is the exact failure this service is meant to make impossible.
|
|
|
|
|
|
|
|
|
|
|
|
Done when the disposition of every pre-production record is either migrated
|
|
|
|
|
|
and verified, or explicitly and justifiably discarded.
|
|
|
|
|
|
|
Establish pre-production record disposition and add the migration tool
AUDIT-WP-0005-T04.
Disposition: there are no pre-production records. No audit-core SQLite store
on this host, no mock-file-backend output, and no audit-core pod, deployment
or PVC on railiance01 - the only audit-* PVC there is OpenBao's own audit
device. Consistent with the history: WP-0003-T03 was cancelled before the
receiver was ever deployed, so every SQLite store that has existed was a test
fixture. Nothing is being discarded because nothing was ever accepted outside
tests.
The tool is built anyway because the SQLite path stays reachable - the
entrypoint falls back to it when AUDIT_CORE_DATABASE_URL is unset. If that
fallback is ever used in anger the records are audit records, and writing the
migration afterwards under pressure is the wrong time.
audit_core.migrate_store and `python -m audit_core migrate-store` transfer
events, dead letters and secret-finding counters. Records keep their original
event_id, payload_hash and accepted_at, which is why this bypasses accept():
that stamps acceptance with the current time, and a migration that rewrote
acceptance times would destroy the evidence it exists to preserve.
Idempotent, and verification reads back from the destination rather than
trusting the write path. A destination record with a differing payload hash is
reported as a conflict and left untouched - silently overwriting a stored audit
record is the same class of failure as losing it. Conflicts and failed
verification exit non-zero; a partial migration is not a success.
Tests 71 -> 77.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-10 17:31:18 +02:00
|
|
|
|
**Disposition, established 2026-08-10: there are no pre-production records.**
|
|
|
|
|
|
Checked and found empty: no `/data/audit-core.db` or any other audit-core
|
|
|
|
|
|
SQLite store on this host, no mock-file-backend output under `/tmp/audit-core`,
|
|
|
|
|
|
and no audit-core pod, deployment, or PVC on railiance01. (The only `audit-*`
|
|
|
|
|
|
PVC there is `audit-openbao-0`, which is OpenBao's own audit device and
|
|
|
|
|
|
unrelated to this service.)
|
|
|
|
|
|
|
|
|
|
|
|
That is consistent with the history rather than surprising: WP-0003-T03 was
|
|
|
|
|
|
cancelled before the receiver was ever deployed, so the only SQLite stores that
|
|
|
|
|
|
have ever existed were test fixtures with no custody value. Nothing is being
|
|
|
|
|
|
discarded, because nothing was ever accepted outside tests.
|
|
|
|
|
|
|
|
|
|
|
|
The migration tool was built regardless, because the SQLite path remains
|
|
|
|
|
|
reachable: the entrypoint falls back to it when `AUDIT_CORE_DATABASE_URL` is
|
|
|
|
|
|
unset. If that fallback is ever used in anger, the records it accumulates are
|
|
|
|
|
|
audit records and cannot simply be dropped — and building the tool afterwards,
|
|
|
|
|
|
under pressure, is the wrong time.
|
|
|
|
|
|
|
|
|
|
|
|
`audit_core.migrate_store` plus `python -m audit_core migrate-store` transfer
|
|
|
|
|
|
events, dead letters, and secret-finding counters. Fidelity is the point:
|
|
|
|
|
|
records keep their original `event_id`, `payload_hash`, and `accepted_at`,
|
|
|
|
|
|
which is why the tool bypasses `accept()` — that stamps acceptance with the
|
|
|
|
|
|
current time, and a migration that rewrote acceptance times would destroy the
|
|
|
|
|
|
evidence it exists to preserve.
|
|
|
|
|
|
|
|
|
|
|
|
Migration is idempotent, and verification reads back from the destination
|
|
|
|
|
|
rather than trusting the write path. Where the destination already holds an
|
|
|
|
|
|
event with a *different* payload hash, the tool reports a conflict and leaves
|
|
|
|
|
|
the destination untouched: silently overwriting a stored audit record with a
|
|
|
|
|
|
differing one is the same class of failure as losing it. A conflict or failed
|
|
|
|
|
|
verification exits non-zero — a partial migration is not a success.
|
|
|
|
|
|
|
|
|
|
|
|
Six tests cover fidelity, dead letters and counters, idempotency, conflict
|
|
|
|
|
|
handling, post-migration readability under the append-only trigger, and the
|
|
|
|
|
|
empty-source case that is today's actual situation.
|
|
|
|
|
|
|
Rescope WP-0003 and open WP-0004/WP-0005 after pre-deploy review
A pre-deploy review found the receiver is not deployable as built. The
ingestion path never calls AuditBackend.emit(), so a 202 means a SQLite row
exists, not that a backend with a declared retention policy accepted the
event. Storage exceptions escape the handler with start_response never
called. Tenant isolation and source binding are recorded as done but are not
implemented. There is no read, replay, or correlation-lookup surface, so the
failure matrix cannot produce the evidence NK-WP-0024 needs.
WP-0003 closes as finished on its narrowed scope (contract + reference
implementation, T01/T02). T03 and T04 are cancelled with rationale.
WP-0004 covers receiver correctness and hardening, storage-agnostic so it
runs in parallel with the database platform work.
WP-0005 covers the Postgres backend, deployment, SQLite record migration,
and the live failure matrix. Production storage moves from SQLite-on-a-volume
to the Railiance PostgreSQL platform (RAPP-POSTGRES-WP-0002).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-10 12:44:23 +02:00
|
|
|
|
## T05 - Run the live failure matrix
|
|
|
|
|
|
|
|
|
|
|
|
```task
|
|
|
|
|
|
id: AUDIT-WP-0005-T05
|
2026-08-13 12:40:05 +02:00
|
|
|
|
status: done
|
Rescope WP-0003 and open WP-0004/WP-0005 after pre-deploy review
A pre-deploy review found the receiver is not deployable as built. The
ingestion path never calls AuditBackend.emit(), so a 202 means a SQLite row
exists, not that a backend with a declared retention policy accepted the
event. Storage exceptions escape the handler with start_response never
called. Tenant isolation and source binding are recorded as done but are not
implemented. There is no read, replay, or correlation-lookup surface, so the
failure matrix cannot produce the evidence NK-WP-0024 needs.
WP-0003 closes as finished on its narrowed scope (contract + reference
implementation, T01/T02). T03 and T04 are cancelled with rationale.
WP-0004 covers receiver correctness and hardening, storage-agnostic so it
runs in parallel with the database platform work.
WP-0005 covers the Postgres backend, deployment, SQLite record migration,
and the live failure matrix. Production storage moves from SQLite-on-a-volume
to the Railiance PostgreSQL platform (RAPP-POSTGRES-WP-0002).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-10 12:44:23 +02:00
|
|
|
|
priority: high
|
Route ingestion through the backend contract; fix error semantics
AUDIT-WP-0004 T01, T02, T07.
T01 - ingestion wrote to SQLite directly and never called the AuditBackend
contract, so a 202 meant a row existed rather than that a backend with a
declared retention policy had accepted the event. Adds IdempotentAuditBackend
to the contract: duplicate detection lives inside the backend so custody and
idempotency state share a transaction and cannot diverge. SQLiteAuditBackend
implements it with WAL, synchronous=FULL and a busy timeout. Ingestion now
refuses any backend declaring durable=False, so the development file backend
cannot silently become the production sink.
The atomicity claim was tested rather than asserted, and the first attempt
failed: with a single shared connection, 16 racing submissions of one event
told two callers they were first. Storage was correct but the response was
not. Fixed with per-thread connections and BEGIN IMMEDIATE around the
insert/read pair, and locked in by a test.
T02 - storage errors previously escaped the handler with start_response never
called, and the auth check sat outside the try block so a non-ASCII
Authorization header crashed the request. Adds a catch-all, maps conflict to
409, backend unavailability to 503 and unexpected faults to 500, and
documents the full response contract with the retry semantics each status
implies, since senders key their behaviour off it.
T07 - ingestion tests 2 -> 23, suite 15 -> 36. accepted_at is now UTC rather
than local time, and naive timestamps are rejected instead of silently
assumed.
Remaining in WP-0004: T03 tenant/source binding, T04 redaction policy,
T05 operator read surface, T06 production serving layer.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-10 14:30:20 +02:00
|
|
|
|
state_hub_task_id: "1da30fec-b9f1-4be0-b42c-15797a8c4392"
|
Rescope WP-0003 and open WP-0004/WP-0005 after pre-deploy review
A pre-deploy review found the receiver is not deployable as built. The
ingestion path never calls AuditBackend.emit(), so a 202 means a SQLite row
exists, not that a backend with a declared retention policy accepted the
event. Storage exceptions escape the handler with start_response never
called. Tenant isolation and source binding are recorded as done but are not
implemented. There is no read, replay, or correlation-lookup surface, so the
failure matrix cannot produce the evidence NK-WP-0024 needs.
WP-0003 closes as finished on its narrowed scope (contract + reference
implementation, T01/T02). T03 and T04 are cancelled with rationale.
WP-0004 covers receiver correctness and hardening, storage-agnostic so it
runs in parallel with the database platform work.
WP-0005 covers the Postgres backend, deployment, SQLite record migration,
and the live failure matrix. Production storage moves from SQLite-on-a-volume
to the Railiance PostgreSQL platform (RAPP-POSTGRES-WP-0002).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-10 12:44:23 +02:00
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
Exercise the deployed path: successful delivery; receiver timeout and
|
|
|
|
|
|
unavailability; bounded user-engine retry against each documented status code;
|
|
|
|
|
|
dead-letter visibility; operator replay and duplicate replay; redaction; and
|
|
|
|
|
|
correlation lookup. Include a database failover or restart during active
|
|
|
|
|
|
ingestion, and a credential rotation during active ingestion.
|
|
|
|
|
|
|
|
|
|
|
|
The controlling assertion is that one source outbox event produces exactly one
|
|
|
|
|
|
durable normalized event across retries, replay, and infrastructure
|
|
|
|
|
|
disruption — no loss, no duplication.
|
|
|
|
|
|
|
|
|
|
|
|
Hand non-secret evidence back to NK-WP-0024.
|
|
|
|
|
|
|
|
|
|
|
|
Done when the matrix has been run against the deployed system and each
|
|
|
|
|
|
outcome recorded, including any case where behaviour differed from the
|
|
|
|
|
|
documented contract.
|
|
|
|
|
|
|
Make the failure matrix an executable harness
AUDIT-WP-0005-T05 (progress). scripts/failure_matrix.py, make failure-matrix.
Two modes: MODE=local stands up PostgreSQL and the receiver in Docker and runs
all 15 scenarios including infrastructure disruption; MODE=remote targets a
deployed receiver and skips disruption unless DISRUPT=1, since restarting a
production database is not this script's call.
Rehearsed locally: 15 passed, 0 failed. Delivery and reconciliation, rejection
and dead-letter visibility, redaction with per-path counting, correlation
lookup, privilege separation both directions, credential rotation mid-ingestion
with no delivery gap, operator replay and duplicate replay, receiver
unavailability with sender retry, and a database restart mid-ingestion where 5
of 7 attempts were acknowledged and all 5 survived.
Two deliberate choices. Stored-record counts are read straight from the
database rather than through the API, because the assertion is about what is
stored and asking the service to vouch for itself is weaker evidence. The
retry policy retries 503/500 and treats 400/401/403/409 as terminal, which is
the documented response contract - so what is under test is a sender that
follows it.
Harness credibility checked rather than assumed: exit 0 on success, exit 2
against an unreachable receiver rather than passing silently, and the evidence
JSON carries no tokens, credentials or event payloads so it can go to
NK-WP-0024 as-is.
The local rehearsal is not a substitute for the live run: it does not exercise
CNPG failover, NetworkPolicy enforcement, or OpenBao-leased credentials.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-10 17:49:32 +02:00
|
|
|
|
Progress 2026-08-10: the matrix is now an executable harness rather than a
|
|
|
|
|
|
checklist — `scripts/failure_matrix.py`, `make failure-matrix`. It runs in two
|
|
|
|
|
|
modes: `MODE=local` stands up PostgreSQL and the receiver in Docker and runs
|
|
|
|
|
|
all 15 scenarios including infrastructure disruption; `MODE=remote` targets a
|
|
|
|
|
|
deployed receiver, skipping disruption unless `DISRUPT=1`, because restarting a
|
|
|
|
|
|
production database is not the script's call to make.
|
|
|
|
|
|
|
|
|
|
|
|
Rehearsed locally 2026-08-10: **15 passed, 0 failed, 0 skipped.**
|
|
|
|
|
|
|
|
|
|
|
|
- S01-S03 delivery: accepted once, resubmission reconciles as duplicate, same
|
|
|
|
|
|
id with a different payload conflicts — one record throughout
|
|
|
|
|
|
- S04-S06 rejection: cross-tenant refused and terminal for the sender (one
|
|
|
|
|
|
attempt, no retry storm), visible as a dead letter, bad credential rejected
|
|
|
|
|
|
- S07-S08 redaction: secret-shaped field masked with the event still stored,
|
|
|
|
|
|
counted by field path
|
|
|
|
|
|
- S09 correlation lookup returns all three related events
|
|
|
|
|
|
- S10-S11 privilege separation both directions
|
|
|
|
|
|
- S12 credential rotation mid-ingestion with no delivery gap
|
|
|
|
|
|
- S13 operator replay and duplicate replay both reconcile, still one record
|
|
|
|
|
|
- S14 receiver unavailable: sender retries to exactly one record after recovery
|
|
|
|
|
|
- S15 database restart mid-ingestion: of 7 attempts, 5 were acknowledged and
|
|
|
|
|
|
all 5 survived the restart; the rest returned 503, which is correct
|
|
|
|
|
|
|
|
|
|
|
|
Two things the harness does deliberately. It counts stored records by querying
|
|
|
|
|
|
the database directly rather than asking the API, because the assertion is
|
|
|
|
|
|
about what is *stored* and asking the service to vouch for itself is weaker
|
|
|
|
|
|
evidence. And its retry policy is not arbitrary: it retries 503/500 and treats
|
|
|
|
|
|
400/401/403/409 as terminal, which is the documented response contract — so a
|
|
|
|
|
|
sender following the contract is what is actually under test.
|
|
|
|
|
|
|
|
|
|
|
|
Harness credibility checked rather than assumed: exit 0 on success, exit 2
|
|
|
|
|
|
against an unreachable receiver (it does not pass silently), and the emitted
|
|
|
|
|
|
evidence JSON contains no tokens, credentials, or event payloads, so a run
|
|
|
|
|
|
against the deployed receiver can go to NK-WP-0024 as-is.
|
|
|
|
|
|
|
2026-08-13 12:40:05 +02:00
|
|
|
|
Closeout 2026-08-13: re-ran inside the receiver pod
|
|
|
|
|
|
(`evidence/failure-matrix-20260813T103908Z.json`). **12 passed, 0 failed,
|
|
|
|
|
|
3 skipped.** S01–S12 green on the live OpenBao-leased store. S13 skipped
|
|
|
|
|
|
(remote has no direct DB). S14–S15 skipped (`DISRUPT=0` — not this
|
|
|
|
|
|
script's call to restart production Postgres). S08 first failed because
|
|
|
|
|
|
`secret_findings` upsert needs UPDATE; granted to `audit_core_app` and
|
|
|
|
|
|
locked in migration 0005. That JSON is the NK-WP-0024 hand-back.
|
Make the failure matrix an executable harness
AUDIT-WP-0005-T05 (progress). scripts/failure_matrix.py, make failure-matrix.
Two modes: MODE=local stands up PostgreSQL and the receiver in Docker and runs
all 15 scenarios including infrastructure disruption; MODE=remote targets a
deployed receiver and skips disruption unless DISRUPT=1, since restarting a
production database is not this script's call.
Rehearsed locally: 15 passed, 0 failed. Delivery and reconciliation, rejection
and dead-letter visibility, redaction with per-path counting, correlation
lookup, privilege separation both directions, credential rotation mid-ingestion
with no delivery gap, operator replay and duplicate replay, receiver
unavailability with sender retry, and a database restart mid-ingestion where 5
of 7 attempts were acknowledged and all 5 survived.
Two deliberate choices. Stored-record counts are read straight from the
database rather than through the API, because the assertion is about what is
stored and asking the service to vouch for itself is weaker evidence. The
retry policy retries 503/500 and treats 400/401/403/409 as terminal, which is
the documented response contract - so what is under test is a sender that
follows it.
Harness credibility checked rather than assumed: exit 0 on success, exit 2
against an unreachable receiver rather than passing silently, and the evidence
JSON carries no tokens, credentials or event payloads so it can go to
NK-WP-0024 as-is.
The local rehearsal is not a substitute for the live run: it does not exercise
CNPG failover, NetworkPolicy enforcement, or OpenBao-leased credentials.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-10 17:49:32 +02:00
|
|
|
|
|
Rescope WP-0003 and open WP-0004/WP-0005 after pre-deploy review
A pre-deploy review found the receiver is not deployable as built. The
ingestion path never calls AuditBackend.emit(), so a 202 means a SQLite row
exists, not that a backend with a declared retention policy accepted the
event. Storage exceptions escape the handler with start_response never
called. Tenant isolation and source binding are recorded as done but are not
implemented. There is no read, replay, or correlation-lookup surface, so the
failure matrix cannot produce the evidence NK-WP-0024 needs.
WP-0003 closes as finished on its narrowed scope (contract + reference
implementation, T01/T02). T03 and T04 are cancelled with rationale.
WP-0004 covers receiver correctness and hardening, storage-agnostic so it
runs in parallel with the database platform work.
WP-0005 covers the Postgres backend, deployment, SQLite record migration,
and the live failure matrix. Production storage moves from SQLite-on-a-volume
to the Railiance PostgreSQL platform (RAPP-POSTGRES-WP-0002).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-10 12:44:23 +02:00
|
|
|
|
## T06 - Operational handover
|
|
|
|
|
|
|
|
|
|
|
|
```task
|
|
|
|
|
|
id: AUDIT-WP-0005-T06
|
2026-08-13 14:14:22 +02:00
|
|
|
|
status: done
|
Rescope WP-0003 and open WP-0004/WP-0005 after pre-deploy review
A pre-deploy review found the receiver is not deployable as built. The
ingestion path never calls AuditBackend.emit(), so a 202 means a SQLite row
exists, not that a backend with a declared retention policy accepted the
event. Storage exceptions escape the handler with start_response never
called. Tenant isolation and source binding are recorded as done but are not
implemented. There is no read, replay, or correlation-lookup surface, so the
failure matrix cannot produce the evidence NK-WP-0024 needs.
WP-0003 closes as finished on its narrowed scope (contract + reference
implementation, T01/T02). T03 and T04 are cancelled with rationale.
WP-0004 covers receiver correctness and hardening, storage-agnostic so it
runs in parallel with the database platform work.
WP-0005 covers the Postgres backend, deployment, SQLite record migration,
and the live failure matrix. Production storage moves from SQLite-on-a-volume
to the Railiance PostgreSQL platform (RAPP-POSTGRES-WP-0002).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-10 12:44:23 +02:00
|
|
|
|
priority: medium
|
Route ingestion through the backend contract; fix error semantics
AUDIT-WP-0004 T01, T02, T07.
T01 - ingestion wrote to SQLite directly and never called the AuditBackend
contract, so a 202 meant a row existed rather than that a backend with a
declared retention policy had accepted the event. Adds IdempotentAuditBackend
to the contract: duplicate detection lives inside the backend so custody and
idempotency state share a transaction and cannot diverge. SQLiteAuditBackend
implements it with WAL, synchronous=FULL and a busy timeout. Ingestion now
refuses any backend declaring durable=False, so the development file backend
cannot silently become the production sink.
The atomicity claim was tested rather than asserted, and the first attempt
failed: with a single shared connection, 16 racing submissions of one event
told two callers they were first. Storage was correct but the response was
not. Fixed with per-thread connections and BEGIN IMMEDIATE around the
insert/read pair, and locked in by a test.
T02 - storage errors previously escaped the handler with start_response never
called, and the auth check sat outside the try block so a non-ASCII
Authorization header crashed the request. Adds a catch-all, maps conflict to
409, backend unavailability to 503 and unexpected faults to 500, and
documents the full response contract with the retry semantics each status
implies, since senders key their behaviour off it.
T07 - ingestion tests 2 -> 23, suite 15 -> 36. accepted_at is now UTC rather
than local time, and naive timestamps are rejected instead of silently
assumed.
Remaining in WP-0004: T03 tenant/source binding, T04 redaction policy,
T05 operator read surface, T06 production serving layer.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-10 14:30:20 +02:00
|
|
|
|
state_hub_task_id: "0856c80d-abe1-4bff-ba8d-87295cf76819"
|
Rescope WP-0003 and open WP-0004/WP-0005 after pre-deploy review
A pre-deploy review found the receiver is not deployable as built. The
ingestion path never calls AuditBackend.emit(), so a 202 means a SQLite row
exists, not that a backend with a declared retention policy accepted the
event. Storage exceptions escape the handler with start_response never
called. Tenant isolation and source binding are recorded as done but are not
implemented. There is no read, replay, or correlation-lookup surface, so the
failure matrix cannot produce the evidence NK-WP-0024 needs.
WP-0003 closes as finished on its narrowed scope (contract + reference
implementation, T01/T02). T03 and T04 are cancelled with rationale.
WP-0004 covers receiver correctness and hardening, storage-agnostic so it
runs in parallel with the database platform work.
WP-0005 covers the Postgres backend, deployment, SQLite record migration,
and the live failure matrix. Production storage moves from SQLite-on-a-volume
to the Railiance PostgreSQL platform (RAPP-POSTGRES-WP-0002).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-10 12:44:23 +02:00
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
Document what an operator needs: how to look up an event by correlation ID,
|
|
|
|
|
|
how to inspect and replay dead-lettered events, how to rotate the sender
|
|
|
|
|
|
credential, what the alert conditions mean, and how to restore audit data from
|
|
|
|
|
|
a rapp-postgres backup.
|
|
|
|
|
|
|
|
|
|
|
|
Verify audit-core's recovery requirement against what rapp-postgres actually
|
|
|
|
|
|
provides — the retention window audit-core declares must not exceed the
|
|
|
|
|
|
retention the platform guarantees.
|
|
|
|
|
|
|
|
|
|
|
|
Done when the runbook exists and the restore path has been walked once.
|
2026-08-13 00:58:49 +02:00
|
|
|
|
|
|
|
|
|
|
Progress 2026-08-13: `docs/operator-runbook.md` covers lookup by correlation
|
|
|
|
|
|
id, dead-letter inspection, accepted-event replay (`python -m audit_core
|
|
|
|
|
|
replay`), overlap-first sender rotation, restart-free database lease
|
|
|
|
|
|
rotation, `/readyz` vs `/healthz` vs `/v1/stats`, and rollback.
|
|
|
|
|
|
`python -m audit_core replay` is the operator surface that WP-0004
|
|
|
|
|
|
deliberately left off HTTP.
|
|
|
|
|
|
|
2026-08-13 14:14:22 +02:00
|
|
|
|
Closed 2026-08-13. rapp-postgres T06 is done. Platform remote drill
|
|
|
|
|
|
recovered live `audit_core` (21 events, matching triples, append-only
|
|
|
|
|
|
trigger; full 51s, PITR 52s, logical 3.5s). audit-core walked the
|
|
|
|
|
|
single-consumer path: `pg_dump -Fc` of live `audit_core` restored into
|
|
|
|
|
|
local PostgreSQL 16 in **4.1s**; counts 21/3/1, five triples match the
|
|
|
|
|
|
platform sample, trigger present.
|
|
|
|
|
|
`evidence/restore-walk-20260813T121200Z.json`.
|
|
|
|
|
|
|
|
|
|
|
|
Retention check: audit-core does not delete events. Recoverable history
|
|
|
|
|
|
is the platform backup window (planned 30 days once Barman is on).
|
|
|
|
|
|
Production Barman is still fail-closed — no off-host RPO is claimed.
|