audit-core/WORK-RECORDS.md

71 lines
6.3 KiB
Markdown
Raw Permalink Normal View History

# Work Records — audit-core
> Generated by `statehub fix-consistency` (CUST-WP-0061-T04, work-record
> stage 3). Do not edit by hand — edit the source file/block listed for
> each record and re-run fix-consistency to refresh this index. Archived
> workplans are omitted; closed decisions/intakes/engagements stay listed
> so recently-resolved work is still visible. [auto]
| Kind | ID | Status | Lane | Source |
| --- | --- | --- | --- | --- |
| workplan | AUDIT-WP-0001 | finished | — | workplans/AUDIT-WP-0001-statehub-bootstrap.md |
| workplan | AUDIT-WP-0002 | finished | — | workplans/AUDIT-WP-0002-pluggable-audit-backend.md |
Route ingestion through the backend contract; fix error semantics AUDIT-WP-0004 T01, T02, T07. T01 - ingestion wrote to SQLite directly and never called the AuditBackend contract, so a 202 meant a row existed rather than that a backend with a declared retention policy had accepted the event. Adds IdempotentAuditBackend to the contract: duplicate detection lives inside the backend so custody and idempotency state share a transaction and cannot diverge. SQLiteAuditBackend implements it with WAL, synchronous=FULL and a busy timeout. Ingestion now refuses any backend declaring durable=False, so the development file backend cannot silently become the production sink. The atomicity claim was tested rather than asserted, and the first attempt failed: with a single shared connection, 16 racing submissions of one event told two callers they were first. Storage was correct but the response was not. Fixed with per-thread connections and BEGIN IMMEDIATE around the insert/read pair, and locked in by a test. T02 - storage errors previously escaped the handler with start_response never called, and the auth check sat outside the try block so a non-ASCII Authorization header crashed the request. Adds a catch-all, maps conflict to 409, backend unavailability to 503 and unexpected faults to 500, and documents the full response contract with the retry semantics each status implies, since senders key their behaviour off it. T07 - ingestion tests 2 -> 23, suite 15 -> 36. accepted_at is now UTC rather than local time, and naive timestamps are rejected instead of silently assumed. Remaining in WP-0004: T03 tenant/source binding, T04 redaction policy, T05 operator read surface, T06 production serving layer. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-10 14:30:20 +02:00
| workplan | AUDIT-WP-0003 | finished | — | workplans/AUDIT-WP-0003-user-engine-event-ingestion-service.md |
Add the PostgreSQL audit backend and a shared conformance suite AUDIT-WP-0005-T01, built and verified against PostgreSQL 16 locally in Docker; the Railiance cluster was not needed. tests/test_backend_conformance.py is one suite run against every backend, so "the Postgres backend is done" means it satisfies the same contract SQLite already does rather than having its own green tests. It skips cleanly with no server reachable; make pg-test-up and make test-pg run it. Suite 50 -> 71. RetentionPolicy declares immutable=True and earns it: migration 0002 installs a trigger rejecting UPDATE and DELETE on the events table, so a leaked runtime credential can append but cannot rewrite or erase the trail. That materially narrows the residual risk ADR-0001 section 5 called out. tamper_evidence stays False because nothing here would prove a database owner had dropped the trigger - hash-chaining or external anchoring would be needed and is not implemented. Idempotency is one statement (INSERT ... ON CONFLICT DO NOTHING RETURNING), verified to behave identically to the SQLite backend under 12 concurrent submissions of the same event. Migrations are ordered, recorded and idempotent. Replay reconciles rather than duplicating - the piece deferred out of WP-0004-T05 - and is tested to leave exactly one custody record. Backend selection is by AUDIT_CORE_DATABASE_URL; the SQLite fallback logs a warning so a deployment that lost its URL is visible rather than quietly running on the wrong store. Also fixed: ingestion had no __main__ guard, so python -m audit_core.ingestion silently did nothing. Found during end-to-end smoke. Counting semantics documented: occurrences counts transmissions, not stored events, so a retry of a secret-shaped field increments it again. That is the sender behaviour being optimized away. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-10 17:09:46 +02:00
| workplan | AUDIT-WP-0004 | finished | — | workplans/AUDIT-WP-0004-receiver-correctness-and-hardening.md |
| workplan | AUDIT-WP-0005 | finished | — | workplans/AUDIT-WP-0005-postgres-store-and-production-deployment.md |
| workplan | AUDIT-WP-0006 | finished | — | workplans/AUDIT-WP-0006-honest-custody-and-canon-join.md |
| workplan | AUDIT-WP-0007 | finished | — | workplans/AUDIT-WP-0007-integrity-verification.md |
| workplan | AUDIT-WP-0008 | active | — | workplans/AUDIT-WP-0008-tenancy-posture-alignment.md |
| workplan | AUDIT-WP-0009 | ready | — | workplans/AUDIT-WP-0009-evidence-role-conformance.md |
| task | AUDIT-WP-0001-T01 | done | — | workplans/AUDIT-WP-0001-statehub-bootstrap.md |
| task | AUDIT-WP-0001-T02 | done | — | workplans/AUDIT-WP-0001-statehub-bootstrap.md |
| task | AUDIT-WP-0001-T03 | done | — | workplans/AUDIT-WP-0001-statehub-bootstrap.md |
| task | AUDIT-WP-0002-T01 | done | — | workplans/AUDIT-WP-0002-pluggable-audit-backend.md |
| task | AUDIT-WP-0003-T01 | done | — | workplans/AUDIT-WP-0003-user-engine-event-ingestion-service.md |
| task | AUDIT-WP-0003-T02 | done | — | workplans/AUDIT-WP-0003-user-engine-event-ingestion-service.md |
Route ingestion through the backend contract; fix error semantics AUDIT-WP-0004 T01, T02, T07. T01 - ingestion wrote to SQLite directly and never called the AuditBackend contract, so a 202 meant a row existed rather than that a backend with a declared retention policy had accepted the event. Adds IdempotentAuditBackend to the contract: duplicate detection lives inside the backend so custody and idempotency state share a transaction and cannot diverge. SQLiteAuditBackend implements it with WAL, synchronous=FULL and a busy timeout. Ingestion now refuses any backend declaring durable=False, so the development file backend cannot silently become the production sink. The atomicity claim was tested rather than asserted, and the first attempt failed: with a single shared connection, 16 racing submissions of one event told two callers they were first. Storage was correct but the response was not. Fixed with per-thread connections and BEGIN IMMEDIATE around the insert/read pair, and locked in by a test. T02 - storage errors previously escaped the handler with start_response never called, and the auth check sat outside the try block so a non-ASCII Authorization header crashed the request. Adds a catch-all, maps conflict to 409, backend unavailability to 503 and unexpected faults to 500, and documents the full response contract with the retry semantics each status implies, since senders key their behaviour off it. T07 - ingestion tests 2 -> 23, suite 15 -> 36. accepted_at is now UTC rather than local time, and naive timestamps are rejected instead of silently assumed. Remaining in WP-0004: T03 tenant/source binding, T04 redaction policy, T05 operator read surface, T06 production serving layer. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-10 14:30:20 +02:00
| task | AUDIT-WP-0003-T03 | cancel | — | workplans/AUDIT-WP-0003-user-engine-event-ingestion-service.md |
| task | AUDIT-WP-0003-T04 | cancel | — | workplans/AUDIT-WP-0003-user-engine-event-ingestion-service.md |
Bind sender identities, add operator read surface, real serving layer AUDIT-WP-0004 T03, T05, T06. T03 - the receiver accepted whatever tenant and source a caller sent as long as it held the one shared token, despite WP-0003 recording tenant isolation as delivered. audit_core.senders binds each credential to the sources and tenants it may assert, driven by AUDIT_CORE_SENDERS rather than literals. Identities hold a list of tokens so rotation publishes the replacement alongside the incumbent and needs no delivery gap. Read is a separate privilege from write, so a sender credential cannot read the audit trail back. T05 - lookup by event id, lookup by correlation id, and a dead-letter view. Rejections are recorded rather than silently dropped. An event rejected for carrying secret-shaped material has its payload withheld: storing it would write that material into the audit store, which is what the rejection exists to prevent. Reason and payload hash are kept so it stays traceable. Replay is deliberately not built here. Idempotent replay is a property of the durable store and building it against SQLite would produce a second implementation to throw away; it lands with the Postgres backend in AUDIT-WP-0005-T01. T06 - serving moves to waitress with configurable threads and channel timeout, installed in the image via the serve extra. Without it the entrypoint falls back to a threaded wsgiref server with a socket timeout and graceful shutdown on SIGTERM, and logs a warning so a deployment cannot quietly land on the fallback. Metric counters deferred to WP-0005-T03 to be designed against the real scrape path. Tests 36 -> 46, covering cross-tenant and cross-source refusal, token rotation, read/write privilege separation, correlation lookup, and payload withholding on secret rejection. Remaining in WP-0004: T04 redaction policy, which needs a decision on whether a secret-shaped field is a rejection or a redaction. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-10 14:50:02 +02:00
| task | AUDIT-WP-0004-T01 | done | — | workplans/AUDIT-WP-0004-receiver-correctness-and-hardening.md |
| task | AUDIT-WP-0004-T02 | done | — | workplans/AUDIT-WP-0004-receiver-correctness-and-hardening.md |
2026-08-10 16:02:22 +02:00
| task | AUDIT-WP-0004-T03 | done | — | workplans/AUDIT-WP-0004-receiver-correctness-and-hardening.md |
Add the PostgreSQL audit backend and a shared conformance suite AUDIT-WP-0005-T01, built and verified against PostgreSQL 16 locally in Docker; the Railiance cluster was not needed. tests/test_backend_conformance.py is one suite run against every backend, so "the Postgres backend is done" means it satisfies the same contract SQLite already does rather than having its own green tests. It skips cleanly with no server reachable; make pg-test-up and make test-pg run it. Suite 50 -> 71. RetentionPolicy declares immutable=True and earns it: migration 0002 installs a trigger rejecting UPDATE and DELETE on the events table, so a leaked runtime credential can append but cannot rewrite or erase the trail. That materially narrows the residual risk ADR-0001 section 5 called out. tamper_evidence stays False because nothing here would prove a database owner had dropped the trigger - hash-chaining or external anchoring would be needed and is not implemented. Idempotency is one statement (INSERT ... ON CONFLICT DO NOTHING RETURNING), verified to behave identically to the SQLite backend under 12 concurrent submissions of the same event. Migrations are ordered, recorded and idempotent. Replay reconciles rather than duplicating - the piece deferred out of WP-0004-T05 - and is tested to leave exactly one custody record. Backend selection is by AUDIT_CORE_DATABASE_URL; the SQLite fallback logs a warning so a deployment that lost its URL is visible rather than quietly running on the wrong store. Also fixed: ingestion had no __main__ guard, so python -m audit_core.ingestion silently did nothing. Found during end-to-end smoke. Counting semantics documented: occurrences counts transmissions, not stored events, so a retry of a secret-shaped field increments it again. That is the sender behaviour being optimized away. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-10 17:09:46 +02:00
| task | AUDIT-WP-0004-T04 | done | — | workplans/AUDIT-WP-0004-receiver-correctness-and-hardening.md |
2026-08-10 16:02:22 +02:00
| task | AUDIT-WP-0004-T05 | done | — | workplans/AUDIT-WP-0004-receiver-correctness-and-hardening.md |
| task | AUDIT-WP-0004-T06 | done | — | workplans/AUDIT-WP-0004-receiver-correctness-and-hardening.md |
Bind sender identities, add operator read surface, real serving layer AUDIT-WP-0004 T03, T05, T06. T03 - the receiver accepted whatever tenant and source a caller sent as long as it held the one shared token, despite WP-0003 recording tenant isolation as delivered. audit_core.senders binds each credential to the sources and tenants it may assert, driven by AUDIT_CORE_SENDERS rather than literals. Identities hold a list of tokens so rotation publishes the replacement alongside the incumbent and needs no delivery gap. Read is a separate privilege from write, so a sender credential cannot read the audit trail back. T05 - lookup by event id, lookup by correlation id, and a dead-letter view. Rejections are recorded rather than silently dropped. An event rejected for carrying secret-shaped material has its payload withheld: storing it would write that material into the audit store, which is what the rejection exists to prevent. Reason and payload hash are kept so it stays traceable. Replay is deliberately not built here. Idempotent replay is a property of the durable store and building it against SQLite would produce a second implementation to throw away; it lands with the Postgres backend in AUDIT-WP-0005-T01. T06 - serving moves to waitress with configurable threads and channel timeout, installed in the image via the serve extra. Without it the entrypoint falls back to a threaded wsgiref server with a socket timeout and graceful shutdown on SIGTERM, and logs a warning so a deployment cannot quietly land on the fallback. Metric counters deferred to WP-0005-T03 to be designed against the real scrape path. Tests 36 -> 46, covering cross-tenant and cross-source refusal, token rotation, read/write privilege separation, correlation lookup, and payload withholding on secret rejection. Remaining in WP-0004: T04 redaction policy, which needs a decision on whether a secret-shaped field is a rejection or a redaction. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-10 14:50:02 +02:00
| task | AUDIT-WP-0004-T07 | done | — | workplans/AUDIT-WP-0004-receiver-correctness-and-hardening.md |
Establish pre-production record disposition and add the migration tool AUDIT-WP-0005-T04. Disposition: there are no pre-production records. No audit-core SQLite store on this host, no mock-file-backend output, and no audit-core pod, deployment or PVC on railiance01 - the only audit-* PVC there is OpenBao's own audit device. Consistent with the history: WP-0003-T03 was cancelled before the receiver was ever deployed, so every SQLite store that has existed was a test fixture. Nothing is being discarded because nothing was ever accepted outside tests. The tool is built anyway because the SQLite path stays reachable - the entrypoint falls back to it when AUDIT_CORE_DATABASE_URL is unset. If that fallback is ever used in anger the records are audit records, and writing the migration afterwards under pressure is the wrong time. audit_core.migrate_store and `python -m audit_core migrate-store` transfer events, dead letters and secret-finding counters. Records keep their original event_id, payload_hash and accepted_at, which is why this bypasses accept(): that stamps acceptance with the current time, and a migration that rewrote acceptance times would destroy the evidence it exists to preserve. Idempotent, and verification reads back from the destination rather than trusting the write path. A destination record with a differing payload hash is reported as a conflict and left untouched - silently overwriting a stored audit record is the same class of failure as losing it. Conflicts and failed verification exit non-zero; a partial migration is not a success. Tests 71 -> 77. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-10 17:31:18 +02:00
| task | AUDIT-WP-0005-T01 | done | — | workplans/AUDIT-WP-0005-postgres-store-and-production-deployment.md |
| task | AUDIT-WP-0005-T02 | done | — | workplans/AUDIT-WP-0005-postgres-store-and-production-deployment.md |
| task | AUDIT-WP-0005-T03 | done | — | workplans/AUDIT-WP-0005-postgres-store-and-production-deployment.md |
Add deployment manifests, custody-class guard and request counters AUDIT-WP-0005-T03 (progress). Manifests validated --dry-run=server --validate=strict against railiance01; not applied, since deployment is gated on RAPP-POSTGRES-WP-0002 and T02 credentials. Nothing here mutates the cluster. Conventions read off the deployed user-engine workload rather than invented: digest-pinned image from forgejo.coulomb.social, runAsNonRoot with RuntimeDefault seccomp, no privilege escalation, all capabilities dropped, readOnlyRootFilesystem, probes on a named http port, same resource envelope. The namespace carries railiance.io/postgres-client: platform-pg, which is what platform-pg-consumer-ingress in rapp-postgres admits; without that label the pod cannot reach the database at all. NetworkPolicies default-deny both directions, then permit ingress from the user-engine namespace only, a separately labelled operator read path, and egress to PostgreSQL in databases plus DNS. Three decisions worth naming. Liveness is /healthz while readiness is /readyz, so a database outage drops the pod from the Service rather than restarting it in a loop. readOnlyRootFilesystem enforces the empty-filesystem property rather than trusting it, so the SQLite fallback physically cannot accumulate audit records on ephemeral storage. AUDIT_CORE_REQUIRE_CUSTODY_CLASS=archive makes a missing database URL a startup failure instead of a silent downgrade to the development store. Counters deferred from WP-0004-T06 are exposed as JSON at /v1/stats behind the read privilege, not as Prometheus exposition format: the cluster runs no Prometheus, no ServiceMonitor CRD and no other scrape target, so an exposition endpoint would target a scrape path that does not exist. Usable with curl now and a small step from /metrics later. Tests 77 -> 80. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-10 17:42:43 +02:00
| task | AUDIT-WP-0005-T04 | done | — | workplans/AUDIT-WP-0005-postgres-store-and-production-deployment.md |
| task | AUDIT-WP-0005-T05 | done | — | workplans/AUDIT-WP-0005-postgres-store-and-production-deployment.md |
| task | AUDIT-WP-0005-T06 | done | — | workplans/AUDIT-WP-0005-postgres-store-and-production-deployment.md |
| task | AUDIT-WP-0006-T01 | done | — | workplans/AUDIT-WP-0006-honest-custody-and-canon-join.md |
| task | AUDIT-WP-0006-T02 | done | — | workplans/AUDIT-WP-0006-honest-custody-and-canon-join.md |
| task | AUDIT-WP-0006-T03 | done | — | workplans/AUDIT-WP-0006-honest-custody-and-canon-join.md |
| task | AUDIT-WP-0006-T04 | done | — | workplans/AUDIT-WP-0006-honest-custody-and-canon-join.md |
| task | AUDIT-WP-0006-T05 | done | — | workplans/AUDIT-WP-0006-honest-custody-and-canon-join.md |
| task | AUDIT-WP-0007-T01 | done | — | workplans/AUDIT-WP-0007-integrity-verification.md |
| task | AUDIT-WP-0007-T02 | done | — | workplans/AUDIT-WP-0007-integrity-verification.md |
| task | AUDIT-WP-0007-T03 | done | — | workplans/AUDIT-WP-0007-integrity-verification.md |
| task | AUDIT-WP-0007-T04 | done | — | workplans/AUDIT-WP-0007-integrity-verification.md |
| task | AUDIT-WP-0007-T05 | done | — | workplans/AUDIT-WP-0007-integrity-verification.md |
| task | AUDIT-WP-0008-T01 | done | — | workplans/AUDIT-WP-0008-tenancy-posture-alignment.md |
| task | AUDIT-WP-0008-T02 | done | — | workplans/AUDIT-WP-0008-tenancy-posture-alignment.md |
| task | AUDIT-WP-0008-T03 | done | — | workplans/AUDIT-WP-0008-tenancy-posture-alignment.md |
| task | AUDIT-WP-0008-T04 | done | — | workplans/AUDIT-WP-0008-tenancy-posture-alignment.md |
| task | AUDIT-WP-0008-T05 | done | — | workplans/AUDIT-WP-0008-tenancy-posture-alignment.md |
| task | AUDIT-WP-0008-T06 | done | — | workplans/AUDIT-WP-0008-tenancy-posture-alignment.md |
| task | AUDIT-WP-0008-T07 | progress | — | workplans/AUDIT-WP-0008-tenancy-posture-alignment.md |
| task | AUDIT-WP-0008-T08 | done | — | workplans/AUDIT-WP-0008-tenancy-posture-alignment.md |
| task | AUDIT-WP-0009-T01 | todo | — | workplans/AUDIT-WP-0009-evidence-role-conformance.md |
| task | AUDIT-WP-0009-T02 | todo | — | workplans/AUDIT-WP-0009-evidence-role-conformance.md |
| task | AUDIT-WP-0009-T03 | todo | — | workplans/AUDIT-WP-0009-evidence-role-conformance.md |
| task | AUDIT-WP-0009-T04 | todo | — | workplans/AUDIT-WP-0009-evidence-role-conformance.md |
| task | AUDIT-WP-0009-T05 | wait | — | workplans/AUDIT-WP-0009-evidence-role-conformance.md |
| task | AUDIT-WP-0009-T06 | todo | — | workplans/AUDIT-WP-0009-evidence-role-conformance.md |
| task | AUDIT-WP-0009-T07 | todo | — | workplans/AUDIT-WP-0009-evidence-role-conformance.md |
| task | AUDIT-WP-0009-T08 | todo | — | workplans/AUDIT-WP-0009-evidence-role-conformance.md |
| task | AUDIT-WP-0009-T09 | todo | — | workplans/AUDIT-WP-0009-evidence-role-conformance.md |
| task | AUDIT-WP-0009-T10 | todo | — | workplans/AUDIT-WP-0009-evidence-role-conformance.md |
| intake | AUDIT-IN-0001 | closed | — | intakes/intakes.md |
| intake | AUDIT-IN-0002 | open | — | intakes/intakes.md |