coulomb/audit-core
5
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

AUDIT-WP-0017: propose production audit sources, correlation id, restore drill (NK-WP-0055-T11)

Browse source 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Assistant: claude-code
Assistant-Model: opus
Assistant-Process: 4515@bnt-lap001
Assistant-Session: 0bf96127-6e04-41be-bfda-d11717c3f354
This commit is contained in:
tegwick 2026-10-27 22:31:48 +01:00
parent 3ea5a63d8c
commit ead0f09f38
1 changed files with 177 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

177
workplans/AUDIT-WP-0017-production-audit-sources-and-correlation.md Normal file
View file

@ -0,0 +1,177 @@
---
id: AUDIT-WP-0017
type: workplan
title: "Production audit sources, correlation-id contract, restore drill and offsite chain head"
domain: infotech
repo: audit-core
status: proposed
flavor: implementation
owner: audit-core
topic_slug: audit-evidence-dependencies
created: "2026-10-27"
updated: "2026-10-27"
related:
- NK-WP-0055
- AUDIT-WP-0013
- AUDIT-WP-0015
---
# AUDIT-WP-0017 — Production audit sources, correlation-id contract, restore drill and offsite chain head
Receiving workplan in `audit-core` for NetKingdom NK-WP-0055-T11 (production readiness, area 9 "Audit sink"
in `net-kingdom/docs/evidence/2026-10-27-production-readiness-ledger.md`). The goal is that every production
audit source lands in the durable, hash-chained sink and that records across streams can be joined by one
correlation id.
Topaz is not a source: NetKingdom ADR-0018 makes the access-engine Go core the production PDP.
Owner split:
- audit-core (this plan): sender admission, envelope, scope rows, retention, restore drill, chain-head export.
- railiance-platform: OpenBao audit-device forwarding.
- railiance-cluster: Kubernetes API audit policy and forwarding.
- access-engine (and flex-auth for its pins): decision records, tracked in AUDIT-WP-0013-T02.
- NetKingdom: identity-provider event forwarding and the cross-stream correlation contract.
- resource-control / rapp-postgres: offsite chain-head custody, tracked in AUDIT-WP-0015-T01.
This file plans work only. It activates no sender, mints no credential and makes no live change. Each sender
task stays `wait` until its owner returns the inputs listed. The general admission rules are in
`docs/cross-repo-evidence-admission.md` and `docs/cadence-registration.md`.
## Common intake package for every new sender
Each sender owner must return, before audit-core prepares an overlay:
1. **Source registration**: stable `source` name, owning repo, namespace/pod selectors, the emitting
identity, and the tenant policy (platform-scoped or per-tenant).
2. **Envelope mapping**: how native records map to the envelope in `docs/event-envelope.md`
(`event_id` derivation for idempotent retry, `observed_at`, `actor`, `action`, `outcome`, `tenant`,
`correlation_id`). The raw native record may travel as payload but must hold no secret value.
3. **Scope rows**: the action vocabulary the source may emit, so refusals are explicit.
4. **Volume and retention**: expected events/day and peak rate, `emission_cadence` with heartbeat and
the allowed silence threshold, and the required retention (default 30-day query window plus archive).
5. **Credential path**: the sender secret is minted through the attended OpenBao path routed by
`warden route`; audit-core never holds or prints it. `may_read` stays false.
## Admit OpenBao audit forwarding
```task
id: AUDIT-WP-0017-T01
status: wait
priority: high
blocking_reason: "railiance-platform must supply a forwarder design for the OpenBao audit device (today audit only lands on a PVC) and the common intake package."
blocked_on: message-from:railiance-platform
```
Needs from railiance-platform: the forwarder (socket or file-tail device to the receiver), HMAC'd fields kept
HMAC'd (no plaintext secret or token in the payload), request-id mapping to `event_id`, and the volume estimate.
Fail-closed question for the owner: whether OpenBao should block on audit-sink loss (OpenBao's own semantics)
or buffer locally; audit-core needs the answer recorded, not a particular choice.
**Done when:** a scoped overlay is prepared, acceptance/retry/refusal/heartbeat are verified on synthetic
fixtures, and an owner-run activation receipt lands.
## Admit Kubernetes API audit
```task
id: AUDIT-WP-0017-T02
status: wait
priority: high
blocking_reason: "railiance-cluster must supply the k3s audit policy (levels and excluded noise) and a forwarding path (webhook backend or log shipper) plus the common intake package."
blocked_on: message-from:railiance-cluster
```
Needs from railiance-cluster: audit policy that records Secret access at Metadata level only (never
RequestResponse for Secrets), `auditID` as `event_id`, user/impersonation fields as `actor`, and a volume
estimate after noise exclusion. audit-core checks the receiver can sustain that rate and sizes retention.
**Done when:** overlay prepared, fixture tests pass, owner-run activation receipt lands.
## Admit access-engine decision records
```task
id: AUDIT-WP-0017-T03
status: wait
priority: high
depends_on: [AUDIT-WP-0013-T02]
blocking_reason: "Tracked by AUDIT-WP-0013-T02; waits on flex-auth/access-engine registrations and attended custody."
blocked_on: message-from:access-engine
```
No separate admission here. This task closes when AUDIT-WP-0013-T02 is done and the decision records carry the
correlation id from T05 (decision id plus the caller's correlation id).
## Admit identity-provider events
```task
id: AUDIT-WP-0017-T04
status: wait
priority: high
blocking_reason: "NetKingdom must supply one intake package per provider (KeyCape, Authelia, LLDAP, privacyIDEA) and say which events are in scope."
blocked_on: message-from:net-kingdom
```
Needs from NetKingdom: per provider, the event set (login success/failure, MFA enrolment and challenge, token
issue/revoke, user and group changes, admin actions), the export mechanism (native webhook, log tail or
collector), and how the provider's session or request id maps to `correlation_id`. Four separate `source`
names, not one merged stream, so silence in one provider is detectable.
**Done when:** four overlays prepared, fixture tests pass, owner-run activation receipts land.
## Define the correlation_id field and propagation contract
```task
id: AUDIT-WP-0017-T05
status: todo
priority: high
```
The envelope already carries `correlation_id` (free-form string) and the receiver serves
`GET ...?correlation_id=` across tenants (`audit_core/ingestion.py`). Proposal for the envelope side:
- **Format**: an opaque string of at most 128 chars from `[A-Za-z0-9._:-]`, no PII and no secret. Prefer a
W3C trace-context `trace-id` (32 lowercase hex) when the caller has one, so the same value can sit in logs
and traces.
- **Required** for the new production sources; the receiver refuses an empty value for those sources and
keeps today's behaviour for already-admitted senders until they migrate.
- **Origin rule**: the first component in a chain (normally the IdP login or the onboarding request at
user-engine/tenant-engine) mints it; every later hop copies it unchanged. A component that has none mints one
and records `causation` metadata pointing at the native request id.
- **Optional `parent_event_id`** for the direct cause, so a chain can be ordered without trusting clocks.
NetKingdom owns the cross-stream contract (which hop mints, the propagation header name, and the onboarding chain
test). audit-core owns the field definition, validation and query. Deliverables: update `docs/event-envelope.md`,
receiver validation behind a per-source flag, tests, and a synthetic onboarding-chain fixture
(IdP login, tenant-engine onboarding, access-engine decision, OpenBao read, Kubernetes API call) that is joined
by one id through the query endpoint.
**Done when:** NetKingdom accepts the field definition, the doc and tests are committed, and the fixture
query returns the whole chain.
## Replace the single restore walk with a recurring drill
```task
id: AUDIT-WP-0017-T06
status: todo
priority: high
```
The only restore evidence is the 2026-08-13 walk. Define a recurring drill: quarterly, and after any schema or
backend change. Each drill restores the newest backup into an isolated namespace or database (never over the
live store), runs the integrity verifier over the restored chain, compares head digest, event count and newest
`observed_at` with the live head, and records a dated non-secret receipt under `evidence/`. The drill is
founder- or operator-run; audit-core supplies the script, the runbook section in `docs/operator-runbook.md`
and the receipt schema, tested against a synthetic store. A missed drill is a stale-evidence finding.
**Done when:** the procedure and script are committed, tested on fixtures, and the first owner-run drill
receipt lands.
## Close the offsite chain head (AUDIT-WP-0015-T01)
```task
id: AUDIT-WP-0017-T07
status: wait
priority: high
depends_on: [AUDIT-WP-0015-T01]
blocking_reason: "AUDIT-WP-0015-T01 still waits on resource-control and rapp-postgres to name the chain-head custody owner and return offsite upload/readback receipts."
blocked_on: message-from:resource-control
```
Status 2026-10-27: AUDIT-WP-0015-T01 is `wait`. No owner or task for chain-head custody has been named and no
offsite upload/readback receipt exists. This task closes when that one does; it adds only the T06 link: each
drill also checks the restored head against the latest offsite head.