audit-core/docs/operator-runbook.md

audit-core operator runbook

Production receiver on railiance01. Custody is PostgreSQL (platform-pg, database audit_core). This runbook is the T06 handover: lookup, dead letters, replay, credential rotation, alerts, and restore.

In-cluster URL:

http://audit-core.audit-core.svc.cluster.local:8080

Reachability is part of the threat model. Ingress is allowed only from the user-engine namespace (write) and from namespaces labelled railiance.io/audit-core-reader=true (operator read). From a workstation, kubectl -n audit-core port-forward svc/audit-core 8080:8080 is the usual path and does not require that label.

Do not paste sender tokens, database passwords, or OpenBao tokens into Git, State Hub, workplans, logs, or chat. Route first:

warden route find "database credential" --json
warden route show database-dynamic-credentials --json

Readiness and health

Check	Meaning
GET /healthz	Process is up. Liveness uses this. A database outage must not restart the pod.
GET /readyz	Custody is reachable and custody_class=archive. Readiness uses this; the pod leaves the Service rather than accept events it cannot store.
GET /v1/stats	In-process counters since start (accepted, duplicate, conflict, rejected, unauthorized, forbidden, unavailable, error). Resets on restart. Requires may_read.

A missing AUDIT_CORE_DATABASE_URL / credential directory is a startup failure (AUDIT_CORE_REQUIRE_CUSTODY_CLASS=archive), not a silent downgrade to SQLite.

Lookup

All read routes require a sender identity with may_read: true. A write credential must not be able to read the trail back.

# One event
curl -sS -H "Authorization: Bearer $OPERATOR_TOKEN" \
  http://127.0.0.1:8080/v1/events/<event-id>

# Every event sharing a correlation id
curl -sS -H "Authorization: Bearer $OPERATOR_TOKEN" \
  'http://127.0.0.1:8080/v1/events?correlation_id=<id>&limit=100'

Stored records keep the original event_id, payload_hash, and accepted_at. Redacted fields appear in details.redaction.paths; the payload itself is stored already masked.

Dead letters and replay

curl -sS -H "Authorization: Bearer $OPERATOR_TOKEN" \
  'http://127.0.0.1:8080/v1/dead-letters?limit=100'

A dead letter is a rejected submission (400). Retrying it unchanged will not help. Typical reasons: invalid_event, source_not_allowed, tenant_not_allowed, secret_shaped_field. Payloads rejected for secret shape are withheld (payload_withheld: true).

To replay after the sender is fixed, POST the corrected event with the same id and Idempotency-Key. The store is idempotent: an already-accepted identical payload returns 200 duplicate and does not create a second record. A different payload under the same id returns 409.

Reconciling an already stored event (the WP-0005-T05 S13 case):

# From a shell that already has a migration or runtime lease.
python3 -m audit_core replay --event-id <event-id>

That command must report "duplicate": true. A first-acceptance result is a custody defect: stop and investigate.

Sender credential rotation

First deploy mints the registry in-cluster as Secret audit-core-senders (ops-mason plan audit-core-openbao-runtime-custody). The OpenBao path platform/workloads/audit-core/senders is the later authority after a wrap-migrate; do not bao kv put it by hand. Rotation is overlap-first:

1. Add the replacement token to the sender's tokens list. Both work.
2. Move the sender to the new token.
3. Drop the predecessor from the list.
4. Write the updated document back (bao kv put from a mode-0600 file).
5. Either wait for the 1h refresh or annotate the ExternalSecret to force a sync, then restart the pod so it re-reads AUDIT_CORE_SENDERS.

The write token is bound to source=user-engine and the tenants that identity may claim. The operator token is a separate identity with may_read: true. Do not reuse one token for both.

A shape (values are placeholders) is in docs/senders.example.json.

Database credential rotation

Runtime leases come from database/creds/audit-core-runtime. ESO refreshes every 15 minutes into Secret audit-core-database, mounted at /etc/audit-core/db. The process re-reads those files on every new connection. No restart is required; a restart would be a delivery gap.

Rotation is logged by password fingerprint, never by value:

database credential rotated (fingerprint <12 hex chars>)

A revoked lease surfaces as /readyz 503 and request 503 unavailable. The sender contract retries 503/500 and treats 400/401/403/409 as terminal.

Schema changes are a Job (deploy/migrate-job.yaml) using database/creds/audit-core-migration. The runtime Deployment sets AUDIT_CORE_AUTO_MIGRATE=0. After a new migration ships, delete the old Job and re-apply.

Alert conditions

There is no Prometheus scrape path on railiance01 today (/v1/stats is JSON behind the read privilege). Watch:

Signal	Why
/readyz not 200	Custody is down; senders will retry. Do not bounce the pod.
counts.unavailable or counts.error climbing	Database or unexpected fault. Check platform-pg and the mounted lease.
counts.unauthorized climbing	Sender using a dropped token, or registry not refreshed.
counts.conflict non-zero	Same event_id, different payload. Sender bug or id reuse.
GET /v1/secret-findings	Secret-shaped fields by path. Fix the sending service.
Pod not Ready after rotate	Stale or revoked lease; check ExternalSecret status, not the password.

platform-pg operations, including what to scrape on port 9187, live in rapp-postgres/docs/operations.md.

Restore

audit-core does not expire events (retention_days unset). Recovery is bounded by what rapp-postgres can restore, not by an audit-core deletion window. The platform's planned Barman window is 30 days. Production Barman is still fail-closed (no governed off-host target). Do not promise an RPO until that target exists. Local WAL on the node is not an off-host copy.

Physical restore is instance-wide. A consumer-only restore is a logical export of audit_core from a scratch physical restore, then a controlled import. Never recover in place. Procedure: rapp-postgres/docs/restore.md.

Walked 2026-08-13:

Path	Evidence	Elapsed	Result
Platform full restore + PITR of live audit_core (21 events)	rapp-postgres/docs/evidence/backup-restore-20260813T111651Z-remote.json	51.0s / 52.5s	triples + events_append_only
Platform logical single-consumer export/import	same	3.5s	21 events
audit-core logical dump of live audit_core → local PG 16	evidence/restore-walk-20260813T121200Z.json	4.1s	21/3/1 counts, 5 triples, trigger present

Verify after any restore:

SELECT count(*) FROM audit_core.events;
SELECT event_id, payload_hash, accepted_at FROM audit_core.events
  WHERE event_id IN (/* sample from pre-restore */);
SELECT tgname FROM pg_trigger WHERE tgname = 'events_append_only';

Rollback

The Deployment annotation audit-core.railiance.io/rollback-note is the source of truth. Migrations 0001-0005 are additive; kubectl -n audit-core rollout undo deploy/audit-core returns to the previous digest and an older image runs against the newer schema. A future migration that drops or narrows a column must replace that note before release.

Deploy order

1. Image published and digest-pinned.
2. Namespace (label railiance.io/postgres-client: platform-pg), NetworkPolicies, ClusterSecretStores, ExternalSecrets.
3. Sender registry is Secret audit-core-senders (in-cluster mint). Database leases come from openbao-audit-core-database.
4. Job audit-core-migrate with AUDIT_CORE_MIGRATE_ROLE=audit_core_migrate.
5. Deployment. /readyz must report custody_class=archive.
6. In-pod MODE=remote DISRUPT=0 failure matrix. Evidence goes to NK-WP-0024.