audit-core/docs/operator-runbook.md

audit-core operator runbook

Production receiver on railiance01. Custody is PostgreSQL (platform-pg, database audit_core). This runbook is the T06 handover: lookup, dead letters, replay, credential rotation, alerts, and restore.

In-cluster URL:

http://audit-core.audit-core.svc.cluster.local:8080

Reachability is part of the threat model. Ingress is allowed only from the user-engine namespace (write) and from namespaces labelled railiance.io/audit-core-reader=true (operator read). From a workstation, kubectl -n audit-core port-forward svc/audit-core 8080:8080 is the usual path and does not require that label.

Do not paste sender tokens, database passwords, or OpenBao tokens into Git, State Hub, workplans, logs, or chat. Route first:

warden route find "database credential" --json
warden route show database-dynamic-credentials --json

Readiness and health

Check	Meaning
GET /healthz	Process is up. Liveness uses this. A database outage must not restart the pod.
GET /readyz	Custody is reachable and custody_class=archive. Readiness uses this; the pod leaves the Service rather than accept events it cannot store.
GET /v1/stats	In-process counters since start (accepted, duplicate, conflict, rejected, unauthorized, forbidden, unavailable, error). Resets on restart. Requires may_read.

A missing AUDIT_CORE_DATABASE_URL / credential directory is a startup failure (AUDIT_CORE_REQUIRE_CUSTODY_CLASS=archive), not a silent downgrade to SQLite.

Lookup

All read routes require a sender identity with may_read: true. A write credential must not be able to read the trail back.

# One event
curl -sS -H "Authorization: Bearer $OPERATOR_TOKEN" \
  http://127.0.0.1:8080/v1/events/<event-id>

# Every event sharing a correlation id
curl -sS -H "Authorization: Bearer $OPERATOR_TOKEN" \
  'http://127.0.0.1:8080/v1/events?correlation_id=<id>&limit=100'

Stored records keep the original event_id, payload_hash, and accepted_at. Redacted fields appear in details.redaction.paths; the payload itself is stored already masked.

Dead letters and replay

curl -sS -H "Authorization: Bearer $OPERATOR_TOKEN" \
  'http://127.0.0.1:8080/v1/dead-letters?limit=100'

A dead letter is a rejected submission (400). Retrying it unchanged will not help. Typical reasons: invalid_event, source_not_allowed, tenant_not_allowed, secret_shaped_field. Payloads rejected for secret shape are withheld (payload_withheld: true).

To replay after the sender is fixed, POST the corrected event with the same id and Idempotency-Key. The store is idempotent: an already-accepted identical payload returns 200 duplicate and does not create a second record. A different payload under the same id returns 409.

Reconciling an already stored event (the WP-0005-T05 S13 case):

# From a shell that already has a migration or runtime lease.
python3 -m audit_core replay --event-id <event-id>

That command must report "duplicate": true. A first-acceptance result is a custody defect: stop and investigate.

Sender credential rotation

First deploy mints the registry in-cluster as Secret audit-core-senders (ops-mason plan audit-core-openbao-runtime-custody). The OpenBao path platform/workloads/audit-core/senders is the later authority after a wrap-migrate; do not bao kv put it by hand. Rotation is overlap-first:

1. Add the replacement token to the sender's tokens list. Both work.
2. Move the sender to the new token.
3. Drop the predecessor from the list.
4. Write the updated document back (bao kv put from a mode-0600 file).
5. Either wait for the 1h refresh or annotate the ExternalSecret to force a sync, then restart the pod so it re-reads AUDIT_CORE_SENDERS.

The write token is bound to source=user-engine and the tenants that identity may claim. The operator token is a separate identity with may_read: true. Do not reuse one token for both.

A shape (values are placeholders) is in docs/senders.example.json.

Database credential rotation

Runtime leases come from database/creds/audit-core-runtime. ESO refreshes every 15 minutes into Secret audit-core-database, mounted at /etc/audit-core/db. The process re-reads those files on every new connection. No restart is required; a restart would be a delivery gap.

Rotation is logged by password fingerprint, never by value:

database credential rotated (fingerprint <12 hex chars>)

A revoked lease surfaces as /readyz 503 and request 503 unavailable. The sender contract retries 503/500 and treats 400/401/403/409 as terminal.

Schema changes are a Job (deploy/migrate-job.yaml) using database/creds/audit-core-migration. The runtime Deployment sets AUDIT_CORE_AUTO_MIGRATE=0. After a new migration ships, delete the old Job and re-apply.

Alert conditions

There is no Prometheus scrape path on railiance01 today (/v1/stats is JSON behind the read privilege). Watch:

Signal	Why
/readyz not 200	Custody is down; senders will retry. Do not bounce the pod.
counts.unavailable or counts.error climbing	Database or unexpected fault. Check platform-pg and the mounted lease.
counts.unauthorized climbing	Sender using a dropped token, or registry not refreshed.
counts.conflict non-zero	Same event_id, different payload. Sender bug or id reuse.
GET /v1/secret-findings	Secret-shaped fields by path. Fix the sending service.
Pod not Ready after rotate	Stale or revoked lease; check ExternalSecret status, not the password.

platform-pg operations, including what to scrape on port 9187, live in rapp-postgres/docs/operations.md.

Restore

audit-core declares custody_class=archive and immutable=True (the append-only trigger). It does not declare a retention window shorter than the platform's, because it currently declares none (retention_days unset = keep). That is only honest if the platform's backup retention is also unbounded or is an explicit accepted loss.

As of 2026-08-13, RAPP-POSTGRES-WP-0002-T06 is wait. Production backup is fail-closed until an S3-compatible Barman target exists. The governed Nextcloud logical-dump lane cannot provide WAL archiving or PITR. Do not claim an RPO/RTO for audit-core until those drills have been recorded in rapp-postgres.

When the platform restore path is open, the audit-core walk is:

1. Restore platform-pg to a scratch cluster (platform-pg-restore-full or PITR). Never recover in place.
2. Physical restore is instance-wide. A single-consumer restore is a logical dump of audit_core from the scratch cluster, then a controlled import.
3. Verify: event counts, a sample of event_id/payload_hash/accepted_at triples, and that the append-only trigger is still installed (SELECT tgname FROM pg_trigger WHERE tgname = 'events_append_only').
4. Record elapsed time. That number is what this service can promise.

Until that walk exists, a workstation-side logical dump of an empty audit_core is not evidence of recovery.

Rollback

The Deployment annotation audit-core.railiance.io/rollback-note is the source of truth. Migrations 0001-0004 are additive; kubectl -n audit-core rollout undo deploy/audit-core returns to the previous digest and an older image runs against the newer schema. A future migration that drops or narrows a column must replace that note before release.

Deploy order

1. Image published, digest pinned in deploy/audit-core.yaml and deploy/migrate-job.yaml.
2. Attended scripts/openbao-eso-token-apply.sh.
3. bao kv put platform/workloads/audit-core/senders senders.json=@file from a mode-0600 file; shred the file.
4. Apply namespace (carries railiance.io/postgres-client: platform-pg), NetworkPolicies, ClusterSecretStore, VaultDynamicSecrets, ExternalSecrets.
5. Wait until Secrets audit-core-database, audit-core-database-migrate, and audit-core-senders exist.
6. Apply and wait for Job audit-core-migrate.
7. Apply the Deployment. Confirm /readyz reports custody_class=archive.
8. Run MODE=remote BASE_URL=… make failure-matrix and hand the JSON to NK-WP-0024.