audit-core/docs/operator-runbook.md

audit-core operator runbook

Production receiver on railiance01. Custody is PostgreSQL (platform-pg, database audit_core). This runbook is the T06 handover: lookup, dead letters, replay, credential rotation, alerts, and restore.

In-cluster URL:

http://audit-core.audit-core.svc.cluster.local:8080

Reachability is part of the threat model. Ingress is allowed only from the user-engine namespace (write) and from namespaces labelled railiance.io/audit-core-reader=true (operator read). From a workstation, kubectl -n audit-core port-forward svc/audit-core 8080:8080 is the usual path and does not require that label.

Do not paste sender tokens, database passwords, or OpenBao tokens into Git, State Hub, workplans, logs, or chat. Route first:

warden route find "database credential" --json
warden route show database-dynamic-credentials --json

Readiness and health

Check	Meaning
GET /healthz	Process is up. Liveness uses this. A database outage must not restart the pod.
GET /readyz	Custody is reachable and custody_class=operational. Also reports recoverable_days and tamper_evidence. Readiness uses this; the pod leaves the Service rather than accept events it cannot store.
GET /v1/stats	In-process counters since start (accepted, duplicate, conflict, rejected, unauthorized, forbidden, unavailable, error). Resets on restart. Requires may_read and full tenant scope.
GET /v1/integrity	Hash-chain walk: {intact, events, head, first_break}. No payloads. Requires may_read and full tenant scope. A break is a custody defect, not a sender retry.

A missing AUDIT_CORE_DATABASE_URL / credential directory is a startup failure (AUDIT_CORE_REQUIRE_CUSTODY_CLASS=operational), not a silent downgrade to SQLite. An older manifest that still requires archive is accepted as an alias for one mixed rollout.

After a node reboot, /readyz failing for tens of seconds is expected: CoreDNS and platform-pg come up after the receiver. Liveness stays on /healthz so that window does not restart the pod. Walked 2026-08-16 (~40s unreadiness, then Ready). That window is not a chain break.

Integrity

Each accept extends a single hash chain for the schema (not per tenant). python -m audit_core verify-chain exits 0 only if the walk is intact. python -m audit_core attest-chain --output docs/evidence/chain-head-<ts>.json writes the live head outside platform-pg. Do not put that file in the Barman prefix; a second copy may follow the RESOURCE-WP-0002-T06 logical-offsite path. verify-chain --against <file> fails if the cited head is not in the live chain.

A break is a custody defect. Do not "fix" it by accepting the same event again. Reboot unreadiness is unrelated.

Lookup

All read routes require a sender identity with may_read: true. A write credential must not be able to read the trail back.

Read is also tenant-scoped (AUDIT-WP-0008-T04). A reader sees only the tenants its identity may act for, and the two rules differ by surface:

Surface	Behaviour for a scoped reader
GET /v1/events/<id>	Another tenant's event returns 404, identical to an id that does not exist. A distinguishable 403 would confirm the event exists and whose it is.
GET /v1/events?correlation_id=	Filtered to the readable slice. A correlation id legitimately spans services and tenants, so this is filtered rather than refused.
/v1/stats, /v1/integrity, /v1/dead-letters, /v1/secret-findings	403 full_tenant_scope_required. These carry no tenant key, so there is nothing to filter on; they are reserved for an identity with tenants: ["*"].

An operator identity holding tenants: ["*"] is unaffected and sees everything, which is what $OPERATOR_TOKEN below assumes. A reader scoped to one tenant is the case this protects: before T04, may_read was the only gate and any reader could read every tenant.

# One event
curl -sS -H "Authorization: Bearer $OPERATOR_TOKEN" \
  http://127.0.0.1:8080/v1/events/<event-id>

# Every event sharing a correlation id
curl -sS -H "Authorization: Bearer $OPERATOR_TOKEN" \
  'http://127.0.0.1:8080/v1/events?correlation_id=<id>&limit=100'

Stored records keep the original event_id, payload_hash, and accepted_at. Redacted fields appear in details.redaction.paths; the payload itself is stored already masked.

Dead letters and replay

curl -sS -H "Authorization: Bearer $OPERATOR_TOKEN" \
  'http://127.0.0.1:8080/v1/dead-letters?limit=100'

A dead letter is a rejected submission (400). Retrying it unchanged will not help. Typical reasons: invalid_event, source_not_allowed, tenant_not_allowed, secret_shaped_field. Payloads rejected for secret shape are withheld (payload_withheld: true).

To replay after the sender is fixed, POST the corrected event with the same id and Idempotency-Key. The store is idempotent: an already-accepted identical payload returns 200 duplicate and does not create a second record. A different payload under the same id returns 409.

Reconciling an already stored event (the WP-0005-T05 S13 case):

# From a shell that already has a migration or runtime lease.
python3 -m audit_core replay --event-id <event-id>

That command must report "duplicate": true. A first-acceptance result is a custody defect: stop and investigate.

Sender credential rotation

Governed E2 evidence identities

The whitehat test plane has a dedicated ingress policy. It admits only pods in namespace whitehat carrying both whitehat.security/plane: "true" and whitehat.security/target: audit-core; bearer authentication and sender scope still apply to every request. The policy is connectivity, not authorization.

A dated live-e2 engagement may request exactly two temporary identities. Each identity is bound to source=whitehat-security, one named synthetic tenant, may_write=true, may_read=true, and an RFC3339 expires_at no later than the engagement end. Audit-core evaluates expiry on every authentication, so a stale registry copy cannot extend the bearer lifetime. Provision token values only through the sanctioned sender-registry custody lane and the plane's mount-only credential projection. Never reuse the production user-engine identity. Revoke both identities and remove their registry entries when the plane lease ends; application expiry is a backstop, not a substitute for evidenced cleanup.

OpenBao path platform/workloads/audit-core/senders is the authority for ExternalSecret audit-core-senders. The initial in-cluster registry was wrap-migrated on 2026-08-14 without printing or staging its values; do not replace that path with a founder paste. Rotation is overlap-first:

1. Add the replacement token to the sender's tokens list. Both work.
2. Move the sender to the new token.
3. Drop the predecessor from the list.
4. Write the updated document back (bao kv put from a mode-0600 file).
5. Either wait for the 1h refresh or annotate the ExternalSecret to force a sync, then restart the pod so it re-reads AUDIT_CORE_SENDERS.

The write token is bound to source=user-engine. Tenant scope for that identity is not a secret: deploy/senders-scope.json (ConfigMap audit-core-senders-scope) overlays tenants: ["*"] onto the Secret at start. An ExternalSecret refresh cannot revert it to a single tenant. Tokens stay in Secret audit-core-senders / OpenBao KV.

The operator token is a separate identity with may_read: true. Do not reuse one token for both.

A shape (values are placeholders) is in docs/senders.example.json.

Database credential rotation

Runtime leases come from database/creds/audit-core-runtime. ESO refreshes every 15 minutes into Secret audit-core-database, mounted at /etc/audit-core/db. The process re-reads those files on every new connection. No restart is required; a restart would be a delivery gap.

Rotation is logged by password fingerprint, never by value:

database credential rotated (fingerprint <12 hex chars>)

A revoked lease surfaces as /readyz 503 and request 503 unavailable. The sender contract retries 503/500 and treats 400/401/403/409 as terminal.

Schema changes are a Job (deploy/migrate-job.yaml) using database/creds/audit-core-migration. The runtime Deployment sets AUDIT_CORE_AUTO_MIGRATE=0. After a new migration ships, delete the old Job and re-apply.

Alert conditions

There is no Prometheus scrape path on railiance01 today (/v1/stats is JSON behind the read privilege). Watch:

Signal	Why
/readyz not 200	Custody is down; senders will retry. Do not bounce the pod.
counts.unavailable or counts.error climbing	Database or unexpected fault. Check platform-pg and the mounted lease.
counts.unauthorized climbing	Sender using a dropped token, or registry not refreshed.
counts.conflict non-zero	Same event_id, different payload. Sender bug or id reuse.
GET /v1/secret-findings	Secret-shaped fields by path. Fix the sending service.
Pod not Ready after rotate	Stale or revoked lease; check ExternalSecret status, not the password.

platform-pg operations, including what to scrape on port 9187, live in rapp-postgres/docs/operations.md.

Restore

audit-core does not expire events (retention_days unset). That is a lifecycle policy statement, not a recovery guarantee. Recoverable history is the platform data.backup window: 30 days, prefix platform-pg/, bucket owned by resource:platform:audit-storage. Cite, do not copy:

• resource-control/data/capability/platform-audit-storage.json
• rapp-postgres/docs/restore.md

The platform requirement is data.backup profile database at D5 (RPO 5 min, RTO 60 min, 30-day retention, not in the railiance01 / host-europe failure domain). The live provision is D4, not D5: resource-control scored one backup, one full restore, one PITR, and does not yet emit wal_archive_gap_minutes. RPO/RTO numbers are theirs (measured, single observation). audit-core does not claim a better grade.

Rows older than the 30-day window are not promised after a restore. Local WAL on the node is not a second copy.

Physical restore is instance-wide. A consumer-only restore is a logical export of audit_core from a scratch physical restore, then a controlled import. Never recover in place. Procedure: rapp-postgres/docs/restore.md.

The 2026-08-13 fail-closed sentence is superseded by RESOURCE-WP-0002-T05 (2026-08-14): production Barman to Scaleway, audit_core.events 30=30, full restore 65 s, PITR 65 s.

Walked 2026-08-13 (pre-commissioning historical evidence):

Path	Evidence	Elapsed	Result
Platform full restore + PITR of live audit_core (21 events)	rapp-postgres/docs/evidence/backup-restore-20260813T111651Z-remote.json	51.0s / 52.5s	triples + events_append_only
Platform logical single-consumer export/import	same	3.5s	21 events
audit-core logical dump of live audit_core → local PG 16	evidence/restore-walk-20260813T121200Z.json	4.1s	21/3/1 counts, 5 triples, trigger present

Verify after any restore:

SELECT count(*) FROM audit_core.events;
SELECT event_id, payload_hash, accepted_at FROM audit_core.events
  WHERE event_id IN (/* sample from pre-restore */);
SELECT tgname FROM pg_trigger WHERE tgname = 'events_append_only';

Rollback

The Deployment annotation audit-core.railiance.io/rollback-note is the source of truth. Migrations 0001-0005 are additive; kubectl -n audit-core rollout undo deploy/audit-core returns to the previous digest and an older image runs against the newer schema. A future migration that drops or narrows a column must replace that note before release.

Deploy order

1. Image published and digest-pinned.
2. Namespace (label railiance.io/postgres-client: platform-pg), NetworkPolicies, ClusterSecretStores, ExternalSecrets.
3. Sender registry is Secret audit-core-senders (in-cluster mint). Database leases come from openbao-audit-core-database.
4. Job audit-core-migrate with AUDIT_CORE_MIGRATE_ROLE=audit_core_migrate.
5. Deployment. /readyz must report custody_class=operational and recoverable_days=30.
6. In-pod MODE=remote DISRUPT=0 failure matrix. Evidence goes to NK-WP-0024.