# audit-core operator runbook Production receiver on railiance01. Custody is PostgreSQL (`platform-pg`, database `audit_core`). This runbook is the T06 handover: lookup, dead letters, replay, credential rotation, alerts, and restore. In-cluster URL: ```text http://audit-core.audit-core.svc.cluster.local:8080 ``` Reachability is part of the threat model. Ingress is allowed from the declared producer peers in `deploy/networkpolicies.yaml` (write) and from namespaces labelled `railiance.io/audit-core-reader=true` (operator read). From a workstation, `kubectl -n audit-core port-forward svc/audit-core 8080:8080` is the usual path and does not require that label. Do not paste sender tokens, database passwords, or OpenBao tokens into Git, State Hub, workplans, logs, or chat. Route first: ```bash warden route find "database credential" --json warden route show database-dynamic-credentials --json ``` ## Readiness and health | Check | Meaning | | --- | --- | | `GET /healthz` | Process is up. Liveness uses this. A database outage must **not** restart the pod. | | `GET /readyz` | Custody is reachable (`SELECT 1`) within a 1.5s budget. Reports `custody_class`, `recoverable_days`, and last-known `tamper_evidence`. Does **not** walk the hash chain — kubelet `timeoutSeconds` is 2, and a hung walk takes the pod out of the Service. Readiness uses this; the pod leaves the Service rather than accept events it cannot store. Integrity evaluation is `/v1/integrity`. | | `GET /v1/stats` | In-process counters since start (`accepted`, `duplicate`, `conflict`, `rejected`, `unauthorized`, `forbidden`, `unavailable`, `error`). Resets on restart. Requires `may_read` and full tenant scope. | | `GET /v1/integrity` | Hash-chain walk: `{intact, events, head, first_break}`. No payloads. Requires `may_read` and full tenant scope. A break is a custody defect, not a sender retry. | A missing `AUDIT_CORE_DATABASE_URL` / credential directory is a startup failure (`AUDIT_CORE_REQUIRE_CUSTODY_CLASS=operational`), not a silent downgrade to SQLite. An older manifest that still requires `archive` is accepted as an alias for one mixed rollout. After a node reboot, `/readyz` failing for tens of seconds is expected: CoreDNS and `platform-pg` come up after the receiver. Liveness stays on `/healthz` so that window does not restart the pod. Walked 2026-08-16 (~40s unreadiness, then Ready). That window is not a chain break. ## Integrity Each accept extends a single hash chain for the schema (not per tenant). `python -m audit_core verify-chain` exits 0 only if the walk is intact. `python -m audit_core attest-chain --output docs/evidence/chain-head-.json` writes the live head **outside** `platform-pg`. Do not put that file in the Barman prefix; a second copy may follow the RESOURCE-WP-0002-T06 logical-offsite path. `verify-chain --against ` fails if the cited head is not in the live chain. A break is a custody defect. Do not "fix" it by accepting the same event again. Reboot unreadiness is unrelated. ## Lookup All read routes require a sender identity with `may_read: true`. A write credential must not be able to read the trail back. **Read is also tenant-scoped** (AUDIT-WP-0008-T04). A reader sees only the tenants its identity may act for, and the two rules differ by surface: | Surface | Behaviour for a scoped reader | |---|---| | `GET /v1/events/` | Another tenant's event returns **404**, identical to an id that does not exist. A distinguishable 403 would confirm the event exists and whose it is. | | `GET /v1/events?correlation_id=` | Filtered to the readable slice. A correlation id legitimately spans services and tenants, so this is filtered rather than refused. | | `/v1/stats`, `/v1/integrity`, `/v1/dead-letters`, `/v1/secret-findings` | **403 `full_tenant_scope_required`.** These carry no tenant key, so there is nothing to filter on; they are reserved for an identity with `tenants: ["*"]`. | An operator identity holding `tenants: ["*"]` is unaffected and sees everything, which is what `$OPERATOR_TOKEN` below assumes. A reader scoped to one tenant is the case this protects: before T04, `may_read` was the only gate and any reader could read every tenant. ```bash # One event curl -sS -H "Authorization: Bearer $OPERATOR_TOKEN" \ http://127.0.0.1:8080/v1/events/ # Every event sharing a correlation id curl -sS -H "Authorization: Bearer $OPERATOR_TOKEN" \ 'http://127.0.0.1:8080/v1/events?correlation_id=&limit=100' # Independent per-class counts for ANY source's window (AUDIT-WP-0013-T06). # Needs may_read AND tenants ["*"]; a scoped reader gets 403 # full_tenant_scope_required, a writer 403 source_not_allowed. # Window is half-open on accepted_at: since <= accepted_at < until. curl -sS -H "Authorization: Bearer $OPERATOR_TOKEN" \ 'http://127.0.0.1:8080/v1/reconciliation?source=&tenant=&since=&until=&stream_instance_id=' ``` The independent count response is the own-source shape plus `basis: "independent_reader"`: `source`, `tenant`, `since`, `until`, `counts` (`[{class, count}]`), optional `stream_instance_id`, and `means`. Counts only. Sequence and lag are not aggregated; take them from the exact-id reads above (`accepted_at`, `observed_at`, `details.data`). Stored records keep the original `event_id`, `payload_hash`, and `accepted_at`. Redacted fields appear in `details.redaction.paths`; the payload itself is stored already masked. ## Dead letters and replay ```bash curl -sS -H "Authorization: Bearer $OPERATOR_TOKEN" \ 'http://127.0.0.1:8080/v1/dead-letters?limit=100' ``` A dead letter is a **rejected** submission (400). Retrying it unchanged will not help. Typical reasons: `invalid_event`, `source_not_allowed`, `tenant_not_allowed`, `secret_shaped_field`. Payloads rejected for secret shape are withheld (`payload_withheld: true`). To replay after the sender is fixed, POST the corrected event with the same `id` and `Idempotency-Key`. The store is idempotent: an already-accepted identical payload returns 200 `duplicate` and does not create a second record. A different payload under the same id returns 409. Reconciling an **already stored** event (the WP-0005-T05 S13 case): ```bash # From a shell that already has a migration or runtime lease. python3 -m audit_core replay --event-id ``` That command must report `"duplicate": true`. A first-acceptance result is a custody defect: stop and investigate. ## Sender credential rotation ### Governed E2 evidence identities The `whitehat` test plane has a dedicated ingress policy. It admits only pods in namespace `whitehat` carrying both `whitehat.security/plane: "true"` and `whitehat.security/target: audit-core`; bearer authentication and sender scope still apply to every request. The policy is connectivity, not authorization. A dated `live-e2` engagement may request exactly two temporary identities. Each identity is bound to `source=whitehat-security`, one named synthetic tenant, `may_write=true`, `may_read=true`, and an RFC3339 `expires_at` no later than the engagement end. Audit-core evaluates expiry on every authentication, so a stale registry copy cannot extend the bearer lifetime. Provision token values only through the sanctioned sender-registry custody lane and the plane's mount-only credential projection. Never reuse the production `user-engine` identity. Revoke both identities and remove their registry entries when the plane lease ends; application expiry is a backstop, not a substitute for evidenced cleanup. OpenBao path `platform/workloads/audit-core/senders` is the authority for ExternalSecret `audit-core-senders`. The initial in-cluster registry was wrap-migrated on 2026-08-14 without printing or staging its values; do not replace that path with a founder paste. Rotation is overlap-first: 1. Add the replacement token to the sender's `tokens` list. Both work. 2. Move the sender to the new token. 3. Drop the predecessor from the list. 4. Write the updated document back (`bao kv put` from a mode-0600 file). 5. Either wait for the 1h refresh or annotate the ExternalSecret to force a sync, then restart the pod so it re-reads `AUDIT_CORE_SENDERS`. The write token is bound to `source=user-engine`. Tenant scope for that identity is **not a secret**: `deploy/senders-scope.json` (ConfigMap `audit-core-senders-scope`) overlays `tenants: ["*"]` onto the Secret at start. An ExternalSecret refresh cannot revert it to a single tenant. Tokens stay in Secret `audit-core-senders` / OpenBao KV. The operator token is a separate identity with `may_read: true`. Do not reuse one token for both. A shape (values are placeholders) is in `docs/senders.example.json`. ## Database credential rotation Runtime leases come from `database/creds/audit-core-runtime`. ESO refreshes every 15 minutes into Secret `audit-core-database`, mounted at `/etc/audit-core/db`. The process re-reads those files on every new connection. No restart is required; a restart would be a delivery gap. Rotation is logged by password fingerprint, never by value: ```text database credential rotated (fingerprint <12 hex chars>) ``` A revoked lease surfaces as `/readyz` 503 and request 503 `unavailable`. The sender contract retries 503/500 and treats 400/401/403/409 as terminal. Schema changes are a Job (`deploy/migrate-job.yaml`) using `database/creds/audit-core-migration`. The runtime Deployment sets `AUDIT_CORE_AUTO_MIGRATE=0`. After a new migration ships, delete the old Job and re-apply. ## Alert conditions There is no Prometheus scrape path on railiance01 today (`/v1/stats` is JSON behind the read privilege). Watch: | Signal | Why | | --- | --- | | `/readyz` not 200 | Custody is down; senders will retry. Do not bounce the pod. | | `counts.unavailable` or `counts.error` climbing | Database or unexpected fault. Check `platform-pg` and the mounted lease. | | `counts.unauthorized` climbing | Sender using a dropped token, or registry not refreshed. | | `counts.conflict` non-zero | Same `event_id`, different payload. Sender bug or id reuse. | | `GET /v1/secret-findings` | Secret-shaped fields by path. Fix the sending service. | | Pod not Ready after rotate | Stale or revoked lease; check ExternalSecret status, not the password. | `platform-pg` operations, including what to scrape on port 9187, live in `rapp-postgres/docs/operations.md`. ## Restore audit-core does **not** expire events (`retention_days` unset). That is a lifecycle policy statement, not a recovery guarantee. Recoverable history is the platform `data.backup` window: **30 days**, prefix `platform-pg/`, bucket owned by `resource:platform:audit-storage`. Cite, do not copy: - `resource-control/data/capability/platform-audit-storage.json` - `rapp-postgres/docs/restore.md` The platform requirement is `data.backup` profile `database` at D5 (RPO 5 min, RTO 60 min, 30-day retention, not in the railiance01 / host-europe failure domain). The live provision is **D4**, not D5: resource-control scored one backup, one full restore, one PITR, and does not yet emit `wal_archive_gap_minutes`. RPO/RTO numbers are theirs (`measured`, single observation). audit-core does not claim a better grade. Rows older than the 30-day window are not promised after a restore. Local WAL on the node is not a second copy. Physical restore is instance-wide. A consumer-only restore is a logical export of `audit_core` from a scratch physical restore, then a controlled import. Never recover in place. Procedure: `rapp-postgres/docs/restore.md`. The 2026-08-13 fail-closed sentence is superseded by RESOURCE-WP-0002-T05 (2026-08-14): production Barman to Scaleway, `audit_core.events` 30=30, full restore 65 s, PITR 65 s. Walked 2026-08-13 (pre-commissioning historical evidence): | Path | Evidence | Elapsed | Result | | --- | --- | --- | --- | | Platform full restore + PITR of live `audit_core` (21 events) | `rapp-postgres/docs/evidence/backup-restore-20260813T111651Z-remote.json` | 51.0s / 52.5s | triples + `events_append_only` | | Platform logical single-consumer export/import | same | 3.5s | 21 events | | audit-core logical dump of live `audit_core` → local PG 16 | `evidence/restore-walk-20260813T121200Z.json` | 4.1s | 21/3/1 counts, 5 triples, trigger present | Verify after any restore: ```sql SELECT count(*) FROM audit_core.events; SELECT event_id, payload_hash, accepted_at FROM audit_core.events WHERE event_id IN (/* sample from pre-restore */); SELECT tgname FROM pg_trigger WHERE tgname = 'events_append_only'; ``` ## Rollback The Deployment annotation `audit-core.railiance.io/rollback-note` is the source of truth. Migrations 0001-0005 are additive; `kubectl -n audit-core rollout undo deploy/audit-core` returns to the previous digest and an older image runs against the newer schema. A future migration that drops or narrows a column must replace that note before release. ## Deploy order 1. Image published and digest-pinned. 2. Namespace (label `railiance.io/postgres-client: platform-pg`), NetworkPolicies, ClusterSecretStores, ExternalSecrets. 3. Sender registry is Secret `audit-core-senders` (in-cluster mint). Database leases come from `openbao-audit-core-database`. 4. Job `audit-core-migrate` with `AUDIT_CORE_MIGRATE_ROLE=audit_core_migrate`. 5. Deployment. `/readyz` must report `custody_class=operational` and `recoverable_days=30`. 6. In-pod `MODE=remote DISRUPT=0` failure matrix. Evidence goes to NK-WP-0024. ## Factory sender receiver release (2026-09-11) Build from a committed source with `make image-build`. `requirements.lock` pins the five runtime packages with hashes; refresh intentionally with `uv pip compile pyproject.toml --extra serve --extra postgres --generate-hashes --output-file requirements.lock`. The image uses the factory Python 3.12 Alpine baseline, non-root UID 10001 and no runtime package installer. Scan every candidate; a clean vulnerability result is time-bound and not a claim that all future defects are absent. The release check script uses only a labelled disposable local PostgreSQL container, synthetic sender identities, and a fresh schema. It proves exact source/tenant refusal, all seven evidence read routes denied to writers, independent readback/redaction, duplicate handling, and a database write in flight during SIGTERM. Waitress has a bounded five-second worker drain. Requests without an acknowledgement still require idempotent retry; do not promise that every queued or long-running request finishes inside a shutdown window. The original c2fe39a image and this release have identical migrations 0001-0006. This receiver upgrade requires no schema migration job. Rollback to c2fe39a is structurally compatible, but loses evidence classification support: stop the two new factory producers first and retain the custody records, then re-admit their sender contract before resuming. Do not claim load-bearing admission while the legacy receiver is running. The historical SQLite importer has five PostgreSQL test failures tracked by AUDIT-IN-0004. It is outside the live receiver path; do not use it as a recovery procedure until that record closes. The release test result must state this exclusion explicitly, never describe the entire suite as green. ### Historical SQLite import `python -m audit_core migrate-store --from-sqlite --source-id ` uses PostgreSQL custody with schema migration `0007-import-receipts`. Run this with the migration owner's database authority, not a runtime writer identity. Keep `source-id` unchanged across file relocation and retries; its default is the absolute file path. Replacing a source at that path requires a new identity. No source content is copied into the receipt table: it holds hashed source and row identities, content fingerprints and previously imported finding counts. Import reads one SQLite snapshot and writes under the native append advisory lock. Historical events extend the stored link chain with original IDs, hashes, records and exact acceptance timestamps. Existing differing evidence is reported as a conflict and retained; no trigger is disabled and no existing chain link is rewritten. Retries skip imported dead letters and add only the incremental source finding count. Decreasing counts or changed historical dead letters are conflicts. Verification checks events and the resulting chain independently. The new PostgreSQL receiver and attestor follow links rather than sorting by historical timestamps. Deploy that code to all PostgreSQL append/verify workers before importing historical rows into a populated destination; older workers still use timestamp order. Source tests do not authorize a production import.