audit-core/docs/operator-runbook.md

# audit-core operator runbook

Production receiver on railiance01. Custody is PostgreSQL (`platform-pg`,
database `audit_core`). This runbook is the T06 handover: lookup, dead
letters, replay, credential rotation, alerts, and restore.

In-cluster URL:

```text
http://audit-core.audit-core.svc.cluster.local:8080
```

Reachability is part of the threat model. Ingress is allowed only from the
`user-engine` namespace (write) and from namespaces labelled
`railiance.io/audit-core-reader=true` (operator read). From a workstation,
`kubectl -n audit-core port-forward svc/audit-core 8080:8080` is the usual
path and does not require that label.

Do not paste sender tokens, database passwords, or OpenBao tokens into Git,
State Hub, workplans, logs, or chat. Route first:

```bash
warden route find "database credential" --json
warden route show database-dynamic-credentials --json
```

## Readiness and health

| Check | Meaning |
| --- | --- |
| `GET /healthz` | Process is up. Liveness uses this. A database outage must **not** restart the pod. |
| `GET /readyz` | Custody is reachable and `custody_class=operational`. Also reports `recoverable_days` and `tamper_evidence`. Readiness uses this; the pod leaves the Service rather than accept events it cannot store. |
| `GET /v1/stats` | In-process counters since start (`accepted`, `duplicate`, `conflict`, `rejected`, `unauthorized`, `forbidden`, `unavailable`, `error`). Resets on restart. Requires `may_read` and full tenant scope. |
| `GET /v1/integrity` | Hash-chain walk: `{intact, events, head, first_break}`. No payloads. Requires `may_read` and full tenant scope. A break is a custody defect, not a sender retry. |

A missing `AUDIT_CORE_DATABASE_URL` / credential directory is a startup
failure (`AUDIT_CORE_REQUIRE_CUSTODY_CLASS=operational`), not a silent
downgrade to SQLite. An older manifest that still requires `archive` is
accepted as an alias for one mixed rollout.

After a node reboot, `/readyz` failing for tens of seconds is expected:
CoreDNS and `platform-pg` come up after the receiver. Liveness stays on
`/healthz` so that window does not restart the pod. Walked 2026-08-16
(~40s unreadiness, then Ready). That window is not a chain break.

## Integrity

Each accept extends a single hash chain for the schema (not per tenant).
`python -m audit_core verify-chain` exits 0 only if the walk is intact.
`python -m audit_core attest-chain --output docs/evidence/chain-head-<ts>.json`
writes the live head **outside** `platform-pg`. Do not put that file in
the Barman prefix; a second copy may follow the RESOURCE-WP-0002-T06
logical-offsite path. `verify-chain --against <file>` fails if the cited
head is not in the live chain.

A break is a custody defect. Do not "fix" it by accepting the same event
again. Reboot unreadiness is unrelated.

## Lookup

All read routes require a sender identity with `may_read: true`. A write
credential must not be able to read the trail back.

**Read is also tenant-scoped** (AUDIT-WP-0008-T04). A reader sees only the
tenants its identity may act for, and the two rules differ by surface:

| Surface | Behaviour for a scoped reader |
|---|---|
| `GET /v1/events/<id>` | Another tenant's event returns **404**, identical to an id that does not exist. A distinguishable 403 would confirm the event exists and whose it is. |
| `GET /v1/events?correlation_id=` | Filtered to the readable slice. A correlation id legitimately spans services and tenants, so this is filtered rather than refused. |
| `/v1/stats`, `/v1/integrity`, `/v1/dead-letters`, `/v1/secret-findings` | **403 `full_tenant_scope_required`.** These carry no tenant key, so there is nothing to filter on; they are reserved for an identity with `tenants: ["*"]`. |

An operator identity holding `tenants: ["*"]` is unaffected and sees everything,
which is what `$OPERATOR_TOKEN` below assumes. A reader scoped to one tenant is
the case this protects: before T04, `may_read` was the only gate and any reader
could read every tenant.

```bash
# One event
curl -sS -H "Authorization: Bearer $OPERATOR_TOKEN" \
  http://127.0.0.1:8080/v1/events/<event-id>

# Every event sharing a correlation id
curl -sS -H "Authorization: Bearer $OPERATOR_TOKEN" \
  'http://127.0.0.1:8080/v1/events?correlation_id=<id>&limit=100'
```

Stored records keep the original `event_id`, `payload_hash`, and
`accepted_at`. Redacted fields appear in `details.redaction.paths`; the
payload itself is stored already masked.

## Dead letters and replay

```bash
curl -sS -H "Authorization: Bearer $OPERATOR_TOKEN" \
  'http://127.0.0.1:8080/v1/dead-letters?limit=100'
```

A dead letter is a **rejected** submission (400). Retrying it unchanged will
not help. Typical reasons: `invalid_event`, `source_not_allowed`,
`tenant_not_allowed`, `secret_shaped_field`. Payloads rejected for secret
shape are withheld (`payload_withheld: true`).

To replay after the sender is fixed, POST the corrected event with the same
`id` and `Idempotency-Key`. The store is idempotent: an already-accepted
identical payload returns 200 `duplicate` and does not create a second
record. A different payload under the same id returns 409.

Reconciling an **already stored** event (the WP-0005-T05 S13 case):

```bash
# From a shell that already has a migration or runtime lease.
python3 -m audit_core replay --event-id <event-id>
```

That command must report `"duplicate": true`. A first-acceptance result is a
custody defect: stop and investigate.

## Sender credential rotation

### Governed E2 evidence identities

The `whitehat` test plane has a dedicated ingress policy. It admits only pods
in namespace `whitehat` carrying both `whitehat.security/plane: "true"` and
`whitehat.security/target: audit-core`; bearer authentication and sender scope
still apply to every request. The policy is connectivity, not authorization.

A dated `live-e2` engagement may request exactly two temporary identities. Each
identity is bound to `source=whitehat-security`, one named synthetic tenant,
`may_write=true`, `may_read=true`, and an RFC3339 `expires_at` no later than the
engagement end. Audit-core evaluates expiry on every authentication, so a stale
registry copy cannot extend the bearer lifetime. Provision token values only
through the sanctioned sender-registry custody lane and the plane's mount-only
credential projection. Never reuse the production `user-engine` identity.
Revoke both identities and remove their registry entries when the plane lease
ends; application expiry is a backstop, not a substitute for evidenced cleanup.

OpenBao path `platform/workloads/audit-core/senders` is the authority for
ExternalSecret `audit-core-senders`. The initial in-cluster registry was
wrap-migrated on 2026-08-14 without printing or staging its values; do not
replace that path with a founder paste. Rotation is overlap-first:

1. Add the replacement token to the sender's `tokens` list. Both work.
2. Move the sender to the new token.
3. Drop the predecessor from the list.
4. Write the updated document back (`bao kv put` from a mode-0600 file).
5. Either wait for the 1h refresh or annotate the ExternalSecret to force a
   sync, then restart the pod so it re-reads `AUDIT_CORE_SENDERS`.

The write token is bound to `source=user-engine`. Tenant scope for that
identity is **not a secret**: `deploy/senders-scope.json` (ConfigMap
`audit-core-senders-scope`) overlays `tenants: ["*"]` onto the Secret
at start. An ExternalSecret refresh cannot revert it to a single tenant.
Tokens stay in Secret `audit-core-senders` / OpenBao KV.

The operator token is a separate identity with `may_read: true`. Do not
reuse one token for both.

A shape (values are placeholders) is in `docs/senders.example.json`.

## Database credential rotation

Runtime leases come from `database/creds/audit-core-runtime`. ESO refreshes
every 15 minutes into Secret `audit-core-database`, mounted at
`/etc/audit-core/db`. The process re-reads those files on every new
connection. No restart is required; a restart would be a delivery gap.

Rotation is logged by password fingerprint, never by value:

```text
database credential rotated (fingerprint <12 hex chars>)
```

A revoked lease surfaces as `/readyz` 503 and request 503 `unavailable`.
The sender contract retries 503/500 and treats 400/401/403/409 as terminal.

Schema changes are a Job (`deploy/migrate-job.yaml`) using
`database/creds/audit-core-migration`. The runtime Deployment sets
`AUDIT_CORE_AUTO_MIGRATE=0`. After a new migration ships, delete the old Job
and re-apply.

## Alert conditions

There is no Prometheus scrape path on railiance01 today (`/v1/stats` is JSON
behind the read privilege). Watch:

| Signal | Why |
| --- | --- |
| `/readyz` not 200 | Custody is down; senders will retry. Do not bounce the pod. |
| `counts.unavailable` or `counts.error` climbing | Database or unexpected fault. Check `platform-pg` and the mounted lease. |
| `counts.unauthorized` climbing | Sender using a dropped token, or registry not refreshed. |
| `counts.conflict` non-zero | Same `event_id`, different payload. Sender bug or id reuse. |
| `GET /v1/secret-findings` | Secret-shaped fields by path. Fix the sending service. |
| Pod not Ready after rotate | Stale or revoked lease; check ExternalSecret status, not the password. |

`platform-pg` operations, including what to scrape on port 9187, live in
`rapp-postgres/docs/operations.md`.

## Restore

audit-core does **not** expire events (`retention_days` unset). That is a
lifecycle policy statement, not a recovery guarantee. Recoverable history
is the platform `data.backup` window: **30 days**, prefix `platform-pg/`,
bucket owned by `resource:platform:audit-storage`. Cite, do not copy:

- `resource-control/data/capability/platform-audit-storage.json`
- `rapp-postgres/docs/restore.md`

The platform requirement is `data.backup` profile `database` at D5
(RPO 5 min, RTO 60 min, 30-day retention, not in the railiance01 /
host-europe failure domain). The live provision is **D4**, not D5:
resource-control scored one backup, one full restore, one PITR, and
does not yet emit `wal_archive_gap_minutes`. RPO/RTO numbers are
theirs (`measured`, single observation). audit-core does not claim a
better grade.

Rows older than the 30-day window are not promised after a restore.
Local WAL on the node is not a second copy.

Physical restore is instance-wide. A consumer-only restore is a logical
export of `audit_core` from a scratch physical restore, then a controlled
import. Never recover in place. Procedure: `rapp-postgres/docs/restore.md`.

The 2026-08-13 fail-closed sentence is superseded by RESOURCE-WP-0002-T05
(2026-08-14): production Barman to Scaleway, `audit_core.events` 30=30,
full restore 65 s, PITR 65 s.

Walked 2026-08-13 (pre-commissioning historical evidence):

| Path | Evidence | Elapsed | Result |
| --- | --- | --- | --- |
| Platform full restore + PITR of live `audit_core` (21 events) | `rapp-postgres/docs/evidence/backup-restore-20260813T111651Z-remote.json` | 51.0s / 52.5s | triples + `events_append_only` |
| Platform logical single-consumer export/import | same | 3.5s | 21 events |
| audit-core logical dump of live `audit_core` → local PG 16 | `evidence/restore-walk-20260813T121200Z.json` | 4.1s | 21/3/1 counts, 5 triples, trigger present |

Verify after any restore:

```sql
SELECT count(*) FROM audit_core.events;
SELECT event_id, payload_hash, accepted_at FROM audit_core.events
  WHERE event_id IN (/* sample from pre-restore */);
SELECT tgname FROM pg_trigger WHERE tgname = 'events_append_only';
```

## Rollback

The Deployment annotation `audit-core.railiance.io/rollback-note` is the
source of truth. Migrations 0001-0005 are additive; `kubectl -n audit-core
rollout undo deploy/audit-core` returns to the previous digest and an older
image runs against the newer schema. A future migration that drops or
narrows a column must replace that note before release.

## Deploy order

1. Image published and digest-pinned.
2. Namespace (label `railiance.io/postgres-client: platform-pg`),
   NetworkPolicies, ClusterSecretStores, ExternalSecrets.
3. Sender registry is Secret `audit-core-senders` (in-cluster mint).
   Database leases come from `openbao-audit-core-database`.
4. Job `audit-core-migrate` with `AUDIT_CORE_MIGRATE_ROLE=audit_core_migrate`.
5. Deployment. `/readyz` must report `custody_class=operational` and
   `recoverable_days=30`.
6. In-pod `MODE=remote DISRUPT=0` failure matrix. Evidence goes to NK-WP-0024.