coulomb/audit-core
4
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
audit-core/docs/operator-runbook.md

247 lines
11 KiB
Markdown
Raw Normal View History

Prepare railiance01 delivery: dynamic leases, migrate Job, operator runbook VaultDynamicSecret pulls database/creds/* so a rotating lease is not frozen into KV. Runtime sets AUDIT_CORE_AUTO_MIGRATE=0; schema is a Job with the migration lease. Image base is digest-pinned. Namespace and NetworkPolicies are on the cluster; Deployment waits for the attended OpenBao ESO token.
2026-08-13 00:58:49 +02:00
# audit-core operator runbook
Production receiver on railiance01. Custody is PostgreSQL (`platform-pg`,
database `audit_core`). This runbook is the T06 handover: lookup, dead
letters, replay, credential rotation, alerts, and restore.
In-cluster URL:
```text
http://audit-core.audit-core.svc.cluster.local:8080
```
Reachability is part of the threat model. Ingress is allowed only from the
`user-engine` namespace (write) and from namespaces labelled
`railiance.io/audit-core-reader=true` (operator read). From a workstation,
`kubectl -n audit-core port-forward svc/audit-core 8080:8080` is the usual
path and does not require that label.
Do not paste sender tokens, database passwords, or OpenBao tokens into Git,
State Hub, workplans, logs, or chat. Route first:
```bash
warden route find "database credential" --json
warden route show database-dynamic-credentials --json
```
## Readiness and health
| Check | Meaning |
| --- | --- |
| `GET /healthz` | Process is up. Liveness uses this. A database outage must **not** restart the pod. |
Implement AUDIT-WP-0007 hash-chain integrity. Accept now extends a single-schema chain. Verify walks it; a rewritten payload_hash is a break. Tamper evidence is that detector plus an external chain-head attestation, not WORM.
2026-08-16 01:18:30 +02:00
| `GET /readyz` | Custody is reachable and `custody_class=operational`. Also reports `recoverable_days` and `tamper_evidence`. Readiness uses this; the pod leaves the Service rather than accept events it cannot store. |
Scope the read path by tenant (AUDIT-WP-0008-T04). _read gated on may_read alone and never called permits_tenant, so any reader credential could read every tenant through /v1/events, /v1/events/<id>, /v1/dead-letters and /v1/secret-findings. Deployment bounded the exposure -- the only production sender holds may_read: false -- but the boundary was not in the code, which is the difference between E2 and E1 on the tenancy posture enforcement ladder. Two rules, because the surfaces divide cleanly. Event reads are filtered to the tenants the credential may act for. Surfaces with no tenant key to filter on -- stats, integrity, dead letters, secret findings -- require full tenant scope and are refused rather than served instance-wide facts to a scoped reader. A cross-tenant fetch returns 404 rather than 403. A distinguishable forbidden would confirm that an event id exists and which tenant holds it, turning the read surface into an existence oracle. Correlation lookup is filtered rather than refused, since a correlation id legitimately spans tenants. _readable_by fails closed: a record with no tenant is readable only at full scope. Three existing tests read instance-wide surfaces with a scoped credential, which this makes a 403; bound_app now carries an unrestricted operator identity and those reads use it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-17 22:05:52 +02:00
| `GET /v1/stats` | In-process counters since start (`accepted`, `duplicate`, `conflict`, `rejected`, `unauthorized`, `forbidden`, `unavailable`, `error`). Resets on restart. Requires `may_read` and full tenant scope. |
| `GET /v1/integrity` | Hash-chain walk: `{intact, events, head, first_break}`. No payloads. Requires `may_read` and full tenant scope. A break is a custody defect, not a sender retry. |
Prepare railiance01 delivery: dynamic leases, migrate Job, operator runbook VaultDynamicSecret pulls database/creds/* so a rotating lease is not frozen into KV. Runtime sets AUDIT_CORE_AUTO_MIGRATE=0; schema is a Job with the migration lease. Image base is digest-pinned. Namespace and NetworkPolicies are on the cluster; Deployment waits for the attended OpenBao ESO token.
2026-08-13 00:58:49 +02:00
A missing `AUDIT_CORE_DATABASE_URL` / credential directory is a startup
Implement AUDIT-WP-0006 honest operational custody. Postgres now reports custody_class=operational with a cited 30-day recoverable window. Join ITC-CAP operations.audit at D4, publish the interface card, and overlay user-engine tenants [*] from Git so an ExternalSecret refresh cannot shrink it.
2026-08-16 00:24:33 +02:00
failure (`AUDIT_CORE_REQUIRE_CUSTODY_CLASS=operational`), not a silent
downgrade to SQLite. An older manifest that still requires `archive` is
accepted as an alias for one mixed rollout.
Prepare railiance01 delivery: dynamic leases, migrate Job, operator runbook VaultDynamicSecret pulls database/creds/* so a rotating lease is not frozen into KV. Runtime sets AUDIT_CORE_AUTO_MIGRATE=0; schema is a Job with the migration lease. Image base is digest-pinned. Namespace and NetworkPolicies are on the cluster; Deployment waits for the attended OpenBao ESO token.
2026-08-13 00:58:49 +02:00
Cut over AUDIT-WP-0006 image and add AUDIT-WP-0007. Live receiver now reports custody_class=operational with a cited 30-day recoverable window (sha256:05fe1c06). Next workplan is integrity verification so tamper_evidence can become honest.
2026-08-16 00:57:59 +02:00
After a node reboot, `/readyz` failing for tens of seconds is expected:
CoreDNS and `platform-pg` come up after the receiver. Liveness stays on
`/healthz` so that window does not restart the pod. Walked 2026-08-16
Implement AUDIT-WP-0007 hash-chain integrity. Accept now extends a single-schema chain. Verify walks it; a rewritten payload_hash is a break. Tamper evidence is that detector plus an external chain-head attestation, not WORM.
2026-08-16 01:18:30 +02:00
(~40s unreadiness, then Ready). That window is not a chain break.
## Integrity
Each accept extends a single hash chain for the schema (not per tenant).
`python -m audit_core verify-chain` exits 0 only if the walk is intact.
`python -m audit_core attest-chain --output docs/evidence/chain-head-<ts>.json`
writes the live head **outside** `platform-pg`. Do not put that file in
the Barman prefix; a second copy may follow the RESOURCE-WP-0002-T06
logical-offsite path. `verify-chain --against <file>` fails if the cited
head is not in the live chain.
A break is a custody defect. Do not "fix" it by accepting the same event
again. Reboot unreadiness is unrelated.
Cut over AUDIT-WP-0006 image and add AUDIT-WP-0007. Live receiver now reports custody_class=operational with a cited 30-day recoverable window (sha256:05fe1c06). Next workplan is integrity verification so tamper_evidence can become honest.
2026-08-16 00:57:59 +02:00
Prepare railiance01 delivery: dynamic leases, migrate Job, operator runbook VaultDynamicSecret pulls database/creds/* so a rotating lease is not frozen into KV. Runtime sets AUDIT_CORE_AUTO_MIGRATE=0; schema is a Job with the migration lease. Image base is digest-pinned. Namespace and NetworkPolicies are on the cluster; Deployment waits for the attended OpenBao ESO token.
2026-08-13 00:58:49 +02:00
## Lookup
All read routes require a sender identity with `may_read: true`. A write
credential must not be able to read the trail back.
Scope the read path by tenant (AUDIT-WP-0008-T04). _read gated on may_read alone and never called permits_tenant, so any reader credential could read every tenant through /v1/events, /v1/events/<id>, /v1/dead-letters and /v1/secret-findings. Deployment bounded the exposure -- the only production sender holds may_read: false -- but the boundary was not in the code, which is the difference between E2 and E1 on the tenancy posture enforcement ladder. Two rules, because the surfaces divide cleanly. Event reads are filtered to the tenants the credential may act for. Surfaces with no tenant key to filter on -- stats, integrity, dead letters, secret findings -- require full tenant scope and are refused rather than served instance-wide facts to a scoped reader. A cross-tenant fetch returns 404 rather than 403. A distinguishable forbidden would confirm that an event id exists and which tenant holds it, turning the read surface into an existence oracle. Correlation lookup is filtered rather than refused, since a correlation id legitimately spans tenants. _readable_by fails closed: a record with no tenant is readable only at full scope. Three existing tests read instance-wide surfaces with a scoped credential, which this makes a 403; bound_app now carries an unrestricted operator identity and those reads use it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-17 22:05:52 +02:00
**Read is also tenant-scoped** (AUDIT-WP-0008-T04). A reader sees only the
tenants its identity may act for, and the two rules differ by surface:
| Surface | Behaviour for a scoped reader |
|---|---|
| `GET /v1/events/<id>` | Another tenant's event returns **404**, identical to an id that does not exist. A distinguishable 403 would confirm the event exists and whose it is. |
| `GET /v1/events?correlation_id=` | Filtered to the readable slice. A correlation id legitimately spans services and tenants, so this is filtered rather than refused. |
| `/v1/stats`, `/v1/integrity`, `/v1/dead-letters`, `/v1/secret-findings` | **403 `full_tenant_scope_required`.** These carry no tenant key, so there is nothing to filter on; they are reserved for an identity with `tenants: ["*"]`. |
An operator identity holding `tenants: ["*"]` is unaffected and sees everything,
which is what `$OPERATOR_TOKEN` below assumes. A reader scoped to one tenant is
the case this protects: before T04, `may_read` was the only gate and any reader
could read every tenant.
Prepare railiance01 delivery: dynamic leases, migrate Job, operator runbook VaultDynamicSecret pulls database/creds/* so a rotating lease is not frozen into KV. Runtime sets AUDIT_CORE_AUTO_MIGRATE=0; schema is a Job with the migration lease. Image base is digest-pinned. Namespace and NetworkPolicies are on the cluster; Deployment waits for the attended OpenBao ESO token.
2026-08-13 00:58:49 +02:00
```bash
# One event
curl -sS -H "Authorization: Bearer $OPERATOR_TOKEN" \
http://127.0.0.1:8080/v1/events/<event-id>
# Every event sharing a correlation id
curl -sS -H "Authorization: Bearer $OPERATOR_TOKEN" \
'http://127.0.0.1:8080/v1/events?correlation_id=<id>&limit=100'
```
Stored records keep the original `event_id`, `payload_hash`, and
`accepted_at`. Redacted fields appear in `details.redaction.paths`; the
payload itself is stored already masked.
## Dead letters and replay
```bash
curl -sS -H "Authorization: Bearer $OPERATOR_TOKEN" \
'http://127.0.0.1:8080/v1/dead-letters?limit=100'
```
A dead letter is a **rejected** submission (400). Retrying it unchanged will
not help. Typical reasons: `invalid_event`, `source_not_allowed`,
`tenant_not_allowed`, `secret_shaped_field`. Payloads rejected for secret
shape are withheld (`payload_withheld: true`).
To replay after the sender is fixed, POST the corrected event with the same
`id` and `Idempotency-Key`. The store is idempotent: an already-accepted
identical payload returns 200 `duplicate` and does not create a second
record. A different payload under the same id returns 409.
Reconciling an **already stored** event (the WP-0005-T05 S13 case):
```bash
# From a shell that already has a migration or runtime lease.
python3 -m audit_core replay --event-id <event-id>
```
That command must report `"duplicate": true`. A first-acceptance result is a
custody defect: stop and investigate.
## Sender credential rotation
Record audit sender OpenBao authority
2026-08-14 20:12:33 +02:00
OpenBao path `platform/workloads/audit-core/senders` is the authority for
ExternalSecret `audit-core-senders`. The initial in-cluster registry was
wrap-migrated on 2026-08-14 without printing or staging its values; do not
replace that path with a founder paste. Rotation is overlap-first:
Prepare railiance01 delivery: dynamic leases, migrate Job, operator runbook VaultDynamicSecret pulls database/creds/* so a rotating lease is not frozen into KV. Runtime sets AUDIT_CORE_AUTO_MIGRATE=0; schema is a Job with the migration lease. Image base is digest-pinned. Namespace and NetworkPolicies are on the cluster; Deployment waits for the attended OpenBao ESO token.
2026-08-13 00:58:49 +02:00
1. Add the replacement token to the sender's `tokens` list. Both work.
2. Move the sender to the new token.
3. Drop the predecessor from the list.
4. Write the updated document back (`bao kv put` from a mode-0600 file).
5. Either wait for the 1h refresh or annotate the ExternalSecret to force a
sync, then restart the pod so it re-reads `AUDIT_CORE_SENDERS`.
Implement AUDIT-WP-0006 honest operational custody. Postgres now reports custody_class=operational with a cited 30-day recoverable window. Join ITC-CAP operations.audit at D4, publish the interface card, and overlay user-engine tenants [*] from Git so an ExternalSecret refresh cannot shrink it.
2026-08-16 00:24:33 +02:00
The write token is bound to `source=user-engine`. Tenant scope for that
identity is **not a secret**: `deploy/senders-scope.json` (ConfigMap
`audit-core-senders-scope`) overlays `tenants: ["*"]` onto the Secret
at start. An ExternalSecret refresh cannot revert it to a single tenant.
Tokens stay in Secret `audit-core-senders` / OpenBao KV.
The operator token is a separate identity with `may_read: true`. Do not
reuse one token for both.
Prepare railiance01 delivery: dynamic leases, migrate Job, operator runbook VaultDynamicSecret pulls database/creds/* so a rotating lease is not frozen into KV. Runtime sets AUDIT_CORE_AUTO_MIGRATE=0; schema is a Job with the migration lease. Image base is digest-pinned. Namespace and NetworkPolicies are on the cluster; Deployment waits for the attended OpenBao ESO token.
2026-08-13 00:58:49 +02:00
A shape (values are placeholders) is in `docs/senders.example.json`.
## Database credential rotation
Runtime leases come from `database/creds/audit-core-runtime`. ESO refreshes
every 15 minutes into Secret `audit-core-database`, mounted at
`/etc/audit-core/db`. The process re-reads those files on every new
connection. No restart is required; a restart would be a delivery gap.
Rotation is logged by password fingerprint, never by value:
```text
database credential rotated (fingerprint <12 hex chars>)
```
A revoked lease surfaces as `/readyz` 503 and request 503 `unavailable`.
The sender contract retries 503/500 and treats 400/401/403/409 as terminal.
Schema changes are a Job (`deploy/migrate-job.yaml`) using
`database/creds/audit-core-migration`. The runtime Deployment sets
`AUDIT_CORE_AUTO_MIGRATE=0`. After a new migration ships, delete the old Job
and re-apply.
## Alert conditions
There is no Prometheus scrape path on railiance01 today (`/v1/stats` is JSON
behind the read privilege). Watch:
| Signal | Why |
| --- | --- |
| `/readyz` not 200 | Custody is down; senders will retry. Do not bounce the pod. |
| `counts.unavailable` or `counts.error` climbing | Database or unexpected fault. Check `platform-pg` and the mounted lease. |
| `counts.unauthorized` climbing | Sender using a dropped token, or registry not refreshed. |
| `counts.conflict` non-zero | Same `event_id`, different payload. Sender bug or id reuse. |
| `GET /v1/secret-findings` | Secret-shaped fields by path. Fix the sending service. |
| Pod not Ready after rotate | Stale or revoked lease; check ExternalSecret status, not the password. |
`platform-pg` operations, including what to scrape on port 9187, live in
`rapp-postgres/docs/operations.md`.
## Restore
Implement AUDIT-WP-0006 honest operational custody. Postgres now reports custody_class=operational with a cited 30-day recoverable window. Join ITC-CAP operations.audit at D4, publish the interface card, and overlay user-engine tenants [*] from Git so an ExternalSecret refresh cannot shrink it.
2026-08-16 00:24:33 +02:00
audit-core does **not** expire events (`retention_days` unset). That is a
lifecycle policy statement, not a recovery guarantee. Recoverable history
is the platform `data.backup` window: **30 days**, prefix `platform-pg/`,
bucket owned by `resource:platform:audit-storage`. Cite, do not copy:
- `resource-control/data/capability/platform-audit-storage.json`
- `rapp-postgres/docs/restore.md`
The platform requirement is `data.backup` profile `database` at D5
(RPO 5 min, RTO 60 min, 30-day retention, not in the railiance01 /
host-europe failure domain). The live provision is **D4**, not D5:
resource-control scored one backup, one full restore, one PITR, and
does not yet emit `wal_archive_gap_minutes`. RPO/RTO numbers are
theirs (`measured`, single observation). audit-core does not claim a
better grade.
Rows older than the 30-day window are not promised after a restore.
Local WAL on the node is not a second copy.
Finish AUDIT-WP-0005: walk restore and close T06 Logical dump of live audit_core restored locally in 4.1s; counts and sample triples match the rapp-postgres drill. Retention is platform backup, not an audit-core deletion window. Workplan finished.
2026-08-13 14:14:22 +02:00
Physical restore is instance-wide. A consumer-only restore is a logical
export of `audit_core` from a scratch physical restore, then a controlled
import. Never recover in place. Procedure: `rapp-postgres/docs/restore.md`.
Implement AUDIT-WP-0006 honest operational custody. Postgres now reports custody_class=operational with a cited 30-day recoverable window. Join ITC-CAP operations.audit at D4, publish the interface card, and overlay user-engine tenants [*] from Git so an ExternalSecret refresh cannot shrink it.
2026-08-16 00:24:33 +02:00
The 2026-08-13 fail-closed sentence is superseded by RESOURCE-WP-0002-T05
(2026-08-14): production Barman to Scaleway, `audit_core.events` 30=30,
full restore 65 s, PITR 65 s.
Walked 2026-08-13 (pre-commissioning historical evidence):
Finish AUDIT-WP-0005: walk restore and close T06 Logical dump of live audit_core restored locally in 4.1s; counts and sample triples match the rapp-postgres drill. Retention is platform backup, not an audit-core deletion window. Workplan finished.
2026-08-13 14:14:22 +02:00
| Path | Evidence | Elapsed | Result |
| --- | --- | --- | --- |
| Platform full restore + PITR of live `audit_core` (21 events) | `rapp-postgres/docs/evidence/backup-restore-20260813T111651Z-remote.json` | 51.0s / 52.5s | triples + `events_append_only` |
| Platform logical single-consumer export/import | same | 3.5s | 21 events |
| audit-core logical dump of live `audit_core` → local PG 16 | `evidence/restore-walk-20260813T121200Z.json` | 4.1s | 21/3/1 counts, 5 triples, trigger present |
Verify after any restore:
```sql
SELECT count(*) FROM audit_core.events;
SELECT event_id, payload_hash, accepted_at FROM audit_core.events
WHERE event_id IN (/* sample from pre-restore */);
SELECT tgname FROM pg_trigger WHERE tgname = 'events_append_only';
```
Prepare railiance01 delivery: dynamic leases, migrate Job, operator runbook VaultDynamicSecret pulls database/creds/* so a rotating lease is not frozen into KV. Runtime sets AUDIT_CORE_AUTO_MIGRATE=0; schema is a Job with the migration lease. Image base is digest-pinned. Namespace and NetworkPolicies are on the cluster; Deployment waits for the attended OpenBao ESO token.
2026-08-13 00:58:49 +02:00
## Rollback
The Deployment annotation `audit-core.railiance.io/rollback-note` is the
Finish AUDIT-WP-0005: walk restore and close T06 Logical dump of live audit_core restored locally in 4.1s; counts and sample triples match the rapp-postgres drill. Retention is platform backup, not an audit-core deletion window. Workplan finished.
2026-08-13 14:14:22 +02:00
source of truth. Migrations 0001-0005 are additive; `kubectl -n audit-core
Prepare railiance01 delivery: dynamic leases, migrate Job, operator runbook VaultDynamicSecret pulls database/creds/* so a rotating lease is not frozen into KV. Runtime sets AUDIT_CORE_AUTO_MIGRATE=0; schema is a Job with the migration lease. Image base is digest-pinned. Namespace and NetworkPolicies are on the cluster; Deployment waits for the attended OpenBao ESO token.
2026-08-13 00:58:49 +02:00
rollout undo deploy/audit-core` returns to the previous digest and an older
image runs against the newer schema. A future migration that drops or
narrows a column must replace that note before release.
## Deploy order
Finish AUDIT-WP-0005: walk restore and close T06 Logical dump of live audit_core restored locally in 4.1s; counts and sample triples match the rapp-postgres drill. Retention is platform backup, not an audit-core deletion window. Workplan finished.
2026-08-13 14:14:22 +02:00
1. Image published and digest-pinned.
2. Namespace (label `railiance.io/postgres-client: platform-pg`),
NetworkPolicies, ClusterSecretStores, ExternalSecrets.
3. Sender registry is Secret `audit-core-senders` (in-cluster mint).
Database leases come from `openbao-audit-core-database`.
4. Job `audit-core-migrate` with `AUDIT_CORE_MIGRATE_ROLE=audit_core_migrate`.
Implement AUDIT-WP-0006 honest operational custody. Postgres now reports custody_class=operational with a cited 30-day recoverable window. Join ITC-CAP operations.audit at D4, publish the interface card, and overlay user-engine tenants [*] from Git so an ExternalSecret refresh cannot shrink it.
2026-08-16 00:24:33 +02:00
5. Deployment. `/readyz` must report `custody_class=operational` and
`recoverable_days=30`.
Finish AUDIT-WP-0005: walk restore and close T06 Logical dump of live audit_core restored locally in 4.1s; counts and sample triples match the rapp-postgres drill. Retention is platform backup, not an audit-core deletion window. Workplan finished.
2026-08-13 14:14:22 +02:00
6. In-pod `MODE=remote DISRUPT=0` failure matrix. Evidence goes to NK-WP-0024.
Reference in a new issue Copy permalink