Prepare railiance01 delivery: dynamic leases, migrate Job, operator runbook
VaultDynamicSecret pulls database/creds/* so a rotating lease is not frozen into KV. Runtime sets AUDIT_CORE_AUTO_MIGRATE=0; schema is a Job with the migration lease. Image base is digest-pinned. Namespace and NetworkPolicies are on the cluster; Deployment waits for the attended OpenBao ESO token.
This commit is contained in:
parent
bbf86b8373
commit
3a7d63e18f
18 changed files with 826 additions and 33 deletions
192
docs/operator-runbook.md
Normal file
192
docs/operator-runbook.md
Normal file
|
|
@ -0,0 +1,192 @@
|
|||
# audit-core operator runbook
|
||||
|
||||
Production receiver on railiance01. Custody is PostgreSQL (`platform-pg`,
|
||||
database `audit_core`). This runbook is the T06 handover: lookup, dead
|
||||
letters, replay, credential rotation, alerts, and restore.
|
||||
|
||||
In-cluster URL:
|
||||
|
||||
```text
|
||||
http://audit-core.audit-core.svc.cluster.local:8080
|
||||
```
|
||||
|
||||
Reachability is part of the threat model. Ingress is allowed only from the
|
||||
`user-engine` namespace (write) and from namespaces labelled
|
||||
`railiance.io/audit-core-reader=true` (operator read). From a workstation,
|
||||
`kubectl -n audit-core port-forward svc/audit-core 8080:8080` is the usual
|
||||
path and does not require that label.
|
||||
|
||||
Do not paste sender tokens, database passwords, or OpenBao tokens into Git,
|
||||
State Hub, workplans, logs, or chat. Route first:
|
||||
|
||||
```bash
|
||||
warden route find "database credential" --json
|
||||
warden route show database-dynamic-credentials --json
|
||||
```
|
||||
|
||||
## Readiness and health
|
||||
|
||||
| Check | Meaning |
|
||||
| --- | --- |
|
||||
| `GET /healthz` | Process is up. Liveness uses this. A database outage must **not** restart the pod. |
|
||||
| `GET /readyz` | Custody is reachable and `custody_class=archive`. Readiness uses this; the pod leaves the Service rather than accept events it cannot store. |
|
||||
| `GET /v1/stats` | In-process counters since start (`accepted`, `duplicate`, `conflict`, `rejected`, `unauthorized`, `forbidden`, `unavailable`, `error`). Resets on restart. Requires `may_read`. |
|
||||
|
||||
A missing `AUDIT_CORE_DATABASE_URL` / credential directory is a startup
|
||||
failure (`AUDIT_CORE_REQUIRE_CUSTODY_CLASS=archive`), not a silent downgrade
|
||||
to SQLite.
|
||||
|
||||
## Lookup
|
||||
|
||||
All read routes require a sender identity with `may_read: true`. A write
|
||||
credential must not be able to read the trail back.
|
||||
|
||||
```bash
|
||||
# One event
|
||||
curl -sS -H "Authorization: Bearer $OPERATOR_TOKEN" \
|
||||
http://127.0.0.1:8080/v1/events/<event-id>
|
||||
|
||||
# Every event sharing a correlation id
|
||||
curl -sS -H "Authorization: Bearer $OPERATOR_TOKEN" \
|
||||
'http://127.0.0.1:8080/v1/events?correlation_id=<id>&limit=100'
|
||||
```
|
||||
|
||||
Stored records keep the original `event_id`, `payload_hash`, and
|
||||
`accepted_at`. Redacted fields appear in `details.redaction.paths`; the
|
||||
payload itself is stored already masked.
|
||||
|
||||
## Dead letters and replay
|
||||
|
||||
```bash
|
||||
curl -sS -H "Authorization: Bearer $OPERATOR_TOKEN" \
|
||||
'http://127.0.0.1:8080/v1/dead-letters?limit=100'
|
||||
```
|
||||
|
||||
A dead letter is a **rejected** submission (400). Retrying it unchanged will
|
||||
not help. Typical reasons: `invalid_event`, `source_not_allowed`,
|
||||
`tenant_not_allowed`, `secret_shaped_field`. Payloads rejected for secret
|
||||
shape are withheld (`payload_withheld: true`).
|
||||
|
||||
To replay after the sender is fixed, POST the corrected event with the same
|
||||
`id` and `Idempotency-Key`. The store is idempotent: an already-accepted
|
||||
identical payload returns 200 `duplicate` and does not create a second
|
||||
record. A different payload under the same id returns 409.
|
||||
|
||||
Reconciling an **already stored** event (the WP-0005-T05 S13 case):
|
||||
|
||||
```bash
|
||||
# From a shell that already has a migration or runtime lease.
|
||||
python3 -m audit_core replay --event-id <event-id>
|
||||
```
|
||||
|
||||
That command must report `"duplicate": true`. A first-acceptance result is a
|
||||
custody defect: stop and investigate.
|
||||
|
||||
## Sender credential rotation
|
||||
|
||||
The registry is OpenBao KV `platform/workloads/audit-core/senders`, field
|
||||
`senders.json`, refreshed by ExternalSecret `audit-core-senders` every hour
|
||||
and injected as `AUDIT_CORE_SENDERS`. Rotation is overlap-first:
|
||||
|
||||
1. Add the replacement token to the sender's `tokens` list. Both work.
|
||||
2. Move the sender to the new token.
|
||||
3. Drop the predecessor from the list.
|
||||
4. Write the updated document back (`bao kv put` from a mode-0600 file).
|
||||
5. Either wait for the 1h refresh or annotate the ExternalSecret to force a
|
||||
sync, then restart the pod so it re-reads `AUDIT_CORE_SENDERS`.
|
||||
|
||||
The write token is bound to `source=user-engine` and the tenants that
|
||||
identity may claim. The operator token is a separate identity with
|
||||
`may_read: true`. Do not reuse one token for both.
|
||||
|
||||
A shape (values are placeholders) is in `docs/senders.example.json`.
|
||||
|
||||
## Database credential rotation
|
||||
|
||||
Runtime leases come from `database/creds/audit-core-runtime`. ESO refreshes
|
||||
every 15 minutes into Secret `audit-core-database`, mounted at
|
||||
`/etc/audit-core/db`. The process re-reads those files on every new
|
||||
connection. No restart is required; a restart would be a delivery gap.
|
||||
|
||||
Rotation is logged by password fingerprint, never by value:
|
||||
|
||||
```text
|
||||
database credential rotated (fingerprint <12 hex chars>)
|
||||
```
|
||||
|
||||
A revoked lease surfaces as `/readyz` 503 and request 503 `unavailable`.
|
||||
The sender contract retries 503/500 and treats 400/401/403/409 as terminal.
|
||||
|
||||
Schema changes are a Job (`deploy/migrate-job.yaml`) using
|
||||
`database/creds/audit-core-migration`. The runtime Deployment sets
|
||||
`AUDIT_CORE_AUTO_MIGRATE=0`. After a new migration ships, delete the old Job
|
||||
and re-apply.
|
||||
|
||||
## Alert conditions
|
||||
|
||||
There is no Prometheus scrape path on railiance01 today (`/v1/stats` is JSON
|
||||
behind the read privilege). Watch:
|
||||
|
||||
| Signal | Why |
|
||||
| --- | --- |
|
||||
| `/readyz` not 200 | Custody is down; senders will retry. Do not bounce the pod. |
|
||||
| `counts.unavailable` or `counts.error` climbing | Database or unexpected fault. Check `platform-pg` and the mounted lease. |
|
||||
| `counts.unauthorized` climbing | Sender using a dropped token, or registry not refreshed. |
|
||||
| `counts.conflict` non-zero | Same `event_id`, different payload. Sender bug or id reuse. |
|
||||
| `GET /v1/secret-findings` | Secret-shaped fields by path. Fix the sending service. |
|
||||
| Pod not Ready after rotate | Stale or revoked lease; check ExternalSecret status, not the password. |
|
||||
|
||||
`platform-pg` operations, including what to scrape on port 9187, live in
|
||||
`rapp-postgres/docs/operations.md`.
|
||||
|
||||
## Restore
|
||||
|
||||
audit-core declares `custody_class=archive` and `immutable=True` (the
|
||||
append-only trigger). It does **not** declare a retention window shorter
|
||||
than the platform's, because it currently declares none (`retention_days`
|
||||
unset = keep). That is only honest if the platform's backup retention is
|
||||
also unbounded or is an explicit accepted loss.
|
||||
|
||||
As of 2026-08-13, `RAPP-POSTGRES-WP-0002-T06` is `wait`. Production backup
|
||||
is fail-closed until an S3-compatible Barman target exists. The governed
|
||||
Nextcloud logical-dump lane cannot provide WAL archiving or PITR. Do not
|
||||
claim an RPO/RTO for audit-core until those drills have been recorded in
|
||||
rapp-postgres.
|
||||
|
||||
When the platform restore path is open, the audit-core walk is:
|
||||
|
||||
1. Restore `platform-pg` to a scratch cluster (`platform-pg-restore-full` or
|
||||
PITR). Never recover in place.
|
||||
2. Physical restore is instance-wide. A single-consumer restore is a logical
|
||||
dump of `audit_core` from the scratch cluster, then a controlled import.
|
||||
3. Verify: event counts, a sample of `event_id`/`payload_hash`/`accepted_at`
|
||||
triples, and that the append-only trigger is still installed
|
||||
(`SELECT tgname FROM pg_trigger WHERE tgname = 'events_append_only'`).
|
||||
4. Record elapsed time. That number is what this service can promise.
|
||||
|
||||
Until that walk exists, a workstation-side logical dump of an empty
|
||||
`audit_core` is not evidence of recovery.
|
||||
|
||||
## Rollback
|
||||
|
||||
The Deployment annotation `audit-core.railiance.io/rollback-note` is the
|
||||
source of truth. Migrations 0001-0004 are additive; `kubectl -n audit-core
|
||||
rollout undo deploy/audit-core` returns to the previous digest and an older
|
||||
image runs against the newer schema. A future migration that drops or
|
||||
narrows a column must replace that note before release.
|
||||
|
||||
## Deploy order
|
||||
|
||||
1. Image published, digest pinned in `deploy/audit-core.yaml` and
|
||||
`deploy/migrate-job.yaml`.
|
||||
2. Attended `scripts/openbao-eso-token-apply.sh`.
|
||||
3. `bao kv put platform/workloads/audit-core/senders senders.json=@file`
|
||||
from a mode-0600 file; shred the file.
|
||||
4. Apply namespace (carries `railiance.io/postgres-client: platform-pg`),
|
||||
NetworkPolicies, ClusterSecretStore, VaultDynamicSecrets, ExternalSecrets.
|
||||
5. Wait until Secrets `audit-core-database`, `audit-core-database-migrate`,
|
||||
and `audit-core-senders` exist.
|
||||
6. Apply and wait for Job `audit-core-migrate`.
|
||||
7. Apply the Deployment. Confirm `/readyz` reports `custody_class=archive`.
|
||||
8. Run `MODE=remote BASE_URL=… make failure-matrix` and hand the JSON to
|
||||
NK-WP-0024.
|
||||
Loading…
Add table
Add a link
Reference in a new issue