audit-core/docs/audit-backend-contract.md

# Audit Backend Contract

Audit Core separates **event producers** (integrations, CLI, future HTTP API) from
**audit backends** (sinks that persist normalized events). This document defines
the replaceable backend interface, the current event schema, retention guarantees,
and how to migrate from the development mock file backend to durable custody.

## AuditBackend protocol

Every backend implements `audit_core.interface.AuditBackend`:

```python
class AuditBackend(Protocol):
    @property
    def retention_policy(self) -> RetentionPolicy: ...

    def emit(self, event: AuditEvent) -> str: ...
```

### `emit(event) -> str`

Persist one normalized `AuditEvent` and return a backend-specific reference.
Callers use the reference for debugging and correlation; it is not a stable
cross-backend identifier.

| Backend kind | Typical return value |
| --- | --- |
| Mock file | Absolute path to the hourly JSONL file |
| Archive (planned) | Batch URI or object key |
| Hot search (planned) | Stream offset or index document id |

Requirements:

- **Idempotent references:** Re-emitting the same logical event (same
  `event_id`) must not corrupt prior records. Backends may append duplicates
  unless deduplication is documented.
- **No secret dumping:** Backends must not log or persist plaintext secrets,
  tokens, keys, or passwords from `details` or future payload fields.
- **Redaction is recorded, not silent:** where ingestion masks a field, the
  stored record says so (see [Secret-shaped fields](#secret-shaped-fields)).
- **Visible failure:** Raise on persistence failure; do not silently drop events.
- **Normalization only:** Backends receive `AuditEvent` instances. Source-specific
  adapters run upstream.

### `retention_policy`

Each backend exposes a frozen `RetentionPolicy` describing what it guarantees.
Integrators and readiness checks use this to decide whether a sink satisfies a
scope's policy. See [Retention policy](#retention-policy).

## Event schema (`audit-core.event.v1alpha1`)

The first implementation uses a flat JSON record produced by `AuditEvent.as_record()`.
This is a deliberate simplification for local wiring; the long-term envelope in
`spec/ProductRequirementsDefinition.md` (`audit-core.event.v1`) nests source,
tenant, scope, actor, and result objects.

### Required fields

| Field | Type | Description |
| --- | --- | --- |
| `schema_version` | string | Must be `audit-core.event.v1alpha1` for this contract |
| `event_id` | string | UUID or source-stable identifier |
| `observed_at` | string | UTC ISO-8601 timestamp (no subsecond precision) |
| `tenant` | string | Tenant id; default `platform` for control-plane events |
| `scope` | string | Scope id; default `platform-control-plane` |
| `source` | string | Emitter id (e.g. `openbao`, `audit-core`) |
| `action` | string | Namespaced action (e.g. `openbao.audit.list`) |
| `resource` | string | Affected resource path or id |
| `outcome` | string | Result label (e.g. `success`, `failure`, `denied`) |

### Optional fields

| Field | Type | Description |
| --- | --- | --- |
| `actor` | string or null | Subject performing the action |
| `reason` | string or null | Human-readable result explanation |
| `details` | object | Source-specific extension map; must not contain secrets. May carry a `redaction` entry — see below |

### Example record

```json
{
  "action": "openbao.authenticated_readiness_proof",
  "actor": null,
  "details": {"backend": "mock-file", "file_audit_visible": true},
  "event_id": "6f3e2b1a-4c5d-6e7f-8a9b-0c1d2e3f4a5b",
  "observed_at": "2026-06-01T20:30:00+00:00",
  "outcome": "success",
  "reason": null,
  "resource": "openbao/openbao-0",
  "schema_version": "audit-core.event.v1alpha1",
  "scope": "platform-control-plane",
  "source": "openbao",
  "tenant": "platform"
}
```

### Validation

Use `audit_core.interface.validate_event()` before emit. It checks required
fields, `schema_version`, and rejects empty strings on required identifiers.

### Evolution to `audit-core.event.v1`

Future backends will accept the nested v1 envelope at the HTTP ingestion layer.
Module-level backends may continue using `AuditEvent` with adapter translation.
Compatibility rules (not yet implemented):

- v1alpha1 records remain readable in archive export.
- New required v1 fields get sensible defaults during adapter migration.
- `schema_version` gates parser selection.

## Retention policy

`RetentionPolicy` is declarative metadata; enforcement is backend-specific.

| Field | Meaning |
| --- | --- |
| `custody_class` | `development`, `operational`, `archive`, or `hot_search` |
| `retention_days` | Maximum age before eligible deletion; `None` is a lifecycle statement (no expiry), not a recovery guarantee |
| `immutable` | Whether stored records are protected from in-place alteration |
| `tamper_evidence` | Whether manifests, hash chains, or signatures exist |
| `durable` | Whether survival is expected across process restarts and host reboots |
| `recoverable_days` | Cited platform backup window; `None` if not declared |
| `recoverable_source` | Where the recoverable window is cited from |
| `recoverable_basis` | ITC-GOV EvidenceBasis of that citation (`measured`, `quoted`, …) |

### Custody classes

| Class | Purpose | Guarantees |
| --- | --- | --- |
| `development` | Local integration and bootstrap wiring | Ephemeral local files; best-effort cleanup; **not audit custody** |
| `operational` | Durable production custody | Append-only Postgres; recoverable through the platform `data.backup` provision; not ITC-CAP `data.archive` |
| `archive` | Long-term evidence (future sink) | Reserved for a backend that can satisfy `data.archive` hooks (retention policy, integrity verification, retrieval test) |
| `hot_search` | Operational investigation (planned) | Shorter retention; searchable; not the evidence record |

`AUDIT_CORE_REQUIRE_CUSTODY_CLASS=operational` is the production fail-closed gate. `archive` is accepted as an alias of `operational` for one mixed rollout so an old manifest cannot refuse a new image. A `development` backend satisfies neither.

### Mock file backend policy

`MockFileAuditBackend.retention_policy`:

- `custody_class`: `development`
- `retention_days`: 7 (override via `AUDIT_CORE_MOCK_RETENTION_DAYS` or constructor)
- `immutable`: false
- `tamper_evidence`: false
- `durable`: false

Enforcement:

- Events append to hourly JSONL files under `AUDIT_CORE_MOCK_DIR` (default
  `/tmp/audit-core`).
- `cleanup_old_files()` deletes `audit-*.jsonl` whose mtime is older than
  `retention_days`. Cleanup runs on each `emit` and via `python3 -m audit_core cleanup`.
- Negative `retention_days` disables automatic deletion.

**Not guaranteed:** crash-safe writes, replication, encryption, tenant isolation,
integrity proofs, or survival of `/tmp` across reboots.

### Production operational policy (Postgres, live)

`PostgresAuditBackend.retention_policy`:

- `custody_class`: `operational`
- `retention_days`: unset in production (the service does not expire rows)
- `immutable`: true (trigger `events_append_only`; not a claim against the database owner)
- `tamper_evidence`: true (hash chain + verify + external head attestation; not WORM)
- `durable`: true
- `recoverable_days`: 30, cited from the platform `data.backup` provision
- `recoverable_source`: `resource-control/data/capability/platform-audit-storage.json#provisions[capability=data.backup]`
- `recoverable_basis`: `measured`

`None` retention is a lifecycle statement, not unbounded archive. Rows older than the recoverable window are not promised after a restore. A future `archive` backend that satisfies ITC-CAP `data.archive` is not implemented.

## Migration path: mock file → durable backend

### Phase 0 — today (mock file)

Use `MockFileAuditBackend` or the CLI:

```bash
python3 -m audit_core emit \
  --source my-service \
  --action my_service.audit.smoke \
  --resource my-service/instance-0 \
  --outcome success
```

Integrations import the protocol, not the mock:

```python
from audit_core import AuditBackend, AuditEvent, MockFileAuditBackend

backend: AuditBackend = MockFileAuditBackend()
backend.emit(AuditEvent(source="...", action="...", resource="...", outcome="success"))
```

### Phase 1 — dual-write readiness (planned)

1. Register a durable archive backend implementing `AuditBackend`.
2. Configure routing: development scopes may keep mock; production scopes require
   `custody_class=operational` (the live Postgres backend).
3. Readiness checks compare `backend.retention_policy` against scope policy and
   fail closed when custody is insufficient.

### Phase 2 — operational primary (live)

1. HTTP ingestion writes through `PostgresAuditBackend` (`custody_class=operational`).
2. Retain mock only for local `make mock-audit-smoke` and unit tests.
3. A future `data.archive` sink is a separate backend, not a rename of Postgres.

### Phase 3 — hot search adjunct (planned)

Add a second `AuditBackend` with `custody_class=hot_search` for investigation.
Archive remains the evidence record; hot search may use shorter `retention_days`.

### Code migration checklist

| Step | Action |
| --- | --- |
| 1 | Depend on `AuditBackend`, not `MockFileAuditBackend`, in integration code |
| 2 | Build `AuditEvent` with explicit `tenant`, `scope`, and `source` |
| 3 | Call `validate_event()` before emit |
| 4 | Inspect `retention_policy` in readiness gates |
| 5 | Replace mock construction with injected backend from configuration |
| 6 | Verify export/manifest workflow before decommissioning mock files |

## Reference implementations

| Backend | Module | Custody class |
| --- | --- | --- |
| Mock file JSONL | `audit_core.mock_file_backend.MockFileAuditBackend` | `development` |
| SQLite (local / test) | `audit_core.sqlite_backend.SQLiteAuditBackend` | `development` |
| PostgreSQL (production) | `audit_core.postgres_backend.PostgresAuditBackend` | `operational` |
| Archive (planned `data.archive` sink) | TBD | `archive` |
| Hot search (planned) | TBD | `hot_search` |

## Related documents

- `INTENT.md` — product purpose and principles
- `spec/ProductRequirementsDefinition.md` — full v1 envelope and API requirements
- `docs/integrity.md` — hash chain, proof bound, verify/attest
- `registry/capabilities/capability.audit.event-retain.md` — capability registry entry
## Secret-shaped fields

Ingestion detects fields whose *key name* contains `password`, `secret`,
`token`, `credential`, or `private_key`, at any depth in the payload. Values
are not inspected: a value-shape heuristic produces false positives on
legitimate identifiers, and a false positive here silently mangles an audit
record.

### Policy

The default is **redact and accept**. Rejecting an otherwise valid event
because of one field loses the audit record entirely, which is a worse outcome
than storing it with that field masked.

Policy is set **per sender identity** via `secret_policy` in
`AUDIT_CORE_SENDERS`, so a higher-assurance channel can be switched to
`reject` without changing the posture for every other sender:

```json
[{"name": "user-engine", "tokens": ["..."], "sources": ["user-engine"],
  "secret_policy": "redact"},
 {"name": "payments-engine", "tokens": ["..."], "sources": ["payments-engine"],
  "secret_policy": "reject"}]
```

| Policy | Response | Effect |
|---|---|---|
| `redact` (default) | `202` / `200` | Value replaced with `[redacted]`; key preserved; event stored |
| `reject` | `400 secret_shaped_field` | Event not stored; dead-lettered with its payload withheld |

Keys are preserved under redaction. Dropping them would hide the fact that the
sender transmitted the field at all — which is exactly what an operator needs
in order to stop it.

### Recorded redaction

A redacted record carries the fact in `details.redaction`, so a reader never
has to infer whether what they are looking at is what the sender sent:

```json
"details": {
  "correlation_id": "corr-1",
  "data": {"membership_id": "m-1", "auth_token": "[redacted]"},
  "redaction": {"policy": "redact", "paths": ["data.auth_token"]}
}
```

Idempotency is unaffected: the payload hash is taken over the original request
body, so a resubmission of the same original is still recognised as a
duplicate and redaction is deterministic.

### Counting

Both outcomes are counted durably, aggregated by sender, source, action, and
**field path** — because the actionable unit is "stop emitting
`data.auth.token` on `membership.added`", not "there were 47 redactions".
Counters survive restart, since the fix they drive lives in another service.

Read them at `GET /v1/secret-findings` (requires the read privilege):

```json
{"secret_findings": [
  {"sender": "user-engine", "source": "user-engine",
   "action": "membership.added", "field_path": "data.auth_token",
   "outcome": "redacted", "persisted": true, "occurrences": 3,
   "first_seen": "...", "last_seen": "..."}]}
```

`occurrences` counts **transmissions, not stored events**: a retry resubmitting
the same secret-shaped field increments it again, even though the event
reconciles as a duplicate and produces no second custody record. That is
deliberate — the number measures how often the sender emitted the field, which
is the behaviour being optimized away.

`persisted` distinguishes a field that reached the stored record from one that
sat elsewhere in the envelope and was dropped by normalization anyway. A
healthy sender trends to zero occurrences; a non-empty list is a backlog item
for the sending service, not a steady state.