Redact secret-shaped fields by default, countable per field path
All checks were successful
CI Smoke / host-smoke (push) Successful in 0s
CI Smoke / container-smoke (push) Successful in 1s

AUDIT-WP-0004-T04, closing the workplan.

Decision (Bernd): default to redaction, allow rejection per sender. Losing an
audit record over one field is worse than storing it masked, but a
higher-assurance channel must be able to refuse rather than mask. secret_policy
is set per sender identity in AUDIT_CORE_SENDERS and defaults to redact.

Detection now covers the whole payload at any depth, including lists, rather
than only the top level of data. Under redaction the value is masked and the
key is preserved: dropping the key would hide that the sender transmitted the
field at all, which is exactly what an operator needs in order to stop it. The
stored record carries details.redaction with policy and affected paths, so a
reader never has to infer whether what they see is what was sent.

Idempotency is unaffected - the payload hash is taken over the original request
body, so redaction is deterministic and a resubmission still reconciles as a
duplicate.

Both outcomes are counted durably by sender, source, action and field path,
exposed at GET /v1/secret-findings. Per-path aggregation is the point: the
actionable unit is "stop emitting data.auth.token on membership.added", not
"there were 47 redactions". Counters survive restart because the fix they drive
lives in another service.

Contract doc updated to match. Tests 46 -> 50. WP-0004 is finished.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
tegwick 2026-08-10 16:02:22 +02:00
parent 0ad526c2d8
commit 576caa2665
8 changed files with 441 additions and 29 deletions

View file

@ -36,6 +36,8 @@ Requirements:
unless deduplication is documented.
- **No secret dumping:** Backends must not log or persist plaintext secrets,
tokens, keys, or passwords from `details` or future payload fields.
- **Redaction is recorded, not silent:** where ingestion masks a field, the
stored record says so (see [Secret-shaped fields](#secret-shaped-fields)).
- **Visible failure:** Raise on persistence failure; do not silently drop events.
- **Normalization only:** Backends receive `AuditEvent` instances. Source-specific
adapters run upstream.
@ -73,7 +75,7 @@ tenant, scope, actor, and result objects.
| --- | --- | --- |
| `actor` | string or null | Subject performing the action |
| `reason` | string or null | Human-readable result explanation |
| `details` | object | Source-specific extension map; must not contain secrets |
| `details` | object | Source-specific extension map; must not contain secrets. May carry a `redaction` entry — see below |
### Example record
@ -225,4 +227,76 @@ Archive remains the evidence record; hot search may use shorter `retention_days`
- `INTENT.md` — product purpose and principles
- `spec/ProductRequirementsDefinition.md` — full v1 envelope and API requirements
- `registry/capabilities/capability.audit.event-retain.md` — capability registry entry
- `registry/capabilities/capability.audit.event-retain.md` — capability registry entry
## Secret-shaped fields
Ingestion detects fields whose *key name* contains `password`, `secret`,
`token`, `credential`, or `private_key`, at any depth in the payload. Values
are not inspected: a value-shape heuristic produces false positives on
legitimate identifiers, and a false positive here silently mangles an audit
record.
### Policy
The default is **redact and accept**. Rejecting an otherwise valid event
because of one field loses the audit record entirely, which is a worse outcome
than storing it with that field masked.
Policy is set **per sender identity** via `secret_policy` in
`AUDIT_CORE_SENDERS`, so a higher-assurance channel can be switched to
`reject` without changing the posture for every other sender:
```json
[{"name": "user-engine", "tokens": ["..."], "sources": ["user-engine"],
"secret_policy": "redact"},
{"name": "payments-engine", "tokens": ["..."], "sources": ["payments-engine"],
"secret_policy": "reject"}]
```
| Policy | Response | Effect |
|---|---|---|
| `redact` (default) | `202` / `200` | Value replaced with `[redacted]`; key preserved; event stored |
| `reject` | `400 secret_shaped_field` | Event not stored; dead-lettered with its payload withheld |
Keys are preserved under redaction. Dropping them would hide the fact that the
sender transmitted the field at all — which is exactly what an operator needs
in order to stop it.
### Recorded redaction
A redacted record carries the fact in `details.redaction`, so a reader never
has to infer whether what they are looking at is what the sender sent:
```json
"details": {
"correlation_id": "corr-1",
"data": {"membership_id": "m-1", "auth_token": "[redacted]"},
"redaction": {"policy": "redact", "paths": ["data.auth_token"]}
}
```
Idempotency is unaffected: the payload hash is taken over the original request
body, so a resubmission of the same original is still recognised as a
duplicate and redaction is deterministic.
### Counting
Both outcomes are counted durably, aggregated by sender, source, action, and
**field path** — because the actionable unit is "stop emitting
`data.auth.token` on `membership.added`", not "there were 47 redactions".
Counters survive restart, since the fix they drive lives in another service.
Read them at `GET /v1/secret-findings` (requires the read privilege):
```json
{"secret_findings": [
{"sender": "user-engine", "source": "user-engine",
"action": "membership.added", "field_path": "data.auth_token",
"outcome": "redacted", "persisted": true, "occurrences": 3,
"first_seen": "...", "last_seen": "..."}]}
```
`persisted` distinguishes a field that reached the stored record from one that
sat elsewhere in the envelope and was dropped by normalization anyway. A
healthy sender trends to zero occurrences; a non-empty list is a backlog item
for the sending service, not a steady state.