Redact secret-shaped fields by default, countable per field path
AUDIT-WP-0004-T04, closing the workplan. Decision (Bernd): default to redaction, allow rejection per sender. Losing an audit record over one field is worse than storing it masked, but a higher-assurance channel must be able to refuse rather than mask. secret_policy is set per sender identity in AUDIT_CORE_SENDERS and defaults to redact. Detection now covers the whole payload at any depth, including lists, rather than only the top level of data. Under redaction the value is masked and the key is preserved: dropping the key would hide that the sender transmitted the field at all, which is exactly what an operator needs in order to stop it. The stored record carries details.redaction with policy and affected paths, so a reader never has to infer whether what they see is what was sent. Idempotency is unaffected - the payload hash is taken over the original request body, so redaction is deterministic and a resubmission still reconciles as a duplicate. Both outcomes are counted durably by sender, source, action and field path, exposed at GET /v1/secret-findings. Per-path aggregation is the point: the actionable unit is "stop emitting data.auth.token on membership.added", not "there were 47 redactions". Counters survive restart because the fix they drive lives in another service. Contract doc updated to match. Tests 46 -> 50. WP-0004 is finished. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
0ad526c2d8
commit
576caa2665
8 changed files with 441 additions and 29 deletions
|
|
@ -36,6 +36,8 @@ Requirements:
|
|||
unless deduplication is documented.
|
||||
- **No secret dumping:** Backends must not log or persist plaintext secrets,
|
||||
tokens, keys, or passwords from `details` or future payload fields.
|
||||
- **Redaction is recorded, not silent:** where ingestion masks a field, the
|
||||
stored record says so (see [Secret-shaped fields](#secret-shaped-fields)).
|
||||
- **Visible failure:** Raise on persistence failure; do not silently drop events.
|
||||
- **Normalization only:** Backends receive `AuditEvent` instances. Source-specific
|
||||
adapters run upstream.
|
||||
|
|
@ -73,7 +75,7 @@ tenant, scope, actor, and result objects.
|
|||
| --- | --- | --- |
|
||||
| `actor` | string or null | Subject performing the action |
|
||||
| `reason` | string or null | Human-readable result explanation |
|
||||
| `details` | object | Source-specific extension map; must not contain secrets |
|
||||
| `details` | object | Source-specific extension map; must not contain secrets. May carry a `redaction` entry — see below |
|
||||
|
||||
### Example record
|
||||
|
||||
|
|
@ -225,4 +227,76 @@ Archive remains the evidence record; hot search may use shorter `retention_days`
|
|||
|
||||
- `INTENT.md` — product purpose and principles
|
||||
- `spec/ProductRequirementsDefinition.md` — full v1 envelope and API requirements
|
||||
- `registry/capabilities/capability.audit.event-retain.md` — capability registry entry
|
||||
- `registry/capabilities/capability.audit.event-retain.md` — capability registry entry
|
||||
## Secret-shaped fields
|
||||
|
||||
Ingestion detects fields whose *key name* contains `password`, `secret`,
|
||||
`token`, `credential`, or `private_key`, at any depth in the payload. Values
|
||||
are not inspected: a value-shape heuristic produces false positives on
|
||||
legitimate identifiers, and a false positive here silently mangles an audit
|
||||
record.
|
||||
|
||||
### Policy
|
||||
|
||||
The default is **redact and accept**. Rejecting an otherwise valid event
|
||||
because of one field loses the audit record entirely, which is a worse outcome
|
||||
than storing it with that field masked.
|
||||
|
||||
Policy is set **per sender identity** via `secret_policy` in
|
||||
`AUDIT_CORE_SENDERS`, so a higher-assurance channel can be switched to
|
||||
`reject` without changing the posture for every other sender:
|
||||
|
||||
```json
|
||||
[{"name": "user-engine", "tokens": ["..."], "sources": ["user-engine"],
|
||||
"secret_policy": "redact"},
|
||||
{"name": "payments-engine", "tokens": ["..."], "sources": ["payments-engine"],
|
||||
"secret_policy": "reject"}]
|
||||
```
|
||||
|
||||
| Policy | Response | Effect |
|
||||
|---|---|---|
|
||||
| `redact` (default) | `202` / `200` | Value replaced with `[redacted]`; key preserved; event stored |
|
||||
| `reject` | `400 secret_shaped_field` | Event not stored; dead-lettered with its payload withheld |
|
||||
|
||||
Keys are preserved under redaction. Dropping them would hide the fact that the
|
||||
sender transmitted the field at all — which is exactly what an operator needs
|
||||
in order to stop it.
|
||||
|
||||
### Recorded redaction
|
||||
|
||||
A redacted record carries the fact in `details.redaction`, so a reader never
|
||||
has to infer whether what they are looking at is what the sender sent:
|
||||
|
||||
```json
|
||||
"details": {
|
||||
"correlation_id": "corr-1",
|
||||
"data": {"membership_id": "m-1", "auth_token": "[redacted]"},
|
||||
"redaction": {"policy": "redact", "paths": ["data.auth_token"]}
|
||||
}
|
||||
```
|
||||
|
||||
Idempotency is unaffected: the payload hash is taken over the original request
|
||||
body, so a resubmission of the same original is still recognised as a
|
||||
duplicate and redaction is deterministic.
|
||||
|
||||
### Counting
|
||||
|
||||
Both outcomes are counted durably, aggregated by sender, source, action, and
|
||||
**field path** — because the actionable unit is "stop emitting
|
||||
`data.auth.token` on `membership.added`", not "there were 47 redactions".
|
||||
Counters survive restart, since the fix they drive lives in another service.
|
||||
|
||||
Read them at `GET /v1/secret-findings` (requires the read privilege):
|
||||
|
||||
```json
|
||||
{"secret_findings": [
|
||||
{"sender": "user-engine", "source": "user-engine",
|
||||
"action": "membership.added", "field_path": "data.auth_token",
|
||||
"outcome": "redacted", "persisted": true, "occurrences": 3,
|
||||
"first_seen": "...", "last_seen": "..."}]}
|
||||
```
|
||||
|
||||
`persisted` distinguishes a field that reached the stored record from one that
|
||||
sat elsewhere in the envelope and was dropped by normalization anyway. A
|
||||
healthy sender trends to zero occurrences; a non-empty list is a backlog item
|
||||
for the sending service, not a steady state.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue