Implement AUDIT-WP-0006 honest operational custody.
Postgres now reports custody_class=operational with a cited 30-day recoverable window. Join ITC-CAP operations.audit at D4, publish the interface card, and overlay user-engine tenants [*] from Git so an ExternalSecret refresh cannot shrink it.
This commit is contained in:
parent
0a3d05ff1c
commit
ded432a63f
25 changed files with 832 additions and 94 deletions
|
|
@ -117,20 +117,26 @@ Compatibility rules (not yet implemented):
|
|||
|
||||
| Field | Meaning |
|
||||
| --- | --- |
|
||||
| `custody_class` | `development`, `archive`, or `hot_search` |
|
||||
| `retention_days` | Maximum age before eligible deletion; `None` means indefinite |
|
||||
| `custody_class` | `development`, `operational`, `archive`, or `hot_search` |
|
||||
| `retention_days` | Maximum age before eligible deletion; `None` is a lifecycle statement (no expiry), not a recovery guarantee |
|
||||
| `immutable` | Whether stored records are protected from in-place alteration |
|
||||
| `tamper_evidence` | Whether manifests, hash chains, or signatures exist |
|
||||
| `durable` | Whether survival is expected across process restarts and host reboots |
|
||||
| `recoverable_days` | Cited platform backup window; `None` if not declared |
|
||||
| `recoverable_source` | Where the recoverable window is cited from |
|
||||
| `recoverable_basis` | ITC-GOV EvidenceBasis of that citation (`measured`, `quoted`, …) |
|
||||
|
||||
### Custody classes
|
||||
|
||||
| Class | Purpose | Guarantees |
|
||||
| --- | --- | --- |
|
||||
| `development` | Local integration and bootstrap wiring | Ephemeral local files; best-effort cleanup; **not audit custody** |
|
||||
| `archive` | Long-term evidence (planned) | Durable object storage, batch manifests, explicit retention |
|
||||
| `operational` | Durable production custody | Append-only Postgres; recoverable through the platform `data.backup` provision; not ITC-CAP `data.archive` |
|
||||
| `archive` | Long-term evidence (future sink) | Reserved for a backend that can satisfy `data.archive` hooks (retention policy, integrity verification, retrieval test) |
|
||||
| `hot_search` | Operational investigation (planned) | Shorter retention; searchable; not the evidence record |
|
||||
|
||||
`AUDIT_CORE_REQUIRE_CUSTODY_CLASS=operational` is the production fail-closed gate. `archive` is accepted as an alias of `operational` for one mixed rollout so an old manifest cannot refuse a new image. A `development` backend satisfies neither.
|
||||
|
||||
### Mock file backend policy
|
||||
|
||||
`MockFileAuditBackend.retention_policy`:
|
||||
|
|
@ -152,15 +158,20 @@ Enforcement:
|
|||
**Not guaranteed:** crash-safe writes, replication, encryption, tenant isolation,
|
||||
integrity proofs, or survival of `/tmp` across reboots.
|
||||
|
||||
### Production archive policy (planned)
|
||||
### Production operational policy (Postgres, live)
|
||||
|
||||
Target guarantees for the first durable backend:
|
||||
`PostgresAuditBackend.retention_policy`:
|
||||
|
||||
- `custody_class`: `archive`
|
||||
- `retention_days`: scope policy (often years, sometimes indefinite)
|
||||
- `immutable`: true (WORM / object lock where available)
|
||||
- `tamper_evidence`: true (batch manifests with content hashes)
|
||||
- `custody_class`: `operational`
|
||||
- `retention_days`: unset in production (the service does not expire rows)
|
||||
- `immutable`: true (trigger `events_append_only`; not a claim against the database owner)
|
||||
- `tamper_evidence`: false (a superuser can drop the trigger; no hash-chain)
|
||||
- `durable`: true
|
||||
- `recoverable_days`: 30, cited from the platform `data.backup` provision
|
||||
- `recoverable_source`: `resource-control/data/capability/platform-audit-storage.json#provisions[capability=data.backup]`
|
||||
- `recoverable_basis`: `measured`
|
||||
|
||||
`None` retention is a lifecycle statement, not unbounded archive. Rows older than the recoverable window are not promised after a restore. A future `archive` backend that satisfies ITC-CAP `data.archive` is not implemented.
|
||||
|
||||
## Migration path: mock file → durable backend
|
||||
|
||||
|
|
@ -189,15 +200,15 @@ backend.emit(AuditEvent(source="...", action="...", resource="...", outcome="suc
|
|||
|
||||
1. Register a durable archive backend implementing `AuditBackend`.
|
||||
2. Configure routing: development scopes may keep mock; production scopes require
|
||||
`custody_class=archive`.
|
||||
`custody_class=operational` (the live Postgres backend).
|
||||
3. Readiness checks compare `backend.retention_policy` against scope policy and
|
||||
fail closed when custody is insufficient.
|
||||
|
||||
### Phase 2 — archive primary (planned)
|
||||
### Phase 2 — operational primary (live)
|
||||
|
||||
1. Point `emit` calls (or HTTP ingestion) at the archive backend.
|
||||
1. HTTP ingestion writes through `PostgresAuditBackend` (`custody_class=operational`).
|
||||
2. Retain mock only for local `make mock-audit-smoke` and unit tests.
|
||||
3. Export historical mock JSONL into archive batches with manifest generation.
|
||||
3. A future `data.archive` sink is a separate backend, not a rename of Postgres.
|
||||
|
||||
### Phase 3 — hot search adjunct (planned)
|
||||
|
||||
|
|
@ -220,7 +231,9 @@ Archive remains the evidence record; hot search may use shorter `retention_days`
|
|||
| Backend | Module | Custody class |
|
||||
| --- | --- | --- |
|
||||
| Mock file JSONL | `audit_core.mock_file_backend.MockFileAuditBackend` | `development` |
|
||||
| Archive (planned) | TBD | `archive` |
|
||||
| SQLite (local / test) | `audit_core.sqlite_backend.SQLiteAuditBackend` | `development` |
|
||||
| PostgreSQL (production) | `audit_core.postgres_backend.PostgresAuditBackend` | `operational` |
|
||||
| Archive (planned `data.archive` sink) | TBD | `archive` |
|
||||
| Hot search (planned) | TBD | `hot_search` |
|
||||
|
||||
## Related documents
|
||||
|
|
|
|||
|
|
@ -27,10 +27,11 @@
|
|||
"platform_logical_seconds": 3.468,
|
||||
"retention": {
|
||||
"audit_core_retention_days": null,
|
||||
"audit_core_meaning": "does not expire or delete events",
|
||||
"audit_core_meaning": "does not expire or delete events; lifecycle statement, not a recovery guarantee",
|
||||
"platform_planned_window_days": 30,
|
||||
"production_barman": "fail-closed; no off-host copy yet",
|
||||
"recovery_bound": "platform backup retention, not audit-core deletion"
|
||||
"production_barman": "superseded 2026-08-14 by RESOURCE-WP-0002-T05: live Scaleway Barman, 30-day window, audit_core.events 30=30, full 65s / PITR 65s",
|
||||
"recovery_bound": "platform data.backup window (resource:platform:audit-storage), not audit-core deletion",
|
||||
"provision_honesty": "requirement D5, provision D4; cited, not re-scored here"
|
||||
},
|
||||
"verified": true
|
||||
}
|
||||
|
|
|
|||
99
docs/interface-card.yaml
Normal file
99
docs/interface-card.yaml
Normal file
|
|
@ -0,0 +1,99 @@
|
|||
schema: info-tech-canon.interface-card.v1
|
||||
id: audit-core/interface-card
|
||||
title: audit-core Canon Interface Card
|
||||
consumer: audit-core
|
||||
consumer_profile:
|
||||
repo: audit-core
|
||||
domain: infotech
|
||||
owner: audit-core
|
||||
intent: >
|
||||
Provide durable, tenant-aware operational custody for audit events so
|
||||
senders can treat a 202 as evidence-in-store, not a log-forwarding hint.
|
||||
scope:
|
||||
- audit event ingestion
|
||||
- append-only operational custody
|
||||
- sender binding
|
||||
- recovery bound to platform backup
|
||||
purposes:
|
||||
- id: audit-core/operational-custody
|
||||
use_case: Accept normalized events from registered senders and retain them in an append-only store.
|
||||
consumer_need: A joinable operations.audit provision with an honest recovery claim.
|
||||
demand_signals:
|
||||
- user-engine delivers platform and tenant events over POST /v1/events
|
||||
- neighbours already require data.backup in ITC-CAP terms
|
||||
canon_surfaces:
|
||||
- model/capability
|
||||
- model/governance
|
||||
- model/data
|
||||
- model/security
|
||||
surfaces:
|
||||
implemented_profiles: []
|
||||
consumed_artifacts:
|
||||
- model/capability
|
||||
- model/governance
|
||||
- model/data
|
||||
- model/security
|
||||
owned_concepts: []
|
||||
produced_concepts:
|
||||
- Evidence
|
||||
- AuditRecord
|
||||
consumed_concepts:
|
||||
- Evidence
|
||||
- EvidenceBasis
|
||||
- CapabilityProvision
|
||||
- RetentionRuleReference
|
||||
mappings:
|
||||
- from: stored event
|
||||
to: Evidence / AuditRecord
|
||||
note: This service stores evidence. It is not an independent Audit-as-assessment.
|
||||
- from: capability.audit.event-retain
|
||||
to: operations.audit
|
||||
note: data/capability/audit-core-operational.json
|
||||
validation_expectations:
|
||||
commands:
|
||||
- PYTHONPATH=src python3 -m info_tech_canon capability-review /home/worsch/audit-core/data/capability/audit-core-operational.json
|
||||
evidence_required:
|
||||
- data/capability/audit-core-operational.json
|
||||
- docs/operator-runbook.md Restore section
|
||||
- docs/evidence/restore-walk-20260813T121200Z.json
|
||||
known_gaps:
|
||||
- id: data.archive-unprovided
|
||||
owner: audit-core
|
||||
disposition: unmet requirement recorded on the ITC-CAP case; do not build the sink in AUDIT-WP-0006
|
||||
- id: tamper-evidence-false
|
||||
owner: audit-core
|
||||
disposition: integrity_verification hook is unknown; trigger is not a proof
|
||||
- id: no-hash-chain
|
||||
owner: audit-core
|
||||
disposition: INTENT residual, not this workplan
|
||||
- id: single-sender
|
||||
owner: audit-core
|
||||
disposition: NetworkPolicy admits user-engine only; other sources remain adapters
|
||||
- id: no-rapp-yaml
|
||||
owner: railiance-master
|
||||
disposition: schema requires rapp-*; extraction is a first-wave family decision
|
||||
- id: historic-archive-overclaim
|
||||
owner: audit-core
|
||||
disposition: closed by AUDIT-WP-0006-T01; /readyz reports operational
|
||||
purpose_fit:
|
||||
state: partial
|
||||
matched_capabilities:
|
||||
- operations.audit
|
||||
scope_pressure: >
|
||||
INTENT describes a control plane, object archive, hot search, and export.
|
||||
The live service is a single-sender operational custody receiver.
|
||||
recommended_disposition: keep the operational provision honest; do not inflate to data.archive
|
||||
consumer_needs:
|
||||
current:
|
||||
- Honest custody_class and recoverable window on /readyz
|
||||
- Joinable operations.audit provision
|
||||
requested_extensions:
|
||||
- data.archive sink if events must survive past the 30-day backup window
|
||||
feedback: []
|
||||
known_deviations:
|
||||
- no data.archive sink
|
||||
- tamper_evidence=False
|
||||
- no hash-chain
|
||||
- single sender user-engine
|
||||
- no rapp.yaml (not a rapp-* repo)
|
||||
- /readyz historically overclaimed archive (closed by T01)
|
||||
|
|
@ -29,12 +29,13 @@ warden route show database-dynamic-credentials --json
|
|||
| Check | Meaning |
|
||||
| --- | --- |
|
||||
| `GET /healthz` | Process is up. Liveness uses this. A database outage must **not** restart the pod. |
|
||||
| `GET /readyz` | Custody is reachable and `custody_class=archive`. Readiness uses this; the pod leaves the Service rather than accept events it cannot store. |
|
||||
| `GET /readyz` | Custody is reachable and `custody_class=operational`. Also reports `recoverable_days` (cited platform backup window). Readiness uses this; the pod leaves the Service rather than accept events it cannot store. |
|
||||
| `GET /v1/stats` | In-process counters since start (`accepted`, `duplicate`, `conflict`, `rejected`, `unauthorized`, `forbidden`, `unavailable`, `error`). Resets on restart. Requires `may_read`. |
|
||||
|
||||
A missing `AUDIT_CORE_DATABASE_URL` / credential directory is a startup
|
||||
failure (`AUDIT_CORE_REQUIRE_CUSTODY_CLASS=archive`), not a silent downgrade
|
||||
to SQLite.
|
||||
failure (`AUDIT_CORE_REQUIRE_CUSTODY_CLASS=operational`), not a silent
|
||||
downgrade to SQLite. An older manifest that still requires `archive` is
|
||||
accepted as an alias for one mixed rollout.
|
||||
|
||||
## Lookup
|
||||
|
||||
|
|
@ -96,9 +97,14 @@ replace that path with a founder paste. Rotation is overlap-first:
|
|||
5. Either wait for the 1h refresh or annotate the ExternalSecret to force a
|
||||
sync, then restart the pod so it re-reads `AUDIT_CORE_SENDERS`.
|
||||
|
||||
The write token is bound to `source=user-engine` and the tenants that
|
||||
identity may claim. The operator token is a separate identity with
|
||||
`may_read: true`. Do not reuse one token for both.
|
||||
The write token is bound to `source=user-engine`. Tenant scope for that
|
||||
identity is **not a secret**: `deploy/senders-scope.json` (ConfigMap
|
||||
`audit-core-senders-scope`) overlays `tenants: ["*"]` onto the Secret
|
||||
at start. An ExternalSecret refresh cannot revert it to a single tenant.
|
||||
Tokens stay in Secret `audit-core-senders` / OpenBao KV.
|
||||
|
||||
The operator token is a separate identity with `may_read: true`. Do not
|
||||
reuse one token for both.
|
||||
|
||||
A shape (values are placeholders) is in `docs/senders.example.json`.
|
||||
|
||||
|
|
@ -142,17 +148,34 @@ behind the read privilege). Watch:
|
|||
|
||||
## Restore
|
||||
|
||||
audit-core does **not** expire events (`retention_days` unset). Recovery is
|
||||
bounded by what rapp-postgres can restore, not by an audit-core deletion
|
||||
window. The platform's planned Barman window is 30 days. Production Barman
|
||||
is still fail-closed (no governed off-host target). Do not promise an RPO
|
||||
until that target exists. Local WAL on the node is not an off-host copy.
|
||||
audit-core does **not** expire events (`retention_days` unset). That is a
|
||||
lifecycle policy statement, not a recovery guarantee. Recoverable history
|
||||
is the platform `data.backup` window: **30 days**, prefix `platform-pg/`,
|
||||
bucket owned by `resource:platform:audit-storage`. Cite, do not copy:
|
||||
|
||||
- `resource-control/data/capability/platform-audit-storage.json`
|
||||
- `rapp-postgres/docs/restore.md`
|
||||
|
||||
The platform requirement is `data.backup` profile `database` at D5
|
||||
(RPO 5 min, RTO 60 min, 30-day retention, not in the railiance01 /
|
||||
host-europe failure domain). The live provision is **D4**, not D5:
|
||||
resource-control scored one backup, one full restore, one PITR, and
|
||||
does not yet emit `wal_archive_gap_minutes`. RPO/RTO numbers are
|
||||
theirs (`measured`, single observation). audit-core does not claim a
|
||||
better grade.
|
||||
|
||||
Rows older than the 30-day window are not promised after a restore.
|
||||
Local WAL on the node is not a second copy.
|
||||
|
||||
Physical restore is instance-wide. A consumer-only restore is a logical
|
||||
export of `audit_core` from a scratch physical restore, then a controlled
|
||||
import. Never recover in place. Procedure: `rapp-postgres/docs/restore.md`.
|
||||
|
||||
Walked 2026-08-13:
|
||||
The 2026-08-13 fail-closed sentence is superseded by RESOURCE-WP-0002-T05
|
||||
(2026-08-14): production Barman to Scaleway, `audit_core.events` 30=30,
|
||||
full restore 65 s, PITR 65 s.
|
||||
|
||||
Walked 2026-08-13 (pre-commissioning historical evidence):
|
||||
|
||||
| Path | Evidence | Elapsed | Result |
|
||||
| --- | --- | --- | --- |
|
||||
|
|
@ -185,5 +208,6 @@ narrows a column must replace that note before release.
|
|||
3. Sender registry is Secret `audit-core-senders` (in-cluster mint).
|
||||
Database leases come from `openbao-audit-core-database`.
|
||||
4. Job `audit-core-migrate` with `AUDIT_CORE_MIGRATE_ROLE=audit_core_migrate`.
|
||||
5. Deployment. `/readyz` must report `custody_class=archive`.
|
||||
5. Deployment. `/readyz` must report `custody_class=operational` and
|
||||
`recoverable_days=30`.
|
||||
6. In-pod `MODE=remote DISRUPT=0` failure matrix. Evidence goes to NK-WP-0024.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue