Define retained PDP audit admission and complete-ledger reconciliation

Assistant: codex
Assistant-Model: gpt-6-astra
Assistant-Session: 01a140c4-354a-7823-8f0d-8f717397f786

docs/flex-auth-secrets-engine-admission.md

@@ -0,0 +1,183 @@
+# First protected workflow: Audit Core admission packet
+
+Prepared 2026-10-16 for TSA-09, AUDIT-WP-0013-T02, FLEX-WP-0031-T02/T06/T08,
+SECRETS-WP-0013 and RPF-WP-0035-T12. This packet grants no sender, observer,
+credential, deployment or protected effect authority. T02 remains waiting.
+
+## Exact source and receiver compatibility
+
+Reviewed Access source `0a05383195213e7eaacd08388ce1d7892ce756df`, particularly
+`internal/emission/outbox.go` and `delivery.go`; Audit base
+`2e97f37c01b583ac779310a2aa0d938bb9816604`. The protected consumer pin is
+`flex-auth.secrets-engine`, distinct from generic `flex-auth` and other pins.
+Access emits the existing eight-field contract in `docs/event-envelope.md`.
+No translating adapter or second event schema is needed. Preserve event id,
+Idempotency-Key, source, tenant, correlation, original time and data; the archive
+does not authenticate a decision signature or establish approval validity.
+
+Inventory: `flex-auth.decision.allow`, `.deny`, `.redact`, `.not_applicable`,
+`.audit_only`, plus transport class `audit-core.heartbeat`. Heartbeat data.class
+names one exact rare class, with assertion, committed and released_uncommitted
+counts. Four heartbeats are four retained events, not four decision counts.
+Outbox sequence and committed_at are local entry fields; the current wire event
+has no stream_instance_id or sequence. Do not promise native receiver instance
+filtering or wire sequence-gap detection for this source merely because the
+receiver has that optional filter. Scope, lifetime and boundary are required
+before comparing cumulative source counts.
+
+The new isolated receiver tests exercise all five decision types: first202,
+same-id duplicate200, changed-body409, write-only read403, independent fixture
+readback, exact source/tenant, once-only counts and half-open acceptance-time
+boundaries. They exercise wrong scope refusals and class-specific silence.
+The identities and narrowed tenant:platform in tests are synthetic fixtures,
+not proposed production grants. No live record or real decision validation is
+claimed by these tests.
+
+## Existing deployed evidence and its limit
+
+Audit `docs/evidence/2026-10-15-cadence-admission/receiver-cadence-release-native.json`
+records image `sha256:b479801b0bc76df66a36a85376c45aaf399c6d82feade9ba81430b774c4a8271`,
+Deployment UID `b85fe3d0-75c9-4e0d-8c34-c6f1df0881bb`, source5227fb1 and
+synthetic native cadence/heartbeat/exact-scope/SQLite/PostgreSQL feature probes.
+The same folder records migration0007 and admitted Qonto/Approval metadata.
+These are historical receiver capability receipts, not fresh flex-auth admission
+or independent protected-workflow acceptance. Revalidate deployed image, readyz,
+configuration metadata and migration compatibility before using them for native
+acceptance. No production inspection was performed for this packet.
+
+## Metadata prerequisites before any custody change
+
+Access owner must return one reviewed per-pin registration payload and canonical
+cadence bytes/digest for this pin, plus authoritative retained deployment source,
+image, namespace `flex-auth`, exact AND namespace/pod selectors and configured
+emission source. Do not derive pod selectors solely from a Helm release name.
+Existing six-pin commitment in docs/cross-repo-evidence-admission.md requests
+write true/read false, exact per-pin source, tenants ["*"] on the recorded
+deny-before-tenant-creation justification, explicit redact and load-bearing.
+This packet neither narrows that accepted scope to the fixture tenant nor widens
+it; a revised scope requires its owner review. Retention/access/export policy
+and owner must be explicit; an intake accepting custody is not that policy.
+
+Current `docs/contracts/decision-emission-cadence.json` declares generic
+source flex-auth, four heartbeat intervals86400 and allow expected_min1 per3600;
+its source state and observer status remain pending. It is not a final per-pin
+payload. Its daily source heartbeat cannot be registered as hourly f3600 without
+an Access-owned runtime/declaration update and provenance. If hourly frame is
+adopted, record f3600/wait7200/effective10800 and prove actual runtime adoption.
+Do not silently apply Approval Engine's `cadence_registration.prepare`: that
+helper deliberately validates source=approval-engine and its own class set.
+The generic SenderRegistry parser already supports the reviewed flex-auth
+metadata; custody tooling can validate an in-memory candidate with it.
+
+Platform owns contained sender seed/delivery, registry CAS/readback/ESO reload,
+transport and PVC admission. Resolve the actual usable custody route with
+`warden route find` and `show` before any credential/access request. Historical
+AUDIT-WP-0013-T02 names an unusable draft audit-core-senders route; do not assume
+it is corrected. Obtain current route evidence and required scoped approval.
+Never copy a registry/token to this packet or give a writer archive-read scope.
+Receiver ingress must pair namespace AND exact pod selector to port8080;
+source egress and receiver service DNS are separately admitted. Source-owned
+single-writer flock/64MiB bound is source progress; actual mounted durable PVC,
+restart, full/fenced failure, bounded drain and retained cursor need native proof.
+
+## Native writer acceptance, followed by independent readback
+
+1. Retain source/deployment/configuration pins, registry metadata version/digest,
+   source declaration digest, original exact change authorization and rollback.
+   Check accepted source/class/tenant/retention/secret policy inventory against
+   actual receiver configuration without exporting credentials.
+2. Owner runs the approved bounded fixture through its contained existing lane.
+   Record durable event IDs, original correlation/decision/approval/operation
+   references, source class/tenant, local outbox identity/lifetime and entry
+   sequence/committed_at; retain before/after cursor, pending, lag, counters and
+   sanitized first202/duplicate200/refusal receipts. No new metadata operation
+   is authorized by this packet; where native effect is unadmitted, retain wait.
+3. Exercise wrong source/tenant400, invalid/expired bearer401, writer read403,
+   payload conflict409 and formerly-valid bearer refusal after owner revocation.
+   A dead letter is not accepted custody and a source stdout202 is not readback.
+4. Existing independently authorized read-only owner retrieves each exact ID via
+   GET /v1/events/{id}; compare source, tenant, action, correlation and normalized
+   original payload/hash, plus accepted_at. GET /v1/events?correlation_id=... can
+   enumerate related records within that reader's actual authorized scope.
+   Record reader identity/role separation metadata and original receiver receipt
+   references; credentials stay contained. Existing operator reader is evidence
+   of a past lane, not automatic current access. If no admitted scoped reader
+   exists, this gate waits on that owner; never upgrade the source writer.
+5. Establish the full producer denominator before matching any chosen IDs. The
+   independent owner retains the complete producer outbox.jsonl snapshot, source
+   revision, outbox/PVC lifetime identity and bytes/SHA256/EOF high-water H. Check
+   every newline-terminated entry, contiguous seq1..H, exact source, unique event
+   IDs, valid committed_at and complete inventory including zero classes. Pin
+   drain cursor seq/offset, committed counters and pending/lag to that snapshot;
+   cursor must refer to a real entry boundary and never exceed H. This is an
+   independently retained full ledger, not a producer-selected list of successful
+   IDs. An arbitrary chosen-ID readback cannot satisfy this denominator. Retain
+   independent transition/effect-owner receipts too: an omitted transition never
+   queued by a compromised producer remains outside the archive guarantee.
+   Compare all ledger IDs versus independently retained receiver records, after
+   explicitly declared bounded drain lag. Missing pending
+   IDs are lag until the bound expires, then a finding; acknowledged-but-missing
+   IDs, conflicts and unexpected IDs are separate findings. Exclude four heartbeat
+   transport events from decision-effect totals. Preserve duplicate attempt counts
+   while accepted IDs count once. Keep nonzero released_uncommitted as a finding;
+   zero observed after restart does not establish its historical absence.
+6. For aggregate GET /v1/reconciliation name exact source/tenant and explicit
+   [since,until) accepted_at bounds. Derive the local comparison set by the same
+   independently read accepted_at of the exact retained IDs, not occurred_at,
+   local commit time or a cumulative GET /v1/emission snapshot. Report unresolved
+   boundary arrivals separately and repeat after bounded drain. Source status
+   lacks tenant/window splits and persisted remote acceptance times; its raw
+   total cannot be compared to an arbitrary receiver time-window total. This
+   event-ID/accepted_at method supports a bounded admitted run without claiming a
+   general unattended reconciliation implementation. Compare the complete map of
+   all classes, including unexpected classes, against receiver aggregate counts:
+   all expected IDs must exist, and any receiver count above the independently
+   enumerated expected map is an unaccounted-record finding. This detects extras
+   without pretending GET-by-ID itself enumerates the receiver. Correlation search
+   has a1000 ceiling and no pagination; it is not a complete source export.
+   Concurrent arrivals require a second independently retained complete ledger
+   watermark H2 covering the same receiver window, or an approved quiescent
+   bounded run. Do not report agreement while that boundary remains unsettled.
+   Independently retained full snapshots, repeated cadence execution, actual
+   missing-class observation and independent observer adoption remain required
+   for operational TSA-09 acceptance; this packet alone closes none of them.
+7. Independently observe four distinct rare-class heartbeats, actual configured
+   cadence and lag. A controlled withheld-heartbeat check belongs in an admitted
+   non-production exercise; do not suppress production evidence. Keep never-seen,
+   stale, missing-class, duplicate, count and boundary findings distinct. King's
+   Guard owns independent operational observer acceptance under KG-WP-0005.
+
+Rollback restores only the exact earlier admitted scope/configuration, records
+uncertain delivery and retains outbox/cursor/events. It never deletes evidence,
+replays an uncertain protected effect, accepts unsigned decisions, widens a grant
+or revives a revoked credential. Audit arrival proves custody after arrival;
+neither arrival nor an empty finding set proves source completeness. Independent
+chain-head/offsite custody remains AUDIT-WP-0015-T01's separate open boundary.
+
+## Cross-owner accounting review — 2026-10-16
+
+Access owner confirms the initial lifecycle signer slice keeps auditDelivery
+disabled. Its current native package has no wire/accounting extension and makes
+no G2/hourly-adoption claim. Audit owner accepts the complete independently
+retained ledger + exact-ID readback + all-class receiver-count protocol above
+for a bounded source-to-receiver reconciliation. A source self-selected receipt
+list is insufficient. This is an evidence protocol using existing APIs, not a
+new assertion format or permission. Audit has no current source-wide paginated
+export endpoint or canonical producer-accounting receipt schema. Original owner
+ledger bytes and receiver records are authoritative; a Custodian summary is not.
+
+For future unattended monitoring, Access may separately propose durable
+outbox-lifetime stream identity and sequence metadata, but must preserve signed
+DecisionEnvelope bytes. A receiver-compatible wrapper would need a source-owned
+versioned wire contract and fixture changes; post-sign insertion into decision
+data is refused. An emitter restart is not automatically a new outbox lifetime.
+No producer extension is mandatory for the bounded full-ledger protocol above;
+repeated complete producer ledger capture/independent retention and source-owned
+per-pin cadence/runtime adoption are mandatory operational work still missing.
+
+The previously skipped existing cross-owner test was run directly from the real
+canonical Audit checkout using its actual sibling NetKingdom/Approval fixtures:
+`python3 -m pytest -q tests/test_cadence_registration.py::test_real_owner_proposal_when_checkout_available`
+passed1. This makes114 tested cases across the focused isolated run and the
+canonical owner-fixture check; the isolated run itself remains113 pass/1 skip.
+No foreign source was changed (pytest cache writes were denied by the sandbox).

tests/test_flex_auth_native_contract.py

@@ -0,0 +1,90 @@
+"""AUDIT-WP-0013-T02: wire compatibility, never live sender admission.
+
+Fixture field mapping is pinned in docs/flex-auth-secrets-engine-admission.md.
+All identities/tokens below exist only in the isolated test receiver.
+"""
+from datetime import datetime, timedelta, timezone
+from urllib.parse import urlencode
+
+import pytest
+
+from audit_core.ingestion import IngestionApplication
+from audit_core.senders import SenderIdentity, SenderRegistry
+from audit_core.sqlite_backend import SQLiteAuditBackend
+from audit_core.stream_findings import evaluate, NEVER_HEARTBEAT
+from test_stream_findings import invoke
+
+SOURCE = "flex-auth.secrets-engine"
+RARE = tuple("flex-auth.decision." + name for name in ("deny", "redact", "not_applicable", "audit_only"))
+CLASSES = RARE + ("flex-auth.decision.allow",)
+
+
+@pytest.fixture
+def receiver(tmp_path):
+    writer = SenderIdentity(name=SOURCE, tokens=("fixture-writer",), sources=frozenset({SOURCE}),
+                            tenants=frozenset({"tenant:platform"}), may_read=False,
+                            evidence_kind="load-bearing", heartbeat_classes=tuple((c, 86400) for c in RARE))
+    reader = SenderIdentity(name="fixture-independent-reader", tokens=("fixture-reader",),
+                            sources=frozenset({SOURCE}), tenants=frozenset({"tenant:platform"}),
+                            may_write=False, may_read=True)
+    return IngestionApplication(SQLiteAuditBackend(str(tmp_path / "events.db")), SenderRegistry([writer, reader])), writer
+
+
+def envelope(kind, event_id="fixture-event"):
+    # outbox.go Event + event(): no receiver-derived fields or invented sequence.
+    return dict(id=event_id, type=kind, source=SOURCE, subject="openbao/metadata/fixture",
+                tenant="tenant:platform", correlation_id="fixture-decision",
+                occurred_at="2026-10-16T00:00:00.123456789Z",
+                data={"id": "fixture-decision", "effect": kind.rsplit(".", 1)[-1]})
+
+
+def post(app, event):
+    return invoke(app, "/v1/events", token="fixture-writer", method="POST", payload=event, key=event["id"])
+
+
+def count(app, since, until, source=SOURCE, tenant="tenant:platform"):
+    return invoke(app, "/v1/reconciliation?" + urlencode(dict(source=source, tenant=tenant, since=since, until=until)), token="fixture-writer")
+
+
+@pytest.mark.parametrize("kind", CLASSES)
+def test_exact_pin_wire_duplicate_and_independent_readback(receiver, kind):
+    app, _ = receiver
+    event = envelope(kind)
+    assert post(app, event)[0].startswith("202")
+    assert post(app, event)[0].startswith("200")
+    assert post(app, dict(event, subject="changed"))[0].startswith("409")
+    assert invoke(app, "/v1/events/fixture-event", token="fixture-writer")[0].startswith("403")
+    status, retained = invoke(app, "/v1/events/fixture-event", token="fixture-reader")
+    assert status.startswith("200")
+    assert (retained["source"], retained["tenant"], retained["action"]) == (SOURCE, "tenant:platform", kind)
+    assert retained["details"]["data"] == event["data"]
+    accepted = datetime.fromisoformat(retained["accepted_at"])
+    before, after = ((accepted + timedelta(seconds=n)).isoformat() for n in (-1, 1))
+    status, counts = count(app, before, after)
+    assert status.startswith("200")
+    assert counts["counts"] == [{"class": kind, "count": 1}]
+    # Acceptance-time [since, until): source occurred_at is deliberately older.
+    assert count(app, before, accepted.isoformat())[1]["counts"] == []
+    assert count(app, accepted.isoformat(), after)[1]["counts"] == [{"class": kind, "count": 1}]
+
+
+def test_wrong_scope_refusals_are_not_empty_counts(receiver):
+    app, _ = receiver
+    assert post(app, dict(envelope(CLASSES[0]), source="flex-auth.other"))[0].startswith("400")
+    assert post(app, dict(envelope(CLASSES[0]), tenant="tenant:other"))[0].startswith("400")
+    window = ("2026-10-15T00:00:00Z", "2027-01-01T00:00:00Z")
+    assert count(app, *window, source="flex-auth.other")[0].startswith("403")
+    assert count(app, *window, tenant="tenant:other")[0].startswith("403")
+
+
+def test_each_rare_class_must_have_its_own_positive_claim(receiver):
+    app, writer = receiver
+    now = datetime.now(timezone.utc)
+    covered = RARE[0]
+    event = dict(envelope("audit-core.heartbeat"), subject=SOURCE,
+                 data={"class": covered, "assertion": "nothing-to-report", "committed": 0, "released_uncommitted": 0})
+    assert post(app, event)[0].startswith("202")
+    # Source/class finding evaluator; a heartbeat for deny cannot cover redact.
+    findings = evaluate([writer], {(SOURCE, covered): now.isoformat()}, now=now)
+    assert {f.event_class for f in findings} == set(RARE) - {covered}
+    assert all(f.kind == NEVER_HEARTBEAT for f in findings)

workplans/AUDIT-WP-0013-cross-repo-sender-admission.md

@@ -131,3 +131,26 @@ not independent receiver readback. Real elapsed hourly observation, independent
 active-instance/class-silence/sequence/count/lag assessment and Approval lifecycle
 acceptance remain with NK-WP-0035/KG-WP-0005. Source-global receiver findings must
 not misclassify intentional Qonto scale-to-zero as an active-instance omission.
+
+### First protected pin admission preparation — 2026-10-16
+
+T02 remains wait. Prepared docs/flex-auth-secrets-engine-admission.md against
+Access0a05383195213e7eaacd08388ce1d7892ce756df and Audit2e97f37. Existing wire
+contracts need no adapter. Seven local tests cover all five decision classes,
+retry/conflict, independent fixture readback, writer-read and wrong-scope refusal,
+half-open acceptance-time accounting and rare-class silence. This does not grant
+or admit a sender. Per-pin source/cadence/policy/selectors, usable attended custody
+route, source deployment/PVC and independent native readback remain owner gates.
+Generic source cadence remains daily and cannot stand in for admitted hourly
+runtime. Source outbox sequence is not on the wire; exact event-ID/accepted_at
+comparison is the bounded readback method, not cumulative-time-window equality.
+
+Cross-owner follow-up: Access confirms auditDelivery remains disabled in the
+initial signer slice. The packet now requires independently retained complete
+producer JSONL with contiguous sequence/high-water and lifetime identity, not a
+selected-ID list. Existing exact-ID reads plus all-class accepted-time count
+comparison support bounded full-ledger reconciliation; correlation search is
+capped and cannot serve as complete export. Operational repeated ledger custody,
+per-pin source cadence/runtime adoption and independent observation still wait.
+The canonical sibling-owner fixture check passed1, resolving the isolated
+checkout's expected skip without modifying foreign source.