approval-engine/workplans/APPROVAL-WP-0002-production-readiness-and-consumer-adoption.md

---
id: APPROVAL-WP-0002
type: workplan
title: "Production readiness and consumer adoption"
domain: infotech
repo: approval-engine
status: active
owner: codex
topic_slug: netkingdom
created: "2026-09-01"
updated: "2026-09-06"
reviewed_at: "2026-09-01"
reviewed_against_commit: "ebce5abb276c01ab29ce2526f3b8abb332dc9e90"
reviewed_note: >-
  Reviewed against approval-engine's finished spine, KeyCape's RS256/JWKS and
  service-token contract, access-engine caller-auth/binding surface,
  audit-core's deployed authenticated ingestion plus open AUDIT-WP-0009
  approval-source/cadence/reconciliation tasks, and secrets-engine's waiting
  exact-action/decision-consumption tasks. Repo-owned implementation can
  proceed; live T03-T05 closure remains evidence-gated on those owners.
origin: residual
origin_ref: APPROVAL-WP-0001
state_hub_workstream_id: "4fa25ad5-f5d0-5592-aa59-085f8ee3edaf"
---

# APPROVAL-WP-0002 — Production readiness and consumer adoption

Move the completed first-cut engine spine into an authenticated, durable,
observable production service and prove one PEP integration end to end. This is
the residual production scope deliberately excluded from APPROVAL-WP-0001.

The workplan is proposed pending review against the deployment estate and the
current key-cape, access-engine, audit-core, and secrets-engine contracts.

Review completed 2026-09-01. The plan is active. Production mode will verify
KeyCape JWT signatures and exact issuer/audience/scopes; local caller-supplied
identity never becomes authenticated evidence. SQLite remains the first
production store only as a single-replica StatefulSet with explicit migration,
backup, integrity, and restore gates. audit-core delivery can be implemented
against its existing authenticated idempotent ingest, while heartbeat findings,
count reconciliation, and sender registration remain external gates in
`AUDIT-WP-0009` T04/T06/T09. The first live PEP proof remains jointly gated on
secrets-engine T04/T02 and deployment of this service.

## Authenticate lifecycle mutations and approver evidence

```task
id: APPROVAL-WP-0002-T01
status: progress
priority: high
state_hub_task_id: "dc4523f5-e0af-5734-a0ef-07aa7f2b2a27"
```

Bind create, approval-entry, revoke, supersede, and consume callers to
authenticated identities. An API-supplied `subject_id`, `actor`, or
`decision_id` is provenance only until independently authenticated. Keep
authorization decisions in access-engine and approval doctrine in gate-house.

Acceptance: production startup requires a signature-valid KeyCape JWT verifier;
every non-health API route requires an explicit scope; approval entry identity
and assurance come only from verified claims; create binds `binding.actor` to
the authenticated subject; consume is restricted to service/agent principals;
missing, expired, wrong-issuer, wrong-audience, wrong-scope, or unverifiable
tokens fail closed without mutation.

Repository implementation complete 2026-09-02: RS256/JWKS verification,
issuer/audience/time/profile validation, exact scopes, store-tenant isolation,
verified approver evidence, and a deny-all default are covered by tests.

2026-09-02 follow-up: fail-closed coverage now includes wrong signature, HS256,
empty/invalid profile claims, deny-all mutation refusal, human-principal consume
rejection, and production CLI refusal of static tokens / missing audit
delivery. Requested KeyCape registrations are in
`docs/keycape-service-registrations.md` (`aud` MUST be the resource server
`approval-engine`, not the OAuth client id). Remains `progress` until KeyCape
owns and proves those audience/client/scope registrations.

## Harden durable storage and migrations

```task
id: APPROVAL-WP-0002-T02
status: done
priority: high
state_hub_task_id: "2bef94ca-9482-5eff-9bd9-39adef292a78"
```

Define the production persistence, backup/restore, migration, concurrency, and
recovery posture. Prove schema upgrades preserve existing approvals and that
crash recovery cannot separate mutations from outbox evidence.

Acceptance: schema version is explicit; production serve refuses an unmigrated
or in-memory store; migration is a separate repeatable command; backup uses
SQLite's online backup API and integrity verification; restore is documented as
a stopped-single-writer operation; tests cover legacy upgrade, backup/restore,
and outbox atomicity after restart.

Completed 2026-09-02: schema v2 migration, production no-auto-migrate gate,
integrity/status surface, online mode-0600 backup, stopped-writer restore runbook,
retry-attempt state, and migration/backup/atomicity tests are in place.

## Package and deploy the service

```task
id: APPROVAL-WP-0002-T03
status: wait
priority: high
state_hub_task_id: "f0aa2e6d-19e6-5b43-886c-efa4e3de5f22"
```

Add the governed image/deployment surface, health and readiness behavior,
resource bounds, and fail-closed caller configuration. A local WSGI development
server is not production evidence.

Acceptance: a digest-pin-ready image and single-writer StatefulSet manifest
exist with non-root/read-only-root controls, PVC, migration init container,
resource bounds, probes, and default-deny network policy. Live completion also
requires an immutable image digest, KeyCape registrations, audit sender
credential, successful rollout, and restart/restore evidence.

Repository implementation complete 2026-09-02: the digest-pin-ready non-root
image builds and runs; the single-writer StatefulSet, PVC, migration init,
read-only root, resources, probes, and default-deny policies pass client dry-run.
Waiting on release digest, KeyCape/audit registrations and credentials, rollout,
restart, and restore evidence.

## Wire outbox delivery and reconciliation

```task
id: APPROVAL-WP-0002-T04
status: wait
priority: high
state_hub_task_id: "777a5cf5-c1db-5e07-8de6-ab109db5ccdc"
```

Deliver the local outbox asynchronously to audit-core, preserve event-id
deduplication, publish lag/depth signals, emit the declared heartbeat, and prove
the Gate House reconciliation contract against accepted event counts.

Acceptance: the sender adapts the local event to audit-core's authenticated
HTTP ingest, reuses event id as idempotency key, rereads a mounted token file,
marks drained only on accepted/duplicate, retains retryable failures, exposes
attempt/lag state, and emits the declared heartbeat. Live reconciliation waits
on `AUDIT-WP-0009-T04/T06/T09`; do not invent that receiver surface here.

Repository implementation complete 2026-09-02: audit-core envelope adaptation,
event-id idempotency, mounted-token reread, accepted/duplicate handling,
retry-attempt/lag metrics, and periodic heartbeats are tested.

2026-09-02 follow-up: drain tests now cover HTTP 200 duplicate as drained and
urllib `HTTPError` 503 as pending. Waiting on the audit-core sender
registration/ingress and receiver-owned reconciliation work
(`AUDIT-WP-0009-T04/T06/T09`).

## Prove one live PEP consumption path

```task
id: APPROVAL-WP-0002-T05
status: wait
priority: high
state_hub_task_id: "3196fb77-3df0-5358-ae12-2d07bac12041"
```

Integrate one protected-system consumer under `GH-DEC-2026-003`: claim before
decision, CAS consume after ALLOW and before side effect, same-digest retry,
different-digest conflict, spent-on-failure behavior, and no protected action
when approval-engine is unavailable.

Acceptance: a repeatable harness proves the PEP sequence against the real HTTP
surface without performing a protected action; live closure requires a
secrets-engine-owned handler and evidence that no OpenBao call occurs on every
failure case. This repo may supply the protocol client and fixture, but may not
claim the consumer's side effect.

Repository implementation complete 2026-09-02: the HTTP PEP client and
fail-closed sequencing harness prove claim-before-decision and CAS-consume-before
callback, including unavailable, DENY, digest mismatch, and conflict paths.

2026-09-02 follow-up: secrets-engine shipped the PEP consume-before-OpenBao
handler (`src/secrets_engine/approval_consume.py`; inbox `0e04b4e2`). This
engine's client now maps HTTP 409/401/403/404/503 and unreachability, requires
the canonical digest, and refuses a decision-shaped consume payload. The
repeatable harness in `tests/test_pep.py` drives the real HTTP surface: first
consume then same-digest retry, different-digest conflict, spent-on-failure
claim refusal, and unreachable-engine callback suppression. Waiting on live
closure: this service deployed (T03) and a durable consume binding served
(`SECRETS-WP-0007-T04` / `SECRETS-WP-0008-T02`). This repo does not claim the
OpenBao side effect.

2026-09-06 follow-up: secrets-engine reports its PIP join is implemented and
blocked on deployment, not contract (inbox `61ae1174`). Review of its
`validate_action_authorization` shows a real envelope divergence: it expects a
`state-hub`-authority `ActionAuthorization` (`id`, `status`, `superseded_by`,
`request`, `approvals.entries`, policy pin) while this engine serves the
governed approval-claim (`approval_id`, `state`/`valid_now`, `binding`,
`freshness`, `reason_code`, `issuer: approval-engine`). Both declare
`schema_version` `0.1`, so the mismatch surfaces as a field/authority error
rather than a version error. Recorded in `docs/approval-consumption.md`;
reconciling the envelopes is a `GH-DEC-2026-003` cross-repo change, not a
unilateral edit here. T05 stays `wait`: still no deployed base URL (T03).

2026-09-06 ruling: the claim-envelope question is settled. `GH-DEC-2026-005`
confirms all three requested dispositions — the approval-claim is the step-1
artifact, `ActionAuthorization` is not required and MUST NOT be served from the
claim endpoint, and a PEP validates across the claim and the step-2
`DecisionEnvelope`. Gate House recorded the split as doctrine (a PIP must not
republish the PDP's decision) and struck the `provenance.authority ==
"state-hub"` requirement explicitly. flex-auth accepted as `FLEX-DEC-2026-006`,
argued against its own proposal, and traced the bad authority constant to a
fixture rather than prose. **This engine changes nothing: the claim schema
stands as published.** `APPROVAL-IN-0002` is closed. T05 remains `wait` on T03
deployment plus the secrets-engine validator split and its
`secrets-engine-approval` KeyCape registration (already requested verbatim in
`docs/keycape-service-registrations.md`).

2026-09-06 follow-on (T01): `GH-DEC-2026-005` assigned this engine a
compensating obligation. secrets-engine has implemented the split and reports
it no longer verifies the distinct-approver threshold independently; Gate House
accepted that as correct on layering and as a real reduction in defence in
depth, requiring instead that the threshold evaluation be reconstructable from
this engine's state transitions and its use outbox row under §9.6. Implemented:
`approval.issuance` and `approval.use` now carry a `threshold` object
(`required_count`, `distinct_approver_count`, `threshold_met`, and `approvers`
with `approved_at` plus assurance/evidence refs). Tests prove reconstruction
from the use row alone and that the claim still discloses no approver
identities. Documented in `docs/outbox-contract.md`.

2026-09-06 follow-on (T05): flex-auth asked whether this engine should publish
an action/target vocabulary mapping between the claim binding and their policy
package before `SECRETS-WP-0007-T04` makes destroy reachable. Answered no, and
recorded why in `docs/approval-claim.md`: a PIP asserting that one vocabulary's
action *means* another's would author policy semantics it does not own, and a
wrong mapping silently accepts a claim approved for a different action.
`binding.pdp_digest` is the correspondence — it compares the PDP's digest to
the PDP's digest with no translation. Implemented the part that was ours:
`pdp_digest` is now always present on the claim and `null` when the approval
was not issued against a PDP decision, so absence is a stated fact rather than
a missing key, and the schema requires it as nullable. A PEP on a privileged
lane must refuse a null. Both published examples were contradicting the schema;
fixed, and `tests/test_examples.py` now validates every example against it.