approval-engine/workplans/APPROVAL-WP-0002-production-readiness-and-consumer-adoption.md

---
id: APPROVAL-WP-0002
type: workplan
title: "Production readiness and consumer adoption"
domain: infotech
repo: approval-engine
status: active
owner: codex
topic_slug: netkingdom
created: "2026-09-01"
updated: "2026-09-06"
reviewed_at: "2026-09-01"
reviewed_against_commit: "ebce5abb276c01ab29ce2526f3b8abb332dc9e90"
reviewed_note: >-
  Reviewed against approval-engine's finished spine, KeyCape's RS256/JWKS and
  service-token contract, access-engine caller-auth/binding surface,
  audit-core's deployed authenticated ingestion plus open AUDIT-WP-0009
  approval-source/cadence/reconciliation tasks, and secrets-engine's waiting
  exact-action/decision-consumption tasks. Repo-owned implementation can
  proceed; live T03-T05 closure remains evidence-gated on those owners.
origin: residual
origin_ref: APPROVAL-WP-0001
state_hub_workstream_id: "4fa25ad5-f5d0-5592-aa59-085f8ee3edaf"
---

# APPROVAL-WP-0002 — Production readiness and consumer adoption

Move the completed first-cut engine spine into an authenticated, durable,
observable production service and prove one PEP integration end to end. This is
the residual production scope deliberately excluded from APPROVAL-WP-0001.

The workplan is proposed pending review against the deployment estate and the
current key-cape, access-engine, audit-core, and secrets-engine contracts.

Review completed 2026-09-01. The plan is active. Production mode will verify
KeyCape JWT signatures and exact issuer/audience/scopes; local caller-supplied
identity never becomes authenticated evidence. SQLite remains the first
production store only as a single-replica StatefulSet with explicit migration,
backup, integrity, and restore gates. audit-core delivery can be implemented
against its existing authenticated idempotent ingest, while heartbeat findings,
count reconciliation, and sender registration remain external gates in
`AUDIT-WP-0009` T04/T06/T09. The first live PEP proof remains jointly gated on
secrets-engine T04/T02 and deployment of this service.

## Authenticate lifecycle mutations and approver evidence

```task
id: APPROVAL-WP-0002-T01
status: progress
priority: high
state_hub_task_id: "dc4523f5-e0af-5734-a0ef-07aa7f2b2a27"
```

Bind create, approval-entry, revoke, supersede, and consume callers to
authenticated identities. An API-supplied `subject_id`, `actor`, or
`decision_id` is provenance only until independently authenticated. Keep
authorization decisions in access-engine and approval doctrine in gate-house.

Acceptance: production startup requires a signature-valid KeyCape JWT verifier;
every non-health API route requires an explicit scope; approval entry identity
and assurance come only from verified claims; create binds `binding.actor` to
the authenticated subject; consume is restricted to service/agent principals;
missing, expired, wrong-issuer, wrong-audience, wrong-scope, or unverifiable
tokens fail closed without mutation.

Repository implementation complete 2026-09-02: RS256/JWKS verification,
issuer/audience/time/profile validation, exact scopes, store-tenant isolation,
verified approver evidence, and a deny-all default are covered by tests.

2026-09-02 follow-up: fail-closed coverage now includes wrong signature, HS256,
empty/invalid profile claims, deny-all mutation refusal, human-principal consume
rejection, and production CLI refusal of static tokens / missing audit
delivery. Requested KeyCape registrations are in
`docs/keycape-service-registrations.md` (`aud` MUST be the resource server
`approval-engine`, not the OAuth client id). Remains `progress` until KeyCape
owns and proves those audience/client/scope registrations.

2026-09-06 follow-up: a tenant collision in the deployment inputs is now
recorded in `docs/keycape-service-registrations.md`. The manifest serves
`--tenant platform` while the requested client registrations issue
`tenant: tenant:coulomb`, and `ApiApplication.identity` compares the two with
exact string equality before any object lookup — so tokens issued under the
current registrations would be denied `403` on every non-health route.
flex-auth's `tenant:platform` CheckRequest subject is a PDP input this engine
never reads and cannot participate in the comparison. Denial evidence:
`tests/test_auth.py::test_wrong_tenant_is_forbidden`. The values are left as-is
deliberately: resolving it requires an owner statement on whether `platform` and
`tenant:coulomb` name the same layer, and guessing grants cross-tenant access to
the approval store. The registrations doc's stale issuer
(`https://auth.netkingdom.local`) is corrected to the live
`https://kc.coulomb.social` from `06544b0`. T01 stays `progress`.

2026-09-06 resolution: the operator accepted `tenant:platform` as the platform
management/administration/services tenant (landlord zone) — decision
`5ed3fb35-eca9-413a-82b9-95171ba85bf6`,
`glas-harness/docs/platform-tenant-decision.md`, relayed by `glas-harness`. The
collision above is closed by setting the store tenant to exactly
`tenant:platform`: `deploy/approval-engine.yaml` `--tenant`, the
`approval_engine/cli.py` `--tenant` default, and the `Engine(tenant=…)` default
all move off bare `platform`, and the requested client registrations now ask for
`tenant: tenant:platform`. Exact JWT/store equality is retained — no alias, no
normalisation, no prefix handling, and no implicit cross-tenant grant.

Moving the *defaults* rather than only the manifest is deliberate: a default of
`platform` under a sanctioned value of `tenant:platform` is a trap, because a
`serve` that omits `--tenant` would come up healthy and then refuse every
authenticated call.

The default change broke ten tests whose identity fixtures hard-coded
`platform`, which is the fixture-consistency hazard flex-auth reported as
`FLEX-DEC-2026-008` (29 fixtures all carrying one tenant proved nothing about
the field). Fixtures are aligned to the exact spelling, and the field is now
*varied* rather than merely present:
`tests/test_auth.py::test_near_miss_tenant_spellings_are_forbidden` refuses
`platform`, `tenant:coulomb`, case variants, whitespace variants and empty
against a `tenant:platform` store, and
`test_exact_sanctioned_tenant_is_admitted` pins the other half so a
reject-everything bug cannot pass. 111 tests pass.

This resolves the choice of value only. T01 stays `progress`: KeyCape still has
to own and prove these registrations, and credential materialization is
unchanged.

## Harden durable storage and migrations

```task
id: APPROVAL-WP-0002-T02
status: done
priority: high
state_hub_task_id: "2bef94ca-9482-5eff-9bd9-39adef292a78"
```

Define the production persistence, backup/restore, migration, concurrency, and
recovery posture. Prove schema upgrades preserve existing approvals and that
crash recovery cannot separate mutations from outbox evidence.

Acceptance: schema version is explicit; production serve refuses an unmigrated
or in-memory store; migration is a separate repeatable command; backup uses
SQLite's online backup API and integrity verification; restore is documented as
a stopped-single-writer operation; tests cover legacy upgrade, backup/restore,
and outbox atomicity after restart.

Completed 2026-09-02: schema v2 migration, production no-auto-migrate gate,
integrity/status surface, online mode-0600 backup, stopped-writer restore runbook,
retry-attempt state, and migration/backup/atomicity tests are in place.

## Package and deploy the service

```task
id: APPROVAL-WP-0002-T03
status: wait
priority: high
state_hub_task_id: "f0aa2e6d-19e6-5b43-886c-efa4e3de5f22"
```

Add the governed image/deployment surface, health and readiness behavior,
resource bounds, and fail-closed caller configuration. A local WSGI development
server is not production evidence.

Acceptance: a digest-pin-ready image and single-writer StatefulSet manifest
exist with non-root/read-only-root controls, PVC, migration init container,
resource bounds, probes, and default-deny network policy. Live completion also
requires an immutable image digest, KeyCape registrations, audit sender
credential, successful rollout, and restart/restore evidence.

Repository implementation complete 2026-09-02: the digest-pin-ready non-root
image builds and runs; the single-writer StatefulSet, PVC, migration init,
read-only root, resources, probes, and default-deny policies pass client dry-run.
Waiting on release digest, KeyCape/audit registrations and credentials, rollout,
restart, and restore evidence.

2026-09-06 image preparation (GLAS-WP-0015 request; credential-independent half):
built and validated locally against the agreed tenant and current schema.

- Build: `make image-build` off the digest-pinned base
  `python:3.12-slim@sha256:d764629c…`. Local manifest-list digest
  `sha256:85e46ddf47b3ac0cdab620163a7b034027451dff286b63ecff6a2493d7a57ecc`.
  **This is a local build digest, not a release digest** — the manifest still
  carries `REPLACE_WITH_RELEASE_DIGEST` because pinning requires a push to
  `forgejo.coulomb.social`, which needs registry credentials this session does
  not hold.
- Runtime identity: `uid=10001(approval) gid=10001(approval)`, non-root as
  required.
- Schema: image carries `LATEST_SCHEMA_VERSION = 3`, matching the migrated
  store, and `Engine` tenant default `tenant:platform`.
- First-install migration (no prior DB): `migrate` then `verify` on a fresh
  volume both report `schema_version: 3`, `schema_current: true`,
  `integrity: ["ok"]`, `foreign_key_violations: 0`, `persistent: true`.
- Fail-closed configuration proven in the image, not only in tests:
  `serve --production --db :memory:` refuses with "production requires a
  persistent database"; `serve --production` on a real DB without audit
  configuration refuses with "production requires authenticated audit
  delivery".
- Manifest inputs: `kubectl apply --dry-run=client` passes for the namespace,
  service, and StatefulSet with the new `--tenant tenant:platform`.

Scan gate NOT met: no scanner (`trivy`, `grype`, `docker scout`) is installed on
this workstation, so no vulnerability scan was run and none is claimed. The
inventory a scanner needs is the pinned base above plus
`cryptography==50.0.1`, `PyJWT==2.13.0`, `waitress==3.0.2`, `cffi==2.1.1`,
`pycparser==3.0`, `pip==25.0.1`.

T03 stays `wait`: still no release digest, no KeyCape/audit credentials, no
rollout, and no restart/restore evidence. Nothing here is a deploy.

2026-09-06 scan gate — run, and it failed on the sanctioned base. Full record in
`docs/image-scan-2026-09-06.md`. Scanned with trivy (no scanner was installed;
this one needed none). Findings are all inherited from the base image, none in
`approval_engine` code.

Two fixes applied. The base pin was stale at Debian 13.5 while upstream is
13.6 — bumping the digest removed 30 HIGH and 47 MEDIUM. And pip, which held all
10 MEDIUM Python findings, is now absent from the runtime image via a two-stage
build; it is a build-time tool and had no business in a running approval
service.

What remains is a decision, not a task. Three CRITICALs persist on Debian, all
`perl-base` (`CVE-2026-13221`, `CVE-2026-42496`, `CVE-2026-8376`), **none with
an upstream fix**, in a package this service never invokes and which Debian
marks `Essential: yes`. An Alpine variant carries no perl and scans 0 CRITICAL /
7 HIGH / 1 MEDIUM against Debian's 3 / 51 / 56. It is proven viable — musl
wheels resolve without a toolchain, the full suite passes on musl at 111, and
non-root identity, schema v3, `tenant:platform`, fresh-store migrate/verify and
both production fail-closed gates all hold in the image. It is parked in
`Containerfile.alpine` as a candidate; `Containerfile` remains the sanctioned
base.

Nothing was pushed. Pushing the Debian variant would pin three unfixable
CRITICALs into a release digest, and choosing the runtime C library for this
service is not a call to make silently. `deploy/approval-engine.yaml` still
carries `REPLACE_WITH_RELEASE_DIGEST`. Awaiting an owner answer on the base
(Alpine, a documented perl exception on glibc, or distroless — unevaluated) and
on which scanner is sanctioned.

2026-09-06 both answered; release candidate scans clean. The operator adopted
Alpine/musl as the sanctioned base and trivy as the sanctioned scanner.
`Containerfile.alpine` is promoted to `Containerfile` and the Debian variant is
retired rather than kept as an option.

Promoting Alpine left 7 HIGH and 1 MEDIUM, all `libuuid` 2.42.1-r0 and all with
fixes in 2.42.3. The runtime stage now requires `libuuid>=2.42.3-r1` — a version
floor rather than a floating upgrade, so the base stays digest-pinned and a
vulnerable libuuid fails the build instead of shipping. **The candidate now
scans 0 CRITICAL / 0 HIGH / 0 MEDIUM / 0 LOW.**

Verified on that exact artifact: non-root `uid=10001`, pip absent, schema v3,
tenant default `tenant:platform`, fresh-store `migrate`+`verify` clean, restart
persistence via re-`verify` on the same volume, both production fail-closed
refusals, 111 tests on musl, and `kubectl --dry-run` passing.

The gate is now reproducible rather than a one-off: `make image-scan` fails on
any CRITICAL or HIGH, and `make image-release` runs build then scan then push so
a failing scan blocks the push by construction.

Still `wait`, and now for exactly one reason: `docker push` was attempted and
refused by this session's sandbox as an outward-facing publish. It was not
worked around. No release digest exists, so the manifest keeps
`REPLACE_WITH_RELEASE_DIGEST`; an operator must run `make image-release` (or the
push and `docker inspect`) and pin the returned registry manifest digest — never
the tag, never the local image id
`sha256:73333f5ceb55e48192e3095cb2e2a741cdc6ff0be2f18128301072b4a6b6eb9d`.
Rollout, restart and restore evidence remain gated on the KeyCape registrations
and audit sender credential.

## Wire outbox delivery and reconciliation

```task
id: APPROVAL-WP-0002-T04
status: wait
priority: high
state_hub_task_id: "777a5cf5-c1db-5e07-8de6-ab109db5ccdc"
```

Deliver the local outbox asynchronously to audit-core, preserve event-id
deduplication, publish lag/depth signals, emit the declared heartbeat, and prove
the Gate House reconciliation contract against accepted event counts.

Acceptance: the sender adapts the local event to audit-core's authenticated
HTTP ingest, reuses event id as idempotency key, rereads a mounted token file,
marks drained only on accepted/duplicate, retains retryable failures, exposes
attempt/lag state, and emits the declared heartbeat. Live reconciliation waits
on `AUDIT-WP-0009-T04/T06/T09`; do not invent that receiver surface here.

Repository implementation complete 2026-09-02: audit-core envelope adaptation,
event-id idempotency, mounted-token reread, accepted/duplicate handling,
retry-attempt/lag metrics, and periodic heartbeats are tested.

2026-09-02 follow-up: drain tests now cover HTTP 200 duplicate as drained and
urllib `HTTPError` 503 as pending. Waiting on the audit-core sender
registration/ingress and receiver-owned reconciliation work
(`AUDIT-WP-0009-T04/T06/T09`).

## Prove one live PEP consumption path

```task
id: APPROVAL-WP-0002-T05
status: wait
priority: high
state_hub_task_id: "3196fb77-3df0-5358-ae12-2d07bac12041"
```

Integrate one protected-system consumer under `GH-DEC-2026-003`: claim before
decision, CAS consume after ALLOW and before side effect, same-digest retry,
different-digest conflict, spent-on-failure behavior, and no protected action
when approval-engine is unavailable.

Acceptance: a repeatable harness proves the PEP sequence against the real HTTP
surface without performing a protected action; live closure requires a
secrets-engine-owned handler and evidence that no OpenBao call occurs on every
failure case. This repo may supply the protocol client and fixture, but may not
claim the consumer's side effect.

Repository implementation complete 2026-09-02: the HTTP PEP client and
fail-closed sequencing harness prove claim-before-decision and CAS-consume-before
callback, including unavailable, DENY, digest mismatch, and conflict paths.

2026-09-02 follow-up: secrets-engine shipped the PEP consume-before-OpenBao
handler (`src/secrets_engine/approval_consume.py`; inbox `0e04b4e2`). This
engine's client now maps HTTP 409/401/403/404/503 and unreachability, requires
the canonical digest, and refuses a decision-shaped consume payload. The
repeatable harness in `tests/test_pep.py` drives the real HTTP surface: first
consume then same-digest retry, different-digest conflict, spent-on-failure
claim refusal, and unreachable-engine callback suppression. Waiting on live
closure: this service deployed (T03) and a durable consume binding served
(`SECRETS-WP-0007-T04` / `SECRETS-WP-0008-T02`). This repo does not claim the
OpenBao side effect.

2026-09-06 follow-up: secrets-engine reports its PIP join is implemented and
blocked on deployment, not contract (inbox `61ae1174`). Review of its
`validate_action_authorization` shows a real envelope divergence: it expects a
`state-hub`-authority `ActionAuthorization` (`id`, `status`, `superseded_by`,
`request`, `approvals.entries`, policy pin) while this engine serves the
governed approval-claim (`approval_id`, `state`/`valid_now`, `binding`,
`freshness`, `reason_code`, `issuer: approval-engine`). Both declare
`schema_version` `0.1`, so the mismatch surfaces as a field/authority error
rather than a version error. Recorded in `docs/approval-consumption.md`;
reconciling the envelopes is a `GH-DEC-2026-003` cross-repo change, not a
unilateral edit here. T05 stays `wait`: still no deployed base URL (T03).

2026-09-06 ruling: the claim-envelope question is settled. `GH-DEC-2026-005`
confirms all three requested dispositions — the approval-claim is the step-1
artifact, `ActionAuthorization` is not required and MUST NOT be served from the
claim endpoint, and a PEP validates across the claim and the step-2
`DecisionEnvelope`. Gate House recorded the split as doctrine (a PIP must not
republish the PDP's decision) and struck the `provenance.authority ==
"state-hub"` requirement explicitly. flex-auth accepted as `FLEX-DEC-2026-006`,
argued against its own proposal, and traced the bad authority constant to a
fixture rather than prose. **This engine changes nothing: the claim schema
stands as published.** `APPROVAL-IN-0002` is closed. T05 remains `wait` on T03
deployment plus the secrets-engine validator split and its
`secrets-engine-approval` KeyCape registration (already requested verbatim in
`docs/keycape-service-registrations.md`).

2026-09-06 follow-on (T01): `GH-DEC-2026-005` assigned this engine a
compensating obligation. secrets-engine has implemented the split and reports
it no longer verifies the distinct-approver threshold independently; Gate House
accepted that as correct on layering and as a real reduction in defence in
depth, requiring instead that the threshold evaluation be reconstructable from
this engine's state transitions and its use outbox row under §9.6. Implemented:
`approval.issuance` and `approval.use` now carry a `threshold` object
(`required_count`, `distinct_approver_count`, `threshold_met`, and `approvers`
with `approved_at` plus assurance/evidence refs). Tests prove reconstruction
from the use row alone and that the claim still discloses no approver
identities. Documented in `docs/outbox-contract.md`.

2026-09-06 follow-on (T05): flex-auth asked whether this engine should publish
an action/target vocabulary mapping between the claim binding and their policy
package before `SECRETS-WP-0007-T04` makes destroy reachable. Answered no, and
recorded why in `docs/approval-claim.md`: a PIP asserting that one vocabulary's
action *means* another's would author policy semantics it does not own, and a
wrong mapping silently accepts a claim approved for a different action.
`binding.pdp_digest` is the correspondence — it compares the PDP's digest to
the PDP's digest with no translation. Implemented the part that was ours:
`pdp_digest` is now always present on the claim and `null` when the approval
was not issued against a PDP decision, so absence is a stated fact rather than
a missing key, and the schema requires it as nullable. A PEP on a privileged
lane must refuse a null. Both published examples were contradicting the schema;
fixed, and `tests/test_examples.py` now validates every example against it.

2026-09-06 follow-on (T01/T05): implemented `GH-DEC-2026-008`, which requires
`binding.pdp_digest` on the `GH-DEC-2026-003` path and directed this engine to
record the PDP digest at issue and have the claim state which approvals those
are. Schema v3 adds `approvals.pdp_path`; `create` refuses `pdp_path: true`
without a `pdp_digest`, so the failure lands at issue rather than at the
protected side effect. The claim exposes `binding.pdp_path`, making
`pdp_path: true` a guarantee that `pdp_digest` is non-null. Intent is declared,
never inferred from an incidental digest, and never back-filled: legacy rows
migrate to `false` and a successor inherits its predecessor's declaration.
Schema, both examples, and a v2→v3 migration test cover it (102 tests).

2026-09-06 follow-up (envelope target): flex-auth published
`binding.approval_binding_digest` (`FLEX-DEC-2026-007`) — the request digest
material with `context.approval` removed — because a claim-bearing request's
`request_digest` covers the carried claim and can therefore never equal a
`pdp_digest` recorded at issue. A consumer obeying `GH-DEC-2026-008` against
`request_digest` would have failed closed permanently on every claim, which is
the defect secrets-engine and flex-auth both hit. The value this engine records
at issue is unchanged and correct; only the consumer-side comparison target
needed naming. `schemas/approval_claim.schema.json` and `docs/approval-claim.md`
now name `approval_binding_digest` as the execute-time target and state that
`request_digest` is never it. No code change was required, so T05 stays `wait`
on the deployed base URL (T03).

## Production preflight — 2026-09-06 Glas deployment session

User authorized production deployment. Live cluster inspection confirms no
approval-engine workload/service. KeyCape exists at service keycape in sso;
live issuer is https://kc.coulomb.social. Corrected those two stale deployment
inputs. Neither secrets-engine-approval nor approval-engine-operator appears
in the live KeyCape client configuration. The audit sender-scope ConfigMap
currently registers only user-engine. No credential values were emitted.

T03 remains wait: provision the two KeyCape registrations and protected client
credentials, register/custody the approval-engine audit sender, align the
store/client tenant contract, then build/scan and pin the release image. The
manifest currently says platform while requested KeyCape registrations say
tenant:coulomb (and the secrets-engine policy says tenant:platform); exact
claim/store comparison requires an owner-consistent choice before activation.
Do not substitute guessed values or start production without audit delivery.
The secrets-engine PDP was independently deployed by FLEX-WP-0021-T04; this
does not satisfy approval service readiness.