approval-engine/workplans/APPROVAL-WP-0002-production-readiness-and-consumer-adoption.md

id	type	title	domain	repo	status	owner	topic_slug	created	updated	reviewed_at	reviewed_against_commit	reviewed_note	origin	origin_ref	state_hub_workstream_id
APPROVAL-WP-0002	workplan	Production readiness and consumer adoption	infotech	approval-engine	active	codex	netkingdom	2026-09-01	2026-09-06	2026-09-01	ebce5abb27	Reviewed against approval-engine's finished spine, KeyCape's RS256/JWKS and service-token contract, access-engine caller-auth/binding surface, audit-core's deployed authenticated ingestion plus open AUDIT-WP-0009 approval-source/cadence/reconciliation tasks, and secrets-engine's waiting exact-action/decision-consumption tasks. Repo-owned implementation can proceed; live T03-T05 closure remains evidence-gated on those owners.	residual	APPROVAL-WP-0001	4fa25ad5-f5d0-5592-aa59-085f8ee3edaf

APPROVAL-WP-0002 — Production readiness and consumer adoption

Move the completed first-cut engine spine into an authenticated, durable, observable production service and prove one PEP integration end to end. This is the residual production scope deliberately excluded from APPROVAL-WP-0001.

The workplan is proposed pending review against the deployment estate and the current key-cape, access-engine, audit-core, and secrets-engine contracts.

Review completed 2026-09-01. The plan is active. Production mode will verify KeyCape JWT signatures and exact issuer/audience/scopes; local caller-supplied identity never becomes authenticated evidence. SQLite remains the first production store only as a single-replica StatefulSet with explicit migration, backup, integrity, and restore gates. audit-core delivery can be implemented against its existing authenticated idempotent ingest, while heartbeat findings, count reconciliation, and sender registration remain external gates in AUDIT-WP-0009 T04/T06/T09. The first live PEP proof remains jointly gated on secrets-engine T04/T02 and deployment of this service.

Authenticate lifecycle mutations and approver evidence

id: APPROVAL-WP-0002-T01
status: progress
priority: high
state_hub_task_id: "dc4523f5-e0af-5734-a0ef-07aa7f2b2a27"

Bind create, approval-entry, revoke, supersede, and consume callers to authenticated identities. An API-supplied subject_id, actor, or decision_id is provenance only until independently authenticated. Keep authorization decisions in access-engine and approval doctrine in gate-house.

Acceptance: production startup requires a signature-valid KeyCape JWT verifier; every non-health API route requires an explicit scope; approval entry identity and assurance come only from verified claims; create binds binding.actor to the authenticated subject; consume is restricted to service/agent principals; missing, expired, wrong-issuer, wrong-audience, wrong-scope, or unverifiable tokens fail closed without mutation.

Repository implementation complete 2026-09-02: RS256/JWKS verification, issuer/audience/time/profile validation, exact scopes, store-tenant isolation, verified approver evidence, and a deny-all default are covered by tests.

2026-09-02 follow-up: fail-closed coverage now includes wrong signature, HS256, empty/invalid profile claims, deny-all mutation refusal, human-principal consume rejection, and production CLI refusal of static tokens / missing audit delivery. Requested KeyCape registrations are in docs/keycape-service-registrations.md (aud MUST be the resource server approval-engine, not the OAuth client id). Remains progress until KeyCape owns and proves those audience/client/scope registrations.

2026-09-06 follow-up: a tenant collision in the deployment inputs is now recorded in docs/keycape-service-registrations.md. The manifest serves --tenant platform while the requested client registrations issue tenant: tenant:coulomb, and ApiApplication.identity compares the two with exact string equality before any object lookup — so tokens issued under the current registrations would be denied 403 on every non-health route. flex-auth's tenant:platform CheckRequest subject is a PDP input this engine never reads and cannot participate in the comparison. Denial evidence: tests/test_auth.py::test_wrong_tenant_is_forbidden. The values are left as-is deliberately: resolving it requires an owner statement on whether platform and tenant:coulomb name the same layer, and guessing grants cross-tenant access to the approval store. The registrations doc's stale issuer (https://auth.netkingdom.local) is corrected to the live https://kc.coulomb.social from 06544b0. T01 stays progress.

2026-09-06 resolution: the operator accepted tenant:platform as the platform management/administration/services tenant (landlord zone) — decision 5ed3fb35-eca9-413a-82b9-95171ba85bf6, glas-harness/docs/platform-tenant-decision.md, relayed by glas-harness. The collision above is closed by setting the store tenant to exactly tenant:platform: deploy/approval-engine.yaml --tenant, the approval_engine/cli.py --tenant default, and the Engine(tenant=…) default all move off bare platform, and the requested client registrations now ask for tenant: tenant:platform. Exact JWT/store equality is retained — no alias, no normalisation, no prefix handling, and no implicit cross-tenant grant.

Moving the defaults rather than only the manifest is deliberate: a default of platform under a sanctioned value of tenant:platform is a trap, because a serve that omits --tenant would come up healthy and then refuse every authenticated call.

The default change broke ten tests whose identity fixtures hard-coded platform, which is the fixture-consistency hazard flex-auth reported as FLEX-DEC-2026-008 (29 fixtures all carrying one tenant proved nothing about the field). Fixtures are aligned to the exact spelling, and the field is now varied rather than merely present: tests/test_auth.py::test_near_miss_tenant_spellings_are_forbidden refuses platform, tenant:coulomb, case variants, whitespace variants and empty against a tenant:platform store, and test_exact_sanctioned_tenant_is_admitted pins the other half so a reject-everything bug cannot pass. 111 tests pass.

This resolves the choice of value only. T01 stays progress: KeyCape still has to own and prove these registrations, and credential materialization is unchanged.

Harden durable storage and migrations

id: APPROVAL-WP-0002-T02
status: done
priority: high
state_hub_task_id: "2bef94ca-9482-5eff-9bd9-39adef292a78"

Define the production persistence, backup/restore, migration, concurrency, and recovery posture. Prove schema upgrades preserve existing approvals and that crash recovery cannot separate mutations from outbox evidence.

Acceptance: schema version is explicit; production serve refuses an unmigrated or in-memory store; migration is a separate repeatable command; backup uses SQLite's online backup API and integrity verification; restore is documented as a stopped-single-writer operation; tests cover legacy upgrade, backup/restore, and outbox atomicity after restart.

Completed 2026-09-02: schema v2 migration, production no-auto-migrate gate, integrity/status surface, online mode-0600 backup, stopped-writer restore runbook, retry-attempt state, and migration/backup/atomicity tests are in place.

Package and deploy the service

id: APPROVAL-WP-0002-T03
status: wait
priority: high
state_hub_task_id: "f0aa2e6d-19e6-5b43-886c-efa4e3de5f22"

Add the governed image/deployment surface, health and readiness behavior, resource bounds, and fail-closed caller configuration. A local WSGI development server is not production evidence.

Acceptance: a digest-pin-ready image and single-writer StatefulSet manifest exist with non-root/read-only-root controls, PVC, migration init container, resource bounds, probes, and default-deny network policy. Live completion also requires an immutable image digest, KeyCape registrations, audit sender credential, successful rollout, and restart/restore evidence.

Repository implementation complete 2026-09-02: the digest-pin-ready non-root image builds and runs; the single-writer StatefulSet, PVC, migration init, read-only root, resources, probes, and default-deny policies pass client dry-run. Waiting on release digest, KeyCape/audit registrations and credentials, rollout, restart, and restore evidence.

2026-09-06 image preparation (GLAS-WP-0015 request; credential-independent half): built and validated locally against the agreed tenant and current schema.

• Build: make image-build off the digest-pinned base python:3.12-slim@sha256:d764629c…. Local manifest-list digest sha256:85e46ddf47b3ac0cdab620163a7b034027451dff286b63ecff6a2493d7a57ecc. This is a local build digest, not a release digest — the manifest still carries REPLACE_WITH_RELEASE_DIGEST because pinning requires a push to forgejo.coulomb.social, which needs registry credentials this session does not hold.
• Runtime identity: uid=10001(approval) gid=10001(approval), non-root as required.
• Schema: image carries LATEST_SCHEMA_VERSION = 3, matching the migrated store, and Engine tenant default tenant:platform.
• First-install migration (no prior DB): migrate then verify on a fresh volume both report schema_version: 3, schema_current: true, integrity: ["ok"], foreign_key_violations: 0, persistent: true.
• Fail-closed configuration proven in the image, not only in tests: serve --production --db :memory: refuses with "production requires a persistent database"; serve --production on a real DB without audit configuration refuses with "production requires authenticated audit delivery".
• Manifest inputs: kubectl apply --dry-run=client passes for the namespace, service, and StatefulSet with the new --tenant tenant:platform.

Scan gate NOT met: no scanner (trivy, grype, docker scout) is installed on this workstation, so no vulnerability scan was run and none is claimed. The inventory a scanner needs is the pinned base above plus cryptography==50.0.1, PyJWT==2.13.0, waitress==3.0.2, cffi==2.1.1, pycparser==3.0, pip==25.0.1.

T03 stays wait: still no release digest, no KeyCape/audit credentials, no rollout, and no restart/restore evidence. Nothing here is a deploy.

2026-09-06 scan gate — run, and it failed on the sanctioned base. Full record in docs/image-scan-2026-09-06.md. Scanned with trivy (no scanner was installed; this one needed none). Findings are all inherited from the base image, none in approval_engine code.

Two fixes applied. The base pin was stale at Debian 13.5 while upstream is 13.6 — bumping the digest removed 30 HIGH and 47 MEDIUM. And pip, which held all 10 MEDIUM Python findings, is now absent from the runtime image via a two-stage build; it is a build-time tool and had no business in a running approval service.

What remains is a decision, not a task. Three CRITICALs persist on Debian, all perl-base (CVE-2026-13221, CVE-2026-42496, CVE-2026-8376), none with an upstream fix, in a package this service never invokes and which Debian marks Essential: yes. An Alpine variant carries no perl and scans 0 CRITICAL / 7 HIGH / 1 MEDIUM against Debian's 3 / 51 / 56. It is proven viable — musl wheels resolve without a toolchain, the full suite passes on musl at 111, and non-root identity, schema v3, tenant:platform, fresh-store migrate/verify and both production fail-closed gates all hold in the image. It is parked in Containerfile.alpine as a candidate; Containerfile remains the sanctioned base.

Nothing was pushed. Pushing the Debian variant would pin three unfixable CRITICALs into a release digest, and choosing the runtime C library for this service is not a call to make silently. deploy/approval-engine.yaml still carries REPLACE_WITH_RELEASE_DIGEST. Awaiting an owner answer on the base (Alpine, a documented perl exception on glibc, or distroless — unevaluated) and on which scanner is sanctioned.

2026-09-06 both answered; release candidate scans clean. The operator adopted Alpine/musl as the sanctioned base and trivy as the sanctioned scanner. Containerfile.alpine is promoted to Containerfile and the Debian variant is retired rather than kept as an option.

Promoting Alpine left 7 HIGH and 1 MEDIUM, all libuuid 2.42.1-r0 and all with fixes in 2.42.3. The runtime stage now requires libuuid>=2.42.3-r1 — a version floor rather than a floating upgrade, so the base stays digest-pinned and a vulnerable libuuid fails the build instead of shipping. The candidate now scans 0 CRITICAL / 0 HIGH / 0 MEDIUM / 0 LOW.

Verified on that exact artifact: non-root uid=10001, pip absent, schema v3, tenant default tenant:platform, fresh-store migrate+verify clean, restart persistence via re-verify on the same volume, both production fail-closed refusals, 111 tests on musl, and kubectl --dry-run passing.

The gate is now reproducible rather than a one-off: make image-scan fails on any CRITICAL or HIGH, and make image-release runs build then scan then push so a failing scan blocks the push by construction.

Still wait, and now for exactly one reason: docker push was attempted and refused by this session's sandbox as an outward-facing publish. It was not worked around. No release digest exists, so the manifest keeps REPLACE_WITH_RELEASE_DIGEST; an operator must run make image-release (or the push and docker inspect) and pin the returned registry manifest digest — never the tag, never the local image id sha256:73333f5ceb55e48192e3095cb2e2a741cdc6ff0be2f18128301072b4a6b6eb9d. Rollout, restart and restore evidence remain gated on the KeyCape registrations and audit sender credential.

Wire outbox delivery and reconciliation

id: APPROVAL-WP-0002-T04
status: wait
priority: high
state_hub_task_id: "777a5cf5-c1db-5e07-8de6-ab109db5ccdc"

Deliver the local outbox asynchronously to audit-core, preserve event-id deduplication, publish lag/depth signals, emit the declared heartbeat, and prove the Gate House reconciliation contract against accepted event counts.

Acceptance: the sender adapts the local event to audit-core's authenticated HTTP ingest, reuses event id as idempotency key, rereads a mounted token file, marks drained only on accepted/duplicate, retains retryable failures, exposes attempt/lag state, and emits the declared heartbeat. Live reconciliation waits on AUDIT-WP-0009-T04/T06/T09; do not invent that receiver surface here.

Repository implementation complete 2026-09-02: audit-core envelope adaptation, event-id idempotency, mounted-token reread, accepted/duplicate handling, retry-attempt/lag metrics, and periodic heartbeats are tested.

2026-09-02 follow-up: drain tests now cover HTTP 200 duplicate as drained and urllib HTTPError 503 as pending. Waiting on the audit-core sender registration/ingress and receiver-owned reconciliation work (AUDIT-WP-0009-T04/T06/T09).

Prove one live PEP consumption path

id: APPROVAL-WP-0002-T05
status: wait
priority: high
state_hub_task_id: "3196fb77-3df0-5358-ae12-2d07bac12041"

Integrate one protected-system consumer under GH-DEC-2026-003: claim before decision, CAS consume after ALLOW and before side effect, same-digest retry, different-digest conflict, spent-on-failure behavior, and no protected action when approval-engine is unavailable.

Acceptance: a repeatable harness proves the PEP sequence against the real HTTP surface without performing a protected action; live closure requires a secrets-engine-owned handler and evidence that no OpenBao call occurs on every failure case. This repo may supply the protocol client and fixture, but may not claim the consumer's side effect.

Repository implementation complete 2026-09-02: the HTTP PEP client and fail-closed sequencing harness prove claim-before-decision and CAS-consume-before callback, including unavailable, DENY, digest mismatch, and conflict paths.

2026-09-02 follow-up: secrets-engine shipped the PEP consume-before-OpenBao handler (src/secrets_engine/approval_consume.py; inbox 0e04b4e2). This engine's client now maps HTTP 409/401/403/404/503 and unreachability, requires the canonical digest, and refuses a decision-shaped consume payload. The repeatable harness in tests/test_pep.py drives the real HTTP surface: first consume then same-digest retry, different-digest conflict, spent-on-failure claim refusal, and unreachable-engine callback suppression. Waiting on live closure: this service deployed (T03) and a durable consume binding served (SECRETS-WP-0007-T04 / SECRETS-WP-0008-T02). This repo does not claim the OpenBao side effect.

2026-09-06 follow-up: secrets-engine reports its PIP join is implemented and blocked on deployment, not contract (inbox 61ae1174). Review of its validate_action_authorization shows a real envelope divergence: it expects a state-hub-authority ActionAuthorization (id, status, superseded_by, request, approvals.entries, policy pin) while this engine serves the governed approval-claim (approval_id, state/valid_now, binding, freshness, reason_code, issuer: approval-engine). Both declare schema_version 0.1, so the mismatch surfaces as a field/authority error rather than a version error. Recorded in docs/approval-consumption.md; reconciling the envelopes is a GH-DEC-2026-003 cross-repo change, not a unilateral edit here. T05 stays wait: still no deployed base URL (T03).

2026-09-06 ruling: the claim-envelope question is settled. GH-DEC-2026-005 confirms all three requested dispositions — the approval-claim is the step-1 artifact, ActionAuthorization is not required and MUST NOT be served from the claim endpoint, and a PEP validates across the claim and the step-2 DecisionEnvelope. Gate House recorded the split as doctrine (a PIP must not republish the PDP's decision) and struck the provenance.authority == "state-hub" requirement explicitly. flex-auth accepted as FLEX-DEC-2026-006, argued against its own proposal, and traced the bad authority constant to a fixture rather than prose. This engine changes nothing: the claim schema stands as published. APPROVAL-IN-0002 is closed. T05 remains wait on T03 deployment plus the secrets-engine validator split and its secrets-engine-approval KeyCape registration (already requested verbatim in docs/keycape-service-registrations.md).

2026-09-06 follow-on (T01): GH-DEC-2026-005 assigned this engine a compensating obligation. secrets-engine has implemented the split and reports it no longer verifies the distinct-approver threshold independently; Gate House accepted that as correct on layering and as a real reduction in defence in depth, requiring instead that the threshold evaluation be reconstructable from this engine's state transitions and its use outbox row under §9.6. Implemented: approval.issuance and approval.use now carry a threshold object (required_count, distinct_approver_count, threshold_met, and approvers with approved_at plus assurance/evidence refs). Tests prove reconstruction from the use row alone and that the claim still discloses no approver identities. Documented in docs/outbox-contract.md.

2026-09-06 follow-on (T05): flex-auth asked whether this engine should publish an action/target vocabulary mapping between the claim binding and their policy package before SECRETS-WP-0007-T04 makes destroy reachable. Answered no, and recorded why in docs/approval-claim.md: a PIP asserting that one vocabulary's action means another's would author policy semantics it does not own, and a wrong mapping silently accepts a claim approved for a different action. binding.pdp_digest is the correspondence — it compares the PDP's digest to the PDP's digest with no translation. Implemented the part that was ours: pdp_digest is now always present on the claim and null when the approval was not issued against a PDP decision, so absence is a stated fact rather than a missing key, and the schema requires it as nullable. A PEP on a privileged lane must refuse a null. Both published examples were contradicting the schema; fixed, and tests/test_examples.py now validates every example against it.

2026-09-06 follow-on (T01/T05): implemented GH-DEC-2026-008, which requires binding.pdp_digest on the GH-DEC-2026-003 path and directed this engine to record the PDP digest at issue and have the claim state which approvals those are. Schema v3 adds approvals.pdp_path; create refuses pdp_path: true without a pdp_digest, so the failure lands at issue rather than at the protected side effect. The claim exposes binding.pdp_path, making pdp_path: true a guarantee that pdp_digest is non-null. Intent is declared, never inferred from an incidental digest, and never back-filled: legacy rows migrate to false and a successor inherits its predecessor's declaration. Schema, both examples, and a v2→v3 migration test cover it (102 tests).

2026-09-06 follow-up (envelope target): flex-auth published binding.approval_binding_digest (FLEX-DEC-2026-007) — the request digest material with context.approval removed — because a claim-bearing request's request_digest covers the carried claim and can therefore never equal a pdp_digest recorded at issue. A consumer obeying GH-DEC-2026-008 against request_digest would have failed closed permanently on every claim, which is the defect secrets-engine and flex-auth both hit. The value this engine records at issue is unchanged and correct; only the consumer-side comparison target needed naming. schemas/approval_claim.schema.json and docs/approval-claim.md now name approval_binding_digest as the execute-time target and state that request_digest is never it. No code change was required, so T05 stays wait on the deployed base URL (T03).

Production preflight — 2026-09-06 Glas deployment session

User authorized production deployment. Live cluster inspection confirms no approval-engine workload/service. KeyCape exists at service keycape in sso; live issuer is https://kc.coulomb.social. Corrected those two stale deployment inputs. Neither secrets-engine-approval nor approval-engine-operator appears in the live KeyCape client configuration. The audit sender-scope ConfigMap currently registers only user-engine. No credential values were emitted.

T03 remains wait: provision the two KeyCape registrations and protected client credentials, register/custody the approval-engine audit sender, align the store/client tenant contract, then build/scan and pin the release image. The manifest currently says platform while requested KeyCape registrations say tenant:coulomb (and the secrets-engine policy says tenant:platform); exact claim/store comparison requires an owner-consistent choice before activation. Do not substitute guessed values or start production without audit delivery. The secrets-engine PDP was independently deployed by FLEX-WP-0021-T04; this does not satisfy approval service readiness.