--- id: FLEX-WP-0013 type: workplan title: "Restore the seven-action tenant-engine policy pin" domain: infotech repo: flex-auth status: finished owner: grok topic_slug: netkingdom planning_priority: P1 planning_order: 130 depends_on_workplans: - FLEX-WP-0010 - FLEX-WP-0011 related_workplans: - TEN-WP-0005 - TEN-WP-0006 created: "2026-08-16" updated: "2026-08-16" state_hub_workstream_id: "41df7845-a144-420d-9852-adabde0949b3" --- # FLEX-WP-0013 - Restore the seven-action tenant-engine policy pin `flex-auth-tenant-engine` on railiance01 is serving the pre-FLEX-WP-0010 four-action image. `tenant.update` / `tenant.retire` / `tenant.reactivate` deny `unknown_action`. That undoes the TEN-WP-0005-T05 production authority tenant-engine confirmed on 2026-08-13. This is a pin restore, not a policy change. The seven-action package is already in source (`FLEX-WP-0010`) and already exists as image `sha256:9320df394a642eff24da8af4a0ee8886a7bb78b0f14d8ee1deeb30ea8eeeaba7` (CI-built from `e9911eb`). That image was live 2026-08-11 through at least 2026-08-13; its ReplicaSet is still present at desired=0. Do **not** add the TEN-WP-0006 guardrail actions here. Those land on a package that is behind; extending it first would compound the regression. ## Why production is on four actions | Date | What happened | |---|---| | 2026-08-11 | `3dc6223` pinned and applied `9320df39` (seven actions). tenant-engine confirmed live lifecycle writes. | | 2026-08-16 | Live probe during `FLEX-WP-0011` found the Deployment back on `c25fc34a`. Revision 4 reused the 2026-08-08 four-action ReplicaSet. | | 2026-08-16 | Overlay commit `1d58f13` rewrote `deploy/flex-auth-tenant-engine.yaml` and `values/tenant-engine.yaml` to match the rolled-back live pin, so the files now *encode* the regression. | `FLEX-WP-0011` explicitly did not re-promote. tenant-engine left the restore as the operator's call (`5572d198`). This workplan is that call. ## Candidate and rollback | Role | Digest | Policy | |---|---|---| | Candidate | `sha256:9320df394a642eff24da8af4a0ee8886a7bb78b0f14d8ee1deeb30ea8eeeaba7` | seven-action, previously live | | Rollback | `sha256:c25fc34a6cd7e64d955f8723ec70e176a583d5ae71d76280c4e2d89fba0fe0aa` | four-action, live at start of this workplan | Restore the known-good digest rather than building a new image. The tenant-engine policy has not changed since `e9911eb`. A new image would re-bake unrelated later packages and would not be the artifact tenant-engine already verified. Apply via the emergency kubectl path. Production has no Helm release; `helm upgrade --install` would adopt the live objects and is out of scope. Do not move `flex-auth-user-engine`. ## T01 - Re-pin the overlay and emergency manifests ```task id: FLEX-WP-0013-T01 status: done priority: high state_hub_task_id: "22db4198-b9c3-4ebf-975a-5c44c6a75928" ``` Set the tenant-engine digest to `9320df39` in: - `values/tenant-engine.yaml` (overlay pin; leaving this on `c25fc34a` would re-encode the regression and invite the next overlay apply to undo T02) - `deploy/flex-auth-tenant-engine.yaml` (emergency kubectl path, what production actually applies) Update the last-known-good tables in `deploy/README.md` and `railiance/README.md`. Leave `flex-auth-user-engine` untouched. Done when both pin files name the seven-action digest and the runbooks list `c25fc34a` as the rollback target. Done 2026-08-16: `values/tenant-engine.yaml` and `deploy/flex-auth-tenant-engine.yaml` pin `9320df39`. Runbook tables list that digest as live and `c25fc34a` as rollback. ## T02 - Apply and prove the seven actions live ```task id: FLEX-WP-0013-T02 status: done priority: high state_hub_task_id: "e5e76f15-369a-4573-8095-46ba05cd6b14" ``` `kubectl apply -f deploy/flex-auth-tenant-engine.yaml` against railiance01, then `rollout status`. Prove the policy, not just that the pod started. Required live `/v1/check` outcomes against the Service: | Action | Expected | |---|---| | `tenant.create` | allow `write_api_policy_matched` | | `tenant.update` | allow `write_api_policy_matched` | | `tenant.retire` | allow `write_api_policy_matched` | | `tenant.reactivate` | allow `write_api_policy_matched` | | misspelled lifecycle action | deny `unknown_action` | | unknown subject | deny `unknown_subject` | Confirm `flex-auth-user-engine` is still on `sha256:1f5290376dc5fcf456dc7a785e394d8b90949dabecd1d3e856f38557149bb5f4`. Done when the three lifecycle actions allow from the live Service and the user-engine pin is unchanged. Done 2026-08-16 on railiance01. `kubectl apply -f deploy/flex-auth-tenant-engine.yaml` reused ReplicaSet `flex-auth-tenant-engine-56d5bbd6bf` (desired 0 → 1). Approval `f3fbe4cf-efb2-4e4a-99ca-a83050dac6ed`. Decision `c8e39a19`. | Action | Subject | Effect | Decision | |---|---|---|---| | `tenant.create` | tenant-engine | allow `write_api_policy_matched` | `decision:2a94b9ee0dcb4050` | | `tenant.role.grant` | tenant-engine | allow `write_api_policy_matched` | `decision:845d6aa2d5cf6ae7` | | `tenant.role.revoke` | tenant-engine | allow `write_api_policy_matched` | `decision:bd0e331d9443b0bf` | | `tenant.plan.assign` | tenant-engine | allow `write_api_policy_matched` | `decision:b900c1e96b5534ad` | | `tenant.update` | tenant-engine | allow `write_api_policy_matched` | `decision:422fe875467610fb` | | `tenant.retire` | tenant-engine | allow `write_api_policy_matched` | `decision:799301d3443fd6a4` | | `tenant.reactivate` | tenant-engine | allow `write_api_policy_matched` | `decision:4160478ca5499457` | | `tenant.retired` | tenant-engine | deny `unknown_action` | `decision:30946c43cbe80bfc` | | `tenant.retire` | ops | deny `unknown_subject` | `decision:7b107e73cf16fab9` | `flex-auth-user-engine` remained `sha256:1f5290376dc5fcf456dc7a785e394d8b90949dabecd1d3e856f38557149bb5f4`. ## T03 - Handoff and close ```task id: FLEX-WP-0013-T03 status: done priority: medium state_hub_task_id: "df616e44-cfea-4811-a986-90ea0f72b68b" ``` Notify `tenant-engine` that TEN-WP-0005-T05 authority is restored, naming digest `9320df39` and the live decision ids. State plainly that TEN-WP-0006 guardrail actions (`tenant.guardrail.read` / `tenant.guardrail.set`) are **not** in this image and still need a separate policy revision. Update `examples/tenant-engine/README.md` so it no longer says the lifecycle actions are not in production. Done when the handoff is sent and this workplan is `finished`. Done 2026-08-16: handoff sent to `tenant-engine`. TEN-WP-0005-T05 authority is restored on digest `9320df39`. TEN-WP-0006 guardrail actions are not in this image.