--- id: TEN-WP-0007 type: workplan title: "Production rollout of the guardrail surface" domain: infotech repo: tenant-engine status: ready owner: claude topic_slug: tenant-guardrails created: "2026-08-16" updated: "2026-08-16" depends_on: - TEN-WP-0006 unblocks: [] state_hub_workstream_id: "9c4d1bb8-47a4-47ff-841f-e8f2ddd01b59" --- # TEN-WP-0007 - Guardrail production rollout TEN-WP-0006 deliberately left production rollout out of scope. The operator has now authorized proceeding, so this workplan carries the deploy that TEN-WP-0006 would not. ## Sequencing (agreed with flex-auth, FLEX-WP-0014) 1. **flex-auth first** — CI image from the nine-action commit, pin `flex-auth-tenant-engine`. 2. **tenant-engine second** — pin the guardrail image. 3. **Joint live verification** against a disposable tenant. flex-auth goes first on purpose. If we pin first, the guardrail surface is live but every call denies with `write_denied` — a state indistinguishable from a broken deployment to anyone reading logs. Their step landing first makes each step independently observable. ## What is already true - Source revision `e5c1061` on `main`, TEN-WP-0006 finished, 231 tests passing. - flex-auth's nine-action package is verified **in source** (17/17 Rego tests, 23/23 fixtures, end-to-end against our unmodified `FlexAuthWriteAuthorizer`). - Production flex-auth serves the **seven**-action image `@sha256:9320df39…` — the four-action regression is resolved, but the guardrail actions are not in that image. - Production tenant-engine still runs the TEN-WP-0005 lifecycle image. ## T01 - Confirm the CI image and its digest ```task id: TEN-WP-0007-T01 status: done priority: high state_hub_task_id: "2f977044-7de1-4dba-aba6-d8a4da6cdcac" ``` `src/**` is a trigger path in `.forgejo/workflows/image.yaml` and the guardrail commits are pushed, so CI should have built an image for `main-e5c1061`. Confirm it exists and record the immutable digest. Blocked from this workstation: no registry credentials. Needs a shell with forge access, or the digest relayed by whoever has one. Done when the digest is recorded here and the image is confirmed to contain the guardrail routes (`GET /openapi.json` on the built image lists `/tenants/{tenant_id}/guardrails`). Done 2026-08-16. The forge API turned out to be readable without credentials, so this was not blocked after all. | Fact | Value | | --- | --- | | Immutable image | `forgejo.coulomb.social/coulomb/tenant-engine@sha256:44ca65f3cdd5967b0124e16b6aba10bf1ac2747e1dde96cbef3dd11ae7cc9574` | | CI tag | `main-955fe33` (workflow run #2, `success`) | | Built from | `955fe33` — "Finish TEN-WP-0006: guardrail and quota policy" | | API version | `0.1.0` (unchanged — additive routes only) | **Why `955fe33` and not `main` HEAD.** Three commits landed after it (`e5c1061`, `9f1e58a`, `4d29b8c`), so the newest commit is *not* the newest image. None of the three touches a CI trigger path — they are `WORK-RECORDS.md`, a workplan file, and `.custodian-brief.md` — so no build was triggered and none was needed. `955fe33` is the last commit containing source, and therefore the correct image. Checked rather than assumed, because "latest commit" and "latest image" diverging silently is exactly how a stale artifact ships. `latest` resolves to the same digest, which corroborates it. Verified by running the pulled image locally, not by trusting the tag: ``` /health {"status":"ok","version":"0.1.0"} /tenants/{tenant_id}/guardrails GET /tenants/{tenant_id}/guardrails/{limit_key} PUT, DELETE ``` 12 paths total: the two guardrail paths plus all ten pre-existing ones (create, role read/grant/revoke, roles/live, plan, PATCH, retire, reactivate, health) — so the additive claim holds against the artifact, not just the diff. Note for future runs: the image listens on **8090**, not 8000. ## T02 - Wait for the flex-auth nine-action pin ```task id: TEN-WP-0007-T02 status: done priority: high state_hub_task_id: "a5e6f0e5-0a77-4075-b0f6-8ed9bd3b5e2c" ``` Waiting on FLEX-WP-0014 step 1. Two things to collect when it lands: - the new flex-auth image digest; - confirmation that the **seven pre-existing actions still allow** after the pin. The last regression surfaced as a silent rollback, and lifecycle authority is what user-engine is already shipping against — a second silent rollback would break a live consumer, not just this workplan. Done when guardrail checks return allow instead of `deny unknown_action` against the live decision surface. Done 2026-08-16 by flex-auth (FLEX-WP-0014 step 1). | Fact | Value | | --- | --- | | flex-auth image | `@sha256:1bf060e61122693ce98359c167cc5fe8bdafc84e097e090eaa71af94d0f27cbc` | | CI tag | `main-f304688` | | Package | `tenant-engine.write-api.mutate` v1, **nine** actions | | Approval | `8d1efcea-93c7-436b-a749-e5d464e94a33` | | Rollback target | `@sha256:9320df39…` (seven actions) | Live `/v1/check` after their pin: ``` tenant.guardrail.read actor=flex-auth allow decision:9b9e540de13cf7cf tenant.guardrail.set actor=tenant-engine allow decision:20c82bc38501a343 tenant.guardrail.set actor=flex-auth deny action_not_granted misspelled guardrail deny unknown_action unknown subject deny unknown_subject tenant.create / tenant.retire allow (still) ``` Both asks answered: **the seven pre-existing actions still allow**, so the silent-rollback failure mode that hit TEN-WP-0005 did not recur, and `flex-auth-user-engine` was not moved. The read/write split is real, not nominal — `flex-auth` itself is denied the write. ## T03 - Apply the migration and pin the tenant-engine image ```task id: TEN-WP-0007-T03 status: wait priority: high state_hub_task_id: "ea509a74-2030-4484-a6e0-f4543a8d394f" ``` The guardrail schema change is two new tables created by the same `CREATE TABLE IF NOT EXISTS` script fresh and existing databases both take, so the migration is forward-only and idempotent and needs no separate step. What it does need is evidence that the existing SQLite database on the PVC survives it with tenants, grants, and plans untouched. Rollback is `kubectl -n tenant-engine rollout undo`; last-known-good is the TEN-WP-0005 lifecycle digest `@sha256:08be0b1dcdc65575592b7be665c28e09a82316ea3d4c9b551ccb753f25360612`. Rolling back is safe in one direction only: the new tables are additive and the old image ignores them, but any override written while the new image is live stops being enforced on rollback. Nothing reads guardrails yet, so today that is inert — record it before that stops being true. Blocked from this workstation: `kubectl` returns `Unauthorized`. This is an operator step. Done when the deployment is pinned to the T01 digest and the pre-existing tenants are intact. Status 2026-08-16: **repo side done, cluster apply outstanding.** `deploy/tenant-engine.yaml` now pins `@sha256:44ca65f3cdd5967b0124e16b6aba10bf1ac2747e1dde96cbef3dd11ae7cc9574` (was the TEN-WP-0005 lifecycle digest `08be0b1d…`). The manifest already targets `flex-auth-tenant-engine.flex-auth.svc.cluster.local:8080`, which is the deployment flex-auth just re-pinned to nine actions, so no env change is needed. Remaining, and it needs an operator with cluster credentials: ```bash kubectl -n tenant-engine apply -f deploy/tenant-engine.yaml kubectl -n tenant-engine rollout status deploy/tenant-engine # confirm the pre-existing tenant survived the forward-only migration kubectl -n tenant-engine exec deploy/tenant-engine -- \ curl -s localhost:8090/tenants/tenant:trial:portalcheck ``` `kubectl` returns `Unauthorized` from this workstation, so I have not applied it and cannot verify the live result. ## T04 - Live verification and consumer confirmation ```task id: TEN-WP-0007-T04 status: wait priority: high state_hub_task_id: "6e4961e2-25e4-43c4-9f39-113610306fe9" ``` Against a disposable tenant, in the TEN-WP-0005-T05 evidence style: - guardrail read returns the grouping default with provenance `grouping`; - a `trial` tenant reads a **zero** spend ceiling — the ADR-0013 mandate, and the one thing here that is enforceable with no meter; - override set → read reflects it with provenance `override`; - override cleared → falls back to the grouping default; - `actor=ops` denied on both read and write; - `flex-auth` allowed on read, denied on write (`action_not_granted`); - retired tenant still reads, clamps to floor, loosening refused; - idempotent replay returns the original result unchanged. Then confirm to flex-auth from our side — they verified the decision surface, not our endpoints in production, and TEN-WP-0005 established that those are different claims. Done when live evidence is recorded here and the confirmation is sent. ## Out of scope - **Metering.** Still unowned by any repo. The guardrail read stays ceiling-only until canon names a meter. - **The plan-derived layer.** `adaptive-pricing` exposes no plan-derived ceilings; the precedence layer exists and is tested but has no feed. - **Grouping ceiling values.** The non-`trial` defaults are conservative opening numbers pending product sign-off. Rolling out does not ratify them — they are config, changeable without a deploy.