tenant-engine/workplans/TEN-WP-0007-guardrail-production-rollout.md

id	type	title	domain	repo	status	owner	topic_slug	created	updated	depends_on	unblocks	state_hub_workstream_id
TEN-WP-0007	workplan	Production rollout of the guardrail surface	infotech	tenant-engine	finished	claude	tenant-guardrails	2026-08-16	2026-08-16	TEN-WP-0006		9c4d1bb8-47a4-47ff-841f-e8f2ddd01b59

TEN-WP-0007 - Guardrail production rollout

TEN-WP-0006 deliberately left production rollout out of scope. The operator has now authorized proceeding, so this workplan carries the deploy that TEN-WP-0006 would not.

Sequencing (agreed with flex-auth, FLEX-WP-0014)

1. flex-auth first — CI image from the nine-action commit, pin flex-auth-tenant-engine.
2. tenant-engine second — pin the guardrail image.
3. Joint live verification against a disposable tenant.

flex-auth goes first on purpose. If we pin first, the guardrail surface is live but every call denies with write_denied — a state indistinguishable from a broken deployment to anyone reading logs. Their step landing first makes each step independently observable.

What is already true

• Source revision e5c1061 on main, TEN-WP-0006 finished, 231 tests passing.
• flex-auth's nine-action package is verified in source (17/17 Rego tests, 23/23 fixtures, end-to-end against our unmodified FlexAuthWriteAuthorizer).
• Production flex-auth serves the seven-action image @sha256:9320df39… — the four-action regression is resolved, but the guardrail actions are not in that image.
• Production tenant-engine still runs the TEN-WP-0005 lifecycle image.

T01 - Confirm the CI image and its digest

id: TEN-WP-0007-T01
status: done
priority: high
state_hub_task_id: "2f977044-7de1-4dba-aba6-d8a4da6cdcac"

src/** is a trigger path in .forgejo/workflows/image.yaml and the guardrail commits are pushed, so CI should have built an image for main-e5c1061. Confirm it exists and record the immutable digest.

Blocked from this workstation: no registry credentials. Needs a shell with forge access, or the digest relayed by whoever has one.

Done when the digest is recorded here and the image is confirmed to contain the guardrail routes (GET /openapi.json on the built image lists /tenants/{tenant_id}/guardrails).

Done 2026-08-16. The forge API turned out to be readable without credentials, so this was not blocked after all.

Fact	Value
Immutable image	forgejo.coulomb.social/coulomb/tenant-engine@sha256:44ca65f3cdd5967b0124e16b6aba10bf1ac2747e1dde96cbef3dd11ae7cc9574
CI tag	main-955fe33 (workflow run #2, success)
Built from	955fe33 — "Finish TEN-WP-0006: guardrail and quota policy"
API version	0.1.0 (unchanged — additive routes only)

Why 955fe33 and not main HEAD. Three commits landed after it (e5c1061, 9f1e58a, 4d29b8c), so the newest commit is not the newest image. None of the three touches a CI trigger path — they are WORK-RECORDS.md, a workplan file, and .custodian-brief.md — so no build was triggered and none was needed. 955fe33 is the last commit containing source, and therefore the correct image. Checked rather than assumed, because "latest commit" and "latest image" diverging silently is exactly how a stale artifact ships.

latest resolves to the same digest, which corroborates it.

Verified by running the pulled image locally, not by trusting the tag:

/health                                     {"status":"ok","version":"0.1.0"}
/tenants/{tenant_id}/guardrails             GET
/tenants/{tenant_id}/guardrails/{limit_key} PUT, DELETE

12 paths total: the two guardrail paths plus all ten pre-existing ones (create, role read/grant/revoke, roles/live, plan, PATCH, retire, reactivate, health) — so the additive claim holds against the artifact, not just the diff.

Note for future runs: the image listens on 8090, not 8000.

T02 - Wait for the flex-auth nine-action pin

id: TEN-WP-0007-T02
status: done
priority: high
state_hub_task_id: "a5e6f0e5-0a77-4075-b0f6-8ed9bd3b5e2c"

Waiting on FLEX-WP-0014 step 1. Two things to collect when it lands:

• the new flex-auth image digest;
• confirmation that the seven pre-existing actions still allow after the pin. The last regression surfaced as a silent rollback, and lifecycle authority is what user-engine is already shipping against — a second silent rollback would break a live consumer, not just this workplan.

Done when guardrail checks return allow instead of deny unknown_action against the live decision surface.

Done 2026-08-16 by flex-auth (FLEX-WP-0014 step 1).

Fact	Value
flex-auth image	@sha256:1bf060e61122693ce98359c167cc5fe8bdafc84e097e090eaa71af94d0f27cbc
CI tag	main-f304688
Package	tenant-engine.write-api.mutate v1, nine actions
Approval	8d1efcea-93c7-436b-a749-e5d464e94a33
Rollback target	@sha256:9320df39… (seven actions)

Live /v1/check after their pin:

tenant.guardrail.read  actor=flex-auth      allow   decision:9b9e540de13cf7cf
tenant.guardrail.set   actor=tenant-engine  allow   decision:20c82bc38501a343
tenant.guardrail.set   actor=flex-auth      deny action_not_granted
misspelled guardrail                        deny unknown_action
unknown subject                             deny unknown_subject
tenant.create / tenant.retire               allow   (still)

Both asks answered: the seven pre-existing actions still allow, so the silent-rollback failure mode that hit TEN-WP-0005 did not recur, and flex-auth-user-engine was not moved. The read/write split is real, not nominal — flex-auth itself is denied the write.

T03 - Apply the migration and pin the tenant-engine image

id: TEN-WP-0007-T03
status: done
priority: high
state_hub_task_id: "ea509a74-2030-4484-a6e0-f4543a8d394f"

The guardrail schema change is two new tables created by the same CREATE TABLE IF NOT EXISTS script fresh and existing databases both take, so the migration is forward-only and idempotent and needs no separate step. What it does need is evidence that the existing SQLite database on the PVC survives it with tenants, grants, and plans untouched.

Rollback is kubectl -n tenant-engine rollout undo; last-known-good is the TEN-WP-0005 lifecycle digest @sha256:08be0b1dcdc65575592b7be665c28e09a82316ea3d4c9b551ccb753f25360612. Rolling back is safe in one direction only: the new tables are additive and the old image ignores them, but any override written while the new image is live stops being enforced on rollback. Nothing reads guardrails yet, so today that is inert — record it before that stops being true.

Blocked from this workstation: kubectl returns Unauthorized. This is an operator step.

Done when the deployment is pinned to the T01 digest and the pre-existing tenants are intact.

Status 2026-08-16: repo side done, cluster apply outstanding. deploy/tenant-engine.yaml now pins @sha256:44ca65f3cdd5967b0124e16b6aba10bf1ac2747e1dde96cbef3dd11ae7cc9574 (was the TEN-WP-0005 lifecycle digest 08be0b1d…). The manifest already targets flex-auth-tenant-engine.flex-auth.svc.cluster.local:8080, which is the deployment flex-auth just re-pinned to nine actions, so no env change is needed.

Remaining, and it needs an operator with cluster credentials:

kubectl -n tenant-engine apply -f deploy/tenant-engine.yaml
kubectl -n tenant-engine rollout status deploy/tenant-engine
# confirm the pre-existing tenant survived the forward-only migration
kubectl -n tenant-engine exec deploy/tenant-engine -- \
  curl -s localhost:8090/tenants/tenant:trial:portalcheck

kubectl returns Unauthorized from this workstation, so I have not applied it and cannot verify the live result.

Done 2026-08-16. flex-auth attended the apply as operator (explicitly not as an ownership claim). Deployment is 1/1 on @sha256:44ca65f3…, /health returns 0.1.0.

Correction to the rollback target recorded above. The image actually replaced was @sha256:2249e8c6…85149207 — the TEN-WP-0004 image — not the TEN-WP-0005 lifecycle digest 08be0b1d… this task assumed. Production had been sitting on a pre-lifecycle image, so the lifecycle routes verified live on 2026-08-13 were not being served. That is a second silent rollback, parallel to the flex-auth four-action one, and nobody had noticed because the two failures looked identical from outside: user-engine's portal would have seen 404 on the lifecycle routes rather than the 403 a policy denial produces. This rollout fixed both surfaces at once, since the new image carries lifecycle and guardrail routes together.

So the real last-known-good ladder is: 44ca65f3… (current, lifecycle + guardrails) → 08be0b1d… (lifecycle only) → 2249e8c6… (neither). Rolling back past 08be0b1d… removes lifecycle authority that user-engine depends on.

Pre-existing tenants survived the forward-only migration:

• tenant:trial:portalcheck — active, version 1, untouched
• tenant:trial:ten-wp-0005-t05 — retired, version 5, rename and lifecycle history intact

T04 - Live verification and consumer confirmation

id: TEN-WP-0007-T04
status: done
priority: high
state_hub_task_id: "6e4961e2-25e4-43c4-9f39-113610306fe9"

Against a disposable tenant, in the TEN-WP-0005-T05 evidence style:

• guardrail read returns the grouping default with provenance grouping;
• a trial tenant reads a zero spend ceiling — the ADR-0013 mandate, and the one thing here that is enforceable with no meter;
• override set → read reflects it with provenance override;
• override cleared → falls back to the grouping default;
• actor=ops denied on both read and write;
• flex-auth allowed on read, denied on write (action_not_granted);
• retired tenant still reads, clamps to floor, loosening refused;
• idempotent replay returns the original result unchanged.

Then confirm to flex-auth from our side — they verified the decision surface, not our endpoints in production, and TEN-WP-0005 established that those are different claims.

Done when live evidence is recorded here and the confirmation is sent.

Done 2026-08-16. Live against production, disposable tenants tenant:small:ten-wp-0007-t04 (left retired) and tenant:trial:ten-wp-0007-t04 (left active).

Check	Result
grouping default, actor=flex-auth	200 — 25000 EUR P1M, provenance grouping
trial reads zero spend	200 — amount 0, provenance grouping
read actor=ops	403
write actor=flex-auth	403 — read/write split holds live
write actor=ops	403
set override actor=tenant-engine	200 — 9000, provenance override, ETag "2"
idempotent replay, same key	Idempotent-Replay: true, version still 2, same change_id
clear override	200 — falls back to 25000, provenance grouping
read while retired	200 — amount 0, provenance lifecycle
loosen while retired	409 guardrail_loosening_denied
tighten while retired	200 — reduce-privilege stays available
unregistered key	404 unknown_limit_key

The ADR-0013 mandate is now enforced in production. A trial tenant reads a zero spend ceiling; before this rollout it had no ceiling at all. That is the one guardrail that needs no meter to be meaningful, which is why it lands first.

Two things this run incidentally proved beyond the guardrail surface: the lifecycle routes are serving again (the retire above returned 200, which the pre-rollout image could not have done), and the change_id derivation holds across a real replay rather than only in tests.

Out of scope

• Metering. Still unowned by any repo. The guardrail read stays ceiling-only until canon names a meter.
• The plan-derived layer. adaptive-pricing exposes no plan-derived ceilings; the precedence layer exists and is tested but has no feed.
• Grouping ceiling values. The non-trial defaults are conservative opening numbers pending product sign-off. Rolling out does not ratify them — they are config, changeable without a deploy.