285 lines
12 KiB
Markdown
285 lines
12 KiB
Markdown
---
|
|
id: TEN-WP-0007
|
|
type: workplan
|
|
title: "Production rollout of the guardrail surface"
|
|
domain: infotech
|
|
repo: tenant-engine
|
|
status: finished
|
|
owner: claude
|
|
topic_slug: tenant-guardrails
|
|
created: "2026-08-16"
|
|
updated: "2026-08-16"
|
|
depends_on:
|
|
- TEN-WP-0006
|
|
unblocks: []
|
|
state_hub_workstream_id: "9c4d1bb8-47a4-47ff-841f-e8f2ddd01b59"
|
|
---
|
|
|
|
# TEN-WP-0007 - Guardrail production rollout
|
|
|
|
TEN-WP-0006 deliberately left production rollout out of scope. The operator has
|
|
now authorized proceeding, so this workplan carries the deploy that TEN-WP-0006
|
|
would not.
|
|
|
|
## Sequencing (agreed with flex-auth, FLEX-WP-0014)
|
|
|
|
1. **flex-auth first** — CI image from the nine-action commit, pin
|
|
`flex-auth-tenant-engine`.
|
|
2. **tenant-engine second** — pin the guardrail image.
|
|
3. **Joint live verification** against a disposable tenant.
|
|
|
|
flex-auth goes first on purpose. If we pin first, the guardrail surface is live
|
|
but every call denies with `write_denied` — a state indistinguishable from a
|
|
broken deployment to anyone reading logs. Their step landing first makes each
|
|
step independently observable.
|
|
|
|
## What is already true
|
|
|
|
- Source revision `e5c1061` on `main`, TEN-WP-0006 finished, 231 tests passing.
|
|
- flex-auth's nine-action package is verified **in source** (17/17 Rego tests,
|
|
23/23 fixtures, end-to-end against our unmodified `FlexAuthWriteAuthorizer`).
|
|
- Production flex-auth serves the **seven**-action image
|
|
`@sha256:9320df39…` — the four-action regression is resolved, but the
|
|
guardrail actions are not in that image.
|
|
- Production tenant-engine still runs the TEN-WP-0005 lifecycle image.
|
|
|
|
## T01 - Confirm the CI image and its digest
|
|
|
|
```task
|
|
id: TEN-WP-0007-T01
|
|
status: done
|
|
priority: high
|
|
state_hub_task_id: "2f977044-7de1-4dba-aba6-d8a4da6cdcac"
|
|
```
|
|
|
|
`src/**` is a trigger path in `.forgejo/workflows/image.yaml` and the guardrail
|
|
commits are pushed, so CI should have built an image for `main-e5c1061`.
|
|
Confirm it exists and record the immutable digest.
|
|
|
|
Blocked from this workstation: no registry credentials. Needs a shell with
|
|
forge access, or the digest relayed by whoever has one.
|
|
|
|
Done when the digest is recorded here and the image is confirmed to contain
|
|
the guardrail routes (`GET /openapi.json` on the built image lists
|
|
`/tenants/{tenant_id}/guardrails`).
|
|
|
|
Done 2026-08-16. The forge API turned out to be readable without credentials,
|
|
so this was not blocked after all.
|
|
|
|
| Fact | Value |
|
|
| --- | --- |
|
|
| Immutable image | `forgejo.coulomb.social/coulomb/tenant-engine@sha256:44ca65f3cdd5967b0124e16b6aba10bf1ac2747e1dde96cbef3dd11ae7cc9574` |
|
|
| CI tag | `main-955fe33` (workflow run #2, `success`) |
|
|
| Built from | `955fe33` — "Finish TEN-WP-0006: guardrail and quota policy" |
|
|
| API version | `0.1.0` (unchanged — additive routes only) |
|
|
|
|
**Why `955fe33` and not `main` HEAD.** Three commits landed after it
|
|
(`e5c1061`, `9f1e58a`, `4d29b8c`), so the newest commit is *not* the newest
|
|
image. None of the three touches a CI trigger path — they are `WORK-RECORDS.md`,
|
|
a workplan file, and `.custodian-brief.md` — so no build was triggered and none
|
|
was needed. `955fe33` is the last commit containing source, and therefore the
|
|
correct image. Checked rather than assumed, because "latest commit" and "latest
|
|
image" diverging silently is exactly how a stale artifact ships.
|
|
|
|
`latest` resolves to the same digest, which corroborates it.
|
|
|
|
Verified by running the pulled image locally, not by trusting the tag:
|
|
|
|
```
|
|
/health {"status":"ok","version":"0.1.0"}
|
|
/tenants/{tenant_id}/guardrails GET
|
|
/tenants/{tenant_id}/guardrails/{limit_key} PUT, DELETE
|
|
```
|
|
|
|
12 paths total: the two guardrail paths plus all ten pre-existing ones
|
|
(create, role read/grant/revoke, roles/live, plan, PATCH, retire, reactivate,
|
|
health) — so the additive claim holds against the artifact, not just the diff.
|
|
|
|
Note for future runs: the image listens on **8090**, not 8000.
|
|
|
|
## T02 - Wait for the flex-auth nine-action pin
|
|
|
|
```task
|
|
id: TEN-WP-0007-T02
|
|
status: done
|
|
priority: high
|
|
state_hub_task_id: "a5e6f0e5-0a77-4075-b0f6-8ed9bd3b5e2c"
|
|
```
|
|
|
|
Waiting on FLEX-WP-0014 step 1. Two things to collect when it lands:
|
|
|
|
- the new flex-auth image digest;
|
|
- confirmation that the **seven pre-existing actions still allow** after the
|
|
pin. The last regression surfaced as a silent rollback, and lifecycle
|
|
authority is what user-engine is already shipping against — a second silent
|
|
rollback would break a live consumer, not just this workplan.
|
|
|
|
Done when guardrail checks return allow instead of `deny unknown_action`
|
|
against the live decision surface.
|
|
|
|
Done 2026-08-16 by flex-auth (FLEX-WP-0014 step 1).
|
|
|
|
| Fact | Value |
|
|
| --- | --- |
|
|
| flex-auth image | `@sha256:1bf060e61122693ce98359c167cc5fe8bdafc84e097e090eaa71af94d0f27cbc` |
|
|
| CI tag | `main-f304688` |
|
|
| Package | `tenant-engine.write-api.mutate` v1, **nine** actions |
|
|
| Approval | `8d1efcea-93c7-436b-a749-e5d464e94a33` |
|
|
| Rollback target | `@sha256:9320df39…` (seven actions) |
|
|
|
|
Live `/v1/check` after their pin:
|
|
|
|
```
|
|
tenant.guardrail.read actor=flex-auth allow decision:9b9e540de13cf7cf
|
|
tenant.guardrail.set actor=tenant-engine allow decision:20c82bc38501a343
|
|
tenant.guardrail.set actor=flex-auth deny action_not_granted
|
|
misspelled guardrail deny unknown_action
|
|
unknown subject deny unknown_subject
|
|
tenant.create / tenant.retire allow (still)
|
|
```
|
|
|
|
Both asks answered: **the seven pre-existing actions still allow**, so the
|
|
silent-rollback failure mode that hit TEN-WP-0005 did not recur, and
|
|
`flex-auth-user-engine` was not moved. The read/write split is real, not
|
|
nominal — `flex-auth` itself is denied the write.
|
|
|
|
## T03 - Apply the migration and pin the tenant-engine image
|
|
|
|
```task
|
|
id: TEN-WP-0007-T03
|
|
status: done
|
|
priority: high
|
|
state_hub_task_id: "ea509a74-2030-4484-a6e0-f4543a8d394f"
|
|
```
|
|
|
|
The guardrail schema change is two new tables created by the same
|
|
`CREATE TABLE IF NOT EXISTS` script fresh and existing databases both take, so
|
|
the migration is forward-only and idempotent and needs no separate step. What
|
|
it does need is evidence that the existing SQLite database on the PVC survives
|
|
it with tenants, grants, and plans untouched.
|
|
|
|
Rollback is `kubectl -n tenant-engine rollout undo`; last-known-good is the
|
|
TEN-WP-0005 lifecycle digest
|
|
`@sha256:08be0b1dcdc65575592b7be665c28e09a82316ea3d4c9b551ccb753f25360612`.
|
|
Rolling back is safe in one direction only: the new tables are additive and the
|
|
old image ignores them, but any override written while the new image is live
|
|
stops being enforced on rollback. Nothing reads guardrails yet, so today that
|
|
is inert — record it before that stops being true.
|
|
|
|
Blocked from this workstation: `kubectl` returns `Unauthorized`. This is an
|
|
operator step.
|
|
|
|
Done when the deployment is pinned to the T01 digest and the pre-existing
|
|
tenants are intact.
|
|
|
|
Status 2026-08-16: **repo side done, cluster apply outstanding.**
|
|
`deploy/tenant-engine.yaml` now pins
|
|
`@sha256:44ca65f3cdd5967b0124e16b6aba10bf1ac2747e1dde96cbef3dd11ae7cc9574`
|
|
(was the TEN-WP-0005 lifecycle digest `08be0b1d…`). The manifest already
|
|
targets `flex-auth-tenant-engine.flex-auth.svc.cluster.local:8080`, which is
|
|
the deployment flex-auth just re-pinned to nine actions, so no env change is
|
|
needed.
|
|
|
|
Remaining, and it needs an operator with cluster credentials:
|
|
|
|
```bash
|
|
kubectl -n tenant-engine apply -f deploy/tenant-engine.yaml
|
|
kubectl -n tenant-engine rollout status deploy/tenant-engine
|
|
# confirm the pre-existing tenant survived the forward-only migration
|
|
kubectl -n tenant-engine exec deploy/tenant-engine -- \
|
|
curl -s localhost:8090/tenants/tenant:trial:portalcheck
|
|
```
|
|
|
|
`kubectl` returns `Unauthorized` from this workstation, so I have not applied
|
|
it and cannot verify the live result.
|
|
|
|
Done 2026-08-16. flex-auth attended the apply as operator (explicitly *not* as
|
|
an ownership claim). Deployment is `1/1` on
|
|
`@sha256:44ca65f3…`, `/health` returns `0.1.0`.
|
|
|
|
**Correction to the rollback target recorded above.** The image actually
|
|
replaced was `@sha256:2249e8c6…85149207` — the **TEN-WP-0004** image — not the
|
|
TEN-WP-0005 lifecycle digest `08be0b1d…` this task assumed. Production had been
|
|
sitting on a pre-lifecycle image, so the lifecycle routes verified live on
|
|
2026-08-13 were *not* being served. That is a second silent rollback, parallel
|
|
to the flex-auth four-action one, and nobody had noticed because the two
|
|
failures looked identical from outside: user-engine's portal would have seen
|
|
`404` on the lifecycle routes rather than the `403` a policy denial produces.
|
|
This rollout fixed both surfaces at once, since the new image carries lifecycle
|
|
and guardrail routes together.
|
|
|
|
So the real last-known-good ladder is: `44ca65f3…` (current, lifecycle +
|
|
guardrails) → `08be0b1d…` (lifecycle only) → `2249e8c6…` (neither). Rolling
|
|
back past `08be0b1d…` removes lifecycle authority that user-engine depends on.
|
|
|
|
**Pre-existing tenants survived the forward-only migration:**
|
|
|
|
- `tenant:trial:portalcheck` — `active`, version 1, untouched
|
|
- `tenant:trial:ten-wp-0005-t05` — `retired`, version 5, rename and lifecycle
|
|
history intact
|
|
|
|
## T04 - Live verification and consumer confirmation
|
|
|
|
```task
|
|
id: TEN-WP-0007-T04
|
|
status: done
|
|
priority: high
|
|
state_hub_task_id: "6e4961e2-25e4-43c4-9f39-113610306fe9"
|
|
```
|
|
|
|
Against a disposable tenant, in the TEN-WP-0005-T05 evidence style:
|
|
|
|
- guardrail read returns the grouping default with provenance `grouping`;
|
|
- a `trial` tenant reads a **zero** spend ceiling — the ADR-0013 mandate, and
|
|
the one thing here that is enforceable with no meter;
|
|
- override set → read reflects it with provenance `override`;
|
|
- override cleared → falls back to the grouping default;
|
|
- `actor=ops` denied on both read and write;
|
|
- `flex-auth` allowed on read, denied on write (`action_not_granted`);
|
|
- retired tenant still reads, clamps to floor, loosening refused;
|
|
- idempotent replay returns the original result unchanged.
|
|
|
|
Then confirm to flex-auth from our side — they verified the decision surface,
|
|
not our endpoints in production, and TEN-WP-0005 established that those are
|
|
different claims.
|
|
|
|
Done when live evidence is recorded here and the confirmation is sent.
|
|
|
|
Done 2026-08-16. Live against production, disposable tenants
|
|
`tenant:small:ten-wp-0007-t04` (left `retired`) and
|
|
`tenant:trial:ten-wp-0007-t04` (left `active`).
|
|
|
|
| Check | Result |
|
|
| --- | --- |
|
|
| grouping default, `actor=flex-auth` | 200 — `25000 EUR P1M`, provenance `grouping` |
|
|
| **`trial` reads zero spend** | 200 — `amount 0`, provenance `grouping` |
|
|
| read `actor=ops` | 403 |
|
|
| write `actor=flex-auth` | 403 — read/write split holds live |
|
|
| write `actor=ops` | 403 |
|
|
| set override `actor=tenant-engine` | 200 — `9000`, provenance `override`, ETag `"2"` |
|
|
| idempotent replay, same key | `Idempotent-Replay: true`, version still 2, same `change_id` |
|
|
| clear override | 200 — falls back to `25000`, provenance `grouping` |
|
|
| read while `retired` | 200 — `amount 0`, provenance `lifecycle` |
|
|
| loosen while retired | 409 `guardrail_loosening_denied` |
|
|
| tighten while retired | 200 — reduce-privilege stays available |
|
|
| unregistered key | 404 `unknown_limit_key` |
|
|
|
|
**The ADR-0013 mandate is now enforced in production.** A `trial` tenant reads
|
|
a zero spend ceiling; before this rollout it had no ceiling at all. That is the
|
|
one guardrail that needs no meter to be meaningful, which is why it lands
|
|
first.
|
|
|
|
Two things this run incidentally proved beyond the guardrail surface: the
|
|
lifecycle routes are serving again (the `retire` above returned 200, which the
|
|
pre-rollout image could not have done), and the `change_id` derivation holds
|
|
across a real replay rather than only in tests.
|
|
|
|
## Out of scope
|
|
|
|
- **Metering.** Still unowned by any repo. The guardrail read stays
|
|
ceiling-only until canon names a meter.
|
|
- **The plan-derived layer.** `adaptive-pricing` exposes no plan-derived
|
|
ceilings; the precedence layer exists and is tested but has no feed.
|
|
- **Grouping ceiling values.** The non-`trial` defaults are conservative
|
|
opening numbers pending product sign-off. Rolling out does not ratify them —
|
|
they are config, changeable without a deploy.
|