tenant-engine/workplans/TEN-WP-0006-guardrail-quota-policy.md
tegwick 955fe339fd
All checks were successful
Build and Publish Container Image / build-and-push (push) Successful in 43s
Finish TEN-WP-0006: guardrail and quota policy
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-16 02:19:48 +02:00

383 lines
19 KiB
Markdown

---
id: TEN-WP-0006
type: workplan
title: "Guardrail and quota policy for tenants"
domain: infotech
repo: tenant-engine
status: finished
owner: claude
topic_slug: tenant-guardrails
created: "2026-08-16"
updated: "2026-08-16"
depends_on:
- TEN-WP-0005
unblocks: []
state_hub_workstream_id: "bddb6699-cb53-472a-9757-a3ed0eb0ce27"
---
# TEN-WP-0006 - Guardrail and quota policy
Implement the guardrail/quota concern that `SCOPE.md`, `INTENT.md`, and
`.claude/rules/architecture.md` have reserved since the repo was created but
never built. `tenant-engine` owns *policy* — what a tenant is permitted to
spend and how many entities/actions it may hold — not metering, not billing,
and not the enforcement decision itself.
The concrete hole this closes: ADR-0013 specifies that `trial`-grouped tenants
default to a **zero** spend budget. No such default exists today, so every
trial tenant currently carries no ceiling at all.
## Boundary constraints
These are not negotiable without a new ADR, and every task below is bound by
them:
- **`tenant-engine` is a data source, never a PDP.** It answers "what is this
tenant's limit" and "what is its recorded consumption". `flex-auth` decides
whether an action is allowed. Never invert this.
- **No metering.** Consumption counters are written *to* this service by
whoever meters; `tenant-engine` does not observe spend or count actions
itself.
- **No plan terms.** A guardrail may be *derived from* an `adaptive-pricing`
plan id, but plan definitions are never copied here.
- **Fail closed.** An unknown tenant, an unreadable store, or a missing
guardrail resolves to the most restrictive answer, never to "unlimited".
## T01 - Specify the guardrail model and its boundary
```task
id: TEN-WP-0006-T01
status: done
priority: high
state_hub_task_id: "b0693ec3-b3a7-48fb-9f4f-baeedb488e9d"
```
Write the guardrail domain contract before any code. Decide and document:
- The **limit kinds** in scope. Start with spend budget (a currency amount
over a period) and entity/action counts; leave rate limiting out unless it
falls out for free.
- **Precedence.** A tenant may inherit a limit from its grouping (ADR-0013),
from its assigned plan, and from an explicit per-tenant override. Define the
resolution order and make it total — every tenant resolves to exactly one
effective guardrail set, with no ambiguity and no "unset means unlimited".
- **The `trial` = zero-spend default**, and what the default is for each other
grouping. An unmapped grouping must resolve restrictively, not openly.
- **Reserved identifiers.** `tenant:platform` and `tenant:coulomb` are
ungrouped (see `domain.py`); state explicitly what they resolve to.
- Whether consumption lives in this repo at all. If it does, it is an opaque
counter written by an external meter, with the writer named in the contract.
Document the consumer-facing shape in `docs/tenant-guardrail-policy.md`,
matching the style of `docs/tenant-lifecycle-api.md`.
Done when the precedence rules are unambiguous, the fail-closed default is
explicit for every grouping, and the doc states plainly what this service does
*not* do (meter, bill, decide).
Done 2026-08-16: `docs/tenant-guardrail-policy.md`. Decisions worth carrying:
- **Limit kinds:** `spend` (integer minor units + ISO-4217, never floats),
`entity_count`, `action_count`. Rate limiting stays out — it is a gateway
concern with different latency and storage needs, not a free by-product.
- **Registry, not open keys.** A limit key must be registered to resolve; an
unregistered key errors (`unknown_limit_key`) rather than resolving to zero
or unlimited. Zero would break a caller who merely misspelled a key;
unlimited would fail open. Startup validation requires a default for every
grouping, so an unmapped grouping cannot reach production.
- **Precedence, per key:** override → plan → grouping → fail-closed floor. Per
key, not per set, so a plan supplying `spend.monthly` does not wipe out a
grouping-derived `entity.*`. The floor is only reachable if the registry is
internally inconsistent, and reports provenance `fail_closed` so that state
is visible rather than mistaken for policy.
- **`unlimited` is an explicit sentinel**, never a default and never the result
of absence. The rule is "no unset means unlimited" — that bans *inferring* an
open ceiling, not deliberately declaring one.
- **Reserved identifiers** get a `reserved` profile at layer 3 rather than
falling through: spend 0 (infrastructure identities are not billable
spenders), counts explicitly unlimited. Falling through would clamp the
platform's own identity to zero and take the platform down with it.
- **Lifecycle clamp** applies after resolution and may only reduce; `retired`
clamps every limit to the floor while leaving reads working.
- **Consumption does not live in this repo.** tenant-engine is a low-write
policy authority with audit and CAS semantics; consumption is high-frequency
telemetry with opposite needs, and holding it here would drag metering into a
service `SCOPE.md` disowns it from. Reads return limits plus provenance only.
Two things flagged rather than settled: the non-`trial` grouping ceilings are
conservative opening values needing product sign-off (only `trial` = 0 is
canon), and **no repo owns metering** — the same gap `SCOPE.md` records for
payment processing. Until canon names one, `flex-auth` can enforce
ceiling-presence semantics but not consumption-relative ones. A zero budget is
fully enforceable with no meter at all, which is why the ADR-0013 `trial`
default lands immediately.
## T02 - Implement the guardrail domain model
```task
id: TEN-WP-0006-T02
status: done
priority: high
state_hub_task_id: "1b28d836-11d9-4ef3-967b-05bfa74c304a"
```
Add a `guardrail/` module (the namespace `architecture.md` reserves) holding
pure domain types with no framework dependency, mirroring how `domain.py` is
structured: frozen dataclasses, validation that raises typed errors, and
resolution logic as a pure function over (grouping, plan_id, overrides).
Guardrail changes are **audited like role grants** — append-only, carrying
actor, reason, and correlation id. A limit change is a privilege change and
must be as reconstructible as a `RoleGrant`.
Done when effective-guardrail resolution is a total, side-effect-free function
with unit tests covering every grouping, the reserved identifiers, missing
plans, and conflicting overrides.
Done 2026-08-16: `src/tenant_engine/guardrail/``model.py` (frozen
dataclasses, typed errors), `registry.py` (the key registry and grouping
defaults), `resolution.py` (the pure resolution function). 42 new tests, 166
total, all passing.
Design notes worth carrying:
- **Registry totality is enforced at construction.** `LimitDefinition` raises
`GuardrailRegistryInvalidError` if any grouping in `GROUPINGS` lacks a
default, so an unmapped grouping cannot reach production. The `fail_closed`
branch in `resolve_limit` is the second line of defence, not the first — its
test has to bypass the constructor to reach it.
- **A floor may never be `unlimited`,** also enforced at construction. A floor
that fails open is not a floor.
- **Currency is deliberately excluded from `compatible_with`.** Precedence
resolves per key and the winning layer takes the value whole, so two
currencies never combine — which is why there is no exchange rate anywhere
in this service, and why there must not be one.
- **The lifecycle clamp only relabels when it actually reduces.** A retired
`trial` tenant already at zero keeps provenance `grouping`; a retired
`large` tenant becomes `lifecycle`. Relabelling unconditionally would hide
where the value came from.
- **`bool` is rejected as an amount** — `True` is an `int` in Python and would
otherwise pass the non-negative check as a ceiling of 1.
Lint: the new files are clean. `make lint` still fails on **19 pre-existing
E501s** in files this workplan does not touch (`tests/test_store.py` and
friends) — that baseline predates TEN-WP-0006 and is left alone rather than
folded into this diff.
## T03 - Persist guardrails in both stores
```task
id: TEN-WP-0006-T03
status: done
priority: high
state_hub_task_id: "f1c93573-6322-4505-b738-7d66d67e60a8"
```
Extend the `TenantStore` Protocol and both implementations (`InMemoryTenantStore`
and the SQLite store) with guardrail persistence and its audit trail. Follow the
precedent set by `mutate_tenant()` in TEN-WP-0005: the mutation, the audit
event, and any idempotency receipt commit **in one transaction** — a crash must
never leave a changed limit with no audit record.
Add a forward-only, idempotent migration that backfills existing tenants with
their grouping-derived defaults. Existing tenants must not silently gain a
looser limit than a fresh tenant of the same grouping would get.
Interaction with lifecycle: decide and test what a `retired` tenant's
guardrails resolve to. Follow the TEN-WP-0005 precedent — operations that only
*reduce* privilege stay available while retired; loosening does not.
Done when the store-conformance suite (already parametrised over both backends)
covers guardrails, so the durable store cannot diverge from the reference
semantics.
Done 2026-08-16: `TenantStore` gains `guardrail_overrides`, `guardrail_changes`,
and `set_guardrail_override`, implemented in both backends.
`tests/test_guardrail_store_conformance.py` parametrises 35 tests over
in-memory and SQLite. 201 tests total, all passing.
Deviations and decisions, both deliberate:
- **No backfill migration, because grouping defaults are resolved rather than
materialised.** The task anticipated backfilling existing tenants with their
grouping-derived defaults. Storing them would have been worse: it duplicates
a value that already has a single source, and it makes changing a default a
data migration instead of a config edit. Since a tenant with no override
resolves through the same function whether it predates guardrails or not,
"existing tenants must not silently gain a looser limit than a fresh tenant
of the same grouping" holds by construction, not by a migration step that
could be skipped. The forward-only, idempotent part is the two new tables,
created by the same `CREATE TABLE IF NOT EXISTS` script fresh and existing
databases both take — no second schema definition to drift.
- **A guardrail change bumps the tenant version.** It reuses the lifecycle
ETag, so a guardrail write invalidates a stale reader's `If-Match` exactly
as a metadata edit does, rather than introducing a second, separately
versioned resource that a caller could race against the first.
Retired tenants, following the TEN-WP-0005 reduce-privilege precedent:
- Guardrails stay **readable**; it is the values that clamp, not the endpoint.
- A retired tenant's effective limits clamp to the floor, provenance
`lifecycle`.
- **Tightening is allowed while retired, loosening is refused.** The check
compares the before/after value computed *as if the tenant were active*
because the clamp makes every candidate look inert today, and the real
effect of a loosening override lands the moment the tenant is reactivated.
That deferred loosening is the thing worth refusing. Clearing an override is
evaluated the same way, so clearing a tightening override on a retired
tenant is refused too, since fall-through would loosen.
Stored amounts are `TEXT NOT NULL` with an explicit `"unlimited"` token, never
NULL. A NULL that meant unlimited would make an open ceiling the result of
absence — the one thing the contract forbids.
## T04 - Expose guardrail read and write APIs
```task
id: TEN-WP-0006-T04
status: done
priority: high
state_hub_task_id: "4a256517-2773-4679-ad57-2909f22ac8a4"
```
Add the API surface, authorized through the existing flex-auth `WriteAuthorizer`
with **distinct actions** so policy can permit reading a limit without
permitting a change to it — the same split TEN-WP-0005 used for
`tenant.update` / `tenant.retire` / `tenant.reactivate`:
- a read returning a tenant's effective guardrails plus their provenance
(which layer each limit came from), for `flex-auth` to consume;
- a write to set or clear a per-tenant override, requiring `Idempotency-Key`,
`If-Match`, actor, reason, and correlation id, consistent with the lifecycle
mutation contract.
Authorization runs **before** the store is touched, so an unauthorized caller
cannot probe which tenants exist. Errors are redacted: never reflect policy
internals or store paths in `detail`.
New flex-auth actions mean a policy-package change outside this repo. Name the
required actions explicitly in the task record so the flex-auth side is a
tracked handoff and not a surprise — this is exactly what stalled
TEN-WP-0005-T05.
Done when the routes are authorized, versioned, idempotent, correlated, and
provider-neutral, and the OpenAPI document makes the semantics unambiguous.
Done 2026-08-16. Three routes: `GET /tenants/{id}/guardrails` (effective
limits plus provenance), `PUT` and `DELETE
/tenants/{id}/guardrails/{limit_key}`. 26 API tests; 227 total, all passing.
Contract documented in `docs/tenant-guardrail-policy.md`.
**Required flex-auth actions — the tracked handoff this task exists to make
explicit:**
| Action | Resource type | Purpose |
| --- | --- | --- |
| `tenant.guardrail.read` | `guardrail` | read a tenant's effective ceilings |
| `tenant.guardrail.set` | `guardrail` | set or clear a per-tenant override |
Both are **new** and need a policy-package change in `flex-auth` before this
surface functions in production. Until then every check correctly resolves to
deny. See the regression note below — these land on a package that is
currently *behind*, not merely one that needs extending.
Decisions:
- **The read is authorized too**, unlike the existing `/roles` and
`/roles/live` reads. A ceiling is policy about a tenant's commercial
exposure, not a capability claim, and the task's whole point was that policy
must be able to permit reading a limit without permitting a change. `actor`
is a required query parameter, so there is no anonymous caller. Tested both
ways: read-only permission cannot write, write-only permission cannot read.
- **`amount` is a string on the wire.** It lets the `unlimited` sentinel and
an integer share one field without a JSON union, and keeps a spend amount in
minor units from ever round-tripping through a float.
- **`change_id` is derived** from `(tenant_id, limit_key, Idempotency-Key)`
rather than random, so a genuine retry replays the original audit record
instead of minting a second one for a mutation that happened once.
- **Authorization precedes every store touch**, so a denied read of a real
tenant and of a nonexistent one are byte-identical — no probing existence or
registry contents through status codes.
- A retired tenant's refused loosening gets its own code,
`guardrail_loosening_denied`, rather than reusing
`invalid_lifecycle_transition`: the tenant's lifecycle is not in transition,
and a caller should be able to tell "your tenant is retired, tighten instead"
from "your retire/reactivate call was invalid".
## T05 - Conformance and consumer handoff
```task
id: TEN-WP-0006-T05
status: done
priority: medium
state_hub_task_id: "92036fa3-9031-4b42-a67e-93196e236e08"
```
Cover: every grouping's default, `trial` resolving to zero spend, precedence
between grouping/plan/override, unmapped grouping failing closed, reserved
identifiers, guardrail reads on a retired tenant, stale-version conflict,
idempotent replay, cross-tenant authorization denial, store outage, and error
redaction. Prove the existing create/role/plan/lifecycle clients are unaffected.
Then write the consumer handoff for `flex-auth` — the guardrail read is only
useful once a PDP consults it — naming the API version, the authorization
actions, and the contract doc, in the table format TEN-WP-0005-T05 used.
**Do not** deploy to production as part of this workplan. Cluster rollout is a
separate, human-authorized step; see the note below.
Done when the full suite passes and the handoff is sent.
Done 2026-08-16. 231 tests pass (was 166 before this workplan; 65 new). Every
item on the coverage list is exercised:
| Case | Where |
| --- | --- |
| every grouping's default | `test_guardrail_domain` (parametrised over `GROUPINGS`) |
| `trial` resolves to zero spend | `test_guardrail_domain`, `test_api_guardrails` |
| precedence grouping/plan/override | `test_guardrail_domain` |
| precedence is per key, not per set | `test_guardrail_domain` |
| unmapped grouping fails closed | `test_guardrail_domain` (bypasses the constructor to reach it) |
| reserved identifiers | `test_guardrail_domain` (parametrised over `RESERVED_IDENTIFIERS`) |
| guardrail read on a retired tenant | `test_guardrail_store_conformance`, `test_api_guardrails` |
| stale-version conflict | both conformance suites |
| idempotent replay | both conformance suites |
| cross-tenant authorization denial | `test_api_guardrails` |
| store outage, read **and** write path | `test_api_guardrails` |
| error redaction | `test_api_guardrails` |
| existing clients unaffected | `test_api_guardrails`, plus all 166 prior tests unchanged |
Consumer handoff to `flex-auth`:
| Fact | Value |
| --- | --- |
| Contract | `docs/tenant-guardrail-policy.md` |
| API version | `0.1.0` (unchanged — additive routes only) |
| Read route | `GET /tenants/{id}/guardrails?actor=<actor>` |
| Write routes | `PUT` / `DELETE /tenants/{id}/guardrails/{limit_key}` |
| Required actions | `tenant.guardrail.read`, `tenant.guardrail.set` (resource type `guardrail`) |
| Policy package | `tenant-engine.write-api.mutate` — extends the existing seven actions to nine |
| Source revision | `d1d9c9a` |
| Immutable image | **none — not built.** Production rollout is out of scope; see below |
Two things the consumer must know, both stated plainly in the message sent:
1. **`flex-auth` cannot use this yet.** Both actions are new, so until the
policy package carries them every check denies — correctly, but the surface
is inert in production until that lands.
2. **The read alone is not enforcement.** No repo owns metering, so a PDP can
enforce ceiling-presence semantics (is there a limit? is it zero?) but not
consumption-relative ones (has the tenant used it up?). The ADR-0013 `trial`
= zero default is fully enforceable today precisely because it needs no
meter.
## Out of scope / explicitly deferred
- **Production rollout.** Image build and cluster apply need credentials and a
human authorization decision. Track separately.
- **The live flex-auth regression** (`flex-auth-tenant-engine` rolled back to
the four-action image, so `tenant.update` / `tenant.retire` /
`tenant.reactivate` currently deny with `unknown_action`). Unrelated to this
workplan, but it means any *new* actions from T04 land on a policy package
that is already behind. Resolve the regression before shipping T04's actions.
- **Metering and billing.** Not this repo, at any point.