workplan: FIN-WP-0007 AI plan cost, token estimates, and effectiveness
Record the booked-vs-usage-vs-entitlement-vs-work split for AI plans so missing session tokens can be estimated without becoming booked spend.
This commit is contained in:
parent
cfe152f19c
commit
6f0eb7f7f8
1 changed files with 278 additions and 0 deletions
278
workplans/FIN-WP-0007-ai-plan-token-effectiveness.md
Normal file
278
workplans/FIN-WP-0007-ai-plan-token-effectiveness.md
Normal file
|
|
@ -0,0 +1,278 @@
|
|||
---
|
||||
id: FIN-WP-0007
|
||||
type: workplan
|
||||
title: "AI plan cost, token estimates, and effectiveness"
|
||||
domain: financials
|
||||
repo: fin-hub
|
||||
status: proposed
|
||||
owner: grok
|
||||
topic_slug: financials
|
||||
created: "2026-08-15"
|
||||
updated: "2026-08-15"
|
||||
related:
|
||||
- FIN-WP-0002
|
||||
- FIN-WP-0004
|
||||
- STATE-WP-0045
|
||||
- ITC-CAP-0.2.0
|
||||
origin: assessment
|
||||
origin_ref: "info-tech-canon 0.3.0 review — AI token data-gap"
|
||||
---
|
||||
|
||||
# FIN-WP-0007 — AI plan cost, token estimates, and effectiveness
|
||||
|
||||
## Goal
|
||||
|
||||
Build a provenance-bearing dataset that lets fin-hub compare **what AI
|
||||
plans cost**, **what token capacity they entitle**, **what was actually
|
||||
or estimably consumed**, and **what work was delivered**, without treating
|
||||
missing session usage as zero and without turning estimates into booked
|
||||
spend.
|
||||
|
||||
The motivating failure is operational, not theoretical: most coding-agent
|
||||
sessions do not emit trustworthy token counts or token costs. The facts
|
||||
that *do* exist are monthly plan invoices, plan entitlements, occasional
|
||||
provider or State Hub measurements, and independent work outcomes. Those
|
||||
must stay distinct and joinable so effectiveness can be watched over
|
||||
time.
|
||||
|
||||
This workplan applies the InfoTechCanon 0.3.0 / ITC-CAP 0.2.0 split
|
||||
already used for infrastructure in `FIN-WP-0004`: booked euros are not
|
||||
usage; usage is recorded in native units (`token` for class `I`);
|
||||
`unknown` is never stored as `0`; currency is an overlay. It does **not**
|
||||
wait for a new commerce or intelligence domain model.
|
||||
|
||||
## Boundaries
|
||||
|
||||
| Concept | Authoritative writer | Fin-hub role |
|
||||
| --- | --- | --- |
|
||||
| Provider invoice / plan subscription charge | fin-hub | Book once as `booked_cost` |
|
||||
| Recurring plan as a financial commitment | fin-hub | Record cadence and amount; not usage |
|
||||
| Plan entitlement (included tokens or plan capacity) | fin-hub | Commercial fact, same discipline as engagement prices |
|
||||
| Session token evidence (`measured` / `allocated` / `estimated` / `superseded`) | State Hub (`STATE-WP-0045`) | Consume aggregates; do not re-parse sessions |
|
||||
| Technical infrastructure usage and resource identity | resource-control | Out of scope here (class `S`/`C`/`P`, `FIN-WP-0004`) |
|
||||
| Work done (tasks, progress, outcomes) | State Hub / domain repos | Join key for effectiveness; not a cost driver |
|
||||
| Implied €/token and work-per-token / work-per-euro | fin-hub | Reporting overlay; never a booked price |
|
||||
| Vendor parsers, transcripts, model names, list prices | State Hub / llm-connect / operators | Not invented here |
|
||||
|
||||
Reuse `docs/fin-resource-authority-contract-v0.1.md` shapes
|
||||
(`booked_cost`, `usage_observation`, `AllocationEvidence`). Resource-control
|
||||
is the wrong *producer* for AI-session usage; the *envelope* is the right
|
||||
one. A new AI-plan producer (State Hub aggregates, plus fin-hub's own
|
||||
entitlement and booked facts) fills that role.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- Legal invoices, DATEV postings, or payment execution (`FIN-WP-0005`).
|
||||
- Completing the Scaleway `platform:audit-storage` booked-cost round trip
|
||||
(`FIN-WP-0004-T05`). That is class `S`, not class `I`.
|
||||
- Owning session parsers or storing transcripts.
|
||||
- A canonical token↔hour or token↔euro exchange rate (ITC-CAP forbids
|
||||
asserting one; substitution is observed from the time series).
|
||||
- A new definition of “work” beyond joining to existing task / progress
|
||||
outcomes.
|
||||
- Editing InfoTechCanon inline. Canon pressure is a demand message, not a
|
||||
kernel patch from this repo.
|
||||
|
||||
## Measurement layers
|
||||
|
||||
Keep these four quantities in separate records, joined by period,
|
||||
provider/plan, and optional workplan/repo:
|
||||
|
||||
1. **Booked plan cost** — euros for the subscription month.
|
||||
2. **Entitlement** — included tokens or declared plan capacity.
|
||||
3. **Consumption of class `I`** — tokens, each row marked
|
||||
`measured` | `inferred` | `unknown`. Missing is `unknown`, never `0`.
|
||||
4. **Work done** — tasks or accepted outcomes in the same period.
|
||||
|
||||
Derived overlays, never booked facts:
|
||||
|
||||
- implied €/token = booked ÷ (measured + estimated + residual)
|
||||
- effectiveness = work / measured tokens, work / (measured+estimated)
|
||||
tokens, and work / euro
|
||||
- coverage = measured / (measured + estimated + residual)
|
||||
|
||||
A month with worse coverage must not look more efficient. Changing an
|
||||
estimator method starts a new series.
|
||||
|
||||
## Record the authority and measurement split
|
||||
|
||||
```task
|
||||
id: FIN-WP-0007-T00
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
Write a short design note in `docs/` that records the four-layer split,
|
||||
the authority table above, reuse of the v0.1 exchange envelopes, and
|
||||
the State Hub join (token aggregates by
|
||||
`measurement_kind`, period, repo/workplan). Confirm `SCOPE.md` already
|
||||
covers token-spend models; if the current wording implies session-metered
|
||||
CSV rows are the only token surface, update it so plan invoices,
|
||||
entitlements, estimated usage, and effectiveness reports are in scope
|
||||
while session parsing stays out.
|
||||
|
||||
Done when the note is committed, `SCOPE.md` matches the boundary, and
|
||||
no task below has an ambiguous owner for booked cost, entitlement,
|
||||
session tokens, or work.
|
||||
|
||||
## Stop treating missing tokens as zero
|
||||
|
||||
```task
|
||||
id: FIN-WP-0007-T01
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
After T00, fix `TokenSpend` and `parse_anthropic_billing_csv` so absent
|
||||
token fields are `null` / omitted, not `0`. `0` means measured and none.
|
||||
Align ledger import: token quantities that are unknown must not become
|
||||
zero ledger attributes, and a cost-only row must remain a cost-only row.
|
||||
|
||||
Add regression tests for missing `input_tokens` / `output_tokens`,
|
||||
cost-only provider exports, and the invariant that incomplete months
|
||||
cannot under-report consumption as zero.
|
||||
|
||||
Done when the existing fixture still parses, a missing-token fixture
|
||||
does not emit zeros, and CAP-R8 (`unknown` never stored as zero) holds
|
||||
on the token path.
|
||||
|
||||
## Book AI-plan invoices as facts and commitments
|
||||
|
||||
```task
|
||||
id: FIN-WP-0007-T02
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
After T00, ingest monthly AI-plan / subscription charges as
|
||||
`booked_cost` facts (one current fact per provider-account × plan ×
|
||||
accounting period) and as recurring `Commitment` rows where the plan is
|
||||
ongoing. Use the existing financial-fact identity, fingerprint,
|
||||
correction, and unknown-tax rules from `FIN-WP-0004`.
|
||||
|
||||
Do not explode a flat monthly invoice into synthetic per-session costs.
|
||||
Session-level euros, if shown at all, come from a later allocation
|
||||
(T05), not from the booked ledger.
|
||||
|
||||
Done when a fixture for at least two providers (for example a flat
|
||||
subscription and a usage-top-up) books exactly once per period,
|
||||
corrects append-only, and appears on burn/runway like any other
|
||||
commitment.
|
||||
|
||||
## Record plan entitlements
|
||||
|
||||
```task
|
||||
id: FIN-WP-0007-T03
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
After T02, add an append-only plan-entitlement record: provider, plan
|
||||
identity, period, entitled quantity, unit (`token` or a declared plan
|
||||
capacity such as a multiplier), source provenance, and revision link.
|
||||
Follow the engagement-price discipline from `FIN-WP-0002-T02`: a
|
||||
revision supersedes, a price or capacity is not an invoice, currency
|
||||
mismatches cannot silently join.
|
||||
|
||||
When entitled tokens are themselves unknown (common for “Max 20×”
|
||||
style plans), store `unknown` plus the declared plan label. Do not
|
||||
invent a token ceiling.
|
||||
|
||||
Done when a plan month can be reported as booked euros + entitled
|
||||
capacity (or explicit unknown) without reading session logs.
|
||||
|
||||
## Consume State Hub token aggregates as usage observations
|
||||
|
||||
```task
|
||||
id: FIN-WP-0007-T04
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
After T00, ingest State Hub token summaries as `usage_observation`
|
||||
records (or a compatible typed sibling). Preserve `measurement_kind`
|
||||
(`measured`, `allocated`, `estimated`, `superseded`), period, model,
|
||||
repo/workplan association, confidence, and coverage. Do not re-parse
|
||||
Codex/Claude/Cursor/Grok session files in this repo.
|
||||
|
||||
Superseded rows must not enter totals. Estimated and measured must
|
||||
remain separable. Months with no State Hub events produce an explicit
|
||||
unknown residual, not a zero-consumption observation.
|
||||
|
||||
Done when a recorded fixture of mixed measured/estimated/superseded
|
||||
aggregates stores as usage evidence, reconciles by kind, and cannot be
|
||||
posted as booked spend.
|
||||
|
||||
## Allocate booked plan cost across work
|
||||
|
||||
```task
|
||||
id: FIN-WP-0007-T05
|
||||
status: todo
|
||||
priority: medium
|
||||
```
|
||||
|
||||
After T02 and T04, allocate each AI-plan `financial_fact_id` across
|
||||
repos/workplans using `AllocationEvidence`: named method, shares, and
|
||||
explicit residual. Allowed first methods: measured-token share,
|
||||
measured+estimated-token share, session-count share, even split.
|
||||
Method and estimator version are part of the record identity.
|
||||
|
||||
Shares plus residual must equal 1. Unattributed and unmeasured shares
|
||||
stay visible. Allocation never writes a new booked fact.
|
||||
|
||||
Done when a one-plan, two-workplan fixture reconciles to the booked
|
||||
effective amount, names its method, and keeps residual when coverage
|
||||
is incomplete.
|
||||
|
||||
## Publish an effectiveness report
|
||||
|
||||
```task
|
||||
id: FIN-WP-0007-T06
|
||||
status: todo
|
||||
priority: medium
|
||||
```
|
||||
|
||||
After T03–T05, add a period report (CLI and, if cheap, the existing
|
||||
read API) that emits, per plan and currency:
|
||||
|
||||
- booked effective amount
|
||||
- entitled capacity (or unknown)
|
||||
- tokens measured / estimated / residual-unknown
|
||||
- coverage ratio
|
||||
- implied €/token, labelled inferred
|
||||
- work joined (task counts or accepted outcomes) for the same period
|
||||
- work per measured token, work per (measured+estimated) token, work
|
||||
per euro — three series, not one blend
|
||||
|
||||
Refuse to present a single blended “efficiency” number. A month whose
|
||||
coverage is below a documented threshold is marked unfit for trend
|
||||
comparison, not interpolated.
|
||||
|
||||
Done when two fixture months with different coverage cannot produce a
|
||||
chart in which the worse-measured month looks cheaper per unit of
|
||||
work unless that is also true on the measured-only series.
|
||||
|
||||
## Send canon feedback
|
||||
|
||||
```task
|
||||
id: FIN-WP-0007-T07
|
||||
status: todo
|
||||
priority: low
|
||||
```
|
||||
|
||||
After T00, send a demand signal to `info-tech-canon` (State Hub
|
||||
message and/or a `demand/` note they can assimilate). Ask for:
|
||||
|
||||
- a `measured | inferred | declared | unknown` qualifier on
|
||||
`CapabilityConsumption` (reuse ITC-DATA lineage states; do not
|
||||
invent a local dialect);
|
||||
- enough `commerce.entitlement` / `commerce.metering` structure for
|
||||
plan capacity and incomplete meters (OQ-5 still open for invoices
|
||||
and entitlements);
|
||||
- dropping or re-homing the `cost` quality dimension on
|
||||
`intelligence.generation` so it does not fight ITC-CAP §4.12.
|
||||
|
||||
Do not patch the canon from this repo.
|
||||
|
||||
Done when the demand is sent, referenced from this workplan, and
|
||||
explicitly does not block T01–T06.
|
||||
Loading…
Add table
Add a link
Reference in a new issue