Reserve worker spend durably before governed dispatch
Some checks failed
Governed runtime contract / contract (push) Has been cancelled
Some checks failed
Governed runtime contract / contract (push) Has been cancelled
Assistant: codex Assistant-Model: gpt-5.6-luna Assistant-Session: 01a07ff8-19d0-7820-b4d0-1353833cb7fc
This commit is contained in:
parent
4ffb8acb19
commit
38b25fbc26
14 changed files with 1378 additions and 9 deletions
133
docs/spend-admission.md
Normal file
133
docs/spend-admission.md
Normal file
|
|
@ -0,0 +1,133 @@
|
|||
# Durable worker spend admission
|
||||
|
||||
The claim worker can reserve a declared maximum liability before invoking Glas.
|
||||
This is an opt-in control for a single host and one immutable operating envelope.
|
||||
It adds no service, provider calls or credentials. llm-connect remains the usage
|
||||
reporting owner; its reporting ledger and bundled FX snapshot are not admission
|
||||
inputs. Reuse discovery on 2026-09-09 found no reservation capability in the fresh
|
||||
hosted reuse-surface index (65 capabilities, 61 sources), or in the inspected
|
||||
llm-connect, Railiance Fabric and agentic-resources implementations. This is a
|
||||
bounded discovery result, not proof that every fleet capability is registered.
|
||||
|
||||
No factory policy is provisioned by this change. HFACT-WP-0001-T01 still owns
|
||||
provider enforcement/overrun proof, accepted conservative FX, final operating
|
||||
admission and the deployment return under REINAH-WP-0003-T05/T06.
|
||||
|
||||
## Admission and accounting
|
||||
|
||||
The worker replays pending terminal closes first, then checks spend capacity
|
||||
before claiming. A configured worker refuses profile-absent work. Immediately
|
||||
before invoking Glas it checks the queue owner, ActivityDefinition, resolved
|
||||
repository, project, actor, repository grant, exact profile and descriptor digests,
|
||||
and the native USD/turn limits. The same cached catalog supplies the checked
|
||||
profile to Glas. Policy, ledger and project come from trusted worker configuration;
|
||||
queue execution references and prompt text cannot supply them.
|
||||
|
||||
An SQLite `BEGIN IMMEDIATE` transaction inserts the reservation before dispatch.
|
||||
The ledger opens in existing-file mode, with synchronous FULL transactions. A
|
||||
missing/corrupt ledger, changed policy, invalid FX/amount, expired policy, backward
|
||||
clock, exhausted daily/total capacity or unresolved reservation refuses execution.
|
||||
Concurrent processes sharing the file cannot admit two active runs. A run ID or
|
||||
ActivityDefinition/idempotency-key pair cannot be spent again, even on a new
|
||||
claim attempt, after reconciliation or under a replacement queue row ID.
|
||||
|
||||
The reservation is `ceil(max_liability_usd * eur_per_usd * 1,000,000)` integer
|
||||
micro-euros; envelope ceilings round down. Monetary arithmetic uses decimal
|
||||
values without reporting-FX defaults. `max_budget_usd` is the native CLI stop
|
||||
threshold and must fit inside the declared maximum liability. These two values
|
||||
are separate because admission must account for the provider's demonstrated
|
||||
maximum exposure, including any bounded in-flight overshoot.
|
||||
|
||||
On complete success with finite accounting and confirmed session cleanup and
|
||||
sandbox destruction, the ledger charges the **full reserved amount**. It never
|
||||
refunds capacity based on the sandbox's self-reported cost. Missing accounting,
|
||||
execution failure, cancellation, lost lease or crash leaves the reservation held
|
||||
and blocks further admission, including after midnight or restart. An observed
|
||||
cost above declared liability is recorded conservatively and permanently marks
|
||||
the envelope breached. Reconciliation does not clear that breach.
|
||||
|
||||
A completed or reconciled reservation counts in total once and, conservatively,
|
||||
against every local calendar day from admission through completion/reconciliation.
|
||||
A run crossing midnight therefore uses its full reservation in both daily totals.
|
||||
Unresolved work blocks new admission globally; no timer silently releases it.
|
||||
This intentionally sacrifices some utilization for the first bounded pilot.
|
||||
More exact billing reconciliation can follow measured use without weakening the
|
||||
unknown-outcome boundary.
|
||||
|
||||
## Operator configuration and recovery
|
||||
|
||||
Use a private directory on durable local storage outside every sandbox mount and
|
||||
target checkout. The directory must be worker-owned mode 0700; policy and ledger
|
||||
must be worker-owned regular files mode 0600, without hardlinks or symlinks.
|
||||
All admitted processes must use this one ledger. Separate copies, network-file
|
||||
locking, multi-host admission, rollback to stale backups and hostile host owners
|
||||
are outside v1. The protected deployment must keep both files and the launch
|
||||
configuration inaccessible to the workload and preserve them across restarts.
|
||||
|
||||
Configure the following only after the operating envelope is accepted:
|
||||
|
||||
```text
|
||||
AGENT_HARNESS_REQUIRE_SPEND_ADMISSION=1
|
||||
AGENT_HARNESS_SPEND_POLICY=/absolute/private/spend-policy.json
|
||||
AGENT_HARNESS_SPEND_LEDGER=/absolute/private/spend.sqlite3
|
||||
AGENT_HARNESS_EXECUTION_PROJECT=<accepted-project>
|
||||
```
|
||||
|
||||
Any nonempty REQUIRE setting requires admission; missing policy/ledger then
|
||||
refuses before claim. With all spend settings absent, the existing worker
|
||||
compatibility behavior remains. The factory deployment must set REQUIRE and
|
||||
verify missing-configuration denial as part of its owner acceptance.
|
||||
|
||||
The strict JSON `SpendPolicy` fields are:
|
||||
|
||||
| Fields | Contract |
|
||||
| --- | --- |
|
||||
| `version`, `envelope_id`, `authority_ref` | Version `1`, unique envelope identity, reference to the accepted owner record. Local configuration is trusted; this module does not verify signatures or grant authority. |
|
||||
| `valid_from`, `expires_at`, `timezone` | Explicit timezone-aware validity interval and budget calendar, for example Europe/Berlin. |
|
||||
| `worker_id`, `activity_definition_id`, `target_repo`, `project` | Exact admitted scope; target is an absolute path. Actor is `agt`. |
|
||||
| `profile_ref`, `profile_sha256`, `descriptor_sha256`, `repository_grant_id` | Versioned profile plus SHA-256 of canonical `model_dump(mode="json")` for the resolved profile and descriptor; `spend_admission.digest` supplies the canonical serializer. Grant identity is the existing `RepositoryGrant.grant_id`. |
|
||||
| `max_budget_usd`, `max_liability_usd`, `eur_per_usd`, `per_run_eur`, `daily_eur`, `total_eur` | Positive decimal **strings**. Explicit liability/FX must fit per-run, daily and total ceilings. No implicit exchange rate or provider price. |
|
||||
| `max_turns` | Positive integer matching the resolved native profile. |
|
||||
|
||||
Provision the ledger once using `rein-aharness spend init --policy <path>
|
||||
--ledger <path>`. Initialization refuses an existing file. Worker execution never
|
||||
creates an empty replacement. Keep the accepted policy immutable; replacing its
|
||||
contents under the same ledger refuses, rather than resetting spent capacity.
|
||||
|
||||
Inspect with `rein-aharness spend status --policy <path> --ledger <path>`.
|
||||
If a reservation is held, the operator must first confirm provider execution has
|
||||
stopped and obtain final accounting. Record the evidence in the owning work record,
|
||||
then run:
|
||||
|
||||
```text
|
||||
rein-aharness spend reconcile --policy <path> --ledger <path> \
|
||||
--run-id <id> --cost-usd <final-usd> --receipt <bounded-evidence-reference> \
|
||||
--provider-stopped
|
||||
```
|
||||
|
||||
The flag is an operator attestation, not an automatic provider query. The local
|
||||
receipt must refer to reviewed termination and final accounting; it must contain
|
||||
no secrets. Reconciliation charges at least the full reservation and any higher
|
||||
observed liability, cannot discard a known higher cost, is idempotent for the same
|
||||
receipt/cost, and never authorizes rerunning that demand. There is no refund,
|
||||
reset, policy migration or breach-clear command. A breached envelope requires a
|
||||
new owner decision that accounts for existing spend before any further admission.
|
||||
|
||||
## Proof and remaining boundary
|
||||
|
||||
`tests/test_spend_admission.py` covers pre-dispatch denial, concurrent processes,
|
||||
crash/reopen, identity/profile/grant changes, replay, unknown and overrun outcomes,
|
||||
midnight, total exhaustion, decimal rounding and operator reconciliation. It is
|
||||
included in both runtime-contract and recovery gates.
|
||||
|
||||
The opt-in `REIN_REAL_BWRAP=1` test also runs actual bwrap, sandbox owner transport,
|
||||
Glas, repository import and durable-close replay with admission enabled. Queue
|
||||
responses, accounting and authoring are deterministic fixtures; no model is
|
||||
called. The fixture's policy and FX have no production authority.
|
||||
|
||||
This ledger enforces allocation against **declared** maximum liability. It cannot
|
||||
make an opaque provider/tool loop honor that maximum, establish a hard EUR limit,
|
||||
or verify the operator's FX assumption. G0 remains blocked until those semantics
|
||||
are proven for the pinned provider/CLI, including retries, cache, subagents and
|
||||
in-flight work. Exact credential/identity/egress/placement and protected-runtime
|
||||
installation are also still required before natural factory execution.
|
||||
Loading…
Add table
Add a link
Reference in a new issue