Some checks failed
Governed runtime contract / contract (push) Failing after 31s
Assistant: codex Assistant-Model: gpt-5.6-luna Assistant-Session: 01a07ff8-19d0-7820-b4d0-1353833cb7fc
142 lines
8.7 KiB
Markdown
142 lines
8.7 KiB
Markdown
# Durable worker spend admission
|
|
|
|
The claim worker can reserve a declared maximum liability before invoking Glas.
|
|
This is an opt-in control for a single host and one immutable operating envelope.
|
|
It adds no service, provider calls or credentials. llm-connect remains the usage
|
|
reporting owner; its reporting ledger and bundled FX snapshot are not admission
|
|
inputs. Reuse discovery on 2026-09-09 found no reservation capability in the fresh
|
|
hosted reuse-surface index (65 capabilities, 61 sources), or in the inspected
|
|
llm-connect, Railiance Fabric and agentic-resources implementations. This is a
|
|
bounded discovery result, not proof that every fleet capability is registered.
|
|
|
|
No factory policy is provisioned by this change. HFACT-WP-0001-T01 still owns
|
|
provider enforcement/overrun proof, accepted conservative FX, final operating
|
|
admission and the deployment return under REINAH-WP-0003-T05/T06.
|
|
|
|
## Admission and accounting
|
|
|
|
The worker replays pending terminal closes first, then checks spend capacity
|
|
before claiming. A configured worker refuses profile-absent work. Immediately
|
|
before invoking Glas it checks the queue owner, ActivityDefinition, resolved
|
|
repository, project, actor, repository grant, exact profile and descriptor digests,
|
|
and the native USD/turn limits. The same cached catalog supplies the checked
|
|
profile to Glas. Policy, ledger and project come from trusted worker configuration;
|
|
queue execution references and prompt text cannot supply them.
|
|
|
|
An SQLite `BEGIN IMMEDIATE` transaction inserts the reservation before dispatch.
|
|
The ledger opens in existing-file mode, with synchronous FULL transactions. A
|
|
missing/corrupt ledger, changed policy, invalid FX/amount, expired policy, backward
|
|
clock, exhausted daily/total capacity or unresolved reservation refuses execution.
|
|
Concurrent processes sharing the file cannot admit two active runs. A run ID or
|
|
ActivityDefinition/idempotency-key pair cannot be spent again, even on a new
|
|
claim attempt, after reconciliation or under a replacement queue row ID.
|
|
|
|
The reservation is `ceil(max_liability_usd * eur_per_usd * 1,000,000)` integer
|
|
micro-euros; envelope ceilings round down. Monetary arithmetic uses decimal
|
|
values without reporting-FX defaults. `max_budget_usd` is the native CLI stop
|
|
threshold and must fit inside the declared maximum liability. These two values
|
|
are separate because admission must account for the provider's demonstrated
|
|
maximum exposure, including any bounded in-flight overshoot.
|
|
|
|
On complete success with finite accounting and confirmed session cleanup and
|
|
sandbox destruction, the ledger charges the **full reserved amount**. It never
|
|
refunds capacity based on the sandbox's self-reported cost. Missing accounting,
|
|
execution failure, cancellation, lost lease or crash leaves the reservation held
|
|
and blocks further admission, including after midnight or restart. An observed
|
|
cost above declared liability is recorded conservatively and permanently marks
|
|
the envelope breached. Reconciliation does not clear that breach.
|
|
|
|
A completed or reconciled reservation counts in total once and, conservatively,
|
|
against every local calendar day from admission through completion/reconciliation.
|
|
A run crossing midnight therefore uses its full reservation in both daily totals.
|
|
Unresolved work blocks new admission globally; no timer silently releases it.
|
|
This intentionally sacrifices some utilization for the first bounded pilot.
|
|
More exact billing reconciliation can follow measured use without weakening the
|
|
unknown-outcome boundary.
|
|
|
|
## Operator configuration and recovery
|
|
|
|
Use a private directory on durable local storage outside every sandbox mount and
|
|
target checkout. The directory must be worker-owned mode 0700; policy and ledger
|
|
must be worker-owned regular files mode 0600, without hardlinks or symlinks.
|
|
All admitted processes must use this one ledger. Separate copies, network-file
|
|
locking, multi-host admission, rollback to stale backups and hostile host owners
|
|
are outside v1. The protected deployment must keep both files and the launch
|
|
configuration inaccessible to the workload and preserve them across restarts.
|
|
|
|
Configure the following only after the operating envelope is accepted:
|
|
|
|
```text
|
|
AGENT_HARNESS_REQUIRE_SPEND_ADMISSION=1
|
|
AGENT_HARNESS_SPEND_POLICY=/absolute/private/spend-policy.json
|
|
AGENT_HARNESS_SPEND_LEDGER=/absolute/private/spend.sqlite3
|
|
AGENT_HARNESS_EXECUTION_PROJECT=<accepted-project>
|
|
```
|
|
|
|
Any nonempty REQUIRE setting requires admission; missing policy/ledger then
|
|
refuses before claim. With all spend settings absent, the existing worker
|
|
compatibility behavior remains. The factory deployment must set REQUIRE and
|
|
verify missing-configuration denial as part of its owner acceptance.
|
|
|
|
The strict JSON `SpendPolicy` fields are:
|
|
|
|
| Fields | Contract |
|
|
| --- | --- |
|
|
| `version`, `envelope_id`, `authority_ref` | Version `1`, unique envelope identity, reference to the accepted owner record. Local configuration is trusted; this module does not verify signatures or grant authority. |
|
|
| `valid_from`, `expires_at`, `timezone` | Explicit timezone-aware validity interval and budget calendar, for example Europe/Berlin. |
|
|
| `worker_id`, `activity_definition_id`, `target_repo`, `project` | Exact admitted scope; target is an absolute path. Actor is `agt`. |
|
|
| `profile_ref`, `profile_sha256`, `descriptor_sha256`, `repository_grant_id` | Versioned profile plus SHA-256 of canonical `model_dump(mode="json")` for the resolved profile and descriptor; `spend_admission.digest` supplies the canonical serializer. Grant identity is the existing `RepositoryGrant.grant_id`. |
|
|
| `max_budget_usd`, `max_liability_usd`, `eur_per_usd`, `per_run_eur`, `daily_eur`, `total_eur` | Positive decimal **strings**. Explicit liability/FX must fit per-run, daily and total ceilings. No implicit exchange rate or provider price. |
|
|
| `max_turns` | Positive integer matching the resolved native profile. |
|
|
|
|
Provision the ledger once using `rein-aharness spend init --policy <path>
|
|
--ledger <path>`. Initialization refuses an existing file. Worker execution never
|
|
creates an empty replacement. Keep the accepted policy immutable; replacing its
|
|
contents under the same ledger refuses, rather than resetting spent capacity.
|
|
|
|
Inspect with `rein-aharness spend status --policy <path> --ledger <path>`.
|
|
If a reservation is held, the operator must first confirm provider execution has
|
|
stopped and obtain final accounting. Record the evidence in the owning work record,
|
|
then run:
|
|
|
|
```text
|
|
rein-aharness spend reconcile --policy <path> --ledger <path> \
|
|
--run-id <id> --cost-usd <final-usd> --receipt <bounded-evidence-reference> \
|
|
--provider-stopped
|
|
```
|
|
|
|
The flag is an operator attestation, not an automatic provider query. The local
|
|
receipt must refer to reviewed termination and final accounting; it must contain
|
|
no secrets. Reconciliation charges at least the full reservation and any higher
|
|
observed liability, cannot discard a known higher cost, is idempotent for the same
|
|
receipt/cost, and never authorizes rerunning that demand. There is no refund,
|
|
reset, policy migration or breach-clear command. A breached envelope requires a
|
|
new owner decision that accounts for existing spend before any further admission.
|
|
|
|
## Proof and remaining boundary
|
|
|
|
`tests/test_spend_admission.py` covers pre-dispatch denial, concurrent processes,
|
|
crash/reopen, identity/profile/grant changes, replay, unknown and overrun outcomes,
|
|
midnight, total exhaustion, decimal rounding and operator reconciliation. It is
|
|
included in both runtime-contract and recovery gates.
|
|
|
|
The opt-in `REIN_REAL_BWRAP=1` test also runs actual bwrap, sandbox owner transport,
|
|
Glas, repository import and durable-close replay with admission enabled. Queue
|
|
responses, accounting and authoring are deterministic fixtures; no model is
|
|
called. The fixture's policy and FX have no production authority.
|
|
|
|
This ledger enforces allocation against **declared** maximum liability. It cannot
|
|
make an opaque provider/tool loop honor that maximum, establish a hard EUR limit,
|
|
or verify the operator's FX assumption. G0 remains blocked until those semantics
|
|
are proven for the pinned provider/CLI, including retries, cache, subagents and
|
|
in-flight work. Exact credential/identity/egress/placement and protected-runtime
|
|
installation are also still required before natural factory execution.
|
|
|
|
The [native CLI proof](native-cli-boundary-proof.md) now demonstrates the missing
|
|
pre-request bound: one fixture response overshot a USD 0.01 threshold to an
|
|
estimated USD 0.18015. The [request extension](request-admission.md) now provides
|
|
child holds and the llm-connect transport proves reserve-before-forward with the
|
|
installed CLI. HFACT T01 retains owner hosting, actual lease/token integration,
|
|
accepted tariffs/FX and direct-provider bypass denial. Parent observation also
|
|
revokes request routes and retains unresolved children; operator reconciliation
|
|
keeps full liability while closing the route.
|