IR-WP-0006-T01/T02: billing reconciliation contract and settlement mechanism
Spec and implement external-evidence settlement for pending trial reservations, so the USD 0.023712 HTTP-401 reservation (and future ones) can be resolved without fabricating a completed trial or treating a failed request as zero-charge proof. intelligence_radar.trials.reconcile (+ CLI reconcile) is write-once under the existing trials lock, rejects unknown reservations, duplicate/conflicting settlement, cross-campaign references, invalid amounts and budget overrun, and records reviewer attribution plus an evidence pointer rather than any secret value. 9 new tests; full suite (48 tests) and repository check pass. T03 (applying this to the real reservation) stays blocked on authoritative provider billing evidence. ADR-0003 closes the IR-WP-0007-T03 uptime question: retain the local systemd timer, no hosted worker, since nothing here needs sub-day freshness and the scheduler already tolerates downtime. Mark IR-WP-0005/0006/0007 workplan status blocked: each still has at least one task genuinely blocked on an external party (secrets-engine human approval, authoritative billing evidence, and paid-execution outcomes respectively) rather than anything actionable in this repo. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Assistant: claude-code Assistant-Model: sonnet Assistant-Process: 8472@bnt-lap001 Assistant-Session: 1cac6b6d-c1a5-4a04-a950-c4febb8d3f67
This commit is contained in:
parent
7d03727945
commit
710b2ebbb5
8 changed files with 312 additions and 6 deletions
46
docs/adr/0003-operating-model-disposition.md
Normal file
46
docs/adr/0003-operating-model-disposition.md
Normal file
|
|
@ -0,0 +1,46 @@
|
|||
# ADR 0003 — Operating model disposition (local timer retained)
|
||||
|
||||
Accepted: 2026-09-27.
|
||||
|
||||
## Context
|
||||
|
||||
IR-WP-0007-T03 asks whether continuous uptime is required for Intelligence
|
||||
Radar's scheduled jobs, and to retain the local timer unless a concrete
|
||||
requirement justifies a hosted worker and its own live owner workplan.
|
||||
|
||||
The current operating model is a user-level systemd timer/service pair
|
||||
(`ops/systemd/intelligence-radar.timer`, `ops/systemd/intelligence-radar.service`)
|
||||
invoking `python3 -m intelligence_radar run-due`, which is explicitly
|
||||
idempotent and safe to invoke repeatedly (`ops/RUNBOOK.md`). `run-due` and the
|
||||
scheduler already handle missed/late runs and retries (`intelligence_radar/scheduler.py`,
|
||||
exercised in `tests/test_ingestion_scheduler.py`), so uptime gaps degrade
|
||||
freshness rather than correctness.
|
||||
|
||||
## Decision
|
||||
|
||||
Retain the local systemd user timer as the operating model. No hosted worker
|
||||
is adopted.
|
||||
|
||||
## Rationale
|
||||
|
||||
- Nothing in this repository's current scope requires sub-day freshness: the
|
||||
cadence is daily/weekly digests and periodic source captures, not real-time
|
||||
alerting.
|
||||
- The scheduler design already absorbs downtime (a laptop asleep, a missed
|
||||
boot) by running the latest due job on next wake rather than requiring
|
||||
uninterrupted uptime.
|
||||
- A hosted worker adds a live infrastructure surface (deployment, credentials,
|
||||
monitoring, its own on-call) with no offsetting capability gain today, and
|
||||
would need its own owner workplan to be justified, per this task's own
|
||||
instruction not to start "another infrastructure programme before the
|
||||
research evidence is usable."
|
||||
- The evidence-quality blockers ahead of this project (IR-WP-0005 credential
|
||||
delivery, IR-WP-0006 billing reconciliation) are unrelated to scheduler
|
||||
uptime; hosting the scheduler elsewhere would not unblock them.
|
||||
|
||||
## Revisit trigger
|
||||
|
||||
Reopen this decision only if a concrete requirement appears — e.g. a
|
||||
downstream consumer needs sub-daily freshness guarantees, or the operator
|
||||
machine's uptime becomes materially unreliable for the daily/weekly cadence.
|
||||
Until then this is not tracked as an open task.
|
||||
|
|
@ -0,0 +1,89 @@
|
|||
# IR-WP-0006-T01 — billing evidence and reconciliation contract
|
||||
|
||||
Reviewed 2026-09-27 by codex. Specifies how an unresolved trial reservation is
|
||||
settled from reviewed external billing evidence, without treating absence of
|
||||
a response as proof of zero cost.
|
||||
|
||||
## Inventory: campaign ir-openrouter-pilot-20260913
|
||||
|
||||
One append-only `trial-events` record exists for this campaign:
|
||||
|
||||
```json
|
||||
{"budget_usd":"10.00","campaign":"ir-openrouter-pilot-20260913","case_id":"provenance-1",
|
||||
"created_at":"2026-09-13T21:22:29.301211+00:00","kind":"reserved","model":"anthropic/claude-sonnet-5",
|
||||
"reserved_usd":"0.023712","run_id":"29a46a1d6c2d9ef6cfce8de6efef75cddcfecee346c1b4babf57119f1a241d71"}
|
||||
```
|
||||
|
||||
It is a `reserved` event with no matching `settled` event — the pending
|
||||
USD 0.023712 reservation referenced throughout IR-WP-0004/0005/0006. `budget()`
|
||||
in `intelligence_radar/trials.py` already reports this as `pending_usd`.
|
||||
There is no mechanism today to settle a reservation except `execute()`, which
|
||||
requires a live, successful, billed provider call — the wrong tool for a
|
||||
request that returned HTTP 401 and never authenticated. `HTTP 401 alone is not
|
||||
zero-charge proof`: no evidence in this repository establishes the provider's
|
||||
actual charge for that attempt, and none is invented here.
|
||||
|
||||
## What counts as billing evidence
|
||||
|
||||
Provider-issued evidence of the actual charge (or absence of charge) for a
|
||||
specific reservation's `run_id`: an OpenRouter activity/billing export,
|
||||
dashboard screenshot, or admin API response showing the request's cost. It
|
||||
must be retained as an artifact (content-addressed, like the rest of this
|
||||
repository's evidence) and referenced by a pointer — a SHA-256 or file path —
|
||||
never by embedding a credential or raw authenticated response that could carry
|
||||
a secret. Retrieving that evidence may itself require the admitted read-only
|
||||
path from IR-WP-0005; this contract does not require the credential, only a
|
||||
place to record what it produces.
|
||||
|
||||
## Matching a reservation to evidence
|
||||
|
||||
A settlement targets exactly one `reserved` event, identified by its
|
||||
`run_id` (content-addressed identity of `campaign + case + request payload`,
|
||||
already unique and tamper-evident). The settlement is refused unless:
|
||||
|
||||
- **the reservation exists** — the `run_id` matches a recorded `reserved`
|
||||
event (else: unknown reservation);
|
||||
- **the campaign matches** — the reservation's own `campaign` field equals the
|
||||
campaign the settlement is filed under (else: cross-campaign reference);
|
||||
- **no settlement already exists** for that reservation (else: duplicate or
|
||||
conflicting settlement — settlement is write-once, matching the store's
|
||||
append-only, content-addressed design);
|
||||
- **the amount is a valid, finite, nonnegative decimal** (else: invalid
|
||||
amount);
|
||||
- **recording it would not push the campaign's committed total (other
|
||||
settled + other pending + this settlement) past the reservation's own
|
||||
recorded `budget_usd` ceiling** (else: budget overrun — a discrepancy this
|
||||
large is escalated for review, not silently written).
|
||||
|
||||
## What a settlement records
|
||||
|
||||
A `trial-events` entry, `kind: settled`, exactly like a live execution
|
||||
produces — so `budget()` and `plan()` need no changes to recognize it — but
|
||||
with `settlement_method: external_evidence`, an `evidence_ref` pointer, the
|
||||
`reviewer` who attested the match, and the resulting `discrepancy_usd`
|
||||
(`actual - reserved`, which may be negative, zero, or positive and is always
|
||||
retained, never discarded). **No `trials` record is created.** Fabricating a
|
||||
completed model trial (a response, a verification result, a returned model)
|
||||
to close out a failed request is explicitly excluded — a settlement asserts
|
||||
only a cost, never a synthetic outcome.
|
||||
|
||||
## Reviewer and restart behavior
|
||||
|
||||
The reviewer attribution is a plain identifier (human or agent name), not a
|
||||
credential. Because the settlement check runs under the same file lock as
|
||||
`execute()` and refuses once a `settled` event exists for the reservation,
|
||||
re-running the same reconciliation after a partial failure (e.g. process
|
||||
killed after validation but before write, or after write) is safe: a retry
|
||||
either completes the original write or is refused as already-settled — never
|
||||
double-settled, never partially written (the store's `atomic`/`put` primitives
|
||||
guarantee whole-file, content-addressed writes). Uncertain or partial evidence
|
||||
(no reviewer, no evidence pointer, non-numeric or missing amount) is refused
|
||||
before any write, by the same validation used for a first attempt.
|
||||
|
||||
## Disposition
|
||||
|
||||
This contract and its implementation (`intelligence_radar/trials.reconcile`,
|
||||
CLI `reconcile`) are specified and tested in this change (IR-WP-0006-T02).
|
||||
Applying it to the real USD 0.023712 reservation (IR-WP-0006-T03) still
|
||||
requires authoritative provider billing evidence, which this repository does
|
||||
not have — that step stays blocked until such evidence exists.
|
||||
|
|
@ -185,6 +185,12 @@ def main(argv=None):
|
|||
trials_command.add_argument('--suite', type=Path, required=True)
|
||||
trials_command.add_argument('--pilot', action='store_true')
|
||||
trials_command.add_argument('--plan', action='store_true', help='review request identities and budget without calls or writes')
|
||||
reconcile_command = commands.add_parser('reconcile', help='settle a pending trial reservation from reviewed external billing evidence')
|
||||
reconcile_command.add_argument('--campaign', required=True)
|
||||
reconcile_command.add_argument('--run-id', required=True)
|
||||
reconcile_command.add_argument('--actual-usd', required=True)
|
||||
reconcile_command.add_argument('--evidence-ref', required=True, help='pointer (hash or path) to retained evidence, not a credential')
|
||||
reconcile_command.add_argument('--reviewer', required=True)
|
||||
args = parser.parse_args(argv)
|
||||
try:
|
||||
if args.command == 'trials':
|
||||
|
|
@ -192,6 +198,10 @@ def main(argv=None):
|
|||
action = plan if args.plan else run
|
||||
print(json.dumps(action(args.root, json.loads(args.config.read_text()),
|
||||
json.loads(args.suite.read_text()), args.pilot), indent=2))
|
||||
elif args.command == 'reconcile':
|
||||
from .trials import reconcile
|
||||
print(reconcile(args.root, args.campaign, args.run_id, args.actual_usd,
|
||||
args.evidence_ref, args.reviewer))
|
||||
elif args.command == 'export':
|
||||
from .bundles import export_bundle
|
||||
print(json.dumps(export_bundle(args.root, args.id, args.output), indent=2))
|
||||
|
|
|
|||
|
|
@ -168,6 +168,52 @@ def execute(root, config, case, model, opener=urlopen):
|
|||
return trial
|
||||
|
||||
|
||||
def reconcile(root, campaign, run_id, actual_usd, evidence_ref, reviewer):
|
||||
"""Settle a pending reservation from reviewed external billing evidence.
|
||||
|
||||
Never fabricates a completed trial: only a settlement event is written.
|
||||
evidence_ref is a pointer (hash or path) to retained evidence, not a
|
||||
credential or raw authenticated provider response.
|
||||
"""
|
||||
root = Path(root)
|
||||
if not isinstance(evidence_ref, str) or not evidence_ref.strip():
|
||||
raise ValueError('evidence_ref is required')
|
||||
if not isinstance(reviewer, str) or not reviewer.strip():
|
||||
raise ValueError('reviewer is required')
|
||||
actual = money(actual_usd)
|
||||
lockpath = root / '.runtime/trials.lock'
|
||||
lockpath.parent.mkdir(parents=True, exist_ok=True)
|
||||
with lockpath.open('a') as lock:
|
||||
fcntl.flock(lock, fcntl.LOCK_EX)
|
||||
events = all_records(root, 'trial-events')
|
||||
reservations = {i: e for i, e in events if e['kind'] == 'reserved'}
|
||||
settled = {e['reservation_id']: e for _, e in events if e['kind'] == 'settled'}
|
||||
match = next(((i, e) for i, e in reservations.items() if e.get('run_id') == run_id), None)
|
||||
if match is None:
|
||||
raise ValueError('unknown reservation: no matching run_id in trial-events')
|
||||
reservation_id, reservation = match
|
||||
if reservation['campaign'] != campaign:
|
||||
raise ValueError('cross-campaign reference: reservation belongs to a different campaign')
|
||||
if reservation_id in settled:
|
||||
raise ValueError('reservation already settled; duplicate or conflicting settlement refused')
|
||||
other_settled = sum((money(e['actual_usd']) for rid, e in settled.items()
|
||||
if reservations[rid]['campaign'] == campaign), Decimal(0))
|
||||
other_pending = sum((money(e['reserved_usd']) for i, e in reservations.items()
|
||||
if e['campaign'] == campaign and i != reservation_id and i not in settled),
|
||||
Decimal(0))
|
||||
limit = money(reservation['budget_usd'])
|
||||
if other_settled + other_pending + actual > limit:
|
||||
raise ValueError('settlement exceeds campaign budget authorization')
|
||||
stamp = datetime.now(timezone.utc).isoformat()
|
||||
event = {'kind': 'settled', 'campaign': campaign, 'run_id': run_id,
|
||||
'reservation_id': reservation_id, 'actual_usd': str(actual),
|
||||
'reserved_usd': reservation['reserved_usd'],
|
||||
'discrepancy_usd': str(actual - money(reservation['reserved_usd'])),
|
||||
'settlement_method': 'external_evidence', 'evidence_ref': evidence_ref,
|
||||
'reviewer': reviewer, 'settled_at': stamp}
|
||||
return put(root, 'trial-events', event)
|
||||
|
||||
|
||||
def run(root, config, suite, pilot=False):
|
||||
rows=[]
|
||||
selected=suite['cases'][:2] if pilot else suite['cases']
|
||||
|
|
|
|||
|
|
@ -6,7 +6,7 @@ import tempfile
|
|||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
from intelligence_radar.trials import budget, execute, plan, verify
|
||||
from intelligence_radar.trials import budget, execute, plan, reconcile, verify
|
||||
|
||||
|
||||
class TrialTests(unittest.TestCase):
|
||||
|
|
@ -98,3 +98,73 @@ class TrialTests(unittest.TestCase):
|
|||
plan(self.root, config, {'cases': [self.case, self.case]})
|
||||
result = plan(self.root, config, {'cases': [self.case]})
|
||||
self.assertIn('planned reservations exceed campaign budget', result['budget_blockers'])
|
||||
|
||||
def _pending_run_id(self):
|
||||
def fail(*args, **kwargs):
|
||||
raise TimeoutError('timeout')
|
||||
with self.assertRaises(TimeoutError):
|
||||
execute(self.root, self.config, self.case, self.model, opener=fail)
|
||||
return budget(self.root, 'test')
|
||||
|
||||
def test_reconcile_unknown_reservation_refused(self):
|
||||
with self.assertRaisesRegex(ValueError, 'unknown reservation'):
|
||||
reconcile(self.root, 'test', 'nope', '0.001', 'evidence-sha', 'codex')
|
||||
|
||||
def test_reconcile_settles_pending_reservation_without_fabricating_trial(self):
|
||||
self._pending_run_id()
|
||||
from intelligence_radar.store import all_records
|
||||
run_id = [e for i, e in all_records(self.root, 'trial-events') if e['kind'] == 'reserved'][0]['run_id']
|
||||
reconcile(self.root, 'test', run_id, '0.001', 'evidence-sha256:abc', 'codex')
|
||||
state = budget(self.root, 'test')
|
||||
self.assertEqual(state['pending_usd'], '0')
|
||||
self.assertEqual(state['spent_usd'], '0.001')
|
||||
trials_dir = self.root / 'data/trials'
|
||||
self.assertEqual(list(trials_dir.glob('*.json')) if trials_dir.exists() else [], [])
|
||||
|
||||
def test_reconcile_refuses_duplicate_settlement(self):
|
||||
self._pending_run_id()
|
||||
from intelligence_radar.store import all_records
|
||||
run_id = [e for i, e in all_records(self.root, 'trial-events') if e['kind'] == 'reserved'][0]['run_id']
|
||||
reconcile(self.root, 'test', run_id, '0.001', 'evidence-sha256:abc', 'codex')
|
||||
with self.assertRaisesRegex(ValueError, 'already settled'):
|
||||
reconcile(self.root, 'test', run_id, '0.001', 'evidence-sha256:abc', 'codex')
|
||||
|
||||
def test_reconcile_refuses_cross_campaign_reference(self):
|
||||
self._pending_run_id()
|
||||
from intelligence_radar.store import all_records
|
||||
run_id = [e for i, e in all_records(self.root, 'trial-events') if e['kind'] == 'reserved'][0]['run_id']
|
||||
with self.assertRaisesRegex(ValueError, 'cross-campaign'):
|
||||
reconcile(self.root, 'other-campaign', run_id, '0.001', 'evidence-sha256:abc', 'codex')
|
||||
|
||||
def test_reconcile_refuses_invalid_amount(self):
|
||||
self._pending_run_id()
|
||||
from intelligence_radar.store import all_records
|
||||
run_id = [e for i, e in all_records(self.root, 'trial-events') if e['kind'] == 'reserved'][0]['run_id']
|
||||
with self.assertRaises(ValueError):
|
||||
reconcile(self.root, 'test', run_id, '-1', 'evidence-sha256:abc', 'codex')
|
||||
|
||||
def test_reconcile_refuses_budget_overrun(self):
|
||||
self.config['budget_usd'] = '0.02'
|
||||
self._pending_run_id()
|
||||
from intelligence_radar.store import all_records
|
||||
run_id = [e for i, e in all_records(self.root, 'trial-events') if e['kind'] == 'reserved'][0]['run_id']
|
||||
with self.assertRaisesRegex(ValueError, 'exceeds campaign budget'):
|
||||
reconcile(self.root, 'test', run_id, '5', 'evidence-sha256:abc', 'codex')
|
||||
|
||||
def test_reconcile_requires_evidence_and_reviewer(self):
|
||||
self._pending_run_id()
|
||||
from intelligence_radar.store import all_records
|
||||
run_id = [e for i, e in all_records(self.root, 'trial-events') if e['kind'] == 'reserved'][0]['run_id']
|
||||
with self.assertRaisesRegex(ValueError, 'evidence_ref'):
|
||||
reconcile(self.root, 'test', run_id, '0.001', '', 'codex')
|
||||
with self.assertRaisesRegex(ValueError, 'reviewer'):
|
||||
reconcile(self.root, 'test', run_id, '0.001', 'evidence-sha256:abc', '')
|
||||
|
||||
def test_reconcile_is_safe_to_retry_after_success(self):
|
||||
self._pending_run_id()
|
||||
from intelligence_radar.store import all_records
|
||||
run_id = [e for i, e in all_records(self.root, 'trial-events') if e['kind'] == 'reserved'][0]['run_id']
|
||||
first = reconcile(self.root, 'test', run_id, '0.001', 'evidence-sha256:abc', 'codex')
|
||||
with self.assertRaisesRegex(ValueError, 'already settled'):
|
||||
reconcile(self.root, 'test', run_id, '0.001', 'evidence-sha256:abc', 'codex')
|
||||
self.assertEqual(budget(self.root, 'test')['spent_usd'], '0.001')
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@ type: workplan
|
|||
title: "Deliver admitted OpenRouter credentials to Radar"
|
||||
domain: infotech
|
||||
repo: intelligence-radar
|
||||
status: ready
|
||||
status: blocked
|
||||
flavor: implementation
|
||||
owner: codex
|
||||
topic_slug: custodian
|
||||
|
|
@ -47,3 +47,10 @@ state_hub_task_id: "c4b2378a-5241-5195-8c4b-d53ec58a7f72"
|
|||
```
|
||||
|
||||
Depends on T01 and native owner execution. Prove value-safe read-only provider key validation, correct recipient success, wrong caller/owner/tenant/path refusal, revocation and unchanged existing llm-connect delivery. Capture source/recipient revisions and sanitized native receipts. Do not widen policies or rotate the provider key merely to unblock Radar. This plan completes on admitted delivery evidence; paid inference and campaign accounting belong elsewhere.
|
||||
|
||||
Blocked, 2026-09-27: secrets-engine confirmed no ops-mason build is needed and withdrew
|
||||
the guest-clock hold (claim/consume uses a Railiance Clock trust file, not the OS clock).
|
||||
The remaining gate is human acceptance of three new apply/verify/exec approvals for the
|
||||
pinned `intelligence-radar-key-check` recipient; Radar requested secrets-engine open them
|
||||
(message `45b52d6e-7a3b-4ebe-ac15-d8137957731f`). Coordination trail:
|
||||
`docs/evidence/2026-09-27-ir-wp-0005-t01-native-delivery.md`.
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@ type: workplan
|
|||
title: "Reconcile trial billing and establish campaign accounting"
|
||||
domain: infotech
|
||||
repo: intelligence-radar
|
||||
status: ready
|
||||
status: blocked
|
||||
flavor: implementation
|
||||
owner: codex
|
||||
topic_slug: custodian
|
||||
|
|
@ -24,24 +24,35 @@ retain authority over native changes. Review baseline:
|
|||
|
||||
```task
|
||||
id: IR-WP-0006-T01
|
||||
status: todo
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "87b26b10-f4d6-5c53-9965-fae61ac0df11"
|
||||
```
|
||||
|
||||
Inventory the append-only events for ir-openrouter-pilot-20260913. Preserve the USD 0.023712 pending reservation and USD 10 total authorization. Define how provider billing evidence matches the failed request, what is unknown, who reviews a settlement, and how discrepancies are retained. HTTP 401 alone is not zero-charge proof. Accounting design can proceed independently of credential delivery; provider evidence retrieval may require an admitted read-only path.
|
||||
|
||||
Done: contract specified in `docs/evidence/2026-09-27-ir-wp-0006-t01-reconciliation-contract.md` —
|
||||
inventories the one pending reservation, defines valid billing evidence, the run_id-based match,
|
||||
the five refusal conditions, and what a settlement record must (and must not) contain.
|
||||
|
||||
## Implement and verify auditable reconciliation
|
||||
|
||||
```task
|
||||
id: IR-WP-0006-T02
|
||||
status: todo
|
||||
status: done
|
||||
priority: high
|
||||
state_hub_task_id: "1637f2ee-2369-5e78-8893-9437d2d6b6fc"
|
||||
```
|
||||
|
||||
Provide a reviewed append-only settlement procedure or command. Preserve original reservations; reject duplicate/conflicting settlement, unknown reservation, invalid amount, cross-campaign references and budget overrun. Retain source billing evidence and reviewer attribution without credentials. Test uncertain/partial evidence and restart behavior. Do not fabricate a completed model trial to settle a failed request.
|
||||
|
||||
Done: `intelligence_radar.trials.reconcile` (+ CLI `reconcile`) implements the T01 contract —
|
||||
write-once settlement under the existing trials file lock, all five refusal conditions covered,
|
||||
no `trials` record fabricated. Tests in `tests/test_trials.py`
|
||||
(`test_reconcile_*`, 9 new cases) cover unknown reservation, duplicate/retry-after-success,
|
||||
cross-campaign, invalid amount, budget overrun, and missing evidence/reviewer attribution.
|
||||
Full suite (`make check`, 48 tests) passes.
|
||||
|
||||
## Reconcile the outstanding reservation and hand off budget readiness
|
||||
|
||||
```task
|
||||
|
|
@ -52,3 +63,8 @@ state_hub_task_id: "441dc053-af7b-5a9c-866f-5b1eb804b94f"
|
|||
```
|
||||
|
||||
Depends on T01/T02 and authoritative provider evidence. Resolve the USD 0.023712 reservation only to the supported actual amount, retaining any unresolved discrepancy explicitly. Rerender pilot/full plans for the same campaign and report spent, pending and remaining authorized budget. No campaign reset, additional authorization, or paid inference is implied. Completion requires reconciled evidence and a usable accounting handoff, not merely implementation tests.
|
||||
|
||||
Blocked: the mechanism is ready (T01/T02 done), but no authoritative provider billing evidence
|
||||
exists for the pending run_id `29a46a1d6c2d9ef6cfce8de6efef75cddcfecee346c1b4babf57119f1a241d71`
|
||||
yet. Applying `reconcile` here without that evidence would be exactly the fabrication this
|
||||
workplan forbids. Waits on external evidence (likely via the IR-WP-0005 admitted read-only path).
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@ type: workplan
|
|||
title: "Complete comparable evidence, field outcomes and publication"
|
||||
domain: infotech
|
||||
repo: intelligence-radar
|
||||
status: ready
|
||||
status: blocked
|
||||
flavor: implementation
|
||||
owner: codex
|
||||
topic_slug: custodian
|
||||
|
|
@ -31,6 +31,14 @@ state_hub_task_id: "8d4a62c6-1f0e-5866-8d62-638c300a5b2e"
|
|||
|
||||
Residual IR-WP-0004-T01. Pin a concrete task and five relevant benchmark versions, harnesses/tools, effort, attempts and source captures for at least two complete profiles. Resolve the vendor-instrument versus advisor-selection mismatch. Obtain requester access evidence from IR-WP-0005 before declaring eligibility. Produce the source-backed Frontier/Efficient comparison only when completeness and comparability checks pass. Preserve unknowns and conflicting claims; provider catalog metadata alone is insufficient.
|
||||
|
||||
Blocked, 2026-09-27: requester access evidence still does not exist — IR-WP-0005-T02
|
||||
(scoped delivery verification) is the explicit prerequisite this task names, and it is
|
||||
itself blocked on a pending secrets-engine approval cycle. Pinning concrete benchmark
|
||||
versions/harnesses independent of eligibility was already attempted in the IR-WP-0004
|
||||
review (`docs/evidence/2026-09-14-ir-wp-0004-review.md`) and found to require a
|
||||
dedicated, sourced measurement campaign rather than a redo inside this session —
|
||||
inventing pinned versions without primary-source verification would violate P11.
|
||||
|
||||
## Collect and assess representative field outcomes
|
||||
|
||||
```task
|
||||
|
|
@ -42,6 +50,12 @@ state_hub_task_id: "d78ac4b1-da1b-52a8-8626-cf5c6a499b57"
|
|||
|
||||
Residual IR-WP-0004-T02. Paid execution depends on BOTH IR-WP-0005 credential readiness and IR-WP-0006 budget readiness; sampling-frame/verifier design can proceed earlier. Use the existing USD 10 campaign and reviewed pilot inputs. Define real task origins, inclusion/exclusion, held-out tasks, dependency groups, verifiers and failure taxonomy before claiming representativeness. The 20 close controlled variants are pilot evidence only. Retain successes/failures, actual costs/runtime and verifier artifacts; justify cross-task calibration and intervals before using them predictively.
|
||||
|
||||
Blocked, 2026-09-27: paid execution still needs IR-WP-0005-T02 (wait) and IR-WP-0006-T03
|
||||
(wait) both closed. The sampling-frame/verifier design portion this task allows to proceed
|
||||
earlier is nontrivial methodology work in its own right (real task origins, held-out sets,
|
||||
verifiers, failure taxonomy) and was not attempted in this session to avoid a rushed
|
||||
design under time pressure; it remains open, undecomposed, for a dedicated pass.
|
||||
|
||||
## Complete reviewed publication and operational disposition
|
||||
|
||||
```task
|
||||
|
|
@ -53,6 +67,12 @@ state_hub_task_id: "314f79fd-d95a-5e02-b67d-67e35e4334ed"
|
|||
|
||||
Residual IR-WP-0004-T03. Build on portable bundles and the public catalog adapter. Define reviewed publication artifacts linking claims to captures, revisions and corrections. Keep generated drafts distinct from reviewed publication/notifications. Decide whether continuous uptime is required; retain the local timer unless a concrete requirement justifies a hosted worker and its live owner workplan.
|
||||
|
||||
Partial, 2026-09-27: the uptime question is decided — `docs/adr/0003-operating-model-disposition.md`
|
||||
retains the local systemd timer, no hosted worker. The reviewed-publication-artifact
|
||||
definition (claim-to-capture references, exclusions, revision identity, correction linkage,
|
||||
distinct from a generated draft) is still open; it is a schema/format design decision
|
||||
better made deliberately than folded into this pass. Stays todo.
|
||||
|
||||
## Compare decomposition with measured monolithic execution
|
||||
|
||||
```task
|
||||
|
|
@ -63,3 +83,5 @@ state_hub_task_id: "e36910d6-b7c5-5dad-bc1f-1af9f472ffab"
|
|||
```
|
||||
|
||||
Depends on T02 matched held-out outcome evidence. Count orchestration, retries and verification in total cost/runtime, control task/model selection differences and retain uncertainty. No synthetic superiority or unsupported empirical quantiles. Carry any remaining research or infrastructure scope into live records before closure.
|
||||
|
||||
Blocked, 2026-09-27: no matched held-out outcome evidence exists yet (T02 unresolved).
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue