Integrate founding explorations and bootstrap operational radar
Assistant: codex Assistant-Model: gpt-6-astra Assistant-Session: 01a09c77-b1a9-7eb3-b501-6cb24b458c97
This commit is contained in:
parent
775f93b920
commit
d0e6438609
57 changed files with 4723 additions and 2 deletions
4
.claude/rules/session-protocol.md
Normal file
4
.claude/rules/session-protocol.md
Normal file
|
|
@ -0,0 +1,4 @@
|
|||
# Session protocol
|
||||
|
||||
Follow AGENTS.md. Read .custodian-brief.md and active workplans. Preserve archive
|
||||
bytes and append observations. Update file-backed work and run make sync on close.
|
||||
5
.claude/rules/stack-and-commands.md
Normal file
5
.claude/rules/stack-and-commands.md
Normal file
|
|
@ -0,0 +1,5 @@
|
|||
# Stack and commands
|
||||
|
||||
Python 3.11+ standard library. No install required. make check runs validation and
|
||||
tests; make demo executes a temporary synthetic workflow; make sync reconciles
|
||||
Statehub. python3 -m intelligence_radar --help lists operations. No deployed service.
|
||||
5
.claude/rules/workplan-convention.md
Normal file
5
.claude/rules/workplan-convention.md
Normal file
|
|
@ -0,0 +1,5 @@
|
|||
# Workplans
|
||||
|
||||
Use workplans/IR-WP-NNNN-<slug>.md with YAML identity/status frontmatter and fenced
|
||||
task blocks. Follow the format generated by statehub register. Files are canonical;
|
||||
never invent Statehub UUIDs. Keep residual work in an explicit follow-up workplan.
|
||||
27
.custodian-brief.md
Normal file
27
.custodian-brief.md
Normal file
|
|
@ -0,0 +1,27 @@
|
|||
<!-- custodian-brief: generated by statehub register; fix-consistency may replace this file -->
|
||||
# Custodian Brief - intelligence-radar
|
||||
|
||||
**Project:** intelligence-radar
|
||||
**Domain:** infotech
|
||||
**State Hub:** http://127.0.0.1:8000
|
||||
**Topic ID:** `cee7bedf-2b48-46ef-8601-006474f2ad7a`
|
||||
|
||||
## Open Workplans
|
||||
|
||||
### Bootstrap State Hub integration
|
||||
|
||||
Workplan file: `workplans/IR-WP-0001-statehub-bootstrap.md`
|
||||
|
||||
Open tasks:
|
||||
- T01 - Review generated integration files
|
||||
- T02 - Verify local developer workflow
|
||||
- T03 - Seed first real workplan
|
||||
|
||||
## Session Start
|
||||
|
||||
1. Read `INTENT.md`, `SCOPE.md`, and `AGENTS.md`.
|
||||
2. Check inbox: `GET /messages/?to_agent=intelligence-radar&unread_only=true`.
|
||||
3. Scan `workplans/`.
|
||||
4. Update task statuses in workplan files as work progresses.
|
||||
|
||||
Last generated: 2026-09-13
|
||||
7
.gitignore
vendored
Normal file
7
.gitignore
vendored
Normal file
|
|
@ -0,0 +1,7 @@
|
|||
__pycache__/
|
||||
*.py[cod]
|
||||
.venv/
|
||||
.pytest_cache/
|
||||
build/
|
||||
dist/
|
||||
*.egg-info/
|
||||
12
.repo-classification.yaml
Normal file
12
.repo-classification.yaml
Normal file
|
|
@ -0,0 +1,12 @@
|
|||
repo_classification:
|
||||
standard: Repo Classification Standard
|
||||
version: '1.0'
|
||||
classified_at: '2026-09-13'
|
||||
classified_by: codex
|
||||
category: product
|
||||
domain: infotech
|
||||
secondary_domains: [agents]
|
||||
capability_tags: [research, evaluation, documentation]
|
||||
business_stake: [technology, execution]
|
||||
business_mechanics: [operation]
|
||||
notes: Epistemic observatory with a local evidence workflow and manual capability advisor.
|
||||
222
AGENTS.md
Normal file
222
AGENTS.md
Normal file
|
|
@ -0,0 +1,222 @@
|
|||
# intelligence-radar — Agent Instructions
|
||||
|
||||
## Repo Identity
|
||||
|
||||
**Purpose:** Epistemic observatory and task-specific machine intelligence capability and economics advisor.
|
||||
|
||||
**Domain:** infotech
|
||||
**Repo slug:** intelligence-radar
|
||||
**Topic ID:** `cee7bedf-2b48-46ef-8601-006474f2ad7a`
|
||||
**Workplan prefix:** `IR-WP-`
|
||||
|
||||
---
|
||||
|
||||
## State Hub Integration
|
||||
|
||||
The Custodian State Hub tracks work across all domains. Codex uses HTTP REST and
|
||||
the `statehub` CLI by default. MCP is opt-in because the current Codex MCP bridge
|
||||
adds severe call latency; the full administrative MCP surface remains available
|
||||
to clients that need it.
|
||||
|
||||
| Context | URL |
|
||||
|---------|-----|
|
||||
| Local workstation | `http://127.0.0.1:8000` |
|
||||
| Remote via tunnel | `http://127.0.0.1:18000` |
|
||||
| Optional local edge relay | http://127.0.0.1:18080 |
|
||||
|
||||
When an operator has enabled the edge relay, set API_BASE to the relay URL.
|
||||
Queueable writes return an explicit queued receipt if the central hub is
|
||||
unreachable. Treat that as pending local evidence, then ask the operator to run
|
||||
statehub outbox status/replay after connectivity returns.
|
||||
|
||||
Codex workspace-write sandboxes need network access enabled to reach the host's
|
||||
loopback listener. Bootstrap this once with `make -C ~/state-hub configure-codex`
|
||||
and restart Codex. The canonical REST health endpoint is `/state/health`, not
|
||||
`/health`. If a sandboxed loopback probe fails, retry it with escalated execution
|
||||
before declaring State Hub unavailable; a managed Codex permission profile may
|
||||
still enforce isolated networking. Experimental MCP can be enabled explicitly
|
||||
with `make -C ~/state-hub configure-codex WITH_MCP=1`.
|
||||
|
||||
### Orient at session start
|
||||
|
||||
```bash
|
||||
# Offline brief — works without hub connection
|
||||
cat .custodian-brief.md
|
||||
|
||||
# Active workplans for this domain
|
||||
curl -s "http://127.0.0.1:8000/workplans/?topic_id=cee7bedf-2b48-46ef-8601-006474f2ad7a&status=active" \
|
||||
| python3 -m json.tool
|
||||
|
||||
# Check inbox
|
||||
curl -s "http://127.0.0.1:8000/messages/?to_agent=intelligence-radar&unread_only=true" \
|
||||
| python3 -m json.tool
|
||||
```
|
||||
|
||||
Mark a message read:
|
||||
```bash
|
||||
curl -s -X PATCH "http://127.0.0.1:8000/messages/<id>/read" \
|
||||
-H "Content-Type: application/json" -d '{}'
|
||||
```
|
||||
|
||||
### Log progress (required at session close)
|
||||
|
||||
```bash
|
||||
curl -s -X POST http://127.0.0.1:8000/progress/ \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"summary": "what was done",
|
||||
"event_type": "note",
|
||||
"author": "codex",
|
||||
"workplan_id": "<uuid>",
|
||||
"task_id": "<uuid>"
|
||||
}'
|
||||
```
|
||||
|
||||
Omit `workplan_id` / `task_id` when not applicable.
|
||||
|
||||
### Update task status
|
||||
|
||||
```bash
|
||||
curl -s -X PATCH "http://127.0.0.1:8000/tasks/<task_id>" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"status": "progress"}'
|
||||
# values: wait | todo | progress | done | cancel
|
||||
```
|
||||
|
||||
### Flag a task for human review
|
||||
|
||||
```bash
|
||||
curl -s -X PATCH "http://127.0.0.1:8000/tasks/<task_id>" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"needs_human": true, "intervention_note": "reason"}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Session Protocol
|
||||
|
||||
**Start:**
|
||||
1. `cat .custodian-brief.md` — domain goal and open workplans (offline-safe)
|
||||
2. Check inbox: `GET /messages/?to_agent=intelligence-radar&unread_only=true`; mark read
|
||||
3. Scan workplans: `ls workplans/` — note `status: ready`, `active`, or `blocked` files and open tasks
|
||||
4. Check human-needed tasks: `GET /tasks/?needs_human=true`
|
||||
|
||||
**During work:**
|
||||
- Update task statuses in workplan files as tasks progress
|
||||
- Record significant decisions via `POST /decisions/`
|
||||
|
||||
**Close:**
|
||||
1. Update workplan file task statuses to reflect progress
|
||||
2. If finishing a workplan: hand off **residuals** as live work records first
|
||||
(intake with `origin: residual` + `origin_ref: <WP-id>`, or a next workplan /
|
||||
decision / engagement). Do not park leftovers only in prose or `SCOPE.md`.
|
||||
Canon: `the-custodian/canon/standards/work-record-types_v0.1.md` § Residuals.
|
||||
3. Log: `POST /progress/` with a summary of what changed (name handoff ids)
|
||||
4. After workplan file changes, run:
|
||||
```bash
|
||||
uv run --project ~/repo-manager rmgr sync --path . --push
|
||||
```
|
||||
This assigns only missing deterministic identifiers, verifies the pushed
|
||||
Forgejo commit and `primary/railliance01`, then requests one central
|
||||
reconciliation. A queued receipt is pending evidence; rerun after
|
||||
connectivity returns. Use `statehub fix-consistency` for a separate deep audit.
|
||||
|
||||
---
|
||||
|
||||
## Credential routing
|
||||
|
||||
Before requesting credentials, use `warden route find "<need>" --json` and
|
||||
`warden route show <catalog-id> --json`. ops-warden issues SSH certificates;
|
||||
OpenBao owns API/DB secrets. Never put credentials in observations, Git or Statehub.
|
||||
The local v0.1 workflow requires no credentials.
|
||||
|
||||
<!-- REPO-AGENTS-EXTENSIONS -->
|
||||
<!-- Append repo-specific agent instructions below this marker.
|
||||
The state-hub template sync preserves content after this line. -->
|
||||
|
||||
---
|
||||
|
||||
## Workplan Convention (ADR-001)
|
||||
|
||||
Work items originate as files in this repo — not in the hub. The hub is a
|
||||
read/cache/index layer that rebuilds from files.
|
||||
|
||||
**File location:** `workplans/IR-WP-NNNN-<slug>.md`
|
||||
|
||||
**Archived location:** finished workplans may move to
|
||||
`workplans/archived/YYMMDD-IR-WP-NNNN-<slug>.md`. The `YYMMDD` prefix is
|
||||
the completion/archive date; the frontmatter `id` does not change.
|
||||
|
||||
**Ad Hoc Tasks:** small opportunistic fixes discovered during a session use
|
||||
`workplans/ADHOC-YYYY-MM-DD.md`, workplan id
|
||||
`IR-WP-ADHOC-YYYY-MM-DD`, and task ids
|
||||
`IR-WP-ADHOC-YYYY-MM-DD-T01`, etc. `IR-WP` includes its final `-WP`
|
||||
token. Unqualified historic `ADHOC-*` ids are grandfathered and must not be
|
||||
copied into new records. Use this only for low-risk work completed directly;
|
||||
create a normal workplan for anything needing analysis, design, approval,
|
||||
dependencies, or multiple phases.
|
||||
|
||||
**Frontmatter:**
|
||||
|
||||
```yaml
|
||||
---
|
||||
id: IR-WP-NNNN
|
||||
type: workplan
|
||||
title: "..."
|
||||
domain: infotech
|
||||
repo: intelligence-radar
|
||||
status: proposed | ready | active | blocked | backlog | finished | archived
|
||||
owner: codex
|
||||
topic_slug: ...
|
||||
created: "YYYY-MM-DD"
|
||||
updated: "YYYY-MM-DD"
|
||||
state_hub_workstream_id: "<uuid>" # deterministic UUIDv5; managed by Repo Manager
|
||||
---
|
||||
```
|
||||
|
||||
Use `proposed` for a new draft, `ready` after review against current repo
|
||||
state, and `finished` after implementation. `stalled` and `needs_review` are
|
||||
derived health labels, not frontmatter statuses.
|
||||
|
||||
**Terminology:** workplan is the fleet term; `workstream` appears only in legacy
|
||||
API/MCP/frontmatter bridges until `STATE-WP-0069` retires them — see
|
||||
`the-custodian/canon/standards/workplan-terminology-fleet_v0.1.md`.
|
||||
|
||||
**Task block format** (one per `##` section):
|
||||
|
||||
```
|
||||
## Task Title
|
||||
|
||||
` ` `task
|
||||
id: IR-WP-NNNN-T01
|
||||
status: wait | todo | progress | done | cancel
|
||||
priority: high | medium | low
|
||||
state_hub_task_id: "<uuid>" # deterministic UUIDv5; managed by Repo Manager
|
||||
` ` `
|
||||
|
||||
Task description text.
|
||||
```
|
||||
|
||||
Status progression: `todo` → `progress` → `done`; use `wait` for waiting/blocked work and `cancel` for stopped work.
|
||||
|
||||
**Residuals when finishing:** actionable leftovers become live work records
|
||||
before `status: finished` — usually an intake (`origin: residual`,
|
||||
`origin_ref: IR-WP-NNNN`) or a spawned workplan. Residual is a *role*,
|
||||
not a kind. Fleet list lives on State Hub, not in `SCOPE.md`.
|
||||
|
||||
To create a new workplan:
|
||||
1. Write the file following the format above
|
||||
2. Run `uv run --project ~/repo-manager rmgr sync --path . --push`.
|
||||
3. Run `statehub fix-consistency` only when a separate deep audit is needed.
|
||||
|
||||
## Intelligence Radar workflow
|
||||
|
||||
Read INTENT.md → GOAL.md → SCOPE.md → docs/integration-review.md.
|
||||
Python 3.11+ standard library; no installation required. `make check` runs tests
|
||||
and integrity checks; `make demo` runs isolated synthetic evidence; `make sync`
|
||||
uses Repo Manager to push and project workplans. No service is deployed.
|
||||
|
||||
Preserve history ZIP bytes. Treat imported catalogs and dated prose as unverified
|
||||
research leads. Append corrections as observations; never overwrite old records.
|
||||
Use TaskModelEvaluationMethod.md and versioned policies as normative; task-router
|
||||
is its manual interface. Do not equate benchmark fitness with task success.
|
||||
464
ArchitectureBlueprint.md
Normal file
464
ArchitectureBlueprint.md
Normal file
|
|
@ -0,0 +1,464 @@
|
|||
> Target architecture. The implemented local v0.1 boundary is in [SCOPE.md](SCOPE.md) and [ADR 0001](docs/adr/0001-initial-operational-baseline.md).
|
||||
|
||||
# Intelligence Radar — Architecture Blueprint
|
||||
|
||||
**Status:** Draft 0.1
|
||||
**Date:** 2026-09-13
|
||||
|
||||
## 1. Architectural objective
|
||||
|
||||
Build a system that can continuously ingest evidence about intelligence and AI capability, preserve provenance, update a structured knowledge model, and answer task-specific model-selection questions reproducibly.
|
||||
|
||||
The design separates:
|
||||
|
||||
1. **observations** — immutable source-derived facts;
|
||||
2. **knowledge** — claims and relationships built from observations;
|
||||
3. **policy** — versioned interpretation and ranking rules;
|
||||
4. **recommendations** — time-indexed outputs produced from observations + policy;
|
||||
5. **outcomes** — real task results used for calibration.
|
||||
|
||||
## 2. High-level architecture
|
||||
|
||||
```text
|
||||
┌──────────────────────┐
|
||||
│ Primary Sources │
|
||||
│ papers/docs/repos │
|
||||
│ labs/regulators/etc. │
|
||||
└──────────┬───────────┘
|
||||
│
|
||||
discovery / fetch / poll
|
||||
│
|
||||
┌─────────────────▼─────────────────┐
|
||||
│ Ingestion Plane │
|
||||
│ source registry · fetchers │
|
||||
│ dedupe · provenance · extraction │
|
||||
└─────────────────┬─────────────────┘
|
||||
│
|
||||
immutable observations
|
||||
│
|
||||
┌───────────────────────────▼───────────────────────────┐
|
||||
│ Knowledge Plane │
|
||||
│ events · claims · evidence · models · deployments │
|
||||
│ benchmarks · prices · hardware · regulations │
|
||||
│ task outcomes · historical versions │
|
||||
└───────────────┬───────────────────────┬──────────────┘
|
||||
│ │
|
||||
compendium synthesis capability data
|
||||
│ │
|
||||
┌─────────────▼─────────┐ ┌────────▼────────────────┐
|
||||
│ Intelligence │ │ Capability Advisor │
|
||||
│ Compendium │ │ task profiler │
|
||||
│ claim graph │ │ benchmark selector │
|
||||
│ theory maps │ │ model scorer │
|
||||
└─────────────┬────────┘ │ economics estimator │
|
||||
│ │ workplan optimizer │
|
||||
│ └──────────┬───────────────┘
|
||||
│ │
|
||||
┌─────────▼───────────────────────▼────────┐
|
||||
│ Publication Plane │
|
||||
│ daily delta · weekly digest · query API │
|
||||
│ recommendation report · dashboards │
|
||||
└───────────────────┬──────────────────────┘
|
||||
│
|
||||
▼
|
||||
execution / feedback
|
||||
│
|
||||
┌───────────▼───────────┐
|
||||
│ Outcome & Calibration│
|
||||
│ task-performance │
|
||||
│ ledger │
|
||||
└───────────┬───────────┘
|
||||
└──────► model calibration
|
||||
```
|
||||
|
||||
## 3. Data layers
|
||||
|
||||
### 3.1 Source registry
|
||||
|
||||
A versioned registry describing what should be monitored.
|
||||
|
||||
Fields:
|
||||
|
||||
- source id;
|
||||
- publisher/person;
|
||||
- source type;
|
||||
- URL/API/feed;
|
||||
- topic tags;
|
||||
- trust tier;
|
||||
- primary/secondary;
|
||||
- expected cadence;
|
||||
- retrieval method;
|
||||
- license/copyright constraints;
|
||||
- last successful observation;
|
||||
- importance;
|
||||
- notes.
|
||||
|
||||
### 3.2 Raw observation store
|
||||
|
||||
Append-only.
|
||||
|
||||
Stores:
|
||||
|
||||
- retrieval timestamp;
|
||||
- publication/event timestamp;
|
||||
- source URI;
|
||||
- content hash;
|
||||
- document metadata;
|
||||
- extracted structured facts;
|
||||
- retrieval agent/version;
|
||||
- link to raw artifact where legally appropriate;
|
||||
- provenance.
|
||||
|
||||
Never overwrite a historical observation because a source page later changes.
|
||||
|
||||
### 3.3 Event store
|
||||
|
||||
Normalized events such as:
|
||||
|
||||
- model released;
|
||||
- model deprecated;
|
||||
- benchmark version released;
|
||||
- benchmark score published;
|
||||
- price changed;
|
||||
- provider added/removed;
|
||||
- paper published;
|
||||
- regulation enacted/applies;
|
||||
- hardware announced/shipped;
|
||||
- safety incident;
|
||||
- capability claim;
|
||||
- benchmark invalidation/saturation evidence.
|
||||
|
||||
### 3.4 Claim graph
|
||||
|
||||
Compendium entities and evidence relationships described in `IntelligenceConceptModel.md`.
|
||||
|
||||
### 3.5 Capability catalog
|
||||
|
||||
Contains separate entities for:
|
||||
|
||||
- `model`;
|
||||
- `execution_profile`;
|
||||
- `deployment`;
|
||||
- `benchmark`;
|
||||
- `benchmark_version`;
|
||||
- `benchmark_result`;
|
||||
- `price_observation`;
|
||||
- `availability_observation`;
|
||||
- `hardware/system_result`.
|
||||
|
||||
### 3.6 Task-performance ledger
|
||||
|
||||
Stores real task predictions and outcomes.
|
||||
|
||||
This becomes the calibration dataset for task-specific performance prediction.
|
||||
|
||||
## 4. Suggested initial physical implementation
|
||||
|
||||
Start boring and auditable:
|
||||
|
||||
- **Git** for normative Markdown, policies, schemas, hand-curated catalog seeds, and digest artifacts.
|
||||
- **PostgreSQL** for indexed structured observations, claim relationships, benchmark results, model/deployment catalog, recommendation history, and task-outcome ledger.
|
||||
- **Object storage** for larger source artifacts and generated reports.
|
||||
- **Parquet/Iceberg later** when observation volume, historical analytics, or multi-engine processing warrants it.
|
||||
- **Embeddings/vector search** as an index over evidence, never as the canonical source of truth.
|
||||
|
||||
The architecture should not require a graph database initially. PostgreSQL can represent graph edges cleanly; a specialized graph engine can be added when query patterns justify it.
|
||||
|
||||
## 5. Ingestion pipeline
|
||||
|
||||
```text
|
||||
discover
|
||||
→ fetch
|
||||
→ fingerprint
|
||||
→ deduplicate
|
||||
→ classify
|
||||
→ extract candidate facts/events
|
||||
→ resolve entities
|
||||
→ attach provenance
|
||||
→ confidence checks
|
||||
→ append observation
|
||||
→ emit event
|
||||
→ candidate claim update
|
||||
```
|
||||
|
||||
### Agentic extraction rule
|
||||
|
||||
Agents may propose structured facts and claims.
|
||||
|
||||
They must not silently turn their inference into source fact.
|
||||
|
||||
Every extracted field should carry one of:
|
||||
|
||||
- `explicit_source_statement`
|
||||
- `derived_from_source`
|
||||
- `cross_source_inference`
|
||||
- `system_estimate`
|
||||
|
||||
## 6. Daily Radar pipeline
|
||||
|
||||
Daily runs should answer:
|
||||
|
||||
1. What new source artifacts appeared?
|
||||
2. What facts/events are genuinely new?
|
||||
3. What existing observations were corrected or contradicted?
|
||||
4. Did any benchmark materially change?
|
||||
5. Did price, availability, context limit, or model status change?
|
||||
6. Did the capability frontier move?
|
||||
7. Did the cost/performance Pareto frontier move?
|
||||
8. Did any event affect a Compendium claim?
|
||||
9. Did any current recommendation policy become questionable?
|
||||
|
||||
Output should be a **delta**, not a generic summary of the news.
|
||||
|
||||
## 7. Weekly synthesis pipeline
|
||||
|
||||
The weekly digest ranks events by estimated significance.
|
||||
|
||||
Recommended significance dimensions:
|
||||
|
||||
- epistemic impact;
|
||||
- capability impact;
|
||||
- economic impact;
|
||||
- breadth of affected tasks;
|
||||
- confidence;
|
||||
- novelty;
|
||||
- persistence;
|
||||
- governance/social impact.
|
||||
|
||||
The weekly synthesis should explicitly state:
|
||||
|
||||
> What do we have more reason to believe than seven days ago?
|
||||
|
||||
and:
|
||||
|
||||
> What do we have less reason to believe?
|
||||
|
||||
## 8. Capability Advisor pipeline
|
||||
|
||||
```text
|
||||
task
|
||||
↓
|
||||
Task Interpreter
|
||||
↓
|
||||
Clarity + constraints + risk class
|
||||
↓
|
||||
Capability Profile
|
||||
↓
|
||||
Benchmark Candidate Retrieval
|
||||
↓
|
||||
Five-Benchmark Selector
|
||||
↓
|
||||
Result Freshness / Validity Filter
|
||||
↓
|
||||
Execution Profile Candidate Builder
|
||||
↓
|
||||
Task-Fit Estimator
|
||||
↓
|
||||
Availability Filter
|
||||
↓
|
||||
Cost + Runtime Estimator
|
||||
↓
|
||||
Frontier / Topnotch / Efficient / Sufficient / Budget
|
||||
↓
|
||||
Confidence decomposition
|
||||
↓
|
||||
optional Workplan Optimizer
|
||||
```
|
||||
|
||||
## 9. Workplan optimizer
|
||||
|
||||
The optimizer converts a task into a DAG.
|
||||
|
||||
Each node contains:
|
||||
|
||||
- subtask;
|
||||
- inputs/outputs;
|
||||
- capability profile;
|
||||
- dependency edges;
|
||||
- acceptance/verifier;
|
||||
- parallelism constraints;
|
||||
- privacy/tool constraints;
|
||||
- candidate execution profiles;
|
||||
- predicted success;
|
||||
- cost distribution;
|
||||
- runtime distribution.
|
||||
|
||||
It should generate several plans, at minimum:
|
||||
|
||||
1. **Monolithic Frontier**
|
||||
2. **Monolithic Efficient**
|
||||
3. **Decomposed Quality**
|
||||
4. **Decomposed Efficient**
|
||||
5. **Decomposed Budget**
|
||||
|
||||
Optimization objectives can include:
|
||||
|
||||
- maximize predicted success subject to budget;
|
||||
- minimize expected cost subject to success threshold;
|
||||
- minimize p90 runtime subject to quality threshold;
|
||||
- maximize quality-adjusted value.
|
||||
|
||||
### Verification as a first-class subtask
|
||||
|
||||
Do not allocate all budget to generation.
|
||||
|
||||
A cheaper producer plus an independent verifier can outperform an expensive monolithic attempt when errors are detectable.
|
||||
|
||||
## 10. Confidence model
|
||||
|
||||
Store components independently:
|
||||
|
||||
- `task_clarity`
|
||||
- `benchmark_coverage`
|
||||
- `benchmark_validity`
|
||||
- `measurement_reliability`
|
||||
- `result_freshness`
|
||||
- `deployment_match`
|
||||
- `price_confidence`
|
||||
- `task_calibration_evidence`
|
||||
- `overall_recommendation_confidence`
|
||||
|
||||
The overall confidence number is a convenience. The components are the actual explanation.
|
||||
|
||||
## 11. Temporal semantics
|
||||
|
||||
Every mutable external fact must support:
|
||||
|
||||
- `observed_at`
|
||||
- `valid_from` if known
|
||||
- `valid_to` if known
|
||||
- `source_published_at`
|
||||
- `supersedes_observation_id` where applicable.
|
||||
|
||||
A model price without a timestamp is not valid capability-economics data.
|
||||
|
||||
## 12. Policy versioning
|
||||
|
||||
Examples:
|
||||
|
||||
- benchmark selection policy;
|
||||
- task capability ontology;
|
||||
- benchmark-score normalization;
|
||||
- Topnotch threshold;
|
||||
- sufficiency threshold inference;
|
||||
- cost model;
|
||||
- confidence aggregation;
|
||||
- source trust;
|
||||
- model eligibility.
|
||||
|
||||
Every recommendation stores policy hashes/versions.
|
||||
|
||||
Historical recommendations therefore remain reproducible.
|
||||
|
||||
## 13. API surface — first draft
|
||||
|
||||
### `POST /evaluate-task`
|
||||
|
||||
Returns task profile and model recommendation.
|
||||
|
||||
### `POST /optimize-workplan`
|
||||
|
||||
Returns model-allocated DAG alternatives.
|
||||
|
||||
### `POST /record-outcome`
|
||||
|
||||
Adds verified outcome evidence.
|
||||
|
||||
### `GET /models`
|
||||
|
||||
Current model catalog.
|
||||
|
||||
### `GET /deployments`
|
||||
|
||||
Provider-specific availability and economics.
|
||||
|
||||
### `GET /benchmarks`
|
||||
|
||||
Benchmark catalog and health.
|
||||
|
||||
### `GET /radar/events`
|
||||
|
||||
Normalized recent events.
|
||||
|
||||
### `GET /digests/{period}`
|
||||
|
||||
Daily/weekly Radar products.
|
||||
|
||||
### `GET /compendium/claims/{id}`
|
||||
|
||||
Claim + evidence graph.
|
||||
|
||||
## 14. Minimal recommendation record
|
||||
|
||||
```yaml
|
||||
task_id: task-...
|
||||
evaluated_at: 2026-09-13T...
|
||||
policy:
|
||||
task_profile: v0.1
|
||||
benchmark_selection: v0.1
|
||||
model_ranking: v0.1
|
||||
clarity:
|
||||
score: 0.78
|
||||
ambiguities: []
|
||||
benchmarks:
|
||||
- id: ...
|
||||
weight: 0.27
|
||||
frontier:
|
||||
execution_profile: ...
|
||||
topnotch: [...]
|
||||
efficient: ...
|
||||
sufficient: [...]
|
||||
budget: ...
|
||||
economics:
|
||||
currency: USD
|
||||
expected: ...
|
||||
p90: ...
|
||||
runtime:
|
||||
expected_seconds: ...
|
||||
confidence:
|
||||
overall: ...
|
||||
components: ...
|
||||
evidence_gaps: [...]
|
||||
```
|
||||
|
||||
## 15. Initial milestones
|
||||
|
||||
### M0 — Repository kernel
|
||||
|
||||
Documents, schemas, policies, source catalog.
|
||||
|
||||
### M1 — Dated model/economics catalog
|
||||
|
||||
Automated collection of model releases, availability, context limits, and API prices from major providers.
|
||||
|
||||
### M2 — Benchmark health catalog
|
||||
|
||||
Benchmark metadata, versions, results, saturation and contamination notes.
|
||||
|
||||
### M3 — Task profiler
|
||||
|
||||
Produces stable capability profiles and clarity assessment.
|
||||
|
||||
### M4 — Five-benchmark selector
|
||||
|
||||
Produces explainable benchmark mapping with independence penalties.
|
||||
|
||||
### M5 — Model ranking
|
||||
|
||||
Frontier / Topnotch / Efficient / Sufficient / Budget with cost intervals.
|
||||
|
||||
### M6 — Workplan optimizer
|
||||
|
||||
DAG decomposition, heterogeneous model allocation, verification, parallel runtime estimate.
|
||||
|
||||
### M7 — Outcome ledger
|
||||
|
||||
Capture actual task success, cost, runtime, and failure modes.
|
||||
|
||||
### M8 — Calibration engine
|
||||
|
||||
Learn task-cluster performance models from the outcome ledger.
|
||||
|
||||
### M9 — Self-improving Radar
|
||||
|
||||
Detect benchmark drift/saturation and automatically propose policy/catalog changes for review.
|
||||
262
BenchmarkCatalog.md
Normal file
262
BenchmarkCatalog.md
Normal file
|
|
@ -0,0 +1,262 @@
|
|||
> Imported research draft. Benchmark versions, availability, and source claims require verification before operational use.
|
||||
|
||||
# Benchmark Catalog — Initial Canon
|
||||
|
||||
**Status:** Draft 0.1
|
||||
**Snapshot date:** 2026-09-13
|
||||
|
||||
## 1. Purpose
|
||||
|
||||
The Benchmark Catalog is not a list of scores. It is a registry of **measurement instruments**.
|
||||
|
||||
Each benchmark record should describe:
|
||||
|
||||
- intended construct;
|
||||
- task distribution;
|
||||
- modality;
|
||||
- interaction style;
|
||||
- agent/tool environment;
|
||||
- scoring;
|
||||
- versions;
|
||||
- freshness;
|
||||
- saturation;
|
||||
- contamination risk;
|
||||
- reproducibility;
|
||||
- ecological validity;
|
||||
- scaffold dependence;
|
||||
- cost to run;
|
||||
- public/private test status;
|
||||
- observed correlations with other benchmarks;
|
||||
- observed predictive value for Intelligence Radar task outcomes.
|
||||
|
||||
## 2. Health dimensions
|
||||
|
||||
Suggested 0–1 dimensions:
|
||||
|
||||
- `construct_relevance`
|
||||
- `ecological_validity`
|
||||
- `measurement_reliability`
|
||||
- `discriminative_power`
|
||||
- `freshness`
|
||||
- `reproducibility`
|
||||
- `contamination_resistance`
|
||||
- `execution_profile_similarity`
|
||||
- `coverage_breadth`
|
||||
|
||||
No single health score should replace the components, though a summary can be generated.
|
||||
|
||||
## 3. Initial benchmark candidates
|
||||
|
||||
### Humanity's Last Exam (HLE)
|
||||
|
||||
**Constructs:** broad expert academic knowledge/reasoning, multimodal closed-ended reasoning.
|
||||
**Strength:** deliberately difficult broad frontier evaluation.
|
||||
**Risk:** closed-ended academic performance is not equivalent to agentic work; rapid frontier gains can reduce discrimination over time.
|
||||
**Status:** active/watch for saturation.
|
||||
|
||||
Source: Nature, *A benchmark of expert-level academic questions to assess AI capabilities*, published 2026-01-28.
|
||||
|
||||
### ARC-AGI-2
|
||||
|
||||
**Constructs:** abstraction, novel-rule induction, test-time adaptation, sample-efficient generalization.
|
||||
**Strength:** novel tasks with minimal prior task-specific knowledge; explicit efficiency orientation.
|
||||
**Risk:** narrow task formalism; uncertain transfer to many real workloads.
|
||||
**Status:** active.
|
||||
|
||||
Source: ARC Prize official ARC-AGI-2 materials and 2026 competition.
|
||||
|
||||
### SWE-bench Verified
|
||||
|
||||
**Constructs:** repository-level software issue resolution.
|
||||
**Environment:** code repository + tests; agent/scaffold heavily matters.
|
||||
**Strength:** real GitHub issues; executable verification.
|
||||
**Risk:** benchmark and scaffold familiarity; Python-heavy Verified set; high scores reduce discrimination.
|
||||
**Status:** active/watch.
|
||||
|
||||
Official site states Verified is a human-filtered 500-instance subset.
|
||||
|
||||
### SWE-bench Multilingual
|
||||
|
||||
**Constructs:** real software engineering across multiple languages.
|
||||
**Strength:** 300 curated tasks from 42 repositories across 9 languages.
|
||||
**Risk:** still a specific issue-resolution distribution.
|
||||
**Status:** active.
|
||||
|
||||
### Terminal-Bench
|
||||
|
||||
**Constructs:** agent competence on hard terminal tasks.
|
||||
**Strength:** execution-based tasks in sandboxed terminal environments.
|
||||
**Risk:** version comparability and agent harness effects must be tracked carefully.
|
||||
**Status:** active; version must always be recorded.
|
||||
|
||||
Artificial Analysis moved its aggregate index to Terminal-Bench v4.0 in September 2026, illustrating rapid benchmark version churn.
|
||||
|
||||
### Berkeley Function Calling Leaderboard (BFCL) V4
|
||||
|
||||
**Constructs:** tool/function calling and broader agentic tool use.
|
||||
**Strength:** explicit function/tool-use evaluation, real-world contributed functions, multi-turn and agentic evolution.
|
||||
**Risk:** native function calling vs prompted emulation are distinct execution conditions.
|
||||
**Status:** active.
|
||||
|
||||
Official leaderboard describes V4 as moving from function calling toward holistic agentic evaluation.
|
||||
|
||||
### τ³-bench
|
||||
|
||||
**Constructs:** tool-agent-user interaction; knowledge-aware and voice variants.
|
||||
**Strength:** evaluates coordination with dynamic users/tools rather than isolated single-turn answers.
|
||||
**Risk:** domain/task versioning materially affects comparability.
|
||||
**Status:** active.
|
||||
|
||||
A July 2026 grading update explicitly made affected pre-1.0.1 and post-1.0.1 results non-comparable.
|
||||
|
||||
### OSWorld 2.0
|
||||
|
||||
**Constructs:** multimodal computer use in real operating-system environments.
|
||||
**Strength:** execution in computer environments.
|
||||
**Risk:** environment configuration, UI drift, and agent harness matter.
|
||||
**Status:** active.
|
||||
|
||||
OSWorld announced version 2.0 in June 2026.
|
||||
|
||||
### BrowseComp
|
||||
|
||||
**Constructs:** persistent web research/navigation to find difficult information.
|
||||
**Strength:** tests search persistence and information discovery.
|
||||
**Risk:** short verifiable answers omit much of real research quality, synthesis, ambiguity handling, and citation practice.
|
||||
**Status:** active.
|
||||
|
||||
The original dataset contains 1,266 difficult browsing questions.
|
||||
|
||||
### METR Task-Completion Time Horizon
|
||||
|
||||
**Constructs:** autonomy/reliability versus human-expert task duration.
|
||||
**Strength:** converts heterogeneous software tasks into an interpretable time-horizon measure at fixed success probability.
|
||||
**Risk:** software-task distribution; estimates above the tested horizon become unreliable.
|
||||
**Status:** active.
|
||||
|
||||
METR's Time Horizon 1.1 page was last updated 2026-05-08 and explicitly warns that measurements above 16 hours are unreliable with the current task suite.
|
||||
|
||||
### GDPval / GDPval-AA v2
|
||||
|
||||
**Constructs:** economically valuable professional knowledge work across occupations.
|
||||
**Strength:** realistic work products including documents, spreadsheets, diagrams, and slides.
|
||||
**Risk:** judge-model sensitivity and agent harness effects; GDPval-AA is an independent evaluation implementation rather than the original benchmark alone.
|
||||
**Status:** active.
|
||||
|
||||
Artificial Analysis' GDPval-AA v2 uses 220 tasks from the larger GDPval dataset, shell/web tools, and pairwise judging.
|
||||
|
||||
### AA-Briefcase
|
||||
|
||||
**Constructs:** long-horizon agentic knowledge work.
|
||||
**Strength:** private evaluation set helps contamination resistance.
|
||||
**Risk:** externally non-reproducible/private; methodology provider dependence.
|
||||
**Status:** active as independent evidence, not canonical truth.
|
||||
|
||||
Included in Artificial Analysis Intelligence Index v4.2+.
|
||||
|
||||
### AutomationBench-AA
|
||||
|
||||
**Constructs:** agentic business workflow automation.
|
||||
**Strength:** hundreds of workflows across simulated business applications with a private question set.
|
||||
**Risk:** implementation-specific; private test set.
|
||||
**Status:** active as independent evidence.
|
||||
|
||||
Added to Artificial Analysis Intelligence Index v4.3 in September 2026.
|
||||
|
||||
### SciCode
|
||||
|
||||
**Constructs:** scientific reasoning translated into research-level code.
|
||||
**Strength:** scientist-curated problems from realistic scientific workflows.
|
||||
**Risk:** specialized scientific-programming distribution.
|
||||
**Status:** active.
|
||||
|
||||
Original benchmark contains research problems across physics, mathematics, materials science, biology, and chemistry.
|
||||
|
||||
### MLPerf Inference
|
||||
|
||||
**Constructs:** systems-level inference performance and efficiency, not semantic intelligence directly.
|
||||
**Strength:** standardized hardware/software serving measurement; indispensable for self-hosting economics.
|
||||
**Risk:** workload benchmark results are not direct task-quality rankings.
|
||||
**Status:** active.
|
||||
|
||||
MLPerf Inference v6.0 added/updated major LLM workloads including GPT-OSS 120B and an interactive DeepSeek-R1 scenario.
|
||||
|
||||
## 4. Benchmarks intentionally not treated as timeless
|
||||
|
||||
A benchmark can lose selection priority when:
|
||||
|
||||
- frontier systems approach ceiling;
|
||||
- training contamination becomes plausible;
|
||||
- test cases become widely optimized against;
|
||||
- scorer errors are discovered;
|
||||
- benchmark version changes invalidate comparability;
|
||||
- the benchmark ceases to predict real Intelligence Radar outcomes;
|
||||
- a more ecologically valid instrument becomes available.
|
||||
|
||||
The 2026 Stanford AI Index explicitly reports rapid benchmark saturation and evaluation-quality concerns. That is a design requirement for Intelligence Radar, not background trivia.
|
||||
|
||||
## 5. Five-benchmark selection rule
|
||||
|
||||
Given task `T`, candidate benchmark `b`, and already-selected set `S`:
|
||||
|
||||
```text
|
||||
base(b,T) =
|
||||
0.30 relevance_to_task
|
||||
+ 0.15 ecological_validity
|
||||
+ 0.12 measurement_reliability
|
||||
+ 0.12 discriminative_power
|
||||
+ 0.10 freshness
|
||||
+ 0.10 execution_profile_similarity
|
||||
+ 0.06 reproducibility
|
||||
+ 0.05 contamination_resistance
|
||||
|
||||
marginal(b,T,S) =
|
||||
base(b,T)
|
||||
- redundancy_penalty(b,S)
|
||||
- known_bias_penalty(b,T)
|
||||
```
|
||||
|
||||
Select greedily or via constrained optimization until five measurements are chosen.
|
||||
|
||||
Then normalize task weights across the five:
|
||||
|
||||
`weight_i = positive_relevance_i / Σ positive_relevance`
|
||||
|
||||
The system must return:
|
||||
|
||||
- raw relevance;
|
||||
- final weight;
|
||||
- why selected;
|
||||
- what it fails to measure;
|
||||
- independence/redundancy notes;
|
||||
- benchmark version;
|
||||
- most recent result date used.
|
||||
|
||||
## 6. Capability-to-benchmark examples
|
||||
|
||||
| Task characteristic | Strong candidate evidence |
|
||||
|---|---|
|
||||
| Repository issue resolution | SWE-bench Verified / Multilingual |
|
||||
| Terminal/DevOps work | Terminal-Bench |
|
||||
| Tool/API orchestration | BFCL V4 |
|
||||
| User+tool interactive workflow | τ³-bench |
|
||||
| Computer GUI use | OSWorld 2.0 |
|
||||
| Deep web research | BrowseComp |
|
||||
| Long autonomous software tasks | METR time horizon |
|
||||
| Professional deliverables | GDPval-AA v2 / AA-Briefcase |
|
||||
| Business-app automation | AutomationBench-AA |
|
||||
| Scientific programming | SciCode |
|
||||
| Novel abstract induction | ARC-AGI-2 |
|
||||
| Broad expert knowledge | HLE |
|
||||
| Self-hosted inference economics | MLPerf Inference |
|
||||
|
||||
## 7. Required next catalog work
|
||||
|
||||
1. Add machine-readable records for every benchmark.
|
||||
2. Capture version-specific score semantics.
|
||||
3. Estimate benchmark correlations from shared model results.
|
||||
4. Add saturation indicators.
|
||||
5. Add benchmark-cost estimates.
|
||||
6. Add contamination/public-test exposure indicators.
|
||||
7. Record model/scaffold/provider exact configuration for every result.
|
||||
8. Add predictive validity against the Task → Model Performance Ledger.
|
||||
7
CLAUDE.md
Normal file
7
CLAUDE.md
Normal file
|
|
@ -0,0 +1,7 @@
|
|||
# Intelligence Radar
|
||||
|
||||
Read @AGENTS.md, @INTENT.md and @SCOPE.md.
|
||||
|
||||
@.claude/rules/session-protocol.md
|
||||
@.claude/rules/workplan-convention.md
|
||||
@.claude/rules/stack-and-commands.md
|
||||
12
GOAL.md
Normal file
12
GOAL.md
Normal file
|
|
@ -0,0 +1,12 @@
|
|||
# Current goal
|
||||
|
||||
Establish a reproducible, manually operated v0.1 research facility from the two
|
||||
2026-09-13 explorations, with source provenance, one coherent advisory method,
|
||||
working evidence/digest tooling, and Coulomb/Statehub lifecycle integration.
|
||||
|
||||
Acceptance: archives retain checksums; governing documents have explicit precedence;
|
||||
evidence can be recorded and checked; daily and weekly draft digests can be generated;
|
||||
tests pass; repo identity and file-backed workplans are registered and synced.
|
||||
|
||||
Next stage: IR-WP-0003 covers verified source ingestion, calibrated advisory data,
|
||||
and unattended scheduling. The larger architecture remains a target.
|
||||
296
INTENT.md
Normal file
296
INTENT.md
Normal file
|
|
@ -0,0 +1,296 @@
|
|||
# INTENT — Intelligence Radar
|
||||
|
||||
**Status:** Adopted baseline 0.1 (2026-09-13)
|
||||
**Date:** 2026-09-13
|
||||
|
||||
## 1. Intent
|
||||
|
||||
Intelligence Radar exists to improve our ability to reason about **intelligence itself** and to make practical decisions about **where, when, and at what price machine intelligence can be applied**.
|
||||
|
||||
The facility shall continuously answer two different classes of questions:
|
||||
|
||||
### Epistemic questions
|
||||
|
||||
- What do humans mean when they use the word *intelligence*?
|
||||
- Which models of intelligence exist across biology, cognitive science, cybernetics, neuroscience, psychology, computer science, AI, economics, organizational theory, collective intelligence, and related disciplines?
|
||||
- Which claims are supported, contested, speculative, superseded, or refuted?
|
||||
- Which observations or experiments would distinguish competing models?
|
||||
- Where are we confusing measurement convenience with the underlying construct?
|
||||
|
||||
### Operational questions
|
||||
|
||||
For a concrete task:
|
||||
|
||||
- What capabilities does the task require?
|
||||
- Which five current benchmarks provide the most useful evidence about those capabilities, and with what relevance?
|
||||
- Which available execution profile is the **Frontier** choice?
|
||||
- Which profiles are **Topnotch** — at most 10% predicted degradation from Frontier?
|
||||
- Which Topnotch profile is the cheapest expected solution — the **Efficient** choice?
|
||||
- Which profiles are credibly **Sufficient** to satisfy the task?
|
||||
- Which Sufficient profile is the cheapest — the **Budget** choice?
|
||||
- How clear is the task?
|
||||
- How confident are we in the benchmark mapping, capability estimate, availability, cost estimate, and final recommendation?
|
||||
- What is the expected cost and runtime?
|
||||
- Would decomposition into subtasks and collaboration among specialized models improve quality, confidence, cost, or runtime?
|
||||
|
||||
## 2. Scope
|
||||
|
||||
Intelligence Radar spans:
|
||||
|
||||
- biological intelligence;
|
||||
- cognition and neuroscience;
|
||||
- psychometrics and measurement;
|
||||
- cybernetics, control, adaptation, and viability;
|
||||
- embodied and situated intelligence;
|
||||
- collective, organizational, swarm, and social intelligence;
|
||||
- artificial intelligence and machine learning;
|
||||
- foundation models and generative systems;
|
||||
- agents and multi-agent systems;
|
||||
- software and systems engineering for intelligent systems;
|
||||
- AI accelerators, inference systems, networks, storage, and energy;
|
||||
- capability evaluation and benchmark science;
|
||||
- economics, price, availability, and accessibility of intelligence;
|
||||
- safety, security, alignment, reliability, and misuse;
|
||||
- economic and social effects;
|
||||
- law, regulation, standards, and public policy.
|
||||
|
||||
The facility must not equate *AI progress* with *progress in understanding intelligence*.
|
||||
|
||||
## 3. Three planes
|
||||
|
||||
### 3.1 Intelligence Compendium
|
||||
|
||||
A durable, versioned epistemic graph describing concepts, theories, claims, evidence, objections, thinkers, experiments, systems, and historical development.
|
||||
|
||||
It answers:
|
||||
|
||||
> What does humanity currently have reason to believe about intelligence?
|
||||
|
||||
### 3.2 Intelligence Radar
|
||||
|
||||
A high-frequency event and delta layer.
|
||||
|
||||
It answers:
|
||||
|
||||
> What changed, and which previous beliefs, measurements, policies, capability estimates, or economic assumptions should change because of it?
|
||||
|
||||
### 3.3 Capability Advisor
|
||||
|
||||
A task-specific decision system.
|
||||
|
||||
It answers:
|
||||
|
||||
> What intelligence should be used for this task, under these conditions, today?
|
||||
|
||||
## 4. Normative vocabulary
|
||||
|
||||
### Model
|
||||
|
||||
A base trained model, e.g. a named model family/version.
|
||||
|
||||
### Execution Profile
|
||||
|
||||
A model plus the conditions that materially affect measured capability:
|
||||
|
||||
- reasoning/effort level;
|
||||
- prompting/scaffold;
|
||||
- system instructions;
|
||||
- agent harness;
|
||||
- available tools;
|
||||
- context strategy;
|
||||
- retrieval;
|
||||
- memory;
|
||||
- sampling parameters;
|
||||
- number of attempts or ensemble strategy.
|
||||
|
||||
### Deployment
|
||||
|
||||
An execution profile bound to practical availability:
|
||||
|
||||
- provider;
|
||||
- API/product;
|
||||
- region;
|
||||
- service tier;
|
||||
- rate limits;
|
||||
- context/output limits;
|
||||
- quantization or serving stack;
|
||||
- price;
|
||||
- latency;
|
||||
- data-handling constraints;
|
||||
- availability date/status.
|
||||
|
||||
**Benchmarks primarily measure execution profiles, not Platonic model capability.**
|
||||
|
||||
### Frontier
|
||||
|
||||
The eligible execution profile with the highest predicted task performance, regardless of cost.
|
||||
|
||||
### Topnotch
|
||||
|
||||
Eligible profiles whose predicted task performance is at most 10% below Frontier under the active ranking policy.
|
||||
|
||||
Default policy:
|
||||
|
||||
`task_score(profile) >= 0.90 × task_score(frontier)`
|
||||
|
||||
### Efficient
|
||||
|
||||
The Topnotch profile with the lowest expected total task cost.
|
||||
|
||||
### Sufficient
|
||||
|
||||
A profile whose predicted probability of satisfying the task's acceptance criteria exceeds the task's declared or inferred sufficiency threshold.
|
||||
|
||||
### Budget
|
||||
|
||||
The Sufficient profile with the lowest expected total task cost.
|
||||
|
||||
### Available Frontier
|
||||
|
||||
The highest-performing profile the requester can actually use under jurisdiction, access, privacy, product, and deployment constraints.
|
||||
|
||||
## 5. Foundational principles
|
||||
|
||||
### P1 — Intelligence is contextual
|
||||
|
||||
A capability exists relative to an environment, objective, resource budget, interface, and observation method.
|
||||
|
||||
### P2 — Do not collapse intelligence into one scalar
|
||||
|
||||
Aggregate indices are useful summaries, not ontologies.
|
||||
|
||||
### P3 — Benchmarks are instruments
|
||||
|
||||
Every benchmark has a construct, validity envelope, error model, saturation state, contamination risk, ecological validity, and expiration risk.
|
||||
|
||||
### P4 — Five benchmarks are selected per task
|
||||
|
||||
There is no permanent canonical five. The selector chooses five task-relevant measurements while penalizing redundancy.
|
||||
|
||||
If evidence coverage is poor, the system still returns its best five candidates but explicitly marks the mapping as weak.
|
||||
|
||||
### P5 — Prefer independent evidence
|
||||
|
||||
Five correlated benchmarks do not constitute five independent observations.
|
||||
|
||||
### P6 — Distinguish observed fact from policy
|
||||
|
||||
Raw benchmark results, prices, availability observations, and task outcomes are facts with provenance.
|
||||
|
||||
`topnotch degradation <= 10%`, benchmark weighting, sufficiency thresholds, and confidence aggregation are policies and must be versioned.
|
||||
|
||||
### P7 — Price is time indexed
|
||||
|
||||
Promotional rates, negotiated rates, region, provider, caching, batch/flex tiers, token lengths, and tool charges can change the economics drastically.
|
||||
|
||||
### P8 — Cost per token is not cost per solved task
|
||||
|
||||
Expected cost includes retries, reasoning tokens, tool calls, cache behavior, failures, and verification.
|
||||
|
||||
### P9 — Outcome evidence outranks leaderboard intuition
|
||||
|
||||
Public benchmarks seed predictions. Verified real-world outcomes on structurally similar tasks should progressively calibrate or override generic benchmark priors.
|
||||
|
||||
### P10 — Preserve disagreement
|
||||
|
||||
The Compendium records competing hypotheses and unresolved contradictions rather than prematurely synthesizing them away.
|
||||
|
||||
### P11 — Primary sources first
|
||||
|
||||
Secondary reporting discovers events. Durable claims should resolve to primary research, official documentation, benchmark maintainers, regulator texts, standards, or first-party measurements whenever possible.
|
||||
|
||||
### P12 — Decomposition is an optimization problem
|
||||
|
||||
A task may be better solved by a workplan of heterogeneous models than by one global Frontier model.
|
||||
|
||||
## 6. Core service contract
|
||||
|
||||
Input:
|
||||
|
||||
- task/question;
|
||||
- supplied context;
|
||||
- constraints;
|
||||
- required output;
|
||||
- acceptance criteria, if known;
|
||||
- risk/reliability requirements;
|
||||
- environment/tool constraints;
|
||||
- optional budget/runtime limits.
|
||||
|
||||
Output:
|
||||
|
||||
1. **Task interpretation**
|
||||
2. **Task clarity score** and missing/ambiguous information
|
||||
3. **Capability profile**
|
||||
4. **Five selected benchmarks**
|
||||
- relevance weight;
|
||||
- rationale;
|
||||
- measurement quality;
|
||||
- data freshness;
|
||||
- known weaknesses
|
||||
5. **Candidate execution profiles**
|
||||
6. **Frontier**
|
||||
7. **Topnotch set**
|
||||
8. **Efficient**
|
||||
9. **Sufficient set**
|
||||
10. **Budget**
|
||||
11. **Expected cost** with p50/p90 or equivalent interval
|
||||
12. **Expected runtime** with uncertainty
|
||||
13. **Recommendation confidence**
|
||||
14. **Evidence gaps**
|
||||
15. **Optional optimized workplan**
|
||||
16. **Comparison against monolithic execution**
|
||||
17. **Post-execution outcome record** when results become available
|
||||
|
||||
## 7. Feedback loop
|
||||
|
||||
Every completed task should be eligible to create a calibration observation:
|
||||
|
||||
- canonical task profile;
|
||||
- selected benchmarks and policy versions;
|
||||
- predicted performance;
|
||||
- chosen execution profile;
|
||||
- actual model/provider configuration;
|
||||
- input/output/cached/reasoning token use;
|
||||
- tool calls;
|
||||
- retries;
|
||||
- elapsed and active runtime;
|
||||
- acceptance-test results;
|
||||
- verifier/judge outcomes;
|
||||
- human assessment;
|
||||
- failure mode;
|
||||
- environmental context.
|
||||
|
||||
These observations form the **Task → Model Performance Ledger**.
|
||||
|
||||
The long-term goal is:
|
||||
|
||||
> Replace generic benchmark inference with empirically calibrated predictions for classes of real work while retaining benchmarks as transferable prior evidence.
|
||||
|
||||
## 8. Success criteria
|
||||
|
||||
Intelligence Radar succeeds when it can:
|
||||
|
||||
- explain why a recommendation was made;
|
||||
- reproduce a historical recommendation from its dated evidence and policy versions;
|
||||
- detect when a benchmark ceases to predict real outcomes;
|
||||
- distinguish model capability from scaffolding and deployment effects;
|
||||
- identify cost/performance Pareto frontiers rather than merely rank models;
|
||||
- provide actionable model choices for tasks with explicit uncertainty;
|
||||
- reduce cost without materially reducing task success;
|
||||
- improve its own predictions from observed outcomes;
|
||||
- maintain a useful, non-dogmatic map of scientific and philosophical models of intelligence.
|
||||
|
||||
## 9. Non-goals
|
||||
|
||||
Intelligence Radar is not:
|
||||
|
||||
- a universal AGI score;
|
||||
- a model fan leaderboard;
|
||||
- a news-content farm;
|
||||
- an automatic claim that benchmark performance equals real-world competence;
|
||||
- an authority deciding what intelligence “really is”;
|
||||
- a system that hides uncertainty behind precise-looking numbers.
|
||||
|
||||
## 10. One-line description
|
||||
|
||||
> **Intelligence Radar continuously maps what intelligence appears to be, what intelligent systems can demonstrably do, and the cheapest reliable way to apply those capabilities to a concrete task.**
|
||||
377
IntelligenceConceptModel.md
Normal file
377
IntelligenceConceptModel.md
Normal file
|
|
@ -0,0 +1,377 @@
|
|||
# Intelligence Concept Model
|
||||
|
||||
**Status:** Draft 0.1
|
||||
**Purpose:** Provide a substrate-neutral conceptual frame for the Intelligence Compendium without freezing a single definition of intelligence.
|
||||
|
||||
## 1. Starting stance
|
||||
|
||||
Intelligence should be modeled first as an **observed relation among system, environment, objective/viability conditions, feedback, adaptation, and resource constraints**, not as a single scalar substance residing inside an agent.
|
||||
|
||||
A useful minimal frame is:
|
||||
|
||||
`System ↔ Environment ↔ Objective/Viability ↔ Feedback ↔ Adaptation ↔ Resource Budget`
|
||||
|
||||
Different disciplines emphasize different parts of that relation.
|
||||
|
||||
Psychometrics may operationalize intelligence through performance covariance.
|
||||
Cybernetics emphasizes regulation and requisite variety.
|
||||
Neuroscience investigates biological mechanisms.
|
||||
Cognitive science studies representation, learning, prediction, decision, and action.
|
||||
AI engineering studies produced task capability.
|
||||
Evolutionary theory can treat intelligence as adaptive strategy.
|
||||
Collective-intelligence research may assign intelligent behavior to groups whose components individually possess much less capability.
|
||||
|
||||
The Compendium should make these differences explicit.
|
||||
|
||||
## 2. Core entities
|
||||
|
||||
### Concept
|
||||
|
||||
A named idea whose meaning may vary by discipline.
|
||||
|
||||
Examples: intelligence, cognition, agency, learning, representation, reasoning, memory, adaptation.
|
||||
|
||||
### Definition
|
||||
|
||||
A scoped interpretation of a Concept associated with an author, tradition, field, or measurement practice.
|
||||
|
||||
### Theory / Model of Intelligence
|
||||
|
||||
A structured explanatory model connecting concepts and making claims about intelligent behavior.
|
||||
|
||||
### Claim
|
||||
|
||||
An addressable proposition that can be supported, challenged, refined, or superseded.
|
||||
|
||||
Example:
|
||||
|
||||
> Generalization to novel tasks is a necessary component of intelligence.
|
||||
|
||||
The system does not mark this merely `true/false`; it records provenance and epistemic state.
|
||||
|
||||
### Evidence
|
||||
|
||||
Observation, experiment, dataset, mathematical result, benchmark result, replication, case study, or other artifact relevant to a claim.
|
||||
|
||||
### Objection / Counterevidence
|
||||
|
||||
Evidence or reasoning that weakens, scopes, or contradicts a claim.
|
||||
|
||||
### Measurement
|
||||
|
||||
An operationalization intended to estimate one or more constructs.
|
||||
|
||||
A benchmark is a Measurement.
|
||||
|
||||
### System
|
||||
|
||||
The entity whose behavior is being discussed or measured.
|
||||
|
||||
Examples:
|
||||
|
||||
- human;
|
||||
- animal;
|
||||
- brain subsystem;
|
||||
- organization;
|
||||
- swarm;
|
||||
- language model;
|
||||
- model+agent scaffold;
|
||||
- robot;
|
||||
- multi-agent collective;
|
||||
- market;
|
||||
- hybrid human-AI team.
|
||||
|
||||
### Environment
|
||||
|
||||
The relevant external state and interaction surface.
|
||||
|
||||
### Objective / Viability Condition
|
||||
|
||||
What counts as success, persistence, regulation, fitness, goal achievement, or desired behavior.
|
||||
|
||||
Not all theories require an explicit goal.
|
||||
|
||||
### Resource Budget
|
||||
|
||||
Compute, energy, time, memory, data, money, communication, physical resources, attention.
|
||||
|
||||
### Thinker / Organization / Tradition
|
||||
|
||||
Provenance and intellectual context, not automatic epistemic authority.
|
||||
|
||||
## 3. Epistemic states
|
||||
|
||||
Recommended claim states:
|
||||
|
||||
- `observed` — direct measurement/observation with stated conditions;
|
||||
- `supported` — substantial converging evidence;
|
||||
- `contested` — credible conflicting evidence or interpretations;
|
||||
- `speculative` — plausible but weakly supported;
|
||||
- `historical` — important in development of the field, not necessarily current;
|
||||
- `superseded` — substantially replaced by a stronger formulation;
|
||||
- `refuted` — evidence strongly contradicts the stated claim within its scope;
|
||||
- `unknown` — unresolved / insufficient evidence.
|
||||
|
||||
Confidence belongs to the **claim-evidence relationship**, not merely the claim.
|
||||
|
||||
## 4. Substrate classes
|
||||
|
||||
The model should be neutral across at least:
|
||||
|
||||
- biological;
|
||||
- artificial;
|
||||
- embodied artificial;
|
||||
- collective/social;
|
||||
- organizational;
|
||||
- swarm;
|
||||
- ecological/evolutionary;
|
||||
- hybrid human-machine;
|
||||
- synthetic/virtual;
|
||||
- hypothesized non-human/non-terrestrial.
|
||||
|
||||
The classification is descriptive and non-exclusive.
|
||||
|
||||
## 5. Capability dimensions
|
||||
|
||||
These dimensions are not asserted to be independent. They are indexing coordinates that let competing theories be compared.
|
||||
|
||||
### 5.1 Perception
|
||||
|
||||
Acquiring task-relevant information from the environment.
|
||||
|
||||
### 5.2 Representation
|
||||
|
||||
Maintaining internal or distributed state that supports useful discrimination or action.
|
||||
|
||||
Some theories deny that explicit representations are necessary; this disagreement should be preserved.
|
||||
|
||||
### 5.3 Memory
|
||||
|
||||
Persistence and retrieval of information across time.
|
||||
|
||||
### 5.4 Prediction / World Modeling
|
||||
|
||||
Estimating possible or likely future states and latent causes.
|
||||
|
||||
### 5.5 Reasoning / Inference
|
||||
|
||||
Transforming information into conclusions or decisions under stated rules or learned structure.
|
||||
|
||||
### 5.6 Learning / Adaptation
|
||||
|
||||
Changing behavior or internal organization as a consequence of experience or feedback.
|
||||
|
||||
### 5.7 Generalization / Transfer
|
||||
|
||||
Applying acquired structure successfully outside the exact distribution or tasks in which it was learned.
|
||||
|
||||
### 5.8 Planning
|
||||
|
||||
Selecting sequences of actions against future consequences.
|
||||
|
||||
### 5.9 Control / Regulation
|
||||
|
||||
Maintaining desired variables or viability conditions under disturbance.
|
||||
|
||||
### 5.10 Tool Use
|
||||
|
||||
Extending capability through external artifacts and interfaces.
|
||||
|
||||
### 5.11 Agency
|
||||
|
||||
Coherent pursuit of objectives over time.
|
||||
|
||||
Agency can be graded and context-dependent.
|
||||
|
||||
### 5.12 Autonomy Horizon
|
||||
|
||||
How long or complex a trajectory a system can execute without corrective external intervention at a specified success probability.
|
||||
|
||||
### 5.13 Embodiment
|
||||
|
||||
Dependence on physical interaction, morphology, sensorimotor coupling, or situatedness.
|
||||
|
||||
### 5.14 Social / Cooperative Intelligence
|
||||
|
||||
Modeling, communicating with, coordinating with, teaching, persuading, or learning from other agents.
|
||||
|
||||
### 5.15 Collective Intelligence
|
||||
|
||||
Capability emerging at group/system level that cannot be reduced trivially to a representative component.
|
||||
|
||||
### 5.16 Metacognition / Self-Modeling
|
||||
|
||||
Estimating one's knowledge, uncertainty, limits, goals, or state.
|
||||
|
||||
### 5.17 Creativity / Search
|
||||
|
||||
Producing useful novelty or discovering structures not directly specified.
|
||||
|
||||
### 5.18 Robustness
|
||||
|
||||
Maintaining useful behavior under perturbation, uncertainty, adversarial inputs, or distribution shift.
|
||||
|
||||
### 5.19 Reliability
|
||||
|
||||
Probability of producing acceptable outcomes under repeated comparable conditions.
|
||||
|
||||
### 5.20 Resource Efficiency
|
||||
|
||||
Capability achieved per unit of energy, compute, memory, data, time, money, or other scarce resource.
|
||||
|
||||
## 6. Systems-theory lens
|
||||
|
||||
A reusable analytical frame:
|
||||
|
||||
| Element | Question |
|
||||
|---|---|
|
||||
| Boundary | What is considered part of the intelligent system? |
|
||||
| Environment | What variety and disturbances must it handle? |
|
||||
| Sensors | What information can cross into the system? |
|
||||
| Actuators | What can the system change? |
|
||||
| State | What relevant information persists? |
|
||||
| Model | What regularities does it exploit? |
|
||||
| Feedback | How is error/success observed? |
|
||||
| Adaptation | What changes after feedback? |
|
||||
| Objective | What states count as desirable or viable? |
|
||||
| Timescale | Over what horizon is capability evaluated? |
|
||||
| Resources | What scarcity constrains behavior? |
|
||||
| Observer | Who defines success and what can they measure? |
|
||||
|
||||
This makes explicit an important problem:
|
||||
|
||||
> Apparent intelligence can move across the chosen boundary.
|
||||
|
||||
A language model with search, code execution, memory, retrieval, and an agent harness is a different system from the underlying frozen network.
|
||||
|
||||
Likewise, a human with notebooks, search engines, colleagues, and institutions is not operationally equivalent to an isolated human.
|
||||
|
||||
## 7. Intelligence as Pareto surface
|
||||
|
||||
Avoid asking only:
|
||||
|
||||
> How intelligent is system X?
|
||||
|
||||
Prefer:
|
||||
|
||||
> What capability surface does system X exhibit over tasks, environments, reliability levels, time horizons, and resource budgets?
|
||||
|
||||
Conceptually:
|
||||
|
||||
`Capability = f(task, environment, tools, horizon, resources, reliability_target, system_configuration)`
|
||||
|
||||
This turns “intelligence” from a scalar-ranking temptation into a multidimensional empirical object.
|
||||
|
||||
## 8. Distinguish capability, competence, performance, and outcome
|
||||
|
||||
### Capability
|
||||
|
||||
Latent ability inferred from evidence.
|
||||
|
||||
### Competence
|
||||
|
||||
Capability under reasonably favorable and relevant conditions.
|
||||
|
||||
### Performance
|
||||
|
||||
Observed result in a specific evaluation instance.
|
||||
|
||||
### Outcome
|
||||
|
||||
Whether the actual real-world objective was achieved.
|
||||
|
||||
A benchmark observes performance and is used to infer capability. It does not directly reveal capability without assumptions.
|
||||
|
||||
## 9. Measurement model
|
||||
|
||||
Every measurement record should state:
|
||||
|
||||
- construct(s) intended to be measured;
|
||||
- target system/execution profile;
|
||||
- environment;
|
||||
- task distribution;
|
||||
- scoring rule;
|
||||
- reliability;
|
||||
- sampling uncertainty;
|
||||
- contamination risk;
|
||||
- saturation state;
|
||||
- ecological validity;
|
||||
- scaffold dependence;
|
||||
- reproducibility;
|
||||
- known failure modes;
|
||||
- version/date.
|
||||
|
||||
A benchmark result without these qualifiers is incomplete evidence.
|
||||
|
||||
## 10. Compendium graph relationships
|
||||
|
||||
Recommended edges:
|
||||
|
||||
- `defines`
|
||||
- `refines`
|
||||
- `contradicts`
|
||||
- `supports`
|
||||
- `challenges`
|
||||
- `operationalizes`
|
||||
- `measures`
|
||||
- `predicts`
|
||||
- `depends_on`
|
||||
- `generalizes`
|
||||
- `supersedes`
|
||||
- `historically_influenced`
|
||||
- `replicates`
|
||||
- `fails_to_replicate`
|
||||
- `applies_to_substrate`
|
||||
- `assumes_boundary`
|
||||
- `requires_resource`
|
||||
- `observed_in`
|
||||
|
||||
## 11. First research clusters
|
||||
|
||||
### A — Adaptation, control, and cybernetics
|
||||
|
||||
Ashby, Wiener, Beer, requisite variety, viable-system thinking, adaptive control.
|
||||
|
||||
### B — Psychometrics and cognitive abilities
|
||||
|
||||
g, hierarchical models, working memory, processing speed, reasoning, measurement validity.
|
||||
|
||||
### C — Computation and learning
|
||||
|
||||
Turing, algorithmic views, statistical learning, representation learning, reinforcement learning.
|
||||
|
||||
### D — Predictive and embodied cognition
|
||||
|
||||
Active inference, ecological cognition, enactivism, sensorimotor approaches.
|
||||
|
||||
### E — Biological intelligence
|
||||
|
||||
Neural computation, learning systems, evolution, animal cognition.
|
||||
|
||||
### F — Collective and organizational intelligence
|
||||
|
||||
Swarm systems, markets, institutions, organizations, cultural accumulation.
|
||||
|
||||
### G — Artificial generality and transfer
|
||||
|
||||
Generalization, abstraction, program induction, ARC-style reasoning, meta-learning.
|
||||
|
||||
### H — Foundation models and agents
|
||||
|
||||
Scaling, in-context learning, reasoning, tool use, memory, planning, computer use, multi-agent systems.
|
||||
|
||||
### I — Measurement science
|
||||
|
||||
Benchmark validity, contamination, saturation, gaming, judge reliability, reproducibility.
|
||||
|
||||
### J — Economics of intelligence
|
||||
|
||||
Marginal cost, inference efficiency, substitution, complements, availability, latency, energy, and quality-adjusted price.
|
||||
|
||||
## 12. Open question that should remain open
|
||||
|
||||
A central research question for the Compendium is deliberately unresolved:
|
||||
|
||||
> Is intelligence best understood as a property, a capability distribution, a process, a relation, an adaptive control phenomenon, an emergent systems property, or some useful combination of these views?
|
||||
|
||||
Intelligence Radar should become better at mapping the space of answers rather than prematurely choosing one.
|
||||
151
LICENSE
Normal file
151
LICENSE
Normal file
|
|
@ -0,0 +1,151 @@
|
|||
# Target Revenue Source License
|
||||
|
||||
**Version 1.0, Candidate 1 (V1C1)**
|
||||
|
||||
---
|
||||
|
||||
> **PRELIMINARY CANDIDATE — SUBJECT TO CHANGE — NOT FINAL**
|
||||
>
|
||||
> This repository is governed by the Target Revenue Source License
|
||||
> (TRSL), Version 1, Candidate 1. This is the framework's first working
|
||||
> candidate, adopted as the org's preliminary operating license across
|
||||
> all repos (maintainer decision, 2026-07-29) during the build/alpha
|
||||
> stage — see `SCOPE.md` §1 and Appendix A of the canonical text (link
|
||||
> below) for the alpha/beta risk-acceptance decision this reflects and
|
||||
> what it does and does not mean. Full specialist legal review is
|
||||
> explicitly deferred until the framework moves out of beta.
|
||||
>
|
||||
> **Canonical source, full candidate-status banner, and Appendix A
|
||||
> (non-normative candidate notes tracking every open item):** the
|
||||
> `coulomb` org's `target-revenue` repository,
|
||||
> `specs/TargetRevenueSourceLicense-V1C1.md` — this file is the operative
|
||||
> legal text (Preamble through Section 11) only; the canonical document
|
||||
> is authoritative if this copy and that document ever diverge.
|
||||
|
||||
---
|
||||
|
||||
## Preamble
|
||||
|
||||
This Target Revenue Source License ("**License**") governs the Software identified in the applicable Phase Manifest. It implements the Target Revenue Framework: a defined development Phase accumulates Development Credit and Remission Credit against an immutable Initial Target until the Milestone Release automatically and irrevocably converts to a declared permissive Future License.
|
||||
|
||||
Commercial beneficiaries fund the creation and early availability of a software improvement; once the declared target is satisfied, the governed release becomes permissively open source.
|
||||
|
||||
## 1. Definitions
|
||||
|
||||
Capitalized terms used in this License have the meanings given below. Where a term is also defined in the Phase Manifest or Target Ledger for a specific Phase, the Phase Manifest and Target Ledger govern the *values* (amounts, dates, identifiers) and this License governs the *legal effect* of those values — the two must not be read as conflicting definitions of the same concept.
|
||||
|
||||
**"Commercial Entitlement"** means a right, purchased or otherwise granted under a Commercial Use Agreement, to make Commercial Use of the Software during a Phase.
|
||||
|
||||
**"Commercial Use"** means billing, invoicing, or otherwise charging any customer a fee, subscription, license fee, or other consideration for or in connection with use of the Software, at any time before the Conversion Event for the applicable Phase, regardless of whether the person or organization billed would otherwise qualify for Noncommercial Use. Commercial Use occurs by virtue of such billing alone, whether or not the resulting payment is registered with the Trust Service; in particular, billing a customer for pre-conversion use of the Software without recording the corresponding payment in the applicable Phase's Target Ledger is Commercial Use without a valid Commercial Entitlement — a violation of Section 3, addressed under Section 7 and, where applicable, the Enforcement Network described in the canonical repository's `specs/EnforcementNetworkConcept.md`.
|
||||
|
||||
**"Commercial Use Agreement"** means the separate agreement, referenced by the applicable Phase Manifest, under which a Commercial Entitlement is purchased or granted. This License does not itself set pricing, metering, or payment terms — those are governed by the Commercial Use Agreement.
|
||||
|
||||
**"Conversion Event"** means the moment the Outstanding Target for a Phase reaches zero, as computed from the Phase Manifest and Target Ledger per the Target Ledger Specification. The Conversion Event occurs automatically and is not conditioned on any declaration, attestation, or other act by the Licensor or any Trust Service.
|
||||
|
||||
**"Development Credit"** means the portion of a collected and settled payment explicitly allocated toward satisfying the Initial Target of a specific Phase, as recorded in that Phase's Target Ledger.
|
||||
|
||||
**"Future License"** means the permissive license identified in the applicable Phase Manifest, being either the MIT License or the Apache License, Version 2.0, which applies to the Milestone Release upon the Conversion Event.
|
||||
|
||||
**"Initial Target"** means the immutable monetary target declared for a Phase in its Phase Manifest.
|
||||
|
||||
**"Licensor"** means **Binky Hedgehog GmbH**, the party that publishes the Phase Manifest and holds the rights necessary to grant this License and the Future License for the Milestone Release.
|
||||
|
||||
**"Milestone Release"** means the precisely identified software release designated in the applicable Phase Manifest, identified by an immutable source revision, release artifact, or cryptographic digest.
|
||||
|
||||
**"Noncommercial Use"** means use of the Software for personal purposes, private study, hobby or amateur projects; use by any charitable organization, educational institution, public research organization, or government institution acting in a non-revenue-generating capacity; or other use of a materially similar character.
|
||||
|
||||
**"Outstanding Target"** means, at any time, `max(0, Initial Target − cumulative Development Credit − cumulative Remission Credit)` for a Phase, as computed from that Phase's Target Ledger.
|
||||
|
||||
**"Phase"** means a bounded development undertaking governed by one Initial Target, one Milestone Release, one degeneration policy, and one Future License declaration, as declared in a Phase Manifest.
|
||||
|
||||
**"Phase Manifest"** means the published, immutable declaration identifying a Phase, its Milestone Release, Initial Target, Future License, degeneration policy, and Target Ledger location, as specified in the Phase Manifest Specification.
|
||||
|
||||
**"Remission Credit"** means a transparent, non-revenue reduction of a Phase's Outstanding Target, generated under that Phase's published degeneration policy and recorded in the Target Ledger.
|
||||
|
||||
**"Settled Payment"** means a payment that has cleared through its payment processor and is no longer subject to reversal in the ordinary course (chargeback, dispute, or equivalent), as further specified by the applicable Commercial Use Agreement or monetization extension.
|
||||
|
||||
**"Software"** means the source code, object code, and associated documentation of the Milestone Release identified in the applicable Phase Manifest.
|
||||
|
||||
**"Target Ledger"** means the append-only record of Development Credit, Remission Credit, and correction entries for a Phase, as specified in the Target Ledger Specification.
|
||||
|
||||
**"You"** or **"Licensee"** means the individual or entity exercising rights under this License.
|
||||
|
||||
## 2. Grant of Rights for Noncommercial Use
|
||||
|
||||
Subject to the terms of this License, the Licensor grants You a worldwide, royalty-free, non-exclusive license, during the applicable Phase, to:
|
||||
|
||||
(a) use, reproduce, and study the Software for any Noncommercial Use;
|
||||
|
||||
(b) modify the Software and create derivative works of it for any Noncommercial Use; and
|
||||
|
||||
(c) redistribute the Software and Your modifications, in source or object form, for any Noncommercial Use, provided that You include this License, unmodified, with any such redistribution, and that You do not remove or alter any copyright, patent, trademark, or attribution notices contained in the Software.
|
||||
|
||||
This grant does not extend to Commercial Use. Commercial Use requires a Commercial Entitlement under Section 3.
|
||||
|
||||
## 3. Commercial Use
|
||||
|
||||
You may not make Commercial Use of the Software during the applicable Phase unless You hold a valid, current Commercial Entitlement under a Commercial Use Agreement with the Licensor covering the applicable Phase. A Commercial Entitlement granted under one Phase's Commercial Use Agreement does not extend to a later Phase's Milestone Release unless the Commercial Use Agreement expressly says so.
|
||||
|
||||
This Section 3 states the existence and boundary of the commercial-use restriction. It does not itself set pricing, invoicing, metering, audit rights, or payment terms — those are governed exclusively by the applicable Commercial Use Agreement.
|
||||
|
||||
## 4. Patent License
|
||||
|
||||
Subject to the terms of this License, each contributor to the Software grants You, during the applicable Phase and solely to the extent of rights granted under Sections 2 and 3, a perpetual (subject to the termination below), worldwide, non-exclusive, no-charge, royalty-free patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Software, limited to those patent claims licensable by that contributor that are necessarily infringed by their contribution(s) alone or by combination of their contribution(s) with the Software.
|
||||
|
||||
If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Software or a contribution incorporated within it constitutes direct or contributory patent infringement, then any patent licenses granted to You under this Section 4 for the Software shall terminate as of the date such litigation is filed.
|
||||
|
||||
## 5. Automatic Conversion to the Future License
|
||||
|
||||
**5.1 Automatic effect.** Upon the Conversion Event for a Phase, the rights and restrictions in Sections 3 (Commercial Use) of this License, as they apply to that Phase's Milestone Release, terminate automatically. In their place, the Milestone Release is licensed under the Future License identified in that Phase's Phase Manifest, effective as of the Conversion Event, without any further act, declaration, or attestation required by the Licensor, any Trust Service, or any other party.
|
||||
|
||||
**5.2 Irrevocability.** Once a valid Conversion Event has occurred for a Phase, no subsequent refund, chargeback, accounting correction, dispute, or termination of this License for an unrelated breach shall revoke, suspend, or otherwise impair the Future License grant for that Phase's Milestone Release. Any shortfall or dispute arising after a Conversion Event is a commercial or accounting matter between the relevant parties and does not reinstate a commercial-use restriction over already-converted Software.
|
||||
|
||||
**5.3 Prior freedom preserved.** A later Phase covering subsequent improvements to the Software does not restrict, withdraw, or otherwise affect the rights granted under the Future License for an earlier Phase's already-converted Milestone Release.
|
||||
|
||||
**5.4 Evidence, not cause.** A Trust Service may publish a Conversion Attestation documenting a Conversion Event. Such an attestation is evidence that the Conversion Event occurred; it is not a condition of, and its absence or delay does not postpone, the automatic effect described in Section 5.1. Any person may independently verify whether a Conversion Event has occurred directly from the Phase Manifest and Target Ledger.
|
||||
|
||||
## 6. Successive Phases
|
||||
|
||||
The Licensor may declare a new Phase covering subsequent improvements to the Software following a Milestone Release's Conversion Event. Each Phase is independently governed by its own Phase Manifest, Initial Target, degeneration policy, and Target Ledger. Nothing in a later Phase's Phase Manifest may be construed to reduce or withdraw rights already granted under Section 5 for an earlier Phase's Milestone Release.
|
||||
|
||||
## 7. Term and Termination
|
||||
|
||||
**7.1 Term.** This License applies to the Software for the duration of the applicable Phase, and, for the Milestone Release, indefinitely following that Phase's Conversion Event under the Future License.
|
||||
|
||||
**7.2 Termination for breach.** If You breach Section 3 (Commercial Use) or Section 2(c) (redistribution notice requirement), the Licensor may terminate this License as to You. Before such termination becomes effective, the Licensor shall provide You written notice of the breach; if You cure the breach within thirty (30) days of that notice, this License continues in effect. A second breach of the same provision within twelve (12) months may be terminated immediately without a further cure opportunity.
|
||||
|
||||
**7.3 Effect of termination.** Termination under this Section 7 affects only Your rights under Sections 2 and 3 for the Phase in which the breach occurred. It does not affect any rights already vested under Section 5 (Automatic Conversion) for a Milestone Release whose Conversion Event has already occurred, per Section 5.2.
|
||||
|
||||
**7.4 Public record of breach and resolution.** The Licensor shall cause the Trust Service to publish, as part of the public record for the affected Phase, notice of: (a) any breach notice issued under Section 7.2, stating the general nature of the breach and the date of notice; (b) whether the breach was cured within the applicable cure period, and the date of cure; and (c) any termination determination made under this Section 7, including its effective date and scope. This public record exists to give the ecosystem a transparent, verifiable conformity signal for the Phase, distinct from and in addition to the Development Credit and Remission Credit facts already published under Section 5.4 and the Target Ledger Specification.
|
||||
|
||||
A breach that You dispute, and that has not been finally determined, shall be recorded as **alleged**; it shall be recorded as **determined** only once the cure period has run without cure, or the dispute has been resolved against You under the applicable Commercial Use Agreement's dispute process, if any. The Trust Service shall update the record promptly upon resolution in either direction. Recording an alleged or determined breach under this Section 7.4 is a ministerial act of publishing the Licensor's determination (or a dispute process's outcome); it does not give the Trust Service discretionary authority to decide whether a breach occurred, consistent with Section 5.4's evidence-not-cause principle.
|
||||
|
||||
Whether, and under what conditions, the public record identifies a Commercial Entitlement holder by name is governed exclusively by the applicable Commercial Use Agreement, which the Licensor and that Commercial Entitlement holder negotiate and agree to directly. This License does not itself set a naming default. Where no Commercial Use Agreement addresses the question, or where the affected party has no Commercial Use Agreement at all (for example, a Section 2(c) breach by a Noncommercial Use licensee), the public record states the Phase and breach category only, without naming the party.
|
||||
|
||||
## 8. Disclaimer of Warranty
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NONINFRINGEMENT. THE LICENSOR DOES NOT WARRANT THAT THE SOFTWARE WILL BE ERROR-FREE OR THAT ANY PHASE WILL REACH ITS CONVERSION EVENT.
|
||||
|
||||
## 9. Limitation of Liability
|
||||
|
||||
IN NO EVENT SHALL THE LICENSOR OR ANY CONTRIBUTOR BE LIABLE FOR ANY CLAIM, DAMAGES, OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT, OR OTHERWISE, ARISING FROM, OUT OF, OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE, EXCEPT TO THE EXTENT SUCH LIMITATION IS PROHIBITED BY APPLICABLE LAW.
|
||||
|
||||
## 10. Trademarks
|
||||
|
||||
This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary attribution.
|
||||
|
||||
## 11. General Provisions
|
||||
|
||||
**11.1 Governing law and venue.** Adopted for alpha/beta 2026-07-29: disputes arising under this License shall be resolved by binding arbitration, seated at a neutral, arbitration-mature venue (Singapore or London are the two candidates concretely supported by current research), rather than by litigation in a national court. The specific arbitral institution and substantive governing law remain a per-deployment blank pending final selection; they are not fixed by this candidate. See the canonical repository's `history/260729-TRSL-Jurisdiction-Synthesis.md` §2.
|
||||
|
||||
**11.2 Severability.** If any provision of this License is held unenforceable, the remaining provisions remain in full force, and the unenforceable provision shall be reformed to the minimum extent necessary to make it enforceable.
|
||||
|
||||
**11.3 No waiver.** Failure to enforce any provision of this License is not a waiver of future enforcement of that or any other provision.
|
||||
|
||||
**11.4 Entire agreement (as to licensing).** This License, together with the applicable Phase Manifest and, where applicable, the Commercial Use Agreement, constitutes the entire agreement between You and the Licensor regarding the Software's licensing terms. Operations, service, and consulting arrangements are governed by separate agreements, if any, and are not part of this License.
|
||||
|
||||
**11.5 Definitions control.** Marketing materials, documentation, or other non-normative communications about the Software must not describe pre-Conversion-Event Software as "Open Source," "free software," or "open core." Pre-conversion Noncommercial Use is **source-available**; pre-conversion Commercial Use requires a **Commercial Entitlement**; only post-conversion Software may be described as Open Source, under the Future License.
|
||||
|
||||
---
|
||||
|
||||
**No Phase is currently declared for this repository under this License.** Until a Phase Manifest is published and registered with the Trust Service for a Milestone Release in this repository, Sections 2–7 above have no operative subject matter here — this License establishes the governing framework in advance of that declaration, consistent with the org-wide rollout decision recorded in `target-revenue`'s `workplans/TREV-WP-0008-governance-and-pilot-rollout.md`.
|
||||
12
Makefile
Normal file
12
Makefile
Normal file
|
|
@ -0,0 +1,12 @@
|
|||
.PHONY: check test demo sync
|
||||
check: test
|
||||
python3 -m intelligence_radar check
|
||||
|
||||
test:
|
||||
python3 -m unittest discover -s tests -v
|
||||
|
||||
demo:
|
||||
python3 tools/demo.py
|
||||
|
||||
sync:
|
||||
uv run --project /home/worsch/repo-manager rmgr sync --path . --push
|
||||
47
README.md
47
README.md
|
|
@ -1,3 +1,46 @@
|
|||
# intelligence-radar
|
||||
# Intelligence Radar
|
||||
|
||||
Epistemic observatory and continuously updated machine intelligence utility predictor.
|
||||
Epistemic observatory and task-specific machine intelligence capability and economics advisor.
|
||||
|
||||
The initial operational version combines a **Compendium**, a manually curated
|
||||
**Radar**, and an on-demand **Capability Advisor** (also called Task Router).
|
||||
Read [INTENT](INTENT.md), [current goal](GOAL.md), and [scope](SCOPE.md).
|
||||
|
||||
## Run locally
|
||||
|
||||
Requires Python 3.11+ and Make. No service, API key, or package installation is needed.
|
||||
|
||||
```bash
|
||||
make check
|
||||
make demo
|
||||
python3 -m intelligence_radar --help
|
||||
```
|
||||
|
||||
`make demo` exercises an isolated temporary ledger using synthetic evidence.
|
||||
For real work, follow [the runbook](ops/RUNBOOK.md): capture source evidence,
|
||||
record an observation, produce a dated digest draft, review it, and evaluate tasks
|
||||
with [the routing method](task-router/METHOD.md) and [ticket](task-router/TEMPLATE.md).
|
||||
The CLI supports recording and validating evidence and drafting daily/weekly digests.
|
||||
Model ranking and task calibration are currently analyst workflows.
|
||||
|
||||
## Knowledge and operations
|
||||
|
||||
| Path | Purpose |
|
||||
| --- | --- |
|
||||
| `IntelligenceConceptModel.md`, `compendium/` | Concept model and accessible theory maps |
|
||||
| `ArchitectureBlueprint.md`, `ResearchProgram.md` | Target architecture and research program |
|
||||
| `BenchmarkCatalog.md`, `catalog/` | Candidate measurement instruments; imported seeds need verification |
|
||||
| `SourceCatalog.md`, `watchlist/` | Source discovery and editorial coverage |
|
||||
| `TaskModelEvaluationMethod.md`, `policies/`, `task-router/` | Authoritative method, policies, and operational templates |
|
||||
| `data/observations/` | Immutable, content-addressed observation envelopes |
|
||||
| `digests/`, `frontier/`, `research/` | Dated editorial artifacts; imported drafts are marked unverified |
|
||||
| `ops/` | Runbook, cadence, and scheduling status |
|
||||
| `history/` | Unmodified source archives and checksums |
|
||||
| `docs/integration-review.md` | Evaluation, conflicts, and integration decisions |
|
||||
| `workplans/` | Canonical Coulomb work records projected into Statehub |
|
||||
|
||||
Imported September 2026 rankings and prices are research leads, not validated
|
||||
recommendations. No external automation has been activated by this repository.
|
||||
|
||||
Statehub: domain `infotech`, topic `custodian`, prefix `IR-WP-`.
|
||||
Commit workplan edits, then run `make sync` (pushes commits and reconciles Statehub). See [agent instructions](AGENTS.md).
|
||||
|
|
|
|||
412
ResearchProgram.md
Normal file
412
ResearchProgram.md
Normal file
|
|
@ -0,0 +1,412 @@
|
|||
# Intelligence Radar Research Program
|
||||
|
||||
**Status:** Draft 0.1
|
||||
**Date:** 2026-09-13
|
||||
|
||||
## 1. Research objective
|
||||
|
||||
Maintain a continuously improving map of:
|
||||
|
||||
1. what intelligence is proposed to mean;
|
||||
2. how intelligence is generated and constrained across substrates;
|
||||
3. how it can be measured;
|
||||
4. what artificial systems can currently do;
|
||||
5. what those capabilities cost and where they are available;
|
||||
6. what the consequences are for engineering, science, economics, society, safety, law, and governance.
|
||||
|
||||
The program intentionally joins **slow epistemic research** with **fast operational monitoring**.
|
||||
|
||||
## 2. Research streams
|
||||
|
||||
### R1 — Models of intelligence
|
||||
|
||||
Questions:
|
||||
|
||||
- What definitions and operationalizations exist?
|
||||
- Which phenomena do they explain well?
|
||||
- Which assumptions do they make about goals, representation, embodiment, learning, or agency?
|
||||
- Which theories are complementary versus genuinely contradictory?
|
||||
- What evidence could discriminate among them?
|
||||
|
||||
### R2 — Biological intelligence
|
||||
|
||||
- neural computation;
|
||||
- memory and learning;
|
||||
- animal cognition;
|
||||
- development;
|
||||
- evolution;
|
||||
- perception/action loops;
|
||||
- energetic efficiency;
|
||||
- social cognition.
|
||||
|
||||
### R3 — Systems, cybernetics, and control
|
||||
|
||||
- requisite variety;
|
||||
- regulation;
|
||||
- viability;
|
||||
- feedback;
|
||||
- adaptation;
|
||||
- recursive organization;
|
||||
- observer dependence;
|
||||
- self-reference;
|
||||
- autopoiesis/enaction.
|
||||
|
||||
### R4 — Artificial architectures
|
||||
|
||||
- deep learning;
|
||||
- transformers and alternatives;
|
||||
- recurrent/state-space systems;
|
||||
- world models;
|
||||
- reinforcement learning;
|
||||
- memory;
|
||||
- retrieval;
|
||||
- planning;
|
||||
- tool use;
|
||||
- program synthesis;
|
||||
- neuro-symbolic systems;
|
||||
- robotics;
|
||||
- multi-agent systems.
|
||||
|
||||
### R5 — Capability measurement
|
||||
|
||||
- construct validity;
|
||||
- benchmark design;
|
||||
- contamination;
|
||||
- saturation;
|
||||
- evaluator/judge reliability;
|
||||
- reproducibility;
|
||||
- ecological validity;
|
||||
- long-horizon evaluation;
|
||||
- hidden/private tests;
|
||||
- human baselines;
|
||||
- task distributions.
|
||||
|
||||
### R6 — Capability frontier
|
||||
|
||||
Track:
|
||||
|
||||
- model releases;
|
||||
- execution profiles;
|
||||
- agent scaffolds;
|
||||
- context and memory;
|
||||
- multimodality;
|
||||
- tool/computer use;
|
||||
- reliability;
|
||||
- autonomy horizon;
|
||||
- coding;
|
||||
- research;
|
||||
- scientific reasoning;
|
||||
- professional knowledge work;
|
||||
- security-relevant capability.
|
||||
|
||||
### R7 — Economics of intelligence
|
||||
|
||||
Track:
|
||||
|
||||
- input/output/cache/reasoning token prices;
|
||||
- tool charges;
|
||||
- cost per solved task;
|
||||
- runtime;
|
||||
- throughput;
|
||||
- latency;
|
||||
- energy;
|
||||
- accelerator economics;
|
||||
- self-hosting economics;
|
||||
- open-weight availability;
|
||||
- batching/flex/priority tiers;
|
||||
- geography and access restrictions.
|
||||
|
||||
Goal:
|
||||
|
||||
> Build a quality-adjusted price surface for intelligence rather than a token-price table.
|
||||
|
||||
### R8 — Intelligence engineering
|
||||
|
||||
- inference infrastructure;
|
||||
- accelerators;
|
||||
- memory bandwidth;
|
||||
- networking;
|
||||
- compilers;
|
||||
- quantization;
|
||||
- serving;
|
||||
- speculative decoding;
|
||||
- distributed inference;
|
||||
- edge intelligence;
|
||||
- agent runtimes;
|
||||
- observability;
|
||||
- evaluation harnesses.
|
||||
|
||||
### R9 — Safety, security, and reliability
|
||||
|
||||
- dangerous capabilities;
|
||||
- cybersecurity;
|
||||
- biosecurity;
|
||||
- alignment;
|
||||
- deception/manipulation;
|
||||
- robustness;
|
||||
- privacy;
|
||||
- misuse;
|
||||
- incident evidence;
|
||||
- control and monitoring methods.
|
||||
|
||||
### R10 — Economic and social effects
|
||||
|
||||
- productivity;
|
||||
- task substitution/complementarity;
|
||||
- labor;
|
||||
- organizational design;
|
||||
- education;
|
||||
- science;
|
||||
- inequality;
|
||||
- access;
|
||||
- market concentration;
|
||||
- new business models.
|
||||
|
||||
### R11 — Law, policy, standards, and institutions
|
||||
|
||||
Track original texts and official implementation material for:
|
||||
|
||||
- AI regulation;
|
||||
- safety obligations;
|
||||
- privacy/data law interactions;
|
||||
- copyright;
|
||||
- liability;
|
||||
- competition;
|
||||
- standards;
|
||||
- procurement;
|
||||
- sector regulation;
|
||||
- export controls and compute policy where relevant.
|
||||
|
||||
## 3. Monitoring cadence
|
||||
|
||||
### Daily
|
||||
|
||||
Detect high-value deltas.
|
||||
|
||||
Output:
|
||||
|
||||
- event;
|
||||
- primary source;
|
||||
- what changed;
|
||||
- why it matters;
|
||||
- affected catalog entities;
|
||||
- affected claims/policies;
|
||||
- confidence;
|
||||
- follow-up required.
|
||||
|
||||
### Weekly
|
||||
|
||||
Synthesize rather than concatenate.
|
||||
|
||||
Recommended sections:
|
||||
|
||||
1. **Five most consequential changes**
|
||||
2. **Capability frontier**
|
||||
3. **Cost and availability frontier**
|
||||
4. **Benchmark/evaluation health**
|
||||
5. **Science of intelligence**
|
||||
6. **Agents/software/hardware**
|
||||
7. **Safety and security**
|
||||
8. **Economics and society**
|
||||
9. **Politics, law, regulation, standards**
|
||||
10. **Compendium changes**
|
||||
11. **What became less certain**
|
||||
12. **Watchlist for next week**
|
||||
|
||||
### Monthly
|
||||
|
||||
Perform maintenance that daily/weekly news cannot:
|
||||
|
||||
- benchmark health review;
|
||||
- source quality review;
|
||||
- thinker/source additions/removals;
|
||||
- stale claims;
|
||||
- pricing/catalog reconciliation;
|
||||
- calibration analysis from real tasks;
|
||||
- research-gap prioritization.
|
||||
|
||||
### Quarterly
|
||||
|
||||
Produce a deeper **State of Intelligence** revision:
|
||||
|
||||
- major model/theory changes;
|
||||
- frontier movement;
|
||||
- economics;
|
||||
- evaluation validity;
|
||||
- capability-to-cost curves;
|
||||
- autonomy horizon;
|
||||
- open vs closed availability;
|
||||
- important regulatory changes;
|
||||
- newly emerging research questions.
|
||||
|
||||
## 4. Evidence discipline
|
||||
|
||||
### Tier A — Primary evidence
|
||||
|
||||
Preferred for durable facts:
|
||||
|
||||
- peer-reviewed papers / primary preprints;
|
||||
- official datasets/evaluation repos;
|
||||
- benchmark maintainers;
|
||||
- model/system cards;
|
||||
- provider documentation;
|
||||
- official pricing;
|
||||
- source code/releases;
|
||||
- regulator/legislator texts;
|
||||
- standards bodies;
|
||||
- original experiment data.
|
||||
|
||||
### Tier B — Independent measurement / synthesis
|
||||
|
||||
Examples:
|
||||
|
||||
- independent evaluation organizations;
|
||||
- systematic benchmark comparisons;
|
||||
- high-quality academic syntheses;
|
||||
- major index reports.
|
||||
|
||||
Useful for triangulation and cross-provider comparison.
|
||||
|
||||
### Tier C — Expert interpretation
|
||||
|
||||
Researchers, engineers, economists, legal experts, and other thinkers.
|
||||
|
||||
Useful for hypotheses, interpretation, and discovery.
|
||||
|
||||
Not equivalent to primary evidence.
|
||||
|
||||
### Tier D — General reporting/social signal
|
||||
|
||||
Useful as an early warning or discovery stream.
|
||||
|
||||
A durable Radar claim should normally resolve to stronger evidence.
|
||||
|
||||
## 5. Research item template
|
||||
|
||||
```yaml
|
||||
id: research-item-...
|
||||
title: ...
|
||||
research_streams: [R5, R6]
|
||||
question: ...
|
||||
status: open
|
||||
priority: high
|
||||
claims:
|
||||
- ...
|
||||
evidence_needed:
|
||||
- ...
|
||||
sources:
|
||||
- ...
|
||||
next_actions:
|
||||
- ...
|
||||
last_reviewed: ...
|
||||
```
|
||||
|
||||
## 6. Claim lifecycle
|
||||
|
||||
```text
|
||||
candidate
|
||||
→ sourced
|
||||
→ structured
|
||||
→ challenged
|
||||
→ triangulated
|
||||
→ compendium claim
|
||||
→ periodically re-evaluated
|
||||
```
|
||||
|
||||
No claim becomes durable merely because many secondary articles repeat it.
|
||||
|
||||
## 7. Benchmark research lifecycle
|
||||
|
||||
Every benchmark gets a health record.
|
||||
|
||||
Review:
|
||||
|
||||
- intended construct;
|
||||
- real task distribution;
|
||||
- scoring correctness;
|
||||
- ceiling/saturation;
|
||||
- public-test contamination;
|
||||
- benchmark gaming incentives;
|
||||
- scaffold sensitivity;
|
||||
- judge sensitivity;
|
||||
- version comparability;
|
||||
- human baseline quality;
|
||||
- cost to run;
|
||||
- licensing/access;
|
||||
- predictive correlation with Intelligence Radar task outcomes.
|
||||
|
||||
A benchmark can be:
|
||||
|
||||
- `candidate`
|
||||
- `active`
|
||||
- `watch`
|
||||
- `degraded`
|
||||
- `saturated`
|
||||
- `retired`
|
||||
|
||||
Retirement does not delete historical results.
|
||||
|
||||
## 8. Frontier research question
|
||||
|
||||
Do not ask only:
|
||||
|
||||
> What is the best model?
|
||||
|
||||
Track multiple frontiers:
|
||||
|
||||
- maximal quality;
|
||||
- quality per dollar;
|
||||
- quality per joule;
|
||||
- quality per second;
|
||||
- quality per GB of memory;
|
||||
- quality under local/self-hosted constraints;
|
||||
- quality under privacy/ZDR constraints;
|
||||
- autonomy horizon;
|
||||
- open-weight frontier;
|
||||
- deployable frontier by region/provider;
|
||||
- multimodal frontier;
|
||||
- specialized domain frontiers.
|
||||
|
||||
## 9. Calibration research
|
||||
|
||||
The most strategically important proprietary dataset will be the Task → Model Performance Ledger.
|
||||
|
||||
Research questions:
|
||||
|
||||
- Which public benchmarks best predict which task clusters?
|
||||
- When do model-family/vendor effects remain after benchmark scores?
|
||||
- How stable are rankings across scaffolds?
|
||||
- Which task features predict failures?
|
||||
- How much does decomposition help?
|
||||
- When is a verifier more valuable than a larger generator?
|
||||
- How do costs scale with ambiguity?
|
||||
- How well does model self-estimated confidence predict actual success?
|
||||
- Which benchmarks lose predictive value first as the frontier moves?
|
||||
|
||||
## 10. Anti-goals
|
||||
|
||||
Avoid:
|
||||
|
||||
- benchmark tourism;
|
||||
- leaderboard worship;
|
||||
- treating press-release evaluation tables as neutral evidence;
|
||||
- silently mixing model-only and agent-scaffold results;
|
||||
- using stale prices;
|
||||
- measuring task success only with model judges;
|
||||
- collapsing heterogeneous scientific concepts into a single AGI narrative;
|
||||
- overreacting to every daily release.
|
||||
|
||||
## 11. First twelve research tasks
|
||||
|
||||
1. Build a genealogy of definitions of intelligence across psychometrics, cybernetics, cognitive science, neuroscience, AI, and collective intelligence.
|
||||
2. Establish a benchmark construct taxonomy.
|
||||
3. Create a benchmark health scorecard.
|
||||
4. Model `execution_profile` explicitly and normalize existing leaderboard evidence into it.
|
||||
5. Build price-observation ingestion for major providers.
|
||||
6. Build a first capability-vector ontology from 50 representative real tasks.
|
||||
7. Test the five-benchmark selector manually on those 50 tasks.
|
||||
8. Establish initial success-probability calibration using repeated model trials.
|
||||
9. Compare monolithic vs decomposed execution on at least 20 complex tasks.
|
||||
10. Measure benchmark correlation and redundancy.
|
||||
11. Define availability/privacy constraints as first-class ranking filters.
|
||||
12. Publish the first quarterly State of Intelligence report from accumulated Radar evidence.
|
||||
23
SCOPE.md
Normal file
23
SCOPE.md
Normal file
|
|
@ -0,0 +1,23 @@
|
|||
# Scope
|
||||
|
||||
## Implemented
|
||||
|
||||
- Governing intent, concept model, architecture, research program, benchmark and source catalogs.
|
||||
- Compendium, watchlist, historical draft digests and frontier snapshots from both explorations.
|
||||
- One authoritative task evaluation method, policy baseline, and manual ticket workflow.
|
||||
- Python CLI: validate and record immutable observation envelopes; validate the ledger;
|
||||
generate daily or ISO-weekly digest drafts by observation event date.
|
||||
- Local checks and a synthetic end-to-end demonstration.
|
||||
- Coulomb workplans, agent entry points, classification and capability registry.
|
||||
|
||||
## Boundaries
|
||||
|
||||
Statehub tracks repository work; `data/observations/` holds research evidence.
|
||||
A digest records what was observed; editorial review determines whether a claim
|
||||
is supported. A content hash proves local integrity, not truth or source authenticity.
|
||||
Imported claims and source catalog status labels remain unverified research leads.
|
||||
|
||||
The advisor is manually operated. No calibrated success predictor, provider adapter,
|
||||
automated benchmark selector, live price feed, API server, PostgreSQL deployment,
|
||||
scheduled worker, or production UI exists yet. These belong to IR-WP-0003.
|
||||
No live model recommendation is warranted by the synthetic demo.
|
||||
233
SourceCatalog.md
Normal file
233
SourceCatalog.md
Normal file
|
|
@ -0,0 +1,233 @@
|
|||
> Imported research draft. Benchmark versions, availability, and source claims require verification before operational use.
|
||||
|
||||
# Intelligence Radar — Source Catalog
|
||||
|
||||
**Status:** Seed catalog 0.1
|
||||
**Date:** 2026-09-13
|
||||
|
||||
## 1. Source policy
|
||||
|
||||
The monitoring system should aggressively discover information but conservatively promote claims.
|
||||
|
||||
Priority order for durable facts:
|
||||
|
||||
1. original research/data;
|
||||
2. benchmark maintainers;
|
||||
3. official model/system cards and provider documentation;
|
||||
4. official standards/regulatory/legal sources;
|
||||
5. independent evaluators and systematic syntheses;
|
||||
6. expert interpretation;
|
||||
7. journalism/social discussion for discovery.
|
||||
|
||||
A thinker being influential is a reason to monitor them, not a reason to accept their claims.
|
||||
|
||||
## 2. Scientific research streams
|
||||
|
||||
Monitor primary publications and proceedings in:
|
||||
|
||||
- machine learning and AI;
|
||||
- cognitive science;
|
||||
- neuroscience;
|
||||
- psychology/psychometrics;
|
||||
- robotics;
|
||||
- complex systems;
|
||||
- cybernetics/control;
|
||||
- collective intelligence;
|
||||
- computational biology;
|
||||
- philosophy of mind where empirically relevant.
|
||||
|
||||
High-signal discovery surfaces include arXiv categories, major peer-reviewed journals, and major conference proceedings. The source registry should resolve interesting discoveries to the actual paper/repository/data.
|
||||
|
||||
## 3. AI labs and model providers — official sources
|
||||
|
||||
Seed set:
|
||||
|
||||
- OpenAI — releases, model/system cards, API/model docs, safety/deployment material.
|
||||
- Anthropic — research, model releases, system cards, API docs, pricing.
|
||||
- Google DeepMind / Google AI — research, model cards, Gemini API docs.
|
||||
- Meta AI — model/research releases.
|
||||
- xAI — model releases and technical documentation.
|
||||
- DeepSeek — model/research releases.
|
||||
- Alibaba/Qwen — model/research releases.
|
||||
- Mistral AI — model/research releases.
|
||||
- Cohere — model/research releases.
|
||||
- NVIDIA — model, inference, accelerator, and serving research.
|
||||
- major open-weight research organizations as they emerge.
|
||||
|
||||
## 4. Evaluation and benchmark maintainers
|
||||
|
||||
Seed set:
|
||||
|
||||
- SWE-bench
|
||||
- Berkeley Function Calling Leaderboard (BFCL)
|
||||
- METR
|
||||
- ARC Prize / ARC-AGI
|
||||
- OSWorld
|
||||
- Sierra research / τ-bench family
|
||||
- MLCommons / MLPerf
|
||||
- SciCode
|
||||
- Humanity's Last Exam maintainers
|
||||
- major benchmark authors/repositories discovered through current research
|
||||
|
||||
Each benchmark needs its own version/change monitor.
|
||||
|
||||
## 5. Independent capability/economics measurement
|
||||
|
||||
### Artificial Analysis
|
||||
|
||||
Useful for:
|
||||
|
||||
- cross-provider model measurements;
|
||||
- benchmark results;
|
||||
- cost per task;
|
||||
- token use;
|
||||
- speed/latency;
|
||||
- provider comparisons;
|
||||
- evolving aggregate and domain indices.
|
||||
|
||||
Treat its private tests and methodology as independent evidence, not fully reproducible primary benchmark truth.
|
||||
|
||||
### Stanford HAI AI Index
|
||||
|
||||
Useful periodic synthesis of:
|
||||
|
||||
- capability trends;
|
||||
- benchmark health;
|
||||
- economics;
|
||||
- adoption;
|
||||
- social/policy trends.
|
||||
|
||||
### Additional independent evaluators
|
||||
|
||||
Add only after documenting methodology, independence, update cadence, and data access.
|
||||
|
||||
## 6. Hardware and inference engineering
|
||||
|
||||
Seed primary sources:
|
||||
|
||||
- MLCommons / MLPerf;
|
||||
- NVIDIA;
|
||||
- AMD;
|
||||
- Intel;
|
||||
- Google Cloud/TPU material;
|
||||
- AWS accelerator/inference infrastructure;
|
||||
- Microsoft/Azure infrastructure;
|
||||
- hyperscaler research and engineering publications;
|
||||
- compiler/serving projects relevant to inference economics.
|
||||
|
||||
Track:
|
||||
|
||||
- throughput;
|
||||
- latency;
|
||||
- power;
|
||||
- memory;
|
||||
- interconnect;
|
||||
- quantization;
|
||||
- batching;
|
||||
- speculative decoding;
|
||||
- serving frameworks;
|
||||
- price/performance.
|
||||
|
||||
## 7. Regulation, standards, and institutions
|
||||
|
||||
### European Union
|
||||
|
||||
Monitor original European Commission / AI Office / EUR-Lex materials for AI Act implementation, delegated/implementing acts, codes, guidance, and enforcement milestones.
|
||||
|
||||
### United States
|
||||
|
||||
Monitor relevant NIST, federal agency, legislative, standards, and official policy sources.
|
||||
|
||||
### International
|
||||
|
||||
Monitor:
|
||||
|
||||
- OECD AI policy materials;
|
||||
- ISO/IEC JTC 1/SC 42 where accessible;
|
||||
- IEEE standards initiatives;
|
||||
- Council of Europe AI convention implementation;
|
||||
- other jurisdictions as capability or market relevance requires.
|
||||
|
||||
## 8. Thinker watchlist — seed, not canon
|
||||
|
||||
The watchlist should span disagreement.
|
||||
|
||||
Candidate clusters:
|
||||
|
||||
### Foundations / learning / AI
|
||||
|
||||
- Geoffrey Hinton
|
||||
- Yann LeCun
|
||||
- Yoshua Bengio
|
||||
- Richard Sutton
|
||||
- Demis Hassabis
|
||||
- Ilya Sutskever
|
||||
- Dario Amodei
|
||||
- François Chollet
|
||||
|
||||
### Cognitive science / neuroscience / intelligence
|
||||
|
||||
- active researchers in psychometrics, cognitive architectures, predictive processing, memory, development, animal cognition, and computational neuroscience.
|
||||
|
||||
### Systems / cybernetics lineage
|
||||
|
||||
Historical compendium coverage should include:
|
||||
|
||||
- Norbert Wiener
|
||||
- W. Ross Ashby
|
||||
- Stafford Beer
|
||||
- Heinz von Foerster
|
||||
- Humberto Maturana
|
||||
- Francisco Varela
|
||||
|
||||
Current monitoring should focus on contemporary researchers extending relevant ideas rather than treating historical figures as news sources.
|
||||
|
||||
### Critique / alternative interpretations
|
||||
|
||||
Monitor credible researchers who challenge:
|
||||
|
||||
- scaling narratives;
|
||||
- benchmark validity;
|
||||
- claims of generality;
|
||||
- economic assumptions;
|
||||
- safety assumptions;
|
||||
- anthropomorphic interpretations.
|
||||
|
||||
The purpose is epistemic diversity, not artificial balance.
|
||||
|
||||
## 9. Source record schema
|
||||
|
||||
```yaml
|
||||
id: source-...
|
||||
name: ...
|
||||
kind: paper_feed|lab|benchmark|provider_docs|regulator|thinker|journal|evaluator
|
||||
primary: true
|
||||
trust_tier: A
|
||||
topics: [...]
|
||||
url: ...
|
||||
retrieval:
|
||||
mode: rss|html|api|github|manual
|
||||
cadence: daily
|
||||
importance: 0.9
|
||||
last_reviewed: 2026-09-13
|
||||
notes: ...
|
||||
```
|
||||
|
||||
## 10. Source-health policy
|
||||
|
||||
Periodically score:
|
||||
|
||||
- authority for the claimed fact;
|
||||
- independence;
|
||||
- reproducibility;
|
||||
- provenance quality;
|
||||
- update timeliness;
|
||||
- correction transparency;
|
||||
- historical reliability;
|
||||
- machine-retrieval stability.
|
||||
|
||||
A source can be excellent for one class of fact and weak for another.
|
||||
|
||||
Example:
|
||||
|
||||
A provider's API page is authoritative for its **current list price**, but not neutral evidence that its model is the world's best.
|
||||
446
TaskModelEvaluationMethod.md
Normal file
446
TaskModelEvaluationMethod.md
Normal file
|
|
@ -0,0 +1,446 @@
|
|||
# Task → Benchmark → Model Evaluation Method
|
||||
|
||||
**Status:** Draft 0.1
|
||||
**Policy baseline:** 2026-09-13
|
||||
|
||||
## 1. Purpose
|
||||
|
||||
Provide a reproducible method for answering:
|
||||
|
||||
> For this particular task, which AI execution profile is Frontier, Topnotch, Efficient, Sufficient, or Budget — and how confident are we?
|
||||
|
||||
The method must avoid pretending that a global leaderboard directly answers a task-specific question.
|
||||
|
||||
## 2. Step 1 — Interpret the task
|
||||
|
||||
Extract:
|
||||
|
||||
- objective;
|
||||
- required deliverable;
|
||||
- input/context;
|
||||
- success criteria;
|
||||
- constraints;
|
||||
- allowed tools;
|
||||
- modalities;
|
||||
- privacy/security restrictions;
|
||||
- time horizon;
|
||||
- interaction requirements;
|
||||
- expected autonomy;
|
||||
- domain;
|
||||
- consequence of failure;
|
||||
- budget/runtime constraints.
|
||||
|
||||
### Task clarity
|
||||
|
||||
Score six components from 0–1:
|
||||
|
||||
- objective clarity;
|
||||
- context sufficiency;
|
||||
- constraint clarity;
|
||||
- acceptance-criterion clarity;
|
||||
- output-contract clarity;
|
||||
- environment/tool clarity.
|
||||
|
||||
Default:
|
||||
|
||||
`clarity = weighted_mean(components)`
|
||||
|
||||
Return both the score and concrete ambiguities.
|
||||
|
||||
Do not confuse `unclear` with `difficult`.
|
||||
|
||||
## 3. Step 2 — Build a Capability Profile
|
||||
|
||||
Initial dimensions:
|
||||
|
||||
```yaml
|
||||
knowledge:
|
||||
reasoning:
|
||||
instruction_following:
|
||||
long_context:
|
||||
coding:
|
||||
scientific_reasoning:
|
||||
quantitative:
|
||||
web_research:
|
||||
retrieval:
|
||||
tool_use:
|
||||
computer_use:
|
||||
planning:
|
||||
autonomy_horizon:
|
||||
user_interaction:
|
||||
multimodal:
|
||||
writing:
|
||||
artifact_creation:
|
||||
non_hallucination:
|
||||
robustness:
|
||||
security_awareness:
|
||||
domain_specialization:
|
||||
latency_sensitivity:
|
||||
cost_sensitivity:
|
||||
```
|
||||
|
||||
Values are normalized 0–1 relevance/requirement levels.
|
||||
|
||||
Also derive hard gates, e.g.:
|
||||
|
||||
- must accept images;
|
||||
- must access browser;
|
||||
- must support tool calls;
|
||||
- must support >300k context;
|
||||
- must be locally deployable;
|
||||
- must satisfy regional/data restrictions.
|
||||
|
||||
## 4. Step 3 — Select five benchmarks
|
||||
|
||||
Retrieve benchmark candidates tagged with relevant constructs.
|
||||
|
||||
Rank using the active benchmark-selection policy.
|
||||
|
||||
Selection must balance:
|
||||
|
||||
- task relevance;
|
||||
- ecological validity;
|
||||
- current discriminative power;
|
||||
- reliability;
|
||||
- freshness;
|
||||
- execution-profile similarity;
|
||||
- reproducibility;
|
||||
- contamination resistance;
|
||||
- independence from the other selected measurements.
|
||||
|
||||
Output example:
|
||||
|
||||
| Benchmark | Relevance | Weight | Main reason |
|
||||
|---|---:|---:|---|
|
||||
| SWE-bench Multilingual | 0.94 | 0.31 | repository engineering |
|
||||
| Terminal-Bench | 0.87 | 0.25 | terminal/tool execution |
|
||||
| BFCL V4 | 0.77 | 0.18 | API/tool orchestration |
|
||||
| METR Time Horizon | 0.72 | 0.15 | long-horizon autonomy |
|
||||
| BrowseComp | 0.58 | 0.11 | external research |
|
||||
|
||||
If all candidates are weak, return low benchmark-coverage confidence instead of disguising the weakness.
|
||||
|
||||
## 5. Step 4 — Normalize benchmark results
|
||||
|
||||
Raw benchmark scores are generally not directly comparable.
|
||||
|
||||
Each benchmark version needs a normalization function producing:
|
||||
|
||||
`p_b(m) ∈ [0,1]`
|
||||
|
||||
where possible this should be calibrated toward **probability of acceptable performance on the benchmark's target construct**, not merely leaderboard percentile.
|
||||
|
||||
Early versions may use normalized score/percentile transformations, but the method must label them as proxies.
|
||||
|
||||
Important:
|
||||
|
||||
- score model+scaffold results as execution profiles;
|
||||
- do not transfer a tool-enabled score to model-only use without a penalty/uncertainty;
|
||||
- preserve missing data;
|
||||
- do not automatically impute a model result from a related model family.
|
||||
|
||||
## 6. Step 5 — Estimate task fitness
|
||||
|
||||
Baseline weighted model:
|
||||
|
||||
`S(m,T) = Σ_i w_i · p_i(m)`
|
||||
|
||||
This is intentionally simple for v0.1.
|
||||
|
||||
Later calibration should learn task-cluster-specific mappings using outcome data.
|
||||
|
||||
### Weakest-link modifier
|
||||
|
||||
For tasks with hard multi-capability requirements, arithmetic averaging can hide a catastrophic weakness.
|
||||
|
||||
Optional policy:
|
||||
|
||||
`S'(m,T) = (1-λ)·weighted_mean + λ·weighted_geometric_mean`
|
||||
|
||||
or explicit gates such as:
|
||||
|
||||
`tool_use >= threshold`
|
||||
|
||||
Use policy-defined `λ`.
|
||||
|
||||
## 7. Step 6 — Estimate task success probability
|
||||
|
||||
`S` is a comparative task-fit score.
|
||||
|
||||
For **Sufficient**, we need:
|
||||
|
||||
`P(success | task, execution_profile, environment)`
|
||||
|
||||
Initially estimate this from benchmark calibration plus task similarity and expose high uncertainty.
|
||||
|
||||
As the outcome ledger grows, fit empirical calibration models.
|
||||
|
||||
Recommended task risk classes:
|
||||
|
||||
- `exploratory`
|
||||
- `routine`
|
||||
- `important`
|
||||
- `high_assurance`
|
||||
|
||||
Sufficiency threshold should be explicit.
|
||||
|
||||
Illustrative defaults only:
|
||||
|
||||
- exploratory: 0.60
|
||||
- routine: 0.80
|
||||
- important: 0.90
|
||||
- high_assurance: 0.97 plus independent verification requirements
|
||||
|
||||
These defaults are policy, not facts.
|
||||
|
||||
## 8. Step 7 — Apply eligibility filters
|
||||
|
||||
Before ranking, remove deployments that violate hard constraints:
|
||||
|
||||
- unavailable to requester;
|
||||
- wrong modality;
|
||||
- inadequate context/output limit;
|
||||
- prohibited geography;
|
||||
- unacceptable data retention/privacy;
|
||||
- insufficient tool support;
|
||||
- incompatible hosting requirement;
|
||||
- incompatible license;
|
||||
- rate limit below required throughput.
|
||||
|
||||
This yields the **Available** candidate set.
|
||||
|
||||
## 9. Step 8 — Define Frontier and Topnotch
|
||||
|
||||
For eligible set `M`:
|
||||
|
||||
`frontier = argmax_m S(m,T)`
|
||||
|
||||
Topnotch default:
|
||||
|
||||
`S(m,T) >= 0.90 × S(frontier,T)`
|
||||
|
||||
The 10% rule is a versioned policy and can be replaced by a calibrated success-probability delta later.
|
||||
|
||||
## 10. Step 9 — Estimate total task economics
|
||||
|
||||
### API deployments
|
||||
|
||||
Expected total cost:
|
||||
|
||||
```text
|
||||
C =
|
||||
input_tokens × input_rate
|
||||
+ cached_read_tokens × cached_read_rate
|
||||
+ cache_write_tokens × cache_write_rate
|
||||
+ output_and_reasoning_tokens × output_rate
|
||||
+ tool_call_cost
|
||||
+ search/retrieval cost
|
||||
+ expected_retry_cost
|
||||
+ verifier_cost
|
||||
+ orchestration overhead
|
||||
```
|
||||
|
||||
Provider semantics differ; cost adapters must reflect the actual billing model.
|
||||
|
||||
### Self-hosted/open-weight deployments
|
||||
|
||||
Estimate:
|
||||
|
||||
```text
|
||||
C =
|
||||
accelerator_seconds × amortized_accelerator_rate
|
||||
+ CPU/RAM/storage/network
|
||||
+ energy
|
||||
+ idle/utilization overhead
|
||||
+ orchestration
|
||||
+ operations allocation
|
||||
```
|
||||
|
||||
### Distribution, not false precision
|
||||
|
||||
Return at least:
|
||||
|
||||
- expected/p50;
|
||||
- p90;
|
||||
- main cost drivers;
|
||||
- price observation timestamp.
|
||||
|
||||
## 11. Step 10 — Define Efficient, Sufficient, Budget
|
||||
|
||||
### Efficient
|
||||
|
||||
`argmin expected_cost(m)` among Topnotch profiles.
|
||||
|
||||
### Sufficient
|
||||
|
||||
All eligible profiles whose estimated success probability exceeds the task threshold and satisfy all hard gates.
|
||||
|
||||
### Budget
|
||||
|
||||
`argmin expected_cost(m)` among Sufficient profiles.
|
||||
|
||||
If no profile meets sufficiency with adequate confidence, state:
|
||||
|
||||
`no_evidenced_sufficient_model`
|
||||
|
||||
rather than relaxing the threshold invisibly.
|
||||
|
||||
## 12. Step 11 — Runtime estimate
|
||||
|
||||
Runtime includes:
|
||||
|
||||
- queue/TTFT;
|
||||
- token generation;
|
||||
- tool latency;
|
||||
- serial agent turns;
|
||||
- code execution;
|
||||
- browsing;
|
||||
- retries;
|
||||
- verification.
|
||||
|
||||
For a DAG workplan:
|
||||
|
||||
`runtime ≈ critical_path_length`, not sum of node runtimes.
|
||||
|
||||
Return p50/p90 where possible.
|
||||
|
||||
## 13. Step 12 — Confidence decomposition
|
||||
|
||||
Recommended components:
|
||||
|
||||
```yaml
|
||||
task_clarity:
|
||||
benchmark_coverage:
|
||||
benchmark_validity:
|
||||
measurement_reliability:
|
||||
result_freshness:
|
||||
deployment_match:
|
||||
price_confidence:
|
||||
task_calibration_evidence:
|
||||
```
|
||||
|
||||
Overall recommendation confidence should use a conservative aggregation, e.g. weighted geometric mean with caps when a critical dimension is weak.
|
||||
|
||||
Example cap:
|
||||
|
||||
If `benchmark_coverage < 0.4`, then `overall_confidence <= 0.6`.
|
||||
|
||||
## 14. Step 13 — Workplan optimization
|
||||
|
||||
### Decompose
|
||||
|
||||
Create subtasks where specialization or parallelism may help.
|
||||
|
||||
Examples:
|
||||
|
||||
- requirements interpretation;
|
||||
- research;
|
||||
- data extraction;
|
||||
- architecture;
|
||||
- implementation;
|
||||
- testing;
|
||||
- adversarial review;
|
||||
- synthesis.
|
||||
|
||||
### Profile each subtask
|
||||
|
||||
Run the same Capability Profile and five-benchmark mapping per node where justified.
|
||||
|
||||
### Allocate execution profiles
|
||||
|
||||
Minimize expected cost or runtime subject to quality constraints.
|
||||
|
||||
### Verification
|
||||
|
||||
Add independent checks when:
|
||||
|
||||
- task consequence is high;
|
||||
- model uncertainty is high;
|
||||
- outputs are mechanically verifiable;
|
||||
- using a cheaper generator becomes economical when paired with verification.
|
||||
|
||||
### Escalation
|
||||
|
||||
A useful policy:
|
||||
|
||||
1. try Budget/Sufficient model;
|
||||
2. verify;
|
||||
3. escalate failed/uncertain nodes to Efficient;
|
||||
4. escalate irreducible hard nodes to Frontier.
|
||||
|
||||
This often dominates “Frontier everywhere.”
|
||||
|
||||
## 15. Compare plans
|
||||
|
||||
Recommendation report:
|
||||
|
||||
| Strategy | Predicted success | Confidence | p50 cost | p90 cost | p50 runtime | Calls |
|
||||
|---|---:|---:|---:|---:|---:|---:|
|
||||
| Monolithic Frontier | ... | ... | ... | ... | ... | 1 |
|
||||
| Monolithic Efficient | ... | ... | ... | ... | ... | 1 |
|
||||
| Decomposed Efficient | ... | ... | ... | ... | ... | ... |
|
||||
| Decomposed Budget+Escalation | ... | ... | ... | ... | ... | ... |
|
||||
|
||||
Also show:
|
||||
|
||||
- expected cost delta;
|
||||
- expected runtime delta;
|
||||
- expected quality/confidence delta.
|
||||
|
||||
## 16. Post-execution learning
|
||||
|
||||
After task completion, record:
|
||||
|
||||
- whether acceptance criteria passed;
|
||||
- verifier results;
|
||||
- human evaluation;
|
||||
- model calls;
|
||||
- tokens;
|
||||
- cost;
|
||||
- runtime;
|
||||
- retries;
|
||||
- failure modes;
|
||||
- which subtasks required escalation.
|
||||
|
||||
This observation updates calibration for similar tasks.
|
||||
|
||||
## 17. Recommended user-facing result shape
|
||||
|
||||
```text
|
||||
Task
|
||||
Interpretation: ...
|
||||
Clarity: 78%
|
||||
|
||||
Relevant evidence
|
||||
1. Benchmark A — 31% weight
|
||||
2. Benchmark B — 24%
|
||||
3. Benchmark C — 19%
|
||||
4. Benchmark D — 15%
|
||||
5. Benchmark E — 11%
|
||||
Benchmark coverage confidence: 74%
|
||||
|
||||
Recommendation
|
||||
Frontier: ...
|
||||
Topnotch: ...
|
||||
Efficient: ...
|
||||
Sufficient: ...
|
||||
Budget: ...
|
||||
|
||||
Economics
|
||||
Frontier: $... expected / $... p90
|
||||
Efficient: $... expected / $... p90
|
||||
Budget: $... expected / $... p90
|
||||
|
||||
Evaluation confidence
|
||||
Overall: ...
|
||||
Main uncertainty: ...
|
||||
|
||||
Optimization opportunity
|
||||
...
|
||||
```
|
||||
|
||||
## 18. Critical caveat
|
||||
|
||||
Until Intelligence Radar accumulates substantial real task outcome data, numerical task-success probabilities are **estimates with model risk**.
|
||||
|
||||
The system should be precise about its evidence without pretending that precision of arithmetic equals certainty of knowledge.
|
||||
63
catalog/benchmark-seed.yaml
Normal file
63
catalog/benchmark-seed.yaml
Normal file
|
|
@ -0,0 +1,63 @@
|
|||
catalog_version: 0.1
|
||||
snapshot_date: 2026-09-13
|
||||
benchmarks:
|
||||
- id: hle
|
||||
name: Humanity's Last Exam
|
||||
constructs: [expert_knowledge, reasoning, multimodal]
|
||||
status: watch
|
||||
- id: arc-agi-2
|
||||
name: ARC-AGI-2
|
||||
constructs: [abstraction, generalization, novel_rule_induction]
|
||||
status: active
|
||||
- id: swe-bench-verified
|
||||
name: SWE-bench Verified
|
||||
constructs: [software_engineering, repository_issue_resolution]
|
||||
status: watch
|
||||
- id: swe-bench-multilingual
|
||||
name: SWE-bench Multilingual
|
||||
constructs: [software_engineering, multilingual_programming]
|
||||
status: active
|
||||
- id: terminal-bench
|
||||
name: Terminal-Bench
|
||||
constructs: [terminal_use, agentic_execution]
|
||||
status: active
|
||||
- id: bfcl-v4
|
||||
name: Berkeley Function Calling Leaderboard V4
|
||||
constructs: [tool_use, function_calling, agentic_execution]
|
||||
status: active
|
||||
- id: tau3
|
||||
name: tau3-bench
|
||||
constructs: [tool_agent_user_interaction, knowledge, voice]
|
||||
status: active
|
||||
- id: osworld-2
|
||||
name: OSWorld 2.0
|
||||
constructs: [computer_use, multimodal_agent]
|
||||
status: active
|
||||
- id: browsecomp
|
||||
name: BrowseComp
|
||||
constructs: [web_research, browsing]
|
||||
status: active
|
||||
- id: metr-time-horizon
|
||||
name: METR Task-Completion Time Horizon
|
||||
constructs: [autonomy_horizon, reliability, software_tasks]
|
||||
status: active
|
||||
- id: gdpval-aa-v2
|
||||
name: GDPval-AA v2
|
||||
constructs: [professional_work, artifact_creation, agentic_knowledge_work]
|
||||
status: active
|
||||
- id: aa-briefcase
|
||||
name: AA-Briefcase
|
||||
constructs: [long_horizon_knowledge_work]
|
||||
status: active_private
|
||||
- id: automationbench-aa
|
||||
name: AutomationBench-AA
|
||||
constructs: [business_workflows, tool_use]
|
||||
status: active_private
|
||||
- id: scicode
|
||||
name: SciCode
|
||||
constructs: [scientific_reasoning, scientific_programming]
|
||||
status: active
|
||||
- id: mlperf-inference
|
||||
name: MLPerf Inference
|
||||
constructs: [inference_system_performance, efficiency]
|
||||
status: active
|
||||
38
compendium/00-what-this-is.md
Normal file
38
compendium/00-what-this-is.md
Normal file
|
|
@ -0,0 +1,38 @@
|
|||
> Imported exploration, 2026-09-13. Source claims have not been independently verified by this repository. See [integration review](../docs/integration-review.md).
|
||||
|
||||
# Compendium — what this is
|
||||
|
||||
Version 0.1 · 13 September 2026
|
||||
|
||||
This folder is the **introductory map** Intelligence Radar keeps of how humans currently *model* intelligence. It is meant to improve over time: weekly digests propose edits; accepted edits bump the version and leave a short changelog.
|
||||
|
||||
It is not a complete literature review. It is a working ontology plus the disputes that matter for monitoring the frontier.
|
||||
|
||||
## Scope of “intelligence” here
|
||||
|
||||
Intelligence Radar tracks four overlapping objects:
|
||||
|
||||
1. **Biological intelligence** — nervous systems and the cognitive capacities they support in humans and other animals.
|
||||
2. **Artificial intelligence** — engineered systems that perform cognitive work, currently dominated by large trained models and agent scaffolds.
|
||||
3. **Collective / institutional intelligence** — markets, science, firms, states, open-source communities: systems that know and decide without a single mind.
|
||||
4. **Otherwise generated** — hybrid human–machine systems, in-silico cognitive science, digital twins of brains, proposed non-neural substrates (quantum, analog, neuromorphic, latent-space recurrent architectures).
|
||||
|
||||
The facility’s special focus is the **capability frontier of artificial systems** and the **price and availability** of those capabilities. The other three objects exist so the frontier is not mistaken for the whole phenomenon.
|
||||
|
||||
## What “humanity knows”
|
||||
|
||||
A useful humility: humanity does not have one settled theory of intelligence. It has:
|
||||
|
||||
- **Measurement traditions** (psychometrics, animal cognition batteries, AI benchmarks, economic productivity studies).
|
||||
- **Mechanistic sketches** (predictive processing, global workspace, network neuroscience, scaling laws, Bayesian program learning).
|
||||
- **Engineering recipes** that work without a complete theory (pretrain–reason–act loops, RL from verifiable rewards, tool use, long-horizon agents).
|
||||
- **Open disputes** that are load-bearing for policy and investment (is general intelligence a real factor? is consciousness computational? do current agents generalize or memorize? is the bottleneck data, compute, architecture, or evaluation?).
|
||||
|
||||
The compendium tracks those four layers separately so a new model release does not get written up as a new theory of mind.
|
||||
|
||||
## Reading order
|
||||
|
||||
1. `01-maps-of-intelligence.md` — the main human models
|
||||
2. `02-measurement.md` — how intelligence is scored
|
||||
3. `03-frontier-as-of-2026-09.md` — snapshot of the artificial frontier
|
||||
4. Changelog at the bottom of each file
|
||||
85
compendium/01-maps-of-intelligence.md
Normal file
85
compendium/01-maps-of-intelligence.md
Normal file
|
|
@ -0,0 +1,85 @@
|
|||
> Imported exploration, 2026-09-13. Source claims have not been independently verified by this repository. See [integration review](../docs/integration-review.md).
|
||||
|
||||
# Maps of intelligence
|
||||
|
||||
Version 0.1 · 13 September 2026
|
||||
|
||||
Humans have invented several mutually incompatible but productive ways to talk about intelligence. Intelligence Radar treats them as **maps**, not as a single territory.
|
||||
|
||||
## 1. Psychometric / difference map
|
||||
|
||||
**Claim:** Cognitive performances covary. A general factor *g* plus group factors (verbal, spatial, processing speed) describe individual differences.
|
||||
|
||||
**What it is good for:** Predicting educational and occupational outcomes; grounding “general” as a statistical regularity rather than a slogan.
|
||||
|
||||
**Limits:** *g* is a covariance structure, not a mechanism. It does not say how a brain or a model produces competence. Transfer to machines is metaphorical unless the same battery is used.
|
||||
|
||||
**2026 note:** Network Neuroscience Theory treats *g* as a property of global connectome topology (distributed processing, weak long-range links, modal control hubs, small-world communication) rather than a single region. That is a live empirical program, not settled doctrine.
|
||||
|
||||
## 2. Cognitive-architecture map
|
||||
|
||||
**Claim:** Intelligence is a set of processes — perception, working memory, attention, planning, learning, language, motor control — arranged in an architecture (ACT-R, SOAR, dual-process, global workspace, predictive processing).
|
||||
|
||||
**What it is good for:** Task analysis and decomposition. This is the map the Task Router inherits when it splits a job into subtasks.
|
||||
|
||||
**Limits:** Architectures multiply. “System 1 / System 2” is a useful cartoon that current reasoning models have made commercially real without proving the cartoon.
|
||||
|
||||
**2026 note:** Complementary-intelligence programs distinguish *cognitive AI* (built to model human processes) from *machine AI* (optimized for scale). Integration routes being discussed: shared embeddings, instruction encoding, training agents on cognitive-AI data, and coevolving agents.
|
||||
|
||||
## 3. Computational / Bayesian map
|
||||
|
||||
**Claim:** Intelligence is approximate rational inference under resource limits: priors, likelihoods, programs, search.
|
||||
|
||||
**Champions:** Tenenbaum and the probabilistic-programs tradition; the “language is not all you need” line that wants LLMs as *language modules* attached to a world-model / program-induction core.
|
||||
|
||||
**What it is good for:** Explaining few-shot concept learning, causal reasoning, and why pure next-token prediction still fails some structured tasks.
|
||||
|
||||
**Limits:** Full Bayesian program learning does not yet scale to frontier product systems. The engineering winning streak is still large transformers plus RL and tools.
|
||||
|
||||
## 4. Connectionist / scaling map
|
||||
|
||||
**Claim:** Competence emerges from gradient-trained networks at sufficient scale of data, parameters, and compute, plus the right post-training (instruction, preference, RL, tools).
|
||||
|
||||
**What it is good for:** Predicting that many new skills appear as a function of scale and that evaluation sets saturate. This is the map the AI industry actually ships against.
|
||||
|
||||
**Limits:** Poor sample and energy efficiency versus brains; weak accounts of agency, grounded world models, and long-horizon reliability; benchmarks leak.
|
||||
|
||||
**2026 note:** Alternative architectures (latent-space recurrent “brain-inspired” models such as Pathway BDH) are trying to move reasoning *off* the token tape. Watch these as potential cost-curve breakers, not as proven replacements.
|
||||
|
||||
## 5. Neuroscientific / mechanistic map
|
||||
|
||||
**Claim:** Intelligence is what certain recurrent, sparse, multi-scale biological circuits do — especially prefrontal control interacting with specialized systems — under severe energy and developmental constraints.
|
||||
|
||||
**2026 note:** Proposals for a “foundational brain model of intelligence” want connectome + cell types + recurrent dynamics + multi-task training, with PFC as the control core. Separate work (Ganguli and others) calls for a science of intelligence that unifies physics of complex systems, neuroscience, and AI.
|
||||
|
||||
**Limits:** We still lack a causal, multi-scale model that predicts both a mouse navigating and GPT-class language.
|
||||
|
||||
## 6. Consciousness / sentience map (disputed)
|
||||
|
||||
**Claim variants:** Integrated information; global workspace broadcast; higher-order thought; life-first / enactive views that deny substrate-neutral computation.
|
||||
|
||||
**Radar rule:** Track the *dispute* and any policy consequence (welfare of models, “critical” capability designations). Do not treat a lab’s anthropomorphic demo as evidence of sentience.
|
||||
|
||||
## 7. Collective / economic map
|
||||
|
||||
**Claim:** A lot of “what humanity knows” lives in institutions, prices, papers, and code, not in individual skulls. AI changes the cost of contributing to that collective.
|
||||
|
||||
**What it is good for:** The weekly digest’s economic, legal, and political sections. Capability without deployment, capital, chips, or permission is not a social fact.
|
||||
|
||||
## Working synthesis used by this facility
|
||||
|
||||
For routing work to models, Intelligence Radar uses a **capability stack**, not a theory of mind:
|
||||
|
||||
1. Perception and representation
|
||||
2. Knowledge and retrieval
|
||||
3. Reasoning and search
|
||||
4. Tool and environment action
|
||||
5. Planning over long horizons
|
||||
6. Coordination (multi-agent / human-in-the-loop)
|
||||
7. Reliability under distribution shift and adversarial pressure
|
||||
|
||||
A task is scored by which layers it actually needs. Benchmarks are chosen to hit those layers. Theories from sections 1–7 explain *why* a layer is hard; they do not replace the score.
|
||||
|
||||
## Changelog
|
||||
|
||||
- 2026-09-13: v0.1 initial map.
|
||||
58
compendium/02-measurement.md
Normal file
58
compendium/02-measurement.md
Normal file
|
|
@ -0,0 +1,58 @@
|
|||
> Imported exploration, 2026-09-13. Source claims have not been independently verified by this repository. See [integration review](../docs/integration-review.md).
|
||||
|
||||
# Measurement of intelligence
|
||||
|
||||
Version 0.1 · 13 September 2026
|
||||
|
||||
If the maps in `01` are theories, this file is the **instrument panel**. Intelligence Radar refuses to rank models from a single number, but it also refuses to treat every eval as equally informative.
|
||||
|
||||
## Human measurement (background)
|
||||
|
||||
- **Psychometric batteries** — WAIS-class tests, Raven matrices, specialized factor batteries. Human *g* research lives here.
|
||||
- **Expert work samples** — the closest human analogue to GDPval / OSWorld: can a person produce the artifact a job requires.
|
||||
- **Animal batteries** — delayed match, tool use, social inference. Relevant when someone claims “animal-level AGI.”
|
||||
|
||||
These are monitored for conceptual transfer, not for daily model ranking.
|
||||
|
||||
## Artificial measurement — what still discriminates in September 2026
|
||||
|
||||
Classic exams (MMLU, GSM8K, HumanEval, HellaSwag, ARC-Challenge) are largely **saturated** at the frontier. They remain useful as regression tests and as historical series, not as rankers.
|
||||
|
||||
Instruments that still move, with caveats:
|
||||
|
||||
| Instrument | Layer it hits | Caveat |
|
||||
| --- | --- | --- |
|
||||
| Humanity’s Last Exam (HLE) | expert knowledge + reasoning | tool-use variants change the number |
|
||||
| FrontierMath / FrontierMath Erdős | deep mathematical research | tiny N; one problem moves the headline |
|
||||
| GPQA Diamond | graduate science QA | approaching saturation at the very top |
|
||||
| SciCode | scientific coding | scaffold-sensitive |
|
||||
| SWE-bench Verified / Pro, DeepSWE, Frontier Code | software engineering agents | harness and test-contamination risk |
|
||||
| Terminal-Bench 4.0 | CLI / sysadmin / research agents | version jumps rewrite history |
|
||||
| OSWorld 2.0 | long-horizon computer use | live desktop; expensive to run |
|
||||
| ARC-AGI-3 | abstract interactive reasoning | wrapper/harness can swing scores by tens of points |
|
||||
| GDPval-AA v2, GDP.pdf, AA-Briefcase | professional knowledge work | closer to economic value |
|
||||
| AutomationBench-AA | agentic office workflows | private set; good contamination resistance |
|
||||
| AA-Omniscience, SimpleQA, CritPt | factuality / calibration | “smart but wrong” detector |
|
||||
| Artificial Analysis Intelligence Index v4.3 | composite of the above class | re-bases; never compare raw scores across index versions |
|
||||
| Epoch Capabilities Index (ECI) | long-run capability scale | best for *history*, not shopping |
|
||||
| LMSYS / Arena-style preference | human taste | not a substitute for task success |
|
||||
|
||||
## Rules for using scores
|
||||
|
||||
1. **Same harness, same date, same effort setting.** “Max” vs “medium” reasoning is a different product.
|
||||
2. **Price per completed task beats price per million tokens** when the model burns hidden reasoning tokens.
|
||||
3. **Availability is part of the score.** A preview-only or capacity-gated model is tagged, not treated as generally usable.
|
||||
4. **Open weights vs API** are different goods (latency, privacy, sovereignty, fine-tune rights).
|
||||
5. **A 10% degradation** in Task Router language means: on the *task-weighted* blend of the five mapped benchmarks, comparative task-fit proxy is within 10% relative of the Frontier model (see `task-router/METHOD.md`). It is not “10 Elo” and it is not “10 index points.”
|
||||
|
||||
## What we do not pretend to measure well
|
||||
|
||||
- Scientific novelty (a model can ace HLE and still not find a new theorem).
|
||||
- Taste, aesthetics, and institutional judgment.
|
||||
- Long-run reliability in production (weeks of agent autonomy).
|
||||
- Alignment, refusal quality, and dual-use except via lab preparedness designations and independent red-team reports.
|
||||
- Consciousness.
|
||||
|
||||
## Changelog
|
||||
|
||||
- 2026-09-13: v0.1 instrument panel.
|
||||
50
compendium/03-frontier-as-of-2026-09.md
Normal file
50
compendium/03-frontier-as-of-2026-09.md
Normal file
|
|
@ -0,0 +1,50 @@
|
|||
> Imported exploration, 2026-09-13. Source claims have not been independently verified by this repository. See [integration review](../docs/integration-review.md).
|
||||
|
||||
# Artificial capability frontier — snapshot 13 September 2026
|
||||
|
||||
This is a **dated reading**, assembled from public leaderboards and lab announcements. Independent aggregators disagree. Treat ranks as a cloud, not a podium.
|
||||
|
||||
## What “frontier” means here
|
||||
|
||||
A model is **frontier-class** if it is inside the current top cluster on at least one of: Artificial Analysis Intelligence Index (current version), Epoch ECI, a hard agentic coding/computer-use board, or a hard science/math board — *and* it is a product someone can actually call or download.
|
||||
|
||||
Peak-score models that are invitation-only or unreleased (e.g. some Mythos-class preview lanes) are **frontier-adjacent**, not generally usable Frontier.
|
||||
|
||||
## Cluster as of this week
|
||||
|
||||
Rough usable cluster (not a single official ranking):
|
||||
|
||||
| Role this week | Model | Lab | Notes |
|
||||
| --- | --- | --- | --- |
|
||||
| Joint intelligence-index lead (AA v4.3, max effort) | Claude Fable 5.1 and GPT-6 Astra | Anthropic / OpenAI | Tied at 53 on AA Intelligence Index v4.3 after the 4–7 Sep rebase |
|
||||
| Strong general / coding workhorse at mid price | Claude Opus 5 | Anthropic | Same $5/$25 list as older Opus; SWE-bench Verified ~96, Pro ~79.2 (lab-reported) |
|
||||
| Previous OpenAI flagship still widely deployed | GPT-5.6 Sol / Terra / Luna | OpenAI | Sol remains a reference; Terra often the efficiency pick inside the family |
|
||||
| Cheap multimodal workhorse | Gemini 3.8 Flash | Google | GA 2 Sep; Gemini 3.5 Pro still not generally shipped |
|
||||
| Real-time / long-horizon value hosted model | Grok 4.6 | xAI | Current xAI flagship; Grok 4.7 rumored, not shipped as of 13 Sep |
|
||||
| Closed Meta line | Muse Spark 1.3 | Meta Superintelligence Labs | Low list price vs US lab peers on some boards |
|
||||
| Leading open / exportable cluster | Kimi K3, Qwen3.8-Max, DeepSeek V4.x (V4-Pro and V4.1 Flash), GLM-5.3 | Moonshot, Alibaba, DeepSeek, Z.ai | Open or semi-open paths; distillation and export-control fights are political facts |
|
||||
|
||||
**Headline capability claims to verify, not swallow:** GPT-6 Astra’s ARC-AGI-3 number is wrapper-sensitive (public reports of ~63% on ARC Prize harness vs 99.9% on a provider adapter). That gap is itself an Intelligence Radar finding: **the harness is part of the model.**
|
||||
|
||||
## Economics and availability (the other half of the frontier)
|
||||
|
||||
- **Price span** among near-frontier models is two orders of magnitude on blended $/M tokens and still large on *cost per completed hard task*.
|
||||
- **Token cost ≠ task cost.** Reasoning models that emit fewer hidden tokens can beat a cheaper list price.
|
||||
- **Capacity is a constraint.** Mid-September reporting: OpenAI paused some $200 ChatGPT Pro signups/upgrades amid Astra demand. A sold-out Frontier is not an available Frontier.
|
||||
- **Access tiers matter:** Fable 5.1 general vs Mythos 5.1 restricted; Astra first to cybersecurity “Daybreak” customers then broader; some Chinese weights are legally or practically unavailable in parts of the West.
|
||||
- **Hardware sovereignty** is now a first-class story (DeepSeek training paths on non-NVIDIA stacks; EU/US compute buildout; export controls).
|
||||
|
||||
Detailed prices live in `../frontier/ledger-2026-09.md` and will drift. Re-check before any buying decision.
|
||||
|
||||
## What moved this month (for the weekly series)
|
||||
|
||||
- 1 Sep — Claude Fable 5.1 / Mythos 5.1
|
||||
- 2 Sep — Gemini 3.8 Flash; Muse Spark 1.3
|
||||
- 3 Sep — GPT-6 Astra
|
||||
- 4–7 Sep — Artificial Analysis Index v4.2 then v4.3 rebase (harder terminal bench + AutomationBench-AA)
|
||||
- 10 Sep — DeepSeek V4.1 Flash; Cognition SWE-2 (near-frontier coding at lower cost)
|
||||
- Ongoing — Grok 4.7 slip; Gemini 3.5 Pro still unshipped; policy noise in Washington and Beijing
|
||||
|
||||
## Changelog
|
||||
|
||||
- 2026-09-13: v0.1 snapshot.
|
||||
10
data/README.md
Normal file
10
data/README.md
Normal file
|
|
@ -0,0 +1,10 @@
|
|||
# Evidence ledger
|
||||
|
||||
`observations/<sha256>.json` contains immutable envelopes created by the CLI.
|
||||
Each envelope stores a validated record and SHA-256 of its canonical JSON.
|
||||
Source content is represented by its own hash; retain source artifacts separately
|
||||
where permitted. Hashes provide integrity, not truth. Corrections are new records
|
||||
with `supersedes` pointing to the prior observation ID, leaving history intact.
|
||||
|
||||
No model scores or prices from the explorations have been accepted as observations.
|
||||
Synthetic fixtures live in `examples/` and are used only in temporary demo storage.
|
||||
42
digests/daily/2026-09-13.md
Normal file
42
digests/daily/2026-09-13.md
Normal file
|
|
@ -0,0 +1,42 @@
|
|||
> Imported exploration, 2026-09-13. Source claims have not been independently verified by this repository. See [integration review](../../docs/integration-review.md).
|
||||
|
||||
# Daily brief — 13 September 2026
|
||||
|
||||
Standing-up day. This brief is the first radar sweep, not a full 40-minute harvest of every primary.
|
||||
|
||||
## Frontier and markets
|
||||
|
||||
- **GPT-6 Astra** (OpenAI, 3 Sep) remains the week’s gravitational event. OpenAI frames it as a generational step for computer use and has used “Critical” cybersecurity language under its preparedness framework. Independent boards after Artificial Analysis v4.3 put Astra in a **tie with Claude Fable 5.1** on the Intelligence Index (53) at materially lower reported cost-per-index-task. ARC-AGI-3 headlines (99.9% with a provider adapter vs ~63% on the ARC Prize harness in public write-ups) are a measurement scandal and a capability signal at once.
|
||||
- **Claude Fable 5.1** (1 Sep) is generally available; **Mythos 5.1** is a restricted sibling with safeguards lifted for verified programs. Cache-read pricing cut is an economics event, not only a model event.
|
||||
- **Gemini 3.8 Flash** GA (2 Sep). **Gemini 3.5 Pro** still not generally shipped — availability gap.
|
||||
- **Grok 4.7** still not shipped as of tonight; **Grok 4.6** (12 Aug) remains xAI’s current flagship. Musk comments last week pointed at RL length-penalty issues.
|
||||
- **DeepSeek V4.1 Flash** and **Cognition SWE-2** (10 Sep) compress the coding-agent cost curve: near-frontier coding at a discount.
|
||||
- Capacity: mid-September reporting that OpenAI paused some ChatGPT Pro signups/upgrades. Treat **availability as a first-class ledger field**.
|
||||
|
||||
## Science of intelligence
|
||||
|
||||
- Daedalus 2026 intelligence essays remain the best current bridge texts: Tenenbaum (language + probabilistic programs), Ganguli (physics + neuroscience + AI as one science).
|
||||
- NeuroAI: Wang’s “foundational brain model of intelligence” program (PFC-centric, multi-task recurrent circuits) and Pathway’s BDH latent-space reasoning architecture (cost-efficiency claim on ARC-AGI-1) are the two architectural stories worth tracking beside transformers.
|
||||
- In-silico cognitive science: AutoCog-on-Centaur work circulating this month argues theories discovered on behavioral foundation models can generalize to humans. Watch replication, not the press title.
|
||||
|
||||
## Engineering
|
||||
|
||||
- Agent evals, not chat evals, are the ruler: Terminal-Bench 4.0, OSWorld 2.0, AutomationBench-AA, SWE-Pro / DeepSWE / Frontier Code.
|
||||
- Index rebases (AA v4.2 on 4 Sep, v4.3 on 7 Sep) rewrote last week’s rank story. Never quote an Intelligence Index number without the version.
|
||||
|
||||
## Political / legal / social
|
||||
|
||||
- US congressional leadership publicly agrees risks are rising and that there is no quick statute. Track actual text, not Sunday shows.
|
||||
- Amodei slowdown essay and Anthropic distillation accusations toward some Chinese labs are both policy objects. Treat accusations as claims until evidence is public.
|
||||
- OpenAI IPO timing chatter is not a capability event.
|
||||
|
||||
## Ledger actions
|
||||
|
||||
1. Freeze Task Router recipes onto **AA v4.3 + work-sample evals**.
|
||||
2. Tag Astra and Fable 5.1 as joint general Frontier until a task-specific eval breaks the tie.
|
||||
3. Add DeepSeek V4.1 Flash and SWE-2 to the Budget / Efficient watch for coding tasks.
|
||||
4. Do not list Grok 4.7.
|
||||
|
||||
## Sources to re-open tomorrow
|
||||
|
||||
Artificial Analysis Intelligence Index v4.3 article; Epoch benchmark hub; OpenAI Astra card; Anthropic Fable 5.1 card; ARC Prize Astra measurement notes; lab pricing pages.
|
||||
28
digests/weekly/2026-W37.md
Normal file
28
digests/weekly/2026-W37.md
Normal file
|
|
@ -0,0 +1,28 @@
|
|||
> Imported exploration, 2026-09-13. Source claims have not been independently verified by this repository. See [integration review](../../docs/integration-review.md).
|
||||
|
||||
# Weekly digest — 2026-W37 (7–13 September)
|
||||
|
||||
Inaugural week. Ranked by decision relevance for Intelligence Radar.
|
||||
|
||||
1. **Artificial Analysis rebased the Intelligence Index twice (v4.2 then v4.3).** Terminal-Bench 4.0 and AutomationBench-AA entered the composite. Headline: Claude Fable 5.1 and GPT-6 Astra tied at 53. Old scores are not comparable.
|
||||
2. **GPT-6 Astra launched (3 Sep).** Computer-use and cybersecurity-preparedness framing; wrapper-sensitive ARC-AGI-3 numbers; capacity constraints on Pro plans reported later in the week.
|
||||
3. **Claude Fable 5.1 generally available (1 Sep); Mythos 5.1 restricted.** Cache-read price cut changes long-agent economics.
|
||||
4. **Gemini 3.8 Flash GA; Gemini 3.5 Pro still missing.** Google’s usable frontier is a Flash-class model this week.
|
||||
5. **DeepSeek V4.1 Flash + Cognition SWE-2 (10 Sep).** Coding-agent Pareto moved; Budget/Efficient picks for software work must be revisited.
|
||||
6. **Grok 4.7 did not ship.** Grok 4.6 remains the xAI reference.
|
||||
7. **Meta Muse Spark 1.3** continues Meta’s closed frontier line at aggressive prices on some trackers.
|
||||
8. **Measurement politics:** harness gaps (ARC-AGI-3) are now large enough to mislead buyers. Radar rule: name the harness.
|
||||
9. **Policy:** US leadership acknowledges rising risk without a bill; Amodei slowdown proposal; cross-border distillation allegations. No new binding global rule this week.
|
||||
10. **Science-of-intelligence stack** is alive independently of product launches: NeuroAI foundational-brain proposals, complementary cognitive/machine AI frameworks, latent-space recurrent architectures.
|
||||
|
||||
## Frontier one-pager
|
||||
|
||||
Joint general Frontier: **Fable 5.1 ↔ Astra**.
|
||||
Default Efficient generalist: **Opus 5** (task-dependent: Terra or Grok 4.6).
|
||||
Default Budget with evidence: **DeepSeek V4.1 Flash / GLM-5.3-Flash** depending on jurisdiction.
|
||||
|
||||
## Compendium edit proposals
|
||||
|
||||
- Add “harness is part of the model” as a measurement principle. (Accept.)
|
||||
- Do not add Grok 4.7 until a model card exists. (Accept.)
|
||||
- Watch Pathway BDH as an architecture outlier, not a product Frontier. (Accept.)
|
||||
16
docs/adr/0001-initial-operational-baseline.md
Normal file
16
docs/adr/0001-initial-operational-baseline.md
Normal file
|
|
@ -0,0 +1,16 @@
|
|||
# ADR 0001 — Local evidence workflow and normative precedence
|
||||
|
||||
Status: accepted, 2026-09-13.
|
||||
|
||||
Use ChatGPT's INTENT, concept model, and evaluation method as the governing design.
|
||||
Integrate Grok's editorial structure and reading material. Keep source archives
|
||||
immutable. Runtime records are content-addressed JSON envelopes; analysts supply
|
||||
source material and review claims. This needs no hosted infrastructure to operate.
|
||||
|
||||
Precedence: INTENT → TaskModelEvaluationMethod → versioned policies → operational
|
||||
templates. ArchitectureBlueprint describes the target; SCOPE describes implementation.
|
||||
All imported current-world claims require verification before recommendations.
|
||||
Benchmark fitness must not be reported as calibrated task success.
|
||||
|
||||
Domain `infotech`, topic `custodian` fits the existing cognitive infrastructure
|
||||
umbrella. `IR-WP-` identifies file-backed workplans; Statehub is their projection.
|
||||
14
docs/evidence/bootstrap.md
Normal file
14
docs/evidence/bootstrap.md
Normal file
|
|
@ -0,0 +1,14 @@
|
|||
# Bootstrap evidence — 2026-09-13
|
||||
|
||||
- Both source archive SHA-256 checks pass against history/MANIFEST.json.
|
||||
- `make check`: seven behavior tests passed; core files and empty operational ledger valid.
|
||||
- `make demo`: isolated synthetic record, duplicate import, daily draft and weekly draft passed.
|
||||
- Statehub registration: repository `3130bd49-a43f-4682-a25d-68b4d4de9b1f`,
|
||||
topic `cee7bedf-2b48-46ef-8601-006474f2ad7a` (`infotech/custodian`), host `bnt-lap001`.
|
||||
- Source ZIPs contain unverified research claims; no automated facts, live recommendations,
|
||||
scheduler, notifications or paid inference were enabled.
|
||||
- LICENSE is inherited unchanged from the current Coulomb repo-seed template;
|
||||
no repository Phase has been declared.
|
||||
|
||||
Projection verification is in progress. IR-WP-0002-T03 remains open until a primary
|
||||
Statehub reconciliation receipt binds the pushed source commit.
|
||||
48
docs/integration-review.md
Normal file
48
docs/integration-review.md
Normal file
|
|
@ -0,0 +1,48 @@
|
|||
# Founding integration review — 2026-09-13
|
||||
|
||||
## Inputs and evaluation
|
||||
|
||||
Both source ZIPs remain unchanged in `history/`; `history/MANIFEST.json` records
|
||||
SHA-256 checksums. ChatGPT contributes 13 files and Grok contributes 17 files
|
||||
(including a DOCX charter and its JavaScript generator). Neither is a runtime.
|
||||
The DOCX and generator remain recoverable from the Grok archive; its Markdown
|
||||
content supplies the working charter without introducing a document-build dependency.
|
||||
The archive READMEs and ChatGPT MANIFEST remain in the ZIPs as provenance.
|
||||
|
||||
ChatGPT is stronger on entity boundaries, historical reproducibility, source discipline,
|
||||
benchmark validity, task-specific ranking, cost accounting, and calibration.
|
||||
Its architecture is aspirational and its catalogs lack versioned observed results.
|
||||
Grok is stronger on accessible theory maps, editorial cadence, a practical watchlist,
|
||||
and a usable ticket shape. Its illustrative routing arithmetic overstates confidence.
|
||||
Both include dated factual claims without captured retrieval evidence; neither can
|
||||
seed a trusted production model/price database without verification.
|
||||
|
||||
## Resolution
|
||||
|
||||
| Conflict or gap | Integrated decision |
|
||||
| --- | --- |
|
||||
| Two descriptions and methods | INTENT governs; TaskModelEvaluationMethod and policies govern the advisor; task-router is the operator interface |
|
||||
| Fixed Grok weights vs task-specific selection | Select five distinct instruments using benchmark-selection-v0.1; normalize task-specific weights and report weak coverage |
|
||||
| Benchmark percent equals P(success) | Preserve task-fit proxy separately; no Sufficient/Budget claim without adequate calibration evidence |
|
||||
| Grok 0.70/0.85 vs ChatGPT risk thresholds | Adopt exploratory .60, routine .80, important .90, high_assurance .97 as explicit provisional policies |
|
||||
| Less than 10% vs inclusive 0.90 formula | Adopt at most 10% degradation (inclusive .90 boundary) |
|
||||
| Price conflicts, e.g. Terra ranges; blended vs separate rates | Retain historical drafts, do not choose an unsupported price; require dated deployment-specific evidence |
|
||||
| Fixed p25/p50/p75 vs p50/p90 | Use p50/p90 where justified; otherwise a labeled assumption range, never fabricated quantiles |
|
||||
| Aggregate index in worked example | Supersede operational example; retain original in ZIP; composite index cannot replace one of the five instruments |
|
||||
| External automation IDs described as live | Archive-only references; scheduling here is planned and disabled |
|
||||
| PostgreSQL/object storage target with no implementation | Start with auditable Git + content-addressed JSON; retain migration as future work |
|
||||
|
||||
## Imported file treatment
|
||||
|
||||
ChatGPT root design files, policy YAML, benchmark seed and landscape were promoted.
|
||||
Grok compendium, watchlist, frontier snapshots and dated digests were imported with
|
||||
unverified-draft banners. Router method/template/example and ops instructions were
|
||||
rewritten to agree with the governing method. The atlas remains candidate discovery,
|
||||
not a source of verified benchmark versions. Raw originals remain in the ZIPs.
|
||||
|
||||
## Acceptance evidence
|
||||
|
||||
`make check` checks archive integrity, core files, and the observation ledger and
|
||||
runs behavior tests. `make demo` exercises record → validate → daily/weekly draft
|
||||
without polluting the operational ledger. Statehub results are recorded in
|
||||
`docs/evidence/bootstrap.md` after synchronization.
|
||||
12
examples/observation.json
Normal file
12
examples/observation.json
Normal file
|
|
@ -0,0 +1,12 @@
|
|||
{
|
||||
"schema_version": 1,
|
||||
"title": "Synthetic workflow check",
|
||||
"claim": "Synthetic fixture: a fictional benchmark result was published.",
|
||||
"source_url": "https://example.invalid/synthetic",
|
||||
"source_kind": "synthetic",
|
||||
"content_sha256": "39631bb5e5dc9326b23f8c00bf43440a84ce711315c6b799de554c56aa04d133",
|
||||
"observed_at": "2026-09-13T20:00:00+02:00",
|
||||
"event_date": "2026-09-13",
|
||||
"status": "reported",
|
||||
"observer": "demo"
|
||||
}
|
||||
50
frontier/ledger-2026-09.md
Normal file
50
frontier/ledger-2026-09.md
Normal file
|
|
@ -0,0 +1,50 @@
|
|||
> Imported exploration, 2026-09-13. Source claims have not been independently verified by this repository. See [integration review](../docs/integration-review.md).
|
||||
|
||||
# Frontier ledger — September 2026
|
||||
|
||||
Last human pass: 13 September 2026. Prices and ranks rot quickly. Re-verify against Artificial Analysis, lab pricing pages, and Epoch before any purchase.
|
||||
|
||||
Figures below are **indicative public list prices** in USD per million tokens and recent public board readings. Effort settings change everything.
|
||||
|
||||
## Usable cluster (hosted)
|
||||
|
||||
| Model | Input / output $/M (list, public reports) | Role in Radar language | Availability note |
|
||||
| --- | --- | --- | --- |
|
||||
| Claude Fable 5.1 | ~$10 / $50; cache reads cut to ~$0.25 | Frontier-class; often joint AA lead | Generally available; Mythos 5.1 is restricted |
|
||||
| GPT-6 Astra | reports vary; treat ~$10 / $50 as working assumption until card is stable | Frontier-class; computer-use / ARC-AGI-3 specialist claims | Rollout gated; Pro capacity incidents mid-Sep |
|
||||
| Claude Opus 5 | ~$5 / $25 | Topnotch generalist; often Efficient vs Fable | Broad |
|
||||
| GPT-5.6 Sol | promotional cuts reported (~$4 / $20 in some windows) | Topnotch; previous flagship | Broad |
|
||||
| GPT-5.6 Terra | lower than Sol (reports ~$2.5 / $12–15) | Often Efficient inside OpenAI family | Broad |
|
||||
| GPT-5.6 Luna | lower still | Sufficient / Budget for many knowledge tasks | Broad |
|
||||
| Gemini 3.8 Flash | reports ~$0.75 blended class | Sufficient multimodal workhorse; sometimes Budget | GA 2 Sep; watch 2027 price-change notes |
|
||||
| Grok 4.6 | reports ~$2 / $6 | Value hosted frontier-adjacent; long-horizon agents | Current xAI flagship |
|
||||
| Muse Spark 1.3 | very low list vs US peers on some trackers | Watch for agentic coding efficiency | Closed Meta line |
|
||||
| Claude Sonnet 5 | cheaper Claude tier | Sufficient for many writing/coding tasks | Broad |
|
||||
|
||||
## Open / exportable cluster
|
||||
|
||||
| Model | Role | Note |
|
||||
| --- | --- | --- |
|
||||
| Kimi K3 | often strongest open-weight intelligence reading | License and endpoint availability vary by region |
|
||||
| Qwen3.8-Max | multilingual + strong general | Open-weight flagship path |
|
||||
| DeepSeek V4-Pro / V4.1 Flash | Budget-to-Sufficient coding & general | V4.1 Flash: smaller active params, MIT, 1M context |
|
||||
| GLM-5.3 / Flash | open-weight efficiency | Frequent Pareto mention on cost/intelligence |
|
||||
| Mistral Medium 3.5 | EU-sovereign option | Not always peak score |
|
||||
|
||||
## Composite boards (do not mix versions)
|
||||
|
||||
- **AA Intelligence Index v4.3** (10 evals including Terminal-Bench 4.0 + AutomationBench-AA): Fable 5.1 max ≈ Astra max ≈ 53; Opus 5 max ≈ 51; Fable 5 ≈ 50; Muse Spark 1.3 max ≈ 48; Sol max ≈ 47. Cost per AA task reported far lower for Astra than Fable 5.1 in AA’s own methodology (~$3.26 vs ~$7.63 in one published comparison).
|
||||
- **Epoch ECI:** GPT-6 Astra reported at the top of the current staircase (~169), then Fable 5 / Opus 5 cluster.
|
||||
- **Coding / computer use:** SWE-bench Pro, DeepSWE, Terminal-Bench 4.0, OSWorld 2.0, Frontier Code 1.1 — Astra, Fable 5.1, Opus 5, SWE-2, Grok 4.6 trade blows depending on harness.
|
||||
|
||||
## Radar labels (how the Task Router will speak)
|
||||
|
||||
- **Frontier** — best expected success on the task’s five-benchmark blend, ignoring price. If two models are within noise, name both and pick the one with better availability.
|
||||
- **Topnotch** — expected success within 10% *relative* of Frontier on that blend.
|
||||
- **Efficient** — lowest expected *task* cost among Topnotch (tokens × list price × typical traces, not list price alone).
|
||||
- **Sufficient** — judged able to complete the task to an acceptable bar (see METHOD).
|
||||
- **Budget** — lowest expected task cost among Sufficient.
|
||||
|
||||
## Changelog
|
||||
|
||||
- 2026-09-13: v0.1 ledger from public boards. Not an audit.
|
||||
BIN
history/20260913-intelligence-radar-starter-2026-09-13_chatgpt.zip
Executable file
BIN
history/20260913-intelligence-radar-starter-2026-09-13_chatgpt.zip
Executable file
Binary file not shown.
BIN
history/20260913-intelligence-radar_grok.zip
Executable file
BIN
history/20260913-intelligence-radar_grok.zip
Executable file
Binary file not shown.
12
history/MANIFEST.json
Normal file
12
history/MANIFEST.json
Normal file
|
|
@ -0,0 +1,12 @@
|
|||
{
|
||||
"archives": [
|
||||
{
|
||||
"path": "history/20260913-intelligence-radar-starter-2026-09-13_chatgpt.zip",
|
||||
"sha256": "ce8f73d6bc3e73e4ad6820d742346c6cd3f32191b9cf63c46b55e548b9522769"
|
||||
},
|
||||
{
|
||||
"path": "history/20260913-intelligence-radar_grok.zip",
|
||||
"sha256": "fcb2f52fbf9f332f1bb187de3ff437545b75cb6d6510bea36f72107b885d0f8f"
|
||||
}
|
||||
]
|
||||
}
|
||||
1
intelligence_radar/__init__.py
Normal file
1
intelligence_radar/__init__.py
Normal file
|
|
@ -0,0 +1 @@
|
|||
"""Local evidence workflow for Intelligence Radar."""
|
||||
174
intelligence_radar/__main__.py
Normal file
174
intelligence_radar/__main__.py
Normal file
|
|
@ -0,0 +1,174 @@
|
|||
"""Run with python3 -m intelligence_radar. Standard library only."""
|
||||
import argparse
|
||||
from datetime import date, datetime
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import re
|
||||
import sys
|
||||
from urllib.parse import urlparse
|
||||
|
||||
|
||||
def canonical(value):
|
||||
return json.dumps(value, sort_keys=True, separators=(',', ':'), ensure_ascii=False, allow_nan=False).encode()
|
||||
|
||||
|
||||
def digest(value):
|
||||
return hashlib.sha256(canonical(value)).hexdigest()
|
||||
|
||||
|
||||
def validate(record):
|
||||
if not isinstance(record, dict):
|
||||
raise ValueError('record must be an object')
|
||||
required = {'schema_version', 'title', 'claim', 'source_url', 'source_kind',
|
||||
'content_sha256', 'observed_at', 'event_date', 'status', 'observer'}
|
||||
allowed = required | {'supersedes', 'notes'}
|
||||
if set(record) - allowed or required - set(record):
|
||||
raise ValueError('record has missing or unknown fields')
|
||||
if type(record['schema_version']) is not int or record['schema_version'] != 1:
|
||||
raise ValueError('schema_version must be 1')
|
||||
for key in allowed - {'schema_version'}:
|
||||
if key in record and (not isinstance(record[key], str) or not record[key].strip()):
|
||||
raise ValueError(f'{key} must be nonempty text')
|
||||
if record['source_kind'] not in {'primary', 'secondary', 'local', 'synthetic'}:
|
||||
raise ValueError('invalid source_kind')
|
||||
if record['status'] not in {'reported', 'corroborated', 'disputed', 'retracted'}:
|
||||
raise ValueError('invalid status')
|
||||
url = urlparse(record['source_url'])
|
||||
if url.scheme not in {'https', 'http', 'file'} or (url.scheme != 'file' and not url.netloc):
|
||||
raise ValueError('source_url must be an http(s) or file URL')
|
||||
if url.username or url.password:
|
||||
raise ValueError('source_url must not contain credentials')
|
||||
for key in ('content_sha256', 'supersedes'):
|
||||
if key in record and not re.fullmatch('[0-9a-f]{64}', record[key]):
|
||||
raise ValueError(f'{key} must be a lowercase SHA-256')
|
||||
observed = datetime.fromisoformat(record['observed_at'].replace('Z', '+00:00'))
|
||||
if observed.tzinfo is None:
|
||||
raise ValueError('observed_at requires a timezone')
|
||||
event = date.fromisoformat(record['event_date'])
|
||||
if event.isoformat() != record['event_date']:
|
||||
raise ValueError('event_date requires YYYY-MM-DD')
|
||||
if event > observed.date():
|
||||
raise ValueError('event_date cannot follow observation date; describe future plans in claim')
|
||||
|
||||
|
||||
def ledger(root):
|
||||
result = []
|
||||
for path in sorted((root / 'data/observations').glob('*.json')):
|
||||
envelope = json.loads(path.read_text())
|
||||
if not isinstance(envelope, dict) or set(envelope) != {'id', 'record'}:
|
||||
raise ValueError(f'invalid envelope: {path}')
|
||||
validate(envelope['record'])
|
||||
ident = digest(envelope['record'])
|
||||
if envelope['id'] != ident or path.stem != ident:
|
||||
raise ValueError(f'integrity mismatch: {path}')
|
||||
result.append(envelope)
|
||||
ids = {item['id'] for item in result}
|
||||
for item in result:
|
||||
parent = item['record'].get('supersedes')
|
||||
if parent and parent not in ids:
|
||||
raise ValueError(f'unknown superseded observation: {parent}')
|
||||
return result
|
||||
|
||||
|
||||
def record(root, value):
|
||||
validate(value)
|
||||
items = ledger(root)
|
||||
if value.get('supersedes') and value['supersedes'] not in {x['id'] for x in items}:
|
||||
raise ValueError('supersedes must reference an existing observation')
|
||||
ident = digest(value)
|
||||
path = root / 'data/observations' / f'{ident}.json'
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
envelope = {'id': ident, 'record': value}
|
||||
try:
|
||||
with path.open('x') as stream:
|
||||
stream.write(json.dumps(envelope, indent=2, ensure_ascii=False) + '\n')
|
||||
except FileExistsError:
|
||||
if json.loads(path.read_text()) != envelope:
|
||||
raise ValueError('existing observation differs; refusing overwrite') from None
|
||||
return ident
|
||||
|
||||
|
||||
def draft(root, period):
|
||||
if re.fullmatch(r'\d{4}-W\d{2}', period):
|
||||
year, week = period.split('-W')
|
||||
start = date.fromisocalendar(int(year), int(week), 1)
|
||||
matches = lambda d: d.isocalendar()[:2] == start.isocalendar()[:2]
|
||||
kind = 'weekly'
|
||||
else:
|
||||
start = date.fromisoformat(period)
|
||||
if start.isoformat() != period:
|
||||
raise ValueError('use YYYY-MM-DD or YYYY-Www')
|
||||
matches = lambda d: d == start
|
||||
kind = 'daily'
|
||||
items = [x for x in ledger(root) if matches(date.fromisoformat(x['record']['event_date']))]
|
||||
lines = [f'# Radar {kind} draft — {period}', '',
|
||||
'Status: draft; editorial review required. Claims retain their reported status.', '',
|
||||
'Selection: event date; observation timestamps below preserve when evidence was received.', '']
|
||||
if not items:
|
||||
lines += ['No recorded observations for this period. This does not establish that no events occurred.', '']
|
||||
for item in items:
|
||||
r = item['record']
|
||||
lines += [f"## {r['title']}", '', r['claim'], '',
|
||||
f"- Status: {r['status']}; source kind: {r['source_kind']}",
|
||||
f"- Source: {r['source_url']}", f"- Observed: {r['observed_at']}",
|
||||
f"- Observation: {item['id']}"]
|
||||
if r.get('supersedes'):
|
||||
lines.append(f"- Supersedes: {r['supersedes']}")
|
||||
lines.append('')
|
||||
lines += ['## Editorial follow-up', '',
|
||||
'Review source claims, contradictions, frontier/price impacts, compendium edits,',
|
||||
'benchmark health, and evidence gaps. Draft generation is not fact verification.', '']
|
||||
path = root / 'digests' / kind / f'{period}.md'
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with path.open('x') as stream:
|
||||
stream.write('\n'.join(lines))
|
||||
return path
|
||||
|
||||
|
||||
def check(root):
|
||||
required = ['INTENT.md', 'GOAL.md', 'SCOPE.md', 'AGENTS.md', 'README.md',
|
||||
'TaskModelEvaluationMethod.md', 'task-router/METHOD.md',
|
||||
'policies/model-ranking-v0.1.yaml', 'policies/benchmark-selection-v0.1.yaml',
|
||||
'ops/RUNBOOK.md', 'history/MANIFEST.json']
|
||||
for name in required:
|
||||
if not (root / name).is_file():
|
||||
raise ValueError(f'missing {name}')
|
||||
manifest = json.loads((root / 'history/MANIFEST.json').read_text())
|
||||
for item in manifest['archives']:
|
||||
path = (root / item['path']).resolve()
|
||||
if not path.is_relative_to((root / 'history').resolve()):
|
||||
raise ValueError('archive path escapes history')
|
||||
if hashlib.sha256(path.read_bytes()).hexdigest() != item['sha256']:
|
||||
raise ValueError(f'archive checksum mismatch: {path}')
|
||||
return len(ledger(root))
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument('--root', type=Path, default=Path.cwd(), help='repository or isolated ledger root')
|
||||
commands = parser.add_subparsers(dest='command', required=True)
|
||||
commands.add_parser('check', help='check repository, archive integrity, and ledger')
|
||||
commands.add_parser('validate', help='validate every observation and integrity hash')
|
||||
add = commands.add_parser('record', help='record a source observation JSON without overwriting history')
|
||||
add.add_argument('input', type=Path)
|
||||
render = commands.add_parser('digest', help='create a daily or ISO-week draft, refusing overwrite')
|
||||
render.add_argument('period')
|
||||
args = parser.parse_args(argv)
|
||||
try:
|
||||
if args.command == 'record':
|
||||
print(record(args.root, json.loads(args.input.read_text())))
|
||||
elif args.command == 'digest':
|
||||
print(draft(args.root, args.period))
|
||||
elif args.command == 'validate':
|
||||
print(f'{len(ledger(args.root))} valid observations')
|
||||
else:
|
||||
print(f'Repository and archives valid; {check(args.root)} observations')
|
||||
except (ValueError, OSError, KeyError, TypeError) as exc:
|
||||
print(f'error: {exc}', file=sys.stderr)
|
||||
return 1
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
sys.exit(main())
|
||||
15
ops/AUTOMATION-PROMPTS.md
Normal file
15
ops/AUTOMATION-PROMPTS.md
Normal file
|
|
@ -0,0 +1,15 @@
|
|||
# Scheduling status and prompts
|
||||
|
||||
Repository scheduling status: **not configured**. The source Grok archive contains
|
||||
external task/schedule UUIDs; ownership and execution were not verified. They are
|
||||
historical references only. No app notifications or email have been enabled here.
|
||||
|
||||
Daily operator prompt: Review the watchlist for primary-source changes since the
|
||||
previous pass. Record evidence with provenance, publication/event and observation
|
||||
timestamps. Generate a draft for the event date, review each claim and mark gaps.
|
||||
|
||||
Weekly operator prompt: Generate the ISO-week draft. Synthesize decision-relevant
|
||||
changes, unresolved contradictions, frontier evidence gaps, compendium edit proposals,
|
||||
and benchmark health. Link observation IDs. Do not pad an empty week with invented news.
|
||||
|
||||
IR-WP-0003 tracks source adapters and an explicitly configured scheduler.
|
||||
45
ops/CADENCE.md
Normal file
45
ops/CADENCE.md
Normal file
|
|
@ -0,0 +1,45 @@
|
|||
# Operating cadence
|
||||
|
||||
Planned editorial targets; manually operated in v0.1. No scheduler is installed.
|
||||
|
||||
## Daily — 21:30 Europe/Berlin
|
||||
|
||||
Planned job name: `intelligence-radar-daily`
|
||||
|
||||
Produce a brief (800–1,400 words) and store the pattern in `digests/daily/YYYY-MM-DD.md`.
|
||||
|
||||
Must include:
|
||||
|
||||
1. **Frontier & markets** — model releases, price or availability changes, benchmark rebases, capacity incidents.
|
||||
2. **Science of intelligence** — papers or results that change how intelligence is *modeled*, not just applied.
|
||||
3. **Engineering** — software, agents, chips, energy, inference stacks.
|
||||
4. **Political / legal / social** — statutes, suits, labor, safety designations.
|
||||
5. **Watchlist hits** — named thinkers or labs with a primary artifact.
|
||||
6. **Ledger actions** — what should be patched in `frontier/` or `task-router/` tomorrow.
|
||||
7. **Sources** — links to primaries only.
|
||||
|
||||
Ignore rumor without a primary. Tag confidence per item: confirmed / reported / rumor.
|
||||
|
||||
## Weekly — Sunday 20:00 Europe/Berlin
|
||||
|
||||
Planned job name: `intelligence-radar-weekly`
|
||||
|
||||
Produce `digests/weekly/YYYY-Www.md` (ISO week).
|
||||
|
||||
Must include:
|
||||
|
||||
- Top 10 events of the week, ranked by *decision relevance* (does it change who is Frontier, what things cost, what is legal, or what the theory of intelligence is).
|
||||
- Updated one-page frontier snapshot.
|
||||
- Compendium edit proposals (accept/reject next pass).
|
||||
- Open questions for the Task Router atlas (which evals saturated, which recipes broke).
|
||||
|
||||
## On demand
|
||||
|
||||
Any user task → Task Router ticket using `task-router/METHOD.md`.
|
||||
|
||||
## Quality bar
|
||||
|
||||
- No composite rank without naming the index version.
|
||||
- No “best model” without a task.
|
||||
- No copyrighted dumps; summarize.
|
||||
- Separate lab claim from independent eval.
|
||||
42
ops/RUNBOOK.md
Normal file
42
ops/RUNBOOK.md
Normal file
|
|
@ -0,0 +1,42 @@
|
|||
# Operator runbook
|
||||
|
||||
Run commands at the repository root with Python 3.11+. No installation is needed.
|
||||
|
||||
1. Read INTENT, the source catalogs and the last reviewed digest. Choose a primary
|
||||
artifact to inspect. Record event date separately from retrieval time.
|
||||
2. Capture the source hash (`sha256sum path/to/artifact`). Retain the artifact where
|
||||
permitted; summarize rather than reproducing copyrighted text. A hash alone
|
||||
cannot reconstruct an artifact or certify a claim.
|
||||
3. Prepare an observation JSON with the fields shown in
|
||||
[the synthetic fixture](../examples/observation.json). Replace every synthetic
|
||||
value with actual evidence. `observed_at` requires a timezone; `event_date` is
|
||||
the event's date, not a future effective date. Record future plans in the claim.
|
||||
Use source_kind primary/secondary/local; status reported/corroborated/disputed/retracted.
|
||||
These are analyst assertions, not automatic verification results.
|
||||
4. Record and check:
|
||||
|
||||
```bash
|
||||
python3 -m intelligence_radar record /tmp/observation.json
|
||||
python3 -m intelligence_radar validate
|
||||
python3 -m intelligence_radar digest 2026-09-14
|
||||
python3 -m intelligence_radar digest 2026-W38
|
||||
```
|
||||
|
||||
The CLI prints the immutable observation ID. Re-recording identical JSON is
|
||||
idempotent. Changes create a new ID; for corrections add `supersedes` with the
|
||||
old ID. The original remains available. Operators serialize writes; this local
|
||||
prototype is not a concurrent database or a tamper-proof audit service.
|
||||
5. Review the generated draft and its linked evidence. Digests include all records
|
||||
by event date, including disputes and corrections. A weekly draft is an evidence
|
||||
listing, not an automatically synthesized newsletter. Add editorial synthesis,
|
||||
distinguish source claims from independent evidence, and record evidence gaps.
|
||||
Existing digests are never overwritten by the CLI; edit reviewed prose in Git.
|
||||
Late evidence requires an explicit revision or follow-up, not a silent rewrite.
|
||||
6. For recommendations, complete task-router/TEMPLATE.md under the governing method.
|
||||
Preserve unknowns. Real task outcomes later support calibration.
|
||||
7. Run `make check`, review changes, commit the evidence and artifacts. Update
|
||||
workplan files and run `make sync` for Statehub projection.
|
||||
|
||||
`make demo` uses a temporary directory; its synthetic observation never enters the
|
||||
repository ledger. No crawler, scheduled job, network fetch, or paid inference is
|
||||
triggered by these commands. The planned editorial cadence is in CADENCE.md.
|
||||
20
policies/benchmark-selection-v0.1.yaml
Normal file
20
policies/benchmark-selection-v0.1.yaml
Normal file
|
|
@ -0,0 +1,20 @@
|
|||
policy_id: benchmark-selection-v0.1
|
||||
effective_from: 2026-09-13
|
||||
selection_count: 5
|
||||
weights:
|
||||
task_relevance: 0.30
|
||||
ecological_validity: 0.15
|
||||
measurement_reliability: 0.12
|
||||
discriminative_power: 0.12
|
||||
freshness: 0.10
|
||||
execution_profile_similarity: 0.10
|
||||
reproducibility: 0.06
|
||||
contamination_resistance: 0.05
|
||||
penalties:
|
||||
redundancy: enabled
|
||||
known_task_bias: enabled
|
||||
requirements:
|
||||
preserve_benchmark_version: true
|
||||
preserve_execution_profile: true
|
||||
report_weak_coverage: true
|
||||
never_substitute_aggregate_index_for_task_mapping: true
|
||||
29
policies/model-ranking-v0.1.yaml
Normal file
29
policies/model-ranking-v0.1.yaml
Normal file
|
|
@ -0,0 +1,29 @@
|
|||
policy_id: model-ranking-v0.1
|
||||
effective_from: 2026-09-13
|
||||
frontier:
|
||||
objective: maximize_task_fit
|
||||
topnotch:
|
||||
relative_to_frontier_minimum: 0.90
|
||||
efficient:
|
||||
objective: minimize_expected_total_task_cost
|
||||
eligible_set: topnotch
|
||||
sufficient:
|
||||
use_success_probability: true
|
||||
default_thresholds:
|
||||
exploratory: 0.60
|
||||
routine: 0.80
|
||||
important: 0.90
|
||||
high_assurance: 0.97
|
||||
note: >
|
||||
Defaults are policy placeholders and must be replaced or calibrated with
|
||||
task-outcome evidence. High-assurance work may additionally require
|
||||
independent verification.
|
||||
budget:
|
||||
objective: minimize_expected_total_task_cost
|
||||
eligible_set: sufficient
|
||||
economics:
|
||||
require_price_observed_at: true
|
||||
include_retries: true
|
||||
include_tools: true
|
||||
include_verification: true
|
||||
return_uncertainty_interval: true
|
||||
4
registry/README.md
Normal file
4
registry/README.md
Normal file
|
|
@ -0,0 +1,4 @@
|
|||
# Capability registry
|
||||
|
||||
The local evidence workflow is discoverable through indexes/capabilities.yaml.
|
||||
The Capability Advisor remains manual; no runtime API is advertised.
|
||||
|
|
@ -0,0 +1,24 @@
|
|||
---
|
||||
id: capability.infotech.intelligence-radar
|
||||
name: Intelligence Radar evidence workflow
|
||||
summary: Local provenance-preserving observation ledger and draft intelligence digests.
|
||||
owner: intelligence-radar
|
||||
status: draft
|
||||
domain: infotech
|
||||
tags: [research, evidence, evaluation]
|
||||
discovery:
|
||||
intent: Support auditable intelligence research and task-specific capability advice.
|
||||
includes: [observation recording, integrity validation, draft digests, manual advisory method]
|
||||
excludes: [live model rankings, calibrated predictor, hosted API, scheduled crawling]
|
||||
availability:
|
||||
consumption_modes: [cli, informational]
|
||||
current_artifacts: [intelligence_radar/, ops/RUNBOOK.md, task-router/METHOD.md]
|
||||
evidence:
|
||||
documentation: [README.md, SCOPE.md, ops/RUNBOOK.md]
|
||||
tests: [tests/test_workflow.py]
|
||||
---
|
||||
|
||||
# Intelligence Radar evidence workflow
|
||||
|
||||
Run `python3 -m intelligence_radar --help` from a checkout. Python 3.11+ only.
|
||||
The source catalogs contain research leads. Digests require editorial review.
|
||||
13
registry/indexes/capabilities.yaml
Normal file
13
registry/indexes/capabilities.yaml
Normal file
|
|
@ -0,0 +1,13 @@
|
|||
version: 1
|
||||
updated: '2026-09-13'
|
||||
domain: infotech
|
||||
capabilities:
|
||||
- id: capability.infotech.intelligence-radar
|
||||
name: Intelligence Radar evidence workflow
|
||||
summary: Record source observations and draft daily or weekly intelligence digests locally.
|
||||
domain: infotech
|
||||
status: draft
|
||||
owner: intelligence-radar
|
||||
path: registry/capabilities/capability.infotech.intelligence-radar.md
|
||||
tags: [research, evidence, evaluation]
|
||||
consumption_modes: [cli, informational]
|
||||
335
research/2026-09-13-landscape.md
Normal file
335
research/2026-09-13-landscape.md
Normal file
|
|
@ -0,0 +1,335 @@
|
|||
> Imported exploration, 2026-09-13. Source claims have not been independently verified by this repository. See [integration review](../docs/integration-review.md).
|
||||
|
||||
# Intelligence Radar — Initial Landscape Snapshot
|
||||
|
||||
**As of:** 2026-09-13
|
||||
**Purpose:** Seed observation for the Capability/Economics Radar.
|
||||
**Important:** This is a dated snapshot, not a timeless ranking.
|
||||
|
||||
## 1. Main observation
|
||||
|
||||
The current frontier is better described as a **dense capability/cost Pareto surface** than as one universally dominant model.
|
||||
|
||||
At the same time, the measurement regime is changing rapidly: benchmark suites are replacing saturated tests with harder, more agentic, longer-horizon, and increasingly private evaluations.
|
||||
|
||||
This directly validates Intelligence Radar's design decision to make:
|
||||
|
||||
- benchmark selection task-specific;
|
||||
- benchmark versions explicit;
|
||||
- price observations time-indexed;
|
||||
- execution profiles distinct from base models;
|
||||
- real-task outcomes a future calibration layer.
|
||||
|
||||
## 2. Broad independent capability snapshot
|
||||
|
||||
Artificial Analysis announced Intelligence Index v4.3 on 2026-09-07.
|
||||
|
||||
Its v4.3 aggregate includes ten evaluations:
|
||||
|
||||
- AA-Briefcase
|
||||
- GDPval-AA v2
|
||||
- AutomationBench-AA
|
||||
- Terminal-Bench v4.0
|
||||
- SciCode
|
||||
- Humanity's Last Exam
|
||||
- GDP.pdf
|
||||
- CritPt
|
||||
- AA-Omniscience
|
||||
- AA-LCR v1.1
|
||||
|
||||
Reported top aggregate results in that index:
|
||||
|
||||
| Execution profile | Index v4.3 |
|
||||
|---|---:|
|
||||
| Claude Fable 5.1, max with fallback | 53 |
|
||||
| GPT-6 Astra, max | 53 |
|
||||
| Claude Opus 5, max | 51 |
|
||||
| Claude Fable 5, with fallback | 50 |
|
||||
| Muse Spark 1.3, max | 48 |
|
||||
| GPT-5.6 Sol, max | 47 |
|
||||
|
||||
Interpretation:
|
||||
|
||||
- There is a tie on this particular broad aggregate, not proof of universal task equivalence.
|
||||
- Artificial Analysis reports GPT-6 Astra reaching the same aggregate score as Fable 5.1 at roughly 40% of the measured cost per task in its methodology.
|
||||
- Therefore Astra sits on a compelling **aggregate cost/capability frontier** in this measurement regime.
|
||||
- Task-specific Frontier/Efficient conclusions can differ.
|
||||
|
||||
Sources:
|
||||
|
||||
- https://artificialanalysis.ai/articles/artificial-analysis-intelligence-index-v4-3
|
||||
- https://artificialanalysis.ai/articles/benchmarking-gpt-6-astra
|
||||
|
||||
## 3. OpenAI — current observed deployment economics
|
||||
|
||||
OpenAI's current model documentation lists GPT-6 Astra as its most capable model for hard end-to-end work.
|
||||
|
||||
Observed API properties:
|
||||
|
||||
### GPT-6 Astra
|
||||
|
||||
- input: **$10 / 1M tokens**
|
||||
- cached input: **$1 / 1M**
|
||||
- cache write: **$12.50 / 1M**
|
||||
- output: **$50 / 1M**
|
||||
- context: **1.05M**
|
||||
- maximum output: **128K**
|
||||
- knowledge cutoff: **2026-04-30**
|
||||
- reasoning levels include low through max.
|
||||
|
||||
Long prompts above 272K input tokens use higher rates according to OpenAI's model documentation.
|
||||
|
||||
### GPT-5.6 family
|
||||
|
||||
Current OpenAI model listing shows:
|
||||
|
||||
- GPT-5.6 Sol: **$4 input / $20 output per 1M**
|
||||
- GPT-5.6 Terra: **$2 input / $12 output per 1M**
|
||||
- GPT-5.6 Luna: **$0.20 input / $1.20 output per 1M**
|
||||
|
||||
GPT-5.6 Sol pricing is currently promotional according to OpenAI documentation and therefore must be stored with validity metadata.
|
||||
|
||||
Sources:
|
||||
|
||||
- https://developers.openai.com/api/docs/models/gpt-6-astra
|
||||
- https://developers.openai.com/api/docs/models
|
||||
- https://openai.com/index/gpt-5-6/
|
||||
|
||||
## 4. Anthropic — current observed deployment economics
|
||||
|
||||
Anthropic released Claude Fable 5.1 in September 2026.
|
||||
|
||||
Observed Fable 5.1 API pricing:
|
||||
|
||||
- input: **$10 / 1M tokens**
|
||||
- output: **$50 / 1M**
|
||||
- cache reads: **$0.25 / 1M**
|
||||
|
||||
Anthropic states that reduced cache-read pricing lowers typical workload cost relative to Fable 5 and can matter even more for highly agentic workloads.
|
||||
|
||||
Claude Sonnet 5 is currently documented at:
|
||||
|
||||
- input: **$2 / 1M**
|
||||
- output: **$10 / 1M**
|
||||
|
||||
This is an example of why token list prices alone are insufficient: cache-heavy agents can reverse simple nominal-price comparisons.
|
||||
|
||||
Sources:
|
||||
|
||||
- https://www.anthropic.com/claude-fable-and-mythos-5-1
|
||||
- https://www.anthropic.com/claude/fable
|
||||
- https://www.anthropic.com/news/claude-sonnet-5
|
||||
|
||||
## 5. Google — current observed deployment economics
|
||||
|
||||
Gemini 3.8 Flash became generally available in September 2026.
|
||||
|
||||
Observed properties:
|
||||
|
||||
- 1,048,576 input-token limit;
|
||||
- 65,536 output-token limit;
|
||||
- text/image/video/audio/PDF input;
|
||||
- code execution;
|
||||
- computer use in preview;
|
||||
- function calling;
|
||||
- search grounding;
|
||||
- structured outputs;
|
||||
- tunable thinking.
|
||||
|
||||
Introductory standard API pricing through **2026-12-31**:
|
||||
|
||||
- input: **$0.75 / 1M**
|
||||
- output (including thinking tokens): **$3.75 / 1M**
|
||||
- context cache: **$0.075 / 1M**
|
||||
|
||||
Google states standard pricing doubles starting 2027-01-01.
|
||||
|
||||
This is exactly the kind of temporary economic regime Intelligence Radar must represent explicitly.
|
||||
|
||||
Sources:
|
||||
|
||||
- https://ai.google.dev/gemini-api/docs/models/gemini-3.8-flash
|
||||
- https://ai.google.dev/gemini-api/docs/latest-model
|
||||
- https://ai.google.dev/gemini-api/docs/pricing
|
||||
|
||||
## 6. Benchmark regime is moving toward agentic work
|
||||
|
||||
Artificial Analysis' index changes in 2026 are instructive:
|
||||
|
||||
- v4.1 shifted weight toward agentic workloads;
|
||||
- IFBench was removed because of saturation;
|
||||
- Terminal-Bench and τ-bench variants were upgraded;
|
||||
- GDPval methodology was upgraded for longer trajectories and more robust judging;
|
||||
- v4.2 added AA-Briefcase and removed GPQA Diamond after saturation;
|
||||
- v4.3 moved to Terminal-Bench v4 and added AutomationBench-AA with a private test set.
|
||||
|
||||
This suggests a general benchmark lifecycle:
|
||||
|
||||
`novel → discriminative → widely optimized → saturated/gamed → reduced predictive value → replace or re-scope`
|
||||
|
||||
Source:
|
||||
|
||||
- https://artificialanalysis.ai/articles/artificial-analysis-intelligence-index-v4-1
|
||||
- https://artificialanalysis.ai/articles/artificial-analysis-intelligence-index-v4-2
|
||||
- https://artificialanalysis.ai/articles/artificial-analysis-intelligence-index-v4-3
|
||||
|
||||
## 7. Stanford AI Index 2026 reinforces measurement concerns
|
||||
|
||||
Stanford HAI's 2026 AI Index technical-performance chapter highlights:
|
||||
|
||||
- rapid saturation of evaluations;
|
||||
- convergence among leading model providers on some aggregate measures;
|
||||
- benchmark reliability and gaming concerns;
|
||||
- large progress in agentic/computer-use systems;
|
||||
- continuing jaggedness across apparently simple and hard tasks.
|
||||
|
||||
For Intelligence Radar, the important inference is not a specific leaderboard position. It is:
|
||||
|
||||
> Benchmark health itself is a continuously changing research object.
|
||||
|
||||
Source:
|
||||
|
||||
- https://hai.stanford.edu/ai-index/2026-ai-index-report/technical-performance
|
||||
|
||||
## 8. Long-horizon autonomy is becoming directly measurable
|
||||
|
||||
METR's current Time Horizon 1.1 methodology estimates the human-expert task duration at which an AI agent reaches a specified probability of success.
|
||||
|
||||
This is valuable because it reframes autonomy from a vague adjective into a reliability-conditioned horizon.
|
||||
|
||||
METR explicitly warns that measurements above 16 hours are unreliable with its current task suite.
|
||||
|
||||
Intelligence Radar should therefore treat autonomy-horizon measurements with:
|
||||
|
||||
- task-distribution scope;
|
||||
- confidence intervals;
|
||||
- horizon validity limit;
|
||||
- model+agent configuration;
|
||||
- version.
|
||||
|
||||
Source:
|
||||
|
||||
- https://metr.org/time-horizons/
|
||||
|
||||
## 9. Tool use and interactive agents need separate evidence
|
||||
|
||||
BFCL V4 explicitly evaluates function/tool calling and describes its evolution toward holistic agentic evaluation.
|
||||
|
||||
The τ³-bench family expands interactive evaluation to knowledge-aware and voice interaction. Its July 2026 grading update notes that affected results before and after version 1.0.1 are not comparable.
|
||||
|
||||
This strongly supports benchmark versioning as a hard requirement.
|
||||
|
||||
Sources:
|
||||
|
||||
- https://gorilla.cs.berkeley.edu/leaderboard
|
||||
- https://github.com/sierra-research/tau2-bench
|
||||
- https://research.sierra.ai/
|
||||
|
||||
## 10. Software engineering evidence is scaffold-dependent
|
||||
|
||||
SWE-bench evaluates real software issues, but the leaderboard shows results attached to both a **model and an agent/scaffold**.
|
||||
|
||||
SWE-bench Verified is a 500-instance human-filtered set. SWE-bench Multilingual contains 300 curated tasks across 42 repositories and 9 programming languages.
|
||||
|
||||
Intelligence Radar must therefore never transform:
|
||||
|
||||
`Model X + Agent Y = 79%`
|
||||
|
||||
into:
|
||||
|
||||
`Model X = 79% coding ability`
|
||||
|
||||
without preserving the execution profile.
|
||||
|
||||
Sources:
|
||||
|
||||
- https://www.swebench.com/
|
||||
- https://www.swebench.com/multilingual.html
|
||||
|
||||
## 11. Scientific and abstract reasoning remain distinct constructs
|
||||
|
||||
### SciCode
|
||||
|
||||
Scientist-curated scientific-programming tasks test the ability to combine scientific concepts, reasoning, and code synthesis.
|
||||
|
||||
Source:
|
||||
|
||||
- https://scicode-bench.github.io/
|
||||
|
||||
### ARC-AGI-2
|
||||
|
||||
ARC-AGI-2 deliberately stresses abstraction and novel problem solving under efficiency constraints. The 2026 competition sets 85% on the private evaluation set as a major target.
|
||||
|
||||
Source:
|
||||
|
||||
- https://arcprize.org/competitions/2026/arc-agi-2
|
||||
|
||||
### Humanity's Last Exam
|
||||
|
||||
HLE is an expert-level, broad, multimodal academic benchmark published in Nature in January 2026.
|
||||
|
||||
Source:
|
||||
|
||||
- https://www.nature.com/articles/s41586-025-09962-4
|
||||
|
||||
These should not be collapsed into one “reasoning” number without retaining the constructs they measure.
|
||||
|
||||
## 12. Hardware economics belongs inside capability economics
|
||||
|
||||
MLPerf Inference v6.0 substantially updated its inference suite and added/expanded LLM workloads including GPT-OSS 120B and interactive DeepSeek-R1 scenarios.
|
||||
|
||||
For hosted proprietary models, provider pricing can approximate economic cost to the user.
|
||||
|
||||
For open/self-hosted models, Intelligence Radar needs system-level throughput, latency, power, accelerator utilization, and memory measurements.
|
||||
|
||||
MLPerf therefore belongs in the same economic decision graph as model benchmarks.
|
||||
|
||||
Sources:
|
||||
|
||||
- https://mlcommons.org/2026/04/mlperf-inference-v6-0-results/
|
||||
- https://docs.mlcommons.org/inference/index_gh/
|
||||
|
||||
## 13. Current initial hypotheses
|
||||
|
||||
### H1 — “Best model” is increasingly task-local
|
||||
|
||||
**Confidence:** high.
|
||||
|
||||
Evidence: top models are clustered on broad measures while specialized benchmark and cost differences remain large.
|
||||
|
||||
### H2 — Scaffold quality is becoming part of measured intelligence
|
||||
|
||||
**Confidence:** high operationally; philosophical interpretation remains open.
|
||||
|
||||
Agentic benchmarks increasingly measure model+tools+loop rather than frozen next-token competence.
|
||||
|
||||
### H3 — Benchmark half-life is shrinking at the frontier
|
||||
|
||||
**Confidence:** medium-high.
|
||||
|
||||
Repeated benchmark upgrades/removals due to saturation support the claim, but the exact rate needs measurement.
|
||||
|
||||
### H4 — Economics will become as important as aggregate capability
|
||||
|
||||
**Confidence:** high.
|
||||
|
||||
Top performance is clustering while task cost can differ by multiples.
|
||||
|
||||
### H5 — Real task outcome data will be more valuable than generic leaderboards for repeated organizational work
|
||||
|
||||
**Confidence:** hypothesis with strong rationale, not yet empirically proven by Intelligence Radar.
|
||||
|
||||
This should be tested through the Task → Model Performance Ledger.
|
||||
|
||||
## 14. Immediate catalog consequences
|
||||
|
||||
1. Store benchmark versions.
|
||||
2. Store model + scaffold + effort as execution profiles.
|
||||
3. Store provider deployment separately.
|
||||
4. Timestamp all prices.
|
||||
5. Add cache economics.
|
||||
6. Add cost-per-task observations where methodology is known.
|
||||
7. Track benchmark saturation and replacement events.
|
||||
8. Do not globally designate one current Frontier model.
|
||||
9. Designate Frontier only relative to task and active policy.
|
||||
10. Use this snapshot solely as dated seed evidence.
|
||||
58
task-router/BENCHMARK-ATLAS.md
Normal file
58
task-router/BENCHMARK-ATLAS.md
Normal file
|
|
@ -0,0 +1,58 @@
|
|||
> Imported discovery draft; recipes are non-normative. Verify versions and use METHOD.md for selection.
|
||||
|
||||
# Benchmark atlas — Task Router
|
||||
|
||||
Version 0.1 · 13 September 2026
|
||||
Pick five. Recipes at the bottom.
|
||||
|
||||
## Living instruments (preferred)
|
||||
|
||||
| ID | Benchmark | Best-used-for layers | Still discriminating? |
|
||||
| --- | --- | --- | --- |
|
||||
| HLE | Humanity’s Last Exam | know + reason (expert) | yes |
|
||||
| FMath | FrontierMath / Erdős | reason (research math) | yes, tiny N |
|
||||
| GPQA | GPQA Diamond | know + reason (grad science) | fading at the very top |
|
||||
| SciCode | SciCode | reason + tool (scientific code) | yes |
|
||||
| SWE-V | SWE-bench Verified | tool + plan (repo repair) | yes, some saturation at top |
|
||||
| SWE-P | SWE-bench Pro | tool + plan (harder engineering) | yes |
|
||||
| DeepSWE | DeepSWE | tool + plan (agent coding) | yes |
|
||||
| FCode | Frontier Code | tool + plan (front of coding agents) | yes |
|
||||
| Term4 | Terminal-Bench 4.0 | tool + plan (CLI / research ops) | yes (v4 reset) |
|
||||
| OSW2 | OSWorld 2.0 | perceive + tool + plan (desktop) | yes, lots of headroom |
|
||||
| ARC3 | ARC-AGI-3 | reason (interactive abstraction) | yes; **harness-sensitive** |
|
||||
| GDPv | GDPval-AA v2 | know + reason (occupation work) | yes |
|
||||
| GDPpdf | GDP.pdf | know + reason (long professional docs) | yes |
|
||||
| Brief | AA-Briefcase | know + reason (knowledge work pack) | yes |
|
||||
| AutoB | AutomationBench-AA | tool + plan (office agents) | yes |
|
||||
| Omni | AA-Omniscience / SimpleQA | know + reliable (facts) | yes |
|
||||
| Crit | CritPt | reliable (critique / catch errors) | yes |
|
||||
| LCR | AA-LCR | reason (long-context) | yes |
|
||||
| AAIX | AA Intelligence Index (current ver.) | composite | yes, version-locked |
|
||||
| ECI | Epoch Capabilities Index | long-run general capability | yes, for history |
|
||||
| LiveC | LiveCodeBench | tool (contest coding, contamination-aware) | yes |
|
||||
| Browse | BrowseComp / web-research evals | tool + know (search) | use when task is open-web |
|
||||
|
||||
## Saturated / secondary (at most one of five)
|
||||
|
||||
MMLU-Pro, MATH-500, AIME (year-locked), HumanEval, IFEval, Arena Elo.
|
||||
|
||||
## Default five-benchmark recipes
|
||||
|
||||
| Task family | Five |
|
||||
| --- | --- |
|
||||
| Hard general question / research memo | HLE, GDPpdf, Brief, Omni, Crit |
|
||||
| Graduate science explanation | GPQA, HLE, SciCode, Crit, Omni |
|
||||
| Research mathematics | FMath, HLE, SciCode, LCR, Crit |
|
||||
| Production software change | SWE-P, DeepSWE, Term4, LiveC, Crit |
|
||||
| Greenfield coding / scripts | LiveC, SWE-V, Term4, Crit, Omni |
|
||||
| Computer-use / “do this in the GUI” | OSW2, Term4, AutoB, ARC3, Crit |
|
||||
| Office automation / agent workflow | AutoB, GDPv, Term4, OSW2, Brief |
|
||||
| Long-document diligence | GDPpdf, LCR, Brief, Crit, Omni |
|
||||
| Open-web investigation | Browse, Omni, Crit, Brief, HLE |
|
||||
| Multimodal perception + act | OSW2, ARC3, AutoB, GDPv, Crit |
|
||||
| Cheap classification / extraction | Omni, Crit, Brief, GDPv, AAIX |
|
||||
| “What model should we buy?” meta | AAIX, GDPv, SWE-P, Term4, OSW2 |
|
||||
|
||||
## Changelog
|
||||
|
||||
- 2026-09-13: v0.1 atlas. Rebuild whenever a major eval saturates or an index rebases.
|
||||
24
task-router/METHOD.md
Normal file
24
task-router/METHOD.md
Normal file
|
|
@ -0,0 +1,24 @@
|
|||
# Capability Advisor — operational method
|
||||
|
||||
The normative method is [TaskModelEvaluationMethod](../TaskModelEvaluationMethod.md).
|
||||
Use policy IDs `benchmark-selection-v0.1` and `model-ranking-v0.1`.
|
||||
|
||||
1. Record task, deliverable, acceptance criteria, risk class, constraints, tools,
|
||||
budget and requester-accessible deployments. Score the six clarity components 0–1.
|
||||
2. Map required capabilities and hard gates. Select five distinct benchmark versions
|
||||
with task-specific weights summing to one; document relevance, redundancy,
|
||||
weaknesses, provenance and dates. The atlas is only a discovery aid.
|
||||
3. Preserve model, effort, harness, provider and deployment identity for results.
|
||||
Treat missing results as missing. Label normalized benchmark scores as proxies.
|
||||
4. Rank eligible profiles by comparative task fitness. Frontier maximizes fitness;
|
||||
Topnotch includes fitness >= .90 of Frontier. Efficient minimizes expected total
|
||||
task cost within Topnotch, using dated prices and explicit workload assumptions.
|
||||
5. Evaluate Sufficient against the risk threshold using calibrated success evidence.
|
||||
If missing, report `no_evidenced_sufficient_model`; Budget is then undetermined.
|
||||
Do not derive retries by dividing by an uncalibrated benchmark score.
|
||||
6. Explain confidence dimensions and evidence gaps. Estimate cost/runtime p50/p90
|
||||
only where evidence supports quantiles; otherwise label a scenario range.
|
||||
7. Compare optional decomposition with a monolithic baseline including coordination,
|
||||
verification and failure costs. Record actual outcomes for later calibration.
|
||||
|
||||
Use [TEMPLATE](TEMPLATE.md). This is a manual workflow in v0.1, not an API promise.
|
||||
40
task-router/TEMPLATE.md
Normal file
40
task-router/TEMPLATE.md
Normal file
|
|
@ -0,0 +1,40 @@
|
|||
# Capability Advisor ticket
|
||||
|
||||
Status: draft / reviewed
|
||||
Date and evidence cutoff:
|
||||
Policy IDs: benchmark-selection-v0.1; model-ranking-v0.1
|
||||
Task / deliverable / acceptance criteria:
|
||||
Risk class and explicit sufficiency threshold:
|
||||
Hard gates / tools / privacy / region / budget:
|
||||
Clarity components (0–1): objective, context, constraints, acceptance, output, environment
|
||||
Capability profile:
|
||||
|
||||
## Five measurements
|
||||
|
||||
| Benchmark + version | Observation ID / source / date | Weight | Relevance and validity | Gaps / redundancy |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| 1 | | | | |
|
||||
| 2 | | | | |
|
||||
| 3 | | | | |
|
||||
| 4 | | | | |
|
||||
| 5 | | | | |
|
||||
|
||||
Weights sum to one. No composite index substituted for a measurement.
|
||||
|
||||
## Candidates and decision
|
||||
|
||||
| Profile + deployment + harness + effort | Eligibility | Fitness proxy | Calibrated success / evidence | Expected cost |
|
||||
| --- | --- | --- | --- | --- |
|
||||
|
||||
Frontier:
|
||||
Topnotch:
|
||||
Efficient:
|
||||
Sufficient: no_evidenced_sufficient_model until supported
|
||||
Budget: undetermined until supported
|
||||
|
||||
Price observation IDs and dates; input/cache/output/reasoning/tool/retry/verifier assumptions:
|
||||
Cost and runtime p50/p90 (or explicitly labeled scenario ranges):
|
||||
Confidence by component and what would change the decision:
|
||||
Evidence gaps:
|
||||
Optional decomposition vs monolithic baseline:
|
||||
Outcome: actual configuration, acceptance result, cost, tokens, runtime, retries, failure mode:
|
||||
24
task-router/examples/2026-09-13-worked-example.md
Normal file
24
task-router/examples/2026-09-13-worked-example.md
Normal file
|
|
@ -0,0 +1,24 @@
|
|||
# Worked example — evidence-limited routing
|
||||
|
||||
Task: produce a sourced technical diligence memo and migration plan for a coding-agent stack.
|
||||
This is a manual workflow illustration, not a live model recommendation.
|
||||
|
||||
Missing context: current stack, task distribution, acceptable failure rate, region,
|
||||
and access to candidate deployments. Acceptance: reviewed memo plus reproducible cost model.
|
||||
Risk policy: important, provisional threshold .90.
|
||||
|
||||
Candidate instruments: SWE-bench Multilingual (repository work), Terminal-Bench
|
||||
(shell work), BrowseComp (source discovery), GDPval (deliverables), BFCL (tool use).
|
||||
Their exact versions, result dates, independence and requester harness match need
|
||||
verification. No weights or model results are invented to fill the ticket.
|
||||
|
||||
Frontier / Topnotch / Efficient: undetermined pending eligible profile evidence.
|
||||
Sufficient: `no_evidenced_sufficient_model`. Budget: undetermined.
|
||||
Cost/runtime: unknown until prices and representative traces are observed.
|
||||
|
||||
Next: collect versioned results and deployment prices, run representative work samples,
|
||||
then compare retrieval → analysis → independent verification against monolithic execution.
|
||||
Record measured outcomes; do not assert an unsupported percentage saving.
|
||||
|
||||
The original Grok example, with named winners and assumed savings, is preserved in
|
||||
the source ZIP and superseded by this evidence-limited example.
|
||||
79
tests/test_workflow.py
Normal file
79
tests/test_workflow.py
Normal file
|
|
@ -0,0 +1,79 @@
|
|||
import json
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
|
||||
from intelligence_radar.__main__ import draft, ledger, record, validate
|
||||
|
||||
FIXTURE = Path(__file__).resolve().parents[1] / 'examples/observation.json'
|
||||
|
||||
|
||||
class WorkflowTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.temp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(self.temp.cleanup)
|
||||
self.root = Path(self.temp.name)
|
||||
self.value = json.loads(FIXTURE.read_text())
|
||||
|
||||
def test_idempotence_and_append_only_correction(self):
|
||||
original = record(self.root, self.value)
|
||||
self.assertEqual(record(self.root, self.value), original)
|
||||
replacement = dict(self.value, claim='Corrected synthetic claim', supersedes=original)
|
||||
correction = record(self.root, replacement)
|
||||
self.assertNotEqual(original, correction)
|
||||
self.assertEqual(len(ledger(self.root)), 2)
|
||||
text = draft(self.root, '2026-09-13').read_text()
|
||||
self.assertIn(f'Supersedes: {original}', text)
|
||||
with self.assertRaises(FileExistsError):
|
||||
draft(self.root, '2026-09-13')
|
||||
|
||||
def test_tampered_record_is_rejected_by_reader_and_writer(self):
|
||||
ident = record(self.root, self.value)
|
||||
path = self.root / 'data/observations' / f'{ident}.json'
|
||||
envelope = json.loads(path.read_text())
|
||||
envelope['record']['claim'] = 'tampered'
|
||||
path.write_text(json.dumps(envelope))
|
||||
for fn in (lambda: ledger(self.root), lambda: record(self.root, self.value)):
|
||||
with self.assertRaisesRegex(ValueError, 'integrity'):
|
||||
fn()
|
||||
|
||||
def test_timezone_date_hash_and_shape_validation(self):
|
||||
for change in ({'observed_at': '2026-09-13T20:00:00'},
|
||||
{'event_date': '2026-09-14'}, {'content_sha256': 'bad'},
|
||||
{'source_url': 'https://user:password@example.invalid'},
|
||||
{'schema_version': True}, {'status': 'verified_by_magic'},
|
||||
{'unexpected': 'field'}):
|
||||
with self.subTest(change=change), self.assertRaises(ValueError):
|
||||
validate(dict(self.value, **change))
|
||||
|
||||
def test_unknown_correction_target_rejected(self):
|
||||
with self.assertRaisesRegex(ValueError, 'existing observation'):
|
||||
record(self.root, dict(self.value, supersedes='0' * 64))
|
||||
|
||||
def test_iso_week_year_boundary_and_event_date_filter(self):
|
||||
ident = record(self.root, dict(self.value, event_date='2025-12-29',
|
||||
observed_at='2026-01-02T12:00:00Z'))
|
||||
self.assertIn(ident, draft(self.root, '2026-W01').read_text())
|
||||
self.assertIn('No recorded observations', draft(self.root, '2026-01-02').read_text())
|
||||
with self.assertRaises(ValueError):
|
||||
draft(self.root, '2026-W99')
|
||||
|
||||
def test_invalid_cli_input_exits_nonzero_without_traceback(self):
|
||||
invalid = self.root / 'invalid.json'
|
||||
invalid.write_text('[]')
|
||||
result = subprocess.run([sys.executable, '-m', 'intelligence_radar',
|
||||
'--root', str(self.root), 'record', str(invalid)],
|
||||
capture_output=True, text=True)
|
||||
self.assertEqual(result.returncode, 1)
|
||||
self.assertIn('record must be an object', result.stderr)
|
||||
self.assertNotIn('Traceback', result.stderr)
|
||||
|
||||
def test_empty_period_does_not_invent_news(self):
|
||||
self.assertIn('does not establish that no events occurred',
|
||||
draft(self.root, '2026-09-13').read_text())
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
20
tools/demo.py
Normal file
20
tools/demo.py
Normal file
|
|
@ -0,0 +1,20 @@
|
|||
"""Isolated, synthetic workflow demonstration."""
|
||||
import json
|
||||
from pathlib import Path
|
||||
import sys
|
||||
import tempfile
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
||||
from intelligence_radar.__main__ import draft, ledger, record
|
||||
|
||||
fixture = json.loads((Path(__file__).resolve().parents[1] / 'examples/observation.json').read_text())
|
||||
with tempfile.TemporaryDirectory(prefix='intelligence-radar-demo-') as directory:
|
||||
root = Path(directory)
|
||||
ident = record(root, fixture)
|
||||
assert record(root, fixture) == ident
|
||||
assert len(ledger(root)) == 1
|
||||
for period in ('2026-09-13', '2026-W37'):
|
||||
path = draft(root, period)
|
||||
assert ident in path.read_text()
|
||||
print(path.read_text())
|
||||
print('Demo passed; temporary synthetic ledger removed.')
|
||||
79
watchlist/SOURCES.md
Normal file
79
watchlist/SOURCES.md
Normal file
|
|
@ -0,0 +1,79 @@
|
|||
> Imported exploration, 2026-09-13. Source claims have not been independently verified by this repository. See [integration review](../docs/integration-review.md).
|
||||
|
||||
# Watchlist — original outlets and thinkers
|
||||
|
||||
Version 0.1 · 13 September 2026
|
||||
|
||||
Daily monitoring prefers **primary outlets**. Commentary is used to find primaries, not as the record.
|
||||
|
||||
## A. Labs and model builders (primary)
|
||||
|
||||
- OpenAI — research, product, preparedness, system cards
|
||||
- Anthropic — research, alignment, economic index, model cards
|
||||
- Google DeepMind / Google AI — Gemini, science models, weather/bio
|
||||
- xAI — Grok model cards and API docs
|
||||
- Meta Superintelligence Labs / Meta AI — Muse Spark, Llama, research
|
||||
- Mistral, Cohere
|
||||
- DeepSeek, Alibaba Qwen, Moonshot Kimi, Z.ai / Zhipu GLM, MiniMax, Tencent Hunyuan, Thinking Machines
|
||||
- Cognition, Adept-class agent firms, World Labs, Pathway (architecture outliers)
|
||||
|
||||
## B. Measurement houses (primary for the ledger)
|
||||
|
||||
- Artificial Analysis
|
||||
- Epoch AI (benchmarks, ECI, compute/data trackers)
|
||||
- ARC Prize
|
||||
- SWE-bench / Terminal-Bench / OSWorld maintainers
|
||||
- Stanford HAI AI Index, HELM
|
||||
- LMSYS / Arena (preference, not task success)
|
||||
- BenchLM, UnifyBench, and other evidence-tracing boards (secondary corroboration)
|
||||
|
||||
## C. Science of intelligence (primary)
|
||||
|
||||
- arXiv cs.AI, cs.LG, cs.CL, q-bio.NC; Nature / Science / Neuron / Nat. Commun. when they land
|
||||
- Cognitive science venues (Psychological Science, Cognitive Science, Daedalus intelligence issues)
|
||||
- NeuroAI programs: Wang-style foundational brain models, connectomics, PFC control
|
||||
- Tenenbaum / Ganguli / Gonzalez complementary-intelligence lines
|
||||
- Papers With Code only as an index, not as a source of truth
|
||||
|
||||
## D. Hardware, systems, energy
|
||||
|
||||
- NVIDIA, AMD, TSMC, ASML, Huawei Ascend coverage from primary filings
|
||||
- Hyperscaler capex letters (MSFT, GOOG, AMZN, META, ORCL)
|
||||
- National compute plans (US, EU, CN, IN, Gulf)
|
||||
|
||||
## E. Economy, firms, markets
|
||||
|
||||
- Lab blogs + The Information / Semafor / Bloomberg for deal flow (verify)
|
||||
- Earnings transcripts
|
||||
- Pricing pages and status dashboards (availability is data)
|
||||
|
||||
## F. Politics, law, regulation
|
||||
|
||||
- Official texts: US executive orders and agency rules, EU AI Act implementing acts, China interim measures, UK AISI / DSIT, OECD, UN
|
||||
- Court dockets on copyright, product liability, export controls
|
||||
- Preparedness / safety institute evaluations (US AISI, EU equivalents, independent red teams)
|
||||
|
||||
## G. Thinkers to read in the original (not the clip)
|
||||
|
||||
Keep the list short and rotate. Current standing set:
|
||||
|
||||
**Builders / research leads:** Dario Amodei, Demis Hassabis, Shane Legg, Ilya Sutskever, Noam Brown, Jared Kaplan, Jack Clark, Sepp Hochreiter, Yann LeCun, Geoffrey Hinton, Yoshua Bengio, Fei-Fei Li, Oriol Vinyals, Chelsea Finn, Jim Fan, John Schulman-class post-training leads, Moonshot / DeepSeek / Qwen chief scientists when they publish.
|
||||
|
||||
**Measurement / forecasting:** Epoch AI authors, Artificial Analysis, ARC Prize (Chollet and successors), METR / eval orgs, Ajeya Cotra–class compute forecasts, Stanford HAI index leads.
|
||||
|
||||
**Cognitive science / NeuroAI:** Joshua Tenenbaum, Surya Ganguli, Xiao-Jing Wang, Karl Friston (predictive processing — track influence, do not endorse), Nancy Kanwisher / Ev Fedorenko language-brain line, Anne Collins / Tom Griffiths resource-rational line.
|
||||
|
||||
**Political economy / law:** Jack Clark (Import AI), Helen Toner–class policy, EU and US agency technical staff papers, competition-authority reports.
|
||||
|
||||
**Public squares (secondary, high noise):** X accounts of the above; Hacker News; LessWrong / Alignment Forum only when a primary artifact is linked.
|
||||
|
||||
## H. Daily scan order (30–40 minutes of model time)
|
||||
|
||||
1. Lab blogs + X posts from standing thinkers that link a primary
|
||||
2. Artificial Analysis / Epoch / ARC / agent-bench leaderboards for *deltas*
|
||||
3. arXiv cs.AI/LG frontpage + Nature/Science news
|
||||
4. Regulator and court feeds
|
||||
5. Pricing/status/capacity incidents
|
||||
6. One pass of quality press to catch missed primaries (MIT TR, Ars, The Information, Axios, Semafor, ThursdAI release log)
|
||||
|
||||
Reject: product roundups with no source, leaked slides without corroboration, “AGI declared” quotes without a measurement.
|
||||
57
workplans/IR-WP-0001-statehub-bootstrap.md
Normal file
57
workplans/IR-WP-0001-statehub-bootstrap.md
Normal file
|
|
@ -0,0 +1,57 @@
|
|||
---
|
||||
id: IR-WP-0001
|
||||
type: workplan
|
||||
title: "Bootstrap State Hub integration"
|
||||
domain: infotech
|
||||
repo: intelligence-radar
|
||||
status: finished
|
||||
owner: codex
|
||||
topic_slug: custodian
|
||||
created: "2026-09-13"
|
||||
updated: "2026-09-13"
|
||||
---
|
||||
|
||||
# Bootstrap State Hub integration
|
||||
|
||||
Epistemic observatory and task-specific machine intelligence capability and economics advisor.
|
||||
|
||||
## Review Generated Integration Files
|
||||
|
||||
```task
|
||||
id: IR-WP-0001-T01
|
||||
status: done
|
||||
priority: high
|
||||
```
|
||||
|
||||
Review `INTENT.md`, `SCOPE.md`, `AGENTS.md`, and `.custodian-brief.md`.
|
||||
Replace generated placeholders with repo-specific facts where needed.
|
||||
|
||||
## Verify Local Developer Workflow
|
||||
|
||||
```task
|
||||
id: IR-WP-0001-T02
|
||||
status: done
|
||||
priority: high
|
||||
```
|
||||
|
||||
Identify the repo's install, test, lint, build, and run commands. Add or refine
|
||||
those commands in the agent instructions so future coding sessions can verify
|
||||
changes confidently.
|
||||
|
||||
## Seed First Real Workplan
|
||||
|
||||
```task
|
||||
id: IR-WP-0001-T03
|
||||
status: done
|
||||
priority: medium
|
||||
```
|
||||
|
||||
Create the first implementation workplan for the repository's most important
|
||||
next change. After workplan file updates, run the sync locally from this repo
|
||||
checkout:
|
||||
|
||||
```bash
|
||||
make sync
|
||||
```
|
||||
|
||||
Completion: governing files reviewed, developer commands tested, IR-WP-0002 and residual IR-WP-0003 created.
|
||||
47
workplans/IR-WP-0002-integrated-operational-baseline.md
Normal file
47
workplans/IR-WP-0002-integrated-operational-baseline.md
Normal file
|
|
@ -0,0 +1,47 @@
|
|||
---
|
||||
id: IR-WP-0002
|
||||
type: workplan
|
||||
title: "Integrate explorations into an operational baseline"
|
||||
domain: infotech
|
||||
repo: intelligence-radar
|
||||
status: active
|
||||
owner: codex
|
||||
topic_slug: custodian
|
||||
created: "2026-09-13"
|
||||
updated: "2026-09-13"
|
||||
---
|
||||
|
||||
# Integrate explorations into an operational baseline
|
||||
|
||||
## Evaluate and integrate founding material
|
||||
|
||||
```task
|
||||
id: IR-WP-0002-T01
|
||||
status: done
|
||||
priority: medium
|
||||
```
|
||||
|
||||
Preserve archive checksums; promote INTENT and base documents; reconcile methods and label unverified research. Evidence: docs/integration-review.md and ADR 0001.
|
||||
|
||||
## Implement local evidence and digest workflow
|
||||
|
||||
```task
|
||||
id: IR-WP-0002-T02
|
||||
status: done
|
||||
priority: medium
|
||||
```
|
||||
|
||||
Provide immutable records, validation, daily/weekly drafts, runbook, tests, and isolated synthetic demonstration. Acceptance: make check and make demo pass.
|
||||
|
||||
## Complete Coulomb scaffold and synchronization
|
||||
|
||||
```task
|
||||
id: IR-WP-0002-T03
|
||||
status: progress
|
||||
priority: medium
|
||||
```
|
||||
|
||||
Provide agent entry points, classification, license inherited from repo-seed, registry, and file-backed workplans. Verify registration and Statehub projection. Record evidence in docs/evidence/bootstrap.md. Residual capabilities are tracked in IR-WP-0003.
|
||||
|
||||
|
||||
Validation: make check passed seven behavior tests and archive/ledger checks; make demo passed daily/weekly rendering in temporary storage.
|
||||
45
workplans/IR-WP-0003-verified-radar-and-advisor.md
Normal file
45
workplans/IR-WP-0003-verified-radar-and-advisor.md
Normal file
|
|
@ -0,0 +1,45 @@
|
|||
---
|
||||
id: IR-WP-0003
|
||||
type: workplan
|
||||
title: "Verified sources and calibrated advisory capability"
|
||||
domain: infotech
|
||||
repo: intelligence-radar
|
||||
status: backlog
|
||||
owner: codex
|
||||
topic_slug: custodian
|
||||
created: "2026-09-13"
|
||||
updated: "2026-09-13"
|
||||
---
|
||||
|
||||
# Verified sources and calibrated advisory capability
|
||||
|
||||
## Verify and structure source evidence
|
||||
|
||||
```task
|
||||
id: IR-WP-0003-T01
|
||||
status: todo
|
||||
priority: medium
|
||||
```
|
||||
|
||||
Residual of IR-WP-0002: validate imported benchmark versions, prices and availability against primary artifacts; add source IDs and exact execution profiles. Acceptance: one sourced task-specific comparison with explicit missing-data handling.
|
||||
|
||||
## Implement capability advisor and outcome calibration
|
||||
|
||||
```task
|
||||
id: IR-WP-0003-T02
|
||||
status: todo
|
||||
priority: medium
|
||||
```
|
||||
|
||||
Residual of IR-WP-0002: implement five-instrument selection, eligibility gates, policy-versioned ranking, workload economics and actual outcome records. Acceptance: reproducible historical recommendation; no unsupported success probability.
|
||||
|
||||
## Automate ingestion and editorial cadence
|
||||
|
||||
```task
|
||||
id: IR-WP-0003-T03
|
||||
status: todo
|
||||
priority: medium
|
||||
```
|
||||
|
||||
Residual of IR-WP-0002: add source adapters, deduplication, retrieval artifacts and a configured scheduler with retry/health evidence. Confirm ownership of any external automation before reuse; repository scheduler must be explicit and testable.
|
||||
|
||||
Loading…
Add table
Add a link
Reference in a new issue