Establish Freedom Intelligence lab foundation and baseline research.

Add INTENT/SCOPE, daily-brief playbook, activity-core definition (disabled), workplans FI-WP-0001..0003, baseline field survey with open-weight collection recommendations, and inventory catalog candidates for the model reserve.
2026-07-24 00:15:27 +02:00 · 2026-07-24 00:15:27 +02:00 · b5f911140b
commit b5f911140b
parent 1dc1e09517
28 changed files with 2651 additions and 1 deletions
--- a/research/2026-07-24-baseline-field-survey.md
+++ b/research/2026-07-24-baseline-field-survey.md
@ -0,0 +1,279 @@
+# Baseline field survey — Freedom Intelligence starting point
+
+**Date:** 2026-07-24  
+**Type:** deep baseline (not a daily delta brief)  
+**Purpose:** Establish longitudinal memory so the **first daily research brief**
+can be a true delta, not a full survey. Also recommend the **initial open-weight
+collection** for the lab reserve.  
+**Axes:** A frontier · B edge/local · C training · D harness/fleet  
+**Sources:** public web surveys, HF/model docs, pricing roundups, harness project
+sites (mid–late 2026 public reporting). Prices and leaderboard claims move weekly —
+treat numbers as **order-of-magnitude** and re-check on first daily.
+
+---
+
+## 1. Executive snapshot
+
+| Theme | State of the field (mid-2026) | Lab implication |
+| ----- | ----------------------------- | --------------- |
+| Frontier | Multi-vendor race: OpenAI GPT-5.x, Anthropic Claude 4.x/Opus, Google Gemini 3.x, xAI Grok 4.x; coding agents dominate product narrative | Keep API access via `llm-connect`; do not try to host frontier closed weights |
+| Price collapse | DeepSeek-class APIs often **10–100×** cheaper than flagship closed models for many tasks | Route cheap/easy work to low-cost APIs; reserve local for privacy, offline, specialization |
+| Open weights | Strong mid-size dense + MoE story (Qwen3, DeepSeek distill/V3-class, Gemma/Llama small, Mistral small) | Homelab can be *useful* without 8×H100 |
+| Local stack | llama.cpp / vLLM / MLX / Ollama mature; Q4–Q5 quants standard | Axis B is operationally ready |
+| Training | QLoRA/Unsloth/TRL still the homelab path; full pretrain not lab-realistic | Axis C = adapters + domain SFT, not from-scratch |
+| Harness | OpenHands, Aider, SWE-agent, OpenCode, Cline; closed CLIs (Claude Code, Codex) lead quality | Axis D: watch + integrate with sand-boxer / agent-harness, don’t fork everything |
+
+**Central proposition for the reserve:** collect a **compact spine** of (1) small
+general instruct, (2) mid-size code/reason distill, (3) multilingual embed,
+(4) optional larger “lab ceiling” base — not a mirror of Hugging Face.
+
+---
+
+## 2. Axis A — Frontier capabilities, benchmarks, price
+
+### 2.1 Capability landscape
+
+Public 2026 reporting groups the closed frontier into:
+
+- **Flagship general / agentic** — OpenAI GPT-5.x family, Anthropic Claude Opus/Sonnet 4.x+, Google Gemini 3.x Pro
+- **Fast / cheap tiers** — GPT mini/nano, Claude Haiku, Gemini Flash
+- **Reasoning-heavy** — “thinking” / extended variants; cost dominated by output tokens
+- **Coding productization** — agent harnesses matter as much as base model (SWE-bench Verified numbers often quoted 70%+ with strong closed models + good harness; novel-issue sets much lower ~18–20%)
+
+Benchmark caveats for the lab:
+
+- Arena and public leaderboards are **gamed and lag** product quality
+- SWE-bench Verified ≠ “our monorepos + NetKingdom tools”
+- Prefer **internal harness scores** once we have them; use public numbers only as directional
+
+### 2.2 Price-point trends (API, ~per 1M tokens)
+
+Synthesized from 2026 pricing roundups (verify on provider pages before budget decisions):
+
+| Band | Examples (reported ranges) | Role |
+| ---- | -------------------------- | ---- |
+| **Ultra-cheap** | DeepSeek V3.x / V4 Flash-class (~$0.14–0.28 in / ~$0.28–0.42 out; cache hits lower) | Bulk classify, draft, simple tools |
+| **Budget closed** | Gemini Flash, GPT mini/nano, Haiku-class | Default automation when quality bar is modest |
+| **Mid flagship** | Claude Sonnet-class, GPT mid-tier (~$2–5 in / ~$10–15 out) | Daily agent coding, serious writing |
+| **Top flagship** | Opus / GPT pro tiers (~$5+ in / $25–30+ out; pro tiers much higher) | Hard reasoning, high-stakes review |
+
+**Trend:** continuous compression of mid-tier quality cost; flagship **output**
+tokens remain expensive. Caching and batch APIs swing real cost 50%+.
+
+**Lab stance:**
+
+- Frontier APIs stay first-class for hard work (via llm-connect / OpenRouter-class)
+- Daily *research* about price moves is high value for routing policy
+- Local models win on **privacy, offline, predictable unit cost, specialization** — not always raw SOTA
+
+### 2.3 Open near-frontier (API or weights)
+
+DeepSeek open-weight + cheap API continues to be the structural disruptor:
+MIT-leaning releases for R1/V3-class artifacts, distill series for single-GPU
+reasoning. Track **V3/V4 Flash vs Pro** naming carefully — product lines rename often.
+
+---
+
+## 3. Axis B — Edge, local, open compute
+
+### 3.1 What “homelab useful” means now
+
+| Class | Params (ballpark) | Typical local use |
+| ----- | ----------------- | ----------------- |
+| Edge / CPU-OK | ≤3B Q4 | Chat toys, classification, always-on small agents |
+| Consumer GPU | 7–14B Q4/Q5 | Daily coding assist, RAG chat, tool loops |
+| Single enthusiast GPU | 32B Q4 | Stronger reason/code; slower |
+| Lab multi-GPU | 70B Q4 or sparse MoE | Optional ceiling; not required for v1 reserve |
+
+Runtimes to track: **llama.cpp**, **vLLM**, **MLX** (Apple), **Ollama** (UX),
+TensorRT-LLM (NVIDIA optimize). Speculative decoding + good quant matter more
+than chasing every new 70B.
+
+### 3.2 Open model families to watch (weights)
+
+| Family | Why it matters | License caution |
+| ------ | -------------- | --------------- |
+| **Qwen3** (Alibaba) | Dense + MoE line; strong multilingual & tool use; embeddings/VL variants | Apache-class for many; check model card |
+| **DeepSeek** (V3/R1 + distills) | Reasoning + cost; MIT for key R1/distill artifacts | Confirm per-revision card |
+| **Llama 3.x / 4 text** | Ecosystem, tooling, small 1B–3B instruct | Llama community license (not pure MIT) |
+| **Gemma 3** | Google small/efficient; EmbeddingGemma | Gemma terms |
+| **Mistral Small / Ministral** | EU-friendly commercial open options | Apache / research licenses vary |
+| **Phi / Smol** class | Edge CPU experiments | Check commercial terms |
+
+### 3.3 Embeddings & retrieval (local RAG spine)
+
+For NetKingdom docs, code, and ops notes:
+
+| Model | Role | Notes |
+| ----- | ---- | ----- |
+| **BGE-M3** (BAAI) | Default multilingual dense (+ multi-granularity) | MIT; ~0.5B; self-host staple |
+| **Nomic Embed** text v1/v2 | Strong English/multilingual text | Apache |
+| **Nomic Embed Code** / code-specialized | Code retrieval | Larger; optional P1 |
+| **EmbeddingGemma-300M** | Tiny multilingual | Edge RAG |
+| **Qwen3 embedding / VL-small** | Multimodal retrieval experiments | Watch cards |
+
+Rerankers (BGE reranker, mini cross-encoders) are high leverage and small — good
+companions once embed base is reserved.
+
+---
+
+## 4. Axis C — Homelab training & domain specialization
+
+### 4.1 Realistic lab methods
+
+| Method | Homelab fit | Use |
+| ------ | ----------- | --- |
+| QLoRA / LoRA (Unsloth, PEFT, TRL) | Excellent | Domain assistants (security, ops, railiance) |
+| Preference (DPO/ORPO/KTO) | Good on 7–14B | Style and safety for agents |
+| Continued pretrain | Possible on small bases | Domain language (logs, policies) |
+| Full SFT of 70B+ | Poor without multi-GPU budget | Prefer distill/adapter on smaller |
+| From-scratch pretrain | Out of scope | — |
+
+### 4.2 Bases worth specializing
+
+1. **7–8B instruct** (Qwen3-8B or Llama-3.1-8B-Instruct class) — default FT target  
+2. **14B** — quality step if VRAM allows  
+3. **DeepSeek-R1-Distill 14B/32B** — reason-heavy agents after FT discipline  
+4. **Code-specialized mid** — if code assistants dominate workload  
+
+Data: prefer **synthetic + curated NetKingdom docs** over scraping; keep PII out
+of training sets (separate policy later).
+
+### 4.3 Tooling to track (not collect as weights)
+
+Unsloth, Axolotl, TRL, llama-factory, mergekit, preference datasets (UltraFeedback-class),
+eval harnesses (lm-eval, custom domain suites).
+
+---
+
+## 5. Axis D — Agent harnesses, orchestration, sandboxing, fleet
+
+### 5.1 Open harness landscape (2026)
+
+| Project | Shape | Why watch |
+| ------- | ----- | --------- |
+| **OpenHands** | Full agent platform + Canvas; MIT; multi-backend | Leading open coding-agent platform; multi-agent |
+| **Aider** | Git-native CLI pair programmer | Atomic commits; simple; great for ops scripts |
+| **SWE-agent / mini-SWE-agent** | Research harness, clean ACI | Benchmark + security CTF modes |
+| **OpenCode** | Open Claude Code alternative | High interest community |
+| **Cline** | IDE agent | Editor-centric workflows |
+| **Closed CLIs** | Claude Code, Codex, Gemini CLI | Quality bar for comparison |
+
+SWE-bench: harness + model co-determine scores; Mini-SWE-Agent claims strong
+results in very little code — architecture lessons for Coulomb harnesses.
+
+### 5.2 NetKingdom adjacency
+
+| Coulomb piece | Relation to D |
+| ------------- | ------------- |
+| **sand-boxer** | Where agents run safely |
+| **activity-core** | When work is due (this lab’s daily brief) |
+| **agent-harness / kaizen** | Who executes with profiles |
+| **state-hub** | What changed; task spawn completion |
+| **llm-connect** | Which model backend |
+
+Freedom Intelligence **reports** on harness tech; it does not replace sand-boxer.
+
+### 5.3 Sandboxing & fleet themes to track
+
+- Default-deny network + TTL sandboxes  
+- Credential routing (OpenBao / ops-warden) separate from model weights  
+- Multi-agent kanban / workplan-tied loops (Ralph-workplan class)  
+- Cost and rate-limit fleet management for API models  
+
+---
+
+## 6. Collection recommendations (open-weight reserve)
+
+### 6.1 Decision framework (recap)
+
+- Prefer **B/C** leverage + clear license + stable org provenance  
+- Size gates per `inventory/collection-policy.md`  
+- Backup path must be pinned (FI-WP-0001-T04) before large pulls  
+- Quants: prefer **official or well-known GGUF/AWQ** only after base id is cataloged; v1 recommends **safetensors official** first, quant as companion later  
+
+### 6.2 P0 — seed now (approve after storage pin)
+
+| # | Model (representative HF id) | Est. size class | Axes | Why |
+| - | ---------------------------- | --------------- | ---- | --- |
+| 1 | `Qwen/Qwen3-8B` (or current 8B-Instruct sibling on card) | ~16 GB fp16 / ~5 GB Q4 | B, C | Best default general + FT base; multilingual; tool-friendly |
+| 2 | `meta-llama/Llama-3.2-3B-Instruct` | ~6 GB fp16 / ~2 GB Q4 | B | Tiny always-on / edge; huge ecosystem |
+| 3 | `BAAI/bge-m3` | ~2 GB | B | Multilingual RAG staple; MIT |
+| 4 | `deepseek-ai/DeepSeek-R1-Distill-Qwen-14B` (or 8B if VRAM tight) | ~28 GB fp16 / ~9 GB Q4 | B, C | Local reasoning without full R1; MIT distill story |
+| 5 | `nomic-ai/nomic-embed-text-v1.5` (or current v2 text) | &lt;1 GB | B | Lightweight embed alternative / A/B with BGE |
+
+**Rough P0 cold storage (fp16 worst case):** ~50–60 GB.  
+**With Q4 preference for LLMs + fp for embeds:** often **~20–30 GB**. Prefer quants only when catalog notes the exact quant provenance.
+
+### 6.3 P1 — next wave (quota permitting)
+
+| # | Model | Why |
+| - | ----- | --- |
+| 6 | `Qwen/Qwen3-14B` (instruct) | Quality step for single-GPU chat/code |
+| 7 | `google/gemma-3-4b-it` or EmbeddingGemma-300M | Efficient Google line; embed edge |
+| 8 | `mistralai/Mistral-Small-3.x` instruct (current card) | EU-adjacent open option |
+| 9 | `deepseek-ai/DeepSeek-R1-Distill-Qwen-32B` | Stronger local reason if ≥24 GB VRAM class |
+| 10 | Code embed: `nomic-ai/nomic-embed-code` or Jina code embed | Code RAG for repos |
+| 11 | Small reranker: `BAAI/bge-reranker-v2-m3` | Cheap quality win for RAG |
+
+### 6.4 P2 — watch / large (candidate or docs only)
+
+| Item | Guidance |
+| ---- | -------- |
+| Full **DeepSeek-V3 / R1** MoE (~600B+ class, hundreds of GB even quantized) | **Do not auto-collect.** Separate capacity + multi-GPU decision |
+| **Llama 70B / Qwen 72B** class | Only if lab gains multi-GPU and clear offline need |
+| Every new quant repack on HF | Skip; track base official only |
+
+### 6.5 Explicit non-recommendations (for now)
+
+- Closed API-only weights  
+- Anonymous “uncensored” reuploads without provenance  
+- Duplicate quants of the same revision  
+- Giant pretraining corpora as “models”  
+
+### 6.6 Catalog seed
+
+YAML candidates for P0–P1 are under `inventory/catalog/` with
+`status: candidate` and `brief_refs` pointing at this survey. Approval and
+download are **FI-WP-0003**.
+
+---
+
+## 7. Implications for the first daily brief
+
+The first `briefs/YYYY/MM/YYYY-MM-DD.md` after this survey should:
+
+1. **Not** restate this whole document  
+2. Cover **only deltas** since 2026-07-24 (new releases, price changes, harness versions)  
+3. Revisit P0/P1 if a **new open model** clearly displaces a candidate  
+4. Note any **license or access shocks** (takedowns, geo blocks)  
+5. Optionally mark FI-WP-0001-T04 / FI-WP-0003 progress under Lab implications  
+
+Automation: activity-core def `fi-daily-research-brief` (disabled until FI-WP-0002
+resolver + dry-run). Manual playbook: `docs/daily-brief-playbook.md`.
+
+---
+
+## 8. Open questions for later briefs
+
+1. Exact **hardware envelope** measured VRAM on lab hosts?  
+2. Prefer **GGUF Q5_K_M** vs official safetensors for P0?  
+3. Is **German-first** multilingual quality a hard requirement (affects embed + instruct choice)?  
+4. Will local models serve **air-gapped NetKingdom** scenarios or only cost/privacy convenience?  
+5. Single **code model** vs general instruct for domain FT?
+
+---
+
+## 9. Source anchors (non-exhaustive)
+
+- DeepSeek R1 release / MIT notes — DeepSeek API docs & HF model cards  
+- Pricing roundups 2026 — IntuitionLabs, MorphLLM, provider pricing pages  
+- Embeddings 2026 comparisons — BentoML, Milvus/Zilliz, BGE-M3 cards  
+- Harness landscape — OpenHands, Aider, SWE-agent comparisons (LocalAI Master, vendor blogs)  
+- Open-weight commercial lists — community curations (verify each card)
+
+Re-verify any number before spend or multi-hundred-GB download.
+
+---
+
+*Baseline established 2026-07-24 for Freedom Intelligence. Daily briefs begin as deltas from here.*