Establish Freedom Intelligence lab foundation and baseline research.

Add INTENT/SCOPE, daily-brief playbook, activity-core definition (disabled), workplans FI-WP-0001..0003, baseline field survey with open-weight collection recommendations, and inventory catalog candidates for the model reserve.
2026-07-24 00:15:27 +02:00 · 2026-07-24 00:15:27 +02:00 · b5f911140b
commit b5f911140b
parent 1dc1e09517
28 changed files with 2651 additions and 1 deletions
--- a/docs/backup-storage-policy.md
+++ b/docs/backup-storage-policy.md
@ -0,0 +1,161 @@
+# Backup storage policy — open-weight model reserve
+
+**Status:** foundation draft (concrete path and budget **TBD**)  
+**Related:** `INTENT.md`, `SCOPE.md`, `inventory/collection-policy.md`  
+**Adjacent:** `disaster-control` BackupPolicy (platform backups — different concern)
+
+---
+
+## Purpose
+
+Define where Freedom Intelligence stores **open-weight model blobs** (and justified
+companions), how much capacity we allow, and how this reserve relates to other
+lab backup facilities.
+
+Git never stores weight tensors. Git stores:
+
+* inventory catalog metadata (`inventory/catalog/`)
+* policies and schemas
+* research briefs
+
+---
+
+## Storage class
+
+| Requirement | Policy |
+| ----------- | ------ |
+| **Class** | Backup / bulk durable storage — **not** hot cluster PVCs or app disks |
+| **Durability** | Same or better retention posture as other lab bulk assets; prefer media that survives workstation rebuilds |
+| **Performance** | Sequential read for restore/training pull is enough; low latency not required |
+| **Access** | Operator and approved lab hosts only; not a public mirror |
+| **Separation** | Do not co-mingle with age-encrypted operational backups (Forgejo dumps, k3s state) without a clear subdirectory and different lifecycle rules |
+
+Platform backup paths such as `/opt/backup/railiance/{infra,cluster}/` and
+`~/.cache/railiance/backups/` are **operational recovery** lanes. The model
+reserve may live on the **same physical facility** only if isolated by path and
+quota so large weight pulls cannot crowd out restore media.
+
+---
+
+## Target location (to pin)
+
+| Field | Value |
+| ----- | ----- |
+| **Host / facility** | `TBD` — operator pin (e.g. backup NAS, off-cluster bulk volume, dedicated disk) |
+| **Base path or bucket** | `TBD` — suggested shape: `…/freedom-intelligence/models/` |
+| **Layout under base** | See [On-disk layout](#on-disk-layout) |
+| **Mount on lab hosts** | `TBD` |
+| **Credentials** | `TBD` — if remote: OpenBao or existing backup credential lane; never commit secrets |
+
+Until pinned, **do not** bulk-download multi-GB models into this git workspace or
+into hot root filesystems.
+
+### Suggested path shape (non-binding)
+
+```text
+{BACKUP_ROOT}/freedom-intelligence/
+├── models/
+│   └── {org}__{name}/
+│       └── {revision}/
+│           ├── blobs/          # weight files, shards
+│           └── MANIFEST.txt    # optional local copy of hashes
+├── companions/                 # adapters, tokenizers only when separate
+└── staging/                    # incomplete downloads; not catalog-ready
+```
+
+Map each `{org}__{name}/{revision}` to an inventory catalog entry.
+
+---
+
+## On-disk layout
+
+| Path element | Rule |
+| ------------ | ---- |
+| `org__name` | Hugging Face-style id with `/` → `__` |
+| `revision` | Git commit SHA, tag, or release id used at download time |
+| `blobs/` | Actual files; prefer original names from source |
+| `staging/` | Incomplete transfers; purge or resume; never mark collected until complete + verified |
+
+---
+
+## Capacity budget
+
+| Parameter | Policy |
+| --------- | ------ |
+| **Soft quota** | `TBD` GiB/TiB — operator pin based on free backup capacity |
+| **Hard stop** | No new collection when soft quota exceeded unless operator raises budget |
+| **Per-pull threshold** | See `inventory/collection-policy.md` (size gates approval) |
+| **Growth review** | Revisit quota when catalog total exceeds 70% of soft quota |
+| **Eviction** | Prefer archive/delete lowest-priority, easily re-obtainable revisions first; record eviction in catalog history |
+
+---
+
+## Integrity and provenance
+
+For every completed collection:
+
+1. Record source URL and revision in the catalog entry.
+2. Store checksums (`sha256` of each blob or upstream manifest digest).
+3. Record download date (UTC) and downloader identity.
+4. Prefer official org releases over anonymous re-uploads.
+5. Keep license text or SPDX id in catalog; refuse unclear licenses.
+
+Verification command examples belong in tooling later; policy only requires that
+**catalog claims match on-disk checksums** before status `collected`.
+
+---
+
+## Retention
+
+| Class | Retention |
+| ----- | --------- |
+| **Strategic reserve** (high priority, hard to re-obtain) | Keep until explicit deprecation |
+| **Working set** (common bases for training experiments) | Keep while in active use + one superseded revision optional |
+| **Staging** | Max 14 days incomplete, then purge |
+| **Deprecated** | Metadata retained in catalog with status `evicted` or `superseded`; blobs may be deleted |
+
+---
+
+## Encryption and offsite
+
+| Topic | Policy |
+| ----- | ------ |
+| **At rest** | Follow host/facility default; extra age/GPG of multi-hundred-GB trees is optional and costly |
+| **In transit** | HTTPS or trusted lab network only |
+| **Offsite copy** | Optional later; not required for foundation. If added, coordinate with `disaster-control` so model reserve does not break operational backup SLAs |
+
+---
+
+## What must not live here
+
+* Closed weights or artifacts whose terms forbid offline retention
+* Secrets, API keys, customer data, or training corpora with personal data (domain datasets need their own policy)
+* Operational backups (databases, k3s state, Forgejo dumps)
+* Git LFS dumps of full model trees as a substitute for backup storage
+
+---
+
+## Operator checklist to pin TBD fields
+
+- [ ] Choose facility and base path
+- [ ] Confirm free capacity and set soft quota
+- [ ] Document mount/access for lab hosts
+- [ ] Confirm credential path (if any)
+- [ ] Create empty `models/` and `staging/` directories
+- [ ] Update this file: replace `TBD` rows with concrete values
+- [ ] Note pin date and operator in revision history below
+
+### Pin log
+
+| Date | Operator | Change |
+| ---- | -------- | ------ |
+| 2026-07-23 | foundation | Policy created; path and quota unpinned |
+
+---
+
+## Relationship to disaster-control
+
+`disaster-control` owns **platform resilience** (what to restore after loss).
+Freedom Intelligence owns **which open weights we choose to retain** and their
+inventory. Shared facilities are fine; shared lifecycle rules are not automatic —
+model reserve is large, slow-changing, and rarely needed for emergency restore.
--- a/docs/daily-brief-playbook.md
+++ b/docs/daily-brief-playbook.md
@ -0,0 +1,111 @@
+# Daily research brief — playbook
+
+**Audience:** operator or agent executing `target_repo: freedom-intelligence`  
+**Template:** `briefs/_template.md`  
+**Baseline memory:** `research/2026-07-24-baseline-field-survey.md`  
+**Sources:** `docs/sources-allowlist.md`
+
+---
+
+## Purpose
+
+Produce **one** curated brief per active day covering axes A–D when there is
+signal. Briefs are **deltas** from the baseline survey and prior briefs — not
+full resurveys.
+
+---
+
+## Idempotence
+
+| Rule | Detail |
+| ---- | ------ |
+| One brief per date | Path `briefs/YYYY/MM/YYYY-MM-DD.md` |
+| Calendar | Prefer **Europe/Berlin** date in front matter |
+| Skip empty days | Either omit file or write `*No material delta.*` in Headline deltas |
+| Automation due bit | Cleared only by State Hub event `fi_daily_brief` (below) |
+
+---
+
+## Manual procedure
+
+1. **Orient**
+   - Read last 1–2 briefs if any; skim baseline § executive snapshot if first week
+   - `git pull` on freedom-intelligence
+
+2. **Scan** (`docs/sources-allowlist.md`) — 30–60 minutes cap  
+   - Capture only items that change capability, cost, access, or operability
+
+3. **Draft**
+   ```bash
+   mkdir -p briefs/$(date +%Y/%m)
+   cp briefs/_template.md briefs/$(date +%Y/%m)/$(date +%Y-%m-%d).md
+   ```
+   - Fill front matter (`status: final` when done)
+   - Leave empty sections as `*(none)*`
+   - Cite primary URLs
+
+4. **Collection candidates**
+   - Use inventory field names; set priority
+   - If nominating: add or update `inventory/catalog/{id}.yaml` with
+     `status: candidate` and `collection.brief_refs`
+
+5. **Lab implications**
+   - Link open workplans (`FI-WP-*`) only when action is real
+
+6. **Commit** (operator policy) — brief + any catalog YAML
+
+7. **Completion evidence** (required for activity-core rhythm)
+
+   Post State Hub progress (HTTP or MCP), shape:
+
+   ```yaml
+   event_type: fi_daily_brief
+   author: <agent-or-operator>
+   summary: "FI daily brief YYYY-MM-DD"
+   detail:
+     repo: freedom-intelligence
+     date: "YYYY-MM-DD"
+     path: "briefs/YYYY/MM/YYYY-MM-DD.md"
+     collection_candidates: 0
+   ```
+
+   Workstation API often `http://127.0.0.1:8000`; remote tunnel ports per
+   Custodian docs. See `docs/state-hub-delivery.md`.
+
+---
+
+## Automation consumer
+
+When activity-core emits `activity_task_spawn` with
+`target_repo: freedom-intelligence` and labels including `research-brief`:
+
+1. Checkout this repo on the execution host  
+2. Run this playbook end-to-end  
+3. Prefer **llm-connect / OpenRouter** for any LLM drafting (Binky lesson: do
+   not install coding-agent CLIs on railiance01 solely for this)  
+4. Always post `fi_daily_brief` even if the brief is “no material delta”  
+5. Idempotency key: do not open a second brief file for the same `date`
+
+Activity definition: `activity-definitions/fi-daily-research-brief.md`  
+Workplan: `workplans/FI-WP-0002-activity-core-daily-research.md`
+
+---
+
+## Quality bar
+
+| Check | Pass |
+| ----- | ---- |
+| Signal | No filler paragraphs |
+| Sources | At least one primary link per non-empty axis row |
+| Honesty | Benchmarks marked directional; prices “verify before budget” |
+| Collection | License noted for any candidate |
+| Length | Prefer one screen to three; deep dives go under `research/` |
+
+---
+
+## First daily after baseline
+
+The baseline survey (2026-07-24) already covers the field. The **first daily**
+should only list changes **after** that document’s research window. If nothing
+moved, a short no-delta brief still completes the rhythm and posts
+`fi_daily_brief`.
--- a/docs/hardware-envelope.md
+++ b/docs/hardware-envelope.md
@ -0,0 +1,65 @@
+# Hardware envelope — homelab accessible (draft)
+
+**Status:** assumptions + TBD measurements  
+**Used by:** inventory `hardware_class`, axis B/C prioritization, collection size decisions
+
+---
+
+## Purpose
+
+Define what Freedom Intelligence means by **homelab accessible** so we do not
+collect models that only make sense on hyperscale clusters.
+
+---
+
+## Working tiers (until hosts are measured)
+
+| Tier | Assumed capacity | Target model class |
+| ---- | ---------------- | ------------------ |
+| **T0 — CPU / edge** | 16–64 GB system RAM, no GPU | ≤3B Q4; embeddings ≤0.5B |
+| **T1 — consumer GPU** | 8–12 GB VRAM | 7–8B Q4/Q5; 3B fp16 |
+| **T2 — enthusiast GPU** | 16–24 GB VRAM | 14B Q4; 8B fp16; light 32B Q4 |
+| **T3 — lab multi-GPU** | 2×24 GB+ or 48 GB+ | 32B fp16 / 70B Q4; small MoE |
+| **T4 — out of envelope** | multi-node / 8×A100-class | Full DeepSeek-V3 MoE — P2 watch only |
+
+P0 collection targets **T0–T2**. P2 full MoE is **T4**.
+
+---
+
+## Lab hosts (fill in)
+
+| Host | Role | GPU | VRAM | RAM | Notes |
+| ---- | ---- | --- | ---- | --- | ----- |
+| TBD | primary local inference | TBD | TBD | TBD | |
+| TBD | training experiments | TBD | TBD | TBD | |
+| railiance01 | cluster (not weight store) | n/a | n/a | n/a | Prefer not to fill hot disks with weights |
+
+---
+
+## Inference runtime defaults (intent)
+
+| Runtime | When |
+| ------- | ---- |
+| llama.cpp / GGUF | Desktop & edge |
+| vLLM | Server multi-request |
+| MLX | Apple silicon if present |
+| Ollama | Quick operator UX only |
+
+---
+
+## Training defaults (intent)
+
+| Method | Envelope |
+| ------ | -------- |
+| QLoRA 7–8B | T1–T2 |
+| QLoRA 14B | T2 |
+| Full FT 7B | T2–T3 |
+| 70B+ FT | T3+ only with explicit plan |
+
+---
+
+## Pin log
+
+| Date | Change |
+| ---- | ------ |
+| 2026-07-24 | Draft tiers; host table empty |
--- a/docs/sources-allowlist.md
+++ b/docs/sources-allowlist.md
@ -0,0 +1,62 @@
+# Sources allowlist — daily research
+
+Standing channels for Freedom Intelligence briefs (axes A–D). Prefer **primary**
+sources over aggregators. This is a watchlist, not a scrape mandate.
+
+---
+
+## Axis A — Frontier & commercial
+
+| Channel | What to extract |
+| ------- | --------------- |
+| OpenAI / Anthropic / Google / xAI / DeepSeek blogs & release notes | Capability claims, deprecations |
+| Official pricing pages | $/MTok in/out, cache, batch |
+| Model cards for API models | Context limits, tool use, policies |
+| LMSYS / Arena / artificialanalysis-class charts | Directional only; note gaming risk |
+| SWE-bench Verified / Live leaderboards | Harness+model pairs, not raw IQ |
+
+## Axis B — Edge / local / open
+
+| Channel | What to extract |
+| ------- | --------------- |
+| Hugging Face org feeds: `Qwen`, `deepseek-ai`, `meta-llama`, `google`, `mistralai`, `BAAI`, `nomic-ai` | New weights, licenses, sizes |
+| llama.cpp, vLLM, MLX, Ollama release notes | Runtime breakthroughs |
+| GGUF / official quant announcements | Only after base model known |
+
+## Axis C — Training & specialization
+
+| Channel | What to extract |
+| ------- | --------------- |
+| arXiv cs.LG, cs.CL, cs.AI (recent) | Methods that fit homelab |
+| Unsloth, Hugging Face TRL, Axolotl, LLaMA-Factory | Practical recipes |
+| Preference-optimization papers/releases | DPO/ORPO/RLVR-class |
+
+## Axis D — Harness & fleet
+
+| Channel | What to extract |
+| ------- | --------------- |
+| OpenHands, Aider, SWE-agent, OpenCode, Cline repos/releases | Harness capability |
+| sand-boxer / agent-harness / activity-core (internal) | Coulomb adjacency |
+| MCP / tool-protocol standards | Interop shifts |
+
+## Internal memory
+
+| Artifact | Role |
+| -------- | ---- |
+| `research/2026-07-24-baseline-field-survey.md` | Baseline; dailies are deltas |
+| Prior `briefs/YYYY/MM/*.md` | Avoid duplicate headlines |
+| `inventory/catalog/` | Collection status |
+
+## Do not
+
+- Treat Twitter/X rumor as confirmed without primary link  
+- Mirror entire HF orgs  
+- Cite paywalled claims without noting access limit  
+- Collect weights that fail the license gate (`inventory/collection-policy.md`)  
+- Store API keys or customer data in briefs  
+
+## Cadence hint
+
+For a 30–60 minute manual brief: scan A pricing+releases (10m), B HF/org
+releases (10m), C one methods thread (10m), D harness releases (10m), write
+(15m). Skip empty sections.
--- a/docs/state-hub-delivery.md
+++ b/docs/state-hub-delivery.md
@ -0,0 +1,62 @@
+# State Hub / operator delivery
+
+**Status:** contract defined; optional distribution not fully wired  
+**Related:** `docs/daily-brief-playbook.md`, `workplans/FI-WP-0002-activity-core-daily-research.md`
+
+---
+
+## Required: completion evidence for daily rhythm
+
+Every completed brief day (including “no material delta”) should post:
+
+```yaml
+event_type: fi_daily_brief
+author: <agent-or-operator>
+summary: "FI daily brief YYYY-MM-DD"
+detail:
+  repo: freedom-intelligence
+  date: "YYYY-MM-DD"
+  path: "briefs/YYYY/MM/YYYY-MM-DD.md"
+  collection_candidates: 0
+```
+
+This is the **idempotence signal** for activity-core context query
+`fi_brief_status` (FI-WP-0002-T03): `due=false` when an event exists for today
+(Europe/Berlin) with `detail.repo=freedom-intelligence`.
+
+Inbound automation uses `activity_task_spawn` from activity-core (state-hub sink).
+See `activity-core/docs/task-emission-consumer-contract.md`.
+
+---
+
+## Optional: operator visibility
+
+| Channel | Use |
+| ------- | --- |
+| **Progress event** (above) | Minimum — always for automation health |
+| **Inbox message** | Only when high-priority collection candidates appear |
+| **Wiki / Nextcloud** | Optional human mirror later |
+
+---
+
+## Non-goals
+
+* Freedom Intelligence is **not** required to be a full State Hub domain yet
+* Do not use `ISSUE_SINK_TYPE=rest` / Forgejo issues for this rhythm
+* Do not register workplans by hand in the hub (`fix-consistency` after commit)
+
+---
+
+## Fallback without MCP
+
+HTTP against State Hub API (workstation `http://127.0.0.1:8000` or remote
+tunnel ports). Check health before declaring hub down.
+
+---
+
+## Pin log
+
+| Date | Change |
+| ---- | ------ |
+| 2026-07-23 | Stub created |
+| 2026-07-24 | `fi_daily_brief` contract + activity-core linkage documented |