coulomb/freedom-intelligence
4
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Register reserve under agents hub and pin VAULT storage.

Browse source 
Classify freedom-intelligence for State Hub (agents domain), sync FI-WP-0001..0003 with hub IDs, enrich catalog entries with profile/SWOT, and pin the open-weight reserve to D:\vault\coulomb\freedom-intelligence\ with strategic S-tier policy.
This commit is contained in:
tegwick 2026-07-28 00:25:21 +02:00
parent 00f469662b
commit a83ef0a79a
29 changed files with 1343 additions and 279 deletions
Download patch file Download diff file Expand all files Collapse all files

119
docs/backup-storage-policy.md
View file

@ -1,7 +1,8 @@
# Backup storage policy — open-weight model reserve
**Status:** foundation draft (concrete path and budget **TBD**)
**Related:** `INTENT.md`, `SCOPE.md`, `inventory/collection-policy.md`
**Status:** facility + path pinned (workstation VAULT HD)
**Related:** `INTENT.md`, `SCOPE.md`, `inventory/collection-policy.md`,
`docs/decisions/2026-07-24-nas-strategic-reserve.md`
**Adjacent:** `disaster-control` BackupPolicy (platform backups — different concern)
---
@ -25,35 +26,42 @@ Git never stores weight tensors. Git stores:
| Requirement | Policy |
| ----------- | ------ |
| **Class** | Backup / bulk durable storage — **not** hot cluster PVCs or app disks |
| **Durability** | Same or better retention posture as other lab bulk assets; prefer media that survives workstation rebuilds |
| **Facility** | Workstation VAULT HD (`D:`) under `D:\vault\coulomb\` (interim pin 2026-07-28) |
| **Durability** | Survives WSL rebuilds; weights live on the VAULT volume, not the git/WSL root |
| **Performance** | Sequential read for restore/training pull is enough; low latency not required |
| **Access** | Operator and approved lab hosts only; not a public mirror |
| **Separation** | Do not co-mingle with age-encrypted operational backups (Forgejo dumps, k3s state) without a clear subdirectory and different lifecycle rules |
Platform backup paths such as `/opt/backup/railiance/{infra,cluster}/` and
`~/.cache/railiance/backups/` are **operational recovery** lanes. The model
reserve may live on the **same physical facility** only if isolated by path and
quota so large weight pulls cannot crowd out restore media.
`~/.cache/railiance/backups/` are **operational recovery** lanes. Model weights
belong on the **VAULT model tree**, not on railiance hot disks.
---
## Target location (to pin)
## Target location
| Field | Value |
| ----- | ----- |
| **Host / facility** | `TBD` — operator pin (e.g. backup NAS, off-cluster bulk volume, dedicated disk) |
| **Base path or bucket** | `TBD` — suggested shape: `…/freedom-intelligence/models/` |
| **Host / facility** | Workstation-attached bulk HD **VAULT** (Windows `D:`) |
| **Raw capacity** | ~1.86 TB total (~1.76 TB free as of 2026-07-28 pin) |
| **Base path (Windows)** | `D:\vault\coulomb\freedom-intelligence\` |
| **Base path (WSL, when D: automounted)** | `/mnt/d/vault/coulomb/freedom-intelligence/` |
| **Layout under base** | See [On-disk layout](#on-disk-layout) |
| **Mount on lab hosts** | `TBD` |
| **Credentials** | `TBD` — if remote: OpenBao or existing backup credential lane; never commit secrets |
| **Mount on lab hosts** | Local attach on operator workstation; WSL via drvfs `D:` → `/mnt/d` when enabled |
| **Credentials** | Local filesystem ACL on the workstation HD; never commit secrets |
Until pinned, **do not** bulk-download multi-GB models into this git workspace or
into hot root filesystems.
**Interim note:** Earlier policy assumed a dedicated ~1 TB NAS. Operator chose this
VAULT volume under `D:\vault\coulomb\` for now. Soft quota **850 GiB** for the
model tree still applies (self-imposed discipline; disk is larger).
### Suggested path shape (non-binding)
Do **not** bulk-download multi-GB models into this git workspace or into hot
root filesystems (`/` on WSL, `C:`).
### Path shape (pinned)
```text
{BACKUP_ROOT}/freedom-intelligence/
D:\vault\coulomb\freedom-intelligence\ # Windows
/mnt/d/vault/coulomb/freedom-intelligence/ # WSL when D: mounted
├── models/
│ └── {org}__{name}/
│ └── {revision}/
@ -65,6 +73,8 @@ into hot root filesystems.
Map each `{org}__{name}/{revision}` to an inventory catalog entry.
Directories `models/`, `staging/`, and `companions/` were created 2026-07-28.
---
## On-disk layout
@ -76,17 +86,33 @@ Map each `{org}__{name}/{revision}` to an inventory catalog entry.
| `blobs/` | Actual files; prefer original names from source |
| `staging/` | Incomplete transfers; purge or resume; never mark collected until complete + verified |
Optional layout tags (directory name or catalog field):
| Tag | Meaning |
| --- | ------- |
| runnable | Fits current run envelope (R tier) |
| strategic | Capability reserve; may exceed current hardware (S tier) |
---
## Capacity budget
| Parameter | Policy |
| --------- | ------ |
| **Soft quota** | `TBD` GiB/TiB — operator pin based on free backup capacity |
| **Hard stop** | No new collection when soft quota exceeded unless operator raises budget |
| **Per-pull threshold** | See `inventory/collection-policy.md` (size gates approval) |
| **Growth review** | Revisit quota when catalog total exceeds 70% of soft quota |
| **Eviction** | Prefer archive/delete lowest-priority, easily re-obtainable revisions first; record eviction in catalog history |
| **Device** | VAULT HD (`D:`) — model tree under `D:\vault\coulomb\freedom-intelligence\` |
| **Soft quota (model reserve)** | **850 GiB** (self-imposed; disk is larger) |
| **Hard stop** | **920 GiB** used under the freedom-intelligence base path |
| **Growth review** | When catalog total ≥ **70% soft quota (~595 GiB)** |
| **Per-pull threshold** | See `inventory/collection-policy.md` |
| **Eviction** | Prefer drop **W** (watch) and easily re-obtained quants; protect unique **S** SOTA bases and the **R** spine |
### Portfolio guidance on 1 TB
* Always keep the **runnable spine (R / former P0)** resident.
* Use remaining space for a **small number of most-capable open models (S)**,
preferably well-provenance compressed weights when full precision would exhaust the disk.
* Do **not** attempt to mirror entire HF orgs.
* Full bf16 of multiple 600B-class models will **not** fit; prioritize top capability identities.
---
@ -100,8 +126,7 @@ For every completed collection:
4. Prefer official org releases over anonymous re-uploads.
5. Keep license text or SPDX id in catalog; refuse unclear licenses.
Verification command examples belong in tooling later; policy only requires that
**catalog claims match on-disk checksums** before status `collected`.
Verification: **catalog claims must match on-disk checksums** before status `collected`.
---
@ -109,10 +134,11 @@ Verification command examples belong in tooling later; policy only requires that
| Class | Retention |
| ----- | --------- |
| **Strategic reserve** (high priority, hard to re-obtain) | Keep until explicit deprecation |
| **Working set** (common bases for training experiments) | Keep while in active use + one superseded revision optional |
| **Strategic capability (S)** | Keep until explicit deprecation or displacement by a clearly stronger open successor |
| **Runnable spine (R)** | Keep while still the lab default for local ops / FT |
| **Working set** | Active experiment bases + optional one superseded revision |
| **Staging** | Max 14 days incomplete, then purge |
| **Deprecated** | Metadata retained in catalog with status `evicted` or `superseded`; blobs may be deleted |
| **Deprecated / evicted** | Metadata retained in catalog; blobs may be deleted |
---
@ -120,42 +146,35 @@ Verification command examples belong in tooling later; policy only requires that
| Topic | Policy |
| ----- | ------ |
| **At rest** | Follow host/facility default; extra age/GPG of multi-hundred-GB trees is optional and costly |
| **In transit** | HTTPS or trusted lab network only |
| **Offsite copy** | Optional later; not required for foundation. If added, coordinate with `disaster-control` so model reserve does not break operational backup SLAs |
| **At rest** | NAS default; extra age/GPG of multi-hundred-GB trees optional |
| **In transit** | Trusted lab network / local attach |
| **Offsite copy** | Optional later; coordinate with disaster-control if added so model bulk does not break ops backup SLAs |
---
## What must not live here
* Closed weights or artifacts whose terms forbid offline retention
* Secrets, API keys, customer data, or training corpora with personal data (domain datasets need their own policy)
* Operational backups (databases, k3s state, Forgejo dumps)
* Git LFS dumps of full model trees as a substitute for backup storage
* Closed weights or artifacts whose terms forbid offline retention
* Secrets, API keys, customer data, or ungoverned training corpora with personal data
* Operational backups (databases, k3s state, Forgejo dumps) as the primary home
* Git LFS dumps of full model trees as a substitute for the NAS
---
## Operator checklist to pin TBD fields
## Operator checklist
- [ ] Choose facility and base path
- [ ] Confirm free capacity and set soft quota
- [ ] Document mount/access for lab hosts
- [ ] Confirm credential path (if any)
- [ ] Create empty `models/` and `staging/` directories
- [ ] Update this file: replace `TBD` rows with concrete values
- [ ] Note pin date and operator in revision history below
- [x] Choose facility: bulk durable storage for model reserve (2026-07-24 intent)
- [x] Set soft quota: **850 GiB** / hard stop **920 GiB**
- [x] Pin path: **`D:\vault\coulomb\freedom-intelligence\`** (2026-07-28)
- [x] Create `models/`, `staging/`, `companions/`
- [x] Document Windows + WSL path forms above
- [x] Ensure WSL can see `D:` (`/mnt/d`) when downloads run from Linux (verified 2026-07-28)
### Pin log
| Date | Operator | Change |
| ---- | -------- | ------ |
| 2026-07-23 | foundation | Policy created; path and quota unpinned |
---
## Relationship to disaster-control
`disaster-control` owns **platform resilience** (what to restore after loss).
Freedom Intelligence owns **which open weights we choose to retain** and their
inventory. Shared facilities are fine; shared lifecycle rules are not automatic —
model reserve is large, slow-changing, and rarely needed for emergency restore.
| 2026-07-24 | foundation | Policy created; path unpinned |
| 2026-07-24 | operator direction | Facility intent = 1 TB local NAS; soft quota 850 GiB; strategic unrunnable models in scope |
| 2026-07-28 | operator direction | **Interim pin:** workstation VAULT HD `D:\vault\coulomb\freedom-intelligence\`; layout created; soft quota retained |
| 2026-07-28 | operator | WSL mount verified: `D:` → `/mnt/d` (1.9T, writable) |

84
docs/decisions/2026-07-24-nas-strategic-reserve.md Normal file
View file

@ -0,0 +1,84 @@
# Decision: 1TB local NAS + strategic capability reserve
**Date:** 2026-07-24
**Status:** accepted
**Affects:** `docs/backup-storage-policy.md`, `inventory/collection-policy.md`,
`docs/hardware-envelope.md`, collection tiers, FI-WP-0001 / 0003
---
## Decision
1. **Storage backend** for the open-weight reserve is a **1 TB minimal local NAS**
attached to the lab (operator facility). Weights live there; git keeps catalog only.
2. **Reserve intent** is explicitly **not limited to currently runnable models**.
We capture the **most capable open-weight models** that license allows us to
retain, even when they exceed today’s inference/training hardware (T4 and beyond).
3. **Runnability** and **reservability** are separate axes:
- *Run envelope* → what we can serve or fine-tune now (T0–T3)
- *Reserve envelope* → what we store for future ops, training facility, or access loss
---
## Capacity reality check (1 TB)
| Use | Guidance |
| --- | -------- |
| Raw device | ~1 TB |
| Soft quota for model reserve | **850 GiB** (leave ~15% for FS overhead, staging, other lab bulk) |
| Hard stop | **920 GiB** used on the NAS model tree — stop new pulls |
| Growth review | At **70% of soft quota (~595 GiB)** re-prioritize; prefer drop easily re-obtained quants over unique SOTA bases |
Full-precision 600B-class MoE weights can be **multi-hundreds of GiB to >1 TB each**.
On a 1 TB NAS we therefore:
* Prefer **official or well-provenance compressed** distributions (FP8, INT4/GGUF,
published quant) when the card supports them for the *strategic* tier
* Still catalog the **base model id + revision** as the capability identity
* Accept that only a **small number of top-tier giants** fit at once — quality over quantity
* Keep the **runnable spine** (P0) always resident so local work does not depend on giants
Rough fit examples (order-of-magnitude; re-measure at download):
| Portfolio sketch | Est. total |
| ---------------- | ---------- |
| P0 quant spine only | ~25–55 GiB |
| P0 + mid P1 | ~80–150 GiB |
| P0 + **one** SOTA open MoE quant (~300–400 GiB) | ~350–450 GiB |
| P0 + two large MoE quants | can approach or exceed soft quota — choose carefully |
---
## Collection tiering (updated)
| Tier | Name | Rule |
| ---- | ---- | ---- |
| **R** | Runnable spine | Fits current/near hardware; always keep |
| **S** | Strategic capability | Best open SOTA / near-SOTA; **may be unrunnable today**; high retention |
| **W** | Watch / optional | Nice-to-have; only if soft quota headroom |
Supersedes the earlier “do not collect full DeepSeek MoE without a separate
decision” posture: **strategic collection of top open giants is in scope** for
this NAS, subject to license + soft quota.
---
## Non-goals (unchanged)
* Public model CDN or Hugging Face mirror
* Collecting closed weights
* Guaranteeing inference of every reserved model on current hosts
---
## Follow-ups
- [x] Operator: pin storage path and create `freedom-intelligence/{models,staging,companions}/`
- **2026-07-28 interim:** `D:\vault\coulomb\freedom-intelligence\` on workstation VAULT HD (~1.86 TB)
- Soft quota **850 GiB** retained (self-imposed; disk larger than original 1 TB sketch)
- [x] Update pin log in `docs/backup-storage-policy.md` with exact mount path
- [x] Seed/update catalog entries for strategic tier (S)
- [ ] FI-WP-0003: prioritize R spine + top S giants under 850 GiB soft quota
- [x] WSL: `D:` available as `/mnt/d` (verified 2026-07-28)

57
docs/hardware-envelope.md
View file

@ -1,28 +1,47 @@
# Hardware envelope — homelab accessible (draft)
# Hardware envelope — run vs reserve
**Status:** assumptions + TBD measurements
**Used by:** inventory `hardware_class`, axis B/C prioritization, collection size decisions
**Status:** updated 2026-07-24 (1 TB NAS strategic reserve)
**Used by:** inventory `hardware_class`, axis B/C, collection tiers R/S
---
## Purpose
Define what Freedom Intelligence means by **homelab accessible** so we do not
collect models that only make sense on hyperscale clusters.
Separate two questions that used to be conflated:
1. **What can we run or fine-tune now?** → *run envelope*
2. **What should we store on the NAS just in case?** → *reserve envelope* (1 TB)
Collecting a model **does not** require it to fit the run envelope.
See `docs/decisions/2026-07-24-nas-strategic-reserve.md`.
---
## Working tiers (until hosts are measured)
## Run envelope (inference / training *today*)
| Tier | Assumed capacity | Target model class |
| ---- | ---------------- | ------------------ |
| Tier | Assumed capacity | Typical runnable class |
| ---- | ---------------- | ---------------------- |
| **T0 — CPU / edge** | 16–64 GB system RAM, no GPU | ≤3B Q4; embeddings ≤0.5B |
| **T1 — consumer GPU** | 8–12 GB VRAM | 7–8B Q4/Q5; 3B fp16 |
| **T2 — enthusiast GPU** | 16–24 GB VRAM | 14B Q4; 8B fp16; light 32B Q4 |
| **T3 — lab multi-GPU** | 2×24 GB+ or 48 GB+ | 32B fp16 / 70B Q4; small MoE |
| **T4 — out of envelope** | multi-node / 8×A100-class | Full DeepSeek-V3 MoE — P2 watch only |
| **T4 — beyond current lab** | multi-node / datacenter class | Full large MoE / 70B+ fp16 dense |
P0 collection targets **T0–T2**. P2 full MoE is **T4**.
**Runnable spine (tier R)** targets **T0–T2** (stretch T3 when hardware exists).
**Strategic reserve (tier S)** may be **T3–T4 or above** for *execution* — still
valid for NAS collection.
---
## Reserve envelope (NAS)
| Parameter | Value |
| --------- | ----- |
| Facility | VAULT HD `D:\vault\coulomb\freedom-intelligence\` (soft quota 850 GiB) |
| Soft quota | 850 GiB |
| Hard stop | 920 GiB |
| Policy | `docs/backup-storage-policy.md` |
---
@ -30,13 +49,14 @@ P0 collection targets **T0–T2**. P2 full MoE is **T4**.
| Host | Role | GPU | VRAM | RAM | Notes |
| ---- | ---- | --- | ---- | --- | ----- |
| TBD | primary local inference | TBD | TBD | TBD | |
| TBD | training experiments | TBD | TBD | TBD | |
| railiance01 | cluster (not weight store) | n/a | n/a | n/a | Prefer not to fill hot disks with weights |
| TBD | primary local inference | TBD | TBD | TBD | Run envelope |
| TBD | training experiments | TBD | TBD | TBD | Run envelope |
| **VAULT HD (D:)** | weight store | n/a | n/a | n/a | **Reserve envelope** — soft 850 GiB |
| railiance01 | cluster | n/a | n/a | n/a | Do not store weight trees on hot disks |
---
## Inference runtime defaults (intent)
## Inference runtime defaults (when runnable)
| Runtime | When |
| ------- | ---- |
@ -45,16 +65,18 @@ P0 collection targets **T0–T2**. P2 full MoE is **T4**.
| MLX | Apple silicon if present |
| Ollama | Quick operator UX only |
Strategic giants may remain **cold storage only** until hardware upgrades.
---
## Training defaults (intent)
## Training defaults (when runnable)
| Method | Envelope |
| ------ | -------- |
| QLoRA 7–8B | T1–T2 |
| QLoRA 14B | T2 |
| Full FT 7B | T2–T3 |
| 70B+ FT | T3+ only with explicit plan |
| 70B+ / large MoE FT | Future facility — weights may already sit on NAS |
---
@ -62,4 +84,5 @@ P0 collection targets **T0–T2**. P2 full MoE is **T4**.
| Date | Change |
| ---- | ------ |
| 2026-07-24 | Draft tiers; host table empty |
| 2026-07-24 | Draft tiers |
| 2026-07-24 | Split run vs reserve; NAS 1 TB; S tier may exceed run envelope |

12
docs/state-hub-delivery.md
View file

@ -3,6 +3,18 @@
**Status:** contract defined; optional distribution not fully wired
**Related:** `docs/daily-brief-playbook.md`, `workplans/FI-WP-0002-activity-core-daily-research.md`
## Repo registration
| Field | Value |
| ----- | ----- |
| Hub domain | **agents** |
| Repo slug | `freedom-intelligence` |
| Classification | `.repo-classification.yaml` (primary `agents`, secondary `infotech`) |
| Sync | `statehub fix-consistency --repo freedom-intelligence` after workplan edits |
Workplans live in git; the hub indexes them after registration + fix-consistency.
Do not create workplans via hub `create_workplan` APIs.
---
## Required: completion evidence for daily rhythm