Register reserve under agents hub and pin VAULT storage.
Classify freedom-intelligence for State Hub (agents domain), sync FI-WP-0001..0003 with hub IDs, enrich catalog entries with profile/SWOT, and pin the open-weight reserve to D:\vault\coulomb\freedom-intelligence\ with strategic S-tier policy.
This commit is contained in:
parent
00f469662b
commit
a83ef0a79a
29 changed files with 1343 additions and 279 deletions
|
|
@ -1,48 +1,71 @@
|
|||
# Collection policy — open-weight reserve
|
||||
|
||||
**Status:** foundation
|
||||
**Related:** `schema.yaml`, `docs/backup-storage-policy.md`, `INTENT.md`
|
||||
**Status:** updated for 1 TB NAS + strategic capability reserve (2026-07-24)
|
||||
**Related:** `schema.yaml`, `docs/backup-storage-policy.md`,
|
||||
`docs/decisions/2026-07-24-nas-strategic-reserve.md`, `INTENT.md`
|
||||
|
||||
---
|
||||
|
||||
## Purpose
|
||||
|
||||
Decide **what** enters the open-weight reserve, **who** may approve it, and
|
||||
**when** a daily-brief candidate becomes a catalog entry with blobs on backup
|
||||
storage.
|
||||
**when** a daily-brief candidate becomes a catalog entry with blobs on the NAS.
|
||||
|
||||
---
|
||||
|
||||
## Goals
|
||||
|
||||
* Keep a **small, high-leverage** reserve — not a Hugging Face mirror
|
||||
* Hold a **capability-first strategic reserve** of the best open weights we may
|
||||
need later — **including models too large to run on current lab hardware**
|
||||
* Also hold a **runnable spine** for day-to-day local ops and fine-tunes
|
||||
* Enforce **license and integrity** before download completes
|
||||
* Match capacity to the backup storage soft quota
|
||||
* Prefer models that serve axes **B** and **C**, plus strategic **A** open releases
|
||||
* Stay within the **1 TB NAS soft quota (850 GiB)** — quality over mirror volume
|
||||
* Prefer official provenance; not a full Hugging Face scrape
|
||||
|
||||
---
|
||||
|
||||
## Two envelopes (do not conflate)
|
||||
|
||||
| Envelope | Question | Effect on collection |
|
||||
| -------- | -------- | -------------------- |
|
||||
| **Run** | Can we infer / FT this *now*? | Tags `hardware_class`, prioritizes R tier for local work |
|
||||
| **Reserve** | Is this among the best open artifacts worth keeping *just in case*? | **Not gated by current VRAM** |
|
||||
|
||||
**Runnability is not an eligibility gate.** An unrunnable SOTA open MoE can be
|
||||
priority **high** if license and quota allow.
|
||||
|
||||
---
|
||||
|
||||
## Eligibility (must pass all)
|
||||
|
||||
1. **Open weights** — weights obtainable under terms that allow offline retention for lab use
|
||||
2. **Clear license** — SPDX or linkable license text; `allows_offline_retention: true`
|
||||
3. **Stable provenance** — official org, tagged release, or commit revision (not anonymous drive-by reupload as sole source)
|
||||
4. **Lab rationale** — written `reason` tied to at least one axis A–D (usually B/C)
|
||||
5. **Capacity** — estimated size fits under remaining soft quota (see backup storage policy)
|
||||
1. **Open weights** — obtainable under terms that allow offline retention for lab use
|
||||
2. **Clear license** — SPDX or linkable license text; `allows_offline_retention: true`
|
||||
3. **Stable provenance** — official org, tagged release, or commit revision
|
||||
4. **Lab rationale** — written `reason` (capability SOTA, runnable spine, embed, FT base, …)
|
||||
5. **Capacity** — estimated size fits under remaining soft quota (850 GiB)
|
||||
|
||||
Fail any gate → status `rejected` with reason, or never enter catalog.
|
||||
Fail any gate → `rejected` or never enter catalog.
|
||||
|
||||
---
|
||||
|
||||
## Priority rubric
|
||||
## Collection tiers
|
||||
|
||||
| Tier | Code | Meaning |
|
||||
| ---- | ---- | ------- |
|
||||
| **Runnable spine** | **R** | Default local chat/code/embed/FT bases; keep resident |
|
||||
| **Strategic capability** | **S** | Most capable open models (often large); may be T4+ only to *run* |
|
||||
| **Watch / optional** | **W** | Secondary; collect only with clear headroom |
|
||||
|
||||
Catalog field: use `tags` including `tier-r` / `tier-s` / `tier-w` and
|
||||
`priority: high|medium|low`.
|
||||
|
||||
### Priority rubric
|
||||
|
||||
| Priority | Guidance |
|
||||
| -------- | -------- |
|
||||
| **high** | Rare or strategically important; license/access risk of disappearance; uniquely strong for B/C at our hardware class; hard to re-obtain |
|
||||
| **medium** | Clear lab use within 1–2 quarters; good quality/cost; easy enough to re-download but worth having cold |
|
||||
| **low** | Nice to have; only collect if quota headroom is large and pull is cheap |
|
||||
|
||||
Daily brief **collection candidates** should set a suggested priority; approval may change it.
|
||||
| **high** | Top open capability (S) or essential runnable spine (R); hard to re-obtain; license/access risk |
|
||||
| **medium** | Strong but not unique; mid-size upgrades; companions (rerankers) |
|
||||
| **low** | Nice-to-have; W tier |
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -50,16 +73,48 @@ Daily brief **collection candidates** should set a suggested priority; approval
|
|||
|
||||
| Estimated total size | Approval |
|
||||
| -------------------- | -------- |
|
||||
| **< 5 GiB** | Operator or lab agent may collect after license check; catalog entry required before or immediately after |
|
||||
| **5–40 GiB** | Explicit operator approval (chat, workplan task, or signed catalog `approved_by`) |
|
||||
| **> 40 GiB** | Operator approval **plus** check against soft quota and whether a smaller quant/variant suffices |
|
||||
| **Any size if quota ≥ 70% used** | Operator approval required regardless of size |
|
||||
| **Unclear license or ToS risk** | Do not collect; status `rejected` |
|
||||
| **< 5 GiB** | Operator or lab agent after license check |
|
||||
| **5–40 GiB** | Explicit operator approval |
|
||||
| **> 40 GiB** | Operator approval + remaining soft-quota check |
|
||||
| **> 200 GiB (typical S giants)** | Operator approval + written note on which other S models may need to wait |
|
||||
| **Any size if quota ≥ 70% used** | Operator approval required |
|
||||
| **Unclear license or ToS risk** | Do not collect; `rejected` |
|
||||
|
||||
“Operator” means the human lab owner (or a documented delegate). Agents may
|
||||
**nominate** (`status: candidate`) freely from briefs; they may **collect** only
|
||||
within the < 5 GiB band when licenses are unambiguous — otherwise stop at
|
||||
`candidate` / `approved`.
|
||||
Agents may **nominate** freely; they **collect** only in the < 5 GiB band with
|
||||
unambiguous licenses unless the operator has approved the catalog entry.
|
||||
|
||||
---
|
||||
|
||||
## What we prefer to collect
|
||||
|
||||
### Runnable spine (R)
|
||||
|
||||
* Small/mid instruct and code models that fit the run envelope
|
||||
* Strong embedding / rerank models for local RAG
|
||||
* Bases known to fine-tune well under QLoRA on lab GPUs
|
||||
|
||||
### Strategic capability (S)
|
||||
|
||||
* **Best available open general / reasoning / code weights** at the frontier of open
|
||||
* Large MoE or dense models even if current infra cannot serve them
|
||||
* Prefer official compressed releases (FP8, published quant) when full precision
|
||||
would exhaust the 1 TB NAS
|
||||
* One clear “best open” per capability niche is better than five near-duplicates
|
||||
|
||||
### Companions
|
||||
|
||||
Tokenizers, LoRA adapters, small eval fixtures when required to use a reserved base.
|
||||
|
||||
---
|
||||
|
||||
## What we usually skip
|
||||
|
||||
* Duplicate quants of the same revision already reserved
|
||||
* Anonymous merges/repacks without provenance
|
||||
* Closed weights
|
||||
* Entire org mirrors
|
||||
* Giant pretraining corpora (default out of band unless separately justified)
|
||||
* Anything whose license forbids offline retention
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -67,68 +122,30 @@ within the < 5 GiB band when licenses are unambiguous — otherwise stop at
|
|||
|
||||
```text
|
||||
brief nominates
|
||||
→ candidate (catalog YAML, no blobs required)
|
||||
→ approved (license + size + quota OK)
|
||||
→ collecting (download in staging/)
|
||||
→ collected (blobs complete, checksums recorded, storage_path set)
|
||||
→ verified (optional re-hash / smoke load)
|
||||
→ superseded|evicted (replaced or removed; metadata kept)
|
||||
→ candidate
|
||||
→ approved
|
||||
→ collecting (NAS staging/)
|
||||
→ collected
|
||||
→ verified
|
||||
→ superseded|evicted
|
||||
```
|
||||
|
||||
Rejected candidates stay in catalog only if useful as a decision record; otherwise omit.
|
||||
|
||||
---
|
||||
|
||||
## What we prefer to collect
|
||||
|
||||
* Small/mid instruct and code models that fit the hardware envelope
|
||||
* Strong embedding / rerank models for local RAG
|
||||
* Base models known to fine-tune well under QLoRA/LoRA on lab GPUs
|
||||
* Official quant releases when they are the supported distribution
|
||||
* Adapters and tokenizers that unlock a reserved base (as companions)
|
||||
|
||||
## What we usually skip
|
||||
|
||||
* Duplicate quants of the same revision already reserved
|
||||
* Huge models with no near-term local run/train path and no access-risk story
|
||||
* Merges/repacks without provenance
|
||||
* Datasets larger than model weights unless separately justified (default: out of band)
|
||||
* Anything requiring acceptance flows we cannot satisfy offline
|
||||
|
||||
---
|
||||
|
||||
## Companions
|
||||
|
||||
Tokenizers, LoRA adapters, and small eval fixtures may be collected when:
|
||||
|
||||
* they are required to use a reserved base, or
|
||||
* they are small (< 1 GiB) and high leverage
|
||||
|
||||
Link via `companions` in the catalog schema.
|
||||
|
||||
---
|
||||
|
||||
## Brief integration
|
||||
|
||||
1. Brief section **Collection candidates** nominates items.
|
||||
2. Operator/agent opens `inventory/catalog/{id}.yaml` with `status: candidate`.
|
||||
3. Approval and download follow this policy and `docs/backup-storage-policy.md`.
|
||||
4. Brief `brief_refs` on the entry point back to the nominating day(s).
|
||||
1. Brief **Collection candidates** nominates R/S/W.
|
||||
2. Catalog YAML under `inventory/catalog/`.
|
||||
3. Download only after storage path is pinned (`docs/backup-storage-policy.md`) and approval rules pass.
|
||||
4. `collection.brief_refs` / research refs for provenance of the nomination.
|
||||
|
||||
---
|
||||
|
||||
## Eviction rule of thumb
|
||||
|
||||
When over quota or cleaning:
|
||||
When over soft quota:
|
||||
|
||||
1. `low` priority, easily re-obtainable from still-live official URLs
|
||||
2. Superseded revisions with a newer `verified` replacement
|
||||
3. Never silent-delete: set `status: evicted`, clear or note `storage_path`, append `history`
|
||||
|
||||
---
|
||||
|
||||
## Non-goals
|
||||
|
||||
* Automatic bulk mirrors of entire orgs
|
||||
* Collecting on every brief mention without priority
|
||||
* Bypassing license gates for “research only” convenience
|
||||
1. **W** tier and easily re-obtainable duplicates
|
||||
2. Superseded revisions with a stronger verified successor
|
||||
3. Never silent-delete **S** SOTA or sole **R** spine without operator note
|
||||
4. Always set `status: evicted` and append `history`
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue