tegwick
b5f911140b
Add INTENT/SCOPE, daily-brief playbook, activity-core definition (disabled), workplans FI-WP-0001..0003, baseline field survey with open-weight collection recommendations, and inventory catalog candidates for the model reserve.
161 lines
6.1 KiB
Markdown
161 lines
6.1 KiB
Markdown
# Backup storage policy — open-weight model reserve
|
|
|
|
**Status:** foundation draft (concrete path and budget **TBD**)
|
|
**Related:** `INTENT.md`, `SCOPE.md`, `inventory/collection-policy.md`
|
|
**Adjacent:** `disaster-control` BackupPolicy (platform backups — different concern)
|
|
|
|
---
|
|
|
|
## Purpose
|
|
|
|
Define where Freedom Intelligence stores **open-weight model blobs** (and justified
|
|
companions), how much capacity we allow, and how this reserve relates to other
|
|
lab backup facilities.
|
|
|
|
Git never stores weight tensors. Git stores:
|
|
|
|
* inventory catalog metadata (`inventory/catalog/`)
|
|
* policies and schemas
|
|
* research briefs
|
|
|
|
---
|
|
|
|
## Storage class
|
|
|
|
| Requirement | Policy |
|
|
| ----------- | ------ |
|
|
| **Class** | Backup / bulk durable storage — **not** hot cluster PVCs or app disks |
|
|
| **Durability** | Same or better retention posture as other lab bulk assets; prefer media that survives workstation rebuilds |
|
|
| **Performance** | Sequential read for restore/training pull is enough; low latency not required |
|
|
| **Access** | Operator and approved lab hosts only; not a public mirror |
|
|
| **Separation** | Do not co-mingle with age-encrypted operational backups (Forgejo dumps, k3s state) without a clear subdirectory and different lifecycle rules |
|
|
|
|
Platform backup paths such as `/opt/backup/railiance/{infra,cluster}/` and
|
|
`~/.cache/railiance/backups/` are **operational recovery** lanes. The model
|
|
reserve may live on the **same physical facility** only if isolated by path and
|
|
quota so large weight pulls cannot crowd out restore media.
|
|
|
|
---
|
|
|
|
## Target location (to pin)
|
|
|
|
| Field | Value |
|
|
| ----- | ----- |
|
|
| **Host / facility** | `TBD` — operator pin (e.g. backup NAS, off-cluster bulk volume, dedicated disk) |
|
|
| **Base path or bucket** | `TBD` — suggested shape: `…/freedom-intelligence/models/` |
|
|
| **Layout under base** | See [On-disk layout](#on-disk-layout) |
|
|
| **Mount on lab hosts** | `TBD` |
|
|
| **Credentials** | `TBD` — if remote: OpenBao or existing backup credential lane; never commit secrets |
|
|
|
|
Until pinned, **do not** bulk-download multi-GB models into this git workspace or
|
|
into hot root filesystems.
|
|
|
|
### Suggested path shape (non-binding)
|
|
|
|
```text
|
|
{BACKUP_ROOT}/freedom-intelligence/
|
|
├── models/
|
|
│ └── {org}__{name}/
|
|
│ └── {revision}/
|
|
│ ├── blobs/ # weight files, shards
|
|
│ └── MANIFEST.txt # optional local copy of hashes
|
|
├── companions/ # adapters, tokenizers only when separate
|
|
└── staging/ # incomplete downloads; not catalog-ready
|
|
```
|
|
|
|
Map each `{org}__{name}/{revision}` to an inventory catalog entry.
|
|
|
|
---
|
|
|
|
## On-disk layout
|
|
|
|
| Path element | Rule |
|
|
| ------------ | ---- |
|
|
| `org__name` | Hugging Face-style id with `/` → `__` |
|
|
| `revision` | Git commit SHA, tag, or release id used at download time |
|
|
| `blobs/` | Actual files; prefer original names from source |
|
|
| `staging/` | Incomplete transfers; purge or resume; never mark collected until complete + verified |
|
|
|
|
---
|
|
|
|
## Capacity budget
|
|
|
|
| Parameter | Policy |
|
|
| --------- | ------ |
|
|
| **Soft quota** | `TBD` GiB/TiB — operator pin based on free backup capacity |
|
|
| **Hard stop** | No new collection when soft quota exceeded unless operator raises budget |
|
|
| **Per-pull threshold** | See `inventory/collection-policy.md` (size gates approval) |
|
|
| **Growth review** | Revisit quota when catalog total exceeds 70% of soft quota |
|
|
| **Eviction** | Prefer archive/delete lowest-priority, easily re-obtainable revisions first; record eviction in catalog history |
|
|
|
|
---
|
|
|
|
## Integrity and provenance
|
|
|
|
For every completed collection:
|
|
|
|
1. Record source URL and revision in the catalog entry.
|
|
2. Store checksums (`sha256` of each blob or upstream manifest digest).
|
|
3. Record download date (UTC) and downloader identity.
|
|
4. Prefer official org releases over anonymous re-uploads.
|
|
5. Keep license text or SPDX id in catalog; refuse unclear licenses.
|
|
|
|
Verification command examples belong in tooling later; policy only requires that
|
|
**catalog claims match on-disk checksums** before status `collected`.
|
|
|
|
---
|
|
|
|
## Retention
|
|
|
|
| Class | Retention |
|
|
| ----- | --------- |
|
|
| **Strategic reserve** (high priority, hard to re-obtain) | Keep until explicit deprecation |
|
|
| **Working set** (common bases for training experiments) | Keep while in active use + one superseded revision optional |
|
|
| **Staging** | Max 14 days incomplete, then purge |
|
|
| **Deprecated** | Metadata retained in catalog with status `evicted` or `superseded`; blobs may be deleted |
|
|
|
|
---
|
|
|
|
## Encryption and offsite
|
|
|
|
| Topic | Policy |
|
|
| ----- | ------ |
|
|
| **At rest** | Follow host/facility default; extra age/GPG of multi-hundred-GB trees is optional and costly |
|
|
| **In transit** | HTTPS or trusted lab network only |
|
|
| **Offsite copy** | Optional later; not required for foundation. If added, coordinate with `disaster-control` so model reserve does not break operational backup SLAs |
|
|
|
|
---
|
|
|
|
## What must not live here
|
|
|
|
* Closed weights or artifacts whose terms forbid offline retention
|
|
* Secrets, API keys, customer data, or training corpora with personal data (domain datasets need their own policy)
|
|
* Operational backups (databases, k3s state, Forgejo dumps)
|
|
* Git LFS dumps of full model trees as a substitute for backup storage
|
|
|
|
---
|
|
|
|
## Operator checklist to pin TBD fields
|
|
|
|
- [ ] Choose facility and base path
|
|
- [ ] Confirm free capacity and set soft quota
|
|
- [ ] Document mount/access for lab hosts
|
|
- [ ] Confirm credential path (if any)
|
|
- [ ] Create empty `models/` and `staging/` directories
|
|
- [ ] Update this file: replace `TBD` rows with concrete values
|
|
- [ ] Note pin date and operator in revision history below
|
|
|
|
### Pin log
|
|
|
|
| Date | Operator | Change |
|
|
| ---- | -------- | ------ |
|
|
| 2026-07-23 | foundation | Policy created; path and quota unpinned |
|
|
|
|
---
|
|
|
|
## Relationship to disaster-control
|
|
|
|
`disaster-control` owns **platform resilience** (what to restore after loss).
|
|
Freedom Intelligence owns **which open weights we choose to retain** and their
|
|
inventory. Shared facilities are fine; shared lifecycle rules are not automatic —
|
|
model reserve is large, slow-changing, and rarely needed for emergency restore.
|