2026-07-24 00:15:27 +02:00
|
|
|
# Backup storage policy — open-weight model reserve
|
|
|
|
|
|
2026-09-13 22:45:43 +02:00
|
|
|
**Status:** facility pin **Scaleway Object Storage** (2026-09-13)
|
2026-07-28 00:25:21 +02:00
|
|
|
**Related:** `INTENT.md`, `SCOPE.md`, `inventory/collection-policy.md`,
|
2026-09-13 22:45:43 +02:00
|
|
|
`docs/decisions/2026-09-13-scaleway-object-reserve.md`
|
|
|
|
|
**Supersedes storage pin:** 2026-07-28 VAULT HD (now staging only)
|
|
|
|
|
**Adjacent:** `disaster-control` BackupPolicy and `rapp-postgres` WAL
|
|
|
|
|
archive — **different buckets, different lifecycles**
|
2026-07-24 00:15:27 +02:00
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## Purpose
|
|
|
|
|
|
2026-09-13 22:45:43 +02:00
|
|
|
Define where Freedom Intelligence stores **open-weight model blobs** (and
|
|
|
|
|
justified companions), how much we allow ourselves to spend, and how this
|
|
|
|
|
reserve relates to other lab backup facilities.
|
2026-07-24 00:15:27 +02:00
|
|
|
|
|
|
|
|
Git never stores weight tensors. Git stores:
|
|
|
|
|
|
|
|
|
|
* inventory catalog metadata (`inventory/catalog/`)
|
|
|
|
|
* policies and schemas
|
|
|
|
|
* research briefs
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## Storage class
|
|
|
|
|
|
|
|
|
|
| Requirement | Policy |
|
|
|
|
|
| ----------- | ------ |
|
2026-09-13 22:45:43 +02:00
|
|
|
| **Class** | Backup / bulk durable object storage — **not** hot cluster PVCs |
|
|
|
|
|
| **Facility** | Scaleway Object Storage `nl-ams` via `reef-storage` |
|
|
|
|
|
| **Durability** | Provider 11-nines claim; independent of Host Europe `railiance01` and of the workstation HD |
|
|
|
|
|
| **Performance** | Sequential restore for training/inference pull is enough |
|
|
|
|
|
| **Access** | Operator and approved lab hosts; not a public mirror |
|
|
|
|
|
| **Separation** | **Never** the postgres backup bucket (`railiance-platform-pg-backup` expires at 30 days) |
|
2026-07-24 00:15:27 +02:00
|
|
|
|
|
|
|
|
Platform backup paths such as `/opt/backup/railiance/{infra,cluster}/` and
|
2026-09-13 22:45:43 +02:00
|
|
|
`~/.cache/railiance/backups/` are **operational recovery** lanes. Model
|
|
|
|
|
weights belong in the FI object bucket.
|
2026-07-24 00:15:27 +02:00
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
2026-07-28 00:25:21 +02:00
|
|
|
## Target location
|
2026-07-24 00:15:27 +02:00
|
|
|
|
|
|
|
|
| Field | Value |
|
|
|
|
|
| ----- | ----- |
|
2026-09-13 22:45:43 +02:00
|
|
|
| **Facility** | Scaleway Object Storage, region `nl-ams` |
|
|
|
|
|
| **Endpoint** | `https://s3.nl-ams.scw.cloud` |
|
|
|
|
|
| **Bucket** | `railiance-fi-open-weight-reserve` (planned until FI-WP-0004-T08) |
|
|
|
|
|
| **Attributes** | `reef:storage/substrate/object-stores/fi-open-weight-reserve.yaml` |
|
|
|
|
|
| **Credentials** | OpenBao / `railiance-platform` — never git |
|
2026-07-24 00:15:27 +02:00
|
|
|
|
2026-09-13 22:45:43 +02:00
|
|
|
**Local staging (not SoT):** workstation VAULT HD
|
|
|
|
|
`D:\vault\coulomb\freedom-intelligence\` /
|
|
|
|
|
`/mnt/d/vault/coulomb/freedom-intelligence/` for `huggingface_hub`
|
|
|
|
|
downloads and an optional R-spine cache.
|
2026-07-24 00:15:27 +02:00
|
|
|
|
2026-09-13 22:45:43 +02:00
|
|
|
Do **not** bulk-download multi-GB models into this git workspace or into
|
|
|
|
|
hot root filesystems (`/` on WSL, `C:`, railiance PVCs).
|
2026-07-28 00:25:21 +02:00
|
|
|
|
2026-09-13 22:45:43 +02:00
|
|
|
### Prefix shape (pinned)
|
2026-07-24 00:15:27 +02:00
|
|
|
|
|
|
|
|
```text
|
2026-09-13 22:45:43 +02:00
|
|
|
s3://railiance-fi-open-weight-reserve/
|
|
|
|
|
├── models/{org}__{name}/{revision}/ # R / W (One Zone IA)
|
|
|
|
|
├── strategic/{org}__{name}/{revision}/ # S (Glacier)
|
|
|
|
|
├── companions/
|
|
|
|
|
├── staging/
|
|
|
|
|
└── manifests/{org}__{name}/{revision}/MANIFEST.json
|
2026-07-24 00:15:27 +02:00
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Map each `{org}__{name}/{revision}` to an inventory catalog entry.
|
2026-09-13 22:45:43 +02:00
|
|
|
`collection.storage_path` is the `s3://` URI.
|
2026-07-28 00:25:21 +02:00
|
|
|
|
2026-07-24 00:15:27 +02:00
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## Capacity budget
|
|
|
|
|
|
|
|
|
|
| Parameter | Policy |
|
|
|
|
|
| --------- | ------ |
|
2026-09-13 22:45:43 +02:00
|
|
|
| **Gate** | **Euros per month**, not GiB |
|
|
|
|
|
| **Soft budget** | **€15 / month** ex VAT for this bucket |
|
|
|
|
|
| **Hard backstop** | Founder Scaleway project billing alert (no provider euro hard-stop) |
|
|
|
|
|
| **Growth review** | When the running month is ≥ **70% of soft budget (~€10.50)** |
|
2026-07-28 00:25:21 +02:00
|
|
|
| **Per-pull threshold** | See `inventory/collection-policy.md` |
|
2026-09-13 22:45:43 +02:00
|
|
|
| **Eviction** | Prefer drop **W** and easily re-obtained quants; protect unique **S** SOTA bases and the **R** spine |
|
|
|
|
|
|
|
|
|
|
Rough cost (2026-09-13 Scaleway list prices):
|
2026-07-28 00:25:21 +02:00
|
|
|
|
2026-09-13 22:45:43 +02:00
|
|
|
| Mix | Class | ≈ € / month |
|
|
|
|
|
| --- | ----- | ----------: |
|
|
|
|
|
| R-spine ~45 GiB | One Zone IA | 0.54 |
|
|
|
|
|
| V4-Flash ~167 GiB | Glacier | 0.42 |
|
|
|
|
|
| Kimi K3 ~1454 GiB | Glacier | 3.69 |
|
|
|
|
|
| Those three together | | **~4.65** |
|
2026-07-28 00:25:21 +02:00
|
|
|
|
2026-09-13 22:45:43 +02:00
|
|
|
Glacier restore is €0.009/GB — pay it when we actually need the weights
|
|
|
|
|
on a GPU host.
|
2026-07-24 00:15:27 +02:00
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## Integrity and provenance
|
|
|
|
|
|
|
|
|
|
For every completed collection:
|
|
|
|
|
|
|
|
|
|
1. Record source URL and revision in the catalog entry.
|
2026-09-13 22:45:43 +02:00
|
|
|
2. Store checksums (`sha256` of each blob or upstream manifest digest)
|
|
|
|
|
in `MANIFEST.json` (local staging **and** `manifests/` prefix).
|
2026-07-24 00:15:27 +02:00
|
|
|
3. Record download date (UTC) and downloader identity.
|
|
|
|
|
4. Prefer official org releases over anonymous re-uploads.
|
|
|
|
|
5. Keep license text or SPDX id in catalog; refuse unclear licenses.
|
|
|
|
|
|
2026-09-13 22:45:43 +02:00
|
|
|
Verification: **catalog claims must match object checksums** before
|
|
|
|
|
status `collected`.
|
2026-07-24 00:15:27 +02:00
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## Retention
|
|
|
|
|
|
|
|
|
|
| Class | Retention |
|
|
|
|
|
| ----- | --------- |
|
2026-09-13 22:45:43 +02:00
|
|
|
| **Strategic capability (S)** | Keep until explicit deprecation; Glacier |
|
|
|
|
|
| **Runnable spine (R)** | Keep while still the lab default; One Zone IA |
|
2026-07-28 00:25:21 +02:00
|
|
|
| **Working set** | Active experiment bases + optional one superseded revision |
|
2026-07-24 00:15:27 +02:00
|
|
|
| **Staging** | Max 14 days incomplete, then purge |
|
2026-09-13 22:45:43 +02:00
|
|
|
| **Deprecated / evicted** | Metadata retained in catalog; objects may be deleted |
|
2026-07-24 00:15:27 +02:00
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## Encryption and offsite
|
|
|
|
|
|
|
|
|
|
| Topic | Policy |
|
|
|
|
|
| ----- | ------ |
|
2026-09-13 22:45:43 +02:00
|
|
|
| **At rest** | Scaleway provider-managed |
|
|
|
|
|
| **In transit** | TLS to `s3.nl-ams.scw.cloud` |
|
|
|
|
|
| **Workstation copy** | Optional R-spine on VAULT; not a second SoT |
|
|
|
|
|
| **Ops backups** | Do not mix with Forgejo/k3s dumps |
|
2026-07-24 00:15:27 +02:00
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## What must not live here
|
|
|
|
|
|
2026-09-13 22:45:43 +02:00
|
|
|
* Closed weights or artifacts whose terms forbid offline retention
|
|
|
|
|
* Secrets, API keys, customer data, or ungoverned training corpora
|
|
|
|
|
* Operational backups (databases, k3s state, Forgejo dumps)
|
|
|
|
|
* Git LFS dumps of full model trees
|
|
|
|
|
* Objects in `railiance-platform-pg-backup`
|
2026-07-24 00:15:27 +02:00
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
2026-07-28 00:25:21 +02:00
|
|
|
## Operator checklist
|
2026-07-24 00:15:27 +02:00
|
|
|
|
2026-09-13 22:45:43 +02:00
|
|
|
- [x] Choose facility: Scaleway object storage (2026-09-13)
|
|
|
|
|
- [x] Set capacity gate: **€15 / month** soft
|
|
|
|
|
- [x] Pin prefix layout above
|
|
|
|
|
- [ ] Create bucket + scoped key (FI-WP-0004-T08)
|
|
|
|
|
- [ ] Smoke PUT/GET
|
|
|
|
|
- [ ] First S collect: DeepSeek-V4-Flash-0731 (FI-WP-0004-T09)
|
2026-07-24 00:15:27 +02:00
|
|
|
|
|
|
|
|
### Pin log
|
|
|
|
|
|
|
|
|
|
| Date | Operator | Change |
|
|
|
|
|
| ---- | -------- | ------ |
|
2026-07-28 00:25:21 +02:00
|
|
|
| 2026-07-24 | foundation | Policy created; path unpinned |
|
2026-09-13 22:45:43 +02:00
|
|
|
| 2026-07-24 | operator direction | Facility intent = 1 TB local NAS; soft quota 850 GiB |
|
|
|
|
|
| 2026-07-28 | operator direction | **Interim pin:** workstation VAULT HD |
|
|
|
|
|
| 2026-09-13 | FI-WP-0004 | **SoT pin:** Scaleway `nl-ams`; VAULT demoted to staging; euro gate replaces GiB quota |
|