coulomb/freedom-intelligence
4
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
freedom-intelligence/docs/backup-storage-policy.md
tegwick a78187c33e Make the sensing loop durable and pin the reserve to Scaleway 
The weekday brief only counts when the file is on origin/main. Hub
events without git objects are failures. The open-weight SoT moves
from the VAULT 850 GiB quota to a dedicated Scaleway bucket (One Zone
IA for R, Glacier for S) so V4-Flash and K3 are economically in-scope.

Catalogs the real DeepSeek-V4-Flash-0731 (MIT ~167 GiB MoE, not 12B)
and nominates Qwen3.8-27B. Bucket create remains FI-WP-0004-T08.

Assistant: grok
Assistant-Session: 01a09c6a-1cfc-75b1-a78d-c13eaf22241d
2026-09-13 22:45:43 +02:00

5.8 KiB
Raw Blame History

Backup storage policy — open-weight model reserve

Status: facility pin Scaleway Object Storage (2026-09-13)
Related: INTENT.md, SCOPE.md, inventory/collection-policy.md, docs/decisions/2026-09-13-scaleway-object-reserve.md
Supersedes storage pin: 2026-07-28 VAULT HD (now staging only)
Adjacent: disaster-control BackupPolicy and rapp-postgres WAL archive — different buckets, different lifecycles

Purpose

Define where Freedom Intelligence stores open-weight model blobs (and justified companions), how much we allow ourselves to spend, and how this reserve relates to other lab backup facilities.

Git never stores weight tensors. Git stores:

  • inventory catalog metadata (inventory/catalog/)
  • policies and schemas
  • research briefs

Storage class

Requirement Policy
Class Backup / bulk durable object storage — not hot cluster PVCs
Facility Scaleway Object Storage nl-ams via reef-storage
Durability Provider 11-nines claim; independent of Host Europe railiance01 and of the workstation HD
Performance Sequential restore for training/inference pull is enough
Access Operator and approved lab hosts; not a public mirror
Separation Never the postgres backup bucket (railiance-platform-pg-backup expires at 30 days)

Platform backup paths such as /opt/backup/railiance/{infra,cluster}/ and ~/.cache/railiance/backups/ are operational recovery lanes. Model weights belong in the FI object bucket.

Target location

Field Value
Facility Scaleway Object Storage, region nl-ams
Endpoint https://s3.nl-ams.scw.cloud
Bucket railiance-fi-open-weight-reserve (planned until FI-WP-0004-T08)
Attributes reef:storage/substrate/object-stores/fi-open-weight-reserve.yaml
Credentials OpenBao / railiance-platform — never git

Local staging (not SoT): workstation VAULT HD D:\vault\coulomb\freedom-intelligence\ / /mnt/d/vault/coulomb/freedom-intelligence/ for huggingface_hub downloads and an optional R-spine cache.

Do not bulk-download multi-GB models into this git workspace or into hot root filesystems (/ on WSL, C:, railiance PVCs).

Prefix shape (pinned)

s3://railiance-fi-open-weight-reserve/
├── models/{org}__{name}/{revision}/      # R / W  (One Zone IA)
├── strategic/{org}__{name}/{revision}/   # S      (Glacier)
├── companions/
├── staging/
└── manifests/{org}__{name}/{revision}/MANIFEST.json

Map each {org}__{name}/{revision} to an inventory catalog entry. collection.storage_path is the s3:// URI.

Capacity budget

Parameter Policy
Gate Euros per month, not GiB
Soft budget €15 / month ex VAT for this bucket
Hard backstop Founder Scaleway project billing alert (no provider euro hard-stop)
Growth review When the running month is ≥ 70% of soft budget (~€10.50)
Per-pull threshold See inventory/collection-policy.md
Eviction Prefer drop W and easily re-obtained quants; protect unique S SOTA bases and the R spine

Rough cost (2026-09-13 Scaleway list prices):

Mix Class ≈ € / month
R-spine ~45 GiB One Zone IA 0.54
V4-Flash ~167 GiB Glacier 0.42
Kimi K3 ~1454 GiB Glacier 3.69
Those three together ~4.65

Glacier restore is €0.009/GB — pay it when we actually need the weights on a GPU host.

Integrity and provenance

For every completed collection:

  1. Record source URL and revision in the catalog entry.
  2. Store checksums (sha256 of each blob or upstream manifest digest) in MANIFEST.json (local staging and manifests/ prefix).
  3. Record download date (UTC) and downloader identity.
  4. Prefer official org releases over anonymous re-uploads.
  5. Keep license text or SPDX id in catalog; refuse unclear licenses.

Verification: catalog claims must match object checksums before status collected.

Retention

Class Retention
Strategic capability (S) Keep until explicit deprecation; Glacier
Runnable spine (R) Keep while still the lab default; One Zone IA
Working set Active experiment bases + optional one superseded revision
Staging Max 14 days incomplete, then purge
Deprecated / evicted Metadata retained in catalog; objects may be deleted

Encryption and offsite

Topic Policy
At rest Scaleway provider-managed
In transit TLS to s3.nl-ams.scw.cloud
Workstation copy Optional R-spine on VAULT; not a second SoT
Ops backups Do not mix with Forgejo/k3s dumps

What must not live here

  • Closed weights or artifacts whose terms forbid offline retention
  • Secrets, API keys, customer data, or ungoverned training corpora
  • Operational backups (databases, k3s state, Forgejo dumps)
  • Git LFS dumps of full model trees
  • Objects in railiance-platform-pg-backup

Operator checklist

  • Choose facility: Scaleway object storage (2026-09-13)
  • Set capacity gate: €15 / month soft
  • Pin prefix layout above
  • Create bucket + scoped key (FI-WP-0004-T08)
  • Smoke PUT/GET
  • First S collect: DeepSeek-V4-Flash-0731 (FI-WP-0004-T09)

Pin log

Date Operator Change
2026-07-24 foundation Policy created; path unpinned
2026-07-24 operator direction Facility intent = 1 TB local NAS; soft quota 850 GiB
2026-07-28 operator direction Interim pin: workstation VAULT HD
2026-09-13 FI-WP-0004 SoT pin: Scaleway nl-ams; VAULT demoted to staging; euro gate replaces GiB quota