Stream model reserve to Scaleway without local weight staging

Assistant: codex
Assistant-Model: gpt-6-astra
Assistant-Session: 01a09cbd-43c1-79f3-809e-1ee97b40b64d
This commit is contained in:
tegwick 2026-09-14 00:17:15 +02:00
parent ad88d16a52
commit 8a65297d56
16 changed files with 805 additions and 145 deletions

View file

@ -51,13 +51,10 @@ weights belong in the FI object bucket.
| **Attributes** | `reef:storage/substrate/object-stores/fi-open-weight-reserve.yaml` |
| **Credentials** | OpenBao / `railiance-platform` — never git |
**Local staging (not SoT):** workstation VAULT HD
`D:\vault\coulomb\freedom-intelligence\` /
`/mnt/d/vault/coulomb/freedom-intelligence/` for `huggingface_hub`
downloads and an optional R-spine cache.
Do **not** bulk-download multi-GB models into this git workspace or into
hot root filesystems (`/` on WSL, `C:`, railiance PVCs).
**Bulk transfer:** [diskless streaming on Railiance](streaming-reserve.md),
with 64 MiB parts and a 512 MiB process memory limit. Weight files are never
staged on WSL or on the remote host's root disk. VAULT is an optional explicit
R-spine cache only (`--local-download`), not an intermediate reserve step.
### Prefix shape (pinned)
@ -106,13 +103,14 @@ For every completed collection:
1. Record source URL and revision in the catalog entry.
2. Store checksums (`sha256` of each blob or upstream manifest digest)
in `MANIFEST.json` (local staging **and** `manifests/` prefix).
in `MANIFEST.json` under the STANDARD `manifests/` prefix.
3. Record download date (UTC) and downloader identity.
4. Prefer official org releases over anonymous re-uploads.
5. Keep license text or SPDX id in catalog; refuse unclear licenses.
Verification: **catalog claims must match object checksums** before
status `collected`.
Verification: source hashes, explicit part/composite MD5 ETags and remote object identities
must match before `collected`. `verified` requires restore/readback checksums;
Glacier HEAD alone is insufficient. See the streaming transfer contract.
---