Stream model reserve to Scaleway without local weight staging
Assistant: codex Assistant-Model: gpt-6-astra Assistant-Session: 01a09cbd-43c1-79f3-809e-1ee97b40b64d
This commit is contained in:
parent
ad88d16a52
commit
8a65297d56
16 changed files with 805 additions and 145 deletions
|
|
@ -51,13 +51,10 @@ weights belong in the FI object bucket.
|
|||
| **Attributes** | `reef:storage/substrate/object-stores/fi-open-weight-reserve.yaml` |
|
||||
| **Credentials** | OpenBao / `railiance-platform` — never git |
|
||||
|
||||
**Local staging (not SoT):** workstation VAULT HD
|
||||
`D:\vault\coulomb\freedom-intelligence\` /
|
||||
`/mnt/d/vault/coulomb/freedom-intelligence/` for `huggingface_hub`
|
||||
downloads and an optional R-spine cache.
|
||||
|
||||
Do **not** bulk-download multi-GB models into this git workspace or into
|
||||
hot root filesystems (`/` on WSL, `C:`, railiance PVCs).
|
||||
**Bulk transfer:** [diskless streaming on Railiance](streaming-reserve.md),
|
||||
with 64 MiB parts and a 512 MiB process memory limit. Weight files are never
|
||||
staged on WSL or on the remote host's root disk. VAULT is an optional explicit
|
||||
R-spine cache only (`--local-download`), not an intermediate reserve step.
|
||||
|
||||
### Prefix shape (pinned)
|
||||
|
||||
|
|
@ -106,13 +103,14 @@ For every completed collection:
|
|||
|
||||
1. Record source URL and revision in the catalog entry.
|
||||
2. Store checksums (`sha256` of each blob or upstream manifest digest)
|
||||
in `MANIFEST.json` (local staging **and** `manifests/` prefix).
|
||||
in `MANIFEST.json` under the STANDARD `manifests/` prefix.
|
||||
3. Record download date (UTC) and downloader identity.
|
||||
4. Prefer official org releases over anonymous re-uploads.
|
||||
5. Keep license text or SPDX id in catalog; refuse unclear licenses.
|
||||
|
||||
Verification: **catalog claims must match object checksums** before
|
||||
status `collected`.
|
||||
Verification: source hashes, explicit part/composite MD5 ETags and remote object identities
|
||||
must match before `collected`. `verified` requires restore/readback checksums;
|
||||
Glacier HEAD alone is insufficient. See the streaming transfer contract.
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
118
docs/streaming-reserve.md
Normal file
118
docs/streaming-reserve.md
Normal file
|
|
@ -0,0 +1,118 @@
|
|||
# Streaming the model reserve
|
||||
|
||||
Decision: 2026-09-14. Bulk reserve collection must not stage weights in WSL,
|
||||
on a workstation disk, or on the remote host's root disk.
|
||||
|
||||
```text
|
||||
Hugging Face HTTPS → Railiance worker RAM → Scaleway multipart upload
|
||||
64 MiB parts
|
||||
no model files on disk
|
||||
```
|
||||
|
||||
`collect_model.py --s3-bucket ...` now uses this path. A full local snapshot
|
||||
requires `--local-download` without `--s3-bucket`, for an intentional local
|
||||
cache only. The streaming path never invokes `snapshot_download` or creates
|
||||
an HF weights cache. Python dependencies and bounded service logs use disk.
|
||||
|
||||
## Transfer contract
|
||||
|
||||
- Resolve the requested source revision to an immutable commit before upload.
|
||||
S3 keys use that commit, not mutable `main`.
|
||||
- Apply the existing collection allow/ignore patterns. Validate every selected
|
||||
file's size and upstream digest before starting the model.
|
||||
- Read and upload one part at a time. Default buffer: 64 MiB; configurable
|
||||
5–128 MiB. Reject files that would exceed S3's 10,000-part limit.
|
||||
- Calculate SHA256 while reading. Check LFS SHA256 or the Git blob SHA1 for
|
||||
ordinary repository files before completing each multipart upload.
|
||||
- Send Content-MD5, but do not rely on its enforcement: the live smoke test
|
||||
found Scaleway accepts a wrong value. Explicitly compare each returned part
|
||||
ETag with local MD5 and the assembled ETag with the calculated multipart
|
||||
composite. Fail closed on a mismatch or an unexpected ETag format. Check
|
||||
destination size, metadata and version identity. ETags are not SHA256.
|
||||
Small receipts and manifests also receive a full JSON readback check.
|
||||
- Write a small STANDARD receipt per completed object. Restarts skip only
|
||||
receipts matching the source and current remote object identity. An
|
||||
interrupted file starts again at byte zero. No model-sized local checkpoint.
|
||||
- SIGTERM aborts the current multipart upload. On restart, abort orphan uploads
|
||||
only under this model's immutable prefix. Run one worker per destination
|
||||
prefix; the supplied service uses a host lock. Do not launch another host
|
||||
against the same prefix. The bucket's existing `staging/` abort lifecycle
|
||||
does not cover `strategic/`; restart/manual cleanup is required after a
|
||||
permanent worker loss.
|
||||
- Publish `manifests/{local_name}/{commit}/MANIFEST.json` only after all files
|
||||
complete. It includes source hashes, computed SHA256, object versions and
|
||||
the verification method. Credentials and signed URLs are never logged.
|
||||
- R/W objects use ONEZONE_IA; S objects use GLACIER. Manifests and receipts
|
||||
remain STANDARD. `collected` requires the complete manifest; `verified`
|
||||
additionally requires a restore/readback checksum check. A Glacier HEAD
|
||||
check alone does not establish restore verification.
|
||||
|
||||
Provider references: [Scaleway multipart uploads](https://www.scaleway.com/en/docs/object-storage/api-cli/multipart-uploads/),
|
||||
[Hugging Face model metadata](https://huggingface.co/docs/huggingface_hub/package_reference/hf_api).
|
||||
|
||||
## Railiance worker
|
||||
|
||||
Files: `~/.local/share/fi-reserve/scripts/`; venv alongside them. User service:
|
||||
`fi-reserve-collect.service`. It pins the V4-Flash source commit to
|
||||
`7872f01b1d1fe23eabc4c98b48bffcef5a386062` (67 selected files,
|
||||
166,898,547,054 bytes; approximately 155.4 GiB).
|
||||
|
||||
Limits: MemoryHigh=384M, MemoryMax=512M, MemorySwapMax=0, CPUQuota=50%,
|
||||
TasksMax=32. One worker, no background parallel download pool. Three service
|
||||
starts/hour maximum; each file has three attempts. Resource limits are a
|
||||
backstop: a killed worker retries without filling the host's filesystem.
|
||||
|
||||
Install/update (stop an existing worker before replacing code):
|
||||
|
||||
```bash
|
||||
ssh railiance01 'mkdir -p ~/.local/share/fi-reserve/scripts ~/.config/systemd/user && python3 -m venv ~/.local/share/fi-reserve/venv'
|
||||
scp scripts/collect_model.py scripts/stream_model.py scripts/smoke_stream_model.py scripts/requirements-collect.txt railiance01:.local/share/fi-reserve/scripts/
|
||||
scp scripts/fi-reserve-collect.service railiance01:.config/systemd/user/
|
||||
ssh railiance01 '~/.local/share/fi-reserve/venv/bin/pip install -r ~/.local/share/fi-reserve/scripts/requirements-collect.txt'
|
||||
python3 scripts/provision-reserve-credentials.py --host railiance01 --allow-bootstrap
|
||||
ssh railiance01 'systemctl --user daemon-reload && systemctl --user start fi-reserve-collect'
|
||||
```
|
||||
|
||||
The credential helper first checks the dedicated FI OpenBao path. The
|
||||
`--allow-bootstrap` flag allows the pre-existing fallback while the dedicated
|
||||
key is absent. It sends only S3 keys over SSH, never the OpenBao token. Remote
|
||||
keys live at `/run/user/1000/fi-reserve/s3.json` (0600), lost on reboot. The
|
||||
service is deliberately not enabled at boot. Re-provision keys before restart
|
||||
after a reboot. Replace the bootstrap with a bucket-scoped key when available.
|
||||
|
||||
```bash
|
||||
ssh railiance01 'systemctl --user status fi-reserve-collect --no-pager'
|
||||
ssh railiance01 'journalctl --user -u fi-reserve-collect -n 20 --no-pager'
|
||||
ssh railiance01 'systemctl --user show fi-reserve-collect -p MemoryCurrent -p MemoryPeak -p MemoryMax -p MemorySwapMax -p NRestarts'
|
||||
ssh railiance01 'systemctl --user stop fi-reserve-collect'
|
||||
```
|
||||
|
||||
Tests: `python -m unittest discover -s scripts -p 'test_*.py'` in the collector
|
||||
venv. `smoke_stream_model.py` exercises live multipart STANDARD/ONEZONE_IA
|
||||
readback, GLACIER upload/HEAD, and behavior with a deliberately incorrect
|
||||
Content-MD5. All three storage classes passed with explicit ETag checks. It deletes only the smoke versions it created.
|
||||
|
||||
## Migration from WSL
|
||||
|
||||
The old snapshot-first process was absent on inspection. Its partial
|
||||
`/home/worsch/vault/freedom-intelligence/staging/deepseek-ai__DeepSeek-V4-Flash-0731/main`
|
||||
occupied about 39 GiB. After the first remote weight shard completed, the
|
||||
operator-approved cleanup removed this abandoned snapshot. Do not resume the
|
||||
old command. The remote stream starts from Hugging Face; no local model files
|
||||
are sent to Railiance. Deletion frees space inside WSL; shrinking the Windows
|
||||
VHDX file, if needed, is a separate host operation.
|
||||
|
||||
Read-only object progress:
|
||||
|
||||
```bash
|
||||
scp scripts/reserve_progress.py railiance01:.local/share/fi-reserve/scripts/
|
||||
ssh railiance01 'FI_S3_CREDENTIAL_FILE=/run/user/1000/fi-reserve/s3.json ~/.local/share/fi-reserve/venv/bin/python ~/.local/share/fi-reserve/scripts/reserve_progress.py'
|
||||
```
|
||||
|
||||
Initial live evidence (2026-09-14 Berlin): first weight shard plus 16 metadata
|
||||
objects completed (1,059,102,756 object bytes); second shard uploading.
|
||||
Worker active with zero restarts, observed peak 136,933,376 bytes (~131 MiB).
|
||||
Cgroup memory.max=536870912, memory.swap.max=0. Worker installation (code and
|
||||
dependencies) occupies 77 MiB, with no model-weight staging. Thirteen automated
|
||||
tests pass, including checksum failures, interruptions, resume identity and
|
||||
the rule that failed transfers cannot publish a complete manifest.
|
||||
Loading…
Add table
Add a link
Reference in a new issue