freedom-intelligence/docs/streaming-reserve.md

Streaming the model reserve

Decision: 2026-09-14. Bulk reserve collection must not stage weights in WSL, on a workstation disk, or on the remote host's root disk.

Hugging Face HTTPS → Railiance worker RAM → Scaleway multipart upload
                          64 MiB parts
                    no model files on disk

collect_model.py --s3-bucket ... now uses this path. A full local snapshot requires --local-download without --s3-bucket, for an intentional local cache only. The streaming path never invokes snapshot_download or creates an HF weights cache. Python dependencies and bounded service logs use disk.

Transfer contract

• Resolve the requested source revision to an immutable commit before upload. S3 keys use that commit, not mutable main.
• Apply the existing collection allow/ignore patterns. Validate every selected file's size and upstream digest before starting the model.
• Read and upload one part at a time. Default buffer: 64 MiB; configurable 5–128 MiB. Reject files that would exceed S3's 10,000-part limit.
• Calculate SHA256 while reading. Check LFS SHA256 or the Git blob SHA1 for ordinary repository files before completing each multipart upload.
• Send Content-MD5, but do not rely on its enforcement: the live smoke test found Scaleway accepts a wrong value. Explicitly compare each returned part ETag with local MD5 and the assembled ETag with the calculated multipart composite. Fail closed on a mismatch or an unexpected ETag format. Check destination size, metadata and version identity. ETags are not SHA256. Small receipts and manifests also receive a full JSON readback check.
• Write a small STANDARD receipt per completed object. Restarts skip only receipts matching the source and current remote object identity. An interrupted file starts again at byte zero. No model-sized local checkpoint.
• SIGTERM aborts the current multipart upload. On restart, abort orphan uploads only under this model's immutable prefix. Run one worker per destination prefix; the supplied service uses a host lock. Do not launch another host against the same prefix. The bucket's existing staging/ abort lifecycle does not cover strategic/; restart/manual cleanup is required after a permanent worker loss.
• Publish manifests/{local_name}/{commit}/MANIFEST.json only after all files complete. It includes source hashes, computed SHA256, object versions and the verification method. Credentials and signed URLs are never logged.
• R/W objects use ONEZONE_IA; S objects use GLACIER. Manifests and receipts remain STANDARD. collected requires the complete manifest; verified additionally requires a restore/readback checksum check. A Glacier HEAD check alone does not establish restore verification.

Provider references: Scaleway multipart uploads, Hugging Face model metadata.

Railiance worker

Files: ~/.local/share/fi-reserve/scripts/; venv alongside them. User service: fi-reserve-collect.service. It pins the V4-Flash source commit to 7872f01b1d1fe23eabc4c98b48bffcef5a386062 (67 selected files, 166,898,547,054 bytes; approximately 155.4 GiB).

Limits: MemoryHigh=384M, MemoryMax=512M, MemorySwapMax=0, CPUQuota=50%, TasksMax=32. One worker, no background parallel download pool. Three service starts/hour maximum; each file has three attempts. Resource limits are a backstop: a killed worker retries without filling the host's filesystem.

Install/update (stop an existing worker before replacing code):

ssh railiance01 'mkdir -p ~/.local/share/fi-reserve/scripts ~/.config/systemd/user && python3 -m venv ~/.local/share/fi-reserve/venv'
scp scripts/collect_model.py scripts/stream_model.py scripts/smoke_stream_model.py scripts/requirements-collect.txt railiance01:.local/share/fi-reserve/scripts/
scp scripts/fi-reserve-collect.service railiance01:.config/systemd/user/
ssh railiance01 '~/.local/share/fi-reserve/venv/bin/pip install -r ~/.local/share/fi-reserve/scripts/requirements-collect.txt'
python3 scripts/provision-reserve-credentials.py --host railiance01 --allow-bootstrap
ssh railiance01 'systemctl --user daemon-reload && systemctl --user start fi-reserve-collect'

The credential helper first checks the dedicated FI OpenBao path. The --allow-bootstrap flag allows the pre-existing fallback while the dedicated key is absent. It sends only S3 keys over SSH, never the OpenBao token. Remote keys live at /run/user/1000/fi-reserve/s3.json (0600), lost on reboot. The service is deliberately not enabled at boot. Re-provision keys before restart after a reboot. Replace the bootstrap with a bucket-scoped key when available.

ssh railiance01 'systemctl --user status fi-reserve-collect --no-pager'
ssh railiance01 'journalctl --user -u fi-reserve-collect -n 20 --no-pager'
ssh railiance01 'systemctl --user show fi-reserve-collect -p MemoryCurrent -p MemoryPeak -p MemoryMax -p MemorySwapMax -p NRestarts'
ssh railiance01 'systemctl --user stop fi-reserve-collect'

Tests: python -m unittest discover -s scripts -p 'test_*.py' in the collector venv. smoke_stream_model.py exercises live multipart STANDARD/ONEZONE_IA readback, GLACIER upload/HEAD, and behavior with a deliberately incorrect Content-MD5. All three storage classes passed with explicit ETag checks. It deletes only the smoke versions it created.

Migration from WSL

The old snapshot-first process was absent on inspection. Its partial /home/worsch/vault/freedom-intelligence/staging/deepseek-ai__DeepSeek-V4-Flash-0731/main occupied about 39 GiB. After the first remote weight shard completed, the operator-approved cleanup removed this abandoned snapshot. Do not resume the old command. The remote stream starts from Hugging Face; no local model files are sent to Railiance. Deletion frees space inside WSL; shrinking the Windows VHDX file, if needed, is a separate host operation.

Read-only object progress:

scp scripts/reserve_progress.py railiance01:.local/share/fi-reserve/scripts/
ssh railiance01 'FI_S3_CREDENTIAL_FILE=/run/user/1000/fi-reserve/s3.json ~/.local/share/fi-reserve/venv/bin/python ~/.local/share/fi-reserve/scripts/reserve_progress.py'

Initial live evidence (2026-09-14 Berlin): first weight shard plus 16 metadata objects completed (1,059,102,756 object bytes); second shard uploading. Worker active with zero restarts, observed peak 136,933,376 bytes (~131 MiB). Cgroup memory.max=536870912, memory.swap.max=0. Worker installation (code and dependencies) occupies 77 MiB, with no model-weight staging. Thirteen automated tests pass, including checksum failures, interruptions, resume identity and the rule that failed transfers cannot publish a complete manifest.