Stream model reserve to Scaleway without local weight staging
Assistant: codex Assistant-Model: gpt-6-astra Assistant-Session: 01a09cbd-43c1-79f3-809e-1ee97b40b64d
This commit is contained in:
parent
ad88d16a52
commit
8a65297d56
16 changed files with 805 additions and 145 deletions
|
|
@ -1,9 +1,9 @@
|
|||
# Open-weight reserve status
|
||||
|
||||
**As of:** 2026-09-13
|
||||
**As of:** 2026-09-14
|
||||
**SoT store:** Scaleway Object Storage `nl-ams` bucket
|
||||
`railiance-fi-open-weight-reserve` (**planned** — FI-WP-0004-T08)
|
||||
**Staging:** `/mnt/d/vault/coulomb/freedom-intelligence/` (VAULT HD)
|
||||
`railiance-fi-open-weight-reserve` (**live** — FI-WP-0004-T08)
|
||||
**Transfer:** diskless streaming on Railiance; [operations](../docs/streaming-reserve.md)
|
||||
**Capacity gate:** **€15 / month** soft (not 850 GiB)
|
||||
|
||||
---
|
||||
|
|
@ -14,7 +14,8 @@
|
|||
| ----- | ----: | ---------- |
|
||||
| Catalog entries | 15 | + V4-Flash S, + Qwen3.8-27B W/R |
|
||||
| **verified** on VAULT | 4 | R embeds + Qwen3-8B + R1-Distill-14B |
|
||||
| **approved**, blobs missing | S giants + Llama-3.2-3B + **V4-Flash** | HF gate / bucket not live |
|
||||
| **approved**, blobs missing | S giants + Llama-3.2-3B | HF gate / deferred |
|
||||
| **collecting** | 1 | V4-Flash → Scaleway Glacier, remote stream |
|
||||
| **candidate** | 4 | W + Qwen3.8-27B |
|
||||
|
||||
### Disk / object use
|
||||
|
|
@ -25,7 +26,7 @@
|
|||
| VAULT `models/BAAI__bge-m3/main` | R embed (staging cache) | ~2.1 GiB verified |
|
||||
| VAULT `models/Qwen__Qwen3-8B/main` | R instruct (staging cache) | ~15.3 GiB verified |
|
||||
| VAULT `models/deepseek-ai__DeepSeek-R1-Distill-Qwen-14B/main` | R reason (staging cache) | ~27.5 GiB verified |
|
||||
| Scaleway bucket | SoT | **empty** (not created) |
|
||||
| Scaleway bucket | SoT | V4-Flash streaming; final manifest pending |
|
||||
|
||||
VAULT ~45 GiB used remains a **cache**, not the reserve.
|
||||
|
||||
|
|
@ -40,7 +41,7 @@ VAULT ~45 GiB used remains a **cache**, not the reserve.
|
|||
| BGE-M3 | R | **verified** on VAULT cache | local only |
|
||||
| R1-Distill-Qwen-14B | R | **verified** on VAULT cache | local only |
|
||||
| nomic-embed-text-v1.5 | R | **verified** on VAULT cache | local only |
|
||||
| **DeepSeek-V4-Flash-0731** | **S1** | **approved** — **first Scaleway pull** | no — ~167 GiB Glacier |
|
||||
| **DeepSeek-V4-Flash-0731** | **S1** | **collecting** — **first Scaleway pull** | partial — 166.9 GB / 155.4 GiB source, Glacier |
|
||||
| Kimi-K3 | S1 secondary | **approved**, deferred until after V4-Flash | no — ~1454 GiB Glacier ≈ €3.69/mo |
|
||||
| DeepSeek-V3 | S superseded identity | **approved**, deferred | no — V4-Flash is the collectable S1 |
|
||||
| DeepSeek-R1 full | S2 | **approved**, deferred | no |
|
||||
|
|
@ -65,17 +66,17 @@ that used that id were wrong. Do not collect under that name.
|
|||
| Edge micro-agent | Llama-3.2-3B blocked on HF auth |
|
||||
| Multilingual RAG embed | **Covered** (BGE-M3 + nomic) |
|
||||
| Local reason | **Covered** (R1-Distill-14B) |
|
||||
| Frontier-open MIT MoE offline | **Cataloged** V4-Flash — pull blocked on bucket |
|
||||
| Frontier-open MIT MoE offline | **Collecting** V4-Flash — remote diskless stream |
|
||||
| Full open giant (K3) | Cataloged; Glacier-affordable; after V4-Flash |
|
||||
|
||||
---
|
||||
|
||||
## Next pulls (operator)
|
||||
|
||||
1. **FI-WP-0004-T08** create `railiance-fi-open-weight-reserve` (no 30-day expiry).
|
||||
2. **FI-WP-0004-T09** collect `deepseek-ai/DeepSeek-V4-Flash-0731` → `strategic/` Glacier.
|
||||
1. **FI-WP-0004-T08** bucket live; dedicated IAM key remains outstanding.
|
||||
2. **FI-WP-0004-T09** monitor the active remote stream; require the complete manifest before marking collected.
|
||||
3. Set `HF_TOKEN` and pull Llama-3.2-3B-Instruct into `models/` One Zone IA.
|
||||
4. Decide Qwen3.8-27B vs keeping Qwen3-8B as the R instruct default.
|
||||
5. Optional: Kimi K3 on Glacier if the month is still under €15.
|
||||
|
||||
Tool: `scripts/collect_model.py` (weights-only, sequential, MANIFEST, optional `--s3-bucket`).
|
||||
Tool: `scripts/collect_model.py` (`--s3-bucket` streams without disk staging; explicit `--local-download` for cache only).
|
||||
|
|
|
|||
|
|
@ -5,7 +5,7 @@ org: deepseek-ai
|
|||
source:
|
||||
kind: huggingface
|
||||
url: https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731
|
||||
revision: main
|
||||
revision: 7872f01b1d1fe23eabc4c98b48bffcef5a386062
|
||||
model_card_url: https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731
|
||||
project_url: https://www.deepseek.com/
|
||||
paper_url: https://arxiv.org/abs/2606.19348
|
||||
|
|
@ -54,8 +54,8 @@ swot:
|
|||
- "Successor V4.x could land while we still have no object on disk"
|
||||
- "Quota-era deferral already cost a month of optionality"
|
||||
size:
|
||||
total_bytes: 179000000000
|
||||
total_human: "~167 GiB official HF tree (card file listing 2026-09-13); 304B params / ~13B active"
|
||||
total_bytes: 166898547054
|
||||
total_human: "166.9 GB / 155.4 GiB; 67 selected files at pinned commit, HF metadata 2026-09-14"
|
||||
hardware_class:
|
||||
min_vram_gb_q4: 0
|
||||
min_vram_gb_fp16: 0
|
||||
|
|
@ -96,3 +96,7 @@ history:
|
|||
event: collecting
|
||||
by: grok
|
||||
detail: "T08 bucket live. snapshot_download + Glacier upload started on workstation staging /home/worsch/vault/freedom-intelligence."
|
||||
- at: "2026-09-14"
|
||||
event: collecting
|
||||
by: codex
|
||||
detail: "Replaced WSL snapshot staging with diskless Railiance service fi-reserve-collect. 64 MiB parts, 512 MiB memory cap, no swap. Source pinned to 7872f01b1d1fe23eabc4c98b48bffcef5a386062; 67 selected files / 166898547054 bytes. Destination strategic/deepseek-ai__DeepSeek-V4-Flash-0731/<commit>/. Complete manifest pending; not collected or restore-verified."
|
||||
|
|
|
|||
|
|
@ -10,7 +10,7 @@
|
|||
|
||||
Decide **what** enters the open-weight reserve, **who** may approve it, and
|
||||
**when** a daily-brief candidate becomes a catalog entry with blobs on
|
||||
Scaleway (staged locally first).
|
||||
Scaleway (streamed through a remote worker with bounded RAM; no local staging).
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -127,7 +127,7 @@ Tokenizers, LoRA adapters, small eval fixtures when required to use a reserved b
|
|||
brief nominates
|
||||
→ candidate
|
||||
→ approved
|
||||
→ collecting (local staging/ then S3 upload)
|
||||
→ collecting (remote HTTP stream → S3 multipart)
|
||||
→ collected
|
||||
→ verified
|
||||
→ superseded|evicted
|
||||
|
|
@ -140,7 +140,7 @@ brief nominates
|
|||
1. Brief **Collection candidates** nominates R/S/W.
|
||||
2. Catalog YAML under `inventory/catalog/`.
|
||||
3. Download only after the Scaleway bucket is live (FI-WP-0004-T08) and
|
||||
approval rules pass. Stage on VAULT; SoT is `s3://`.
|
||||
approval rules pass. Use [diskless streaming](../docs/streaming-reserve.md); SoT is `s3://`.
|
||||
4. `collection.brief_refs` / research refs for provenance of the nomination.
|
||||
|
||||
---
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue