diff --git a/docs/streaming-reserve.md b/docs/streaming-reserve.md index 2b8a0e2..3f5707d 100644 --- a/docs/streaming-reserve.md +++ b/docs/streaming-reserve.md @@ -116,3 +116,11 @@ Cgroup memory.max=536870912, memory.swap.max=0. Worker installation (code and dependencies) occupies 77 MiB, with no model-weight staging. Thirteen automated tests pass, including checksum failures, interruptions, resume identity and the rule that failed transfers cannot publish a complete manifest. + +Completion (2026-09-13 23:18 UTC): all 67 objects uploaded and +`manifests/deepseek-ai__DeepSeek-V4-Flash-0731/7872f01b1d1fe23eabc4c98b48bffcef5a386062/MANIFEST.json` +published with `complete: true`. Worker peak 159.3 MiB, zero swap. Re-check on +2026-09-22: 166,898,547,054 object bytes, no open multipart uploads, all +objects GLACIER; the 48 LFS SHA256 source digests equal the computed SHA256; +HEAD size, ETag and version match the manifest for every object. Catalog +status is `collected`; `verified` still requires a restore readback. diff --git a/inventory/RESERVE-STATUS.md b/inventory/RESERVE-STATUS.md index fb1f400..2914fe0 100644 --- a/inventory/RESERVE-STATUS.md +++ b/inventory/RESERVE-STATUS.md @@ -1,6 +1,6 @@ # Open-weight reserve status -**As of:** 2026-09-14 +**As of:** 2026-09-22 **SoT store:** Scaleway Object Storage `nl-ams` bucket `railiance-fi-open-weight-reserve` (**live** — FI-WP-0004-T08) **Transfer:** diskless streaming on Railiance; [operations](../docs/streaming-reserve.md) @@ -15,7 +15,7 @@ | Catalog entries | 15 | + V4-Flash S, + Qwen3.8-27B W/R | | **verified** on VAULT | 4 | R embeds + Qwen3-8B + R1-Distill-14B | | **approved**, blobs missing | S giants + Llama-3.2-3B | HF gate / deferred | -| **collecting** | 1 | V4-Flash → Scaleway Glacier, remote stream | +| **collected** on Scaleway | 1 | V4-Flash, Glacier; manifest complete, no restore readback yet | | **candidate** | 4 | W + Qwen3.8-27B | ### Disk / object use @@ -26,7 +26,7 @@ | VAULT `models/BAAI__bge-m3/main` | R embed (staging cache) | ~2.1 GiB verified | | VAULT `models/Qwen__Qwen3-8B/main` | R instruct (staging cache) | ~15.3 GiB verified | | VAULT `models/deepseek-ai__DeepSeek-R1-Distill-Qwen-14B/main` | R reason (staging cache) | ~27.5 GiB verified | -| Scaleway bucket | SoT | V4-Flash streaming; final manifest pending | +| Scaleway bucket `strategic/` | SoT | V4-Flash 166.9 GB / 155.4 GiB, 67 objects, GLACIER, manifest complete | VAULT ~45 GiB used remains a **cache**, not the reserve. @@ -41,7 +41,7 @@ VAULT ~45 GiB used remains a **cache**, not the reserve. | BGE-M3 | R | **verified** on VAULT cache | local only | | R1-Distill-Qwen-14B | R | **verified** on VAULT cache | local only | | nomic-embed-text-v1.5 | R | **verified** on VAULT cache | local only | -| **DeepSeek-V4-Flash-0731** | **S1** | **collecting** — **first Scaleway pull** | partial — 166.9 GB / 155.4 GiB source, Glacier | +| **DeepSeek-V4-Flash-0731** | **S1** | **collected** — first Scaleway pull (2026-09-13) | yes — 166.9 GB / 155.4 GiB, Glacier; restore readback pending | | Kimi-K3 | S1 secondary | **approved**, deferred until after V4-Flash | no — ~1454 GiB Glacier ≈ €3.69/mo | | DeepSeek-V3 | S superseded identity | **approved**, deferred | no — V4-Flash is the collectable S1 | | DeepSeek-R1 full | S2 | **approved**, deferred | no | @@ -66,7 +66,7 @@ that used that id were wrong. Do not collect under that name. | Edge micro-agent | Llama-3.2-3B blocked on HF auth | | Multilingual RAG embed | **Covered** (BGE-M3 + nomic) | | Local reason | **Covered** (R1-Distill-14B) | -| Frontier-open MIT MoE offline | **Collecting** V4-Flash — remote diskless stream | +| Frontier-open MIT MoE offline | **Collected** V4-Flash on Glacier (not yet restore-verified) | | Full open giant (K3) | Cataloged; Glacier-affordable; after V4-Flash | --- @@ -74,7 +74,7 @@ that used that id were wrong. Do not collect under that name. ## Next pulls (operator) 1. **FI-WP-0004-T08** bucket live; dedicated IAM key remains outstanding. -2. **FI-WP-0004-T09** monitor the active remote stream; require the complete manifest before marking collected. +2. V4-Flash is collected. Mark it `verified` only after a Glacier restore + SHA256 readback. 3. Set `HF_TOKEN` and pull Llama-3.2-3B-Instruct into `models/` One Zone IA. 4. Decide Qwen3.8-27B vs keeping Qwen3-8B as the R instruct default. 5. Optional: Kimi K3 on Glacier if the month is still under €15. diff --git a/inventory/catalog/deepseek-ai__DeepSeek-V4-Flash-0731__strategic.yaml b/inventory/catalog/deepseek-ai__DeepSeek-V4-Flash-0731__strategic.yaml index 36bf90c..b359d13 100644 --- a/inventory/catalog/deepseek-ai__DeepSeek-V4-Flash-0731__strategic.yaml +++ b/inventory/catalog/deepseek-ai__DeepSeek-V4-Flash-0731__strategic.yaml @@ -1,5 +1,5 @@ id: deepseek-ai__DeepSeek-V4-Flash-0731__strategic -status: collecting +status: collected name: DeepSeek-V4-Flash-0731 org: deepseek-ai source: @@ -56,6 +56,208 @@ swot: size: total_bytes: 166898547054 total_human: "166.9 GB / 155.4 GiB; 67 selected files at pinned commit, HF metadata 2026-09-14" +artifacts: + - path: "LICENSE" + sha256: "f2c6c602815669d292889e5be8c802f2ed950653b77999b1584e8e6aed25d040" + bytes: 1084 + - path: "README.md" + sha256: "252acafdc9204d0dba3fde1b0a93d71cd1664a4ceadfe222b60117ed0ccc56ff" + bytes: 7238 + - path: "config.json" + sha256: "6c8f3d2d3b48707541b88f32f22ef3f0f8a6b57d8523281e2b8d3cdb0ae9a023" + bytes: 1888 + - path: "encoding/README.md" + sha256: "294ed6493ebfe5a487f172154d60c494899997fed8806fdd3352dd8e3c33f9de" + bytes: 9168 + - path: "encoding/tests/test_input_1.json" + sha256: "10e0c074c977c3a80daab758af28219c6b1c2bd7f3f5cf2890c84b361cc32897" + bytes: 2748 + - path: "encoding/tests/test_input_2.json" + sha256: "c44ae0db20fafff38a6021e8068d7ed6e28605d76cafd20be80b03398509f447" + bytes: 526 + - path: "encoding/tests/test_input_3.json" + sha256: "37bf8ef95e0411ea5f411be0b02fbafec7363438b6ccefddca0c52ec9aeaf69a" + bytes: 4546 + - path: "encoding/tests/test_input_4.json" + sha256: "c45bbd0a1b7a2f75033d8db4ba74ee5b7653bd01114045a52f79d3b387663465" + bytes: 2730 + - path: "encoding/tests/test_output_1.txt" + sha256: "9b366d9d2eac842a6e890594aac0b58648e5623717202b33497afadf03e26540" + bytes: 2390 + - path: "encoding/tests/test_output_2.txt" + sha256: "ca66b01a1ac3a204bb032c928fb607d4877171e998a50f7f89f39fa821b75665" + bytes: 342 + - path: "encoding/tests/test_output_3.txt" + sha256: "b3b1cd8748b7b90d3c6be6da3f786f12e4d70be073bd445ea162dfad4dc01a64" + bytes: 3313 + - path: "encoding/tests/test_output_4.txt" + sha256: "60e1643840ba9e4aeede450feb7b0498fa66ee24e4a939d48855ce04ec6fc375" + bytes: 2552 + - path: "generation_config.json" + sha256: "5fccff80f55a4d455bbe516bdd552edf3e9623df95e99fbf2a3c3389fdf91af0" + bytes: 170 + - path: "inference/README.md" + sha256: "b9fe8027f91f2160b46582b61eaaf7b9873b3da36ee9f98bc76a900e7c1fed94" + bytes: 951 + - path: "inference/config.json" + sha256: "c90861f3d10a9e4ef5954f8f1a34c529d480da1c5799f84660028f4e38e14e71" + bytes: 1162 + - path: "inference/requirements.txt" + sha256: "857e0b8b58e41cabe16e55bf4ab7ff791677c53b25f0f3e104ef85227cd11eab" + bytes: 92 + - path: "model-00001-of-00048.safetensors" + sha256: "f3668ba4cccf1ca6a7eb84e888fb92c1cdc7204d472ba9db771e6fd3abf6b874" + bytes: 1059061856 + - path: "model-00002-of-00048.safetensors" + sha256: "77b26c939a0e25b3113c8d6bb04e1901a748bd4a7d2589e3bfdaabdf1e9bba14" + bytes: 3566321192 + - path: "model-00003-of-00048.safetensors" + sha256: "412abf4c906faadc221ef0cb50f90fe20bde8454a08ad4dc2364b6b79e7fda5c" + bytes: 3566321192 + - path: "model-00004-of-00048.safetensors" + sha256: "9610f56bc587fb0ff9a8b68a60299482ee8c433fe5b5587e4257aca98add4a2e" + bytes: 3596229272 + - path: "model-00005-of-00048.safetensors" + sha256: "f87a5ac7b8becc31f9c3169afd3a6f33fb82b4af9e21022e3755a10bc28f0180" + bytes: 3568768976 + - path: "model-00006-of-00048.safetensors" + sha256: "4a4f3764e3fc772b9fba67f0a44ef68e18f178b6f00faa80b75db549e51894cd" + bytes: 3590024776 + - path: "model-00007-of-00048.safetensors" + sha256: "df81bb80e27a689e01fa579eebd6499f86e0b6105f7fea18961aa5eebbbee9bc" + bytes: 3568768976 + - path: "model-00008-of-00048.safetensors" + sha256: "224968d2b27f8669365ec08657a768dfec40da0585f85f302a31495931f6a526" + bytes: 3590024776 + - path: "model-00009-of-00048.safetensors" + sha256: "04d69ef1071fff8721c62968c200a5583122b59b015e9ef9b2978bfed271b2b7" + bytes: 3568768976 + - path: "model-00010-of-00048.safetensors" + sha256: "627145f4ebeb1cc3f5bdd03416b8cb7370b3c96974853cbeb8e5516ad5713e49" + bytes: 3590024776 + - path: "model-00011-of-00048.safetensors" + sha256: "e4b8e601dcbebe902e0102e7b098b670a121cb8b9564dd719fc41d782c8416e0" + bytes: 3568768976 + - path: "model-00012-of-00048.safetensors" + sha256: "64ed4e5f6126ba029c462c9d5fca0fc907c5f855b4ba01194d79560f6db16e42" + bytes: 3590026352 + - path: "model-00013-of-00048.safetensors" + sha256: "8dfe199d07c07ddd141c2c0136a2237f1161250a1a03ebe8deaabac93440da1d" + bytes: 3568770544 + - path: "model-00014-of-00048.safetensors" + sha256: "45db2f540f825f92453c50335e49aede58cca56bc578d1787c12a0fbca6593e5" + bytes: 3590026352 + - path: "model-00015-of-00048.safetensors" + sha256: "5810381a0f05b7381c002d299ed6ac19e42eba8070dd17e2703546944d84f292" + bytes: 3568770544 + - path: "model-00016-of-00048.safetensors" + sha256: "e0530b7024771b0ce2df9b40bcc2232578f3300178487ec216863b0b2835617b" + bytes: 3590026352 + - path: "model-00017-of-00048.safetensors" + sha256: "ed11130247118b185ade893c0109bad896dd394cb1e066ce4fce044176261d94" + bytes: 3568770544 + - path: "model-00018-of-00048.safetensors" + sha256: "e393fea96da2a3414ef089354fc32e1c8891954de40958c84d2c2ecf80365b25" + bytes: 3590026352 + - path: "model-00019-of-00048.safetensors" + sha256: "a74ca4d3e8e82ce20c458bb8b1900110b753793ad4c58d08f38995f719c616f7" + bytes: 3568770544 + - path: "model-00020-of-00048.safetensors" + sha256: "9f556769926e60309e8defe45ab59fc8b26ae460d30c190cd746a3d78c11e2c2" + bytes: 3590026352 + - path: "model-00021-of-00048.safetensors" + sha256: "1671cce7f90d781f796b5ca6bf32dd1aeb740abcb2735e41ffd28f62485ce005" + bytes: 3568770544 + - path: "model-00022-of-00048.safetensors" + sha256: "decd67a4bd97a75fa36861d2ad3067afeefa6a04a20da997fe6c19f171e70132" + bytes: 3590026352 + - path: "model-00023-of-00048.safetensors" + sha256: "c61a3e179cdbee19bfb8cbc4e111928ce2f1e1f0f4729d7c0cd5634354a4689d" + bytes: 3568770544 + - path: "model-00024-of-00048.safetensors" + sha256: "fc27aeb4233534f6f7781dcfe57127a3908ae10fc025c5d86dc0682057f8b2fe" + bytes: 3590026352 + - path: "model-00025-of-00048.safetensors" + sha256: "a66b6b8d5821b68f5b511e4f91e12025cd07d0fa6d0b71e722d825a2d6d878ca" + bytes: 3568770544 + - path: "model-00026-of-00048.safetensors" + sha256: "657b89314fbaf6eee4acce24b3baf7e5fd2c5986a96ad85b08d90539cde869fe" + bytes: 3590026352 + - path: "model-00027-of-00048.safetensors" + sha256: "fb01f21a0da0446b0bdf25a127ab19a6b06006acd8735f06a9ebfe34423fd7f5" + bytes: 3568770544 + - path: "model-00028-of-00048.safetensors" + sha256: "b2fd5cbbb639f16e673bc484e5cca16b52a58bf2ab4bd62592e0c5408712ad7c" + bytes: 3590026352 + - path: "model-00029-of-00048.safetensors" + sha256: "9ec2fdf900275daeac0980490c5c731cc7868b151ce1de5698f48418de4fa5f0" + bytes: 3568770544 + - path: "model-00030-of-00048.safetensors" + sha256: "9ed3c317bf967d32133ad3a068ee4c56aae9784bc8b7da694482437f37dc1782" + bytes: 3590026352 + - path: "model-00031-of-00048.safetensors" + sha256: "d5078c3fca3e6370043606ead7856e0b8fe67a9aab52c415769f29934c4d7f5d" + bytes: 3568770544 + - path: "model-00032-of-00048.safetensors" + sha256: "163653848f002718d3deaa6ce48885483fc1f2e12e50e44a47477c73ccd91393" + bytes: 3590026352 + - path: "model-00033-of-00048.safetensors" + sha256: "f2cffd43f2a5f491f4691f8694e6bc08239158e143ae7063dc04f0eb0259214a" + bytes: 3568770544 + - path: "model-00034-of-00048.safetensors" + sha256: "0f94945121474cfdb6a9ab175914d3811ffbf08e6cc54082e14e473c755d18d8" + bytes: 3590026352 + - path: "model-00035-of-00048.safetensors" + sha256: "9cb6a316989f7c7385e3ec2bd42ffe766ee126c70ef3466742849982ee1b0f0f" + bytes: 3568770544 + - path: "model-00036-of-00048.safetensors" + sha256: "7e6761421fe944c2143eb897b983085891b421c148c6c17fb5cd8eaa9bdaa497" + bytes: 3590026352 + - path: "model-00037-of-00048.safetensors" + sha256: "a59d662f1143596d56c452a1230b717ce43edf678207398c573a2503b0f72c91" + bytes: 3568770544 + - path: "model-00038-of-00048.safetensors" + sha256: "137fa617a74ba8e73fd76bb1010c7a85d791aaa150006cb66faa04a83e9e730f" + bytes: 3590026352 + - path: "model-00039-of-00048.safetensors" + sha256: "a29af1aa519d7ce726235ea2c2b38146d756290cda7a82d90c4d4438155b53e4" + bytes: 3568770544 + - path: "model-00040-of-00048.safetensors" + sha256: "8bc93d8a7d1987dc86b14e22b1d8f42ec31da92c56edd9f312daf43f33a6a206" + bytes: 3590026352 + - path: "model-00041-of-00048.safetensors" + sha256: "fd312e7fdd6cb5796df356a7f0314f124851dc149991e9fb02c5bed45cc4ba05" + bytes: 3568770544 + - path: "model-00042-of-00048.safetensors" + sha256: "4d19bf368083c9a183cb0849f316ec17b62f859ca824c0586e779657efb6e6a6" + bytes: 3590026352 + - path: "model-00043-of-00048.safetensors" + sha256: "b7103842ceb70848f9804f55c193d6a57f43174a587cba42b61b5c1bc4e1303d" + bytes: 3568770544 + - path: "model-00044-of-00048.safetensors" + sha256: "422d3889fa20c238b7f97464c14df0bcf3328f189c294f41a3a334421dc560c7" + bytes: 3590026352 + - path: "model-00045-of-00048.safetensors" + sha256: "a5be6aed7b84fc87ec42b5d24ba0b0d67f253a3906fcd99c13f4f7be5958fc00" + bytes: 1059332516 + - path: "model-00046-of-00048.safetensors" + sha256: "5db924ca907e0d93acd975bd5079c3662717f9ac709f23d079bd8f816d29d9dd" + bytes: 3610455184 + - path: "model-00047-of-00048.safetensors" + sha256: "62816173f9f6e136b20b48e3b6f16613ac9ea02b5603f636928b253244a548bd" + bytes: 3560111960 + - path: "model-00048-of-00048.safetensors" + sha256: "cc43742bd24ae6bcdea343a91442f6f66aed2cfebcc6b235470204851ce2f8a9" + bytes: 3692775244 + - path: "model.safetensors.index.json" + sha256: "98efab455cf08dfbbbaaba6f570e1bf10bf927d2b4c3c453a59c2f6f0e3be92b" + bytes: 5602871 + - path: "tokenizer.json" + sha256: "8f9f37ca37fdc4f5fd36d5cf4d3b0e8392edb4e894fd10cc0d70b4957c8633cf" + bytes: 6367146 + - path: "tokenizer_config.json" + sha256: "6ac8c8dc065ed118161d02dd532749ae3f52c243deac27872134fae2f50d8547" + bytes: 801 hardware_class: min_vram_gb_q4: 0 min_vram_gb_fp16: 0 @@ -65,9 +267,10 @@ priority: high collection: approved_by: "grok" approved_at: "2026-09-13" - downloaded_at: null - downloaded_by: "" - storage_path: "" + downloaded_at: "2026-09-13T23:18:26Z" + downloaded_by: "fi-reserve-collect (railiance01)" + storage_path: "s3://railiance-fi-open-weight-reserve/strategic/deepseek-ai__DeepSeek-V4-Flash-0731/7872f01b1d1fe23eabc4c98b48bffcef5a386062/" + manifest: "s3://railiance-fi-open-weight-reserve/manifests/deepseek-ai__DeepSeek-V4-Flash-0731/7872f01b1d1fe23eabc4c98b48bffcef5a386062/MANIFEST.json" brief_refs: - briefs/2026/08/2026-08-03.md - briefs/2026/09/2026-09-13.md @@ -100,3 +303,7 @@ history: event: collecting by: codex detail: "Replaced WSL snapshot staging with diskless Railiance service fi-reserve-collect. 64 MiB parts, 512 MiB memory cap, no swap. Source pinned to 7872f01b1d1fe23eabc4c98b48bffcef5a386062; 67 selected files / 166898547054 bytes. Destination strategic/deepseek-ai__DeepSeek-V4-Flash-0731//. Complete manifest pending; not collected or restore-verified." + - at: "2026-09-22" + event: collected + by: claude + detail: "Transfer finished 2026-09-13T23:18:26Z; complete MANIFEST.json published. Re-checked 2026-09-22: 67/67 objects, 166898547054 bytes, all GLACIER; 48 LFS SHA256 source digests equal computed SHA256, 19 git-sha1 files verified in-stream; HEAD size/ETag/version match manifest for every object. No restore readback yet, so not verified." diff --git a/workplans/FI-WP-0004-operational-loop-and-scaleway-reserve.md b/workplans/FI-WP-0004-operational-loop-and-scaleway-reserve.md index 9670d5d..eac3ad0 100644 --- a/workplans/FI-WP-0004-operational-loop-and-scaleway-reserve.md +++ b/workplans/FI-WP-0004-operational-loop-and-scaleway-reserve.md @@ -4,13 +4,13 @@ type: workplan title: "Make the sensing loop durable and move the reserve to Scaleway" domain: agents repo: freedom-intelligence -status: active +status: finished flavor: implementation owner: grok topic_slug: freedom-intelligence priority: high created: "2026-09-13" -updated: "2026-09-13" +updated: "2026-09-22" depends_on: - FI-WP-0001 - FI-WP-0002 @@ -224,7 +224,7 @@ Operator / railiance-platform: ```task id: FI-WP-0004-T09 -status: progress +status: done priority: medium state_hub_task_id: "304816de-5065-5928-b9bd-da1b5019c3d4" ``` @@ -239,6 +239,14 @@ See `docs/streaming-reserve.md`. Transfer completion and final manifest remain pending. Update catalog `collected` only after the manifest is complete; `verified` requires restore/readback. +**2026-09-22:** Done. The service finished 2026-09-13T23:18:26Z and published +a complete MANIFEST.json. Re-check: 67/67 objects, 166,898,547,054 bytes, +all GLACIER, no open multipart uploads. The 48 LFS SHA256 source digests +match the computed SHA256, and HEAD size, ETag and version match the manifest +for every object. The catalog is `collected` with the S3 prefix as +`storage_path`. Restore readback (`verified`) is a follow-up outside this +task. + **Done when:** catalog `storage_path` is the S3 prefix and checksums match. ## Acceptance (workplan-level) @@ -249,7 +257,7 @@ pending. Update catalog `collected` only after the manifest is complete; - [x] Executor publishes to origin or fails the day - [x] V4-Flash cataloged at the real size - [x] Bucket live (T08) -- [ ] First S-tier object on Scaleway (T09) +- [x] First S-tier object on Scaleway (T09) ## Out of scope