Make the sensing loop durable and pin the reserve to Scaleway

The weekday brief only counts when the file is on origin/main. Hub
events without git objects are failures. The open-weight SoT moves
from the VAULT 850 GiB quota to a dedicated Scaleway bucket (One Zone
IA for R, Glacier for S) so V4-Flash and K3 are economically in-scope.

Catalogs the real DeepSeek-V4-Flash-0731 (MIT ~167 GiB MoE, not 12B)
and nominates Qwen3.8-27B. Bucket create remains FI-WP-0004-T08.

Assistant: grok
Assistant-Session: 01a09c6a-1cfc-75b1-a78d-c13eaf22241d
This commit is contained in:
tegwick 2026-09-13 22:45:43 +02:00
parent 8832652ad3
commit a78187c33e
18 changed files with 1125 additions and 226 deletions

View file

@ -23,7 +23,7 @@ Standing lenses: **A** frontier & price · **B** edge/local open · **C** homela
| [`briefs/`](briefs/) | Daily briefs + template | | [`briefs/`](briefs/) | Daily briefs + template |
| [`inventory/`](inventory/) | Model catalog & collection policy | | [`inventory/`](inventory/) | Model catalog & collection policy |
| [`workplans/`](workplans/) | Setup workplans (incl. activity-core) | | [`workplans/`](workplans/) | Setup workplans (incl. activity-core) |
| [`activity-definitions/`](activity-definitions/) | Cron definition for daily brief (disabled until FI-WP-0002) | | [`activity-definitions/`](activity-definitions/) | Weekday brief schedule (enabled; completion = origin/main) |
--- ---
@ -31,23 +31,26 @@ Standing lenses: **A** frontier & price · **B** edge/local open · **C** homela
| ID | Title | Status | | ID | Title | Status |
| -- | ----- | ------ | | -- | ----- | ------ |
| [FI-WP-0001](workplans/FI-WP-0001-lab-operating-foundation.md) | Lab operating foundation | active (storage pin open) | | [FI-WP-0001](workplans/FI-WP-0001-lab-operating-foundation.md) | Lab operating foundation | done |
| [FI-WP-0002](workplans/FI-WP-0002-activity-core-daily-research.md) | Activity-core daily research rhythm | active (resolver/ops open) | | [FI-WP-0002](workplans/FI-WP-0002-activity-core-daily-research.md) | Activity-core daily research rhythm | finished |
| [FI-WP-0003](workplans/FI-WP-0003-seed-open-weight-reserve.md) | Seed open-weight reserve | active (catalog seeded) | | [FI-WP-0003](workplans/FI-WP-0003-seed-open-weight-reserve.md) | Seed open-weight reserve | finished |
| [FI-WP-0004](workplans/FI-WP-0004-operational-loop-and-scaleway-reserve.md) | Durable loop + Scaleway reserve | **active** |
--- ---
## Collection plan (bulk reserve) ## Collection plan (bulk reserve)
**Storage:** `D:\vault\coulomb\freedom-intelligence\` (VAULT HD) · soft quota **850 GiB** **SoT:** Scaleway Object Storage `nl-ams` · bucket `railiance-fi-open-weight-reserve`
**Gate:** **€15 / month** · R = One Zone IA · S = Glacier
**Staging:** VAULT HD `D:\vault\coulomb\freedom-intelligence\`
**Policy:** store the **most capable open weights** even if unrunnable today **Policy:** store the **most capable open weights** even if unrunnable today
(`docs/decisions/2026-07-24-nas-strategic-reserve.md`). (`docs/decisions/2026-09-13-scaleway-object-reserve.md`).
| Tier | Examples | | Tier | Examples |
| ---- | -------- | | ---- | -------- |
| **R — runnable spine** | Qwen3-8B, Llama-3.2-3B-Instruct, BGE-M3, R1-Distill-14B, nomic-embed | | **R — runnable spine** | Qwen3-8B, Llama-3.2-3B-Instruct, BGE-M3, R1-Distill-14B, nomic-embed |
| **S — strategic** | DeepSeek-V3 (class), DeepSeek-R1 full, Llama-3.3-70B / Qwen3-72B dense | | **S — strategic** | **DeepSeek-V4-Flash-0731** (first pull), Kimi K3, V3/R1 full, 70B-class |
| **W — optional** | 14B/32B upgrades, reranker, … | | **W — optional** | Qwen3.8-27B, 14B/32B upgrades, reranker |
Plan detail: `research/2026-07-24-nas-strategic-collection-plan.md` · catalog: `inventory/catalog/`. Plan detail: `research/2026-07-24-nas-strategic-collection-plan.md` · catalog: `inventory/catalog/`.
@ -58,7 +61,7 @@ Plan detail: `research/2026-07-24-nas-strategic-collection-plan.md` · catalog:
```text ```text
freedom-intelligence/ freedom-intelligence/
├── INTENT.md / SCOPE.md / README.md ├── INTENT.md / SCOPE.md / README.md
├── activity-definitions/ # fi-daily-research-brief (enabled: false) ├── activity-definitions/ # fi-daily-research-brief (enabled; origin publish required)
├── briefs/ # daily deltas ├── briefs/ # daily deltas
├── research/ # deep surveys (baseline lives here) ├── research/ # deep surveys (baseline lives here)
├── inventory/catalog/ # YAML candidates / collected ├── inventory/catalog/ # YAML candidates / collected
@ -79,8 +82,14 @@ cp briefs/_template.md briefs/$(date +%Y/%m)/$(date +%Y-%m-%d).md
# follow docs/daily-brief-playbook.md # follow docs/daily-brief-playbook.md
``` ```
**Do not** bulk-download models until `docs/backup-storage-policy.md` path/quota **Do not** bulk-download S-tier models until the Scaleway bucket is live
are pinned (FI-WP-0001-T04). (FI-WP-0004-T08). Stage on VAULT; never into this git tree.
**Durability check** (hub events vs git files):
```bash
python3 scripts/verify_brief_durability.py --require-hub
```
--- ---

View file

@ -91,17 +91,17 @@ Git holds briefs, schemas, policies, and catalog metadata. Model weight blobs st
| Area | Status | | Area | Status |
| ---- | ------ | | ---- | ------ |
| Overall | **Operable lab** — foundation + **live** weekday brief schedule + R-spine reserve | | Overall | **Repairing the operating loop** — foundation exists; origin publication + Scaleway SoT in FI-WP-0004 |
| INTENT / SCOPE | Drafted | | INTENT / SCOPE | Live; SCOPE current-state refreshed 2026-09-13 |
| Baseline survey | `research/2026-07-24-baseline-field-survey.md` | | Baseline survey | `research/2026-07-24-baseline-field-survey.md` |
| Briefs | 2026-07-24, 07-28, **08-03** (week catch-up); playbook + recurrence ops | | Briefs | Origin last 2026-08-14; catch-up **2026-09-13**; weekday schedule still enabled |
| Backup storage | **VAULT HD** `D:\vault\coulomb\freedom-intelligence\`; soft quota 850 GiB | | Backup storage | **Scaleway `nl-ams`** (bucket planned); VAULT HD is staging; €15/mo gate |
| Inventory | R verified; **Kimi K3** S1 approved+deferred (~1454 GiB); other S deferred; W candidate | | Inventory | R verified on VAULT cache; **V4-Flash** S1 approved (not collected); K3 Glacier-affordable; Qwen3.8-27B candidate |
| Reserve policy | Capability-first — not gated by current VRAM | | Reserve policy | Capability-first — not gated by current VRAM; **not** gated by 850 GiB |
| Activity-core | Definition **enabled on railiance**; Temporal schedule weekdays 07:30 Berlin | | Activity-core | Definition **enabled**; completion = brief on `origin/main` |
| Executor | rein-aharness `fi-research-brief` via railiance user timer 07:35 (not workstation cron) | | Executor | rein-aharness `fi-research-brief` **must push** or the day fails |
| State Hub | Repo under **agents**; workplans finished + hub-indexed | | State Hub | Repo under **agents**; `fi_daily_brief` is not evidence without the git file |
| Workplans | FI-WP-0001…0003 **finished** | | Workplans | FI-WP-0001…0003 finished; **FI-WP-0004 active** |
--- ---
@ -121,12 +121,14 @@ Git holds briefs, schemas, policies, and catalog metadata. Model weight blobs st
1. ✅ `INTENT.md` / `SCOPE.md` / foundation docs 1. ✅ `INTENT.md` / `SCOPE.md` / foundation docs
2. ✅ Brief template + playbook + sources allowlist 2. ✅ Brief template + playbook + sources allowlist
3. ✅ Baseline field survey + collection recommendations 3. ✅ Baseline field survey + collection recommendations
4. ✅ Catalog candidates (P0/P1) 4. ✅ Catalog candidates (P0/P1) + V4-Flash correction
5. ✅ ActivityDefinition draft (disabled) 5. ✅ ActivityDefinition enabled
6. ⬜ Pin backup storage path and capacity budget (FI-WP-0001-T04) 6. ✅ Pin backup storage (VAULT staging 2026-07-28; **Scaleway SoT 2026-09-13**)
7. ⬜ activity-core resolver + enable schedule (FI-WP-0002) 7. ✅ activity-core resolver + weekday schedule
8. ⬜ Approve/download P0 reserve (FI-WP-0003) 8. ✅ R-spine verified on VAULT cache (FI-WP-0003)
9. ⬜ First daily delta brief (manual or after automation) 9. ⬜ Origin publication proven end-to-end on railiance (FI-WP-0004-T04 deploy)
10. ⬜ Scaleway bucket live (FI-WP-0004-T08)
11. ⬜ First S collect: V4-Flash (FI-WP-0004-T09)
--- ---

View file

@ -18,9 +18,10 @@ context_sources:
bind_to: context.fi_brief bind_to: context.fi_brief
# Resolver (FI-WP-0002-T03, activity-core): expected shape # Resolver (FI-WP-0002-T03, activity-core): expected shape
# {items: [{kind: "daily_brief", due: bool, date: "YYYY-MM-DD", # {items: [{kind: "daily_brief", due: bool, date: "YYYY-MM-DD",
# last_run_at}]} — due is true when no fi_daily_brief progress event with # last_run_at}]} — due is true when no *successful origin-published*
# detail.repo=freedom-intelligence exists for today (Europe/Berlin). # fi_daily_brief exists for today (Europe/Berlin).
# Executing session must record that event on completion (idempotence). # Completion event is valid only with detail.pushed=true and the file on
# origin/main. Local-only commits must not clear due (FI-WP-0004).
--- ---
# Freedom Intelligence Daily Research Brief # Freedom Intelligence Daily Research Brief
@ -46,8 +47,10 @@ action:
description: > description: >
Produce briefs/YYYY/MM/YYYY-MM-DD.md per docs/daily-brief-playbook.md and Produce briefs/YYYY/MM/YYYY-MM-DD.md per docs/daily-brief-playbook.md and
briefs/_template.md. Cite primary sources. Flag collection candidates. briefs/_template.md. Cite primary sources. Flag collection candidates.
On completion post State Hub progress event_type=fi_daily_brief with Commit AND push to origin/main. Post fi_daily_brief only if
detail.repo=freedom-intelligence and detail.date. detail.pushed=true and detail.origin_sha is set. If push fails, post
executor_run ok=false so due stays true. Do not re-announce cataloged
models. Do not invent parameter counts.
target_repo: freedom-intelligence target_repo: freedom-intelligence
priority: medium priority: medium
labels: ["freedom-intelligence", "research-brief", "automated"] labels: ["freedom-intelligence", "research-brief", "automated"]

View file

@ -0,0 +1,81 @@
---
date: 2026-09-13
timezone: Europe/Berlin
author: grok
status: final
sources_checked:
- docs/sources-allowlist.md
- inventory/RESERVE-STATUS.md
- huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731
- huggingface.co/Qwen/Qwen3.8-27B (card identity via public trackers)
- https://www.scaleway.com/en/pricing/storage/
---
# Freedom Intelligence Daily Brief — 2026-09-13
> **Month catch-up.** Last file on `origin/main` is 2026-08-14. Hub
> `fi_daily_brief` events exist for 2026-09-07…11 with `committed=True`
> and no git objects. Those days are **not** recovered here. This file
> is a field + ops delta from 2026-08-14, not a resurrection of the
> missing executor commits.
## Headline deltas
- **Loop failure:** sensing ran locally on railiance and cleared `due` without publishing to origin. Rein-aharness dropped `git push` on 2026-09-04 as an integrity guard; FI now re-grants origin publication as a named completion requirement (FI-WP-0004).
- **DeepSeek-V4-Flash-0731 is not 12B.** Official card: MIT, 284B base / ~13B active (304B with DSpark), ~167 GiB HF tree, 2026-07-31. Automated briefs 08-04…14 re-announced it as a new 12B dense model (license flipped MIT/Apache) and 08-07 claimed it was already on disk. Catalog now records the real model as **S, approved, not collected**.
- **Qwen3.8-27B** (Apache-2.0 dense multimodal, 2026-08-14) and **Qwen3.8-2.4T-A95B** (custom Max licence, multi-TB, ~08-12) shipped in the hole. 27B is a W/R candidate; Max is S-watch only until euro budget is live.
- **Reserve facility:** VAULT HD 850 GiB quota is retired as SoT. Scaleway `nl-ams` object storage is the reserve (One Zone IA for R, Glacier for S). Kimi K3 (~1454 GiB) becomes a few euros/month on Glacier instead of a quota wall.
- **Next pull** after the bucket exists: V4-Flash official tree into `strategic/` Glacier — it always fitted even the old 850 GiB quota.
## Frontier & commercial (axis A)
| Item | Delta | Sources | Lab relevance |
| ---- | ----- | ------- | ------------- |
| DeepSeek-V4-Flash-0731 | Official open MIT MoE; API `deepseek-v4-flash`. Card scores Terminal Bench 2.1 82.7. | [HF card](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731), [arXiv 2606.19348](https://arxiv.org/abs/2606.19348) | Closed-bar comparison via llm-connect; open identity is the S pull |
| Qwen3.8-Max family | API GA early Aug; open 2.4T-A95B under **bespoke** licence (revenue threshold), not Apache | Qwen card / trackers (verify licence text at pin) | Do not collect the 2.4T tree until T08 budget + licence read |
| GLM-5.3-Flash (late Aug) | 320B-A18B, MIT, 1M ctx, reported $0.15 / $0.50 per MTok | Z.ai release notes (verify) | Watch vs V4-Flash as MIT S alternative; not displacing V4-Flash as first pull |
## Edge / local / open (axis B)
| Item | Delta | Sources | Lab relevance |
| ---- | ----- | ------- | ------------- |
| DeepSeek-V4-Flash-0731 size class | ~167 GiB official tree, ~13B active — **S, not R** | HF file listing ~167 GB; card “304B params” | First Scaleway Glacier collect |
| Qwen3.8-27B | Dense multimodal Apache-2.0, ~27B, 2026-08-14 — same calendar day as last origin brief | [HF Qwen/Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B) | Candidate R-spine upgrade; cataloged `candidate` |
| Muse Glimmer 30B (Meta, ~08-10) | Apache-2.0 dense multimodal, ~2432 GB class | Meta/HF trackers (verify card) | Watch; do not pull until V4-Flash and 27B decision |
| Kimi K3 | Unchanged identity; capacity objection is now euros not GiB | existing catalog | Glacier ~€3.69/mo; still deferred until V4-Flash lands |
## Training & specialization (axis C)
| Item | Delta | Sources | Lab relevance |
| ---- | ----- | ------- | ------------- |
| Qwen3.8-27B as FT base | Dense 27B Apache-2.0 is a more serious QLoRA target than 8B if T2T3 VRAM exists | Qwen3.8-27B card | Hardware envelope hosts still TBD — do not pretend we can FT it today |
| No new homelab recipe displacement | Unsloth / TRL / Axolotl still the working set from baseline | allowlist | No stack change |
## Harness & fleet (axis D)
| Item | Delta | Sources | Lab relevance |
| ---- | ----- | ------- | ------------- |
| FI executor publication | 2026-09-04 rein-aharness removed `git push`; hub still cleared `due` | `rein-aharness` REINAH-WP-0003, hub events 2026-09-07…11 | **Broken durable memory.** FI-WP-0004 re-grants origin push and verifier |
| V4-Flash agentic stack | Official vLLM DSpark + SGLang recipes on the card | HF card “How to Run” | Axis D watch; serve is not this repo |
## Collection candidates
| id | org | name | priority | reason | approx size | license |
| -- | --- | ---- | -------- | ------ | ----------- | ------- |
| deepseek-ai__DeepSeek-V4-Flash-0731__strategic | deepseek-ai | DeepSeek-V4-Flash-0731 | high | Corrected S1; MIT; ~167 GiB; first Scaleway pull | ~167 GiB | MIT |
| Qwen__Qwen3.8-27B__candidate | Qwen | Qwen3.8-27B | medium | Apache-2.0 dense 27B R-upgrade | ~56 GiB class | Apache-2.0 |
Already in catalog, **not** new: Kimi K3, DeepSeek-V3, Qwen3-8B, R1-Distill-14B, BGE-M3, nomic-embed.
## Lab implications
- Finish FI-WP-0004: origin publish in the executor, durability verifier in CI/ops, Scaleway bucket (T08), then collect V4-Flash (T09).
- Do not treat hub `fi_daily_brief` as evidence the brief exists until `scripts/verify_brief_durability.py` is green.
- Recovering 2026-09-07…11 files from the railiance clone is optional; they were produced by the same failing loop and are not trusted.
- Hardware envelope hosts are still TBD — R-upgrade of Qwen3.8-27B waits on that as much as on disk.
## Notes / caveats
- Parameter counts and €/GB from primary cards/pricing pages today; re-measure at pull.
- Qwen3.8-Max open tree licence is **not** Apache-2.0; do not copy the 27B licence onto it.
- `DeepSeek-V4-Flash-0731-12B` is not a real repo id.

View file

@ -1,17 +1,19 @@
# Backup storage policy — open-weight model reserve # Backup storage policy — open-weight model reserve
**Status:** facility + path pinned (workstation VAULT HD) **Status:** facility pin **Scaleway Object Storage** (2026-09-13)
**Related:** `INTENT.md`, `SCOPE.md`, `inventory/collection-policy.md`, **Related:** `INTENT.md`, `SCOPE.md`, `inventory/collection-policy.md`,
`docs/decisions/2026-07-24-nas-strategic-reserve.md` `docs/decisions/2026-09-13-scaleway-object-reserve.md`
**Adjacent:** `disaster-control` BackupPolicy (platform backups — different concern) **Supersedes storage pin:** 2026-07-28 VAULT HD (now staging only)
**Adjacent:** `disaster-control` BackupPolicy and `rapp-postgres` WAL
archive — **different buckets, different lifecycles**
--- ---
## Purpose ## Purpose
Define where Freedom Intelligence stores **open-weight model blobs** (and justified Define where Freedom Intelligence stores **open-weight model blobs** (and
companions), how much capacity we allow, and how this reserve relates to other justified companions), how much we allow ourselves to spend, and how this
lab backup facilities. reserve relates to other lab backup facilities.
Git never stores weight tensors. Git stores: Git never stores weight tensors. Git stores:
@ -25,16 +27,16 @@ Git never stores weight tensors. Git stores:
| Requirement | Policy | | Requirement | Policy |
| ----------- | ------ | | ----------- | ------ |
| **Class** | Backup / bulk durable storage — **not** hot cluster PVCs or app disks | | **Class** | Backup / bulk durable object storage — **not** hot cluster PVCs |
| **Facility** | Workstation VAULT HD (`D:`) under `D:\vault\coulomb\` (interim pin 2026-07-28) | | **Facility** | Scaleway Object Storage `nl-ams` via `reef-storage` |
| **Durability** | Survives WSL rebuilds; weights live on the VAULT volume, not the git/WSL root | | **Durability** | Provider 11-nines claim; independent of Host Europe `railiance01` and of the workstation HD |
| **Performance** | Sequential read for restore/training pull is enough; low latency not required | | **Performance** | Sequential restore for training/inference pull is enough |
| **Access** | Operator and approved lab hosts only; not a public mirror | | **Access** | Operator and approved lab hosts; not a public mirror |
| **Separation** | Do not co-mingle with age-encrypted operational backups (Forgejo dumps, k3s state) without a clear subdirectory and different lifecycle rules | | **Separation** | **Never** the postgres backup bucket (`railiance-platform-pg-backup` expires at 30 days) |
Platform backup paths such as `/opt/backup/railiance/{infra,cluster}/` and Platform backup paths such as `/opt/backup/railiance/{infra,cluster}/` and
`~/.cache/railiance/backups/` are **operational recovery** lanes. Model weights `~/.cache/railiance/backups/` are **operational recovery** lanes. Model
belong on the **VAULT model tree**, not on railiance hot disks. weights belong in the FI object bucket.
--- ---
@ -42,56 +44,33 @@ belong on the **VAULT model tree**, not on railiance hot disks.
| Field | Value | | Field | Value |
| ----- | ----- | | ----- | ----- |
| **Host / facility** | Workstation-attached bulk HD **VAULT** (Windows `D:`) | | **Facility** | Scaleway Object Storage, region `nl-ams` |
| **Raw capacity** | ~1.86 TB total (~1.76 TB free as of 2026-07-28 pin) | | **Endpoint** | `https://s3.nl-ams.scw.cloud` |
| **Base path (Windows)** | `D:\vault\coulomb\freedom-intelligence\` | | **Bucket** | `railiance-fi-open-weight-reserve` (planned until FI-WP-0004-T08) |
| **Base path (WSL, when D: automounted)** | `/mnt/d/vault/coulomb/freedom-intelligence/` | | **Attributes** | `reef:storage/substrate/object-stores/fi-open-weight-reserve.yaml` |
| **Layout under base** | See [On-disk layout](#on-disk-layout) | | **Credentials** | OpenBao / `railiance-platform` — never git |
| **Mount on lab hosts** | Local attach on operator workstation; WSL via drvfs `D:``/mnt/d` when enabled |
| **Credentials** | Local filesystem ACL on the workstation HD; never commit secrets |
**Interim note:** Earlier policy assumed a dedicated ~1 TB NAS. Operator chose this **Local staging (not SoT):** workstation VAULT HD
VAULT volume under `D:\vault\coulomb\` for now. Soft quota **850 GiB** for the `D:\vault\coulomb\freedom-intelligence\` /
model tree still applies (self-imposed discipline; disk is larger). `/mnt/d/vault/coulomb/freedom-intelligence/` for `huggingface_hub`
downloads and an optional R-spine cache.
Do **not** bulk-download multi-GB models into this git workspace or into hot Do **not** bulk-download multi-GB models into this git workspace or into
root filesystems (`/` on WSL, `C:`). hot root filesystems (`/` on WSL, `C:`, railiance PVCs).
### Path shape (pinned) ### Prefix shape (pinned)
```text ```text
D:\vault\coulomb\freedom-intelligence\ # Windows s3://railiance-fi-open-weight-reserve/
/mnt/d/vault/coulomb/freedom-intelligence/ # WSL when D: mounted ├── models/{org}__{name}/{revision}/ # R / W (One Zone IA)
├── models/ ├── strategic/{org}__{name}/{revision}/ # S (Glacier)
│ └── {org}__{name}/ ├── companions/
│ └── {revision}/ ├── staging/
│ ├── blobs/ # weight files, shards └── manifests/{org}__{name}/{revision}/MANIFEST.json
│ └── MANIFEST.txt # optional local copy of hashes
├── companions/ # adapters, tokenizers only when separate
└── staging/ # incomplete downloads; not catalog-ready
``` ```
Map each `{org}__{name}/{revision}` to an inventory catalog entry. Map each `{org}__{name}/{revision}` to an inventory catalog entry.
`collection.storage_path` is the `s3://` URI.
Directories `models/`, `staging/`, and `companions/` were created 2026-07-28.
---
## On-disk layout
| Path element | Rule |
| ------------ | ---- |
| `org__name` | Hugging Face-style id with `/``__` |
| `revision` | Git commit SHA, tag, or release id used at download time |
| `blobs/` | Actual files; prefer original names from source |
| `staging/` | Incomplete transfers; purge or resume; never mark collected until complete + verified |
Optional layout tags (directory name or catalog field):
| Tag | Meaning |
| --- | ------- |
| runnable | Fits current run envelope (R tier) |
| strategic | Capability reserve; may exceed current hardware (S tier) |
--- ---
@ -99,20 +78,24 @@ Optional layout tags (directory name or catalog field):
| Parameter | Policy | | Parameter | Policy |
| --------- | ------ | | --------- | ------ |
| **Device** | VAULT HD (`D:`) — model tree under `D:\vault\coulomb\freedom-intelligence\` | | **Gate** | **Euros per month**, not GiB |
| **Soft quota (model reserve)** | **850 GiB** (self-imposed; disk is larger) | | **Soft budget** | **€15 / month** ex VAT for this bucket |
| **Hard stop** | **920 GiB** used under the freedom-intelligence base path | | **Hard backstop** | Founder Scaleway project billing alert (no provider euro hard-stop) |
| **Growth review** | When catalog total ≥ **70% soft quota (~595 GiB)** | | **Growth review** | When the running month is ≥ **70% of soft budget (~€10.50)** |
| **Per-pull threshold** | See `inventory/collection-policy.md` | | **Per-pull threshold** | See `inventory/collection-policy.md` |
| **Eviction** | Prefer drop **W** (watch) and easily re-obtained quants; protect unique **S** SOTA bases and the **R** spine | | **Eviction** | Prefer drop **W** and easily re-obtained quants; protect unique **S** SOTA bases and the **R** spine |
### Portfolio guidance on 1 TB Rough cost (2026-09-13 Scaleway list prices):
* Always keep the **runnable spine (R / former P0)** resident. | Mix | Class | ≈ € / month |
* Use remaining space for a **small number of most-capable open models (S)**, | --- | ----- | ----------: |
preferably well-provenance compressed weights when full precision would exhaust the disk. | R-spine ~45 GiB | One Zone IA | 0.54 |
* Do **not** attempt to mirror entire HF orgs. | V4-Flash ~167 GiB | Glacier | 0.42 |
* Full bf16 of multiple 600B-class models will **not** fit; prioritize top capability identities. | Kimi K3 ~1454 GiB | Glacier | 3.69 |
| Those three together | | **~4.65** |
Glacier restore is €0.009/GB — pay it when we actually need the weights
on a GPU host.
--- ---
@ -121,12 +104,14 @@ Optional layout tags (directory name or catalog field):
For every completed collection: For every completed collection:
1. Record source URL and revision in the catalog entry. 1. Record source URL and revision in the catalog entry.
2. Store checksums (`sha256` of each blob or upstream manifest digest). 2. Store checksums (`sha256` of each blob or upstream manifest digest)
in `MANIFEST.json` (local staging **and** `manifests/` prefix).
3. Record download date (UTC) and downloader identity. 3. Record download date (UTC) and downloader identity.
4. Prefer official org releases over anonymous re-uploads. 4. Prefer official org releases over anonymous re-uploads.
5. Keep license text or SPDX id in catalog; refuse unclear licenses. 5. Keep license text or SPDX id in catalog; refuse unclear licenses.
Verification: **catalog claims must match on-disk checksums** before status `collected`. Verification: **catalog claims must match object checksums** before
status `collected`.
--- ---
@ -134,11 +119,11 @@ Verification: **catalog claims must match on-disk checksums** before status `col
| Class | Retention | | Class | Retention |
| ----- | --------- | | ----- | --------- |
| **Strategic capability (S)** | Keep until explicit deprecation or displacement by a clearly stronger open successor | | **Strategic capability (S)** | Keep until explicit deprecation; Glacier |
| **Runnable spine (R)** | Keep while still the lab default for local ops / FT | | **Runnable spine (R)** | Keep while still the lab default; One Zone IA |
| **Working set** | Active experiment bases + optional one superseded revision | | **Working set** | Active experiment bases + optional one superseded revision |
| **Staging** | Max 14 days incomplete, then purge | | **Staging** | Max 14 days incomplete, then purge |
| **Deprecated / evicted** | Metadata retained in catalog; blobs may be deleted | | **Deprecated / evicted** | Metadata retained in catalog; objects may be deleted |
--- ---
@ -146,35 +131,37 @@ Verification: **catalog claims must match on-disk checksums** before status `col
| Topic | Policy | | Topic | Policy |
| ----- | ------ | | ----- | ------ |
| **At rest** | NAS default; extra age/GPG of multi-hundred-GB trees optional | | **At rest** | Scaleway provider-managed |
| **In transit** | Trusted lab network / local attach | | **In transit** | TLS to `s3.nl-ams.scw.cloud` |
| **Offsite copy** | Optional later; coordinate with disaster-control if added so model bulk does not break ops backup SLAs | | **Workstation copy** | Optional R-spine on VAULT; not a second SoT |
| **Ops backups** | Do not mix with Forgejo/k3s dumps |
--- ---
## What must not live here ## What must not live here
* Closed weights or artifacts whose terms forbid offline retention * Closed weights or artifacts whose terms forbid offline retention
* Secrets, API keys, customer data, or ungoverned training corpora with personal data * Secrets, API keys, customer data, or ungoverned training corpora
* Operational backups (databases, k3s state, Forgejo dumps) as the primary home * Operational backups (databases, k3s state, Forgejo dumps)
* Git LFS dumps of full model trees as a substitute for the NAS * Git LFS dumps of full model trees
* Objects in `railiance-platform-pg-backup`
--- ---
## Operator checklist ## Operator checklist
- [x] Choose facility: bulk durable storage for model reserve (2026-07-24 intent) - [x] Choose facility: Scaleway object storage (2026-09-13)
- [x] Set soft quota: **850 GiB** / hard stop **920 GiB** - [x] Set capacity gate: **€15 / month** soft
- [x] Pin path: **`D:\vault\coulomb\freedom-intelligence\`** (2026-07-28) - [x] Pin prefix layout above
- [x] Create `models/`, `staging/`, `companions/` - [ ] Create bucket + scoped key (FI-WP-0004-T08)
- [x] Document Windows + WSL path forms above - [ ] Smoke PUT/GET
- [x] Ensure WSL can see `D:` (`/mnt/d`) when downloads run from Linux (verified 2026-07-28) - [ ] First S collect: DeepSeek-V4-Flash-0731 (FI-WP-0004-T09)
### Pin log ### Pin log
| Date | Operator | Change | | Date | Operator | Change |
| ---- | -------- | ------ | | ---- | -------- | ------ |
| 2026-07-24 | foundation | Policy created; path unpinned | | 2026-07-24 | foundation | Policy created; path unpinned |
| 2026-07-24 | operator direction | Facility intent = 1 TB local NAS; soft quota 850 GiB; strategic unrunnable models in scope | | 2026-07-24 | operator direction | Facility intent = 1 TB local NAS; soft quota 850 GiB |
| 2026-07-28 | operator direction | **Interim pin:** workstation VAULT HD `D:\vault\coulomb\freedom-intelligence\`; layout created; soft quota retained | | 2026-07-28 | operator direction | **Interim pin:** workstation VAULT HD |
| 2026-07-28 | operator | WSL mount verified: `D:``/mnt/d` (1.9T, writable) | | 2026-09-13 | FI-WP-0004 | **SoT pin:** Scaleway `nl-ams`; VAULT demoted to staging; euro gate replaces GiB quota |

View file

@ -22,7 +22,8 @@ full resurveys.
| One brief per date | Path `briefs/YYYY/MM/YYYY-MM-DD.md` | | One brief per date | Path `briefs/YYYY/MM/YYYY-MM-DD.md` |
| Calendar | Prefer **Europe/Berlin** date in front matter | | Calendar | Prefer **Europe/Berlin** date in front matter |
| Skip empty days | Either omit file or write `*No material delta.*` in Headline deltas | | Skip empty days | Either omit file or write `*No material delta.*` in Headline deltas |
| Automation due bit | Cleared only by State Hub event `fi_daily_brief` (below) | | Automation due bit | Cleared only by State Hub event `fi_daily_brief` **after** the brief exists on `origin/main` |
| Hub without git | A `fi_daily_brief` event with no matching file is a **false completion**. Run `scripts/verify_brief_durability.py`. |
--- ---
@ -46,17 +47,20 @@ full resurveys.
4. **Collection candidates** 4. **Collection candidates**
- Use inventory field names; set priority - Use inventory field names; set priority
- Do **not** re-nominate a model already in `inventory/catalog/`
- Do **not** invent parameter counts; cite the model card or mark unverified
- If nominating: add or update `inventory/catalog/{id}.yaml` with - If nominating: add or update `inventory/catalog/{id}.yaml` with
`status: candidate` and `collection.brief_refs` `status: candidate` and `collection.brief_refs`
5. **Lab implications** 5. **Lab implications**
- Link open workplans (`FI-WP-*`) only when action is real - Link open workplans (`FI-WP-*`) only when action is real
6. **Commit** (operator policy) — brief + any catalog YAML 6. **Commit and push** — brief + any catalog YAML must land on `origin/main`.
A local-only commit does **not** complete the day.
7. **Completion evidence** (required for activity-core rhythm) 7. **Completion evidence** (required for activity-core rhythm)
Post State Hub progress (HTTP or MCP), shape: Post State Hub progress **only after** origin has the commit:
```yaml ```yaml
event_type: fi_daily_brief event_type: fi_daily_brief
@ -67,8 +71,13 @@ full resurveys.
date: "YYYY-MM-DD" date: "YYYY-MM-DD"
path: "briefs/YYYY/MM/YYYY-MM-DD.md" path: "briefs/YYYY/MM/YYYY-MM-DD.md"
collection_candidates: 0 collection_candidates: 0
pushed: true
origin_sha: "<full sha>"
``` ```
If the brief is written but push fails, post `event_type: executor_run`
with `ok: false` so `due` stays true.
Workstation API often `http://127.0.0.1:8000`; remote tunnel ports per Workstation API often `http://127.0.0.1:8000`; remote tunnel ports per
Custodian docs. See `docs/state-hub-delivery.md`. Custodian docs. See `docs/state-hub-delivery.md`.
@ -85,8 +94,10 @@ When activity-core emits `activity_task_spawn` with
1. Railiance host timer (or manual) runs 1. Railiance host timer (or manual) runs
`rein-aharness fi-research-brief --target-repo ~/freedom-intelligence` `rein-aharness fi-research-brief --target-repo ~/freedom-intelligence`
2. llm-connect drafts the brief; harness commits and posts **`fi_daily_brief`** 2. llm-connect drafts the brief; harness **commits, pushes to origin**, then
3. Idempotency: one brief file per date; resolver clears `due` on completion event posts **`fi_daily_brief`** with `pushed=true`
3. Idempotency: one brief file per date on origin; resolver clears `due` only
then. Local commit without push must not clear `due`.
Do **not** install Claude Code / Codex on railiance01 for this rhythm. Do **not** install Claude Code / Codex on railiance01 for this rhythm.
@ -116,9 +127,10 @@ make -C ~/activity-core sync-activity-definitions
| Check | Pass | | Check | Pass |
| ----- | ---- | | ----- | ---- |
| Signal | No filler paragraphs | | Signal | No filler paragraphs |
| Sources | At least one primary link per non-empty axis row | | Sources | At least one primary URL per non-empty axis row |
| Honesty | Benchmarks marked directional; prices “verify before budget” | | Honesty | Benchmarks marked directional; prices “verify before budget”; sizes from the card |
| Collection | License noted for any candidate | | Collection | License + real size/class; never re-announce a cataloged id as a new release |
| Axes | C and D must be scanned; `*(none)*` is allowed only after a look |
| Length | Prefer one screen to three; deep dives go under `research/` | | Length | Prefer one screen to three; deep dives go under `research/` |
--- ---

View file

@ -0,0 +1,136 @@
# Decision: Scaleway object storage as the open-weight reserve
**Date:** 2026-09-13
**Status:** accepted
**Affects:** `docs/backup-storage-policy.md`, `inventory/collection-policy.md`,
`docs/hardware-envelope.md`, `scripts/collect_model.py`, FI-WP-0004
**Supersedes (storage backend only):**
`docs/decisions/2026-07-24-nas-strategic-reserve.md` § storage facility.
The **capability-first** rule from that decision stands.
---
## Decision
1. **Source of truth for weight blobs** is a dedicated Scaleway Object
Storage bucket in `nl-ams`, operated as `reef-storage`, consumed by
this repo. Git still holds catalog metadata only.
2. **VAULT HD** (`D:\vault\coulomb\freedom-intelligence\`) is **staging
and optional local R-spine cache**. It is not the strategic envelope.
3. **Storage class follows tier:**
- **R (runnable spine):** One Zone IA — cheap, GET is immediate.
- **S (strategic):** Glacier — cheapest just-in-case; restore cost
and latency are acceptable.
- **W:** only with euro-budget headroom; One Zone IA or Glacier by size.
4. **Capacity gate is euros/month, not a 850 GiB disk quota.** The 850 /
920 GiB numbers were a 1 TB NAS artifact. Usage-based object storage
removes that wall. Soft infrastructure budget: **€15 / month**
(ex VAT) for this bucket. Founder Scaleway billing alert remains the
hard backstop (`RESOURCE-WP-0002`).
5. **Do not reuse** `railiance-platform-pg-backup`. That bucket expires
objects at 30 days. Model weights must not.
---
## Why Scaleway (economics)
Org already selected Scaleway over Hetzner / Host Europe for S3
(`resource-control` RESOURCE-WP-0002, 2026-08-14). Same project, same
failure domain independence from Host Europe `railiance01`.
Prices from [Scaleway storage pricing](https://www.scaleway.com/en/pricing/storage/)
as of 2026-09-13, ex tax:
| Class | € / GB-month | 45 GiB (R spine) | 167 GiB (V4-Flash) | 1454 GiB (Kimi K3) |
| ----- | -----------: | ---------------: | -----------------: | -----------------: |
| Multi-AZ Standard | ~0.0146 | €0.66 | €2.44 | €21.23 |
| One Zone IA | ~0.012 | €0.54 | €2.00 | €17.45 |
| Glacier | ~0.00254 | €0.11 | €0.42 | **€3.69** |
Glacier restore to Standard is €0.009/GB (V4-Flash restore ≈ €1.50;
K3 restore ≈ €13). Ingress is included. Egress is 75 GB/month free,
then €0.01/GB — relevant when we actually pull a giant to a GPU host,
not when we park it.
**Portfolio that the 850 GiB NAS could not hold:**
| Item | Class | Est. monthly |
| ---- | ----- | -----------: |
| R-spine resident (One Zone IA) | R | €0.54 |
| DeepSeek-V4-Flash-0731 (Glacier) | S | €0.42 |
| Kimi K3 official tree (Glacier) | S | €3.69 |
| **Total** | | **~€4.65** |
That is inside the €15 soft budget with room for a second compressed
giant or a Qwen3.8-27B R upgrade. Buying another multi-TB disk to
fit K3 would spend the same money in a year and stay on the
workstation failure domain.
VAULT remains useful: sequential staging for `huggingface_hub`
downloads, and a local copy of the R-spine if the workstation is
doing inference. It is no longer the thing INTENT means by “reserve.”
---
## Bucket contract
| Field | Value |
| ----- | ----- |
| Provider | Scaleway Object Storage |
| Region | `nl-ams` |
| Endpoint | `https://s3.nl-ams.scw.cloud` |
| Bucket (planned) | `railiance-fi-open-weight-reserve` |
| Reef attributes | `reef-storage/substrate/object-stores/fi-open-weight-reserve.yaml` |
| Public access | disabled |
| Versioning | on |
| Object expiry | **none** |
| Lifecycle | `strategic/` → Glacier after 1 day |
| Encryption | provider-managed at rest |
| Identity | bucket-scoped key, not the postgres backup key |
| Secrets | `railiance-platform` / OpenBao — never this git repo |
### Prefix layout
```text
s3://railiance-fi-open-weight-reserve/
├── models/{org}__{name}/{revision}/ # R / W (One Zone IA at PUT)
├── strategic/{org}__{name}/{revision}/ # S (Glacier via lifecycle)
├── companions/
├── staging/ # incomplete uploads; purge after 14d
└── manifests/{org}__{name}/{revision}/MANIFEST.json
```
Each prefix maps to one catalog YAML. `collection.storage_path` is the
`s3://` URI, not a VAULT path.
---
## What stays from 2026-07-24
- Run envelope ≠ reserve envelope.
- License and integrity first.
- Catalog in git, blobs off git.
- No public mirror, no closed weights.
- R-spine always kept; S is a small number of top open identities.
## What changes
| Before | After |
| ------ | ----- |
| Facility = workstation VAULT HD | Facility = Scaleway `nl-ams` |
| Soft quota 850 GiB | Soft budget €15/month |
| K3 deferred on capacity | K3 economically in-scope on Glacier |
| V4-Flash uncollectable-as-12B | V4-Flash (~167 GiB MIT MoE) is the first S pull |
---
## Follow-ups
- [x] FI-WP-0004-T01 this file
- [ ] FI-WP-0004-T08 operator creates the bucket
- [ ] FI-WP-0004-T09 first collect: DeepSeek-V4-Flash-0731
- [ ] resource-control inventory record (optional; FI consumes, resource-control costs)

View file

@ -1,6 +1,6 @@
# Hardware envelope — run vs reserve # Hardware envelope — run vs reserve
**Status:** updated 2026-07-24 (1 TB NAS strategic reserve) **Status:** updated 2026-09-13 (Scaleway object reserve; run envelope unchanged)
**Used by:** inventory `hardware_class`, axis B/C, collection tiers R/S **Used by:** inventory `hardware_class`, axis B/C, collection tiers R/S
--- ---
@ -10,10 +10,10 @@
Separate two questions that used to be conflated: Separate two questions that used to be conflated:
1. **What can we run or fine-tune now?***run envelope* 1. **What can we run or fine-tune now?***run envelope*
2. **What should we store on the NAS just in case?** → *reserve envelope* (1 TB) 2. **What should we store just in case?** → *reserve envelope* (Scaleway Glacier / One Zone IA)
Collecting a model **does not** require it to fit the run envelope. Collecting a model **does not** require it to fit the run envelope.
See `docs/decisions/2026-07-24-nas-strategic-reserve.md`. See `docs/decisions/2026-09-13-scaleway-object-reserve.md`.
--- ---
@ -34,13 +34,15 @@ valid for NAS collection.
--- ---
## Reserve envelope (NAS) ## Reserve envelope (object store)
| Parameter | Value | | Parameter | Value |
| --------- | ----- | | --------- | ----- |
| Facility | VAULT HD `D:\vault\coulomb\freedom-intelligence\` (soft quota 850 GiB) | | Facility | Scaleway Object Storage `nl-ams` (`railiance-fi-open-weight-reserve`) |
| Soft quota | 850 GiB | | Soft budget | **€15 / month** |
| Hard stop | 920 GiB | | R class | One Zone IA |
| S class | Glacier |
| Staging | VAULT HD `D:\vault\coulomb\freedom-intelligence\` |
| Policy | `docs/backup-storage-policy.md` | | Policy | `docs/backup-storage-policy.md` |
--- ---
@ -51,7 +53,8 @@ valid for NAS collection.
| ---- | ---- | --- | ---- | --- | ----- | | ---- | ---- | --- | ---- | --- | ----- |
| TBD | primary local inference | TBD | TBD | TBD | Run envelope | | TBD | primary local inference | TBD | TBD | TBD | Run envelope |
| TBD | training experiments | TBD | TBD | TBD | Run envelope | | TBD | training experiments | TBD | TBD | TBD | Run envelope |
| **VAULT HD (D:)** | weight store | n/a | n/a | n/a | **Reserve envelope** — soft 850 GiB | | **VAULT HD (D:)** | staging / R cache | n/a | n/a | n/a | **Not SoT** — downloads and optional local R |
| **Scaleway nl-ams** | weight SoT | n/a | n/a | n/a | **Reserve envelope** — €15/mo soft |
| railiance01 | cluster | n/a | n/a | n/a | Do not store weight trees on hot disks | | railiance01 | cluster | n/a | n/a | n/a | Do not store weight trees on hot disks |
--- ---
@ -76,7 +79,7 @@ Strategic giants may remain **cold storage only** until hardware upgrades.
| QLoRA 78B | T1T2 | | QLoRA 78B | T1T2 |
| QLoRA 14B | T2 | | QLoRA 14B | T2 |
| Full FT 7B | T2T3 | | Full FT 7B | T2T3 |
| 70B+ / large MoE FT | Future facility — weights may already sit on NAS | | 70B+ / large MoE FT | Future facility — weights may already sit on Glacier |
--- ---
@ -86,3 +89,4 @@ Strategic giants may remain **cold storage only** until hardware upgrades.
| ---- | ------ | | ---- | ------ |
| 2026-07-24 | Draft tiers | | 2026-07-24 | Draft tiers |
| 2026-07-24 | Split run vs reserve; NAS 1 TB; S tier may exceed run envelope | | 2026-07-24 | Split run vs reserve; NAS 1 TB; S tier may exceed run envelope |
| 2026-09-13 | Reserve envelope moved to Scaleway; VAULT is staging; euro gate |

View file

@ -19,7 +19,8 @@ activity-core (when / what / where)
rein-aharness on railiance01 (does the work) rein-aharness on railiance01 (does the work)
primary: claim-loop → ops_run → fi-research-brief (REIN-A-0002) primary: claim-loop → ops_run → fi-research-brief (REIN-A-0002)
break-glass only: host timer 07:35 (disabled 2026-08-05; re-enable if claim loop down) break-glass only: host timer 07:35 (disabled 2026-08-05; re-enable if claim loop down)
llm-connect drafts brief → commit → push → fi_daily_brief llm-connect drafts brief → commit → git push origin → fi_daily_brief
(fi_daily_brief is forbidden until origin has the file)
Operator morning review (no AI) from this repo: Operator morning review (no AI) from this repo:
```bash ```bash
@ -118,7 +119,19 @@ detail:
collection_candidates: <int> collection_candidates: <int>
``` ```
Posted automatically by `rein-aharness fi-research-brief` on success. Posted automatically by `rein-aharness fi-research-brief` **only after**
`git push origin` succeeds. `detail.pushed` and `detail.origin_sha` are
required. A local commit with a failed push is `executor_run` / `ok=false`.
FI is the definition owner: origin publication is a **granted** capability
of this rhythm (FI-WP-0004-T04), not an ungoverned env widening of
`FI_RESEARCH_BRIEF_PUSH`.
Verify drift:
```bash
python3 scripts/verify_brief_durability.py --require-hub
```
--- ---
@ -127,4 +140,6 @@ Posted automatically by `rein-aharness fi-research-brief` on success.
- Workstation `crontab` that writes the brief (bypasses activity-core) - Workstation `crontab` that writes the brief (bypasses activity-core)
- Installing Claude Code / Codex on railiance01 for this rhythm - Installing Claude Code / Codex on railiance01 for this rhythm
- Treating host timers as the scheduler of record - Treating host timers as the scheduler of record
- Skipping the completion event (leaves `due=true` forever) - Skipping the completion event (leaves `due=true` forever)
- Posting `fi_daily_brief` for a local-only commit (clears `due`, loses the day)
- Reusing `railiance-platform-pg-backup` for weights (30-day expiry)

View file

@ -19,7 +19,8 @@ Do not create workplans via hub `create_workplan` APIs.
## Required: completion evidence for daily rhythm ## Required: completion evidence for daily rhythm
Every completed brief day (including “no material delta”) should post: Every completed brief day (including “no material delta”) should post
**after the file is on `origin/main`**:
```yaml ```yaml
event_type: fi_daily_brief event_type: fi_daily_brief
@ -30,11 +31,15 @@ detail:
date: "YYYY-MM-DD" date: "YYYY-MM-DD"
path: "briefs/YYYY/MM/YYYY-MM-DD.md" path: "briefs/YYYY/MM/YYYY-MM-DD.md"
collection_candidates: 0 collection_candidates: 0
pushed: true
origin_sha: "<full sha>"
``` ```
This is the **idempotence signal** for activity-core context query This is the **idempotence signal** for activity-core context query
`fi_brief_status` (FI-WP-0002-T03): `due=false` when an event exists for today `fi_brief_status` (FI-WP-0002-T03): `due=false` when an event exists for today
(Europe/Berlin) with `detail.repo=freedom-intelligence`. (Europe/Berlin) with `detail.repo=freedom-intelligence` **and**
`detail.pushed=true`. A hub event without a git file is a false completion
(`scripts/verify_brief_durability.py`).
Inbound automation uses `activity_task_spawn` from activity-core (state-hub sink). Inbound automation uses `activity_task_spawn` from activity-core (state-hub sink).
See `activity-core/docs/task-emission-consumer-contract.md`. See `activity-core/docs/task-emission-consumer-contract.md`.

View file

@ -1,8 +1,8 @@
# Model inventory # Model inventory
In-repo **catalog of open-weight revisions** reserved (or nominated) for the lab. In-repo **catalog of open-weight revisions** reserved (or nominated) for the lab.
Weight blobs live on the **VAULT HD** at `D:\vault\coulomb\freedom-intelligence\` Weight blobs live on **Scaleway Object Storage** (`s3://railiance-fi-open-weight-reserve/…`).
see `docs/backup-storage-policy.md`. VAULT HD is staging only — see `docs/backup-storage-policy.md`.
## Layout ## Layout

View file

@ -1,8 +1,10 @@
# Open-weight reserve status # Open-weight reserve status
**As of:** 2026-08-03 **As of:** 2026-09-13
**Store:** `/mnt/d/vault/coulomb/freedom-intelligence/` (`D:\vault\coulomb\freedom-intelligence\`) **SoT store:** Scaleway Object Storage `nl-ams` bucket
**Soft quota:** 850 GiB · **Hard stop:** 920 GiB `railiance-fi-open-weight-reserve` (**planned** — FI-WP-0004-T08)
**Staging:** `/mnt/d/vault/coulomb/freedom-intelligence/` (VAULT HD)
**Capacity gate:** **€15 / month** soft (not 850 GiB)
--- ---
@ -10,64 +12,70 @@
| Class | Count | Status mix | | Class | Count | Status mix |
| ----- | ----: | ---------- | | ----- | ----: | ---------- |
| Catalog entries | 13 | + Kimi K3 S1 | | Catalog entries | 15 | + V4-Flash S, + Qwen3.8-27B W/R |
| **verified** on disk | 4 | R embeds + Qwen3-8B + R1-Distill-14B | | **verified** on VAULT | 4 | R embeds + Qwen3-8B + R1-Distill-14B |
| **approved** (pull deferred) | S giants + Llama edge + **Kimi K3** | capacity / HF gate | | **approved**, blobs missing | S giants + Llama-3.2-3B + **V4-Flash** | HF gate / bucket not live |
| **candidate** (W) | 3 | headroom-gated | | **candidate** | 4 | W + Qwen3.8-27B |
### Disk use (model tree) ### Disk / object use
| Path | Role | Approx | | Location | Role | Approx |
| ---- | ---- | ------ | | -------- | ---- | ------ |
| `models/nomic-ai__nomic-embed-text-v1.5/main` | R embed | ~0.5 GiB verified | | VAULT `models/nomic-ai__nomic-embed-text-v1.5/main` | R embed (staging cache) | ~0.5 GiB verified |
| `models/BAAI__bge-m3/main` | R embed | ~2.1 GiB verified | | VAULT `models/BAAI__bge-m3/main` | R embed (staging cache) | ~2.1 GiB verified |
| `models/Qwen__Qwen3-8B/main` | R instruct | ~15.3 GiB verified | | VAULT `models/Qwen__Qwen3-8B/main` | R instruct (staging cache) | ~15.3 GiB verified |
| `models/deepseek-ai__DeepSeek-R1-Distill-Qwen-14B/main` | R reason | ~27.5 GiB verified | | VAULT `models/deepseek-ai__DeepSeek-R1-Distill-Qwen-14B/main` | R reason (staging cache) | ~27.5 GiB verified |
| Scaleway bucket | SoT | **empty** (not created) |
**~45 GiB** used → **~800+ GiB** soft-quota headroom (insufficient for full Kimi K3 ~1454 GiB). VAULT ~45 GiB used remains a **cache**, not the reserve.
VAULT free (2026-08-03): ~1.7 TB free of 1.9 TB — full K3 would still dominate the volume and **violate soft quota**.
--- ---
## Decision matrix (2026-08-03) ## Decision matrix (2026-09-13)
| Id | Tier | Decision | Disk | | Id | Tier | Decision | Blobs |
| -- | ---- | -------- | ---- | | -- | ---- | -------- | ----- |
| Qwen3-8B | R | **verified** | yes | | Qwen3-8B | R | **verified** on VAULT cache | local only |
| Llama-3.2-3B-Instruct | R | **approved** | deferred — HF gate | | Llama-3.2-3B-Instruct | R | **approved** | deferred — HF gate |
| BGE-M3 | R | **verified** | yes | | BGE-M3 | R | **verified** on VAULT cache | local only |
| R1-Distill-Qwen-14B | R | **verified** | yes | | R1-Distill-Qwen-14B | R | **verified** on VAULT cache | local only |
| nomic-embed-text-v1.5 | R | **verified** | yes | | nomic-embed-text-v1.5 | R | **verified** on VAULT cache | local only |
| **Kimi-K3** | **S1** | **approved**, **collection deferred** | no — ~1454 GiB > soft quota | | **DeepSeek-V4-Flash-0731** | **S1** | **approved****first Scaleway pull** | no — ~167 GiB Glacier |
| DeepSeek-V3 | S1→secondary | **approved**, deferred | no — K3 is primary open SOTA identity | | Kimi-K3 | S1 secondary | **approved**, deferred until after V4-Flash | no — ~1454 GiB Glacier ≈ €3.69/mo |
| DeepSeek-V3 | S superseded identity | **approved**, deferred | no — V4-Flash is the collectable S1 |
| DeepSeek-R1 full | S2 | **approved**, deferred | no | | DeepSeek-R1 full | S2 | **approved**, deferred | no |
| Qwen3-72B | S4 | **approved**, deferred | no | | Qwen3-72B | S4 | **approved**, deferred | no |
| Llama-3.3-70B-Instruct | S4 | **approved**, deferred | no | | Llama-3.3-70B-Instruct | S4 | **approved**, deferred | no |
| Qwen3.8-27B | W/R | **candidate** | no |
| Qwen3-14B | W | candidate | no | | Qwen3-14B | W | candidate | no |
| R1-Distill-32B | W | candidate | no | | R1-Distill-32B | W | candidate | no |
| bge-reranker-v2-m3 | W | candidate | no | | bge-reranker-v2-m3 | W | candidate | no |
**Correction:** there is no `DeepSeek-V4-Flash-0731-12B`. Automated briefs
that used that id were wrong. Do not collect under that name.
--- ---
## Hardware coverage gaps ## Hardware coverage gaps
| Need | Status | | Need | Status |
| ---- | ------ | | ---- | ------ |
| Default local instruct (8B) | **Covered** (Qwen3-8B) | | Default local instruct (8B) | **Covered** (Qwen3-8B cache) |
| Stronger local instruct (27B) | **Candidate** Qwen3.8-27B |
| Edge micro-agent | Llama-3.2-3B blocked on HF auth | | Edge micro-agent | Llama-3.2-3B blocked on HF auth |
| Multilingual RAG embed | **Covered** (BGE-M3 + nomic) | | Multilingual RAG embed | **Covered** (BGE-M3 + nomic) |
| Local reason | **Covered** (R1-Distill-14B) | | Local reason | **Covered** (R1-Distill-14B) |
| Frontier-open general MoE offline | **Cataloged** Kimi K3 — pull blocked on capacity | | Frontier-open MIT MoE offline | **Cataloged** V4-Flash — pull blocked on bucket |
| Full open reasoner MoE | Deferred S2 | | Full open giant (K3) | Cataloged; Glacier-affordable; after V4-Flash |
--- ---
## Next pulls (operator) ## Next pulls (operator)
1. **Capacity decision for Kimi K3** (raise soft quota / dedicate disk / official compressed only). 1. **FI-WP-0004-T08** create `railiance-fi-open-weight-reserve` (no 30-day expiry).
2. Set `HF_TOKEN` and pull Llama-3.2-3B-Instruct. 2. **FI-WP-0004-T09** collect `deepseek-ai/DeepSeek-V4-Flash-0731``strategic/` Glacier.
3. Optional S4: one of Qwen3-72B vs Llama-3.3-70B under remaining quota. 3. Set `HF_TOKEN` and pull Llama-3.2-3B-Instruct into `models/` One Zone IA.
4. DeepSeek V4-Flash-0731 as mid open value candidate (catalog later if desired). 4. Decide Qwen3.8-27B vs keeping Qwen3-8B as the R instruct default.
5. Optional: Kimi K3 on Glacier if the month is still under €15.
Tool: `scripts/collect_model.py` (weights-only, sequential, writes `MANIFEST.json`). Tool: `scripts/collect_model.py` (weights-only, sequential, MANIFEST, optional `--s3-bucket`).

View file

@ -0,0 +1,74 @@
id: Qwen__Qwen3.8-27B__candidate
status: candidate
name: Qwen3.8-27B
org: Qwen
source:
kind: huggingface
url: https://huggingface.co/Qwen/Qwen3.8-27B
revision: main
model_card_url: https://huggingface.co/Qwen/Qwen3.8-27B
project_url: https://qwen.ai/
license:
spdx: Apache-2.0
url: https://huggingface.co/Qwen/Qwen3.8-27B
allows_offline_retention: true
allows_local_ops: true
allows_fine_tune: true
notes: "Apache 2.0 on the 27B dense multimodal card (2026-08-14). Confirm at pin. Not the Qwen3.8-Max bespoke licence."
profile:
summary: "Dense 27B multimodal Apache-2.0 — candidate R-spine upgrade vs Qwen3-8B."
original_source: https://huggingface.co/Qwen/Qwen3.8-27B
use_cases:
- "Local instruct / vision on T2T3 (24 GB class at Q4)"
- "Coding and office agents that need more than 8B"
- "Homelab QLoRA base"
sweet_spots:
- "Apache-2.0, ungated, actually runnable unlike V4-Flash / K3"
- "Native vision; 262k context (YaRN toward 1M)"
not_ideal_for:
- "Replacing the 8B always-on default until measured"
- "Standing in for Qwen3.8-2.4T-A95B (different licence, multi-TB)"
capability_notes: >
Released 2026-08-14, the day the automated brief last landed on
origin — so the sensing loop never recorded it. Dense 27.8B,
multimodal, Apache 2.0. Distinct from Qwen3.8-2.4T-A95B (custom
Max licence, ~4.89 TB). Nominate as W/R; do not pull until euro
budget and R-spine review after V4-Flash lands.
swot:
strengths:
- "Apache-2.0 dense multimodal at a size we can actually serve later"
- "Likely better local default than Qwen3-8B if VRAM allows"
weaknesses:
- "Larger than current verified R instruct (8B)"
- "Hardware envelope hosts still TBD"
opportunities:
- "R-spine upgrade path that does not need Glacier"
- "Vision in the local working set"
threats:
- "Qwen 4.x could supersede before we pull"
size:
total_bytes: 0
total_human: "~56 GiB class (reports); measure at pull — Q4 much smaller"
hardware_class:
min_vram_gb_q4: 20
min_vram_gb_fp16: 56
notes: "T2 stretch / T3. Not T1."
axes: [B, C]
priority: medium
collection:
approved_by: ""
approved_at: null
downloaded_at: null
downloaded_by: ""
storage_path: ""
brief_refs:
- briefs/2026/09/2026-09-13.md
reason: "Missed by the Aug 14 brief cutoff; Apache-2.0 dense 27B is the obvious R-upgrade candidate."
tags: [tier-w, tier-r, instruct, vision, apache]
companions: []
notes: "Do not confuse with Qwen3.8-2.4T-A95B (S-class, custom licence, multi-TB)."
history:
- at: "2026-09-13"
event: nominated
by: grok
detail: "Catch-up brief after cadence hole. Candidate only."

View file

@ -0,0 +1,94 @@
id: deepseek-ai__DeepSeek-V4-Flash-0731__strategic
status: approved
name: DeepSeek-V4-Flash-0731
org: deepseek-ai
source:
kind: huggingface
url: https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731
revision: main
model_card_url: https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731
project_url: https://www.deepseek.com/
paper_url: https://arxiv.org/abs/2606.19348
license:
spdx: MIT
url: https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731
allows_offline_retention: true
allows_local_ops: true
allows_fine_tune: true
notes: "Official card: repository and weights MIT. Confirm at pin."
profile:
summary: "S-tier open MIT MoE — 284B base / ~13B active (304B with DSpark); not a 12B dense model."
original_source: https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731
use_cases:
- "Primary open-weight agentic / coding identity after Jul 2026"
- "Offline optionality if DeepSeek API access changes"
- "Future multi-GPU serve (vLLM DSpark / SGLang); not current R-spine"
- "Capability bar for daily briefs against closed frontier"
sweet_spots:
- "Best MIT-licensed open near-frontier that actually fits cheap Glacier (~167 GiB HF tree)"
- "Fits the €15/month Scaleway budget with room for K3"
- "Displaces DeepSeek-V3 as the collectable S1 identity"
not_ideal_for:
- "Current single-consumer-GPU daily ops (use R spine / Qwen3.8-27B)"
- "Treating 'Flash' as a 12B dense model (that reading was an executor error)"
capability_notes: >
Official V4-Flash release (2026-07-31). Same structure as
DeepSeek-V4-Flash-DSpark (speculative decoding module attached).
HF card reports 304B params (284B base + draft). ~13B active.
1M context. Terminal Bench 2.1 82.7. HF tree listed ~167 GB.
Automated briefs 2026-08-04…14 repeatedly called this a 12B dense
consumer-GPU model and once claimed it was already collected. It
was not. Size class is S, not R.
swot:
strengths:
- "MIT, official org, actually downloadable"
- "Near-frontier agentic scores at ~13B active"
- "Footprint (~167 GiB) is cheap on Glacier and was always inside the old 850 GiB quota"
weaknesses:
- "Serve needs multi-GPU / high unified memory, not T1T2"
- "Executor hallucinated it as 12B for two weeks — catalog must stay the authority"
opportunities:
- "First S pull onto Scaleway (FI-WP-0004-T09)"
- "Compressed / GGUF community packs exist if we want a later R experiment (not a substitute for the official tree)"
threats:
- "Successor V4.x could land while we still have no object on disk"
- "Quota-era deferral already cost a month of optionality"
size:
total_bytes: 179000000000
total_human: "~167 GiB official HF tree (card file listing 2026-09-13); 304B params / ~13B active"
hardware_class:
min_vram_gb_q4: 0
min_vram_gb_fp16: 0
notes: "Beyond current lab run envelope (T3T4). Community 2-bit reports exist for 128 GB unified memory; not an R-spine claim."
axes: [A, B, D]
priority: high
collection:
approved_by: "grok"
approved_at: "2026-09-13"
downloaded_at: null
downloaded_by: ""
storage_path: ""
brief_refs:
- briefs/2026/08/2026-08-03.md
- briefs/2026/09/2026-09-13.md
- docs/decisions/2026-09-13-scaleway-object-reserve.md
reason: "S1 collectable — strongest MIT open MoE that fits the Scaleway euro budget; first pull after bucket create."
tags: [tier-s, strategic, moe, mit, beyond-run-envelope]
companions: []
notes: >
Do not confuse with a 12B dense checkpoint. There is no
deepseek-ai/DeepSeek-V4-Flash-0731-12B. Pull the official card repo.
Storage class at upload: strategic/ → Glacier.
history:
- at: "2026-08-03"
event: nominated
by: grok
detail: "Catch-up brief named V4-Flash-0731 as mid-tier open value leader."
- at: "2026-08-04"
event: misidentified
by: rein-aharness
detail: "Automated briefs treated it as 12B dense; never cataloged; 08-07 claimed on-disk collection that did not happen."
- at: "2026-09-13"
event: approved
by: grok
detail: "Corrected to 304B-class MoE ~167 GiB MIT. Approved for Scaleway Glacier once FI-WP-0004-T08 bucket exists."

View file

@ -1,15 +1,16 @@
# Collection policy — open-weight reserve # Collection policy — open-weight reserve
**Status:** updated for 1 TB NAS + strategic capability reserve (2026-07-24) **Status:** updated for Scaleway object reserve (2026-09-13)
**Related:** `schema.yaml`, `docs/backup-storage-policy.md`, **Related:** `schema.yaml`, `docs/backup-storage-policy.md`,
`docs/decisions/2026-07-24-nas-strategic-reserve.md`, `INTENT.md` `docs/decisions/2026-09-13-scaleway-object-reserve.md`, `INTENT.md`
--- ---
## Purpose ## Purpose
Decide **what** enters the open-weight reserve, **who** may approve it, and Decide **what** enters the open-weight reserve, **who** may approve it, and
**when** a daily-brief candidate becomes a catalog entry with blobs on the NAS. **when** a daily-brief candidate becomes a catalog entry with blobs on
Scaleway (staged locally first).
--- ---
@ -19,7 +20,7 @@ Decide **what** enters the open-weight reserve, **who** may approve it, and
need later — **including models too large to run on current lab hardware** need later — **including models too large to run on current lab hardware**
* Also hold a **runnable spine** for day-to-day local ops and fine-tunes * Also hold a **runnable spine** for day-to-day local ops and fine-tunes
* Enforce **license and integrity** before download completes * Enforce **license and integrity** before download completes
* Stay within the **1 TB NAS soft quota (850 GiB)** — quality over mirror volume * Stay within the **€15 / month** soft budget — quality over mirror volume
* Prefer official provenance; not a full Hugging Face scrape * Prefer official provenance; not a full Hugging Face scrape
--- ---
@ -32,7 +33,7 @@ Decide **what** enters the open-weight reserve, **who** may approve it, and
| **Reserve** | Is this among the best open artifacts worth keeping *just in case*? | **Not gated by current VRAM** | | **Reserve** | Is this among the best open artifacts worth keeping *just in case*? | **Not gated by current VRAM** |
**Runnability is not an eligibility gate.** An unrunnable SOTA open MoE can be **Runnability is not an eligibility gate.** An unrunnable SOTA open MoE can be
priority **high** if license and quota allow. priority **high** if license and euro budget allow.
--- ---
@ -42,7 +43,8 @@ priority **high** if license and quota allow.
2. **Clear license** — SPDX or linkable license text; `allows_offline_retention: true` 2. **Clear license** — SPDX or linkable license text; `allows_offline_retention: true`
3. **Stable provenance** — official org, tagged release, or commit revision 3. **Stable provenance** — official org, tagged release, or commit revision
4. **Lab rationale** — written `reason` (capability SOTA, runnable spine, embed, FT base, …) 4. **Lab rationale** — written `reason` (capability SOTA, runnable spine, embed, FT base, …)
5. **Capacity** — estimated size fits under remaining soft quota (850 GiB) 5. **Capacity** — estimated monthly storage cost fits under remaining
€15 soft budget (Glacier for S, One Zone IA for R)
Fail any gate → `rejected` or never enter catalog. Fail any gate → `rejected` or never enter catalog.
@ -75,9 +77,9 @@ Catalog field: use `tags` including `tier-r` / `tier-s` / `tier-w` and
| -------------------- | -------- | | -------------------- | -------- |
| **&lt; 5 GiB** | Operator or lab agent after license check | | **&lt; 5 GiB** | Operator or lab agent after license check |
| **540 GiB** | Explicit operator approval | | **540 GiB** | Explicit operator approval |
| **&gt; 40 GiB** | Operator approval + remaining soft-quota check | | **&gt; 40 GiB** | Operator approval + remaining **euro-budget** check |
| **&gt; 200 GiB (typical S giants)** | Operator approval + written note on which other S models may need to wait | | **&gt; 200 GiB (typical S giants)** | Operator approval + written note that the object goes to **Glacier** |
| **Any size if quota ≥ 70% used** | Operator approval required | | **Any size if month ≥ 70% of €15** | Operator approval required |
| **Unclear license or ToS risk** | Do not collect; `rejected` | | **Unclear license or ToS risk** | Do not collect; `rejected` |
Agents may **nominate** freely; they **collect** only in the &lt; 5 GiB band with Agents may **nominate** freely; they **collect** only in the &lt; 5 GiB band with
@ -97,8 +99,9 @@ unambiguous licenses unless the operator has approved the catalog entry.
* **Best available open general / reasoning / code weights** at the frontier of open * **Best available open general / reasoning / code weights** at the frontier of open
* Large MoE or dense models even if current infra cannot serve them * Large MoE or dense models even if current infra cannot serve them
* Prefer official compressed releases (FP8, published quant) when full precision * Prefer official compressed releases (FP8, published quant) when full
would exhaust the 1 TB NAS precision would blow the euro budget; Glacier makes full official trees
of the *top* S identities affordable (V4-Flash ~€0.42/mo, K3 ~€3.69/mo)
* One clear “best open” per capability niche is better than five near-duplicates * One clear “best open” per capability niche is better than five near-duplicates
### Companions ### Companions
@ -124,7 +127,7 @@ Tokenizers, LoRA adapters, small eval fixtures when required to use a reserved b
brief nominates brief nominates
→ candidate → candidate
→ approved → approved
→ collecting (NAS staging/) → collecting (local staging/ then S3 upload)
→ collected → collected
→ verified → verified
→ superseded|evicted → superseded|evicted
@ -136,14 +139,15 @@ brief nominates
1. Brief **Collection candidates** nominates R/S/W. 1. Brief **Collection candidates** nominates R/S/W.
2. Catalog YAML under `inventory/catalog/`. 2. Catalog YAML under `inventory/catalog/`.
3. Download only after storage path is pinned (`docs/backup-storage-policy.md`) and approval rules pass. 3. Download only after the Scaleway bucket is live (FI-WP-0004-T08) and
approval rules pass. Stage on VAULT; SoT is `s3://`.
4. `collection.brief_refs` / research refs for provenance of the nomination. 4. `collection.brief_refs` / research refs for provenance of the nomination.
--- ---
## Eviction rule of thumb ## Eviction rule of thumb
When over soft quota: When over the euro soft budget:
1. **W** tier and easily re-obtainable duplicates 1. **W** tier and easily re-obtainable duplicates
2. Superseded revisions with a stronger verified successor 2. Superseded revisions with a stronger verified successor

View file

@ -1,13 +1,27 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
"""Collect one HF model into the VAULT reserve tree (weights-first). """Collect one HF model into local staging, optionally upload to Scaleway.
Usage: Usage:
/tmp/fi-hf/bin/python scripts/collect_model.py \\ python3 scripts/collect_model.py \\
--repo-id Qwen/Qwen3-8B --local-name Qwen__Qwen3-8B --revision main --repo-id Qwen/Qwen3-8B --local-name Qwen__Qwen3-8B --revision main
Layout: python3 scripts/collect_model.py \\
{base}/models/{local_name}/{revision}/ # HF files (no full onnx dump) --repo-id deepseek-ai/DeepSeek-V4-Flash-0731 \\
{base}/staging/... # incomplete --local-name deepseek-ai__DeepSeek-V4-Flash-0731 \\
--tier s \\
--s3-bucket railiance-fi-open-weight-reserve
Layout (local staging, still required for huggingface_hub):
{base}/models/{local_name}/{revision}/ # R / W
{base}/strategic/{local_name}/{revision}/ # S
{base}/staging/...
S3 prefix (SoT once bucket is live):
s3://{bucket}/models|strategic/{local_name}/{revision}/
s3://{bucket}/manifests/{local_name}/{revision}/MANIFEST.json
Credentials: AWS_ACCESS_KEY_ID + AWS_SECRET_ACCESS_KEY, or SCW_ACCESS_KEY +
SCW_SECRET_KEY. Never commit keys.
""" """
from __future__ import annotations from __future__ import annotations
@ -80,44 +94,62 @@ def main() -> int:
ap.add_argument( ap.add_argument(
"--base", "--base",
default="/mnt/d/vault/coulomb/freedom-intelligence", default="/mnt/d/vault/coulomb/freedom-intelligence",
help="local staging root (VAULT HD)",
)
ap.add_argument(
"--tier",
choices=("r", "s", "w"),
default="r",
help="r/w → models/ (One Zone IA); s → strategic/ (Glacier)",
)
ap.add_argument("--s3-bucket", default=os.environ.get("FI_S3_BUCKET") or "")
ap.add_argument(
"--s3-endpoint",
default=os.environ.get("FI_S3_ENDPOINT", "https://s3.nl-ams.scw.cloud"),
)
ap.add_argument(
"--s3-region",
default=os.environ.get("FI_S3_REGION", "nl-ams"),
) )
ap.add_argument("--token", default=os.environ.get("HF_TOKEN") or None) ap.add_argument("--token", default=os.environ.get("HF_TOKEN") or None)
args = ap.parse_args() args = ap.parse_args()
base = Path(args.base) base = Path(args.base)
dest_kind = "strategic" if args.tier == "s" else "models"
stage = base / "staging" / args.local_name / args.revision stage = base / "staging" / args.local_name / args.revision
final = base / "models" / args.local_name / args.revision final = base / dest_kind / args.local_name / args.revision
if final.exists() and any(final.rglob("*.safetensors")) or ( already = final.exists() and (
final.exists() and any(final.rglob("pytorch_model*.bin")) any(final.rglob("*.safetensors")) or any(final.rglob("pytorch_model*.bin"))
):
print(f"already present with weights: {final}")
return 0
if stage.exists():
shutil.rmtree(stage, ignore_errors=True)
stage.mkdir(parents=True, exist_ok=True)
print(f"downloading {args.repo_id}@{args.revision} -> {stage}")
snapshot_download(
repo_id=args.repo_id,
revision=args.revision,
local_dir=str(stage),
allow_patterns=list(DEFAULT_ALLOW),
ignore_patterns=list(IGNORE),
token=args.token,
max_workers=1,
) )
if already:
print(f"already present with weights: {final}", file=sys.stderr)
if not args.s3_bucket:
return 0
else:
if stage.exists():
shutil.rmtree(stage, ignore_errors=True)
stage.mkdir(parents=True, exist_ok=True)
# strip hub cache dir if present print(f"downloading {args.repo_id}@{args.revision} -> {stage}")
cache = stage / ".cache" snapshot_download(
if cache.exists(): repo_id=args.repo_id,
shutil.rmtree(cache, ignore_errors=True) revision=args.revision,
local_dir=str(stage),
allow_patterns=list(DEFAULT_ALLOW),
ignore_patterns=list(IGNORE),
token=args.token,
max_workers=1,
)
final.parent.mkdir(parents=True, exist_ok=True) cache = stage / ".cache"
if final.exists(): if cache.exists():
shutil.rmtree(final) shutil.rmtree(cache, ignore_errors=True)
shutil.move(str(stage), str(final))
final.parent.mkdir(parents=True, exist_ok=True)
if final.exists():
shutil.rmtree(final)
shutil.move(str(stage), str(final))
artifacts = [] artifacts = []
total = 0 total = 0
@ -145,9 +177,99 @@ def main() -> int:
"artifacts": artifacts, "artifacts": artifacts,
} }
(final / "MANIFEST.json").write_text(json.dumps(manifest, indent=2) + "\n") (final / "MANIFEST.json").write_text(json.dumps(manifest, indent=2) + "\n")
print(json.dumps({"ok": True, "total_bytes": total, "path": str(final)}, indent=2)) result = {"ok": True, "total_bytes": total, "path": str(final), "tier": args.tier}
if args.s3_bucket:
prefix = f"{dest_kind}/{args.local_name}/{args.revision}"
storage_class = "GLACIER" if args.tier == "s" else "ONEZONE_IA"
uploaded = upload_tree(
final,
bucket=args.s3_bucket,
prefix=prefix,
endpoint=args.s3_endpoint,
region=args.s3_region,
storage_class=storage_class,
)
man_key = f"manifests/{args.local_name}/{args.revision}/MANIFEST.json"
upload_file(
final / "MANIFEST.json",
bucket=args.s3_bucket,
key=man_key,
endpoint=args.s3_endpoint,
region=args.s3_region,
storage_class="STANDARD",
)
result["s3"] = {
"bucket": args.s3_bucket,
"prefix": f"s3://{args.s3_bucket}/{prefix}/",
"manifest": f"s3://{args.s3_bucket}/{man_key}",
"storage_class": storage_class,
"objects": uploaded,
}
print(json.dumps(result, indent=2))
return 0 return 0
def _s3_client(endpoint: str, region: str):
try:
import boto3
except ImportError as exc:
raise SystemExit("boto3 required for --s3-bucket (pip install boto3)") from exc
access = os.environ.get("AWS_ACCESS_KEY_ID") or os.environ.get("SCW_ACCESS_KEY")
secret = os.environ.get("AWS_SECRET_ACCESS_KEY") or os.environ.get("SCW_SECRET_KEY")
if not access or not secret:
raise SystemExit("set AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY or SCW_ACCESS_KEY/SCW_SECRET_KEY")
return boto3.client(
"s3",
region_name=region,
endpoint_url=endpoint,
aws_access_key_id=access,
aws_secret_access_key=secret,
)
def upload_file(
path: Path,
*,
bucket: str,
key: str,
endpoint: str,
region: str,
storage_class: str,
) -> None:
client = _s3_client(endpoint, region)
extra = {"StorageClass": storage_class} if storage_class else {}
client.upload_file(str(path), bucket, key, ExtraArgs=extra)
def upload_tree(
root: Path,
*,
bucket: str,
prefix: str,
endpoint: str,
region: str,
storage_class: str,
) -> int:
n = 0
for p in sorted(root.rglob("*")):
if not p.is_file() or p.name.startswith("."):
continue
rel = p.relative_to(root).as_posix()
key = f"{prefix.rstrip('/')}/{rel}"
upload_file(
p,
bucket=bucket,
key=key,
endpoint=endpoint,
region=region,
storage_class=storage_class,
)
n += 1
print(f"uploaded {key}", file=sys.stderr)
return n
if __name__ == "__main__": if __name__ == "__main__":
sys.exit(main()) sys.exit(main())

View file

@ -0,0 +1,114 @@
#!/usr/bin/env python3
"""Fail if State Hub fi_daily_brief events are not on origin/main.
A brief day is complete only when briefs/YYYY/MM/YYYY-MM-DD.md exists in
this git repository (the clone should be origin/main). Hub events that
claim wrote/committed without a matching file are the 2026-08/09 failure
mode: local commit on railiance, due-bit cleared, durable memory empty.
Usage:
python3 scripts/verify_brief_durability.py
STATE_HUB_URL=http://127.0.0.1:8000 python3 scripts/verify_brief_durability.py
Exit 0 = every hub success event has a brief file.
Exit 1 = mismatch (or hub unreachable when --require-hub is set).
"""
from __future__ import annotations
import argparse
import json
import os
import sys
import urllib.error
import urllib.request
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
def brief_path(date: str) -> Path:
year, month, _ = date.split("-")
return ROOT / "briefs" / year / month / f"{date}.md"
def load_events(url: str, timeout: float) -> list[dict]:
req = urllib.request.Request(
f"{url.rstrip('/')}/progress/?event_type=fi_daily_brief",
headers={"Accept": "application/json"},
)
with urllib.request.urlopen(req, timeout=timeout) as resp:
data = json.loads(resp.read().decode())
if isinstance(data, list):
return data
if isinstance(data, dict):
for key in ("items", "events", "results"):
if isinstance(data.get(key), list):
return data[key]
raise RuntimeError(f"unexpected hub payload type {type(data)}")
def main() -> int:
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument(
"--hub",
default=os.environ.get("STATE_HUB_URL", "http://127.0.0.1:8000"),
)
ap.add_argument("--timeout", type=float, default=8.0)
ap.add_argument(
"--require-hub",
action="store_true",
help="exit 1 if the hub cannot be reached (default: warn and check git only)",
)
args = ap.parse_args()
git_dates = sorted(
p.stem
for p in (ROOT / "briefs").rglob("20*.md")
if p.name[0].isdigit()
)
print(f"git briefs: {len(git_dates)} last={git_dates[-1] if git_dates else '-'}")
try:
events = load_events(args.hub, args.timeout)
except (urllib.error.URLError, TimeoutError, json.JSONDecodeError, RuntimeError) as exc:
msg = f"hub unreachable or unreadable at {args.hub}: {exc}"
if args.require_hub:
print(f"FAIL {msg}", file=sys.stderr)
return 1
print(f"WARN {msg}")
return 0
missing: list[str] = []
present: list[str] = []
for ev in events:
detail = ev.get("detail") or {}
date = str(detail.get("date") or "")
if not date:
continue
repo = detail.get("repo")
if repo and repo != "freedom-intelligence":
continue
path = brief_path(date)
if path.is_file():
present.append(date)
else:
missing.append(date)
missing = sorted(set(missing))
present = sorted(set(present))
print(f"hub fi_daily_brief dates on git: {len(present)}")
if missing:
print("FAIL hub events with no brief file:")
for date in missing:
print(f" {date} expected {brief_path(date).relative_to(ROOT)}")
print(
"A local-only commit must not clear due. "
"Push to origin, or treat the hub event as a false completion."
)
return 1
print("ok: every hub fi_daily_brief date has a brief file")
return 0
if __name__ == "__main__":
sys.exit(main())

View file

@ -0,0 +1,229 @@
---
id: FI-WP-0004
type: workplan
title: "Make the sensing loop durable and move the reserve to Scaleway"
domain: agents
repo: freedom-intelligence
status: active
owner: grok
topic_slug: freedom-intelligence
priority: high
created: "2026-09-13"
updated: "2026-09-13"
depends_on:
- FI-WP-0001
- FI-WP-0002
- FI-WP-0003
---
# FI-WP-0004 — Operational loop + Scaleway reserve
## Goal
The lab already has INTENT, a brief template, a catalog, and a weekday
schedule. It does **not** yet have a trusted daily field picture or a
reserve that can hold the models INTENT says to keep.
This workplan closes that gap:
1. **Sensing loop is durable.** A brief day is complete only when the file
exists on `origin/main`. Hub `fi_daily_brief` events that are not on
git are failures, not completions.
2. **Sensing quality is gated.** The executor must not re-announce cataloged
models, invent sizes, or skip axes C/D by default.
3. **Reserve uses Scaleway economics.** VAULT HD is local staging. The
source of truth for weight blobs is a dedicated Scaleway bucket on
`reef-storage` (One Zone IA for the R-spine, Glacier for S-tier).
The 850 GiB NAS quota is retired as the capacity gate; euros/month
become the gate.
## Why now
Review 2026-09-13: last brief on `origin/main` is 2026-08-14; hub events
for 2026-09-07…11 claim `committed=True` with no origin objects. Automated
briefs mis-sized DeepSeek-V4-Flash-0731 as a 12B dense model (it is a
~304B / ~13B-active MIT MoE, ~167 GiB) and never cataloged it. Kimi K3
was deferred only because a 1 TB disk quota could not hold ~1454 GiB;
on Scaleway Glacier that footprint is a few euros per month.
Org storage selection is already Scaleway (`RESOURCE-WP-0002`,
`reef-storage`). Do **not** reuse the postgres backup bucket (30-day
expiry). FI needs its own long-retention bucket.
## Tasks
### T01 — Decision: Scaleway object reserve
```task
id: FI-WP-0004-T01
status: done
priority: high
```
Write `docs/decisions/2026-09-13-scaleway-object-reserve.md`. Pin:
- SoT for blobs: Scaleway Object Storage `nl-ams` via `reef-storage`
- R-spine: One Zone IA (immediate get)
- S-tier: Glacier (restore is acceptable for just-in-case)
- VAULT HD: staging / optional local R cache, not SoT
- Soft euro budget (not GiB quota) as the capacity gate
- Dedicated bucket, never `railiance-platform-pg-backup`
**Done when:** decision file exists and is linked from storage policy.
### T02 — Policy and operator docs
```task
id: FI-WP-0004-T02
status: done
priority: high
```
Update `docs/backup-storage-policy.md`, `inventory/collection-policy.md`,
`docs/hardware-envelope.md`, `README.md`, `SCOPE.md`,
`docs/recurrence-ops.md` so they describe the live dual-loop contract
and Scaleway pin. Retire “workplans still active” drift from README.
**Done when:** an operator can find the bucket contract, euro gate, and
completion rule without reading INTENT.
### T03 — Brief durability contract + verifier
```task
id: FI-WP-0004-T03
status: done
priority: high
```
- Document: `fi_daily_brief` may fire only after the brief is on
`origin/main`.
- Add `scripts/verify_brief_durability.py` comparing hub events to git.
- Tighten `activity-definitions/fi-daily-research-brief.md` and the
playbook so a local-only commit does not clear `due`.
**Done when:** `python3 scripts/verify_brief_durability.py` fails on the
known Sep 2026 hub/git mismatch (or reports it as debt until those
events are reconciled).
### T04 — Executor origin publication (rein-aharness)
```task
id: FI-WP-0004-T04
status: done
priority: high
```
FI is the definition owner. Grant origin publication as a named
capability of `fi-research-brief` (reverses the 2026-09-04 local-only
commit):
- After commit, `git push origin HEAD:<branch>`
- `fi_daily_brief` only if push succeeded (`detail.pushed=true`,
`detail.origin_sha`)
- Push failure → `executor_run` failure, `due` stays true
- Prompt/context: catalog ids, no re-announce, no invented sizes
**Done when:** rein-aharness tests prove push is attempted and a failed
push does not look like a completed brief day.
### T05 — Catch-up brief + catalog the missed SOTA
```task
id: FI-WP-0004-T05
status: done
priority: high
```
Write `briefs/2026/09/2026-09-13.md` correcting V4-Flash and covering
the Aug 15Sep 12 gap. Catalog:
- `deepseek-ai/DeepSeek-V4-Flash-0731` as S (MIT, ~167 GiB) — **primary
next pull** once the bucket exists
- `Qwen/Qwen3.8-27B` as R/W candidate (Apache-2.0, dense ~27B)
Update `inventory/RESERVE-STATUS.md`.
**Done when:** catalog entries exist with correct size/class; catch-up
brief cites primary cards.
### T06 — `collect_model.py` S3 backend
```task
id: FI-WP-0004-T06
status: done
priority: medium
```
Local snapshot still stages on disk. After MANIFEST, optional upload to
the FI bucket with an explicit storage class. Credentials from env
(`AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` or `SCW_*`), never git.
**Done when:** `--s3-bucket` uploads and prints the object prefix.
### T07 — reef-storage bucket declaration (planned)
```task
id: FI-WP-0004-T07
status: done
priority: high
```
Add `reef-storage/substrate/object-stores/fi-open-weight-reserve.yaml`
(`status: planned`) and a create script that **must not** copy the
postgres 30-day expiry. Bind freedom-intelligence as a potential
consumer.
**Done when:** YAML validates against the reef-storage schema; create
script is documented. Bucket create itself is T08 (operator).
### T08 — Operator: create bucket and scoped key
```task
id: FI-WP-0004-T08
status: todo
priority: high
```
Operator / railiance-platform:
1. Create private bucket `railiance-fi-open-weight-reserve` in `nl-ams`.
2. Versioning on. **No 30-day object expiry.**
3. Lifecycle: prefix `strategic/` → Glacier after 1 day.
4. Scoped IAM key in OpenBao (not the postgres backup key).
5. Flip the reef-storage YAML to `status: active`.
6. Confirm a 1-byte PUT/GET.
**Done when:** attributes are live and a smoke object round-trips.
### T09 — First Scaleway collect (V4-Flash)
```task
id: FI-WP-0004-T09
status: todo
priority: medium
```
After T08: pull `deepseek-ai/DeepSeek-V4-Flash-0731` via
`scripts/collect_model.py` into staging, upload as `strategic/` Glacier
(or Standard then lifecycle). Verify MANIFEST vs objects. Update catalog
`status: collected` / `verified` and RESERVE-STATUS.
**Done when:** catalog `storage_path` is the S3 prefix and checksums match.
## Acceptance (workplan-level)
- [x] Scaleway reserve decision written
- [x] Docs no longer claim VAULT-as-SoT or “workplans still active”
- [x] Durability verifier exists
- [x] Executor publishes to origin or fails the day
- [x] V4-Flash cataloged at the real size
- [ ] Bucket live (T08, operator)
- [ ] First S-tier object on Scaleway (T09)
## Out of scope
- Reusing `railiance-platform-pg-backup`
- 24/7 inference of the reserve
- Changing Scaleway org billing (resource-control / founder alert)
- Recovering the unpushed 2026-09-07…11 railiance-local commits
(note in the catch-up brief; do not invent those files)