diff --git a/README.md b/README.md index db4ed0c..c7f4505 100644 --- a/README.md +++ b/README.md @@ -23,7 +23,7 @@ Standing lenses: **A** frontier & price · **B** edge/local open · **C** homela | [`briefs/`](briefs/) | Daily briefs + template | | [`inventory/`](inventory/) | Model catalog & collection policy | | [`workplans/`](workplans/) | Setup workplans (incl. activity-core) | -| [`activity-definitions/`](activity-definitions/) | Cron definition for daily brief (disabled until FI-WP-0002) | +| [`activity-definitions/`](activity-definitions/) | Weekday brief schedule (enabled; completion = origin/main) | --- @@ -31,23 +31,26 @@ Standing lenses: **A** frontier & price · **B** edge/local open · **C** homela | ID | Title | Status | | -- | ----- | ------ | -| [FI-WP-0001](workplans/FI-WP-0001-lab-operating-foundation.md) | Lab operating foundation | active (storage pin open) | -| [FI-WP-0002](workplans/FI-WP-0002-activity-core-daily-research.md) | Activity-core daily research rhythm | active (resolver/ops open) | -| [FI-WP-0003](workplans/FI-WP-0003-seed-open-weight-reserve.md) | Seed open-weight reserve | active (catalog seeded) | +| [FI-WP-0001](workplans/FI-WP-0001-lab-operating-foundation.md) | Lab operating foundation | done | +| [FI-WP-0002](workplans/FI-WP-0002-activity-core-daily-research.md) | Activity-core daily research rhythm | finished | +| [FI-WP-0003](workplans/FI-WP-0003-seed-open-weight-reserve.md) | Seed open-weight reserve | finished | +| [FI-WP-0004](workplans/FI-WP-0004-operational-loop-and-scaleway-reserve.md) | Durable loop + Scaleway reserve | **active** | --- ## Collection plan (bulk reserve) -**Storage:** `D:\vault\coulomb\freedom-intelligence\` (VAULT HD) · soft quota **850 GiB** +**SoT:** Scaleway Object Storage `nl-ams` · bucket `railiance-fi-open-weight-reserve` +**Gate:** **€15 / month** · R = One Zone IA · S = Glacier +**Staging:** VAULT HD `D:\vault\coulomb\freedom-intelligence\` **Policy:** store the **most capable open weights** even if unrunnable today -(`docs/decisions/2026-07-24-nas-strategic-reserve.md`). +(`docs/decisions/2026-09-13-scaleway-object-reserve.md`). | Tier | Examples | | ---- | -------- | | **R — runnable spine** | Qwen3-8B, Llama-3.2-3B-Instruct, BGE-M3, R1-Distill-14B, nomic-embed | -| **S — strategic** | DeepSeek-V3 (class), DeepSeek-R1 full, Llama-3.3-70B / Qwen3-72B dense | -| **W — optional** | 14B/32B upgrades, reranker, … | +| **S — strategic** | **DeepSeek-V4-Flash-0731** (first pull), Kimi K3, V3/R1 full, 70B-class | +| **W — optional** | Qwen3.8-27B, 14B/32B upgrades, reranker | Plan detail: `research/2026-07-24-nas-strategic-collection-plan.md` · catalog: `inventory/catalog/`. @@ -58,7 +61,7 @@ Plan detail: `research/2026-07-24-nas-strategic-collection-plan.md` · catalog: ```text freedom-intelligence/ ├── INTENT.md / SCOPE.md / README.md -├── activity-definitions/ # fi-daily-research-brief (enabled: false) +├── activity-definitions/ # fi-daily-research-brief (enabled; origin publish required) ├── briefs/ # daily deltas ├── research/ # deep surveys (baseline lives here) ├── inventory/catalog/ # YAML candidates / collected @@ -79,8 +82,14 @@ cp briefs/_template.md briefs/$(date +%Y/%m)/$(date +%Y-%m-%d).md # follow docs/daily-brief-playbook.md ``` -**Do not** bulk-download models until `docs/backup-storage-policy.md` path/quota -are pinned (FI-WP-0001-T04). +**Do not** bulk-download S-tier models until the Scaleway bucket is live +(FI-WP-0004-T08). Stage on VAULT; never into this git tree. + +**Durability check** (hub events vs git files): + +```bash +python3 scripts/verify_brief_durability.py --require-hub +``` --- diff --git a/SCOPE.md b/SCOPE.md index cec0918..1ea1a77 100644 --- a/SCOPE.md +++ b/SCOPE.md @@ -91,17 +91,17 @@ Git holds briefs, schemas, policies, and catalog metadata. Model weight blobs st | Area | Status | | ---- | ------ | -| Overall | **Operable lab** — foundation + **live** weekday brief schedule + R-spine reserve | -| INTENT / SCOPE | Drafted | +| Overall | **Repairing the operating loop** — foundation exists; origin publication + Scaleway SoT in FI-WP-0004 | +| INTENT / SCOPE | Live; SCOPE current-state refreshed 2026-09-13 | | Baseline survey | `research/2026-07-24-baseline-field-survey.md` | -| Briefs | 2026-07-24, 07-28, **08-03** (week catch-up); playbook + recurrence ops | -| Backup storage | **VAULT HD** `D:\vault\coulomb\freedom-intelligence\`; soft quota 850 GiB | -| Inventory | R verified; **Kimi K3** S1 approved+deferred (~1454 GiB); other S deferred; W candidate | -| Reserve policy | Capability-first — not gated by current VRAM | -| Activity-core | Definition **enabled on railiance**; Temporal schedule weekdays 07:30 Berlin | -| Executor | rein-aharness `fi-research-brief` via railiance user timer 07:35 (not workstation cron) | -| State Hub | Repo under **agents**; workplans finished + hub-indexed | -| Workplans | FI-WP-0001…0003 **finished** | +| Briefs | Origin last 2026-08-14; catch-up **2026-09-13**; weekday schedule still enabled | +| Backup storage | **Scaleway `nl-ams`** (bucket planned); VAULT HD is staging; €15/mo gate | +| Inventory | R verified on VAULT cache; **V4-Flash** S1 approved (not collected); K3 Glacier-affordable; Qwen3.8-27B candidate | +| Reserve policy | Capability-first — not gated by current VRAM; **not** gated by 850 GiB | +| Activity-core | Definition **enabled**; completion = brief on `origin/main` | +| Executor | rein-aharness `fi-research-brief` **must push** or the day fails | +| State Hub | Repo under **agents**; `fi_daily_brief` is not evidence without the git file | +| Workplans | FI-WP-0001…0003 finished; **FI-WP-0004 active** | --- @@ -121,12 +121,14 @@ Git holds briefs, schemas, policies, and catalog metadata. Model weight blobs st 1. ✅ `INTENT.md` / `SCOPE.md` / foundation docs 2. ✅ Brief template + playbook + sources allowlist 3. ✅ Baseline field survey + collection recommendations -4. ✅ Catalog candidates (P0/P1) -5. ✅ ActivityDefinition draft (disabled) -6. ⬜ Pin backup storage path and capacity budget (FI-WP-0001-T04) -7. ⬜ activity-core resolver + enable schedule (FI-WP-0002) -8. ⬜ Approve/download P0 reserve (FI-WP-0003) -9. ⬜ First daily delta brief (manual or after automation) +4. ✅ Catalog candidates (P0/P1) + V4-Flash correction +5. ✅ ActivityDefinition enabled +6. ✅ Pin backup storage (VAULT staging 2026-07-28; **Scaleway SoT 2026-09-13**) +7. ✅ activity-core resolver + weekday schedule +8. ✅ R-spine verified on VAULT cache (FI-WP-0003) +9. ⬜ Origin publication proven end-to-end on railiance (FI-WP-0004-T04 deploy) +10. ⬜ Scaleway bucket live (FI-WP-0004-T08) +11. ⬜ First S collect: V4-Flash (FI-WP-0004-T09) --- diff --git a/activity-definitions/fi-daily-research-brief.md b/activity-definitions/fi-daily-research-brief.md index 9425929..5e4a752 100644 --- a/activity-definitions/fi-daily-research-brief.md +++ b/activity-definitions/fi-daily-research-brief.md @@ -18,9 +18,10 @@ context_sources: bind_to: context.fi_brief # Resolver (FI-WP-0002-T03, activity-core): expected shape # {items: [{kind: "daily_brief", due: bool, date: "YYYY-MM-DD", -# last_run_at}]} — due is true when no fi_daily_brief progress event with -# detail.repo=freedom-intelligence exists for today (Europe/Berlin). -# Executing session must record that event on completion (idempotence). +# last_run_at}]} — due is true when no *successful origin-published* +# fi_daily_brief exists for today (Europe/Berlin). +# Completion event is valid only with detail.pushed=true and the file on +# origin/main. Local-only commits must not clear due (FI-WP-0004). --- # Freedom Intelligence Daily Research Brief @@ -46,8 +47,10 @@ action: description: > Produce briefs/YYYY/MM/YYYY-MM-DD.md per docs/daily-brief-playbook.md and briefs/_template.md. Cite primary sources. Flag collection candidates. - On completion post State Hub progress event_type=fi_daily_brief with - detail.repo=freedom-intelligence and detail.date. + Commit AND push to origin/main. Post fi_daily_brief only if + detail.pushed=true and detail.origin_sha is set. If push fails, post + executor_run ok=false so due stays true. Do not re-announce cataloged + models. Do not invent parameter counts. target_repo: freedom-intelligence priority: medium labels: ["freedom-intelligence", "research-brief", "automated"] diff --git a/briefs/2026/09/2026-09-13.md b/briefs/2026/09/2026-09-13.md new file mode 100644 index 0000000..1edb8c1 --- /dev/null +++ b/briefs/2026/09/2026-09-13.md @@ -0,0 +1,81 @@ +--- +date: 2026-09-13 +timezone: Europe/Berlin +author: grok +status: final +sources_checked: + - docs/sources-allowlist.md + - inventory/RESERVE-STATUS.md + - huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731 + - huggingface.co/Qwen/Qwen3.8-27B (card identity via public trackers) + - https://www.scaleway.com/en/pricing/storage/ +--- + +# Freedom Intelligence Daily Brief — 2026-09-13 + +> **Month catch-up.** Last file on `origin/main` is 2026-08-14. Hub +> `fi_daily_brief` events exist for 2026-09-07…11 with `committed=True` +> and no git objects. Those days are **not** recovered here. This file +> is a field + ops delta from 2026-08-14, not a resurrection of the +> missing executor commits. + +## Headline deltas + +- **Loop failure:** sensing ran locally on railiance and cleared `due` without publishing to origin. Rein-aharness dropped `git push` on 2026-09-04 as an integrity guard; FI now re-grants origin publication as a named completion requirement (FI-WP-0004). +- **DeepSeek-V4-Flash-0731 is not 12B.** Official card: MIT, 284B base / ~13B active (304B with DSpark), ~167 GiB HF tree, 2026-07-31. Automated briefs 08-04…14 re-announced it as a new 12B dense model (license flipped MIT/Apache) and 08-07 claimed it was already on disk. Catalog now records the real model as **S, approved, not collected**. +- **Qwen3.8-27B** (Apache-2.0 dense multimodal, 2026-08-14) and **Qwen3.8-2.4T-A95B** (custom Max licence, multi-TB, ~08-12) shipped in the hole. 27B is a W/R candidate; Max is S-watch only until euro budget is live. +- **Reserve facility:** VAULT HD 850 GiB quota is retired as SoT. Scaleway `nl-ams` object storage is the reserve (One Zone IA for R, Glacier for S). Kimi K3 (~1454 GiB) becomes a few euros/month on Glacier instead of a quota wall. +- **Next pull** after the bucket exists: V4-Flash official tree into `strategic/` Glacier — it always fitted even the old 850 GiB quota. + +## Frontier & commercial (axis A) + +| Item | Delta | Sources | Lab relevance | +| ---- | ----- | ------- | ------------- | +| DeepSeek-V4-Flash-0731 | Official open MIT MoE; API `deepseek-v4-flash`. Card scores Terminal Bench 2.1 82.7. | [HF card](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731), [arXiv 2606.19348](https://arxiv.org/abs/2606.19348) | Closed-bar comparison via llm-connect; open identity is the S pull | +| Qwen3.8-Max family | API GA early Aug; open 2.4T-A95B under **bespoke** licence (revenue threshold), not Apache | Qwen card / trackers (verify licence text at pin) | Do not collect the 2.4T tree until T08 budget + licence read | +| GLM-5.3-Flash (late Aug) | 320B-A18B, MIT, 1M ctx, reported $0.15 / $0.50 per MTok | Z.ai release notes (verify) | Watch vs V4-Flash as MIT S alternative; not displacing V4-Flash as first pull | + +## Edge / local / open (axis B) + +| Item | Delta | Sources | Lab relevance | +| ---- | ----- | ------- | ------------- | +| DeepSeek-V4-Flash-0731 size class | ~167 GiB official tree, ~13B active — **S, not R** | HF file listing ~167 GB; card “304B params” | First Scaleway Glacier collect | +| Qwen3.8-27B | Dense multimodal Apache-2.0, ~27B, 2026-08-14 — same calendar day as last origin brief | [HF Qwen/Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B) | Candidate R-spine upgrade; cataloged `candidate` | +| Muse Glimmer 30B (Meta, ~08-10) | Apache-2.0 dense multimodal, ~24–32 GB class | Meta/HF trackers (verify card) | Watch; do not pull until V4-Flash and 27B decision | +| Kimi K3 | Unchanged identity; capacity objection is now euros not GiB | existing catalog | Glacier ~€3.69/mo; still deferred until V4-Flash lands | + +## Training & specialization (axis C) + +| Item | Delta | Sources | Lab relevance | +| ---- | ----- | ------- | ------------- | +| Qwen3.8-27B as FT base | Dense 27B Apache-2.0 is a more serious QLoRA target than 8B if T2–T3 VRAM exists | Qwen3.8-27B card | Hardware envelope hosts still TBD — do not pretend we can FT it today | +| No new homelab recipe displacement | Unsloth / TRL / Axolotl still the working set from baseline | allowlist | No stack change | + +## Harness & fleet (axis D) + +| Item | Delta | Sources | Lab relevance | +| ---- | ----- | ------- | ------------- | +| FI executor publication | 2026-09-04 rein-aharness removed `git push`; hub still cleared `due` | `rein-aharness` REINAH-WP-0003, hub events 2026-09-07…11 | **Broken durable memory.** FI-WP-0004 re-grants origin push and verifier | +| V4-Flash agentic stack | Official vLLM DSpark + SGLang recipes on the card | HF card “How to Run” | Axis D watch; serve is not this repo | + +## Collection candidates + +| id | org | name | priority | reason | approx size | license | +| -- | --- | ---- | -------- | ------ | ----------- | ------- | +| deepseek-ai__DeepSeek-V4-Flash-0731__strategic | deepseek-ai | DeepSeek-V4-Flash-0731 | high | Corrected S1; MIT; ~167 GiB; first Scaleway pull | ~167 GiB | MIT | +| Qwen__Qwen3.8-27B__candidate | Qwen | Qwen3.8-27B | medium | Apache-2.0 dense 27B R-upgrade | ~56 GiB class | Apache-2.0 | + +Already in catalog, **not** new: Kimi K3, DeepSeek-V3, Qwen3-8B, R1-Distill-14B, BGE-M3, nomic-embed. + +## Lab implications + +- Finish FI-WP-0004: origin publish in the executor, durability verifier in CI/ops, Scaleway bucket (T08), then collect V4-Flash (T09). +- Do not treat hub `fi_daily_brief` as evidence the brief exists until `scripts/verify_brief_durability.py` is green. +- Recovering 2026-09-07…11 files from the railiance clone is optional; they were produced by the same failing loop and are not trusted. +- Hardware envelope hosts are still TBD — R-upgrade of Qwen3.8-27B waits on that as much as on disk. + +## Notes / caveats + +- Parameter counts and €/GB from primary cards/pricing pages today; re-measure at pull. +- Qwen3.8-Max open tree licence is **not** Apache-2.0; do not copy the 27B licence onto it. +- `DeepSeek-V4-Flash-0731-12B` is not a real repo id. diff --git a/docs/backup-storage-policy.md b/docs/backup-storage-policy.md index 796c1da..e9d0a7f 100644 --- a/docs/backup-storage-policy.md +++ b/docs/backup-storage-policy.md @@ -1,17 +1,19 @@ # Backup storage policy — open-weight model reserve -**Status:** facility + path pinned (workstation VAULT HD) +**Status:** facility pin **Scaleway Object Storage** (2026-09-13) **Related:** `INTENT.md`, `SCOPE.md`, `inventory/collection-policy.md`, -`docs/decisions/2026-07-24-nas-strategic-reserve.md` -**Adjacent:** `disaster-control` BackupPolicy (platform backups — different concern) +`docs/decisions/2026-09-13-scaleway-object-reserve.md` +**Supersedes storage pin:** 2026-07-28 VAULT HD (now staging only) +**Adjacent:** `disaster-control` BackupPolicy and `rapp-postgres` WAL +archive — **different buckets, different lifecycles** --- ## Purpose -Define where Freedom Intelligence stores **open-weight model blobs** (and justified -companions), how much capacity we allow, and how this reserve relates to other -lab backup facilities. +Define where Freedom Intelligence stores **open-weight model blobs** (and +justified companions), how much we allow ourselves to spend, and how this +reserve relates to other lab backup facilities. Git never stores weight tensors. Git stores: @@ -25,16 +27,16 @@ Git never stores weight tensors. Git stores: | Requirement | Policy | | ----------- | ------ | -| **Class** | Backup / bulk durable storage — **not** hot cluster PVCs or app disks | -| **Facility** | Workstation VAULT HD (`D:`) under `D:\vault\coulomb\` (interim pin 2026-07-28) | -| **Durability** | Survives WSL rebuilds; weights live on the VAULT volume, not the git/WSL root | -| **Performance** | Sequential read for restore/training pull is enough; low latency not required | -| **Access** | Operator and approved lab hosts only; not a public mirror | -| **Separation** | Do not co-mingle with age-encrypted operational backups (Forgejo dumps, k3s state) without a clear subdirectory and different lifecycle rules | +| **Class** | Backup / bulk durable object storage — **not** hot cluster PVCs | +| **Facility** | Scaleway Object Storage `nl-ams` via `reef-storage` | +| **Durability** | Provider 11-nines claim; independent of Host Europe `railiance01` and of the workstation HD | +| **Performance** | Sequential restore for training/inference pull is enough | +| **Access** | Operator and approved lab hosts; not a public mirror | +| **Separation** | **Never** the postgres backup bucket (`railiance-platform-pg-backup` expires at 30 days) | Platform backup paths such as `/opt/backup/railiance/{infra,cluster}/` and -`~/.cache/railiance/backups/` are **operational recovery** lanes. Model weights -belong on the **VAULT model tree**, not on railiance hot disks. +`~/.cache/railiance/backups/` are **operational recovery** lanes. Model +weights belong in the FI object bucket. --- @@ -42,56 +44,33 @@ belong on the **VAULT model tree**, not on railiance hot disks. | Field | Value | | ----- | ----- | -| **Host / facility** | Workstation-attached bulk HD **VAULT** (Windows `D:`) | -| **Raw capacity** | ~1.86 TB total (~1.76 TB free as of 2026-07-28 pin) | -| **Base path (Windows)** | `D:\vault\coulomb\freedom-intelligence\` | -| **Base path (WSL, when D: automounted)** | `/mnt/d/vault/coulomb/freedom-intelligence/` | -| **Layout under base** | See [On-disk layout](#on-disk-layout) | -| **Mount on lab hosts** | Local attach on operator workstation; WSL via drvfs `D:` → `/mnt/d` when enabled | -| **Credentials** | Local filesystem ACL on the workstation HD; never commit secrets | +| **Facility** | Scaleway Object Storage, region `nl-ams` | +| **Endpoint** | `https://s3.nl-ams.scw.cloud` | +| **Bucket** | `railiance-fi-open-weight-reserve` (planned until FI-WP-0004-T08) | +| **Attributes** | `reef:storage/substrate/object-stores/fi-open-weight-reserve.yaml` | +| **Credentials** | OpenBao / `railiance-platform` — never git | -**Interim note:** Earlier policy assumed a dedicated ~1 TB NAS. Operator chose this -VAULT volume under `D:\vault\coulomb\` for now. Soft quota **850 GiB** for the -model tree still applies (self-imposed discipline; disk is larger). +**Local staging (not SoT):** workstation VAULT HD +`D:\vault\coulomb\freedom-intelligence\` / +`/mnt/d/vault/coulomb/freedom-intelligence/` for `huggingface_hub` +downloads and an optional R-spine cache. -Do **not** bulk-download multi-GB models into this git workspace or into hot -root filesystems (`/` on WSL, `C:`). +Do **not** bulk-download multi-GB models into this git workspace or into +hot root filesystems (`/` on WSL, `C:`, railiance PVCs). -### Path shape (pinned) +### Prefix shape (pinned) ```text -D:\vault\coulomb\freedom-intelligence\ # Windows -/mnt/d/vault/coulomb/freedom-intelligence/ # WSL when D: mounted -├── models/ -│ └── {org}__{name}/ -│ └── {revision}/ -│ ├── blobs/ # weight files, shards -│ └── MANIFEST.txt # optional local copy of hashes -├── companions/ # adapters, tokenizers only when separate -└── staging/ # incomplete downloads; not catalog-ready +s3://railiance-fi-open-weight-reserve/ +├── models/{org}__{name}/{revision}/ # R / W (One Zone IA) +├── strategic/{org}__{name}/{revision}/ # S (Glacier) +├── companions/ +├── staging/ +└── manifests/{org}__{name}/{revision}/MANIFEST.json ``` Map each `{org}__{name}/{revision}` to an inventory catalog entry. - -Directories `models/`, `staging/`, and `companions/` were created 2026-07-28. - ---- - -## On-disk layout - -| Path element | Rule | -| ------------ | ---- | -| `org__name` | Hugging Face-style id with `/` → `__` | -| `revision` | Git commit SHA, tag, or release id used at download time | -| `blobs/` | Actual files; prefer original names from source | -| `staging/` | Incomplete transfers; purge or resume; never mark collected until complete + verified | - -Optional layout tags (directory name or catalog field): - -| Tag | Meaning | -| --- | ------- | -| runnable | Fits current run envelope (R tier) | -| strategic | Capability reserve; may exceed current hardware (S tier) | +`collection.storage_path` is the `s3://` URI. --- @@ -99,20 +78,24 @@ Optional layout tags (directory name or catalog field): | Parameter | Policy | | --------- | ------ | -| **Device** | VAULT HD (`D:`) — model tree under `D:\vault\coulomb\freedom-intelligence\` | -| **Soft quota (model reserve)** | **850 GiB** (self-imposed; disk is larger) | -| **Hard stop** | **920 GiB** used under the freedom-intelligence base path | -| **Growth review** | When catalog total ≥ **70% soft quota (~595 GiB)** | +| **Gate** | **Euros per month**, not GiB | +| **Soft budget** | **€15 / month** ex VAT for this bucket | +| **Hard backstop** | Founder Scaleway project billing alert (no provider euro hard-stop) | +| **Growth review** | When the running month is ≥ **70% of soft budget (~€10.50)** | | **Per-pull threshold** | See `inventory/collection-policy.md` | -| **Eviction** | Prefer drop **W** (watch) and easily re-obtained quants; protect unique **S** SOTA bases and the **R** spine | +| **Eviction** | Prefer drop **W** and easily re-obtained quants; protect unique **S** SOTA bases and the **R** spine | -### Portfolio guidance on 1 TB +Rough cost (2026-09-13 Scaleway list prices): -* Always keep the **runnable spine (R / former P0)** resident. -* Use remaining space for a **small number of most-capable open models (S)**, - preferably well-provenance compressed weights when full precision would exhaust the disk. -* Do **not** attempt to mirror entire HF orgs. -* Full bf16 of multiple 600B-class models will **not** fit; prioritize top capability identities. +| Mix | Class | ≈ € / month | +| --- | ----- | ----------: | +| R-spine ~45 GiB | One Zone IA | 0.54 | +| V4-Flash ~167 GiB | Glacier | 0.42 | +| Kimi K3 ~1454 GiB | Glacier | 3.69 | +| Those three together | | **~4.65** | + +Glacier restore is €0.009/GB — pay it when we actually need the weights +on a GPU host. --- @@ -121,12 +104,14 @@ Optional layout tags (directory name or catalog field): For every completed collection: 1. Record source URL and revision in the catalog entry. -2. Store checksums (`sha256` of each blob or upstream manifest digest). +2. Store checksums (`sha256` of each blob or upstream manifest digest) + in `MANIFEST.json` (local staging **and** `manifests/` prefix). 3. Record download date (UTC) and downloader identity. 4. Prefer official org releases over anonymous re-uploads. 5. Keep license text or SPDX id in catalog; refuse unclear licenses. -Verification: **catalog claims must match on-disk checksums** before status `collected`. +Verification: **catalog claims must match object checksums** before +status `collected`. --- @@ -134,11 +119,11 @@ Verification: **catalog claims must match on-disk checksums** before status `col | Class | Retention | | ----- | --------- | -| **Strategic capability (S)** | Keep until explicit deprecation or displacement by a clearly stronger open successor | -| **Runnable spine (R)** | Keep while still the lab default for local ops / FT | +| **Strategic capability (S)** | Keep until explicit deprecation; Glacier | +| **Runnable spine (R)** | Keep while still the lab default; One Zone IA | | **Working set** | Active experiment bases + optional one superseded revision | | **Staging** | Max 14 days incomplete, then purge | -| **Deprecated / evicted** | Metadata retained in catalog; blobs may be deleted | +| **Deprecated / evicted** | Metadata retained in catalog; objects may be deleted | --- @@ -146,35 +131,37 @@ Verification: **catalog claims must match on-disk checksums** before status `col | Topic | Policy | | ----- | ------ | -| **At rest** | NAS default; extra age/GPG of multi-hundred-GB trees optional | -| **In transit** | Trusted lab network / local attach | -| **Offsite copy** | Optional later; coordinate with disaster-control if added so model bulk does not break ops backup SLAs | +| **At rest** | Scaleway provider-managed | +| **In transit** | TLS to `s3.nl-ams.scw.cloud` | +| **Workstation copy** | Optional R-spine on VAULT; not a second SoT | +| **Ops backups** | Do not mix with Forgejo/k3s dumps | --- ## What must not live here -* Closed weights or artifacts whose terms forbid offline retention -* Secrets, API keys, customer data, or ungoverned training corpora with personal data -* Operational backups (databases, k3s state, Forgejo dumps) as the primary home -* Git LFS dumps of full model trees as a substitute for the NAS +* Closed weights or artifacts whose terms forbid offline retention +* Secrets, API keys, customer data, or ungoverned training corpora +* Operational backups (databases, k3s state, Forgejo dumps) +* Git LFS dumps of full model trees +* Objects in `railiance-platform-pg-backup` --- ## Operator checklist -- [x] Choose facility: bulk durable storage for model reserve (2026-07-24 intent) -- [x] Set soft quota: **850 GiB** / hard stop **920 GiB** -- [x] Pin path: **`D:\vault\coulomb\freedom-intelligence\`** (2026-07-28) -- [x] Create `models/`, `staging/`, `companions/` -- [x] Document Windows + WSL path forms above -- [x] Ensure WSL can see `D:` (`/mnt/d`) when downloads run from Linux (verified 2026-07-28) +- [x] Choose facility: Scaleway object storage (2026-09-13) +- [x] Set capacity gate: **€15 / month** soft +- [x] Pin prefix layout above +- [ ] Create bucket + scoped key (FI-WP-0004-T08) +- [ ] Smoke PUT/GET +- [ ] First S collect: DeepSeek-V4-Flash-0731 (FI-WP-0004-T09) ### Pin log | Date | Operator | Change | | ---- | -------- | ------ | | 2026-07-24 | foundation | Policy created; path unpinned | -| 2026-07-24 | operator direction | Facility intent = 1 TB local NAS; soft quota 850 GiB; strategic unrunnable models in scope | -| 2026-07-28 | operator direction | **Interim pin:** workstation VAULT HD `D:\vault\coulomb\freedom-intelligence\`; layout created; soft quota retained | -| 2026-07-28 | operator | WSL mount verified: `D:` → `/mnt/d` (1.9T, writable) | +| 2026-07-24 | operator direction | Facility intent = 1 TB local NAS; soft quota 850 GiB | +| 2026-07-28 | operator direction | **Interim pin:** workstation VAULT HD | +| 2026-09-13 | FI-WP-0004 | **SoT pin:** Scaleway `nl-ams`; VAULT demoted to staging; euro gate replaces GiB quota | diff --git a/docs/daily-brief-playbook.md b/docs/daily-brief-playbook.md index b128945..11fb03c 100644 --- a/docs/daily-brief-playbook.md +++ b/docs/daily-brief-playbook.md @@ -22,7 +22,8 @@ full resurveys. | One brief per date | Path `briefs/YYYY/MM/YYYY-MM-DD.md` | | Calendar | Prefer **Europe/Berlin** date in front matter | | Skip empty days | Either omit file or write `*No material delta.*` in Headline deltas | -| Automation due bit | Cleared only by State Hub event `fi_daily_brief` (below) | +| Automation due bit | Cleared only by State Hub event `fi_daily_brief` **after** the brief exists on `origin/main` | +| Hub without git | A `fi_daily_brief` event with no matching file is a **false completion**. Run `scripts/verify_brief_durability.py`. | --- @@ -46,17 +47,20 @@ full resurveys. 4. **Collection candidates** - Use inventory field names; set priority + - Do **not** re-nominate a model already in `inventory/catalog/` + - Do **not** invent parameter counts; cite the model card or mark unverified - If nominating: add or update `inventory/catalog/{id}.yaml` with `status: candidate` and `collection.brief_refs` 5. **Lab implications** - Link open workplans (`FI-WP-*`) only when action is real -6. **Commit** (operator policy) — brief + any catalog YAML +6. **Commit and push** — brief + any catalog YAML must land on `origin/main`. + A local-only commit does **not** complete the day. 7. **Completion evidence** (required for activity-core rhythm) - Post State Hub progress (HTTP or MCP), shape: + Post State Hub progress **only after** origin has the commit: ```yaml event_type: fi_daily_brief @@ -67,8 +71,13 @@ full resurveys. date: "YYYY-MM-DD" path: "briefs/YYYY/MM/YYYY-MM-DD.md" collection_candidates: 0 + pushed: true + origin_sha: "" ``` + If the brief is written but push fails, post `event_type: executor_run` + with `ok: false` so `due` stays true. + Workstation API often `http://127.0.0.1:8000`; remote tunnel ports per Custodian docs. See `docs/state-hub-delivery.md`. @@ -85,8 +94,10 @@ When activity-core emits `activity_task_spawn` with 1. Railiance host timer (or manual) runs `rein-aharness fi-research-brief --target-repo ~/freedom-intelligence` -2. llm-connect drafts the brief; harness commits and posts **`fi_daily_brief`** -3. Idempotency: one brief file per date; resolver clears `due` on completion event +2. llm-connect drafts the brief; harness **commits, pushes to origin**, then + posts **`fi_daily_brief`** with `pushed=true` +3. Idempotency: one brief file per date on origin; resolver clears `due` only + then. Local commit without push must not clear `due`. Do **not** install Claude Code / Codex on railiance01 for this rhythm. @@ -116,9 +127,10 @@ make -C ~/activity-core sync-activity-definitions | Check | Pass | | ----- | ---- | | Signal | No filler paragraphs | -| Sources | At least one primary link per non-empty axis row | -| Honesty | Benchmarks marked directional; prices “verify before budget” | -| Collection | License noted for any candidate | +| Sources | At least one primary URL per non-empty axis row | +| Honesty | Benchmarks marked directional; prices “verify before budget”; sizes from the card | +| Collection | License + real size/class; never re-announce a cataloged id as a new release | +| Axes | C and D must be scanned; `*(none)*` is allowed only after a look | | Length | Prefer one screen to three; deep dives go under `research/` | --- diff --git a/docs/decisions/2026-09-13-scaleway-object-reserve.md b/docs/decisions/2026-09-13-scaleway-object-reserve.md new file mode 100644 index 0000000..3aea491 --- /dev/null +++ b/docs/decisions/2026-09-13-scaleway-object-reserve.md @@ -0,0 +1,136 @@ +# Decision: Scaleway object storage as the open-weight reserve + +**Date:** 2026-09-13 +**Status:** accepted +**Affects:** `docs/backup-storage-policy.md`, `inventory/collection-policy.md`, +`docs/hardware-envelope.md`, `scripts/collect_model.py`, FI-WP-0004 +**Supersedes (storage backend only):** +`docs/decisions/2026-07-24-nas-strategic-reserve.md` § storage facility. +The **capability-first** rule from that decision stands. + +--- + +## Decision + +1. **Source of truth for weight blobs** is a dedicated Scaleway Object + Storage bucket in `nl-ams`, operated as `reef-storage`, consumed by + this repo. Git still holds catalog metadata only. + +2. **VAULT HD** (`D:\vault\coulomb\freedom-intelligence\`) is **staging + and optional local R-spine cache**. It is not the strategic envelope. + +3. **Storage class follows tier:** + - **R (runnable spine):** One Zone IA — cheap, GET is immediate. + - **S (strategic):** Glacier — cheapest just-in-case; restore cost + and latency are acceptable. + - **W:** only with euro-budget headroom; One Zone IA or Glacier by size. + +4. **Capacity gate is euros/month, not a 850 GiB disk quota.** The 850 / + 920 GiB numbers were a 1 TB NAS artifact. Usage-based object storage + removes that wall. Soft infrastructure budget: **€15 / month** + (ex VAT) for this bucket. Founder Scaleway billing alert remains the + hard backstop (`RESOURCE-WP-0002`). + +5. **Do not reuse** `railiance-platform-pg-backup`. That bucket expires + objects at 30 days. Model weights must not. + +--- + +## Why Scaleway (economics) + +Org already selected Scaleway over Hetzner / Host Europe for S3 +(`resource-control` RESOURCE-WP-0002, 2026-08-14). Same project, same +failure domain independence from Host Europe `railiance01`. + +Prices from [Scaleway storage pricing](https://www.scaleway.com/en/pricing/storage/) +as of 2026-09-13, ex tax: + +| Class | € / GB-month | 45 GiB (R spine) | 167 GiB (V4-Flash) | 1454 GiB (Kimi K3) | +| ----- | -----------: | ---------------: | -----------------: | -----------------: | +| Multi-AZ Standard | ~0.0146 | €0.66 | €2.44 | €21.23 | +| One Zone IA | ~0.012 | €0.54 | €2.00 | €17.45 | +| Glacier | ~0.00254 | €0.11 | €0.42 | **€3.69** | + +Glacier restore to Standard is €0.009/GB (V4-Flash restore ≈ €1.50; +K3 restore ≈ €13). Ingress is included. Egress is 75 GB/month free, +then €0.01/GB — relevant when we actually pull a giant to a GPU host, +not when we park it. + +**Portfolio that the 850 GiB NAS could not hold:** + +| Item | Class | Est. monthly | +| ---- | ----- | -----------: | +| R-spine resident (One Zone IA) | R | €0.54 | +| DeepSeek-V4-Flash-0731 (Glacier) | S | €0.42 | +| Kimi K3 official tree (Glacier) | S | €3.69 | +| **Total** | | **~€4.65** | + +That is inside the €15 soft budget with room for a second compressed +giant or a Qwen3.8-27B R upgrade. Buying another multi-TB disk to +fit K3 would spend the same money in a year and stay on the +workstation failure domain. + +VAULT remains useful: sequential staging for `huggingface_hub` +downloads, and a local copy of the R-spine if the workstation is +doing inference. It is no longer the thing INTENT means by “reserve.” + +--- + +## Bucket contract + +| Field | Value | +| ----- | ----- | +| Provider | Scaleway Object Storage | +| Region | `nl-ams` | +| Endpoint | `https://s3.nl-ams.scw.cloud` | +| Bucket (planned) | `railiance-fi-open-weight-reserve` | +| Reef attributes | `reef-storage/substrate/object-stores/fi-open-weight-reserve.yaml` | +| Public access | disabled | +| Versioning | on | +| Object expiry | **none** | +| Lifecycle | `strategic/` → Glacier after 1 day | +| Encryption | provider-managed at rest | +| Identity | bucket-scoped key, not the postgres backup key | +| Secrets | `railiance-platform` / OpenBao — never this git repo | + +### Prefix layout + +```text +s3://railiance-fi-open-weight-reserve/ +├── models/{org}__{name}/{revision}/ # R / W (One Zone IA at PUT) +├── strategic/{org}__{name}/{revision}/ # S (Glacier via lifecycle) +├── companions/ +├── staging/ # incomplete uploads; purge after 14d +└── manifests/{org}__{name}/{revision}/MANIFEST.json +``` + +Each prefix maps to one catalog YAML. `collection.storage_path` is the +`s3://` URI, not a VAULT path. + +--- + +## What stays from 2026-07-24 + +- Run envelope ≠ reserve envelope. +- License and integrity first. +- Catalog in git, blobs off git. +- No public mirror, no closed weights. +- R-spine always kept; S is a small number of top open identities. + +## What changes + +| Before | After | +| ------ | ----- | +| Facility = workstation VAULT HD | Facility = Scaleway `nl-ams` | +| Soft quota 850 GiB | Soft budget €15/month | +| K3 deferred on capacity | K3 economically in-scope on Glacier | +| V4-Flash uncollectable-as-12B | V4-Flash (~167 GiB MIT MoE) is the first S pull | + +--- + +## Follow-ups + +- [x] FI-WP-0004-T01 this file +- [ ] FI-WP-0004-T08 operator creates the bucket +- [ ] FI-WP-0004-T09 first collect: DeepSeek-V4-Flash-0731 +- [ ] resource-control inventory record (optional; FI consumes, resource-control costs) diff --git a/docs/hardware-envelope.md b/docs/hardware-envelope.md index 2324fa0..1c31220 100644 --- a/docs/hardware-envelope.md +++ b/docs/hardware-envelope.md @@ -1,6 +1,6 @@ # Hardware envelope — run vs reserve -**Status:** updated 2026-07-24 (1 TB NAS strategic reserve) +**Status:** updated 2026-09-13 (Scaleway object reserve; run envelope unchanged) **Used by:** inventory `hardware_class`, axis B/C, collection tiers R/S --- @@ -10,10 +10,10 @@ Separate two questions that used to be conflated: 1. **What can we run or fine-tune now?** → *run envelope* -2. **What should we store on the NAS just in case?** → *reserve envelope* (1 TB) +2. **What should we store just in case?** → *reserve envelope* (Scaleway Glacier / One Zone IA) Collecting a model **does not** require it to fit the run envelope. -See `docs/decisions/2026-07-24-nas-strategic-reserve.md`. +See `docs/decisions/2026-09-13-scaleway-object-reserve.md`. --- @@ -34,13 +34,15 @@ valid for NAS collection. --- -## Reserve envelope (NAS) +## Reserve envelope (object store) | Parameter | Value | | --------- | ----- | -| Facility | VAULT HD `D:\vault\coulomb\freedom-intelligence\` (soft quota 850 GiB) | -| Soft quota | 850 GiB | -| Hard stop | 920 GiB | +| Facility | Scaleway Object Storage `nl-ams` (`railiance-fi-open-weight-reserve`) | +| Soft budget | **€15 / month** | +| R class | One Zone IA | +| S class | Glacier | +| Staging | VAULT HD `D:\vault\coulomb\freedom-intelligence\` | | Policy | `docs/backup-storage-policy.md` | --- @@ -51,7 +53,8 @@ valid for NAS collection. | ---- | ---- | --- | ---- | --- | ----- | | TBD | primary local inference | TBD | TBD | TBD | Run envelope | | TBD | training experiments | TBD | TBD | TBD | Run envelope | -| **VAULT HD (D:)** | weight store | n/a | n/a | n/a | **Reserve envelope** — soft 850 GiB | +| **VAULT HD (D:)** | staging / R cache | n/a | n/a | n/a | **Not SoT** — downloads and optional local R | +| **Scaleway nl-ams** | weight SoT | n/a | n/a | n/a | **Reserve envelope** — €15/mo soft | | railiance01 | cluster | n/a | n/a | n/a | Do not store weight trees on hot disks | --- @@ -76,7 +79,7 @@ Strategic giants may remain **cold storage only** until hardware upgrades. | QLoRA 7–8B | T1–T2 | | QLoRA 14B | T2 | | Full FT 7B | T2–T3 | -| 70B+ / large MoE FT | Future facility — weights may already sit on NAS | +| 70B+ / large MoE FT | Future facility — weights may already sit on Glacier | --- @@ -86,3 +89,4 @@ Strategic giants may remain **cold storage only** until hardware upgrades. | ---- | ------ | | 2026-07-24 | Draft tiers | | 2026-07-24 | Split run vs reserve; NAS 1 TB; S tier may exceed run envelope | +| 2026-09-13 | Reserve envelope moved to Scaleway; VAULT is staging; euro gate | diff --git a/docs/recurrence-ops.md b/docs/recurrence-ops.md index 7b0b5f9..f880bcc 100644 --- a/docs/recurrence-ops.md +++ b/docs/recurrence-ops.md @@ -19,7 +19,8 @@ activity-core (when / what / where) rein-aharness on railiance01 (does the work) primary: claim-loop → ops_run → fi-research-brief (REIN-A-0002) break-glass only: host timer 07:35 (disabled 2026-08-05; re-enable if claim loop down) - llm-connect drafts brief → commit → push → fi_daily_brief + llm-connect drafts brief → commit → git push origin → fi_daily_brief + (fi_daily_brief is forbidden until origin has the file) Operator morning review (no AI) from this repo: ```bash @@ -118,7 +119,19 @@ detail: collection_candidates: ``` -Posted automatically by `rein-aharness fi-research-brief` on success. +Posted automatically by `rein-aharness fi-research-brief` **only after** +`git push origin` succeeds. `detail.pushed` and `detail.origin_sha` are +required. A local commit with a failed push is `executor_run` / `ok=false`. + +FI is the definition owner: origin publication is a **granted** capability +of this rhythm (FI-WP-0004-T04), not an ungoverned env widening of +`FI_RESEARCH_BRIEF_PUSH`. + +Verify drift: + +```bash +python3 scripts/verify_brief_durability.py --require-hub +``` --- @@ -127,4 +140,6 @@ Posted automatically by `rein-aharness fi-research-brief` on success. - Workstation `crontab` that writes the brief (bypasses activity-core) - Installing Claude Code / Codex on railiance01 for this rhythm - Treating host timers as the scheduler of record -- Skipping the completion event (leaves `due=true` forever) +- Skipping the completion event (leaves `due=true` forever) +- Posting `fi_daily_brief` for a local-only commit (clears `due`, loses the day) +- Reusing `railiance-platform-pg-backup` for weights (30-day expiry) diff --git a/docs/state-hub-delivery.md b/docs/state-hub-delivery.md index a45a8dc..44f6408 100644 --- a/docs/state-hub-delivery.md +++ b/docs/state-hub-delivery.md @@ -19,7 +19,8 @@ Do not create workplans via hub `create_workplan` APIs. ## Required: completion evidence for daily rhythm -Every completed brief day (including “no material delta”) should post: +Every completed brief day (including “no material delta”) should post +**after the file is on `origin/main`**: ```yaml event_type: fi_daily_brief @@ -30,11 +31,15 @@ detail: date: "YYYY-MM-DD" path: "briefs/YYYY/MM/YYYY-MM-DD.md" collection_candidates: 0 + pushed: true + origin_sha: "" ``` This is the **idempotence signal** for activity-core context query `fi_brief_status` (FI-WP-0002-T03): `due=false` when an event exists for today -(Europe/Berlin) with `detail.repo=freedom-intelligence`. +(Europe/Berlin) with `detail.repo=freedom-intelligence` **and** +`detail.pushed=true`. A hub event without a git file is a false completion +(`scripts/verify_brief_durability.py`). Inbound automation uses `activity_task_spawn` from activity-core (state-hub sink). See `activity-core/docs/task-emission-consumer-contract.md`. diff --git a/inventory/README.md b/inventory/README.md index 0ee40d2..b184243 100644 --- a/inventory/README.md +++ b/inventory/README.md @@ -1,8 +1,8 @@ # Model inventory In-repo **catalog of open-weight revisions** reserved (or nominated) for the lab. -Weight blobs live on the **VAULT HD** at `D:\vault\coulomb\freedom-intelligence\` — -see `docs/backup-storage-policy.md`. +Weight blobs live on **Scaleway Object Storage** (`s3://railiance-fi-open-weight-reserve/…`). +VAULT HD is staging only — see `docs/backup-storage-policy.md`. ## Layout diff --git a/inventory/RESERVE-STATUS.md b/inventory/RESERVE-STATUS.md index 1387022..4ff537d 100644 --- a/inventory/RESERVE-STATUS.md +++ b/inventory/RESERVE-STATUS.md @@ -1,8 +1,10 @@ # Open-weight reserve status -**As of:** 2026-08-03 -**Store:** `/mnt/d/vault/coulomb/freedom-intelligence/` (`D:\vault\coulomb\freedom-intelligence\`) -**Soft quota:** 850 GiB · **Hard stop:** 920 GiB +**As of:** 2026-09-13 +**SoT store:** Scaleway Object Storage `nl-ams` bucket +`railiance-fi-open-weight-reserve` (**planned** — FI-WP-0004-T08) +**Staging:** `/mnt/d/vault/coulomb/freedom-intelligence/` (VAULT HD) +**Capacity gate:** **€15 / month** soft (not 850 GiB) --- @@ -10,64 +12,70 @@ | Class | Count | Status mix | | ----- | ----: | ---------- | -| Catalog entries | 13 | + Kimi K3 S1 | -| **verified** on disk | 4 | R embeds + Qwen3-8B + R1-Distill-14B | -| **approved** (pull deferred) | S giants + Llama edge + **Kimi K3** | capacity / HF gate | -| **candidate** (W) | 3 | headroom-gated | +| Catalog entries | 15 | + V4-Flash S, + Qwen3.8-27B W/R | +| **verified** on VAULT | 4 | R embeds + Qwen3-8B + R1-Distill-14B | +| **approved**, blobs missing | S giants + Llama-3.2-3B + **V4-Flash** | HF gate / bucket not live | +| **candidate** | 4 | W + Qwen3.8-27B | -### Disk use (model tree) +### Disk / object use -| Path | Role | Approx | -| ---- | ---- | ------ | -| `models/nomic-ai__nomic-embed-text-v1.5/main` | R embed | ~0.5 GiB verified | -| `models/BAAI__bge-m3/main` | R embed | ~2.1 GiB verified | -| `models/Qwen__Qwen3-8B/main` | R instruct | ~15.3 GiB verified | -| `models/deepseek-ai__DeepSeek-R1-Distill-Qwen-14B/main` | R reason | ~27.5 GiB verified | +| Location | Role | Approx | +| -------- | ---- | ------ | +| VAULT `models/nomic-ai__nomic-embed-text-v1.5/main` | R embed (staging cache) | ~0.5 GiB verified | +| VAULT `models/BAAI__bge-m3/main` | R embed (staging cache) | ~2.1 GiB verified | +| VAULT `models/Qwen__Qwen3-8B/main` | R instruct (staging cache) | ~15.3 GiB verified | +| VAULT `models/deepseek-ai__DeepSeek-R1-Distill-Qwen-14B/main` | R reason (staging cache) | ~27.5 GiB verified | +| Scaleway bucket | SoT | **empty** (not created) | -**~45 GiB** used → **~800+ GiB** soft-quota headroom (insufficient for full Kimi K3 ~1454 GiB). - -VAULT free (2026-08-03): ~1.7 TB free of 1.9 TB — full K3 would still dominate the volume and **violate soft quota**. +VAULT ~45 GiB used remains a **cache**, not the reserve. --- -## Decision matrix (2026-08-03) +## Decision matrix (2026-09-13) -| Id | Tier | Decision | Disk | -| -- | ---- | -------- | ---- | -| Qwen3-8B | R | **verified** | yes | +| Id | Tier | Decision | Blobs | +| -- | ---- | -------- | ----- | +| Qwen3-8B | R | **verified** on VAULT cache | local only | | Llama-3.2-3B-Instruct | R | **approved** | deferred — HF gate | -| BGE-M3 | R | **verified** | yes | -| R1-Distill-Qwen-14B | R | **verified** | yes | -| nomic-embed-text-v1.5 | R | **verified** | yes | -| **Kimi-K3** | **S1** | **approved**, **collection deferred** | no — ~1454 GiB > soft quota | -| DeepSeek-V3 | S1→secondary | **approved**, deferred | no — K3 is primary open SOTA identity | +| BGE-M3 | R | **verified** on VAULT cache | local only | +| R1-Distill-Qwen-14B | R | **verified** on VAULT cache | local only | +| nomic-embed-text-v1.5 | R | **verified** on VAULT cache | local only | +| **DeepSeek-V4-Flash-0731** | **S1** | **approved** — **first Scaleway pull** | no — ~167 GiB Glacier | +| Kimi-K3 | S1 secondary | **approved**, deferred until after V4-Flash | no — ~1454 GiB Glacier ≈ €3.69/mo | +| DeepSeek-V3 | S superseded identity | **approved**, deferred | no — V4-Flash is the collectable S1 | | DeepSeek-R1 full | S2 | **approved**, deferred | no | | Qwen3-72B | S4 | **approved**, deferred | no | | Llama-3.3-70B-Instruct | S4 | **approved**, deferred | no | +| Qwen3.8-27B | W/R | **candidate** | no | | Qwen3-14B | W | candidate | no | | R1-Distill-32B | W | candidate | no | | bge-reranker-v2-m3 | W | candidate | no | +**Correction:** there is no `DeepSeek-V4-Flash-0731-12B`. Automated briefs +that used that id were wrong. Do not collect under that name. + --- ## Hardware coverage gaps | Need | Status | | ---- | ------ | -| Default local instruct (8B) | **Covered** (Qwen3-8B) | +| Default local instruct (8B) | **Covered** (Qwen3-8B cache) | +| Stronger local instruct (27B) | **Candidate** Qwen3.8-27B | | Edge micro-agent | Llama-3.2-3B blocked on HF auth | | Multilingual RAG embed | **Covered** (BGE-M3 + nomic) | | Local reason | **Covered** (R1-Distill-14B) | -| Frontier-open general MoE offline | **Cataloged** Kimi K3 — pull blocked on capacity | -| Full open reasoner MoE | Deferred S2 | +| Frontier-open MIT MoE offline | **Cataloged** V4-Flash — pull blocked on bucket | +| Full open giant (K3) | Cataloged; Glacier-affordable; after V4-Flash | --- ## Next pulls (operator) -1. **Capacity decision for Kimi K3** (raise soft quota / dedicate disk / official compressed only). -2. Set `HF_TOKEN` and pull Llama-3.2-3B-Instruct. -3. Optional S4: one of Qwen3-72B vs Llama-3.3-70B under remaining quota. -4. DeepSeek V4-Flash-0731 as mid open value candidate (catalog later if desired). +1. **FI-WP-0004-T08** create `railiance-fi-open-weight-reserve` (no 30-day expiry). +2. **FI-WP-0004-T09** collect `deepseek-ai/DeepSeek-V4-Flash-0731` → `strategic/` Glacier. +3. Set `HF_TOKEN` and pull Llama-3.2-3B-Instruct into `models/` One Zone IA. +4. Decide Qwen3.8-27B vs keeping Qwen3-8B as the R instruct default. +5. Optional: Kimi K3 on Glacier if the month is still under €15. -Tool: `scripts/collect_model.py` (weights-only, sequential, writes `MANIFEST.json`). +Tool: `scripts/collect_model.py` (weights-only, sequential, MANIFEST, optional `--s3-bucket`). diff --git a/inventory/catalog/Qwen__Qwen3.8-27B__candidate.yaml b/inventory/catalog/Qwen__Qwen3.8-27B__candidate.yaml new file mode 100644 index 0000000..d3af458 --- /dev/null +++ b/inventory/catalog/Qwen__Qwen3.8-27B__candidate.yaml @@ -0,0 +1,74 @@ +id: Qwen__Qwen3.8-27B__candidate +status: candidate +name: Qwen3.8-27B +org: Qwen +source: + kind: huggingface + url: https://huggingface.co/Qwen/Qwen3.8-27B + revision: main + model_card_url: https://huggingface.co/Qwen/Qwen3.8-27B + project_url: https://qwen.ai/ +license: + spdx: Apache-2.0 + url: https://huggingface.co/Qwen/Qwen3.8-27B + allows_offline_retention: true + allows_local_ops: true + allows_fine_tune: true + notes: "Apache 2.0 on the 27B dense multimodal card (2026-08-14). Confirm at pin. Not the Qwen3.8-Max bespoke licence." +profile: + summary: "Dense 27B multimodal Apache-2.0 — candidate R-spine upgrade vs Qwen3-8B." + original_source: https://huggingface.co/Qwen/Qwen3.8-27B + use_cases: + - "Local instruct / vision on T2–T3 (24 GB class at Q4)" + - "Coding and office agents that need more than 8B" + - "Homelab QLoRA base" + sweet_spots: + - "Apache-2.0, ungated, actually runnable unlike V4-Flash / K3" + - "Native vision; 262k context (YaRN toward 1M)" + not_ideal_for: + - "Replacing the 8B always-on default until measured" + - "Standing in for Qwen3.8-2.4T-A95B (different licence, multi-TB)" + capability_notes: > + Released 2026-08-14, the day the automated brief last landed on + origin — so the sensing loop never recorded it. Dense 27.8B, + multimodal, Apache 2.0. Distinct from Qwen3.8-2.4T-A95B (custom + Max licence, ~4.89 TB). Nominate as W/R; do not pull until euro + budget and R-spine review after V4-Flash lands. +swot: + strengths: + - "Apache-2.0 dense multimodal at a size we can actually serve later" + - "Likely better local default than Qwen3-8B if VRAM allows" + weaknesses: + - "Larger than current verified R instruct (8B)" + - "Hardware envelope hosts still TBD" + opportunities: + - "R-spine upgrade path that does not need Glacier" + - "Vision in the local working set" + threats: + - "Qwen 4.x could supersede before we pull" +size: + total_bytes: 0 + total_human: "~56 GiB class (reports); measure at pull — Q4 much smaller" +hardware_class: + min_vram_gb_q4: 20 + min_vram_gb_fp16: 56 + notes: "T2 stretch / T3. Not T1." +axes: [B, C] +priority: medium +collection: + approved_by: "" + approved_at: null + downloaded_at: null + downloaded_by: "" + storage_path: "" + brief_refs: + - briefs/2026/09/2026-09-13.md +reason: "Missed by the Aug 14 brief cutoff; Apache-2.0 dense 27B is the obvious R-upgrade candidate." +tags: [tier-w, tier-r, instruct, vision, apache] +companions: [] +notes: "Do not confuse with Qwen3.8-2.4T-A95B (S-class, custom licence, multi-TB)." +history: + - at: "2026-09-13" + event: nominated + by: grok + detail: "Catch-up brief after cadence hole. Candidate only." diff --git a/inventory/catalog/deepseek-ai__DeepSeek-V4-Flash-0731__strategic.yaml b/inventory/catalog/deepseek-ai__DeepSeek-V4-Flash-0731__strategic.yaml new file mode 100644 index 0000000..a8b0753 --- /dev/null +++ b/inventory/catalog/deepseek-ai__DeepSeek-V4-Flash-0731__strategic.yaml @@ -0,0 +1,94 @@ +id: deepseek-ai__DeepSeek-V4-Flash-0731__strategic +status: approved +name: DeepSeek-V4-Flash-0731 +org: deepseek-ai +source: + kind: huggingface + url: https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731 + revision: main + model_card_url: https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731 + project_url: https://www.deepseek.com/ + paper_url: https://arxiv.org/abs/2606.19348 +license: + spdx: MIT + url: https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731 + allows_offline_retention: true + allows_local_ops: true + allows_fine_tune: true + notes: "Official card: repository and weights MIT. Confirm at pin." +profile: + summary: "S-tier open MIT MoE — 284B base / ~13B active (304B with DSpark); not a 12B dense model." + original_source: https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731 + use_cases: + - "Primary open-weight agentic / coding identity after Jul 2026" + - "Offline optionality if DeepSeek API access changes" + - "Future multi-GPU serve (vLLM DSpark / SGLang); not current R-spine" + - "Capability bar for daily briefs against closed frontier" + sweet_spots: + - "Best MIT-licensed open near-frontier that actually fits cheap Glacier (~167 GiB HF tree)" + - "Fits the €15/month Scaleway budget with room for K3" + - "Displaces DeepSeek-V3 as the collectable S1 identity" + not_ideal_for: + - "Current single-consumer-GPU daily ops (use R spine / Qwen3.8-27B)" + - "Treating 'Flash' as a 12B dense model (that reading was an executor error)" + capability_notes: > + Official V4-Flash release (2026-07-31). Same structure as + DeepSeek-V4-Flash-DSpark (speculative decoding module attached). + HF card reports 304B params (284B base + draft). ~13B active. + 1M context. Terminal Bench 2.1 82.7. HF tree listed ~167 GB. + Automated briefs 2026-08-04…14 repeatedly called this a 12B dense + consumer-GPU model and once claimed it was already collected. It + was not. Size class is S, not R. +swot: + strengths: + - "MIT, official org, actually downloadable" + - "Near-frontier agentic scores at ~13B active" + - "Footprint (~167 GiB) is cheap on Glacier and was always inside the old 850 GiB quota" + weaknesses: + - "Serve needs multi-GPU / high unified memory, not T1–T2" + - "Executor hallucinated it as 12B for two weeks — catalog must stay the authority" + opportunities: + - "First S pull onto Scaleway (FI-WP-0004-T09)" + - "Compressed / GGUF community packs exist if we want a later R experiment (not a substitute for the official tree)" + threats: + - "Successor V4.x could land while we still have no object on disk" + - "Quota-era deferral already cost a month of optionality" +size: + total_bytes: 179000000000 + total_human: "~167 GiB official HF tree (card file listing 2026-09-13); 304B params / ~13B active" +hardware_class: + min_vram_gb_q4: 0 + min_vram_gb_fp16: 0 + notes: "Beyond current lab run envelope (T3–T4). Community 2-bit reports exist for 128 GB unified memory; not an R-spine claim." +axes: [A, B, D] +priority: high +collection: + approved_by: "grok" + approved_at: "2026-09-13" + downloaded_at: null + downloaded_by: "" + storage_path: "" + brief_refs: + - briefs/2026/08/2026-08-03.md + - briefs/2026/09/2026-09-13.md + - docs/decisions/2026-09-13-scaleway-object-reserve.md +reason: "S1 collectable — strongest MIT open MoE that fits the Scaleway euro budget; first pull after bucket create." +tags: [tier-s, strategic, moe, mit, beyond-run-envelope] +companions: [] +notes: > + Do not confuse with a 12B dense checkpoint. There is no + deepseek-ai/DeepSeek-V4-Flash-0731-12B. Pull the official card repo. + Storage class at upload: strategic/ → Glacier. +history: + - at: "2026-08-03" + event: nominated + by: grok + detail: "Catch-up brief named V4-Flash-0731 as mid-tier open value leader." + - at: "2026-08-04" + event: misidentified + by: rein-aharness + detail: "Automated briefs treated it as 12B dense; never cataloged; 08-07 claimed on-disk collection that did not happen." + - at: "2026-09-13" + event: approved + by: grok + detail: "Corrected to 304B-class MoE ~167 GiB MIT. Approved for Scaleway Glacier once FI-WP-0004-T08 bucket exists." diff --git a/inventory/collection-policy.md b/inventory/collection-policy.md index 635cb35..a31731b 100644 --- a/inventory/collection-policy.md +++ b/inventory/collection-policy.md @@ -1,15 +1,16 @@ # Collection policy — open-weight reserve -**Status:** updated for 1 TB NAS + strategic capability reserve (2026-07-24) +**Status:** updated for Scaleway object reserve (2026-09-13) **Related:** `schema.yaml`, `docs/backup-storage-policy.md`, -`docs/decisions/2026-07-24-nas-strategic-reserve.md`, `INTENT.md` +`docs/decisions/2026-09-13-scaleway-object-reserve.md`, `INTENT.md` --- ## Purpose Decide **what** enters the open-weight reserve, **who** may approve it, and -**when** a daily-brief candidate becomes a catalog entry with blobs on the NAS. +**when** a daily-brief candidate becomes a catalog entry with blobs on +Scaleway (staged locally first). --- @@ -19,7 +20,7 @@ Decide **what** enters the open-weight reserve, **who** may approve it, and need later — **including models too large to run on current lab hardware** * Also hold a **runnable spine** for day-to-day local ops and fine-tunes * Enforce **license and integrity** before download completes -* Stay within the **1 TB NAS soft quota (850 GiB)** — quality over mirror volume +* Stay within the **€15 / month** soft budget — quality over mirror volume * Prefer official provenance; not a full Hugging Face scrape --- @@ -32,7 +33,7 @@ Decide **what** enters the open-weight reserve, **who** may approve it, and | **Reserve** | Is this among the best open artifacts worth keeping *just in case*? | **Not gated by current VRAM** | **Runnability is not an eligibility gate.** An unrunnable SOTA open MoE can be -priority **high** if license and quota allow. +priority **high** if license and euro budget allow. --- @@ -42,7 +43,8 @@ priority **high** if license and quota allow. 2. **Clear license** — SPDX or linkable license text; `allows_offline_retention: true` 3. **Stable provenance** — official org, tagged release, or commit revision 4. **Lab rationale** — written `reason` (capability SOTA, runnable spine, embed, FT base, …) -5. **Capacity** — estimated size fits under remaining soft quota (850 GiB) +5. **Capacity** — estimated monthly storage cost fits under remaining + €15 soft budget (Glacier for S, One Zone IA for R) Fail any gate → `rejected` or never enter catalog. @@ -75,9 +77,9 @@ Catalog field: use `tags` including `tier-r` / `tier-s` / `tier-w` and | -------------------- | -------- | | **< 5 GiB** | Operator or lab agent after license check | | **5–40 GiB** | Explicit operator approval | -| **> 40 GiB** | Operator approval + remaining soft-quota check | -| **> 200 GiB (typical S giants)** | Operator approval + written note on which other S models may need to wait | -| **Any size if quota ≥ 70% used** | Operator approval required | +| **> 40 GiB** | Operator approval + remaining **euro-budget** check | +| **> 200 GiB (typical S giants)** | Operator approval + written note that the object goes to **Glacier** | +| **Any size if month ≥ 70% of €15** | Operator approval required | | **Unclear license or ToS risk** | Do not collect; `rejected` | Agents may **nominate** freely; they **collect** only in the < 5 GiB band with @@ -97,8 +99,9 @@ unambiguous licenses unless the operator has approved the catalog entry. * **Best available open general / reasoning / code weights** at the frontier of open * Large MoE or dense models even if current infra cannot serve them -* Prefer official compressed releases (FP8, published quant) when full precision - would exhaust the 1 TB NAS +* Prefer official compressed releases (FP8, published quant) when full + precision would blow the euro budget; Glacier makes full official trees + of the *top* S identities affordable (V4-Flash ~€0.42/mo, K3 ~€3.69/mo) * One clear “best open” per capability niche is better than five near-duplicates ### Companions @@ -124,7 +127,7 @@ Tokenizers, LoRA adapters, small eval fixtures when required to use a reserved b brief nominates → candidate → approved - → collecting (NAS staging/) + → collecting (local staging/ then S3 upload) → collected → verified → superseded|evicted @@ -136,14 +139,15 @@ brief nominates 1. Brief **Collection candidates** nominates R/S/W. 2. Catalog YAML under `inventory/catalog/`. -3. Download only after storage path is pinned (`docs/backup-storage-policy.md`) and approval rules pass. +3. Download only after the Scaleway bucket is live (FI-WP-0004-T08) and + approval rules pass. Stage on VAULT; SoT is `s3://`. 4. `collection.brief_refs` / research refs for provenance of the nomination. --- ## Eviction rule of thumb -When over soft quota: +When over the euro soft budget: 1. **W** tier and easily re-obtainable duplicates 2. Superseded revisions with a stronger verified successor diff --git a/scripts/collect_model.py b/scripts/collect_model.py index f53b782..bd4c3d5 100755 --- a/scripts/collect_model.py +++ b/scripts/collect_model.py @@ -1,13 +1,27 @@ #!/usr/bin/env python3 -"""Collect one HF model into the VAULT reserve tree (weights-first). +"""Collect one HF model into local staging, optionally upload to Scaleway. Usage: - /tmp/fi-hf/bin/python scripts/collect_model.py \\ + python3 scripts/collect_model.py \\ --repo-id Qwen/Qwen3-8B --local-name Qwen__Qwen3-8B --revision main -Layout: - {base}/models/{local_name}/{revision}/ # HF files (no full onnx dump) - {base}/staging/... # incomplete + python3 scripts/collect_model.py \\ + --repo-id deepseek-ai/DeepSeek-V4-Flash-0731 \\ + --local-name deepseek-ai__DeepSeek-V4-Flash-0731 \\ + --tier s \\ + --s3-bucket railiance-fi-open-weight-reserve + +Layout (local staging, still required for huggingface_hub): + {base}/models/{local_name}/{revision}/ # R / W + {base}/strategic/{local_name}/{revision}/ # S + {base}/staging/... + +S3 prefix (SoT once bucket is live): + s3://{bucket}/models|strategic/{local_name}/{revision}/ + s3://{bucket}/manifests/{local_name}/{revision}/MANIFEST.json + +Credentials: AWS_ACCESS_KEY_ID + AWS_SECRET_ACCESS_KEY, or SCW_ACCESS_KEY + +SCW_SECRET_KEY. Never commit keys. """ from __future__ import annotations @@ -80,44 +94,62 @@ def main() -> int: ap.add_argument( "--base", default="/mnt/d/vault/coulomb/freedom-intelligence", + help="local staging root (VAULT HD)", + ) + ap.add_argument( + "--tier", + choices=("r", "s", "w"), + default="r", + help="r/w → models/ (One Zone IA); s → strategic/ (Glacier)", + ) + ap.add_argument("--s3-bucket", default=os.environ.get("FI_S3_BUCKET") or "") + ap.add_argument( + "--s3-endpoint", + default=os.environ.get("FI_S3_ENDPOINT", "https://s3.nl-ams.scw.cloud"), + ) + ap.add_argument( + "--s3-region", + default=os.environ.get("FI_S3_REGION", "nl-ams"), ) ap.add_argument("--token", default=os.environ.get("HF_TOKEN") or None) args = ap.parse_args() base = Path(args.base) + dest_kind = "strategic" if args.tier == "s" else "models" stage = base / "staging" / args.local_name / args.revision - final = base / "models" / args.local_name / args.revision + final = base / dest_kind / args.local_name / args.revision - if final.exists() and any(final.rglob("*.safetensors")) or ( - final.exists() and any(final.rglob("pytorch_model*.bin")) - ): - print(f"already present with weights: {final}") - return 0 - - if stage.exists(): - shutil.rmtree(stage, ignore_errors=True) - stage.mkdir(parents=True, exist_ok=True) - - print(f"downloading {args.repo_id}@{args.revision} -> {stage}") - snapshot_download( - repo_id=args.repo_id, - revision=args.revision, - local_dir=str(stage), - allow_patterns=list(DEFAULT_ALLOW), - ignore_patterns=list(IGNORE), - token=args.token, - max_workers=1, + already = final.exists() and ( + any(final.rglob("*.safetensors")) or any(final.rglob("pytorch_model*.bin")) ) + if already: + print(f"already present with weights: {final}", file=sys.stderr) + if not args.s3_bucket: + return 0 + else: + if stage.exists(): + shutil.rmtree(stage, ignore_errors=True) + stage.mkdir(parents=True, exist_ok=True) - # strip hub cache dir if present - cache = stage / ".cache" - if cache.exists(): - shutil.rmtree(cache, ignore_errors=True) + print(f"downloading {args.repo_id}@{args.revision} -> {stage}") + snapshot_download( + repo_id=args.repo_id, + revision=args.revision, + local_dir=str(stage), + allow_patterns=list(DEFAULT_ALLOW), + ignore_patterns=list(IGNORE), + token=args.token, + max_workers=1, + ) - final.parent.mkdir(parents=True, exist_ok=True) - if final.exists(): - shutil.rmtree(final) - shutil.move(str(stage), str(final)) + cache = stage / ".cache" + if cache.exists(): + shutil.rmtree(cache, ignore_errors=True) + + final.parent.mkdir(parents=True, exist_ok=True) + if final.exists(): + shutil.rmtree(final) + shutil.move(str(stage), str(final)) artifacts = [] total = 0 @@ -145,9 +177,99 @@ def main() -> int: "artifacts": artifacts, } (final / "MANIFEST.json").write_text(json.dumps(manifest, indent=2) + "\n") - print(json.dumps({"ok": True, "total_bytes": total, "path": str(final)}, indent=2)) + result = {"ok": True, "total_bytes": total, "path": str(final), "tier": args.tier} + + if args.s3_bucket: + prefix = f"{dest_kind}/{args.local_name}/{args.revision}" + storage_class = "GLACIER" if args.tier == "s" else "ONEZONE_IA" + uploaded = upload_tree( + final, + bucket=args.s3_bucket, + prefix=prefix, + endpoint=args.s3_endpoint, + region=args.s3_region, + storage_class=storage_class, + ) + man_key = f"manifests/{args.local_name}/{args.revision}/MANIFEST.json" + upload_file( + final / "MANIFEST.json", + bucket=args.s3_bucket, + key=man_key, + endpoint=args.s3_endpoint, + region=args.s3_region, + storage_class="STANDARD", + ) + result["s3"] = { + "bucket": args.s3_bucket, + "prefix": f"s3://{args.s3_bucket}/{prefix}/", + "manifest": f"s3://{args.s3_bucket}/{man_key}", + "storage_class": storage_class, + "objects": uploaded, + } + + print(json.dumps(result, indent=2)) return 0 +def _s3_client(endpoint: str, region: str): + try: + import boto3 + except ImportError as exc: + raise SystemExit("boto3 required for --s3-bucket (pip install boto3)") from exc + access = os.environ.get("AWS_ACCESS_KEY_ID") or os.environ.get("SCW_ACCESS_KEY") + secret = os.environ.get("AWS_SECRET_ACCESS_KEY") or os.environ.get("SCW_SECRET_KEY") + if not access or not secret: + raise SystemExit("set AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY or SCW_ACCESS_KEY/SCW_SECRET_KEY") + return boto3.client( + "s3", + region_name=region, + endpoint_url=endpoint, + aws_access_key_id=access, + aws_secret_access_key=secret, + ) + + +def upload_file( + path: Path, + *, + bucket: str, + key: str, + endpoint: str, + region: str, + storage_class: str, +) -> None: + client = _s3_client(endpoint, region) + extra = {"StorageClass": storage_class} if storage_class else {} + client.upload_file(str(path), bucket, key, ExtraArgs=extra) + + +def upload_tree( + root: Path, + *, + bucket: str, + prefix: str, + endpoint: str, + region: str, + storage_class: str, +) -> int: + n = 0 + for p in sorted(root.rglob("*")): + if not p.is_file() or p.name.startswith("."): + continue + rel = p.relative_to(root).as_posix() + key = f"{prefix.rstrip('/')}/{rel}" + upload_file( + p, + bucket=bucket, + key=key, + endpoint=endpoint, + region=region, + storage_class=storage_class, + ) + n += 1 + print(f"uploaded {key}", file=sys.stderr) + return n + + if __name__ == "__main__": sys.exit(main()) diff --git a/scripts/verify_brief_durability.py b/scripts/verify_brief_durability.py new file mode 100755 index 0000000..f6afe91 --- /dev/null +++ b/scripts/verify_brief_durability.py @@ -0,0 +1,114 @@ +#!/usr/bin/env python3 +"""Fail if State Hub fi_daily_brief events are not on origin/main. + +A brief day is complete only when briefs/YYYY/MM/YYYY-MM-DD.md exists in +this git repository (the clone should be origin/main). Hub events that +claim wrote/committed without a matching file are the 2026-08/09 failure +mode: local commit on railiance, due-bit cleared, durable memory empty. + +Usage: + python3 scripts/verify_brief_durability.py + STATE_HUB_URL=http://127.0.0.1:8000 python3 scripts/verify_brief_durability.py + +Exit 0 = every hub success event has a brief file. +Exit 1 = mismatch (or hub unreachable when --require-hub is set). +""" +from __future__ import annotations + +import argparse +import json +import os +import sys +import urllib.error +import urllib.request +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[1] + + +def brief_path(date: str) -> Path: + year, month, _ = date.split("-") + return ROOT / "briefs" / year / month / f"{date}.md" + + +def load_events(url: str, timeout: float) -> list[dict]: + req = urllib.request.Request( + f"{url.rstrip('/')}/progress/?event_type=fi_daily_brief", + headers={"Accept": "application/json"}, + ) + with urllib.request.urlopen(req, timeout=timeout) as resp: + data = json.loads(resp.read().decode()) + if isinstance(data, list): + return data + if isinstance(data, dict): + for key in ("items", "events", "results"): + if isinstance(data.get(key), list): + return data[key] + raise RuntimeError(f"unexpected hub payload type {type(data)}") + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__) + ap.add_argument( + "--hub", + default=os.environ.get("STATE_HUB_URL", "http://127.0.0.1:8000"), + ) + ap.add_argument("--timeout", type=float, default=8.0) + ap.add_argument( + "--require-hub", + action="store_true", + help="exit 1 if the hub cannot be reached (default: warn and check git only)", + ) + args = ap.parse_args() + + git_dates = sorted( + p.stem + for p in (ROOT / "briefs").rglob("20*.md") + if p.name[0].isdigit() + ) + print(f"git briefs: {len(git_dates)} last={git_dates[-1] if git_dates else '-'}") + + try: + events = load_events(args.hub, args.timeout) + except (urllib.error.URLError, TimeoutError, json.JSONDecodeError, RuntimeError) as exc: + msg = f"hub unreachable or unreadable at {args.hub}: {exc}" + if args.require_hub: + print(f"FAIL {msg}", file=sys.stderr) + return 1 + print(f"WARN {msg}") + return 0 + + missing: list[str] = [] + present: list[str] = [] + for ev in events: + detail = ev.get("detail") or {} + date = str(detail.get("date") or "") + if not date: + continue + repo = detail.get("repo") + if repo and repo != "freedom-intelligence": + continue + path = brief_path(date) + if path.is_file(): + present.append(date) + else: + missing.append(date) + + missing = sorted(set(missing)) + present = sorted(set(present)) + print(f"hub fi_daily_brief dates on git: {len(present)}") + if missing: + print("FAIL hub events with no brief file:") + for date in missing: + print(f" {date} expected {brief_path(date).relative_to(ROOT)}") + print( + "A local-only commit must not clear due. " + "Push to origin, or treat the hub event as a false completion." + ) + return 1 + print("ok: every hub fi_daily_brief date has a brief file") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/workplans/FI-WP-0004-operational-loop-and-scaleway-reserve.md b/workplans/FI-WP-0004-operational-loop-and-scaleway-reserve.md new file mode 100644 index 0000000..f7b71fa --- /dev/null +++ b/workplans/FI-WP-0004-operational-loop-and-scaleway-reserve.md @@ -0,0 +1,229 @@ +--- +id: FI-WP-0004 +type: workplan +title: "Make the sensing loop durable and move the reserve to Scaleway" +domain: agents +repo: freedom-intelligence +status: active +owner: grok +topic_slug: freedom-intelligence +priority: high +created: "2026-09-13" +updated: "2026-09-13" +depends_on: + - FI-WP-0001 + - FI-WP-0002 + - FI-WP-0003 +--- + +# FI-WP-0004 — Operational loop + Scaleway reserve + +## Goal + +The lab already has INTENT, a brief template, a catalog, and a weekday +schedule. It does **not** yet have a trusted daily field picture or a +reserve that can hold the models INTENT says to keep. + +This workplan closes that gap: + +1. **Sensing loop is durable.** A brief day is complete only when the file + exists on `origin/main`. Hub `fi_daily_brief` events that are not on + git are failures, not completions. +2. **Sensing quality is gated.** The executor must not re-announce cataloged + models, invent sizes, or skip axes C/D by default. +3. **Reserve uses Scaleway economics.** VAULT HD is local staging. The + source of truth for weight blobs is a dedicated Scaleway bucket on + `reef-storage` (One Zone IA for the R-spine, Glacier for S-tier). + The 850 GiB NAS quota is retired as the capacity gate; euros/month + become the gate. + +## Why now + +Review 2026-09-13: last brief on `origin/main` is 2026-08-14; hub events +for 2026-09-07…11 claim `committed=True` with no origin objects. Automated +briefs mis-sized DeepSeek-V4-Flash-0731 as a 12B dense model (it is a +~304B / ~13B-active MIT MoE, ~167 GiB) and never cataloged it. Kimi K3 +was deferred only because a 1 TB disk quota could not hold ~1454 GiB; +on Scaleway Glacier that footprint is a few euros per month. + +Org storage selection is already Scaleway (`RESOURCE-WP-0002`, +`reef-storage`). Do **not** reuse the postgres backup bucket (30-day +expiry). FI needs its own long-retention bucket. + +## Tasks + +### T01 — Decision: Scaleway object reserve + +```task +id: FI-WP-0004-T01 +status: done +priority: high +``` + +Write `docs/decisions/2026-09-13-scaleway-object-reserve.md`. Pin: + +- SoT for blobs: Scaleway Object Storage `nl-ams` via `reef-storage` +- R-spine: One Zone IA (immediate get) +- S-tier: Glacier (restore is acceptable for just-in-case) +- VAULT HD: staging / optional local R cache, not SoT +- Soft euro budget (not GiB quota) as the capacity gate +- Dedicated bucket, never `railiance-platform-pg-backup` + +**Done when:** decision file exists and is linked from storage policy. + +### T02 — Policy and operator docs + +```task +id: FI-WP-0004-T02 +status: done +priority: high +``` + +Update `docs/backup-storage-policy.md`, `inventory/collection-policy.md`, +`docs/hardware-envelope.md`, `README.md`, `SCOPE.md`, +`docs/recurrence-ops.md` so they describe the live dual-loop contract +and Scaleway pin. Retire “workplans still active” drift from README. + +**Done when:** an operator can find the bucket contract, euro gate, and +completion rule without reading INTENT. + +### T03 — Brief durability contract + verifier + +```task +id: FI-WP-0004-T03 +status: done +priority: high +``` + +- Document: `fi_daily_brief` may fire only after the brief is on + `origin/main`. +- Add `scripts/verify_brief_durability.py` comparing hub events to git. +- Tighten `activity-definitions/fi-daily-research-brief.md` and the + playbook so a local-only commit does not clear `due`. + +**Done when:** `python3 scripts/verify_brief_durability.py` fails on the +known Sep 2026 hub/git mismatch (or reports it as debt until those +events are reconciled). + +### T04 — Executor origin publication (rein-aharness) + +```task +id: FI-WP-0004-T04 +status: done +priority: high +``` + +FI is the definition owner. Grant origin publication as a named +capability of `fi-research-brief` (reverses the 2026-09-04 local-only +commit): + +- After commit, `git push origin HEAD:` +- `fi_daily_brief` only if push succeeded (`detail.pushed=true`, + `detail.origin_sha`) +- Push failure → `executor_run` failure, `due` stays true +- Prompt/context: catalog ids, no re-announce, no invented sizes + +**Done when:** rein-aharness tests prove push is attempted and a failed +push does not look like a completed brief day. + +### T05 — Catch-up brief + catalog the missed SOTA + +```task +id: FI-WP-0004-T05 +status: done +priority: high +``` + +Write `briefs/2026/09/2026-09-13.md` correcting V4-Flash and covering +the Aug 15–Sep 12 gap. Catalog: + +- `deepseek-ai/DeepSeek-V4-Flash-0731` as S (MIT, ~167 GiB) — **primary + next pull** once the bucket exists +- `Qwen/Qwen3.8-27B` as R/W candidate (Apache-2.0, dense ~27B) + +Update `inventory/RESERVE-STATUS.md`. + +**Done when:** catalog entries exist with correct size/class; catch-up +brief cites primary cards. + +### T06 — `collect_model.py` S3 backend + +```task +id: FI-WP-0004-T06 +status: done +priority: medium +``` + +Local snapshot still stages on disk. After MANIFEST, optional upload to +the FI bucket with an explicit storage class. Credentials from env +(`AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` or `SCW_*`), never git. + +**Done when:** `--s3-bucket` uploads and prints the object prefix. + +### T07 — reef-storage bucket declaration (planned) + +```task +id: FI-WP-0004-T07 +status: done +priority: high +``` + +Add `reef-storage/substrate/object-stores/fi-open-weight-reserve.yaml` +(`status: planned`) and a create script that **must not** copy the +postgres 30-day expiry. Bind freedom-intelligence as a potential +consumer. + +**Done when:** YAML validates against the reef-storage schema; create +script is documented. Bucket create itself is T08 (operator). + +### T08 — Operator: create bucket and scoped key + +```task +id: FI-WP-0004-T08 +status: todo +priority: high +``` + +Operator / railiance-platform: + +1. Create private bucket `railiance-fi-open-weight-reserve` in `nl-ams`. +2. Versioning on. **No 30-day object expiry.** +3. Lifecycle: prefix `strategic/` → Glacier after 1 day. +4. Scoped IAM key in OpenBao (not the postgres backup key). +5. Flip the reef-storage YAML to `status: active`. +6. Confirm a 1-byte PUT/GET. + +**Done when:** attributes are live and a smoke object round-trips. + +### T09 — First Scaleway collect (V4-Flash) + +```task +id: FI-WP-0004-T09 +status: todo +priority: medium +``` + +After T08: pull `deepseek-ai/DeepSeek-V4-Flash-0731` via +`scripts/collect_model.py` into staging, upload as `strategic/` Glacier +(or Standard then lifecycle). Verify MANIFEST vs objects. Update catalog +`status: collected` / `verified` and RESERVE-STATUS. + +**Done when:** catalog `storage_path` is the S3 prefix and checksums match. + +## Acceptance (workplan-level) + +- [x] Scaleway reserve decision written +- [x] Docs no longer claim VAULT-as-SoT or “workplans still active” +- [x] Durability verifier exists +- [x] Executor publishes to origin or fails the day +- [x] V4-Flash cataloged at the real size +- [ ] Bucket live (T08, operator) +- [ ] First S-tier object on Scaleway (T09) + +## Out of scope + +- Reusing `railiance-platform-pg-backup` +- 24/7 inference of the reserve +- Changing Scaleway org billing (resource-control / founder alert) +- Recovering the unpushed 2026-09-07…11 railiance-local commits + (note in the catch-up brief; do not invent those files)