Track WSL host disk incident and add capacity guard
Assistant: codex Assistant-Model: gpt-6.1-sol Assistant-Session: 01a1b4e3-7aa4-75b2-b432-114153ff6879
This commit is contained in:
codex
parent
8519459e6f
commit
6a758c5241
10 changed files with 647 additions and 0 deletions
|
|
@ -53,6 +53,7 @@
|
|||
| workplan | RPF-WP-0071 | backlog | — | workplans/RPF-WP-0071-attended-credential-rotation-lane.md |
|
||||
| workplan | RPF-WP-0072 | backlog | — | workplans/RPF-WP-0072-audit-core-recovery-compatibility.md |
|
||||
| workplan | RPF-WP-0073 | blocked | — | workplans/RPF-WP-0073-phase-b-contained-openbao-restore.md |
|
||||
| workplan | RPF-WP-0074 | active | — | workplans/RPF-WP-0074-workstation-disk-capacity.md |
|
||||
| task | RPF-WP-ADHOC-2026-10-10-T01 | done | — | workplans/ADHOC-2026-10-10.md |
|
||||
| task | RPF-WP-ADHOC-2026-10-11-T01 | done | — | workplans/ADHOC-2026-10-11.md |
|
||||
| task | RPF-WP-ADHOC-2026-10-11-T02 | done | — | workplans/ADHOC-2026-10-11.md |
|
||||
|
|
@ -247,6 +248,11 @@
|
|||
| task | RPF-WP-0072-T01 | todo | — | workplans/RPF-WP-0072-audit-core-recovery-compatibility.md |
|
||||
| task | RPF-WP-0073-T01 | wait | — | workplans/RPF-WP-0073-phase-b-contained-openbao-restore.md |
|
||||
| task | RPF-WP-0073-T02 | wait | — | workplans/RPF-WP-0073-phase-b-contained-openbao-restore.md |
|
||||
| task | RPF-WP-0074-T01 | done | — | workplans/RPF-WP-0074-workstation-disk-capacity.md |
|
||||
| task | RPF-WP-0074-T02 | done | — | workplans/RPF-WP-0074-workstation-disk-capacity.md |
|
||||
| task | RPF-WP-0074-T03 | todo | — | workplans/RPF-WP-0074-workstation-disk-capacity.md |
|
||||
| task | RPF-WP-0074-T04 | todo | — | workplans/RPF-WP-0074-workstation-disk-capacity.md |
|
||||
| task | RPF-WP-0074-T05 | todo | — | workplans/RPF-WP-0074-workstation-disk-capacity.md |
|
||||
| intake | RPF-IN-0001 | closed | — | intakes/intakes.md |
|
||||
| intake | RPF-IN-0002 | open | — | intakes/intakes.md |
|
||||
| intake | RPF-IN-0003 | open | — | intakes/intakes.md |
|
||||
|
|
|
|||
115
docs/evidence/2026-11-07-workstation-disk/README.md
Normal file
115
docs/evidence/2026-11-07-workstation-disk/README.md
Normal file
|
|
@ -0,0 +1,115 @@
|
|||
# Workstation disk incident — 2026-11-07
|
||||
|
||||
Scope: Windows `bnt-lap001`, WSL `Ubuntu-24.04`, local development data.
|
||||
Related: RAPPS-WP-0016; follow-through: [RPF-WP-0074](../../../workplans/RPF-WP-0074-workstation-disk-capacity.md).
|
||||
Sizes below are GiB (1024³ bytes). `du -x -B1` measures allocated directory
|
||||
blocks; Windows VHD sizes are file lengths. Measurements are sequential.
|
||||
|
||||
## Incident and immediate relief
|
||||
|
||||
The supplied [bus-error.png](bus-error.png) shows a synthetic mount registry lock
|
||||
failing with a read-only filesystem error, then `Bus error (core dumped)`.
|
||||
The WSL terminal reports failed passwd/locale reads and `I/O error @util.cpp`.
|
||||
[bus-error-1.png](bus-error-1.png) documents the interrupted earlier investigation
|
||||
and Windows free space of 243,466,247 bytes (about 232 MiB). Both originals
|
||||
were copied unchanged from Downloads. These establish symptoms, not a kernel cause.
|
||||
|
||||
This session measured C: initially at about 2.4 GiB free, falling to about 1.1 GiB
|
||||
before relief. Linux simultaneously reported about 783 GiB available inside its
|
||||
virtual filesystem. Physical Windows capacity was the immediate risk.
|
||||
|
||||
With operator approval through the command approval prompt, removed exactly
|
||||
`C:\windream Demo 2022_1`. Checked exact target, absence of VMware processes,
|
||||
absence of reparse points throughout the tree, and exclusive read access to the
|
||||
VMDK. No `.lck` directories were present. Windows deletion receipt:
|
||||
|
||||
```json
|
||||
{"Target":"C:\\windream Demo 2022_1","Removed":true,"FileBytes":55611519565,"FreeBefore":1147469824,"FreeAfter":56759525376,"UTC":"2026-11-07T09:52:29.5217532Z"}
|
||||
```
|
||||
|
||||
**51.8 GiB reclaimed; 52.9 GiB free immediately afterward.** Later samples
|
||||
fluctuate; [capacity-after.json](capacity-after.json) records a later measurement.
|
||||
|
||||
Ubuntu-24.04's VHDX is 222,345,297,920 bytes (207.1 GiB). Other registered VHDX
|
||||
lengths: custodian-agents 4.57 GiB, Ubuntu 1.39 GiB, archlinux 0.80 GiB,
|
||||
Debian 0.73 GiB. Ubuntu-18.04 is registered but no VHDX was found at its base
|
||||
path; this does not imply that distribution is empty or removable.
|
||||
|
||||
The **real root is ext4 rw**, verified outside the agent sandbox. Its
|
||||
`errors=remount-ro` option does not mean it is currently read-only. The sandbox
|
||||
itself presents `/` read-only; do not mistake that for a damaged guest root.
|
||||
Retained recent kernel journals show ext4 journal/orphan recovery after unclean
|
||||
shutdowns, but searched boots did not expose a matching block I/O or ext4-remount
|
||||
error. Journald also replaced an unclean/corrupt journal. Host exhaustion is a
|
||||
strong working explanation, not proof that every bus error has this cause.
|
||||
No WSL shutdown, filesystem repair, resize or compaction was performed.
|
||||
|
||||
## Measured data producers
|
||||
|
||||
| Location / producer | GiB | Finding |
|
||||
|---|---:|---|
|
||||
| `/home/worsch` | 56.2 | Point-in-time unprivileged scan |
|
||||
| `clay-borg/target` | 5.78 | Rust output, mostly debug; largest generated tree inside a repo |
|
||||
| `content-explorer` | 1.75 | Two node_modules trees total 1.09 GiB; `.next` adds 0.25 GiB |
|
||||
| `open-cmis-tck/.local` | 0.58 | Local toolchains and test runtime |
|
||||
| `repo-scoping/var` | 0.36 | Mostly managed checkout data |
|
||||
| `evidence-anchor/node_modules` | 0.31 | Dependencies |
|
||||
| `.npm/_cacache` | 4.71 | Shared npm cache |
|
||||
| `go/pkg` | 5.27 | Shared Go package/module data |
|
||||
| `.cache/uv` | 3.04 | Shared Python cache |
|
||||
| `.cache/ms-playwright` | 1.89 | Browser downloads |
|
||||
| `.cache/trivy` | 1.33 | Scanner data; some nested data unreadable |
|
||||
| `.codex/sessions` | 2.96 | Conversation/recovery records; retention needs review |
|
||||
| `.config/railiance-platform/coulomb-social-recovery` | 4.78 | Mostly three large recovery drill directories |
|
||||
| `.cache/railiance/backups` | 3.08 | Forgejo staging alone 2.78 GiB |
|
||||
| `/var/log` | 0.52 | Journal 0.51 GiB; logs are not dominant |
|
||||
| `/tmp` | <0.001 | Almost empty after restart; previous peaks unknown |
|
||||
|
||||
Do not add parent and child rows. Protected directories may be omitted.
|
||||
`/var/lib/docker` and `/var/lib/containerd` require root; noninteractive sudo
|
||||
requires a password. Thus these measurements do **not** reconcile all 173.4 GiB
|
||||
used inside the guest. Complete byte attribution remains open in T04.
|
||||
|
||||
[Docker metadata](docker-inventory.json) records **394 image rows, 345 unique
|
||||
image IDs and 30 containers**. Most containers are stopped; `infra-postgres-1`
|
||||
is running and healthy. Repository tag counts include user-engine 32, audit-core
|
||||
25, informed-decision 24, coulomb-social 23, flex-auth 21, key-cape 19,
|
||||
state-hub 18, vergabe-teilnahme 15 and activity-core 12. Playwright images display
|
||||
around 3.6 GB each (Docker's decimal units). Tags and images share layers, so
|
||||
neither these counts nor summed display sizes establish physical/reclaimable
|
||||
bytes. `docker system df -v` timed out. No Docker objects were deleted.
|
||||
|
||||
No repo outputs, shared caches, database volumes, backups or session records
|
||||
were removed. In particular, recovery drill data and backup staging are protected
|
||||
pending existing custody/retention checks, regardless of a `cache` path component.
|
||||
|
||||
## RAPPS-WP-0016 growth mechanism
|
||||
|
||||
`railiance-apps` itself measured only 18,214,912 bytes. Its
|
||||
`tools/forgejo-release-build.py` requires a fresh `/tmp` workspace per attempt
|
||||
(lines 70, 289), extracts source/Go/Node, optionally **copies** Go/npm cache seeds
|
||||
(86–87), installs dependencies and retains failed logs/prior verification outputs.
|
||||
OCI builds use `--no-cache`. This preserves isolation/evidence but has no
|
||||
host-space admission or bounded workspace retention. Repeat builds multiply the
|
||||
working set. Previous workspace sizes cannot be recovered from today's nearly
|
||||
empty `/tmp`; save evidence durably before any restart or selected cleanup.
|
||||
|
||||
## Capacity check available now
|
||||
|
||||
```bash
|
||||
python3 scripts/workstation_disk_guard.py
|
||||
python3 scripts/workstation_disk_guard.py --reserve-gib 25
|
||||
```
|
||||
|
||||
25 GiB is an **illustration**, not a measured Forgejo build budget. Reservation
|
||||
is deducted from both host and guest capacity. Starting floors: C: warning below
|
||||
60 GiB, critical below 30 GiB; guest warning below 40 GiB, critical below 20 GiB.
|
||||
Exit 0/1/2/3 means healthy/warning/critical/unknown. Missing host mount or unreadable
|
||||
capacity is unknown. Current C: deliberately remains a warning. Seven regression
|
||||
scenarios pass. Build integration and a recurring timer are not installed yet.
|
||||
|
||||
Microsoft documents that [WSL's advertised guest capacity can exceed physical
|
||||
Windows capacity](https://learn.microsoft.com/en-us/windows/wsl/disk-space).
|
||||
That guide also covers offline repair for a genuinely read-only VHD. Any repair
|
||||
or compaction is a separate attended maintenance step with recoverable backups;
|
||||
increasing the guest maximum does not address today's host exhaustion.
|
||||
BIN
docs/evidence/2026-11-07-workstation-disk/bus-error-1.png
Normal file
BIN
docs/evidence/2026-11-07-workstation-disk/bus-error-1.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 150 KiB |
BIN
docs/evidence/2026-11-07-workstation-disk/bus-error.png
Normal file
BIN
docs/evidence/2026-11-07-workstation-disk/bus-error.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 180 KiB |
|
|
@ -0,0 +1,27 @@
|
|||
{
|
||||
"observed_at": "2026-11-07T10:19:28.249862+00:00",
|
||||
"schema": "platform.workstation-disk-guard.v1",
|
||||
"status": "warning",
|
||||
"volumes": [
|
||||
{
|
||||
"critical_bytes": 32212254720,
|
||||
"free_after_reserve_bytes": 56835805184,
|
||||
"free_bytes": 56835805184,
|
||||
"path": "/mnt/c",
|
||||
"reserve_bytes": 0,
|
||||
"status": "warning",
|
||||
"total_bytes": 510770802688,
|
||||
"warning_bytes": 64424509440
|
||||
},
|
||||
{
|
||||
"critical_bytes": 21474836480,
|
||||
"free_after_reserve_bytes": 839868833792,
|
||||
"free_bytes": 839868833792,
|
||||
"path": "/",
|
||||
"reserve_bytes": 0,
|
||||
"status": "ok",
|
||||
"total_bytes": 1081101176832,
|
||||
"warning_bytes": 42949672960
|
||||
}
|
||||
]
|
||||
}
|
||||
232
docs/evidence/2026-11-07-workstation-disk/docker-inventory.json
Normal file
232
docs/evidence/2026-11-07-workstation-disk/docker-inventory.json
Normal file
|
|
@ -0,0 +1,232 @@
|
|||
{
|
||||
"observed_at": "2026-11-07T10:19:26.929538+00:00",
|
||||
"physical_docker_bytes": "unknown: system df timed out; root directory requires sudo",
|
||||
"image_rows": 394,
|
||||
"unique_image_ids": 345,
|
||||
"image_tag_counts_by_repository": {
|
||||
"user-engine": 32,
|
||||
"audit-core": 25,
|
||||
"informed-decision": 24,
|
||||
"coulomb-social": 23,
|
||||
"flex-auth": 21,
|
||||
"key-cape": 19,
|
||||
"state-hub": 18,
|
||||
"vergabe-teilnahme": 15,
|
||||
"mas-live": 12,
|
||||
"activity-core": 12,
|
||||
"hub-core": 12,
|
||||
"identity-provisioner": 11,
|
||||
"email-connect": 10,
|
||||
"core-hub": 9,
|
||||
"bounded-executor": 8,
|
||||
"policy-nexus": 8,
|
||||
"approval-engine": 7,
|
||||
"canned-prompts": 7
|
||||
},
|
||||
"large_image_examples": [
|
||||
{
|
||||
"Repository": "forgejo.coulomb.social/coulomb/rapp-playwright",
|
||||
"Tag": "0.1.0",
|
||||
"Size": "3.61GB"
|
||||
},
|
||||
{
|
||||
"Repository": "mcr.microsoft.com/playwright",
|
||||
"Tag": "v1.64.0-noble",
|
||||
"Size": "3.59GB"
|
||||
},
|
||||
{
|
||||
"Repository": "vergabe-teilnahme",
|
||||
"Tag": "pilot-access-tests-20260911",
|
||||
"Size": "1.11GB"
|
||||
},
|
||||
{
|
||||
"Repository": "vergabe-factory-tests",
|
||||
"Tag": "20260908",
|
||||
"Size": "1.11GB"
|
||||
},
|
||||
{
|
||||
"Repository": "grafana/grafana",
|
||||
"Tag": "<none>",
|
||||
"Size": "1.85GB"
|
||||
},
|
||||
{
|
||||
"Repository": "cnpg-option-a-backup",
|
||||
"Tag": "v1",
|
||||
"Size": "1.36GB"
|
||||
},
|
||||
{
|
||||
"Repository": "forgejo.coulomb.social/coulomb/cnpg-option-a-backup",
|
||||
"Tag": "v1",
|
||||
"Size": "1.36GB"
|
||||
},
|
||||
{
|
||||
"Repository": "node",
|
||||
"Tag": "20-bookworm",
|
||||
"Size": "1.59GB"
|
||||
},
|
||||
{
|
||||
"Repository": "ghcr.io/cloudnative-pg/postgresql",
|
||||
"Tag": "16.10-system-trixie",
|
||||
"Size": "1.15GB"
|
||||
},
|
||||
{
|
||||
"Repository": "elasticsearch",
|
||||
"Tag": "7.17.27",
|
||||
"Size": "1.01GB"
|
||||
}
|
||||
],
|
||||
"image_sizes_caveat": "Image sizes include shared layers; do not sum to infer physical or reclaimable bytes.",
|
||||
"container_count": 30,
|
||||
"containers": [
|
||||
{
|
||||
"Names": "csoc-recovery-20261101202213",
|
||||
"Status": "Exited (0) 5 days ago",
|
||||
"Image": "sha256:ff23cdce56cac62ada6f66013e1a50864c0abbe79d132d40b6e05bd80f378a70"
|
||||
},
|
||||
{
|
||||
"Names": "csoc-recovery-20261031233616",
|
||||
"Status": "Exited (0) 5 days ago",
|
||||
"Image": "sha256:ff23cdce56cac62ada6f66013e1a50864c0abbe79d132d40b6e05bd80f378a70"
|
||||
},
|
||||
{
|
||||
"Names": "csoc-recovery-20261031202727",
|
||||
"Status": "Exited (0) 6 days ago",
|
||||
"Image": "sha256:ff23cdce56cac62ada6f66013e1a50864c0abbe79d132d40b6e05bd80f378a70"
|
||||
},
|
||||
{
|
||||
"Names": "csoc-recovery-20261031073947",
|
||||
"Status": "Exited (0) 6 days ago",
|
||||
"Image": "sha256:ff23cdce56cac62ada6f66013e1a50864c0abbe79d132d40b6e05bd80f378a70"
|
||||
},
|
||||
{
|
||||
"Names": "csoc-recovery-20261031073157",
|
||||
"Status": "Exited (0) 6 days ago",
|
||||
"Image": "sha256:ff23cdce56cac62ada6f66013e1a50864c0abbe79d132d40b6e05bd80f378a70"
|
||||
},
|
||||
{
|
||||
"Names": "csoc-recovery-20261031001528",
|
||||
"Status": "Exited (0) 6 days ago",
|
||||
"Image": "sha256:ff23cdce56cac62ada6f66013e1a50864c0abbe79d132d40b6e05bd80f378a70"
|
||||
},
|
||||
{
|
||||
"Names": "csoc-recovery-20261029214329",
|
||||
"Status": "Exited (0) 6 days ago",
|
||||
"Image": "sha256:ff23cdce56cac62ada6f66013e1a50864c0abbe79d132d40b6e05bd80f378a70"
|
||||
},
|
||||
{
|
||||
"Names": "csoc-recovery-20261029212702",
|
||||
"Status": "Exited (0) 6 days ago",
|
||||
"Image": "sha256:ff23cdce56cac62ada6f66013e1a50864c0abbe79d132d40b6e05bd80f378a70"
|
||||
},
|
||||
{
|
||||
"Names": "rapp-playwright-final",
|
||||
"Status": "Exited (255) 3 weeks ago",
|
||||
"Image": "forgejo.coulomb.social/coulomb/rapp-playwright:0.1.0"
|
||||
},
|
||||
{
|
||||
"Names": "rapp-playwright-release-check",
|
||||
"Status": "Exited (0) 3 weeks ago",
|
||||
"Image": "sha256:f4ea70c4ec505fdd63d5bfdda95f92510c987949bbf98bdce4da1e9baf473889"
|
||||
},
|
||||
{
|
||||
"Names": "rapp-playwright-dev",
|
||||
"Status": "Exited (0) 3 weeks ago",
|
||||
"Image": "sha256:d07d0f180dc8cc17c5c57f96a1b24f35b2eb48c525319a1e131d7faddc30a714"
|
||||
},
|
||||
{
|
||||
"Names": "blissful_driscoll",
|
||||
"Status": "Exited (1) 3 weeks ago",
|
||||
"Image": "sha256:aa5b1f9b87948795fe6b09eaf25542b4294a40c75bcf1e6fd281ec441c68d698"
|
||||
},
|
||||
{
|
||||
"Names": "masmx",
|
||||
"Status": "Exited (255) 4 weeks ago",
|
||||
"Image": "forgejo.coulomb.social/coulomb/mas-mx:wp0003"
|
||||
},
|
||||
{
|
||||
"Names": "rtel-ack-smoke",
|
||||
"Status": "Exited (0) 5 weeks ago",
|
||||
"Image": "railiance-telemetry-ack:20260928"
|
||||
},
|
||||
{
|
||||
"Names": "p06-postgres-acceptance",
|
||||
"Status": "Exited (255) 7 weeks ago",
|
||||
"Image": "postgres:16-alpine"
|
||||
},
|
||||
{
|
||||
"Names": "infra-postgres-1",
|
||||
"Status": "Up 3 hours (healthy)",
|
||||
"Image": "postgres:16-alpine"
|
||||
},
|
||||
{
|
||||
"Names": "flamboyant_leavitt",
|
||||
"Status": "Created",
|
||||
"Image": "forgejo.coulomb.social/coulomb/flex-auth:latest"
|
||||
},
|
||||
{
|
||||
"Names": "trf-pilot-pg",
|
||||
"Status": "Exited (255) 2 months ago",
|
||||
"Image": "sha256:6a36eac34375eb521d588ea811b80822cf3b7aac79d07f9180ccd5b5a2be4fa1"
|
||||
},
|
||||
{
|
||||
"Names": "keycape-inspect",
|
||||
"Status": "Created",
|
||||
"Image": "92.205.130.254:32166/coulomb/key-cape:main-e877d27-2"
|
||||
},
|
||||
{
|
||||
"Names": "temporal-admin-tools",
|
||||
"Status": "Exited (255) 3 months ago",
|
||||
"Image": "docker.io/temporalio/admin-tools:1.29.1-tctl-1.18.4-cli-1.5.0"
|
||||
},
|
||||
{
|
||||
"Names": "temporal-ui",
|
||||
"Status": "Exited (255) 3 months ago",
|
||||
"Image": "docker.io/temporalio/ui:2.34.0"
|
||||
},
|
||||
{
|
||||
"Names": "temporal",
|
||||
"Status": "Exited (255) 3 months ago",
|
||||
"Image": "docker.io/temporalio/auto-setup:1.29.1"
|
||||
},
|
||||
{
|
||||
"Names": "actcore-app-db",
|
||||
"Status": "Exited (255) 3 months ago",
|
||||
"Image": "docker.io/postgres:16.6"
|
||||
},
|
||||
{
|
||||
"Names": "temporal-db",
|
||||
"Status": "Exited (255) 3 months ago",
|
||||
"Image": "docker.io/postgres:16.6"
|
||||
},
|
||||
{
|
||||
"Names": "actcore-nats",
|
||||
"Status": "Exited (255) 3 months ago",
|
||||
"Image": "docker.io/nats:2.10-alpine"
|
||||
},
|
||||
{
|
||||
"Names": "temporal-elasticsearch",
|
||||
"Status": "Exited (255) 3 months ago",
|
||||
"Image": "docker.io/elasticsearch:7.17.27"
|
||||
},
|
||||
{
|
||||
"Names": "sweet_kare",
|
||||
"Status": "Created",
|
||||
"Image": "92.205.130.254:32166/coulomb/key-cape:main-1d68639"
|
||||
},
|
||||
{
|
||||
"Names": "heuristic_einstein",
|
||||
"Status": "Exited (1) 5 months ago",
|
||||
"Image": "sha256:039d29654ccb3754c6ecdbe497c6364bbd8452edcdcb7fa937dd9debf5b734ff"
|
||||
},
|
||||
{
|
||||
"Names": "priceless_hopper",
|
||||
"Status": "Exited (255) 5 months ago",
|
||||
"Image": "sha256:039d29654ccb3754c6ecdbe497c6364bbd8452edcdcb7fa937dd9debf5b734ff"
|
||||
},
|
||||
{
|
||||
"Names": "vergabe-teilnahme-db-1",
|
||||
"Status": "Exited (255) 5 months ago",
|
||||
"Image": "sha256:6a36eac34375eb521d588ea811b80822cf3b7aac79d07f9180ccd5b5a2be4fa1"
|
||||
}
|
||||
]
|
||||
}
|
||||
59
scripts/workstation_disk_guard.py
Normal file
59
scripts/workstation_disk_guard.py
Normal file
|
|
@ -0,0 +1,59 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Read-only WSL host/guest capacity check. Exit 0/1/2/3: OK/warn/critical/unknown."""
|
||||
|
||||
import argparse
|
||||
from datetime import datetime, timezone
|
||||
import json
|
||||
import math
|
||||
from pathlib import Path
|
||||
import shutil
|
||||
|
||||
GIB = 1024 ** 3
|
||||
LEVELS = {"ok": 0, "warning": 1, "critical": 2, "unknown": 3}
|
||||
|
||||
|
||||
def nonnegative_gib(value):
|
||||
number = float(value)
|
||||
if not math.isfinite(number) or number < 0:
|
||||
raise argparse.ArgumentTypeError("reserve must be finite and nonnegative")
|
||||
return number
|
||||
|
||||
|
||||
def check_volume(path, warning_gib, critical_gib, reserve_bytes, require_mount=False):
|
||||
result = {"path": str(path), "warning_bytes": warning_gib * GIB,
|
||||
"critical_bytes": critical_gib * GIB, "reserve_bytes": reserve_bytes}
|
||||
try:
|
||||
if require_mount and not Path(path).is_mount():
|
||||
raise OSError("host drive is not mounted")
|
||||
usage = shutil.disk_usage(path)
|
||||
remaining = usage.free - reserve_bytes
|
||||
result.update(total_bytes=usage.total, free_bytes=usage.free,
|
||||
free_after_reserve_bytes=remaining)
|
||||
result["status"] = ("critical" if remaining < critical_gib * GIB else
|
||||
"warning" if remaining < warning_gib * GIB else "ok")
|
||||
except OSError:
|
||||
result.update(status="unknown", error="capacity_unavailable_or_drive_unmounted")
|
||||
return result
|
||||
|
||||
|
||||
def collect(reserve_bytes=0):
|
||||
volumes = [check_volume("/mnt/c", 60, 30, reserve_bytes, require_mount=True),
|
||||
check_volume("/", 40, 20, reserve_bytes)]
|
||||
status = max((v["status"] for v in volumes), key=LEVELS.get)
|
||||
return {"schema": "platform.workstation-disk-guard.v1",
|
||||
"observed_at": datetime.now(timezone.utc).isoformat(),
|
||||
"status": status, "volumes": volumes}
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--reserve-gib", type=nonnegative_gib, default=0,
|
||||
help="expected additional build use; subtract from BOTH host and guest free space")
|
||||
args = parser.parse_args()
|
||||
report = collect(int(args.reserve_gib * GIB))
|
||||
print(json.dumps(report, indent=2, sort_keys=True))
|
||||
return LEVELS[report["status"]]
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
57
tests/test_workstation_disk_guard.py
Normal file
57
tests/test_workstation_disk_guard.py
Normal file
|
|
@ -0,0 +1,57 @@
|
|||
"""Regression scenarios for host exhaustion hidden by a roomy WSL filesystem."""
|
||||
|
||||
import importlib.util
|
||||
from pathlib import Path
|
||||
import shutil
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
SPEC = importlib.util.spec_from_file_location(
|
||||
"workstation_disk_guard", Path(__file__).parents[1] / "scripts/workstation_disk_guard.py")
|
||||
guard = importlib.util.module_from_spec(SPEC)
|
||||
SPEC.loader.exec_module(guard)
|
||||
|
||||
|
||||
class DiskGuardTests(unittest.TestCase):
|
||||
def report(self, host_free, guest_free=783, reserve=0):
|
||||
usage_type = type(shutil.disk_usage("/"))
|
||||
def usage(path):
|
||||
free = (host_free if path == "/mnt/c" else guest_free) * guard.GIB
|
||||
return usage_type(1024 * guard.GIB, 1024 * guard.GIB - free, free)
|
||||
with patch.object(guard.Path, "is_mount", return_value=True), \
|
||||
patch.object(guard.shutil, "disk_usage", side_effect=usage):
|
||||
return guard.collect(reserve * guard.GIB)
|
||||
|
||||
def test_crash_capacity_is_critical_despite_roomy_guest(self):
|
||||
report = self.report(1)
|
||||
self.assertEqual(report["status"], "critical")
|
||||
self.assertEqual(report["volumes"][1]["status"], "ok")
|
||||
|
||||
def test_reclaimed_capacity_warns_and_projected_build_can_refuse(self):
|
||||
self.assertEqual(self.report(53)["status"], "warning")
|
||||
self.assertEqual(self.report(53, reserve=25)["status"], "critical")
|
||||
|
||||
def test_guest_exhaustion_is_also_critical(self):
|
||||
self.assertEqual(self.report(100, guest_free=5)["status"], "critical")
|
||||
|
||||
def test_thresholds_and_healthy_capacity(self):
|
||||
self.assertEqual(self.report(30)["status"], "warning")
|
||||
self.assertEqual(self.report(60)["status"], "ok")
|
||||
|
||||
def test_unmounted_host_cannot_silently_use_guest_capacity(self):
|
||||
with patch.object(guard.Path, "is_mount", return_value=False):
|
||||
self.assertEqual(guard.collect()["status"], "unknown")
|
||||
|
||||
def test_unreadable_capacity_is_unknown(self):
|
||||
with patch.object(guard.Path, "is_mount", return_value=True), \
|
||||
patch.object(guard.shutil, "disk_usage", side_effect=OSError):
|
||||
self.assertEqual(guard.collect()["status"], "unknown")
|
||||
|
||||
def test_bad_reserve_inputs_are_rejected(self):
|
||||
for value in ("nan", "inf", "-1"):
|
||||
with self.subTest(value=value), self.assertRaises(guard.argparse.ArgumentTypeError):
|
||||
guard.nonnegative_gib(value)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
|
@ -6,6 +6,7 @@ Completed plans retain their IDs and managed UUIDs, including archived plans.
|
|||
|
||||
| Workplan | Open tasks | Next work or dependency |
|
||||
| --- | --- | --- |
|
||||
| [RPF-WP-0074](RPF-WP-0074-workstation-disk-capacity.md) (active) | T03–T05 | Windows/WSL build space gates, selected Docker/cache retention, recurring checks; obsolete windream VM removed |
|
||||
| [RPF-WP-0015](RPF-WP-0015-audit-core-custody-and-recovery-coordination.md) (active) | T02, T03 | Recovery windows, sender/abort owner and snapshot/quorum/access evidence |
|
||||
| [RPF-WP-0027](RPF-WP-0027-keycape-live-secret-exposure-recovery.md) (blocked) | T03, T06 | Resolver and custody owner contracts |
|
||||
| [RPF-WP-0035](RPF-WP-0035-credential-lane-implementation.md) (blocked) | T02, T03, T07, T08, T10, T12 | Execution integration and owner acceptance; verified-source compiler correction tested October 13 |
|
||||
|
|
|
|||
150
workplans/RPF-WP-0074-workstation-disk-capacity.md
Normal file
150
workplans/RPF-WP-0074-workstation-disk-capacity.md
Normal file
|
|
@ -0,0 +1,150 @@
|
|||
---
|
||||
id: RPF-WP-0074
|
||||
type: workplan
|
||||
title: "Prevent Windows host disk exhaustion from crashing WSL development"
|
||||
domain: financials
|
||||
repo: railiance-platform
|
||||
status: active
|
||||
flavor: implementation
|
||||
owner: codex
|
||||
topic_slug: railiance
|
||||
created: "2026-11-07"
|
||||
updated: "2026-11-07"
|
||||
related_repos:
|
||||
- railiance-apps
|
||||
- railiance-infra
|
||||
- coulomb-social
|
||||
- clay-borg
|
||||
related:
|
||||
- RAPPS-WP-0016
|
||||
- RPF-WP-0053
|
||||
---
|
||||
|
||||
# RPF-WP-0074 — Workstation capacity and bounded development data
|
||||
|
||||
User request 2026-11-07: investigate repeated WSL bus-error/I/O failures during
|
||||
RAPPS-WP-0016, dispose of the unwanted windream demo, identify repo growth and
|
||||
plan limits. C: fell to about 1.1 GiB free while WSL advertised 783 GiB available.
|
||||
Approved obsolete VM removal reclaimed 51.8 GiB. Host exhaustion is a strong
|
||||
hypothesis; it does not prove that every bus error came from capacity.
|
||||
|
||||
[Incident report, screenshots, measurements and deletion receipt](../docs/evidence/2026-11-07-workstation-disk/README.md).
|
||||
|
||||
## T01 — Preserve the incident and remove the obsolete appliance
|
||||
|
||||
```task
|
||||
id: RPF-WP-0074-T01
|
||||
status: done
|
||||
priority: high
|
||||
```
|
||||
|
||||
Done 2026-11-07: copied both screenshots unchanged, measured host/guest capacity,
|
||||
checked and removed exactly `C:\windream Demo 2022_1` with command approval.
|
||||
Windows confirmed absence and free-space gain. Verified the real WSL root is
|
||||
writable; logs show unclean recovery without conclusive initiating I/O evidence.
|
||||
|
||||
## T02 — Attribute growth and provide a host-aware capacity check
|
||||
|
||||
```task
|
||||
id: RPF-WP-0074-T02
|
||||
status: done
|
||||
priority: high
|
||||
```
|
||||
|
||||
Done 2026-11-07: inventoried largest repo outputs, shared caches, private recovery/
|
||||
backup staging, WSL VHD sizes and Docker image/container metadata. Complete
|
||||
Docker byte accounting remains explicitly open in T04; this is no full guest
|
||||
byte reconciliation.
|
||||
|
||||
Added `scripts/workstation_disk_guard.py`: read-only JSON; exit 0/1/2/3 means
|
||||
healthy/warning/critical/unknown. It checks host and guest independently and
|
||||
deducts prospective build reservation from both. Missing host mount fails closed.
|
||||
Starting host warning/critical floors 60/30 GiB, guest 40/20 GiB. Seven regression
|
||||
scenarios pass, including roomy Linux plus exhausted C:, projected build growth,
|
||||
and missing/unreadable host capacity. Build integration and scheduling remain.
|
||||
|
||||
## T03 — Bound Forgejo build working sets at the Apps entry points
|
||||
|
||||
```task
|
||||
id: RPF-WP-0074-T03
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
Owner follow-through: railiance-apps RAPPS-WP-0016. Measure a representative full
|
||||
attempt's peak allocated bytes, including toolchains, source, npm, Go caches,
|
||||
tests, OCI export and concurrent verification. Gate prepare/build/container
|
||||
export on **host and guest** free bytes minus the phase's additional reservation;
|
||||
refuse critical/unknown before allocating. Include concurrent writers and Windows
|
||||
reserve. Limit concurrent full builds initially to one; do not guess a small
|
||||
reservation from the source tree size.
|
||||
|
||||
Define a workspace manifest with owner/run/workplan, lifecycle and generated
|
||||
subtrees eligible for cleanup. Save input locks, source revision, logs, hashes,
|
||||
review/admission receipts and needed rollback packages durably before selected
|
||||
cleanup of finished trials. Keep PID/network isolation and pinned public-cache
|
||||
provenance; do not substitute an unchecked shared writable cache. Review bounded
|
||||
read-only seeds or export/reuse against the existing build security contract.
|
||||
|
||||
Acceptance: low-host/unknown-host probes block allocation; a measured full build
|
||||
fits its reservation; cleanup is scoped; reviewed evidence survives WSL restart.
|
||||
|
||||
## T04 — Reconcile Docker and apply selected repo/cache retention
|
||||
|
||||
```task
|
||||
id: RPF-WP-0074-T04
|
||||
status: todo
|
||||
priority: high
|
||||
```
|
||||
|
||||
The saved baseline contains 345 unique image IDs across 394 image rows, plus
|
||||
30 containers; retain IDs as well as tags when constructing cleanup candidates.
|
||||
First finish attribution: Docker's detailed size query timed out and root-owned
|
||||
storage is inaccessible without operator sudo. Obtain a read-only report covering
|
||||
DockerRootDir, build caches, unique image layers, container writable layers,
|
||||
volumes and deleted-but-open files; reconcile the guest total. Do not sum image
|
||||
display sizes as physical/reclaimable storage.
|
||||
|
||||
Prioritize repeated Docker iterations from user-engine, audit-core,
|
||||
informed-decision, coulomb-social, state-hub and activity-core; Playwright bases;
|
||||
then `clay-borg/target` (5.78 GiB), content-explorer frontend output, repo-scoping
|
||||
managed checkouts and shared npm/Go/uv/browser caches. Record owners, current and
|
||||
rollback pins, live consumers, regeneration cost, size/age budgets and an exact
|
||||
candidate list. Remove only reviewed inactive generated data. Preserve running
|
||||
`infra-postgres-1` and all data volumes absent separate owner disposal authority.
|
||||
No blanket `docker system prune --volumes`, Git-history rewrite or home sweep.
|
||||
|
||||
Protect `.config/railiance-platform/coulomb-social-recovery` (4.78 GiB),
|
||||
`.cache/railiance/backups` (3.08 GiB) and session records. Review staging under
|
||||
RPF-WP-0053 and source recovery workplans; verify offsite custody and restore
|
||||
evidence before expiry. Paths containing `cache`, `drill` or `recovery` do not
|
||||
confer disposal authority.
|
||||
|
||||
Acceptance: byte reconciliation or bounded documented residual; measured gain
|
||||
from selected cleanup; current/rollback packages, data and evidence preserved.
|
||||
|
||||
## T05 — Make checks routine and verify stable recovery
|
||||
|
||||
```task
|
||||
id: RPF-WP-0074-T05
|
||||
status: todo
|
||||
priority: medium
|
||||
```
|
||||
|
||||
Choose/install a workstation-local recurring capacity check (suggested every
|
||||
15 minutes), visible notification route and bounded report retention; inventory
|
||||
producer sizes weekly. Do not run full home/Docker scans every 15 minutes.
|
||||
Agree floors/budgets; distinguish warning/critical/unknown. The checker exists;
|
||||
no timer, notification route or automatic cleanup is installed by this investigation.
|
||||
|
||||
Observe host/VHD growth across resumed RAPPS-WP-0016 work and an ordinary restart.
|
||||
If I/O errors/read-only mounts recur despite host reserve, capture kernel and
|
||||
Windows storage evidence and plan an attended offline ext4 check with backup
|
||||
first. Guest deletion may need a separate trim/offline VHD compaction to return
|
||||
host space; measure instead of assuming immediate one-for-one reclamation.
|
||||
Do not stop active WSL sessions for compaction outside an attended maintenance
|
||||
window. No guest resize is indicated for this host-capacity incident.
|
||||
|
||||
Acceptance: scheduled report and visible warning/critical/unknown probes work;
|
||||
inventory retention is bounded; development respects admitted floors; recurrence
|
||||
ceases over the observation window or receives evidence-based follow-through.
|
||||
Loading…
Add table
Add a link
Reference in a new issue