diff --git a/WORK-RECORDS.md b/WORK-RECORDS.md index b97ec58..3b8aad2 100644 --- a/WORK-RECORDS.md +++ b/WORK-RECORDS.md @@ -53,6 +53,7 @@ | workplan | RPF-WP-0071 | backlog | — | workplans/RPF-WP-0071-attended-credential-rotation-lane.md | | workplan | RPF-WP-0072 | backlog | — | workplans/RPF-WP-0072-audit-core-recovery-compatibility.md | | workplan | RPF-WP-0073 | blocked | — | workplans/RPF-WP-0073-phase-b-contained-openbao-restore.md | +| workplan | RPF-WP-0074 | active | — | workplans/RPF-WP-0074-workstation-disk-capacity.md | | task | RPF-WP-ADHOC-2026-10-10-T01 | done | — | workplans/ADHOC-2026-10-10.md | | task | RPF-WP-ADHOC-2026-10-11-T01 | done | — | workplans/ADHOC-2026-10-11.md | | task | RPF-WP-ADHOC-2026-10-11-T02 | done | — | workplans/ADHOC-2026-10-11.md | @@ -247,6 +248,11 @@ | task | RPF-WP-0072-T01 | todo | — | workplans/RPF-WP-0072-audit-core-recovery-compatibility.md | | task | RPF-WP-0073-T01 | wait | — | workplans/RPF-WP-0073-phase-b-contained-openbao-restore.md | | task | RPF-WP-0073-T02 | wait | — | workplans/RPF-WP-0073-phase-b-contained-openbao-restore.md | +| task | RPF-WP-0074-T01 | done | — | workplans/RPF-WP-0074-workstation-disk-capacity.md | +| task | RPF-WP-0074-T02 | done | — | workplans/RPF-WP-0074-workstation-disk-capacity.md | +| task | RPF-WP-0074-T03 | todo | — | workplans/RPF-WP-0074-workstation-disk-capacity.md | +| task | RPF-WP-0074-T04 | todo | — | workplans/RPF-WP-0074-workstation-disk-capacity.md | +| task | RPF-WP-0074-T05 | todo | — | workplans/RPF-WP-0074-workstation-disk-capacity.md | | intake | RPF-IN-0001 | closed | — | intakes/intakes.md | | intake | RPF-IN-0002 | open | — | intakes/intakes.md | | intake | RPF-IN-0003 | open | — | intakes/intakes.md | diff --git a/docs/evidence/2026-11-07-workstation-disk/README.md b/docs/evidence/2026-11-07-workstation-disk/README.md new file mode 100644 index 0000000..b85553e --- /dev/null +++ b/docs/evidence/2026-11-07-workstation-disk/README.md @@ -0,0 +1,115 @@ +# Workstation disk incident — 2026-11-07 + +Scope: Windows `bnt-lap001`, WSL `Ubuntu-24.04`, local development data. +Related: RAPPS-WP-0016; follow-through: [RPF-WP-0074](../../../workplans/RPF-WP-0074-workstation-disk-capacity.md). +Sizes below are GiB (1024³ bytes). `du -x -B1` measures allocated directory +blocks; Windows VHD sizes are file lengths. Measurements are sequential. + +## Incident and immediate relief + +The supplied [bus-error.png](bus-error.png) shows a synthetic mount registry lock +failing with a read-only filesystem error, then `Bus error (core dumped)`. +The WSL terminal reports failed passwd/locale reads and `I/O error @util.cpp`. +[bus-error-1.png](bus-error-1.png) documents the interrupted earlier investigation +and Windows free space of 243,466,247 bytes (about 232 MiB). Both originals +were copied unchanged from Downloads. These establish symptoms, not a kernel cause. + +This session measured C: initially at about 2.4 GiB free, falling to about 1.1 GiB +before relief. Linux simultaneously reported about 783 GiB available inside its +virtual filesystem. Physical Windows capacity was the immediate risk. + +With operator approval through the command approval prompt, removed exactly +`C:\windream Demo 2022_1`. Checked exact target, absence of VMware processes, +absence of reparse points throughout the tree, and exclusive read access to the +VMDK. No `.lck` directories were present. Windows deletion receipt: + +```json +{"Target":"C:\\windream Demo 2022_1","Removed":true,"FileBytes":55611519565,"FreeBefore":1147469824,"FreeAfter":56759525376,"UTC":"2026-11-07T09:52:29.5217532Z"} +``` + +**51.8 GiB reclaimed; 52.9 GiB free immediately afterward.** Later samples +fluctuate; [capacity-after.json](capacity-after.json) records a later measurement. + +Ubuntu-24.04's VHDX is 222,345,297,920 bytes (207.1 GiB). Other registered VHDX +lengths: custodian-agents 4.57 GiB, Ubuntu 1.39 GiB, archlinux 0.80 GiB, +Debian 0.73 GiB. Ubuntu-18.04 is registered but no VHDX was found at its base +path; this does not imply that distribution is empty or removable. + +The **real root is ext4 rw**, verified outside the agent sandbox. Its +`errors=remount-ro` option does not mean it is currently read-only. The sandbox +itself presents `/` read-only; do not mistake that for a damaged guest root. +Retained recent kernel journals show ext4 journal/orphan recovery after unclean +shutdowns, but searched boots did not expose a matching block I/O or ext4-remount +error. Journald also replaced an unclean/corrupt journal. Host exhaustion is a +strong working explanation, not proof that every bus error has this cause. +No WSL shutdown, filesystem repair, resize or compaction was performed. + +## Measured data producers + +| Location / producer | GiB | Finding | +|---|---:|---| +| `/home/worsch` | 56.2 | Point-in-time unprivileged scan | +| `clay-borg/target` | 5.78 | Rust output, mostly debug; largest generated tree inside a repo | +| `content-explorer` | 1.75 | Two node_modules trees total 1.09 GiB; `.next` adds 0.25 GiB | +| `open-cmis-tck/.local` | 0.58 | Local toolchains and test runtime | +| `repo-scoping/var` | 0.36 | Mostly managed checkout data | +| `evidence-anchor/node_modules` | 0.31 | Dependencies | +| `.npm/_cacache` | 4.71 | Shared npm cache | +| `go/pkg` | 5.27 | Shared Go package/module data | +| `.cache/uv` | 3.04 | Shared Python cache | +| `.cache/ms-playwright` | 1.89 | Browser downloads | +| `.cache/trivy` | 1.33 | Scanner data; some nested data unreadable | +| `.codex/sessions` | 2.96 | Conversation/recovery records; retention needs review | +| `.config/railiance-platform/coulomb-social-recovery` | 4.78 | Mostly three large recovery drill directories | +| `.cache/railiance/backups` | 3.08 | Forgejo staging alone 2.78 GiB | +| `/var/log` | 0.52 | Journal 0.51 GiB; logs are not dominant | +| `/tmp` | <0.001 | Almost empty after restart; previous peaks unknown | + +Do not add parent and child rows. Protected directories may be omitted. +`/var/lib/docker` and `/var/lib/containerd` require root; noninteractive sudo +requires a password. Thus these measurements do **not** reconcile all 173.4 GiB +used inside the guest. Complete byte attribution remains open in T04. + +[Docker metadata](docker-inventory.json) records **394 image rows, 345 unique +image IDs and 30 containers**. Most containers are stopped; `infra-postgres-1` +is running and healthy. Repository tag counts include user-engine 32, audit-core +25, informed-decision 24, coulomb-social 23, flex-auth 21, key-cape 19, +state-hub 18, vergabe-teilnahme 15 and activity-core 12. Playwright images display +around 3.6 GB each (Docker's decimal units). Tags and images share layers, so +neither these counts nor summed display sizes establish physical/reclaimable +bytes. `docker system df -v` timed out. No Docker objects were deleted. + +No repo outputs, shared caches, database volumes, backups or session records +were removed. In particular, recovery drill data and backup staging are protected +pending existing custody/retention checks, regardless of a `cache` path component. + +## RAPPS-WP-0016 growth mechanism + +`railiance-apps` itself measured only 18,214,912 bytes. Its +`tools/forgejo-release-build.py` requires a fresh `/tmp` workspace per attempt +(lines 70, 289), extracts source/Go/Node, optionally **copies** Go/npm cache seeds +(86–87), installs dependencies and retains failed logs/prior verification outputs. +OCI builds use `--no-cache`. This preserves isolation/evidence but has no +host-space admission or bounded workspace retention. Repeat builds multiply the +working set. Previous workspace sizes cannot be recovered from today's nearly +empty `/tmp`; save evidence durably before any restart or selected cleanup. + +## Capacity check available now + +```bash +python3 scripts/workstation_disk_guard.py +python3 scripts/workstation_disk_guard.py --reserve-gib 25 +``` + +25 GiB is an **illustration**, not a measured Forgejo build budget. Reservation +is deducted from both host and guest capacity. Starting floors: C: warning below +60 GiB, critical below 30 GiB; guest warning below 40 GiB, critical below 20 GiB. +Exit 0/1/2/3 means healthy/warning/critical/unknown. Missing host mount or unreadable +capacity is unknown. Current C: deliberately remains a warning. Seven regression +scenarios pass. Build integration and a recurring timer are not installed yet. + +Microsoft documents that [WSL's advertised guest capacity can exceed physical +Windows capacity](https://learn.microsoft.com/en-us/windows/wsl/disk-space). +That guide also covers offline repair for a genuinely read-only VHD. Any repair +or compaction is a separate attended maintenance step with recoverable backups; +increasing the guest maximum does not address today's host exhaustion. diff --git a/docs/evidence/2026-11-07-workstation-disk/bus-error-1.png b/docs/evidence/2026-11-07-workstation-disk/bus-error-1.png new file mode 100644 index 0000000..eae7e0e Binary files /dev/null and b/docs/evidence/2026-11-07-workstation-disk/bus-error-1.png differ diff --git a/docs/evidence/2026-11-07-workstation-disk/bus-error.png b/docs/evidence/2026-11-07-workstation-disk/bus-error.png new file mode 100644 index 0000000..2941c3a Binary files /dev/null and b/docs/evidence/2026-11-07-workstation-disk/bus-error.png differ diff --git a/docs/evidence/2026-11-07-workstation-disk/capacity-after.json b/docs/evidence/2026-11-07-workstation-disk/capacity-after.json new file mode 100644 index 0000000..9214286 --- /dev/null +++ b/docs/evidence/2026-11-07-workstation-disk/capacity-after.json @@ -0,0 +1,27 @@ +{ + "observed_at": "2026-11-07T10:19:28.249862+00:00", + "schema": "platform.workstation-disk-guard.v1", + "status": "warning", + "volumes": [ + { + "critical_bytes": 32212254720, + "free_after_reserve_bytes": 56835805184, + "free_bytes": 56835805184, + "path": "/mnt/c", + "reserve_bytes": 0, + "status": "warning", + "total_bytes": 510770802688, + "warning_bytes": 64424509440 + }, + { + "critical_bytes": 21474836480, + "free_after_reserve_bytes": 839868833792, + "free_bytes": 839868833792, + "path": "/", + "reserve_bytes": 0, + "status": "ok", + "total_bytes": 1081101176832, + "warning_bytes": 42949672960 + } + ] +} diff --git a/docs/evidence/2026-11-07-workstation-disk/docker-inventory.json b/docs/evidence/2026-11-07-workstation-disk/docker-inventory.json new file mode 100644 index 0000000..bf9a56d --- /dev/null +++ b/docs/evidence/2026-11-07-workstation-disk/docker-inventory.json @@ -0,0 +1,232 @@ +{ + "observed_at": "2026-11-07T10:19:26.929538+00:00", + "physical_docker_bytes": "unknown: system df timed out; root directory requires sudo", + "image_rows": 394, + "unique_image_ids": 345, + "image_tag_counts_by_repository": { + "user-engine": 32, + "audit-core": 25, + "informed-decision": 24, + "coulomb-social": 23, + "flex-auth": 21, + "key-cape": 19, + "state-hub": 18, + "vergabe-teilnahme": 15, + "mas-live": 12, + "activity-core": 12, + "hub-core": 12, + "identity-provisioner": 11, + "email-connect": 10, + "core-hub": 9, + "bounded-executor": 8, + "policy-nexus": 8, + "approval-engine": 7, + "canned-prompts": 7 + }, + "large_image_examples": [ + { + "Repository": "forgejo.coulomb.social/coulomb/rapp-playwright", + "Tag": "0.1.0", + "Size": "3.61GB" + }, + { + "Repository": "mcr.microsoft.com/playwright", + "Tag": "v1.64.0-noble", + "Size": "3.59GB" + }, + { + "Repository": "vergabe-teilnahme", + "Tag": "pilot-access-tests-20260911", + "Size": "1.11GB" + }, + { + "Repository": "vergabe-factory-tests", + "Tag": "20260908", + "Size": "1.11GB" + }, + { + "Repository": "grafana/grafana", + "Tag": "", + "Size": "1.85GB" + }, + { + "Repository": "cnpg-option-a-backup", + "Tag": "v1", + "Size": "1.36GB" + }, + { + "Repository": "forgejo.coulomb.social/coulomb/cnpg-option-a-backup", + "Tag": "v1", + "Size": "1.36GB" + }, + { + "Repository": "node", + "Tag": "20-bookworm", + "Size": "1.59GB" + }, + { + "Repository": "ghcr.io/cloudnative-pg/postgresql", + "Tag": "16.10-system-trixie", + "Size": "1.15GB" + }, + { + "Repository": "elasticsearch", + "Tag": "7.17.27", + "Size": "1.01GB" + } + ], + "image_sizes_caveat": "Image sizes include shared layers; do not sum to infer physical or reclaimable bytes.", + "container_count": 30, + "containers": [ + { + "Names": "csoc-recovery-20261101202213", + "Status": "Exited (0) 5 days ago", + "Image": "sha256:ff23cdce56cac62ada6f66013e1a50864c0abbe79d132d40b6e05bd80f378a70" + }, + { + "Names": "csoc-recovery-20261031233616", + "Status": "Exited (0) 5 days ago", + "Image": "sha256:ff23cdce56cac62ada6f66013e1a50864c0abbe79d132d40b6e05bd80f378a70" + }, + { + "Names": "csoc-recovery-20261031202727", + "Status": "Exited (0) 6 days ago", + "Image": "sha256:ff23cdce56cac62ada6f66013e1a50864c0abbe79d132d40b6e05bd80f378a70" + }, + { + "Names": "csoc-recovery-20261031073947", + "Status": "Exited (0) 6 days ago", + "Image": "sha256:ff23cdce56cac62ada6f66013e1a50864c0abbe79d132d40b6e05bd80f378a70" + }, + { + "Names": "csoc-recovery-20261031073157", + "Status": "Exited (0) 6 days ago", + "Image": "sha256:ff23cdce56cac62ada6f66013e1a50864c0abbe79d132d40b6e05bd80f378a70" + }, + { + "Names": "csoc-recovery-20261031001528", + "Status": "Exited (0) 6 days ago", + "Image": "sha256:ff23cdce56cac62ada6f66013e1a50864c0abbe79d132d40b6e05bd80f378a70" + }, + { + "Names": "csoc-recovery-20261029214329", + "Status": "Exited (0) 6 days ago", + "Image": "sha256:ff23cdce56cac62ada6f66013e1a50864c0abbe79d132d40b6e05bd80f378a70" + }, + { + "Names": "csoc-recovery-20261029212702", + "Status": "Exited (0) 6 days ago", + "Image": "sha256:ff23cdce56cac62ada6f66013e1a50864c0abbe79d132d40b6e05bd80f378a70" + }, + { + "Names": "rapp-playwright-final", + "Status": "Exited (255) 3 weeks ago", + "Image": "forgejo.coulomb.social/coulomb/rapp-playwright:0.1.0" + }, + { + "Names": "rapp-playwright-release-check", + "Status": "Exited (0) 3 weeks ago", + "Image": "sha256:f4ea70c4ec505fdd63d5bfdda95f92510c987949bbf98bdce4da1e9baf473889" + }, + { + "Names": "rapp-playwright-dev", + "Status": "Exited (0) 3 weeks ago", + "Image": "sha256:d07d0f180dc8cc17c5c57f96a1b24f35b2eb48c525319a1e131d7faddc30a714" + }, + { + "Names": "blissful_driscoll", + "Status": "Exited (1) 3 weeks ago", + "Image": "sha256:aa5b1f9b87948795fe6b09eaf25542b4294a40c75bcf1e6fd281ec441c68d698" + }, + { + "Names": "masmx", + "Status": "Exited (255) 4 weeks ago", + "Image": "forgejo.coulomb.social/coulomb/mas-mx:wp0003" + }, + { + "Names": "rtel-ack-smoke", + "Status": "Exited (0) 5 weeks ago", + "Image": "railiance-telemetry-ack:20260928" + }, + { + "Names": "p06-postgres-acceptance", + "Status": "Exited (255) 7 weeks ago", + "Image": "postgres:16-alpine" + }, + { + "Names": "infra-postgres-1", + "Status": "Up 3 hours (healthy)", + "Image": "postgres:16-alpine" + }, + { + "Names": "flamboyant_leavitt", + "Status": "Created", + "Image": "forgejo.coulomb.social/coulomb/flex-auth:latest" + }, + { + "Names": "trf-pilot-pg", + "Status": "Exited (255) 2 months ago", + "Image": "sha256:6a36eac34375eb521d588ea811b80822cf3b7aac79d07f9180ccd5b5a2be4fa1" + }, + { + "Names": "keycape-inspect", + "Status": "Created", + "Image": "92.205.130.254:32166/coulomb/key-cape:main-e877d27-2" + }, + { + "Names": "temporal-admin-tools", + "Status": "Exited (255) 3 months ago", + "Image": "docker.io/temporalio/admin-tools:1.29.1-tctl-1.18.4-cli-1.5.0" + }, + { + "Names": "temporal-ui", + "Status": "Exited (255) 3 months ago", + "Image": "docker.io/temporalio/ui:2.34.0" + }, + { + "Names": "temporal", + "Status": "Exited (255) 3 months ago", + "Image": "docker.io/temporalio/auto-setup:1.29.1" + }, + { + "Names": "actcore-app-db", + "Status": "Exited (255) 3 months ago", + "Image": "docker.io/postgres:16.6" + }, + { + "Names": "temporal-db", + "Status": "Exited (255) 3 months ago", + "Image": "docker.io/postgres:16.6" + }, + { + "Names": "actcore-nats", + "Status": "Exited (255) 3 months ago", + "Image": "docker.io/nats:2.10-alpine" + }, + { + "Names": "temporal-elasticsearch", + "Status": "Exited (255) 3 months ago", + "Image": "docker.io/elasticsearch:7.17.27" + }, + { + "Names": "sweet_kare", + "Status": "Created", + "Image": "92.205.130.254:32166/coulomb/key-cape:main-1d68639" + }, + { + "Names": "heuristic_einstein", + "Status": "Exited (1) 5 months ago", + "Image": "sha256:039d29654ccb3754c6ecdbe497c6364bbd8452edcdcb7fa937dd9debf5b734ff" + }, + { + "Names": "priceless_hopper", + "Status": "Exited (255) 5 months ago", + "Image": "sha256:039d29654ccb3754c6ecdbe497c6364bbd8452edcdcb7fa937dd9debf5b734ff" + }, + { + "Names": "vergabe-teilnahme-db-1", + "Status": "Exited (255) 5 months ago", + "Image": "sha256:6a36eac34375eb521d588ea811b80822cf3b7aac79d07f9180ccd5b5a2be4fa1" + } + ] +} diff --git a/scripts/workstation_disk_guard.py b/scripts/workstation_disk_guard.py new file mode 100644 index 0000000..ae63cff --- /dev/null +++ b/scripts/workstation_disk_guard.py @@ -0,0 +1,59 @@ +#!/usr/bin/env python3 +"""Read-only WSL host/guest capacity check. Exit 0/1/2/3: OK/warn/critical/unknown.""" + +import argparse +from datetime import datetime, timezone +import json +import math +from pathlib import Path +import shutil + +GIB = 1024 ** 3 +LEVELS = {"ok": 0, "warning": 1, "critical": 2, "unknown": 3} + + +def nonnegative_gib(value): + number = float(value) + if not math.isfinite(number) or number < 0: + raise argparse.ArgumentTypeError("reserve must be finite and nonnegative") + return number + + +def check_volume(path, warning_gib, critical_gib, reserve_bytes, require_mount=False): + result = {"path": str(path), "warning_bytes": warning_gib * GIB, + "critical_bytes": critical_gib * GIB, "reserve_bytes": reserve_bytes} + try: + if require_mount and not Path(path).is_mount(): + raise OSError("host drive is not mounted") + usage = shutil.disk_usage(path) + remaining = usage.free - reserve_bytes + result.update(total_bytes=usage.total, free_bytes=usage.free, + free_after_reserve_bytes=remaining) + result["status"] = ("critical" if remaining < critical_gib * GIB else + "warning" if remaining < warning_gib * GIB else "ok") + except OSError: + result.update(status="unknown", error="capacity_unavailable_or_drive_unmounted") + return result + + +def collect(reserve_bytes=0): + volumes = [check_volume("/mnt/c", 60, 30, reserve_bytes, require_mount=True), + check_volume("/", 40, 20, reserve_bytes)] + status = max((v["status"] for v in volumes), key=LEVELS.get) + return {"schema": "platform.workstation-disk-guard.v1", + "observed_at": datetime.now(timezone.utc).isoformat(), + "status": status, "volumes": volumes} + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--reserve-gib", type=nonnegative_gib, default=0, + help="expected additional build use; subtract from BOTH host and guest free space") + args = parser.parse_args() + report = collect(int(args.reserve_gib * GIB)) + print(json.dumps(report, indent=2, sort_keys=True)) + return LEVELS[report["status"]] + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tests/test_workstation_disk_guard.py b/tests/test_workstation_disk_guard.py new file mode 100644 index 0000000..b7c6dd0 --- /dev/null +++ b/tests/test_workstation_disk_guard.py @@ -0,0 +1,57 @@ +"""Regression scenarios for host exhaustion hidden by a roomy WSL filesystem.""" + +import importlib.util +from pathlib import Path +import shutil +import unittest +from unittest.mock import patch + +SPEC = importlib.util.spec_from_file_location( + "workstation_disk_guard", Path(__file__).parents[1] / "scripts/workstation_disk_guard.py") +guard = importlib.util.module_from_spec(SPEC) +SPEC.loader.exec_module(guard) + + +class DiskGuardTests(unittest.TestCase): + def report(self, host_free, guest_free=783, reserve=0): + usage_type = type(shutil.disk_usage("/")) + def usage(path): + free = (host_free if path == "/mnt/c" else guest_free) * guard.GIB + return usage_type(1024 * guard.GIB, 1024 * guard.GIB - free, free) + with patch.object(guard.Path, "is_mount", return_value=True), \ + patch.object(guard.shutil, "disk_usage", side_effect=usage): + return guard.collect(reserve * guard.GIB) + + def test_crash_capacity_is_critical_despite_roomy_guest(self): + report = self.report(1) + self.assertEqual(report["status"], "critical") + self.assertEqual(report["volumes"][1]["status"], "ok") + + def test_reclaimed_capacity_warns_and_projected_build_can_refuse(self): + self.assertEqual(self.report(53)["status"], "warning") + self.assertEqual(self.report(53, reserve=25)["status"], "critical") + + def test_guest_exhaustion_is_also_critical(self): + self.assertEqual(self.report(100, guest_free=5)["status"], "critical") + + def test_thresholds_and_healthy_capacity(self): + self.assertEqual(self.report(30)["status"], "warning") + self.assertEqual(self.report(60)["status"], "ok") + + def test_unmounted_host_cannot_silently_use_guest_capacity(self): + with patch.object(guard.Path, "is_mount", return_value=False): + self.assertEqual(guard.collect()["status"], "unknown") + + def test_unreadable_capacity_is_unknown(self): + with patch.object(guard.Path, "is_mount", return_value=True), \ + patch.object(guard.shutil, "disk_usage", side_effect=OSError): + self.assertEqual(guard.collect()["status"], "unknown") + + def test_bad_reserve_inputs_are_rejected(self): + for value in ("nan", "inf", "-1"): + with self.subTest(value=value), self.assertRaises(guard.argparse.ArgumentTypeError): + guard.nonnegative_gib(value) + + +if __name__ == "__main__": + unittest.main() diff --git a/workplans/README.md b/workplans/README.md index 8875148..ef0c6f0 100644 --- a/workplans/README.md +++ b/workplans/README.md @@ -6,6 +6,7 @@ Completed plans retain their IDs and managed UUIDs, including archived plans. | Workplan | Open tasks | Next work or dependency | | --- | --- | --- | +| [RPF-WP-0074](RPF-WP-0074-workstation-disk-capacity.md) (active) | T03–T05 | Windows/WSL build space gates, selected Docker/cache retention, recurring checks; obsolete windream VM removed | | [RPF-WP-0015](RPF-WP-0015-audit-core-custody-and-recovery-coordination.md) (active) | T02, T03 | Recovery windows, sender/abort owner and snapshot/quorum/access evidence | | [RPF-WP-0027](RPF-WP-0027-keycape-live-secret-exposure-recovery.md) (blocked) | T03, T06 | Resolver and custody owner contracts | | [RPF-WP-0035](RPF-WP-0035-credential-lane-implementation.md) (blocked) | T02, T03, T07, T08, T10, T12 | Execution integration and owner acceptance; verified-source compiler correction tested October 13 | diff --git a/workplans/RPF-WP-0074-workstation-disk-capacity.md b/workplans/RPF-WP-0074-workstation-disk-capacity.md new file mode 100644 index 0000000..7816fba --- /dev/null +++ b/workplans/RPF-WP-0074-workstation-disk-capacity.md @@ -0,0 +1,150 @@ +--- +id: RPF-WP-0074 +type: workplan +title: "Prevent Windows host disk exhaustion from crashing WSL development" +domain: financials +repo: railiance-platform +status: active +flavor: implementation +owner: codex +topic_slug: railiance +created: "2026-11-07" +updated: "2026-11-07" +related_repos: + - railiance-apps + - railiance-infra + - coulomb-social + - clay-borg +related: + - RAPPS-WP-0016 + - RPF-WP-0053 +--- + +# RPF-WP-0074 — Workstation capacity and bounded development data + +User request 2026-11-07: investigate repeated WSL bus-error/I/O failures during +RAPPS-WP-0016, dispose of the unwanted windream demo, identify repo growth and +plan limits. C: fell to about 1.1 GiB free while WSL advertised 783 GiB available. +Approved obsolete VM removal reclaimed 51.8 GiB. Host exhaustion is a strong +hypothesis; it does not prove that every bus error came from capacity. + +[Incident report, screenshots, measurements and deletion receipt](../docs/evidence/2026-11-07-workstation-disk/README.md). + +## T01 — Preserve the incident and remove the obsolete appliance + +```task +id: RPF-WP-0074-T01 +status: done +priority: high +``` + +Done 2026-11-07: copied both screenshots unchanged, measured host/guest capacity, +checked and removed exactly `C:\windream Demo 2022_1` with command approval. +Windows confirmed absence and free-space gain. Verified the real WSL root is +writable; logs show unclean recovery without conclusive initiating I/O evidence. + +## T02 — Attribute growth and provide a host-aware capacity check + +```task +id: RPF-WP-0074-T02 +status: done +priority: high +``` + +Done 2026-11-07: inventoried largest repo outputs, shared caches, private recovery/ +backup staging, WSL VHD sizes and Docker image/container metadata. Complete +Docker byte accounting remains explicitly open in T04; this is no full guest +byte reconciliation. + +Added `scripts/workstation_disk_guard.py`: read-only JSON; exit 0/1/2/3 means +healthy/warning/critical/unknown. It checks host and guest independently and +deducts prospective build reservation from both. Missing host mount fails closed. +Starting host warning/critical floors 60/30 GiB, guest 40/20 GiB. Seven regression +scenarios pass, including roomy Linux plus exhausted C:, projected build growth, +and missing/unreadable host capacity. Build integration and scheduling remain. + +## T03 — Bound Forgejo build working sets at the Apps entry points + +```task +id: RPF-WP-0074-T03 +status: todo +priority: high +``` + +Owner follow-through: railiance-apps RAPPS-WP-0016. Measure a representative full +attempt's peak allocated bytes, including toolchains, source, npm, Go caches, +tests, OCI export and concurrent verification. Gate prepare/build/container +export on **host and guest** free bytes minus the phase's additional reservation; +refuse critical/unknown before allocating. Include concurrent writers and Windows +reserve. Limit concurrent full builds initially to one; do not guess a small +reservation from the source tree size. + +Define a workspace manifest with owner/run/workplan, lifecycle and generated +subtrees eligible for cleanup. Save input locks, source revision, logs, hashes, +review/admission receipts and needed rollback packages durably before selected +cleanup of finished trials. Keep PID/network isolation and pinned public-cache +provenance; do not substitute an unchecked shared writable cache. Review bounded +read-only seeds or export/reuse against the existing build security contract. + +Acceptance: low-host/unknown-host probes block allocation; a measured full build +fits its reservation; cleanup is scoped; reviewed evidence survives WSL restart. + +## T04 — Reconcile Docker and apply selected repo/cache retention + +```task +id: RPF-WP-0074-T04 +status: todo +priority: high +``` + +The saved baseline contains 345 unique image IDs across 394 image rows, plus +30 containers; retain IDs as well as tags when constructing cleanup candidates. +First finish attribution: Docker's detailed size query timed out and root-owned +storage is inaccessible without operator sudo. Obtain a read-only report covering +DockerRootDir, build caches, unique image layers, container writable layers, +volumes and deleted-but-open files; reconcile the guest total. Do not sum image +display sizes as physical/reclaimable storage. + +Prioritize repeated Docker iterations from user-engine, audit-core, +informed-decision, coulomb-social, state-hub and activity-core; Playwright bases; +then `clay-borg/target` (5.78 GiB), content-explorer frontend output, repo-scoping +managed checkouts and shared npm/Go/uv/browser caches. Record owners, current and +rollback pins, live consumers, regeneration cost, size/age budgets and an exact +candidate list. Remove only reviewed inactive generated data. Preserve running +`infra-postgres-1` and all data volumes absent separate owner disposal authority. +No blanket `docker system prune --volumes`, Git-history rewrite or home sweep. + +Protect `.config/railiance-platform/coulomb-social-recovery` (4.78 GiB), +`.cache/railiance/backups` (3.08 GiB) and session records. Review staging under +RPF-WP-0053 and source recovery workplans; verify offsite custody and restore +evidence before expiry. Paths containing `cache`, `drill` or `recovery` do not +confer disposal authority. + +Acceptance: byte reconciliation or bounded documented residual; measured gain +from selected cleanup; current/rollback packages, data and evidence preserved. + +## T05 — Make checks routine and verify stable recovery + +```task +id: RPF-WP-0074-T05 +status: todo +priority: medium +``` + +Choose/install a workstation-local recurring capacity check (suggested every +15 minutes), visible notification route and bounded report retention; inventory +producer sizes weekly. Do not run full home/Docker scans every 15 minutes. +Agree floors/budgets; distinguish warning/critical/unknown. The checker exists; +no timer, notification route or automatic cleanup is installed by this investigation. + +Observe host/VHD growth across resumed RAPPS-WP-0016 work and an ordinary restart. +If I/O errors/read-only mounts recur despite host reserve, capture kernel and +Windows storage evidence and plan an attended offline ext4 check with backup +first. Guest deletion may need a separate trim/offline VHD compaction to return +host space; measure instead of assuming immediate one-for-one reclamation. +Do not stop active WSL sessions for compaction outside an attended maintenance +window. No guest resize is indicated for this host-capacity incident. + +Acceptance: scheduled report and visible warning/critical/unknown probes work; +inventory retention is bounded; development respects admitted floors; recurrence +ceases over the observation window or receives evidence-based follow-through.