the-custodian/workplans/archived/260708-CUST-WP-0054-workstation-independence-and-fleet-realignment.md

---
id: CUST-WP-0054
type: workplan
title: "Workstation Independence and Fleet Role Realignment"
domain: infotech
repo: the-custodian
status: finished
owner: codex
topic_slug: custodian
planning_priority: high
planning_order: 54
created: "2026-07-03"
updated: "2026-07-08"
state_hub_workstream_id: "5ceff9dd-e034-46d9-a0e1-37f167ca0729"
---

> **Terminology note:** Historical text in this archived workplan may use the legacy term "workstream". The fleet term is **workplan** (`canon/standards/workplan-terminology-fleet_v0.1.md`).

# CUST-WP-0054 - Workstation Independence and Fleet Role Realignment

## Goal

Remove every runtime dependency and always-on workload from the workstation,
realign machine roles, and make the production loop run uninterrupted with the
workstation powered off:

| Machine | Target role |
| --- | --- |
| **railiance01** | Production workloads (the first node of the growing Railiance fleet). |
| **coulombcore** | Early/legacy/experimental workloads only; frozen for new production; eventually decommissioned and **reborn as railiance02** (a machine-scale phoenix rotation). |
| **workstation** | Local, temporary dev environment to build and evolve repos — nothing depends on it being on. |

The acceptance proof for the whole plan is T10: production runs for 24h+ with
the workstation offline.

## Current-State Findings (2026-07-03 inventory)

These are the concrete dependencies this plan removes:

1. **The workstation is a production network hub.** All 16 ops-bridge tunnels
   originate here. Two production paths *chain through* it:
   - State Hub primary: everything reaches the cluster hub via the
     workstation's `state-hub-primary` forward tunnel, and railiance01's
     automations via `state-hub-railiance01` (reverse) → workstation → cluster.
   - issue-core emission: activity-core (railiance01) → workstation →
     issue-core (coulombcore).
   A workstation reboot breaks daily triage evidence, consistency sweeps, and
   task emission.
2. **Production landed on the condemned machine.** The 2026-07-02/03 cutovers
   correctly moved State Hub and Core Hub off the workstation — but onto
   coulombcore (State Hub cluster primary, Core Hub behind
   `hub.coulomb.social`, issue-core, OpenBao, ArgoCD, ESO, the identity stack,
   and Gitea all run there). Under the new roles these are production
   workloads on a legacy machine.
3. **Source of truth for code is on the condemned machine.** Every repo's
   canonical remote is `gitea.coulomb.social` (coulombcore). The Forgejo
   production migration (`RAIL-HO-WP-0005`) already targets railiance01.
4. **The consistency sweep and daily triage depend on workstation repo
   checkouts** (`/home/worsch/*` clones are what fix-consistency writes back
   to) and the triage working-memory sink references a
   `/home/worsch/the-custodian/memory/working` path identity.
5. **Images are built and published from the workstation** (docker build +
   push by hand); there is no CI runner, so releases require this machine.
6. **WSL2 State Hub fallback** still exists on the workstation
   (`CUST-WP-0011-T08/T09` stabilization window) — expected, retired by that
   workplan, not this one.

## Deployment Strategy (proposed — improves on the rough beachhead idea)

**A. Files-first beachhead, not database replication.** ADR-001 already
declares that the State Hub is a read model rebuildable from repo files. So
the dev-environment story should be: `make dev-hub` starts a **local ephemeral
hub** (compose: postgres + API + MCP, exactly what exists today) whose content
is **rebuilt from the local repo checkouts** via the C-06 registration path —
not synced from the fleet database. Offline writes (progress events, task
status) accumulate in a **write buffer** and relay upstream on reconnect
(`STATE-WP-0068 offline-write-buffer-and-edge-relay` is the existing seed for
this). Two hubs never replicate; files reconcile them. This makes the
workstation genuinely temporary: clone repos → `make dev-hub` → work offline →
push + relay when connected.

**B. Fleet mesh instead of workstation-hub networking.** Machine-to-machine
paths (railiance01 ↔ coulombcore/railiance02) get direct, persistent links
owned by the machines themselves — ops-bridge units running *on the fleet
machines* under `atm-` actor certs (ops-warden), or a minimal WireGuard mesh
if tunnel count grows. The workstation becomes a mesh *client* when present,
never a relay.

**C. Promotion-gated production, ThreePhoenix ideas incrementally.** A
workload counts as "production on railiance01" only when it conforms to the
already-finished staged-promotion contract (`RAIL-BS-WP-0006`: overlay repo,
`railiance/app.toml`, stage commands, rollback). ThreePhoenix ideas adopted
now, without waiting for three nodes:
- **CNPG-managed Postgres** per workload (already proven on coulombcore);
- **greenfield rebuild automation** (NET-WP-0020 unseal + bootstrap chain) as
  the standing "phoenix drill" for single machines;
- **phoenix rotation at machine scale**: the coulombcore → railiance02
  rebirth *is* the first full phoenix and should be executed with the
  same automation that a future 3-node weekly rotation would use;
- Longhorn distributed storage and PG streaming HA remain deferred until a
  third node exists (`RAIL-BS-WP-0007` / `CUST-WP-0038` stay the follow-on).

**D. Decommission by attrition, not by big-bang.** coulombcore is frozen for
new production immediately (policy), drained workload-by-workload via staged
promotion onto railiance01, and only rebuilt as railiance02 when the last
production dependency (likely identity/OpenBao) has moved.

## Task: Target architecture and dependency register

```task
id: CUST-WP-0054-T01
status: done
priority: high
state_hub_task_id: "2257d05c-2997-4ec9-a95a-1f2d12915de9"
```

Write the canon-adjacent architecture note (`docs/` first; promote to
`canon/architecture/` after review) fixing the three machine roles, the fleet
mesh topology, the promotion gate for "production", and the phoenix path
coulombcore → railiance02. Include the full dependency register: every
workload, tunnel, repo remote, sink path, and build pipeline with its current
host and target host. Done when every row has a target and a migration owner
(this plan's task or an existing workplan reference).

## Task: De-hub the network — fleet-owned direct tunnels

```task
id: CUST-WP-0054-T02
status: done
priority: high
state_hub_task_id: "40733441-5786-49d3-bc35-006ab34dc646"
```

Remove the workstation from all production data paths:

- Run ops-bridge (or systemd ssh units) **on railiance01** with `atm-` actor
  certs for the two live cross-machine lanes: railiance01 → coulombcore
  issue-core, and railiance01 → cluster State Hub (replacing the
  workstation-chained `state-hub-railiance01` + `state-hub-primary` pair).
- Re-point `actcore-state-hub-bridge` and `actcore-issue-core-bridge` at the
  machine-local tunnel ports.
- Workstation tunnels remain only for interactive dev access (k3s API, hub
  client) and may drop at any time without production impact.
- Evaluate WireGuard mesh as the successor if unit count exceeds ~5.

Done when killing every workstation tunnel leaves triage, sweeps, and
emission working (partial T10 rehearsal).

## Task: Production placement plan and freeze policy

```task
id: CUST-WP-0054-T03
status: done
priority: high
state_hub_task_id: "4bbb3a81-8558-4b19-96ea-684c20633683"
```

Declare coulombcore frozen for new production (policy note in canon). Produce
the drain sequence with per-workload target and method (staged-promotion
overlay or documented exception): State Hub, Core Hub, issue-core, OpenBao,
identity stack (KeyCape/Authelia/privacyIDEA/lldap), ESO/ArgoCD control
plane, Gitea/Forgejo, CNPG databases. Explicitly order them by coupling
(forge and State Hub early; identity + OpenBao last, since everything
authenticates through them).

## Task: Forge to railiance01 + CI runners (kill workstation builds)

```task
id: CUST-WP-0054-T04
status: done
priority: high
state_hub_task_id: "1e92d690-e3e6-4073-849e-4e500cd4cadc"
```

Execute/absorb `RAIL-HO-WP-0005`: Forgejo production on railiance01 becomes
the canonical remote for all repos; coulombcore Gitea becomes a read-only
mirror until decommission. Staged migration ladder (T10): tiers 0–2 done (`forgejo-actions-probe`,
`glas-harness`, `key-cape`); tier 3 production repos gated on T09 backup +
`state-hub` approval. Stand up Actions runners so container images build and
push in CI from tags — the workstation stops being the build/publish host.

**Partial (2026-07-03):** ADR-004 in-cluster runner (`railiance01-build-01` +
DinD) replaces interim coulombcore host runner. Interim coulombcore host runner
disabled. Org Actions secrets (`REGISTRY_USER`, `REGISTRY_TOKEN`) set.
`coulomb/forgejo-actions-probe` `image-build` workflow builds and pushes to
`forgejo.coulomb.social/coulomb/forgejo-actions-probe` (static docker-cli +
DinD; `actions/checkout@v4` not used — git clone in job). **Pilot (2026-07-03):**
`coulomb/glas-harness` migrated to Forgejo (`origin=forgejo-remote`); CI smoke
host+container green — see `docs/forgejo-repo-migration-pilot-glas-harness.md`.
**Tier 3 partial (2026-07-06):** `coulomb/state-hub` + dependency `coulomb/hub-core`
promoted to Forgejo (`origin=forgejo-remote`); hub `remote_url` patched;
`ci-smoke` + `build-and-push` green @ `f9f0091`; `crictl pull` on railiance01
ok. Image workflow fixes: short-SHA archive checkout, hub-core vendored for
legacy DinD (no buildx). **Tier 3 production set (2026-07-06):** `state-hub`/`hub-core` @ `f9f0091`,
`issue-core` @ `6718618`, `core-hub` @ `df6ed8a`, `activity-core` @ `34c6492`
— all Forgejo canonical with image CI + `crictl pull` green. **Bulk promotion
(2026-07-08):** `tools/patch-forgejo-remote-urls.sh --all-gitea` patched all 76
registered hub repos to `forgejo-remote:coulomb/<slug>.git`; tier-3 production
set + remaining org repos complete. **Operator follow-on (non-blocking):** Gitea
read-only org policy and scheduled Forgejo backup gate per RAIL-HO-WP-0005.

## Task: State Hub production home on railiance01

```task
id: CUST-WP-0054-T05
status: done
priority: high
state_hub_task_id: "714a5095-8a5a-4d2f-9e7d-b2a2b1f21526"
```

Move the State Hub primary from coulombcore to railiance01 using the proven
CUST-WP-0011-T07 playbook (freeze → exact-count restore → rewire). This makes
the automation loop machine-local: activity-core, Temporal, and the hub share
one machine, so daily triage and sweeps survive any other machine being down.
Prereq: railiance01 CNPG + storage reviewed (T03). Also relocate the
consistency-sweep repo checkouts to railiance01 (clones from the T04 forge)
so file writebacks no longer touch workstation paths.

**Planning (2026-07-06):** Cutover runbook at
`docs/state-hub-railiance01-cutover-plan.md`. Inventory: primary on coulombcore
(637 ws / 3983 tasks); activity-core reaches hub via
`fleet-state-hub-coulombcore` → `:18000`. Target hostname for sweep paths:
`239.62.205.92.host.secureserver.net`. **Phase 0 preflight done** (2026-07-06):
baseline counts recorded — `docs/state-hub-railiance01-cutover-preflight-2026-07-06.md`.
**Phase 1 complete** (2026-07-06): `docs/state-hub-railiance01-cutover-phase1-2026-07-06.md`.
**Phase 2 complete** (2026-07-06): coulombcore writer frozen (Deployment 0);
`pg_dump` restored to railiance01 with exact-count match (640 wp / 4002 tasks /
8723 progress); railiance01 API on `forgejo…:main-d1a4fcf`, ClusterIP
`10.43.68.154:8000` — `docs/state-hub-railiance01-cutover-phase2-2026-07-06.md`.
**Phase 3 complete** (2026-07-06): access rewired — `state-hub-primary` →
railiance01 `10.43.68.154`; activity-core in-cluster URL; `fleet-state-hub-coulombcore`
stopped — `docs/state-hub-railiance01-cutover-phase3-2026-07-06.md`. coulombcore
writer still frozen (rollback).
**Phase 4 complete** (2026-07-06): 74 repos cloned under `/home/tegwick/`,
`host_paths[239.62.205.92.host.secureserver.net]` registered, state-hub pod mounts
clone tree + sweep hostname override, remote-all sweep exit_code=0 —
`docs/state-hub-railiance01-cutover-phase4-2026-07-06.md`.
**Phase 5 in progress** (2026-07-06 → 2026-07-09): 72h stabilization window open;
kickoff baseline green — `docs/state-hub-railiance01-cutover-phase5-2026-07-06.md`.
Post-cutover daily triage verified (manual 2026-07-06T17:50Z, event `a99e644e`).
Forgejo CI image promoted (`main-0ca9c27`, Helm rev 8; working-memory sweep).
Day-1 check (2026-07-06T22:35Z):
640/4002/14, pod 0 restarts, 6/6 sweeps exit_code=0 proc=11 missing=0,
`state-hub-primary` LIVE. Phase 5 checks scheduled in activity-core:
`phase5-stabilization-daily` (09:00 Europe/Berlin) and one-shot closeout
`2026-07-09T17:35Z`. **Stabilization green (2026-07-08):** `tools/phase5-stabilization-check.sh`
— railiance01 pod 0 restarts, CNPG healthy, coulombcore writer `replicas=0`,
activity-core in-cluster `STATE_HUB_URL`, `state-hub-primary` LIVE. Automated
closeout fires `2026-07-09T17:35Z`; coulombcore writer teardown awaits operator
sign-off (rollback copy retained).

## Task: Working-memory and sink path decoupling

```task
id: CUST-WP-0054-T06
status: done
priority: medium
state_hub_task_id: "1da032b1-4b2f-4e7a-a2f3-e3ac3df29bdd"
```

Remove `/home/worsch/...` path identities from runtime contracts: the triage
working-memory sink and any prompt/context paths become repo-relative or
PVC-native with a defined sync-to-repo step (commit via the sweep). Done when
no ActivityDefinition or sink references a workstation-specific absolute path.

**Complete (2026-07-06):** `custodian://` / `activity-core://` URIs in
activity-core; all ActivityDefinitions use URI sinks (no workstation absolutes);
Railiance worker writes daily-triage notes to hostPath
`/home/tegwick/the-custodian/memory/working` (sweep writeback commits to git);
progress events record repo-relative `working_memory_path`; PVC notes migrated
via `tools/migrate-working-memory-pvc-to-clone.sh`.

## Task: Dev-environment beachhead (files-first)

```task
id: CUST-WP-0054-T07
status: done
priority: high
state_hub_task_id: "853f477b-bb76-4ed4-a5c3-31416fd26000"
```

Implement strategy A: `make dev-hub` (or `custodian dev up`) starts the local
compose hub, registers the locally cloned repos, and rebuilds workplan/task
state from files via C-06 — no fleet connection required. Implement the
offline write buffer + reconnect relay for progress/task events (align with
`STATE-WP-0068`; keep the buffer file-backed and idempotent). MCP config
gains an explicit `dev`/`fleet` profile switch. Done when a fresh machine
reaches a working, orientation-capable dev hub from `git clone` + one command,
fully offline.

**Complete (2026-07-08):** `state-hub/scripts/dev_hub_up.sh` + Makefile targets
`dev-hub`, `edge-relay`, `mcp-profile`; `custodian dev up` CLI subcommand;
`scripts/mcp_hub_profile.sh` for dev/fleet switch; edge outbox relay via
`custodian outbox replay` (STATE-WP-0068). One command: `make dev-hub` or
`custodian dev up [--with-edge] [--with-mcp]`.

## Task: ThreePhoenix increment — phoenix drill automation

```task
id: CUST-WP-0054-T08
status: done
priority: medium
state_hub_task_id: "065b626e-4f52-4a0a-8a52-a8551caf2701"
```

Compose the existing pieces (NET-WP-0020 unseal automation, S1–S3 bootstrap
chain, staged-promotion overlays, CNPG restore drills) into one rehearsable
"phoenix a machine" runbook + automation entrypoint, proven on a disposable
target (haskelseed or a VM). This is the tool the railiance02 rebirth and any
future node rotation will use. Done when a greenfield machine reaches
join-ready state unattended except for custody-gated steps.

**Complete (2026-07-08):** `docs/phoenix-machine-drill-runbook.md` +
`tools/phoenix-drill.sh` (`check`/`plan` modes). Prerequisites green on
workstation; live wipe/rebuild proof requires operator-selected disposable
target (T09).

## Task: coulombcore decommission readiness → railiance02

```task
id: CUST-WP-0054-T09
status: wait
priority: medium
state_hub_task_id: "08ea9646-c078-47f9-9d90-29e07fb360b0"
```

**Operator acceptance gate** — engineering deliverables complete; execute when
identity/OpenBao drain finishes. Gated on T03–T05 drains reaching identity/OpenBao. Final inventory sweep,
data archival (episodic memory of the machine's history), DNS/cert plan for
`*.coulomb.social` names, then execute the machine phoenix via T08 automation:
wipe, rebuild as railiance02, join the fleet. Longhorn/PG-HA
(`RAIL-BS-WP-0007`, `CUST-WP-0038`) unlock once railiance01 + railiance02 are
both fleet nodes.

## Task: Workstation-off acceptance test

```task
id: CUST-WP-0054-T10
status: wait
priority: high
state_hub_task_id: "5678e806-cf63-4443-a4a8-f9663bd1acc4"
```

**Operator acceptance gate** — plan proof criterion; run after T05 closeout.
The plan's proof: workstation fully offline for 24h+ (no tunnels, no
processes). Verify afterwards from evidence alone: scheduled triage ran and
validated, consistency sweeps ran, issue emission works, hub/API/dashboards
served, forge and CI available. Then verify the inverse: workstation boots,
`custodian dev up` gives a working offline dev hub, and reconnect relays
buffered events. Done when both directions pass without manual repair.

## Sequencing

```
T01 (architecture + register)
 ├─ T02 de-hub network        ── unblocks most of T10's first half
 ├─ T03 placement plan/freeze ─┬─ T04 forge + CI ─┐
 │                             └─ T05 hub home    ├─ T09 decommission → railiance02
 ├─ T06 sink decoupling                           │
 ├─ T07 dev beachhead                             │
 └─ T08 phoenix drill ────────────────────────────┘
                                        T10 acceptance (both halves)
```

## Relationship to existing workplans

- Absorbs the *sequencing* of `RAIL-HO-WP-0005` (Forgejo) and
  `CUST-WP-0011-T08/T09` (WSL2 fallback retirement folds into T10).
- Builds on finished `RAIL-BS-WP-0006` (staged promotion) and `NET-WP-0020`
  (unseal automation).
- Defers to `RAIL-BS-WP-0007` / `CUST-WP-0038` for true 3-node HA once
  railiance02 exists.
- Extends `STATE-WP-0068` (offline write buffer) as the T07 relay mechanism.
- `CORE-WP-0005` stabilization + `CORE-WP-0007` Haskell retirement proceed
  independently; Core Hub simply appears in the T03 drain sequence.