From 5921b65c49048aebbf5d08403288fc81c243783a Mon Sep 17 00:00:00 2001 From: tegwick Date: Tue, 11 Aug 2026 02:32:14 +0200 Subject: [PATCH] feat(RMGR-WP-0003): production pilot dual-run config, bulk, push Config file dual-run (env override), writeback_push, bulk-status facade on State Hub, pilot example config, evidence and finished workplan. --- config/dual-run.pilot.example.yaml | 10 ++ docs/dual-run.md | 83 ++++++----- docs/evidence/wp0003-completion.md | 26 ++++ src/repo_manager/dual_run.py | 130 +++++++++++++++--- tests/test_dual_run.py | 37 ++++- .../RMGR-WP-0003-production-pilot-dual-run.md | 93 +++++++++++++ 6 files changed, 324 insertions(+), 55 deletions(-) create mode 100644 config/dual-run.pilot.example.yaml create mode 100644 docs/evidence/wp0003-completion.md create mode 100644 workplans/RMGR-WP-0003-production-pilot-dual-run.md diff --git a/config/dual-run.pilot.example.yaml b/config/dual-run.pilot.example.yaml new file mode 100644 index 0000000..27a6311 --- /dev/null +++ b/config/dual-run.pilot.example.yaml @@ -0,0 +1,10 @@ +# Example dual-run pilot config (RMGR-WP-0003). +# Install: cp config/dual-run.pilot.example.yaml ~/.repo-manager/dual-run.yaml +# Env vars still override when set. Rollback: writeback/reconcile false or delete file. + +writeback: true +reconcile: true +writeback_push: false # set true only if pilot checkouts can push safely +pilot_repos: + - repo-manager +meter_path: ~/.repo-manager/mutation-meter.jsonl diff --git a/docs/dual-run.md b/docs/dual-run.md index 75cdaab..ab8cce7 100644 --- a/docs/dual-run.md +++ b/docs/dual-run.md @@ -1,50 +1,69 @@ -# Dual-run flags (RMGR-WP-0002) +# Dual-run flags (RMGR-WP-0002 / 0003) -State Hub remains the MCP/REST entrypoint. When flags are on, **checkout +State Hub remains the MCP/REST entrypoint. When dual-run is on, **checkout mutation** is executed by Repo Manager. -## Environment variables +## Config file (preferred for production pilot) + +```bash +mkdir -p ~/.repo-manager +cp ~/repo-manager/config/dual-run.pilot.example.yaml ~/.repo-manager/dual-run.yaml +# edit pilot_repos / writeback_push as needed +``` + +```yaml +writeback: true +reconcile: true +writeback_push: false +pilot_repos: + - repo-manager +meter_path: ~/.repo-manager/mutation-meter.jsonl +``` + +Optional path override: `RM_DUAL_RUN_CONFIG=/path/to/dual-run.yaml` + +**Restart State Hub API** after config changes so the process reloads +(settings are cached per process). + +## Environment variables (override config when set) | Variable | Meaning | | --- | --- | | `RM_WRITEBACK=1` | Task file writeback via `rmgr update-task-status` | | `RM_RECONCILE=1` | `fix-consistency` also runs `rmgr reconcile` for pilots | -| `RM_PILOT_REPOS=repo-manager` | Comma-separated slugs; **unset** = all repos when flags on | -| `RM_METER_PATH` | JSONL meter path (default `~/.repo-manager/mutation-meter.jsonl`) | -| `REPO_MANAGER_SRC` | Optional path to `repo-manager/src` for SH adapter | -| `RMGR_BIN` | Optional override command prefix for `rmgr` | +| `RM_WRITEBACK_PUSH=1` | `git push` after RM writeback commit (best-effort) | +| `RM_PILOT_REPOS=slug1,slug2` | Restrict to slugs; **unset** = all when flags on | +| `RM_METER_PATH` | JSONL meter path | +| `REPO_MANAGER_SRC` | Path to `repo-manager/src` for SH adapter | +| `RMGR_BIN` | Override `rmgr` command | ## Rollback ```bash -unset RM_WRITEBACK RM_RECONCILE RM_PILOT_REPOS -# or -export RM_WRITEBACK=0 RM_RECONCILE=0 +# config: set writeback/reconcile false, or: +rm ~/.repo-manager/dual-run.yaml +unset RM_WRITEBACK RM_RECONCILE RM_WRITEBACK_PUSH RM_PILOT_REPOS +# restart SH API ``` -State Hub immediately uses native C-15 writeback again. - -## Pilot (repo-manager) - -```bash -export RM_WRITEBACK=1 -export RM_RECONCILE=1 -export RM_PILOT_REPOS=repo-manager -export REPO_MANAGER_SRC=$HOME/repo-manager/src - -# From API host environment, PATCH a task or: -cd ~/repo-manager -rmgr update-task-status --path . --task-id --status progress - -cd ~/repo-manager && statehub fix-consistency -rmgr dual-run-status -``` - -## Meter +## Operator checks ```bash rmgr dual-run-status -# or -wc -l ~/.repo-manager/mutation-meter.jsonl -grep repo-manager ~/.repo-manager/mutation-meter.jsonl | tail +# or inspect meter +tail -20 ~/.repo-manager/mutation-meter.jsonl ``` + +## Paths covered + +| Entry | Dual-run when flagged | +| --- | --- | +| `PATCH /tasks/{id}` | RM writeback (+ optional push) | +| `POST /tasks/bulk-status-sync` | RM writeback per task | +| C-15 fix-consistency writeback | RM writeback, else native | +| fix-consistency fix_repo | RM reconcile for pilots | + +## Pilot expand (STATE-WP-0079) + +Add slugs to `pilot_repos`, restart SH, watch meter for `source=repo-manager` +writebacks. Expand only when failures are absent under normal agent load. diff --git a/docs/evidence/wp0003-completion.md b/docs/evidence/wp0003-completion.md new file mode 100644 index 0000000..fd8cdb2 --- /dev/null +++ b/docs/evidence/wp0003-completion.md @@ -0,0 +1,26 @@ +# RMGR-WP-0003 completion + +**Date:** 2026-08-11 + +## Delivered + +| Item | Location | +| --- | --- | +| Config-file dual-run | `dual_run.py` (RM + SH) | +| Example pilot config | `config/dual-run.pilot.example.yaml` | +| Bulk dual-run | SH `POST /tasks/bulk-status-sync` | +| writeback_push | config/env + facade | +| Runbook | `docs/dual-run.md` | + +## Enable production pilot + +```bash +mkdir -p ~/.repo-manager +cp ~/repo-manager/config/dual-run.pilot.example.yaml ~/.repo-manager/dual-run.yaml +# restart State Hub API process to load config +rmgr dual-run-status +``` + +## Expand for STATE-WP-0079 + +Edit `pilot_repos` list, restart SH, watch `~/.repo-manager/mutation-meter.jsonl`. diff --git a/src/repo_manager/dual_run.py b/src/repo_manager/dual_run.py index f2c5799..c4c4311 100644 --- a/src/repo_manager/dual_run.py +++ b/src/repo_manager/dual_run.py @@ -1,13 +1,15 @@ -"""Dual-run flags and checkout-mutation meter (RMGR-WP-0002 / ArchitectureBlueprint Stage B). +"""Dual-run flags and checkout-mutation meter (RMGR-WP-0002/0003). -Environment variables (shared with State Hub adapter): +Precedence (highest first): + 1. Environment variables when set + 2. Config file (~/.repo-manager/dual-run.yaml or RM_DUAL_RUN_CONFIG) + 3. Defaults (all off) - RM_WRITEBACK=1|true|yes — file+git task writeback via repo-manager - RM_RECONCILE=1|true|yes — reconcile path prefers repo-manager for pilot repos - RM_PILOT_REPOS=slug1,slug2 — if set, flags only apply to these slugs; empty = all - RM_METER_PATH=~/.repo-manager/mutation-meter.jsonl — append-only meter log +Environment variables: + RM_WRITEBACK, RM_RECONCILE, RM_WRITEBACK_PUSH, RM_PILOT_REPOS, RM_METER_PATH, + RM_DUAL_RUN_CONFIG -Rollback: unset flags or set to 0 → State Hub native path only. +Rollback: writeback/reconcile false in config and unset env → State Hub native. """ from __future__ import annotations @@ -15,33 +17,106 @@ from __future__ import annotations import json import os from datetime import datetime, timezone +from functools import lru_cache from pathlib import Path from typing import Any, Literal +import yaml + Source = Literal["state-hub", "repo-manager"] +_TRUE = frozenset({"1", "true", "yes", "on"}) +_FALSE = frozenset({"0", "false", "no", "off"}) -def _truthy(name: str) -> bool: - return os.environ.get(name, "").strip().lower() in {"1", "true", "yes", "on"} + +def config_path() -> Path: + raw = os.environ.get("RM_DUAL_RUN_CONFIG", "~/.repo-manager/dual-run.yaml") + return Path(raw).expanduser() + + +@lru_cache(maxsize=1) +def _load_file_config() -> dict[str, Any]: + path = config_path() + if not path.is_file(): + return {} + try: + data = yaml.safe_load(path.read_text(encoding="utf-8")) or {} + except (OSError, yaml.YAMLError): + return {} + return data if isinstance(data, dict) else {} + + +def reload_config() -> None: + """Clear cached config (tests / SIGHUP-style).""" + _load_file_config.cache_clear() + + +def _env_bool(name: str) -> bool | None: + raw = os.environ.get(name) + if raw is None or raw.strip() == "": + return None + v = raw.strip().lower() + if v in _TRUE: + return True + if v in _FALSE: + return False + return None + + +def _cfg_bool(key: str, default: bool = False) -> bool: + env_map = { + "writeback": "RM_WRITEBACK", + "reconcile": "RM_RECONCILE", + "writeback_push": "RM_WRITEBACK_PUSH", + } + env_name = env_map.get(key) + if env_name: + ev = _env_bool(env_name) + if ev is not None: + return ev + cfg = _load_file_config() + if key in cfg: + val = cfg[key] + if isinstance(val, bool): + return val + if isinstance(val, str): + return val.strip().lower() in _TRUE + return default def writeback_enabled() -> bool: - return _truthy("RM_WRITEBACK") + return _cfg_bool("writeback", False) def reconcile_enabled() -> bool: - return _truthy("RM_RECONCILE") + return _cfg_bool("reconcile", False) + + +def writeback_push_enabled() -> bool: + return _cfg_bool("writeback_push", False) def pilot_slugs() -> set[str] | None: - """None means all repos; empty set after parse of blank list means none.""" + """None = all repos; empty set = none.""" raw = os.environ.get("RM_PILOT_REPOS") - if raw is None: + if raw is not None: + raw = raw.strip() + if not raw: + return set() + return {s.strip() for s in raw.split(",") if s.strip()} + cfg = _load_file_config() + if "pilot_repos" not in cfg: return None - raw = raw.strip() - if not raw: - return set() - return {s.strip() for s in raw.split(",") if s.strip()} + val = cfg["pilot_repos"] + if val is None: + return None + if isinstance(val, str): + if not val.strip(): + return set() + return {s.strip() for s in val.split(",") if s.strip()} + if isinstance(val, list): + return {str(s).strip() for s in val if str(s).strip()} + return None def slug_allowed(slug: str | None) -> bool: @@ -62,8 +137,13 @@ def reconcile_for_repo(slug: str | None) -> bool: def meter_path() -> Path: - raw = os.environ.get("RM_METER_PATH", "~/.repo-manager/mutation-meter.jsonl") - return Path(raw).expanduser() + raw = os.environ.get("RM_METER_PATH") + if raw: + return Path(raw).expanduser() + cfg = _load_file_config() + if cfg.get("meter_path"): + return Path(str(cfg["meter_path"])).expanduser() + return Path("~/.repo-manager/mutation-meter.jsonl").expanduser() def record_mutation( @@ -73,7 +153,6 @@ def record_mutation( repo_slug: str | None, detail: dict[str, Any] | None = None, ) -> None: - """Append one meter line. Best-effort; never raises to callers.""" try: path = meter_path() path.parent.mkdir(parents=True, exist_ok=True) @@ -112,9 +191,18 @@ def meter_summary(path: Path | None = None) -> dict[str, int]: def flags_status() -> dict[str, Any]: return { + "config_path": str(config_path()), + "config_exists": config_path().is_file(), + "writeback": writeback_enabled(), + "reconcile": reconcile_enabled(), + "writeback_push": writeback_push_enabled(), + "pilot_repos": sorted(pilot_slugs()) if pilot_slugs() is not None else None, + "meter_path": str(meter_path()), + "meter": meter_summary(), + # legacy env-style keys for operators "RM_WRITEBACK": writeback_enabled(), "RM_RECONCILE": reconcile_enabled(), + "RM_WRITEBACK_PUSH": writeback_push_enabled(), "RM_PILOT_REPOS": sorted(pilot_slugs()) if pilot_slugs() is not None else None, "RM_METER_PATH": str(meter_path()), - "meter": meter_summary(), } diff --git a/tests/test_dual_run.py b/tests/test_dual_run.py index 28c6bf3..1570038 100644 --- a/tests/test_dual_run.py +++ b/tests/test_dual_run.py @@ -1,4 +1,4 @@ -"""Dual-run flags, meter, and hardened task status command.""" +"""Dual-run flags, config file, meter, and hardened task status command.""" from __future__ import annotations @@ -48,10 +48,19 @@ x return repo +@pytest.fixture(autouse=True) +def _clear_dual_run_cache(): + dual_run.reload_config() + yield + dual_run.reload_config() + + def test_flags_and_pilot(monkeypatch: pytest.MonkeyPatch): monkeypatch.delenv("RM_WRITEBACK", raising=False) monkeypatch.delenv("RM_RECONCILE", raising=False) monkeypatch.delenv("RM_PILOT_REPOS", raising=False) + monkeypatch.delenv("RM_DUAL_RUN_CONFIG", raising=False) + dual_run.reload_config() assert dual_run.writeback_enabled() is False monkeypatch.setenv("RM_WRITEBACK", "1") @@ -63,6 +72,31 @@ def test_flags_and_pilot(monkeypatch: pytest.MonkeyPatch): assert dual_run.writeback_for_repo("any") is True +def test_config_file(tmp_path: Path, monkeypatch: pytest.MonkeyPatch): + cfg = tmp_path / "dual-run.yaml" + cfg.write_text( + "writeback: true\nreconcile: true\nwriteback_push: true\n" + "pilot_repos:\n - repo-manager\n", + encoding="utf-8", + ) + monkeypatch.delenv("RM_WRITEBACK", raising=False) + monkeypatch.delenv("RM_RECONCILE", raising=False) + monkeypatch.delenv("RM_WRITEBACK_PUSH", raising=False) + monkeypatch.delenv("RM_PILOT_REPOS", raising=False) + monkeypatch.setenv("RM_DUAL_RUN_CONFIG", str(cfg)) + dual_run.reload_config() + assert dual_run.writeback_enabled() is True + assert dual_run.reconcile_enabled() is True + assert dual_run.writeback_push_enabled() is True + assert dual_run.writeback_for_repo("repo-manager") is True + assert dual_run.writeback_for_repo("other") is False + + # env overrides file + monkeypatch.setenv("RM_WRITEBACK", "0") + dual_run.reload_config() + assert dual_run.writeback_enabled() is False + + def test_meter(tmp_path: Path, monkeypatch: pytest.MonkeyPatch): meter = tmp_path / "meter.jsonl" monkeypatch.setenv("RM_METER_PATH", str(meter)) @@ -106,7 +140,6 @@ def test_update_by_uuid_and_idempotency(tmp_path: Path, monkeypatch: pytest.Monk assert r2.status == "applied" assert r2.evidence.get("git_sha") == r1.evidence.get("git_sha") - # Different payload same key → conflict r3 = update_task_status( repo, "11111111-1111-4111-8111-111111111111", diff --git a/workplans/RMGR-WP-0003-production-pilot-dual-run.md b/workplans/RMGR-WP-0003-production-pilot-dual-run.md new file mode 100644 index 0000000..4d8015f --- /dev/null +++ b/workplans/RMGR-WP-0003-production-pilot-dual-run.md @@ -0,0 +1,93 @@ +--- +id: RMGR-WP-0003 +type: workplan +title: "Production pilot dual-run: config, bulk path, push-seal" +domain: infotech +repo: repo-manager +status: finished +owner: codex +topic_slug: repo-manager +created: "2026-08-11" +updated: "2026-08-11" +parent_project: prj-state-hub-retirement +parent_workplan: SHR-WP-0001 +stream: S1 +related: + - RMGR-WP-0002 + - STATE-WP-0079 + - specs/ArchitectureBlueprint.md +--- + +# Production pilot dual-run: config, bulk path, push-seal + +## Goal + +Make Stage B dual-run **operable in production for a pilot set** without relying +only on ad-hoc API process env vars, and close remaining high-frequency mutation +gaps (bulk status, post-writeback push) so STATE-WP-0079 can expand cutover. + +**Not in scope:** full C-rule port, Postgres registry, MCP rewrite, Stage C +host_paths SoR, dashboards. + +## Dual-run config file (shared RM + SH) + +```task +id: RMGR-WP-0003-T01 +status: done +priority: high +``` + +Load dual-run settings from `~/.repo-manager/dual-run.yaml` (and optional +`RM_DUAL_RUN_CONFIG` path), with env vars still overriding. Fields: writeback, +reconcile, pilot_repos, writeback_push, meter_path. Document and unit-test. + +**Result:** Config loader in RM + SH dual_run modules; env overrides file. + +## Bulk task-status dual-run on State Hub + +```task +id: RMGR-WP-0003-T02 +status: done +priority: high +``` + +When dual-run writeback is on for the task's repo, `POST /tasks/bulk-status-sync` +delegates each status change to RM (same as PATCH). Preserve progress events on +SH. Failures log and fall back to native DB-only (file via later C-15). + +**Result:** bulk_status_sync hooks try_writeback_for_task per status change. + +## Push-seal after RM writeback + +```task +id: RMGR-WP-0003-T03 +status: done +priority: high +``` + +When `writeback_push: true` (or `RM_WRITEBACK_PUSH=1`), RM writeback path pushes +after commit (best-effort, never force). SH facade passes push=true for pilot. +Meter push_ok/fail. Align with C-16/C-17 semantics (skip push issues as meter +only; do not force). + +**Result:** writeback_push config/env; SH facade sets push=writeback_push_enabled(). + +## Pilot defaults + operator runbook + +```task +id: RMGR-WP-0003-T04 +status: done +priority: medium +``` + +Ship example config for pilot `repo-manager`, update `docs/dual-run.md`, capture +`docs/evidence/wp0003-*.md` with meter summary command for STATE-WP-0079. + +**Result:** `config/dual-run.pilot.example.yaml`, dual-run.md, evidence note. + +## Acceptance + +- [x] Config file drives dual-run (env override still works) +- [x] Bulk status uses RM writeback when flagged +- [x] Optional push after writeback +- [x] Evidence + runbook for pilot expansion