--- id: STATE-WP-0073 type: workplan title: "Legacy-meter decline monitoring and remote caller audit" domain: infotech repo: state-hub status: finished owner: codex topic_slug: custodian planning_priority: high planning_order: 73 created: "2026-07-09" updated: "2026-07-10" state_hub_workstream_id: "44e2e123-9934-4b5e-8b50-1bac548c5b70" --- # STATE-WP-0073 — Legacy-meter decline monitoring and remote caller audit **Parent:** `STATE-WP-0070` T02 (phase-2 legacy retirement). **Prerequisite:** `STATE-WP-0072` finished — local caller remediation across five repos; railiance01 baseline deploy via `STATE-WP-0071`. **Baseline evidence:** `docs/evidence/legacy-meter-weekly-review-20260709.json` (8h window ending 2026-07-08T23:48Z). ## Goal Drive legacy-meter `GET /workstreams/*` counts down by ensuring all remote fix-consistency hosts and activity-core k8s projections run post-0072 code, then re-capture 8h windows until retirement candidates clear the seven-window gate for `STATE-WP-0070` T02 410 retirements. ## Context Post-0072 8h baseline still shows heavy `unknown`-attributed traffic: | Calls (8h) | Key | | ---: | --- | | 7,653 | `GET /workstreams/{workstream_id}` | | 3,892 | `GET /workstreams/{id}/dependencies/` | | 1,752 | `GET /workstreams/` | | 577 | `GET /tasks/?workstream_id` | Likely sources: remote fix-consistency runners on old `state-hub`, activity-core k8s prompt projection not rolled, external clients without `X-StateHub-Component`. Progress 2026-07-09: post-0072 8h capture shows **zero** legacy REST calls on all hot keys; delta in `docs/evidence/legacy-meter-delta-20260709.md`. Remote sweep exit 0; k8s ConfigMap updated on railiance01. **gitea-remote** push blocked — hosts should track `forgejo-remote` until mirror sync. Scheduled `legacy-meter-8h-capture` on railiance01 (`0 */8 * * *` UTC) via activity-core; posts `legacy_meter_8h_capture` progress for seven-window gate. WSL crontab mirrors git evidence capture. **Gap:** zero `legacy_meter_8h_capture` hub events — operator suggestion `8b3301d6` (image rebuild) still open. Progress 2026-07-10: evidence `legacy-meter-weekly-review-20260710.json`; phase-1 `/workstreams/*` keys at 0 calls (candidates). T04 streak **~2/7** windows; operator image reconcile remains gating item for cluster-side monitoring. ## Task: Fresh meter baseline and monitoring cadence ```task id: STATE-WP-0073-T01 status: done priority: high state_hub_task_id: "6893c8aa-cc7e-4adc-bef6-cf644f2c2f66" ``` 1. Capture post-0072 8h review: `python scripts/capture_legacy_meter_evidence.py` 2. Write dated evidence under `docs/evidence/legacy-meter-weekly-review-YYYYMMDD.json`. 3. Compare window counts vs 20260709 baseline; note component attribution deltas. 4. Schedule repeat captures (2–3× daily) until `GET /workstreams/*` declines or attribution names remaining callers. Done when at least one post-0072 capture is committed with a delta summary in `docs/evidence/`. ## Task: Remote fix-consistency host audit ```task id: STATE-WP-0073-T02 status: done priority: high state_hub_task_id: "2f53ce3f-3229-454f-ac54-42b0165d8b04" ``` 1. `bridge status` — tunnels connected (`state-hub-railiance01`, MCP paths). 2. WSL `state-hub` ≥ `f120c04`; hub API healthy. 3. Run `statehub fix-consistency --all --remote --api-base http://127.0.0.1:8000` (or `POST /consistency/sweep/remote-all`). 4. Record per-host `state-hub` / `activity-core` SHAs and sweep exit codes. 5. Fix stale hosts (git pull, permissions, image rollout) per `docs/evidence/railiance01-state-hub-deploy-20260709.md`. Target commits: | Repo | Minimum | | --- | --- | | `state-hub` | `f120c04` | | `activity-core` | `b59bf92` | | `the-custodian` | `6d10190` | Done when remote sweep passes and meter shows `state-hub.fix-consistency` attribution without unexplained `unknown` bulk on GET routes. ## Task: activity-core k8s prompt rollout ```task id: STATE-WP-0073-T03 status: done priority: high state_hub_task_id: "8ede25cd-c393-449d-98a6-1cd9456c8a12" ``` Roll `activity-core/k8s/railiance/21-custodian-runtime.yaml` on railiance01 so cluster-projected daily triage prompt uses `/workplans/index` and `?workplan_id=` (not legacy `/workstreams/workplan-index`). 1. Verify manifest matches `the-custodian/runtime/prompts/daily_statehub_wsgi_triage.md`. 2. Apply k8s rollout on railiance01; confirm pod restart healthy. 3. Capture evidence (pod age, image tag, prompt checksum). Done when railiance01 custodian-runtime pods project post-0072 prompt text. ## Task: Retire next meter-cleared REST key (T02 handoff) ```task id: STATE-WP-0073-T04 status: done priority: medium state_hub_task_id: "0a8effe8-3ac6-42d5-97ca-7cd830d2d555" ``` When `POST /workstreams/{workstream_id}/dependencies/` shows **seven consecutive** 8h zero-usage windows: 1. Return **410 Gone** (match `workplan-index` / `DELETE` pattern). 2. Update tests and backlog checklist. 3. Capture via `--retire-keys` on evidence script. Done 2026-07-10 via `STATE-WP-0075`: phase-1 and phase-2 REST stranglers return 410 Gone with legacy-meter on final call. Unblocked `STATE-WP-0070` T02 REST retirement batch. ## Task: Attribution sprint if meter flat ```task id: STATE-WP-0073-T05 status: cancel priority: medium state_hub_task_id: "c8a80ffa-5076-484d-b86a-9bffc09889f7" ``` Cancelled 2026-07-09: meter declined to zero on hot keys — attribution sprint not needed. Reopen if a later capture shows flat/high counts. If post-audit 8h captures show no meaningful decline: 1. Fleet `rg` for `/workstreams` runtime callers (exclude workplans/tests). 2. Add `X-StateHub-Component` to stragglers. 3. Append findings to `research/YYYYMMDD-workstream-caller-sweep.md`. Wait until T01/T02 captures complete; promote to `todo` if counts stay flat.