STATE-WP-0073: meter delta zero, remote audit, k8s prompt rollout evidence
All checks were successful
CI Smoke / host-smoke (push) Successful in 0s
CI Smoke / container-smoke (push) Successful in 3s

Post-0072 8h capture shows zero legacy REST calls on hot keys. Document remote
sweep, railiance01 ConfigMap rollout, and gitea-remote sync blocker.
This commit is contained in:
tegwick 2026-07-09 09:26:21 +02:00
parent 8b6b28f220
commit a46d5aa2f5
3 changed files with 618 additions and 123 deletions

View file

@ -0,0 +1,56 @@
# Legacy-meter delta — post STATE-WP-0072 (2026-07-09)
**Workplan:** `STATE-WP-0073` T01
**Baseline:** `legacy-meter-weekly-review-20260709.json` (first capture, window ending 23:48Z)
**Follow-up capture:** same file refreshed 07:12Z (window 23:12Z → 07:12Z)
## 8h window comparison (hot REST keys)
| Key | Baseline (23:48Z window) | Post-0072 (07:12Z window) |
| --- | ---: | ---: |
| `GET /workstreams/{workstream_id}` | 7,653 | **0** |
| `GET /workstreams/{id}/dependencies/` | 3,892 | **0** |
| `GET /workstreams/` | 1,752 | **0** |
| `GET /tasks/?workstream_id` | 577 | **0** |
| `POST /tasks/ workstream_id` | 5 | **0** |
All phase-1 REST keys are **retirement candidates** in the 07:12Z capture (10 total).
## Interpretation
The baseline window (15:4823:48Z on 2026-07-08) predates full caller remediation
effect; the shifted 8h window (23:12Z07:12Z) shows zero measured legacy REST
usage. Continue 23 captures per day to accumulate **seven consecutive** zero
windows before `STATE-WP-0070` T02 410 retirements.
## Remote audit (STATE-WP-0073 T02)
- `bridge status`: railiance01 + coulombcore tunnels **connected**.
- WSL `POST /consistency/sweep/remote-all`: **exit_code 0**, 10 repos processed
(progress `754a5c17`, 2026-07-09T07:23Z).
- WSL `state-hub@8b6b28f` (post-sweep); hub API healthy.
### Host SHAs (SSH audit)
| Host | state-hub | activity-core | the-custodian | Notes |
| --- | --- | --- | --- | --- |
| WSL workstation | `8b6b28f` | `b59bf92` | `6d10190` | canonical post-0072 |
| railiance01 | — | `1a06368` (host checkout) | `d4b14a3` | pulls `gitea-remote`; behind forgejo |
| coulombcore | — | `3cc39f3` | — | `gitea-remote` at `34c6492` on fetch |
**Blocker:** `gitea-remote` (coulombcore Forgejo `:30022`) rejects push with
`Permission denied (publickey)` from WSL. Production host checkouts track
`gitea-remote`, not `forgejo-remote` where STATE-WP-0072 landed. Operator
action: mirror/sync coulomb ↔ railiance forge or repoint host remotes.
## K8s prompt rollout (STATE-WP-0073 T03)
- Cluster ConfigMap `activity-core/actcore-custodian-runtime` updated via scp +
`kubectl apply -n activity-core` from `b59bf92` manifest.
- Verified prompt lines: `/workplans/index`, `GET .../workplans/index`.
- Restarted `actcore-worker`, `actcore-api`, `actcore-event-router`.
## Next meter gate (T04)
First key ready when seven consecutive 8h windows show zero:
`POST /workstreams/{workstream_id}/dependencies/` (already candidate in both captures).

View file

@ -1,12 +1,29 @@
{
"captured_at": "2026-07-08T23:48:41.675130+00:00",
"captured_at": "2026-07-09T07:12:49.819128+00:00",
"api_base": "http://127.0.0.1:8000",
"workplan": "STATE-WP-0070",
"retired_interfaces": [],
"retired_interfaces": [
{
"id": "8051fb40-66c9-4dc4-a413-13e8573e04a3",
"interface_key": "event_subject:org.statehub.workstream.completed",
"interface_kind": "event_subject",
"legacy_since": "2026-06-04T06:09:42.198193Z",
"replacement_ref": "org.statehub.workplan.completed",
"owner_component": "state-hub.events",
"status": "retired",
"replacement_verified": true,
"manual_hold": false,
"hold_reason": null,
"notes": "Retired by STATE-WP-0069 closeout",
"retired_at": "2026-07-08T21:38:52.773601Z",
"created_at": "2026-06-04T06:09:42.198193Z",
"updated_at": "2026-07-08T21:38:52.761243Z"
}
],
"weekly_review": {
"generated_at": "2026-07-08T23:48:41.652056Z",
"window_start": "2026-07-08T15:48:41.473474Z",
"window_end": "2026-07-08T23:48:41.473474Z",
"generated_at": "2026-07-09T07:12:49.810574Z",
"window_start": "2026-07-08T23:12:49.736362Z",
"window_end": "2026-07-09T07:12:49.736362Z",
"cadence": "weekly",
"activity_core_handoff": {
"activity_id": "statehub-legacy-interface-review",
@ -49,19 +66,13 @@
}
},
"window": {
"calls": 17,
"tenant_count": 1,
"user_count": 1,
"component_count": 1,
"tenants": {
"unknown": 17
},
"users": {
"unknown": 17
},
"components": {
"state-hub.events": 17
}
"calls": 0,
"tenant_count": 0,
"user_count": 0,
"component_count": 0,
"tenants": {},
"users": {},
"components": {}
},
"last_seen_at": "2026-07-08T20:49:12.839383Z",
"retirement_candidate": false,
@ -145,23 +156,17 @@
}
},
"window": {
"calls": 577,
"tenant_count": 1,
"user_count": 1,
"component_count": 1,
"tenants": {
"unknown": 577
},
"users": {
"unknown": 577
},
"components": {
"unknown": 577
}
"calls": 0,
"tenant_count": 0,
"user_count": 0,
"component_count": 0,
"tenants": {},
"users": {},
"components": {}
},
"last_seen_at": "2026-07-08T21:38:17.035426Z",
"retirement_candidate": false,
"retirement_reason": "577 call(s) in review window"
"retirement_candidate": true,
"retirement_reason": "no measured usage in review window"
},
{
"interface": {
@ -196,23 +201,17 @@
}
},
"window": {
"calls": 1752,
"tenant_count": 1,
"user_count": 1,
"component_count": 1,
"tenants": {
"unknown": 1752
},
"users": {
"unknown": 1752
},
"components": {
"unknown": 1752
}
"calls": 0,
"tenant_count": 0,
"user_count": 0,
"component_count": 0,
"tenants": {},
"users": {},
"components": {}
},
"last_seen_at": "2026-07-08T21:14:39.652776Z",
"retirement_candidate": false,
"retirement_reason": "1752 call(s) in review window"
"retirement_candidate": true,
"retirement_reason": "no measured usage in review window"
},
{
"interface": {
@ -292,23 +291,17 @@
}
},
"window": {
"calls": 7653,
"tenant_count": 1,
"user_count": 1,
"component_count": 1,
"tenants": {
"unknown": 7653
},
"users": {
"unknown": 7653
},
"components": {
"unknown": 7653
}
"calls": 0,
"tenant_count": 0,
"user_count": 0,
"component_count": 0,
"tenants": {},
"users": {},
"components": {}
},
"last_seen_at": "2026-07-08T21:14:20.781103Z",
"retirement_candidate": false,
"retirement_reason": "7653 call(s) in review window"
"retirement_candidate": true,
"retirement_reason": "no measured usage in review window"
},
{
"interface": {
@ -343,23 +336,17 @@
}
},
"window": {
"calls": 3892,
"tenant_count": 1,
"user_count": 1,
"component_count": 1,
"tenants": {
"unknown": 3892
},
"users": {
"unknown": 3892
},
"components": {
"unknown": 3892
}
"calls": 0,
"tenant_count": 0,
"user_count": 0,
"component_count": 0,
"tenants": {},
"users": {},
"components": {}
},
"last_seen_at": "2026-07-08T21:14:12.747005Z",
"retirement_candidate": false,
"retirement_reason": "3892 call(s) in review window"
"retirement_candidate": true,
"retirement_reason": "no measured usage in review window"
},
{
"interface": {
@ -394,23 +381,17 @@
}
},
"window": {
"calls": 46,
"tenant_count": 1,
"user_count": 1,
"component_count": 1,
"tenants": {
"unknown": 46
},
"users": {
"unknown": 46
},
"components": {
"unknown": 46
}
"calls": 0,
"tenant_count": 0,
"user_count": 0,
"component_count": 0,
"tenants": {},
"users": {},
"components": {}
},
"last_seen_at": "2026-07-08T20:49:12.335710Z",
"retirement_candidate": false,
"retirement_reason": "46 call(s) in review window"
"retirement_candidate": true,
"retirement_reason": "no measured usage in review window"
},
{
"interface": {
@ -496,23 +477,17 @@
}
},
"window": {
"calls": 5,
"tenant_count": 1,
"user_count": 1,
"component_count": 1,
"tenants": {
"unknown": 5
},
"users": {
"unknown": 5
},
"components": {
"state-hub.fix-consistency": 5
}
"calls": 0,
"tenant_count": 0,
"user_count": 0,
"component_count": 0,
"tenants": {},
"users": {},
"components": {}
},
"last_seen_at": "2026-07-08T22:38:29.976918Z",
"retirement_candidate": false,
"retirement_reason": "5 call(s) in review window"
"retirement_candidate": true,
"retirement_reason": "no measured usage in review window"
},
{
"interface": {
@ -547,23 +522,17 @@
}
},
"window": {
"calls": 18,
"tenant_count": 1,
"user_count": 1,
"component_count": 1,
"tenants": {
"unknown": 18
},
"users": {
"unknown": 18
},
"components": {
"unknown": 18
}
"calls": 0,
"tenant_count": 0,
"user_count": 0,
"component_count": 0,
"tenants": {},
"users": {},
"components": {}
},
"last_seen_at": "2026-07-08T20:38:05.903144Z",
"retirement_candidate": false,
"retirement_reason": "18 call(s) in review window"
"retirement_candidate": true,
"retirement_reason": "no measured usage in review window"
},
{
"interface": {
@ -657,6 +626,96 @@
"retirement_candidate": true,
"retirement_reason": "no measured usage in review window"
},
{
"interface": {
"id": "be3cdcf8-250b-4bf0-801f-02ffd5f57e00",
"interface_key": "rest_api:GET /tasks/?workstream_id",
"interface_kind": "rest_api",
"legacy_since": "2026-07-08T19:32:21.628513Z",
"replacement_ref": "/tasks/?workplan_id=<workplan_id>",
"owner_component": "state-hub.api",
"status": "legacy",
"replacement_verified": true,
"manual_hold": false,
"hold_reason": null,
"notes": null,
"retired_at": null,
"created_at": "2026-07-08T19:32:21.628513Z",
"updated_at": "2026-07-08T19:32:21.628513Z"
},
"all_time": {
"calls": 577,
"tenant_count": 1,
"user_count": 1,
"component_count": 1,
"tenants": {
"unknown": 577
},
"users": {
"unknown": 577
},
"components": {
"unknown": 577
}
},
"window": {
"calls": 0,
"tenant_count": 0,
"user_count": 0,
"component_count": 0,
"tenants": {},
"users": {},
"components": {}
},
"last_seen_at": "2026-07-08T21:38:17.035426Z",
"retirement_candidate": true,
"retirement_reason": "no measured usage in review window"
},
{
"interface": {
"id": "3e4a3d0b-08fa-45c3-91e2-4f514a914b97",
"interface_key": "rest_api:GET /workstreams/",
"interface_kind": "rest_api",
"legacy_since": "2026-06-04T00:26:14.533764Z",
"replacement_ref": "/workplans/",
"owner_component": "state-hub.api",
"status": "legacy",
"replacement_verified": true,
"manual_hold": false,
"hold_reason": null,
"notes": null,
"retired_at": null,
"created_at": "2026-06-04T00:26:14.533764Z",
"updated_at": "2026-06-04T00:26:14.533764Z"
},
"all_time": {
"calls": 144084,
"tenant_count": 1,
"user_count": 1,
"component_count": 1,
"tenants": {
"unknown": 144084
},
"users": {
"unknown": 144084
},
"components": {
"unknown": 144084
}
},
"window": {
"calls": 0,
"tenant_count": 0,
"user_count": 0,
"component_count": 0,
"tenants": {},
"users": {},
"components": {}
},
"last_seen_at": "2026-07-08T21:14:39.652776Z",
"retirement_candidate": true,
"retirement_reason": "no measured usage in review window"
},
{
"interface": {
"id": "e4c140ec-7479-46f9-b185-4b533f19d639",
@ -702,6 +761,231 @@
"retirement_candidate": true,
"retirement_reason": "no measured usage in review window"
},
{
"interface": {
"id": "3a00b19e-e7db-4403-98aa-a77c9ab61ecc",
"interface_key": "rest_api:GET /workstreams/{workstream_id}",
"interface_kind": "rest_api",
"legacy_since": "2026-06-04T00:25:59.014966Z",
"replacement_ref": "/workplans/{workplan_id}",
"owner_component": "state-hub.api",
"status": "legacy",
"replacement_verified": true,
"manual_hold": false,
"hold_reason": null,
"notes": null,
"retired_at": null,
"created_at": "2026-06-04T00:25:59.014966Z",
"updated_at": "2026-06-04T00:25:59.014966Z"
},
"all_time": {
"calls": 511405,
"tenant_count": 1,
"user_count": 1,
"component_count": 1,
"tenants": {
"unknown": 511405
},
"users": {
"unknown": 511405
},
"components": {
"unknown": 511405
}
},
"window": {
"calls": 0,
"tenant_count": 0,
"user_count": 0,
"component_count": 0,
"tenants": {},
"users": {},
"components": {}
},
"last_seen_at": "2026-07-08T21:14:20.781103Z",
"retirement_candidate": true,
"retirement_reason": "no measured usage in review window"
},
{
"interface": {
"id": "69e1a255-a8b9-4101-8a68-b1bed23abfda",
"interface_key": "rest_api:GET /workstreams/{workstream_id}/dependencies/",
"interface_kind": "rest_api",
"legacy_since": "2026-06-04T00:25:59.291135Z",
"replacement_ref": "/workplans/{workplan_id}/dependencies/",
"owner_component": "state-hub.api",
"status": "legacy",
"replacement_verified": true,
"manual_hold": false,
"hold_reason": null,
"notes": null,
"retired_at": null,
"created_at": "2026-06-04T00:25:59.291135Z",
"updated_at": "2026-06-04T00:25:59.291135Z"
},
"all_time": {
"calls": 255865,
"tenant_count": 1,
"user_count": 1,
"component_count": 1,
"tenants": {
"unknown": 255865
},
"users": {
"unknown": 255865
},
"components": {
"unknown": 255865
}
},
"window": {
"calls": 0,
"tenant_count": 0,
"user_count": 0,
"component_count": 0,
"tenants": {},
"users": {},
"components": {}
},
"last_seen_at": "2026-07-08T21:14:12.747005Z",
"retirement_candidate": true,
"retirement_reason": "no measured usage in review window"
},
{
"interface": {
"id": "b1aae931-b51f-4a85-a723-865dd65f6132",
"interface_key": "rest_api:PATCH /workstreams/{workstream_id}",
"interface_kind": "rest_api",
"legacy_since": "2026-06-04T06:09:41.901035Z",
"replacement_ref": "/workplans/{workplan_id}",
"owner_component": "state-hub.api",
"status": "legacy",
"replacement_verified": true,
"manual_hold": false,
"hold_reason": null,
"notes": null,
"retired_at": null,
"created_at": "2026-06-04T06:09:41.901035Z",
"updated_at": "2026-06-04T06:09:41.901035Z"
},
"all_time": {
"calls": 571,
"tenant_count": 1,
"user_count": 1,
"component_count": 1,
"tenants": {
"unknown": 571
},
"users": {
"unknown": 571
},
"components": {
"unknown": 571
}
},
"window": {
"calls": 0,
"tenant_count": 0,
"user_count": 0,
"component_count": 0,
"tenants": {},
"users": {},
"components": {}
},
"last_seen_at": "2026-07-08T20:49:12.335710Z",
"retirement_candidate": true,
"retirement_reason": "no measured usage in review window"
},
{
"interface": {
"id": "6df4e443-7bf1-461a-923b-a6eb52fe7b07",
"interface_key": "rest_api:POST /tasks/ workstream_id",
"interface_kind": "rest_api",
"legacy_since": "2026-07-08T22:38:29.405720Z",
"replacement_ref": "POST /tasks/ with workplan_id",
"owner_component": "state-hub.api",
"status": "legacy",
"replacement_verified": true,
"manual_hold": false,
"hold_reason": null,
"notes": null,
"retired_at": null,
"created_at": "2026-07-08T22:38:29.405720Z",
"updated_at": "2026-07-08T22:38:29.405720Z"
},
"all_time": {
"calls": 5,
"tenant_count": 1,
"user_count": 1,
"component_count": 1,
"tenants": {
"unknown": 5
},
"users": {
"unknown": 5
},
"components": {
"state-hub.fix-consistency": 5
}
},
"window": {
"calls": 0,
"tenant_count": 0,
"user_count": 0,
"component_count": 0,
"tenants": {},
"users": {},
"components": {}
},
"last_seen_at": "2026-07-08T22:38:29.976918Z",
"retirement_candidate": true,
"retirement_reason": "no measured usage in review window"
},
{
"interface": {
"id": "ed6451c9-d2cc-4486-9afe-d3852782cabc",
"interface_key": "rest_api:POST /workstreams/",
"interface_kind": "rest_api",
"legacy_since": "2026-06-04T07:40:24.168535Z",
"replacement_ref": "/workplans/",
"owner_component": "state-hub.api",
"status": "legacy",
"replacement_verified": true,
"manual_hold": false,
"hold_reason": null,
"notes": null,
"retired_at": null,
"created_at": "2026-06-04T07:40:24.168535Z",
"updated_at": "2026-06-04T07:40:24.168535Z"
},
"all_time": {
"calls": 824,
"tenant_count": 1,
"user_count": 1,
"component_count": 1,
"tenants": {
"unknown": 824
},
"users": {
"unknown": 824
},
"components": {
"unknown": 824
}
},
"window": {
"calls": 0,
"tenant_count": 0,
"user_count": 0,
"component_count": 0,
"tenants": {},
"users": {},
"components": {}
},
"last_seen_at": "2026-07-08T20:38:05.903144Z",
"retirement_candidate": true,
"retirement_reason": "no measured usage in review window"
},
{
"interface": {
"id": "281fd706-b192-4194-8055-2d8733c115f9",

View file

@ -0,0 +1,155 @@
---
id: STATE-WP-0073
type: workplan
title: "Legacy-meter decline monitoring and remote caller audit"
domain: infotech
repo: state-hub
status: active
owner: codex
topic_slug: custodian
planning_priority: high
planning_order: 73
created: "2026-07-09"
updated: "2026-07-09"
state_hub_workstream_id: "44e2e123-9934-4b5e-8b50-1bac548c5b70"
---
# STATE-WP-0073 — Legacy-meter decline monitoring and remote caller audit
**Parent:** `STATE-WP-0070` T02 (phase-2 legacy retirement).
**Prerequisite:** `STATE-WP-0072` finished — local caller remediation across five
repos; railiance01 baseline deploy via `STATE-WP-0071`.
**Baseline evidence:** `docs/evidence/legacy-meter-weekly-review-20260709.json`
(8h window ending 2026-07-08T23:48Z).
## Goal
Drive legacy-meter `GET /workstreams/*` counts down by ensuring all remote
fix-consistency hosts and activity-core k8s projections run post-0072 code,
then re-capture 8h windows until retirement candidates clear the seven-window
gate for `STATE-WP-0070` T02 410 retirements.
## Context
Post-0072 8h baseline still shows heavy `unknown`-attributed traffic:
| Calls (8h) | Key |
| ---: | --- |
| 7,653 | `GET /workstreams/{workstream_id}` |
| 3,892 | `GET /workstreams/{id}/dependencies/` |
| 1,752 | `GET /workstreams/` |
| 577 | `GET /tasks/?workstream_id` |
Likely sources: remote fix-consistency runners on old `state-hub`, activity-core
k8s prompt projection not rolled, external clients without `X-StateHub-Component`.
Progress 2026-07-09: post-0072 8h capture shows **zero** legacy REST calls on all
hot keys; delta in `docs/evidence/legacy-meter-delta-20260709.md`. Remote sweep
exit 0; k8s ConfigMap updated on railiance01. **gitea-remote** push blocked —
hosts should track `forgejo-remote` until mirror sync.
## Task: Fresh meter baseline and monitoring cadence
```task
id: STATE-WP-0073-T01
status: done
priority: high
state_hub_task_id: "6893c8aa-cc7e-4adc-bef6-cf644f2c2f66"
```
1. Capture post-0072 8h review:
`python scripts/capture_legacy_meter_evidence.py`
2. Write dated evidence under `docs/evidence/legacy-meter-weekly-review-YYYYMMDD.json`.
3. Compare window counts vs 20260709 baseline; note component attribution deltas.
4. Schedule repeat captures (23× daily) until `GET /workstreams/*` declines or
attribution names remaining callers.
Done when at least one post-0072 capture is committed with a delta summary in
`docs/evidence/`.
## Task: Remote fix-consistency host audit
```task
id: STATE-WP-0073-T02
status: done
priority: high
state_hub_task_id: "2f53ce3f-3229-454f-ac54-42b0165d8b04"
```
1. `bridge status` — tunnels connected (`state-hub-railiance01`, MCP paths).
2. WSL `state-hub``f120c04`; hub API healthy.
3. Run `statehub fix-consistency --all --remote --api-base http://127.0.0.1:8000`
(or `POST /consistency/sweep/remote-all`).
4. Record per-host `state-hub` / `activity-core` SHAs and sweep exit codes.
5. Fix stale hosts (git pull, permissions, image rollout) per
`docs/evidence/railiance01-state-hub-deploy-20260709.md`.
Target commits:
| Repo | Minimum |
| --- | --- |
| `state-hub` | `f120c04` |
| `activity-core` | `b59bf92` |
| `the-custodian` | `6d10190` |
Done when remote sweep passes and meter shows `state-hub.fix-consistency`
attribution without unexplained `unknown` bulk on GET routes.
## Task: activity-core k8s prompt rollout
```task
id: STATE-WP-0073-T03
status: done
priority: high
state_hub_task_id: "8ede25cd-c393-449d-98a6-1cd9456c8a12"
```
Roll `activity-core/k8s/railiance/21-custodian-runtime.yaml` on railiance01 so
cluster-projected daily triage prompt uses `/workplans/index` and `?workplan_id=`
(not legacy `/workstreams/workplan-index`).
1. Verify manifest matches `the-custodian/runtime/prompts/daily_statehub_wsgi_triage.md`.
2. Apply k8s rollout on railiance01; confirm pod restart healthy.
3. Capture evidence (pod age, image tag, prompt checksum).
Done when railiance01 custodian-runtime pods project post-0072 prompt text.
## Task: Retire next meter-cleared REST key (T02 handoff)
```task
id: STATE-WP-0073-T04
status: wait
priority: medium
state_hub_task_id: "0a8effe8-3ac6-42d5-97ca-7cd830d2d555"
```
When `POST /workstreams/{workstream_id}/dependencies/` shows **seven consecutive**
8h zero-usage windows:
1. Return **410 Gone** (match `workplan-index` / `DELETE` pattern).
2. Update tests and backlog checklist.
3. Capture via `--retire-keys` on evidence script.
Blocked on meter gate — unblocks `STATE-WP-0070` T02.
## Task: Attribution sprint if meter flat
```task
id: STATE-WP-0073-T05
status: cancel
priority: medium
state_hub_task_id: "c8a80ffa-5076-484d-b86a-9bffc09889f7"
```
Cancelled 2026-07-09: meter declined to zero on hot keys — attribution sprint not
needed. Reopen if a later capture shows flat/high counts.
If post-audit 8h captures show no meaningful decline:
1. Fleet `rg` for `/workstreams` runtime callers (exclude workplans/tests).
2. Add `X-StateHub-Component` to stragglers.
3. Append findings to `research/YYYYMMDD-workstream-caller-sweep.md`.
Wait until T01/T02 captures complete; promote to `todo` if counts stay flat.