From d98b8263309198a407784041361e92a8a852889e Mon Sep 17 00:00:00 2001 From: tegwick Date: Fri, 21 Aug 2026 13:50:02 +0200 Subject: [PATCH] ops(RMGR-WP-0005): contain stale-lineage production sweep --- WORK-RECORDS.md | 20 +++++- ...gistrar-consolidation-deterministic-ids.md | 68 ++++++++++++++++++- 2 files changed, 86 insertions(+), 2 deletions(-) diff --git a/WORK-RECORDS.md b/WORK-RECORDS.md index d1dd87d..482ad24 100644 --- a/WORK-RECORDS.md +++ b/WORK-RECORDS.md @@ -15,6 +15,8 @@ | workplan | RMGR-WP-0005 | active | — | workplans/RMGR-WP-0005-registrar-consolidation-deterministic-ids.md | | workplan | RMGR-WP-0006 | finished | — | workplans/RMGR-WP-0006-railiance-app-wrapper-setup.md | | workplan | RMGR-WP-0007 | finished | — | workplans/RMGR-WP-0007-greenfield-rapp-wrap-efficiency.md | +| workplan | RMGR-WP-0008 | proposed | — | workplans/RMGR-WP-0008-work-record-and-register-receiving-surface.md | +| workplan | RMGR-WP-0009 | proposed | — | workplans/RMGR-WP-0009-coding-assistant-commit-provenance.md | | task | RMGR-WP-0001-T01 | done | — | workplans/RMGR-WP-0001-foundation.md | | task | RMGR-WP-0001-T02 | done | — | workplans/RMGR-WP-0001-foundation.md | | task | RMGR-WP-0001-T03 | done | — | workplans/RMGR-WP-0001-foundation.md | @@ -41,7 +43,7 @@ | task | RMGR-WP-0004-T09 | wait | — | workplans/RMGR-WP-0004-repository-standards-conformance.md | | task | RMGR-WP-0005-T01 | done | — | workplans/RMGR-WP-0005-registrar-consolidation-deterministic-ids.md | | task | RMGR-WP-0005-T02 | wait | — | workplans/RMGR-WP-0005-registrar-consolidation-deterministic-ids.md | -| task | RMGR-WP-0005-T03 | wait | — | workplans/RMGR-WP-0005-registrar-consolidation-deterministic-ids.md | +| task | RMGR-WP-0005-T03 | todo | — | workplans/RMGR-WP-0005-registrar-consolidation-deterministic-ids.md | | task | RMGR-WP-0005-T04 | wait | — | workplans/RMGR-WP-0005-registrar-consolidation-deterministic-ids.md | | task | RMGR-WP-0005-T05 | wait | — | workplans/RMGR-WP-0005-registrar-consolidation-deterministic-ids.md | | task | RMGR-WP-0005-T06 | done | — | workplans/RMGR-WP-0005-registrar-consolidation-deterministic-ids.md | @@ -49,6 +51,8 @@ | task | RMGR-WP-0005-T08 | wait | — | workplans/RMGR-WP-0005-registrar-consolidation-deterministic-ids.md | | task | RMGR-WP-0005-T09 | wait | — | workplans/RMGR-WP-0005-registrar-consolidation-deterministic-ids.md | | task | RMGR-WP-0005-T10 | wait | — | workplans/RMGR-WP-0005-registrar-consolidation-deterministic-ids.md | +| task | RMGR-WP-0005-T11 | done | — | workplans/RMGR-WP-0005-registrar-consolidation-deterministic-ids.md | +| task | RMGR-WP-0005-T12 | wait | — | workplans/RMGR-WP-0005-registrar-consolidation-deterministic-ids.md | | task | RMGR-WP-0006-T01 | done | — | workplans/RMGR-WP-0006-railiance-app-wrapper-setup.md | | task | RMGR-WP-0006-T02 | done | — | workplans/RMGR-WP-0006-railiance-app-wrapper-setup.md | | task | RMGR-WP-0006-T03 | done | — | workplans/RMGR-WP-0006-railiance-app-wrapper-setup.md | @@ -61,3 +65,17 @@ | task | RMGR-WP-0007-T05 | done | — | workplans/RMGR-WP-0007-greenfield-rapp-wrap-efficiency.md | | task | RMGR-WP-0007-T06 | done | — | workplans/RMGR-WP-0007-greenfield-rapp-wrap-efficiency.md | | task | RMGR-WP-0007-T07 | done | — | workplans/RMGR-WP-0007-greenfield-rapp-wrap-efficiency.md | +| task | RMGR-WP-0008-T01 | todo | — | workplans/RMGR-WP-0008-work-record-and-register-receiving-surface.md | +| task | RMGR-WP-0008-T02 | todo | — | workplans/RMGR-WP-0008-work-record-and-register-receiving-surface.md | +| task | RMGR-WP-0008-T03 | todo | — | workplans/RMGR-WP-0008-work-record-and-register-receiving-surface.md | +| task | RMGR-WP-0008-T04 | todo | — | workplans/RMGR-WP-0008-work-record-and-register-receiving-surface.md | +| task | RMGR-WP-0008-T05 | todo | — | workplans/RMGR-WP-0008-work-record-and-register-receiving-surface.md | +| task | RMGR-WP-0008-T06 | todo | — | workplans/RMGR-WP-0008-work-record-and-register-receiving-surface.md | +| task | RMGR-WP-0008-T07 | wait | — | workplans/RMGR-WP-0008-work-record-and-register-receiving-surface.md | +| task | RMGR-WP-0009-T01 | todo | — | workplans/RMGR-WP-0009-coding-assistant-commit-provenance.md | +| task | RMGR-WP-0009-T02 | todo | — | workplans/RMGR-WP-0009-coding-assistant-commit-provenance.md | +| task | RMGR-WP-0009-T03 | todo | — | workplans/RMGR-WP-0009-coding-assistant-commit-provenance.md | +| task | RMGR-WP-0009-T04 | todo | — | workplans/RMGR-WP-0009-coding-assistant-commit-provenance.md | +| task | RMGR-WP-0009-T05 | todo | — | workplans/RMGR-WP-0009-coding-assistant-commit-provenance.md | +| task | RMGR-WP-0009-T06 | todo | — | workplans/RMGR-WP-0009-coding-assistant-commit-provenance.md | +| task | RMGR-WP-0009-T07 | todo | — | workplans/RMGR-WP-0009-coding-assistant-commit-provenance.md | diff --git a/workplans/RMGR-WP-0005-registrar-consolidation-deterministic-ids.md b/workplans/RMGR-WP-0005-registrar-consolidation-deterministic-ids.md index cae8881..1d83647 100644 --- a/workplans/RMGR-WP-0005-registrar-consolidation-deterministic-ids.md +++ b/workplans/RMGR-WP-0005-registrar-consolidation-deterministic-ids.md @@ -8,7 +8,7 @@ status: active owner: codex topic_slug: infotech created: "2026-08-17" -updated: "2026-08-18" +updated: "2026-08-21" parent_project: prj-state-hub-retirement parent_workplan: SHR-WP-0001 related: @@ -142,6 +142,72 @@ Reversing the order mints into the wrong lineage at fleet scale. those 70 stale checkouts. Establish what it has been pushing to `gitea-remote` before changing anything. +## Contain the stale-lineage production sweep + +```task +id: RMGR-WP-0005-T11 +status: done +priority: high +``` + +Audit the active `state-hub` sweep workload on `railiance01` before enabling the +registrar. Establish which repositories it has fetched from or pushed to +`gitea-remote`, preserve non-secret evidence of the workload configuration and +recent Git outcomes, and prevent further stale-lineage writes with the smallest +reversible control. + +Do not rewrite remote URLs as containment: the checkouts have diverged and need +the governed reconciliation described above. Do not enable +`STATEHUB_REGISTRAR` while any swept checkout still targets `gitea-remote`. + +Done when the production sweep cannot push repository changes to the stale +lineage, normal State Hub serving remains available, the SSH hostPath exposure +is recorded for remediation, and the control and rollback are documented. + +**Result (2026-08-21):** contained without interrupting the State Hub API. + +- Production evidence showed the 15-minute Temporal schedule had fired 5,501 + times. The final runs processed 12–15 repositories each, but every Git fetch + and push failed with `Resource temporarily unavailable`; C-16/C-17 prevented + further writes where repositories were behind or had unpushed commits. No + checkout commit or reflog activity was found after 2026-08-19. The latent + stale-lineage write path nevertheless remained live. +- Paused Temporal schedule + `activity-schedule-7c4e9a12-8f3b-4d5e-9c6a-1b2d3e4f5a6b`. Its last run is + fixed at `2026-08-21T10:00:00Z`; subsequent intervals did not fire. +- Rolled State Hub deployment revision 11 with `/home/tegwick` read-only and + the `/home/tegwick/.ssh` mount removed. The replacement pod is Ready and + `/state/health` remains healthy. +- Made containment durable: State Hub production Helm sweep disabled in commit + `2841bf3`; activity-core projection disabled in `5793eb3`; Custodian-owned + definition disabled in `22d9366`. All three commits are on Forgejo `main`. +- Live activity-core ConfigMap and definition row say `enabled: false`; the + Helm chart passes lint and renders no sweep/SSH mounts; activity-core targeted + tests pass (16 tests). + +Rollback is deliberately gated: do not unpause the Temporal schedule or enable +the Helm sweep until the `railiance01` checkouts are reconciled to +`forgejo-remote`, the registrar preflight passes, and T12 provides a scoped +credential path that does not mount an operator home or private-key directory. + +## Replace the host-wide sweep credential with a scoped identity + +```task +id: RMGR-WP-0005-T12 +status: wait +priority: high +``` + +The retired sweep design mounted all of `/home/tegwick` read-write and mounted +`/home/tegwick/.ssh` into the State Hub container. Before any remote sweep is +re-enabled, replace that host-wide authority with a workload-specific identity +and explicit repository scope. The runtime must not receive an operator private +key, an operator home directory, or implicit write access to every checkout. + +Coordinate credential custody with the platform owner and keep the schedule +disabled until positive allowed-repository and negative unrelated-repository +push evidence exist without exposing credential values. + ## Re-register identifiers minted outside the registrar ```task