--- id: CUST-WP-0068 type: workplan title: "Recover or disposition the cache-only work records so the retired hub can be discarded" domain: infotech repo: the-custodian status: active owner: codex created: "2026-08-25" updated: "2026-08-25" quality_dor: DoR-Ok quality_dor_at: "2026-08-25" quality_dor_by: codex quality_dor_note: "Divergence measured exactly (297 workplans, 23 of them orphans), every remaining record classified with a named cause, per-repository blockers captured from real error messages, and the mechanism proven on live repositories before rollout." state_hub_workstream_id: "c930b3ec-68b5-596e-ae72-d4ecaf7380e7" --- # Recover or disposition the cache-only work records so the retired hub can be discarded ## Goal Finish what `CUST-WP-0067-T10` started: get every work record that exists only in the retired local hub either onto central or into a written disposition, so the cache database stops being load-bearing and can be discarded. `ADR-010` described this divergence in August 2026 and it has never been closed. ## Context The local hub instance was retired under `CUST-WP-0067`, and its repository records were fully recovered. Its **work records were not**. The database is still running solely because 297 workplans and their tasks exist nowhere else. Measured 2026-08-25, after recovering 129 of the original 426: | Count | Class | Needs | |---|---|---| | 176 | non-empty projection on central | C-06 for the deterministic subset; the rest await an identifier decision | | 58 | file does not parse as a workplan | content fix — 50 are `clay-borg` | | 24 | bootstrap-eligible but blocked | per-repository unblocking | | 23 | orphan, no backing file | written disposition | | 16 | flagged as non-conforming ids | see the correction below | **Correction carried forward.** Ten of those 16 are `ADHOC-YYYY-MM-DD` identifiers, which are the documented ad-hoc convention — the classifier's pattern was wrong, not the files. Only six are genuinely non-conforming: `GLAS-0001`, `REIN-A-0001`–`0004`, `REIN-OW-0001`, plus `RAIL-BS-ADHOC-2026-07-27` on a prefix the registry has retired. The ten adhocs should be re-tested and will most likely fall to the class-F treatment. **What already works.** `--bootstrap-empty-projection` registers a repository's workplans while **preserving their existing identifiers**, verified on a live repository before rollout: workplan and all four tasks landed with original UUIDs and statuses, no file rewritten and no commit. All 403 cache-only workplans that record a backing path still have that file on disk, so this is recovery, not reconstruction. **What does not work, and why.** Only 57 of 403 carry deterministic UUIDv5 identifiers. C-06 refuses non-derived identifiers as `C-03 FAIL — stale reference`, so the ordinary consistency path cannot re-derive most of them. Rewriting 345 identifiers fleet-wide was considered and rejected: it overlaps `RMGR-WP-0005`'s ownership of identifier migration and would churn IDs across ~40 repositories to solve a problem bootstrap already solves for part of the set. ## Repair the registrar's self-defeating precondition ```task id: CUST-WP-0068-T01 status: todo priority: high state_hub_task_id: "e8fbfa1a-3abf-5ec0-8abb-f63604dc23d5" ``` The registrar generates `WORK-RECORDS.md`, which dirties the worktree and then fails its own git precondition. Several runs registered their records successfully and *then* reported failure for the file they had just created; retries report `git_precondition_failed` instead, which points away from the real cause. Any operator running this at scale will misread it exactly as this workplan's predecessor did. Also surface error *messages*, not only codes. `bootstrap_source_invalid` covers at least two unrelated causes — a repository with no parseable workplan, and a task missing its projection UUID — and reading only the code merged them into one wrong diagnosis. Acceptance: a repository whose only change is the generated index completes in one run; failure reports name the specific record and cause. ## Unblock the bootstrap-eligible repositories ```task id: CUST-WP-0068-T02 status: todo priority: high state_hub_task_id: "a3564d51-2a9a-511a-8053-1a346a979bef" ``` Twenty-four workplans sit in repositories whose projection is empty and would bootstrap cleanly but for a specific blocker: | Repository | Cause | |---|---| | `rapp-openbao`, `policy-nexus`, `whitehat-security` | a task has no authoritative projection UUID; task ids must be minted first | | `ops-mason` | projection is no longer empty — partly registered already, so it needs the class-F path | | `ground-game` | genuine uncommitted work (`SCOPE.md`, `Modes.csv`); owner must land or stash it | | `zone-engine` | untracked `DECISIONS.md`; owner content, not ours to commit | Acceptance: each repository either bootstraps or has its blocker recorded as an owner action. ## Rule the six non-conforming record identifiers ```task id: CUST-WP-0068-T03 status: todo priority: medium state_hub_task_id: "e1eb31d5-cece-59ac-9ccc-8631fc025870" ``` `GLAS-0001`, `REIN-A-0001`–`0004`, `REIN-OW-0001` and `RAIL-BS-ADHOC-2026-07-27` are rejected by the canon registry. Renaming a work record changes its identity, so this is an owner decision per repository, not a rewrite. Note `rein-aharness` and `rein-openweights` are the `rein` concept that `agent-harness` was dropped in favour of (`CUST-WP-0067-T05`), so their records matter more than their age suggests. Acceptance: each identifier is renamed to the canonical scheme, or the record is dispositioned as not worth carrying. ## Bring clay-borg's workplans onto the convention ```task id: CUST-WP-0068-T04 status: todo priority: medium state_hub_task_id: "554bf550-3357-580b-a54c-78e4be777911" ``` Fifty workplans — the single largest block — use `kind: product` instead of `type: workplan` and `status: done`, which is not a workplan status. Nothing parses them as workplans, so no tooling sees them at all. This is a content pass with the repository owner, not a mechanical rewrite: the status mapping (`done` → `finished`?) and the missing `domain`, `repo` and `owner` fields are judgements about what those records mean. Acceptance: `clay-borg` workplans parse, and either bootstrap or take the class-F path. ## Recover the 176 with a non-empty projection ```task id: CUST-WP-0068-T05 status: todo priority: high state_hub_task_id: "0947f7fe-9432-567b-8fc0-4b5e3fadccf4" ``` Thirty-two repositories already have workplans on central, so bootstrap refuses them. The deterministic subset can go through C-06 unchanged. Largest: `railiance-platform` (18), `activity-core` (17), `net-kingdom` (13), `state-hub` (12), `flex-auth` (11), `the-custodian` (10), `ops-warden` (10). Run C-06 first and measure what remains before assuming any of it needs T06. Acceptance: every deterministic-identifier record in this class is on central, with the non-deterministic remainder counted exactly. ## Rule the identifier question for the remainder ```task id: CUST-WP-0068-T06 status: todo priority: medium state_hub_task_id: "fd1bf8e8-d03c-5f26-99c4-aef676b39644" ``` Whatever survives T05 has a non-deterministic identifier in a repository whose projection is not empty — the one case neither bootstrap nor C-06 handles. The options are to migrate those identifiers to derived UUIDv5 (`ADR-007` decision 2, and `state-hub` already has `apply_repository_identifier_migration` with a reversible plan), or to relax C-06 for records whose backing file is committed and pushed, or to disposition them as closed history. **This overlaps `RMGR-WP-0005` and must be agreed with Repo Manager rather than decided here.** Do not begin before T05 reports its count: the population may be small enough that disposition beats migration. Acceptance: a ruling recorded as a decision, with the chosen path executed or explicitly deferred. ## Repair the binding gap before counting any orphans ```task id: CUST-WP-0068-T07 status: done priority: high state_hub_task_id: "f44537f2-a738-5f39-b4c1-3711de06632e" ``` **The orphan count is not trustworthy and must be re-measured first.** `backing_filename` is written only by `PUT /workplans/index/bindings`, which `fix-consistency` calls for workplans that already carry a UUID. The documented registration flow mints the UUID *after* that sync — `fix-consistency` runs, C-06 skips because the workstation is not the registrar, then `registrar-reconcile` mints — so a newly registered workplan is left with no backing file recorded. Reproduced on `CUST-WP-0068` itself: it registered with `backing_filename: None`, and a second `fix-consistency` run populated it. Consequence: **278 of 800 workplans on central (35%) record no backing file**, including 8 `proposed`, 4 `ready` and 4 `active`. By the read model's own definition those are hub-only records, and `ADR-010` predicted exactly this as "broken links — a file exists but `backing_filename` was never recorded … likely the largest class". The 23 cache-only records classified as orphans were counted with the same unreliable field, so that number is a starting point for investigation, not a work list. Order of work: 1. **Done (2026-08-25).** The registrar now syncs bindings after minting, in the same pass (`repo-manager` `bdb981b`). Chosen over adding a step to the session protocol: a step everyone must remember is how this gap opened. Verified end to end — `ADHOC-2026-08-25` went from file to registered to bound with no second `fix-consistency` run. It binds on the `noop` path too. Without that, a record whose earlier bind failed would stay unbound forever, because every later run returns early. Binding never fails the registration: identifiers are already minted and committed, so a bind can be retried. Two further findings are captured in tests — the schema validates `status` against the enum *without* normalising, so a single legacy value `422`s the whole batch, and status is now sent only when already canonical. 2. **`bindings: synced 63/73` is explained.** The unbound ten are workplans whose identifiers central does not hold — the cache-only records this workplan exists to recover. The number will fall as `T02`–`T06` land, and is a progress indicator rather than a fault. A run against `the-custodian` after the fix reports `updated 64, sent 74, unbound 10`. 3. Re-sync bindings fleet-wide and re-measure the 278. Most are expected to be this same ordering gap and should resolve without any content change. 3b. **Done 2026-08-25.** The fleet-wide re-sync ran across 120 repositories: 1075 bindings sent, 1015 written, and workplans recording no backing file fell from **265 to 29**. `ADR-010` predicted this precisely — "broken links, where a file exists but `backing_filename` was never recorded … likely the largest class" — and it was: 89% of the apparent orphans were files that had existed all along and were never recorded. The 29 that remain are all closed — 23 `finished`, 6 `archived`, none open. `vergabe-teilnahme` holds 16 of them, then `railiance-cluster` (3), `railiance-platform` (3) and singles elsewhere. 4. **Done 2026-08-25.** `ADR-010`'s orphan rules applied to the 29 that remain. Every one is closed — 23 `finished`, 6 `archived`, none `proposed`, `ready`, `active` or `blocked`. None is an `ADR-001` violation: there are no open hub-first records needing a file written or an explicit drop. Under the rule "closed hub-first records may be retained as provenance where cheap", all 29 are retained. They record work that happened and cost nothing to keep. `vergabe-teilnahme` holds 16, then `railiance-cluster` (3), `railiance-platform` (3) and singles elsewhere. Four of them — `railiance-wp-0017`, `-0018`, `-0022`, `-0023` — will clear on their own when `T09`'s reset runs, since they are prefix-migration residue rather than genuine hub-first records. Acceptance: registration records a backing file in one pass; the fleet-wide count of file-less workplans is re-measured and explained; every remaining true orphan is repaired, given a file, or dropped in writing. ## Outcome so far (2026-08-25) **964 of 1075 work records recovered. Central went from 670 to 1018 workplans; the cache-only gap fell from 426 to 111.** `T06` was scoped as an identifier-migration decision. That framing was wrong, and the re-key it authorised turned out to be the smaller half. Two defects were doing most of the blocking, and both were invisible behind a generic `"State Hub reconciliation did not assign every requested identifier"` — the real message sat in a child process's stderr the whole time: 1. **50 repositories carried retired `gitea-remote:` URLs on central.** `fix-consistency` matches a checkout to its record *by* `remote_url`, so it could not find them and refused to register any of their work records. The forge migration updated every working copy and never the hub. `repo-onboard` now corrects a stale remote from the working copy's origin; 53 were fixed. 2. **The registrar was blind to "identified but absent".** It looked only for *missing* identifiers, so a record whose derived identifier was already in the file but absent from central short-circuited to `noop`. C-06 handles that case and marks it fixable; only the early return stood in the way. The re-key itself was then necessary but narrow: C-06 registers an absent record only when its identifier is the derived one, so 139 pre-`ADR-007` identifiers had to be re-keyed before they could be registered. **A mistake worth recording.** The first re-key script ran repo-wide and overwrote the identifiers of eight workplans on `kaizen-agentic` that were correctly registered, severing them from their live records. They were restored by matching canonical ids against central's slugs. The rule — *never re-key a record the hub already holds* — is now enforced in the script rather than remembered, and it correctly refused five duplicate-registration cases on the full run. `kaizen-agentic` deserves an independent look, since the repair was verified only by the person who caused the damage. **The 111 that remain**, all in eight repositories: | Count | Repository | Cause | |---|---|---| | 50 | `clay-borg` | files use `kind: product`, not `type: workplan` — nothing parses them (`T04`) | | 12 | `glas-harness`, `rein-aharness`, `rein-openweights` | identifiers the canon registry rejects (`T03`) | | 6 | `evidence-anchor`, `the-custodian`, `sand-boxer`, `state-hub` | mixed residue, needs case-by-case review | | — | `railiance-platform`, `railiance-apps` | the retired `RAILIANCE-WP` family prefix: files in two repositories whose records central attributes to a third. An identity decision, not a re-key | By status the remainder is overwhelmingly closed history — 96 `finished`, 10 `archived` — with only 2 `active`, 2 `blocked` and 1 `proposed` still live. ## Clear the stale hub rows that no longer derive from any file ```task id: CUST-WP-0068-T09 status: wait priority: medium ``` Migrating the active workplans off the retired `RAILIANCE-WP` prefix (2026-08-25, owner decision) changed their canonical identifiers, so the records registered under the old ones no longer have a backing file. **26 orphaned rows remain on central**, mostly `finished` and `archived`. This debt was accepted deliberately rather than discovered: the alternative was leaving one repository carrying two prefixes indefinitely, which is more confusing than a visible, bounded set of stale rows. C-07 detects them and is explicitly `fixable=False`, and there is no sanctioned delete path — reaching past the tooling to remove hub rows by hand is the direct central edit `ADR-010` decision 6 forbids. The correct instrument is `ADR-012` decision 7's per-repository reset from the forge, which rebuilds `railiance-platform` and `railiance-apps` from what Forgejo holds and drops whatever no longer derives. Status is `wait`, not `todo`: this is blocked on that reset existing, and doing it any other way would violate the ADR accepted the same day. Note the prefix collision this resolved was worse than a naming inconsistency. Three repositories used one number space for unrelated work — `RAILIANCE-WP-0012` was an openbao extraction, a cnpg backup, and a deploy-verify simultaneously — and `railiance-platform` carried two files both numbered `0016`. ### The same blocker covers every stale row, not just these 26 Widened 2026-08-25 by owner decision after an attempt to purge stale task rows established that **the hub has no hard-delete for work records at all**. `DELETE /tasks/{id}` is implemented as `cancel_task` — it sets status to `cancel` and keeps the row. `DELETE /workstreams/{id}` returns `410 Gone`. No `session.delete` exists for either. Removal by re-derivation is not one option among several; it is the only one the hub permits. That is a deliberate property worth respecting rather than routing around: a read model that cannot drop a record except by re-deriving cannot be quietly edited into disagreeing with its source. Full scope of rows that no longer derive from any file: | Count | What | Origin | |---|---|---| | 4 | `railiance-wp-*` workplan rows | the prefix migration above | | 305 | task rows across 44 workplans | superseded task sets, drafts registered before a workplan was retargeted | **Corrected 2026-08-25.** The migration was first reported as leaving 26 orphaned rows. It left four. The other 22 are still backed by the *archived* `RAILIANCE-WP` files, which this migration deliberately did not rename — so those rows continue to derive from a real file and are not orphans at all. The overstatement came from counting rows by slug prefix instead of by whether a file still backs them, and only surfaced when the fleet-wide binding re-sync bound 22 of them. The four genuinely unbacked are `railiance-wp-0017`, `-0018`, `-0022` and `-0023`, all `finished`. Of the 305 task rows, only **9 currently show open**; the rest are already `finished` or `cancel`, so the misleading surface is small. `kaizen-agentic` (8 workplans), `the-custodian` (5), `ops-warden` (4) and `state-hub` (4) hold the largest shares. **Nothing was mutated.** Eight cancel calls were issued against `STATE-WP-0080` before the endpoint's real behaviour was understood; all eight tasks were already `cancel`, so the calls were no-ops. No other workplan was touched. Acceptance: no `railiance-wp-*` workplan row and no orphaned task row remains on central, all removed through re-derivation rather than deletion. ## Disposition of duplicate task identities (2026-08-26) Establishing task identity (`STATE-WP-0083-T06`) exposed 35 duplicate identities on central. Owner decisions, and what each actually achieved: **`KONT-WP-0013` — renumbered.** `kontextual-engine` carried two different workplans under one identifier: blob-storage content streaming, registered 2026-05-07, and cmis-browser binding TCK compatibility, 2026-05-08. The later one became `KONT-WP-0018`. Ten colliding task identifiers resolved at source. **Archived `RAILIANCE-WP` — migrated.** The 2026-08-25 decision covered active workplans only; the archived copies kept historical identifiers and three repositories shared one number space. All 17 archived files are now on their repository's successor prefix, numbered on from each repository's active sequence so nothing is reused. Forge-side collisions fell from 45 to 16. **Unqualified `ADHOC-*` — accepted.** `CUST-WP-0066` already ruled that `{PREFIX}-WP-ADHOC-YYYY-MM-DD` is canonical and the unqualified historic form is grandfathered compatibility, explicitly declining to reinterpret repositories as namespaces. Five remain and are correct as they stand. **`OPS-WP-*` — newly visible, undecided.** Another two-repositories-one-prefix case, only apparent once task identity existed. Not yet dispositioned. ### What renaming files does not do Central still reports 35 duplicates, and the reason is worth recording rather than retrying: **archived workplans are never registered.** The registrar scans `workplans/*.md` at the top level only — closed archives are frozen under `ADR-007` and are not registration work — so renaming an archived file changes the source without touching the hub. The renames also re-derived each file's projection UUIDs, so those files no longer claim the rows they used to. The old rows are now unclaimed by any file: precisely the "no longer derives" condition that `ADR-012` decision 7's reset exists to retire. So the duplicates clear when `STATE-WP-0083-T03`'s reset runs against central, not before. The work above is what makes that reset produce the right answer. ## Converge and retire the cache database ```task id: CUST-WP-0068-T08 status: todo priority: high state_hub_task_id: "cbb8bf98-9b66-5827-992c-f22c3a6c1bb5" ``` Confirm no work record exists only in the cache, then stop and remove the local Postgres instance and its volume. Verify by direct comparison against the database rather than by assuming the preceding tasks covered everything — the predecessor workplan twice reported convergence that a direct count contradicted. Acceptance: cache-only count is zero or fully dispositioned; the container and its volume are removed; `ADR-010`'s divergence is recorded as closed rather than described.