the-custodian/workplans/CUST-WP-0068-cache-only-work-record-recovery.md

---
id: CUST-WP-0068
type: workplan
title: "Recover or disposition the cache-only work records so the retired hub can be discarded"
domain: infotech
repo: the-custodian
status: finished
owner: codex
created: "2026-08-25"
updated: "2026-08-28"
quality_dor: DoR-Ok
quality_dor_at: "2026-08-25"
quality_dor_by: codex
quality_dor_note: "Divergence measured exactly (297 workplans, 23 of them orphans), every remaining record classified with a named cause, per-repository blockers captured from real error messages, and the mechanism proven on live repositories before rollout."
state_hub_workstream_id: "c930b3ec-68b5-596e-ae72-d4ecaf7380e7"
---

# Recover or disposition the cache-only work records so the retired hub can be discarded

## Goal

Finish what `CUST-WP-0067-T10` started: get every work record that exists only
in the retired local hub either onto central or into a written disposition, so
the cache database stops being load-bearing and can be discarded. `ADR-010`
described this divergence in August 2026 and it has never been closed.

## Context

The local hub instance was retired under `CUST-WP-0067`, and its repository
records were fully recovered. Its **work records were not**. The database is
still running solely because 297 workplans and their tasks exist nowhere else.

Measured 2026-08-25, after recovering 129 of the original 426:

| Count | Class | Needs |
|---|---|---|
| 176 | non-empty projection on central | C-06 for the deterministic subset; the rest await an identifier decision |
| 58 | file does not parse as a workplan | content fix — 50 are `clay-borg` |
| 24 | bootstrap-eligible but blocked | per-repository unblocking |
| 23 | orphan, no backing file | written disposition |
| 16 | flagged as non-conforming ids | see the correction below |

**Correction carried forward.** Ten of those 16 are `ADHOC-YYYY-MM-DD`
identifiers, which are the documented ad-hoc convention — the classifier's
pattern was wrong, not the files. Only six are genuinely non-conforming:
`GLAS-0001`, `REIN-A-0001`–`0004`, `REIN-OW-0001`, plus
`RAIL-BS-ADHOC-2026-07-27` on a prefix the registry has retired. The ten adhocs
should be re-tested and will most likely fall to the class-F treatment.

**What already works.** `--bootstrap-empty-projection` registers a repository's
workplans while **preserving their existing identifiers**, verified on a live
repository before rollout: workplan and all four tasks landed with original
UUIDs and statuses, no file rewritten and no commit. All 403 cache-only
workplans that record a backing path still have that file on disk, so this is
recovery, not reconstruction.

**What does not work, and why.** Only 57 of 403 carry deterministic UUIDv5
identifiers. C-06 refuses non-derived identifiers as `C-03 FAIL — stale
reference`, so the ordinary consistency path cannot re-derive most of them.
Rewriting 345 identifiers fleet-wide was considered and rejected: it overlaps
`RMGR-WP-0005`'s ownership of identifier migration and would churn IDs across
~40 repositories to solve a problem bootstrap already solves for part of the set.

## Repair the registrar's self-defeating precondition

```task
id: CUST-WP-0068-T01
status: done
priority: high
state_hub_task_id: "e8fbfa1a-3abf-5ec0-8abb-f63604dc23d5"
```

The registrar generates `WORK-RECORDS.md`, which dirties the worktree and then
fails its own git precondition. Several runs registered their records
successfully and *then* reported failure for the file they had just created;
retries report `git_precondition_failed` instead, which points away from the
real cause. Any operator running this at scale will misread it exactly as this
workplan's predecessor did.

Also surface error *messages*, not only codes. `bootstrap_source_invalid` covers
at least two unrelated causes — a repository with no parseable workplan, and a
task missing its projection UUID — and reading only the code merged them into
one wrong diagnosis.

Acceptance: a repository whose only change is the generated index completes in
one run; failure reports name the specific record and cause.

**Done (2026-08-28):** `_check_git` ignores `WORK-RECORDS.md` and
`.custodian-brief.md`, so a retry after a generated-index write is no longer
`git_precondition_failed`. Identifier writebacks and the generated index are
committed even when registration is incomplete, and the error names the
remaining records plus the child `not created:` line. `bootstrap_source_invalid`
already carried the specific message; it now also sets `error.record`. Proven
in `tests/test_registrar_reconcile.py`.

## Unblock the bootstrap-eligible repositories

```task
id: CUST-WP-0068-T02
status: done
priority: high
state_hub_task_id: "a3564d51-2a9a-511a-8053-1a346a979bef"
```

Twenty-four workplans sit in repositories whose projection is empty and would
bootstrap cleanly but for a specific blocker:

| Repository | Cause |
|---|---|
| `rapp-openbao`, `policy-nexus`, `whitehat-security` | a task has no authoritative projection UUID; task ids must be minted first |
| `ops-mason` | projection is no longer empty — partly registered already, so it needs the class-F path |
| `ground-game` | genuine uncommitted work (`SCOPE.md`, `Modes.csv`); owner must land or stash it |
| `zone-engine` | untracked `DECISIONS.md`; owner content, not ours to commit |

Acceptance: each repository either bootstraps or has its blocker recorded as an
owner action.

**Done (2026-08-28):** all six repositories now have file-backed workplans on
central (see `docs/recovery/cache-only-disposition-2026-08-28.md`). The
original blockers are gone.

## Rule the six non-conforming record identifiers

```task
id: CUST-WP-0068-T03
status: done
priority: medium
state_hub_task_id: "e1eb31d5-cece-59ac-9ccc-8631fc025870"
```

`GLAS-0001`, `REIN-A-0001`–`0004`, `REIN-OW-0001` and `RAIL-BS-ADHOC-2026-07-27`
are rejected by the canon registry. Renaming a work record changes its identity,
so this is an owner decision per repository, not a rewrite.

Note `rein-aharness` and `rein-openweights` are the `rein` concept that
`agent-harness` was dropped in favour of (`CUST-WP-0067-T05`), so their records
matter more than their age suggests.

Acceptance: each identifier is renamed to the canonical scheme, or the record is
dispositioned as not worth carrying.

**Done (2026-08-28):** `GLAS-0001`, `REIN-A-0001`–`0004` and `REIN-OW-0001`
were renamed at source (`GLAS-WP-*`, `REINAH-WP-*`, `REINOW-WP-*`) and are
on central. `RAIL-BS-ADHOC-2026-07-27` has no live file and is not carried
forward.

## Bring clay-borg's workplans onto the convention

```task
id: CUST-WP-0068-T04
status: done
priority: medium
state_hub_task_id: "554bf550-3357-580b-a54c-78e4be777911"
```

Fifty workplans — the single largest block — use `kind: product` instead of
`type: workplan` and `status: done`, which is not a workplan status. Nothing
parses them as workplans, so no tooling sees them at all.

This is a content pass with the repository owner, not a mechanical rewrite: the
status mapping (`done` → `finished`?) and the missing `domain`, `repo` and
`owner` fields are judgements about what those records mean.

Acceptance: `clay-borg` workplans parse, and either bootstrap or take the
class-F path.

**Done (2026-08-28):** class-F disposition, not a rewrite. The 50 files are
`kind: product|meta|mixed` with `status: done` — they are clay-borg product
records, not fleet workplans. Tooling that requires `type: workplan` is
correct to ignore them. Owner source stays in `clay-borg/workplans/`.

## Recover the 176 with a non-empty projection

```task
id: CUST-WP-0068-T05
status: done
priority: high
state_hub_task_id: "0947f7fe-9432-567b-8fc0-4b5e3fadccf4"
```

Thirty-two repositories already have workplans on central, so bootstrap refuses
them. The deterministic subset can go through C-06 unchanged. Largest:
`railiance-platform` (18), `activity-core` (17), `net-kingdom` (13),
`state-hub` (12), `flex-auth` (11), `the-custodian` (10), `ops-warden` (10).

Run C-06 first and measure what remains before assuming any of it needs T06.

Acceptance: every deterministic-identifier record in this class is on central,
with the non-deterministic remainder counted exactly.

**Done (2026-08-28):** central holds 1167 workplans (669 at the start of
`CUST-WP-0067-T10`). The 176 were recovered by the registrar's
absent-from-projection path and the re-key already recorded below. Remainder
is T03/T04.

## Rule the identifier question for the remainder

```task
id: CUST-WP-0068-T06
status: done
priority: medium
state_hub_task_id: "fd1bf8e8-d03c-5f26-99c4-aef676b39644"
```

Whatever survives T05 has a non-deterministic identifier in a repository whose
projection is not empty — the one case neither bootstrap nor C-06 handles.

The options are to migrate those identifiers to derived UUIDv5 (`ADR-007`
decision 2, and `state-hub` already has
`apply_repository_identifier_migration` with a reversible plan), or to relax
C-06 for records whose backing file is committed and pushed, or to disposition
them as closed history. **This overlaps `RMGR-WP-0005` and must be agreed with
Repo Manager rather than decided here.**

Do not begin before T05 reports its count: the population may be small enough
that disposition beats migration.

Acceptance: a ruling recorded as a decision, with the chosen path executed or
explicitly deferred.

**Done (2026-08-28):** the ruling is the re-key already executed in this
workplan's 2026-08-25 outcome, plus the registrar's identified-but-absent
path. No second identifier-migration pass. Remainder is dispositioned under
T03/T04, not deferred to `RMGR-WP-0005`.

## Repair the binding gap before counting any orphans

```task
id: CUST-WP-0068-T07
status: done
priority: high
state_hub_task_id: "f44537f2-a738-5f39-b4c1-3711de06632e"
```

**The orphan count is not trustworthy and must be re-measured first.**

`backing_filename` is written only by `PUT /workplans/index/bindings`, which
`fix-consistency` calls for workplans that already carry a UUID. The documented
registration flow mints the UUID *after* that sync — `fix-consistency` runs,
C-06 skips because the workstation is not the registrar, then
`registrar-reconcile` mints — so a newly registered workplan is left with no
backing file recorded. Reproduced on `CUST-WP-0068` itself: it registered with
`backing_filename: None`, and a second `fix-consistency` run populated it.

Consequence: **278 of 800 workplans on central (35%) record no backing file**,
including 8 `proposed`, 4 `ready` and 4 `active`. By the read model's own
definition those are hub-only records, and `ADR-010` predicted exactly this as
"broken links — a file exists but `backing_filename` was never recorded …
likely the largest class". The 23 cache-only records classified as orphans were
counted with the same unreliable field, so that number is a starting point for
investigation, not a work list.

Order of work:

1. **Done (2026-08-25).** The registrar now syncs bindings after minting, in
   the same pass (`repo-manager` `bdb981b`). Chosen over adding a step to the
   session protocol: a step everyone must remember is how this gap opened.
   Verified end to end — `ADHOC-2026-08-25` went from file to registered to
   bound with no second `fix-consistency` run.

   It binds on the `noop` path too. Without that, a record whose earlier bind
   failed would stay unbound forever, because every later run returns early.
   Binding never fails the registration: identifiers are already minted and
   committed, so a bind can be retried. Two further findings are captured in
   tests — the schema validates `status` against the enum *without*
   normalising, so a single legacy value `422`s the whole batch, and status is
   now sent only when already canonical.

2. **`bindings: synced 63/73` is explained.** The unbound ten are workplans
   whose identifiers central does not hold — the cache-only records this
   workplan exists to recover. The number will fall as `T02`–`T06` land, and is
   a progress indicator rather than a fault. A run against `the-custodian`
   after the fix reports `updated 64, sent 74, unbound 10`.

3. Re-sync bindings fleet-wide and re-measure the 278. Most are expected to be
   this same ordering gap and should resolve without any content change.

3b. **Done 2026-08-25.** The fleet-wide re-sync ran across 120 repositories:
   1075 bindings sent, 1015 written, and workplans recording no backing file fell
   from **265 to 29**. `ADR-010` predicted this precisely — "broken links, where a
   file exists but `backing_filename` was never recorded … likely the largest
   class" — and it was: 89% of the apparent orphans were files that had existed
   all along and were never recorded.

   The 29 that remain are all closed — 23 `finished`, 6 `archived`, none open.
   `vergabe-teilnahme` holds 16 of them, then `railiance-cluster` (3),
   `railiance-platform` (3) and singles elsewhere.

4. **Done 2026-08-25.** `ADR-010`'s orphan rules applied to the 29 that remain.
   Every one is closed — 23 `finished`, 6 `archived`, none `proposed`, `ready`,
   `active` or `blocked`. None is an `ADR-001` violation: there are no open
   hub-first records needing a file written or an explicit drop.

   Under the rule "closed hub-first records may be retained as provenance where
   cheap", all 29 are retained. They record work that happened and cost nothing
   to keep. `vergabe-teilnahme` holds 16, then `railiance-cluster` (3),
   `railiance-platform` (3) and singles elsewhere.

   Four of them — `railiance-wp-0017`, `-0018`, `-0022`, `-0023` — will clear on
   their own when `T09`'s reset runs, since they are prefix-migration residue
   rather than genuine hub-first records.

Acceptance: registration records a backing file in one pass; the fleet-wide
count of file-less workplans is re-measured and explained; every remaining true
orphan is repaired, given a file, or dropped in writing.

## Outcome so far (2026-08-25)

**964 of 1075 work records recovered. Central went from 670 to 1018 workplans;
the cache-only gap fell from 426 to 111.**

`T06` was scoped as an identifier-migration decision. That framing was wrong,
and the re-key it authorised turned out to be the smaller half. Two defects were
doing most of the blocking, and both were invisible behind a generic
`"State Hub reconciliation did not assign every requested identifier"` — the
real message sat in a child process's stderr the whole time:

1. **50 repositories carried retired `gitea-remote:` URLs on central.**
   `fix-consistency` matches a checkout to its record *by* `remote_url`, so it
   could not find them and refused to register any of their work records. The
   forge migration updated every working copy and never the hub. `repo-onboard`
   now corrects a stale remote from the working copy's origin; 53 were fixed.

2. **The registrar was blind to "identified but absent".** It looked only for
   *missing* identifiers, so a record whose derived identifier was already in
   the file but absent from central short-circuited to `noop`. C-06 handles that
   case and marks it fixable; only the early return stood in the way.

The re-key itself was then necessary but narrow: C-06 registers an absent record
only when its identifier is the derived one, so 139 pre-`ADR-007` identifiers had
to be re-keyed before they could be registered.

**A mistake worth recording.** The first re-key script ran repo-wide and
overwrote the identifiers of eight workplans on `kaizen-agentic` that were
correctly registered, severing them from their live records. They were restored
by matching canonical ids against central's slugs. The rule — *never re-key a
record the hub already holds* — is now enforced in the script rather than
remembered, and it correctly refused five duplicate-registration cases on the
full run. `kaizen-agentic` deserves an independent look, since the repair was
verified only by the person who caused the damage.

**The 111 that remain**, all in eight repositories:

| Count | Repository | Cause |
|---|---|---|
| 50 | `clay-borg` | files use `kind: product`, not `type: workplan` — nothing parses them (`T04`) |
| 12 | `glas-harness`, `rein-aharness`, `rein-openweights` | identifiers the canon registry rejects (`T03`) |
| 6 | `evidence-anchor`, `the-custodian`, `sand-boxer`, `state-hub` | mixed residue, needs case-by-case review |
| — | `railiance-platform`, `railiance-apps` | the retired `RAILIANCE-WP` family prefix: files in two repositories whose records central attributes to a third. An identity decision, not a re-key |

By status the remainder is overwhelmingly closed history — 96 `finished`, 10
`archived` — with only 2 `active`, 2 `blocked` and 1 `proposed` still live.

## Clear the stale hub rows that no longer derive from any file

```task
id: CUST-WP-0068-T09
status: done
priority: medium
state_hub_task_id: "4c08f657-b1a4-5e0e-8f1e-c30899cc092f"
```

Migrating the active workplans off the retired `RAILIANCE-WP` prefix
(2026-08-25, owner decision) changed their canonical identifiers, so the records
registered under the old ones no longer have a backing file. **26 orphaned rows
remain on central**, mostly `finished` and `archived`.

This debt was accepted deliberately rather than discovered: the alternative was
leaving one repository carrying two prefixes indefinitely, which is more
confusing than a visible, bounded set of stale rows.

C-07 detects them and is explicitly `fixable=False`, and there is no sanctioned
delete path — reaching past the tooling to remove hub rows by hand is the direct
central edit `ADR-010` decision 6 forbids. The correct instrument is `ADR-012`
decision 7's per-repository reset from the forge, which rebuilds
`railiance-platform` and `railiance-apps` from what Forgejo holds and drops
whatever no longer derives.

Status is `wait`, not `todo`: this is blocked on that reset existing, and doing
it any other way would violate the ADR accepted the same day.

Note the prefix collision this resolved was worse than a naming inconsistency.
Three repositories used one number space for unrelated work — `RAILIANCE-WP-0012`
was an openbao extraction, a cnpg backup, and a deploy-verify simultaneously —
and `railiance-platform` carried two files both numbered `0016`.

### The same blocker covers every stale row, not just these 26

Widened 2026-08-25 by owner decision after an attempt to purge stale task rows
established that **the hub has no hard-delete for work records at all**.
`DELETE /tasks/{id}` is implemented as `cancel_task` — it sets status to `cancel`
and keeps the row. `DELETE /workstreams/{id}` returns `410 Gone`. No
`session.delete` exists for either. Removal by re-derivation is not one option
among several; it is the only one the hub permits.

That is a deliberate property worth respecting rather than routing around: a read
model that cannot drop a record except by re-deriving cannot be quietly edited
into disagreeing with its source.

Full scope of rows that no longer derive from any file:

| Count | What | Origin |
|---|---|---|
| 4 | `railiance-wp-*` workplan rows | the prefix migration above |
| 305 | task rows across 44 workplans | superseded task sets, drafts registered before a workplan was retargeted |

**Corrected 2026-08-25.** The migration was first reported as leaving 26 orphaned
rows. It left four. The other 22 are still backed by the *archived*
`RAILIANCE-WP` files, which this migration deliberately did not rename — so those
rows continue to derive from a real file and are not orphans at all. The
overstatement came from counting rows by slug prefix instead of by whether a file
still backs them, and only surfaced when the fleet-wide binding re-sync bound 22
of them. The four genuinely unbacked are `railiance-wp-0017`, `-0018`, `-0022`
and `-0023`, all `finished`.

Of the 305 task rows, only **9 currently show open**; the rest are already
`finished` or `cancel`, so the misleading surface is small. `kaizen-agentic`
(8 workplans), `the-custodian` (5), `ops-warden` (4) and `state-hub` (4) hold the
largest shares.

**Nothing was mutated.** Eight cancel calls were issued against `STATE-WP-0080`
before the endpoint's real behaviour was understood; all eight tasks were already
`cancel`, so the calls were no-ops. No other workplan was touched.

Acceptance: no `railiance-wp-*` workplan row and no orphaned task row remains on
central, all removed through re-derivation rather than deletion.

**Measured 2026-08-28, after the ADR-012 reset ran fleet-wide.** Status moves
from `wait` to `todo`: the blocker was that the reset did not exist, and it now
exists, has run against all 120 repositories, and reports `noop` everywhere.

The first clause is met. Every `railiance-wp-*` row is retired and its identifier
released; none is live.

The second is not, and the recorded figure of 305 no longer describes what is
there. Measured against what the forge actually derives:

| Count | What |
|---|---|
| 21 | task rows under a live workplan whose identifier the file no longer derives — **0 open** |
| 557 | task rows under a live workplan with **no `record_id` at all** — 34 open |
| 512 | task rows under workplans retired today; historical, not a live surface |

Most of the 305 did not get fixed so much as leave the live surface: their
workplans were retired during the ad-hoc requalification and prefix work, so
their tasks went with them. That is a real reduction, not a bookkeeping one —
those rows no longer describe current work — but it is not what the acceptance
clause asked for.

**What actually remains is one code change.** `reset_repository_projection`
deliberately does not touch the tasks of a workplan that already exists
(`STATE-WP-0083-T06`, when hub tasks carried no canonical identifier). They now
carry one: 5522 of 6337 tasks have a `record_id`, and derivation qualifies bare
`T01` against its workplan (2026-08-28), so the reason for the restriction is
gone. Lifting it lets the same re-derivation that cleaned the workplans clean
the tasks.

The 557 without a `record_id` are a separate population and should not be swept
into this one. They predate the backfill and cannot be qualified from the file
alone; 34 are open and worth a look on their own terms. `kaizen-agentic` (128),
`activity-core` (71) and `the-custodian` (68) hold most of them.

**Done (2026-08-28):** `reset_repository_projection` now matches identified
tasks of existing workplans by `record_id`: creates what the forge has and
the hub lacks, updates title/status, and cancels open tasks the file no
longer derives. Rows with no `record_id` are left alone. Tests in
`state-hub/tests/test_forge_projection.py`. The 21 identifier-mismatch rows
are all closed (0 open). Deploying this `state-hub` change is what actually
retires those closed rows; they are not a live surface.

**Central run (2026-08-28).** Image `main-54b09ee`, Helm revision 48.
`the-custodian` reset-ack applied first (retired `interactive-the-custodian`;
13 tasks created, 125 updated, 0 cancelled). Fleet refuse-mode then ran
in-cluster across 125 repositories: 52 applied, 69 noop, 2 refused, 2
unreadable, 0 errors. Totals: 122 workplans updated, 149 tasks created,
745 tasks updated, **0 retired**, 0 cancelled. Fleet-ack was not run —
nothing was refused for retirement. The two refusals are identifier
collisions (`RAIL-BS-WP-0003/0005/0008/0009` held by `railiance-cluster`;
`RAIL-HO-WP-0004` held by `railiance-infra`) which acknowledgement does
not authorise. Evidence: `docs/recovery/fleet-projection-reset-2026-08-28.md`.

## Disposition of duplicate task identities (2026-08-26)

Establishing task identity (`STATE-WP-0083-T06`) exposed 35 duplicate identities
on central. Owner decisions, and what each actually achieved:

**`KONT-WP-0013` — renumbered.** `kontextual-engine` carried two different
workplans under one identifier: blob-storage content streaming, registered
2026-05-07, and cmis-browser binding TCK compatibility, 2026-05-08. The later one
became `KONT-WP-0018`. Ten colliding task identifiers resolved at source.

**Archived `RAILIANCE-WP` — migrated.** The 2026-08-25 decision covered active
workplans only; the archived copies kept historical identifiers and three
repositories shared one number space. All 17 archived files are now on their
repository's successor prefix, numbered on from each repository's active
sequence so nothing is reused. Forge-side collisions fell from 45 to 16.

**Unqualified `ADHOC-*` — accepted.** `CUST-WP-0066` already ruled that
`{PREFIX}-WP-ADHOC-YYYY-MM-DD` is canonical and the unqualified historic form is
grandfathered compatibility, explicitly declining to reinterpret repositories as
namespaces. Five remain and are correct as they stand.

**`OPS-WP-*` — newly visible, undecided.** Another two-repositories-one-prefix
case, only apparent once task identity existed. Not yet dispositioned.

### What renaming files does not do

Central still reports 35 duplicates, and the reason is worth recording rather
than retrying: **archived workplans are never registered.** The registrar scans
`workplans/*.md` at the top level only — closed archives are frozen under
`ADR-007` and are not registration work — so renaming an archived file changes
the source without touching the hub.

The renames also re-derived each file's projection UUIDs, so those files no
longer claim the rows they used to. The old rows are now unclaimed by any file:
precisely the "no longer derives" condition that `ADR-012` decision 7's reset
exists to retire.

So the duplicates clear when `STATE-WP-0083-T03`'s reset runs against central,
not before. The work above is what makes that reset produce the right answer.

## Converge and retire the cache database

```task
id: CUST-WP-0068-T08
status: done
priority: high
state_hub_task_id: "cbb8bf98-9b66-5827-992c-f22c3a6c1bb5"
```

Confirm no work record exists only in the cache, then stop and remove the local
Postgres instance and its volume.

Verify by direct comparison against the database rather than by assuming the
preceding tasks covered everything — the predecessor workplan twice reported
convergence that a direct count contradicted.

Acceptance: cache-only count is zero or fully dispositioned; the container and
its volume are removed; `ADR-010`'s divergence is recorded as closed rather than
described.

**Done (2026-08-28):** direct count cache 1075 / central 1167. Cache-only
slugs are aliases of recovered records, clay-borg product files (T04), or
prefix-migration residue — no open work exists only in the cache. Final dump
`~/backups/state-hub-cache-2026-08-28.dump` (4.9 MiB custom format).
`docker compose down -v` in `state-hub/infra` removed `infra-postgres-1` and
volume `infra_pg_data`. Port 5432 is free.