the-custodian/workplans/CUST-WP-0068-cache-only-work-record-recovery.md

id	type	title	domain	repo	status	owner	created	updated	quality_dor	quality_dor_at	quality_dor_by	quality_dor_note	state_hub_workstream_id
CUST-WP-0068	workplan	Recover or disposition the cache-only work records so the retired hub can be discarded	infotech	the-custodian	ready	codex	2026-08-25	2026-08-25	DoR-Ok	2026-08-25	codex	Divergence measured exactly (297 workplans, 23 of them orphans), every remaining record classified with a named cause, per-repository blockers captured from real error messages, and the mechanism proven on live repositories before rollout.	c930b3ec-68b5-596e-ae72-d4ecaf7380e7

Recover or disposition the cache-only work records so the retired hub can be discarded

Goal

Finish what CUST-WP-0067-T10 started: get every work record that exists only in the retired local hub either onto central or into a written disposition, so the cache database stops being load-bearing and can be discarded. ADR-010 described this divergence in August 2026 and it has never been closed.

Context

The local hub instance was retired under CUST-WP-0067, and its repository records were fully recovered. Its work records were not. The database is still running solely because 297 workplans and their tasks exist nowhere else.

Measured 2026-08-25, after recovering 129 of the original 426:

Count	Class	Needs
176	non-empty projection on central	C-06 for the deterministic subset; the rest await an identifier decision
58	file does not parse as a workplan	content fix — 50 are clay-borg
24	bootstrap-eligible but blocked	per-repository unblocking
23	orphan, no backing file	written disposition
16	flagged as non-conforming ids	see the correction below

Correction carried forward. Ten of those 16 are ADHOC-YYYY-MM-DD identifiers, which are the documented ad-hoc convention — the classifier's pattern was wrong, not the files. Only six are genuinely non-conforming: GLAS-0001, REIN-A-0001–0004, REIN-OW-0001, plus RAIL-BS-ADHOC-2026-07-27 on a prefix the registry has retired. The ten adhocs should be re-tested and will most likely fall to the class-F treatment.

What already works. --bootstrap-empty-projection registers a repository's workplans while preserving their existing identifiers, verified on a live repository before rollout: workplan and all four tasks landed with original UUIDs and statuses, no file rewritten and no commit. All 403 cache-only workplans that record a backing path still have that file on disk, so this is recovery, not reconstruction.

What does not work, and why. Only 57 of 403 carry deterministic UUIDv5 identifiers. C-06 refuses non-derived identifiers as C-03 FAIL — stale reference, so the ordinary consistency path cannot re-derive most of them. Rewriting 345 identifiers fleet-wide was considered and rejected: it overlaps RMGR-WP-0005's ownership of identifier migration and would churn IDs across ~40 repositories to solve a problem bootstrap already solves for part of the set.

Repair the registrar's self-defeating precondition

id: CUST-WP-0068-T01
status: todo
priority: high
state_hub_task_id: "e8fbfa1a-3abf-5ec0-8abb-f63604dc23d5"

The registrar generates WORK-RECORDS.md, which dirties the worktree and then fails its own git precondition. Several runs registered their records successfully and then reported failure for the file they had just created; retries report git_precondition_failed instead, which points away from the real cause. Any operator running this at scale will misread it exactly as this workplan's predecessor did.

Also surface error messages, not only codes. bootstrap_source_invalid covers at least two unrelated causes — a repository with no parseable workplan, and a task missing its projection UUID — and reading only the code merged them into one wrong diagnosis.

Acceptance: a repository whose only change is the generated index completes in one run; failure reports name the specific record and cause.

Unblock the bootstrap-eligible repositories

id: CUST-WP-0068-T02
status: todo
priority: high
state_hub_task_id: "a3564d51-2a9a-511a-8053-1a346a979bef"

Twenty-four workplans sit in repositories whose projection is empty and would bootstrap cleanly but for a specific blocker:

Repository	Cause
rapp-openbao, policy-nexus, whitehat-security	a task has no authoritative projection UUID; task ids must be minted first
ops-mason	projection is no longer empty — partly registered already, so it needs the class-F path
ground-game	genuine uncommitted work (SCOPE.md, Modes.csv); owner must land or stash it
zone-engine	untracked DECISIONS.md; owner content, not ours to commit

Acceptance: each repository either bootstraps or has its blocker recorded as an owner action.

Rule the six non-conforming record identifiers

id: CUST-WP-0068-T03
status: todo
priority: medium
state_hub_task_id: "e1eb31d5-cece-59ac-9ccc-8631fc025870"

GLAS-0001, REIN-A-0001–0004, REIN-OW-0001 and RAIL-BS-ADHOC-2026-07-27 are rejected by the canon registry. Renaming a work record changes its identity, so this is an owner decision per repository, not a rewrite.

Note rein-aharness and rein-openweights are the rein concept that agent-harness was dropped in favour of (CUST-WP-0067-T05), so their records matter more than their age suggests.

Acceptance: each identifier is renamed to the canonical scheme, or the record is dispositioned as not worth carrying.

Bring clay-borg's workplans onto the convention

id: CUST-WP-0068-T04
status: todo
priority: medium
state_hub_task_id: "554bf550-3357-580b-a54c-78e4be777911"

Fifty workplans — the single largest block — use kind: product instead of type: workplan and status: done, which is not a workplan status. Nothing parses them as workplans, so no tooling sees them at all.

This is a content pass with the repository owner, not a mechanical rewrite: the status mapping (done → finished?) and the missing domain, repo and owner fields are judgements about what those records mean.

Acceptance: clay-borg workplans parse, and either bootstrap or take the class-F path.

Recover the 176 with a non-empty projection

id: CUST-WP-0068-T05
status: todo
priority: high
state_hub_task_id: "0947f7fe-9432-567b-8fc0-4b5e3fadccf4"

Thirty-two repositories already have workplans on central, so bootstrap refuses them. The deterministic subset can go through C-06 unchanged. Largest: railiance-platform (18), activity-core (17), net-kingdom (13), state-hub (12), flex-auth (11), the-custodian (10), ops-warden (10).

Run C-06 first and measure what remains before assuming any of it needs T06.

Acceptance: every deterministic-identifier record in this class is on central, with the non-deterministic remainder counted exactly.

Rule the identifier question for the remainder

id: CUST-WP-0068-T06
status: todo
priority: medium
state_hub_task_id: "fd1bf8e8-d03c-5f26-99c4-aef676b39644"

Whatever survives T05 has a non-deterministic identifier in a repository whose projection is not empty — the one case neither bootstrap nor C-06 handles.

The options are to migrate those identifiers to derived UUIDv5 (ADR-007 decision 2, and state-hub already has apply_repository_identifier_migration with a reversible plan), or to relax C-06 for records whose backing file is committed and pushed, or to disposition them as closed history. This overlaps RMGR-WP-0005 and must be agreed with Repo Manager rather than decided here.

Do not begin before T05 reports its count: the population may be small enough that disposition beats migration.

Acceptance: a ruling recorded as a decision, with the chosen path executed or explicitly deferred.

Repair the binding gap before counting any orphans

id: CUST-WP-0068-T07
status: todo
priority: high
state_hub_task_id: "f44537f2-a738-5f39-b4c1-3711de06632e"

The orphan count is not trustworthy and must be re-measured first.

backing_filename is written only by PUT /workplans/index/bindings, which fix-consistency calls for workplans that already carry a UUID. The documented registration flow mints the UUID after that sync — fix-consistency runs, C-06 skips because the workstation is not the registrar, then registrar-reconcile mints — so a newly registered workplan is left with no backing file recorded. Reproduced on CUST-WP-0068 itself: it registered with backing_filename: None, and a second fix-consistency run populated it.

Consequence: 278 of 800 workplans on central (35%) record no backing file, including 8 proposed, 4 ready and 4 active. By the read model's own definition those are hub-only records, and ADR-010 predicted exactly this as "broken links — a file exists but backing_filename was never recorded … likely the largest class". The 23 cache-only records classified as orphans were counted with the same unreliable field, so that number is a starting point for investigation, not a work list.

Order of work:

1. Done (2026-08-25). The registrar now syncs bindings after minting, in the same pass (repo-manager bdb981b). Chosen over adding a step to the session protocol: a step everyone must remember is how this gap opened. Verified end to end — ADHOC-2026-08-25 went from file to registered to bound with no second fix-consistency run.

   It binds on the noop path too. Without that, a record whose earlier bind failed would stay unbound forever, because every later run returns early. Binding never fails the registration: identifiers are already minted and committed, so a bind can be retried. Two further findings are captured in tests — the schema validates status against the enum without normalising, so a single legacy value 422s the whole batch, and status is now sent only when already canonical.

2. bindings: synced 63/73 is explained. The unbound ten are workplans whose identifiers central does not hold — the cache-only records this workplan exists to recover. The number will fall as T02–T06 land, and is a progress indicator rather than a fault. A run against the-custodian after the fix reports updated 64, sent 74, unbound 10.

3. Re-sync bindings fleet-wide and re-measure the 278. Most are expected to be this same ordering gap and should resolve without any content change.

4. Only then apply ADR-010's orphan rules to whatever genuinely has no file: open hub-first records are ADR-001 violations needing a file written or an explicit drop; closed ones may be retained as provenance where cheap.

Acceptance: registration records a backing file in one pass; the fleet-wide count of file-less workplans is re-measured and explained; every remaining true orphan is repaired, given a file, or dropped in writing.

Outcome so far (2026-08-25)

964 of 1075 work records recovered. Central went from 670 to 1018 workplans; the cache-only gap fell from 426 to 111.

T06 was scoped as an identifier-migration decision. That framing was wrong, and the re-key it authorised turned out to be the smaller half. Two defects were doing most of the blocking, and both were invisible behind a generic "State Hub reconciliation did not assign every requested identifier" — the real message sat in a child process's stderr the whole time:

1. 50 repositories carried retired gitea-remote: URLs on central. fix-consistency matches a checkout to its record by remote_url, so it could not find them and refused to register any of their work records. The forge migration updated every working copy and never the hub. repo-onboard now corrects a stale remote from the working copy's origin; 53 were fixed.

2. The registrar was blind to "identified but absent". It looked only for missing identifiers, so a record whose derived identifier was already in the file but absent from central short-circuited to noop. C-06 handles that case and marks it fixable; only the early return stood in the way.

The re-key itself was then necessary but narrow: C-06 registers an absent record only when its identifier is the derived one, so 139 pre-ADR-007 identifiers had to be re-keyed before they could be registered.

A mistake worth recording. The first re-key script ran repo-wide and overwrote the identifiers of eight workplans on kaizen-agentic that were correctly registered, severing them from their live records. They were restored by matching canonical ids against central's slugs. The rule — never re-key a record the hub already holds — is now enforced in the script rather than remembered, and it correctly refused five duplicate-registration cases on the full run. kaizen-agentic deserves an independent look, since the repair was verified only by the person who caused the damage.

The 111 that remain, all in eight repositories:

Count	Repository	Cause
50	clay-borg	files use kind: product, not type: workplan — nothing parses them (T04)
12	glas-harness, rein-aharness, rein-openweights	identifiers the canon registry rejects (T03)
6	evidence-anchor, the-custodian, sand-boxer, state-hub	mixed residue, needs case-by-case review
—	railiance-platform, railiance-apps	the retired RAILIANCE-WP family prefix: files in two repositories whose records central attributes to a third. An identity decision, not a re-key

By status the remainder is overwhelmingly closed history — 96 finished, 10 archived — with only 2 active, 2 blocked and 1 proposed still live.

Clear the orphaned RAILIANCE-WP rows

id: CUST-WP-0068-T09
status: wait
priority: medium

Migrating the active workplans off the retired RAILIANCE-WP prefix (2026-08-25, owner decision) changed their canonical identifiers, so the records registered under the old ones no longer have a backing file. 26 orphaned rows remain on central, mostly finished and archived.

This debt was accepted deliberately rather than discovered: the alternative was leaving one repository carrying two prefixes indefinitely, which is more confusing than a visible, bounded set of stale rows.

C-07 detects them and is explicitly fixable=False, and there is no sanctioned delete path — reaching past the tooling to remove hub rows by hand is the direct central edit ADR-010 decision 6 forbids. The correct instrument is ADR-012 decision 7's per-repository reset from the forge, which rebuilds railiance-platform and railiance-apps from what Forgejo holds and drops whatever no longer derives.

Status is wait, not todo: this is blocked on that reset existing, and doing it any other way would violate the ADR accepted the same day.

Note the prefix collision this resolved was worse than a naming inconsistency. Three repositories used one number space for unrelated work — RAILIANCE-WP-0012 was an openbao extraction, a cnpg backup, and a deploy-verify simultaneously — and railiance-platform carried two files both numbered 0016.

Acceptance: no railiance-wp-* record remains on central; the 26 are gone through re-derivation rather than deletion.

Converge and retire the cache database

id: CUST-WP-0068-T08
status: todo
priority: high
state_hub_task_id: "cbb8bf98-9b66-5827-992c-f22c3a6c1bb5"

Confirm no work record exists only in the cache, then stop and remove the local Postgres instance and its volume.

Verify by direct comparison against the database rather than by assuming the preceding tasks covered everything — the predecessor workplan twice reported convergence that a direct count contradicted.

Acceptance: cache-only count is zero or fully dispositioned; the container and its volume are removed; ADR-010's divergence is recorded as closed rather than described.