the-custodian/workplans/CUST-WP-0068-cache-only-work-record-recovery.md

id	type	title	domain	repo	status	owner	created	updated	quality_dor	quality_dor_at	quality_dor_by	quality_dor_note	state_hub_workstream_id
CUST-WP-0068	workplan	Recover or disposition the cache-only work records so the retired hub can be discarded	infotech	the-custodian	ready	codex	2026-08-25	2026-08-25	DoR-Ok	2026-08-25	codex	Divergence measured exactly (297 workplans, 23 of them orphans), every remaining record classified with a named cause, per-repository blockers captured from real error messages, and the mechanism proven on live repositories before rollout.	c930b3ec-68b5-596e-ae72-d4ecaf7380e7

Recover or disposition the cache-only work records so the retired hub can be discarded

Goal

Finish what CUST-WP-0067-T10 started: get every work record that exists only in the retired local hub either onto central or into a written disposition, so the cache database stops being load-bearing and can be discarded. ADR-010 described this divergence in August 2026 and it has never been closed.

Context

The local hub instance was retired under CUST-WP-0067, and its repository records were fully recovered. Its work records were not. The database is still running solely because 297 workplans and their tasks exist nowhere else.

Measured 2026-08-25, after recovering 129 of the original 426:

Count	Class	Needs
176	non-empty projection on central	C-06 for the deterministic subset; the rest await an identifier decision
58	file does not parse as a workplan	content fix — 50 are clay-borg
24	bootstrap-eligible but blocked	per-repository unblocking
23	orphan, no backing file	written disposition
16	flagged as non-conforming ids	see the correction below

Correction carried forward. Ten of those 16 are ADHOC-YYYY-MM-DD identifiers, which are the documented ad-hoc convention — the classifier's pattern was wrong, not the files. Only six are genuinely non-conforming: GLAS-0001, REIN-A-0001–0004, REIN-OW-0001, plus RAIL-BS-ADHOC-2026-07-27 on a prefix the registry has retired. The ten adhocs should be re-tested and will most likely fall to the class-F treatment.

What already works. --bootstrap-empty-projection registers a repository's workplans while preserving their existing identifiers, verified on a live repository before rollout: workplan and all four tasks landed with original UUIDs and statuses, no file rewritten and no commit. All 403 cache-only workplans that record a backing path still have that file on disk, so this is recovery, not reconstruction.

What does not work, and why. Only 57 of 403 carry deterministic UUIDv5 identifiers. C-06 refuses non-derived identifiers as C-03 FAIL — stale reference, so the ordinary consistency path cannot re-derive most of them. Rewriting 345 identifiers fleet-wide was considered and rejected: it overlaps RMGR-WP-0005's ownership of identifier migration and would churn IDs across ~40 repositories to solve a problem bootstrap already solves for part of the set.

Repair the registrar's self-defeating precondition

id: CUST-WP-0068-T01
status: todo
priority: high
state_hub_task_id: "e8fbfa1a-3abf-5ec0-8abb-f63604dc23d5"

The registrar generates WORK-RECORDS.md, which dirties the worktree and then fails its own git precondition. Several runs registered their records successfully and then reported failure for the file they had just created; retries report git_precondition_failed instead, which points away from the real cause. Any operator running this at scale will misread it exactly as this workplan's predecessor did.

Also surface error messages, not only codes. bootstrap_source_invalid covers at least two unrelated causes — a repository with no parseable workplan, and a task missing its projection UUID — and reading only the code merged them into one wrong diagnosis.

Acceptance: a repository whose only change is the generated index completes in one run; failure reports name the specific record and cause.

Unblock the bootstrap-eligible repositories

id: CUST-WP-0068-T02
status: todo
priority: high
state_hub_task_id: "a3564d51-2a9a-511a-8053-1a346a979bef"

Twenty-four workplans sit in repositories whose projection is empty and would bootstrap cleanly but for a specific blocker:

Repository	Cause
rapp-openbao, policy-nexus, whitehat-security	a task has no authoritative projection UUID; task ids must be minted first
ops-mason	projection is no longer empty — partly registered already, so it needs the class-F path
ground-game	genuine uncommitted work (SCOPE.md, Modes.csv); owner must land or stash it
zone-engine	untracked DECISIONS.md; owner content, not ours to commit

Acceptance: each repository either bootstraps or has its blocker recorded as an owner action.

Rule the six non-conforming record identifiers

id: CUST-WP-0068-T03
status: todo
priority: medium
state_hub_task_id: "e1eb31d5-cece-59ac-9ccc-8631fc025870"

GLAS-0001, REIN-A-0001–0004, REIN-OW-0001 and RAIL-BS-ADHOC-2026-07-27 are rejected by the canon registry. Renaming a work record changes its identity, so this is an owner decision per repository, not a rewrite.

Note rein-aharness and rein-openweights are the rein concept that agent-harness was dropped in favour of (CUST-WP-0067-T05), so their records matter more than their age suggests.

Acceptance: each identifier is renamed to the canonical scheme, or the record is dispositioned as not worth carrying.

Bring clay-borg's workplans onto the convention

id: CUST-WP-0068-T04
status: todo
priority: medium
state_hub_task_id: "554bf550-3357-580b-a54c-78e4be777911"

Fifty workplans — the single largest block — use kind: product instead of type: workplan and status: done, which is not a workplan status. Nothing parses them as workplans, so no tooling sees them at all.

This is a content pass with the repository owner, not a mechanical rewrite: the status mapping (done → finished?) and the missing domain, repo and owner fields are judgements about what those records mean.

Acceptance: clay-borg workplans parse, and either bootstrap or take the class-F path.

Recover the 176 with a non-empty projection

id: CUST-WP-0068-T05
status: todo
priority: high
state_hub_task_id: "0947f7fe-9432-567b-8fc0-4b5e3fadccf4"

Thirty-two repositories already have workplans on central, so bootstrap refuses them. The deterministic subset can go through C-06 unchanged. Largest: railiance-platform (18), activity-core (17), net-kingdom (13), state-hub (12), flex-auth (11), the-custodian (10), ops-warden (10).

Run C-06 first and measure what remains before assuming any of it needs T06.

Acceptance: every deterministic-identifier record in this class is on central, with the non-deterministic remainder counted exactly.

Rule the identifier question for the remainder

id: CUST-WP-0068-T06
status: todo
priority: medium
state_hub_task_id: "fd1bf8e8-d03c-5f26-99c4-aef676b39644"

Whatever survives T05 has a non-deterministic identifier in a repository whose projection is not empty — the one case neither bootstrap nor C-06 handles.

The options are to migrate those identifiers to derived UUIDv5 (ADR-007 decision 2, and state-hub already has apply_repository_identifier_migration with a reversible plan), or to relax C-06 for records whose backing file is committed and pushed, or to disposition them as closed history. This overlaps RMGR-WP-0005 and must be agreed with Repo Manager rather than decided here.

Do not begin before T05 reports its count: the population may be small enough that disposition beats migration.

Acceptance: a ruling recorded as a decision, with the chosen path executed or explicitly deferred.

Repair the binding gap before counting any orphans

id: CUST-WP-0068-T07
status: todo
priority: high
state_hub_task_id: "f44537f2-a738-5f39-b4c1-3711de06632e"

The orphan count is not trustworthy and must be re-measured first.

backing_filename is written only by PUT /workplans/index/bindings, which fix-consistency calls for workplans that already carry a UUID. The documented registration flow mints the UUID after that sync — fix-consistency runs, C-06 skips because the workstation is not the registrar, then registrar-reconcile mints — so a newly registered workplan is left with no backing file recorded. Reproduced on CUST-WP-0068 itself: it registered with backing_filename: None, and a second fix-consistency run populated it.

Consequence: 278 of 800 workplans on central (35%) record no backing file, including 8 proposed, 4 ready and 4 active. By the read model's own definition those are hub-only records, and ADR-010 predicted exactly this as "broken links — a file exists but backing_filename was never recorded … likely the largest class". The 23 cache-only records classified as orphans were counted with the same unreliable field, so that number is a starting point for investigation, not a work list.

Order of work:

1. Done (2026-08-25). The registrar now syncs bindings after minting, in the same pass (repo-manager bdb981b). Chosen over adding a step to the session protocol: a step everyone must remember is how this gap opened. Verified end to end — ADHOC-2026-08-25 went from file to registered to bound with no second fix-consistency run.

   It binds on the noop path too. Without that, a record whose earlier bind failed would stay unbound forever, because every later run returns early. Binding never fails the registration: identifiers are already minted and committed, so a bind can be retried. Two further findings are captured in tests — the schema validates status against the enum without normalising, so a single legacy value 422s the whole batch, and status is now sent only when already canonical.

2. bindings: synced 63/73 is explained. The unbound ten are workplans whose identifiers central does not hold — the cache-only records this workplan exists to recover. The number will fall as T02–T06 land, and is a progress indicator rather than a fault. A run against the-custodian after the fix reports updated 64, sent 74, unbound 10.

3. Re-sync bindings fleet-wide and re-measure the 278. Most are expected to be this same ordering gap and should resolve without any content change.

4. Only then apply ADR-010's orphan rules to whatever genuinely has no file: open hub-first records are ADR-001 violations needing a file written or an explicit drop; closed ones may be retained as provenance where cheap.

Acceptance: registration records a backing file in one pass; the fleet-wide count of file-less workplans is re-measured and explained; every remaining true orphan is repaired, given a file, or dropped in writing.

Converge and retire the cache database

id: CUST-WP-0068-T08
status: todo
priority: high
state_hub_task_id: "cbb8bf98-9b66-5827-992c-f22c3a6c1bb5"

Confirm no work record exists only in the cache, then stop and remove the local Postgres instance and its volume.

Verify by direct comparison against the database rather than by assuming the preceding tasks covered everything — the predecessor workplan twice reported convergence that a direct count contradicted.

Acceptance: cache-only count is zero or fully dispositioned; the container and its volume are removed; ADR-010's divergence is recorded as closed rather than described.