the-custodian/workplans/CUST-WP-0068-cache-only-work-record-recovery.md

id	type	title	domain	repo	status	owner	created	updated	quality_dor	quality_dor_at	quality_dor_by	quality_dor_note	state_hub_workstream_id
CUST-WP-0068	workplan	Recover or disposition the cache-only work records so the retired hub can be discarded	infotech	the-custodian	finished	codex	2026-08-25	2026-08-28	DoR-Ok	2026-08-25	codex	Divergence measured exactly (297 workplans, 23 of them orphans), every remaining record classified with a named cause, per-repository blockers captured from real error messages, and the mechanism proven on live repositories before rollout.	c930b3ec-68b5-596e-ae72-d4ecaf7380e7

Recover or disposition the cache-only work records so the retired hub can be discarded

Goal

Finish what CUST-WP-0067-T10 started: get every work record that exists only in the retired local hub either onto central or into a written disposition, so the cache database stops being load-bearing and can be discarded. ADR-010 described this divergence in August 2026 and it has never been closed.

Context

The local hub instance was retired under CUST-WP-0067, and its repository records were fully recovered. Its work records were not. The database is still running solely because 297 workplans and their tasks exist nowhere else.

Measured 2026-08-25, after recovering 129 of the original 426:

Count	Class	Needs
176	non-empty projection on central	C-06 for the deterministic subset; the rest await an identifier decision
58	file does not parse as a workplan	content fix — 50 are clay-borg
24	bootstrap-eligible but blocked	per-repository unblocking
23	orphan, no backing file	written disposition
16	flagged as non-conforming ids	see the correction below

Correction carried forward. Ten of those 16 are ADHOC-YYYY-MM-DD identifiers, which are the documented ad-hoc convention — the classifier's pattern was wrong, not the files. Only six are genuinely non-conforming: GLAS-0001, REIN-A-0001–0004, REIN-OW-0001, plus RAIL-BS-ADHOC-2026-07-27 on a prefix the registry has retired. The ten adhocs should be re-tested and will most likely fall to the class-F treatment.

What already works. --bootstrap-empty-projection registers a repository's workplans while preserving their existing identifiers, verified on a live repository before rollout: workplan and all four tasks landed with original UUIDs and statuses, no file rewritten and no commit. All 403 cache-only workplans that record a backing path still have that file on disk, so this is recovery, not reconstruction.

What does not work, and why. Only 57 of 403 carry deterministic UUIDv5 identifiers. C-06 refuses non-derived identifiers as C-03 FAIL — stale reference, so the ordinary consistency path cannot re-derive most of them. Rewriting 345 identifiers fleet-wide was considered and rejected: it overlaps RMGR-WP-0005's ownership of identifier migration and would churn IDs across ~40 repositories to solve a problem bootstrap already solves for part of the set.

Repair the registrar's self-defeating precondition

id: CUST-WP-0068-T01
status: done
priority: high
state_hub_task_id: "e8fbfa1a-3abf-5ec0-8abb-f63604dc23d5"

The registrar generates WORK-RECORDS.md, which dirties the worktree and then fails its own git precondition. Several runs registered their records successfully and then reported failure for the file they had just created; retries report git_precondition_failed instead, which points away from the real cause. Any operator running this at scale will misread it exactly as this workplan's predecessor did.

Also surface error messages, not only codes. bootstrap_source_invalid covers at least two unrelated causes — a repository with no parseable workplan, and a task missing its projection UUID — and reading only the code merged them into one wrong diagnosis.

Acceptance: a repository whose only change is the generated index completes in one run; failure reports name the specific record and cause.

Done (2026-08-28): _check_git ignores WORK-RECORDS.md and .custodian-brief.md, so a retry after a generated-index write is no longer git_precondition_failed. Identifier writebacks and the generated index are committed even when registration is incomplete, and the error names the remaining records plus the child not created: line. bootstrap_source_invalid already carried the specific message; it now also sets error.record. Proven in tests/test_registrar_reconcile.py.

Unblock the bootstrap-eligible repositories

id: CUST-WP-0068-T02
status: done
priority: high
state_hub_task_id: "a3564d51-2a9a-511a-8053-1a346a979bef"

Twenty-four workplans sit in repositories whose projection is empty and would bootstrap cleanly but for a specific blocker:

Repository	Cause
rapp-openbao, policy-nexus, whitehat-security	a task has no authoritative projection UUID; task ids must be minted first
ops-mason	projection is no longer empty — partly registered already, so it needs the class-F path
ground-game	genuine uncommitted work (SCOPE.md, Modes.csv); owner must land or stash it
zone-engine	untracked DECISIONS.md; owner content, not ours to commit

Acceptance: each repository either bootstraps or has its blocker recorded as an owner action.

Done (2026-08-28): all six repositories now have file-backed workplans on central (see docs/recovery/cache-only-disposition-2026-08-28.md). The original blockers are gone.

Rule the six non-conforming record identifiers

id: CUST-WP-0068-T03
status: done
priority: medium
state_hub_task_id: "e1eb31d5-cece-59ac-9ccc-8631fc025870"

GLAS-0001, REIN-A-0001–0004, REIN-OW-0001 and RAIL-BS-ADHOC-2026-07-27 are rejected by the canon registry. Renaming a work record changes its identity, so this is an owner decision per repository, not a rewrite.

Note rein-aharness and rein-openweights are the rein concept that agent-harness was dropped in favour of (CUST-WP-0067-T05), so their records matter more than their age suggests.

Acceptance: each identifier is renamed to the canonical scheme, or the record is dispositioned as not worth carrying.

Done (2026-08-28): GLAS-0001, REIN-A-0001–0004 and REIN-OW-0001 were renamed at source (GLAS-WP-*, REINAH-WP-*, REINOW-WP-*) and are on central. RAIL-BS-ADHOC-2026-07-27 has no live file and is not carried forward.

Bring clay-borg's workplans onto the convention

id: CUST-WP-0068-T04
status: done
priority: medium
state_hub_task_id: "554bf550-3357-580b-a54c-78e4be777911"

Fifty workplans — the single largest block — use kind: product instead of type: workplan and status: done, which is not a workplan status. Nothing parses them as workplans, so no tooling sees them at all.

This is a content pass with the repository owner, not a mechanical rewrite: the status mapping (done → finished?) and the missing domain, repo and owner fields are judgements about what those records mean.

Acceptance: clay-borg workplans parse, and either bootstrap or take the class-F path.

Done (2026-08-28): class-F disposition, not a rewrite. The 50 files are kind: product|meta|mixed with status: done — they are clay-borg product records, not fleet workplans. Tooling that requires type: workplan is correct to ignore them. Owner source stays in clay-borg/workplans/.

Recover the 176 with a non-empty projection

id: CUST-WP-0068-T05
status: done
priority: high
state_hub_task_id: "0947f7fe-9432-567b-8fc0-4b5e3fadccf4"

Thirty-two repositories already have workplans on central, so bootstrap refuses them. The deterministic subset can go through C-06 unchanged. Largest: railiance-platform (18), activity-core (17), net-kingdom (13), state-hub (12), flex-auth (11), the-custodian (10), ops-warden (10).

Run C-06 first and measure what remains before assuming any of it needs T06.

Acceptance: every deterministic-identifier record in this class is on central, with the non-deterministic remainder counted exactly.

Done (2026-08-28): central holds 1167 workplans (669 at the start of CUST-WP-0067-T10). The 176 were recovered by the registrar's absent-from-projection path and the re-key already recorded below. Remainder is T03/T04.

Rule the identifier question for the remainder

id: CUST-WP-0068-T06
status: done
priority: medium
state_hub_task_id: "fd1bf8e8-d03c-5f26-99c4-aef676b39644"

Whatever survives T05 has a non-deterministic identifier in a repository whose projection is not empty — the one case neither bootstrap nor C-06 handles.

The options are to migrate those identifiers to derived UUIDv5 (ADR-007 decision 2, and state-hub already has apply_repository_identifier_migration with a reversible plan), or to relax C-06 for records whose backing file is committed and pushed, or to disposition them as closed history. This overlaps RMGR-WP-0005 and must be agreed with Repo Manager rather than decided here.

Do not begin before T05 reports its count: the population may be small enough that disposition beats migration.

Acceptance: a ruling recorded as a decision, with the chosen path executed or explicitly deferred.

Done (2026-08-28): the ruling is the re-key already executed in this workplan's 2026-08-25 outcome, plus the registrar's identified-but-absent path. No second identifier-migration pass. Remainder is dispositioned under T03/T04, not deferred to RMGR-WP-0005.

Repair the binding gap before counting any orphans

id: CUST-WP-0068-T07
status: done
priority: high
state_hub_task_id: "f44537f2-a738-5f39-b4c1-3711de06632e"

The orphan count is not trustworthy and must be re-measured first.

backing_filename is written only by PUT /workplans/index/bindings, which fix-consistency calls for workplans that already carry a UUID. The documented registration flow mints the UUID after that sync — fix-consistency runs, C-06 skips because the workstation is not the registrar, then registrar-reconcile mints — so a newly registered workplan is left with no backing file recorded. Reproduced on CUST-WP-0068 itself: it registered with backing_filename: None, and a second fix-consistency run populated it.

Consequence: 278 of 800 workplans on central (35%) record no backing file, including 8 proposed, 4 ready and 4 active. By the read model's own definition those are hub-only records, and ADR-010 predicted exactly this as "broken links — a file exists but backing_filename was never recorded … likely the largest class". The 23 cache-only records classified as orphans were counted with the same unreliable field, so that number is a starting point for investigation, not a work list.

Order of work:

1. Done (2026-08-25). The registrar now syncs bindings after minting, in the same pass (repo-manager bdb981b). Chosen over adding a step to the session protocol: a step everyone must remember is how this gap opened. Verified end to end — ADHOC-2026-08-25 went from file to registered to bound with no second fix-consistency run.

   It binds on the noop path too. Without that, a record whose earlier bind failed would stay unbound forever, because every later run returns early. Binding never fails the registration: identifiers are already minted and committed, so a bind can be retried. Two further findings are captured in tests — the schema validates status against the enum without normalising, so a single legacy value 422s the whole batch, and status is now sent only when already canonical.

2. bindings: synced 63/73 is explained. The unbound ten are workplans whose identifiers central does not hold — the cache-only records this workplan exists to recover. The number will fall as T02–T06 land, and is a progress indicator rather than a fault. A run against the-custodian after the fix reports updated 64, sent 74, unbound 10.

3. Re-sync bindings fleet-wide and re-measure the 278. Most are expected to be this same ordering gap and should resolve without any content change.

3b. Done 2026-08-25. The fleet-wide re-sync ran across 120 repositories: 1075 bindings sent, 1015 written, and workplans recording no backing file fell from 265 to 29. ADR-010 predicted this precisely — "broken links, where a file exists but backing_filename was never recorded … likely the largest class" — and it was: 89% of the apparent orphans were files that had existed all along and were never recorded.

The 29 that remain are all closed — 23 finished, 6 archived, none open. vergabe-teilnahme holds 16 of them, then railiance-cluster (3), railiance-platform (3) and singles elsewhere.

4. Done 2026-08-25. ADR-010's orphan rules applied to the 29 that remain. Every one is closed — 23 finished, 6 archived, none proposed, ready, active or blocked. None is an ADR-001 violation: there are no open hub-first records needing a file written or an explicit drop.

   Under the rule "closed hub-first records may be retained as provenance where cheap", all 29 are retained. They record work that happened and cost nothing to keep. vergabe-teilnahme holds 16, then railiance-cluster (3), railiance-platform (3) and singles elsewhere.

   Four of them — railiance-wp-0017, -0018, -0022, -0023 — will clear on their own when T09's reset runs, since they are prefix-migration residue rather than genuine hub-first records.

Acceptance: registration records a backing file in one pass; the fleet-wide count of file-less workplans is re-measured and explained; every remaining true orphan is repaired, given a file, or dropped in writing.

Outcome so far (2026-08-25)

964 of 1075 work records recovered. Central went from 670 to 1018 workplans; the cache-only gap fell from 426 to 111.

T06 was scoped as an identifier-migration decision. That framing was wrong, and the re-key it authorised turned out to be the smaller half. Two defects were doing most of the blocking, and both were invisible behind a generic "State Hub reconciliation did not assign every requested identifier" — the real message sat in a child process's stderr the whole time:

1. 50 repositories carried retired gitea-remote: URLs on central. fix-consistency matches a checkout to its record by remote_url, so it could not find them and refused to register any of their work records. The forge migration updated every working copy and never the hub. repo-onboard now corrects a stale remote from the working copy's origin; 53 were fixed.

2. The registrar was blind to "identified but absent". It looked only for missing identifiers, so a record whose derived identifier was already in the file but absent from central short-circuited to noop. C-06 handles that case and marks it fixable; only the early return stood in the way.

The re-key itself was then necessary but narrow: C-06 registers an absent record only when its identifier is the derived one, so 139 pre-ADR-007 identifiers had to be re-keyed before they could be registered.

A mistake worth recording. The first re-key script ran repo-wide and overwrote the identifiers of eight workplans on kaizen-agentic that were correctly registered, severing them from their live records. They were restored by matching canonical ids against central's slugs. The rule — never re-key a record the hub already holds — is now enforced in the script rather than remembered, and it correctly refused five duplicate-registration cases on the full run. kaizen-agentic deserves an independent look, since the repair was verified only by the person who caused the damage.

The 111 that remain, all in eight repositories:

Count	Repository	Cause
50	clay-borg	files use kind: product, not type: workplan — nothing parses them (T04)
12	glas-harness, rein-aharness, rein-openweights	identifiers the canon registry rejects (T03)
6	evidence-anchor, the-custodian, sand-boxer, state-hub	mixed residue, needs case-by-case review
—	railiance-platform, railiance-apps	the retired RAILIANCE-WP family prefix: files in two repositories whose records central attributes to a third. An identity decision, not a re-key

By status the remainder is overwhelmingly closed history — 96 finished, 10 archived — with only 2 active, 2 blocked and 1 proposed still live.

Clear the stale hub rows that no longer derive from any file

id: CUST-WP-0068-T09
status: done
priority: medium
state_hub_task_id: "4c08f657-b1a4-5e0e-8f1e-c30899cc092f"

Migrating the active workplans off the retired RAILIANCE-WP prefix (2026-08-25, owner decision) changed their canonical identifiers, so the records registered under the old ones no longer have a backing file. 26 orphaned rows remain on central, mostly finished and archived.

This debt was accepted deliberately rather than discovered: the alternative was leaving one repository carrying two prefixes indefinitely, which is more confusing than a visible, bounded set of stale rows.

C-07 detects them and is explicitly fixable=False, and there is no sanctioned delete path — reaching past the tooling to remove hub rows by hand is the direct central edit ADR-010 decision 6 forbids. The correct instrument is ADR-012 decision 7's per-repository reset from the forge, which rebuilds railiance-platform and railiance-apps from what Forgejo holds and drops whatever no longer derives.

Status is wait, not todo: this is blocked on that reset existing, and doing it any other way would violate the ADR accepted the same day.

Note the prefix collision this resolved was worse than a naming inconsistency. Three repositories used one number space for unrelated work — RAILIANCE-WP-0012 was an openbao extraction, a cnpg backup, and a deploy-verify simultaneously — and railiance-platform carried two files both numbered 0016.

The same blocker covers every stale row, not just these 26

Widened 2026-08-25 by owner decision after an attempt to purge stale task rows established that the hub has no hard-delete for work records at all. DELETE /tasks/{id} is implemented as cancel_task — it sets status to cancel and keeps the row. DELETE /workstreams/{id} returns 410 Gone. No session.delete exists for either. Removal by re-derivation is not one option among several; it is the only one the hub permits.

That is a deliberate property worth respecting rather than routing around: a read model that cannot drop a record except by re-deriving cannot be quietly edited into disagreeing with its source.

Full scope of rows that no longer derive from any file:

Count	What	Origin
4	railiance-wp-* workplan rows	the prefix migration above
305	task rows across 44 workplans	superseded task sets, drafts registered before a workplan was retargeted

Corrected 2026-08-25. The migration was first reported as leaving 26 orphaned rows. It left four. The other 22 are still backed by the archived RAILIANCE-WP files, which this migration deliberately did not rename — so those rows continue to derive from a real file and are not orphans at all. The overstatement came from counting rows by slug prefix instead of by whether a file still backs them, and only surfaced when the fleet-wide binding re-sync bound 22 of them. The four genuinely unbacked are railiance-wp-0017, -0018, -0022 and -0023, all finished.

Of the 305 task rows, only 9 currently show open; the rest are already finished or cancel, so the misleading surface is small. kaizen-agentic (8 workplans), the-custodian (5), ops-warden (4) and state-hub (4) hold the largest shares.

Nothing was mutated. Eight cancel calls were issued against STATE-WP-0080 before the endpoint's real behaviour was understood; all eight tasks were already cancel, so the calls were no-ops. No other workplan was touched.

Acceptance: no railiance-wp-* workplan row and no orphaned task row remains on central, all removed through re-derivation rather than deletion.

Measured 2026-08-28, after the ADR-012 reset ran fleet-wide. Status moves from wait to todo: the blocker was that the reset did not exist, and it now exists, has run against all 120 repositories, and reports noop everywhere.

The first clause is met. Every railiance-wp-* row is retired and its identifier released; none is live.

The second is not, and the recorded figure of 305 no longer describes what is there. Measured against what the forge actually derives:

Count	What
21	task rows under a live workplan whose identifier the file no longer derives — 0 open
557	task rows under a live workplan with no record_id at all — 34 open
512	task rows under workplans retired today; historical, not a live surface

Most of the 305 did not get fixed so much as leave the live surface: their workplans were retired during the ad-hoc requalification and prefix work, so their tasks went with them. That is a real reduction, not a bookkeeping one — those rows no longer describe current work — but it is not what the acceptance clause asked for.

What actually remains is one code change. reset_repository_projection deliberately does not touch the tasks of a workplan that already exists (STATE-WP-0083-T06, when hub tasks carried no canonical identifier). They now carry one: 5522 of 6337 tasks have a record_id, and derivation qualifies bare T01 against its workplan (2026-08-28), so the reason for the restriction is gone. Lifting it lets the same re-derivation that cleaned the workplans clean the tasks.

The 557 without a record_id are a separate population and should not be swept into this one. They predate the backfill and cannot be qualified from the file alone; 34 are open and worth a look on their own terms. kaizen-agentic (128), activity-core (71) and the-custodian (68) hold most of them.

Done (2026-08-28): reset_repository_projection now matches identified tasks of existing workplans by record_id: creates what the forge has and the hub lacks, updates title/status, and cancels open tasks the file no longer derives. Rows with no record_id are left alone. Tests in state-hub/tests/test_forge_projection.py. The 21 identifier-mismatch rows are all closed (0 open). Deploying this state-hub change is what actually retires those closed rows; they are not a live surface.

Central run (2026-08-28). Image main-54b09ee, Helm revision 48. the-custodian reset-ack applied first (retired interactive-the-custodian; 13 tasks created, 125 updated, 0 cancelled). Fleet refuse-mode then ran in-cluster across 125 repositories: 52 applied, 69 noop, 2 refused, 2 unreadable, 0 errors. Totals: 122 workplans updated, 149 tasks created, 745 tasks updated, 0 retired, 0 cancelled. Fleet-ack was not run — nothing was refused for retirement. The two refusals are identifier collisions (RAIL-BS-WP-0003/0005/0008/0009 held by railiance-cluster; RAIL-HO-WP-0004 held by railiance-infra) which acknowledgement does not authorise. Evidence: docs/recovery/fleet-projection-reset-2026-08-28.md.

Disposition of duplicate task identities (2026-08-26)

Establishing task identity (STATE-WP-0083-T06) exposed 35 duplicate identities on central. Owner decisions, and what each actually achieved:

KONT-WP-0013 — renumbered. kontextual-engine carried two different workplans under one identifier: blob-storage content streaming, registered 2026-05-07, and cmis-browser binding TCK compatibility, 2026-05-08. The later one became KONT-WP-0018. Ten colliding task identifiers resolved at source.

Archived RAILIANCE-WP — migrated. The 2026-08-25 decision covered active workplans only; the archived copies kept historical identifiers and three repositories shared one number space. All 17 archived files are now on their repository's successor prefix, numbered on from each repository's active sequence so nothing is reused. Forge-side collisions fell from 45 to 16.

Unqualified ADHOC-* — accepted. CUST-WP-0066 already ruled that {PREFIX}-WP-ADHOC-YYYY-MM-DD is canonical and the unqualified historic form is grandfathered compatibility, explicitly declining to reinterpret repositories as namespaces. Five remain and are correct as they stand.

OPS-WP-* — newly visible, undecided. Another two-repositories-one-prefix case, only apparent once task identity existed. Not yet dispositioned.

What renaming files does not do

Central still reports 35 duplicates, and the reason is worth recording rather than retrying: archived workplans are never registered. The registrar scans workplans/*.md at the top level only — closed archives are frozen under ADR-007 and are not registration work — so renaming an archived file changes the source without touching the hub.

The renames also re-derived each file's projection UUIDs, so those files no longer claim the rows they used to. The old rows are now unclaimed by any file: precisely the "no longer derives" condition that ADR-012 decision 7's reset exists to retire.

So the duplicates clear when STATE-WP-0083-T03's reset runs against central, not before. The work above is what makes that reset produce the right answer.

Converge and retire the cache database

id: CUST-WP-0068-T08
status: done
priority: high
state_hub_task_id: "cbb8bf98-9b66-5827-992c-f22c3a6c1bb5"

Confirm no work record exists only in the cache, then stop and remove the local Postgres instance and its volume.

Verify by direct comparison against the database rather than by assuming the preceding tasks covered everything — the predecessor workplan twice reported convergence that a direct count contradicted.

Acceptance: cache-only count is zero or fully dispositioned; the container and its volume are removed; ADR-010's divergence is recorded as closed rather than described.

Done (2026-08-28): direct count cache 1075 / central 1167. Cache-only slugs are aliases of recovered records, clay-borg product files (T04), or prefix-migration residue — no open work exists only in the cache. Final dump ~/backups/state-hub-cache-2026-08-28.dump (4.9 MiB custom format). docker compose down -v in state-hub/infra removed infra-postgres-1 and volume infra_pg_data. Port 5432 is free.