state-hub/workplans/STATE-WP-0083-forge-derived-projection-reset.md

id	type	title	domain	repo	status	owner	topic_slug	created	updated	related
STATE-WP-0083	workplan	Forge-derived projection reset, per repository	infotech	state-hub	active	codex	infotech	2026-08-25	2026-08-25	CUST-ADR-012 CUST-WP-0068

Forge-derived projection reset, per repository

Goal

Implement ADR-012 decision 7: discard a repository's projection and rebuild it from what Forgejo holds. Per repository, routine, idempotent, and verifiable.

This is the only sanctioned way to remove a hub record. The hub has no hard-delete for work records — DELETE /tasks/{id} is cancel_task, DELETE /workstreams/{id} is 410 Gone, and no session.delete exists for either. Re-derivation is not one option among several; it is the mechanism.

Why here and not in Repo Manager

ADR-012 decision 1 says central derives from the forge, and the pod can reach it: git ls-remote https://forgejo.coulomb.social/... succeeds anonymously from inside the cluster, and git is already in the image.

Driving the reset from Repo Manager would mean computing a projection on a workstation and pushing it into central — precisely the injection of derived state ADR-010 decision 5 forbids. Central must do its own reading. The cost is that workplan parsing exists in two places; scripts/consistency_check.py already parses frontmatter and task blocks here, so the capability is present and should be extracted rather than rewritten.

What it must not do

• Not restore preliminary overlay records. They exist because the forge does not hold them; a rebuild from the forge cannot reproduce them and must not pretend to.
• Not destroy hub-native records. Progress events, decisions and inbox messages originate in the hub (ADR-010 decision 4). They are not forge-derived and must survive a rebuild of forge-derived state.
• Not proceed silently when records would be lost. Stop, name them, and require an explicit acknowledgement.

Derive a repository's projection from the forge

id: STATE-WP-0083-T01
status: done
priority: high

Given a repository slug, clone or fetch its default branch from Forgejo into a temporary location, parse the workplan files, and compute the projection that state would produce: workplans, tasks, their identifiers, statuses and backing paths.

Read-only and side-effect free. Record the commit the derivation came from — ADR-012 decision 2 requires provenance, and the existing git_fingerprint is worthless precisely because nothing ever wrote it correctly.

Acceptance: deriving the-custodian twice from the same commit yields identical output, and the commit is reported.

Done (2026-08-25). api/services/forge_projection.py. Central clones the default branch from Forgejo and derives 69 workplans and 459 tasks from the-custodian at commit d5013ae, identically across runs. Identifiers are derived from the canonical record id in the ADR-007 namespace, verified against live records — so a forge-derived projection and a preliminary overlay agree on identity without reconciliation. Eight tests, including that identifiers are derived rather than inherited from whatever a file happens to carry.

A fresh shallow clone each time is deliberate: reusing a working copy is how a projection ends up reflecting someone's local state instead of the forge.

Report the difference before changing anything

id: STATE-WP-0083-T02
status: progress
priority: high

Compare the derived projection against what the hub currently holds for that repository, and report: records the forge has that the hub lacks, records the hub has that the forge does not, and records whose fields differ.

This is the half that is immediately useful without any destructive capability, and it is what makes decision 7's "verifiable" real. It also answers the question CUST-WP-0068-T09 is waiting on — which stale rows would actually clear — before anyone commits to clearing them.

Acceptance: a dry-run diff for a repository with known drift matches what manual inspection shows.

Workplan-level diff done; task-level blocked (2026-08-25).

The first implementation matched hub records to derived ones by UUID and was badly wrong: most hub records still carry pre-ADR-007 random identifiers, so nearly every record appeared simultaneously missing and stale. A reset built on that comparison would have destroyed and recreated the whole projection. It now matches on canonical record id, falling back to the backing file.

Workplan-level results are trustworthy and inspectable:

Repository	missing	stale	differing
whitehat-security	0	0	0
the-custodian	3	4	52
railiance-platform	18	25	2

whitehat-security reporting clean is the control: it was bootstrapped directly from its files, so a forge derivation must agree with it exactly. The four stale workplans on the-custodian were checked by hand and are real — CUST-WP-0023/0024 have no file in the repository at all, and state-hub-v0.1/v0.2 carry slugs that were never canonical ids. All four are genuine hub-first records, the class ADR-010 says to disposition.

Blocked: hub tasks carry no canonical record id. The task schema is id, workplan_id, title, status, priority, … with nothing holding CUST-WP-0067-T01. A file task and a hub task can therefore only be matched by title, which is why the task diff reports 149 missing and 131 stale for the-custodian where the workplan diff reports 3 and 4. That is the matching failing, not drift.

Task-level reset must not be built on title matching — renaming a task heading would silently destroy and recreate its record. T03 is limited to workplans until tasks carry their canonical id, which is T06.

Apply the reset transactionally

id: STATE-WP-0083-T03
status: done
priority: high

Replace the repository's forge-derived records with the derived projection in one transaction: create what is missing, update what differs, remove what no longer derives. Hub-native records are untouched.

Refuse by default when removal would destroy a record with no counterpart in the forge; report what would be lost and require an explicit acknowledgement to proceed. That refusal is evaluated per repository, so one unresolved repository never blocks the rest.

Acceptance: reset twice produces the same projection; a repository holding records the forge lacks is refused with those records named; hub-native record counts are unchanged across a reset.

Done (2026-08-26). reset_repository_projection() in api/services/forge_projection.py. Verified against live data, rolled back: whitehat-security applied 5 updates and 0 retirements; the-custodian refused, naming cust-wp-0023, cust-wp-0024, state-hub-v0.1 and state-hub-v0.2 — the four records confirmed by hand as genuine hub-first records with no file — and changed nothing.

Refusal is the default because a record that stops deriving is ambiguous: the file may have been deleted deliberately, or the caller may have pointed at the wrong branch. Only the caller can say which, so only the caller may authorise it.

Retirement never deletes, with a test asserting the row survives and session.delete is never called. A record that derives again is un-retired rather than left contradicting the forge.

The first execution of this write path was run against the cache database rather than central, and rolled back. A write path's first run belongs on the discardable copy.

Scope limit, enforced in the code rather than documented beside it. Tasks of existing workplans are untouched; tasks are created only alongside a new workplan, where nothing exists to mis-match. The outcome carries that as a note so partial convergence cannot be mistaken for full. 678 tests pass.

Fleet form as a loop over the repository form

id: STATE-WP-0083-T04
status: todo
priority: medium

The fleet-wide reset iterates the per-repository reset and shares its implementation, so the rarely-used dangerous path is exercised by the frequently used safe one. Repositories that refuse are skipped and reported, never aborting the pass.

Acceptance: a fleet dry-run reports per-repository outcomes including refusals, and completes despite them.

Retire what the reset replaces

id: STATE-WP-0083-T05
status: wait
priority: medium

Once reset is trusted, CUST-WP-0068-T09 can clear its stale rows — 4 workplan rows from the prefix migration and 305 task rows across 44 workplans.

ADR-003 decision 2's mtime-based fingerprints should also be replaced by the source commit at this point; ADR-012 invalidated that composition and the replacement belongs with the provenance work in T01.

Acceptance: CUST-WP-0068-T09 closes; no fingerprint input depends on a local filesystem.

Give hub tasks their canonical record id

id: STATE-WP-0083-T06
status: done
priority: high

Hub task rows hold no canonical identifier — only a title — so nothing reliably connects CUST-WP-0067-T01 in a file to its row. Every other record type has a stable identity; tasks do not, and that gap is what stops the reset from covering them.

Add the canonical id to the task record and populate it during derivation and registration. Once present, task matching becomes identity-based like workplans, and T03 can extend to tasks safely.

Until then a task's identity is its title, which changes whenever someone edits a heading. That is not a foundation for deletion.

Acceptance: task rows carry their canonical record id; the task diff for whitehat-security reports clean; the-custodian's task diff falls to something manual inspection confirms.

Built (2026-08-26). Migration e2b3c4d5f6a7 adds tasks.record_id, nullable because no migration can invent an identity for an existing row — only the repository files hold the mapping.

The backfill reads that mapping rather than inferring it: a file task declares both its canonical id and the projection UUID it was registered under, so the pairing is stated, not guessed. Across 121 repositories and 1104 files it recovered 5516 pairs with zero conflicts, and identified 4456 of 6073 task rows in the cache database.

Diff and reset now key on record_id where present, falling back to a title:-prefixed key so an unidentified row is visibly unidentified rather than silently title-matched.

Two refusals are tested rather than documented: a row the files do not claim keeps no identity, and a row that already has one is never overwritten — a mismatch is recorded as a conflict, not resolved. Title matching would have "worked" and destroyed a record every time someone edited a heading.

Remaining, in order. Deploy the migration to central; then run the backfill against central as an explicit operation — deliberately not inside the Helm hook, where a partial failure would silently leave half the tasks identified. The cache matched 4456 of 6073 and central has a different history, so the numbers will differ and a dry run should be compared before applying.

Only then may T03 be extended to tasks of existing workplans, which is a further change with its own verification. CUST-WP-0068-T09 is therefore two moves away, not one.

Backfill run against central (2026-08-26), and a defect it exposed.

The first implementation read local filesystem paths. Central has no workstation checkout and must not depend on one — ADR-012 decision 1 makes the forge the projection source — and the point became concrete rather than theoretical: central's Postgres is unreachable from the workstation, since the tunnel carries HTTP only. A forge-sourced variant now clones each repository in-cluster and reads the pairing from what the forge holds.

Dry run across 121 repositories: 1094 files, 5620 pairs, 5248 to update, 372 file-declared UUIDs central does not hold, zero conflicts. The count differing from the cache's 4456 of 6073 was the check that mattered — central holds the re-keyed records the cache never had, so a matching number would have meant cache state was leaking in. The forge scan reading 1094 files against the workstation's 1104 is the preliminary-overlay distinction showing up in the numbers.

Then applied, and wrong. 5248 rows were identified but only 5100 identities were distinct. A task id written as a bare T01 is unique only inside its own workplan; stored as canonical it gave every workplan's first task one identity. 51 such ids landed on 148 rows.

The backfill's own conflict detection could not see it: it checks one UUID claimed by two ids, and this was the inverse. It surfaced only because the identified count and the distinct count did not reconcile — a check that could easily have been skipped after a dry run reporting zero conflicts.

Short ids are now qualified as WORKPLAN-ID-T01, and a short id in a file with no workplan id gets no identity at all. The 136 affected rows on central were cleared for reassignment, since the backfill never overwrites an existing identity.

Central now: 5112 of 5974 identified, 5077 distinct. The remaining 34 duplicates are not from this work. They are unqualified ad-hoc ids reused across repositories — ADHOC-2026-07-02-T01 exists in three workplans — which is the problem CUST-WP-0066 closed for new records while existing ones kept their unqualified ids. They need their own disposition.

Done (2026-08-26). Corrected backfill applied to central: 5248 of 5974 tasks identified, 5213 distinct, 35 duplicate identities.

The check before applying was uniqueness, not volume — the lesson from the first attempt, where a clean dry run reporting zero conflicts concealed 51 non-unique ids. All 136 pending updates were verified to take identities held by nothing else before a single row was written.

The 35 remaining duplicates predate this work and fall into three classes:

Count	Class	Origin
18	KONT-WP-0013-T00N and similar	a workplan registered twice, so both copies claim the same task ids
12	RAILIANCE-WP-00NN-T0N	archived RAILIANCE-WP files, left unrenamed by the owner decision of 2026-08-25 — the same numbers exist in three repositories, so their task ids collide
4	ADHOC-*-T0N	unqualified ad-hoc ids reused across repositories, the class CUST-WP-0066 closed for new records only

The RAILIANCE-WP twelve are worth noting as a consequence rather than a surprise: migrating only the active workplans was the right call for blast radius, and this is the price — the archived copies keep colliding identities until they are migrated too, and task-level reset cannot act on those workplans in the meantime.

Task-level reset remains blocked, now on a narrower and better-understood problem: 35 identities, not 5974 unidentified rows.

Restore a migration mechanism for central

id: STATE-WP-0083-T07
status: done
priority: high

Central's schema is behind the code it is running, and nothing detects it. Discovered 2026-08-25 while preparing the retirement columns.

Central is at revision b8d4f0a2c6e1; the chain is b8d4f0a2c6e1 → c9e5a1b3d7f2 → d1a2b3c4e5f6. The review_contracts table does not exist on central even though the migration creating it ships inside the image currently serving traffic. The code and the database disagree silently.

There is no migration mechanism. The image CMD is bare uvicorn, the Helm chart declares no migration job, and the only evidence migrations ever ran was an ad-hoc state-hub-alembic-init Job created outside the chart — deleted earlier the same day as a retired-forge leftover. Deleting a completed Job removed a historical record rather than a working mechanism, since Jobs do not re-run, but it also removed the last visible sign that this gap existed.

This blocks T03: the retirement columns cannot reach central without it.

Needed: migrations as a declared part of the release — a chart-managed job or init container that runs alembic upgrade head before the API serves — and a check that surfaces a schema/code mismatch instead of leaving it silent. A deployment that can run against a schema it was not built for is the same class of defect as a projection that cannot name its source commit.

Acceptance: central reaches head; review_contracts exists; the mechanism is declared in the chart; a mismatch between code and schema is reported rather than tolerated.

Done (2026-08-26). Central reports schema: {status: ok, applied: d1a2b3c4e5f6, expected: d1a2b3c4e5f6}. review_contracts exists. Both pods run main-97c8762.

The first execution is worth recording, because it failed in a way that proved the design. The pre-upgrade hook ran, applied both migrations and deleted itself on success — then the k3s API tunnel dropped (unexpected EOF → connection refused) and Helm could not read the deployment to roll the API. The release wedged at pending-upgrade; no pod rolled.

The schema was therefore briefly ahead of the running code, which is the safe direction and the reason additive migrations were the right shape: the old image served correctly against the new schema throughout. A destructive migration in the same circumstance would have taken the service down.

Recovery was helm rollback to the last deployed revision — which clears the wedge without touching the schema, there being nothing to undo — then re-running the upgrade. The hook is idempotent, so the second run's alembic upgrade head was a no-op.

Two things this exposed, neither yet addressed:

• The hook's hook-succeeded delete policy removes the Job on success, so a successful migration leaves no trace. That made it look as though nothing had run when in fact everything had. Retaining a short-lived record of successful migrations, not only failed ones, would have answered the question immediately.
• bridge status reported k3s-api-railiance01 as connected while holding a stale pid, so its health signal did not reflect the dropped connection. A tunnel monitor that cannot detect the failure it exists to detect is the same defect class as a projection that cannot name its source commit.

Repair workplan files whose frontmatter never terminates

id: STATE-WP-0083-T08
status: done
priority: high

Blocks the first production reset. Found 2026-08-26 by reviewing the reset's diff before applying it — the diff proposed retiring kont-wp-0015, a live and correctly registered workplan, which is what prompted the investigation.

Eighteen workplan files across the fleet have a closing --- fused onto the last frontmatter value:

depends_on_workplans:
  - KONT-WP-0016---

There is no delimiter line, so the frontmatter never terminates. The file declares a dependency on a workplan literally named KONT-WP-0016--- and the entire body is swallowed as frontmatter. Any parser selecting on type: workplan sees nothing, so the file is not merely unparsed but invisible — it appears as neither a workplan nor an error.

Affected: citation-evidence (5), infospace-bench (2), kontextual-engine, net-kingdom (2 incl. archived), and others — 18 in total.

The consequence for reset is severe and must not be worked around: a repository containing one of these files derives a projection missing that workplan, so the reset proposes retiring a record that is entirely correct. Running it before this is repaired would retire live work.

That the refusal-by-default caught this is the design working, but the diff would have looked plausible to anyone not checking a specific record — created=0, would_retire=N reads like a tidy-up.

Repair the frontmatter, then re-derive and confirm each affected repository's diff no longer proposes retiring live workplans.

Acceptance: no workplan file yields empty frontmatter while starting with ---; the reset diff for each affected repository proposes retiring only records whose backing file genuinely no longer exists.

Done (2026-08-26). All 18 repaired across 8 repositories; zero workplan files now yield empty frontmatter. Previously invisible workplans derive again: KONT-WP-0015, CUST-WP-0000, CE-WP-0001 and the rest.

Two shapes of one defect, both a missing newline:

Count	Shape	Example
16	closing delimiter fused onto the last value	- KONT-WP-0016---
2	a value fused onto the following key	…v1.3.0state_hub_task_id: …

Only the newline was inserted; no value was altered, and the diff on each file is one line split into two.

A correction this forces. state-hub-v0.1 and state-hub-v0.2 were reported earlier as genuine hub-first records with no backing file, and dispositioned on that basis. They are CUST-WP-0000 and CUST-WP-0000b — files that existed all along and were unreadable. The claim was wrong because "no file backs this record" was inferred from a parser's silence rather than checked against the directory.

The ordering bug in the repair itself is worth recording, since it is the same mistake in miniature. The first version only looked for a fused delimiter when no \n--- existed anywhere in the file. Five citation-evidence files have a horizontal rule in the body, so a delimiter was found — just the wrong one — and the real defect was hidden behind it. Checking whether the frontmatter parses before trusting the delimiter position fixed it, taking the count from 11 repairable to 18.

Reset diff after the frontmatter repairs (2026-08-26)

Re-run against central once T08 landed. The false retirements are gone:

Repository	before	after
kontextual-engine	1 (a live workplan)	0 retire, 18 updated
citation-evidence	not yet measured	0 retire, 12 updated
the-custodian	4	2 retire
net-kingdom	—	errors

Two findings the re-run produced, both worth more than the diff itself.

The four "hub-first records" were never hub-first

cust-wp-0023 and cust-wp-0024 were reported as records with no backing file, verified by hand, and dispositioned on that basis. They have files — CUST-WP-0023-tpsc.md and CUST-WP-0024-repo-doi-gate.md — which declare type: feature, so the derivation correctly excludes them while the hub holds them as workplans.

Together with state-hub-v0.1/v0.2, which turned out to be CUST-WP-0000 and CUST-WP-0000b with unterminated frontmatter, all four claims were wrong, by three different mechanisms: broken frontmatter, a mismatched type, and in both cases an inference from a parser's silence rather than a check against the directory. "The tool did not find it" was repeatedly reported as "it does not exist".

Ten files in workplans/ directories declare something other than type: workplan — feature, api, target, runbook, domain, extension-workplan. Some are registered as workplans on the hub. Each needs a disposition: either the file's type is wrong, or the hub record should not exist. Until then a reset will keep proposing to retire them.

The reset crashes where it should refuse

net-kingdom raises IntegrityError: Key (id)=(e25ba5fa…) already exists. Its ADHOC-2026-08-23 derives to an identifier another repository already holds — the exact collision CUST-WP-0066 was written about, where two repositories created the same daily identifier on the same day.

The reset must detect that a derived identifier belongs to another repository and refuse with that stated, rather than failing on a database constraint. A refusal is a decision the caller can act on; a constraint violation is a stack trace.

Fixed and verified live (2026-08-26, main-5e4d0be). net-kingdom now returns refused with reason "derived identifier already belongs to another repository", naming ADHOC-2026-08-23 and the holder.

Acknowledging retirements deliberately does not authorise a collision: those are different decisions — one says the work is gone, the other says take an identifier another repository owns — and conflating them would let a routine acknowledgement carry an identity change through. Covered by test.

Worth recording how nearly this was mis-reported. A first check run against the cache returned refused and was almost reported as proof; it had exercised the retirement path instead, because the cache does not hold the colliding record. The output looked like success while testing nothing of the sort.