<divclass="wrap"><header><divclass="eyebrow"><span>CUST-ADR-003</span><spanclass="stat">accepted · accepted-2</span><span>the-custodian</span><span>reviewed 2026-08-31</span><span>generated from canonical source — do not edit</span></div><h1>Materialized Derived State with Fingerprint Invalidation for Repo-Sourced Data</h1><pclass="sub">Source: <code>the-custodian · canon/architecture/adr-003-materialized-derived-state.md · f9435cd605cc5b3cb0f2e957ce6287d9f3129aac</code></p><pclass="sub">Review due: 2027-02-28</p></header><divclass="layout"><navclass="rail"aria-label="Sections"><ol><li><ahref="#status"><spanclass="n">·</span>Status</a></li><li><ahref="#context"><spanclass="n">·</span>Context</a></li><li><ahref="#pattern-name"><spanclass="n">·</span>Pattern Name</a></li><li><ahref="#decision"><spanclass="n">·</span>Decision</a></li><li><ahref="#consequences"><spanclass="n">·</span>Consequences</a></li><li><ahref="#implementation-checklist"><spanclass="n">·</span>Implementation Checklist</a></li><li><ahref="#current-implementations"><spanclass="n">·</span>Current Implementations</a></li><li><ahref="#planned-applications"><spanclass="n">·</span>Planned Applications</a></li><li><ahref="#related"><spanclass="n">·</span>Related</a></li></ol></nav><main><sectionid="status"><h2>Status</h2>
<p>Accepted, and <strong>partially superseded by <code>ADR-012</code></strong> (accepted 2026-08-25). Decision 2's fingerprint composition is invalidated in part; decision 5's rebuild principle is given a concrete source and a required operation. See the notes on each.</p>
<p>The Custodian State Hub is a <strong>read model</strong> (CQRS terminology) — its data is fully derivable from canonical sources that live in repositories and the filesystem. No state-hub data is authoritative; it is always a derived view of what the repos contain.</p>
<p>Several categories of data fit this description:</p>
<divclass="scroll"><table><thead><tr><th>Data</th><th>Canonical source</th><th>State-hub table</th></tr></thead><tbody><tr><td>SBOM dependencies</td><td><code>uv.lock</code>, <code>package-lock.json</code>, etc.</td><td><code>sbom_entries</code></td></tr><tr><td>Third-party service declarations</td><td><code>tpsc.yaml</code></td><td><code>tpsc_entries</code></td></tr><tr><td>Provided capabilities</td><td><code>SCOPE.md</code><code>capability</code> blocks</td><td><code>capability_catalog</code></td></tr><tr><td>DoI compliance tier</td><td>14 criteria across repo files + DB</td><td><code>doi_cache</code></td></tr><tr><td>Workplan task status</td><td><code>workplans/*.md</code></td><td><code>tasks</code></td></tr></tbody></table></div>
<p>Early implementations either recomputed this data on every request (too slow) or ingested it once without invalidation (stale data goes undetected). Neither is acceptable for a system designed to give accurate, fast orientation.</p>
<p>The <code>doi_cache</code> table, introduced in CUST-WP-0024, demonstrated a pattern that solves both problems. This ADR formalises that pattern and mandates its use for all repo-sourced derived data.</p>
</section>
<sectionid="pattern-name"><h2>Pattern Name</h2>
<p><strong>Materialized Derived State with Fingerprint Invalidation.</strong></p>
<p>This pattern is known under several names in the literature:</p>
<ul><li><strong>Materialized View</strong> (SQL standard, PostgreSQL) — the stored result of a query or computation, refreshed on demand when source data changes.</li><li><strong>Derived Data Store</strong> (Kleppmann, <em>Designing Data-Intensive Applications</em>, Ch. 3 & 11) — a system whose entire dataset can be rebuilt from upstream sources; it is never the source of truth.</li><li><strong>Read Model / Projection</strong> (CQRS / Event Sourcing) — a pre-computed view maintained alongside a write model, rebuilt when relevant events occur.</li><li><strong>Fingerprint-based / Content-addressed invalidation</strong> — analogous to HTTP ETags: a cache entry is valid as long as a composite hash/timestamp of its inputs matches the stored value.</li></ul>
<p>The State Hub already documents itself as a read model. This ADR extends that principle to specify <em>how</em> the read model stays fresh.</p>
</section>
<sectionid="decision"><h2>Decision</h2>
<h3>1. All repo-sourced derived data MUST be materialised in the DB</h3>
<p>Data computed from repository files or repo records must be stored in a dedicated table rather than recomputed per request. Direct computation on every API call is only permissible for development tooling or when explicitly forced by the caller.</p>
<h3>2. Each materialised table MUST carry a <code>fingerprint</code> column</h3>
<p>The fingerprint is a deterministic string encoding all inputs that affect the computed result. It is compared on each read; if unchanged, the stored result is returned without recomputation. If changed, the result is recomputed and the stored value is updated.</p>
<ul><li>Include the <code>updated_at</code> timestamp of every DB record that feeds the computation (repo record, related domain, goals, snapshots).</li><li>Include the <code>mtime</code> (filesystem modification time) of every file that feeds the computation (<code>SCOPE.md</code>, <code>CLAUDE.md</code>, lockfiles, <code>tpsc.yaml</code>, etc.).</li></ul>
<divclass="rule-quote"><p><strong>Invalidated in part 2026-08-25 by <code>ADR-012</code> decisions 1 and 2.</strong> Filesystem <code>mtime</code> is not a property of the source. It differs between machines, changes on a fresh clone, and says nothing about content — so a fingerprint built from it describes one workstation's filesystem rather than the repository. Under <code>ADR-012</code> the projection derives from the forge, and the commit that produced a record is both the correct input and the auditable one. This was not merely theoretical drift. <code>git_fingerprint</code> for <code>the-custodian</code> held the repository's <em>initial</em> commit while <code>last_state_synced_at</code> was minutes old: the field meant to identify what a projection reflects was wrong by the entire history of the repository, and nothing noticed. Replace <code>mtime</code> inputs with the source commit.</p></div>
<ul><li>Join all components with <code>|</code> as a pipe-separated string — no hashing needed since the string is compared by equality, not transmitted to clients.</li><li>If a file is absent, encode <code>filename:absent</code> rather than omitting it, so file creation also triggers invalidation.</li></ul>
<h3>3. Every materialised endpoint MUST support <code>?force_refresh=true</code></h3>
<p>Callers must always be able to bypass the cache and trigger a fresh computation. This is the escape hatch for debugging, post-ingest verification, and scheduled background refresh jobs.</p>
<h3>4. Writes to source data SHOULD update the repo record's <code>updated_at</code></h3>
<p>Operations that change source data (SBOM ingest, TPSC ingest, capability ingest) must ensure <code>managed_repos.updated_at</code> is refreshed so the fingerprint detects the change on the next read. Where data lives in a related table (e.g. <code>tpsc_snapshots</code>), the fingerprint must include that table's <code>max(snapshot_at)</code> directly rather than relying on the repo record.</p>
<h3>5. The DB is never the source of truth — the rebuild principle holds</h3>
<p>Per ADR-001, the state-hub must be rebuildable from scratch by re-ingesting all canonical sources. Materialised tables are <strong>caches</strong>, not records of authority. They may be wiped and repopulated at any time without data loss.</p>
<divclass="rule-quote"><p><strong>Given concrete form 2026-08-25 by <code>ADR-012</code> decision 7.</strong> This principle was correct and, until now, never exercised — an untested rebuild path is an assumption rather than a capability, and this one was believed for long enough that a divergence survived seven weeks behind it. <code>ADR-012</code> requires the reconstruction to exist as a routine operation, scoped per repository, sourced from the forge, and verifiable against it. The claim "without data loss" also needs its precondition stated: it holds only while the rule immediately below does. On 2026-08-25, 111 work records existed only in the hub, so a rebuild at that moment would have destroyed them. <code>ADR-012</code> therefore requires reset to refuse, per repository, when records have no counterpart in the forge.</p></div>
<ul><li>No materialised table may be the only copy of any information.</li><li>Schema migrations that wipe a materialised table are safe and expected.</li><li>Background jobs that periodically re-ingest all repos are valid and encouraged.</li></ul>
</section>
<sectionid="consequences"><h2>Consequences</h2>
<h3>Positive</h3>
<ul><li><strong>Fast reads in steady state</strong> — after the first computation, subsequent reads hit the DB with no filesystem or subprocess overhead.</li><li><strong>Accurate on change</strong> — fingerprint invalidation ensures stale data is never silently served; the cache refreshes exactly when needed.</li><li><strong>Debuggable</strong> — <code>force_refresh=true</code> and <code>checked_at</code> timestamps make it easy to see when a value was last computed and to trigger a recheck.</li><li><strong>Consistent with the read model principle</strong> — the pattern makes explicit what was always implied: state-hub data is derived, not authoritative.</li></ul>
<h3>Negative / Trade-offs</h3>
<ul><li><strong>First-call latency</strong> — cache misses are expensive (filesystem reads, subprocess calls, HTTP self-calls). Mitigated by pre-warming caches at startup or after ingest.</li><li><strong>Fingerprint completeness</strong> — if a new input is added to a computation and not added to the fingerprint, stale results will be silently returned. The fingerprint must be kept in sync with the computation.</li><li><strong>Filesystem dependency</strong> — file mtimes are volatile (e.g. <code>git checkout</code> rewrites mtimes). In practice this means a cache miss after every checkout, not a correctness problem.</li></ul>
<p>When adding a new category of repo-sourced derived data:</p>
<ul><li>[ ] Create a <code>_cache</code> or <code>_snapshots</code> table with <code>fingerprint</code> and <code>checked_at</code> columns.</li><li>[ ] Implement <code>compute_fingerprint(repo, ...)</code> in the relevant module.</li><li>[ ] Add <code>?force_refresh=true</code> query parameter to the read endpoint.</li><li>[ ] Ensure the ingest script (or write path) touches <code>managed_repos.updated_at</code> or includes a related table's <code>max(timestamp)</code> in the fingerprint.</li><li>[ ] Verify the cache can be wiped and repopulated without data loss.</li><li>[ ] Document which inputs are included in the fingerprint in a comment alongside <code>compute_fingerprint</code>.</li></ul>
