markitect-tool/docs/proposed-stale-derived-and-include-pinning.md

Proposal: stale detection, pinned derived blocks, replace-or-keep

Field	Value
Date	2026-08-13
Status	proposal (not implemented)
Origin	coulomb.social product needs + gap analysis of include/cache/provenance
Consumer	coulomb-social, agents, any markitect host UI

Problem

Markitect already has:

• live includes (resolve_includes, selectors, provenance events),
• content hashes / cache fingerprints / backend refresh plans,
• processing provenance envelopes for extensions and workflows.

It does not yet define a single, document-native way to:

1. keep a stored snapshot of derived content (include, pipeline, AI, rule),
2. detect that snapshot is stale vs current inputs,
3. generate an alternative,
4. let the user replace or keep (and lock).

Live resolve alone is not enough for co-creation products: users need
exportable, offline, reviewable snapshots that can still be refreshed
explicitly.

Goals

• One derived-slot model for: includes, projections (host app),
  pipelines, external data, classic compute, AI.
• Deterministic staleness from input fingerprints.
• No silent overwrite of locked or user-accepted content by default.
• CLI + library APIs hosts can use for “may be stale” UX.
• Stay Markdown-native and reversible; no required database.

Non-goals (v1)

• Full workflow UI inside markitect-tool.
• Automatic background regeneration of pinned snapshots without user consent.
• Replacing live mkt:include as the default (live resolve remains the default product mode — spreadsheet-like).

Product policy (coulomb): live includes by default; pin is the exception
(“paste values”). Stale → alternative → replace|keep applies to pinned
derived slots and to projection fields, not to pure live resolve.

Proposed document syntax

Live include (existing)

<!-- mkt:include path="src.md" selector="sections[heading=API]" -->

Always resolves from current target at evaluation time.

Pinned derived block (new)

<!-- mkt:derived id="api-excerpt"
     kind="include"
     path="src.md"
     selector="sections[heading=API]"
     heading_delta="0"
     input_hash="sha256:…"
     locked="false"
     generator="include@v1"
     generated_at="2026-08-13T12:00:00Z" -->
…snapshot body (markdown)…
<!-- /mkt:derived -->

Other kinds:

<!-- mkt:derived id="sum-1" kind="ai" recipe="summarize@v2"
     inputs="body" input_hash="sha256:…" locked="false" -->
…
<!-- /mkt:derived -->

<!-- mkt:derived id="tbl" kind="pipeline" recipe="metrics@v1"
     inputs="data/metrics.json" input_hash="sha256:…" -->
…
<!-- /mkt:derived -->

Attribute	Meaning
id	Stable slot id within the document
kind	include | pipeline | external | ai | rule | compute | …
input_hash	Fingerprint of inputs when snapshot was accepted
locked	If true, refuse refresh/replace until unlocked
recipe / generator	How to recompute
path / selector / …	Kind-specific params (include reuses include attrs)

Readers that do not understand mkt:derived still see the snapshot body
as normal markdown between comments (comments ignored by most renderers).

APIs (proposed)

list_derived_slots(markdown) -> list[DerivedSlot]
fingerprint_slot(slot, context) -> str
is_stale(slot, context) -> bool
evaluate_slot(slot, context) -> Candidate   # body + provenance + new hash
apply_replace(markdown, slot_id, candidate) -> str
apply_keep(markdown, slot_id, *, ack: bool) -> str
set_lock(markdown, slot_id, locked: bool) -> str

context supplies base_dir, body text, variable bindings, optional
extension registry for ai / pipeline recipes.

CLI (proposed)

mkt derived status DOC.md
mkt derived refresh DOC.md --id ID --dry-run    # print candidate + stale?
mkt derived refresh DOC.md --id ID --replace
mkt derived refresh DOC.md --id ID --keep --ack # keep text, update hash
mkt derived lock DOC.md --id ID
mkt derived unlock DOC.md --id ID

Include integration

• resolve_includes stays live.
• New optional mode: materialize_includes(markdown) -> markdown
  converts live includes to pinned mkt:derived snapshots (initial pin).
• is_stale for kind=include hashes target file(s) after selector extract
  (reuse query/extract + cache fingerprint).

Stale → alternative → replace | keep

status: stale
  evaluate_slot → candidate
  host shows Current | Alternative
    replace → apply_replace
    keep    → apply_keep (optional ack fingerprint)
    cancel  → no write

Library returns both bodies; host owns UI. Markitect does not open GUIs.

Relation to existing pieces

Existing	Reuse
OperationProvenance / ProcessingProvenance	Emit on evaluate/replace
mkt cache fingerprint	Input hashing for files
resolve_includes + selectors	kind=include evaluate path
Extension framework	Register pipeline/ai recipes
Backend refresh-plan	Filesystem/index stale; derived slots are document-local

Implementation sketch (workplan later)

1. Parse/serialize mkt:derived regions (ignore inside fences).
2. list / fingerprint / is_stale for kind=include only.
3. evaluate + apply_replace / apply_keep / lock.
4. CLI mkt derived *.
5. materialize_includes helper.
6. Extension hook for non-include kinds.
7. Tests: cycle-free, path sandbox, lock enforcement, fence safety.

Acceptance (when implemented)

• Pinned include snapshot detects source change as stale.
• Dry-run alternative differs when source changed.
• Replace updates body + input_hash.
• Keep + ack clears stale without changing body.
• Locked slot rejects replace.
• Live mkt:include unchanged for existing callers.

References

• coulomb: docs/architecture/stale-derived-content.md
• coulomb: docs/architecture/authoring-modes-and-lazy-projections.md
• docs/transform-compose-include.md
• docs/cache-incremental.md
• docs/content-references.md
• docs/backend-fabric.md