From cb3bd78b982df982271f4b7a469398da2a28f77c Mon Sep 17 00:00:00 2001 From: tegwick Date: Thu, 13 Aug 2026 00:19:54 +0200 Subject: [PATCH] Propose pinned derived blocks with stale detect and replace-or-keep Document mkt:derived snapshot syntax, fingerprint/stale APIs, and CLI for include/pipeline/AI slots. Driven by coulomb co-creation needs; not implemented. --- ...posed-stale-derived-and-include-pinning.md | 179 ++++++++++++++++++ 1 file changed, 179 insertions(+) create mode 100644 docs/proposed-stale-derived-and-include-pinning.md diff --git a/docs/proposed-stale-derived-and-include-pinning.md b/docs/proposed-stale-derived-and-include-pinning.md new file mode 100644 index 0000000..e39691b --- /dev/null +++ b/docs/proposed-stale-derived-and-include-pinning.md @@ -0,0 +1,179 @@ +# Proposal: stale detection, pinned derived blocks, replace-or-keep + +| Field | Value | +|-------|--------| +| Date | 2026-08-13 | +| Status | **proposal** (not implemented) | +| Origin | coulomb.social product needs + gap analysis of include/cache/provenance | +| Consumer | coulomb-social, agents, any markitect host UI | + +## Problem + +Markitect already has: + +- live **includes** (`resolve_includes`, selectors, provenance events), +- **content hashes** / cache fingerprints / backend refresh plans, +- **processing provenance** envelopes for extensions and workflows. + +It does **not** yet define a single, document-native way to: + +1. keep a **stored snapshot** of derived content (include, pipeline, AI, rule), +2. detect that snapshot is **stale** vs current inputs, +3. **generate an alternative**, +4. let the user **replace** or **keep** (and **lock**). + +Live resolve alone is not enough for co-creation products: users need +exportable, offline, reviewable snapshots that can still be refreshed +explicitly. + +## Goals + +- One **derived-slot** model for: includes, projections (host app), + pipelines, external data, classic compute, AI. +- Deterministic **staleness** from input fingerprints. +- **No silent overwrite** of locked or user-accepted content by default. +- CLI + library APIs hosts can use for “may be stale” UX. +- Stay Markdown-native and reversible; no required database. + +## Non-goals (v1) + +- Full workflow UI inside markitect-tool. +- Automatic background regeneration as default policy. +- Replacing live `mkt:include` (live resolve remains valid). + +## Proposed document syntax + +### Live include (existing) + +```markdown + +``` + +Always resolves from current target at evaluation time. + +### Pinned derived block (new) + +```markdown + +…snapshot body (markdown)… + +``` + +Other kinds: + +```markdown + +… + + + +… + +``` + +| Attribute | Meaning | +|-----------|---------| +| `id` | Stable slot id within the document | +| `kind` | `include` \| `pipeline` \| `external` \| `ai` \| `rule` \| `compute` \| … | +| `input_hash` | Fingerprint of inputs when snapshot was accepted | +| `locked` | If true, refuse refresh/replace until unlocked | +| `recipe` / `generator` | How to recompute | +| `path` / `selector` / … | Kind-specific params (include reuses include attrs) | + +**Readers** that do not understand `mkt:derived` still see the snapshot body +as normal markdown between comments (comments ignored by most renderers). + +## APIs (proposed) + +```text +list_derived_slots(markdown) -> list[DerivedSlot] +fingerprint_slot(slot, context) -> str +is_stale(slot, context) -> bool +evaluate_slot(slot, context) -> Candidate # body + provenance + new hash +apply_replace(markdown, slot_id, candidate) -> str +apply_keep(markdown, slot_id, *, ack: bool) -> str +set_lock(markdown, slot_id, locked: bool) -> str +``` + +`context` supplies base_dir, body text, variable bindings, optional +extension registry for `ai` / `pipeline` recipes. + +### CLI (proposed) + +```bash +mkt derived status DOC.md +mkt derived refresh DOC.md --id ID --dry-run # print candidate + stale? +mkt derived refresh DOC.md --id ID --replace +mkt derived refresh DOC.md --id ID --keep --ack # keep text, update hash +mkt derived lock DOC.md --id ID +mkt derived unlock DOC.md --id ID +``` + +### Include integration + +- `resolve_includes` stays **live**. +- New optional mode: `materialize_includes(markdown) -> markdown` + converts live includes to pinned `mkt:derived` snapshots (initial pin). +- `is_stale` for `kind=include` hashes target file(s) after selector extract + (reuse query/extract + cache fingerprint). + +## Stale → alternative → replace | keep + +```text +status: stale + evaluate_slot → candidate + host shows Current | Alternative + replace → apply_replace + keep → apply_keep (optional ack fingerprint) + cancel → no write +``` + +Library returns both bodies; **host owns UI**. Markitect does not open GUIs. + +## Relation to existing pieces + +| Existing | Reuse | +|----------|--------| +| `OperationProvenance` / `ProcessingProvenance` | Emit on evaluate/replace | +| `mkt cache fingerprint` | Input hashing for files | +| `resolve_includes` + selectors | `kind=include` evaluate path | +| Extension framework | Register pipeline/ai recipes | +| Backend refresh-plan | Filesystem/index stale; derived slots are document-local | + +## Implementation sketch (workplan later) + +1. Parse/serialize `mkt:derived` regions (ignore inside fences). +2. `list` / `fingerprint` / `is_stale` for `kind=include` only. +3. `evaluate` + `apply_replace` / `apply_keep` / lock. +4. CLI `mkt derived *`. +5. `materialize_includes` helper. +6. Extension hook for non-include kinds. +7. Tests: cycle-free, path sandbox, lock enforcement, fence safety. + +## Acceptance (when implemented) + +- [ ] Pinned include snapshot detects source change as stale. +- [ ] Dry-run alternative differs when source changed. +- [ ] Replace updates body + `input_hash`. +- [ ] Keep + ack clears stale without changing body. +- [ ] Locked slot rejects replace. +- [ ] Live `mkt:include` unchanged for existing callers. + +## References + +- coulomb: `docs/architecture/stale-derived-content.md` +- coulomb: `docs/architecture/authoring-modes-and-lazy-projections.md` +- `docs/transform-compose-include.md` +- `docs/cache-incremental.md` +- `docs/content-references.md` +- `docs/backend-fabric.md`