markitect-tool/docs/proposed-stale-derived-and-include-pinning.md

# Proposal: stale detection, pinned derived blocks, replace-or-keep

| Field | Value |
|-------|--------|
| Date | 2026-08-13 |
| Status | **proposal** (not implemented) |
| Origin | coulomb.social product needs + gap analysis of include/cache/provenance |
| Consumer | coulomb-social, agents, any markitect host UI |

## Problem

Markitect already has:

- live **includes** (`resolve_includes`, selectors, provenance events),
- **content hashes** / cache fingerprints / backend refresh plans,
- **processing provenance** envelopes for extensions and workflows.

It does **not** yet define a single, document-native way to:

1. keep a **stored snapshot** of derived content (include, pipeline, AI, rule),
2. detect that snapshot is **stale** vs current inputs,
3. **generate an alternative**,
4. let the user **replace** or **keep** (and **lock**).

Live resolve alone is not enough for co-creation products: users need  
exportable, offline, reviewable snapshots that can still be refreshed  
explicitly.

## Goals

- One **derived-slot** model for: includes, projections (host app),  
  pipelines, external data, classic compute, AI.
- Deterministic **staleness** from input fingerprints.
- **No silent overwrite** of locked or user-accepted content by default.
- CLI + library APIs hosts can use for “may be stale” UX.
- Stay Markdown-native and reversible; no required database.

## Non-goals (v1)

- Full workflow UI inside markitect-tool.
- Automatic background regeneration as default policy.
- Replacing live `mkt:include` (live resolve remains valid).

## Proposed document syntax

### Live include (existing)

```markdown
<!-- mkt:include path="src.md" selector="sections[heading=API]" -->
```

Always resolves from current target at evaluation time.

### Pinned derived block (new)

```markdown
<!-- mkt:derived id="api-excerpt"
     kind="include"
     path="src.md"
     selector="sections[heading=API]"
     heading_delta="0"
     input_hash="sha256:…"
     locked="false"
     generator="include@v1"
     generated_at="2026-08-13T12:00:00Z" -->
…snapshot body (markdown)…
<!-- /mkt:derived -->
```

Other kinds:

```markdown
<!-- mkt:derived id="sum-1" kind="ai" recipe="summarize@v2"
     inputs="body" input_hash="sha256:…" locked="false" -->
…
<!-- /mkt:derived -->

<!-- mkt:derived id="tbl" kind="pipeline" recipe="metrics@v1"
     inputs="data/metrics.json" input_hash="sha256:…" -->
…
<!-- /mkt:derived -->
```

| Attribute | Meaning |
|-----------|---------|
| `id` | Stable slot id within the document |
| `kind` | `include` \| `pipeline` \| `external` \| `ai` \| `rule` \| `compute` \| … |
| `input_hash` | Fingerprint of inputs when snapshot was accepted |
| `locked` | If true, refuse refresh/replace until unlocked |
| `recipe` / `generator` | How to recompute |
| `path` / `selector` / … | Kind-specific params (include reuses include attrs) |

**Readers** that do not understand `mkt:derived` still see the snapshot body  
as normal markdown between comments (comments ignored by most renderers).

## APIs (proposed)

```text
list_derived_slots(markdown) -> list[DerivedSlot]
fingerprint_slot(slot, context) -> str
is_stale(slot, context) -> bool
evaluate_slot(slot, context) -> Candidate   # body + provenance + new hash
apply_replace(markdown, slot_id, candidate) -> str
apply_keep(markdown, slot_id, *, ack: bool) -> str
set_lock(markdown, slot_id, locked: bool) -> str
```

`context` supplies base_dir, body text, variable bindings, optional  
extension registry for `ai` / `pipeline` recipes.

### CLI (proposed)

```bash
mkt derived status DOC.md
mkt derived refresh DOC.md --id ID --dry-run    # print candidate + stale?
mkt derived refresh DOC.md --id ID --replace
mkt derived refresh DOC.md --id ID --keep --ack # keep text, update hash
mkt derived lock DOC.md --id ID
mkt derived unlock DOC.md --id ID
```

### Include integration

- `resolve_includes` stays **live**.
- New optional mode: `materialize_includes(markdown) -> markdown`  
  converts live includes to pinned `mkt:derived` snapshots (initial pin).
- `is_stale` for `kind=include` hashes target file(s) after selector extract  
  (reuse query/extract + cache fingerprint).

## Stale → alternative → replace | keep

```text
status: stale
  evaluate_slot → candidate
  host shows Current | Alternative
    replace → apply_replace
    keep    → apply_keep (optional ack fingerprint)
    cancel  → no write
```

Library returns both bodies; **host owns UI**. Markitect does not open GUIs.

## Relation to existing pieces

| Existing | Reuse |
|----------|--------|
| `OperationProvenance` / `ProcessingProvenance` | Emit on evaluate/replace |
| `mkt cache fingerprint` | Input hashing for files |
| `resolve_includes` + selectors | `kind=include` evaluate path |
| Extension framework | Register pipeline/ai recipes |
| Backend refresh-plan | Filesystem/index stale; derived slots are document-local |

## Implementation sketch (workplan later)

1. Parse/serialize `mkt:derived` regions (ignore inside fences).  
2. `list` / `fingerprint` / `is_stale` for `kind=include` only.  
3. `evaluate` + `apply_replace` / `apply_keep` / lock.  
4. CLI `mkt derived *`.  
5. `materialize_includes` helper.  
6. Extension hook for non-include kinds.  
7. Tests: cycle-free, path sandbox, lock enforcement, fence safety.

## Acceptance (when implemented)

- [ ] Pinned include snapshot detects source change as stale.  
- [ ] Dry-run alternative differs when source changed.  
- [ ] Replace updates body + `input_hash`.  
- [ ] Keep + ack clears stale without changing body.  
- [ ] Locked slot rejects replace.  
- [ ] Live `mkt:include` unchanged for existing callers.

## References

- coulomb: `docs/architecture/stale-derived-content.md`
- coulomb: `docs/architecture/authoring-modes-and-lazy-projections.md`
- `docs/transform-compose-include.md`
- `docs/cache-incremental.md`
- `docs/content-references.md`
- `docs/backend-fabric.md`