coulomb/evidence-source
4
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Implement ESRC-WP-0002/0003/0004: HTML/MD ingest, metadata, recovery
Some checks failed
CI Smoke / host-smoke (push) Successful in 0s
Details
CI Smoke / container-smoke (push) Failing after 15m18s
Details

Browse source 
Add ingestHtml and ingestMarkdown with ADR-0003 pageless offset semantics,
PDF intrinsic metadata extraction with caller-wins merge (WP-0003), and
citation recovery primitives including re-ingest reconcile, local quote
search, and pluggable discovery hooks (ADR-0004). Mark all three workplans
finished with contract tests (80 passing).
This commit is contained in:
tegwick 2026-07-09 01:48:28 +02:00
parent df6acad828
commit 97437dfa18
28 changed files with 1295 additions and 69 deletions
Download patch file Download diff file Expand all files Collapse all files

24
workplans/ESRC-WP-0002-html-markdown-representations.md
View file

@ -4,14 +4,15 @@ type: workplan
title: "HTML and Markdown source representations"
domain: infotech
repo: evidence-source
status: proposed
status: finished
owner: codex
topic_slug: citation_evidence_mvp
created: "2026-07-08"
updated: "2026-07-08"
updated: "2026-07-09"
spec_refs:
- README.md
- docs/ADR-0001-extraction-boundary.md
- docs/ADR-0003-html-markdown-representation.md
- ../citation-engine/src/shared/document.ts
state_hub_workstream_id: "c0d28886-e02f-4df2-abb6-ebe179171c94"
---
@ -22,18 +23,19 @@ The PDF slice (ESRC-WP-0001) established the ingest boundary and contract shape.
This workplan extends ingest to HTML and Markdown sources, producing the same
`{ document, representation }` pair over engine-owned contracts.
## Open questions
## Decisions (ADR-0003)
- Which `representationType` values do HTML/MD map to, and do they need new
`DocumentRepresentation` fields in `citation-engine`?
- How is canonical text + offset map derived for reflowable formats that have
no fixed pages? (Page map may be empty or synthetic.)
- HTML maps to `representationType: "html-dom"`; Markdown to
`"markdown-rendered"`.
- Pageless formats omit `pageMap`; `offsetMap` is a single synthetic page-1
range covering `[0, canonicalText.length)`.
- Inline fixtures owned by this repo (no PII); PDF corpus rules unchanged.
## Sketch
```task
id: ESRC-WP-0002-T01
status: todo
status: done
priority: medium
state_hub_task_id: "bf1d2a2f-41b4-44b5-b1d1-4f0eec5fbf59"
```
@ -42,7 +44,7 @@ Define the HTML/MD representation contract with `citation-engine`; agree
```task
id: ESRC-WP-0002-T02
status: todo
status: done
priority: medium
state_hub_task_id: "66320911-a4b2-404f-b025-8b23dc0c9680"
```
@ -51,9 +53,9 @@ reusing `fingerprintBytes` and the canonical-text normalizer.
```task
id: ESRC-WP-0002-T03
status: todo
status: done
priority: medium
state_hub_task_id: "266d168c-d93e-4387-9527-70aa0d5b2a6b"
```
Add fixture-driven contract tests mirroring the PDF suite; decide fixture
ownership per ADR-0002.
ownership per ADR-0002.

13
workplans/ESRC-WP-0003-metadata-enrichment.md
View file

@ -4,14 +4,15 @@ type: workplan
title: "Document metadata enrichment beyond pass-through options"
domain: infotech
repo: evidence-source
status: proposed
status: finished
owner: codex
topic_slug: citation_evidence_mvp
created: "2026-07-08"
updated: "2026-07-08"
updated: "2026-07-09"
spec_refs:
- README.md
- src/pdf/ingest.ts
- src/pdf/metadata.ts
state_hub_workstream_id: "831267a1-fc93-4872-a25b-73793d9c5df4"
---
@ -26,7 +27,7 @@ it into the `Document` record without breaking the pure-over-bytes contract.
```task
id: ESRC-WP-0003-T01
status: todo
status: done
priority: low
state_hub_task_id: "45eeef15-41a9-4abd-9436-ed77e2c6b3a4"
```
@ -35,7 +36,7 @@ precedence vs. caller-supplied options (caller wins).
```task
id: ESRC-WP-0003-T02
status: todo
status: done
priority: low
state_hub_task_id: "41ca090f-c709-420a-b7c5-8332e4e82bd1"
```
@ -44,9 +45,9 @@ minimal-metadata PDFs still ingest cleanly.
```task
id: ESRC-WP-0003-T03
status: todo
status: done
priority: low
state_hub_task_id: "0004881b-6577-4a62-a2ff-cbb5c4706b1b"
```
Contract tests over the fixture corpus asserting extracted vs. overridden
metadata precedence.
metadata precedence.

20
workplans/ESRC-WP-0004-citation-recovery.md
View file

@ -4,14 +4,15 @@ type: workplan
title: "Local citation recovery and external source discovery hooks"
domain: infotech
repo: evidence-source
status: proposed
status: finished
owner: codex
topic_slug: citation_evidence_mvp
created: "2026-07-08"
updated: "2026-07-08"
updated: "2026-07-09"
spec_refs:
- INTENT.md
- README.md
- docs/ADR-0004-citation-recovery-boundary.md
state_hub_workstream_id: "2ce8b799-d67c-4dbf-aac5-0ff1f0363d62"
---
@ -22,18 +23,17 @@ selector goes stale and discovering the external source a document was drawn
from. This is the largest deferred strand and depends on the anchoring layer's
selector model.
## Open questions
## Decisions (ADR-0004)
- What is the recovery contract — does this repo *detect* stale selectors, or
only provide the re-ingest/re-fingerprint primitives the anchor layer calls?
- Where does external source discovery (URL/DOI resolution) belong relative to
`citation-engine` and the umbrella?
- `evidence-source` owns re-ingest/re-fingerprint, local quote search, and a
pluggable discovery hook registry (no network in core).
- Stale selector detection and resolution order stay in `evidence-anchor`.
## Sketch
```task
id: ESRC-WP-0004-T01
status: todo
status: done
priority: low
state_hub_task_id: "e7ff29b9-f04f-44f5-a23a-b028fc20a80b"
```
@ -42,9 +42,9 @@ Design the recovery boundary with the anchor layer: define what
```task
id: ESRC-WP-0004-T02
status: todo
status: done
priority: low
state_hub_task_id: "88a512d1-70e0-4bfa-b6b0-c1e4e31abb97"
```
Prototype re-ingest/re-fingerprint recovery primitives and a discovery hook
interface (no network coupling baked into the core).
interface (no network coupling baked into the core).