Implement ESRC-WP-0002/0003/0004: HTML/MD ingest, metadata, recovery
Add ingestHtml and ingestMarkdown with ADR-0003 pageless offset semantics, PDF intrinsic metadata extraction with caller-wins merge (WP-0003), and citation recovery primitives including re-ingest reconcile, local quote search, and pluggable discovery hooks (ADR-0004). Mark all three workplans finished with contract tests (80 passing).
This commit is contained in:
parent
df6acad828
commit
97437dfa18
28 changed files with 1295 additions and 69 deletions
58
docs/ADR-0003-html-markdown-representation.md
Normal file
58
docs/ADR-0003-html-markdown-representation.md
Normal file
|
|
@ -0,0 +1,58 @@
|
|||
# ADR-0003 — HTML and Markdown representation contract
|
||||
|
||||
- **Status:** accepted
|
||||
- **Date:** 2026-07-09
|
||||
- **Workplan:** ESRC-WP-0002 (T01)
|
||||
|
||||
## Context
|
||||
|
||||
ESRC-WP-0001 established the PDF ingest boundary. ESRC-WP-0002 extends ingest to
|
||||
HTML and Markdown while reusing engine-owned contracts from
|
||||
`@citation-evidence/engine/shared`.
|
||||
|
||||
`RepresentationType` already reserves `html-dom` and `markdown-rendered` in
|
||||
`citation-engine`; no engine schema change is required for this slice.
|
||||
|
||||
## Decision
|
||||
|
||||
### Representation types
|
||||
|
||||
| Source format | `mediaType` | `representationType` |
|
||||
| ------------- | ------------------ | ---------------------- |
|
||||
| HTML | `text/html` | `html-dom` |
|
||||
| Markdown | `text/markdown` | `markdown-rendered` |
|
||||
|
||||
### Pageless offset semantics
|
||||
|
||||
Reflowable formats have no fixed pages:
|
||||
|
||||
- `pageMap` is **omitted** (undefined).
|
||||
- `offsetMap` is a **single synthetic range** on page 1 covering
|
||||
`[0, canonicalText.length)` with no gaps. This keeps
|
||||
`TextPositionSelector` usable without inventing physical page geometry.
|
||||
- `structureMap` remains `never` in the engine contract and is not populated.
|
||||
|
||||
### Canonical text
|
||||
|
||||
Both pipelines decode source bytes as UTF-8, derive plain text from the format,
|
||||
then apply `normalize()` from `@citation-evidence/engine/shared`. HTML strips
|
||||
active content (`script`, `style`, `iframe`, `object`, `embed`, `noscript`)
|
||||
before text extraction.
|
||||
|
||||
### Content hash
|
||||
|
||||
`contentHash` is the SHA-256 fingerprint of the **source bytes** (same rule as
|
||||
PDF ingest), not a hash of canonical text.
|
||||
|
||||
### Fixtures
|
||||
|
||||
HTML/Markdown contract tests use small inline fixtures owned by this repo. They
|
||||
contain no PII and do not duplicate the upstream PDF corpus (ADR-0002).
|
||||
|
||||
## Consequences
|
||||
|
||||
- `ingestHtml` and `ingestMarkdown` mirror the PDF `{ document, representation }`
|
||||
shape.
|
||||
- Anchoring layers may add DOM/structural selectors later; this slice delivers
|
||||
canonical text and position offsets only.
|
||||
- Sanitization is extraction-time stripping, not a sandboxed renderer.
|
||||
Loading…
Add table
Add a link
Reference in a new issue