tegwick
97437dfa18
Add ingestHtml and ingestMarkdown with ADR-0003 pageless offset semantics, PDF intrinsic metadata extraction with caller-wins merge (WP-0003), and citation recovery primitives including re-ingest reconcile, local quote search, and pluggable discovery hooks (ADR-0004). Mark all three workplans finished with contract tests (80 passing).
58 lines
No EOL
2 KiB
Markdown
58 lines
No EOL
2 KiB
Markdown
# ADR-0003 — HTML and Markdown representation contract
|
|
|
|
- **Status:** accepted
|
|
- **Date:** 2026-07-09
|
|
- **Workplan:** ESRC-WP-0002 (T01)
|
|
|
|
## Context
|
|
|
|
ESRC-WP-0001 established the PDF ingest boundary. ESRC-WP-0002 extends ingest to
|
|
HTML and Markdown while reusing engine-owned contracts from
|
|
`@citation-evidence/engine/shared`.
|
|
|
|
`RepresentationType` already reserves `html-dom` and `markdown-rendered` in
|
|
`citation-engine`; no engine schema change is required for this slice.
|
|
|
|
## Decision
|
|
|
|
### Representation types
|
|
|
|
| Source format | `mediaType` | `representationType` |
|
|
| ------------- | ------------------ | ---------------------- |
|
|
| HTML | `text/html` | `html-dom` |
|
|
| Markdown | `text/markdown` | `markdown-rendered` |
|
|
|
|
### Pageless offset semantics
|
|
|
|
Reflowable formats have no fixed pages:
|
|
|
|
- `pageMap` is **omitted** (undefined).
|
|
- `offsetMap` is a **single synthetic range** on page 1 covering
|
|
`[0, canonicalText.length)` with no gaps. This keeps
|
|
`TextPositionSelector` usable without inventing physical page geometry.
|
|
- `structureMap` remains `never` in the engine contract and is not populated.
|
|
|
|
### Canonical text
|
|
|
|
Both pipelines decode source bytes as UTF-8, derive plain text from the format,
|
|
then apply `normalize()` from `@citation-evidence/engine/shared`. HTML strips
|
|
active content (`script`, `style`, `iframe`, `object`, `embed`, `noscript`)
|
|
before text extraction.
|
|
|
|
### Content hash
|
|
|
|
`contentHash` is the SHA-256 fingerprint of the **source bytes** (same rule as
|
|
PDF ingest), not a hash of canonical text.
|
|
|
|
### Fixtures
|
|
|
|
HTML/Markdown contract tests use small inline fixtures owned by this repo. They
|
|
contain no PII and do not duplicate the upstream PDF corpus (ADR-0002).
|
|
|
|
## Consequences
|
|
|
|
- `ingestHtml` and `ingestMarkdown` mirror the PDF `{ document, representation }`
|
|
shape.
|
|
- Anchoring layers may add DOM/structural selectors later; this slice delivers
|
|
canonical text and position offsets only.
|
|
- Sanitization is extraction-time stripping, not a sandboxed renderer. |