Implement ESRC-WP-0002/0003/0004: HTML/MD ingest, metadata, recovery
Add ingestHtml and ingestMarkdown with ADR-0003 pageless offset semantics, PDF intrinsic metadata extraction with caller-wins merge (WP-0003), and citation recovery primitives including re-ingest reconcile, local quote search, and pluggable discovery hooks (ADR-0004). Mark all three workplans finished with contract tests (80 passing).
This commit is contained in:
parent
df6acad828
commit
97437dfa18
28 changed files with 1295 additions and 69 deletions
58
docs/ADR-0003-html-markdown-representation.md
Normal file
58
docs/ADR-0003-html-markdown-representation.md
Normal file
|
|
@ -0,0 +1,58 @@
|
|||
# ADR-0003 — HTML and Markdown representation contract
|
||||
|
||||
- **Status:** accepted
|
||||
- **Date:** 2026-07-09
|
||||
- **Workplan:** ESRC-WP-0002 (T01)
|
||||
|
||||
## Context
|
||||
|
||||
ESRC-WP-0001 established the PDF ingest boundary. ESRC-WP-0002 extends ingest to
|
||||
HTML and Markdown while reusing engine-owned contracts from
|
||||
`@citation-evidence/engine/shared`.
|
||||
|
||||
`RepresentationType` already reserves `html-dom` and `markdown-rendered` in
|
||||
`citation-engine`; no engine schema change is required for this slice.
|
||||
|
||||
## Decision
|
||||
|
||||
### Representation types
|
||||
|
||||
| Source format | `mediaType` | `representationType` |
|
||||
| ------------- | ------------------ | ---------------------- |
|
||||
| HTML | `text/html` | `html-dom` |
|
||||
| Markdown | `text/markdown` | `markdown-rendered` |
|
||||
|
||||
### Pageless offset semantics
|
||||
|
||||
Reflowable formats have no fixed pages:
|
||||
|
||||
- `pageMap` is **omitted** (undefined).
|
||||
- `offsetMap` is a **single synthetic range** on page 1 covering
|
||||
`[0, canonicalText.length)` with no gaps. This keeps
|
||||
`TextPositionSelector` usable without inventing physical page geometry.
|
||||
- `structureMap` remains `never` in the engine contract and is not populated.
|
||||
|
||||
### Canonical text
|
||||
|
||||
Both pipelines decode source bytes as UTF-8, derive plain text from the format,
|
||||
then apply `normalize()` from `@citation-evidence/engine/shared`. HTML strips
|
||||
active content (`script`, `style`, `iframe`, `object`, `embed`, `noscript`)
|
||||
before text extraction.
|
||||
|
||||
### Content hash
|
||||
|
||||
`contentHash` is the SHA-256 fingerprint of the **source bytes** (same rule as
|
||||
PDF ingest), not a hash of canonical text.
|
||||
|
||||
### Fixtures
|
||||
|
||||
HTML/Markdown contract tests use small inline fixtures owned by this repo. They
|
||||
contain no PII and do not duplicate the upstream PDF corpus (ADR-0002).
|
||||
|
||||
## Consequences
|
||||
|
||||
- `ingestHtml` and `ingestMarkdown` mirror the PDF `{ document, representation }`
|
||||
shape.
|
||||
- Anchoring layers may add DOM/structural selectors later; this slice delivers
|
||||
canonical text and position offsets only.
|
||||
- Sanitization is extraction-time stripping, not a sandboxed renderer.
|
||||
54
docs/ADR-0004-citation-recovery-boundary.md
Normal file
54
docs/ADR-0004-citation-recovery-boundary.md
Normal file
|
|
@ -0,0 +1,54 @@
|
|||
# ADR-0004 — Citation recovery boundary
|
||||
|
||||
- **Status:** accepted
|
||||
- **Date:** 2026-07-09
|
||||
- **Workplan:** ESRC-WP-0004 (T01)
|
||||
|
||||
## Context
|
||||
|
||||
`evidence-source` owns ingestion and representation generation. Citation
|
||||
recovery spans stale-selector detection, local quote search, external source
|
||||
discovery, and human confirmation — but selector resolution algorithms live in
|
||||
`evidence-anchor`, and the `CitationRecoveryAttempt` type vocabulary lives in
|
||||
`citation-engine`.
|
||||
|
||||
## Decision
|
||||
|
||||
### Owned by `evidence-source`
|
||||
|
||||
| Capability | Module | Notes |
|
||||
| ---------- | ------ | ----- |
|
||||
| Re-ingest + re-fingerprint | `src/recovery/re-ingest.ts` | Compare prior vs. fresh `Document.fingerprint`; flag `requiresReanchor` |
|
||||
| Local canonical quote search | `src/recovery/local-search.ts` | Search `DocumentRepresentation.canonicalText` |
|
||||
| Recovery attempt scaffold | `src/recovery/attempt.ts` | Local record using SharedContracts state vocabulary |
|
||||
| Discovery hook interface | `src/recovery/discovery.ts` | Pluggable providers; **no network in core** |
|
||||
|
||||
### Owned elsewhere
|
||||
|
||||
| Capability | Owner |
|
||||
| ---------- | ----- |
|
||||
| Stale selector detection / resolution order | `evidence-anchor` |
|
||||
| `CitationRecoveryAttempt` canonical type | `citation-engine` (future) |
|
||||
| Human confirmation UX | `citation-work` / umbrella |
|
||||
| External HTTP/API lookup implementations | Deployment-specific providers registered on the hook |
|
||||
|
||||
### Recovery flow
|
||||
|
||||
```text
|
||||
CitationClue
|
||||
→ createRecoveryAttempt (evidence-source)
|
||||
→ local library search via searchCanonicalQuote (evidence-source)
|
||||
→ optional SourceDiscoveryHook providers (registered by host)
|
||||
→ reIngestAndCompare when fresh bytes arrive (evidence-source)
|
||||
→ anchor layer re-resolves selectors when requiresReanchor is true
|
||||
```
|
||||
|
||||
`evidence-source` **does not** decide whether a selector is stale. It supplies
|
||||
primitives the anchor layer calls after it detects mismatch.
|
||||
|
||||
## Consequences
|
||||
|
||||
- No network coupling in the recovery core.
|
||||
- Host apps register discovery providers explicitly (local-first default).
|
||||
- Recovery state enum mirrors `wiki/SharedContracts.md` §2.6 locally until the
|
||||
engine exports a shared `CitationRecoveryAttempt` type.
|
||||
Loading…
Add table
Add a link
Reference in a new issue