Implement ESRC-WP-0002/0003/0004: HTML/MD ingest, metadata, recovery
Some checks failed
CI Smoke / host-smoke (push) Successful in 0s
CI Smoke / container-smoke (push) Failing after 15m18s

Add ingestHtml and ingestMarkdown with ADR-0003 pageless offset semantics,
PDF intrinsic metadata extraction with caller-wins merge (WP-0003), and
citation recovery primitives including re-ingest reconcile, local quote
search, and pluggable discovery hooks (ADR-0004). Mark all three workplans
finished with contract tests (80 passing).
This commit is contained in:
tegwick 2026-07-09 01:48:28 +02:00
parent df6acad828
commit 97437dfa18
28 changed files with 1295 additions and 69 deletions

View file

@ -0,0 +1,58 @@
# ADR-0003 — HTML and Markdown representation contract
- **Status:** accepted
- **Date:** 2026-07-09
- **Workplan:** ESRC-WP-0002 (T01)
## Context
ESRC-WP-0001 established the PDF ingest boundary. ESRC-WP-0002 extends ingest to
HTML and Markdown while reusing engine-owned contracts from
`@citation-evidence/engine/shared`.
`RepresentationType` already reserves `html-dom` and `markdown-rendered` in
`citation-engine`; no engine schema change is required for this slice.
## Decision
### Representation types
| Source format | `mediaType` | `representationType` |
| ------------- | ------------------ | ---------------------- |
| HTML | `text/html` | `html-dom` |
| Markdown | `text/markdown` | `markdown-rendered` |
### Pageless offset semantics
Reflowable formats have no fixed pages:
- `pageMap` is **omitted** (undefined).
- `offsetMap` is a **single synthetic range** on page 1 covering
`[0, canonicalText.length)` with no gaps. This keeps
`TextPositionSelector` usable without inventing physical page geometry.
- `structureMap` remains `never` in the engine contract and is not populated.
### Canonical text
Both pipelines decode source bytes as UTF-8, derive plain text from the format,
then apply `normalize()` from `@citation-evidence/engine/shared`. HTML strips
active content (`script`, `style`, `iframe`, `object`, `embed`, `noscript`)
before text extraction.
### Content hash
`contentHash` is the SHA-256 fingerprint of the **source bytes** (same rule as
PDF ingest), not a hash of canonical text.
### Fixtures
HTML/Markdown contract tests use small inline fixtures owned by this repo. They
contain no PII and do not duplicate the upstream PDF corpus (ADR-0002).
## Consequences
- `ingestHtml` and `ingestMarkdown` mirror the PDF `{ document, representation }`
shape.
- Anchoring layers may add DOM/structural selectors later; this slice delivers
canonical text and position offsets only.
- Sanitization is extraction-time stripping, not a sandboxed renderer.

View file

@ -0,0 +1,54 @@
# ADR-0004 — Citation recovery boundary
- **Status:** accepted
- **Date:** 2026-07-09
- **Workplan:** ESRC-WP-0004 (T01)
## Context
`evidence-source` owns ingestion and representation generation. Citation
recovery spans stale-selector detection, local quote search, external source
discovery, and human confirmation — but selector resolution algorithms live in
`evidence-anchor`, and the `CitationRecoveryAttempt` type vocabulary lives in
`citation-engine`.
## Decision
### Owned by `evidence-source`
| Capability | Module | Notes |
| ---------- | ------ | ----- |
| Re-ingest + re-fingerprint | `src/recovery/re-ingest.ts` | Compare prior vs. fresh `Document.fingerprint`; flag `requiresReanchor` |
| Local canonical quote search | `src/recovery/local-search.ts` | Search `DocumentRepresentation.canonicalText` |
| Recovery attempt scaffold | `src/recovery/attempt.ts` | Local record using SharedContracts state vocabulary |
| Discovery hook interface | `src/recovery/discovery.ts` | Pluggable providers; **no network in core** |
### Owned elsewhere
| Capability | Owner |
| ---------- | ----- |
| Stale selector detection / resolution order | `evidence-anchor` |
| `CitationRecoveryAttempt` canonical type | `citation-engine` (future) |
| Human confirmation UX | `citation-work` / umbrella |
| External HTTP/API lookup implementations | Deployment-specific providers registered on the hook |
### Recovery flow
```text
CitationClue
→ createRecoveryAttempt (evidence-source)
→ local library search via searchCanonicalQuote (evidence-source)
→ optional SourceDiscoveryHook providers (registered by host)
→ reIngestAndCompare when fresh bytes arrive (evidence-source)
→ anchor layer re-resolves selectors when requiresReanchor is true
```
`evidence-source` **does not** decide whether a selector is stale. It supplies
primitives the anchor layer calls after it detects mismatch.
## Consequences
- No network coupling in the recovery core.
- Host apps register discovery providers explicitly (local-first default).
- Recovery state enum mirrors `wiki/SharedContracts.md` §2.6 locally until the
engine exports a shared `CitationRecoveryAttempt` type.