Implement ESRC-WP-0002/0003/0004: HTML/MD ingest, metadata, recovery
Some checks failed
CI Smoke / host-smoke (push) Successful in 0s
CI Smoke / container-smoke (push) Failing after 15m18s

Add ingestHtml and ingestMarkdown with ADR-0003 pageless offset semantics,
PDF intrinsic metadata extraction with caller-wins merge (WP-0003), and
citation recovery primitives including re-ingest reconcile, local quote
search, and pluggable discovery hooks (ADR-0004). Mark all three workplans
finished with contract tests (80 passing).
This commit is contained in:
tegwick 2026-07-09 01:48:28 +02:00
parent df6acad828
commit 97437dfa18
28 changed files with 1295 additions and 69 deletions

View file

@ -8,8 +8,8 @@
## One-liner
Headless PDF ingest that turns document bytes into engine-owned evidence
contracts (fingerprint, canonical text, page/offset maps).
Headless document ingest that turns PDF, HTML, and Markdown bytes into
engine-owned evidence contracts (fingerprint, canonical text, page/offset maps).
---
@ -18,25 +18,30 @@ contracts (fingerprint, canonical text, page/offset maps).
The rest of the citation-evidence ecosystem (anchoring, evidence linking,
binders) needs a stable, runtime-agnostic way to go from *raw document bytes*
to a `Document` + `DocumentRepresentation`. This repo owns that transformation
for PDFs — pure over bytes, no persistence, no viewer, no UI framework.
for PDF, HTML, and Markdown — pure over bytes, no persistence, no viewer,
no UI framework.
---
## In Scope
- PDF byte ingest → `{ document, representation }` (`ingestPdf`)
- HTML byte ingest → `{ document, representation }` (`ingestHtml`)
- Markdown byte ingest → `{ document, representation }` (`ingestMarkdown`)
- PDF text extraction → canonical text + page map + gap-free offset map (`extractPdf`)
- PDF intrinsic metadata extraction with caller-wins merge (`extractPdfMetadata`)
- SHA-256 byte fingerprinting (`fingerprintBytes`)
- Citation recovery primitives: re-ingest/reconcile, local quote search, discovery hooks
- Browser upload helpers behind a separate entry point (`createPdfByteStore`, `ingestPdfFromFile`)
---
## Out of Scope
- HTML / Markdown representations (deferred — see `workplans/`)
- Persisting documents or representations (caller's job)
- Viewer URL resolution and blob-vs-fixture policy (stays in `citation-evidence`)
- Citation recovery / external source discovery (deferred)
- Stale selector detection and anchor resolution (stays in `evidence-anchor`)
- Network-backed external source discovery implementations (host registers hooks)
- Defining domain contracts — those are owned by `citation-engine`
---
@ -51,7 +56,7 @@ for PDFs — pure over bytes, no persistence, no viewer, no UI framework.
## Not Relevant When
- You need HTML/Markdown ingest (not yet implemented)
- You need formats beyond PDF/HTML/Markdown (not yet implemented)
- You need viewer/session/UI behavior (see `citation-evidence`)
- You are changing the `Document`/`DocumentRepresentation` contract (see `citation-engine`)
@ -60,7 +65,7 @@ for PDFs — pure over bytes, no persistence, no viewer, no UI framework.
## Current State
- Status: active
- Implementation: partial (PDF slice implemented; HTML/MD and recovery deferred)
- Implementation: partial (PDF/HTML/MD ingest, metadata enrichment, recovery primitives)
- Stability: evolving
- Usage: internal (consumed by `citation-evidence` across the repo boundary)
@ -92,7 +97,7 @@ for PDFs — pure over bytes, no persistence, no viewer, no UI framework.
## Getting Oriented
- Start with: `README.md`, then `docs/ADR-0001-extraction-boundary.md`
- Key files / directories: `src/pdf/` (headless core), `src/browser/` (upload helpers)
- Key files / directories: `src/pdf/`, `src/html/`, `src/markdown/`, `src/recovery/`, `src/browser/`
- Entry points: `src/index.ts` (headless), `src/browser/index.ts` (browser)
---