Implement ESRC-WP-0002/0003/0004: HTML/MD ingest, metadata, recovery
Add ingestHtml and ingestMarkdown with ADR-0003 pageless offset semantics, PDF intrinsic metadata extraction with caller-wins merge (WP-0003), and citation recovery primitives including re-ingest reconcile, local quote search, and pluggable discovery hooks (ADR-0004). Mark all three workplans finished with contract tests (80 passing).
This commit is contained in:
parent
df6acad828
commit
97437dfa18
28 changed files with 1295 additions and 69 deletions
21
SCOPE.md
21
SCOPE.md
|
|
@ -8,8 +8,8 @@
|
|||
|
||||
## One-liner
|
||||
|
||||
Headless PDF ingest that turns document bytes into engine-owned evidence
|
||||
contracts (fingerprint, canonical text, page/offset maps).
|
||||
Headless document ingest that turns PDF, HTML, and Markdown bytes into
|
||||
engine-owned evidence contracts (fingerprint, canonical text, page/offset maps).
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -18,25 +18,30 @@ contracts (fingerprint, canonical text, page/offset maps).
|
|||
The rest of the citation-evidence ecosystem (anchoring, evidence linking,
|
||||
binders) needs a stable, runtime-agnostic way to go from *raw document bytes*
|
||||
to a `Document` + `DocumentRepresentation`. This repo owns that transformation
|
||||
for PDFs — pure over bytes, no persistence, no viewer, no UI framework.
|
||||
for PDF, HTML, and Markdown — pure over bytes, no persistence, no viewer,
|
||||
no UI framework.
|
||||
|
||||
---
|
||||
|
||||
## In Scope
|
||||
|
||||
- PDF byte ingest → `{ document, representation }` (`ingestPdf`)
|
||||
- HTML byte ingest → `{ document, representation }` (`ingestHtml`)
|
||||
- Markdown byte ingest → `{ document, representation }` (`ingestMarkdown`)
|
||||
- PDF text extraction → canonical text + page map + gap-free offset map (`extractPdf`)
|
||||
- PDF intrinsic metadata extraction with caller-wins merge (`extractPdfMetadata`)
|
||||
- SHA-256 byte fingerprinting (`fingerprintBytes`)
|
||||
- Citation recovery primitives: re-ingest/reconcile, local quote search, discovery hooks
|
||||
- Browser upload helpers behind a separate entry point (`createPdfByteStore`, `ingestPdfFromFile`)
|
||||
|
||||
---
|
||||
|
||||
## Out of Scope
|
||||
|
||||
- HTML / Markdown representations (deferred — see `workplans/`)
|
||||
- Persisting documents or representations (caller's job)
|
||||
- Viewer URL resolution and blob-vs-fixture policy (stays in `citation-evidence`)
|
||||
- Citation recovery / external source discovery (deferred)
|
||||
- Stale selector detection and anchor resolution (stays in `evidence-anchor`)
|
||||
- Network-backed external source discovery implementations (host registers hooks)
|
||||
- Defining domain contracts — those are owned by `citation-engine`
|
||||
|
||||
---
|
||||
|
|
@ -51,7 +56,7 @@ for PDFs — pure over bytes, no persistence, no viewer, no UI framework.
|
|||
|
||||
## Not Relevant When
|
||||
|
||||
- You need HTML/Markdown ingest (not yet implemented)
|
||||
- You need formats beyond PDF/HTML/Markdown (not yet implemented)
|
||||
- You need viewer/session/UI behavior (see `citation-evidence`)
|
||||
- You are changing the `Document`/`DocumentRepresentation` contract (see `citation-engine`)
|
||||
|
||||
|
|
@ -60,7 +65,7 @@ for PDFs — pure over bytes, no persistence, no viewer, no UI framework.
|
|||
## Current State
|
||||
|
||||
- Status: active
|
||||
- Implementation: partial (PDF slice implemented; HTML/MD and recovery deferred)
|
||||
- Implementation: partial (PDF/HTML/MD ingest, metadata enrichment, recovery primitives)
|
||||
- Stability: evolving
|
||||
- Usage: internal (consumed by `citation-evidence` across the repo boundary)
|
||||
|
||||
|
|
@ -92,7 +97,7 @@ for PDFs — pure over bytes, no persistence, no viewer, no UI framework.
|
|||
## Getting Oriented
|
||||
|
||||
- Start with: `README.md`, then `docs/ADR-0001-extraction-boundary.md`
|
||||
- Key files / directories: `src/pdf/` (headless core), `src/browser/` (upload helpers)
|
||||
- Key files / directories: `src/pdf/`, `src/html/`, `src/markdown/`, `src/recovery/`, `src/browser/`
|
||||
- Entry points: `src/index.ts` (headless), `src/browser/index.ts` (browser)
|
||||
|
||||
---
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue