Implement ESRC-WP-0002/0003/0004: HTML/MD ingest, metadata, recovery
Some checks failed
CI Smoke / host-smoke (push) Successful in 0s
CI Smoke / container-smoke (push) Failing after 15m18s

Add ingestHtml and ingestMarkdown with ADR-0003 pageless offset semantics,
PDF intrinsic metadata extraction with caller-wins merge (WP-0003), and
citation recovery primitives including re-ingest reconcile, local quote
search, and pluggable discovery hooks (ADR-0004). Mark all three workplans
finished with contract tests (80 passing).
This commit is contained in:
tegwick 2026-07-09 01:48:28 +02:00
parent df6acad828
commit 97437dfa18
28 changed files with 1295 additions and 69 deletions

View file

@ -1,18 +1,16 @@
# evidence-source
Headless document ingest for the citation-evidence ecosystem. Turns raw PDF
bytes into engine-owned evidence contracts — a `Document` (media type,
SHA-256 fingerprint, optional title/uri/metadata) and a
`DocumentRepresentation` (`pdf-text`: canonical text, page map, gap-free
offset map). Ingest is pure over bytes: no persistence, no viewer state, no
React.
Headless document ingest for the citation-evidence ecosystem. Turns raw PDF,
HTML, and Markdown bytes into engine-owned evidence contracts — a `Document`
(media type, SHA-256 fingerprint, optional title/uri/metadata) and a
`DocumentRepresentation` (canonical text, page/offset maps where applicable).
Ingest is pure over bytes: no persistence, no viewer state, no React.
## Status
**Implemented: the PDF slice.** As of ESRC-WP-0001 this repo hosts the
extracted PDF ingest core that previously lived in `citation-evidence/src/source/`.
HTML/Markdown representations, richer metadata enrichment, and citation
recovery are deferred to follow-on workplans (see `workplans/`).
**Implemented:** PDF ingest (ESRC-WP-0001), HTML/Markdown ingest
(ESRC-WP-0002), PDF metadata enrichment (ESRC-WP-0003), and citation recovery
primitives (ESRC-WP-0004). See `workplans/` for history.
## Install
@ -26,13 +24,17 @@ pnpm install # resolves @citation-evidence/engine via link:../citation-engine
## Usage
```ts
import { ingestPdf } from "@citation-evidence/evidence-source";
import {
ingestPdf,
ingestHtml,
ingestMarkdown,
} from "@citation-evidence/evidence-source";
const { document, representation } = await ingestPdf(pdfBytes, {
filename: "contract.pdf",
});
// document.fingerprint -> SHA-256 hex
// representation.canonicalText / pageMap / offsetMap
const pdf = await ingestPdf(pdfBytes, { filename: "contract.pdf" });
const html = await ingestHtml(htmlBytes, { filename: "brief.html" });
const md = await ingestMarkdown("# Title\n\nBody text.", { filename: "notes.md" });
// document.fingerprint -> SHA-256 hex
// representation.canonicalText / pageMap / offsetMap (format-dependent)
```
Browser upload helpers (in-memory `blob:` byte store + `ingestPdfFromFile`)
@ -61,12 +63,14 @@ Point `EVIDENCE_SOURCE_FIXTURE_DIR` at the PDF corpus when the sibling
## Architecture
- `src/pdf/` — headless core: `ingest`, `extract`, `fingerprint`. Runtime-agnostic.
- `src/pdf/` — PDF ingest, extraction, fingerprinting, metadata enrichment.
- `src/html/`, `src/markdown/` — reflowable-format ingest (ADR-0003).
- `src/recovery/` — re-ingest/reconcile, local quote search, discovery hooks (ADR-0004).
- `src/browser/` — browser upload surface: `byte-store`, `upload`. Separated by
an ESLint boundary so the core never imports it.
- Domain contracts come from `@citation-evidence/engine/shared`; none are
copied locally.
- Boundary and fixture decisions: `docs/ADR-0001`, `docs/ADR-0002`.
- Boundary and fixture decisions: `docs/ADR-0001` through `docs/ADR-0004`.
`viewer-url` resolution stays in the consuming app (`citation-evidence`), which
imports this package's ingest core through a thin façade.