diff --git a/README.md b/README.md index 0f88113..652a780 100644 --- a/README.md +++ b/README.md @@ -1,18 +1,16 @@ # evidence-source -Headless document ingest for the citation-evidence ecosystem. Turns raw PDF -bytes into engine-owned evidence contracts — a `Document` (media type, -SHA-256 fingerprint, optional title/uri/metadata) and a -`DocumentRepresentation` (`pdf-text`: canonical text, page map, gap-free -offset map). Ingest is pure over bytes: no persistence, no viewer state, no -React. +Headless document ingest for the citation-evidence ecosystem. Turns raw PDF, +HTML, and Markdown bytes into engine-owned evidence contracts — a `Document` +(media type, SHA-256 fingerprint, optional title/uri/metadata) and a +`DocumentRepresentation` (canonical text, page/offset maps where applicable). +Ingest is pure over bytes: no persistence, no viewer state, no React. ## Status -**Implemented: the PDF slice.** As of ESRC-WP-0001 this repo hosts the -extracted PDF ingest core that previously lived in `citation-evidence/src/source/`. -HTML/Markdown representations, richer metadata enrichment, and citation -recovery are deferred to follow-on workplans (see `workplans/`). +**Implemented:** PDF ingest (ESRC-WP-0001), HTML/Markdown ingest +(ESRC-WP-0002), PDF metadata enrichment (ESRC-WP-0003), and citation recovery +primitives (ESRC-WP-0004). See `workplans/` for history. ## Install @@ -26,13 +24,17 @@ pnpm install # resolves @citation-evidence/engine via link:../citation-engine ## Usage ```ts -import { ingestPdf } from "@citation-evidence/evidence-source"; +import { + ingestPdf, + ingestHtml, + ingestMarkdown, +} from "@citation-evidence/evidence-source"; -const { document, representation } = await ingestPdf(pdfBytes, { - filename: "contract.pdf", -}); -// document.fingerprint -> SHA-256 hex -// representation.canonicalText / pageMap / offsetMap +const pdf = await ingestPdf(pdfBytes, { filename: "contract.pdf" }); +const html = await ingestHtml(htmlBytes, { filename: "brief.html" }); +const md = await ingestMarkdown("# Title\n\nBody text.", { filename: "notes.md" }); +// document.fingerprint -> SHA-256 hex +// representation.canonicalText / pageMap / offsetMap (format-dependent) ``` Browser upload helpers (in-memory `blob:` byte store + `ingestPdfFromFile`) @@ -61,12 +63,14 @@ Point `EVIDENCE_SOURCE_FIXTURE_DIR` at the PDF corpus when the sibling ## Architecture -- `src/pdf/` — headless core: `ingest`, `extract`, `fingerprint`. Runtime-agnostic. +- `src/pdf/` — PDF ingest, extraction, fingerprinting, metadata enrichment. +- `src/html/`, `src/markdown/` — reflowable-format ingest (ADR-0003). +- `src/recovery/` — re-ingest/reconcile, local quote search, discovery hooks (ADR-0004). - `src/browser/` — browser upload surface: `byte-store`, `upload`. Separated by an ESLint boundary so the core never imports it. - Domain contracts come from `@citation-evidence/engine/shared`; none are copied locally. -- Boundary and fixture decisions: `docs/ADR-0001`, `docs/ADR-0002`. +- Boundary and fixture decisions: `docs/ADR-0001` through `docs/ADR-0004`. `viewer-url` resolution stays in the consuming app (`citation-evidence`), which imports this package's ingest core through a thin façade. diff --git a/SCOPE.md b/SCOPE.md index ca66d5b..b886d7b 100644 --- a/SCOPE.md +++ b/SCOPE.md @@ -8,8 +8,8 @@ ## One-liner -Headless PDF ingest that turns document bytes into engine-owned evidence -contracts (fingerprint, canonical text, page/offset maps). +Headless document ingest that turns PDF, HTML, and Markdown bytes into +engine-owned evidence contracts (fingerprint, canonical text, page/offset maps). --- @@ -18,25 +18,30 @@ contracts (fingerprint, canonical text, page/offset maps). The rest of the citation-evidence ecosystem (anchoring, evidence linking, binders) needs a stable, runtime-agnostic way to go from *raw document bytes* to a `Document` + `DocumentRepresentation`. This repo owns that transformation -for PDFs — pure over bytes, no persistence, no viewer, no UI framework. +for PDF, HTML, and Markdown — pure over bytes, no persistence, no viewer, +no UI framework. --- ## In Scope - PDF byte ingest → `{ document, representation }` (`ingestPdf`) +- HTML byte ingest → `{ document, representation }` (`ingestHtml`) +- Markdown byte ingest → `{ document, representation }` (`ingestMarkdown`) - PDF text extraction → canonical text + page map + gap-free offset map (`extractPdf`) +- PDF intrinsic metadata extraction with caller-wins merge (`extractPdfMetadata`) - SHA-256 byte fingerprinting (`fingerprintBytes`) +- Citation recovery primitives: re-ingest/reconcile, local quote search, discovery hooks - Browser upload helpers behind a separate entry point (`createPdfByteStore`, `ingestPdfFromFile`) --- ## Out of Scope -- HTML / Markdown representations (deferred — see `workplans/`) - Persisting documents or representations (caller's job) - Viewer URL resolution and blob-vs-fixture policy (stays in `citation-evidence`) -- Citation recovery / external source discovery (deferred) +- Stale selector detection and anchor resolution (stays in `evidence-anchor`) +- Network-backed external source discovery implementations (host registers hooks) - Defining domain contracts — those are owned by `citation-engine` --- @@ -51,7 +56,7 @@ for PDFs — pure over bytes, no persistence, no viewer, no UI framework. ## Not Relevant When -- You need HTML/Markdown ingest (not yet implemented) +- You need formats beyond PDF/HTML/Markdown (not yet implemented) - You need viewer/session/UI behavior (see `citation-evidence`) - You are changing the `Document`/`DocumentRepresentation` contract (see `citation-engine`) @@ -60,7 +65,7 @@ for PDFs — pure over bytes, no persistence, no viewer, no UI framework. ## Current State - Status: active -- Implementation: partial (PDF slice implemented; HTML/MD and recovery deferred) +- Implementation: partial (PDF/HTML/MD ingest, metadata enrichment, recovery primitives) - Stability: evolving - Usage: internal (consumed by `citation-evidence` across the repo boundary) @@ -92,7 +97,7 @@ for PDFs — pure over bytes, no persistence, no viewer, no UI framework. ## Getting Oriented - Start with: `README.md`, then `docs/ADR-0001-extraction-boundary.md` -- Key files / directories: `src/pdf/` (headless core), `src/browser/` (upload helpers) +- Key files / directories: `src/pdf/`, `src/html/`, `src/markdown/`, `src/recovery/`, `src/browser/` - Entry points: `src/index.ts` (headless), `src/browser/index.ts` (browser) --- diff --git a/docs/ADR-0003-html-markdown-representation.md b/docs/ADR-0003-html-markdown-representation.md new file mode 100644 index 0000000..f06214b --- /dev/null +++ b/docs/ADR-0003-html-markdown-representation.md @@ -0,0 +1,58 @@ +# ADR-0003 — HTML and Markdown representation contract + +- **Status:** accepted +- **Date:** 2026-07-09 +- **Workplan:** ESRC-WP-0002 (T01) + +## Context + +ESRC-WP-0001 established the PDF ingest boundary. ESRC-WP-0002 extends ingest to +HTML and Markdown while reusing engine-owned contracts from +`@citation-evidence/engine/shared`. + +`RepresentationType` already reserves `html-dom` and `markdown-rendered` in +`citation-engine`; no engine schema change is required for this slice. + +## Decision + +### Representation types + +| Source format | `mediaType` | `representationType` | +| ------------- | ------------------ | ---------------------- | +| HTML | `text/html` | `html-dom` | +| Markdown | `text/markdown` | `markdown-rendered` | + +### Pageless offset semantics + +Reflowable formats have no fixed pages: + +- `pageMap` is **omitted** (undefined). +- `offsetMap` is a **single synthetic range** on page 1 covering + `[0, canonicalText.length)` with no gaps. This keeps + `TextPositionSelector` usable without inventing physical page geometry. +- `structureMap` remains `never` in the engine contract and is not populated. + +### Canonical text + +Both pipelines decode source bytes as UTF-8, derive plain text from the format, +then apply `normalize()` from `@citation-evidence/engine/shared`. HTML strips +active content (`script`, `style`, `iframe`, `object`, `embed`, `noscript`) +before text extraction. + +### Content hash + +`contentHash` is the SHA-256 fingerprint of the **source bytes** (same rule as +PDF ingest), not a hash of canonical text. + +### Fixtures + +HTML/Markdown contract tests use small inline fixtures owned by this repo. They +contain no PII and do not duplicate the upstream PDF corpus (ADR-0002). + +## Consequences + +- `ingestHtml` and `ingestMarkdown` mirror the PDF `{ document, representation }` + shape. +- Anchoring layers may add DOM/structural selectors later; this slice delivers + canonical text and position offsets only. +- Sanitization is extraction-time stripping, not a sandboxed renderer. \ No newline at end of file diff --git a/docs/ADR-0004-citation-recovery-boundary.md b/docs/ADR-0004-citation-recovery-boundary.md new file mode 100644 index 0000000..7bbb980 --- /dev/null +++ b/docs/ADR-0004-citation-recovery-boundary.md @@ -0,0 +1,54 @@ +# ADR-0004 — Citation recovery boundary + +- **Status:** accepted +- **Date:** 2026-07-09 +- **Workplan:** ESRC-WP-0004 (T01) + +## Context + +`evidence-source` owns ingestion and representation generation. Citation +recovery spans stale-selector detection, local quote search, external source +discovery, and human confirmation — but selector resolution algorithms live in +`evidence-anchor`, and the `CitationRecoveryAttempt` type vocabulary lives in +`citation-engine`. + +## Decision + +### Owned by `evidence-source` + +| Capability | Module | Notes | +| ---------- | ------ | ----- | +| Re-ingest + re-fingerprint | `src/recovery/re-ingest.ts` | Compare prior vs. fresh `Document.fingerprint`; flag `requiresReanchor` | +| Local canonical quote search | `src/recovery/local-search.ts` | Search `DocumentRepresentation.canonicalText` | +| Recovery attempt scaffold | `src/recovery/attempt.ts` | Local record using SharedContracts state vocabulary | +| Discovery hook interface | `src/recovery/discovery.ts` | Pluggable providers; **no network in core** | + +### Owned elsewhere + +| Capability | Owner | +| ---------- | ----- | +| Stale selector detection / resolution order | `evidence-anchor` | +| `CitationRecoveryAttempt` canonical type | `citation-engine` (future) | +| Human confirmation UX | `citation-work` / umbrella | +| External HTTP/API lookup implementations | Deployment-specific providers registered on the hook | + +### Recovery flow + +```text +CitationClue + → createRecoveryAttempt (evidence-source) + → local library search via searchCanonicalQuote (evidence-source) + → optional SourceDiscoveryHook providers (registered by host) + → reIngestAndCompare when fresh bytes arrive (evidence-source) + → anchor layer re-resolves selectors when requiresReanchor is true +``` + +`evidence-source` **does not** decide whether a selector is stale. It supplies +primitives the anchor layer calls after it detects mismatch. + +## Consequences + +- No network coupling in the recovery core. +- Host apps register discovery providers explicitly (local-first default). +- Recovery state enum mirrors `wiki/SharedContracts.md` §2.6 locally until the + engine exports a shared `CitationRecoveryAttempt` type. \ No newline at end of file diff --git a/src/html/extract.ts b/src/html/extract.ts new file mode 100644 index 0000000..887cb00 --- /dev/null +++ b/src/html/extract.ts @@ -0,0 +1,58 @@ +/** + * HTML text extraction → canonical text + gap-free offset map. + * + * Implements ADR-0003 for the `html-dom` representation. Active content is + * stripped before text extraction; canonical text is produced via `normalize()`. + */ + +import { normalize, type OffsetMap } from "@citation-evidence/engine/shared"; +import { buildWholeTextOffsetMap } from "../shared/offset-map"; + +const ACTIVE_CONTENT = + /<(script|style|iframe|object|embed|noscript)\b[^>]*>[\s\S]*?<\/\1>/gi; +const BLOCK_BREAK = /<\/(p|div|h[1-6]|li|tr|section|article|blockquote|pre)>/gi; +const BR_TAG = //gi; +const TAG = /<[^>]+>/g; + +export interface HtmlExtractionResult { + readonly canonicalText: string; + readonly sanitizedHtml: string; + readonly offsetMap: OffsetMap; +} + +export function extractHtml(source: string): HtmlExtractionResult { + const sanitizedHtml = sanitizeHtml(source); + const rawText = htmlToPlainText(sanitizedHtml); + const canonicalText = normalize(rawText).text; + return { + canonicalText, + sanitizedHtml, + offsetMap: buildWholeTextOffsetMap(canonicalText), + }; +} + +export function sanitizeHtml(source: string): string { + return source.replace(ACTIVE_CONTENT, ""); +} + +function htmlToPlainText(html: string): string { + let text = html.replace(BR_TAG, "\n").replace(BLOCK_BREAK, "\n\n"); + text = text.replace(TAG, ""); + return decodeHtmlEntities(text); +} + +function decodeHtmlEntities(text: string): string { + return text + .replace(/ /gi, " ") + .replace(/&/gi, "&") + .replace(/</gi, "<") + .replace(/>/gi, ">") + .replace(/"/gi, '"') + .replace(/'/g, "'") + .replace(/&#x([0-9a-f]+);/gi, (_, hex: string) => + String.fromCodePoint(Number.parseInt(hex, 16)), + ) + .replace(/&#(\d+);/g, (_, dec: string) => + String.fromCodePoint(Number.parseInt(dec, 10)), + ); +} \ No newline at end of file diff --git a/src/html/ingest.test.ts b/src/html/ingest.test.ts new file mode 100644 index 0000000..83fdd30 --- /dev/null +++ b/src/html/ingest.test.ts @@ -0,0 +1,54 @@ +import { describe, expect, it } from "vitest"; + +import { fingerprintBytes } from "../pdf/fingerprint"; +import { HTML_FIXTURE } from "../../tests/fixtures-text"; +import { ingestHtml } from "./ingest"; +import { extractHtml, sanitizeHtml } from "./extract"; + +describe("extractHtml", () => { + it("strips active content before text extraction", () => { + const { sanitizedHtml } = extractHtml(HTML_FIXTURE); + expect(sanitizeHtml(HTML_FIXTURE)).not.toMatch(/ + +`; + +export const MARKDOWN_FIXTURE = `# Hello World + +Known good quote for Markdown ingest testing. + +**bold** and _italic_ text. +`; + +export const TEXT_FIXTURES: readonly TextFixture[] = [ + { + id: "html-basic", + knownGoodQuote: "Known good quote for HTML ingest testing.", + }, + { + id: "markdown-basic", + knownGoodQuote: "Known good quote for Markdown ingest testing.", + }, +]; \ No newline at end of file diff --git a/workplans/ESRC-WP-0002-html-markdown-representations.md b/workplans/ESRC-WP-0002-html-markdown-representations.md index c156b2d..cd7e2a2 100644 --- a/workplans/ESRC-WP-0002-html-markdown-representations.md +++ b/workplans/ESRC-WP-0002-html-markdown-representations.md @@ -4,14 +4,15 @@ type: workplan title: "HTML and Markdown source representations" domain: infotech repo: evidence-source -status: proposed +status: finished owner: codex topic_slug: citation_evidence_mvp created: "2026-07-08" -updated: "2026-07-08" +updated: "2026-07-09" spec_refs: - README.md - docs/ADR-0001-extraction-boundary.md + - docs/ADR-0003-html-markdown-representation.md - ../citation-engine/src/shared/document.ts state_hub_workstream_id: "c0d28886-e02f-4df2-abb6-ebe179171c94" --- @@ -22,18 +23,19 @@ The PDF slice (ESRC-WP-0001) established the ingest boundary and contract shape. This workplan extends ingest to HTML and Markdown sources, producing the same `{ document, representation }` pair over engine-owned contracts. -## Open questions +## Decisions (ADR-0003) -- Which `representationType` values do HTML/MD map to, and do they need new - `DocumentRepresentation` fields in `citation-engine`? -- How is canonical text + offset map derived for reflowable formats that have - no fixed pages? (Page map may be empty or synthetic.) +- HTML maps to `representationType: "html-dom"`; Markdown to + `"markdown-rendered"`. +- Pageless formats omit `pageMap`; `offsetMap` is a single synthetic page-1 + range covering `[0, canonicalText.length)`. +- Inline fixtures owned by this repo (no PII); PDF corpus rules unchanged. ## Sketch ```task id: ESRC-WP-0002-T01 -status: todo +status: done priority: medium state_hub_task_id: "bf1d2a2f-41b4-44b5-b1d1-4f0eec5fbf59" ``` @@ -42,7 +44,7 @@ Define the HTML/MD representation contract with `citation-engine`; agree ```task id: ESRC-WP-0002-T02 -status: todo +status: done priority: medium state_hub_task_id: "66320911-a4b2-404f-b025-8b23dc0c9680" ``` @@ -51,9 +53,9 @@ reusing `fingerprintBytes` and the canonical-text normalizer. ```task id: ESRC-WP-0002-T03 -status: todo +status: done priority: medium state_hub_task_id: "266d168c-d93e-4387-9527-70aa0d5b2a6b" ``` Add fixture-driven contract tests mirroring the PDF suite; decide fixture -ownership per ADR-0002. +ownership per ADR-0002. \ No newline at end of file diff --git a/workplans/ESRC-WP-0003-metadata-enrichment.md b/workplans/ESRC-WP-0003-metadata-enrichment.md index 12bbc39..4232327 100644 --- a/workplans/ESRC-WP-0003-metadata-enrichment.md +++ b/workplans/ESRC-WP-0003-metadata-enrichment.md @@ -4,14 +4,15 @@ type: workplan title: "Document metadata enrichment beyond pass-through options" domain: infotech repo: evidence-source -status: proposed +status: finished owner: codex topic_slug: citation_evidence_mvp created: "2026-07-08" -updated: "2026-07-08" +updated: "2026-07-09" spec_refs: - README.md - src/pdf/ingest.ts + - src/pdf/metadata.ts state_hub_workstream_id: "831267a1-fc93-4872-a25b-73793d9c5df4" --- @@ -26,7 +27,7 @@ it into the `Document` record without breaking the pure-over-bytes contract. ```task id: ESRC-WP-0003-T01 -status: todo +status: done priority: low state_hub_task_id: "45eeef15-41a9-4abd-9436-ed77e2c6b3a4" ``` @@ -35,7 +36,7 @@ precedence vs. caller-supplied options (caller wins). ```task id: ESRC-WP-0003-T02 -status: todo +status: done priority: low state_hub_task_id: "41ca090f-c709-420a-b7c5-8332e4e82bd1" ``` @@ -44,9 +45,9 @@ minimal-metadata PDFs still ingest cleanly. ```task id: ESRC-WP-0003-T03 -status: todo +status: done priority: low state_hub_task_id: "0004881b-6577-4a62-a2ff-cbb5c4706b1b" ``` Contract tests over the fixture corpus asserting extracted vs. overridden -metadata precedence. +metadata precedence. \ No newline at end of file diff --git a/workplans/ESRC-WP-0004-citation-recovery.md b/workplans/ESRC-WP-0004-citation-recovery.md index 4f32ae1..f324872 100644 --- a/workplans/ESRC-WP-0004-citation-recovery.md +++ b/workplans/ESRC-WP-0004-citation-recovery.md @@ -4,14 +4,15 @@ type: workplan title: "Local citation recovery and external source discovery hooks" domain: infotech repo: evidence-source -status: proposed +status: finished owner: codex topic_slug: citation_evidence_mvp created: "2026-07-08" -updated: "2026-07-08" +updated: "2026-07-09" spec_refs: - INTENT.md - README.md + - docs/ADR-0004-citation-recovery-boundary.md state_hub_workstream_id: "2ce8b799-d67c-4dbf-aac5-0ff1f0363d62" --- @@ -22,18 +23,17 @@ selector goes stale and discovering the external source a document was drawn from. This is the largest deferred strand and depends on the anchoring layer's selector model. -## Open questions +## Decisions (ADR-0004) -- What is the recovery contract — does this repo *detect* stale selectors, or - only provide the re-ingest/re-fingerprint primitives the anchor layer calls? -- Where does external source discovery (URL/DOI resolution) belong relative to - `citation-engine` and the umbrella? +- `evidence-source` owns re-ingest/re-fingerprint, local quote search, and a + pluggable discovery hook registry (no network in core). +- Stale selector detection and resolution order stay in `evidence-anchor`. ## Sketch ```task id: ESRC-WP-0004-T01 -status: todo +status: done priority: low state_hub_task_id: "e7ff29b9-f04f-44f5-a23a-b028fc20a80b" ``` @@ -42,9 +42,9 @@ Design the recovery boundary with the anchor layer: define what ```task id: ESRC-WP-0004-T02 -status: todo +status: done priority: low state_hub_task_id: "88a512d1-70e0-4bfa-b6b0-c1e4e31abb97" ``` Prototype re-ingest/re-fingerprint recovery primitives and a discovery hook -interface (no network coupling baked into the core). +interface (no network coupling baked into the core). \ No newline at end of file