tegwick 97437dfa18

CI Smoke / host-smoke (push) Successful in 0s

Details

CI Smoke / container-smoke (push) Failing after 15m18s

Details

Implement ESRC-WP-0002/0003/0004: HTML/MD ingest, metadata, recovery

Add ingestHtml and ingestMarkdown with ADR-0003 pageless offset semantics,
PDF intrinsic metadata extraction with caller-wins merge (WP-0003), and
citation recovery primitives including re-ingest reconcile, local quote
search, and pluggable discovery hooks (ADR-0004). Mark all three workplans
finished with contract tests (80 passing).

2026-07-09 01:48:28 +02:00

3.7 KiB

Raw Permalink Blame History

SCOPE

This file helps you quickly understand what this repository is about, when it is relevant, and when it is not. It is intentionally lightweight and may be incomplete.

One-liner

Headless document ingest that turns PDF, HTML, and Markdown bytes into engine-owned evidence contracts (fingerprint, canonical text, page/offset maps).

Core Idea

The rest of the citation-evidence ecosystem (anchoring, evidence linking, binders) needs a stable, runtime-agnostic way to go from raw document bytes to a Document + DocumentRepresentation. This repo owns that transformation for PDF, HTML, and Markdown — pure over bytes, no persistence, no viewer, no UI framework.

In Scope

PDF byte ingest → { document, representation } (ingestPdf)
HTML byte ingest → { document, representation } (ingestHtml)
Markdown byte ingest → { document, representation } (ingestMarkdown)
PDF text extraction → canonical text + page map + gap-free offset map (extractPdf)
PDF intrinsic metadata extraction with caller-wins merge (extractPdfMetadata)
SHA-256 byte fingerprinting (fingerprintBytes)
Citation recovery primitives: re-ingest/reconcile, local quote search, discovery hooks
Browser upload helpers behind a separate entry point (createPdfByteStore, ingestPdfFromFile)

Out of Scope

Persisting documents or representations (caller's job)
Viewer URL resolution and blob-vs-fixture policy (stays in citation-evidence)
Stale selector detection and anchor resolution (stays in evidence-anchor)
Network-backed external source discovery implementations (host registers hooks)
Defining domain contracts — those are owned by citation-engine

Relevant When

You have PDF bytes and need engine-shaped evidence contracts
You need a deterministic content fingerprint of document bytes
You are wiring a new consumer to ingest without pulling in a viewer

Not Relevant When

You need formats beyond PDF/HTML/Markdown (not yet implemented)
You need viewer/session/UI behavior (see citation-evidence)
You are changing the Document/DocumentRepresentation contract (see citation-engine)

Current State

Status: active
Implementation: partial (PDF/HTML/MD ingest, metadata enrichment, recovery primitives)
Stability: evolving
Usage: internal (consumed by citation-evidence across the repo boundary)

How It Fits

Upstream dependencies: citation-engine (@citation-evidence/engine/shared)
Downstream consumers: citation-evidence (umbrella app)
Often used with: citation-engine, citation-evidence

Terminology

Preferred terms: ingest, canonical text, fingerprint, representation
Also known as: "source" (its former location was citation-evidence/src/source/)
Potentially confusing terms: "representation" (a derived view of a document, not the document itself)

citation-engine — owns the shared Document/DocumentRepresentation contracts this repo produces
citation-evidence — umbrella app that consumes this ingest core; still owns viewer-url resolution

Getting Oriented

Start with: README.md, then docs/ADR-0001-extraction-boundary.md
Key files / directories: src/pdf/, src/html/, src/markdown/, src/recovery/, src/browser/
Entry points: src/index.ts (headless), src/browser/index.ts (browser)

Provided Capabilities

capability.infotech.pdf-evidence-ingest — headless PDF → evidence contracts (see registry/capabilities/)

Notes

Fixture corpus is owned upstream by citation-evidence and resolved from a sibling checkout (ADR-0002); it is not duplicated here.

3.7 KiB Raw Permalink Blame History