feat(pdf): extract standalone PDF ingest package (ESRC-WP-0001)
Bootstrap evidence-source from the citation-evidence src/source PDF slice. - Headless core (src/pdf): ingest/extract/fingerprint, importing domain contracts from @citation-evidence/engine/shared (no local copies) - Browser upload helpers isolated under src/browser behind a ./browser entry point, with an eslint boundary keeping the core browser-free - pnpm/TS/vitest/eslint scaffold; 52 tests (contract + determinism) - Fixtures resolved from the sibling citation-evidence checkout, not duplicated (real PII) — see docs/ADR-0002; suites skip when absent - Boundary + fixture decisions recorded as docs/ADR-0001 / ADR-0002 - README/SCOPE rewritten; capability.infotech.pdf-evidence-ingest registered, NO_CAPABILITIES removed - Follow-on workplans ESRC-WP-0002..0004 queued; ESRC-WP-0001 finished Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
parent
2fd715ba45
commit
cb93c322c0
31 changed files with 5066 additions and 105 deletions
|
|
@ -1,4 +1,4 @@
|
||||||
**Purpose:** Document ingestion, extraction, fingerprinting, citation recovery. Depends only on citation-engine. INTENT-only during umbrella-first MVP.
|
**Purpose:** Document ingestion, extraction, fingerprinting, citation recovery. Depends only on citation-engine. The PDF ingest slice is implemented and consumed by citation-evidence (ESRC-WP-0001); HTML/MD, metadata enrichment, and citation recovery are deferred to ESRC-WP-0002..0004.
|
||||||
|
|
||||||
**Domain:** infotech
|
**Domain:** infotech
|
||||||
**Repo slug:** evidence-source
|
**Repo slug:** evidence-source
|
||||||
|
|
|
||||||
6
.gitignore
vendored
6
.gitignore
vendored
|
|
@ -174,3 +174,9 @@ cython_debug/
|
||||||
# PyPI configuration file
|
# PyPI configuration file
|
||||||
.pypirc
|
.pypirc
|
||||||
|
|
||||||
|
|
||||||
|
# ---> Node / TypeScript
|
||||||
|
node_modules/
|
||||||
|
coverage/
|
||||||
|
*.tsbuildinfo
|
||||||
|
.vitest/
|
||||||
|
|
|
||||||
78
README.md
78
README.md
|
|
@ -1,16 +1,72 @@
|
||||||
# evidence-source
|
# evidence-source
|
||||||
|
|
||||||
Document source, ingestion, extraction, metadata, and citation recovery —
|
Headless document ingest for the citation-evidence ecosystem. Turns raw PDF
|
||||||
PDF/HTML/MD ingest, fingerprinting, page-/offset-map construction,
|
bytes into engine-owned evidence contracts — a `Document` (media type,
|
||||||
canonical-text extraction, and the recovery behavior for stale selectors.
|
SHA-256 fingerprint, optional title/uri/metadata) and a
|
||||||
|
`DocumentRepresentation` (`pdf-text`: canonical text, page map, gap-free
|
||||||
|
offset map). Ingest is pure over bytes: no persistence, no viewer state, no
|
||||||
|
React.
|
||||||
|
|
||||||
## MVP status: INTENT only
|
## Status
|
||||||
|
|
||||||
During the citation-evidence MVP, code lives upstream in
|
**Implemented: the PDF slice.** As of ESRC-WP-0001 this repo hosts the
|
||||||
[`citation-evidence`](../citation-evidence/) under `src/source/`. This repo
|
extracted PDF ingest core that previously lived in `citation-evidence/src/source/`.
|
||||||
currently holds `INTENT.md` describing what will move here. Contract
|
HTML/Markdown representations, richer metadata enrichment, and citation
|
||||||
changes belong in
|
recovery are deferred to follow-on workplans (see `workplans/`).
|
||||||
[`citation-evidence/wiki/SharedContracts.md`](../citation-evidence/wiki/SharedContracts.md),
|
|
||||||
not here.
|
|
||||||
|
|
||||||
Per the dependency map, source depends on `shared/` and `engine/` only.
|
## Install
|
||||||
|
|
||||||
|
Requires a sibling checkout of `citation-engine` (domain contracts) and, for
|
||||||
|
tests, `citation-evidence` (fixture corpus — see `docs/ADR-0002-fixture-ownership.md`).
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pnpm install # resolves @citation-evidence/engine via link:../citation-engine
|
||||||
|
```
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
|
||||||
|
```ts
|
||||||
|
import { ingestPdf } from "@citation-evidence/evidence-source";
|
||||||
|
|
||||||
|
const { document, representation } = await ingestPdf(pdfBytes, {
|
||||||
|
filename: "contract.pdf",
|
||||||
|
});
|
||||||
|
// document.fingerprint -> SHA-256 hex
|
||||||
|
// representation.canonicalText / pageMap / offsetMap
|
||||||
|
```
|
||||||
|
|
||||||
|
Browser upload helpers (in-memory `blob:` byte store + `ingestPdfFromFile`)
|
||||||
|
live behind a separate, clearly-isolated entry point so headless consumers do
|
||||||
|
not pull in browser machinery:
|
||||||
|
|
||||||
|
```ts
|
||||||
|
import { createPdfByteStore, ingestPdfFromFile }
|
||||||
|
from "@citation-evidence/evidence-source/browser";
|
||||||
|
```
|
||||||
|
|
||||||
|
Extraction requires the host to configure the PDF.js worker
|
||||||
|
(`GlobalWorkerOptions.workerSrc`) before calling `extractPdf`/`ingestPdf`; the
|
||||||
|
module itself does no worker setup so it loads cleanly in Node and browsers.
|
||||||
|
|
||||||
|
## Dev commands
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pnpm test # vitest (fixture suites skip if the corpus is absent)
|
||||||
|
pnpm typecheck # tsc --noEmit
|
||||||
|
pnpm lint # eslint
|
||||||
|
```
|
||||||
|
|
||||||
|
Point `EVIDENCE_SOURCE_FIXTURE_DIR` at the PDF corpus when the sibling
|
||||||
|
`citation-evidence` checkout is not at `../citation-evidence`.
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
- `src/pdf/` — headless core: `ingest`, `extract`, `fingerprint`. Runtime-agnostic.
|
||||||
|
- `src/browser/` — browser upload surface: `byte-store`, `upload`. Separated by
|
||||||
|
an ESLint boundary so the core never imports it.
|
||||||
|
- Domain contracts come from `@citation-evidence/engine/shared`; none are
|
||||||
|
copied locally.
|
||||||
|
- Boundary and fixture decisions: `docs/ADR-0001`, `docs/ADR-0002`.
|
||||||
|
|
||||||
|
`viewer-url` resolution stays in the consuming app (`citation-evidence`), which
|
||||||
|
imports this package's ingest core through a thin façade.
|
||||||
|
|
|
||||||
107
SCOPE.md
107
SCOPE.md
|
|
@ -8,130 +8,103 @@
|
||||||
|
|
||||||
## One-liner
|
## One-liner
|
||||||
|
|
||||||
<!-- Describe the purpose of this repository in one precise sentence. -->
|
Headless PDF ingest that turns document bytes into engine-owned evidence
|
||||||
<!-- Example: "Provides a lightweight event router for Kubernetes-native systems." -->
|
contracts (fingerprint, canonical text, page/offset maps).
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Core Idea
|
## Core Idea
|
||||||
|
|
||||||
<!-- What is the main capability or idea behind this repository? -->
|
The rest of the citation-evidence ecosystem (anchoring, evidence linking,
|
||||||
<!-- What problem does it try to solve? -->
|
binders) needs a stable, runtime-agnostic way to go from *raw document bytes*
|
||||||
|
to a `Document` + `DocumentRepresentation`. This repo owns that transformation
|
||||||
|
for PDFs — pure over bytes, no persistence, no viewer, no UI framework.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## In Scope
|
## In Scope
|
||||||
|
|
||||||
<!-- What this repository is responsible for. -->
|
- PDF byte ingest → `{ document, representation }` (`ingestPdf`)
|
||||||
<!-- Be explicit and concrete. -->
|
- PDF text extraction → canonical text + page map + gap-free offset map (`extractPdf`)
|
||||||
|
- SHA-256 byte fingerprinting (`fingerprintBytes`)
|
||||||
-
|
- Browser upload helpers behind a separate entry point (`createPdfByteStore`, `ingestPdfFromFile`)
|
||||||
-
|
|
||||||
-
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Out of Scope
|
## Out of Scope
|
||||||
|
|
||||||
<!-- What this repository deliberately does NOT do. -->
|
- HTML / Markdown representations (deferred — see `workplans/`)
|
||||||
<!-- This is often more important than "In Scope". -->
|
- Persisting documents or representations (caller's job)
|
||||||
|
- Viewer URL resolution and blob-vs-fixture policy (stays in `citation-evidence`)
|
||||||
-
|
- Citation recovery / external source discovery (deferred)
|
||||||
-
|
- Defining domain contracts — those are owned by `citation-engine`
|
||||||
-
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Relevant When
|
## Relevant When
|
||||||
|
|
||||||
<!-- When should someone consider using or exploring this repository? -->
|
- You have PDF bytes and need engine-shaped evidence contracts
|
||||||
|
- You need a deterministic content fingerprint of document bytes
|
||||||
-
|
- You are wiring a new consumer to ingest without pulling in a viewer
|
||||||
-
|
|
||||||
-
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Not Relevant When
|
## Not Relevant When
|
||||||
|
|
||||||
<!-- When should someone ignore this repository? -->
|
- You need HTML/Markdown ingest (not yet implemented)
|
||||||
|
- You need viewer/session/UI behavior (see `citation-evidence`)
|
||||||
-
|
- You are changing the `Document`/`DocumentRepresentation` contract (see `citation-engine`)
|
||||||
-
|
|
||||||
-
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Current State
|
## Current State
|
||||||
|
|
||||||
<!-- Rough indication of maturity. No strict format required. -->
|
- Status: active
|
||||||
|
- Implementation: partial (PDF slice implemented; HTML/MD and recovery deferred)
|
||||||
- Status: <!-- e.g. concept / experimental / active / stable / deprecated -->
|
- Stability: evolving
|
||||||
- Implementation: <!-- e.g. idea / partial / substantial / complete -->
|
- Usage: internal (consumed by `citation-evidence` across the repo boundary)
|
||||||
- Stability: <!-- e.g. unstable / evolving / stable -->
|
|
||||||
- Usage: <!-- e.g. none / personal / internal / production -->
|
|
||||||
|
|
||||||
<!-- Add any notes that help set expectations. -->
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## How It Fits
|
## How It Fits
|
||||||
|
|
||||||
<!-- Where does this repository sit in the bigger picture? -->
|
- Upstream dependencies: `citation-engine` (`@citation-evidence/engine/shared`)
|
||||||
|
- Downstream consumers: `citation-evidence` (umbrella app)
|
||||||
- Upstream dependencies:
|
- Often used with: `citation-engine`, `citation-evidence`
|
||||||
- Downstream consumers:
|
|
||||||
- Often used with:
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Terminology
|
## Terminology
|
||||||
|
|
||||||
<!-- Terms that are important to understand this repo. -->
|
- Preferred terms: ingest, canonical text, fingerprint, representation
|
||||||
<!-- Especially useful if naming differs from other repos. -->
|
- Also known as: "source" (its former location was `citation-evidence/src/source/`)
|
||||||
|
- Potentially confusing terms: "representation" (a derived view of a document, not the document itself)
|
||||||
- Preferred terms:
|
|
||||||
- Also known as:
|
|
||||||
- Potentially confusing terms:
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Related / Overlapping Repositories
|
## Related / Overlapping Repositories
|
||||||
|
|
||||||
<!-- List repositories that have similar or adjacent responsibilities. -->
|
- `citation-engine` — owns the shared `Document`/`DocumentRepresentation` contracts this repo produces
|
||||||
<!-- Helps detect duplication and navigate the ecosystem. -->
|
- `citation-evidence` — umbrella app that consumes this ingest core; still owns viewer-url resolution
|
||||||
|
|
||||||
- <repo-name> — <!-- how it relates -->
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Getting Oriented
|
## Getting Oriented
|
||||||
|
|
||||||
<!-- If someone decides to look deeper, where should they start? -->
|
- Start with: `README.md`, then `docs/ADR-0001-extraction-boundary.md`
|
||||||
|
- Key files / directories: `src/pdf/` (headless core), `src/browser/` (upload helpers)
|
||||||
- Start with:
|
- Entry points: `src/index.ts` (headless), `src/browser/index.ts` (browser)
|
||||||
- Key files / directories:
|
|
||||||
- Entry points:
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Provided Capabilities
|
## Provided Capabilities
|
||||||
|
|
||||||
<!-- What can this repo's domain provide to other domains on request? -->
|
- `capability.infotech.pdf-evidence-ingest` — headless PDF → evidence contracts
|
||||||
<!-- Each capability block is parsed by the state-hub capability catalog ingest. -->
|
(see `registry/capabilities/`)
|
||||||
<!-- Remove the examples and add your own, or leave empty if none. -->
|
|
||||||
|
|
||||||
<!--
|
|
||||||
```capability
|
|
||||||
type: infrastructure
|
|
||||||
title: Example capability title
|
|
||||||
description: What this capability provides, in one or two sentences.
|
|
||||||
keywords: [keyword1, keyword2, keyword3]
|
|
||||||
```
|
|
||||||
-->
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Notes
|
## Notes
|
||||||
|
|
||||||
<!-- Anything else worth knowing. Keep it short. -->
|
Fixture corpus is owned upstream by `citation-evidence` and resolved from a
|
||||||
|
sibling checkout (ADR-0002); it is not duplicated here.
|
||||||
|
|
|
||||||
78
docs/ADR-0001-extraction-boundary.md
Normal file
78
docs/ADR-0001-extraction-boundary.md
Normal file
|
|
@ -0,0 +1,78 @@
|
||||||
|
# ADR-0001 — PDF extraction boundary for `evidence-source`
|
||||||
|
|
||||||
|
- **Status:** accepted
|
||||||
|
- **Date:** 2026-07-08
|
||||||
|
- **Workplan:** ESRC-WP-0001 (T01)
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
`evidence-source` is being bootstrapped from the PDF slice that currently
|
||||||
|
lives in `../citation-evidence/src/source/`. That slice already implements a
|
||||||
|
working, tested headless ingest pipeline plus a set of browser/session helpers.
|
||||||
|
This ADR locks *which* files cross the boundary, *where* they land, and *which
|
||||||
|
contracts* they may depend on, so the extraction is mechanical rather than a
|
||||||
|
series of ad-hoc judgement calls.
|
||||||
|
|
||||||
|
The shared domain contracts (`Document`, `DocumentRepresentation`, `PageMap`,
|
||||||
|
`OffsetMap`, `newId`, `normalize`, …) are owned by `citation-engine` and are
|
||||||
|
published from its package as `@citation-evidence/engine/shared`. Nothing in
|
||||||
|
this extraction copies those types locally.
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
### In-scope — headless ingest core (`src/pdf/`)
|
||||||
|
|
||||||
|
Moved into this repo immediately; these are pure, runtime-agnostic, and the
|
||||||
|
reason this repo exists:
|
||||||
|
|
||||||
|
| Upstream file | New home | Notes |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `src/source/pdf/ingest.ts` | `src/pdf/ingest.ts` | `ingestPdf` |
|
||||||
|
| `src/source/pdf/extract.ts` | `src/pdf/extract.ts` | `extractPdf` |
|
||||||
|
| `src/source/pdf/fingerprint.ts` | `src/pdf/fingerprint.ts` | `fingerprintBytes` |
|
||||||
|
|
||||||
|
The only change to these files is the import specifier: `@shared/*` becomes the
|
||||||
|
published package subpath `@citation-evidence/engine/shared`.
|
||||||
|
|
||||||
|
### In-scope but explicitly isolated — browser helpers (`src/browser/`)
|
||||||
|
|
||||||
|
Useful for browser upload flows, but *not* headless ingest. They are kept in
|
||||||
|
this repo behind a separate `./browser` entry point and documented as
|
||||||
|
browser-facing so they never blur into the headless core:
|
||||||
|
|
||||||
|
| Upstream file | New home | Notes |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `src/source/pdf/byte-store.ts` | `src/browser/byte-store.ts` | in-memory `blob:` byte store |
|
||||||
|
| `src/source/pdf/upload.ts` | `src/browser/upload.ts` | `ingestPdfFromFile` — thin wrapper over `ingestPdf` |
|
||||||
|
|
||||||
|
### Out of scope — stays in `citation-evidence`
|
||||||
|
|
||||||
|
| Upstream file | Reason |
|
||||||
|
| --- | --- |
|
||||||
|
| `src/source/pdf/viewer-url.ts` | Pure viewer/app concern: it encodes the umbrella's `/fixtures/pdfs/…` URL convention and the app's blob-vs-fixture fallback policy. It consumes `PdfByteStore` (a type this repo now owns) but the resolution *policy* belongs to the viewer. It stays upstream and imports the `PdfByteStore` type from this package. |
|
||||||
|
| `tests/integration/anchor-source-roundtrip.test.ts` | Cross-subsystem (source ↔ anchor) contract. Remains an umbrella integration test; it is not a private test of this repo. |
|
||||||
|
|
||||||
|
### Contracts
|
||||||
|
|
||||||
|
All domain contracts come from `@citation-evidence/engine/shared`. This repo
|
||||||
|
declares `@citation-evidence/engine` as a `link:../citation-engine` dependency
|
||||||
|
and never re-declares `Document`/`DocumentRepresentation`/etc. locally.
|
||||||
|
|
||||||
|
### Fixture corpus
|
||||||
|
|
||||||
|
The PDF fixture corpus (`../citation-evidence/fixtures/pdfs/`) is **not** copied
|
||||||
|
into this repo. The files are real personal/legal documents (utility statements,
|
||||||
|
court letters, admission forms) containing PII; duplicating them into a package
|
||||||
|
that advertises reusable ingest capability is undesirable. Instead this repo's
|
||||||
|
tests resolve the corpus from the sibling checkout (default
|
||||||
|
`../citation-evidence/fixtures/pdfs`, override via
|
||||||
|
`EVIDENCE_SOURCE_FIXTURE_DIR`). This mirrors the sibling-checkout `link:` model
|
||||||
|
already used for the engine dependency and keeps the corpus single-owned
|
||||||
|
upstream. See ADR-0002.
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
- The extraction is a 5-file move (3 core + 2 browser) plus import rewrites.
|
||||||
|
- `citation-evidence` keeps `viewer-url.ts` and the round-trip integration test.
|
||||||
|
- Tests here require a sibling `citation-evidence` checkout for fixtures — an
|
||||||
|
accepted coupling, identical in spirit to the engine `link:` dependency.
|
||||||
43
docs/ADR-0002-fixture-ownership.md
Normal file
43
docs/ADR-0002-fixture-ownership.md
Normal file
|
|
@ -0,0 +1,43 @@
|
||||||
|
# ADR-0002 — Fixture corpus ownership
|
||||||
|
|
||||||
|
- **Status:** accepted
|
||||||
|
- **Date:** 2026-07-08
|
||||||
|
- **Workplan:** ESRC-WP-0001 (T05)
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
The PDF ingest contract tests are driven by a fixture corpus
|
||||||
|
(`fixtures/pdfs/` + `manifest.json`) currently owned by `citation-evidence`.
|
||||||
|
The corpus is made of **real** personal and legal documents — utility-cost
|
||||||
|
statements, a settlement letter, court correspondence, cemetery admission
|
||||||
|
forms — containing names, addresses, and case details.
|
||||||
|
|
||||||
|
The workplan (T05) asks us to either copy the corpus into this repo or define a
|
||||||
|
stable shared-fixture path.
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
Do **not** copy the corpus into `evidence-source`. Resolve it from the sibling
|
||||||
|
`citation-evidence` checkout instead:
|
||||||
|
|
||||||
|
- default: `<repo>/../citation-evidence/fixtures/pdfs`
|
||||||
|
- override: `EVIDENCE_SOURCE_FIXTURE_DIR` environment variable
|
||||||
|
|
||||||
|
## Rationale
|
||||||
|
|
||||||
|
- **Privacy.** `evidence-source` advertises reusable ingest capability and is a
|
||||||
|
candidate for wider distribution. Duplicating real PII into it multiplies the
|
||||||
|
places that data lives and can be leaked.
|
||||||
|
- **Single ownership.** The corpus and its `manifest.json` (page counts,
|
||||||
|
known-good quotes) were curated and re-verified upstream. One owner avoids
|
||||||
|
drift between two copies.
|
||||||
|
- **Consistent coupling.** This repo already assumes a sibling checkout for its
|
||||||
|
`@citation-evidence/engine` (`link:../citation-engine`) dependency. Reading
|
||||||
|
fixtures from `../citation-evidence` is the same coupling model, not a new one.
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
- Tests require a sibling `citation-evidence` checkout. In CI or a standalone
|
||||||
|
clone, point `EVIDENCE_SOURCE_FIXTURE_DIR` at wherever the corpus is mounted.
|
||||||
|
- When the corpus is absent, the fixture-driven suites skip with a clear message
|
||||||
|
rather than failing spuriously, so unit tests that need no corpus still run.
|
||||||
58
eslint.config.js
Normal file
58
eslint.config.js
Normal file
|
|
@ -0,0 +1,58 @@
|
||||||
|
// ESLint flat config for evidence-source.
|
||||||
|
//
|
||||||
|
// Enforces the one boundary that matters in this repo: the browser upload
|
||||||
|
// surface (`src/browser/**`) may depend on the headless core (`src/pdf/**`),
|
||||||
|
// but the headless core must never import from `src/browser/**`. That keeps
|
||||||
|
// `src/pdf/` runtime-agnostic (no `URL.createObjectURL`, no `Blob` upload
|
||||||
|
// path) so it loads cleanly in Node and browsers alike.
|
||||||
|
|
||||||
|
import js from "@eslint/js";
|
||||||
|
import tseslint from "typescript-eslint";
|
||||||
|
import importPlugin from "eslint-plugin-import";
|
||||||
|
import globals from "globals";
|
||||||
|
|
||||||
|
export default tseslint.config(
|
||||||
|
{
|
||||||
|
ignores: ["dist/", "node_modules/", "coverage/", "**/*.d.ts"],
|
||||||
|
},
|
||||||
|
js.configs.recommended,
|
||||||
|
...tseslint.configs.recommended,
|
||||||
|
{
|
||||||
|
files: ["src/**/*.ts", "tests/**/*.ts"],
|
||||||
|
languageOptions: {
|
||||||
|
ecmaVersion: 2022,
|
||||||
|
sourceType: "module",
|
||||||
|
globals: { ...globals.node },
|
||||||
|
},
|
||||||
|
plugins: {
|
||||||
|
import: importPlugin,
|
||||||
|
},
|
||||||
|
settings: {
|
||||||
|
"import/resolver": {
|
||||||
|
typescript: { project: "./tsconfig.json" },
|
||||||
|
},
|
||||||
|
},
|
||||||
|
rules: {
|
||||||
|
"no-restricted-imports": [
|
||||||
|
"error",
|
||||||
|
{
|
||||||
|
patterns: [
|
||||||
|
{
|
||||||
|
group: ["**/browser/**", "../browser/*", "./browser/*"],
|
||||||
|
message:
|
||||||
|
"The headless core (src/pdf) must not depend on browser helpers (src/browser).",
|
||||||
|
},
|
||||||
|
],
|
||||||
|
},
|
||||||
|
],
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
// Browser helpers are allowed to reach into the headless core; relax the
|
||||||
|
// guard for tests, which import from anywhere.
|
||||||
|
files: ["src/browser/**/*.ts", "tests/**/*.ts", "src/index.ts"],
|
||||||
|
rules: {
|
||||||
|
"no-restricted-imports": "off",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
);
|
||||||
43
package.json
Normal file
43
package.json
Normal file
|
|
@ -0,0 +1,43 @@
|
||||||
|
{
|
||||||
|
"name": "@citation-evidence/evidence-source",
|
||||||
|
"version": "0.1.0",
|
||||||
|
"private": true,
|
||||||
|
"description": "Headless document ingest, fingerprinting, and canonical-text extraction for the citation-evidence ecosystem. PDF slice.",
|
||||||
|
"license": "Apache-2.0",
|
||||||
|
"type": "module",
|
||||||
|
"packageManager": "pnpm@9.15.0",
|
||||||
|
"engines": {
|
||||||
|
"node": ">=20.10.0"
|
||||||
|
},
|
||||||
|
"files": [
|
||||||
|
"src",
|
||||||
|
"docs",
|
||||||
|
"README.md",
|
||||||
|
"LICENSE"
|
||||||
|
],
|
||||||
|
"exports": {
|
||||||
|
".": "./src/index.ts",
|
||||||
|
"./browser": "./src/browser/index.ts"
|
||||||
|
},
|
||||||
|
"scripts": {
|
||||||
|
"test": "vitest run",
|
||||||
|
"test:watch": "vitest",
|
||||||
|
"lint": "eslint .",
|
||||||
|
"typecheck": "tsc --noEmit"
|
||||||
|
},
|
||||||
|
"dependencies": {
|
||||||
|
"@citation-evidence/engine": "link:../citation-engine",
|
||||||
|
"pdfjs-dist": "^4.4.168"
|
||||||
|
},
|
||||||
|
"devDependencies": {
|
||||||
|
"@eslint/js": "^9.7.0",
|
||||||
|
"@types/node": "^20.14.0",
|
||||||
|
"eslint": "^9.7.0",
|
||||||
|
"eslint-import-resolver-typescript": "^3.6.3",
|
||||||
|
"eslint-plugin-import": "^2.30.0",
|
||||||
|
"globals": "^15.9.0",
|
||||||
|
"typescript": "^5.5.4",
|
||||||
|
"typescript-eslint": "^8.0.0",
|
||||||
|
"vitest": "^2.0.5"
|
||||||
|
}
|
||||||
|
}
|
||||||
3491
pnpm-lock.yaml
generated
Normal file
3491
pnpm-lock.yaml
generated
Normal file
File diff suppressed because it is too large
Load diff
|
|
@ -1,15 +0,0 @@
|
||||||
---
|
|
||||||
repo: evidence-source
|
|
||||||
reason: >
|
|
||||||
Only INTENT/SCOPE/registry/workplans; the ingestion/extraction layer is described but not implemented.
|
|
||||||
reviewed: "2026-07-06"
|
|
||||||
reviewed_by: claude-code
|
|
||||||
revisit: >
|
|
||||||
Revisit once the ingestion/extraction layer has a real implementation.
|
|
||||||
---
|
|
||||||
|
|
||||||
# No reusable capability
|
|
||||||
|
|
||||||
This repo was reviewed for the `reuse-surface` capability registry
|
|
||||||
(REUSE-WP-0017 coverage campaign) and has no capability to register at this
|
|
||||||
time. See the `reason` field above.
|
|
||||||
179
registry/capabilities/capability.infotech.pdf-evidence-ingest.md
Normal file
179
registry/capabilities/capability.infotech.pdf-evidence-ingest.md
Normal file
|
|
@ -0,0 +1,179 @@
|
||||||
|
---
|
||||||
|
id: capability.infotech.pdf-evidence-ingest
|
||||||
|
name: Headless PDF Evidence Ingest
|
||||||
|
summary: Turns raw PDF bytes into an engine-shaped Document + DocumentRepresentation — SHA-256 fingerprint, canonical text, page map, and gap-free offset map — as a runtime-agnostic library with no viewer, persistence, or React coupling.
|
||||||
|
owner: evidence-source
|
||||||
|
status: draft
|
||||||
|
domain: infotech
|
||||||
|
tags:
|
||||||
|
- pdf
|
||||||
|
- ingest
|
||||||
|
- fingerprint
|
||||||
|
- canonical-text
|
||||||
|
- evidence
|
||||||
|
|
||||||
|
maturity:
|
||||||
|
discovery:
|
||||||
|
current: D2
|
||||||
|
target: D5
|
||||||
|
confidence: medium
|
||||||
|
rationale: >
|
||||||
|
Intent, scope, and boundary are documented (ADR-0001/0002) and the
|
||||||
|
surface is exercised by a fixture-driven contract suite plus a
|
||||||
|
cross-repo consumer round-trip. HTML/MD ingest and citation recovery
|
||||||
|
are declared out of scope for now, so discovery is bounded to the PDF
|
||||||
|
slice.
|
||||||
|
availability:
|
||||||
|
current: A1
|
||||||
|
target: A3
|
||||||
|
confidence: medium
|
||||||
|
rationale: >
|
||||||
|
Consumable today as a linked library (`link:../evidence-source`) via the
|
||||||
|
package root (`@citation-evidence/evidence-source`) and browser subpath.
|
||||||
|
Reaching A3 needs a published/versioned artifact rather than a
|
||||||
|
sibling-checkout link.
|
||||||
|
|
||||||
|
external_evidence:
|
||||||
|
completeness:
|
||||||
|
level: C1
|
||||||
|
confidence: medium
|
||||||
|
basis: scope_vs_intent_and_consumer_expectations
|
||||||
|
satisfied_expectations:
|
||||||
|
- PDF bytes -> Document + DocumentRepresentation
|
||||||
|
- deterministic SHA-256 fingerprint
|
||||||
|
- canonical text with gap-free page offset map
|
||||||
|
broken_expectations: []
|
||||||
|
out_of_scope_expectations:
|
||||||
|
- HTML / Markdown ingest
|
||||||
|
- metadata enrichment beyond pass-through options
|
||||||
|
- citation recovery / external source discovery
|
||||||
|
reliability:
|
||||||
|
level: R1
|
||||||
|
confidence: medium
|
||||||
|
basis: consumer_quality_signals
|
||||||
|
known_reliability_risks:
|
||||||
|
- PDF.js worker must be configured by the host before extraction
|
||||||
|
- text-extraction fidelity varies on scanned/OCR-noisy PDFs
|
||||||
|
|
||||||
|
discovery:
|
||||||
|
intent: >
|
||||||
|
Let any consumer turn a PDF into engine-owned evidence contracts without
|
||||||
|
inheriting a viewer, a store, or a UI framework.
|
||||||
|
includes:
|
||||||
|
- PDF byte ingest (ingestPdf)
|
||||||
|
- PDF text extraction to canonical text + page/offset maps (extractPdf)
|
||||||
|
- SHA-256 byte fingerprint (fingerprintBytes)
|
||||||
|
- browser upload helpers behind a separate entry point (createPdfByteStore, ingestPdfFromFile)
|
||||||
|
excludes:
|
||||||
|
- HTML / Markdown ingest
|
||||||
|
- persistence of documents/representations
|
||||||
|
- viewer URL resolution (stays in the consuming app)
|
||||||
|
- citation recovery
|
||||||
|
assumptions:
|
||||||
|
- domain contracts come from '@citation-evidence/engine/shared'
|
||||||
|
- host configures the PDF.js worker before extraction
|
||||||
|
use_cases:
|
||||||
|
- citation-evidence umbrella consumes ingest across the repo boundary
|
||||||
|
research_memos: []
|
||||||
|
|
||||||
|
availability:
|
||||||
|
current_level: A1
|
||||||
|
target_level: A3
|
||||||
|
current_artifacts:
|
||||||
|
- '@citation-evidence/evidence-source (package root, headless core)'
|
||||||
|
- '@citation-evidence/evidence-source/browser (upload helpers)'
|
||||||
|
target_artifacts:
|
||||||
|
- published versioned package
|
||||||
|
consumption_modes:
|
||||||
|
- library import
|
||||||
|
|
||||||
|
relations:
|
||||||
|
depends_on: []
|
||||||
|
supports: []
|
||||||
|
related_to: []
|
||||||
|
|
||||||
|
evidence:
|
||||||
|
documentation:
|
||||||
|
- README.md
|
||||||
|
- docs/ADR-0001-extraction-boundary.md
|
||||||
|
- docs/ADR-0002-fixture-ownership.md
|
||||||
|
tests:
|
||||||
|
- src/pdf/ingest.test.ts
|
||||||
|
- src/pdf/fingerprint.test.ts
|
||||||
|
- src/browser/byte-store.test.ts
|
||||||
|
- src/browser/upload.test.ts
|
||||||
|
consumer_feedback: []
|
||||||
|
bug_reports: []
|
||||||
|
incidents: []
|
||||||
|
|
||||||
|
consumer_guidance:
|
||||||
|
recommended_for:
|
||||||
|
- headless PDF -> evidence-contract ingest in Node or the browser
|
||||||
|
- deterministic content fingerprinting of document bytes
|
||||||
|
not_recommended_for:
|
||||||
|
- non-PDF formats (not yet implemented)
|
||||||
|
- high-fidelity extraction from scanned/OCR-only PDFs
|
||||||
|
known_limitations:
|
||||||
|
- PDF only for now
|
||||||
|
- requires host-configured PDF.js worker
|
||||||
|
- fixtures resolved from sibling citation-evidence checkout (ADR-0002)
|
||||||
|
|
||||||
|
promotion_history:
|
||||||
|
- date: "2026-07-08"
|
||||||
|
dimension: discovery
|
||||||
|
from: D0
|
||||||
|
to: D2
|
||||||
|
rationale: PDF ingest slice extracted, documented (ADR-0001/0002), and covered by contract tests plus a cross-repo consumer round-trip (ESRC-WP-0001).
|
||||||
|
author: claude-code
|
||||||
|
---
|
||||||
|
|
||||||
|
# Headless PDF Evidence Ingest
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
`evidence-source` turns raw PDF bytes into the two engine-owned contracts that
|
||||||
|
the rest of the citation-evidence ecosystem builds on — a `Document` (media
|
||||||
|
type, SHA-256 fingerprint, optional title/uri/metadata) and a
|
||||||
|
`DocumentRepresentation` (`pdf-text`: canonical text, page map, and a gap-free
|
||||||
|
offset map). Ingest is pure over bytes: no persistence, no viewer state, no
|
||||||
|
React.
|
||||||
|
|
||||||
|
The YAML front matter above is the machine-readable source of truth for
|
||||||
|
maturity, evidence, relations, and consumer guidance.
|
||||||
|
|
||||||
|
## Assessment notes
|
||||||
|
|
||||||
|
### Discovery
|
||||||
|
|
||||||
|
The boundary is fixed in ADR-0001 (which files are headless core vs. browser
|
||||||
|
helpers vs. app-only viewer concerns) and ADR-0002 (fixture ownership). The
|
||||||
|
public surface is small and stable: `ingestPdf`, `extractPdf`,
|
||||||
|
`fingerprintBytes`, plus browser upload helpers behind a separate entry point.
|
||||||
|
|
||||||
|
### Availability
|
||||||
|
|
||||||
|
Consumed today as a linked library via `link:../evidence-source`. The umbrella
|
||||||
|
app (`citation-evidence`) imports the headless core and browser helpers through
|
||||||
|
a thin façade and runs its source ↔ anchor round-trip across the boundary.
|
||||||
|
Target A3 requires a published, versioned artifact.
|
||||||
|
|
||||||
|
### Completeness
|
||||||
|
|
||||||
|
PDF only. HTML/Markdown representations, metadata enrichment, and citation
|
||||||
|
recovery are explicitly deferred to follow-on workplans (see README) and are
|
||||||
|
recorded as out-of-scope expectations rather than gaps.
|
||||||
|
|
||||||
|
### Reliability
|
||||||
|
|
||||||
|
Backed by a fixture-driven contract suite (fingerprint determinism, page/offset
|
||||||
|
invariants, known-good quote presence) and a downstream consumer round-trip.
|
||||||
|
The main reliability caveat is extraction fidelity on scanned/OCR-noisy PDFs,
|
||||||
|
which is inherent to text-layer extraction.
|
||||||
|
|
||||||
|
## Promotion checklist
|
||||||
|
|
||||||
|
- [x] ID follows `capability.<domain>.<name>` pattern
|
||||||
|
- [x] Maturity enums match `specs/CapabilityMaturityStandard.md`
|
||||||
|
- [x] `external_evidence` is populated separately from `maturity`
|
||||||
|
- [x] Relations reference valid capability IDs
|
||||||
|
- [x] Index entry added or updated in `registry/indexes/capabilities.yaml`
|
||||||
|
|
@ -1,4 +1,22 @@
|
||||||
version: 1
|
version: 1
|
||||||
updated: '2026-06-16'
|
updated: '2026-07-08'
|
||||||
domain: helix_forge
|
domain: infotech
|
||||||
capabilities: []
|
capabilities:
|
||||||
|
- id: capability.infotech.pdf-evidence-ingest
|
||||||
|
name: Headless PDF Evidence Ingest
|
||||||
|
summary: Turns raw PDF bytes into an engine-shaped Document + DocumentRepresentation
|
||||||
|
— SHA-256 fingerprint, canonical text, page map, and gap-free offset map — as a
|
||||||
|
runtime-agnostic library with no viewer, persistence, or React coupling.
|
||||||
|
vector: D2 / A1 / C1 / R1
|
||||||
|
domain: infotech
|
||||||
|
status: draft
|
||||||
|
owner: evidence-source
|
||||||
|
path: registry/capabilities/capability.infotech.pdf-evidence-ingest.md
|
||||||
|
tags:
|
||||||
|
- pdf
|
||||||
|
- ingest
|
||||||
|
- fingerprint
|
||||||
|
- canonical-text
|
||||||
|
- evidence
|
||||||
|
consumption_modes:
|
||||||
|
- library import
|
||||||
|
|
|
||||||
97
src/browser/byte-store.test.ts
Normal file
97
src/browser/byte-store.test.ts
Normal file
|
|
@ -0,0 +1,97 @@
|
||||||
|
import { describe, expect, it, vi } from "vitest";
|
||||||
|
|
||||||
|
import type { DocumentId } from "@citation-evidence/engine/shared";
|
||||||
|
|
||||||
|
import { createPdfByteStore } from "./byte-store";
|
||||||
|
|
||||||
|
function stubUrlHelpers() {
|
||||||
|
let counter = 0;
|
||||||
|
const created: string[] = [];
|
||||||
|
const revoked: string[] = [];
|
||||||
|
const createObjectURL = vi.fn(() => {
|
||||||
|
const url = `blob:stub-${++counter}`;
|
||||||
|
created.push(url);
|
||||||
|
return url;
|
||||||
|
});
|
||||||
|
const revokeObjectURL = vi.fn((url: string) => {
|
||||||
|
revoked.push(url);
|
||||||
|
});
|
||||||
|
return { createObjectURL, revokeObjectURL, created, revoked };
|
||||||
|
}
|
||||||
|
|
||||||
|
describe("PdfByteStore", () => {
|
||||||
|
it("put / get round-trips bytes and exposes a blob URL", () => {
|
||||||
|
const helpers = stubUrlHelpers();
|
||||||
|
const store = createPdfByteStore(helpers);
|
||||||
|
const bytes = new Uint8Array([0x25, 0x50, 0x44, 0x46]); // %PDF
|
||||||
|
const record = store.put("doc_a" as DocumentId, bytes);
|
||||||
|
expect(record.blobUrl).toBe("blob:stub-1");
|
||||||
|
expect(store.get("doc_a" as DocumentId)?.bytes).toBe(bytes);
|
||||||
|
expect(store.has("doc_a" as DocumentId)).toBe(true);
|
||||||
|
expect(store.list()).toEqual(["doc_a"]);
|
||||||
|
expect(store.size()).toBe(4);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("put replaces an existing entry and revokes the old URL", () => {
|
||||||
|
const helpers = stubUrlHelpers();
|
||||||
|
const store = createPdfByteStore(helpers);
|
||||||
|
const id = "doc_a" as DocumentId;
|
||||||
|
const first = store.put(id, new Uint8Array([1, 2]));
|
||||||
|
const second = store.put(id, new Uint8Array([3, 4, 5]));
|
||||||
|
expect(helpers.revoked).toEqual([first.blobUrl]);
|
||||||
|
expect(store.get(id)?.bytes).toHaveLength(3);
|
||||||
|
expect(second.blobUrl).not.toBe(first.blobUrl);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("delete revokes the blob URL exactly once and is idempotent", () => {
|
||||||
|
const helpers = stubUrlHelpers();
|
||||||
|
const store = createPdfByteStore(helpers);
|
||||||
|
const id = "doc_a" as DocumentId;
|
||||||
|
const record = store.put(id, new Uint8Array([1, 2, 3]));
|
||||||
|
expect(store.delete(id)).toBe(true);
|
||||||
|
expect(helpers.revoked).toEqual([record.blobUrl]);
|
||||||
|
expect(store.delete(id)).toBe(false);
|
||||||
|
expect(helpers.revoked).toHaveLength(1);
|
||||||
|
expect(store.get(id)).toBeNull();
|
||||||
|
expect(store.has(id)).toBe(false);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("clear revokes every URL and empties the store", () => {
|
||||||
|
const helpers = stubUrlHelpers();
|
||||||
|
const store = createPdfByteStore(helpers);
|
||||||
|
const a = store.put("doc_a" as DocumentId, new Uint8Array([1]));
|
||||||
|
const b = store.put("doc_b" as DocumentId, new Uint8Array([2]));
|
||||||
|
store.clear();
|
||||||
|
expect(helpers.revoked.sort()).toEqual([a.blobUrl, b.blobUrl].sort());
|
||||||
|
expect(store.list()).toEqual([]);
|
||||||
|
expect(store.size()).toBe(0);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("uses URL.createObjectURL by default when no override is supplied", () => {
|
||||||
|
const createObjectURL = vi.fn(() => "blob:built-in");
|
||||||
|
const revokeObjectURL = vi.fn();
|
||||||
|
const originalURL = globalThis.URL;
|
||||||
|
Object.defineProperty(globalThis, "URL", {
|
||||||
|
configurable: true,
|
||||||
|
writable: true,
|
||||||
|
value: Object.assign(Object.create(originalURL.prototype as object), {
|
||||||
|
createObjectURL,
|
||||||
|
revokeObjectURL,
|
||||||
|
}),
|
||||||
|
});
|
||||||
|
try {
|
||||||
|
const store = createPdfByteStore();
|
||||||
|
const rec = store.put("doc_z" as DocumentId, new Uint8Array([9]));
|
||||||
|
expect(rec.blobUrl).toBe("blob:built-in");
|
||||||
|
expect(createObjectURL).toHaveBeenCalledTimes(1);
|
||||||
|
store.delete("doc_z" as DocumentId);
|
||||||
|
expect(revokeObjectURL).toHaveBeenCalledWith("blob:built-in");
|
||||||
|
} finally {
|
||||||
|
Object.defineProperty(globalThis, "URL", {
|
||||||
|
configurable: true,
|
||||||
|
writable: true,
|
||||||
|
value: originalURL,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
});
|
||||||
|
});
|
||||||
115
src/browser/byte-store.ts
Normal file
115
src/browser/byte-store.ts
Normal file
|
|
@ -0,0 +1,115 @@
|
||||||
|
/**
|
||||||
|
* `PdfByteStore` — in-memory store for uploaded PDF bytes, keyed by
|
||||||
|
* `DocumentId`.
|
||||||
|
*
|
||||||
|
* BROWSER SURFACE — not part of the headless ingest core. This mints and
|
||||||
|
* revokes `blob:` URLs and is meant to back an interactive upload/viewer
|
||||||
|
* session. Headless callers should use `src/pdf/` directly and never import
|
||||||
|
* this module.
|
||||||
|
*
|
||||||
|
* Uploaded PDFs are stored in memory only. Bytes survive within a tab
|
||||||
|
* session; reloading the page loses them unless the session was exported.
|
||||||
|
*
|
||||||
|
* One store instance per active session. The session-management layer is
|
||||||
|
* responsible for swapping the active store when the user switches sessions.
|
||||||
|
* The store also owns a small registry of issued `blob:` URLs so it can
|
||||||
|
* revoke them on delete/clear — no cross-cutting cleanup at the app layer.
|
||||||
|
*/
|
||||||
|
|
||||||
|
import type { DocumentId } from "@citation-evidence/engine/shared";
|
||||||
|
|
||||||
|
export interface PdfByteRecord {
|
||||||
|
readonly bytes: Uint8Array;
|
||||||
|
/** A `blob:` URL the viewer can consume directly. */
|
||||||
|
readonly blobUrl: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface PdfByteStore {
|
||||||
|
put(documentId: DocumentId, bytes: Uint8Array): PdfByteRecord;
|
||||||
|
get(documentId: DocumentId): PdfByteRecord | null;
|
||||||
|
has(documentId: DocumentId): boolean;
|
||||||
|
delete(documentId: DocumentId): boolean;
|
||||||
|
list(): readonly DocumentId[];
|
||||||
|
/** Revoke every blob URL and clear the store. */
|
||||||
|
clear(): void;
|
||||||
|
/** Total bytes currently held — useful for UI dashboards. */
|
||||||
|
size(): number;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface CreatePdfByteStoreOptions {
|
||||||
|
/**
|
||||||
|
* Mint a URL for the given bytes. Defaults to `URL.createObjectURL` in
|
||||||
|
* environments that have it; tests can inject a deterministic stub.
|
||||||
|
*/
|
||||||
|
readonly createObjectURL?: (blob: Blob) => string;
|
||||||
|
/** Revoke a URL previously minted by `createObjectURL`. */
|
||||||
|
readonly revokeObjectURL?: (url: string) => void;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function createPdfByteStore(
|
||||||
|
options: CreatePdfByteStoreOptions = {},
|
||||||
|
): PdfByteStore {
|
||||||
|
const createUrl =
|
||||||
|
options.createObjectURL ??
|
||||||
|
((blob: Blob) => {
|
||||||
|
if (typeof URL === "undefined" || typeof URL.createObjectURL !== "function") {
|
||||||
|
throw new Error(
|
||||||
|
"createPdfByteStore: URL.createObjectURL not available — inject a stub via options",
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return URL.createObjectURL(blob);
|
||||||
|
});
|
||||||
|
const revokeUrl =
|
||||||
|
options.revokeObjectURL ??
|
||||||
|
((url: string) => {
|
||||||
|
if (typeof URL !== "undefined" && typeof URL.revokeObjectURL === "function") {
|
||||||
|
URL.revokeObjectURL(url);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
const records = new Map<DocumentId, PdfByteRecord>();
|
||||||
|
|
||||||
|
return {
|
||||||
|
put(documentId, bytes) {
|
||||||
|
// Replace previous record (revoking the prior URL) if any.
|
||||||
|
const prior = records.get(documentId);
|
||||||
|
if (prior) revokeUrl(prior.blobUrl);
|
||||||
|
// Cast: Blob() does accept Uint8Array at runtime, but TS narrows the
|
||||||
|
// buffer type to ArrayBufferLike (could be SharedArrayBuffer) and
|
||||||
|
// refuses without help. The bytes here always come from a fresh
|
||||||
|
// arrayBuffer() call, so a regular ArrayBuffer is guaranteed.
|
||||||
|
const blob = new Blob([bytes as unknown as ArrayBuffer], {
|
||||||
|
type: "application/pdf",
|
||||||
|
});
|
||||||
|
const blobUrl = createUrl(blob);
|
||||||
|
const record: PdfByteRecord = { bytes, blobUrl };
|
||||||
|
records.set(documentId, record);
|
||||||
|
return record;
|
||||||
|
},
|
||||||
|
get(documentId) {
|
||||||
|
return records.get(documentId) ?? null;
|
||||||
|
},
|
||||||
|
has(documentId) {
|
||||||
|
return records.has(documentId);
|
||||||
|
},
|
||||||
|
delete(documentId) {
|
||||||
|
const record = records.get(documentId);
|
||||||
|
if (!record) return false;
|
||||||
|
revokeUrl(record.blobUrl);
|
||||||
|
records.delete(documentId);
|
||||||
|
return true;
|
||||||
|
},
|
||||||
|
list() {
|
||||||
|
return [...records.keys()];
|
||||||
|
},
|
||||||
|
clear() {
|
||||||
|
for (const record of records.values()) revokeUrl(record.blobUrl);
|
||||||
|
records.clear();
|
||||||
|
},
|
||||||
|
size() {
|
||||||
|
let total = 0;
|
||||||
|
for (const r of records.values()) total += r.bytes.byteLength;
|
||||||
|
return total;
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
20
src/browser/index.ts
Normal file
20
src/browser/index.ts
Normal file
|
|
@ -0,0 +1,20 @@
|
||||||
|
/**
|
||||||
|
* `evidence-source/browser` — browser-facing upload helpers.
|
||||||
|
*
|
||||||
|
* BROWSER SURFACE. These helpers assume a browser session: they mint and
|
||||||
|
* revoke `blob:` URLs and hold uploaded bytes in memory for a viewer to
|
||||||
|
* mount. They are deliberately separated from the headless core (the package
|
||||||
|
* root, `src/index.ts`) so that headless consumers never pull in
|
||||||
|
* `URL.createObjectURL`/`Blob` upload machinery.
|
||||||
|
*/
|
||||||
|
|
||||||
|
export {
|
||||||
|
createPdfByteStore,
|
||||||
|
type CreatePdfByteStoreOptions,
|
||||||
|
type PdfByteRecord,
|
||||||
|
type PdfByteStore,
|
||||||
|
} from "./byte-store";
|
||||||
|
export {
|
||||||
|
ingestPdfFromFile,
|
||||||
|
type IngestPdfFromFileOptions,
|
||||||
|
} from "./upload";
|
||||||
82
src/browser/upload.test.ts
Normal file
82
src/browser/upload.test.ts
Normal file
|
|
@ -0,0 +1,82 @@
|
||||||
|
/**
|
||||||
|
* `ingestPdfFromFile` end-to-end: pipes a fixture PDF through the upload path,
|
||||||
|
* asserts the byte store keeps the bytes and the document record carries the
|
||||||
|
* minted `blob:` URL. Corpus resolution/skip lives in `tests/fixtures.ts`.
|
||||||
|
*/
|
||||||
|
|
||||||
|
import { describe, expect, it, vi } from "vitest";
|
||||||
|
|
||||||
|
import { createPdfByteStore } from "./byte-store";
|
||||||
|
import { ingestPdfFromFile } from "./upload";
|
||||||
|
import { fixtureBytes, fixtureCorpusAvailable } from "../../tests/fixtures";
|
||||||
|
|
||||||
|
const hasCorpus = fixtureCorpusAvailable();
|
||||||
|
const FIXTURE_FILE = "Fristsetzung zur Bezifferung GÜ an Gegenseite 3 Wochen.pdf";
|
||||||
|
|
||||||
|
class FakeFile {
|
||||||
|
readonly name: string;
|
||||||
|
private readonly bytes: Uint8Array;
|
||||||
|
constructor(bytes: Uint8Array, name: string) {
|
||||||
|
this.bytes = bytes;
|
||||||
|
this.name = name;
|
||||||
|
}
|
||||||
|
async arrayBuffer(): Promise<ArrayBuffer> {
|
||||||
|
const out = new ArrayBuffer(this.bytes.byteLength);
|
||||||
|
new Uint8Array(out).set(this.bytes);
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
describe.skipIf(!hasCorpus)("ingestPdfFromFile", () => {
|
||||||
|
it("round-trips a fixture PDF through ingest + byte store + blob URL", async () => {
|
||||||
|
const bytes = fixtureBytes(FIXTURE_FILE);
|
||||||
|
const file = new FakeFile(bytes, "demo.pdf") as unknown as File;
|
||||||
|
let counter = 0;
|
||||||
|
const store = createPdfByteStore({
|
||||||
|
createObjectURL: () => `blob:upload-stub-${++counter}`,
|
||||||
|
revokeObjectURL: () => {},
|
||||||
|
});
|
||||||
|
|
||||||
|
const { document, representation } = await ingestPdfFromFile(file, store);
|
||||||
|
|
||||||
|
const stored = store.get(document.id);
|
||||||
|
expect(stored).not.toBeNull();
|
||||||
|
expect(stored!.bytes.byteLength).toBe(bytes.byteLength);
|
||||||
|
|
||||||
|
expect(document.uri).toBe(`blob:upload-stub-${counter}`);
|
||||||
|
expect(document.title).toBe("demo.pdf");
|
||||||
|
expect(document.fingerprint).toMatch(/^[0-9a-f]{64}$/);
|
||||||
|
|
||||||
|
expect(representation.representationType).toBe("pdf-text");
|
||||||
|
expect((representation.canonicalText ?? "").length).toBeGreaterThan(0);
|
||||||
|
}, 30_000);
|
||||||
|
|
||||||
|
it("falls through to ingestPdf with no filename when given a plain Blob", async () => {
|
||||||
|
const bytes = fixtureBytes(FIXTURE_FILE);
|
||||||
|
const blob = {
|
||||||
|
async arrayBuffer() {
|
||||||
|
const out = new ArrayBuffer(bytes.byteLength);
|
||||||
|
new Uint8Array(out).set(bytes);
|
||||||
|
return out;
|
||||||
|
},
|
||||||
|
} as Blob;
|
||||||
|
const store = createPdfByteStore({
|
||||||
|
createObjectURL: () => "blob:no-name",
|
||||||
|
revokeObjectURL: () => {},
|
||||||
|
});
|
||||||
|
const { document } = await ingestPdfFromFile(blob, store);
|
||||||
|
expect(document.title).toBeUndefined();
|
||||||
|
expect(document.uri).toBe("blob:no-name");
|
||||||
|
}, 30_000);
|
||||||
|
|
||||||
|
it("explicit title option overrides the filename", async () => {
|
||||||
|
const bytes = fixtureBytes(FIXTURE_FILE);
|
||||||
|
const file = new FakeFile(bytes, "anonymous-name.pdf") as unknown as File;
|
||||||
|
const store = createPdfByteStore({
|
||||||
|
createObjectURL: vi.fn(() => "blob:override"),
|
||||||
|
revokeObjectURL: vi.fn(),
|
||||||
|
});
|
||||||
|
const { document } = await ingestPdfFromFile(file, store, { title: "Custom" });
|
||||||
|
expect(document.title).toBe("Custom");
|
||||||
|
}, 30_000);
|
||||||
|
});
|
||||||
43
src/browser/upload.ts
Normal file
43
src/browser/upload.ts
Normal file
|
|
@ -0,0 +1,43 @@
|
||||||
|
/**
|
||||||
|
* Upload-side ingest path.
|
||||||
|
*
|
||||||
|
* BROWSER SURFACE — not part of the headless ingest core. This is a thin
|
||||||
|
* wrapper over the headless `ingestPdf` that additionally pushes the bytes
|
||||||
|
* into a per-session `PdfByteStore`, which mints a `blob:` URL and stamps it
|
||||||
|
* onto `document.uri` so a viewer adapter can mount the PDF directly.
|
||||||
|
*
|
||||||
|
* 1. Read `file.arrayBuffer()` once into a `Uint8Array`.
|
||||||
|
* 2. Run the existing `ingestPdf(bytes, { filename })` pipeline to produce
|
||||||
|
* `{document, representation}`.
|
||||||
|
* 3. Push the bytes into the per-session `PdfByteStore`.
|
||||||
|
* 4. Hand the engine inputs back to the caller, which wires them via
|
||||||
|
* `engine.documents.register(...)`.
|
||||||
|
*
|
||||||
|
* Keeping URL-minting inside the byte store (rather than at the call site)
|
||||||
|
* means there is exactly one place that creates `blob:` URLs and exactly one
|
||||||
|
* place that revokes them.
|
||||||
|
*/
|
||||||
|
|
||||||
|
import { ingestPdf, type IngestPdfResult } from "../pdf/ingest";
|
||||||
|
import type { PdfByteStore } from "./byte-store";
|
||||||
|
|
||||||
|
export interface IngestPdfFromFileOptions {
|
||||||
|
/** Override the filename used as the document title. */
|
||||||
|
readonly title?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function ingestPdfFromFile(
|
||||||
|
file: File | Blob,
|
||||||
|
store: PdfByteStore,
|
||||||
|
options: IngestPdfFromFileOptions = {},
|
||||||
|
): Promise<IngestPdfResult> {
|
||||||
|
const bytes = new Uint8Array(await file.arrayBuffer());
|
||||||
|
const filename = "name" in file && typeof file.name === "string" ? file.name : undefined;
|
||||||
|
const ingested = await ingestPdf(bytes, {
|
||||||
|
...(filename !== undefined ? { filename } : {}),
|
||||||
|
...(options.title !== undefined ? { title: options.title } : {}),
|
||||||
|
});
|
||||||
|
const record = store.put(ingested.document.id, bytes);
|
||||||
|
const document = { ...ingested.document, uri: record.blobUrl };
|
||||||
|
return { document, representation: ingested.representation };
|
||||||
|
}
|
||||||
20
src/index.ts
Normal file
20
src/index.ts
Normal file
|
|
@ -0,0 +1,20 @@
|
||||||
|
/**
|
||||||
|
* `evidence-source` — headless document ingest core (PDF slice).
|
||||||
|
*
|
||||||
|
* This entry point exposes only runtime-agnostic ingest: pass PDF bytes,
|
||||||
|
* receive a `{ document, representation }` pair built entirely from
|
||||||
|
* engine-owned contracts (`@citation-evidence/engine/shared`). No browser,
|
||||||
|
* viewer, or persistence concerns leak through here.
|
||||||
|
*
|
||||||
|
* Browser-facing upload helpers live behind the separate
|
||||||
|
* `@citation-evidence/evidence-source/browser` entry point.
|
||||||
|
*/
|
||||||
|
|
||||||
|
export {
|
||||||
|
ingestPdf,
|
||||||
|
type IngestPdfInput,
|
||||||
|
type IngestPdfOptions,
|
||||||
|
type IngestPdfResult,
|
||||||
|
} from "./pdf/ingest";
|
||||||
|
export { extractPdf, type PdfExtractionResult } from "./pdf/extract";
|
||||||
|
export { fingerprintBytes } from "./pdf/fingerprint";
|
||||||
122
src/pdf/extract.ts
Normal file
122
src/pdf/extract.ts
Normal file
|
|
@ -0,0 +1,122 @@
|
||||||
|
/**
|
||||||
|
* PDF text extraction → canonical text + PageMap + OffsetMap.
|
||||||
|
*
|
||||||
|
* Implements `wiki/ArchitectureOverview.md` §3.4 ("extract canonical text /
|
||||||
|
* build format-specific maps") for the `pdf-text` representation
|
||||||
|
* (`wiki/SharedContracts.md` §1, §3) and §6 (canonical normalization).
|
||||||
|
*
|
||||||
|
* Runtime independence: the PDF.js worker must be configured by the host
|
||||||
|
* application (`GlobalWorkerOptions.workerSrc`) before this module is
|
||||||
|
* called. In Vite/browser code the worker is bundled via the viewer; in
|
||||||
|
* Node tests the test setup file points it at
|
||||||
|
* `pdfjs-dist/legacy/build/pdf.worker.mjs`. No worker setup happens here
|
||||||
|
* so the same module loads cleanly in both runtimes.
|
||||||
|
*
|
||||||
|
* Page boundary semantics: canonical text concatenates per-page normalized
|
||||||
|
* text with a single "\n\n" paragraph separator. The separator is treated
|
||||||
|
* as belonging to the *preceding* page in `OffsetMap`, so the map covers
|
||||||
|
* `[0, canonicalText.length)` with no gaps. The last page has no trailing
|
||||||
|
* separator. This means `pageLength = globalEnd - globalStart` for
|
||||||
|
* every page; for non-last pages it equals (normalized page text length +
|
||||||
|
* 2). See `PageOffsetRange` in the engine's shared document contract.
|
||||||
|
*/
|
||||||
|
|
||||||
|
import { getDocument } from "pdfjs-dist";
|
||||||
|
import type { PDFPageProxy } from "pdfjs-dist";
|
||||||
|
import {
|
||||||
|
normalize,
|
||||||
|
type OffsetMap,
|
||||||
|
type PageInfo,
|
||||||
|
type PageMap,
|
||||||
|
type PageOffsetRange,
|
||||||
|
} from "@citation-evidence/engine/shared";
|
||||||
|
|
||||||
|
const PAGE_SEPARATOR = "\n\n";
|
||||||
|
|
||||||
|
export interface PdfExtractionResult {
|
||||||
|
readonly canonicalText: string;
|
||||||
|
readonly pageMap: PageMap;
|
||||||
|
readonly offsetMap: OffsetMap;
|
||||||
|
readonly pageCount: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function extractPdf(bytes: Uint8Array): Promise<PdfExtractionResult> {
|
||||||
|
// PDF.js mutates the bytes buffer (transfers ownership). Pass a fresh copy
|
||||||
|
// so the caller's Uint8Array stays usable for fingerprinting after extract.
|
||||||
|
const data = new Uint8Array(bytes);
|
||||||
|
const loadingTask = getDocument({ data });
|
||||||
|
const doc = await loadingTask.promise;
|
||||||
|
|
||||||
|
try {
|
||||||
|
const pageCount = doc.numPages;
|
||||||
|
const pageInfos: PageInfo[] = [];
|
||||||
|
const pageNormalizedTexts: string[] = [];
|
||||||
|
|
||||||
|
for (let pageNumber = 1; pageNumber <= pageCount; pageNumber++) {
|
||||||
|
const page = await doc.getPage(pageNumber);
|
||||||
|
try {
|
||||||
|
const viewport = page.getViewport({ scale: 1 });
|
||||||
|
pageInfos.push({
|
||||||
|
page: pageNumber,
|
||||||
|
width: viewport.width,
|
||||||
|
height: viewport.height,
|
||||||
|
});
|
||||||
|
|
||||||
|
const rawText = await extractPageText(page);
|
||||||
|
pageNormalizedTexts.push(normalize(rawText).text);
|
||||||
|
} finally {
|
||||||
|
page.cleanup();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const { canonicalText, offsetMap } = buildOffsetMap(pageNormalizedTexts);
|
||||||
|
|
||||||
|
return {
|
||||||
|
canonicalText,
|
||||||
|
pageMap: pageInfos,
|
||||||
|
offsetMap,
|
||||||
|
pageCount,
|
||||||
|
};
|
||||||
|
} finally {
|
||||||
|
await doc.destroy();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function extractPageText(page: PDFPageProxy): Promise<string> {
|
||||||
|
const content = await page.getTextContent();
|
||||||
|
// textContent.items are TextItem | TextMarkedContent. We want only the
|
||||||
|
// TextItem strings (those have a `str` field); marked-content entries are
|
||||||
|
// structural anchors and have no visible text.
|
||||||
|
const parts: string[] = [];
|
||||||
|
for (const item of content.items) {
|
||||||
|
if ("str" in item) {
|
||||||
|
parts.push(item.str);
|
||||||
|
if (item.hasEOL) parts.push("\n");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return parts.join("");
|
||||||
|
}
|
||||||
|
|
||||||
|
function buildOffsetMap(pageTexts: readonly string[]): {
|
||||||
|
canonicalText: string;
|
||||||
|
offsetMap: OffsetMap;
|
||||||
|
} {
|
||||||
|
const ranges: PageOffsetRange[] = [];
|
||||||
|
let offset = 0;
|
||||||
|
for (let i = 0; i < pageTexts.length; i++) {
|
||||||
|
const text = pageTexts[i]!;
|
||||||
|
const isLast = i === pageTexts.length - 1;
|
||||||
|
const segmentLength = text.length + (isLast ? 0 : PAGE_SEPARATOR.length);
|
||||||
|
const globalStart = offset;
|
||||||
|
const globalEnd = offset + segmentLength;
|
||||||
|
ranges.push({
|
||||||
|
page: i + 1,
|
||||||
|
globalStart,
|
||||||
|
globalEnd,
|
||||||
|
pageLength: segmentLength,
|
||||||
|
});
|
||||||
|
offset = globalEnd;
|
||||||
|
}
|
||||||
|
const canonicalText = pageTexts.join(PAGE_SEPARATOR);
|
||||||
|
return { canonicalText, offsetMap: ranges };
|
||||||
|
}
|
||||||
37
src/pdf/fingerprint.test.ts
Normal file
37
src/pdf/fingerprint.test.ts
Normal file
|
|
@ -0,0 +1,37 @@
|
||||||
|
import { describe, expect, it } from "vitest";
|
||||||
|
|
||||||
|
import { fingerprintBytes } from "./fingerprint";
|
||||||
|
|
||||||
|
describe("fingerprintBytes", () => {
|
||||||
|
it("produces a 64-char lowercase hex SHA-256", async () => {
|
||||||
|
const digest = await fingerprintBytes(new Uint8Array([0x25, 0x50, 0x44, 0x46]));
|
||||||
|
expect(digest).toMatch(/^[0-9a-f]{64}$/);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("matches the known SHA-256 of the empty input", async () => {
|
||||||
|
// Well-known constant: SHA-256 of zero bytes.
|
||||||
|
const digest = await fingerprintBytes(new Uint8Array(0));
|
||||||
|
expect(digest).toBe(
|
||||||
|
"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("is deterministic across repeated calls on the same bytes", async () => {
|
||||||
|
const bytes = new Uint8Array([1, 2, 3, 4, 5, 6, 7, 8]);
|
||||||
|
const a = await fingerprintBytes(bytes);
|
||||||
|
const b = await fingerprintBytes(bytes);
|
||||||
|
expect(a).toBe(b);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("does not mutate or consume the input buffer", async () => {
|
||||||
|
const bytes = new Uint8Array([9, 8, 7]);
|
||||||
|
await fingerprintBytes(bytes);
|
||||||
|
expect([...bytes]).toEqual([9, 8, 7]);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("distinguishes single-bit differences", async () => {
|
||||||
|
const a = await fingerprintBytes(new Uint8Array([0]));
|
||||||
|
const b = await fingerprintBytes(new Uint8Array([1]));
|
||||||
|
expect(a).not.toBe(b);
|
||||||
|
});
|
||||||
|
});
|
||||||
31
src/pdf/fingerprint.ts
Normal file
31
src/pdf/fingerprint.ts
Normal file
|
|
@ -0,0 +1,31 @@
|
||||||
|
/**
|
||||||
|
* SHA-256 fingerprint of raw document bytes.
|
||||||
|
*
|
||||||
|
* Implements the fingerprint half of `wiki/ArchitectureOverview.md` §3.4
|
||||||
|
* (the "compute fingerprint" pipeline step) and populates
|
||||||
|
* `Document.fingerprint` (`wiki/SharedContracts.md` §1).
|
||||||
|
*
|
||||||
|
* Uses Web Crypto's `crypto.subtle.digest`, which is available in browsers
|
||||||
|
* and in Node ≥ 20 (where it is exposed on `globalThis.crypto`). No
|
||||||
|
* platform branching — the API is the same in both environments.
|
||||||
|
*/
|
||||||
|
|
||||||
|
export async function fingerprintBytes(bytes: Uint8Array): Promise<string> {
|
||||||
|
// Copy into a fresh ArrayBuffer (not SharedArrayBuffer) so the digest call
|
||||||
|
// satisfies TS's updated `BufferSource` type, which excludes
|
||||||
|
// `SharedArrayBuffer`. The copy is O(n) — fine even for large PDFs since
|
||||||
|
// SHA-256 itself is already O(n).
|
||||||
|
const ab = new ArrayBuffer(bytes.byteLength);
|
||||||
|
new Uint8Array(ab).set(bytes);
|
||||||
|
const digest = await crypto.subtle.digest("SHA-256", ab);
|
||||||
|
return bytesToHex(new Uint8Array(digest));
|
||||||
|
}
|
||||||
|
|
||||||
|
function bytesToHex(bytes: Uint8Array): string {
|
||||||
|
let hex = "";
|
||||||
|
for (let i = 0; i < bytes.length; i++) {
|
||||||
|
const b = bytes[i]!;
|
||||||
|
hex += (b < 0x10 ? "0" : "") + b.toString(16);
|
||||||
|
}
|
||||||
|
return hex;
|
||||||
|
}
|
||||||
119
src/pdf/ingest.test.ts
Normal file
119
src/pdf/ingest.test.ts
Normal file
|
|
@ -0,0 +1,119 @@
|
||||||
|
/**
|
||||||
|
* Fixture-driven contract tests for the PDF ingest pipeline.
|
||||||
|
*
|
||||||
|
* For each fixture in the shared corpus manifest (ADR-0002):
|
||||||
|
* 1. Read the PDF bytes from disk.
|
||||||
|
* 2. Run `ingestPdf` end-to-end.
|
||||||
|
* 3. Assert the resulting Document + DocumentRepresentation honour the
|
||||||
|
* manifest contract: media type is application/pdf, fingerprint is a
|
||||||
|
* 64-hex SHA-256, pageMap matches `page_count`, canonicalText contains
|
||||||
|
* `known_good_quote`, and the offsetMap covers `[0, canonicalText.length)`
|
||||||
|
* with no gaps.
|
||||||
|
*
|
||||||
|
* Corpus resolution and skip-on-absent behaviour live in `tests/fixtures.ts`.
|
||||||
|
*/
|
||||||
|
|
||||||
|
import { describe, expect, it } from "vitest";
|
||||||
|
|
||||||
|
import { ingestPdf } from "./ingest";
|
||||||
|
import { fingerprintBytes } from "./fingerprint";
|
||||||
|
import {
|
||||||
|
fixtureBytes,
|
||||||
|
fixtureCorpusAvailable,
|
||||||
|
loadFixtures,
|
||||||
|
} from "../../tests/fixtures";
|
||||||
|
|
||||||
|
const hasCorpus = fixtureCorpusAvailable();
|
||||||
|
const FIXTURES = hasCorpus ? loadFixtures() : [];
|
||||||
|
|
||||||
|
describe.skipIf(!hasCorpus)("ingestPdf — fixture corpus", { timeout: 30_000 }, () => {
|
||||||
|
for (const fixture of FIXTURES) {
|
||||||
|
describe(fixture.id, () => {
|
||||||
|
const bytes = fixtureBytes(fixture.filename);
|
||||||
|
|
||||||
|
it("produces a Document with PDF media type and SHA-256 fingerprint", async () => {
|
||||||
|
const { document } = await ingestPdf(bytes, { filename: fixture.filename });
|
||||||
|
expect(document.mediaType).toBe("application/pdf");
|
||||||
|
expect(document.fingerprint).toMatch(/^[0-9a-f]{64}$/);
|
||||||
|
expect(document.title).toBe(fixture.filename);
|
||||||
|
const expected = await fingerprintBytes(bytes);
|
||||||
|
expect(document.fingerprint).toBe(expected);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("produces a pdf-text representation with the expected page count", async () => {
|
||||||
|
const { representation } = await ingestPdf(bytes);
|
||||||
|
expect(representation.representationType).toBe("pdf-text");
|
||||||
|
expect(representation.pageMap?.length).toBe(fixture.page_count);
|
||||||
|
expect(representation.offsetMap?.length).toBe(fixture.page_count);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("canonical text contains the manifest's known-good quote", async () => {
|
||||||
|
const { representation } = await ingestPdf(bytes);
|
||||||
|
const text = representation.canonicalText ?? "";
|
||||||
|
expect(text).toContain(fixture.known_good_quote);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("offsetMap is gap-free and covers [0, canonicalText.length)", async () => {
|
||||||
|
const { representation } = await ingestPdf(bytes);
|
||||||
|
const text = representation.canonicalText ?? "";
|
||||||
|
const offsets = representation.offsetMap ?? [];
|
||||||
|
expect(offsets.length).toBeGreaterThan(0);
|
||||||
|
expect(offsets[0]!.globalStart).toBe(0);
|
||||||
|
expect(offsets.at(-1)!.globalEnd).toBe(text.length);
|
||||||
|
for (let i = 0; i < offsets.length; i++) {
|
||||||
|
const r = offsets[i]!;
|
||||||
|
expect(r.page).toBe(i + 1);
|
||||||
|
expect(r.globalEnd - r.globalStart).toBe(r.pageLength);
|
||||||
|
if (i > 0) expect(r.globalStart).toBe(offsets[i - 1]!.globalEnd);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
it("pageMap entries have positive width and height in user-space points", async () => {
|
||||||
|
const { representation } = await ingestPdf(bytes);
|
||||||
|
const pages = representation.pageMap ?? [];
|
||||||
|
for (let i = 0; i < pages.length; i++) {
|
||||||
|
const p = pages[i]!;
|
||||||
|
expect(p.page).toBe(i + 1);
|
||||||
|
expect(p.width).toBeGreaterThan(0);
|
||||||
|
expect(p.height).toBeGreaterThan(0);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
});
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
describe.skipIf(!hasCorpus)("ingestPdf — option handling", { timeout: 30_000 }, () => {
|
||||||
|
const fixture = FIXTURES[0];
|
||||||
|
const bytes = fixture ? fixtureBytes(fixture.filename) : new Uint8Array();
|
||||||
|
|
||||||
|
it("uses explicit title over filename", async () => {
|
||||||
|
const { document } = await ingestPdf(bytes, {
|
||||||
|
filename: fixture!.filename,
|
||||||
|
title: "Custom Title",
|
||||||
|
});
|
||||||
|
expect(document.title).toBe("Custom Title");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("omits title entirely when neither filename nor title is supplied", async () => {
|
||||||
|
const { document } = await ingestPdf(bytes);
|
||||||
|
expect(document.title).toBeUndefined();
|
||||||
|
});
|
||||||
|
|
||||||
|
it("propagates uri and metadata when supplied", async () => {
|
||||||
|
const { document } = await ingestPdf(bytes, {
|
||||||
|
uri: "file:///example.pdf",
|
||||||
|
metadata: { source: "test" },
|
||||||
|
});
|
||||||
|
expect(document.uri).toBe("file:///example.pdf");
|
||||||
|
expect(document.metadata).toEqual({ source: "test" });
|
||||||
|
});
|
||||||
|
|
||||||
|
it("accepts ArrayBuffer input", async () => {
|
||||||
|
const ab = bytes.buffer.slice(
|
||||||
|
bytes.byteOffset,
|
||||||
|
bytes.byteOffset + bytes.byteLength,
|
||||||
|
) as ArrayBuffer;
|
||||||
|
const { document } = await ingestPdf(ab);
|
||||||
|
expect(document.fingerprint).toMatch(/^[0-9a-f]{64}$/);
|
||||||
|
});
|
||||||
|
});
|
||||||
88
src/pdf/ingest.ts
Normal file
88
src/pdf/ingest.ts
Normal file
|
|
@ -0,0 +1,88 @@
|
||||||
|
/**
|
||||||
|
* PDF ingest pipeline → `{ document, representation }`.
|
||||||
|
*
|
||||||
|
* Implements `wiki/ArchitectureOverview.md` §3.4 ("Raw Source → identify
|
||||||
|
* media type → compute fingerprint → extract metadata → extract canonical
|
||||||
|
* text → build format-specific maps → persist Document +
|
||||||
|
* DocumentRepresentation") for the PDF source format.
|
||||||
|
*
|
||||||
|
* Ingest is a pure function over bytes: it does not persist anything. The
|
||||||
|
* caller (engine repositories, or the app layer) writes the returned
|
||||||
|
* Document + DocumentRepresentation into the chosen store.
|
||||||
|
*/
|
||||||
|
|
||||||
|
import {
|
||||||
|
newId,
|
||||||
|
type Document,
|
||||||
|
type DocumentRepresentation,
|
||||||
|
} from "@citation-evidence/engine/shared";
|
||||||
|
import { extractPdf } from "./extract";
|
||||||
|
import { fingerprintBytes } from "./fingerprint";
|
||||||
|
|
||||||
|
const PDF_MEDIA_TYPE = "application/pdf";
|
||||||
|
|
||||||
|
export interface IngestPdfOptions {
|
||||||
|
/** Original filename, used as the default title when no title is given. */
|
||||||
|
readonly filename?: string;
|
||||||
|
/** Optional pre-existing title (overrides filename). */
|
||||||
|
readonly title?: string;
|
||||||
|
/** Optional source URI (e.g. file:// or https://). */
|
||||||
|
readonly uri?: string;
|
||||||
|
/** Free-form metadata persisted on the Document record. */
|
||||||
|
readonly metadata?: Readonly<Record<string, unknown>>;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface IngestPdfResult {
|
||||||
|
readonly document: Document;
|
||||||
|
readonly representation: DocumentRepresentation;
|
||||||
|
}
|
||||||
|
|
||||||
|
export type IngestPdfInput = Uint8Array | ArrayBuffer | Blob;
|
||||||
|
|
||||||
|
export async function ingestPdf(
|
||||||
|
input: IngestPdfInput,
|
||||||
|
options: IngestPdfOptions = {},
|
||||||
|
): Promise<IngestPdfResult> {
|
||||||
|
const bytes = await toBytes(input);
|
||||||
|
const [fingerprint, extraction] = await Promise.all([
|
||||||
|
fingerprintBytes(bytes),
|
||||||
|
extractPdf(bytes),
|
||||||
|
]);
|
||||||
|
|
||||||
|
const now = new Date().toISOString();
|
||||||
|
const documentId = newId("document");
|
||||||
|
const representationId = newId("representation");
|
||||||
|
const title = options.title ?? options.filename;
|
||||||
|
|
||||||
|
const document: Document = {
|
||||||
|
id: documentId,
|
||||||
|
mediaType: PDF_MEDIA_TYPE,
|
||||||
|
fingerprint,
|
||||||
|
createdAt: now,
|
||||||
|
updatedAt: now,
|
||||||
|
...(title !== undefined ? { title } : {}),
|
||||||
|
...(options.uri !== undefined ? { uri: options.uri } : {}),
|
||||||
|
...(options.metadata !== undefined ? { metadata: options.metadata } : {}),
|
||||||
|
};
|
||||||
|
|
||||||
|
const representation: DocumentRepresentation = {
|
||||||
|
id: representationId,
|
||||||
|
documentId,
|
||||||
|
representationType: "pdf-text",
|
||||||
|
contentHash: fingerprint,
|
||||||
|
canonicalText: extraction.canonicalText,
|
||||||
|
pageMap: extraction.pageMap,
|
||||||
|
offsetMap: extraction.offsetMap,
|
||||||
|
generatedAt: now,
|
||||||
|
};
|
||||||
|
|
||||||
|
return { document, representation };
|
||||||
|
}
|
||||||
|
|
||||||
|
async function toBytes(input: IngestPdfInput): Promise<Uint8Array> {
|
||||||
|
if (input instanceof Uint8Array) return input;
|
||||||
|
if (input instanceof ArrayBuffer) return new Uint8Array(input);
|
||||||
|
// Blob (covers `File` in browsers — File extends Blob).
|
||||||
|
const buf = await input.arrayBuffer();
|
||||||
|
return new Uint8Array(buf);
|
||||||
|
}
|
||||||
51
tests/fixtures.ts
Normal file
51
tests/fixtures.ts
Normal file
|
|
@ -0,0 +1,51 @@
|
||||||
|
/**
|
||||||
|
* Fixture corpus access for evidence-source tests.
|
||||||
|
*
|
||||||
|
* The corpus is owned by the sibling `citation-evidence` checkout (ADR-0002);
|
||||||
|
* it is not duplicated into this repo. Resolution order:
|
||||||
|
* 1. `EVIDENCE_SOURCE_FIXTURE_DIR` env var, if set.
|
||||||
|
* 2. `<repo>/../citation-evidence/fixtures/pdfs` (sibling-checkout default).
|
||||||
|
*
|
||||||
|
* When the corpus is unavailable, `fixtureCorpusAvailable()` returns false so
|
||||||
|
* fixture-driven suites can skip cleanly instead of failing.
|
||||||
|
*/
|
||||||
|
|
||||||
|
import { existsSync, readFileSync } from "node:fs";
|
||||||
|
import { dirname, resolve } from "node:path";
|
||||||
|
import { fileURLToPath } from "node:url";
|
||||||
|
|
||||||
|
const __dirname = dirname(fileURLToPath(import.meta.url));
|
||||||
|
|
||||||
|
export const FIXTURE_DIR = process.env.EVIDENCE_SOURCE_FIXTURE_DIR
|
||||||
|
? resolve(process.env.EVIDENCE_SOURCE_FIXTURE_DIR)
|
||||||
|
: resolve(__dirname, "../../citation-evidence/fixtures/pdfs");
|
||||||
|
|
||||||
|
export interface Fixture {
|
||||||
|
readonly id: string;
|
||||||
|
readonly filename: string;
|
||||||
|
readonly page_count: number;
|
||||||
|
readonly known_good_quote: string;
|
||||||
|
readonly known_good_quote_page: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
interface Manifest {
|
||||||
|
readonly fixtures: readonly Fixture[];
|
||||||
|
}
|
||||||
|
|
||||||
|
export function fixturePath(filename: string): string {
|
||||||
|
return resolve(FIXTURE_DIR, filename);
|
||||||
|
}
|
||||||
|
|
||||||
|
export function fixtureBytes(filename: string): Uint8Array {
|
||||||
|
return new Uint8Array(readFileSync(fixturePath(filename)));
|
||||||
|
}
|
||||||
|
|
||||||
|
export function fixtureCorpusAvailable(): boolean {
|
||||||
|
return existsSync(resolve(FIXTURE_DIR, "manifest.json"));
|
||||||
|
}
|
||||||
|
|
||||||
|
export function loadFixtures(): readonly Fixture[] {
|
||||||
|
const manifestPath = resolve(FIXTURE_DIR, "manifest.json");
|
||||||
|
const manifest = JSON.parse(readFileSync(manifestPath, "utf8")) as Manifest;
|
||||||
|
return manifest.fixtures;
|
||||||
|
}
|
||||||
17
tests/setup-pdf-worker.ts
Normal file
17
tests/setup-pdf-worker.ts
Normal file
|
|
@ -0,0 +1,17 @@
|
||||||
|
/**
|
||||||
|
* Vitest setup: point PDF.js at its legacy worker bundle for Node runs.
|
||||||
|
*
|
||||||
|
* `extractPdf` deliberately does no worker setup (so the same module loads in
|
||||||
|
* browsers and Node). In Node tests we configure `GlobalWorkerOptions.workerSrc`
|
||||||
|
* here, once per test file, before any extraction runs. The legacy bundle ships
|
||||||
|
* as plain JS and works through PDF.js's fake-worker fallback when no real
|
||||||
|
* Worker is available.
|
||||||
|
*/
|
||||||
|
|
||||||
|
import { createRequire } from "node:module";
|
||||||
|
import * as pdfjs from "pdfjs-dist";
|
||||||
|
|
||||||
|
const require = createRequire(import.meta.url);
|
||||||
|
pdfjs.GlobalWorkerOptions.workerSrc = require.resolve(
|
||||||
|
"pdfjs-dist/legacy/build/pdf.worker.mjs",
|
||||||
|
);
|
||||||
25
tsconfig.json
Normal file
25
tsconfig.json
Normal file
|
|
@ -0,0 +1,25 @@
|
||||||
|
{
|
||||||
|
"compilerOptions": {
|
||||||
|
"target": "ES2022",
|
||||||
|
"lib": ["ES2022", "DOM"],
|
||||||
|
"module": "ESNext",
|
||||||
|
"moduleResolution": "Bundler",
|
||||||
|
"strict": true,
|
||||||
|
"noImplicitOverride": true,
|
||||||
|
"noFallthroughCasesInSwitch": true,
|
||||||
|
"noUncheckedIndexedAccess": true,
|
||||||
|
"exactOptionalPropertyTypes": true,
|
||||||
|
"noUnusedLocals": true,
|
||||||
|
"noUnusedParameters": true,
|
||||||
|
"noEmit": true,
|
||||||
|
"esModuleInterop": true,
|
||||||
|
"forceConsistentCasingInFileNames": true,
|
||||||
|
"isolatedModules": true,
|
||||||
|
"skipLibCheck": true,
|
||||||
|
"resolveJsonModule": true,
|
||||||
|
"verbatimModuleSyntax": true,
|
||||||
|
"types": ["node"]
|
||||||
|
},
|
||||||
|
"include": ["src", "tests", "vitest.config.ts", "eslint.config.js"],
|
||||||
|
"exclude": ["node_modules", "dist"]
|
||||||
|
}
|
||||||
14
vitest.config.ts
Normal file
14
vitest.config.ts
Normal file
|
|
@ -0,0 +1,14 @@
|
||||||
|
import { defineConfig } from "vitest/config";
|
||||||
|
|
||||||
|
// Headless-only test surface: every test runs in Node, which has full
|
||||||
|
// pdfjs-dist legacy-worker support. There is no DOM/React surface in this
|
||||||
|
// repo, so unlike the umbrella app we do not register a happy-dom
|
||||||
|
// environment. The browser upload helpers (`src/browser/`) are exercised
|
||||||
|
// with injected URL stubs, so they run in Node too.
|
||||||
|
export default defineConfig({
|
||||||
|
test: {
|
||||||
|
globals: false,
|
||||||
|
include: ["src/**/*.test.ts", "tests/**/*.test.ts"],
|
||||||
|
setupFiles: ["tests/setup-pdf-worker.ts"],
|
||||||
|
},
|
||||||
|
});
|
||||||
|
|
@ -4,7 +4,7 @@ type: workplan
|
||||||
title: "Bootstrap standalone source repo from citation-evidence PDF slice"
|
title: "Bootstrap standalone source repo from citation-evidence PDF slice"
|
||||||
domain: infotech
|
domain: infotech
|
||||||
repo: evidence-source
|
repo: evidence-source
|
||||||
status: ready
|
status: finished
|
||||||
owner: codex
|
owner: codex
|
||||||
topic_slug: citation_evidence_mvp
|
topic_slug: citation_evidence_mvp
|
||||||
created: "2026-06-21"
|
created: "2026-06-21"
|
||||||
|
|
@ -64,7 +64,7 @@ T01 extraction boundary and import plan
|
||||||
|
|
||||||
```task
|
```task
|
||||||
id: ESRC-WP-0001-T01
|
id: ESRC-WP-0001-T01
|
||||||
status: todo
|
status: done
|
||||||
priority: high
|
priority: high
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|
@ -93,7 +93,7 @@ the extraction relies on copying `citation-engine` shared types locally.
|
||||||
|
|
||||||
```task
|
```task
|
||||||
id: ESRC-WP-0001-T02
|
id: ESRC-WP-0001-T02
|
||||||
status: todo
|
status: done
|
||||||
priority: high
|
priority: high
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|
@ -119,7 +119,7 @@ runtime host.
|
||||||
|
|
||||||
```task
|
```task
|
||||||
id: ESRC-WP-0001-T03
|
id: ESRC-WP-0001-T03
|
||||||
status: todo
|
status: done
|
||||||
priority: high
|
priority: high
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|
@ -147,7 +147,7 @@ Acceptance: a consumer can pass PDF bytes to this repo and receive a valid
|
||||||
|
|
||||||
```task
|
```task
|
||||||
id: ESRC-WP-0001-T04
|
id: ESRC-WP-0001-T04
|
||||||
status: todo
|
status: done
|
||||||
priority: high
|
priority: high
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|
@ -176,7 +176,7 @@ browser/session-only behavior after extraction.
|
||||||
|
|
||||||
```task
|
```task
|
||||||
id: ESRC-WP-0001-T05
|
id: ESRC-WP-0001-T05
|
||||||
status: todo
|
status: done
|
||||||
priority: high
|
priority: high
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|
@ -203,7 +203,7 @@ source-to-anchor round-trip remains an umbrella integration test.
|
||||||
|
|
||||||
```task
|
```task
|
||||||
id: ESRC-WP-0001-T06
|
id: ESRC-WP-0001-T06
|
||||||
status: todo
|
status: done
|
||||||
priority: high
|
priority: high
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|
@ -227,7 +227,7 @@ package rather than the local `src/source/` copy for the PDF slice.
|
||||||
|
|
||||||
```task
|
```task
|
||||||
id: ESRC-WP-0001-T07
|
id: ESRC-WP-0001-T07
|
||||||
status: todo
|
status: done
|
||||||
priority: medium
|
priority: medium
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|
|
||||||
55
workplans/ESRC-WP-0002-html-markdown-representations.md
Normal file
55
workplans/ESRC-WP-0002-html-markdown-representations.md
Normal file
|
|
@ -0,0 +1,55 @@
|
||||||
|
---
|
||||||
|
id: ESRC-WP-0002
|
||||||
|
type: workplan
|
||||||
|
title: "HTML and Markdown source representations"
|
||||||
|
domain: infotech
|
||||||
|
repo: evidence-source
|
||||||
|
status: proposed
|
||||||
|
owner: codex
|
||||||
|
topic_slug: citation_evidence_mvp
|
||||||
|
created: "2026-07-08"
|
||||||
|
updated: "2026-07-08"
|
||||||
|
spec_refs:
|
||||||
|
- README.md
|
||||||
|
- docs/ADR-0001-extraction-boundary.md
|
||||||
|
- ../citation-engine/src/shared/document.ts
|
||||||
|
---
|
||||||
|
|
||||||
|
# ESRC-WP-0002 — HTML and Markdown source representations
|
||||||
|
|
||||||
|
The PDF slice (ESRC-WP-0001) established the ingest boundary and contract shape.
|
||||||
|
This workplan extends ingest to HTML and Markdown sources, producing the same
|
||||||
|
`{ document, representation }` pair over engine-owned contracts.
|
||||||
|
|
||||||
|
## Open questions
|
||||||
|
|
||||||
|
- Which `representationType` values do HTML/MD map to, and do they need new
|
||||||
|
`DocumentRepresentation` fields in `citation-engine`?
|
||||||
|
- How is canonical text + offset map derived for reflowable formats that have
|
||||||
|
no fixed pages? (Page map may be empty or synthetic.)
|
||||||
|
|
||||||
|
## Sketch
|
||||||
|
|
||||||
|
```task
|
||||||
|
id: ESRC-WP-0002-T01
|
||||||
|
status: todo
|
||||||
|
priority: medium
|
||||||
|
```
|
||||||
|
Define the HTML/MD representation contract with `citation-engine`; agree
|
||||||
|
`representationType` and any offset/page semantics for pageless formats.
|
||||||
|
|
||||||
|
```task
|
||||||
|
id: ESRC-WP-0002-T02
|
||||||
|
status: todo
|
||||||
|
priority: medium
|
||||||
|
```
|
||||||
|
Implement `ingestHtml` / `ingestMarkdown` in `src/html/` and `src/markdown/`,
|
||||||
|
reusing `fingerprintBytes` and the canonical-text normalizer.
|
||||||
|
|
||||||
|
```task
|
||||||
|
id: ESRC-WP-0002-T03
|
||||||
|
status: todo
|
||||||
|
priority: medium
|
||||||
|
```
|
||||||
|
Add fixture-driven contract tests mirroring the PDF suite; decide fixture
|
||||||
|
ownership per ADR-0002.
|
||||||
48
workplans/ESRC-WP-0003-metadata-enrichment.md
Normal file
48
workplans/ESRC-WP-0003-metadata-enrichment.md
Normal file
|
|
@ -0,0 +1,48 @@
|
||||||
|
---
|
||||||
|
id: ESRC-WP-0003
|
||||||
|
type: workplan
|
||||||
|
title: "Document metadata enrichment beyond pass-through options"
|
||||||
|
domain: infotech
|
||||||
|
repo: evidence-source
|
||||||
|
status: proposed
|
||||||
|
owner: codex
|
||||||
|
topic_slug: citation_evidence_mvp
|
||||||
|
created: "2026-07-08"
|
||||||
|
updated: "2026-07-08"
|
||||||
|
spec_refs:
|
||||||
|
- README.md
|
||||||
|
- src/pdf/ingest.ts
|
||||||
|
---
|
||||||
|
|
||||||
|
# ESRC-WP-0003 — Metadata enrichment
|
||||||
|
|
||||||
|
Today `ingestPdf` only passes through caller-supplied `title`/`uri`/`metadata`.
|
||||||
|
This workplan extracts intrinsic document metadata (PDF info dictionary,
|
||||||
|
embedded XMP, page-derived signals like author/creation date/title) and folds
|
||||||
|
it into the `Document` record without breaking the pure-over-bytes contract.
|
||||||
|
|
||||||
|
## Sketch
|
||||||
|
|
||||||
|
```task
|
||||||
|
id: ESRC-WP-0003-T01
|
||||||
|
status: todo
|
||||||
|
priority: low
|
||||||
|
```
|
||||||
|
Enumerate available PDF metadata sources (info dict, XMP) via PDF.js and decide
|
||||||
|
precedence vs. caller-supplied options (caller wins).
|
||||||
|
|
||||||
|
```task
|
||||||
|
id: ESRC-WP-0003-T02
|
||||||
|
status: todo
|
||||||
|
priority: low
|
||||||
|
```
|
||||||
|
Implement extraction into a normalized metadata shape; keep it optional so
|
||||||
|
minimal-metadata PDFs still ingest cleanly.
|
||||||
|
|
||||||
|
```task
|
||||||
|
id: ESRC-WP-0003-T03
|
||||||
|
status: todo
|
||||||
|
priority: low
|
||||||
|
```
|
||||||
|
Contract tests over the fixture corpus asserting extracted vs. overridden
|
||||||
|
metadata precedence.
|
||||||
47
workplans/ESRC-WP-0004-citation-recovery.md
Normal file
47
workplans/ESRC-WP-0004-citation-recovery.md
Normal file
|
|
@ -0,0 +1,47 @@
|
||||||
|
---
|
||||||
|
id: ESRC-WP-0004
|
||||||
|
type: workplan
|
||||||
|
title: "Local citation recovery and external source discovery hooks"
|
||||||
|
domain: infotech
|
||||||
|
repo: evidence-source
|
||||||
|
status: proposed
|
||||||
|
owner: codex
|
||||||
|
topic_slug: citation_evidence_mvp
|
||||||
|
created: "2026-07-08"
|
||||||
|
updated: "2026-07-08"
|
||||||
|
spec_refs:
|
||||||
|
- INTENT.md
|
||||||
|
- README.md
|
||||||
|
---
|
||||||
|
|
||||||
|
# ESRC-WP-0004 — Citation recovery and source discovery hooks
|
||||||
|
|
||||||
|
`evidence-source`'s original intent includes recovering citations when a stored
|
||||||
|
selector goes stale and discovering the external source a document was drawn
|
||||||
|
from. This is the largest deferred strand and depends on the anchoring layer's
|
||||||
|
selector model.
|
||||||
|
|
||||||
|
## Open questions
|
||||||
|
|
||||||
|
- What is the recovery contract — does this repo *detect* stale selectors, or
|
||||||
|
only provide the re-ingest/re-fingerprint primitives the anchor layer calls?
|
||||||
|
- Where does external source discovery (URL/DOI resolution) belong relative to
|
||||||
|
`citation-engine` and the umbrella?
|
||||||
|
|
||||||
|
## Sketch
|
||||||
|
|
||||||
|
```task
|
||||||
|
id: ESRC-WP-0004-T01
|
||||||
|
status: todo
|
||||||
|
priority: low
|
||||||
|
```
|
||||||
|
Design the recovery boundary with the anchor layer: define what
|
||||||
|
`evidence-source` owns vs. what stays in anchoring.
|
||||||
|
|
||||||
|
```task
|
||||||
|
id: ESRC-WP-0004-T02
|
||||||
|
status: todo
|
||||||
|
priority: low
|
||||||
|
```
|
||||||
|
Prototype re-ingest/re-fingerprint recovery primitives and a discovery hook
|
||||||
|
interface (no network coupling baked into the core).
|
||||||
Loading…
Add table
Add a link
Reference in a new issue