feat(pdf): extract standalone PDF ingest package (ESRC-WP-0001)
Bootstrap evidence-source from the citation-evidence src/source PDF slice. - Headless core (src/pdf): ingest/extract/fingerprint, importing domain contracts from @citation-evidence/engine/shared (no local copies) - Browser upload helpers isolated under src/browser behind a ./browser entry point, with an eslint boundary keeping the core browser-free - pnpm/TS/vitest/eslint scaffold; 52 tests (contract + determinism) - Fixtures resolved from the sibling citation-evidence checkout, not duplicated (real PII) — see docs/ADR-0002; suites skip when absent - Boundary + fixture decisions recorded as docs/ADR-0001 / ADR-0002 - README/SCOPE rewritten; capability.infotech.pdf-evidence-ingest registered, NO_CAPABILITIES removed - Follow-on workplans ESRC-WP-0002..0004 queued; ESRC-WP-0001 finished Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
parent
2fd715ba45
commit
cb93c322c0
31 changed files with 5066 additions and 105 deletions
|
|
@ -1,4 +1,4 @@
|
|||
**Purpose:** Document ingestion, extraction, fingerprinting, citation recovery. Depends only on citation-engine. INTENT-only during umbrella-first MVP.
|
||||
**Purpose:** Document ingestion, extraction, fingerprinting, citation recovery. Depends only on citation-engine. The PDF ingest slice is implemented and consumed by citation-evidence (ESRC-WP-0001); HTML/MD, metadata enrichment, and citation recovery are deferred to ESRC-WP-0002..0004.
|
||||
|
||||
**Domain:** infotech
|
||||
**Repo slug:** evidence-source
|
||||
|
|
|
|||
6
.gitignore
vendored
6
.gitignore
vendored
|
|
@ -174,3 +174,9 @@ cython_debug/
|
|||
# PyPI configuration file
|
||||
.pypirc
|
||||
|
||||
|
||||
# ---> Node / TypeScript
|
||||
node_modules/
|
||||
coverage/
|
||||
*.tsbuildinfo
|
||||
.vitest/
|
||||
|
|
|
|||
78
README.md
78
README.md
|
|
@ -1,16 +1,72 @@
|
|||
# evidence-source
|
||||
|
||||
Document source, ingestion, extraction, metadata, and citation recovery —
|
||||
PDF/HTML/MD ingest, fingerprinting, page-/offset-map construction,
|
||||
canonical-text extraction, and the recovery behavior for stale selectors.
|
||||
Headless document ingest for the citation-evidence ecosystem. Turns raw PDF
|
||||
bytes into engine-owned evidence contracts — a `Document` (media type,
|
||||
SHA-256 fingerprint, optional title/uri/metadata) and a
|
||||
`DocumentRepresentation` (`pdf-text`: canonical text, page map, gap-free
|
||||
offset map). Ingest is pure over bytes: no persistence, no viewer state, no
|
||||
React.
|
||||
|
||||
## MVP status: INTENT only
|
||||
## Status
|
||||
|
||||
During the citation-evidence MVP, code lives upstream in
|
||||
[`citation-evidence`](../citation-evidence/) under `src/source/`. This repo
|
||||
currently holds `INTENT.md` describing what will move here. Contract
|
||||
changes belong in
|
||||
[`citation-evidence/wiki/SharedContracts.md`](../citation-evidence/wiki/SharedContracts.md),
|
||||
not here.
|
||||
**Implemented: the PDF slice.** As of ESRC-WP-0001 this repo hosts the
|
||||
extracted PDF ingest core that previously lived in `citation-evidence/src/source/`.
|
||||
HTML/Markdown representations, richer metadata enrichment, and citation
|
||||
recovery are deferred to follow-on workplans (see `workplans/`).
|
||||
|
||||
Per the dependency map, source depends on `shared/` and `engine/` only.
|
||||
## Install
|
||||
|
||||
Requires a sibling checkout of `citation-engine` (domain contracts) and, for
|
||||
tests, `citation-evidence` (fixture corpus — see `docs/ADR-0002-fixture-ownership.md`).
|
||||
|
||||
```bash
|
||||
pnpm install # resolves @citation-evidence/engine via link:../citation-engine
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
```ts
|
||||
import { ingestPdf } from "@citation-evidence/evidence-source";
|
||||
|
||||
const { document, representation } = await ingestPdf(pdfBytes, {
|
||||
filename: "contract.pdf",
|
||||
});
|
||||
// document.fingerprint -> SHA-256 hex
|
||||
// representation.canonicalText / pageMap / offsetMap
|
||||
```
|
||||
|
||||
Browser upload helpers (in-memory `blob:` byte store + `ingestPdfFromFile`)
|
||||
live behind a separate, clearly-isolated entry point so headless consumers do
|
||||
not pull in browser machinery:
|
||||
|
||||
```ts
|
||||
import { createPdfByteStore, ingestPdfFromFile }
|
||||
from "@citation-evidence/evidence-source/browser";
|
||||
```
|
||||
|
||||
Extraction requires the host to configure the PDF.js worker
|
||||
(`GlobalWorkerOptions.workerSrc`) before calling `extractPdf`/`ingestPdf`; the
|
||||
module itself does no worker setup so it loads cleanly in Node and browsers.
|
||||
|
||||
## Dev commands
|
||||
|
||||
```bash
|
||||
pnpm test # vitest (fixture suites skip if the corpus is absent)
|
||||
pnpm typecheck # tsc --noEmit
|
||||
pnpm lint # eslint
|
||||
```
|
||||
|
||||
Point `EVIDENCE_SOURCE_FIXTURE_DIR` at the PDF corpus when the sibling
|
||||
`citation-evidence` checkout is not at `../citation-evidence`.
|
||||
|
||||
## Architecture
|
||||
|
||||
- `src/pdf/` — headless core: `ingest`, `extract`, `fingerprint`. Runtime-agnostic.
|
||||
- `src/browser/` — browser upload surface: `byte-store`, `upload`. Separated by
|
||||
an ESLint boundary so the core never imports it.
|
||||
- Domain contracts come from `@citation-evidence/engine/shared`; none are
|
||||
copied locally.
|
||||
- Boundary and fixture decisions: `docs/ADR-0001`, `docs/ADR-0002`.
|
||||
|
||||
`viewer-url` resolution stays in the consuming app (`citation-evidence`), which
|
||||
imports this package's ingest core through a thin façade.
|
||||
|
|
|
|||
107
SCOPE.md
107
SCOPE.md
|
|
@ -8,130 +8,103 @@
|
|||
|
||||
## One-liner
|
||||
|
||||
<!-- Describe the purpose of this repository in one precise sentence. -->
|
||||
<!-- Example: "Provides a lightweight event router for Kubernetes-native systems." -->
|
||||
Headless PDF ingest that turns document bytes into engine-owned evidence
|
||||
contracts (fingerprint, canonical text, page/offset maps).
|
||||
|
||||
---
|
||||
|
||||
## Core Idea
|
||||
|
||||
<!-- What is the main capability or idea behind this repository? -->
|
||||
<!-- What problem does it try to solve? -->
|
||||
The rest of the citation-evidence ecosystem (anchoring, evidence linking,
|
||||
binders) needs a stable, runtime-agnostic way to go from *raw document bytes*
|
||||
to a `Document` + `DocumentRepresentation`. This repo owns that transformation
|
||||
for PDFs — pure over bytes, no persistence, no viewer, no UI framework.
|
||||
|
||||
---
|
||||
|
||||
## In Scope
|
||||
|
||||
<!-- What this repository is responsible for. -->
|
||||
<!-- Be explicit and concrete. -->
|
||||
|
||||
-
|
||||
-
|
||||
-
|
||||
- PDF byte ingest → `{ document, representation }` (`ingestPdf`)
|
||||
- PDF text extraction → canonical text + page map + gap-free offset map (`extractPdf`)
|
||||
- SHA-256 byte fingerprinting (`fingerprintBytes`)
|
||||
- Browser upload helpers behind a separate entry point (`createPdfByteStore`, `ingestPdfFromFile`)
|
||||
|
||||
---
|
||||
|
||||
## Out of Scope
|
||||
|
||||
<!-- What this repository deliberately does NOT do. -->
|
||||
<!-- This is often more important than "In Scope". -->
|
||||
|
||||
-
|
||||
-
|
||||
-
|
||||
- HTML / Markdown representations (deferred — see `workplans/`)
|
||||
- Persisting documents or representations (caller's job)
|
||||
- Viewer URL resolution and blob-vs-fixture policy (stays in `citation-evidence`)
|
||||
- Citation recovery / external source discovery (deferred)
|
||||
- Defining domain contracts — those are owned by `citation-engine`
|
||||
|
||||
---
|
||||
|
||||
## Relevant When
|
||||
|
||||
<!-- When should someone consider using or exploring this repository? -->
|
||||
|
||||
-
|
||||
-
|
||||
-
|
||||
- You have PDF bytes and need engine-shaped evidence contracts
|
||||
- You need a deterministic content fingerprint of document bytes
|
||||
- You are wiring a new consumer to ingest without pulling in a viewer
|
||||
|
||||
---
|
||||
|
||||
## Not Relevant When
|
||||
|
||||
<!-- When should someone ignore this repository? -->
|
||||
|
||||
-
|
||||
-
|
||||
-
|
||||
- You need HTML/Markdown ingest (not yet implemented)
|
||||
- You need viewer/session/UI behavior (see `citation-evidence`)
|
||||
- You are changing the `Document`/`DocumentRepresentation` contract (see `citation-engine`)
|
||||
|
||||
---
|
||||
|
||||
## Current State
|
||||
|
||||
<!-- Rough indication of maturity. No strict format required. -->
|
||||
|
||||
- Status: <!-- e.g. concept / experimental / active / stable / deprecated -->
|
||||
- Implementation: <!-- e.g. idea / partial / substantial / complete -->
|
||||
- Stability: <!-- e.g. unstable / evolving / stable -->
|
||||
- Usage: <!-- e.g. none / personal / internal / production -->
|
||||
|
||||
<!-- Add any notes that help set expectations. -->
|
||||
- Status: active
|
||||
- Implementation: partial (PDF slice implemented; HTML/MD and recovery deferred)
|
||||
- Stability: evolving
|
||||
- Usage: internal (consumed by `citation-evidence` across the repo boundary)
|
||||
|
||||
---
|
||||
|
||||
## How It Fits
|
||||
|
||||
<!-- Where does this repository sit in the bigger picture? -->
|
||||
|
||||
- Upstream dependencies:
|
||||
- Downstream consumers:
|
||||
- Often used with:
|
||||
- Upstream dependencies: `citation-engine` (`@citation-evidence/engine/shared`)
|
||||
- Downstream consumers: `citation-evidence` (umbrella app)
|
||||
- Often used with: `citation-engine`, `citation-evidence`
|
||||
|
||||
---
|
||||
|
||||
## Terminology
|
||||
|
||||
<!-- Terms that are important to understand this repo. -->
|
||||
<!-- Especially useful if naming differs from other repos. -->
|
||||
|
||||
- Preferred terms:
|
||||
- Also known as:
|
||||
- Potentially confusing terms:
|
||||
- Preferred terms: ingest, canonical text, fingerprint, representation
|
||||
- Also known as: "source" (its former location was `citation-evidence/src/source/`)
|
||||
- Potentially confusing terms: "representation" (a derived view of a document, not the document itself)
|
||||
|
||||
---
|
||||
|
||||
## Related / Overlapping Repositories
|
||||
|
||||
<!-- List repositories that have similar or adjacent responsibilities. -->
|
||||
<!-- Helps detect duplication and navigate the ecosystem. -->
|
||||
|
||||
- <repo-name> — <!-- how it relates -->
|
||||
- `citation-engine` — owns the shared `Document`/`DocumentRepresentation` contracts this repo produces
|
||||
- `citation-evidence` — umbrella app that consumes this ingest core; still owns viewer-url resolution
|
||||
|
||||
---
|
||||
|
||||
## Getting Oriented
|
||||
|
||||
<!-- If someone decides to look deeper, where should they start? -->
|
||||
|
||||
- Start with:
|
||||
- Key files / directories:
|
||||
- Entry points:
|
||||
- Start with: `README.md`, then `docs/ADR-0001-extraction-boundary.md`
|
||||
- Key files / directories: `src/pdf/` (headless core), `src/browser/` (upload helpers)
|
||||
- Entry points: `src/index.ts` (headless), `src/browser/index.ts` (browser)
|
||||
|
||||
---
|
||||
|
||||
## Provided Capabilities
|
||||
|
||||
<!-- What can this repo's domain provide to other domains on request? -->
|
||||
<!-- Each capability block is parsed by the state-hub capability catalog ingest. -->
|
||||
<!-- Remove the examples and add your own, or leave empty if none. -->
|
||||
|
||||
<!--
|
||||
```capability
|
||||
type: infrastructure
|
||||
title: Example capability title
|
||||
description: What this capability provides, in one or two sentences.
|
||||
keywords: [keyword1, keyword2, keyword3]
|
||||
```
|
||||
-->
|
||||
- `capability.infotech.pdf-evidence-ingest` — headless PDF → evidence contracts
|
||||
(see `registry/capabilities/`)
|
||||
|
||||
---
|
||||
|
||||
## Notes
|
||||
|
||||
<!-- Anything else worth knowing. Keep it short. -->
|
||||
Fixture corpus is owned upstream by `citation-evidence` and resolved from a
|
||||
sibling checkout (ADR-0002); it is not duplicated here.
|
||||
|
|
|
|||
78
docs/ADR-0001-extraction-boundary.md
Normal file
78
docs/ADR-0001-extraction-boundary.md
Normal file
|
|
@ -0,0 +1,78 @@
|
|||
# ADR-0001 — PDF extraction boundary for `evidence-source`
|
||||
|
||||
- **Status:** accepted
|
||||
- **Date:** 2026-07-08
|
||||
- **Workplan:** ESRC-WP-0001 (T01)
|
||||
|
||||
## Context
|
||||
|
||||
`evidence-source` is being bootstrapped from the PDF slice that currently
|
||||
lives in `../citation-evidence/src/source/`. That slice already implements a
|
||||
working, tested headless ingest pipeline plus a set of browser/session helpers.
|
||||
This ADR locks *which* files cross the boundary, *where* they land, and *which
|
||||
contracts* they may depend on, so the extraction is mechanical rather than a
|
||||
series of ad-hoc judgement calls.
|
||||
|
||||
The shared domain contracts (`Document`, `DocumentRepresentation`, `PageMap`,
|
||||
`OffsetMap`, `newId`, `normalize`, …) are owned by `citation-engine` and are
|
||||
published from its package as `@citation-evidence/engine/shared`. Nothing in
|
||||
this extraction copies those types locally.
|
||||
|
||||
## Decision
|
||||
|
||||
### In-scope — headless ingest core (`src/pdf/`)
|
||||
|
||||
Moved into this repo immediately; these are pure, runtime-agnostic, and the
|
||||
reason this repo exists:
|
||||
|
||||
| Upstream file | New home | Notes |
|
||||
| --- | --- | --- |
|
||||
| `src/source/pdf/ingest.ts` | `src/pdf/ingest.ts` | `ingestPdf` |
|
||||
| `src/source/pdf/extract.ts` | `src/pdf/extract.ts` | `extractPdf` |
|
||||
| `src/source/pdf/fingerprint.ts` | `src/pdf/fingerprint.ts` | `fingerprintBytes` |
|
||||
|
||||
The only change to these files is the import specifier: `@shared/*` becomes the
|
||||
published package subpath `@citation-evidence/engine/shared`.
|
||||
|
||||
### In-scope but explicitly isolated — browser helpers (`src/browser/`)
|
||||
|
||||
Useful for browser upload flows, but *not* headless ingest. They are kept in
|
||||
this repo behind a separate `./browser` entry point and documented as
|
||||
browser-facing so they never blur into the headless core:
|
||||
|
||||
| Upstream file | New home | Notes |
|
||||
| --- | --- | --- |
|
||||
| `src/source/pdf/byte-store.ts` | `src/browser/byte-store.ts` | in-memory `blob:` byte store |
|
||||
| `src/source/pdf/upload.ts` | `src/browser/upload.ts` | `ingestPdfFromFile` — thin wrapper over `ingestPdf` |
|
||||
|
||||
### Out of scope — stays in `citation-evidence`
|
||||
|
||||
| Upstream file | Reason |
|
||||
| --- | --- |
|
||||
| `src/source/pdf/viewer-url.ts` | Pure viewer/app concern: it encodes the umbrella's `/fixtures/pdfs/…` URL convention and the app's blob-vs-fixture fallback policy. It consumes `PdfByteStore` (a type this repo now owns) but the resolution *policy* belongs to the viewer. It stays upstream and imports the `PdfByteStore` type from this package. |
|
||||
| `tests/integration/anchor-source-roundtrip.test.ts` | Cross-subsystem (source ↔ anchor) contract. Remains an umbrella integration test; it is not a private test of this repo. |
|
||||
|
||||
### Contracts
|
||||
|
||||
All domain contracts come from `@citation-evidence/engine/shared`. This repo
|
||||
declares `@citation-evidence/engine` as a `link:../citation-engine` dependency
|
||||
and never re-declares `Document`/`DocumentRepresentation`/etc. locally.
|
||||
|
||||
### Fixture corpus
|
||||
|
||||
The PDF fixture corpus (`../citation-evidence/fixtures/pdfs/`) is **not** copied
|
||||
into this repo. The files are real personal/legal documents (utility statements,
|
||||
court letters, admission forms) containing PII; duplicating them into a package
|
||||
that advertises reusable ingest capability is undesirable. Instead this repo's
|
||||
tests resolve the corpus from the sibling checkout (default
|
||||
`../citation-evidence/fixtures/pdfs`, override via
|
||||
`EVIDENCE_SOURCE_FIXTURE_DIR`). This mirrors the sibling-checkout `link:` model
|
||||
already used for the engine dependency and keeps the corpus single-owned
|
||||
upstream. See ADR-0002.
|
||||
|
||||
## Consequences
|
||||
|
||||
- The extraction is a 5-file move (3 core + 2 browser) plus import rewrites.
|
||||
- `citation-evidence` keeps `viewer-url.ts` and the round-trip integration test.
|
||||
- Tests here require a sibling `citation-evidence` checkout for fixtures — an
|
||||
accepted coupling, identical in spirit to the engine `link:` dependency.
|
||||
43
docs/ADR-0002-fixture-ownership.md
Normal file
43
docs/ADR-0002-fixture-ownership.md
Normal file
|
|
@ -0,0 +1,43 @@
|
|||
# ADR-0002 — Fixture corpus ownership
|
||||
|
||||
- **Status:** accepted
|
||||
- **Date:** 2026-07-08
|
||||
- **Workplan:** ESRC-WP-0001 (T05)
|
||||
|
||||
## Context
|
||||
|
||||
The PDF ingest contract tests are driven by a fixture corpus
|
||||
(`fixtures/pdfs/` + `manifest.json`) currently owned by `citation-evidence`.
|
||||
The corpus is made of **real** personal and legal documents — utility-cost
|
||||
statements, a settlement letter, court correspondence, cemetery admission
|
||||
forms — containing names, addresses, and case details.
|
||||
|
||||
The workplan (T05) asks us to either copy the corpus into this repo or define a
|
||||
stable shared-fixture path.
|
||||
|
||||
## Decision
|
||||
|
||||
Do **not** copy the corpus into `evidence-source`. Resolve it from the sibling
|
||||
`citation-evidence` checkout instead:
|
||||
|
||||
- default: `<repo>/../citation-evidence/fixtures/pdfs`
|
||||
- override: `EVIDENCE_SOURCE_FIXTURE_DIR` environment variable
|
||||
|
||||
## Rationale
|
||||
|
||||
- **Privacy.** `evidence-source` advertises reusable ingest capability and is a
|
||||
candidate for wider distribution. Duplicating real PII into it multiplies the
|
||||
places that data lives and can be leaked.
|
||||
- **Single ownership.** The corpus and its `manifest.json` (page counts,
|
||||
known-good quotes) were curated and re-verified upstream. One owner avoids
|
||||
drift between two copies.
|
||||
- **Consistent coupling.** This repo already assumes a sibling checkout for its
|
||||
`@citation-evidence/engine` (`link:../citation-engine`) dependency. Reading
|
||||
fixtures from `../citation-evidence` is the same coupling model, not a new one.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Tests require a sibling `citation-evidence` checkout. In CI or a standalone
|
||||
clone, point `EVIDENCE_SOURCE_FIXTURE_DIR` at wherever the corpus is mounted.
|
||||
- When the corpus is absent, the fixture-driven suites skip with a clear message
|
||||
rather than failing spuriously, so unit tests that need no corpus still run.
|
||||
58
eslint.config.js
Normal file
58
eslint.config.js
Normal file
|
|
@ -0,0 +1,58 @@
|
|||
// ESLint flat config for evidence-source.
|
||||
//
|
||||
// Enforces the one boundary that matters in this repo: the browser upload
|
||||
// surface (`src/browser/**`) may depend on the headless core (`src/pdf/**`),
|
||||
// but the headless core must never import from `src/browser/**`. That keeps
|
||||
// `src/pdf/` runtime-agnostic (no `URL.createObjectURL`, no `Blob` upload
|
||||
// path) so it loads cleanly in Node and browsers alike.
|
||||
|
||||
import js from "@eslint/js";
|
||||
import tseslint from "typescript-eslint";
|
||||
import importPlugin from "eslint-plugin-import";
|
||||
import globals from "globals";
|
||||
|
||||
export default tseslint.config(
|
||||
{
|
||||
ignores: ["dist/", "node_modules/", "coverage/", "**/*.d.ts"],
|
||||
},
|
||||
js.configs.recommended,
|
||||
...tseslint.configs.recommended,
|
||||
{
|
||||
files: ["src/**/*.ts", "tests/**/*.ts"],
|
||||
languageOptions: {
|
||||
ecmaVersion: 2022,
|
||||
sourceType: "module",
|
||||
globals: { ...globals.node },
|
||||
},
|
||||
plugins: {
|
||||
import: importPlugin,
|
||||
},
|
||||
settings: {
|
||||
"import/resolver": {
|
||||
typescript: { project: "./tsconfig.json" },
|
||||
},
|
||||
},
|
||||
rules: {
|
||||
"no-restricted-imports": [
|
||||
"error",
|
||||
{
|
||||
patterns: [
|
||||
{
|
||||
group: ["**/browser/**", "../browser/*", "./browser/*"],
|
||||
message:
|
||||
"The headless core (src/pdf) must not depend on browser helpers (src/browser).",
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
{
|
||||
// Browser helpers are allowed to reach into the headless core; relax the
|
||||
// guard for tests, which import from anywhere.
|
||||
files: ["src/browser/**/*.ts", "tests/**/*.ts", "src/index.ts"],
|
||||
rules: {
|
||||
"no-restricted-imports": "off",
|
||||
},
|
||||
},
|
||||
);
|
||||
43
package.json
Normal file
43
package.json
Normal file
|
|
@ -0,0 +1,43 @@
|
|||
{
|
||||
"name": "@citation-evidence/evidence-source",
|
||||
"version": "0.1.0",
|
||||
"private": true,
|
||||
"description": "Headless document ingest, fingerprinting, and canonical-text extraction for the citation-evidence ecosystem. PDF slice.",
|
||||
"license": "Apache-2.0",
|
||||
"type": "module",
|
||||
"packageManager": "pnpm@9.15.0",
|
||||
"engines": {
|
||||
"node": ">=20.10.0"
|
||||
},
|
||||
"files": [
|
||||
"src",
|
||||
"docs",
|
||||
"README.md",
|
||||
"LICENSE"
|
||||
],
|
||||
"exports": {
|
||||
".": "./src/index.ts",
|
||||
"./browser": "./src/browser/index.ts"
|
||||
},
|
||||
"scripts": {
|
||||
"test": "vitest run",
|
||||
"test:watch": "vitest",
|
||||
"lint": "eslint .",
|
||||
"typecheck": "tsc --noEmit"
|
||||
},
|
||||
"dependencies": {
|
||||
"@citation-evidence/engine": "link:../citation-engine",
|
||||
"pdfjs-dist": "^4.4.168"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@eslint/js": "^9.7.0",
|
||||
"@types/node": "^20.14.0",
|
||||
"eslint": "^9.7.0",
|
||||
"eslint-import-resolver-typescript": "^3.6.3",
|
||||
"eslint-plugin-import": "^2.30.0",
|
||||
"globals": "^15.9.0",
|
||||
"typescript": "^5.5.4",
|
||||
"typescript-eslint": "^8.0.0",
|
||||
"vitest": "^2.0.5"
|
||||
}
|
||||
}
|
||||
3491
pnpm-lock.yaml
generated
Normal file
3491
pnpm-lock.yaml
generated
Normal file
File diff suppressed because it is too large
Load diff
|
|
@ -1,15 +0,0 @@
|
|||
---
|
||||
repo: evidence-source
|
||||
reason: >
|
||||
Only INTENT/SCOPE/registry/workplans; the ingestion/extraction layer is described but not implemented.
|
||||
reviewed: "2026-07-06"
|
||||
reviewed_by: claude-code
|
||||
revisit: >
|
||||
Revisit once the ingestion/extraction layer has a real implementation.
|
||||
---
|
||||
|
||||
# No reusable capability
|
||||
|
||||
This repo was reviewed for the `reuse-surface` capability registry
|
||||
(REUSE-WP-0017 coverage campaign) and has no capability to register at this
|
||||
time. See the `reason` field above.
|
||||
179
registry/capabilities/capability.infotech.pdf-evidence-ingest.md
Normal file
179
registry/capabilities/capability.infotech.pdf-evidence-ingest.md
Normal file
|
|
@ -0,0 +1,179 @@
|
|||
---
|
||||
id: capability.infotech.pdf-evidence-ingest
|
||||
name: Headless PDF Evidence Ingest
|
||||
summary: Turns raw PDF bytes into an engine-shaped Document + DocumentRepresentation — SHA-256 fingerprint, canonical text, page map, and gap-free offset map — as a runtime-agnostic library with no viewer, persistence, or React coupling.
|
||||
owner: evidence-source
|
||||
status: draft
|
||||
domain: infotech
|
||||
tags:
|
||||
- pdf
|
||||
- ingest
|
||||
- fingerprint
|
||||
- canonical-text
|
||||
- evidence
|
||||
|
||||
maturity:
|
||||
discovery:
|
||||
current: D2
|
||||
target: D5
|
||||
confidence: medium
|
||||
rationale: >
|
||||
Intent, scope, and boundary are documented (ADR-0001/0002) and the
|
||||
surface is exercised by a fixture-driven contract suite plus a
|
||||
cross-repo consumer round-trip. HTML/MD ingest and citation recovery
|
||||
are declared out of scope for now, so discovery is bounded to the PDF
|
||||
slice.
|
||||
availability:
|
||||
current: A1
|
||||
target: A3
|
||||
confidence: medium
|
||||
rationale: >
|
||||
Consumable today as a linked library (`link:../evidence-source`) via the
|
||||
package root (`@citation-evidence/evidence-source`) and browser subpath.
|
||||
Reaching A3 needs a published/versioned artifact rather than a
|
||||
sibling-checkout link.
|
||||
|
||||
external_evidence:
|
||||
completeness:
|
||||
level: C1
|
||||
confidence: medium
|
||||
basis: scope_vs_intent_and_consumer_expectations
|
||||
satisfied_expectations:
|
||||
- PDF bytes -> Document + DocumentRepresentation
|
||||
- deterministic SHA-256 fingerprint
|
||||
- canonical text with gap-free page offset map
|
||||
broken_expectations: []
|
||||
out_of_scope_expectations:
|
||||
- HTML / Markdown ingest
|
||||
- metadata enrichment beyond pass-through options
|
||||
- citation recovery / external source discovery
|
||||
reliability:
|
||||
level: R1
|
||||
confidence: medium
|
||||
basis: consumer_quality_signals
|
||||
known_reliability_risks:
|
||||
- PDF.js worker must be configured by the host before extraction
|
||||
- text-extraction fidelity varies on scanned/OCR-noisy PDFs
|
||||
|
||||
discovery:
|
||||
intent: >
|
||||
Let any consumer turn a PDF into engine-owned evidence contracts without
|
||||
inheriting a viewer, a store, or a UI framework.
|
||||
includes:
|
||||
- PDF byte ingest (ingestPdf)
|
||||
- PDF text extraction to canonical text + page/offset maps (extractPdf)
|
||||
- SHA-256 byte fingerprint (fingerprintBytes)
|
||||
- browser upload helpers behind a separate entry point (createPdfByteStore, ingestPdfFromFile)
|
||||
excludes:
|
||||
- HTML / Markdown ingest
|
||||
- persistence of documents/representations
|
||||
- viewer URL resolution (stays in the consuming app)
|
||||
- citation recovery
|
||||
assumptions:
|
||||
- domain contracts come from '@citation-evidence/engine/shared'
|
||||
- host configures the PDF.js worker before extraction
|
||||
use_cases:
|
||||
- citation-evidence umbrella consumes ingest across the repo boundary
|
||||
research_memos: []
|
||||
|
||||
availability:
|
||||
current_level: A1
|
||||
target_level: A3
|
||||
current_artifacts:
|
||||
- '@citation-evidence/evidence-source (package root, headless core)'
|
||||
- '@citation-evidence/evidence-source/browser (upload helpers)'
|
||||
target_artifacts:
|
||||
- published versioned package
|
||||
consumption_modes:
|
||||
- library import
|
||||
|
||||
relations:
|
||||
depends_on: []
|
||||
supports: []
|
||||
related_to: []
|
||||
|
||||
evidence:
|
||||
documentation:
|
||||
- README.md
|
||||
- docs/ADR-0001-extraction-boundary.md
|
||||
- docs/ADR-0002-fixture-ownership.md
|
||||
tests:
|
||||
- src/pdf/ingest.test.ts
|
||||
- src/pdf/fingerprint.test.ts
|
||||
- src/browser/byte-store.test.ts
|
||||
- src/browser/upload.test.ts
|
||||
consumer_feedback: []
|
||||
bug_reports: []
|
||||
incidents: []
|
||||
|
||||
consumer_guidance:
|
||||
recommended_for:
|
||||
- headless PDF -> evidence-contract ingest in Node or the browser
|
||||
- deterministic content fingerprinting of document bytes
|
||||
not_recommended_for:
|
||||
- non-PDF formats (not yet implemented)
|
||||
- high-fidelity extraction from scanned/OCR-only PDFs
|
||||
known_limitations:
|
||||
- PDF only for now
|
||||
- requires host-configured PDF.js worker
|
||||
- fixtures resolved from sibling citation-evidence checkout (ADR-0002)
|
||||
|
||||
promotion_history:
|
||||
- date: "2026-07-08"
|
||||
dimension: discovery
|
||||
from: D0
|
||||
to: D2
|
||||
rationale: PDF ingest slice extracted, documented (ADR-0001/0002), and covered by contract tests plus a cross-repo consumer round-trip (ESRC-WP-0001).
|
||||
author: claude-code
|
||||
---
|
||||
|
||||
# Headless PDF Evidence Ingest
|
||||
|
||||
## Overview
|
||||
|
||||
`evidence-source` turns raw PDF bytes into the two engine-owned contracts that
|
||||
the rest of the citation-evidence ecosystem builds on — a `Document` (media
|
||||
type, SHA-256 fingerprint, optional title/uri/metadata) and a
|
||||
`DocumentRepresentation` (`pdf-text`: canonical text, page map, and a gap-free
|
||||
offset map). Ingest is pure over bytes: no persistence, no viewer state, no
|
||||
React.
|
||||
|
||||
The YAML front matter above is the machine-readable source of truth for
|
||||
maturity, evidence, relations, and consumer guidance.
|
||||
|
||||
## Assessment notes
|
||||
|
||||
### Discovery
|
||||
|
||||
The boundary is fixed in ADR-0001 (which files are headless core vs. browser
|
||||
helpers vs. app-only viewer concerns) and ADR-0002 (fixture ownership). The
|
||||
public surface is small and stable: `ingestPdf`, `extractPdf`,
|
||||
`fingerprintBytes`, plus browser upload helpers behind a separate entry point.
|
||||
|
||||
### Availability
|
||||
|
||||
Consumed today as a linked library via `link:../evidence-source`. The umbrella
|
||||
app (`citation-evidence`) imports the headless core and browser helpers through
|
||||
a thin façade and runs its source ↔ anchor round-trip across the boundary.
|
||||
Target A3 requires a published, versioned artifact.
|
||||
|
||||
### Completeness
|
||||
|
||||
PDF only. HTML/Markdown representations, metadata enrichment, and citation
|
||||
recovery are explicitly deferred to follow-on workplans (see README) and are
|
||||
recorded as out-of-scope expectations rather than gaps.
|
||||
|
||||
### Reliability
|
||||
|
||||
Backed by a fixture-driven contract suite (fingerprint determinism, page/offset
|
||||
invariants, known-good quote presence) and a downstream consumer round-trip.
|
||||
The main reliability caveat is extraction fidelity on scanned/OCR-noisy PDFs,
|
||||
which is inherent to text-layer extraction.
|
||||
|
||||
## Promotion checklist
|
||||
|
||||
- [x] ID follows `capability.<domain>.<name>` pattern
|
||||
- [x] Maturity enums match `specs/CapabilityMaturityStandard.md`
|
||||
- [x] `external_evidence` is populated separately from `maturity`
|
||||
- [x] Relations reference valid capability IDs
|
||||
- [x] Index entry added or updated in `registry/indexes/capabilities.yaml`
|
||||
|
|
@ -1,4 +1,22 @@
|
|||
version: 1
|
||||
updated: '2026-06-16'
|
||||
domain: helix_forge
|
||||
capabilities: []
|
||||
updated: '2026-07-08'
|
||||
domain: infotech
|
||||
capabilities:
|
||||
- id: capability.infotech.pdf-evidence-ingest
|
||||
name: Headless PDF Evidence Ingest
|
||||
summary: Turns raw PDF bytes into an engine-shaped Document + DocumentRepresentation
|
||||
— SHA-256 fingerprint, canonical text, page map, and gap-free offset map — as a
|
||||
runtime-agnostic library with no viewer, persistence, or React coupling.
|
||||
vector: D2 / A1 / C1 / R1
|
||||
domain: infotech
|
||||
status: draft
|
||||
owner: evidence-source
|
||||
path: registry/capabilities/capability.infotech.pdf-evidence-ingest.md
|
||||
tags:
|
||||
- pdf
|
||||
- ingest
|
||||
- fingerprint
|
||||
- canonical-text
|
||||
- evidence
|
||||
consumption_modes:
|
||||
- library import
|
||||
|
|
|
|||
97
src/browser/byte-store.test.ts
Normal file
97
src/browser/byte-store.test.ts
Normal file
|
|
@ -0,0 +1,97 @@
|
|||
import { describe, expect, it, vi } from "vitest";
|
||||
|
||||
import type { DocumentId } from "@citation-evidence/engine/shared";
|
||||
|
||||
import { createPdfByteStore } from "./byte-store";
|
||||
|
||||
function stubUrlHelpers() {
|
||||
let counter = 0;
|
||||
const created: string[] = [];
|
||||
const revoked: string[] = [];
|
||||
const createObjectURL = vi.fn(() => {
|
||||
const url = `blob:stub-${++counter}`;
|
||||
created.push(url);
|
||||
return url;
|
||||
});
|
||||
const revokeObjectURL = vi.fn((url: string) => {
|
||||
revoked.push(url);
|
||||
});
|
||||
return { createObjectURL, revokeObjectURL, created, revoked };
|
||||
}
|
||||
|
||||
describe("PdfByteStore", () => {
|
||||
it("put / get round-trips bytes and exposes a blob URL", () => {
|
||||
const helpers = stubUrlHelpers();
|
||||
const store = createPdfByteStore(helpers);
|
||||
const bytes = new Uint8Array([0x25, 0x50, 0x44, 0x46]); // %PDF
|
||||
const record = store.put("doc_a" as DocumentId, bytes);
|
||||
expect(record.blobUrl).toBe("blob:stub-1");
|
||||
expect(store.get("doc_a" as DocumentId)?.bytes).toBe(bytes);
|
||||
expect(store.has("doc_a" as DocumentId)).toBe(true);
|
||||
expect(store.list()).toEqual(["doc_a"]);
|
||||
expect(store.size()).toBe(4);
|
||||
});
|
||||
|
||||
it("put replaces an existing entry and revokes the old URL", () => {
|
||||
const helpers = stubUrlHelpers();
|
||||
const store = createPdfByteStore(helpers);
|
||||
const id = "doc_a" as DocumentId;
|
||||
const first = store.put(id, new Uint8Array([1, 2]));
|
||||
const second = store.put(id, new Uint8Array([3, 4, 5]));
|
||||
expect(helpers.revoked).toEqual([first.blobUrl]);
|
||||
expect(store.get(id)?.bytes).toHaveLength(3);
|
||||
expect(second.blobUrl).not.toBe(first.blobUrl);
|
||||
});
|
||||
|
||||
it("delete revokes the blob URL exactly once and is idempotent", () => {
|
||||
const helpers = stubUrlHelpers();
|
||||
const store = createPdfByteStore(helpers);
|
||||
const id = "doc_a" as DocumentId;
|
||||
const record = store.put(id, new Uint8Array([1, 2, 3]));
|
||||
expect(store.delete(id)).toBe(true);
|
||||
expect(helpers.revoked).toEqual([record.blobUrl]);
|
||||
expect(store.delete(id)).toBe(false);
|
||||
expect(helpers.revoked).toHaveLength(1);
|
||||
expect(store.get(id)).toBeNull();
|
||||
expect(store.has(id)).toBe(false);
|
||||
});
|
||||
|
||||
it("clear revokes every URL and empties the store", () => {
|
||||
const helpers = stubUrlHelpers();
|
||||
const store = createPdfByteStore(helpers);
|
||||
const a = store.put("doc_a" as DocumentId, new Uint8Array([1]));
|
||||
const b = store.put("doc_b" as DocumentId, new Uint8Array([2]));
|
||||
store.clear();
|
||||
expect(helpers.revoked.sort()).toEqual([a.blobUrl, b.blobUrl].sort());
|
||||
expect(store.list()).toEqual([]);
|
||||
expect(store.size()).toBe(0);
|
||||
});
|
||||
|
||||
it("uses URL.createObjectURL by default when no override is supplied", () => {
|
||||
const createObjectURL = vi.fn(() => "blob:built-in");
|
||||
const revokeObjectURL = vi.fn();
|
||||
const originalURL = globalThis.URL;
|
||||
Object.defineProperty(globalThis, "URL", {
|
||||
configurable: true,
|
||||
writable: true,
|
||||
value: Object.assign(Object.create(originalURL.prototype as object), {
|
||||
createObjectURL,
|
||||
revokeObjectURL,
|
||||
}),
|
||||
});
|
||||
try {
|
||||
const store = createPdfByteStore();
|
||||
const rec = store.put("doc_z" as DocumentId, new Uint8Array([9]));
|
||||
expect(rec.blobUrl).toBe("blob:built-in");
|
||||
expect(createObjectURL).toHaveBeenCalledTimes(1);
|
||||
store.delete("doc_z" as DocumentId);
|
||||
expect(revokeObjectURL).toHaveBeenCalledWith("blob:built-in");
|
||||
} finally {
|
||||
Object.defineProperty(globalThis, "URL", {
|
||||
configurable: true,
|
||||
writable: true,
|
||||
value: originalURL,
|
||||
});
|
||||
}
|
||||
});
|
||||
});
|
||||
115
src/browser/byte-store.ts
Normal file
115
src/browser/byte-store.ts
Normal file
|
|
@ -0,0 +1,115 @@
|
|||
/**
|
||||
* `PdfByteStore` — in-memory store for uploaded PDF bytes, keyed by
|
||||
* `DocumentId`.
|
||||
*
|
||||
* BROWSER SURFACE — not part of the headless ingest core. This mints and
|
||||
* revokes `blob:` URLs and is meant to back an interactive upload/viewer
|
||||
* session. Headless callers should use `src/pdf/` directly and never import
|
||||
* this module.
|
||||
*
|
||||
* Uploaded PDFs are stored in memory only. Bytes survive within a tab
|
||||
* session; reloading the page loses them unless the session was exported.
|
||||
*
|
||||
* One store instance per active session. The session-management layer is
|
||||
* responsible for swapping the active store when the user switches sessions.
|
||||
* The store also owns a small registry of issued `blob:` URLs so it can
|
||||
* revoke them on delete/clear — no cross-cutting cleanup at the app layer.
|
||||
*/
|
||||
|
||||
import type { DocumentId } from "@citation-evidence/engine/shared";
|
||||
|
||||
export interface PdfByteRecord {
|
||||
readonly bytes: Uint8Array;
|
||||
/** A `blob:` URL the viewer can consume directly. */
|
||||
readonly blobUrl: string;
|
||||
}
|
||||
|
||||
export interface PdfByteStore {
|
||||
put(documentId: DocumentId, bytes: Uint8Array): PdfByteRecord;
|
||||
get(documentId: DocumentId): PdfByteRecord | null;
|
||||
has(documentId: DocumentId): boolean;
|
||||
delete(documentId: DocumentId): boolean;
|
||||
list(): readonly DocumentId[];
|
||||
/** Revoke every blob URL and clear the store. */
|
||||
clear(): void;
|
||||
/** Total bytes currently held — useful for UI dashboards. */
|
||||
size(): number;
|
||||
}
|
||||
|
||||
export interface CreatePdfByteStoreOptions {
|
||||
/**
|
||||
* Mint a URL for the given bytes. Defaults to `URL.createObjectURL` in
|
||||
* environments that have it; tests can inject a deterministic stub.
|
||||
*/
|
||||
readonly createObjectURL?: (blob: Blob) => string;
|
||||
/** Revoke a URL previously minted by `createObjectURL`. */
|
||||
readonly revokeObjectURL?: (url: string) => void;
|
||||
}
|
||||
|
||||
export function createPdfByteStore(
|
||||
options: CreatePdfByteStoreOptions = {},
|
||||
): PdfByteStore {
|
||||
const createUrl =
|
||||
options.createObjectURL ??
|
||||
((blob: Blob) => {
|
||||
if (typeof URL === "undefined" || typeof URL.createObjectURL !== "function") {
|
||||
throw new Error(
|
||||
"createPdfByteStore: URL.createObjectURL not available — inject a stub via options",
|
||||
);
|
||||
}
|
||||
return URL.createObjectURL(blob);
|
||||
});
|
||||
const revokeUrl =
|
||||
options.revokeObjectURL ??
|
||||
((url: string) => {
|
||||
if (typeof URL !== "undefined" && typeof URL.revokeObjectURL === "function") {
|
||||
URL.revokeObjectURL(url);
|
||||
}
|
||||
});
|
||||
|
||||
const records = new Map<DocumentId, PdfByteRecord>();
|
||||
|
||||
return {
|
||||
put(documentId, bytes) {
|
||||
// Replace previous record (revoking the prior URL) if any.
|
||||
const prior = records.get(documentId);
|
||||
if (prior) revokeUrl(prior.blobUrl);
|
||||
// Cast: Blob() does accept Uint8Array at runtime, but TS narrows the
|
||||
// buffer type to ArrayBufferLike (could be SharedArrayBuffer) and
|
||||
// refuses without help. The bytes here always come from a fresh
|
||||
// arrayBuffer() call, so a regular ArrayBuffer is guaranteed.
|
||||
const blob = new Blob([bytes as unknown as ArrayBuffer], {
|
||||
type: "application/pdf",
|
||||
});
|
||||
const blobUrl = createUrl(blob);
|
||||
const record: PdfByteRecord = { bytes, blobUrl };
|
||||
records.set(documentId, record);
|
||||
return record;
|
||||
},
|
||||
get(documentId) {
|
||||
return records.get(documentId) ?? null;
|
||||
},
|
||||
has(documentId) {
|
||||
return records.has(documentId);
|
||||
},
|
||||
delete(documentId) {
|
||||
const record = records.get(documentId);
|
||||
if (!record) return false;
|
||||
revokeUrl(record.blobUrl);
|
||||
records.delete(documentId);
|
||||
return true;
|
||||
},
|
||||
list() {
|
||||
return [...records.keys()];
|
||||
},
|
||||
clear() {
|
||||
for (const record of records.values()) revokeUrl(record.blobUrl);
|
||||
records.clear();
|
||||
},
|
||||
size() {
|
||||
let total = 0;
|
||||
for (const r of records.values()) total += r.bytes.byteLength;
|
||||
return total;
|
||||
},
|
||||
};
|
||||
}
|
||||
20
src/browser/index.ts
Normal file
20
src/browser/index.ts
Normal file
|
|
@ -0,0 +1,20 @@
|
|||
/**
|
||||
* `evidence-source/browser` — browser-facing upload helpers.
|
||||
*
|
||||
* BROWSER SURFACE. These helpers assume a browser session: they mint and
|
||||
* revoke `blob:` URLs and hold uploaded bytes in memory for a viewer to
|
||||
* mount. They are deliberately separated from the headless core (the package
|
||||
* root, `src/index.ts`) so that headless consumers never pull in
|
||||
* `URL.createObjectURL`/`Blob` upload machinery.
|
||||
*/
|
||||
|
||||
export {
|
||||
createPdfByteStore,
|
||||
type CreatePdfByteStoreOptions,
|
||||
type PdfByteRecord,
|
||||
type PdfByteStore,
|
||||
} from "./byte-store";
|
||||
export {
|
||||
ingestPdfFromFile,
|
||||
type IngestPdfFromFileOptions,
|
||||
} from "./upload";
|
||||
82
src/browser/upload.test.ts
Normal file
82
src/browser/upload.test.ts
Normal file
|
|
@ -0,0 +1,82 @@
|
|||
/**
|
||||
* `ingestPdfFromFile` end-to-end: pipes a fixture PDF through the upload path,
|
||||
* asserts the byte store keeps the bytes and the document record carries the
|
||||
* minted `blob:` URL. Corpus resolution/skip lives in `tests/fixtures.ts`.
|
||||
*/
|
||||
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
|
||||
import { createPdfByteStore } from "./byte-store";
|
||||
import { ingestPdfFromFile } from "./upload";
|
||||
import { fixtureBytes, fixtureCorpusAvailable } from "../../tests/fixtures";
|
||||
|
||||
const hasCorpus = fixtureCorpusAvailable();
|
||||
const FIXTURE_FILE = "Fristsetzung zur Bezifferung GÜ an Gegenseite 3 Wochen.pdf";
|
||||
|
||||
class FakeFile {
|
||||
readonly name: string;
|
||||
private readonly bytes: Uint8Array;
|
||||
constructor(bytes: Uint8Array, name: string) {
|
||||
this.bytes = bytes;
|
||||
this.name = name;
|
||||
}
|
||||
async arrayBuffer(): Promise<ArrayBuffer> {
|
||||
const out = new ArrayBuffer(this.bytes.byteLength);
|
||||
new Uint8Array(out).set(this.bytes);
|
||||
return out;
|
||||
}
|
||||
}
|
||||
|
||||
describe.skipIf(!hasCorpus)("ingestPdfFromFile", () => {
|
||||
it("round-trips a fixture PDF through ingest + byte store + blob URL", async () => {
|
||||
const bytes = fixtureBytes(FIXTURE_FILE);
|
||||
const file = new FakeFile(bytes, "demo.pdf") as unknown as File;
|
||||
let counter = 0;
|
||||
const store = createPdfByteStore({
|
||||
createObjectURL: () => `blob:upload-stub-${++counter}`,
|
||||
revokeObjectURL: () => {},
|
||||
});
|
||||
|
||||
const { document, representation } = await ingestPdfFromFile(file, store);
|
||||
|
||||
const stored = store.get(document.id);
|
||||
expect(stored).not.toBeNull();
|
||||
expect(stored!.bytes.byteLength).toBe(bytes.byteLength);
|
||||
|
||||
expect(document.uri).toBe(`blob:upload-stub-${counter}`);
|
||||
expect(document.title).toBe("demo.pdf");
|
||||
expect(document.fingerprint).toMatch(/^[0-9a-f]{64}$/);
|
||||
|
||||
expect(representation.representationType).toBe("pdf-text");
|
||||
expect((representation.canonicalText ?? "").length).toBeGreaterThan(0);
|
||||
}, 30_000);
|
||||
|
||||
it("falls through to ingestPdf with no filename when given a plain Blob", async () => {
|
||||
const bytes = fixtureBytes(FIXTURE_FILE);
|
||||
const blob = {
|
||||
async arrayBuffer() {
|
||||
const out = new ArrayBuffer(bytes.byteLength);
|
||||
new Uint8Array(out).set(bytes);
|
||||
return out;
|
||||
},
|
||||
} as Blob;
|
||||
const store = createPdfByteStore({
|
||||
createObjectURL: () => "blob:no-name",
|
||||
revokeObjectURL: () => {},
|
||||
});
|
||||
const { document } = await ingestPdfFromFile(blob, store);
|
||||
expect(document.title).toBeUndefined();
|
||||
expect(document.uri).toBe("blob:no-name");
|
||||
}, 30_000);
|
||||
|
||||
it("explicit title option overrides the filename", async () => {
|
||||
const bytes = fixtureBytes(FIXTURE_FILE);
|
||||
const file = new FakeFile(bytes, "anonymous-name.pdf") as unknown as File;
|
||||
const store = createPdfByteStore({
|
||||
createObjectURL: vi.fn(() => "blob:override"),
|
||||
revokeObjectURL: vi.fn(),
|
||||
});
|
||||
const { document } = await ingestPdfFromFile(file, store, { title: "Custom" });
|
||||
expect(document.title).toBe("Custom");
|
||||
}, 30_000);
|
||||
});
|
||||
43
src/browser/upload.ts
Normal file
43
src/browser/upload.ts
Normal file
|
|
@ -0,0 +1,43 @@
|
|||
/**
|
||||
* Upload-side ingest path.
|
||||
*
|
||||
* BROWSER SURFACE — not part of the headless ingest core. This is a thin
|
||||
* wrapper over the headless `ingestPdf` that additionally pushes the bytes
|
||||
* into a per-session `PdfByteStore`, which mints a `blob:` URL and stamps it
|
||||
* onto `document.uri` so a viewer adapter can mount the PDF directly.
|
||||
*
|
||||
* 1. Read `file.arrayBuffer()` once into a `Uint8Array`.
|
||||
* 2. Run the existing `ingestPdf(bytes, { filename })` pipeline to produce
|
||||
* `{document, representation}`.
|
||||
* 3. Push the bytes into the per-session `PdfByteStore`.
|
||||
* 4. Hand the engine inputs back to the caller, which wires them via
|
||||
* `engine.documents.register(...)`.
|
||||
*
|
||||
* Keeping URL-minting inside the byte store (rather than at the call site)
|
||||
* means there is exactly one place that creates `blob:` URLs and exactly one
|
||||
* place that revokes them.
|
||||
*/
|
||||
|
||||
import { ingestPdf, type IngestPdfResult } from "../pdf/ingest";
|
||||
import type { PdfByteStore } from "./byte-store";
|
||||
|
||||
export interface IngestPdfFromFileOptions {
|
||||
/** Override the filename used as the document title. */
|
||||
readonly title?: string;
|
||||
}
|
||||
|
||||
export async function ingestPdfFromFile(
|
||||
file: File | Blob,
|
||||
store: PdfByteStore,
|
||||
options: IngestPdfFromFileOptions = {},
|
||||
): Promise<IngestPdfResult> {
|
||||
const bytes = new Uint8Array(await file.arrayBuffer());
|
||||
const filename = "name" in file && typeof file.name === "string" ? file.name : undefined;
|
||||
const ingested = await ingestPdf(bytes, {
|
||||
...(filename !== undefined ? { filename } : {}),
|
||||
...(options.title !== undefined ? { title: options.title } : {}),
|
||||
});
|
||||
const record = store.put(ingested.document.id, bytes);
|
||||
const document = { ...ingested.document, uri: record.blobUrl };
|
||||
return { document, representation: ingested.representation };
|
||||
}
|
||||
20
src/index.ts
Normal file
20
src/index.ts
Normal file
|
|
@ -0,0 +1,20 @@
|
|||
/**
|
||||
* `evidence-source` — headless document ingest core (PDF slice).
|
||||
*
|
||||
* This entry point exposes only runtime-agnostic ingest: pass PDF bytes,
|
||||
* receive a `{ document, representation }` pair built entirely from
|
||||
* engine-owned contracts (`@citation-evidence/engine/shared`). No browser,
|
||||
* viewer, or persistence concerns leak through here.
|
||||
*
|
||||
* Browser-facing upload helpers live behind the separate
|
||||
* `@citation-evidence/evidence-source/browser` entry point.
|
||||
*/
|
||||
|
||||
export {
|
||||
ingestPdf,
|
||||
type IngestPdfInput,
|
||||
type IngestPdfOptions,
|
||||
type IngestPdfResult,
|
||||
} from "./pdf/ingest";
|
||||
export { extractPdf, type PdfExtractionResult } from "./pdf/extract";
|
||||
export { fingerprintBytes } from "./pdf/fingerprint";
|
||||
122
src/pdf/extract.ts
Normal file
122
src/pdf/extract.ts
Normal file
|
|
@ -0,0 +1,122 @@
|
|||
/**
|
||||
* PDF text extraction → canonical text + PageMap + OffsetMap.
|
||||
*
|
||||
* Implements `wiki/ArchitectureOverview.md` §3.4 ("extract canonical text /
|
||||
* build format-specific maps") for the `pdf-text` representation
|
||||
* (`wiki/SharedContracts.md` §1, §3) and §6 (canonical normalization).
|
||||
*
|
||||
* Runtime independence: the PDF.js worker must be configured by the host
|
||||
* application (`GlobalWorkerOptions.workerSrc`) before this module is
|
||||
* called. In Vite/browser code the worker is bundled via the viewer; in
|
||||
* Node tests the test setup file points it at
|
||||
* `pdfjs-dist/legacy/build/pdf.worker.mjs`. No worker setup happens here
|
||||
* so the same module loads cleanly in both runtimes.
|
||||
*
|
||||
* Page boundary semantics: canonical text concatenates per-page normalized
|
||||
* text with a single "\n\n" paragraph separator. The separator is treated
|
||||
* as belonging to the *preceding* page in `OffsetMap`, so the map covers
|
||||
* `[0, canonicalText.length)` with no gaps. The last page has no trailing
|
||||
* separator. This means `pageLength = globalEnd - globalStart` for
|
||||
* every page; for non-last pages it equals (normalized page text length +
|
||||
* 2). See `PageOffsetRange` in the engine's shared document contract.
|
||||
*/
|
||||
|
||||
import { getDocument } from "pdfjs-dist";
|
||||
import type { PDFPageProxy } from "pdfjs-dist";
|
||||
import {
|
||||
normalize,
|
||||
type OffsetMap,
|
||||
type PageInfo,
|
||||
type PageMap,
|
||||
type PageOffsetRange,
|
||||
} from "@citation-evidence/engine/shared";
|
||||
|
||||
const PAGE_SEPARATOR = "\n\n";
|
||||
|
||||
export interface PdfExtractionResult {
|
||||
readonly canonicalText: string;
|
||||
readonly pageMap: PageMap;
|
||||
readonly offsetMap: OffsetMap;
|
||||
readonly pageCount: number;
|
||||
}
|
||||
|
||||
export async function extractPdf(bytes: Uint8Array): Promise<PdfExtractionResult> {
|
||||
// PDF.js mutates the bytes buffer (transfers ownership). Pass a fresh copy
|
||||
// so the caller's Uint8Array stays usable for fingerprinting after extract.
|
||||
const data = new Uint8Array(bytes);
|
||||
const loadingTask = getDocument({ data });
|
||||
const doc = await loadingTask.promise;
|
||||
|
||||
try {
|
||||
const pageCount = doc.numPages;
|
||||
const pageInfos: PageInfo[] = [];
|
||||
const pageNormalizedTexts: string[] = [];
|
||||
|
||||
for (let pageNumber = 1; pageNumber <= pageCount; pageNumber++) {
|
||||
const page = await doc.getPage(pageNumber);
|
||||
try {
|
||||
const viewport = page.getViewport({ scale: 1 });
|
||||
pageInfos.push({
|
||||
page: pageNumber,
|
||||
width: viewport.width,
|
||||
height: viewport.height,
|
||||
});
|
||||
|
||||
const rawText = await extractPageText(page);
|
||||
pageNormalizedTexts.push(normalize(rawText).text);
|
||||
} finally {
|
||||
page.cleanup();
|
||||
}
|
||||
}
|
||||
|
||||
const { canonicalText, offsetMap } = buildOffsetMap(pageNormalizedTexts);
|
||||
|
||||
return {
|
||||
canonicalText,
|
||||
pageMap: pageInfos,
|
||||
offsetMap,
|
||||
pageCount,
|
||||
};
|
||||
} finally {
|
||||
await doc.destroy();
|
||||
}
|
||||
}
|
||||
|
||||
async function extractPageText(page: PDFPageProxy): Promise<string> {
|
||||
const content = await page.getTextContent();
|
||||
// textContent.items are TextItem | TextMarkedContent. We want only the
|
||||
// TextItem strings (those have a `str` field); marked-content entries are
|
||||
// structural anchors and have no visible text.
|
||||
const parts: string[] = [];
|
||||
for (const item of content.items) {
|
||||
if ("str" in item) {
|
||||
parts.push(item.str);
|
||||
if (item.hasEOL) parts.push("\n");
|
||||
}
|
||||
}
|
||||
return parts.join("");
|
||||
}
|
||||
|
||||
function buildOffsetMap(pageTexts: readonly string[]): {
|
||||
canonicalText: string;
|
||||
offsetMap: OffsetMap;
|
||||
} {
|
||||
const ranges: PageOffsetRange[] = [];
|
||||
let offset = 0;
|
||||
for (let i = 0; i < pageTexts.length; i++) {
|
||||
const text = pageTexts[i]!;
|
||||
const isLast = i === pageTexts.length - 1;
|
||||
const segmentLength = text.length + (isLast ? 0 : PAGE_SEPARATOR.length);
|
||||
const globalStart = offset;
|
||||
const globalEnd = offset + segmentLength;
|
||||
ranges.push({
|
||||
page: i + 1,
|
||||
globalStart,
|
||||
globalEnd,
|
||||
pageLength: segmentLength,
|
||||
});
|
||||
offset = globalEnd;
|
||||
}
|
||||
const canonicalText = pageTexts.join(PAGE_SEPARATOR);
|
||||
return { canonicalText, offsetMap: ranges };
|
||||
}
|
||||
37
src/pdf/fingerprint.test.ts
Normal file
37
src/pdf/fingerprint.test.ts
Normal file
|
|
@ -0,0 +1,37 @@
|
|||
import { describe, expect, it } from "vitest";
|
||||
|
||||
import { fingerprintBytes } from "./fingerprint";
|
||||
|
||||
describe("fingerprintBytes", () => {
|
||||
it("produces a 64-char lowercase hex SHA-256", async () => {
|
||||
const digest = await fingerprintBytes(new Uint8Array([0x25, 0x50, 0x44, 0x46]));
|
||||
expect(digest).toMatch(/^[0-9a-f]{64}$/);
|
||||
});
|
||||
|
||||
it("matches the known SHA-256 of the empty input", async () => {
|
||||
// Well-known constant: SHA-256 of zero bytes.
|
||||
const digest = await fingerprintBytes(new Uint8Array(0));
|
||||
expect(digest).toBe(
|
||||
"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
|
||||
);
|
||||
});
|
||||
|
||||
it("is deterministic across repeated calls on the same bytes", async () => {
|
||||
const bytes = new Uint8Array([1, 2, 3, 4, 5, 6, 7, 8]);
|
||||
const a = await fingerprintBytes(bytes);
|
||||
const b = await fingerprintBytes(bytes);
|
||||
expect(a).toBe(b);
|
||||
});
|
||||
|
||||
it("does not mutate or consume the input buffer", async () => {
|
||||
const bytes = new Uint8Array([9, 8, 7]);
|
||||
await fingerprintBytes(bytes);
|
||||
expect([...bytes]).toEqual([9, 8, 7]);
|
||||
});
|
||||
|
||||
it("distinguishes single-bit differences", async () => {
|
||||
const a = await fingerprintBytes(new Uint8Array([0]));
|
||||
const b = await fingerprintBytes(new Uint8Array([1]));
|
||||
expect(a).not.toBe(b);
|
||||
});
|
||||
});
|
||||
31
src/pdf/fingerprint.ts
Normal file
31
src/pdf/fingerprint.ts
Normal file
|
|
@ -0,0 +1,31 @@
|
|||
/**
|
||||
* SHA-256 fingerprint of raw document bytes.
|
||||
*
|
||||
* Implements the fingerprint half of `wiki/ArchitectureOverview.md` §3.4
|
||||
* (the "compute fingerprint" pipeline step) and populates
|
||||
* `Document.fingerprint` (`wiki/SharedContracts.md` §1).
|
||||
*
|
||||
* Uses Web Crypto's `crypto.subtle.digest`, which is available in browsers
|
||||
* and in Node ≥ 20 (where it is exposed on `globalThis.crypto`). No
|
||||
* platform branching — the API is the same in both environments.
|
||||
*/
|
||||
|
||||
export async function fingerprintBytes(bytes: Uint8Array): Promise<string> {
|
||||
// Copy into a fresh ArrayBuffer (not SharedArrayBuffer) so the digest call
|
||||
// satisfies TS's updated `BufferSource` type, which excludes
|
||||
// `SharedArrayBuffer`. The copy is O(n) — fine even for large PDFs since
|
||||
// SHA-256 itself is already O(n).
|
||||
const ab = new ArrayBuffer(bytes.byteLength);
|
||||
new Uint8Array(ab).set(bytes);
|
||||
const digest = await crypto.subtle.digest("SHA-256", ab);
|
||||
return bytesToHex(new Uint8Array(digest));
|
||||
}
|
||||
|
||||
function bytesToHex(bytes: Uint8Array): string {
|
||||
let hex = "";
|
||||
for (let i = 0; i < bytes.length; i++) {
|
||||
const b = bytes[i]!;
|
||||
hex += (b < 0x10 ? "0" : "") + b.toString(16);
|
||||
}
|
||||
return hex;
|
||||
}
|
||||
119
src/pdf/ingest.test.ts
Normal file
119
src/pdf/ingest.test.ts
Normal file
|
|
@ -0,0 +1,119 @@
|
|||
/**
|
||||
* Fixture-driven contract tests for the PDF ingest pipeline.
|
||||
*
|
||||
* For each fixture in the shared corpus manifest (ADR-0002):
|
||||
* 1. Read the PDF bytes from disk.
|
||||
* 2. Run `ingestPdf` end-to-end.
|
||||
* 3. Assert the resulting Document + DocumentRepresentation honour the
|
||||
* manifest contract: media type is application/pdf, fingerprint is a
|
||||
* 64-hex SHA-256, pageMap matches `page_count`, canonicalText contains
|
||||
* `known_good_quote`, and the offsetMap covers `[0, canonicalText.length)`
|
||||
* with no gaps.
|
||||
*
|
||||
* Corpus resolution and skip-on-absent behaviour live in `tests/fixtures.ts`.
|
||||
*/
|
||||
|
||||
import { describe, expect, it } from "vitest";
|
||||
|
||||
import { ingestPdf } from "./ingest";
|
||||
import { fingerprintBytes } from "./fingerprint";
|
||||
import {
|
||||
fixtureBytes,
|
||||
fixtureCorpusAvailable,
|
||||
loadFixtures,
|
||||
} from "../../tests/fixtures";
|
||||
|
||||
const hasCorpus = fixtureCorpusAvailable();
|
||||
const FIXTURES = hasCorpus ? loadFixtures() : [];
|
||||
|
||||
describe.skipIf(!hasCorpus)("ingestPdf — fixture corpus", { timeout: 30_000 }, () => {
|
||||
for (const fixture of FIXTURES) {
|
||||
describe(fixture.id, () => {
|
||||
const bytes = fixtureBytes(fixture.filename);
|
||||
|
||||
it("produces a Document with PDF media type and SHA-256 fingerprint", async () => {
|
||||
const { document } = await ingestPdf(bytes, { filename: fixture.filename });
|
||||
expect(document.mediaType).toBe("application/pdf");
|
||||
expect(document.fingerprint).toMatch(/^[0-9a-f]{64}$/);
|
||||
expect(document.title).toBe(fixture.filename);
|
||||
const expected = await fingerprintBytes(bytes);
|
||||
expect(document.fingerprint).toBe(expected);
|
||||
});
|
||||
|
||||
it("produces a pdf-text representation with the expected page count", async () => {
|
||||
const { representation } = await ingestPdf(bytes);
|
||||
expect(representation.representationType).toBe("pdf-text");
|
||||
expect(representation.pageMap?.length).toBe(fixture.page_count);
|
||||
expect(representation.offsetMap?.length).toBe(fixture.page_count);
|
||||
});
|
||||
|
||||
it("canonical text contains the manifest's known-good quote", async () => {
|
||||
const { representation } = await ingestPdf(bytes);
|
||||
const text = representation.canonicalText ?? "";
|
||||
expect(text).toContain(fixture.known_good_quote);
|
||||
});
|
||||
|
||||
it("offsetMap is gap-free and covers [0, canonicalText.length)", async () => {
|
||||
const { representation } = await ingestPdf(bytes);
|
||||
const text = representation.canonicalText ?? "";
|
||||
const offsets = representation.offsetMap ?? [];
|
||||
expect(offsets.length).toBeGreaterThan(0);
|
||||
expect(offsets[0]!.globalStart).toBe(0);
|
||||
expect(offsets.at(-1)!.globalEnd).toBe(text.length);
|
||||
for (let i = 0; i < offsets.length; i++) {
|
||||
const r = offsets[i]!;
|
||||
expect(r.page).toBe(i + 1);
|
||||
expect(r.globalEnd - r.globalStart).toBe(r.pageLength);
|
||||
if (i > 0) expect(r.globalStart).toBe(offsets[i - 1]!.globalEnd);
|
||||
}
|
||||
});
|
||||
|
||||
it("pageMap entries have positive width and height in user-space points", async () => {
|
||||
const { representation } = await ingestPdf(bytes);
|
||||
const pages = representation.pageMap ?? [];
|
||||
for (let i = 0; i < pages.length; i++) {
|
||||
const p = pages[i]!;
|
||||
expect(p.page).toBe(i + 1);
|
||||
expect(p.width).toBeGreaterThan(0);
|
||||
expect(p.height).toBeGreaterThan(0);
|
||||
}
|
||||
});
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
describe.skipIf(!hasCorpus)("ingestPdf — option handling", { timeout: 30_000 }, () => {
|
||||
const fixture = FIXTURES[0];
|
||||
const bytes = fixture ? fixtureBytes(fixture.filename) : new Uint8Array();
|
||||
|
||||
it("uses explicit title over filename", async () => {
|
||||
const { document } = await ingestPdf(bytes, {
|
||||
filename: fixture!.filename,
|
||||
title: "Custom Title",
|
||||
});
|
||||
expect(document.title).toBe("Custom Title");
|
||||
});
|
||||
|
||||
it("omits title entirely when neither filename nor title is supplied", async () => {
|
||||
const { document } = await ingestPdf(bytes);
|
||||
expect(document.title).toBeUndefined();
|
||||
});
|
||||
|
||||
it("propagates uri and metadata when supplied", async () => {
|
||||
const { document } = await ingestPdf(bytes, {
|
||||
uri: "file:///example.pdf",
|
||||
metadata: { source: "test" },
|
||||
});
|
||||
expect(document.uri).toBe("file:///example.pdf");
|
||||
expect(document.metadata).toEqual({ source: "test" });
|
||||
});
|
||||
|
||||
it("accepts ArrayBuffer input", async () => {
|
||||
const ab = bytes.buffer.slice(
|
||||
bytes.byteOffset,
|
||||
bytes.byteOffset + bytes.byteLength,
|
||||
) as ArrayBuffer;
|
||||
const { document } = await ingestPdf(ab);
|
||||
expect(document.fingerprint).toMatch(/^[0-9a-f]{64}$/);
|
||||
});
|
||||
});
|
||||
88
src/pdf/ingest.ts
Normal file
88
src/pdf/ingest.ts
Normal file
|
|
@ -0,0 +1,88 @@
|
|||
/**
|
||||
* PDF ingest pipeline → `{ document, representation }`.
|
||||
*
|
||||
* Implements `wiki/ArchitectureOverview.md` §3.4 ("Raw Source → identify
|
||||
* media type → compute fingerprint → extract metadata → extract canonical
|
||||
* text → build format-specific maps → persist Document +
|
||||
* DocumentRepresentation") for the PDF source format.
|
||||
*
|
||||
* Ingest is a pure function over bytes: it does not persist anything. The
|
||||
* caller (engine repositories, or the app layer) writes the returned
|
||||
* Document + DocumentRepresentation into the chosen store.
|
||||
*/
|
||||
|
||||
import {
|
||||
newId,
|
||||
type Document,
|
||||
type DocumentRepresentation,
|
||||
} from "@citation-evidence/engine/shared";
|
||||
import { extractPdf } from "./extract";
|
||||
import { fingerprintBytes } from "./fingerprint";
|
||||
|
||||
const PDF_MEDIA_TYPE = "application/pdf";
|
||||
|
||||
export interface IngestPdfOptions {
|
||||
/** Original filename, used as the default title when no title is given. */
|
||||
readonly filename?: string;
|
||||
/** Optional pre-existing title (overrides filename). */
|
||||
readonly title?: string;
|
||||
/** Optional source URI (e.g. file:// or https://). */
|
||||
readonly uri?: string;
|
||||
/** Free-form metadata persisted on the Document record. */
|
||||
readonly metadata?: Readonly<Record<string, unknown>>;
|
||||
}
|
||||
|
||||
export interface IngestPdfResult {
|
||||
readonly document: Document;
|
||||
readonly representation: DocumentRepresentation;
|
||||
}
|
||||
|
||||
export type IngestPdfInput = Uint8Array | ArrayBuffer | Blob;
|
||||
|
||||
export async function ingestPdf(
|
||||
input: IngestPdfInput,
|
||||
options: IngestPdfOptions = {},
|
||||
): Promise<IngestPdfResult> {
|
||||
const bytes = await toBytes(input);
|
||||
const [fingerprint, extraction] = await Promise.all([
|
||||
fingerprintBytes(bytes),
|
||||
extractPdf(bytes),
|
||||
]);
|
||||
|
||||
const now = new Date().toISOString();
|
||||
const documentId = newId("document");
|
||||
const representationId = newId("representation");
|
||||
const title = options.title ?? options.filename;
|
||||
|
||||
const document: Document = {
|
||||
id: documentId,
|
||||
mediaType: PDF_MEDIA_TYPE,
|
||||
fingerprint,
|
||||
createdAt: now,
|
||||
updatedAt: now,
|
||||
...(title !== undefined ? { title } : {}),
|
||||
...(options.uri !== undefined ? { uri: options.uri } : {}),
|
||||
...(options.metadata !== undefined ? { metadata: options.metadata } : {}),
|
||||
};
|
||||
|
||||
const representation: DocumentRepresentation = {
|
||||
id: representationId,
|
||||
documentId,
|
||||
representationType: "pdf-text",
|
||||
contentHash: fingerprint,
|
||||
canonicalText: extraction.canonicalText,
|
||||
pageMap: extraction.pageMap,
|
||||
offsetMap: extraction.offsetMap,
|
||||
generatedAt: now,
|
||||
};
|
||||
|
||||
return { document, representation };
|
||||
}
|
||||
|
||||
async function toBytes(input: IngestPdfInput): Promise<Uint8Array> {
|
||||
if (input instanceof Uint8Array) return input;
|
||||
if (input instanceof ArrayBuffer) return new Uint8Array(input);
|
||||
// Blob (covers `File` in browsers — File extends Blob).
|
||||
const buf = await input.arrayBuffer();
|
||||
return new Uint8Array(buf);
|
||||
}
|
||||
51
tests/fixtures.ts
Normal file
51
tests/fixtures.ts
Normal file
|
|
@ -0,0 +1,51 @@
|
|||
/**
|
||||
* Fixture corpus access for evidence-source tests.
|
||||
*
|
||||
* The corpus is owned by the sibling `citation-evidence` checkout (ADR-0002);
|
||||
* it is not duplicated into this repo. Resolution order:
|
||||
* 1. `EVIDENCE_SOURCE_FIXTURE_DIR` env var, if set.
|
||||
* 2. `<repo>/../citation-evidence/fixtures/pdfs` (sibling-checkout default).
|
||||
*
|
||||
* When the corpus is unavailable, `fixtureCorpusAvailable()` returns false so
|
||||
* fixture-driven suites can skip cleanly instead of failing.
|
||||
*/
|
||||
|
||||
import { existsSync, readFileSync } from "node:fs";
|
||||
import { dirname, resolve } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
const __dirname = dirname(fileURLToPath(import.meta.url));
|
||||
|
||||
export const FIXTURE_DIR = process.env.EVIDENCE_SOURCE_FIXTURE_DIR
|
||||
? resolve(process.env.EVIDENCE_SOURCE_FIXTURE_DIR)
|
||||
: resolve(__dirname, "../../citation-evidence/fixtures/pdfs");
|
||||
|
||||
export interface Fixture {
|
||||
readonly id: string;
|
||||
readonly filename: string;
|
||||
readonly page_count: number;
|
||||
readonly known_good_quote: string;
|
||||
readonly known_good_quote_page: number;
|
||||
}
|
||||
|
||||
interface Manifest {
|
||||
readonly fixtures: readonly Fixture[];
|
||||
}
|
||||
|
||||
export function fixturePath(filename: string): string {
|
||||
return resolve(FIXTURE_DIR, filename);
|
||||
}
|
||||
|
||||
export function fixtureBytes(filename: string): Uint8Array {
|
||||
return new Uint8Array(readFileSync(fixturePath(filename)));
|
||||
}
|
||||
|
||||
export function fixtureCorpusAvailable(): boolean {
|
||||
return existsSync(resolve(FIXTURE_DIR, "manifest.json"));
|
||||
}
|
||||
|
||||
export function loadFixtures(): readonly Fixture[] {
|
||||
const manifestPath = resolve(FIXTURE_DIR, "manifest.json");
|
||||
const manifest = JSON.parse(readFileSync(manifestPath, "utf8")) as Manifest;
|
||||
return manifest.fixtures;
|
||||
}
|
||||
17
tests/setup-pdf-worker.ts
Normal file
17
tests/setup-pdf-worker.ts
Normal file
|
|
@ -0,0 +1,17 @@
|
|||
/**
|
||||
* Vitest setup: point PDF.js at its legacy worker bundle for Node runs.
|
||||
*
|
||||
* `extractPdf` deliberately does no worker setup (so the same module loads in
|
||||
* browsers and Node). In Node tests we configure `GlobalWorkerOptions.workerSrc`
|
||||
* here, once per test file, before any extraction runs. The legacy bundle ships
|
||||
* as plain JS and works through PDF.js's fake-worker fallback when no real
|
||||
* Worker is available.
|
||||
*/
|
||||
|
||||
import { createRequire } from "node:module";
|
||||
import * as pdfjs from "pdfjs-dist";
|
||||
|
||||
const require = createRequire(import.meta.url);
|
||||
pdfjs.GlobalWorkerOptions.workerSrc = require.resolve(
|
||||
"pdfjs-dist/legacy/build/pdf.worker.mjs",
|
||||
);
|
||||
25
tsconfig.json
Normal file
25
tsconfig.json
Normal file
|
|
@ -0,0 +1,25 @@
|
|||
{
|
||||
"compilerOptions": {
|
||||
"target": "ES2022",
|
||||
"lib": ["ES2022", "DOM"],
|
||||
"module": "ESNext",
|
||||
"moduleResolution": "Bundler",
|
||||
"strict": true,
|
||||
"noImplicitOverride": true,
|
||||
"noFallthroughCasesInSwitch": true,
|
||||
"noUncheckedIndexedAccess": true,
|
||||
"exactOptionalPropertyTypes": true,
|
||||
"noUnusedLocals": true,
|
||||
"noUnusedParameters": true,
|
||||
"noEmit": true,
|
||||
"esModuleInterop": true,
|
||||
"forceConsistentCasingInFileNames": true,
|
||||
"isolatedModules": true,
|
||||
"skipLibCheck": true,
|
||||
"resolveJsonModule": true,
|
||||
"verbatimModuleSyntax": true,
|
||||
"types": ["node"]
|
||||
},
|
||||
"include": ["src", "tests", "vitest.config.ts", "eslint.config.js"],
|
||||
"exclude": ["node_modules", "dist"]
|
||||
}
|
||||
14
vitest.config.ts
Normal file
14
vitest.config.ts
Normal file
|
|
@ -0,0 +1,14 @@
|
|||
import { defineConfig } from "vitest/config";
|
||||
|
||||
// Headless-only test surface: every test runs in Node, which has full
|
||||
// pdfjs-dist legacy-worker support. There is no DOM/React surface in this
|
||||
// repo, so unlike the umbrella app we do not register a happy-dom
|
||||
// environment. The browser upload helpers (`src/browser/`) are exercised
|
||||
// with injected URL stubs, so they run in Node too.
|
||||
export default defineConfig({
|
||||
test: {
|
||||
globals: false,
|
||||
include: ["src/**/*.test.ts", "tests/**/*.test.ts"],
|
||||
setupFiles: ["tests/setup-pdf-worker.ts"],
|
||||
},
|
||||
});
|
||||
|
|
@ -4,7 +4,7 @@ type: workplan
|
|||
title: "Bootstrap standalone source repo from citation-evidence PDF slice"
|
||||
domain: infotech
|
||||
repo: evidence-source
|
||||
status: ready
|
||||
status: finished
|
||||
owner: codex
|
||||
topic_slug: citation_evidence_mvp
|
||||
created: "2026-06-21"
|
||||
|
|
@ -64,7 +64,7 @@ T01 extraction boundary and import plan
|
|||
|
||||
```task
|
||||
id: ESRC-WP-0001-T01
|
||||
status: todo
|
||||
status: done
|
||||
priority: high
|
||||
```
|
||||
|
||||
|
|
@ -93,7 +93,7 @@ the extraction relies on copying `citation-engine` shared types locally.
|
|||
|
||||
```task
|
||||
id: ESRC-WP-0001-T02
|
||||
status: todo
|
||||
status: done
|
||||
priority: high
|
||||
```
|
||||
|
||||
|
|
@ -119,7 +119,7 @@ runtime host.
|
|||
|
||||
```task
|
||||
id: ESRC-WP-0001-T03
|
||||
status: todo
|
||||
status: done
|
||||
priority: high
|
||||
```
|
||||
|
||||
|
|
@ -147,7 +147,7 @@ Acceptance: a consumer can pass PDF bytes to this repo and receive a valid
|
|||
|
||||
```task
|
||||
id: ESRC-WP-0001-T04
|
||||
status: todo
|
||||
status: done
|
||||
priority: high
|
||||
```
|
||||
|
||||
|
|
@ -176,7 +176,7 @@ browser/session-only behavior after extraction.
|
|||
|
||||
```task
|
||||
id: ESRC-WP-0001-T05
|
||||
status: todo
|
||||
status: done
|
||||
priority: high
|
||||
```
|
||||
|
||||
|
|
@ -203,7 +203,7 @@ source-to-anchor round-trip remains an umbrella integration test.
|
|||
|
||||
```task
|
||||
id: ESRC-WP-0001-T06
|
||||
status: todo
|
||||
status: done
|
||||
priority: high
|
||||
```
|
||||
|
||||
|
|
@ -227,7 +227,7 @@ package rather than the local `src/source/` copy for the PDF slice.
|
|||
|
||||
```task
|
||||
id: ESRC-WP-0001-T07
|
||||
status: todo
|
||||
status: done
|
||||
priority: medium
|
||||
```
|
||||
|
||||
|
|
|
|||
55
workplans/ESRC-WP-0002-html-markdown-representations.md
Normal file
55
workplans/ESRC-WP-0002-html-markdown-representations.md
Normal file
|
|
@ -0,0 +1,55 @@
|
|||
---
|
||||
id: ESRC-WP-0002
|
||||
type: workplan
|
||||
title: "HTML and Markdown source representations"
|
||||
domain: infotech
|
||||
repo: evidence-source
|
||||
status: proposed
|
||||
owner: codex
|
||||
topic_slug: citation_evidence_mvp
|
||||
created: "2026-07-08"
|
||||
updated: "2026-07-08"
|
||||
spec_refs:
|
||||
- README.md
|
||||
- docs/ADR-0001-extraction-boundary.md
|
||||
- ../citation-engine/src/shared/document.ts
|
||||
---
|
||||
|
||||
# ESRC-WP-0002 — HTML and Markdown source representations
|
||||
|
||||
The PDF slice (ESRC-WP-0001) established the ingest boundary and contract shape.
|
||||
This workplan extends ingest to HTML and Markdown sources, producing the same
|
||||
`{ document, representation }` pair over engine-owned contracts.
|
||||
|
||||
## Open questions
|
||||
|
||||
- Which `representationType` values do HTML/MD map to, and do they need new
|
||||
`DocumentRepresentation` fields in `citation-engine`?
|
||||
- How is canonical text + offset map derived for reflowable formats that have
|
||||
no fixed pages? (Page map may be empty or synthetic.)
|
||||
|
||||
## Sketch
|
||||
|
||||
```task
|
||||
id: ESRC-WP-0002-T01
|
||||
status: todo
|
||||
priority: medium
|
||||
```
|
||||
Define the HTML/MD representation contract with `citation-engine`; agree
|
||||
`representationType` and any offset/page semantics for pageless formats.
|
||||
|
||||
```task
|
||||
id: ESRC-WP-0002-T02
|
||||
status: todo
|
||||
priority: medium
|
||||
```
|
||||
Implement `ingestHtml` / `ingestMarkdown` in `src/html/` and `src/markdown/`,
|
||||
reusing `fingerprintBytes` and the canonical-text normalizer.
|
||||
|
||||
```task
|
||||
id: ESRC-WP-0002-T03
|
||||
status: todo
|
||||
priority: medium
|
||||
```
|
||||
Add fixture-driven contract tests mirroring the PDF suite; decide fixture
|
||||
ownership per ADR-0002.
|
||||
48
workplans/ESRC-WP-0003-metadata-enrichment.md
Normal file
48
workplans/ESRC-WP-0003-metadata-enrichment.md
Normal file
|
|
@ -0,0 +1,48 @@
|
|||
---
|
||||
id: ESRC-WP-0003
|
||||
type: workplan
|
||||
title: "Document metadata enrichment beyond pass-through options"
|
||||
domain: infotech
|
||||
repo: evidence-source
|
||||
status: proposed
|
||||
owner: codex
|
||||
topic_slug: citation_evidence_mvp
|
||||
created: "2026-07-08"
|
||||
updated: "2026-07-08"
|
||||
spec_refs:
|
||||
- README.md
|
||||
- src/pdf/ingest.ts
|
||||
---
|
||||
|
||||
# ESRC-WP-0003 — Metadata enrichment
|
||||
|
||||
Today `ingestPdf` only passes through caller-supplied `title`/`uri`/`metadata`.
|
||||
This workplan extracts intrinsic document metadata (PDF info dictionary,
|
||||
embedded XMP, page-derived signals like author/creation date/title) and folds
|
||||
it into the `Document` record without breaking the pure-over-bytes contract.
|
||||
|
||||
## Sketch
|
||||
|
||||
```task
|
||||
id: ESRC-WP-0003-T01
|
||||
status: todo
|
||||
priority: low
|
||||
```
|
||||
Enumerate available PDF metadata sources (info dict, XMP) via PDF.js and decide
|
||||
precedence vs. caller-supplied options (caller wins).
|
||||
|
||||
```task
|
||||
id: ESRC-WP-0003-T02
|
||||
status: todo
|
||||
priority: low
|
||||
```
|
||||
Implement extraction into a normalized metadata shape; keep it optional so
|
||||
minimal-metadata PDFs still ingest cleanly.
|
||||
|
||||
```task
|
||||
id: ESRC-WP-0003-T03
|
||||
status: todo
|
||||
priority: low
|
||||
```
|
||||
Contract tests over the fixture corpus asserting extracted vs. overridden
|
||||
metadata precedence.
|
||||
47
workplans/ESRC-WP-0004-citation-recovery.md
Normal file
47
workplans/ESRC-WP-0004-citation-recovery.md
Normal file
|
|
@ -0,0 +1,47 @@
|
|||
---
|
||||
id: ESRC-WP-0004
|
||||
type: workplan
|
||||
title: "Local citation recovery and external source discovery hooks"
|
||||
domain: infotech
|
||||
repo: evidence-source
|
||||
status: proposed
|
||||
owner: codex
|
||||
topic_slug: citation_evidence_mvp
|
||||
created: "2026-07-08"
|
||||
updated: "2026-07-08"
|
||||
spec_refs:
|
||||
- INTENT.md
|
||||
- README.md
|
||||
---
|
||||
|
||||
# ESRC-WP-0004 — Citation recovery and source discovery hooks
|
||||
|
||||
`evidence-source`'s original intent includes recovering citations when a stored
|
||||
selector goes stale and discovering the external source a document was drawn
|
||||
from. This is the largest deferred strand and depends on the anchoring layer's
|
||||
selector model.
|
||||
|
||||
## Open questions
|
||||
|
||||
- What is the recovery contract — does this repo *detect* stale selectors, or
|
||||
only provide the re-ingest/re-fingerprint primitives the anchor layer calls?
|
||||
- Where does external source discovery (URL/DOI resolution) belong relative to
|
||||
`citation-engine` and the umbrella?
|
||||
|
||||
## Sketch
|
||||
|
||||
```task
|
||||
id: ESRC-WP-0004-T01
|
||||
status: todo
|
||||
priority: low
|
||||
```
|
||||
Design the recovery boundary with the anchor layer: define what
|
||||
`evidence-source` owns vs. what stays in anchoring.
|
||||
|
||||
```task
|
||||
id: ESRC-WP-0004-T02
|
||||
status: todo
|
||||
priority: low
|
||||
```
|
||||
Prototype re-ingest/re-fingerprint recovery primitives and a discovery hook
|
||||
interface (no network coupling baked into the core).
|
||||
Loading…
Add table
Add a link
Reference in a new issue