artifact-store/docs/REVIEW-2026-05-15-intent-and-blueprint.md

# Review — INTENT and Architecture Blueprint

Date: 2026-05-15
Reviewer: claude (opus-4-7)
Inputs: `INTENT.md`, `docs/ARCHITECTURE-BLUEPRINT.md`,
`workplans/ARTIFACT-STORE-WP-0001-service-baseline.md`, `SCOPE.md`, `AGENTS.md`

This review reframes the current scoped-internal-service design against a
longer-horizon ambition: make `artifact-store` the leading open source
substrate for generic artifact storage in the same sense that VLC and ffmpeg
lead their domain. See `docs/PLATFORM-AMBITION.md` for the ambition framing
this review is in service of.

## SWOT

### Strengths

- Clean separation between artifact *identity / lifecycle* and *bytes*.
  Registry owns metadata; storage adapter owns persistence. This is the single
  most consequential architectural decision and the docs get it right.
- Retention is a first-class concept from day one, not bolted on later.
- Audit log designed in from the start, with explicit room for signed events.
- Storage adapter contract is minimal and well-shaped
  (`put / get / head / delete / health`).
- Pilot-first discipline (`guide-board` / OpenCMIS TCK) anchors the work in a
  real producer rather than a hypothetical one.
- Manifest portability is an explicit goal — a package should be understandable
  without calling its producer.
- Boundary statements are explicit (will not replace StateHub, will not encode
  producer semantics).

### Weaknesses

- Storage is keyed by logical path, not by content hash. Blocks global
  deduplication, Merkle integrity proofs, partial replication, federation.
- No streaming, chunked, or resumable upload story. Multipart REST will cap
  throughput at the slowest Python/WSGI hop for multi-GB packages.
- No content-defined chunking (CDC). Evidence packages with logs are highly
  dedup-able; current design captures none of that.
- SHA-256 is the right *compatibility* digest but the wrong *throughput*
  digest at platform scale.
- Single-writer SQLite is a real concurrency ceiling; PostgreSQL helps but no
  partitioning / sharding story exists.
- No event / CDC stream for downstream consumers — Statehub, search, UIs would
  have to poll.
- No signing / attestation story (Sigstore, in-toto, SLSA). Evidence storage
  without signed attestations leaves half the value on the table.
- Metadata is open-ended JSON without a schema-registration path. Hard to
  build typed tooling on top.
- No multi-tenancy, quota, or rate-limiting primitives. Painful to retrofit.
- No observability targets (latency / throughput SLOs, metrics, traces).
  Platform-grade claims will eventually require numbers.
- No OCI / `oras` artifact compatibility — leaves the largest existing
  artifact ecosystem off the table.

### Opportunities

- **OCI Artifact + ORAS compatibility.** Inherit Helm, ML model, SBOM, cosign
  tooling for free. Probably the single highest-leverage external move.
- **Sigstore + in-toto + SLSA.** Evidence packages should be signed by
  default; this is exactly the gap most generic registries leave unfilled.
- **Content-addressed CAS + Merkle DAG** (Git / IPFS / restic pattern):
  enables global dedup, integrity proofs, federation, partial mirroring.
- **BLAKE3** as native digest with SHA-256 retained for interop:
  orders-of-magnitude faster hashing, and BLAKE3's construction *is* a Merkle
  tree — package-level integrity comes for free.
- **WASM plugin surface for transforms, extractors, indexers, redactors.**
  The "ffmpeg moment" for this domain: a stable host API that ecosystem
  contributors can extend without forking the core.
- **Federation / mirroring** between artifact-store instances via signed
  manifests. Nothing comparable exists in the evidence space today.
- **FUSE / NFS / S3-gateway frontends.** Legacy producers ingest without code
  changes.
- **Embeddable mode.** A single static binary like `restic`, plus a server
  mode. Embedding is what makes ffmpeg ubiquitous.

### Threats

- Crowded adjacency: MinIO, Pulp, Harbor / Zot, Artifactory / Nexus, restic,
  IPFS, Sigstore, plain S3. None are exactly this, but each chips at the
  value proposition.
- Scope creep vs the carefully-scoped INTENT. The platform ambition pulls
  toward "do everything"; the INTENT pulls toward "ship the pilot." Resolve
  this tension explicitly or you get neither.
- Python performance ceiling on the data plane (ingestion of multi-GB
  packages, hashing, chunking).
- Governance / maintenance debt. VLC and ffmpeg have decades of contributor
  depth; underestimating that is a project-killer.

## Architecture optimizations worth taking now

Each of these is cheap to lock in before code lands, and expensive (or
breaking) to add later.

1. **Split control plane from data plane.** Registry / API / retention stays
   in Python with PostgreSQL. Ingestion + hashing + storage I/O becomes a
   separate process (Rust sidecar, eventually with hot kernels in C / asm)
   that can scale and be rewritten independently. Pin the contract now (Unix
   socket, gRPC or framed bincode). See `docs/PLATFORM-AMBITION.md`.
2. **Make content the primary address.** Internal object key
   `blake3:<digest>` (or `sha256:<digest>` for compat). `relative_path`
   becomes logical metadata in the manifest. Unlocks dedup, integrity,
   federation, OCI compatibility.
3. **Append-only WAL as the source of truth.** Metadata DB is a materialized
   view rebuildable from the log. Same pattern as Kafka / EventStore /
   Datomic. Cheap audit, replication, point-in-time recovery.
4. **OCI artifact spec as a wire format**, even if the native API is richer.
   Buys instant interop with `oras`, `cosign`, `crane`, Helm.
5. **Signed manifests from day one.** Pin a signing format (cosign / Sigstore)
   and a canonicalization (JCS or canonical CBOR). Post-hoc signing means
   every legacy manifest is unsigned forever.
6. **Resumable, chunked uploads on the wire.** Upload session resource
   (`POST /uploads` → `PATCH /uploads/{id}` ranges → `POST /uploads/{id}/complete`).
   `tus.io` is a reasonable reference. v1 implementation can still be
   single-shot multipart.
7. **Event stream out.** A monotonic-sequence `events` table; consumers
   tail via long-poll, NATS, or Kafka. Trivial to add now, expensive later.
8. **Schema-typed metadata with escape hatch.** Producers register a JSON
   Schema for their metadata variant (`guide-board.run.v1`). Stored as open
   JSON, validated at ingest, queryable by typed views.
9. **Tiering as a first-class column of `storage_location`.** Promote
   `retrieval_tier` and `restore_status` into the schema now (nullable,
   default `hot`).
10. **Ship a great CLI before any UI.** ffmpeg ships a binary, not a GUI.

## Performance hotspots — where native code actually matters

Ranked by realistic impact for this workload. Adopting libraries that already
contain hand-tuned assembly is the cheap path; writing fresh assembly is an
explicit research line — see `docs/ASSEMBLY-EXPERIMENT.md`.

1. **Hashing (dominant ingest cost).** SHA-256 with SHA-NI: ~1.5–2 GB/s/core.
   BLAKE3 with AVX-512: 6–10+ GB/s/core, parallelizable, free Merkle tree.
   Adopt BLAKE3 as native; retain SHA-256 for SLSA / OCI interop.
2. **Content-defined chunking (FastCDC / Gear).** Rolling hash over every
   byte; pure-Python is unusable, optimized C / Rust hits 5–10 GB/s.
   Mandatory if dedup is on the roadmap.
3. **Compression.** Zstd with bundled SIMD reaches multi-GB/s. Evidence logs
   typically compress 5–20×. Apply at chunk level so dedup still works.
4. **I/O path.** Linux: `io_uring` for ingest writes; `sendfile(2)` /
   `splice(2)` for download zero-copy; `O_DIRECT` for very large objects.
5. **Encryption.** AES-GCM with AES-NI: ~5 GB/s/core. ChaCha20-Poly1305
   vector implementations for non-AES-NI hardware. Use Ring, BoringSSL, or
   AWS-LC. Never write crypto by hand.
6. **Metadata hot paths.** Bloom or Cuckoo filter in front of the
   "have I seen this hash?" lookup. ~50 lines of Rust, ~100× win.
7. **Manifest canonicalization.** Signed manifests canonicalize on every
   ingest and every verify. Pick a fast canonical CBOR / JCS impl.

Not worth native code: HTTP layer, retention engine, audit log, DB access,
orchestration, workflow logic. Keep Python.

## Concrete suggestions before WP-0001 lands

- Add `digest_algorithm` to `artifact_files` (default `sha256`, allow
  `blake3`).
- Add `content_address` (e.g., `blake3:…`) as canonical storage key, with
  `relative_path` retained as logical metadata.
- Add `retrieval_tier` and `restore_status` to `storage_locations` now,
  nullable.
- Define the upload session resource shape even if v1 implements only
  single-shot multipart.
- Pin a manifest canonicalization (recommend JCS or canonical CBOR) and a
  signing format target (cosign / Sigstore). Decide, do not implement.
- Add an `events` table with a monotonic sequence number so a CDC feed is
  trivial later.
- Decide explicitly whether OCI artifact compatibility is a v2 goal or out of
  scope. Either is fine; ambiguity will distort schema decisions.

## What this review does not change

INTENT and SCOPE remain correctly scoped for v1. The pilot path through
WP-0001 should ship as planned. The schema annotations above are additive,
not redirective. The platform ambition lives in `docs/PLATFORM-AMBITION.md`
so it can guide later decisions without expanding the current workplan.