artifact-store/workplans/ARTIFACT-STORE-WP-0004-s3-compatible-backend.md

---
id: ARTIFACT-STORE-WP-0004
type: workplan
title: "S3-Compatible Backend (Ceph RGW Target)"
repo: artifact-store
domain: infotech
status: done
owner: codex
topic_slug: stack
planning_priority: medium
planning_order: 4
created: "2026-05-15"
updated: "2026-05-17"
state_hub_workstream_id: "d0526cfc-e532-431f-970d-f3e548d27a80"
---

# ARTIFACT-STORE-WP-0004: S3-Compatible Backend

## Purpose

Add a second concrete storage backend that speaks the S3 protocol.
Validated targets: Ceph RGW (primary self-hosted production target),
MinIO (dev / CI), AWS S3 (interop check). The backend must satisfy
the storage SPI without any leaks of S3-specific concepts into the
registry.

## Constraints

- `storage.spi.StorageBackend` Protocol from WP-0001 is the contract.
- No S3 vocabulary leaks into `registry.*` or `api.*`.
- `docs/ARCHITECTURE-BLUEPRINT.md` storage-backend section.

## Prerequisites

- WP-0001 done (SPI exists, local backend exists as a reference).

## D4.1 - Configuration Surface

```task
id: ARTIFACT-STORE-WP-0004-T001
status: done
priority: high
state_hub_task_id: "1db0d548-cdac-4b07-962b-bcafa3aae30e"
```

Acceptance:

- `s3` backend configuration accepts: `endpoint_url`, `region`,
  `bucket`, `key_prefix`, `access_key_ref`, `secret_key_ref`,
  `storage_class`, `sse` (optional), `multipart_threshold_bytes`,
  `multipart_chunk_bytes`.
- Credential references resolve from env vars or mounted files; never
  from request bodies.
- Documented Ceph RGW configuration example checked in under
  `docs/OPERATOR.md`.

## D4.2 - S3 Backend Implementation

```task
id: ARTIFACT-STORE-WP-0004-T002
status: done
priority: high
state_hub_task_id: "14b50595-5820-4369-b037-b015fcbddcc4"
```

Acceptance:

- `storage.backends.s3.S3Backend` implements the SPI using `aioboto3`
  or `aiobotocore` (decision recorded in the workplan; whichever is
  better-maintained at implementation time).
- Object key layout
  `<key_prefix>/<digest_algorithm>/<hex[0:2]>/<hex[2:4]>/<hex>`.
- `put` uses multipart for objects above the configured threshold.
- `get` supports `Range`.
- `head`, `delete`, `health` implemented.
- `delete` is idempotent (delete-of-missing returns success).

Decision: use `aioboto3` as the optional S3 client dependency. The
backend imports it lazily so local-only deployments do not need S3
dependencies installed.

## D4.3 - Backend Selection And Routing

```task
id: ARTIFACT-STORE-WP-0004-T003
status: done
priority: medium
state_hub_task_id: "725dafd6-3337-4f81-b221-bb9f3a564d7e"
```

Acceptance:

- A registry can have multiple backends configured; package creation
  records which backend a file is stored in.
- Per-package backend selection rule: configurable function of
  `retention_class` + producer; default routes everything to a single
  backend.
- `storage_locations.backend_id` reflects the actual storage.

## D4.4 - Test Strategy: MinIO In CI, RGW As Documented Manual Smoke

```task
id: ARTIFACT-STORE-WP-0004-T004
status: done
priority: high
state_hub_task_id: "4fd7b73b-7058-4edd-b5e3-edca396760d4"
```

Acceptance:

- Integration tests run against MinIO via `testcontainers-python`
  (or a docker-compose fixture if testcontainers fights the WSL2
  environment).
- A documented manual procedure tests against a real Ceph RGW
  endpoint; results recorded in `docs/OPERATOR.md`.
- No CI dependency on a live Ceph or AWS account.

Closure note: the S3 backend implementation and local verification
for artifact-store are complete. MinIO-specific compatibility,
testcontainers/bootstrap, and community-fork assessment have been moved
to ARTIFACT-STORE-WP-0007 so this backend workstream can close without
hiding the remaining external-platform work.

## D4.5 - Verification Pass

```task
id: ARTIFACT-STORE-WP-0004-T005
status: done
priority: medium
state_hub_task_id: "5a55546f-288f-4da0-a646-3d9319908279"
```

Acceptance:

- `artifactstore storage verify --backend s3` re-reads every object in
  the backend, recomputes its primary digest, and emits
  `v1.storage.location_verified` events.
- Mismatches are reported as `failed` locations and surfaced via the
  health endpoint.

## Success criteria

- The same package ingestion flow that worked against `local` in
  WP-0001 works unchanged against `s3`.
- Switching backend by config — without code changes in the registry
  or API layers — is the smoke test.