diff --git a/INTENT.md b/INTENT.md index f4ee860..35c9947 100644 --- a/INTENT.md +++ b/INTENT.md @@ -82,6 +82,8 @@ The rebuild is deliberately **product-faithful first**: UI and content parity ma - `docs/resource-evidence/` — workload demand forecasts, observations, labor - `docs/capability/` — product capability/feature model; stage-1 cutover must set - `docs/decisions/2026-08-12-feature-cut-stage-1.md` — partial rebuild for productive transfer +- `docs/architecture/ArchitectureBlueprint.md` — content plane, NetKingdom, assets (discussion) +- `docs/architecture/spike-checklist.md` — evaluation spikes for store choice - `docs/architecture/2026-08-12-content-substrate-exploration.md` — markdown SoR, markitect-tool, shard-wiki eval - `the-custodian/docs/coulomb-social-rebuild-seed.md` — original workplan seed (CUST-WP-0058-T08) - `the-custodian/canon/standards/business-app-service-contract_v0.1.md` diff --git a/docs/architecture/2026-08-12-content-substrate-exploration.md b/docs/architecture/2026-08-12-content-substrate-exploration.md index 3e389bf..a863643 100644 --- a/docs/architecture/2026-08-12-content-substrate-exploration.md +++ b/docs/architecture/2026-08-12-content-substrate-exploration.md @@ -146,11 +146,12 @@ Rebuild / reimport should not require coulomb’s Postgres. Cache tables are dis ## Open evaluation tasks (if we continue) -1. Spike: coulomb → markitect-tool parse/serialize of a stage-1 page document. -2. Spike: shard-wiki single-space InformationSpace + FolderAdapter or native engine; implement copy/transfer as engine ops or app-level compose. -3. Compare authz model: shard-wiki page ACLs vs coulomb SpaceMembership vs NetKingdom groups. -4. Decide cache invalidation: write to markdown first, then reindex cards. -5. Promote winners into **ADR-0003** (information model + substrate) and revise capability model feature wording. +Executable checklist + blueprint: + +- [`ArchitectureBlueprint.md`](./ArchitectureBlueprint.md) +- [`spike-checklist.md`](./spike-checklist.md) + +Includes kontextual-engine option, asset-store extraction thesis, and NetKingdom PolicyGateway gap. ## Summary decisions from this note diff --git a/docs/architecture/ArchitectureBlueprint.md b/docs/architecture/ArchitectureBlueprint.md new file mode 100644 index 0000000..1c03434 --- /dev/null +++ b/docs/architecture/ArchitectureBlueprint.md @@ -0,0 +1,279 @@ +# Architecture Blueprint — coulomb.social content & platform + +| Field | Value | +|-------|--------| +| Status | **draft for discussion** (2026-08-12) | +| Product | coulomb.social rebuild (`coulomb-social`) | +| Related | stage-1 cut, capability model, content-substrate exploration | +| Non-goals | Final store vendor lock-in; implementing UI chrome in this doc | + +## 1. Architectural aim + +Deliver a co-creation product where: + +1. **Identity** (users, groups, tenants) is **NetKingdom** — not reimplemented. +2. **Runtime** is **Railiance** — not reimplemented. +3. **User-owned content** is **markdown (+ assets)** that can be exported and reused without the app database. +4. The **database is a cache / projection** for cards, memberships, and speed. +5. **markitect-tool** enables efficient structured markdown (lib/service), not a product host. +6. **coulomb** owns co-creation UI (playing-card pages, spaces, transfer/copy) and product verbs. + +## 2. Non-negotiable rules + +| # | Rule | +|---|------| +| R1 | Markdown chunks (frontmatter + body + backmatter) + assets are the **exportable SoR** for content. | +| R2 | Postgres (or any app DB) holds **projections and app linkage only** for content; disposable for bodies. | +| R3 | Authn/IdP = **NetKingdom**; coulomb maps OIDC → Member; no password store. | +| R4 | Authz for product actions may compose NetKingdom groups, app membership, and content-plane ACLs — **no second user directory**. | +| R5 | **markitect-main** is legacy reference, not a runtime foundation. | +| R6 | **markitect-tool** is the markdown structure dependency (parse/validate/transform). | +| R7 | Content-plane choice (shard-wiki / kontextual / thin store) is behind a **narrow page-ops interface** so coulomb UI does not couple to federation or ECM graphs. | +| R8 | Stage-1 capability set remains: onboarding, spaces/pages with Title/Abstract/Visual, page copy, page transfer. | + +## 3. System shape + +```text + ┌──────────────────────────┐ + │ NetKingdom │ + │ OIDC, groups, tenants │ + └────────────┬─────────────┘ + │ principal + ▼ +┌─────────────────────────────────────────────────────────────┐ +│ coulomb.social (product) │ +│ • Card UI, spaces chrome, stage-1 product flows │ +│ • Session / Member / SpaceMembership projections │ +│ • PageOps façade: list/get/create/update/archive/copy/move │ +│ • Railiance deploy (Helm, env, probes) │ +└───────────────┬─────────────────────────────┬───────────────┘ + │ │ + │ projections │ content ops + ▼ ▼ + ┌───────────────┐ ┌───────────────────────────┐ + │ App DB cache │ │ Content plane (choose one │ + │ (Postgres) │ │ primary adapter) │ + └───────────────┘ │ A thin folder/git store │ + │ B shard-wiki │ + │ C kontextual-engine │ + └─────────────┬─────────────┘ + │ structure + ▼ + ┌───────────────────────────┐ + │ markitect-tool │ + │ front/body/back, schema │ + └───────────────────────────┘ + │ + ┌─────────────▼─────────────┐ + │ Asset bytes (see §6) │ + │ content-addressed blobs │ + └───────────────────────────┘ +``` + +**Product verbs (PageOps)** — store-agnostic: + +| Op | Meaning | +|----|---------| +| `list_pages(space)` | Card projections | +| `get_page(id)` | Full markdown + resolved visual | +| `put_page(...)` | Create/update attributes + body | +| `archive_page` | Soft-delete / lifecycle | +| `copy_page` | Independent duplicate | +| `transfer_page` | Re-home to another space | +| `put_visual` / `get_asset` | Binary visual/attachment | + +Coulomb never talks to Forgejo/shard/kontextual APIs ad hoc from templates — only through PageOps. + +## 4. Information model (product) + +| Concept | Representation | +|---------|----------------| +| **Page** | Markdown document: frontmatter (title, abstract, visual ref, space, type, …) + body + backmatter (relations, provenance) | +| **Space** | Container context; may be a page of `type: space` or a first-class container id in frontmatter | +| **Visual** | Asset ref in frontmatter → blob/asset service | +| **Card** | UI projection: visual + title + abstract + actions (not a storage type) | +| **Member** | NetKingdom subject ↔ app Member row | +| **Tenant** | Instance / `tenant_id` (DR-1 C) | + +## 5. Content-plane options (comparison) + +### 5.1 Option A — Thin folder/git store + markitect-tool + +| | | +|--|--| +| **Idea** | Layout of `.md` files + assets; optional git; markitect validates; app indexes cards | +| **Pros** | Fastest stage-1; trivial user export; low structure surface | +| **Cons** | Reimplements copy/transfer/versioning carefully; federation later is a rewrite | +| **When** | Need product velocity; content plane maturity unclear | + +### 5.2 Option B — shard-wiki (thin slice) + +| | | +|--|--| +| **Idea** | Headless wiki orchestration; pages as union of shards; IdP external; coulomb = UI | +| **Pros** | Built for wiki pages, git coordination, overlays, multi-shard (personal + space); INTENT names Coulomb; no UI ownership | +| **Cons** | Early product maturity for multi-tenant ops; federation API can over-structure if not sliced thin | +| **When** | Want page federation / export shards / git-addressable coordination as first-class | +| **Thin-slice rule** | One InformationSpace per instance; one primary shard; hide federation from card UI | + +### 5.3 Option C — kontextual-engine (knowledge ops / CMS-ECM backend) + +| | | +|--|--| +| **Idea** | Headless knowledge operations: durable **KnowledgeAsset** identity, representations, metadata, relationships, workflows, audit, blob storage, optional CMIS; markdown via markitect-tool adapter | +| **Pros** | Strong identity for assets beyond path; lineage, versioning, export packages, multi-format future; **blob layer already implemented** (memory/fs/S3, content-addressed); PolicyGateway port ready for external authz; aligns with “ECM without being the UI” | +| **Cons** | Heavier than pure wiki; **not streamlined for NetKingdom** today (actors are engine-local; default `AllowAllPolicyGateway`); markdown is *a* format among many — SoR discipline must force “source representation is markdown file users can take” | +| **When** | Need governed asset registry, visuals/blobs, audit, multi-format, CMIS-ish interoperability; willing to invest PolicyGateway → flex-auth / NetKingdom | +| **Auth gap** | `Actor` + `external_ref` can map to OIDC subject; `PolicyGateway.authorize` must call **flex-auth** (or equivalent) with NetKingdom groups — not reimplement IAM inside kontextual | + +**kontextual already has (relevant):** + +- Asset registry (create, metadata, lifecycle, versions, relationships, audit) +- BlobStorage port + local/S3/memory adapters, dedupe by sha256 +- Representation kinds (source / normalized / derived) +- markitect-tool reuse boundary docs + adapter package +- Service API surface for assets/retrieval/workflows +- Policy port (pluggable); default allow-all for tests + +**kontextual does not yet have (for coulomb):** + +- First-class “space + card product model” +- Wired NetKingdom / KeyCape session → Actor +- Production flex-auth policy adapter +- Guaranteed “markdown file tree export is the happy path” product packaging (export exists as concept; productize for coulomb) + +### 5.4 How options relate (not exclusive forever) + +```text +coulomb PageOps + │ + ├─ short term: A or thin B/C adapter + │ + └─ adapters can evolve: + folder/git shard ──► shard-wiki InformationSpace + markdown assets ──► kontextual KnowledgeAsset + BlobStorage + markitect-tool ──► always structure layer +``` + +shard-wiki optimizes **wiki federation and page union**. +kontextual optimizes **governed multi-format knowledge assets + blobs + audit**. +coulomb optimizes **co-creation product**. + +A hybrid is possible later: shard-wiki as page navigation plane, kontextual as asset/blob + governance — only if spikes show value without double-structure tax. + +## 6. Asset management — extract to its own repo? + +### 6.1 What exists today + +| Source | Asset story | +|--------|-------------| +| **kontextual-engine** | Clean **BlobStorage** protocol + content-addressed store + representation linkage; small, intentional, portable adapters (~memory/fs/S3) | +| **markitect-main** `markitect/assets/` | Large (~0.5M+ tree) manager/registry/packager/optimizer/analytics — **orphan** in succession map; do not run coulomb on it | +| **coulomb** | No first-class visual/blob store yet (stage-1 Visual is todo) | + +### 6.2 Extraction thesis + +**Yes, worth evaluating a dedicated `asset-store` (name TBD) capability** if: + +1. **coulomb**, **kontextual**, **shard-wiki**, and future apps all need content-addressed bytes + metadata without pulling full ECM or full wiki. +2. The core contract is small: `put/get/stat/delete_unreferenced` + digest + media type + optional ACLs. +3. We want Railiance-wide reuse (space visuals, wiki attachments, kontextual representations). + +**Prefer extracting from kontextual’s BlobStorage** (clean ports), **not** from markitect-main’s asset mega-package (mine ideas only: dedupe, mdpkg packaging). + +### 6.3 Suggested extraction shape (discussion) + +```text +asset-store (new repo or helix capability) + ports: BlobStorage (digest-addressed) + adapters: fs, s3, memory + optional: reference accounting, GC, HTTP byte range + NOT: full asset registry, workflows, CMIS, markdown parsing + +kontextual-engine + keeps KnowledgeAsset + Representation pointing at storage_ref + depends on asset-store library/service + +coulomb / shard-wiki + may depend on asset-store for Visual without full kontextual +``` + +**Decision gate for extraction:** after coulomb stage-1 Visual needs a real store **and** a second consumer wants the same API. Until then, use kontextual blob adapters **in-process** or vendor the port as a small shared package without full service. + +## 7. Identity & access control + +```text +Browser → coulomb (session after NetKingdom OIDC) + │ + ├─ product authz: SpaceMembership, tenant_id + │ + └─ content plane actor: + Actor(external_ref=issuer|subject, groups=OIDC groups) + PolicyGateway → flex-auth (preferred) or engine rules +``` + +| Plane | Responsibility | +|-------|----------------| +| NetKingdom | Who is the user; groups/roles claims | +| coulomb | Session; Member; space membership product rules | +| flex-auth | Optional PDP for resource actions | +| Content plane | Enforce policy on get/put content; fail closed | + +**Spike required:** map one OIDC login → kontextual `Actor` → deny-by-default policy for foreign tenant assets. + +## 8. Deploy sketch (Railiance) + +| Component | Deploy | +|-----------|--------| +| coulomb-social | Existing chart / app.coulomb.social | +| Content plane | Sidecar or separate service in `coulomb-social` ns or shared platform ns | +| asset bytes | PVC and/or S3-compatible; secrets via existing env-secret pattern | +| NetKingdom | Existing KeyCape / Authelia | + +No apex DNS change until stage-1 cut. + +## 9. ADR roadmap (proposed) + +| ADR | Topic | +|-----|--------| +| ADR-0001 | NetKingdom identity (done) | +| ADR-0002 | Forgejo markdown binding (provisional slice → demote to adapter) | +| **ADR-0003** | Page-centric model + markdown SoR + DB cache (to write after spikes) | +| **ADR-0004** | Content-plane choice (A/B/C) + PageOps boundary | +| **ADR-0005** | Asset-store extraction (if/when) | + +## 10. Spike checklist + +See **[spike-checklist.md](./spike-checklist.md)** — executable evaluation tasks. + +## 11. Discussion prompts + +1. Is **user export as markdown tree** more important than **multi-format ECM** in the next 6 months? (Biases A/B vs C.) +2. Do we need **multi-shard federation** (personal vault + space) at stage 1, or only later? (Biases B.) +3. Should **Visual** block on asset-store extraction, or ship with kontextual/local blob first? +4. Who owns **PolicyGateway** production adapter — coulomb, kontextual, or flex-auth team? +5. Accept **hybrid**: kontextual for assets+registry, thin git for human-facing markdown tree sync? + +## 12. Working recommendation (to debate, not freeze) + +| Layer | Recommendation | +|-------|----------------| +| Product UI | coulomb cards + spaces | +| Markdown structure | markitect-tool | +| Identity / deploy | NetKingdom / Railiance | +| Content plane shortlist | Spike **B thin** and **C** in parallel; keep **A** as fallback | +| SoR narrative | User-facing SoR = markdown export; engine may add durable IDs/lineage **without** trapping content in opaque DB-only forms | +| Assets | Prefer kontextual BlobStorage now; extract **asset-store** when second consumer appears | +| markitect-main | Avoid | + +--- + +## References + +- `docs/architecture/2026-08-12-content-substrate-exploration.md` +- `docs/capability/stage-1-cutover.md` +- `docs/decisions/2026-08-12-feature-cut-stage-1.md` +- kontextual: `docs/architecture-blueprint.md`, `asset-registry-implementation.md`, `blob-storage-content-streaming-workplan.md`, `markitect-tool-reuse-boundary.md` +- shard-wiki: `INTENT.md`, `SCOPE.md` +- markitect-tool: `docs/markitect-main-scope-assessment.md` +- markitect-main succession: `docs/successor-gap-assessment.md` diff --git a/docs/architecture/spike-checklist.md b/docs/architecture/spike-checklist.md new file mode 100644 index 0000000..2308414 --- /dev/null +++ b/docs/architecture/spike-checklist.md @@ -0,0 +1,205 @@ +# Spike checklist — content plane & assets + +| Field | Value | +|-------|--------| +| Blueprint | [ArchitectureBlueprint.md](./ArchitectureBlueprint.md) | +| Goal | Falsify or confirm options A/B/C and asset extraction before ADR-0003/0004 | +| Time box | Prefer ≤2–3 working days per major spike; stop when decision quality is enough | + +Scoring legend for each spike: **Pass** / **Partial** / **Fail** / **Skip** with one-line evidence. + +--- + +## Spike 0 — Shared page document fixture + +**Owner:** coulomb-social +**Depends on:** none + +- [ ] Define one **stage-1 page** markdown fixture: + - frontmatter: `id`, `type`, `title`, `abstract`, `visual`, `space` + - body: short markdown + - backmatter: e.g. `relations` or provenance stub +- [ ] Define one **space** container convention (page-type vs directory). +- [ ] Define **export package** layout (`pages/**`, `assets/**`, `MANIFEST.json`). +- [ ] Commit fixtures under `docs/space-fixtures/` or `docs/architecture/fixtures/`. + +**Pass when:** fixture is stable enough that all later spikes use the same bytes. + +--- + +## Spike 1 — markitect-tool as structure lib + +**Owner:** coulomb-social (+ markitect-tool if API gaps) +**Depends on:** Spike 0 + +- [ ] From Python in coulomb (or throwaway script): parse fixture with markitect-tool public API. +- [ ] Round-trip serialize without losing title/abstract/visual fields. +- [ ] Validate required frontmatter fields (even minimal schema). +- [ ] Note missing APIs (backmatter support, field maps) as markitect-tool gaps. + +**Pass when:** coulomb can treat markitect-tool as dependency for read/write structure without shelling out to `mkt` for the happy path. + +**Fail when:** only CLI works or public API cannot preserve frontmatter stably. + +--- + +## Spike 2 — Thin folder/git store (Option A) + +**Owner:** coulomb-social +**Depends on:** Spike 0–1 + +- [ ] Layout: `spaces//pages/.md` + `assets/`. +- [ ] Implement throwaway PageOps: list, get, put, copy, transfer (filesystem). +- [ ] Reindex card projection into SQLite/Postgres (title, abstract, visual path, space). +- [ ] Export = tar/zip of tree; reimport without DB. +- [ ] Measure: copy/transfer LOC and failure modes (broken links, name collision). + +**Pass when:** stage-1 verbs work on files alone; export/import demonstrated. + +**Decision input:** Is A enough for 3–6 months without federation? + +--- + +## Spike 3 — shard-wiki thin slice (Option B) + +**Owner:** coulomb-social + shard-wiki +**Depends on:** Spike 0 + +- [ ] Boot one InformationSpace with **single** FolderAdapter or native engine shard. +- [ ] Ingest/edit stage-1 page; read via headless API (no UI). +- [ ] Implement or map: list, get, put; prototype copy/transfer (engine op or app-level). +- [ ] Attach **fake Actor** from static OIDC-like subject; confirm authz hooks exist or gap list. +- [ ] Document API surface coulomb would depend on (count endpoints/concepts). +- [ ] Stress “structure tax”: can card UI ignore federation/overlays entirely? + +**Pass when:** thin slice delivers page CRUD without coulomb importing federation UX concepts. + +**Fail when:** product code must understand multi-shard resolution for basic CRUD. + +**Decision input:** structure surface area vs future multi-shard value. + +--- + +## Spike 4 — kontextual-engine page + markdown SoR (Option C) + +**Owner:** coulomb-social + kontextual-engine +**Depends on:** Spike 0–1 + +- [ ] Create `KnowledgeAsset` for stage-1 page; store **source** representation as markdown bytes (BlobStorage). +- [ ] Put title/abstract/visual in metadata and/or inside markdown frontmatter (document which is authoritative for export). +- [ ] Get content stream; reconstruct `.md` file bit-identical (or document lossiness). +- [ ] Run markitect-tool adapter on stored markdown (existing boundary tests as reference). +- [ ] Prototype copy = new asset + blob refs; transfer = metadata/relationship change to space context entity. +- [ ] Export package: write assets + markdown to filesystem from engine API. + +**Pass when:** user-visible export is clean markdown+assets; engine IDs are additive not mandatory for reuse. + +**Fail when:** content is only queryable as opaque registry rows without faithful markdown export. + +**Decision input:** governance/lineage value vs stage-1 complexity. + +--- + +## Spike 5 — NetKingdom / PolicyGateway wiring + +**Owner:** coulomb + kontextual (and/or flex-auth) +**Depends on:** Spike 3 or 4 + +- [ ] Map live or mocked OIDC claims → `Actor(external_ref=issuer|sub, groups=...)`. +- [ ] Implement **deny-by-default** PolicyGateway stub that allows only same-tenant resource metadata. +- [ ] Optional: call flex-auth `check` with coulomb vocabulary (if service available). +- [ ] Prove fail-closed: missing policy → no content bytes. +- [ ] Document gap list for production adapter ownership. + +**Pass when:** written design + failing/passing test for cross-tenant deny. + +**Fail when:** only AllowAll can run the stack. + +--- + +## Spike 6 — Asset / Visual path + +**Owner:** coulomb (+ kontextual or extracted store) +**Depends on:** Spike 2 or 4 + +- [ ] Store a cover image via content-addressed put; reference from page frontmatter `visual: sha256:…` or `asset:…`. +- [ ] Serve image for card UI (signed URL, app proxy, or static mount — pick one). +- [ ] GC/unreferenced dry-run once. +- [ ] Compare effort: in-process kontextual BlobStorage vs copy port to shared package. + +**Pass when:** card shows visual without storing image bytes in Postgres. + +**Decision input for extraction:** is a second consumer ready? If no → keep in kontextual for now. + +--- + +## Spike 7 — Asset-store extraction feasibility + +**Owner:** platform / kontextual +**Depends on:** Spike 6 + +- [ ] Inventory kontextual `BlobStorage` + adapters (fs/s3/memory) public surface. +- [ ] Inventory markitect-main `markitect/assets/` for **ideas only** (dedupe, packaging); do not port wholesale. +- [ ] Draft package name + repo boundary: bytes only vs registry. +- [ ] Estimate extraction PR: move port+adapters, leave KnowledgeAsset in kontextual. +- [ ] List consumers: coulomb, kontextual, shard-wiki (optional). + +**Pass when:** one-page extraction proposal with “extract now / later / never” recommendation. + +**Default bias:** **later** until two consumers need it. + +--- + +## Spike 8 — Cross-option decision workshop + +**Owner:** founder + implementers +**Depends on:** Spikes 1–6 results + +- [ ] Fill scorecard (below). +- [ ] Choose primary content plane for stage 1 + migration path. +- [ ] Choose asset strategy (in-kontextual vs extract vs folder-only). +- [ ] File ADR-0003 (page model + SoR) and ADR-0004 (content plane). +- [ ] Update `docs/capability/model-v0.yaml` feature wording if PageOps replaces Forgejo-centric features. + +### Scorecard (1–5, higher better) + +| Criterion | A thin store | B shard-wiki | C kontextual | +|-----------|:------------:|:------------:|:------------:| +| Markdown export fidelity | | | | +| Stage-1 speed | | | | +| Structure tax (inverse) | | | | +| Copy/transfer naturalness | | | | +| Visual/blob readiness | | | | +| NetKingdom/authz fit | | | | +| Multi-shard / federation later | | | | +| Multi-format / ECM later | | | | +| Railiance operability | | | | +| Aligns with fleet layers | | | | +| **Total** | | | | + +--- + +## Suggested order + +```text +0 fixture → 1 markitect → 2 thin store (always, baseline) + ├→ 3 shard-wiki thin + └→ 4 kontextual page+blob + → 5 authz on winner(s) + → 6 visual + → 7 extraction feasibility + → 8 decision / ADRs +``` + +Spike 2 is the **control experiment**: even if B or C wins, A defines the export contract users feel. + +--- + +## Exit criteria for this checklist + +Stop spiking when: + +1. One primary content plane is chosen for stage 1 with explicit residuals, and +2. Asset strategy is chosen (where Visual bytes live), and +3. NetKingdom policy gap is owned (who implements PolicyGateway adapter), and +4. ADR-0003/0004 drafts exist or are scheduled.