From c27a50e6d3055ab110822377a30e79ac6cee51e2 Mon Sep 17 00:00:00 2001 From: tegwick Date: Thu, 13 Aug 2026 00:27:44 +0200 Subject: [PATCH] Revise ArchitectureBlueprint with full discussion synthesis Fold page-first/transclusion, dual authoring, lazy projections, stale replace-or-keep, and open content-plane decisions into the main draft. --- docs/architecture/ArchitectureBlueprint.md | 353 +++++++++++---------- 1 file changed, 189 insertions(+), 164 deletions(-) diff --git a/docs/architecture/ArchitectureBlueprint.md b/docs/architecture/ArchitectureBlueprint.md index 8e84676..6b74c12 100644 --- a/docs/architecture/ArchitectureBlueprint.md +++ b/docs/architecture/ArchitectureBlueprint.md @@ -2,10 +2,33 @@ | Field | Value | |-------|--------| -| Status | **draft for discussion** (2026-08-12) | +| Status | **draft for discussion** (revised 2026-08-13) | | Product | coulomb.social rebuild (`coulomb-social`) | -| Related | stage-1 cut, capability model, content-substrate exploration | -| Non-goals | Final store vendor lock-in; implementing UI chrome in this doc | +| Related | stage-1 cut, capability maps, substrate exploration, authoring, stale/derived | +| Non-goals | Final store vendor lock-in; full UI chrome; implementing markitect features in this doc | + +## 0. Review summary (2026-08-13) + +This revision folds in topics settled after the first draft: + +| Topic | Stance | +|-------|--------| +| Identity / tenants / groups | NetKingdom only | +| Deploy | Railiance | +| Content SoR | Markdown + assets; DB = cache | +| markitect | **tool** as structure lib; **not** “run on markitect”; **main** = legacy | +| Page vs chunk | **Page-first**; chunks = address / transclude / promote | +| Authoring | Top-down **and** bottom-up; no forced mode | +| Title / Abstractor / Visual | Real fields; may start **lazy**; then persist; **regenerate + lock** | +| Stale derived content | Badge → generate **alternative** → **replace or keep** | +| Includes | Live resolve **and** (proposed) pinned snapshots with same stale grammar | +| Content plane | Still open: thin store / shard-wiki / kontextual (spikes) | +| Assets | kontextual BlobStorage now; extract asset-store when 2nd consumer | +| Live Bubble | Mapped; S1 aligns with Title/Abstractor/Visual/spaces/pages/cards | + +**Still open (decisions, not re-litigate principles):** content-plane pick (A/B/C), visual storage short-term, PolicyGateway owner, chunk migration strategy. + +--- ## 1. Architectural aim @@ -13,23 +36,26 @@ Deliver a co-creation product where: 1. **Identity** (users, groups, tenants) is **NetKingdom** — not reimplemented. 2. **Runtime** is **Railiance** — not reimplemented. -3. **User-owned content** is **markdown (+ assets)** that can be exported and reused without the app database. -4. The **database is a cache / projection** for cards, memberships, and speed. -5. **markitect-tool** enables efficient structured markdown (lib/service), not a product host. -6. **coulomb** owns co-creation UI (playing-card pages, spaces, transfer/copy) and product verbs. +3. **User-owned content** is **markdown (+ assets)** exportable without the app database. +4. The **database is a cache / projection** for cards, memberships, search speed. +5. **markitect-tool** makes markdown an efficient structured substrate (lib/CLI/API) — **not** the product host. +6. **coulomb** owns co-creation UI (cards, spaces, copy/transfer, authoring UX) and product verbs. +7. Knowledge work stays **fluent**: top-down and bottom-up authoring; structure and reuse appear when needed (transclusion, lazy projections), not as mandatory a priori chunk graphs. ## 2. Non-negotiable rules | # | Rule | |---|------| -| R1 | Markdown chunks (frontmatter + body + backmatter) + assets are the **exportable SoR** for content. | -| R2 | Postgres (or any app DB) holds **projections and app linkage only** for content; disposable for bodies. | +| R1 | Markdown (frontmatter + body + backmatter) + assets are the **exportable SoR** for content. | +| R2 | App DB holds **projections and app linkage only** for content; disposable for bodies. | | R3 | Authn/IdP = **NetKingdom**; coulomb maps OIDC → Member; no password store. | -| R4 | Authz for product actions may compose NetKingdom groups, app membership, and content-plane ACLs — **no second user directory**. | +| R4 | Authz composes NetKingdom groups, app membership, content-plane ACLs — **no second user directory**. | | R5 | **markitect-main** is legacy reference, not a runtime foundation. | -| R6 | **markitect-tool** is the markdown structure dependency (parse/validate/transform). | -| R7 | Content-plane choice (shard-wiki / kontextual / thin store) is behind a **narrow page-ops interface** so coulomb UI does not couple to federation or ECM graphs. | -| R8 | Stage-1 capability set remains: onboarding, spaces/pages with Title/Abstract/Visual, page copy, page transfer. | +| R6 | **markitect-tool** is the markdown structure dependency (parse, validate, query, include, later derived slots). | +| R7 | Content plane (shard-wiki / kontextual / thin store) sits behind a **narrow PageOps** interface. | +| R8 | Stage-1 must: onboarding; spaces/pages with **Title / Abstractor / Visual**; page copy; page transfer. | +| R9 | **Pages first** — do not model documents as a priori chunk assemblies; use address/transclude/promote. | +| R10 | Derived content (projections, pinned includes, pipelines, AI) uses **stale badge → alternative → replace\|keep** and **lock**; no silent overwrite of locked/user content. | ## 3. System shape @@ -42,58 +68,94 @@ Deliver a co-creation product where: ▼ ┌─────────────────────────────────────────────────────────────┐ │ coulomb.social (product) │ -│ • Card UI, spaces chrome, stage-1 product flows │ -│ • Session / Member / SpaceMembership projections │ -│ • PageOps façade: list/get/create/update/archive/copy/move │ -│ • Railiance deploy (Helm, env, probes) │ +│ • Card UI, spaces chrome, dual authoring flows │ +│ • Session / Member / SpaceMembership │ +│ • PageOps + projection UX (lazy / stale / lock) │ +│ • Railiance deploy │ └───────────────┬─────────────────────────────┬───────────────┘ - │ │ │ projections │ content ops ▼ ▼ ┌───────────────┐ ┌───────────────────────────┐ - │ App DB cache │ │ Content plane (choose one │ - │ (Postgres) │ │ primary adapter) │ - └───────────────┘ │ A thin folder/git store │ - │ B shard-wiki │ + │ App DB cache │ │ Content plane (adapter) │ + │ (Postgres) │ │ A thin folder/git │ + └───────────────┘ │ B shard-wiki (thin) │ │ C kontextual-engine │ └─────────────┬─────────────┘ │ structure ▼ ┌───────────────────────────┐ │ markitect-tool │ - │ front/body/back, schema │ - └───────────────────────────┘ - │ - ┌─────────────▼─────────────┐ - │ Asset bytes (see §6) │ - │ content-addressed blobs │ + │ parse, FM, query, include │ + │ derived/stale (proposed) │ + └─────────────┬─────────────┘ + ▼ + ┌───────────────────────────┐ + │ Asset bytes (blobs) │ + │ content-addressed │ └───────────────────────────┘ ``` -**Product verbs (PageOps)** — store-agnostic: +### PageOps (store-agnostic) | Op | Meaning | |----|---------| | `list_pages(space)` | Card projections | -| `get_page(id)` | Full markdown + resolved visual | +| `get_page(id)` | Full markdown + resolved visual; optional include resolve | | `put_page(...)` | Create/update attributes + body | | `archive_page` | Soft-delete / lifecycle | -| `copy_page` | Independent duplicate | +| `copy_page` | Independent duplicate (rewrite includes as needed) | | `transfer_page` | Re-home to another space | | `put_visual` / `get_asset` | Binary visual/attachment | +| `project_field` / `regenerate` / `lock` | Lazy & stale projection actions | +| `derived_refresh` | Stale include/pipeline: alternative → replace\|keep (when markitect supports) | -Coulomb never talks to Forgejo/shard/kontextual APIs ad hoc from templates — only through PageOps. +Coulomb templates/services do **not** call Forgejo/shard/kontextual APIs ad hoc — only PageOps. ## 4. Information model (product) | Concept | Representation | |---------|----------------| -| **Page** | Markdown document: frontmatter (title, abstract, visual ref, space, type, …) + body + backmatter (relations, provenance) | -| **Space** | Container context; may be a page of `type: space` or a first-class container id in frontmatter | -| **Visual** | Asset ref in frontmatter → blob/asset service | -| **Card** | UI projection: visual + title + abstract + actions (not a storage type) | -| **Member** | NetKingdom subject ↔ app Member row | -| **Tenant** | Instance / `tenant_id` (DR-1 C) | +| **Page** | Markdown: frontmatter (title, abstractor, visual, space, projections lifecycle, …) + body + backmatter | +| **Space** | Container; may be `type: space` page or container id in frontmatter | +| **Visual** | Asset ref → blob store | +| **Card** | UI only: visual + title + abstractor + actions (+ stale badges) | +| **Region / “chunk”** | Addressable unit of a page (heading, `mkt:region`, selector) — **not** a required storage type | +| **Transclusion** | Live `mkt:include` and/or pinned `mkt:derived` snapshot (proposed) | +| **Member / tenant** | NetKingdom + app session | + +### 4.1 Page-first vs Bubble chunks + +Live Bubble has `custom.chunk` graphs. Rebuild stance: + +```text +Write fluently in a page + → address regions when needed + → transclude / extract / promote + → materialize a new page only when reuse earns independence +``` + +See `transclusion-and-chunks.md`. Migration may flatten Bubble chunks into sections/regions/pages+includes. + +### 4.2 Dual authoring + +| Mode | Flow | +|------|------| +| **Top-down** | Title → abstractor → outline (headings) → elaborate body → visual | +| **Bottom-up** | Body first; title/abstractor/visual/structure later | + +No mandatory mode toggle. Empty body with metadata is valid; body-only is valid. + +### 4.3 Lazy projections + lock + stale alternatives + +For **title**, **abstractor**, **visual** (and similar display fields): + +1. Missing on display need → **generate** (lazy) → **persist** as normal field values. +2. **Regenerate** and **lock** are per-field. +3. If inputs changed → **May be stale** → **Generate updated version** → compare Current \| Alternative → **Keep** or **Use alternative**. + +Detail: `authoring-modes-and-lazy-projections.md`, `stale-derived-content.md`. + +Same grammar for **pinned includes** and **pipeline/AI/rule** blocks (markitect proposal: `mkt:derived`). ## 5. Content-plane options (comparison) @@ -101,182 +163,145 @@ Coulomb never talks to Forgejo/shard/kontextual APIs ad hoc from templates — o | | | |--|--| -| **Idea** | Layout of `.md` files + assets; optional git; markitect validates; app indexes cards | -| **Pros** | Fastest stage-1; trivial user export; low structure surface | -| **Cons** | Reimplements copy/transfer/versioning carefully; federation later is a rewrite | -| **When** | Need product velocity; content plane maturity unclear | +| **Idea** | `.md` tree + assets; markitect validates; app indexes cards | +| **Pros** | Fastest S1; trivial export; low structure tax | +| **Cons** | Copy/transfer/versioning are app work; federation later costs more | +| **When** | Velocity first; plane maturity unclear | ### 5.2 Option B — shard-wiki (thin slice) | | | |--|--| -| **Idea** | Headless wiki orchestration; pages as union of shards; IdP external; coulomb = UI | -| **Pros** | Built for wiki pages, git coordination, overlays, multi-shard (personal + space); INTENT names Coulomb; no UI ownership | -| **Cons** | Early product maturity for multi-tenant ops; federation API can over-structure if not sliced thin | -| **When** | Want page federation / export shards / git-addressable coordination as first-class | -| **Thin-slice rule** | One InformationSpace per instance; one primary shard; hide federation from card UI | +| **Idea** | Headless wiki orchestration; IdP external; coulomb = UI | +| **Pros** | Page union, git coordination, multi-shard later; INTENT names Coulomb | +| **Cons** | Structure tax if federation leaks into product; multi-tenant ops still maturing | +| **Thin-slice rule** | One InformationSpace / instance; one primary shard; hide federation from cards | -### 5.3 Option C — kontextual-engine (knowledge ops / CMS-ECM backend) +### 5.3 Option C — kontextual-engine | | | |--|--| -| **Idea** | Headless knowledge operations: durable **KnowledgeAsset** identity, representations, metadata, relationships, workflows, audit, blob storage, optional CMIS; markdown via markitect-tool adapter | -| **Pros** | Strong identity for assets beyond path; lineage, versioning, export packages, multi-format future; **blob layer already implemented** (memory/fs/S3, content-addressed); PolicyGateway port ready for external authz; aligns with “ECM without being the UI” | -| **Cons** | Heavier than pure wiki; **not streamlined for NetKingdom** today (actors are engine-local; default `AllowAllPolicyGateway`); markdown is *a* format among many — SoR discipline must force “source representation is markdown file users can take” | -| **When** | Need governed asset registry, visuals/blobs, audit, multi-format, CMIS-ish interoperability; willing to invest PolicyGateway → flex-auth / NetKingdom | -| **Auth gap** | `Actor` + `external_ref` can map to OIDC subject; `PolicyGateway.authorize` must call **flex-auth** (or equivalent) with NetKingdom groups — not reimplement IAM inside kontextual | +| **Idea** | Knowledge ops: asset identity, representations, blobs, audit, workflows; markdown via markitect adapter | +| **Pros** | Blobs ready; lineage; multi-format future; PolicyGateway port | +| **Cons** | Heavier; NetKingdom not wired; must enforce markdown export as happy path | +| **Auth gap** | Map OIDC → Actor; PolicyGateway → flex-auth | -**kontextual already has (relevant):** - -- Asset registry (create, metadata, lifecycle, versions, relationships, audit) -- BlobStorage port + local/S3/memory adapters, dedupe by sha256 -- Representation kinds (source / normalized / derived) -- markitect-tool reuse boundary docs + adapter package -- Service API surface for assets/retrieval/workflows -- Policy port (pluggable); default allow-all for tests - -**kontextual does not yet have (for coulomb):** - -- First-class “space + card product model” -- Wired NetKingdom / KeyCape session → Actor -- Production flex-auth policy adapter -- Guaranteed “markdown file tree export is the happy path” product packaging (export exists as concept; productize for coulomb) - -### 5.4 How options relate (not exclusive forever) +### 5.4 Evolution (not exclusive forever) ```text -coulomb PageOps - │ - ├─ short term: A or thin B/C adapter - │ - └─ adapters can evolve: - folder/git shard ──► shard-wiki InformationSpace - markdown assets ──► kontextual KnowledgeAsset + BlobStorage - markitect-tool ──► always structure layer +PageOps → A or thin B/C now + → folder/git shard can become shard-wiki + → assets can move to kontextual / asset-store + → markitect-tool always structure layer ``` -shard-wiki optimizes **wiki federation and page union**. -kontextual optimizes **governed multi-format knowledge assets + blobs + audit**. -coulomb optimizes **co-creation product**. +**Working shortlist (unchanged):** spike **A** as control + **B thin** and **C** in parallel; pick after scorecard (`spike-checklist.md`). -A hybrid is possible later: shard-wiki as page navigation plane, kontextual as asset/blob + governance — only if spikes show value without double-structure tax. +## 6. Asset management -## 6. Asset management — extract to its own repo? +| Source | Role | +|--------|------| +| kontextual BlobStorage | Prefer for Visual/bytes now (fs/S3/memory, content-addressed) | +| markitect-main assets | Ideas only — do not runtime-depend | +| Extract `asset-store` | When **second** consumer needs same API; seed from kontextual port | -### 6.1 What exists today - -| Source | Asset story | -|--------|-------------| -| **kontextual-engine** | Clean **BlobStorage** protocol + content-addressed store + representation linkage; small, intentional, portable adapters (~memory/fs/S3) | -| **markitect-main** `markitect/assets/` | Large (~0.5M+ tree) manager/registry/packager/optimizer/analytics — **orphan** in succession map; do not run coulomb on it | -| **coulomb** | No first-class visual/blob store yet (stage-1 Visual is todo) | - -### 6.2 Extraction thesis - -**Yes, worth evaluating a dedicated `asset-store` (name TBD) capability** if: - -1. **coulomb**, **kontextual**, **shard-wiki**, and future apps all need content-addressed bytes + metadata without pulling full ECM or full wiki. -2. The core contract is small: `put/get/stat/delete_unreferenced` + digest + media type + optional ACLs. -3. We want Railiance-wide reuse (space visuals, wiki attachments, kontextual representations). - -**Prefer extracting from kontextual’s BlobStorage** (clean ports), **not** from markitect-main’s asset mega-package (mine ideas only: dedupe, mdpkg packaging). - -### 6.3 Suggested extraction shape (discussion) - -```text -asset-store (new repo or helix capability) - ports: BlobStorage (digest-addressed) - adapters: fs, s3, memory - optional: reference accounting, GC, HTTP byte range - NOT: full asset registry, workflows, CMIS, markdown parsing - -kontextual-engine - keeps KnowledgeAsset + Representation pointing at storage_ref - depends on asset-store library/service - -coulomb / shard-wiki - may depend on asset-store for Visual without full kontextual -``` - -**Decision gate for extraction:** after coulomb stage-1 Visual needs a real store **and** a second consumer wants the same API. Until then, use kontextual blob adapters **in-process** or vendor the port as a small shared package without full service. +Visual field stores a **ref** (path or `sha256:…`); bytes never live only in Postgres. ## 7. Identity & access control ```text -Browser → coulomb (session after NetKingdom OIDC) - │ - ├─ product authz: SpaceMembership, tenant_id - │ - └─ content plane actor: - Actor(external_ref=issuer|subject, groups=OIDC groups) - PolicyGateway → flex-auth (preferred) or engine rules +Browser → NetKingdom OIDC → coulomb session + ├─ product authz: tenant_id, SpaceMembership + └─ content plane: Actor(external_ref=iss|sub, groups=…) + PolicyGateway → flex-auth (preferred) ``` -| Plane | Responsibility | -|-------|----------------| -| NetKingdom | Who is the user; groups/roles claims | -| coulomb | Session; Member; space membership product rules | -| flex-auth | Optional PDP for resource actions | -| Content plane | Enforce policy on get/put content; fail closed | - -**Spike required:** map one OIDC login → kontextual `Actor` → deny-by-default policy for foreign tenant assets. +Fail closed on ambiguous access. Spike: OIDC → Actor → cross-tenant deny. ## 8. Deploy sketch (Railiance) | Component | Deploy | |-----------|--------| -| coulomb-social | Existing chart / app.coulomb.social | -| Content plane | Sidecar or separate service in `coulomb-social` ns or shared platform ns | -| asset bytes | PVC and/or S3-compatible; secrets via existing env-secret pattern | +| coulomb-social | Existing chart; `app.coulomb.social` | +| Content plane | In-process, sidecar, or platform service | +| Asset bytes | PVC and/or S3; secrets via env-secret pattern | | NetKingdom | Existing KeyCape / Authelia | -No apex DNS change until stage-1 cut. +Apex DNS stays Bubble until stage-1 cut + transfer readiness. -## 9. ADR roadmap (proposed) +## 9. Stage-1 product scope (capability cut) + +**Must for productive transfer** (Bubble → reference-only): + +- Member onboarding (NetKingdom; public reg may lag via provisioned users) +- Spaces + pages with **Title, Abstractor, Visual** + markdown body +- Page **copy** and **transfer** +- Card presentation +- Dual authoring + lazy projections (at least title/abstractor); stale replace|keep preferred early + +**Later:** social, ventures/invest, challenges marketplace, monetization, full chunk-stage machine, multi-shard personal vaults. + +Live Bubble map: `docs/capability/existing-bubble-capability-map.md`. + +## 10. ADR roadmap | ADR | Topic | |-----|--------| -| ADR-0001 | NetKingdom identity (done) | -| ADR-0002 | Forgejo markdown binding (provisional slice → demote to adapter) | -| **ADR-0003** | Page-centric model + markdown SoR + DB cache (to write after spikes) | -| **ADR-0004** | Content-plane choice (A/B/C) + PageOps boundary | +| ADR-0001 | NetKingdom identity (**done**) | +| ADR-0002 | Forgejo binding — **demote** to one adapter/export shape | +| **ADR-0003** | Page-centric model + SoR + DB cache + page-first/transclusion | +| **ADR-0004** | Content-plane choice + PageOps | | **ADR-0005** | Asset-store extraction (if/when) | +| **ADR-0006** | Authoring modes + projections + stale/derived (or fold into 0003) | -## 10. Spike checklist +Markitect-tool: implement `proposed-stale-derived-and-include-pinning.md` as its own work. -See **[spike-checklist.md](./spike-checklist.md)** — executable evaluation tasks. +## 11. Spike checklist -## 11. Discussion prompts +See **[spike-checklist.md](./spike-checklist.md)**. +Add when scheduling: dual-authoring smoke, stale projection compare UX, include pin (after markitect). -1. Is **user export as markdown tree** more important than **multi-format ECM** in the next 6 months? (Biases A/B vs C.) -2. Do we need **multi-shard federation** (personal vault + space) at stage 1, or only later? (Biases B.) -3. Should **Visual** block on asset-store extraction, or ship with kontextual/local blob first? -4. Who owns **PolicyGateway** production adapter — coulomb, kontextual, or flex-auth team? -5. Accept **hybrid**: kontextual for assets+registry, thin git for human-facing markdown tree sync? +## 12. Discussion prompts (remaining) -## 12. Working recommendation (to debate, not freeze) +1. Content plane for next 3–6 months: **A velocity**, **B federation path**, or **C governance/blobs**? +2. Visual: kontextual/local blob first, or block on asset-store extract? +3. PolicyGateway production owner? +4. Bubble **chunk** migration: sections/regions only, or keep a chunk list UX as a *view*? +5. Default include mode in product: **live** vs **pinned** for shared excerpts? + +## 13. Working recommendation (principles frozen; plane open) | Layer | Recommendation | |-------|----------------| -| Product UI | coulomb cards + spaces | -| Markdown structure | markitect-tool | +| Product UI | coulomb cards + spaces + dual authoring | +| Markdown structure | markitect-tool (includes + future derived slots) | | Identity / deploy | NetKingdom / Railiance | -| Content plane shortlist | Spike **B thin** and **C** in parallel; keep **A** as fallback | -| SoR narrative | User-facing SoR = markdown export; engine may add durable IDs/lineage **without** trapping content in opaque DB-only forms | -| Assets | Prefer kontextual BlobStorage now; extract **asset-store** when second consumer appears | +| Content plane | Spike A+B+C; choose with export fidelity as hard constraint | +| SoR | Markdown export always works; engines add IDs/lineage without trapping content | +| Assets | kontextual blobs now; extract later if shared | +| Chunks | Transclusion/extract, not a priori graph | +| Derived/stale | Badge → alternative → replace\|keep + lock | | markitect-main | Avoid | --- ## References +### coulomb-social + - `docs/architecture/2026-08-12-content-substrate-exploration.md` -- `docs/architecture/transclusion-and-chunks.md` — page-first model; chunks as extract/transclude -- `docs/architecture/authoring-modes-and-lazy-projections.md` — top-down/bottom-up; lazy Title/Abstractor/Visual + lock -- `docs/architecture/stale-derived-content.md` — stale badge; alternative; replace|keep; includes/pipelines +- `docs/architecture/transclusion-and-chunks.md` +- `docs/architecture/authoring-modes-and-lazy-projections.md` +- `docs/architecture/stale-derived-content.md` +- `docs/architecture/spike-checklist.md` - `docs/capability/stage-1-cutover.md` +- `docs/capability/model-v0.yaml` +- `docs/capability/existing-bubble-capability-map.md` - `docs/decisions/2026-08-12-feature-cut-stage-1.md` -- kontextual: `docs/architecture-blueprint.md`, `asset-registry-implementation.md`, `blob-storage-content-streaming-workplan.md`, `markitect-tool-reuse-boundary.md` -- shard-wiki: `INTENT.md`, `SCOPE.md` -- markitect-tool: `docs/markitect-main-scope-assessment.md`, `docs/transform-compose-include.md`, `docs/content-references.md` -- markitect-main succession: `docs/successor-gap-assessment.md` + +### fleet + +- kontextual: architecture-blueprint, asset-registry, blob-storage, markitect reuse boundary +- shard-wiki: INTENT, SCOPE +- markitect-tool: transform-compose-include, content-references, cache-incremental, **proposed-stale-derived-and-include-pinning.md** +- markitect-main: successor-gap-assessment (legacy)