coulomb-social/docs/migration/corpus-import.md

Local corpus import and recovery prerequisites

import_content_corpus imports prepared ADR-0004 trees. It does not contact Bubble, provision accounts or verify the completeness of a Bubble export. Use private local trees and manifests outside the checkout. Never commit member mappings, corpus content, DB dumps or backup archives.

All-space order and original dates — 2026-10-27

The user specifies eventual transfer of all coulomb.social spaces, using deterministic tooling and newest space creation first (LIFO guidance). Preserve creation dates of spaces, pages and every nested chunk. Source modification date, export time and filesystem mtime must not determine this order.

Plan an existing private capture without contacting Bubble or an inference service:

uv run python scripts/plan-bubble-transfer.py /private/capture --output /private/new-plan.json

Input is the exporter's source-ID-keyed spaces.json and articles.json. The new output is mode 0600 and refuses overwrite; its parent directory should be private. It contains private IDs and normalized/raw dates, no article bodies or identities. Input hashes bind it to the exact capture. Every space appears, sorted by creation descending with a stable ID tie-breaker. Missing dates sort last and are flagged. Recursively follow chunks_list_custom_article, including children without their own space reference; cycles do not duplicate records. Report missing children/parents, foreign-space chunks, missing bodies/dates and unassigned captured articles. Never fabricate dates from Bubble IDs or import time.

captured_graph_complete means only that those checks pass for captured records. It does not prove pagination/source completeness, asset/link fidelity, original access rules, approved grants or productive acceptance. Fresh source capture, all-tree conversion and per-space asset/link reconciliation remain required. The August retained inventory has 71 spaces/109 articles, with 11 incomplete captured graphs: 16 missing chunk references, 29 missing parent references and one space missing both creation/modification dates. All remain in the plan.

Prepared Markdown uses source_created_at for each space index/page/chunk. The importer accepts timezone-aware ISO values or Bubble epoch milliseconds, rejects invalid present dates before writes, copies page/chunk metadata unchanged, and restores the space's SQL created_at (including repeat import). Normal edits retain page/chunk source metadata. Bubble-marked trees require original creation dates for the space index and every page/chunk before any writes; the importer refuses to synthesize an index with an unknown original space date. Generic hand-authored, non-Bubble trees can still omit source metadata. The capture exporter now includes original space dates in its inventory/index, alongside the existing original article dates.

When refreshing an older inventory, add --inventory-spaces /private/old-capture/spaces.json to the planning command. It keeps every known space queued, including absent records, explicitly marked source_record_fresh: false and space_not_refreshed. Retained timestamps may guide ordering; they never certify current source completeness or transfer readiness.

After renewing the private Bubble session with scripts/capture-auth-state.sh, the read-only capture can discover every visible space root instead of using the historical fixed space-name list:

COULOMB_SOCIAL_ALL_SPACES=1 COULOMB_SOCIAL_CORPUS_DIR=/private/new-capture \
  node scripts/export-bubble-corpus.mjs

This mode never seeds old responses, ranks discovered roots by their captured space creation date, retains unknown dates last and records each visited/missing/ failed root in the private index. It still uses bounded UI scrolling and does not certify pagination/source completeness. The source JSON capture covers all visited roots; the legacy exporter still produces only one sample tree. Run the planner on the JSON and reconcile missing references before all-tree conversion; do not interpret that one sample tree as migration of the entire capture.

Optionally set COULOMB_SOCIAL_ORDER_SPACES_FILE=/private/old-capture/spaces.json to rank by retained original creation dates before every space's fresh metadata has been fetched. That file is used for ordering only, with its hash recorded; older articles/bodies are never seeded in all-space mode. Private source JSON and capture-state.json checkpoints preserve partial successful capture. Three consecutive navigation failures stop the pass with explicit private step diagnostics. Bubble's hidden root anchors identify the correct card; capture clicks its visible title rather than the hidden anchor. A visited card is not pagination proof.

Resume a partial capture through explicit missing references, without another full space walk or using older article bodies:

node scripts/capture-bubble-references.mjs --capture /private/checkpoint \
  --inventory-spaces /private/old-capture/spaces.json --output /private/new-reference-pass

This follows only source-referenced article/root addresses, including missing chunk/parent records, absent bodies/dates and unrefreshed space roots. Input hashes, attempts and unresolved references remain in private checkpoints. Output must be new and outside the repository. Response reads are bounded; a completed pass is not a completeness claim. Plan its resulting source files again. The October27 pass captured 70 fresh spaces/162 articles; all 71 known spaces remain represented, with 67 captured graphs passing and the remaining source/ownership gaps explicit.

Prepare all captured trees and a private reconciliation bundle offline:

uv run python scripts/prepare-bubble-bundle.py /private/checkpoint \
  --inventory-spaces /private/old-capture/spaces.json --output /private/new-review-bundle

The output directory must be new and outside the checkout. Its mode is 0700; files are 0600. reconciliation.json is written last: without that completed receipt the output is incomplete. Capture hashes and exact input snapshots bind the bundle to the source. Every known space remains queued in LIFO order. Only spaces whose captured graph checks pass receive review-trees/; blocked spaces retain their gaps. source-bodies/ preserves every captured text body exactly as UTF-8, including blocked/unassigned records and leading/trailing whitespace. Source-ID-derived slugs remain stable across label changes and subsequent captures. These are proposed destinations; reconcile the already accepted cohort and its local edits/copies/aliases before using any mapping.

Each draft index preserves the space creation date and links to a separate source root article. Each article/chunk preserves its own creation/modification dates and parent/chunk/contents references; ordered lists retain duplicates. The private report includes per-page body/file hashes and raw/UTC dates. The root article's birthdate never substitutes for the space's birthdate.

The bundle inventories ordinary Markdown inline/reference links and images, HTML link/resource attributes and source Visuals, without fetching URLs or rewriting bodies. Internal Bubble article links receive candidate destinations; ambiguous, absent, queried and fragment targets remain for reconciliation. External URLs, local resources, unsupported HTML/style/srcset, attachment flags and uncaptured contents_list_custom_article relations remain explicit review items. This is not an exhaustive parser for custom Bubble conventions, an asset download, or proof of rendering/relationship fidelity. Source capture can now follow missing contents relations as well as chunks/parents; their product semantics require review before conversion.

No import manifest or membership mapping is generated. Every space remains import_ready: false until source completeness, assets/links, rendering/workflows, existing destinations and explicit tenant/identity/grants are reconciled. Keep all drafts private. Do not feed the whole bundle to import_content_corpus or replace a live tree to backfill dates.

Prepare private source/product rendering cards without contacting Bubble:

uv run python scripts/audit-bubble-rendering.py /private/checkpoint/articles.json \
  --output /private/new-rendering-review

The tool invokes the actual product renderer under integration-disabled test settings. It records body/render/card hashes and structural sanitizer changes, retains missing-body records, and writes private index.html/cards plus a final audit.json completion receipt. Output is new, outside the checkout, mode 0700 with files mode 0600. Samples have a restrictive resource policy and inert body links/forms. The original Bubble view opens only through an explicit review link. An unchanged sanitizer structure is not an original-rendering comparison or productive fidelity acceptance. Use the UC-003–006 review walkthrough for verdicts.

Existing accepted content: date reconciliation without replacement

Use scripts/plan-source-date-backfill.py for an explicitly selected existing destination. It plans corrections; it has no execution/import mode:

uv run python scripts/plan-source-date-backfill.py /private/fresh-capture \
  --observation /private/current-date-observation.json \
  --selection /private/accepted-space-selection.json --output /private/new-date-plan.json

The private observation schema is coulomb.source-date-observation.v1: spaces contains observed SQL id, slug, tenant_id, created_at, and pages containing slug, full-file sha256 and source frontmatter meta. Capture it through an authorized read-only application context without emitting bodies/identities into logs. This observation is not a quiesced recovery point. The selection must pin exactly space_id, space_slug, tenant_id, source_space_id; neither a title nor source provenance on a copied page is authority to select a destination.

The planner normalizes old typed/prefixed Bubble IDs, binds all four inputs by hash and refuses mismatched SQL/tenant/slug/index/source identities. It compares article creation dates but never proposes article changes, including when a copy retains duplicate provenance. Local pages remain excluded. Existing index date conflicts/malformed values and source/article date gaps require review. For a missing index field it proposes inserting only source_created_at, preserving every existing metadata field and body byte. SQL changes target only the selected space's created_at, with its observed value as a precondition.

Output is a new mode-0600 JSON outside the checkout. It contains private IDs, hashes and proposed values; stdout contains counts only. Every plan remains ready_to_apply: false: apply requires a fresh paired rollback/recovery point, quiesced writers, live file-hash/SQL preconditions and verified postflight. Never use tree replacement/reimport to perform this backfill.

The October28 accepted-space observation found 36 imported article dates already matching fresh source, plus two preserved local pages. Its plan has only the SQL space date correction and missing index field insertion; no productive writes. Counts/hashes: docs/release/accepted-source-date-reconciliation-2026-10-28.json.

Packaged SQL migration rehearsal (internal disposable PostgreSQL, no production):

uv run python scripts/rehearse-storage-migration.py \
  --app-image forgejo.coulomb.social/coulomb/coulomb-social:c11300a --packaged-source

Packaged mode uses the image's application and Django; only test tools are copied into a read-only mount. Its test database/container/network are disposable.

Manifest

Version 1 JSON. Tree paths are relative to the manifest directory or absolute. Every space requires an existing active member mapped as owner. Identity is matched by tenant + issuer + subject, never email or a password.

{
  "version": 1,
  "spaces": [
    {
      "tree": "trees/example",
      "slug": "example",
      "tenant": "tenant:example",
      "memberships": [
        {"issuer": "https://issuer.example", "subject": "existing-subject", "role": "owner"}
      ]
    }
  ]
}

Roles: owner, member, viewer. Product authorization denies writes for viewer grants and permits owner/member writes; reads require an explicit membership or creator link. Tenant-wide bootstrap visibility has been removed. The manifest supplies owner and explicit grants, but does not certify that they match original Bubble privacy. Review the effect on existing spaces before executing a real cohort migration.

Run

First provision accounts through NetKingdom and have users sign in to create local Members. Reconcile the corpus and approved grants. Pause all application writers; run with the deployment tenant's private CONTENT_ROOT and DB settings.

uv run manage.py import_content_corpus /private/path/manifest.json --dry-run
uv run manage.py import_content_corpus /private/path/manifest.json
# Explicit replacement only after reviewing the existing content and backup:
uv run manage.py import_content_corpus /private/path/manifest.json --replace

Preflight validates all entries before any writes: local pages, normalized filenames, metadata bounds, source symlinks, source/destination overlap, destination collisions, active identity/tenant mapping and at least one owner. A missing owner or identity in the last tree prevents the first tree being written. Dry-run performs that same preflight and creates no directories or rows.

Execution stages each tree beside its destination, then applies metadata and memberships and publishes by rename inside a DB transaction. Ordinary errors restore the prior tree and roll back that space's DB changes. The whole batch is not atomic: previously committed spaces remain if a later space fails. Receipts give the completed count. Reconcile local state and either import a manifest containing only remaining entries, or deliberately use --replace. Replacement removes the previous content tree, including assets not in source; it adds/updates mapped memberships and does not revoke other existing grants or overwrite a pre-existing creator. Review those extra accesses explicitly.

Writers must remain paused. A process crash, rename/DB commit race or failed recovery filesystem operation requires manual recovery; this is not a crash-safe journal or concurrent import protocol. Keep content-root and instance-per-client tenant boundaries intact: filesystem paths remain slug-keyed.

The single-tree command shares validation and staging. --member-subject must resolve to exactly one tenant-local Member; use the corpus manifest when issuer ambiguity is possible. Single-tree imports may still omit membership, in which case the space is hidden unless it has an existing membership/creator grant. They are not a replacement for approved cohort mapping.

Paired backup/restore protocol (acceptance still pending)

1. Quiesce app and import writers, including Forgejo refresh consumers where relevant. Record recovery point, exact image SHA, migration level and tenant.
2. Through the DB owner's documented credential lane, capture metadata DB; separately snapshot/archive the entire CONTENT_ROOT (pages and assets, including archived trees). Keep private data in the approved backup store.
3. Record SHA-256 file inventory, row counts and backup identities privately. Do not log credentials, content bodies or identity lists into this repo/hub.
4. Restore DB and CONTENT_ROOT together into an isolated instance. Disable outgoing integrations and use restore-specific settings and credentials.
5. Compare Space/Member/membership counts, file hashes, page list/render, visual assets, copied/moved content and denied reads by an ungranted member. Measure achieved recovery time and recovery-point age; record the non-secret receipts.
6. Resume writers only after accepted recovery checks. If import fails, restore both stores from that same recovery point rather than only rolling back DB.

This is an operational protocol, not evidence that scheduled backup or a restore drill exists. CSOC-WP-0008-T02 owns automation and measured acceptance; T01 owns storage admission, T03 the actual private cohort rehearsal.