coulomb-social/docs/migration/corpus-import.md

# Local corpus import and recovery prerequisites

`import_content_corpus` imports prepared ADR-0004 trees. It does not contact
Bubble, provision accounts or verify the completeness of a Bubble export.
Use private local trees and manifests outside the checkout. Never commit member
mappings, corpus content, DB dumps or backup archives.

## All-space order and original dates — 2026-10-27

The user specifies eventual transfer of **all** coulomb.social spaces, using
deterministic tooling and newest space creation first (LIFO guidance). Preserve
creation dates of spaces, pages and every nested chunk. Source modification
date, export time and filesystem mtime must not determine this order.

Plan an existing private capture without contacting Bubble or an inference service:

```bash
uv run python scripts/plan-bubble-transfer.py /private/capture --output /private/new-plan.json
```

Input is the exporter's source-ID-keyed `spaces.json` and `articles.json`.
The new output is mode 0600 and refuses overwrite; its parent directory should
be private. It contains private IDs and normalized/raw dates, no article bodies
or identities. Input hashes bind it to the exact capture. Every space appears,
sorted by creation descending with a stable ID tie-breaker. Missing dates sort
last and are flagged. Recursively follow `chunks_list_custom_article`, including
children without their own space reference; cycles do not duplicate records.
Report missing children/parents, foreign-space chunks, missing bodies/dates and
unassigned captured articles. Never fabricate dates from Bubble IDs or import time.

`captured_graph_complete` means only that those checks pass for captured records.
It does not prove pagination/source completeness, asset/link fidelity, original
access rules, approved grants or productive acceptance. Fresh source capture,
all-tree conversion and per-space asset/link reconciliation remain required.
The August retained inventory has 71 spaces/109 articles, with 11 incomplete
captured graphs: 16 missing chunk references, 29 missing parent references and
one space missing both creation/modification dates. All remain in the plan.

Prepared Markdown uses `source_created_at` for each space index/page/chunk.
The importer accepts timezone-aware ISO values or Bubble epoch milliseconds,
rejects invalid present dates before writes, copies page/chunk metadata unchanged,
and restores the space's SQL `created_at` (including repeat import). Normal edits
retain page/chunk source metadata. Bubble-marked trees require original creation
dates for the space index and every page/chunk before any writes; the importer refuses to synthesize an index
with an unknown original space date. Generic hand-authored, non-Bubble trees
can still omit source metadata.
The capture exporter now includes original space dates in its inventory/index,
alongside the existing original article dates.

When refreshing an older inventory, add
`--inventory-spaces /private/old-capture/spaces.json` to the planning command.
It keeps every known space queued, including absent records, explicitly marked
`source_record_fresh: false` and `space_not_refreshed`. Retained timestamps may
guide ordering; they never certify current source completeness or transfer readiness.

After renewing the private Bubble session with `scripts/capture-auth-state.sh`,
the read-only capture can discover every visible space root instead of using
the historical fixed space-name list:

```bash
COULOMB_SOCIAL_ALL_SPACES=1 COULOMB_SOCIAL_CORPUS_DIR=/private/new-capture \
  node scripts/export-bubble-corpus.mjs
```

This mode never seeds old responses, ranks discovered roots by their captured
space creation date, retains unknown dates last and records each visited/missing/
failed root in the private index. It still uses bounded UI scrolling and does
not certify pagination/source completeness. The source JSON capture covers all
visited roots; the legacy exporter still produces only one sample tree. Run the
planner on the JSON and reconcile missing references before all-tree conversion;
do not interpret that one sample tree as migration of the entire capture.

Optionally set `COULOMB_SOCIAL_ORDER_SPACES_FILE=/private/old-capture/spaces.json`
to rank by retained original creation dates before every space's fresh metadata
has been fetched. That file is used for ordering only, with its hash recorded;
older articles/bodies are never seeded in all-space mode. Private source JSON
and `capture-state.json` checkpoints preserve partial successful capture. Three
consecutive navigation failures stop the pass with explicit private step diagnostics.
Bubble's hidden root anchors identify the correct card; capture clicks its visible
title rather than the hidden anchor. A visited card is not pagination proof.

Resume a partial capture through explicit missing references, without another full
space walk or using older article bodies:

```bash
node scripts/capture-bubble-references.mjs --capture /private/checkpoint \
  --inventory-spaces /private/old-capture/spaces.json --output /private/new-reference-pass
```

This follows only source-referenced article/root addresses, including missing
chunk/parent records, absent bodies/dates and unrefreshed space roots. Input hashes,
attempts and unresolved references remain in private checkpoints. Output must be
new and outside the repository. Response reads are bounded; a completed pass is
not a completeness claim. Plan its resulting source files again. The October27
pass captured 70 fresh spaces/162 articles; all 71 known spaces remain represented,
with 67 captured graphs passing and the remaining source/ownership gaps explicit.

Prepare **all** captured trees and a private reconciliation bundle offline:

```bash
uv run python scripts/prepare-bubble-bundle.py /private/checkpoint \
  --inventory-spaces /private/old-capture/spaces.json --output /private/new-review-bundle
```

The output directory must be new and outside the checkout. Its mode is 0700;
files are 0600. `reconciliation.json` is written last: without that completed
receipt the output is incomplete. Capture hashes and exact input snapshots bind
the bundle to the source. Every known space remains queued in LIFO order.
Only spaces whose captured graph checks pass receive `review-trees/`; blocked
spaces retain their gaps. `source-bodies/` preserves every captured text body
exactly as UTF-8, including blocked/unassigned records and leading/trailing
whitespace. Source-ID-derived slugs remain stable across label changes and
subsequent captures. These are proposed destinations; reconcile the already
accepted cohort and its local edits/copies/aliases before using any mapping.

Each draft index preserves the **space** creation date and links to a separate
source root article. Each article/chunk preserves its own creation/modification
dates and parent/chunk/contents references; ordered lists retain duplicates.
The private report includes per-page body/file hashes and raw/UTC dates. The
root article's birthdate never substitutes for the space's birthdate.

The bundle inventories ordinary Markdown inline/reference links and images,
HTML link/resource attributes and source Visuals, without fetching URLs or
rewriting bodies. Internal Bubble article links receive candidate destinations;
ambiguous, absent, queried and fragment targets remain for reconciliation.
External URLs, local resources, unsupported HTML/style/srcset, attachment flags
and uncaptured `contents_list_custom_article` relations remain explicit review
items. This is **not an exhaustive parser for custom Bubble conventions**, an
asset download, or proof of rendering/relationship fidelity. Source capture can
now follow missing contents relations as well as chunks/parents; their product
semantics require review before conversion.

No import manifest or membership mapping is generated. Every space remains
`import_ready: false` until source completeness, assets/links, rendering/workflows,
existing destinations and explicit tenant/identity/grants are reconciled. Keep
all drafts private. Do not feed the whole bundle to `import_content_corpus` or
replace a live tree to backfill dates.

Prepare private source/product rendering cards without contacting Bubble:

```bash
uv run python scripts/audit-bubble-rendering.py /private/checkpoint/articles.json \
  --output /private/new-rendering-review
```

The tool invokes the actual product renderer under integration-disabled test
settings. It records body/render/card hashes and structural sanitizer changes,
retains missing-body records, and writes private `index.html`/cards plus a final
`audit.json` completion receipt. Output is new, outside the checkout, mode 0700
with files mode 0600. Samples have a restrictive resource policy and inert body
links/forms. The original Bubble view opens only through an explicit review link.
An unchanged sanitizer structure is not an original-rendering comparison or
productive fidelity acceptance. Use the UC-003–006 review walkthrough for verdicts.

## Existing accepted content: date reconciliation without replacement

Use `scripts/plan-source-date-backfill.py` for an explicitly selected **existing**
destination. It plans corrections; it has no execution/import mode:

```bash
uv run python scripts/plan-source-date-backfill.py /private/fresh-capture \
  --observation /private/current-date-observation.json \
  --selection /private/accepted-space-selection.json --output /private/new-date-plan.json
```

The private observation schema is `coulomb.source-date-observation.v1`: `spaces`
contains observed SQL `id`, `slug`, `tenant_id`, `created_at`, and `pages` containing
`slug`, full-file `sha256` and source frontmatter `meta`. Capture it through an
authorized read-only application context without emitting bodies/identities into
logs. This observation is not a quiesced recovery point. The selection must pin
exactly `space_id`, `space_slug`, `tenant_id`, `source_space_id`; neither a title
nor source provenance on a copied page is authority to select a destination.

The planner normalizes old typed/prefixed Bubble IDs, binds all four inputs by
hash and refuses mismatched SQL/tenant/slug/index/source identities. It compares
article creation dates but never proposes article changes, including when a
copy retains duplicate provenance. Local pages remain excluded. Existing index
date conflicts/malformed values and source/article date gaps require review.
For a missing index field it proposes **inserting only** `source_created_at`,
preserving every existing metadata field and body byte. SQL changes target only
the selected space's `created_at`, with its observed value as a precondition.

Output is a new mode-0600 JSON outside the checkout. It contains private IDs,
hashes and proposed values; stdout contains counts only. Every plan remains
`ready_to_apply: false`: apply requires a fresh paired rollback/recovery point,
quiesced writers, live file-hash/SQL preconditions and verified postflight. Never
use tree replacement/reimport to perform this backfill.

The October28 accepted-space observation found 36 imported article dates already
matching fresh source, plus two preserved local pages. Its plan has only the SQL
space date correction and missing index field insertion; no productive writes.
Counts/hashes: `docs/release/accepted-source-date-reconciliation-2026-10-28.json`.

Packaged SQL migration rehearsal (internal disposable PostgreSQL, no production):

```bash
uv run python scripts/rehearse-storage-migration.py \
  --app-image forgejo.coulomb.social/coulomb/coulomb-social:c11300a --packaged-source
```

Packaged mode uses the image's application and Django; only test tools are copied
into a read-only mount. Its test database/container/network are disposable.

## Manifest

Version 1 JSON. Tree paths are relative to the manifest directory or absolute.
Every space requires an existing active member mapped as owner. Identity is
matched by **tenant + issuer + subject**, never email or a password.

```json
{
  "version": 1,
  "spaces": [
    {
      "tree": "trees/example",
      "slug": "example",
      "tenant": "tenant:example",
      "memberships": [
        {"issuer": "https://issuer.example", "subject": "existing-subject", "role": "owner"}
      ]
    }
  ]
}
```

Roles: `owner`, `member`, `viewer`. Product authorization denies writes for viewer grants and permits
owner/member writes; reads require an explicit membership or creator link.
Tenant-wide bootstrap visibility has been removed. The manifest supplies owner
and explicit grants, but does not certify that they match original Bubble
privacy. Review the effect on existing spaces before executing a real cohort migration.

## Run

First provision accounts through NetKingdom and have users sign in to create
local Members. Reconcile the corpus and approved grants. Pause all application
writers; run with the deployment tenant's private CONTENT_ROOT and DB settings.

```bash
uv run manage.py import_content_corpus /private/path/manifest.json --dry-run
uv run manage.py import_content_corpus /private/path/manifest.json
# Explicit replacement only after reviewing the existing content and backup:
uv run manage.py import_content_corpus /private/path/manifest.json --replace
```

Preflight validates **all** entries before any writes: local pages, normalized
filenames, metadata bounds, source symlinks, source/destination overlap,
destination collisions, active identity/tenant mapping and at least one owner.
A missing owner or identity in the last tree prevents the first tree being
written. Dry-run performs that same preflight and creates no directories or rows.

Execution stages each tree beside its destination, then applies metadata and
memberships and publishes by rename inside a DB transaction. Ordinary errors
restore the prior tree and roll back that space's DB changes. **The whole batch
is not atomic**: previously committed spaces remain if a later space fails.
Receipts give the completed count. Reconcile local state and either import a
manifest containing only remaining entries, or deliberately use `--replace`.
Replacement removes the previous content tree, including assets not in source;
it adds/updates mapped memberships and **does not revoke** other existing grants
or overwrite a pre-existing creator. Review those extra accesses explicitly.

Writers must remain paused. A process crash, rename/DB commit race or failed
recovery filesystem operation requires manual recovery; this is not a crash-safe
journal or concurrent import protocol. Keep content-root and instance-per-client
tenant boundaries intact: filesystem paths remain slug-keyed.

The single-tree command shares validation and staging. `--member-subject` must
resolve to exactly one tenant-local Member; use the corpus manifest when issuer
ambiguity is possible. Single-tree imports may still omit membership,
in which case the space is hidden unless it has an existing membership/creator
grant. They are not a replacement for approved cohort mapping.

## Paired backup/restore protocol (acceptance still pending)

1. Quiesce app and import writers, including Forgejo refresh consumers where
   relevant. Record recovery point, exact image SHA, migration level and tenant.
2. Through the DB owner's documented credential lane, capture metadata DB;
   separately snapshot/archive the **entire** CONTENT_ROOT (pages and assets,
   including archived trees). Keep private data in the approved backup store.
3. Record SHA-256 file inventory, row counts and backup identities privately.
   Do not log credentials, content bodies or identity lists into this repo/hub.
4. Restore DB and CONTENT_ROOT together into an isolated instance. Disable
   outgoing integrations and use restore-specific settings and credentials.
5. Compare Space/Member/membership counts, file hashes, page list/render, visual
   assets, copied/moved content and denied reads by an ungranted member. Measure
   achieved recovery time and recovery-point age; record the non-secret receipts.
6. Resume writers only after accepted recovery checks. If import fails, restore
   both stores from that same recovery point rather than only rolling back DB.

This is an operational protocol, **not** evidence that scheduled backup or a
restore drill exists. CSOC-WP-0008-T02 owns automation and measured acceptance;
T01 owns storage admission, T03 the actual private cohort rehearsal.