From b897aa1ee6fdc9bf411235bbb7c4f8a4c18b9418 Mon Sep 17 00:00:00 2001 From: tegwick Date: Tue, 6 Oct 2026 22:11:42 +0200 Subject: [PATCH] feat: prepare isolated recovery and guarded fresh ReichelAg capture Assistant: codex Assistant-Model: gpt-6.1-sol Assistant-Session: 01a10e2d-a5ef-7e92-b03d-b1694e915680 --- SCOPE.md | 8 + docs/migration/bubble-auth-and-export.md | 17 ++ docs/release/reichelag-transfer-2026-10-06.md | 83 +++++++++ ...6Z-reichelag-source-and-paired-recovery.md | 83 +++++++++ scripts/bubble-corpus-utils.mjs | 30 ++++ scripts/export-bubble-corpus.mjs | 91 ++++++++-- scripts/restore-paired-snapshot.py | 160 ++++++++++++++++++ scripts/tests/bubble-corpus-utils.test.mjs | 38 +++++ tests/test_bubble_capture_guards.py | 12 ++ tests/test_restore_recovery.py | 41 +++++ ...-WP-0008-production-transfer-acceptance.md | 25 ++- .../CSOC-WP-0011-ui-framework-architecture.md | 68 ++++++++ 12 files changed, 642 insertions(+), 14 deletions(-) create mode 100644 docs/release/reichelag-transfer-2026-10-06.md create mode 100644 history/20261006T200726Z-reichelag-source-and-paired-recovery.md create mode 100644 scripts/bubble-corpus-utils.mjs create mode 100644 scripts/restore-paired-snapshot.py create mode 100644 scripts/tests/bubble-corpus-utils.test.mjs create mode 100644 tests/test_bubble_capture_guards.py create mode 100644 tests/test_restore_recovery.py create mode 100644 workplans/CSOC-WP-0011-ui-framework-architecture.md diff --git a/SCOPE.md b/SCOPE.md index 0d2d774..539ba7c 100644 --- a/SCOPE.md +++ b/SCOPE.md @@ -112,3 +112,11 @@ and platform mailbox custody exist outside this repo; Authelia reset SMTP is now active from platform@coulomb.social; actual inbox receipt is confirmed. Personal-account login acceptance is also confirmed. See [platform mail assessment](docs/release/platform-email-2026-10-06.md). + +Paired recovery preparation (2026-10-06): guarded app-quiesced PostgreSQL/content +snapshot and isolated restore helpers are implemented. Live pre-import pair +restored13tables/102rows with empty content; a synthetic populated pair restored +13tables/111rows and4files with rendering and permission checks. Retained backup +custody/retention and real corpus/rollback acceptance stay CSOC-WP-0008-T02. +Fresh real ReichelAg transfer is waiting for renewed Bubble authentication; +expired/public-only source captures are rejected by the targeted exporter. diff --git a/docs/migration/bubble-auth-and-export.md b/docs/migration/bubble-auth-and-export.md index df4a931..b40deae 100644 --- a/docs/migration/bubble-auth-and-export.md +++ b/docs/migration/bubble-auth-and-export.md @@ -125,3 +125,20 @@ Redacted notes for the repo go under `docs/migration/` after review. - `scripts/capture-auth-state.sh` - `scripts/export-bubble-session.sh` - CSOC-WP-0001-T02 + +## Targeted fresh first-use-case capture + +For the nominated ReichelAg transfer, use explicit target mode after renewing +the session. This excludes August seed data, selects the space in the Pages +filter then applies Search, rejects expired signature cookies, and refuses +a one-page or body-incomplete target. A captured tree still needs reconciliation +against the latest expected protocol and private membership map before import. + +```bash +COULOMB_SOCIAL_TARGET_SPACE_LABEL=ReichelAG node scripts/export-bubble-corpus.mjs +``` + +Set COULOMB_SOCIAL_CORPUS_DIR to a private directory if needed. Do not put raw +exports in the checkout. Default target route is /vw_pages; override via +COULOMB_SOCIAL_CAPTURE_ROUTE only after confirming the actual authenticated +protocol surface. A successful public-route load is not authentication proof. diff --git a/docs/release/reichelag-transfer-2026-10-06.md b/docs/release/reichelag-transfer-2026-10-06.md new file mode 100644 index 0000000..78019af --- /dev/null +++ b/docs/release/reichelag-transfer-2026-10-06.md @@ -0,0 +1,83 @@ +# ReichelAg first transfer: recovery prepared, fresh source needs login + +## User authorization and source state + +Founder answered “yes go on” to current Bubble capture and first ReichelAg +transfer following the empty-workspace assessment. New tegwick app sign-in and +reset inbox receipt remain accepted. Authority for bounded app snapshot/quiesce: +ADMINISTER @ realm:kubernetes/railiance01, activation=APPROVED; continue existing +personal-app/brief-interruption approval and the current first-transfer go. + +Initial Bubble capture returned public records and looked usable, but no private +weekly protocols. Subsequent metadata inspection found expired Bubble signature +cookie. No fresh ReichelAg protocol capture or live import is claimed. User asked +to run scripts/capture-auth-state.sh privately and save a new authenticated +session after opening the real protocol list. No password/session/reset values +read into chat, Git or State Hub. Raw source dumps remain private temporary files. + +## Capture fixes + +Existing exporter guessed the largest captured space, mixed an old seed by +default, overwrote full records with partial responses and did not validate +expired signature cookies. Added explicit target mode using the Space filter +and Search action, disabling old seeds, retaining full fields across partial +fresh responses, normalizing typed Bubble lookup references, awaiting response +handlers and rejecting missing target protocols/body fields. Target mode defaults +to Pages following the user-confirmed Spaces -> ReichelAg -> Pages journey; +actual fresh authenticated protocol list still needs reconciliation. + +Four synthetic Node tests cover typed references, expired signature/analytics +distinction, partial-record preservation, and explicit target selection rejecting +larger unrelated spaces or incomplete bodies. A pytest wrapper includes them +in the repo gate. Current expired-session run exits2 before creating any output +directory, so public inventory cannot silently become a real import tree. + +## Paired snapshot and isolated restore evidence + +Node snapshot helper scripts/snapshot-content-and-db.py default server dry-run +passed against the pinned cluster UID and revision11 image. Execute scopes +pg_dump only to coulomb_social_db; app writers are paused, retained PVC mounted +read-only by a separate pod with no credential projection or service selector. +Pod removed and app1replica restored. Snapshot started19:34:18Z; total operation +53.73seconds; completed/app resumed. Public app health200 afterward. + +Database dump49843bytes SHA256 +26a1630db0a0603dff33a4f2cb128e5ca3bd914d953c335a0f941162dbddf2b3; +content archive186bytes SHA256 +7645781eca69efff473f67dcc736fde7e2a8a66a0172f59056ac57b651b19a25. +Snapshot files mode0600, enclosing directory0700, private node/workstation/tmp. +These are temporary rehearsal artifacts, not durable scheduled backup custody. + +Restore helper scripts/restore-paired-snapshot.py validates artifact checksums, +restores native PostgreSQL in a Docker internal network without published ports, +compares every COPY row/table and sequence value, checks archive paths and file +hashes. Exact app image retained; offline instance disables identity integrations. +Initial production-pair restore:13tables/102rows, all data/sequences match, empty +content confirmed, measured database/content restore4.26seconds. + +Populate that isolated instance with the synthetic established weekly-call +fixture: full copy, independently edited copy, an asset, owner/viewer/ungranted +actors. Snapshot both stores with no writers running; restore into a second +isolated PostgreSQL instance. Result:13tables/111rows,4content files, all logical +data/sequence/file hashes match, measured restore3.92seconds. Restored Django +checks: original and edited body matches; asset bytes and markdown render match; +owner can read; viewer write with validCSRF denied403; outsider read denied. +Both temporary database containers/internal networks removed afterward. + +Four snapshot safety tests exercise default dry-run/no writes, corrupt dump and +cleanup failures resuming the app, and private paired artifact creation. Restore +unit checks exercise changed rows/sequences, incomplete COPY and archive escape. +Full make check:96passed, including established Chromium workflows. Focused +Ruff and Node syntax/capture tests pass. + +## Remaining live work + +CSOC-WP-0008-T02 remains progress: establish retained backup custody/retention, +measured real-content recovery and production rollback procedure; successful +empty/synthetic recovery does not close actual corpus/visual/privacy acceptance. +T03 waits fresh authenticated Bubble capture. After renewal, capture the correct +current ReichelAg protocols, reconcile latest expected call and assets/privacy, +rehearse import with tenant:platform and only the existing tegwick owner mapping, +refresh paired snapshot before a reviewed live import, then validate visible +space, latest protocol and explicit denied access. T09 is the user's copy/edit/ +clipboard/mail-client acceptance; Bubble remains authoritative until accepted. diff --git a/history/20261006T200726Z-reichelag-source-and-paired-recovery.md b/history/20261006T200726Z-reichelag-source-and-paired-recovery.md new file mode 100644 index 0000000..78019af --- /dev/null +++ b/history/20261006T200726Z-reichelag-source-and-paired-recovery.md @@ -0,0 +1,83 @@ +# ReichelAg first transfer: recovery prepared, fresh source needs login + +## User authorization and source state + +Founder answered “yes go on” to current Bubble capture and first ReichelAg +transfer following the empty-workspace assessment. New tegwick app sign-in and +reset inbox receipt remain accepted. Authority for bounded app snapshot/quiesce: +ADMINISTER @ realm:kubernetes/railiance01, activation=APPROVED; continue existing +personal-app/brief-interruption approval and the current first-transfer go. + +Initial Bubble capture returned public records and looked usable, but no private +weekly protocols. Subsequent metadata inspection found expired Bubble signature +cookie. No fresh ReichelAg protocol capture or live import is claimed. User asked +to run scripts/capture-auth-state.sh privately and save a new authenticated +session after opening the real protocol list. No password/session/reset values +read into chat, Git or State Hub. Raw source dumps remain private temporary files. + +## Capture fixes + +Existing exporter guessed the largest captured space, mixed an old seed by +default, overwrote full records with partial responses and did not validate +expired signature cookies. Added explicit target mode using the Space filter +and Search action, disabling old seeds, retaining full fields across partial +fresh responses, normalizing typed Bubble lookup references, awaiting response +handlers and rejecting missing target protocols/body fields. Target mode defaults +to Pages following the user-confirmed Spaces -> ReichelAg -> Pages journey; +actual fresh authenticated protocol list still needs reconciliation. + +Four synthetic Node tests cover typed references, expired signature/analytics +distinction, partial-record preservation, and explicit target selection rejecting +larger unrelated spaces or incomplete bodies. A pytest wrapper includes them +in the repo gate. Current expired-session run exits2 before creating any output +directory, so public inventory cannot silently become a real import tree. + +## Paired snapshot and isolated restore evidence + +Node snapshot helper scripts/snapshot-content-and-db.py default server dry-run +passed against the pinned cluster UID and revision11 image. Execute scopes +pg_dump only to coulomb_social_db; app writers are paused, retained PVC mounted +read-only by a separate pod with no credential projection or service selector. +Pod removed and app1replica restored. Snapshot started19:34:18Z; total operation +53.73seconds; completed/app resumed. Public app health200 afterward. + +Database dump49843bytes SHA256 +26a1630db0a0603dff33a4f2cb128e5ca3bd914d953c335a0f941162dbddf2b3; +content archive186bytes SHA256 +7645781eca69efff473f67dcc736fde7e2a8a66a0172f59056ac57b651b19a25. +Snapshot files mode0600, enclosing directory0700, private node/workstation/tmp. +These are temporary rehearsal artifacts, not durable scheduled backup custody. + +Restore helper scripts/restore-paired-snapshot.py validates artifact checksums, +restores native PostgreSQL in a Docker internal network without published ports, +compares every COPY row/table and sequence value, checks archive paths and file +hashes. Exact app image retained; offline instance disables identity integrations. +Initial production-pair restore:13tables/102rows, all data/sequences match, empty +content confirmed, measured database/content restore4.26seconds. + +Populate that isolated instance with the synthetic established weekly-call +fixture: full copy, independently edited copy, an asset, owner/viewer/ungranted +actors. Snapshot both stores with no writers running; restore into a second +isolated PostgreSQL instance. Result:13tables/111rows,4content files, all logical +data/sequence/file hashes match, measured restore3.92seconds. Restored Django +checks: original and edited body matches; asset bytes and markdown render match; +owner can read; viewer write with validCSRF denied403; outsider read denied. +Both temporary database containers/internal networks removed afterward. + +Four snapshot safety tests exercise default dry-run/no writes, corrupt dump and +cleanup failures resuming the app, and private paired artifact creation. Restore +unit checks exercise changed rows/sequences, incomplete COPY and archive escape. +Full make check:96passed, including established Chromium workflows. Focused +Ruff and Node syntax/capture tests pass. + +## Remaining live work + +CSOC-WP-0008-T02 remains progress: establish retained backup custody/retention, +measured real-content recovery and production rollback procedure; successful +empty/synthetic recovery does not close actual corpus/visual/privacy acceptance. +T03 waits fresh authenticated Bubble capture. After renewal, capture the correct +current ReichelAg protocols, reconcile latest expected call and assets/privacy, +rehearse import with tenant:platform and only the existing tegwick owner mapping, +refresh paired snapshot before a reviewed live import, then validate visible +space, latest protocol and explicit denied access. T09 is the user's copy/edit/ +clipboard/mail-client acceptance; Bubble remains authoritative until accepted. diff --git a/scripts/bubble-corpus-utils.mjs b/scripts/bubble-corpus-utils.mjs new file mode 100644 index 0000000..db8ac1b --- /dev/null +++ b/scripts/bubble-corpus-utils.mjs @@ -0,0 +1,30 @@ +/** Pure capture guards; fixtures must never contain real cookies or notes. */ +export function normalizeBubbleRef(value) { + if (typeof value !== 'string') return value; + if (!value.startsWith('LOOKUP__')) return value; + return value.match(/[0-9]+x[0-9]+/)?.[0] || value; +} + +export function assertSessionFresh(state, now = Date.now() / 1000) { + if ((state.cookies || []).some(c => c.name.endsWith('.sig') && + c.expires > 0 && c.expires <= now)) { + throw new Error('bubble_session_expired'); + } +} + +export function mergeCapturedRecord(existing, source, id) { + return { ...(existing || {}), ...source, _id: normalizeBubbleRef(id) }; +} + +export function selectTargetArticles(articles, rootId) { + const root = articles[normalizeBubbleRef(rootId)]; + const spaceId = normalizeBubbleRef(root?.space_custom_space); + if (!root || !spaceId) throw new Error('target_space_identity_missing'); + const selected = Object.values(articles).filter(a => + normalizeBubbleRef(a.space_custom_space) === spaceId); + if (selected.length <= 1) throw new Error('target_protocols_not_captured'); + if (selected.some(a => !Object.hasOwn(a, 'content_text'))) { + throw new Error('target_protocol_bodies_incomplete'); + } + return [spaceId, selected]; +} diff --git a/scripts/export-bubble-corpus.mjs b/scripts/export-bubble-corpus.mjs index e55bd8d..1cdfaa4 100755 --- a/scripts/export-bubble-corpus.mjs +++ b/scripts/export-bubble-corpus.mjs @@ -7,6 +7,8 @@ * node scripts/export-bubble-corpus.mjs */ import { chromium } from "playwright"; +import { assertSessionFresh, normalizeBubbleRef, mergeCapturedRecord, + selectTargetArticles } from "./bubble-corpus-utils.mjs"; import { mkdirSync, writeFileSync, @@ -25,6 +27,8 @@ const STATE = process.env.COULOMB_SOCIAL_STORAGE_STATE || join(homedir(), ".config", "coulomb-social", "auth", "storage-state.json"); const BASE = process.env.COULOMB_SOCIAL_URL || "https://coulomb.social"; +const TARGET_LABEL = process.env.COULOMB_SOCIAL_TARGET_SPACE_LABEL || ""; +const TARGET_ROUTE = process.env.COULOMB_SOCIAL_CAPTURE_ROUTE || "/vw_pages"; const SEED_NET = process.env.COULOMB_SOCIAL_SEED_NETWORK || join( @@ -73,11 +77,8 @@ function slugify(s) { .slice(0, 80) || "untitled" ); } -function normId(id) { - if (!id || typeof id !== "string") return id; - const m = id.match(/LOOKUP__([0-9]+x[0-9]+)/); - return m ? m[1] : id; -} +function normId(id) { return normalizeBubbleRef(id); } + function ingest(obj, spaces, articles) { if (Array.isArray(obj)) { for (const x of obj) ingest(x, spaces, articles); @@ -88,10 +89,10 @@ function ingest(obj, spaces, articles) { const src = obj._source && typeof obj._source === "object" ? obj._source : null; const id = obj._id || (src && src._id); if (t === "custom.space" && src && id) { - spaces[normId(id)] = { ...src, _id: normId(id) }; + spaces[normId(id)] = mergeCapturedRecord(spaces[normId(id)], src, id); } if (t === "custom.article" && src && id) { - const a = { ...src, _id: normId(id) }; + const a = mergeCapturedRecord(articles[normId(id)], src, id); if (a.space_custom_space) a.space_custom_space = normId(a.space_custom_space); articles[normId(id)] = a; } @@ -162,9 +163,16 @@ if (!existsSync(STATE)) { process.exit(1); } +try { + assertSessionFresh(JSON.parse(readFileSync(STATE, "utf8"))); +} catch { + console.error("Bubble session unavailable or expired. Run ./scripts/capture-auth-state.sh."); + process.exit(2); +} + const spaces = {}; const articles = {}; -if (existsSync(SEED_NET)) { +if (!TARGET_LABEL && process.env.COULOMB_SOCIAL_FRESH_ONLY !== "1" && existsSync(SEED_NET)) { for (const name of readdirSync(SEED_NET)) { if (!name.endsWith(".body.txt")) continue; parseBody(readFileSync(join(SEED_NET, name), "utf8"), spaces, articles); @@ -176,7 +184,9 @@ ensure(OUT); const browser = await chromium.launch({ headless: true }); const context = await browser.newContext({ storageState: STATE }); const page = await context.newPage(); -page.on("response", async (response) => { +const pendingResponses = []; +page.on("response", (response) => { + pendingResponses.push((async () => { try { if (!response.url().includes("elasticsearch")) return; const text = await response.text(); @@ -184,6 +194,7 @@ page.on("response", async (response) => { } catch { /* ignore */ } + })()); }); async function loadSpacesList() { @@ -195,6 +206,46 @@ async function loadSpacesList() { } } +let targetRootId = ""; +if (TARGET_LABEL) { + try { + await loadSpacesList(); + const link = page.getByText(TARGET_LABEL, { exact: true }); + if (await link.count() !== 1) throw new Error("target_space_link_not_unique"); + const href = await link.getAttribute("href"); + if (!href || !new URL(href, BASE).pathname.startsWith("/pg_chunk/")) { + throw new Error("target_space_root_not_found"); + } + targetRootId = new URL(href, BASE).pathname.split("/").pop(); + await page.goto(`${BASE}${TARGET_ROUTE}`, { waitUntil: "domcontentloaded", timeout: 60000 }); + await page.waitForTimeout(4000); + const filter = page.locator("button").filter({ + has: page.locator("svg text").filter({ hasText: /^\s*filter_list\s*$/ }), + }); + await filter.first().click(); + const selects = page.locator("select"); + let selected = false; + for (let i = 0; i < await selects.count(); i++) { + const matches = await selects.nth(i).evaluate((el, label) => + [...el.options].some(o => o.textContent.trim() === label), TARGET_LABEL); + if (!matches) continue; + await selects.nth(i).selectOption({ label: TARGET_LABEL }); + selected = true; + break; + } + if (!selected) throw new Error("target_space_filter_not_found"); + await page.getByRole("button", { name: "Search", exact: true }).click(); + await page.waitForTimeout(5000); + for (let i = 0; i < 24; i++) { + await page.mouse.wheel(0, 1800); + await page.waitForTimeout(500); + } + } catch { + await browser.close(); + console.error("Target capture failed; no import tree published."); + process.exit(3); + } +} else { for (const name of SPACE_CLICKS) { try { await loadSpacesList(); @@ -213,9 +264,11 @@ for (const name of SPACE_CLICKS) { } console.log("ok", name, "articles", Object.keys(articles).length); } catch (e) { - console.log("fail", name, String(e.message || e).slice(0, 60)); + console.log("fail", name, "capture_step_failed"); } } +} +await Promise.allSettled(pendingResponses); await browser.close(); const bySpace = {}; @@ -239,11 +292,25 @@ function spaceLabel(sid) { } const ranked = Object.entries(bySpace).sort((a, b) => b[1].length - a[1].length); -const [sampleId, sampleArts] = ranked[0] || [null, []]; -const lab = spaceLabel(sampleId); +let sampleId, sampleArts; +if (TARGET_LABEL) { + try { + [sampleId, sampleArts] = selectTargetArticles(articles, targetRootId); + } catch { + console.error("Target protocols unavailable; no import tree published. Renew Bubble login."); + process.exit(4); + } +} else { + [sampleId, sampleArts] = ranked[0] || [null, []]; +} +const lab = TARGET_LABEL ? { ...spaceLabel(sampleId), slug: slugify(TARGET_LABEL), + title: TARGET_LABEL } : spaceLabel(sampleId); const index = { exported_at: new Date().toISOString(), + target_requested: TARGET_LABEL || null, + source_route: TARGET_LABEL ? TARGET_ROUTE : null, + old_seed_used: !TARGET_LABEL && process.env.COULOMB_SOCIAL_FRESH_ONLY !== "1" && existsSync(SEED_NET), space_count: Object.keys(spaces).length, article_count: Object.keys(articles).length, sample_tree: sampleId ? `trees/${lab.slug}` : null, diff --git a/scripts/restore-paired-snapshot.py b/scripts/restore-paired-snapshot.py new file mode 100644 index 0000000..644114e --- /dev/null +++ b/scripts/restore-paired-snapshot.py @@ -0,0 +1,160 @@ +#!/usr/bin/env python3 +"""Restore a private snapshot into isolated Docker PostgreSQL; never production. + +No published ports; an internal network blocks external integrations. Compare +all restored COPY rows and sequence state, and all archived file hashes. A +successful empty-content restore alone does not prove real corpus recovery. +""" +from __future__ import annotations + +import argparse +import hashlib +import json +import os +import re +import subprocess +import sys +import tarfile +import tempfile +import time +from pathlib import Path + + +class Refused(RuntimeError): + pass + + +def run(args: list[str], *, payload: bytes | None = None) -> bytes: + p = subprocess.run(args, input=payload, capture_output=True, timeout=180) + if p.returncode: + raise Refused('command_failed') + return p.stdout + + +def data_fingerprint(sql: bytes) -> dict: + """Ignore SQL boilerplate, compare COPY data and explicit sequence state.""" + tables = {} + sequences = [] + current = None + rows = [] + for line in sql.splitlines(): + if current is not None: + if line == b'\\.': + tables[current] = { + 'rows': len(rows), + 'sha256': hashlib.sha256(b'\n'.join(sorted(rows))).hexdigest(), + } + current, rows = None, [] + else: + rows.append(line) + elif line.startswith(b'COPY ') and line.endswith(b' FROM stdin;'): + current = line.decode('utf-8') + elif line.startswith(b'SELECT pg_catalog.setval('): + sequences.append(line.decode('utf-8')) + if current is not None: + raise Refused('unterminated_copy') + if not tables: + raise Refused('no_tables_in_dump') + return {'tables': tables, 'sequences': sorted(sequences)} + + +def content_fingerprint(archive: Path, destination: Path) -> dict[str, str]: + hashes = {} + with tarfile.open(archive, 'r:gz') as t: + for entry in t.getmembers(): + parts = Path(entry.name).parts + if (not parts or parts[0] != 'content' or '..' in parts + or Path(entry.name).is_absolute()): + raise Refused('invalid_archive_path') + if not (entry.isfile() or entry.isdir()): + raise Refused('unsupported_archive_entry') + if entry.isfile(): + hashes[entry.name] = hashlib.sha256(t.extractfile(entry).read()).hexdigest() + t.extractall(destination, filter='data') + actual = {str(p.relative_to(destination)): hashlib.sha256(p.read_bytes()).hexdigest() + for p in (destination / 'content').rglob('*') if p.is_file()} + if actual != hashes: + raise Refused('content_hash_mismatch') + return hashes + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('snapshot', type=Path) + parser.add_argument('--keep-instance', action='store_true', + help='Keep the isolated internal network/database for a content rehearsal') + args = parser.parse_args() + source = args.snapshot.resolve() + receipt = json.loads((source / 'receipt.json').read_text()) + if receipt.get('status') != 'captured' or not receipt.get('writers_quiesced'): + raise Refused('snapshot_not_paired') + for name, key in [('database.dump', 'database_sha256'), ('content.tar.gz', 'content_sha256')]: + if hashlib.sha256((source / name).read_bytes()).hexdigest() != receipt[key]: + raise Refused('artifact_checksum_mismatch') + started = time.monotonic() + name = 'csoc-recovery-' + time.strftime('%Y%m%d%H%M%S', time.gmtime()) + network = name + '-net' + work = Path(tempfile.mkdtemp(prefix=name + '-', dir='/tmp')) + image = run(['docker', 'image', 'inspect', 'postgres:16', + '--format', '{{.Id}}']).decode().strip() + if not re.fullmatch(r'sha256:[a-f0-9]{64}', image): + raise Refused('unresolved_postgres_image') + created = False + network_created = False + verified = False + try: + run(['docker', 'network', 'create', '--internal', network]) + network_created = True + run(['docker', 'run', '-d', '--name', name, '--network', network, + '--network-alias', 'recovery-db', '-e', 'POSTGRES_HOST_AUTH_METHOD=trust', + '-e', 'POSTGRES_DB=coulomb_social_restore', image]) + created = True + for _ in range(45): + p = subprocess.run(['docker', 'exec', name, 'pg_isready', '-U', 'postgres'], + capture_output=True) + if p.returncode == 0: + break + time.sleep(1) + else: + raise Refused('restore_db_not_ready') + archive = (source / 'database.dump').read_bytes() + original_sql = run(['docker', 'exec', '-i', name, 'pg_restore', '--data-only', '-f', '-'], + payload=archive) + baseline = data_fingerprint(original_sql) + run(['docker', 'exec', '-i', name, 'pg_restore', '-U', 'postgres', + '-d', 'coulomb_social_restore', '--no-owner', '--no-acl', '--exit-on-error'], + payload=archive) + restored_sql = run(['docker', 'exec', name, 'pg_dump', '-U', 'postgres', + '-d', 'coulomb_social_restore', '--data-only', + '--no-owner', '--no-acl']) + if data_fingerprint(restored_sql) != baseline: + raise Refused('database_fingerprint_mismatch') + files = content_fingerprint(source / 'content.tar.gz', work) + result = {'status': 'restored_verified', 'database_tables': len(baseline['tables']), + 'database_rows': sum(t['rows'] for t in baseline['tables'].values()), + 'database_rows_and_sequences_match': True, 'content_files': len(files), + 'content_hashes_match': True, 'network_internal': True, 'published_ports': 0, + 'postgres_image_id': image, 'elapsed_seconds': round(time.monotonic()-started, 2), + 'content_was_empty': not files, + 'database_container': name, 'network': network, 'restored_directory': str(work)} + path = work / 'restore-receipt.json' + fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600) + with os.fdopen(fd, 'w') as f: + json.dump(result, f, indent=2) + verified = True + print(json.dumps(result)) + finally: + if not (args.keep_instance and verified): + if created: + run(['docker', 'rm', '-f', '-v', name]) + if network_created: + run(['docker', 'network', 'rm', network]) + return 0 + + +if __name__ == '__main__': + try: + sys.exit(main()) + except Exception as exc: + print(json.dumps({'status': 'failed', 'failure_type': type(exc).__name__})) + sys.exit(1) diff --git a/scripts/tests/bubble-corpus-utils.test.mjs b/scripts/tests/bubble-corpus-utils.test.mjs new file mode 100644 index 0000000..d671cbf --- /dev/null +++ b/scripts/tests/bubble-corpus-utils.test.mjs @@ -0,0 +1,38 @@ +import test from 'node:test'; +import assert from 'node:assert/strict'; +import { normalizeBubbleRef, assertSessionFresh, mergeCapturedRecord, + selectTargetArticles } from '../bubble-corpus-utils.mjs'; + +test('typed lookup references normalize to the same space identifier', () => { + const id = '1700000000000x1234567890'; + assert.equal(normalizeBubbleRef(`LOOKUP__custom.space__${id}`), id); + assert.equal(normalizeBubbleRef(`LOOKUP__${id}`), id); + assert.equal(normalizeBubbleRef(id), id); +}); + +test('expired signature blocks capture; an expired analytics cookie does not', () => { + assert.throws(() => assertSessionFresh({cookies:[{name:'app.sig',expires:9}]},10), + /bubble_session_expired/); + assert.doesNotThrow(() => assertSessionFresh({cookies:[ + {name:'analytics',expires:9},{name:'app.sig',expires:11}]},10)); +}); + +test('partial network responses retain full notes captured in this run', () => { + const first = mergeCapturedRecord(null,{content_text:'synthetic full notes'},'root'); + assert.equal(mergeCapturedRecord(first,{nomer_text:'Synthetic call'},'root').content_text, + 'synthetic full notes'); +}); + +test('explicit target selection never substitutes the largest space', () => { + const records = {root:{space_custom_space:'target',content_text:'synthetic index'}, + call:{space_custom_space:'target',content_text:'synthetic protocol'}, + a:{space_custom_space:'other'},b:{space_custom_space:'other'},c:{space_custom_space:'other'}}; + const [id, selected] = selectTargetArticles(records,'root'); + assert.equal(id,'target'); + assert.equal(selected.length,2); + delete records.call.content_text; + assert.throws(() => selectTargetArticles(records,'root'), /target_protocol_bodies_incomplete/); + records.call.content_text = 'synthetic protocol'; + delete records.call; + assert.throws(() => selectTargetArticles(records,'root'), /target_protocols_not_captured/); +}); diff --git a/tests/test_bubble_capture_guards.py b/tests/test_bubble_capture_guards.py new file mode 100644 index 0000000..8f7a3d8 --- /dev/null +++ b/tests/test_bubble_capture_guards.py @@ -0,0 +1,12 @@ +"""Run synthetic source-capture guards in the same Node runtime as the exporter.""" +import subprocess +from pathlib import Path + + +def test_bubble_capture_guards(): + root = Path(__file__).resolve().parents[1] + result = subprocess.run( + ['node', '--test', 'scripts/tests/bubble-corpus-utils.test.mjs'], + cwd=root, capture_output=True, text=True, check=False, + ) + assert result.returncode == 0, result.stdout + result.stderr diff --git a/tests/test_restore_recovery.py b/tests/test_restore_recovery.py new file mode 100644 index 0000000..a3b0c8e --- /dev/null +++ b/tests/test_restore_recovery.py @@ -0,0 +1,41 @@ +"""Logical backup comparisons must catch changed rows, sequences and paths.""" +import importlib.util +import io +import tarfile +from pathlib import Path + +import pytest + + +@pytest.fixture +def helper(): + path = Path(__file__).resolve().parents[1] / 'scripts/restore-paired-snapshot.py' + spec = importlib.util.spec_from_file_location('csoc_restore', path) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +def test_data_comparison_ignores_order_but_detects_changes(helper): + a = b'COPY public.demo (id, value) FROM stdin;\n1\tone\n2\ttwo\n\\.\n' + b = b'-- SQL header\nCOPY public.demo (id, value) FROM stdin;\n2\ttwo\n1\tone\n\\.\n' + assert helper.data_fingerprint(a) == helper.data_fingerprint(b) + assert helper.data_fingerprint(a) != helper.data_fingerprint(b.replace(b'two', b'changed')) + assert helper.data_fingerprint(a) != helper.data_fingerprint( + a + b"SELECT pg_catalog.setval('public.demo_seq', 9, true);\n") + + +def test_partial_copy_is_rejected(helper): + with pytest.raises(helper.Refused): + helper.data_fingerprint(b'COPY public.demo (id) FROM stdin;\n1\n') + + +def test_content_archive_rejects_escape(helper, tmp_path): + archive = tmp_path / 'bad.tar.gz' + with tarfile.open(archive, 'w:gz') as t: + info = tarfile.TarInfo('content/../../outside') + info.size = 4 + t.addfile(info, io.BytesIO(b'data')) + with pytest.raises(helper.Refused): + helper.content_fingerprint(archive, tmp_path / 'restored') + assert not (tmp_path / 'outside').exists() diff --git a/workplans/CSOC-WP-0008-production-transfer-acceptance.md b/workplans/CSOC-WP-0008-production-transfer-acceptance.md index 121de03..270d341 100644 --- a/workplans/CSOC-WP-0008-production-transfer-acceptance.md +++ b/workplans/CSOC-WP-0008-production-transfer-acceptance.md @@ -102,7 +102,7 @@ snapshots; no private archive in git or State Hub. ```task id: CSOC-WP-0008-T03 -status: progress +status: wait priority: high state_hub_task_id: "0615eff2-bf2b-503b-8f15-0591d4666446" ``` @@ -395,7 +395,7 @@ prerequisite. Diagnostic receipt: history/20261006T190725Z-empty-personal-spaces ## Fresh ReichelAg transfer execution — 2026-10-06 Founder answered “yes go on” to current Bubble capture and first ReichelAg -transfer. Existing private browser session valid. Old exporter selects largest +transfer. Saved browser session probed; expired signature cookie prevents private protocol access. Old exporter selects largest captured space and assumes title click sets active space; neither proves target selection. Fresh source capture must explicitly select ReichelAG in Pages filter and apply Search, with old network seed disabled. Retain source-ID provenance, @@ -406,3 +406,24 @@ server dry-run passed. Quiesce app, scope pg_dump to coulomb_social_db, read-onl PVC archive, restore app even on failure; isolated PostgreSQL restore must pass before real import. Four focused safety tests pass (dry-run/no writes, corrupt dump/app-resume, cleanup failure/app-resume, private paired artifacts). + +Paired production snapshot completed; app resumed/healthy. Isolated PostgreSQL +restore matches13tables/102rows with empty content. A second synthetic populated +pair restores13tables/111rows and4files; full-protocol copy/independent edit, +asset/render fidelity, viewer validCSRF write denial and outsider read denial +pass after recovery.96tests pass. T02 remains progress for retained backup +custody/retention and real corpus/rollback review. + +T03 wait: Bubble signature cookie expired; public-only captures are not accepted +source. User must privately renew via scripts/capture-auth-state.sh and open +actual ReichelAg protocols. Targeted exporter now fails before output on expired +session, uses explicit target/Space filter/Search, excludes old seeds, merges +partial fresh records and rejects missing protocol bodies. Source receipt: +docs/release/reichelag-transfer-2026-10-06.md. No productive import performed. + +User confirmed the old journey: Spaces -> ReichelAg -> Pages, ordered by latest +changes. Capture defaults to /vw_pages. Prioritize current content migration; +UI framework architecture is a separate next phase (CSOC-WP-0011), before broad +UX rebuilding. Existing browser session renewal or a dedicated Bubble account +with read access can provide the source; never accept password/session contents +in chat. User renewal remains the next input needed for private protocol capture. diff --git a/workplans/CSOC-WP-0011-ui-framework-architecture.md b/workplans/CSOC-WP-0011-ui-framework-architecture.md new file mode 100644 index 0000000..2ec46db --- /dev/null +++ b/workplans/CSOC-WP-0011-ui-framework-architecture.md @@ -0,0 +1,68 @@ +--- +id: CSOC-WP-0011 +type: workplan +title: "Establish UI framework architecture before broad Bubble UX rebuild" +domain: communication +repo: coulomb-social +status: backlog +owner: codex +topic_slug: coulomb-social +created: "2026-10-06" +updated: "2026-10-06" +--- + +# User milestone and priority + +After the first content transfer, the interface has a consistent foundation for +finding spaces, opening pages ordered by latest changes, editing/copying protocols +and distributing notes by clipboard. New Bubble use cases can reuse the same +navigation, forms, interaction and permission patterns. The framework decision +is explicit and reviewed before broad UX investment; no new frontend stack is +assumed merely to make the current migration work. + +User steering on 2026-10-06: prioritize content migration, then establish framework +architecture before rebuilding many UI elements. CSOC-WP-0008-T02/T03/T09 remain +first; this backlog workplan does not block source capture or the first transfer. + +## Inventory existing UI and observed Bubble interaction patterns + +```task +id: CSOC-WP-0011-T01 +status: todo +priority: medium +``` + +Review current Django templates, routes, forms, permission checks and clipboard +JavaScript together with user-reviewed Bubble journeys. Describe navigation, +latest-change ordering, protocol copying and editing, reusable components and +accessibility needs. Use captured UI evidence separately from private content. + +## Decide the UI framework boundaries and extension contract + +```task +id: CSOC-WP-0011-T02 +status: todo +priority: medium +``` + +Prepare an ADR comparing the existing server-rendered foundation with progressive +interaction or a distinct frontend only where user needs justify it. Specify +component ownership, routing/state, server validation, auth/permission boundary, +asset delivery and regression test contract. Review with the user before broad +UX implementation. Avoid coupling the content migration format to the UI stack. + +## Demonstrate the foundation on the accepted ReichelAg journey + +```task +id: CSOC-WP-0011-T03 +status: todo +priority: medium +``` + +After content acceptance and framework review, implement one representative +space/page interaction on the decided foundation. Keep the full-protocol copy, +independent edit, latest-protocol discovery and native clipboard workflows under +established regression coverage. User reviews consistency and usability before +other Bubble UI surfaces are rebuilt. Register later UI gaps as live follow-ups. + +Founder content-first decision: e67cfd2e-e55d-4b42-947a-b279b6287c55.