feat: prepare isolated recovery and guarded fresh ReichelAg capture
Some checks failed
Use-case regression gate / usecase-regressions (push) Has been cancelled
Some checks failed
Use-case regression gate / usecase-regressions (push) Has been cancelled
Assistant: codex Assistant-Model: gpt-6.1-sol Assistant-Session: 01a10e2d-a5ef-7e92-b03d-b1694e915680
This commit is contained in:
parent
1e7b94df1a
commit
b897aa1ee6
12 changed files with 642 additions and 14 deletions
8
SCOPE.md
8
SCOPE.md
|
|
@ -112,3 +112,11 @@ and platform mailbox custody exist outside this repo; Authelia reset SMTP
|
|||
is now active from platform@coulomb.social; actual inbox receipt is confirmed.
|
||||
Personal-account login acceptance is also confirmed. See
|
||||
[platform mail assessment](docs/release/platform-email-2026-10-06.md).
|
||||
|
||||
Paired recovery preparation (2026-10-06): guarded app-quiesced PostgreSQL/content
|
||||
snapshot and isolated restore helpers are implemented. Live pre-import pair
|
||||
restored13tables/102rows with empty content; a synthetic populated pair restored
|
||||
13tables/111rows and4files with rendering and permission checks. Retained backup
|
||||
custody/retention and real corpus/rollback acceptance stay CSOC-WP-0008-T02.
|
||||
Fresh real ReichelAg transfer is waiting for renewed Bubble authentication;
|
||||
expired/public-only source captures are rejected by the targeted exporter.
|
||||
|
|
|
|||
|
|
@ -125,3 +125,20 @@ Redacted notes for the repo go under `docs/migration/` after review.
|
|||
- `scripts/capture-auth-state.sh`
|
||||
- `scripts/export-bubble-session.sh`
|
||||
- CSOC-WP-0001-T02
|
||||
|
||||
## Targeted fresh first-use-case capture
|
||||
|
||||
For the nominated ReichelAg transfer, use explicit target mode after renewing
|
||||
the session. This excludes August seed data, selects the space in the Pages
|
||||
filter then applies Search, rejects expired signature cookies, and refuses
|
||||
a one-page or body-incomplete target. A captured tree still needs reconciliation
|
||||
against the latest expected protocol and private membership map before import.
|
||||
|
||||
```bash
|
||||
COULOMB_SOCIAL_TARGET_SPACE_LABEL=ReichelAG node scripts/export-bubble-corpus.mjs
|
||||
```
|
||||
|
||||
Set COULOMB_SOCIAL_CORPUS_DIR to a private directory if needed. Do not put raw
|
||||
exports in the checkout. Default target route is /vw_pages; override via
|
||||
COULOMB_SOCIAL_CAPTURE_ROUTE only after confirming the actual authenticated
|
||||
protocol surface. A successful public-route load is not authentication proof.
|
||||
|
|
|
|||
83
docs/release/reichelag-transfer-2026-10-06.md
Normal file
83
docs/release/reichelag-transfer-2026-10-06.md
Normal file
|
|
@ -0,0 +1,83 @@
|
|||
# ReichelAg first transfer: recovery prepared, fresh source needs login
|
||||
|
||||
## User authorization and source state
|
||||
|
||||
Founder answered “yes go on” to current Bubble capture and first ReichelAg
|
||||
transfer following the empty-workspace assessment. New tegwick app sign-in and
|
||||
reset inbox receipt remain accepted. Authority for bounded app snapshot/quiesce:
|
||||
ADMINISTER @ realm:kubernetes/railiance01, activation=APPROVED; continue existing
|
||||
personal-app/brief-interruption approval and the current first-transfer go.
|
||||
|
||||
Initial Bubble capture returned public records and looked usable, but no private
|
||||
weekly protocols. Subsequent metadata inspection found expired Bubble signature
|
||||
cookie. No fresh ReichelAg protocol capture or live import is claimed. User asked
|
||||
to run scripts/capture-auth-state.sh privately and save a new authenticated
|
||||
session after opening the real protocol list. No password/session/reset values
|
||||
read into chat, Git or State Hub. Raw source dumps remain private temporary files.
|
||||
|
||||
## Capture fixes
|
||||
|
||||
Existing exporter guessed the largest captured space, mixed an old seed by
|
||||
default, overwrote full records with partial responses and did not validate
|
||||
expired signature cookies. Added explicit target mode using the Space filter
|
||||
and Search action, disabling old seeds, retaining full fields across partial
|
||||
fresh responses, normalizing typed Bubble lookup references, awaiting response
|
||||
handlers and rejecting missing target protocols/body fields. Target mode defaults
|
||||
to Pages following the user-confirmed Spaces -> ReichelAg -> Pages journey;
|
||||
actual fresh authenticated protocol list still needs reconciliation.
|
||||
|
||||
Four synthetic Node tests cover typed references, expired signature/analytics
|
||||
distinction, partial-record preservation, and explicit target selection rejecting
|
||||
larger unrelated spaces or incomplete bodies. A pytest wrapper includes them
|
||||
in the repo gate. Current expired-session run exits2 before creating any output
|
||||
directory, so public inventory cannot silently become a real import tree.
|
||||
|
||||
## Paired snapshot and isolated restore evidence
|
||||
|
||||
Node snapshot helper scripts/snapshot-content-and-db.py default server dry-run
|
||||
passed against the pinned cluster UID and revision11 image. Execute scopes
|
||||
pg_dump only to coulomb_social_db; app writers are paused, retained PVC mounted
|
||||
read-only by a separate pod with no credential projection or service selector.
|
||||
Pod removed and app1replica restored. Snapshot started19:34:18Z; total operation
|
||||
53.73seconds; completed/app resumed. Public app health200 afterward.
|
||||
|
||||
Database dump49843bytes SHA256
|
||||
26a1630db0a0603dff33a4f2cb128e5ca3bd914d953c335a0f941162dbddf2b3;
|
||||
content archive186bytes SHA256
|
||||
7645781eca69efff473f67dcc736fde7e2a8a66a0172f59056ac57b651b19a25.
|
||||
Snapshot files mode0600, enclosing directory0700, private node/workstation/tmp.
|
||||
These are temporary rehearsal artifacts, not durable scheduled backup custody.
|
||||
|
||||
Restore helper scripts/restore-paired-snapshot.py validates artifact checksums,
|
||||
restores native PostgreSQL in a Docker internal network without published ports,
|
||||
compares every COPY row/table and sequence value, checks archive paths and file
|
||||
hashes. Exact app image retained; offline instance disables identity integrations.
|
||||
Initial production-pair restore:13tables/102rows, all data/sequences match, empty
|
||||
content confirmed, measured database/content restore4.26seconds.
|
||||
|
||||
Populate that isolated instance with the synthetic established weekly-call
|
||||
fixture: full copy, independently edited copy, an asset, owner/viewer/ungranted
|
||||
actors. Snapshot both stores with no writers running; restore into a second
|
||||
isolated PostgreSQL instance. Result:13tables/111rows,4content files, all logical
|
||||
data/sequence/file hashes match, measured restore3.92seconds. Restored Django
|
||||
checks: original and edited body matches; asset bytes and markdown render match;
|
||||
owner can read; viewer write with validCSRF denied403; outsider read denied.
|
||||
Both temporary database containers/internal networks removed afterward.
|
||||
|
||||
Four snapshot safety tests exercise default dry-run/no writes, corrupt dump and
|
||||
cleanup failures resuming the app, and private paired artifact creation. Restore
|
||||
unit checks exercise changed rows/sequences, incomplete COPY and archive escape.
|
||||
Full make check:96passed, including established Chromium workflows. Focused
|
||||
Ruff and Node syntax/capture tests pass.
|
||||
|
||||
## Remaining live work
|
||||
|
||||
CSOC-WP-0008-T02 remains progress: establish retained backup custody/retention,
|
||||
measured real-content recovery and production rollback procedure; successful
|
||||
empty/synthetic recovery does not close actual corpus/visual/privacy acceptance.
|
||||
T03 waits fresh authenticated Bubble capture. After renewal, capture the correct
|
||||
current ReichelAg protocols, reconcile latest expected call and assets/privacy,
|
||||
rehearse import with tenant:platform and only the existing tegwick owner mapping,
|
||||
refresh paired snapshot before a reviewed live import, then validate visible
|
||||
space, latest protocol and explicit denied access. T09 is the user's copy/edit/
|
||||
clipboard/mail-client acceptance; Bubble remains authoritative until accepted.
|
||||
|
|
@ -0,0 +1,83 @@
|
|||
# ReichelAg first transfer: recovery prepared, fresh source needs login
|
||||
|
||||
## User authorization and source state
|
||||
|
||||
Founder answered “yes go on” to current Bubble capture and first ReichelAg
|
||||
transfer following the empty-workspace assessment. New tegwick app sign-in and
|
||||
reset inbox receipt remain accepted. Authority for bounded app snapshot/quiesce:
|
||||
ADMINISTER @ realm:kubernetes/railiance01, activation=APPROVED; continue existing
|
||||
personal-app/brief-interruption approval and the current first-transfer go.
|
||||
|
||||
Initial Bubble capture returned public records and looked usable, but no private
|
||||
weekly protocols. Subsequent metadata inspection found expired Bubble signature
|
||||
cookie. No fresh ReichelAg protocol capture or live import is claimed. User asked
|
||||
to run scripts/capture-auth-state.sh privately and save a new authenticated
|
||||
session after opening the real protocol list. No password/session/reset values
|
||||
read into chat, Git or State Hub. Raw source dumps remain private temporary files.
|
||||
|
||||
## Capture fixes
|
||||
|
||||
Existing exporter guessed the largest captured space, mixed an old seed by
|
||||
default, overwrote full records with partial responses and did not validate
|
||||
expired signature cookies. Added explicit target mode using the Space filter
|
||||
and Search action, disabling old seeds, retaining full fields across partial
|
||||
fresh responses, normalizing typed Bubble lookup references, awaiting response
|
||||
handlers and rejecting missing target protocols/body fields. Target mode defaults
|
||||
to Pages following the user-confirmed Spaces -> ReichelAg -> Pages journey;
|
||||
actual fresh authenticated protocol list still needs reconciliation.
|
||||
|
||||
Four synthetic Node tests cover typed references, expired signature/analytics
|
||||
distinction, partial-record preservation, and explicit target selection rejecting
|
||||
larger unrelated spaces or incomplete bodies. A pytest wrapper includes them
|
||||
in the repo gate. Current expired-session run exits2 before creating any output
|
||||
directory, so public inventory cannot silently become a real import tree.
|
||||
|
||||
## Paired snapshot and isolated restore evidence
|
||||
|
||||
Node snapshot helper scripts/snapshot-content-and-db.py default server dry-run
|
||||
passed against the pinned cluster UID and revision11 image. Execute scopes
|
||||
pg_dump only to coulomb_social_db; app writers are paused, retained PVC mounted
|
||||
read-only by a separate pod with no credential projection or service selector.
|
||||
Pod removed and app1replica restored. Snapshot started19:34:18Z; total operation
|
||||
53.73seconds; completed/app resumed. Public app health200 afterward.
|
||||
|
||||
Database dump49843bytes SHA256
|
||||
26a1630db0a0603dff33a4f2cb128e5ca3bd914d953c335a0f941162dbddf2b3;
|
||||
content archive186bytes SHA256
|
||||
7645781eca69efff473f67dcc736fde7e2a8a66a0172f59056ac57b651b19a25.
|
||||
Snapshot files mode0600, enclosing directory0700, private node/workstation/tmp.
|
||||
These are temporary rehearsal artifacts, not durable scheduled backup custody.
|
||||
|
||||
Restore helper scripts/restore-paired-snapshot.py validates artifact checksums,
|
||||
restores native PostgreSQL in a Docker internal network without published ports,
|
||||
compares every COPY row/table and sequence value, checks archive paths and file
|
||||
hashes. Exact app image retained; offline instance disables identity integrations.
|
||||
Initial production-pair restore:13tables/102rows, all data/sequences match, empty
|
||||
content confirmed, measured database/content restore4.26seconds.
|
||||
|
||||
Populate that isolated instance with the synthetic established weekly-call
|
||||
fixture: full copy, independently edited copy, an asset, owner/viewer/ungranted
|
||||
actors. Snapshot both stores with no writers running; restore into a second
|
||||
isolated PostgreSQL instance. Result:13tables/111rows,4content files, all logical
|
||||
data/sequence/file hashes match, measured restore3.92seconds. Restored Django
|
||||
checks: original and edited body matches; asset bytes and markdown render match;
|
||||
owner can read; viewer write with validCSRF denied403; outsider read denied.
|
||||
Both temporary database containers/internal networks removed afterward.
|
||||
|
||||
Four snapshot safety tests exercise default dry-run/no writes, corrupt dump and
|
||||
cleanup failures resuming the app, and private paired artifact creation. Restore
|
||||
unit checks exercise changed rows/sequences, incomplete COPY and archive escape.
|
||||
Full make check:96passed, including established Chromium workflows. Focused
|
||||
Ruff and Node syntax/capture tests pass.
|
||||
|
||||
## Remaining live work
|
||||
|
||||
CSOC-WP-0008-T02 remains progress: establish retained backup custody/retention,
|
||||
measured real-content recovery and production rollback procedure; successful
|
||||
empty/synthetic recovery does not close actual corpus/visual/privacy acceptance.
|
||||
T03 waits fresh authenticated Bubble capture. After renewal, capture the correct
|
||||
current ReichelAg protocols, reconcile latest expected call and assets/privacy,
|
||||
rehearse import with tenant:platform and only the existing tegwick owner mapping,
|
||||
refresh paired snapshot before a reviewed live import, then validate visible
|
||||
space, latest protocol and explicit denied access. T09 is the user's copy/edit/
|
||||
clipboard/mail-client acceptance; Bubble remains authoritative until accepted.
|
||||
30
scripts/bubble-corpus-utils.mjs
Normal file
30
scripts/bubble-corpus-utils.mjs
Normal file
|
|
@ -0,0 +1,30 @@
|
|||
/** Pure capture guards; fixtures must never contain real cookies or notes. */
|
||||
export function normalizeBubbleRef(value) {
|
||||
if (typeof value !== 'string') return value;
|
||||
if (!value.startsWith('LOOKUP__')) return value;
|
||||
return value.match(/[0-9]+x[0-9]+/)?.[0] || value;
|
||||
}
|
||||
|
||||
export function assertSessionFresh(state, now = Date.now() / 1000) {
|
||||
if ((state.cookies || []).some(c => c.name.endsWith('.sig') &&
|
||||
c.expires > 0 && c.expires <= now)) {
|
||||
throw new Error('bubble_session_expired');
|
||||
}
|
||||
}
|
||||
|
||||
export function mergeCapturedRecord(existing, source, id) {
|
||||
return { ...(existing || {}), ...source, _id: normalizeBubbleRef(id) };
|
||||
}
|
||||
|
||||
export function selectTargetArticles(articles, rootId) {
|
||||
const root = articles[normalizeBubbleRef(rootId)];
|
||||
const spaceId = normalizeBubbleRef(root?.space_custom_space);
|
||||
if (!root || !spaceId) throw new Error('target_space_identity_missing');
|
||||
const selected = Object.values(articles).filter(a =>
|
||||
normalizeBubbleRef(a.space_custom_space) === spaceId);
|
||||
if (selected.length <= 1) throw new Error('target_protocols_not_captured');
|
||||
if (selected.some(a => !Object.hasOwn(a, 'content_text'))) {
|
||||
throw new Error('target_protocol_bodies_incomplete');
|
||||
}
|
||||
return [spaceId, selected];
|
||||
}
|
||||
|
|
@ -7,6 +7,8 @@
|
|||
* node scripts/export-bubble-corpus.mjs
|
||||
*/
|
||||
import { chromium } from "playwright";
|
||||
import { assertSessionFresh, normalizeBubbleRef, mergeCapturedRecord,
|
||||
selectTargetArticles } from "./bubble-corpus-utils.mjs";
|
||||
import {
|
||||
mkdirSync,
|
||||
writeFileSync,
|
||||
|
|
@ -25,6 +27,8 @@ const STATE =
|
|||
process.env.COULOMB_SOCIAL_STORAGE_STATE ||
|
||||
join(homedir(), ".config", "coulomb-social", "auth", "storage-state.json");
|
||||
const BASE = process.env.COULOMB_SOCIAL_URL || "https://coulomb.social";
|
||||
const TARGET_LABEL = process.env.COULOMB_SOCIAL_TARGET_SPACE_LABEL || "";
|
||||
const TARGET_ROUTE = process.env.COULOMB_SOCIAL_CAPTURE_ROUTE || "/vw_pages";
|
||||
const SEED_NET =
|
||||
process.env.COULOMB_SOCIAL_SEED_NETWORK ||
|
||||
join(
|
||||
|
|
@ -73,11 +77,8 @@ function slugify(s) {
|
|||
.slice(0, 80) || "untitled"
|
||||
);
|
||||
}
|
||||
function normId(id) {
|
||||
if (!id || typeof id !== "string") return id;
|
||||
const m = id.match(/LOOKUP__([0-9]+x[0-9]+)/);
|
||||
return m ? m[1] : id;
|
||||
}
|
||||
function normId(id) { return normalizeBubbleRef(id); }
|
||||
|
||||
function ingest(obj, spaces, articles) {
|
||||
if (Array.isArray(obj)) {
|
||||
for (const x of obj) ingest(x, spaces, articles);
|
||||
|
|
@ -88,10 +89,10 @@ function ingest(obj, spaces, articles) {
|
|||
const src = obj._source && typeof obj._source === "object" ? obj._source : null;
|
||||
const id = obj._id || (src && src._id);
|
||||
if (t === "custom.space" && src && id) {
|
||||
spaces[normId(id)] = { ...src, _id: normId(id) };
|
||||
spaces[normId(id)] = mergeCapturedRecord(spaces[normId(id)], src, id);
|
||||
}
|
||||
if (t === "custom.article" && src && id) {
|
||||
const a = { ...src, _id: normId(id) };
|
||||
const a = mergeCapturedRecord(articles[normId(id)], src, id);
|
||||
if (a.space_custom_space) a.space_custom_space = normId(a.space_custom_space);
|
||||
articles[normId(id)] = a;
|
||||
}
|
||||
|
|
@ -162,9 +163,16 @@ if (!existsSync(STATE)) {
|
|||
process.exit(1);
|
||||
}
|
||||
|
||||
try {
|
||||
assertSessionFresh(JSON.parse(readFileSync(STATE, "utf8")));
|
||||
} catch {
|
||||
console.error("Bubble session unavailable or expired. Run ./scripts/capture-auth-state.sh.");
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
const spaces = {};
|
||||
const articles = {};
|
||||
if (existsSync(SEED_NET)) {
|
||||
if (!TARGET_LABEL && process.env.COULOMB_SOCIAL_FRESH_ONLY !== "1" && existsSync(SEED_NET)) {
|
||||
for (const name of readdirSync(SEED_NET)) {
|
||||
if (!name.endsWith(".body.txt")) continue;
|
||||
parseBody(readFileSync(join(SEED_NET, name), "utf8"), spaces, articles);
|
||||
|
|
@ -176,7 +184,9 @@ ensure(OUT);
|
|||
const browser = await chromium.launch({ headless: true });
|
||||
const context = await browser.newContext({ storageState: STATE });
|
||||
const page = await context.newPage();
|
||||
page.on("response", async (response) => {
|
||||
const pendingResponses = [];
|
||||
page.on("response", (response) => {
|
||||
pendingResponses.push((async () => {
|
||||
try {
|
||||
if (!response.url().includes("elasticsearch")) return;
|
||||
const text = await response.text();
|
||||
|
|
@ -184,6 +194,7 @@ page.on("response", async (response) => {
|
|||
} catch {
|
||||
/* ignore */
|
||||
}
|
||||
})());
|
||||
});
|
||||
|
||||
async function loadSpacesList() {
|
||||
|
|
@ -195,6 +206,46 @@ async function loadSpacesList() {
|
|||
}
|
||||
}
|
||||
|
||||
let targetRootId = "";
|
||||
if (TARGET_LABEL) {
|
||||
try {
|
||||
await loadSpacesList();
|
||||
const link = page.getByText(TARGET_LABEL, { exact: true });
|
||||
if (await link.count() !== 1) throw new Error("target_space_link_not_unique");
|
||||
const href = await link.getAttribute("href");
|
||||
if (!href || !new URL(href, BASE).pathname.startsWith("/pg_chunk/")) {
|
||||
throw new Error("target_space_root_not_found");
|
||||
}
|
||||
targetRootId = new URL(href, BASE).pathname.split("/").pop();
|
||||
await page.goto(`${BASE}${TARGET_ROUTE}`, { waitUntil: "domcontentloaded", timeout: 60000 });
|
||||
await page.waitForTimeout(4000);
|
||||
const filter = page.locator("button").filter({
|
||||
has: page.locator("svg text").filter({ hasText: /^\s*filter_list\s*$/ }),
|
||||
});
|
||||
await filter.first().click();
|
||||
const selects = page.locator("select");
|
||||
let selected = false;
|
||||
for (let i = 0; i < await selects.count(); i++) {
|
||||
const matches = await selects.nth(i).evaluate((el, label) =>
|
||||
[...el.options].some(o => o.textContent.trim() === label), TARGET_LABEL);
|
||||
if (!matches) continue;
|
||||
await selects.nth(i).selectOption({ label: TARGET_LABEL });
|
||||
selected = true;
|
||||
break;
|
||||
}
|
||||
if (!selected) throw new Error("target_space_filter_not_found");
|
||||
await page.getByRole("button", { name: "Search", exact: true }).click();
|
||||
await page.waitForTimeout(5000);
|
||||
for (let i = 0; i < 24; i++) {
|
||||
await page.mouse.wheel(0, 1800);
|
||||
await page.waitForTimeout(500);
|
||||
}
|
||||
} catch {
|
||||
await browser.close();
|
||||
console.error("Target capture failed; no import tree published.");
|
||||
process.exit(3);
|
||||
}
|
||||
} else {
|
||||
for (const name of SPACE_CLICKS) {
|
||||
try {
|
||||
await loadSpacesList();
|
||||
|
|
@ -213,9 +264,11 @@ for (const name of SPACE_CLICKS) {
|
|||
}
|
||||
console.log("ok", name, "articles", Object.keys(articles).length);
|
||||
} catch (e) {
|
||||
console.log("fail", name, String(e.message || e).slice(0, 60));
|
||||
console.log("fail", name, "capture_step_failed");
|
||||
}
|
||||
}
|
||||
}
|
||||
await Promise.allSettled(pendingResponses);
|
||||
await browser.close();
|
||||
|
||||
const bySpace = {};
|
||||
|
|
@ -239,11 +292,25 @@ function spaceLabel(sid) {
|
|||
}
|
||||
|
||||
const ranked = Object.entries(bySpace).sort((a, b) => b[1].length - a[1].length);
|
||||
const [sampleId, sampleArts] = ranked[0] || [null, []];
|
||||
const lab = spaceLabel(sampleId);
|
||||
let sampleId, sampleArts;
|
||||
if (TARGET_LABEL) {
|
||||
try {
|
||||
[sampleId, sampleArts] = selectTargetArticles(articles, targetRootId);
|
||||
} catch {
|
||||
console.error("Target protocols unavailable; no import tree published. Renew Bubble login.");
|
||||
process.exit(4);
|
||||
}
|
||||
} else {
|
||||
[sampleId, sampleArts] = ranked[0] || [null, []];
|
||||
}
|
||||
const lab = TARGET_LABEL ? { ...spaceLabel(sampleId), slug: slugify(TARGET_LABEL),
|
||||
title: TARGET_LABEL } : spaceLabel(sampleId);
|
||||
|
||||
const index = {
|
||||
exported_at: new Date().toISOString(),
|
||||
target_requested: TARGET_LABEL || null,
|
||||
source_route: TARGET_LABEL ? TARGET_ROUTE : null,
|
||||
old_seed_used: !TARGET_LABEL && process.env.COULOMB_SOCIAL_FRESH_ONLY !== "1" && existsSync(SEED_NET),
|
||||
space_count: Object.keys(spaces).length,
|
||||
article_count: Object.keys(articles).length,
|
||||
sample_tree: sampleId ? `trees/${lab.slug}` : null,
|
||||
|
|
|
|||
160
scripts/restore-paired-snapshot.py
Normal file
160
scripts/restore-paired-snapshot.py
Normal file
|
|
@ -0,0 +1,160 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Restore a private snapshot into isolated Docker PostgreSQL; never production.
|
||||
|
||||
No published ports; an internal network blocks external integrations. Compare
|
||||
all restored COPY rows and sequence state, and all archived file hashes. A
|
||||
successful empty-content restore alone does not prove real corpus recovery.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import tarfile
|
||||
import tempfile
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
class Refused(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
def run(args: list[str], *, payload: bytes | None = None) -> bytes:
|
||||
p = subprocess.run(args, input=payload, capture_output=True, timeout=180)
|
||||
if p.returncode:
|
||||
raise Refused('command_failed')
|
||||
return p.stdout
|
||||
|
||||
|
||||
def data_fingerprint(sql: bytes) -> dict:
|
||||
"""Ignore SQL boilerplate, compare COPY data and explicit sequence state."""
|
||||
tables = {}
|
||||
sequences = []
|
||||
current = None
|
||||
rows = []
|
||||
for line in sql.splitlines():
|
||||
if current is not None:
|
||||
if line == b'\\.':
|
||||
tables[current] = {
|
||||
'rows': len(rows),
|
||||
'sha256': hashlib.sha256(b'\n'.join(sorted(rows))).hexdigest(),
|
||||
}
|
||||
current, rows = None, []
|
||||
else:
|
||||
rows.append(line)
|
||||
elif line.startswith(b'COPY ') and line.endswith(b' FROM stdin;'):
|
||||
current = line.decode('utf-8')
|
||||
elif line.startswith(b'SELECT pg_catalog.setval('):
|
||||
sequences.append(line.decode('utf-8'))
|
||||
if current is not None:
|
||||
raise Refused('unterminated_copy')
|
||||
if not tables:
|
||||
raise Refused('no_tables_in_dump')
|
||||
return {'tables': tables, 'sequences': sorted(sequences)}
|
||||
|
||||
|
||||
def content_fingerprint(archive: Path, destination: Path) -> dict[str, str]:
|
||||
hashes = {}
|
||||
with tarfile.open(archive, 'r:gz') as t:
|
||||
for entry in t.getmembers():
|
||||
parts = Path(entry.name).parts
|
||||
if (not parts or parts[0] != 'content' or '..' in parts
|
||||
or Path(entry.name).is_absolute()):
|
||||
raise Refused('invalid_archive_path')
|
||||
if not (entry.isfile() or entry.isdir()):
|
||||
raise Refused('unsupported_archive_entry')
|
||||
if entry.isfile():
|
||||
hashes[entry.name] = hashlib.sha256(t.extractfile(entry).read()).hexdigest()
|
||||
t.extractall(destination, filter='data')
|
||||
actual = {str(p.relative_to(destination)): hashlib.sha256(p.read_bytes()).hexdigest()
|
||||
for p in (destination / 'content').rglob('*') if p.is_file()}
|
||||
if actual != hashes:
|
||||
raise Refused('content_hash_mismatch')
|
||||
return hashes
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument('snapshot', type=Path)
|
||||
parser.add_argument('--keep-instance', action='store_true',
|
||||
help='Keep the isolated internal network/database for a content rehearsal')
|
||||
args = parser.parse_args()
|
||||
source = args.snapshot.resolve()
|
||||
receipt = json.loads((source / 'receipt.json').read_text())
|
||||
if receipt.get('status') != 'captured' or not receipt.get('writers_quiesced'):
|
||||
raise Refused('snapshot_not_paired')
|
||||
for name, key in [('database.dump', 'database_sha256'), ('content.tar.gz', 'content_sha256')]:
|
||||
if hashlib.sha256((source / name).read_bytes()).hexdigest() != receipt[key]:
|
||||
raise Refused('artifact_checksum_mismatch')
|
||||
started = time.monotonic()
|
||||
name = 'csoc-recovery-' + time.strftime('%Y%m%d%H%M%S', time.gmtime())
|
||||
network = name + '-net'
|
||||
work = Path(tempfile.mkdtemp(prefix=name + '-', dir='/tmp'))
|
||||
image = run(['docker', 'image', 'inspect', 'postgres:16',
|
||||
'--format', '{{.Id}}']).decode().strip()
|
||||
if not re.fullmatch(r'sha256:[a-f0-9]{64}', image):
|
||||
raise Refused('unresolved_postgres_image')
|
||||
created = False
|
||||
network_created = False
|
||||
verified = False
|
||||
try:
|
||||
run(['docker', 'network', 'create', '--internal', network])
|
||||
network_created = True
|
||||
run(['docker', 'run', '-d', '--name', name, '--network', network,
|
||||
'--network-alias', 'recovery-db', '-e', 'POSTGRES_HOST_AUTH_METHOD=trust',
|
||||
'-e', 'POSTGRES_DB=coulomb_social_restore', image])
|
||||
created = True
|
||||
for _ in range(45):
|
||||
p = subprocess.run(['docker', 'exec', name, 'pg_isready', '-U', 'postgres'],
|
||||
capture_output=True)
|
||||
if p.returncode == 0:
|
||||
break
|
||||
time.sleep(1)
|
||||
else:
|
||||
raise Refused('restore_db_not_ready')
|
||||
archive = (source / 'database.dump').read_bytes()
|
||||
original_sql = run(['docker', 'exec', '-i', name, 'pg_restore', '--data-only', '-f', '-'],
|
||||
payload=archive)
|
||||
baseline = data_fingerprint(original_sql)
|
||||
run(['docker', 'exec', '-i', name, 'pg_restore', '-U', 'postgres',
|
||||
'-d', 'coulomb_social_restore', '--no-owner', '--no-acl', '--exit-on-error'],
|
||||
payload=archive)
|
||||
restored_sql = run(['docker', 'exec', name, 'pg_dump', '-U', 'postgres',
|
||||
'-d', 'coulomb_social_restore', '--data-only',
|
||||
'--no-owner', '--no-acl'])
|
||||
if data_fingerprint(restored_sql) != baseline:
|
||||
raise Refused('database_fingerprint_mismatch')
|
||||
files = content_fingerprint(source / 'content.tar.gz', work)
|
||||
result = {'status': 'restored_verified', 'database_tables': len(baseline['tables']),
|
||||
'database_rows': sum(t['rows'] for t in baseline['tables'].values()),
|
||||
'database_rows_and_sequences_match': True, 'content_files': len(files),
|
||||
'content_hashes_match': True, 'network_internal': True, 'published_ports': 0,
|
||||
'postgres_image_id': image, 'elapsed_seconds': round(time.monotonic()-started, 2),
|
||||
'content_was_empty': not files,
|
||||
'database_container': name, 'network': network, 'restored_directory': str(work)}
|
||||
path = work / 'restore-receipt.json'
|
||||
fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)
|
||||
with os.fdopen(fd, 'w') as f:
|
||||
json.dump(result, f, indent=2)
|
||||
verified = True
|
||||
print(json.dumps(result))
|
||||
finally:
|
||||
if not (args.keep_instance and verified):
|
||||
if created:
|
||||
run(['docker', 'rm', '-f', '-v', name])
|
||||
if network_created:
|
||||
run(['docker', 'network', 'rm', network])
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
try:
|
||||
sys.exit(main())
|
||||
except Exception as exc:
|
||||
print(json.dumps({'status': 'failed', 'failure_type': type(exc).__name__}))
|
||||
sys.exit(1)
|
||||
38
scripts/tests/bubble-corpus-utils.test.mjs
Normal file
38
scripts/tests/bubble-corpus-utils.test.mjs
Normal file
|
|
@ -0,0 +1,38 @@
|
|||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { normalizeBubbleRef, assertSessionFresh, mergeCapturedRecord,
|
||||
selectTargetArticles } from '../bubble-corpus-utils.mjs';
|
||||
|
||||
test('typed lookup references normalize to the same space identifier', () => {
|
||||
const id = '1700000000000x1234567890';
|
||||
assert.equal(normalizeBubbleRef(`LOOKUP__custom.space__${id}`), id);
|
||||
assert.equal(normalizeBubbleRef(`LOOKUP__${id}`), id);
|
||||
assert.equal(normalizeBubbleRef(id), id);
|
||||
});
|
||||
|
||||
test('expired signature blocks capture; an expired analytics cookie does not', () => {
|
||||
assert.throws(() => assertSessionFresh({cookies:[{name:'app.sig',expires:9}]},10),
|
||||
/bubble_session_expired/);
|
||||
assert.doesNotThrow(() => assertSessionFresh({cookies:[
|
||||
{name:'analytics',expires:9},{name:'app.sig',expires:11}]},10));
|
||||
});
|
||||
|
||||
test('partial network responses retain full notes captured in this run', () => {
|
||||
const first = mergeCapturedRecord(null,{content_text:'synthetic full notes'},'root');
|
||||
assert.equal(mergeCapturedRecord(first,{nomer_text:'Synthetic call'},'root').content_text,
|
||||
'synthetic full notes');
|
||||
});
|
||||
|
||||
test('explicit target selection never substitutes the largest space', () => {
|
||||
const records = {root:{space_custom_space:'target',content_text:'synthetic index'},
|
||||
call:{space_custom_space:'target',content_text:'synthetic protocol'},
|
||||
a:{space_custom_space:'other'},b:{space_custom_space:'other'},c:{space_custom_space:'other'}};
|
||||
const [id, selected] = selectTargetArticles(records,'root');
|
||||
assert.equal(id,'target');
|
||||
assert.equal(selected.length,2);
|
||||
delete records.call.content_text;
|
||||
assert.throws(() => selectTargetArticles(records,'root'), /target_protocol_bodies_incomplete/);
|
||||
records.call.content_text = 'synthetic protocol';
|
||||
delete records.call;
|
||||
assert.throws(() => selectTargetArticles(records,'root'), /target_protocols_not_captured/);
|
||||
});
|
||||
12
tests/test_bubble_capture_guards.py
Normal file
12
tests/test_bubble_capture_guards.py
Normal file
|
|
@ -0,0 +1,12 @@
|
|||
"""Run synthetic source-capture guards in the same Node runtime as the exporter."""
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def test_bubble_capture_guards():
|
||||
root = Path(__file__).resolve().parents[1]
|
||||
result = subprocess.run(
|
||||
['node', '--test', 'scripts/tests/bubble-corpus-utils.test.mjs'],
|
||||
cwd=root, capture_output=True, text=True, check=False,
|
||||
)
|
||||
assert result.returncode == 0, result.stdout + result.stderr
|
||||
41
tests/test_restore_recovery.py
Normal file
41
tests/test_restore_recovery.py
Normal file
|
|
@ -0,0 +1,41 @@
|
|||
"""Logical backup comparisons must catch changed rows, sequences and paths."""
|
||||
import importlib.util
|
||||
import io
|
||||
import tarfile
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def helper():
|
||||
path = Path(__file__).resolve().parents[1] / 'scripts/restore-paired-snapshot.py'
|
||||
spec = importlib.util.spec_from_file_location('csoc_restore', path)
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
def test_data_comparison_ignores_order_but_detects_changes(helper):
|
||||
a = b'COPY public.demo (id, value) FROM stdin;\n1\tone\n2\ttwo\n\\.\n'
|
||||
b = b'-- SQL header\nCOPY public.demo (id, value) FROM stdin;\n2\ttwo\n1\tone\n\\.\n'
|
||||
assert helper.data_fingerprint(a) == helper.data_fingerprint(b)
|
||||
assert helper.data_fingerprint(a) != helper.data_fingerprint(b.replace(b'two', b'changed'))
|
||||
assert helper.data_fingerprint(a) != helper.data_fingerprint(
|
||||
a + b"SELECT pg_catalog.setval('public.demo_seq', 9, true);\n")
|
||||
|
||||
|
||||
def test_partial_copy_is_rejected(helper):
|
||||
with pytest.raises(helper.Refused):
|
||||
helper.data_fingerprint(b'COPY public.demo (id) FROM stdin;\n1\n')
|
||||
|
||||
|
||||
def test_content_archive_rejects_escape(helper, tmp_path):
|
||||
archive = tmp_path / 'bad.tar.gz'
|
||||
with tarfile.open(archive, 'w:gz') as t:
|
||||
info = tarfile.TarInfo('content/../../outside')
|
||||
info.size = 4
|
||||
t.addfile(info, io.BytesIO(b'data'))
|
||||
with pytest.raises(helper.Refused):
|
||||
helper.content_fingerprint(archive, tmp_path / 'restored')
|
||||
assert not (tmp_path / 'outside').exists()
|
||||
|
|
@ -102,7 +102,7 @@ snapshots; no private archive in git or State Hub.
|
|||
|
||||
```task
|
||||
id: CSOC-WP-0008-T03
|
||||
status: progress
|
||||
status: wait
|
||||
priority: high
|
||||
state_hub_task_id: "0615eff2-bf2b-503b-8f15-0591d4666446"
|
||||
```
|
||||
|
|
@ -395,7 +395,7 @@ prerequisite. Diagnostic receipt: history/20261006T190725Z-empty-personal-spaces
|
|||
## Fresh ReichelAg transfer execution — 2026-10-06
|
||||
|
||||
Founder answered “yes go on” to current Bubble capture and first ReichelAg
|
||||
transfer. Existing private browser session valid. Old exporter selects largest
|
||||
transfer. Saved browser session probed; expired signature cookie prevents private protocol access. Old exporter selects largest
|
||||
captured space and assumes title click sets active space; neither proves target
|
||||
selection. Fresh source capture must explicitly select ReichelAG in Pages filter
|
||||
and apply Search, with old network seed disabled. Retain source-ID provenance,
|
||||
|
|
@ -406,3 +406,24 @@ server dry-run passed. Quiesce app, scope pg_dump to coulomb_social_db, read-onl
|
|||
PVC archive, restore app even on failure; isolated PostgreSQL restore must pass
|
||||
before real import. Four focused safety tests pass (dry-run/no writes, corrupt
|
||||
dump/app-resume, cleanup failure/app-resume, private paired artifacts).
|
||||
|
||||
Paired production snapshot completed; app resumed/healthy. Isolated PostgreSQL
|
||||
restore matches13tables/102rows with empty content. A second synthetic populated
|
||||
pair restores13tables/111rows and4files; full-protocol copy/independent edit,
|
||||
asset/render fidelity, viewer validCSRF write denial and outsider read denial
|
||||
pass after recovery.96tests pass. T02 remains progress for retained backup
|
||||
custody/retention and real corpus/rollback review.
|
||||
|
||||
T03 wait: Bubble signature cookie expired; public-only captures are not accepted
|
||||
source. User must privately renew via scripts/capture-auth-state.sh and open
|
||||
actual ReichelAg protocols. Targeted exporter now fails before output on expired
|
||||
session, uses explicit target/Space filter/Search, excludes old seeds, merges
|
||||
partial fresh records and rejects missing protocol bodies. Source receipt:
|
||||
docs/release/reichelag-transfer-2026-10-06.md. No productive import performed.
|
||||
|
||||
User confirmed the old journey: Spaces -> ReichelAg -> Pages, ordered by latest
|
||||
changes. Capture defaults to /vw_pages. Prioritize current content migration;
|
||||
UI framework architecture is a separate next phase (CSOC-WP-0011), before broad
|
||||
UX rebuilding. Existing browser session renewal or a dedicated Bubble account
|
||||
with read access can provide the source; never accept password/session contents
|
||||
in chat. User renewal remains the next input needed for private protocol capture.
|
||||
|
|
|
|||
68
workplans/CSOC-WP-0011-ui-framework-architecture.md
Normal file
68
workplans/CSOC-WP-0011-ui-framework-architecture.md
Normal file
|
|
@ -0,0 +1,68 @@
|
|||
---
|
||||
id: CSOC-WP-0011
|
||||
type: workplan
|
||||
title: "Establish UI framework architecture before broad Bubble UX rebuild"
|
||||
domain: communication
|
||||
repo: coulomb-social
|
||||
status: backlog
|
||||
owner: codex
|
||||
topic_slug: coulomb-social
|
||||
created: "2026-10-06"
|
||||
updated: "2026-10-06"
|
||||
---
|
||||
|
||||
# User milestone and priority
|
||||
|
||||
After the first content transfer, the interface has a consistent foundation for
|
||||
finding spaces, opening pages ordered by latest changes, editing/copying protocols
|
||||
and distributing notes by clipboard. New Bubble use cases can reuse the same
|
||||
navigation, forms, interaction and permission patterns. The framework decision
|
||||
is explicit and reviewed before broad UX investment; no new frontend stack is
|
||||
assumed merely to make the current migration work.
|
||||
|
||||
User steering on 2026-10-06: prioritize content migration, then establish framework
|
||||
architecture before rebuilding many UI elements. CSOC-WP-0008-T02/T03/T09 remain
|
||||
first; this backlog workplan does not block source capture or the first transfer.
|
||||
|
||||
## Inventory existing UI and observed Bubble interaction patterns
|
||||
|
||||
```task
|
||||
id: CSOC-WP-0011-T01
|
||||
status: todo
|
||||
priority: medium
|
||||
```
|
||||
|
||||
Review current Django templates, routes, forms, permission checks and clipboard
|
||||
JavaScript together with user-reviewed Bubble journeys. Describe navigation,
|
||||
latest-change ordering, protocol copying and editing, reusable components and
|
||||
accessibility needs. Use captured UI evidence separately from private content.
|
||||
|
||||
## Decide the UI framework boundaries and extension contract
|
||||
|
||||
```task
|
||||
id: CSOC-WP-0011-T02
|
||||
status: todo
|
||||
priority: medium
|
||||
```
|
||||
|
||||
Prepare an ADR comparing the existing server-rendered foundation with progressive
|
||||
interaction or a distinct frontend only where user needs justify it. Specify
|
||||
component ownership, routing/state, server validation, auth/permission boundary,
|
||||
asset delivery and regression test contract. Review with the user before broad
|
||||
UX implementation. Avoid coupling the content migration format to the UI stack.
|
||||
|
||||
## Demonstrate the foundation on the accepted ReichelAg journey
|
||||
|
||||
```task
|
||||
id: CSOC-WP-0011-T03
|
||||
status: todo
|
||||
priority: medium
|
||||
```
|
||||
|
||||
After content acceptance and framework review, implement one representative
|
||||
space/page interaction on the decided foundation. Keep the full-protocol copy,
|
||||
independent edit, latest-protocol discovery and native clipboard workflows under
|
||||
established regression coverage. User reviews consistency and usability before
|
||||
other Bubble UI surfaces are rebuilt. Register later UI gaps as live follow-ups.
|
||||
|
||||
Founder content-first decision: e67cfd2e-e55d-4b42-947a-b279b6287c55.
|
||||
Loading…
Add table
Add a link
Reference in a new issue