diff --git a/docs/migration/README.md b/docs/migration/README.md index 6d3aede..49fc2a1 100644 --- a/docs/migration/README.md +++ b/docs/migration/README.md @@ -6,6 +6,7 @@ CSOC-WP-0001-T02/T04 home for Bubble → rebuild mapping and rehearsal notes. |-----|--------| | `bubble-auth-and-export.md` | **Safe read access for agents** (session file; no secrets in chat) | | `field-map-from-session-2026-08-13.md` | **Field map from live session export** (article/space/user) | +| `export-progress-2026-08-13.md` | Corpus counts + sample tree path (no private bodies) | | `schema-mapping-sketch-2026-08-12.md` | Provisional mapping (supersede with field map) | | *(later)* field-level map | After authenticated dump | | *(later)* rehearsal log | Import of one space to thin git tree (ADR-0004) | diff --git a/docs/migration/export-progress-2026-08-13.md b/docs/migration/export-progress-2026-08-13.md new file mode 100644 index 0000000..a60f05f --- /dev/null +++ b/docs/migration/export-progress-2026-08-13.md @@ -0,0 +1,82 @@ +# Export progress — 2026-08-13 + +| Field | Value | +|-------|--------| +| Workplan | CSOC-WP-0001-T02 | +| Auth | Local Playwright session (XDG; not in git) | +| Local corpus | `~/.config/coulomb-social/auth/exports/corpus-202608131015057/` | + +## Results (no content bodies in git) + +| Metric | Count | +|--------|------:| +| Spaces | 71 | +| Articles (unique) | 109 | +| Spaces with ≥1 article in dump | 70 | +| Sample tree pages | 30 (29 articles + index) | +| Sample visuals downloaded | 29 (~1.6 MB) | + +### Articles per space (histogram) + +| Articles in space | # spaces | +|------------------:|---------:| +| 0 | 1 | +| 1 | 62 | +| 2 | 4 | +| 3 | 2 | +| 4 | 1 | +| 29 | 1 (sample) | + +Most spaces are thin (one home/article). One dense space was used as the +**rehearsal tree** (local only). + +## Sample tree shape (ADR-0004) + +```text +trees// + pages/ + index.md # space landing + .md # title/abstractor/visual + body + assets/ + -visual.* # logo_image downloads +``` + +Frontmatter keys (per page): + +```yaml +id: bubble: +title: "…" # nomer_text +abstractor: "…" # abstractor_text +visual: "assets/…" # logo_image → local file +space: "" +bubble_type: article +archived: false +parent: "" # context_custom_article when set +``` + +## Methods used + +1. Seed from first session network capture (ES msearch/mget responses). +2. UI walk: open named spaces on `/vw_spaces` then `/vw_pages` + scroll (Bubble ES + request bodies are **encrypted** — cannot craft free-form API queries). +3. Emit JSON corpus + markdown tree under XDG only. + +## Limits + +- Article coverage beyond the “active” space is incomplete (many spaces only + have 1 article in the dump; more may exist server-side). +- Space **display title** is not a first-class field on `custom.space` (only + `Slug`); UI names come from related articles / presentation. +- Dense sample content is **private** — tree stays out of git. + +## Next + +1. Optional: longer crawl / pagination to fill more per-space articles. +2. **T04 rehearsal:** PageOps import of local tree into app content dir (CSOC-WP-0006). +3. Member → NetKingdom identity mapping for membership lists (no passwords). + +## Related + +- `docs/migration/field-map-from-session-2026-08-13.md` +- `docs/migration/bubble-auth-and-export.md` +- ADR-0003, ADR-0004 diff --git a/docs/migration/field-map-from-session-2026-08-13.md b/docs/migration/field-map-from-session-2026-08-13.md index 186951a..9188a37 100644 --- a/docs/migration/field-map-from-session-2026-08-13.md +++ b/docs/migration/field-map-from-session-2026-08-13.md @@ -140,9 +140,9 @@ archived: false ## Next actions -1. Extend exporter to walk **all spaces** the user can access and dump article JSON redacted/normalized into a private corpus. -2. Download visuals for sample space. -3. Emit one space as ADR-0004 tree (rehearsal dry-run for T04). +1. ~~Extend exporter / sample tree~~ — see `export-progress-2026-08-13.md` + `scripts/export-bubble-corpus.mjs`. +2. Optional: deeper crawl for spaces with only 1 article if more body pages exist. +3. **T04:** import local sample tree via PageOps (CSOC-WP-0006). 4. Keep raw export **only** under XDG auth dir. ## Privacy diff --git a/scripts/export-bubble-corpus.mjs b/scripts/export-bubble-corpus.mjs new file mode 100755 index 0000000..e55bd8d --- /dev/null +++ b/scripts/export-bubble-corpus.mjs @@ -0,0 +1,334 @@ +#!/usr/bin/env node +/** + * Build a Bubble article/space corpus from a Playwright session and emit + * one ADR-0004 markdown tree. Output stays under XDG auth exports (not git). + * + * ./scripts/capture-auth-state.sh + * node scripts/export-bubble-corpus.mjs + */ +import { chromium } from "playwright"; +import { + mkdirSync, + writeFileSync, + chmodSync, + existsSync, + readFileSync, + readdirSync, + createWriteStream, +} from "node:fs"; +import { homedir } from "node:os"; +import { join } from "node:path"; +import https from "node:https"; +import http from "node:http"; + +const STATE = + process.env.COULOMB_SOCIAL_STORAGE_STATE || + join(homedir(), ".config", "coulomb-social", "auth", "storage-state.json"); +const BASE = process.env.COULOMB_SOCIAL_URL || "https://coulomb.social"; +const SEED_NET = + process.env.COULOMB_SOCIAL_SEED_NETWORK || + join( + homedir(), + ".config", + "coulomb-social", + "auth", + "exports", + "20260813-083747", + "network", + ); +const stamp = new Date().toISOString().replace(/[-:TZ.]/g, "").slice(0, 15); +const OUT = + process.env.COULOMB_SOCIAL_CORPUS_DIR || + join(homedir(), ".config", "coulomb-social", "auth", "exports", `corpus-${stamp}`); + +const SPACE_CLICKS = ( + process.env.COULOMB_SOCIAL_SPACE_NAMES || + "DeepNews,RepositoryScoping,Sandbox,CoulombConfig,IntentRoyal,DecisionGuidesLibrary,NetKingdom,TinkerBase,OpsBridge,ReichelAG,SpaceCampaign,AiFeedEditor" +).split(",").map((s) => s.trim()).filter(Boolean); + +function ensure(p, mode = 0o700) { + mkdirSync(p, { recursive: true, mode }); + try { + chmodSync(p, mode); + } catch { + /* best effort */ + } +} +function write(p, data) { + writeFileSync(p, data, { mode: 0o600 }); + try { + chmodSync(p, 0o600); + } catch { + /* best effort */ + } +} +function slugify(s) { + return ( + String(s || "untitled") + .toLowerCase() + .normalize("NFKD") + .replace(/[\u0300-\u036f]/g, "") + .replace(/[^a-z0-9]+/g, "-") + .replace(/^-+|-+$/g, "") + .slice(0, 80) || "untitled" + ); +} +function normId(id) { + if (!id || typeof id !== "string") return id; + const m = id.match(/LOOKUP__([0-9]+x[0-9]+)/); + return m ? m[1] : id; +} +function ingest(obj, spaces, articles) { + if (Array.isArray(obj)) { + for (const x of obj) ingest(x, spaces, articles); + return; + } + if (!obj || typeof obj !== "object") return; + const t = obj._type; + const src = obj._source && typeof obj._source === "object" ? obj._source : null; + const id = obj._id || (src && src._id); + if (t === "custom.space" && src && id) { + spaces[normId(id)] = { ...src, _id: normId(id) }; + } + if (t === "custom.article" && src && id) { + const a = { ...src, _id: normId(id) }; + if (a.space_custom_space) a.space_custom_space = normId(a.space_custom_space); + articles[normId(id)] = a; + } + if (Array.isArray(obj.responses)) { + for (const r of obj.responses) ingest(r, spaces, articles); + } + if (obj.hits && typeof obj.hits === "object") { + for (const h of obj.hits.hits || []) ingest(h, spaces, articles); + } + for (const v of Object.values(obj)) { + if (v && typeof v === "object") ingest(v, spaces, articles); + } +} +function parseBody(text, spaces, articles) { + try { + ingest(JSON.parse(text), spaces, articles); + return; + } catch { + /* try NDJSON */ + } + for (const line of text.split("\n")) { + try { + ingest(JSON.parse(line), spaces, articles); + } catch { + /* ignore */ + } + } +} +function download(url, dest) { + if (!url || typeof url !== "string") return Promise.resolve(null); + let u = url; + if (u.startsWith("//")) u = "https:" + u; + if (u.startsWith("/")) u = BASE.replace(/\/$/, "") + u; + if (!u.startsWith("http")) return Promise.resolve(null); + return new Promise((resolve) => { + const lib = u.startsWith("https") ? https : http; + const file = createWriteStream(dest, { mode: 0o600 }); + const req = lib.get(u, { headers: { "User-Agent": "coulomb-export/1.0" } }, (res) => { + if (res.statusCode && res.statusCode >= 300 && res.statusCode < 400 && res.headers.location) { + file.close(); + download(res.headers.location, dest).then(resolve); + return; + } + if (res.statusCode && res.statusCode >= 400) { + file.close(); + resolve(null); + return; + } + res.pipe(file); + file.on("finish", () => { + file.close(); + resolve(dest); + }); + }); + req.on("error", () => { + try { + file.close(); + } catch { + /* ignore */ + } + resolve(null); + }); + }); +} + +if (!existsSync(STATE)) { + console.error("No session. Run ./scripts/capture-auth-state.sh first."); + process.exit(1); +} + +const spaces = {}; +const articles = {}; +if (existsSync(SEED_NET)) { + for (const name of readdirSync(SEED_NET)) { + if (!name.endsWith(".body.txt")) continue; + parseBody(readFileSync(join(SEED_NET, name), "utf8"), spaces, articles); + } + console.log("seeded", Object.keys(spaces).length, "spaces", Object.keys(articles).length, "articles"); +} + +ensure(OUT); +const browser = await chromium.launch({ headless: true }); +const context = await browser.newContext({ storageState: STATE }); +const page = await context.newPage(); +page.on("response", async (response) => { + try { + if (!response.url().includes("elasticsearch")) return; + const text = await response.text(); + if (text && text.length < 5_000_000) parseBody(text, spaces, articles); + } catch { + /* ignore */ + } +}); + +async function loadSpacesList() { + await page.goto(`${BASE}/vw_spaces`, { waitUntil: "domcontentloaded", timeout: 60000 }); + await page.waitForTimeout(3500); + for (let i = 0; i < 6; i++) { + await page.mouse.wheel(0, 2000); + await page.waitForTimeout(600); + } +} + +for (const name of SPACE_CLICKS) { + try { + await loadSpacesList(); + const loc = page.getByText(name, { exact: true }).first(); + if ((await loc.count()) === 0) { + console.log("skip", name); + continue; + } + await loc.click({ timeout: 5000 }); + await page.waitForTimeout(2500); + await page.goto(`${BASE}/vw_pages`, { waitUntil: "domcontentloaded", timeout: 60000 }); + await page.waitForTimeout(3000); + for (let i = 0; i < 4; i++) { + await page.mouse.wheel(0, 1800); + await page.waitForTimeout(400); + } + console.log("ok", name, "articles", Object.keys(articles).length); + } catch (e) { + console.log("fail", name, String(e.message || e).slice(0, 60)); + } +} +await browser.close(); + +const bySpace = {}; +for (const a of Object.values(articles)) { + const sid = a.space_custom_space; + if (!sid) continue; + (bySpace[sid] ||= []).push(a); +} + +function spaceLabel(sid) { + const sp = spaces[sid] || {}; + const rawSlug = sp.Slug || ""; + const arts = bySpace[sid] || []; + const nomers = arts.map((a) => a.nomer_text).filter(Boolean); + let title = + nomers.find((n) => n === "ReichelAG") || + nomers.find((n) => !/^\d{6}-/.test(n) && !/^new /i.test(n) && n.length < 48) || + rawSlug || + sid.slice(-12); + return { slug: slugify(title), title, rawSlug }; +} + +const ranked = Object.entries(bySpace).sort((a, b) => b[1].length - a[1].length); +const [sampleId, sampleArts] = ranked[0] || [null, []]; +const lab = spaceLabel(sampleId); + +const index = { + exported_at: new Date().toISOString(), + space_count: Object.keys(spaces).length, + article_count: Object.keys(articles).length, + sample_tree: sampleId ? `trees/${lab.slug}` : null, + spaces: Object.entries(spaces).map(([id]) => { + const L = spaceLabel(id); + return { + id, + slug: L.slug, + bubble_slug: L.rawSlug, + title_guess: L.title, + article_count: (bySpace[id] || []).length, + }; + }), +}; +write(join(OUT, "index.json"), JSON.stringify(index, null, 2)); +write(join(OUT, "spaces.json"), JSON.stringify(spaces, null, 2)); +write(join(OUT, "articles.json"), JSON.stringify(articles, null, 2)); + +const treeRoot = join(OUT, "trees", lab.slug); +ensure(join(treeRoot, "pages")); +ensure(join(treeRoot, "assets")); +const used = new Set(); +let visuals = 0; +for (const a of sampleArts) { + let pageSlug = slugify(a.nomer_text || a._id); + let n = 2; + while (used.has(pageSlug)) pageSlug = `${slugify(a.nomer_text || a._id)}-${n++}`; + used.add(pageSlug); + let visualRel = ""; + if (a.logo_image) { + const lower = String(a.logo_image).toLowerCase(); + const ext = lower.includes(".png") + ? "png" + : lower.includes(".jpg") || lower.includes(".jpeg") + ? "jpg" + : lower.includes(".webp") + ? "webp" + : "bin"; + const assetName = `${pageSlug}-visual.${ext}`; + const dest = join(treeRoot, "assets", assetName); + const ok = await download(a.logo_image, dest); + if (ok && existsSync(dest)) { + visualRel = `assets/${assetName}`; + visuals++; + } + } + const md = `--- +id: bubble:${a._id} +title: ${JSON.stringify(a.nomer_text || "")} +abstractor: ${JSON.stringify(a.abstractor_text || "")} +visual: ${JSON.stringify(visualRel)} +space: ${JSON.stringify(lab.slug)} +bubble_type: article +archived: ${Boolean(a.archived_boolean)} +parent: ${JSON.stringify(a.context_custom_article ? `bubble:${normId(String(a.context_custom_article))}` : "")} +--- + +${a.content_text || ""} +`; + write(join(treeRoot, "pages", `${pageSlug}.md`), md); +} +write( + join(treeRoot, "pages", "index.md"), + `--- +id: bubble-space:${sampleId} +title: ${JSON.stringify(lab.title)} +abstractor: ${JSON.stringify(`Imported sample (${sampleArts.length} articles)`)} +visual: "" +space: ${JSON.stringify(lab.slug)} +bubble_type: space +--- + +# ${lab.title} +`, +); +write( + join(OUT, "SUMMARY.md"), + `# Corpus ${stamp} + +- spaces: ${Object.keys(spaces).length} +- articles: ${Object.keys(articles).length} +- sample: trees/${lab.slug}/ (${sampleArts.length} pages, ${visuals} visuals) + +Do not commit — may contain private content. +`, +); +console.log("OUT", OUT); +console.log("tree", treeRoot, "pages", sampleArts.length, "visuals", visuals); diff --git a/workplans/CSOC-WP-0001-bubble-io-exit-assessment.md b/workplans/CSOC-WP-0001-bubble-io-exit-assessment.md index 139d420..fe0b59e 100644 --- a/workplans/CSOC-WP-0001-bubble-io-exit-assessment.md +++ b/workplans/CSOC-WP-0001-bubble-io-exit-assessment.md @@ -85,8 +85,15 @@ Operator captures Playwright session once (`capture-auth-state.sh`); agents use 2026-08-13: **Session export run** (probe `vw_pages`, 147 network captures). Research content is primarily **`custom.article`** (nomer/abstractor/content/ logo + space link), not empty public meta. Field map (redacted): -`docs/migration/field-map-from-session-2026-08-13.md`. Remaining: multi-space -walk, visual download, emit sample tree, then T04 rehearsal. +`docs/migration/field-map-from-session-2026-08-13.md`. + +2026-08-13: **Corpus + sample tree** (local only, not git): +`~/.config/coulomb-social/auth/exports/corpus-202608131015057/` — +71 spaces, 109 articles; sample ADR-0004 tree `trees/reichelag/` with 30 +pages + 29 visuals (~1.6 MB). Progress note: +`docs/migration/export-progress-2026-08-13.md`. Script: +`scripts/export-bubble-corpus.mjs`. **T02** largely unblocked for mapping; +**T04** next when PageOps thin-git landing pad exists (CSOC-WP-0006). ## Feature cut decision (human gate)