Add safe Bubble session export path for CSOC-WP-0001-T02
Document XDG/OpenBao credential custody (no secrets in chat) and add Playwright inventory export using storage-state for automated read access.
This commit is contained in:
parent
638166cff4
commit
a160ad400b
5 changed files with 390 additions and 1 deletions
1
.gitignore
vendored
1
.gitignore
vendored
|
|
@ -8,6 +8,7 @@
|
||||||
.auth/
|
.auth/
|
||||||
**/storage-state.json
|
**/storage-state.json
|
||||||
**/*cookie*.txt
|
**/*cookie*.txt
|
||||||
|
docs/migration/samples/_local/
|
||||||
scripts/node_modules/
|
scripts/node_modules/
|
||||||
scripts/package-lock.json
|
scripts/package-lock.json
|
||||||
|
|
||||||
|
|
|
||||||
127
docs/migration/bubble-auth-and-export.md
Normal file
127
docs/migration/bubble-auth-and-export.md
Normal file
|
|
@ -0,0 +1,127 @@
|
||||||
|
# Bubble auth for automated export (CSOC-WP-0001-T02)
|
||||||
|
|
||||||
|
## Do not put credentials in chat or git
|
||||||
|
|
||||||
|
| Artefact | Where it belongs |
|
||||||
|
|----------|------------------|
|
||||||
|
| Email / password | Local `0600` files and/or OpenBao — **never** chat, PRs, workplans |
|
||||||
|
| Playwright `storage-state.json` | `~/.config/coulomb-social/auth/` mode `0600` (gitignored) |
|
||||||
|
| Export dumps | Local under auth dir or `docs/migration/samples/_local/` (gitignored) |
|
||||||
|
|
||||||
|
Agents will **not** accept a password pasted into the conversation. That would
|
||||||
|
land secrets in session logs.
|
||||||
|
|
||||||
|
## What a member session can do
|
||||||
|
|
||||||
|
With **your coulomb.social member** login, automation can:
|
||||||
|
|
||||||
|
- Open authenticated Bubble routes (`/vw_pages`, `/vw_spaces`, …)
|
||||||
|
- Capture HTML/text/screenshots for mapping
|
||||||
|
- Intercept browser network calls (sometimes useful for JSON payloads)
|
||||||
|
|
||||||
|
A **member** session usually **cannot**:
|
||||||
|
|
||||||
|
- Full Bubble **editor** data-type field export
|
||||||
|
- Admin “export all data” unless your account is an app admin with that power
|
||||||
|
|
||||||
|
If you need full schema dumps, use Bubble editor as the same human, or grant
|
||||||
|
editor access to a dedicated automation identity (still via OpenBao, not chat).
|
||||||
|
|
||||||
|
## Path A — Interactive session (recommended if MFA/CAPTCHA)
|
||||||
|
|
||||||
|
On the workstation:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd ~/coulomb-social
|
||||||
|
./scripts/capture-auth-state.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
1. Browser opens `https://coulomb.social`.
|
||||||
|
2. Log in as **you**.
|
||||||
|
3. Open a real app surface (e.g. Research / Pages — not only the landing root).
|
||||||
|
4. Press **Enter** in the terminal.
|
||||||
|
|
||||||
|
Session file:
|
||||||
|
|
||||||
|
```text
|
||||||
|
~/.config/coulomb-social/auth/storage-state.json # mode 0600
|
||||||
|
```
|
||||||
|
|
||||||
|
Probe (should **not** stay on bare `/` if login worked):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd ~/coulomb-social/scripts
|
||||||
|
node ./probe-auth-session.mjs
|
||||||
|
# expect something like …/vw_pages or another app path
|
||||||
|
```
|
||||||
|
|
||||||
|
Then:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./scripts/export-bubble-session.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
## Path B — Password files (only if no interactive MFA)
|
||||||
|
|
||||||
|
You write secrets **yourself** on disk (agent never sees the values in chat):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
AUTH="$HOME/.config/coulomb-social/auth"
|
||||||
|
mkdir -p "$AUTH"
|
||||||
|
chmod 700 "$AUTH"
|
||||||
|
|
||||||
|
# Replace with your real values — run in your terminal only
|
||||||
|
printf '%s' 'your@email.example' > "$AUTH/email"
|
||||||
|
printf '%s' 'your-password-here' > "$AUTH/password"
|
||||||
|
chmod 600 "$AUTH/email" "$AUTH/password"
|
||||||
|
|
||||||
|
cd ~/coulomb-social
|
||||||
|
./scripts/capture-auth-state.sh --from-password-files
|
||||||
|
# If Bubble shows MFA/CAPTCHA, fall back to Path A.
|
||||||
|
```
|
||||||
|
|
||||||
|
Optional OpenBao (when CCR/policy applied — see `docs/design-extract-auth.md`):
|
||||||
|
|
||||||
|
```text
|
||||||
|
tenants/binky/coulomb-social/bubble-member → EMAIL, PASSWORD
|
||||||
|
```
|
||||||
|
|
||||||
|
## Path C — Keep session warm for agents
|
||||||
|
|
||||||
|
Once `storage-state.json` exists and probe looks authenticated:
|
||||||
|
|
||||||
|
1. Tell the agent: *“Session is captured under the default XDG auth path; run export.”*
|
||||||
|
2. Do **not** paste the file contents.
|
||||||
|
3. Re-run `capture-auth-state.sh` when the session expires (probe lands on `/` only).
|
||||||
|
|
||||||
|
Agents with local filesystem access can then run export without further human
|
||||||
|
login **until cookies die**.
|
||||||
|
|
||||||
|
## Export outputs
|
||||||
|
|
||||||
|
Default dump directory (not for commit of raw HTML):
|
||||||
|
|
||||||
|
```text
|
||||||
|
~/.config/coulomb-social/auth/exports/YYYYMMDD-HHMMSS/
|
||||||
|
probe-url.txt
|
||||||
|
routes/
|
||||||
|
network/
|
||||||
|
SUMMARY.md
|
||||||
|
```
|
||||||
|
|
||||||
|
Redacted notes for the repo go under `docs/migration/` after review.
|
||||||
|
|
||||||
|
## Security checklist
|
||||||
|
|
||||||
|
- [ ] Secrets only in `~/.config/coulomb-social/auth/` (700/600)
|
||||||
|
- [ ] No password in chat, Slack, or git
|
||||||
|
- [ ] No `storage-state.json` committed (`.gitignore`)
|
||||||
|
- [ ] Prefer a **dedicated** Bubble user with least privilege if this becomes long-lived automation
|
||||||
|
- [ ] Rotate password if it was ever pasted into an unsafe channel
|
||||||
|
|
||||||
|
## Related
|
||||||
|
|
||||||
|
- `docs/design-extract-auth.md` — designlang / OpenBao detail
|
||||||
|
- `scripts/capture-auth-state.sh`
|
||||||
|
- `scripts/export-bubble-session.sh`
|
||||||
|
- CSOC-WP-0001-T02
|
||||||
248
scripts/export-bubble-inventory.mjs
Executable file
248
scripts/export-bubble-inventory.mjs
Executable file
|
|
@ -0,0 +1,248 @@
|
||||||
|
#!/usr/bin/env node
|
||||||
|
/**
|
||||||
|
* Authenticated Bubble inventory export for CSOC-WP-0001-T02.
|
||||||
|
*
|
||||||
|
* Uses Playwright storageState (never prints secrets). Captures routes, page
|
||||||
|
* text/HTML samples, and network JSON-ish responses for offline mapping.
|
||||||
|
*
|
||||||
|
* Usage:
|
||||||
|
* ./scripts/capture-auth-state.sh # once
|
||||||
|
* node ./scripts/export-bubble-inventory.mjs
|
||||||
|
*
|
||||||
|
* Env:
|
||||||
|
* COULOMB_SOCIAL_STORAGE_STATE default ~/.config/coulomb-social/auth/storage-state.json
|
||||||
|
* COULOMB_SOCIAL_URL default https://coulomb.social
|
||||||
|
* COULOMB_SOCIAL_EXPORT_DIR default ~/.config/coulomb-social/auth/exports/<stamp>
|
||||||
|
*/
|
||||||
|
import { chromium } from "playwright";
|
||||||
|
import {
|
||||||
|
mkdirSync,
|
||||||
|
writeFileSync,
|
||||||
|
chmodSync,
|
||||||
|
existsSync,
|
||||||
|
appendFileSync,
|
||||||
|
} from "node:fs";
|
||||||
|
import { homedir } from "node:os";
|
||||||
|
import { join } from "node:path";
|
||||||
|
|
||||||
|
const BASE_URL = process.env.COULOMB_SOCIAL_URL || "https://coulomb.social";
|
||||||
|
const AUTH_DIR =
|
||||||
|
process.env.COULOMB_SOCIAL_AUTH_DIR ||
|
||||||
|
join(homedir(), ".config", "coulomb-social", "auth");
|
||||||
|
const STATE_PATH =
|
||||||
|
process.env.COULOMB_SOCIAL_STORAGE_STATE ||
|
||||||
|
join(AUTH_DIR, "storage-state.json");
|
||||||
|
|
||||||
|
const ROUTES = [
|
||||||
|
"/",
|
||||||
|
"/vw_pages",
|
||||||
|
"/vw_spaces",
|
||||||
|
"/vw_chunks",
|
||||||
|
"/vw_search",
|
||||||
|
"/vw_feed",
|
||||||
|
"/vw_invites",
|
||||||
|
"/vw_likes",
|
||||||
|
"/vw_new",
|
||||||
|
"/reset_pw",
|
||||||
|
];
|
||||||
|
|
||||||
|
function stamp() {
|
||||||
|
const d = new Date();
|
||||||
|
const p = (n) => String(n).padStart(2, "0");
|
||||||
|
return (
|
||||||
|
d.getUTCFullYear() +
|
||||||
|
p(d.getUTCMonth() + 1) +
|
||||||
|
p(d.getUTCDate()) +
|
||||||
|
"-" +
|
||||||
|
p(d.getUTCHours()) +
|
||||||
|
p(d.getUTCMinutes()) +
|
||||||
|
p(d.getUTCSeconds())
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
function ensureDir(path, mode = 0o700) {
|
||||||
|
mkdirSync(path, { recursive: true, mode });
|
||||||
|
try {
|
||||||
|
chmodSync(path, mode);
|
||||||
|
} catch {
|
||||||
|
/* best effort */
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function writeText(path, text) {
|
||||||
|
writeFileSync(path, text, { mode: 0o600 });
|
||||||
|
try {
|
||||||
|
chmodSync(path, 0o600);
|
||||||
|
} catch {
|
||||||
|
/* best effort */
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function main() {
|
||||||
|
if (!existsSync(STATE_PATH)) {
|
||||||
|
console.error(`ERROR: No session file at ${STATE_PATH}`);
|
||||||
|
console.error("Capture a session first (do not paste passwords into chat):");
|
||||||
|
console.error(" ./scripts/capture-auth-state.sh");
|
||||||
|
console.error("Docs: docs/migration/bubble-auth-and-export.md");
|
||||||
|
process.exit(1);
|
||||||
|
}
|
||||||
|
|
||||||
|
const outRoot =
|
||||||
|
process.env.COULOMB_SOCIAL_EXPORT_DIR ||
|
||||||
|
join(AUTH_DIR, "exports", stamp());
|
||||||
|
const routesDir = join(outRoot, "routes");
|
||||||
|
const netDir = join(outRoot, "network");
|
||||||
|
ensureDir(outRoot);
|
||||||
|
ensureDir(routesDir);
|
||||||
|
ensureDir(netDir);
|
||||||
|
|
||||||
|
console.log(`Session: ${STATE_PATH} (present; not printed)`);
|
||||||
|
console.log(`Export → ${outRoot}`);
|
||||||
|
|
||||||
|
const networkLog = [];
|
||||||
|
const browser = await chromium.launch({ headless: true });
|
||||||
|
try {
|
||||||
|
const context = await browser.newContext({ storageState: STATE_PATH });
|
||||||
|
const page = await context.newPage();
|
||||||
|
|
||||||
|
page.on("response", async (response) => {
|
||||||
|
try {
|
||||||
|
const url = response.url();
|
||||||
|
const ct = (response.headers()["content-type"] || "").toLowerCase();
|
||||||
|
const status = response.status();
|
||||||
|
if (status < 200 || status >= 400) return;
|
||||||
|
const interesting =
|
||||||
|
ct.includes("json") ||
|
||||||
|
url.includes("/api/") ||
|
||||||
|
url.includes("elasticsearch") ||
|
||||||
|
url.includes("userdata") ||
|
||||||
|
url.includes("obj");
|
||||||
|
if (!interesting) return;
|
||||||
|
let body = "";
|
||||||
|
try {
|
||||||
|
body = await response.text();
|
||||||
|
} catch {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if (!body || body.length > 2_000_000) return;
|
||||||
|
const entry = {
|
||||||
|
url: url.slice(0, 500),
|
||||||
|
status,
|
||||||
|
contentType: ct.slice(0, 120),
|
||||||
|
bytes: body.length,
|
||||||
|
at: new Date().toISOString(),
|
||||||
|
};
|
||||||
|
networkLog.push(entry);
|
||||||
|
const safe = String(networkLog.length).padStart(3, "0");
|
||||||
|
writeText(join(netDir, `${safe}.meta.json`), JSON.stringify(entry, null, 2));
|
||||||
|
writeText(join(netDir, `${safe}.body.txt`), body);
|
||||||
|
} catch {
|
||||||
|
/* ignore single response failures */
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
// Warm session
|
||||||
|
await page.goto(BASE_URL, { waitUntil: "domcontentloaded", timeout: 60_000 });
|
||||||
|
await page.waitForTimeout(2500);
|
||||||
|
const probeUrl = page.url();
|
||||||
|
const probeTitle = await page.title().catch(() => "");
|
||||||
|
writeText(
|
||||||
|
join(outRoot, "probe.txt"),
|
||||||
|
`url=${probeUrl}\ntitle=${probeTitle}\n`,
|
||||||
|
);
|
||||||
|
console.log(`Probe: ${probeUrl} — ${probeTitle}`);
|
||||||
|
try {
|
||||||
|
const path = new URL(probeUrl).pathname;
|
||||||
|
if (path === "/" || path === "") {
|
||||||
|
console.warn(
|
||||||
|
"WARNING: Still on site root — session may be guest/expired. Re-run capture-auth-state.sh",
|
||||||
|
);
|
||||||
|
}
|
||||||
|
} catch {
|
||||||
|
/* ignore */
|
||||||
|
}
|
||||||
|
|
||||||
|
const summary = [];
|
||||||
|
summary.push(`# Bubble export ${stamp()}`);
|
||||||
|
summary.push("");
|
||||||
|
summary.push(`- base: ${BASE_URL}`);
|
||||||
|
summary.push(`- probe: ${probeUrl}`);
|
||||||
|
summary.push(`- title: ${probeTitle}`);
|
||||||
|
summary.push(`- session file: present (path not echoed in summary for safety)`);
|
||||||
|
summary.push("");
|
||||||
|
summary.push("## Routes");
|
||||||
|
summary.push("");
|
||||||
|
|
||||||
|
for (const route of ROUTES) {
|
||||||
|
const url = route.startsWith("http") ? route : `${BASE_URL.replace(/\/$/, "")}${route}`;
|
||||||
|
const slug = route.replace(/\W+/g, "_").replace(/^_|_$/g, "") || "root";
|
||||||
|
console.log(` GET ${route}`);
|
||||||
|
try {
|
||||||
|
const resp = await page.goto(url, {
|
||||||
|
waitUntil: "domcontentloaded",
|
||||||
|
timeout: 60_000,
|
||||||
|
});
|
||||||
|
await page.waitForTimeout(3500);
|
||||||
|
const finalUrl = page.url();
|
||||||
|
const title = await page.title().catch(() => "");
|
||||||
|
const status = resp?.status() ?? 0;
|
||||||
|
const html = await page.content();
|
||||||
|
const text = await page.evaluate(() => document.body?.innerText || "");
|
||||||
|
writeText(join(routesDir, `${slug}.html`), html);
|
||||||
|
writeText(join(routesDir, `${slug}.txt`), text);
|
||||||
|
writeText(
|
||||||
|
join(routesDir, `${slug}.meta.json`),
|
||||||
|
JSON.stringify(
|
||||||
|
{ route, url, finalUrl, title, status, htmlBytes: html.length, textBytes: text.length },
|
||||||
|
null,
|
||||||
|
2,
|
||||||
|
),
|
||||||
|
);
|
||||||
|
summary.push(
|
||||||
|
`- \`${route}\` → ${status} ${title} (${text.length} chars text, final ${finalUrl})`,
|
||||||
|
);
|
||||||
|
} catch (err) {
|
||||||
|
summary.push(`- \`${route}\` → ERROR ${err.message}`);
|
||||||
|
console.warn(` failed: ${err.message}`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Public meta (no auth needed, still useful)
|
||||||
|
try {
|
||||||
|
const metaResp = await page.request.get(`${BASE_URL.replace(/\/$/, "")}/api/1.1/meta`);
|
||||||
|
const metaText = await metaResp.text();
|
||||||
|
writeText(join(outRoot, "api-meta.json"), metaText);
|
||||||
|
summary.push("");
|
||||||
|
summary.push("## API meta");
|
||||||
|
summary.push(`- status ${metaResp.status()} (${metaText.length} bytes)`);
|
||||||
|
} catch (err) {
|
||||||
|
summary.push(`- api meta ERROR ${err.message}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
writeText(
|
||||||
|
join(outRoot, "network-index.json"),
|
||||||
|
JSON.stringify(networkLog, null, 2),
|
||||||
|
);
|
||||||
|
summary.push("");
|
||||||
|
summary.push("## Network captures");
|
||||||
|
summary.push(`- ${networkLog.length} JSON/API-ish responses under network/`);
|
||||||
|
summary.push("");
|
||||||
|
summary.push("## Next");
|
||||||
|
summary.push("- Review routes/*.txt for Title/Abstractor/Visual/copy cues");
|
||||||
|
summary.push("- Map into docs/migration/schema-mapping (redacted notes only in git)");
|
||||||
|
summary.push("- Do not commit this export directory");
|
||||||
|
summary.push("");
|
||||||
|
|
||||||
|
writeText(join(outRoot, "SUMMARY.md"), summary.join("\n"));
|
||||||
|
console.log("");
|
||||||
|
console.log(`Done. Summary: ${join(outRoot, "SUMMARY.md")}`);
|
||||||
|
console.log(`Network hits: ${networkLog.length}`);
|
||||||
|
} finally {
|
||||||
|
await browser.close();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
main().catch((err) => {
|
||||||
|
console.error(err.message || err);
|
||||||
|
process.exit(1);
|
||||||
|
});
|
||||||
13
scripts/export-bubble-session.sh
Executable file
13
scripts/export-bubble-session.sh
Executable file
|
|
@ -0,0 +1,13 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
# Authenticated Bubble inventory export (CSOC-WP-0001-T02).
|
||||||
|
# Uses storage-state only — never prints secrets.
|
||||||
|
set -euo pipefail
|
||||||
|
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
ROOT="$(cd "$DIR/.." && pwd)"
|
||||||
|
cd "$DIR"
|
||||||
|
if [[ ! -d node_modules/playwright ]]; then
|
||||||
|
echo "Installing playwright in scripts/ …"
|
||||||
|
npm install
|
||||||
|
npx playwright install chromium
|
||||||
|
fi
|
||||||
|
exec node ./export-bubble-inventory.mjs "$@"
|
||||||
|
|
@ -8,7 +8,7 @@ status: active
|
||||||
owner: bernd
|
owner: bernd
|
||||||
topic_slug: coulomb-social
|
topic_slug: coulomb-social
|
||||||
created: "2026-08-09"
|
created: "2026-08-09"
|
||||||
updated: "2026-08-12"
|
updated: "2026-08-13"
|
||||||
origin: residual
|
origin: residual
|
||||||
origin_ref: the-custodian/docs/coulomb-social-rebuild-seed.md
|
origin_ref: the-custodian/docs/coulomb-social-rebuild-seed.md
|
||||||
state_hub_workstream_id: "ecefaeef-908a-4072-ae34-6406862d4cad"
|
state_hub_workstream_id: "ecefaeef-908a-4072-ae34-6406862d4cad"
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue