Add safe Bubble session export path for CSOC-WP-0001-T02
Document XDG/OpenBao credential custody (no secrets in chat) and add Playwright inventory export using storage-state for automated read access.
This commit is contained in:
parent
638166cff4
commit
a160ad400b
5 changed files with 390 additions and 1 deletions
1
.gitignore
vendored
1
.gitignore
vendored
|
|
@ -8,6 +8,7 @@
|
|||
.auth/
|
||||
**/storage-state.json
|
||||
**/*cookie*.txt
|
||||
docs/migration/samples/_local/
|
||||
scripts/node_modules/
|
||||
scripts/package-lock.json
|
||||
|
||||
|
|
|
|||
127
docs/migration/bubble-auth-and-export.md
Normal file
127
docs/migration/bubble-auth-and-export.md
Normal file
|
|
@ -0,0 +1,127 @@
|
|||
# Bubble auth for automated export (CSOC-WP-0001-T02)
|
||||
|
||||
## Do not put credentials in chat or git
|
||||
|
||||
| Artefact | Where it belongs |
|
||||
|----------|------------------|
|
||||
| Email / password | Local `0600` files and/or OpenBao — **never** chat, PRs, workplans |
|
||||
| Playwright `storage-state.json` | `~/.config/coulomb-social/auth/` mode `0600` (gitignored) |
|
||||
| Export dumps | Local under auth dir or `docs/migration/samples/_local/` (gitignored) |
|
||||
|
||||
Agents will **not** accept a password pasted into the conversation. That would
|
||||
land secrets in session logs.
|
||||
|
||||
## What a member session can do
|
||||
|
||||
With **your coulomb.social member** login, automation can:
|
||||
|
||||
- Open authenticated Bubble routes (`/vw_pages`, `/vw_spaces`, …)
|
||||
- Capture HTML/text/screenshots for mapping
|
||||
- Intercept browser network calls (sometimes useful for JSON payloads)
|
||||
|
||||
A **member** session usually **cannot**:
|
||||
|
||||
- Full Bubble **editor** data-type field export
|
||||
- Admin “export all data” unless your account is an app admin with that power
|
||||
|
||||
If you need full schema dumps, use Bubble editor as the same human, or grant
|
||||
editor access to a dedicated automation identity (still via OpenBao, not chat).
|
||||
|
||||
## Path A — Interactive session (recommended if MFA/CAPTCHA)
|
||||
|
||||
On the workstation:
|
||||
|
||||
```bash
|
||||
cd ~/coulomb-social
|
||||
./scripts/capture-auth-state.sh
|
||||
```
|
||||
|
||||
1. Browser opens `https://coulomb.social`.
|
||||
2. Log in as **you**.
|
||||
3. Open a real app surface (e.g. Research / Pages — not only the landing root).
|
||||
4. Press **Enter** in the terminal.
|
||||
|
||||
Session file:
|
||||
|
||||
```text
|
||||
~/.config/coulomb-social/auth/storage-state.json # mode 0600
|
||||
```
|
||||
|
||||
Probe (should **not** stay on bare `/` if login worked):
|
||||
|
||||
```bash
|
||||
cd ~/coulomb-social/scripts
|
||||
node ./probe-auth-session.mjs
|
||||
# expect something like …/vw_pages or another app path
|
||||
```
|
||||
|
||||
Then:
|
||||
|
||||
```bash
|
||||
./scripts/export-bubble-session.sh
|
||||
```
|
||||
|
||||
## Path B — Password files (only if no interactive MFA)
|
||||
|
||||
You write secrets **yourself** on disk (agent never sees the values in chat):
|
||||
|
||||
```bash
|
||||
AUTH="$HOME/.config/coulomb-social/auth"
|
||||
mkdir -p "$AUTH"
|
||||
chmod 700 "$AUTH"
|
||||
|
||||
# Replace with your real values — run in your terminal only
|
||||
printf '%s' 'your@email.example' > "$AUTH/email"
|
||||
printf '%s' 'your-password-here' > "$AUTH/password"
|
||||
chmod 600 "$AUTH/email" "$AUTH/password"
|
||||
|
||||
cd ~/coulomb-social
|
||||
./scripts/capture-auth-state.sh --from-password-files
|
||||
# If Bubble shows MFA/CAPTCHA, fall back to Path A.
|
||||
```
|
||||
|
||||
Optional OpenBao (when CCR/policy applied — see `docs/design-extract-auth.md`):
|
||||
|
||||
```text
|
||||
tenants/binky/coulomb-social/bubble-member → EMAIL, PASSWORD
|
||||
```
|
||||
|
||||
## Path C — Keep session warm for agents
|
||||
|
||||
Once `storage-state.json` exists and probe looks authenticated:
|
||||
|
||||
1. Tell the agent: *“Session is captured under the default XDG auth path; run export.”*
|
||||
2. Do **not** paste the file contents.
|
||||
3. Re-run `capture-auth-state.sh` when the session expires (probe lands on `/` only).
|
||||
|
||||
Agents with local filesystem access can then run export without further human
|
||||
login **until cookies die**.
|
||||
|
||||
## Export outputs
|
||||
|
||||
Default dump directory (not for commit of raw HTML):
|
||||
|
||||
```text
|
||||
~/.config/coulomb-social/auth/exports/YYYYMMDD-HHMMSS/
|
||||
probe-url.txt
|
||||
routes/
|
||||
network/
|
||||
SUMMARY.md
|
||||
```
|
||||
|
||||
Redacted notes for the repo go under `docs/migration/` after review.
|
||||
|
||||
## Security checklist
|
||||
|
||||
- [ ] Secrets only in `~/.config/coulomb-social/auth/` (700/600)
|
||||
- [ ] No password in chat, Slack, or git
|
||||
- [ ] No `storage-state.json` committed (`.gitignore`)
|
||||
- [ ] Prefer a **dedicated** Bubble user with least privilege if this becomes long-lived automation
|
||||
- [ ] Rotate password if it was ever pasted into an unsafe channel
|
||||
|
||||
## Related
|
||||
|
||||
- `docs/design-extract-auth.md` — designlang / OpenBao detail
|
||||
- `scripts/capture-auth-state.sh`
|
||||
- `scripts/export-bubble-session.sh`
|
||||
- CSOC-WP-0001-T02
|
||||
248
scripts/export-bubble-inventory.mjs
Executable file
248
scripts/export-bubble-inventory.mjs
Executable file
|
|
@ -0,0 +1,248 @@
|
|||
#!/usr/bin/env node
|
||||
/**
|
||||
* Authenticated Bubble inventory export for CSOC-WP-0001-T02.
|
||||
*
|
||||
* Uses Playwright storageState (never prints secrets). Captures routes, page
|
||||
* text/HTML samples, and network JSON-ish responses for offline mapping.
|
||||
*
|
||||
* Usage:
|
||||
* ./scripts/capture-auth-state.sh # once
|
||||
* node ./scripts/export-bubble-inventory.mjs
|
||||
*
|
||||
* Env:
|
||||
* COULOMB_SOCIAL_STORAGE_STATE default ~/.config/coulomb-social/auth/storage-state.json
|
||||
* COULOMB_SOCIAL_URL default https://coulomb.social
|
||||
* COULOMB_SOCIAL_EXPORT_DIR default ~/.config/coulomb-social/auth/exports/<stamp>
|
||||
*/
|
||||
import { chromium } from "playwright";
|
||||
import {
|
||||
mkdirSync,
|
||||
writeFileSync,
|
||||
chmodSync,
|
||||
existsSync,
|
||||
appendFileSync,
|
||||
} from "node:fs";
|
||||
import { homedir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
|
||||
const BASE_URL = process.env.COULOMB_SOCIAL_URL || "https://coulomb.social";
|
||||
const AUTH_DIR =
|
||||
process.env.COULOMB_SOCIAL_AUTH_DIR ||
|
||||
join(homedir(), ".config", "coulomb-social", "auth");
|
||||
const STATE_PATH =
|
||||
process.env.COULOMB_SOCIAL_STORAGE_STATE ||
|
||||
join(AUTH_DIR, "storage-state.json");
|
||||
|
||||
const ROUTES = [
|
||||
"/",
|
||||
"/vw_pages",
|
||||
"/vw_spaces",
|
||||
"/vw_chunks",
|
||||
"/vw_search",
|
||||
"/vw_feed",
|
||||
"/vw_invites",
|
||||
"/vw_likes",
|
||||
"/vw_new",
|
||||
"/reset_pw",
|
||||
];
|
||||
|
||||
function stamp() {
|
||||
const d = new Date();
|
||||
const p = (n) => String(n).padStart(2, "0");
|
||||
return (
|
||||
d.getUTCFullYear() +
|
||||
p(d.getUTCMonth() + 1) +
|
||||
p(d.getUTCDate()) +
|
||||
"-" +
|
||||
p(d.getUTCHours()) +
|
||||
p(d.getUTCMinutes()) +
|
||||
p(d.getUTCSeconds())
|
||||
);
|
||||
}
|
||||
|
||||
function ensureDir(path, mode = 0o700) {
|
||||
mkdirSync(path, { recursive: true, mode });
|
||||
try {
|
||||
chmodSync(path, mode);
|
||||
} catch {
|
||||
/* best effort */
|
||||
}
|
||||
}
|
||||
|
||||
function writeText(path, text) {
|
||||
writeFileSync(path, text, { mode: 0o600 });
|
||||
try {
|
||||
chmodSync(path, 0o600);
|
||||
} catch {
|
||||
/* best effort */
|
||||
}
|
||||
}
|
||||
|
||||
async function main() {
|
||||
if (!existsSync(STATE_PATH)) {
|
||||
console.error(`ERROR: No session file at ${STATE_PATH}`);
|
||||
console.error("Capture a session first (do not paste passwords into chat):");
|
||||
console.error(" ./scripts/capture-auth-state.sh");
|
||||
console.error("Docs: docs/migration/bubble-auth-and-export.md");
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const outRoot =
|
||||
process.env.COULOMB_SOCIAL_EXPORT_DIR ||
|
||||
join(AUTH_DIR, "exports", stamp());
|
||||
const routesDir = join(outRoot, "routes");
|
||||
const netDir = join(outRoot, "network");
|
||||
ensureDir(outRoot);
|
||||
ensureDir(routesDir);
|
||||
ensureDir(netDir);
|
||||
|
||||
console.log(`Session: ${STATE_PATH} (present; not printed)`);
|
||||
console.log(`Export → ${outRoot}`);
|
||||
|
||||
const networkLog = [];
|
||||
const browser = await chromium.launch({ headless: true });
|
||||
try {
|
||||
const context = await browser.newContext({ storageState: STATE_PATH });
|
||||
const page = await context.newPage();
|
||||
|
||||
page.on("response", async (response) => {
|
||||
try {
|
||||
const url = response.url();
|
||||
const ct = (response.headers()["content-type"] || "").toLowerCase();
|
||||
const status = response.status();
|
||||
if (status < 200 || status >= 400) return;
|
||||
const interesting =
|
||||
ct.includes("json") ||
|
||||
url.includes("/api/") ||
|
||||
url.includes("elasticsearch") ||
|
||||
url.includes("userdata") ||
|
||||
url.includes("obj");
|
||||
if (!interesting) return;
|
||||
let body = "";
|
||||
try {
|
||||
body = await response.text();
|
||||
} catch {
|
||||
return;
|
||||
}
|
||||
if (!body || body.length > 2_000_000) return;
|
||||
const entry = {
|
||||
url: url.slice(0, 500),
|
||||
status,
|
||||
contentType: ct.slice(0, 120),
|
||||
bytes: body.length,
|
||||
at: new Date().toISOString(),
|
||||
};
|
||||
networkLog.push(entry);
|
||||
const safe = String(networkLog.length).padStart(3, "0");
|
||||
writeText(join(netDir, `${safe}.meta.json`), JSON.stringify(entry, null, 2));
|
||||
writeText(join(netDir, `${safe}.body.txt`), body);
|
||||
} catch {
|
||||
/* ignore single response failures */
|
||||
}
|
||||
});
|
||||
|
||||
// Warm session
|
||||
await page.goto(BASE_URL, { waitUntil: "domcontentloaded", timeout: 60_000 });
|
||||
await page.waitForTimeout(2500);
|
||||
const probeUrl = page.url();
|
||||
const probeTitle = await page.title().catch(() => "");
|
||||
writeText(
|
||||
join(outRoot, "probe.txt"),
|
||||
`url=${probeUrl}\ntitle=${probeTitle}\n`,
|
||||
);
|
||||
console.log(`Probe: ${probeUrl} — ${probeTitle}`);
|
||||
try {
|
||||
const path = new URL(probeUrl).pathname;
|
||||
if (path === "/" || path === "") {
|
||||
console.warn(
|
||||
"WARNING: Still on site root — session may be guest/expired. Re-run capture-auth-state.sh",
|
||||
);
|
||||
}
|
||||
} catch {
|
||||
/* ignore */
|
||||
}
|
||||
|
||||
const summary = [];
|
||||
summary.push(`# Bubble export ${stamp()}`);
|
||||
summary.push("");
|
||||
summary.push(`- base: ${BASE_URL}`);
|
||||
summary.push(`- probe: ${probeUrl}`);
|
||||
summary.push(`- title: ${probeTitle}`);
|
||||
summary.push(`- session file: present (path not echoed in summary for safety)`);
|
||||
summary.push("");
|
||||
summary.push("## Routes");
|
||||
summary.push("");
|
||||
|
||||
for (const route of ROUTES) {
|
||||
const url = route.startsWith("http") ? route : `${BASE_URL.replace(/\/$/, "")}${route}`;
|
||||
const slug = route.replace(/\W+/g, "_").replace(/^_|_$/g, "") || "root";
|
||||
console.log(` GET ${route}`);
|
||||
try {
|
||||
const resp = await page.goto(url, {
|
||||
waitUntil: "domcontentloaded",
|
||||
timeout: 60_000,
|
||||
});
|
||||
await page.waitForTimeout(3500);
|
||||
const finalUrl = page.url();
|
||||
const title = await page.title().catch(() => "");
|
||||
const status = resp?.status() ?? 0;
|
||||
const html = await page.content();
|
||||
const text = await page.evaluate(() => document.body?.innerText || "");
|
||||
writeText(join(routesDir, `${slug}.html`), html);
|
||||
writeText(join(routesDir, `${slug}.txt`), text);
|
||||
writeText(
|
||||
join(routesDir, `${slug}.meta.json`),
|
||||
JSON.stringify(
|
||||
{ route, url, finalUrl, title, status, htmlBytes: html.length, textBytes: text.length },
|
||||
null,
|
||||
2,
|
||||
),
|
||||
);
|
||||
summary.push(
|
||||
`- \`${route}\` → ${status} ${title} (${text.length} chars text, final ${finalUrl})`,
|
||||
);
|
||||
} catch (err) {
|
||||
summary.push(`- \`${route}\` → ERROR ${err.message}`);
|
||||
console.warn(` failed: ${err.message}`);
|
||||
}
|
||||
}
|
||||
|
||||
// Public meta (no auth needed, still useful)
|
||||
try {
|
||||
const metaResp = await page.request.get(`${BASE_URL.replace(/\/$/, "")}/api/1.1/meta`);
|
||||
const metaText = await metaResp.text();
|
||||
writeText(join(outRoot, "api-meta.json"), metaText);
|
||||
summary.push("");
|
||||
summary.push("## API meta");
|
||||
summary.push(`- status ${metaResp.status()} (${metaText.length} bytes)`);
|
||||
} catch (err) {
|
||||
summary.push(`- api meta ERROR ${err.message}`);
|
||||
}
|
||||
|
||||
writeText(
|
||||
join(outRoot, "network-index.json"),
|
||||
JSON.stringify(networkLog, null, 2),
|
||||
);
|
||||
summary.push("");
|
||||
summary.push("## Network captures");
|
||||
summary.push(`- ${networkLog.length} JSON/API-ish responses under network/`);
|
||||
summary.push("");
|
||||
summary.push("## Next");
|
||||
summary.push("- Review routes/*.txt for Title/Abstractor/Visual/copy cues");
|
||||
summary.push("- Map into docs/migration/schema-mapping (redacted notes only in git)");
|
||||
summary.push("- Do not commit this export directory");
|
||||
summary.push("");
|
||||
|
||||
writeText(join(outRoot, "SUMMARY.md"), summary.join("\n"));
|
||||
console.log("");
|
||||
console.log(`Done. Summary: ${join(outRoot, "SUMMARY.md")}`);
|
||||
console.log(`Network hits: ${networkLog.length}`);
|
||||
} finally {
|
||||
await browser.close();
|
||||
}
|
||||
}
|
||||
|
||||
main().catch((err) => {
|
||||
console.error(err.message || err);
|
||||
process.exit(1);
|
||||
});
|
||||
13
scripts/export-bubble-session.sh
Executable file
13
scripts/export-bubble-session.sh
Executable file
|
|
@ -0,0 +1,13 @@
|
|||
#!/usr/bin/env bash
|
||||
# Authenticated Bubble inventory export (CSOC-WP-0001-T02).
|
||||
# Uses storage-state only — never prints secrets.
|
||||
set -euo pipefail
|
||||
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT="$(cd "$DIR/.." && pwd)"
|
||||
cd "$DIR"
|
||||
if [[ ! -d node_modules/playwright ]]; then
|
||||
echo "Installing playwright in scripts/ …"
|
||||
npm install
|
||||
npx playwright install chromium
|
||||
fi
|
||||
exec node ./export-bubble-inventory.mjs "$@"
|
||||
|
|
@ -8,7 +8,7 @@ status: active
|
|||
owner: bernd
|
||||
topic_slug: coulomb-social
|
||||
created: "2026-08-09"
|
||||
updated: "2026-08-12"
|
||||
updated: "2026-08-13"
|
||||
origin: residual
|
||||
origin_ref: the-custodian/docs/coulomb-social-rebuild-seed.md
|
||||
state_hub_workstream_id: "ecefaeef-908a-4072-ae34-6406862d4cad"
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue