Advance Bubble export: corpus walk and ADR-0004 sample tree

Add corpus exporter (session + ES capture), record 71 spaces / 109 articles
and local reichelag markdown tree with visuals. Redacted progress only in git.
This commit is contained in:
tegwick 2026-08-13 12:24:13 +02:00
parent 440719b9eb
commit a0fd3cb7ea
5 changed files with 429 additions and 5 deletions

View file

@ -6,6 +6,7 @@ CSOC-WP-0001-T02/T04 home for Bubble → rebuild mapping and rehearsal notes.
|-----|--------|
| `bubble-auth-and-export.md` | **Safe read access for agents** (session file; no secrets in chat) |
| `field-map-from-session-2026-08-13.md` | **Field map from live session export** (article/space/user) |
| `export-progress-2026-08-13.md` | Corpus counts + sample tree path (no private bodies) |
| `schema-mapping-sketch-2026-08-12.md` | Provisional mapping (supersede with field map) |
| *(later)* field-level map | After authenticated dump |
| *(later)* rehearsal log | Import of one space to thin git tree (ADR-0004) |

View file

@ -0,0 +1,82 @@
# Export progress — 2026-08-13
| Field | Value |
|-------|--------|
| Workplan | CSOC-WP-0001-T02 |
| Auth | Local Playwright session (XDG; not in git) |
| Local corpus | `~/.config/coulomb-social/auth/exports/corpus-202608131015057/` |
## Results (no content bodies in git)
| Metric | Count |
|--------|------:|
| Spaces | 71 |
| Articles (unique) | 109 |
| Spaces with ≥1 article in dump | 70 |
| Sample tree pages | 30 (29 articles + index) |
| Sample visuals downloaded | 29 (~1.6MB) |
### Articles per space (histogram)
| Articles in space | # spaces |
|------------------:|---------:|
| 0 | 1 |
| 1 | 62 |
| 2 | 4 |
| 3 | 2 |
| 4 | 1 |
| 29 | 1 (sample) |
Most spaces are thin (one home/article). One dense space was used as the
**rehearsal tree** (local only).
## Sample tree shape (ADR-0004)
```text
trees/<space-slug>/
pages/
index.md # space landing
<article-slug>.md # title/abstractor/visual + body
assets/
<article-slug>-visual.* # logo_image downloads
```
Frontmatter keys (per page):
```yaml
id: bubble:<id>
title: "…" # nomer_text
abstractor: "…" # abstractor_text
visual: "assets/…" # logo_image → local file
space: "<slug>"
bubble_type: article
archived: false
parent: "" # context_custom_article when set
```
## Methods used
1. Seed from first session network capture (ES msearch/mget responses).
2. UI walk: open named spaces on `/vw_spaces` then `/vw_pages` + scroll (Bubble ES
request bodies are **encrypted** — cannot craft free-form API queries).
3. Emit JSON corpus + markdown tree under XDG only.
## Limits
- Article coverage beyond the “active” space is incomplete (many spaces only
have 1 article in the dump; more may exist server-side).
- Space **display title** is not a first-class field on `custom.space` (only
`Slug`); UI names come from related articles / presentation.
- Dense sample content is **private** — tree stays out of git.
## Next
1. Optional: longer crawl / pagination to fill more per-space articles.
2. **T04 rehearsal:** PageOps import of local tree into app content dir (CSOC-WP-0006).
3. Member → NetKingdom identity mapping for membership lists (no passwords).
## Related
- `docs/migration/field-map-from-session-2026-08-13.md`
- `docs/migration/bubble-auth-and-export.md`
- ADR-0003, ADR-0004

View file

@ -140,9 +140,9 @@ archived: false
## Next actions
1. Extend exporter to walk **all spaces** the user can access and dump article JSON redacted/normalized into a private corpus.
2. Download visuals for sample space.
3. Emit one space as ADR-0004 tree (rehearsal dry-run for T04).
1. ~~Extend exporter / sample tree~~ — see `export-progress-2026-08-13.md` + `scripts/export-bubble-corpus.mjs`.
2. Optional: deeper crawl for spaces with only 1 article if more body pages exist.
3. **T04:** import local sample tree via PageOps (CSOC-WP-0006).
4. Keep raw export **only** under XDG auth dir.
## Privacy