Advance Bubble export: corpus walk and ADR-0004 sample tree
Add corpus exporter (session + ES capture), record 71 spaces / 109 articles and local reichelag markdown tree with visuals. Redacted progress only in git.
This commit is contained in:
parent
440719b9eb
commit
a0fd3cb7ea
5 changed files with 429 additions and 5 deletions
|
|
@ -6,6 +6,7 @@ CSOC-WP-0001-T02/T04 home for Bubble → rebuild mapping and rehearsal notes.
|
|||
|-----|--------|
|
||||
| `bubble-auth-and-export.md` | **Safe read access for agents** (session file; no secrets in chat) |
|
||||
| `field-map-from-session-2026-08-13.md` | **Field map from live session export** (article/space/user) |
|
||||
| `export-progress-2026-08-13.md` | Corpus counts + sample tree path (no private bodies) |
|
||||
| `schema-mapping-sketch-2026-08-12.md` | Provisional mapping (supersede with field map) |
|
||||
| *(later)* field-level map | After authenticated dump |
|
||||
| *(later)* rehearsal log | Import of one space to thin git tree (ADR-0004) |
|
||||
|
|
|
|||
82
docs/migration/export-progress-2026-08-13.md
Normal file
82
docs/migration/export-progress-2026-08-13.md
Normal file
|
|
@ -0,0 +1,82 @@
|
|||
# Export progress — 2026-08-13
|
||||
|
||||
| Field | Value |
|
||||
|-------|--------|
|
||||
| Workplan | CSOC-WP-0001-T02 |
|
||||
| Auth | Local Playwright session (XDG; not in git) |
|
||||
| Local corpus | `~/.config/coulomb-social/auth/exports/corpus-202608131015057/` |
|
||||
|
||||
## Results (no content bodies in git)
|
||||
|
||||
| Metric | Count |
|
||||
|--------|------:|
|
||||
| Spaces | 71 |
|
||||
| Articles (unique) | 109 |
|
||||
| Spaces with ≥1 article in dump | 70 |
|
||||
| Sample tree pages | 30 (29 articles + index) |
|
||||
| Sample visuals downloaded | 29 (~1.6 MB) |
|
||||
|
||||
### Articles per space (histogram)
|
||||
|
||||
| Articles in space | # spaces |
|
||||
|------------------:|---------:|
|
||||
| 0 | 1 |
|
||||
| 1 | 62 |
|
||||
| 2 | 4 |
|
||||
| 3 | 2 |
|
||||
| 4 | 1 |
|
||||
| 29 | 1 (sample) |
|
||||
|
||||
Most spaces are thin (one home/article). One dense space was used as the
|
||||
**rehearsal tree** (local only).
|
||||
|
||||
## Sample tree shape (ADR-0004)
|
||||
|
||||
```text
|
||||
trees/<space-slug>/
|
||||
pages/
|
||||
index.md # space landing
|
||||
<article-slug>.md # title/abstractor/visual + body
|
||||
assets/
|
||||
<article-slug>-visual.* # logo_image downloads
|
||||
```
|
||||
|
||||
Frontmatter keys (per page):
|
||||
|
||||
```yaml
|
||||
id: bubble:<id>
|
||||
title: "…" # nomer_text
|
||||
abstractor: "…" # abstractor_text
|
||||
visual: "assets/…" # logo_image → local file
|
||||
space: "<slug>"
|
||||
bubble_type: article
|
||||
archived: false
|
||||
parent: "" # context_custom_article when set
|
||||
```
|
||||
|
||||
## Methods used
|
||||
|
||||
1. Seed from first session network capture (ES msearch/mget responses).
|
||||
2. UI walk: open named spaces on `/vw_spaces` then `/vw_pages` + scroll (Bubble ES
|
||||
request bodies are **encrypted** — cannot craft free-form API queries).
|
||||
3. Emit JSON corpus + markdown tree under XDG only.
|
||||
|
||||
## Limits
|
||||
|
||||
- Article coverage beyond the “active” space is incomplete (many spaces only
|
||||
have 1 article in the dump; more may exist server-side).
|
||||
- Space **display title** is not a first-class field on `custom.space` (only
|
||||
`Slug`); UI names come from related articles / presentation.
|
||||
- Dense sample content is **private** — tree stays out of git.
|
||||
|
||||
## Next
|
||||
|
||||
1. Optional: longer crawl / pagination to fill more per-space articles.
|
||||
2. **T04 rehearsal:** PageOps import of local tree into app content dir (CSOC-WP-0006).
|
||||
3. Member → NetKingdom identity mapping for membership lists (no passwords).
|
||||
|
||||
## Related
|
||||
|
||||
- `docs/migration/field-map-from-session-2026-08-13.md`
|
||||
- `docs/migration/bubble-auth-and-export.md`
|
||||
- ADR-0003, ADR-0004
|
||||
|
|
@ -140,9 +140,9 @@ archived: false
|
|||
|
||||
## Next actions
|
||||
|
||||
1. Extend exporter to walk **all spaces** the user can access and dump article JSON redacted/normalized into a private corpus.
|
||||
2. Download visuals for sample space.
|
||||
3. Emit one space as ADR-0004 tree (rehearsal dry-run for T04).
|
||||
1. ~~Extend exporter / sample tree~~ — see `export-progress-2026-08-13.md` + `scripts/export-bubble-corpus.mjs`.
|
||||
2. Optional: deeper crawl for spaces with only 1 article if more body pages exist.
|
||||
3. **T04:** import local sample tree via PageOps (CSOC-WP-0006).
|
||||
4. Keep raw export **only** under XDG auth dir.
|
||||
|
||||
## Privacy
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue