An advanced markdown engine
tegwick
e3e5b8ecc1
Three coordinated changes that let the pipeline produce a clean chapter-by-chapter git history on long texts without archaeology after the fact. 1. Richer commit messages. `SourcePipeline._git_commit` now diffs the staged changes, buckets added files by output subdirectory (entities, evaluations, classifications, mappings, analyses, metrics, logs), and includes counts in the commit body. So `git log` reads "entities: +23, evaluations: +23" per chapter instead of the same generic blurb on every commit. Zero behaviour change when no output changed; falls back to the original message if the diff query fails. 2. --eval-after-source / --classify-after-source on `infospace process`. After a source's stages succeed, the pipeline identifies which entity files are *new* (set diff of entity slugs before vs after), loads their EntityMeta, and runs per-entity evaluation and/or classification scoped to just those slugs before the per-source git commit lands. Result: each chapter's commit is self-contained — extraction + evaluation + classification in one atomic unit. Gated behind explicit flags because the cost is real (LLM latency per chapter rather than amortised across one bulk batch). 3. `markitect infospace chapters` subcommand. Lists source files in canonical order with entity count, evaluated count, classified count, and mean per-entity score per source. Text or JSON output. Natural triage surface for long-text infospaces — spot chapters that under-extracted or evaluated poorly. Also: `docs/advanced-usage.md` gets a new "Systematic processing of long texts" section with the recommended flag combo and the tradeoff note on cost. 11 new unit tests cover the chapters command (text/json/no-sources), the process flag wiring (help + provider requirement), and the commit-body bucket logic. Full infospace+llm unit suite (315 tests) green; 3 pre-existing infospace failures unchanged. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|---|---|---|
| .claude | ||
| .github/workflows | ||
| .issues | ||
| .venv_old | ||
| _issue-tracking | ||
| agents | ||
| application | ||
| assets | ||
| capabilities | ||
| config | ||
| cost_notes | ||
| docs | ||
| domain | ||
| examples | ||
| guides | ||
| history | ||
| infrastructure | ||
| issue_tracker/cli | ||
| markitect | ||
| migrations/prompts | ||
| node_modules | ||
| reports | ||
| roadmap | ||
| scripts | ||
| services | ||
| src | ||
| testdata | ||
| tests | ||
| tools | ||
| wiki@8818df03d3 | ||
| .clinerules | ||
| .custodian-brief.md | ||
| .gitignore | ||
| .gitmodules | ||
| aliases.sh | ||
| asset_registry.json | ||
| CHANGELOG.md | ||
| CLAUDE.md | ||
| demo_plugin_integration.py | ||
| DEPENDENCIES.md | ||
| GUARDRAILS.md | ||
| install | ||
| install.py | ||
| install.sh | ||
| INTRODUCTION.md | ||
| Makefile | ||
| package-lock.json | ||
| package.json | ||
| pyproject.toml | ||
| pytest-timeout.ini | ||
| pytest.ini | ||
| SCOPE.md | ||
| test_asset_deployment.py | ||
| test_browser_ready.py | ||
| test_cli_integration.py | ||
| test_cli_plugin.md | ||
| test_cli_simple.py | ||
| test_cli_with_assets.py | ||
| test_complete_integration.py | ||
| test_integration.md | ||
| test_plugin_discovery.py | ||
| test_strict_mode.html | ||
| TODO.md | ||
MarkiTect Documentation
Welcome to the MarkiTect documentation. This directory contains comprehensive documentation for developers, users, and contributors.
Documentation Structure
📐 Architecture Documentation (architecture/)
Deep technical documentation about system design, performance, and implementation details.
- Capabilities Architecture - Critical: How capabilities work as independent git submodules and separation of concerns
- Caching System - Why and how MarkiTect's AST caching delivers 60-85% performance improvements
- Coming soon: Database Schema, CLI Architecture
👥 User Guides (user-guides/)
End-user documentation for working with MarkiTect CLI and features.
- Coming soon: Getting Started, Command Reference, Best Practices
🔧 Development Documentation (development/)
Documentation for contributors and developers extending MarkiTect.
- Coming soon: Contributing Guide, Testing Strategy, Release Process
Quick Links
For Users
- Installation & Setup
- Command Reference (coming soon)
- Performance Guide (coming soon)
For Developers
- Architecture Overview - System design and component relationships
- Development Setup - Local development environment
- API Documentation (coming soon)
Project Management
- Project Status - Current development status
- Roadmap - Strategic development plan
- Current Tasks - Task management using Keep a Todofile format
Key Concepts
Core Architecture Principles
- Parse Once, Use Many Times - AST caching for 60-85% performance improvement
- Convention Over Configuration - Sensible defaults with minimal setup
- Schema-Driven Processing - Structured markdown with validation
- Relational Metadata - Database-powered document relationships
Performance Philosophy
MarkiTect treats markdown documents as structured, queryable data rather than plain text. This approach enables:
- Lightning-fast document processing through intelligent caching
- Complex querying and relationship management
- Schema validation and consistency enforcement
- Scalable performance that grows with your content
Contributing to Documentation
Documentation follows the same quality standards as code:
- Clear Structure - Logical organization and navigation
- Practical Examples - Real-world usage patterns
- Performance Context - Why architectural decisions matter
- User-Focused - Written for the intended audience
Documentation Standards
- Use clear, concise language
- Include practical examples
- Explain the "why" behind design decisions
- Keep technical accuracy as the highest priority
- Update docs when changing functionality
This documentation is maintained alongside the codebase. For the most current information, always refer to the latest version in the repository.