markitect-main/TODO.md

205 lines
9.5 KiB
Markdown
Raw Normal View History

# Todofile
This is a "to do next" file, particularly useful to keep the human and a coding assistant in sync.
The format is based on [Keep a Todofile V0.0.1](https://coulomb.social/open/KeepaTodofile).
The structure organizes **future tasks** by their impact, just as a changelog organizes past changes by their impact.
***
## [Unreleased] - *Active Vibe-Coding State* 💡
This section is for tasks currently being discussed with or worked on by the coding assistant. These are the ephemeral, flow-of-thought tasks.
feat: implement Phase 3 - Schema-for-Schemas Metaschema Completed Phase 3 of the schema-of-schemas implementation with a comprehensive metaschema that validates all MarkiTect schema files against conventions and standards. Metaschema Implementation (schema-schema-v1.0.md - 650+ lines): - Validates core JSON Schema fields ($schema, $id, title, description) - Validates MarkiTect version field (SemVer: major.minor.patch) - Validates $id URL format (HTTPS with version path) - Validates MarkiTect extensions: - x-markitect-sections: section classifications and content rules - x-markitect-content-control: pattern and quality validation - x-markitect-metadata: status, authors, tags - x-markitect-source: loader metadata (auto-added) - Section classification validation (required, recommended, optional, discouraged, improper) - Content control pattern validation - Comprehensive documentation with examples and usage guides CLI Command (markitect schema-validate): - Validates schema files against metaschema - Supports both markdown and JSON schema files - Detailed error reporting with schema paths - Structure validation recommendations - Exit codes for CI/CD integration Test Coverage (tests/test_schema_metaschema.py - 12 tests, 100% passing): - Metaschema self-validation - Manpage schema validation - Required fields enforcement - Version format validation (valid and invalid cases) - $id format validation (valid and invalid cases) - Section classification validation - Complete schema with all extensions Validation Results: - ✅ Metaschema validates itself successfully - ✅ Manpage schema (v1.0.md) validates successfully - ⚠️ Terminology schema needs migration (missing version, incorrect $id) Progress Tracking: - Updated TODO.md with Phase 3 completion - Updated CHANGELOG.md with implementation details - Next: Phase 4 - Schema Migration 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-05 03:10:49 +01:00
### Schema-of-Schemas Implementation (Active - Phase 3)
feat: implement Phase 3 - Schema-for-Schemas Metaschema Completed Phase 3 of the schema-of-schemas implementation with a comprehensive metaschema that validates all MarkiTect schema files against conventions and standards. Metaschema Implementation (schema-schema-v1.0.md - 650+ lines): - Validates core JSON Schema fields ($schema, $id, title, description) - Validates MarkiTect version field (SemVer: major.minor.patch) - Validates $id URL format (HTTPS with version path) - Validates MarkiTect extensions: - x-markitect-sections: section classifications and content rules - x-markitect-content-control: pattern and quality validation - x-markitect-metadata: status, authors, tags - x-markitect-source: loader metadata (auto-added) - Section classification validation (required, recommended, optional, discouraged, improper) - Content control pattern validation - Comprehensive documentation with examples and usage guides CLI Command (markitect schema-validate): - Validates schema files against metaschema - Supports both markdown and JSON schema files - Detailed error reporting with schema paths - Structure validation recommendations - Exit codes for CI/CD integration Test Coverage (tests/test_schema_metaschema.py - 12 tests, 100% passing): - Metaschema self-validation - Manpage schema validation - Required fields enforcement - Version format validation (valid and invalid cases) - $id format validation (valid and invalid cases) - Section classification validation - Complete schema with all extensions Validation Results: - ✅ Metaschema validates itself successfully - ✅ Manpage schema (v1.0.md) validates successfully - ⚠️ Terminology schema needs migration (missing version, incorrect $id) Progress Tracking: - Updated TODO.md with Phase 3 completion - Updated CHANGELOG.md with implementation details - Next: Phase 4 - Schema Migration 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-05 03:10:49 +01:00
**Status:** Phase 3 - Schema-for-Schemas Metaschema (Completed ✅)
chore: establish schema-of-schemas workplan and reorganize roadmap This commit sets up the comprehensive workplan for implementing a markdown-first schema management system with naming conventions, versioning, and self-validation capabilities. ## Directory Reorganization - Renamed `todo/` → `roadmap/` for better organization - Created `roadmap/schema-of-schemas/` subdirectory - Moved schema management planning artifacts to dedicated directory ## Planning Artifacts Created ### Workplan & Documentation - **WORKPLAN.md** (19KB) - Comprehensive 6-phase implementation plan - **SCHEMA_MANAGEMENT_PROPOSAL.md** - Full analysis with 4 options - **SCHEMA_MANAGEMENT_SUMMARY.md** - Executive summary - **README.md** - Quick reference guide ### Example Schema - **examples/schemas/manpage-schema-v1.md** - Demonstrates markdown format ## Schema Management System Design ### Naming Convention **Format:** `{domain}-schema-v{major}.{minor}.md` **Examples:** - `manpage-schema-v1.0.md` - `terminology-schema-v1.0.md` - `api-documentation-schema-v1.0.md` ### Markdown-First Format Schemas will be markdown files with: - YAML frontmatter for metadata - Rich documentation sections - Embedded JSON schema in code block - Version history and examples ### Implementation Phases (8-10 days) **Phase 0:** Planning & Setup ✅ (0.5 days) - COMPLETE **Phase 1:** Filename Convention (1 day) - NEXT **Phase 2:** Markdown Loader (2-3 days) **Phase 3:** Schema-for-Schemas (2 days) **Phase 4:** Schema Migration (1-2 days) **Phase 5:** CLI & Documentation (1 day) **Phase 6:** Testing & Validation (1 day) ### Goals 1. ✅ Establish naming convention 2. ⏳ Implement filename validation 3. ⏳ Create markdown schema loader 4. ⏳ Build schema-for-schemas metaschema 5. ⏳ Migrate 5 existing schemas (remove 2 duplicates) 6. ⏳ Update CLI and documentation ## Updated Tracking ### TODO.md - Added Schema-of-Schemas as active work item - Documented Phase 1 tasks and timeline - Paused capability extraction work ### CHANGELOG.md - Added schema management system to [Unreleased] - Documented directory reorganization - Added "In Progress" section for current work ## Next Steps Begin Phase 1: 1. Implement schema_naming.py with validation 2. Add unit tests 3. Update CLI schema-ingest command 4. Create naming specification document ## Files Changed - CHANGELOG.md - Added unreleased schema management features - TODO.md - Updated active work tracking - roadmap/ - Reorganized from todo/ - roadmap/schema-of-schemas/ - New planning directory - examples/schemas/ - Example markdown schema 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-04 23:47:02 +01:00
**Workplan:** See `roadmap/schema-of-schemas/WORKPLAN.md`
**Current Goals:**
1. ✅ Establish naming convention: `{domain}-schema-v{major}.{minor}.md`
feat: implement Phase 2 - Markdown Schema Loader Completed Phase 2 of the schema-of-schemas implementation with full markdown schema support. This enables schemas to be authored as markdown files with rich documentation and embedded JSON schemas. Core Implementation (markitect/schema_loader.py): - MarkdownSchemaLoader class with comprehensive parsing capabilities - YAML frontmatter extraction with error handling - JSON code block extraction with section preference (## Schema Definition) - Metadata merging with x-markitect-source tracking - Schema saving with template support and round-trip capability - Helper methods: list_json_blocks(), validate_schema_structure() Test Coverage (tests/test_schema_loader.py): - 35 comprehensive unit tests (100% passing) - Tests for loading, parsing, saving, round-trip conversion - Edge case handling (empty files, binary files, malformed blocks) - Fixed binary file test to use invalid UTF-8 sequences Example Schema (markitect/schemas/manpage-schema-v1.0.md): - First markdown schema following naming convention - Complete manpage schema with frontmatter + documentation + JSON - Demonstrates section classification and content control - Shows proper structure for future schema authors Documentation (roadmap/schema-of-schemas/SCHEMA_LOADER_GUIDE.md): - Comprehensive user guide (600+ lines) - API reference with examples - Best practices and troubleshooting - Integration patterns for CLI and validator Progress Tracking: - Updated TODO.md with Phase 2 completion - Updated CHANGELOG.md with implementation details - Next: Phase 3 - Schema-for-Schemas Metaschema 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-05 00:02:15 +01:00
2. ✅ Implement filename validation logic
3. ✅ Create markdown schema loader
4. ✅ Create example markdown schema
feat: implement Phase 3 - Schema-for-Schemas Metaschema Completed Phase 3 of the schema-of-schemas implementation with a comprehensive metaschema that validates all MarkiTect schema files against conventions and standards. Metaschema Implementation (schema-schema-v1.0.md - 650+ lines): - Validates core JSON Schema fields ($schema, $id, title, description) - Validates MarkiTect version field (SemVer: major.minor.patch) - Validates $id URL format (HTTPS with version path) - Validates MarkiTect extensions: - x-markitect-sections: section classifications and content rules - x-markitect-content-control: pattern and quality validation - x-markitect-metadata: status, authors, tags - x-markitect-source: loader metadata (auto-added) - Section classification validation (required, recommended, optional, discouraged, improper) - Content control pattern validation - Comprehensive documentation with examples and usage guides CLI Command (markitect schema-validate): - Validates schema files against metaschema - Supports both markdown and JSON schema files - Detailed error reporting with schema paths - Structure validation recommendations - Exit codes for CI/CD integration Test Coverage (tests/test_schema_metaschema.py - 12 tests, 100% passing): - Metaschema self-validation - Manpage schema validation - Required fields enforcement - Version format validation (valid and invalid cases) - $id format validation (valid and invalid cases) - Section classification validation - Complete schema with all extensions Validation Results: - ✅ Metaschema validates itself successfully - ✅ Manpage schema (v1.0.md) validates successfully - ⚠️ Terminology schema needs migration (missing version, incorrect $id) Progress Tracking: - Updated TODO.md with Phase 3 completion - Updated CHANGELOG.md with implementation details - Next: Phase 4 - Schema Migration 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-05 03:10:49 +01:00
5. ✅ Build schema-for-schemas metaschema
6. ⏳ Migrate existing schemas to new format (Next: Phase 4)
chore: establish schema-of-schemas workplan and reorganize roadmap This commit sets up the comprehensive workplan for implementing a markdown-first schema management system with naming conventions, versioning, and self-validation capabilities. ## Directory Reorganization - Renamed `todo/` → `roadmap/` for better organization - Created `roadmap/schema-of-schemas/` subdirectory - Moved schema management planning artifacts to dedicated directory ## Planning Artifacts Created ### Workplan & Documentation - **WORKPLAN.md** (19KB) - Comprehensive 6-phase implementation plan - **SCHEMA_MANAGEMENT_PROPOSAL.md** - Full analysis with 4 options - **SCHEMA_MANAGEMENT_SUMMARY.md** - Executive summary - **README.md** - Quick reference guide ### Example Schema - **examples/schemas/manpage-schema-v1.md** - Demonstrates markdown format ## Schema Management System Design ### Naming Convention **Format:** `{domain}-schema-v{major}.{minor}.md` **Examples:** - `manpage-schema-v1.0.md` - `terminology-schema-v1.0.md` - `api-documentation-schema-v1.0.md` ### Markdown-First Format Schemas will be markdown files with: - YAML frontmatter for metadata - Rich documentation sections - Embedded JSON schema in code block - Version history and examples ### Implementation Phases (8-10 days) **Phase 0:** Planning & Setup ✅ (0.5 days) - COMPLETE **Phase 1:** Filename Convention (1 day) - NEXT **Phase 2:** Markdown Loader (2-3 days) **Phase 3:** Schema-for-Schemas (2 days) **Phase 4:** Schema Migration (1-2 days) **Phase 5:** CLI & Documentation (1 day) **Phase 6:** Testing & Validation (1 day) ### Goals 1. ✅ Establish naming convention 2. ⏳ Implement filename validation 3. ⏳ Create markdown schema loader 4. ⏳ Build schema-for-schemas metaschema 5. ⏳ Migrate 5 existing schemas (remove 2 duplicates) 6. ⏳ Update CLI and documentation ## Updated Tracking ### TODO.md - Added Schema-of-Schemas as active work item - Documented Phase 1 tasks and timeline - Paused capability extraction work ### CHANGELOG.md - Added schema management system to [Unreleased] - Documented directory reorganization - Added "In Progress" section for current work ## Next Steps Begin Phase 1: 1. Implement schema_naming.py with validation 2. Add unit tests 3. Update CLI schema-ingest command 4. Create naming specification document ## Files Changed - CHANGELOG.md - Added unreleased schema management features - TODO.md - Updated active work tracking - roadmap/ - Reorganized from todo/ - roadmap/schema-of-schemas/ - New planning directory - examples/schemas/ - Example markdown schema 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-04 23:47:02 +01:00
feat: implement schema filename validation (Phase 1 complete) Implements filename convention enforcement for schema files as part of the schema-of-schemas implementation. All schemas must now follow the naming pattern: {domain}-schema-v{major}.{minor}.md ## Phase 1 Deliverables ### Schema Naming Module **File:** `markitect/schema_naming.py` (380 lines) **Functions:** - `validate_schema_filename()` - Validate filename against pattern - `suggest_schema_filename()` - Generate valid filename from domain/version - `extract_schema_metadata()` - Extract domain and version from filename - `get_validation_errors()` - Detailed error messages for invalid filenames - `is_valid_schema_filename()` - Simple boolean validation - `format_validation_message()` - User-friendly error formatting **Features:** - Regex-based pattern matching - Automatic normalization (spaces → hyphens, lowercase) - Detailed error reporting - Domain validation (must start with letter) - Version validation (major.minor format) ### Comprehensive Test Suite **File:** `tests/test_schema_naming.py` (500+ lines, 50 tests) **Test Coverage:** - ✅ Valid filename variations (simple, hyphenated, with numbers) - ✅ Invalid filenames (wrong extension, missing components, wrong case) - ✅ Filename suggestion with normalization - ✅ Metadata extraction - ✅ Error message generation - ✅ Edge cases (long names, many hyphens, large versions) - ✅ Pattern regex validation **Results:** 50/50 tests passing (100%) ### Specification Document **File:** `roadmap/schema-of-schemas/SCHEMA_NAMING_SPEC.md` **Contents:** - Formal specification of naming convention - Regular expression pattern with explanation - Valid and invalid examples - Version numbering guidelines - Domain naming best practices - Normalization rules - Migration strategy from legacy naming - Implementation guide ## Naming Convention ### Format ``` {domain}-schema-v{major}.{minor}.md ``` ### Examples ``` ✓ manpage-schema-v1.0.md ✓ api-documentation-schema-v1.0.md ✓ terminology-schema-v1.0.md ✓ arc42-schema-v2.1.md ✗ manpage.json (wrong extension) ✗ ManPage-schema-v1.0.md (uppercase) ✗ manpage-v1.0.md (missing 'schema') ✗ manpage-schema-v1.md (missing minor version) ``` ### Components - **domain**: Lowercase, hyphen-separated, starts with letter - **schema**: Literal keyword - **version**: v{major}.{minor} (SemVer simplified) - **extension**: .md (markdown) ## Implementation Highlights ### Automatic Normalization ```python suggest_schema_filename("API Documentation", "2.1") # → "api-documentation-schema-v2.1.md" suggest_schema_filename("My_Custom Type", "1.0") # → "my-custom-type-schema-v1.0.md" ``` ### Detailed Error Reporting ```python format_validation_message("invalid.json") # → Detailed error list + suggested fix ``` ### Metadata Extraction ```python extract_schema_metadata("manpage-schema-v1.0.md") # → {'domain': 'manpage', 'version': '1.0', 'major': 1, 'minor': 0} ``` ## Migration Plan Current schemas will be renamed: ``` Old → New ──────────────────────────────────────────────────────── terminology-schema.json → terminology-schema-v1.0.md api-documentation → api-documentation-schema-v1.0.md enhanced-manpage → manpage-schema-v2.0.md markdown-manpage → DELETE (duplicate) markdown-manpage-schema.json → DELETE (duplicate) ``` ## Phase 1 Status: ✅ COMPLETE ### Completed - [x] Schema naming module implementation - [x] Comprehensive test suite (50 tests, 100% passing) - [x] Specification document - [x] TODO.md updated ### Next: Phase 2 - [ ] Update CLI schema-ingest with validation - [ ] Implement markdown schema loader - [ ] Parse frontmatter and JSON code blocks - [ ] Update SchemaValidator for .md support ## Testing ```bash # Run tests pytest tests/test_schema_naming.py -v # → 50 passed in 0.48s # Test interactively python -c " from markitect.schema_naming import validate_schema_filename print(validate_schema_filename('manpage-schema-v1.0.md')) " # → (True, {'domain': 'manpage', 'version': '1.0', ...}) ``` ## Files Changed - markitect/schema_naming.py (NEW, 380 lines) - tests/test_schema_naming.py (NEW, 500+ lines) - roadmap/schema-of-schemas/SCHEMA_NAMING_SPEC.md (NEW) - TODO.md (updated progress tracking) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-04 23:51:29 +01:00
**Phase 1 Tasks (Completed ✅):**
- [x] Write `markitect/schema_naming.py` with validation logic
- [x] Add unit tests for filename validation (50 tests, 100% passing)
- [x] Create SCHEMA_NAMING_SPEC.md documentation
chore: establish schema-of-schemas workplan and reorganize roadmap This commit sets up the comprehensive workplan for implementing a markdown-first schema management system with naming conventions, versioning, and self-validation capabilities. ## Directory Reorganization - Renamed `todo/` → `roadmap/` for better organization - Created `roadmap/schema-of-schemas/` subdirectory - Moved schema management planning artifacts to dedicated directory ## Planning Artifacts Created ### Workplan & Documentation - **WORKPLAN.md** (19KB) - Comprehensive 6-phase implementation plan - **SCHEMA_MANAGEMENT_PROPOSAL.md** - Full analysis with 4 options - **SCHEMA_MANAGEMENT_SUMMARY.md** - Executive summary - **README.md** - Quick reference guide ### Example Schema - **examples/schemas/manpage-schema-v1.md** - Demonstrates markdown format ## Schema Management System Design ### Naming Convention **Format:** `{domain}-schema-v{major}.{minor}.md` **Examples:** - `manpage-schema-v1.0.md` - `terminology-schema-v1.0.md` - `api-documentation-schema-v1.0.md` ### Markdown-First Format Schemas will be markdown files with: - YAML frontmatter for metadata - Rich documentation sections - Embedded JSON schema in code block - Version history and examples ### Implementation Phases (8-10 days) **Phase 0:** Planning & Setup ✅ (0.5 days) - COMPLETE **Phase 1:** Filename Convention (1 day) - NEXT **Phase 2:** Markdown Loader (2-3 days) **Phase 3:** Schema-for-Schemas (2 days) **Phase 4:** Schema Migration (1-2 days) **Phase 5:** CLI & Documentation (1 day) **Phase 6:** Testing & Validation (1 day) ### Goals 1. ✅ Establish naming convention 2. ⏳ Implement filename validation 3. ⏳ Create markdown schema loader 4. ⏳ Build schema-for-schemas metaschema 5. ⏳ Migrate 5 existing schemas (remove 2 duplicates) 6. ⏳ Update CLI and documentation ## Updated Tracking ### TODO.md - Added Schema-of-Schemas as active work item - Documented Phase 1 tasks and timeline - Paused capability extraction work ### CHANGELOG.md - Added schema management system to [Unreleased] - Documented directory reorganization - Added "In Progress" section for current work ## Next Steps Begin Phase 1: 1. Implement schema_naming.py with validation 2. Add unit tests 3. Update CLI schema-ingest command 4. Create naming specification document ## Files Changed - CHANGELOG.md - Added unreleased schema management features - TODO.md - Updated active work tracking - roadmap/ - Reorganized from todo/ - roadmap/schema-of-schemas/ - New planning directory - examples/schemas/ - Example markdown schema 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-04 23:47:02 +01:00
feat: implement Phase 2 - Markdown Schema Loader Completed Phase 2 of the schema-of-schemas implementation with full markdown schema support. This enables schemas to be authored as markdown files with rich documentation and embedded JSON schemas. Core Implementation (markitect/schema_loader.py): - MarkdownSchemaLoader class with comprehensive parsing capabilities - YAML frontmatter extraction with error handling - JSON code block extraction with section preference (## Schema Definition) - Metadata merging with x-markitect-source tracking - Schema saving with template support and round-trip capability - Helper methods: list_json_blocks(), validate_schema_structure() Test Coverage (tests/test_schema_loader.py): - 35 comprehensive unit tests (100% passing) - Tests for loading, parsing, saving, round-trip conversion - Edge case handling (empty files, binary files, malformed blocks) - Fixed binary file test to use invalid UTF-8 sequences Example Schema (markitect/schemas/manpage-schema-v1.0.md): - First markdown schema following naming convention - Complete manpage schema with frontmatter + documentation + JSON - Demonstrates section classification and content control - Shows proper structure for future schema authors Documentation (roadmap/schema-of-schemas/SCHEMA_LOADER_GUIDE.md): - Comprehensive user guide (600+ lines) - API reference with examples - Best practices and troubleshooting - Integration patterns for CLI and validator Progress Tracking: - Updated TODO.md with Phase 2 completion - Updated CHANGELOG.md with implementation details - Next: Phase 3 - Schema-for-Schemas Metaschema 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-05 00:02:15 +01:00
**Phase 2 Tasks (Completed ✅):**
- [x] Implement MarkdownSchemaLoader class (markitect/schema_loader.py, 515 lines)
- [x] Add frontmatter extraction (YAML)
- [x] Add JSON code block extraction with section preference
- [x] Add metadata merging with x-markitect-source tracking
- [x] Write comprehensive unit tests (35 tests, 100% passing)
- [x] Create example markdown schema (manpage-schema-v1.0.md)
- [x] Create SCHEMA_LOADER_GUIDE.md documentation
feat: implement Phase 3 - Schema-for-Schemas Metaschema Completed Phase 3 of the schema-of-schemas implementation with a comprehensive metaschema that validates all MarkiTect schema files against conventions and standards. Metaschema Implementation (schema-schema-v1.0.md - 650+ lines): - Validates core JSON Schema fields ($schema, $id, title, description) - Validates MarkiTect version field (SemVer: major.minor.patch) - Validates $id URL format (HTTPS with version path) - Validates MarkiTect extensions: - x-markitect-sections: section classifications and content rules - x-markitect-content-control: pattern and quality validation - x-markitect-metadata: status, authors, tags - x-markitect-source: loader metadata (auto-added) - Section classification validation (required, recommended, optional, discouraged, improper) - Content control pattern validation - Comprehensive documentation with examples and usage guides CLI Command (markitect schema-validate): - Validates schema files against metaschema - Supports both markdown and JSON schema files - Detailed error reporting with schema paths - Structure validation recommendations - Exit codes for CI/CD integration Test Coverage (tests/test_schema_metaschema.py - 12 tests, 100% passing): - Metaschema self-validation - Manpage schema validation - Required fields enforcement - Version format validation (valid and invalid cases) - $id format validation (valid and invalid cases) - Section classification validation - Complete schema with all extensions Validation Results: - ✅ Metaschema validates itself successfully - ✅ Manpage schema (v1.0.md) validates successfully - ⚠️ Terminology schema needs migration (missing version, incorrect $id) Progress Tracking: - Updated TODO.md with Phase 3 completion - Updated CHANGELOG.md with implementation details - Next: Phase 4 - Schema Migration 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-05 03:10:49 +01:00
**Phase 3 Tasks (Completed ✅):**
- [x] Design schema-for-schemas metaschema (schema-schema-v1.0.md)
- [x] Implement metaschema with validation rules for MarkiTect conventions
- [x] Add schema-validate CLI command with detailed error reporting
- [x] Write comprehensive unit tests (12 tests, 100% passing)
- [x] Test metaschema self-validation
- [x] Validate existing schemas against metaschema
chore: establish schema-of-schemas workplan and reorganize roadmap This commit sets up the comprehensive workplan for implementing a markdown-first schema management system with naming conventions, versioning, and self-validation capabilities. ## Directory Reorganization - Renamed `todo/` → `roadmap/` for better organization - Created `roadmap/schema-of-schemas/` subdirectory - Moved schema management planning artifacts to dedicated directory ## Planning Artifacts Created ### Workplan & Documentation - **WORKPLAN.md** (19KB) - Comprehensive 6-phase implementation plan - **SCHEMA_MANAGEMENT_PROPOSAL.md** - Full analysis with 4 options - **SCHEMA_MANAGEMENT_SUMMARY.md** - Executive summary - **README.md** - Quick reference guide ### Example Schema - **examples/schemas/manpage-schema-v1.md** - Demonstrates markdown format ## Schema Management System Design ### Naming Convention **Format:** `{domain}-schema-v{major}.{minor}.md` **Examples:** - `manpage-schema-v1.0.md` - `terminology-schema-v1.0.md` - `api-documentation-schema-v1.0.md` ### Markdown-First Format Schemas will be markdown files with: - YAML frontmatter for metadata - Rich documentation sections - Embedded JSON schema in code block - Version history and examples ### Implementation Phases (8-10 days) **Phase 0:** Planning & Setup ✅ (0.5 days) - COMPLETE **Phase 1:** Filename Convention (1 day) - NEXT **Phase 2:** Markdown Loader (2-3 days) **Phase 3:** Schema-for-Schemas (2 days) **Phase 4:** Schema Migration (1-2 days) **Phase 5:** CLI & Documentation (1 day) **Phase 6:** Testing & Validation (1 day) ### Goals 1. ✅ Establish naming convention 2. ⏳ Implement filename validation 3. ⏳ Create markdown schema loader 4. ⏳ Build schema-for-schemas metaschema 5. ⏳ Migrate 5 existing schemas (remove 2 duplicates) 6. ⏳ Update CLI and documentation ## Updated Tracking ### TODO.md - Added Schema-of-Schemas as active work item - Documented Phase 1 tasks and timeline - Paused capability extraction work ### CHANGELOG.md - Added schema management system to [Unreleased] - Documented directory reorganization - Added "In Progress" section for current work ## Next Steps Begin Phase 1: 1. Implement schema_naming.py with validation 2. Add unit tests 3. Update CLI schema-ingest command 4. Create naming specification document ## Files Changed - CHANGELOG.md - Added unreleased schema management features - TODO.md - Updated active work tracking - roadmap/ - Reorganized from todo/ - roadmap/schema-of-schemas/ - New planning directory - examples/schemas/ - Example markdown schema 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-04 23:47:02 +01:00
**Next Phases:**
- Phase 4: Schema Migration (1-2 days)
- Phase 5: CLI & Documentation Updates (1 day)
- Phase 6: Testing & Validation (1 day)
feat: implement Phase 3 - Schema-for-Schemas Metaschema Completed Phase 3 of the schema-of-schemas implementation with a comprehensive metaschema that validates all MarkiTect schema files against conventions and standards. Metaschema Implementation (schema-schema-v1.0.md - 650+ lines): - Validates core JSON Schema fields ($schema, $id, title, description) - Validates MarkiTect version field (SemVer: major.minor.patch) - Validates $id URL format (HTTPS with version path) - Validates MarkiTect extensions: - x-markitect-sections: section classifications and content rules - x-markitect-content-control: pattern and quality validation - x-markitect-metadata: status, authors, tags - x-markitect-source: loader metadata (auto-added) - Section classification validation (required, recommended, optional, discouraged, improper) - Content control pattern validation - Comprehensive documentation with examples and usage guides CLI Command (markitect schema-validate): - Validates schema files against metaschema - Supports both markdown and JSON schema files - Detailed error reporting with schema paths - Structure validation recommendations - Exit codes for CI/CD integration Test Coverage (tests/test_schema_metaschema.py - 12 tests, 100% passing): - Metaschema self-validation - Manpage schema validation - Required fields enforcement - Version format validation (valid and invalid cases) - $id format validation (valid and invalid cases) - Section classification validation - Complete schema with all extensions Validation Results: - ✅ Metaschema validates itself successfully - ✅ Manpage schema (v1.0.md) validates successfully - ⚠️ Terminology schema needs migration (missing version, incorrect $id) Progress Tracking: - Updated TODO.md with Phase 3 completion - Updated CHANGELOG.md with implementation details - Next: Phase 4 - Schema Migration 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-05 03:10:49 +01:00
**Expected Completion:** 4-5 days remaining
chore: establish schema-of-schemas workplan and reorganize roadmap This commit sets up the comprehensive workplan for implementing a markdown-first schema management system with naming conventions, versioning, and self-validation capabilities. ## Directory Reorganization - Renamed `todo/` → `roadmap/` for better organization - Created `roadmap/schema-of-schemas/` subdirectory - Moved schema management planning artifacts to dedicated directory ## Planning Artifacts Created ### Workplan & Documentation - **WORKPLAN.md** (19KB) - Comprehensive 6-phase implementation plan - **SCHEMA_MANAGEMENT_PROPOSAL.md** - Full analysis with 4 options - **SCHEMA_MANAGEMENT_SUMMARY.md** - Executive summary - **README.md** - Quick reference guide ### Example Schema - **examples/schemas/manpage-schema-v1.md** - Demonstrates markdown format ## Schema Management System Design ### Naming Convention **Format:** `{domain}-schema-v{major}.{minor}.md` **Examples:** - `manpage-schema-v1.0.md` - `terminology-schema-v1.0.md` - `api-documentation-schema-v1.0.md` ### Markdown-First Format Schemas will be markdown files with: - YAML frontmatter for metadata - Rich documentation sections - Embedded JSON schema in code block - Version history and examples ### Implementation Phases (8-10 days) **Phase 0:** Planning & Setup ✅ (0.5 days) - COMPLETE **Phase 1:** Filename Convention (1 day) - NEXT **Phase 2:** Markdown Loader (2-3 days) **Phase 3:** Schema-for-Schemas (2 days) **Phase 4:** Schema Migration (1-2 days) **Phase 5:** CLI & Documentation (1 day) **Phase 6:** Testing & Validation (1 day) ### Goals 1. ✅ Establish naming convention 2. ⏳ Implement filename validation 3. ⏳ Create markdown schema loader 4. ⏳ Build schema-for-schemas metaschema 5. ⏳ Migrate 5 existing schemas (remove 2 duplicates) 6. ⏳ Update CLI and documentation ## Updated Tracking ### TODO.md - Added Schema-of-Schemas as active work item - Documented Phase 1 tasks and timeline - Paused capability extraction work ### CHANGELOG.md - Added schema management system to [Unreleased] - Documented directory reorganization - Added "In Progress" section for current work ## Next Steps Begin Phase 1: 1. Implement schema_naming.py with validation 2. Add unit tests 3. Update CLI schema-ingest command 4. Create naming specification document ## Files Changed - CHANGELOG.md - Added unreleased schema management features - TODO.md - Updated active work tracking - roadmap/ - Reorganized from todo/ - roadmap/schema-of-schemas/ - New planning directory - examples/schemas/ - Example markdown schema 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-04 23:47:02 +01:00
---
### Extract Capability-Capability from Issue-Facade (Paused)
**Context:** Issue-facade currently provides two capabilities:
1. **issue-tracking** (explicit in CAPABILITY-issue-tracking.yaml) - Issue management across platforms
2. **capability-capability** (implicit) - Patterns and tools for creating/managing capabilities
The **capability-capability** includes:
- Feedback pattern (feedback/ directory, .capability/feedback CLI tool, documentation)
- Detachment facility (.capability/detach script for clean capability removal)
- Integration pattern (.capability/integrate.sh for project integration)
- CAPABILITY-*.yaml specification format
- ReusableCapabilitiesArchitecture.md (complete specification)
- Directory conventions (_family/implementation, visible/hidden patterns)
**Goal:** Extract capability-capability to separate `reusable-capability` repository so it can be used by any capability in the markitect ecosystem.
**Approach:** Step-by-step extraction, starting with specification.
#### Phase 1: Specification & Planning (Current)
- [ ] Create CAPABILITY-capability.yaml in issue-facade to explicitly declare the implicit capability
- [ ] Define what belongs to capability-capability family vs issue-tracking family
- [ ] Document the capability-capability API surface (what tools/patterns it provides)
- [ ] Identify all files/directories to extract
- [ ] Plan extraction strategy (copy vs move, how to maintain during transition)
#### Phase 2: Repository Creation
- [ ] Create reusable-capability repository structure
- [ ] Extract ReusableCapabilitiesArchitecture.md to new repo
- [ ] Extract feedback pattern (directory structure, CLI tool, README)
- [ ] Extract detachment facility (.capability/detach)
- [ ] Extract integration scripts (.capability/integrate.sh, integration-checklist.md)
- [ ] Create CAPABILITY-capability.yaml in new repo (canonical version)
- [ ] Add README.md for reusable-capability repo
#### Phase 3: Integration & Testing
- [ ] Update issue-facade to depend on reusable-capability (as integrated capability)
- [ ] Integrate reusable-capability into issue-facade using _capability/reusable-capability pattern
- [ ] Test that issue-facade still works with extracted capability
- [ ] Update issue-facade documentation to reference both capabilities it provides/uses
- [ ] Verify feedback system still works
- [ ] Verify detachment still works
#### Phase 4: Dogfooding & Validation
- [ ] Choose another markitect capability for dogfooding
- [ ] Integrate reusable-capability into that capability
- [ ] Add feedback system to new capability
- [ ] Add detachment facility to new capability
- [ ] Document learnings and refine reusable-capability based on real-world usage
- [ ] Update ReusableCapabilitiesArchitecture.md with insights
**Current Step:** Phase 1, Task 1 - Create CAPABILITY-capability.yaml
***
## Completed Tasks
*Recent completed tasks have been documented in _issue-tracking/issue-facade/CHANGELOG.md following Keep a Changelog format.*
feat: add terminology schema example and improve schema-list command This commit completes Phase 2 of schema evolution work and establishes a new example demonstrating schema usage for terminology documents. ## New Features ### Terminology Validation Example (examples/terminology/) - Complete example terminology document with proper structure - JSON schema with MarkiTect extensions for validation - Demonstrates schema usage beyond manpages (glossaries, lexicons) - Validates term structure: Definition, Synonyms, Related Terms, Examples - Includes content control and quality validation rules - Full documentation with usage examples and best practices ### Schema Registration System - Registered terminology schema in markitect database - Created schema catalog (markitect/schemas/schema-catalog.yaml) - Copied schema to official location (markitect/schemas/) - Provides metadata, features, and usage info for all schemas ### Improved schema-list Command - Now displays creation timestamps in default output - Table format includes Created/Updated columns - Cleaner timestamp formatting (removed microseconds) - Better visibility into when schemas were added ## Files Changed Added: - examples/terminology/README.md - Complete documentation - examples/terminology/terminology-example.md - Example glossary - examples/terminology/terminology-schema.json - Validation schema - markitect/schemas/terminology-schema.json - Registered schema - markitect/schemas/schema-catalog.yaml - Schema registry Modified: - markitect/cli.py - Enhanced schema-list with timestamps - TODO.md - Documented Phase 2 completion and new example Moved: - SCHEMA_EVOLUTION_WORKPLAN.md → todo/ directory ## Schema Features Demonstrated - Heading hierarchy validation (H1 → H2 → H3) - Term structure validation with required/optional fields - Content quality metrics (word counts, readability targets) - MarkiTect extensions (x-markitect-sections, x-markitect-content-control) - Classification system (required/recommended/optional/discouraged/improper) ## Usage ```bash # List schemas with timestamps markitect schema-list # Validate terminology document markitect validate glossary.md --schema terminology-schema.json # View in table format markitect schema-list --format table ``` 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-04 23:07:36 +01:00
### 2026-01-04 - Phase 2: Schema Refinement Tools & Terminology Example
- ✅ Implemented schema-analyze command to detect rigidity issues
- ✅ Implemented schema-refine command with automatic loosening logic
- ✅ Added interactive mode to schema-refine for fine-grained control
- ✅ Created comprehensive test suite (33 unit tests, 100% passing)
- ✅ Wrote user guide documentation with examples and workflows
- ✅ Successfully tested on example schemas (reduced rigidity from 60/100 to 24/100)
- ✅ Integrated into CLI with proper exit codes and error handling
feat: add terminology schema example and improve schema-list command This commit completes Phase 2 of schema evolution work and establishes a new example demonstrating schema usage for terminology documents. ## New Features ### Terminology Validation Example (examples/terminology/) - Complete example terminology document with proper structure - JSON schema with MarkiTect extensions for validation - Demonstrates schema usage beyond manpages (glossaries, lexicons) - Validates term structure: Definition, Synonyms, Related Terms, Examples - Includes content control and quality validation rules - Full documentation with usage examples and best practices ### Schema Registration System - Registered terminology schema in markitect database - Created schema catalog (markitect/schemas/schema-catalog.yaml) - Copied schema to official location (markitect/schemas/) - Provides metadata, features, and usage info for all schemas ### Improved schema-list Command - Now displays creation timestamps in default output - Table format includes Created/Updated columns - Cleaner timestamp formatting (removed microseconds) - Better visibility into when schemas were added ## Files Changed Added: - examples/terminology/README.md - Complete documentation - examples/terminology/terminology-example.md - Example glossary - examples/terminology/terminology-schema.json - Validation schema - markitect/schemas/terminology-schema.json - Registered schema - markitect/schemas/schema-catalog.yaml - Schema registry Modified: - markitect/cli.py - Enhanced schema-list with timestamps - TODO.md - Documented Phase 2 completion and new example Moved: - SCHEMA_EVOLUTION_WORKPLAN.md → todo/ directory ## Schema Features Demonstrated - Heading hierarchy validation (H1 → H2 → H3) - Term structure validation with required/optional fields - Content quality metrics (word counts, readability targets) - MarkiTect extensions (x-markitect-sections, x-markitect-content-control) - Classification system (required/recommended/optional/discouraged/improper) ## Usage ```bash # List schemas with timestamps markitect schema-list # Validate terminology document markitect validate glossary.md --schema terminology-schema.json # View in table format markitect schema-list --format table ``` 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-04 23:07:36 +01:00
- ✅ Moved SCHEMA_EVOLUTION_WORKPLAN.md to todo/ directory
- ✅ Created terminology validation example (examples/terminology/)
**Key Features Delivered:**
- Rigidity score calculation (0-100 scale)
- Automatic detection of exact counts, const values, overly specific numbers
- Path navigation for nested schema properties
- Dry-run mode for previewing changes
- Interactive approval workflow
- Comprehensive reporting (normal and verbose modes)
feat: add terminology schema example and improve schema-list command This commit completes Phase 2 of schema evolution work and establishes a new example demonstrating schema usage for terminology documents. ## New Features ### Terminology Validation Example (examples/terminology/) - Complete example terminology document with proper structure - JSON schema with MarkiTect extensions for validation - Demonstrates schema usage beyond manpages (glossaries, lexicons) - Validates term structure: Definition, Synonyms, Related Terms, Examples - Includes content control and quality validation rules - Full documentation with usage examples and best practices ### Schema Registration System - Registered terminology schema in markitect database - Created schema catalog (markitect/schemas/schema-catalog.yaml) - Copied schema to official location (markitect/schemas/) - Provides metadata, features, and usage info for all schemas ### Improved schema-list Command - Now displays creation timestamps in default output - Table format includes Created/Updated columns - Cleaner timestamp formatting (removed microseconds) - Better visibility into when schemas were added ## Files Changed Added: - examples/terminology/README.md - Complete documentation - examples/terminology/terminology-example.md - Example glossary - examples/terminology/terminology-schema.json - Validation schema - markitect/schemas/terminology-schema.json - Registered schema - markitect/schemas/schema-catalog.yaml - Schema registry Modified: - markitect/cli.py - Enhanced schema-list with timestamps - TODO.md - Documented Phase 2 completion and new example Moved: - SCHEMA_EVOLUTION_WORKPLAN.md → todo/ directory ## Schema Features Demonstrated - Heading hierarchy validation (H1 → H2 → H3) - Term structure validation with required/optional fields - Content quality metrics (word counts, readability targets) - MarkiTect extensions (x-markitect-sections, x-markitect-content-control) - Classification system (required/recommended/optional/discouraged/improper) ## Usage ```bash # List schemas with timestamps markitect schema-list # Validate terminology document markitect validate glossary.md --schema terminology-schema.json # View in table format markitect schema-list --format table ``` 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-04 23:07:36 +01:00
**Terminology Example:**
- Complete terminology document structure (terminology-example.md)
- JSON schema with MarkiTect extensions (terminology-schema.json)
- Demonstrates schema usage for non-manpage documents
- Validates term definitions, synonyms, related terms, examples
- Includes content control and validation rules
- Full documentation and usage examples (README.md)
feat: implement Phase 2 - Markdown Schema Loader Completed Phase 2 of the schema-of-schemas implementation with full markdown schema support. This enables schemas to be authored as markdown files with rich documentation and embedded JSON schemas. Core Implementation (markitect/schema_loader.py): - MarkdownSchemaLoader class with comprehensive parsing capabilities - YAML frontmatter extraction with error handling - JSON code block extraction with section preference (## Schema Definition) - Metadata merging with x-markitect-source tracking - Schema saving with template support and round-trip capability - Helper methods: list_json_blocks(), validate_schema_structure() Test Coverage (tests/test_schema_loader.py): - 35 comprehensive unit tests (100% passing) - Tests for loading, parsing, saving, round-trip conversion - Edge case handling (empty files, binary files, malformed blocks) - Fixed binary file test to use invalid UTF-8 sequences Example Schema (markitect/schemas/manpage-schema-v1.0.md): - First markdown schema following naming convention - Complete manpage schema with frontmatter + documentation + JSON - Demonstrates section classification and content control - Shows proper structure for future schema authors Documentation (roadmap/schema-of-schemas/SCHEMA_LOADER_GUIDE.md): - Comprehensive user guide (600+ lines) - API reference with examples - Best practices and troubleshooting - Integration patterns for CLI and validator Progress Tracking: - Updated TODO.md with Phase 2 completion - Updated CHANGELOG.md with implementation details - Next: Phase 3 - Schema-for-Schemas Metaschema 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-05 00:02:15 +01:00
### 2026-01-04 - Phase 2: Markdown Schema Loader
- ✅ Implemented MarkdownSchemaLoader class (markitect/schema_loader.py, 515 lines)
- ✅ YAML frontmatter extraction with validation
- ✅ JSON code block extraction with "Schema Definition" section preference
- ✅ Metadata merging with x-markitect-source tracking
- ✅ Schema saving with template support and round-trip capability
- ✅ Comprehensive test suite (35 unit tests, 100% passing)
- ✅ Created example markdown schema (manpage-schema-v1.0.md)
- ✅ Created SCHEMA_LOADER_GUIDE.md with complete usage documentation
**Key Features Delivered:**
- Markdown-first schema format with embedded JSON
- Frontmatter metadata merges into schema ($id, version, status)
- Automatic detection of multiple JSON blocks
- Schema structure validation helper
- Error handling for binary files and invalid formats
- List JSON blocks helper for debugging
- Full round-trip save/load capability
**Example Markdown Schema:**
- manpage-schema-v1.0.md demonstrating complete format
- Includes frontmatter, documentation, and JSON schema
- Shows section classification and content control
- Follows naming convention: {domain}-schema-v{major}.{minor}.md
feat: implement Phase 3 - Schema-for-Schemas Metaschema Completed Phase 3 of the schema-of-schemas implementation with a comprehensive metaschema that validates all MarkiTect schema files against conventions and standards. Metaschema Implementation (schema-schema-v1.0.md - 650+ lines): - Validates core JSON Schema fields ($schema, $id, title, description) - Validates MarkiTect version field (SemVer: major.minor.patch) - Validates $id URL format (HTTPS with version path) - Validates MarkiTect extensions: - x-markitect-sections: section classifications and content rules - x-markitect-content-control: pattern and quality validation - x-markitect-metadata: status, authors, tags - x-markitect-source: loader metadata (auto-added) - Section classification validation (required, recommended, optional, discouraged, improper) - Content control pattern validation - Comprehensive documentation with examples and usage guides CLI Command (markitect schema-validate): - Validates schema files against metaschema - Supports both markdown and JSON schema files - Detailed error reporting with schema paths - Structure validation recommendations - Exit codes for CI/CD integration Test Coverage (tests/test_schema_metaschema.py - 12 tests, 100% passing): - Metaschema self-validation - Manpage schema validation - Required fields enforcement - Version format validation (valid and invalid cases) - $id format validation (valid and invalid cases) - Section classification validation - Complete schema with all extensions Validation Results: - ✅ Metaschema validates itself successfully - ✅ Manpage schema (v1.0.md) validates successfully - ⚠️ Terminology schema needs migration (missing version, incorrect $id) Progress Tracking: - Updated TODO.md with Phase 3 completion - Updated CHANGELOG.md with implementation details - Next: Phase 4 - Schema Migration 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-05 03:10:49 +01:00
### 2026-01-04 - Phase 3: Schema-for-Schemas Metaschema
- ✅ Created schema-schema-v1.0.md metaschema (650+ lines)
- ✅ Validates core JSON Schema fields ($schema, $id, title, description)
- ✅ Validates MarkiTect version field (SemVer: major.minor.patch)
- ✅ Validates $id URL format (HTTPS with version)
- ✅ Validates MarkiTect extensions (x-markitect-sections, x-markitect-content-control, x-markitect-metadata)
- ✅ Implemented schema-validate CLI command with detailed error reporting
- ✅ Comprehensive test suite (12 unit tests, 100% passing)
- ✅ Metaschema self-validation successful
**Key Features Delivered:**
- Complete metaschema for validating all MarkiTect schemas
- Section classification validation (required, recommended, optional, discouraged, improper)
- Content control pattern validation
- Version format enforcement (SemVer)
- $id URL format enforcement (HTTPS with version)
- CLI command for easy schema validation
- Detailed error messages with schema paths
**Validation Results:**
- ✅ Metaschema validates itself
- ✅ Manpage schema validates successfully
- ⚠️ Terminology schema needs migration (missing version field, incorrect $id format)
### 2025-12-17 - Architecture Refactoring
- ✅ Implemented ReusableCapabilitiesArchitecture v0.1
- ✅ Added feedback capability to issue-facade
- ✅ Created detachment facility
- ✅ Refactored to family-based directory structure (_issue-tracking/issue-facade)
- ✅ Made feedback directory visible (feedback/ not .feedback/)
- ✅ Renamed to explicit family declaration (CAPABILITY-issue-tracking.yaml)
- ✅ Created CHANGELOG.md documenting v1.0.0