research: work orchestration reconciliation — fleet survey, best practices, architecture draft
Trigger: binky-control automation queues (AWQ/DEC/OH) emerged as an unindexed parallel work-item ontology (AWQ-010 reintegration case). Three artefacts: - research/2026-07-19-work-orchestration-infrastructure-survey.md — inventory of the ADR-001/005 backbone, state models, task-flow engine, hub generations, execution stack, and the >=9 live work-item species - research/2026-07-19-work-orchestration-best-practices.md — external 2025/26 practice: GitOps reconciliation, risk-tiered HITL, flow metrics/WIP, lean budget guardrails; anti-patterns to skip - research/WorkOrchestrationArchitectureDraft.md — v0.1 proposal: unified work-record ontology, type registry + UUIDv7/canonical-name layering, lanes to fleet canon, policy/derived tags, queues as generated views, staged migration; open questions for discussion Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
codex
parent
435731231a
commit
f4a76fae5f
3 changed files with 688 additions and 0 deletions
274
research/WorkOrchestrationArchitectureDraft.md
Normal file
274
research/WorkOrchestrationArchitectureDraft.md
Normal file
|
|
@ -0,0 +1,274 @@
|
|||
# Work Orchestration Architecture — Draft for Discussion
|
||||
|
||||
> Draft v0.1 — 2026-07-19. Synthesis of
|
||||
> `2026-07-19-work-orchestration-infrastructure-survey.md` (internal) and
|
||||
> `2026-07-19-work-orchestration-best-practices.md` (external).
|
||||
> Trigger: reconciling the binky-control automation queues with the
|
||||
> suggestion/task/workplan backbone (AWQ-010 case). Status: **proposal** —
|
||||
> nothing here is canon until ratified; terminology extensions are marked
|
||||
> ⟨new⟩ and would land as a canon standard v0.1 after discussion.
|
||||
|
||||
## 0. Thesis
|
||||
|
||||
The queues were not a mistake — they were a **market signal**. Intake
|
||||
through a one-YAML-block append beat the hub suggestion flow because its
|
||||
friction was an order of magnitude lower, and it carried routing information
|
||||
(lanes) the backbone lacks. The error was ontological, not practical: the
|
||||
queue items became a second, unindexed species of work item.
|
||||
|
||||
The reconciliation is therefore not "move queues into the hub" but:
|
||||
**make the backbone cheap enough at intake that the queue pattern becomes a
|
||||
schema-valid, hub-indexed instance of it** — and make every future species
|
||||
impossible to create outside the reconciliation loop without tripping an
|
||||
alarm.
|
||||
|
||||
One sentence: *everything with an ID is a work record, every work record is
|
||||
a repo file, every repo file is indexed, and every list anyone works from is
|
||||
a derived view.*
|
||||
|
||||
## 1. The unified ontology
|
||||
|
||||
### 1.1 Work record ⟨new umbrella term⟩
|
||||
|
||||
A **work record** is any identified, lifecycle-bearing coordination artefact.
|
||||
All existing species become *kinds* of work record:
|
||||
|
||||
```
|
||||
work record
|
||||
├── intake item ⟨new⟩ — spark: idea, mail finding, observation, request
|
||||
│ (absorbs: AWQ items, hub suggestions, "actionable" mail-triage rows)
|
||||
├── task — smallest executable unit (canon lifecycle)
|
||||
│ (absorbs: workplan tasks, adhoc tasks, issue-core issues-as-interface)
|
||||
├── workplan — structured, dependency-bearing body of work
|
||||
├── decision — approval package + resolution
|
||||
│ (absorbs: DEC items; hub decision records become the index)
|
||||
├── engagement ⟨new⟩ — scheduled external interaction
|
||||
│ (absorbs: OH items — office-hour/command-day bookings)
|
||||
└── register entry — risk, technical debt, capability request, CCR,
|
||||
interface change (existing species, unchanged
|
||||
semantics, brought under the same conventions)
|
||||
```
|
||||
|
||||
Every work record, regardless of kind, carries the same **conventional
|
||||
spine** (aspect 4 — one framework, many demands):
|
||||
|
||||
| Field | Convention |
|
||||
| --- | --- |
|
||||
| `id` | canonical name per type registry (§1.3) |
|
||||
| `uuid` | UUIDv7, hub-assigned at first index, written back to the file (like `state_hub_*_id` today) |
|
||||
| `kind` | from the closed kind list above |
|
||||
| `lane` | green/blue/yellow/orange/red (§2) |
|
||||
| `status` | abstract canon lifecycle for its kind (§1.4) |
|
||||
| `owner` | repo-anchored owner (agent identity or human) |
|
||||
| `repo` | owning repo (ADR-001/ADR-005 anchor) |
|
||||
| `tags` | policy + derived tags (§3) |
|
||||
| `created`/`updated` | dates; UUIDv7 provides fine-grained internal time |
|
||||
|
||||
Kind-specific fields (budget, evidence, deadline_pressure,
|
||||
attention_cost, …) extend the spine; they never replace it.
|
||||
|
||||
### 1.2 File-first, one reconciliation loop (aspect 3)
|
||||
|
||||
- Every work record originates in a repo file: workplans as today; intake
|
||||
items, decisions, engagements as YAML blocks inside per-repo registry
|
||||
documents (`queues/…` or the existing live documents — location is
|
||||
convention, the schema is what matters).
|
||||
- **Forgejo CI validates schemas** on push (yaml-block schema per kind —
|
||||
the GitOps "validate before apply" rule).
|
||||
- `fix-consistency` grows checks per kind (as C-06 does for workplans):
|
||||
registers new records, writes UUIDs back, syncs status *from file to hub*.
|
||||
The hub stays a read model; runtime operations data (logs, metrics, run
|
||||
histories, token events) is the sanctioned exception and lives only in
|
||||
hub/activity-core/monitoring stores.
|
||||
- Derived views (§4) are generated *from the hub*, never hand-maintained:
|
||||
the "Completed" tail of AutopilotWorkQueue.md, daily-brief queue summaries
|
||||
etc. become materialized view output (ADR-003 pattern), clearly marked
|
||||
`[auto]`.
|
||||
|
||||
### 1.3 Type registry and layered identity (aspect 5)
|
||||
|
||||
A single canon file (`canon/standards/work-record-types_v0.1.md` + machine
|
||||
copy `work-record-types.yaml`) registers every id scheme:
|
||||
|
||||
```yaml
|
||||
- code: WP # {PREFIX}-WP-NNNN workplan
|
||||
- code: T # {WP-ID}-TNN task
|
||||
- code: IN # {PREFIX}-IN-NNNN intake item ⟨new; AWQ-* grandfathered⟩
|
||||
- code: DEC # {PREFIX}-DEC-YYYY-NNN decision
|
||||
- code: ENG # {PREFIX}-ENG-YYYY-NNN engagement ⟨OH-* grandfathered⟩
|
||||
- code: RISK, TD, CAP, CCR, ADR … register entries (existing)
|
||||
```
|
||||
|
||||
- **UUIDv7 is primary** for bookkeeping, relations, dependencies, history —
|
||||
stable across renames, migrations, and terminology transitions
|
||||
(the workstream→workplan migration cost is the proof this matters).
|
||||
- **Canonical names are primary for humans and agents** in UI/CLI/API
|
||||
output, prose, and commits; UUIDs inspectable on demand. This is already
|
||||
practiced for WP/T — the registry universalizes it.
|
||||
- Grandfathering rule (from the terminology standard): existing ids
|
||||
(AWQ-, OH-) are never renamed; the registry maps legacy prefixes to kinds
|
||||
and the legacy-meter discipline governs retirement of the *schemes*, not
|
||||
the records.
|
||||
|
||||
### 1.4 Abstract states + flow profiles (aspect 4)
|
||||
|
||||
- The **abstract lifecycles stay minimal and canon-fixed**: task
|
||||
`wait/todo/progress/done/cancel` (InfoTechCanon); workplan
|
||||
`proposed/ready/active/blocked/backlog/finished/archived`; intake item
|
||||
`open → vetted → routed → closed(promoted|declined|absorbed)` ⟨new,
|
||||
replaces both suggestion stages and queue-item ad-hoc states⟩; decision
|
||||
`prepared → resolved(approved|rejected|revised|deferred)`; engagement
|
||||
`queued → prepared → done`.
|
||||
- **Different work kinds** (development, ops, compliance, billing, sales…)
|
||||
get their demands met by **flow profiles** on the existing task-flow
|
||||
engine: named workstation definitions whose entry/exit assertions encode
|
||||
the kind-specific gates (e.g. a compliance task cannot exit `progress`
|
||||
without an evidence link; a sales engagement cannot enter `prepared`
|
||||
without prepared_material). Profiles are YAML in repos → same
|
||||
reconciliation loop. States stay abstract; rigor lives in assertions.
|
||||
|
||||
## 2. Lanes become fleet canon (aspects 1, 6, 7)
|
||||
|
||||
Promote binky-control's AutonomyPolicy lane model to
|
||||
`canon/standards/autonomy-lanes_v0.1.md`, unchanged in substance:
|
||||
|
||||
- Green (safe autonomous) · Blue (reversible, logged) · Yellow (deferred
|
||||
approval) · Orange (time-window approval) · Red (human-only).
|
||||
- `lane` is a **mandatory field on every work record** and on every agent
|
||||
manifest; the harness already enforces lane-scoped tool profiles — this
|
||||
closes the loop from item to execution permission.
|
||||
- Human attention is modeled as a **WIP-limited workstation**: the founder
|
||||
attention budget (minutes/day, max decisions/day) is a lane capacity;
|
||||
Yellow routes to the decision queue view, Orange to the engagement view,
|
||||
both subject to that capacity. External research is unanimous that
|
||||
risk-tiered gating is the only scalable human-oversight pattern.
|
||||
- Time-boxing (`needed_by`, `fallback_if_no_response`) is hub-owned: the
|
||||
hub (not a human reading a file) raises the escalation when a window
|
||||
lapses — via the existing NATS → activity-core delegation.
|
||||
|
||||
## 3. Policy tags, not states (aspect 6)
|
||||
|
||||
Two tag classes, both in the `tags` field, never in `status`:
|
||||
|
||||
- **Derived tags** — computed by hub/activity-core from data, read-only in
|
||||
files: `stalled` (work-item age over threshold — the flow-metrics form),
|
||||
`stale` (reviewed against older repo state), `overdue-decision`,
|
||||
`budget-breached`, `orphaned` (no goal/workplan link).
|
||||
- **Policy tags** — assigned by agents/humans under tagging policies:
|
||||
`automatable`, `needs-human`, `unclear-implementation`, `red-adjacent`
|
||||
(touches a Red boundary), `compliance-relevant`. Tagging policies are
|
||||
YAML rules (activity-core `ActivityDefinition`s may auto-apply them on
|
||||
events) — auditable and evolvable without schema migrations.
|
||||
|
||||
Tags drive the optimization loop: `automatable` feeds the autopilot view;
|
||||
`needs-human` + lane feeds attention routing; `stalled`+`unclear` feeds the
|
||||
kaizen coach. This is how "what should be automated next" becomes a query
|
||||
instead of a discovery.
|
||||
|
||||
## 4. Queues are views (the reconciliation itself)
|
||||
|
||||
The three binky queues are retained **as products, redefined as views**:
|
||||
|
||||
| Today | Becomes |
|
||||
| --- | --- |
|
||||
| AutopilotWorkQueue.md (hand-edited) | *Autopilot view*: `kind=intake∨task, lane∈{green,blue}, status∈{open,todo}, tag=automatable`, WSJF-ordered; "never empty" becomes a monitored invariant with an alert, not a plea |
|
||||
| DecisionQueue.md open section | *Decision view*: `kind=decision, status=prepared`, ordered by needed_by; resolved log is generated |
|
||||
| OfficeHourQueue.md | *Engagement view*: `kind=engagement`, batched per command day; runbook generation stays (officehour/) |
|
||||
|
||||
Mechanics: records are authored in files (cheap intake preserved — one YAML
|
||||
block, schema-checked); `fix-consistency` indexes them; view documents are
|
||||
regenerated `[auto]` sections. Pull protocol becomes a status transition
|
||||
(`todo → progress` with owner) instead of a prose note — which makes claims
|
||||
visible fleet-wide and collision-free (issue-core remains the claim
|
||||
*interface* for harness runtime; the record is the truth).
|
||||
|
||||
Promotion (the AWQ-010 lesson) becomes a **first-class transition**:
|
||||
`intake.routed → workplan|task|decision|engagement` with back-links
|
||||
(`promoted_to`/`origin`), performable by one CLI/MCP call that writes both
|
||||
files and re-syncs — never a manual transcription again.
|
||||
|
||||
## 5. Visibility, tenancy, scale (aspect 2)
|
||||
|
||||
- **state-hub** remains the dev-coordination read model (localhost,
|
||||
file-first, fix-consistency).
|
||||
- **core-hub** is the production visibility layer: dashboards, flow metrics
|
||||
(WIP, cycle time, item age, throughput — all computable from UUIDv7 +
|
||||
status transitions), cross-domain rollups, operator console. The work
|
||||
record spine becomes part of the core-hub contract/IR so Gen-3 ingests
|
||||
ADR-001 artefacts natively (its workplan-coordination spec already
|
||||
commits to this).
|
||||
- **Tenancy**: repo → domain → tenant, derived from repo classification
|
||||
(never a parallel spine). agent-harness is already multi-tenant with
|
||||
per-tenant credential lanes (ops-warden/OpenBao); dashboards get
|
||||
tenant/domain scoping from the same derivation. Separation is by
|
||||
classification + credential lanes; centralization is by shared runtime
|
||||
(harness, activity-core, hubs) — efficient utilization with clean blast
|
||||
radii.
|
||||
- **Budgets as guardrail envelopes** (aspect 1): per-run token budgets
|
||||
(harness manifests, enforced today) roll up to per-workplan and per-lane
|
||||
envelopes recorded on the work record; `record_token_event` meters,
|
||||
`budget-breached` tag + escalation on breach. Complex-project risk
|
||||
control = dependency edges (existing) + lane gates + budget envelopes +
|
||||
flow metrics, all on one spine.
|
||||
|
||||
## 6. Opinionated by construction (aspect 7)
|
||||
|
||||
- **The registry is closed.** An id pattern not in the type registry fails
|
||||
CI and fix-consistency (the "sidetrack detector" — a warning that names
|
||||
the reintegration cost explicitly). New kinds are added by canon PR, which
|
||||
is deliberately cheap (one YAML entry + one schema) so the legitimate
|
||||
path is easier than the workaround — the core lesson of the queues.
|
||||
- **AGENTS.md generation** (state-hub templates) teaches the conventions in
|
||||
every repo; the guidance and the enforcement come from the same source.
|
||||
- **Intake friction budget**: creating an intake item must never cost more
|
||||
than the AWQ pattern did (append one block). If a future workaround
|
||||
appears anyway, the postmortem question is fixed: *which friction did we
|
||||
fail to remove?*
|
||||
|
||||
## 7. Migration path (staged, history-preserving)
|
||||
|
||||
1. **Canonize** (docs only): work-record umbrella + type registry +
|
||||
autonomy-lanes standard + intake lifecycle → canon standards v0.1;
|
||||
extend `workplan-terminology-fleet` rather than fork it.
|
||||
2. **Schema + CI**: YAML schemas per kind; Forgejo CI check; binky-control
|
||||
is the pilot (its queues become schema-valid without renaming ids).
|
||||
3. **Index**: fix-consistency checks for intake/decision/engagement kinds
|
||||
(C-2x series); hub tables reuse existing entities where they fit
|
||||
(suggestions table ⇢ intake items — suggestions gain file artefacts,
|
||||
resolving their own ADR-001 tension; decisions already dual-exist,
|
||||
the check just automates the mirroring both ways).
|
||||
4. **Views**: generated `[auto]` sections replace hand-maintained queue
|
||||
tails; hub owns needed_by clocks via NATS → activity-core.
|
||||
5. **Tags + metrics**: derived-tag computation and the four flow metrics in
|
||||
state-hub first, surfaced in core-hub dashboards as the contract lands.
|
||||
6. **Retire** legacy singularities per legacy-meter discipline (e.g. the
|
||||
hub-first suggestion create path) only after zero measured use.
|
||||
|
||||
Each stage is independently valuable; stopping after any stage leaves the
|
||||
system consistent (no big-bang).
|
||||
|
||||
## 8. Open questions for discussion
|
||||
|
||||
1. **Naming**: "work record" vs. "work item" as umbrella; "intake item" vs.
|
||||
keeping "suggestion" fleet-wide; German-friendly alternatives?
|
||||
2. **Where intake blocks live**: per-repo `queues/intake.md` vs. today's
|
||||
thematic live documents (AutopilotWorkQueue as a *view* argues for
|
||||
separating authored records from generated views into distinct files).
|
||||
3. **Engagements** as their own kind vs. decisions-with-a-time-window —
|
||||
the Orange lane is behaviorally distinct (batching), which argues for
|
||||
the separate kind, but the spine could carry it as a decision subtype.
|
||||
4. **WSJF adoption**: reuse the 2026-06-04 calibration or start simpler
|
||||
(lane + priority + age) and let kaizen metrics justify WSJF later?
|
||||
5. **issue-core's long-term role**: claim interface only (proposed here),
|
||||
or should issues become file-backed work records too?
|
||||
6. **Suggestion table reuse vs. new intake entity** in state-hub — schema
|
||||
archaeology needed before stage 3.
|
||||
7. **Budget envelope placement**: frontmatter on workplans vs. a separate
|
||||
`budgets.yaml` per repo (compliance kinds may want the latter).
|
||||
|
||||
---
|
||||
|
||||
*Prepared from live fleet data (AWQ-010 reintegration, DEC-2026-003/004
|
||||
dual bookkeeping, BINKY-WP-0002 slug drift) and the two research artefacts
|
||||
alongside this file. Next step after discussion: canon standard drafts +
|
||||
a CUST workplan for stage 1–2.*
|
||||
Loading…
Add table
Add a link
Reference in a new issue