tegwick
047a6a2d31
Assistant: codex Assistant-Model: gpt-6.1-sol Assistant-Session: 01a1045b-933d-7aa1-b2db-104557487a79
85 lines
3.7 KiB
Markdown
85 lines
3.7 KiB
Markdown
---
|
|
id: FEP-WP-0004
|
|
type: workplan
|
|
title: "Evaluate retrieval and real repository applicability"
|
|
domain: infotech
|
|
repo: frontend-patterns
|
|
status: blocked
|
|
owner: codex
|
|
topic_slug: canon
|
|
created: "2026-09-24"
|
|
updated: "2026-10-04"
|
|
state_hub_workstream_id: "831bbc30-30c0-5df3-a315-5355cd3b8ee5"
|
|
---
|
|
|
|
# Evaluate retrieval and real repository applicability
|
|
|
|
Part of `research/plan/ResearchProgram.md`. Phase prerequisite: FEP-WP-0003's evidence
|
|
gate. This is planned follow-up, not an active background execution request.
|
|
Individual task descriptions identify any work that can start earlier.
|
|
|
|
## Validate metadata and export the graph
|
|
|
|
```task
|
|
id: FEP-WP-0004-T01
|
|
status: wait
|
|
depends_on: [FEP-WP-0009-T03]
|
|
blocking_reason: "After schema and seed examples pass M1 (FEP-WP-0009-T03)."
|
|
priority: high
|
|
state_hub_task_id: "4fec98f6-fee2-52fc-a01c-f490c79ee0f8"
|
|
```
|
|
|
|
After the schema and seed examples pass M1, implement the smallest validator/exporter needed to check unique IDs, allowed values, relation targets and evidence references. Add meaningful invalid and valid fixtures; preserve prose and source provenance.
|
|
|
|
## Evaluate held-out retrieval cases
|
|
|
|
```task
|
|
id: FEP-WP-0004-T02
|
|
status: wait
|
|
depends_on: [FEP-WP-0004-T01, FEP-WP-0003-T01, FEP-WP-0003-T02]
|
|
blocking_reason: "Needs canonical export (FEP-WP-0004-T01) and an accepted M2 reassessment following FEP-WP-0003-T01/T02; completed editorial review currently says revise/defer."
|
|
priority: medium
|
|
state_hub_task_id: "4bdd4c53-fcdb-5dfd-9a37-ebc3fee4d445"
|
|
```
|
|
|
|
After M2, freeze a corpus revision and evaluate at least six unseen prompts following ValidationCases.md. Record expected alternatives, abstentions, unsupported claims and missed constraints; compare with manual navigation. Address Q07.
|
|
|
|
## Test against real repository revisions
|
|
|
|
```task
|
|
id: FEP-WP-0004-T03
|
|
status: wait
|
|
depends_on: [FEP-WP-0004-T02]
|
|
blocking_reason: "After the held-out retrieval evaluation (FEP-WP-0004-T02)."
|
|
priority: medium
|
|
state_hub_task_id: "ab66848b-1045-54dd-9711-52c5560faafa"
|
|
```
|
|
|
|
Study at least two contrasting accessible repositories at recorded revisions. Separate observed implementation from inferred intent. Write research/reviews/M3-EmpiricalReview.md, publish limitations, and hand remaining findings into live work records.
|
|
|
|
## User-directed early reuse loop — 2026-09-27
|
|
|
|
WP-0008 owns an early qualified-reference pilot and consumer outcome instrumentation,
|
|
authorized by the user's request for autonomous ecosystem improvement. It may proceed
|
|
before M1/M2. It is distinct from this workplan's canonical export and held-out M3
|
|
claims; it does not rename consumed feedback into unseen evaluation cases. This plan
|
|
remains proposed with its original task gates. Routine reuse must not await those
|
|
claims or require a human survey; preserve evidence provenance and honest unknowns.
|
|
|
|
## Closeout audit — 2026-09-27
|
|
|
|
Workplan blocked. T01 waits for the schema/seed M1 acceptance; T02 and T03 wait for
|
|
M2 under the phase prerequisite. The M1 and M2 reviews both say revise/defer. Reopen
|
|
each task only on the recorded prerequisite evidence. Existing preview validation
|
|
and development cases are useful outputs of WP-0007/WP-0008, not canonical export,
|
|
held-out evaluation or two complete real-repository applicability studies.
|
|
|
|
|
|
## Blocked-task review — 2026-10-04
|
|
|
|
[Dependency requirements](../docs/DependencyRequirements.md) establish exact owner
|
|
returns and local evidence gates for every remaining task. All unfinished tasks
|
|
stay `wait`; this plan is `blocked` because no currently available implementation
|
|
can meet its remaining acceptance. Completed deliverables remain done with their
|
|
recorded limits. Cross-repo source/synthetic proofs and finished editorial reviews
|
|
are not the missing runtime admission, actual consumer or human evidence.
|