28 lines
1.6 KiB
Markdown
28 lines
1.6 KiB
Markdown
|
# Compatibility executor defect log
|
||
|
|
|
||
|
|
Baseline for the FI-WP-0005-T07 canary comparison: defects produced by the
|
||
|
|
current `rein-aharness fi-research-brief` path (single llm-connect call, no
|
||
|
|
network access, no primary sources). The canary must not reproduce these.
|
||
|
|
|
||
|
|
| Date | Defect | Class | Evidence |
|
||
|
|
| --- | --- | --- | --- |
|
||
|
|
| 2026-08-04…14 | DeepSeek-V4-Flash-0731 reported as a 12B dense model; once claimed as already collected | invented facts; false collection claim | Corrected in `inventory/catalog/deepseek-ai__DeepSeek-V4-Flash-0731__strategic.yaml` history (FI-WP-0004-T05) |
|
||
|
|
| 2026-09-23 | "Qwen3.8-27B has been moved from 'candidate' to 'approved'" | invented state change; contradicts the catalog | Catalog unchanged since `a78187c`; `approved_by` empty. Correction appended to `briefs/2026/09/2026-09-23.md` |
|
||
|
|
|
||
|
|
## Pattern
|
||
|
|
|
||
|
|
Both defects are assertions about **lab state** that the brief had in its own
|
||
|
|
context and still got wrong, not stale knowledge about the outside world. The
|
||
|
|
executor receives the catalog and the previous brief, then narrates a change
|
||
|
|
that no file records.
|
||
|
|
|
||
|
|
Consequences for the FI-WP-0005 acceptance run (T07):
|
||
|
|
|
||
|
|
1. Compare canary output against the catalog for every claim about collection
|
||
|
|
status, sizes and licenses. Any assertion of a state change must cite the
|
||
|
|
file and revision that made it.
|
||
|
|
2. Briefs with no external delta should say so, rather than manufacturing an
|
||
|
|
internal one. Both defect days had genuinely quiet outside channels.
|
||
|
|
3. A false claim propagates: the previous brief is context for the next one.
|
||
|
|
Corrections belong in the brief file, not only in the workplan.
|