From 776579c660ee7386ce8f26e5d20e6c2cbc4f1b63 Mon Sep 17 00:00:00 2001 From: tegwick Date: Sat, 5 Sep 2026 20:50:43 +0200 Subject: [PATCH] Extend PQRST-WP-0002: land the practice in hall-of-helix MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The fleet already has a session-closing ritual — the operator asks for a hall-of-helix entry at wind-down — but it is unspecified, and PQRST is not part of it. The one estimate in the hall was added by hand as `30/23/18/19/10`, a format the spec does not define. Two tasks added: T04 — Specify the closing routine in a new CLOSING.md, quoting the operator's wind-down sentence so an agent recognises the situation, and link it from README.md, ENTRY.md, and the AGENTS.md Close protocol. Takes a position on two things the routine is silent on: the estimate covers the substantive session and excludes the closing ritual itself (otherwise the ritual inflates T on every session that has a seat), and the prompt is reached by path with only the output block inlined. Constrains the change against both repos' INTENT.md: a record on a seat, never a score on a worker. T05 — Carry the record in entries as a quoted canonical signature in frontmatter plus a `## PQRST estimate` section holding the full stored record, with templates/entry.md, ENTRY.md, and check-entries.py updated to match. Required for agent-session seats after adoption; human seats exempt; the 102 existing seats grandfathered. Explicitly forbids backfilling estimates for sessions nobody observed — that is the fabricated evidence spec R4 forbids. Only the one manual estimate is normalised, and it already sums to 100. T01 gains a sixth pilot question — whether a session estimating itself inflates P — since the hall entry is written by the same session being audited. T02 now evaluates hall-of-helix as the default answer to the storage question rather than weighing it from scratch, with its sampling bias stated. T03 picks up the ritual-scope and self-estimation findings. Execution order T01 → T04 → T05 → T02 → T03 is stated in the workplan; task ids are identity, not sequence. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01SQ6oF1DtVDKcD1FCpvRVLx Assistant: claude-code Assistant-Model: opus Assistant-Process: 272883@bnt-lap001 Assistant-Session: f40c8f53-fb65-4980-9d29-bcdb3dd946f7 --- ...0002-validate-v01-against-real-sessions.md | 197 ++++++++++++++++-- 1 file changed, 177 insertions(+), 20 deletions(-) diff --git a/workplans/PQRST-WP-0002-validate-v01-against-real-sessions.md b/workplans/PQRST-WP-0002-validate-v01-against-real-sessions.md index bcdbfff..0cdbbfc 100644 --- a/workplans/PQRST-WP-0002-validate-v01-against-real-sessions.md +++ b/workplans/PQRST-WP-0002-validate-v01-against-real-sessions.md @@ -1,7 +1,7 @@ --- id: PQRST-WP-0002 type: workplan -title: "Validate spec v0.1 against real sessions and cut v0.2" +title: "Validate spec v0.1 in real session closes and land PQRST in hall-of-helix" domain: agents repo: pqrst-practice status: proposed @@ -13,25 +13,55 @@ updated: "2026-09-05" quality_dor: DoR-Ok quality_dor_at: "2026-09-05" quality_dor_by: claude-code -quality_dor_note: "v0.1 consolidates two drafts that were written without either being applied to a session. Every open question is empirical — whether the prompt returns a valid record unassisted, whether attribution by primary purpose is decidable in practice, and where records are stored — so the next change is a bounded pilot, not more specification." +quality_dor_note: "v0.1 consolidates two drafts that were written without either being applied to a session. Every open question is empirical — whether the prompt returns a valid record unassisted, whether attribution by primary purpose is decidable in practice, and where records are stored. hall-of-helix supplies both the pilot population and the answer to the storage question: the operator already asks agents for a closing entry there, and one estimate has been added by hand, so the routine exists and is simply unspecified." related: - PQRST-WP-0001 +repos: + - pqrst-practice + - hall-of-helix state_hub_workstream_id: "6a875b19-5a76-55c1-bd1f-2f5005cd416b" --- -# Validate spec v0.1 against real sessions and cut v0.2 +# Validate spec v0.1 in real session closes and land PQRST in hall-of-helix Spec v0.1 was consolidated from two independent drafts, neither of which had been applied to an actual session. It is internally consistent and entirely unvalidated. -The purpose of this workplan is to find out where the practice is wrong by using -it, then cut v0.2 from the findings. It deliberately does **not** build tooling -or a record store — both are out of scope for this repository (`SCOPE.md`). +There is already a session-closing ritual in the fleet. The operator ends a +session with some form of: -**Done when:** the prompt has been run unassisted at the end of at least eight -real sessions, the friction is written down, the storage boundary is decided, -and v0.2 either incorporates the findings or records why v0.1 stands. +> "Ok, lets close down this session then. Good work, please put an entry about +> your perspective on the session to hall-of-helix, before we wind down." + +That ritual is real, repeated, and under-specified. An agent receiving it has to +infer the routine from `ENTRY.md`, and PQRST is not part of that routine at all — +the one estimate in the hall +(`entries/2026-09-05T16:36:12.000Z-codex-statehub-snapshot-and-signature.md`, +`pqrst_estimate: 30/23/18/19/10`) was added by hand, in a format that is not the +one the specification defines. + +This workplan closes both gaps together: use real session closes as the pilot +population, and make the closing routine explicit in hall-of-helix so that an +agent reading that prompt finds — without being told twice — that closing a +session means running the PQRST prompt and recording the result in its entry. + +It deliberately does **not** build tooling or a record store in *this* +repository. Both are out of scope (`SCOPE.md`); the hall is the store, and the +hall owns its own validation. + +**Done when:** the closing routine is specified in hall-of-helix and reachable +from the prompt above, entries carry a validated PQRST record, the prompt has +been run unassisted at the end of at least eight real sessions, and v0.2 either +incorporates the findings or records why v0.1 stands. + +**Execution order:** T01 → T04 → T05 → T02 → T03. Task ids are identity, not +sequence; the hall changes (T04, T05) land early so the pilot runs through the +real routine rather than beside it. + +**Cross-repo:** T04 and T05 change `hall-of-helix`, not this repo. Open a +companion `HOH-WP` (or an `ADHOC` record) there referencing `PQRST-WP-0002`, and +land the edits under it. ## Pilot the closing prompt on real sessions @@ -44,7 +74,9 @@ state_hub_task_id: "dd259b3f-e9ae-5bdd-8e02-07ae0083930a" Paste `PqrstPrompt.md` unmodified at the end of at least **eight** real agentic coding sessions, spread across at least three repositories and at least two task -classes (e.g. `feature`, `bugfix`, `explore`, `harden`). +classes (e.g. `feature`, `bugfix`, `explore`, `harden`). Once T04 has landed, +these are ordinary hall closes — the pilot is observation of the routine, not a +separate exercise. Do not coach the model, do not repair a malformed answer, and do not re-run the prompt for a better result. A rejected record is the finding. @@ -67,6 +99,10 @@ The specific questions to answer: 4. Do estimates cluster at coarse increments (R7), or does false precision reappear? 5. Is `Confidence` used meaningfully, or is it always `medium`? +6. **Does an agent estimating its own session inflate P?** The hall entry is a + first-person account written by the same session being audited. Self-report + bias is a live threat to the whole practice and the hall is where it will + first be visible. ## Decide the record-storage boundary @@ -82,16 +118,29 @@ records go instead. Until that is answered the practice produces records with nowhere to live, and the trend analysis that justifies the whole exercise (spec §8, §9) cannot happen. -Decide and record the answer as a decision (`POST /decisions/`, then an ADR in -this repo if the decision has repo-local consequences). Options to weigh: +**hall-of-helix is now the leading candidate and should be evaluated as the +default**, not weighed from scratch: one record per session, written at the +moment of closing, already carrying `repos`, `session_id`, `exact_model`, +`harness`, and a first-person account of what the session actually did. That is +richer session metadata than a purpose-built store would likely collect, and +spec §9 asks for exactly those companions. -- a State Hub surface (progress events already carry free text; a dedicated - record type would need a hub change and is a bigger commitment); -- an append-only file in each consuming repository; -- a separate collection repository outside this one. +Weigh honestly what the hall is **not**: -Whatever is chosen, this repository still ships no collector — the decision -names the owner, it does not move the work here. +- it is a hall of remembrance, not a metrics warehouse — spec §8 trend queries + mean grepping Markdown, and `INTENT.md` forbids turning entries into a + leaderboard; +- seats are voluntary and sometimes drafts, so the record set is a biased + sample of sessions, not a census; +- not every session ends with a seat, and forcing one would corrupt both + practices. + +Decide, and record the answer as a decision (`POST /decisions/`, then an ADR in +this repo if there are repo-local consequences). If the hall is chosen, say +plainly whether it is the *only* store or the human-facing one alongside a +machine-facing collector owned elsewhere. Whatever is chosen, this repository +still ships no collector — the decision names the owner, it does not move the +work here. ## Cut spec v0.2 @@ -102,16 +151,22 @@ priority: medium state_hub_task_id: "0a0bce15-ac0c-582a-842c-ae1bf0307afb" ``` -From the pilot findings, revise the specification and prompt together: +From the pilot findings and the hall integration, revise the specification and +prompt together: - fix rules the pilot showed to be undecidable or routinely ignored; - tighten or relax the validation rules based on what actually failed; +- settle whether the closing ritual itself (writing the entry, rendering the + portrait, syncing) is inside or outside the estimate — T04 takes a position; + the pilot says whether it holds; +- state how self-estimation bias is to be handled, if T01 finds any; - add the storage-boundary decision from T02 as a `SCOPE.md` cross-reference; - bump to **v0.2** and add a row to the spec's Appendix B. If a change alters a dimension's meaning, the validation rules, or the stored record format, say plainly in Appendix B that records made under v0.1 are less -comparable. +comparable. A record-format change also means a matching change in +hall-of-helix — name it in the handoff. If the pilot surfaces nothing substantive, record that outcome and leave v0.1 standing. "No change needed, and here is the evidence" is a valid result and a @@ -123,6 +178,108 @@ Run the verification checklist in `AGENTS.md` before committing, then sync: uv run --project ~/repo-manager rmgr sync --path . --push ``` +## Specify the session-closing routine in hall-of-helix + +```task +id: PQRST-WP-0002-T04 +status: todo +priority: high +``` + +Make the closing routine explicit and findable, so an agent that receives the +operator's wind-down prompt knows the whole sequence without asking. + +**Deliverable:** `CLOSING.md` in the hall-of-helix root — the routine in order, +one page, no prose detour. Quote the operator's actual wind-down sentence near +the top so an agent scanning for it recognises the situation it is in. + +Link it from the three places an agent lands: + +- `README.md`, beside "How to leave a seat"; +- `ENTRY.md`, at the top — `ENTRY.md` says how to *write a seat*, `CLOSING.md` + says what *closing a session* consists of, and the entry is one step of it; +- `AGENTS.md`, in the **Close** section of the session protocol. + +**The routine to specify:** + +1. Finish the substantive work. Update workplan task statuses. +2. **Run the PQRST prompt** — `~/pqrst-practice/PqrstPrompt.md` — unmodified and + uncoached. Validate the result against spec §5.5 before using it: five + integers summing to 100, `Confidence` present, `Dominant factors` naming + concrete session facts rather than restating the numbers. +3. Write the entry per `ENTRY.md`, carrying the record (format per T05). +4. `make check`; add the seat to the `README.md` list. +5. Log progress to State Hub; sync the repo. + +**Decide and state two things the routine is otherwise silent on:** + +- **Scope of the estimate.** The estimate covers the *substantive session*, not + the closing ritual — writing the entry, rendering the portrait, and syncing + are excluded. Spec R1 says the estimate is produced after the substantive work + is complete, and letting the ritual score itself would inflate T on every + session that has a seat. State this explicitly in `CLOSING.md`; T03 revisits + it if the pilot disagrees. +- **How the prompt is reached.** Referencing `~/pqrst-practice/PqrstPrompt.md` + keeps one source of truth but assumes the checkout exists; inlining the prompt + in the hall makes closes self-contained but forks the canonical text. The + recommendation is to **reference the path and inline nothing but the output + block**, so a session with no `pqrst-practice` checkout can still produce a + well-formed record. Whichever is chosen, `pqrst-practice` stays canonical and + the hall says so. + +**Constraint:** the hall is not a metrics programme. `INTENT.md` there is +explicit that seats are not ranked and the hall is not a leaderboard, and +`INTENT.md` here is explicit that PQRST fails the moment anyone scores well on +it. The routine adds a record to a seat; it does not add a score to a worker. +Do not introduce comparison, aggregation, or ranking of estimates into +hall-of-helix. + +## Carry the PQRST record in hall entries + +```task +id: PQRST-WP-0002-T05 +status: todo +priority: high +``` + +Give the record a defined home in an entry, so it is machine-collectable without +losing the evidence that makes it auditable. + +**Format** — both halves, since either alone is useless: + +- **Frontmatter:** `pqrst_estimate: "P30 Q23 R18 S19 T10"` — the canonical + signature from spec §5.2, quoted. Prefer it over the existing ad-hoc + `30/23/18/19/10`: it is self-describing, survives reordering, and is matched by + the same regex the `pqrst-practice` verification checklist already uses. This + means normalising the one existing entry (see below). +- **Body:** a `## PQRST estimate` section holding the full stored record from + spec §5.1 — the block, `Confidence`, and `Dominant factors`. Spec §10 rule 8 + is blunt that a signature without its dominant-factors sentence is not + auditable, and `ENTRY.md` already refuses "various improvements" for the same + reason. + +**Changes:** + +- `templates/entry.md` — add the frontmatter field and the section, with the + section placed after "Durable legacy" and before "Visual prompt". +- `ENTRY.md` — add `pqrst_estimate` to the Required frontmatter table and + "PQRST estimate" to the Required sections list, each with one line on what + makes it valid, and link `CLOSING.md`. +- `scripts/check-entries.py` — when `pqrst_estimate` is present, validate the + signature format and that it sums to 100; make it **required for + `worker_kind: agent-session` entries created after the adoption date**, and + add "PQRST estimate" to `REQUIRED_HEADINGS` under the same condition. Human + seats are exempt. `make check` must still pass on all 102 existing seats. + +**Do not backfill.** Existing seats predate the practice and nobody observed +those sessions; inventing estimates for them would be exactly the fabricated +evidence spec R4 and §5.5 rule 4 forbid, and it would poison the first trend +data the practice ever produces. The only record touched is the manual +`30/23/18/19/10` in the 2026-09-05 Codex seat, normalised to +`"P30 Q23 R18 S19 T10"` — same numbers, canonical spelling, and it already sums +to 100. Ask its author before rewriting the seat, or leave it and note the +exception in `CLOSING.md`. + ## Pilot findings _Populated by T01. One subsection per session: repository, task class, the