fix: preserve unknown observed model evidence
All checks were successful
ci / validate (push) Successful in 4m11s
All checks were successful
ci / validate (push) Successful in 4m11s
Assistant: codex Assistant-Model: gpt-6.1-sol Assistant-Session: 01a10238-7ecc-78e2-903d-075f50e913cc
This commit is contained in:
parent
dc49415c9f
commit
5ce4f38aed
9 changed files with 114 additions and 5 deletions
|
|
@ -0,0 +1,38 @@
|
|||
# EMERGENT-WP-0002: session proof and readiness review
|
||||
|
||||
Knowledge 0.2.1; package digest d8ef41bacd2c6551ae3b87628004a5df6943928be1c93eb01e09efc6ad782767; emergent-security commit f9cba6fb35425f74e34401b1baf09f6057ace493.
|
||||
Scoped technical review by delegated Codex worker, 2026-10-03. No profile or
|
||||
credential activated and no production request made.
|
||||
|
||||
GLAS-WP-0008 is finished: blocked profiles refuse before sandbox creation;
|
||||
unverified means a labeled bounded attempt; ready requires positive evidence.
|
||||
GLAS-WP-0012 remains blocked on the owner delivery and real-provider acceptance
|
||||
already recorded in T02–T05. A direct Codex/Claude/Grok CLI comprehension run
|
||||
cannot close those gates or establish credential isolation.
|
||||
|
||||
Current contract supports start_session, dispatch_tool, end_session and cleanup;
|
||||
it supplies a generated task file and opaque bounded rein run. There is no
|
||||
provider resume, compaction or provider-switch event API in this boundary.
|
||||
The generic task description can request orientation loading, but it is no
|
||||
attestation that the provider loaded or retained knowledge. Session state in
|
||||
the Python adapter is not a provider conversation/session identifier.
|
||||
|
||||
Local correction: configured model pins no longer fill missing resolved_model
|
||||
in gateway or either adapter. Provider/rein-reported model remains evidence;
|
||||
absence remains null. A rein report may still be an alias, not an exact model
|
||||
revision. No session identifier or exact revision is fabricated from a profile.
|
||||
Tests check missing reports and distinct observed models on both adapters.
|
||||
|
||||
Acceptance requirements retained in GLAS-WP-0012 T04: after owner readiness,
|
||||
record profile/runtime/rein versions, provider-reported exact model and session
|
||||
ID (unknown if unavailable), package digest and source status, startup retrieval
|
||||
and comprehension, then provider-observed resume/compaction and repository /
|
||||
provider transitions with post-event reload receipts and scoped comprehension.
|
||||
Do not simulate an event and label it actual continuity. Preserve failed runs
|
||||
and proof scope. Until native event evidence exists, EMERGENT T05/T07 remains
|
||||
incomplete. These conditions extend existing proof requirements, not readiness.
|
||||
|
||||
Validation: focused gateway/adapter tests 68 passed; full `tests/` suite
|
||||
139 passed with `.venv/bin/python -m pytest tests/ -q -p no:cacheprovider`.
|
||||
`git diff --check` passed. These are deterministic regression tests, not
|
||||
native provider continuity, readiness or isolation proof.
|
||||
|
|
@ -178,3 +178,12 @@ of the returned sandbox. Errors preparing its reachability or SandboxHandle
|
|||
fail at session_start, with destruction still attempted. Readiness here is the
|
||||
owner lifecycle state; it does not promote a Glas profile's operational
|
||||
readiness or independently prove runtime/provider health.
|
||||
|
||||
|
||||
## Observed model evidence (2026-10-03)
|
||||
|
||||
`resolved_model` is supplied only by a rein execution report. A configured
|
||||
model pin is a requested route, and does not fill absent observed metadata.
|
||||
Unknown remains null, including failure before provider execution. A reported
|
||||
alias does not attest an exact provider revision or conversation identifier.
|
||||
See `docs/evidence/2026-10-03-emergent-session-readiness-review.md`.
|
||||
|
|
|
|||
|
|
@ -308,7 +308,6 @@ def _build_result(
|
|||
resolved_model = (
|
||||
(summary.resolved_model if summary else None)
|
||||
or (tool_result.resolved_model if tool_result else None)
|
||||
or (profile.model.model if profile else None)
|
||||
)
|
||||
tokens_spent = (
|
||||
summary.tokens_spent if summary and summary.tokens_spent is not None
|
||||
|
|
|
|||
|
|
@ -122,7 +122,7 @@ class ReinAharness(Rein):
|
|||
tokens_spent=last_result.get("tokens_spent"),
|
||||
cost_usd=last_result.get("cost_usd"),
|
||||
duration_s=last_result.get("execution_time_s"),
|
||||
resolved_model=last_result.get("model") or self.model,
|
||||
resolved_model=last_result.get("model"),
|
||||
metadata={
|
||||
"tool_profile": last_result.get("tool_profile") or self.tool_profile,
|
||||
},
|
||||
|
|
@ -167,7 +167,7 @@ class ReinAharness(Rein):
|
|||
tokens_spent=last_result.get("tokens_spent"),
|
||||
cost_usd=last_result.get("cost_usd"),
|
||||
duration_s=last_result.get("execution_time_s"),
|
||||
resolved_model=last_result.get("model") or self.model,
|
||||
resolved_model=last_result.get("model"),
|
||||
artifacts=[head_after] if committed and head_after else [],
|
||||
metadata={
|
||||
"tool_profile": last_result.get("tool_profile") or self.tool_profile,
|
||||
|
|
|
|||
|
|
@ -101,7 +101,7 @@ class ReinOpenWeights(Rein):
|
|||
events_completeness="unavailable",
|
||||
tokens_spent=last_result.get("tokens_spent"),
|
||||
duration_s=last_result.get("execution_time_s"),
|
||||
resolved_model=last_result.get("model") or self.model,
|
||||
resolved_model=last_result.get("model"),
|
||||
metadata={
|
||||
"turns": last_result.get("turns"),
|
||||
"tool_profile": last_result.get("tool_profile") or self.tool_profile,
|
||||
|
|
@ -125,7 +125,7 @@ class ReinOpenWeights(Rein):
|
|||
reason=reason,
|
||||
tokens_spent=last_result.get("tokens_spent"),
|
||||
duration_s=last_result.get("execution_time_s"),
|
||||
resolved_model=last_result.get("model") or self.model,
|
||||
resolved_model=last_result.get("model"),
|
||||
artifacts=[head_after] if committed and head_after else [],
|
||||
metadata={
|
||||
"turns": last_result.get("turns"),
|
||||
|
|
|
|||
|
|
@ -597,3 +597,21 @@ def test_cleanup_failure_never_captures_artifact():
|
|||
assert not result.ok
|
||||
capture.assert_not_called()
|
||||
manager.destroy.assert_called_once()
|
||||
|
||||
|
||||
def test_missing_provider_model_does_not_become_configured_pin() -> None:
|
||||
class UnknownModelRein(_FakeRein):
|
||||
def dispatch_tool(self, session, tool_call):
|
||||
return ToolResult(ok=True)
|
||||
|
||||
def end_session(self, session):
|
||||
return ExecutionSummary(outcome="succeeded")
|
||||
|
||||
manager = MagicMock()
|
||||
manager.create.return_value = _fake_status()
|
||||
result = run_execution(
|
||||
_request(), catalog=_catalog_with_readiness(),
|
||||
rein=UnknownModelRein(), manager=manager,
|
||||
)
|
||||
assert result.evidence.resolved_model is None
|
||||
assert result.evidence.model_route is not None
|
||||
|
|
|
|||
|
|
@ -226,3 +226,20 @@ def test_profile_native_limits_are_carried_in_owned_task_file():
|
|||
assert transport.write_task_file.call_args.args[0]["max_turns"] == 3
|
||||
with pytest.raises(ValueError,match="Glas-generated"):
|
||||
rein.start_session(profile,{"task_file":"/untrusted.json"},object())
|
||||
|
||||
|
||||
@pytest.mark.parametrize("reported", [None, "provider-observed-model"])
|
||||
def test_resolved_model_requires_rein_report(reported) -> None:
|
||||
rein = ReinAharness(model="configured-pin")
|
||||
transport = MagicMock(spec=ExecutionTransport)
|
||||
payload = {"ok": True}
|
||||
if reported is not None:
|
||||
payload["model"] = reported
|
||||
transport.run.return_value = MagicMock(returncode=0, stdout=json.dumps(payload), stderr="")
|
||||
transport.git_head.return_value = "new-head"
|
||||
session = {"task_file": "/tmp/task.json", "transport": transport,
|
||||
"timeout_seconds": 30, "head_before": "old-head"}
|
||||
with patch.object(rein, "_bin", return_value="/usr/bin/rein_aharness"):
|
||||
result = rein.dispatch_tool(session, ToolCall(name="run_task"))
|
||||
assert result.resolved_model == reported
|
||||
assert rein.end_session(session).resolved_model == reported
|
||||
|
|
|
|||
|
|
@ -174,3 +174,20 @@ def test_cleanup_session_removes_generated_task_file() -> None:
|
|||
{"transport": transport, "task_file": "/sandbox/task.json", "generated_task_file": True}
|
||||
)
|
||||
transport.remove_file.assert_called_once_with("/sandbox/task.json")
|
||||
|
||||
|
||||
@pytest.mark.parametrize("reported", [None, "provider-observed-model"])
|
||||
def test_resolved_model_requires_rein_report(reported) -> None:
|
||||
rein = ReinOpenWeights(model="configured-pin")
|
||||
transport = MagicMock(spec=ExecutionTransport)
|
||||
payload = {"ok": True}
|
||||
if reported is not None:
|
||||
payload["model"] = reported
|
||||
transport.run.return_value = MagicMock(returncode=0, stdout=json.dumps(payload), stderr="")
|
||||
transport.git_head.return_value = "new-head"
|
||||
session = {"task_file": "/tmp/task.json", "transport": transport,
|
||||
"timeout_seconds": 30, "head_before": "old-head"}
|
||||
with patch.object(rein, "_bin", return_value="/usr/bin/rein_openweights"):
|
||||
result = rein.dispatch_tool(session, ToolCall(name="run_task"))
|
||||
assert result.resolved_model == reported
|
||||
assert rein.end_session(session).resolved_model == reported
|
||||
|
|
|
|||
|
|
@ -412,3 +412,14 @@ started through its owner on railiance01, SECRETS-WP-0009-T03 step B2) and
|
|||
T04/T05 all require that external owner delivery. Nothing in this repo can
|
||||
advance them today; `status: blocked` stands unchanged. Re-review when the
|
||||
runtime rebuild/placement or owner authentication lands.
|
||||
|
||||
|
||||
## EMERGENT-WP-0002 proof requirements — 2026-10-03
|
||||
|
||||
Scoped review and local observed-model correction:
|
||||
`docs/evidence/2026-10-03-emergent-session-readiness-review.md`.
|
||||
T04 retains actual native-provider startup and continuity acceptance, including
|
||||
reload after real resume/compaction, provider/repository changes and exact
|
||||
available model/session metadata. The current lifecycle exposes no continuity
|
||||
event API. Existing owner-delivery blockers and all task/readiness states remain;
|
||||
local deterministic tests are not governed provider or isolation acceptance.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue