diff --git a/.custodian-brief.md b/.custodian-brief.md index d8ed0fd..d7301eb 100644 --- a/.custodian-brief.md +++ b/.custodian-brief.md @@ -2,13 +2,13 @@ # Custodian Brief — glas-harness **Domain:** infotech -**Last synced:** 2026-09-05 17:22 UTC +**Last synced:** 2026-09-05 17:34 UTC **State Hub:** http://127.0.0.1:8000 *(adjust if running on a remote machine)* ## Active Workstreams ### Prove the first local rein profile end to end -Progress: 1/5 done | workplan_id: `170bf1ae-337f-5553-8d1e-03b07100e08f` +Progress: 2/6 done | workplan_id: `170bf1ae-337f-5553-8d1e-03b07100e08f` **Open tasks:** - ! Resolve and verify the owner runtime, credential, and egress contract `4209f564` diff --git a/WORK-RECORDS.md b/WORK-RECORDS.md index 085a7d9..351aa22 100644 --- a/WORK-RECORDS.md +++ b/WORK-RECORDS.md @@ -66,6 +66,7 @@ | task | GLAS-WP-0012-T03 | wait | — | workplans/GLAS-WP-0012-first-local-profile-production-proof.md | | task | GLAS-WP-0012-T04 | wait | — | workplans/GLAS-WP-0012-first-local-profile-production-proof.md | | task | GLAS-WP-0012-T05 | wait | — | workplans/GLAS-WP-0012-first-local-profile-production-proof.md | +| task | GLAS-WP-0012-T06 | done | — | workplans/GLAS-WP-0012-first-local-profile-production-proof.md | | intake | GLAS-IN-0001 | done | blue | docs/intakes/residuals.md | | intake | GLAS-IN-0002 | todo | red | docs/intakes/residuals.md | | intake | GLAS-IN-0003 | todo | green | docs/intakes/residuals.md | diff --git a/docs/local-profile-acceptance.md b/docs/local-profile-acceptance.md index f13bdc4..72a8882 100644 --- a/docs/local-profile-acceptance.md +++ b/docs/local-profile-acceptance.md @@ -123,3 +123,39 @@ retarget existing consumers or enable production schedules. The open-weight profile remains blocked until its own route and real-rein proof pass. Keep that portion of `GLAS-IN-0002` live after this workplan closes. + +## Prepared acceptance runner + +After the owner requirements and candidate review pass, run: + +```bash +.venv/bin/python scripts/prove-local-profile.py \ + --harness-profile harness.agent-dev-local@1.1.0 +``` + +Use `--profile-dir ` if the versioned candidate is staged +outside the packaged catalog. The runner requires an explicit version, the +local Claude route, nonempty credential route references, and bounded token/time +limits. It preserves catalog readiness and refuses blocked profiles before +creating even the disposable fixture. There is no force or readiness override. + +The fixture wraps the catalog-built `ReinAharness` to observe its result and +cleanup, while delegating actual session setup and agent dispatch. It checks +the committed and working-tree file contents, direct parent commit, changed +paths, clean tree (including ignored files), source absence, exact child +identity, and task mode inside owner execution. Task removal is checked before +destroy; host source integrity and workspace absence are checked afterward. + +Output includes configuration references, requested model, commit id, and +boolean checks. It excludes raw model/provider output, task text, and exception +messages. Exit zero means this real-run artifact/lifecycle acceptance passed; +`readiness_review_required` remains true. The owner runtime/deployment pins, +credential-delivery proof, enforced egress and negative-route checks remain +separate prerequisites for T02/T05. The runner does not update the catalog or +publish artifacts. + +Validation of the runner itself: its focused tests use local disposable Git +repositories and mocked adapter boundaries, covering extra commits, extra +paths, bad content, untracked/ignored artifacts, source visibility, file mode, +strict boolean inspection results, redacted failures, and profile refusal. +These tests do not count as the pending real-rein acceptance run. diff --git a/scripts/prove-local-profile.py b/scripts/prove-local-profile.py new file mode 100644 index 0000000..ff2173e --- /dev/null +++ b/scripts/prove-local-profile.py @@ -0,0 +1,212 @@ +"""Real-rein acceptance fixture for GLAS-WP-0012; never overrides readiness.""" + +import argparse +import json +import subprocess +import tempfile +from pathlib import Path + +from sandboxer.core.manager import SandboxManager +from sandboxer.lifecycle.store import SandboxStore +from sandboxer.payments.credits import CreditsStore +from sandboxer.snapshots.store import SnapshotStore + +from glas_harness.contract import ExecutionRequest, Rein +from glas_harness.gateway import run_execution +from glas_harness.profiles import ProfileCatalog +from glas_harness.reins.rein_aharness import ReinAharness + + +# Executed inside the sandbox; results contain only booleans and a Git object id. +INSPECTOR = r''' +import json, os, stat, subprocess, sys +from pathlib import Path +baseline, source, task, request_id = sys.argv[1:] +def git(*args): + return subprocess.check_output(['git', *args], stderr=subprocess.DEVNULL, + timeout=10) +head = git('rev-parse', 'HEAD').decode().strip() +artifact = Path('PROOF.md') +expected = b'Glas local profile proof.\n' +checks = { + 'one_commit': git('rev-list', '--parents', '-n', '1', 'HEAD').decode().split() + == [head, baseline], + 'only_expected_path': git('diff-tree', '--no-commit-id', '--name-only', '-r', + '-z', 'HEAD') == b'PROOF.md\x00', + 'clean_worktree': not git('status', '--porcelain', '--untracked-files=all', + '--ignored=matching').strip(), + 'regular_artifact': artifact.is_file() and not artifact.is_symlink(), + 'worktree_content': artifact.is_file() and artifact.read_bytes() == expected, + 'committed_content': git('show', 'HEAD:PROOF.md') == expected, + 'source_absent': not Path(source).exists(), + 'task_private': stat.S_IMODE(Path(task).stat().st_mode) == 0o600, + 'identity_exact': os.environ.get('SANDBOXER_ACTOR') == 'agt' + and os.environ.get('SANDBOXER_PROJECT') == 'glas-local-proof' + and os.environ.get('SANDBOXER_RUN_ID') == request_id, +} +print(json.dumps({'head': head, 'checks': checks})) +''' + +CHECKS = { + "one_commit", "only_expected_path", "clean_worktree", "regular_artifact", + "worktree_content", "committed_content", "source_absent", "task_private", + "identity_exact", +} + + +class ObservedRein(Rein): + """Observe the catalog-built adapter without replacing its agent dispatch.""" + + def __init__(self, delegate, source): + self.delegate = delegate + self.source = source + self.workspace = None + self.request_id = None + self.checks = {} + self.task_removed = False + + def start_session(self, profile, inputs, sandbox): + reachability = sandbox.reachability + if not reachability.get("workspace_dir") or reachability.get("ssh"): + raise RuntimeError("local proof requires an owner-managed local workspace") + self.workspace = reachability["workspace_dir"] + self.request_id = inputs["request_id"] + return self.delegate.start_session(profile, inputs, sandbox) + + def dispatch_tool(self, session, tool_call): + return self.delegate.dispatch_tool(session, tool_call) + + def end_session(self, session): + summary = self.delegate.end_session(session) + if summary.outcome != "succeeded": + return summary + result = session["transport"].run( + ["/usr/bin/python3", "-c", INSPECTOR, session["head_before"], + str(self.source), session["task_file"], self.request_id], timeout=30, + ) + if result.returncode: + raise RuntimeError("sandbox artifact inspection failed") + try: + report = json.loads(result.stdout) + checks = report["checks"] + self.checks = {key: checks.get(key) is True for key in CHECKS} + self.checks["head_matches_summary"] = report["head"] == summary.commit_sha + except (ValueError, KeyError, TypeError, AttributeError) as exc: + raise RuntimeError("invalid sandbox artifact inspection") from exc + if not all(self.checks.values()): + raise RuntimeError("sandbox artifact acceptance failed") + return summary + + def cleanup_session(self, session): + self.delegate.cleanup_session(session) + proc = session["transport"].run( + ["test", "!", "-e", session["task_file"]], timeout=15 + ) + self.task_removed = proc.returncode == 0 + if not self.task_removed: + raise RuntimeError("generated task file survived cleanup") + + +def select_candidate(reference, catalog): + if "@" not in reference: + raise ValueError("proof requires an exact version-pinned profile") + profile, descriptor = catalog.resolve(reference) + catalog.require_operational(profile) + if (profile.id != "harness.agent-dev-local" or descriptor.id != "rein-aharness" + or profile.model.route != "claude-code-cli" + or profile.model.provider != "anthropic"): + raise ValueError("proof requires the reviewed local Claude route") + if not profile.credential_route_refs: + raise ValueError("proof requires declared credential route references") + if (not profile.limits.timeout_seconds or profile.limits.timeout_seconds > 900 + or not profile.limits.budget_tokens or profile.limits.budget_tokens > 60000): + raise ValueError("proof requires explicit bounded token and time limits") + delegate = catalog.build_rein(profile, descriptor) + if type(delegate) is not ReinAharness: + raise ValueError("proof requires the actual ReinAharness adapter") + return profile, delegate + + +def git(source, *args): + return subprocess.check_output(["git", *args], cwd=source, + stderr=subprocess.DEVNULL, timeout=15) + + +def prove(reference, catalog): + # This gate precedes even the disposable fixture and sandbox manager. + profile, delegate = select_candidate(reference, catalog) + with tempfile.TemporaryDirectory(prefix="glas-real-rein-proof-") as directory: + root = Path(directory) + source = root / "source" + source.mkdir() + git(source, "init", "-q") + git(source, "config", "user.name", "Glas Acceptance") + git(source, "config", "user.email", "proof@example.invalid") + (source / "sentinel").write_bytes(b"unchanged\n") + git(source, "add", "sentinel") + git(source, "commit", "-qm", "fixture") + source_head = git(source, "rev-parse", "HEAD") + manager = SandboxManager( + store=SandboxStore(path=root / "sandboxes.json"), + credits=CreditsStore(path=root / "credits.json"), + snapshots=SnapshotStore(path=root / "snapshots.json"), + ) + observed = ObservedRein(delegate, source) + result = run_execution(ExecutionRequest( + harness_profile_ref=reference, repo=str(source), title="local profile proof", + description=("Create only PROOF.md containing exactly 'Glas local profile proof.' " + "followed by one newline. Make exactly one local commit. " + "Leave the working tree clean. Do not push or change other files."), + actor="agt", project="glas-local-proof", report_to_hub=False, + ), catalog=catalog, rein=observed, manager=manager) + source_unchanged = ( + (source / "sentinel").read_bytes() == b"unchanged\n" + and git(source, "rev-parse", "HEAD") == source_head + and not git(source, "status", "--porcelain", "--untracked-files=all", + "--ignored=matching").strip() + ) + status = manager.store.get(result.evidence.sandbox_id) if result.evidence.sandbox_id else None + destroyed = status is not None and status.state.value == "destroyed" + workspace_removed = bool(observed.workspace) and not Path(observed.workspace).exists() + passed = (result.ok and bool(observed.checks) and all(observed.checks.values()) + and observed.task_removed and source_unchanged and destroyed + and workspace_removed) + # Do not print tool_output, tool_error, exceptions, prompts, or raw owner stdout. + return { + "acceptance_passed": passed, "profile_ref": reference, + "profile_readiness": profile.operational_readiness.status, + "sandbox_profile": profile.sandbox_profile, + "requested_model": profile.model.model, + "model_route": profile.model.route, + "credential_route_refs": profile.credential_route_refs, + "limits": profile.limits.model_dump(exclude_none=True), + "request_id": result.evidence.request_id, + "sandbox_id": result.evidence.sandbox_id, + "commit_sha": result.evidence.commit_sha, + "outcome": result.evidence.outcome, + "failure_stage": result.evidence.failure_stage, + "artifact_checks": observed.checks, + "source_unchanged": source_unchanged, + "task_removed_before_teardown": observed.task_removed, + "destroyed": destroyed, "workspace_removed": workspace_removed, + "readiness_review_required": True, + } + + +def main(argv=None): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--harness-profile", required=True) + parser.add_argument("--profile-dir", type=Path) + args = parser.parse_args(argv) + try: + report = prove(args.harness_profile, ProfileCatalog(profile_dir=args.profile_dir)) + except Exception as exc: + # Configuration/provider errors can contain secret values; expose only the type. + report = {"acceptance_passed": False, "error_type": type(exc).__name__, + "readiness_review_required": True} + print(json.dumps(report, indent=2)) + return 0 if report["acceptance_passed"] else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tests/test_local_profile_proof.py b/tests/test_local_profile_proof.py new file mode 100644 index 0000000..96f9a14 --- /dev/null +++ b/tests/test_local_profile_proof.py @@ -0,0 +1,172 @@ +import importlib.util +import json +import subprocess +from pathlib import Path +from unittest.mock import MagicMock, patch + +import pytest + +from glas_harness.contract import ExecutionSummary, OperationalReadiness, ToolCall +from glas_harness.profiles import OperationallyBlockedProfileError, ProfileCatalog +from glas_harness.reins.rein_aharness import ReinAharness + + +spec = importlib.util.spec_from_file_location( + "local_profile_proof", Path(__file__).parents[1] / "scripts/prove-local-profile.py" +) +proof = importlib.util.module_from_spec(spec) +spec.loader.exec_module(proof) + + +def test_blocked_profile_refuses_before_fixture_or_manager_creation(): + with patch.object(proof.tempfile, "TemporaryDirectory") as temporary, \ + patch.object(proof, "SandboxManager") as manager: + with pytest.raises(OperationallyBlockedProfileError): + proof.prove("harness.agent-dev-local@1.0.0", ProfileCatalog()) + temporary.assert_not_called() + manager.assert_not_called() + + +def test_unpinned_profile_is_never_a_proof_candidate(): + with pytest.raises(ValueError, match="version-pinned"): + proof.select_candidate("harness.agent-dev-local", ProfileCatalog()) + + +@pytest.mark.parametrize("changes", [ + {"credential_route_refs": []}, + {"timeout_seconds": 901}, + {"timeout_seconds": None}, + {"budget_tokens": 60001}, + {"budget_tokens": None}, +]) +def test_candidate_requires_credentials_and_explicit_proof_bounds(changes): + catalog = ProfileCatalog() + profile, _ = catalog.resolve("harness.agent-dev-local@1.0.0") + updates = { + "operational_readiness": OperationalReadiness(status="unverified"), + "credential_route_refs": ["test-claude-route"], + "limits": profile.limits.model_copy(update={ + key: value for key, value in changes.items() if key != "credential_route_refs" + }), + } + if "credential_route_refs" in changes: + updates["credential_route_refs"] = changes["credential_route_refs"] + catalog.profiles()[(profile.id, profile.version)] = profile.model_copy(update=updates) + with pytest.raises(ValueError): + proof.select_candidate(str(profile.ref), catalog) + + +def test_observer_delegates_actual_dispatch_and_cleanup(): + delegate = MagicMock(spec=ReinAharness) + observer = proof.ObservedRein(delegate, Path("/absent-source")) + session = {"transport": MagicMock(), "task_file": "/workspace/.git/task.json"} + call = ToolCall(name="run_task") + assert observer.dispatch_tool(session, call) is delegate.dispatch_tool.return_value + delegate.dispatch_tool.assert_called_once_with(session, call) + session["transport"].run.return_value.returncode = 0 + observer.cleanup_session(session) + delegate.cleanup_session.assert_called_once_with(session) + session["transport"].run.assert_called_once_with( + ["test", "!", "-e", session["task_file"]], timeout=15 + ) + assert observer.task_removed + + +@pytest.mark.parametrize("report", [ + {"head": "abc", "checks": {}}, + {"head": "wrong", "checks": {key: True for key in proof.CHECKS}}, + {"head": "abc", "checks": {key: "true" for key in proof.CHECKS}}, +]) +def test_observer_rejects_incomplete_mismatched_or_untyped_inspection(report): + delegate = MagicMock(spec=ReinAharness) + delegate.end_session.return_value = ExecutionSummary( + committed=True, outcome="succeeded", commit_sha="abc" + ) + observer = proof.ObservedRein(delegate, Path("/absent-source")) + observer.request_id = "proof-request" + transport = MagicMock() + transport.run.return_value = subprocess.CompletedProcess([], 0, json.dumps(report), "") + with pytest.raises(RuntimeError, match="acceptance failed"): + observer.end_session({"transport": transport, "head_before": "before", + "task_file": "/workspace/.git/task.json"}) + + +@pytest.fixture +def repository(tmp_path): + repo = tmp_path / "sandbox" + repo.mkdir() + proof.git(repo, "init", "-q") + proof.git(repo, "config", "user.name", "Glas Proof Tests") + proof.git(repo, "config", "user.email", "proof@example.invalid") + (repo / "sentinel").write_text("unchanged\n") + proof.git(repo, "add", "sentinel") + proof.git(repo, "commit", "-qm", "baseline") + baseline = proof.git(repo, "rev-parse", "HEAD").decode().strip() + task = repo / ".git/task.json" + task.write_text("{}") + task.chmod(0o600) + (repo / "PROOF.md").write_text("Glas local profile proof.\n") + proof.git(repo, "add", "PROOF.md") + proof.git(repo, "commit", "-qm", "proof") + return repo, baseline, task + + +def inspect(repository, source): + repo, baseline, task = repository + result = subprocess.run( + ["/usr/bin/python3", "-c", proof.INSPECTOR, baseline, str(source), str(task), "req"], + cwd=repo, capture_output=True, text=True, timeout=15, + env={"PATH": "/usr/bin:/bin", "HOME": str(repo), + "SANDBOXER_ACTOR": "agt", "SANDBOXER_PROJECT": "glas-local-proof", + "SANDBOXER_RUN_ID": "req"}, + ) + assert result.returncode == 0, result.stderr + return json.loads(result.stdout) + + +def test_artifact_inspector_validates_content_and_parent_commit(repository, tmp_path): + report = inspect(repository, tmp_path / "absent-source") + assert set(report["checks"]) == proof.CHECKS + assert all(report["checks"].values()) + + +@pytest.mark.parametrize("fault,failed_check", [ + ("extra_commit", "one_commit"), + ("wrong_content", "committed_content"), + ("extra_path", "only_expected_path"), + ("untracked", "clean_worktree"), + ("ignored", "clean_worktree"), + ("source_visible", "source_absent"), + ("task_mode", "task_private"), +]) +def test_artifact_inspector_rejects_false_positive_commits(repository, tmp_path, + fault, failed_check): + repo, _, task = repository + source = tmp_path / "absent-source" + if fault == "extra_commit": + proof.git(repo, "commit", "--allow-empty", "-qm", "extra") + elif fault == "wrong_content": + (repo / "PROOF.md").write_text("wrong\n") + proof.git(repo, "commit", "-qam", "wrong", "--amend") + elif fault == "extra_path": + (repo / "extra").write_text("unwanted\n") + proof.git(repo, "add", "extra") + proof.git(repo, "commit", "-qm", "extra path", "--amend") + elif fault in {"untracked", "ignored"}: + (repo / "private-cache").write_text("must not survive\n") + if fault == "ignored": + (repo / ".git/info/exclude").write_text("private-cache\n") + elif fault == "source_visible": + source.mkdir() + elif fault == "task_mode": + task.chmod(0o644) + report = inspect(repository, source) + assert report["checks"][failed_check] is False + + +def test_cli_never_prints_raw_provider_errors(capsys): + with patch.object(proof, "prove", side_effect=RuntimeError("secret-provider-response")): + assert proof.main(["--harness-profile", "harness.agent-dev-local@1.0.0"]) == 1 + output = capsys.readouterr().out + assert "secret-provider-response" not in output + assert json.loads(output)["error_type"] == "RuntimeError" diff --git a/workplans/GLAS-WP-0012-first-local-profile-production-proof.md b/workplans/GLAS-WP-0012-first-local-profile-production-proof.md index 3b398ef..2fe9582 100644 --- a/workplans/GLAS-WP-0012-first-local-profile-production-proof.md +++ b/workplans/GLAS-WP-0012-first-local-profile-production-proof.md @@ -152,6 +152,34 @@ portion live, or split that remaining work into a separately registered record before closing the intake. Synchronize work records and log progress. Finish this workplan only when the first profile is proven and residuals remain live. +## Prepare the real-rein acceptance runner + +```task +id: GLAS-WP-0012-T06 +status: done +priority: high +state_hub_task_id: "d0c26709-549f-540a-8250-1fd1b2e4960a" +``` + +Independent preparation for T04 while T02 remains blocked. Add a disposable +fixture that delegates actual dispatch to the catalog-built ReinAharness, +validates the commit and artifact through owner execution before teardown, +checks the host source and cleanup, and emits bounded evidence without model +output or credential values. Refuse blocked or unpinned profiles before +creating a sandbox. Test negative artifact cases and the refusal path. This +runner does not itself satisfy T04 or change readiness. + +Completed 2026-09-05. `scripts/prove-local-profile.py` delegates real dispatch +and observes content/commit acceptance before teardown. It verifies exactly one +new commit, expected paths and committed/worktree content, a clean tree, +source absence, exact child identity, private task mode/removal, and host-source +integrity plus workspace destruction. It emits bounded evidence and has no +readiness override. `tests/test_local_profile_proof.py`: 20 focused tests pass; +full suite: 101 passed; catalog validation passes. A real CLI invocation against +`harness.agent-dev-local@1.0.0` returned `OperationallyBlockedProfileError` +without creating a sandbox. T02–T05 remain waiting on the owner dependencies; +no real model run or readiness promotion is claimed. + ## Acceptance One version-pinned local profile runs its actual rein/model task entirely in