diff --git a/README.md b/README.md index 922f840..52e3f1e 100644 --- a/README.md +++ b/README.md @@ -86,3 +86,5 @@ speaks in lanes and guarded plans. not write. - Temporary bootstrap OpenBao credentials must live outside repos, use mode 0600, be revocable, and be removed after narrower auth is working. + +Packaged CLI deployment uses the reviewed [wheel + verifier bundle](docs/cli-release-bundle.md). A Python wheel alone omits the mandatory signature helper. diff --git a/docs/cli-release-bundle.md b/docs/cli-release-bundle.md new file mode 100644 index 0000000..8634eda --- /dev/null +++ b/docs/cli-release-bundle.md @@ -0,0 +1,82 @@ +# CLI release bundle with authenticated decision verifier + +The supported packaged CLI path is an explicit wheel-plus-verifier bundle. The +existing editable `uv` workflow remains for development. Installing the Python +wheel alone omits the Go helper and is incomplete for production authorization; +it continues to fail closed. This is a CLI package, not a service or the retired +`rapp-secrets-engine` deployment. + +The bundle contains the Secrets wheel, the existing PyYAML runtime wheel pinned +by `uv.lock`, the verifier executable, producer license attribution, a narrow CLI +entry point and an offline installer. It never contains the test signer, signing +seed, caller credential, runtime trust/key config or backend entitlement. Optional +`clock` extras are outside this base bundle; configuring clock trust without the +separately reviewed extra continues to refuse, rather than fall back. + +## Build + +Use Python >=3.11, uv and the Go toolchain required by `verifier/go.mod` on the +intended Linux architecture. Supply a local compatible PyYAML wheel whose exact +filename and SHA-256 appear in `uv.lock`. The builder never discovers/downloads a +runtime dependency. The existing Hatchling build backend is now version-pinned. + +```bash +python3 scripts/build_release_bundle.py \ + --output /tmp/secrets-cli-reviewed-build \ + --dependency-wheel /path/to/locked-compatible-pyyaml.whl +``` + +Builds are offline by default. In a reviewed build environment, the explicit +`--allow-build-downloads` flag permits uv to fetch the existing pinned build +backend and its build dependencies. It does not change the runtime dependency +contract. Set writable uv/Go caches as required by that build environment. + +The builder first verifies the exact producer API/signature source and native +fixture hashes, builds only the verifier root package with CGO disabled, explicit +host architecture, trimmed paths, disabled VCS stamping and an empty build ID, +then builds the wheel with commit-derived `SOURCE_DATE_EPOCH`. Two builds using +the same source snapshot/toolchain/dependency wheel produce matching artifacts +and manifest. The manifest records exact source-file and artifact SHA-256s, base +commit, producer snapshot, platform and tool versions. It contains no observed +build timestamp; retain observed execution time in a separate receipt. + +## Review and offline install + +**Checksums prove artifact integrity, not independent package admission.** The +owner must review/admit the source, toolchain, complete bundle and manifest hash, +and control the bundle/install prefix. A manifest obtained from an untrusted +bundle does not establish that admission. Key/trust admission remains a distinct +WP14 gate. None of these steps authorizes protected OpenBao work. + +Execute the reviewed installer with the independently retained manifest digest: + +```bash +python3 /path/to/reviewed-bundle/install.py \ + --prefix /path/to/new-owner-controlled-prefix \ + --manifest-sha256 REVIEWED_MANIFEST_SHA256 +/path/to/new-owner-controlled-prefix/bin/secrets-engine --help +``` + +The parent directory must already be owner controlled. The installer validates +platform, manifest and the entire artifact set before creating a private new +prefix, creates its venv through the existing uv convention, and installs entirely +offline from bundled wheels. It never replaces an existing prefix or installs +into standard user/system tools. A failed partial prefix is retained and marked; +retry with a new prefix after diagnosis. Removing a failed/disposable prefix is +an explicit owner operation, not an automatic cleanup of an existing path. + +Use `PREFIX/bin/secrets-engine` as the admitted CLI entry point. It checks the +bundled helper's SHA-256 before CLI import or credential access and fixes +`SECRETS_ENGINE_PDP_VERIFIER` to that helper; ambient executable overrides do not +select another verifier. Missing or changed helpers refuse. There is no runtime +compilation or unsigned fallback. The internal venv scripts/programmatic API are +not a second admitted entry point and do not create package/trust admission. +Catalog/policy and independently reviewed responder trust configuration still +belong to the deploying owner. Neither is silently copied from a developer repo. + +For source verification, supply `SECRETS_ENGINE_TEST_DEPENDENCY_WHEEL` and run +`pytest tests/test_release_bundle.py`. The disposable proof compares two builds, +installs offline, invokes the installed wheel's signed HTTP boundary with public +synthetic fixtures, rejects unsigned decisions, checks helper/bundle tampering, +source drift and existing-prefix refusal. No native credentials or protected +backend are involved. diff --git a/pyproject.toml b/pyproject.toml index fd25de6..aa74fda 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -20,7 +20,7 @@ dev = [ secrets-engine = "secrets_engine.cli:main" [build-system] -requires = ["hatchling"] +requires = ["hatchling==1.27.0"] build-backend = "hatchling.build" [tool.hatch.build.targets.wheel] diff --git a/scripts/build_release_bundle.py b/scripts/build_release_bundle.py new file mode 100644 index 0000000..7f8f0df --- /dev/null +++ b/scripts/build_release_bundle.py @@ -0,0 +1,86 @@ +#!/usr/bin/env python3 +"""Build wheel + pinned producer verifier; install is a separate reviewed step.""" +import argparse +import hashlib +import json +import os +from pathlib import Path +import platform +import shutil +import subprocess +import sys +import tomllib + + +def digest(path): + return hashlib.sha256(path.read_bytes()).hexdigest() + + +def run(command, **kwargs): + return subprocess.check_output(command,text=True,**kwargs).strip() + + +def build(root, output, dependency_wheel, uv, allow_build_downloads=False): + if platform.system() != 'Linux': + raise ValueError('this bundle supports Linux CLI only') + # No discovered keys, source rewriting, or runtime compilation. + subprocess.run([sys.executable,str(root/'scripts/check_pdp_producer_source.py')],check=True,stdout=sys.stderr) + locked=next(p for p in tomllib.loads((root/'uv.lock').read_text())['package'] if p['name']=='pyyaml') + admitted={Path(item['url']).name:item['hash'].removeprefix('sha256:') for item in locked['wheels']} + if dependency_wheel.name not in admitted or digest(dependency_wheel)!=admitted[dependency_wheel.name]: + raise ValueError('supplied PyYAML wheel does not match uv.lock') + output.mkdir(mode=0o700,parents=False,exist_ok=False) + epoch=run(['git','log','-1','--format=%ct'],cwd=root) + commit=run(['git','rev-parse','HEAD'],cwd=root) + architecture={'x86_64':'amd64','aarch64':'arm64'}.get(platform.machine()) + if architecture is None: + raise ValueError('unsupported Linux architecture') + env={**os.environ,'CGO_ENABLED':'0','GOOS':'linux','GOARCH':architecture, + 'GOFLAGS':'','GOWORK':'off','SOURCE_DATE_EPOCH':epoch} + (output/'bin').mkdir();(output/'wheels').mkdir() + subprocess.run(['go','build','-trimpath','-buildvcs=false','-ldflags=-buildid=', + '-o',str(output/'bin/secrets-engine-pdp-verify'),'.'],cwd=root/'verifier',env=env,check=True) + command=[uv,'--no-config','build','--wheel','--out-dir',str(output/'wheels'),str(root)] + if not allow_build_downloads:command.append('--offline') + subprocess.run(command,env=env,check=True) + wheel,=list((output/'wheels').glob('secrets_engine-*.whl')) + shutil.copyfile(dependency_wheel,output/'wheels'/dependency_wheel.name) + helper_hash=digest(output/'bin/secrets-engine-pdp-verify') + launcher=(root/'scripts/release_cli.py').read_text().replace('@HELPER_SHA256@',helper_hash) + (output/'release_cli.py').write_text(launcher) + shutil.copyfile(root/'scripts/install_release_bundle.py',output/'install.py') + shutil.copyfile(root/'verifier/LICENSE.access-engine',output/'LICENSE.access-engine') + sources=[root/'pyproject.toml',root/'uv.lock',root/'verifier/producer-source.json'] + sources+=list((root/'src/secrets_engine').glob('*.py')) + sources+=list((root/'verifier/pkg/api').glob('*.go'))+list((root/'verifier/internal/sign').glob('*.go')) + sources+=[root/'verifier/main.go',root/'scripts/check_pdp_producer_source.py', + root/'scripts/build_release_bundle.py',root/'scripts/install_release_bundle.py',root/'scripts/release_cli.py'] + source_hashes={str(path.relative_to(root)):digest(path) for path in sorted(sources)} + manifest={"schema":"secrets-engine.cli-bundle.v1","source_commit":commit,"source_files":source_hashes, + "source_date_epoch":int(epoch),"system":platform.system(),"machine":platform.machine(), + "python":platform.python_version(),"go":run(['go','version']),"uv":run([uv,'--version']), + "build_backend":"hatchling==1.27.0","wheel":str(wheel.relative_to(output)), + "producer":json.loads((root/'verifier/producer-source.json').read_text()), + "artifacts":{str(path.relative_to(output)):digest(path) for path in sorted(output.rglob('*')) if path.is_file()}, + "admission":"checksums are integrity evidence; owner must admit the bundle/prefix independently"} + (output/'manifest.json').write_text(json.dumps(manifest,sort_keys=True,indent=2)+'\n') + return {'bundle':str(output),'manifest_sha256':digest(output/'manifest.json'),'helper_sha256':helper_hash} + + +def main(): + parser=argparse.ArgumentParser(description=__doc__) + parser.add_argument('--output',type=Path,required=True) + parser.add_argument('--dependency-wheel',type=Path,required=True) + parser.add_argument('--uv',default='uv') + parser.add_argument('--allow-build-downloads',action='store_true',help='permit uv to fetch existing build-system dependencies') + args=parser.parse_args() + try: + print(json.dumps(build(Path(__file__).resolve().parents[1],args.output.resolve(),args.dependency_wheel.resolve(), + args.uv,args.allow_build_downloads),sort_keys=True)) + except (OSError,ValueError,KeyError,subprocess.SubprocessError) as exc: + print('bundle build refused: '+str(exc),file=sys.stderr);return 1 + return 0 + + +if __name__=='__main__': + raise SystemExit(main()) diff --git a/scripts/install_release_bundle.py b/scripts/install_release_bundle.py new file mode 100644 index 0000000..9d718fc --- /dev/null +++ b/scripts/install_release_bundle.py @@ -0,0 +1,82 @@ +#!/usr/bin/env python3 +"""Install one reviewed CLI bundle offline into a new owner-controlled prefix.""" +import argparse +import hashlib +import json +from pathlib import Path +import platform +import shutil +import subprocess +import sys + + +def digest(path): + return hashlib.sha256(path.read_bytes()).hexdigest() + + +def install(bundle, prefix, expected_manifest, python, uv): + manifest_path = bundle / "manifest.json" + if digest(manifest_path) != expected_manifest: + raise ValueError("bundle manifest differs from reviewed digest") + manifest = json.loads(manifest_path.read_text()) + if manifest["schema"] != "secrets-engine.cli-bundle.v1": + raise ValueError("unsupported bundle") + if manifest['system'] != platform.system() or manifest['machine'] != platform.machine(): + raise ValueError("bundle platform differs from installer host") + expected_files = set(manifest['artifacts']) | {'manifest.json'} + actual_files = {str(p.relative_to(bundle)) for p in bundle.rglob('*') if p.is_file()} + if actual_files != expected_files: + raise ValueError("bundle file set differs") + for name, expected in manifest['artifacts'].items(): + relative = Path(name) + if relative.is_absolute() or '..' in relative.parts: + raise ValueError("invalid artifact path") + artifact = bundle / relative + if any(parent.is_symlink() for parent in (artifact,*artifact.parents)) or digest(artifact) != expected: + raise ValueError("bundle artifact missing or changed") + # Reserve once; never replace an installed prefix or follow an existing link. + prefix.mkdir(mode=0o700, parents=False, exist_ok=False) + try: + subprocess.run([uv,'--no-config','venv','--python',python,'--no-python-downloads',str(prefix/'venv')],check=True) + subprocess.run([uv,'--no-config','pip','install','--python',str(prefix/'venv/bin/python'), + '--no-index','--find-links',str(bundle/'wheels'),str(bundle/manifest['wheel'])],check=True) + (prefix/'libexec').mkdir(mode=0o700) + helper=prefix/'libexec/secrets-engine-pdp-verify' + shutil.copyfile(bundle/'bin/secrets-engine-pdp-verify',helper);helper.chmod(0o700) + (prefix/'bin').mkdir(mode=0o700) + launcher=prefix/'bin/secrets-engine' + # An absolute venv interpreter fixes the installed dependency environment. + launcher.write_text('#!'+str(prefix/'venv/bin/python')+'\n'+(bundle/'release_cli.py').read_text()) + launcher.chmod(0o700) + receipt={"schema":"secrets-engine.cli-install.v1","manifest_sha256":expected_manifest, + "helper_sha256":digest(helper),"launcher_sha256":digest(launcher),"prefix":str(prefix), + "admission":"artifact integrity only; trust and keys not admitted"} + (prefix/'install-receipt.json').write_text(json.dumps(receipt,sort_keys=True,indent=2)+'\n') + return receipt + except BaseException: + # Preserve a failed prefix as evidence; never retry into a partial install. + (prefix/'INSTALL-FAILED').write_text('Incomplete installation; use a new prefix.\n') + raise + + +def main(): + parser=argparse.ArgumentParser(description=__doc__) + parser.add_argument('--bundle',type=Path,default=Path(__file__).resolve().parent) + parser.add_argument('--prefix',type=Path,required=True) + parser.add_argument('--manifest-sha256',required=True) + parser.add_argument('--python',default=sys.executable) + parser.add_argument('--uv',default='uv') + args=parser.parse_args() + try: + if not args.prefix.is_absolute() or args.prefix.parent.is_symlink(): + raise ValueError('prefix must be absolute with an admitted parent') + receipt=install(args.bundle.resolve(),args.prefix,args.manifest_sha256,args.python,args.uv) + print(json.dumps(receipt,sort_keys=True)) + except (OSError,ValueError,KeyError,subprocess.SubprocessError) as exc: + print('bundle install refused: '+str(exc),file=sys.stderr) + return 1 + return 0 + + +if __name__=='__main__': + raise SystemExit(main()) diff --git a/scripts/release_cli.py b/scripts/release_cli.py new file mode 100644 index 0000000..c7a2d08 --- /dev/null +++ b/scripts/release_cli.py @@ -0,0 +1,29 @@ +"""Narrow installed CLI entry point; rendered by build_release_bundle.py.""" +import hashlib +import os +from pathlib import Path +import sys + +EXPECTED_HELPER_SHA256 = "@HELPER_SHA256@" + + +def main(): + prefix = Path(__file__).resolve().parent.parent + helper = prefix / "libexec" / "secrets-engine-pdp-verify" + try: + if helper.is_symlink() or not helper.is_file(): + raise ValueError("missing helper") + if hashlib.sha256(helper.read_bytes()).hexdigest() != EXPECTED_HELPER_SHA256: + raise ValueError("changed helper") + except (OSError, ValueError): + print("secrets-engine: installed verifier missing or changed", file=sys.stderr) + return 1 + # The bundle always uses its reviewed helper; ambient overrides do not select + # another executable. The owner admits runtime trust separately. + os.environ["SECRETS_ENGINE_PDP_VERIFIER"] = str(helper) + from secrets_engine.cli import main as cli_main + return cli_main() + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tests/test_release_bundle.py b/tests/test_release_bundle.py new file mode 100644 index 0000000..c481cd5 --- /dev/null +++ b/tests/test_release_bundle.py @@ -0,0 +1,137 @@ +"""Actual offline installation proof; only supplied locked public dependency wheel. + +Set SECRETS_ENGINE_TEST_DEPENDENCY_WHEEL to a local PyYAML wheel from uv.lock. +No production install, trust admission, backend or credential is used. +""" +import hashlib +import json +import os +from pathlib import Path +import shutil +import subprocess +import sys + +import pytest + +from tests.test_decision_signature import FIXTURE, PUBLIC +from tests.test_decision_transport import server + +ROOT = Path(__file__).resolve().parents[1] + + +def run(command, **kwargs): + return subprocess.run(command, text=True, capture_output=True, timeout=120, **kwargs) + + +@pytest.fixture(scope='module') +def bundles(tmp_path_factory): + dependency=os.environ.get('SECRETS_ENGINE_TEST_DEPENDENCY_WHEEL') + if not dependency: + pytest.skip('provide the locked public PyYAML wheel for actual bundle build/install tests') + directory=tmp_path_factory.mktemp('cli-bundles') + result=[] + for name in ['one','two']: + bundle=directory/name + built=run([sys.executable,str(ROOT/'scripts/build_release_bundle.py'),'--output',str(bundle), + '--dependency-wheel',dependency]) + assert built.returncode==0,built.stderr + result.append(bundle) + return result + + +def manifest_hash(bundle): + return hashlib.sha256((bundle/'manifest.json').read_bytes()).hexdigest() + + +def install(bundle,prefix): + return run([sys.executable,str(bundle/'install.py'),'--bundle',str(bundle),'--prefix',str(prefix), + '--manifest-sha256',manifest_hash(bundle)]) + + +@pytest.fixture +def installed(bundles,tmp_path): + prefix=tmp_path/'installed' + result=install(bundles[0],prefix) + assert result.returncode==0,result.stderr + return prefix + + +def test_two_builds_match_and_contain_no_signer(bundles): + one,two=bundles + names={str(path.relative_to(one)) for path in one.rglob('*') if path.is_file()} + assert names=={str(path.relative_to(two)) for path in two.rglob('*') if path.is_file()} + for name in names: + assert (one/name).read_bytes()==(two/name).read_bytes(),name + assert not any('signer' in name or 'testsupport' in name for name in names) + assert (one/'bin/secrets-engine-pdp-verify').is_file() + assert len(list((one/'wheels').glob('secrets_engine-*.whl')))==1 + + +def test_clean_install_uses_installed_wheel_and_signed_boundary(installed,tmp_path): + env={**os.environ,'SECRETS_ENGINE_PDP_VERIFIER':'/must-not-use-ambient-override'} + help_result=run([str(installed/'bin/secrets-engine'),'--help'],env=env,cwd='/tmp') + assert help_result.returncode==0,help_result.stderr + envelope=json.loads(FIXTURE.read_bytes()) + trust={name:envelope['provenance'][name] for name in ('evaluator','mode','policy_package','policy_version','policy_package_digest')} + trust.update(schema='secrets-engine.pdp-trust.v1',keys={'testdata-ed25519':PUBLIC}) + trust_file=tmp_path/'synthetic-trust.json';trust_file.write_text(json.dumps(trust)) + token=tmp_path/'synthetic.token';token.write_text('synthetic-pdp-credential');token.chmod(0o600) + script='''import pathlib,sys +import secrets_engine.decision_check as installed +from secrets_engine.errors import DecisionError +assert str(pathlib.Path(installed.__file__).resolve()).startswith(sys.argv[1]+'/venv/') +try: + result=installed.check_decision(base_url=sys.argv[2],token_file=pathlib.Path(sys.argv[3]),request={},trust_file=pathlib.Path(sys.argv[4]),verifier=pathlib.Path(sys.argv[1]+'/libexec/secrets-engine-pdp-verify')) +except DecisionError: + if sys.argv[5]=='refuse':sys.exit(0) + raise +assert sys.argv[5]=='accept' and result['effect']=='allow' +''' + clean_env={key:value for key,value in os.environ.items() if key not in ('PYTHONPATH','PYTHONHOME')} + for body,expect in [(FIXTURE.read_bytes(),'accept'),(b'{"effect":"allow"}','refuse')]: + with server(body=body) as (origin,seen): + result=run([str(installed/'venv/bin/python'),'-c',script,str(installed),origin,str(token),str(trust_file),expect],env=clean_env,cwd='/tmp') + assert result.returncode==0,result.stderr + assert seen==[True] + + +@pytest.mark.parametrize('change',['missing','tampered']) +def test_installed_cli_refuses_missing_or_tampered_helper(installed,change): + helper=installed/'libexec/secrets-engine-pdp-verify' + if change=='missing':helper.rename(helper.with_suffix('.held')) + else:helper.write_bytes(helper.read_bytes()+b'tampered') + result=run([str(installed/'bin/secrets-engine'),'--help'],cwd='/tmp') + assert result.returncode==1 + assert 'verifier missing or changed' in result.stderr + assert not result.stdout + + +def test_installer_refuses_changed_artifact_and_existing_prefix(bundles,tmp_path): + altered=tmp_path/'altered';shutil.copytree(bundles[0],altered) + helper=altered/'bin/secrets-engine-pdp-verify';helper.write_bytes(helper.read_bytes()+b'tampered') + prefix=tmp_path/'must-not-create' + result=install(altered,prefix) + assert result.returncode==1 and not prefix.exists() + existing=tmp_path/'existing';existing.mkdir();marker=existing/'keep';marker.write_text('unchanged') + result=install(bundles[0],existing) + assert result.returncode==1 and marker.read_text()=='unchanged' + assert sorted(path.name for path in existing.iterdir())==['keep'] + + +def test_installer_requires_exact_reviewed_manifest(bundles,tmp_path): + prefix=tmp_path/'must-not-create' + result=run([sys.executable,str(bundles[0]/'install.py'),'--prefix',str(prefix),'--manifest-sha256','0'*64]) + assert result.returncode==1 and not prefix.exists() + + +def test_changed_producer_source_stops_before_build(tmp_path): + root=tmp_path/'source';(root/'scripts').mkdir(parents=True) + for name in ['build_release_bundle.py','check_pdp_producer_source.py']: + shutil.copyfile(ROOT/'scripts'/name,root/'scripts'/name) + shutil.copytree(ROOT/'verifier',root/'verifier') + shutil.copytree(ROOT/'tests/fixtures/pdp-signature',root/'tests/fixtures/pdp-signature') + source=root/'verifier/internal/sign/sign.go';source.write_bytes(source.read_bytes()+b'// drift\n') + output=tmp_path/'must-not-build' + result=run([sys.executable,str(root/'scripts/build_release_bundle.py'),'--output',str(output), + '--dependency-wheel',str(tmp_path/'not-reached.whl')]) + assert result.returncode==1 and not output.exists() diff --git a/verifier/README.md b/verifier/README.md index 073cace..a63b2a0 100644 --- a/verifier/README.md +++ b/verifier/README.md @@ -28,3 +28,8 @@ The Check API defaults to mandatory authentication. Its deliberately named `allow_unsigned_test_fixture` argument is used only by synthetic transport tests; production `authorize_action` never passes it. The helper runs with an empty environment, so ambient backend and identity credentials cannot reach it. + +For the complete reproducible CLI build and offline fresh-prefix installation, +use [the release bundle contract](../docs/cli-release-bundle.md). This replaces +manual helper copying for packaged deployments; package and key admission remain +separate owner decisions. diff --git a/workplans/SECRETS-WP-0014-pdp-transport-and-responder-authentication.md b/workplans/SECRETS-WP-0014-pdp-transport-and-responder-authentication.md index a3e55a9..886a0f6 100644 --- a/workplans/SECRETS-WP-0014-pdp-transport-and-responder-authentication.md +++ b/workplans/SECRETS-WP-0014-pdp-transport-and-responder-authentication.md @@ -77,6 +77,15 @@ key admission remains part of its scope; T03 stays wait. WP13 execution and identity-only CCR38 are unchanged. Custodian source/verification handoff: `docs/changes/netkingdom-target-2026-10-16/decision-auth/` in the Custodian repo. +2026-10-16 packaging preparation: `scripts/build_release_bundle.py` and its +separate offline installer deliver the existing wheel plus the reviewed producer +verifier together into a new owner-controlled prefix. Source/hash drift checks, +artifact/toolchain receipts and a narrow fixed-helper launcher remove the silent +wheel-only deployment gap. Disposable build/install tests prove deterministic +artifacts, signed HTTP consumption and missing/tampered artifact refusal; they do +not admit packages, keys or a production prefix. T02 remains progress and T03 +wait; no native installation, trust change or protected entitlement is claimed. + ## Verify the admitted native consumer and rotation path ```task