feat(security): add content-silent staged secret scanner

Assistant: codex
Assistant-Model: gpt-6-astra
Assistant-Session: 01a12d13-27f2-7163-b8c0-f682af02772c
This commit is contained in:
tegwick 2026-10-12 15:41:41 +02:00
parent 78d91efc76
commit 77fc4c9b97
5 changed files with 535 additions and 0 deletions

82
docs/secret-detection.md Normal file
View file

@ -0,0 +1,82 @@
# Repository secret detection
Repo Manager provides a control adapter to an external scanner. Credential
custody, incident response and authorization remain with their existing owners.
The adapter is initially source tooling; it does not install or activate hooks.
Work: RMGR-WP-0021, coordinated by Custodian CUST-WP-0080 G07.
## Scan the proposed index
```bash
rmgr secret-scan --path /path/to/repo --binary /reviewed/path/to/gitleaks
```
The supported executable is Gitleaks 8.30.1 Linux x64, SHA256
`88f91962aa2f93ac6ab281d553b9e125f5197bbbce38f9f2437f7299c32e5509`.
Install/admit it separately; this command never downloads or executes an
unverified replacement. The reviewed test archive and checksum came from the
[official release](https://github.com/gitleaks/gitleaks/releases/tag/v8.30.1).
Checksum comparison alone is not a detached signature verification or fleet
installation receipt. Other platforms/versions need an explicitly reviewed pin.
The JSON result contains a fixed status, counts and affected Git blob IDs. It
contains no findings, paths, matched lines or credential values. Raw scanner
stdout/stderr are discarded even on errors; no findings report is written.
Inspect blocked content only in an approved private review environment. Never
paste it into chat, State Hub, issue bodies or CI logs.
| Exit | Result | Required caller behavior |
| --- | --- | --- |
| 0 | `clean` | No detection in the declared changed staged blobs |
| 1 | `blocked` | Reject the proposed commit and privately review the indicated objects |
| 2 | `unverified` | Reject; fix missing tool, unreadable index, limits or other verification failure |
`clean` is not a statement about working-tree files, existing HEAD/history, other
branches, remote refs, logs, artifacts, production credentials or complete absence
of secrets. An empty staged change scans zero objects and says nothing about
history. The scanner uses the upstream default detection rules; heuristic false
negatives and false positives remain possible.
## Input and configuration boundaries
The adapter scans complete changed staged blobs, including an initial commit,
renames and partial staging. It honors Git's explicit alternate index (including
`git commit --only` behavior), ignores Git replacement objects and refuses index
changes observed during scanning. Symlink text is scanned without reading the
external target. Changed gitlinks, conflicts and unreadable entries are refused.
A reviewed configuration extending the scanner's built-in defaults is supplied
from an isolated temporary directory. Repository `.gitleaks.toml`, ignore files,
inline `gitleaks:allow` comments and ambient scanner environment do not suppress
this check. There is no implicit baseline or exception switch. False positives
need a reviewed follow-up mechanism with exact scope and real approval; do not
silently add broad path exclusions.
The current resource bounds are 500 distinct blobs, 8 MiB per blob and 64 MiB in
total; exceeding a bound refuses instead of skipping. Individual Git and scanner
calls have timeouts. The executable is hash-checked and invoked through its open
file descriptor. These measures are not isolation from a hostile same-UID writer,
a malicious Git installation, hook bypass or a process with repository authority.
## Enforcement and response acceptance
Before activating a hook, preserve the existing generated-commit and assistant
provenance checks; validate exact repository coverage, executable availability,
clean commits, synthetic detections and scanner failures. Do not replace the
shared hook or change global `core.hooksPath` merely to run this command.
Local pre-commit can be bypassed. Forge/CI enforcement must independently check
all introduced objects/commits, including merge/base-history cases and unavailable
scanner failures. Required checks and writers able to alter enforcement need
separate admission. An actual finding needs owner classification and a governed
revocation/rotation path; source removal or rejection of one commit does not
revoke credentials that were already exposed. CUST-WP-0073-T05's existing founder
deferral is not overridden by this adapter.
## Tests
Set `RMGR_TEST_GITLEAKS` to the exact reviewed test executable when running
`pytest tests/test_secret_scan.py`. Tests use disposable repositories and keys,
verify commit refusal and preserved index semantics, and never contact a forge
or issuer. Without the test executable, scanner-dependent tests explicitly skip;
those skips are not acceptance evidence.

View file

@ -461,6 +461,10 @@ def main(argv: list[str] | None = None) -> int:
help="Replace the State Hub access table in AGENTS.md from config/state-hub-access.yaml",
)
p_secret_scan = sub.add_parser("secret-scan", help="Content-silent scan of changed staged blobs")
p_secret_scan.add_argument("--path", type=Path, default=Path("."))
p_secret_scan.add_argument("--binary", type=Path, required=True)
p_provenance = sub.add_parser(
"assistant-provenance",
help="Install or report coding-assistant commit provenance",
@ -1052,6 +1056,13 @@ def main(argv: list[str] | None = None) -> int:
print(json.dumps(result.to_dict(), indent=2))
return 0 if result.status == "applied" else 1
if args.command == "secret-scan":
from repo_manager.secret_scan import exit_code, scan_staged
result = scan_staged(args.path, args.binary)
print(json.dumps(result, sort_keys=True))
return exit_code(result)
if args.command == "assistant-provenance":
if not args.provenance_command:
p_provenance.print_help()

View file

@ -0,0 +1,144 @@
"""Content-silent adapter for staged-blob secret detection with pinned Gitleaks.
This is a repository control adapter, not credential custody or an authorization
boundary. It does not install hooks, alter indexes, send content off-host, or
claim history/push/CI coverage. Its caller must treat unverified as refusal.
"""
from __future__ import annotations
import argparse
import hashlib
import json
import os
import re
import stat
import subprocess
import tempfile
from pathlib import Path
VERSION = '8.30.1'
BINARY_SHA256 = '88f91962aa2f93ac6ab281d553b9e125f5197bbbce38f9f2437f7299c32e5509'
MAX_BLOB_BYTES = 8 * 1024 * 1024
MAX_TOTAL_BYTES = 64 * 1024 * 1024
MAX_BLOBS = 500
CONFIG = '[extend]\nuseDefault = true\n'
class Unverified(Exception):
"""A fixed reason code, never a tool exception or repository content."""
def git(repo: Path, *args: str) -> bytes:
env = {**os.environ, 'GIT_NO_REPLACE_OBJECTS': '1'}
result = subprocess.run(['git', '-C', str(repo), *args], stdout=subprocess.PIPE,
stderr=subprocess.DEVNULL, env=env, timeout=20, check=False)
if result.returncode:
raise Unverified('git-read-failed')
return result.stdout
def staged_objects(repo: Path) -> tuple[bytes, list[str]]:
snapshot = git(repo, 'ls-files', '--stage', '-z')
changed = set(git(repo, 'diff', '--cached', '--name-only', '-z', '--no-renames',
'--no-ext-diff', '--no-textconv', '--diff-filter=ACMRTUXB').split(b'\0')) - {b''}
objects: set[str] = set()
seen: set[bytes] = set()
for record in snapshot.split(b'\0'):
if not record:
continue
header, path = record.split(b'\t', 1)
if path not in changed:
continue
mode, oid, stage = header.split()
if stage != b'0' or mode not in (b'100644', b'100755', b'120000'):
raise Unverified('unsupported-index-entry')
if not re.fullmatch(rb'[a-f0-9]{40}|[a-f0-9]{64}', oid):
raise Unverified('invalid-object-id')
objects.add(oid.decode('ascii'))
seen.add(path)
if changed != seen:
raise Unverified('incomplete-index-enumeration')
if len(objects) > MAX_BLOBS:
raise Unverified('staged-blob-limit')
return snapshot, sorted(objects)
def scan_staged(repo: Path, binary: Path) -> dict:
"""Return only fixed statuses/counts and Git blob IDs, never finding contents."""
scanned = 0
descriptor = None
try:
descriptor = os.open(binary, os.O_RDONLY | os.O_NOFOLLOW | os.O_NONBLOCK)
info = os.fstat(descriptor)
if (not stat.S_ISREG(info.st_mode) or info.st_mode & 0o022
or not info.st_mode & 0o111 or info.st_uid not in (0, os.getuid())
or info.st_size > 64 * 1024 * 1024):
raise Unverified('scanner-file-refused')
digest = hashlib.sha256()
while chunk := os.read(descriptor, 1024 * 1024):
digest.update(chunk)
if digest.hexdigest() != BINARY_SHA256:
raise Unverified('scanner-digest-mismatch')
snapshot, objects = staged_objects(repo)
blocked = []
total = 0
# Do not honor repository-local config/ignore files, comments or ambient
# GITLEAKS settings. No scanner report or credential-bearing diagnostic is kept.
with tempfile.TemporaryDirectory(prefix='rmgr-secret-scan-') as temporary:
directory = Path(temporary)
config = directory / 'scanner.toml'; config.write_text(CONFIG)
ignore = directory / '.gitleaksignore'; ignore.write_text('')
env = {'PATH': '/usr/bin:/bin', 'HOME': temporary, 'LANG': 'C', 'TMPDIR': temporary}
for oid in objects:
size = int(git(repo, 'cat-file', '-s', oid))
total += size
if size < 0 or size > MAX_BLOB_BYTES or total > MAX_TOTAL_BYTES:
raise Unverified('staged-byte-limit')
content = git(repo, 'cat-file', 'blob', oid)
if len(content) != size:
raise Unverified('blob-size-mismatch')
command = [str(binary), 'stdin', '--config', str(config),
'--gitleaks-ignore-path', str(ignore), '--ignore-gitleaks-allow',
'--redact=100', '--no-banner', '--no-color', '--log-level', 'fatal',
'--exit-code', '20', '--timeout', '15']
result = subprocess.run(command, executable=f'/proc/self/fd/{descriptor}',
pass_fds=(descriptor,), input=content, cwd=directory, env=env,
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
timeout=20, check=False)
if result.returncode not in (0, 20):
raise Unverified('scanner-failed')
scanned += 1
if result.returncode == 20:
blocked.append(oid)
if git(repo, 'ls-files', '--stage', '-z') != snapshot:
raise Unverified('index-changed-during-scan')
return {'status': 'blocked' if blocked else 'clean', 'scope': 'changed-staged-blobs',
'scanner_version': VERSION, 'scanner_sha256': BINARY_SHA256,
'scanned_blobs': scanned, 'blocked_blob_ids': blocked}
except Unverified as error:
reason = error.args[0]
except (OSError, ValueError, subprocess.SubprocessError):
reason = 'scan-unavailable'
finally:
if descriptor is not None:
os.close(descriptor)
return {'status': 'unverified', 'scope': 'changed-staged-blobs',
'reason': reason, 'scanned_blobs': scanned}
def exit_code(result: dict) -> int:
return {'clean': 0, 'blocked': 1, 'unverified': 2}[result['status']]
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('--path', type=Path, default=Path('.'))
parser.add_argument('--binary', type=Path, required=True)
args = parser.parse_args(argv)
result = scan_staged(args.path, args.binary)
print(json.dumps(result, sort_keys=True))
return exit_code(result)
if __name__ == '__main__':
raise SystemExit(main())

212
tests/test_secret_scan.py Normal file
View file

@ -0,0 +1,212 @@
"""Real scanner integration on disposable Git indexes and never-valid credentials."""
import hashlib
import json
import os
import shlex
import subprocess
import sys
from pathlib import Path
import pytest
from repo_manager import secret_scan
@pytest.fixture
def binary():
configured = os.environ.get('RMGR_TEST_GITLEAKS')
if not configured:
pytest.skip('set RMGR_TEST_GITLEAKS to the reviewed v8.30.1 Linux x64 binary')
path = Path(configured)
assert hashlib.sha256(path.read_bytes()).hexdigest() == secret_scan.BINARY_SHA256
return path
@pytest.fixture
def repo(tmp_path):
repo = tmp_path / 'repo'; repo.mkdir()
git(repo, 'init', '-q')
hooks = tmp_path / 'hooks'; hooks.mkdir()
git(repo, 'config', 'core.hooksPath', str(hooks))
git(repo, 'config', 'user.name', 'Disposable Test')
git(repo, 'config', 'user.email', 'test@example.test')
return repo
def git(repo, *args):
return subprocess.check_output(['git', '-C', str(repo), *args], stderr=subprocess.DEVNULL)
@pytest.fixture
def private_key(tmp_path):
path = tmp_path / 'disposable-key'
subprocess.run(['ssh-keygen', '-q', '-t', 'ed25519', '-N', '', '-f', str(path)],
check=True, capture_output=True)
return path.read_bytes()
def stage(repo, data, name='application.txt'):
(repo / name).write_bytes(data)
git(repo, 'add', '--', name)
def test_clean_staged_blob(binary, repo):
stage(repo, b'print("hello")\n')
result = secret_scan.scan_staged(repo, binary)
assert result['status'] == 'clean'
assert result['scanned_blobs'] == 1
def test_real_secret_commit_is_refused_without_disclosure(binary, repo, private_key):
hook = Path(git(repo, 'rev-parse', '--git-path', 'hooks/pre-commit').decode().strip())
script = str(Path(secret_scan.__file__))
hook.write_text('#!/bin/sh\nexec ' + ' '.join(shlex.quote(s) for s in
[sys.executable, script, '--path', str(repo), '--binary', str(binary)]) + '\n')
hook.chmod(0o700)
stage(repo, private_key)
result = subprocess.run(['git', 'commit', '-qm', 'synthetic detection probe'],
cwd=repo, capture_output=True, check=False)
assert result.returncode != 0
assert bool(private_key.strip() in result.stdout + result.stderr) is False
assert b'"status": "blocked"' in result.stdout + result.stderr
assert subprocess.run(['git', 'rev-parse', '--verify', 'HEAD'], cwd=repo,
capture_output=True, check=False).returncode != 0
def test_staged_secret_refused_even_if_worktree_is_clean(binary, repo, private_key):
stage(repo, private_key)
(repo / 'application.txt').write_text('clean worktree\n')
assert secret_scan.scan_staged(repo, binary)['status'] == 'blocked'
def test_unstaged_secret_not_misrepresented_as_index_scan(binary, repo, private_key):
stage(repo, b'clean staged data\n')
(repo / 'application.txt').write_bytes(private_key)
result = secret_scan.scan_staged(repo, binary)
assert result['status'] == 'clean'
assert result['scope'] == 'changed-staged-blobs'
def test_repo_config_environment_and_inline_allow_cannot_disable_scan(binary, repo, private_key, monkeypatch):
(repo / '.gitleaks.toml').write_text('[allowlist]\nregexes=[".*"]\n')
(repo / '.gitleaksignore').write_text('*\n')
monkeypatch.setenv('GITLEAKS_CONFIG', str(repo / '.gitleaks.toml'))
monkeypatch.setenv('GITLEAKS_CONFIG_TOML', '[allowlist]\nregexes=[".*"]\n')
stage(repo, b'# gitleaks:allow\n' + private_key)
assert secret_scan.scan_staged(repo, binary)['status'] == 'blocked'
def test_symlink_content_is_not_dereferenced(binary, repo, private_key, tmp_path):
target = tmp_path / 'outside-secret'; target.write_bytes(private_key)
(repo / 'link').symlink_to(target)
git(repo, 'add', '--', 'link')
result = secret_scan.scan_staged(repo, binary)
assert result['status'] == 'clean'
assert result['scanned_blobs'] == 1
def test_missing_scanner_is_unverified(repo):
stage(repo, b'clean\n')
result = secret_scan.scan_staged(repo, repo / 'missing')
assert result['status'] == 'unverified'
assert secret_scan.exit_code(result) == 2
def test_wrong_digest_is_refused_before_execution(repo):
marker = repo / 'executed'
fake = repo / 'fake'; fake.write_text('#!/bin/sh\ntouch ' + str(marker) + '\n'); fake.chmod(0o700)
result = secret_scan.scan_staged(repo, fake)
assert result['reason'] == 'scanner-digest-mismatch'
assert not marker.exists()
def test_changed_submodule_is_unverified(binary, repo):
stage(repo, b'clean\n'); git(repo, 'commit', '-qm', 'seed')
commit = git(repo, 'rev-parse', 'HEAD').decode().strip()
git(repo, 'update-index', '--add', '--cacheinfo', '160000,' + commit + ',submodule')
assert secret_scan.scan_staged(repo, binary)['reason'] == 'unsupported-index-entry'
def test_oversize_blob_refuses_instead_of_skipping(binary, repo, monkeypatch):
monkeypatch.setattr(secret_scan, 'MAX_BLOB_BYTES', 8)
stage(repo, b'0123456789')
assert secret_scan.scan_staged(repo, binary)['reason'] == 'staged-byte-limit'
def test_git_replace_cannot_hide_staged_secret(binary, repo, private_key):
stage(repo, private_key)
secret = git(repo, 'rev-parse', ':application.txt').decode().strip()
clean = subprocess.check_output(['git', 'hash-object', '-w', '--stdin'], input=b'clean\n', cwd=repo).decode().strip()
git(repo, 'replace', secret, clean)
assert secret_scan.scan_staged(repo, binary)['status'] == 'blocked'
def test_index_change_during_scan_refuses(binary, repo, monkeypatch):
stage(repo, b'clean\n')
original = secret_scan.git
count = 0
def changing(repo, *args):
nonlocal count
if args == ('ls-files', '--stage', '-z'):
count += 1
if count == 2: stage(repo, b'changed after scan\n')
return original(repo, *args)
monkeypatch.setattr(secret_scan, 'git', changing)
assert secret_scan.scan_staged(repo, binary)['reason'] == 'index-changed-during-scan'
def test_explicit_git_index_file_is_honored(binary, repo, private_key, monkeypatch, tmp_path):
stage(repo, b'clean default index\n')
default = Path(git(repo, 'rev-parse', '--git-path', 'index').decode().strip())
if not default.is_absolute(): default = repo / default
other = tmp_path / 'alternate-index'; other.write_bytes(default.read_bytes())
monkeypatch.setenv('GIT_INDEX_FILE', str(other))
stage(repo, private_key)
assert secret_scan.scan_staged(repo, binary)['status'] == 'blocked'
monkeypatch.delenv('GIT_INDEX_FILE')
assert secret_scan.scan_staged(repo, binary)['status'] == 'clean'
def test_cli_outputs_only_summary(binary, repo, private_key):
stage(repo, private_key)
result = subprocess.run([sys.executable, '-m', 'repo_manager.secret_scan', '--path', str(repo),
'--binary', str(binary)], capture_output=True, check=False)
assert result.returncode == 1
assert json.loads(result.stdout)['status'] == 'blocked'
assert result.stderr == b''
assert bool(private_key.strip() in result.stdout) is False
@pytest.mark.parametrize('failure', ['exit', 'timeout'])
def test_scanner_failure_never_passes_or_emits_diagnostics(binary, repo, monkeypatch, failure):
stage(repo, b'clean\n')
original = secret_scan.subprocess.run
def broken(command, **kwargs):
if len(command) > 1 and command[1] == 'stdin':
if failure == 'timeout':
raise subprocess.TimeoutExpired(command, 20, output=b'synthetic-sensitive-diagnostic')
return subprocess.CompletedProcess(command, 2)
return original(command, **kwargs)
monkeypatch.setattr(secret_scan.subprocess, 'run', broken)
result = secret_scan.scan_staged(repo, binary)
assert result['status'] == 'unverified'
assert 'synthetic-sensitive-diagnostic' not in json.dumps(result)
def test_actual_commit_only_hook_scans_its_temporary_index(binary, repo, private_key):
stage(repo, b'clean\n'); stage(repo, b'also clean\n', 'unrelated.txt')
git(repo, 'commit', '-qm', 'clean baseline')
hook = Path(git(repo, 'rev-parse', '--git-path', 'hooks/pre-commit').decode().strip())
hook.write_text('#!/bin/sh\nexec ' + ' '.join(shlex.quote(s) for s in
[sys.executable, str(Path(secret_scan.__file__)), '--path', str(repo),
'--binary', str(binary)]) + '\n')
hook.chmod(0o700)
stage(repo, b'unrelated staged change\n', 'unrelated.txt')
(repo / 'application.txt').write_bytes(private_key)
before = git(repo, 'rev-parse', 'HEAD')
result = subprocess.run(['git', 'commit', '--only', '-qm', 'synthetic selective commit', '--',
'application.txt'], cwd=repo, capture_output=True, check=False)
assert result.returncode != 0
assert git(repo, 'rev-parse', 'HEAD') == before
assert bool(private_key.strip() in result.stdout + result.stderr) is False
assert b'"status": "blocked"' in result.stdout + result.stderr

View file

@ -0,0 +1,86 @@
---
id: RMGR-WP-0021
type: workplan
title: "Secret-detection adapter and repository enforcement acceptance"
domain: infotech
repo: repo-manager
status: active
owner: codex
topic_slug: repo-manager
created: "2026-10-12"
updated: "2026-10-12"
flavor: implementation
depends_on: []
---
Native repository-control work for CUST-WP-0080 G07 / DSM-DETECT. Existing
RMGR-WP-0009 provenance remains separate and nonblocking by its contract.
Detection is an additional security check; this plan does not redefine provenance,
credential custody, PDP authority, Forge ownership or incident authorization.
## Provide a bounded content-silent staged-object scanner adapter
```task
id: RMGR-WP-0021-T01
status: done
priority: high
```
2026-10-12: all 21 reviewed estate checkouts resolve their pre-commit hook to the
same generated-only guard. A disposable valid private key was committed through
that inherited hook, proving no local prevention for that fixture. No real
credential was used. The new `rmgr secret-scan` adapter pins Gitleaks 8.30.1 Linux
x64, reads complete changed index blobs, isolates configuration, suppresses raw
output, verifies index stability and refuses unknown binaries, oversized inputs,
changed gitlinks and other unverified cases. Real scanner tests prove refusal of
a disposable private-key commit, partial/alternate-index correctness, local
configuration/inline suppression refusal and Git replacement-object resistance.
Evidence: Custodian `docs/evidence/2026-10-12-netkingdom-security/secret-detection.json`.
This is source/tool behavior, not workstation or forge activation.
## Admit the tool and activate local prevention for the reviewed repositories
```task
id: RMGR-WP-0021-T02
status: todo
priority: high
```
Review durable executable installation/version custody and exact repository
coverage. Preserve existing shared generated-only and provenance hooks and the
Custodian's separate local hook. Do not replace unknown hooks, alter global
routing or silently affect unreviewed repositories. Test actual configured hooks
with clean/disposable-secret/scanner-failure cases without changing production
source or committing a credential. Record observed coverage and explicit
uncovered checkouts. Installer rollback restores exact prior content/routing.
## Establish independent forge and CI detection coverage
```task
id: RMGR-WP-0021-T03
status: todo
priority: high
```
Under CUST-WP-0080 coordination with railiance-forge, inspect existing server-side
and required-check controls; define the production repository denominator and
owner-native enforcement record before activating changes. Cover introduced
history/merge cases and bypasses, missing scanner, malformed input and scanner
failure. Record actual rejected/accepted push or required-check receipts. Local
hook success is insufficient and this staged adapter alone does not scan history.
## Verify private finding-to-response handling
```task
id: RMGR-WP-0021-T04
status: todo
priority: high
```
Exercise a synthetic finding through metadata-only owner routing and approved
private review. Define exact false-positive exception governance; require actual
approval, scope, expiry/review and regression coverage. For genuine exposed
credentials, retain incident owner and revoke/replace/adopt/validate gates. Do not
execute founder-deferred rotations or treat suppressed output as remediation.
Close only after native coverage and response evidence are adequate for the
applicable DSM-DETECT requirements; retain residuals as live work records.