From 6d5c9e2abf794cffeea5d3debdc23f5acf556af2 Mon Sep 17 00:00:00 2001 From: tegwick Date: Mon, 12 Oct 2026 18:39:27 +0200 Subject: [PATCH] security: add bounded receive-path secret detection adapter Assistant: codex Assistant-Model: gpt-6-astra Assistant-Session: 01a12d13-27f2-7163-b8c0-f682af02772c --- Makefile | 10 + docs/receive-secret-detection.md | 107 +++++ tools/receive-secret-scan/go.mod | 3 + tools/receive-secret-scan/main.go | 404 +++++++++++++++++ tools/receive-secret-scan/main_test.go | 425 ++++++++++++++++++ ...GE-WP-0006-independent-secret-detection.md | 86 ++++ 6 files changed, 1035 insertions(+) create mode 100644 docs/receive-secret-detection.md create mode 100644 tools/receive-secret-scan/go.mod create mode 100644 tools/receive-secret-scan/main.go create mode 100644 tools/receive-secret-scan/main_test.go create mode 100644 workplans/FORGE-WP-0006-independent-secret-detection.md diff --git a/Makefile b/Makefile index 004b566..bfbd6cc 100644 --- a/Makefile +++ b/Makefile @@ -121,3 +121,13 @@ help: ## Show this help /^##@/ { printf "\n\033[1m%s\033[0m\n", substr($$0, 5) }' $(MAKEFILE_LIST) .PHONY: check-tools check-sops registry-docs evidence-docs runner-docs runner-status forgejo-runner-status forge-observe test-unit check-runner-tools gitea-deploy gitea-ingress-deploy gitea-status help + +##@ Secret detection + +GO ?= go + +test-secret-detection: ## Verify receive-path refusal with an explicitly admitted test scanner + @test -n "$${FORGE_TEST_GITLEAKS:-}" || { echo "FORGE_TEST_GITLEAKS must name the admitted test binary" >&2; exit 2; } + cd tools/receive-secret-scan && $(GO) test -count=1 ./... + +.PHONY: test-secret-detection diff --git a/docs/receive-secret-detection.md b/docs/receive-secret-detection.md new file mode 100644 index 0000000..2c06a3f --- /dev/null +++ b/docs/receive-secret-detection.md @@ -0,0 +1,107 @@ +# Forge receive-path secret detection + +FORGE-WP-0006 implements the forge-owned part of RMGR-WP-0021-T03 and +Custodian CUST-WP-0080 G07. The local Repo Manager scanner remains separate. +The source adapter is not evidence of installed server enforcement. + +## Boundary and ownership + +The forge-owned, standalone Linux adapter composes with Forgejo's generated +pre-receive dispatcher. It uses pinned Gitleaks 8.30.1 Linux amd64 +(SHA256 `88f91962aa2f93ac6ab281d553b9e125f5197bbbce38f9f2437f7299c32e5509`). +It requires `/usr/bin/git` and `/proc` but no Python runtime, issuer credential, +network access, source checkout, or repository-controlled workflow. The current +Forgejo 11.0.3 image has no Python interpreter; this keeps the runtime change to +two reviewed executables and a hook fragment, subject to native admission. + +The forge owns tool admission, hook installation and recovery. Repo Manager +coordinates repository-control coverage; enablement owns reusable CI templates; +source repositories own their workflows. Credential classification and +revocation remain with existing incident/custody owners. + +Do not overwrite generated `hooks/pre-receive` or `pre-receive.d/gitea`, or enable +arbitrary user Git-hook editing. A reviewed additional executable in the existing +`pre-receive.d/` dispatcher is the proposed integration. Server writers, frontend +merge/edit/import paths, SSH and HTTP pushes, hook resynchronization, pod restart +and new repository creation all need separate native acceptance. No admission +or production coverage is inferred from disposable bare-repository tests. + +## What is checked + +Input is the exact `old-object new-object ref-name` batch from receive-pack. +Malformed, duplicate, oversized, inconsistent or unsupported updates refuse. +All updated ref names are scanned without echoing them. Ref exclusions are frozen +from the pre-push snapshot; the traversal scans complete introduced reachable +objects across every new tip and every parent, including annotated tags and +direct object tags. A secret added and removed before the tip is still examined. + +Blob data, raw commit/tag contents and tree names are scanned. Symlink contents +are read as Git blobs; their filesystem targets are never followed. Changed trees +containing external gitlinks refuse because the referenced repository is not +covered. Git replacement objects and grafts cannot hide history. Partial/shallow +repositories, missing objects and unverified traversal refuse. Receive-pack's +quarantine object locations are preserved only for Git reads; the adapter never +updates refs or copies objects into the primary store. + +The scanner executable is opened without following a final symlink, checked for +ownership/mode/regular-file shape and exact SHA256, and executed through its open +file descriptor. Scanner config and ignore files are private temporary files; +repository configuration, inline allow comments and ambient scanner settings do +not disable detection. Child diagnostics/findings are discarded. Output contains +only a fixed status/reason, counts, version and, for an object detection, its ID. +Privately review the identified object; never paste its contents into an issue, +State Hub, CI logs or chat. Standard Git output may independently display ref +names; the adapter does not make Git's transport diagnostics a secret-safe log. + +Exit 0 is `clean` in this declared scope; exit 1 is `blocked`; exit 2 is +`unverified`. Both nonzero statuses must refuse the complete push. There is no +implicit baseline/ignore or override switch. A false-positive exception needs a +separate reviewed mechanism with exact scope, approval and expiry. + +## Bounds and remaining coverage + +Current bounds: 256 updates, 10,000 existing refs, 2,000 introduced objects, +8 MiB per object, 64 MiB total, 512 KiB update input and 90 seconds for the check. +Scanner calls also use the scanner's 15-second timeout. Object sizes and total +bytes are checked before scanning payloads. Limits reject rather +than skip. Large imports need an admitted path; do not disable the gate to pass +an unreviewed import. Git/scanner versions and limits require native load testing. + +The adapter does not rescan objects already reachable before the push, scan +unreachable extra pack objects, LFS payloads, other repositories, runtime logs, +artifacts or credentials outside Git. It is not an authorization boundary against +a forge administrator or hostile same-UID process. Default scanner rules remain +heuristic. Historical discovery and finding-to-response are independent work; +removing a source value or rejecting a push does not revoke an exposed credential. +CUST-WP-0073-T05's founder-deferred rotation is unchanged. + +The 165 observed on-disk repository directories are a discovery count, not an +admitted production-software denominator. The 22 reviewed repositories provide a +named rollout starting scope; the owner must reconcile all production repositories, +wikis, archives, mirrors and new-repository lifecycle before claiming DSM coverage. + +## Build and disposable acceptance + +Use the admitted Go toolchain (the current candidate uses official Go 1.26.9, +verified against the SHA256 from go.dev release metadata). No third-party Go +modules are required. Keep binaries and logs outside the repository: + +```sh +cd tools/receive-secret-scan +FORGE_TEST_GITLEAKS=/reviewed/gitleaks go test -count=1 -v ./... +CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -trimpath -buildvcs=false -o /tmp/receive-secret-scan . +``` + +The root `make test-secret-detection` target requires `FORGE_TEST_GITLEAKS` +and runs this module; set `GO` to the admitted compiler path where needed. + +Tests use disposable Git servers, actual receive-pack quarantine and generated +keys. Scanner-dependent tests explicitly skip without `FORGE_TEST_GITLEAKS`; +skips are not acceptance. Native activation additionally needs exact original +hook hashes, tool provenance, dry-run/diff, abort owner, rollback and value-free +receipts under the accepted build-mode change rule. + +References: [Git receive-pack and quarantine](https://git-scm.com/docs/git-receive-pack), +[Git object traversal](https://git-scm.com/docs/git-rev-list), +[Forgejo 11.0.3 generated hooks](https://codeberg.org/forgejo/forgejo/src/tag/v11.0.3/modules/repository/hooks.go), +[Gitleaks release](https://github.com/gitleaks/gitleaks/releases/tag/v8.30.1). diff --git a/tools/receive-secret-scan/go.mod b/tools/receive-secret-scan/go.mod new file mode 100644 index 0000000..7fd58b7 --- /dev/null +++ b/tools/receive-secret-scan/go.mod @@ -0,0 +1,3 @@ +module railiance-forge/receive-secret-scan + +go 1.22 diff --git a/tools/receive-secret-scan/main.go b/tools/receive-secret-scan/main.go new file mode 100644 index 0000000..61b16cc --- /dev/null +++ b/tools/receive-secret-scan/main.go @@ -0,0 +1,404 @@ +// receive-secret-scan is a content-silent, Linux pre-receive control adapter. +// It never changes refs, runs repository programs, or retrieves credentials. +package main + +import ( + "bytes" + "context" + "crypto/sha256" + "encoding/hex" + "encoding/json" + "errors" + "flag" + "fmt" + "io" + "os" + "os/exec" + "path/filepath" + "regexp" + "sort" + "strconv" + "strings" + "syscall" + "time" +) + +const scannerHash = "88f91962aa2f93ac6ab281d553b9e125f5197bbbce38f9f2437f7299c32e5509" +const scannerVersion = "8.30.1" +const maxInput = 512 * 1024 +const maxObject = 8 * 1024 * 1024 +const maxTotal = 64 * 1024 * 1024 +const maxObjects = 2000 +const maxRefs = 10000 + +var oidPattern = regexp.MustCompile(`^(?:[0-9a-f]{40}|[0-9a-f]{64})$`) +var errLimit = errors.New("output-limit") + +type result struct { + Status string `json:"status"` + Scope string `json:"scope"` + Reason string `json:"reason,omitempty"` + Scanned int `json:"scanned_objects"` + ScannedRefs int `json:"scanned_refs"` + BlockedID string `json:"blocked_object_id,omitempty"` + Version string `json:"scanner_version"` +} +type update struct{ Old, New, Ref string } +type cappedBuffer struct { + bytes.Buffer + limit int +} + +func (b *cappedBuffer) Write(p []byte) (int, error) { + if len(p) > b.limit-b.Len() { + return 0, errLimit + } + return b.Buffer.Write(p) +} +func zero(s string) bool { return strings.Trim(s, "0") == "" } +func parseUpdates(raw []byte) ([]update, error) { + if len(raw) == 0 || len(raw) > maxInput || raw[len(raw)-1] != '\n' { + return nil, errors.New("invalid-update-input") + } + lines := strings.Split(string(raw[:len(raw)-1]), "\n") + if len(lines) > 256 { + return nil, errors.New("update-limit") + } + seen := map[string]bool{} + var out []update + for _, line := range lines { + fields := strings.Split(line, " ") + if len(fields) != 3 || !oidPattern.MatchString(fields[0]) || !oidPattern.MatchString(fields[1]) || len(fields[0]) != len(fields[1]) || len(fields[2]) > 1024 || !strings.HasPrefix(fields[2], "refs/") || strings.ContainsAny(fields[2], "\x00\r\n\t") || seen[fields[2]] || zero(fields[0]) && zero(fields[1]) { + return nil, errors.New("invalid-update-input") + } + seen[fields[2]] = true + out = append(out, update{fields[0], fields[1], fields[2]}) + } + return out, nil +} + +// Only receive-pack object-location variables cross into Git. Scanner processes +// get none. Never inherit arbitrary GIT_CONFIG_* or credential/provider settings. +func gitEnvironment() []string { + env := []string{"PATH=/usr/local/bin:/usr/bin:/bin", "HOME=/nonexistent", "LANG=C", "GIT_CONFIG_NOSYSTEM=1", "GIT_CONFIG_GLOBAL=/dev/null", "GIT_NO_REPLACE_OBJECTS=1", "GIT_GRAFT_FILE=/dev/null", "GIT_NO_LAZY_FETCH=1", "GIT_TERMINAL_PROMPT=0", "GIT_ASKPASS=/bin/false", "GIT_SSH_COMMAND=/bin/false"} + for _, key := range []string{"GIT_DIR", "GIT_COMMON_DIR", "GIT_OBJECT_DIRECTORY", "GIT_ALTERNATE_OBJECT_DIRECTORIES", "GIT_QUARANTINE_PATH"} { + if val, ok := os.LookupEnv(key); ok { + env = append(env, key+"="+val) + } + } + return env +} +func git(ctx context.Context, input []byte, limit int, args ...string) ([]byte, error) { + cmd := exec.CommandContext(ctx, "/usr/bin/git", args...) + cmd.Env = gitEnvironment() + cmd.Stdin = bytes.NewReader(input) + b := &cappedBuffer{limit: limit} + cmd.Stdout = b + cmd.Stderr = io.Discard + if cmd.Run() != nil { + return nil, errors.New("git-" + args[0] + "-failed") + } + return b.Bytes(), nil +} +func refs(ctx context.Context) ([]byte, map[string]string, error) { + raw, err := git(ctx, nil, 2*1024*1024, "for-each-ref", "--format=%(objectname) %(refname)") + if err != nil { + return nil, nil, err + } + found := map[string]string{} + for _, line := range strings.Split(strings.TrimSuffix(string(raw), "\n"), "\n") { + if line == "" { + continue + } + parts := strings.Split(line, " ") + if len(parts) != 2 || !oidPattern.MatchString(parts[0]) || !strings.HasPrefix(parts[1], "refs/") { + return nil, nil, errors.New("invalid-ref-snapshot") + } + found[parts[1]] = parts[0] + } + if len(found) > maxRefs { + return nil, nil, errors.New("ref-limit") + } + return raw, found, nil +} +func pinScanner(path string) (*os.File, error) { + fd, err := syscall.Open(path, syscall.O_RDONLY|syscall.O_NOFOLLOW|syscall.O_NONBLOCK, 0) + if err != nil { + return nil, errors.New("scanner-file-refused") + } + f := os.NewFile(uintptr(fd), path) + ok := false + defer func() { + if !ok { + f.Close() + } + }() + st, err := f.Stat() + if err != nil { + return nil, errors.New("scanner-file-refused") + } + native, valid := st.Sys().(*syscall.Stat_t) + if !valid || !st.Mode().IsRegular() || st.Mode().Perm()&022 != 0 || st.Mode().Perm()&0111 == 0 || st.Size() > 64*1024*1024 || native.Uid != 0 && native.Uid != uint32(os.Getuid()) { + return nil, errors.New("scanner-file-refused") + } + hash := sha256.New() + if _, err = io.Copy(hash, f); err != nil || hex.EncodeToString(hash.Sum(nil)) != scannerHash { + return nil, errors.New("scanner-digest-mismatch") + } + ok = true + return f, nil +} +func scan(ctx context.Context, f *os.File, dir string, content []byte) (bool, error) { + cmd := exec.CommandContext(ctx, "/proc/self/fd/3", "stdin", "--config", filepath.Join(dir, "scanner.toml"), "--gitleaks-ignore-path", filepath.Join(dir, "ignore"), "--ignore-gitleaks-allow", "--redact=100", "--no-banner", "--no-color", "--log-level", "fatal", "--exit-code", "20", "--timeout", "15") + cmd.ExtraFiles = []*os.File{f} + cmd.Dir = dir + cmd.Env = []string{"PATH=/usr/bin:/bin", "HOME=" + dir, "TMPDIR=" + dir, "LANG=C"} + cmd.Stdin = bytes.NewReader(content) + cmd.Stdout = io.Discard + cmd.Stderr = io.Discard + err := cmd.Run() + if err == nil { + return false, nil + } + var exit *exec.ExitError + if errors.As(err, &exit) && exit.ExitCode() == 20 { + return true, nil + } + return false, errors.New("scanner-failed") +} + +// Tree object names are covered too. Gitlinks refer to an external repository, +// so this version refuses them rather than claiming transitive coverage. +func treeContent(raw []byte, oidBytes int) ([]byte, error) { + var names []byte + for len(raw) > 0 { + space := bytes.IndexByte(raw, ' ') + nul := bytes.IndexByte(raw, 0) + if space <= 0 || nul <= space || len(raw) < nul+1+oidBytes { + return nil, errors.New("invalid-tree") + } + mode := string(raw[:space]) + if mode == "160000" { + return nil, errors.New("external-gitlink-unverified") + } + if mode != "100644" && mode != "100755" && mode != "120000" && mode != "40000" { + return nil, errors.New("unsupported-tree-entry") + } + names = append(names, raw[space+1:nul]...) + names = append(names, '\n') + raw = raw[nul+1+oidBytes:] + } + return names, nil +} +func check(ctx context.Context, updates []update, binary string) (out result) { + out = result{Status: "unverified", Scope: "introduced-reachable-git-objects-and-updated-ref-names", Version: scannerVersion} + fail := func(err error) result { out.Reason = err.Error(); return out } + scanner, err := pinScanner(binary) + if err != nil { + return fail(err) + } + defer scanner.Close() + format, err := git(ctx, nil, 128, "rev-parse", "--show-object-format") + if err != nil { + return fail(err) + } + width := 40 + if string(format) == "sha256\n" { + width = 64 + } else if string(format) != "sha1\n" { + return fail(errors.New("unsupported-object-format")) + } + shallow, err := git(ctx, nil, 128, "rev-parse", "--is-shallow-repository") + if err != nil { + return fail(err) + } + if string(shallow) != "false\n" { + return fail(errors.New("shallow-repository-unverified")) + } + // Partial repositories may perform network retrieval while reading objects. + config, err := git(ctx, nil, 1024*1024, "config", "--local", "--includes", "--name-only", "--list") + if err != nil { + return fail(err) + } + for _, key := range strings.Split(strings.ToLower(string(config)), "\n") { + if key == "extensions.partialclone" || strings.HasSuffix(key, ".promisor") { + return fail(errors.New("partial-repository-unverified")) + } + } + snapshot, existing, err := refs(ctx) + if err != nil { + return fail(err) + } + tips := map[string]bool{} + for _, u := range updates { + if len(u.Old) != width { + return fail(errors.New("object-format-mismatch")) + } + if _, err = git(ctx, nil, 128, "check-ref-format", u.Ref); err != nil { + return fail(errors.New("invalid-ref-name")) + } + current, exists := existing[u.Ref] + if zero(u.Old) && exists || !zero(u.Old) && current != u.Old { + return fail(errors.New("old-ref-mismatch")) + } + if !zero(u.New) { + tips[u.New] = true + } + } + // Freeze exclusions from the pre-push snapshot, never a mutable --all read. + // Scan all introduced reachable objects, not just a tip diff or first parent. + var revisions strings.Builder + for tip := range tips { + revisions.WriteString(tip + "\n") + } + objects := map[string]bool{} + if len(tips) > 0 { + excluded := map[string]bool{} + for _, oid := range existing { + excluded[oid] = true + } + for oid := range excluded { + revisions.WriteString("^" + oid + "\n") + } + raw, e := git(ctx, []byte(revisions.String()), (maxObjects+1)*(width+1), "rev-list", "--objects", "--no-object-names", "--stdin") + if e != nil { + return fail(e) + } + for _, oid := range strings.Fields(string(raw)) { + if !oidPattern.MatchString(oid) || len(oid) != width { + return fail(errors.New("invalid-object-list")) + } + objects[oid] = true + } + } + if len(objects) > maxObjects { + return fail(errors.New("object-limit")) + } + ids := make([]string, 0, len(objects)) + for oid := range objects { + ids = append(ids, oid) + } + sort.Strings(ids) + var metadata []byte + if len(ids) > 0 { + metadata, err = git(ctx, []byte(strings.Join(ids, "\n")+"\n"), (maxObjects+1)*100, "cat-file", "--batch-check=%(objectname) %(objecttype) %(objectsize)") + if err != nil { + return fail(err) + } + } + lines := strings.Split(strings.TrimSuffix(string(metadata), "\n"), "\n") + if len(ids) == 0 { + lines = nil + } + if len(lines) != len(ids) { + return fail(errors.New("incomplete-object-metadata")) + } + // Reject oversized batches before reading/scanning any object payload. + type objectInfo struct { + kind string + size int64 + } + var objectMetadata []objectInfo + total := int64(0) + for i, line := range lines { + parts := strings.Split(line, " ") + if len(parts) != 3 || parts[0] != ids[i] { + return fail(errors.New("invalid-object-metadata")) + } + kind := parts[1] + if kind != "blob" && kind != "commit" && kind != "tag" && kind != "tree" { + return fail(errors.New("unsupported-object-type")) + } + size, e := strconv.ParseInt(parts[2], 10, 64) + if e != nil || size < 0 || size > maxObject || total+size > maxTotal { + return fail(errors.New("object-byte-limit")) + } + total += size + objectMetadata = append(objectMetadata, objectInfo{kind, size}) + } + dir, err := os.MkdirTemp("", "forge-secret-scan-") + if err != nil { + return fail(errors.New("temporary-directory-unavailable")) + } + defer os.RemoveAll(dir) + if os.WriteFile(filepath.Join(dir, "scanner.toml"), []byte("[extend]\nuseDefault = true\n"), 0600) != nil || os.WriteFile(filepath.Join(dir, "ignore"), nil, 0600) != nil { + return fail(errors.New("scanner-config-unavailable")) + } + // Ref names can themselves contain recognized tokens. Never echo them. + for _, u := range updates { + blocked, e := scan(ctx, scanner, dir, []byte(u.Ref)) + if e != nil { + return fail(e) + } + out.ScannedRefs++ + if blocked { + out.Status = "blocked" + out.Reason = "ref-name-detection" + return out + } + } + for i, info := range objectMetadata { + kind, size := info.kind, info.size + raw, e := git(ctx, nil, int(size)+1, "cat-file", kind, ids[i]) + if e != nil { + return fail(e) + } + if int64(len(raw)) != size { + return fail(errors.New("object-size-mismatch")) + } + if kind == "tree" { + raw, e = treeContent(raw, width/2) + if e != nil { + return fail(e) + } + } + blocked, e := scan(ctx, scanner, dir, raw) + if e != nil { + return fail(e) + } + out.Scanned++ + if blocked { + out.Status = "blocked" + out.BlockedID = ids[i] + return out + } + } + after, _, err := refs(ctx) + if err != nil { + return fail(err) + } + if !bytes.Equal(snapshot, after) { + return fail(errors.New("refs-changed-during-scan")) + } + out.Status = "clean" + return out +} +func main() { + flags := flag.NewFlagSet("receive-secret-scan", flag.ContinueOnError) + flags.SetOutput(io.Discard) + binary := flags.String("scanner", "", "reviewed Gitleaks executable") + out := result{Status: "unverified", Scope: "introduced-reachable-git-objects-and-updated-ref-names", Version: scannerVersion, Reason: "invalid-invocation"} + if flags.Parse(os.Args[1:]) == nil && *binary != "" && flags.NArg() == 0 { + raw, err := io.ReadAll(io.LimitReader(os.Stdin, maxInput+1)) + if err == nil { + updates, e := parseUpdates(raw) + if e != nil { + out.Reason = e.Error() + } else { + ctx, cancel := context.WithTimeout(context.Background(), 90*time.Second) + out = check(ctx, updates, *binary) + cancel() + } + } + } + // Only fixed reason codes, counts and an object ID may reach the caller. + encoded, _ := json.Marshal(out) + fmt.Println(string(encoded)) + if out.Status == "clean" { + return + } + if out.Status == "blocked" { + os.Exit(1) + } + os.Exit(2) +} diff --git a/tools/receive-secret-scan/main_test.go b/tools/receive-secret-scan/main_test.go new file mode 100644 index 0000000..b5abedd --- /dev/null +++ b/tools/receive-secret-scan/main_test.go @@ -0,0 +1,425 @@ +package main + +import ( + "bytes" + "context" + "crypto/ed25519" + "crypto/rand" + "crypto/x509" + "encoding/hex" + "encoding/json" + "encoding/pem" + "fmt" + "os" + "os/exec" + "path/filepath" + "regexp" + "strings" + "testing" + "time" +) + +func TestUpdateInputRefusesAmbiguity(t *testing.T) { + z := strings.Repeat("0", 40) + a := strings.Repeat("a", 40) + valid := z + " " + a + " refs/heads/main\n" + for _, s := range []string{"", "\n", strings.TrimSuffix(valid, "\n"), z + " " + z + " refs/heads/x\n", valid + valid, z + " " + a + " --all\n", z + " " + a + " refs/heads/a b\n", strings.Repeat(valid, 257)} { + if _, err := parseUpdates([]byte(s)); err == nil { + t.Fatal("accepted invalid input") + } + } + if _, err := parseUpdates([]byte(valid)); err != nil { + t.Fatal(err) + } +} +func TestTreeRefusesMalformedAndExternalEntries(t *testing.T) { + for _, raw := range [][]byte{[]byte("bad"), []byte("100644 a\x00short"), append([]byte("160000 submodule\x00"), make([]byte, 20)...)} { + if _, err := treeContent(raw, 20); err == nil { + t.Fatal("accepted unsupported tree") + } + } +} +func TestGitEnvironmentDoesNotInheritConfigOrSecrets(t *testing.T) { + t.Setenv("GIT_CONFIG_COUNT", "1") + t.Setenv("GIT_CONFIG_KEY_0", "core.hooksPath") + t.Setenv("GIT_CONFIG_VALUE_0", "/untrusted") + t.Setenv("SCANNER_CREDENTIAL", "synthetic") + t.Setenv("GIT_OBJECT_DIRECTORY", "/reviewed/quarantine") + env := strings.Join(gitEnvironment(), "\n") + if strings.Contains(env, "untrusted") || strings.Contains(env, "CREDENTIAL") || strings.Contains(env, "GIT_CONFIG_COUNT") || !strings.Contains(env, "GIT_OBJECT_DIRECTORY=/reviewed/quarantine") { + t.Fatal("environment boundary failed") + } +} +func TestCappedOutput(t *testing.T) { + b := &cappedBuffer{limit: 3} + if _, err := b.Write([]byte("four")); err == nil || b.Len() != 0 { + t.Fatal("unbounded output") + } +} +func run(t *testing.T, dir string, input []byte, args ...string) []byte { + t.Helper() + cmd := exec.Command(args[0], args[1:]...) + cmd.Dir = dir + cmd.Stdin = bytes.NewReader(input) + cmd.Env = append(os.Environ(), "GIT_CONFIG_NOSYSTEM=1", "GIT_CONFIG_GLOBAL=/dev/null", "GIT_TERMINAL_PROMPT=0") + out, err := cmd.CombinedOutput() + if err != nil { + t.Fatalf("disposable command failed (output withheld), executable %s", filepath.Base(args[0])) + } + return out +} +func write(t *testing.T, path string, b []byte) { + t.Helper() + if os.WriteFile(path, b, 0600) != nil { + t.Fatal("fixture write failed") + } +} +func key(t *testing.T) []byte { + _, priv, err := ed25519.GenerateKey(rand.Reader) + if err != nil { + t.Fatal(err) + } + der, err := x509.MarshalPKCS8PrivateKey(priv) + if err != nil { + t.Fatal(err) + } + return pem.EncodeToMemory(&pem.Block{Type: "PRIVATE KEY", Bytes: der}) +} +func binary(t *testing.T) (string, string) { + t.Helper() + scanner := os.Getenv("FORGE_TEST_GITLEAKS") + if scanner == "" { + t.Skip("requires explicitly admitted disposable Gitleaks binary") + } + if _, err := os.Stat(scanner); err != nil { + t.Fatal("configured test scanner missing") + } + path := filepath.Join(t.TempDir(), "adapter") + run(t, "", nil, "go", "build", "-buildvcs=false", "-o", path, ".") + return path, scanner +} + +type fixture struct{ client, server, adapter, scanner string } + +func setup(t *testing.T, adapter, scanner string) fixture { + t.Helper() + area := t.TempDir() + f := fixture{filepath.Join(area, "client"), filepath.Join(area, "remote.git"), adapter, scanner} + run(t, area, nil, "git", "init", "-q", "--bare", f.server) + run(t, area, nil, "git", "init", "-q", "-b", "main", f.client) + for _, dir := range []string{f.client, f.server} { + run(t, dir, nil, "git", "config", "core.hooksPath", filepath.Join(dir, "empty-hooks")) + } + run(t, f.client, nil, "git", "config", "user.name", "Disposable Test") + run(t, f.client, nil, "git", "config", "user.email", "probe@example.test") + f.commit(t, "source.txt", []byte("clean baseline\n"), "baseline") + run(t, f.client, nil, "git", "push", "-q", f.server, "HEAD:refs/heads/main") + run(t, f.server, nil, "git", "config", "core.hooksPath", filepath.Join(f.server, "hooks")) + f.install(t, scanner) + return f +} +func (f fixture) commit(t *testing.T, name string, content []byte, message string) string { + t.Helper() + write(t, filepath.Join(f.client, name), content) + run(t, f.client, nil, "git", "add", "--", name) + run(t, f.client, []byte(message), "git", "commit", "-q", "-F", "-") + return strings.TrimSpace(string(run(t, f.client, nil, "git", "rev-parse", "HEAD"))) +} +func quote(s string) string { return "'" + strings.ReplaceAll(s, "'", "'\\''") + "'" } +func (f fixture) install(t *testing.T, scanner string) { + t.Helper() + p := filepath.Join(f.server, "hooks", "pre-receive") + write(t, p, []byte("#!/bin/sh\nexec "+quote(f.adapter)+" --scanner "+quote(scanner)+"\n")) + if os.Chmod(p, 0755) != nil { + t.Fatal("chmod") + } +} +func (f fixture) push(t *testing.T, want int, args ...string) []byte { + t.Helper() + full := append([]string{"push", "--porcelain", f.server}, args...) + cmd := exec.Command("git", full...) + cmd.Dir = f.client + cmd.Env = append(os.Environ(), "GIT_CONFIG_NOSYSTEM=1", "GIT_CONFIG_GLOBAL=/dev/null") + out, err := cmd.CombinedOutput() + var observed result + for _, line := range bytes.Split(out, []byte("\n")) { + data := regexp.MustCompile(`\{"status":.*\}`).Find(line) + if len(data) > 0 { + _ = json.Unmarshal(data, &observed) + } + } + if want == 0 && err != nil || want != 0 && err == nil { + t.Fatalf("unexpected push outcome, want %d; status=%s reason=%s", want, observed.Status, observed.Reason) + } + expected := "clean" + if want == 1 { + expected = "blocked" + } + if want == 2 { + expected = "unverified" + } + if observed.Status != expected { + t.Fatalf("wrong control result, want %s; status=%s reason=%s", expected, observed.Status, observed.Reason) + } + if bytes.Contains(out, []byte("PRIVATE KEY")) { + t.Fatal("synthetic content leaked") + } + return out +} +func TestNativePushBoundaries(t *testing.T) { + adapter, scanner := binary(t) + t.Run("clean-existing-and-new-branch-and-deletion", func(t *testing.T) { + f := setup(t, adapter, scanner) + f.commit(t, "source.txt", []byte("changed clean\n"), "clean") + f.push(t, 0, "HEAD:refs/heads/main") + f.push(t, 0, "HEAD:refs/heads/feature") + f.push(t, 0, ":refs/heads/feature") + }) + t.Run("initial-repository-clean-and-secret", func(t *testing.T) { + for _, secret := range []bool{false, true} { + t.Run(fmt.Sprint(secret), func(t *testing.T) { + f := setup(t, adapter, scanner) + fresh := filepath.Join(t.TempDir(), "fresh.git") + run(t, "", nil, "git", "init", "-q", "--bare", fresh) + run(t, fresh, nil, "git", "config", "core.hooksPath", filepath.Join(fresh, "hooks")) + f.server = fresh + f.install(t, scanner) + want := 0 + if secret { + f.commit(t, "secret.txt", key(t), "fixture") + want = 1 + } + f.push(t, want, "HEAD:refs/heads/main") + }) + } + }) + t.Run("sha256-object-format", func(t *testing.T) { + t.Setenv("GIT_DEFAULT_HASH", "sha256") + f := setup(t, adapter, scanner) + f.commit(t, "source.txt", []byte("clean change\n"), "clean") + f.push(t, 0, "HEAD:refs/heads/main") + f.commit(t, "secret.txt", key(t), "fixture") + f.push(t, 1, "HEAD:refs/heads/main") + }) + t.Run("secret-in-tip", func(t *testing.T) { + f := setup(t, adapter, scanner) + oid := f.commit(t, "secret.txt", key(t), "synthetic fixture") + output := f.push(t, 1, "HEAD:refs/heads/main") + if !bytes.Contains(output, []byte(`"status":"blocked"`)) { + t.Fatal("not a detection") + } + c := exec.Command("git", "cat-file", "-e", oid) + c.Dir = f.server + if c.Run() == nil { + t.Fatal("rejected quarantined commit entered store") + } + }) + t.Run("secret-added-then-deleted", func(t *testing.T) { + f := setup(t, adapter, scanner) + f.commit(t, "secret.txt", key(t), "temporary fixture") + run(t, f.client, nil, "git", "rm", "-q", "secret.txt") + run(t, f.client, nil, "git", "commit", "-qm", "removed fixture") + out := f.push(t, 1, "HEAD:refs/heads/main") + if !bytes.Contains(out, []byte(`"status":"blocked"`)) { + t.Fatal("history not scanned") + } + }) + t.Run("new-branch-and-force-update", func(t *testing.T) { + f := setup(t, adapter, scanner) + f.commit(t, "secret.txt", key(t), "fixture") + f.push(t, 1, "HEAD:refs/heads/new") + f.push(t, 1, "+HEAD:refs/heads/main") + }) + t.Run("merge-second-parent-history", func(t *testing.T) { + f := setup(t, adapter, scanner) + run(t, f.client, nil, "git", "checkout", "-qb", "side") + f.commit(t, "secret.txt", key(t), "fixture") + run(t, f.client, nil, "git", "rm", "-q", "secret.txt") + run(t, f.client, nil, "git", "commit", "-qm", "remove fixture") + run(t, f.client, nil, "git", "checkout", "-q", "main") + f.commit(t, "main.txt", []byte("clean\n"), "main") + run(t, f.client, nil, "git", "merge", "-q", "--no-ff", "-m", "merge", "side") + f.push(t, 1, "HEAD:refs/heads/main") + }) + t.Run("commit-message", func(t *testing.T) { + f := setup(t, adapter, scanner) + f.commit(t, "source.txt", []byte("clean\n"), string(key(t))) + f.push(t, 1, "HEAD:refs/heads/main") + }) + t.Run("annotated-tag-message", func(t *testing.T) { + f := setup(t, adapter, scanner) + run(t, f.client, key(t), "git", "tag", "-a", "synthetic", "-F", "-") + f.push(t, 1, "refs/tags/synthetic") + }) + t.Run("tag-direct-to-blob", func(t *testing.T) { + f := setup(t, adapter, scanner) + oid := strings.TrimSpace(string(run(t, f.client, key(t), "git", "hash-object", "-w", "--stdin"))) + run(t, f.client, nil, "git", "update-ref", "refs/tags/blob", oid) + f.push(t, 1, "refs/tags/blob") + }) + t.Run("multi-ref-all-refused", func(t *testing.T) { + f := setup(t, adapter, scanner) + run(t, f.client, nil, "git", "branch", "clean") + f.commit(t, "secret.txt", key(t), "fixture") + f.push(t, 1, "clean:refs/heads/clean-new", "HEAD:refs/heads/main") + c := exec.Command("git", "show-ref", "--verify", "refs/heads/clean-new") + c.Dir = f.server + if c.Run() == nil { + t.Fatal("partial ref update") + } + }) + t.Run("missing-scanner-and-deletion-refused", func(t *testing.T) { + f := setup(t, adapter, scanner) + f.install(t, filepath.Join(t.TempDir(), "missing")) + f.commit(t, "source.txt", []byte("clean\n"), "clean") + out := f.push(t, 2, "HEAD:refs/heads/main") + if !bytes.Contains(out, []byte(`"status":"unverified"`)) { + t.Fatal("missing scanner accepted") + } + f.push(t, 2, ":refs/heads/main") + }) + t.Run("wrong-executable-refused", func(t *testing.T) { + f := setup(t, adapter, scanner) + fake := filepath.Join(t.TempDir(), "fake") + write(t, fake, []byte("#!/bin/sh\nexit 0\n")) + os.Chmod(fake, 0755) + f.install(t, fake) + f.commit(t, "source.txt", []byte("clean\n"), "clean") + f.push(t, 2, "HEAD:refs/heads/main") + }) + t.Run("inline-and-config-suppression-refused", func(t *testing.T) { + f := setup(t, adapter, scanner) + f.commit(t, ".gitleaks.toml", []byte("[allowlist]\nregexes = ['.*']\n"), "config") + f.commit(t, "secret.txt", append([]byte("# gitleaks:allow\n"), key(t)...), "fixture") + f.push(t, 1, "HEAD:refs/heads/main") + }) + t.Run("replacement-objects-cannot-hide-secret", func(t *testing.T) { + f := setup(t, adapter, scanner) + clean := strings.TrimSpace(string(run(t, f.client, nil, "git", "rev-parse", "HEAD"))) + bad := f.commit(t, "secret.txt", key(t), "fixture") + run(t, f.server, nil, "git", "fetch", "-q", f.client, "HEAD") + run(t, f.server, nil, "git", "replace", bad, clean) + f.push(t, 1, "HEAD:refs/heads/main") + }) + t.Run("grafts-cannot-hide-introduced-history", func(t *testing.T) { + f := setup(t, adapter, scanner) + f.commit(t, "secret.txt", key(t), "fixture") + run(t, f.client, nil, "git", "rm", "-q", "secret.txt") + run(t, f.client, nil, "git", "commit", "-qm", "remove") + tip := strings.TrimSpace(string(run(t, f.client, nil, "git", "rev-parse", "HEAD"))) + write(t, filepath.Join(f.server, "info/grafts"), []byte(tip+"\n")) + f.push(t, 1, "HEAD:refs/heads/main") + }) + t.Run("ref-name-token", func(t *testing.T) { + f := setup(t, adapter, scanner) + random := make([]byte, 10) + rand.Read(random) + f.push(t, 1, "HEAD:refs/heads/glpat-"+hex.EncodeToString(random)) + }) + t.Run("partial-config-include-refused", func(t *testing.T) { + f := setup(t, adapter, scanner) + config := filepath.Join(t.TempDir(), "partial-config") + write(t, config, []byte("[remote \"origin\"]\n promisor = true\n")) + run(t, f.server, nil, "git", "config", "include.path", config) + f.commit(t, "source.txt", []byte("clean\n"), "clean") + f.push(t, 2, "HEAD:refs/heads/main") + }) + t.Run("object-count-limit-refused", func(t *testing.T) { + f := setup(t, adapter, scanner) + for i := 0; i < maxObjects; i++ { + write(t, filepath.Join(f.client, fmt.Sprintf("file-%d.txt", i)), []byte(fmt.Sprintf("unique clean %d\n", i))) + } + run(t, f.client, nil, "git", "add", ".") + run(t, f.client, nil, "git", "commit", "-qm", "large object set") + f.push(t, 2, "HEAD:refs/heads/main") + }) + t.Run("external-gitlink-refused", func(t *testing.T) { + f := setup(t, adapter, scanner) + head := strings.TrimSpace(string(run(t, f.client, nil, "git", "rev-parse", "HEAD"))) + run(t, f.client, nil, "git", "update-index", "--add", "--cacheinfo", "160000,"+head+",external") + run(t, f.client, nil, "git", "commit", "-qm", "external gitlink") + f.push(t, 2, "HEAD:refs/heads/main") + }) + t.Run("partial-repository-refused", func(t *testing.T) { + f := setup(t, adapter, scanner) + run(t, f.server, nil, "git", "config", "remote.origin.promisor", "true") + f.commit(t, "source.txt", []byte("clean\n"), "clean") + f.push(t, 2, "HEAD:refs/heads/main") + }) + t.Run("aggregate-byte-limit-refused", func(t *testing.T) { + f := setup(t, adapter, scanner) + for i := 0; i < 9; i++ { + line := []byte(fmt.Sprintf("clean input %d\n", i)) + data := bytes.Repeat(line, (maxObject-1024)/len(line)) + write(t, filepath.Join(f.client, fmt.Sprintf("large-%d.txt", i)), data) + } + run(t, f.client, nil, "git", "add", ".") + run(t, f.client, nil, "git", "commit", "-qm", "large aggregate") + out := f.push(t, 2, "HEAD:refs/heads/main") + if !bytes.Contains(out, []byte(`"reason":"object-byte-limit"`)) || !bytes.Contains(out, []byte(`"scanned_objects":0`)) { + t.Fatal("aggregate bytes not bounded") + } + }) + t.Run("oversized-object-refused", func(t *testing.T) { + f := setup(t, adapter, scanner) + f.commit(t, "large.txt", bytes.Repeat([]byte("x"), maxObject+1), "large") + f.push(t, 2, "HEAD:refs/heads/main") + }) +} +func TestScannerFileAndTimeoutRefusal(t *testing.T) { + adapter, scanner := binary(t) + f := setup(t, adapter, scanner) + t.Run("nonexecutable", func(t *testing.T) { + p := filepath.Join(t.TempDir(), "scanner") + b, _ := os.ReadFile(scanner) + write(t, p, b) + if file, err := pinScanner(p); err == nil { + file.Close() + t.Fatal("accepted") + } + }) + t.Run("symlink", func(t *testing.T) { + p := filepath.Join(t.TempDir(), "scanner") + os.Symlink(scanner, p) + if file, err := pinScanner(p); err == nil { + file.Close() + t.Fatal("accepted") + } + }) + t.Run("expired-context", func(t *testing.T) { + old, _ := os.Getwd() + os.Chdir(f.server) + defer os.Chdir(old) + before := strings.TrimSpace(string(run(t, f.server, nil, "git", "rev-parse", "refs/heads/main"))) + ctx, cancel := context.WithDeadline(context.Background(), time.Now().Add(-time.Second)) + defer cancel() + got := check(ctx, []update{{before, before, "refs/heads/main"}}, scanner) + if got.Status != "unverified" { + t.Fatal("timeout accepted") + } + }) + t.Run("scanner-process-failure", func(t *testing.T) { + pinned, err := pinScanner(scanner) + if err != nil { + t.Fatal(err) + } + defer pinned.Close() + _, err = scan(context.Background(), pinned, t.TempDir(), []byte("clean")) + if err == nil || err.Error() != "scanner-failed" { + t.Fatal("scanner failure accepted") + } + }) + t.Run("unknown-old-ref", func(t *testing.T) { + cmd := exec.Command(adapter, "--scanner", scanner) + cmd.Dir = f.server + cmd.Stdin = strings.NewReader(strings.Repeat("b", 40) + " " + strings.Repeat("a", 40) + " refs/heads/main\n") + raw, err := cmd.Output() + if err == nil { + t.Fatal("accepted mismatch") + } + var got result + if json.Unmarshal(raw, &got) != nil || got.Reason != "old-ref-mismatch" { + t.Fatal("wrong refusal") + } + }) +} diff --git a/workplans/FORGE-WP-0006-independent-secret-detection.md b/workplans/FORGE-WP-0006-independent-secret-detection.md new file mode 100644 index 0000000..fe3a51c --- /dev/null +++ b/workplans/FORGE-WP-0006-independent-secret-detection.md @@ -0,0 +1,86 @@ +--- +id: FORGE-WP-0006 +type: workplan +title: "Enforce secret detection independently in the forge receive path" +domain: financials +repo: railiance-forge +status: active +owner: codex +topic_slug: railiance +created: "2026-10-12" +updated: "2026-10-12" +flavor: implementation +depends_on: [] +--- + +Own the native forge boundary of RMGR-WP-0021-T03 / CUST-WP-0080 G07. +Reuse the existing local scanner admission; do not duplicate incident custody, +source-workflow ownership or founder-deferred rotation. The prior runner/resource +workplans are finished and do not cover this missing control. This plan is +source/preparation authorization, not a claim of independent owner assent or +native installation. + +## Implement introduced-history refusal and verify the protocol + +```task +id: FORGE-WP-0006-T01 +status: done +priority: high +``` + +Native baseline: railiance01 Forgejo 11.0.3 stores 165 repository directories; +all 22 reviewed repositories have only upstream-generated receive hooks. Evidence: +Custodian `docs/evidence/2026-10-12-netkingdom-security/forge-secret-detection-baseline.json`. +The standalone adapter must cover all introduced reachable objects and ref names, +intermediate/deleted secrets, multiple parents, messages/tags, quarantine, force +updates, multi-ref refusal and scanner/tool failures. Preserve generated Forgejo +hooks and suppress finding contents. Record real push tests and precise limits. + +2026-10-12 source acceptance: the Linux Go adapter passes 33 leaf cases +(36 passing test nodes), no skips, using pinned Gitleaks and verified official +Go 1.26.9. Actual disposable Git pushes cover initial/SHA-256 repositories, clean/new/deleted branches, secret +blobs, added-then-deleted history, second-parent merges, commit/tag messages, +direct blob tags, force/multi-ref pushes, replacement/graft bypass refusal, +ref-name detections, config/inline suppression refusal, missing/wrong scanner, +partial clones (including included config), gitlinks and object count/byte limits. +Batch bytes are refused before scanning payloads. +Rejected quarantined commits were absent from the server object store. Scanner +mode/symlink/process failures, timeout and invalid old-ref tests also pass. These +prove adapter/protocol behavior only; no production hook is installed by T01. +Initial test-build setup and unsupported Git flag failures were corrected and +retained in Custodian's receipt; a rejected push without the expected detection +status does not count as a successful secret-detection test. + +## Admit the inventory and activate a reviewed server profile + +```task +id: FORGE-WP-0006-T02 +status: todo +priority: high +``` + +Classify the production-software denominator from the current forge metadata; +165 directories are not sufficient. Admit exact scanner/adapter build and runtime +versions, executable custody, bounded resource cost, reviewed hook composition, +abort owner, backup/rollback and receipt. Start with an explicit reviewed set, +record uncovered scope, and reconcile new repositories/mirrors/wikis/archives. +Do not enable user-authored arbitrary hooks. Prove clean acceptance, secret and +unavailable-scanner refusal, quarantine cleanup, unchanged authorization and +other hook behavior. Native acceptance must include SSH/HTTP/frontend creation +and merge paths plus restart/hook regeneration. Keep a missing proof open. + +## Verify CI independence, detection response and complete coverage + +```task +id: FORGE-WP-0006-T03 +status: todo +priority: high +``` + +Coordinate existing RMGR-WP-0021-T03/T04 without inventing owner acceptance. +Review who can alter workflows, required checks and enforcement. Exercise private +synthetic finding classification/routing with a revocation/rotation owner, false +positive governance, exact receipt and recovery path. Historical, log/artifact, +runtime and LFS gaps stay explicit. Complete only with native prevention and +response evidence for the admitted denominator; hand off residuals as live records +before finishing. Do not claim all DSM-DETECT requirements from hook tests alone.