diff --git a/docs/receive-secret-detection.md b/docs/receive-secret-detection.md index 2c06a3f..26c0afe 100644 --- a/docs/receive-secret-detection.md +++ b/docs/receive-secret-detection.md @@ -9,7 +9,8 @@ The source adapter is not evidence of installed server enforcement. The forge-owned, standalone Linux adapter composes with Forgejo's generated pre-receive dispatcher. It uses pinned Gitleaks 8.30.1 Linux amd64 (SHA256 `88f91962aa2f93ac6ab281d553b9e125f5197bbbce38f9f2437f7299c32e5509`). -It requires `/usr/bin/git` and `/proc` but no Python runtime, issuer credential, +It requires `/usr/bin/git`, `/proc`, a private writable tmpfs at `/dev/shm`, +and Linux pidfd support (5.3+ on amd64/arm64) but no Python runtime, issuer credential, network access, source checkout, or repository-controlled workflow. The current Forgejo 11.0.3 image has no Python interpreter; this keeps the runtime change to two reviewed executables and a hook fragment, subject to native admission. @@ -67,6 +68,26 @@ bytes are checked before scanning payloads. Limits reject rather than skip. Large imports need an admitted path; do not disable the gate to pass an unreviewed import. Git/scanner versions and limits require native load testing. +FORGE-WP-0010 corrects per-small-input scanner startup overhead without changing +these limits. It reuses Repo Manager's bounded profile: at most 64 inputs of at +most 4096 bytes per directory invocation. Larger inputs retain the existing stdin +reader and fragment behavior. Raw inputs use 0700 private tmpfs directories and +0600 files with safe object/ref identifiers; no source path controls a filename. +Missing or nonvolatile storage refuses instead of falling back to ordinary disk. + +A separately supervised child pins its receiver with pidfd, acknowledges readiness +before payload writes, and cleans up on pipe EOF. A 180-second backstop terminates +the exact stalled receiver before cleanup. Git/scanner children die when their +creating receiver thread dies; it is pinned to its OS thread during each child. +Normal cleanup and failed cleanup are checked. Directory identity replacement +refuses deletion. This is not protection against a hostile same-UID host process. + +Directory scans use the same pinned scanner/default config and explicit empty +ignore file. Bounded reports contain only known safe payload identifiers. Missing, +malformed, oversized or inconsistent reports, unknown diagnostics and scanner or +watchdog failures refuse. Only the exact pinned scanner's positive-finding summary +is accepted as a warning. The original stdin route is otherwise unchanged. + The adapter does not rescan objects already reachable before the push, scan unreachable extra pack objects, LFS payloads, other repositories, runtime logs, artifacts or credentials outside Git. It is not an authorization boundary against @@ -82,8 +103,10 @@ wikis, archives, mirrors and new-repository lifecycle before claiming DSM covera ## Build and disposable acceptance -Use the admitted Go toolchain (the current candidate uses official Go 1.26.9, -verified against the SHA256 from go.dev release metadata). No third-party Go +Use an admitted Go toolchain. The original deployed revision used Go 1.26.9, +verified against the SHA256 from go.dev release metadata. The FORGE-WP-0010 +source candidate is separately tested with installed Go 1.26.7; its new source, +compiler and exact executable still require native admission. No third-party Go modules are required. Keep binaries and logs outside the repository: ```sh diff --git a/tools/receive-secret-scan/batch.go b/tools/receive-secret-scan/batch.go new file mode 100644 index 0000000..429c226 --- /dev/null +++ b/tools/receive-secret-scan/batch.go @@ -0,0 +1,192 @@ +package main + +import ( + "bytes" + "context" + "crypto/sha256" + "encoding/hex" + "errors" + "io" + "os" + "os/exec" + "path/filepath" + "regexp" + "strings" + "syscall" +) + +// Same bounded small-input profile as Repo Manager's staged adapter. Above +// PIPE_BUF, keep the original stdin reader/fragment path byte-for-byte. +const directoryInputBytes = 4096 +const directoryBatchInputs = 64 +const maxReportBytes = 1024 * 1024 + +type scanInput struct { + name string + content []byte +} + +func verifyScannerFD(f *os.File) error { + st, err := f.Stat() + if err != nil { + return errors.New("scanner-file-refused") + } + native, ok := st.Sys().(*syscall.Stat_t) + if !ok || !st.Mode().IsRegular() || st.Mode().Perm()&022 != 0 || st.Mode().Perm()&0111 == 0 || st.Size() > 64*1024*1024 || (native.Uid != 0 && native.Uid != uint32(os.Getuid())) { + return errors.New("scanner-file-refused") + } + if _, err = f.Seek(0, 0); err != nil { + return errors.New("scanner-file-refused") + } + hash := sha256.New() + if _, err = io.Copy(hash, f); err != nil || hex.EncodeToString(hash.Sum(nil)) != scannerHash { + return errors.New("scanner-digest-mismatch") + } + if _, err = f.Seek(0, 0); err != nil { + return errors.New("scanner-file-refused") + } + return nil +} + +// Trusted child mode establishes resource limits before exec of the exact +// independently pinned scanner descriptor. It accepts no repository commands. +func batchScannerProcess(args []string) int { + if len(args) != 2 { + return 127 + } + if _, err := volatileIdentity(args[1]); err != nil { + return 127 + } + f := os.NewFile(3, "pinned-scanner") + if f == nil || verifyScannerFD(f) != nil { + return 127 + } + defer f.Close() + for resource, limit := range map[int]uint64{syscall.RLIMIT_FSIZE: maxReportBytes, syscall.RLIMIT_CPU: 20, syscall.RLIMIT_AS: 8 * 1024 * 1024 * 1024} { + if syscall.Setrlimit(resource, &syscall.Rlimit{Cur: limit, Max: limit}) != nil { + return 127 + } + } + if os.Chdir(args[1]) != nil { + return 127 + } + argv := []string{"gitleaks", "dir", "--config", filepath.Join(args[0], "scanner.toml"), "--gitleaks-ignore-path", filepath.Join(args[0], "ignore"), "--ignore-gitleaks-allow", "--redact=100", "--no-banner", "--no-color", "--log-level", "warn", "--exit-code", "20", "--timeout", "15", "--report-format", "template", "--report-template", filepath.Join(args[0], "template"), "--report-path", filepath.Join(args[0], "report.paths"), "--max-target-megabytes", "0", "payload"} + if syscall.Exec("/proc/self/fd/3", argv, []string{"PATH=/usr/bin:/bin", "HOME=" + args[1], "TMPDIR=" + args[1], "LANG=C"}) != nil { + return 127 + } + return 127 +} + +var safeInputName = regexp.MustCompile(`^(?:[0-9a-f]{40}|[0-9a-f]{64}|ref-[0-9]{1,3})$`) +var leakSummary = regexp.MustCompile(`^[0-9]{1,2}:[0-9]{2}(?:AM|PM) WRN leaks found: [1-9][0-9]*\n$`) + +func validateBatchDiagnostics(code int, diagnostics []byte) error { + if code != 0 && code != 20 || len(diagnostics) > 0 && !(code == 20 && leakSummary.Match(diagnostics)) { + return errors.New("scanner-failed") + } + return nil +} + +func batchFindings(code int, raw []byte, known map[string]string) (map[string]bool, error) { + findings := map[string]bool{} + if len(raw) > 0 { + if !bytes.HasSuffix(raw, []byte("\n")) { + return nil, errors.New("scanner-report-refused") + } + for _, line := range strings.Split(string(raw[:len(raw)-1]), "\n") { + name, ok := known[line] + if !ok { + return nil, errors.New("scanner-report-refused") + } + findings[name] = true + } + } + if (code != 0 && code != 20) || (len(findings) > 0) != (code == 20) { + return nil, errors.New("scanner-report-refused") + } + return findings, nil +} + +func scanSmall(ctx context.Context, scanner *os.File, config string, storage *volatileStorage, inputs []scanInput) (blocked string, err error) { + if len(inputs) == 0 || len(inputs) > directoryBatchInputs || !storage.live() { + return "", errors.New("cleanup-watchdog-unavailable") + } + id, e := volatileIdentity(storage.path) + if e != nil || id != storage.id { + return "", errors.New("volatile-storage-identity-changed") + } + payload := filepath.Join(storage.path, "payload") + if os.Mkdir(payload, 0700) != nil { + return "", errors.New("volatile-storage-unavailable") + } + defer func() { + if os.RemoveAll(payload) != nil && err == nil { + err = errors.New("volatile-storage-cleanup-failed") + } + }() + known := map[string]string{} + for _, item := range inputs { + if !safeInputName.MatchString(item.name) || len(item.content) > directoryInputBytes || known["payload/"+item.name] != "" { + return "", errors.New("invalid-scan-batch") + } + known["payload/"+item.name] = item.name + file, e := os.OpenFile(filepath.Join(payload, item.name), os.O_WRONLY|os.O_CREATE|os.O_EXCL|syscall.O_NOFOLLOW, 0600) + if e != nil { + return "", errors.New("volatile-storage-write-failed") + } + _, e = file.Write(item.content) + closed := file.Close() + if e != nil || closed != nil { + return "", errors.New("volatile-storage-write-failed") + } + } + if os.WriteFile(filepath.Join(config, "template"), []byte("{{range .}}{{.File}}\n{{end}}"), 0600) != nil { + return "", errors.New("scanner-config-unavailable") + } + report := filepath.Join(config, "report.paths") + if e = os.Remove(report); e != nil && !os.IsNotExist(e) { + return "", errors.New("scanner-report-refused") + } + cmd := exec.CommandContext(ctx, "/proc/self/exe", "--batch-scanner", config, storage.path) + cmd.ExtraFiles = []*os.File{scanner} + cmd.Env = []string{"PATH=/usr/bin:/bin", "LANG=C"} + diagnostics := &cappedBuffer{limit: maxReportBytes} + cmd.Stdout = io.Discard + cmd.Stderr = diagnostics + runErr := runChild(cmd) + if ctx.Err() != nil { + return "", errors.New("scan-deadline-exceeded") + } + code := 0 + var exit *exec.ExitError + if runErr != nil { + if !errors.As(runErr, &exit) { + return "", errors.New("scanner-failed") + } + code = exit.ExitCode() + } + if e = validateBatchDiagnostics(code, diagnostics.Bytes()); e != nil { + return "", e + } + st, e := os.Lstat(report) + if e != nil || !st.Mode().IsRegular() || st.Size() > maxReportBytes { + return "", errors.New("scanner-report-refused") + } + raw, e := os.ReadFile(report) + if e != nil { + return "", errors.New("scanner-report-refused") + } + findings, e := batchFindings(code, raw, known) + if e != nil { + return "", e + } + if !storage.live() { + return "", errors.New("scanner-report-refused") + } + for _, item := range inputs { + if findings[item.name] { + return item.name, nil + } + } + return "", nil +} diff --git a/tools/receive-secret-scan/batch_test.go b/tools/receive-secret-scan/batch_test.go new file mode 100644 index 0000000..d2b5278 --- /dev/null +++ b/tools/receive-secret-scan/batch_test.go @@ -0,0 +1,323 @@ +package main + +import ( + "bufio" + "bytes" + "context" + "crypto/rand" + "encoding/hex" + "encoding/json" + "fmt" + "os" + "os/exec" + "path/filepath" + "strings" + "testing" + "time" +) + +func TestMain(m *testing.M) { + if code, handled := internalProcess(os.Args[1:]); handled { + os.Exit(code) + } + if len(os.Args) == 2 && os.Args[1] == "--test-storage-refusal" { + storage, err := newVolatileStorage() + if err == nil { + storage.close() + os.Exit(3) + } + json.NewEncoder(os.Stdout).Encode(map[string]string{"status": "unverified", "reason": err.Error()}) + os.Exit(0) + } + if len(os.Args) == 2 && os.Args[1] == "--test-storage-parent" { + storage, err := newVolatileStorage() + if err != nil { + os.Exit(3) + } + if os.WriteFile(filepath.Join(storage.path, "input"), []byte("synthetic private test input"), 0600) != nil { + os.Exit(3) + } + json.NewEncoder(os.Stdout).Encode(map[string]any{"path": storage.path, "watchdog": storage.command.Process.Pid}) + child := exec.Command("/proc/self/exe", "--test-storage-reader", storage.path) + child.Stdout = os.Stdout + if runChild(child) != nil { + os.Exit(4) + } + os.Exit(0) + } + if len(os.Args) == 3 && os.Args[1] == "--test-storage-reader" { + f, err := os.Open(filepath.Join(os.Args[2], "input")) + if err != nil { + os.Exit(3) + } + defer f.Close() + fmt.Fprintln(os.Stdout, os.Getpid()) + time.Sleep(time.Minute) + os.Exit(0) + } + os.Exit(m.Run()) +} + +func TestVolatileStorageLifecycle(t *testing.T) { + storage, err := newVolatileStorage() + if err != nil { + t.Fatal(err) + } + if !storage.live() { + t.Fatal("watchdog did not remain live") + } + if err = os.WriteFile(filepath.Join(storage.path, "input"), []byte("synthetic input"), 0600); err != nil { + t.Fatal("cannot write private fixture") + } + if err = storage.close(); err != nil { + t.Fatal(err) + } + if _, err = os.Lstat(storage.path); !os.IsNotExist(err) { + t.Fatal("private storage retained") + } +} + +func TestVolatileIdentityRefusesReplacement(t *testing.T) { + storage, err := newVolatileStorage() + if err != nil { + t.Fatal(err) + } + // Keep the original inode alive so a new directory cannot reuse it. + original := storage.path + "-original" + if os.Rename(storage.path, original) != nil || os.Mkdir(storage.path, 0700) != nil { + t.Fatal("cannot prepare replacement") + } + defer os.RemoveAll(original) + defer os.RemoveAll(storage.path) + if err = storage.close(); err == nil { + t.Fatal("replacement directory accepted") + } + if _, err = os.Stat(storage.path); err != nil { + t.Fatal("replacement directory was removed") + } +} + +func TestReceiverDeathCleansStorageAndStopsReader(t *testing.T) { + cmd := exec.Command("/proc/self/exe", "--test-storage-parent") + pipe, err := cmd.StdoutPipe() + if err != nil || cmd.Start() != nil { + t.Fatal("cannot start disposable parent") + } + defer cmd.Process.Kill() + reader := bufio.NewReader(pipe) + line, err := reader.ReadBytes('\n') + var receipt struct { + Path string + Watchdog int + } + if err != nil || json.Unmarshal(line, &receipt) != nil || receipt.Path == "" { + t.Fatal("disposable parent did not prepare storage") + } + var childPID int + if _, err = fmt.Fscanln(reader, &childPID); err != nil || childPID < 2 { + t.Fatal("disposable reader did not open input") + } + if cmd.Process.Kill() != nil { + t.Fatal("cannot kill disposable parent") + } + cmd.Wait() + deadline := time.Now().Add(5 * time.Second) + for time.Now().Before(deadline) { + _, pathErr := os.Lstat(receipt.Path) + stat, procErr := os.ReadFile(fmt.Sprintf("/proc/%d/stat", childPID)) + terminal := os.IsNotExist(procErr) + if procErr == nil { + // Zombies are terminal; the private PID namespace reaps them. + fields := strings.Fields(string(stat)[strings.LastIndex(string(stat), ")")+1:]) + terminal = len(fields) > 0 && fields[0] == "Z" + } + if os.IsNotExist(pathErr) && terminal { + return + } + time.Sleep(10 * time.Millisecond) + } + t.Fatal("receiver loss retained private input or live reader") +} + +func TestSmallInputParity(t *testing.T) { + scanner := os.Getenv("FORGE_TEST_GITLEAKS") + if scanner == "" { + t.Skip("requires explicitly admitted disposable Gitleaks binary") + } + f, err := pinScanner(scanner) + if err != nil { + t.Fatal(err) + } + defer f.Close() + storage, err := newVolatileStorage() + if err != nil { + t.Fatal(err) + } + defer func() { + if err := storage.close(); err != nil { + t.Error(err) + } + }() + config := t.TempDir() + write(t, filepath.Join(config, "scanner.toml"), []byte("[extend]\nuseDefault = true\n")) + write(t, filepath.Join(config, "ignore"), nil) + privateKey := key(t) + var random [32]byte + if _, err := rand.Read(random[:]); err != nil { + t.Fatal("cannot generate synthetic token") + } + canary := []byte("glpat-" + hex.EncodeToString(random[:])) + for _, tc := range []struct { + name string + input []byte + want bool + }{ + {"empty", nil, false}, + {"clean", []byte("reviewed clean input\n"), false}, + {"exact-limit", bytes.Repeat([]byte("x"), directoryInputBytes), false}, + {"private-key", privateKey, true}, + {"exact-limit-private-key", append(bytes.Repeat([]byte("\n"), directoryInputBytes-len(privateKey)), privateKey...), true}, + {"binary-private-key", append([]byte{0, 1, 0}, privateKey...), true}, + {"token", canary, true}, + {"binary-token", append([]byte{0, 0}, canary...), true}, + {"inline-suppression", append([]byte("# gitleaks:allow\n"), privateKey...), true}, + } { + t.Run(tc.name, func(t *testing.T) { + stdin, e := scan(context.Background(), f, config, tc.input) + if e != nil || stdin != tc.want { + t.Fatal("stdin reference outcome differs", e) + } + name := strings.Repeat("a", 40) + blocked, e := scanSmall(context.Background(), f, config, storage, []scanInput{{name, tc.input}}) + if e != nil || (blocked != "") != stdin || blocked != "" && blocked != name { + t.Fatal("directory and original stdin detection differ", e) + } + }) + } + t.Run("sixty-four-distinct-inputs", func(t *testing.T) { + var inputs []scanInput + for i := 0; i < directoryBatchInputs; i++ { + inputs = append(inputs, scanInput{fmt.Sprintf("%040x", i), []byte("clean")}) + } + inputs[63].content = privateKey + blocked, e := scanSmall(context.Background(), f, config, storage, inputs) + if e != nil || blocked != inputs[63].name { + t.Fatal("last batch input was not detected", e) + } + }) + t.Run("expired-context", func(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + cancel() + _, e := scanSmall(ctx, f, config, storage, []scanInput{{strings.Repeat("b", 40), []byte("clean")}}) + if e == nil || e.Error() != "scan-deadline-exceeded" { + t.Fatal("expired scan did not fail closed") + } + }) + t.Run("invalid-inputs", func(t *testing.T) { + name := strings.Repeat("c", 40) + for _, inputs := range [][]scanInput{ + {{"../escape", []byte("clean")}}, + {{name, bytes.Repeat([]byte("x"), directoryInputBytes+1)}}, + {{name, []byte("clean")}, {name, []byte("clean")}}, + } { + if _, e := scanSmall(context.Background(), f, config, storage, inputs); e == nil { + t.Fatal("invalid batch input accepted") + } + if _, e := os.Lstat(filepath.Join(storage.path, "payload")); !os.IsNotExist(e) { + t.Fatal("invalid batch left input files") + } + } + }) + t.Run("scanner-failure", func(t *testing.T) { + write(t, filepath.Join(config, "scanner.toml"), []byte("invalid = [")) + if _, e := scanSmall(context.Background(), f, config, storage, []scanInput{{strings.Repeat("d", 40), []byte("clean")}}); e == nil { + t.Fatal("scanner failure accepted") + } + }) +} + +func TestBatchAndLargerInputPushBoundaries(t *testing.T) { + adapter, scanner := binary(t) + for _, tc := range []struct { + name string + size int + secret bool + }{ + {"exact-limit-key", directoryInputBytes, true}, + {"larger-stdin-key", directoryInputBytes + 1, true}, + {"larger-stdin-clean", 70 * 1024, false}, + } { + t.Run(tc.name, func(t *testing.T) { + f := setup(t, adapter, scanner) + content := []byte("reviewed clean input\n") + want := 0 + if tc.secret { + content = key(t) + want = 1 + } + content = append(content, bytes.Repeat([]byte("\n"), tc.size-len(content))...) + f.commit(t, "input.bin", content, "disposable boundary fixture") + f.push(t, want, "HEAD:refs/heads/main") + }) + } + t.Run("multiple-ref-name-detection", func(t *testing.T) { + f := setup(t, adapter, scanner) + var random [32]byte + if _, err := rand.Read(random[:]); err != nil { + t.Fatal("fixture generation failed") + } + out := f.push(t, 1, "HEAD:refs/heads/clean-new", "HEAD:refs/heads/glpat-"+hex.EncodeToString(random[:])) + if !bytes.Contains(out, []byte(`"reason":"ref-name-detection"`)) { + t.Fatal("multi-ref name was not scanned") + } + }) +} + +func TestBatchRefusesUnverifiedReportsAndDiagnostics(t *testing.T) { + known := map[string]string{"payload/ref-0": "ref-0"} + for _, tc := range []struct { + code int + raw string + }{ + {0, "payload/ref-0\n"}, {20, ""}, {0, "\n"}, + {20, "payload/ref-0"}, {20, "payload/ref-0\n\n"}, + {20, "payload/ref-unknown\n"}, {20, "../payload/ref-0\n"}, + {20, "payload/ref-0\r\n"}, {2, ""}, + } { + if _, err := batchFindings(tc.code, []byte(tc.raw), known); err == nil { + t.Fatal("unverified report accepted") + } + } + for _, tc := range []struct { + code int + raw string + }{ + {0, ""}, {20, "payload/ref-0\n"}, {20, "payload/ref-0\npayload/ref-0\n"}, + } { + if _, err := batchFindings(tc.code, []byte(tc.raw), known); err != nil { + t.Fatal("valid report refused") + } + } + for _, tc := range []struct { + code int + text string + }{ + {0, "12:03PM WRN incomplete scan\n"}, {20, "12:03PM WRN incomplete scan\n"}, + {20, "12:03PM WRN leaks found: 1\n12:03PM WRN incomplete scan\n"}, + {0, "12:03PM WRN leaks found: 1\n"}, {2, ""}, + } { + if validateBatchDiagnostics(tc.code, []byte(tc.text)) == nil { + t.Fatal("unverified diagnostic accepted") + } + } + for _, tc := range []struct { + code int + text string + }{ + {0, ""}, {20, ""}, {20, "12:03PM WRN leaks found: 1\n"}, + } { + if validateBatchDiagnostics(tc.code, []byte(tc.text)) != nil { + t.Fatal("valid diagnostic refused") + } + } +} diff --git a/tools/receive-secret-scan/main.go b/tools/receive-secret-scan/main.go index 61b16cc..ec28e58 100644 --- a/tools/receive-secret-scan/main.go +++ b/tools/receive-secret-scan/main.go @@ -5,8 +5,6 @@ package main import ( "bytes" "context" - "crypto/sha256" - "encoding/hex" "encoding/json" "errors" "flag" @@ -95,7 +93,7 @@ func git(ctx context.Context, input []byte, limit int, args ...string) ([]byte, b := &cappedBuffer{limit: limit} cmd.Stdout = b cmd.Stderr = io.Discard - if cmd.Run() != nil { + if runChild(cmd) != nil { return nil, errors.New("git-" + args[0] + "-failed") } return b.Bytes(), nil @@ -133,17 +131,8 @@ func pinScanner(path string) (*os.File, error) { f.Close() } }() - st, err := f.Stat() - if err != nil { - return nil, errors.New("scanner-file-refused") - } - native, valid := st.Sys().(*syscall.Stat_t) - if !valid || !st.Mode().IsRegular() || st.Mode().Perm()&022 != 0 || st.Mode().Perm()&0111 == 0 || st.Size() > 64*1024*1024 || native.Uid != 0 && native.Uid != uint32(os.Getuid()) { - return nil, errors.New("scanner-file-refused") - } - hash := sha256.New() - if _, err = io.Copy(hash, f); err != nil || hex.EncodeToString(hash.Sum(nil)) != scannerHash { - return nil, errors.New("scanner-digest-mismatch") + if err = verifyScannerFD(f); err != nil { + return nil, err } ok = true return f, nil @@ -156,7 +145,10 @@ func scan(ctx context.Context, f *os.File, dir string, content []byte) (bool, er cmd.Stdin = bytes.NewReader(content) cmd.Stdout = io.Discard cmd.Stderr = io.Discard - err := cmd.Run() + err := runChild(cmd) + if ctx.Err() != nil { + return false, errors.New("scan-deadline-exceeded") + } if err == nil { return false, nil } @@ -321,22 +313,86 @@ func check(ctx context.Context, updates []update, binary string) (out result) { return fail(errors.New("temporary-directory-unavailable")) } defer os.RemoveAll(dir) + var storage *volatileStorage + defer func() { + if storage != nil { + if e := storage.close(); e != nil { + out.Status = "unverified" + out.Reason = e.Error() + } + } + }() + getStorage := func() error { + if storage == nil { + var e error + storage, e = newVolatileStorage() + return e + } + if !storage.live() { + return errors.New("cleanup-watchdog-unavailable") + } + return nil + } if os.WriteFile(filepath.Join(dir, "scanner.toml"), []byte("[extend]\nuseDefault = true\n"), 0600) != nil || os.WriteFile(filepath.Join(dir, "ignore"), nil, 0600) != nil { return fail(errors.New("scanner-config-unavailable")) } // Ref names can themselves contain recognized tokens. Never echo them. - for _, u := range updates { - blocked, e := scan(ctx, scanner, dir, []byte(u.Ref)) + for start := 0; start < len(updates); start += directoryBatchInputs { + end := start + directoryBatchInputs + if end > len(updates) { + end = len(updates) + } + if end-start == 1 { + blocked, e := scan(ctx, scanner, dir, []byte(updates[start].Ref)) + if e != nil { + return fail(e) + } + out.ScannedRefs++ + if blocked { + out.Status = "blocked" + out.Reason = "ref-name-detection" + return out + } + continue + } + if e := getStorage(); e != nil { + return fail(e) + } + var batch []scanInput + for i := start; i < end; i++ { + batch = append(batch, scanInput{fmt.Sprintf("ref-%d", i), []byte(updates[i].Ref)}) + } + blocked, e := scanSmall(ctx, scanner, dir, storage, batch) if e != nil { return fail(e) } - out.ScannedRefs++ - if blocked { + out.ScannedRefs += len(batch) + if blocked != "" { out.Status = "blocked" out.Reason = "ref-name-detection" return out } } + var pending []scanInput + flush := func() error { + if len(pending) == 0 { + return nil + } + if e := getStorage(); e != nil { + return e + } + blocked, e := scanSmall(ctx, scanner, dir, storage, pending) + if e != nil { + return e + } + out.Scanned += len(pending) + pending = nil + if blocked != "" { + out.Status = "blocked" + out.BlockedID = blocked + } + return nil + } for i, info := range objectMetadata { kind, size := info.kind, info.size raw, e := git(ctx, nil, int(size)+1, "cat-file", kind, ids[i]) @@ -352,6 +408,18 @@ func check(ctx context.Context, updates []update, binary string) (out result) { return fail(e) } } + if len(raw) <= directoryInputBytes { + pending = append(pending, scanInput{ids[i], raw}) + if len(pending) == directoryBatchInputs { + if e = flush(); e != nil { + return fail(e) + } + if out.Status == "blocked" { + return out + } + } + continue + } blocked, e := scan(ctx, scanner, dir, raw) if e != nil { return fail(e) @@ -363,6 +431,12 @@ func check(ctx context.Context, updates []update, binary string) (out result) { return out } } + if err = flush(); err != nil { + return fail(err) + } + if out.Status == "blocked" { + return out + } after, _, err := refs(ctx) if err != nil { return fail(err) @@ -374,6 +448,9 @@ func check(ctx context.Context, updates []update, binary string) (out result) { return out } func main() { + if code, handled := internalProcess(os.Args[1:]); handled { + os.Exit(code) + } flags := flag.NewFlagSet("receive-secret-scan", flag.ContinueOnError) flags.SetOutput(io.Discard) binary := flags.String("scanner", "", "reviewed Gitleaks executable") diff --git a/tools/receive-secret-scan/volatile.go b/tools/receive-secret-scan/volatile.go new file mode 100644 index 0000000..4da2201 --- /dev/null +++ b/tools/receive-secret-scan/volatile.go @@ -0,0 +1,268 @@ +// Linux volatile input custody; no raw input is written to ordinary disk. +package main + +import ( + "errors" + "fmt" + "io" + "os" + "os/exec" + "path/filepath" + "runtime" + "strconv" + "strings" + "syscall" + "time" +) + +type storageIdentity struct { + dev uint64 + ino uint64 + uid uint32 +} + +func volatileIdentity(path string) (storageIdentity, error) { + var id storageIdentity + if filepath.Dir(path) != "/dev/shm" || !strings.HasPrefix(filepath.Base(path), "forge-receive-scan-") { + return id, errors.New("volatile-storage-refused") + } + fd, err := syscall.Open(path, syscall.O_RDONLY|syscall.O_DIRECTORY|syscall.O_NOFOLLOW|syscall.O_CLOEXEC, 0) + if err != nil { + return id, errors.New("volatile-storage-unavailable") + } + defer syscall.Close(fd) + var st syscall.Stat_t + var fs syscall.Statfs_t + if syscall.Fstat(fd, &st) != nil || syscall.Fstatfs(fd, &fs) != nil || fs.Type != 0x01021994 || st.Mode&07777 != 0700 || st.Uid != uint32(os.Getuid()) { + return id, errors.New("volatile-storage-refused") + } + return storageIdentity{uint64(st.Dev), st.Ino, st.Uid}, nil +} + +func removeVolatile(path string, expected storageIdentity) error { + if _, err := os.Lstat(path); os.IsNotExist(err) { + return nil + } + id, err := volatileIdentity(path) + if err != nil || id != expected { + return errors.New("volatile-storage-identity-changed") + } + if os.RemoveAll(path) != nil { + return errors.New("volatile-storage-cleanup-failed") + } + return nil +} + +// Pin the exact parent using Linux pidfd, not a reusable numeric process ID. +func parentDescriptor(parent int) (*os.File, error) { + if runtime.GOARCH != "amd64" && runtime.GOARCH != "arm64" { + return nil, errors.New("cleanup-platform-unsupported") + } + if parent <= 1 || os.Getppid() != parent { + return nil, errors.New("cleanup-parent-refused") + } + fd, _, errno := syscall.Syscall(434, uintptr(parent), 0, 0) // pidfd_open, Linux 5.3+ + if errno != 0 { + return nil, errors.New("cleanup-parent-unavailable") + } + f := os.NewFile(fd, "parent-pidfd") + if os.Getppid() != parent || fd >= 1024 { + f.Close() + return nil, errors.New("cleanup-parent-refused") + } + return f, nil +} + +func waitParent(f *os.File, duration time.Duration) bool { + fd := int(f.Fd()) + var deadline time.Time + if duration >= 0 { + deadline = time.Now().Add(duration) + } + for { + // select may mutate both arguments, including when interrupted. + var set syscall.FdSet + set.Bits[fd/64] |= 1 << uint(fd%64) + var timeout *syscall.Timeval + if duration >= 0 { + remaining := time.Until(deadline) + if remaining < 0 { + remaining = 0 + } + value := syscall.NsecToTimeval(remaining.Nanoseconds()) + timeout = &value + } + n, err := syscall.Select(fd+1, &set, nil, nil, timeout) + if err == syscall.EINTR { + continue + } + return err == nil && n > 0 + } +} + +func signalParent(f *os.File, signal syscall.Signal) { + _, _, _ = syscall.Syscall6(424, f.Fd(), uintptr(signal), 0, 0, 0, 0) // pidfd_send_signal +} + +func cleanupProcess(args []string) int { + if len(args) != 5 { + return 4 + } + dev, e1 := strconv.ParseUint(args[1], 10, 64) + ino, e2 := strconv.ParseUint(args[2], 10, 64) + uid, e3 := strconv.ParseUint(args[3], 10, 32) + parent, e4 := strconv.Atoi(args[4]) + if e1 != nil || e2 != nil || e3 != nil || e4 != nil { + return 4 + } + expected := storageIdentity{dev, ino, uint32(uid)} + id, err := volatileIdentity(args[0]) + if err != nil || id != expected { + return 3 + } + process, err := parentDescriptor(parent) + if err != nil { + return 4 + } + defer process.Close() + if _, err = os.Stdout.Write([]byte("R")); err != nil { + return 4 + } + eof := make(chan bool, 1) + go func() { var b [1]byte; n, e := os.Stdin.Read(b[:]); eof <- n == 0 && e == io.EOF }() + select { + case ok := <-eof: + if !ok { + return 4 + } + case <-time.After(180 * time.Second): + signalParent(process, syscall.SIGTERM) + if !waitParent(process, 5*time.Second) { + signalParent(process, syscall.SIGKILL) + if !waitParent(process, -1) { + return 4 + } + } + } + if removeVolatile(args[0], expected) != nil { + return 3 + } + return 0 +} + +type volatileStorage struct { + path string + id storageIdentity + pipe io.WriteCloser + command *exec.Cmd + done chan error +} + +func newVolatileStorage() (*volatileStorage, error) { + root, err := syscall.Open("/dev/shm", syscall.O_RDONLY|syscall.O_DIRECTORY|syscall.O_NOFOLLOW|syscall.O_CLOEXEC, 0) + if err != nil { + return nil, errors.New("volatile-storage-unavailable") + } + var fs syscall.Statfs_t + err = syscall.Fstatfs(root, &fs) + syscall.Close(root) + if err != nil || fs.Type != 0x01021994 { + return nil, errors.New("volatile-storage-unavailable") + } + path, err := os.MkdirTemp("/dev/shm", "forge-receive-scan-") + if err != nil { + return nil, errors.New("volatile-storage-unavailable") + } + id, err := volatileIdentity(path) + if err != nil { + os.RemoveAll(path) + return nil, err + } + cmd := exec.Command("/proc/self/exe", "--cleanup-storage", path, fmt.Sprint(id.dev), fmt.Sprint(id.ino), fmt.Sprint(id.uid), strconv.Itoa(os.Getpid())) + cmd.Env = []string{"PATH=/usr/bin:/bin", "LANG=C"} + cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true} + writer, err := cmd.StdinPipe() + if err != nil { + removeVolatile(path, id) + return nil, errors.New("cleanup-watchdog-unavailable") + } + reader, err := cmd.StdoutPipe() + if err != nil { + writer.Close() + removeVolatile(path, id) + return nil, errors.New("cleanup-watchdog-unavailable") + } + if err = cmd.Start(); err != nil { + writer.Close() + reader.Close() + removeVolatile(path, id) + return nil, errors.New("cleanup-watchdog-unavailable") + } + v := &volatileStorage{path, id, writer, cmd, make(chan error, 1)} + go func() { v.done <- cmd.Wait() }() + ready := make(chan bool, 1) + go func() { + var b [1]byte + _, e := io.ReadFull(reader, b[:]) + reader.Close() + ready <- e == nil && b[0] == 'R' + }() + select { + case ok := <-ready: + if ok && v.live() { + return v, nil + } + case <-time.After(5 * time.Second): + } + v.close() + return nil, errors.New("cleanup-watchdog-unavailable") +} + +func (v *volatileStorage) live() bool { + select { + case err := <-v.done: + v.done <- err + return false + default: + return true + } +} + +func (v *volatileStorage) close() error { + err := removeVolatile(v.path, v.id) + v.pipe.Close() + select { + case outcome := <-v.done: + if outcome != nil && err == nil { + err = errors.New("cleanup-watchdog-failed") + } + case <-time.After(5 * time.Second): + v.command.Process.Kill() + <-v.done + if err == nil { + err = errors.New("cleanup-watchdog-failed") + } + } + return err +} + +// Source-reading/writing children cannot survive loss of their exact parent. +func runChild(cmd *exec.Cmd) error { + runtime.LockOSThread() + defer runtime.UnlockOSThread() + cmd.SysProcAttr = &syscall.SysProcAttr{Pdeathsig: syscall.SIGKILL} + return cmd.Run() +} + +func internalProcess(args []string) (int, bool) { + if len(args) == 0 { + return 0, false + } + switch args[0] { + case "--cleanup-storage": + return cleanupProcess(args[1:]), true + case "--batch-scanner": + return batchScannerProcess(args[1:]), true + } + return 0, false +} diff --git a/workplans/FORGE-WP-0010-receive-scanner-budget.md b/workplans/FORGE-WP-0010-receive-scanner-budget.md new file mode 100644 index 0000000..bd452d1 --- /dev/null +++ b/workplans/FORGE-WP-0010-receive-scanner-budget.md @@ -0,0 +1,90 @@ +--- +id: FORGE-WP-0010 +type: workplan +title: "Complete bounded receive scans without per-small-input startup overhead" +domain: financials +repo: railiance-forge +status: proposed +owner: codex +topic_slug: railiance-forge +created: "2026-10-18" +updated: "2026-10-18" +flavor: implementation +depends_on: [] +state_hub_workstream_id: "04d8ebb9-e9fe-58d8-bdc8-2aa985098c7a" +--- + +CUST-WP-0080 discovered an availability defect in the already enforced receive +control: the normal 141-object public Custodian source delta exhausts the +unchanged 90-second budget, with no secret finding. The unmodified owner source +reproduces the refusal in an isolated bare repository. This is a new correction, +not reopening finished FORGE-WP-0006 or promoting FORGE-WP-0008 residual coverage. + +Reuse the existing Repo Manager small-input profile: at most 64 inputs of at most +4096 bytes per directory scan, identical pinned Gitleaks/default rules and strict +report/diagnostic validation. Larger inputs retain the original stdin fragment +path. Preserve every traversal/ref/object/byte/time bound, no override or new +runtime service. Raw batch inputs require checked private tmpfs, independent +parent-bound cleanup and child termination on receiver loss. Source preparation +does not grant native installation, package/owner acceptance or production proof. + +## Reproduce and implement the bounded adapter correction + +```task +id: FORGE-WP-0010-T01 +status: done +priority: high +state_hub_task_id: "c61c0392-6c0a-5a36-bbd1-4ee6aeb2afea" +``` + +Retain the exact original failure and source delta. Implement the existing +small-input batching contract in the Linux receive adapter. Verify synthetic +positive/negative parity, real receive-pack quarantine, all prior boundaries, +limits, expired context, diagnostic/report refusal, normal cleanup, receiver +death and changed storage identity. Complete the exact original clean delta +within the same deadline. Failures remain evidence; source proof closes no +native acceptance gate. Independent package/owner review remains under T02. + +Source result: the exact previously refused 141-object/one-ref delta completes +in 52.53 seconds with all objects counted, under the unchanged 90-second budget. +The complete suite passes 49 leaf cases with zero skips; five additional boundary +cases bring the combined distinct leaf coverage to 54. These include 4096-byte +and larger-input detection, multi-ref-name detection, normal and receiver-loss +cleanup, scanner/report/diagnostic failures and changed directory identity. +Unavailable and ordinary-disk storage fail closed in separate actual helper +processes. The original low-entropy canary failure and misconfigured storage +fixture/setup failures are retained rather than counted as successful tests. +Source/proof: Custodian `docs/changes/netkingdom-target-2026-10-18/receive-scanner-batch-r1/`. + +## Prepare and approve the immutable native revision + +```task +id: FORGE-WP-0010-T02 +status: progress +priority: high +state_hub_task_id: "b49d34ef-e6e5-5674-b523-ebcf80e0dcca" +``` + +Obtain independent source/package review. Pin exact source/build tool provenance, executable digest, startup-guard and +hook composition. Capture read-only current state, no-op checks and exact +before/after/rollback. Retain permissions, scanner digest, hook ordering and +fail-closed behavior. Obtain the founder's explicit go for this live change +under the existing environment rule; previous activation is not approval for +this different executable. Preserve the original revision and every refinement +in the current supervised-action record. + +## Verify admitted native behavior and ordinary publication + +```task +id: FORGE-WP-0010-T03 +status: wait +priority: high +state_hub_task_id: "863c91e4-7c25-575f-b8b6-f98d0074ce5a" +``` + +Waiting on T02: exact executable/startup admission and founder-authorized native +activation. Verify actual hook clean/refusal/unavailable/cleanup behavior and +the legitimate ordinary source publication without splitting or seeding refs. +Reconcile owner task/source status and independent evidence. Existing remaining +coverage, private incident response and target-wide acceptance retain their +original owners; no residual implementation is inferred.