the-custodian/docs/changes/netkingdom-target-2026-10-16/scanner-batch-r1

TSA15 scanner batching candidate, revision 1

This is an isolated review candidate against repo-manager commit 97ea2489257de08911cc5da1e1b84d8e15780dc3. The canonical owner checkout is unchanged. manifest.json fixes the base, candidate and patch hashes; candidate.patch applies without changing repository control ownership. This package does not admit or install the candidate and does not change hook, receive-path, CI or independent source-control coverage.

Behavior and attribution

One metadata query checks every unique staged blob, then bounded cat-file --batch reads replace per-object size/content subprocess pairs. Every returned object is checked for exact framing, type, size, requested OID and recomputed Git blob hash. Index snapshots still bracket the scan. Existing stage/submodule, byte/count limits, alternate-index support, no-replace behavior and content-silent refusal remain in force.

Blobs of at most 4096 bytes use one directory invocation per batch of at most 64 objects. Each object remains a separate regular file with a neutral OID filename; content is never concatenated and original filenames cannot activate ignore patterns. Larger blobs retain the original individual stdin route. At the small-file cutoff, Python's stdin writer uses one atomic PIPE_BUF write and the scanner gets one fragment in both routes. This avoids changing larger inputs' pipe-dependent fragmentation. The fixed template emits only neutral file paths; unknown paths, missing/oversized reports, exit/report disagreement, warnings, timeouts and failures all refuse. The pinned scanner's single fixed successful WRN leaks found: N summary is the only accepted diagnostic. No finding content is emitted or retained in reports.

Both routes use Gitleaks 8.30.1 and the original exact SHA256, default-rule extend config, explicit empty ignore file, --ignore-gitleaks-allow, full redaction and an explicit environment. The executable is now copied into a sealed memfd while hashing and executed from that immutable snapshot. Path replacement or in-place modification of the original after verification cannot substitute executable bytes.

Storage and process lifecycle

All new source-bearing temporary files are restricted to a private 0700 directory on fixed /dev/shm, verified with fstatfs against TMPFS_MAGIC. There is no disk-backed fallback. Scanner reports and diagnostics use the same volatile directory. Tests use generated disposable fixtures only.

Before reading any blob content, a separate isolated Python watchdog registers an exact parent pidfd and confirms readiness. Normal cleanup happens after all synchronous source children exit. Hard parent death also triggers guarded cleanup; every Git/scanner child installs Linux parent-death SIGKILL and closes the fork/registration race. At the exceptional 180-second watchdog deadline, the watchdog sends TERM to the exact parent pidfd, allows five seconds, sends KILL if needed and waits for proven death before cleanup. If kernel quiescence stalls, the cleanup observer retains its duty until death is established. It never signals a shell, user process group or tool harness. The shortened deadline test targets only a newly created disposable Python process.

Deletion verifies directory device, inode, owner and 0700 mode through a directory fd, then uses symlink-safe rmtree. Normal cleanup verifies watcher completion; watchdog loss/failure refuses. This is process and memory lifecycle handling, not a new service or authority boundary.

Bounds

• Unique staged objects: 500; individual source blob: 8 MiB; total source: 64 MiB.
• Git content batches: at most 64 objects and 8 MiB, plus at most 128 framing bytes per object. Git output is capped before capture with RLIMIT_FSIZE.
• Materialized small-file payload: at most 64 × 4096 bytes per batch. Large blobs remain individual stdin inputs. An anonymous content-batch file can hold 8 MiB plus framing, and in-memory batch contents are bounded by 8 MiB.
• Each report and diagnostic file: at most 1 MiB. Exceeding a limit refuses.
• Child virtual address space: 8 GiB (a virtual address bound, not a resident memory guarantee); CPU: 20 seconds per Git/scanner child; scanner Go parallelism fixed at two. The pinned WASM regex engine reserves a 4 GiB virtual region.
• Each subprocess has a 20-second caller timeout; scanner internal timeout is 15 seconds. Clean/blocked acceptance requires the entire scan to finish within 120 seconds. Cleanup uses five-second waits; the separate exceptional watchdog deadline is 180 seconds plus five seconds TERM grace and proven-death wait.
• The scanner snapshot is capped at 64 MiB and sealed before execution. Existing index-enumeration output bounds are unchanged; staged paths are not file content.

Validation evidence

run_validation.sh is the exact generated-fixture driver. Its three stages run 42 tests, 13 old/new exact-summary parity cases, then the same five benchmark fixture families and repetition/percentile method as the retained baseline. The coordinator executed the exact reviewed runner successfully: 42 tests passed in 32.95 seconds, all 13 old/new exact-summary parity cases matched, all benchmark cases and missing/wrong scanner refusals passed, and generated fixtures were removed. The final source and pinned scanner hashes were unchanged. The hard-loss and shortened-watchdog-deadline probes affected only disposable Python processes. See test-results.txt, parity-results.json and scanner-candidate.json.

Final warm p50 results, using the same retained fixture families and seven warm samples per case:

Generated case	Earlier baseline	Final candidate	Comparison
1 distinct 1 KiB blob	702.358 ms	835.286 ms	18.9% slower
16 distinct 1 KiB blobs	12065.334 ms	884.239 ms	13.6× faster
64 distinct 1 KiB blobs	49171.849 ms	904.024 ms	54.4× faster
1 × 1 MiB blob	2548.887 ms	1902.956 ms	1.34× faster
Generated private key	854.193 ms	882.714 ms	3.3% slower

These are separate descriptive local fixture samples rather than a controlled interleaved trial or production tail estimate. They establish bounded local multi-object startup amortization, with explicit single-object regressions; there is no aggregate performance, remote receive, CI or whole-TSA15 claim.

scanner-candidate-pre-tmpfs.json is retained superseded evidence from the initial disk-temporary revision, not proof of the final storage/lifecycle design. Its p50 for 1/16/64 distinct 1 KiB blobs was 1.058/1.048/1.136 seconds, compared with the earlier baseline's 0.702/12.065/49.172 seconds. The initial one-blob case regressed; multi-object amortization improved. Those samples are descriptive local fixtures, not workload or production tail estimates.

Failed preparation experiments are preserved here in prose: a 4 GiB virtual address limit caused the pinned WASM allocator to refuse startup; the candidate therefore uses 8 GiB. The pinned CLI required explicit --report-format template despite help claiming --report-template implied it. Successful blocked scans emit a warning-level fixed summary, so blanket rejection of all nonempty stderr was refined to accept only that exact summary. An initial test discovery run resolved the installed package instead of the isolated namespace copy; it was excluded and rerun after copying __init__.py and printing the candidate import. The subsequent 17 original and 34 expanded tests passed on the pre-tmpfs candidate; none of those runs establishes the final revision's lifecycle result.

Pinned upstream behavior reviewed

Gitleaks source at tag v8.30.1:

• stdin command
• directory command
• shared file fragment reader
• directory traversal and skip diagnostics
• default config and path allowlists
• template reporter

Both commands use the same file-fragment scanner. Directory mode adds path-based allowlists and warning/error skip behavior; neutral relative OID paths, restricted small-file batching and refusal on unexpected diagnostics address those additional conditions. The generated parity matrix tests representative binary, encoded, path-collision and fragment-boundary cases; it is not an exhaustive proof over all possible scanner inputs or a promise that the scanner detects every secret.

Owner refinement and publication preparation

owner-workplan.patch proposes RMGR-WP-0021-T05 without changing the existing done tasks or claiming source publication. RMGR-WP-0021.proposed.md is the exact reviewable owner-file result. runtime-routing-readonly.json records the actual hook and CLI source paths; no hook was executed against a real user index. publication-plan.md distinguishes source publication from configured-hook and CLI acceptance. The globally advertised ~/.local/bin/rmgr executable was absent; the verified repo-local .venv/bin/rmgr imports the canonical source.