Why a weekly file scan stalled over the VPN, and how the fix brought it to 24 minutes

A root cause analysis of a compliance scan that ran at about 178 items per second on the office LAN and about 7 over the VPN, and the four changes that made it reliable.

FileGate audits every item on the agency file share against the naming and records policy once a week, about 67,000 files and folders. The scan worked in testing, then stalled in production.

Measure before fixing

The first step was a benchmark, not a code change. The same scan ran against the same share from two places: on the office LAN and over the WireGuard VPN. On the LAN it processed about 178 items per second. Over the VPN it processed about 7, roughly 25 times slower. At that rate the directory walk alone takes hours, and a dropped connection could cost the entire run.

The code wasn't slow; the network path was. A scan like this is dominated by small metadata operations: list a folder, read a name, check a timestamp. Each one is a network round trip, so extra latency is multiplied by every item in the tree. Bandwidth barely matters when no single request moves much data.

What changed

A network gate. The scan now checks that it is on the office network before it starts, and refuses to run over the VPN instead of limping along.
Incremental commits. Results are written in batches as the scan goes, so an interruption loses minutes of work instead of the whole run.
Explicit failure states. A run ends as completed, failed or interrupted, with the reason recorded. "Still running" is no longer a state that can last forever.
The right host. The weekly job moved to an always-on machine on the office network, so nobody's laptop has to stay awake and connected.

Result

The production scan on September 22, 2026 processed 67,240 items in 24 minutes with 0 errors. The benchmark also changed a broader decision: bulk file workloads now run on-network by default, and the VPN is for people, not batch jobs.

Takeaways

Benchmark the same workload on both paths before touching code. The ratio tells you whether you have a software problem or a placement problem.
Metadata-heavy jobs are latency-bound. Move the job to the data rather than the data to the job.
Long-running jobs need checkpoints and a defined way to fail.

The policy rule engine is available as a standalone open-source tool: naming-policy-validator.