PowerShell: executing C# Class (FindFiles) as an Add-Type definition on a remote system using Invoke-Command is extremely slow

Question

I want to find all Package.json files on a remote system using PowerShell. In order to optimize this, I found a C# class on the internet adjusted it and now would like to use it.

This code runs really fast whenever I run it locally on my machine, however when I use the code via Invoke-Command, it's super slow on the remote machine (which is a VM on VmWare).

Here is the code:

 
 $Result = Invoke-Command -ComputerName "RemoteSystem" -ScriptBlock {
 Add-Type -TypeDefinition @"
            using System;
            using System.IO;
            using System.Linq;
            using System.Collections.Concurrent;
            using System.Runtime.InteropServices;
            using System.Text.RegularExpressions;
            
            public class FileSearch {
                public struct WIN32_FIND_DATA {
                    public uint dwFileAttributes;
                    public System.Runtime.InteropServices.ComTypes.FILETIME ftCreationTime;
                    public System.Runtime.InteropServices.ComTypes.FILETIME ftLastAccessTime;
                    public System.Runtime.InteropServices.ComTypes.FILETIME ftLastWriteTime;
                    public uint nFileSizeHigh;
                    public uint nFileSizeLow;
                    public uint dwReserved0;
                    public uint dwReserved1;
                    [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 260)]
                    public string cFileName;
                    [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 14)]
                    public string cAlternateFileName;
                }

                [DllImport("kernel32.dll", SetLastError = true, CharSet = CharSet.Ansi)]
                static extern IntPtr FindFirstFile(string lpFileName, out WIN32_FIND_DATA lpFindFileData);

                [DllImport("kernel32.dll", SetLastError = true, CharSet = CharSet.Ansi)]
                static extern bool FindNextFile(IntPtr hFindFile, out WIN32_FIND_DATA lpFindFileData);

                [DllImport("kernel32.dll", SetLastError = true, CharSet = CharSet.Ansi)]
                static extern bool FindClose(IntPtr hFindFile);

                static IntPtr INVALID_HANDLE_VALUE = new IntPtr(-1);
                static BlockingCollection<string> fileList {get;set;}

                public static BlockingCollection<string> GetFiles(string searchDir, string searchFile) {
                    bool isPattern = false;
                    if (searchFile.Contains(@"?") | searchFile.Contains(@"*")) {
                        searchFile = @"^" + searchFile.Replace(@".",@"\.").Replace(@"*",@".*").Replace(@"?",@".") + @"$";
                        isPattern = true;
                    }
                    fileList = new BlockingCollection<string>();
                    SearchDirectory(searchDir, searchFile, isPattern);
                    return fileList;
                }

                private static void SearchDirectory(string path, string searchFile, bool isPattern) {
                    IntPtr handle = INVALID_HANDLE_VALUE;
                    WIN32_FIND_DATA fileData;

                    path = path.EndsWith(@"\") ? path : path + @"\";
                    handle = FindFirstFile(path + @"*", out fileData);
                    if (handle != INVALID_HANDLE_VALUE) {
                        FindNextFile(handle, out fileData); // Skip "." entry
                        while (FindNextFile(handle, out fileData)) {
                            if ((fileData.dwFileAttributes & 0x10) > 0) { // Directory
                                string fullPath = path + fileData.cFileName;
                                SearchDirectory(fullPath, searchFile, isPattern);
                            } else { // File
                                if (isPattern) {
                                    if (Regex.IsMatch(fileData.cFileName, searchFile, RegexOptions.IgnoreCase)) {
                                        string fullPath = path + fileData.cFileName;
                                        fileList.TryAdd(fullPath);
                                    }
                                } else {
                                    if (fileData.cFileName.Equals(searchFile, StringComparison.OrdinalIgnoreCase)) {
                                        string fullPath = path + fileData.cFileName;
                                        fileList.TryAdd(fullPath);
                                    }
                                }
                            }
                        }
                        FindClose(handle);
                    }
                }
            }
"@ -IgnoreWarnings

$searchFile = "package.json"
$lookupPath = "C:"
$Files      = @()
$Files      = @([FileSearch]::GetFiles($lookupPath,$searchFile))

return $Files


}

$Result

I rewrote the code with Get-Childitem which works fine on the Remote Machine and Locally. I tried using the Script via PsExec (copying the Script to the Remote Machine and running it there), Invoke-command, Interactive Powershell Session but the results are the same. The only things left would be to disable the Virus Protection but that is not really an Option. You can see when i run the Code locally that there is 80% Disk Usage and 30% CPU Usage but with invoke command just 30% CPU Usage and 0-1% Disk Usage. So the runtime locally (When i sign into the Remote Machine via RDP) is around 10Seconds and when ran from my machine with invoke-command on the Remote Machine is around 10-20min. Notable is also that when run locally the System Process does the Disk searching (has the high DiskUsage) and when run Remotly System doesnt even seem to be involved.

What i would like to know is just why that is. GCI gets the Job done but i dont care about that. I really really would like to know why this code is so very slow ONLY on the Remote System.

Answer 1

Following comments and advise from Charlieface, .NET already performs a great job enumerating files with its EnumerateFiles method however, since your remote host is running PowerShell 5.1, there is a caveat, you must enumerate the IEnumerable (returned by EnumerateFiles) manually otherwise the enumeration stops as soon as there is an inaccessible file / folder. You can see an example in here: Powershell getting list from large directories fast.

Applied to your use case, you can do:

$Result = Invoke-Command -ComputerName RemoteSystem -ScriptBlock {
    try {
        $ienumerable = [System.IO.Directory]::EnumerateFiles(
            'C:\', 'package.json', [System.IO.SearchOption]::AllDirectories)

        $enumerator = $ienumerable.GetEnumerator()
        while ($true) {
            try {
                if (-not $enumerator.MoveNext()) {
                    break
                }

                $enumerator.Current
            }
            catch {
                # leave empty to ignore any error
            }
        }
    }
    finally {
        if ($enumerator) { $enumerator.Dispose() }
    }
}

---

For your current low level implementation, there are a few issues, one of them also named by Charlieface, using a BlockingCollection<> doesn't make sense there. You could also use a Queue<> instead of recursion.

I'd personally use this approach, it uses a compiled WildcardPattern, though not needed for your current use case since you want to match the file name exactly, instead of you manually creating the regex pattern.

using System;
using System.IO;
using System.Runtime.InteropServices;
using System.Collections.Generic;
using System.Management.Automation;

[StructLayout(LayoutKind.Sequential, CharSet = CharSet.Unicode)]
internal struct WIN32_FIND_DATAW
{
    internal uint dwFileAttributes;
    internal FILETIME ftCreationTime;
    internal FILETIME ftLastAccessTime;
    internal FILETIME ftLastWriteTime;
    internal uint nFileSizeHigh;
    internal uint nFileSizeLow;
    internal uint dwReserved0;
    internal uint dwReserved1;

    [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 260)]
    internal string cFileName;

    [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 14)]
    internal string cAlternateFileName;

    internal uint dwFileType;      // Obsolete. Do not use.
    internal uint dwCreatorType;   // Obsolete. Do not use.
    internal ushort wFinderFlags;  // Obsolete. Do not use.
}

[StructLayout(LayoutKind.Sequential)]
internal struct FILETIME
{
    internal uint dwLowDateTime;
    internal uint dwHighDateTime;
}

internal static class Native
{
    internal static readonly IntPtr INVALID_HANDLE_VALUE = new IntPtr(-1);
    internal const uint FILE_ATTRIBUTE_DIRECTORY = 0x10;

    [DllImport("kernel32.dll", SetLastError = true, CharSet = CharSet.Unicode)]
    internal static extern IntPtr FindFirstFileW(
        string lpFileName, out WIN32_FIND_DATAW lpFindFileData);

    [DllImport("kernel32.dll", SetLastError = true, CharSet = CharSet.Unicode)]
    internal static extern bool FindNextFileW(
        IntPtr hFindFile, out WIN32_FIND_DATAW lpFindFileData);

    [DllImport("kernel32.dll", SetLastError = true)]
    internal static extern bool FindClose(IntPtr hFindFile);
}

public static class FileSearch
{
    private static List<string> s_result;
    private static WildcardPattern s_pattern;
    private static Queue<string> s_queue;
    private const WildcardOptions _options = WildcardOptions.Compiled
        | WildcardOptions.CultureInvariant
        | WildcardOptions.IgnoreCase;

    public static string[] SearchDirectory(string path, string searchFile)
    {
        WIN32_FIND_DATAW fileData;
        IntPtr handle;

        s_result = new List<string>();
        s_queue = new Queue<string>();
        s_pattern = WildcardPattern.ContainsWildcardCharacters(searchFile)
            ? new WildcardPattern(searchFile, _options) : null;

        s_queue.Enqueue(path);
        while (s_queue.Count > 0)
        {
            string current = s_queue.Dequeue();
            if (!FindFirstFile(current, out handle, out fileData))
            {
                continue;
            }

            try
            {
                Walk(searchFile, current, fileData, handle);
            }
            finally
            {
                Native.FindClose(handle);
            }
        }

        return s_result.ToArray();
    }

    private static void Walk(
        string searchFile,
        string current,
        WIN32_FIND_DATAW fileData,
        IntPtr handle)
    {
        do
        {
            // Skip "." and ".." entries
            if (fileData.cFileName == "." || fileData.cFileName == "..")
            {
                continue;
            }

            string fullName = Path.Combine(current, fileData.cFileName);
            if ((fileData.dwFileAttributes & Native.FILE_ATTRIBUTE_DIRECTORY) != 0)
            {
                s_queue.Enqueue(fullName);
                continue;
            }

            bool isMatch = s_pattern == null
                ? fileData.cFileName.Equals(searchFile, StringComparison.InvariantCultureIgnoreCase)
                : s_pattern.IsMatch(fileData.cFileName);

            if (isMatch)
            {
                s_result.Add(fullName);
            }
        }
        while (Native.FindNextFileW(handle, out fileData));
    }

    private static bool FindFirstFile(string path, out IntPtr handle, out WIN32_FIND_DATAW data)
    {
        handle = Native.FindFirstFileW(Path.Combine(path, "*"), out data);
        return handle != Native.INVALID_HANDLE_VALUE;
    }
}