Multithreading DirectoryEntry Searches slow to complete

Question

I have about 50,000 IDs I need to lookup in AD and so I setup 10 threads using TPL to handle the list concurrently. It takes over 5 hours to go through these IDs and get the needed info as it is below. Any idea why? Is this normal or am I slowing things down with my code?

I tested and it could take an ID anywhere from 3-4 seconds to sometimes 70 or 80 seconds to complete an iteration for any given ID.

Thanks

maxThreads = 10;
int hdCount = UserProfileDictionary.Count;
int completedCount = 0;
foreach (var intKey in UserProfileDictionary.Keys.ToList())
{
    String ID = intKey;

    MyTasks.Add(System.Threading.Tasks.Task.Factory.StartNew(() => GetADInfo(ID, ref UserProfileDictionary, supportedOUCNs), TaskCreationOptions.LongRunning));

    completedCount++;
    Console.Write("\rCompleted " + completedCount + " out of " + hdCount);

    lock (numActiveThreadLock)
        numActiveThreads++;

    bool continuethreads = false;
    while (continuethreads == false)
    {
        if (numActiveThreads < maxThreads)
            continuethreads = true;
        else
            System.Threading.Thread.Sleep(1000);
    }
}
Task.WaitAll(MyTasks.ToArray(), -1);
MyTasks.Clear();

protected static void GetADInfo(String ID, ref Dictionary<string, UserProfile> UserProfileDictionary, List<string> supportedOUCNs)
{
    using (DirectoryEntry entry = new DirectoryEntry("LDAP://DC=A,DC=B,DC=C"))
    {
        using (DirectorySearcher mySearcher = new DirectorySearcher(entry))
        {
            mySearcher.SearchScope = SearchScope.Subtree;
            mySearcher.CacheResults = false;
            mySearcher.PropertiesToLoad.AddRange(new string[] { "cn", "displayName", "canonicalName", "userAccountControl", "distinguishedName"});
            mySearcher.Filter = ("(&(samAccountType=805306368)(sAMAccountName=" + ID + "))");

            foreach (SearchResult result in mySearcher.FindAll())
            {
                String displayname = "";
                String acctstatus = "N/A";
                String acctLocation = "N/A";

                String strUserDN = result.Properties["distinguishedName"][0].ToString();

                try
                {
                    displayname = result.Properties["displayName"][0].ToString();
                }
                catch
                {
                    displayname = "N/A";
                }

                acctLocation = result.Properties["canonicalName"][0].ToString().Replace(@"/" + result.Properties["cn"][0].ToString(), "");

                int userAccountControl = Convert.ToInt32(result.Properties["userAccountControl"][0]);
                bool disabled = ((userAccountControl & 2) > 0);
                if (disabled == true)
                    acctstatus = "Disabled";
                else
                    acctstatus = "Enabled";

                String suptUser = "NOT SUPPORTED";
                foreach (String CN in supportedOUCNs)
                {
                    if (acctLocation.ToLower().Contains(CN.ToLower()) == true)
                    {
                        suptUser = "SUPPORTED";
                        break;
                    }
                }

                Dictionary<string, string> usermemberOfDictionary = new Dictionary<string, string>();
                List<ResourceInfo> resInfoList = new List<ResourceInfo>();

                entry.Path = "LDAP://" + strUserDN;
                entry.RefreshCache(new string[] { "msds-memberOfTransitive" });
                foreach (String strDN in entry.Properties["msds-memberOfTransitive"])
                {
                    usermemberOfDictionary.Add(strDN, "GROUP");
                }

                String userOU = strUserDN;
                String[] OUArray = userOU.Split(',');
                foreach (String OU in OUArray)
                {
                    userOU = userOU.Replace(OU + ",", "");
                    if (userOU != "DC=net")
                    {
                        usermemberOfDictionary.Add(userOU, "OU");
                    }
                }

                foreach (KeyValuePair<string, string> DNEntry in usermemberOfDictionary)
                {
                    String strObject = "";
                    entry.Path = "LDAP://" + DNEntry.Key;
                    entry.RefreshCache(new string[] { "cn", "DriveMapping", "Priority" });
                    if (DNEntry.Value == "GROUP")
                        strObject = entry.Properties["cn"][0].ToString();
                    else
                        strObject = DNEntry.Key;

                    try
                    {
                        if (entry.Properties["DriveMapping"].Count > 0)
                        {
                            String resPriority = "";
                            try
                            {
                                resPriority = entry.Properties["Priority"][0].ToString();
                            }
                            catch
                            {
                                resPriority = "N/A";
                            }

                            PropertyValueCollection driveResources = entry.Properties["DriveMapping"];
                            for (int DRindex = 0; DRindex < driveResources.Count; DRindex++)
                            {
                                if (driveResources[DRindex].ToString().ToLower().Contains("<username>") == true)
                                {
                                    String[] driveResourceArray = driveResources[DRindex].ToString().Split(',');
                                    String resLetter = driveResourceArray[0] + @":\";
                                    String resServer = driveResourceArray[1];
                                    String resShare = driveResourceArray[2];

                                    resInfoList.Add(new ResourceInfo
                                    {
                                        resObject = strObject,
                                        resLetter = resLetter,
                                        resServer = resServer,
                                        resShare = resShare,
                                        resPriority = resPriority,
                                        resTargetLink = @"\\" + resServer + @"\" + resShare.ToLower().Replace("<username>", ID.ToLower()),
                                        resServerSupportStatus = "NOT SUPPORTED",
                                        resServerStatus = "OFFLINE",
                                        resShareStatus = "NOT ACTIVE"
                                    });
                                }
                            }
                        }
                    }
                    catch (Exception e)
                    {
                        Console.WriteLine(e.Message);
                    }
                }

                lock(UserProfileDictionaryLock)
                {
                    UserProfile userProfile = UserProfileDictionary[ID.ToLower()];
                    userProfile.displayname = displayname;
                    userProfile.acctstatus = acctstatus;
                    userProfile.acctLocation = acctLocation;
                    userProfile.resources = resInfoList;
                    userProfile.userSupportStatus = suptUser;
                    UserProfileDictionary[ID.ToLower()] = userProfile;
                }
            }
            mySearcher.FindAll().Dispose();
        }
    }

    lock (numActiveThreadLock)
    {
        numActiveThreads--;
    }
}

Answer 1

The most glaring issue to me is this:

mySearcher.FindAll().Dispose();

Yes, you should Dispose of the SearchResultCollection, but you have to dispose the one you created for the loop. Calling FindAll() a second time just repeats the search. Then you're just discarding the results, and still leaving your previous SearchResultCollection undisposed.

You should use something more like this:

using (var results = mySearcher.FindAll()) {
    foreach (SearchResult result in results) {

    }
}

Making that change should speed things up since it removes an unnecessary call out to AD for each account.

I found your reuse of entry a bit odd, but I guess it works :)

Is there a reason you are not including msds-memberOfTransitive in the PropertiesToLoad? That would save another call out to AD.

mySearcher.PropertiesToLoad.AddRange(new string[] { "cn", "displayName", "canonicalName", "userAccountControl", "distinguishedName", "msds-memberOfTransitive"});
...
//These lines no longer needed
//entry.Path = "LDAP://" + strUserDN;
//entry.RefreshCache(new string[] { "msds-memberOfTransitive" });

foreach (String strDN in result.Properties["msds-memberOfTransitive"]) {
    ...
}

I see that's a new attribute in Windows Server 2012. I don't have access to a domain that runs on 2012, so I can't test to make sure that works, so maybe it won't. But I know it will return other constructed attributes (like canonicalName), so it should work.

Edit: Also - I don't know if this will help speed, but it will help simplify your code - instead of using lock(UserProfileDictionaryLock) you can just make UserProfileDictionary a ConcurrentDictionary, which is designed to be thread-safe.

Edit 2: If it's worth the effort, you can actually search for multiple accounts in one query:

(&(samAccountType=805306368)(|(sAMAccountName=username1)(sAMAccountName=username2)(sAMAccountName=username3)))

The maximum length of the LDAP query is apparently very big, but you could do them in batches of 50 or even 100 (or more?).

So you could pass a list of usernames to your GetADInfo method instead of just a single one. That could really cut down on the number connections you're making to AD.