Read / Process large number of data using DirectorySearcher API

Question

Context

• Active Directory instance with 200k Users in an OU

• Potential to grow upto 1 M in our domain

Not an expert in this ground. I am trying to come up with a solution design for a scenario where I need to poll AD and

• Check PwdLastSet and LastLogonDate property

• Take 6 different decisions (strategies) based on their values per user (lock account, send email etc.)

Ideally, if it was database, I would have the option to

• Open a connection
• Read a small page worth of data
• Close connection
• Process them in memory via the application
• Repeat

(and thus leave it to the connection pool to juggle stuff and allow others to do their stuff).

I am really interested in knowing the best practice / approach in this case which is scaleable. I only need to fetch those 2 properties for all users (of course we have filters - e.g. remove inactive)

Personally, I was wondering if I should

• Use our custom scheduler service to run Powershell (or .NET), use DirectorySearcher, open a connection (ssl), read 100 / 1000 users at a time using paging and process them in memory. Connection to AD remain open

• Open connection to AD, get a dump of all users in a CSV (paged), close connection, write that to database for other tasks to process. But then this will have to be a nightly job with potential volume rights to the DB.

• Replicate those two properties in a Database table and keep them in sync whenever they change in AD via our application. Consume data from here.

and so on.

Suggestions?

Answer 1

Filter at source wherever possible. Return minimal property sets wherever possible.

You could leverage DirectorySynchronization, but I would say there is only value in doing so where you have a significant amount of client-side calculation to do. I use this to technique to manage photos in AD. I have a offline synchronized set which has the photo as a hash that can be used to ensure I only update where required, and when I update I only pull down changes from the directory since last execution.

For actions based on pwdLastSet or lastLogonTimeStamp I will always generate (LDAP) filters that allow me to ask AD for the smallest result set. I will always request the smallest number of attributes I actually need to work with.