CODEWITHSUNDEEP
phplaravelldapldap-queryadldap
Fawwash Khan
Fawwash Khan

Reputation: 55

How to Fetch All Users from AD LDAP in Laravel Without Timeout or Memory Issues?

I’m currently working on a Laravel 10 application that integrates with Active Directory (AD) using the AdLdap2 package and PHP's LDAP functions (PHP 8). However, I’m facing challenges when trying to fetch all users (over 20,000) from the LDAP server.

Here are the two methods I've tried: Method 1: Using AdLdap2 Package

private function handleAdLdapUsersEx($provider): void
{
    try {
        $pageSize = 200;

        if ($this->searchBase->filter) {
            $provider->where($this->searchBase->filter);
        }

        $pagedUsers = $provider->select('distinguishedname', 'givenname', 'name', 'objectguid', 'samaccountname', 'userprincipalname', 'mail')
            ->whereEnabled()
            ->limit($pageSize)
            ->paginate($pageSize, $this->currentPage);
        
        dump($pagedUsers->count());
        
        if ($pagedUsers->count() > 0) {
            collect($pagedUsers->getResults())->chunk(100)->each(function ($chunkedUsers) {
                $this->handleBulk($chunkedUsers);
                unset($chunkedUsers);
                gc_collect_cycles();
            });

            if ($pagedUsers->count() === $pageSize) {
                ImportUsersJob::dispatch($this->searchBase, $this->ldapConnector, $this->ldapConfig, $this->currentPage + 1);
            }
        }
    } catch (\Exception $e) {
        dd($e->getMessage());
    }
}

In this method, I set a limit for pagination, even with limit and pagination I am getting all the records in one page, but I'm still experiencing timeouts. Setting public $timeout = 600; works, but I’d like to avoid hardcoding a timeout.

Method 2: Using PHP LDAP Functions

private function handleAdLdapUsers($provider, $ldapConnector): void
{
    try {
        $usersFetched = 0;
        $lastUserEntry = null;

        $ldapConnection = ldap_connect($provider->getConnection()->getHost());
        ldap_set_option($ldapConnection, LDAP_OPT_PROTOCOL_VERSION, 3);
        ldap_bind($ldapConnection, $ldapConnector->getUsernames(), $ldapConnector->getPassword());

        $filter = !empty($this->searchBase->filter) ? $this->searchBase->filter : "(objectClass=*)";

        $result = @ldap_search($ldapConnection, $provider->getDn(), $filter, [], 0, 1);
        $firstEntry = ldap_first_entry($ldapConnection, $result);

        while ($firstEntry) {
            $attributes = ldap_get_attributes($ldapConnection, $firstEntry);

            $users = $provider->select('distinguishedname', 'givenname', 'name', 'objectguid', 'samaccountname', 'userprincipalname', 'mail', 'usncreated')
                ->whereEnabled()
                ->get($attributes);

            if ($users->count() > 0) {
                $this->handleBulk($users);
                $usersFetched = $users->count();
            } else {
                break;
            }

            $lastUserEntry = ldap_next_entry($ldapConnection, $firstEntry);
            $firstEntry = $lastUserEntry;
        }

        ldap_unbind($ldapConnection);
    } catch (\Exception $e) {
        dd($e->getMessage());
    }
}

This method returns a "Sizelimit exceeded" warning and only fetches 1000 records. I suppressed the warning with @ldap_search, but I still need a way to fetch all users (potentially over 50,000) without hitting size limits or running into memory issues.

  1. How can I modify my queries or pagination to handle fetching all users efficiently?
  2. Are there best practices for dealing with large datasets in Laravel when querying LDAP?
  3. Is there a way to configure the LDAP server to allow fetching more than the default size limit?

Any help or guidance would be greatly appreciated!

Upvotes: 1

Views: 90

Answers (1)

grawity_u1686
grawity_u1686

Reputation: 16122

Do not suppress the warning. Suppressing it using @ doesn't help in any way with making the code fetch more results – it still gets incomplete data, except now it gets incomplete data quietly – and once the code is fixed, the warning will not be there anyway.

ldap_search() takes a $controls parameter, which is how you can request a paged search result. See LDAP Controls. The results, parsed by ldap_parse_result(), will have a corresponding control with a 'cookie' that needs to be passed to another ldap_search() to retrieve the next page.

$req_controls = [
    ["oid" => LDAP_CONTROL_PAGEDRESULTS, "value" => ["size" => 50"]],
];

Do not filter for (objectClass=*) if you specifically want users; filter for (objectCategory=user) (this is an AD-specific attribute) to reduce the amount of useless results.

ldap_search() also takes an $attributes parameter. Right now you're retrieving all attributes, then selecting just the few that you want, which is like doing a SELECT * FROM when all you want is two columns. Instead of doing so, pass the list of desired attributes to ldap_search() to reduce the amount of data transferred (even if not necessarily the amount of entries).

Upvotes: 0

Related Questions