How to do a paged search on an Ldap server with lots of users?

Question

I need to create a LDAP client that works on linux using .NET Core. I searched trought the internet and the only library that support .Net Standard is Novell.Directory.Ldap (open source, iei - https://github.com/dsbenghe/Novell.Directory.Ldap.NETStandard). Directory Service library does not have support in .Net Core for Linux, only on Windows.

I looked to the documentation and succeded to create a basic Ldap client application.

Now the problem: I need to synchronize a lot of users (10.000, 200.000 users) and by default my ldap server has a maximum size page 1000 (and I don't want to change it). I used a VirtualListControl in order to create pages and it works ok for 10k users.

For 200k user it crashes with error 53 - unwilling to perform on the LdapSortControl response. The Novell library need a LdapSortControl in order to perform paging operation (for index) and I think that my ldap is not able to sort 200k. The code that I used:

        int startIndex = 1;
        int contentCount = 0;
        int afterIndex = 10;
        int count = 0;
        do
        {
            LdapVirtualListControl ctrl = new LdapVirtualListControl(startIndex, 0, afterIndex, contentCount);
            LdapSortKey[] keys = new LdapSortKey[1];
            keys[0] = new LdapSortKey("sn");
            LdapSortControl sort = new LdapSortControl(keys, true);

            LdapSearchConstraints constraints = _ldapConnection.SearchConstraints;
            constraints.setControls(new LdapControl[] { ctrl, sort});

            _ldapConnection.Constraints = constraints;
            LdapSearchResults lsc = _ldapConnection.Search(searchBase, searchScope, searchFilter, attributes, typesOnly, cons);
            while (lsc.HasMore())
            {
                try
                {
                    LdapEntry nextEntry = lsc.Next();
                    Console.WriteLine( nextEntry.DN);
                }
                catch (LdapException e)
                {
                    Console.WriteLine($"Error: {e.LdapErrorMessage}");
                    //Exception is thrown, go for next entry
                    continue;
                }

            }

            LdapControl[] controls = lsc.ResponseControls;
            if (controls == null)
            {
                Console.Out.WriteLine("No controls returned");
            }
            else
            {
                foreach (LdapControl control in controls)
                {
                    if (control.ID == "2.16.840.1.113730.3.4.10")
                    {
                        LdapVirtualListResponse response = new LdapVirtualListResponse(control.ID, control.Critical, control.getValue());
                        startIndex += afterIndex + 1;
                        contentCount = response.ContentCount;
                        count += afterIndex;
                    }
                }
            }
            Console.WriteLine(i);

        } while (count <= contentCount);

The documentation is small and does not have enough information and I have no idea how to use Novell library for paging in a better way. Is anybody here that use Novell Ldap library and have any experience with paging and also can help me? I'm disperate

Thank you

Answer 1

To make a paged query with Novell.Directory.Ldap you have to use LdapVirtualListControl as "Request" Control.

LdapVirtualListControl respects the arguments of the Ldap Sorting Request Control: VLV (Virtual List View) that are:

before:after:index:content_count

where "before" is the number of items that you want to be returned BEFORE the index, "after" the number of items that you want to be returned AFTER the index and "content_count" is the expected total count of items in the server. In case you do not know it, you have to use 0 as value.

If you want to return the first 5 elements via "ldapsearch" cli, you have to use: '0:4:1:0' and '0:4:5:0' for the subsequent five ones.

The LdapVirtualListControl owns a constructor with the same arguments but with a different order:

LdapVirtualListControl(int startIndex, int beforeCount, int afterCount, int contentCount)

Personally, I use this function to set the arguments correctly:

public static LdapSearchConstraints AddPagination(this LdapSearchConstraints constraints, int page,
            int pageSize)
{
    int startIndex = (page - 1) * pageSize;
    startIndex++;
    int beforeCount = 0;
    int afterCount = pageSize - 1;
    int contentCount = 0; //0 means that i don't know the total count

    var lvlc = new LdapVirtualListControl(startIndex, beforeCount, afterCount, contentCount);
    constraints.setControls(lvlc);
    return constraints;
}

After that, another issue needs your attention: if you ask for a set of data, which are positioned after the end of the data-set, you will receive back the first items of the dataset.

Explanation:

Example of data present in ldap:
| 1 | 2 | 3 | 4 |
if we ask for the set
________| 3 | 4 | 5 | <- 5 does not exists
Ldap returns:
________| 3 | 4 | 1 | <- it starts over from the beginning

To fix this issue, I remove the exceeded elements before return:

var lastIndex = (page * pageSize);
if (lastIndex > result.Total)
{
    var itemsToReturn = (int) (result.Total - (lastIndex - pageSize));
    if (itemsToReturn < 1)
    {
        items = new List<LdapQueryItem>();
    }
    else
    {
        items = items.Take(itemsToReturn).ToList();
    }
}   

Finally, the function to get the totals (to execute AFTER the searchResults.HasMore() method)

protected int? GetTotal(LdapSearchResults searchResult)
{
    if (searchResult == null) {
       throw new ArgumentNullException(nameof(searchResult));
    }
    if (searchResult.ResponseControls != null && searchResult.ResponseControls.Any())
    {
        foreach (LdapControl control in searchResult.ResponseControls)
        {
            if (control.ID == "2.16.840.1.113730.3.4.10") // the id of the response control
            {
                LdapVirtualListResponse response =
                    new LdapVirtualListResponse(control.ID, control.Critical, control.getValue());
                return response.ContentCount;
            }
        }
    }

    return null;
}

You can get further insights and info on this book: Understanding and Deploying LDAP Directory Services