Openldap sync issue in multi-master replication

Question

We are facing a synchronise issue in openldap setup we have. Following are the details:

• We have 2 openldap servers configed with dynamic configurations and those are setup with multi-master replication.
• There is a loadbalancer setup to direct requests to the ldap servers.
• Used version is : openldap 2.4.56.

Following is the ldap replication configs that's in the <openldap_location>/etc/openldap/slapd.d/cn=config/olcDatabase={1}mdb.ldif file.

olcSyncrepl: {0}rid=001 provider=ldaps://ldaptest1.com binddn="cn
 =admin,dc=test,dc=com" bindmethod=simple credentials="test" search
 base="dc=test,dc=com" type=refreshAndPersist retry="5 5 300 +" timeout=
 1
olcSyncrepl: {1}rid=002 provider=ldaps://ldaptest2.com binddn="cn
 =admin,dc=test,dc=com" bindmethod=simple credentials="test" search
 base="dc=test,dc=com" type=refreshAndPersist retry="5 5 300 +" timeout=
 1
olcMirrorMode: TRUE

How we're testing:

We have a script to create user (through ldapadd command) and assign him to role-A, role-B and role-c (through ldapmodify command). We are doing this for bulk number of users (1000 users) and check the memberOf and member attributes to verify the user is correctly assign to the role (hasRole) and to verify user exists in the role (hasMember).

The issue we are facing is:

In this test we are getting results with inconsistencies. For an example, the script output is like this.

User 1:
 hasMemberA=1, hasRoleA=1 | hasMemberB=1, hasRoleB=1 | hasMemberC=1, hasRoleC=1
...
User 22:
 hasMemberA=1, hasRoleA=0 | hasMemberB=1, hasRoleB=1 | hasMemberC=1, hasRoleC=1
...
User 56:
 hasMemberA=1, hasRoleA=1 | hasMemberB=1, hasRoleB=1 | hasMemberC=0, hasRoleC=0

As you can see for some users (user 1) the expected outcome has given correctly. But intermittently it has not worked as expected. For example in User 22, the memberOf attribute is missing for role-A. When I check the log in this kind of failing scenario, I could see in openldap-1 server there's a ' delete memberOf' log immediately after 'add memberOf' for the role A. But in openldap-2 server there's no relevant log for 'add memberOf' for role A. In this case when we manually browse the ldap server and check, there also the user 22 is not assigned to the role-A.

In some cases we found that when browse the ldap server there it has the role assigned correctly. In such cases, I could see in one server 'add memberOf' log exists and the other server 'add memberOf', 'delete memberOf' and again 'add memberOf' log exists for that particular role. It seems that after deleting the role on one server, since the role still exists on the other server, it gets added back again while synchronising.

As a summary we could see the sync not working well in our scenario in some cases. So that we are getting unexpected output.

We assume that because both servers act as masters, and due to the load balancing and since we're making changes simultaneously, this issue arises.

Appreciate your support on resolving this issue.

Also what's the good approach for openldap replication, is it multi-master or master-slave?