OpenLDAP N-Way Multi-Master Replication

N-Way Multi Master Replication, supports writing to multiple nodes, and could be used for OpenLDAP servers spread widely, in example different countries.
For OpenLDAP it is really recommended to go with the latest version!
I asume, that the replication is setup before any data is present, so be careful with dataloss!
Backup before doing this!

Prerequisites

NTP Must be correctly configured, and servers must be in sync.
The servers must be able to lookup each server.
Create entries in host files, or use DNS! We will use these names later on!

/etc/hosts
10.0.2.1 ldap1.test.local
10.0.2.2 ldap2.test.local
10.0.2.3 ldap3.test.local

Make sure that /etc/default/slapd have entries matching, ldap1.test.local, ldap2.test.local, ldap3.test.local or you might get this error!
read_config: no serverID / URL match found. Check slapd -h arguments.

Master1

/etc/default/slapd
SLAPD_SERVICES="ldapi:// ldap://ldap1.test.local"

Master2

/etc/default/slapd
SLAPD_SERVICES="ldapi:// ldap://ldap2.test.local"

Master3

/etc/default/slapd
SLAPD_SERVICES="ldapi:// ldap://ldap3.test.local"

Configurating Replication

All below examples, can be used with "ldapmodify -Y EXTERNAL -H ldapi://". (CTRL-D, to save and exit).

Make sure that the syncprov module is loaded, should be done on both Master1,Master2 and Master3

dn: cn=module{0},cn=config
changetype: modify
add: olcModuleLoad
olcModuleLoad: syncprov

On Master1,Master2 and Master3, setup the config database for replication, and set a password if not already set!
Replace olcServerID for Master2 to 2, and Master3 to 3

dn: cn=config
changeType: modify
add: olcServerID
olcServerID: 1

On Master1, Master2 and Master3 add config pass, if not already set, so we can bind to the config.
Password in this example is "test", you can create a more decent one using slappasswd

dn: cn=config
changeType: modify

dn: olcDatabase={0}config,cn=config
add: olcRootPW
olcRootPW: {SSHA}SGq737yNactRCyMY70TDTQs6V1wzMRD6

Now test, that you can login with ldapmodify, asuming this was done on Master1.

ldapmodify -H ldap://ldap1.test.local -D "cn=admin,cn=config" -W

On all masters add config replication

dn: cn=config
changetype: modify
replace: olcServerID
olcServerID: 1 ldap://ldap1.test.local
olcServerID: 2 ldap://ldap3.test.local
olcServerID: 3 ldap://ldap3.test.local

and On all masters add syncproc to the config.

dn: olcOverlay=syncprov,olcDatabase={0}config,cn=config
changetype: add
objectClass: olcOverlayConfig
objectClass: olcSyncProvConfig
olcOverlay: syncprov

On all masters adding SyncRepl, between the servers. WARNING:
The password is cleartext, and simplebind does NOT support other than cleartext passwords.

dn: olcDatabase={0}config,cn=config
changetype: modify
add: olcSyncRepl
olcSyncRepl: rid=001 provider=ldap://ldap1.test.local binddn="cn=admin,cn=config" bindmethod=simple 
  credentials=test searchbase="cn=config" type=refreshAndPersist
  retry="5 5 300 5" timeout=1
olcSyncRepl: rid=002 provider=ldap://ldap2.test.local binddn="cn=admin,cn=config" bindmethod=simple 
  credentials=test searchbase="cn=config" type=refreshAndPersist
  retry="5 5 300 5" timeout=1
olcSyncRepl: rid=003 provider=ldap://ldap3.test.local binddn="cn=admin,cn=config" bindmethod=simple 
  credentials=test searchbase="cn=config" type=refreshAndPersist
  retry="5 5 300 5" timeout=1
-
add: olcMirrorMode
olcMirrorMode: TRUE

Now the Configuration part should be working!
You can test this by changing a configuration paramter on either Master1, Master2 or Master3 and confirm the configuration replicates.

Adding other databases for replication

Since, the configuration does already replicate, we only need to do this on ONE of the servers. Add syncproc module to "data" hdb database.
It is likely that you will get errors about olcRootDN, olcRootPW, and olcSuffix.
If you do this, change "add" with "replace". You wil get this if the parameter is already set in the config.
I have given an example,under "Errors and Fixes".

dn: olcOverlay=syncprov,olcDatabase={1}hdb,cn=config
changetype: add
objectClass: olcOverlayConfig
objectClass: olcSyncProvConfig
olcOverlay: syncprov

Add SyncRepl, between the servers. Make sure that olcSuffix, olcRootDN, and olcRootPW is correct

dn: olcDatabase={1}hdb,cn=config
changetype: modify
add: olcSuffix
olcSuffix: dc=test,dc=local
-
add: olcRootDN
olcRootDN: cn=admin,dc=test,dc=local
-
add: olcRootPW
olcRootPW: {SSHA}SGq737yNactRCyMY70TDTQs6V1wzMRD6
-
add: olcSyncRepl
olcSyncRepl: rid=004 provider=ldap://ldap1.test.local binddn="cn=admin,dc=test,dc=local" bindmethod=simple
  credentials=test searchbase="dc=test,dc=local" type=refreshOnly
  interval=00:00:00:10 retry="5 5 300 5" timeout=1
olcSyncRepl: rid=005 provider=ldap://ldap2.test.local binddn="cn=admin,dc=test,dc=local" bindmethod=simple
  credentials=test searchbase="dc=test,dc=local" type=refreshOnly
  interval=00:00:00:10 retry="5 5 300 5" timeout=1
olcSyncRepl: rid=006 provider=ldap://ldap3.test.local binddn="cn=admin,dc=test,dc=local" bindmethod=simple
  credentials=test searchbase="dc=test,dc=local" type=refreshOnly
  interval=00:00:00:10 retry="5 5 300 5" timeout=1
-
add: olcMirrorMode
olcMirrorMode: TRUE

Setup indexes for the hdb.

dn: olcDatabase={1}hdb,cn=config
changetype:  modify
add: olcDbIndex
olcDbIndex: entryCSN eq
olcDbIndex: entryUUID eq

If everything went well, on all of the 3 Master servers, you should now have a working replication.

Errors and Fixes

ldap_add: Server is unwilling to perform (53)
	additional info: shadow context; no update referral

Make sure that the module that olcMirrorMode is TRUE on the database

ldap_modify: Other (e.g., implementation specific) error (80)
	additional info: Base DN "00:00:00:10" is not within the database naming context

Make sure the olcSuffix, oldRootDN and olcRootPW is existing, and are correct, so add them, or replace them!

These example uses the hdb "data" database.
Adding if they are NOT present.

dn: olcDatabase={1}hdb,cn=config
changetype: modify
add: olcSuffix
olcSuffix: dc=test,dc=local
-
add: olcRootDN
olcRootDN: cn=admin,dc=test,dc=local
-
add: olcRootPW
olcRootPW: test

Replacing, if they ARE present.

dn: olcDatabase={1}hdb,cn=config
changetype: modify
replace: olcSuffix
olcSuffix: dc=test,dc=local
-
replace: olcRootDN
olcRootDN: cn=admin,dc=test,dc=local
-
replace: olcRootPW
olcRootPW: test

Refs.:
http://www.openldap.org/doc/admin24/replication.html#MirrorMode
http://www.openldap.org/doc/admin24/syncrepl.html
http://www.openldap.org/doc/admin24/replication.html#Set up the provider slapd