How to set up MongoDB for multiple Data Center?

Question

Here is the environment:

[ Data Center 1 ]
   [ load balancer, ip: 45.45.45.45] 
     [ Server 1-A, internal ip: 10.0.0.1, external ip: 200.0.0.1 ]
     [ Server 1-B, internal ip: 10.0.0.2, external ip: 200.0.0.2 ]
     [ Server 1-C, internal ip: 10.0.0.3, external ip: 200.0.0.3 ]

[ Data Center 2 ]
   [ load balancer, ip: 90.90.90.90] 
     [ Server 2-A, internal ip: 10.0.0.1, external ip: 201.0.0.1 ]
     [ Server 2-B, internal ip: 10.0.0.2, external ip: 201.0.0.2 ]
     [ Server 2-C, internal ip: 10.0.0.3, external ip: 201.0.0.3 ]

[ Data Center 3 ]
   [ load balancer, ip: 88.88.88.88] 
     [ Server 3-A, internal ip: 10.0.0.1, external ip: 221.0.0.1 ]
     [ Server 3-B, internal ip: 10.0.0.2, external ip: 221.0.0.2 ]
     [ Server 3-C, internal ip: 10.0.0.3, external ip: 221.0.0.3 ]

---

What I would like to achieve is that each server install one mongo server, and only Data Center 1 servers (1-A, 1-B, and 1-C) are allowed to be primary. MongoDB servers in Data Center 2 and Data Center 3 are only secondary. Applications can specifically read data from Data Center 2, because those applications may be in the same network as Data Center 2 which has a faster access than connecting to Data Cetner 1. Server is using replicate Sets. There is no sharding.

Here are my questions:

1. When setting up the replicate set, do I need to use external real IP address to specify the host from Data Center 2 and 3? e.g.

   config = {  _id: 'foo', members: [
                         // data center 1
                         {_id: 0, host: '10.0.0.1:27017'},
                         {_id: 1, host: '10.0.0.2:27017'},
                         {_id: 2, host: '10.0.0.3:27017'},
                         // data center 2
                         {_id: 3, host: '201.0.0.1:27017'},
                         {_id: 4, host: '201.0.0.2:27017'},
                         {_id: 5, host: '201.0.0.3:27017'},
                         // data center 3
                         {_id: 6, host: '221.0.0.1:27017'},
                         {_id: 7, host: '221.0.0.2:27017'},
                         {_id: 8, host: '221.0.0.3:27017'}
                        ]
          }
   

2. Because the servers in Data Center 2 will behind load balancer and firewall, the IP address may not be able to explode to outside world. Is it possible to use the load balancer IP address? if so, how to do it?

3. I should use priorty=0 to make servers in Data Center 2 & Data Center 3 never be primary, right?

4. How other application connect to the MongoDB in Data Center 2 and 3 (for read data only)? Let say, the application (a server side app) is also in Data Center 2 network. What IP address should the application use to connect to the MongoDB in DC2? Should I use the internal IP or external IP? If an application is outside of the DB internal network and would like to connect to MongoDB in DC2 to read data only, should I use the load balancer IP or external IP?

5. Because I want Servers in DC2 and DC3 able to allow read data, how to set slaveOk? Should I set it in the primary "slaveOK", or should I set it in each server in DC2 and DC3?

6. One step further, let say DC1, DC2 and DC3 are all individual replicate set. e.g. DC 1 has its only primary and secondary. DC2 has its own primary and secondary, so far DC3. Is there an easy and real-time way to update DC2 and DC3 when there is change (new record or update) in DC1?

Answer 1

1. Depends. They need to be hostnames/IPs that your application servers can access. If your app servers run on the same DC your mongod process are running they can and should be internal. Any other setup has major security issues.
2. Technically, yes. Assign each mongod process to a different port. However, I don't understand why you'd ever want a load balancer to distribute mongo queries. That's what the driver would do for you and there might even be side effects to letting something other than the driver decide which RS member the query goes to.
3. Correct.
4. Internal IPs are the way to go. App servers should initialize their driver with all internal IPs of the local members (read: the members on the same DC)
5. You don't set servers to slaveOk. SlaveOk functionality is on the driver end of things. Typically set per connection but most drivers have a global and per query flag for it as well. Obviously your setup will not work without this flag enabled.
6. No. There is no master-master replication in MongoDB atm.