Creating and maintaining MongoDB replica sets with pymongo

Question

I am trying to replicate (for a teaching activity) the Docker and MongoDB Sharded Cluster recipe in an IPython notebook using pymongo to set up several mongo replica sets.

The recipe suggests creating a replica set from the mongo prompt by connecting to one member of the proposed replica set as follows:

mongo --port <port>

rs.initiate()
rs.add("<IP_of_rs1_srv2>:27017")
rs.add("<IP_of_rs1_srv3>:27017")
rs.status()

The pymongo documentation suggests the following initialisation route:

$ hostname
morton.local
$ mongod --replSet foo/morton.local:27018,morton.local:27019 --rest
$ mongod --port 27018 --dbpath /data/db1 --replSet foo/morton.local:27017 --rest
$ mongod --port 27019 --dbpath /data/db2 --replSet foo/morton.local:27017 --rest

from pymongo import MongoClient, ReadPreference
c = MongoClient("morton.local:27017", read_preference=ReadPreference.SECONDARY)

c.admin.command("replSetInitiate")

This requires knowing the port ids and using them in the initialisation of the database servers via the --replSet flag, rather than just declaring a simple single label for the replica set and then applying it to each member. How can I script the initialisation in pymongo to follow the original recipe?

The original also modifies the configuration by changing the hostname to include the container IP address:

cfg = rs.conf()
cfg.members[0].host = "<IP_of_rs1_srv1>:27017"
rs.reconfig(cfg)
rs.status()

Again, pymongo doesn't support the rs helpers, so how can I update the replica set configuration?

Answer 1

One solution to this is to create a configuration for the replica set based on the attributes of the mongodb containers and then use that to initiate the cluster set.

So for example, a configuration might take the form:

rsc = {'_id': 'rs4',
 'members': [{'_id': 0, 'host': '172.17.0.2:27017'},
  {'_id': 1, 'host': '172.17.0.3:27017'},
  {'_id': 2, 'host': '172.17.0.4:27017'}]}

and then be called with the replSetInitiate admin command.

import docker

#Connect to docker
c = docker.Client(base_url='unix://var/run/docker.sock',
                  version='1.10',
                  timeout=10)

Create a set of database nodes to run as the replica set:

def createReplicaSetNode(c,stub,num=0):
    ''' Create and run a specified number of mongo database servers as a replica set '''
    name='{stub}_srv{num}'.format(stub=stub,num=num)
    command='--replSet {stub}  --noprealloc --smallfiles'.format(stub=stub)
    c.create_container('dev24/mongodb',name=name,command=command)
    c.start(name,publish_all_ports=True)
    return name

def createReplicaSetNodes(c,stub,numNodes):
    ''' Create and run a specified number of mongo database servers as a replica set '''
    names=[]
    for i in range(0,numNodes):
        name=createReplicaSetNode(c,stub,i)
        names.append(name)
    return names


def getContainIPaddress(c,container):
    ''' Get the IP address of the container '''
    cConfig = c.inspect_container(container)
    return cConfig['NetworkSettings']['IPAddress']

def rs_config(c,rsid,num=3):
    ''' Create a replica set of nodes and then define a configuration file for that replica set '''
    createReplicaSetNodes(c,rsid,num)
    _rs_config={"_id" : rsid, 'members':[] }
    #This is scrappy - should really return something better from the creation
    for i in range(0,num):
        name='{stub}_srv{num}'.format(stub=rsid,num=i)
        #c.inspect_container(name)
        #get IP and port
        _rs_config['members'].append({"_id":i,"host":'{0}:{1}'.format(getContainIPaddress(c,name),27017)})
    return _rs_config

We can then start the nodes and create a config file for the replica set:

rsc=rs_config(c,'rs4')
rsc

'''
{'_id': 'rs4',
 'members': [{'_id': 0, 'host': '172.17.0.2:27017'},
  {'_id': 1, 'host': '172.17.0.3:27017'},
  {'_id': 2, 'host': '172.17.0.4:27017'}]}
'''

This configuration data is used to initialise the replica set:

from pymongo import MongoClient

#Find the local port bound for 27017/tcp for each server in the replica set
def get27017tcp_port(c,container):
    cConfig = c.inspect_container(container)
    return int(cConfig['NetworkSettings']['Ports']['27017/tcp'][0]['HostPort'])

#We'll use the 0th server in the set as a the node
mc = MongoClient('localhost', get27017tcp_port(c,'rs4_srv0'))


#In mongo console, we'd typically use command rs.config() to initialise replica set
#Here, use replSetInitiate admin command, applying it with desired configuration
mc.admin.command( "replSetInitiate",rsc);