mongodb readpreference and read semantics

Question

I am trying to understand mongodb read semantics. According to docs it's always primary server in replica set unless you change the readpreferences.

My Setup: Linux VM: 4GB RAM, 1 COre, Running 3 mongo instances on same VM (I know it's a bad setup! but that's how I got from ops for now)

I am just monitoring memory usage, logs etc while I run following mongo query from Java Driver.

Here's what I see:

ReadPreference.PrimaryPreferred:

top

PID                USER      PR  NI VIRT  RES  SHR  S %CPU %MEM    TIME+  COMMAND 
17874(primary)     mongod    20   0 43.0g 1.3g 1.1g S 14.3 36.0 109:06.71 mongod                                                                                                                                                                    
18048(secondary-1) mongod    20   0 42.7g 690m  88m S  0.3 18.0  28:45.09 mongod                                                                                                                                                                    
18207(secondary-2) mongod    20   0 42.7g 641m  60m S  0.3 16.7  28:53.13 mongod   

logs

Primary-1

Fri Aug  2 21:35:39.416 [initandlisten] connection accepted from 10.10.1.57:54306 #11927 (40 connections now open)
Fri Aug  2 21:44:16.332 [initandlisten] connection accepted from 10.10.1.57:54376 #11963 (41 connections now open)
Fri Aug  2 21:44:16.348 [initandlisten] connection accepted from 10.10.1.57:54377 #11964 (42 connections now open)
Fri Aug  2 21:44:16.414 [initandlisten] connection accepted from 10.10.1.57:54380 #11965 (43 connections now open)

Secondary-1

Fri Aug  2 20:03:43.066 [conn11270] end connection 10.10.1.57:53547 (20 connections now open)
Fri Aug  2 20:03:43.066 [conn11271] end connection 10.10.1.57:53551 (19 connections now open)
Fri Aug  2 21:44:16.331 [initandlisten] connection accepted from 10.10.1.57:54374 #11776 (19 connections now open)
Fri Aug  2 21:44:16.350 [initandlisten] connection accepted from 10.10.1.57:54378 #11777 (20 connections now open)

Secondary-2

Fri Aug  2 20:03:43.065 [conn11271] end connection 10.10.1.57:53552 (20 connections now open)
Fri Aug  2 20:03:43.066 [conn11270] end connection 10.10.1.57:53549 (19 connections now open)
Fri Aug  2 21:44:16.331 [initandlisten] connection accepted from 10.10.1.57:54375 #11776 (19 connections now open)
Fri Aug  2 21:44:16.352 [initandlisten] connection accepted from 10.10.1.57:54379 #11777 (20 connections now open)

ReadPreference.Secondary:

top

PID                USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                                                  
18048(secondary-1) mongod    20   0 42.7g 1.1g 517m S  7.3 29.2  29:28.00 mongod                                                                                                                                                                    
17874(primary)     mongod    20   0 43.0g 1.0g 777m S  0.3 26.7 112:57.05 mongod                                                                                                                                                                    
18207(secondary-2) mongod    20   0 42.7g 617m  37m S  0.3 16.1  28:59.08 mongod

logs

==> mongod_rs0-0.log <==

Fri Aug  2 22:19:01.056 [conn12118] end connection 10.10.10.201:52558 (34 connections now open)
Fri Aug  2 22:19:01.057 [initandlisten] connection accepted from 10.10.10.201:52564 #12120 (35 connections now open)

==> mongod_rs0-1.log <==

Fri Aug  2 22:19:04.038 [conn11925] end connection 10.10.10.201:46443 (15 connections now open)
Fri Aug  2 22:19:04.039 [initandlisten] connection accepted from 10.10.10.201:46449 #11927 (16 connections now open)

==> mongod_rs0-2.log <==

Fri Aug  2 22:19:04.050 [conn11925] end connection 10.10.10.201:37641 (14 connections now open)
Fri Aug  2 22:19:04.050 [initandlisten] connection accepted from 10.10.10.201:37647 #11927 (15 connections now open)

My Questions are:

1. In both scenario CPU usage says reflects your readpreferences but I don't understand logs very well. Why it opens connection on each instance when I query?
2. Why primary maintains more no. of connections even if readpreference is SecondaryPreferred.
3. tcpdump on each instance is active. 2 of them should be idle, right? There's no other client/processes calling this cluster.

Answer 1

The connections to the primary and secondary servers might not even be your client connecting and querying the server. The primary and secondary servers need to stay connected to each other so that a) writes on the primary can be replicated on the secondaries and b) all servers can make sure there still is a primary available.

For example, I have a three server replica set up and running on my laptop right now, but there are no processes connected to it or querying it, and the log on the primary says:

[conn15939] end connection 127.0.0.1:50584 (6 connections now open)
[initandlisten] connection accepted from 127.0.0.1:50597 #15941 (7 connections now open)

On the secondaries I have half that number of connections. This is because the secondary only needs to be connected to its primary, but the primary will have connections from all secondaries.