symfony2 app: mongodb replica set failover requires apache restart

Question

My Symfony2 app connects to a MongoDB replica set (2 full nodes and an arbiter). After failover is complete (a new primary has successfully been elected), many web requests result in server errors. If I restart Apache (but make no other changes), the exceptions go away and the app works as expected (queries the new primary, no problem).

Before restarting Apache I get MongoCursorException with the message not master and slaveOk=false or couldn't determine master. Not consistently: it seems like it depends on which Apache worker I hit. Or something. Anyway, restarting Apache seems to fix the app immediately and all queries succeed normally.

My replica set is for redundancy, not performance, so I never use slaveOk=true.

These options are passed to the Mongo constructor:

• connect => TRUE
• replicaSet => foo

I'm using:

• PHP 5.3.10
• APC 3.1.7
• MongoDB 2.2.1
• Symfony2 version 2.0.18

From my deps file:

• doctrine 2.1.7
• doctrine-mongodb 1.0.0-BETA1
• doctrine-mongodb-odm 1.0.0-BETA5
• doctrine-common 2.2.0
• doctrine-dbal 2.1.7

It smells like the Symfony2 app is trying to re-use stale MongoDB connections. The primary's log supports this guess: connections add up as I hit the app web pages and it makes queries, and when I restart apache many connections are released.

• Do I need to forcibly clear my APC cache?
• Do I just need to restart all my webservers after a failover event? (what the heck do other folks do?)
• Do I need to play with mongo.is_master_interval?
• I thought retry_query and retry_connect would smooth out failover automatically, but that doesn't appear to be the case. Do I need to add try/catch blocks to my code?

Related:

• Doctrine ODM / MongoDB not retrying queries? (I'm trying out the new retry_query and retry_connect options)

Possibly related:

• PHP MongoDB errors when using automatic failover

Answer 1

Am I correct in assuming that you were using version 1.2.12 of the PHP driver? Given that restarting Apache seems to fix the problem, it looks like the driver is simply re-using the wrong connection in its pool. Restarting Apache and its workers effectively clears out all of the pools, as each worker has its own connection pool to be shared across requests served in its lifetime. It's entirely likely that some workers refreshed their connections after the failover while others did not, which would lead you to still get an exception depending on which worker was hit.

The connection handling was overhauled in 1.3.0, so you should see some improvement in failover support once you upgrade. Doctrine MongoDB should support 1.3.0 properly once PR #81 is merged, with ODM following soon after.

The retry_ options in Doctrine do not support delays between successive attempts, which makes them less suited for dealing with failovers, which can take 10-30 seconds. I believe Jon's original intention for them was to deal with dropped connections and network blips.

I'll also note that until PHP driver 1.3.0, the mongo.is_master_interval INI setting was never actually used. This was fixed in PHP-576.