Reputation: 1532
I've got a Mongo replica set running. Here's the config:
It's been humming along smoothly for months now. No real problems. Late last night our monitoring notified me that one of the secondaries was falling behind in replication. It's currently about an hour behind and continuously falling more behind (at a slow rate). None of the other secondaries have this problem - they're keeping up with the primary with no problem.
I can't find anything in the logs on any of these machines to indicate what the cause might be. I've had experience with replication lag in the past, but it always affected EVERY secondary, not just a single one. This is feeling like a hardware problem but these are physical machines (no virtualization) and our vendor ran checks that seem to indicate no issue.
A few more pieces of info in case they're helpful:
I appreciate any guidance given. Thanks!
Upvotes: 2
Views: 287
Reputation: 1532
Proving, once again, that computers are magic, the offending secondary magically caught up while I was weeping in the shower.
Happy to give points to anybody who has solid guesses about why this may have happened. I'm chalking it up to ghosts.
Upvotes: 2