CODEWITHSUNDEEP
mongodbshardingreplicaset
N.B.
N.B.

Reputation: 11

MongoDB storageEngine from MMAPv1 to wiredTiger fassert() failure

I am upgrading my cluster to wiredTiger using this site: https://docs.mongodb.org/manual/tutorial/change-replica-set-wiredtiger/

I have been having the following issue:

Environment details:

MongoDB 3.0.9 in a sharded cluster on Red Hat Enterprise Linux Server release 6.2 (Santiago). I have 4 shards, each one is a replica set with 3 members. I just recently upgraded all binaries from 2.4 to 3.0.9. Every server has updated binaries, I tried converting each replica set to wired tiger storage engine, but I was getting the following error when upgrading the secondary on one member server (shard 1):

2016-02-09T12:36:39.366-0500 F REPL     [rsSync] replication oplog stream went back in time. previous timestamp: 56b9c217:ab newest timestamp: 56b9b429:60. Op being applied: { ts: Timestamp 1455010857000|96, h: 2267356763748731326, v: 2, op: "d", ns: "General.Tickets", fromMigrate: true, b: true, o: { _id: ObjectId('566aec7bdfd4b700e73d64db') }
2016-02-09T12:36:39.366-0500 I -        [rsSync] Fatal Assertion 18905
2016-02-09T12:36:39.366-0500 I -        [rsSync]
***aborting after fassert() failure

This is an open bug with replication: https://jira.mongodb.org/browse/SERVER-17081

Every other part of the cluster, the upgrade went flawlessly, however, now I am stuck with only the primary and one secondary on shard 1. I've attempted resyncing the broken member using MMAPv1 and Wired Tiger, but I continually get the error above. Because of this, one shard is stuck using MMAPV1, and that shard happens to have most of the data (700 GB).

I have also tried rebooting, re-installing the binaries, to no avail.

Any help is appreciated.

Upvotes: 0

Views: 259

Answers (1)

N.B.
N.B.

Reputation: 11

I solved this by dropping our giant collection. The rssync must have been hitting some limit since the giant collection had 4.5 billion documents.

Upvotes: 0

Related Questions