CODEWITHSUNDEEP
mongodb
Paul McMaster
Paul McMaster

Reputation: 71

Mongo DB Invariant failure

Our DB of +- 400Gb is stopping on our one server.

From the logs:

2015-07-07T09:09:51.072+0200 I STORAGE  [conn10] _getOpenFile() invalid file index requested 8388701
2015-07-07T09:09:51.072+0200 I -        [conn10] Invariant failure false src/mongo/db/storage/mmap_v1/mmap_v1_extent_manager.cpp 201
2015-07-07T09:09:51.082+0200 I CONTROL  [conn10]

Any idea in what are I should start looking? Storage issue?

Upvotes: 7

Views: 11396

Answers (3)

Akseli Palén
Akseli Palén

Reputation: 28161

In my case this happened in a development setting with MongoDB 3.6.20 on macOS 10.14.6. Another program restarted the mac and close any open terminals, including the terminal that ran the mongod process. After the OS restart, I could not restart the mongod because the Invariant failure. The error also mentioned a bad lockfile.

I was able to solve the issue with the following steps, yet I am not exactly sure which did the job:

  • remove corrupted lock file: rm -rf data/db/mongod.lock
    • direct outcome: mongod still failed due to Invariant failure but at least no mention about the lockfile anymore.
  • run mongod --repair
    • direct outcome: repair still failed due to Invariant failure. Error output mentions SocketException: Address already in use.
  • restart the machine again to free the socket.
    • direct outcome: mongod starts and runs without problems. Yay.

The first successful mongod run after the issue gave the following output:

[ftdc] Unclean full-time diagnostic data capture shutdown detected, found interim file, some metrics may have been lost.

Thus, it runs smoothly again. Maybe I was fortunate. I hope the same approach helps some of you.

Upvotes: 0

moonkey
moonkey

Reputation: 321

I am just answering this question in case some people make the same non-technical mistake again:

I tried to scp all the files in the /data/db directory to the server. As the files are many (dbname.1 to dbname.55, about 100GB), it was interrupted in the middle (last successful file dbname.22), and I restarted and uploaded dbname.23 to dbname.55. And when I run queries in mongo client, it worked for some cases, and failed for some others showing the error message the same as in the question. I thought it might be some file broken in the file transferring, but the md5 check was all right. Only after I spent a long time finishing all the md5 check I found the reason.

It turned out to be that scp uploads dbname.21 to dbname.29 after it uploads dbname.2, so dbname.3 to dbname.9 was never uploaded to the server. I am going to upload them, and this should solve the problem.

Upvotes: 2

QuickDanger
QuickDanger

Reputation: 1112

I ran into a variant of this today as well. Mysteriously one of my data files disappeared (or didn't make it in a migration from another server). None of the repair/recovery procedures would work, failing on the same error you reference. Luckily I have a separate mongod that has a collection with the same name, so as a cheap hack I copied the (admittedly wrong) data file to the other server, and while I knew I wouldn't get any data back, the repair tools (such as mongod --repair) were then able to work their magic, but as expected, they recovered some data from the bad file I copied in, so I had to weed out some docs. Luckily it was the "mycollection.1" file, which is only 128MB.

I don't think this applies in your case since index of the missing data file your log is talking about is ridiculously high. Your log is essentially saying it can't find /data/dbname/mycollection.8388701. You said your data-set is only 400GB, so an index that high just doesn't make sense. You should have only roughly 200 data files since most of them are 2GB each by default. What is the result of db.stats() (specifically the fileSize attribute)?

This mongolab blog entry helped me understand the data file structure.

My advice for where you should start looking:

  1. run the db.stats() command to get an idea of how big your data on disk actually is.
  2. Does it make sense for your server to be looking for a data file with a crazy high index? If not, the issue isn't really with storage, but with the extents and the metadata of your collection/database.
  3. Do your repair tools work? If you have at least enough free disk space as the size of your data set (on disk), try the mongod --repair, or db.repairDatabase() tools to start a repair. I'm assuming it won't work since my repair attempts crashed with the same invalid file index requested error.
  4. Try copying a "bad" file like I did that roughly matches what the missing file would look like (keeping in mind how the file sizes of the data files aren't all the same, do your best to match it up and try a repair). If this works, your data files will be cleaned up (but it does take a lot of disk space).

Hope that helps point you in the right direction.

Upvotes: 1

Related Questions