Reputation: 71
Our DB of +- 400Gb is stopping on our one server.
From the logs:
2015-07-07T09:09:51.072+0200 I STORAGE [conn10] _getOpenFile() invalid file index requested 8388701
2015-07-07T09:09:51.072+0200 I - [conn10] Invariant failure false src/mongo/db/storage/mmap_v1/mmap_v1_extent_manager.cpp 201
2015-07-07T09:09:51.082+0200 I CONTROL [conn10]
Any idea in what are I should start looking? Storage issue?
Upvotes: 7
Views: 11396
Reputation: 28161
In my case this happened in a development setting with MongoDB 3.6.20 on macOS 10.14.6. Another program restarted the mac and close any open terminals, including the terminal that ran the mongod process. After the OS restart, I could not restart the mongod because the Invariant failure. The error also mentioned a bad lockfile.
I was able to solve the issue with the following steps, yet I am not exactly sure which did the job:
rm -rf data/db/mongod.lock
mongod --repair
SocketException: Address already in use
.The first successful mongod run after the issue gave the following output:
[ftdc] Unclean full-time diagnostic data capture shutdown detected, found interim file, some metrics may have been lost.
Thus, it runs smoothly again. Maybe I was fortunate. I hope the same approach helps some of you.
Upvotes: 0
Reputation: 321
I am just answering this question in case some people make the same non-technical mistake again:
I tried to scp
all the files in the /data/db
directory to the server. As the files are many (dbname.1
to dbname.55
, about 100GB), it was interrupted in the middle (last successful file dbname.22
), and I restarted and uploaded dbname.23
to dbname.55
. And when I run queries in mongo
client, it worked for some cases, and failed for some others showing the error message the same as in the question. I thought it might be some file broken in the file transferring, but the md5 check was all right. Only after I spent a long time finishing all the md5 check I found the reason.
It turned out to be that scp
uploads dbname.21
to dbname.29
after it uploads dbname.2
, so dbname.3
to dbname.9
was never uploaded to the server. I am going to upload them, and this should solve the problem.
Upvotes: 2
Reputation: 1112
I ran into a variant of this today as well. Mysteriously one of my data files disappeared (or didn't make it in a migration from another server). None of the repair/recovery procedures would work, failing on the same error you reference. Luckily I have a separate mongod that has a collection with the same name, so as a cheap hack I copied the (admittedly wrong) data file to the other server, and while I knew I wouldn't get any data back, the repair tools (such as mongod --repair
) were then able to work their magic, but as expected, they recovered some data from the bad file I copied in, so I had to weed out some docs. Luckily it was the "mycollection.1" file, which is only 128MB.
I don't think this applies in your case since index of the missing data file your log is talking about is ridiculously high. Your log is essentially saying it can't find /data/dbname/mycollection.8388701
. You said your data-set is only 400GB, so an index that high just doesn't make sense. You should have only roughly 200 data files since most of them are 2GB each by default. What is the result of db.stats()
(specifically the fileSize attribute)?
This mongolab blog entry helped me understand the data file structure.
My advice for where you should start looking:
db.stats()
command to get an idea of how big your data on
disk actually is.mongod --repair
, or db.repairDatabase()
tools to start a repair. I'm assuming it won't work since my repair attempts crashed with the same invalid file index requested
error.Hope that helps point you in the right direction.
Upvotes: 1