Mongo service crashed. Need to find reason of crash

Question

Today morning I found the following error with mongo, on my server.

*** System restart required ***
You have mail.
ubuntu@ip-xxx-xx-xx-xx:~$ mongo
MongoDB shell version: 2.4.9
connecting to: test
Wed Nov 25 05:18:51.381 Error: couldn't connect to server 127.0.0.1:27017 at src/mongo/shell/mongo.js:145
exception: connect failed

It crashed my website. How can I find what stopped mongo service? Do mongo have logs for this? Where can I find them? Also How to make sure if if mongodb is crashed then it restart service as well?

I restarted mongodb service and website started working again.

Output of ps -ef | grep mongod:

mongodb   14756      1  0 05:20 ?        00:00:14 /usr/bin/mongod --config /etc/mongodb.conf
ubuntu    36865  14123  0 06:18 pts/2    00:00:00 grep --color=auto mongod

Answer 1

My server also crashed for no obvious reason. The other answers did not throw anything. But I found the following looking at the mongod.conf file in /etc/.

There are also logs inside /var/log/mongodb/mongod.log. It seems that by default it has a 500 Mb limit. But is still a big file.

There I found that my server crashed at a certain time, but couldn't find the reason.

2019-04-17T19:39:11.700+0000 I COMMAND  [conn460] command cge_info_system.expedientes command: find { find: "expedientes", filter: { NoExpediente: "090311-497449" }, limit: 1, $db: "cge_info_system", $readPreference: { mode: "primaryPreferred" }, lsid: { id: UUID("f18d952b-37e7-4a45-bf31-b89a596057d1") } } planSummary: COLLSCAN keysExamined:0 docsExamined:49360 cursorExhausted:1 numYields:386 nreturned:1 reslen:3224 locks:{ Global: { acquireCount: { r: 387 } }, Database: { acquireCount: { r: 387 } }, Collection: { acquireCount: { r: 387 } } } protocol:op_msg 1416ms
2019-04-23T16:35:19.578+0000 I CONTROL  [main] ***** SERVER RESTARTED *****
2019-04-23T16:35:19.599+0000 I CONTROL  [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'
2019-04-23T16:35:19.958+0000 I CONTROL  [initandlisten] MongoDB starting : pid=741 port=27017 dbpath=/var/lib/mongodb 64-bit host=bb7dd22f-cb77-4c6b-9c4c-9fddcfc2cc00.clouding.host
2019-04-23T16:35:19.958+0000 I CONTROL  [initandlisten] db version v4.0.5
2019-04-23T16:35:19.958+0000 I CONTROL  [initandlisten] git version: 3739429dd92b92d1b0ab120911a23d50bf03c412
2019-04-23T16:35:19.958+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.1.0g  2 Nov 2017
2019-04-23T16:35:19.958+0000 I CONTROL  [initandlisten] allocator: tcmalloc
2019-04-23T16:35:19.958+0000 I CONTROL  [initandlisten] modules: none
2019-04-23T16:35:19.958+0000 I CONTROL  [initandlisten] build environment:
2019-04-23T16:35:19.958+0000 I CONTROL  [initandlisten]     distmod: ubuntu1804
2019-04-23T16:35:19.958+0000 I CONTROL  [initandlisten]     distarch: x86_64
2019-04-23T16:35:19.958+0000 I CONTROL  [initandlisten]     target_arch: x86_64
2019-04-23T16:35:19.958+0000 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1", port: 27017 }, processManagement: { timeZoneInfo: "/usr/share/zoneinfo" }, storage: { dbPath: "/var/lib/mongodb", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2019-04-23T16:35:19.961+0000 W STORAGE  [initandlisten] Detected unclean shutdown - /var/lib/mongodb/mongod.lock is not empty.
2019-04-23T16:35:19.962+0000 I STORAGE  [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2019-04-23T16:35:19.962+0000 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
2019-04-23T16:35:19.962+0000 I STORAGE  [initandlisten]

In there is also a reference to a directory called /var/lib/mongodb/diagnostic.data but the data in there are binary files. I haven't got much info from there.

Answer 2

I had the same issue. After some searching I guessed it was os that sacrificed mongod for memory. If kernel killed your mongod you can find the logs in kernel logs

Mostly kernel logs will be in /var/log/kern.log, The logs might be rotated.

So better to do a grep

grep mongod /var/log/kern.log*

if the logs are archived find -name \*kern.log.*.gz -print0 | xargs -0 zgrep "mongod"

./kern.log.1:Nov 12 12:55:04 ip-xxx-xxx-xxx-xxx kernel: [8413229.118361] mongod invoked oom-killer: gfp_mask=0x24201ca, order=0, oom_score_adj=0
./kern.log.1:Nov 12 12:55:04 ip-xxx-xxx-xxx-xxx kernel: [8413229.118365] mongod cpuset=/ mems_allowed=0
./kern.log.1:Nov 12 12:55:04 ip-xxx-xxx-xxx-xxx kernel: [8413229.118371] CPU: 0 PID: 1830 Comm: mongod Not tainted 4.4.0-1062-aws #71-Ubuntu
./kern.log.1:Nov 12 12:55:04 ip-xxx-xxx-xxx-xxx kernel: [8413229.118531] [ 1301]   112  1301   300787   181207     485       4        0             0 mongod
./kern.log.1:Nov 12 12:55:04 ip-xxx-xxx-xxx-xxx kernel: [8413229.118680] Out of memory: Kill process 1301 (mongod) score 355 or sacrifice child
./kern.log.1:Nov 12 12:55:04 ip-xxx-xxx-xxx-xxx kernel: [8413229.130161] Killed process 1301 (mongod) total-vm:1203148kB, anon-rss:724828kB, file-rss:0kB

Answer 3

According to MongoDB documentation

The getLog command returns a document with a log array that contains recent messages from the mongod process log.

Please try executing following command in mongo shell to return logs which consist of information related to errors or warnings while initialization of mongod service

db.adminCommand({getLog: "startupWarnings" })

For detailed description regarding getLog method please refer the documentation mentioned in following URL

https://docs.mongodb.org/manual/reference/command/getLog/