CODEWITHSUNDEEP
linuxmongodbwiredtiger
Sandip Pingle
Sandip Pingle

Reputation: 665

Error while Recovering a WiredTiger collection from a corrupt MongoDB installation

We have implemented MongoDB master-slave replication and database got deleted.

We restored deleted files from Mongo data directory using extundelete command.

Undelete Blog https://github.com/RIKSOF/development/wiki/Restoring-deleted-files-on-Linux

We wanted to restore at least one collection for which we got the WT (wired timer) file. We are trying to restore that file using the following commands and we are getting the following errors

Recovering a WiredTiger collection from a corrupt MongoDB installation

wget http://source.wiredtiger.com/releases/wiredtiger-2.7.0.tar.bz2 tar xvf wiredtiger-2.7.0.tar.bz2 cd wiredtiger-2.7.0 sudo apt-get install libsnappy-dev build-essential ./configure --enable-snappy make

./wt -v -h ../mongo-bak -C "extensions=[./ext/compressors/snappy/.libs/libwiredtiger_snappy.so]" -R salvage collection-2657--1723320556100349955.wt

Error -

./wt -v -h ../mongo-bak -C "extensions=[./ext/compressors/snappy/.libs/libwiredtiger_snappy.so]" -R salvage collection-246--7553069514495955510.wt [1488888117:36780][13536:0x7f7b633cd740], file:WiredTiger.wt, connection: read checksum error for 4096B block at offset 12288: block header checksum of 1955562709 doesn't match expected checksum of 3146787951 [1488888117:36809][13536:0x7f7b633cd740], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value [1488888117:36817][13536:0x7f7b633cd740], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic lt-wt: WT_PANIC: WiredTiger library panic

Upvotes: 2

Views: 11141

Answers (3)

CMLDMR
CMLDMR

Reputation: 344

I got this error and trying a lot of way to resolve but not reached to successfully.

I am currently using 5.0.1 mongodb standalone version,

  • removed currently installed mongodb ( 5.0.1 )
  • installed latest version ( 6.0 )

and reapaired all corrupted *.wt file and verified.

mongod --repair --dbpath /path/to/data/db

Upvotes: 0

Krishna Vedula
Krishna Vedula

Reputation: 1791

In my case, I had recovered mac from Time Machine. I had mongodb installed via brew install. When I started mongo, the service did not start cleanly. 

vi /usr/local/var/log/mongodb/mongo.log
enter code here
2020-01-30T17:06:10.655-0500 E  STORAGE  [initandlisten] WiredTiger error (0) [1580421970:655376][62181:0x112233dc0], file:WiredTiger.wt, connection: __wt_block_read_off, 283: WiredTiger.wt: read checksum error for 4096B block at offset 12288: block header checksum of 0x325e318 doesn't match expected checksum of 0x65ef0e12 Raw: [1580421970:655376][62181:0x112233dc0], file:WiredTiger.wt, connection: __wt_block_read_off, 283: WiredTiger.wt: read checksum error for 4096B block at offset 12288: block header checksum of 0x325e318 doesn't match expected checksum of 0x65ef0e12

What I saw was that because of a corrupt mongodb collection. So, I ran the cleanup command for the brew install, which has the db in a different location.

mongod --repair --dbpath /usr/local/var/mongodb

    2020-01-30T17:06:11.741-0500 I  STORAGE  [initandlisten] Verify failed on uri table:collection-12--8361386101775862971. Running a salvage operation.
2020-01-30T17:06:11.745-0500 I  STORAGE  [initandlisten] Repairing collection wlchat.jobs
2020-01-30T17:06:11.765-0500 I  STORAGE  [initandlisten] WiredTiger progress WT_SESSION.verify 100
2020-01-30T17:06:11.769-0500 I  STORAGE  [initandlisten] WiredTiger progress WT_SESSION.verify 200
2020-01-30T17:06:11.774-0500 I  STORAGE  [initandlisten] WiredTiger progress WT_SESSION.verify 300
2020-01-30T17:06:11.783-0500 I  STORAGE  [initandlisten] WiredTiger progress WT_SESSION.verify 400
2020-01-30T17:06:11.789-0500 I  STORAGE  [initandlisten] WiredTiger progress WT_SESSION.verify 500
2020-01-30T17:06:11.793-0500 I  STORAGE  [initandlisten] WiredTiger progress WT_SESSION.verify 600
2020-01-30T17:06:11.795-0500 I  STORAGE  [initandlisten] Verify succeeded on uri table:collection-10--8361386101775862971. Not salvaging.
2020-01-30T17:06:11.795-0500 I  STORAGE  [initandlisten] Repairing collection wlchat.SovrenJobs
2020-01-30T17:06:11.801-0500 I  STORAGE  [initandlisten] Verify succeeded on uri table:collection-0-8794251073515214768. Not salvaging.
2020-01-30T17:06:11.814-0500 I  INDEX    [initandlisten] index build: starting on wlchat.users properties: { v: 2, key: { _id: 1 }, name: "_id_", ns: "wlchat.users" } using method: Foreground

and restart mongodb-community

brew services restart mongodb-community.
==> Successfully started `mongodb-community` (label: homebrew.mxcl.mongodb-community)

Things are back to normal.

Upvotes: 2

kenorb
kenorb

Reputation: 166409

This could be related to corrupted .wt files (e.g. WiredTiger.wt/WiredTiger.turtle) as per SERVER-31076 bug report.

Try to run repair on all databases by the following command:

mongod --repair --dbpath /path/to/data/db

Also make sure all data/db files have the right read and write permission.

Upvotes: 5

Related Questions