CODEWITHSUNDEEP
mongodbsharding
Avinash
Avinash

Reputation: 801

MongoDB: One of the shard is not equally balanced like other shards

I have a sharded cluster setup for my app but unfortunately one of the shard is taking 17 GB of data size and others are taking average 3 GB of data size. What could be the issue?

10 shards

sh.status() gives me huge output. Shared here: https://www.dropbox.com/s/qqsucbm6q9egbhf/shard.txt?dl=0

My bad collection shard distribution details is below. 

mongos> db.MyCollection_1_100000.getShardDistribution()

Shard shard_0 at shard_0/mongo-11.2816.mongodbdns.com:270                                                                              

00,mongo-12.2816.mongodbdns.com:27000,mongo-13.2816.                                                                              mongodbdns.com:27000,mongo-3.2816.mongodbdns.com:27003
     data : 143.86MiB docs : 281828 chunks : 4
     estimated data per chunk : 35.96MiB
     estimated docs per chunk : 70457

    Shard shard_1 at shard_1/mongo-10.2816.mongodbdns.com:270                                                                              00,mongo-11.2816.mongodbdns.com:27002,mongo-19.2816.                                                                              mongodbdns.com:27001,mongo-9.2816.mongodbdns.com:27005
     data : 107.66MiB docs : 211180 chunks : 3
     estimated data per chunk : 35.88MiB
     estimated docs per chunk : 70393

    Shard shard_2 at shard_2/mongo-14.2816.mongodbdns.com:270                                                                              00,mongo-3.2816.mongodbdns.com:27000,mongo-4.2816.mo                                                                              ngodbdns.com:27000,mongo-6.2816.mongodbdns.com:27002
     data : 107.55MiB docs : 210916 chunks : 3
     estimated data per chunk : 35.85MiB
     estimated docs per chunk : 70305

    Shard shard_3 at shard_3/mongo-14.2816.mongodbdns.com:270                                                                              04,mongo-18.2816.mongodbdns.com:27002,mongo-6.2816.m                                                                              ongodbdns.com:27000,mongo-8.2816.mongodbdns.com:27000
     data : 107.99MiB docs : 211506 chunks : 3
     estimated data per chunk : 35.99MiB
     estimated docs per chunk : 70502

    Shard shard_4 at shard_4/mongo-12.2816.mongodbdns.com:270                                                                              01,mongo-13.2816.mongodbdns.com:27001,mongo-17.2816.                                                                              mongodbdns.com:27002,mongo-6.2816.mongodbdns.com:27003
     data : 107.92MiB docs : 211440 chunks : 3
     estimated data per chunk : 35.97MiB
     estimated docs per chunk : 70480

    Shard shard_5 at shard_5/mongo-17.2816.mongodbdns.com:270                                                                              01,mongo-18.2816.mongodbdns.com:27001,mongo-19.2816.                                                                              mongodbdns.com:27000
     data : 728.64MiB docs : 1423913 chunks : 4
     estimated data per chunk : 182.16MiB
     estimated docs per chunk : 355978

    Shard shard_6 at shard_6/mongo-10.2816.mongodbdns.com:270                                                                              01,mongo-14.2816.mongodbdns.com:27005,mongo-3.2816.m                                                                              ongodbdns.com:27001,mongo-8.2816.mongodbdns.com:27003
     data : 107.52MiB docs : 211169 chunks : 3
     estimated data per chunk : 35.84MiB
     estimated docs per chunk : 70389

    Shard shard_7 at shard_7/mongo-17.2816.mongodbdns.com:270                                                                              00,mongo-18.2816.mongodbdns.com:27000,mongo-19.2816.                                                                              mongodbdns.com:27003,mongo-9.2816.mongodbdns.com:27003
     data : 107.87MiB docs : 211499 chunks : 3
     estimated data per chunk : 35.95MiB
     estimated docs per chunk : 70499

    Shard shard_8 at shard_8/mongo-19.2816.mongodbdns.com:270                                                                              02,mongo-4.2816.mongodbdns.com:27002,mongo-8.2816.mo                                                                              ngodbdns.com:27001,mongo-9.2816.mongodbdns.com:27001
     data : 107.83MiB docs : 211154 chunks : 3
     estimated data per chunk : 35.94MiB
     estimated docs per chunk : 70384

    Shard shard_9 at shard_9/mongo-10.2816.mongodbdns.com:270                                                                              02,mongo-11.2816.mongodbdns.com:27003,mongo-12.2816.                                                                              mongodbdns.com:27002,mongo-13.2816.mongodbdns.com:27002
     data : 107.84MiB docs : 211483 chunks : 3
     estimated data per chunk : 35.94MiB
     estimated docs per chunk : 70494

    Totals
     data : 1.69GiB docs : 3396088 chunks : 32
     Shard shard_0 contains 8.29% data, 8.29% docs in cluster, avg obj size on shard :                                                                               535B
     Shard shard_1 contains 6.2% data, 6.21% docs in cluster, avg obj size on shard : 5                                                                              34B
     Shard shard_2 contains 6.2% data, 6.21% docs in cluster, avg obj size on shard : 5                                                                              34B
     Shard shard_3 contains 6.22% data, 6.22% docs in cluster, avg obj size on shard :                                                                               535B
     Shard shard_4 contains 6.22% data, 6.22% docs in cluster, avg obj size on shard :                                                                               535B
     Shard shard_5 contains 42% data, 41.92% docs in cluster, avg obj size on shard : 5                                                                              36B
     Shard shard_6 contains 6.19% data, 6.21% docs in cluster, avg obj size on shard :                                                                               533B
     Shard shard_7 contains 6.21% data, 6.22% docs in cluster, avg obj size on shard :                                                                               534B
     Shard shard_8 contains 6.21% data, 6.21% docs in cluster, avg obj size on shard :                                                                               535B
     Shard shard_9 contains 6.21% data, 6.22% docs in cluster, avg obj size on shard : 534B

I have 150+ similar collections where I have divided data by user_id's

e.g. MyCollection_1_100000
MyCollection_100001_200000
MyCollection_200001_300000

Here I have divided data of user id's ranging from 1 to 100000 in MyCollection_1_100000 likewise for other collections

shard key for all 150+ collection is sequential number but it is hashed. Applied by following way

db.MyCollection_1_100000.ensureIndex({"column": "hashed"})
sh.shardCollection("dbName.MyCollection_1_100000", { "column": "hashed" })

Please suggest me corrective steps to get rid of unbalanced shard problem.

Upvotes: 4

Views: 1703

Answers (1)

Gerald Mücke
Gerald Mücke

Reputation: 11132

Unshared Collections

Shard 5 is the primary shard in your cluster, which means it will take all unsharded collections and therefore grows bigger in size. You should check for that. See here.

Chunk Split

As Markus pointed out, distribution is done by chunk and not by documents. Chunks may grow up to their defined chunk size. When they exceed the chunk size they are split and redistributed. In your case there seems to be at least one collection that has 1 additional chunk than all the other shards. The reason could be that either the chunk has not yet reached it's chunk limit (check db.settings.find( { _id:"chunksize" }) default size is 64MB, see also here) or that the chunk can not be split because the range represented by the chunk can not be further split automatically. You should check the ranges using the sh.status(true) command (the output of the ranges is omitted for some collections in the large output you posted) However you may split the chunk manually. There is also a quite good answer on the dba forum.

Shard Key

If you have no unsharded collections, the problem may be the shard key itself. Mongo suggest to use a shard key with high cardinality and a high degree of randomness. Without knowing the value range of your columns, I assume the cardinality is rather low (i.e. 1000 columns) compared to, lets say a timestamp (1 for every single entry, making up to a LOT of different values).

Further, the data should be evenly distributed. So lets say you have 10 possible columns. But there are a lot more entries with a particular value for the column name all that entries would be written to the same shard. For example

  • entries.count({column: "A"} = 10 -> shard 0
  • entries.count({column: "B"} = 10 -> shard 1
  • ...
  • entries.count({column: "F"} = 100 -> shard 5

The sh.status() command should give you some more information about the chunks.

If you use the object id or a timestamp - which are values that are monotonically increasing - will lead to data being written to the same chunk as well. So Mongo suggests to use a compound key which will lead to a higher cardinality (value-range of field1 x value-range of field2). In your case you could combine the column name with a timestamp.

But either way, you're out of luck for your current installation, as you can not change the shard key afterwards.

DB Design

The verbose output you printed also indicates, you have several dbs/collections with same schema or purpose which occur to me to be sort of manually partitioned. Is there a particular reason for this? This could have an effect on the distribution of the data in the cluster as well as every collection start to be filled on the primary node. There is at least one collection with just a single chunk in the primary, and some with 3 or 4 chunks in total, all having at least one chunk on the primary (i.e. the z_best_times_*). Preferrably you should only have a single collection for one purpose and probably use a compound shard key (i.e. hashed timestamp in addition).

Upvotes: 6

Related Questions