CODEWITHSUNDEEP
mongodbsharding
Jiby Jose
Jiby Jose

Reputation: 3845

defining dynamic ranges to mongodb shard tags

I have a sharded mongodb database setup,where i am using shard tags to constraint certain data to certain shards.

What i am trying to do is give dynamic ranges for the shard tags. I want data of last 5 days be always in a shard tag and others in the other shard tag.I cant specify a fixed date here as it changes every day.

An example would be

Shard tag "LATEST" data from last 5 days to today Shard tag "OLD" data from minimum to last 5 days

How can i defined such a range on shard tag rather than change the shard tag range every day.

Thanks in Advance

Upvotes: 2

Views: 354

Answers (2)

Mark Boon
Mark Boon

Reputation: 113

The problem with trying to make shard tagging dynamic is that the tag is only honored when inserting. In your case, all new data will end up in one set of tagged shards as all data starts off as less than five days old when inserted. Once five days pass since insertion, nothing will happen. It does not provide a magical capability of deleting documents older than five days from the tagged shards and distribute them among another set of untagged shards.

Upvotes: 0

Dyin
Dyin

Reputation: 5366

In your example, with your sharding key distribution, the shard with the "latest" data will get all the new writes, as further "latest" data would be injected into it. The problem is, that you need to move the older data to the other shard manually.

Lets suppose you have a timestamp-like shard key or ObjectID. (The ObjectID gets constructed automatically with a timestamp component.) These are increasing monotonic, so the linearly balanced chucks will get unbalanced every time as you insert data into the corresponding collection.

I would not force the old or the new data to one specific shard, since sharding is for load balancing (mostly), which you would avoid in that case. You can have a nice distribution of your data with monotonic key types, since MongoDB version 2.4 with Hashed Sharding.

Learn how to Shard a Collection Using a Hashed Shard Key

Example: sh.shardCollection( "records.active", { a: "hashed" } )

Where a should be a monotonically increasing type of field.

Upvotes: 1

Related Questions