CODEWITHSUNDEEP
redisspring-data-redis
achowdhury
achowdhury

Reputation: 21

Any alternate search option other that SCAN when search is based on key pattern

I am using redis as database. Here is the example of key and value I am using. Both key and value are of type string. Key - shipping_rate:64:MNR:Home-Delivery. It is combination of 3 fields like item_id (64), item_type (MNR) , delivery service (Home-Delivery). And I am appending service name (shipping_rate) before the key.

The values is a json string - {"PRD_CAT_NAME" : "PTO PARTS", "SHIPMENT_CHG" : "20.9", "ITEM_CHG" : "12.99", "CONSOLIDATION" : "N", "AREA_UPCHARGE" : "0", "MISC_UPCHARGE" : "0", "ACTIVE_FLAG" : "Y"}'

Now I want to get all the keys using the pattern like shipping_rate:64:MNR:*. Here my searching is based on the pattern only because that is the requirement. I am using the SCAN for searching. In my spring boot java code I am using ScanOptions. Then want to query redis db for the list of keys I get from the SCAN operation.

Now I want to know for this scenario is there any other better solution approach than SCAN?

I was looking for secondary indexing but is it a good idea to add secondary indexing on the key? And even if it is possible and I keep all the keys in a set (for indexing) then also I need to use pattern to search from the set and that is not possible because SMEMBERS does not work for pattern searching.

If scan is the only approach for this scenario and if my redis DB has 50 million record then how would be performance of SCAN? is there any metrics anyone aware of for this scenario

Upvotes: 1

Views: 84

Answers (4)

avifen
avifen

Reputation: 1017

Generally not a lot, for each key you have its checking the key in a hash table.
If you have 50 million keys and you know that the pattern you scan is a portion from it, send it with high count so you don't pay big prices for empty responses.
But i believe that there's ways with better performance.
In case you don't take another direction:
If you use pattern in Valkey 8 released lately there's massive improvements for scannings with pattern using the hashing function that sort keys into their slots.
Meaning the scan would not check every key, but the one fitting the hash, and will cost much less.

Upvotes: 0

Ofir Luzon
Ofir Luzon

Reputation: 10947

One option is to use ReJSON and RediSearch As Lior suggested.

Another is to index them, not sure what was your idea to index, if you'll use a set for each item_id:item_type object, something like shipping_rate:64:MNR:all, it should also eliminate the need to scan or sscan.

One alternative option, store your options in a hash instead of a string. e.g.

HSET shipping_rate:64:MNR Home-Delivery '{"PRD_CAT_NAME" : "PTO PARTS", "SHIPMENT_CHG" : "20.9", "ITEM_CHG" : "12.99", "CONSOLIDATION" : "N", "AREA_UPCHARGE" : "0", "MISC_UPCHARGE" : "0", "ACTIVE_FLAG" : "Y"}'

And depending on your usecase, you can hkeys to get the options, hgetall to get all the JSONs, hget a spesific one, or even hscan if you have many entries.

I would even consider adding another hash field that has a duplicated value of one of the methods with default name (and maybe adding the delivery method name into the JSON), just to reduce guesses and make sure you can get some data without guessing or checking. Something like:

HSET shipping_rate:64:MNR default '{"delivery_service" : "Home-Delivery", "PRD_CAT_NAME" : "PTO PARTS", "SHIPMENT_CHG" : "20.9", "ITEM_CHG" : "12.99", "CONSOLIDATION" : "N", "AREA_UPCHARGE" : "0", "MISC_UPCHARGE" : "0", "ACTIVE_FLAG" : "Y"}'

Upvotes: 0

Ritu Yadav
Ritu Yadav

Reputation: 1

An alternative to SCAN for key pattern searches in databases like Redis is the KEYS command, which can quickly return all keys matching a specific pattern. However, KEYS is not recommended for large datasets due to performance issues, as it blocks the database while searching. Another option is SSCAN, which works well with sets and allows for more efficient, incremental searches. You could also consider using ZSCAN for sorted sets or HSCAN for hash fields, depending on the data structure in use. These commands provide non-blocking, scalable searches over large datasets.

Upvotes: -1

Lior Kogan
Lior Kogan

Reputation: 20658

Instead of using strings containing JSONs, consider using JSON keys. You can then index your JSON keys and use Redis Query Engine to search for specific keys based on their content instead of their name.

Upvotes: 1

Related Questions