CODEWITHSUNDEEP
performanceredisinsertion
zhihui
zhihui

Reputation: 11

read redis when redis mass insertion

I have a server running all day using redis as the data storage. There is one huge data updating (nearly 10 million rows) on a specific time (3 am) every day and a lot of real time but few data (nearly 100 rows) updating at other time.

I choose the redis mass insertion mode to accelerate the data insertion, which costs 30 seconds. But at that time, the redis query performance is really bad. Any ideas to avoid this problem?

If I use redis master-slave mode to separate the read and write, and the master writes as well as the slave reads. But when the batch insertion happens on the master, and there are also a lot of data needed to be synced to slave, and I doubt that it is still the hot spot on the slave redis for querying.

Any suggestion on this kind of senario?

Thank you.

Upvotes: 1

Views: 513

Answers (1)

Tw Bert
Tw Bert

Reputation: 3809

First of all, I'd investigate where the bottleneck is. Is it network I/O? You might want to setup a dedicated route with (virtual) nic's, and even a dedicated internet connection to address that. Is it CPU? You might want to spread out the mass insertion.

If you use simple non-transactional pipelining, it should give redis time to breathe, so clients won't notice the mass insertion.

An alternative could be using client-side connections. Let the client connect to a different slave, which you promote to master temporarily. After the mass insertion is complete, you let the client connect to the 'real' slave again. You might be able to use redis sentinel for this, but with redis pub/sub you can accomplish the same result but with more control. Use a separate small redis instance with redis connection hosts/ports stored in a HSET.

Hope this helps, TW

Upvotes: 2

Related Questions