CODEWITHSUNDEEP
phpredisstoragecounterkey-value
Kirzilla
Kirzilla

Reputation: 16596

Key-value storage for counters with ability of fast dumping data

Let's imagine that, we have 300K keys with simple counters in our storage Counter key names, for example: counter1_2014-03-25_00:01 counter2_2014-03-25_00:01 counter3_2014-03-25_00:01

Each counter is collecting data for 1 minute, thus one key per one minute: counter1_2014-03-25_00:01, counter1_2014-03-25_00:02, counter1_2014-03-25_00:03 etc.

Every next minute I have to dump all counters for previous minute and remove them from storage. My current implementation is really really simple and using Redis hashes (HINCR, HGETALL, DEL).

//incrementing counter $Redis->hincr('counters_2014-03-25_00:02', 'counter1');

//dumping and removing $result = $Redis->multi() ->hgetall('counters_2014-03-25_00:02') //dumping ->del('counters_2014-03-25_00:02') //removing hash ->exec()

Everything is fine with Redis, but dumping with HGETALL is getting very and very slow with large number of counters because HGETALL have O(N) complexity.

Right now I'm looking for approach that will allow:

UPDATE: Fast dumping is required because I need to transfer all locally collected data from backend servers to master server.

Upvotes: 0

Views: 223

Answers (1)

Chhavi Gangwal
Chhavi Gangwal

Reputation: 1176

One solution can be to create simple keys, increment using INCR and each time you create a key set an expiry time with it using EXPIRE.

Upvotes: 1

Related Questions