CODEWITHSUNDEEP
c#c#-4.0redisstackexchange.redis
Neo
Neo

Reputation: 16219

How to convert Redis Value[] into Deserialize object c#?

I'm working on list sorting in webjob. It is working fine when i used c# List. But to increase performance i'm saving my data into Redis cache in the form of list.

my final target is only take out most recent last 5 minutes data.

working c# code -

public class MyObject
    {
        public uint InstrumentID { get; set; }
        public decimal Close { get; set; }
        public decimal High { get; set; }
        public decimal Low { get; set; }
        public decimal Open { get; set; }
        public DateTime TimeStamp { get; set; }
        public uint Volume { get; set; }

        public DateTime Created { get; } = DateTime.Now;
        public DateTime Ttl { get; } = DateTime.Now.AddMinutes(5);
        public DateTime? Persisted { get; set; }

        public bool IsDead => DateTime.Now > Ttl;
        public bool IsPersisted => Persisted.HasValue;
        public bool TimeToPersist => IsPersisted == false && DateTime.Now > Created.AddMinutes(5);

        public DateTime GetStartOfPeriodByMins(int numMinutes)
        {
            int oldMinutes = TimeStamp.Minute;
            int newMinutes = (oldMinutes / numMinutes) * numMinutes;

            DateTime startOfPeriod = new DateTime(TimeStamp.Year, TimeStamp.Month, TimeStamp.Day, TimeStamp.Hour, newMinutes, 0);

            return startOfPeriod;
        }
    }

var inputList = new SortedSet<MyObject>(new MyObjectComparer());

            inputList.Add(new MyObject() { TimeStamp = DateTime.Now, Open = 9, High = 12, Low = 8, Close = 11, InstrumentID = 2526 });
            Thread.Sleep(10000);
            inputList.Add(new MyObject() { TimeStamp = DateTime.Now, Open = 9, High = 12, Low = 8, Close = 11, InstrumentID = 2526 });
            Thread.Sleep(10000);
            inputList.Add(new MyObject() { TimeStamp = DateTime.Now, Open = 9, High = 12, Low = 8, Close = 11, InstrumentID = 2526 });
            Thread.Sleep(50000);
            inputList.Add(new MyObject() { TimeStamp = DateTime.Now, Open = 9, High = 12, Low = 8, Close = 11, InstrumentID = 2526 });

            var resultSet = inputList
                .GroupBy(i => i.GetStartOfPeriodByMins(5))
                .Select(gr =>
               new
               {
                   StartOfPeriod = gr.Key,
                   Min = gr.Min(item => item.Open),
                   Max = gr.Max(item => item.Open),
                   Open = gr.OrderBy(item => item.TimeStamp).First().Open,
                   Close = gr.OrderBy(item => item.TimeStamp).Last().Open
               });

Now same records i'm continuously inserting into redis cache. and while tried to take last 5 minutes data I was thiking to use same GetStartOfPeriodByMins concept but it needs a list of MyObject class and redis return RedisValue[].

Redis code - using StackExchange.Redis package

var cache = RedisConnectorHelper.Connection.GetDatabase();

            //int i = 0;
            for (int i = 0; i < 10000; i++)
            {
                var tickDataHis = new MyObject()
                {
                    InstrumentID = 2526,
                    Close = 14 + i,
                    High = 16 + i,
                    Low = 11 + i,
                    Open = 12 + i,
                    TimeStamp = DateTime.Now,
                    Volume = 11111
                };

                // insert into redis
                cache.ListRightPush("tickData", JsonConvert.SerializeObject(tickDataHis));

                Thread.Sleep(3000);
            }

            var inputList = cache.ListRange("tickData");

or is there any other way to get latest 5 minutes data from redis cache?

enter image description here

Upvotes: 2

Views: 9938

Answers (1)

maradev21
maradev21

Reputation: 614

I used Redis to store timeseries data in the past. In order to optimize data retrieval, I used a sorted set (in that case there were more than one, but the concept is the same) where the score was the unix timestamp of when the data were recorded, and I serialized my data using Newtonsoft.Json library.

The code was something like this:

var myData = new MyObject() { SomeProperty = "my text" };
long dataTimestamp = DateTimeOffset.Now.ToUnixTimeMilliseconds();
string serializedData = Newtonsoft.Json.JsonConvert.Convert(myData);
redisDb.SortedSetAdd("mySortedSet", dataTimestamp, serializedData);

Doing in this way, if you want to retrieve only the data of the last 5 minutes, you can directly filter the data loaded from Redis using SortedSetRangeByScore method and passing "now - 5 minutes" as starting score, so that you can deserialize only what you need (which is of course less expensive than deserializing the whole list):

var minutesToTake = 5;
long startingTime = DateTimeOffset.Now.AddMinutes(-minutesToTake).ToUnixTimeMilliseconds();
RedisValue[] redisData = redisDb.SortedSetRangeByScore("mySortedSet", startingTime);

After that you can easily deserialize your data with the help of linq (consider that RedisValue implements operator to string conversion):

MyObject[] myData = redisData.Select(d => JsonConvert.DeserializeObject<MyObject>(d)).ToArray();

Edit: I didn't use the package suggested by @Hasan Emrah Süngü. Maybe it is more efficient, I'm just explaining what I did at that time.

Edit 2: redisDb is my StackExchange.Redis.IDatabase instance.

Edit 3: Useful reference links:

Upvotes: 3

Related Questions