CODEWITHSUNDEEP
c#.netc#-4.0azure-cosmosdb
Neo
Neo

Reputation: 16219

How to get exact latest 5 minutes data from cosmos db in C#

I have a webjob which run continuously and read data from web socket api.

Below is the code which automatically run after every 1 second and add tick data into cosmos db.

private static void OnTick(Tick TickData)
        {

            var latestTickData = new MyObject()
            {
                InstrumentID = TickData.InstrumentToken,
                Close = TickData.LastPrice,
                High = TickData.LastPrice,
                Low = TickData.LastPrice,
                Open = TickData.LastPrice,
                TimeStamp = TickData.Timestamp.HasValue ? TickData.Timestamp.Value : DateTime.Now
            };

            // add data into cosmos

            Task.Run(() =>
            {
                Program.documentClient.CreateDocumentAsync(UriFactory.CreateDocumentCollectionUri("tickerDatabase", "tickerContainer"), latestTickData);
            }).Wait();
        }

Now I want to read most recent 5 minutes data and take open high low close values.

For latest 5 minutes data currently I'm running a timer job every 5 minutes which read data from cosmos and calculate open high low close but issue is time here.

If timer job runs 1 minute late also the widow of that data will change and getting wrong values.

My question is, how to get exact 5 minutes latest data from cosmos?

Current timer job code -

myobject.cs

public class MyObject
    {
        public uint InstrumentID { get; set; }
        public decimal Close { get; set; }
        public decimal High { get; set; }
        public decimal Low { get; set; }
        public decimal Open { get; set; }
        public DateTime TimeStamp { get; set; }
        public uint Volume { get; set; }

        public DateTime GetStartOfPeriodByMins(int numMinutes)
        {
            int oldMinutes = TimeStamp.Minute;
            int newMinutes = (oldMinutes / numMinutes) * numMinutes;

            DateTime startOfPeriod = new DateTime(TimeStamp.Year, TimeStamp.Month, TimeStamp.Day, TimeStamp.Hour, newMinutes, 0);

            return startOfPeriod;
        }
    }

myfunction.cs

public static void ExecuteProcess([TimerTrigger("0 */5 * * * *")] TimerInfo myTimer)
        {
            var option = new FeedOptions { EnableCrossPartitionQuery = true };
            var queryable = Program.documentClient.CreateDocumentQuery<MyObject>
            (UriFactory.CreateDocumentCollectionUri("tickerDatabase", "tickerContainer"), option).ToList();


            var resultSet = queryable.GroupBy(i => i.GetStartOfPeriodByMins(5))
               .Select(gr =>
              new
              {
                  StartOfPeriod = gr.Key,
                  Low = gr.Min(item => item.Low),
                  High = gr.Max(item => item.High),
                  Open = gr.OrderBy(item => item.TimeStamp).First().Open,
                  Close = gr.OrderBy(item => item.TimeStamp).Last().Close
              });

            var my5min = resultSet.LastOrDefault();

            Console.WriteLine("time " + my5min.StartOfPeriod + " open " + my5min.Open + " high " + my5min.High + " low " + my5min.Low + " close " + my5min.Close);

Let me explain issue with 1 minutes data (though I originally want 5 minute).

Below are the sample records -

List<MyObject> test = new List<MyObject>();
            test.Add(new MyObject() { Open = 2939, High = 2939, Low = 2939, Close = 2939, TimeStamp = new DateTime(2020, 10, 15, 10, 01, 01) });
            test.Add(new MyObject() { Open = 2933, High = 2933, Low = 2933, Close = 2933, TimeStamp = new DateTime(2020, 10, 15, 10, 01, 01) });
            test.Add(new MyObject() { Open = 2936, High = 2936, Low = 2936, Close = 2936, TimeStamp = new DateTime(2020, 10, 15, 10, 01, 03) });
            test.Add(new MyObject() { Open = 2944, High = 2944, Low = 2944, Close = 2944, TimeStamp = new DateTime(2020, 10, 15, 10, 01, 05) });
            test.Add(new MyObject() { Open = 2944, High = 2944, Low = 2944, Close = 2944, TimeStamp = new DateTime(2020, 10, 15, 10, 01, 08) });
            test.Add(new MyObject() { Open = 2939, High = 2939, Low = 2939, Close = 2939, TimeStamp = new DateTime(2020, 10, 15, 10, 01, 10) });
            test.Add(new MyObject() { Open = 2939, High = 2939, Low = 2939, Close = 2939, TimeStamp = new DateTime(2020, 10, 15, 10, 01, 15) });
            test.Add(new MyObject() { Open = 2932, High = 2932, Low = 2932, Close = 2932, TimeStamp = new DateTime(2020, 10, 15, 10, 01, 25) });
            test.Add(new MyObject() { Open = 2939, High = 2939, Low = 2939, Close = 2939, TimeStamp = new DateTime(2020, 10, 15, 10, 01, 26) });
            test.Add(new MyObject() { Open = 2939, High = 2939, Low = 2939, Close = 2939, TimeStamp = new DateTime(2020, 10, 15, 10, 01, 28) });
            test.Add(new MyObject() { Open = 2932, High = 2932, Low = 2932, Close = 2932, TimeStamp = new DateTime(2020, 10, 15, 10, 01, 30) });
            test.Add(new MyObject() { Open = 2941, High = 2941, Low = 2941, Close = 2941, TimeStamp = new DateTime(2020, 10, 15, 10, 01, 32) });
            test.Add(new MyObject() { Open = 2939, High = 2939, Low = 2939, Close = 2939, TimeStamp = new DateTime(2020, 10, 15, 10, 01, 35) });
            test.Add(new MyObject() { Open = 2941, High = 2941, Low = 2941, Close = 2941, TimeStamp = new DateTime(2020, 10, 15, 10, 01, 40) });
            test.Add(new MyObject() { Open = 2937, High = 2937, Low = 2937, Close = 2937, TimeStamp = new DateTime(2020, 10, 15, 10, 01, 42) });
            test.Add(new MyObject() { Open = 2939, High = 2939, Low = 2939, Close = 2939, TimeStamp = new DateTime(2020, 10, 15, 10, 01, 45) });
            test.Add(new MyObject() { Open = 2937, High = 2937, Low = 2937, Close = 2937, TimeStamp = new DateTime(2020, 10, 15, 10, 01, 48) });
            test.Add(new MyObject() { Open = 2939, High = 2939, Low = 2939, Close = 2939, TimeStamp = new DateTime(2020, 10, 15, 10, 01, 50) });
            test.Add(new MyObject() { Open = 2939, High = 2939, Low = 2939, Close = 2939, TimeStamp = new DateTime(2020, 10, 15, 10, 01, 52) });
            test.Add(new MyObject() { Open = 2937, High = 2937, Low = 2937, Close = 2937, TimeStamp = new DateTime(2020, 10, 15, 10, 01, 54) });
            test.Add(new MyObject() { Open = 2935, High = 2935, Low = 2935, Close = 2935, TimeStamp = new DateTime(2020, 10, 15, 10, 01, 56) });

            test.Add(new MyObject() { Open = 2935, High = 2935, Low = 2935, Close = 2935, TimeStamp = new DateTime(2020, 10, 15, 10, 02, 12) });

Passed 1 minutes as a parameter to GetStartOfPeriodByMins(1) -

Here records are for time 10:1:56 and 10:2:12.

Now you can observe last we will get 2 record set into resultSet

If timer trigger function is not run at specific time it will only take average of last record for 10:2:12 and it is not right

So question is how to match running time?

Same will happen with 5 minutes as well.

Output of above sample data for 1 minutes

enter image description here

So we need to ensure data should be complete from that timeframe.

Upvotes: 1

Views: 1229

Answers (1)

Harshita Singh
Harshita Singh

Reputation: 4870

One possible solution here can be to leverage _ts property of documents. You can directly have a SQL query with start date and end date with 5 minutes span. In addition to this, you can save start date as last used end date in DB(negligible cost). The query shall look like this:

SELECT * FROM c where c._ts <= 1601890740 AND c._ts >= 1601890585

Also note that, you will have to do some to and fro conversions of POSIX to DateTime.

Upvotes: 1

Related Questions