CODEWITHSUNDEEP
pythoneventsmonitoring
maphilli14
maphilli14

Reputation: 19

Use python to count the number of events in a specified amount of time

I'm monitoring a json stream for specific events. I'd like to count the number of events that occur in a specific amount of time, say events per minute, then compare to a high water mark to take action.

for l in s.makefile() :
m = json.loads(l)
if 'Event' in m and m['Event'] == 'StarLost' :
    print 'Star Lost at time= ' + str(datetime.datetime.fromtimestamp(m['Timestamp']))

Upvotes: 0

Views: 2306

Answers (1)

Oliver W.
Oliver W.

Reputation: 13459

Keep a double-ended queue and insert and pop items from it based on the timestamp value that you get from the json stream.

import datetime
from collections import deque

threshold = 4
queue = deque()
for l in s.makefile() :
    # fill the queue
    record = json.loads(l)
    try:
        if record['Event'] == 'StarLost':
            timestamp = datetime.datetime.fromtimestamp(record['Timestamp'])
            print('Star Lost at time {}.'.format(timestamp.isoformat()))
            queue.append((timestamp, record))
    except KeyError:
        pass # there was no 'Event' in the record
    # clear the queue from old records
    try:
        while queue[0][0] < datetime.datetime.now() - datetime.timedelta(seconds=60):
            queue.popleft()
    except IndexError:
        pass # there are no records in the queue.
    # analyze the queue
    if len(queue) > threshold:
        print('There were more than {} events over the last minute!'.format(threshold))

This solution assumes you've got s.makefile() generating json data non-stop. Ideally, you'd put the part that clears the queue and the part that analyzes the queue in different threads from the thread that fills the queue, but if you can't come up with the solution above yourself, then threading is not for you now, although I gave the start for it by using the thread-safe deque. Do read up on it though if you want to make this solution better.

Upvotes: 1

Related Questions