Reputation: 1249
Advice on storing this data (JSON) in Redis without overwriting existing?
I'm new to ALL aspects of what I am about to ask.
I'm trying to determine if I can successfully store some polled metrics which come as JSON data in Redis.
In one way I have managed to do it already. The following python script will poll the device, pull back the JSON data and then using (RedisJSON - https://redislabs.com/blog/redis-as-a-json-store/ ) I can push that json data to Redis.
Here is the code:
# !/usr/bin/env python
from __future__ import print_function
from functools import wraps
from pprint import pprint
import time
import sys
import acos_client as acos
import redis
import json
c = acos.Client('10.10.10.10', acos.AXAPI_21, 'username', 'password')
data = c.slb.service_group.stats('SG_SITE1.BUSINESS.COM_443')
# Open a connection to redis
r = redis.StrictRedis()
# Store the document
r.execute_command('JSON.SET', 'data', '.', json.dumps(data))
# Retrieve it
reply = json.loads(r.execute_command('JSON.GET', 'data'))
This is the data that is pulled and then stored in Redis:
{u'service_group_stat': {u'cur_conns': 344,
'date': '04-25-2017',
u'member_stat_list': [{u'cur_conns': 66,
u'req_bytes': 1476212423,
u'req_pkts': 10449342,
u'resp_bytes': 33132743858,
u'resp_pkts': 25652317,
u'server': u'WWW0006',
u'tot_conns': 172226},
{u'cur_conns': 64,
u'req_bytes': 1666275823,
u'req_pkts': 11982676,
u'resp_bytes': 37575461036,
u'resp_pkts': 29175599,
u'server': u'WWW0005',
u'tot_conns': 205244},
{u'cur_conns': 89,
u'req_bytes': 1671222671,
u'req_pkts': 11940864,
u'resp_bytes': 37064038202,
u'resp_pkts': 28747313,
u'server': u'WWW0004',
u'tot_conns': 195789},
{u'cur_conns': 37,
u'req_bytes': 94117510958,
u'req_pkts': 585916896,
u'resp_bytes': 1860691638618,
u'resp_pkts': 1439228725,
u'server': u'WWW0003',
u'tot_conns': 7366402},
{u'cur_conns': 42,
u'req_bytes': 98580368121,
u'req_pkts': 642797814,
u'resp_bytes': 1934241923560,
u'resp_pkts': 1498242871,
u'server': u'WWW0002',
u'tot_conns': 7221995},
{u'cur_conns': 46,
u'req_bytes': 94886760323,
u'req_pkts': 593577169,
u'resp_bytes': 1863028601218,
u'resp_pkts': 1441197389,
u'server': u'WWW0001',
u'tot_conns': 7260787}],
u'name': u'SG_SITE1.BUSINESS.COM_443',
u'req_bytes': 292398350319,
u'req_pkts': 1856664761,
u'resp_bytes': 5765734406492,
u'resp_pkts': 4462244214,
'time': '11:00:00',
u'tot_conns': 22422443}}
When I check for "data" in Redis I see the data/json that was pushed:
127.0.0.1:6379> json.get data
"{\"service_group_stat\":{\"status\":3,\"protocol\":2,\"name\":\"SG_SITE1.BUSINESS.COM_443\",\"cur_reqs\":0,\"resp_pkts\":4472667457,\"cur_conns\":306,\"total_reqs\":0,\"total_reqs_succ\":0,\"req_pkts\":1860880754,\"req_bytes\":293088668161,\"member_stat_list\":[{\"status\":1,\"resp_bytes\":35542787582,\"cur_reqs\":0,\"resp_pkts\":27513984,\"cur_conns\":72,\"total_reqs\":0,\"server\":\"WWW0006\",\"req_pkts\":11202756,\"req_bytes\":1601613790,\"total_reqs_succ\":0,\"tot_conns\":181675,\"port\":81},{\"status\":1,\"resp_bytes\":39991941399,\"cur_reqs\":0,\"resp_pkts\":31033672,\"cur_conns\":45,\"total_reqs\":0,\"server\":\"WWW0005\",\"req_pkts\":12730038,\"req_bytes\":1788901352,\"total_reqs_succ\":0,\"tot_conns\":214379,\"port\":81},{\"status\":1,\"resp_bytes\":38939383008,\"cur_reqs\":0,\"resp_pkts\":30207188,\"cur_conns\":41,\"total_reqs\":0,\"server\":\"WWW0004\",\"req_pkts\":12545969,\"req_bytes\":1770117987,\"total_reqs_succ\":0,\"tot_conns\":203326,\"port\":81},{\"status\":1,\"resp_bytes\":1863483461334,\"cur_reqs\":0,\"resp_pkts\":1441396536,\"cur_conns\":34,\"total_reqs\":0,\"server\":\"WWW0003\",\"req_pkts\":586827719,\"req_bytes\":94258771440,\"total_reqs_succ\":0,\"tot_conns\":7378462,\"port\":81},{\"status\":1,\"resp_bytes\":1935723181356,\"cur_reqs\":0,\"resp_pkts\":1499389206,\"cur_conns\":48,\"total_reqs\":0,\"server\":\"WWW0002\",\"req_pkts\":643286283,\"req_bytes\":98658443365,\"total_reqs_succ\":0,\"tot_conns\":7228301,\"port\":81},{\"status\":1,\"resp_bytes\":1865535919223,\"cur_reqs\":0,\"resp_pkts\":1443126871,\"cur_conns\":66,\"total_reqs\":0,\"server\":\"WWW0001\",\"req_pkts\":594287989,\"req_bytes\":95010820227,\"total_reqs_succ\":0,\"tot_conns\":7271565,\"port\":81},{\"status\":0,\"resp_bytes\":0,\"cur_reqs\":0,\"resp_pkts\":0,\"cur_conns\":0,\"total_reqs\":0,\"server\":\"WWW03\",\"req_pkts\":0,\"req_bytes\":0,\"total_reqs_succ\":0,\"tot_conns\":0,\"port\":81},{\"status\":0,\"resp_bytes\":0,\"cur_reqs\":0,\"resp_pkts\":0,\"cur_conns\":0,\"total_reqs\":0,\"server\":\"WWW01\",\"req_pkts\":0,\"req_bytes\":0,\"total_reqs_succ\":0,\"tot_conns\":0,\"port\":81},{\"status\":0,\"resp_bytes\":0,\"cur_reqs\":0,\"resp_pkts\":0,\"cur_conns\":0,\"total_reqs\":0,\"server\":\"WWW02\",\"req_pkts\":0,\"req_bytes\":0,\"total_reqs_succ\":0,\"tot_conns\":0,\"port\":81}],\"time\":\"13:20:57\",\"date\":\"04-25-2017\",\"resp_bytes\":5779216673902,\"tot_conns\":22477708}}"
So far so good it would appear?
The issue is I'm not sure how to properly handle pushing the next set of data without overwriting the existing "data"?
I realize that this is what saves it to redis as "data":
r.execute_command('JSON.SET', 'data', '.', json.dumps(data))
The goal is to have multiple sets of this data as it includes date and time.
I'm hoping to use this for storing one hour of these metrics pulled every 5 minutes for each of the "service_groups" and the servers included with them.
So I am very interested in learning and understanding how I might go about accomplishing this?
** Additional Details / Responses to Questions **
The ultimate goal is to be able to use some sort of graphing / reporting tool that could access the data and show the metrics for "service_group" in a given time period as well as the ability to expand the "service_group" and see the metrics associated with each member/server within that "service_group".
Upvotes: 0
Views: 2111
Answers (1)
Reputation: 49972
Every key in Redis (data) is a data structure, and in your case the data structure is a JSON document. There are at least two ways that you can "save multiple sets": 1) use multiple keys and/or 2) use JSON nesting inside a key.
Redis can hold a lot of keys (2^32-1 to be exact). You can have each of your JSON documents be stored under a different key. The key name should be unique, so a combination of the name and the timestamp could do it, i.e.:
...
name = 'SG_SITE1.BUSINESS.COM_443'
data = c.slb.service_group.stats(name)
...
keyname = '{}:{}'.format(name, data['service_group_stat']['date'])
r.execute_command('JSON.SET', keyname, '.', json.dumps(data))
...
Using the other way, namely use one key (e.g. data or SG_SITE1.BUSINESS.COM_443) but nesting multiple documents in it, could be implemented for example if you think about the document as an object where each key is the date perhaps, i.e.:
{'04-25-2017':
{u'service_group_stat': {u'cur_conns': 344,
'date': '04-25-2017',
...
},
'04-24-2017':
{u'service_group_stat': {u'cur_conns': 344,
'date': '04-24-2017',
...
So basically, it would be something like:
...
name = 'SG_SITE1.BUSINESS.COM_443'
data = c.slb.service_group.stats(name)
...
path = '.["{}"]'.format(data['service_group_stat']['date'])
r.execute_command('JSON.SET', name, path, json.dumps(data))
...
Of course, the real question is what do you want to do with the data once it's in place ;)
P.S. my apologies for my delay over email - this is a much better medium to discuss the topic because it may benefit future generations :P
Upvotes: 2
Related Questions
- How to store a Complex Nested JSON in Redis using Python
- How should I store JSON in redis?
- how to store a complex object in redis (using redis-py)
- Storing multiple nested objects in redis
- How to structure python code when storing redis hashes and json
- How to save complicated data structure in Redis?
- redis json best practice
- How would I store this data in redis properly?
- Advice on best data structure to use in Redis?
- How should I store this object in Redis?