CODEWITHSUNDEEP
csvpython-3.xdictionaryiteration
Peter S
Peter S

Reputation: 575

Iterate over . csv file --> write into two dictionaries

The csv looks like this:

2016-04-01 02:17    16.7    51.9
2016-04-01 02:18    16.7    51.9
2016-04-01 02:19    16.6    52.0
2016-04-01 02:20    16.6    52.2
2016-04-01 02:21    16.7    52.2
2016-04-01 02:22    16.7    52.1
2016-04-01 02:23    16.6    52.2
2016-04-01 02:24    16.6    52.2
2016-04-01 02:25    16.6    52.2

The File logs Temp and Humidity from a Sensor every Minute. Every Measurement is a row in the cvsfile.

I'd like to iterate over this file and add rows, depending on the time to dictionaries. So one dictionary should hold all rows from nowuntil now - 24hours and the other one should hold all rows from now until now - 1week. Nowis the current time, the script was started. I'm not certain on how to do that with on iteration (if possible). This is how I open the file and define the TimeRow:

for row in f_reader:
    stringRowDate = row[0]
    Date = datetime.datetime.strptime(stringRowDate,"%Y-%m-%d %H:%M")
    stringDateNow = datetime.datetime.now().strftime("%Y-%m-%d %H:%M")
    DateNow = datetime.datetime.strptime(stringDateNow,"%Y-%m-%d %H:%M")

My Code with the csvFile open:

if Date >= DateNow:
    continue
elif Date >= (DateNow - timedelta(weeks=1)):
    data_Week = {row[0]:row[1:] for row in f_reader}
rownum +=1

data_Weekis the Dictionary that now holds all rows from the file from now to 1 Week before now.

Can I, by using the same Iteration add every row within 24hoursto another Dictionary (data_24)?

Upvotes: 0

Views: 42

Answers (1)

Tadhg McDonald-Jensen
Tadhg McDonald-Jensen

Reputation: 21463

You could just check each line individually instead of sending all the rest of the data into one dict:

week_ago = (DateNow - timedelta(weeks=1))
day_ago = (DateNow - timedelta(days=1))
data_Week = {}
data_24 = {}
#data_before = {}

for row in f_reader:
    stringRowDate = row[0]
    Date = datetime.datetime.strptime(stringRowDate,"%Y-%m-%d %H:%M")
    if Date >= day_ago:
        data_24[row[0]] = row[1:]
    elif Date >= week_ago:
        data_Week[row[0]] = row[1:]
    else:
        continue
        #data_before[row[0]] = row[1:]

although this means that a lot of conditionals are checked unnecessarily if the file is conveniently sorted, in which case you can just keep track of the current split_time to indicate when to switch dict and current_data which will point to one of the dicts to put the dates into:

#first is the value to signal to go to next group
data_sorter = iter([(week_ago,None), #None placeholder to not keep data from before week_ago
                    (day_ago,data_Week),
                    (None,data_24)]) #no split for last one, just take the rest of data until end of reader
split_time,current_data = next(data_sorter)
for row in f_reader:
    stringRowDate = row[0]
    Date = datetime.datetime.strptime(stringRowDate,"%Y-%m-%d %H:%M")
    if split_time is not None and (Date > split_time):
        split_time,current_data = next(data_sorter)
    if current_data is not None: #make sure there is no placeholder
        current_data[row[0]] = row[1:]

Depending what else is involved in your program it might be more convenient to use a generator so that there is no need for placeholders:

from operator import lt #used only for the default check_f
def split_by_time(reader,*split_times, check_f=lt):
    split_times = iter(split_times)
    split_time = next(split_times,None)
    current_data = {}
    for line in reader:
        row_time,*row_data = line
        if split_time is not None and check_f(split_time,row_time):
            yield current_data
            split_time = next(split_times,None)
            current_data = {}
        current_data[row_time] = row_data
    yield current_data


def check_limit(split_date, date_from_file):
    "convert the string date from file to a datetime object for comparison"
    return split_date < datetime.datetime.strptime(date_from_file,"%Y-%m-%d %H:%M")

times = list(split_by_time(f_reader,
                        week_ago,day_ago, 
                        check_f=check_limit))

print("from before a week ago:\n",times[0])
print("from a week ago to a day ago:\n",times[1])
print("since yesterday:\n",times[2])

but that's probably more then you really need, I just really like generators. :D

Upvotes: 1

Related Questions