CODEWITHSUNDEEP
pythondictionary
tlqn
tlqn

Reputation: 399

Retrieve the values from a dictionary in a text (log) file

I have a output.log file with the following format as follows:

{'loss': 2.1327, 'learning_rate': 3.5e-05, 'epoch': 0.0}
{'loss': 2.1133, 'learning_rate': 4.9981292413606026e-05, 'epoch': 0.01}
{'loss': 2.1522, 'learning_rate': 4.994855413741657e-05, 'epoch': 0.01}
{'loss': 2.1394, 'learning_rate': 4.991581586122713e-05, 'epoch': 0.01}
{'loss': 2.1501, 'learning_rate': 4.9883077585037675e-05, 'epoch': 0.02}

I would really like to just get the values in the log file for plotting, and have it like this:

2.1327   3.5e-05                  0.0
2.1133   4.9981292413606026e-05   0.01
2.1522   4.994855413741657e-05    0.01
2.1394   4.991581586122713e-05    0.01
2.1501   4.9883077585037675e-05   0.02

I am really not sure how to tackle this or what approach should I try/ look up. Since it is a text file, I tried to import with json and convert the string to dictionary and then get the values but I received the error message:

JSONDecodeError: Expecting value: line 1 column 1 (char 0)

I tried to append the values but also received error:

X,Y = [] , [] 
    for line in open(output.log, 'r'):
        values = [float(s) for s in line.split()]
        X.append(values[2])
        Y.append(values[0])

ValueError: could not convert string to float: "{'loss':"

My final purpose is to plot the values, so maybe there could be a better way to plot them without having to get rid of the texts first? Any suggestions or direction would be greatly appreciated.

Upvotes: 0

Views: 170

Answers (1)

Krishna Chaurasia
Krishna Chaurasia

Reputation: 9572

You can use json.loads after replacing the single quotes ' with " quotes after loading each line:

import json
X,Y = [] , [] 
for line in open('log.json', 'r'):
    x = json.loads(line.replace("'", '"'))
    values = [v for v in x.values()]
    X.append(values[2])
    Y.append(values[0])
    
print(X, Y)

Output:

[0.0, 0.01, 0.01, 0.01, 0.02] [2.1327, 2.1133, 2.1522, 2.1394, 2.1501]

Upvotes: 1

Related Questions