CODEWITHSUNDEEP
pythonnumpy
Mader n fader
Mader n fader

Reputation: 146

File .npy created, which should contains a 200MB dataset, is almost empty

I'm following this tutorial and the main goal is to balance data and save them into a second training data sheet (the first one contains data not balanced). This is the code:

import numpy as np
import pandas as pd
from collections import Counter
from random import shuffle

train_data = np.load('training_data.npy')

df = pd.DataFrame(train_data)
print(df.head())
print(Counter(df[1].apply(str)))

lefts = []
rights = []
forwards = []

shuffle(train_data)

for data in train_data:
    img = data[0]
    choice = data[1]

    if choice == [1,0,0]:
        lefts.append([img,choice])
    elif choice == [0,1,0]:
        forwards.append([img,choice])
    elif choice == [0,0,1]:
        rights.append([img,choice])
    else:
        print('no matches')


forwards = forwards[:len(lefts)][:len(rights)]
lefts = lefts[:len(forwards)]
rights = rights[:len(forwards)]

final_data = forwards + lefts + rights
shuffle(final_data)

np.save('training_data_v2.npy', final_data)

I really don't understand whhy it does create a 120B file while dataset weighs 200MB..

Upvotes: 0

Views: 187

Answers (1)

Shreyas Pimpalgaonkar
Shreyas Pimpalgaonkar

Reputation: 350

So the main problem is in these three lines

forwards = forwards[:len(lefts)][:len(rights)]
lefts = lefts[:len(forwards)]
rights = rights[:len(forwards)]

you are truncating the arrays.

So to confirm the final shapes of the arrays do -

print(len(forwards),len(lefts),len(rights))
// those 3 lines
print(len(forwards),len(lefts),len(rights))

You'll see the difference.

Also, try running the code without those three lines, the arrays will be 200 MB :)

P.S. I would advise you to do truncation manually - 

forwards = forwards[:my_number]

and so on..

Upvotes: 1

Related Questions