CODEWITHSUNDEEP
pythonarraysnumpykeras
Boris
Boris

Reputation: 301

Any workaround to getting a list to numpy array for keras input data?

I'm trying to train a CNN model on keras, my data looks like this

type(datain)
<class 'list'>
len(datain)
35000
type(datain[0])
<class 'numpy.ndarray'>
datain[0].shape
(256,256,1)

And being my input data a list of arrays I get this error when trying to train the network

AttributeError: 'list' object has no attribute 'shape'

but when trying to do something like np.array(datain) as suggested here https://github.com/keras-team/keras/issues/4823 my computer freezes/crashes. defining my input using python list takes like 60 seconds in total, but if I try as numpy arrays from the beginning but takes like 1 sec per (256,256,1) array, and is way too much time if I intent of doing various test and modifications to my network,
is there any work around for this problem?
any way to use lists for keras?
a different way to define a numpy array?
or am I misunderstanding something?

Upvotes: 0

Views: 2071

Answers (1)

Daniel M&#246;ller
Daniel M&#246;ller

Reputation: 86600

Creating a generator from your data.

A generator is a python concept, it loops and yields results. For Keras, your generator should yield batches of X_train and y_train indefinitely.

So, a simple generator that you can make is:

def generator(batch_size,from_list_x,from_list_y):

    assert len(from_list_x) == len(from_list_y)
    total_size = len(from_list_x)

    while True #keras generators should be infinite

        for i in range(0,total_size,batch_size):
            yield np.array(from_list_x[i:i+batch_size]), np.array(from_list_y[i:i+batch_size])

Use the generator in training:

model.fit_generator(generator(size,datain,dataout),
                    steps_per_epoch=len(datain)//size, 
                    epochs=...,...)

Upvotes: 4

Related Questions