CODEWITHSUNDEEP
pythoncsvkeraslstm
Jishnu Vijayan
Jishnu Vijayan

Reputation: 103

python - Reading multiple CSVs for Keras LSTM

I'm trying to implement a LSTM network using Keras but I'm having problems with taking input. My dataset is in the form of multiple CSV files (all files have same dimensions 68x250 with each entry containing 2 values). There are about 200 CSV files, between various classes. Preview of one of the CSVs

How do i take these multiple CSVs as input?

Upvotes: 1

Views: 1113

Answers (2)

ixeption
ixeption

Reputation: 2060

I did something similar recently, as Pedro said you shoudl use fit_generator and write your custom generator.

Here is an example of a generator:

def generator(files):
    print('start generator')
    while 1:        
        print('loop generator')
        for file in files:
            try:                 
                df = pd.read_csv(file)
                batches = int(np.ceil(len(df)/batch_size))      
                for i in range(0, batches):                                   
                    yield pad_batch(df[i*batch_size:min(len(df), i*batch_size+batch_size)])

            except EOFError:
                print("error" + file)

Where you pass the list of filename to the generator and it iterates through the files and returns the content in batches. load_data is in my case a function which reads csvs in pandas and does some preprocessing. pad_batch does the padding for the LSTM.

Usage:

model.fit_generator(
      generator=generator(trainingFiles),   
      steps_per_epoch=steps,
      epochs=num_epochs,
      validation_data=[x_test, y_test],
      verbose=1)

Upvotes: 2

Pedro Marques
Pedro Marques

Reputation: 2682

Define a class that implements the interface defined in: https://keras.io/utils/#sequence

and use the method model.fit_generator.

Upvotes: 0

Related Questions