CODEWITHSUNDEEP
pythonscipy
hasti
hasti

Reputation: 21

How to read/import/load many .mat files for training in python?

i want to load many .mat files in python for training. scipy.io.loadmat() loads one file! there is flow_from_directory() for .jpg files. but for .mat files what can i do? some body help me please.

Upvotes: 2

Views: 722

Answers (1)

Djib2011
Djib2011

Reputation: 7432

You can always write a generator on your own.

mat_locations = [...]  # list containing paths to all mat files

def mat_generator():
  indices = np.random.permutation(len(mat_locations))  # shuffle the data
  ind = 0
  while i < len(indices):
    yield scipy.io.loadmat(mat_locations[i])
    i+= 1

gen = mat_generator()

Then you can iterate over gen and load the mat files one-by-one. Depending on keys your mat files have and the shape of the arrays you want returned you can concatenate multiple arrays together and return them as a batch.

mat_locations = [...]

def load_mat_files(list_of_mat_files):
  '''
  Loads a series of mat files from a list and returns them as a batch
  '''
  out = []
  for file in list_of_mat_files:
    mat = scipy.io.loadmat(file)
    processed_mat = ...  # do your magic here
    out.append(processed_mat)
  return np.stack(out)

def mat_generator(batch_size=128):
  '''
  Generator that loads 
  '''
  indices = np.random.permutation(len(mat_locations))  # shuffle the data
  ind = 0
  while ind < len(indices):
    yield load_mat_files(mat_locations[ind:ind+batch_size])
    ind += batch_size

gen = mat_generator()

Now gen can be iterated over and return batches of data. You can train your model like this:

for e in range(epochs):

  gen = mat_generator()  # initialize the generator at each epoch

  for x in gen:

    # train model

Upvotes: 1

Related Questions