CODEWITHSUNDEEP
pythonhyperopt
bleand
bleand

Reputation: 376

How to find the source of a MemoryError in Python?

I'm running a hyperparameter optimization using Hyperopt for a Neural Network. While doing so, after some iterations, I get a MemoryError exception

So far, I tried clearing all variables after they had been used (assigning None or empty lists to them, is there a better way for this?) and printing all locals(), dirs() and globals() with their sizes, but those counts never increase and the sizes are quite small.

The structure looks like this:

def create_model(params):
    ## load data from temp files
    ## pre-process data accordingly
    ## Train NN with crossvalidation clearing Keras' session every time
    ## save stats and clean all variables (assigning None or empty lists to them)

def Optimize():
    for model in models: #I have multiple models
        ## load data
        ## save data to temp files
        trials = Trials()
        best_run = fmin(create_model,
                        space,
                        algo=tpe.suggest,
                        max_evals=100,
                        trials=trials)

After X number of iterations (sometimes it completes the firsts 100 and shifts to the second model) it throws a memory error. My guess is that some variables remain in memory and I'm not clearing them, but I wasn't able to detect them.

EDIT:

Traceback (most recent call last):
  File "Main.py", line 32, in <module>
    optimal = Optimize(training_sets)
  File "/home/User1/Optimizer/optimization2.py", line 394, in Optimize
    trials=trials)
  File "/usr/local/lib/python3.5/dist-packages/hyperopt/fmin.py", line 307, in fmin
    return_argmin=return_argmin,
  File "/usr/local/lib/python3.5/dist-packages/hyperopt/base.py", line 635, in fmin
    return_argmin=return_argmin)
  File "/usr/local/lib/python3.5/dist-packages/hyperopt/fmin.py", line 320, in fmin
    rval.exhaust()
  File "/usr/local/lib/python3.5/dist-packages/hyperopt/fmin.py", line 199, in exhaust
    self.run(self.max_evals - n_done, block_until_done=self.async)
  File "/usr/local/lib/python3.5/dist-packages/hyperopt/fmin.py", line 173, in run
    self.serial_evaluate()
  File "/usr/local/lib/python3.5/dist-packages/hyperopt/fmin.py", line 92, in serial_evaluate
    result = self.domain.evaluate(spec, ctrl)
  File "/usr/local/lib/python3.5/dist-packages/hyperopt/base.py", line 840, in evaluate
    rval = self.fn(pyll_rval)
  File "/home/User1/Optimizer/optimization2.py", line 184, in create_model
    x_train, x_test = x[train_indices], x[val_indices]
MemoryError

Upvotes: 1

Views: 418

Answers (1)

bleand
bleand

Reputation: 376

It took me a couple of days to figure this out so I'll answer my own question to save whoever encounters this issue some time.

Usually, when using Hyperopt for Keras, the suggested return of the create_model function is something like this:

return {'loss': -acc, 'status': STATUS_OK, 'model': model}

But in large models, with many evaluations, you don't want to return and save in memory every model, all you need is the set of Hyperparameters that gave the lowest loss

By simply removing the model from the returned dict, the issue of memory increasing with each evaluation is resolved.

return {'loss': -acc, 'status': STATUS_OK}

Upvotes: 2

Related Questions