MemoryError with python using keras and theano

Question

I tried this keras tutorial. I'm using theano on my other project, so I changed keras to use theano and not tenorflow. But when I run this tutorial I get this error first:

nvcc fatal : Cannot find compiler 'cl.exe' in PATH

And after some time, in the first Epoch (out of 3), on the sample number 13056 (out of 25000) i get this error:

File "test_keras.py", line 28, in model.fit(X_train, y_train, validation_data=(X_test, y_test), epochs=3, batch_size=64)

File "C:\Users\domi1_000\Anaconda3\envs\Pyhon27\lib\site-packages\keras\models.py", line 845, in fit initial_epoch=initial_epoch)

File "C:\Users\domi1_000\Anaconda3\envs\Pyhon27\lib\site-packages\keras\engine\training.py", line 1485, in fit initial_epoch=initial_epoch)

File "C:\Users\domi1_000\Anaconda3\envs\Pyhon27\lib\site-packages\keras\engine\training.py", line 1140, in _fit_loop outs = f(ins_batch)

File "C:\Users\domi1_000\Anaconda3\envs\Pyhon27\lib\site-packages\keras\backend\theano_backend.py", line 1094, in call return self.function(*inputs)

File "C:\Users\domi1_000\Anaconda3\envs\Pyhon27\lib\site-packages\theano\compile\function_module.py", line 898, in call storage_map=getattr(self.fn, 'storage_map', None))

File "C:\Users\domi1_000\Anaconda3\envs\Pyhon27\lib\site-packages\theano\gof\link.py", line 325, in raise_with_op reraise(exc_type, exc_value, exc_trace)

File "C:\Users\domi1_000\Anaconda3\envs\Pyhon27\lib\site-packages\theano\compile\function_module.py", line 884, in call self.fn() if output_subset is None else\

File "C:\Users\domi1_000\Anaconda3\envs\Pyhon27\lib\site-packages\theano\scan_module\scan_op.py", line 989, in rval r = p(n, [x[0] for x in i], o)

File "C:\Users\domi1_000\Anaconda3\envs\Pyhon27\lib\site-packages\theano\scan_module\scan_op.py", line 978, in p self, node)

File "theano/scan_module/scan_perform.pyx", line 445, in theano.scan_module.scan_perform.perform (C:\Users\domi1_000\AppData\Local\Theano\compiledir_Windows-8.1-6.3.9600-Intel64_Family_6_Model_58_St epping_9_GenuineIntel-2.7.12-64\scan_perform\mod.cpp:5259)

MemoryError:

Apply node that caused the error: forall_inplace,cpu,grad_of_scan_fn}(TensorConstant{500}, Subtensor{int64:int64:int64}.0, Elemwise{tanh}.0, Alloc.0, InplaceDimShuffle{0,2,1}.0, Elemwise{Composite{(i0 - sqr(i1))}}.0, Subtensor{int64:int64:int64}.0, Subtensor{int64:int64:int64}.0, Subtensor{int64:int64:int64}.0, Alloc.0, Alloc.0, Alloc.0, TensorConstant{500}, Subtensor{::, int64:int64:}.0, Subtenso r{::, :int64:}.0, Subtensor{::, int64::}.0, Subtensor{::, int64:int64:}.0, InplaceDimShuffle{1,0}.0, InplaceDimShuffle{1,0}.0, InplaceDimShuffle{1,0}.0, Alloc.0, InplaceDimShuffle{1,0}.0)

Toposort index: 148 Inputs types: [TensorType(int64, scalar), TensorType(float32, 3D), TensorType(float32, 3D), TensorType(float32, 3D), TensorType(float32, 3D), TensorType(float32, 3D), TensorType(float32, 3D), TensorTy pe(float32, 3D), TensorType(float32, 3D), TensorType(float32, 3D), TensorType(float32, 3D), TensorType(float32, 3D), TensorType(int64, scalar), TensorType(float32, matrix), TensorType(float32, matrix) , TensorType(float32, matrix), TensorType(float32, matrix), TensorType(float32, matrix), TensorType(float32, matrix), TensorType(float32, matrix), TensorType(float32, matrix), TensorType(float32, matr ix)]

Inputs shapes: [(), (500L, 64L, 100L), (500L, 64L, 100L), (500L, 64L, 400L), (500L, 100L, 64L), (500L, 64L, 100L), (500L, 64L, 400L), (500L, 64L, 100L), (500L, 64L, 100L), (501L, 64L, 100L), (501L, 64 L, 100L), (2L, 100L, 400L), (), (100L, 100L), (100L, 100L), (100L, 100L), (100L, 100L), (100L, 100L), (100L, 100L), (100L, 100L), (100L, 400L), (100L, 100L)] Inputs strides: [(), (-25600L, 400L, 4L), (25600L, 400L, 4L), (102400L, 1600L, 4L), (-25600L, 4L, 400L), (25600L, 400L, 4L), (-1600L, 800000L, 4L), (-25600L, 400L, 4L), (-25600L, 400L, 4L), (25600L, 4 00L, 4L), (25600L, 400L, 4L), (160000L, 1600L, 4L), (), (1600L, 4L), (1600L, 4L), (1600L, 4L), (1600L, 4L), (4L, 1600L), (4L, 1600L), (4L, 1600L), (1600L, 4L), (4L, 1600L)] Inputs values: [array(500L, dtype=int64), 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', array(500L, dty pe=int64), 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown', 'not shown'] Outputs clients: [[], [], [Subtensor{int64}(forall_inplace,cpu,grad_of_scan_fn}.2, Constant{1})], [Subtensor{::int64}(forall_inplace,cpu,grad_of_scan_fn}.3, Constant{-1})]]

HINT: Re-running with most Theano optimization disabled could give you a back-trace of when this node was created. This can be done with by setting the Theano flag 'optimizer=fast_compile'. If that do es not work, Theano optimizations can be disabled with 'optimizer=None'.

This also happened to me on my other project, but there problem was dimension mismatch. It couldn't match node with dimensions 300x200 to node with dimension 100x100.

Any help would be really appreciated.

Answer 1

it says what it says. your GPU reached memory limit . If you are using Theano as backend try setting CNMeM and cuDNN. how much memory does your GPU have?