CODEWITHSUNDEEP
cudaneural-networktheanodeep-learningconv-neural-network
Cheng Guo
Cheng Guo

Reputation: 353

Why does my dropout function in Theano slow down convolution greatly?

I am learning Theano. I wrote a simple dropout function as below:

import theano.tensor as T
srng = T.shared_randomstreams.RandomStreams()

def drop(input, dropout=0.0):
    if T.gt(dropout, 0.):
        retain_prob = 1 - dropout.astype('floatX')
        mask = srng.binomial(n=1, p=retain_prob, size=input.shape, dtype='floatX')
        return input * mask / retain_prob
    else:
        return input

When I apply this function to the input of the first two convolutional layers, the average time spent on each image increases from 0.5ms to about 2.5ms! Does anyone know what could be the reason for such drastic slow down?

I am using a GTX 980 card with cuDNN installed.

Upvotes: 2

Views: 1129

Answers (2)

Morten Kolbæk
Morten Kolbæk

Reputation: 31

It seems like a similar problem as mine (Lasagne dropoutlayer does not utilize GPU efficiently). Have you checked that you code sets cuda_enabled = True somewhere? otherwise you can manually set it in line 93 of https://github.com/Theano/Theano/blob/master/theano/sandbox/cuda/init.py. I know this is not a elegant solution but it solved my problem for now. :)

Upvotes: 0

Cheng Guo
Cheng Guo

Reputation: 353

RandomStream only works on the CPU. Therefore mask has to be copied from CPU to GPU every time drop is called, which is the reason for the drastic slow down. To avoid this, I now use random stream implementation which works on GPU.

Upvotes: 8

Related Questions