CODEWITHSUNDEEP
tensorflowkeraspython-3.5
Boppity Bop
Boppity Bop

Reputation: 10463

Keras on GPU - low GPU utilisation

I am training a model using Keras 2.2.4 and python 3.5.3 and Tensorflow on GCP virtual machine with K80 GPU.

GPU utilisation oscillates between 25 and 50% while CPU process with python eats 98%

I assume python is too slow to feed K80 with data.

The code as below.

There are multiple days of data for each epoch. Each day has around 20K samples - number is a bit different for each. Batch size is fixed by variable window_size=900 So I feed it around 19K batches for a day. Batch 0 starts with sample 0 and takes 900 samples, batch 1 starts from sample 1 and takes 900 samples and so on until the day ends.

So I have 3 loops - epoch, days, batches. I feel the epoch and days loops should be preserved for clarity. I don't think they are the problem

I think the most inner loop should be looked at.

The implementation of the inner loop is naïve. Is there some trickery that can make work with arrays faster? 

# d is tuple from groupby - d[0] = date, d[1] = values
for epoch in epochs:
    print('epoch: ', epoch)

    for d in days : 
        print('  day: ', d[0])

        # get arrays for the day
        features = np.asarray(d[1])[:,2:9].astype(dtype = 'float32')
        print(len(features), len(features[0]), features[1].dtype)

        labels = np.asarray(d[1])[:, 9:].astype(dtype = 'int8')
        print(len(labels), len(labels[0]), labels[1].dtype)

        for batch in range(len(features) - window_size):

            # # # can these be optimised?
            fb = features[batch:batch+window_size,:]
            lb = labels[batch:batch+window_size,:]

            fb = fb.reshape(1, fb.shape[0], fb.shape[1])
            lb = lb.reshape(1, lb.shape[0], lb.shape[1])
            # # #

            model.train_on_batch(fb, lb)

        #for batches

        #model.reset_states()

    #for days
#for epoch

Upvotes: 0

Views: 288

Answers (1)

20r25g22e
20r25g22e

Reputation: 97

try wrapping your script with: 

import tensorflow as tf

with tf.device('/device:GPU:0'):
    <your code>

Check out the Tensorflow guide on using GPUs for more information

Upvotes: 1

Related Questions