CODEWITHSUNDEEP
pythontensorflowkeras
the_cheff
the_cheff

Reputation: 5040

Keras seems slower than tensorflow while feeding training data

I am currently converting a project from tensorflow to keras.

Everything seems fine and I am very impressed by how easy it is to build models with keras. However, training is much slower with Keras, where my GPU is significantly less utilized.

I am using a Tensorflow generator Dataset to load my training data. Luckily keras seems to accept that with no problems.

The problem

However while using tensorflow to train on the dataset I archieve an average GPU utilization of ~70%. When I am training the same network with the same dataset generator using Keras I only archieve ~35% GPU utilization

The problem seems to be that I have a very simple network, hence I need to feed data to the GPU as fast as possible since much time is spent here, compared to actually doing backpropagation.

Using tensorflow the key here seemed to be to not use feed-dicts but instead use the tensor from my dataset as input to the graph. Basically, this can be reduced to 

x, y = iterator.get_next()                     # Get the dataset tensors
loss = tf.reduce_sum(tf.square(y - model_out)) # Use the y tensor directly for loss
# Use x as the input layer in my model <- Implememntation omitted

I would like to achieve the same thing with keras, hence I did something like this, where i set x as the input and y as the target tensor. (Can I somehow get rid of having to put y in a list for the target tensor?)

x, y = iterator.get_next()                                    # Get the dataset tensors

model_input = keras.Input(tensor=x)
# Build model with model_input as input layer and something as output layer. <- Implememntation omitted
model = tf.keras.Model(inputs=model_input, outputs=something) # Insert the dataset tensor directly as input

model.compile(loss='mean_squared_error',                                                                                                                   
              optimizer=#something,
              metrics=['accuracy'],                                                                                                                        
              target_tensors=[y]) # Input the dataset y tensor directly for use in the loss calculation

Basically that should set x as the input tensor and y as the tensor used directly for loss, just like in the tensorflow version. I can now cal model.fit without providing x and y arguments explicitly since they are used directly in the graph

model.fit(validation_data=validation_iterator,
          steps_per_epoch=5000,
          validation_steps=1)

To me it seems like I am doing the same thing now with keras and tensorflow, however, keras is way slower with about half the GPU utilization of the pure tensorflow implementation Am I doing something wrong here, or should i just accept this slowdown if I want to use keras?

Upvotes: 0

Views: 374

Answers (1)

elad
elad

Reputation: 31

I experienced the same issue on TensorFlow 1.13 and solved it by upgrading to TensorFlow 1.14 / 2.0.0.

For a sanity check, I wrapped the TensorFlow graph (as is) as a Keras model and trained the model using model.fit(). When using TensorFlow 1.13, I got a slowdown in throughput of 50% relative to the throughput of training the pure TensorFlow implementation. In both cases, I used the same tf.data.dataset input pipeline.

Using TensorFlow version 1.14 solved the issue (now I get the ~same throughput for both cases mentioned above). Later I migrated to TensorFlow 2.0.0 (alpha) and also got the same throughput for both cases.

Upvotes: 1

Related Questions