CODEWITHSUNDEEP
tensorflowgpu
sonam agarwal
sonam agarwal

Reputation: 3

Tensorflow gpu error: Dst tensor not initialized

It is my first time training a model on GPU. I am using tensorflow. I am getting an error: InternalError: Failed copying input tensor from /job:localhost/replica:0/task:0/device:CPU:0 to /job:localhost/replica:0/task:0/device:GPU:0 in order to run AssignVariableOp: Dst tensor is not initialized. [Op:AssignVariableOp]

I have tried for solutions like reducing batch size, use tf-nightly but to no avail. I am using Nvidia GeForce GTX 1080 8 Gb. I am trying to train an image classification model using Keras Application(Xception).

Upvotes: 0

Views: 290

Answers (1)

iinquisitiveTechie
iinquisitiveTechie

Reputation: 88

Note: You can get more information about each of suggestions on official guide - https://www.tensorflow.org/guide. I always find this really helpful

Dealing with memory issues is really challenging and you can try one of the suggested approaches to identify and solve the problem.

Try Reduce Batch Size Further But I see that since you are already trying a batch size of one ensure that no other part of your code is increasing the batch size or increasing load on the memory.

Use Mixed Precision Training You are already using mixed precision with mixed_precision.set_global_policy('mixed_float16'). Ensure that your model and optimizer are compatible with mixed precision. If you encounter issues, consider disabling mixed precision temporarily to see if it affects the memory usage.

Clear GPU Memory Before Training You are already using tf.keras.backend.clear_session() to clear the session. You could call this at the beginning of your training code to free up memory before loading the model.

Limit GPU Memory Growth You are already setting memory growth with:

tf.config.experimental.set_memory_growth(gpu, True) Another good approach is to ensure this is code executed before any model processing.

Check for Memory Leaks Ensure that your dataset is not loading all images into memory. Check the return_dataset_image_embedding function to ensure it only processes a batch of images.

Adjusting the Prefetching or Buffer Sizes You are currently using:

patch_dataset = patch_dataset.prefetch(buffer_size=prefetch_buffer_size) Consider increasing the prefetch_buffer_size to a larger number to allow better data pipeline performance.

Profile Memory Usage Use TensorFlow profiling tools to monitor memory usage. You can enable profiling with TensorBoard.

Use tf.data correctly An example - using .cache() after .shuffle() if your dataset fits in memory, which might improve performance.

Some popular tips:

Use nvidia-smi to monitor GPU memory usage at the time of training. Try to incorporate Smaller Input Images VGG16 expects 224x224 input we you can still experiment with smaller sizes. Let me know if this helps else I can help you to fix your code.

Upvotes: 0

Related Questions