TensorFlow : slow startup

Question

I use PyCharm and I'm a new one in python.

After 2 days to figure out how tensorflow works I succeed, but the startup time is slow. Everything is ok before the sentence : 'Adding visible gpu device : 0' which takes like 7/8 minutes.

I searched and tried things like the export CUDA_CACHE_MAXSIZE and CUDA_FORCE_PTX_JIT=1 but it doesn't work, I'm a beginner in python specially in tensorflow so I'm looking for a pretty clear solution of course if the solution exists.

So if someone has a solution please let me know I will be grateful for this.

Have a nice day and sorry for my English.

System info :
Windows 10 x64, Gtx 1060, i5, 16Go RAM
Python 3.8.7
Cuda v10.1
Tensorflow 2.2.0
cuDNN 7.6

EDIT : I'm learning from 'freecodchamp' in YouTube so I followed a little bit the beginning code

here is my source code :

import tensorflow as tf
import numpy
import tensorflow_datasets
import os

os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2'

tensor1 = tf.ones([2, 3])  # shape 2x3 
print('shape tensor 1 = ', tf.shape(tensor1))  # Doing some test for learning
tensor2 = tf.reshape(tensor1, [1, 2, 3])
print('shape tensor 2 = ', tf.shape(tensor2))
tensor1 = tf.reshape(tensor1, [1, 1, 1, 1, 1, 6])
print('shape tensor 1 reshaped = ', tf.shape(tensor1))

output :

2021-01-04 10:04:31.355144: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
2021-01-04 10:04:43.401510: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library nvcuda.dll
2021-01-04 10:04:43.498316: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: GeForce GTX 1060 computeCapability: 6.1
coreClock: 1.6705GHz coreCount: 10 deviceMemorySize: 3.00GiB deviceMemoryBandwidth: 178.99GiB/s
2021-01-04 10:04:43.499195: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
2021-01-04 10:04:43.548183: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll
2021-01-04 10:04:43.583967: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cufft64_10.dll
2021-01-04 10:04:43.595863: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library curand64_10.dll
2021-01-04 10:04:43.637187: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusolver64_10.dll
2021-01-04 10:04:43.668089: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusparse64_10.dll
2021-01-04 10:04:43.773909: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2021-01-04 10:04:43.774488: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2021-01-04 10:04:43.779002: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
2021-01-04 10:04:43.820840: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x16676079d90 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2021-01-04 10:04:43.821794: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2021-01-04 10:04:43.824342: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: GeForce GTX 1060 computeCapability: 6.1
coreClock: 1.6705GHz coreCount: 10 deviceMemorySize: 3.00GiB deviceMemoryBandwidth: 178.99GiB/s
2021-01-04 10:04:43.825340: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
2021-01-04 10:04:43.825837: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll
2021-01-04 10:04:43.826331: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cufft64_10.dll
2021-01-04 10:04:43.826816: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library curand64_10.dll
2021-01-04 10:04:43.827303: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusolver64_10.dll
2021-01-04 10:04:43.827801: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusparse64_10.dll
2021-01-04 10:04:43.828297: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2021-01-04 10:04:43.828914: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2021-01-04 10:16:39.045025: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-01-04 10:16:39.045537: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108]      0 
2021-01-04 10:16:39.045836: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0:   N 
2021-01-04 10:16:39.047684: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2095 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060, pci bus id: 0000:01:00.0, compute capability: 6.1)
2021-01-04 10:16:39.056201: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1661e481430 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2021-01-04 10:16:39.056798: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): GeForce GTX 1060, Compute Capability 6.1
shape tensor 1 =  tf.Tensor([2 3], shape=(2,), dtype=int32)
shape tensor 2 =  tf.Tensor([1 2 3], shape=(3,), dtype=int32)
shape tensor 1 reshaped =  tf.Tensor([1 1 1 1 1 6], shape=(6,), dtype=int32)

Process finished with exit code 0

As you can see between line "Adding visible gpu devices :0 " and the next one it took 12 minutes Everything is running ok I just want to run it faster because I can't want that long every time I run this program.

Answer 1

Had the same error with Win10 and tensorflow==2.3. Could be fixed by switching to tensorflow==2.4

Alternatively, you could probably switch to Linux. I have the feeling that it is simply much better supported (and tested) by tensorflow.