CODEWITHSUNDEEP
multithreadingtensorflowxeon-phi
grok
grok

Reputation: 131

Is training in tensorflow multithreaded?

Suppose I'm trying to train the following network for the cifar-10

https://www.tensorflow.org/tutorials/images/deep_cnn

I would like to know if the basic operations involved in stochastic gradient descent ( or some optimization technique) like computation of gradient etc multithreaded?

More precisely, if I run the above code on a single core machine and on many core machine like Intel Xeon Phi, will it run faster on the many-core machine? ( One can assume one core in both the machines are similar) If yes, what is the exact cause of the speed-up or which computations run faster on the many-core machine?

Upvotes: 0

Views: 1585

Answers (1)

Kaihong Zhang
Kaihong Zhang

Reputation: 419

There are 3 kinds of parallelism in tensorflow.

  • input pipeline: input, preprocessing and network are run in different thread. This can be achieved in tf.QueueRunner or tf.data.Dataset in newer version of tensorflow.
  • inter op parallelism: if one node is independent from another, then they can be executed at the same time. So this depends on the structure of your network.
  • intra op parallelism: one node can use multi-thread, for example, ops implemented with Eigen can use multi-thread in one node.

BTW, inter and intra op parallelism threads appear in Session config, most cpu ops can benefit from it. However, GPU for training is highly recommended because of the high speed.

Upvotes: 1

Related Questions