Training neural networks in parallel, theoretical framework

Question

I'm working with neural networks and currently exploring other ways than my cpu to accelerate the training process. Using Keras with Tensorflow on a GPU machine in AWS I was able to really speed up the process.

How does this parallelization work in theory? (I am not considering cluster parallelization)

I struggled to find a reference so I ask you for an insight into how it works.

Thank you in advance

Answer 1

While I'm not familiar with the relevant implementations, training a neural network means optimizing that neural network's parameters, e.g. the weights of neural connections.

Common optimization methods include quasi-Newton methods, which are rate-limited by matrix math operations like matrix inversion. GPU's can greatly improve calculation speed here.

References:

• "Matrix computations on the GPU", Nvidia (2013-08), discusses how their GPU's can perform large matrix operations in parallel.

• "Using GPUs", TensorFlow, talks about how TensorFlow can be configured to use GPU's.