Which layers in a neural network use activation functions?

Question

Does the input layer of a neural network use activation functions, or is it just the hidden and output layers?

Answer 1

Here's a figure of a 3-layer neural network from Stanford's class on NN for visual recognition.

The two hidden layers will always have an activation function. The output layer may or may not have an activation function: for binary classification, you might have a sigmoid function to squash your output, but for regression, you typically will not have an activation function.

For clarity, the hidden layers compute:

output = activation_function(W x inputs + b)

As you probably know, the activation_function() may be a sigmoid, tanh, ReLU, or other.

Answer 2

Hidden and output layer neurons possess activation functions, but input layer neurons do not, input layer just gets input and multiply it with unique weights. Activation functions perform a transformation on the input received.