CODEWITHSUNDEEP
pythonneural-networktheanolstm
Marek Židek
Marek Židek

Reputation: 819

Theano lstm - what is initial hidden state

I'm having trouble understanding this line in a piece of code I've found: 

def has_hidden(layer):
    """
    Whether a layer has a trainable
    initial hidden state.
    """
    return hasattr(layer, 'initial_hidden_state')

My question is what is that initial hidden state? What is its use? Or what is a state of layer? I am familiar with hidden layers, RNNs, LSTMs from papers and videos, but I cannot find anything about this thing. Thanks for help.

Upvotes: 0

Views: 252

Answers (1)

Pranav Rai
Pranav Rai

Reputation: 359

The state of a layer of neurons is the set of all the weights (of its connections) that describe it at that point in time.

To get good training performance its necessary that you dont start off with 0's for all the weights for a layer of neurons. The most common solution to this problems is to initialize all the weights to small but non zero numbers. This would describe the initial state of the neural network.

Upvotes: 1

Related Questions