CODEWITHSUNDEEP
tensorflowkeraskeras-layer
valentin
valentin

Reputation: 1123

Order of LSTM weights in Keras

I am trying to do a simple evaluation (i.e. forward pass) for a learned LSTM model and I cannot figure out in what order can f_t, i_t, o_t, c_in be extracted from z. It is my understanding that they are computed in bulk. Here is the model architecture obtained using Keras: enter image description here

My input sequence is:

input_seq = np.array([[[0.725323664],
                   [0.7671179],
                   [0.805884672]]])

The output should be:

[ 0.83467698]

Using Keras, I have obtained the following parameters for the first LSTM layer:

lstm_1_kernel_0 = np.array([[-0.40927699, -0.53539848, 0.40065038, -0.07722378, 0.30405849, 0.54959822, -0.23097005, 0.4720422, 0.05197877, -0.52746099, -0.5856396, -0.43691438]])

lstm_1_recurrent_kernel_0 = np.array([[-0.25504839, -0.0823682, 0.11609183,  0.41123426, 0.03409858, -0.0647027, -0.59183347, -0.15359771,  0.21647622,  0.24863823, 0.46169096, -0.21100986],
                                  [0.29160395,  0.46513283,  0.33996364, -0.31195125, -0.24458826, -0.09762905, 0.16202784, -0.01602131, 0.34460208, 0.39724654, 0.31806156, 0.1102117],
                                  [-0.15919448, -0.33053166, -0.22857222, -0.04912394, -0.21862955,  0.55346996, 0.38505834, 0.18110731, 0.270677, -0.02759281, 0.42814475, -0.13496138]])
lstm_1_bias_0 = np.array([0., 0., 0., 1., 1., 1., 0., 0., 0., 0., 0., 0.])

# LSTM 1
z_1_lstm_1 = np.dot(x_1_lstm_1, lstm_1_kernel_0) + np.dot(h_0_lstm_1, lstm_1_recurrent_kernel_0) + lstm_1_bias_0
i_1_lstm_1 = z_1_lstm_1[0, 0:3]
f_1_lstm_1 = z_1_lstm_1[0, 3:6]
input_to_c_1_lstm_1 = z_1_lstm_1[0, 6:9]
o_1_lstm_1 = z_1_lstm_1[0, 9:12]

So the question is what is the correct order for i_1_lstm_1, f_1_lstm_1, input_to_c_1_lstm_1, o_1_lstm_1 ?

Upvotes: 1

Views: 1061

Answers (1)

Yu-Yang
Yu-Yang

Reputation: 14619

It's (i, f, c, o). In recurrent.py, in LSTMCell, the weights are constructed by:

    self.kernel_i = self.kernel[:, :self.units]
    self.kernel_f = self.kernel[:, self.units: self.units * 2]
    self.kernel_c = self.kernel[:, self.units * 2: self.units * 3]
    self.kernel_o = self.kernel[:, self.units * 3:]

    self.recurrent_kernel_i = self.recurrent_kernel[:, :self.units]
    self.recurrent_kernel_f = self.recurrent_kernel[:, self.units: self.units * 2]
    self.recurrent_kernel_c = self.recurrent_kernel[:, self.units * 2: self.units * 3]
    self.recurrent_kernel_o = self.recurrent_kernel[:, self.units * 3:]

    if self.use_bias:
        self.bias_i = self.bias[:self.units]
        self.bias_f = self.bias[self.units: self.units * 2]
        self.bias_c = self.bias[self.units * 2: self.units * 3]
        self.bias_o = self.bias[self.units * 3:]

Upvotes: 1

Related Questions