Aniket Saxena
Aniket Saxena

Reputation: 23

Seeking Masking support for Dense Layer in Keras

I did a small proof-of-concept to know whether or not Dense layer in Keras supports Masking. Below is my code:-

import keras
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense,Masking,Flatten
import numpy as np

input_shape = (125,)
model = Sequential()
model.add(Masking(mask_value=-1,input_shape=input_shape))
model.add(Dense(16, activation='relu', kernel_initializer='he_uniform'))
model.add(Dense(8, activation='relu', kernel_initializer='he_uniform'))
model.add(Flatten())
model.add(Dense(1, activation='linear'))
model.compile(loss='mean_absolute_error', optimizer='adam', metrics=['mean_absolute_error'])
history = model.fit(x_train, y_train, epochs=5, batch_size=1, verbose=1)

My dataset contains only 2 samples, each having 125 features, which can be seen below:-

[[  1.   2.   3.   4.   5.   6.   7.   8.   9.  10.  11.  12.  13.  14.
   15.  16.  17.  18.  19.  20.  21.  22.  23.  24.  25.  26.  27.  -1.
   -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.
   -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.
   -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.
   -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.
   -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.
   -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.
   -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.]
 [ 82.  83.  84.  85.  86.  87.  88.  89.  90.  91.  92.  93.  94.  95.
   96.  97.  98.  99. 100. 101. 102. 103. 104. 105. 106. 107. 108. 109.
  110. 111. 112. 113. 114. 115. 116. 117. 118. 119. 120. 121. 122. 123.
  124. 125. 126. 127. 128. 129. 130. 131. 132. 133. 134. 135. 136. 137.
  138. 139. 140. 141. 142. 143. 144. 145. 146. 147. 148. 149. 150. 151.
  152. 153. 154. 155. 156. 157. 158. 159. 160. 161. 162.  -1.  -1.  -1.
   -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.
   -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.
   -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.  -1.]]

With Masking layer, I want Dense layer to ignore all the '-1' values while training the network. Hence I wrote model.add(Masking(mask_value=-1,input_shape=input_shape)).

I also headed over to Keras Github code (https://github.com/keras-team/keras/blob/master/keras/layers/core.py) for Dense layer and found that Dense layer class has self.supports_masking = True but I am not sure that it propagates the mask to other layers in the network.

So, my questions are as follows:-

  1. will my network ignore these '-1' values during training time if I use Masking layer and mask these dummy (padded) '-1' values?
  2. will dense layer propagates its mask to other layers in the network so that the network, during training time, will not train on these dummy values?
  3. Since Dense layer in my implementation doesn't throw dense_18 doesn't support masking error, does this mean that it does support Masking?

Any help would be highly appreciated. Thanks...

Upvotes: 2

Views: 2836

Answers (1)

Nathan H
Nathan H

Reputation: 56

Not sure if you ever received an answer, but in case you haven't, you might look at this: Masking Guide

I was just going to look to see if Dense supports masking also. Here is the relevant quote from the linked guide:

If you have a custom layer that does not modify the time dimension, and if you want it to be able to propagate the current input mask, you should set self.supports_masking = True in the layer constructor. In this case, the default behavior of compute_mask() is to just pass the current mask through.

This to me, says that Dense will propagate the mask.

Upvotes: 1

Related Questions