CODEWITHSUNDEEP
pythonpython-3.xtensorflowkerastensorflow2.0
Grzegorz Rut
Grzegorz Rut

Reputation: 215

Custom regularization in keras with tf.gather

I'm trying to implement a custom regularizer in keras. The idea is that the scope of the regularization is limited to two columns out of the dataset. Here's a toy dataset:

# dataset
from sklearn.datasets import load_breast_cancer
data = load_breast_cancer()
x = data.data # x.shape() == (569, 30)
y = data.target

Here's how I've written the regularizer:

import tensorflow as tf # tf.__version__ == 2.0.0

class MyRegularizer(tf.keras.regularizers.Regularizer):

    def __init__(self, strength):
        self.strength = strength

    def __call__(self, x):
        # print(tf.shape(x))
        return self.strength * tf.reduce_sum(tf.subtract(tf.gather(params=x,indices=[29],axis=1),
                                                         tf.gather(params=x,indices=[28],axis=1)
                                                        )
                                            )

Here's a toy model:

# model
inputs = tf.keras.layers.Input(shape=x.shape[1])
dense = tf.keras.layers.Dense(1, kernel_regularizer=MyRegularizer(0.01)
                             )(inputs)
model = tf.keras.models.Model(inputs = inputs, outputs = dense)
model.compile(loss='binary_crossentropy')
model.summary()

model.fit(x,y)

The error I get is as follows:

InvalidArgumentError: segment_ids[0] = 28 is out of range [0, 1)

I did try to check the regularizer's output function outside of the model

tf.reduce_sum(tf.subtract(tf.gather(x,[29],axis=1),tf.gather(x,[28],axis=1)))

and it runs fine.

So probably there's a problem with the shape of the tensor that's sent to the regularizer. I have no idea how to solve this (played with variable names, data types, input shapes, all without luck). The model without the regularizer fits without any errors.

The threads on the internet regarding the above-mentioned error revolve around embedding dimensions and I failed to find a solution that would work for me.

Upvotes: 0

Views: 418

Answers (1)

Tou You
Tou You

Reputation: 1194

In your subclass, The argument "x" passed in the 'call()' method is the layer kernel (weights). Since you have one cell in the Dence layer, the "tf.gather" method cant find element of index[28] on the second axis in the kernel.

InvalidArgumentError: segment_ids[0] = 28 is out of range [0, 1)

if you want to get the weight that corresponds to the [28] input ;I think the code below can work (change the axis value to zero):

import tensorflow as tf # tf.__version__ == 2.0.0

class MyRegularizer(tf.keras.regularizers.Regularizer):

def __init__(self, strength):
    self.strength = strength

def __call__(self, x):
    # print(tf.shape(x))
    return self.strength * tf.reduce_sum(tf.subtract(tf.gather(params=x,indices=[29],axis=0),
                                                     tf.gather(params=x,indices=[28],axis=0)
                                                    )
                                        )

Upvotes: 1

Related Questions