What is K Max Pooling? How to implement it in Keras?

I have to add a k-max pooling layer in CNN model to detect fake reviews. Please can you let me know how to implement it using keras.

I searched the internet but I got no good resources.

Upvotes: 2

Answers (5)

Rania El'Oueslati

Reputation: 47

Here is my implementation of k-max pooling as explained in the comment of @Anubhav Singh above (the order of topk is preserved)

def test60_simple_test(a):
# swap last two dimensions since top_k will be applied along the last dimension
    #shifted_input = tf.transpose(a) #[0, 2, 1]

    # extract top_k, returns two tensors [values, indices]
    res = tf.nn.top_k(a, k=3, sorted=True, name=None)
    b = tf.sort(res[1],axis=0,direction='ASCENDING',name=None)
    e=tf.gather(a,b)
    #e=e[0:3]
    return (e)

a = tf.constant([7, 2, 3, 9, 5], dtype = tf.float64) 
print('*input:',a)
print('**output', test60_simple_test(a))

The result:

*input: tf.Tensor([7. 2. 3. 9. 5.], shape=(5,), dtype=float64)
**output tf.Tensor([7. 9. 5.], shape=(3,), dtype=float64)

Upvotes: 0

Vineet Suryan

Reputation: 101

Check this out. Not thoroughly tested but works fine for me. Let me know what you think. P.S. Latest tensorflow version.

tf.nn.top_k does not preserve the order of occurrence of values. So, that is the think that need to be worked upon

import tensorflow as tf
from tensorflow.keras import layers
class KMaxPooling(layers.Layer):
    """
    K-max pooling layer that extracts the k-highest activations from a sequence (2nd dimension).
    TensorFlow backend.
    """
    def __init__(self, k=1, axis=1, **kwargs):
        super(KMaxPooling, self).__init__(**kwargs)
        self.input_spec = layers.InputSpec(ndim=3)
        self.k = k

        assert axis in [1,2],  'expected dimensions (samples, filters, convolved_values),\
                   cannot fold along samples dimension or axis not in list [1,2]'
        self.axis = axis

        # need to switch the axis with the last elemnet
        # to perform transpose for tok k elements since top_k works in last axis
        self.transpose_perm = [0,1,2] #default
        self.transpose_perm[self.axis] = 2
        self.transpose_perm[2] = self.axis

    def compute_output_shape(self, input_shape):
        input_shape_list = list(input_shape)
        input_shape_list[self.axis] = self.k
        return tuple(input_shape_list)

    def call(self, x):
        # swap sequence dimension to get top k elements along axis=1
        transposed_for_topk = tf.transpose(x, perm=self.transpose_perm)

        # extract top_k, returns two tensors [values, indices]
        top_k_vals, top_k_indices = tf.math.top_k(transposed_for_topk,
                                                  k=self.k, sorted=True,
                                                  name=None)
        # maintain the order of values as in the paper
        # sort indices
        sorted_top_k_ind = tf.sort(top_k_indices)
        flatten_seq = tf.reshape(transposed_for_topk, (-1,))
        shape_seq = tf.shape(transposed_for_topk)
        len_seq = tf.shape(flatten_seq)[0]
        indices_seq = tf.range(len_seq)
        indices_seq = tf.reshape(indices_seq, shape_seq)
        indices_gather = tf.gather(indices_seq, 0, axis=-1)
        indices_sum = tf.expand_dims(indices_gather, axis=-1)
        sorted_top_k_ind += indices_sum
        k_max_out = tf.gather(flatten_seq, sorted_top_k_ind)
        # return back to normal dimension but now sequence dimension has only k elements
        # performing another transpose will get the tensor back to its original shape
        # but will have k as its axis_1 size
        transposed_back = tf.transpose(k_max_out, perm=self.transpose_perm)

        return transposed_back

Upvotes: 0

pandalai

Reputation: 426

Here is a Pytorch version implementation of k-max pooling:

import torch
def kmax_pooling(x, dim, k):
    index = x.topk(k, dim = dim)[1].sort(dim = dim)[0]
    return x.gather(dim, index)

Hope it would help.

Upvotes: -1

CypherX

Reputation: 7353

Keras implementation of K-Max Pooling with TensorFlow Backend

There seems to be a solution here as @Anubhav_Singh suggested. This response got almost 5 times more thumbs up (24) than thumbs down (5) on the github keras issues link. I am just quoting it as-is here and let people try it out and say whether it worked for them or not.

Original author: arbackus

from keras.engine import Layer, InputSpec
from keras.layers import Flatten
import tensorflow as tf

class KMaxPooling(Layer):
    """
    K-max pooling layer that extracts the k-highest activations from a sequence (2nd dimension).
    TensorFlow backend.
    """
    def __init__(self, k=1, **kwargs):
        super().__init__(**kwargs)
        self.input_spec = InputSpec(ndim=3)
        self.k = k

    def compute_output_shape(self, input_shape):
        return (input_shape[0], (input_shape[2] * self.k))

    def call(self, inputs):

        # swap last two dimensions since top_k will be applied along the last dimension
        shifted_input = tf.transpose(inputs, [0, 2, 1])

        # extract top_k, returns two tensors [values, indices]
        top_k = tf.nn.top_k(shifted_input, k=self.k, sorted=True, name=None)[0]

        # return flattened output
        return Flatten()(top_k)

Note: it was reported to be running very slow (though it worked for people).

Upvotes: 2

Anubhav Singh

Reputation: 8699

As per this paper, k-Max Pooling is a pooling operation that is a generalisation of the max pooling over the time dimension used in the Max-TDNN sentence model and different from the local max pooling operations applied in a convolutional network for object recognition (LeCun et al., 1998).

The k-max pooling operation makes it possible to pool the k most active features in p that may be a number of positions apart; it preserves the order of the features, but is insensitive to their specific positions.

There are few resources which show how to implement it in tensorflow or keras:

Upvotes: 5

What is K Max Pooling? How to implement it in Keras?

Answers (5)

Keras implementation of K-Max Pooling with TensorFlow Backend

Related Questions