CODEWITHSUNDEEP
pythonnumpyfor-loopmultidimensional-arrayensemble-learning
arilwan
arilwan

Reputation: 3973

IndexError: boolean index vs indexed array mismatch in numpy ndarray

I am working on a multiclass problem (5-classes, highly imbalanced dataset). I would like to implement an ensemble of convolutional auto-encoders where each auto-encoder is trained on a single class, and then ensemble to obtain the final classification results.

I am however stuck at a point to train each encoder per class. I'm getting the error that I believe has to do with my logic dealing with arrays of the class labels:

IndexError: boolean index did not match indexed array along dimension 1; dimension is 1 but corresponding boolean dimension is 5

I am working with really huge dataset, but I provide an MWE for a 3-class problem to reproduce similar situation below:

#..scikitlearn, keras, numpy ....libraries import
class SingleAED:

    def __init__(self, train, test):
        self.x_train = train
        self.x_test = test

    def setSingleModel(self):
        autoencoder = Sequential()
        activ = 'relu'
        autoencoder.add(Conv2D(32, (1, 3), strides=(1, 1), padding='same', activation=activ, input_shape=(1, Threshold, 4)))
        autoencoder.add(BatchNormalization(axis = 3))
        autoencoder.add(Conv2D(32, (1, 3), strides=(1, 1), padding='same', activation=activ ))
        autoencoder.add(BatchNormalization(axis = 3))
        autoencoder.add(MaxPooling2D(pool_size=(1, 2) ))

        autoencoder.compile(optimizer='adam', loss='mae', metrics=['mean_squared_error'])

        filepath = "weights.best.hdf5"
        checkpoint = ModelCheckpoint(filepath, monitor='loss', verbose=1, save_best_only=True, mode='max')
        callbacks_list = [checkpoint]

        autoencoder.fit(self.x_train, self.x_train, epochs=250, batch_size=256, shuffle=True,callbacks=callbacks_list)

        return autoencoder

#generate dummy data
X = np.random.randn(20, 1, 5, 4)
a,b,c = np.repeat(0, 7), np.repeat(1, 7), np.repeat(2, 6)
y = np.hstack((a,b,c))

LABELS= list(set(np.ndarray.flatten(y)))
Threshold = len(X[0, 0, :, 0])
NoClass = len(LABELS)

#train-test split
x_train, x_test, y_train, y_test = train_test_split(X, y, 
                                                    test_size=0.20, random_state=7)
#...to categorical
y_train = keras.utils.to_categorical(y_train, num_classes=NoClass)
y_test = keras.utils.to_categorical(y_test, num_classes=NoClass)

#train an auto-encoder per class
ensemble = []
for i in range(len(LABELS)):
    print(LABELS[i])
    sub_train = x_train[y_train == i]
    sub_test = x_test[y_test == i]

    autoencoder = SingleAED(sub_train, sub_test)
    autoencoder = autoencoder.setSingleModel()
    ensemble.append(autoencoder)

Error:

0
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-98-e00f5454d8b5> in <module>()
      2 for i in range(len(LABELS)):
      3     print(LABELS[i])
----> 4     sub_train = x_train[y_train == i]
      5     sub_test = x_test[y_test == i]
      6 

IndexError: boolean index did not match indexed array along dimension 1; dimension is 1 but corresponding boolean dimension is 3

In this case, I want to loop through the classes 0..2 to train an encoder per class. I am not sure why I get this error, can someone help sort this out?

Upvotes: 0

Views: 157

Answers (1)

Michael Gruner
Michael Gruner

Reputation: 640

You want to index the x_train array with y_train before it is converted to categorical.

x_train, x_test, y_train, y_test = train_test_split(X, y, 
                                                    test_size=0.20,
                                                    random_state=7)
# EDIT: DO NOT OVERRIDE!
y_train_cat = keras.utils.to_categorical(y_train, num_classes=NoClass)
y_test_cat = keras.utils.to_categorical(y_test, num_classes=NoClass)

#train an auto-encoder per class
ensemble = []
for i in range(len(LABELS)):
    print(LABELS[I])
    # EDIT: USE NON-CATEGORICAL
    sub_train = x_train[y_train == i]
    sub_test = x_test[y_test == i]

    autoencoder = SingleAED(sub_train, sub_test)
    autoencoder = autoencoder.setSingleModel()
    ensemble.append(autoencoder)

Upvotes: 1

Related Questions