CODEWITHSUNDEEP
pythontensorflowkerasneural-networkdeep-learning
maurera
maurera

Reputation: 1659

How to add custom metric to keras? (Percent Mean Absolute Error)

I am trying to add percent mean absolute error (pmae) as a custom metric in keras. This is defined as (MAE divided by the mean absolute y-value multiplied by 100). I have tried:

def pmae(y_true,y_pred):
    return K.mean(K.abs(y_pred - y_true)) / K.mean(K.abs(y_true)) * 100
...
model.compile(loss='mse', optimizer=Adam(),metrics=[pmae])

which runs, but the value is many orders of magnitude off (when I look at model.history.history.pmae)

The working numpy version (on a test sample) is:

y_pred = model.predict(X_test)
pmae = abs(y_pred - y_test).mean() / abs(y_true).mean() * 100

I've also tried adding , axis=-1 to the K.mean() calls, with no improvement (as suggested in other stackoverflow answers). Does anyone know what's wrong?

Resources

  1. The keras website gives mean value of y as an example:
import keras.backend as K

def mean_pred(y_true, y_pred):
    return K.mean(y_pred)

model.compile(optimizer='rmsprop',
              loss='binary_crossentropy',
              metrics=['accuracy', mean_pred])
  1. Others have answered on stackoverflow about other custom metrics (eg Keras custom RMSLE metric and how to implement custom metric in keras?), but the responses there haven't helped me with pmae calculation.

Upvotes: 1

Views: 2861

Answers (1)

IonicSolutions
IonicSolutions

Reputation: 2599

Let's compare your implementation with the mean_absolute_percentage_error in Keras:

def mean_absolute_percentage_error(y_true, y_pred):
    if not K.is_tensor(y_pred):
        y_pred = K.constant(y_pred)
    y_true = K.cast(y_true, y_pred.dtype)
    diff = K.abs((y_true - y_pred) / K.clip(K.abs(y_true),
                                            K.epsilon(),
                                            None))
    return 100. * K.mean(diff, axis=-1)

Based on this, the following should work for your case:

def percent_mean_absolute_error(y_true, y_pred):
    if not K.is_tensor(y_pred):
        y_pred = K.constant(y_pred)
    y_true = K.cast(y_true, y_pred.dtype)
    diff = K.mean(K.abs((y_pred - y_true)) / K.mean(K.clip(K.abs(y_true),
                                                           K.epsilon(),
                                                           None)))
    return 100. * K.mean(diff)

The main difference to your attempt is that here both y_true and y_pred are cast to the same datatype and the denominator is at least K.epsilon() (which is set to 1e-7 by default), so the error does not go to infinity if y_true approaches 0.

Upvotes: 1

Related Questions