CODEWITHSUNDEEP
pythontensorflowkerasdeep-learningmetrics
Tobias Scheck
Tobias Scheck

Reputation: 633

Keras/Tensorflow calculate mean_iou for batches

I'm trying to calculate the mean_iou and update a confusion matrix for each batch. But after 30 steps I get a SIGKILL event. The images which I use in my generator have the resolution of 2048x1024 because of this my batch_size is 2. It seems that I can't release the memory after one step is finished. I tested the generator while I'm iterating over all images but everything works well.

I'm using Keras 2.1.2 with Tensorflow 1.4.1 as Backend on a GTX 1080. It would be really nice if someone have an advice. 

def calculate_iou_tf(model, generator, steps, num_classes):
    conf_m = K.tf.zeros((num_classes, num_classes), dtype=K.tf.float64)
    generator.reset()
    pb = Progbar(steps)
    for i in range(0, steps):
        x, y_true = generator.next()
        y_pred = model.predict_on_batch(x)

        # num_classes = K.int_shape(y_pred)[-1]
        y_pred = K.flatten(K.argmax(y_pred, axis=-1))
        y_true = K.reshape(y_true, (-1,))

        mask = K.less_equal(y_true, num_classes - 1)
        y_true = K.tf.to_int32(K.tf.boolean_mask(y_true, mask))
        y_pred = K.tf.to_int32(K.tf.boolean_mask(y_pred, mask))

        mIoU, up_op = K.tf.contrib.metrics.streaming_mean_iou(y_pred, y_true, num_classes, updates_collections=[conf_m])
        K.get_session().run(K.tf.local_variables_initializer())
        with K.tf.control_dependencies([up_op]):
            score = K.eval(mIoU)
            print(score)

        pb.update(i + 1)

    conf_m = K.eval(conf_m)
    return conf_m, K.eval(mIoU)

Upvotes: 1

Views: 1848

Answers (1)

Marcin Możejko
Marcin Możejko

Reputation: 40516

The problem lied in using keras.backend functions instead of numpy ones. Every time when a function was called - a new tensor was created. Unfortunately - with the current implementation of tf - there is no a systematic garbage collection of tensors - so this made memory full error. Switching to numpy solved the problem.

Upvotes: 1

Related Questions