CODEWITHSUNDEEP
tensorflow
bxshi
bxshi

Reputation: 2292

Get sum of a batch of model.evaluate results in TensorFlow

In a sentence, I have several customized evaluation metrics, a large set of evaluation data that have to run in batches during evaluation, and I want to get the sum of these metrics of all evaluation batches.

More details:

I build a model using TensorFlow's Estimator class, and it works fine for training and evaluation. The problem is, my evaluation data is too big and I need to split them into batches using a customized input_fn.

The call looks like

eval_res = model.evaluate(input_fn=lambda:eval_input_fn(params['eval_data'], params['batch_size']))

In this case, I'll get the evaluation metric dict (let's call it eval_dict) output every step through the terminal output, but eval_res only has the result of the last evaluation batch.

Let's assume eval_res actually contains a list of eval_dict, then what I want is the sum across all the elements in eval_res that has the same key.

An example is like

eval_dict = {'metric1': 0.0, 'metric2': 0.0}

eval_list = [{'metric1': 0.1, 'metric2': 0.0}, {'metric1': 0.5, 'metric2': 1.0}]

desired_eval_result = {'metric1': 0.6, 'metric2': 1.0}

I tried to store the metric ops into a graph collection using tf.add_to_collection, and then use tf.get_collection to get them. But the result of tf.get_collection is always empty. I suppose I need to specify the graph name to it but I'm not sure how to find the graph of an Estimator class.

Upvotes: 0

Views: 677

Answers (1)

bxshi
bxshi

Reputation: 2292

Every time model.evaluate get called, it will run the model_fn again. So if I create several variables using tf.Variable and update the metric score of each evaluation batch to a variable using assign_add, then I can successfully get the accumulated metric score for all evaluation batches.

And because every time model_fn get called, the variables get reset to 0 by the tf.Variable(0., dtype=tf.float32) call, previous evaluation will not affect the result of next evaluation.

A code sample looks like:

hr_mean_rank_var = tf.Variable(0, name="hr_mean_rank", dtype=tf.float32, trainable=False)
hr_mr_update = hr_mean_rank_var.assign_add(mean_rank_eval_op)
eval_metric_dict = {'hr_mr' : hr_mr_update}

This time when I call 

eval_res = model.evaluate(input_fn=lambda:eval_input_fn(params['eval_data'], params['batch_size']))

I'll get the accumulated metric scores.

Upvotes: 1

Related Questions