How to print Accuracy and other metrics in Tensorflow 2.x?

Question

I am working on an object detection project with a custom dataset. My problem is that, it is really difficult to understand how and where should I make changes to evaluate my training set(accuracy, mAP metrics). I am using the tensorflow 2.3.0 on colab and for now I am getting only the loss value as it is shown in the following picture: loss .

Also, this is a picture of my tensorboard: .

To train the model I use the model_main_tf2.py,

!python /content/gdrive/My\ Drive/models/research/object_detection/model_main_tf2.py \
    --pipeline_config_path={pipeline_file} \
    --model_dir={model_dir} \
    --alsologtostderr \
    --num_train_steps={num_steps} \
    --sample_1_of_n_eval_examples=10 \
    --eval_training_data=True \
    --sample_1_of_n_eval_on_train_examples=10 \
    --num_eval_steps={num_eval_steps}

In the configuration file I have: eval_config { metrics_set: "coco_detection_metrics" use_moving_averages: false }

I have tried various things, just like with eval.py (which I read works with tensorflow 1.x), but I encounter a lot of errors or just like with other scripts in the object detection repository from github, object_detection(repository).

The most important thing for now, is the accuracy. I found that the loss is probably defined at the model_lib_v2.py at 845-858 line:

 eval_metrics = {}

  for evaluator in evaluators:
    eval_metrics.update(evaluator.evaluate())
  for loss_key in loss_metrics:
    eval_metrics[loss_key] = loss_metrics[loss_key].result()

  eval_metrics = {str(k): v for k, v in eval_metrics.items()}
  tf.logging.info('Eval metrics at step %d', global_step)
  for k in eval_metrics:
    tf.compat.v2.summary.scalar(k, eval_metrics[k], step=global_step)
    tf.logging.info('	+ %s: %f', k, eval_metrics[k])

  return eval_metrics

but I do not know how to change the code to add accuracy.

In case that helps, I use the ssd_mobilenet_v2_fpnlite_640x640 model and gdrive to load the data and the run the scripts.

Update: The config file I use is the following:

model {
  ssd {
    num_classes: 18
    image_resizer {
      fixed_shape_resizer {
        height: 640
        width: 640
      }
    }
    feature_extractor {
      type: "ssd_mobilenet_v2_fpn_keras"
      depth_multiplier: 1.0
      min_depth: 16
      conv_hyperparams {
        regularizer {
          l2_regularizer {
            weight: 3.9999998989515007e-05
          }
        }
        initializer {
          random_normal_initializer {
            mean: 0.0
            stddev: 0.009999999776482582
          }
        }
        activation: RELU_6
        batch_norm {
          decay: 0.996999979019165
          scale: true
          epsilon: 0.0010000000474974513
        }
      }
      use_depthwise: true
      override_base_feature_extractor_hyperparams: true
      fpn {
        min_level: 3
        max_level: 7
        additional_layer_depth: 128
      }
    }
    box_coder {
      faster_rcnn_box_coder {
        y_scale: 10.0
        x_scale: 10.0
        height_scale: 5.0
        width_scale: 5.0
      }
    }
    matcher {
      argmax_matcher {
        matched_threshold: 0.75
        unmatched_threshold: 0.25
        ignore_thresholds: false
        negatives_lower_than_unmatched: true
        force_match_for_each_row: true
        use_matmul_gather: true
      }
    }
    similarity_calculator {
      iou_similarity {
      }
    }
    box_predictor {
      weight_shared_convolutional_box_predictor {
        conv_hyperparams {
          regularizer {
            l2_regularizer {
              weight: 3.9999998989515007e-05
            }
          }
          initializer {
            random_normal_initializer {
              mean: 0.0
              stddev: 0.009999999776482582
            }
          }
          activation: RELU_6
          batch_norm {
            decay: 0.996999979019165
            scale: true
            epsilon: 0.0010000000474974513
          }
        }
        depth: 128
        num_layers_before_predictor: 4
        kernel_size: 3
        class_prediction_bias_init: -4.599999904632568
        share_prediction_tower: true
        use_depthwise: true
      }
    }
    anchor_generator {
      multiscale_anchor_generator {
        min_level: 3
        max_level: 7
        anchor_scale: 4.0
        aspect_ratios: 1.0
        aspect_ratios: 2.0
        aspect_ratios: 0.5
        scales_per_octave: 2
      }
    }
    post_processing {
      batch_non_max_suppression {
        score_threshold: 9.99999993922529e-09
        iou_threshold: 0.6000000238418579
        max_detections_per_class: 100
        max_total_detections: 100
        use_static_shapes: false
      }
      score_converter: SIGMOID
    }
    normalize_loss_by_num_matches: true
    loss {
      localization_loss {
        weighted_smooth_l1 {
        }
      }
      classification_loss {
        weighted_sigmoid_focal {
          gamma: 2.0
          alpha: 0.25
        }
      }
      classification_weight: 1.0
      localization_weight: 1.0
    }
    encode_background_as_zeros: true
    normalize_loc_loss_by_codesize: true
    inplace_batchnorm_update: true
    freeze_batchnorm: false
  }
}
train_config {
  batch_size: 16
  data_augmentation_options {
    random_horizontal_flip {
    }
  }
  data_augmentation_options {
    random_crop_image {
      min_object_covered: 0.0
      min_aspect_ratio: 0.75
      max_aspect_ratio: 3.0
      min_area: 0.75
      max_area: 1.0
      overlap_thresh: 0.0
    }
  }
  sync_replicas: true
  optimizer {
    momentum_optimizer {
      learning_rate {
        cosine_decay_learning_rate {
          learning_rate_base: 0.07999999821186066
          total_steps: 50000
          warmup_learning_rate: 0.026666000485420227
          warmup_steps: 1000
        }
      }
      momentum_optimizer_value: 0.8999999761581421
    }
    use_moving_average: false
  }
  fine_tune_checkpoint: "/content/gdrive/My Drive/models/research/deploy/ssd_mobilenet_v2_fpnlite_640x640_coco17_tpu-8/checkpoint/ckpt-0"
  num_steps: 20000
  startup_delay_steps: 0.0
  replicas_to_aggregate: 8
  max_number_of_boxes: 1
  unpad_groundtruth_tensors: false
  fine_tune_checkpoint_type: "detection"
  fine_tune_checkpoint_version: V2
}
train_input_reader {
  label_map_path: "/content/gdrive/My Drive/models/research/deploy/label_map.pbtxt"
  tf_record_input_reader {
    input_path: "/content/gdrive/My Drive/models/research/object_detection/data/train.record"
  }
}
eval_config {
  metrics_set: "coco_detection_metrics"
  use_moving_averages: false
}
eval_input_reader {
  label_map_path: "/content/gdrive/My Drive/models/research/deploy/label_map.pbtxt"
  shuffle: false
  num_epochs: 1
  tf_record_input_reader {
    input_path: "/content/gdrive/My Drive/models/research/object_detection/data/test.record"
  }
}

papaya · Accepted Answer

You already seem to have found the solution to your question. Thats great, but you just need to wait a little more to actually get evaludation metrics showing up in your tensorboard.

Start with the defaults and work your way up the training.

First run Training as a dameon:

!python object_detection/model_main_tf2.py \
--pipeline_config_path={pipeline_file} \
--model_dir='object_detection/training' \
--alsologtostderr &

Then run Evaluation on a different console or shell. The evaluation will automatically pick up the newer checkpoints (by default this is at every 1000th step. So you gotta wait for training to reach 1000th step + wait for evaluation to complete 1 epoch or number of images in your test.record)

!python object_detection/model_main_tf2.py \
--pipeline_config_path={pipeline_file} \
--model_dir='object_detection/training' \
--alsologtostderr \
--checkpoint_dir='object_detection/training'

Here's a sample configuration which you can use to visualise.

eval_config: {
  metrics_set: "coco_detection_metrics"
  use_moving_averages: false
  batch_size: 1
  num_visualizations: 10
  max_num_boxes_to_visualize: 5
  visualize_groundtruth_boxes: true
  eval_interval_secs: 30
}

eval_input_reader: {
  label_map_path: "path/to/label_map.pbtxt"
  shuffle: true
  queue_capacity: 100 #depending on your GPU/TPU/CPU
  num_epochs: 1 #what encompasses when to upload results for mAP and AR, if you rather have a number provide that under [num_examples <= test.record total size else error]
  tf_record_input_reader {
    input_path: "path/to/test.record"
  }
}

Quite a few things to note here: TF20 issue

batch_size: 1 #This has to be 1. TF2 throws errors

num_epochs: 1 #provide this as 1

eval_interval_secs: #something based on your dataset & gpu config. Default is 300

num_epochs: 1 #what encompasses when to upload results for mAP and AR, if you rather have a number provide that under [num_examples <= test.record total size else error]

Most importantly, wait. Wait for the evaluation to be complete atleast twice (the first time usually, it basically runs for step 0 which is useless (as your training has to reach a checkpoint (default as mentioned above at 1000th)))

How to print Accuracy and other metrics in Tensorflow 2.x?

Answers (2)

Related Questions