convert from saved model to quant. tflite, 'Quantization not yet supported for op: CUSTOM'

Question

I read similar question, Tensorflow (TF2) quantization to full integer error with TFLiteConverter RuntimeError: Quantization not yet supported for op: 'CUSTOM'
However it cannot resolve this at TF 2.4.1.

I referred this tensorflow site to convert using integer-only quantization. https://tensorflow.google.cn/lite/performance/post_training_integer_quant
However, it returns this error:

RuntimeError: Quantization not yet supported for op: 'CUSTOM'.

Code:

import tensorflow as tf
import numpy as np

def representative_data_gen():
   for input_value in tf.data.Dataset.from_tensor_slices(train_images).batch(1).take(100):
       yield [input_value]

converter = tf.lite.TFLiteConverter.from_saved_model(model)

# This enables quantization
converter.optimizations = [tf.lite.Optimize.DEFAULT]
# This ensures that if any ops can't be quantized, the converter throws an error
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
# Set the input and output tensors to uint8
converter.inference_input_type = tf.uint8
converter.inference_output_type = tf.uint8
# set the representative dataset for the converter so we can quantize the activations
converter.representative_dataset = representative_data_gen
tflite_model = converter.convert()
#write the quantized tflite model to a file
with open('my_quant.tflite', 'wb') as f:
  f.write(tflite_model)

How to resolve this issue?
Thanks

Answer 1

The CUSTOM ops normally is the TFLitepostProcess, there are 3 ways working around for this:

**Note that, I use tf 2.4.0 for the workaround, I havent tested on tf 1.15, but the tf.compat.v1 should be the same with it as I quantized by frozen graph protocol

1/ You can remove this op in the tensorflow graph, and then do NMS normally in numpy. You can check here how to delete a node in tensorflow graph

2/ Because the TFLitepostProcess replace NMS, it shouldn't be quantized. If you insist on quantizing it, you should place converter._experimental_new_quantizer = True. However this can lead to poor result as the bounding boxes after NMS activation can vary a lot

3/ Else, you should try to converter with converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8,tf.lite.OpsSet.TFLITE_BUILTINS], this will allow the TFLitepostProcess run in non quantized values

Answer 2

Can you try to use the flags

converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS]
converter.experimental_new_quantizer = True

instead.

"TFLITE_BUILTINS_INT8" indicates a fully quantized op set and we don't have the quantized kernel for the custom op.