CODEWITHSUNDEEP
androidtensorflowobject-detection
Guido Forrestor
Guido Forrestor

Reputation: 49

Android / TFlite call results in a NPE

we are trying to do object detection in an android app. To do this we used a ssdlite_mobilenet_v2_coco pretrained network and continued training our own dataset.

To train we used Ubuntu 18.04 & TensorFlow 1.11.0 + GPU support. We created the tflite model using these scripts: 

python3 ~/tensorflow/models/research/object_detection/export_tflite_ssd_graph.py \
--pipeline_config_path input/ssdlite_mobilenet_v2_coco/pipeline.config \
--trained_checkpoint_prefix input/ssdlite_mobilenet_v2_coco/model.ckpt-381700 \
--output_directory output/ \
--add_postprocessing_op=true

tflite_convert \
--output_file=output/ssdlite_mobilenet_v2_coco.tflite \
--graph_def_file=input/ssdlite_mobilenet_v2_coco.pb \
--input_arrays=FLOAT \
--output_arrays=concat,concat_1 \
--input_shape=1,300,300,3

What this app basically does, it takes a pre-recorded video, decodes it frame-by-frame using FFmpegMediaMetadataRetriever and passes the bitmap into tflite to detect objects there. The app is built with gradle and we are using 'org.tensorflow:tensorflow-lite:1.12.0', but we basically get the same error with 1.11.

We scale the bitmap down to 300x300 and convert it from ARGB into 3 float channels and call tflite like this:

Log.v(TAG, "Feeding TFLite")
outputLocations = Array(1) { Array(NUM_DETECTIONS) { FloatArray(4) } }
outputClasses = Array(1) { FloatArray(NUM_DETECTIONS) }
outputScores = Array(1) {FloatArray(NUM_DETECTIONS)}
numDetections = FloatArray(1)

val inputArray = arrayOf<Any>(imgData!!)
val outputMap = HashMap<Int, Any>()
outputMap.put(0, outputLocations!!)
outputMap.put(1, outputClasses!!)
outputMap.put(2, outputScores!!)
outputMap.put(3, numDetections!!)


Log.v(TAG, "Running TFLite")
tflite!!.runForMultipleInputsOutputs(inputArray, outputMap)
Log.v(TAG, "Returning from TFLite")

val recognitions = ArrayList<Recognition>(NUM_DETECTIONS)

The error we are getting is:

2019-02-28 11:52:33.486 26807-26879/com.package.xxxxxx A/libc: Fatal signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x0 in tid 26879 (.xxxxxx), pid 26807 (.xxxxxx)
2019-02-28 11:52:33.600 26890-26890/? A/DEBUG: *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
2019-02-28 11:52:33.601 26890-26890/? A/DEBUG: Build fingerprint: 'google/sdk_gphone_x86/generic_x86:9/PSR1.180720.075/5124027:userdebug/dev-keys'
2019-02-28 11:52:33.601 26890-26890/? A/DEBUG: Revision: '0'
2019-02-28 11:52:33.601 26890-26890/? A/DEBUG: ABI: 'x86'
2019-02-28 11:52:33.604 26890-26890/? A/DEBUG: pid: 26807, tid: 26879, name: .xxxxxx  >>> com.package.xxxxxx <<<
2019-02-28 11:52:33.604 26890-26890/? A/DEBUG: signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x0
2019-02-28 11:52:33.604 26890-26890/? A/DEBUG: Cause: null pointer dereference
2019-02-28 11:52:33.604 26890-26890/? A/DEBUG:     eax 00000000  ebx 00000000  ecx 00000000  edx 00000000
2019-02-28 11:52:33.604 26890-26890/? A/DEBUG:     edi c75094a8  esi 00000000
2019-02-28 11:52:33.604 26890-26890/? A/DEBUG:     ebp c75090f8  esp c7509070  eip c757c77b
2019-02-28 11:52:33.606 26890-26890/? A/DEBUG: backtrace:
2019-02-28 11:52:33.606 26890-26890/? A/DEBUG:     #00 pc 0007277b  /data/app/com.package.xxxxxx-yGil4a0ylttBBhPIo26SWA==/lib/x86/libtensorflowlite_jni.so
2019-02-28 11:52:33.606 26890-26890/? A/DEBUG:     #01 pc 00074fe0  /data/app/com.package.xxxxxx-yGil4a0ylttBBhPIo26SWA==/lib/x86/libtensorflowlite_jni.so
2019-02-28 11:52:33.606 26890-26890/? A/DEBUG:     #02 pc 0007590d  /data/app/com.package.xxxxxx-yGil4a0ylttBBhPIo26SWA==/lib/x86/libtensorflowlite_jni.so
2019-02-28 11:52:33.606 26890-26890/? A/DEBUG:     #03 pc 000755b0  /data/app/com.package.xxxxxx-yGil4a0ylttBBhPIo26SWA==/lib/x86/libtensorflowlite_jni.so
2019-02-28 11:52:33.606 26890-26890/? A/DEBUG:     #04 pc 00076322  /data/app/com.package.xxxxxx-yGil4a0ylttBBhPIo26SWA==/lib/x86/libtensorflowlite_jni.so
2019-02-28 11:52:33.606 26890-26890/? A/DEBUG:     #05 pc 0013389c  /data/app/com.package.xxxxxx-yGil4a0ylttBBhPIo26SWA==/lib/x86/libtensorflowlite_jni.so
2019-02-28 11:52:33.607 26890-26890/? A/DEBUG:     #06 pc 00132fa7  /data/app/com.package.xxxxxx-yGil4a0ylttBBhPIo26SWA==/lib/x86/libtensorflowlite_jni.so
2019-02-28 11:52:33.607 26890-26890/? A/DEBUG:     #07 pc 00132e37  /data/app/com.package.xxxxxx-yGil4a0ylttBBhPIo26SWA==/lib/x86/libtensorflowlite_jni.so
2019-02-28 11:52:33.607 26890-26890/? A/DEBUG:     #08 pc 0016550e  /data/app/com.package.xxxxxx-yGil4a0ylttBBhPIo26SWA==/lib/x86/libtensorflowlite_jni.so
2019-02-28 11:52:33.607 26890-26890/? A/DEBUG:     #09 pc 0008f065  /system/lib/libc.so (__pthread_start(void*)+53)
2019-02-28 11:52:33.607 26890-26890/? A/DEBUG:     #10 pc 0002485b  /system/lib/libc.so (__start_thread+75)
2019-02-28 11:52:34.197 1761-1761/? E//system/bin/tombstoned: Tombstone written to: /data/tombstones/tombstone_35

As you can see the NPE occurs deep in the libtensorflow and we are basically running out of ideas what we could do to fix it, so any help is appreciated. It happens on both a physical device and the android sandbox (API 28)

We used this as a starting point and also the Tensorflow tflite demo form the tensorflow repository.

Upvotes: 0

Views: 521

Answers (1)

Guido Forrestor
Guido Forrestor

Reputation: 49

It turns out the input-arrays and output-arrays in the tflite_convert script were wrong.

We managed to successfully (without NullPointer) call tflite using a tflite graph generated by the following command:

tflite_convert \ 
--graph_def_file=tflite_graph.pb \ 
--output_file=output.tflite \ 
--input_shapes=1,300,300,3 \ 
--std_dev_values=128 \ 
--mean_values=128 \ 
--input_arrays=normalized_input_image_tensor \ 
--output_arrays='TFLite_Detection_PostProcess','TFLite_Detection_PostProcess:1','TFLite_Detection_PostProcess:2','TFLite_Detection_PostProcess:3' \ 
--allow_custom_ops

Upvotes: 1

Related Questions