CODEWITHSUNDEEP
androidandroid-jetpack-composecoordinatesgoogle-mlkitpose-detection
isifzade
isifzade

Reputation: 576

ML-Kit Android pose detection landmark coordinate matching with preview

I am trying to draw points on landmarks by using the result of ML-Kit Pose Detection on Android app with Jetpack Compose.

First, I draw points (red points) by using the pure coordinates that obtained from analyzer. The coordinates are representing the smaller scale of the image and points drawn on left corner of screen.

I observed that the processed image (processImage) size was scaled to 640x480. Here is analyzer code.

class PoseAnalyzer(
    private val onSuccess: (result: AnalyzeResultData) -> Unit,
): ImageAnalysis.Analyzer {
    private val options = PoseDetectorOptions.Builder()
        .setDetectorMode(PoseDetectorOptions.STREAM_MODE)
        .build()

    private val poseDetector = PoseDetection.getClient(options)

    @ExperimentalGetImage
    override fun analyze(imageProxy: ImageProxy) {
        val mediaImage = imageProxy.image
        if (mediaImage != null) {
            val rotationDegrees = imageProxy.imageInfo.rotationDegrees
            val processImage = InputImage.fromMediaImage(mediaImage, rotationDegrees)
            poseDetector.process(processImage)
                .addOnSuccessListener { pose ->
                    val result = AnalyzeResultData(pose, processImage.toSizeData())
                    onSuccess(result)
                }
                .addOnFailureListener {
                }
                .addOnCompleteListener {
                    imageProxy.close()
                }
        }
    }
}

Then I scaled the coordinated according to the preview size (1080x2125) by using below code.

private fun scaleAndGetOffset(
        point: SizeData,
        analyzedImage: SizeData,
        screenSize: SizeData,
        cameraSelector: CameraSelector,
    ): Offset {
        val rateX = screenSize.x/analyzedImage.x
        val rateY = screenSize.y/analyzedImage.y
        return Offset(
            x = if (cameraSelector == CameraSelector.DEFAULT_BACK_CAMERA) point.x * rateX else screenSize.x-point.x * rateX,
            y = point.y * rateY
        )
}

The point were not on correct place.

How get or convert the analyzer coordinates that will match to the preview?

Upvotes: 0

Views: 194

Answers (1)

Thương Nguyễn ngọc
Thương Nguyễn ngọc

Reputation: 1

I have handled a similar issue when mapping coordinates for use with the front camera. Below is my code, which you can apply:

private fun coordinateMapping(
    points: List<PointF>,
    imageHeight: Int,
    scale: Float
): List<PointF> {
    return points.map { point ->
        PointF(
            (imageHeight - point.x) * scale,
            point.y * scale
        )
    }
}

imageWidth = 640, imageHeight = 480 (the image size processed by ML Kit). The x-coordinate is mapped as (imageHeight - point.x) * scale instead of (imageWidth - point.x) * scale because the camera image for ML Kit processing is rotated 90 degrees to the right. For each point, you need to further adjust the mapping with

mappingX = x - (imageHeight * (2125/640) - screenWidth) / 2

, because based on your screen’s aspect ratio, the preview image has been cropped slightly on the width side. Please ask me again if you need more information

Upvotes: 0

Related Questions