CODEWITHSUNDEEP
androidkotlinqr-codebarcode-scannergoogle-mlkit
iknow
iknow

Reputation: 9862

ML Kit Barcode Scanning doesn't detect QR codes in the photo of the monitor screen

I am using com.google.mlkit:barcode-scanning:17.0.2 to detect QR codes in the pictures.

After getting URI from the gallery I create InputImage and then process this image with BarcodeScanner to find QR codes. When I select a photo of QR codes on paper code is found. But when I take a photo of the QR code on the monitor screen code is never found. What I should do to be able to detect a QR code in a photo of a monitor screen?

(When I use the same scanner with CameraX to do live QR code detection it finds code on the monitor screen)

val image = InputImage.fromFilePath(context, uri)

val scanOptions =
    BarcodeScannerOptions.Builder()
        .setBarcodeFormats(
            Barcode.FORMAT_QR_CODE,
        )
        .build()

val scanner = BarcodeScanning.getClient(scanOptions)

scanner.process(image)
    .addOnSuccessListener {
        val code = it.getOrNull(0)?.rawValue
        if (code == null) {
            // code NOT found
        } else {
            // code was found
        }
    }

Example of QR code on paper which is found

enter image description here

Example of QR code on the monitor screen which is NOT found

enter image description here

Upvotes: 3

Views: 7488

Answers (1)

LSerni
LSerni

Reputation: 57388

Chances are that you're fighting against Moiré effect. Depending on the QR detection algorithm, the high frequencies introduced by the Moiré effect can throw the detector off its track. Frustratingly, it is often the better QRcode detectors that are defeated by Moiré patterns.

A good workaround is:

  • take the picture at the highest resolution you can
  • perform a blurring of the picture
  • increase contrast to the max, if possible
  • (optionally) run a sigma thresholding, or just rewrite all pixels with a luma component below 32 to 0, all those above 224 to 255.

Another way of doing approximately the same operation is

  • take the picture at the highest resolution you can
  • increase contrast to the max, if possible
  • downsample the picture to a resolution which is way lower

The second method gives worse results, but usually can be implemented with device primitives.

Another source of problems with monitors (not in your picture as far as I can see) is the refresh rate. Sometimes, you'll find that the QR code is actually an overexposed QRcode in the upper half of the picture and an underexposed QRcode in the bottom half of the picture. Neither are recognized. This effect is due to the monitor's refresh rate and strategy and is not easy to solve - you can try lowering the monitor's luminosity to increase exposure time, until it exceeds 1/50th or 1/25th of a second, or take the picture from farther away and use digital zooming. Modern monitors have higher refresh rates and actually refresh at more than their own dwell time, so this should not happen; with old analog monitors however it will happen every time.

A third, crazy way

This was discovered half by chance, but it works really well even on cheap hardware provided the QR SDK or library supplies some small extra frills.

  1. Take a video of about 1 second length at the highest frame rate you can get (25 fps?).
  2. From the middle (e.g. 13th) frame, extract the three QR "waypoints" - there might be a low-level function in your SDK called "containsQRCode()" that does this. If it returns true, the waypoints were found and their coordinates are returned to allow performing scaling/estimates. It might return a confidence figure ("this picture seems to contain a QR code with probability X%"). These are the APIs used by apps to show a frame or red dots around candidate QR codes. If your SDK doesn't have these APIs, sorry... you're out of luck.
  3. Get the frames immediately before and after (12th and 14th), then the 11th and 15th, and so on. If any of these returns a valid QR code, you're home free.
  4. If the QR code is found (even if not correctly decoded) in enough frames, but the waypoint coordinates vary much, the hand is not steady - say so to the user.
  5. If you have enough frames with coordinates that vary little, you can center and align on those, and average the frames. Then run the real QRCode recognition on the resulting image. This gets rid of 100% of the Moiré effect, and also drastically reduces monitor dwell noise with next to no information loss. The results are way better than the resolution change, which isn't easy to perform on (some) devices that reset the camera upon resolution change.

This worked on a $19 ESP32 IoT device operating in a noisy, vibration-rich environment (it acquires QR codes from a camera image of carton boxes on a moving transport ribbon).

Upvotes: 3

Related Questions