Ideas to process challenging image

Question

I'm working with Infra Red image that is an output of a 3D sensor. This sensors project a Infra Red pattern in order to draw a depth map, and, because of this, the IR image has a lot of white spots that reduce its quality. So, I want to process this image to make it smoother in order to make it possible to detect objects laying in the surface.

The original image looks like this:

My objective is to have something like this (which I obtained by blocking the IR projecter with my hand) :

An "open" morphological operation does remove some noise, but I think first there should be some noise removal operation that addresses the white dots.

Any ideas?

I should mention that the algorithm to reduce the noise has to run on real time.

Answer 1

My main point on the noise removal was to have a cleaner image so it would be easier to detect objects. However, as I tried to find a solution for the problem, I realized that it was unrealistic to remove all noise from the image using on-the-fly noise removal algorithms, since most of the image is actually noise.. So I had to find the objects despite those conditions. Here is my aproach

1 - Initial image

2 - Background subtraction followed by opening operation to smooth noise

3 - Binary threshold

4 - Morphological operation close to make sure object has no edge discontinuities (necessary for thin objects)

5 - Fill holes + opening morphological operations to remove small noise blobs

6 - Detection

Answer 2

Is the IR projected pattern fixed or changes over time? In the second case, you could try to take advantage of the movement of the dots.

For instance, you could acquire a sequence of images and assign each pixel of the result image to the minimum (or a very low percentile) value of the sequence.

Edit: here is a Python script you might want to try

Answer 3

The main difficulty in your images is the large radius of the white dots. Median and morphologic filters should be of little help here.

Usually I'm not a big fan of these algorithms, but you seem to have a perfect use case for a decomposition of your images on a functional space with a sketch and an oscillatary component. Basically, these algorithms aim at solving for the cartoon-like image X that approaches the observed image, and that differs from Y only through the removal of some oscillatory texture. You can find a list of related papers and algorithms here.

(Disclaimer: I'm not Jérôme Gilles, but I know him, and I know that most of his algorithms were implemented in plain C, so I think most of them are practical to implement with OpenCV.)

What you can try otherwise, if you want to try simpler implementations first:

• taking the difference between the input image and a blurred version to see if it emphasizes the dots, in which case you have an easy way to find and mark them. The output of this part may be enough, but you may also want to fill the previous place of the dots using inpainting,
• or applying anisotropic diffusion (like the Rudin-Osher-Fatemi equation) to see if the dots disappear. Despite its apparent complexity, this diffusion can be implemented easily and efficiently in OpenCV by applying the algorithms in this paper. TV diffusion can also be used for the inpainting step of the previous item.

Answer 4

A median filter would be my first attempt .... possibly followed by a Gaussian blur. It really depends what you want to do with it afterwards.

For example, here's your original image after a 5x5 median filter and 5x5 Gaussian blur: