How to blend 80x60 thermal and 640x480 RGB image?

Question

How do I blend two images - thermal(80x60) and RGB(640x480) efficiently? If I scale the thermal to 640x480 it doesn't scale up evenly or doesn't have enough quality to do any processing on it. Any ideas would be really helpful.

RGB image - http://postimg.org/image/66f9hnaj1/

• 

Thermal image - http://postimg.org/image/6g1oxbm5n/

• 

Answer 1

If you scale the resolution of the thermal image up by a factor of 8 and use Bilinear Interpolation you should get a smoother, less-blocky result.

When combining satellite images of different resolution, (I talk about satellite imagery because that is my speciality), you would normally use the highest resolution imagery as the Lightness or L channel to give you apparent resolution and detail in the shapes because the human eye is good at detecting contrast and then use the lower resolution imagery to fill in the Hue and Saturation, or a and b channels to give you the colour graduations you are hoping to see.

So, in concrete terms, I would consider converting the RGB to Lab or HSL colourspace and retaining the L channel. The take the thermal image and up-res it by 8 using bilinear interpolation and use the result as the a, or b or H or S and maybe fill in the remaining channel with the one from the RGB that has the most variance. Then convert the result back to RGB for a false-colour image. It is hard to tell without seeing the images or knowing what you are hoping to find in them. But in general terms, that would be my approach. HTH.

Note: Given that a of Lab colourspace controls the red/green relationship, I would probably try putting the thermal data in that channel so it tends to show more red the "hotter" the thermal channel is.

Updated Answer

Ok, now I can see your images and you have a couple more problems... firstly the images are not aligned, or registered, with each other which is not going to help - try using a tripod ;-) Secondly, your RGB image is very poorly exposed so it is not really going to contribute that much detail - especially in the shadows - to the combined image.

So, firstly, I used ImageMagick at the commandline to up-size the thermal image like this:

convert thermal.png -resize 640x480 thermal.png

Then, I used Photoshop to do a crude alignment/registration. If you want to try this, the easiest way is to put the two images into separate layers of the same document and set the Blending mode of the upper layer to Difference. Then use the Move Tool (shortcut v) to move the upper image around till the screen goes black which means that the details are on top of each other and when subtracted they come to zero, i.e. black. Then crop so the images are aligned and turn off one layer and save, then turn that layer back on and the other layer off and save again.

Now, I used ImageMagick again to separate the two images into Lab layers:

convert bigthermalaligned.png -colorspace Lab -separate thermal.png
convert rgbaligned.png -colorspace Lab -separate rgb.png

which gives me

thermal-0.png => L channel
thermal-1.png => a channel
thermal-2.png => b channel

rgb-0.png => L channel
rgb-1.png => a channel
rgb-2.png => b channel

Now I can take the L channel of the RGB image and the a and b channels of the thermal image and put them together:

convert rgba-0.png thermal-1.png thermal-2.png -normalize -set colorpsace lab -combine result.png

And you get this monstrosity! Obviously you can play around with the channels and colourpsaces and a tripod and proper exposures, but you should be able to see some of the details of the RGB image - especially the curtains on the left, the lights, the camera on the cellphone and the label on the water bottle - have come through into the final image.

Answer 2

Assuming that the images were not captured using a single camera, you need to note that the two cameras may have different parameters. Also, if it's two cameras, they are probably not located in the same world position (offset).

In order to resolve this, you need to get the intrinsic calibration matrix of each of the cameras, and find the offset between them.

Then, you can find a transformation between a pixel in one camera and the other. Unfortunately, if you don't have any depth information about the scene, the most you can do with the calibration matrix is get a ray direction from the camera position to the world.

The easy approach would be to ignore the offset (assuming the scene is not too close to the camera), and just transform the pixel.

p2=K2*(K1^-1 * p1)

Using this you can construct a new image that is a composite of both.

The more difficult approach would be to reconstruct the 3D structure of the scene by finding features that you can match between both images, and then triangulate the point with both rays.