Overlaping raw thermal image and embedded image from FLIR

Question

I have two images: a thermal image from FLIR e an embedded image which I extract from the metadata embedded in the thermal image. Example:

All metadata available in the thermal image are:

File Name: FLIR2721.jpg
File Size: 1000 kB
File Type: JPEG
File Type Extension: jpg
Mime Type: image/jpeg
Jfif Version: 1.01
Exif Byte Order: Little-endian (Intel, II)
Make: FLIR Systems AB
Model: FLIR T620
Orientation: Horizontal (normal)
X Resolution: 72
Y Resolution: 72
Resolution Unit: inches
Software: 4.8.2
Modify Date: 2017:11:29 08:34:08
Y Cb Cr Positioning: Centered
Exposure Time: 1/46
Exif Version: 220
Create Date: 2017:11:29 08:34:08
Components Configuration: -, Cr, Cb, Y
Subject Distance: 1 m
Focal Length: 41.3 mm
Image Temperature Max: 332
Image Temperature Min: 235
Flashpix Version: 100
Color Spaces: RGB
Exif Image Width: 640
Exif Image Height: 480
Digital Zoom Ratio: 1
Image Unique Id: DF9BB2EBB15922366AB9059A1D8D53A9
Gps Version Id: 2.2.0.0
Gps Altitude Ref: Above Sea Level
Gps Time Stamp: 11:48:03
Gps Satellites: 0
Gps Img Direction Ref: Magnetic North
Gps Img Direction: 160
Compression: JPEG (old-style)
Thumbnail Offset: 2172
Thumbnail Length: 5973
Emissivity: 0.75
Object Distance: 1.00 m
Reflected Apparent Temperature: 20.0 C
Atmospheric Temperature: 28.0 C
Ir Window Temperature: 20.0 C
Ir Window Transmission: 1
Relative Humidity: 60.0 %
Planck R: 115965.437
Planck B: 1417.1
Planck F: 1
Atmospheric Trans Alpha1: 0.006569
Atmospheric Trans Alpha2: 0.01262
Atmospheric Trans Beta1: -0.002276
Atmospheric Trans Beta2: -0.00667
Atmospheric Trans X: 1.9
Camera Temperature Range Max: 150.0 C
Camera Temperature Range Min: -40.0 C
Camera Temperature Max Clip: 160.0 C
Camera Temperature Min Clip: -60.0 C
Camera Temperature Max Warn: 150.0 C
Camera Temperature Min Warn: -40.0 C
Camera Temperature Max Saturated: 180.0 C
Camera Temperature Min Saturated: -60.0 C
Camera Model: FLIR T620
Camera Part Number: 55903-5022 
Camera Serial Number: 55906523
Camera Software: 25.0.0
Lens Model: FOL41
Lens Part Number: T197524
Lens Serial Number: 56702775
Field Of View: 15.0 deg
Planck O: -3949
Planck R: 20.01401533
Raw Value Range Min: 5427
Raw Value Range Max: 56178
Raw Value Median: 13350
Raw Value Range: 10060
Date Time Original: 2017:11:29 08:34:08.131+00:00
Focus Step Count: 5925
Focus Distance: 8.9 m
Frame Rate: 30
Palette Colors: 224
Above Color: 170 128 128
Below Color: 50 128 128
Overflow Color: 67 216 98
Underflow Color: 41 110 240
Isotherm1 Color: 100 128 128
Isotherm2 Color: 100 110 240
Palette Method: 0
Palette Stretch: 2
Palette File Name: \FlashBFS\system\iron.pal
Palette Name: Iron
Palette: (Binary data 672 bytes)
Raw Thermal Image Width: 640
Raw Thermal Image Height: 480
Raw Thermal Image Type: PNG
Raw Thermal Image: (Binary data 362104 bytes)
Real2 Ir: 3.7806735038757
Offset X: -35
Offset Y: -131
Pi Px1: 160
Pi Px2: 479
Pi Py1: 120
Pi Py2: 359
Gps Valid: Yes
Gps Latitude Ref: South
Gps Longitude Ref: West
Gpsdop: 0.53
Gps Map Datum: WGS84
Embedded Image Width: 2592
Embedded Image Height: 1944
Embedded Image Type: JPG
Embedded Image: (Binary data 600954 bytes)
Image Width: 640
Image Height: 480
Encoding Process: Baseline DCT, Huffman coding
Bits Per Sample: 8
Color Components: 3
Y Cb Cr Sub Sampling: YCbCr4:2:0 (2 2)
Gps Altitude: 0 m Above Sea Level
Gps Latitude: 26 deg 18' 1.14" S
Gps Longitude: 48 deg 51' 2.82" W
Gps Position: 26 deg 18' 1.14" S, 48 deg 51' 2.82" W
Image Size: 640x480
Megapixels: 0.307
Peak Spectral Sensitivity: 10.2 um
Shutter Speed: 1/46
Thumbnail Image: (Binary data 5973 bytes)
Focal Length35Efl: 41.3 mm
Category: image
Raw Header: FF D8 FF E0 00 10 4A 46 49 46 00 01 01 00 00 01 00 01 00 00 FF E1 1F BC 45 78 69 66 00 00 49 49 2A 00 08 00 00 00 0B 00 0F 01 02 00 10 00 00 00 92 00 00 00 10 01 02 00 0A 00 00 00 A2 00 00 00

I want to "overlap" the thermal image in the embedded image but I don't know how to do this since the thermal image is not centered with the embedded image.

I've tried to use matchTemplate form OpenCV but the result is not quite good.

Any suggestions about how to solve it?

Answer 1

Have you looked to see if astrophotography stacking software could help, or could be a pointer to a similar technology. Long exposure images of stars are rarely a single exposure, but consist of hundreds of images. Normally the software automatically detects key stars to help align the images, but maybe there is a way to choose key points on each image to tell the software how to carry out the alignment.

Answer 2

I've been playing with Photoshop and the embedded image, i believe this may work:

import cv2
import  flirimageextractor
flir = flirimageextractor.FlirImageExtractor()
flir.process_image('C:\Temp\FLIR14107.jpg',RGB=True)
c = flir.get_metadata('C:\Temp\FLIR1234.jpg')
img = flir.extract_embedded_image()
Scale = c['Real2IR']
OffsetX = c['OffsetX']
OffsetY = c['OffsetY']
PiPX1 = c['PiPX1']
PiPX2 = c['PiPX2']
PiPY1 = c['PiPY1']
PiPY2 = c['PiPY2']
Width = c['EmbeddedImageWidth']
Height = c['EmbeddedImageHeight']
scaled_img = cv2.resize(img.astype('uint8'), (int(Width * Scale), int(Height * 
                         Scale)), interpolation=cv2.INTER_AREA)

ct = [int(scaled_img.shape[1] / 2), int(scaled_img.shape[0] / 2)]
cx = ct[0] + OffsetX * Scale
cy = ct[1] + OffsetY * Scale
img1 = scaled_img[int(cy - Height/2):int(cy + Height/2),
              int(cx - Width/2):int(cx + Width/2)]
cv2.imshow('1',cv2.resize(img1,(PiPX1 + PiPX2,PiPY1 + 
PiPY2),interpolation=cv2.INTER_AREA))
cv2.waitKey(0)

Answer 3

I have found a solution but have no ideia of why it works.

Sample code:

import math
import cv2

emb_img = cv2.imread(<path_to_embedded_image>, cv2.IMREAD_COLOR)

emb_height, emb_width, emb_channels = emb_img.shape
emb_center = {
    'x': math.floor(emb_width / 2),
    'y': math.floor(emb_height / 2)
}

raw_img = cv2.imread(<path_to_raw_image>, cv2.IMREAD_GRAYSCALE)

raw_height, raw_width = raw_img.shape
raw_center = {
    'x': math.floor(raw_width / 2),
    'y': math.floor(raw_height / 2)
}

j = {
    'OffsetX': <some number>,
    'OffsetY': <some number>
}

x = emb_center.get('x') - raw_center.get('x') + j.get('OffsetX') - 45
y = emb_center.get('y') - raw_center.get('y') + j.get('OffsetY') - 45

cropped_img = emb_img[y:y+540, x:x+720, :]

cropped_img = cv2.resize(cropped_img, (640, 480))

cv2.imwrite('path_you_want.jpg', cropped_img)

I have no ideia why to write cropped_img = emb_img[y:y+540, x:x+720, :] instead of cropped_img = emb_img[y:y+480, x:x+680, :] since raw size is 680x540. Another thing, 45 came from where??? I just know it worked.

Answer 4

I think here is all data you need:

Raw Thermal Image Width: 640
Raw Thermal Image Height: 480
Raw Thermal Image Type: PNG
Raw Thermal Image: (Binary data 362104 bytes)
Real2 Ir: 3.7806735038757
Offset X: -35
Offset Y: -131

Real2 Ir value looks like zoom ratio of raw IR image to JPEG photo. 640x480*3.7806735038757=2420x1815 which is very close to 2592x1944. So you just need to test some combinations of what Pi Px/y coordinates and Offset X/Y means. Enlarge IR image (or reduce JPEG), center both images manually in graphics editor and check what X/Y distance between upper left corners of both images is. I think it will be same as some of these values.