Mapping RGB data to values in legend

Question

This is a follow-up to my previous question here

I've been trying to convert the color data in a heatmap to RGB values.

source image

In the below image, to the left is a subplot present in panel D of the source image. This has 6 x 6 cells (6 rows and 6 columns). On the right, we see the binarized image, with white color highlighted in the cell that is clicked after running the code below. The input for running the code is the below image. The ouput is(mean = [ 27.72 26.83 144.17])is the mean of BGR color in the cell that is highlighted in white on the right image below.

A really nice solution that was provided as an answer to my previous question is the following (ref)

import cv2
import numpy as np


# print pixel value on click
def mouse_callback(event, x, y, flags, params):
    if event == cv2.EVENT_LBUTTONDOWN:
        # get specified color
        row = y
        column = x
        color = image[row, column]
        print('color = ', color)

        # calculate range
        thr = 20  # ± color range
        up_thr = color + thr
        up_thr[up_thr < color] = 255
        down_thr = color - thr
        down_thr[down_thr > color] = 0

        # find points in range
        img_thr = cv2.inRange(image, down_thr, up_thr)  # accepted range
        height, width, _ = image.shape
        left_bound = x - (x % round(width/6))
        right_bound = left_bound + round(width/6)
        up_bound = y - (y % round(height/6))
        down_bound = up_bound + round(height/6)
        img_rect = np.zeros((height, width), np.uint8)  # bounded by rectangle
        cv2.rectangle(img_rect, (left_bound, up_bound), (right_bound, down_bound), (255,255,255), -1)
        img_thr = cv2.bitwise_and(img_thr, img_rect)

        # get points around specified point
        img_spec = np.zeros((height, width), np.uint8)  # specified mask
        last_img_spec = np.copy(img_spec)
        img_spec[row, column] = 255
        kernel = np.ones((3,3), np.uint8)  # dilation structuring element
        while cv2.bitwise_xor(img_spec, last_img_spec).any():
            last_img_spec = np.copy(img_spec)
            img_spec = cv2.dilate(img_spec, kernel)
            img_spec = cv2.bitwise_and(img_spec, img_thr)
            cv2.imshow('mask', img_spec)
            cv2.waitKey(10)
        avg = cv2.mean(image, img_spec)[:3]
        mean.append(np.around(np.array(avg), 2))
        print('mean = ', np.around(np.array(avg), 2))
        # print(mean) # appends data to variable mean


if __name__ == '__main__':

    mean = []  #np.zeros((6, 6))
    # create window and callback
    winname = 'img'
    cv2.namedWindow(winname)
    cv2.setMouseCallback(winname, mouse_callback)

    # read & display image
    image = cv2.imread('ip2.png', 1)
    #image = image[3:62, 2:118]  # crop the image to 6x6 cells

    #---- resize image--------------------------------------------------
    # appended this to the original code

    print('Original Dimensions : ', image.shape)

    scale_percent = 220  # percent of original size
    width = int(image.shape[1] * scale_percent / 100)
    height = int(image.shape[0] * scale_percent / 100)
    dim = (width, height)
    # resize image
    image = cv2.resize(image, dim, interpolation=cv2.INTER_AREA)

    # ----------------------------------------------------------------------
    cv2.imshow(winname, image)
    cv2.waitKey()  # press any key to exit
    cv2.destroyAllWindows()

What do I want to do next?

The mean of the RGB values thus obtained has to be mapped to the values in the following legend provided in the source image,

I would like to ask for suggestions on how to map the RGB data to the values in the legend.

Note: In my previous post it has been suggested that one could

fit the RGB values into an equation which gives continuous results.

Any suggestions in this direction will also be helpful.

EDIT: Answering the comment below

I did the following to measure the RGB values of legend Input image:

This image has 8 cells in columns width and 1 cell in rows height

Changed these lines of code:

left_bound = x - (x % round(width/8)) # 6 replaced with 8
right_bound = left_bound + round(width/8) # 6 replaced with 8
up_bound = y - (y % round(height/1)) # 6 replaced with 1
down_bound = up_bound + round(height/1) # 6 replaced with 1

Mean obtained for each cell/ each color in legend from left to right:

mean =  [ 82.15 174.95  33.66]
mean =  [45.55 87.01 17.51]
mean =  [8.88 8.61 5.97]
mean =  [16.79 17.96 74.46]
mean =  [ 35.59  30.53 167.14]
mean =  [ 37.9   32.39 233.74]
mean =  [120.29 118.   240.34]
mean =  [238.33 239.56 248.04]

Answer 1

You can try to apply piece wise approach, make pair wise transitions between colors:

c[i->i+1](t)=t*(R[i+1],G[i+1],B[i+1])+(1-t)*(R[i],G[i],B[i]) 

Do the same for these values:

val[i->i+1](t)=t*val[i+1]+(1-t)*val[i]

Where i - index of color in legend scale, t - parameter in [0:1] range.

So, you have continuous mapping of 2 values, and just need to find color parameters i and t closest to sample and find value from mapping.

Update:

To find the color parameters you can think about every pair of neighbour legend colors as a pair of 3d points, and your queried color as external 3d point. Now you just meed to find a length of perpendicular from the external point to a line, then, iterating over legend color pairs, find the shortest perpendicular (now you have i).

Then find intersection point of the perpendicular and the line. This point will be located at the distance A from line start and if line length is L then parameter value t=A/L.

Update2:

Simple brutforce solution to illustrate piece wise approach:

#include "opencv2/opencv.hpp"
#include <string>
#include <iostream>

using namespace std;
using namespace cv;

int main(int argc, char* argv[])
{
    Mat Image=cv::Mat::zeros(100,250,CV_32FC3);
    std::vector<cv::Scalar> Legend;
    Legend.push_back(cv::Scalar(82.15,174.95,33.66));
    Legend.push_back(cv::Scalar(45.55, 87.01, 17.51));
    Legend.push_back(cv::Scalar(8.88, 8.61, 5.97));
    Legend.push_back(cv::Scalar(16.79, 17.96, 74.46));
    Legend.push_back(cv::Scalar(35.59, 30.53, 167.14));
    Legend.push_back(cv::Scalar(37.9, 32.39, 233.74));
    Legend.push_back(cv::Scalar(120.29, 118., 240.34));
    Legend.push_back(cv::Scalar(238.33, 239.56, 248.04));

    std::vector<float> Values;
    Values.push_back(-4);
    Values.push_back(-2);
    Values.push_back(0);
    Values.push_back(2);
    Values.push_back(4);
    Values.push_back(8);
    Values.push_back(16);
    Values.push_back(32);

    int w = 30;
    int h = 10;

    for (int i = 0; i < Legend.size(); ++i)
    {
        cv::rectangle(Image, Rect(i * w, 0, w, h), Legend[i]/255, -1);
    }

    std::vector<cv::Scalar> Smooth_Legend;
    std::vector<float> Smooth_Values;
    for (int i = 0; i < Legend.size()-1; ++i)
    {
        cv::Scalar c1 = Legend[i];
        cv::Scalar c2 = Legend[i + 1];
        float v1 = Values[i];
        float v2 = Values[i+1];
        for (int j = 0; j < w; ++j)
        {
            float t = (float)j / (float)w;
            Scalar c = c2 * t + c1 * (1 - t);
            float v = v2 * t + v1 * (1 - t);
            float x = i * w + j;
            line(Image, Point(x, h), Point(x, h + h), c/255, 1);
            Smooth_Values.push_back(v);
            Smooth_Legend.push_back(c);
        }
    }

    Scalar qp = cv::Scalar(5, 0, 200);
    float d_min = FLT_MAX;
    int ind = -1;
    for (int i = 0; i < Smooth_Legend.size(); ++i)
    {
        float d = cv::norm(qp- Smooth_Legend[i]);
        if (d < d_min)
        {
            ind = i;
            d_min = d;
        }
    }
    std::cout << Smooth_Values[ind] << std::endl;

    line(Image, Point(ind, 3 * h), Point(ind, 4 * h), Scalar::all(255), 2);
    circle(Image, Point(ind, 4 * h), 3, qp/255,-1);
    putText(Image, std::to_string(Smooth_Values[ind]), Point(ind, 70), FONT_HERSHEY_DUPLEX, 1, Scalar(0, 0.5, 0.5), 0.002);


    cv::imshow("Legend", Image);
    cv::imwrite("result.png", Image*255);
    cv::waitKey();

}

The result:

Python:

import cv2
import numpy as np
height=100
width=250
Image = np.zeros((height, width,3), np.float)
legend =  np.array([ (82.15,174.95,33.66),
          (45.55,87.01,17.51),
          (8.88,8.61,5.97),
          (16.79,17.96,74.46),
          ( 35.59,0.53,167.14),
          ( 37.9,32.39,233.74),
          (120.29,118.,240.34),
          (238.33,239.56,248.04)], np.float)

values = np.array([-4,-2,0,2,4,8,16,32], np.float)

# width of cell, also defines number 
# of one segment transituin subdivisions.
# Larger values will give more accuracy, but will woek slower.
w = 30 
# Only fo displaying purpose. Height of bars in result image.
h = 10


# Plot legend cells ( to check correcrness only )
for i in range(len(legend)):
    col=legend[i]
    cv2.rectangle(Image, (i * w, 0, w, h), col/255, -1)

# Start form smoorhed scales for color and according values
Smooth_Legend=[]
Smooth_Values=[]
for i in range(len(legend)-1): # iterate known knots
    c1 = legend[i] # start color point
    c2 = legend[i + 1] # end color point
    v1 = values[i] # start value 
    v2 = values[i+1] # emd va;ie
    for j in range(w): # slide inside [start:end] interval.
        t = float(j) / float(w) # map it to [0:1] interval
        c = c2 * t + c1 * (1 - t) # transition between c1 and c2
        v = v2 * t + v1 * (1 - t) # transition between v1 and v2
        x = i * w + j # global scale coordinate (for drawing)
        cv2.line(Image, (x, h), (x, h + h), c/255, 1) # draw one tick of smoothed scale
        Smooth_Values.append(v) # append smoothed values for next step
        Smooth_Legend.append(c) # append smoothed color for next step

# queried color    
qp = np.array([5, 0, 200])
# initial value for minimal distance set to large value
d_min = 1e7
# index for clolor search
ind = -1
# search for minimal distance from queried color to smoothed scale color
for i in range(len(Smooth_Legend)):
    # distance
    d = cv2.norm(qp-Smooth_Legend[i])
    if (d < d_min):    
        ind = i
        d_min = d
# ind contains index of the closest color in smoothed scale
# and now we can extract according value from smoothed values scale
print(Smooth_Values[ind]) # value mapped to queried color.
# plot pointer (to check ourself)
cv2.line(Image, (ind, 3 * h), (ind, 4 * h), (255,255,255), 2);
cv2.circle(Image, (ind, 4 * h), 3, qp/255,-1);
cv2.putText(Image, str(Smooth_Values[ind]), (ind, 70), cv2.FONT_HERSHEY_DUPLEX, 1, (0, 0.5, 0.5), 1);
# show window
cv2.imshow("Legend", Image)
# save to file
cv2.imwrite("result.png", Image*255)
cv2.waitKey()